2019-12-05T04:42:22.518 INFO:root:teuthology version: 1.0.0-e0b5789 2019-12-05T04:42:22.519 DEBUG:teuthology.run:Teuthology command: teuthology /tmp/teuthology-worker.NUBevT.tmp -- --verbose --lock --description fs/thrash/{begin.yaml ceph-thrash/mds.yaml clusters/1-mds-1-client-coloc.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} mount/fuse.yaml msgr-failures/osd-mds-delay.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} supported-random-distros$/{ubuntu_latest.yaml} tasks/cfuse_workunit_snaptests.yaml} --name pdonnell-2019-12-05_02:12:53-fs-wip-pdonnell-testing-20191204.231200-distro-basic-smithi --block --owner scheduled_pdonnell@teuthology --archive /home/teuthworker/archive/pdonnell-2019-12-05_02:12:53-fs-wip-pdonnell-testing-20191204.231200-distro-basic-smithi/4567617 2019-12-05T04:42:22.564 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2019-12-05T04:42:22.628 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/pdonnell-2019-12-05_02:12:53-fs-wip-pdonnell-testing-20191204.231200-distro-basic-smithi/4567617 branch: wip-pdonnell-testing-20191204.231200 description: fs/thrash/{begin.yaml ceph-thrash/mds.yaml clusters/1-mds-1-client-coloc.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} mount/fuse.yaml msgr-failures/osd-mds-delay.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} supported-random-distros$/{ubuntu_latest.yaml} tasks/cfuse_workunit_snaptests.yaml} email: pdonnell@redhat.com first_in_suite: false job_id: '4567617' kernel: kdb: true sha1: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: pdonnell-2019-12-05_02:12:53-fs-wip-pdonnell-testing-20191204.231200-distro-basic-smithi nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: ubuntu os_version: '18.04' overrides: admin_socket: branch: wip-pdonnell-testing-20191204.231200 ceph: cephfs_ec_profile: - m=2 - k=2 - crush-failure-domain=osd conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 global: mon client directed command retry: 5 ms inject delay max: 1 ms inject delay probability: 0.005 ms inject delay type: osd mds ms inject socket failures: 2500 mds: debug mds: 20 debug ms: 1 mds bal frag: true mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug filestore: 20 debug journal: 20 debug ms: 1 debug osd: 25 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 fs: xfs log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - not responding, replacing - \(OSD_SLOW_PING_TIME - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding sha1: c5290805033483ba0cec165c06bde07dae5f19ec ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: sha1: c5290805033483ba0cec165c06bde07dae5f19ec thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-pdonnell-testing-20191204.231200 sha1: c5290805033483ba0cec165c06bde07dae5f19ec owner: scheduled_pdonnell@teuthology priority: 75 repo: git://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - - mon.b - mon.c - mgr.x - mds.b - osd.4 - osd.5 - osd.6 - osd.7 sha1: c5290805033483ba0cec165c06bde07dae5f19ec suite: fs suite_branch: wip-pdonnell-testing-20191204.231200 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_wip-pdonnell-testing-20191204.231200/qa suite_relpath: qa suite_repo: git://git.ceph.com/ceph-ci.git suite_sha1: c5290805033483ba0cec165c06bde07dae5f19ec tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell rpm: - python3-cephfs extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - ceph: null - mds_thrash: null - ceph-fuse: null - workunit: clients: all: - fs/snaps teuthology_branch: master tube: smithi verbose: false worker_log: /home/teuthworker/archive/worker_logs/worker.smithi.9180 2019-12-05T04:42:22.628 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_wip-pdonnell-testing-20191204.231200/qa; will attempt to use it 2019-12-05T04:42:22.630 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_wip-pdonnell-testing-20191204.231200/qa/tasks 2019-12-05T04:42:22.630 INFO:teuthology.run_tasks:Running task internal.check_packages... 2019-12-05T04:42:22.632 INFO:teuthology.task.internal:Checking packages... 2019-12-05T04:42:22.656 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'basic' and ceph hash 'c5290805033483ba0cec165c06bde07dae5f19ec' 2019-12-05T04:42:22.657 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2019-12-05T04:42:22.657 INFO:teuthology.packaging:ref: None 2019-12-05T04:42:22.657 INFO:teuthology.packaging:tag: None 2019-12-05T04:42:22.657 INFO:teuthology.packaging:branch: wip-pdonnell-testing-20191204.231200 2019-12-05T04:42:22.657 INFO:teuthology.packaging:sha1: c5290805033483ba0cec165c06bde07dae5f19ec 2019-12-05T04:42:22.657 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&ref=wip-pdonnell-testing-20191204.231200 2019-12-05T04:42:22.842 INFO:teuthology.task.internal:Found packages for ceph version 15.0.0-8014-gc529080-1bionic 2019-12-05T04:42:22.909 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2019-12-05T04:42:22.993 INFO:teuthology.task.internal:no buildpackages task found 2019-12-05T04:42:22.993 INFO:teuthology.run_tasks:Running task internal.lock_machines... 2019-12-05T04:42:23.076 INFO:teuthology.task.internal.lock_machines:Locking machines... 2019-12-05T04:42:23.076 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2019-12-05T04:42:23.131 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-12-05T04:42:33.150 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-12-05T04:42:43.164 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-12-05T04:42:53.182 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-12-05T04:43:03.198 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-12-05T04:43:13.238 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-12-05T04:43:23.252 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-12-05T04:43:33.269 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-12-05T04:43:43.287 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-12-05T04:43:53.304 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-12-05T04:44:03.328 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-12-05T04:44:13.353 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-12-05T04:44:23.371 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-12-05T04:44:33.390 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-12-05T04:44:43.411 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-12-05T04:44:53.427 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-12-05T04:45:03.450 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-12-05T04:45:13.465 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-12-05T04:45:23.485 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-12-05T04:45:33.502 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-12-05T04:45:43.519 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-12-05T04:45:53.535 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-12-05T04:46:03.553 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-12-05T04:46:13.567 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-12-05T04:46:23.580 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-12-05T04:46:33.598 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-12-05T04:46:43.613 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 3)... 2019-12-05T04:46:53.629 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 3)... 2019-12-05T04:47:03.644 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 1)... 2019-12-05T04:47:13.663 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 1)... 2019-12-05T04:47:23.680 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 1)... 2019-12-05T04:47:33.696 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 1)... 2019-12-05T04:47:43.714 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 3)... 2019-12-05T04:47:53.732 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 3)... 2019-12-05T04:48:03.749 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 3)... 2019-12-05T04:48:13.765 DEBUG:teuthology.lock.ops:lock_many request: {'count': 2, 'locked_by': 'scheduled_pdonnell@teuthology', 'description': '/home/teuthworker/archive/pdonnell-2019-12-05_02:12:53-fs-wip-pdonnell-testing-20191204.231200-distro-basic-smithi/4567617', 'machine_type': 'smithi'} 2019-12-05T04:48:13.780 DEBUG:teuthology.lock.ops:locked ubuntu@smithi040.front.sepia.ceph.com, ubuntu@smithi042.front.sepia.ceph.com 2019-12-05T04:48:13.921 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=, signals=[15]) 2019-12-05T04:48:14.145 INFO:teuthology.provision.fog.smithi042:Scheduling deploy of ubuntu 18.04 2019-12-05T04:48:14.150 INFO:teuthology.provision.fog.smithi040:Scheduling deploy of ubuntu 18.04 2019-12-05T04:48:14.546 INFO:teuthology.orchestra.console:Power off smithi042 2019-12-05T04:48:14.547 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi042.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2019-12-05T04:48:14.585 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi042.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-12-05T04:48:14.644 INFO:teuthology.orchestra.console:Power off smithi040 2019-12-05T04:48:14.644 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi040.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2019-12-05T04:48:14.684 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi040.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-12-05T04:48:14.775 INFO:teuthology.orchestra.console:Power off for smithi042 completed 2019-12-05T04:48:14.876 INFO:teuthology.orchestra.console:Power on smithi042 2019-12-05T04:48:14.877 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi042.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2019-12-05T04:48:14.916 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi042.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-12-05T04:48:18.724 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi040.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-12-05T04:48:18.958 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi042.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-12-05T04:48:22.860 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi040.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-12-05T04:48:23.088 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi042.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-12-05T04:48:23.330 INFO:teuthology.orchestra.console:Power on for smithi042 completed 2019-12-05T04:48:23.434 INFO:teuthology.provision.fog.smithi042:Waiting for deploy to finish 2019-12-05T04:48:27.016 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi040.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-12-05T04:48:27.246 INFO:teuthology.orchestra.console:Power off for smithi040 completed 2019-12-05T04:48:27.347 INFO:teuthology.orchestra.console:Power on smithi040 2019-12-05T04:48:27.348 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi040.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2019-12-05T04:48:27.394 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi040.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-12-05T04:48:31.445 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi040.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-12-05T04:48:35.576 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi040.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-12-05T04:48:35.808 INFO:teuthology.orchestra.console:Power on for smithi040 completed 2019-12-05T04:48:35.912 INFO:teuthology.provision.fog.smithi040:Waiting for deploy to finish 2019-12-05T04:51:09.750 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi042.front.sepia.ceph.com', 'timeout': 60} 2019-12-05T04:51:18.829 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi042.front.sepia.ceph.com', 'timeout': 60} 2019-12-05T04:51:22.244 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi040.front.sepia.ceph.com', 'timeout': 60} 2019-12-05T04:51:27.882 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi042.front.sepia.ceph.com', 'timeout': 60} 2019-12-05T04:51:31.305 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi040.front.sepia.ceph.com', 'timeout': 60} 2019-12-05T04:51:36.937 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi042.front.sepia.ceph.com', 'timeout': 60} 2019-12-05T04:51:40.361 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi040.front.sepia.ceph.com', 'timeout': 60} 2019-12-05T04:51:45.994 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi042.front.sepia.ceph.com', 'timeout': 60} 2019-12-05T04:51:49.421 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi040.front.sepia.ceph.com', 'timeout': 60} 2019-12-05T04:51:55.051 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi042.front.sepia.ceph.com', 'timeout': 60} 2019-12-05T04:51:58.473 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi040.front.sepia.ceph.com', 'timeout': 60} 2019-12-05T04:52:04.110 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi042.front.sepia.ceph.com', 'timeout': 60} 2019-12-05T04:52:07.529 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi040.front.sepia.ceph.com', 'timeout': 60} 2019-12-05T04:52:16.593 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi040.front.sepia.ceph.com', 'timeout': 60} 2019-12-05T04:52:41.647 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi042.front.sepia.ceph.com', 'timeout': 60} 2019-12-05T04:52:41.656 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2019-12-05T04:52:47.659 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi042.front.sepia.ceph.com', 'timeout': 60} 2019-12-05T04:52:47.670 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2019-12-05T04:52:53.679 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi042.front.sepia.ceph.com', 'timeout': 60} 2019-12-05T04:52:53.690 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2019-12-05T04:52:59.700 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi042.front.sepia.ceph.com', 'timeout': 60} 2019-12-05T04:52:59.711 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2019-12-05T04:53:05.721 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi042.front.sepia.ceph.com', 'timeout': 60} 2019-12-05T04:53:05.806 INFO:teuthology.orchestra.run.smithi042:> hostname 2019-12-05T04:53:06.170 INFO:teuthology.orchestra.run.smithi042.stdout:smithi042.front.sepia.ceph.com 2019-12-05T04:53:06.170 INFO:teuthology.orchestra.run.smithi042:> grep smithi042.front.sepia.ceph.com /etc/hosts 2019-12-05T04:53:06.243 INFO:teuthology.orchestra.run.smithi042.stdout:172.21.15.42 smithi042.front.sepia.ceph.com smithi042 2019-12-05T04:53:06.244 INFO:teuthology.orchestra.run.smithi042:> sudo hostname smithi042 2019-12-05T04:53:06.369 INFO:teuthology.orchestra.run.smithi042:> sudo sed -i -e 's/smithi042.front.sepia.ceph.com/smithi042/g' /etc/hosts 2019-12-05T04:53:06.395 INFO:teuthology.orchestra.run.smithi042:> sudo sed -i -e 's/172.21.15.42/172.21.15.42/g' /etc/hosts 2019-12-05T04:53:06.476 INFO:teuthology.orchestra.run.smithi042:> sudo sed -i -e 's/smithi042.front.sepia.ceph.com/smithi042/g' /etc/hostname 2019-12-05T04:53:06.564 INFO:teuthology.orchestra.run.smithi042:> sudo hostname smithi042 2019-12-05T04:53:06.648 INFO:teuthology.provision.fog.smithi042:Deploy complete! 2019-12-05T04:53:06.650 DEBUG:teuthology.parallel:result is None 2019-12-05T04:53:22.596 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi040.front.sepia.ceph.com', 'timeout': 60} 2019-12-05T04:53:22.684 INFO:teuthology.orchestra.run.smithi040:> hostname 2019-12-05T04:53:23.107 INFO:teuthology.orchestra.run.smithi040.stdout:smithi040.front.sepia.ceph.com 2019-12-05T04:53:23.107 INFO:teuthology.orchestra.run.smithi040:> grep smithi040.front.sepia.ceph.com /etc/hosts 2019-12-05T04:53:23.178 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.15.40 smithi040.front.sepia.ceph.com smithi040 2019-12-05T04:53:23.179 INFO:teuthology.orchestra.run.smithi040:> sudo hostname smithi040 2019-12-05T04:53:23.307 INFO:teuthology.orchestra.run.smithi040:> sudo sed -i -e 's/smithi040.front.sepia.ceph.com/smithi040/g' /etc/hosts 2019-12-05T04:53:23.324 INFO:teuthology.orchestra.run.smithi040:> sudo sed -i -e 's/172.21.15.40/172.21.15.40/g' /etc/hosts 2019-12-05T04:53:23.385 INFO:teuthology.orchestra.run.smithi040:> sudo sed -i -e 's/smithi040.front.sepia.ceph.com/smithi040/g' /etc/hostname 2019-12-05T04:53:23.481 INFO:teuthology.orchestra.run.smithi040:> sudo hostname smithi040 2019-12-05T04:53:23.538 INFO:teuthology.provision.fog.smithi040:Deploy complete! 2019-12-05T04:53:23.540 DEBUG:teuthology.parallel:result is None 2019-12-05T04:53:23.736 INFO:teuthology.lock.ops:Checking smithi042.front.sepia.ceph.com 2019-12-05T04:53:23.736 INFO:teuthology.lock.ops:New key found. Updating... 2019-12-05T04:53:23.823 INFO:teuthology.lock.ops:Checking smithi040.front.sepia.ceph.com 2019-12-05T04:53:23.823 INFO:teuthology.lock.ops:New key found. Updating... 2019-12-05T04:53:23.889 INFO:teuthology.lock.ops:Updating [smithi042.front.sepia.ceph.com]: set os type and version on server 2019-12-05T04:53:23.889 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2019-12-05T04:53:23.890 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi042.front.sepia.ceph.com', 'timeout': 60} 2019-12-05T04:53:23.974 INFO:teuthology.orchestra.run.smithi042:> true 2019-12-05T04:53:24.254 INFO:teuthology.orchestra.run.smithi042:> uname -m 2019-12-05T04:53:24.325 INFO:teuthology.orchestra.run.smithi042.stdout:x86_64 2019-12-05T04:53:24.325 INFO:teuthology.orchestra.run.smithi042:> cat /etc/os-release 2019-12-05T04:53:24.400 INFO:teuthology.orchestra.run.smithi042.stdout:NAME="Ubuntu" 2019-12-05T04:53:24.401 INFO:teuthology.orchestra.run.smithi042.stdout:VERSION="18.04.3 LTS (Bionic Beaver)" 2019-12-05T04:53:24.401 INFO:teuthology.orchestra.run.smithi042.stdout:ID=ubuntu 2019-12-05T04:53:24.401 INFO:teuthology.orchestra.run.smithi042.stdout:ID_LIKE=debian 2019-12-05T04:53:24.401 INFO:teuthology.orchestra.run.smithi042.stdout:PRETTY_NAME="Ubuntu 18.04.3 LTS" 2019-12-05T04:53:24.402 INFO:teuthology.orchestra.run.smithi042.stdout:VERSION_ID="18.04" 2019-12-05T04:53:24.402 INFO:teuthology.orchestra.run.smithi042.stdout:HOME_URL="https://www.ubuntu.com/" 2019-12-05T04:53:24.402 INFO:teuthology.orchestra.run.smithi042.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2019-12-05T04:53:24.402 INFO:teuthology.orchestra.run.smithi042.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2019-12-05T04:53:24.402 INFO:teuthology.orchestra.run.smithi042.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2019-12-05T04:53:24.403 INFO:teuthology.orchestra.run.smithi042.stdout:VERSION_CODENAME=bionic 2019-12-05T04:53:24.403 INFO:teuthology.orchestra.run.smithi042.stdout:UBUNTU_CODENAME=bionic 2019-12-05T04:53:24.404 INFO:teuthology.lock.ops:Updating smithi042.front.sepia.ceph.com on lock server 2019-12-05T04:53:24.434 INFO:teuthology.lock.ops:Updating [smithi040.front.sepia.ceph.com]: set os type and version on server 2019-12-05T04:53:24.435 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2019-12-05T04:53:24.435 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi040.front.sepia.ceph.com', 'timeout': 60} 2019-12-05T04:53:24.527 INFO:teuthology.orchestra.run.smithi040:> true 2019-12-05T04:53:24.770 INFO:teuthology.orchestra.run.smithi040:> uname -m 2019-12-05T04:53:24.840 INFO:teuthology.orchestra.run.smithi040.stdout:x86_64 2019-12-05T04:53:24.841 INFO:teuthology.orchestra.run.smithi040:> cat /etc/os-release 2019-12-05T04:53:24.930 INFO:teuthology.orchestra.run.smithi040.stdout:NAME="Ubuntu" 2019-12-05T04:53:24.931 INFO:teuthology.orchestra.run.smithi040.stdout:VERSION="18.04.3 LTS (Bionic Beaver)" 2019-12-05T04:53:24.931 INFO:teuthology.orchestra.run.smithi040.stdout:ID=ubuntu 2019-12-05T04:53:24.931 INFO:teuthology.orchestra.run.smithi040.stdout:ID_LIKE=debian 2019-12-05T04:53:24.931 INFO:teuthology.orchestra.run.smithi040.stdout:PRETTY_NAME="Ubuntu 18.04.3 LTS" 2019-12-05T04:53:24.931 INFO:teuthology.orchestra.run.smithi040.stdout:VERSION_ID="18.04" 2019-12-05T04:53:24.932 INFO:teuthology.orchestra.run.smithi040.stdout:HOME_URL="https://www.ubuntu.com/" 2019-12-05T04:53:24.932 INFO:teuthology.orchestra.run.smithi040.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2019-12-05T04:53:24.936 INFO:teuthology.orchestra.run.smithi040.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2019-12-05T04:53:24.936 INFO:teuthology.orchestra.run.smithi040.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2019-12-05T04:53:24.936 INFO:teuthology.orchestra.run.smithi040.stdout:VERSION_CODENAME=bionic 2019-12-05T04:53:24.936 INFO:teuthology.orchestra.run.smithi040.stdout:UBUNTU_CODENAME=bionic 2019-12-05T04:53:24.937 INFO:teuthology.lock.ops:Updating smithi040.front.sepia.ceph.com on lock server 2019-12-05T04:53:24.968 INFO:teuthology.task.internal.lock_machines:2 smithi machines locked this try, 2/2 locked so far 2019-12-05T04:53:25.005 INFO:teuthology.task.internal.lock_machines:Locked targets: smithi040.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCgyJ01L5AbcRj/1XTjdYFJIHzwMpnOxBZONLQTXqlhq7lTSfDfiDodGztGQJajHviERj+qqE3vO3E2x/P4ZU/5IT/LJv50GUX9BdxIIGuuynvuvR3O4HMkop0bQsDgw4a+v8zt37zDNjCytnTl6Y0aNBnnauTeyzzx8XCbIkVH57yJls5rMIZoie6zgVfE/UL9aQcrsK5nhq9TLY5i0RLGuE2QTYnNfqm1/t5qdnfBNjQSp6ISqMi46VLqiKOjU6TyYT8fz1zQHZ8TtMIPHTf63z2yJ/KNgMZdAMzNehRjmvpzMD0FKJQSSeop8QREHqaamxu76t1F6npDEIlKsBOF smithi042.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDMZ4TtLKXWjwgZ56TLIpfpPIWboZCfXhbjBb/Qn2NiHen0ZAuEBJ8bNv4Slz4g63tJrnNOqlGQITIsdjbLZ6BdoOpeYWlGQDSZzSLvX6WxTtQpdl+F7OthfarPxPX8MZ1Lhd1wsPG5ZYnuuFgg9yrsPqbCr/1G5QbVU+cTl0FB1F7aMwEcpUiE6TV8y2/qTJZbCAR3NnSFjYUWUR6PVmYFg3fWT33wz0h3cZIHuEx1HTORW0f6XvJ543BNSlik9/7k67PuxHo4HIZkohNLUHHV3sYGS9yO7zP4tdD9B9MfjRG9lHpyoC0L1J55CFASYoZnxblDrh6ETrGW21tU83hd 2019-12-05T04:53:25.005 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2019-12-05T04:53:25.050 INFO:teuthology.run_tasks:Running task internal.save_config... 2019-12-05T04:53:25.786 INFO:teuthology.task.internal:Saving configuration 2019-12-05T04:53:25.804 INFO:teuthology.run_tasks:Running task internal.check_lock... 2019-12-05T04:53:25.820 INFO:teuthology.task.internal.check_lock:Checking locks... 2019-12-05T04:53:25.836 DEBUG:teuthology.task.internal.check_lock:machine status is {u'is_vm': False, u'locked': True, u'locked_since': u'2019-12-05 04:48:13.770413', u'locked_by': u'scheduled_pdonnell@teuthology', u'up': True, u'mac_address': None, u'name': u'smithi042.front.sepia.ceph.com', u'os_version': u'18.04', u'machine_type': u'smithi', u'vm_host': None, u'os_type': u'ubuntu', u'arch': u'x86_64', u'ssh_pub_key': u'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGj6VaMgFTSBuojE9mMI2piqDCVUAO1VmFZZBeMQlCOa', u'description': u'/home/teuthworker/archive/pdonnell-2019-12-05_02:12:53-fs-wip-pdonnell-testing-20191204.231200-distro-basic-smithi/4567617'} 2019-12-05T04:53:25.850 DEBUG:teuthology.task.internal.check_lock:machine status is {u'is_vm': False, u'locked': True, u'locked_since': u'2019-12-05 04:48:13.770433', u'locked_by': u'scheduled_pdonnell@teuthology', u'up': True, u'mac_address': None, u'name': u'smithi040.front.sepia.ceph.com', u'os_version': u'18.04', u'machine_type': u'smithi', u'vm_host': None, u'os_type': u'ubuntu', u'arch': u'x86_64', u'ssh_pub_key': u'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAILLTgUfR5wrYbiXfqoPSw/j+nLED7OngRkIcExXDEcdd', u'description': u'/home/teuthworker/archive/pdonnell-2019-12-05_02:12:53-fs-wip-pdonnell-testing-20191204.231200-distro-basic-smithi/4567617'} 2019-12-05T04:53:25.850 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2019-12-05T04:53:25.859 INFO:teuthology.task.internal:roles: ubuntu@smithi042.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2019-12-05T04:53:25.860 INFO:teuthology.task.internal:roles: ubuntu@smithi040.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2019-12-05T04:53:25.860 INFO:teuthology.run_tasks:Running task console_log... 2019-12-05T04:53:26.021 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=, signals=[15]) 2019-12-05T04:53:26.021 INFO:teuthology.run_tasks:Running task internal.connect... 2019-12-05T04:53:26.744 INFO:teuthology.task.internal:Opening connections... 2019-12-05T04:53:26.744 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi042.front.sepia.ceph.com 2019-12-05T04:53:26.745 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi042.front.sepia.ceph.com', 'timeout': 60} 2019-12-05T04:53:26.853 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi040.front.sepia.ceph.com 2019-12-05T04:53:26.855 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi040.front.sepia.ceph.com', 'timeout': 60} 2019-12-05T04:53:26.993 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2019-12-05T04:53:27.328 INFO:teuthology.orchestra.run.smithi042:> uname -m 2019-12-05T04:53:27.336 INFO:teuthology.orchestra.run.smithi042.stdout:x86_64 2019-12-05T04:53:27.337 INFO:teuthology.orchestra.run.smithi042:> cat /etc/os-release 2019-12-05T04:53:27.408 INFO:teuthology.orchestra.run.smithi042.stdout:NAME="Ubuntu" 2019-12-05T04:53:27.408 INFO:teuthology.orchestra.run.smithi042.stdout:VERSION="18.04.3 LTS (Bionic Beaver)" 2019-12-05T04:53:27.408 INFO:teuthology.orchestra.run.smithi042.stdout:ID=ubuntu 2019-12-05T04:53:27.408 INFO:teuthology.orchestra.run.smithi042.stdout:ID_LIKE=debian 2019-12-05T04:53:27.409 INFO:teuthology.orchestra.run.smithi042.stdout:PRETTY_NAME="Ubuntu 18.04.3 LTS" 2019-12-05T04:53:27.409 INFO:teuthology.orchestra.run.smithi042.stdout:VERSION_ID="18.04" 2019-12-05T04:53:27.409 INFO:teuthology.orchestra.run.smithi042.stdout:HOME_URL="https://www.ubuntu.com/" 2019-12-05T04:53:27.409 INFO:teuthology.orchestra.run.smithi042.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2019-12-05T04:53:27.409 INFO:teuthology.orchestra.run.smithi042.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2019-12-05T04:53:27.409 INFO:teuthology.orchestra.run.smithi042.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2019-12-05T04:53:27.410 INFO:teuthology.orchestra.run.smithi042.stdout:VERSION_CODENAME=bionic 2019-12-05T04:53:27.410 INFO:teuthology.orchestra.run.smithi042.stdout:UBUNTU_CODENAME=bionic 2019-12-05T04:53:27.410 INFO:teuthology.lock.ops:Updating smithi042.front.sepia.ceph.com on lock server 2019-12-05T04:53:27.439 INFO:teuthology.orchestra.run.smithi040:> uname -m 2019-12-05T04:53:27.449 INFO:teuthology.orchestra.run.smithi040.stdout:x86_64 2019-12-05T04:53:27.450 INFO:teuthology.orchestra.run.smithi040:> cat /etc/os-release 2019-12-05T04:53:27.521 INFO:teuthology.orchestra.run.smithi040.stdout:NAME="Ubuntu" 2019-12-05T04:53:27.521 INFO:teuthology.orchestra.run.smithi040.stdout:VERSION="18.04.3 LTS (Bionic Beaver)" 2019-12-05T04:53:27.521 INFO:teuthology.orchestra.run.smithi040.stdout:ID=ubuntu 2019-12-05T04:53:27.521 INFO:teuthology.orchestra.run.smithi040.stdout:ID_LIKE=debian 2019-12-05T04:53:27.521 INFO:teuthology.orchestra.run.smithi040.stdout:PRETTY_NAME="Ubuntu 18.04.3 LTS" 2019-12-05T04:53:27.522 INFO:teuthology.orchestra.run.smithi040.stdout:VERSION_ID="18.04" 2019-12-05T04:53:27.522 INFO:teuthology.orchestra.run.smithi040.stdout:HOME_URL="https://www.ubuntu.com/" 2019-12-05T04:53:27.522 INFO:teuthology.orchestra.run.smithi040.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2019-12-05T04:53:27.522 INFO:teuthology.orchestra.run.smithi040.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2019-12-05T04:53:27.522 INFO:teuthology.orchestra.run.smithi040.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2019-12-05T04:53:27.522 INFO:teuthology.orchestra.run.smithi040.stdout:VERSION_CODENAME=bionic 2019-12-05T04:53:27.522 INFO:teuthology.orchestra.run.smithi040.stdout:UBUNTU_CODENAME=bionic 2019-12-05T04:53:27.523 INFO:teuthology.lock.ops:Updating smithi040.front.sepia.ceph.com on lock server 2019-12-05T04:53:27.537 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2019-12-05T04:53:27.611 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2019-12-05T04:53:27.652 INFO:teuthology.task.internal:Checking for old test directory... 2019-12-05T04:53:27.652 INFO:teuthology.orchestra.run.smithi040:> test '!' -e /home/ubuntu/cephtest 2019-12-05T04:53:27.657 INFO:teuthology.orchestra.run.smithi042:> test '!' -e /home/ubuntu/cephtest 2019-12-05T04:53:27.663 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2019-12-05T04:53:27.685 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2019-12-05T04:53:27.685 INFO:teuthology.orchestra.run.smithi040:> test -z $(ls -A /var/lib/ceph) 2019-12-05T04:53:27.703 INFO:teuthology.orchestra.run.smithi042:> test -z $(ls -A /var/lib/ceph) 2019-12-05T04:53:27.727 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2019-12-05T04:53:27.766 INFO:teuthology.run_tasks:Running task kernel... 2019-12-05T04:53:29.455 DEBUG:teuthology.task.kernel:normalized config {'mon.a': {'sha1': 'distro', 'kdb': True}, 'mon.c': {'sha1': 'distro', 'kdb': True}, 'client.0': {'sha1': 'distro', 'kdb': True}, 'mds.a': {'sha1': 'distro', 'kdb': True}, 'mds.b': {'sha1': 'distro', 'kdb': True}, 'osd.1': {'sha1': 'distro', 'kdb': True}, 'osd.0': {'sha1': 'distro', 'kdb': True}, 'osd.3': {'sha1': 'distro', 'kdb': True}, 'osd.2': {'sha1': 'distro', 'kdb': True}, 'osd.5': {'sha1': 'distro', 'kdb': True}, 'osd.4': {'sha1': 'distro', 'kdb': True}, 'osd.7': {'sha1': 'distro', 'kdb': True}, 'osd.6': {'sha1': 'distro', 'kdb': True}, 'mon.b': {'sha1': 'distro', 'kdb': True}, 'mgr.y': {'sha1': 'distro', 'kdb': True}, 'mgr.x': {'sha1': 'distro', 'kdb': True}} 2019-12-05T04:53:29.455 INFO:teuthology.task.kernel:config {'mon.a': {'sha1': 'distro', 'kdb': True}, 'mon.b': {'sha1': 'distro', 'kdb': True}}, timeout 300 2019-12-05T04:53:29.455 INFO:teuthology.orchestra.run.smithi042:> uname -r 2019-12-05T04:53:29.464 INFO:teuthology.orchestra.run.smithi042.stdout:4.15.0-66-generic 2019-12-05T04:53:29.464 INFO:teuthology.task.kernel:Running kernel on smithi042: 4.15.0-66-generic 2019-12-05T04:53:29.465 INFO:teuthology.orchestra.run.smithi042:> sudo apt-get clean 2019-12-05T04:53:29.589 INFO:teuthology.orchestra.run.smithi042:> sudo apt-get update 2019-12-05T04:53:29.730 INFO:teuthology.orchestra.run.smithi042.stdout:Get:1 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB] 2019-12-05T04:53:29.856 INFO:teuthology.orchestra.run.smithi042.stdout:Hit:2 http://archive.ubuntu.com/ubuntu bionic InRelease 2019-12-05T04:53:29.937 INFO:teuthology.orchestra.run.smithi042.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB] 2019-12-05T04:53:29.991 INFO:teuthology.orchestra.run.smithi042.stdout:Get:4 http://security.ubuntu.com/ubuntu bionic-security/main i386 Packages [413 kB] 2019-12-05T04:53:30.065 INFO:teuthology.orchestra.run.smithi042.stdout:Get:5 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages [590 kB] 2019-12-05T04:53:30.085 INFO:teuthology.orchestra.run.smithi042.stdout:Get:6 http://security.ubuntu.com/ubuntu bionic-security/main Translation-en [194 kB] 2019-12-05T04:53:30.090 INFO:teuthology.orchestra.run.smithi042.stdout:Get:7 http://security.ubuntu.com/ubuntu bionic-security/restricted amd64 Packages [14.6 kB] 2019-12-05T04:53:30.091 INFO:teuthology.orchestra.run.smithi042.stdout:Get:8 http://security.ubuntu.com/ubuntu bionic-security/restricted Translation-en [4,600 B] 2019-12-05T04:53:30.091 INFO:teuthology.orchestra.run.smithi042.stdout:Get:9 http://security.ubuntu.com/ubuntu bionic-security/universe i386 Packages [600 kB] 2019-12-05T04:53:30.100 INFO:teuthology.orchestra.run.smithi042.stdout:Get:10 http://security.ubuntu.com/ubuntu bionic-security/universe amd64 Packages [626 kB] 2019-12-05T04:53:30.116 INFO:teuthology.orchestra.run.smithi042.stdout:Get:11 http://security.ubuntu.com/ubuntu bionic-security/universe Translation-en [209 kB] 2019-12-05T04:53:30.116 INFO:teuthology.orchestra.run.smithi042.stdout:Get:12 http://security.ubuntu.com/ubuntu bionic-security/multiverse i386 Packages [4,292 B] 2019-12-05T04:53:30.117 INFO:teuthology.orchestra.run.smithi042.stdout:Get:13 http://security.ubuntu.com/ubuntu bionic-security/multiverse amd64 Packages [6,116 B] 2019-12-05T04:53:30.117 INFO:teuthology.orchestra.run.smithi042.stdout:Get:14 http://security.ubuntu.com/ubuntu bionic-security/multiverse Translation-en [2,600 B] 2019-12-05T04:53:30.245 INFO:teuthology.orchestra.run.smithi042.stdout:Get:15 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB] 2019-12-05T04:53:30.378 INFO:teuthology.orchestra.run.smithi042.stdout:Get:16 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages [814 kB] 2019-12-05T04:53:30.646 INFO:teuthology.orchestra.run.smithi042.stdout:Get:17 http://archive.ubuntu.com/ubuntu bionic-updates/main i386 Packages [624 kB] 2019-12-05T04:53:30.719 INFO:teuthology.orchestra.run.smithi042.stdout:Get:18 http://archive.ubuntu.com/ubuntu bionic-updates/main Translation-en [287 kB] 2019-12-05T04:53:30.739 INFO:teuthology.orchestra.run.smithi042.stdout:Get:19 http://archive.ubuntu.com/ubuntu bionic-updates/restricted i386 Packages [9,260 B] 2019-12-05T04:53:30.740 INFO:teuthology.orchestra.run.smithi042.stdout:Get:20 http://archive.ubuntu.com/ubuntu bionic-updates/restricted amd64 Packages [23.4 kB] 2019-12-05T04:53:30.741 INFO:teuthology.orchestra.run.smithi042.stdout:Get:21 http://archive.ubuntu.com/ubuntu bionic-updates/restricted Translation-en [6,436 B] 2019-12-05T04:53:30.743 INFO:teuthology.orchestra.run.smithi042.stdout:Get:22 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 Packages [1,032 kB] 2019-12-05T04:53:30.790 INFO:teuthology.orchestra.run.smithi042.stdout:Get:23 http://archive.ubuntu.com/ubuntu bionic-updates/universe i386 Packages [996 kB] 2019-12-05T04:53:30.830 INFO:teuthology.orchestra.run.smithi042.stdout:Get:24 http://archive.ubuntu.com/ubuntu bionic-updates/universe Translation-en [319 kB] 2019-12-05T04:53:30.846 INFO:teuthology.orchestra.run.smithi042.stdout:Get:25 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse i386 Packages [6,924 B] 2019-12-05T04:53:30.846 INFO:teuthology.orchestra.run.smithi042.stdout:Get:26 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse amd64 Packages [8,736 B] 2019-12-05T04:53:30.846 INFO:teuthology.orchestra.run.smithi042.stdout:Get:27 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse Translation-en [4,080 B] 2019-12-05T04:53:30.890 INFO:teuthology.orchestra.run.smithi042.stdout:Get:28 http://archive.ubuntu.com/ubuntu bionic-backports/universe i386 Packages [4,028 B] 2019-12-05T04:53:30.890 INFO:teuthology.orchestra.run.smithi042.stdout:Get:29 http://archive.ubuntu.com/ubuntu bionic-backports/universe amd64 Packages [4,028 B] 2019-12-05T04:53:31.410 INFO:teuthology.orchestra.run.smithi042.stdout:Fetched 7,053 kB in 2s (4,418 kB/s) 2019-12-05T04:53:32.739 INFO:teuthology.orchestra.run.smithi042.stdout:Reading package lists... 2019-12-05T04:53:32.785 INFO:teuthology.orchestra.run.smithi042:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-current-generic 2019-12-05T04:53:32.872 INFO:teuthology.orchestra.run.smithi042.stdout:Reading package lists... 2019-12-05T04:53:33.124 INFO:teuthology.orchestra.run.smithi042.stdout:Building dependency tree... 2019-12-05T04:53:33.125 INFO:teuthology.orchestra.run.smithi042.stdout:Reading state information... 2019-12-05T04:53:33.209 INFO:teuthology.orchestra.run.smithi042.stderr:E: Unable to locate package linux-image-current-generic 2019-12-05T04:53:33.211 DEBUG:teuthology.orchestra.run:got remote process result: 100 2019-12-05T04:53:33.212 INFO:teuthology.orchestra.run.smithi042:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2019-12-05T04:53:33.295 INFO:teuthology.orchestra.run.smithi042.stdout:Reading package lists... 2019-12-05T04:53:33.524 INFO:teuthology.orchestra.run.smithi042.stdout:Building dependency tree... 2019-12-05T04:53:33.524 INFO:teuthology.orchestra.run.smithi042.stdout:Reading state information... 2019-12-05T04:53:33.698 INFO:teuthology.orchestra.run.smithi042.stdout:The following additional packages will be installed: 2019-12-05T04:53:33.698 INFO:teuthology.orchestra.run.smithi042.stdout: linux-generic linux-headers-4.15.0-72 linux-headers-4.15.0-72-generic 2019-12-05T04:53:33.698 INFO:teuthology.orchestra.run.smithi042.stdout: linux-headers-generic linux-image-4.15.0-72-generic 2019-12-05T04:53:33.698 INFO:teuthology.orchestra.run.smithi042.stdout: linux-modules-4.15.0-72-generic linux-modules-extra-4.15.0-72-generic 2019-12-05T04:53:33.704 INFO:teuthology.orchestra.run.smithi042.stdout:Suggested packages: 2019-12-05T04:53:33.705 INFO:teuthology.orchestra.run.smithi042.stdout: fdutils linux-doc-4.15.0 | linux-source-4.15.0 linux-tools 2019-12-05T04:53:33.705 INFO:teuthology.orchestra.run.smithi042.stdout:Recommended packages: 2019-12-05T04:53:33.705 INFO:teuthology.orchestra.run.smithi042.stdout: thermald 2019-12-05T04:53:33.731 INFO:teuthology.orchestra.run.smithi042.stdout:The following NEW packages will be installed: 2019-12-05T04:53:33.731 INFO:teuthology.orchestra.run.smithi042.stdout: linux-headers-4.15.0-72 linux-headers-4.15.0-72-generic 2019-12-05T04:53:33.732 INFO:teuthology.orchestra.run.smithi042.stdout: linux-image-4.15.0-72-generic linux-modules-4.15.0-72-generic 2019-12-05T04:53:33.732 INFO:teuthology.orchestra.run.smithi042.stdout: linux-modules-extra-4.15.0-72-generic 2019-12-05T04:53:33.733 INFO:teuthology.orchestra.run.smithi042.stdout:The following packages will be upgraded: 2019-12-05T04:53:33.734 INFO:teuthology.orchestra.run.smithi042.stdout: linux-generic linux-headers-generic linux-image-generic 2019-12-05T04:53:33.912 INFO:teuthology.orchestra.run.smithi042.stdout:3 upgraded, 5 newly installed, 0 to remove and 56 not upgraded. 2019-12-05T04:53:33.913 INFO:teuthology.orchestra.run.smithi042.stdout:Need to get 65.8 MB of archives. 2019-12-05T04:53:33.913 INFO:teuthology.orchestra.run.smithi042.stdout:After this operation, 332 MB of additional disk space will be used. 2019-12-05T04:53:33.913 INFO:teuthology.orchestra.run.smithi042.stdout:Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-modules-4.15.0-72-generic amd64 4.15.0-72.81 [13.0 MB] 2019-12-05T04:53:34.926 INFO:teuthology.orchestra.run.smithi042.stdout:Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-image-4.15.0-72-generic amd64 4.15.0-72.81 [7,991 kB] 2019-12-05T04:53:35.120 INFO:teuthology.orchestra.run.smithi042.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-modules-extra-4.15.0-72-generic amd64 4.15.0-72.81 [32.7 MB] 2019-12-05T04:53:36.018 INFO:teuthology.orchestra.run.smithi042.stdout:Get:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-generic amd64 4.15.0.72.74 [1,868 B] 2019-12-05T04:53:36.018 INFO:teuthology.orchestra.run.smithi042.stdout:Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-image-generic amd64 4.15.0.72.74 [2,392 B] 2019-12-05T04:53:36.018 INFO:teuthology.orchestra.run.smithi042.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-4.15.0-72 all 4.15.0-72.81 [10.9 MB] 2019-12-05T04:53:36.370 INFO:teuthology.orchestra.run.smithi042.stdout:Get:7 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-4.15.0-72-generic amd64 4.15.0-72.81 [1,107 kB] 2019-12-05T04:53:36.397 INFO:teuthology.orchestra.run.smithi042.stdout:Get:8 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-generic amd64 4.15.0.72.74 [2,348 B] 2019-12-05T04:53:36.875 INFO:teuthology.orchestra.run.smithi042.stdout:Fetched 65.8 MB in 3s (24.8 MB/s) 2019-12-05T04:53:37.026 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package linux-modules-4.15.0-72-generic. 2019-12-05T04:53:37.062 INFO:teuthology.orchestra.run.smithi042.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 101661 files and directories currently installed.) 2019-12-05T04:53:37.065 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../0-linux-modules-4.15.0-72-generic_4.15.0-72.81_amd64.deb ... 2019-12-05T04:53:37.122 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking linux-modules-4.15.0-72-generic (4.15.0-72.81) ... 2019-12-05T04:53:38.769 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package linux-image-4.15.0-72-generic. 2019-12-05T04:53:38.782 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../1-linux-image-4.15.0-72-generic_4.15.0-72.81_amd64.deb ... 2019-12-05T04:53:38.876 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking linux-image-4.15.0-72-generic (4.15.0-72.81) ... 2019-12-05T04:53:39.389 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package linux-modules-extra-4.15.0-72-generic. 2019-12-05T04:53:39.400 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../2-linux-modules-extra-4.15.0-72-generic_4.15.0-72.81_amd64.deb ... 2019-12-05T04:53:39.439 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking linux-modules-extra-4.15.0-72-generic (4.15.0-72.81) ... 2019-12-05T04:53:43.189 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../3-linux-generic_4.15.0.72.74_amd64.deb ... 2019-12-05T04:53:43.315 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking linux-generic (4.15.0.72.74) over (4.15.0.66.68) ... 2019-12-05T04:53:43.672 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../4-linux-image-generic_4.15.0.72.74_amd64.deb ... 2019-12-05T04:53:43.798 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking linux-image-generic (4.15.0.72.74) over (4.15.0.66.68) ... 2019-12-05T04:53:44.134 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package linux-headers-4.15.0-72. 2019-12-05T04:53:44.146 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../5-linux-headers-4.15.0-72_4.15.0-72.81_all.deb ... 2019-12-05T04:53:44.175 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking linux-headers-4.15.0-72 (4.15.0-72.81) ... 2019-12-05T04:53:48.974 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package linux-headers-4.15.0-72-generic. 2019-12-05T04:53:48.997 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../6-linux-headers-4.15.0-72-generic_4.15.0-72.81_amd64.deb ... 2019-12-05T04:53:49.041 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking linux-headers-4.15.0-72-generic (4.15.0-72.81) ... 2019-12-05T04:53:52.125 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../7-linux-headers-generic_4.15.0.72.74_amd64.deb ... 2019-12-05T04:53:52.248 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking linux-headers-generic (4.15.0.72.74) over (4.15.0.66.68) ... 2019-12-05T04:53:52.573 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up linux-modules-4.15.0-72-generic (4.15.0-72.81) ... 2019-12-05T04:53:53.224 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up linux-headers-4.15.0-72 (4.15.0-72.81) ... 2019-12-05T04:53:53.342 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up linux-image-4.15.0-72-generic (4.15.0-72.81) ... 2019-12-05T04:53:53.798 INFO:teuthology.orchestra.run.smithi042.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-4.15.0-72-generic 2019-12-05T04:53:53.799 INFO:teuthology.orchestra.run.smithi042.stdout:I: /initrd.img is now a symlink to boot/initrd.img-4.15.0-72-generic 2019-12-05T04:53:53.929 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up linux-headers-4.15.0-72-generic (4.15.0-72.81) ... 2019-12-05T04:53:54.063 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up linux-headers-generic (4.15.0.72.74) ... 2019-12-05T04:53:54.174 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up linux-modules-extra-4.15.0-72-generic (4.15.0-72.81) ... 2019-12-05T04:53:54.741 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up linux-image-generic (4.15.0.72.74) ... 2019-12-05T04:53:54.869 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up linux-generic (4.15.0.72.74) ... 2019-12-05T04:53:54.989 INFO:teuthology.orchestra.run.smithi042.stdout:Processing triggers for linux-image-4.15.0-72-generic (4.15.0-72.81) ... 2019-12-05T04:53:55.193 INFO:teuthology.orchestra.run.smithi042.stdout:/etc/kernel/postinst.d/initramfs-tools: 2019-12-05T04:53:55.194 INFO:teuthology.orchestra.run.smithi042.stdout:update-initramfs: Generating /boot/initrd.img-4.15.0-72-generic 2019-12-05T04:54:11.762 INFO:teuthology.orchestra.run.smithi042.stdout:/etc/kernel/postinst.d/zz-update-grub: 2019-12-05T04:54:11.763 INFO:teuthology.orchestra.run.smithi042.stdout:Sourcing file `/etc/default/grub' 2019-12-05T04:54:11.791 INFO:teuthology.orchestra.run.smithi042.stdout:Generating grub configuration file ... 2019-12-05T04:54:12.112 INFO:teuthology.orchestra.run.smithi042.stdout:Found linux image: /boot/vmlinuz-4.15.0-72-generic 2019-12-05T04:54:12.131 INFO:teuthology.orchestra.run.smithi042.stdout:Found initrd image: /boot/initrd.img-4.15.0-72-generic 2019-12-05T04:54:12.456 INFO:teuthology.orchestra.run.smithi042.stdout:Found linux image: /boot/vmlinuz-4.15.0-66-generic 2019-12-05T04:54:12.483 INFO:teuthology.orchestra.run.smithi042.stdout:Found initrd image: /boot/initrd.img-4.15.0-66-generic 2019-12-05T04:54:13.901 INFO:teuthology.orchestra.run.smithi042.stdout:done 2019-12-05T04:54:14.281 INFO:teuthology.orchestra.run.smithi042:> dpkg -s linux-image-generic 2019-12-05T04:54:14.307 INFO:teuthology.orchestra.run.smithi042.stdout:Package: linux-image-generic 2019-12-05T04:54:14.307 INFO:teuthology.orchestra.run.smithi042.stdout:Status: install ok installed 2019-12-05T04:54:14.307 INFO:teuthology.orchestra.run.smithi042.stdout:Priority: optional 2019-12-05T04:54:14.307 INFO:teuthology.orchestra.run.smithi042.stdout:Section: kernel 2019-12-05T04:54:14.308 INFO:teuthology.orchestra.run.smithi042.stdout:Installed-Size: 15 2019-12-05T04:54:14.308 INFO:teuthology.orchestra.run.smithi042.stdout:Maintainer: Ubuntu Kernel Team 2019-12-05T04:54:14.308 INFO:teuthology.orchestra.run.smithi042.stdout:Architecture: amd64 2019-12-05T04:54:14.309 INFO:teuthology.orchestra.run.smithi042.stdout:Source: linux-meta 2019-12-05T04:54:14.309 INFO:teuthology.orchestra.run.smithi042.stdout:Version: 4.15.0.72.74 2019-12-05T04:54:14.309 INFO:teuthology.orchestra.run.smithi042.stdout:Depends: linux-image-4.15.0-72-generic, linux-modules-extra-4.15.0-72-generic, linux-firmware, intel-microcode, amd64-microcode 2019-12-05T04:54:14.309 INFO:teuthology.orchestra.run.smithi042.stdout:Recommends: thermald 2019-12-05T04:54:14.309 INFO:teuthology.orchestra.run.smithi042.stdout:Description: Generic Linux kernel image 2019-12-05T04:54:14.309 INFO:teuthology.orchestra.run.smithi042.stdout: This package will always depend on the latest generic kernel image 2019-12-05T04:54:14.309 INFO:teuthology.orchestra.run.smithi042.stdout: available. 2019-12-05T04:54:14.310 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.15.0-66-generic Expected: 4.15.0-72-generic 2019-12-05T04:54:14.310 INFO:teuthology.orchestra.run.smithi040:> uname -r 2019-12-05T04:54:14.318 INFO:teuthology.orchestra.run.smithi040.stdout:4.15.0-66-generic 2019-12-05T04:54:14.319 INFO:teuthology.task.kernel:Running kernel on smithi040: 4.15.0-66-generic 2019-12-05T04:54:14.319 INFO:teuthology.orchestra.run.smithi040:> sudo apt-get clean 2019-12-05T04:54:14.444 INFO:teuthology.orchestra.run.smithi040:> sudo apt-get update 2019-12-05T04:54:14.574 INFO:teuthology.orchestra.run.smithi040.stdout:Get:1 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB] 2019-12-05T04:54:14.699 INFO:teuthology.orchestra.run.smithi040.stdout:Hit:2 http://archive.ubuntu.com/ubuntu bionic InRelease 2019-12-05T04:54:14.780 INFO:teuthology.orchestra.run.smithi040.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB] 2019-12-05T04:54:14.831 INFO:teuthology.orchestra.run.smithi040.stdout:Get:4 http://security.ubuntu.com/ubuntu bionic-security/main i386 Packages [413 kB] 2019-12-05T04:54:14.908 INFO:teuthology.orchestra.run.smithi040.stdout:Get:5 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages [590 kB] 2019-12-05T04:54:14.928 INFO:teuthology.orchestra.run.smithi040.stdout:Get:6 http://security.ubuntu.com/ubuntu bionic-security/main Translation-en [194 kB] 2019-12-05T04:54:14.934 INFO:teuthology.orchestra.run.smithi040.stdout:Get:7 http://security.ubuntu.com/ubuntu bionic-security/restricted amd64 Packages [14.6 kB] 2019-12-05T04:54:14.935 INFO:teuthology.orchestra.run.smithi040.stdout:Get:8 http://security.ubuntu.com/ubuntu bionic-security/restricted Translation-en [4,600 B] 2019-12-05T04:54:14.935 INFO:teuthology.orchestra.run.smithi040.stdout:Get:9 http://security.ubuntu.com/ubuntu bionic-security/universe amd64 Packages [626 kB] 2019-12-05T04:54:14.949 INFO:teuthology.orchestra.run.smithi040.stdout:Get:10 http://security.ubuntu.com/ubuntu bionic-security/universe i386 Packages [600 kB] 2019-12-05T04:54:14.956 INFO:teuthology.orchestra.run.smithi040.stdout:Get:11 http://security.ubuntu.com/ubuntu bionic-security/universe Translation-en [209 kB] 2019-12-05T04:54:14.956 INFO:teuthology.orchestra.run.smithi040.stdout:Get:12 http://security.ubuntu.com/ubuntu bionic-security/multiverse amd64 Packages [6,116 B] 2019-12-05T04:54:14.957 INFO:teuthology.orchestra.run.smithi040.stdout:Get:13 http://security.ubuntu.com/ubuntu bionic-security/multiverse i386 Packages [4,292 B] 2019-12-05T04:54:14.957 INFO:teuthology.orchestra.run.smithi040.stdout:Get:14 http://security.ubuntu.com/ubuntu bionic-security/multiverse Translation-en [2,600 B] 2019-12-05T04:54:15.055 INFO:teuthology.orchestra.run.smithi040.stdout:Get:15 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB] 2019-12-05T04:54:15.182 INFO:teuthology.orchestra.run.smithi040.stdout:Get:16 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages [814 kB] 2019-12-05T04:54:15.420 INFO:teuthology.orchestra.run.smithi040.stdout:Get:17 http://archive.ubuntu.com/ubuntu bionic-updates/main i386 Packages [624 kB] 2019-12-05T04:54:15.479 INFO:teuthology.orchestra.run.smithi040.stdout:Get:18 http://archive.ubuntu.com/ubuntu bionic-updates/main Translation-en [287 kB] 2019-12-05T04:54:15.501 INFO:teuthology.orchestra.run.smithi040.stdout:Get:19 http://archive.ubuntu.com/ubuntu bionic-updates/restricted amd64 Packages [23.4 kB] 2019-12-05T04:54:15.502 INFO:teuthology.orchestra.run.smithi040.stdout:Get:20 http://archive.ubuntu.com/ubuntu bionic-updates/restricted i386 Packages [9,260 B] 2019-12-05T04:54:15.503 INFO:teuthology.orchestra.run.smithi040.stdout:Get:21 http://archive.ubuntu.com/ubuntu bionic-updates/restricted Translation-en [6,436 B] 2019-12-05T04:54:15.503 INFO:teuthology.orchestra.run.smithi040.stdout:Get:22 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 Packages [1,032 kB] 2019-12-05T04:54:15.552 INFO:teuthology.orchestra.run.smithi040.stdout:Get:23 http://archive.ubuntu.com/ubuntu bionic-updates/universe i386 Packages [996 kB] 2019-12-05T04:54:15.589 INFO:teuthology.orchestra.run.smithi040.stdout:Get:24 http://archive.ubuntu.com/ubuntu bionic-updates/universe Translation-en [319 kB] 2019-12-05T04:54:15.599 INFO:teuthology.orchestra.run.smithi040.stdout:Get:25 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse i386 Packages [6,924 B] 2019-12-05T04:54:15.599 INFO:teuthology.orchestra.run.smithi040.stdout:Get:26 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse amd64 Packages [8,736 B] 2019-12-05T04:54:15.600 INFO:teuthology.orchestra.run.smithi040.stdout:Get:27 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse Translation-en [4,080 B] 2019-12-05T04:54:15.600 INFO:teuthology.orchestra.run.smithi040.stdout:Get:28 http://archive.ubuntu.com/ubuntu bionic-backports/universe amd64 Packages [4,028 B] 2019-12-05T04:54:15.662 INFO:teuthology.orchestra.run.smithi040.stdout:Get:29 http://archive.ubuntu.com/ubuntu bionic-backports/universe i386 Packages [4,028 B] 2019-12-05T04:54:16.161 INFO:teuthology.orchestra.run.smithi040.stdout:Fetched 7,053 kB in 1s (4,716 kB/s) 2019-12-05T04:54:17.503 INFO:teuthology.orchestra.run.smithi040.stdout:Reading package lists... 2019-12-05T04:54:17.553 INFO:teuthology.orchestra.run.smithi040:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-current-generic 2019-12-05T04:54:17.634 INFO:teuthology.orchestra.run.smithi040.stdout:Reading package lists... 2019-12-05T04:54:17.868 INFO:teuthology.orchestra.run.smithi040.stdout:Building dependency tree... 2019-12-05T04:54:17.868 INFO:teuthology.orchestra.run.smithi040.stdout:Reading state information... 2019-12-05T04:54:17.956 INFO:teuthology.orchestra.run.smithi040.stderr:E: Unable to locate package linux-image-current-generic 2019-12-05T04:54:17.957 DEBUG:teuthology.orchestra.run:got remote process result: 100 2019-12-05T04:54:17.958 INFO:teuthology.orchestra.run.smithi040:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2019-12-05T04:54:18.053 INFO:teuthology.orchestra.run.smithi040.stdout:Reading package lists... 2019-12-05T04:54:18.298 INFO:teuthology.orchestra.run.smithi040.stdout:Building dependency tree... 2019-12-05T04:54:18.299 INFO:teuthology.orchestra.run.smithi040.stdout:Reading state information... 2019-12-05T04:54:18.467 INFO:teuthology.orchestra.run.smithi040.stdout:The following additional packages will be installed: 2019-12-05T04:54:18.468 INFO:teuthology.orchestra.run.smithi040.stdout: linux-generic linux-headers-4.15.0-72 linux-headers-4.15.0-72-generic 2019-12-05T04:54:18.468 INFO:teuthology.orchestra.run.smithi040.stdout: linux-headers-generic linux-image-4.15.0-72-generic 2019-12-05T04:54:18.468 INFO:teuthology.orchestra.run.smithi040.stdout: linux-modules-4.15.0-72-generic linux-modules-extra-4.15.0-72-generic 2019-12-05T04:54:18.481 INFO:teuthology.orchestra.run.smithi040.stdout:Suggested packages: 2019-12-05T04:54:18.481 INFO:teuthology.orchestra.run.smithi040.stdout: fdutils linux-doc-4.15.0 | linux-source-4.15.0 linux-tools 2019-12-05T04:54:18.482 INFO:teuthology.orchestra.run.smithi040.stdout:Recommended packages: 2019-12-05T04:54:18.482 INFO:teuthology.orchestra.run.smithi040.stdout: thermald 2019-12-05T04:54:18.501 INFO:teuthology.orchestra.run.smithi040.stdout:The following NEW packages will be installed: 2019-12-05T04:54:18.501 INFO:teuthology.orchestra.run.smithi040.stdout: linux-headers-4.15.0-72 linux-headers-4.15.0-72-generic 2019-12-05T04:54:18.502 INFO:teuthology.orchestra.run.smithi040.stdout: linux-image-4.15.0-72-generic linux-modules-4.15.0-72-generic 2019-12-05T04:54:18.502 INFO:teuthology.orchestra.run.smithi040.stdout: linux-modules-extra-4.15.0-72-generic 2019-12-05T04:54:18.503 INFO:teuthology.orchestra.run.smithi040.stdout:The following packages will be upgraded: 2019-12-05T04:54:18.504 INFO:teuthology.orchestra.run.smithi040.stdout: linux-generic linux-headers-generic linux-image-generic 2019-12-05T04:54:18.684 INFO:teuthology.orchestra.run.smithi040.stdout:3 upgraded, 5 newly installed, 0 to remove and 56 not upgraded. 2019-12-05T04:54:18.684 INFO:teuthology.orchestra.run.smithi040.stdout:Need to get 65.8 MB of archives. 2019-12-05T04:54:18.685 INFO:teuthology.orchestra.run.smithi040.stdout:After this operation, 332 MB of additional disk space will be used. 2019-12-05T04:54:18.685 INFO:teuthology.orchestra.run.smithi040.stdout:Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-modules-4.15.0-72-generic amd64 4.15.0-72.81 [13.0 MB] 2019-12-05T04:54:19.692 INFO:teuthology.orchestra.run.smithi040.stdout:Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-image-4.15.0-72-generic amd64 4.15.0-72.81 [7,991 kB] 2019-12-05T04:54:19.891 INFO:teuthology.orchestra.run.smithi040.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-modules-extra-4.15.0-72-generic amd64 4.15.0-72.81 [32.7 MB] 2019-12-05T04:54:20.824 INFO:teuthology.orchestra.run.smithi040.stdout:Get:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-generic amd64 4.15.0.72.74 [1,868 B] 2019-12-05T04:54:20.824 INFO:teuthology.orchestra.run.smithi040.stdout:Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-image-generic amd64 4.15.0.72.74 [2,392 B] 2019-12-05T04:54:20.995 INFO:teuthology.orchestra.run.smithi040.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-4.15.0-72 all 4.15.0-72.81 [10.9 MB] 2019-12-05T04:54:21.640 INFO:teuthology.orchestra.run.smithi040.stdout:Get:7 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-4.15.0-72-generic amd64 4.15.0-72.81 [1,107 kB] 2019-12-05T04:54:21.716 INFO:teuthology.orchestra.run.smithi040.stdout:Get:8 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-generic amd64 4.15.0.72.74 [2,348 B] 2019-12-05T04:54:22.167 INFO:teuthology.orchestra.run.smithi040.stdout:Fetched 65.8 MB in 3s (20.6 MB/s) 2019-12-05T04:54:22.319 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package linux-modules-4.15.0-72-generic. 2019-12-05T04:54:22.357 INFO:teuthology.orchestra.run.smithi040.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 101661 files and directories currently installed.) 2019-12-05T04:54:22.360 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../0-linux-modules-4.15.0-72-generic_4.15.0-72.81_amd64.deb ... 2019-12-05T04:54:22.420 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking linux-modules-4.15.0-72-generic (4.15.0-72.81) ... 2019-12-05T04:54:24.040 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package linux-image-4.15.0-72-generic. 2019-12-05T04:54:24.053 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../1-linux-image-4.15.0-72-generic_4.15.0-72.81_amd64.deb ... 2019-12-05T04:54:24.158 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking linux-image-4.15.0-72-generic (4.15.0-72.81) ... 2019-12-05T04:54:24.681 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package linux-modules-extra-4.15.0-72-generic. 2019-12-05T04:54:24.690 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../2-linux-modules-extra-4.15.0-72-generic_4.15.0-72.81_amd64.deb ... 2019-12-05T04:54:24.727 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking linux-modules-extra-4.15.0-72-generic (4.15.0-72.81) ... 2019-12-05T04:54:28.517 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../3-linux-generic_4.15.0.72.74_amd64.deb ... 2019-12-05T04:54:28.630 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking linux-generic (4.15.0.72.74) over (4.15.0.66.68) ... 2019-12-05T04:54:29.004 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../4-linux-image-generic_4.15.0.72.74_amd64.deb ... 2019-12-05T04:54:29.100 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking linux-image-generic (4.15.0.72.74) over (4.15.0.66.68) ... 2019-12-05T04:54:29.428 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package linux-headers-4.15.0-72. 2019-12-05T04:54:29.440 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../5-linux-headers-4.15.0-72_4.15.0-72.81_all.deb ... 2019-12-05T04:54:29.478 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking linux-headers-4.15.0-72 (4.15.0-72.81) ... 2019-12-05T04:54:34.284 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package linux-headers-4.15.0-72-generic. 2019-12-05T04:54:34.307 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../6-linux-headers-4.15.0-72-generic_4.15.0-72.81_amd64.deb ... 2019-12-05T04:54:34.351 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking linux-headers-4.15.0-72-generic (4.15.0-72.81) ... 2019-12-05T04:54:37.438 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../7-linux-headers-generic_4.15.0.72.74_amd64.deb ... 2019-12-05T04:54:37.554 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking linux-headers-generic (4.15.0.72.74) over (4.15.0.66.68) ... 2019-12-05T04:54:37.877 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up linux-modules-4.15.0-72-generic (4.15.0-72.81) ... 2019-12-05T04:54:38.481 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up linux-headers-4.15.0-72 (4.15.0-72.81) ... 2019-12-05T04:54:38.595 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up linux-image-4.15.0-72-generic (4.15.0-72.81) ... 2019-12-05T04:54:39.062 INFO:teuthology.orchestra.run.smithi040.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-4.15.0-72-generic 2019-12-05T04:54:39.062 INFO:teuthology.orchestra.run.smithi040.stdout:I: /initrd.img is now a symlink to boot/initrd.img-4.15.0-72-generic 2019-12-05T04:54:39.203 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up linux-headers-4.15.0-72-generic (4.15.0-72.81) ... 2019-12-05T04:54:39.374 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up linux-headers-generic (4.15.0.72.74) ... 2019-12-05T04:54:39.496 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up linux-modules-extra-4.15.0-72-generic (4.15.0-72.81) ... 2019-12-05T04:54:40.043 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up linux-image-generic (4.15.0.72.74) ... 2019-12-05T04:54:40.163 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up linux-generic (4.15.0.72.74) ... 2019-12-05T04:54:40.293 INFO:teuthology.orchestra.run.smithi040.stdout:Processing triggers for linux-image-4.15.0-72-generic (4.15.0-72.81) ... 2019-12-05T04:54:40.511 INFO:teuthology.orchestra.run.smithi040.stdout:/etc/kernel/postinst.d/initramfs-tools: 2019-12-05T04:54:40.511 INFO:teuthology.orchestra.run.smithi040.stdout:update-initramfs: Generating /boot/initrd.img-4.15.0-72-generic 2019-12-05T04:54:56.806 INFO:teuthology.orchestra.run.smithi040.stdout:/etc/kernel/postinst.d/zz-update-grub: 2019-12-05T04:54:56.807 INFO:teuthology.orchestra.run.smithi040.stdout:Sourcing file `/etc/default/grub' 2019-12-05T04:54:56.836 INFO:teuthology.orchestra.run.smithi040.stdout:Generating grub configuration file ... 2019-12-05T04:54:57.288 INFO:teuthology.orchestra.run.smithi040.stdout:Found linux image: /boot/vmlinuz-4.15.0-72-generic 2019-12-05T04:54:57.319 INFO:teuthology.orchestra.run.smithi040.stdout:Found initrd image: /boot/initrd.img-4.15.0-72-generic 2019-12-05T04:54:57.691 INFO:teuthology.orchestra.run.smithi040.stdout:Found linux image: /boot/vmlinuz-4.15.0-66-generic 2019-12-05T04:54:57.726 INFO:teuthology.orchestra.run.smithi040.stdout:Found initrd image: /boot/initrd.img-4.15.0-66-generic 2019-12-05T04:54:59.158 INFO:teuthology.orchestra.run.smithi040.stdout:done 2019-12-05T04:54:59.595 INFO:teuthology.orchestra.run.smithi040:> dpkg -s linux-image-generic 2019-12-05T04:54:59.620 INFO:teuthology.orchestra.run.smithi040.stdout:Package: linux-image-generic 2019-12-05T04:54:59.621 INFO:teuthology.orchestra.run.smithi040.stdout:Status: install ok installed 2019-12-05T04:54:59.621 INFO:teuthology.orchestra.run.smithi040.stdout:Priority: optional 2019-12-05T04:54:59.621 INFO:teuthology.orchestra.run.smithi040.stdout:Section: kernel 2019-12-05T04:54:59.621 INFO:teuthology.orchestra.run.smithi040.stdout:Installed-Size: 15 2019-12-05T04:54:59.621 INFO:teuthology.orchestra.run.smithi040.stdout:Maintainer: Ubuntu Kernel Team 2019-12-05T04:54:59.622 INFO:teuthology.orchestra.run.smithi040.stdout:Architecture: amd64 2019-12-05T04:54:59.622 INFO:teuthology.orchestra.run.smithi040.stdout:Source: linux-meta 2019-12-05T04:54:59.623 INFO:teuthology.orchestra.run.smithi040.stdout:Version: 4.15.0.72.74 2019-12-05T04:54:59.623 INFO:teuthology.orchestra.run.smithi040.stdout:Depends: linux-image-4.15.0-72-generic, linux-modules-extra-4.15.0-72-generic, linux-firmware, intel-microcode, amd64-microcode 2019-12-05T04:54:59.623 INFO:teuthology.orchestra.run.smithi040.stdout:Recommends: thermald 2019-12-05T04:54:59.623 INFO:teuthology.orchestra.run.smithi040.stdout:Description: Generic Linux kernel image 2019-12-05T04:54:59.623 INFO:teuthology.orchestra.run.smithi040.stdout: This package will always depend on the latest generic kernel image 2019-12-05T04:54:59.623 INFO:teuthology.orchestra.run.smithi040.stdout: available. 2019-12-05T04:54:59.624 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.15.0-66-generic Expected: 4.15.0-72-generic 2019-12-05T04:54:59.624 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2019-12-05T04:54:59.624 DEBUG:teuthology.task.kernel:src is distro, skipping download 2019-12-05T04:54:59.624 DEBUG:teuthology.task.kernel:src is distro, skipping download 2019-12-05T04:54:59.624 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2019-12-05T04:54:59.624 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi042.front.sepia.ceph.com, path=None, version=distro) 2019-12-05T04:54:59.625 INFO:teuthology.orchestra.run.smithi042:> sudo apt-get clean 2019-12-05T04:54:59.699 INFO:teuthology.orchestra.run.smithi042:> sudo apt-get update 2019-12-05T04:54:59.963 INFO:teuthology.orchestra.run.smithi042.stdout:Hit:1 http://security.ubuntu.com/ubuntu bionic-security InRelease 2019-12-05T04:54:59.963 INFO:teuthology.orchestra.run.smithi042.stdout:Hit:2 http://archive.ubuntu.com/ubuntu bionic InRelease 2019-12-05T04:55:00.033 INFO:teuthology.orchestra.run.smithi042.stdout:Hit:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease 2019-12-05T04:55:00.114 INFO:teuthology.orchestra.run.smithi042.stdout:Hit:4 http://archive.ubuntu.com/ubuntu bionic-backports InRelease 2019-12-05T04:55:01.793 INFO:teuthology.orchestra.run.smithi042.stdout:Reading package lists... 2019-12-05T04:55:01.834 INFO:teuthology.orchestra.run.smithi042:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-current-generic 2019-12-05T04:55:01.913 INFO:teuthology.orchestra.run.smithi042.stdout:Reading package lists... 2019-12-05T04:55:02.147 INFO:teuthology.orchestra.run.smithi042.stdout:Building dependency tree... 2019-12-05T04:55:02.148 INFO:teuthology.orchestra.run.smithi042.stdout:Reading state information... 2019-12-05T04:55:02.233 INFO:teuthology.orchestra.run.smithi042.stderr:E: Unable to locate package linux-image-current-generic 2019-12-05T04:55:02.235 DEBUG:teuthology.orchestra.run:got remote process result: 100 2019-12-05T04:55:02.236 INFO:teuthology.orchestra.run.smithi042:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2019-12-05T04:55:02.318 INFO:teuthology.orchestra.run.smithi042.stdout:Reading package lists... 2019-12-05T04:55:02.570 INFO:teuthology.orchestra.run.smithi042.stdout:Building dependency tree... 2019-12-05T04:55:02.571 INFO:teuthology.orchestra.run.smithi042.stdout:Reading state information... 2019-12-05T04:55:02.775 INFO:teuthology.orchestra.run.smithi042.stdout:linux-image-generic is already the newest version (4.15.0.72.74). 2019-12-05T04:55:02.775 INFO:teuthology.orchestra.run.smithi042.stdout:0 upgraded, 0 newly installed, 0 to remove and 56 not upgraded. 2019-12-05T04:55:02.777 INFO:teuthology.orchestra.run.smithi042:> dpkg -s linux-image-generic 2019-12-05T04:55:02.801 INFO:teuthology.orchestra.run.smithi042.stdout:Package: linux-image-generic 2019-12-05T04:55:02.801 INFO:teuthology.orchestra.run.smithi042.stdout:Status: install ok installed 2019-12-05T04:55:02.801 INFO:teuthology.orchestra.run.smithi042.stdout:Priority: optional 2019-12-05T04:55:02.802 INFO:teuthology.orchestra.run.smithi042.stdout:Section: kernel 2019-12-05T04:55:02.802 INFO:teuthology.orchestra.run.smithi042.stdout:Installed-Size: 15 2019-12-05T04:55:02.802 INFO:teuthology.orchestra.run.smithi042.stdout:Maintainer: Ubuntu Kernel Team 2019-12-05T04:55:02.802 INFO:teuthology.orchestra.run.smithi042.stdout:Architecture: amd64 2019-12-05T04:55:02.802 INFO:teuthology.orchestra.run.smithi042.stdout:Source: linux-meta 2019-12-05T04:55:02.802 INFO:teuthology.orchestra.run.smithi042.stdout:Version: 4.15.0.72.74 2019-12-05T04:55:02.803 INFO:teuthology.orchestra.run.smithi042.stdout:Depends: linux-image-4.15.0-72-generic, linux-modules-extra-4.15.0-72-generic, linux-firmware, intel-microcode, amd64-microcode 2019-12-05T04:55:02.803 INFO:teuthology.orchestra.run.smithi042.stdout:Recommends: thermald 2019-12-05T04:55:02.803 INFO:teuthology.orchestra.run.smithi042.stdout:Description: Generic Linux kernel image 2019-12-05T04:55:02.803 INFO:teuthology.orchestra.run.smithi042.stdout: This package will always depend on the latest generic kernel image 2019-12-05T04:55:02.803 INFO:teuthology.orchestra.run.smithi042.stdout: available. 2019-12-05T04:55:02.804 INFO:teuthology.orchestra.run.smithi042:> mktemp 2019-12-05T04:55:02.877 INFO:teuthology.orchestra.run.smithi042.stdout:/tmp/tmp.bkhbO6287c 2019-12-05T04:55:02.877 INFO:teuthology.orchestra.run.smithi042:> sudo cp /boot/grub/grub.cfg /tmp/tmp.bkhbO6287c 2019-12-05T04:55:02.962 INFO:teuthology.orchestra.run.smithi042:> sudo chmod 0666 /tmp/tmp.bkhbO6287c 2019-12-05T04:55:03.139 DEBUG:teuthology.orchestra.remote:smithi042:/tmp/tmp.bkhbO6287c is 8KB 2019-12-05T04:55:03.190 INFO:teuthology.orchestra.run.smithi042:> rm -fr /tmp/tmp.bkhbO6287c 2019-12-05T04:55:03.197 INFO:teuthology.orchestra.run.smithi042:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2019-12-05T04:55:03.278 INFO:teuthology.orchestra.run.smithi042:> sudo sh -c 'cat > /etc/grub.d/01_ceph_kernel' && sudo chmod 755 /etc/grub.d/01_ceph_kernel 2019-12-05T04:55:03.404 INFO:teuthology.task.kernel:Distro Kernel Version: 4.15.0-72-generic 2019-12-05T04:55:03.404 INFO:teuthology.orchestra.run.smithi042:> sudo update-grub 2019-12-05T04:55:04.731 INFO:teuthology.orchestra.run.smithi042.stderr:Sourcing file `/etc/default/grub' 2019-12-05T04:55:04.742 INFO:teuthology.orchestra.run.smithi042.stderr:Generating grub configuration file ... 2019-12-05T04:55:04.977 INFO:teuthology.orchestra.run.smithi042.stderr:Found linux image: /boot/vmlinuz-4.15.0-72-generic 2019-12-05T04:55:04.987 INFO:teuthology.orchestra.run.smithi042.stderr:Found initrd image: /boot/initrd.img-4.15.0-72-generic 2019-12-05T04:55:05.271 INFO:teuthology.orchestra.run.smithi042.stderr:Found linux image: /boot/vmlinuz-4.15.0-66-generic 2019-12-05T04:55:05.299 INFO:teuthology.orchestra.run.smithi042.stderr:Found initrd image: /boot/initrd.img-4.15.0-66-generic 2019-12-05T04:55:06.411 INFO:teuthology.orchestra.run.smithi042.stderr:done 2019-12-05T04:55:06.411 INFO:teuthology.orchestra.run.smithi042:> sudo shutdown -r now 2019-12-05T04:55:06.418 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2019-12-05T04:55:06.418 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi040.front.sepia.ceph.com, path=None, version=distro) 2019-12-05T04:55:06.419 INFO:teuthology.orchestra.run.smithi040:> sudo apt-get clean 2019-12-05T04:55:06.465 INFO:teuthology.orchestra.run.smithi040:> sudo apt-get update 2019-12-05T04:55:06.669 INFO:teuthology.orchestra.run.smithi040.stdout:Hit:1 http://security.ubuntu.com/ubuntu bionic-security InRelease 2019-12-05T04:55:06.783 INFO:teuthology.orchestra.run.smithi040.stdout:Hit:2 http://archive.ubuntu.com/ubuntu bionic InRelease 2019-12-05T04:55:06.864 INFO:teuthology.orchestra.run.smithi040.stdout:Hit:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease 2019-12-05T04:55:06.949 INFO:teuthology.orchestra.run.smithi040.stdout:Hit:4 http://archive.ubuntu.com/ubuntu bionic-backports InRelease 2019-12-05T04:55:08.514 INFO:teuthology.orchestra.run.smithi040.stdout:Reading package lists... 2019-12-05T04:55:08.559 INFO:teuthology.orchestra.run.smithi040:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-current-generic 2019-12-05T04:55:08.643 INFO:teuthology.orchestra.run.smithi040.stdout:Reading package lists... 2019-12-05T04:55:08.894 INFO:teuthology.orchestra.run.smithi040.stdout:Building dependency tree... 2019-12-05T04:55:08.895 INFO:teuthology.orchestra.run.smithi040.stdout:Reading state information... 2019-12-05T04:55:08.980 INFO:teuthology.orchestra.run.smithi040.stderr:E: Unable to locate package linux-image-current-generic 2019-12-05T04:55:08.982 DEBUG:teuthology.orchestra.run:got remote process result: 100 2019-12-05T04:55:08.982 INFO:teuthology.orchestra.run.smithi040:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2019-12-05T04:55:09.063 INFO:teuthology.orchestra.run.smithi040.stdout:Reading package lists... 2019-12-05T04:55:09.290 INFO:teuthology.orchestra.run.smithi040.stdout:Building dependency tree... 2019-12-05T04:55:09.290 INFO:teuthology.orchestra.run.smithi040.stdout:Reading state information... 2019-12-05T04:55:09.488 INFO:teuthology.orchestra.run.smithi040.stdout:linux-image-generic is already the newest version (4.15.0.72.74). 2019-12-05T04:55:09.488 INFO:teuthology.orchestra.run.smithi040.stdout:0 upgraded, 0 newly installed, 0 to remove and 56 not upgraded. 2019-12-05T04:55:09.490 INFO:teuthology.orchestra.run.smithi040:> dpkg -s linux-image-generic 2019-12-05T04:55:09.511 INFO:teuthology.orchestra.run.smithi040.stdout:Package: linux-image-generic 2019-12-05T04:55:09.511 INFO:teuthology.orchestra.run.smithi040.stdout:Status: install ok installed 2019-12-05T04:55:09.512 INFO:teuthology.orchestra.run.smithi040.stdout:Priority: optional 2019-12-05T04:55:09.512 INFO:teuthology.orchestra.run.smithi040.stdout:Section: kernel 2019-12-05T04:55:09.512 INFO:teuthology.orchestra.run.smithi040.stdout:Installed-Size: 15 2019-12-05T04:55:09.512 INFO:teuthology.orchestra.run.smithi040.stdout:Maintainer: Ubuntu Kernel Team 2019-12-05T04:55:09.512 INFO:teuthology.orchestra.run.smithi040.stdout:Architecture: amd64 2019-12-05T04:55:09.512 INFO:teuthology.orchestra.run.smithi040.stdout:Source: linux-meta 2019-12-05T04:55:09.513 INFO:teuthology.orchestra.run.smithi040.stdout:Version: 4.15.0.72.74 2019-12-05T04:55:09.513 INFO:teuthology.orchestra.run.smithi040.stdout:Depends: linux-image-4.15.0-72-generic, linux-modules-extra-4.15.0-72-generic, linux-firmware, intel-microcode, amd64-microcode 2019-12-05T04:55:09.513 INFO:teuthology.orchestra.run.smithi040.stdout:Recommends: thermald 2019-12-05T04:55:09.521 INFO:teuthology.orchestra.run.smithi040.stdout:Description: Generic Linux kernel image 2019-12-05T04:55:09.521 INFO:teuthology.orchestra.run.smithi040.stdout: This package will always depend on the latest generic kernel image 2019-12-05T04:55:09.521 INFO:teuthology.orchestra.run.smithi040.stdout: available. 2019-12-05T04:55:09.522 INFO:teuthology.orchestra.run.smithi040:> mktemp 2019-12-05T04:55:09.563 INFO:teuthology.orchestra.run.smithi040.stdout:/tmp/tmp.CNmzRMabbh 2019-12-05T04:55:09.563 INFO:teuthology.orchestra.run.smithi040:> sudo cp /boot/grub/grub.cfg /tmp/tmp.CNmzRMabbh 2019-12-05T04:55:09.643 INFO:teuthology.orchestra.run.smithi040:> sudo chmod 0666 /tmp/tmp.CNmzRMabbh 2019-12-05T04:55:09.814 DEBUG:teuthology.orchestra.remote:smithi040:/tmp/tmp.CNmzRMabbh is 8KB 2019-12-05T04:55:09.865 INFO:teuthology.orchestra.run.smithi040:> rm -fr /tmp/tmp.CNmzRMabbh 2019-12-05T04:55:09.872 INFO:teuthology.orchestra.run.smithi040:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2019-12-05T04:55:09.956 INFO:teuthology.orchestra.run.smithi040:> sudo sh -c 'cat > /etc/grub.d/01_ceph_kernel' && sudo chmod 755 /etc/grub.d/01_ceph_kernel 2019-12-05T04:55:10.083 INFO:teuthology.task.kernel:Distro Kernel Version: 4.15.0-72-generic 2019-12-05T04:55:10.084 INFO:teuthology.orchestra.run.smithi040:> sudo update-grub 2019-12-05T04:55:11.466 INFO:teuthology.orchestra.run.smithi040.stderr:Sourcing file `/etc/default/grub' 2019-12-05T04:55:11.474 INFO:teuthology.orchestra.run.smithi040.stderr:Generating grub configuration file ... 2019-12-05T04:55:11.727 INFO:teuthology.orchestra.run.smithi040.stderr:Found linux image: /boot/vmlinuz-4.15.0-72-generic 2019-12-05T04:55:11.737 INFO:teuthology.orchestra.run.smithi040.stderr:Found initrd image: /boot/initrd.img-4.15.0-72-generic 2019-12-05T04:55:12.000 INFO:teuthology.orchestra.run.smithi040.stderr:Found linux image: /boot/vmlinuz-4.15.0-66-generic 2019-12-05T04:55:12.012 INFO:teuthology.orchestra.run.smithi040.stderr:Found initrd image: /boot/initrd.img-4.15.0-66-generic 2019-12-05T04:55:13.142 INFO:teuthology.orchestra.run.smithi040.stderr:done 2019-12-05T04:55:13.143 INFO:teuthology.orchestra.run.smithi040:> sudo shutdown -r now 2019-12-05T04:55:13.147 INFO:teuthology.misc:Re-opening connections... 2019-12-05T04:55:13.147 INFO:teuthology.misc:trying to connect to ubuntu@smithi042.front.sepia.ceph.com 2019-12-05T04:55:13.147 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2019-12-05T04:55:13.148 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi042.front.sepia.ceph.com', 'timeout': 60} 2019-12-05T04:56:13.149 DEBUG:teuthology.orchestra.remote:timed out 2019-12-05T04:56:13.173 INFO:teuthology.misc:trying to connect to ubuntu@smithi040.front.sepia.ceph.com 2019-12-05T04:56:13.173 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2019-12-05T04:56:13.174 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi040.front.sepia.ceph.com', 'timeout': 60} 2019-12-05T04:57:13.184 DEBUG:teuthology.orchestra.remote:timed out 2019-12-05T04:57:13.185 DEBUG:teuthology.misc:waited 120.037328959 2019-12-05T04:57:14.185 INFO:teuthology.misc:trying to connect to ubuntu@smithi042.front.sepia.ceph.com 2019-12-05T04:57:14.185 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2019-12-05T04:57:14.186 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi042.front.sepia.ceph.com', 'timeout': 60} 2019-12-05T04:57:14.331 INFO:teuthology.orchestra.run.smithi042:> true 2019-12-05T04:57:14.817 DEBUG:teuthology.misc:waited 121.669676781 2019-12-05T04:57:15.819 INFO:teuthology.misc:trying to connect to ubuntu@smithi040.front.sepia.ceph.com 2019-12-05T04:57:15.819 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2019-12-05T04:57:15.819 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi040.front.sepia.ceph.com', 'timeout': 60} 2019-12-05T04:57:15.973 INFO:teuthology.orchestra.run.smithi040:> true 2019-12-05T04:57:16.562 DEBUG:teuthology.misc:waited 123.415241957 2019-12-05T04:57:17.563 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2019-12-05T04:57:17.564 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "4.15.0-72-generic"... 2019-12-05T04:57:17.564 INFO:teuthology.orchestra.run.smithi042:> uname -r 2019-12-05T04:57:17.571 INFO:teuthology.orchestra.run.smithi042.stdout:4.15.0-72-generic 2019-12-05T04:57:17.572 DEBUG:teuthology.task.kernel:current kernel version is 4.15.0-72-generic 2019-12-05T04:57:17.572 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2019-12-05T04:57:17.572 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2019-12-05T04:57:17.572 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "4.15.0-72-generic"... 2019-12-05T04:57:17.572 INFO:teuthology.orchestra.run.smithi040:> uname -r 2019-12-05T04:57:17.578 INFO:teuthology.orchestra.run.smithi040.stdout:4.15.0-72-generic 2019-12-05T04:57:17.579 DEBUG:teuthology.task.kernel:current kernel version is 4.15.0-72-generic 2019-12-05T04:57:17.579 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2019-12-05T04:57:18.584 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2019-12-05T04:57:18.584 INFO:teuthology.orchestra.run.smithi042:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2019-12-05T04:57:18.610 INFO:teuthology.orchestra.run.smithi042.stdout:ttyS1 2019-12-05T04:57:18.648 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2019-12-05T04:57:18.648 INFO:teuthology.orchestra.run.smithi040:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2019-12-05T04:57:18.701 INFO:teuthology.orchestra.run.smithi040.stdout:ttyS1 2019-12-05T04:57:18.730 INFO:teuthology.run_tasks:Running task internal.base... 2019-12-05T04:57:18.787 INFO:teuthology.task.internal:Creating test directory... 2019-12-05T04:57:18.788 INFO:teuthology.orchestra.run.smithi040:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2019-12-05T04:57:18.793 INFO:teuthology.orchestra.run.smithi042:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2019-12-05T04:57:18.802 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2019-12-05T04:57:18.812 INFO:teuthology.run_tasks:Running task internal.archive... 2019-12-05T04:57:18.875 INFO:teuthology.task.internal:Creating archive directory... 2019-12-05T04:57:18.877 INFO:teuthology.orchestra.run.smithi040:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2019-12-05T04:57:18.910 INFO:teuthology.orchestra.run.smithi042:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2019-12-05T04:57:18.941 INFO:teuthology.run_tasks:Running task internal.coredump... 2019-12-05T04:57:18.958 INFO:teuthology.task.internal:Enabling coredump saving... 2019-12-05T04:57:18.958 INFO:teuthology.orchestra.run.smithi040:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2019-12-05T04:57:18.994 INFO:teuthology.orchestra.run.smithi042:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2019-12-05T04:57:19.015 INFO:teuthology.orchestra.run.smithi040.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2019-12-05T04:57:19.019 INFO:teuthology.orchestra.run.smithi042.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2019-12-05T04:57:19.020 INFO:teuthology.run_tasks:Running task internal.sudo... 2019-12-05T04:57:19.031 INFO:teuthology.task.internal:Configuring sudo... 2019-12-05T04:57:19.031 INFO:teuthology.orchestra.run.smithi040:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2019-12-05T04:57:19.070 INFO:teuthology.orchestra.run.smithi042:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2019-12-05T04:57:19.087 INFO:teuthology.run_tasks:Running task internal.syslog... 2019-12-05T04:57:19.106 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2019-12-05T04:57:19.106 INFO:teuthology.orchestra.run.smithi040:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2019-12-05T04:57:19.131 INFO:teuthology.orchestra.run.smithi042:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2019-12-05T04:57:19.139 INFO:teuthology.orchestra.run.smithi042:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2019-12-05T04:57:19.189 INFO:teuthology.orchestra.run.smithi042:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2019-12-05T04:57:19.265 INFO:teuthology.orchestra.run.smithi042:> sudo sh -c 'cat > /etc/rsyslog.d/80-cephtest.conf' 2019-12-05T04:57:19.397 INFO:teuthology.orchestra.run.smithi040:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2019-12-05T04:57:19.429 INFO:teuthology.orchestra.run.smithi040:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2019-12-05T04:57:19.508 INFO:teuthology.orchestra.run.smithi040:> sudo sh -c 'cat > /etc/rsyslog.d/80-cephtest.conf' 2019-12-05T04:57:19.638 INFO:teuthology.orchestra.run.smithi040:> sudo service rsyslog restart 2019-12-05T04:57:19.709 INFO:teuthology.orchestra.run.smithi042:> sudo service rsyslog restart 2019-12-05T04:57:19.761 INFO:teuthology.run_tasks:Running task internal.timer... 2019-12-05T04:57:19.774 INFO:teuthology.task.internal:Starting timer... 2019-12-05T04:57:19.774 INFO:teuthology.run_tasks:Running task pcp... 2019-12-05T04:57:19.831 INFO:teuthology.run_tasks:Running task selinux... 2019-12-05T04:57:19.884 DEBUG:teuthology.task.selinux:Excluding smithi042: OS 'ubuntu' does not support SELinux 2019-12-05T04:57:19.887 DEBUG:teuthology.task.selinux:Excluding smithi040: OS 'ubuntu' does not support SELinux 2019-12-05T04:57:19.887 DEBUG:teuthology.task.selinux:Getting current SELinux state 2019-12-05T04:57:19.889 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2019-12-05T04:57:19.890 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2019-12-05T04:57:19.890 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2019-12-05T04:57:19.905 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master was just updated; assuming it is current 2019-12-05T04:57:19.905 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master to branch origin/master 2019-12-05T04:57:19.976 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}] 2019-12-05T04:57:19.987 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi042.front.sepia.ceph.com,smithi040.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/cephlab.yml 2019-12-05T04:57:20.440 INFO:teuthology.task.ansible.out:Using /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/ansible.cfg as config file 2019-12-05T04:57:24.550 INFO:teuthology.task.ansible.out: PLAY [all] ********************************************************************* 2019-12-05T04:57:24.575 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:57:24 +0000 (0:00:00.078) 0:00:00.078 ***** 2019-12-05T04:57:24.587 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:57:24 +0000 (0:00:00.012) 0:00:00.090 ***** 2019-12-05T04:57:27.359 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi042.front.sepia.ceph.com] 2019-12-05T04:57:27.436 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:57:27 +0000 (0:00:02.848) 0:00:02.939 ***** 2019-12-05T04:57:27.457 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] 2019-12-05T04:57:27.541 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:57:27 +0000 (0:00:00.105) 0:00:03.044 ***** 2019-12-05T04:57:27.730 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the sudo group.] ******************************** ok: [smithi040.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} 2019-12-05T04:57:27.828 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:57:27 +0000 (0:00:00.278) 0:00:03.323 ***** ok: [smithi042.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} 2019-12-05T04:57:27.868 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:57:27 +0000 (0:00:00.047) 0:00:03.370 ***** 2019-12-05T04:57:28.354 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the ansible user.] ****************************** 2019-12-05T04:57:28.361 INFO:teuthology.task.ansible.out:changed: [smithi040.front.sepia.ceph.com] => {"append": false, "changed": true, "comment": "cm,,,", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2019-12-05T04:57:28.421 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:57:28 +0000 (0:00:00.547) 0:00:03.918 ***** 2019-12-05T04:57:28.433 INFO:teuthology.task.ansible.out:changed: [smithi042.front.sepia.ceph.com] => {"append": false, "changed": true, "comment": "cm,,,", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2019-12-05T04:57:28.512 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:57:28 +0000 (0:00:00.080) 0:00:03.999 ***** 2019-12-05T04:57:28.782 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Delete the ansible users password.] ******************** changed: [smithi040.front.sepia.ceph.com] => {"changed": true, "cmd": ["passwd", "-d", "cm"], "delta": "0:00:00.080733", "end": "2019-12-05 04:57:28.379422", "rc": 0, "start": "2019-12-05 04:57:28.298689", "stderr": "", "stderr_lines": [], "stdout": "passwd: password expiry information changed.", "stdout_lines": ["passwd: password expiry information changed."]} 2019-12-05T04:57:28.822 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:57:28 +0000 (0:00:00.326) 0:00:04.325 ***** 2019-12-05T04:57:28.851 INFO:teuthology.task.ansible.out:changed: [smithi042.front.sepia.ceph.com] => {"changed": true, "cmd": ["passwd", "-d", "cm"], "delta": "0:00:00.076321", "end": "2019-12-05 04:57:29.153148", "rc": 0, "start": "2019-12-05 04:57:29.076827", "stderr": "", "stderr_lines": [], "stdout": "passwd: password expiry information changed.", "stdout_lines": ["passwd: password expiry information changed."]} 2019-12-05T04:57:28.871 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:57:28 +0000 (0:00:00.048) 0:00:04.373 ***** 2019-12-05T04:57:29.149 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Ensure includedir is present in sudoers.] ************** ok: [smithi040.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2019-12-05T04:57:29.169 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:57:29 +0000 (0:00:00.298) 0:00:04.671 ***** 2019-12-05T04:57:29.178 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2019-12-05T04:57:29.227 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:57:29 +0000 (0:00:00.058) 0:00:04.730 ***** 2019-12-05T04:57:30.006 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the cephlab_sudo sudoers.d file.] *************** changed: [smithi042.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1575521849.25-169912698600534/source", "state": "file", "uid": 0} 2019-12-05T04:57:30.072 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:57:30 +0000 (0:00:00.844) 0:00:05.575 ***** 2019-12-05T04:57:30.078 INFO:teuthology.task.ansible.out:changed: [smithi040.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1575521849.19-85355523279430/source", "state": "file", "uid": 0} 2019-12-05T04:57:30.125 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:57:30 +0000 (0:00:00.048) 0:00:05.623 ***** 2019-12-05T04:57:30.447 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Add authorized keys for the ansible user.] ************* ok: [smithi042.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} ok: [smithi040.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn 2019-12-05T04:57:30.448 INFO:teuthology.task.ansible.out:0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2019-12-05T04:57:30.565 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2019-12-05T04:57:30.595 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2019-12-05T04:57:30.700 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2019-12-05T04:57:30.767 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2019-12-05T04:57:30.831 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2019-12-05T04:57:30.912 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2019-12-05T04:57:30.969 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2019-12-05T04:57:31.072 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2019-12-05T04:57:31.116 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2019-12-05T04:57:31.219 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2019-12-05T04:57:31.262 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2019-12-05T04:57:31.377 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2019-12-05T04:57:31.485 INFO:teuthology.task.ansible.out: PLAY [teuthology] ************************************************************** skipping: no hosts matched 2019-12-05T04:57:31.490 INFO:teuthology.task.ansible.out: PLAY [testnodes] *************************************************************** 2019-12-05T04:57:31.709 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:57:31 +0000 (0:00:01.588) 0:00:07.212 ***** 2019-12-05T04:57:31.763 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:57:31 +0000 (0:00:00.054) 0:00:07.266 ***** 2019-12-05T04:57:32.496 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi040.front.sepia.ceph.com] 2019-12-05T04:57:32.596 INFO:teuthology.task.ansible.out: [WARNING]: Using run_once with the free strategy is not currently supported. This task will still be executed for every host in the inventory list. Thursday 05 December 2019 04:57:32 +0000 (0:00:00.829) 0:00:08.096 ***** 2019-12-05T04:57:32.615 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] 2019-12-05T04:57:32.760 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:57:32 +0000 (0:00:00.166) 0:00:08.263 ***** 2019-12-05T04:57:33.035 INFO:teuthology.task.ansible.out: 2019-12-05T04:57:33.043 INFO:teuthology.task.ansible.out:TASK [users : Check keys_repo HEAD sha1] *************************************** changed: [smithi040.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.250429", "end": "2019-12-05 04:57:33.004361", "rc": 0, "start": "2019-12-05 04:57:32.753932", "stderr": "", "stderr_lines": [], "stdout": "7b315d00bbff69e651613e1408370c0721c0c099", "stdout_lines": ["7b315d00bbff69e651613e1408370c0721c0c099"]} 2019-12-05T04:57:33.086 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:57:33 +0000 (0:00:00.325) 0:00:08.588 ***** 2019-12-05T04:57:33.213 INFO:teuthology.task.ansible.out:changed: [smithi042.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.237277", "end": "2019-12-05 04:57:33.172841", "rc": 0, "start": "2019-12-05 04:57:32.935564", "stderr": "", "stderr_lines": [], "stdout": "7b315d00bbff69e651613e1408370c0721c0c099", "stdout_lines": ["7b315d00bbff69e651613e1408370c0721c0c099"]} 2019-12-05T04:57:33.269 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:57:33 +0000 (0:00:00.177) 0:00:08.766 ***** 2019-12-05T04:57:33.271 INFO:teuthology.task.ansible.out: TASK [users : Check host's /keys-repo-sha1 sentinel file] ********************** changed: [smithi040.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.019756", "end": "2019-12-05 04:57:32.853094", "failed_when_result": false, "rc": 0, "start": "2019-12-05 04:57:32.833338", "stderr": "", "stderr_lines": [], "stdout": "7b088106a1031364e6869fc2888a0e2542579008", "stdout_lines": ["7b088106a1031364e6869fc2888a0e2542579008"]} 2019-12-05T04:57:33.314 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:57:33 +0000 (0:00:00.050) 0:00:08.816 ***** 2019-12-05T04:57:33.346 INFO:teuthology.task.ansible.out: TASK [users : Determine if we can skip users and pubkeys updates] ************** skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:57:33.387 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:57:33 +0000 (0:00:00.072) 0:00:08.889 ***** 2019-12-05T04:57:33.417 INFO:teuthology.task.ansible.out: TASK [users : Merge extra_admin_users into managed_admin_users] **************** skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:57:33.456 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:57:33 +0000 (0:00:00.066) 0:00:08.956 ***** 2019-12-05T04:57:33.461 INFO:teuthology.task.ansible.out: TASK [users : Check host's /keys-repo-sha1 sentinel file] ********************** changed: [smithi042.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.016282", "end": "2019-12-05 04:57:33.788796", "failed_when_result": false, "rc": 0, "start": "2019-12-05 04:57:33.772514", "stderr": "", "stderr_lines": [], "stdout": "7b088106a1031364e6869fc2888a0e2542579008", "stdout_lines": ["7b088106a1031364e6869fc2888a0e2542579008"]} 2019-12-05T04:57:33.508 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:57:33 +0000 (0:00:00.054) 0:00:09.010 ***** 2019-12-05T04:57:33.514 INFO:teuthology.task.ansible.out: TASK [users : Remove managed_admin_users from managed_users] ******************* 2019-12-05T04:57:33.516 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:57:33.558 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:57:33 +0000 (0:00:00.049) 0:00:09.060 ***** 2019-12-05T04:57:33.565 INFO:teuthology.task.ansible.out: TASK [users : Determine if we can skip users and pubkeys updates] ************** skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:57:33.609 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:57:33 +0000 (0:00:00.051) 0:00:09.111 ***** 2019-12-05T04:57:33.618 INFO:teuthology.task.ansible.out: TASK [users : Filter the managed_users list] *********************************** skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:57:33.659 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:57:33 +0000 (0:00:00.048) 0:00:09.160 ***** 2019-12-05T04:57:33.664 INFO:teuthology.task.ansible.out: TASK [users : Merge extra_admin_users into managed_admin_users] **************** skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:57:33.717 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:57:33 +0000 (0:00:00.059) 0:00:09.219 ***** 2019-12-05T04:57:33.723 INFO:teuthology.task.ansible.out: TASK [users : Filter the managed_admin_users list] ***************************** skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:57:33.771 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:57:33 +0000 (0:00:00.053) 0:00:09.273 ***** 2019-12-05T04:57:33.779 INFO:teuthology.task.ansible.out: TASK [users : Remove managed_admin_users from managed_users] ******************* skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:57:33.820 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:57:33 +0000 (0:00:00.049) 0:00:09.323 ***** 2019-12-05T04:57:33.878 INFO:teuthology.task.ansible.out: TASK [users : Filter the managed_users list] *********************************** 2019-12-05T04:57:33.880 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:57:33.922 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:57:33 +0000 (0:00:00.101) 0:00:09.424 ***** 2019-12-05T04:57:33.928 INFO:teuthology.task.ansible.out: TASK [users : Set UID_MIN to 1001] ********************************************* 2019-12-05T04:57:33.935 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2019-12-05T04:57:33.972 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:57:33 +0000 (0:00:00.050) 0:00:09.475 ***** 2019-12-05T04:57:33.980 INFO:teuthology.task.ansible.out: TASK [users : Filter the managed_admin_users list] ***************************** skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:57:34.031 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:57:34 +0000 (0:00:00.058) 0:00:09.533 ***** 2019-12-05T04:57:34.174 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9', u'name': u'andrewschoen'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1101, "groups": "sudo", "home": "/home/andrewschoen", "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "move_home": false, "name": "andrewschoen", "shell": "/bin/bash", "state": "present", "uid": 1101} 2019-12-05T04:57:34.183 INFO:teuthology.task.ansible.out: TASK [users : Set UID_MIN to 1001] ********************************************* 2019-12-05T04:57:34.185 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2019-12-05T04:57:34.240 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:57:34 +0000 (0:00:00.204) 0:00:09.738 ***** 2019-12-05T04:57:34.305 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n', u'name': u'zack'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1102, "groups": "sudo", "home": "/home/zack", "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "move_home": false, "name": "zack", "shell": "/bin/bash", "state": "present", "uid": 1102} 2019-12-05T04:57:34.449 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9', u'name': u'andrewschoen'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1101, "groups": "sudo", "home": "/home/andrewschoen", "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "move_home": false, "name": "andrewschoen", "shell": "/bin/bash", "state": "present", "uid": 1101} 2019-12-05T04:57:34.486 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd', u'name': u'kdreyer'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1103, "groups": "sudo", "home": "/home/kdreyer", "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "move_home": false, "name": "kdreyer", "shell": "/bin/bash", "state": "present", "uid": 1103} 2019-12-05T04:57:34.586 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n', u'name': u'zack'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1102, "groups": "sudo", "home": "/home/zack", "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "move_home": false, "name": "zack", "shell": "/bin/bash", "state": "present", "uid": 1102} 2019-12-05T04:57:34.628 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'alfredo@papaya.local V5VkRT2Du88WwQrYeK196w 4dff4d9bdc4d9ab60532814c338abe4534cb45a9ca729bdbc2411ad513516b47', u'name': u'adeza'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1104, "groups": "sudo", "home": "/home/adeza", "item": {"name": "adeza", "ovpn": "alfredo@papaya.local V5VkRT2Du88WwQrYeK196w 4dff4d9bdc4d9ab60532814c338abe4534cb45a9ca729bdbc2411ad513516b47"}, "move_home": false, "name": "adeza", "shell": "/bin/bash", "state": "present", "uid": 1104} 2019-12-05T04:57:34.756 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd', u'name': u'kdreyer'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1103, "groups": "sudo", "home": "/home/kdreyer", "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "move_home": false, "name": "kdreyer", "shell": "/bin/bash", "state": "present", "uid": 1103} 2019-12-05T04:57:34.757 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n', u'name': u'dmick'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1105, "groups": "sudo", "home": "/home/dmick", "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n"}, "move_home": false, "name": "dmick", "shell": "/bin/bash", "state": "present", "uid": 1105} 2019-12-05T04:57:34.898 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n', u'name': u'sage'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1106, "groups": "sudo", "home": "/home/sage", "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "move_home": false, "name": "sage", "shell": "/bin/bash", "state": "present", "uid": 1106} ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'alfredo@papaya.local V5VkRT2Du88WwQrYeK196w 4dff4d9bdc4d9ab60532814c338abe4534cb45a9ca729bdbc2411ad513516b47', u'name': u'adeza'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1104, "groups": "sudo", "home": "/home/adeza", "item": {"name": "adeza", "ovpn": "alfredo@papaya.local V5VkRT2Du88WwQrYeK196w 4dff4d9bdc4d9ab60532814c338abe4534cb45a9ca729bdbc2411ad513516b47"}, "move_home": false, "name": "adeza", "shell": "/bin/bash", "state": "present", "uid": 1104} 2019-12-05T04:57:35.030 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n', u'name': u'dgalloway'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1107, "groups": "sudo", "home": "/home/dgalloway", "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "move_home": false, "name": "dgalloway", "shell": "/bin/bash", "state": "present", "uid": 1107} 2019-12-05T04:57:35.080 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n', u'name': u'dmick'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1105, "groups": "sudo", "home": "/home/dmick", "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n"}, "move_home": false, "name": "dmick", "shell": "/bin/bash", "state": "present", "uid": 1105} 2019-12-05T04:57:35.169 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045', u'name': u'akraitma'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1108, "groups": "sudo", "home": "/home/akraitma", "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "move_home": false, "name": "akraitma", "shell": "/bin/bash", "state": "present", "uid": 1108} 2019-12-05T04:57:35.202 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n', u'name': u'sage'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1106, "groups": "sudo", "home": "/home/sage", "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "move_home": false, "name": "sage", "shell": "/bin/bash", "state": "present", "uid": 1106} 2019-12-05T04:57:35.322 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e', u'name': u'trociny'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1109, "groups": "sudo", "home": "/home/trociny", "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "move_home": false, "name": "trociny", "shell": "/bin/bash", "state": "present", "uid": 1109} 2019-12-05T04:57:35.376 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n', u'name': u'dgalloway'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1107, "groups": "sudo", "home": "/home/dgalloway", "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "move_home": false, "name": "dgalloway", "shell": "/bin/bash", "state": "present", "uid": 1107} 2019-12-05T04:57:35.460 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007', u'name': u'smithfarm', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1110, "groups": "sudo", "home": "/home/smithfarm", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "move_home": false, "name": "smithfarm", "shell": "/bin/bash", "state": "present", "uid": 1110} 2019-12-05T04:57:35.508 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045', u'name': u'akraitma'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1108, "groups": "sudo", "home": "/home/akraitma", "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "move_home": false, "name": "akraitma", "shell": "/bin/bash", "state": "present", "uid": 1108} 2019-12-05T04:57:35.649 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e', u'name': u'trociny'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1109, "groups": "sudo", "home": "/home/trociny", "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "move_home": false, "name": "trociny", "shell": "/bin/bash", "state": "present", "uid": 1109} 2019-12-05T04:57:35.655 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n', u'name': u'mbenjamin'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1111, "groups": "sudo", "home": "/home/mbenjamin", "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "move_home": false, "name": "mbenjamin", "shell": "/bin/bash", "state": "present", "uid": 1111} 2019-12-05T04:57:35.807 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01', u'name': u'aemerson', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1112, "groups": "sudo", "home": "/home/aemerson", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, "move_home": false, "name": "aemerson", "shell": "/bin/bash", "state": "present", "uid": 1112} 2019-12-05T04:57:35.808 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007', u'name': u'smithfarm', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1110, "groups": "sudo", "home": "/home/smithfarm", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "move_home": false, "name": "smithfarm", "shell": "/bin/bash", "state": "present", "uid": 1110} 2019-12-05T04:57:35.958 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78', u'name': u'amaredia', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1114, "groups": "sudo", "home": "/home/amaredia", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "move_home": false, "name": "amaredia", "shell": "/bin/bash", "state": "present", "uid": 1114} 2019-12-05T04:57:35.960 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n', u'name': u'mbenjamin'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1111, "groups": "sudo", "home": "/home/mbenjamin", "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "move_home": false, "name": "mbenjamin", "shell": "/bin/bash", "state": "present", "uid": 1111} 2019-12-05T04:57:36.094 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5', u'name': u'tserlin', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1115, "groups": "sudo", "home": "/home/tserlin", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "move_home": false, "name": "tserlin", "shell": "/bin/bash", "state": "present", "uid": 1115} 2019-12-05T04:57:36.107 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01', u'name': u'aemerson', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1112, "groups": "sudo", "home": "/home/aemerson", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, "move_home": false, "name": "aemerson", "shell": "/bin/bash", "state": "present", "uid": 1112} 2019-12-05T04:57:36.276 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28', u'name': u'dis'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1116, "groups": "sudo", "home": "/home/dis", "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "move_home": false, "name": "dis", "shell": "/bin/bash", "state": "present", "uid": 1116} 2019-12-05T04:57:36.288 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78', u'name': u'amaredia', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1114, "groups": "sudo", "home": "/home/amaredia", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "move_home": false, "name": "amaredia", "shell": "/bin/bash", "state": "present", "uid": 1114} 2019-12-05T04:57:36.422 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n', u'name': u'gregf'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1117, "groups": "sudo", "home": "/home/gregf", "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "move_home": false, "name": "gregf", "shell": "/bin/bash", "state": "present", "uid": 1117} 2019-12-05T04:57:36.434 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5', u'name': u'tserlin', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1115, "groups": "sudo", "home": "/home/tserlin", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "move_home": false, "name": "tserlin", "shell": "/bin/bash", "state": "present", "uid": 1115} 2019-12-05T04:57:36.573 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28', u'name': u'dis'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1116, "groups": "sudo", "home": "/home/dis", "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "move_home": false, "name": "dis", "shell": "/bin/bash", "state": "present", "uid": 1116} 2019-12-05T04:57:36.575 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n', u'name': u'joshd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1118, "groups": "sudo", "home": "/home/joshd", "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "move_home": false, "name": "joshd", "shell": "/bin/bash", "state": "present", "uid": 1118} 2019-12-05T04:57:36.705 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n', u'name': u'gregf'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1117, "groups": "sudo", "home": "/home/gregf", "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "move_home": false, "name": "gregf", "shell": "/bin/bash", "state": "present", "uid": 1117} 2019-12-05T04:57:36.708 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n', u'name': u'davidz'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1119, "groups": "sudo", "home": "/home/davidz", "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "move_home": false, "name": "davidz", "shell": "/bin/bash", "state": "present", "uid": 1119} 2019-12-05T04:57:36.843 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n', u'name': u'joshd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1118, "groups": "sudo", "home": "/home/joshd", "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "move_home": false, "name": "joshd", "shell": "/bin/bash", "state": "present", "uid": 1118} 2019-12-05T04:57:36.852 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc', u'name': u'gmeno'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1120, "groups": "sudo", "home": "/home/gmeno", "item": {"name": "gmeno", "ovpn": "gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc"}, "move_home": false, "name": "gmeno", "shell": "/bin/bash", "state": "present", "uid": 1120} 2019-12-05T04:57:36.984 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n', u'name': u'davidz'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1119, "groups": "sudo", "home": "/home/davidz", "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "move_home": false, "name": "davidz", "shell": "/bin/bash", "state": "present", "uid": 1119} 2019-12-05T04:57:37.020 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1', u'name': u'ivancich'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1121, "groups": "sudo", "home": "/home/ivancich", "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "move_home": false, "name": "ivancich", "shell": "/bin/bash", "state": "present", "uid": 1121} 2019-12-05T04:57:37.125 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc', u'name': u'gmeno'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1120, "groups": "sudo", "home": "/home/gmeno", "item": {"name": "gmeno", "ovpn": "gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc"}, "move_home": false, "name": "gmeno", "shell": "/bin/bash", "state": "present", "uid": 1120} 2019-12-05T04:57:37.158 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n', u'name': u'wusui'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1122, "groups": "sudo", "home": "/home/wusui", "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "move_home": false, "name": "wusui", "shell": "/bin/bash", "state": "present", "uid": 1122} 2019-12-05T04:57:37.279 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1', u'name': u'ivancich'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1121, "groups": "sudo", "home": "/home/ivancich", "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "move_home": false, "name": "ivancich", "shell": "/bin/bash", "state": "present", "uid": 1121} 2019-12-05T04:57:37.311 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304', u'name': u'zyan'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1123, "groups": "sudo", "home": "/home/zyan", "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "move_home": false, "name": "zyan", "shell": "/bin/bash", "state": "present", "uid": 1123} 2019-12-05T04:57:37.425 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n', u'name': u'wusui'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1122, "groups": "sudo", "home": "/home/wusui", "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "move_home": false, "name": "wusui", "shell": "/bin/bash", "state": "present", "uid": 1122} 2019-12-05T04:57:37.465 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n', u'name': u'yuriw'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1124, "groups": "sudo", "home": "/home/yuriw", "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n"}, "move_home": false, "name": "yuriw", "shell": "/bin/bash", "state": "present", "uid": 1124} 2019-12-05T04:57:37.586 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304', u'name': u'zyan'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1123, "groups": "sudo", "home": "/home/zyan", "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "move_home": false, "name": "zyan", "shell": "/bin/bash", "state": "present", "uid": 1123} 2019-12-05T04:57:37.611 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n', u'name': u'tamil'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1125, "groups": "sudo", "home": "/home/tamil", "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "move_home": false, "name": "tamil", "shell": "/bin/bash", "state": "present", "uid": 1125} 2019-12-05T04:57:37.746 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n', u'name': u'yuriw'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1124, "groups": "sudo", "home": "/home/yuriw", "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n"}, "move_home": false, "name": "yuriw", "shell": "/bin/bash", "state": "present", "uid": 1124} 2019-12-05T04:57:37.769 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n', u'name': u'jowilkin'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1126, "groups": "sudo", "home": "/home/jowilkin", "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "move_home": false, "name": "jowilkin", "shell": "/bin/bash", "state": "present", "uid": 1126} 2019-12-05T04:57:37.908 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n', u'name': u'tamil'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1125, "groups": "sudo", "home": "/home/tamil", "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "move_home": false, "name": "tamil", "shell": "/bin/bash", "state": "present", "uid": 1125} 2019-12-05T04:57:37.953 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4', u'name': u'bhubbard'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1127, "groups": "sudo", "home": "/home/bhubbard", "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "move_home": false, "name": "bhubbard", "shell": "/bin/bash", "state": "present", "uid": 1127} 2019-12-05T04:57:38.044 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n', u'name': u'jowilkin'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1126, "groups": "sudo", "home": "/home/jowilkin", "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "move_home": false, "name": "jowilkin", "shell": "/bin/bash", "state": "present", "uid": 1126} 2019-12-05T04:57:38.089 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n', u'name': u'yehudasa'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1128, "groups": "sudo", "home": "/home/yehudasa", "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "move_home": false, "name": "yehudasa", "shell": "/bin/bash", "state": "present", "uid": 1128} 2019-12-05T04:57:38.185 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4', u'name': u'bhubbard'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1127, "groups": "sudo", "home": "/home/bhubbard", "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "move_home": false, "name": "bhubbard", "shell": "/bin/bash", "state": "present", "uid": 1127} 2019-12-05T04:57:38.241 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4', u'name': u'dang'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1129, "groups": "sudo", "home": "/home/dang", "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "move_home": false, "name": "dang", "shell": "/bin/bash", "state": "present", "uid": 1129} 2019-12-05T04:57:38.326 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n', u'name': u'yehudasa'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1128, "groups": "sudo", "home": "/home/yehudasa", "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "move_home": false, "name": "yehudasa", "shell": "/bin/bash", "state": "present", "uid": 1128} 2019-12-05T04:57:38.391 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb', u'name': u'branto'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1130, "groups": "sudo", "home": "/home/branto", "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "move_home": false, "name": "branto", "shell": "/bin/bash", "state": "present", "uid": 1130} 2019-12-05T04:57:38.466 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4', u'name': u'dang'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1129, "groups": "sudo", "home": "/home/dang", "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "move_home": false, "name": "dang", "shell": "/bin/bash", "state": "present", "uid": 1129} 2019-12-05T04:57:38.541 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6', u'name': u'xiaoxichen'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1131, "groups": "sudo", "home": "/home/xiaoxichen", "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "move_home": false, "name": "xiaoxichen", "shell": "/bin/bash", "state": "present", "uid": 1131} 2019-12-05T04:57:38.609 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb', u'name': u'branto'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1130, "groups": "sudo", "home": "/home/branto", "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "move_home": false, "name": "branto", "shell": "/bin/bash", "state": "present", "uid": 1130} 2019-12-05T04:57:38.706 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e', u'name': u'ffilz'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1132, "groups": "sudo", "home": "/home/ffilz", "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "move_home": false, "name": "ffilz", "shell": "/bin/bash", "state": "present", "uid": 1132} 2019-12-05T04:57:38.772 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6', u'name': u'xiaoxichen'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1131, "groups": "sudo", "home": "/home/xiaoxichen", "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "move_home": false, "name": "xiaoxichen", "shell": "/bin/bash", "state": "present", "uid": 1131} 2019-12-05T04:57:38.866 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n', u'name': u'joao'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1133, "groups": "sudo", "home": "/home/joao", "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "move_home": false, "name": "joao", "shell": "/bin/bash", "state": "present", "uid": 1133} 2019-12-05T04:57:38.923 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e', u'name': u'ffilz'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1132, "groups": "sudo", "home": "/home/ffilz", "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "move_home": false, "name": "ffilz", "shell": "/bin/bash", "state": "present", "uid": 1132} 2019-12-05T04:57:38.980 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n', u'name': u'nhm'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1134, "groups": "sudo", "home": "/home/nhm", "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "move_home": false, "name": "nhm", "shell": "/bin/bash", "state": "present", "uid": 1134} 2019-12-05T04:57:39.076 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n', u'name': u'joao'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1133, "groups": "sudo", "home": "/home/joao", "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "move_home": false, "name": "joao", "shell": "/bin/bash", "state": "present", "uid": 1133} 2019-12-05T04:57:39.130 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313', u'name': u'nwatkins'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1136, "groups": "sudo", "home": "/home/nwatkins", "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "move_home": false, "name": "nwatkins", "shell": "/bin/bash", "state": "present", "uid": 1136} 2019-12-05T04:57:39.243 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n', u'name': u'nhm'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1134, "groups": "sudo", "home": "/home/nhm", "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "move_home": false, "name": "nhm", "shell": "/bin/bash", "state": "present", "uid": 1134} 2019-12-05T04:57:39.295 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22', u'name': u'mkidd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1137, "groups": "sudo", "home": "/home/mkidd", "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "move_home": false, "name": "mkidd", "shell": "/bin/bash", "state": "present", "uid": 1137} 2019-12-05T04:57:39.441 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313', u'name': u'nwatkins'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1136, "groups": "sudo", "home": "/home/nwatkins", "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "move_home": false, "name": "nwatkins", "shell": "/bin/bash", "state": "present", "uid": 1136} 2019-12-05T04:57:39.478 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af', u'name': u'jlopez'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/jlopez", "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "move_home": false, "name": "jlopez", "shell": "/bin/bash", "state": "present", "uid": 1138} 2019-12-05T04:57:39.625 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22', u'name': u'mkidd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1137, "groups": "sudo", "home": "/home/mkidd", "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "move_home": false, "name": "mkidd", "shell": "/bin/bash", "state": "present", "uid": 1137} 2019-12-05T04:57:39.660 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb', u'name': u'haomaiwang'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/haomaiwang", "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "move_home": false, "name": "haomaiwang", "shell": "/bin/bash", "state": "present", "uid": 1139} 2019-12-05T04:57:39.780 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af', u'name': u'jlopez'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/jlopez", "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "move_home": false, "name": "jlopez", "shell": "/bin/bash", "state": "present", "uid": 1138} 2019-12-05T04:57:39.783 INFO:teuthology.task.ansible.out: 2019-12-05T04:57:39.800 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608', u'name': u'jdillaman'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/jdillaman", "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "move_home": false, "name": "jdillaman", "shell": "/bin/bash", "state": "present", "uid": 1140} 2019-12-05T04:57:39.938 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb', u'name': u'haomaiwang'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/haomaiwang", "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "move_home": false, "name": "haomaiwang", "shell": "/bin/bash", "state": "present", "uid": 1139} 2019-12-05T04:57:39.976 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3', u'name': u'kchai'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1141, "groups": "sudo", "home": "/home/kchai", "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "move_home": false, "name": "kchai", "shell": "/bin/bash", "state": "present", "uid": 1141} 2019-12-05T04:57:40.094 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608', u'name': u'jdillaman'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/jdillaman", "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "move_home": false, "name": "jdillaman", "shell": "/bin/bash", "state": "present", "uid": 1140} 2019-12-05T04:57:40.125 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1142, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1142} 2019-12-05T04:57:40.257 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3', u'name': u'kchai'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1141, "groups": "sudo", "home": "/home/kchai", "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "move_home": false, "name": "kchai", "shell": "/bin/bash", "state": "present", "uid": 1141} 2019-12-05T04:57:40.291 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n', u'name': u'dfuller'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1143, "groups": "sudo", "home": "/home/dfuller", "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "move_home": false, "name": "dfuller", "shell": "/bin/bash", "state": "present", "uid": 1143} 2019-12-05T04:57:40.406 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1142, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1142} 2019-12-05T04:57:40.444 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n', u'name': u'owasserm'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1144, "groups": "sudo", "home": "/home/owasserm", "item": {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, "move_home": false, "name": "owasserm", "shell": "/bin/bash", "state": "present", "uid": 1144} 2019-12-05T04:57:40.594 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n', u'name': u'dfuller'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1143, "groups": "sudo", "home": "/home/dfuller", "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "move_home": false, "name": "dfuller", "shell": "/bin/bash", "state": "present", "uid": 1143} 2019-12-05T04:57:40.596 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n', u'name': u'vasu'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/vasu", "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "move_home": false, "name": "vasu", "shell": "/bin/bash", "state": "present", "uid": 1146} 2019-12-05T04:57:40.755 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37', u'name': u'smohan'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/smohan", "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "move_home": false, "name": "smohan", "shell": "/bin/bash", "state": "present", "uid": 1147} ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n', u'name': u'owasserm'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1144, "groups": "sudo", "home": "/home/owasserm", "item": {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, "move_home": false, "name": "owasserm", "shell": "/bin/bash", "state": "present", "uid": 1144} 2019-12-05T04:57:40.925 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114', u'name': u'abhi'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/abhi", "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "move_home": false, "name": "abhi", "shell": "/bin/bash", "state": "present", "uid": 1148} 2019-12-05T04:57:40.936 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n', u'name': u'vasu'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/vasu", "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "move_home": false, "name": "vasu", "shell": "/bin/bash", "state": "present", "uid": 1146} 2019-12-05T04:57:41.070 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611', u'name': u'cbodley'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1150, "groups": "sudo", "home": "/home/cbodley", "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "move_home": false, "name": "cbodley", "shell": "/bin/bash", "state": "present", "uid": 1150} 2019-12-05T04:57:41.121 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37', u'name': u'smohan'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/smohan", "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "move_home": false, "name": "smohan", "shell": "/bin/bash", "state": "present", "uid": 1147} 2019-12-05T04:57:41.276 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195', u'name': u'mwatts'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1153, "groups": "sudo", "home": "/home/mwatts", "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "move_home": false, "name": "mwatts", "shell": "/bin/bash", "state": "present", "uid": 1153} 2019-12-05T04:57:41.310 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114', u'name': u'abhi'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/abhi", "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "move_home": false, "name": "abhi", "shell": "/bin/bash", "state": "present", "uid": 1148} 2019-12-05T04:57:41.415 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1155, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1155} 2019-12-05T04:57:41.461 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611', u'name': u'cbodley'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1150, "groups": "sudo", "home": "/home/cbodley", "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "move_home": false, "name": "cbodley", "shell": "/bin/bash", "state": "present", "uid": 1150} 2019-12-05T04:57:41.584 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea', u'name': u'pdonnell'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1156, "groups": "sudo", "home": "/home/pdonnell", "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "move_home": false, "name": "pdonnell", "shell": "/bin/bash", "state": "present", "uid": 1156} 2019-12-05T04:57:41.616 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195', u'name': u'mwatts'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1153, "groups": "sudo", "home": "/home/mwatts", "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "move_home": false, "name": "mwatts", "shell": "/bin/bash", "state": "present", "uid": 1153} 2019-12-05T04:57:41.745 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868', u'name': u'jlayton'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1157, "groups": "sudo", "home": "/home/jlayton", "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "move_home": false, "name": "jlayton", "shell": "/bin/bash", "state": "present", "uid": 1157} 2019-12-05T04:57:41.817 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1155, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1155} 2019-12-05T04:57:41.940 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f', u'name': u'rzarzynski'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1158, "groups": "sudo", "home": "/home/rzarzynski", "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "move_home": false, "name": "rzarzynski", "shell": "/bin/bash", "state": "present", "uid": 1158} 2019-12-05T04:57:41.964 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea', u'name': u'pdonnell'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1156, "groups": "sudo", "home": "/home/pdonnell", "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "move_home": false, "name": "pdonnell", "shell": "/bin/bash", "state": "present", "uid": 1156} 2019-12-05T04:57:42.081 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e', u'name': u'rdias'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1159, "groups": "sudo", "home": "/home/rdias", "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "move_home": false, "name": "rdias", "shell": "/bin/bash", "state": "present", "uid": 1159} 2019-12-05T04:57:42.138 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868', u'name': u'jlayton'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1157, "groups": "sudo", "home": "/home/jlayton", "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "move_home": false, "name": "jlayton", "shell": "/bin/bash", "state": "present", "uid": 1157} 2019-12-05T04:57:42.283 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953', u'name': u'vshankar'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1161, "groups": "sudo", "home": "/home/vshankar", "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, "move_home": false, "name": "vshankar", "shell": "/bin/bash", "state": "present", "uid": 1161} 2019-12-05T04:57:42.290 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f', u'name': u'rzarzynski'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1158, "groups": "sudo", "home": "/home/rzarzynski", "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "move_home": false, "name": "rzarzynski", "shell": "/bin/bash", "state": "present", "uid": 1158} 2019-12-05T04:57:42.468 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e', u'name': u'rdias'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1159, "groups": "sudo", "home": "/home/rdias", "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "move_home": false, "name": "rdias", "shell": "/bin/bash", "state": "present", "uid": 1159} ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58', u'name': u'akupczyk'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1162, "groups": "sudo", "home": "/home/akupczyk", "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "move_home": false, "name": "akupczyk", "shell": "/bin/bash", "state": "present", "uid": 1162} 2019-12-05T04:57:42.640 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e', u'name': u'nojha'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1163, "groups": "sudo", "home": "/home/nojha", "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e"}, "move_home": false, "name": "nojha", "shell": "/bin/bash", "state": "present", "uid": 1163} 2019-12-05T04:57:42.659 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953', u'name': u'vshankar'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1161, "groups": "sudo", "home": "/home/vshankar", "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, "move_home": false, "name": "vshankar", "shell": "/bin/bash", "state": "present", "uid": 1161} 2019-12-05T04:57:42.830 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58', u'name': u'akupczyk'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1162, "groups": "sudo", "home": "/home/akupczyk", "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "move_home": false, "name": "akupczyk", "shell": "/bin/bash", "state": "present", "uid": 1162} 2019-12-05T04:57:42.832 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43', u'name': u'ifed01'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1164, "groups": "sudo", "home": "/home/ifed01", "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "move_home": false, "name": "ifed01", "shell": "/bin/bash", "state": "present", "uid": 1164} 2019-12-05T04:57:43.028 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4', u'name': u'myoungwon'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1165, "groups": "sudo", "home": "/home/myoungwon", "item": {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, "move_home": false, "name": "myoungwon", "shell": "/bin/bash", "state": "present", "uid": 1165} 2019-12-05T04:57:43.041 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e', u'name': u'nojha'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1163, "groups": "sudo", "home": "/home/nojha", "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e"}, "move_home": false, "name": "nojha", "shell": "/bin/bash", "state": "present", "uid": 1163} 2019-12-05T04:57:43.181 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2', u'name': u'gabrioux'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1167, "groups": "sudo", "home": "/home/gabrioux", "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "move_home": false, "name": "gabrioux", "shell": "/bin/bash", "state": "present", "uid": 1167} 2019-12-05T04:57:43.215 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43', u'name': u'ifed01'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1164, "groups": "sudo", "home": "/home/ifed01", "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "move_home": false, "name": "ifed01", "shell": "/bin/bash", "state": "present", "uid": 1164} 2019-12-05T04:57:43.337 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f', u'name': u'leseb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1168, "groups": "sudo", "home": "/home/leseb", "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "move_home": false, "name": "leseb", "shell": "/bin/bash", "state": "present", "uid": 1168} 2019-12-05T04:57:43.379 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4', u'name': u'myoungwon'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1165, "groups": "sudo", "home": "/home/myoungwon", "item": {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, "move_home": false, "name": "myoungwon", "shell": "/bin/bash", "state": "present", "uid": 1165} 2019-12-05T04:57:43.472 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca', u'name': u'hchen'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1169, "groups": "sudo", "home": "/home/hchen", "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "move_home": false, "name": "hchen", "shell": "/bin/bash", "state": "present", "uid": 1169} 2019-12-05T04:57:43.521 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2', u'name': u'gabrioux'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1167, "groups": "sudo", "home": "/home/gabrioux", "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "move_home": false, "name": "gabrioux", "shell": "/bin/bash", "state": "present", "uid": 1167} 2019-12-05T04:57:43.638 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n', u'name': u'jcollin'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1170, "groups": "sudo", "home": "/home/jcollin", "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "move_home": false, "name": "jcollin", "shell": "/bin/bash", "state": "present", "uid": 1170} 2019-12-05T04:57:43.660 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f', u'name': u'leseb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1168, "groups": "sudo", "home": "/home/leseb", "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "move_home": false, "name": "leseb", "shell": "/bin/bash", "state": "present", "uid": 1168} 2019-12-05T04:57:43.780 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc', u'name': u'xxg'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1171, "groups": "sudo", "home": "/home/xxg", "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "move_home": false, "name": "xxg", "shell": "/bin/bash", "state": "present", "uid": 1171} 2019-12-05T04:57:43.814 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca', u'name': u'hchen'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1169, "groups": "sudo", "home": "/home/hchen", "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "move_home": false, "name": "hchen", "shell": "/bin/bash", "state": "present", "uid": 1169} 2019-12-05T04:57:43.922 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53', u'name': u'pcuzner'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1172, "groups": "sudo", "home": "/home/pcuzner", "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "move_home": false, "name": "pcuzner", "shell": "/bin/bash", "state": "present", "uid": 1172} 2019-12-05T04:57:43.977 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n', u'name': u'jcollin'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1170, "groups": "sudo", "home": "/home/jcollin", "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "move_home": false, "name": "jcollin", "shell": "/bin/bash", "state": "present", "uid": 1170} 2019-12-05T04:57:44.074 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2', u'name': u'mkogan'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1174, "groups": "sudo", "home": "/home/mkogan", "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "move_home": false, "name": "mkogan", "shell": "/bin/bash", "state": "present", "uid": 1174} 2019-12-05T04:57:44.131 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc', u'name': u'xxg'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1171, "groups": "sudo", "home": "/home/xxg", "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "move_home": false, "name": "xxg", "shell": "/bin/bash", "state": "present", "uid": 1171} 2019-12-05T04:57:44.211 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1142, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1142} 2019-12-05T04:57:44.268 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53', u'name': u'pcuzner'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1172, "groups": "sudo", "home": "/home/pcuzner", "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "move_home": false, "name": "pcuzner", "shell": "/bin/bash", "state": "present", "uid": 1172} 2019-12-05T04:57:44.375 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736', u'name': u'henrix'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1177, "groups": "sudo", "home": "/home/henrix", "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "move_home": false, "name": "henrix", "shell": "/bin/bash", "state": "present", "uid": 1177} 2019-12-05T04:57:44.413 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2', u'name': u'mkogan'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1174, "groups": "sudo", "home": "/home/mkogan", "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "move_home": false, "name": "mkogan", "shell": "/bin/bash", "state": "present", "uid": 1174} 2019-12-05T04:57:44.523 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26', u'name': u'pbs1108'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1178, "groups": "sudo", "home": "/home/pbs1108", "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "move_home": false, "name": "pbs1108", "shell": "/bin/bash", "state": "present", "uid": 1178} 2019-12-05T04:57:44.577 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1142, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1142} 2019-12-05T04:57:44.660 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315', u'name': u'clacroix'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1179, "groups": "sudo", "home": "/home/clacroix", "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "move_home": false, "name": "clacroix", "shell": "/bin/bash", "state": "present", "uid": 1179} 2019-12-05T04:57:44.713 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736', u'name': u'henrix'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1177, "groups": "sudo", "home": "/home/henrix", "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "move_home": false, "name": "henrix", "shell": "/bin/bash", "state": "present", "uid": 1177} 2019-12-05T04:57:44.832 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006', u'name': u'epuertat'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1180, "groups": "sudo", "home": "/home/epuertat", "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "move_home": false, "name": "epuertat", "shell": "/bin/bash", "state": "present", "uid": 1180} 2019-12-05T04:57:44.854 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26', u'name': u'pbs1108'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1178, "groups": "sudo", "home": "/home/pbs1108", "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "move_home": false, "name": "pbs1108", "shell": "/bin/bash", "state": "present", "uid": 1178} 2019-12-05T04:57:44.988 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461', u'name': u'tdehler'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1181, "groups": "sudo", "home": "/home/tdehler", "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "move_home": false, "name": "tdehler", "shell": "/bin/bash", "state": "present", "uid": 1181} 2019-12-05T04:57:45.028 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315', u'name': u'clacroix'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1179, "groups": "sudo", "home": "/home/clacroix", "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "move_home": false, "name": "clacroix", "shell": "/bin/bash", "state": "present", "uid": 1179} 2019-12-05T04:57:45.143 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305', u'name': u'laura'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1182, "groups": "sudo", "home": "/home/laura", "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "move_home": false, "name": "laura", "shell": "/bin/bash", "state": "present", "uid": 1182} 2019-12-05T04:57:45.164 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006', u'name': u'epuertat'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1180, "groups": "sudo", "home": "/home/epuertat", "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "move_home": false, "name": "epuertat", "shell": "/bin/bash", "state": "present", "uid": 1180} 2019-12-05T04:57:45.278 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce', u'name': u'yaarit'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1184, "groups": "sudo", "home": "/home/yaarit", "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "move_home": false, "name": "yaarit", "shell": "/bin/bash", "state": "present", "uid": 1184} 2019-12-05T04:57:45.318 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461', u'name': u'tdehler'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1181, "groups": "sudo", "home": "/home/tdehler", "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "move_home": false, "name": "tdehler", "shell": "/bin/bash", "state": "present", "uid": 1181} 2019-12-05T04:57:45.458 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3', u'name': u'rpavani1998'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1185, "groups": "sudo", "home": "/home/rpavani1998", "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "move_home": false, "name": "rpavani1998", "shell": "/bin/bash", "state": "present", "uid": 1185} 2019-12-05T04:57:45.494 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305', u'name': u'laura'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1182, "groups": "sudo", "home": "/home/laura", "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "move_home": false, "name": "laura", "shell": "/bin/bash", "state": "present", "uid": 1182} 2019-12-05T04:57:45.620 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4', u'name': u'rishabh'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1186, "groups": "sudo", "home": "/home/rishabh", "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "move_home": false, "name": "rishabh", "shell": "/bin/bash", "state": "present", "uid": 1186} 2019-12-05T04:57:45.649 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce', u'name': u'yaarit'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1184, "groups": "sudo", "home": "/home/yaarit", "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "move_home": false, "name": "yaarit", "shell": "/bin/bash", "state": "present", "uid": 1184} 2019-12-05T04:57:45.793 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2', u'name': u'skrah'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1187, "groups": "sudo", "home": "/home/skrah", "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "move_home": false, "name": "skrah", "shell": "/bin/bash", "state": "present", "uid": 1187} 2019-12-05T04:57:45.800 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3', u'name': u'rpavani1998'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1185, "groups": "sudo", "home": "/home/rpavani1998", "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "move_home": false, "name": "rpavani1998", "shell": "/bin/bash", "state": "present", "uid": 1185} 2019-12-05T04:57:45.933 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8', u'name': u'smanjara'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1188, "groups": "sudo", "home": "/home/smanjara", "item": {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, "move_home": false, "name": "smanjara", "shell": "/bin/bash", "state": "present", "uid": 1188} 2019-12-05T04:57:45.946 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4', u'name': u'rishabh'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1186, "groups": "sudo", "home": "/home/rishabh", "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "move_home": false, "name": "rishabh", "shell": "/bin/bash", "state": "present", "uid": 1186} 2019-12-05T04:57:46.087 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2', u'name': u'skrah'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1187, "groups": "sudo", "home": "/home/skrah", "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "move_home": false, "name": "skrah", "shell": "/bin/bash", "state": "present", "uid": 1187} 2019-12-05T04:57:46.099 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41', u'name': u'bengland'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1189, "groups": "sudo", "home": "/home/bengland", "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "move_home": false, "name": "bengland", "shell": "/bin/bash", "state": "present", "uid": 1189} 2019-12-05T04:57:46.242 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8', u'name': u'smanjara'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1188, "groups": "sudo", "home": "/home/smanjara", "item": {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, "move_home": false, "name": "smanjara", "shell": "/bin/bash", "state": "present", "uid": 1188} 2019-12-05T04:57:46.257 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d', u'name': u'pnawracay'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1190, "groups": "sudo", "home": "/home/pnawracay", "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "move_home": false, "name": "pnawracay", "shell": "/bin/bash", "state": "present", "uid": 1190} 2019-12-05T04:57:46.402 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8', u'name': u'alfonsomthd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1191, "groups": "sudo", "home": "/home/alfonsomthd", "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "move_home": false, "name": "alfonsomthd", "shell": "/bin/bash", "state": "present", "uid": 1191} 2019-12-05T04:57:46.407 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41', u'name': u'bengland'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1189, "groups": "sudo", "home": "/home/bengland", "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "move_home": false, "name": "bengland", "shell": "/bin/bash", "state": "present", "uid": 1189} 2019-12-05T04:57:46.537 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e', u'name': u'oliveiradan'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1192, "groups": "sudo", "home": "/home/oliveiradan", "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "move_home": false, "name": "oliveiradan", "shell": "/bin/bash", "state": "present", "uid": 1192} 2019-12-05T04:57:46.559 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d', u'name': u'pnawracay'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1190, "groups": "sudo", "home": "/home/pnawracay", "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "move_home": false, "name": "pnawracay", "shell": "/bin/bash", "state": "present", "uid": 1190} 2019-12-05T04:57:46.689 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba', u'name': u'swagner'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1193, "groups": "sudo", "home": "/home/swagner", "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "move_home": false, "name": "swagner", "shell": "/bin/bash", "state": "present", "uid": 1193} 2019-12-05T04:57:46.702 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8', u'name': u'alfonsomthd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1191, "groups": "sudo", "home": "/home/alfonsomthd", "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "move_home": false, "name": "alfonsomthd", "shell": "/bin/bash", "state": "present", "uid": 1191} 2019-12-05T04:57:46.836 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e', u'name': u'yuvalif'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1194, "groups": "sudo", "home": "/home/yuvalif", "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, "move_home": false, "name": "yuvalif", "shell": "/bin/bash", "state": "present", "uid": 1194} 2019-12-05T04:57:46.863 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e', u'name': u'oliveiradan'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1192, "groups": "sudo", "home": "/home/oliveiradan", "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "move_home": false, "name": "oliveiradan", "shell": "/bin/bash", "state": "present", "uid": 1192} 2019-12-05T04:57:46.992 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852', u'name': u'kkeithle'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1195, "groups": "sudo", "home": "/home/kkeithle", "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "move_home": false, "name": "kkeithle", "shell": "/bin/bash", "state": "present", "uid": 1195} 2019-12-05T04:57:47.014 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba', u'name': u'swagner'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1193, "groups": "sudo", "home": "/home/swagner", "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "move_home": false, "name": "swagner", "shell": "/bin/bash", "state": "present", "uid": 1193} 2019-12-05T04:57:47.149 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130', u'name': u'emmericp'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1196, "groups": "sudo", "home": "/home/emmericp", "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "move_home": false, "name": "emmericp", "shell": "/bin/bash", "state": "present", "uid": 1196} 2019-12-05T04:57:47.162 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e', u'name': u'yuvalif'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1194, "groups": "sudo", "home": "/home/yuvalif", "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, "move_home": false, "name": "yuvalif", "shell": "/bin/bash", "state": "present", "uid": 1194} 2019-12-05T04:57:47.299 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0', u'name': u'mchangir'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1197, "groups": "sudo", "home": "/home/mchangir", "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "move_home": false, "name": "mchangir", "shell": "/bin/bash", "state": "present", "uid": 1197} 2019-12-05T04:57:47.368 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852', u'name': u'kkeithle'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1195, "groups": "sudo", "home": "/home/kkeithle", "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "move_home": false, "name": "kkeithle", "shell": "/bin/bash", "state": "present", "uid": 1195} 2019-12-05T04:57:47.480 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286', u'name': u'sidharthanup'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1198, "groups": "sudo", "home": "/home/sidharthanup", "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "move_home": false, "name": "sidharthanup", "shell": "/bin/bash", "state": "present", "uid": 1198} 2019-12-05T04:57:47.556 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130', u'name': u'emmericp'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1196, "groups": "sudo", "home": "/home/emmericp", "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "move_home": false, "name": "emmericp", "shell": "/bin/bash", "state": "present", "uid": 1196} 2019-12-05T04:57:47.653 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31', u'name': u'varsha'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1199, "groups": "sudo", "home": "/home/varsha", "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "move_home": false, "name": "varsha", "shell": "/bin/bash", "state": "present", "uid": 1199} 2019-12-05T04:57:47.703 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0', u'name': u'mchangir'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1197, "groups": "sudo", "home": "/home/mchangir", "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "move_home": false, "name": "mchangir", "shell": "/bin/bash", "state": "present", "uid": 1197} 2019-12-05T04:57:47.809 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n', u'name': u'sjust'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1200, "groups": "sudo", "home": "/home/sjust", "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "move_home": false, "name": "sjust", "shell": "/bin/bash", "state": "present", "uid": 1200} 2019-12-05T04:57:47.857 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286', u'name': u'sidharthanup'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1198, "groups": "sudo", "home": "/home/sidharthanup", "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "move_home": false, "name": "sidharthanup", "shell": "/bin/bash", "state": "present", "uid": 1198} 2019-12-05T04:57:47.967 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'deepika@thinkpad HXRUSerxILmV0ejAD/Zt1g 29b44e741567e7a2e7b664d6533247d79851fc302051c9ba34103c63a2fcfacc', u'name': u'ideepika'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1201, "groups": "sudo", "home": "/home/ideepika", "item": {"name": "ideepika", "ovpn": "deepika@thinkpad HXRUSerxILmV0ejAD/Zt1g 29b44e741567e7a2e7b664d6533247d79851fc302051c9ba34103c63a2fcfacc"}, "move_home": false, "name": "ideepika", "shell": "/bin/bash", "state": "present", "uid": 1201} 2019-12-05T04:57:48.014 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31', u'name': u'varsha'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1199, "groups": "sudo", "home": "/home/varsha", "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "move_home": false, "name": "varsha", "shell": "/bin/bash", "state": "present", "uid": 1199} 2019-12-05T04:57:48.114 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46', u'name': u'gsalomon'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1202, "groups": "sudo", "home": "/home/gsalomon", "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "move_home": false, "name": "gsalomon", "shell": "/bin/bash", "state": "present", "uid": 1202} 2019-12-05T04:57:48.180 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n', u'name': u'sjust'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1200, "groups": "sudo", "home": "/home/sjust", "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "move_home": false, "name": "sjust", "shell": "/bin/bash", "state": "present", "uid": 1200} 2019-12-05T04:57:48.248 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de', u'name': u'soumyakoduri'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1203, "groups": "sudo", "home": "/home/soumyakoduri", "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "move_home": false, "name": "soumyakoduri", "shell": "/bin/bash", "state": "present", "uid": 1203} 2019-12-05T04:57:48.302 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'deepika@thinkpad HXRUSerxILmV0ejAD/Zt1g 29b44e741567e7a2e7b664d6533247d79851fc302051c9ba34103c63a2fcfacc', u'name': u'ideepika'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1201, "groups": "sudo", "home": "/home/ideepika", "item": {"name": "ideepika", "ovpn": "deepika@thinkpad HXRUSerxILmV0ejAD/Zt1g 29b44e741567e7a2e7b664d6533247d79851fc302051c9ba34103c63a2fcfacc"}, "move_home": false, "name": "ideepika", "shell": "/bin/bash", "state": "present", "uid": 1201} 2019-12-05T04:57:48.392 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a', u'name': u'kyr'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1204, "groups": "sudo", "home": "/home/kyr", "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "move_home": false, "name": "kyr", "shell": "/bin/bash", "state": "present", "uid": 1204} 2019-12-05T04:57:48.453 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46', u'name': u'gsalomon'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1202, "groups": "sudo", "home": "/home/gsalomon", "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "move_home": false, "name": "gsalomon", "shell": "/bin/bash", "state": "present", "uid": 1202} 2019-12-05T04:57:48.543 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849', u'name': u'sseshasa'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1205, "groups": "sudo", "home": "/home/sseshasa", "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "move_home": false, "name": "sseshasa", "shell": "/bin/bash", "state": "present", "uid": 1205} 2019-12-05T04:57:48.589 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de', u'name': u'soumyakoduri'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1203, "groups": "sudo", "home": "/home/soumyakoduri", "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "move_home": false, "name": "soumyakoduri", "shell": "/bin/bash", "state": "present", "uid": 1203} 2019-12-05T04:57:48.676 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189', u'name': u'rfriedma'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1206, "groups": "sudo", "home": "/home/rfriedma", "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "move_home": false, "name": "rfriedma", "shell": "/bin/bash", "state": "present", "uid": 1206} 2019-12-05T04:57:48.731 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a', u'name': u'kyr'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1204, "groups": "sudo", "home": "/home/kyr", "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "move_home": false, "name": "kyr", "shell": "/bin/bash", "state": "present", "uid": 1204} 2019-12-05T04:57:48.817 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b', u'name': u'lmb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1207, "groups": "sudo", "home": "/home/lmb", "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "move_home": false, "name": "lmb", "shell": "/bin/bash", "state": "present", "uid": 1207} 2019-12-05T04:57:48.883 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849', u'name': u'sseshasa'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1205, "groups": "sudo", "home": "/home/sseshasa", "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "move_home": false, "name": "sseshasa", "shell": "/bin/bash", "state": "present", "uid": 1205} 2019-12-05T04:57:48.951 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc', u'name': u'ksirivad'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1208, "groups": "sudo", "home": "/home/ksirivad", "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, "move_home": false, "name": "ksirivad", "shell": "/bin/bash", "state": "present", "uid": 1208} 2019-12-05T04:57:49.040 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189', u'name': u'rfriedma'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1206, "groups": "sudo", "home": "/home/rfriedma", "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "move_home": false, "name": "rfriedma", "shell": "/bin/bash", "state": "present", "uid": 1206} 2019-12-05T04:57:49.099 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4', u'name': u'ofriedma'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1209, "groups": "sudo", "home": "/home/ofriedma", "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "move_home": false, "name": "ofriedma", "shell": "/bin/bash", "state": "present", "uid": 1209} 2019-12-05T04:57:49.165 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b', u'name': u'lmb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1207, "groups": "sudo", "home": "/home/lmb", "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "move_home": false, "name": "lmb", "shell": "/bin/bash", "state": "present", "uid": 1207} 2019-12-05T04:57:49.241 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52', u'name': u'h4iresis'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1210, "groups": "sudo", "home": "/home/h4iresis", "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "move_home": false, "name": "h4iresis", "shell": "/bin/bash", "state": "present", "uid": 1210} 2019-12-05T04:57:49.309 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc', u'name': u'ksirivad'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1208, "groups": "sudo", "home": "/home/ksirivad", "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, "move_home": false, "name": "ksirivad", "shell": "/bin/bash", "state": "present", "uid": 1208} 2019-12-05T04:57:49.382 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5', u'name': u'shyukri'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1211, "groups": "sudo", "home": "/home/shyukri", "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "move_home": false, "name": "shyukri", "shell": "/bin/bash", "state": "present", "uid": 1211} 2019-12-05T04:57:49.476 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4', u'name': u'ofriedma'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1209, "groups": "sudo", "home": "/home/ofriedma", "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "move_home": false, "name": "ofriedma", "shell": "/bin/bash", "state": "present", "uid": 1209} 2019-12-05T04:57:49.532 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e', u'name': u'scarvalhojr'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1212, "groups": "sudo", "home": "/home/scarvalhojr", "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "move_home": false, "name": "scarvalhojr", "shell": "/bin/bash", "state": "present", "uid": 1212} 2019-12-05T04:57:49.612 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52', u'name': u'h4iresis'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1210, "groups": "sudo", "home": "/home/h4iresis", "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "move_home": false, "name": "h4iresis", "shell": "/bin/bash", "state": "present", "uid": 1210} 2019-12-05T04:57:49.668 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b', u'name': u'rakesh'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1213, "groups": "sudo", "home": "/home/rakesh", "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "move_home": false, "name": "rakesh", "shell": "/bin/bash", "state": "present", "uid": 1213} 2019-12-05T04:57:49.763 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5', u'name': u'shyukri'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1211, "groups": "sudo", "home": "/home/shyukri", "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "move_home": false, "name": "shyukri", "shell": "/bin/bash", "state": "present", "uid": 1211} 2019-12-05T04:57:49.832 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'ryg@wombat 7WbRbRXPcrQ2++Bww59JKg 1186db58c70662d55cae6d75e54022d1392708d3cf742b10ce5ff9d67dd8486', u'name': u'ryg-'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1214, "groups": "sudo", "home": "/home/ryg-", "item": {"name": "ryg-", "ovpn": "ryg@wombat 7WbRbRXPcrQ2++Bww59JKg 1186db58c70662d55cae6d75e54022d1392708d3cf742b10ce5ff9d67dd8486"}, "move_home": false, "name": "ryg-", "shell": "/bin/bash", "state": "present", "uid": 1214} 2019-12-05T04:57:49.923 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e', u'name': u'scarvalhojr'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1212, "groups": "sudo", "home": "/home/scarvalhojr", "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "move_home": false, "name": "scarvalhojr", "shell": "/bin/bash", "state": "present", "uid": 1212} 2019-12-05T04:57:50.000 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1155, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1155} 2019-12-05T04:57:50.076 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b', u'name': u'rakesh'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1213, "groups": "sudo", "home": "/home/rakesh", "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "move_home": false, "name": "rakesh", "shell": "/bin/bash", "state": "present", "uid": 1213} 2019-12-05T04:57:50.130 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42', u'name': u'zdover'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1215, "groups": "sudo", "home": "/home/zdover", "item": {"name": "zdover", "ovpn": "zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42"}, "move_home": false, "name": "zdover", "shell": "/bin/bash", "state": "present", "uid": 1215} 2019-12-05T04:57:50.233 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'ryg@wombat 7WbRbRXPcrQ2++Bww59JKg 1186db58c70662d55cae6d75e54022d1392708d3cf742b10ce5ff9d67dd8486', u'name': u'ryg-'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1214, "groups": "sudo", "home": "/home/ryg-", "item": {"name": "ryg-", "ovpn": "ryg@wombat 7WbRbRXPcrQ2++Bww59JKg 1186db58c70662d55cae6d75e54022d1392708d3cf742b10ce5ff9d67dd8486"}, "move_home": false, "name": "ryg-", "shell": "/bin/bash", "state": "present", "uid": 1214} 2019-12-05T04:57:50.392 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1155, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1155} 2019-12-05T04:57:50.548 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42', u'name': u'zdover'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1215, "groups": "sudo", "home": "/home/zdover", "item": {"name": "zdover", "ovpn": "zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42"}, "move_home": false, "name": "zdover", "shell": "/bin/bash", "state": "present", "uid": 1215} 2019-12-05T04:57:50.763 INFO:teuthology.task.ansible.out:changed: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b', u'name': u'rraja'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10103, "groups": "sudo", "home": "/home/rraja", "item": {"name": "rraja", "ovpn": "rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b"}, "name": "rraja", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10102} 2019-12-05T04:57:50.900 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:57:50 +0000 (0:00:16.664) 0:00:26.403 ***** 2019-12-05T04:57:50.928 INFO:teuthology.task.ansible.out: TASK [users : Create all users without sudo access.] *************************** 2019-12-05T04:57:50.970 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:57:50 +0000 (0:00:00.069) 0:00:26.473 ***** 2019-12-05T04:57:51.029 INFO:teuthology.task.ansible.out: TASK [users : Merge managed_users and managed_admin users] ********************* 2019-12-05T04:57:51.034 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => {"ansible_facts": {"pubkey_users": [{"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, {"name": "adeza", "ovpn": "alfredo@papaya.local V5VkRT2Du88WwQrYeK196w 4dff4d9bdc4d9ab60532814c338abe4534cb45a9ca729bdbc2411ad513516b47"}, {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n"}, {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, {"name": "tr 2019-12-05T04:57:51.036 INFO:teuthology.task.ansible.out:ociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7 2019-12-05T04:57:51.037 INFO:teuthology.task.ansible.out:L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32 2019-12-05T04:57:51.039 INFO:teuthology.task.ansible.out:f606a33fb488ff621e11aa305ac979501030202658\n"}, {"name": "gmeno", "ovpn": "gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc"}, {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n"}, {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b 2019-12-05T04:57:51.041 INFO:teuthology.task.ansible.out:09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e 2019-12-05T04:57:51.042 INFO:teuthology.task.ansible.out:66b842a921169f20a00481a4cd3d9cb"}, {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa 2019-12-05T04:57:51.044 INFO:teuthology.task.ansible.out:8ea"}, {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e"}, {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 2019-12-05T04:57:51.045 INFO:teuthology.task.ansible.out:6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f8 2019-12-05T04:57:51.046 INFO:teuthology.task.ansible.out:48846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8a 2019-12-05T04:57:51.048 INFO:teuthology.task.ansible.out:a01123ab69cc6f3f11273528ce33e5a31"}, {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, {"name": "ideepika", "ovpn": "deepika@thinkpad HXRUSerxILmV0ejAD/Zt1g 29b44e741567e7a2e7b664d6533247d79851fc302051c9ba34103c63a2fcfacc"}, {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, {"name": "scarvalho 2019-12-05T04:57:51.049 INFO:teuthology.task.ansible.out:jr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, {"name": "ryg-", "ovpn": "ryg@wombat 7WbRbRXPcrQ2++Bww59JKg 1186db58c70662d55cae6d75e54022d1392708d3cf742b10ce5ff9d67dd8486"}, {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, {"name": "zdover", "ovpn": "zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42"}, {"name": "rraja", "ovpn": "rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b"}]}, "changed": false} 2019-12-05T04:57:51.109 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:57:51 +0000 (0:00:00.134) 0:00:26.607 ***** 2019-12-05T04:57:51.121 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ 2019-12-05T04:57:51.122 INFO:teuthology.task.ansible.out:changed: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b', u'name': u'rraja'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10103, "groups": "sudo", "home": "/home/rraja", "item": {"name": "rraja", "ovpn": "rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b"}, "name": "rraja", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10102} 2019-12-05T04:57:51.281 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:57:51 +0000 (0:00:00.176) 0:00:26.783 ***** 2019-12-05T04:57:51.320 INFO:teuthology.task.ansible.out: TASK [users : Create all users without sudo access.] *************************** 2019-12-05T04:57:51.377 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:57:51 +0000 (0:00:00.090) 0:00:26.873 ***** 2019-12-05T04:57:51.428 INFO:teuthology.task.ansible.out: TASK [users : Merge managed_users and managed_admin users] ********************* 2019-12-05T04:57:51.430 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => {"ansible_facts": {"pubkey_users": [{"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, {"name": "adeza", "ovpn": "alfredo@papaya.local V5VkRT2Du88WwQrYeK196w 4dff4d9bdc4d9ab60532814c338abe4534cb45a9ca729bdbc2411ad513516b47"}, {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n"}, {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, {"name": "tr 2019-12-05T04:57:51.431 INFO:teuthology.task.ansible.out:ociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7 2019-12-05T04:57:51.433 INFO:teuthology.task.ansible.out:L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32 2019-12-05T04:57:51.434 INFO:teuthology.task.ansible.out:f606a33fb488ff621e11aa305ac979501030202658\n"}, {"name": "gmeno", "ovpn": "gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc"}, {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n"}, {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b 2019-12-05T04:57:51.436 INFO:teuthology.task.ansible.out:09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e 2019-12-05T04:57:51.437 INFO:teuthology.task.ansible.out:66b842a921169f20a00481a4cd3d9cb"}, {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa 2019-12-05T04:57:51.439 INFO:teuthology.task.ansible.out:8ea"}, {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e"}, {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f 2019-12-05T04:57:51.440 INFO:teuthology.task.ansible.out:826aae1caa53"}, {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, {"name": 2019-12-05T04:57:51.442 INFO:teuthology.task.ansible.out: "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, {"name": "sidharthanup", "ovpn": "sidharth 2019-12-05T04:57:51.443 INFO:teuthology.task.ansible.out:anup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, {"name": "ideepika", "ovpn": "deepika@thinkpad HXRUSerxILmV0ejAD/Zt1g 29b44e741567e7a2e7b664d6533247d79851fc302051c9ba34103c63a2fcfacc"}, {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c3659458 2019-12-05T04:57:51.444 INFO:teuthology.task.ansible.out:5e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, {"name": "ryg-", "ovpn": "ryg@wombat 7WbRbRXPcrQ2++Bww59JKg 1186db58c70662d55cae6d75e54022d1392708d3cf742b10ce5ff9d67dd8486"}, {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, {"name": "zdover", "ovpn": "zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42"}, {"name": "rraja", "ovpn": "rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b"}]}, "changed": false} 2019-12-05T04:57:51.498 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:57:51 +0000 (0:00:00.127) 0:00:27.001 ***** 2019-12-05T04:57:52.094 INFO:teuthology.task.ansible.out: TASK [users : Clone the keys repo] ********************************************* 2019-12-05T04:57:52.095 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com -> localhost] => {"after": "7b315d00bbff69e651613e1408370c0721c0c099", "attempts": 1, "before": "7b315d00bbff69e651613e1408370c0721c0c099", "changed": false, "remote_url_changed": false} 2019-12-05T04:57:52.147 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:57:52 +0000 (0:00:00.645) 0:00:27.647 ***** 2019-12-05T04:57:52.225 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com -> localhost] => {"after": "7b315d00bbff69e651613e1408370c0721c0c099", "attempts": 1, "before": "7b315d00bbff69e651613e1408370c0721c0c099", "changed": false, "remote_url_changed": false} 2019-12-05T04:57:52.271 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:57:52 +0000 (0:00:00.126) 0:00:27.774 ***** 2019-12-05T04:57:52.366 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9', u'name': u'andrewschoen'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com", "key_options": null, "keyfile": "/home/andrewschoen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "andrewschoen", "validate_certs": true} 2019-12-05T04:57:52.461 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9', u'name': u'andrewschoen'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com", "key_options": null, "keyfile": "/home/andrewschoen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "andrewschoen", "validate_certs": true} 2019-12-05T04:57:52.506 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n', u'name': u'zack'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/zack/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zack", "validate_certs": true} 2019-12-05T04:57:52.605 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n', u'name': u'zack'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/zack/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zack", "validate_certs": true} 2019-12-05T04:57:52.627 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd', u'name': u'kdreyer'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4eHmz10szeHNS3dNejKokW85ksB+iR4HGOFsmQM11Ni68Nm5aqEKvkOZU8TpY92vpCQL0A68GlrXB845cACdyk6HUJYyNNNMC43l1FYWOwjMqQBSdj8W3VQDTA6eiG60mt5fgI8fyR38rKzIA1MnTBkSSjuh5kQVJ9bdEp3GuY5oc8vxDNBlGJ6LYnyEWt/pqL2J+mpjqnOjsC+EbE2exhP9O+mvzpQiyo/+dEN1COwX3//pNRXGfOSeOczHNsJE8Eu+j/n/BlW57++sJyFMkzS7bUxMSGM6quvjQZ7RT1c5JM6vLEiQyzQxoRgzY93h1yKlOstBi0NamtpqHQZGP kdreyer@mbp.ktdreyer.com", "key_options": null, "keyfile": "/home/kdreyer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kdreyer", "validate_certs": true} 2019-12-05T04:57:52.756 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd', u'name': u'kdreyer'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4eHmz10szeHNS3dNejKokW85ksB+iR4HGOFsmQM11Ni68Nm5aqEKvkOZU8TpY92vpCQL0A68GlrXB845cACdyk6HUJYyNNNMC43l1FYWOwjMqQBSdj8W3VQDTA6eiG60mt5fgI8fyR38rKzIA1MnTBkSSjuh5kQVJ9bdEp3GuY5oc8vxDNBlGJ6LYnyEWt/pqL2J+mpjqnOjsC+EbE2exhP9O+mvzpQiyo/+dEN1COwX3//pNRXGfOSeOczHNsJE8Eu+j/n/BlW57++sJyFMkzS7bUxMSGM6quvjQZ7RT1c5JM6vLEiQyzQxoRgzY93h1yKlOstBi0NamtpqHQZGP kdreyer@mbp.ktdreyer.com", "key_options": null, "keyfile": "/home/kdreyer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kdreyer", "validate_certs": true} 2019-12-05T04:57:52.767 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'alfredo@papaya.local V5VkRT2Du88WwQrYeK196w 4dff4d9bdc4d9ab60532814c338abe4534cb45a9ca729bdbc2411ad513516b47', u'name': u'adeza'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "adeza", "ovpn": "alfredo@papaya.local V5VkRT2Du88WwQrYeK196w 4dff4d9bdc4d9ab60532814c338abe4534cb45a9ca729bdbc2411ad513516b47"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCWy2i6Iv6IWLh53Z44O4fQ0ctxR8odCAWIZAaSQXAxxC++RS+95Zr+BZ5wJwsBOyP1G4AAhl+9uwnn3NmDDG7fKGexPoQrTZZ27KdJ3bHpZRq5LJf9byWFwz4rDsqIqkGjI9sUkfa6nV7ci2BdEkIgLupTT2uXUzTb/njOjI7Fc+cQ7ac4QfJ+yeIcqxeje/YYDF3068V4z1yynk9q2KRXYdERKQghaWDNJlbdBhn/fmsam6okOtUnsXjE88qGPd+LVHWfK0EfYvYXiAoOaJDK8d4EXnJ4RKJceQPBk8ravuQtKVwnRIr8WtmILHuoztthfwqb2WvZ5iMDt6v7oQZ alfredo@papaya.local", "key_options": null, "keyfile": "/home/adeza/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "adeza", "validate_certs": true} 2019-12-05T04:57:52.901 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'alfredo@papaya.local V5VkRT2Du88WwQrYeK196w 4dff4d9bdc4d9ab60532814c338abe4534cb45a9ca729bdbc2411ad513516b47', u'name': u'adeza'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "adeza", "ovpn": "alfredo@papaya.local V5VkRT2Du88WwQrYeK196w 4dff4d9bdc4d9ab60532814c338abe4534cb45a9ca729bdbc2411ad513516b47"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCWy2i6Iv6IWLh53Z44O4fQ0ctxR8odCAWIZAaSQXAxxC++RS+95Zr+BZ5wJwsBOyP1G4AAhl+9uwnn3NmDDG7fKGexPoQrTZZ27KdJ3bHpZRq5LJf9byWFwz4rDsqIqkGjI9sUkfa6nV7ci2BdEkIgLupTT2uXUzTb/njOjI7Fc+cQ7ac4QfJ+yeIcqxeje/YYDF3068V4z1yynk9q2KRXYdERKQghaWDNJlbdBhn/fmsam6okOtUnsXjE88qGPd+LVHWfK0EfYvYXiAoOaJDK8d4EXnJ4RKJceQPBk8ravuQtKVwnRIr8WtmILHuoztthfwqb2WvZ5iMDt6v7oQZ alfredo@papaya.local", "key_options": null, "keyfile": "/home/adeza/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "adeza", "validate_certs": true} 2019-12-05T04:57:52.913 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n', u'name': u'dmick'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/onMoOCUhkp+UiJAq+ye4P70sfeV+uMhgWKLfLBh/1SYq1xxb9bSmjBnETO/9KvukRhzlReBQ7y+BKRhm037P7LuTUY4r0Y8JB6BXkjjCY1yXZJC6kvB7xFsvyiDum5rrdeWMQ/Rhh7YJpze5KY1nHLkyU9X7Ewak0kLEyduJEKekbsOom6l0wZ2ZYrBEKFWJXUDk5icUuWWg3mhKaKnLNWVg1lUw8jzYTJYfjR9GUgzZWXtzkAaI3ugpLOlQ+KEls03/WAFcWA9F9ExqzIsJpvEks27LeFBGVE3PjVNKMqqBZQlZtIEz1iguWJaWMi5lDIfXkvVOFabtIpX/BZRf dmick@dmick-MacBookAir", "key_options": null, "keyfile": "/home/dmick/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dmick", "va 2019-12-05T04:57:52.915 INFO:teuthology.task.ansible.out:lidate_certs": true} 2019-12-05T04:57:53.042 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n', u'name': u'dmick'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/onMoOCUhkp+UiJAq+ye4P70sfeV+uMhgWKLfLBh/1SYq1xxb9bSmjBnETO/9KvukRhzlReBQ7y+BKRhm037P7LuTUY4r0Y8JB6BXkjjCY1yXZJC6kvB7xFsvyiDum5rrdeWMQ/Rhh7YJpze5KY1nHLkyU9X7Ewak0kLEyduJEKekbsOom6l0wZ2ZYrBEKFWJXUDk5icUuWWg3mhKaKnLNWVg1lUw8jzYTJYfjR9GUgzZWXtzkAaI3ugpLOlQ+KEls03/WAFcWA9F9ExqzIsJpvEks27LeFBGVE3PjVNKMqqBZQlZtIEz1iguWJaWMi5lDIfXkvVOFabtIpX/BZRf dmick@dmick-MacBookAir", "key_options": null, "keyfile": "/home/dmick/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dmick", "va 2019-12-05T04:57:53.044 INFO:teuthology.task.ansible.out:lidate_certs": true} 2019-12-05T04:57:53.074 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n', u'name': u'sage'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJ4z0gLwjexgyixdTuiwaxpPnq7w4FOKxHsQKKAHTlKRu11ynB+9APbtqaAYss8yDqHaHyewPdE+9JMYhZX8rUgcKv2H7jBCQ/Enfl0UyY643VQnuG8lr2Ipk6ELuL5NuuTb7Npo4zKyq/0649kPD46k/X/qLuKzhN49k50S3XJq5WwHi6CJvR8DzPOkjied0o1BHpfcCCIMorUcyfVxqmeu2m+uSE15ncQsd8BIvH4/YNoncUGYxqVhGB+FNUlWdhV0SPRW7V/gmyvFW5qdYX2QhNZAQHChdEH1E9AI7WiCFkyLRx39qYgXgqj8vOOnlwHgy/oJnpnu/Mlvj1uETX sage@montreal\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCVPfoCpv2/9uWCwEo4clWt3i1O8ZPpB2FcQbBLxY3iLQWZJDe74rNWPBj0cJVCytTHxYV0e5HxYyChOG60ca5odHlPNB+/Z4WLU9LVuHiCYsElivJlwvD+l9kzI+UMGJb3HCHqx2yIRGXpEkSMFR1+w5Oz9sInPD7JJmO8Hb9KkoeqZtkJL0aic/M5bDWBut6SSfYlmh8qnQYbkCDiXg/KtOcy1ZJXQ9ZLjsCbpSzeBNbdUOxEQ1+5vUe00vHpl0vOKtRrzgTne2ct1IxbSjdL16ZgHi5BQAREXc9UUG3z7NPOmnwkuKkkpyLmwcOusot5tYHrxks+5i6EoGEJrvJ3 sage@maetl", "key_options": null, "keyfile": "/home/sage/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sage", "validate_certs": true} 2019-12-05T04:57:53.214 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n', u'name': u'sage'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJ4z0gLwjexgyixdTuiwaxpPnq7w4FOKxHsQKKAHTlKRu11ynB+9APbtqaAYss8yDqHaHyewPdE+9JMYhZX8rUgcKv2H7jBCQ/Enfl0UyY643VQnuG8lr2Ipk6ELuL5NuuTb7Npo4zKyq/0649kPD46k/X/qLuKzhN49k50S3XJq5WwHi6CJvR8DzPOkjied0o1BHpfcCCIMorUcyfVxqmeu2m+uSE15ncQsd8BIvH4/YNoncUGYxqVhGB+FNUlWdhV0SPRW7V/gmyvFW5qdYX2QhNZAQHChdEH1E9AI7WiCFkyLRx39qYgXgqj8vOOnlwHgy/oJnpnu/Mlvj1uETX sage@montreal\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCVPfoCpv2/9uWCwEo4clWt3i1O8ZPpB2FcQbBLxY3iLQWZJDe74rNWPBj0cJVCytTHxYV0e5HxYyChOG60ca5odHlPNB+/Z4WLU9LVuHiCYsElivJlwvD+l9kzI+UMGJb3HCHqx2yIRGXpEkSMFR1+w5Oz9sInPD7JJmO8Hb9KkoeqZtkJL0aic/M5bDWBut6SSfYlmh8qnQYbkCDiXg/KtOcy1ZJXQ9ZLjsCbpSzeBNbdUOxEQ1+5vUe00vHpl0vOKtRrzgTne2ct1IxbSjdL16ZgHi5BQAREXc9UUG3z7NPOmnwkuKkkpyLmwcOusot5tYHrxks+5i6EoGEJrvJ3 sage@maetl", "key_options": null, "keyfile": "/home/sage/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sage", "validate_certs": true} 2019-12-05T04:57:53.240 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n', u'name': u'dgalloway'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBAN3MlkYQNsm1uB6uJQ7hWJ1Ro9Gusrhi/b/KshPMFh51VWBQn71LdCCjdrsE6FWs5FYE/hr7MITJe+si7aZCmuf8U9tc+oiYh3spXtRhJVIM80FOVRXUmt2XR8I+oNbpcFDayYu956Volafksk3oYjsO356swj7+Q+AUQVF4bDJRAAAAFQCkB2YZKItSDL7XfAl6ZmJ21vqUMwAAAIEAsZspNIUwssVJGO4r3M3BECWklWLEFviIjGVKdPWulN10eRCczZrH7vMOs8coHKDQqYdS2RejlojT03FjFAqeBiTeAlcmgHoev+vqTg2Yhqx3gROzNc6NmG887owCOU4j7LJQP6SX8I8EpSXCMNOZliv2ZokjX2DuRsuIDxQdc84AAACBAMAp++fcqmGS7x7ic2x4rhyiYmdqlMIl8lrM3GkVR6qGuIVBvOGl38ieJKcTBLeRgyC04uIMfk8P9lLvZw3UF97qH3V0imKfq8aYnBGxCv+IjtdOrR5k4hrUW+P0qipvU+4dgHlpqwTKB4MtD9uDxvhENWvqcCnOS5rI6dddaZT6 dgallowa@dgallowa.csb\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZtt 2019-12-05T04:57:53.242 INFO:teuthology.task.ansible.out:Edz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/dgalloway/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dgalloway", "validate_certs": true} 2019-12-05T04:57:53.376 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045', u'name': u'akraitma'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/akraitma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akraitma", "validate_certs": true} 2019-12-05T04:57:53.383 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n', u'name': u'dgalloway'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBAN3MlkYQNsm1uB6uJQ7hWJ1Ro9Gusrhi/b/KshPMFh51VWBQn71LdCCjdrsE6FWs5FYE/hr7MITJe+si7aZCmuf8U9tc+oiYh3spXtRhJVIM80FOVRXUmt2XR8I+oNbpcFDayYu956Volafksk3oYjsO356swj7+Q+AUQVF4bDJRAAAAFQCkB2YZKItSDL7XfAl6ZmJ21vqUMwAAAIEAsZspNIUwssVJGO4r3M3BECWklWLEFviIjGVKdPWulN10eRCczZrH7vMOs8coHKDQqYdS2RejlojT03FjFAqeBiTeAlcmgHoev+vqTg2Yhqx3gROzNc6NmG887owCOU4j7LJQP6SX8I8EpSXCMNOZliv2ZokjX2DuRsuIDxQdc84AAACBAMAp++fcqmGS7x7ic2x4rhyiYmdqlMIl8lrM3GkVR6qGuIVBvOGl38ieJKcTBLeRgyC04uIMfk8P9lLvZw3UF97qH3V0imKfq8aYnBGxCv+IjtdOrR5k4hrUW+P0qipvU+4dgHlpqwTKB4MtD9uDxvhENWvqcCnOS5rI6dddaZT6 dgallowa@dgallowa.csb\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZtt 2019-12-05T04:57:53.385 INFO:teuthology.task.ansible.out:Edz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/dgalloway/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dgalloway", "validate_certs": true} 2019-12-05T04:57:53.513 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e', u'name': u'trociny'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDSc2rWtLtZM0CBWwaxhSfd0T++ZTX9c8kQrYT0rsQlJ6Hxj4c3+ypwh4iivdtFpyz4clxWkb/wimi48PvjNyZdudB9BeQUBeFfLiro3LnFEbKhM1RwRI89s34mRwmC0LLUZwcdX80zc6/fKuPzvBzR1PiY9tWRyIpYErQAHa2jEOE2MpFDHnlbXdfXFh5Tw3OmC1qBlQvgcrivA9lVaVlf+hfwmOQwXFIG3PCJLNlXDshCSUGAfYKBRvQaKS8sOf9rQaflPrwqJ8hDgVVQkpp69yVsGtahbBmjs6QS6C8EkGUFQmvrIcrQ8+5mRlRAmK2DMTP3cAcsYcMOJ+9KwP+H mgolub@zhuzha", "key_options": null, "keyfile": "/home/trociny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "trociny", "validate_certs": true} 2019-12-05T04:57:53.524 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045', u'name': u'akraitma'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/akraitma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akraitma", "validate_certs": true} 2019-12-05T04:57:53.526 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007', u'name': u'smithfarm', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:57:53.675 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n', u'name': u'mbenjamin'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANarZ+C62DPffRIhaCuCtXLChrfqwXgWqGHOLLkmPTtZ3/EE8ZiN+LY1ZcmP939AxXi8Oe+9LepwAdsoQKZctlC7XToyhjIk+rVSPyn3zl3BTC2AxfCKq2vqj65a1sc35Whrw7OaUa2fr5iHSazP6HuG01DhJ9l8HJsLKmiwWChZAAAAFQCNmunc7BNLWrqyrYwINTfscowkbQAAAIEAyYERcunZYJAZ9Eno18kM+ylq0jcQmnPqyIvssWXTttUZnaGQvbVxABeyr6ZKuUyZQfyetofquOGBKPBuzV/7posl5ShDuAcxbS1zhrmk9glTA1yFLvznDdWdLOWG06tjOc78G2AmldKHCdPsmgh2D7WyRr4QY4rcCvwKfZ6+eEYAAACBAJZrOlOlygfPZ4ZB0+EbdYACz1+L/iYNNAOBp51TRutNceLVrNJNOVdmCMLuF0A1go4kPSD/05aJOkOyW86gYLA5DXaW3961+kPInOFAe8OTyQYE7221ThYHRnCfVFjdknuCDomj69wk4SsR+ZPXvSXs1jSJWKhLfUSe9n4bmw7w anet2@secant.private.linuxbox.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5CMht/crEocrHqGjf0sxE9AhNcW3XBxwlniJwaBJLhekLQEQJoxAy2xUUmPYb2/e7rdmTBxzQipUaAGWVk1FHANac4cbgXzNT+9cnF7EIRkPa/x9jaKNiU0ozMhu/GOoeDXMrHIDgVGWPF3Ag/r17ORpYnuFTjsqoCWyh+mqskV+lmAp04rv2tvqouHtEs6gLebKNtB8c6twhrBKF62Ie+rSr3r7hQaqiZitHbg/isbU5RMBl6H48wHAVSK2/uT 2019-12-05T04:57:53.677 INFO:teuthology.task.ansible.out:fMorbnuEYRG8HvlnPRvQHscJYKiZSt/G7XqZ3x/7c4czEF+OpdQbAgg46vh4fNVMxZk9KHwEkOE06ROuwgAejJ mbenjamin@duel", "key_options": null, "keyfile": "/home/mbenjamin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mbenjamin", "validate_certs": true} 2019-12-05T04:57:53.688 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e', u'name': u'trociny'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDSc2rWtLtZM0CBWwaxhSfd0T++ZTX9c8kQrYT0rsQlJ6Hxj4c3+ypwh4iivdtFpyz4clxWkb/wimi48PvjNyZdudB9BeQUBeFfLiro3LnFEbKhM1RwRI89s34mRwmC0LLUZwcdX80zc6/fKuPzvBzR1PiY9tWRyIpYErQAHa2jEOE2MpFDHnlbXdfXFh5Tw3OmC1qBlQvgcrivA9lVaVlf+hfwmOQwXFIG3PCJLNlXDshCSUGAfYKBRvQaKS8sOf9rQaflPrwqJ8hDgVVQkpp69yVsGtahbBmjs6QS6C8EkGUFQmvrIcrQ8+5mRlRAmK2DMTP3cAcsYcMOJ+9KwP+H mgolub@zhuzha", "key_options": null, "keyfile": "/home/trociny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "trociny", "validate_certs": true} 2019-12-05T04:57:53.689 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01', u'name': u'aemerson', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, "skip_reason": "Conditional result was False"} skipping: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78', u'name': u'amaredia', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYu 2019-12-05T04:57:53.691 INFO:teuthology.task.ansible.out:RIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:57:53.700 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007', u'name': u'smithfarm', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:57:53.711 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5', u'name': u'tserlin', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:57:53.845 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n', u'name': u'mbenjamin'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANarZ+C62DPffRIhaCuCtXLChrfqwXgWqGHOLLkmPTtZ3/EE8ZiN+LY1ZcmP939AxXi8Oe+9LepwAdsoQKZctlC7XToyhjIk+rVSPyn3zl3BTC2AxfCKq2vqj65a1sc35Whrw7OaUa2fr5iHSazP6HuG01DhJ9l8HJsLKmiwWChZAAAAFQCNmunc7BNLWrqyrYwINTfscowkbQAAAIEAyYERcunZYJAZ9Eno18kM+ylq0jcQmnPqyIvssWXTttUZnaGQvbVxABeyr6ZKuUyZQfyetofquOGBKPBuzV/7posl5ShDuAcxbS1zhrmk9glTA1yFLvznDdWdLOWG06tjOc78G2AmldKHCdPsmgh2D7WyRr4QY4rcCvwKfZ6+eEYAAACBAJZrOlOlygfPZ4ZB0+EbdYACz1+L/iYNNAOBp51TRutNceLVrNJNOVdmCMLuF0A1go4kPSD/05aJOkOyW86gYLA5DXaW3961+kPInOFAe8OTyQYE7221ThYHRnCfVFjdknuCDomj69wk4SsR+ZPXvSXs1jSJWKhLfUSe9n4bmw7w anet2@secant.private.linuxbox.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5CMht/crEocrHqGjf0sxE9AhNcW3XBxwlniJwaBJLhekLQEQJoxAy2xUUmPYb2/e7rdmTBxzQipUaAGWVk1FHANac4cbgXzNT+9cnF7EIRkPa/x9jaKNiU0ozMhu/GOoeDXMrHIDgVGWPF3Ag/r17ORpYnuFTjsqoCWyh+mqskV+lmAp04rv2tvqouHtEs6gLebKNtB8c6twhrBKF62Ie+rSr3r7hQaqiZitHbg/isbU5RMBl6H48wHAVSK2/uT 2019-12-05T04:57:53.847 INFO:teuthology.task.ansible.out:fMorbnuEYRG8HvlnPRvQHscJYKiZSt/G7XqZ3x/7c4czEF+OpdQbAgg46vh4fNVMxZk9KHwEkOE06ROuwgAejJ mbenjamin@duel", "key_options": null, "keyfile": "/home/mbenjamin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mbenjamin", "validate_certs": true} ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28', u'name': u'dis'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHk8c8wUTTYolOOmMlniNrsjfRyF9pCCnaEQ2DXR73P0hbFMme5Md5cF3PTtPIOHSNkWk/qg7s6gZD8LGyFmloO42RBgLsW1qM8RWTDbPy5EQdrjxL9byVzBBCDoQPIS2M+0XnhYa6CBj38Sw8XPlS41CebAJEOQeyLG2I6CAI0rTij6gsqDfALbIezd/eWvmbksKYM1Lb4Gvj+k/OL/xDZzIZHF/P+mK+trY45gsUHxKZamkeHyAT4f3kiNJA+UwzTWGhafy6UE7yGqhZjt9xOZSQGuwroL7EtAzntxayPRNxQPfGDV+fYYf0QBbTtHyJeikPUuw7aR1Z3KwuSmWD dis@zambezi", "key_options": null, "keyfile": "/home/dis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dis", "validate_certs": true} skipping: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01', u'name': u'aemerson', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMg 2019-12-05T04:57:53.849 INFO:teuthology.task.ansible.out:cG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:57:53.858 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78', u'name': u'amaredia', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:57:53.885 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5', u'name': u'tserlin', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:57:53.992 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n', u'name': u'gregf'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANaS5gZUnIp5UL/PM3xOrQiNVxfona7sD1XEXGYvNUyoojPWWAvdQRwdA/Ay8JsAjKeWUt0IwfWaPfbEoXU1TKAI+yyv5WzQUZAKmBFuYM0CUQnKrk72Dp6n8P2tnYen0ect22xEiW2+Y0JZ3wfV3IB3BQPoVCC9FQ+P3LHhYii/AAAAFQCgSceiZFeNItwJ2rJeQgVVTQKljwAAAIEA0Gd2VNCbbQ48vAF+m/i/CeQHJ9Dom9IOBN3v8XnSlOhtNzY1pJ0NRzrFQURCCeVQJaywr0et5FCXIDVqTuR4qfCu5XXU1KHpn/AvHstOd93Y0XYMwSXfNszOtpH7w9SEm7uBfKnLQNF843/MHZL4ZIdfxk73YjfMjrYFcvJwvM0AAACBAI9Wi2wG9Rrdbrgk8tqbnKMxxJCYt/Vakn5mQSc/WHqMUn14+Kb7oBtfTsfPFNK952FzU09ptNQvNWTPNMniI8l8lQi2B+Z28XUlFLL76uxJXWko99xFpf/d1kFBL0M7GRGmvvrMfeTCI+WIqh2jBVqbl4o8kjNx4p0VWyoiWeFJ gregf@pudgy\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwSe7szyqMTqKK3D/JsXIa1PtPf1p8Njk0sMDSsJPA0gBGysmSgstke9fglOgeNIbNOBNDOY3vltuNCgUiQgVWOYhUaVpj+ 2019-12-05T04:57:53.996 INFO:teuthology.task.ansible.out:2vtSoAWpXMQbNaWPGKhbxvJWMCLVckLENyoUZUBKLmTvRyKDjjq8CJ1TnfA2Xfo3a796EIOtD6essU3lAVIfIUILGD+HqptiHSB0spUJUVRxxlO2xw9rLYij1eUPRHrr9xWvDamr5JtKVtCLOL3M9QlltHieKlbJSCZq/EVxW9NAGA6IiqOhw2E1k59oGZUq4Qg+YLgYkB80Ms7nb1xpjQ8txvY+73xL/8i7Nqmr7ekKJfyTpRe03nCQ== gfarnum@mozart.thompsonian.net\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzqJQfdAKaggPDgSpk8P6QjLeyQYOz183fVSKnqAa5EeaCJozS1dn8z+iphswT3YL6wqIMqVHMfkSUcEeTBwZmvydaghOMJEIQSP1PCzH0V2XprELrEcBC3QTvaElPjjEYgBVUvEw8sL/RIQ7mgt9vxD0NcCDspsz35fmU222KzFxD41PZ5XzHQ3Lumwdax7mcpqrw8Aa5/tgcEqAIu54HYRf1Qmk6+ueMAvpwkX3eiSEtkVvuV1Kxpc3/0mOfJ7cn+eyPX3qUbo4oY76hIYseoUIwEaVE/pNW+pjwDSy4gSOcuxYDEGKzCwZPZeQtRQWzRt3YNafxi5xsByobGLlZw== gfarnum@GF-Macbook.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1ik3LbyKsPiGwdh+vGrKSYLvhMg7OkqMwDNYU5LqWBSotogxyGQJUuD80ZavxmmdXC4qScddHtWOJjtyX1CTSDUe7INpSgGPGnmSExVFpcYRnmxHAB+9M/c9VorUkQ8IOKLmRO/BdYma3Rl9NzhHI73yd4zJlx1K4NuQnhnj0Z4yXO+k44RkBWDdMicBB5lBP0hETaH2Iq1u5Dai3u6N5N9bSPXv2PGKvvU395K0Wts2CxMhOBc7LAixysNzNWLrnG1ghS4jPVG2TIHOElP5lIXUv1Wlkool5rTUWgnVY/7loWkDus2JryDM7hLGFXqgT7HHcpI4gSITqRhWxzsUF gregf@fedoragreg.localdomain", "key_options": null, "keyfile": "/home/gregf/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gregf", "validate_certs": true} 2019-12-05T04:57:54.014 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28', u'name': u'dis'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHk8c8wUTTYolOOmMlniNrsjfRyF9pCCnaEQ2DXR73P0hbFMme5Md5cF3PTtPIOHSNkWk/qg7s6gZD8LGyFmloO42RBgLsW1qM8RWTDbPy5EQdrjxL9byVzBBCDoQPIS2M+0XnhYa6CBj38Sw8XPlS41CebAJEOQeyLG2I6CAI0rTij6gsqDfALbIezd/eWvmbksKYM1Lb4Gvj+k/OL/xDZzIZHF/P+mK+trY45gsUHxKZamkeHyAT4f3kiNJA+UwzTWGhafy6UE7yGqhZjt9xOZSQGuwroL7EtAzntxayPRNxQPfGDV+fYYf0QBbTtHyJeikPUuw7aR1Z3KwuSmWD dis@zambezi", "key_options": null, "keyfile": "/home/dis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dis", "validate_certs": true} 2019-12-05T04:57:54.136 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n', u'name': u'joshd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDgS5EfJyOxBkl/TtJikfb38rm6muQo7MwYufkLSBw9W2jFzt/ZMbVXyVsdZM4I6hWfZm1/q60UcER4yoA1xwYGOTmbFXrLUqqKxKTnCdyejuQ3miB7aiuisWO4vjb2+ppBwL9Mx775DPeQJehFyFK0qgh33tZO/9LWWYy7NOZdA9L2ei1PgMWa0VBeijcDR5GQSXzh0Yi+h/Ezu8jevu+NkW2IUJF6zPGjd9M9lAy7KG2dQjqqCX4XWmUS+MHJS6f9bsC5i39GWy29FvX/qFEArOMzuYRIFWE8iza95KSBxptb+Hw8PFBjQ1kTh6nC5ZV2X3ETcoj8fToTcgW3dTRF jdurgin@new-angeles", "key_options": null, "keyfile": "/home/joshd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joshd", "validate_certs": true} 2019-12-05T04:57:54.168 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n', u'name': u'gregf'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANaS5gZUnIp5UL/PM3xOrQiNVxfona7sD1XEXGYvNUyoojPWWAvdQRwdA/Ay8JsAjKeWUt0IwfWaPfbEoXU1TKAI+yyv5WzQUZAKmBFuYM0CUQnKrk72Dp6n8P2tnYen0ect22xEiW2+Y0JZ3wfV3IB3BQPoVCC9FQ+P3LHhYii/AAAAFQCgSceiZFeNItwJ2rJeQgVVTQKljwAAAIEA0Gd2VNCbbQ48vAF+m/i/CeQHJ9Dom9IOBN3v8XnSlOhtNzY1pJ0NRzrFQURCCeVQJaywr0et5FCXIDVqTuR4qfCu5XXU1KHpn/AvHstOd93Y0XYMwSXfNszOtpH7w9SEm7uBfKnLQNF843/MHZL4ZIdfxk73YjfMjrYFcvJwvM0AAACBAI9Wi2wG9Rrdbrgk8tqbnKMxxJCYt/Vakn5mQSc/WHqMUn14+Kb7oBtfTsfPFNK952FzU09ptNQvNWTPNMniI8l8lQi2B+Z28XUlFLL76uxJXWko99xFpf/d1kFBL0M7GRGmvvrMfeTCI+WIqh2jBVqbl4o8kjNx4p0VWyoiWeFJ gregf@pudgy\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwSe7szyqMTqKK3D/JsXIa1PtPf1p8Njk0sMDSsJPA0gBGysmSgstke9fglOgeNIbNOBNDOY3vltuNCgUiQgVWOYhUaVpj+ 2019-12-05T04:57:54.170 INFO:teuthology.task.ansible.out:2vtSoAWpXMQbNaWPGKhbxvJWMCLVckLENyoUZUBKLmTvRyKDjjq8CJ1TnfA2Xfo3a796EIOtD6essU3lAVIfIUILGD+HqptiHSB0spUJUVRxxlO2xw9rLYij1eUPRHrr9xWvDamr5JtKVtCLOL3M9QlltHieKlbJSCZq/EVxW9NAGA6IiqOhw2E1k59oGZUq4Qg+YLgYkB80Ms7nb1xpjQ8txvY+73xL/8i7Nqmr7ekKJfyTpRe03nCQ== gfarnum@mozart.thompsonian.net\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzqJQfdAKaggPDgSpk8P6QjLeyQYOz183fVSKnqAa5EeaCJozS1dn8z+iphswT3YL6wqIMqVHMfkSUcEeTBwZmvydaghOMJEIQSP1PCzH0V2XprELrEcBC3QTvaElPjjEYgBVUvEw8sL/RIQ7mgt9vxD0NcCDspsz35fmU222KzFxD41PZ5XzHQ3Lumwdax7mcpqrw8Aa5/tgcEqAIu54HYRf1Qmk6+ueMAvpwkX3eiSEtkVvuV1Kxpc3/0mOfJ7cn+eyPX3qUbo4oY76hIYseoUIwEaVE/pNW+pjwDSy4gSOcuxYDEGKzCwZPZeQtRQWzRt3YNafxi5xsByobGLlZw== gfarnum@GF-Macbook.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1ik3LbyKsPiGwdh+vGrKSYLvhMg7OkqMwDNYU5LqWBSotogxyGQJUuD80ZavxmmdXC4qScddHtWOJjtyX1CTSDUe7INpSgGPGnmSExVFpcYRnmxHAB+9M/c9VorUkQ8IOKLmRO/BdYma3Rl9NzhHI73yd4zJlx1K4NuQnhnj0Z4yXO+k44RkBWDdMicBB5lBP0hETaH2Iq1u5Dai3u6N5N9bSPXv2PGKvvU395K0Wts2CxMhOBc7LAixysNzNWLrnG1ghS4jPVG2TIHOElP5lIXUv1Wlkool5rTUWgnVY/7loWkDus2JryDM7hLGFXqgT7HHcpI4gSITqRhWxzsUF gregf@fedoragreg.localdomain", "key_options": null, "keyfile": "/home/gregf/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gregf", "validate_certs": true} 2019-12-05T04:57:54.277 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n', u'name': u'davidz'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCyOWnAdvzh5zR2ts9e8g0wG4wZ0uNr0YJUCncAOTJB0L8xkQU76jWzX3BRNU5Wk8VtLvpJAQ61P7UXyzgGWSO9FHZjD3RDrc2V9uh/lqRKmx1COtDvcart/JwzwHlsWdrYdcXvEaAiFrZJ6x1jJAjIL6tvMO7bqBz4YNruKpGWsQ6p5P/M5iIEPMj1oKanLoULXqReC6eYFTw3LNAXjberPY5GxYwaKS0HuzySRvSblShyA85jawTSlEbsOSnMqqGu5vGJiC415UFQQ5qKBtzrgAjlUgMbaqGTnTmBYn7L751gIWRetgWqwuHgfIe2ZGM3mmh0DX8E03DUbEt4UxyR dzafman@TrustyTahr\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDDtSwGzH7V5jzppggL0+nYlBSM1HqGjkInwxutra5iEyFoBuV73MNSBk3HrvIVGN13+PCdHf4Oy8ywbhC/4yKbxS1NG76d/GU8ijZFuBlEwxwHZpcThpSTuYrNx7FJVp8Yjl45DtD5s8FMI/0YZZhLlIqT6QCTxlTIPXDb6zefEWvAVziE0bmhRpe4B6AOLOGTB9fivQr5YzxhfaSS5Vu4U1x5gmosouSJ+aMFrQWLqa/eoCy8t9PnxdMjlHjHCZ7Qv1JfpaAPw3qOK30GrVeXR/hPN3aFodTuIJpzdynjk5HO6/VKW5+mWBO7pmQqn9LNQEjADXbLh7PG//nOONbH dzafman@ubuntu16", "key_options": null, "keyfile": "/home/davidz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "davidz", "validate_certs": true} 2019-12-05T04:57:54.334 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n', u'name': u'joshd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDgS5EfJyOxBkl/TtJikfb38rm6muQo7MwYufkLSBw9W2jFzt/ZMbVXyVsdZM4I6hWfZm1/q60UcER4yoA1xwYGOTmbFXrLUqqKxKTnCdyejuQ3miB7aiuisWO4vjb2+ppBwL9Mx775DPeQJehFyFK0qgh33tZO/9LWWYy7NOZdA9L2ei1PgMWa0VBeijcDR5GQSXzh0Yi+h/Ezu8jevu+NkW2IUJF6zPGjd9M9lAy7KG2dQjqqCX4XWmUS+MHJS6f9bsC5i39GWy29FvX/qFEArOMzuYRIFWE8iza95KSBxptb+Hw8PFBjQ1kTh6nC5ZV2X3ETcoj8fToTcgW3dTRF jdurgin@new-angeles", "key_options": null, "keyfile": "/home/joshd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joshd", "validate_certs": true} 2019-12-05T04:57:54.430 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc', u'name': u'gmeno'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gmeno", "ovpn": "gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvWnKfzTd574C4GXg1mr4pbJOInKv4qMRf0DwksTzqtXStlhT+PaO2C8p48U2Xjz5hmPK9J+iHA2xKwI6lKI194wUo3tW283euRyTCN7LlbV6tgD9k78hDaeA+53MHYmwbxjHEqAadqrLU+68LfmL7gaaUSToP/Ap+hdcDC7H9lulMcxkUeahEiDfG2s1mD8IUM6XVuqZM9WEARKpF1oSEBKhMMob9YJZPpJWipuAGkPvDA0yHK1lB58Yr2x0yaezVjQc7V2dwaY1gXR6AscU5WzlwWNNwXHqPdyyWRaGx/9Fk00yfzX4V7ZqTvKdMLKwB7EnamduTMPsaoKBUBJCh gmeno@gmeno-virtual-machine", "key_options": null, "keyfile": "/home/gmeno/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gmeno", "validate_certs": true} 2019-12-05T04:57:54.473 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n', u'name': u'davidz'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCyOWnAdvzh5zR2ts9e8g0wG4wZ0uNr0YJUCncAOTJB0L8xkQU76jWzX3BRNU5Wk8VtLvpJAQ61P7UXyzgGWSO9FHZjD3RDrc2V9uh/lqRKmx1COtDvcart/JwzwHlsWdrYdcXvEaAiFrZJ6x1jJAjIL6tvMO7bqBz4YNruKpGWsQ6p5P/M5iIEPMj1oKanLoULXqReC6eYFTw3LNAXjberPY5GxYwaKS0HuzySRvSblShyA85jawTSlEbsOSnMqqGu5vGJiC415UFQQ5qKBtzrgAjlUgMbaqGTnTmBYn7L751gIWRetgWqwuHgfIe2ZGM3mmh0DX8E03DUbEt4UxyR dzafman@TrustyTahr\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDDtSwGzH7V5jzppggL0+nYlBSM1HqGjkInwxutra5iEyFoBuV73MNSBk3HrvIVGN13+PCdHf4Oy8ywbhC/4yKbxS1NG76d/GU8ijZFuBlEwxwHZpcThpSTuYrNx7FJVp8Yjl45DtD5s8FMI/0YZZhLlIqT6QCTxlTIPXDb6zefEWvAVziE0bmhRpe4B6AOLOGTB9fivQr5YzxhfaSS5Vu4U1x5gmosouSJ+aMFrQWLqa/eoCy8t9PnxdMjlHjHCZ7Qv1JfpaAPw3qOK30GrVeXR/hPN3aFodTuIJpzdynjk5HO6/VKW5+mWBO7pmQqn9LNQEjADXbLh7PG//nOONbH dzafman@ubuntu16", "key_options": null, "keyfile": "/home/davidz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "davidz", "validate_certs": true} 2019-12-05T04:57:54.553 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1', u'name': u'ivancich'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDkIZyWnJXYv/z3EVeKHatjFUAdaomIMqzvFLrA9VskeM0Ctv8a83Wk3BbYvTFQWALjZjS2HVYzi5p6kTQ4qxXlcO7HxRw0j4otNauilvgIc4Lu5N4pUtYf0piEo+nrXz+WhELfmJ8t1Yx00uGEl9pOF04SOuXtpwwrCFAO6WMCcjDuYhoI2VBKO3W/OWcIuQ8ZY3j4iDdofFOUGyyoYVyOi/HILKjluYB+ues5+wBEkDuoi3MafCnIj8M0O5fU8/JoOTksOu779UN6blPiNdEgnwD00jA+RWYY6b12ZVkiF4rUVGCiiKKUwOOWkRhkA5OEcBgZ8cGWFgGfZM1GJA// ivancich@sleepy", "key_options": null, "keyfile": "/home/ivancich/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ivancich", "validate_certs": true} 2019-12-05T04:57:54.661 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc', u'name': u'gmeno'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gmeno", "ovpn": "gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvWnKfzTd574C4GXg1mr4pbJOInKv4qMRf0DwksTzqtXStlhT+PaO2C8p48U2Xjz5hmPK9J+iHA2xKwI6lKI194wUo3tW283euRyTCN7LlbV6tgD9k78hDaeA+53MHYmwbxjHEqAadqrLU+68LfmL7gaaUSToP/Ap+hdcDC7H9lulMcxkUeahEiDfG2s1mD8IUM6XVuqZM9WEARKpF1oSEBKhMMob9YJZPpJWipuAGkPvDA0yHK1lB58Yr2x0yaezVjQc7V2dwaY1gXR6AscU5WzlwWNNwXHqPdyyWRaGx/9Fk00yfzX4V7ZqTvKdMLKwB7EnamduTMPsaoKBUBJCh gmeno@gmeno-virtual-machine", "key_options": null, "keyfile": "/home/gmeno/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gmeno", "validate_certs": true} 2019-12-05T04:57:54.708 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n', u'name': u'wusui'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsTZI+b02boZv8jVLHQoFlloD2WF/3xT+ZfZxzwddL7Q6eqo7j8i21LAbLAMWXYMyhiClrloDeHfCZ42gG5bqSThF8LoCw+GspF6SahTI0lmptGu69/OeHqqCpW1DnsWB3wWqfi90XbQJrpg3cdRaqOCmkGVJrqfFWzff8Zg9IuojIH/aO9drCTRGa1pHHCyEdW9ZNeYFwrMjfP+wJi/mC/ABAoXyDy0NWT0uIA+36abNg2/ptok90rdan0r72hjFEz/1hpupKWri+VXAepiyDMuSgJm9uJCXDFaNTn3jocdX+sbVtltThQ2OBxe1T60vRLbXx3lvhxxZ1fUkcP/FF wusui@ubuntu\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD2UzyfBk1nZ8SLCtW9UjorMeOo4u9AM/YL8ZI74GdprR+odERZR17/QfwT5vmLVNa51+CQt1HcGodlQYfR/8n/OUl/gJyVagBuU2zgmLPO3tOor5RBfms5MZ2VZ9Vm3+WrCO/vrAq/5HVyGbQsKiQHXkIRSL2NIgfKRTOVTNmuTSXtZNDgdWGIBmcOFQv+Im9CRopoI8ZtJLQj7xfxtGtUas7FbWTcf0o92+xvWuGWq5WQgmZ8MUIfvBLYLmicuVfsWjZ1YsC1PNhuhkwrW7MhaD71Ej3ll243tG1QD7eAmo+psIp8phlU5s/5FeA5y5OzDZ2T963oHGhq018TYO0b wusui@teuthology", "key_options": null, "keyfile": "/home/wusui/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "wusui", "validate_certs": true} 2019-12-05T04:57:54.821 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1', u'name': u'ivancich'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDkIZyWnJXYv/z3EVeKHatjFUAdaomIMqzvFLrA9VskeM0Ctv8a83Wk3BbYvTFQWALjZjS2HVYzi5p6kTQ4qxXlcO7HxRw0j4otNauilvgIc4Lu5N4pUtYf0piEo+nrXz+WhELfmJ8t1Yx00uGEl9pOF04SOuXtpwwrCFAO6WMCcjDuYhoI2VBKO3W/OWcIuQ8ZY3j4iDdofFOUGyyoYVyOi/HILKjluYB+ues5+wBEkDuoi3MafCnIj8M0O5fU8/JoOTksOu779UN6blPiNdEgnwD00jA+RWYY6b12ZVkiF4rUVGCiiKKUwOOWkRhkA5OEcBgZ8cGWFgGfZM1GJA// ivancich@sleepy", "key_options": null, "keyfile": "/home/ivancich/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ivancich", "validate_certs": true} 2019-12-05T04:57:54.852 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304', u'name': u'zyan'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCtmxLK2rSomB14d9/fkAshmsXQfd6cJ+vEb0MZDaUT+OLmr1vAq0+H5g9w3hApJ2UEOFSaiz6ywgBAFeBeu/KmUALOhJIKKk0imoTunYclqxVPBK6tG/2b+ME8VE1afoEpMem3Bzfkrq0pDt6DLFBDlt9rRIqtO/ZlqEayWkEwTXJ7pwj+oJS03Z3Y7q0Iqw8PToKqJWwy2OvW1I9Pi0Q8s4YV9I34rSO7Wv4w43AXpFHRlvJWIcI5g5WFI5tnjel+kUAWMZft9ynFoAP5iXdByq897r4jAF74hapaxLMsHBz4MyYmPZqS76Cx2e3kGqXeetPinfcZxC3ViboR7o/1 root@zhyan-desk1", "key_options": null, "keyfile": "/home/zyan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zyan", "validate_certs": true} 2019-12-05T04:57:54.979 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n', u'name': u'wusui'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsTZI+b02boZv8jVLHQoFlloD2WF/3xT+ZfZxzwddL7Q6eqo7j8i21LAbLAMWXYMyhiClrloDeHfCZ42gG5bqSThF8LoCw+GspF6SahTI0lmptGu69/OeHqqCpW1DnsWB3wWqfi90XbQJrpg3cdRaqOCmkGVJrqfFWzff8Zg9IuojIH/aO9drCTRGa1pHHCyEdW9ZNeYFwrMjfP+wJi/mC/ABAoXyDy0NWT0uIA+36abNg2/ptok90rdan0r72hjFEz/1hpupKWri+VXAepiyDMuSgJm9uJCXDFaNTn3jocdX+sbVtltThQ2OBxe1T60vRLbXx3lvhxxZ1fUkcP/FF wusui@ubuntu\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD2UzyfBk1nZ8SLCtW9UjorMeOo4u9AM/YL8ZI74GdprR+odERZR17/QfwT5vmLVNa51+CQt1HcGodlQYfR/8n/OUl/gJyVagBuU2zgmLPO3tOor5RBfms5MZ2VZ9Vm3+WrCO/vrAq/5HVyGbQsKiQHXkIRSL2NIgfKRTOVTNmuTSXtZNDgdWGIBmcOFQv+Im9CRopoI8ZtJLQj7xfxtGtUas7FbWTcf0o92+xvWuGWq5WQgmZ8MUIfvBLYLmicuVfsWjZ1YsC1PNhuhkwrW7MhaD71Ej3ll243tG1QD7eAmo+psIp8phlU5s/5FeA5y5OzDZ2T963oHGhq018TYO0b wusui@teuthology", "key_options": null, "keyfile": "/home/wusui/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "wusui", "validate_certs": true} 2019-12-05T04:57:55.030 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n', u'name': u'yuriw'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCw3MhMNDYjkiQADoC1mg9ga0Yj174FHIFt4RQIC8qK+6cB0N3CM5R4JwP8pR4tYkVFWmnYRScYK5VVdC873Ac2BzYxmse97zLJTbUnYWTdY+OFi1FUozsy3PwtiNHSd48Z/PBvkinQIey2QlMKaW3/yz9Of/Yfm4prxg9lBsd6/tOOFj3AsqIQ1OVhMfgeVEkdCL7AAbEYyMxePZ3UFrDwj05HDLgmFzx1xsNJFOuBwLfU/QNHoXJQrcVy0rZq9fB5rg74iz1JZs0uDWocQQK4Nd3Rv+wRGBuew2fmkrS2bQWXzMoSL+/i7pbf74LKcbWAE0wXEImqQeREHxq4ZZ5J yuriweinstein@Yuris-MacBook-Pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJUmq7btpBMjjODHXp1888xoBuX+rhiZg3ufuo69RgDxUMymJn3CPJ/5NGXoZ3SGAWKk0w0Dx6K/VO4y92mAV+cO81y8wZR4J3fPV3IbWRjN0X3IFoAvJi5n9v5hpG7z3RZCGmQnOZ99m1i9dge/3S8NpuPLIvYkSo9z00d+mNP6D8rGnqOmEBfEhUQXLeH7rbPwaMWPgB/hrpiPCyLR/+sGC5bq9PAsKnocqZ12cnvCtYxpnA7et6nL15ftEpUrKYAmRDxotuoUcsllP66GSszjbrgN6OBNDSqpGt/+RSJJwzZEGQ073gJuMveKy9tvD/PyEttjvVKL2KsBuXGSuL ubuntu@yw\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmXWgRRes8bzekTH2fTcxdAAquFn1SjPhws1 2019-12-05T04:57:55.031 INFO:teuthology.task.ansible.out:Z9CkxNaqKe25XnzPehqrhg9etouUwvORQHdDJdTlmhTSYtn588Omq+1bvniab0KzAYgGZsWdPma/3E9qF7ChXIMik9Umip0drzYTaexCuC/Nu46brTxlnL0vaxnnVjDwmHC0CEJs4qN4e2wSlsA0uBDRWjxWFMdTUgSzII5xasHypUYZ0y3MNdbfY+SVF69DEDH8EOQMGYm8JQXNJFloQvmvLUFqT/63FsdpzRVTph4A8hvI1f8i6ULb7fPQN5ZS62wdXd/KvrBXrCvhOi7R63RM60mkMRXm1XajTGmCKiikfiqgTr yuriw@Yuris-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC41kRr6B/RC32DrQKRByalH832H6BpCQF9XdTvgYi4+LJt+kAuIGLDF4xm4uApTmWRrLspWm0Kksc0ger03uNx5qJqMEUnJvjxJnSDC0/XyxWapVxIviFkvv0BjbhVNOOaMU55ywrZ9hHgMdaFwuMuCOaB7f9Tm6IUXm2k82OXueeIG23Kziq+IQBysWOjvNUy/LLpl7l4+xOkhNUupUiQCqHvFH98VUdYWkHeyamafGVZmYbFLrOXgmviKo0u8LPxcT7psKLk0yFG7fG0DSak6DDdKJqb+G/NqMSPxRSAI4FbXQ6UOZb32Ozr1Sql9yfOXEYWZuatn7W5F8DjSv3p yuriw@yuriw-RH", "key_options": null, "keyfile": "/home/yuriw/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuriw", "validate_certs": true} 2019-12-05T04:57:55.131 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304', u'name': u'zyan'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCtmxLK2rSomB14d9/fkAshmsXQfd6cJ+vEb0MZDaUT+OLmr1vAq0+H5g9w3hApJ2UEOFSaiz6ywgBAFeBeu/KmUALOhJIKKk0imoTunYclqxVPBK6tG/2b+ME8VE1afoEpMem3Bzfkrq0pDt6DLFBDlt9rRIqtO/ZlqEayWkEwTXJ7pwj+oJS03Z3Y7q0Iqw8PToKqJWwy2OvW1I9Pi0Q8s4YV9I34rSO7Wv4w43AXpFHRlvJWIcI5g5WFI5tnjel+kUAWMZft9ynFoAP5iXdByq897r4jAF74hapaxLMsHBz4MyYmPZqS76Cx2e3kGqXeetPinfcZxC3ViboR7o/1 root@zhyan-desk1", "key_options": null, "keyfile": "/home/zyan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zyan", "validate_certs": true} 2019-12-05T04:57:55.184 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n', u'name': u'tamil'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxrS30JmTZbnBd73s9PGZYxCkWH3u4En9Q0bYJan45W+j+y5vMzaNUVsp8We1tphqPQ9B0ygi80OxbHWu5gyv4g+M/JFtrnxv3QgBCYWeBWfh652Vbdlk1thl6btFnHUhVMSn5tkdnFxyVDJHIROoVXkHRaCNAUUy7tB60IhPAMomNcCGu9R8+opONFLXKTIfkYQ1cGbysr90DMs8ASEkvxdv40hIWhK4ufvL7CFFtgXuoC0c1WedBFOkr5LDegzL13xN97K9o5J8CAU0MUKk6H8KNKbQuNq1zANwymMFY599GIlxwai8iluJgfOObDRdCDmKM2f9RAEUXQM7oYoH7 tamil@tamil-VirtualBox\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvwiWVhNLeOLg72cpVIwbUYnmFh+Cqf/mE7Rc/Xci91on3GPrq++dC8t2GhTB5Zyq46EE27SmUkiZpHRnz7xwiJKM/tTgNEkffXDK3j0pA2E22iqV9FpfSA9Zdi1Puq7CPT7/rix4rF3SPvMtg/R1fkyornLFdzhJiMExCKJXGwf+QsKk4NVnr+nO3EEkKOoP1wio1bceZ53rCt9Tyu8NawAB2nL2fvtBPQDAriQ8bLX69XLbOfhFI6GJo815/j6CQxNmvA0l5ABJc8+A71jZmndzumFRgqBuhiCM5ohsGR0JGahUQjFEA4quNTx8EWTGRsqhAYszzpqJzuN+Xb6tJ tamil@macbookair\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEPzEyDGI7poyXVPG70vgQpvlU/lTt6Vvv5CoHWJoSrHklDdJEqekhCB87BX4DTrqbb5xoM7yfxle/z9UzgVwSUWqsPdZHA4EbrNFRKFxi/gc2YrygVSjumTyzf9WJJ5vCZvTQexOj7rjWVnJkjXHKMXZXIAQN/bJf6J21t+4A3XLrl30/4XXyq664NONUNjBHxHtlbS2LM6Th/hK4B23oj2BugX2j/50vWgd9H36bPr8+JGskOi6MH2ndZv5pYtLznFzQmsbixjaN8kCuyQdlRCOndPEFoHFhDla2MeB+h3XQLDXW7J1TR2URFSfcNq5ndDS4i0b+t75Fjhb7AtLT tamil@ubuntu-delldesktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6tJ9y3solDCzFdXcczO8GJEwMqMYXWU5kuMz0rr/US6F72j0spSNmwGemMOZE9s7KtHgior/Ufw8/vMRkR 2019-12-05T04:57:55.187 INFO:teuthology.task.ansible.out:hoBYB0C5KT9utGNqrtDdvH98h43XnKZ3K/yaHv/T+u2EYWNgHtOXMtgIVFnGwSD7RzF6TbwSfurprTV6ZUyyOxRcmHnkmwpbNwiwIoXB03k2PITdYCC3lEBEY0OC+vAW/E3+bFbV3RSEA0Cm31jmMyZyPeYiXiEnPXUIClsoDj34NwJTIgctwIsqM4eYUBg3pHliWYKEWDs55HZYoj3W/6Jw7MASzyAYFSmP8rQu8iS7tO1iEBRGOXFNxHirGQ1WqUn tmuthamizhan@MacBook-Air.local", "key_options": null, "keyfile": "/home/tamil/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tamil", "validate_certs": true} 2019-12-05T04:57:55.300 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n', u'name': u'yuriw'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCw3MhMNDYjkiQADoC1mg9ga0Yj174FHIFt4RQIC8qK+6cB0N3CM5R4JwP8pR4tYkVFWmnYRScYK5VVdC873Ac2BzYxmse97zLJTbUnYWTdY+OFi1FUozsy3PwtiNHSd48Z/PBvkinQIey2QlMKaW3/yz9Of/Yfm4prxg9lBsd6/tOOFj3AsqIQ1OVhMfgeVEkdCL7AAbEYyMxePZ3UFrDwj05HDLgmFzx1xsNJFOuBwLfU/QNHoXJQrcVy0rZq9fB5rg74iz1JZs0uDWocQQK4Nd3Rv+wRGBuew2fmkrS2bQWXzMoSL+/i7pbf74LKcbWAE0wXEImqQeREHxq4ZZ5J yuriweinstein@Yuris-MacBook-Pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJUmq7btpBMjjODHXp1888xoBuX+rhiZg3ufuo69RgDxUMymJn3CPJ/5NGXoZ3SGAWKk0w0Dx6K/VO4y92mAV+cO81y8wZR4J3fPV3IbWRjN0X3IFoAvJi5n9v5hpG7z3RZCGmQnOZ99m1i9dge/3S8NpuPLIvYkSo9z00d+mNP6D8rGnqOmEBfEhUQXLeH7rbPwaMWPgB/hrpiPCyLR/+sGC5bq9PAsKnocqZ12cnvCtYxpnA7et6nL15ftEpUrKYAmRDxotuoUcsllP66GSszjbrgN6OBNDSqpGt/+RSJJwzZEGQ073gJuMveKy9tvD/PyEttjvVKL2KsBuXGSuL ubuntu@yw\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmXWgRRes8bzekTH2fTcxdAAquFn1SjPhws1 2019-12-05T04:57:55.302 INFO:teuthology.task.ansible.out:Z9CkxNaqKe25XnzPehqrhg9etouUwvORQHdDJdTlmhTSYtn588Omq+1bvniab0KzAYgGZsWdPma/3E9qF7ChXIMik9Umip0drzYTaexCuC/Nu46brTxlnL0vaxnnVjDwmHC0CEJs4qN4e2wSlsA0uBDRWjxWFMdTUgSzII5xasHypUYZ0y3MNdbfY+SVF69DEDH8EOQMGYm8JQXNJFloQvmvLUFqT/63FsdpzRVTph4A8hvI1f8i6ULb7fPQN5ZS62wdXd/KvrBXrCvhOi7R63RM60mkMRXm1XajTGmCKiikfiqgTr yuriw@Yuris-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC41kRr6B/RC32DrQKRByalH832H6BpCQF9XdTvgYi4+LJt+kAuIGLDF4xm4uApTmWRrLspWm0Kksc0ger03uNx5qJqMEUnJvjxJnSDC0/XyxWapVxIviFkvv0BjbhVNOOaMU55ywrZ9hHgMdaFwuMuCOaB7f9Tm6IUXm2k82OXueeIG23Kziq+IQBysWOjvNUy/LLpl7l4+xOkhNUupUiQCqHvFH98VUdYWkHeyamafGVZmYbFLrOXgmviKo0u8LPxcT7psKLk0yFG7fG0DSak6DDdKJqb+G/NqMSPxRSAI4FbXQ6UOZb32Ozr1Sql9yfOXEYWZuatn7W5F8DjSv3p yuriw@yuriw-RH", "key_options": null, "keyfile": "/home/yuriw/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuriw", "validate_certs": true} 2019-12-05T04:57:55.328 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n', u'name': u'jowilkin'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDYGVqKZgLx2LJ49x0kGo/Z5r8MVefSUCYhhrK/DK8ht+YK1AqLXErjAedgpVz0WuwirRjh5YkiXGMYiOdkMUYWGk5nwCHEKnnCpne7Xh1OOqge/QGHlUigWVH8VIHBsZ+wTW25mqOrgDSGB8+Qk7Dr2o1sLGUHcx7vITDWOrer4KjF2+H5TQH/7irUVORu6fzeGKtGyOrTwQ+ZT1+fuhfznDULFGyvKGdcXxxNOg+21Fssk2nCNTlgRCGjHSrfelfuMT7QEt+9puWd7MWXYQg7xx8jkJGMpEQRFxJTxnfUlUXf3x2lc9Q/SvKfpVMtc0N+bk2F9+euyvRvjtn4LJNx jowilkin@jowilkin.wilkins", "key_options": null, "keyfile": "/home/jowilkin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jowilkin", "validate_certs": true} 2019-12-05T04:57:55.456 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n', u'name': u'tamil'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxrS30JmTZbnBd73s9PGZYxCkWH3u4En9Q0bYJan45W+j+y5vMzaNUVsp8We1tphqPQ9B0ygi80OxbHWu5gyv4g+M/JFtrnxv3QgBCYWeBWfh652Vbdlk1thl6btFnHUhVMSn5tkdnFxyVDJHIROoVXkHRaCNAUUy7tB60IhPAMomNcCGu9R8+opONFLXKTIfkYQ1cGbysr90DMs8ASEkvxdv40hIWhK4ufvL7CFFtgXuoC0c1WedBFOkr5LDegzL13xN97K9o5J8CAU0MUKk6H8KNKbQuNq1zANwymMFY599GIlxwai8iluJgfOObDRdCDmKM2f9RAEUXQM7oYoH7 tamil@tamil-VirtualBox\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvwiWVhNLeOLg72cpVIwbUYnmFh+Cqf/mE7Rc/Xci91on3GPrq++dC8t2GhTB5Zyq46EE27SmUkiZpHRnz7xwiJKM/tTgNEkffXDK3j0pA2E22iqV9FpfSA9Zdi1Puq7CPT7/rix4rF3SPvMtg/R1fkyornLFdzhJiMExCKJXGwf+QsKk4NVnr+nO3EEkKOoP1wio1bceZ53rCt9Tyu8NawAB2nL2fvtBPQDAriQ8bLX69XLbOfhFI6GJo815/j6CQxNmvA0l5ABJc8+A71jZmndzumFRgqBuhiCM5ohsGR0JGahUQjFEA4quNTx8EWTGRsqhAYszzpqJzuN+Xb6tJ tamil@macbookair\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEPzEyDGI7poyXVPG70vgQpvlU/lTt6Vvv5CoHWJoSrHklDdJEqekhCB87BX4DTrqbb5xoM7yfxle/z9UzgVwSUWqsPdZHA4EbrNFRKFxi/gc2YrygVSjumTyzf9WJJ5vCZvTQexOj7rjWVnJkjXHKMXZXIAQN/bJf6J21t+4A3XLrl30/4XXyq664NONUNjBHxHtlbS2LM6Th/hK4B23oj2BugX2j/50vWgd9H36bPr8+JGskOi6MH2ndZv5pYtLznFzQmsbixjaN8kCuyQdlRCOndPEFoHFhDla2MeB+h3XQLDXW7J1TR2URFSfcNq5ndDS4i0b+t75Fjhb7AtLT tamil@ubuntu-delldesktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6tJ9y3solDCzFdXcczO8GJEwMqMYXWU5kuMz0rr/US6F72j0spSNmwGemMOZE9s7KtHgior/Ufw8/vMRkR 2019-12-05T04:57:55.457 INFO:teuthology.task.ansible.out:hoBYB0C5KT9utGNqrtDdvH98h43XnKZ3K/yaHv/T+u2EYWNgHtOXMtgIVFnGwSD7RzF6TbwSfurprTV6ZUyyOxRcmHnkmwpbNwiwIoXB03k2PITdYCC3lEBEY0OC+vAW/E3+bFbV3RSEA0Cm31jmMyZyPeYiXiEnPXUIClsoDj34NwJTIgctwIsqM4eYUBg3pHliWYKEWDs55HZYoj3W/6Jw7MASzyAYFSmP8rQu8iS7tO1iEBRGOXFNxHirGQ1WqUn tmuthamizhan@MacBook-Air.local", "key_options": null, "keyfile": "/home/tamil/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tamil", "validate_certs": true} 2019-12-05T04:57:55.480 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4', u'name': u'bhubbard'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDP/yVttBEs7mpVfJL+reXb/BfB7DsLrA5ArVqEJjySi3NOz3e/wE8KRWbHD1UdraAU3P8b96GdAQxGOs4n0muOMejmaXWZCP3HkRL+rcd+03ZI1lb6ga9XjBARZfb2h+kg5jTJp/Ll6mwAYbSXvQ4MuDQzwGRDX5Q5Xme7OZ4Wb+vBmmkihHydySyP9YXbWWsBw4jvZCFSKMGsUVy/HIt1coZx/PyCG/K2Kl0nrE5ux2QqfdTIpzYo4c3y6uMIHDvt29lRmk3Gnh2//U5eN8mmRRKJlkvE8DKy3VYsl0JsP9pTWUZEYgV0EX251WTO0akQ2bKuszqd9Ai04RWnXUu5 brad@dhcp-1-165.bne.redhat.com", "key_options": null, "keyfile": "/home/bhubbard/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bhubbard", "validate_certs": true} 2019-12-05T04:57:55.609 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n', u'name': u'jowilkin'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDYGVqKZgLx2LJ49x0kGo/Z5r8MVefSUCYhhrK/DK8ht+YK1AqLXErjAedgpVz0WuwirRjh5YkiXGMYiOdkMUYWGk5nwCHEKnnCpne7Xh1OOqge/QGHlUigWVH8VIHBsZ+wTW25mqOrgDSGB8+Qk7Dr2o1sLGUHcx7vITDWOrer4KjF2+H5TQH/7irUVORu6fzeGKtGyOrTwQ+ZT1+fuhfznDULFGyvKGdcXxxNOg+21Fssk2nCNTlgRCGjHSrfelfuMT7QEt+9puWd7MWXYQg7xx8jkJGMpEQRFxJTxnfUlUXf3x2lc9Q/SvKfpVMtc0N+bk2F9+euyvRvjtn4LJNx jowilkin@jowilkin.wilkins", "key_options": null, "keyfile": "/home/jowilkin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jowilkin", "validate_certs": true} 2019-12-05T04:57:55.640 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n', u'name': u'yehudasa'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAyRDKb+InDsnW4E/S12Pr6rK3wDfUG3j+iw/NsCWwN5xpJEA1/NhVlRzY8/jejc6ddmnNKWbm/UcYVrbo7yClRTGu45g0Qm9lmfvgDgi1k8mwUCy+edbQ/R8FB9cIHHCnUuWzRmE2XGaeDiSVQgiYqq3mOPY/hwe3G7jwFalYPsbmY1jNqpzODmKFMRd1bOwdHEOMTfh0owHWqhhXakU8em30lYraQVYH24FplH2NUBArxmTuctyfweHxx7yH3ZFcXFUpUMSLuVDWb1A7fMiL6eW6wiwh+dD+q5iFNvuioZITP2EjIsb9nS+eM9bOJZuZ+5pln2EsHee27LjunbwgWw== yehudasa@fatty\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDF3PxJYgI3tS7SG+RnUvxMxr0/XwBAio0oiP4AC+o8rUYYyPd/RU27TR1kphbuJhQWVtaZ6kFSBQeNQUxv6rb8hmMzvlP1FDEoGkjaOHOuYEc2DFkkEAfaR2AlNKn3pUQgLE00TLHctCgd57Cn2Ppt8xEi/vTuo0gX3xjWOL4N9QDElCXKNdxdyyJ/RNb/Mid+YxgKKjf8ppmhYsQ+Qqu4YwPzLkfK1Qb3YtR6Hbjb4jxC7C1Z13h0vUgHe7qSmUCYNo9eC+lEgNmmnLb2sXJaUUJM8MEex6v+4muWM9hV0cq/Wplrx8fhMWMa978sxH8yVkE3Svqc/K4ZWN59l3xT yehudasa@yehudasa-XPS-L401X\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDvplQk8xkSXF03uSJh3H4zyZHRMjvgUJBgGN27Kb7gAlIQv306wPKIprOwp+jDhEcPnEWBMRJMUJuBi0vEiqmDHt8mg8ZYcs+Gi7MbUgBokILdJuSG89KAgq7gBCuIhYbSg35pfvXvIZswiXvySqMrJNB495GYj/FrcWEBfQMuedMSkIfNzqPV9yR5YRh8fnSs3fjCzV96vzu7O4AzgwICbM6XmqVLXPax/32Hq0dz7sAawlQFCJYPHzVzjwm 2019-12-05T04:57:55.645 INFO:teuthology.task.ansible.out:ih46TRJnXrxewsRhk9GwKfI4SWaFIRdOdadGFSLgdZ+2iGE7hTZ3sRgkLOj+4zKa+xTbtjjXfw5up7J3hAmrK15Tz yehuda@yehuda-940X3G\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfNxvAw6cb33jg36vw8kJu48/BttmFJ4BFeanYqwRgtWZrKpAAGHEgCiNeyIttMrI3t7Spcdyp6dy6n0P3WUFVgHWEYtGqXwXqDC9UXISkUJVVDZoEynpGC92RghhzH53+7jX1Ytpr30r/xb1ktryTPCoEJRqyrg2j1b167i6PV5tuJCA+77gW5LWBmMXJCprjnfnJ92Gp8ZewBB6gL1x+3RFkMxGsibL0Qw1CTKfyAHNYc/jos6YiL0b42Ha+NmKwvzZ1GgIleRMleisSOyKPUlPdR7UnmtEXa9bl3Gc/hc7Zt+v4iUypr2WL0X4Yl7rJNH/AAd/Z6jPkAX9nHYzh yehudasa@yehudasa-desktop", "key_options": null, "keyfile": "/home/yehudasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yehudasa", "validate_certs": true} 2019-12-05T04:57:55.764 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4', u'name': u'bhubbard'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDP/yVttBEs7mpVfJL+reXb/BfB7DsLrA5ArVqEJjySi3NOz3e/wE8KRWbHD1UdraAU3P8b96GdAQxGOs4n0muOMejmaXWZCP3HkRL+rcd+03ZI1lb6ga9XjBARZfb2h+kg5jTJp/Ll6mwAYbSXvQ4MuDQzwGRDX5Q5Xme7OZ4Wb+vBmmkihHydySyP9YXbWWsBw4jvZCFSKMGsUVy/HIt1coZx/PyCG/K2Kl0nrE5ux2QqfdTIpzYo4c3y6uMIHDvt29lRmk3Gnh2//U5eN8mmRRKJlkvE8DKy3VYsl0JsP9pTWUZEYgV0EX251WTO0akQ2bKuszqd9Ai04RWnXUu5 brad@dhcp-1-165.bne.redhat.com", "key_options": null, "keyfile": "/home/bhubbard/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bhubbard", "validate_certs": true} 2019-12-05T04:57:55.797 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4', u'name': u'dang'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDx37xbP0jYqopygG66LbSMRPoqpGUPzQZ3G8eqJCk2I dang@revan.aa.redhat.com-2015-07-17", "key_options": null, "keyfile": "/home/dang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dang", "validate_certs": true} 2019-12-05T04:57:55.928 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n', u'name': u'yehudasa'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAyRDKb+InDsnW4E/S12Pr6rK3wDfUG3j+iw/NsCWwN5xpJEA1/NhVlRzY8/jejc6ddmnNKWbm/UcYVrbo7yClRTGu45g0Qm9lmfvgDgi1k8mwUCy+edbQ/R8FB9cIHHCnUuWzRmE2XGaeDiSVQgiYqq3mOPY/hwe3G7jwFalYPsbmY1jNqpzODmKFMRd1bOwdHEOMTfh0owHWqhhXakU8em30lYraQVYH24FplH2NUBArxmTuctyfweHxx7yH3ZFcXFUpUMSLuVDWb1A7fMiL6eW6wiwh+dD+q5iFNvuioZITP2EjIsb9nS+eM9bOJZuZ+5pln2EsHee27LjunbwgWw== yehudasa@fatty\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDF3PxJYgI3tS7SG+RnUvxMxr0/XwBAio0oiP4AC+o8rUYYyPd/RU27TR1kphbuJhQWVtaZ6kFSBQeNQUxv6rb8hmMzvlP1FDEoGkjaOHOuYEc2DFkkEAfaR2AlNKn3pUQgLE00TLHctCgd57Cn2Ppt8xEi/vTuo0gX3xjWOL4N9QDElCXKNdxdyyJ/RNb/Mid+YxgKKjf8ppmhYsQ+Qqu4YwPzLkfK1Qb3YtR6Hbjb4jxC7C1Z13h0vUgHe7qSmUCYNo9eC+lEgNmmnLb2sXJaUUJM8MEex6v+4muWM9hV0cq/Wplrx8fhMWMa978sxH8yVkE3Svqc/K4ZWN59l3xT yehudasa@yehudasa-XPS-L401X\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDvplQk8xkSXF03uSJh3H4zyZHRMjvgUJBgGN27Kb7gAlIQv306wPKIprOwp+jDhEcPnEWBMRJMUJuBi0vEiqmDHt8mg8ZYcs+Gi7MbUgBokILdJuSG89KAgq7gBCuIhYbSg35pfvXvIZswiXvySqMrJNB495GYj/FrcWEBfQMuedMSkIfNzqPV9yR5YRh8fnSs3fjCzV96vzu7O4AzgwICbM6XmqVLXPax/32Hq0dz7sAawlQFCJYPHzVzjwm 2019-12-05T04:57:55.930 INFO:teuthology.task.ansible.out:ih46TRJnXrxewsRhk9GwKfI4SWaFIRdOdadGFSLgdZ+2iGE7hTZ3sRgkLOj+4zKa+xTbtjjXfw5up7J3hAmrK15Tz yehuda@yehuda-940X3G\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfNxvAw6cb33jg36vw8kJu48/BttmFJ4BFeanYqwRgtWZrKpAAGHEgCiNeyIttMrI3t7Spcdyp6dy6n0P3WUFVgHWEYtGqXwXqDC9UXISkUJVVDZoEynpGC92RghhzH53+7jX1Ytpr30r/xb1ktryTPCoEJRqyrg2j1b167i6PV5tuJCA+77gW5LWBmMXJCprjnfnJ92Gp8ZewBB6gL1x+3RFkMxGsibL0Qw1CTKfyAHNYc/jos6YiL0b42Ha+NmKwvzZ1GgIleRMleisSOyKPUlPdR7UnmtEXa9bl3Gc/hc7Zt+v4iUypr2WL0X4Yl7rJNH/AAd/Z6jPkAX9nHYzh yehudasa@yehudasa-desktop", "key_options": null, "keyfile": "/home/yehudasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yehudasa", "validate_certs": true} 2019-12-05T04:57:55.960 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb', u'name': u'branto'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2YyDoeKdaCuJif79AxJAorqpUj33WHbHu/eIWpHju+iIF3f94K0qyEXgs4AwZin9pzfyd8N1aKNT8ZFPss6L0Xk68aO/6tdInq+1lTMMUC5v/LBqsIV2btPTFHrQWA0R10v05aZFvFaSqhHNsZaj98vQV7P+jiSs3yPWqCIWUTQ1VXnCeQnB4R5tob0gpMuonhkKq7vrJPhf6ge9Av+yjL7+8oP5LJqFJ5npDxkwZ0u3L0QQ9ftAT/e7Y7ESpwpgaI3Trl0EwAXFmiLjXEUA+FBbE7ko9kLqXqRjc1C0BdRG9sO0QKBGDBEx1W3w8rCPKgaDlVyqsBT1yvIjqqXDOQ== borix@dhcp-31-190.brq.redhat.com", "key_options": null, "keyfile": "/home/branto/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "branto", "validate_certs": true} 2019-12-05T04:57:56.090 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4', u'name': u'dang'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDx37xbP0jYqopygG66LbSMRPoqpGUPzQZ3G8eqJCk2I dang@revan.aa.redhat.com-2015-07-17", "key_options": null, "keyfile": "/home/dang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dang", "validate_certs": true} 2019-12-05T04:57:56.097 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6', u'name': u'xiaoxichen'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDO00mus2XPX/EfUZ/YsKI84TnWj5TbZP7Fs01EkoOTPgNoYysVdf0PwhAooz1qPrv+HUsUMpQT6uNYfCHyqx9rScR+XUlyQuFe7Sjp/dn3y+b1PKa6G4HX7f1eejR3ZaLZXA4R/8oucAXqQS5pDr3Y7prwkrp9cyk8uawBXoDcAlA1SMIPZPrtO1+iZw20NDhixsG7NE5jMFfw3q8xTckZEqBbteybci7X1v4lEYXCVlbSbUhyZ2Af8HEj09JBDOL2bI3yLvj9a5Oj/HtKubyApzZpA7rgeoii2UZQGuSRI1VJktp9YC0zGk5S843xYW92utbALKyQ6DdB2F5gc0DX xiaoxchen@LM-SHC-16501223", "key_options": null, "keyfile": "/home/xiaoxichen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiaoxichen", "validate_certs": true} 2019-12-05T04:57:56.234 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb', u'name': u'branto'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2YyDoeKdaCuJif79AxJAorqpUj33WHbHu/eIWpHju+iIF3f94K0qyEXgs4AwZin9pzfyd8N1aKNT8ZFPss6L0Xk68aO/6tdInq+1lTMMUC5v/LBqsIV2btPTFHrQWA0R10v05aZFvFaSqhHNsZaj98vQV7P+jiSs3yPWqCIWUTQ1VXnCeQnB4R5tob0gpMuonhkKq7vrJPhf6ge9Av+yjL7+8oP5LJqFJ5npDxkwZ0u3L0QQ9ftAT/e7Y7ESpwpgaI3Trl0EwAXFmiLjXEUA+FBbE7ko9kLqXqRjc1C0BdRG9sO0QKBGDBEx1W3w8rCPKgaDlVyqsBT1yvIjqqXDOQ== borix@dhcp-31-190.brq.redhat.com", "key_options": null, "keyfile": "/home/branto/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "branto", "validate_certs": true} ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e', u'name': u'ffilz'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCz7e2G3Ziaus2yMviKh9s65e+yaqy1+ezs5IC6US90/8WumNAWqC4Yr3ckvCc120r3VwpJWcvaAWCF+4IZeO43lSNLadbCnJONed5YklcUuIJvjC1eDbZTMLJ7h0ijgNXgAFmFq0AV2NHuE2L2VKXk2hsIqDVGjZLUpi7W+ic07xnScEoXXDSk5mhJ6bKmFsUYCC91g2roYs9Ui5fGd+ey28SFOADI7osDLpLkiTF2OzGel93o5qobNZwKR0AxGsBIta17mvf3I3307bGnvugDXpNB0FaS1QZhIzMO4G4LZL9zahCvv4wm5GmuO4F0+Ac7dwFu3KZKtHQe3g6WHuAv ffilz@redhat.com", "key_options": null, "keyfile": "/home/ffilz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ffilz", "valida 2019-12-05T04:57:56.236 INFO:teuthology.task.ansible.out:te_certs": true} 2019-12-05T04:57:56.378 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n', u'name': u'joao'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpGZK2cWUtE85Bt7fl+kSeiC0lM/EN+H2X3ePGq3LiD4g5R5n6nJsLaJaIZXdJ96G0JSnj07G8Xu1CkVbGDHuVg2O9tnxLhz6tzDwFYNFrQIpALlQwOq8OiwGDDyFkMXQ3JqCCWvyD0HLysOJQIBIlb/o92ZCSAGOyNX1JjrhXF+mlBk1AUW11UeZOeGZQ34MKHapH7T/DuexCz5gfZI/jY9Rg3hr4jb1zGttvAF2eeLJCjqD/Vi4mowYB/Kn/PJhNixeA2sli2i8r6yO/bFmA/PWnfSl+EqD4mUJWb5ATeJ3hyW8qVLOogY6XVu4Sd1VgL7l1RYzXuxJV3Rb1Yw2T joao@sepia\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDV0jbKpbmXW1bhRIOBIdVV1L73qjls7s5pxiW2EMPvZLYzxLBE92e3Vf8bSMWlm1Lyb7a+u5BR85+2c+h2F59sgU1Saek5ebPpFU5qBn8AzYH18iMVSPciByOzwS0NXwWnksckWxUkQgP9bZP/1ORQ7jWZEOHZ5mEQ0oMkP0bfG1cpFrNTxOhDmwbCiQ8dp6KeZaMg4hErALr5Eh/EESxKIkuwQzk9bDbtjPRDu0F2dE0NCjs0k0BXmDA80GksxuOacRxE95K6/er11qst/VqA7o6BfhxJMTXWMdj8NhzIUNjZ8LqLTG6P/XGR24sA2d2WBhlEKRMZNqeEjzmi1REr joao@tagide", "key_options": null, "keyfile": "/home/joao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joao", "validate_certs": true} 2019-12-05T04:57:56.379 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6', u'name': u'xiaoxichen'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDO00mus2XPX/EfUZ/YsKI84TnWj5TbZP7Fs01EkoOTPgNoYysVdf0PwhAooz1qPrv+HUsUMpQT6uNYfCHyqx9rScR+XUlyQuFe7Sjp/dn3y+b1PKa6G4HX7f1eejR3ZaLZXA4R/8oucAXqQS5pDr3Y7prwkrp9cyk8uawBXoDcAlA1SMIPZPrtO1+iZw20NDhixsG7NE5jMFfw3q8xTckZEqBbteybci7X1v4lEYXCVlbSbUhyZ2Af8HEj09JBDOL2bI3yLvj9a5Oj/HtKubyApzZpA7rgeoii2UZQGuSRI1VJktp9YC0zGk5S843xYW92utbALKyQ6DdB2F5gc0DX xiaoxchen@LM-SHC-16501223", "key_options": null, "keyfile": "/home/xiaoxichen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiaoxichen", "validate_certs": true} 2019-12-05T04:57:56.534 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n', u'name': u'nhm'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHhYb/mGgHbEntAN2iPFTV1k8MYDs+jtsJDPE9MWtUZXksdeG8cEfXDBGhiBsprjELsW4+czJBqqESugFMevEysTVKIo5lc95bf1nApxfiRyMnAxb5wQ2OeLY62x26TWJhPu6HMajV+gb8CKjxDobf1Ix4ELbYlL77SOH1VVDtVZ6HiwtYngXAPMONCdI99cuwJlBpv5RjDoLwljOoxEXIh+1Fsvth+iymp0rIPuJx91DzYYgWZFfCLPHRgDy0H7YRdQWUzhrxTyzuT27vsNh5L8HgdowzOu9f2HwGzlTlxli94IJLMBY2359xsqjSswZzAv+BiPDAZMUwJpYvRvSt nhm@clusterfaq.org\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1JAKymiRDyVar5P75cMycxz9u8VsLuNtf/0t1na4ofa4YmanKswPQQz1Am82pvyPqXlBbdIGVneyMrkEoFAk4nTtfmCl7WdtaxnH87Dr9BOLEp+lIPHy7jpS/dcVmKRPB1qCYZ5GNAZ9g+DrXdaeOJdUOpIK49BP1IK8Uw5eBWAx/ae6/12yz+d/6e9kBvO7OUamMSne6Ghe2pRoI8jFOdWMbQy3ofb5wSHZGPNDn0YTGbNWHlfEAcrFmGQSTwohca4SqCJ1NgKnYu4qPVmb3Gbpm+MpUPs8Gy++rQ8wVZ5xWZW/yReIbJCqqVY4vrQzVywQXRucIrB1VyvShuNW3 nhm@metropolis\nssh-dss AAAAB3NzaC1kc3MAAACBAM7BfQAinQmLlkTgt7IIg3nQyyaJRiWA4agfu/HncrcAmepGqPCwy451g1eLMD3AC4QptHnV80Y47YQTyX1VQixIQ6M9zHG6Bo7IXzyqFHFpqnDZN25DaMM4na5bf1CwC9RtR7VWubojatPylOpflkChUBuAGWnzwkz2cxl4Dc47AAAAFQD35l/Lg4eTqZ83eTgJ5lihzHS/awAAAIB/ZdMSE6M0zEvdNPsWY28bvQ0QXN7w1ncVXGachA1JOEFGYvsbHeoywLpSYoiPLV57/wZT2dUzG70u80IQGildZ1WgQNn7C9r2Z3ZUgKmRGyC+PpYEa6nv+U4CHdMViBRXxNrP0bj2 2019-12-05T04:57:56.537 INFO:teuthology.task.ansible.out:CfQvUxUxrHWtOu/da2W23cpSs6Wz8gtz2QAAAIEAvf4vx/32Sy+kbvqOCYz0wu0ItlECN0qOk3gAMvphYMI2sW2ez2+Tsua4eoPRXT10zFxiRTkHQd5FSIvNWXV2iqed6rpwTu5K8vJz2duMIyI+xWLHo4XudtZdNxggkf+RW9JvPOMLMp4sCzBXxpreHBsb1CynndKeYVrgMLybZ28= mark@Marks-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfDOM2v4h6Dk4pMWPQxxHI5rhFtLjkme/4hAp6ki/kJ1XOSzBQdQmD2UPDKvfO9lKO7v12G2Wt+re+0giAcSkfej4+xqV53u7d2pTbMI5U3zohZt8AgESx22DAxj1wBZYuyGO2TRI7DRQMxC47E8EJgcmZBBnMKk1Rz8k71pTmg+CI2vjWx/2x2bup9Mqe4uMA8ubjg2qRlWEqPKIKcHZyrAPdIlpy2FSZ9SPa0mJpdl6aYcFNgonD8BO6A0u2RPO90KmEWrRWu69vRvbbTDiqE0CkJJlAlu3jeFVD8US9vymBHSI0BDFUbukuyG1tAujytqGBRLzxQDU9JOYMP45v nhm@mocha", "key_options": null, "keyfile": "/home/nhm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nhm", "validate_certs": true} ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e', u'name': u'ffilz'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCz7e2G3Ziaus2yMviKh9s65e+yaqy1+ezs5IC6US90/8WumNAWqC4Yr3ckvCc120r3VwpJWcvaAWCF+4IZeO43lSNLadbCnJONed5YklcUuIJvjC1eDbZTMLJ7h0ijgNXgAFmFq0AV2NHuE2L2VKXk2hsIqDVGjZLUpi7W+ic07xnScEoXXDSk5mhJ6bKmFsUYCC91g2roYs9Ui5fGd+ey28SFOADI7osDLpLkiTF2OzGel93o5qobNZwKR0AxGsBIta17mvf3I3307bGnvugDXpNB0FaS1QZhIzMO4G4LZL9zahCvv4wm5GmuO4F0+Ac7dwFu3KZKtHQe3g6WHuAv ffilz@redhat.com", "key_options": null, "keyfile": "/home/ffilz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ffilz", "validate_certs": true} 2019-12-05T04:57:56.667 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313', u'name': u'nwatkins'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaetIvR7YgpCq142fBQtOYsJrOMW+Wxu1beNXPJ8qzT5MxD05sdKntVE0DPveUydaeI8/vHeNjXDKN8mt1eO7Lf/rDKHOW/Y/36bUfQAODru9sdNCEJfSPgq1y6+X1ISljvLBXOOl+nQXlx3n5OjThNQbmdthgCT2Fk4hEoK7gn4+ol6foTkTdUXIBam8LuPU6f1KIG6ae1oAVFWLeavAbvmxUTmVl04+hKJWSPnvZxWupaPuiHvsImaQ7GwBjKhzL1wSMly0+pa2zbVYZKkLBA7Fx6fzP7jcEieOOPM1YU0jR5JF+WVzeRFJ1V9n75yOw6S4ef/X/zwWx42cdfx3N nwatkins@martini\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDRivdSX6cbPrdMrjz1oYgn0hb8VQo+QU7wD+AcCQi/CH06ny+2QjSr7bfdpgQ81Wgwqt9+e3KcLD5qdt3tneC8sgSAR5QUQrWKhPejrqdV+3990k8k0KoKIOZ9MGFc1FBv5IpuT/2d535iHJb0ZEg8mFKqKDw71sBEoQT4WElatf5DeVC507hP7GnrhZgtTPcrVddOLkmmqTQtLE1zvo9Y/eel8O2JMImMd6mdVBIcFQ1alU1wV4jLdMpagoPelvVTVeNkGRnOgvvQac5RXtGSMP1R5PoGz+1FgRhDCtdTXRfOdtrlJPtUjYiGPpIoKIUymlvQGjUrrK3WiQIIkIfDtFFu48EN4drjgQpCEQ2D0NMKEmk74mjx3CJDOF1Nj/2Fa1kjECiz4EM7sR8dDG5cW5Y8XsDoomgugM+rsOH9kWFFClHUJmdlNGWLhJnu9UQHLYZjQw3zT0F+FXUsUxKJUVTUdxQx5HBO9QFsSe13NPDnX64+SQ2VVStt8oQAvFO+6260zbux0RVaJM5dTNwnIHD/uI3PeR1z2b+iKhyj/9VP2uBRpGydLw/etQu/lS2K30S+kZIDAp0sqwXH89bdq3AJBipcryrnW1Ov9K5Mfl1UlV95/MivrnOFISWN1F9XR1kuIIYdCkI7hq/FHZ9gd84AjTiERfxRv8VCIHBVEQ== nwatkins@redhat.com", "key_options": null, "keyfile": "/home/nwatkins/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nwatkins", "validate_certs": true} 2019-12-05T04:57:56.670 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n', u'name': u'joao'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpGZK2cWUtE85Bt7fl+kSeiC0lM/EN+H2X3ePGq3LiD4g5R5n6nJsLaJaIZXdJ96G0JSnj07G8Xu1CkVbGDHuVg2O9tnxLhz6tzDwFYNFrQIpALlQwOq8OiwGDDyFkMXQ3JqCCWvyD0HLysOJQIBIlb/o92ZCSAGOyNX1JjrhXF+mlBk1AUW11UeZOeGZQ34MKHapH7T/DuexCz5gfZI/jY9Rg3hr4jb1zGttvAF2eeLJCjqD/Vi4mowYB/Kn/PJhNixeA2sli2i8r6yO/bFmA/PWnfSl+EqD4mUJWb5ATeJ3hyW8qVLOogY6XVu4Sd1VgL7l1RYzXuxJV3Rb1Yw2T joao@sepia\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDV0jbKpbmXW1bhRIOBIdVV1L73qjls7s5pxiW2EMPvZLYzxLBE92e3Vf8bSMWlm1Lyb7a+u5BR85+2c+h2F59sgU1Saek5ebPpFU5qBn8AzYH18iMVSPciByOzwS0NXwWnksckWxUkQgP9bZP/1ORQ7jWZEOHZ5mEQ0oMkP0bfG1cpFrNTxOhDmwbCiQ8dp6KeZaMg4hErALr5Eh/EESxKIkuwQzk9bDbtjPRDu0F2dE0NCjs0k0BXmDA80GksxuOacRxE95K6/er11qst/VqA7o6BfhxJMTXWMdj8NhzIUNjZ8LqLTG6P/XGR24sA2d2WBhlEKRMZNqeEjzmi1REr joao@tagide", "key_options": null, "keyfile": "/home/joao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joao", "validate_certs": true} 2019-12-05T04:57:56.796 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22', u'name': u'mkidd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJD+2BHStkaMJ8RStIIz+yDJCGNT1wbwUnmuO71vNlFxzIyHGvCq3ktx2xTwZz/kj6ZayXYWIRH//BhK2/iK/v+u2WQMBURfMHbf0lDMmNfbHCDBjYvEYgFo4Jwvzq4WHus0AHh9TdVt+TpcxyGPhVbUJh8NZut+w4cRIKu2xGoZZGDeLgv9ijXvDPrCdBxtpvQnF99C0bCTdPM7MK1FSCQEaRmNUgzEj0H7R+gmWi0T4HUD8ZY7y0vWbYPqT3Z07My1u2BfRBYpXnv1655/b777TkSlzSasORhSTT0AkW25u90Rsj2JWr39+49/Ia9uOyFtZIirugEv2v7v/l0grN linuxkidd@zenbook", "key_options": null, "keyfile": "/home/mkidd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkidd", "validate_certs": true} 2019-12-05T04:57:56.845 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n', u'name': u'nhm'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHhYb/mGgHbEntAN2iPFTV1k8MYDs+jtsJDPE9MWtUZXksdeG8cEfXDBGhiBsprjELsW4+czJBqqESugFMevEysTVKIo5lc95bf1nApxfiRyMnAxb5wQ2OeLY62x26TWJhPu6HMajV+gb8CKjxDobf1Ix4ELbYlL77SOH1VVDtVZ6HiwtYngXAPMONCdI99cuwJlBpv5RjDoLwljOoxEXIh+1Fsvth+iymp0rIPuJx91DzYYgWZFfCLPHRgDy0H7YRdQWUzhrxTyzuT27vsNh5L8HgdowzOu9f2HwGzlTlxli94IJLMBY2359xsqjSswZzAv+BiPDAZMUwJpYvRvSt nhm@clusterfaq.org\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1JAKymiRDyVar5P75cMycxz9u8VsLuNtf/0t1na4ofa4YmanKswPQQz1Am82pvyPqXlBbdIGVneyMrkEoFAk4nTtfmCl7WdtaxnH87Dr9BOLEp+lIPHy7jpS/dcVmKRPB1qCYZ5GNAZ9g+DrXdaeOJdUOpIK49BP1IK8Uw5eBWAx/ae6/12yz+d/6e9kBvO7OUamMSne6Ghe2pRoI8jFOdWMbQy3ofb5wSHZGPNDn0YTGbNWHlfEAcrFmGQSTwohca4SqCJ1NgKnYu4qPVmb3Gbpm+MpUPs8Gy++rQ8wVZ5xWZW/yReIbJCqqVY4vrQzVywQXRucIrB1VyvShuNW3 nhm@metropolis\nssh-dss AAAAB3NzaC1kc3MAAACBAM7BfQAinQmLlkTgt7IIg3nQyyaJRiWA4agfu/HncrcAmepGqPCwy451g1eLMD3AC4QptHnV80Y47YQTyX1VQixIQ6M9zHG6Bo7IXzyqFHFpqnDZN25DaMM4na5bf1CwC9RtR7VWubojatPylOpflkChUBuAGWnzwkz2cxl4Dc47AAAAFQD35l/Lg4eTqZ83eTgJ5lihzHS/awAAAIB/ZdMSE6M0zEvdNPsWY28bvQ0QXN7w1ncVXGachA1JOEFGYvsbHeoywLpSYoiPLV57/wZT2dUzG70u80IQGildZ1WgQNn7C9r2Z3ZUgKmRGyC+PpYEa6nv+U4CHdMViBRXxNrP0bj2 2019-12-05T04:57:56.849 INFO:teuthology.task.ansible.out:CfQvUxUxrHWtOu/da2W23cpSs6Wz8gtz2QAAAIEAvf4vx/32Sy+kbvqOCYz0wu0ItlECN0qOk3gAMvphYMI2sW2ez2+Tsua4eoPRXT10zFxiRTkHQd5FSIvNWXV2iqed6rpwTu5K8vJz2duMIyI+xWLHo4XudtZdNxggkf+RW9JvPOMLMp4sCzBXxpreHBsb1CynndKeYVrgMLybZ28= mark@Marks-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfDOM2v4h6Dk4pMWPQxxHI5rhFtLjkme/4hAp6ki/kJ1XOSzBQdQmD2UPDKvfO9lKO7v12G2Wt+re+0giAcSkfej4+xqV53u7d2pTbMI5U3zohZt8AgESx22DAxj1wBZYuyGO2TRI7DRQMxC47E8EJgcmZBBnMKk1Rz8k71pTmg+CI2vjWx/2x2bup9Mqe4uMA8ubjg2qRlWEqPKIKcHZyrAPdIlpy2FSZ9SPa0mJpdl6aYcFNgonD8BO6A0u2RPO90KmEWrRWu69vRvbbTDiqE0CkJJlAlu3jeFVD8US9vymBHSI0BDFUbukuyG1tAujytqGBRLzxQDU9JOYMP45v nhm@mocha", "key_options": null, "keyfile": "/home/nhm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nhm", "validate_certs": true} 2019-12-05T04:57:56.933 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af', u'name': u'jlopez'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrRvNeBa+/GXxkSDmaim7U1wFpdBvgcK9/eP1I4fPS5xAWyxXI0s0ryX4Jwb7auhZdcqPjgUPQvIVVRridKJPz7l6jUlbIqdnZu4VHuaGG/DfIVxsSmw5aI5qI9Fg/s16JPwOUuzcLHsSSh8tFxY9L1yvs3ztQnKHW6nN5+zhPbW8UjmTNUksXXAeCM/OZn+IOOzzvfh0vVe/mghIW2JpOjHkZv4qwaWYcHg6+wWE4m8l84VIegDBiONknZXcQaRj82nHtfdnb2FSiV4+gKP18i90ai0iTKKXUkMJNFKp1T5MuF7UuJLA7l10FmgQJsZBS2Loa5oeqbHYeztNAfUxF JCLO@oh-mbp-jcl", "key_options": null, "keyfile": "/home/jlopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlopez", "validate_certs": true} 2019-12-05T04:57:56.987 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313', u'name': u'nwatkins'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaetIvR7YgpCq142fBQtOYsJrOMW+Wxu1beNXPJ8qzT5MxD05sdKntVE0DPveUydaeI8/vHeNjXDKN8mt1eO7Lf/rDKHOW/Y/36bUfQAODru9sdNCEJfSPgq1y6+X1ISljvLBXOOl+nQXlx3n5OjThNQbmdthgCT2Fk4hEoK7gn4+ol6foTkTdUXIBam8LuPU6f1KIG6ae1oAVFWLeavAbvmxUTmVl04+hKJWSPnvZxWupaPuiHvsImaQ7GwBjKhzL1wSMly0+pa2zbVYZKkLBA7Fx6fzP7jcEieOOPM1YU0jR5JF+WVzeRFJ1V9n75yOw6S4ef/X/zwWx42cdfx3N nwatkins@martini\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDRivdSX6cbPrdMrjz1oYgn0hb8VQo+QU7wD+AcCQi/CH06ny+2QjSr7bfdpgQ81Wgwqt9+e3KcLD5qdt3tneC8sgSAR5QUQrWKhPejrqdV+3990k8k0KoKIOZ9MGFc1FBv5IpuT/2d535iHJb0ZEg8mFKqKDw71sBEoQT4WElatf5DeVC507hP7GnrhZgtTPcrVddOLkmmqTQtLE1zvo9Y/eel8O2JMImMd6mdVBIcFQ1alU1wV4jLdMpagoPelvVTVeNkGRnOgvvQac5RXtGSMP1R5PoGz+1FgRhDCtdTXRfOdtrlJPtUjYiGPpIoKIUymlvQGjUrrK3WiQIIkIfDtFFu48EN4drjgQpCEQ2D0NMKEmk74mjx3CJDOF1Nj/2Fa1kjECiz4EM7sR8dDG5cW5Y8XsDoomgugM+rsOH9kWFFClHUJmdlNGWLhJnu9UQHLYZjQw3zT0F+FXUsUxKJUVTUdxQx5HBO9QFsSe13NPDnX64+SQ2VVStt8oQAvFO+6260zbux0RVaJM5dTNwnIHD/uI3PeR1z2b+iKhyj/9VP2uBRpGydLw/etQu/lS2K30S+kZIDAp0sqwXH89bdq3AJBipcryrnW1Ov9K5Mfl1UlV95/MivrnOFISWN1F9XR1kuIIYdCkI7hq/FHZ9gd84AjTiERfxRv8VCIHBVEQ== nwatkins@redhat.com", "key_options": null, "keyfile": "/home/nwatkins/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nwatkins", "validate_certs": true} 2019-12-05T04:57:57.084 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb', u'name': u'haomaiwang'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFkUz8Wr2RPiY8+xf2hL4qMT+jA4mo1gmpIF8q7uVR03vX5NhzQVxVnXiyLCFv6susSMxpBafa7aLutqigXhDGJWUbsJII5yTnFYVer0QmudEsLmoDyvI1HCNk1IVY6aIESE50N3Ndx+agWasIU8TGKqJdDMPyiUtfrvsrGWi/91ZLX6aJSrFqBJtTpxDoNe4XL6QRPNv0wb8GF0az5graXkzVe3GkaQS9lyw8m2ougDvW1SLvoeE+lhUUGhJGkkCZKv1f+QNLhUHyQ3XkXqh4MrP3pPUKIuQ7nKiWDjCP6Db+ZcejjTRC19D2gLjhrw3Z2J7nn/loijlFzV5dp+BL haomaiwang@gmail.com", "key_options": null, "keyfile": "/home/haomaiwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "haomaiwang", "validate_certs": true} 2019-12-05T04:57:57.144 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22', u'name': u'mkidd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJD+2BHStkaMJ8RStIIz+yDJCGNT1wbwUnmuO71vNlFxzIyHGvCq3ktx2xTwZz/kj6ZayXYWIRH//BhK2/iK/v+u2WQMBURfMHbf0lDMmNfbHCDBjYvEYgFo4Jwvzq4WHus0AHh9TdVt+TpcxyGPhVbUJh8NZut+w4cRIKu2xGoZZGDeLgv9ijXvDPrCdBxtpvQnF99C0bCTdPM7MK1FSCQEaRmNUgzEj0H7R+gmWi0T4HUD8ZY7y0vWbYPqT3Z07My1u2BfRBYpXnv1655/b777TkSlzSasORhSTT0AkW25u90Rsj2JWr39+49/Ia9uOyFtZIirugEv2v7v/l0grN linuxkidd@zenbook", "key_options": null, "keyfile": "/home/mkidd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkidd", "validate_certs": true} 2019-12-05T04:57:57.211 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608', u'name': u'jdillaman'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA296in+8Sr9slF8hrzAV3dNz0q94Q4FlFaMYw08II5PQm6HOpabGOGOwOud/PPeh3lMUHhs45N5WnbgdfQdzz9lpDoNBcqxeACdMicdECMHtYWf2GuFUdTAnPBFVl8wFW5JcziQYJePUcAUJRXGGfdMLMppAi8VEuhMw/z3jz89UN+zf51Y9eT3rgAVSCR+Zn47kite4OB2S1aVMEcZBE0fPnh60i+rpN26s80fh3wG2N9JrPBEvst6LbL0aEixrlRUtoFHZuWNCIjntVMANa+idpuWSxN2dzfcoxpGLF1TFzqx0moznp0PnDs7ZAFFS1/EkeSCgBflyhPV6csG3VNw== jdillaman@ceph1.example.com", "key_options": null, "keyfile": "/home/jdillaman/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jdillaman", "validate_certs": true} 2019-12-05T04:57:57.317 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af', u'name': u'jlopez'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrRvNeBa+/GXxkSDmaim7U1wFpdBvgcK9/eP1I4fPS5xAWyxXI0s0ryX4Jwb7auhZdcqPjgUPQvIVVRridKJPz7l6jUlbIqdnZu4VHuaGG/DfIVxsSmw5aI5qI9Fg/s16JPwOUuzcLHsSSh8tFxY9L1yvs3ztQnKHW6nN5+zhPbW8UjmTNUksXXAeCM/OZn+IOOzzvfh0vVe/mghIW2JpOjHkZv4qwaWYcHg6+wWE4m8l84VIegDBiONknZXcQaRj82nHtfdnb2FSiV4+gKP18i90ai0iTKKXUkMJNFKp1T5MuF7UuJLA7l10FmgQJsZBS2Loa5oeqbHYeztNAfUxF JCLO@oh-mbp-jcl", "key_options": null, "keyfile": "/home/jlopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlopez", "validate_certs": true} 2019-12-05T04:57:57.349 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3', u'name': u'kchai'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAuTvkaUtEaLbLcYA9Zw3MoHI5ZPudKf6CAbAZdhp4RMVujimR61TT6LdRF20g6F7pH7GrI/1WP2bu7a3sZyCZ1nCJCOhrTTS+2wwy7oGo/0yY4l+Bw6i7LGmJgqpO5dDxwADMlaHUwJ7Rf1Q+Pd7NmggkAVTmaN6qUGffz9hZsj7ibD/Z8/GFtQsCEMloTLtnz4NB6Gxe54UXEKz9pvjpXBhs0xWsJtDKaPFrbKRKswuH45XPuwJhGr249ZYtwQqcgXSWhOUcPTwf5Y6fj27aL2dx566DxQzqzNbcTcO96MguIy8Ufnc0kIOQQw9oHJZtg82zwM6+2mWr9J/K94Gv9w== kefu@save.node2", "key_options": null, "keyfile": "/home/kchai/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchai", "validate_certs": true} 2019-12-05T04:57:57.473 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb', u'name': u'haomaiwang'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFkUz8Wr2RPiY8+xf2hL4qMT+jA4mo1gmpIF8q7uVR03vX5NhzQVxVnXiyLCFv6susSMxpBafa7aLutqigXhDGJWUbsJII5yTnFYVer0QmudEsLmoDyvI1HCNk1IVY6aIESE50N3Ndx+agWasIU8TGKqJdDMPyiUtfrvsrGWi/91ZLX6aJSrFqBJtTpxDoNe4XL6QRPNv0wb8GF0az5graXkzVe3GkaQS9lyw8m2ougDvW1SLvoeE+lhUUGhJGkkCZKv1f+QNLhUHyQ3XkXqh4MrP3pPUKIuQ7nKiWDjCP6Db+ZcejjTRC19D2gLjhrw3Z2J7nn/loijlFzV5dp+BL haomaiwang@gmail.com", "key_options": null, "keyfile": "/home/haomaiwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "haomaiwang", "validate_certs": true} 2019-12-05T04:57:57.492 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2019-12-05T04:57:57.648 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608', u'name': u'jdillaman'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA296in+8Sr9slF8hrzAV3dNz0q94Q4FlFaMYw08II5PQm6HOpabGOGOwOud/PPeh3lMUHhs45N5WnbgdfQdzz9lpDoNBcqxeACdMicdECMHtYWf2GuFUdTAnPBFVl8wFW5JcziQYJePUcAUJRXGGfdMLMppAi8VEuhMw/z3jz89UN+zf51Y9eT3rgAVSCR+Zn47kite4OB2S1aVMEcZBE0fPnh60i+rpN26s80fh3wG2N9JrPBEvst6LbL0aEixrlRUtoFHZuWNCIjntVMANa+idpuWSxN2dzfcoxpGLF1TFzqx0moznp0PnDs7ZAFFS1/EkeSCgBflyhPV6csG3VNw== jdillaman@ceph1.example.com", "key_options": null, "keyfile": "/home/jdillaman/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jdillaman", "validate_certs": true} 2019-12-05T04:57:57.672 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n', u'name': u'dfuller'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/r/Xf58szqJ3wLdTE+mRK1Z11f/+VABmx6804NO77ncdC5ZvD3Kx6WP/oje01iwWEVXNAgEOaXp6j+U10aPvBMaS30qf/IhUcrXwO1YcGzZ0pV6UZMxcTuJw6w1VXy/YtgW5BlIfSm78zabztES9BqCgjYYM2rml3L1teDF4WU1QiI97iW0R57yc0t/gqXGpxqSFQ+iNYAl0Q1MGUgYltLvPrhm9WI2WhR2/GivdVMEM5lViNlegxneAZRb9YZMCG2kmSP+9Zd2NR367OrlgzmgeY/9u+KGXQiyfaO330s+iksxVwwE3fDl+KnqXuHSJNccJV7iXzZKej2QAire4X dfuller@Douglass-MacBook-Pro.local", "key_options": null, "keyfile": "/home/dfuller/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dfuller", "validate_certs": true} 2019-12-05T04:57:57.796 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3', u'name': u'kchai'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAuTvkaUtEaLbLcYA9Zw3MoHI5ZPudKf6CAbAZdhp4RMVujimR61TT6LdRF20g6F7pH7GrI/1WP2bu7a3sZyCZ1nCJCOhrTTS+2wwy7oGo/0yY4l+Bw6i7LGmJgqpO5dDxwADMlaHUwJ7Rf1Q+Pd7NmggkAVTmaN6qUGffz9hZsj7ibD/Z8/GFtQsCEMloTLtnz4NB6Gxe54UXEKz9pvjpXBhs0xWsJtDKaPFrbKRKswuH45XPuwJhGr249ZYtwQqcgXSWhOUcPTwf5Y6fj27aL2dx566DxQzqzNbcTcO96MguIy8Ufnc0kIOQQw9oHJZtg82zwM6+2mWr9J/K94Gv9w== kefu@save.node2", "key_options": null, "keyfile": "/home/kchai/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchai", "validate_certs": true} 2019-12-05T04:57:57.805 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n', u'name': u'owasserm'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDH/YT1mGZg50qDdngGQ4Ea+jfJGZ582J6pQbYkrvuGsyATMzNwk73sAbn7mgbpw7nkW/ol/vwahXHZ5heC056BFnqiZ7tjoDeotDx/1XxFtkbyIaVflUeD1vBww38v6IslLHcVhPilauPVZSfVzEHI7JB0a+9OwNrEH3+dt1B6vR+WRy2gHG/aEJffPMv4yv/0FgRWy3cEiUvGlLq0KVVTbsqqLQhw/u5k8bBIoibBtZLFAMTrjLYp9Dv8dxJROvmey6O2TuX6gCY1rLGTAArYULpXH73OLIqaksJKc3srgoJrTSZPO6bo3BQeLz4ODJ7uvTpvQIhacBpp/dq1ln7p owasserm@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCiqChxNdielMPBchs0u6bxWstqU9oeY2K9qf1PNwEIrwDYTrpDjsdoVnS750tOBjWY0wr9Gf9ATOJXi89151oao9GaUueQ1rTgmu7GtPlKsuBUVNMFzLEa0mqneqDA4AfknisFt/LnL8fLM4BddT25tmQ4oUGS7F17p4kXqwboK/DsJciCMWWc/5AT4NVwdnYFzIA6ta25/bB94kpktaZ6HDw7G8q/mK62Yza3EZfq8SP7p0sPL7oVsInQ9AutcVonmALgzPa9t1/MPGaXUIqARaWtwe4MSlNEIs4afLFyteBVeq+dgU01yeyjFivwVmuS+aPFTm6sjGGe8/gOsFDIJiVertOqi663m7q1WYGlVANfif7uL2+uJ9yrvOmrFh5s/CJMJBRG1SLfK1j/ivFwvPAQc/uMqEIRzrVZOsuNLPO4zHPWzmBAPUteb+2BqSdUFYMlBxb3lmhFFffroDqnOK1nbdAfmyKiosUizQgX2Gs2wjjtejuhS9dpYkNu+sBOujsxa2mZUouyWf4fXyWGVjthrCZaM+FyeSDpJrgwgRiSKNB8aRxB10ZNo9WZEy6eH2adp0Xf6v2KsKvryDndqBEkBI8zLbGrPVHGSEqiSBWc+NeAB06uSscJd0pHU3hDg0qW0EaxOXaial9qrBiwcFVKEr0g7LDURQ6KZsZy9w== owasserm@owasserm.redhat.com", "key_options": null, "keyfile": "/home/owasserm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "owasserm", "validate_certs": true} 2019-12-05T04:57:57.929 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2019-12-05T04:57:57.958 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n', u'name': u'vasu'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDVFy1sjGvvHwS7Q6E61ykH4dUGfxW9+a7Xo3JMpTBT33/E8bTrYF6ZHSdH+uxScisV9Vll222NJCLXyx2s9EOcv4L88vWDTZGCaUb3iA5dweLm7EYi1NXT/p+jWDA7MAF1FQGSAQafTHpNRxLcrVPgsLDy35weV4xj7ShEU+w7EBDb75jlJXqy0luJLKNBrLzz/wDV9Oe6y3APwQBhtO7vGrVzrWEF/LWLe7U9ad3DQjvLM7+H+QVpNy+nt7us73K8vHDk1KrPlCCjrcN+fSGPD/ILlxrDrDeqmmBCtPMa7PSSjCyAV/EW0fsmArcHrHXKeKWTKXG9sDL0KBqos1kB vasu@vasuSrv\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQConeaJ2kvUOMy+jeKvXZTw655L3KK0ZG0HONIySFrRNkGa/M/hYf8oMNVh9SEax7vLNme0kwk7d71EhgtMGNsvHtQOkshWxM9P0XPk4HgB/OL+2LQ/7M3SwD4vuXPbqvrQS63YMoxcHBue5Ol5hXmM+qSoZajpUrfB2woqbp7aKsCpoK4Cgj4XfLHfh0n239ae38DU6GoAcL0GzMsxMdQhWzBwOO+ZHfowohc1pfY0+xGIMW1OlMN1eO7kuK2gGS/YRVOnLw9uu/2NErIjBzCihNN0PYWur2JE+KmU1x6VMaky49cKmAlc5WUqZ+FpV3gvnb3cLjBClnCkdjIcLeRR vasukulkarni@ovpn-116-227.phx2.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRFyOO7qitrq6Hk2NSl28h4laRTxA2zezy1wwjTqs6UL+oZjNRYQjeWD3dDNPZrUmc9JyI3j1EN11cCsKAp8YXz3ItB46M0BPeJ4TXCpg36vxs7/twSDMq/EoXA5sRBbUJFf+bthcNemwG1WsJoxRVzMezbWBTHvz5i4Hbh4sL3PBbac3FhT/3b5Q4tKdO6bXGXmNLAoALyVZZMfoSz9CWzvrO+g8Houb9iJiaXHp293y+ND9Wicl9wHPckO1VcocZ+ldByujgX1cz6m+pxmf6 2019-12-05T04:57:57.959 INFO:teuthology.task.ansible.out:AKY1OMJj5PjmpN0HrgpV2QLVA+ba9qBiDwi1g8sRHXkOETMZ2jq/2a/mmTtvrsF vasu@teuthology", "key_options": null, "keyfile": "/home/vasu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vasu", "validate_certs": true} 2019-12-05T04:57:58.063 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n', u'name': u'dfuller'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/r/Xf58szqJ3wLdTE+mRK1Z11f/+VABmx6804NO77ncdC5ZvD3Kx6WP/oje01iwWEVXNAgEOaXp6j+U10aPvBMaS30qf/IhUcrXwO1YcGzZ0pV6UZMxcTuJw6w1VXy/YtgW5BlIfSm78zabztES9BqCgjYYM2rml3L1teDF4WU1QiI97iW0R57yc0t/gqXGpxqSFQ+iNYAl0Q1MGUgYltLvPrhm9WI2WhR2/GivdVMEM5lViNlegxneAZRb9YZMCG2kmSP+9Zd2NR367OrlgzmgeY/9u+KGXQiyfaO330s+iksxVwwE3fDl+KnqXuHSJNccJV7iXzZKej2QAire4X dfuller@Douglass-MacBook-Pro.local", "key_options": null, "keyfile": "/home/dfuller/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dfuller", "validate_certs": true} 2019-12-05T04:57:58.107 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37', u'name': u'smohan'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlOImwCV0AGu5L29ABaefV9Vqjv43WYJYFuLTxGQQdeQGcways6iNKGIX/nzf3UWkwgg25OVSpzB7zf6FjBx+HcQU+RJ7FLzeaz0s8ZxBXqWwtOYOvP9cXmAXA07hD5UAfxZKC1twBzL82bKJ2ZSmzfQmBjvpUOqRs7XwsQIClOW/ycY+jLgLRLFFDxzADsgnn6UEGlHeMETYocUbe89P3wbY5fHhqZmn6e5ncuAQMsiLRa9d+QhSdpD07Y/P65EzqRF4UOX+zpOc6gPlgHFir9pvf0bTbbBG/qKN9+KGqzZMRJGKenpHM6EMfLkf0QmrT4s4zd4ZnUQPq3HN3pul3 redhat@unused\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9n4ege98BNmvgUxogdcpptMILFfEisAQaiLUvSKpTsoy2gF7C6xSJXgDaziT5dW4ad2El4xmKQ/e6/e14Vx5i9pGbJz4NvL36Xmb3ent2PtY18i6d3CbnmGI8J7YH4YS9QvR7FC/KuvwLR6AdJ+uQShGhqG8kWJMjbqrtaxu8l1hq8ymBXw3O1Ht+xrZ/r3dEkAIVF3bMCSUnZpEbSsNoeiydEQl0ZCw+meXYmSw15vYVtKpYscZxqADqhSJJxIEHZA4E88dq6aYdV2AZ/s4cJK+tBl/ckBnclI3VGHSGn6oaHoO/fCHtKiENplDuKoLGQPO6xJ2Q/apHSGD7McY3 redhat@network\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDn/XgNfvqytLHgfBUdQIhJiWBlj+JrXqGZYvAZZJIWs32Ugm1/lCQTssGZrEj3+Qv81jLfkP6VQ8gXSWJkzUK4csxRxkZKXqcYLDE1csdmAFDOfWUFJwhMsCiiCyBiW9oapioLpvd40nw8k9qM4QuzaUhVonoGXMDDL1/dCgkbIjiQxxTJPHDEJFjtzul+VuhlhHqE6S1syoJG+UKo3Q1tOBCs8rwaPXEfCXFsyv+ZmcEv1+mpM8QRjdr1D1hfUfwb4CHtaa2AE6B5wL+zoY6+jsrTwZSwCfeIdgEZ8e3V2iXP5F7G6eXotu5r7mGYJClHZFLSfbNPN5J+EbRORl9/ wbhack3r@dhcp35-24.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHYkp1nuRSYSAl6me1xuoTU30DW2YpAR6EH1c2adUC9tNtTZbMnab11aitQLAtZb9SKsRgnoB5KpH+zJ0VNKCuJU227r9vZwcVUruSGwA6X7LOrSkK0gFqNRZkVoKhC9oGHSU0zegYBDxx8VLxSV3CE56b+KEosEo6iJe7TkXy3wXHK6am0uhQPEdSHtdUdD683CGIffdF1RFmAFnzTPn96lC9rLe1mIGO+jVQx4zIo3i33RFQLy++4DtQw6Lrgp26Xr85HmQekjgyiya0VkR9Ypy7fgF2ucBGegnOgZDGCCY4G2Ruq 2019-12-05T04:57:58.108 INFO:teuthology.task.ansible.out:iZyKz2TZaOecDwrLkbYM7EGp8up5uLFRZsV shmohan@shmohan.remote.csb", "key_options": null, "keyfile": "/home/smohan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smohan", "validate_certs": true} 2019-12-05T04:57:58.253 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114', u'name': u'abhi'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZePxJmYerihzA0aQbMt7K7bctPEDcwPHiO8EVuJowp6zwhSmtDnYuFybKJsee4l8+rDKR6T5Ugdl5pLjSzQ+Ba00OSH7+qWC6shvVDYju5mIooJnc+kwx8RgapbGtDdl/yYskBb1KD5g/3+i/2aM+Igk+7nQR7Cny5+WnUEvHd7PHBbUXQSuu+mlUlnXCVMM8g8qlc1I3WQL5eQoIqGjSWRgZUIsn3P5oryYBzbuG9jqLwbaohJvZRJqkimQDe7zCFk6bB3z39pyOWUN+sldRvyL8FPyEmqog4IwpjcYyCuCnPhnb6fNq9yB65RUIde82Kkzx3LZrYOXTrdHgD/lr abhi@trusty", "key_options": null, "keyfile": "/home/abhi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "abhi", "validate_certs": true} 2019-12-05T04:57:58.265 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n', u'name': u'owasserm'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDH/YT1mGZg50qDdngGQ4Ea+jfJGZ582J6pQbYkrvuGsyATMzNwk73sAbn7mgbpw7nkW/ol/vwahXHZ5heC056BFnqiZ7tjoDeotDx/1XxFtkbyIaVflUeD1vBww38v6IslLHcVhPilauPVZSfVzEHI7JB0a+9OwNrEH3+dt1B6vR+WRy2gHG/aEJffPMv4yv/0FgRWy3cEiUvGlLq0KVVTbsqqLQhw/u5k8bBIoibBtZLFAMTrjLYp9Dv8dxJROvmey6O2TuX6gCY1rLGTAArYULpXH73OLIqaksJKc3srgoJrTSZPO6bo3BQeLz4ODJ7uvTpvQIhacBpp/dq1ln7p owasserm@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCiqChxNdielMPBchs0u6bxWstqU9oeY2K9qf1PNwEIrwDYTrpDjsdoVnS750tOBjWY0wr9Gf9ATOJXi89151oao9GaUueQ1rTgmu7GtPlKsuBUVNMFzLEa0mqneqDA4AfknisFt/LnL8fLM4BddT25tmQ4oUGS7F17p4kXqwboK/DsJciCMWWc/5AT4NVwdnYFzIA6ta25/bB94kpktaZ6HDw7G8q/mK62Yza3EZfq8SP7p0sPL7oVsInQ9AutcVonmALgzPa9t1/MPGaXUIqARaWtwe4MSlNEIs4afLFyteBVeq+dgU01yeyjFivwVmuS+aPFTm6sjGGe8/gOsFDIJiVertOqi663m7q1WYGlVANfif7uL2+uJ9yrvOmrFh5s/CJMJBRG1SLfK1j/ivFwvPAQc/uMqEIRzrVZOsuNLPO4zHPWzmBAPUteb+2BqSdUFYMlBxb3lmhFFffroDqnOK1nbdAfmyKiosUizQgX2Gs2wjjtejuhS9dpYkNu+sBOujsxa2mZUouyWf4fXyWGVjthrCZaM+FyeSDpJrgwgRiSKNB8aRxB10ZNo9WZEy6eH2adp0Xf6v2KsKvryDndqBEkBI8zLbGrPVHGSEqiSBWc+NeAB06uSscJd0pHU3hDg0qW0EaxOXaial9qrBiwcFVKEr0g7LDURQ6KZsZy9w== owasserm@owasserm.redhat.com", "key_options": null, "keyfile": "/home/owasserm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "owasserm", "validate_certs": true} 2019-12-05T04:57:58.396 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611', u'name': u'cbodley'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDcsBe6CzgWh64iZAH8zbU6ra4yEiUBSsiUn0AHzyQurvjeRpL2Mut6B1WEi0kJG7BAM6b13JrhnD9BW9yIMIUIPpVTa2o4zaXP90+bvjGrgDeJbxv3T9/TGh13k+WWyampxm2lo90/+Paql/HObQyyIv7Wan0DOdIe2/Ue4ooWH0tu3kj+3unJTuw82oEDEn+9tGs0oleG2c8XN0gQyVl1DzR2Gqf9PCzXK7hInC3niY2huoSAZzVFEgsniKEWqTnVWYi49X2SsJhUyHCj+mqrTG3YnW30nha3EXOWIbQGblXVoTIOc+gNreqdGByOMICKSnQmPSG4DAl3c1Y4h4A71T3Q043IUv7v7S9tAEKq2h6oJ9I/a5z0u4kaL/+B7Cw7l4bitTdcZ0rcqSvOkKUNNgxujbucMSSc4Tsh5L0762xaziD9nQZl5bteb3VtsMAueqhu7L3sJy69+HGg7oF68t5TguErXfI2UNVGOn3eRj+zN/zSwLUXMoROlpM5yHXSzHvpsbgZiHBImM2CeG4qaPtqzFw7VH6NWJv4ypb0htOMWsbXmLshNTca1KXrrBcVozSJC8wKgEOxu4spk26G8x1IVTsi2ypyYX+irtAqZrrpIPdQ0Ss9NjKPhFNXAxJgLHj6TlH8UNh1gRfN/QTyzpFSe4EH6uOlnWuxcuM8ZQ== cbodley@redhat.com", "key_options": null, "keyfile": "/home/cbodley/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cbodley", "validate_certs": true} 2019-12-05T04:57:58.407 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n', u'name': u'vasu'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDVFy1sjGvvHwS7Q6E61ykH4dUGfxW9+a7Xo3JMpTBT33/E8bTrYF6ZHSdH+uxScisV9Vll222NJCLXyx2s9EOcv4L88vWDTZGCaUb3iA5dweLm7EYi1NXT/p+jWDA7MAF1FQGSAQafTHpNRxLcrVPgsLDy35weV4xj7ShEU+w7EBDb75jlJXqy0luJLKNBrLzz/wDV9Oe6y3APwQBhtO7vGrVzrWEF/LWLe7U9ad3DQjvLM7+H+QVpNy+nt7us73K8vHDk1KrPlCCjrcN+fSGPD/ILlxrDrDeqmmBCtPMa7PSSjCyAV/EW0fsmArcHrHXKeKWTKXG9sDL0KBqos1kB vasu@vasuSrv\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQConeaJ2kvUOMy+jeKvXZTw655L3KK0ZG0HONIySFrRNkGa/M/hYf8oMNVh9SEax7vLNme0kwk7d71EhgtMGNsvHtQOkshWxM9P0XPk4HgB/OL+2LQ/7M3SwD4vuXPbqvrQS63YMoxcHBue5Ol5hXmM+qSoZajpUrfB2woqbp7aKsCpoK4Cgj4XfLHfh0n239ae38DU6GoAcL0GzMsxMdQhWzBwOO+ZHfowohc1pfY0+xGIMW1OlMN1eO7kuK2gGS/YRVOnLw9uu/2NErIjBzCihNN0PYWur2JE+KmU1x6VMaky49cKmAlc5WUqZ+FpV3gvnb3cLjBClnCkdjIcLeRR vasukulkarni@ovpn-116-227.phx2.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRFyOO7qitrq6Hk2NSl28h4laRTxA2zezy1wwjTqs6UL+oZjNRYQjeWD3dDNPZrUmc9JyI3j1EN11cCsKAp8YXz3ItB46M0BPeJ4TXCpg36vxs7/twSDMq/EoXA5sRBbUJFf+bthcNemwG1WsJoxRVzMezbWBTHvz5i4Hbh4sL3PBbac3FhT/3b5Q4tKdO6bXGXmNLAoALyVZZMfoSz9CWzvrO+g8Houb9iJiaXHp293y+ND9Wicl9wHPckO1VcocZ+ldByujgX1cz6m+pxmf6 2019-12-05T04:57:58.411 INFO:teuthology.task.ansible.out:AKY1OMJj5PjmpN0HrgpV2QLVA+ba9qBiDwi1g8sRHXkOETMZ2jq/2a/mmTtvrsF vasu@teuthology", "key_options": null, "keyfile": "/home/vasu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vasu", "validate_certs": true} 2019-12-05T04:57:58.547 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195', u'name': u'mwatts'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrtHyRuOSEXAM6mwL4qoM+NM4hpIKhaRS+/Y6RDoFKszkXFw/jC9ZYh7q39rQpWQBKHVe4RyPN9yyGWydFwaR/BjZi14ippmYOqVG4f7b+k39NFbMJRcSxYC+NvkqxK7+zpsnIWC+RdJ+blZi+zCbaaAYcMDSvBZW19FsJSLrO7hh4R1cuzgV+Xrmwdc1/VH+HqPMu3tHGv7N0SfsklWI8Inr0aiL83QqqgKPcVxljSWZGYvvFyTQOrkNQeaLURJNER9WIIac9o57WOINw8kR+ojebcOUBsrxZH9j+gMjsJ752Rph03wdMUpFcnS6yfvI4Ial7x/sGms4FE+wL/vVF mdw@pika", "key_options": null, "keyfile": "/home/mwatts/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mwatts", "validate_certs": true} 2019-12-05T04:57:58.561 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37', u'name': u'smohan'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlOImwCV0AGu5L29ABaefV9Vqjv43WYJYFuLTxGQQdeQGcways6iNKGIX/nzf3UWkwgg25OVSpzB7zf6FjBx+HcQU+RJ7FLzeaz0s8ZxBXqWwtOYOvP9cXmAXA07hD5UAfxZKC1twBzL82bKJ2ZSmzfQmBjvpUOqRs7XwsQIClOW/ycY+jLgLRLFFDxzADsgnn6UEGlHeMETYocUbe89P3wbY5fHhqZmn6e5ncuAQMsiLRa9d+QhSdpD07Y/P65EzqRF4UOX+zpOc6gPlgHFir9pvf0bTbbBG/qKN9+KGqzZMRJGKenpHM6EMfLkf0QmrT4s4zd4ZnUQPq3HN3pul3 redhat@unused\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9n4ege98BNmvgUxogdcpptMILFfEisAQaiLUvSKpTsoy2gF7C6xSJXgDaziT5dW4ad2El4xmKQ/e6/e14Vx5i9pGbJz4NvL36Xmb3ent2PtY18i6d3CbnmGI8J7YH4YS9QvR7FC/KuvwLR6AdJ+uQShGhqG8kWJMjbqrtaxu8l1hq8ymBXw3O1Ht+xrZ/r3dEkAIVF3bMCSUnZpEbSsNoeiydEQl0ZCw+meXYmSw15vYVtKpYscZxqADqhSJJxIEHZA4E88dq6aYdV2AZ/s4cJK+tBl/ckBnclI3VGHSGn6oaHoO/fCHtKiENplDuKoLGQPO6xJ2Q/apHSGD7McY3 redhat@network\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDn/XgNfvqytLHgfBUdQIhJiWBlj+JrXqGZYvAZZJIWs32Ugm1/lCQTssGZrEj3+Qv81jLfkP6VQ8gXSWJkzUK4csxRxkZKXqcYLDE1csdmAFDOfWUFJwhMsCiiCyBiW9oapioLpvd40nw8k9qM4QuzaUhVonoGXMDDL1/dCgkbIjiQxxTJPHDEJFjtzul+VuhlhHqE6S1syoJG+UKo3Q1tOBCs8rwaPXEfCXFsyv+ZmcEv1+mpM8QRjdr1D1hfUfwb4CHtaa2AE6B5wL+zoY6+jsrTwZSwCfeIdgEZ8e3V2iXP5F7G6eXotu5r7mGYJClHZFLSfbNPN5J+EbRORl9/ wbhack3r@dhcp35-24.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHYkp1nuRSYSAl6me1xuoTU30DW2YpAR6EH1c2adUC9tNtTZbMnab11aitQLAtZb9SKsRgnoB5KpH+zJ0VNKCuJU227r9vZwcVUruSGwA6X7LOrSkK0gFqNRZkVoKhC9oGHSU0zegYBDxx8VLxSV3CE56b+KEosEo6iJe7TkXy3wXHK6am0uhQPEdSHtdUdD683CGIffdF1RFmAFnzTPn96lC9rLe1mIGO+jVQx4zIo3i33RFQLy++4DtQw6Lrgp26Xr85HmQekjgyiya0VkR9Ypy7fgF2ucBGegnOgZDGCCY4G2Ruq 2019-12-05T04:57:58.573 INFO:teuthology.task.ansible.out:iZyKz2TZaOecDwrLkbYM7EGp8up5uLFRZsV shmohan@shmohan.remote.csb", "key_options": null, "keyfile": "/home/smohan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smohan", "validate_certs": true} 2019-12-05T04:57:58.692 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options" 2019-12-05T04:57:58.693 INFO:teuthology.task.ansible.out:: null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114', u'name': u'abhi'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZePxJmYerihzA0aQbMt7K7bctPEDcwPHiO8EVuJowp6zwhSmtDnYuFybKJsee4l8+rDKR6T5Ugdl5pLjSzQ+Ba00OSH7+qWC6shvVDYju5mIooJnc+kwx8RgapbGtDdl/yYskBb1KD5g/3+i/2aM+Igk+7nQR7Cny5+WnUEvHd7PHBbUXQSuu+mlUlnXCVMM8g8qlc1I3WQL5eQoIqGjSWRgZUIsn3P5oryYBzbuG9jqLwbaohJvZRJqkimQDe7zCFk6bB3z39pyOWUN+sldRvyL8FPyEmqog4IwpjcYyCuCnPhnb6fNq9yB65RUIde82Kkzx3LZrYOXTrdHgD/lr abhi@trusty", "key_options": null, "keyfile": "/home/abhi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "abhi", "validate_certs": true} 2019-12-05T04:57:58.828 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea', u'name': u'pdonnell'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQD3vdP9QHyNFUNPOdplezEbdYsQfcC91ncATI6Fl/xHifmUbcvjIRqZHkCY6FNWUpMeg9S3+wlHJtzjVqJJ/MlQML3EtwnzQkeWv8bjtuZ07OqULYIv0zEx63XUfSnrdFleJ8+KDSpn7krkYE44q85B88IL+ZBeq5eCXFJ1TtmsDcvsN8WqZNWMdUGqgH3Bb4D8+YKrEfGxoievyxZ1A1DnBHan2l8foTKYggRrnuSbTb59cFpxY3R7Arc5Cn9kDJT1nDVBA5SSGCfRMiNyBOUzQVYe/UjKTaCpngglXC3wYnjoGvGcpksHLQCxyuJPB6djC1tdF+3Jehdf8y8+2TBzWXiiPq2XwS58sER5GcLLLQv26pPL8hgu6CdI0sHxgcgQBgpjoLwUMHmv13CnN/AAcZY4RRDK2X5zd6wFSEhuuCoUqh/m7K219IxI6fVkLaApC851D/ua/pj9OMjiOaJ+D13cPrgfqxva4eK8FQZvqQm7aUK/TT216QNYmn7hkzNjMTkjv/L5yYQ5Ym92AHy44wHzLPrtDB5mk94vHOjG+KlAt9ofuLncMSAXWe+fOqxuIQ6VfbL2LUv0EGYhHe4XOVj98q5utsVQxQbXAZzNQaq9FWXs2A4xcVP7aWj0Fooi17LN/W0ng4Iijir5hyLr7k7M1G6resgiY7p83biNow== pdonnell@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQCpgLVt7Zphmf6nqD+9BC0N91jdmaY6ZpvrYWlzbhxXITQ6juIEtegnlZ+8ohdeEfHFBiyu1xOSQSM1X1ugvpPQE94UqQg7JVb0tuoUCNJHYnIuv8T4YgyP0/vEV48k2G0BBWlP+IJ5f1iDvZuLHWLR5JSgGqh5ELaaHr7Cbyb6JlImldb7SDIoPHMCZ+n6dLO8l5Zy7qWelw6aeA770xmPucBr6t/BVns2g77XgJz6zFd99yEaDy27i8jfzL7WqR9i93yVE9EsL7tuOjg6G9KUzxBF0bg9iVsJXypvcBuG0JwQ+SDKqYw/cN4iLy5qBa6Siqrfpc/LKU21VNmJ0e5eXNgKKbMBpBb8GVmkoLaSRcXw7LclWopjNDA+meUhDAwU7b97N7cqN8BYhBKQyiShXFHMKLuNo2fSuTRt/icfb9VisKNGBdvwgV0oOlrqbXMzZKmj9amb9b+3JDBTGB7ioCsj5x/Evf9srXRrQ9kikIr1aBbVXIWir/lasNWVydcmUYsNi8L7W0vXArLwp+GgOXJFeZUiD3z9LG4CouRxMSwoj3lgWeL2ZR8feH4D/SP/+QmBdU6+aG97au9qreeXi5X8QTR3LND0DwwEKczX+3s+Fa950Oieom/gXC57PVKVh90KVoAGw3HzJTlZrauOtLkYbTUAX6woOMhAtJ5oWuDDT4UrHQj37jRrrl2dETLwKwUJN2+2L07vnnOZ4ABcGWLsU6d3bXeUBlw3pdsYqPebDKUEpjL2kUOw+z 2019-12-05T04:57:58.830 INFO:teuthology.task.ansible.out:mfjEfdVx9BfrvExvDX+3MkxccCJ8FY7od1s5fGEFHsXnIiMnLO/HZb/JN+Aoe+/ExadIcYwJnWcCJyly2/532kGFUlzOliCwPxgJpjWCKQAv6LjWmxoPXiA4JI0bp6y7lMEjmV2Bp3mwphlsW/r8Ie5hg1UPh/ttHRp/VZgcV8e6nNmP/qc6k9b2DWTNFqtWqm1hXLOwq08HzUEeH9mTqvenyAbv2RFMOS18Y2K+4BKUzHwZujrz9qEh/kqtpFScEIXkOJ+HZz3mI6XfjWFIPS487yRGfUvW3+nSWhvHc8vJY21bFRKdjD84qCVlZ2BfOCGUZSKftVItEiqXMTRs3RZd5epfCVZbJuA0hG16jUseeHqqZqDtlbLkzkxtS6v3iGRnPh/Ke/94FHHCzvYY8cmMYAVaM6p/G3edWftbaEtN70Fo7VKxPrpPzuSRZ7Vxmnb+Zad95DxxAC/vtMy/vkBVIP6dyn4HGxqa1oECFMe4oXdhnxZQoPzvWxb8upcXeAeOMMbcIlMq5a+dCBPGFNKhNhrb+9uAEDHwjE7rZlMJVSvt9v1S4hll6jiYIap+7n4reSKrS45aO9U0xiKXJJDohh pdonnell@icewind\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQC8MdUJzIvTCn0fM+hv1HAO/Lwjy/C/V7stQRWGIxyrRHbniyX8iuaU3j7YNFglSQPJq+YRM9UnGauM9/0J9tHKLsMp+zRQYrtxU/ThiTUN8TRJY11mGUol4yLsqkTtm6oq5sl6o+LfuA4hzaYc3XLrFHRJuCUgbtPG71Q0C/y3U+itrDK4wXUxBa4YzZzcIMg+c88VXwoouPbiL0gW5PCXzMaNtKlOqBUKlE935MLn70wIEfN/EwAAbss0ga1moqeqmi9Nlvm20DNIVFmkAGtYoq6CHXtL/uehzai5Sueop8Vmd8+1U8gDTo6JpiagdD8ZNf0zAbRx587KAT5ZMnahJpO276j7RNa85ZFkNkEnBLqBHRkCkTCYq1GehYyRaR69uuKyczFiIoTfAQgPh7VlnpT9525b/4nFsBDOkWxxWAIJR61YiKemPV/n/e9SwhvVsMDb2B/RBriwR7Mi2SZ/lBHGB+z0OA+EE2rGZy4LFVOfUczqIdpltlNotXan4M4F361i+bX19KE+f2RLySn3+GiFv0CdLwslS/y4xysz+FWxw6B0Y/EkpiDU3CdWM61+THGGm/DVhfL/Pvca7k8PonHFYw/x5w1nBcADpkM9JF+m5eOnp+weTEzl58sS+3GnoAuL8YFJwMW1RmE50xM0o5E3SDc8XftUvaSZz8mhII3uwDDukbELfmkLbnIeipQKyjO+GEx0JBp4Itz82qGuao0Wi32U9wHzhSWnX4GQ0pKZnq9frSlyCNwHu/iMiKF8sA2bX3dvplFJ9nKWPefcU8bdl/upr5hYrd2GttjUSkWZa+UuF264yx3ztcN7jotRhEmmdEcHVzKQ26/NyRI4/9DQpDKY7zw4O2wFkXWg5L9kodXwWzgy4cyOr4dMD9LBVLOpHc0CAji5/+lfuYQbu1mwVdfOzCXDqByjt7zCbeN7KuUh403sJ8PTQVp642xA9i4AkPv5g+fqi5C0++1DbZP0lmLfKtTswHhoN+kAqZu2N4Mu6mNVNKyZgW5h1pZzNSGWYcZttyYmMCHhPketReL0LPJ+VUsqQCVohuta3aaSe44InA28GZsXzMj5B0el/84bnE7cN4b+rfGD25NT73JEt3RgPa9W9jqT1hwd/nf53BDKpp7DlHjR77WSWT4h4F7bhbM1RsGSqAfZjCUJT6Rq+IuM2XcrLDY8zIeArkqaZLXMzfGj6KJ7cQes05P5LYfY4osXiq68Gs/7gzF31PQRRGOYAoqMN7752sn0ZMcpnUj3mJutQcncRUN4+naTMjU36WWD7it0YuQBnb2IJ4iKjYLInw8pr2zKgWtgr1TSCM4xKp/v1/ZMmbB 2019-12-05T04:57:58.831 INFO:teuthology.task.ansible.out:mS+dTzNC7Mpkt8vdCNjzZ5KZ5 pdonnell@aglarond", "key_options": null, "keyfile": "/home/pdonnell/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdonnell", "validate_certs": true} 2019-12-05T04:57:58.837 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611', u'name': u'cbodley'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDcsBe6CzgWh64iZAH8zbU6ra4yEiUBSsiUn0AHzyQurvjeRpL2Mut6B1WEi0kJG7BAM6b13JrhnD9BW9yIMIUIPpVTa2o4zaXP90+bvjGrgDeJbxv3T9/TGh13k+WWyampxm2lo90/+Paql/HObQyyIv7Wan0DOdIe2/Ue4ooWH0tu3kj+3unJTuw82oEDEn+9tGs0oleG2c8XN0gQyVl1DzR2Gqf9PCzXK7hInC3niY2huoSAZzVFEgsniKEWqTnVWYi49X2SsJhUyHCj+mqrTG3YnW30nha3EXOWIbQGblXVoTIOc+gNreqdGByOMICKSnQmPSG4DAl3c1Y4h4A71T3Q043IUv7v7S9tAEKq2h6oJ9I/a5z0u4kaL/+B7Cw7l4bitTdcZ0rcqSvOkKUNNgxujbucMSSc4Tsh5L0762xaziD9nQZl5bteb3VtsMAueqhu7L3sJy69+HGg7oF68t5TguErXfI2UNVGOn3eRj+zN/zSwLUXMoROlpM5yHXSzHvpsbgZiHBImM2CeG4qaPtqzFw7VH6NWJv4ypb0htOMWsbXmLshNTca1KXrrBcVozSJC8wKgEOxu4spk26G8x1IVTsi2ypyYX+irtAqZrrpIPdQ0Ss9NjKPhFNXAxJgLHj6TlH8UNh1gRfN/QTyzpFSe4EH6uOlnWuxcuM8ZQ== cbodley@redhat.com", "key_options": null, "keyfile": "/home/cbodley/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cbodley", "validate_certs": true} 2019-12-05T04:57:58.972 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868', u'name': u'jlayton'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKzhyWb3PmU88m8BbgpEuBzYM6+x28z6aZwHSesiyfXMfF4Y7IZjtKaTxH0ph/O9GQCPhRCVhRd8TlgKWJGhts1E+VkQfHfT18CgmrPWzj3gifc2FJhL97vB1Pd3jksGnOPAikgYPHP46SVMTtyvXCkOUuo7E29WrkdpjawcqM1rfpw63OnzHf7Ar7Sqh/othB7ajyVQ92t+XWSEqj7K+NRfTaq5L5RBWbwarM885o8xdI5mimOZ0QjzKOhFDORrI+HEVlkutyB8E20nZgo7TUy+kYeLVhEqb3A6Pi2QbA9GFLT69kA/H9nebhj7Cuim9yrx4OKeZuIncG0UxSQF3QmWflAZHBs8L/jniTz4DJ5id5jR29tvGK2LeddCEMB1L2cHwWHlG0aX5jaNEJe+nG6H5Tcg3+M1RPM8twBkRQmS3UzAyX/MTH1kktyFUHn8CdBXe0WTuoFSvXLB6BCHWzDYZOt0tIV/xMtBwhy+XGYA4oZSsDU5m7VlSVLBGL8DbRX69Z8AGey2y3FoFu/roY6NmDhTxUedJ2ZfoKSwdZD6jjatdO1621nonBnJxrDgaXCWzT7HtnWcc3GwxZoBrqkvgpsx88f4lNi22OpYVPV6A4hECe04u6OJnHSWVk3ft3p5PaLNF8aEXbiIgzZKAwX3LxpvEDFH/t1sdihhpmjw== jlayton@synchrony.poochiereds.net", "key_options": null, "keyfile": "/home/jlayton/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlayton", "validate_certs": true} 2019-12-05T04:57:58.992 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195', u'name': u'mwatts'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrtHyRuOSEXAM6mwL4qoM+NM4hpIKhaRS+/Y6RDoFKszkXFw/jC9ZYh7q39rQpWQBKHVe4RyPN9yyGWydFwaR/BjZi14ippmYOqVG4f7b+k39NFbMJRcSxYC+NvkqxK7+zpsnIWC+RdJ+blZi+zCbaaAYcMDSvBZW19FsJSLrO7hh4R1cuzgV+Xrmwdc1/VH+HqPMu3tHGv7N0SfsklWI8Inr0aiL83QqqgKPcVxljSWZGYvvFyTQOrkNQeaLURJNER9WIIac9o57WOINw8kR+ojebcOUBsrxZH9j+gMjsJ752Rph03wdMUpFcnS6yfvI4Ial7x/sGms4FE+wL/vVF mdw@pika", "key_options": null, "keyfile": "/home/mwatts/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mwatts", "validate_certs": true} 2019-12-05T04:57:59.133 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options" 2019-12-05T04:57:59.134 INFO:teuthology.task.ansible.out:: null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f', u'name': u'rzarzynski'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCijPApnekNl/dsY+fleKYstEzNkKflBEcCoKtjItUCDJCVLVFOsDBrV94EGvDWpr13aj9sSwpUzRbeu/vOW8ZT853HH6Dqg2+eoYwG6vDRprZb93nw6l4vtU59/M0yKkyi9CPp2+YUo31cCFzYWpq8bv2j1xBruDSUCUdn9nJrHiLFqA4QEwSpw+uVCjSKzQL+LjlKmE8Plfd+8mNCQeY/OFsjxAztwN+4M50SZCnOVCIwEXoYRNd82JOXMOaqZJazSWTO8Tml6+Nu0hyEcvHkHxWRDSM+6LmXNnjlVUPyDN2h1/VAYATpOZ2VhmRarDoftCp9Wzi7eQdL/CGiMyuD rzarzyns@redhat.com", "key_options": null, "keyfile": "/home/rzarzynski/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rzarzynski", "validate_certs": true} 2019-12-05T04:57:59.277 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e', u'name': u'rdias'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdGthG4qx90VQ8mJ7TI/IFFzptkSqBHYZsFIVu06IQgHbnFBvjao6heGt4UlkWJkegPCDWpKVAe7wFuXDpPlj/VssRHB5LBnYHKGfkNAzMNiMjpEZsTdy+9w39CCujipz0AlOSfrUdANBZ11AnG9VgQSPEukX5487juob55lFM6J3x+v4opBF3rSlux3+thbr8V0w4oCHMD9/aHvVjFoBnxo4oirR4jc/jgC2fQCMiAJhFKJGhjkKFiNHxcB3npBuRnpy+59BcScOAzSXdJ7DGnwxQLjymnhqsAcWinjjJhOjaq6NaoeKv5I7zQLaVGaQdIyuuAK1Dnb7trpfQU7mR rdias@rdias-suse-laptop", "key_options": null, "keyfile": "/home/rdias/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rdias", "validate_certs": true} 2019-12-05T04:57:59.321 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea', u'name': u'pdonnell'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQD3vdP9QHyNFUNPOdplezEbdYsQfcC91ncATI6Fl/xHifmUbcvjIRqZHkCY6FNWUpMeg9S3+wlHJtzjVqJJ/MlQML3EtwnzQkeWv8bjtuZ07OqULYIv0zEx63XUfSnrdFleJ8+KDSpn7krkYE44q85B88IL+ZBeq5eCXFJ1TtmsDcvsN8WqZNWMdUGqgH3Bb4D8+YKrEfGxoievyxZ1A1DnBHan2l8foTKYggRrnuSbTb59cFpxY3R7Arc5Cn9kDJT1nDVBA5SSGCfRMiNyBOUzQVYe/UjKTaCpngglXC3wYnjoGvGcpksHLQCxyuJPB6djC1tdF+3Jehdf8y8+2TBzWXiiPq2XwS58sER5GcLLLQv26pPL8hgu6CdI0sHxgcgQBgpjoLwUMHmv13CnN/AAcZY4RRDK2X5zd6wFSEhuuCoUqh/m7K219IxI6fVkLaApC851D/ua/pj9OMjiOaJ+D13cPrgfqxva4eK8FQZvqQm7aUK/TT216QNYmn7hkzNjMTkjv/L5yYQ5Ym92AHy44wHzLPrtDB5mk94vHOjG+KlAt9ofuLncMSAXWe+fOqxuIQ6VfbL2LUv0EGYhHe4XOVj98q5utsVQxQbXAZzNQaq9FWXs2A4xcVP7aWj0Fooi17LN/W0ng4Iijir5hyLr7k7M1G6resgiY7p83biNow== pdonnell@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQCpgLVt7Zphmf6nqD+9BC0N91jdmaY6ZpvrYWlzbhxXITQ6juIEtegnlZ+8ohdeEfHFBiyu1xOSQSM1X1ugvpPQE94UqQg7JVb0tuoUCNJHYnIuv8T4YgyP0/vEV48k2G0BBWlP+IJ5f1iDvZuLHWLR5JSgGqh5ELaaHr7Cbyb6JlImldb7SDIoPHMCZ+n6dLO8l5Zy7qWelw6aeA770xmPucBr6t/BVns2g77XgJz6zFd99yEaDy27i8jfzL7WqR9i93yVE9EsL7tuOjg6G9KUzxBF0bg9iVsJXypvcBuG0JwQ+SDKqYw/cN4iLy5qBa6Siqrfpc/LKU21VNmJ0e5eXNgKKbMBpBb8GVmkoLaSRcXw7LclWopjNDA+meUhDAwU7b97N7cqN8BYhBKQyiShXFHMKLuNo2fSuTRt/icfb9VisKNGBdvwgV0oOlrqbXMzZKmj9amb9b+3JDBTGB7ioCsj5x/Evf9srXRrQ9kikIr1aBbVXIWir/lasNWVydcmUYsNi8L7W0vXArLwp+GgOXJFeZUiD3z9LG4CouRxMSwoj3lgWeL2ZR8feH4D/SP/+QmBdU6+aG97au9qreeXi5X8QTR3LND0DwwEKczX+3s+Fa950Oieom/gXC57PVKVh90KVoAGw3HzJTlZrauOtLkYbTUAX6woOMhAtJ5oWuDDT4UrHQj37jRrrl2dETLwKwUJN2+2L07vnnOZ4ABcGWLsU6d3bXeUBlw3pdsYqPebDKUEpjL2kUOw+z 2019-12-05T04:57:59.323 INFO:teuthology.task.ansible.out:mfjEfdVx9BfrvExvDX+3MkxccCJ8FY7od1s5fGEFHsXnIiMnLO/HZb/JN+Aoe+/ExadIcYwJnWcCJyly2/532kGFUlzOliCwPxgJpjWCKQAv6LjWmxoPXiA4JI0bp6y7lMEjmV2Bp3mwphlsW/r8Ie5hg1UPh/ttHRp/VZgcV8e6nNmP/qc6k9b2DWTNFqtWqm1hXLOwq08HzUEeH9mTqvenyAbv2RFMOS18Y2K+4BKUzHwZujrz9qEh/kqtpFScEIXkOJ+HZz3mI6XfjWFIPS487yRGfUvW3+nSWhvHc8vJY21bFRKdjD84qCVlZ2BfOCGUZSKftVItEiqXMTRs3RZd5epfCVZbJuA0hG16jUseeHqqZqDtlbLkzkxtS6v3iGRnPh/Ke/94FHHCzvYY8cmMYAVaM6p/G3edWftbaEtN70Fo7VKxPrpPzuSRZ7Vxmnb+Zad95DxxAC/vtMy/vkBVIP6dyn4HGxqa1oECFMe4oXdhnxZQoPzvWxb8upcXeAeOMMbcIlMq5a+dCBPGFNKhNhrb+9uAEDHwjE7rZlMJVSvt9v1S4hll6jiYIap+7n4reSKrS45aO9U0xiKXJJDohh pdonnell@icewind\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQC8MdUJzIvTCn0fM+hv1HAO/Lwjy/C/V7stQRWGIxyrRHbniyX8iuaU3j7YNFglSQPJq+YRM9UnGauM9/0J9tHKLsMp+zRQYrtxU/ThiTUN8TRJY11mGUol4yLsqkTtm6oq5sl6o+LfuA4hzaYc3XLrFHRJuCUgbtPG71Q0C/y3U+itrDK4wXUxBa4YzZzcIMg+c88VXwoouPbiL0gW5PCXzMaNtKlOqBUKlE935MLn70wIEfN/EwAAbss0ga1moqeqmi9Nlvm20DNIVFmkAGtYoq6CHXtL/uehzai5Sueop8Vmd8+1U8gDTo6JpiagdD8ZNf0zAbRx587KAT5ZMnahJpO276j7RNa85ZFkNkEnBLqBHRkCkTCYq1GehYyRaR69uuKyczFiIoTfAQgPh7VlnpT9525b/4nFsBDOkWxxWAIJR61YiKemPV/n/e9SwhvVsMDb2B/RBriwR7Mi2SZ/lBHGB+z0OA+EE2rGZy4LFVOfUczqIdpltlNotXan4M4F361i+bX19KE+f2RLySn3+GiFv0CdLwslS/y4xysz+FWxw6B0Y/EkpiDU3CdWM61+THGGm/DVhfL/Pvca7k8PonHFYw/x5w1nBcADpkM9JF+m5eOnp+weTEzl58sS+3GnoAuL8YFJwMW1RmE50xM0o5E3SDc8XftUvaSZz8mhII3uwDDukbELfmkLbnIeipQKyjO+GEx0JBp4Itz82qGuao0Wi32U9wHzhSWnX4GQ0pKZnq9frSlyCNwHu/iMiKF8sA2bX3dvplFJ9nKWPefcU8bdl/upr5hYrd2GttjUSkWZa+UuF264yx3ztcN7jotRhEmmdEcHVzKQ26/NyRI4/9DQpDKY7zw4O2wFkXWg5L9kodXwWzgy4cyOr4dMD9LBVLOpHc0CAji5/+lfuYQbu1mwVdfOzCXDqByjt7zCbeN7KuUh403sJ8PTQVp642xA9i4AkPv5g+fqi5C0++1DbZP0lmLfKtTswHhoN+kAqZu2N4Mu6mNVNKyZgW5h1pZzNSGWYcZttyYmMCHhPketReL0LPJ+VUsqQCVohuta3aaSe44InA28GZsXzMj5B0el/84bnE7cN4b+rfGD25NT73JEt3RgPa9W9jqT1hwd/nf53BDKpp7DlHjR77WSWT4h4F7bhbM1RsGSqAfZjCUJT6Rq+IuM2XcrLDY8zIeArkqaZLXMzfGj6KJ7cQes05P5LYfY4osXiq68Gs/7gzF31PQRRGOYAoqMN7752sn0ZMcpnUj3mJutQcncRUN4+naTMjU36WWD7it0YuQBnb2IJ4iKjYLInw8pr2zKgWtgr1TSCM4xKp/v1/ZMmbB 2019-12-05T04:57:59.331 INFO:teuthology.task.ansible.out:mS+dTzNC7Mpkt8vdCNjzZ5KZ5 pdonnell@aglarond", "key_options": null, "keyfile": "/home/pdonnell/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdonnell", "validate_certs": true} 2019-12-05T04:57:59.499 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868', u'name': u'jlayton'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKzhyWb3PmU88m8BbgpEuBzYM6+x28z6aZwHSesiyfXMfF4Y7IZjtKaTxH0ph/O9GQCPhRCVhRd8TlgKWJGhts1E+VkQfHfT18CgmrPWzj3gifc2FJhL97vB1Pd3jksGnOPAikgYPHP46SVMTtyvXCkOUuo7E29WrkdpjawcqM1rfpw63OnzHf7Ar7Sqh/othB7ajyVQ92t+XWSEqj7K+NRfTaq5L5RBWbwarM885o8xdI5mimOZ0QjzKOhFDORrI+HEVlkutyB8E20nZgo7TUy+kYeLVhEqb3A6Pi2QbA9GFLT69kA/H9nebhj7Cuim9yrx4OKeZuIncG0UxSQF3QmWflAZHBs8L/jniTz4DJ5id5jR29tvGK2LeddCEMB1L2cHwWHlG0aX5jaNEJe+nG6H5Tcg3+M1RPM8twBkRQmS3UzAyX/MTH1kktyFUHn8CdBXe0WTuoFSvXLB6BCHWzDYZOt0tIV/xMtBwhy+XGYA4oZSsDU5m7VlSVLBGL8DbRX69Z8AGey2y3FoFu/roY6NmDhTxUedJ2ZfoKSwdZD6jjatdO1621nonBnJxrDgaXCWzT7HtnWcc3GwxZoBrqkvgpsx88f4lNi22OpYVPV6A4hECe04u6OJnHSWVk3ft3p5PaLNF8aEXbiIgzZKAwX3LxpvEDFH/t1sdihhpmjw== jlayton@synchrony.poochiereds.net", "key_options": null, "keyfile": "/home/jlayton/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlayton", "validate_certs": true} 2019-12-05T04:57:59.511 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953', u'name': u'vshankar'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEEjb7yrcY6I6XbMU0hCps1NqBuYmA6HDmMcS/l3i5wJRBitIXYVlv6pefUG9gdBYR5NneccLfVFURu+DC63MJa2ELJvVXsQ6YUOX39bJUT9KlZs8mH+nlTD7tlPmP17Ae3k8sT0H42Q+/5+b5YvdAU+03aPhzntmy38SyvLKrvowRk63wHgcE9p23LY9tY3+HSJu86cb/IdePk1GLjnrMeT9aM+5j2eETrmBjMArP+/2UIsT5eb8MBkQzJ299SCC2gfV/qIVkorFl9xdejctAwNwdT0SPTMxCHxaiK3wZ0e/5HHTbSgR5YlxOlKYq93XBVYIaFKPCePk1At1a3Pjh cephalopod@octopus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDcrTZoKO5e5xs2PB2UuAruvGlJESTHNAtqipbC8yJtRg8WGVDLt//rGu36QNSbpbJYMZgqgKsXBYolJB1ckqLYLv5j8RAoVwbW4QqKaV95y7x0bbXs/YAPnRBhEO1nfpT8GHQusUZvTbz38FDaQwstsYoncmsyZwBrqus9ycFavpY4wfpKoWtzeiK5Wjq7ervThOs98ZJKXJXOOxeI7mDakE3Jv7q2tBNu/rU1kzxIl65kWURzgxox6EmIZXQK3FzKt0jjdKtApVYSlx4ODLPI4iyxOv+MbRe3Nnj6FDhfz8/FeP/U5wWxdYx2WslzZCOI1BJ0VDLu3xEpLlu4vPzp vshankar@h3ckers-pride", "key_options": null, "keyfile": "/home/vshankar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vshankar", "validate_certs": true} 2019-12-05T04:57:59.668 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58', u'name': u'akupczyk'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+CtnvyVRSkIvD15O/q7/vQmD4oE8q7Mo6oT3kqTXaEaQ7iDayC2biUC0PC4OztPlNJxPUcKK5DfWw4F79OKeheqUbZRhsCC3Ge2JGoXHmsf0kBYdioMGiAxm+f8M/v1KESSSxvEnU7o+oi6VtFchh4Dl/WZ54rqZc/oQrlGNTHulFAuyoIAlKkCe3N0GdHN54PB+26QbLXvOuqpWmjVFbcjgRC62mkZl3LuCBhviXS4Rl5/ZE0QIHubCwV5XVrsqvscLS64bi06W8RN2vL56Vn3N9TGV4d2dlbE1PaBBESoTQ7Q0iUN32OAZLo7nF/1fiPY3Xc8r35juya5k+KR9z adam@TP50", "key_options": null, "keyfile": "/home/akupczyk/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akupczyk", "validate_certs": true} 2019-12-05T04:57:59.669 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f', u'name': u'rzarzynski'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCijPApnekNl/dsY+fleKYstEzNkKflBEcCoKtjItUCDJCVLVFOsDBrV94EGvDWpr13aj9sSwpUzRbeu/vOW8ZT853HH6Dqg2+eoYwG6vDRprZb93nw6l4vtU59/M0yKkyi9CPp2+YUo31cCFzYWpq8bv2j1xBruDSUCUdn9nJrHiLFqA4QEwSpw+uVCjSKzQL+LjlKmE8Plfd+8mNCQeY/OFsjxAztwN+4M50SZCnOVCIwEXoYRNd82JOXMOaqZJazSWTO8Tml6+Nu0hyEcvHkHxWRDSM+6LmXNnjlVUPyDN2h1/VAYATpOZ2VhmRarDoftCp9Wzi7eQdL/CGiMyuD rzarzyns@redhat.com", "key_options": null, "keyfile": "/home/rzarzynski/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rzarzynski", "validate_certs": true} 2019-12-05T04:57:59.850 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e', u'name': u'nojha'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDeUJcLeyfqIk3ErPoFqh9kOnvUPsk/2hfaYNC2wZLMOSTIs3zXM8xjki8eN6vle0dmmeeAdwF+e2hJ3tCFKsEUQRvMc7ugIBA7L8/DboShYu7ItyQ3QRvy9pi6jnA/tlHA1LHalvyhMw0yZ4kvPbrfNMhZzOKzvL6jX166rqLA5AuiIfdYMBpE5rYq11GVjm0uQKw4UJpBMooNJeovEfQlNBTSV+IJSTiGBpBD7SgsuW3USMeEF3fEgQOcnvYdv5Fbp/crloyLWYc0HTidkLxpjP7vqY7E4BuBJug0zIB7voR3MYq6rAeuWjByPvq6coDlUjnw/QlrvarCbjjSaHdaRXLj/Dj+QqJFviESBr64ZynhHL21Bl6rexW7AwYyPa73Erbus3dN0aVQQ/ApcDTGHT7KRlDQ2YYg05YM4swpVVnrIdV58FzLWkll+ypCK/+kdtk5eRSe/KzZA3s7z4d4WLgolFWs8Fky8Rx1xDGewvMJgFUWVUbcGC13RocbNSxXG80cbIx8L/sclX4lDfGouu2sUXhP19vq1rJ7gZa6XTPohXjPbFS+En2cdlGdKXhppdvjT4QWKNzgQoF7qgWHMqJnMFJ6beT/2zP34KoM3t0ulee04z88VSn8Th8aqvLlGbdrt5LrE90escSaIAR1vNKZ1hCFwGieSKoRtYJTCw== nojha@redhat.com", "key_options": null, "keyfile": "/home/nojha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nojha", "validate_certs": true} 2019-12-05T04:57:59.851 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e', u'name': u'rdias'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdGthG4qx90VQ8mJ7TI/IFFzptkSqBHYZsFIVu06IQgHbnFBvjao6heGt4UlkWJkegPCDWpKVAe7wFuXDpPlj/VssRHB5LBnYHKGfkNAzMNiMjpEZsTdy+9w39CCujipz0AlOSfrUdANBZ11AnG9VgQSPEukX5487juob55lFM6J3x+v4opBF3rSlux3+thbr8V0w4oCHMD9/aHvVjFoBnxo4oirR4jc/jgC2fQCMiAJhFKJGhjkKFiNHxcB3npBuRnpy+59BcScOAzSXdJ7DGnwxQLjymnhqsAcWinjjJhOjaq6NaoeKv5I7zQLaVGaQdIyuuAK1Dnb7trpfQU7mR rdias@rdias-suse-laptop", "key_options": null, "keyfile": "/home/rdias/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rdias", "validate_certs": true} 2019-12-05T04:57:59.985 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43', u'name': u'ifed01'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+c7zV5QmZb0WsgA2UZogoDNaoHcngRVLA8+BYk/U10ZlvCckJSfIXWMlBnqguZgdm/jBwsw5mGVBgHxUSqFNybwIlU00kyPZU6VbyVP+ZNVUf0XIE1NlemldgPBxxcOzSHkHhWrUqA3IKiCllcQVDQzutfUnq+jDT6NkVn4/cyNzNlargt7FR2L8gvskh62jLTBbhEuHMBp3jLCvaHJOk708gXm9qZ5iLOGV0pu7xdg58KE0p8ILClQBTbvoqPgZE9LA3qaOk1NR1n22kuW8JRHaKfD2Hn0Yu/MfKct783Bxb3mxWBKXxmM4otdmsR/R14NH/hIPce9wSkOOnQAC9 ifed@37526", "key_options": null, "keyfile": "/home/ifed01/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ifed01", "validate_certs": true} 2019-12-05T04:57:59.997 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953', u'name': u'vshankar'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEEjb7yrcY6I6XbMU0hCps1NqBuYmA6HDmMcS/l3i5wJRBitIXYVlv6pefUG9gdBYR5NneccLfVFURu+DC63MJa2ELJvVXsQ6YUOX39bJUT9KlZs8mH+nlTD7tlPmP17Ae3k8sT0H42Q+/5+b5YvdAU+03aPhzntmy38SyvLKrvowRk63wHgcE9p23LY9tY3+HSJu86cb/IdePk1GLjnrMeT9aM+5j2eETrmBjMArP+/2UIsT5eb8MBkQzJ299SCC2gfV/qIVkorFl9xdejctAwNwdT0SPTMxCHxaiK3wZ0e/5HHTbSgR5YlxOlKYq93XBVYIaFKPCePk1At1a3Pjh cephalopod@octopus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDcrTZoKO5e5xs2PB2UuAruvGlJESTHNAtqipbC8yJtRg8WGVDLt//rGu36QNSbpbJYMZgqgKsXBYolJB1ckqLYLv5j8RAoVwbW4QqKaV95y7x0bbXs/YAPnRBhEO1nfpT8GHQusUZvTbz38FDaQwstsYoncmsyZwBrqus9ycFavpY4wfpKoWtzeiK5Wjq7ervThOs98ZJKXJXOOxeI7mDakE3Jv7q2tBNu/rU1kzxIl65kWURzgxox6EmIZXQK3FzKt0jjdKtApVYSlx4ODLPI4iyxOv+MbRe3Nnj6FDhfz8/FeP/U5wWxdYx2WslzZCOI1BJ0VDLu3xEpLlu4vPzp vshankar@h3ckers-pride", "key_options": null, "keyfile": "/home/vshankar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vshankar", "validate_certs": true} 2019-12-05T04:58:00.141 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4', u'name': u'myoungwon'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEudQMGBPqfZBV8bXpdQOgVTDP6GFsr5oL4gdbHFMuJCGe0Sfq8LTaSEBQx0MH7tXc7JJ1fU51U7DSt0kfy/PFeV1Y5XnNATiZu/SzdO3jHe06/8CCr3DLnlEmzXASq+tqDJbY1eV011d/TX1jiC8Qd1VNbpdZYExEDsDzwJXnWdh6rjTL4anGjIgQWcmU0XdbUJXvRP4dGjVvZVHZnflAkxLPBbAx/8BBOuBmv6vP4hNs4poi4VLwR6S6Ha1fgMJmsn6StsCVuOvKBJFuGvaulHgHgdoDJ1ryUFikkXXCWoNx4IbwtpQIkSxWMuAovlwwk9G8ozSs+RrFzCavE0f omw@omw-dev", "key_options": null, "keyfile": "/home/myoungwon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "myoungwon", "validate_certs": true} 2019-12-05T04:58:00.155 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58', u'name': u'akupczyk'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+CtnvyVRSkIvD15O/q7/vQmD4oE8q7Mo6oT3kqTXaEaQ7iDayC2biUC0PC4OztPlNJxPUcKK5DfWw4F79OKeheqUbZRhsCC3Ge2JGoXHmsf0kBYdioMGiAxm+f8M/v1KESSSxvEnU7o+oi6VtFchh4Dl/WZ54rqZc/oQrlGNTHulFAuyoIAlKkCe3N0GdHN54PB+26QbLXvOuqpWmjVFbcjgRC62mkZl3LuCBhviXS4Rl5/ZE0QIHubCwV5XVrsqvscLS64bi06W8RN2vL56Vn3N9TGV4d2dlbE1PaBBESoTQ7Q0iUN32OAZLo7nF/1fiPY3Xc8r35juya5k+KR9z adam@TP50", "key_options": null, "keyfile": "/home/akupczyk/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akupczyk", "validate_certs": true} 2019-12-05T04:58:00.286 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2', u'name': u'gabrioux'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQDZPRW12u7LvaiynB87ad5h/JFtYIVjkKIOym9Y5CAbvatHhZ9fB1WAnYIn2HqiSSPUXT8sFy8w0TmM+aH0/ZGcfXoe0CdTWLtUbXKNlPI5h7TO9rHXESP2iRBpnKs3vK8WnMY0/D5BeESrPfwVScucmzaDrjUqjdZKR/aEeqVvDpQsdaCaoqNg23t4nhQ54BEvK8O+14nQ26Rtmm40QHO1ze+0UK+7nUMMFwxvd1EKjVp21EgYAFEbyQ5Kbbl+fmS/11AGy/yTai0c4qd7ohBzZvx8+sY+c1novG9Q7LhtyxuuYwzYIrGBCkKH5+iC7nt2rGBP+mnTfIgdbJUiXX3cdrMESwOSZy6x6zgbXXME+yfM0myhomKmUkpeZAQInX2KYrnEuPD9kteyK9tZcG6gJSaiUdvlab49Alx5lM2hc62M4s4WKaIgroCoApd8FtNAlTQsdYuycntgNrbPy0LXQ5hw8K8hkvc6RbxISdj/RXWguQSVbJWdS9Pf5NwD3Tq7dsOghJWhx4vBkhW3IECVrPQWVyAmgQ9oj1PoL7/2LZ6h9P8F0KC1qmhm+pWdEPLugOlU6lzQX7RJcukYlsINThM1RGwbPE4h9XcMyHAnODiDF63TDfQ01oKbVXj9XpOGY5eP0EQxKJYjCfWW0Ot6nK2ZCt8OYvoRkWMGnLv11Nub3NNOgh5TB/4nDvEU1cIb/Ikb4BRzVIEbast/pqUJmXe+GQ+ho7mp8y8a+lIxu54BR+VN6isjwODCVt1DYohgixEoul7p5pXvTM+JUY2GD/ddTGwPvVAN66sWnT9bjZZunQwTHqasFCIHNyPhnEnYIIqCo9O1I/D8zRQG3SYNxT6ED9GPARrE/H0TvdOmKaT3e0F2GRJRSToNxol9dpJovUziojOhXyBUjlTqnMY2RVilX90MnzgqR3zGO0aSruTGyNT2Faymqxn6T/50mSdJ8EyWH23BaaXUUW/z3Cjov+yCQjLY48SOWwlxKXw3hGz8xTAxauBYI01ojsVLCpCUGFyKlWS5iFwiLFtr1KPYqeCoS6CoWjiOiFgGe+w3lPPfp2h4rc6auIx2DZ1ByYhWLeHnpkV7sG+q5ckBIRAYmiKyXSGdV4zbXITMWDm+3zu4oj5VPRCfjMlout8OZ4TQm5PUdKZZhSE1jULG9tPDFL7nWwM3z0HET+d9kQYcEO6QB+ma9gZHp1qZJbXvCIC5fQ9y9El++A6N3dBr+ED10Ye88PLoWPeumyAmc24BDdI0skGk89Q0IzoLqagK6HGhPAso8si9wr84CEi4ncpizVLDU2cFKCpef4Sa03HoMbwfTgERtyptKFRbMT+SpuZsnrhF635h9uqFnszowUXV guits@Elisheba", "key_options": null, "keyfile": "/home/gabrioux/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user 2019-12-05T04:58:00.287 INFO:teuthology.task.ansible.out:": "gabrioux", "validate_certs": true} 2019-12-05T04:58:00.342 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e', u'name': u'nojha'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDeUJcLeyfqIk3ErPoFqh9kOnvUPsk/2hfaYNC2wZLMOSTIs3zXM8xjki8eN6vle0dmmeeAdwF+e2hJ3tCFKsEUQRvMc7ugIBA7L8/DboShYu7ItyQ3QRvy9pi6jnA/tlHA1LHalvyhMw0yZ4kvPbrfNMhZzOKzvL6jX166rqLA5AuiIfdYMBpE5rYq11GVjm0uQKw4UJpBMooNJeovEfQlNBTSV+IJSTiGBpBD7SgsuW3USMeEF3fEgQOcnvYdv5Fbp/crloyLWYc0HTidkLxpjP7vqY7E4BuBJug0zIB7voR3MYq6rAeuWjByPvq6coDlUjnw/QlrvarCbjjSaHdaRXLj/Dj+QqJFviESBr64ZynhHL21Bl6rexW7AwYyPa73Erbus3dN0aVQQ/ApcDTGHT7KRlDQ2YYg05YM4swpVVnrIdV58FzLWkll+ypCK/+kdtk5eRSe/KzZA3s7z4d4WLgolFWs8Fky8Rx1xDGewvMJgFUWVUbcGC13RocbNSxXG80cbIx8L/sclX4lDfGouu2sUXhP19vq1rJ7gZa6XTPohXjPbFS+En2cdlGdKXhppdvjT4QWKNzgQoF7qgWHMqJnMFJ6beT/2zP34KoM3t0ulee04z88VSn8Th8aqvLlGbdrt5LrE90escSaIAR1vNKZ1hCFwGieSKoRtYJTCw== nojha@redhat.com", "key_options": null, "keyfile": "/home/nojha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nojha", "validate_certs": true} 2019-12-05T04:58:00.424 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f', u'name': u'leseb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpDFqu0Zg94mCjrA0FTB0vMiSBml1pqYCDKHhKBrfaza8u3bhtsmls9i+6U/vIwCEVk4GGPNhWduTJK0h9R5MS2eCf3GJcE0XHqRp/8msUCDPOVSZwC7hoYHZKunUIUaIUajOnR+L2slA/a9JaqataFH7zdrKbPdSKBzf5CXYrc0fK+Fz5OOsaI9KITbM0nESVpNCDzRqnA/aarF9MlN79vpTtvwQ+i0z2Km3cxqwV70GPMuFVo4CGf8pCGpbMrIg9OtoSC1svQ2vZzLkH2WrbDqTfYlOEYJJXaiBKcGhGnLZXfLIGSgvHjpv6xHo63yaMihocyg2MwTS8TZ/VtAaH seb@redhat.com", "key_options": null, "keyfile": "/home/leseb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "leseb", "validate_certs": true} 2019-12-05T04:58:00.497 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43', u'name': u'ifed01'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+c7zV5QmZb0WsgA2UZogoDNaoHcngRVLA8+BYk/U10ZlvCckJSfIXWMlBnqguZgdm/jBwsw5mGVBgHxUSqFNybwIlU00kyPZU6VbyVP+ZNVUf0XIE1NlemldgPBxxcOzSHkHhWrUqA3IKiCllcQVDQzutfUnq+jDT6NkVn4/cyNzNlargt7FR2L8gvskh62jLTBbhEuHMBp3jLCvaHJOk708gXm9qZ5iLOGV0pu7xdg58KE0p8ILClQBTbvoqPgZE9LA3qaOk1NR1n22kuW8JRHaKfD2Hn0Yu/MfKct783Bxb3mxWBKXxmM4otdmsR/R14NH/hIPce9wSkOOnQAC9 ifed@37526", "key_options": null, "keyfile": "/home/ifed01/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ifed01", "validate_certs": true} 2019-12-05T04:58:00.564 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca', u'name': u'hchen'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzjJ6nlPn/OXP3knQ6orVId2QLswql9gRetKYlicKDBrdFg0MES03EIuAonu0aUR4NEReR7P1iB8eNnezrHyAu5mxIJlZp0rz7ljcBcTLL+11IJU9I9OjBFwzXGqJ1KK4B4Qfw35oc2O6F5V0JP2nHhHgyny37udJLImhxHy3RNmDwc86JhEBCVR3ouYiZuxgxxbA9BXc6+Lkdu9hEMMCDlaEXwNmMyASVU9Wf3dKnMTSCdar2PANquWbpNQZ5cgoTm39xgust5CEyYJ53aOdrti0DYK7qyzxUXre6ckZGrH4oGv2XrDaxpACIiHXt+slvow51gsZnBRHeayMAxQzGQ== hchen@hchen.csb", "key_options": null, "keyfile": "/home/hchen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hchen", "validate_certs": true} 2019-12-05T04:58:00.635 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4', u'name': u'myoungwon'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEudQMGBPqfZBV8bXpdQOgVTDP6GFsr5oL4gdbHFMuJCGe0Sfq8LTaSEBQx0MH7tXc7JJ1fU51U7DSt0kfy/PFeV1Y5XnNATiZu/SzdO3jHe06/8CCr3DLnlEmzXASq+tqDJbY1eV011d/TX1jiC8Qd1VNbpdZYExEDsDzwJXnWdh6rjTL4anGjIgQWcmU0XdbUJXvRP4dGjVvZVHZnflAkxLPBbAx/8BBOuBmv6vP4hNs4poi4VLwR6S6Ha1fgMJmsn6StsCVuOvKBJFuGvaulHgHgdoDJ1ryUFikkXXCWoNx4IbwtpQIkSxWMuAovlwwk9G8ozSs+RrFzCavE0f omw@omw-dev", "key_options": null, "keyfile": "/home/myoungwon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "myoungwon", "validate_certs": true} 2019-12-05T04:58:00.690 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n', u'name': u'jcollin'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCqZKc+/7iPLhapA/7SYmLxwyKEM95D6eFfsDBNKbEUey/B1OlqYSo+yKB/LdFDyUcd7shMDnhFFg47lNMANrsZdtE4iMPVAa1YhcrTgNbV3HSp9cthKixvU/gkt9Fnb6PEckRzr19wEwxtoNg3Fh/R6YCiSjPwqHmArtC1G2hzk560EJpozx6xylMR5IafatNDoZVw2YQ35fZvHbhHD6meQELducHJ1bKEEHxVIRSTcmXDpBz/vAvBoQOEPuH6PmC/oSwL9HVVIsazf65F4QfK98d+8NCK9TemgXyGDk3Sjz6xdm76yBhFzDQEAzO12U1YtS7Q7zGVsjmSDav2dd7 jcollin@earth\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC89dHSsgjgtIjJ3e/pcMthliQlX2vtdXL7pk62nD64230Fm1PfOVCnzn4LqDhBbGgJT+BrlwOPfzStpLgnUT3l67S834SJzhvSTTpQp2pznnQg1rZOiXjvNhbQwVYV4rphXWqay8weWvFIb5QfXE66mYA1etZtcOb1SMY2ofAjUiA8bAQS5aHpzcMVs4N1A9x5GJx/7/Bj1h8zPpSXikDYasJc3++UMGc45C/AAf725YwijK7xMnZNXv8uTBoGzO/Ko24R3VdvtfMVn4Hi4/FwuRd5bRYcJXR/nUdtA3IinqvlnTpFV2LbfgVTHiUMGhvewRTB+8S8ij4nnAlko3H3bhqZZTk/y7d4K/IerY0mozVIc2x8rMLfqhF9OKv1QFB9fskRZcF/ENHITnmLqCJPg9rgmyv1gCQ5Tbds1OI+IYQIowdiwtaViiTSIy7/CedbHqMz6aDnPsJFMs24lcUXe2d8uZ26RqMinMFGXg0zF7tifuVmxj7zG8DT7MFORDm5f/Hhkng05R5XZesStTBxpEU0MGuzWhENt3RuJdstqmY+wkDVW3XwjUvKG/gVYuJYfnBAsRZIZezw2wfvodhc8EmeHELvY1YtB3DGTBXKE1lsKGN7yV5+nl4UtR2IGzBn9GovWVi5ooB8UlSAeUmZix/eAA5teL8ldFrDht7+AQ== jcollin@teuthology\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeApVHu5er+R9h52RLSVYVhXFQSn/nMN38uWoDYPw2bqYBLNEAAMcPl+rnqHubELpXTGXjJYL4n2mjic+2rOLZpDgPLmOOm4PwgYstJjE1KLegKqVzXIa9HfASKMXaHSgwtC6P5vDQkm2MNl8vXuTAITF7VYdDS 2019-12-05T04:58:00.692 INFO:teuthology.task.ansible.out:33Kkke6ViJRZI7szYjbP7z9A1fy26THrvnuBT1Q18yy9vgk/sKo8vjLQfHSLvO90CJzouNEKqmkVpmJEqCPbE+s6Qr4+d2KY7L6Q6GAXxrqXK1jSDrrPsJguE6zZ1oWy/7XFMOr9ssduFWocnjQjPtQj3M+V+7FYAsf5uqQnzeMQTjO6AXYxTI7 jcollin@stratocaster\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDk/DYRp5E8Dla/4XHCF504X20W0u7CijZSCFWT1/5082NgnYeR90Prth3p+4xsw3YC9H5pEAS3XwcbdUb/NtWvZVIAIcMd7NOOEClSIj2Agk7NJxZ/4SfQRLrK3H2Qli/MVnrTxnEa9oeS4zqzv9dKRwFJDUFCoH2CpoumFVAp02uyzk7D/+XqFrXzMJUiSHXfl3YM2rt/vNmCtHtVp04UcG0Kyl2TXY37frRlSOkMMu0zbIC551gOyae6NGYhkYHa7u2SnngXcKM78b8MDvMv3QtuUN5P4WpH5qFRdlKNKsQGrzcQpBDY8AKQ+rX0K1ENhBwer49bihYNMKvjYmvTdLTNWgPCu7dt8JSgJQ8JMMhve0pjvFdpcKDLp0fvXDvJvZdLJM1ID/66kywJspHjShuqQNXRMfd1R9C4F20c+R9LY6eTGwrpLG1DNEez1VBHmOu5hQl03T2PfaThfLfMHRTw8JpGXGLMQarCCTm1PC6Z2Iw7KzorxP1eofGA5q8XoCbwGD01uQRZrwxPq0SHzTAkW2C2M8oEz+k4Mrx7+vGYzSS14hqxrzk91wuYhZCAKuGpqwm1ouJ4wxvWFewjbxLgucMfGSp09zJ2XSYRmtm7SAUifFob6Z9czHGTQerHCOTPuYqzlap2gCCVYtEvQyNB+4IqK/9FtEXTr28x0Q== jcollin@stratocaster", "key_options": null, "keyfile": "/home/jcollin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcollin", "validate_certs": true} 2019-12-05T04:58:00.807 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2', u'name': u'gabrioux'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQDZPRW12u7LvaiynB87ad5h/JFtYIVjkKIOym9Y5CAbvatHhZ9fB1WAnYIn2HqiSSPUXT8sFy8w0TmM+aH0/ZGcfXoe0CdTWLtUbXKNlPI5h7TO9rHXESP2iRBpnKs3vK8WnMY0/D5BeESrPfwVScucmzaDrjUqjdZKR/aEeqVvDpQsdaCaoqNg23t4nhQ54BEvK8O+14nQ26Rtmm40QHO1ze+0UK+7nUMMFwxvd1EKjVp21EgYAFEbyQ5Kbbl+fmS/11AGy/yTai0c4qd7ohBzZvx8+sY+c1novG9Q7LhtyxuuYwzYIrGBCkKH5+iC7nt2rGBP+mnTfIgdbJUiXX3cdrMESwOSZy6x6zgbXXME+yfM0myhomKmUkpeZAQInX2KYrnEuPD9kteyK9tZcG6gJSaiUdvlab49Alx5lM2hc62M4s4WKaIgroCoApd8FtNAlTQsdYuycntgNrbPy0LXQ5hw8K8hkvc6RbxISdj/RXWguQSVbJWdS9Pf5NwD3Tq7dsOghJWhx4vBkhW3IECVrPQWVyAmgQ9oj1PoL7/2LZ6h9P8F0KC1qmhm+pWdEPLugOlU6lzQX7RJcukYlsINThM1RGwbPE4h9XcMyHAnODiDF63TDfQ01oKbVXj9XpOGY5eP0EQxKJYjCfWW0Ot6nK2ZCt8OYvoRkWMGnLv11Nub3NNOgh5TB/4nDvEU1cIb/Ikb4BRzVIEbast/pqUJmXe+GQ+ho7mp8y8a+lIxu54BR+VN6isjwODCVt1DYohgixEoul7p5pXvTM+JUY2GD/ddTGwPvVAN66sWnT9bjZZunQwTHqasFCIHNyPhnEnYIIqCo9O1I/D8zRQG3SYNxT6ED9GPARrE/H0TvdOmKaT3e0F2GRJRSToNxol9dpJovUziojOhXyBUjlTqnMY2RVilX90MnzgqR3zGO0aSruTGyNT2Faymqxn6T/50mSdJ8EyWH23BaaXUUW/z3Cjov+yCQjLY48SOWwlxKXw3hGz8xTAxauBYI01ojsVLCpCUGFyKlWS5iFwiLFtr1KPYqeCoS6CoWjiOiFgGe+w3lPPfp2h4rc6auIx2DZ1ByYhWLeHnpkV7sG+q5ckBIRAYmiKyXSGdV4zbXITMWDm+3zu4oj5VPRCfjMlout8OZ4TQm5PUdKZZhSE1jULG9tPDFL7nWwM3z0HET+d9kQYcEO6QB+ma9gZHp1qZJbXvCIC5fQ9y9El++A6N3dBr+ED10Ye88PLoWPeumyAmc24BDdI0skGk89Q0IzoLqagK6HGhPAso8si9wr84CEi4ncpizVLDU2cFKCpef4Sa03HoMbwfTgERtyptKFRbMT+SpuZsnrhF635h9uqFnszowUXV guits@Elisheba", "key_options": null, "keyfile": "/home/gabrioux/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user 2019-12-05T04:58:00.809 INFO:teuthology.task.ansible.out:": "gabrioux", "validate_certs": true} 2019-12-05T04:58:00.835 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc', u'name': u'xxg'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdxi9xdXNBYctMHH1cuuyNcPC4I4xVAeD3HZsKH+QwIILiKOTHu/6HZg7Yo1WsuwYSA0MDc+5LCSGu7RsmOEVpcSQ4gtNoFVwsThDyI6yhsm9WQqqXwSPs7ZE+urqd/BDfQ/7oMqJm5bViQMP0vmBExHuIFagj+8f6nspkE2Z7lBbaw8bEbAJZFx74oReOAELsL+Pbtf9Iaqrap4Z3WysmhfP6Rm/q7mncBBmo/fl/bejhThr8fghTPJb81M+XLhMk/rAdVrGdKV9iSt6lLiWdvf0lpTZO4LdIfJSBXr1Xv+lHRtaGXID2ahH7zl/W6dM4baz+hU7ZrbP+vdKg/SVH xxg@xxg", "key_options": null, "keyfile": "/home/xxg/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xxg", "validate_certs": true} 2019-12-05T04:58:00.952 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f', u'name': u'leseb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpDFqu0Zg94mCjrA0FTB0vMiSBml1pqYCDKHhKBrfaza8u3bhtsmls9i+6U/vIwCEVk4GGPNhWduTJK0h9R5MS2eCf3GJcE0XHqRp/8msUCDPOVSZwC7hoYHZKunUIUaIUajOnR+L2slA/a9JaqataFH7zdrKbPdSKBzf5CXYrc0fK+Fz5OOsaI9KITbM0nESVpNCDzRqnA/aarF9MlN79vpTtvwQ+i0z2Km3cxqwV70GPMuFVo4CGf8pCGpbMrIg9OtoSC1svQ2vZzLkH2WrbDqTfYlOEYJJXaiBKcGhGnLZXfLIGSgvHjpv6xHo63yaMihocyg2MwTS8TZ/VtAaH seb@redhat.com", "key_options": null, "keyfile": "/home/leseb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "leseb", "validate_certs": true} 2019-12-05T04:58:00.993 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53', u'name': u'pcuzner'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsA63hT3ZAwQ9hsvfY+o2Gy15M1GRnRw1ZBSXEiCs9OJku7T9+LCvvH85Y/7V+mOGcsCZKqVTd5Vy3sHAvAF3M7CYJ+5IIDJKCYslC0wkKcX4jj26aMToAZPms8uvs527BNgKOcvwgIfPjs4WJG06fey5AhsuuL30etbAvbBudxRAKdZSTOsMveneKtXQu3Dx7y0rjfI/RBAhCAvisnMqYFUdajoJIl5gVXl6+8mauUOP5O0mvyvJH2jQveqZXTdsIPyp5mq1nA5psqvivWGz2bEJ+owI0GjfuYbyAi6i+BL4Bsd3GeoAs1G3Dicfr228VIUsT+R3xLBQUj3xIsVAZ pcuzner@redhat.com", "key_options": null, "keyfile": "/home/pcuzner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pcuzner", "validate_certs": true} 2019-12-05T04:58:01.114 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca', u'name': u'hchen'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzjJ6nlPn/OXP3knQ6orVId2QLswql9gRetKYlicKDBrdFg0MES03EIuAonu0aUR4NEReR7P1iB8eNnezrHyAu5mxIJlZp0rz7ljcBcTLL+11IJU9I9OjBFwzXGqJ1KK4B4Qfw35oc2O6F5V0JP2nHhHgyny37udJLImhxHy3RNmDwc86JhEBCVR3ouYiZuxgxxbA9BXc6+Lkdu9hEMMCDlaEXwNmMyASVU9Wf3dKnMTSCdar2PANquWbpNQZ5cgoTm39xgust5CEyYJ53aOdrti0DYK7qyzxUXre6ckZGrH4oGv2XrDaxpACIiHXt+slvow51gsZnBRHeayMAxQzGQ== hchen@hchen.csb", "key_options": null, "keyfile": "/home/hchen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hchen", "validate_certs": true} 2019-12-05T04:58:01.124 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2', u'name': u'mkogan'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaRX+aBLjb/h12ctmvoMvZro6NrrYDuxROF6QMVpEVucA7jHhwCC/j9dJAq0hOVBO8R3S0bvKQ6bmnCb/4y2J6tq18xKUFCrChNii5OQ+AgM5v1353oZ9SrHN8p9ajvS8w313o2OpXhpaPv3khVDjKZsDmeH8YL/Nx3WQKqIJC1mI4jrI0sZGGJLifEy3pPkO0W1KmguN9AwAWK6+kckAjMa3ElkfK/xuAqBYgNKQUxP9oMgTQRcgpWF8Nzb0npN3v+HOUIRrth1INSchkDKtWob2T+0Zqxwm+Vpi0iFWwqjcN/OVdaDGf34//A3M/tsiQ1g9SqhdhZJCfu5XLmM63 mkogan@mkP50.redhat.com", "key_options": null, "keyfile": "/home/mkogan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkogan", "validate_certs": true} 2019-12-05T04:58:01.275 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n', u'name': u'jcollin'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCqZKc+/7iPLhapA/7SYmLxwyKEM95D6eFfsDBNKbEUey/B1OlqYSo+yKB/LdFDyUcd7shMDnhFFg47lNMANrsZdtE4iMPVAa1YhcrTgNbV3HSp9cthKixvU/gkt9Fnb6PEckRzr19wEwxtoNg3Fh/R6YCiSjPwqHmArtC1G2hzk560EJpozx6xylMR5IafatNDoZVw2YQ35fZvHbhHD6meQELducHJ1bKEEHxVIRSTcmXDpBz/vAvBoQOEPuH6PmC/oSwL9HVVIsazf65F4QfK98d+8NCK9TemgXyGDk3Sjz6xdm76yBhFzDQEAzO12U1YtS7Q7zGVsjmSDav2dd7 jcollin@earth\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC89dHSsgjgtIjJ3e/pcMthliQlX2vtdXL7pk62nD64230Fm1PfOVCnzn4LqDhBbGgJT+BrlwOPfzStpLgnUT3l67S834SJzhvSTTpQp2pznnQg1rZOiXjvNhbQwVYV4rphXWqay8weWvFIb5QfXE66mYA1etZtcOb1SMY2ofAjUiA8bAQS5aHpzcMVs4N1A9x5GJx/7/Bj1h8zPpSXikDYasJc3++UMGc45C/AAf725YwijK7xMnZNXv8uTBoGzO/Ko24R3VdvtfMVn4Hi4/FwuRd5bRYcJXR/nUdtA3IinqvlnTpFV2LbfgVTHiUMGhvewRTB+8S8ij4nnAlko3H3bhqZZTk/y7d4K/IerY0mozVIc2x8rMLfqhF9OKv1QFB9fskRZcF/ENHITnmLqCJPg9rgmyv1gCQ5Tbds1OI+IYQIowdiwtaViiTSIy7/CedbHqMz6aDnPsJFMs24lcUXe2d8uZ26RqMinMFGXg0zF7tifuVmxj7zG8DT7MFORDm5f/Hhkng05R5XZesStTBxpEU0MGuzWhENt3RuJdstqmY+wkDVW3XwjUvKG/gVYuJYfnBAsRZIZezw2wfvodhc8EmeHELvY1YtB3DGTBXKE1lsKGN7yV5+nl4UtR2IGzBn9GovWVi5ooB8UlSAeUmZix/eAA5teL8ldFrDht7+AQ== jcollin@teuthology\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeApVHu5er+R9h52RLSVYVhXFQSn/nMN38uWoDYPw2bqYBLNEAAMcPl+rnqHubELpXTGXjJYL4n2mjic+2rOLZpDgPLmOOm4PwgYstJjE1KLegKqVzXIa9HfASKMXaHSgwtC6P5vDQkm2MNl8vXuTAITF7VYdDS 2019-12-05T04:58:01.278 INFO:teuthology.task.ansible.out:33Kkke6ViJRZI7szYjbP7z9A1fy26THrvnuBT1Q18yy9vgk/sKo8vjLQfHSLvO90CJzouNEKqmkVpmJEqCPbE+s6Qr4+d2KY7L6Q6GAXxrqXK1jSDrrPsJguE6zZ1oWy/7XFMOr9ssduFWocnjQjPtQj3M+V+7FYAsf5uqQnzeMQTjO6AXYxTI7 jcollin@stratocaster\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDk/DYRp5E8Dla/4XHCF504X20W0u7CijZSCFWT1/5082NgnYeR90Prth3p+4xsw3YC9H5pEAS3XwcbdUb/NtWvZVIAIcMd7NOOEClSIj2Agk7NJxZ/4SfQRLrK3H2Qli/MVnrTxnEa9oeS4zqzv9dKRwFJDUFCoH2CpoumFVAp02uyzk7D/+XqFrXzMJUiSHXfl3YM2rt/vNmCtHtVp04UcG0Kyl2TXY37frRlSOkMMu0zbIC551gOyae6NGYhkYHa7u2SnngXcKM78b8MDvMv3QtuUN5P4WpH5qFRdlKNKsQGrzcQpBDY8AKQ+rX0K1ENhBwer49bihYNMKvjYmvTdLTNWgPCu7dt8JSgJQ8JMMhve0pjvFdpcKDLp0fvXDvJvZdLJM1ID/66kywJspHjShuqQNXRMfd1R9C4F20c+R9LY6eTGwrpLG1DNEez1VBHmOu5hQl03T2PfaThfLfMHRTw8JpGXGLMQarCCTm1PC6Z2Iw7KzorxP1eofGA5q8XoCbwGD01uQRZrwxPq0SHzTAkW2C2M8oEz+k4Mrx7+vGYzSS14hqxrzk91wuYhZCAKuGpqwm1ouJ4wxvWFewjbxLgucMfGSp09zJ2XSYRmtm7SAUifFob6Z9czHGTQerHCOTPuYqzlap2gCCVYtEvQyNB+4IqK/9FtEXTr28x0Q== jcollin@stratocaster", "key_options": null, "keyfile": "/home/jcollin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcollin", "validate_certs": true} 2019-12-05T04:58:01.299 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2019-12-05T04:58:01.432 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc', u'name': u'xxg'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdxi9xdXNBYctMHH1cuuyNcPC4I4xVAeD3HZsKH+QwIILiKOTHu/6HZg7Yo1WsuwYSA0MDc+5LCSGu7RsmOEVpcSQ4gtNoFVwsThDyI6yhsm9WQqqXwSPs7ZE+urqd/BDfQ/7oMqJm5bViQMP0vmBExHuIFagj+8f6nspkE2Z7lBbaw8bEbAJZFx74oReOAELsL+Pbtf9Iaqrap4Z3WysmhfP6Rm/q7mncBBmo/fl/bejhThr8fghTPJb81M+XLhMk/rAdVrGdKV9iSt6lLiWdvf0lpTZO4LdIfJSBXr1Xv+lHRtaGXID2ahH7zl/W6dM4baz+hU7ZrbP+vdKg/SVH xxg@xxg", "key_options": null, "keyfile": "/home/xxg/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xxg", "validate_certs": true} 2019-12-05T04:58:01.470 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736', u'name': u'henrix'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDZDKRAePFqjugc2ljw4QmM6r7B2dPI39OBB+2HKONtDZI4q+uwgZ1yqnhrbJaU9ux0W5wNzJyp2VfOPhw/9G9jo+efJPg5IySnTonhi1zeCXNpBMY70dJZVCyGEDdWKehvFM9xFBOxBC/h7gOyKqEc28BKoy4cHfULzm4er8PU30MyFBngu8NaoKqyQ3GYAfWOAXF5lUdalbJwl3a+HBb8PjUhbGuznAZF1Ji4JgnNuEZdlNVOKwhEkAFZ1JwXvgLfrEJVQs5b2V9Sp2Vm5ZngsLg7b3RzV5I4B3Lj7QTTraV1zLJGiS5iRdjY7D4W5Uxi2k1X7vaOjySJWWadHyeicNNBhLSfHl6JJ3aMuA8oAAmtgo/8IkJ96GFtgM9j31Wg8wkeTyQrwSeqCuv1uDFr8rp+OvMClhHnDgUklJLWjjYPCDVYR5KV+iOiHL58WJg/b4h6NsoP91wO8P8EcSaiMdq7P77w5nJxDI0KKb//dC8aHtWnRRRlrIS+PNPqKJSNTaSwOTieXvzplhzKFzu2dr79qs+/67iaNE6HBvIX5jVHWCFV3wllyjsC8WYPh4kEVtg2SY00SDOOrMrqANV8DDtHFwwxQTEjz1z5ESuwy9UcUSQfP1/zsAp2OQzPutKs6UipHyBK3pLm6MjaeeaEWu8fmetCqALxYV3h17O+Yw== miguel@hermes.olymp", "key_options": null, "keyfile": "/home/henrix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "henrix", "validate_certs": true} 2019-12-05T04:58:01.581 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53', u'name': u'pcuzner'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsA63hT3ZAwQ9hsvfY+o2Gy15M1GRnRw1ZBSXEiCs9OJku7T9+LCvvH85Y/7V+mOGcsCZKqVTd5Vy3sHAvAF3M7CYJ+5IIDJKCYslC0wkKcX4jj26aMToAZPms8uvs527BNgKOcvwgIfPjs4WJG06fey5AhsuuL30etbAvbBudxRAKdZSTOsMveneKtXQu3Dx7y0rjfI/RBAhCAvisnMqYFUdajoJIl5gVXl6+8mauUOP5O0mvyvJH2jQveqZXTdsIPyp5mq1nA5psqvivWGz2bEJ+owI0GjfuYbyAi6i+BL4Bsd3GeoAs1G3Dicfr228VIUsT+R3xLBQUj3xIsVAZ pcuzner@redhat.com", "key_options": null, "keyfile": "/home/pcuzner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pcuzner", "validate_certs": true} 2019-12-05T04:58:01.618 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26', u'name': u'pbs1108'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNwmgnIsDDJmsmYzkTAEo3iLj7Jd5Ni2R5vD1Z0NRZ39zJMtbicZ0ec3nAMzFyHrmPnKNFltS1mhehbLbtKNKq1fSo2SMHs11z+N8Nl94F4AY8IhXZ8U9GAorEJFVIxHUNEDqwl55Ov6MfP60UI7rNgy2rUt8aOBmAJVhLEFmJJawJgyPhTrUZAT20OoILphrt81o5bTNfmx++Zs5Dc9VtgIh53hexwKgTvG6fxfKdra9m+J2GXpniOsZjPJq/A0je+t7H3UlxCSu0dRy2Un6l5ZAcoeg74SP50dFgGaxkTQqI6wle6NjtPIdssgrdYTgTcOg9M4jWqAr/5drTUSGh root@centos7-bspark", "key_options": null, "keyfile": "/home/pbs1108/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pbs1108", "validate_certs": true} 2019-12-05T04:58:01.729 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2', u'name': u'mkogan'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaRX+aBLjb/h12ctmvoMvZro6NrrYDuxROF6QMVpEVucA7jHhwCC/j9dJAq0hOVBO8R3S0bvKQ6bmnCb/4y2J6tq18xKUFCrChNii5OQ+AgM5v1353oZ9SrHN8p9ajvS8w313o2OpXhpaPv3khVDjKZsDmeH8YL/Nx3WQKqIJC1mI4jrI0sZGGJLifEy3pPkO0W1KmguN9AwAWK6+kckAjMa3ElkfK/xuAqBYgNKQUxP9oMgTQRcgpWF8Nzb0npN3v+HOUIRrth1INSchkDKtWob2T+0Zqxwm+Vpi0iFWwqjcN/OVdaDGf34//A3M/tsiQ1g9SqhdhZJCfu5XLmM63 mkogan@mkP50.redhat.com", "key_options": null, "keyfile": "/home/mkogan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkogan", "validate_certs": true} 2019-12-05T04:58:01.762 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315', u'name': u'clacroix'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDnvVnbK2SNfL8hTmrA09L41RudKW/F7Ty0vvlHjrFjDkdWXVgrafFWGKQZ8dSR6FJ9JLU0X59xiEo4z31ini/Dt2XozFMIr7RdeB3xpiifuUUG5Lgsasd2ezniUoOEukL1a9KkCIxNgmKJuvtlHCt+ZTtyL513yjRICByZbnmBSSpUiett50X10vtvaNBu4LikUJKLc8gP0PHLbaGsU3uOUd655lkFWjWhgExlU8bHWiAZq18Ss63jDlSKFVSk903GKSNcHXMHNPYaXlrYw8NFzejoy/XjxEfmF/Lv3MeAUhHQWx25sh5aDv48mzo8FaIJsL1MrD0QxF1gllobWAfp clacroix@redhat.com", "key_options": null, "keyfile": "/home/clacroix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "clacroix", "validate_certs": true} 2019-12-05T04:58:01.924 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2019-12-05T04:58:01.948 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006', u'name': u'epuertat'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAyiGmj+nOAD+il+FvXvZSpUOJQp6hKh6UPaAIQvQZkdeNRWN7wcDXMUlLj+TBFWa1OO+9xqFlnbgVMDR4qy5orUBYlMapUb1rutjbk64XeY58b6uguHwSoMd81QpCCIN4KOk3cldV8UiyX382baoZ9Je2hsOnnBGMsRtHuVHZ90Bj1rIITkwajq//KRs7kjb/qhjk0xkipDIcaSwKjzPlmmnYn69noODwDexpvLYlCF+LowSZEdrOUvVIigSFG7cLTDaU2tLhxSgbvWu8f0pwvjsyW7gOgM+VeRZ6uHC79F1qvgdgQSxflddIgrQpH0rzf6RNxQjR3SBkZ4V1BESa0w== epuertat@epuertat", "key_options": null, "keyfile": "/home/epuertat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "epuertat", "validate_certs": true} 2019-12-05T04:58:02.095 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736', u'name': u'henrix'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDZDKRAePFqjugc2ljw4QmM6r7B2dPI39OBB+2HKONtDZI4q+uwgZ1yqnhrbJaU9ux0W5wNzJyp2VfOPhw/9G9jo+efJPg5IySnTonhi1zeCXNpBMY70dJZVCyGEDdWKehvFM9xFBOxBC/h7gOyKqEc28BKoy4cHfULzm4er8PU30MyFBngu8NaoKqyQ3GYAfWOAXF5lUdalbJwl3a+HBb8PjUhbGuznAZF1Ji4JgnNuEZdlNVOKwhEkAFZ1JwXvgLfrEJVQs5b2V9Sp2Vm5ZngsLg7b3RzV5I4B3Lj7QTTraV1zLJGiS5iRdjY7D4W5Uxi2k1X7vaOjySJWWadHyeicNNBhLSfHl6JJ3aMuA8oAAmtgo/8IkJ96GFtgM9j31Wg8wkeTyQrwSeqCuv1uDFr8rp+OvMClhHnDgUklJLWjjYPCDVYR5KV+iOiHL58WJg/b4h6NsoP91wO8P8EcSaiMdq7P77w5nJxDI0KKb//dC8aHtWnRRRlrIS+PNPqKJSNTaSwOTieXvzplhzKFzu2dr79qs+/67iaNE6HBvIX5jVHWCFV3wllyjsC8WYPh4kEVtg2SY00SDOOrMrqANV8DDtHFwwxQTEjz1z5ESuwy9UcUSQfP1/zsAp2OQzPutKs6UipHyBK3pLm6MjaeeaEWu8fmetCqALxYV3h17O+Yw== miguel@hermes.olymp", "key_options": null, "keyfile": "/home/henrix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "henrix", "validate_certs": true} 2019-12-05T04:58:02.110 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461', u'name': u'tdehler'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRZyWiDO5JqWHJdUUBr8YReiKXxnL87juFAfkcmw2E9KmANVH2sn3K+OgUCQfBs1Wdg6AbcL2GyMm9alodUruzOm127/bH9OcGmu+8SNsMAYrAF9EzsKJa8Ba/I05rl5a87p1jbYG9Rpo7u3OvLx0LjgqydRYcmjwo/niifYNDTaN0K3v8s1mN4TorUZW/i3XBMBIBjlIiNGs/I7IZreR+1N4V8Nwrp2EWxDv7AwG3nvokL2RTHcsV0KgmAnciv8nXXjIf+uqOPuGA+F2qW/wz92bG8R0O4CjH+QZt4k6UoUIXFRW+LdOVAqmUrb4W0OJ8vsVeukW46SvPnYUTkZnF sonea@think", "key_options": null, "keyfile": "/home/tdehler/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tdehler", "validate_certs": true} 2019-12-05T04:58:02.258 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26', u'name': u'pbs1108'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNwmgnIsDDJmsmYzkTAEo3iLj7Jd5Ni2R5vD1Z0NRZ39zJMtbicZ0ec3nAMzFyHrmPnKNFltS1mhehbLbtKNKq1fSo2SMHs11z+N8Nl94F4AY8IhXZ8U9GAorEJFVIxHUNEDqwl55Ov6MfP60UI7rNgy2rUt8aOBmAJVhLEFmJJawJgyPhTrUZAT20OoILphrt81o5bTNfmx++Zs5Dc9VtgIh53hexwKgTvG6fxfKdra9m+J2GXpniOsZjPJq/A0je+t7H3UlxCSu0dRy2Un6l5ZAcoeg74SP50dFgGaxkTQqI6wle6NjtPIdssgrdYTgTcOg9M4jWqAr/5drTUSGh root@centos7-bspark", "key_options": null, "keyfile": "/home/pbs1108/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pbs1108", "validate_certs": true} 2019-12-05T04:58:02.319 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305', u'name': u'laura'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEBD7CxJJfbLW4jtuLnTwkxVbhCfh/JL3Qj8oeLEkny1tE7QCeXn1bP5bScWSgLH34Wh4TAuH+8/g/koFwcCy1YuoGBdBV4KV7O9MTnFngdqNY6LKzCipm27tnWZEa10hrF8dRFTmhracd6kmmVGHgvGQPVAVONmix+2IZ83wdj8XJSQHeE3PHFLO39tU+ez/T76QaiBLDVo4ctHVt+N9bBbHV6Cr8fyiUUU6YEKMlcedvxGpnMK8oeOSNuDeh+DldcM9KYzNGEOM34jGRIvJp57dxQDQCtPPW5hvBXg7yMyrv6TcJzDd2LFD6AC/Qw9yrEQPXXyBCDkAlCqDR0+rH", "key_options": null, "keyfile": "/home/laura/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "laura", "validate_certs": true} 2019-12-05T04:58:02.433 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315', u'name': u'clacroix'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDnvVnbK2SNfL8hTmrA09L41RudKW/F7Ty0vvlHjrFjDkdWXVgrafFWGKQZ8dSR6FJ9JLU0X59xiEo4z31ini/Dt2XozFMIr7RdeB3xpiifuUUG5Lgsasd2ezniUoOEukL1a9KkCIxNgmKJuvtlHCt+ZTtyL513yjRICByZbnmBSSpUiett50X10vtvaNBu4LikUJKLc8gP0PHLbaGsU3uOUd655lkFWjWhgExlU8bHWiAZq18Ss63jDlSKFVSk903GKSNcHXMHNPYaXlrYw8NFzejoy/XjxEfmF/Lv3MeAUhHQWx25sh5aDv48mzo8FaIJsL1MrD0QxF1gllobWAfp clacroix@redhat.com", "key_options": null, "keyfile": "/home/clacroix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "clacroix", "validate_certs": true} 2019-12-05T04:58:02.456 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce', u'name': u'yaarit'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2k8xsrsLxEqJx3rV6ZZwLH0OjYCDB2uIYzcAyrKEOKlVWO1ICKxkw+gZ9tSiPFNLfq13io9BIrAKJwlrdLCJLmlgCQwpwUQqzmRlRZEm+WmD0dl8LPeQSidyJkPkJgYBEshFPjDHIj10Ei1pJS5svI2l7G1wMWxlt2kHL8Og/4zdFTRctxwqZcvzLmRQcs8lSSpqTkhfeKfmrpR+je8XFbCJJ9m/nJbgyCiZKaGN8nf36pCQDEva4rvaSrK0a6YxL0RWN8W3lqtBPKIehxffr3lYDEkD9EUW5vf6AUKC0ioh4jkPo9WCjaoFsOauMOMVQPHehEAMufEO4Kym6nfrf yaarit@localhost.localdomain", "key_options": null, "keyfile": "/home/yaarit/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yaarit", "validate_certs": true} 2019-12-05T04:58:02.601 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006', u'name': u'epuertat'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAyiGmj+nOAD+il+FvXvZSpUOJQp6hKh6UPaAIQvQZkdeNRWN7wcDXMUlLj+TBFWa1OO+9xqFlnbgVMDR4qy5orUBYlMapUb1rutjbk64XeY58b6uguHwSoMd81QpCCIN4KOk3cldV8UiyX382baoZ9Je2hsOnnBGMsRtHuVHZ90Bj1rIITkwajq//KRs7kjb/qhjk0xkipDIcaSwKjzPlmmnYn69noODwDexpvLYlCF+LowSZEdrOUvVIigSFG7cLTDaU2tLhxSgbvWu8f0pwvjsyW7gOgM+VeRZ6uHC79F1qvgdgQSxflddIgrQpH0rzf6RNxQjR3SBkZ4V1BESa0w== epuertat@epuertat", "key_options": null, "keyfile": "/home/epuertat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "epuertat", "validate_certs": true} 2019-12-05T04:58:02.616 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3', u'name': u'rpavani1998'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUznqo3VDmcKRL64P7sbjuxXCW/0ljFTc5zZoZ9UvtZvU2jrRQ2X9GyxcrcQnAWlF5YZHqeTDU5o8ixn5fq1MSmaPRBNDKKVQOfR/vFQV73LiKx4BFsC1/EdaPie/XqltaLDTtfqhxADJlz1H3a3ZlkeVHMROTqo/TidL1YUYlRH4qHxWWBuFuVS+EYfTegSZdjwp+DuJA0iN6X99+DfH2DhiHENjEmlbKN13+H+EPzm0K3ya2w8D6ANodB3AxMn2tGs8mLtG/P4kCjoDnoUVFzKTyUpTtgOjis47uSY4dL8mTbIK9kZAiFxnT5sMClu6KRStDjRIrxbTBQJ8wvlat admin1@rajula", "key_options": null, "keyfile": "/home/rpavani1998/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rpavani1998", "validate_certs": true} 2019-12-05T04:58:02.754 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4', u'name': u'rishabh'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4NYxUHx8HMgbVIHadruN1kAJS5be0aZA9rvGtYDfwD5siKNfXFRyLG3thkjxtEi7DPYMxgbpFOXW9EQKh5sQ0ohsADZOi8FSIAfQlOQ+7HAEp7DjBtjxoqB31jJ1e293YtO+nqAhcOMyGfDx+71w5sS2EFqy7PscA9gD9aV2iMxIR/JD1LxXabiqj12jU5AOxr7ZQ1+gM9aroQuDpNxY4U8jBTxx5OtIYRkeI/aN2gQj1mXGMKGo0ItFn4lsXGLxGxMpa9K2UFw2dWQVTw9VNm79nixrCX+lC2nyaJPsRYZipXF1ID2NRk7oJ79dJ/uRBBXpunihlEC3aF+nOgEPH ridave@redhat.com", "key_options": null, "keyfile": "/home/rishabh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rishabh", "validate_certs": true} 2019-12-05T04:58:02.761 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461', u'name': u'tdehler'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRZyWiDO5JqWHJdUUBr8YReiKXxnL87juFAfkcmw2E9KmANVH2sn3K+OgUCQfBs1Wdg6AbcL2GyMm9alodUruzOm127/bH9OcGmu+8SNsMAYrAF9EzsKJa8Ba/I05rl5a87p1jbYG9Rpo7u3OvLx0LjgqydRYcmjwo/niifYNDTaN0K3v8s1mN4TorUZW/i3XBMBIBjlIiNGs/I7IZreR+1N4V8Nwrp2EWxDv7AwG3nvokL2RTHcsV0KgmAnciv8nXXjIf+uqOPuGA+F2qW/wz92bG8R0O4CjH+QZt4k6UoUIXFRW+LdOVAqmUrb4W0OJ8vsVeukW46SvPnYUTkZnF sonea@think", "key_options": null, "keyfile": "/home/tdehler/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tdehler", "validate_certs": true} 2019-12-05T04:58:02.952 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305', u'name': u'laura'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEBD7CxJJfbLW4jtuLnTwkxVbhCfh/JL3Qj8oeLEkny1tE7QCeXn1bP5bScWSgLH34Wh4TAuH+8/g/koFwcCy1YuoGBdBV4KV7O9MTnFngdqNY6LKzCipm27tnWZEa10hrF8dRFTmhracd6kmmVGHgvGQPVAVONmix+2IZ83wdj8XJSQHeE3PHFLO39tU+ez/T76QaiBLDVo4ctHVt+N9bBbHV6Cr8fyiUUU6YEKMlcedvxGpnMK8oeOSNuDeh+DldcM9KYzNGEOM34jGRIvJp57dxQDQCtPPW5hvBXg7yMyrv6TcJzDd2LFD6AC/Qw9yrEQPXXyBCDkAlCqDR0+rH", "key_options": null, "keyfile": "/home/laura/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "laura", "validate_certs": true} 2019-12-05T04:58:02.964 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2', u'name': u'skrah'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCeAhoTum3/VmRbxSyw5gwgcPF1EWFK5mEPX3PZyo3xPsUAJazALfge3aEuCrZBbsjOCoj3GfFhr/ziEF+OWvhBoPK5hQzP0V8tE9cWJwvHmD72vGE36o1hFO0ONE4DEuu5KRDqQJUgeyQWXsP26uHTwIcvMpy8egZOfyshqF+lQVXzsAex7Ninmidsf/sRNLhzT1kLrKT+ZMSbZ7Rqh46TMlmLD1LzJZNcb6PGVwlfQZmV6T7m0COu/skeSxMcy8hqMvHFNvvfFSqkfMw0KBGLDxMQr4L+e7sXwQDnEZHdzoh0iS3xDp/Onr1FSmPyEjz51e1sMHBsqklHWidSZVLt/+giB4o6Wdgzc0um/QxW2Ne2qjtDlViOgRE9Fzcw/cRu/Xsn8qtXTxYyxXswjkA8qX7WR22OYgjYChw9siUsKkuD2yAbPMVl+cZWDeBhQnPMbDMVaghS4SAVt5GK16jij4+r3IE9vGHBmGwJYJ/OrtcxBrUrDopwsGLcU+WHdv0FXuiw+YbfnzpthC8rVj4oeJ2IUOQ5JB0ub7NmbR4QWjc33JfD+AeCB75x6io07V2riRjMD9gLC5nwytJNG7Yytf6xI2i/CprW7HhsDs2ykOVcDSBMnL1oGW6FeHaDRW39r6BXSt1jjZQ2E6pDEBTDO41tPEL9EVvv9TK87GTylQ==", "key_options": null, "keyfile": "/home/skrah/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "skrah", "validate_certs": true} 2019-12-05T04:58:03.100 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8', u'name': u'smanjara'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9/Io6M3q8yQqoPNPKAPJ6Fzq18DMfmNtxcD8GUWNo0xcMxd7XOlarvYc7INDQjkpzx6s+aqMme55pSUZuN5plW+nuFmRPqBaIxyeCnNjzCch0wbtEZe5dzT270FhzM1Tu4W0dw+WD6zrGjXayLNs2uxNp3UrWtMTBRzd0IfxJ1R2aJtiCE/HNxIZn+bhS/OmEPV1izz23MvKXD+nRaeJK6Aqrg1zbnpoNskz2fZ/AnmhnAMc4EHl/NUyW1mi4g81KcQMwE8poWmw1NpZje9gWFcISVClLvvhk/A6at/v90P7PisX+/ulTA/QYJYvGBjgbNtURYu/PWzYBOAEZQGfj shilpa@dhcp35-184.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDqRSG3i8QY7t9sp6I8vDZ7LVdk+DDrMhI1QmvU2z+YQhUGDBOvvO6YP0gUXPDyalyj0sJXiwLWPmkrOsbnR96YCUcIt7Ba6R0d0w85JxcJ1WyPcsg2UyOBwq4A+s7VAFZm6FEWzlaubixsNp04gKDqBjSjaqA1Kwx8/Tg39byay8Hn3VJeZ1SkRukvhykz2e6gl7ydfuJQVmEyO3pTrd5157ZqgvQRABT10wc3vR2m4eC65hU/v0ci7HMY9oVtGEdWVBapWMuiocrOPCRWF5zrMJwW4vzPOCHa/47hBEIWfOWJidUVzAawnM3nOe7PykvvDK/iP8B6JpxKTE74eLeh shilpa@fedora.lab.eng.blr.redhat.com", "key_options": null, "keyfile": "/home/smanjara/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smanjara", "validate_certs": true} 2019-12-05T04:58:03.101 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce', u'name': u'yaarit'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2k8xsrsLxEqJx3rV6ZZwLH0OjYCDB2uIYzcAyrKEOKlVWO1ICKxkw+gZ9tSiPFNLfq13io9BIrAKJwlrdLCJLmlgCQwpwUQqzmRlRZEm+WmD0dl8LPeQSidyJkPkJgYBEshFPjDHIj10Ei1pJS5svI2l7G1wMWxlt2kHL8Og/4zdFTRctxwqZcvzLmRQcs8lSSpqTkhfeKfmrpR+je8XFbCJJ9m/nJbgyCiZKaGN8nf36pCQDEva4rvaSrK0a6YxL0RWN8W3lqtBPKIehxffr3lYDEkD9EUW5vf6AUKC0ioh4jkPo9WCjaoFsOauMOMVQPHehEAMufEO4Kym6nfrf yaarit@localhost.localdomain", "key_options": null, "keyfile": "/home/yaarit/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yaarit", "validate_certs": true} 2019-12-05T04:58:03.260 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41', u'name': u'bengland'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDpp2NWr5T6s+jzHiz/FfCKzN7ZBnUSzyE7ieuntzjeGotyupAAjVBoSpCJgNDybm+WkE4dShv5mttPJ2/q1UZCLRj3nN+y3NG333aLZuyugR0ht5qtvGVegg58PwQ8CMobR7ca7bl5UdEy0UIrM3d/qpDOuPkuE4MIJcrtZGICE8NAVnmHNOB9Sm4ovAYkewTuMAv40j2Dl4F9dD6bg+OJfBOJ9+Zj+MpowYlAILtEB9Dc20zB8GGUEtr9yfOVroblTYhvTxGYRaE7aVBe83DNwkWGbDc/qUNR6bpA04w/P+LclYVsVKRoHOs1EXHSwNMFromA8RsKS+rajwu5uRg5 ben@dhcp-185-182.bos.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC93wGgyyQmTxa7f+BZcYPUapTH2N1mnV1ikl9nzJzHhgrtJ6pv00LhtH1HqGhvoqnqZjBD5XRAKyjMRcMBT5MYcXXep5RqZ9sJ6Xz8rHkBjfl/OOQUeZ/ZyG/Ksr1jpGraN2xtrjQQzbqN1YvirCqfLk2exPvVeEk4CGnNybwKwCWob1uKV46ASrYeQK8pfHxrW7pTo+xdErHOvb7uumZm89/YisUJGicO1qSHBeuw1MxjhnD+fIcxHKWY5NNJ4qScW6a5AABk1JJsoHkh9IGCVaf/zJv8D3gil5SiM8LIBKaNa1fn6oObPl/Ky/wkiIo2gmUZawvAVA8iJnX9eUPl ben@bene-laptop", "key_options": null, "keyfile": "/home/bengland/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bengland", "validate_certs": true} 2019-12-05T04:58:03.280 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3', u'name': u'rpavani1998'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUznqo3VDmcKRL64P7sbjuxXCW/0ljFTc5zZoZ9UvtZvU2jrRQ2X9GyxcrcQnAWlF5YZHqeTDU5o8ixn5fq1MSmaPRBNDKKVQOfR/vFQV73LiKx4BFsC1/EdaPie/XqltaLDTtfqhxADJlz1H3a3ZlkeVHMROTqo/TidL1YUYlRH4qHxWWBuFuVS+EYfTegSZdjwp+DuJA0iN6X99+DfH2DhiHENjEmlbKN13+H+EPzm0K3ya2w8D6ANodB3AxMn2tGs8mLtG/P4kCjoDnoUVFzKTyUpTtgOjis47uSY4dL8mTbIK9kZAiFxnT5sMClu6KRStDjRIrxbTBQJ8wvlat admin1@rajula", "key_options": null, "keyfile": "/home/rpavani1998/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rpavani1998", "validate_certs": true} 2019-12-05T04:58:03.414 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d', u'name': u'pnawracay'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC3YhI4am/T6s4rJRvbC/jAvrNcDhE/zaJiVDAjEuA1W/5Xt9ObX52eAnp+tOOtelXrX8ylYF/z5wGNQLjYrWepXhuqaEGxeVayCHzts6LMpmvyZ6sYnAw9HFO9rzrTawm70DuvWqY2Zt4KA6fM+lfXk3Qdpi5RCPDKk1jpXwSCyUtUKSWTDRtYtvvJTcO3H/xCJxQq/F9sU8yqgQcqFPfMNrV4a8gNj9KACqjt4IEIfuxKlKhlMA4yu1VTmGcNuhQS+geJzsEGuxZUQb0Io+q4o0b+40BQtazKgOdTcQbgNtqh3wZpjJsvB4AM+GNPjZY3mCsd+WPGg6dqfU08Voo2b7BfSwyCn14fQHnj1nSwRIea2juFY6AMfVJjstM5iIz7lOH2c3ydWWhzvSmMeoQ47zBXlg8JytWmDrXiyd21BSIpTL0Tn8P95wHYka8U9KRV0ZY7Jr2w/ThPdy2C+Cdir+P+OEL1sKHSnmEyQEeONa9P5yH1Rd1kM/FNESAIRiMLGspzXQeWoJhHswVaKoSBW/7f4b9VENxda8qAQi6ob8A71cAjbpRrkUmd5EBCSFMNs7lscus0/deUVvlpsrMjDAJa8S9gqhGHNU0YvF1WBIz3paLRAHDm9DWrHc5lreizJzX5ul4P/Z4ITjpO+9ZxKOcyi8wnJ8Og+QUYLpqOIw==", "key_options": null, "keyfile": "/home/pnawracay/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pnawracay", "validate_certs": true} 2019-12-05T04:58:03.444 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4', u'name': u'rishabh'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4NYxUHx8HMgbVIHadruN1kAJS5be0aZA9rvGtYDfwD5siKNfXFRyLG3thkjxtEi7DPYMxgbpFOXW9EQKh5sQ0ohsADZOi8FSIAfQlOQ+7HAEp7DjBtjxoqB31jJ1e293YtO+nqAhcOMyGfDx+71w5sS2EFqy7PscA9gD9aV2iMxIR/JD1LxXabiqj12jU5AOxr7ZQ1+gM9aroQuDpNxY4U8jBTxx5OtIYRkeI/aN2gQj1mXGMKGo0ItFn4lsXGLxGxMpa9K2UFw2dWQVTw9VNm79nixrCX+lC2nyaJPsRYZipXF1ID2NRk7oJ79dJ/uRBBXpunihlEC3aF+nOgEPH ridave@redhat.com", "key_options": null, "keyfile": "/home/rishabh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rishabh", "validate_certs": true} 2019-12-05T04:58:03.580 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8', u'name': u'alfonsomthd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC709gIxs1RgiA95ddp2uymzgV84VHJVVMj71FyTlz6pa0gtlaxF+r16p8dnNWjhyJRUXMwBKFOwHx95bcdfiRA+BfC04UnI8VvBxoED5Nt9EbKD5h78ZoNEeQJKoG1dsM/kWWlTd9WCsCyzjRz32+YNWul0LOhrfloHP+LHnfjf/uOnzd8iRHgnpWr5Z9uB0MBu4y/L6AmHnR+8pMZQq6Bfno8ZLwNCzx9v+4MqOz3jY7FYv7SJIftQH0fJygIfCz3bKyUEWk9a/JJacETuJGBnHgNAXUr/FtM1IFiMl2SsTixRgnosEz4pX8EIe5LJyw/Qudro8BPEC8aIGy0XAkerGmTt5iALMnzmeeanZCFm4rF/YiFoHLMS/B0uv42MD64hB3knrV8Bcq+mRjwVOYgecO+xoiBsU6FZeXaBk5IsjmYTwataGv3ILEmNpss0WeqHxF9nd7oTLnw51cZfcaerOnJ5IZJZTXwLNVr7K176BunHggAhjvtvwdeKhphqt9f3ilLr6gzksEoAR4JBOYJJtd8sEqHwBtti+YbMGwH9Y+Z+SujlQBQIsom60xj5OzrwjsIi5JAtOhbc1Mq2fptSXRsR8ZBT8aUFqZ8R7ixEqq2F47OAznbhSQKidzwUNkWNI3WAKi38Oi4aya1k3iqtL04bGf8R/xD5Vq/2J+UlQ== vant alfonsomthd", "key_options": null, "keyfile": "/home/alfonsomthd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "alfonsomthd", "validate_certs": true} 2019-12-05T04:58:03.628 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2', u'name': u'skrah'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCeAhoTum3/VmRbxSyw5gwgcPF1EWFK5mEPX3PZyo3xPsUAJazALfge3aEuCrZBbsjOCoj3GfFhr/ziEF+OWvhBoPK5hQzP0V8tE9cWJwvHmD72vGE36o1hFO0ONE4DEuu5KRDqQJUgeyQWXsP26uHTwIcvMpy8egZOfyshqF+lQVXzsAex7Ninmidsf/sRNLhzT1kLrKT+ZMSbZ7Rqh46TMlmLD1LzJZNcb6PGVwlfQZmV6T7m0COu/skeSxMcy8hqMvHFNvvfFSqkfMw0KBGLDxMQr4L+e7sXwQDnEZHdzoh0iS3xDp/Onr1FSmPyEjz51e1sMHBsqklHWidSZVLt/+giB4o6Wdgzc0um/QxW2Ne2qjtDlViOgRE9Fzcw/cRu/Xsn8qtXTxYyxXswjkA8qX7WR22OYgjYChw9siUsKkuD2yAbPMVl+cZWDeBhQnPMbDMVaghS4SAVt5GK16jij4+r3IE9vGHBmGwJYJ/OrtcxBrUrDopwsGLcU+WHdv0FXuiw+YbfnzpthC8rVj4oeJ2IUOQ5JB0ub7NmbR4QWjc33JfD+AeCB75x6io07V2riRjMD9gLC5nwytJNG7Yytf6xI2i/CprW7HhsDs2ykOVcDSBMnL1oGW6FeHaDRW39r6BXSt1jjZQ2E6pDEBTDO41tPEL9EVvv9TK87GTylQ==", "key_options": null, "keyfile": "/home/skrah/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "skrah", "validate_certs": true} 2019-12-05T04:58:03.722 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e', u'name': u'oliveiradan'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCoL9+aMxLtUG3aRIuT4kbjc1CPuQ+wbS30ocuJ2YIExlKMSxVr+LYqbjcz0T4f8gAKG539dZQgD9BAsHM8yfOVEvei0KsQHn87pzgYGR3DtaMx5IEG5xNyARPu+VuEoe8qlL3cAPwTiaU78datD5Cz9GEClhL1Fhn/Z8o7BMbs5xlcUIQ/J+s9/sKMcqmmBJy5IHQIEUHCl4SFwsev5g5ADlnLNNxjpalExuBCEeoSma9F3b2YEJHvpA5JhCIMZpVfXbgh2UhUsyfAQN6KgOIQ/XlKBMQDtizf2vZ4b8hW4ELcvKSADfTsyyqEs/Ql8jEJu32GFKf2mYPbgTS6E7FZfQz77X+5OBhojFAstB8LSSrH80FrG1gCbVZLnW0jXcSehKnU0oNmUWOKxxaT1b00fo/iZ4P8O1uJRY17/uaVCPNKdGTZzIlzyC16wsFFBxBl0d7HfojQY6qfww7AQJhvDkV2sP0/MIJzwriWOzZEmfr8Osqery5vFiYcSvMbdMZDfMEYd3z7Va2RiTIeyXuEYp5hzf8eyJRP/f2FEw8/BBrm5JCxeSWUx7qwgPHrJoWAVxCIg/M3YS/vfVSFc8Hh88sgc2upq0OqYr0uq6EFzUZi4aoOkOVYn9u/uc/p1AR0KbiQOkERMUDWTyADxrHaL0nQmGy+XyiJQjhq5ckZ+w== doliveira@suse.com", "key_options": null, "keyfile": "/home/oliveiradan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "oliveiradan", "validate_certs": true} 2019-12-05T04:58:03.767 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8', u'name': u'smanjara'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9/Io6M3q8yQqoPNPKAPJ6Fzq18DMfmNtxcD8GUWNo0xcMxd7XOlarvYc7INDQjkpzx6s+aqMme55pSUZuN5plW+nuFmRPqBaIxyeCnNjzCch0wbtEZe5dzT270FhzM1Tu4W0dw+WD6zrGjXayLNs2uxNp3UrWtMTBRzd0IfxJ1R2aJtiCE/HNxIZn+bhS/OmEPV1izz23MvKXD+nRaeJK6Aqrg1zbnpoNskz2fZ/AnmhnAMc4EHl/NUyW1mi4g81KcQMwE8poWmw1NpZje9gWFcISVClLvvhk/A6at/v90P7PisX+/ulTA/QYJYvGBjgbNtURYu/PWzYBOAEZQGfj shilpa@dhcp35-184.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDqRSG3i8QY7t9sp6I8vDZ7LVdk+DDrMhI1QmvU2z+YQhUGDBOvvO6YP0gUXPDyalyj0sJXiwLWPmkrOsbnR96YCUcIt7Ba6R0d0w85JxcJ1WyPcsg2UyOBwq4A+s7VAFZm6FEWzlaubixsNp04gKDqBjSjaqA1Kwx8/Tg39byay8Hn3VJeZ1SkRukvhykz2e6gl7ydfuJQVmEyO3pTrd5157ZqgvQRABT10wc3vR2m4eC65hU/v0ci7HMY9oVtGEdWVBapWMuiocrOPCRWF5zrMJwW4vzPOCHa/47hBEIWfOWJidUVzAawnM3nOe7PykvvDK/iP8B6JpxKTE74eLeh shilpa@fedora.lab.eng.blr.redhat.com", "key_options": null, "keyfile": "/home/smanjara/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smanjara", "validate_certs": true} 2019-12-05T04:58:03.863 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba', u'name': u'swagner'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDh9bzZJulXGES+l9Xh6Aq15RZ8uQCGuDNhlNQtDblE1ISKJ/DqGYXp6wUW54+oVNA7eZiXz+fi3mq5pPEtZZOfd3ixEDzDJ4E3cVXoDZCqeWEmea6KvybeY10YxvB56TEI8U2KKAd56PRl20klXpLCjzXNqG7n0aXFcpCMbXDu369VX6lOk24K/7++7Wc5SttcvVu19sT2kqzsB/S1Y5cxiE6RM6wVtqBoksp/kRCIA16ruNwx3GUabDfbEoUGXlkkUP7+TZgAbHtBsYy6mCQhwi0S2+WG+HhHDUPjHhV+MdN9ffibCtOEGo52itVLVky09VeBocuA6H22JDGXPBgjcgf2NsZIqcKqGHhUkXmH92fhRSFOBLKHstrBq8jWRP/mNrgj8cQksDsakQYQbDg5dyabp+M0/iL2Q3YVq7erZI8aZMA7ZF3WgoQNYZg5E7oejM8URIlFP3x1ne2ClRC9a74phSCxeU/NVamGN3G3dImzEXGOSNyRggHJ4jGrIGc7tLPCzmI5OkomcB5OxReqf0r1TNXuUAqw8M4EoWt+0xoAmH5zVlUHf+psUCJIEV/4pbgtoJiSNq+LVY4jyDEFbvTAL7MqyXorMV7Tqlj+/3d7RXfhW9lR/SHb1Z3jcHvz4ZzYMzKWeJEjz+Y0NwIdFDhcPmUOYtEDmuRRrgYiSQ== sebstian.wagner@it-novum.com", "key_options": null, "keyfile": "/home/swagner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "swagner", "validate_certs": true} 2019-12-05T04:58:03.924 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41', u'name': u'bengland'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDpp2NWr5T6s+jzHiz/FfCKzN7ZBnUSzyE7ieuntzjeGotyupAAjVBoSpCJgNDybm+WkE4dShv5mttPJ2/q1UZCLRj3nN+y3NG333aLZuyugR0ht5qtvGVegg58PwQ8CMobR7ca7bl5UdEy0UIrM3d/qpDOuPkuE4MIJcrtZGICE8NAVnmHNOB9Sm4ovAYkewTuMAv40j2Dl4F9dD6bg+OJfBOJ9+Zj+MpowYlAILtEB9Dc20zB8GGUEtr9yfOVroblTYhvTxGYRaE7aVBe83DNwkWGbDc/qUNR6bpA04w/P+LclYVsVKRoHOs1EXHSwNMFromA8RsKS+rajwu5uRg5 ben@dhcp-185-182.bos.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC93wGgyyQmTxa7f+BZcYPUapTH2N1mnV1ikl9nzJzHhgrtJ6pv00LhtH1HqGhvoqnqZjBD5XRAKyjMRcMBT5MYcXXep5RqZ9sJ6Xz8rHkBjfl/OOQUeZ/ZyG/Ksr1jpGraN2xtrjQQzbqN1YvirCqfLk2exPvVeEk4CGnNybwKwCWob1uKV46ASrYeQK8pfHxrW7pTo+xdErHOvb7uumZm89/YisUJGicO1qSHBeuw1MxjhnD+fIcxHKWY5NNJ4qScW6a5AABk1JJsoHkh9IGCVaf/zJv8D3gil5SiM8LIBKaNa1fn6oObPl/Ky/wkiIo2gmUZawvAVA8iJnX9eUPl ben@bene-laptop", "key_options": null, "keyfile": "/home/bengland/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bengland", "validate_certs": true} 2019-12-05T04:58:04.003 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e', u'name': u'yuvalif'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDzYzqIDimPjMOUOUz1A122eV52fl3mgGgNBRYRlOWl/lOxM5oOat5FPkJlnXTVyQFl0MlnYgIv9fTnjHVTM0wFmFUItlUl151ZmK5XExiGpGVsqG3UWNyKJxZpk9Uh2a0T/5CKzzwXnv2fJMNxzfqgopYDrgq4TFEZcdvgWJ39dcRTXST/rGJQ8wOq9z7UNzKU5VOGxvgi6f/nGevGIHrSaTcY3Uu8BW8kXUrf9vfl+ZSxVjuNHXF/VmrJH5Ne8mVu7DzPMrl8ZZjxsfJKT/eQqF9kjqht5586OSPKIwg/LEOyYj7Wyyf3QbDAIp5bCygQfmBBl+kIxphJYq3EwUZT ylifshit@dhcp-4-102.tlv.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFgwQYD9amVSYq6JzjHyGmKBJATQCM9xsf6QnF3yb0Lvry2nzY9F61evVXQ95TmF/AqasRx5p2WansIvegG5ag5mytvRwFXtHzPs4o/J+XjA9o4+A5l6j9cOFLIDD/TwrwlUjgMa2PhqUztSpI1op906391RLrJAyflfDghJxeVl6dJab6TEsslV0W+yaCS3BBxMXXrCbxCB5Ms3KCRKDz0DqbhFc+Nw05L1O1pC98HBFRIp1SaXlMcT3++CRQ3wugpOMkDMNLbycl+vdMBFY3UZ28rRowS3Ta7V5EARZVVH7KxU5cL+8Hr/8DE1/ITOmrluaEZ1KwYgoxZbZ3w7Ptw0FK6aiCZ49YtqKT1WakoVfO8m7pCdxkqUOA91loMQtxvCwZ/Vfoxaut16gklmPz8N11xvo1H5XB8/ZrvK5yKdl3JDl+wwtfE/fwo4E/fte+rPRisoWBRsmsx8EGxs75sXnb6n3VjZ65vYEKsNMLg+5EAaevWLLeqF0a42UWGJrHdb4VKX/OAplHrOk7kA/VwArow8f9n+iaRp2hgDU9OhQiFVRhgqwahlmCFFXkcP4/4WlxMCuZrsaNMZgMjVhfsp7cskKOezSfZZey2MB2fREMS1wg0j40CdYRn4jTNEBr/LVszsoM7t7AwXUpnqXeYIe+gePx1PJb5XSzSPZ9Pw== yuvalif@yahoo.com", "key_options": null, "keyfile": "/home/yuvalif/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuvalif", "validate_certs": true} 2019-12-05T04:58:04.099 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d', u'name': u'pnawracay'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC3YhI4am/T6s4rJRvbC/jAvrNcDhE/zaJiVDAjEuA1W/5Xt9ObX52eAnp+tOOtelXrX8ylYF/z5wGNQLjYrWepXhuqaEGxeVayCHzts6LMpmvyZ6sYnAw9HFO9rzrTawm70DuvWqY2Zt4KA6fM+lfXk3Qdpi5RCPDKk1jpXwSCyUtUKSWTDRtYtvvJTcO3H/xCJxQq/F9sU8yqgQcqFPfMNrV4a8gNj9KACqjt4IEIfuxKlKhlMA4yu1VTmGcNuhQS+geJzsEGuxZUQb0Io+q4o0b+40BQtazKgOdTcQbgNtqh3wZpjJsvB4AM+GNPjZY3mCsd+WPGg6dqfU08Voo2b7BfSwyCn14fQHnj1nSwRIea2juFY6AMfVJjstM5iIz7lOH2c3ydWWhzvSmMeoQ47zBXlg8JytWmDrXiyd21BSIpTL0Tn8P95wHYka8U9KRV0ZY7Jr2w/ThPdy2C+Cdir+P+OEL1sKHSnmEyQEeONa9P5yH1Rd1kM/FNESAIRiMLGspzXQeWoJhHswVaKoSBW/7f4b9VENxda8qAQi6ob8A71cAjbpRrkUmd5EBCSFMNs7lscus0/deUVvlpsrMjDAJa8S9gqhGHNU0YvF1WBIz3paLRAHDm9DWrHc5lreizJzX5ul4P/Z4ITjpO+9ZxKOcyi8wnJ8Og+QUYLpqOIw==", "key_options": null, "keyfile": "/home/pnawracay/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pnawracay", "validate_certs": true} 2019-12-05T04:58:04.199 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852', u'name': u'kkeithle'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXa9uTf3q+7tGnHzSxv0WbgpRKrBIYjAZ5x3V74vQUSD1aAbmyvOqgPnts7XttLYImjjRqiisyC03L3KgR4b3SOeyL3KFxrk3KkHAJ/AJH+Y/UafyDhHNl3gRNg0B9rjsa7h2dSlcrFKnAX0pDY1HapuNPrzYWrXsEbo7C6wteFh7PyPmTkHu0NzIIPC4FcLl8SGaww/HeBh8v91IymgFSgXG0jNC063FEExWdti54yNLGh3gpW0FytsiRN9eEOKUOgcveT1tejz9ORv8evO4Y9SCQ/lsyOKQKNcG5C7J8nsbwSjqQ2cf6i2aYFZ0Oq1UXuhJ8APJCoiKgLJy7AiYf kkeithle@dhcp-100-19-89.bos.redhat.com", "key_options": null, "keyfile": "/home/kkeithle/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kkeithle", "validate_certs": true} 2019-12-05T04:58:04.288 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8', u'name': u'alfonsomthd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC709gIxs1RgiA95ddp2uymzgV84VHJVVMj71FyTlz6pa0gtlaxF+r16p8dnNWjhyJRUXMwBKFOwHx95bcdfiRA+BfC04UnI8VvBxoED5Nt9EbKD5h78ZoNEeQJKoG1dsM/kWWlTd9WCsCyzjRz32+YNWul0LOhrfloHP+LHnfjf/uOnzd8iRHgnpWr5Z9uB0MBu4y/L6AmHnR+8pMZQq6Bfno8ZLwNCzx9v+4MqOz3jY7FYv7SJIftQH0fJygIfCz3bKyUEWk9a/JJacETuJGBnHgNAXUr/FtM1IFiMl2SsTixRgnosEz4pX8EIe5LJyw/Qudro8BPEC8aIGy0XAkerGmTt5iALMnzmeeanZCFm4rF/YiFoHLMS/B0uv42MD64hB3knrV8Bcq+mRjwVOYgecO+xoiBsU6FZeXaBk5IsjmYTwataGv3ILEmNpss0WeqHxF9nd7oTLnw51cZfcaerOnJ5IZJZTXwLNVr7K176BunHggAhjvtvwdeKhphqt9f3ilLr6gzksEoAR4JBOYJJtd8sEqHwBtti+YbMGwH9Y+Z+SujlQBQIsom60xj5OzrwjsIi5JAtOhbc1Mq2fptSXRsR8ZBT8aUFqZ8R7ixEqq2F47OAznbhSQKidzwUNkWNI3WAKi38Oi4aya1k3iqtL04bGf8R/xD5Vq/2J+UlQ== vant alfonsomthd", "key_options": null, "keyfile": "/home/alfonsomthd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "alfonsomthd", "validate_certs": true} 2019-12-05T04:58:04.384 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130', u'name': u'emmericp'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBYa5uEOlFv1s9K9D1SbcaApLwX3LBwASSnRXDHI7MtU paul.emmerich@croit.io", "key_options": null, "keyfile": "/home/emmericp/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "emmericp", "validate_certs": true} 2019-12-05T04:58:04.469 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e', u'name': u'oliveiradan'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCoL9+aMxLtUG3aRIuT4kbjc1CPuQ+wbS30ocuJ2YIExlKMSxVr+LYqbjcz0T4f8gAKG539dZQgD9BAsHM8yfOVEvei0KsQHn87pzgYGR3DtaMx5IEG5xNyARPu+VuEoe8qlL3cAPwTiaU78datD5Cz9GEClhL1Fhn/Z8o7BMbs5xlcUIQ/J+s9/sKMcqmmBJy5IHQIEUHCl4SFwsev5g5ADlnLNNxjpalExuBCEeoSma9F3b2YEJHvpA5JhCIMZpVfXbgh2UhUsyfAQN6KgOIQ/XlKBMQDtizf2vZ4b8hW4ELcvKSADfTsyyqEs/Ql8jEJu32GFKf2mYPbgTS6E7FZfQz77X+5OBhojFAstB8LSSrH80FrG1gCbVZLnW0jXcSehKnU0oNmUWOKxxaT1b00fo/iZ4P8O1uJRY17/uaVCPNKdGTZzIlzyC16wsFFBxBl0d7HfojQY6qfww7AQJhvDkV2sP0/MIJzwriWOzZEmfr8Osqery5vFiYcSvMbdMZDfMEYd3z7Va2RiTIeyXuEYp5hzf8eyJRP/f2FEw8/BBrm5JCxeSWUx7qwgPHrJoWAVxCIg/M3YS/vfVSFc8Hh88sgc2upq0OqYr0uq6EFzUZi4aoOkOVYn9u/uc/p1AR0KbiQOkERMUDWTyADxrHaL0nQmGy+XyiJQjhq5ckZ+w== doliveira@suse.com", "key_options": null, "keyfile": "/home/oliveiradan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "oliveiradan", "validate_certs": true} 2019-12-05T04:58:04.544 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0', u'name': u'mchangir'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCj28V9sMybkAzhkiaNiDDr8TfNVyr5DNdbOqszdGEVLy1Rxh/UYnxUclsR9n8lt9VnCft3M4NDTnVwlxOHE8+0oolJE9HP+DQomsgj3abP04EGJyOA0X8E7eIFQNLRbMFCD0yX/SNNlYPsS5fPTFhIa/gbBPmDBE07kULts66Tvc0Yo/2yIfwV1DV34bcWWVGi01XybIHBl9qbvlfBKAUrFEcv3ZdpHNhuZw6IkZY8xoYxo8iF+toLYeGX+gFxqCZUUOO+NBVmb0hVL1LI6bU66IrMOQDkFurqpvoraBTJqwZgXvMPceFUkcVIH4Nm7cK1X2WyMtnSw2XBpLMT8C4V mchangir@indraprastha", "key_options": null, "keyfile": "/home/mchangir/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mchangir", "validate_certs": true} 2019-12-05T04:58:04.620 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba', u'name': u'swagner'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDh9bzZJulXGES+l9Xh6Aq15RZ8uQCGuDNhlNQtDblE1ISKJ/DqGYXp6wUW54+oVNA7eZiXz+fi3mq5pPEtZZOfd3ixEDzDJ4E3cVXoDZCqeWEmea6KvybeY10YxvB56TEI8U2KKAd56PRl20klXpLCjzXNqG7n0aXFcpCMbXDu369VX6lOk24K/7++7Wc5SttcvVu19sT2kqzsB/S1Y5cxiE6RM6wVtqBoksp/kRCIA16ruNwx3GUabDfbEoUGXlkkUP7+TZgAbHtBsYy6mCQhwi0S2+WG+HhHDUPjHhV+MdN9ffibCtOEGo52itVLVky09VeBocuA6H22JDGXPBgjcgf2NsZIqcKqGHhUkXmH92fhRSFOBLKHstrBq8jWRP/mNrgj8cQksDsakQYQbDg5dyabp+M0/iL2Q3YVq7erZI8aZMA7ZF3WgoQNYZg5E7oejM8URIlFP3x1ne2ClRC9a74phSCxeU/NVamGN3G3dImzEXGOSNyRggHJ4jGrIGc7tLPCzmI5OkomcB5OxReqf0r1TNXuUAqw8M4EoWt+0xoAmH5zVlUHf+psUCJIEV/4pbgtoJiSNq+LVY4jyDEFbvTAL7MqyXorMV7Tqlj+/3d7RXfhW9lR/SHb1Z3jcHvz4ZzYMzKWeJEjz+Y0NwIdFDhcPmUOYtEDmuRRrgYiSQ== sebstian.wagner@it-novum.com", "key_options": null, "keyfile": "/home/swagner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "swagner", "validate_certs": true} 2019-12-05T04:58:04.705 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286', u'name': u'sidharthanup'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC1OVCo0wMITAp8PNUqsTRPFz0SuBL0r1RzKFJUoxQm7gsP9qttZUQRlfiuZ7YRzkuGLOEBngPuIcV995o0lZ0+LVHA8jxWEJamb6sM9YgnonbGKDHfl4lT8LJd4YtjcWnMGvUUdPiPgIkfrpxXtx+RzJ7eCTmgRjAJziZrRh3AqXuKty+s9MDnmK8EQ2smpLdkMtZG5W3U2PiVaGgq2FIIOAoqr58xGqFgZ4tIpvWFpNxVsDZXIg2NdC+ouuKQvuJRWq9eERRHwq26CR+kDFNZeYSyhYKxNBqauLCsAoXNFCJzrdX5Oi9Ox4lJTsEtYX+SMziVipQS54F7MnRAUccukQ86IbXz8C5Ok2EzNo6smwRZ6UPKRvebWjIvfbeR/iwPRnrWcNgjGcaTR6yhlRJh8pBkW84nyY/bJa4WXsoNLgutdsYkjl18KKhbBldjdf0ovfL7FVgqYfAo/wpQYJJeMebP5QAczxvZIegjHRrAvbs/nbQTmuAkes3WL1hhBjSTwyIbWCzAuBJk8MNTmeZkVBvub7XoIyIySppfoJC/1a+ZsxGkzAlgrIxXPYrrBzTtXd/9YRiIgCYx3UtfdbPkE6hT6GblH6ST6QESfWyyvfGxwlOmu+F+icRyVVJbgM9T/PD2W6I04u2/1mcfoHwaQGdwvbbBkbjWzm+zwNSisQ== sanupkri@redhat.com", "key_options": null, "keyfile": "/home/sidharthanup/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sidharthanup", "validate_certs": true} 2019-12-05T04:58:04.772 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e', u'name': u'yuvalif'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDzYzqIDimPjMOUOUz1A122eV52fl3mgGgNBRYRlOWl/lOxM5oOat5FPkJlnXTVyQFl0MlnYgIv9fTnjHVTM0wFmFUItlUl151ZmK5XExiGpGVsqG3UWNyKJxZpk9Uh2a0T/5CKzzwXnv2fJMNxzfqgopYDrgq4TFEZcdvgWJ39dcRTXST/rGJQ8wOq9z7UNzKU5VOGxvgi6f/nGevGIHrSaTcY3Uu8BW8kXUrf9vfl+ZSxVjuNHXF/VmrJH5Ne8mVu7DzPMrl8ZZjxsfJKT/eQqF9kjqht5586OSPKIwg/LEOyYj7Wyyf3QbDAIp5bCygQfmBBl+kIxphJYq3EwUZT ylifshit@dhcp-4-102.tlv.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFgwQYD9amVSYq6JzjHyGmKBJATQCM9xsf6QnF3yb0Lvry2nzY9F61evVXQ95TmF/AqasRx5p2WansIvegG5ag5mytvRwFXtHzPs4o/J+XjA9o4+A5l6j9cOFLIDD/TwrwlUjgMa2PhqUztSpI1op906391RLrJAyflfDghJxeVl6dJab6TEsslV0W+yaCS3BBxMXXrCbxCB5Ms3KCRKDz0DqbhFc+Nw05L1O1pC98HBFRIp1SaXlMcT3++CRQ3wugpOMkDMNLbycl+vdMBFY3UZ28rRowS3Ta7V5EARZVVH7KxU5cL+8Hr/8DE1/ITOmrluaEZ1KwYgoxZbZ3w7Ptw0FK6aiCZ49YtqKT1WakoVfO8m7pCdxkqUOA91loMQtxvCwZ/Vfoxaut16gklmPz8N11xvo1H5XB8/ZrvK5yKdl3JDl+wwtfE/fwo4E/fte+rPRisoWBRsmsx8EGxs75sXnb6n3VjZ65vYEKsNMLg+5EAaevWLLeqF0a42UWGJrHdb4VKX/OAplHrOk7kA/VwArow8f9n+iaRp2hgDU9OhQiFVRhgqwahlmCFFXkcP4/4WlxMCuZrsaNMZgMjVhfsp7cskKOezSfZZey2MB2fREMS1wg0j40CdYRn4jTNEBr/LVszsoM7t7AwXUpnqXeYIe+gePx1PJb5XSzSPZ9Pw== yuvalif@yahoo.com", "key_options": null, "keyfile": "/home/yuvalif/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuvalif", "validate_certs": true} 2019-12-05T04:58:04.883 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31', u'name': u'varsha'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCkeYXWjU7wVdODC32U40QRFgFH7ZEuEsOeWWaKx3Y5EBuAcwkTHiF9Bq/2mrDu+c9W46S8L38Ogc9GZ0DcCXtN/Z4z8iKOIIi5U1GzvwdbKuiAPCd8bdYP0vo0MOIRgylJo5hcnB5BCwCXRN1hmci1ES5UbSeQg0rydMiyYXHZ64AT47+qVUpn/mVP6PVw00pPqt16l9YxhbPFG74Y/WnY4FyrqXGQJfv0tcP7+UR7GQdFiI4QQV3ZUSRHRLWGcmoiiOmGimLhtzEmP4Jzy4gF4wg9q8jyX54NjfxDRZs+YuaoqqWKz/HSz5ic7W2RGKHrRby2N9wNu19nmC1NzIn3 varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCW58OUR3pniaYgvr9xAExeNm4eMxgaJfFbEfEYPztTWv/xxqqihBYCeFltfgWbu7ERTy+hVNo1PJWyyczjl5Ivc3FdRmhykosygs7X74PPh02c7Em4MsSTr7SSPNN20rkJbWqiYMMOBbqbiZ7JXiSUNVilLHgZqZBEQc5Rf5I1PyBLHHJ2ZruPKnDPHCRLdwJ+MdXy2haKmGrQLhtzIKaDzh8VkfojL454k0aHCIh34Z0oBdP9757mNuYJUVEkKZF5t2oML84lWGMgwZxUWN2yO9ALTti4tNH1fXA6NipS9ztE+3hEJe1fWXEoUQDX9JNu4qEHTeCz0secQMkyZVDZSQk8yDUQxQ76yiett37sQB1WQR/3fclcahmqQ4dUS7M3Onyz6k6zia896PBK6G48TBUbFa1rw5Hx7WfNZluDMkv4YxkQvfz7XVzZx2/lhs45Kjh1UR9n7LaW/NID98vokcvASw1N6Bk/6pOd/45iPyEJemwovnQLs6ecwlMIikM= varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDcQ8xMjwVf+KiFIQARcbFT+wg4NHysBRrFxk3Bx5X2k2hGKcEYR2tY/ApHZYfJ6me3LUvJEXg2c/6xa6ibVTUp/90l4y0cF6RmF5uB45J0xuBYoFJd8xJkVrhDIlZix4aFfb7TrmJEqjleBrkTwam7Q9aE3lss16nVOxU4hiViYiCANJ45wobK6tz7y5btfzrHE/iYpwniDvzLwK7EpkEK336VVd6f9wV0vG+ErEFxnNVxheXLLm05LuvdVQvFqJcTFrc7/GZ54yKg5IgwVwXqSAyOXIhgduUqk+oN1ciA7jnSqdiOKQPS/VmYkbhw2YP85l8wGVreIYUmOG19QEyVLB82oB9KS1LezM2KCH61XW22+85dHf4PvuL9DULhGt6wBCvpO10OHxw0JvHXbEgBPhiyd9j2yJmcQozbRPth/sMUvLKNXOGabZhhkPBYL7xZICHc+UwxrXs0h/j9jOjLyHzBAWMomF2lzgbvtzjcbxJ1a5oclJ6IW8yHJllkOh0= varsha@localhost.lo 2019-12-05T04:58:04.884 INFO:teuthology.task.ansible.out:caldomain", "key_options": null, "keyfile": "/home/varsha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "varsha", "validate_certs": true} 2019-12-05T04:58:04.930 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852', u'name': u'kkeithle'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXa9uTf3q+7tGnHzSxv0WbgpRKrBIYjAZ5x3V74vQUSD1aAbmyvOqgPnts7XttLYImjjRqiisyC03L3KgR4b3SOeyL3KFxrk3KkHAJ/AJH+Y/UafyDhHNl3gRNg0B9rjsa7h2dSlcrFKnAX0pDY1HapuNPrzYWrXsEbo7C6wteFh7PyPmTkHu0NzIIPC4FcLl8SGaww/HeBh8v91IymgFSgXG0jNC063FEExWdti54yNLGh3gpW0FytsiRN9eEOKUOgcveT1tejz9ORv8evO4Y9SCQ/lsyOKQKNcG5C7J8nsbwSjqQ2cf6i2aYFZ0Oq1UXuhJ8APJCoiKgLJy7AiYf kkeithle@dhcp-100-19-89.bos.redhat.com", "key_options": null, "keyfile": "/home/kkeithle/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kkeithle", "validate_certs": true} 2019-12-05T04:58:04.995 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n', u'name': u'sjust'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6RDgqwA0GrvE6KqRaGCgYnFhnxdi9ehjIWvfaMa1oq0CqYxUAee2bNbp/X2Bd18Vqk8iH9fpE0+qrLgAH/8BpD+MtgGikCFMORvazeUtOCF/OCDO0AMg1jvl8aTdvrQU8ZTDPXa65Avpkc9vwugcoplnwrW7lKx++XqCWsSlfHgEdyMV3wrt/ZDFwI2AS2bZBpWM6ViG1KSjXjgTi/T3hSZPY+buYvWjzoEjQO0946iI3iiKKCGohEpCUnG02oJXcS0vvgl8dWE71451lbainEKJ4hbDhfDmy57s6VXfVUANsczwiPWiSC/ZmMlDaCCBeKHO0Gz0xXaDXd9j4SA7P sam@Pondermatic\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCYylAUS12hF6XndDQ3CkrUddKmXNSd8pOlHwO+pWaDTJPKxwtPL6sM3aOtaAPLrGkv1zD2k3iZaxmDwfPYGaXOyb0KS5ZCddtOyNXj5JLrjxh2niqFPP8SIkdHLOP+0Hv1/tB0Jz75RV+tNIT1SJTm7OZ8wZNy0qYycITSooJ14iEgyZ7rOa3EEzbtIVpHS4+YqIuIJdV2oV5egCrJyRv4sPUrS8ERXD4QZSOlHE+W4Tf+WVrEfsHOdDLRlAXJUzhuqgWtVouiYcW5ex294z28eA+12lKaPQpE0cft/NJ6FLs41yqh4ZN588bl8Ada+yTXEaHWioQEEJMlca2jIgrP sjust@rex004.front.sepia.ceph.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC08RrDQ2M7Zjcpx7RL9e+tjv48kBgDNyTenX1QdxW22JA2JR3D3ThA/ej9HtiPWcIVz16oZ4tzrCEVwaCMfb 2019-12-05T04:58:05.005 INFO:teuthology.task.ansible.out:smXq2z08ZXYqwiBuoXJVshU7lIKVqAtadT4F6Q9eQvxEo4zWdBib1W0/cqNLlYzU/DuwlUqYnWBDLGxc4fD5Kxrz366oWUDB/ULCpl77TMqfTKlStCgGXot0zLqiscqzq1W7XsuA7A5vQw/JTq7MvIME+SkUuGWX0WKwTMIRZKNWBGmlf51bNru3GmmAnKBUFsJv9H4G0FPqGY/aH02rr5dAgTTmfKt+XbWRfMTMpjJqVRGX03+1mQNoYQ3kNDz0wl sjust@office\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA5UMPySf/h9r2/OryRXrO9Qq9k5J9Dn9nvJYyaYCE82H/qkBJGboAJgyIXfbsGYO5uPM25d3fGRH4magzAQm2hGWNbbt1YWn9HaIyKCCXo9JX600/dv3OZbHyGGmgXPa4Anl16j1c2vxRRt2+ArjlyyhP7cAcRkilUGdagK9u6ALrwnz9hhXYV+BmXrxMm7zZCmdWHcWz+YxoA65es4+w+aLvVZWmUCZLF8uZrcpPSw+oK3SfZE8JPRc5vWulXbWEEfIoHb9AszPIpSyzbfPWXelmc3XlG0KD66t7w+AnNlQPt3JVePRCfFHPYeTj9B0k8sxPyUqTQIl5pu6GppRMOQ== sam@Deepthought", "key_options": null, "keyfile": "/home/sjust/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sjust", "validate_certs": true} 2019-12-05T04:58:05.088 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130', u'name': u'emmericp'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBYa5uEOlFv1s9K9D1SbcaApLwX3LBwASSnRXDHI7MtU paul.emmerich@croit.io", "key_options": null, "keyfile": "/home/emmericp/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "emmericp", "validate_certs": true} 2019-12-05T04:58:05.157 INFO:teuthology.task.ansible.out:changed: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'deepika@thinkpad HXRUSerxILmV0ejAD/Zt1g 29b44e741567e7a2e7b664d6533247d79851fc302051c9ba34103c63a2fcfacc', u'name': u'ideepika'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ideepika", "ovpn": "deepika@thinkpad HXRUSerxILmV0ejAD/Zt1g 29b44e741567e7a2e7b664d6533247d79851fc302051c9ba34103c63a2fcfacc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCz1UnJy64hXJKl1OsqRPlZ84m/fyHE8/Hqimy5s8sWcvRs2tL5oK30zFataqv1BC7r46+9t1eGUIRAuBg6QPuJnJCnPzS8yTVJ6DqSDGE2NAnpcMK5fSb+qgQ2adHFkjzZ4eKZbnDJMjYBtpUEp1Ve2ncdq5YCy6hGLIstkGq331ULBc/Lm3ZHBwy5I51S8eldkuP975otBYSPZaaN25QDncZzrW36zb3VwoGUAANEfHC1kLgTTFjx0bdsAGYCduXzyyXls8qSv9gbzcRzZZoH5B7tj8VDBoHszPnqGjGqwtWaelifYE/Svp+8fJKLlzRUM+XGNZoNcRl1/1XVhYV/ deepika@deepika", "key_options": null, "keyfile": "/home/ideepika/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ideepika", "validate_certs": true} 2019-12-05T04:58:05.288 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0', u'name': u'mchangir'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCj28V9sMybkAzhkiaNiDDr8TfNVyr5DNdbOqszdGEVLy1Rxh/UYnxUclsR9n8lt9VnCft3M4NDTnVwlxOHE8+0oolJE9HP+DQomsgj3abP04EGJyOA0X8E7eIFQNLRbMFCD0yX/SNNlYPsS5fPTFhIa/gbBPmDBE07kULts66Tvc0Yo/2yIfwV1DV34bcWWVGi01XybIHBl9qbvlfBKAUrFEcv3ZdpHNhuZw6IkZY8xoYxo8iF+toLYeGX+gFxqCZUUOO+NBVmb0hVL1LI6bU66IrMOQDkFurqpvoraBTJqwZgXvMPceFUkcVIH4Nm7cK1X2WyMtnSw2XBpLMT8C4V mchangir@indraprastha", "key_options": null, "keyfile": "/home/mchangir/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mchangir", "validate_certs": true} 2019-12-05T04:58:05.322 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46', u'name': u'gsalomon'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDC0h+TpaUkm2V6Hbf7aPCTys/nsm51CmY9s+bdwEXAiSAWpq/7Xi2LlN0mZgsrAYBHbr5L+Pz8x0YdigvWysUK6U9XRSPM28cbWmh8F25re9HEk8ARESkMZKTv/hKFs1v7kTHbT9hNNynbQGQOqEaazjr0shcli+Byvop27riX3QDDSsd5xfCjjyDz9fiA9YESlF0kbXyFUKPsoaBhRv9FUhtH6DxlygxtxTsKyKgeoiKw2sCfeUMckcJKnGCMtkXW4U8h7NL/fPZFyBaMDRi/TtoBKaVGLaWqF7RuukfjoMCwhuIDmuTG3WhanH4HJrfNenPZkpZUBhzWwZcleao8SD3mXXfZ9gcZb8QB2ymEtTqrAEyDoNWO4q8ljNP3WJXIgF0h0Sjmx/CV/i0ArEYlBMwBD7DFVnPJqCqSyypuQdDGsDNQZxaRCiYg7nIslFOpDr+0PLp78m+CHyYbSFLTvVVJoJ1k0YhMnDPHM29nt7fMGx6Qrigwnuk0yKg1bfgJ/t0TOqEw0bqGKIvHnyoP/n0ObazEsiPl5pRr2ehs/Rqf35Iwhc1i02Hhz/3Sqwc1eAgqm4/OVghxqUJGpRocua5Gf/vO32wwoyGxh2sl7y6K2pNDA3glhKYbZeIh1fjnaTjikgleqIQkCBxbTeAE7n5+T6TcFtnII6RSxYv3tQ== gal.salomon@gmail.com", "key_options": null, "keyfile": "/home/gsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gsalomon", "validate_certs": true} 2019-12-05T04:58:05.453 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286', u'name': u'sidharthanup'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC1OVCo0wMITAp8PNUqsTRPFz0SuBL0r1RzKFJUoxQm7gsP9qttZUQRlfiuZ7YRzkuGLOEBngPuIcV995o0lZ0+LVHA8jxWEJamb6sM9YgnonbGKDHfl4lT8LJd4YtjcWnMGvUUdPiPgIkfrpxXtx+RzJ7eCTmgRjAJziZrRh3AqXuKty+s9MDnmK8EQ2smpLdkMtZG5W3U2PiVaGgq2FIIOAoqr58xGqFgZ4tIpvWFpNxVsDZXIg2NdC+ouuKQvuJRWq9eERRHwq26CR+kDFNZeYSyhYKxNBqauLCsAoXNFCJzrdX5Oi9Ox4lJTsEtYX+SMziVipQS54F7MnRAUccukQ86IbXz8C5Ok2EzNo6smwRZ6UPKRvebWjIvfbeR/iwPRnrWcNgjGcaTR6yhlRJh8pBkW84nyY/bJa4WXsoNLgutdsYkjl18KKhbBldjdf0ovfL7FVgqYfAo/wpQYJJeMebP5QAczxvZIegjHRrAvbs/nbQTmuAkes3WL1hhBjSTwyIbWCzAuBJk8MNTmeZkVBvub7XoIyIySppfoJC/1a+ZsxGkzAlgrIxXPYrrBzTtXd/9YRiIgCYx3UtfdbPkE6hT6GblH6ST6QESfWyyvfGxwlOmu+F+icRyVVJbgM9T/PD2W6I04u2/1mcfoHwaQGdwvbbBkbjWzm+zwNSisQ== sanupkri@redhat.com", "key_options": null, "keyfile": "/home/sidharthanup/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sidharthanup", "validate_certs": true} 2019-12-05T04:58:05.474 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de', u'name': u'soumyakoduri'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1q3k4RvLLCaG874GDJjja4eYqFwrMjsTU+aRzambaVuCmAJT3KQMGuwMjTuCIN7R1Fqr+/dxUClamsfsGqPMe0ASHT9X/0AyNgEkr6na1Fvk9NAwAFelwtiRIHhclzYf6KY8QHZN/eBy43prVKDXd30b15gwQyVqU0/xmny0GrYCN24k3N/R3divt2GVqptYgEwS2KEyFT5EMnJWjdIVG9mBOHrgec2VF/rc6w3BZ0s56iKdpOeCgk5ISYTcom+Xrf+fSDHirlviZv05tBP8ESmampUT/EnwF8Mdaln5XJvZPhrq2JpUisi9fVeR+zlzg6gRGubvokdA3AERrhCQ9 skoduri@rgw_vm", "key_options": null, "keyfile": "/home/soumyakoduri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "soumyakoduri", "validate_certs": true} 2019-12-05T04:58:05.618 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31', u'name': u'varsha'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCkeYXWjU7wVdODC32U40QRFgFH7ZEuEsOeWWaKx3Y5EBuAcwkTHiF9Bq/2mrDu+c9W46S8L38Ogc9GZ0DcCXtN/Z4z8iKOIIi5U1GzvwdbKuiAPCd8bdYP0vo0MOIRgylJo5hcnB5BCwCXRN1hmci1ES5UbSeQg0rydMiyYXHZ64AT47+qVUpn/mVP6PVw00pPqt16l9YxhbPFG74Y/WnY4FyrqXGQJfv0tcP7+UR7GQdFiI4QQV3ZUSRHRLWGcmoiiOmGimLhtzEmP4Jzy4gF4wg9q8jyX54NjfxDRZs+YuaoqqWKz/HSz5ic7W2RGKHrRby2N9wNu19nmC1NzIn3 varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCW58OUR3pniaYgvr9xAExeNm4eMxgaJfFbEfEYPztTWv/xxqqihBYCeFltfgWbu7ERTy+hVNo1PJWyyczjl5Ivc3FdRmhykosygs7X74PPh02c7Em4MsSTr7SSPNN20rkJbWqiYMMOBbqbiZ7JXiSUNVilLHgZqZBEQc5Rf5I1PyBLHHJ2ZruPKnDPHCRLdwJ+MdXy2haKmGrQLhtzIKaDzh8VkfojL454k0aHCIh34Z0oBdP9757mNuYJUVEkKZF5t2oML84lWGMgwZxUWN2yO9ALTti4tNH1fXA6NipS9ztE+3hEJe1fWXEoUQDX9JNu4qEHTeCz0secQMkyZVDZSQk8yDUQxQ76yiett37sQB1WQR/3fclcahmqQ4dUS7M3Onyz6k6zia896PBK6G48TBUbFa1rw5Hx7WfNZluDMkv4YxkQvfz7XVzZx2/lhs45Kjh1UR9n7LaW/NID98vokcvASw1N6Bk/6pOd/45iPyEJemwovnQLs6ecwlMIikM= varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDcQ8xMjwVf+KiFIQARcbFT+wg4NHysBRrFxk3Bx5X2k2hGKcEYR2tY/ApHZYfJ6me3LUvJEXg2c/6xa6ibVTUp/90l4y0cF6RmF5uB45J0xuBYoFJd8xJkVrhDIlZix4aFfb7TrmJEqjleBrkTwam7Q9aE3lss16nVOxU4hiViYiCANJ45wobK6tz7y5btfzrHE/iYpwniDvzLwK7EpkEK336VVd6f9wV0vG+ErEFxnNVxheXLLm05LuvdVQvFqJcTFrc7/GZ54yKg5IgwVwXqSAyOXIhgduUqk+oN1ciA7jnSqdiOKQPS/VmYkbhw2YP85l8wGVreIYUmOG19QEyVLB82oB9KS1LezM2KCH61XW22+85dHf4PvuL9DULhGt6wBCvpO10OHxw0JvHXbEgBPhiyd9j2yJmcQozbRPth/sMUvLKNXOGabZhhkPBYL7xZICHc+UwxrXs0h/j9jOjLyHzBAWMomF2lzgbvtzjcbxJ1a5oclJ6IW8yHJllkOh0= varsha@localhost.lo 2019-12-05T04:58:05.620 INFO:teuthology.task.ansible.out:caldomain", "key_options": null, "keyfile": "/home/varsha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "varsha", "validate_certs": true} 2019-12-05T04:58:05.630 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a', u'name': u'kyr'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDPeuSTD7kv8yQX6dW+7Vpq1mqrcr8JGubPYa1DpGTV/gE50d5L42UspUN2FXmICDMiV1h4Q9WQAFma3168f1I5I94qFkV+/56szCMoILClm7IW/EV4t8fTxoyOpEa9l4qSU3qgJpDbY04CkiyVJFuQj24Zq3gD/70jYgGpy8RfA0BihSR6F+A0WZLh/fESjqfor3/pFWhUCRnPEqrZsXyuB8lRgDTyGp4s1cHyUqmFMuCPDRMtok0rSVpaRZj2Zj8lboah19UI/L89ptQgEdqsOKzlkxiT+1GdyIY+jpPCM0a3yZz75l3Qiu0k6q7qUV8T0hm1GZiBguvST1KMcwnt kyrylo.shatskyy@suse.com", "key_options": null, "keyfile": "/home/kyr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kyr", "validate_certs": true} 2019-12-05T04:58:05.804 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849', u'name': u'sseshasa'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeERrQ0NVXhuTYzI08rlKRbHZVFXSv1YC7sGq6aCWRbTCv7Ad/t/3OJz3vfcd2TjBMvABfrYCFgTFWIaJBaeA3BXUUU9y9TwnzuxKv4C46WJYoalVGyLEpWpcGbJ5v+leA6Je/+lunenplb7TBKGE/YTgggEXyX7KnunYlv+Jdm33TmAZTcHrLk5QR/6FdRGKmdlLna0dXZc3v04+AI8W+p9ghAv1+e8ENxJDTF8p0MawBiKWcSj2mysOhQ2Cj91qZ0qTnLLeu0CPf4rBO+qjYIshiCuTLhYt2UazlzME4nCCsVqK7Nz9olA/lgvJLRaPxgQQ7iCMmIZilS6B4Y3oL sseshasa@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCo0zniUEXuQ4JJuuRtfnEK8WHqr97AmXEniGPY5U23MTxhMuP8CSkGWkUHdmXw0H1tDKbYgk97M6TspJropNGKljla2H1/3e2T1BOI1L2WfcsOh6VPYeyOj2ssMJeEDdQgd0J3g+4oj2QS+guPlT24UhO0CBhRZ6waVlJMgagDYyDLCLGI4ASoBYx0AOaoyjnEQtRd/gyejzv7jntkHc9YNBPsJ37X6hTYNA/s2ORJE7QMXd6xI60mTBuCdD3JcPYoUSvyCUa7rIrDc70lAcW5XtxewvMBogTZJLoFG/d6Ugij49h4AfwrtoUCtLyJOWJX6PTfomXely8Ym6KpN/3981+m70dhcHPDJ66QCWMd1LAgftVXTIsebA+Bxfg4Vsbrv7j1lJXzmMfsOl+1wVPe+Y9h28GyeIlalW3laURZGV7dhexGZ8YNxPuzxtAlWtfw+kddbjw0j4ODtgVXrj4SDxqmTTYYuMBkrQ3H33sOgPLx8DyA6e4pckb6lOOEvb41Yi5tyP7BIJlYN2PpNVDkSQd/t5Q0di4Ct+3vqLRCkPuJiJAUyaBGT4HM7yQc25+Y0B3sXNON7BjeaBXB7OwKZMp9w1jwbJt4JL/aGOw3HVljVyVgu+pIuQ2ISd9VBcmAJK5EGdWe5Q57h5fJkD+TVrXgeNTaufdclh5vWOH1qQ== sseshasa@teuthology", "key_options": null, "keyfile": "/home/sseshasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sseshasa", "validate_certs": true} ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b 2019-12-05T04:58:05.819 INFO:teuthology.task.ansible.out:19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n', u'name': u'sjust'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6RDgqwA0GrvE6KqRaGCgYnFhnxdi9ehjIWvfaMa1oq0CqYxUAee2bNbp/X2Bd18Vqk8iH9fpE0+qrLgAH/8BpD+MtgGikCFMORvazeUtOCF/OCDO0AMg1jvl8aTdvrQU8ZTDPXa65Avpkc9vwugcoplnwrW7lKx++XqCWsSlfHgEdyMV3wrt/ZDFwI2AS2bZBpWM6ViG1KSjXjgTi/T3hSZPY+buYvWjzoEjQO0946iI3iiKKCGohEpCUnG02oJXcS0vvgl8dWE71451lbainEKJ4hbDhfDmy57s6VXfVUANsczwiPWiSC/ZmMlDaCCBeKHO0Gz0xXaDXd9j4SA7P sam@Pondermatic\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCYylAUS12hF6XndDQ3CkrUddKmXNSd8pOlHwO+pWaDTJPKxwtPL6sM3aOtaAPLrGkv1zD2k3iZaxmDwfPYGaXOyb0KS5ZCddtOyNXj5JLrjxh2niqFPP8SIkdHLOP+0Hv1/tB0Jz75RV+tNIT1SJTm7OZ8wZNy0qYycITSooJ14iEgyZ7rOa3EEzbtIVpHS4+YqIuIJdV2oV5egCrJyRv4sPUrS8ERXD4QZSOlHE+W4Tf+WVrEfsHOdDLRlAXJUzhuqgWtVouiYcW5ex294z28eA+12lKaPQpE0cft/NJ6FLs41yqh4ZN588bl8Ada+yTXEaHWioQEEJMlca2jIgrP sjust@rex004.front.sepia.ceph.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC08RrDQ2M7Zjcpx7RL9e+tjv48kBgDNyTenX1QdxW22JA2JR3D3ThA/ej9HtiPWcIVz16oZ4tzrCEVwaCMfbsmXq2z08ZXYqwiBuoXJVshU7lIKVqAtadT4F6Q9eQvxEo4zWdBib1W0/cqNLlYzU/DuwlUqYnWBDLGxc4fD5Kxrz366oWUDB/ULCpl77TMqfTKlStCgGXot0zLqiscqzq1W7XsuA7A5vQw/JTq7MvIME+SkUuGWX0WKwTMIRZKNWBGmlf51bNru3GmmAnKBUFsJv9H4G0FPqGY/aH02rr5dAgTTmfK 2019-12-05T04:58:05.821 INFO:teuthology.task.ansible.out:t+XbWRfMTMpjJqVRGX03+1mQNoYQ3kNDz0wl sjust@office\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA5UMPySf/h9r2/OryRXrO9Qq9k5J9Dn9nvJYyaYCE82H/qkBJGboAJgyIXfbsGYO5uPM25d3fGRH4magzAQm2hGWNbbt1YWn9HaIyKCCXo9JX600/dv3OZbHyGGmgXPa4Anl16j1c2vxRRt2+ArjlyyhP7cAcRkilUGdagK9u6ALrwnz9hhXYV+BmXrxMm7zZCmdWHcWz+YxoA65es4+w+aLvVZWmUCZLF8uZrcpPSw+oK3SfZE8JPRc5vWulXbWEEfIoHb9AszPIpSyzbfPWXelmc3XlG0KD66t7w+AnNlQPt3JVePRCfFHPYeTj9B0k8sxPyUqTQIl5pu6GppRMOQ== sam@Deepthought", "key_options": null, "keyfile": "/home/sjust/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sjust", "validate_certs": true} 2019-12-05T04:58:05.931 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189', u'name': u'rfriedma'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIC6uqbIg9XaTlDZd7KCX7FXSzfGFMEWXhx42ktz3ehOPLviA/yNpVd+LmYHSgzd5Szd3mMxleisiE+JdSq/q5YyPAF6eD6+wfynkpMv5aWNHGI0yVfKbCwdOX8bY6yqhyVwMDo14uoFvdDcM4Lj+vz8F75VJHUU5ihJNOW24EllobTLzR4XI6AcM+mjbXRnBLLVoBSuDaU2D0kWqMn8rDCJKdUJnoiy9oxIok8eMLfGkGsL55oOx9BflMSz/0jM9X5GLQahYiECbXjvCIarCkUFE8txNoXM9Djmg8QqhAQDhY0hzJTauFJJaD8Ax1uFQlXONed7tv7a7Tb3dH+/jb rfriedma@dhcp-4-239.tlv.redhat.com", "key_options": null, "keyfile": "/home/rfriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rfriedma", "validate_certs": true} 2019-12-05T04:58:05.977 INFO:teuthology.task.ansible.out:changed: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'deepika@thinkpad HXRUSerxILmV0ejAD/Zt1g 29b44e741567e7a2e7b664d6533247d79851fc302051c9ba34103c63a2fcfacc', u'name': u'ideepika'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ideepika", "ovpn": "deepika@thinkpad HXRUSerxILmV0ejAD/Zt1g 29b44e741567e7a2e7b664d6533247d79851fc302051c9ba34103c63a2fcfacc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCz1UnJy64hXJKl1OsqRPlZ84m/fyHE8/Hqimy5s8sWcvRs2tL5oK30zFataqv1BC7r46+9t1eGUIRAuBg6QPuJnJCnPzS8yTVJ6DqSDGE2NAnpcMK5fSb+qgQ2adHFkjzZ4eKZbnDJMjYBtpUEp1Ve2ncdq5YCy6hGLIstkGq331ULBc/Lm3ZHBwy5I51S8eldkuP975otBYSPZaaN25QDncZzrW36zb3VwoGUAANEfHC1kLgTTFjx0bdsAGYCduXzyyXls8qSv9gbzcRzZZoH5B7tj8VDBoHszPnqGjGqwtWaelifYE/Svp+8fJKLlzRUM+XGNZoNcRl1/1XVhYV/ deepika@deepika", "key_options": null, "keyfile": "/home/ideepika/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ideepika", "validate_certs": true} 2019-12-05T04:58:06.073 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b', u'name': u'lmb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIXHA8BwV4OnqlaHHyYofsp5X+wanose+aneJVx2Or05PcWA9c7xFEOerkvEn9LvivSRd9G9o/S8SvnGH91tG2bbOAidVZ69FGul9OKa/BCJYUP0vVBX09/Td3n7+R2lVlvu5XiyC5SUzCjaQEZLOpjq23rqaqdCDXE6wFnxfpkmcsccbG/xUiOEBeBGI92lntdx99Sn6CxWWA27lMMaYkCRvX6ziRZEm6O44N2nMm+qYm0jmMyGv1iyQkJxtt0YaLPp+X9t2trUEzEjgBCXqqzEzBjB7T4rbzhAuvU3NRHiAoWXSnrY5AQOTTvC/wpaZV/gF2NDNThjMcpcEdNstleOctXtN4XTTJQewbG+t7I9knR00TPpsoUvUtpbcoFOluGgWTDTFa6GCmDPppx23nZgulOniZUVTI2CbScSpc8sfp1JE7Q6vLECY9lJ+GpVCbDAbIEE9WHXSkJ/6L9yf+ZoHbpqCkEFDcWLfz3B5NFoJ5gne8B8uzd6ptKVtJrP8pMvoOyydyvQlmkgh8tNDsOnp6kaUNG0NHmtfGlCTkbM7oKsKbf87Cv1HgYuZl/HP7go5sOX02LfBfbvWrawZzOaZ3dewEq74w5hVMKEHrmJPstIpcHSxZpq33f2j4Z/uKW9tJat6rgSWK1I+T3tCd++76apxbOYl5TDOaoxCyjQ== lars@marowsky-bree.de", "key_options": null, "keyfile": "/home/lmb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lmb", "validate_certs": true} 2019-12-05T04:58:06.159 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46', u'name': u'gsalomon'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDC0h+TpaUkm2V6Hbf7aPCTys/nsm51CmY9s+bdwEXAiSAWpq/7Xi2LlN0mZgsrAYBHbr5L+Pz8x0YdigvWysUK6U9XRSPM28cbWmh8F25re9HEk8ARESkMZKTv/hKFs1v7kTHbT9hNNynbQGQOqEaazjr0shcli+Byvop27riX3QDDSsd5xfCjjyDz9fiA9YESlF0kbXyFUKPsoaBhRv9FUhtH6DxlygxtxTsKyKgeoiKw2sCfeUMckcJKnGCMtkXW4U8h7NL/fPZFyBaMDRi/TtoBKaVGLaWqF7RuukfjoMCwhuIDmuTG3WhanH4HJrfNenPZkpZUBhzWwZcleao8SD3mXXfZ9gcZb8QB2ymEtTqrAEyDoNWO4q8ljNP3WJXIgF0h0Sjmx/CV/i0ArEYlBMwBD7DFVnPJqCqSyypuQdDGsDNQZxaRCiYg7nIslFOpDr+0PLp78m+CHyYbSFLTvVVJoJ1k0YhMnDPHM29nt7fMGx6Qrigwnuk0yKg1bfgJ/t0TOqEw0bqGKIvHnyoP/n0ObazEsiPl5pRr2ehs/Rqf35Iwhc1i02Hhz/3Sqwc1eAgqm4/OVghxqUJGpRocua5Gf/vO32wwoyGxh2sl7y6K2pNDA3glhKYbZeIh1fjnaTjikgleqIQkCBxbTeAE7n5+T6TcFtnII6RSxYv3tQ== gal.salomon@gmail.com", "key_options": null, "keyfile": "/home/gsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gsalomon", "validate_certs": true} 2019-12-05T04:58:06.238 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc', u'name': u'ksirivad'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMDUcToIhs8tiIozgPudiYyD2upEX+VvsiJRti7zrA/1HPIDARYt2+xyWCASm5x5vdohRM7wnBLUlbBVnvr6ZUQvfShNDsS1Uu7hEfYF38MtW+dVwvGlm14J4m1gbjzqVlZvO6bg6rfGo7Qer0bKUlxtqbisQ/kCg/lVWR04CNxdRtZxEpaQ21y4C5jAlrpFTzxQSuiCLEyY3EAKdZUaZElFEcXlna0ysCP8inlzhF402PkSdl9DkkPpg3IR2dUEGqsblWSdLVk19EFavjtGAAnsvaBWWm2rCLNC5RmhGQaS/Z52Aes1nBuT4cI6qLjbEMqAzLwxiOR7l4IPMpK5EcuRK+FaqQG0wMAjxPc95CmxDzi58ZRJ5ERV8vgps73F1AJXWU9Lb1P77BTsLXqiwko2mvPf6e0LwCxQuxYJWc+xvO7Ji011B+nm5eLCCC/LcDtTPjkimTh7nkvUNi/yTKCXi9KwLISAHHd335KnfrL28rU3GahMXs4NjOxuN8eKM= ksirivad@ksirivad", "key_options": null, "keyfile": "/home/ksirivad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ksirivad", "validate_certs": true} 2019-12-05T04:58:06.309 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de', u'name': u'soumyakoduri'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1q3k4RvLLCaG874GDJjja4eYqFwrMjsTU+aRzambaVuCmAJT3KQMGuwMjTuCIN7R1Fqr+/dxUClamsfsGqPMe0ASHT9X/0AyNgEkr6na1Fvk9NAwAFelwtiRIHhclzYf6KY8QHZN/eBy43prVKDXd30b15gwQyVqU0/xmny0GrYCN24k3N/R3divt2GVqptYgEwS2KEyFT5EMnJWjdIVG9mBOHrgec2VF/rc6w3BZ0s56iKdpOeCgk5ISYTcom+Xrf+fSDHirlviZv05tBP8ESmampUT/EnwF8Mdaln5XJvZPhrq2JpUisi9fVeR+zlzg6gRGubvokdA3AERrhCQ9 skoduri@rgw_vm", "key_options": null, "keyfile": "/home/soumyakoduri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "soumyakoduri", "validate_certs": true} 2019-12-05T04:58:06.387 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4', u'name': u'ofriedma'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0POsA+0NyxecLo3Y821G6W3+jJPKFMuLSIl4FSLi25vuyuM5+mFGmCNWv8gg93eQhG9Y3SOygqmkaxzxDtBS/V6VNC2f8hZsVw9B+HbnfggvCNQSgjJ5rhvf6IHfx/aHq4IHF+stqdcoWhwR/68wIhGNqtP7Nhqnzk2aN1kG9bU2iVk4CH8XGIwqwET3aWNuCMcHi2AhrvDBpZu7bzo/6ARUJDqEfTEC/XIkzv8/mEkJ5igcycARVy6dChuCwfZc8rhfwQpekwx1Y9/WBFdNQlYkN4fVe8yIqT5VsQ3AcRv21GJk8L7mwB4hF9dVAi4p1cSyxatR3kDifsFc8VQk9 toor@localhost.localdomain", "key_options": null, "keyfile": "/home/ofriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ofriedma", "validate_certs": true} 2019-12-05T04:58:06.541 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52', u'name': u'h4iresis'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDc/8sWrMq+bqKBPmDPplUgdMoY2SQL+57jRSh27PAoYW9rNVR/UgK7EA9oxFLfoi9wJ9+VKU4opeoohOXLWQXXatLvfXjGE3fRYIonQfwJqKrasqQJTGWzRZ9t1/2Cb0VHkoAvipCJzWmgB46X6q73XW5Td1H3b+AINd6E4Rsm1QjBOhk0DMjCdfwrVUZIfGdsEo+5u/AwioHwH+NDJvXTWT8ZigeZBNHJSY/EQn1+15ut0QB2rvXkJasvvDlVbEsPQtM4C3VX1ronmAnKMvCp1O4EtAXaoKSOx7XEN++UAvVn7czDCWk/UFgpzZOqTcQJwg82XtS4sZP23CpNe0G9 h4iresis@fedora", "key_options": null, "keyfile": "/home/h4iresis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "h4iresis", "validate_certs": true} 2019-12-05T04:58:06.591 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a', u'name': u'kyr'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDPeuSTD7kv8yQX6dW+7Vpq1mqrcr8JGubPYa1DpGTV/gE50d5L42UspUN2FXmICDMiV1h4Q9WQAFma3168f1I5I94qFkV+/56szCMoILClm7IW/EV4t8fTxoyOpEa9l4qSU3qgJpDbY04CkiyVJFuQj24Zq3gD/70jYgGpy8RfA0BihSR6F+A0WZLh/fESjqfor3/pFWhUCRnPEqrZsXyuB8lRgDTyGp4s1cHyUqmFMuCPDRMtok0rSVpaRZj2Zj8lboah19UI/L89ptQgEdqsOKzlkxiT+1GdyIY+jpPCM0a3yZz75l3Qiu0k6q7qUV8T0hm1GZiBguvST1KMcwnt kyrylo.shatskyy@suse.com", "key_options": null, "keyfile": "/home/kyr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kyr", "validate_certs": true} 2019-12-05T04:58:06.678 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5', u'name': u'shyukri'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCqhWVTosH12864VGY5S3/k0yKc59GbV0m2WxD28gAf+uvJ1/y8gekD2wDzBv0DWqEekYLV7L68gpapDXyGosQeZ6n5OO/NZFZmW+Xq1P7P1QHJil6kVpwk927lzX440pwNf3+PRvTbLzlbuVFWJRpiykv+juJyIFjzXLJRF67wB2qgZVm5cRs/GOi4fnvTaou/Ae51mvsO8M3JL5cphApFKD/AO/X+3pHBhLjvaW33cYsp0Z4Rce1nm8wxSmvuJRF+HAfjBlknhDl5ZaJGA96RYA+g9WbAXimFprg4xeiRp7liPExQwtVgackJweoC4MahC1glmjC3eOh9dvrrxLnALOLMOkyCSK/tOF/iq2PJaLViObanPI8mnNkMLbdJyzUBf8hRGdSLcw98qxaf0sSZ+vz4VQS6e9CwX7ikkNNsTeYXjnJ+1yU3mfCW1CXJDr7kBDVG2OTLefCpMY8WBAYDiEaK0RW83cBcFIKfecKYd0qStHRaOsvmyWdFYgBh2vM1UWjro5WXlb+7aJPrUp8ZruapkUXUwhjHA7FwGA4ysE0tsOVVFm0Dw8tvyVVkVNm/9EObG28dWctioZ2+ZEzTWgMJU26Q+wOJ/U4qZyIvpMRUgPZc5nM9J8kcmvNWtb6DV9k/6p8h28kiOxPkxLtPP+3TlBCHjq//mhEnJZ/d8w== shshyukriev@suse.com", "key_options": null, "keyfile": "/home/shyukri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyukri", "validate_certs": true} 2019-12-05T04:58:06.757 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849', u'name': u'sseshasa'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeERrQ0NVXhuTYzI08rlKRbHZVFXSv1YC7sGq6aCWRbTCv7Ad/t/3OJz3vfcd2TjBMvABfrYCFgTFWIaJBaeA3BXUUU9y9TwnzuxKv4C46WJYoalVGyLEpWpcGbJ5v+leA6Je/+lunenplb7TBKGE/YTgggEXyX7KnunYlv+Jdm33TmAZTcHrLk5QR/6FdRGKmdlLna0dXZc3v04+AI8W+p9ghAv1+e8ENxJDTF8p0MawBiKWcSj2mysOhQ2Cj91qZ0qTnLLeu0CPf4rBO+qjYIshiCuTLhYt2UazlzME4nCCsVqK7Nz9olA/lgvJLRaPxgQQ7iCMmIZilS6B4Y3oL sseshasa@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCo0zniUEXuQ4JJuuRtfnEK8WHqr97AmXEniGPY5U23MTxhMuP8CSkGWkUHdmXw0H1tDKbYgk97M6TspJropNGKljla2H1/3e2T1BOI1L2WfcsOh6VPYeyOj2ssMJeEDdQgd0J3g+4oj2QS+guPlT24UhO0CBhRZ6waVlJMgagDYyDLCLGI4ASoBYx0AOaoyjnEQtRd/gyejzv7jntkHc9YNBPsJ37X6hTYNA/s2ORJE7QMXd6xI60mTBuCdD3JcPYoUSvyCUa7rIrDc70lAcW5XtxewvMBogTZJLoFG/d6Ugij49h4AfwrtoUCtLyJOWJX6PTfomXely8Ym6KpN/3981+m70dhcHPDJ66QCWMd1LAgftVXTIsebA+Bxfg4Vsbrv7j1lJXzmMfsOl+1wVPe+Y9h28GyeIlalW3laURZGV7dhexGZ8YNxPuzxtAlWtfw+kddbjw0j4ODtgVXrj4SDxqmTTYYuMBkrQ3H33sOgPLx8DyA6e4pckb6lOOEvb41Yi5tyP7BIJlYN2PpNVDkSQd/t5Q0di4Ct+3vqLRCkPuJiJAUyaBGT4HM7yQc25+Y0B3sXNON7BjeaBXB7OwKZMp9w1jwbJt4JL/aGOw3HVljVyVgu+pIuQ2ISd9VBcmAJK5EGdWe5Q57h5fJkD+TVrXgeNTaufdclh5vWOH1qQ== sseshasa@teuthology", "key_options": null, "keyfile": "/home/sseshasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sseshasa", "validate_certs": true} 2019-12-05T04:58:06.830 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e', u'name': u'scarvalhojr'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxDJtmmcnrbA4AUsmBBZq/M4xNdIbOT+cQQxhUKeCu8F9mht+RznwyaaJbVIlD+UNv9LMhRyzI5oktSalokNltlrtPQMoP52PXH7KFFGnVnexCHAk4nGvEhuZCniO5T+ENm2wFR47lFEQK7lyJJLp5PFiSM7vM1MK6UkMvRmCTRildjaYLyf3EEvBXifbsGmR0tBIXtml6KITCAaT6l0E94055oEGN2G5NhZIvfLcojt5liEkXEHQNdfkUDXM0Go43JZzgsSWDXC/7RYLrAJwz5mq4qCa9YZL/3Spjso37+1oLq7lOQ0TDJUAQXmnW0So/NkzECx4MolVcylhTzmgj sergio@milestones", "key_options": null, "keyfile": "/home/scarvalhojr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "scarvalhojr", "validate_certs": true} 2019-12-05T04:58:06.915 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189', u'name': u'rfriedma'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIC6uqbIg9XaTlDZd7KCX7FXSzfGFMEWXhx42ktz3ehOPLviA/yNpVd+LmYHSgzd5Szd3mMxleisiE+JdSq/q5YyPAF6eD6+wfynkpMv5aWNHGI0yVfKbCwdOX8bY6yqhyVwMDo14uoFvdDcM4Lj+vz8F75VJHUU5ihJNOW24EllobTLzR4XI6AcM+mjbXRnBLLVoBSuDaU2D0kWqMn8rDCJKdUJnoiy9oxIok8eMLfGkGsL55oOx9BflMSz/0jM9X5GLQahYiECbXjvCIarCkUFE8txNoXM9Djmg8QqhAQDhY0hzJTauFJJaD8Ax1uFQlXONed7tv7a7Tb3dH+/jb rfriedma@dhcp-4-239.tlv.redhat.com", "key_options": null, "keyfile": "/home/rfriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rfriedma", "validate_certs": true} 2019-12-05T04:58:06.979 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b', u'name': u'rakesh'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEOigBGMeElb3uHXX36HxLtPJMG+iG/6D7NDFCVR6AyYsmimBQb6SlSIR6SaFULb6HT/IFVk0Jf/z0HoipAP7Xu+wvP/J6LFoMdguxylh1ov8raqxQJrGLl9n64EI+qWINwEM0MKMUnDQ3tmMKBl78LKQVQTYBx1+9cXsT6DE1yhgmV79XIvOOb+v53agSi/kLJF2i9K04kpi1OejXg95Ub2eLMY69Jm8PnLy1Jq8DhC8pz7nrfzbKtOC7HbytTAO1XTlb90lVhqC/1yTMAUc/rpJwoSEHc2epGeZWdMsJhqhX2EqMwcV53M3qcFSGreGNJ/Ncd4/JHxz7q0IU411t rakesh@localhost.localdomain", "key_options": null, "keyfile": "/home/rakesh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rakesh", "validate_certs": true} 2019-12-05T04:58:07.065 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b', u'name': u'lmb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIXHA8BwV4OnqlaHHyYofsp5X+wanose+aneJVx2Or05PcWA9c7xFEOerkvEn9LvivSRd9G9o/S8SvnGH91tG2bbOAidVZ69FGul9OKa/BCJYUP0vVBX09/Td3n7+R2lVlvu5XiyC5SUzCjaQEZLOpjq23rqaqdCDXE6wFnxfpkmcsccbG/xUiOEBeBGI92lntdx99Sn6CxWWA27lMMaYkCRvX6ziRZEm6O44N2nMm+qYm0jmMyGv1iyQkJxtt0YaLPp+X9t2trUEzEjgBCXqqzEzBjB7T4rbzhAuvU3NRHiAoWXSnrY5AQOTTvC/wpaZV/gF2NDNThjMcpcEdNstleOctXtN4XTTJQewbG+t7I9knR00TPpsoUvUtpbcoFOluGgWTDTFa6GCmDPppx23nZgulOniZUVTI2CbScSpc8sfp1JE7Q6vLECY9lJ+GpVCbDAbIEE9WHXSkJ/6L9yf+ZoHbpqCkEFDcWLfz3B5NFoJ5gne8B8uzd6ptKVtJrP8pMvoOyydyvQlmkgh8tNDsOnp6kaUNG0NHmtfGlCTkbM7oKsKbf87Cv1HgYuZl/HP7go5sOX02LfBfbvWrawZzOaZ3dewEq74w5hVMKEHrmJPstIpcHSxZpq33f2j4Z/uKW9tJat6rgSWK1I+T3tCd++76apxbOYl5TDOaoxCyjQ== lars@marowsky-bree.de", "key_options": null, "keyfile": "/home/lmb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lmb", "validate_certs": true} 2019-12-05T04:58:07.121 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'ryg@wombat 7WbRbRXPcrQ2++Bww59JKg 1186db58c70662d55cae6d75e54022d1392708d3cf742b10ce5ff9d67dd8486', u'name': u'ryg-'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat 7WbRbRXPcrQ2++Bww59JKg 1186db58c70662d55cae6d75e54022d1392708d3cf742b10ce5ff9d67dd8486"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvRwrsN62CReT0/kOeFEXGA3THhzJ+hwZVpUzo7VMRZeSo6LgNlVGdrQchMHnlwBvuwaVNJlBAkQr+5J5BB3ikrmbcnLHfPutXBBBB6uDfSwwbqjzUY7elTbz9ul6AkegR9K65O44ouD9+WC76D+Tp0maqnP1w0O3h1GVxDHe34LpEwBvZJ+/BSYmBpwRZqeXFRADORPzdtW08PSiRErGHb8j+j5i+br37JfOOh+A/9dHwQw9alBVL2DijwvrPMa7dV5yX00dlMSTL2LLo9IdI4aFPmie8oic0eanFbQDq210ilOR65IGeshKHEKOq0KFFz0nSK9OdO03OkSEN/0d5 ryg@tapir", "key_options": null, "keyfile": "/home/ryg-/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ryg-", "validate_certs": true} 2019-12-05T04:58:07.215 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc', u'name': u'ksirivad'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMDUcToIhs8tiIozgPudiYyD2upEX+VvsiJRti7zrA/1HPIDARYt2+xyWCASm5x5vdohRM7wnBLUlbBVnvr6ZUQvfShNDsS1Uu7hEfYF38MtW+dVwvGlm14J4m1gbjzqVlZvO6bg6rfGo7Qer0bKUlxtqbisQ/kCg/lVWR04CNxdRtZxEpaQ21y4C5jAlrpFTzxQSuiCLEyY3EAKdZUaZElFEcXlna0ysCP8inlzhF402PkSdl9DkkPpg3IR2dUEGqsblWSdLVk19EFavjtGAAnsvaBWWm2rCLNC5RmhGQaS/Z52Aes1nBuT4cI6qLjbEMqAzLwxiOR7l4IPMpK5EcuRK+FaqQG0wMAjxPc95CmxDzi58ZRJ5ERV8vgps73F1AJXWU9Lb1P77BTsLXqiwko2mvPf6e0LwCxQuxYJWc+xvO7Ji011B+nm5eLCCC/LcDtTPjkimTh7nkvUNi/yTKCXi9KwLISAHHd335KnfrL28rU3GahMXs4NjOxuN8eKM= ksirivad@ksirivad", "key_options": null, "keyfile": "/home/ksirivad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ksirivad", "validate_certs": true} 2019-12-05T04:58:07.258 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_key 2019-12-05T04:58:07.263 INFO:teuthology.task.ansible.out:s", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2019-12-05T04:58:07.363 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4', u'name': u'ofriedma'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0POsA+0NyxecLo3Y821G6W3+jJPKFMuLSIl4FSLi25vuyuM5+mFGmCNWv8gg93eQhG9Y3SOygqmkaxzxDtBS/V6VNC2f8hZsVw9B+HbnfggvCNQSgjJ5rhvf6IHfx/aHq4IHF+stqdcoWhwR/68wIhGNqtP7Nhqnzk2aN1kG9bU2iVk4CH8XGIwqwET3aWNuCMcHi2AhrvDBpZu7bzo/6ARUJDqEfTEC/XIkzv8/mEkJ5igcycARVy6dChuCwfZc8rhfwQpekwx1Y9/WBFdNQlYkN4fVe8yIqT5VsQ3AcRv21GJk8L7mwB4hF9dVAi4p1cSyxatR3kDifsFc8VQk9 toor@localhost.localdomain", "key_options": null, "keyfile": "/home/ofriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ofriedma", "validate_certs": true} 2019-12-05T04:58:07.405 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42', u'name': u'zdover'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zdover", "ovpn": "zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlnlBzRdWRqns/9VrYuF9tLILUjeGd9z8Tf/dPZmxAt3nnZqpTvI2xXemQpyt/2lsD0U24EuTc1Y/dBsgqvXtZOw51TZE8RK26hvCCaKGfcKONjvR7goHAmkYPHK9+F6m7fgYkRzA0+ZW4HcWNh/xxhhL2G1EGfz5bzIoGQJHSdzc6VeVIsY7HiYEMTxvxdsgV72Kjq97BdmHm56RBzoH24oN0UzNZS8Syx5SuWj9dA3IyFBPwjLlozSk6U2Ofd7qaXuQKl807eD9g6RhRldzHPS+i0EK3EM6Jvho5Y08k1KqV4gIEwMdzDbD3SH0xq+nUPFKcDTUFNp+tkg10GxqN zac.dover@gmail.com", "key_options": null, "keyfile": "/home/zdover/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zdover", "validate_certs": true} 2019-12-05T04:58:07.517 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52', u'name': u'h4iresis'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDc/8sWrMq+bqKBPmDPplUgdMoY2SQL+57jRSh27PAoYW9rNVR/UgK7EA9oxFLfoi9wJ9+VKU4opeoohOXLWQXXatLvfXjGE3fRYIonQfwJqKrasqQJTGWzRZ9t1/2Cb0VHkoAvipCJzWmgB46X6q73XW5Td1H3b+AINd6E4Rsm1QjBOhk0DMjCdfwrVUZIfGdsEo+5u/AwioHwH+NDJvXTWT8ZigeZBNHJSY/EQn1+15ut0QB2rvXkJasvvDlVbEsPQtM4C3VX1ronmAnKMvCp1O4EtAXaoKSOx7XEN++UAvVn7czDCWk/UFgpzZOqTcQJwg82XtS4sZP23CpNe0G9 h4iresis@fedora", "key_options": null, "keyfile": "/home/h4iresis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "h4iresis", "validate_certs": true} 2019-12-05T04:58:07.543 INFO:teuthology.task.ansible.out:changed: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b', u'name': u'rraja'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rraja", "ovpn": "rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC4zwWf+kMEDgaSOrX7AlD+UCOgf4/fRiorQVJqC8GfMNynfiFVZaThngwIrwtQROfXHbzspi+AcRvDLewY25mXyuxVgOfSbuSIEqiUNvzcVq9IItI1J58xeDBpIktAsoHHS6JENV56fAT6krzRAQLnSa1k2b0pyKuPBzFSeNgiaQMhjrCLAaCi9yLsdKpnWgXhyfrdqvN2r/Ctts/zlTPqtRYdvdIL/p/l+vg7Q8S3OhuAGtimaEq3RE70xadaPbGhsLISMTrbfYXbxw84KKZpNYwO4zmQ2ztcvC9D7A/8ov3+A5Jq42G+pq7ORuix2NoTWtADMiwvNe1HyzS/7cX/BuXLFssRY1bzRvp+x2ExZ7b+bFx+TE/RduRh0eyDNLg7SyL83aeTr2Usa+mPoCAMKed9XtYbRnUu7ox4wmfeAgx6jM9uG5uQfauJ1XpDddrVBG8KYBvSd0BM2cyRdcAXqwxg9AVPZcdE7sbBk44CGVhw+ziZ3e3cLBTO4y44KRQre+1Bhe/Ze8OTQWLFugl/XagFlJE/L6fGUIPuFojmQvlw5dBZwtUURnxsu8yHm3iIsj4y8nR1zHMj+zuMk6rF3ZW0epPAECAdq4gG8X2wv3HTq0vLBkKpi1KMAmntEtcVwF0fzMt5XYqRd4jceFgonAAipJNDknHgi6yaUzBYmQ== rraja@redhat.com", "key_options": null, "keyfile": "/home/rraja/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rraja", "validate_certs": true} 2019-12-05T04:58:07.648 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:07 +0000 (0:00:15.376) 0:00:43.151 ***** 2019-12-05T04:58:07.656 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5', u'name': u'shyukri'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCqhWVTosH12864VGY5S3/k0yKc59GbV0m2WxD28gAf+uvJ1/y8gekD2wDzBv0DWqEekYLV7L68gpapDXyGosQeZ6n5OO/NZFZmW+Xq1P7P1QHJil6kVpwk927lzX440pwNf3+PRvTbLzlbuVFWJRpiykv+juJyIFjzXLJRF67wB2qgZVm5cRs/GOi4fnvTaou/Ae51mvsO8M3JL5cphApFKD/AO/X+3pHBhLjvaW33cYsp0Z4Rce1nm8wxSmvuJRF+HAfjBlknhDl5ZaJGA96RYA+g9WbAXimFprg4xeiRp7liPExQwtVgackJweoC4MahC1glmjC3eOh9dvrrxLnALOLMOkyCSK/tOF/iq2PJaLViObanPI8mnNkMLbdJyzUBf8hRGdSLcw98qxaf0sSZ+vz4VQS6e9CwX7ikkNNsTeYXjnJ+1yU3mfCW1CXJDr7kBDVG2OTLefCpMY8WBAYDiEaK0RW83cBcFIKfecKYd0qStHRaOsvmyWdFYgBh2vM1UWjro5WXlb+7aJPrUp8ZruapkUXUwhjHA7FwGA4ysE0tsOVVFm0Dw8tvyVVkVNm/9EObG28dWctioZ2+ZEzTWgMJU26Q+wOJ/U4qZyIvpMRUgPZc5nM9J8kcmvNWtb6DV9k/6p8h28kiOxPkxLtPP+3TlBCHjq//mhEnJZ/d8w== shshyukriev@suse.com", "key_options": null, "keyfile": "/home/shyukri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyukri", "validate_certs": true} 2019-12-05T04:58:07.703 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2019-12-05T04:58:07.704 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9', u'name': u'andrewschoen'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:07.715 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n', u'name': u'zack'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:07.715 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd', u'name': u'kdreyer'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:07.727 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'alfredo@papaya.local V5VkRT2Du88WwQrYeK196w 4dff4d9bdc4d9ab60532814c338abe4534cb45a9ca729bdbc2411ad513516b47', u'name': u'adeza'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "adeza", "ovpn": "alfredo@papaya.local V5VkRT2Du88WwQrYeK196w 4dff4d9bdc4d9ab60532814c338abe4534cb45a9ca729bdbc2411ad513516b47"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:07.739 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n', u'name': u'dmick'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:07.740 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n', u'name': u'sage'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:07.752 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n', u'name': u'dgalloway'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:07.765 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045', u'name': u'akraitma'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:07.767 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e', u'name': u'trociny'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:07.820 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2019-12-05T04:58:07.821 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e', u'name': u'scarvalhojr'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxDJtmmcnrbA4AUsmBBZq/M4xNdIbOT+cQQxhUKeCu8F9mht+RznwyaaJbVIlD+UNv9LMhRyzI5oktSalokNltlrtPQMoP52PXH7KFFGnVnexCHAk4nGvEhuZCniO5T+ENm2wFR47lFEQK7lyJJLp5PFiSM7vM1MK6UkMvRmCTRildjaYLyf3EEvBXifbsGmR0tBIXtml6KITCAaT6l0E94055oEGN2G5NhZIvfLcojt5liEkXEHQNdfkUDXM0Go43JZzgsSWDXC/7RYLrAJwz5mq4qCa9YZL/3Spjso37+1oLq7lOQ0TDJUAQXmnW0So/NkzECx4MolVcylhTzmgj sergio@milestones", "key_options": null, "keyfile": "/home/scarvalhojr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "scarvalhojr", "validate_certs": true} 2019-12-05T04:58:07.905 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2019-12-05T04:58:07.906 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007', u'name': u'smithfarm', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "key_options": null, "keyfile": "/home/smithfarm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smithfarm", "validate_certs": true} 2019-12-05T04:58:07.920 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n', u'name': u'mbenjamin'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:07.967 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2019-12-05T04:58:07.969 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b', u'name': u'rakesh'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEOigBGMeElb3uHXX36HxLtPJMG+iG/6D7NDFCVR6AyYsmimBQb6SlSIR6SaFULb6HT/IFVk0Jf/z0HoipAP7Xu+wvP/J6LFoMdguxylh1ov8raqxQJrGLl9n64EI+qWINwEM0MKMUnDQ3tmMKBl78LKQVQTYBx1+9cXsT6DE1yhgmV79XIvOOb+v53agSi/kLJF2i9K04kpi1OejXg95Ub2eLMY69Jm8PnLy1Jq8DhC8pz7nrfzbKtOC7HbytTAO1XTlb90lVhqC/1yTMAUc/rpJwoSEHc2epGeZWdMsJhqhX2EqMwcV53M3qcFSGreGNJ/Ncd4/JHxz7q0IU411t rakesh@localhost.localdomain", "key_options": null, "keyfile": "/home/rakesh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rakesh", "validate_certs": true} 2019-12-05T04:58:08.062 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01', u'name': u'aemerson', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "key_options": null, "keyfile": "/home/aemerson/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aemerson", "validate_certs": true} 2019-12-05T04:58:08.142 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2019-12-05T04:58:08.144 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'ryg@wombat 7WbRbRXPcrQ2++Bww59JKg 1186db58c70662d55cae6d75e54022d1392708d3cf742b10ce5ff9d67dd8486', u'name': u'ryg-'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat 7WbRbRXPcrQ2++Bww59JKg 1186db58c70662d55cae6d75e54022d1392708d3cf742b10ce5ff9d67dd8486"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvRwrsN62CReT0/kOeFEXGA3THhzJ+hwZVpUzo7VMRZeSo6LgNlVGdrQchMHnlwBvuwaVNJlBAkQr+5J5BB3ikrmbcnLHfPutXBBBB6uDfSwwbqjzUY7elTbz9ul6AkegR9K65O44ouD9+WC76D+Tp0maqnP1w0O3h1GVxDHe34LpEwBvZJ+/BSYmBpwRZqeXFRADORPzdtW08PSiRErGHb8j+j5i+br37JfOOh+A/9dHwQw9alBVL2DijwvrPMa7dV5yX00dlMSTL2LLo9IdI4aFPmie8oic0eanFbQDq210ilOR65IGeshKHEKOq0KFFz0nSK9OdO03OkSEN/0d5 ryg@tapir", "key_options": null, "keyfile": "/home/ryg-/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ryg-", "validate_certs": true} 2019-12-05T04:58:08.195 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78', u'name': u'amaredia', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "key_options": null, "keyfile": "/home/amaredia/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amaredia", "validate_certs": true} 2019-12-05T04:58:08.304 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2019-12-05T04:58:08.306 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_key 2019-12-05T04:58:08.308 INFO:teuthology.task.ansible.out:s", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2019-12-05T04:58:08.354 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2019-12-05T04:58:08.356 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5', u'name': u'tserlin', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "key_options": null, "keyfile": "/home/tserlin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tserlin", "validate_certs": true} 2019-12-05T04:58:08.366 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28', u'name': u'dis'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:08.377 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n', u'name': u'gregf'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:08.389 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n', u'name': u'joshd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:08.390 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n', u'name': u'davidz'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:08.401 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc', u'name': u'gmeno'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gmeno", "ovpn": "gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:08.414 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1', u'name': u'ivancich'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:08.417 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n', u'name': u'wusui'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:08.432 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304', u'name': u'zyan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:08.437 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n', u'name': u'yuriw'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:08.438 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n', u'name': u'tamil'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:08.440 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42', u'name': u'zdover'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zdover", "ovpn": "zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlnlBzRdWRqns/9VrYuF9tLILUjeGd9z8Tf/dPZmxAt3nnZqpTvI2xXemQpyt/2lsD0U24EuTc1Y/dBsgqvXtZOw51TZE8RK26hvCCaKGfcKONjvR7goHAmkYPHK9+F6m7fgYkRzA0+ZW4HcWNh/xxhhL2G1EGfz5bzIoGQJHSdzc6VeVIsY7HiYEMTxvxdsgV72Kjq97BdmHm56RBzoH24oN0UzNZS8Syx5SuWj9dA3IyFBPwjLlozSk6U2Ofd7qaXuQKl807eD9g6RhRldzHPS+i0EK3EM6Jvho5Y08k1KqV4gIEwMdzDbD3SH0xq+nUPFKcDTUFNp+tkg10GxqN zac.dover@gmail.com", "key_options": null, "keyfile": "/home/zdover/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zdover", "validate_certs": true} 2019-12-05T04:58:08.450 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2019-12-05T04:58:08.452 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n', u'name': u'jowilkin'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:08.462 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4', u'name': u'bhubbard'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:08.474 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n', u'name': u'yehudasa'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:08.477 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4', u'name': u'dang'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:08.489 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb', u'name': u'branto'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:08.490 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6', u'name': u'xiaoxichen'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:08.501 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e', u'name': u'ffilz'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:08.513 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n', u'name': u'joao'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:08.514 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n', u'name': u'nhm'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:08.525 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313', u'name': u'nwatkins'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:08.537 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22', u'name': u'mkidd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:08.548 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af', u'name': u'jlopez'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:08.549 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb', u'name': u'haomaiwang'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:08.560 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608', u'name': u'jdillaman'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:08.579 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3', u'name': u'kchai'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:08.593 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:08.612 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** changed: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b', u'name': u'rraja'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rraja", "ovpn": "rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC4zwWf+kMEDgaSOrX7AlD+UCOgf4/fRiorQVJqC8GfMNynfiFVZaThngwIrwtQROfXHbzspi+AcRvDLewY25mXyuxVgOfSbuSIEqiUNvzcVq9IItI1J58xeDBpIktAsoHHS6JENV56fAT6krzRAQLnSa1k2b0pyKuPBzFSeNgiaQMhjrCLAaCi9yLsdKpnWgXhyfrdqvN2r/Ctts/zlTPqtRYdvdIL/p/l+vg7Q8S3OhuAGtimaEq3RE70xadaPbGhsLISMTrbfYXbxw84KKZpNYwO4zmQ2ztcvC9D7A/8ov3+A5Jq42G+pq7ORuix2NoTWtADMiwvNe1HyzS/7cX/BuXLFssRY1bzRvp+x2ExZ7b+bFx+TE/RduRh0eyDNLg7SyL83aeTr2Usa+mPoCAMKed9XtYbRnUu7ox4wmfeAgx6jM9uG5uQfauJ1XpDddrVBG8KYBvSd0BM2cyRdcAXqwxg9AVPZcdE7sbBk44CGVhw+ziZ3e3cLBTO4y44KRQre+1Bhe/Ze8OTQWLFugl/XagFlJE/L6fGUIPuFojmQvlw5dBZwtUURnxsu8yHm3iIsj4y8nR1zHMj+zuMk6rF3ZW0epPAECAdq4gG8X2wv3HTq0vLBkKpi1KMAmntEtcVwF0fzMt5XYqRd4jceFgonAAipJNDknHgi6yaUzBYmQ== rraja@redhat.com", "key_options": null, "keyfile": "/home/rraja/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rraja", "validate_certs": true} 2019-12-05T04:58:08.755 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2019-12-05T04:58:08.758 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n', u'name': u'dfuller'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n', u'name': u'owasserm'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n', u'name': u'vasu'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi040.front.sepi 2019-12-05T04:58:08.759 INFO:teuthology.task.ansible.out:a.ceph.com] => (item={u'ovpn': u'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37', u'name': u'smohan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} skipping: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114', u'name': u'abhi'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} skipping: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611', u'name': u'cbodley'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} skipping: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195', u'name': u'mwatts'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} skipping: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp3 2019-12-05T04:58:08.761 INFO:teuthology.task.ansible.out:5-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "skip_reason": "Conditional result was False"} skipping: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea', u'name': u'pdonnell'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} skipping: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868', u'name': u'jlayton'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} skipping: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f', u'name': u'rzarzynski'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:08.820 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:08 +0000 (0:00:01.169) 0:00:44.320 ***** 2019-12-05T04:58:08.827 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e', u'name': u'rdias'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} skipping: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953', u'name': u'vshankar'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:08.828 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58', u'name': u'akupczyk'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:08.830 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e', u'name': u'nojha'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e"}, "skip_reason": "Conditional result was False"} skipping: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43', u'name': u'ifed01'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:08.846 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4', u'name': u'myoungwon'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:08.847 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2', u'name': u'gabrioux'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:08.848 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f', u'name': u'leseb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:08.850 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca', u'name': u'hchen'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:08.861 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n', u'name': u'jcollin'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:09.310 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9', u'name': u'andrewschoen'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} skipping: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc', u'name': u'xxg'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} skipping: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n', u'name': u'zack'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd', u'name': u'kdreyer'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} skipping: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'pcuz 2019-12-05T04:58:09.311 INFO:teuthology.task.ansible.out:ner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53', u'name': u'pcuzner'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} skipping: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'alfredo@papaya.local V5VkRT2Du88WwQrYeK196w 4dff4d9bdc4d9ab60532814c338abe4534cb45a9ca729bdbc2411ad513516b47', u'name': u'adeza'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "adeza", "ovpn": "alfredo@papaya.local V5VkRT2Du88WwQrYeK196w 4dff4d9bdc4d9ab60532814c338abe4534cb45a9ca729bdbc2411ad513516b47"}, "skip_reason": "Conditional result was False"} skipping: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2', u'name': u'mkogan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} skipping: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n', u'name': u'dmick'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b409344 2019-12-05T04:58:09.313 INFO:teuthology.task.ansible.out:3b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} skipping: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n', u'name': u'sage'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c5408013034 2019-12-05T04:58:09.316 INFO:teuthology.task.ansible.out:2dccc76469fb40\n', u'name': u'dgalloway'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736', u'name': u'henrix'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} skipping: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045', u'name': u'akraitma'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} skipping: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26', u'name': u'pbs1108'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} skipping: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315', u'name': u'clacroix'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redha 2019-12-05T04:58:09.318 INFO:teuthology.task.ansible.out:t.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} skipping: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e', u'name': u'trociny'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} skipping: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006', u'name': u'epuertat'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} skipping: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461', u'name': u'tdehler'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} skipping: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305', u'name': u'laura'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} skipping: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc2585 2019-12-05T04:58:09.319 INFO:teuthology.task.ansible.out:0ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce', u'name': u'yaarit'}) => {"ansible_loop_var": "i 2019-12-05T04:58:09.321 INFO:teuthology.task.ansible.out:tem", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} skipping: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3', u'name': u'rpavani1998'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} skipping: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4', u'name': u'rishabh'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} skipping: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2', u'name': u'skrah'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} skipping: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8', u'name': u'smanjara'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, "skip_reason": "Conditional result was False"} skipping: [smithi040.front.sepia.ceph.com] => (item={u' 2019-12-05T04:58:09.322 INFO:teuthology.task.ansible.out:ovpn': u'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41', u'name': u'bengland'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} skipping: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d', u'name': u'pnawracay'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} skipping: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8', u'name': u'alfonsomthd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "skip_reason": "Conditional result was False"} skipping: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e', u'name': u'oliveiradan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} skipping: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba', u'name': u'swagner'}) => {"ansible_loop_var": "item", "changed": false, "item": {"nam 2019-12-05T04:58:09.324 INFO:teuthology.task.ansible.out:e": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} skipping: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e', u'name': u'yuvalif'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, "skip_reason": "Conditional result was False"} skipping: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852', u'name': u'kkeithle'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} skipping: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130', u'name': u'emmericp'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007', u'name': u'smithfarm', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjN 2019-12-05T04:58:09.326 INFO:teuthology.task.ansible.out:hnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "key_options": null, "keyfile": "/home/smithfarm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smithfarm", "validate_certs": true} skipping: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0', u'name': u'mchangir'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} skipping: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf5926 2019-12-05T04:58:09.327 INFO:teuthology.task.ansible.out:07d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n', u'name': u'mbenjamin'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286', u'name': u'sidharthanup'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} skipping: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31', u'name': u'varsha'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} skipping: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS 2019-12-05T04:58:09.329 INFO:teuthology.task.ansible.out:0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n', u'name': u'sjust'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'deepika@thinkpad HXRUSerxILmV0ejAD/Zt1g 29b44e741567e7a2e7b664d6533247d79851fc302051c9ba34103c63a2fcfacc', u'name': u'ideepika'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@thinkpad HXRUSerxILmV0ejAD/Zt1g 29b44e741567e7a2e7b664d6533247d79851fc302051c9ba34103c63a2fcfacc"}, "skip_reason": "Conditional result was False"} skipping: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46', u'name': u'gsalomon'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} skipping: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de', u'name': u'soumyakoduri'}) => {"ansible_loop_var": "item", "changed": false, "item": {" 2019-12-05T04:58:09.330 INFO:teuthology.task.ansible.out:name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "skip_reason": "Conditional result was False"} skipping: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a', u'name': u'kyr'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} skipping: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849', u'name': u'sseshasa'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01', u'name': u'aemerson', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM 2019-12-05T04:58:09.332 INFO:teuthology.task.ansible.out:6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "key_options": null, "keyfile": "/home/aemerson/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aemerson", "validate_certs": true} 2019-12-05T04:58:09.339 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189', u'name': u'rfriedma'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:09.340 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b', u'name': u'lmb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:09.351 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc', u'name': u'ksirivad'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:09.351 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4', u'name': u'ofriedma'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:09.362 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52', u'name': u'h4iresis'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:09.374 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5', u'name': u'shyukri'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:09.385 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e', u'name': u'scarvalhojr'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:09.386 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b', u'name': u'rakesh'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:09.397 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'ryg@wombat 7WbRbRXPcrQ2++Bww59JKg 1186db58c70662d55cae6d75e54022d1392708d3cf742b10ce5ff9d67dd8486', u'name': u'ryg-'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat 7WbRbRXPcrQ2++Bww59JKg 1186db58c70662d55cae6d75e54022d1392708d3cf742b10ce5ff9d67dd8486"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:09.421 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78', u'name': u'amaredia', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "key_options": null, "keyfile": "/home/amaredia/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amaredia", "validate_certs": true} 2019-12-05T04:58:09.422 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} skipping: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42', u'name': u'zdover'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:09.435 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => (item={u'ovpn': u'rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b', u'name': u'rraja'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:09.587 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:09 +0000 (0:00:00.769) 0:00:45.089 ***** 2019-12-05T04:58:09.600 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5', u'name': u'tserlin', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "key_options": null, "keyfile": "/home/tserlin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tserlin", "validate_certs": true} 2019-12-05T04:58:09.603 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28', u'name': u'dis'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:09.604 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n', u'name': u'gregf'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:09.621 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n', u'name': u'joshd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} TASK [users : Filter the revoked_users list] *********************************** skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:09.674 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:09 +0000 (0:00:00.086) 0:00:45.176 ***** 2019-12-05T04:58:09.689 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n', u'name': u'davidz'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc', u'name': u'gmeno'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gmeno", "ovpn": "gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc"}, "skip_reason": "Conditional result was False"} skipping: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1', u'name': u'ivancich'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} skipping: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc294754 2019-12-05T04:58:09.692 INFO:teuthology.task.ansible.out:10cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n', u'name': u'wusui'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:09.699 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304', u'name': u'zyan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:09.711 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n', u'name': u'yuriw'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:09.723 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n', u'name': u'tamil'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:09.735 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n', u'name': u'jowilkin'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:09.746 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4', u'name': u'bhubbard'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:09.747 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n', u'name': u'yehudasa'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:09.758 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4', u'name': u'dang'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:09.770 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb', u'name': u'branto'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:09.781 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6', u'name': u'xiaoxichen'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:09.793 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e', u'name': u'ffilz'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:09.795 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n', u'name': u'joao'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:09.807 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n', u'name': u'nhm'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:09.818 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313', u'name': u'nwatkins'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:09.819 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22', u'name': u'mkidd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:09.831 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af', u'name': u'jlopez'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:09.843 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb', u'name': u'haomaiwang'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:09.856 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** ok: [smithi040.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "name": "zcerza", "state": "absent"} 2019-12-05T04:58:09.873 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608', u'name': u'jdillaman'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "skip_reason": "Conditional result was False"} skipping: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3', u'name': u'kchai'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} skipping: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:09.887 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n', u'name': u'dfuller'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:09.899 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n', u'name': u'owasserm'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:09.912 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n', u'name': u'vasu'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:09.924 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37', u'name': u'smohan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:09.935 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114', u'name': u'abhi'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:09.947 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611', u'name': u'cbodley'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:09.948 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195', u'name': u'mwatts'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:09.960 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:09.973 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea', u'name': u'pdonnell'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:09.985 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868', u'name': u'jlayton'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:09.990 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f', u'name': u'rzarzynski'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:10.030 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2019-12-05T04:58:10.031 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "name": "aschoen", "state": "absent"} 2019-12-05T04:58:10.035 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e', u'name': u'rdias'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:10.044 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953', u'name': u'vshankar'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, "skip_reason": "Conditional result was False"} skipping: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58', u'name': u'akupczyk'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} skipping: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e', u'name': u'nojha'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e"}, "skip_reason": "Conditional result was False"} skipping: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43', u'name': u'ifed01'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:10.051 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4', u'name': u'myoungwon'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:10.062 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2', u'name': u'gabrioux'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:10.075 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f', u'name': u'leseb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:10.087 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca', u'name': u'hchen'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:10.089 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n', u'name': u'jcollin'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:10.101 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc', u'name': u'xxg'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:10.126 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53', u'name': u'pcuzner'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:10.128 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2', u'name': u'mkogan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} skipping: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:10.142 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736', u'name': u'henrix'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:10.154 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26', u'name': u'pbs1108'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:10.155 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315', u'name': u'clacroix'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:10.170 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006', u'name': u'epuertat'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:10.184 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461', u'name': u'tdehler'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} TASK [users : Remove revoked users] ******************************************** ok: [smithi040.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "name": "andrew", "state": "absent"} 2019-12-05T04:58:10.196 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305', u'name': u'laura'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:10.226 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce', u'name': u'yaarit'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} skipping: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3', u'name': u'rpavani1998'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:10.246 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4', u'name': u'rishabh'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:10.257 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2', u'name': u'skrah'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:10.413 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8', u'name': u'smanjara'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:10.414 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41', u'name': u'bengland'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:10.415 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** ok: [smithi040.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "name": "sweil", "state": "absent"} TASK [users : Update authorized_keys for each user with literal keys] ********** 2019-12-05T04:58:10.417 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d', u'name': u'pnawracay'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:10.418 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8', u'name': u'alfonsomthd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:10.433 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e', u'name': u'oliveiradan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} skipping: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba', u'name': u'swagner'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:10.442 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e', u'name': u'yuvalif'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:10.454 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852', u'name': u'kkeithle'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:10.476 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130', u'name': u'emmericp'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:10.486 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0', u'name': u'mchangir'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:10.500 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286', u'name': u'sidharthanup'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} TASK [users : Remove revoked users] ******************************************** ok: [smithi040.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "name": "brad", "state": "absent"} 2019-12-05T04:58:10.502 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31', u'name': u'varsha'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:10.514 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n', u'name': u'sjust'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:10.525 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'deepika@thinkpad HXRUSerxILmV0ejAD/Zt1g 29b44e741567e7a2e7b664d6533247d79851fc302051c9ba34103c63a2fcfacc', u'name': u'ideepika'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@thinkpad HXRUSerxILmV0ejAD/Zt1g 29b44e741567e7a2e7b664d6533247d79851fc302051c9ba34103c63a2fcfacc"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:10.547 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46', u'name': u'gsalomon'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:10.547 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de', u'name': u'soumyakoduri'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:10.558 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a', u'name': u'kyr'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:10.559 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849', u'name': u'sseshasa'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:10.581 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189', u'name': u'rfriedma'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} skipping: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b', u'name': u'lmb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:10.593 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc', u'name': u'ksirivad'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:10.604 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4', u'name': u'ofriedma'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:10.615 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52', u'name': u'h4iresis'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:10.627 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5', u'name': u'shyukri'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:10.628 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e', u'name': u'scarvalhojr'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:10.639 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** ok: [smithi040.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "name": "kefu", "state": "absent"} 2019-12-05T04:58:10.640 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2019-12-05T04:58:10.642 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b', u'name': u'rakesh'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:10.651 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'ryg@wombat 7WbRbRXPcrQ2++Bww59JKg 1186db58c70662d55cae6d75e54022d1392708d3cf742b10ce5ff9d67dd8486', u'name': u'ryg-'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat 7WbRbRXPcrQ2++Bww59JKg 1186db58c70662d55cae6d75e54022d1392708d3cf742b10ce5ff9d67dd8486"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:10.662 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:10.663 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42', u'name': u'zdover'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:10.688 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => (item={u'ovpn': u'rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b', u'name': u'rraja'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:10.760 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:10 +0000 (0:00:01.086) 0:00:46.263 ***** 2019-12-05T04:58:10.800 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** ok: [smithi040.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "name": "shylesh", "state": "absent"} TASK [users : Filter the revoked_users list] *********************************** skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:10.850 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:10 +0000 (0:00:00.090) 0:00:46.353 ***** 2019-12-05T04:58:10.976 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** ok: [smithi040.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "name": "trhoden", "state": "absent"} 2019-12-05T04:58:11.038 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "name": "zcerza", "state": "absent"} 2019-12-05T04:58:11.114 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "name": "nishtha", "state": "absent"} 2019-12-05T04:58:11.185 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "name": "aschoen", "state": "absent"} 2019-12-05T04:58:11.255 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "name": "yguang", "state": "absent"} 2019-12-05T04:58:11.343 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "name": "andrew", "state": "absent"} 2019-12-05T04:58:11.409 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "name": "sdieffen", "state": "absent"} 2019-12-05T04:58:11.494 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "name": "sweil", "state": "absent"} 2019-12-05T04:58:11.589 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "name": "brian", "state": "absent"} 2019-12-05T04:58:11.660 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "name": "brad", "state": "absent"} 2019-12-05T04:58:11.746 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "name": "pmcgarry", "state": "absent"} 2019-12-05T04:58:11.829 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "name": "kefu", "state": "absent"} 2019-12-05T04:58:11.887 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "name": "karnan", "state": "absent"} 2019-12-05T04:58:11.993 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "name": "shylesh", "state": "absent"} 2019-12-05T04:58:12.037 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "name": "ryneli", "state": "absent"} 2019-12-05T04:58:12.139 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "name": "trhoden", "state": "absent"} 2019-12-05T04:58:12.173 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "name": "dlambrig", "state": "absent"} 2019-12-05T04:58:12.305 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "name": "nishtha", "state": "absent"} 2019-12-05T04:58:12.319 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "name": "icolle", "state": "absent"} 2019-12-05T04:58:12.451 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "name": "yguang", "state": "absent"} 2019-12-05T04:58:12.463 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "name": "soumya", "state": "absent"} 2019-12-05T04:58:12.592 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "name": "sdieffen", "state": "absent"} 2019-12-05T04:58:12.613 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "name": "jspray", "state": "absent"} 2019-12-05T04:58:12.751 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "name": "brian", "state": "absent"} 2019-12-05T04:58:12.753 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item=loic) => {"ansible_loop_var": "item", "changed": false, "item": "loic", "name": "loic", "state": "absent"} 2019-12-05T04:58:12.904 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "name": "pmcgarry", "state": "absent"} 2019-12-05T04:58:12.915 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "name": "erwan", "state": "absent"} 2019-12-05T04:58:13.057 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "name": "karnan", "state": "absent"} 2019-12-05T04:58:13.203 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "name": "ryneli", "state": "absent"} 2019-12-05T04:58:13.359 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "name": "dlambrig", "state": "absent"} 2019-12-05T04:58:13.377 INFO:teuthology.task.ansible.out:changed: [smithi040.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "jj", "name": "jj", "remove": false, "state": "absent"} 2019-12-05T04:58:13.524 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "name": "icolle", "state": "absent"} 2019-12-05T04:58:13.656 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "name": "soumya", "state": "absent"} 2019-12-05T04:58:13.824 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "name": "jspray", "state": "absent"} 2019-12-05T04:58:13.888 INFO:teuthology.task.ansible.out:changed: [smithi040.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "amarangone", "name": "amarangone", "remove": false, "state": "absent"} 2019-12-05T04:58:13.973 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item=loic) => {"ansible_loop_var": "item", "changed": false, "item": "loic", "name": "loic", "state": "absent"} 2019-12-05T04:58:14.113 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "name": "erwan", "state": "absent"} 2019-12-05T04:58:14.351 INFO:teuthology.task.ansible.out:changed: [smithi040.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "oprypin", "name": "oprypin", "remove": false, "state": "absent"} 2019-12-05T04:58:14.572 INFO:teuthology.task.ansible.out:changed: [smithi042.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "jj", "name": "jj", "remove": false, "state": "absent"} 2019-12-05T04:58:14.860 INFO:teuthology.task.ansible.out:changed: [smithi040.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "adamyanova", "name": "adamyanova", "remove": false, "state": "absent"} 2019-12-05T04:58:15.012 INFO:teuthology.task.ansible.out:changed: [smithi042.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "amarangone", "name": "amarangone", "remove": false, "state": "absent"} 2019-12-05T04:58:15.319 INFO:teuthology.task.ansible.out:changed: [smithi040.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "sbillah", "name": "sbillah", "remove": false, "state": "absent"} 2019-12-05T04:58:15.420 INFO:teuthology.task.ansible.out:changed: [smithi042.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "oprypin", "name": "oprypin", "remove": false, "state": "absent"} 2019-12-05T04:58:15.714 INFO:teuthology.task.ansible.out:changed: [smithi040.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "onyb", "name": "onyb", "remove": false, "state": "absent"} 2019-12-05T04:58:15.839 INFO:teuthology.task.ansible.out:changed: [smithi042.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "adamyanova", "name": "adamyanova", "remove": false, "state": "absent"} 2019-12-05T04:58:16.114 INFO:teuthology.task.ansible.out:changed: [smithi040.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "jwilliamson", "name": "jwilliamson", "remove": false, "state": "absent"} 2019-12-05T04:58:16.243 INFO:teuthology.task.ansible.out:changed: [smithi042.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "sbillah", "name": "sbillah", "remove": false, "state": "absent"} 2019-12-05T04:58:16.543 INFO:teuthology.task.ansible.out:changed: [smithi040.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "kmroz", "name": "kmroz", "remove": false, "state": "absent"} 2019-12-05T04:58:16.686 INFO:teuthology.task.ansible.out:changed: [smithi042.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "onyb", "name": "onyb", "remove": false, "state": "absent"} 2019-12-05T04:58:17.020 INFO:teuthology.task.ansible.out:changed: [smithi040.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "shehbazj", "name": "shehbazj", "remove": false, "state": "absent"} 2019-12-05T04:58:17.132 INFO:teuthology.task.ansible.out:changed: [smithi042.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "jwilliamson", "name": "jwilliamson", "remove": false, "state": "absent"} 2019-12-05T04:58:17.461 INFO:teuthology.task.ansible.out:changed: [smithi040.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "abhishekvrshny", "name": "abhishekvrshny", "remove": false, "state": "absent"} 2019-12-05T04:58:17.558 INFO:teuthology.task.ansible.out:changed: [smithi042.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "kmroz", "name": "kmroz", "remove": false, "state": "absent"} 2019-12-05T04:58:17.948 INFO:teuthology.task.ansible.out:changed: [smithi040.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "asheplyakov", "name": "asheplyakov", "remove": false, "state": "absent"} 2019-12-05T04:58:17.991 INFO:teuthology.task.ansible.out:changed: [smithi042.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "shehbazj", "name": "shehbazj", "remove": false, "state": "absent"} 2019-12-05T04:58:18.378 INFO:teuthology.task.ansible.out:changed: [smithi040.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "liupan", "name": "liupan", "remove": false, "state": "absent"} 2019-12-05T04:58:18.453 INFO:teuthology.task.ansible.out:changed: [smithi042.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "abhishekvrshny", "name": "abhishekvrshny", "remove": false, "state": "absent"} 2019-12-05T04:58:18.516 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "name": "nlevine", "state": "absent"} 2019-12-05T04:58:18.656 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "name": "tbrekke", "state": "absent"} 2019-12-05T04:58:18.786 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "name": "taco", "state": "absent"} 2019-12-05T04:58:18.891 INFO:teuthology.task.ansible.out:changed: [smithi042.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "asheplyakov", "name": "asheplyakov", "remove": false, "state": "absent"} 2019-12-05T04:58:18.923 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "name": "louis", "state": "absent"} 2019-12-05T04:58:19.084 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "name": "amarango", "state": "absent"} 2019-12-05T04:58:19.217 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "name": "oobe", "state": "absent"} 2019-12-05T04:58:19.345 INFO:teuthology.task.ansible.out:changed: [smithi042.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "liupan", "name": "liupan", "remove": false, "state": "absent"} ok: [smithi040.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "name": "rturk", "state": "absent"} 2019-12-05T04:58:19.477 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "name": "nlevine", "state": "absent"} 2019-12-05T04:58:19.606 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "name": "tbrekke", "state": "absent"} 2019-12-05T04:58:19.757 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "name": "taco", "state": "absent"} 2019-12-05T04:58:19.792 INFO:teuthology.task.ansible.out:changed: [smithi040.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "fche", "name": "fche", "remove": false, "state": "absent"} 2019-12-05T04:58:19.904 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:19 +0000 (0:00:09.050) 0:00:55.404 ***** 2019-12-05T04:58:19.942 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "name": "louis", "state": "absent"} 2019-12-05T04:58:20.079 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "name": "amarango", "state": "absent"} 2019-12-05T04:58:20.222 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "name": "oobe", "state": "absent"} 2019-12-05T04:58:20.318 INFO:teuthology.task.ansible.out: TASK [users : Write /keys-repo-sha1 sentinel file] ***************************** 2019-12-05T04:58:20.320 INFO:teuthology.task.ansible.out:changed: [smithi040.front.sepia.ceph.com] => {"changed": true, "checksum": "4d6d0da35c32595244f0557c86cc20a1ae54bb67", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "md5sum": "283e36f6f6840b00622b73078c099c15", "mode": "0644", "owner": "root", "size": 40, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1575521899.92-267013448362217/source", "state": "file", "uid": 0} 2019-12-05T04:58:20.358 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:20 +0000 (0:00:00.456) 0:00:55.861 ***** 2019-12-05T04:58:20.365 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2019-12-05T04:58:20.368 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "name": "rturk", "state": "absent"} 2019-12-05T04:58:20.390 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** 2019-12-05T04:58:20.392 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => { "msg": "Host smithi040.front.sepia.ceph.com is running Ubuntu 18.04 (bionic)" } 2019-12-05T04:58:20.430 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:20 +0000 (0:00:00.071) 0:00:55.932 ***** 2019-12-05T04:58:20.480 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** ok: [smithi040.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"} 2019-12-05T04:58:20.522 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:20 +0000 (0:00:00.092) 0:00:56.024 ***** 2019-12-05T04:58:20.548 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** 2019-12-05T04:58:20.550 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:20.589 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:20 +0000 (0:00:00.066) 0:00:56.091 ***** 2019-12-05T04:58:20.616 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:20.663 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:20 +0000 (0:00:00.074) 0:00:56.166 ***** 2019-12-05T04:58:20.705 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:20.745 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:20 +0000 (0:00:00.081) 0:00:56.248 ***** 2019-12-05T04:58:20.771 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** 2019-12-05T04:58:20.777 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:20.810 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:20 +0000 (0:00:00.065) 0:00:56.313 ***** 2019-12-05T04:58:20.831 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2019-12-05T04:58:20.831 INFO:teuthology.task.ansible.out:changed: [smithi042.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "fche", "name": "fche", "remove": false, "state": "absent"} 2019-12-05T04:58:20.873 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** 2019-12-05T04:58:20.875 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:20.925 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:20 +0000 (0:00:00.114) 0:00:56.427 ***** 2019-12-05T04:58:20.962 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:20 +0000 (0:00:00.037) 0:00:56.465 ***** 2019-12-05T04:58:20.967 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** 2019-12-05T04:58:20.969 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:21.010 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:21 +0000 (0:00:00.047) 0:00:56.512 ***** 2019-12-05T04:58:21.036 INFO:teuthology.task.ansible.out: TASK [common : Install yum on Fedora 22 and later] ***************************** 2019-12-05T04:58:21.038 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:21.083 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:21 +0000 (0:00:00.072) 0:00:56.585 ***** 2019-12-05T04:58:21.109 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** 2019-12-05T04:58:21.111 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:21.149 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:21 +0000 (0:00:00.066) 0:00:56.652 ***** 2019-12-05T04:58:21.187 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** 2019-12-05T04:58:21.188 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2019-12-05T04:58:21.198 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2019-12-05T04:58:21.253 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:21 +0000 (0:00:00.095) 0:00:56.747 ***** 2019-12-05T04:58:21.272 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** 2019-12-05T04:58:21.274 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:21.322 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:21 +0000 (0:00:00.077) 0:00:56.825 ***** 2019-12-05T04:58:21.349 INFO:teuthology.task.ansible.out: TASK [users : Write /keys-repo-sha1 sentinel file] ***************************** changed: [smithi042.front.sepia.ceph.com] => {"changed": true, "checksum": "4d6d0da35c32595244f0557c86cc20a1ae54bb67", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "md5sum": "283e36f6f6840b00622b73078c099c15", "mode": "0644", "owner": "root", "size": 40, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1575521900.99-249680831006116/source", "state": "file", "uid": 0} TASK [common : Install CA Cert from Satellite Server] ************************** skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:21.389 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:21 +0000 (0:00:00.066) 0:00:56.892 ***** 2019-12-05T04:58:21.427 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:21 +0000 (0:00:00.036) 0:00:56.928 ***** 2019-12-05T04:58:21.432 INFO:teuthology.task.ansible.out: TASK [common : Determine if node is registered with subscription-manager.] ***** skipping: [smithi040.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2019-12-05T04:58:21.481 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:21 +0000 (0:00:00.055) 0:00:56.984 ***** 2019-12-05T04:58:21.487 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** 2019-12-05T04:58:21.489 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => { "msg": "Host smithi042.front.sepia.ceph.com is running Ubuntu 18.04 (bionic)" } 2019-12-05T04:58:21.533 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:21 +0000 (0:00:00.051) 0:00:57.036 ***** 2019-12-05T04:58:21.538 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** 2019-12-05T04:58:21.540 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:21.582 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:21 +0000 (0:00:00.048) 0:00:57.084 ***** 2019-12-05T04:58:21.598 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** 2019-12-05T04:58:21.600 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"} 2019-12-05T04:58:21.647 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:21 +0000 (0:00:00.061) 0:00:57.146 ***** 2019-12-05T04:58:21.657 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** skipping: [smithi040.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2019-12-05T04:58:21.701 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:21 +0000 (0:00:00.057) 0:00:57.204 ***** 2019-12-05T04:58:21.707 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** 2019-12-05T04:58:21.709 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:21.749 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:21 +0000 (0:00:00.047) 0:00:57.252 ***** 2019-12-05T04:58:21.754 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** 2019-12-05T04:58:21.756 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:21.801 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:21 +0000 (0:00:00.051) 0:00:57.303 ***** 2019-12-05T04:58:21.808 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:21.851 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:21 +0000 (0:00:00.050) 0:00:57.354 ***** 2019-12-05T04:58:21.857 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** 2019-12-05T04:58:21.859 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:21.904 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:21 +0000 (0:00:00.051) 0:00:57.405 ***** 2019-12-05T04:58:21.909 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* 2019-12-05T04:58:21.911 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:21.952 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:21 +0000 (0:00:00.049) 0:00:57.455 ***** 2019-12-05T04:58:21.958 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:22.004 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:21 +0000 (0:00:00.051) 0:00:57.507 ***** 2019-12-05T04:58:22.010 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** 2019-12-05T04:58:22.011 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:22.050 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:22 +0000 (0:00:00.046) 0:00:57.553 ***** 2019-12-05T04:58:22.059 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:22.105 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:22 +0000 (0:00:00.054) 0:00:57.608 ***** 2019-12-05T04:58:22.112 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:22.155 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:22 +0000 (0:00:00.050) 0:00:57.658 ***** 2019-12-05T04:58:22.161 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* 2019-12-05T04:58:22.162 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2019-12-05T04:58:22.206 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:22 +0000 (0:00:00.050) 0:00:57.709 ***** 2019-12-05T04:58:22.220 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:22.258 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:22 +0000 (0:00:00.051) 0:00:57.760 ***** 2019-12-05T04:58:22.263 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2019-12-05T04:58:22.264 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:22.318 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:22 +0000 (0:00:00.060) 0:00:57.820 ***** 2019-12-05T04:58:22.326 INFO:teuthology.task.ansible.out: TASK [common : Install yum on Fedora 22 and later] ***************************** skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:22.370 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:22 +0000 (0:00:00.052) 0:00:57.873 ***** 2019-12-05T04:58:22.376 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* 2019-12-05T04:58:22.377 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:22.418 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:22 +0000 (0:00:00.047) 0:00:57.920 ***** 2019-12-05T04:58:22.424 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** 2019-12-05T04:58:22.426 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:22.465 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:22 +0000 (0:00:00.047) 0:00:57.967 ***** 2019-12-05T04:58:22.470 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ 2019-12-05T04:58:22.470 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:22.528 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:22 +0000 (0:00:00.058) 0:00:58.026 ***** 2019-12-05T04:58:22.535 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** skipping: [smithi042.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} skipping: [smithi042.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2019-12-05T04:58:22.573 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:22 +0000 (0:00:00.049) 0:00:58.076 ***** 2019-12-05T04:58:22.578 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** 2019-12-05T04:58:22.579 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:22.624 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:22 +0000 (0:00:00.050) 0:00:58.126 ***** 2019-12-05T04:58:22.629 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** 2019-12-05T04:58:22.631 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:22.674 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:22 +0000 (0:00:00.049) 0:00:58.176 ***** 2019-12-05T04:58:22.681 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:22.725 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:22 +0000 (0:00:00.051) 0:00:58.227 ***** 2019-12-05T04:58:22.732 INFO:teuthology.task.ansible.out: TASK [common : Install CA Cert from Satellite Server] ************************** 2019-12-05T04:58:22.740 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:22.781 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:22 +0000 (0:00:00.049) 0:00:58.277 ***** TASK [common : Set replace_repos true if newly-subscribed] ********************* skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:22.858 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:22 +0000 (0:00:00.050) 0:00:58.327 ***** TASK [common : Determine if node is registered with subscription-manager.] ***** skipping: [smithi042.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2019-12-05T04:58:22.872 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:22 +0000 (0:00:00.047) 0:00:58.375 ***** 2019-12-05T04:58:22.878 INFO:teuthology.task.ansible.out: TASK [common : Disable all rhsm repos] ***************************************** 2019-12-05T04:58:22.878 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2019-12-05T04:58:22.928 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:22 +0000 (0:00:00.055) 0:00:58.431 ***** 2019-12-05T04:58:22.935 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** 2019-12-05T04:58:22.937 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:22.978 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:22 +0000 (0:00:00.049) 0:00:58.481 ***** 2019-12-05T04:58:22.996 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ 2019-12-05T04:58:23.139 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:23 +0000 (0:00:00.159) 0:00:58.640 ***** 2019-12-05T04:58:23.145 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** skipping: [smithi042.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2019-12-05T04:58:23.190 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:23 +0000 (0:00:00.053) 0:00:58.693 ***** 2019-12-05T04:58:23.196 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* 2019-12-05T04:58:23.198 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:23.240 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:23 +0000 (0:00:00.050) 0:00:58.743 ***** 2019-12-05T04:58:23.249 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:23.289 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:23 +0000 (0:00:00.048) 0:00:58.792 ***** 2019-12-05T04:58:23.295 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:23.345 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:23 +0000 (0:00:00.050) 0:00:58.842 ***** 2019-12-05T04:58:23.347 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** 2019-12-05T04:58:23.349 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:23.399 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:23 +0000 (0:00:00.058) 0:00:58.901 ***** 2019-12-05T04:58:23.405 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** 2019-12-05T04:58:23.406 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => (item={'value': {u'mirrorlist': u'file:///etc/yum.repos.d/epel-mirrorlist', u'name': u'Extra Packages for Enterprise Linux', u'failovermethod': u'priority', u'gpgcheck': 0, u'enabled': u'1', u'baseurl': u'http://dl.fedoraproject.org/pub/epel/18/$basearch'}, 'key': u'epel'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel", "value": {"baseurl": "http://dl.fedoraproject.org/pub/epel/18/$basearch", "enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:23.407 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => (item={'value': {u'mirrorlist': u'file:///etc/yum.repos.d/epel-testing-mirrorlist', u'name': u'Extra Packages for Enterprise Linux - Testing', u'failovermethod': u'priority', u'gpgcheck': 0, u'enabled': 0, u'baseurl': u'http://dl.fedoraproject.org/pub/epel/testing/18/$basearch'}, 'key': u'epel-testing'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel-testing", "value": {"baseurl": "http://dl.fedoraproject.org/pub/epel/testing/18/$basearch", "enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:23.451 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:23 +0000 (0:00:00.052) 0:00:58.953 ***** 2019-12-05T04:58:23.457 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** 2019-12-05T04:58:23.459 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:23.507 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:23 +0000 (0:00:00.056) 0:00:59.010 ***** 2019-12-05T04:58:23.512 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* 2019-12-05T04:58:23.513 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-mirrorlist", "skip_reason": "Conditional result was False"} 2019-12-05T04:58:23.516 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-testing-mirrorlist", "skip_reason": "Conditional result was False"} 2019-12-05T04:58:23.561 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:23 +0000 (0:00:00.054) 0:00:59.064 ***** 2019-12-05T04:58:23.572 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:23.614 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:23 +0000 (0:00:00.052) 0:00:59.116 ***** 2019-12-05T04:58:23.620 INFO:teuthology.task.ansible.out: TASK [common : Temporarily comment mirrorlist so we can update yum cache] ****** 2019-12-05T04:58:23.622 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => (item={'value': {u'mirrorlist': u'file:///etc/yum.repos.d/epel-mirrorlist', u'name': u'Extra Packages for Enterprise Linux', u'failovermethod': u'priority', u'gpgcheck': 0, u'enabled': u'1', u'baseurl': u'http://dl.fedoraproject.org/pub/epel/18/$basearch'}, 'key': u'epel'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel", "value": {"baseurl": "http://dl.fedoraproject.org/pub/epel/18/$basearch", "enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:23.623 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => (item={'value': {u'mirrorlist': u'file:///etc/yum.repos.d/epel-testing-mirrorlist', u'name': u'Extra Packages for Enterprise Linux - Testing', u'failovermethod': u'priority', u'gpgcheck': 0, u'enabled': 0, u'baseurl': u'http://dl.fedoraproject.org/pub/epel/testing/18/$basearch'}, 'key': u'epel-testing'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel-testing", "value": {"baseurl": "http://dl.fedoraproject.org/pub/epel/testing/18/$basearch", "enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:23.667 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:23 +0000 (0:00:00.051) 0:00:59.168 ***** 2019-12-05T04:58:23.672 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* 2019-12-05T04:58:23.673 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2019-12-05T04:58:23.714 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:23 +0000 (0:00:00.049) 0:00:59.217 ***** 2019-12-05T04:58:23.720 INFO:teuthology.task.ansible.out: TASK [common : Update yum cache] *********************************************** 2019-12-05T04:58:23.721 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:23.768 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:23 +0000 (0:00:00.050) 0:00:59.268 ***** 2019-12-05T04:58:23.772 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2019-12-05T04:58:23.774 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:23.824 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:23 +0000 (0:00:00.054) 0:00:59.322 ***** 2019-12-05T04:58:23.825 INFO:teuthology.task.ansible.out: TASK [common : Uncomment mirrorlist] ******************************************* 2019-12-05T04:58:23.827 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => (item={'value': {u'mirrorlist': u'file:///etc/yum.repos.d/epel-mirrorlist', u'name': u'Extra Packages for Enterprise Linux', u'failovermethod': u'priority', u'gpgcheck': 0, u'enabled': u'1', u'baseurl': u'http://dl.fedoraproject.org/pub/epel/18/$basearch'}, 'key': u'epel'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel", "value": {"baseurl": "http://dl.fedoraproject.org/pub/epel/18/$basearch", "enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:23.828 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => (item={'value': {u'mirrorlist': u'file:///etc/yum.repos.d/epel-testing-mirrorlist', u'name': u'Extra Packages for Enterprise Linux - Testing', u'failovermethod': u'priority', u'gpgcheck': 0, u'enabled': 0, u'baseurl': u'http://dl.fedoraproject.org/pub/epel/testing/18/$basearch'}, 'key': u'epel-testing'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel-testing", "value": {"baseurl": "http://dl.fedoraproject.org/pub/epel/testing/18/$basearch", "enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:23.879 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:23 +0000 (0:00:00.051) 0:00:59.373 ***** TASK [common : Get list of enabled RHSM repos] ********************************* skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:23.921 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:23 +0000 (0:00:00.047) 0:00:59.421 ***** 2019-12-05T04:58:23.945 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ 2019-12-05T04:58:23.946 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:23.986 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:23 +0000 (0:00:00.067) 0:00:59.488 ***** 2019-12-05T04:58:24.012 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** 2019-12-05T04:58:24.014 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:24.051 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:24 +0000 (0:00:00.065) 0:00:59.554 ***** 2019-12-05T04:58:24.057 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2019-12-05T04:58:24.058 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => {"changed": false, "cmd": "cat /etc/timezone", "delta": "0:00:00.012189", "end": "2019-12-05 04:58:23.647985", "rc": 0, "start": "2019-12-05 04:58:23.635796", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2019-12-05T04:58:24.099 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:24 +0000 (0:00:00.048) 0:00:59.602 ***** 2019-12-05T04:58:24.105 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** 2019-12-05T04:58:24.106 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:24.148 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:24 +0000 (0:00:00.049) 0:00:59.651 ***** 2019-12-05T04:58:24.154 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** 2019-12-05T04:58:24.154 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:24.198 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:24 +0000 (0:00:00.049) 0:00:59.701 ***** 2019-12-05T04:58:24.209 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:24.246 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:24 +0000 (0:00:00.048) 0:00:59.749 ***** 2019-12-05T04:58:24.252 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** 2019-12-05T04:58:24.254 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:24.294 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:24 +0000 (0:00:00.046) 0:00:59.796 ***** 2019-12-05T04:58:24.299 INFO:teuthology.task.ansible.out: TASK [common : Disable all rhsm repos] ***************************************** 2019-12-05T04:58:24.300 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2019-12-05T04:58:24.360 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:24 +0000 (0:00:00.064) 0:00:59.860 ***** 2019-12-05T04:58:24.364 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:24.409 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:24 +0000 (0:00:00.050) 0:00:59.911 ***** 2019-12-05T04:58:24.414 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ 2019-12-05T04:58:24.456 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:24 +0000 (0:00:00.047) 0:00:59.959 ***** 2019-12-05T04:58:24.461 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** 2019-12-05T04:58:24.463 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:24.505 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:24 +0000 (0:00:00.048) 0:01:00.008 ***** 2019-12-05T04:58:24.511 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* 2019-12-05T04:58:24.514 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:24.554 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:24 +0000 (0:00:00.048) 0:01:00.057 ***** 2019-12-05T04:58:24.560 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ 2019-12-05T04:58:24.564 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:24.604 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:24 +0000 (0:00:00.050) 0:01:00.107 ***** 2019-12-05T04:58:24.610 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** 2019-12-05T04:58:24.611 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:24.653 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:24 +0000 (0:00:00.047) 0:01:00.155 ***** 2019-12-05T04:58:24.658 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* 2019-12-05T04:58:24.659 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:24.716 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:24 +0000 (0:00:00.062) 0:01:00.217 ***** 2019-12-05T04:58:24.725 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** skipping: [smithi042.front.sepia.ceph.com] => (item={'value': {u'mirrorlist': u'file:///etc/yum.repos.d/epel-mirrorlist', u'name': u'Extra Packages for Enterprise Linux', u'failovermethod': u'priority', u'gpgcheck': 0, u'enabled': u'1', u'baseurl': u'http://dl.fedoraproject.org/pub/epel/18/$basearch'}, 'key': u'epel'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel", "value": {"baseurl": "http://dl.fedoraproject.org/pub/epel/18/$basearch", "enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "skip_reason": "Conditional result was False"} skipping: [smithi042.front.sepia.ceph.com] => (item={'value': {u'mirrorlist': u'file:///etc/yum.repos.d/epel-testing-mirrorlist', u'name': u'Extra Packages for Enterprise Linux - Testing', u'failovermethod': u'priority', u'gpgcheck': 0, u'enabled': 0, u'baseurl': u'http://dl.fedoraproject.org/pub/epel/testing/18/$basearch'}, 'key': u'epel-testing'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel-testing", "value": {"baseurl": "http://dl.fedoraproject.org/pub/epel/testing/18/$basearch", "enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:24.767 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:24 +0000 (0:00:00.052) 0:01:00.269 ***** 2019-12-05T04:58:24.775 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:24.824 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:24 +0000 (0:00:00.056) 0:01:00.326 ***** 2019-12-05T04:58:24.832 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* skipping: [smithi042.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-mirrorlist", "skip_reason": "Conditional result was False"} skipping: [smithi042.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-testing-mirrorlist", "skip_reason": "Conditional result was False"} 2019-12-05T04:58:24.891 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:24 +0000 (0:00:00.067) 0:01:00.394 ***** 2019-12-05T04:58:24.952 INFO:teuthology.task.ansible.out: TASK [common : Temporarily comment mirrorlist so we can update yum cache] ****** 2019-12-05T04:58:24.953 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => (item={'value': {u'mirrorlist': u'file:///etc/yum.repos.d/epel-mirrorlist', u'name': u'Extra Packages for Enterprise Linux', u'failovermethod': u'priority', u'gpgcheck': 0, u'enabled': u'1', u'baseurl': u'http://dl.fedoraproject.org/pub/epel/18/$basearch'}, 'key': u'epel'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel", "value": {"baseurl": "http://dl.fedoraproject.org/pub/epel/18/$basearch", "enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:24.964 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => (item={'value': {u'mirrorlist': u'file:///etc/yum.repos.d/epel-testing-mirrorlist', u'name': u'Extra Packages for Enterprise Linux - Testing', u'failovermethod': u'priority', u'gpgcheck': 0, u'enabled': 0, u'baseurl': u'http://dl.fedoraproject.org/pub/epel/testing/18/$basearch'}, 'key': u'epel-testing'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel-testing", "value": {"baseurl": "http://dl.fedoraproject.org/pub/epel/testing/18/$basearch", "enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:25.007 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:25 +0000 (0:00:00.116) 0:01:00.510 ***** 2019-12-05T04:58:25.034 INFO:teuthology.task.ansible.out: TASK [common : Update yum cache] *********************************************** skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:25.077 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:25 +0000 (0:00:00.067) 0:01:00.577 ***** 2019-12-05T04:58:25.113 INFO:teuthology.task.ansible.out: TASK [common : Uncomment mirrorlist] ******************************************* skipping: [smithi042.front.sepia.ceph.com] => (item={'value': {u'mirrorlist': u'file:///etc/yum.repos.d/epel-mirrorlist', u'name': u'Extra Packages for Enterprise Linux', u'failovermethod': u'priority', u'gpgcheck': 0, u'enabled': u'1', u'baseurl': u'http://dl.fedoraproject.org/pub/epel/18/$basearch'}, 'key': u'epel'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel", "value": {"baseurl": "http://dl.fedoraproject.org/pub/epel/18/$basearch", "enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:25.116 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => (item={'value': {u'mirrorlist': u'file:///etc/yum.repos.d/epel-testing-mirrorlist', u'name': u'Extra Packages for Enterprise Linux - Testing', u'failovermethod': u'priority', u'gpgcheck': 0, u'enabled': 0, u'baseurl': u'http://dl.fedoraproject.org/pub/epel/testing/18/$basearch'}, 'key': u'epel-testing'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel-testing", "value": {"baseurl": "http://dl.fedoraproject.org/pub/epel/testing/18/$basearch", "enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:25.165 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:25 +0000 (0:00:00.081) 0:01:00.659 ***** 2019-12-05T04:58:25.331 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2019-12-05T04:58:25.332 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => {"changed": false, "cmd": "cat /etc/timezone", "delta": "0:00:00.009681", "end": "2019-12-05 04:58:25.669711", "rc": 0, "start": "2019-12-05 04:58:25.660030", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2019-12-05T04:58:25.371 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:25 +0000 (0:00:00.213) 0:01:00.872 ***** 2019-12-05T04:58:25.397 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:25.438 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:25 +0000 (0:00:00.067) 0:01:00.940 ***** 2019-12-05T04:58:25.475 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:25.786 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:25 +0000 (0:00:00.083) 0:01:01.023 ***** TASK [common : Get the current timezone] *************************************** skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 05 December 2019 04:58:25 +0000 (0:00:00.074) 0:01:01.098 ***** TASK [common : Set the timezone] *********************************************** skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 05 December 2019 04:58:25 +0000 (0:00:00.069) 0:01:01.167 ***** TASK [common : Add base OpenSUSE Leap repo] ************************************ skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 05 December 2019 04:58:25 +0000 (0:00:00.065) 0:01:01.233 ***** TASK [common : Add updates OpenSUSE Leap repo] ********************************* skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:25.822 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:25 +0000 (0:00:00.090) 0:01:01.324 ***** 2019-12-05T04:58:25.858 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** 2019-12-05T04:58:25.860 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:25.906 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:25 +0000 (0:00:00.084) 0:01:01.409 ***** 2019-12-05T04:58:25.912 INFO:teuthology.task.ansible.out: TASK [common : Set the hardware clock] ***************************************** 2019-12-05T04:58:25.913 INFO:teuthology.task.ansible.out:changed: [smithi040.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.923695", "end": "2019-12-05 04:58:25.505148", "rc": 0, "start": "2019-12-05 04:58:24.581453", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-12-05T04:58:25.960 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:25 +0000 (0:00:00.051) 0:01:01.461 ***** 2019-12-05T04:58:26.006 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** 2019-12-05T04:58:26.019 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:26.048 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:26 +0000 (0:00:00.089) 0:01:01.550 ***** 2019-12-05T04:58:26.172 INFO:teuthology.task.ansible.out: TASK [common : Set the hardware clock] ***************************************** changed: [smithi042.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.132432", "end": "2019-12-05 04:58:26.505190", "rc": 0, "start": "2019-12-05 04:58:26.372758", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-12-05T04:58:26.252 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:26 +0000 (0:00:00.203) 0:01:01.754 ***** 2019-12-05T04:58:26.264 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ [WARNING]: Consider using the apt module rather than running 'apt-get'. If you need to use command because apt is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message. 2019-12-05T04:58:26.265 INFO:teuthology.task.ansible.out:changed: [smithi040.front.sepia.ceph.com] => {"changed": true, "cmd": ["apt-get", "clean"], "delta": "0:00:00.051000", "end": "2019-12-05 04:58:25.866074", "rc": 0, "start": "2019-12-05 04:58:25.815074", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-12-05T04:58:26.315 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:26 +0000 (0:00:00.063) 0:01:01.818 ***** 2019-12-05T04:58:26.328 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:26.365 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:26 +0000 (0:00:00.049) 0:01:01.867 ***** 2019-12-05T04:58:26.565 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ 2019-12-05T04:58:26.566 INFO:teuthology.task.ansible.out:changed: [smithi042.front.sepia.ceph.com] => {"changed": true, "cmd": ["apt-get", "clean"], "delta": "0:00:00.065388", "end": "2019-12-05 04:58:26.908153", "rc": 0, "start": "2019-12-05 04:58:26.842765", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-12-05T04:58:26.608 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:26 +0000 (0:00:00.243) 0:01:02.111 ***** 2019-12-05T04:58:30.018 INFO:teuthology.task.ansible.out: TASK [common : Update apt cache] *********************************************** 2019-12-05T04:58:30.020 INFO:teuthology.task.ansible.out: [WARNING]: Could not find aptitude. Using apt-get instead ok: [smithi040.front.sepia.ceph.com] => {"attempts": 1, "cache_update_time": 1575521656, "cache_updated": false, "changed": false} 2019-12-05T04:58:30.071 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:30 +0000 (0:00:03.461) 0:01:05.572 ***** 2019-12-05T04:58:30.123 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => {"attempts": 1, "cache_update_time": 1575521611, "cache_updated": false, "changed": false} 2019-12-05T04:58:30.164 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:30 +0000 (0:00:00.094) 0:01:05.666 ***** 2019-12-05T04:58:30.975 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** ok: [smithi040.front.sepia.ceph.com] => {"cache_update_time": 1575521656, "cache_updated": false, "changed": false} 2019-12-05T04:58:31.022 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:31 +0000 (0:00:00.858) 0:01:06.525 ***** 2019-12-05T04:58:31.029 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => {"cache_update_time": 1575521611, "cache_updated": false, "changed": false} 2019-12-05T04:58:31.071 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:31 +0000 (0:00:00.048) 0:01:06.573 ***** 2019-12-05T04:58:31.076 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** 2019-12-05T04:58:31.076 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:31.129 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:31 +0000 (0:00:00.058) 0:01:06.632 ***** 2019-12-05T04:58:31.135 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:31.178 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:31 +0000 (0:00:00.049) 0:01:06.681 ***** 2019-12-05T04:58:31.663 INFO:teuthology.task.ansible.out: TASK [common : Add krb5 config file] ******************************************* 2019-12-05T04:58:31.665 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "size": 131, "state": "file", "uid": 0} ok: [smithi042.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "size": 131, "state": "file", "uid": 0} 2019-12-05T04:58:31.666 INFO:teuthology.task.ansible.out: 2019-12-05T04:58:31.708 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:31 +0000 (0:00:00.529) 0:01:07.211 ***** 2019-12-05T04:58:31.754 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:31 +0000 (0:00:00.046) 0:01:07.257 ***** 2019-12-05T04:58:32.568 INFO:teuthology.task.ansible.out: TASK [common : Make sure smartmontools is installed] *************************** ok: [smithi040.front.sepia.ceph.com] => {"cache_update_time": 1575521656, "cache_updated": false, "changed": false} 2019-12-05T04:58:32.628 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:32 +0000 (0:00:00.873) 0:01:08.131 ***** 2019-12-05T04:58:32.730 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => {"cache_update_time": 1575521611, "cache_updated": false, "changed": false} 2019-12-05T04:58:32.772 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:32 +0000 (0:00:00.143) 0:01:08.274 ***** 2019-12-05T04:58:32.967 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** ok: [smithi040.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "size": 2716736, "state": "file", "uid": 0} 2019-12-05T04:58:33.153 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "size": 2716736, "state": "file", "uid": 0} 2019-12-05T04:58:33.293 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "size": 1952038, "state": "file", "uid": 0} 2019-12-05T04:58:33.492 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "size": 1952038, "state": "file", "uid": 0} 2019-12-05T04:58:33.609 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "size": 580432, "state": "file", "uid": 0} 2019-12-05T04:58:33.676 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:33 +0000 (0:00:00.897) 0:01:09.171 ***** 2019-12-05T04:58:33.803 INFO:teuthology.task.ansible.out: TASK [common : Create /usr/libexec.] ******************************************* 2019-12-05T04:58:33.805 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "size": 4096, "state": "directory", "uid": 0} 2019-12-05T04:58:33.848 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:33 +0000 (0:00:00.178) 0:01:09.350 ***** 2019-12-05T04:58:33.855 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2019-12-05T04:58:33.856 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "size": 580432, "state": "file", "uid": 0} 2019-12-05T04:58:33.931 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:33 +0000 (0:00:00.073) 0:01:09.424 ***** 2019-12-05T04:58:34.106 INFO:teuthology.task.ansible.out: TASK [common : Create /usr/libexec.] ******************************************* 2019-12-05T04:58:34.107 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "size": 4096, "state": "directory", "uid": 0} 2019-12-05T04:58:34.176 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:34 +0000 (0:00:00.254) 0:01:09.678 ***** 2019-12-05T04:58:34.256 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2019-12-05T04:58:34.257 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "9a447e184389aadb9aaf1e77f30f0a7f8b716c4e", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "size": 12725, "state": "file", "uid": 0} 2019-12-05T04:58:34.539 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "9a447e184389aadb9aaf1e77f30f0a7f8b716c4e", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "size": 12725, "state": "file", "uid": 0} 2019-12-05T04:58:34.585 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "size": 8602, "state": "file", "uid": 0} 2019-12-05T04:58:34.854 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "size": 8602, "state": "file", "uid": 0} 2019-12-05T04:58:34.920 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9c74425885a83f23bc37817de46979ca5173ccb7", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "size": 3605, "state": "file", "uid": 0} 2019-12-05T04:58:34.966 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:34 +0000 (0:00:00.789) 0:01:10.468 ***** 2019-12-05T04:58:35.004 INFO:teuthology.task.ansible.out: TASK [common : Include apt_system vars] **************************************** ok: [smithi040.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib/nagios/plugins", "nrpe_group": "nagios", "nrpe_packages": ["nagios-nrpe-server", "nagios-plugins-basic"], "nrpe_service_name": "nagios-nrpe-server", "nrpe_user": "nagios"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/apt_systems.yml"], "changed": false} 2019-12-05T04:58:35.072 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:35 +0000 (0:00:00.104) 0:01:10.572 ***** 2019-12-05T04:58:35.101 INFO:teuthology.task.ansible.out: TASK [common : Include yum_systems vars] *************************************** skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:35.144 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:35 +0000 (0:00:00.072) 0:01:10.644 ***** 2019-12-05T04:58:35.158 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2019-12-05T04:58:35.159 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9c74425885a83f23bc37817de46979ca5173ccb7", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "size": 3605, "state": "file", "uid": 0} 2019-12-05T04:58:35.202 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:35 +0000 (0:00:00.060) 0:01:10.704 ***** 2019-12-05T04:58:35.223 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* skipping: [smithi040.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2019-12-05T04:58:35.249 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:35 +0000 (0:00:00.047) 0:01:10.752 ***** 2019-12-05T04:58:35.255 INFO:teuthology.task.ansible.out: TASK [common : Include apt_system vars] **************************************** 2019-12-05T04:58:35.256 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib/nagios/plugins", "nrpe_group": "nagios", "nrpe_packages": ["nagios-nrpe-server", "nagios-plugins-basic"], "nrpe_service_name": "nagios-nrpe-server", "nrpe_user": "nagios"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/apt_systems.yml"], "changed": false} 2019-12-05T04:58:35.299 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:35 +0000 (0:00:00.049) 0:01:10.802 ***** 2019-12-05T04:58:35.307 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** 2019-12-05T04:58:35.309 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2019-12-05T04:58:35.311 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2019-12-05T04:58:35.312 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} skipping: [smithi040.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2019-12-05T04:58:35.360 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:35 +0000 (0:00:00.060) 0:01:10.862 ***** 2019-12-05T04:58:35.366 INFO:teuthology.task.ansible.out: TASK [common : Include yum_systems vars] *************************************** skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:35.417 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:35 +0000 (0:00:00.057) 0:01:10.920 ***** 2019-12-05T04:58:35.419 INFO:teuthology.task.ansible.out: 2019-12-05T04:58:35.429 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:35.471 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:35 +0000 (0:00:00.053) 0:01:10.974 ***** 2019-12-05T04:58:35.477 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* 2019-12-05T04:58:35.477 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2019-12-05T04:58:35.522 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:35 +0000 (0:00:00.049) 0:01:11.023 ***** 2019-12-05T04:58:35.568 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** skipping: [smithi042.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2019-12-05T04:58:35.569 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2019-12-05T04:58:35.579 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2019-12-05T04:58:35.581 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2019-12-05T04:58:35.624 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:35 +0000 (0:00:00.103) 0:01:11.127 ***** 2019-12-05T04:58:35.651 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ 2019-12-05T04:58:35.652 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:35.717 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:35 +0000 (0:00:00.075) 0:01:11.202 ***** 2019-12-05T04:58:36.375 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (Ubuntu)] ***************** ok: [smithi040.front.sepia.ceph.com] => {"cache_update_time": 1575521656, "cache_updated": false, "changed": false} 2019-12-05T04:58:36.433 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:36 +0000 (0:00:00.733) 0:01:11.936 ***** 2019-12-05T04:58:36.616 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => {"cache_update_time": 1575521611, "cache_updated": false, "changed": false} 2019-12-05T04:58:36.662 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:36 +0000 (0:00:00.228) 0:01:12.164 ***** 2019-12-05T04:58:36.762 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios sudoers.d for raid utilities.] ******************** 2019-12-05T04:58:36.762 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => {"changed": false, "checksum": "4880e187ff9f437cc33858eb1d0fa49a06d67be4", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "size": 180, "state": "file", "uid": 0} 2019-12-05T04:58:36.819 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:36 +0000 (0:00:00.157) 0:01:12.322 ***** 2019-12-05T04:58:37.016 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => {"changed": false, "checksum": "4880e187ff9f437cc33858eb1d0fa49a06d67be4", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "size": 180, "state": "file", "uid": 0} 2019-12-05T04:58:37.070 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:37 +0000 (0:00:00.250) 0:01:12.572 ***** 2019-12-05T04:58:37.168 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* 2019-12-05T04:58:37.170 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib/nagios/plugins/check_mem.sh", "size": 1964, "state": "file", "uid": 0} 2019-12-05T04:58:37.221 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:37 +0000 (0:00:00.145) 0:01:12.718 ***** 2019-12-05T04:58:37.365 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2019-12-05T04:58:37.367 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2019-12-05T04:58:37.409 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:37 +0000 (0:00:00.190) 0:01:12.908 ***** 2019-12-05T04:58:37.439 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:37.491 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:37 +0000 (0:00:00.085) 0:01:12.994 ***** 2019-12-05T04:58:37.501 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* ok: [smithi042.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib/nagios/plugins/check_mem.sh", "size": 1964, "state": "file", "uid": 0} 2019-12-05T04:58:37.549 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:37 +0000 (0:00:00.057) 0:01:13.051 ***** 2019-12-05T04:58:37.555 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** 2019-12-05T04:58:37.557 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2019-12-05T04:58:37.606 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:37 +0000 (0:00:00.057) 0:01:13.108 ***** 2019-12-05T04:58:37.641 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:37.680 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:37 +0000 (0:00:00.073) 0:01:13.182 ***** 2019-12-05T04:58:37.695 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2019-12-05T04:58:37.696 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2019-12-05T04:58:37.751 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:37 +0000 (0:00:00.070) 0:01:13.252 ***** 2019-12-05T04:58:37.793 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:37.844 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:37 +0000 (0:00:00.093) 0:01:13.346 ***** 2019-12-05T04:58:37.877 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** skipping: [smithi042.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2019-12-05T04:58:37.915 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:37 +0000 (0:00:00.071) 0:01:13.417 ***** 2019-12-05T04:58:37.952 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:38.010 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:38 +0000 (0:00:00.095) 0:01:13.513 ***** 2019-12-05T04:58:38.075 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios nrpe config.] ************************************* ok: [smithi040.front.sepia.ceph.com] => {"changed": false, "checksum": "9c3d1d79583cf76a6d81bb2ec150bc4af5ca3c5c", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "size": 1166, "state": "file", "uid": 0} 2019-12-05T04:58:38.149 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:38 +0000 (0:00:00.138) 0:01:13.652 ***** 2019-12-05T04:58:38.381 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => {"changed": false, "checksum": "9c3d1d79583cf76a6d81bb2ec150bc4af5ca3c5c", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "size": 1166, "state": "file", "uid": 0} 2019-12-05T04:58:38.423 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:38 +0000 (0:00:00.273) 0:01:13.926 ***** 2019-12-05T04:58:38.847 INFO:teuthology.task.ansible.out: TASK [common : Make sure nagios nrpe service is running.] ********************** 2019-12-05T04:58:38.849 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nagios-nrpe-server", "state": "started", "status": {"ActiveEnterTimestamp": "Thu 2019-12-05 04:56:43 UTC", "ActiveEnterTimestampMonotonic": "18635669", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "-.mount systemd-tmpfiles-setup.service sysinit.target systemd-journald.socket var-run.mount remote-fs.target system.slice time-sync.target network.target local-fs.target nss-lookup.target basic.target", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Thu 2019-12-05 04:56:43 UTC", "AssertTimestampMonotonic": "18634937", "Before": "multi-user.target plymouth-quit.service shutdown.target xdm.service getty@tty1.service", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Thu 2019-12-05 04:56:43 UTC", "ConditionTimestampMonotonic": "18634937", "ConfigurationDirectoryMode": "0755", "Conflicts": "nrpe.socket shutdown.target", "ControlGroup": "/system.slice/nagios-nrpe-ser 2019-12-05T04:58:38.851 INFO:teuthology.task.ansible.out:ver.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Nagios Remote Plugin Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "DynamicUser": "no", "EnvironmentFile": "/etc/default/nagios-nrpe-server (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1097", "ExecMainStartTimestamp": "Thu 2019-12-05 04:56:43 UTC", "ExecMainStartTimestampMonotonic": "18635637", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f $NRPE_OPTS ; ignore_errors=no ; start_time=[Thu 2019-12-05 04:56:43 UTC] ; stop_time=[n/a] ; pid=1097 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /var/run/nagios/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/nagios-nrpe-server.service", "GID": "118", "Group": "nagios", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "nagios-nrpe-server.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Thu 2019-12-05 04:56:43 UTC", "InactiveExitTimestampMonotonic": "18635669", "InvocationID": "38a3f531beea4a8093e27f0990135834", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "i 2019-12-05T04:58:38.852 INFO:teuthology.task.ansible.out:nfinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128059", "LimitNPROCSoft": "128059", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128059", "LimitSIGPENDINGSoft": "128059", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1097", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "nagios-nrpe-server.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "-500", "OnFailureJobMode": "replace", "PIDFile": "/var/run/nagios/nrpe.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice sysinit.target -.mount", "RequiresMountsFor": "/tmp /var/tmp", "Restart": "on-abort", "RestartUSec": "100ms", 2019-12-05T04:58:38.854 INFO:teuthology.task.ansible.out: "RestrictNamespaces": "no", "RestrictRealtime": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Thu 2019-12-05 04:56:43 UTC", "StateChangeTimestampMonotonic": "18635669", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UID": "109", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "User": "nagios", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Thu 2019-12-05 04:56:43 UTC", "WatchdogTimestampMonotonic": "18635668", "WatchdogUSec": "0"}} 2019-12-05T04:58:38.861 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nagios-nrpe-server", "state": "started", "status": {"ActiveEnterTimestamp": "Thu 2019-12-05 04:56:53 UTC", "ActiveEnterTimestampMonotonic": "19446429", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "sysinit.target time-sync.target local-fs.target basic.target -.mount system.slice remote-fs.target systemd-journald.socket var-run.mount nss-lookup.target network.target systemd-tmpfiles-setup.service", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Thu 2019-12-05 04:56:53 UTC", "AssertTimestampMonotonic": "19444506", "Before": "getty@tty1.service plymouth-quit.service xdm.service shutdown.target multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Thu 2019-12-05 04:56:53 UTC", "ConditionTimestampMonotonic": "19444506", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target nrpe.socket", "ControlGroup": "/system.slice/nagios-nrpe-ser 2019-12-05T04:58:38.864 INFO:teuthology.task.ansible.out:ver.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Nagios Remote Plugin Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "DynamicUser": "no", "EnvironmentFile": "/etc/default/nagios-nrpe-server (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1281", "ExecMainStartTimestamp": "Thu 2019-12-05 04:56:53 UTC", "ExecMainStartTimestampMonotonic": "19446364", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f $NRPE_OPTS ; ignore_errors=no ; start_time=[Thu 2019-12-05 04:56:53 UTC] ; stop_time=[n/a] ; pid=1281 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /var/run/nagios/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/nagios-nrpe-server.service", "GID": "118", "Group": "nagios", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "nagios-nrpe-server.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Thu 2019-12-05 04:56:53 UTC", "InactiveExitTimestampMonotonic": "19446429", "InvocationID": "453008d364e949108a3e4ed16bb7d88a", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "i 2019-12-05T04:58:38.866 INFO:teuthology.task.ansible.out:nfinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128059", "LimitNPROCSoft": "128059", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128059", "LimitSIGPENDINGSoft": "128059", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1281", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "nagios-nrpe-server.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "-500", "OnFailureJobMode": "replace", "PIDFile": "/var/run/nagios/nrpe.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "sysinit.target system.slice -.mount", "RequiresMountsFor": "/var/tmp /tmp", "Restart": "on-abort", "RestartUSec": "100ms", 2019-12-05T04:58:38.867 INFO:teuthology.task.ansible.out: "RestrictNamespaces": "no", "RestrictRealtime": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Thu 2019-12-05 04:56:53 UTC", "StateChangeTimestampMonotonic": "19446429", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UID": "109", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "User": "nagios", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Thu 2019-12-05 04:56:53 UTC", "WatchdogTimestampMonotonic": "19446427", "WatchdogUSec": "0"}} 2019-12-05T04:58:38.902 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:38 +0000 (0:00:00.479) 0:01:14.405 ***** 2019-12-05T04:58:38.942 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:38 +0000 (0:00:00.039) 0:01:14.445 ***** 2019-12-05T04:58:38.948 INFO:teuthology.task.ansible.out: 2019-12-05T04:58:38.949 INFO:teuthology.task.ansible.out:TASK [common : Get SELinux status] ********************************************* skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:38.991 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:38 +0000 (0:00:00.049) 0:01:14.494 ***** 2019-12-05T04:58:38.998 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:39.050 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:39 +0000 (0:00:00.055) 0:01:14.550 ***** 2019-12-05T04:58:39.053 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Install SELinux dependencies] **************************** 2019-12-05T04:58:39.053 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:39.095 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:39 +0000 (0:00:00.048) 0:01:14.598 ***** 2019-12-05T04:58:39.112 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:39.160 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:39 +0000 (0:00:00.063) 0:01:14.661 ***** 2019-12-05T04:58:39.175 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Ensure SELinux policy is up to date] ********************* skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:39.229 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:39 +0000 (0:00:00.070) 0:01:14.732 ***** 2019-12-05T04:58:39.236 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:39.281 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:39 +0000 (0:00:00.051) 0:01:14.783 ***** 2019-12-05T04:58:39.287 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Set SELinux boolean nagios_run_sudo true] **************** 2019-12-05T04:58:39.288 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:39.330 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:39 +0000 (0:00:00.049) 0:01:14.833 ***** 2019-12-05T04:58:39.337 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:39.378 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:39 +0000 (0:00:00.046) 0:01:14.880 ***** 2019-12-05T04:58:39.383 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove SELinux policy package] *************************** 2019-12-05T04:58:39.385 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:39.427 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:39 +0000 (0:00:00.049) 0:01:14.929 ***** 2019-12-05T04:58:39.434 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:39.477 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:39 +0000 (0:00:00.049) 0:01:14.979 ***** 2019-12-05T04:58:39.482 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Copy SELinux type enforcement file] ********************** 2019-12-05T04:58:39.483 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:39.525 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:39 +0000 (0:00:00.048) 0:01:15.028 ***** 2019-12-05T04:58:39.531 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:39.580 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:39 +0000 (0:00:00.055) 0:01:15.083 ***** 2019-12-05T04:58:39.592 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Compile SELinux module file] ***************************** 2019-12-05T04:58:39.594 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:39.635 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:39 +0000 (0:00:00.054) 0:01:15.137 ***** 2019-12-05T04:58:39.644 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:39.684 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:39 +0000 (0:00:00.049) 0:01:15.187 ***** 2019-12-05T04:58:39.690 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Build SELinux policy package] **************************** 2019-12-05T04:58:39.690 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:39.738 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:39 +0000 (0:00:00.053) 0:01:15.241 ***** 2019-12-05T04:58:39.744 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:39.786 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:39 +0000 (0:00:00.047) 0:01:15.289 ***** 2019-12-05T04:58:39.791 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Load SELinux policy package] ***************************** 2019-12-05T04:58:39.793 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:39.834 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:39 +0000 (0:00:00.048) 0:01:15.337 ***** 2019-12-05T04:58:39.842 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:39.885 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:39 +0000 (0:00:00.047) 0:01:15.385 ***** 2019-12-05T04:58:39.889 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:39.955 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:39 +0000 (0:00:00.073) 0:01:15.458 ***** 2019-12-05T04:58:39.966 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:40.013 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:40 +0000 (0:00:00.057) 0:01:15.516 ***** 2019-12-05T04:58:40.020 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** 2019-12-05T04:58:40.022 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => {"ansible_facts": {"apt_repos": [], "ceph_packages_to_remove": ["ceph", "ceph-common", "libcephfs1", "radosgw", "python-ceph", "python-rados", "python-cephfs", "python-rbd", "librbd1", "librados2", "ceph-fs-common-dbg", "ceph-fs-common"], "common_apt_repos": [], "common_packages": [], "nfs_service": "nfs-kernel-server", "ntp_service_name": "ntp", "packages": [], "packages_to_remove": ["multipath-tools"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "ssh"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/apt_systems.yml"], "changed": false} 2019-12-05T04:58:40.066 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:40 +0000 (0:00:00.052) 0:01:15.569 ***** 2019-12-05T04:58:40.072 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => {"ansible_facts": {"apt_repos": [], "ceph_packages_to_remove": ["ceph", "ceph-common", "libcephfs1", "radosgw", "python-ceph", "python-rados", "python-cephfs", "python-rbd", "librbd1", "librados2", "ceph-fs-common-dbg", "ceph-fs-common"], "common_apt_repos": [], "common_packages": [], "nfs_service": "nfs-kernel-server", "ntp_service_name": "ntp", "packages": [], "packages_to_remove": ["multipath-tools"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "ssh"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/apt_systems.yml"], "changed": false} 2019-12-05T04:58:40.117 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:40 +0000 (0:00:00.051) 0:01:15.620 ***** 2019-12-05T04:58:40.126 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* ok: [smithi040.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml) => {"ansible_facts": {"common_packages": ["libfcgi0ldbl", "python-virtualenv", "python-dev", "libev-dev", "perl", "libwww-perl", "lsb-release", "build-essential", "sysstat", "gdb", "python-configobj", "python-gevent", "libedit2", "cryptsetup-bin", "xfsprogs", "gdisk", "parted", "libuuid1", "btrfs-tools", "libatomic-ops-dev", "git-core", "attr", "dbench", "bonnie++", "valgrind", "python-nose", "ant", "libtool", "automake", "gettext", "uuid-dev", "libacl1-dev", "bc", "xfsdump", "xfslibs-dev", "libattr1-dev", "quota", "libcap2-bin", "libncurses5-dev", "lvm2", "vim", "pdsh", "collectl", "blktrace", "python-numpy", "python-matplotlib", "genisoimage", "libjson-xs-perl", "xml-twig-tools", "default-jdk", "junit4", "tgt", "open-iscsi", "cifs-utils", "ipcalc", "nfs-common", "nfs-kernel-server", "software-properties-common"], "non_aarch64_common_packages": ["smbios-utils", "libfcgi", "sysprof"], "non_aarch64_packages_to_upgrade": ["libapache2-mod-fastcgi"], "packages_to_remove": ["openmpi-common"], "packages_to_upgrade": ["apt", "apache2"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"} 2019-12-05T04:58:40.175 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:40 +0000 (0:00:00.057) 0:01:15.677 ***** 2019-12-05T04:58:40.182 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml) => {"ansible_facts": {"common_packages": ["libfcgi0ldbl", "python-virtualenv", "python-dev", "libev-dev", "perl", "libwww-perl", "lsb-release", "build-essential", "sysstat", "gdb", "python-configobj", "python-gevent", "libedit2", "cryptsetup-bin", "xfsprogs", "gdisk", "parted", "libuuid1", "btrfs-tools", "libatomic-ops-dev", "git-core", "attr", "dbench", "bonnie++", "valgrind", "python-nose", "ant", "libtool", "automake", "gettext", "uuid-dev", "libacl1-dev", "bc", "xfsdump", "xfslibs-dev", "libattr1-dev", "quota", "libcap2-bin", "libncurses5-dev", "lvm2", "vim", "pdsh", "collectl", "blktrace", "python-numpy", "python-matplotlib", "genisoimage", "libjson-xs-perl", "xml-twig-tools", "default-jdk", "junit4", "tgt", "open-iscsi", "cifs-utils", "ipcalc", "nfs-common", "nfs-kernel-server", "software-properties-common"], "non_aarch64_common_packages": ["smbios-utils", "libfcgi", "sysprof"], "non_aarch64_packages_to_upgrade": ["libapache2-mod-fastcgi"], "packages_to_remove": ["openmpi-common"], "packages_to_upgrade": ["apt", "apache2"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"} 2019-12-05T04:58:40.253 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:40 +0000 (0:00:00.078) 0:01:15.756 ***** 2019-12-05T04:58:40.259 INFO:teuthology.task.ansible.out: TASK [testnode : Including major version specific variables.] ****************** 2019-12-05T04:58:40.260 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml) => {"ansible_facts": {"non_aarch64_packages": ["libgoogle-perftools4", "iozone3", "libssl1.0.0"], "non_aarch64_packages_to_upgrade": [], "packages": ["mpich", "qemu-system-x86", "lttng-tools", "libtool-bin", "docker.io"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml"} 2019-12-05T04:58:40.311 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:40 +0000 (0:00:00.057) 0:01:15.813 ***** 2019-12-05T04:58:40.319 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml) => {"ansible_facts": {"non_aarch64_packages": ["libgoogle-perftools4", "iozone3", "libssl1.0.0"], "non_aarch64_packages_to_upgrade": [], "packages": ["mpich", "qemu-system-x86", "lttng-tools", "libtool-bin", "docker.io"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml"} 2019-12-05T04:58:40.368 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:40 +0000 (0:00:00.056) 0:01:15.870 ***** 2019-12-05T04:58:40.388 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ ok: [smithi040.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2019-12-05T04:58:40.423 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:40 +0000 (0:00:00.054) 0:01:15.925 ***** 2019-12-05T04:58:40.431 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2019-12-05T04:58:40.474 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:40 +0000 (0:00:00.051) 0:01:15.977 ***** 2019-12-05T04:58:40.565 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the sudo group exists.] ******************************** 2019-12-05T04:58:40.567 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} 2019-12-05T04:58:40.614 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:40 +0000 (0:00:00.139) 0:01:16.117 ***** 2019-12-05T04:58:40.628 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} 2019-12-05T04:58:40.676 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:40 +0000 (0:00:00.061) 0:01:16.179 ***** 2019-12-05T04:58:40.756 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* ok: [smithi040.front.sepia.ceph.com] => {"changed": false, "gid": 1216, "name": "ubuntu", "state": "present", "system": false} 2019-12-05T04:58:40.801 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:40 +0000 (0:00:00.124) 0:01:16.304 ***** 2019-12-05T04:58:40.818 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => {"changed": false, "gid": 1216, "name": "ubuntu", "state": "present", "system": false} 2019-12-05T04:58:40.863 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:40 +0000 (0:00:00.059) 0:01:16.363 ***** 2019-12-05T04:58:40.867 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** 2019-12-05T04:58:40.868 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:40.933 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:40 +0000 (0:00:00.073) 0:01:16.436 ***** 2019-12-05T04:58:40.947 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:41.009 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:41 +0000 (0:00:00.075) 0:01:16.512 ***** 2019-12-05T04:58:41.019 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:41.079 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:41 +0000 (0:00:00.060) 0:01:16.573 ***** skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:41.129 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:41 +0000 (0:00:00.058) 0:01:16.631 ***** 2019-12-05T04:58:41.229 INFO:teuthology.task.ansible.out: TASK [testnode : Add a user for xfstests to test user quotas.] ***************** 2019-12-05T04:58:41.231 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/sh", "state": "present", "uid": 10101} 2019-12-05T04:58:41.293 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:41 +0000 (0:00:00.164) 0:01:16.796 ***** 2019-12-05T04:58:41.300 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/sh", "state": "present", "uid": 10101} 2019-12-05T04:58:41.351 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:41 +0000 (0:00:00.051) 0:01:16.847 ***** 2019-12-05T04:58:41.723 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** ok: [smithi040.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "size": 112, "state": "file", "uid": 0} 2019-12-05T04:58:41.770 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:41 +0000 (0:00:00.424) 0:01:17.271 ***** 2019-12-05T04:58:41.788 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "size": 112, "state": "file", "uid": 0} 2019-12-05T04:58:41.830 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:41 +0000 (0:00:00.061) 0:01:17.333 ***** 2019-12-05T04:58:41.836 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* 2019-12-05T04:58:41.837 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi040"}, "changed": false} 2019-12-05T04:58:41.883 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:41 +0000 (0:00:00.052) 0:01:17.385 ***** 2019-12-05T04:58:41.889 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi042"}, "changed": false} 2019-12-05T04:58:41.936 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:41 +0000 (0:00:00.048) 0:01:17.434 ***** 2019-12-05T04:58:42.840 INFO:teuthology.task.ansible.out: TASK [testnode : Set the system's hostname] ************************************ changed: [smithi040.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "front.sepia.ceph.com", "ansible_fqdn": "smithi040.front.sepia.ceph.com", "ansible_hostname": "smithi040", "ansible_nodename": "smithi040"}, "changed": true, "name": "smithi040"} changed: [smithi042.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "front.sepia.ceph.com", "ansible_fqdn": "smithi042.front.sepia.ceph.com", "ansible_hostname": "smithi042", "ansible_nodename": "smithi042"}, "changed": true, "name": "smithi042"} 2019-12-05T04:58:42.882 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:42 +0000 (0:00:00.950) 0:01:18.385 ***** 2019-12-05T04:58:42.926 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:42 +0000 (0:00:00.043) 0:01:18.428 ***** 2019-12-05T04:58:43.226 INFO:teuthology.task.ansible.out: TASK [testnode : Upload distro major version specific sshd_config] ************* 2019-12-05T04:58:43.227 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => {"changed": false, "checksum": "2f7bfea2455a0d9220592aa0c21ac69c312e7471", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "size": 2646, "state": "file", "uid": 0} 2019-12-05T04:58:43.270 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:43 +0000 (0:00:00.344) 0:01:18.773 ***** 2019-12-05T04:58:43.288 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => {"changed": false, "checksum": "2f7bfea2455a0d9220592aa0c21ac69c312e7471", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "size": 2646, "state": "file", "uid": 0} 2019-12-05T04:58:43.330 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:43 +0000 (0:00:00.060) 0:01:18.833 ***** 2019-12-05T04:58:43.599 INFO:teuthology.task.ansible.out: TASK [testnode : Upload ssh_config] ******************************************** 2019-12-05T04:58:43.600 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "size": 570, "state": "file", "uid": 0} 2019-12-05T04:58:43.645 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:43 +0000 (0:00:00.314) 0:01:19.148 ***** 2019-12-05T04:58:43.674 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "size": 570, "state": "file", "uid": 0} 2019-12-05T04:58:43.719 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:43 +0000 (0:00:00.073) 0:01:19.221 ***** 2019-12-05T04:58:44.202 INFO:teuthology.task.ansible.out: TASK [testnode : Add ssh pubkeys] ********************************************** 2019-12-05T04:58:44.210 INFO:teuthology.task.ansible.out: [WARNING]: Module remote_tmp /root/.ansible/tmp did not exist and was created with a mode of 0700, this may cause issues when running as another user. To avoid this, create the remote_tmp dir with the correct permissions manually changed: [smithi040.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2019-12-05T04:58:44.253 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:44 +0000 (0:00:00.534) 0:01:19.756 ***** 2019-12-05T04:58:44.259 INFO:teuthology.task.ansible.out:changed: [smithi042.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2019-12-05T04:58:44.301 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:44 +0000 (0:00:00.047) 0:01:19.803 ***** 2019-12-05T04:58:44.306 INFO:teuthology.task.ansible.out: TASK [testnode : Create remote.conf] ******************************************* 2019-12-05T04:58:44.307 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:44.361 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:44 +0000 (0:00:00.049) 0:01:19.853 ***** skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:44.397 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:44 +0000 (0:00:00.046) 0:01:19.900 ***** 2019-12-05T04:58:44.403 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2019-12-05T04:58:44.404 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:44.446 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:44 +0000 (0:00:00.048) 0:01:19.949 ***** 2019-12-05T04:58:44.461 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:44.492 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:44 +0000 (0:00:00.046) 0:01:19.995 ***** 2019-12-05T04:58:44.497 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2019-12-05T04:58:44.499 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:44.538 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:44 +0000 (0:00:00.045) 0:01:20.041 ***** 2019-12-05T04:58:44.544 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:44.587 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:44 +0000 (0:00:00.049) 0:01:20.090 ***** 2019-12-05T04:58:44.592 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2019-12-05T04:58:44.594 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:44.663 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:44 +0000 (0:00:00.055) 0:01:20.145 ***** skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:44.714 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:44 +0000 (0:00:00.065) 0:01:20.211 ***** TASK [testnode : Configure /etc/sudoers.] ************************************** 2019-12-05T04:58:44.731 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:44.768 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:44 +0000 (0:00:00.056) 0:01:20.268 ***** 2019-12-05T04:58:44.774 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:44.826 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:44 +0000 (0:00:00.060) 0:01:20.329 ***** 2019-12-05T04:58:44.833 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:44.879 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:44 +0000 (0:00:00.052) 0:01:20.381 ***** 2019-12-05T04:58:44.885 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:44.932 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:44 +0000 (0:00:00.053) 0:01:20.435 ***** 2019-12-05T04:58:44.941 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:44.985 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:44 +0000 (0:00:00.052) 0:01:20.487 ***** 2019-12-05T04:58:44.991 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:45.043 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:45 +0000 (0:00:00.057) 0:01:20.545 ***** 2019-12-05T04:58:45.054 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* 2019-12-05T04:58:45.128 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:45 +0000 (0:00:00.085) 0:01:20.631 ***** 2019-12-05T04:58:45.187 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:45 +0000 (0:00:00.058) 0:01:20.690 ***** 2019-12-05T04:58:45.198 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2019-12-05T04:58:45.235 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:45 +0000 (0:00:00.047) 0:01:20.738 ***** 2019-12-05T04:58:45.285 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:45 +0000 (0:00:00.050) 0:01:20.788 ***** 2019-12-05T04:58:45.310 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG] ************************************************** skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:45.338 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:45 +0000 (0:00:00.052) 0:01:20.841 ***** 2019-12-05T04:58:45.347 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:45.390 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:45 +0000 (0:00:00.052) 0:01:20.893 ***** 2019-12-05T04:58:45.396 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG keys] ********************************************* 2019-12-05T04:58:45.398 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2019-12-05T04:58:45.399 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2019-12-05T04:58:45.447 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:45 +0000 (0:00:00.056) 0:01:20.950 ***** 2019-12-05T04:58:45.454 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2019-12-05T04:58:45.455 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2019-12-05T04:58:45.501 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:45 +0000 (0:00:00.049) 0:01:21.000 ***** 2019-12-05T04:58:45.503 INFO:teuthology.task.ansible.out: TASK [testnode : Clean yum cache] ********************************************** 2019-12-05T04:58:45.504 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:45.547 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:45 +0000 (0:00:00.049) 0:01:21.050 ***** 2019-12-05T04:58:45.553 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:45.595 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:45 +0000 (0:00:00.047) 0:01:21.097 ***** 2019-12-05T04:58:45.600 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2019-12-05T04:58:45.603 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:45.643 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:45 +0000 (0:00:00.047) 0:01:21.145 ***** 2019-12-05T04:58:45.650 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:45.690 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:45 +0000 (0:00:00.047) 0:01:21.193 ***** 2019-12-05T04:58:45.696 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure yum_utils is present.] ********************************* 2019-12-05T04:58:45.697 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:45.746 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:45 +0000 (0:00:00.055) 0:01:21.249 ***** 2019-12-05T04:58:45.753 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:45.795 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:45 +0000 (0:00:00.048) 0:01:21.298 ***** 2019-12-05T04:58:45.801 INFO:teuthology.task.ansible.out: TASK [testnode : Removing saved yum transactions] ****************************** 2019-12-05T04:58:45.801 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:45.844 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:45 +0000 (0:00:00.049) 0:01:21.347 ***** 2019-12-05T04:58:45.851 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:45.896 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:45 +0000 (0:00:00.050) 0:01:21.398 ***** 2019-12-05T04:58:45.901 INFO:teuthology.task.ansible.out: TASK [testnode : Check if ceph-debuginfo is installed] ************************* 2019-12-05T04:58:45.904 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:45.950 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:45 +0000 (0:00:00.053) 0:01:21.452 ***** 2019-12-05T04:58:45.956 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:46.007 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:46 +0000 (0:00:00.057) 0:01:21.509 ***** 2019-12-05T04:58:46.014 INFO:teuthology.task.ansible.out: TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** 2019-12-05T04:58:46.014 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:46.071 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:46 +0000 (0:00:00.064) 0:01:21.574 ***** 2019-12-05T04:58:46.080 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:46.133 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:46 +0000 (0:00:00.061) 0:01:21.636 ***** 2019-12-05T04:58:46.141 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2019-12-05T04:58:46.141 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:46.195 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:46 +0000 (0:00:00.061) 0:01:21.698 ***** 2019-12-05T04:58:46.203 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:46.244 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:46 +0000 (0:00:00.048) 0:01:21.746 ***** 2019-12-05T04:58:46.249 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2019-12-05T04:58:46.250 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:46.295 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:46 +0000 (0:00:00.051) 0:01:21.798 ***** 2019-12-05T04:58:46.311 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:46.345 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:46 +0000 (0:00:00.049) 0:01:21.847 ***** 2019-12-05T04:58:46.350 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2019-12-05T04:58:46.352 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:46.398 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:46 +0000 (0:00:00.053) 0:01:21.901 ***** 2019-12-05T04:58:46.405 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:46.448 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:46 +0000 (0:00:00.049) 0:01:21.950 ***** 2019-12-05T04:58:46.453 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** 2019-12-05T04:58:46.454 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:46.507 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:46 +0000 (0:00:00.059) 0:01:22.010 ***** 2019-12-05T04:58:46.515 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:46.556 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:46 +0000 (0:00:00.048) 0:01:22.058 ***** 2019-12-05T04:58:46.562 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2019-12-05T04:58:46.562 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:46.605 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:46 +0000 (0:00:00.049) 0:01:22.108 ***** 2019-12-05T04:58:46.615 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:46.654 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:46 +0000 (0:00:00.049) 0:01:22.157 ***** 2019-12-05T04:58:46.660 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2019-12-05T04:58:46.662 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:46.707 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:46 +0000 (0:00:00.052) 0:01:22.210 ***** 2019-12-05T04:58:46.715 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:46.761 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:46 +0000 (0:00:00.053) 0:01:22.264 ***** 2019-12-05T04:58:46.767 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable firewalld] *********************************** skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:46.827 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:46 +0000 (0:00:00.065) 0:01:22.330 ***** 2019-12-05T04:58:46.835 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:46.878 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:46 +0000 (0:00:00.050) 0:01:22.380 ***** 2019-12-05T04:58:46.884 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable iptables] ************************************ 2019-12-05T04:58:46.886 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:46.929 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:46 +0000 (0:00:00.050) 0:01:22.431 ***** 2019-12-05T04:58:46.944 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:46.984 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:46 +0000 (0:00:00.054) 0:01:22.486 ***** 2019-12-05T04:58:46.990 INFO:teuthology.task.ansible.out: TASK [testnode : Enable SELinux] *********************************************** 2019-12-05T04:58:46.991 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:47.040 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:47 +0000 (0:00:00.056) 0:01:22.543 ***** 2019-12-05T04:58:47.047 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:47.088 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:47 +0000 (0:00:00.048) 0:01:22.591 ***** 2019-12-05T04:58:47.185 INFO:teuthology.task.ansible.out: TASK [testnode : Check for custom repos] *************************************** ok: [smithi040.front.sepia.ceph.com] => {"changed": false, "cmd": "ls -1 /etc/apt/sources.list.d/", "delta": "0:00:00.003472", "end": "2019-12-05 04:58:46.822836", "rc": 0, "start": "2019-12-05 04:58:46.819364", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-12-05T04:58:47.229 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:47 +0000 (0:00:00.140) 0:01:22.731 ***** 2019-12-05T04:58:47.235 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => {"changed": false, "cmd": "ls -1 /etc/apt/sources.list.d/", "delta": "0:00:00.005319", "end": "2019-12-05 04:58:47.562689", "rc": 0, "start": "2019-12-05 04:58:47.557370", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-12-05T04:58:47.279 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:47 +0000 (0:00:00.049) 0:01:22.781 ***** 2019-12-05T04:58:47.285 INFO:teuthology.task.ansible.out: TASK [testnode : Remove custom repos] ****************************************** 2019-12-05T04:58:47.333 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:47 +0000 (0:00:00.054) 0:01:22.836 ***** 2019-12-05T04:58:47.388 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:47 +0000 (0:00:00.054) 0:01:22.890 ***** 2019-12-05T04:58:47.714 INFO:teuthology.task.ansible.out: TASK [testnode : Set apt preferences] ****************************************** 2019-12-05T04:58:47.715 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => {"changed": false, "checksum": "ba101482124806c7f6ea9fc7b4acfda303391ad6", "dest": "/etc/apt/preferences.d/ceph.pref", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/apt/preferences.d/ceph.pref", "size": 52, "state": "file", "uid": 0} 2019-12-05T04:58:47.765 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:47 +0000 (0:00:00.377) 0:01:23.267 ***** 2019-12-05T04:58:47.780 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => {"changed": false, "checksum": "ba101482124806c7f6ea9fc7b4acfda303391ad6", "dest": "/etc/apt/preferences.d/ceph.pref", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/apt/preferences.d/ceph.pref", "size": 52, "state": "file", "uid": 0} 2019-12-05T04:58:47.819 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:47 +0000 (0:00:00.054) 0:01:23.321 ***** 2019-12-05T04:58:47.825 INFO:teuthology.task.ansible.out: TASK [testnode : Add sources list] ********************************************* 2019-12-05T04:58:47.825 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:47.869 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:47 +0000 (0:00:00.050) 0:01:23.372 ***** 2019-12-05T04:58:47.879 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:58:47.939 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:47 +0000 (0:00:00.069) 0:01:23.441 ***** 2019-12-05T04:58:49.333 INFO:teuthology.task.ansible.out: TASK [testnode : Install apt keys] ********************************************* 2019-12-05T04:58:49.334 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc"} 2019-12-05T04:58:49.335 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc"} 2019-12-05T04:58:50.301 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc"} 2019-12-05T04:58:50.353 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:50 +0000 (0:00:02.413) 0:01:25.854 ***** 2019-12-05T04:58:50.360 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc"} 2019-12-05T04:58:50.417 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:50 +0000 (0:00:00.065) 0:01:25.920 ***** 2019-12-05T04:58:51.275 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-apt] ******************************************* 2019-12-05T04:58:51.276 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => {"cache_update_time": 1575521656, "cache_updated": false, "changed": false} 2019-12-05T04:58:51.329 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:51 +0000 (0:00:00.911) 0:01:26.831 ***** 2019-12-05T04:58:51.336 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => {"cache_update_time": 1575521611, "cache_updated": false, "changed": false} 2019-12-05T04:58:51.377 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:51 +0000 (0:00:00.048) 0:01:26.880 ***** 2019-12-05T04:58:51.383 INFO:teuthology.task.ansible.out: TASK [testnode : Add local apt repos.] ***************************************** 2019-12-05T04:58:51.441 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:51 +0000 (0:00:00.059) 0:01:26.940 ***** 2019-12-05T04:58:51.490 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:51 +0000 (0:00:00.053) 0:01:26.993 ***** 2019-12-05T04:58:51.613 INFO:teuthology.task.ansible.out: TASK [testnode : Blast the apt cache.] ***************************************** [WARNING]: Consider using 'become', 'become_method', and 'become_user' rather than running sudo changed: [smithi040.front.sepia.ceph.com] => {"changed": true, "cmd": ["sudo", "apt-get", "clean"], "delta": "0:00:00.037801", "end": "2019-12-05 04:58:51.251150", "rc": 0, "start": "2019-12-05 04:58:51.213349", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-12-05T04:58:51.677 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:51 +0000 (0:00:00.176) 0:01:27.170 ***** changed: [smithi042.front.sepia.ceph.com] => {"changed": true, "cmd": ["sudo", "apt-get", "clean"], "delta": "0:00:00.031616", "end": "2019-12-05 04:58:51.997633", "rc": 0, "start": "2019-12-05 04:58:51.966017", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-12-05T04:58:51.728 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:51 +0000 (0:00:00.060) 0:01:27.230 ***** 2019-12-05T04:58:54.614 INFO:teuthology.task.ansible.out: TASK [testnode : Update apt cache.] ******************************************** ok: [smithi040.front.sepia.ceph.com] => {"cache_update_time": 1575521656, "cache_updated": false, "changed": false} 2019-12-05T04:58:54.656 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:54 +0000 (0:00:02.928) 0:01:30.159 ***** 2019-12-05T04:58:54.864 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => {"cache_update_time": 1575521611, "cache_updated": false, "changed": false} 2019-12-05T04:58:54.916 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:58:54 +0000 (0:00:00.252) 0:01:30.411 ***** 2019-12-05T04:59:04.095 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure packages are not present.] ***************************** 2019-12-05T04:59:04.103 INFO:teuthology.task.ansible.out:changed: [smithi042.front.sepia.ceph.com] => {"changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\nThe following packages were automatically installed and are no longer required:\n cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good\n gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0\n libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1\n libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0\n libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0\n libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3\n libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0\n libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5\n libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils\nUse 'sudo apt autoremove' to remove them.\nThe following packages will be REMOVED:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\n0 upgraded, 0 newly installed, 6 to remove and 52 not upgraded.\nAfter this operation, 47.5 MB disk space will be freed.\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 137174 files and directorie 2019-12-05T04:59:04.114 INFO:teuthology.task.ansible.out:s currently installed.)\r\nRemoving qemu-utils (1:2.11+dfsg-1ubuntu7.19) ...\r\nRemoving qemu-system-x86 (1:2.11+dfsg-1ubuntu7.19) ...\r\nRemoving qemu-system-common (1:2.11+dfsg-1ubuntu7.19) ...\r\nRemoving qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.19) ...\r\nRemoving librbd1 (12.2.12-0ubuntu0.18.04.3) ...\r\nRemoving librados2 (12.2.12-0ubuntu0.18.04.3) ...\r\nProcessing triggers for man-db (2.8.3-2ubuntu0.1) ...\r\nProcessing triggers for libc-bin (2.27-3ubuntu1) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "The following packages were automatically installed and are no longer required:", " cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good", " gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0", " libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1", " libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0", " libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0", " libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3", " libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0", " libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5", " libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils", "Use 'sudo apt autoremove' to remove them.", "The following packages will be REMOVED:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "0 upgraded, 0 newly installed, 6 to remove and 52 not upgraded.", "After this operation, 47.5 MB disk space will be freed.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Readin 2019-12-05T04:59:04.119 INFO:teuthology.task.ansible.out:g database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 137174 files and directories currently installed.)", "Removing qemu-utils (1:2.11+dfsg-1ubuntu7.19) ...", "Removing qemu-system-x86 (1:2.11+dfsg-1ubuntu7.19) ...", "Removing qemu-system-common (1:2.11+dfsg-1ubuntu7.19) ...", "Removing qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.19) ...", "Removing librbd1 (12.2.12-0ubuntu0.18.04.3) ...", "Removing librados2 (12.2.12-0ubuntu0.18.04.3) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...", "Processing triggers for libc-bin (2.27-3ubuntu1) ..."]} 2019-12-05T04:59:04.155 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:04 +0000 (0:00:09.246) 0:01:39.658 ***** 2019-12-05T04:59:05.051 INFO:teuthology.task.ansible.out:changed: [smithi040.front.sepia.ceph.com] => {"changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\nThe following packages were automatically installed and are no longer required:\n cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good\n gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0\n libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1\n libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0\n libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0\n libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3\n libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0\n libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5\n libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils\nUse 'sudo apt autoremove' to remove them.\nThe following packages will be REMOVED:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\n0 upgraded, 0 newly installed, 6 to remove and 52 not upgraded.\nAfter this operation, 47.5 MB disk space will be freed.\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 137174 files and directorie 2019-12-05T04:59:05.053 INFO:teuthology.task.ansible.out:s currently installed.)\r\nRemoving qemu-utils (1:2.11+dfsg-1ubuntu7.19) ...\r\nRemoving qemu-system-x86 (1:2.11+dfsg-1ubuntu7.19) ...\r\nRemoving qemu-system-common (1:2.11+dfsg-1ubuntu7.19) ...\r\nRemoving qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.19) ...\r\nRemoving librbd1 (12.2.12-0ubuntu0.18.04.3) ...\r\nRemoving librados2 (12.2.12-0ubuntu0.18.04.3) ...\r\nProcessing triggers for man-db (2.8.3-2ubuntu0.1) ...\r\nProcessing triggers for libc-bin (2.27-3ubuntu1) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "The following packages were automatically installed and are no longer required:", " cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good", " gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0", " libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1", " libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0", " libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0", " libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3", " libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0", " libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5", " libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils", "Use 'sudo apt autoremove' to remove them.", "The following packages will be REMOVED:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "0 upgraded, 0 newly installed, 6 to remove and 52 not upgraded.", "After this operation, 47.5 MB disk space will be freed.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Readin 2019-12-05T04:59:05.055 INFO:teuthology.task.ansible.out:g database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 137174 files and directories currently installed.)", "Removing qemu-utils (1:2.11+dfsg-1ubuntu7.19) ...", "Removing qemu-system-x86 (1:2.11+dfsg-1ubuntu7.19) ...", "Removing qemu-system-common (1:2.11+dfsg-1ubuntu7.19) ...", "Removing qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.19) ...", "Removing librbd1 (12.2.12-0ubuntu0.18.04.3) ...", "Removing librados2 (12.2.12-0ubuntu0.18.04.3) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...", "Processing triggers for libc-bin (2.27-3ubuntu1) ..."]} 2019-12-05T04:59:05.110 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:05 +0000 (0:00:00.954) 0:01:40.613 ***** 2019-12-05T04:59:05.117 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2019-12-05T04:59:05.119 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => {"cache_update_time": 1575521611, "cache_updated": false, "changed": false} 2019-12-05T04:59:05.187 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:05 +0000 (0:00:00.076) 0:01:40.689 ***** 2019-12-05T04:59:05.226 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:05.269 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:05 +0000 (0:00:00.082) 0:01:40.772 ***** 2019-12-05T04:59:06.042 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2019-12-05T04:59:06.044 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => {"cache_update_time": 1575521656, "cache_updated": false, "changed": false} 2019-12-05T04:59:06.101 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:06 +0000 (0:00:00.828) 0:01:41.600 ***** 2019-12-05T04:59:06.140 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* 2019-12-05T04:59:06.141 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:06.190 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:06 +0000 (0:00:00.092) 0:01:41.693 ***** 2019-12-05T04:59:14.211 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* changed: [smithi042.front.sepia.ceph.com] => {"cache_update_time": 1575521611, "cache_updated": false, "changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\ngit is already the newest version (1:2.17.1-1ubuntu0.4).\nThe following additional packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils\nSuggested packages:\n samba vde2 sgabios ovmf debootstrap\nThe following NEW packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\n0 upgraded, 6 newly installed, 0 to remove and 52 not upgraded.\nNeed to get 10.4 MB of archives.\nAfter this operation, 47.5 MB of additional disk space will be used.\nGet:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librados2 amd64 12.2.12-0ubuntu0.18.04.3 [2702 kB]\nGet:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librbd1 amd64 12.2.12-0ubuntu0.18.04.3 [920 kB]\nGet:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-block-extra amd64 1:2.11+dfsg-1ubuntu7.21 [39.8 kB]\nGet:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-common amd64 1:2.11+dfsg-1ubuntu7.21 [672 kB]\nGet:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-x86 amd64 1:2.11+dfsg-1ubuntu7.21 [5197 kB]\nGet:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-utils amd64 1:2.11+dfsg-1ubuntu7.21 [868 kB]\nFetched 10.4 MB in 1s (9525 kB/s)\nSelecting previously unselected package librados2.\r\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database 2019-12-05T04:59:14.213 INFO:teuthology.task.ansible.out: ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 137032 files and directories currently installed.)\r\nPreparing to unpack .../0-librados2_12.2.12-0ubuntu0.18.04.3_amd64.deb ...\r\nUnpacking librados2 (12.2.12-0ubuntu0.18.04.3) ...\r\nSelecting previously unselected package librbd1.\r\nPreparing to unpack .../1-librbd1_12.2.12-0ubuntu0.18.04.3_amd64.deb ...\r\nUnpacking librbd1 (12.2.12-0ubuntu0.18.04.3) ...\r\nSelecting previously unselected package qemu-block-extra:amd64.\r\nPreparing to unpack .../2-qemu-block-extra_1%3a2.11+dfsg-1ubuntu7.21_amd64.deb ...\r\nUnpacking qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.21) ...\r\nSelecting previously unselected package qemu-system-common.\r\nPreparing to unpack .../3-qemu-system-common_1%3a2.11+dfsg-1ubuntu7.21_amd64.deb ...\r\nUnpacking qemu-system-common (1:2.11+dfsg-1ubuntu7.21) ...\r\nSelecting previously unselected package qemu-system-x86.\r\nPreparing to unpack .../4-qemu-system-x86_1%3a2.11+dfsg-1ubuntu7.21_amd64.deb ...\r\nUnpacking qemu-system-x86 (1:2.11+dfsg-1ubuntu7.21) ...\r\nSelecting previously unselected package qemu-utils.\r\nPreparing to unpack .../5-qemu-utils_1%3a2.11+dfsg-1ubuntu7.21_amd64.deb ...\r\nUnpacking qemu-utils (1:2.11+dfsg-1ubuntu7.21) ...\r\nSetting up librados2 (12.2.12-0ubuntu0.18.04.3) ...\r\nSetting up librbd1 (12.2.12-0ubuntu0.18.04.3) ...\r\nSetting up qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.21) ...\r\nSetting up qemu-utils (1:2.11+dfsg-1ubuntu7.21) ...\r\nSetting up qemu-system-common (1:2.11+dfsg-1ubuntu7.21) ...\r\nSetting up qemu-system-x86 (1:2.11+dfsg-1ubuntu7.21) ...\r\nProcessing trigger 2019-12-05T04:59:14.215 INFO:teuthology.task.ansible.out:s for man-db (2.8.3-2ubuntu0.1) ...\r\nProcessing triggers for libc-bin (2.27-3ubuntu1) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "git is already the newest version (1:2.17.1-1ubuntu0.4).", "The following additional packages will be installed:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils", "Suggested packages:", " samba vde2 sgabios ovmf debootstrap", "The following NEW packages will be installed:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "0 upgraded, 6 newly installed, 0 to remove and 52 not upgraded.", "Need to get 10.4 MB of archives.", "After this operation, 47.5 MB of additional disk space will be used.", "Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librados2 amd64 12.2.12-0ubuntu0.18.04.3 [2702 kB]", "Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librbd1 amd64 12.2.12-0ubuntu0.18.04.3 [920 kB]", "Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-block-extra amd64 1:2.11+dfsg-1ubuntu7.21 [39.8 kB]", "Get:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-common amd64 1:2.11+dfsg-1ubuntu7.21 [672 kB]", "Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-x86 amd64 1:2.11+dfsg-1ubuntu7.21 [5197 kB]", "Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-utils amd64 1:2.11+dfsg-1ubuntu7.21 [868 kB]", "Fetched 10.4 MB in 1s (9525 kB/s)", "Selecting previously unselected package librados2.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Reading database ... 60%", "(Reading database ... 65%", "(Readin 2019-12-05T04:59:14.216 INFO:teuthology.task.ansible.out:g database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 137032 files and directories currently installed.)", "Preparing to unpack .../0-librados2_12.2.12-0ubuntu0.18.04.3_amd64.deb ...", "Unpacking librados2 (12.2.12-0ubuntu0.18.04.3) ...", "Selecting previously unselected package librbd1.", "Preparing to unpack .../1-librbd1_12.2.12-0ubuntu0.18.04.3_amd64.deb ...", "Unpacking librbd1 (12.2.12-0ubuntu0.18.04.3) ...", "Selecting previously unselected package qemu-block-extra:amd64.", "Preparing to unpack .../2-qemu-block-extra_1%3a2.11+dfsg-1ubuntu7.21_amd64.deb ...", "Unpacking qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.21) ...", "Selecting previously unselected package qemu-system-common.", "Preparing to unpack .../3-qemu-system-common_1%3a2.11+dfsg-1ubuntu7.21_amd64.deb ...", "Unpacking qemu-system-common (1:2.11+dfsg-1ubuntu7.21) ...", "Selecting previously unselected package qemu-system-x86.", "Preparing to unpack .../4-qemu-system-x86_1%3a2.11+dfsg-1ubuntu7.21_amd64.deb ...", "Unpacking qemu-system-x86 (1:2.11+dfsg-1ubuntu7.21) ...", "Selecting previously unselected package qemu-utils.", "Preparing to unpack .../5-qemu-utils_1%3a2.11+dfsg-1ubuntu7.21_amd64.deb ...", "Unpacking qemu-utils (1:2.11+dfsg-1ubuntu7.21) ...", "Setting up librados2 (12.2.12-0ubuntu0.18.04.3) ...", "Setting up librbd1 (12.2.12-0ubuntu0.18.04.3) ...", "Setting up qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.21) ...", "Setting up qemu-utils (1:2.11+dfsg-1ubuntu7.21) ...", "Setting up qemu-system-common (1:2.11+dfsg-1ubuntu7.21) ...", "Setting up qemu-system-x86 (1:2.11+dfsg-1ubuntu7.21) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...", "Processing triggers for libc-bin (2.27-3ubuntu1) ..."]} 2019-12-05T04:59:14.254 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:14 +0000 (0:00:08.059) 0:01:49.753 ***** 2019-12-05T04:59:15.189 INFO:teuthology.task.ansible.out: TASK [testnode : Install non aarch64 packages] ********************************* ok: [smithi042.front.sepia.ceph.com] => {"cache_update_time": 1575521611, "cache_updated": false, "changed": false} 2019-12-05T04:59:15.228 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:15 +0000 (0:00:00.977) 0:01:50.731 ***** 2019-12-05T04:59:15.370 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** 2019-12-05T04:59:15.372 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => {"changed": false, "cmd": ["perl", "-pe", "if (m{^([^#]\\S*\\s+/\\s+\\S+\\s+)(\\S+)(\\s+.*)$}) { $_=\"$1$2,user_xattr$3\\n\" unless $2=~m{(^|,)user_xattr(,|$)}; }", "-i.bak", "/etc/fstab"], "rc": 0, "stdout": "skipped, since /etc/fstab.bak exists", "stdout_lines": ["skipped, since /etc/fstab.bak exists"]} 2019-12-05T04:59:15.415 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:15 +0000 (0:00:00.187) 0:01:50.918 ***** 2019-12-05T04:59:15.442 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** 2019-12-05T04:59:15.443 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:15.482 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:15 +0000 (0:00:00.066) 0:01:50.984 ***** 2019-12-05T04:59:15.554 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2019-12-05T04:59:15.555 INFO:teuthology.task.ansible.out:changed: [smithi040.front.sepia.ceph.com] => {"cache_update_time": 1575521656, "cache_updated": false, "changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\ngit is already the newest version (1:2.17.1-1ubuntu0.4).\nThe following additional packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils\nSuggested packages:\n samba vde2 sgabios ovmf debootstrap\nThe following NEW packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\n0 upgraded, 6 newly installed, 0 to remove and 52 not upgraded.\nNeed to get 10.4 MB of archives.\nAfter this operation, 47.5 MB of additional disk space will be used.\nGet:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librados2 amd64 12.2.12-0ubuntu0.18.04.3 [2702 kB]\nGet:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librbd1 amd64 12.2.12-0ubuntu0.18.04.3 [920 kB]\nGet:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-block-extra amd64 1:2.11+dfsg-1ubuntu7.21 [39.8 kB]\nGet:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-common amd64 1:2.11+dfsg-1ubuntu7.21 [672 kB]\nGet:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-x86 amd64 1:2.11+dfsg-1ubuntu7.21 [5197 kB]\nGet:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-utils amd64 1:2.11+dfsg-1ubuntu7.21 [868 kB]\nFetched 10.4 MB in 1s (9847 kB/s)\nSelecting previously unselected package librados2.\r\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database .. 2019-12-05T04:59:15.557 INFO:teuthology.task.ansible.out:. 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 137032 files and directories currently installed.)\r\nPreparing to unpack .../0-librados2_12.2.12-0ubuntu0.18.04.3_amd64.deb ...\r\nUnpacking librados2 (12.2.12-0ubuntu0.18.04.3) ...\r\nSelecting previously unselected package librbd1.\r\nPreparing to unpack .../1-librbd1_12.2.12-0ubuntu0.18.04.3_amd64.deb ...\r\nUnpacking librbd1 (12.2.12-0ubuntu0.18.04.3) ...\r\nSelecting previously unselected package qemu-block-extra:amd64.\r\nPreparing to unpack .../2-qemu-block-extra_1%3a2.11+dfsg-1ubuntu7.21_amd64.deb ...\r\nUnpacking qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.21) ...\r\nSelecting previously unselected package qemu-system-common.\r\nPreparing to unpack .../3-qemu-system-common_1%3a2.11+dfsg-1ubuntu7.21_amd64.deb ...\r\nUnpacking qemu-system-common (1:2.11+dfsg-1ubuntu7.21) ...\r\nSelecting previously unselected package qemu-system-x86.\r\nPreparing to unpack .../4-qemu-system-x86_1%3a2.11+dfsg-1ubuntu7.21_amd64.deb ...\r\nUnpacking qemu-system-x86 (1:2.11+dfsg-1ubuntu7.21) ...\r\nSelecting previously unselected package qemu-utils.\r\nPreparing to unpack .../5-qemu-utils_1%3a2.11+dfsg-1ubuntu7.21_amd64.deb ...\r\nUnpacking qemu-utils (1:2.11+dfsg-1ubuntu7.21) ...\r\nSetting up librados2 (12.2.12-0ubuntu0.18.04.3) ...\r\nSetting up librbd1 (12.2.12-0ubuntu0.18.04.3) ...\r\nSetting up qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.21) ...\r\nSetting up qemu-utils (1:2.11+dfsg-1ubuntu7.21) ...\r\nSetting up qemu-system-common (1:2.11+dfsg-1ubuntu7.21) ...\r\nSetting up qemu-system-x86 (1:2.11+dfsg-1ubuntu7.21) ...\r\nProcessing triggers for man-db (2.8.3-2ubuntu0.1) ...\r\nProcessing triggers for libc-bin (2.27-3ubunt 2019-12-05T04:59:15.558 INFO:teuthology.task.ansible.out:u1) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "git is already the newest version (1:2.17.1-1ubuntu0.4).", "The following additional packages will be installed:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils", "Suggested packages:", " samba vde2 sgabios ovmf debootstrap", "The following NEW packages will be installed:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "0 upgraded, 6 newly installed, 0 to remove and 52 not upgraded.", "Need to get 10.4 MB of archives.", "After this operation, 47.5 MB of additional disk space will be used.", "Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librados2 amd64 12.2.12-0ubuntu0.18.04.3 [2702 kB]", "Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librbd1 amd64 12.2.12-0ubuntu0.18.04.3 [920 kB]", "Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-block-extra amd64 1:2.11+dfsg-1ubuntu7.21 [39.8 kB]", "Get:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-common amd64 1:2.11+dfsg-1ubuntu7.21 [672 kB]", "Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-x86 amd64 1:2.11+dfsg-1ubuntu7.21 [5197 kB]", "Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-utils amd64 1:2.11+dfsg-1ubuntu7.21 [868 kB]", "Fetched 10.4 MB in 1s (9847 kB/s)", "Selecting previously unselected package librados2.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Reading database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Rea 2019-12-05T04:59:15.565 INFO:teuthology.task.ansible.out:ding database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 137032 files and directories currently installed.)", "Preparing to unpack .../0-librados2_12.2.12-0ubuntu0.18.04.3_amd64.deb ...", "Unpacking librados2 (12.2.12-0ubuntu0.18.04.3) ...", "Selecting previously unselected package librbd1.", "Preparing to unpack .../1-librbd1_12.2.12-0ubuntu0.18.04.3_amd64.deb ...", "Unpacking librbd1 (12.2.12-0ubuntu0.18.04.3) ...", "Selecting previously unselected package qemu-block-extra:amd64.", "Preparing to unpack .../2-qemu-block-extra_1%3a2.11+dfsg-1ubuntu7.21_amd64.deb ...", "Unpacking qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.21) ...", "Selecting previously unselected package qemu-system-common.", "Preparing to unpack .../3-qemu-system-common_1%3a2.11+dfsg-1ubuntu7.21_amd64.deb ...", "Unpacking qemu-system-common (1:2.11+dfsg-1ubuntu7.21) ...", "Selecting previously unselected package qemu-system-x86.", "Preparing to unpack .../4-qemu-system-x86_1%3a2.11+dfsg-1ubuntu7.21_amd64.deb ...", "Unpacking qemu-system-x86 (1:2.11+dfsg-1ubuntu7.21) ...", "Selecting previously unselected package qemu-utils.", "Preparing to unpack .../5-qemu-utils_1%3a2.11+dfsg-1ubuntu7.21_amd64.deb ...", "Unpacking qemu-utils (1:2.11+dfsg-1ubuntu7.21) ...", "Setting up librados2 (12.2.12-0ubuntu0.18.04.3) ...", "Setting up librbd1 (12.2.12-0ubuntu0.18.04.3) ...", "Setting up qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.21) ...", "Setting up qemu-utils (1:2.11+dfsg-1ubuntu7.21) ...", "Setting up qemu-system-common (1:2.11+dfsg-1ubuntu7.21) ...", "Setting up qemu-system-x86 (1:2.11+dfsg-1ubuntu7.21) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...", "Processing triggers for libc-bin (2.27-3ubuntu1) ..."]} 2019-12-05T04:59:15.599 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:15 +0000 (0:00:00.115) 0:01:51.100 ***** 2019-12-05T04:59:15.624 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ 2019-12-05T04:59:15.626 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => {"changed": false, "gid": 10102, "name": "fuse", "state": "present", "system": false} 2019-12-05T04:59:15.666 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:15 +0000 (0:00:00.069) 0:01:51.169 ***** 2019-12-05T04:59:16.014 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/fuse.conf.] *************************************** 2019-12-05T04:59:16.016 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => {"changed": false, "checksum": "9edff47031d0f14a9b8c813b44eb4a9296d0439f", "dest": "/etc/fuse.conf", "gid": 10102, "group": "fuse", "mode": "0644", "owner": "root", "path": "/etc/fuse.conf", "size": 366, "state": "file", "uid": 0} 2019-12-05T04:59:16.064 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:16 +0000 (0:00:00.391) 0:01:51.560 ***** 2019-12-05T04:59:16.232 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ ok: [smithi042.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1216, "groups": "fuse,kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2019-12-05T04:59:16.278 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:16 +0000 (0:00:00.216) 0:01:51.776 ***** 2019-12-05T04:59:16.418 INFO:teuthology.task.ansible.out: TASK [testnode : Set up static IP in /etc/hosts.] ****************************** ok: [smithi042.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2019-12-05T04:59:16.458 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:16 +0000 (0:00:00.183) 0:01:51.960 ***** 2019-12-05T04:59:16.490 INFO:teuthology.task.ansible.out: TASK [testnode : Install non aarch64 packages] ********************************* 2019-12-05T04:59:16.491 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => {"cache_update_time": 1575521656, "cache_updated": false, "changed": false} 2019-12-05T04:59:16.547 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:16 +0000 (0:00:00.076) 0:01:52.037 ***** 2019-12-05T04:59:16.698 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** 2019-12-05T04:59:16.699 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => {"changed": false, "cmd": ["perl", "-pe", "if (m{^([^#]\\S*\\s+/\\s+\\S+\\s+)(\\S+)(\\s+.*)$}) { $_=\"$1$2,user_xattr$3\\n\" unless $2=~m{(^|,)user_xattr(,|$)}; }", "-i.bak", "/etc/fstab"], "rc": 0, "stdout": "skipped, since /etc/fstab.bak exists", "stdout_lines": ["skipped, since /etc/fstab.bak exists"]} 2019-12-05T04:59:16.741 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:16 +0000 (0:00:00.206) 0:01:52.244 ***** 2019-12-05T04:59:16.768 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* 2019-12-05T04:59:16.771 INFO:teuthology.task.ansible.out:changed: [smithi042.front.sepia.ceph.com] => {"changed": true, "name": "apache2", "state": "stopped", "status": {"ActiveEnterTimestamp": "Thu 2019-12-05 04:56:44 UTC", "ActiveEnterTimestampMonotonic": "19487649", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "remote-fs.target systemd-journald.socket sysinit.target -.mount systemd-tmpfiles-setup.service basic.target nss-lookup.target network.target system.slice", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Thu 2019-12-05 04:56:43 UTC", "AssertTimestampMonotonic": "18637136", "Before": "multi-user.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Thu 2019-12-05 04:56:43 UTC", "ConditionTimestampMonotonic": "18637135", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target", "ControlGroup": "/system.slice/apache2.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "The Apache HTTP Server", "DevicePolicy": "auto", " 2019-12-05T04:59:16.772 INFO:teuthology.task.ansible.out:DropInPaths": "/lib/systemd/system/apache2.service.d/apache2-systemd.conf", "DynamicUser": "no", "Environment": "APACHE_STARTED_BY_SYSTEMD=true", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1191", "ExecMainStartTimestamp": "Thu 2019-12-05 04:56:44 UTC", "ExecMainStartTimestampMonotonic": "19487628", "ExecMainStatus": "0", "ExecReload": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl graceful ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl start ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStop": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl stop ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/apache2.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "apache2.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Thu 2019-12-05 04:56:43 UTC", "InactiveExitTimestampMonotonic": "18638048", "InvocationID": "33e59b0ad02d4d4cb5508cdc063cac9f", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESof 2019-12-05T04:59:16.774 INFO:teuthology.task.ansible.out:t": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128059", "LimitNPROCSoft": "128059", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128059", "LimitSIGPENDINGSoft": "128059", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1191", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "apache2.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice -.mount sysinit.target", "RequiresMountsFor": "/tmp /var/tmp", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": 2019-12-05T04:59:16.776 INFO:teuthology.task.ansible.out: "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Thu 2019-12-05 04:56:44 UTC", "StateChangeTimestampMonotonic": "19487649", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "55", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Thu 2019-12-05 04:56:44 UTC", "WatchdogTimestampMonotonic": "19487648", "WatchdogUSec": "0"}} TASK [testnode : Enable xattr for this boot.] ********************************** skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:16.815 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:16 +0000 (0:00:00.074) 0:01:52.318 ***** 2019-12-05T04:59:16.859 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:16 +0000 (0:00:00.039) 0:01:52.358 ***** 2019-12-05T04:59:16.884 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:16.926 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:16 +0000 (0:00:00.071) 0:01:52.429 ***** 2019-12-05T04:59:16.953 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ 2019-12-05T04:59:16.954 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => {"changed": false, "gid": 10102, "name": "fuse", "state": "present", "system": false} 2019-12-05T04:59:16.955 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:16.994 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:16 +0000 (0:00:00.068) 0:01:52.497 ***** 2019-12-05T04:59:17.030 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:17 +0000 (0:00:00.035) 0:01:52.533 ***** 2019-12-05T04:59:17.077 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2019-12-05T04:59:17.079 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:17.117 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:17 +0000 (0:00:00.087) 0:01:52.620 ***** 2019-12-05T04:59:17.159 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:17.197 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:17 +0000 (0:00:00.080) 0:01:52.700 ***** 2019-12-05T04:59:17.235 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:17.287 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:17 +0000 (0:00:00.086) 0:01:52.786 ***** 2019-12-05T04:59:17.313 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2019-12-05T04:59:17.315 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:17.358 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:17 +0000 (0:00:00.074) 0:01:52.861 ***** 2019-12-05T04:59:17.376 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/fuse.conf.] *************************************** ok: [smithi040.front.sepia.ceph.com] => {"changed": false, "checksum": "9edff47031d0f14a9b8c813b44eb4a9296d0439f", "dest": "/etc/fuse.conf", "gid": 10102, "group": "fuse", "mode": "0644", "owner": "root", "path": "/etc/fuse.conf", "size": 366, "state": "file", "uid": 0} 2019-12-05T04:59:17.416 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:17 +0000 (0:00:00.057) 0:01:52.918 ***** 2019-12-05T04:59:17.422 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2019-12-05T04:59:17.424 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:17.474 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:17 +0000 (0:00:00.058) 0:01:52.977 ***** 2019-12-05T04:59:17.518 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:17.558 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:17 +0000 (0:00:00.084) 0:01:53.061 ***** 2019-12-05T04:59:17.597 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ 2019-12-05T04:59:17.599 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1216, "groups": "fuse,kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2019-12-05T04:59:17.600 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:17.648 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:17 +0000 (0:00:00.089) 0:01:53.151 ***** 2019-12-05T04:59:17.692 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:17 +0000 (0:00:00.043) 0:01:53.194 ***** 2019-12-05T04:59:17.718 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2019-12-05T04:59:17.720 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:17.764 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:17 +0000 (0:00:00.068) 0:01:53.263 ***** 2019-12-05T04:59:17.788 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:17.858 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:17 +0000 (0:00:00.093) 0:01:53.357 ***** 2019-12-05T04:59:17.864 INFO:teuthology.task.ansible.out: TASK [testnode : Set up static IP in /etc/hosts.] ****************************** ok: [smithi040.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2019-12-05T04:59:17.912 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:17 +0000 (0:00:00.051) 0:01:53.408 ***** TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:17.955 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:17 +0000 (0:00:00.049) 0:01:53.457 ***** 2019-12-05T04:59:18.014 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:18.033 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:18 +0000 (0:00:00.078) 0:01:53.536 ***** 2019-12-05T04:59:18.071 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:18.123 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:18 +0000 (0:00:00.088) 0:01:53.624 ***** 2019-12-05T04:59:18.186 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2019-12-05T04:59:18.193 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:18.234 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:18 +0000 (0:00:00.112) 0:01:53.737 ***** 2019-12-05T04:59:18.244 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* 2019-12-05T04:59:18.247 INFO:teuthology.task.ansible.out:changed: [smithi040.front.sepia.ceph.com] => {"changed": true, "name": "apache2", "state": "stopped", "status": {"ActiveEnterTimestamp": "Thu 2019-12-05 04:56:53 UTC", "ActiveEnterTimestampMonotonic": "20269939", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "systemd-tmpfiles-setup.service systemd-journald.socket sysinit.target -.mount nss-lookup.target remote-fs.target basic.target system.slice network.target", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Thu 2019-12-05 04:56:53 UTC", "AssertTimestampMonotonic": "19441407", "Before": "multi-user.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Thu 2019-12-05 04:56:53 UTC", "ConditionTimestampMonotonic": "19441407", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target", "ControlGroup": "/system.slice/apache2.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "The Apache HTTP Server", "DevicePolicy": "auto", " 2019-12-05T04:59:18.249 INFO:teuthology.task.ansible.out:DropInPaths": "/lib/systemd/system/apache2.service.d/apache2-systemd.conf", "DynamicUser": "no", "Environment": "APACHE_STARTED_BY_SYSTEMD=true", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1366", "ExecMainStartTimestamp": "Thu 2019-12-05 04:56:53 UTC", "ExecMainStartTimestampMonotonic": "20269892", "ExecMainStatus": "0", "ExecReload": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl graceful ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl start ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStop": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl stop ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/apache2.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "apache2.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Thu 2019-12-05 04:56:53 UTC", "InactiveExitTimestampMonotonic": "19442933", "InvocationID": "e1e8e46bd2c248058623053c680fcd04", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESof 2019-12-05T04:59:18.250 INFO:teuthology.task.ansible.out:t": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128059", "LimitNPROCSoft": "128059", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128059", "LimitSIGPENDINGSoft": "128059", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1366", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "apache2.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "sysinit.target system.slice -.mount", "RequiresMountsFor": "/var/tmp /tmp", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": 2019-12-05T04:59:18.252 INFO:teuthology.task.ansible.out: "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Thu 2019-12-05 04:56:53 UTC", "StateChangeTimestampMonotonic": "20269939", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "55", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Thu 2019-12-05 04:56:53 UTC", "WatchdogTimestampMonotonic": "20269937", "WatchdogUSec": "0"}} 2019-12-05T04:59:18.291 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:18 +0000 (0:00:00.056) 0:01:53.793 ***** 2019-12-05T04:59:18.297 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2019-12-05T04:59:18.298 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:18.340 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:18 +0000 (0:00:00.049) 0:01:53.843 ***** 2019-12-05T04:59:18.346 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2019-12-05T04:59:18.348 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:18.387 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:18 +0000 (0:00:00.047) 0:01:53.890 ***** 2019-12-05T04:59:18.394 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:18.436 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:18 +0000 (0:00:00.048) 0:01:53.939 ***** 2019-12-05T04:59:18.441 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2019-12-05T04:59:18.441 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:18.484 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:18 +0000 (0:00:00.048) 0:01:53.987 ***** 2019-12-05T04:59:18.500 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:18.537 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:18 +0000 (0:00:00.051) 0:01:54.039 ***** 2019-12-05T04:59:18.544 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2019-12-05T04:59:18.546 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:18.605 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:18 +0000 (0:00:00.067) 0:01:54.107 ***** 2019-12-05T04:59:18.610 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2019-12-05T04:59:18.612 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:18.654 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:18 +0000 (0:00:00.046) 0:01:54.154 ***** 2019-12-05T04:59:18.658 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:18.706 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:18 +0000 (0:00:00.055) 0:01:54.209 ***** 2019-12-05T04:59:18.715 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:18.760 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:18 +0000 (0:00:00.048) 0:01:54.257 ***** 2019-12-05T04:59:18.762 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:18.811 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:18 +0000 (0:00:00.056) 0:01:54.314 ***** 2019-12-05T04:59:18.819 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:18.865 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:18 +0000 (0:00:00.052) 0:01:54.367 ***** 2019-12-05T04:59:18.871 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:18.916 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:18 +0000 (0:00:00.051) 0:01:54.419 ***** 2019-12-05T04:59:18.923 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2019-12-05T04:59:18.924 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:18.966 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:18 +0000 (0:00:00.050) 0:01:54.469 ***** 2019-12-05T04:59:18.972 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2019-12-05T04:59:18.974 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:19.015 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:19 +0000 (0:00:00.048) 0:01:54.517 ***** 2019-12-05T04:59:19.021 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2019-12-05T04:59:19.022 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:19.069 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:19 +0000 (0:00:00.049) 0:01:54.567 ***** 2019-12-05T04:59:19.073 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:19.113 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:19 +0000 (0:00:00.048) 0:01:54.616 ***** 2019-12-05T04:59:19.127 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:19.162 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:19 +0000 (0:00:00.048) 0:01:54.664 ***** 2019-12-05T04:59:19.167 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2019-12-05T04:59:19.169 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:19.209 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:19 +0000 (0:00:00.047) 0:01:54.712 ***** 2019-12-05T04:59:19.216 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* 2019-12-05T04:59:19.216 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:19.264 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:19 +0000 (0:00:00.054) 0:01:54.766 ***** 2019-12-05T04:59:19.271 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:19.316 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:19 +0000 (0:00:00.052) 0:01:54.819 ***** 2019-12-05T04:59:19.355 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2019-12-05T04:59:19.694 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 05 December 2019 04:59:19 +0000 (0:00:00.081) 0:01:54.900 ***** TASK [testnode : Remove /etc/ceph] ********************************************* ok: [smithi042.front.sepia.ceph.com] => {"changed": false, "path": "/etc/ceph", "state": "absent"} Thursday 05 December 2019 04:59:19 +0000 (0:00:00.059) 0:01:54.960 ***** TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 05 December 2019 04:59:19 +0000 (0:00:00.048) 0:01:55.008 ***** TASK [testnode : Make raid/smart scripts work.] ******************************** skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 05 December 2019 04:59:19 +0000 (0:00:00.071) 0:01:55.080 ***** TASK [testnode : Create FStest ubuntu directory.] ****************************** skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 05 December 2019 04:59:19 +0000 (0:00:00.071) 0:01:55.151 ***** TASK [testnode : Make fsstress same path as ubuntu.] *************************** skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:19.746 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:19 +0000 (0:00:00.097) 0:01:55.249 ***** 2019-12-05T04:59:19.774 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2019-12-05T04:59:19.775 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:19.831 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:19 +0000 (0:00:00.085) 0:01:55.334 ***** 2019-12-05T04:59:19.836 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2019-12-05T04:59:19.839 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => {"changed": false, "checksum": "0b949a4622a315acfc90571fddbd013b4dc0f5f9", "dest": "/etc/exports", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/exports", "size": 578, "state": "file", "uid": 0} 2019-12-05T04:59:19.888 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:19 +0000 (0:00:00.056) 0:01:55.390 ***** 2019-12-05T04:59:19.894 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2019-12-05T04:59:19.896 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:19.940 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:19 +0000 (0:00:00.050) 0:01:55.440 ***** 2019-12-05T04:59:19.943 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2019-12-05T04:59:19.944 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:19.993 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:19 +0000 (0:00:00.055) 0:01:55.495 ***** 2019-12-05T04:59:19.998 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2019-12-05T04:59:20.000 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:20.045 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:20 +0000 (0:00:00.052) 0:01:55.547 ***** 2019-12-05T04:59:20.082 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:20.126 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:20 +0000 (0:00:00.081) 0:01:55.629 ***** 2019-12-05T04:59:20.154 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2019-12-05T04:59:20.156 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:20.196 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:20 +0000 (0:00:00.070) 0:01:55.699 ***** 2019-12-05T04:59:20.235 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2019-12-05T04:59:20.237 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:20.300 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:20 +0000 (0:00:00.103) 0:01:55.803 ***** 2019-12-05T04:59:20.368 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:20.445 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:20 +0000 (0:00:00.144) 0:01:55.947 ***** 2019-12-05T04:59:20.451 INFO:teuthology.task.ansible.out: TASK [testnode : Upload weekly kernel-clean crontab.] ************************** 2019-12-05T04:59:20.452 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => {"changed": false, "checksum": "d5677bbd13857966eb0fc86f66195b18a2b98e20", "dest": "/etc/cron.weekly/kernel-clean", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/cron.weekly/kernel-clean", "size": 752, "state": "file", "uid": 0} 2019-12-05T04:59:20.504 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:20 +0000 (0:00:00.056) 0:01:56.004 ***** 2019-12-05T04:59:20.513 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:20.551 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:20 +0000 (0:00:00.050) 0:01:56.054 ***** 2019-12-05T04:59:20.579 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2019-12-05T04:59:20.581 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:20.639 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:20 +0000 (0:00:00.087) 0:01:56.141 ***** 2019-12-05T04:59:20.680 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:20.742 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:20 +0000 (0:00:00.102) 0:01:56.244 ***** 2019-12-05T04:59:20.875 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* 2019-12-05T04:59:20.876 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => {"changed": false, "path": "/etc/ceph", "state": "absent"} 2019-12-05T04:59:20.947 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:20 +0000 (0:00:00.205) 0:01:56.450 ***** 2019-12-05T04:59:20.960 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* 2019-12-05T04:59:20.962 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => {"changed": false, "checksum": "22e9a1446f43d0440219b9af14a66dd77fb0c2eb", "dest": "/etc/grub.d/02_force_timeout", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/grub.d/02_force_timeout", "size": 115, "state": "file", "uid": 0} 2019-12-05T04:59:21.011 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:21 +0000 (0:00:00.063) 0:01:56.513 ***** 2019-12-05T04:59:21.300 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** ok: [smithi040.front.sepia.ceph.com] => {"changed": false, "checksum": "0b949a4622a315acfc90571fddbd013b4dc0f5f9", "dest": "/etc/exports", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/exports", "size": 578, "state": "file", "uid": 0} 2019-12-05T04:59:21.554 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:21 +0000 (0:00:00.341) 0:01:56.855 ***** TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 05 December 2019 04:59:21 +0000 (0:00:00.092) 0:01:56.947 ***** TASK [testnode : Enable kernel modules to load at boot time.] ****************** ok: [smithi042.front.sepia.ceph.com] => {"changed": false, "checksum": "6449466d762bced58744df80432d189e35329803", "dest": "/etc/modules", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/modules", "size": 315, "state": "file", "uid": 0} Thursday 05 December 2019 04:59:21 +0000 (0:00:00.061) 0:01:57.009 ***** 2019-12-05T04:59:21.689 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** 2019-12-05T04:59:21.697 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2019-12-05T04:59:21.770 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:21 +0000 (0:00:00.263) 0:01:57.273 ***** 2019-12-05T04:59:21.807 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** 2019-12-05T04:59:21.808 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:21.851 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:21 +0000 (0:00:00.080) 0:01:57.353 ***** 2019-12-05T04:59:21.856 INFO:teuthology.task.ansible.out: TASK [testnode : Upload weekly kernel-clean crontab.] ************************** 2019-12-05T04:59:21.857 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => {"changed": false, "checksum": "d5677bbd13857966eb0fc86f66195b18a2b98e20", "dest": "/etc/cron.weekly/kernel-clean", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/cron.weekly/kernel-clean", "size": 752, "state": "file", "uid": 0} 2019-12-05T04:59:21.908 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:21 +0000 (0:00:00.051) 0:01:57.405 ***** 2019-12-05T04:59:21.910 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:21.956 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:21 +0000 (0:00:00.053) 0:01:57.458 ***** 2019-12-05T04:59:21.995 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* 2019-12-05T04:59:21.997 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:22.047 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:22 +0000 (0:00:00.091) 0:01:57.549 ***** 2019-12-05T04:59:22.083 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** 2019-12-05T04:59:22.085 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:22.129 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:22 +0000 (0:00:00.082) 0:01:57.631 ***** 2019-12-05T04:59:22.161 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:22.220 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:22 +0000 (0:00:00.091) 0:01:57.723 ***** 2019-12-05T04:59:22.271 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* 2019-12-05T04:59:22.273 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => {"changed": false, "checksum": "22e9a1446f43d0440219b9af14a66dd77fb0c2eb", "dest": "/etc/grub.d/02_force_timeout", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/grub.d/02_force_timeout", "size": 115, "state": "file", "uid": 0} 2019-12-05T04:59:22.274 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** ok: [smithi042.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2019-12-05T04:59:22.334 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:22 +0000 (0:00:00.113) 0:01:57.836 ***** 2019-12-05T04:59:22.398 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:22 +0000 (0:00:00.063) 0:01:57.900 ***** 2019-12-05T04:59:22.466 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** 2019-12-05T04:59:22.474 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} skipping: [smithi042.front.sepia.ceph.com] => (item=loop3) => {"ansible_loop_var": "item", "changed": false, "item": "loop3", "skip_reason": "Conditional result was False"} 2019-12-05T04:59:22.484 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => (item=loop2) => {"ansible_loop_var": "item", "changed": false, "item": "loop2", "skip_reason": "Conditional result was False"} 2019-12-05T04:59:22.504 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => (item=loop1) => {"ansible_loop_var": "item", "changed": false, "item": "loop1", "skip_reason": "Conditional result was False"} skipping: [smithi042.front.sepia.ceph.com] => (item=loop0) => {"ansible_loop_var": "item", "changed": false, "item": "loop0", "skip_reason": "Conditional result was False"} 2019-12-05T04:59:22.519 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => (item=loop7) => {"ansible_loop_var": "item", "changed": false, "item": "loop7", "skip_reason": "Conditional result was False"} skipping: [smithi042.front.sepia.ceph.com] => (item=loop6) => {"ansible_loop_var": "item", "changed": false, "item": "loop6", "skip_reason": "Conditional result was False"} 2019-12-05T04:59:22.528 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => (item=loop5) => {"ansible_loop_var": "item", "changed": false, "item": "loop5", "skip_reason": "Conditional result was False"} 2019-12-05T04:59:22.542 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => (item=loop4) => {"ansible_loop_var": "item", "changed": false, "item": "loop4", "skip_reason": "Conditional result was False"} 2019-12-05T04:59:22.560 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} 2019-12-05T04:59:22.572 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} 2019-12-05T04:59:22.584 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} 2019-12-05T04:59:22.592 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2019-12-05T04:59:22.610 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} 2019-12-05T04:59:22.616 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2019-12-05T04:59:22.668 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:22 +0000 (0:00:00.269) 0:01:58.170 ***** 2019-12-05T04:59:22.760 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** ok: [smithi040.front.sepia.ceph.com] => {"changed": false, "checksum": "6449466d762bced58744df80432d189e35329803", "dest": "/etc/modules", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/modules", "size": 315, "state": "file", "uid": 0} 2019-12-05T04:59:22.804 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:22 +0000 (0:00:00.136) 0:01:58.307 ***** 2019-12-05T04:59:22.873 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** changed: [smithi042.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.012088", "end": "2019-12-05 04:59:22.832715", "rc": 0, "start": "2019-12-05 04:59:22.820627", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock"]} 2019-12-05T04:59:22.923 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:22 +0000 (0:00:00.117) 0:01:58.424 ***** 2019-12-05T04:59:22.961 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** 2019-12-05T04:59:22.963 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} TASK [testnode : Mark system down if NVMe card missing] ************************ 2019-12-05T04:59:22.965 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:23.008 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:23 +0000 (0:00:00.084) 0:01:58.508 ***** 2019-12-05T04:59:23.059 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:23 +0000 (0:00:00.051) 0:01:58.560 ***** 2019-12-05T04:59:23.063 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** 2019-12-05T04:59:23.066 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:23.109 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:23 +0000 (0:00:00.052) 0:01:58.612 ***** 2019-12-05T04:59:23.116 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** 2019-12-05T04:59:23.118 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:23.170 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:23 +0000 (0:00:00.052) 0:01:58.665 ***** TASK [testnode : Stop collectl] ************************************************ skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:23.212 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:23 +0000 (0:00:00.050) 0:01:58.715 ***** 2019-12-05T04:59:23.221 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:23.263 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:23 +0000 (0:00:00.050) 0:01:58.765 ***** 2019-12-05T04:59:23.273 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:23.311 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:23 +0000 (0:00:00.048) 0:01:58.814 ***** 2019-12-05T04:59:23.343 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:23.384 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:23 +0000 (0:00:00.069) 0:01:58.883 ***** 2019-12-05T04:59:23.416 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:23.460 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:23 +0000 (0:00:00.074) 0:01:58.958 ***** 2019-12-05T04:59:23.483 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** ok: [smithi040.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2019-12-05T04:59:23.523 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:23 +0000 (0:00:00.068) 0:01:59.026 ***** 2019-12-05T04:59:23.572 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** skipping: [smithi040.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} 2019-12-05T04:59:23.595 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => (item=loop3) => {"ansible_loop_var": "item", "changed": false, "item": "loop3", "skip_reason": "Conditional result was False"} 2019-12-05T04:59:23.597 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => (item=loop2) => {"ansible_loop_var": "item", "changed": false, "item": "loop2", "skip_reason": "Conditional result was False"} 2019-12-05T04:59:23.608 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => (item=loop1) => {"ansible_loop_var": "item", "changed": false, "item": "loop1", "skip_reason": "Conditional result was False"} 2019-12-05T04:59:23.618 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => (item=loop0) => {"ansible_loop_var": "item", "changed": false, "item": "loop0", "skip_reason": "Conditional result was False"} 2019-12-05T04:59:23.620 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => (item=loop7) => {"ansible_loop_var": "item", "changed": false, "item": "loop7", "skip_reason": "Conditional result was False"} 2019-12-05T04:59:23.632 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => (item=loop6) => {"ansible_loop_var": "item", "changed": false, "item": "loop6", "skip_reason": "Conditional result was False"} 2019-12-05T04:59:23.653 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => (item=loop5) => {"ansible_loop_var": "item", "changed": false, "item": "loop5", "skip_reason": "Conditional result was False"} 2019-12-05T04:59:23.668 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => (item=loop4) => {"ansible_loop_var": "item", "changed": false, "item": "loop4", "skip_reason": "Conditional result was False"} 2019-12-05T04:59:23.676 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} 2019-12-05T04:59:23.687 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} 2019-12-05T04:59:23.705 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} 2019-12-05T04:59:23.709 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2019-12-05T04:59:23.711 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} 2019-12-05T04:59:23.721 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2019-12-05T04:59:23.779 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:23 +0000 (0:00:00.255) 0:01:59.281 ***** 2019-12-05T04:59:23.965 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** 2019-12-05T04:59:23.967 INFO:teuthology.task.ansible.out:changed: [smithi040.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.012532", "end": "2019-12-05 04:59:23.932856", "rc": 0, "start": "2019-12-05 04:59:23.920324", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock"]} 2019-12-05T04:59:24.013 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:24 +0000 (0:00:00.234) 0:01:59.516 ***** 2019-12-05T04:59:24.062 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ 2019-12-05T04:59:24.063 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:24.128 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:24 +0000 (0:00:00.100) 0:01:59.616 ***** 2019-12-05T04:59:24.161 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** 2019-12-05T04:59:24.163 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Make sure apt dependencies are installed] ********************* 2019-12-05T04:59:24.164 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => {"cache_update_time": 1575521611, "cache_updated": false, "changed": false} 2019-12-05T04:59:24.205 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:24 +0000 (0:00:00.092) 0:01:59.708 ***** 2019-12-05T04:59:24.251 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:24 +0000 (0:00:00.045) 0:01:59.754 ***** 2019-12-05T04:59:24.257 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** 2019-12-05T04:59:24.258 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:24.303 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:24 +0000 (0:00:00.052) 0:01:59.806 ***** 2019-12-05T04:59:24.312 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:24.356 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:24 +0000 (0:00:00.052) 0:01:59.858 ***** 2019-12-05T04:59:24.416 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ ok: [smithi042.front.sepia.ceph.com] => (item={u'block_used': 13715725, u'uuid': u'e17a1a1d-4a16-4f42-a170-a33513a36c81', u'size_total': 983429562368, u'block_total': 240095108, u'mount': u'/', u'block_available': 226379383, u'size_available': 927249952768, u'fstype': u'ext4', u'inode_total': 61054976, u'inode_available': 60905935, u'device': u'/dev/sda1', u'inode_used': 149041, u'block_size': 4096, u'options': u'rw,relatime,errors=remount-ro,data=ordered'}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 226379383, "block_size": 4096, "block_total": 240095108, "block_used": 13715725, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60905935, "inode_total": 61054976, "inode_used": 149041, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927249952768, "size_total": 983429562368, "uuid": "e17a1a1d-4a16-4f42-a170-a33513a36c81"}} 2019-12-05T04:59:24.464 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:24 +0000 (0:00:00.107) 0:01:59.966 ***** 2019-12-05T04:59:24.636 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** changed: [smithi042.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.009888", "end": "2019-12-05 04:59:24.931203", "rc": 0, "start": "2019-12-05 04:59:24.921315", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-12-05T04:59:24.669 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:24 +0000 (0:00:00.205) 0:02:00.172 ***** 2019-12-05T04:59:24.718 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** 2019-12-05T04:59:24.719 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => (item={u'block_used': 13715725, u'uuid': u'e17a1a1d-4a16-4f42-a170-a33513a36c81', u'size_total': 983429562368, u'block_total': 240095108, u'mount': u'/', u'block_available': 226379383, u'size_available': 927249952768, u'fstype': u'ext4', u'inode_total': 61054976, u'inode_available': 60905935, u'device': u'/dev/sda1', u'inode_used': 149041, u'block_size': 4096, u'options': u'rw,relatime,errors=remount-ro,data=ordered'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226379383, "block_size": 4096, "block_total": 240095108, "block_used": 13715725, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60905935, "inode_total": 61054976, "inode_used": 149041, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927249952768, "size_total": 983429562368, "uuid": "e17a1a1d-4a16-4f42-a170-a33513a36c81"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:24.759 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:24 +0000 (0:00:00.090) 0:02:00.262 ***** 2019-12-05T04:59:24.786 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2019-12-05T04:59:24.828 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:24 +0000 (0:00:00.068) 0:02:00.330 ***** 2019-12-05T04:59:24.886 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2019-12-05T04:59:24.887 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': u'ATA', u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W4J2MS', u'wwn-0x5000c500871b7906'], u'uuids': []}, u'sas_device_handle': None, u'host': u'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', u'support_discard': u'0', u'serial': u'Z1W4J2MS', u'wwn': u'0x5000c500871b7906', u'holders': [], u'size': u'931.51 GB', u'scheduler_mode': u'cfq', u'rotational': u'1', u'sectors': u'1953525168', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': u'ST1000NM0033-9ZM', u'partitions': {u'sda1': {u'start': u'2048', u'sectorsize': 512, u'uuid': u'e17a1a1d-4a16-4f42-a170-a33513a36c81', u'sectors': u'1953522688', u'holders': [], u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W4J2MS-part1', u'wwn-0x5000c500871b7906-part1'], u'uuids': [u'e17a1a1d-4a16-4f42-a170-a33513a36c81']}, u'size': u'931.51 GB'}}}, 'key': u'sda'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4J2MS", "wwn-0x5000c500871b7906"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4J2MS-part1", "wwn-0x5000c500871b7906-part1"], "labels": [], "masters": [], "uuids": ["e17a1a1d-4a16-4f42-a170-a33513a36c81"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "e17a1a1d-4a16-4f42-a170-a33513a36c81"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W4J2MS", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "vir 2019-12-05T04:59:24.888 INFO:teuthology.task.ansible.out:tual": 1, "wwn": "0x5000c500871b7906"}}, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:24.890 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:24.901 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:24.914 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:24.916 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:24.928 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:24.941 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:24.954 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:24.969 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:25.235 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* 2019-12-05T04:59:25.237 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => {"cache_update_time": 1575521656, "cache_updated": false, "changed": false} 2019-12-05T04:59:25.280 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:25 +0000 (0:00:00.452) 0:02:00.783 ***** 2019-12-05T04:59:25.318 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:25.385 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:25 +0000 (0:00:00.096) 0:02:00.880 ***** 2019-12-05T04:59:25.445 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ ok: [smithi040.front.sepia.ceph.com] => (item={u'block_used': 13713675, u'uuid': u'e17a1a1d-4a16-4f42-a170-a33513a36c81', u'size_total': 983429562368, u'block_total': 240095108, u'mount': u'/', u'block_available': 226381433, u'size_available': 927258349568, u'fstype': u'ext4', u'inode_total': 61054976, u'inode_available': 60905936, u'device': u'/dev/sda1', u'inode_used': 149040, u'block_size': 4096, u'options': u'rw,relatime,errors=remount-ro,data=ordered'}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 226381433, "block_size": 4096, "block_total": 240095108, "block_used": 13713675, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60905936, "inode_total": 61054976, "inode_used": 149040, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927258349568, "size_total": 983429562368, "uuid": "e17a1a1d-4a16-4f42-a170-a33513a36c81"}} 2019-12-05T04:59:25.509 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:25 +0000 (0:00:00.126) 0:02:01.006 ***** 2019-12-05T04:59:25.661 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** 2019-12-05T04:59:25.662 INFO:teuthology.task.ansible.out:changed: [smithi040.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.009974", "end": "2019-12-05 04:59:25.316900", "rc": 0, "start": "2019-12-05 04:59:25.306926", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-12-05T04:59:25.716 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:25 +0000 (0:00:00.211) 0:02:01.218 ***** 2019-12-05T04:59:25.754 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** 2019-12-05T04:59:25.756 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => (item={u'block_used': 13713675, u'uuid': u'e17a1a1d-4a16-4f42-a170-a33513a36c81', u'size_total': 983429562368, u'block_total': 240095108, u'mount': u'/', u'block_available': 226381433, u'size_available': 927258349568, u'fstype': u'ext4', u'inode_total': 61054976, u'inode_available': 60905936, u'device': u'/dev/sda1', u'inode_used': 149040, u'block_size': 4096, u'options': u'rw,relatime,errors=remount-ro,data=ordered'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226381433, "block_size": 4096, "block_total": 240095108, "block_used": 13713675, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60905936, "inode_total": 61054976, "inode_used": 149040, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927258349568, "size_total": 983429562368, "uuid": "e17a1a1d-4a16-4f42-a170-a33513a36c81"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:25.799 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:25 +0000 (0:00:00.083) 0:02:01.302 ***** 2019-12-05T04:59:25.826 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2019-12-05T04:59:25.891 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:25 +0000 (0:00:00.087) 0:02:01.390 ***** 2019-12-05T04:59:25.941 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** skipping: [smithi040.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': u'ATA', u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W4HQ07', u'wwn-0x5000c500871a8a38'], u'uuids': []}, u'sas_device_handle': None, u'host': u'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', u'support_discard': u'0', u'serial': u'Z1W4HQ07', u'wwn': u'0x5000c500871a8a38', u'holders': [], u'size': u'931.51 GB', u'scheduler_mode': u'cfq', u'rotational': u'1', u'sectors': u'1953525168', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': u'ST1000NM0033-9ZM', u'partitions': {u'sda1': {u'start': u'2048', u'sectorsize': 512, u'uuid': u'e17a1a1d-4a16-4f42-a170-a33513a36c81', u'sectors': u'1953522688', u'holders': [], u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W4HQ07-part1', u'wwn-0x5000c500871a8a38-part1'], u'uuids': [u'e17a1a1d-4a16-4f42-a170-a33513a36c81']}, u'size': u'931.51 GB'}}}, 'key': u'sda'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4HQ07", "wwn-0x5000c500871a8a38"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4HQ07-part1", "wwn-0x5000c500871a8a38-part1"], "labels": [], "masters": [], "uuids": ["e17a1a1d-4a16-4f42-a170-a33513a36c81"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "e17a1a1d-4a16-4f42-a170-a33513a36c81"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "ser 2019-12-05T04:59:25.943 INFO:teuthology.task.ansible.out:ial": "Z1W4HQ07", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c500871a8a38"}}, "skip_r 2019-12-05T04:59:25.947 INFO:teuthology.task.ansible.out:eason": "Conditional result was False"} 2019-12-05T04:59:25.959 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:25.974 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:25.993 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:26.009 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:26.010 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:26.023 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:26.033 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:26.045 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:27.015 INFO:teuthology.task.ansible.out:changed: [smithi042.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [u'dm-0', u'dm-1', u'dm-2', u'dm-3', u'dm-4'], u'labels': [], u'ids': [u'lvm-pv-uuid-us223U-zYGd-W0e4-tLE6-JeIM-Up89-WrMgMa', u'nvme-INTEL_SSDPED1D480GA_PHMB75130021480DGN', u'nvme-nvme.8086-50484d42373531333030323134383044474e-494e54454c2053534450454431443438304741-00000001'], u'uuids': []}, u'sas_device_handle': None, u'host': u'Non-Volatile memory controller: Intel Corporation Optane SSD 900P Series', u'support_discard': u'512', u'model': u'INTEL SSDPED1D480GA', u'size': u'447.13 GB', u'scheduler_mode': u'none', u'rotational': u'0', u'sectors': u'937703088', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [u'vg_nvme-lv_4', u'vg_nvme-lv_1', u'vg_nvme-lv_3', u'vg_nvme-lv_5', u'vg_nvme-lv_2'], u'partitions': {}}, 'key': u'nvme0n1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.907145", "end": "2019-12-05 04:59:27.313260", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_4", "vg_nvme-lv_1", "vg_nvme-lv_3", "vg_nvme-lv_5", "vg_nvme-lv_2"], "host": "Non-Volatile memory controller: Intel Corporation Optane SSD 900P Series", "links": {"ids": ["lvm-pv-uuid-us223U-zYGd-W0e4-tLE6-JeIM-Up89-WrMgMa", "nvme-INTEL_SSDPED1D480GA_PHMB75130021480DGN", "nvme-nvme.8086-50484d42373531333030323134383044474e-494e54454c2053534450454431443438304741-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPED1D480GA", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "937703088", "sectorsize": "512", "size": "447.13 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-12-05 04:59:25.406115", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nGP 2019-12-05T04:59:27.016 INFO:teuthology.task.ansible.out:T data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2019-12-05T04:59:28.004 INFO:teuthology.task.ansible.out:changed: [smithi040.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [u'dm-0', u'dm-1', u'dm-2', u'dm-3', u'dm-4'], u'labels': [], u'ids': [u'lvm-pv-uuid-p51jR8-WU6f-eKe8-qfWs-0sFT-TO0l-gkMZPH', u'nvme-INTEL_SSDPEDMD400G4_CVFT5330009J400BGN', u'nvme-nvme.8086-43564654353333303030394a34303042474e-494e54454c205353445045444d443430304734-00000001'], u'uuids': []}, u'sas_device_handle': None, u'host': u'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', u'support_discard': u'512', u'model': u'INTEL SSDPEDMD400G4', u'size': u'372.61 GB', u'scheduler_mode': u'none', u'rotational': u'0', u'sectors': u'781422768', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [u'vg_nvme-lv_4', u'vg_nvme-lv_1', u'vg_nvme-lv_3', u'vg_nvme-lv_5', u'vg_nvme-lv_2'], u'partitions': {}}, 'key': u'nvme0n1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.816920", "end": "2019-12-05 04:59:27.656697", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_4", "vg_nvme-lv_1", "vg_nvme-lv_3", "vg_nvme-lv_5", "vg_nvme-lv_2"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-p51jR8-WU6f-eKe8-qfWs-0sFT-TO0l-gkMZPH", "nvme-INTEL_SSDPEDMD400G4_CVFT5330009J400BGN", "nvme-nvme.8086-43564654353333303030394a34303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-12-05 04:59:25.839777", "stderr": "", "stderr_lines": [], "stdout": "Creating new GP 2019-12-05T04:59:28.023 INFO:teuthology.task.ansible.out:T entries.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2019-12-05T04:59:28.271 INFO:teuthology.task.ansible.out:changed: [smithi042.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_1', u'dm-uuid-LVM-kGEgBNXyNFrdJsrD7MkvFoTBTW8C1m51PnUroqiSFpgYzOAdIiPrhGUTFehYQwbd'], u'uuids': [u'e3bc4f1c-facf-43d6-b954-3ed3e9f1fc2d']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-4'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.123201", "end": "2019-12-05 04:59:28.569986", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-kGEgBNXyNFrdJsrD7MkvFoTBTW8C1m51PnUroqiSFpgYzOAdIiPrhGUTFehYQwbd"], "labels": [], "masters": [], "uuids": ["e3bc4f1c-facf-43d6-b954-3ed3e9f1fc2d"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-12-05 04:59:27.446785", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2019-12-05T04:59:29.284 INFO:teuthology.task.ansible.out:changed: [smithi040.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_1', u'dm-uuid-LVM-EVON5M0Nga3DT6Bz22oa0c4l4NKx5MI9UDekeENvlKU29ruh126smjMT7Xh6RMfb'], u'uuids': [u'e7617c40-ea15-40e8-bce1-bd8faae80f81']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-4'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.136475", "end": "2019-12-05 04:59:28.931764", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-EVON5M0Nga3DT6Bz22oa0c4l4NKx5MI9UDekeENvlKU29ruh126smjMT7Xh6RMfb"], "labels": [], "masters": [], "uuids": ["e7617c40-ea15-40e8-bce1-bd8faae80f81"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-12-05 04:59:27.795289", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2019-12-05T04:59:29.505 INFO:teuthology.task.ansible.out:changed: [smithi042.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_3', u'dm-uuid-LVM-kGEgBNXyNFrdJsrD7MkvFoTBTW8C1m512vkFgqg3JbojJb10AaGWc1orJGACksdV'], u'uuids': [u'11049691-dfea-4fde-ac44-4ca7ce918ade']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-2'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.097468", "end": "2019-12-05 04:59:29.804198", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-kGEgBNXyNFrdJsrD7MkvFoTBTW8C1m512vkFgqg3JbojJb10AaGWc1orJGACksdV"], "labels": [], "masters": [], "uuids": ["11049691-dfea-4fde-ac44-4ca7ce918ade"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-12-05 04:59:28.706730", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2019-12-05T04:59:30.555 INFO:teuthology.task.ansible.out:changed: [smithi040.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_3', u'dm-uuid-LVM-EVON5M0Nga3DT6Bz22oa0c4l4NKx5MI9ZesOXeekd8Gf9Nsrm8ukSFbKfhmc1RSV'], u'uuids': [u'b147b9bd-00f6-4e74-bde2-8245cabbcd6b']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-2'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.123678", "end": "2019-12-05 04:59:30.191249", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-EVON5M0Nga3DT6Bz22oa0c4l4NKx5MI9ZesOXeekd8Gf9Nsrm8ukSFbKfhmc1RSV"], "labels": [], "masters": [], "uuids": ["b147b9bd-00f6-4e74-bde2-8245cabbcd6b"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-12-05 04:59:29.067571", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2019-12-05T04:59:30.746 INFO:teuthology.task.ansible.out:changed: [smithi042.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_2', u'dm-uuid-LVM-kGEgBNXyNFrdJsrD7MkvFoTBTW8C1m51kSlnQNUeBugaJsg0js1tubFcNSKAA3fr'], u'uuids': [u'f2907880-623c-4750-8f97-579ad55e674c']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-3'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.113263", "end": "2019-12-05 04:59:31.044932", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-kGEgBNXyNFrdJsrD7MkvFoTBTW8C1m51kSlnQNUeBugaJsg0js1tubFcNSKAA3fr"], "labels": [], "masters": [], "uuids": ["f2907880-623c-4750-8f97-579ad55e674c"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-12-05 04:59:29.931669", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2019-12-05T04:59:31.836 INFO:teuthology.task.ansible.out:changed: [smithi040.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_2', u'dm-uuid-LVM-EVON5M0Nga3DT6Bz22oa0c4l4NKx5MI9VvAxZM3Godu4NdEpKLyrQM3GcV4C9mtF'], u'uuids': [u'04bedc7d-0a11-4678-9ade-0cac08c44bb6']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-3'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.161510", "end": "2019-12-05 04:59:31.492157", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-EVON5M0Nga3DT6Bz22oa0c4l4NKx5MI9VvAxZM3Godu4NdEpKLyrQM3GcV4C9mtF"], "labels": [], "masters": [], "uuids": ["04bedc7d-0a11-4678-9ade-0cac08c44bb6"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-12-05 04:59:30.330647", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2019-12-05T04:59:31.985 INFO:teuthology.task.ansible.out:changed: [smithi042.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_5', u'dm-uuid-LVM-kGEgBNXyNFrdJsrD7MkvFoTBTW8C1m51esy2SAkZtPDr03e75SgzET1qhhO7iwvZ'], u'uuids': [u'09e94049-b271-43c8-b15b-6899b9da3cc8']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'14.90 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'31252480', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-0'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.116629", "end": "2019-12-05 04:59:32.292947", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-kGEgBNXyNFrdJsrD7MkvFoTBTW8C1m51esy2SAkZtPDr03e75SgzET1qhhO7iwvZ"], "labels": [], "masters": [], "uuids": ["09e94049-b271-43c8-b15b-6899b9da3cc8"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-12-05 04:59:31.176318", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2019-12-05T04:59:33.109 INFO:teuthology.task.ansible.out:changed: [smithi040.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_5', u'dm-uuid-LVM-EVON5M0Nga3DT6Bz22oa0c4l4NKx5MI9ESTJg2C8cFtyJEeCzbdu9rhXXLvbXzhU'], u'uuids': [u'81639f94-3923-46d6-b8b4-b9391b2da328']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'14.90 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'31252480', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-0'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.124091", "end": "2019-12-05 04:59:32.759939", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-EVON5M0Nga3DT6Bz22oa0c4l4NKx5MI9ESTJg2C8cFtyJEeCzbdu9rhXXLvbXzhU"], "labels": [], "masters": [], "uuids": ["81639f94-3923-46d6-b8b4-b9391b2da328"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-12-05 04:59:31.635848", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2019-12-05T04:59:33.239 INFO:teuthology.task.ansible.out:changed: [smithi042.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_4', u'dm-uuid-LVM-kGEgBNXyNFrdJsrD7MkvFoTBTW8C1m51NmQc5XQdgNCe0DWUlMK1wlZ6J9UtuhMA'], u'uuids': [u'506a7678-9773-4c98-9417-760030db9cc6']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.112282", "end": "2019-12-05 04:59:33.537007", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-kGEgBNXyNFrdJsrD7MkvFoTBTW8C1m51NmQc5XQdgNCe0DWUlMK1wlZ6J9UtuhMA"], "labels": [], "masters": [], "uuids": ["506a7678-9773-4c98-9417-760030db9cc6"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-12-05 04:59:32.424725", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2019-12-05T04:59:33.302 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:33 +0000 (0:00:07.414) 0:02:08.804 ***** 2019-12-05T04:59:33.355 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2019-12-05T04:59:33.357 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': u'ATA', u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W4J2MS', u'wwn-0x5000c500871b7906'], u'uuids': []}, u'sas_device_handle': None, u'host': u'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', u'support_discard': u'0', u'serial': u'Z1W4J2MS', u'wwn': u'0x5000c500871b7906', u'holders': [], u'size': u'931.51 GB', u'scheduler_mode': u'cfq', u'rotational': u'1', u'sectors': u'1953525168', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': u'ST1000NM0033-9ZM', u'partitions': {u'sda1': {u'start': u'2048', u'sectorsize': 512, u'uuid': u'e17a1a1d-4a16-4f42-a170-a33513a36c81', u'sectors': u'1953522688', u'holders': [], u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W4J2MS-part1', u'wwn-0x5000c500871b7906-part1'], u'uuids': [u'e17a1a1d-4a16-4f42-a170-a33513a36c81']}, u'size': u'931.51 GB'}}}, 'key': u'sda'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4J2MS", "wwn-0x5000c500871b7906"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4J2MS-part1", "wwn-0x5000c500871b7906-part1"], "labels": [], "masters": [], "uuids": ["e17a1a1d-4a16-4f42-a170-a33513a36c81"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "e17a1a1d-4a16-4f42-a170-a33513a36c81"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W4J2MS", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "vir 2019-12-05T04:59:33.359 INFO:teuthology.task.ansible.out:tual": 1, "wwn": "0x5000c500871b7906"}}, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:33.372 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:33.388 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:33.401 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:33.414 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:33.416 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:33.430 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:33.460 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} skipping: [smithi042.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:33.638 INFO:teuthology.task.ansible.out:changed: [smithi042.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [u'dm-0', u'dm-1', u'dm-2', u'dm-3', u'dm-4'], u'labels': [], u'ids': [u'lvm-pv-uuid-us223U-zYGd-W0e4-tLE6-JeIM-Up89-WrMgMa', u'nvme-INTEL_SSDPED1D480GA_PHMB75130021480DGN', u'nvme-nvme.8086-50484d42373531333030323134383044474e-494e54454c2053534450454431443438304741-00000001'], u'uuids': []}, u'sas_device_handle': None, u'host': u'Non-Volatile memory controller: Intel Corporation Optane SSD 900P Series', u'support_discard': u'512', u'model': u'INTEL SSDPED1D480GA', u'size': u'447.13 GB', u'scheduler_mode': u'none', u'rotational': u'0', u'sectors': u'937703088', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [u'vg_nvme-lv_4', u'vg_nvme-lv_1', u'vg_nvme-lv_3', u'vg_nvme-lv_5', u'vg_nvme-lv_2'], u'partitions': {}}, 'key': u'nvme0n1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.049283", "end": "2019-12-05 04:59:33.939720", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_4", "vg_nvme-lv_1", "vg_nvme-lv_3", "vg_nvme-lv_5", "vg_nvme-lv_2"], "host": "Non-Volatile memory controller: Intel Corporation Optane SSD 900P Series", "links": {"ids": ["lvm-pv-uuid-us223U-zYGd-W0e4-tLE6-JeIM-Up89-WrMgMa", "nvme-INTEL_SSDPED1D480GA_PHMB75130021480DGN", "nvme-nvme.8086-50484d42373531333030323134383044474e-494e54454c2053534450454431443438304741-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPED1D480GA", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "937703088", "sectorsize": "512", "size": "447.13 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-12-05 04:59:33.890437", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 2019-12-05T04:59:33.640 INFO:teuthology.task.ansible.out: 0.0374365 s, 3.1 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0374365 s, 3.1 GB/s"], "stdout": "", "stdout_lines": []} 2019-12-05T04:59:33.865 INFO:teuthology.task.ansible.out:changed: [smithi042.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_1', u'dm-uuid-LVM-kGEgBNXyNFrdJsrD7MkvFoTBTW8C1m51PnUroqiSFpgYzOAdIiPrhGUTFehYQwbd'], u'uuids': [u'e3bc4f1c-facf-43d6-b954-3ed3e9f1fc2d']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-4'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.095850", "end": "2019-12-05 04:59:34.162971", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-kGEgBNXyNFrdJsrD7MkvFoTBTW8C1m51PnUroqiSFpgYzOAdIiPrhGUTFehYQwbd"], "labels": [], "masters": [], "uuids": ["e3bc4f1c-facf-43d6-b954-3ed3e9f1fc2d"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-12-05 04:59:34.067121", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.092905 s, 1.2 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.092905 s, 1.2 GB/s"], "stdout": "", "stdout_lines": []} 2019-12-05T04:59:34.085 INFO:teuthology.task.ansible.out:changed: [smithi042.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_3', u'dm-uuid-LVM-kGEgBNXyNFrdJsrD7MkvFoTBTW8C1m512vkFgqg3JbojJb10AaGWc1orJGACksdV'], u'uuids': [u'11049691-dfea-4fde-ac44-4ca7ce918ade']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-2'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.100533", "end": "2019-12-05 04:59:34.384174", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-kGEgBNXyNFrdJsrD7MkvFoTBTW8C1m512vkFgqg3JbojJb10AaGWc1orJGACksdV"], "labels": [], "masters": [], "uuids": ["11049691-dfea-4fde-ac44-4ca7ce918ade"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-12-05 04:59:34.283641", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0967315 s, 1.2 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0967315 s, 1.2 GB/s"], "stdout": "", "stdout_lines": []} 2019-12-05T04:59:34.316 INFO:teuthology.task.ansible.out:changed: [smithi042.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_2', u'dm-uuid-LVM-kGEgBNXyNFrdJsrD7MkvFoTBTW8C1m51kSlnQNUeBugaJsg0js1tubFcNSKAA3fr'], u'uuids': [u'f2907880-623c-4750-8f97-579ad55e674c']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-3'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.096864", "end": "2019-12-05 04:59:34.609356", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-kGEgBNXyNFrdJsrD7MkvFoTBTW8C1m51kSlnQNUeBugaJsg0js1tubFcNSKAA3fr"], "labels": [], "masters": [], "uuids": ["f2907880-623c-4750-8f97-579ad55e674c"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-12-05 04:59:34.512492", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0944421 s, 1.2 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0944421 s, 1.2 GB/s"], "stdout": "", "stdout_lines": []} 2019-12-05T04:59:34.393 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** changed: [smithi040.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_4', u'dm-uuid-LVM-EVON5M0Nga3DT6Bz22oa0c4l4NKx5MI92luzPlOnwneEcYnzGwp3iMlyxvq0hXI2'], u'uuids': [u'b6a88c8e-df52-4617-ab53-86b566021aae']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.121861", "end": "2019-12-05 04:59:34.023685", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-EVON5M0Nga3DT6Bz22oa0c4l4NKx5MI92luzPlOnwneEcYnzGwp3iMlyxvq0hXI2"], "labels": [], "masters": [], "uuids": ["b6a88c8e-df52-4617-ab53-86b566021aae"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-12-05 04:59:32.901824", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! 2019-12-05T04:59:34.395 INFO:teuthology.task.ansible.out:You may now partition the disk using fdisk or", "other utilities."]} 2019-12-05T04:59:34.447 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:34 +0000 (0:00:01.145) 0:02:09.949 ***** 2019-12-05T04:59:34.496 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2019-12-05T04:59:34.497 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': u'ATA', u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W4HQ07', u'wwn-0x5000c500871a8a38'], u'uuids': []}, u'sas_device_handle': None, u'host': u'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', u'support_discard': u'0', u'serial': u'Z1W4HQ07', u'wwn': u'0x5000c500871a8a38', u'holders': [], u'size': u'931.51 GB', u'scheduler_mode': u'cfq', u'rotational': u'1', u'sectors': u'1953525168', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': u'ST1000NM0033-9ZM', u'partitions': {u'sda1': {u'start': u'2048', u'sectorsize': 512, u'uuid': u'e17a1a1d-4a16-4f42-a170-a33513a36c81', u'sectors': u'1953522688', u'holders': [], u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W4HQ07-part1', u'wwn-0x5000c500871a8a38-part1'], u'uuids': [u'e17a1a1d-4a16-4f42-a170-a33513a36c81']}, u'size': u'931.51 GB'}}}, 'key': u'sda'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4HQ07", "wwn-0x5000c500871a8a38"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4HQ07-part1", "wwn-0x5000c500871a8a38-part1"], "labels": [], "masters": [], "uuids": ["e17a1a1d-4a16-4f42-a170-a33513a36c81"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "e17a1a1d-4a16-4f42-a170-a33513a36c81"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W4HQ07", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "vir 2019-12-05T04:59:34.500 INFO:teuthology.task.ansible.out:tual": 1, "wwn": "0x5000c500871a8a38"}}, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:34.510 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:34.512 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:34.525 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:34.538 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:34.540 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:34.552 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:34.555 INFO:teuthology.task.ansible.out:changed: [smithi042.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_5', u'dm-uuid-LVM-kGEgBNXyNFrdJsrD7MkvFoTBTW8C1m51esy2SAkZtPDr03e75SgzET1qhhO7iwvZ'], u'uuids': [u'09e94049-b271-43c8-b15b-6899b9da3cc8']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'14.90 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'31252480', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-0'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.104542", "end": "2019-12-05 04:59:34.851727", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-kGEgBNXyNFrdJsrD7MkvFoTBTW8C1m51esy2SAkZtPDr03e75SgzET1qhhO7iwvZ"], "labels": [], "masters": [], "uuids": ["09e94049-b271-43c8-b15b-6899b9da3cc8"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-12-05 04:59:34.747185", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.101904 s, 1.1 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.101904 s, 1.1 GB/s"], "stdout": "", "stdout_lines": []} 2019-12-05T04:59:34.567 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:34.580 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:34.764 INFO:teuthology.task.ansible.out:changed: [smithi040.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [u'dm-0', u'dm-1', u'dm-2', u'dm-3', u'dm-4'], u'labels': [], u'ids': [u'lvm-pv-uuid-p51jR8-WU6f-eKe8-qfWs-0sFT-TO0l-gkMZPH', u'nvme-INTEL_SSDPEDMD400G4_CVFT5330009J400BGN', u'nvme-nvme.8086-43564654353333303030394a34303042474e-494e54454c205353445045444d443430304734-00000001'], u'uuids': []}, u'sas_device_handle': None, u'host': u'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', u'support_discard': u'512', u'model': u'INTEL SSDPEDMD400G4', u'size': u'372.61 GB', u'scheduler_mode': u'none', u'rotational': u'0', u'sectors': u'781422768', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [u'vg_nvme-lv_4', u'vg_nvme-lv_1', u'vg_nvme-lv_3', u'vg_nvme-lv_5', u'vg_nvme-lv_2'], u'partitions': {}}, 'key': u'nvme0n1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.052379", "end": "2019-12-05 04:59:34.421472", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_4", "vg_nvme-lv_1", "vg_nvme-lv_3", "vg_nvme-lv_5", "vg_nvme-lv_2"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-p51jR8-WU6f-eKe8-qfWs-0sFT-TO0l-gkMZPH", "nvme-INTEL_SSDPEDMD400G4_CVFT5330009J400BGN", "nvme-nvme.8086-43564654353333303030394a34303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-12-05 04:59:34.369093", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 11 2019-12-05T04:59:34.766 INFO:teuthology.task.ansible.out:0 MiB) copied, 0.0415715 s, 2.8 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0415715 s, 2.8 GB/s"], "stdout": "", "stdout_lines": []} 2019-12-05T04:59:34.824 INFO:teuthology.task.ansible.out:changed: [smithi042.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_4', u'dm-uuid-LVM-kGEgBNXyNFrdJsrD7MkvFoTBTW8C1m51NmQc5XQdgNCe0DWUlMK1wlZ6J9UtuhMA'], u'uuids': [u'506a7678-9773-4c98-9417-760030db9cc6']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.113101", "end": "2019-12-05 04:59:35.107611", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-kGEgBNXyNFrdJsrD7MkvFoTBTW8C1m51NmQc5XQdgNCe0DWUlMK1wlZ6J9UtuhMA"], "labels": [], "masters": [], "uuids": ["506a7678-9773-4c98-9417-760030db9cc6"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-12-05 04:59:34.994510", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.110276 s, 1.0 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.110276 s, 1.0 GB/s"], "stdout": "", "stdout_lines": []} 2019-12-05T04:59:34.896 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:34 +0000 (0:00:00.449) 0:02:10.398 ***** 2019-12-05T04:59:35.048 INFO:teuthology.task.ansible.out:changed: [smithi040.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_1', u'dm-uuid-LVM-EVON5M0Nga3DT6Bz22oa0c4l4NKx5MI9UDekeENvlKU29ruh126smjMT7Xh6RMfb'], u'uuids': [u'e7617c40-ea15-40e8-bce1-bd8faae80f81']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-4'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.149926", "end": "2019-12-05 04:59:34.703095", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-EVON5M0Nga3DT6Bz22oa0c4l4NKx5MI9UDekeENvlKU29ruh126smjMT7Xh6RMfb"], "labels": [], "masters": [], "uuids": ["e7617c40-ea15-40e8-bce1-bd8faae80f81"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-12-05 04:59:34.553169", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.146815 s, 786 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.146815 s, 786 MB/s"], "stdout": "", "stdout_lines": []} 2019-12-05T04:59:35.216 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** 2019-12-05T04:59:35.218 INFO:teuthology.task.ansible.out:changed: [smithi042.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.154977", "end": "2019-12-05 04:59:35.521066", "rc": 0, "start": "2019-12-05 04:59:35.366089", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-12-05T04:59:35.259 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:35 +0000 (0:00:00.363) 0:02:10.762 ***** 2019-12-05T04:59:35.351 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** changed: [smithi040.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_3', u'dm-uuid-LVM-EVON5M0Nga3DT6Bz22oa0c4l4NKx5MI9ZesOXeekd8Gf9Nsrm8ukSFbKfhmc1RSV'], u'uuids': [u'b147b9bd-00f6-4e74-bde2-8245cabbcd6b']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-2'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.152855", "end": "2019-12-05 04:59:35.012749", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-EVON5M0Nga3DT6Bz22oa0c4l4NKx5MI9ZesOXeekd8Gf9Nsrm8ukSFbKfhmc1RSV"], "labels": [], "masters": [], "uuids": ["b147b9bd-00f6-4e74-bde2-8245cabbcd6b"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-12-05 04:59:34.859894", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.148917 s, 775 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.148917 s, 775 MB/s"], "stdout": "", "stdout_lines": []} 2019-12-05T04:59:35.478 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** 2019-12-05T04:59:35.480 INFO:teuthology.task.ansible.out:changed: [smithi042.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.074032", "end": "2019-12-05 04:59:35.782422", "rc": 0, "start": "2019-12-05 04:59:35.708390", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-12-05T04:59:35.524 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:35 +0000 (0:00:00.265) 0:02:11.027 ***** 2019-12-05T04:59:35.552 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2019-12-05T04:59:35.597 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:35 +0000 (0:00:00.073) 0:02:11.100 ***** 2019-12-05T04:59:35.639 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:35.690 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:35 +0000 (0:00:00.092) 0:02:11.192 ***** 2019-12-05T04:59:35.697 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2019-12-05T04:59:35.699 INFO:teuthology.task.ansible.out:changed: [smithi040.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_2', u'dm-uuid-LVM-EVON5M0Nga3DT6Bz22oa0c4l4NKx5MI9VvAxZM3Godu4NdEpKLyrQM3GcV4C9mtF'], u'uuids': [u'04bedc7d-0a11-4678-9ade-0cac08c44bb6']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-3'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.154615", "end": "2019-12-05 04:59:35.301418", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-EVON5M0Nga3DT6Bz22oa0c4l4NKx5MI9VvAxZM3Godu4NdEpKLyrQM3GcV4C9mtF"], "labels": [], "masters": [], "uuids": ["04bedc7d-0a11-4678-9ade-0cac08c44bb6"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-12-05 04:59:35.146803", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.149995 s, 769 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.149995 s, 769 MB/s"], "stdout": "", "stdout_lines": []} 2019-12-05T04:59:35.724 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** 2019-12-05T04:59:35.727 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:35.772 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:35 +0000 (0:00:00.077) 0:02:11.270 ***** 2019-12-05T04:59:35.795 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:35.846 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:35 +0000 (0:00:00.072) 0:02:11.342 ***** 2019-12-05T04:59:35.895 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** 2019-12-05T04:59:35.897 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:35.940 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:35 +0000 (0:00:00.101) 0:02:11.443 ***** 2019-12-05T04:59:35.978 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2019-12-05T04:59:35.979 INFO:teuthology.task.ansible.out:changed: [smithi040.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_5', u'dm-uuid-LVM-EVON5M0Nga3DT6Bz22oa0c4l4NKx5MI9ESTJg2C8cFtyJEeCzbdu9rhXXLvbXzhU'], u'uuids': [u'81639f94-3923-46d6-b8b4-b9391b2da328']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'14.90 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'31252480', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-0'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.170593", "end": "2019-12-05 04:59:35.626757", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-EVON5M0Nga3DT6Bz22oa0c4l4NKx5MI9ESTJg2C8cFtyJEeCzbdu9rhXXLvbXzhU"], "labels": [], "masters": [], "uuids": ["81639f94-3923-46d6-b8b4-b9391b2da328"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-12-05 04:59:35.456164", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.165707 s, 696 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.165707 s, 696 MB/s"], "stdout": "", "stdout_lines": []} 2019-12-05T04:59:36.081 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ 2019-12-05T04:59:36.082 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2019-12-05T04:59:36.158 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:36 +0000 (0:00:00.217) 0:02:11.661 ***** 2019-12-05T04:59:36.217 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2019-12-05T04:59:36.218 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => (item={u'block_used': 13715725, u'uuid': u'e17a1a1d-4a16-4f42-a170-a33513a36c81', u'size_total': 983429562368, u'block_total': 240095108, u'mount': u'/', u'block_available': 226379383, u'size_available': 927249952768, u'fstype': u'ext4', u'inode_total': 61054976, u'inode_available': 60905935, u'device': u'/dev/sda1', u'inode_used': 149041, u'block_size': 4096, u'options': u'rw,relatime,errors=remount-ro,data=ordered'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226379383, "block_size": 4096, "block_total": 240095108, "block_used": 13715725, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60905935, "inode_total": 61054976, "inode_used": 149041, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927249952768, "size_total": 983429562368, "uuid": "e17a1a1d-4a16-4f42-a170-a33513a36c81"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:36.303 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:36 +0000 (0:00:00.145) 0:02:11.806 ***** 2019-12-05T04:59:36.313 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2019-12-05T04:59:36.315 INFO:teuthology.task.ansible.out:changed: [smithi040.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_4', u'dm-uuid-LVM-EVON5M0Nga3DT6Bz22oa0c4l4NKx5MI92luzPlOnwneEcYnzGwp3iMlyxvq0hXI2'], u'uuids': [u'b6a88c8e-df52-4617-ab53-86b566021aae']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.150007", "end": "2019-12-05 04:59:35.923007", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-EVON5M0Nga3DT6Bz22oa0c4l4NKx5MI92luzPlOnwneEcYnzGwp3iMlyxvq0hXI2"], "labels": [], "masters": [], "uuids": ["b6a88c8e-df52-4617-ab53-86b566021aae"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-12-05 04:59:35.773000", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.145152 s, 795 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.145152 s, 795 MB/s"], "stdout": "", "stdout_lines": []} 2019-12-05T04:59:36.340 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* 2019-12-05T04:59:36.342 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:36.397 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:36 +0000 (0:00:00.093) 0:02:11.900 ***** 2019-12-05T04:59:36.466 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:36 +0000 (0:00:00.059) 0:02:11.959 ***** 2019-12-05T04:59:36.486 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2019-12-05T04:59:36.488 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:36.528 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:36 +0000 (0:00:00.071) 0:02:12.030 ***** 2019-12-05T04:59:36.564 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2019-12-05T04:59:36.565 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:36.621 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:36 +0000 (0:00:00.093) 0:02:12.124 ***** 2019-12-05T04:59:36.648 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ 2019-12-05T04:59:36.650 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:36.692 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:36 +0000 (0:00:00.070) 0:02:12.195 ***** 2019-12-05T04:59:36.698 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** 2019-12-05T04:59:36.700 INFO:teuthology.task.ansible.out:changed: [smithi040.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.132060", "end": "2019-12-05 04:59:36.332502", "rc": 0, "start": "2019-12-05 04:59:36.200442", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-12-05T04:59:36.744 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:36 +0000 (0:00:00.051) 0:02:12.246 ***** 2019-12-05T04:59:36.750 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* 2019-12-05T04:59:36.751 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:36.792 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:36 +0000 (0:00:00.048) 0:02:12.295 ***** 2019-12-05T04:59:36.829 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:36.879 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:36 +0000 (0:00:00.084) 0:02:12.379 ***** 2019-12-05T04:59:36.925 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** 2019-12-05T04:59:36.926 INFO:teuthology.task.ansible.out:changed: [smithi040.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.046822", "end": "2019-12-05 04:59:36.588945", "rc": 0, "start": "2019-12-05 04:59:36.542123", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-12-05T04:59:36.975 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:36 +0000 (0:00:00.097) 0:02:12.477 ***** 2019-12-05T04:59:37.003 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2019-12-05T04:59:37.054 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:37 +0000 (0:00:00.078) 0:02:12.557 ***** 2019-12-05T04:59:37.096 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* 2019-12-05T04:59:37.098 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:37.151 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:37 +0000 (0:00:00.097) 0:02:12.654 ***** 2019-12-05T04:59:37.192 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:37.236 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:37 +0000 (0:00:00.084) 0:02:12.739 ***** 2019-12-05T04:59:37.264 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:37.343 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:37 +0000 (0:00:00.106) 0:02:12.845 ***** 2019-12-05T04:59:37.384 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** 2019-12-05T04:59:37.386 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:37.450 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:37 +0000 (0:00:00.106) 0:02:12.952 ***** 2019-12-05T04:59:37.481 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume group(s)] *************************************** 2019-12-05T04:59:37.483 INFO:teuthology.task.ansible.out:changed: [smithi042.front.sepia.ceph.com] => (item={'value': {u'pvs': u'/dev/nvme0n1'}, 'key': u'vg_nvme'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2019-12-05T04:59:37.530 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:37 +0000 (0:00:00.080) 0:02:13.032 ***** 2019-12-05T04:59:37.588 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ 2019-12-05T04:59:37.590 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2019-12-05T04:59:37.629 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:37 +0000 (0:00:00.099) 0:02:13.132 ***** 2019-12-05T04:59:37.687 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2019-12-05T04:59:37.689 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => (item={u'block_used': 13713675, u'uuid': u'e17a1a1d-4a16-4f42-a170-a33513a36c81', u'size_total': 983429562368, u'block_total': 240095108, u'mount': u'/', u'block_available': 226381433, u'size_available': 927258349568, u'fstype': u'ext4', u'inode_total': 61054976, u'inode_available': 60905936, u'device': u'/dev/sda1', u'inode_used': 149040, u'block_size': 4096, u'options': u'rw,relatime,errors=remount-ro,data=ordered'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226381433, "block_size": 4096, "block_total": 240095108, "block_used": 13713675, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60905936, "inode_total": 61054976, "inode_used": 149040, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927258349568, "size_total": 983429562368, "uuid": "e17a1a1d-4a16-4f42-a170-a33513a36c81"}, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:37.729 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:37 +0000 (0:00:00.100) 0:02:13.232 ***** 2019-12-05T04:59:37.790 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* 2019-12-05T04:59:37.792 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:37.849 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:37 +0000 (0:00:00.119) 0:02:13.352 ***** 2019-12-05T04:59:37.886 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2019-12-05T04:59:37.889 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:37.935 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:37 +0000 (0:00:00.085) 0:02:13.437 ***** 2019-12-05T04:59:37.962 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2019-12-05T04:59:37.964 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:38.005 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:38 +0000 (0:00:00.070) 0:02:13.508 ***** 2019-12-05T04:59:38.054 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2019-12-05T04:59:38.056 INFO:teuthology.task.ansible.out:changed: [smithi042.front.sepia.ceph.com] => (item={'value': {u'vg': u'vg_nvme', u'size': u'14.9G'}, 'key': u'lv_5'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} TASK [testnode : Create volume_groups dict] ************************************ skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:38.102 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:38 +0000 (0:00:00.096) 0:02:13.604 ***** 2019-12-05T04:59:38.131 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:38.173 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:38 +0000 (0:00:00.071) 0:02:13.675 ***** 2019-12-05T04:59:38.210 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** 2019-12-05T04:59:38.212 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:38.254 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:38 +0000 (0:00:00.081) 0:02:13.757 ***** 2019-12-05T04:59:38.260 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2019-12-05T04:59:38.261 INFO:teuthology.task.ansible.out:changed: [smithi042.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_4'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2019-12-05T04:59:38.488 INFO:teuthology.task.ansible.out:changed: [smithi042.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_3'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2019-12-05T04:59:38.592 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume group(s)] *************************************** 2019-12-05T04:59:38.594 INFO:teuthology.task.ansible.out:changed: [smithi040.front.sepia.ceph.com] => (item={'value': {u'pvs': u'/dev/nvme0n1'}, 'key': u'vg_nvme'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2019-12-05T04:59:38.648 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:38 +0000 (0:00:00.394) 0:02:14.151 ***** 2019-12-05T04:59:38.719 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2019-12-05T04:59:38.881 INFO:teuthology.task.ansible.out:changed: [smithi042.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_2'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2019-12-05T04:59:38.951 INFO:teuthology.task.ansible.out:changed: [smithi042.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_1'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2019-12-05T04:59:39.001 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:38 +0000 (0:00:00.352) 0:02:14.504 ***** 2019-12-05T04:59:39.009 INFO:teuthology.task.ansible.out:changed: [smithi040.front.sepia.ceph.com] => (item={'value': {u'vg': u'vg_nvme', u'size': u'14.9G'}, 'key': u'lv_5'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2019-12-05T04:59:39.167 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2019-12-05T04:59:39.169 INFO:teuthology.task.ansible.out:changed: [smithi042.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2019-12-05T04:59:39.209 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:39 +0000 (0:00:00.208) 0:02:14.712 ***** 2019-12-05T04:59:39.248 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* changed: [smithi040.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_4'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2019-12-05T04:59:39.260 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** skipping: [smithi042.front.sepia.ceph.com] => (item={'value': {u'vg': u'vg_nvme', u'size': u'14.9G'}, 'key': u'lv_5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:39.377 INFO:teuthology.task.ansible.out:changed: [smithi042.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_4'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2019-12-05T04:59:39.485 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* changed: [smithi040.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_3'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2019-12-05T04:59:39.518 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2019-12-05T04:59:39.523 INFO:teuthology.task.ansible.out:changed: [smithi042.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_3'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2019-12-05T04:59:39.640 INFO:teuthology.task.ansible.out:changed: [smithi042.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_2'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2019-12-05T04:59:39.746 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2019-12-05T04:59:39.748 INFO:teuthology.task.ansible.out:changed: [smithi040.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_2'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2019-12-05T04:59:39.770 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2019-12-05T04:59:39.772 INFO:teuthology.task.ansible.out:changed: [smithi042.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_1'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2019-12-05T04:59:39.825 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:39 +0000 (0:00:00.615) 0:02:15.328 ***** 2019-12-05T04:59:39.957 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** 2019-12-05T04:59:39.958 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "size": 4096, "state": "directory", "uid": 0} 2019-12-05T04:59:40.008 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:40 +0000 (0:00:00.182) 0:02:15.510 ***** 2019-12-05T04:59:40.016 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* changed: [smithi040.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_1'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2019-12-05T04:59:40.069 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:40 +0000 (0:00:00.061) 0:02:15.572 ***** 2019-12-05T04:59:40.075 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** 2019-12-05T04:59:40.076 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => {"ansible_facts": {"xfs_opts": "-m crc=0,finobt=0"}, "changed": false} 2019-12-05T04:59:40.124 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:40 +0000 (0:00:00.050) 0:02:15.623 ***** 2019-12-05T04:59:40.219 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2019-12-05T04:59:40.221 INFO:teuthology.task.ansible.out:changed: [smithi040.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2019-12-05T04:59:40.268 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:40 +0000 (0:00:00.147) 0:02:15.771 ***** 2019-12-05T04:59:40.311 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2019-12-05T04:59:40.313 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => (item={'value': {u'vg': u'vg_nvme', u'size': u'14.9G'}, 'key': u'lv_5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:40.450 INFO:teuthology.task.ansible.out:changed: [smithi040.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_4'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2019-12-05T04:59:40.576 INFO:teuthology.task.ansible.out:changed: [smithi040.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_3'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2019-12-05T04:59:40.691 INFO:teuthology.task.ansible.out:changed: [smithi040.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_2'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2019-12-05T04:59:40.811 INFO:teuthology.task.ansible.out:changed: [smithi040.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_1'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2019-12-05T04:59:40.855 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:40 +0000 (0:00:00.586) 0:02:16.357 ***** 2019-12-05T04:59:41.008 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** 2019-12-05T04:59:41.009 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "size": 4096, "state": "directory", "uid": 0} 2019-12-05T04:59:41.051 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:41 +0000 (0:00:00.196) 0:02:16.554 ***** 2019-12-05T04:59:41.089 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** 2019-12-05T04:59:41.091 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => {"ansible_facts": {"xfs_opts": "-m crc=0,finobt=0"}, "changed": false} 2019-12-05T04:59:41.132 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:41 +0000 (0:00:00.081) 0:02:16.635 ***** 2019-12-05T04:59:41.377 INFO:teuthology.task.ansible.out: TASK [testnode : Create xfs filesystem on /dev/vg_nvme/lv_5] ******************* 2019-12-05T04:59:41.380 INFO:teuthology.task.ansible.out:changed: [smithi040.front.sepia.ceph.com] => {"changed": true} 2019-12-05T04:59:41.424 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:41 +0000 (0:00:00.291) 0:02:16.927 ***** 2019-12-05T04:59:41.575 INFO:teuthology.task.ansible.out:changed: [smithi042.front.sepia.ceph.com] => {"changed": true} 2019-12-05T04:59:41.617 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:41 +0000 (0:00:00.192) 0:02:17.120 ***** 2019-12-05T04:59:41.918 INFO:teuthology.task.ansible.out: TASK [testnode : Mount /dev/vg_nvme/lv_5 to /var/lib/ceph] ********************* changed: [smithi042.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2019-12-05T04:59:41.964 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:41 +0000 (0:00:00.347) 0:02:17.467 ***** 2019-12-05T04:59:41.971 INFO:teuthology.task.ansible.out:changed: [smithi040.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2019-12-05T04:59:42.018 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:42 +0000 (0:00:00.053) 0:02:17.520 ***** 2019-12-05T04:59:42.109 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the teuthology user.] *********** 2019-12-05T04:59:42.117 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => {"changed": false, "gid": 1216, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 1000} 2019-12-05T04:59:42.152 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:42 +0000 (0:00:00.134) 0:02:17.655 ***** 2019-12-05T04:59:42.169 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => {"changed": false, "gid": 1216, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 1000} 2019-12-05T04:59:42.213 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:42 +0000 (0:00:00.060) 0:02:17.715 ***** 2019-12-05T04:59:42.294 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the root user.] ***************** ok: [smithi042.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 0} 2019-12-05T04:59:42.336 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:42 +0000 (0:00:00.122) 0:02:17.838 ***** 2019-12-05T04:59:42.353 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 0} 2019-12-05T04:59:42.396 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:42 +0000 (0:00:00.059) 0:02:17.898 ***** 2019-12-05T04:59:42.679 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for the teuthology user.] ****************** 2019-12-05T04:59:42.680 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "gid": 1216, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 1000} 2019-12-05T04:59:42.722 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:42 +0000 (0:00:00.326) 0:02:18.225 ***** 2019-12-05T04:59:42.748 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "gid": 1216, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 1000} 2019-12-05T04:59:42.792 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:42 +0000 (0:00:00.070) 0:02:18.295 ***** 2019-12-05T04:59:43.083 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for root.] ********************************* 2019-12-05T04:59:43.084 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/root/.cpan/CPAN/MyConfig.pm", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 0} 2019-12-05T04:59:43.133 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:43 +0000 (0:00:00.337) 0:02:18.633 ***** 2019-12-05T04:59:43.159 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/root/.cpan/CPAN/MyConfig.pm", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 0} 2019-12-05T04:59:43.208 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:43 +0000 (0:00:00.077) 0:02:18.710 ***** 2019-12-05T04:59:44.085 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure perl-doc is installed on apt systems.] ***************** ok: [smithi042.front.sepia.ceph.com] => {"cache_update_time": 1575521611, "cache_updated": false, "changed": false} 2019-12-05T04:59:44.133 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:44 +0000 (0:00:00.920) 0:02:19.630 ***** 2019-12-05T04:59:44.138 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => {"cache_update_time": 1575521656, "cache_updated": false, "changed": false} 2019-12-05T04:59:44.193 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:44 +0000 (0:00:00.063) 0:02:19.694 ***** 2019-12-05T04:59:44.347 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if Amazon::S3 is installed.] ********************* 2019-12-05T04:59:44.348 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => {"changed": false, "cmd": ["perldoc", "-l", "Amazon::S3"], "delta": "0:00:00.047646", "end": "2019-12-05 04:59:44.645964", "rc": 0, "start": "2019-12-05 04:59:44.598318", "stderr": "", "stderr_lines": [], "stdout": "/usr/local/share/perl/5.26.1/Amazon/S3.pm", "stdout_lines": ["/usr/local/share/perl/5.26.1/Amazon/S3.pm"]} 2019-12-05T04:59:44.407 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:44 +0000 (0:00:00.216) 0:02:19.910 ***** 2019-12-05T04:59:44.416 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => {"changed": false, "cmd": ["perldoc", "-l", "Amazon::S3"], "delta": "0:00:00.047819", "end": "2019-12-05 04:59:44.065493", "rc": 0, "start": "2019-12-05 04:59:44.017674", "stderr": "", "stderr_lines": [], "stdout": "/usr/local/share/perl/5.26.1/Amazon/S3.pm", "stdout_lines": ["/usr/local/share/perl/5.26.1/Amazon/S3.pm"]} 2019-12-05T04:59:44.469 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:44 +0000 (0:00:00.057) 0:02:19.967 ***** 2019-12-05T04:59:44.471 INFO:teuthology.task.ansible.out: TASK [testnode : Install Amazon::S3.] ****************************************** skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:44.515 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:44 +0000 (0:00:00.050) 0:02:20.017 ***** 2019-12-05T04:59:44.520 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:44.566 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:44 +0000 (0:00:00.050) 0:02:20.068 ***** 2019-12-05T04:59:44.571 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on rpm based systems.] ******************** 2019-12-05T04:59:44.573 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:44.616 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:44 +0000 (0:00:00.049) 0:02:20.118 ***** 2019-12-05T04:59:44.621 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:44.665 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:44 +0000 (0:00:00.048) 0:02:20.167 ***** 2019-12-05T04:59:45.537 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on apt based systems.] ******************** ok: [smithi040.front.sepia.ceph.com] => {"cache_update_time": 1575521656, "cache_updated": false, "changed": false} 2019-12-05T04:59:45.577 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:45 +0000 (0:00:00.912) 0:02:21.079 ***** 2019-12-05T04:59:45.603 INFO:teuthology.task.ansible.out: TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** 2019-12-05T04:59:45.605 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:45.648 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:45 +0000 (0:00:00.070) 0:02:21.150 ***** 2019-12-05T04:59:45.655 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on apt based systems.] ******************** 2019-12-05T04:59:45.655 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => {"cache_update_time": 1575521611, "cache_updated": false, "changed": false} 2019-12-05T04:59:45.699 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:45 +0000 (0:00:00.050) 0:02:21.201 ***** 2019-12-05T04:59:45.735 INFO:teuthology.task.ansible.out: TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** 2019-12-05T04:59:45.737 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:45.779 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:45 +0000 (0:00:00.079) 0:02:21.281 ***** 2019-12-05T04:59:45.994 INFO:teuthology.task.ansible.out: TASK [testnode : Create the ntp.conf file.] ************************************ 2019-12-05T04:59:45.996 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => {"changed": false, "checksum": "645fd5116197cd9da73b328bc11bf633a40d80e1", "dest": "/etc/ntp.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/ntp.conf", "size": 2767, "state": "file", "uid": 0} 2019-12-05T04:59:46.044 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:46 +0000 (0:00:00.265) 0:02:21.547 ***** 2019-12-05T04:59:46.103 INFO:teuthology.task.ansible.out: TASK [testnode : Create the chrony.conf file] ********************************** 2019-12-05T04:59:46.105 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:46.154 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:46 +0000 (0:00:00.106) 0:02:21.653 ***** 2019-12-05T04:59:46.189 INFO:teuthology.task.ansible.out: TASK [testnode : Create the ntp.conf file.] ************************************ 2019-12-05T04:59:46.190 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => {"changed": false, "checksum": "645fd5116197cd9da73b328bc11bf633a40d80e1", "dest": "/etc/ntp.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/ntp.conf", "size": 2767, "state": "file", "uid": 0} 2019-12-05T04:59:46.235 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:46 +0000 (0:00:00.084) 0:02:21.738 ***** 2019-12-05T04:59:46.272 INFO:teuthology.task.ansible.out: TASK [testnode : Create the chrony.conf file] ********************************** 2019-12-05T04:59:46.273 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:46.317 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:46 +0000 (0:00:00.082) 0:02:21.820 ***** 2019-12-05T04:59:46.453 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ntpd is running.] *********************************** 2019-12-05T04:59:46.455 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "ntp", "state": "started", "status": {"ActiveEnterTimestamp": "Thu 2019-12-05 04:57:01 UTC", "ActiveEnterTimestampMonotonic": "27294164", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "sysinit.target -.mount systemd-journald.socket basic.target system.slice network.target systemd-tmpfiles-setup.service", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Thu 2019-12-05 04:56:52 UTC", "AssertTimestampMonotonic": "19064056", "Before": "multi-user.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Thu 2019-12-05 04:56:52 UTC", "ConditionTimestampMonotonic": "19064056", "ConfigurationDirectoryMode": "0755", "Conflicts": "systemd-timesyncd.service shutdown.target", "ControlGroup": "/system.slice/ntp.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Network Time Service", "DevicePolicy": "auto", "Documen 2019-12-05T04:59:46.457 INFO:teuthology.task.ansible.out:tation": "man:ntpd(8)", "DynamicUser": "no", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1862", "ExecMainStartTimestamp": "Thu 2019-12-05 04:57:01 UTC", "ExecMainStartTimestampMonotonic": "27294147", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/lib/ntp/ntp-systemd-wrapper ; argv[]=/usr/lib/ntp/ntp-systemd-wrapper ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/ntp.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "ntp.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Thu 2019-12-05 04:56:52 UTC", "InactiveExitTimestampMonotonic": "19064953", "InvocationID": "cc7d46c964b3436aaeca6fdb292150c8", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128059", "LimitNPROCSoft": "128059", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "Limi 2019-12-05T04:59:46.458 INFO:teuthology.task.ansible.out:tRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128059", "LimitSIGPENDINGSoft": "128059", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1862", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "ntp.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "-.mount sysinit.target system.slice", "RequiresMountsFor": "/var/tmp /tmp", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Thu 2019-12-05 04:57:01 UTC", "StateChangeTimestampMonoton 2019-12-05T04:59:46.461 INFO:teuthology.task.ansible.out:ic": "27294164", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "2", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Thu 2019-12-05 04:57:01 UTC", "WatchdogTimestampMonotonic": "27294163", "WatchdogUSec": "0"}} 2019-12-05T04:59:46.506 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:46 +0000 (0:00:00.188) 0:02:22.009 ***** 2019-12-05T04:59:46.534 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* 2019-12-05T04:59:46.536 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:46.586 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:46 +0000 (0:00:00.079) 0:02:22.088 ***** 2019-12-05T04:59:46.594 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ntpd is running.] *********************************** 2019-12-05T04:59:46.598 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "ntp", "state": "started", "status": {"ActiveEnterTimestamp": "Thu 2019-12-05 04:56:51 UTC", "ActiveEnterTimestampMonotonic": "26621163", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "network.target systemd-journald.socket systemd-tmpfiles-setup.service basic.target -.mount sysinit.target system.slice", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Thu 2019-12-05 04:56:43 UTC", "AssertTimestampMonotonic": "18284182", "Before": "multi-user.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Thu 2019-12-05 04:56:43 UTC", "ConditionTimestampMonotonic": "18284182", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target systemd-timesyncd.service", "ControlGroup": "/system.slice/ntp.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Network Time Service", "DevicePolicy": "auto", "Documen 2019-12-05T04:59:46.600 INFO:teuthology.task.ansible.out:tation": "man:ntpd(8)", "DynamicUser": "no", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1687", "ExecMainStartTimestamp": "Thu 2019-12-05 04:56:51 UTC", "ExecMainStartTimestampMonotonic": "26621126", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/lib/ntp/ntp-systemd-wrapper ; argv[]=/usr/lib/ntp/ntp-systemd-wrapper ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/ntp.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "ntp.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Thu 2019-12-05 04:56:43 UTC", "InactiveExitTimestampMonotonic": "18284904", "InvocationID": "013a64b72e774fb1ae75e49bcff28668", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128059", "LimitNPROCSoft": "128059", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "Limi 2019-12-05T04:59:46.601 INFO:teuthology.task.ansible.out:tRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128059", "LimitSIGPENDINGSoft": "128059", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1687", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "ntp.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "-.mount system.slice sysinit.target", "RequiresMountsFor": "/tmp /var/tmp", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Thu 2019-12-05 04:56:51 UTC", "StateChangeTimestampMonoton 2019-12-05T04:59:46.603 INFO:teuthology.task.ansible.out:ic": "26621163", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "2", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Thu 2019-12-05 04:56:51 UTC", "WatchdogTimestampMonotonic": "26621161", "WatchdogUSec": "0"}} 2019-12-05T04:59:46.645 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:46 +0000 (0:00:00.058) 0:02:22.147 ***** 2019-12-05T04:59:46.672 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* 2019-12-05T04:59:46.674 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:46.718 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:46 +0000 (0:00:00.073) 0:02:22.221 ***** 2019-12-05T04:59:47.500 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on apt based systems.] ********************* ok: [smithi040.front.sepia.ceph.com] => {"cache_update_time": 1575521656, "cache_updated": false, "changed": false} 2019-12-05T04:59:47.541 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:47 +0000 (0:00:00.822) 0:02:23.043 ***** 2019-12-05T04:59:47.602 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => {"cache_update_time": 1575521611, "cache_updated": false, "changed": false} 2019-12-05T04:59:47.647 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:47 +0000 (0:00:00.106) 0:02:23.150 ***** 2019-12-05T04:59:47.678 INFO:teuthology.task.ansible.out: TASK [testnode : Create the .pip directory for the teuthology user.] *********** 2019-12-05T04:59:47.679 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => {"changed": false, "gid": 1216, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "size": 4096, "state": "directory", "uid": 1000} 2019-12-05T04:59:47.725 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:47 +0000 (0:00:00.078) 0:02:23.228 ***** 2019-12-05T04:59:47.806 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => {"changed": false, "gid": 1216, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "size": 4096, "state": "directory", "uid": 1000} 2019-12-05T04:59:47.868 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:47 +0000 (0:00:00.133) 0:02:23.362 ***** 2019-12-05T04:59:48.065 INFO:teuthology.task.ansible.out: TASK [testnode : Create pip.conf and configure it to use our mirror] *********** 2019-12-05T04:59:48.066 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1216, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "size": 73, "state": "file", "uid": 1000} 2019-12-05T04:59:48.113 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:48 +0000 (0:00:00.253) 0:02:23.615 ***** 2019-12-05T04:59:48.184 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1216, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "size": 73, "state": "file", "uid": 1000} 2019-12-05T04:59:48.237 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:48 +0000 (0:00:00.123) 0:02:23.739 ***** 2019-12-05T04:59:49.224 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages via pip] ************************************* ok: [smithi042.front.sepia.ceph.com] => {"changed": false, "cmd": ["/usr/bin/pip2", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "", "stderr_lines": [], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python2.7/dist-packages\nRequirement already satisfied: execnet in /usr/local/lib/python2.7/dist-packages (from remoto>=0.0.35)\nRequirement already satisfied: apipkg>=1.4 in /usr/local/lib/python2.7/dist-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python2.7/dist-packages", "Requirement already satisfied: execnet in /usr/local/lib/python2.7/dist-packages (from remoto>=0.0.35)", "Requirement already satisfied: apipkg>=1.4 in /usr/local/lib/python2.7/dist-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} ok: [smithi040.front.sepia.ceph.com] => {"changed": false, "cmd": ["/usr/bin/pip2", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "", "stderr_lines": [], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python2.7/dist-packages\nRequirement already satisfied: execnet in /usr/local/lib/python2.7/dist-packages (from remoto>=0.0.35)\nRequirement already satisfied: apipkg>=1.4 in /usr/local/lib/python2.7/dist-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python2.7/dist-packages", "Requirement already satisfied: execnet in /usr/local/lib/python2.7/dist-packages (from remoto>=0.0.35)", "Requirement already satisfied: apipkg>=1.4 in /usr/local/lib/python2.7/dist-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} 2019-12-05T04:59:49.281 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:49 +0000 (0:00:01.034) 0:02:24.774 ***** 2019-12-05T04:59:49.317 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:49 +0000 (0:00:00.045) 0:02:24.820 ***** 2019-12-05T04:59:49.806 INFO:teuthology.task.ansible.out: TASK [testnode : Purge resolvconf] ********************************************* 2019-12-05T04:59:49.807 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => {"changed": false} 2019-12-05T04:59:49.849 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:49 +0000 (0:00:00.532) 0:02:25.352 ***** 2019-12-05T04:59:49.877 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => {"changed": false} 2019-12-05T04:59:49.878 INFO:teuthology.task.ansible.out: TASK [testnode : Set interface] ************************************************ ok: [smithi040.front.sepia.ceph.com] => {"ansible_facts": {"interface": "ens1f0"}, "changed": false} 2019-12-05T04:59:49.919 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:49 +0000 (0:00:00.070) 0:02:25.422 ***** 2019-12-05T04:59:49.957 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:49 +0000 (0:00:00.037) 0:02:25.459 ***** 2019-12-05T04:59:50.008 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => {"ansible_facts": {"interface": "ens1f0"}, "changed": false} 2019-12-05T04:59:50.048 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:50 +0000 (0:00:00.091) 0:02:25.550 ***** 2019-12-05T04:59:50.076 INFO:teuthology.task.ansible.out: TASK [testnode : Check for presence of /etc/network/interfaces] **************** 2019-12-05T04:59:50.077 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => {"changed": false, "stat": {"atime": 1575521589.528042, "attr_flags": "e", "attributes": ["extents"], "block_size": 4096, "blocks": 8, "charset": "us-ascii", "ctime": 1575521589.524042, "dev": 2049, "device_type": 0, "executable": false, "exists": true, "gid": 0, "gr_name": "root", "inode": 68551, "isblk": false, "ischr": false, "isdir": false, "isfifo": false, "isgid": false, "islnk": false, "isreg": true, "issock": false, "isuid": false, "mimetype": "text/plain", "mode": "0644", "mtime": 1575521589.524042, "nlink": 1, "path": "/etc/network/interfaces", "pw_name": "root", "readable": true, "rgrp": true, "roth": true, "rusr": true, "size": 67, "uid": 0, "version": "1369884456", "wgrp": false, "woth": false, "writeable": true, "wusr": true, "xgrp": false, "xoth": false, "xusr": false}} 2019-12-05T04:59:50.119 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:50 +0000 (0:00:00.071) 0:02:25.622 ***** 2019-12-05T04:59:50.201 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => {"changed": false, "stat": {"atime": 1575521575.51312, "attr_flags": "e", "attributes": ["extents"], "block_size": 4096, "blocks": 8, "charset": "us-ascii", "ctime": 1575521575.5091197, "dev": 2049, "device_type": 0, "executable": false, "exists": true, "gid": 0, "gr_name": "root", "inode": 68551, "isblk": false, "ischr": false, "isdir": false, "isfifo": false, "isgid": false, "islnk": false, "isreg": true, "issock": false, "isuid": false, "mimetype": "text/plain", "mode": "0644", "mtime": 1575521575.5091197, "nlink": 1, "path": "/etc/network/interfaces", "pw_name": "root", "readable": true, "rgrp": true, "roth": true, "rusr": true, "size": 67, "uid": 0, "version": "1369884456", "wgrp": false, "woth": false, "writeable": true, "wusr": true, "xgrp": false, "xoth": false, "xusr": false}} 2019-12-05T04:59:50.242 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:50 +0000 (0:00:00.123) 0:02:25.745 ***** 2019-12-05T04:59:50.602 INFO:teuthology.task.ansible.out: TASK [testnode : Rewrite /etc/network/interfaces to use dhcp] ****************** 2019-12-05T04:59:50.603 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => {"changed": false, "msg": ""} ok: [smithi042.front.sepia.ceph.com] => {"changed": false, "msg": ""} 2019-12-05T04:59:50.643 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:50 +0000 (0:00:00.401) 0:02:26.146 ***** 2019-12-05T04:59:50.690 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:50 +0000 (0:00:00.046) 0:02:26.192 ***** 2019-12-05T04:59:50.695 INFO:teuthology.task.ansible.out: TASK [testnode : Set bounce_interface if we just enabled dhcp] ***************** 2019-12-05T04:59:50.696 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2019-12-05T04:59:50.744 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:50 +0000 (0:00:00.054) 0:02:26.247 ***** 2019-12-05T04:59:50.751 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2019-12-05T04:59:50.794 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:50 +0000 (0:00:00.049) 0:02:26.297 ***** 2019-12-05T04:59:50.800 INFO:teuthology.task.ansible.out: TASK [testnode : ifdown and ifup] ********************************************** 2019-12-05T04:59:50.802 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:50.843 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:50 +0000 (0:00:00.048) 0:02:26.345 ***** 2019-12-05T04:59:50.850 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:50.892 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:50 +0000 (0:00:00.048) 0:02:26.394 ***** 2019-12-05T04:59:50.971 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure lab_domain is in search domains in /etc/resolv.conf] *** 2019-12-05T04:59:50.973 INFO:teuthology.task.ansible.out:changed: [smithi040.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2019-12-05T04:59:51.014 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:51 +0000 (0:00:00.122) 0:02:26.517 ***** 2019-12-05T04:59:51.032 INFO:teuthology.task.ansible.out:changed: [smithi042.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2019-12-05T04:59:51.073 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:51 +0000 (0:00:00.059) 0:02:26.576 ***** 2019-12-05T04:59:51.153 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure domain is set in /etc/resolv.conf] ********************* 2019-12-05T04:59:51.154 INFO:teuthology.task.ansible.out:changed: [smithi040.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2019-12-05T04:59:51.197 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:51 +0000 (0:00:00.123) 0:02:26.699 ***** 2019-12-05T04:59:51.225 INFO:teuthology.task.ansible.out:changed: [smithi042.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2019-12-05T04:59:51.226 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ethtool is installed] ******************************* skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:51.269 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:51 +0000 (0:00:00.072) 0:02:26.772 ***** 2019-12-05T04:59:51.309 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:51 +0000 (0:00:00.039) 0:02:26.811 ***** 2019-12-05T04:59:51.316 INFO:teuthology.task.ansible.out: TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* 2019-12-05T04:59:51.317 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => (item=ens1f1) => {"ansible_loop_var": "item", "changed": false, "item": "ens1f1", "skip_reason": "Conditional result was False"} 2019-12-05T04:59:51.318 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => (item=ens1f0) => {"ansible_loop_var": "item", "changed": false, "item": "ens1f0", "skip_reason": "Conditional result was False"} 2019-12-05T04:59:51.329 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} 2019-12-05T04:59:51.331 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} 2019-12-05T04:59:51.342 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ethtool is installed] ******************************* 2019-12-05T04:59:51.344 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* skipping: [smithi040.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} 2019-12-05T04:59:51.395 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:51 +0000 (0:00:00.086) 0:02:26.898 ***** 2019-12-05T04:59:51.478 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:51 +0000 (0:00:00.081) 0:02:26.980 ***** 2019-12-05T04:59:51.483 INFO:teuthology.task.ansible.out: TASK [testnode : Define net_to_configure var] ********************************** 2019-12-05T04:59:51.485 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'ens1f1', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "ens1f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi040.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'ens1f0', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "ens1f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:51.485 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'eno2', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:51.487 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'lo', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi040.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'eno1', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:51.534 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:51 +0000 (0:00:00.056) 0:02:27.037 ***** 2019-12-05T04:59:51.541 INFO:teuthology.task.ansible.out: TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* 2019-12-05T04:59:51.542 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => (item=ens1f1) => {"ansible_loop_var": "item", "changed": false, "item": "ens1f1", "skip_reason": "Conditional result was False"} skipping: [smithi042.front.sepia.ceph.com] => (item=ens1f0) => {"ansible_loop_var": "item", "changed": false, "item": "ens1f0", "skip_reason": "Conditional result was False"} skipping: [smithi042.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} 2019-12-05T04:59:51.554 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} 2019-12-05T04:59:51.555 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:51.598 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:51 +0000 (0:00:00.064) 0:02:27.101 ***** 2019-12-05T04:59:51.604 INFO:teuthology.task.ansible.out: TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* 2019-12-05T04:59:51.606 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} 2019-12-05T04:59:51.647 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:51 +0000 (0:00:00.048) 0:02:27.149 ***** 2019-12-05T04:59:51.653 INFO:teuthology.task.ansible.out: TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:51.697 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:51 +0000 (0:00:00.050) 0:02:27.200 ***** 2019-12-05T04:59:51.704 INFO:teuthology.task.ansible.out: TASK [testnode : Define net_to_configure var] ********************************** 2019-12-05T04:59:51.706 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'ens1f1', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "ens1f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi042.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'ens1f0', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "ens1f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:51.708 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'eno2', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi042.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'lo', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:51.720 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'eno1', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:51.760 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:51 +0000 (0:00:00.062) 0:02:27.263 ***** 2019-12-05T04:59:51.766 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** 2019-12-05T04:59:51.766 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:51.810 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:51 +0000 (0:00:00.049) 0:02:27.313 ***** 2019-12-05T04:59:51.817 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2019-12-05T04:59:51.818 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:51.860 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:51 +0000 (0:00:00.049) 0:02:27.362 ***** 2019-12-05T04:59:51.865 INFO:teuthology.task.ansible.out: TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* 2019-12-05T04:59:51.866 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:51.905 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:51 +0000 (0:00:00.045) 0:02:27.408 ***** 2019-12-05T04:59:51.912 INFO:teuthology.task.ansible.out: TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** 2019-12-05T04:59:51.913 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:51.952 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:51 +0000 (0:00:00.046) 0:02:27.455 ***** 2019-12-05T04:59:51.958 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU on Ubuntu] ******************************************** 2019-12-05T04:59:51.959 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:52.012 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:52 +0000 (0:00:00.059) 0:02:27.514 ***** 2019-12-05T04:59:52.018 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** 2019-12-05T04:59:52.020 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:52.062 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:52 +0000 (0:00:00.050) 0:02:27.565 ***** 2019-12-05T04:59:52.067 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2019-12-05T04:59:52.069 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:52.112 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:52 +0000 (0:00:00.050) 0:02:27.615 ***** 2019-12-05T04:59:52.119 INFO:teuthology.task.ansible.out: TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* 2019-12-05T04:59:52.121 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:52.162 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:52 +0000 (0:00:00.049) 0:02:27.665 ***** 2019-12-05T04:59:52.168 INFO:teuthology.task.ansible.out: TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** 2019-12-05T04:59:52.170 INFO:teuthology.task.ansible.out:skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:52.221 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:52 +0000 (0:00:00.058) 0:02:27.724 ***** 2019-12-05T04:59:52.227 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU on Ubuntu] ******************************************** 2019-12-05T04:59:52.229 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:52.277 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:52 +0000 (0:00:00.050) 0:02:27.774 ***** 2019-12-05T04:59:52.279 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** skipping: [smithi040.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:52.319 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:52 +0000 (0:00:00.047) 0:02:27.822 ***** 2019-12-05T04:59:52.325 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2019-12-05T04:59:52.327 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:52.369 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:52 +0000 (0:00:00.049) 0:02:27.872 ***** 2019-12-05T04:59:52.413 INFO:teuthology.task.ansible.out: TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:52.448 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:52 +0000 (0:00:00.079) 0:02:27.951 ***** 2019-12-05T04:59:52.475 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2019-12-05T04:59:52.476 INFO:teuthology.task.ansible.out:skipping: [smithi042.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T04:59:52.515 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:52 +0000 (0:00:00.066) 0:02:28.018 ***** 2019-12-05T04:59:52.521 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2019-12-05T04:59:52.522 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.010996", "end": "2019-12-05 04:59:52.140807", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2019-12-05 04:59:52.129811", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-12-05T04:59:52.628 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.006112", "end": "2019-12-05 04:59:52.299388", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2019-12-05 04:59:52.293276", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-12-05T04:59:52.681 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.011052", "end": "2019-12-05 04:59:52.968813", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2019-12-05 04:59:52.957761", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-12-05T04:59:52.766 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.005556", "end": "2019-12-05 04:59:52.422672", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2019-12-05 04:59:52.417116", "stderr": "find: '/etc/ceph': No such file or directory", "stderr_lines": ["find: '/etc/ceph': No such file or directory"], "stdout": "", "stdout_lines": []} 2019-12-05T04:59:52.809 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.006069", "end": "2019-12-05 04:59:53.099334", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2019-12-05 04:59:53.093265", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-12-05T04:59:52.882 INFO:teuthology.task.ansible.out:ok: [smithi040.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.006264", "end": "2019-12-05 04:59:52.548743", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2019-12-05 04:59:52.542479", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-12-05T04:59:52.945 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:52 +0000 (0:00:00.429) 0:02:28.447 ***** 2019-12-05T04:59:52.953 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.005604", "end": "2019-12-05 04:59:53.225310", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2019-12-05 04:59:53.219706", "stderr": "find: '/etc/ceph': No such file or directory", "stderr_lines": ["find: '/etc/ceph': No such file or directory"], "stdout": "", "stdout_lines": []} 2019-12-05T04:59:52.965 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2019-12-05T04:59:53.005 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:53 +0000 (0:00:00.060) 0:02:28.508 ***** 2019-12-05T04:59:53.059 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2019-12-05T04:59:53.061 INFO:teuthology.task.ansible.out:ok: [smithi042.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.006902", "end": "2019-12-05 04:59:53.346361", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2019-12-05 04:59:53.339459", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-12-05T04:59:53.103 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:53 +0000 (0:00:00.097) 0:02:28.606 ***** 2019-12-05T04:59:53.130 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2019-12-05T04:59:53.131 INFO:teuthology.task.ansible.out: TASK [testnode : Touch /ceph-qa-ready] ***************************************** 2019-12-05T04:59:53.133 INFO:teuthology.task.ansible.out:changed: [smithi040.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "size": 0, "state": "file", "uid": 0} 2019-12-05T04:59:53.220 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 04:59:53 +0000 (0:00:00.116) 0:02:28.722 ***** 2019-12-05T04:59:53.373 INFO:teuthology.task.ansible.out:changed: [smithi042.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "size": 0, "state": "file", "uid": 0} 2019-12-05T04:59:53.486 INFO:teuthology.task.ansible.out: PLAY [cobbler] ***************************************************************** skipping: no hosts matched 2019-12-05T04:59:53.488 INFO:teuthology.task.ansible.out: PLAY [paddles] ***************************************************************** 2019-12-05T04:59:53.490 INFO:teuthology.task.ansible.out:skipping: no hosts matched 2019-12-05T04:59:53.492 INFO:teuthology.task.ansible.out: PLAY [pulpito] ***************************************************************** skipping: no hosts matched PLAY RECAP ********************************************************************* smithi040.front.sepia.ceph.com : ok=113 changed=32 unreachable=0 failed=0 skipped=167 rescued=0 ignored=0 smithi042.front.sepia.ceph.com : ok=113 changed=32 unreachable=0 failed=0 skipped=167 rescued=0 ignored=0 Thursday 05 December 2019 04:59:53 +0000 (0:00:00.271) 0:02:28.994 ***** =============================================================================== 2019-12-05T04:59:53.493 INFO:teuthology.task.ansible.out:users : Create all admin users with sudo access. ----------------------- 16.66s users : Update authorized_keys using the keys repo --------------------- 15.38s testnode : Ensure packages are not present. ----------------------------- 9.25s users : Remove revoked users -------------------------------------------- 9.05s testnode : Install packages --------------------------------------------- 8.06s testnode : Zap all non-root disks --------------------------------------- 7.41s common : Update apt cache ----------------------------------------------- 3.46s testnode : Update apt cache. -------------------------------------------- 2.93s Gathering Facts --------------------------------------------------------- 2.85s testnode : Install apt keys --------------------------------------------- 2.41s ansible-managed : Add authorized keys for the ansible user. ------------- 1.59s testnode : Install packages via pip ------------------------------------- 1.03s testnode : Set the system's hostname ------------------------------------ 0.95s testnode : Ensure perl-doc is installed on apt systems. ----------------- 0.92s testnode : Install ntp package on apt based systems. -------------------- 0.91s testnode : Install python-apt ------------------------------------------- 0.91s common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/. --- 0.90s common : Make sure smartmontools is installed --------------------------- 0.87s common : Install Kerberos Packages (Debian) ----------------------------- 0.86s ansible-managed : Create the cephlab_sudo sudoers.d file. --------------- 0.84s 2019-12-05T04:59:53.780 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi042.front.sepia.ceph.com'), Remote(name='ubuntu@smithi040.front.sepia.ceph.com')] 2019-12-05T04:59:53.782 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2019-12-05T04:59:53.783 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi042.front.sepia.ceph.com', 'timeout': 60} 2019-12-05T04:59:53.874 INFO:teuthology.orchestra.run.smithi042:> true 2019-12-05T04:59:54.084 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2019-12-05T04:59:54.085 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi040.front.sepia.ceph.com', 'timeout': 60} 2019-12-05T04:59:54.231 INFO:teuthology.orchestra.run.smithi040:> true 2019-12-05T04:59:54.437 INFO:teuthology.run_tasks:Running task clock... 2019-12-05T04:59:54.466 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2019-12-05T04:59:54.466 INFO:teuthology.orchestra.run.smithi042:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2019-12-05T04:59:54.529 INFO:teuthology.orchestra.run.smithi042.stdout: 5 Dec 04:59:54 ntpd[7580]: ntpd 4.2.8p10@1.3728-o (1): Starting 2019-12-05T04:59:54.531 INFO:teuthology.orchestra.run.smithi042.stdout: 5 Dec 04:59:54 ntpd[7580]: Command line: ntpd -gq 2019-12-05T04:59:54.532 INFO:teuthology.orchestra.run.smithi042.stdout: 5 Dec 04:59:54 ntpd[7580]: proto: precision = 0.127 usec (-23) 2019-12-05T04:59:54.533 INFO:teuthology.orchestra.run.smithi042.stdout: 5 Dec 04:59:54 ntpd[7580]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2019-12-05T04:59:54.533 INFO:teuthology.orchestra.run.smithi042.stdout: 5 Dec 04:59:54 ntpd[7580]: restrict ::: KOD does nothing without LIMITED. 2019-12-05T04:59:54.533 INFO:teuthology.orchestra.run.smithi042.stdout: 5 Dec 04:59:54 ntpd[7580]: Listen and drop on 0 v6wildcard [::]:123 2019-12-05T04:59:54.533 INFO:teuthology.orchestra.run.smithi042.stdout: 5 Dec 04:59:54 ntpd[7580]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2019-12-05T04:59:54.533 INFO:teuthology.orchestra.run.smithi042.stdout: 5 Dec 04:59:54 ntpd[7580]: Listen normally on 2 lo 127.0.0.1:123 2019-12-05T04:59:54.533 INFO:teuthology.orchestra.run.smithi042.stdout: 5 Dec 04:59:54 ntpd[7580]: Listen normally on 3 ens1f0 172.21.15.42:123 2019-12-05T04:59:54.533 INFO:teuthology.orchestra.run.smithi042.stdout: 5 Dec 04:59:54 ntpd[7580]: Listen normally on 4 lo [::1]:123 2019-12-05T04:59:54.534 INFO:teuthology.orchestra.run.smithi042.stdout: 5 Dec 04:59:54 ntpd[7580]: Listen normally on 5 ens1f0 [fe80::ec4:7aff:febd:1440%3]:123 2019-12-05T04:59:54.534 INFO:teuthology.orchestra.run.smithi042.stdout: 5 Dec 04:59:54 ntpd[7580]: Listening on routing socket on fd #22 for interface updates 2019-12-05T04:59:54.534 INFO:teuthology.orchestra.run.smithi042.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2019-12-05T04:59:54.534 INFO:teuthology.orchestra.run.smithi042.stderr:restrict ::: KOD does nothing without LIMITED. 2019-12-05T04:59:59.532 INFO:teuthology.orchestra.run.smithi042.stderr: 5 Dec 04:59:59 ntpd[7580]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2019-12-05T04:59:59.533 INFO:teuthology.orchestra.run.smithi042.stderr: 5 Dec 04:59:59 ntpd[7580]: can't open /var/log/ntpstats/rawstats.20191205: Permission denied 2019-12-05T04:59:59.533 INFO:teuthology.orchestra.run.smithi042.stderr: 5 Dec 04:59:59 ntpd[7580]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2019-12-05T04:59:59.533 INFO:teuthology.orchestra.run.smithi042.stderr: 5 Dec 04:59:59 ntpd[7580]: can't open /var/log/ntpstats/peerstats.20191205: Permission denied 2019-12-05T05:00:01.534 INFO:teuthology.orchestra.run.smithi042.stderr: 5 Dec 05:00:01 ntpd[7580]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2019-12-05T05:00:01.534 INFO:teuthology.orchestra.run.smithi042.stderr: 5 Dec 05:00:01 ntpd[7580]: can't open /var/log/ntpstats/rawstats.20191205: Permission denied 2019-12-05T05:00:01.534 INFO:teuthology.orchestra.run.smithi042.stderr: 5 Dec 05:00:01 ntpd[7580]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2019-12-05T05:00:01.534 INFO:teuthology.orchestra.run.smithi042.stderr: 5 Dec 05:00:01 ntpd[7580]: can't open /var/log/ntpstats/peerstats.20191205: Permission denied 2019-12-05T05:00:03.534 INFO:teuthology.orchestra.run.smithi042.stderr: 5 Dec 05:00:03 ntpd[7580]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2019-12-05T05:00:03.535 INFO:teuthology.orchestra.run.smithi042.stderr: 5 Dec 05:00:03 ntpd[7580]: can't open /var/log/ntpstats/rawstats.20191205: Permission denied 2019-12-05T05:00:03.535 INFO:teuthology.orchestra.run.smithi042.stderr: 5 Dec 05:00:03 ntpd[7580]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2019-12-05T05:00:03.535 INFO:teuthology.orchestra.run.smithi042.stderr: 5 Dec 05:00:03 ntpd[7580]: can't open /var/log/ntpstats/peerstats.20191205: Permission denied 2019-12-05T05:00:05.536 INFO:teuthology.orchestra.run.smithi042.stderr: 5 Dec 05:00:05 ntpd[7580]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2019-12-05T05:00:05.537 INFO:teuthology.orchestra.run.smithi042.stderr: 5 Dec 05:00:05 ntpd[7580]: can't open /var/log/ntpstats/rawstats.20191205: Permission denied 2019-12-05T05:00:05.537 INFO:teuthology.orchestra.run.smithi042.stderr: 5 Dec 05:00:05 ntpd[7580]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2019-12-05T05:00:05.537 INFO:teuthology.orchestra.run.smithi042.stderr: 5 Dec 05:00:05 ntpd[7580]: can't open /var/log/ntpstats/peerstats.20191205: Permission denied 2019-12-05T05:00:05.537 INFO:teuthology.orchestra.run.smithi042.stderr: 5 Dec 05:00:05 ntpd[7580]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2019-12-05T05:00:05.537 INFO:teuthology.orchestra.run.smithi042.stderr: 5 Dec 05:00:05 ntpd[7580]: can't open /var/log/ntpstats/loopstats.20191205: Permission denied 2019-12-05T05:00:05.538 INFO:teuthology.orchestra.run.smithi042.stdout: 5 Dec 05:00:05 ntpd[7580]: ntpd: time set -0.313487 s 2019-12-05T05:00:05.538 INFO:teuthology.orchestra.run.smithi042.stdout:ntpd: time set -0.313487s 2019-12-05T05:00:05.602 INFO:teuthology.orchestra.run.smithi042.stdout: remote refid st t when poll reach delay offset jitter 2019-12-05T05:00:05.603 INFO:teuthology.orchestra.run.smithi042.stdout:============================================================================== 2019-12-05T05:00:05.603 INFO:teuthology.orchestra.run.smithi042.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2019-12-05T05:00:05.603 INFO:teuthology.orchestra.run.smithi042.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2019-12-05T05:00:05.603 INFO:teuthology.orchestra.run.smithi042.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2019-12-05T05:00:05.603 INFO:teuthology.orchestra.run.smithi042.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2019-12-05T05:00:05.604 INFO:teuthology.orchestra.run.smithi040:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2019-12-05T05:00:05.664 INFO:teuthology.orchestra.run.smithi040.stdout: 5 Dec 05:00:05 ntpd[7753]: ntpd 4.2.8p10@1.3728-o (1): Starting 2019-12-05T05:00:05.665 INFO:teuthology.orchestra.run.smithi040.stdout: 5 Dec 05:00:05 ntpd[7753]: Command line: ntpd -gq 2019-12-05T05:00:05.665 INFO:teuthology.orchestra.run.smithi040.stdout: 5 Dec 05:00:05 ntpd[7753]: proto: precision = 0.126 usec (-23) 2019-12-05T05:00:05.667 INFO:teuthology.orchestra.run.smithi040.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2019-12-05T05:00:05.667 INFO:teuthology.orchestra.run.smithi040.stderr:restrict ::: KOD does nothing without LIMITED. 2019-12-05T05:00:05.668 INFO:teuthology.orchestra.run.smithi040.stdout: 5 Dec 05:00:05 ntpd[7753]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2019-12-05T05:00:05.668 INFO:teuthology.orchestra.run.smithi040.stdout: 5 Dec 05:00:05 ntpd[7753]: restrict ::: KOD does nothing without LIMITED. 2019-12-05T05:00:05.668 INFO:teuthology.orchestra.run.smithi040.stdout: 5 Dec 05:00:05 ntpd[7753]: Listen and drop on 0 v6wildcard [::]:123 2019-12-05T05:00:05.668 INFO:teuthology.orchestra.run.smithi040.stdout: 5 Dec 05:00:05 ntpd[7753]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2019-12-05T05:00:05.668 INFO:teuthology.orchestra.run.smithi040.stdout: 5 Dec 05:00:05 ntpd[7753]: Listen normally on 2 lo 127.0.0.1:123 2019-12-05T05:00:05.669 INFO:teuthology.orchestra.run.smithi040.stdout: 5 Dec 05:00:05 ntpd[7753]: Listen normally on 3 ens1f0 172.21.15.40:123 2019-12-05T05:00:05.669 INFO:teuthology.orchestra.run.smithi040.stdout: 5 Dec 05:00:05 ntpd[7753]: Listen normally on 4 lo [::1]:123 2019-12-05T05:00:05.669 INFO:teuthology.orchestra.run.smithi040.stdout: 5 Dec 05:00:05 ntpd[7753]: Listen normally on 5 ens1f0 [fe80::ec4:7aff:febd:12cc%3]:123 2019-12-05T05:00:05.669 INFO:teuthology.orchestra.run.smithi040.stdout: 5 Dec 05:00:05 ntpd[7753]: Listening on routing socket on fd #22 for interface updates 2019-12-05T05:00:07.665 INFO:teuthology.orchestra.run.smithi040.stderr: 5 Dec 05:00:07 ntpd[7753]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2019-12-05T05:00:07.665 INFO:teuthology.orchestra.run.smithi040.stderr: 5 Dec 05:00:07 ntpd[7753]: can't open /var/log/ntpstats/rawstats.20191205: Permission denied 2019-12-05T05:00:07.666 INFO:teuthology.orchestra.run.smithi040.stderr: 5 Dec 05:00:07 ntpd[7753]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2019-12-05T05:00:07.666 INFO:teuthology.orchestra.run.smithi040.stderr: 5 Dec 05:00:07 ntpd[7753]: can't open /var/log/ntpstats/peerstats.20191205: Permission denied 2019-12-05T05:00:09.663 INFO:teuthology.orchestra.run.smithi040.stderr: 5 Dec 05:00:09 ntpd[7753]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2019-12-05T05:00:09.664 INFO:teuthology.orchestra.run.smithi040.stderr: 5 Dec 05:00:09 ntpd[7753]: can't open /var/log/ntpstats/rawstats.20191205: Permission denied 2019-12-05T05:00:09.664 INFO:teuthology.orchestra.run.smithi040.stderr: 5 Dec 05:00:09 ntpd[7753]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2019-12-05T05:00:09.664 INFO:teuthology.orchestra.run.smithi040.stderr: 5 Dec 05:00:09 ntpd[7753]: can't open /var/log/ntpstats/peerstats.20191205: Permission denied 2019-12-05T05:00:11.664 INFO:teuthology.orchestra.run.smithi040.stderr: 5 Dec 05:00:11 ntpd[7753]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2019-12-05T05:00:11.665 INFO:teuthology.orchestra.run.smithi040.stderr: 5 Dec 05:00:11 ntpd[7753]: can't open /var/log/ntpstats/rawstats.20191205: Permission denied 2019-12-05T05:00:11.665 INFO:teuthology.orchestra.run.smithi040.stderr: 5 Dec 05:00:11 ntpd[7753]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2019-12-05T05:00:11.665 INFO:teuthology.orchestra.run.smithi040.stderr: 5 Dec 05:00:11 ntpd[7753]: can't open /var/log/ntpstats/peerstats.20191205: Permission denied 2019-12-05T05:00:13.661 INFO:teuthology.orchestra.run.smithi040.stderr: 5 Dec 05:00:13 ntpd[7753]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2019-12-05T05:00:13.661 INFO:teuthology.orchestra.run.smithi040.stderr: 5 Dec 05:00:13 ntpd[7753]: can't open /var/log/ntpstats/rawstats.20191205: Permission denied 2019-12-05T05:00:13.662 INFO:teuthology.orchestra.run.smithi040.stderr: 5 Dec 05:00:13 ntpd[7753]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2019-12-05T05:00:13.662 INFO:teuthology.orchestra.run.smithi040.stderr: 5 Dec 05:00:13 ntpd[7753]: can't open /var/log/ntpstats/peerstats.20191205: Permission denied 2019-12-05T05:00:13.662 INFO:teuthology.orchestra.run.smithi040.stderr: 5 Dec 05:00:13 ntpd[7753]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2019-12-05T05:00:13.662 INFO:teuthology.orchestra.run.smithi040.stderr: 5 Dec 05:00:13 ntpd[7753]: can't open /var/log/ntpstats/loopstats.20191205: Permission denied 2019-12-05T05:00:13.662 INFO:teuthology.orchestra.run.smithi040.stdout: 5 Dec 05:00:13 ntpd[7753]: ntpd: time set +0.293690 s 2019-12-05T05:00:13.662 INFO:teuthology.orchestra.run.smithi040.stdout:ntpd: time set +0.293690s 2019-12-05T05:00:13.710 INFO:teuthology.orchestra.run.smithi040.stdout: remote refid st t when poll reach delay offset jitter 2019-12-05T05:00:13.710 INFO:teuthology.orchestra.run.smithi040.stdout:============================================================================== 2019-12-05T05:00:13.711 INFO:teuthology.orchestra.run.smithi040.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2019-12-05T05:00:13.711 INFO:teuthology.orchestra.run.smithi040.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2019-12-05T05:00:13.711 INFO:teuthology.orchestra.run.smithi040.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2019-12-05T05:00:13.711 INFO:teuthology.orchestra.run.smithi040.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2019-12-05T05:00:13.711 INFO:teuthology.run_tasks:Running task install... 2019-12-05T05:00:37.033 DEBUG:teuthology.task.install:project ceph 2019-12-05T05:00:37.033 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'sha1': 'c5290805033483ba0cec165c06bde07dae5f19ec'}} 2019-12-05T05:00:37.033 DEBUG:teuthology.task.install:config {'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel']}, 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell'], 'rpm': ['python3-cephfs']}, 'sha1': 'c5290805033483ba0cec165c06bde07dae5f19ec'} 2019-12-05T05:00:37.034 INFO:teuthology.task.install:Using flavor: basic 2019-12-05T05:00:37.054 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'ceph-daemon', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python-ceph', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-daemon', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-cloud', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-ssh', 'ceph-fuse', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python-ceph', 'rbd-fuse']} 2019-12-05T05:00:37.055 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell'], 'rpm': ['python3-cephfs']} 2019-12-05T05:00:37.055 INFO:teuthology.orchestra.run.smithi042:> sudo lsb_release -is 2019-12-05T05:00:37.115 INFO:teuthology.orchestra.run.smithi042.stdout:Ubuntu 2019-12-05T05:00:37.121 DEBUG:teuthology.misc:System to be installed: Ubuntu 2019-12-05T05:00:37.122 INFO:teuthology.orchestra.run.smithi040:> sudo lsb_release -is 2019-12-05T05:00:37.123 INFO:teuthology.orchestra.run.smithi042:> sudo apt-key list | grep Ceph 2019-12-05T05:00:37.177 INFO:teuthology.orchestra.run.smithi040.stdout:Ubuntu 2019-12-05T05:00:37.178 DEBUG:teuthology.misc:System to be installed: Ubuntu 2019-12-05T05:00:37.178 INFO:teuthology.orchestra.run.smithi040:> sudo apt-key list | grep Ceph 2019-12-05T05:00:37.224 INFO:teuthology.orchestra.run.smithi042.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2019-12-05T05:00:37.283 INFO:teuthology.orchestra.run.smithi042.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2019-12-05T05:00:37.284 INFO:teuthology.orchestra.run.smithi042.stdout:uid [ unknown] Ceph.com (release key) 2019-12-05T05:00:37.285 INFO:teuthology.orchestra.run.smithi042:> sudo lsb_release -is 2019-12-05T05:00:37.294 INFO:teuthology.orchestra.run.smithi040.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2019-12-05T05:00:37.337 INFO:teuthology.orchestra.run.smithi042.stdout:Ubuntu 2019-12-05T05:00:37.342 DEBUG:teuthology.misc:System to be installed: Ubuntu 2019-12-05T05:00:37.342 INFO:teuthology.task.install.deb:Installing packages: ceph, ceph-daemon, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, radosgw, python-ceph, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, python3-cephfs, cephfs-shell on remote deb x86_64 2019-12-05T05:00:37.342 INFO:teuthology.task.install.deb:Installing system (non-project) packages: bison, flex, libelf-dev, libssl-dev on remote deb x86_64 2019-12-05T05:00:37.343 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1=c5290805033483ba0cec165c06bde07dae5f19ec 2019-12-05T05:00:37.353 INFO:teuthology.orchestra.run.smithi040.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2019-12-05T05:00:37.353 INFO:teuthology.orchestra.run.smithi040.stdout:uid [ unknown] Ceph.com (release key) 2019-12-05T05:00:37.354 INFO:teuthology.orchestra.run.smithi040:> sudo lsb_release -is 2019-12-05T05:00:37.405 INFO:teuthology.orchestra.run.smithi040.stdout:Ubuntu 2019-12-05T05:00:37.410 DEBUG:teuthology.misc:System to be installed: Ubuntu 2019-12-05T05:00:37.411 INFO:teuthology.task.install.deb:Installing packages: ceph, ceph-daemon, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, radosgw, python-ceph, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, python3-cephfs, cephfs-shell on remote deb x86_64 2019-12-05T05:00:37.411 INFO:teuthology.task.install.deb:Installing system (non-project) packages: bison, flex, libelf-dev, libssl-dev on remote deb x86_64 2019-12-05T05:00:37.411 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1=c5290805033483ba0cec165c06bde07dae5f19ec 2019-12-05T05:00:37.523 INFO:teuthology.task.install.deb:Pulling from https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191204.231200/c5290805033483ba0cec165c06bde07dae5f19ec/ubuntu/bionic/flavors/default/ 2019-12-05T05:00:37.523 INFO:teuthology.task.install.deb:Package version is 15.0.0-8014-gc529080-1bionic 2019-12-05T05:00:37.567 INFO:teuthology.task.install.deb:Pulling from https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191204.231200/c5290805033483ba0cec165c06bde07dae5f19ec/ubuntu/bionic/flavors/default/ 2019-12-05T05:00:37.567 INFO:teuthology.task.install.deb:Package version is 15.0.0-8014-gc529080-1bionic 2019-12-05T05:00:37.673 INFO:teuthology.orchestra.run.smithi042:> sudo sh -c 'cat > /etc/apt/sources.list.d/ceph.list' 2019-12-05T05:00:37.715 INFO:teuthology.orchestra.run.smithi040:> sudo sh -c 'cat > /etc/apt/sources.list.d/ceph.list' 2019-12-05T05:00:37.727 INFO:teuthology.orchestra.run.smithi042:> sudo apt-get update 2019-12-05T05:00:37.776 INFO:teuthology.orchestra.run.smithi040:> sudo apt-get update 2019-12-05T05:00:38.033 INFO:teuthology.orchestra.run.smithi042.stdout:Ign:1 https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191204.231200/c5290805033483ba0cec165c06bde07dae5f19ec/ubuntu/bionic/flavors/default bionic InRelease 2019-12-05T05:00:38.060 INFO:teuthology.orchestra.run.smithi042.stdout:Get:2 https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191204.231200/c5290805033483ba0cec165c06bde07dae5f19ec/ubuntu/bionic/flavors/default bionic Release [7,672 B] 2019-12-05T05:00:38.061 INFO:teuthology.orchestra.run.smithi040.stdout:Ign:1 https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191204.231200/c5290805033483ba0cec165c06bde07dae5f19ec/ubuntu/bionic/flavors/default bionic InRelease 2019-12-05T05:00:38.076 INFO:teuthology.orchestra.run.smithi042.stdout:Hit:3 http://security.ubuntu.com/ubuntu bionic-security InRelease 2019-12-05T05:00:38.076 INFO:teuthology.orchestra.run.smithi042.stdout:Hit:4 http://archive.ubuntu.com/ubuntu bionic InRelease 2019-12-05T05:00:38.079 INFO:teuthology.orchestra.run.smithi042.stdout:Ign:5 https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191204.231200/c5290805033483ba0cec165c06bde07dae5f19ec/ubuntu/bionic/flavors/default bionic Release.gpg 2019-12-05T05:00:38.082 INFO:teuthology.orchestra.run.smithi040.stdout:Get:2 https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191204.231200/c5290805033483ba0cec165c06bde07dae5f19ec/ubuntu/bionic/flavors/default bionic Release [7,672 B] 2019-12-05T05:00:38.099 INFO:teuthology.orchestra.run.smithi042.stdout:Get:6 https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191204.231200/c5290805033483ba0cec165c06bde07dae5f19ec/ubuntu/bionic/flavors/default bionic/main i386 Packages [2,865 B] 2019-12-05T05:00:38.122 INFO:teuthology.orchestra.run.smithi040.stdout:Hit:3 http://security.ubuntu.com/ubuntu bionic-security InRelease 2019-12-05T05:00:38.122 INFO:teuthology.orchestra.run.smithi040.stdout:Hit:4 http://archive.ubuntu.com/ubuntu bionic InRelease 2019-12-05T05:00:38.122 INFO:teuthology.orchestra.run.smithi040.stdout:Ign:5 https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191204.231200/c5290805033483ba0cec165c06bde07dae5f19ec/ubuntu/bionic/flavors/default bionic Release.gpg 2019-12-05T05:00:38.150 INFO:teuthology.orchestra.run.smithi042.stdout:Get:7 https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191204.231200/c5290805033483ba0cec165c06bde07dae5f19ec/ubuntu/bionic/flavors/default bionic/main amd64 Packages [16.4 kB] 2019-12-05T05:00:38.151 INFO:teuthology.orchestra.run.smithi040.stdout:Get:6 https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191204.231200/c5290805033483ba0cec165c06bde07dae5f19ec/ubuntu/bionic/flavors/default bionic/main amd64 Packages [16.4 kB] 2019-12-05T05:00:38.151 INFO:teuthology.orchestra.run.smithi042.stdout:Hit:8 http://archive.ubuntu.com/ubuntu bionic-updates InRelease 2019-12-05T05:00:38.169 INFO:teuthology.orchestra.run.smithi040.stdout:Get:7 https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191204.231200/c5290805033483ba0cec165c06bde07dae5f19ec/ubuntu/bionic/flavors/default bionic/main i386 Packages [2,865 B] 2019-12-05T05:00:38.179 INFO:teuthology.orchestra.run.smithi040.stdout:Hit:8 http://archive.ubuntu.com/ubuntu bionic-updates InRelease 2019-12-05T05:00:38.226 INFO:teuthology.orchestra.run.smithi042.stdout:Hit:9 http://archive.ubuntu.com/ubuntu bionic-backports InRelease 2019-12-05T05:00:38.272 INFO:teuthology.orchestra.run.smithi040.stdout:Hit:9 http://archive.ubuntu.com/ubuntu bionic-backports InRelease 2019-12-05T05:00:38.675 INFO:teuthology.orchestra.run.smithi042.stdout:Fetched 26.9 kB in 1s (38.1 kB/s) 2019-12-05T05:00:38.734 INFO:teuthology.orchestra.run.smithi040.stdout:Fetched 26.9 kB in 1s (36.9 kB/s) 2019-12-05T05:00:39.860 INFO:teuthology.orchestra.run.smithi042.stdout:Reading package lists... 2019-12-05T05:00:39.909 INFO:teuthology.orchestra.run.smithi042:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=15.0.0-8014-gc529080-1bionic ceph-daemon=15.0.0-8014-gc529080-1bionic ceph-mds=15.0.0-8014-gc529080-1bionic ceph-mgr=15.0.0-8014-gc529080-1bionic ceph-common=15.0.0-8014-gc529080-1bionic ceph-fuse=15.0.0-8014-gc529080-1bionic ceph-test=15.0.0-8014-gc529080-1bionic radosgw=15.0.0-8014-gc529080-1bionic python-ceph=15.0.0-8014-gc529080-1bionic libcephfs2=15.0.0-8014-gc529080-1bionic libcephfs-dev=15.0.0-8014-gc529080-1bionic librados2=15.0.0-8014-gc529080-1bionic librbd1=15.0.0-8014-gc529080-1bionic rbd-fuse=15.0.0-8014-gc529080-1bionic python3-cephfs=15.0.0-8014-gc529080-1bionic cephfs-shell=15.0.0-8014-gc529080-1bionic 2019-12-05T05:00:39.926 INFO:teuthology.orchestra.run.smithi040.stdout:Reading package lists... 2019-12-05T05:00:39.974 INFO:teuthology.orchestra.run.smithi040:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=15.0.0-8014-gc529080-1bionic ceph-daemon=15.0.0-8014-gc529080-1bionic ceph-mds=15.0.0-8014-gc529080-1bionic ceph-mgr=15.0.0-8014-gc529080-1bionic ceph-common=15.0.0-8014-gc529080-1bionic ceph-fuse=15.0.0-8014-gc529080-1bionic ceph-test=15.0.0-8014-gc529080-1bionic radosgw=15.0.0-8014-gc529080-1bionic python-ceph=15.0.0-8014-gc529080-1bionic libcephfs2=15.0.0-8014-gc529080-1bionic libcephfs-dev=15.0.0-8014-gc529080-1bionic librados2=15.0.0-8014-gc529080-1bionic librbd1=15.0.0-8014-gc529080-1bionic rbd-fuse=15.0.0-8014-gc529080-1bionic python3-cephfs=15.0.0-8014-gc529080-1bionic cephfs-shell=15.0.0-8014-gc529080-1bionic 2019-12-05T05:00:39.989 INFO:teuthology.orchestra.run.smithi042.stdout:Reading package lists... 2019-12-05T05:00:40.060 INFO:teuthology.orchestra.run.smithi040.stdout:Reading package lists... 2019-12-05T05:00:40.245 INFO:teuthology.orchestra.run.smithi042.stdout:Building dependency tree... 2019-12-05T05:00:40.245 INFO:teuthology.orchestra.run.smithi042.stdout:Reading state information... 2019-12-05T05:00:40.309 INFO:teuthology.orchestra.run.smithi040.stdout:Building dependency tree... 2019-12-05T05:00:40.310 INFO:teuthology.orchestra.run.smithi040.stdout:Reading state information... 2019-12-05T05:00:40.415 INFO:teuthology.orchestra.run.smithi042.stdout:The following additional packages will be installed: 2019-12-05T05:00:40.416 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-base ceph-mgr-dashboard ceph-mgr-diskprediction-cloud 2019-12-05T05:00:40.416 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-mgr-diskprediction-local ceph-mgr-k8sevents ceph-mgr-rook ceph-mgr-ssh 2019-12-05T05:00:40.419 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-mon ceph-osd formencode-i18n jq libjq1 libjs-sphinxdoc libjs-underscore 2019-12-05T05:00:40.420 INFO:teuthology.orchestra.run.smithi042.stdout: libleveldb1v5 liblttng-ust0 liboath0 libonig4 librabbitmq4 libradosstriper1 2019-12-05T05:00:40.420 INFO:teuthology.orchestra.run.smithi042.stdout: librdkafka1 librgw2 libsnappy1v5 libxslt1.1 python-attr python-bcrypt 2019-12-05T05:00:40.420 INFO:teuthology.orchestra.run.smithi042.stdout: python-bs4 python-ceph-argparse python-ceph-common python-cephfs 2019-12-05T05:00:40.420 INFO:teuthology.orchestra.run.smithi042.stdout: python-certifi python-chardet python-cherrypy3 python-decorator 2019-12-05T05:00:40.421 INFO:teuthology.orchestra.run.smithi042.stdout: python-dnspython python-fasteners python-formencode python-funcsigs 2019-12-05T05:00:40.421 INFO:teuthology.orchestra.run.smithi042.stdout: python-html5lib python-httplib2 python-joblib python-jwt python-kubernetes 2019-12-05T05:00:40.421 INFO:teuthology.orchestra.run.smithi042.stdout: python-logutils python-lxml python-mako python-markupsafe python-monotonic 2019-12-05T05:00:40.422 INFO:teuthology.orchestra.run.smithi042.stdout: python-oauth2client python-openid python-openssl python-paste 2019-12-05T05:00:40.423 INFO:teuthology.orchestra.run.smithi042.stdout: python-pastedeploy python-pastedeploy-tpl python-pastescript python-pecan 2019-12-05T05:00:40.423 INFO:teuthology.orchestra.run.smithi042.stdout: python-pluggy python-prettytable python-py python-pyasn1 2019-12-05T05:00:40.423 INFO:teuthology.orchestra.run.smithi042.stdout: python-pyasn1-modules python-pyinotify python-pytest python-rados python-rbd 2019-12-05T05:00:40.423 INFO:teuthology.orchestra.run.smithi042.stdout: python-repoze.lru python-requests python-rgw python-routes python-rsa 2019-12-05T05:00:40.424 INFO:teuthology.orchestra.run.smithi042.stdout: python-scgi python-scipy python-simplegeneric python-simplejson 2019-12-05T05:00:40.424 INFO:teuthology.orchestra.run.smithi042.stdout: python-singledispatch python-sklearn python-sklearn-lib python-sqlalchemy 2019-12-05T05:00:40.424 INFO:teuthology.orchestra.run.smithi042.stdout: python-sqlalchemy-ext python-tempita python-uritemplate python-urllib3 2019-12-05T05:00:40.424 INFO:teuthology.orchestra.run.smithi042.stdout: python-waitress python-webencodings python-webob python-websocket 2019-12-05T05:00:40.425 INFO:teuthology.orchestra.run.smithi042.stdout: python-webtest python-werkzeug python-yaml python3-ceph-argparse 2019-12-05T05:00:40.425 INFO:teuthology.orchestra.run.smithi042.stdout: python3-cmd2 python3-colorama python3-pyparsing python3-pyperclip 2019-12-05T05:00:40.425 INFO:teuthology.orchestra.run.smithi042.stdout: python3-rados socat xmlstarlet 2019-12-05T05:00:40.425 INFO:teuthology.orchestra.run.smithi042.stdout:Suggested packages: 2019-12-05T05:00:40.426 INFO:teuthology.orchestra.run.smithi042.stdout: python-influxdb python-attr-doc python-egenix-mxdatetime python-funcsigs-doc 2019-12-05T05:00:40.426 INFO:teuthology.orchestra.run.smithi042.stdout: python-genshi python-lxml-dbg python-lxml-doc python-beaker python-mako-doc 2019-12-05T05:00:40.426 INFO:teuthology.orchestra.run.smithi042.stdout: python-openssl-doc python-openssl-dbg httpd-wsgi libapache2-mod-python 2019-12-05T05:00:40.427 INFO:teuthology.orchestra.run.smithi042.stdout: libapache2-mod-scgi libjs-mochikit python-pastewebkit python-cheetah 2019-12-05T05:00:40.427 INFO:teuthology.orchestra.run.smithi042.stdout: python-cherrypy python-flup python-pecan-doc subversion python-pytest-xdist 2019-12-05T05:00:40.428 INFO:teuthology.orchestra.run.smithi042.stdout: python-pyinotify-doc python-mock python-socks python-scipy-doc python-dap 2019-12-05T05:00:40.428 INFO:teuthology.orchestra.run.smithi042.stdout: python-scikits-optimization python-sklearn-doc ipython python-sqlalchemy-doc 2019-12-05T05:00:40.429 INFO:teuthology.orchestra.run.smithi042.stdout: python-psycopg2 python-pymysql python-fdb python-pymssql python-ntlm 2019-12-05T05:00:40.429 INFO:teuthology.orchestra.run.smithi042.stdout: python-waitress-doc python-webob-doc python-webtest-doc python-pyquery 2019-12-05T05:00:40.429 INFO:teuthology.orchestra.run.smithi042.stdout: python-redis python-pylibmc | python-memcache python-termcolor 2019-12-05T05:00:40.429 INFO:teuthology.orchestra.run.smithi042.stdout: python-watchdog python-werkzeug-doc python-pyparsing-doc 2019-12-05T05:00:40.494 INFO:teuthology.orchestra.run.smithi040.stdout:The following additional packages will be installed: 2019-12-05T05:00:40.495 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-base ceph-mgr-dashboard ceph-mgr-diskprediction-cloud 2019-12-05T05:00:40.495 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-diskprediction-local ceph-mgr-k8sevents ceph-mgr-rook ceph-mgr-ssh 2019-12-05T05:00:40.495 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mon ceph-osd formencode-i18n jq libjq1 libjs-sphinxdoc libjs-underscore 2019-12-05T05:00:40.495 INFO:teuthology.orchestra.run.smithi040.stdout: libleveldb1v5 liblttng-ust0 liboath0 libonig4 librabbitmq4 libradosstriper1 2019-12-05T05:00:40.496 INFO:teuthology.orchestra.run.smithi040.stdout: librdkafka1 librgw2 libsnappy1v5 libxslt1.1 python-attr python-bcrypt 2019-12-05T05:00:40.496 INFO:teuthology.orchestra.run.smithi040.stdout: python-bs4 python-ceph-argparse python-ceph-common python-cephfs 2019-12-05T05:00:40.496 INFO:teuthology.orchestra.run.smithi040.stdout: python-certifi python-chardet python-cherrypy3 python-decorator 2019-12-05T05:00:40.498 INFO:teuthology.orchestra.run.smithi040.stdout: python-dnspython python-fasteners python-formencode python-funcsigs 2019-12-05T05:00:40.498 INFO:teuthology.orchestra.run.smithi040.stdout: python-html5lib python-httplib2 python-joblib python-jwt python-kubernetes 2019-12-05T05:00:40.499 INFO:teuthology.orchestra.run.smithi040.stdout: python-logutils python-lxml python-mako python-markupsafe python-monotonic 2019-12-05T05:00:40.499 INFO:teuthology.orchestra.run.smithi040.stdout: python-oauth2client python-openid python-openssl python-paste 2019-12-05T05:00:40.505 INFO:teuthology.orchestra.run.smithi040.stdout: python-pastedeploy python-pastedeploy-tpl python-pastescript python-pecan 2019-12-05T05:00:40.506 INFO:teuthology.orchestra.run.smithi040.stdout: python-pluggy python-prettytable python-py python-pyasn1 2019-12-05T05:00:40.506 INFO:teuthology.orchestra.run.smithi040.stdout: python-pyasn1-modules python-pyinotify python-pytest python-rados python-rbd 2019-12-05T05:00:40.507 INFO:teuthology.orchestra.run.smithi040.stdout: python-repoze.lru python-requests python-rgw python-routes python-rsa 2019-12-05T05:00:40.507 INFO:teuthology.orchestra.run.smithi040.stdout: python-scgi python-scipy python-simplegeneric python-simplejson 2019-12-05T05:00:40.507 INFO:teuthology.orchestra.run.smithi040.stdout: python-singledispatch python-sklearn python-sklearn-lib python-sqlalchemy 2019-12-05T05:00:40.507 INFO:teuthology.orchestra.run.smithi040.stdout: python-sqlalchemy-ext python-tempita python-uritemplate python-urllib3 2019-12-05T05:00:40.508 INFO:teuthology.orchestra.run.smithi040.stdout: python-waitress python-webencodings python-webob python-websocket 2019-12-05T05:00:40.508 INFO:teuthology.orchestra.run.smithi040.stdout: python-webtest python-werkzeug python-yaml python3-ceph-argparse 2019-12-05T05:00:40.508 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cmd2 python3-colorama python3-pyparsing python3-pyperclip 2019-12-05T05:00:40.508 INFO:teuthology.orchestra.run.smithi040.stdout: python3-rados socat xmlstarlet 2019-12-05T05:00:40.509 INFO:teuthology.orchestra.run.smithi040.stdout:Suggested packages: 2019-12-05T05:00:40.509 INFO:teuthology.orchestra.run.smithi040.stdout: python-influxdb python-attr-doc python-egenix-mxdatetime python-funcsigs-doc 2019-12-05T05:00:40.509 INFO:teuthology.orchestra.run.smithi040.stdout: python-genshi python-lxml-dbg python-lxml-doc python-beaker python-mako-doc 2019-12-05T05:00:40.509 INFO:teuthology.orchestra.run.smithi040.stdout: python-openssl-doc python-openssl-dbg httpd-wsgi libapache2-mod-python 2019-12-05T05:00:40.510 INFO:teuthology.orchestra.run.smithi040.stdout: libapache2-mod-scgi libjs-mochikit python-pastewebkit python-cheetah 2019-12-05T05:00:40.510 INFO:teuthology.orchestra.run.smithi040.stdout: python-cherrypy python-flup python-pecan-doc subversion python-pytest-xdist 2019-12-05T05:00:40.510 INFO:teuthology.orchestra.run.smithi040.stdout: python-pyinotify-doc python-mock python-socks python-scipy-doc python-dap 2019-12-05T05:00:40.510 INFO:teuthology.orchestra.run.smithi040.stdout: python-scikits-optimization python-sklearn-doc ipython python-sqlalchemy-doc 2019-12-05T05:00:40.510 INFO:teuthology.orchestra.run.smithi040.stdout: python-psycopg2 python-pymysql python-fdb python-pymssql python-ntlm 2019-12-05T05:00:40.511 INFO:teuthology.orchestra.run.smithi040.stdout: python-waitress-doc python-webob-doc python-webtest-doc python-pyquery 2019-12-05T05:00:40.511 INFO:teuthology.orchestra.run.smithi040.stdout: python-redis python-pylibmc | python-memcache python-termcolor 2019-12-05T05:00:40.511 INFO:teuthology.orchestra.run.smithi040.stdout: python-watchdog python-werkzeug-doc python-pyparsing-doc 2019-12-05T05:00:40.523 INFO:teuthology.orchestra.run.smithi042.stdout:The following NEW packages will be installed: 2019-12-05T05:00:40.523 INFO:teuthology.orchestra.run.smithi042.stdout: ceph ceph-base ceph-common ceph-daemon ceph-fuse ceph-mds ceph-mgr 2019-12-05T05:00:40.523 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-cloud 2019-12-05T05:00:40.523 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-mgr-diskprediction-local ceph-mgr-k8sevents ceph-mgr-rook ceph-mgr-ssh 2019-12-05T05:00:40.525 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-mon ceph-osd ceph-test cephfs-shell formencode-i18n jq libcephfs-dev 2019-12-05T05:00:40.525 INFO:teuthology.orchestra.run.smithi042.stdout: libcephfs2 libjq1 libjs-sphinxdoc libjs-underscore libleveldb1v5 2019-12-05T05:00:40.526 INFO:teuthology.orchestra.run.smithi042.stdout: liblttng-ust0 liboath0 libonig4 librabbitmq4 libradosstriper1 librdkafka1 2019-12-05T05:00:40.526 INFO:teuthology.orchestra.run.smithi042.stdout: librgw2 libsnappy1v5 libxslt1.1 python-attr python-bcrypt python-bs4 2019-12-05T05:00:40.526 INFO:teuthology.orchestra.run.smithi042.stdout: python-ceph python-ceph-argparse python-ceph-common python-cephfs 2019-12-05T05:00:40.526 INFO:teuthology.orchestra.run.smithi042.stdout: python-certifi python-chardet python-cherrypy3 python-decorator 2019-12-05T05:00:40.526 INFO:teuthology.orchestra.run.smithi042.stdout: python-dnspython python-fasteners python-formencode python-funcsigs 2019-12-05T05:00:40.526 INFO:teuthology.orchestra.run.smithi042.stdout: python-html5lib python-httplib2 python-joblib python-jwt python-kubernetes 2019-12-05T05:00:40.529 INFO:teuthology.orchestra.run.smithi042.stdout: python-logutils python-lxml python-mako python-markupsafe python-monotonic 2019-12-05T05:00:40.529 INFO:teuthology.orchestra.run.smithi042.stdout: python-oauth2client python-openid python-openssl python-paste 2019-12-05T05:00:40.529 INFO:teuthology.orchestra.run.smithi042.stdout: python-pastedeploy python-pastedeploy-tpl python-pastescript python-pecan 2019-12-05T05:00:40.529 INFO:teuthology.orchestra.run.smithi042.stdout: python-pluggy python-prettytable python-py python-pyasn1 2019-12-05T05:00:40.529 INFO:teuthology.orchestra.run.smithi042.stdout: python-pyasn1-modules python-pyinotify python-pytest python-rados python-rbd 2019-12-05T05:00:40.529 INFO:teuthology.orchestra.run.smithi042.stdout: python-repoze.lru python-requests python-rgw python-routes python-rsa 2019-12-05T05:00:40.530 INFO:teuthology.orchestra.run.smithi042.stdout: python-scgi python-scipy python-simplegeneric python-simplejson 2019-12-05T05:00:40.530 INFO:teuthology.orchestra.run.smithi042.stdout: python-singledispatch python-sklearn python-sklearn-lib python-sqlalchemy 2019-12-05T05:00:40.530 INFO:teuthology.orchestra.run.smithi042.stdout: python-sqlalchemy-ext python-tempita python-uritemplate python-urllib3 2019-12-05T05:00:40.531 INFO:teuthology.orchestra.run.smithi042.stdout: python-waitress python-webencodings python-webob python-websocket 2019-12-05T05:00:40.531 INFO:teuthology.orchestra.run.smithi042.stdout: python-webtest python-werkzeug python-yaml python3-ceph-argparse 2019-12-05T05:00:40.531 INFO:teuthology.orchestra.run.smithi042.stdout: python3-cephfs python3-cmd2 python3-colorama python3-pyparsing 2019-12-05T05:00:40.531 INFO:teuthology.orchestra.run.smithi042.stdout: python3-pyperclip python3-rados radosgw rbd-fuse socat xmlstarlet 2019-12-05T05:00:40.532 INFO:teuthology.orchestra.run.smithi042.stdout:The following packages will be upgraded: 2019-12-05T05:00:40.532 INFO:teuthology.orchestra.run.smithi042.stdout: librados2 librbd1 2019-12-05T05:00:40.600 INFO:teuthology.orchestra.run.smithi040.stdout:The following NEW packages will be installed: 2019-12-05T05:00:40.600 INFO:teuthology.orchestra.run.smithi040.stdout: ceph ceph-base ceph-common ceph-daemon ceph-fuse ceph-mds ceph-mgr 2019-12-05T05:00:40.601 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-cloud 2019-12-05T05:00:40.601 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-diskprediction-local ceph-mgr-k8sevents ceph-mgr-rook ceph-mgr-ssh 2019-12-05T05:00:40.601 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mon ceph-osd ceph-test cephfs-shell formencode-i18n jq libcephfs-dev 2019-12-05T05:00:40.601 INFO:teuthology.orchestra.run.smithi040.stdout: libcephfs2 libjq1 libjs-sphinxdoc libjs-underscore libleveldb1v5 2019-12-05T05:00:40.601 INFO:teuthology.orchestra.run.smithi040.stdout: liblttng-ust0 liboath0 libonig4 librabbitmq4 libradosstriper1 librdkafka1 2019-12-05T05:00:40.601 INFO:teuthology.orchestra.run.smithi040.stdout: librgw2 libsnappy1v5 libxslt1.1 python-attr python-bcrypt python-bs4 2019-12-05T05:00:40.602 INFO:teuthology.orchestra.run.smithi040.stdout: python-ceph python-ceph-argparse python-ceph-common python-cephfs 2019-12-05T05:00:40.602 INFO:teuthology.orchestra.run.smithi040.stdout: python-certifi python-chardet python-cherrypy3 python-decorator 2019-12-05T05:00:40.602 INFO:teuthology.orchestra.run.smithi040.stdout: python-dnspython python-fasteners python-formencode python-funcsigs 2019-12-05T05:00:40.602 INFO:teuthology.orchestra.run.smithi040.stdout: python-html5lib python-httplib2 python-joblib python-jwt python-kubernetes 2019-12-05T05:00:40.602 INFO:teuthology.orchestra.run.smithi040.stdout: python-logutils python-lxml python-mako python-markupsafe python-monotonic 2019-12-05T05:00:40.603 INFO:teuthology.orchestra.run.smithi040.stdout: python-oauth2client python-openid python-openssl python-paste 2019-12-05T05:00:40.603 INFO:teuthology.orchestra.run.smithi040.stdout: python-pastedeploy python-pastedeploy-tpl python-pastescript python-pecan 2019-12-05T05:00:40.603 INFO:teuthology.orchestra.run.smithi040.stdout: python-pluggy python-prettytable python-py python-pyasn1 2019-12-05T05:00:40.603 INFO:teuthology.orchestra.run.smithi040.stdout: python-pyasn1-modules python-pyinotify python-pytest python-rados python-rbd 2019-12-05T05:00:40.603 INFO:teuthology.orchestra.run.smithi040.stdout: python-repoze.lru python-requests python-rgw python-routes python-rsa 2019-12-05T05:00:40.604 INFO:teuthology.orchestra.run.smithi040.stdout: python-scgi python-scipy python-simplegeneric python-simplejson 2019-12-05T05:00:40.604 INFO:teuthology.orchestra.run.smithi040.stdout: python-singledispatch python-sklearn python-sklearn-lib python-sqlalchemy 2019-12-05T05:00:40.604 INFO:teuthology.orchestra.run.smithi040.stdout: python-sqlalchemy-ext python-tempita python-uritemplate python-urllib3 2019-12-05T05:00:40.604 INFO:teuthology.orchestra.run.smithi040.stdout: python-waitress python-webencodings python-webob python-websocket 2019-12-05T05:00:40.605 INFO:teuthology.orchestra.run.smithi040.stdout: python-webtest python-werkzeug python-yaml python3-ceph-argparse 2019-12-05T05:00:40.605 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cephfs python3-cmd2 python3-colorama python3-pyparsing 2019-12-05T05:00:40.605 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pyperclip python3-rados radosgw rbd-fuse socat xmlstarlet 2019-12-05T05:00:40.605 INFO:teuthology.orchestra.run.smithi040.stdout:The following packages will be upgraded: 2019-12-05T05:00:40.605 INFO:teuthology.orchestra.run.smithi040.stdout: librados2 librbd1 2019-12-05T05:00:40.677 INFO:teuthology.orchestra.run.smithi042.stdout:2 upgraded, 111 newly installed, 0 to remove and 52 not upgraded. 2019-12-05T05:00:40.677 INFO:teuthology.orchestra.run.smithi042.stdout:Need to get 139 MB of archives. 2019-12-05T05:00:40.677 INFO:teuthology.orchestra.run.smithi042.stdout:After this operation, 658 MB of additional disk space will be used. 2019-12-05T05:00:40.677 INFO:teuthology.orchestra.run.smithi042.stdout:Get:1 https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191204.231200/c5290805033483ba0cec165c06bde07dae5f19ec/ubuntu/bionic/flavors/default bionic/main amd64 librbd1 amd64 15.0.0-8014-gc529080-1bionic [1,516 kB] 2019-12-05T05:00:40.717 INFO:teuthology.orchestra.run.smithi042.stdout:Get:2 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-uritemplate all 0.6-1ubuntu1 [5,042 B] 2019-12-05T05:00:40.749 INFO:teuthology.orchestra.run.smithi040.stdout:2 upgraded, 111 newly installed, 0 to remove and 52 not upgraded. 2019-12-05T05:00:40.749 INFO:teuthology.orchestra.run.smithi040.stdout:Need to get 139 MB of archives. 2019-12-05T05:00:40.749 INFO:teuthology.orchestra.run.smithi040.stdout:After this operation, 658 MB of additional disk space will be used. 2019-12-05T05:00:40.750 INFO:teuthology.orchestra.run.smithi040.stdout:Get:1 https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191204.231200/c5290805033483ba0cec165c06bde07dae5f19ec/ubuntu/bionic/flavors/default bionic/main amd64 librbd1 amd64 15.0.0-8014-gc529080-1bionic [1,516 kB] 2019-12-05T05:00:40.785 INFO:teuthology.orchestra.run.smithi040.stdout:Get:2 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-uritemplate all 0.6-1ubuntu1 [5,042 B] 2019-12-05T05:00:40.798 INFO:teuthology.orchestra.run.smithi042.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyparsing all 2.2.0+dfsg1-2 [52.2 kB] 2019-12-05T05:00:40.843 INFO:teuthology.orchestra.run.smithi042.stdout:Get:4 https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191204.231200/c5290805033483ba0cec165c06bde07dae5f19ec/ubuntu/bionic/flavors/default bionic/main amd64 librados2 amd64 15.0.0-8014-gc529080-1bionic [2,984 kB] 2019-12-05T05:00:40.872 INFO:teuthology.orchestra.run.smithi040.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyparsing all 2.2.0+dfsg1-2 [52.2 kB] 2019-12-05T05:00:40.924 INFO:teuthology.orchestra.run.smithi040.stdout:Get:4 https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191204.231200/c5290805033483ba0cec165c06bde07dae5f19ec/ubuntu/bionic/flavors/default bionic/main amd64 librados2 amd64 15.0.0-8014-gc529080-1bionic [2,984 kB] 2019-12-05T05:00:40.980 INFO:teuthology.orchestra.run.smithi042.stdout:Get:5 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyperclip all 1.6.0-1 [9,376 B] 2019-12-05T05:00:40.980 INFO:teuthology.orchestra.run.smithi042.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-cmd2 all 0.7.9-0ubuntu1 [34.2 kB] 2019-12-05T05:00:41.000 INFO:teuthology.orchestra.run.smithi042.stdout:Get:7 https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191204.231200/c5290805033483ba0cec165c06bde07dae5f19ec/ubuntu/bionic/flavors/default bionic/main amd64 libcephfs2 amd64 15.0.0-8014-gc529080-1bionic [427 kB] 2019-12-05T05:00:41.015 INFO:teuthology.orchestra.run.smithi040.stdout:Get:5 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyperclip all 1.6.0-1 [9,376 B] 2019-12-05T05:00:41.016 INFO:teuthology.orchestra.run.smithi042.stdout:Get:8 http://archive.ubuntu.com/ubuntu bionic/universe amd64 liblttng-ust0 amd64 2.10.1-1 [154 kB] 2019-12-05T05:00:41.028 INFO:teuthology.orchestra.run.smithi042.stdout:Get:9 https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191204.231200/c5290805033483ba0cec165c06bde07dae5f19ec/ubuntu/bionic/flavors/default bionic/main amd64 python-rados amd64 15.0.0-8014-gc529080-1bionic [303 kB] 2019-12-05T05:00:41.045 INFO:teuthology.orchestra.run.smithi040.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-cmd2 all 0.7.9-0ubuntu1 [34.2 kB] 2019-12-05T05:00:41.065 INFO:teuthology.orchestra.run.smithi042.stdout:Get:10 https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191204.231200/c5290805033483ba0cec165c06bde07dae5f19ec/ubuntu/bionic/flavors/default bionic/main amd64 python-ceph-argparse all 15.0.0-8014-gc529080-1bionic [20.2 kB] 2019-12-05T05:00:41.066 INFO:teuthology.orchestra.run.smithi042.stdout:Get:11 https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191204.231200/c5290805033483ba0cec165c06bde07dae5f19ec/ubuntu/bionic/flavors/default bionic/main amd64 python-cephfs amd64 15.0.0-8014-gc529080-1bionic [113 kB] 2019-12-05T05:00:41.066 INFO:teuthology.orchestra.run.smithi042.stdout:Get:12 https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191204.231200/c5290805033483ba0cec165c06bde07dae5f19ec/ubuntu/bionic/flavors/default bionic/main amd64 python-ceph-common all 15.0.0-8014-gc529080-1bionic [33.0 kB] 2019-12-05T05:00:41.066 INFO:teuthology.orchestra.run.smithi042.stdout:Get:13 https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191204.231200/c5290805033483ba0cec165c06bde07dae5f19ec/ubuntu/bionic/flavors/default bionic/main amd64 python-rbd amd64 15.0.0-8014-gc529080-1bionic [262 kB] 2019-12-05T05:00:41.067 INFO:teuthology.orchestra.run.smithi042.stdout:Get:14 https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191204.231200/c5290805033483ba0cec165c06bde07dae5f19ec/ubuntu/bionic/flavors/default bionic/main amd64 librgw2 amd64 15.0.0-8014-gc529080-1bionic [4,538 kB] 2019-12-05T05:00:41.083 INFO:teuthology.orchestra.run.smithi040.stdout:Get:7 http://archive.ubuntu.com/ubuntu bionic/universe amd64 liblttng-ust0 amd64 2.10.1-1 [154 kB] 2019-12-05T05:00:41.117 INFO:teuthology.orchestra.run.smithi040.stdout:Get:8 https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191204.231200/c5290805033483ba0cec165c06bde07dae5f19ec/ubuntu/bionic/flavors/default bionic/main amd64 libcephfs2 amd64 15.0.0-8014-gc529080-1bionic [427 kB] 2019-12-05T05:00:41.128 INFO:teuthology.orchestra.run.smithi042.stdout:Get:15 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-prettytable all 0.7.2-3 [19.6 kB] 2019-12-05T05:00:41.138 INFO:teuthology.orchestra.run.smithi042.stdout:Get:16 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 librabbitmq4 amd64 0.8.0-1ubuntu0.18.04.1 [33.9 kB] 2019-12-05T05:00:41.151 INFO:teuthology.orchestra.run.smithi042.stdout:Get:17 http://archive.ubuntu.com/ubuntu bionic/universe amd64 librdkafka1 amd64 0.11.3-1build1 [293 kB] 2019-12-05T05:00:41.152 INFO:teuthology.orchestra.run.smithi040.stdout:Get:9 https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191204.231200/c5290805033483ba0cec165c06bde07dae5f19ec/ubuntu/bionic/flavors/default bionic/main amd64 python-rados amd64 15.0.0-8014-gc529080-1bionic [303 kB] 2019-12-05T05:00:41.182 INFO:teuthology.orchestra.run.smithi040.stdout:Get:10 https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191204.231200/c5290805033483ba0cec165c06bde07dae5f19ec/ubuntu/bionic/flavors/default bionic/main amd64 python-ceph-argparse all 15.0.0-8014-gc529080-1bionic [20.2 kB] 2019-12-05T05:00:41.183 INFO:teuthology.orchestra.run.smithi040.stdout:Get:11 https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191204.231200/c5290805033483ba0cec165c06bde07dae5f19ec/ubuntu/bionic/flavors/default bionic/main amd64 python-cephfs amd64 15.0.0-8014-gc529080-1bionic [113 kB] 2019-12-05T05:00:41.187 INFO:teuthology.orchestra.run.smithi040.stdout:Get:12 https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191204.231200/c5290805033483ba0cec165c06bde07dae5f19ec/ubuntu/bionic/flavors/default bionic/main amd64 python-ceph-common all 15.0.0-8014-gc529080-1bionic [33.0 kB] 2019-12-05T05:00:41.189 INFO:teuthology.orchestra.run.smithi040.stdout:Get:13 https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191204.231200/c5290805033483ba0cec165c06bde07dae5f19ec/ubuntu/bionic/flavors/default bionic/main amd64 python-rbd amd64 15.0.0-8014-gc529080-1bionic [262 kB] 2019-12-05T05:00:41.190 INFO:teuthology.orchestra.run.smithi040.stdout:Get:14 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-prettytable all 0.7.2-3 [19.6 kB] 2019-12-05T05:00:41.196 INFO:teuthology.orchestra.run.smithi040.stdout:Get:15 https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191204.231200/c5290805033483ba0cec165c06bde07dae5f19ec/ubuntu/bionic/flavors/default bionic/main amd64 librgw2 amd64 15.0.0-8014-gc529080-1bionic [4,538 kB] 2019-12-05T05:00:41.199 INFO:teuthology.orchestra.run.smithi040.stdout:Get:16 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 librabbitmq4 amd64 0.8.0-1ubuntu0.18.04.1 [33.9 kB] 2019-12-05T05:00:41.212 INFO:teuthology.orchestra.run.smithi040.stdout:Get:17 http://archive.ubuntu.com/ubuntu bionic/universe amd64 librdkafka1 amd64 0.11.3-1build1 [293 kB] 2019-12-05T05:00:41.237 INFO:teuthology.orchestra.run.smithi042.stdout:Get:18 http://archive.ubuntu.com/ubuntu bionic/main amd64 libsnappy1v5 amd64 1.1.7-1 [16.0 kB] 2019-12-05T05:00:41.238 INFO:teuthology.orchestra.run.smithi042.stdout:Get:19 http://archive.ubuntu.com/ubuntu bionic/main amd64 libleveldb1v5 amd64 1.20-2 [136 kB] 2019-12-05T05:00:41.265 INFO:teuthology.orchestra.run.smithi042.stdout:Get:20 http://archive.ubuntu.com/ubuntu bionic/universe amd64 liboath0 amd64 2.6.1-1 [44.7 kB] 2019-12-05T05:00:41.266 INFO:teuthology.orchestra.run.smithi042.stdout:Get:21 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-bcrypt amd64 3.1.4-2 [29.8 kB] 2019-12-05T05:00:41.298 INFO:teuthology.orchestra.run.smithi040.stdout:Get:18 http://archive.ubuntu.com/ubuntu bionic/main amd64 libsnappy1v5 amd64 1.1.7-1 [16.0 kB] 2019-12-05T05:00:41.299 INFO:teuthology.orchestra.run.smithi040.stdout:Get:19 http://archive.ubuntu.com/ubuntu bionic/main amd64 libleveldb1v5 amd64 1.20-2 [136 kB] 2019-12-05T05:00:41.299 INFO:teuthology.orchestra.run.smithi042.stdout:Get:22 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-cherrypy3 all 8.9.1-2 [160 kB] 2019-12-05T05:00:41.299 INFO:teuthology.orchestra.run.smithi042.stdout:Get:23 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-jwt all 1.5.3+ds1-1 [15.8 kB] 2019-12-05T05:00:41.299 INFO:teuthology.orchestra.run.smithi042.stdout:Get:24 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-openssl all 17.5.0-1ubuntu1 [41.3 kB] 2019-12-05T05:00:41.300 INFO:teuthology.orchestra.run.smithi042.stdout:Get:25 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-logutils all 0.3.3-5 [16.7 kB] 2019-12-05T05:00:41.300 INFO:teuthology.orchestra.run.smithi042.stdout:Get:26 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-markupsafe amd64 1.0-1build1 [13.0 kB] 2019-12-05T05:00:41.313 INFO:teuthology.orchestra.run.smithi040.stdout:Get:20 http://archive.ubuntu.com/ubuntu bionic/universe amd64 liboath0 amd64 2.6.1-1 [44.7 kB] 2019-12-05T05:00:41.329 INFO:teuthology.orchestra.run.smithi040.stdout:Get:21 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-bcrypt amd64 3.1.4-2 [29.8 kB] 2019-12-05T05:00:41.329 INFO:teuthology.orchestra.run.smithi040.stdout:Get:22 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-cherrypy3 all 8.9.1-2 [160 kB] 2019-12-05T05:00:41.343 INFO:teuthology.orchestra.run.smithi040.stdout:Get:23 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-jwt all 1.5.3+ds1-1 [15.8 kB] 2019-12-05T05:00:41.344 INFO:teuthology.orchestra.run.smithi040.stdout:Get:24 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-openssl all 17.5.0-1ubuntu1 [41.3 kB] 2019-12-05T05:00:41.349 INFO:teuthology.orchestra.run.smithi042.stdout:Get:27 https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191204.231200/c5290805033483ba0cec165c06bde07dae5f19ec/ubuntu/bionic/flavors/default bionic/main amd64 python-rgw amd64 15.0.0-8014-gc529080-1bionic [98.7 kB] 2019-12-05T05:00:41.351 INFO:teuthology.orchestra.run.smithi040.stdout:Get:25 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-logutils all 0.3.3-5 [16.7 kB] 2019-12-05T05:00:41.351 INFO:teuthology.orchestra.run.smithi042.stdout:Get:28 https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191204.231200/c5290805033483ba0cec165c06bde07dae5f19ec/ubuntu/bionic/flavors/default bionic/main amd64 libradosstriper1 amd64 15.0.0-8014-gc529080-1bionic [329 kB] 2019-12-05T05:00:41.352 INFO:teuthology.orchestra.run.smithi040.stdout:Get:26 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-markupsafe amd64 1.0-1build1 [13.0 kB] 2019-12-05T05:00:41.373 INFO:teuthology.orchestra.run.smithi042.stdout:Get:29 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-mako all 1.0.7+ds1-1 [60.1 kB] 2019-12-05T05:00:41.376 INFO:teuthology.orchestra.run.smithi042.stdout:Get:30 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-simplegeneric all 0.8.1-1 [11.5 kB] 2019-12-05T05:00:41.377 INFO:teuthology.orchestra.run.smithi042.stdout:Get:31 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-singledispatch all 3.4.0.3-2 [9,848 B] 2019-12-05T05:00:41.377 INFO:teuthology.orchestra.run.smithi042.stdout:Get:32 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-simplejson amd64 3.13.2-1 [61.2 kB] 2019-12-05T05:00:41.385 INFO:teuthology.orchestra.run.smithi042.stdout:Get:33 https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191204.231200/c5290805033483ba0cec165c06bde07dae5f19ec/ubuntu/bionic/flavors/default bionic/main amd64 ceph-common amd64 15.0.0-8014-gc529080-1bionic [18.0 MB] 2019-12-05T05:00:41.432 INFO:teuthology.orchestra.run.smithi040.stdout:Get:27 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-mako all 1.0.7+ds1-1 [60.1 kB] 2019-12-05T05:00:41.435 INFO:teuthology.orchestra.run.smithi040.stdout:Get:28 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-simplegeneric all 0.8.1-1 [11.5 kB] 2019-12-05T05:00:41.436 INFO:teuthology.orchestra.run.smithi040.stdout:Get:29 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-singledispatch all 3.4.0.3-2 [9,848 B] 2019-12-05T05:00:41.436 INFO:teuthology.orchestra.run.smithi040.stdout:Get:30 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-simplejson amd64 3.13.2-1 [61.2 kB] 2019-12-05T05:00:41.530 INFO:teuthology.orchestra.run.smithi040.stdout:Get:31 https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191204.231200/c5290805033483ba0cec165c06bde07dae5f19ec/ubuntu/bionic/flavors/default bionic/main amd64 python-rgw amd64 15.0.0-8014-gc529080-1bionic [98.7 kB] 2019-12-05T05:00:41.533 INFO:teuthology.orchestra.run.smithi040.stdout:Get:32 https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191204.231200/c5290805033483ba0cec165c06bde07dae5f19ec/ubuntu/bionic/flavors/default bionic/main amd64 libradosstriper1 amd64 15.0.0-8014-gc529080-1bionic [329 kB] 2019-12-05T05:00:41.541 INFO:teuthology.orchestra.run.smithi042.stdout:Get:34 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-webob all 1:1.7.3-2fakesync1 [64.3 kB] 2019-12-05T05:00:41.544 INFO:teuthology.orchestra.run.smithi040.stdout:Get:33 https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191204.231200/c5290805033483ba0cec165c06bde07dae5f19ec/ubuntu/bionic/flavors/default bionic/main amd64 ceph-common amd64 15.0.0-8014-gc529080-1bionic [18.0 MB] 2019-12-05T05:00:41.601 INFO:teuthology.orchestra.run.smithi040.stdout:Get:34 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-webob all 1:1.7.3-2fakesync1 [64.3 kB] 2019-12-05T05:00:41.827 INFO:teuthology.orchestra.run.smithi042.stdout:Get:35 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-bs4 all 4.6.0-1 [67.9 kB] 2019-12-05T05:00:41.886 INFO:teuthology.orchestra.run.smithi042.stdout:Get:36 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-waitress all 1.0.1-1 [53.3 kB] 2019-12-05T05:00:41.899 INFO:teuthology.orchestra.run.smithi040.stdout:Get:35 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-bs4 all 4.6.0-1 [67.9 kB] 2019-12-05T05:00:41.948 INFO:teuthology.orchestra.run.smithi042.stdout:Get:37 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-dnspython all 1.15.0-1 [84.9 kB] 2019-12-05T05:00:41.953 INFO:teuthology.orchestra.run.smithi040.stdout:Get:36 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-waitress all 1.0.1-1 [53.3 kB] 2019-12-05T05:00:42.001 INFO:teuthology.orchestra.run.smithi042.stdout:Get:38 http://archive.ubuntu.com/ubuntu bionic/main amd64 formencode-i18n all 1.3.0-0ubuntu5 [3,382 B] 2019-12-05T05:00:42.001 INFO:teuthology.orchestra.run.smithi042.stdout:Get:39 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-formencode all 1.3.0-0ubuntu5 [69.1 kB] 2019-12-05T05:00:42.015 INFO:teuthology.orchestra.run.smithi040.stdout:Get:37 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-dnspython all 1.15.0-1 [84.9 kB] 2019-12-05T05:00:42.025 INFO:teuthology.orchestra.run.smithi042.stdout:Get:40 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-tempita all 0.5.2-2 [13.9 kB] 2019-12-05T05:00:42.030 INFO:teuthology.orchestra.run.smithi042.stdout:Get:41 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-paste all 2.0.3+dfsg-4ubuntu1 [456 kB] 2019-12-05T05:00:42.067 INFO:teuthology.orchestra.run.smithi040.stdout:Get:38 http://archive.ubuntu.com/ubuntu bionic/main amd64 formencode-i18n all 1.3.0-0ubuntu5 [3,382 B] 2019-12-05T05:00:42.068 INFO:teuthology.orchestra.run.smithi040.stdout:Get:39 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-formencode all 1.3.0-0ubuntu5 [69.1 kB] 2019-12-05T05:00:42.092 INFO:teuthology.orchestra.run.smithi040.stdout:Get:40 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-tempita all 0.5.2-2 [13.9 kB] 2019-12-05T05:00:42.097 INFO:teuthology.orchestra.run.smithi040.stdout:Get:41 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-paste all 2.0.3+dfsg-4ubuntu1 [456 kB] 2019-12-05T05:00:42.135 INFO:teuthology.orchestra.run.smithi042.stdout:Get:42 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-pastedeploy-tpl all 1.5.2-4 [4,796 B] 2019-12-05T05:00:42.136 INFO:teuthology.orchestra.run.smithi042.stdout:Get:43 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-pastedeploy all 1.5.2-4 [25.6 kB] 2019-12-05T05:00:42.140 INFO:teuthology.orchestra.run.smithi042.stdout:Get:44 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-webtest all 2.0.28-1ubuntu1 [27.8 kB] 2019-12-05T05:00:42.144 INFO:teuthology.orchestra.run.smithi042.stdout:Get:45 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-pecan all 1.2.1-2 [86.1 kB] 2019-12-05T05:00:42.157 INFO:teuthology.orchestra.run.smithi042.stdout:Get:46 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-certifi all 2018.1.18-2 [144 kB] 2019-12-05T05:00:42.182 INFO:teuthology.orchestra.run.smithi042.stdout:Get:47 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-chardet all 3.0.4-1 [80.3 kB] 2019-12-05T05:00:42.190 INFO:teuthology.orchestra.run.smithi042.stdout:Get:48 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python-urllib3 all 1.22-1ubuntu0.18.04.1 [85.9 kB] 2019-12-05T05:00:42.198 INFO:teuthology.orchestra.run.smithi042.stdout:Get:49 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python-requests all 2.18.4-2ubuntu0.1 [58.5 kB] 2019-12-05T05:00:42.198 INFO:teuthology.orchestra.run.smithi040.stdout:Get:42 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-pastedeploy-tpl all 1.5.2-4 [4,796 B] 2019-12-05T05:00:42.199 INFO:teuthology.orchestra.run.smithi040.stdout:Get:43 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-pastedeploy all 1.5.2-4 [25.6 kB] 2019-12-05T05:00:42.202 INFO:teuthology.orchestra.run.smithi040.stdout:Get:44 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-webtest all 2.0.28-1ubuntu1 [27.8 kB] 2019-12-05T05:00:42.203 INFO:teuthology.orchestra.run.smithi042.stdout:Get:50 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-werkzeug all 0.14.1+dfsg1-1 [174 kB] 2019-12-05T05:00:42.205 INFO:teuthology.orchestra.run.smithi040.stdout:Get:45 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-pecan all 1.2.1-2 [86.1 kB] 2019-12-05T05:00:42.224 INFO:teuthology.orchestra.run.smithi040.stdout:Get:46 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-certifi all 2018.1.18-2 [144 kB] 2019-12-05T05:00:42.233 INFO:teuthology.orchestra.run.smithi042.stdout:Get:51 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-repoze.lru all 0.7-1 [11.9 kB] 2019-12-05T05:00:42.233 INFO:teuthology.orchestra.run.smithi042.stdout:Get:52 http://archive.ubuntu.com/ubuntu bionic/main amd64 libjs-underscore all 1.8.3~dfsg-1 [59.9 kB] 2019-12-05T05:00:42.236 INFO:teuthology.orchestra.run.smithi042.stdout:Get:53 http://archive.ubuntu.com/ubuntu bionic/main amd64 libjs-sphinxdoc all 1.6.7-1ubuntu1 [85.6 kB] 2019-12-05T05:00:42.241 INFO:teuthology.orchestra.run.smithi042.stdout:Get:54 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-routes all 2.4.1-1 [88.3 kB] 2019-12-05T05:00:42.257 INFO:teuthology.orchestra.run.smithi040.stdout:Get:47 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-chardet all 3.0.4-1 [80.3 kB] 2019-12-05T05:00:42.257 INFO:teuthology.orchestra.run.smithi040.stdout:Get:48 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python-urllib3 all 1.22-1ubuntu0.18.04.1 [85.9 kB] 2019-12-05T05:00:42.269 INFO:teuthology.orchestra.run.smithi040.stdout:Get:49 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python-requests all 2.18.4-2ubuntu0.1 [58.5 kB] 2019-12-05T05:00:42.269 INFO:teuthology.orchestra.run.smithi040.stdout:Get:50 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-werkzeug all 0.14.1+dfsg1-1 [174 kB] 2019-12-05T05:00:42.305 INFO:teuthology.orchestra.run.smithi040.stdout:Get:51 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-repoze.lru all 0.7-1 [11.9 kB] 2019-12-05T05:00:42.306 INFO:teuthology.orchestra.run.smithi040.stdout:Get:52 http://archive.ubuntu.com/ubuntu bionic/main amd64 libjs-underscore all 1.8.3~dfsg-1 [59.9 kB] 2019-12-05T05:00:42.309 INFO:teuthology.orchestra.run.smithi040.stdout:Get:53 http://archive.ubuntu.com/ubuntu bionic/main amd64 libjs-sphinxdoc all 1.6.7-1ubuntu1 [85.6 kB] 2019-12-05T05:00:42.314 INFO:teuthology.orchestra.run.smithi040.stdout:Get:54 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-routes all 2.4.1-1 [88.3 kB] 2019-12-05T05:00:42.408 INFO:teuthology.orchestra.run.smithi042.stdout:Get:55 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-decorator all 4.1.2-1 [9,300 B] 2019-12-05T05:00:42.482 INFO:teuthology.orchestra.run.smithi040.stdout:Get:55 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-decorator all 4.1.2-1 [9,300 B] 2019-12-05T05:00:42.482 INFO:teuthology.orchestra.run.smithi042.stdout:Get:56 https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191204.231200/c5290805033483ba0cec165c06bde07dae5f19ec/ubuntu/bionic/flavors/default bionic/main amd64 ceph-base amd64 15.0.0-8014-gc529080-1bionic [5,087 kB] 2019-12-05T05:00:42.489 INFO:teuthology.orchestra.run.smithi042.stdout:Get:57 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-scipy amd64 0.19.1-2ubuntu1 [9,689 kB] 2019-12-05T05:00:42.587 INFO:teuthology.orchestra.run.smithi040.stdout:Get:56 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-scipy amd64 0.19.1-2ubuntu1 [9,689 kB] 2019-12-05T05:00:42.789 INFO:teuthology.orchestra.run.smithi042.stdout:Get:58 https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191204.231200/c5290805033483ba0cec165c06bde07dae5f19ec/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr amd64 15.0.0-8014-gc529080-1bionic [1,382 kB] 2019-12-05T05:00:42.837 INFO:teuthology.orchestra.run.smithi040.stdout:Get:57 https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191204.231200/c5290805033483ba0cec165c06bde07dae5f19ec/ubuntu/bionic/flavors/default bionic/main amd64 ceph-base amd64 15.0.0-8014-gc529080-1bionic [5,087 kB] 2019-12-05T05:00:42.862 INFO:teuthology.orchestra.run.smithi042.stdout:Get:59 https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191204.231200/c5290805033483ba0cec165c06bde07dae5f19ec/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mon amd64 15.0.0-8014-gc529080-1bionic [5,943 kB] 2019-12-05T05:00:43.207 INFO:teuthology.orchestra.run.smithi040.stdout:Get:58 https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191204.231200/c5290805033483ba0cec165c06bde07dae5f19ec/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr amd64 15.0.0-8014-gc529080-1bionic [1,382 kB] 2019-12-05T05:00:43.235 INFO:teuthology.orchestra.run.smithi042.stdout:Get:60 https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191204.231200/c5290805033483ba0cec165c06bde07dae5f19ec/ubuntu/bionic/flavors/default bionic/main amd64 ceph-osd amd64 15.0.0-8014-gc529080-1bionic [22.5 MB] 2019-12-05T05:00:43.281 INFO:teuthology.orchestra.run.smithi042.stdout:Get:61 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-sklearn-lib amd64 0.19.1-3 [1,521 kB] 2019-12-05T05:00:43.290 INFO:teuthology.orchestra.run.smithi040.stdout:Get:59 https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191204.231200/c5290805033483ba0cec165c06bde07dae5f19ec/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mon amd64 15.0.0-8014-gc529080-1bionic [5,943 kB] 2019-12-05T05:00:43.329 INFO:teuthology.orchestra.run.smithi042.stdout:Get:62 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-joblib all 0.11-1 [112 kB] 2019-12-05T05:00:43.341 INFO:teuthology.orchestra.run.smithi042.stdout:Get:63 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-sklearn all 0.19.1-3 [1,373 kB] 2019-12-05T05:00:43.359 INFO:teuthology.orchestra.run.smithi042.stdout:Get:64 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-monotonic all 1.1-2 [5,262 B] 2019-12-05T05:00:43.840 INFO:teuthology.orchestra.run.smithi042.stdout:Get:65 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-fasteners all 0.12.0-3 [13.4 kB] 2019-12-05T05:00:43.840 INFO:teuthology.orchestra.run.smithi042.stdout:Get:66 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python-httplib2 all 0.9.2+dfsg-1ubuntu0.1 [34.7 kB] 2019-12-05T05:00:43.841 INFO:teuthology.orchestra.run.smithi042.stdout:Get:67 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-pyasn1 all 0.4.2-3 [46.7 kB] 2019-12-05T05:00:43.844 INFO:teuthology.orchestra.run.smithi040.stdout:Get:60 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-sklearn-lib amd64 0.19.1-3 [1,521 kB] 2019-12-05T05:00:43.844 INFO:teuthology.orchestra.run.smithi040.stdout:Get:61 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-joblib all 0.11-1 [112 kB] 2019-12-05T05:00:43.845 INFO:teuthology.orchestra.run.smithi040.stdout:Get:62 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-sklearn all 0.19.1-3 [1,373 kB] 2019-12-05T05:00:43.845 INFO:teuthology.orchestra.run.smithi040.stdout:Get:63 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-monotonic all 1.1-2 [5,262 B] 2019-12-05T05:00:43.845 INFO:teuthology.orchestra.run.smithi040.stdout:Get:64 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-fasteners all 0.12.0-3 [13.4 kB] 2019-12-05T05:00:43.845 INFO:teuthology.orchestra.run.smithi040.stdout:Get:65 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python-httplib2 all 0.9.2+dfsg-1ubuntu0.1 [34.7 kB] 2019-12-05T05:00:43.845 INFO:teuthology.orchestra.run.smithi040.stdout:Get:66 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-pyasn1 all 0.4.2-3 [46.7 kB] 2019-12-05T05:00:43.845 INFO:teuthology.orchestra.run.smithi040.stdout:Get:67 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-pyasn1-modules all 0.2.1-0.2 [32.8 kB] 2019-12-05T05:00:43.845 INFO:teuthology.orchestra.run.smithi040.stdout:Get:68 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-rsa all 3.4.2-1 [28.9 kB] 2019-12-05T05:00:43.846 INFO:teuthology.orchestra.run.smithi040.stdout:Get:69 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-sqlalchemy all 1.1.11+ds1-1ubuntu1 [682 kB] 2019-12-05T05:00:43.846 INFO:teuthology.orchestra.run.smithi040.stdout:Get:70 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-oauth2client all 4.1.2-3 [61.4 kB] 2019-12-05T05:00:43.846 INFO:teuthology.orchestra.run.smithi040.stdout:Get:71 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-websocket all 0.44.0-0ubuntu2 [30.7 kB] 2019-12-05T05:00:43.846 INFO:teuthology.orchestra.run.smithi040.stdout:Get:72 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-yaml amd64 3.12-1build2 [115 kB] 2019-12-05T05:00:43.846 INFO:teuthology.orchestra.run.smithi040.stdout:Get:73 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-kubernetes all 2.0.0-2ubuntu1 [181 kB] 2019-12-05T05:00:43.846 INFO:teuthology.orchestra.run.smithi040.stdout:Get:74 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libonig4 amd64 6.7.0-1 [119 kB] 2019-12-05T05:00:43.846 INFO:teuthology.orchestra.run.smithi040.stdout:Get:75 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libjq1 amd64 1.5+dfsg-2 [111 kB] 2019-12-05T05:00:43.847 INFO:teuthology.orchestra.run.smithi040.stdout:Get:76 http://archive.ubuntu.com/ubuntu bionic/universe amd64 jq amd64 1.5+dfsg-2 [45.6 kB] 2019-12-05T05:00:43.847 INFO:teuthology.orchestra.run.smithi040.stdout:Get:77 http://archive.ubuntu.com/ubuntu bionic/main amd64 socat amd64 1.7.3.2-2ubuntu2 [342 kB] 2019-12-05T05:00:43.847 INFO:teuthology.orchestra.run.smithi040.stdout:Get:78 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libxslt1.1 amd64 1.1.29-5ubuntu0.2 [150 kB] 2019-12-05T05:00:43.847 INFO:teuthology.orchestra.run.smithi040.stdout:Get:79 https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191204.231200/c5290805033483ba0cec165c06bde07dae5f19ec/ubuntu/bionic/flavors/default bionic/main amd64 ceph-osd amd64 15.0.0-8014-gc529080-1bionic [22.5 MB] 2019-12-05T05:00:43.847 INFO:teuthology.orchestra.run.smithi040.stdout:Get:80 http://archive.ubuntu.com/ubuntu bionic/universe amd64 xmlstarlet amd64 1.6.1-2 [382 kB] 2019-12-05T05:00:43.847 INFO:teuthology.orchestra.run.smithi042.stdout:Get:68 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-pyasn1-modules all 0.2.1-0.2 [32.8 kB] 2019-12-05T05:00:43.847 INFO:teuthology.orchestra.run.smithi042.stdout:Get:69 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-rsa all 3.4.2-1 [28.9 kB] 2019-12-05T05:00:43.848 INFO:teuthology.orchestra.run.smithi042.stdout:Get:70 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-sqlalchemy all 1.1.11+ds1-1ubuntu1 [682 kB] 2019-12-05T05:00:43.848 INFO:teuthology.orchestra.run.smithi042.stdout:Get:71 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-oauth2client all 4.1.2-3 [61.4 kB] 2019-12-05T05:00:43.848 INFO:teuthology.orchestra.run.smithi042.stdout:Get:72 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-websocket all 0.44.0-0ubuntu2 [30.7 kB] 2019-12-05T05:00:43.848 INFO:teuthology.orchestra.run.smithi042.stdout:Get:73 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-yaml amd64 3.12-1build2 [115 kB] 2019-12-05T05:00:43.848 INFO:teuthology.orchestra.run.smithi042.stdout:Get:74 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-kubernetes all 2.0.0-2ubuntu1 [181 kB] 2019-12-05T05:00:43.848 INFO:teuthology.orchestra.run.smithi042.stdout:Get:75 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libonig4 amd64 6.7.0-1 [119 kB] 2019-12-05T05:00:43.848 INFO:teuthology.orchestra.run.smithi042.stdout:Get:76 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libjq1 amd64 1.5+dfsg-2 [111 kB] 2019-12-05T05:00:43.849 INFO:teuthology.orchestra.run.smithi042.stdout:Get:77 http://archive.ubuntu.com/ubuntu bionic/universe amd64 jq amd64 1.5+dfsg-2 [45.6 kB] 2019-12-05T05:00:43.849 INFO:teuthology.orchestra.run.smithi042.stdout:Get:78 http://archive.ubuntu.com/ubuntu bionic/main amd64 socat amd64 1.7.3.2-2ubuntu2 [342 kB] 2019-12-05T05:00:43.849 INFO:teuthology.orchestra.run.smithi042.stdout:Get:79 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libxslt1.1 amd64 1.1.29-5ubuntu0.2 [150 kB] 2019-12-05T05:00:43.849 INFO:teuthology.orchestra.run.smithi042.stdout:Get:80 http://archive.ubuntu.com/ubuntu bionic/universe amd64 xmlstarlet amd64 1.6.1-2 [382 kB] 2019-12-05T05:00:44.231 INFO:teuthology.orchestra.run.smithi042.stdout:Get:81 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-colorama all 0.3.7-1 [14.9 kB] 2019-12-05T05:00:44.232 INFO:teuthology.orchestra.run.smithi042.stdout:Get:82 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-attr all 17.4.0-2 [23.7 kB] 2019-12-05T05:00:44.236 INFO:teuthology.orchestra.run.smithi042.stdout:Get:83 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-funcsigs all 1.0.2-4 [13.5 kB] 2019-12-05T05:00:44.238 INFO:teuthology.orchestra.run.smithi042.stdout:Get:84 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-webencodings all 0.5-2 [10.3 kB] 2019-12-05T05:00:44.239 INFO:teuthology.orchestra.run.smithi042.stdout:Get:85 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-html5lib all 0.999999999-1 [83.6 kB] 2019-12-05T05:00:44.263 INFO:teuthology.orchestra.run.smithi042.stdout:Get:86 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python-lxml amd64 4.2.1-1ubuntu0.1 [1,075 kB] 2019-12-05T05:00:44.299 INFO:teuthology.orchestra.run.smithi040.stdout:Get:81 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-colorama all 0.3.7-1 [14.9 kB] 2019-12-05T05:00:44.303 INFO:teuthology.orchestra.run.smithi040.stdout:Get:82 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-attr all 17.4.0-2 [23.7 kB] 2019-12-05T05:00:44.305 INFO:teuthology.orchestra.run.smithi040.stdout:Get:83 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-funcsigs all 1.0.2-4 [13.5 kB] 2019-12-05T05:00:44.307 INFO:teuthology.orchestra.run.smithi040.stdout:Get:84 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-webencodings all 0.5-2 [10.3 kB] 2019-12-05T05:00:44.308 INFO:teuthology.orchestra.run.smithi040.stdout:Get:85 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-html5lib all 0.999999999-1 [83.6 kB] 2019-12-05T05:00:44.319 INFO:teuthology.orchestra.run.smithi040.stdout:Get:86 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python-lxml amd64 4.2.1-1ubuntu0.1 [1,075 kB] 2019-12-05T05:00:44.409 INFO:teuthology.orchestra.run.smithi042.stdout:Get:87 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-openid all 2.2.5-6 [97.8 kB] 2019-12-05T05:00:44.415 INFO:teuthology.orchestra.run.smithi042.stdout:Get:88 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-pastescript all 2.0.2-2 [54.9 kB] 2019-12-05T05:00:44.419 INFO:teuthology.orchestra.run.smithi042.stdout:Get:89 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-pluggy all 0.6.0-1 [12.2 kB] 2019-12-05T05:00:44.420 INFO:teuthology.orchestra.run.smithi042.stdout:Get:90 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-py all 1.5.2-1 [65.1 kB] 2019-12-05T05:00:44.424 INFO:teuthology.orchestra.run.smithi042.stdout:Get:91 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-pyinotify all 0.9.6-1 [24.6 kB] 2019-12-05T05:00:44.426 INFO:teuthology.orchestra.run.smithi042.stdout:Get:92 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-pytest all 3.3.2-2 [142 kB] 2019-12-05T05:00:44.437 INFO:teuthology.orchestra.run.smithi042.stdout:Get:93 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-scgi amd64 1.13-1.1build1 [18.4 kB] 2019-12-05T05:00:44.438 INFO:teuthology.orchestra.run.smithi042.stdout:Get:94 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-sqlalchemy-ext amd64 1.1.11+ds1-1ubuntu1 [13.4 kB] 2019-12-05T05:00:44.488 INFO:teuthology.orchestra.run.smithi040.stdout:Get:87 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-openid all 2.2.5-6 [97.8 kB] 2019-12-05T05:00:44.491 INFO:teuthology.orchestra.run.smithi040.stdout:Get:88 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-pastescript all 2.0.2-2 [54.9 kB] 2019-12-05T05:00:44.496 INFO:teuthology.orchestra.run.smithi040.stdout:Get:89 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-pluggy all 0.6.0-1 [12.2 kB] 2019-12-05T05:00:44.496 INFO:teuthology.orchestra.run.smithi040.stdout:Get:90 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-py all 1.5.2-1 [65.1 kB] 2019-12-05T05:00:44.501 INFO:teuthology.orchestra.run.smithi040.stdout:Get:91 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-pyinotify all 0.9.6-1 [24.6 kB] 2019-12-05T05:00:44.502 INFO:teuthology.orchestra.run.smithi040.stdout:Get:92 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-pytest all 3.3.2-2 [142 kB] 2019-12-05T05:00:44.514 INFO:teuthology.orchestra.run.smithi040.stdout:Get:93 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-scgi amd64 1.13-1.1build1 [18.4 kB] 2019-12-05T05:00:44.514 INFO:teuthology.orchestra.run.smithi040.stdout:Get:94 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-sqlalchemy-ext amd64 1.1.11+ds1-1ubuntu1 [13.4 kB] 2019-12-05T05:00:44.597 INFO:teuthology.orchestra.run.smithi042.stdout:Get:95 https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191204.231200/c5290805033483ba0cec165c06bde07dae5f19ec/ubuntu/bionic/flavors/default bionic/main amd64 ceph amd64 15.0.0-8014-gc529080-1bionic [3,804 B] 2019-12-05T05:00:44.598 INFO:teuthology.orchestra.run.smithi042.stdout:Get:96 https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191204.231200/c5290805033483ba0cec165c06bde07dae5f19ec/ubuntu/bionic/flavors/default bionic/main amd64 ceph-daemon amd64 15.0.0-8014-gc529080-1bionic [18.9 kB] 2019-12-05T05:00:44.599 INFO:teuthology.orchestra.run.smithi042.stdout:Get:97 https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191204.231200/c5290805033483ba0cec165c06bde07dae5f19ec/ubuntu/bionic/flavors/default bionic/main amd64 ceph-fuse amd64 15.0.0-8014-gc529080-1bionic [506 kB] 2019-12-05T05:00:44.639 INFO:teuthology.orchestra.run.smithi042.stdout:Get:98 https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191204.231200/c5290805033483ba0cec165c06bde07dae5f19ec/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mds amd64 15.0.0-8014-gc529080-1bionic [1,829 kB] 2019-12-05T05:00:44.738 INFO:teuthology.orchestra.run.smithi042.stdout:Get:99 https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191204.231200/c5290805033483ba0cec165c06bde07dae5f19ec/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-dashboard all 15.0.0-8014-gc529080-1bionic [3,392 kB] 2019-12-05T05:00:44.954 INFO:teuthology.orchestra.run.smithi042.stdout:Get:100 https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191204.231200/c5290805033483ba0cec165c06bde07dae5f19ec/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-diskprediction-cloud all 15.0.0-8014-gc529080-1bionic [34.1 kB] 2019-12-05T05:00:44.955 INFO:teuthology.orchestra.run.smithi042.stdout:Get:101 https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191204.231200/c5290805033483ba0cec165c06bde07dae5f19ec/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-diskprediction-local all 15.0.0-8014-gc529080-1bionic [7,867 kB] 2019-12-05T05:00:45.310 INFO:teuthology.orchestra.run.smithi040.stdout:Get:95 https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191204.231200/c5290805033483ba0cec165c06bde07dae5f19ec/ubuntu/bionic/flavors/default bionic/main amd64 ceph amd64 15.0.0-8014-gc529080-1bionic [3,804 B] 2019-12-05T05:00:45.310 INFO:teuthology.orchestra.run.smithi040.stdout:Get:96 https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191204.231200/c5290805033483ba0cec165c06bde07dae5f19ec/ubuntu/bionic/flavors/default bionic/main amd64 ceph-daemon amd64 15.0.0-8014-gc529080-1bionic [18.9 kB] 2019-12-05T05:00:45.311 INFO:teuthology.orchestra.run.smithi040.stdout:Get:97 https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191204.231200/c5290805033483ba0cec165c06bde07dae5f19ec/ubuntu/bionic/flavors/default bionic/main amd64 ceph-fuse amd64 15.0.0-8014-gc529080-1bionic [506 kB] 2019-12-05T05:00:45.328 INFO:teuthology.orchestra.run.smithi040.stdout:Get:98 https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191204.231200/c5290805033483ba0cec165c06bde07dae5f19ec/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mds amd64 15.0.0-8014-gc529080-1bionic [1,829 kB] 2019-12-05T05:00:45.427 INFO:teuthology.orchestra.run.smithi042.stdout:Get:102 https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191204.231200/c5290805033483ba0cec165c06bde07dae5f19ec/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-k8sevents all 15.0.0-8014-gc529080-1bionic [3,916 B] 2019-12-05T05:00:45.427 INFO:teuthology.orchestra.run.smithi042.stdout:Get:103 https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191204.231200/c5290805033483ba0cec165c06bde07dae5f19ec/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-rook all 15.0.0-8014-gc529080-1bionic [13.3 kB] 2019-12-05T05:00:45.427 INFO:teuthology.orchestra.run.smithi042.stdout:Get:104 https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191204.231200/c5290805033483ba0cec165c06bde07dae5f19ec/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-ssh all 15.0.0-8014-gc529080-1bionic [18.2 kB] 2019-12-05T05:00:45.427 INFO:teuthology.orchestra.run.smithi042.stdout:Get:105 https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191204.231200/c5290805033483ba0cec165c06bde07dae5f19ec/ubuntu/bionic/flavors/default bionic/main amd64 ceph-test amd64 15.0.0-8014-gc529080-1bionic [36.0 MB] 2019-12-05T05:00:45.465 INFO:teuthology.orchestra.run.smithi040.stdout:Get:99 https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191204.231200/c5290805033483ba0cec165c06bde07dae5f19ec/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-dashboard all 15.0.0-8014-gc529080-1bionic [3,392 kB] 2019-12-05T05:00:45.704 INFO:teuthology.orchestra.run.smithi040.stdout:Get:100 https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191204.231200/c5290805033483ba0cec165c06bde07dae5f19ec/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-diskprediction-cloud all 15.0.0-8014-gc529080-1bionic [34.1 kB] 2019-12-05T05:00:45.705 INFO:teuthology.orchestra.run.smithi040.stdout:Get:101 https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191204.231200/c5290805033483ba0cec165c06bde07dae5f19ec/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-diskprediction-local all 15.0.0-8014-gc529080-1bionic [7,867 kB] 2019-12-05T05:00:46.260 INFO:teuthology.orchestra.run.smithi040.stdout:Get:102 https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191204.231200/c5290805033483ba0cec165c06bde07dae5f19ec/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-k8sevents all 15.0.0-8014-gc529080-1bionic [3,916 B] 2019-12-05T05:00:46.261 INFO:teuthology.orchestra.run.smithi040.stdout:Get:103 https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191204.231200/c5290805033483ba0cec165c06bde07dae5f19ec/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-rook all 15.0.0-8014-gc529080-1bionic [13.3 kB] 2019-12-05T05:00:46.261 INFO:teuthology.orchestra.run.smithi040.stdout:Get:104 https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191204.231200/c5290805033483ba0cec165c06bde07dae5f19ec/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-ssh all 15.0.0-8014-gc529080-1bionic [18.2 kB] 2019-12-05T05:00:46.262 INFO:teuthology.orchestra.run.smithi040.stdout:Get:105 https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191204.231200/c5290805033483ba0cec165c06bde07dae5f19ec/ubuntu/bionic/flavors/default bionic/main amd64 ceph-test amd64 15.0.0-8014-gc529080-1bionic [36.0 MB] 2019-12-05T05:00:47.880 INFO:teuthology.orchestra.run.smithi042.stdout:Get:106 https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191204.231200/c5290805033483ba0cec165c06bde07dae5f19ec/ubuntu/bionic/flavors/default bionic/main amd64 python3-rados amd64 15.0.0-8014-gc529080-1bionic [323 kB] 2019-12-05T05:00:47.890 INFO:teuthology.orchestra.run.smithi042.stdout:Get:107 https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191204.231200/c5290805033483ba0cec165c06bde07dae5f19ec/ubuntu/bionic/flavors/default bionic/main amd64 python3-ceph-argparse all 15.0.0-8014-gc529080-1bionic [20.5 kB] 2019-12-05T05:00:47.890 INFO:teuthology.orchestra.run.smithi042.stdout:Get:108 https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191204.231200/c5290805033483ba0cec165c06bde07dae5f19ec/ubuntu/bionic/flavors/default bionic/main amd64 python3-cephfs amd64 15.0.0-8014-gc529080-1bionic [117 kB] 2019-12-05T05:00:47.899 INFO:teuthology.orchestra.run.smithi042.stdout:Get:109 https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191204.231200/c5290805033483ba0cec165c06bde07dae5f19ec/ubuntu/bionic/flavors/default bionic/main amd64 cephfs-shell all 15.0.0-8014-gc529080-1bionic [14.7 kB] 2019-12-05T05:00:47.911 INFO:teuthology.orchestra.run.smithi042.stdout:Get:110 https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191204.231200/c5290805033483ba0cec165c06bde07dae5f19ec/ubuntu/bionic/flavors/default bionic/main amd64 libcephfs-dev amd64 15.0.0-8014-gc529080-1bionic [17.3 kB] 2019-12-05T05:00:47.912 INFO:teuthology.orchestra.run.smithi042.stdout:Get:111 https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191204.231200/c5290805033483ba0cec165c06bde07dae5f19ec/ubuntu/bionic/flavors/default bionic/main amd64 python-ceph amd64 15.0.0-8014-gc529080-1bionic [3,860 B] 2019-12-05T05:00:47.912 INFO:teuthology.orchestra.run.smithi042.stdout:Get:112 https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191204.231200/c5290805033483ba0cec165c06bde07dae5f19ec/ubuntu/bionic/flavors/default bionic/main amd64 radosgw amd64 15.0.0-8014-gc529080-1bionic [5,194 kB] 2019-12-05T05:00:48.290 INFO:teuthology.orchestra.run.smithi042.stdout:Get:113 https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191204.231200/c5290805033483ba0cec165c06bde07dae5f19ec/ubuntu/bionic/flavors/default bionic/main amd64 rbd-fuse amd64 15.0.0-8014-gc529080-1bionic [62.2 kB] 2019-12-05T05:00:48.373 INFO:teuthology.orchestra.run.smithi040.stdout:Get:106 https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191204.231200/c5290805033483ba0cec165c06bde07dae5f19ec/ubuntu/bionic/flavors/default bionic/main amd64 python3-rados amd64 15.0.0-8014-gc529080-1bionic [323 kB] 2019-12-05T05:00:48.379 INFO:teuthology.orchestra.run.smithi040.stdout:Get:107 https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191204.231200/c5290805033483ba0cec165c06bde07dae5f19ec/ubuntu/bionic/flavors/default bionic/main amd64 python3-ceph-argparse all 15.0.0-8014-gc529080-1bionic [20.5 kB] 2019-12-05T05:00:48.380 INFO:teuthology.orchestra.run.smithi040.stdout:Get:108 https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191204.231200/c5290805033483ba0cec165c06bde07dae5f19ec/ubuntu/bionic/flavors/default bionic/main amd64 python3-cephfs amd64 15.0.0-8014-gc529080-1bionic [117 kB] 2019-12-05T05:00:48.385 INFO:teuthology.orchestra.run.smithi040.stdout:Get:109 https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191204.231200/c5290805033483ba0cec165c06bde07dae5f19ec/ubuntu/bionic/flavors/default bionic/main amd64 cephfs-shell all 15.0.0-8014-gc529080-1bionic [14.7 kB] 2019-12-05T05:00:48.386 INFO:teuthology.orchestra.run.smithi040.stdout:Get:110 https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191204.231200/c5290805033483ba0cec165c06bde07dae5f19ec/ubuntu/bionic/flavors/default bionic/main amd64 libcephfs-dev amd64 15.0.0-8014-gc529080-1bionic [17.3 kB] 2019-12-05T05:00:48.387 INFO:teuthology.orchestra.run.smithi040.stdout:Get:111 https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191204.231200/c5290805033483ba0cec165c06bde07dae5f19ec/ubuntu/bionic/flavors/default bionic/main amd64 python-ceph amd64 15.0.0-8014-gc529080-1bionic [3,860 B] 2019-12-05T05:00:48.387 INFO:teuthology.orchestra.run.smithi040.stdout:Get:112 https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191204.231200/c5290805033483ba0cec165c06bde07dae5f19ec/ubuntu/bionic/flavors/default bionic/main amd64 radosgw amd64 15.0.0-8014-gc529080-1bionic [5,194 kB] 2019-12-05T05:00:48.557 INFO:teuthology.orchestra.run.smithi040.stdout:Get:113 https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191204.231200/c5290805033483ba0cec165c06bde07dae5f19ec/ubuntu/bionic/flavors/default bionic/main amd64 rbd-fuse amd64 15.0.0-8014-gc529080-1bionic [62.2 kB] 2019-12-05T05:00:49.010 INFO:teuthology.orchestra.run.smithi042.stdout:Fetched 139 MB in 8s (17.9 MB/s) 2019-12-05T05:00:49.139 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python-uritemplate. 2019-12-05T05:00:49.192 INFO:teuthology.orchestra.run.smithi042.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 137175 files and directories currently installed.) 2019-12-05T05:00:49.215 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../000-python-uritemplate_0.6-1ubuntu1_all.deb ... 2019-12-05T05:00:49.229 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python-uritemplate (0.6-1ubuntu1) ... 2019-12-05T05:00:49.247 INFO:teuthology.orchestra.run.smithi040.stdout:Fetched 139 MB in 8s (17.5 MB/s) 2019-12-05T05:00:49.366 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python-uritemplate. 2019-12-05T05:00:49.409 INFO:teuthology.orchestra.run.smithi040.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 137175 files and directories currently installed.) 2019-12-05T05:00:49.414 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../000-python-uritemplate_0.6-1ubuntu1_all.deb ... 2019-12-05T05:00:49.449 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python-uritemplate (0.6-1ubuntu1) ... 2019-12-05T05:00:49.507 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-pyparsing. 2019-12-05T05:00:49.521 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../001-python3-pyparsing_2.2.0+dfsg1-2_all.deb ... 2019-12-05T05:00:49.566 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-pyparsing (2.2.0+dfsg1-2) ... 2019-12-05T05:00:49.721 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-pyparsing. 2019-12-05T05:00:49.734 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../001-python3-pyparsing_2.2.0+dfsg1-2_all.deb ... 2019-12-05T05:00:49.769 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-pyparsing (2.2.0+dfsg1-2) ... 2019-12-05T05:00:49.893 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-pyperclip. 2019-12-05T05:00:49.907 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../002-python3-pyperclip_1.6.0-1_all.deb ... 2019-12-05T05:00:49.952 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-pyperclip (1.6.0-1) ... 2019-12-05T05:00:50.063 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-pyperclip. 2019-12-05T05:00:50.079 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../002-python3-pyperclip_1.6.0-1_all.deb ... 2019-12-05T05:00:50.142 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-pyperclip (1.6.0-1) ... 2019-12-05T05:00:50.267 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-cmd2. 2019-12-05T05:00:50.281 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../003-python3-cmd2_0.7.9-0ubuntu1_all.deb ... 2019-12-05T05:00:50.326 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-cmd2 (0.7.9-0ubuntu1) ... 2019-12-05T05:00:50.574 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-cmd2. 2019-12-05T05:00:50.590 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../003-python3-cmd2_0.7.9-0ubuntu1_all.deb ... 2019-12-05T05:00:50.640 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2019-12-05T05:00:50.653 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../004-liblttng-ust0_2.10.1-1_amd64.deb ... 2019-12-05T05:00:50.654 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-cmd2 (0.7.9-0ubuntu1) ... 2019-12-05T05:00:50.716 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking liblttng-ust0:amd64 (2.10.1-1) ... 2019-12-05T05:00:51.024 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2019-12-05T05:00:51.038 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../004-liblttng-ust0_2.10.1-1_amd64.deb ... 2019-12-05T05:00:51.065 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../005-librbd1_15.0.0-8014-gc529080-1bionic_amd64.deb ... 2019-12-05T05:00:51.135 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking liblttng-ust0:amd64 (2.10.1-1) ... 2019-12-05T05:00:51.166 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking librbd1 (15.0.0-8014-gc529080-1bionic) over (12.2.12-0ubuntu0.18.04.3) ... 2019-12-05T05:00:51.508 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../005-librbd1_15.0.0-8014-gc529080-1bionic_amd64.deb ... 2019-12-05T05:00:51.604 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking librbd1 (15.0.0-8014-gc529080-1bionic) over (12.2.12-0ubuntu0.18.04.3) ... 2019-12-05T05:00:51.676 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../006-librados2_15.0.0-8014-gc529080-1bionic_amd64.deb ... 2019-12-05T05:00:51.779 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking librados2 (15.0.0-8014-gc529080-1bionic) over (12.2.12-0ubuntu0.18.04.3) ... 2019-12-05T05:00:52.103 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../006-librados2_15.0.0-8014-gc529080-1bionic_amd64.deb ... 2019-12-05T05:00:52.254 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking librados2 (15.0.0-8014-gc529080-1bionic) over (12.2.12-0ubuntu0.18.04.3) ... 2019-12-05T05:00:52.324 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package libcephfs2. 2019-12-05T05:00:52.340 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../007-libcephfs2_15.0.0-8014-gc529080-1bionic_amd64.deb ... 2019-12-05T05:00:52.389 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking libcephfs2 (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T05:00:52.658 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python-rados. 2019-12-05T05:00:52.680 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../008-python-rados_15.0.0-8014-gc529080-1bionic_amd64.deb ... 2019-12-05T05:00:52.708 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python-rados (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T05:00:52.826 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package libcephfs2. 2019-12-05T05:00:52.842 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../007-libcephfs2_15.0.0-8014-gc529080-1bionic_amd64.deb ... 2019-12-05T05:00:52.868 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking libcephfs2 (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T05:00:53.011 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python-ceph-argparse. 2019-12-05T05:00:53.027 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../009-python-ceph-argparse_15.0.0-8014-gc529080-1bionic_all.deb ... 2019-12-05T05:00:53.069 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python-ceph-argparse (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T05:00:53.177 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python-rados. 2019-12-05T05:00:53.191 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../008-python-rados_15.0.0-8014-gc529080-1bionic_amd64.deb ... 2019-12-05T05:00:53.252 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python-rados (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T05:00:53.357 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python-cephfs. 2019-12-05T05:00:53.372 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../010-python-cephfs_15.0.0-8014-gc529080-1bionic_amd64.deb ... 2019-12-05T05:00:53.406 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python-cephfs (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T05:00:53.535 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python-ceph-argparse. 2019-12-05T05:00:53.550 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../009-python-ceph-argparse_15.0.0-8014-gc529080-1bionic_all.deb ... 2019-12-05T05:00:53.595 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python-ceph-argparse (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T05:00:53.717 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python-ceph-common. 2019-12-05T05:00:53.732 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../011-python-ceph-common_15.0.0-8014-gc529080-1bionic_all.deb ... 2019-12-05T05:00:53.768 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python-ceph-common (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T05:00:53.921 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python-cephfs. 2019-12-05T05:00:53.935 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../010-python-cephfs_15.0.0-8014-gc529080-1bionic_amd64.deb ... 2019-12-05T05:00:53.971 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python-cephfs (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T05:00:54.083 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python-prettytable. 2019-12-05T05:00:54.097 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../012-python-prettytable_0.7.2-3_all.deb ... 2019-12-05T05:00:54.138 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python-prettytable (0.7.2-3) ... 2019-12-05T05:00:54.276 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python-ceph-common. 2019-12-05T05:00:54.286 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../011-python-ceph-common_15.0.0-8014-gc529080-1bionic_all.deb ... 2019-12-05T05:00:54.316 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python-ceph-common (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T05:00:54.344 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python-rbd. 2019-12-05T05:00:54.359 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../013-python-rbd_15.0.0-8014-gc529080-1bionic_amd64.deb ... 2019-12-05T05:00:54.394 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python-rbd (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T05:00:54.602 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python-prettytable. 2019-12-05T05:00:54.615 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../012-python-prettytable_0.7.2-3_all.deb ... 2019-12-05T05:00:54.644 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python-prettytable (0.7.2-3) ... 2019-12-05T05:00:54.756 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package librabbitmq4:amd64. 2019-12-05T05:00:54.769 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../014-librabbitmq4_0.8.0-1ubuntu0.18.04.1_amd64.deb ... 2019-12-05T05:00:54.823 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking librabbitmq4:amd64 (0.8.0-1ubuntu0.18.04.1) ... 2019-12-05T05:00:54.876 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python-rbd. 2019-12-05T05:00:54.889 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../013-python-rbd_15.0.0-8014-gc529080-1bionic_amd64.deb ... 2019-12-05T05:00:54.920 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python-rbd (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T05:00:55.139 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package librdkafka1:amd64. 2019-12-05T05:00:55.153 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../015-librdkafka1_0.11.3-1build1_amd64.deb ... 2019-12-05T05:00:55.197 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking librdkafka1:amd64 (0.11.3-1build1) ... 2019-12-05T05:00:55.229 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package librabbitmq4:amd64. 2019-12-05T05:00:55.243 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../014-librabbitmq4_0.8.0-1ubuntu0.18.04.1_amd64.deb ... 2019-12-05T05:00:55.287 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking librabbitmq4:amd64 (0.8.0-1ubuntu0.18.04.1) ... 2019-12-05T05:00:55.484 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package librgw2. 2019-12-05T05:00:55.497 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../016-librgw2_15.0.0-8014-gc529080-1bionic_amd64.deb ... 2019-12-05T05:00:55.526 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking librgw2 (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T05:00:55.606 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package librdkafka1:amd64. 2019-12-05T05:00:55.622 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../015-librdkafka1_0.11.3-1build1_amd64.deb ... 2019-12-05T05:00:55.665 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking librdkafka1:amd64 (0.11.3-1build1) ... 2019-12-05T05:00:55.993 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package librgw2. 2019-12-05T05:00:56.008 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../016-librgw2_15.0.0-8014-gc529080-1bionic_amd64.deb ... 2019-12-05T05:00:56.047 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking librgw2 (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T05:00:56.128 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python-rgw. 2019-12-05T05:00:56.142 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../017-python-rgw_15.0.0-8014-gc529080-1bionic_amd64.deb ... 2019-12-05T05:00:56.178 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python-rgw (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T05:00:56.489 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package libsnappy1v5:amd64. 2019-12-05T05:00:56.506 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../018-libsnappy1v5_1.1.7-1_amd64.deb ... 2019-12-05T05:00:56.539 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking libsnappy1v5:amd64 (1.1.7-1) ... 2019-12-05T05:00:56.669 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python-rgw. 2019-12-05T05:00:56.683 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../017-python-rgw_15.0.0-8014-gc529080-1bionic_amd64.deb ... 2019-12-05T05:00:56.716 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python-rgw (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T05:00:56.839 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package libleveldb1v5:amd64. 2019-12-05T05:00:56.853 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../019-libleveldb1v5_1.20-2_amd64.deb ... 2019-12-05T05:00:56.899 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking libleveldb1v5:amd64 (1.20-2) ... 2019-12-05T05:00:57.011 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package libsnappy1v5:amd64. 2019-12-05T05:00:57.027 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../018-libsnappy1v5_1.1.7-1_amd64.deb ... 2019-12-05T05:00:57.061 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking libsnappy1v5:amd64 (1.1.7-1) ... 2019-12-05T05:00:57.193 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package liboath0. 2019-12-05T05:00:57.211 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../020-liboath0_2.6.1-1_amd64.deb ... 2019-12-05T05:00:57.234 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking liboath0 (2.6.1-1) ... 2019-12-05T05:00:57.357 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package libleveldb1v5:amd64. 2019-12-05T05:00:57.368 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../019-libleveldb1v5_1.20-2_amd64.deb ... 2019-12-05T05:00:57.405 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking libleveldb1v5:amd64 (1.20-2) ... 2019-12-05T05:00:57.487 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package libradosstriper1. 2019-12-05T05:00:57.501 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../021-libradosstriper1_15.0.0-8014-gc529080-1bionic_amd64.deb ... 2019-12-05T05:00:57.538 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking libradosstriper1 (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T05:00:57.747 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package liboath0. 2019-12-05T05:00:57.761 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../020-liboath0_2.6.1-1_amd64.deb ... 2019-12-05T05:00:57.805 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking liboath0 (2.6.1-1) ... 2019-12-05T05:00:57.950 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package ceph-common. 2019-12-05T05:00:57.961 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../022-ceph-common_15.0.0-8014-gc529080-1bionic_amd64.deb ... 2019-12-05T05:00:58.005 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking ceph-common (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T05:00:58.124 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package libradosstriper1. 2019-12-05T05:00:58.139 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../021-libradosstriper1_15.0.0-8014-gc529080-1bionic_amd64.deb ... 2019-12-05T05:00:58.183 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking libradosstriper1 (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T05:00:58.553 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package ceph-common. 2019-12-05T05:00:58.566 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../022-ceph-common_15.0.0-8014-gc529080-1bionic_amd64.deb ... 2019-12-05T05:00:58.594 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking ceph-common (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T05:00:59.611 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package ceph-base. 2019-12-05T05:00:59.625 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../023-ceph-base_15.0.0-8014-gc529080-1bionic_amd64.deb ... 2019-12-05T05:00:59.691 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking ceph-base (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T05:01:00.239 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package ceph-base. 2019-12-05T05:01:00.253 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../023-ceph-base_15.0.0-8014-gc529080-1bionic_amd64.deb ... 2019-12-05T05:01:00.307 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking ceph-base (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T05:01:00.539 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python-bcrypt. 2019-12-05T05:01:00.553 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../024-python-bcrypt_3.1.4-2_amd64.deb ... 2019-12-05T05:01:00.590 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python-bcrypt (3.1.4-2) ... 2019-12-05T05:01:00.918 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python-cherrypy3. 2019-12-05T05:01:00.938 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../025-python-cherrypy3_8.9.1-2_all.deb ... 2019-12-05T05:01:00.984 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python-cherrypy3 (8.9.1-2) ... 2019-12-05T05:01:01.191 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python-bcrypt. 2019-12-05T05:01:01.207 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../024-python-bcrypt_3.1.4-2_amd64.deb ... 2019-12-05T05:01:01.250 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python-bcrypt (3.1.4-2) ... 2019-12-05T05:01:01.327 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python-jwt. 2019-12-05T05:01:01.341 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../026-python-jwt_1.5.3+ds1-1_all.deb ... 2019-12-05T05:01:01.385 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python-jwt (1.5.3+ds1-1) ... 2019-12-05T05:01:01.569 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python-cherrypy3. 2019-12-05T05:01:01.585 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../025-python-cherrypy3_8.9.1-2_all.deb ... 2019-12-05T05:01:01.619 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python-cherrypy3 (8.9.1-2) ... 2019-12-05T05:01:01.722 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python-openssl. 2019-12-05T05:01:01.736 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../027-python-openssl_17.5.0-1ubuntu1_all.deb ... 2019-12-05T05:01:01.789 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python-openssl (17.5.0-1ubuntu1) ... 2019-12-05T05:01:02.006 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python-jwt. 2019-12-05T05:01:02.019 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../026-python-jwt_1.5.3+ds1-1_all.deb ... 2019-12-05T05:01:02.056 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python-jwt (1.5.3+ds1-1) ... 2019-12-05T05:01:02.117 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python-logutils. 2019-12-05T05:01:02.128 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../028-python-logutils_0.3.3-5_all.deb ... 2019-12-05T05:01:02.181 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python-logutils (0.3.3-5) ... 2019-12-05T05:01:02.381 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python-openssl. 2019-12-05T05:01:02.395 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../027-python-openssl_17.5.0-1ubuntu1_all.deb ... 2019-12-05T05:01:02.463 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python-openssl (17.5.0-1ubuntu1) ... 2019-12-05T05:01:02.501 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python-markupsafe. 2019-12-05T05:01:02.517 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../029-python-markupsafe_1.0-1build1_amd64.deb ... 2019-12-05T05:01:02.559 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python-markupsafe (1.0-1build1) ... 2019-12-05T05:01:02.769 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python-logutils. 2019-12-05T05:01:02.787 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../028-python-logutils_0.3.3-5_all.deb ... 2019-12-05T05:01:02.825 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python-logutils (0.3.3-5) ... 2019-12-05T05:01:02.854 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python-mako. 2019-12-05T05:01:02.869 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../030-python-mako_1.0.7+ds1-1_all.deb ... 2019-12-05T05:01:02.912 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python-mako (1.0.7+ds1-1) ... 2019-12-05T05:01:03.153 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python-markupsafe. 2019-12-05T05:01:03.167 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../029-python-markupsafe_1.0-1build1_amd64.deb ... 2019-12-05T05:01:03.204 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python-markupsafe (1.0-1build1) ... 2019-12-05T05:01:03.288 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python-simplegeneric. 2019-12-05T05:01:03.305 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../031-python-simplegeneric_0.8.1-1_all.deb ... 2019-12-05T05:01:03.352 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python-simplegeneric (0.8.1-1) ... 2019-12-05T05:01:03.497 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python-mako. 2019-12-05T05:01:03.513 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../030-python-mako_1.0.7+ds1-1_all.deb ... 2019-12-05T05:01:03.555 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python-mako (1.0.7+ds1-1) ... 2019-12-05T05:01:03.633 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python-singledispatch. 2019-12-05T05:01:03.648 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../032-python-singledispatch_3.4.0.3-2_all.deb ... 2019-12-05T05:01:03.712 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python-singledispatch (3.4.0.3-2) ... 2019-12-05T05:01:03.942 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python-simplegeneric. 2019-12-05T05:01:03.951 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../031-python-simplegeneric_0.8.1-1_all.deb ... 2019-12-05T05:01:04.017 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python-simplegeneric (0.8.1-1) ... 2019-12-05T05:01:04.044 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python-simplejson. 2019-12-05T05:01:04.058 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../033-python-simplejson_3.13.2-1_amd64.deb ... 2019-12-05T05:01:04.094 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python-simplejson (3.13.2-1) ... 2019-12-05T05:01:04.303 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python-singledispatch. 2019-12-05T05:01:04.323 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../032-python-singledispatch_3.4.0.3-2_all.deb ... 2019-12-05T05:01:04.365 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python-singledispatch (3.4.0.3-2) ... 2019-12-05T05:01:04.470 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python-webob. 2019-12-05T05:01:04.486 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../034-python-webob_1%3a1.7.3-2fakesync1_all.deb ... 2019-12-05T05:01:04.537 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python-webob (1:1.7.3-2fakesync1) ... 2019-12-05T05:01:04.730 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python-simplejson. 2019-12-05T05:01:04.730 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../033-python-simplejson_3.13.2-1_amd64.deb ... 2019-12-05T05:01:04.769 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python-simplejson (3.13.2-1) ... 2019-12-05T05:01:04.848 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python-bs4. 2019-12-05T05:01:04.864 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../035-python-bs4_4.6.0-1_all.deb ... 2019-12-05T05:01:04.916 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python-bs4 (4.6.0-1) ... 2019-12-05T05:01:05.116 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python-webob. 2019-12-05T05:01:05.130 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../034-python-webob_1%3a1.7.3-2fakesync1_all.deb ... 2019-12-05T05:01:05.172 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python-webob (1:1.7.3-2fakesync1) ... 2019-12-05T05:01:05.236 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python-waitress. 2019-12-05T05:01:05.249 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../036-python-waitress_1.0.1-1_all.deb ... 2019-12-05T05:01:05.293 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python-waitress (1.0.1-1) ... 2019-12-05T05:01:05.475 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python-bs4. 2019-12-05T05:01:05.489 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../035-python-bs4_4.6.0-1_all.deb ... 2019-12-05T05:01:05.567 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python-bs4 (4.6.0-1) ... 2019-12-05T05:01:05.613 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python-dnspython. 2019-12-05T05:01:05.629 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../037-python-dnspython_1.15.0-1_all.deb ... 2019-12-05T05:01:05.673 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python-dnspython (1.15.0-1) ... 2019-12-05T05:01:05.891 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python-waitress. 2019-12-05T05:01:05.906 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../036-python-waitress_1.0.1-1_all.deb ... 2019-12-05T05:01:05.950 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python-waitress (1.0.1-1) ... 2019-12-05T05:01:05.983 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package formencode-i18n. 2019-12-05T05:01:05.997 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../038-formencode-i18n_1.3.0-0ubuntu5_all.deb ... 2019-12-05T05:01:06.033 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking formencode-i18n (1.3.0-0ubuntu5) ... 2019-12-05T05:01:06.303 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python-dnspython. 2019-12-05T05:01:06.316 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../037-python-dnspython_1.15.0-1_all.deb ... 2019-12-05T05:01:06.327 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python-formencode. 2019-12-05T05:01:06.341 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../039-python-formencode_1.3.0-0ubuntu5_all.deb ... 2019-12-05T05:01:06.357 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python-dnspython (1.15.0-1) ... 2019-12-05T05:01:06.377 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python-formencode (1.3.0-0ubuntu5) ... 2019-12-05T05:01:06.670 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package formencode-i18n. 2019-12-05T05:01:06.685 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../038-formencode-i18n_1.3.0-0ubuntu5_all.deb ... 2019-12-05T05:01:06.714 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python-tempita. 2019-12-05T05:01:06.726 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../040-python-tempita_0.5.2-2_all.deb ... 2019-12-05T05:01:06.728 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking formencode-i18n (1.3.0-0ubuntu5) ... 2019-12-05T05:01:06.770 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python-tempita (0.5.2-2) ... 2019-12-05T05:01:07.039 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python-formencode. 2019-12-05T05:01:07.053 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../039-python-formencode_1.3.0-0ubuntu5_all.deb ... 2019-12-05T05:01:07.106 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python-formencode (1.3.0-0ubuntu5) ... 2019-12-05T05:01:07.231 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python-paste. 2019-12-05T05:01:07.245 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../041-python-paste_2.0.3+dfsg-4ubuntu1_all.deb ... 2019-12-05T05:01:07.289 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python-paste (2.0.3+dfsg-4ubuntu1) ... 2019-12-05T05:01:07.442 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python-tempita. 2019-12-05T05:01:07.455 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../040-python-tempita_0.5.2-2_all.deb ... 2019-12-05T05:01:07.508 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python-tempita (0.5.2-2) ... 2019-12-05T05:01:07.609 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2019-12-05T05:01:07.624 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../042-python-pastedeploy-tpl_1.5.2-4_all.deb ... 2019-12-05T05:01:07.668 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python-pastedeploy-tpl (1.5.2-4) ... 2019-12-05T05:01:07.808 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python-paste. 2019-12-05T05:01:07.828 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../041-python-paste_2.0.3+dfsg-4ubuntu1_all.deb ... 2019-12-05T05:01:07.866 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python-paste (2.0.3+dfsg-4ubuntu1) ... 2019-12-05T05:01:07.977 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python-pastedeploy. 2019-12-05T05:01:07.992 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../043-python-pastedeploy_1.5.2-4_all.deb ... 2019-12-05T05:01:08.027 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python-pastedeploy (1.5.2-4) ... 2019-12-05T05:01:08.203 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2019-12-05T05:01:08.216 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../042-python-pastedeploy-tpl_1.5.2-4_all.deb ... 2019-12-05T05:01:08.253 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python-pastedeploy-tpl (1.5.2-4) ... 2019-12-05T05:01:08.455 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python-webtest. 2019-12-05T05:01:08.470 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../044-python-webtest_2.0.28-1ubuntu1_all.deb ... 2019-12-05T05:01:08.522 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python-webtest (2.0.28-1ubuntu1) ... 2019-12-05T05:01:08.547 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python-pastedeploy. 2019-12-05T05:01:08.564 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../043-python-pastedeploy_1.5.2-4_all.deb ... 2019-12-05T05:01:08.630 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python-pastedeploy (1.5.2-4) ... 2019-12-05T05:01:08.865 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python-pecan. 2019-12-05T05:01:08.881 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../045-python-pecan_1.2.1-2_all.deb ... 2019-12-05T05:01:08.926 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python-pecan (1.2.1-2) ... 2019-12-05T05:01:08.980 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python-webtest. 2019-12-05T05:01:08.995 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../044-python-webtest_2.0.28-1ubuntu1_all.deb ... 2019-12-05T05:01:09.047 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python-webtest (2.0.28-1ubuntu1) ... 2019-12-05T05:01:09.321 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python-certifi. 2019-12-05T05:01:09.337 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../046-python-certifi_2018.1.18-2_all.deb ... 2019-12-05T05:01:09.382 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python-certifi (2018.1.18-2) ... 2019-12-05T05:01:09.442 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python-pecan. 2019-12-05T05:01:09.455 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../045-python-pecan_1.2.1-2_all.deb ... 2019-12-05T05:01:09.495 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python-pecan (1.2.1-2) ... 2019-12-05T05:01:09.673 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python-chardet. 2019-12-05T05:01:09.689 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../047-python-chardet_3.0.4-1_all.deb ... 2019-12-05T05:01:09.731 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python-chardet (3.0.4-1) ... 2019-12-05T05:01:09.894 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python-certifi. 2019-12-05T05:01:09.907 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../046-python-certifi_2018.1.18-2_all.deb ... 2019-12-05T05:01:09.965 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python-certifi (2018.1.18-2) ... 2019-12-05T05:01:10.091 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python-urllib3. 2019-12-05T05:01:10.105 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../048-python-urllib3_1.22-1ubuntu0.18.04.1_all.deb ... 2019-12-05T05:01:10.149 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python-urllib3 (1.22-1ubuntu0.18.04.1) ... 2019-12-05T05:01:10.304 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python-chardet. 2019-12-05T05:01:10.320 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../047-python-chardet_3.0.4-1_all.deb ... 2019-12-05T05:01:10.362 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python-chardet (3.0.4-1) ... 2019-12-05T05:01:10.475 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python-requests. 2019-12-05T05:01:10.492 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../049-python-requests_2.18.4-2ubuntu0.1_all.deb ... 2019-12-05T05:01:10.542 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python-requests (2.18.4-2ubuntu0.1) ... 2019-12-05T05:01:10.716 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python-urllib3. 2019-12-05T05:01:10.731 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../048-python-urllib3_1.22-1ubuntu0.18.04.1_all.deb ... 2019-12-05T05:01:10.878 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python-werkzeug. 2019-12-05T05:01:10.894 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../050-python-werkzeug_0.14.1+dfsg1-1_all.deb ... 2019-12-05T05:01:10.928 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python-werkzeug (0.14.1+dfsg1-1) ... 2019-12-05T05:01:11.254 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package ceph-mgr. 2019-12-05T05:01:11.270 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../051-ceph-mgr_15.0.0-8014-gc529080-1bionic_amd64.deb ... 2019-12-05T05:01:11.312 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking ceph-mgr (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T05:01:11.499 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python-urllib3 (1.22-1ubuntu0.18.04.1) ... 2019-12-05T05:01:11.815 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package ceph-mon. 2019-12-05T05:01:11.829 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../052-ceph-mon_15.0.0-8014-gc529080-1bionic_amd64.deb ... 2019-12-05T05:01:11.867 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python-requests. 2019-12-05T05:01:11.873 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking ceph-mon (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T05:01:11.882 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../049-python-requests_2.18.4-2ubuntu0.1_all.deb ... 2019-12-05T05:01:11.925 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python-requests (2.18.4-2ubuntu0.1) ... 2019-12-05T05:01:12.269 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python-werkzeug. 2019-12-05T05:01:12.289 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../050-python-werkzeug_0.14.1+dfsg1-1_all.deb ... 2019-12-05T05:01:12.319 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python-werkzeug (0.14.1+dfsg1-1) ... 2019-12-05T05:01:12.686 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package ceph-mgr. 2019-12-05T05:01:12.702 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../051-ceph-mgr_15.0.0-8014-gc529080-1bionic_amd64.deb ... 2019-12-05T05:01:12.768 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package ceph-osd. 2019-12-05T05:01:12.769 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking ceph-mgr (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T05:01:12.785 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../053-ceph-osd_15.0.0-8014-gc529080-1bionic_amd64.deb ... 2019-12-05T05:01:12.818 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking ceph-osd (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T05:01:13.259 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package ceph-mon. 2019-12-05T05:01:13.271 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../052-ceph-mon_15.0.0-8014-gc529080-1bionic_amd64.deb ... 2019-12-05T05:01:13.317 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking ceph-mon (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T05:01:14.225 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package ceph-osd. 2019-12-05T05:01:14.239 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../053-ceph-osd_15.0.0-8014-gc529080-1bionic_amd64.deb ... 2019-12-05T05:01:14.314 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking ceph-osd (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T05:01:14.956 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package ceph. 2019-12-05T05:01:14.972 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../054-ceph_15.0.0-8014-gc529080-1bionic_amd64.deb ... 2019-12-05T05:01:15.014 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking ceph (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T05:01:15.353 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package ceph-daemon. 2019-12-05T05:01:15.367 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../055-ceph-daemon_15.0.0-8014-gc529080-1bionic_amd64.deb ... 2019-12-05T05:01:15.417 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking ceph-daemon (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T05:01:15.766 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package ceph-fuse. 2019-12-05T05:01:15.782 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../056-ceph-fuse_15.0.0-8014-gc529080-1bionic_amd64.deb ... 2019-12-05T05:01:15.841 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking ceph-fuse (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T05:01:16.254 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package ceph-mds. 2019-12-05T05:01:16.269 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../057-ceph-mds_15.0.0-8014-gc529080-1bionic_amd64.deb ... 2019-12-05T05:01:16.316 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking ceph-mds (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T05:01:16.611 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package ceph. 2019-12-05T05:01:16.625 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../054-ceph_15.0.0-8014-gc529080-1bionic_amd64.deb ... 2019-12-05T05:01:16.668 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking ceph (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T05:01:16.871 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python-repoze.lru. 2019-12-05T05:01:16.885 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../058-python-repoze.lru_0.7-1_all.deb ... 2019-12-05T05:01:16.932 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python-repoze.lru (0.7-1) ... 2019-12-05T05:01:17.014 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package ceph-daemon. 2019-12-05T05:01:17.029 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../055-ceph-daemon_15.0.0-8014-gc529080-1bionic_amd64.deb ... 2019-12-05T05:01:17.073 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking ceph-daemon (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T05:01:17.177 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package libjs-underscore. 2019-12-05T05:01:17.191 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../059-libjs-underscore_1.8.3~dfsg-1_all.deb ... 2019-12-05T05:01:17.234 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking libjs-underscore (1.8.3~dfsg-1) ... 2019-12-05T05:01:17.443 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package ceph-fuse. 2019-12-05T05:01:17.459 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../056-ceph-fuse_15.0.0-8014-gc529080-1bionic_amd64.deb ... 2019-12-05T05:01:17.507 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking ceph-fuse (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T05:01:17.517 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package libjs-sphinxdoc. 2019-12-05T05:01:17.532 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../060-libjs-sphinxdoc_1.6.7-1ubuntu1_all.deb ... 2019-12-05T05:01:17.575 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking libjs-sphinxdoc (1.6.7-1ubuntu1) ... 2019-12-05T05:01:17.934 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python-routes. 2019-12-05T05:01:17.948 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../061-python-routes_2.4.1-1_all.deb ... 2019-12-05T05:01:17.956 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package ceph-mds. 2019-12-05T05:01:17.970 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../057-ceph-mds_15.0.0-8014-gc529080-1bionic_amd64.deb ... 2019-12-05T05:01:18.009 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python-routes (2.4.1-1) ... 2019-12-05T05:01:18.019 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking ceph-mds (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T05:01:18.371 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2019-12-05T05:01:18.387 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../062-ceph-mgr-dashboard_15.0.0-8014-gc529080-1bionic_all.deb ... 2019-12-05T05:01:18.431 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking ceph-mgr-dashboard (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T05:01:18.519 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python-repoze.lru. 2019-12-05T05:01:18.534 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../058-python-repoze.lru_0.7-1_all.deb ... 2019-12-05T05:01:18.574 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python-repoze.lru (0.7-1) ... 2019-12-05T05:01:18.834 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package libjs-underscore. 2019-12-05T05:01:18.851 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../059-libjs-underscore_1.8.3~dfsg-1_all.deb ... 2019-12-05T05:01:18.905 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking libjs-underscore (1.8.3~dfsg-1) ... 2019-12-05T05:01:19.217 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package libjs-sphinxdoc. 2019-12-05T05:01:19.231 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../060-libjs-sphinxdoc_1.6.7-1ubuntu1_all.deb ... 2019-12-05T05:01:19.284 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking libjs-sphinxdoc (1.6.7-1ubuntu1) ... 2019-12-05T05:01:19.389 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package ceph-mgr-diskprediction-cloud. 2019-12-05T05:01:19.405 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../063-ceph-mgr-diskprediction-cloud_15.0.0-8014-gc529080-1bionic_all.deb ... 2019-12-05T05:01:19.449 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking ceph-mgr-diskprediction-cloud (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T05:01:19.627 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python-routes. 2019-12-05T05:01:19.642 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../061-python-routes_2.4.1-1_all.deb ... 2019-12-05T05:01:19.694 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python-routes (2.4.1-1) ... 2019-12-05T05:01:19.769 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python-decorator. 2019-12-05T05:01:19.781 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../064-python-decorator_4.1.2-1_all.deb ... 2019-12-05T05:01:19.837 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python-decorator (4.1.2-1) ... 2019-12-05T05:01:20.078 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2019-12-05T05:01:20.094 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../062-ceph-mgr-dashboard_15.0.0-8014-gc529080-1bionic_all.deb ... 2019-12-05T05:01:20.152 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking ceph-mgr-dashboard (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T05:01:20.163 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python-scipy. 2019-12-05T05:01:20.182 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../065-python-scipy_0.19.1-2ubuntu1_amd64.deb ... 2019-12-05T05:01:20.221 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python-scipy (0.19.1-2ubuntu1) ... 2019-12-05T05:01:21.203 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package ceph-mgr-diskprediction-cloud. 2019-12-05T05:01:21.219 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../063-ceph-mgr-diskprediction-cloud_15.0.0-8014-gc529080-1bionic_all.deb ... 2019-12-05T05:01:21.266 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking ceph-mgr-diskprediction-cloud (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T05:01:21.377 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python-sklearn-lib. 2019-12-05T05:01:21.393 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../066-python-sklearn-lib_0.19.1-3_amd64.deb ... 2019-12-05T05:01:21.443 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python-sklearn-lib (0.19.1-3) ... 2019-12-05T05:01:21.631 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python-decorator. 2019-12-05T05:01:21.644 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../064-python-decorator_4.1.2-1_all.deb ... 2019-12-05T05:01:21.684 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python-decorator (4.1.2-1) ... 2019-12-05T05:01:21.941 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python-joblib. 2019-12-05T05:01:21.956 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../067-python-joblib_0.11-1_all.deb ... 2019-12-05T05:01:21.999 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python-joblib (0.11-1) ... 2019-12-05T05:01:22.024 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python-scipy. 2019-12-05T05:01:22.040 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../065-python-scipy_0.19.1-2ubuntu1_amd64.deb ... 2019-12-05T05:01:22.083 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python-scipy (0.19.1-2ubuntu1) ... 2019-12-05T05:01:22.386 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python-sklearn. 2019-12-05T05:01:22.403 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../068-python-sklearn_0.19.1-3_all.deb ... 2019-12-05T05:01:22.462 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python-sklearn (0.19.1-3) ... 2019-12-05T05:01:23.017 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2019-12-05T05:01:23.032 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../069-ceph-mgr-diskprediction-local_15.0.0-8014-gc529080-1bionic_all.deb ... 2019-12-05T05:01:23.074 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking ceph-mgr-diskprediction-local (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T05:01:23.267 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python-sklearn-lib. 2019-12-05T05:01:23.282 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../066-python-sklearn-lib_0.19.1-3_amd64.deb ... 2019-12-05T05:01:23.328 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python-sklearn-lib (0.19.1-3) ... 2019-12-05T05:01:23.836 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python-joblib. 2019-12-05T05:01:23.851 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../067-python-joblib_0.11-1_all.deb ... 2019-12-05T05:01:23.895 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python-joblib (0.11-1) ... 2019-12-05T05:01:24.298 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python-monotonic. 2019-12-05T05:01:24.310 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python-sklearn. 2019-12-05T05:01:24.312 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../070-python-monotonic_1.1-2_all.deb ... 2019-12-05T05:01:24.326 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../068-python-sklearn_0.19.1-3_all.deb ... 2019-12-05T05:01:24.366 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python-monotonic (1.1-2) ... 2019-12-05T05:01:24.368 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python-sklearn (0.19.1-3) ... 2019-12-05T05:01:24.701 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python-fasteners. 2019-12-05T05:01:24.716 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../071-python-fasteners_0.12.0-3_all.deb ... 2019-12-05T05:01:24.767 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python-fasteners (0.12.0-3) ... 2019-12-05T05:01:24.962 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2019-12-05T05:01:24.975 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../069-ceph-mgr-diskprediction-local_15.0.0-8014-gc529080-1bionic_all.deb ... 2019-12-05T05:01:25.046 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking ceph-mgr-diskprediction-local (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T05:01:25.062 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python-httplib2. 2019-12-05T05:01:25.078 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../072-python-httplib2_0.9.2+dfsg-1ubuntu0.1_all.deb ... 2019-12-05T05:01:25.120 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python-httplib2 (0.9.2+dfsg-1ubuntu0.1) ... 2019-12-05T05:01:25.471 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python-pyasn1. 2019-12-05T05:01:25.487 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../073-python-pyasn1_0.4.2-3_all.deb ... 2019-12-05T05:01:25.530 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python-pyasn1 (0.4.2-3) ... 2019-12-05T05:01:25.901 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python-pyasn1-modules. 2019-12-05T05:01:25.918 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../074-python-pyasn1-modules_0.2.1-0.2_all.deb ... 2019-12-05T05:01:25.960 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python-pyasn1-modules (0.2.1-0.2) ... 2019-12-05T05:01:26.221 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python-monotonic. 2019-12-05T05:01:26.234 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../070-python-monotonic_1.1-2_all.deb ... 2019-12-05T05:01:26.272 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python-monotonic (1.1-2) ... 2019-12-05T05:01:26.351 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python-rsa. 2019-12-05T05:01:26.366 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../075-python-rsa_3.4.2-1_all.deb ... 2019-12-05T05:01:26.410 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python-rsa (3.4.2-1) ... 2019-12-05T05:01:26.642 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python-fasteners. 2019-12-05T05:01:26.660 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../071-python-fasteners_0.12.0-3_all.deb ... 2019-12-05T05:01:26.692 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python-fasteners (0.12.0-3) ... 2019-12-05T05:01:26.716 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python-sqlalchemy. 2019-12-05T05:01:26.732 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../076-python-sqlalchemy_1.1.11+ds1-1ubuntu1_all.deb ... 2019-12-05T05:01:26.774 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python-sqlalchemy (1.1.11+ds1-1ubuntu1) ... 2019-12-05T05:01:27.009 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python-httplib2. 2019-12-05T05:01:27.036 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../072-python-httplib2_0.9.2+dfsg-1ubuntu0.1_all.deb ... 2019-12-05T05:01:27.093 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python-httplib2 (0.9.2+dfsg-1ubuntu0.1) ... 2019-12-05T05:01:27.250 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python-oauth2client. 2019-12-05T05:01:27.269 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../077-python-oauth2client_4.1.2-3_all.deb ... 2019-12-05T05:01:27.314 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python-oauth2client (4.1.2-3) ... 2019-12-05T05:01:27.416 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python-pyasn1. 2019-12-05T05:01:27.428 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../073-python-pyasn1_0.4.2-3_all.deb ... 2019-12-05T05:01:27.464 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python-pyasn1 (0.4.2-3) ... 2019-12-05T05:01:27.701 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python-websocket. 2019-12-05T05:01:27.713 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../078-python-websocket_0.44.0-0ubuntu2_all.deb ... 2019-12-05T05:01:27.757 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python-websocket (0.44.0-0ubuntu2) ... 2019-12-05T05:01:27.840 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python-pyasn1-modules. 2019-12-05T05:01:27.855 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../074-python-pyasn1-modules_0.2.1-0.2_all.deb ... 2019-12-05T05:01:27.900 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python-pyasn1-modules (0.2.1-0.2) ... 2019-12-05T05:01:28.108 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python-yaml. 2019-12-05T05:01:28.122 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../079-python-yaml_3.12-1build2_amd64.deb ... 2019-12-05T05:01:28.173 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python-yaml (3.12-1build2) ... 2019-12-05T05:01:28.236 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python-rsa. 2019-12-05T05:01:28.252 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../075-python-rsa_3.4.2-1_all.deb ... 2019-12-05T05:01:28.294 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python-rsa (3.4.2-1) ... 2019-12-05T05:01:28.538 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python-kubernetes. 2019-12-05T05:01:28.552 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../080-python-kubernetes_2.0.0-2ubuntu1_all.deb ... 2019-12-05T05:01:28.646 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python-sqlalchemy. 2019-12-05T05:01:28.659 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../076-python-sqlalchemy_1.1.11+ds1-1ubuntu1_all.deb ... 2019-12-05T05:01:28.693 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python-kubernetes (2.0.0-2ubuntu1) ... 2019-12-05T05:01:28.704 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python-sqlalchemy (1.1.11+ds1-1ubuntu1) ... 2019-12-05T05:01:29.127 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2019-12-05T05:01:29.144 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../081-ceph-mgr-k8sevents_15.0.0-8014-gc529080-1bionic_all.deb ... 2019-12-05T05:01:29.187 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking ceph-mgr-k8sevents (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T05:01:29.225 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python-oauth2client. 2019-12-05T05:01:29.241 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../077-python-oauth2client_4.1.2-3_all.deb ... 2019-12-05T05:01:29.291 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python-oauth2client (4.1.2-3) ... 2019-12-05T05:01:29.498 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package ceph-mgr-rook. 2019-12-05T05:01:29.526 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../082-ceph-mgr-rook_15.0.0-8014-gc529080-1bionic_all.deb ... 2019-12-05T05:01:29.558 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking ceph-mgr-rook (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T05:01:29.679 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python-websocket. 2019-12-05T05:01:29.694 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../078-python-websocket_0.44.0-0ubuntu2_all.deb ... 2019-12-05T05:01:29.739 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python-websocket (0.44.0-0ubuntu2) ... 2019-12-05T05:01:29.881 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package ceph-mgr-ssh. 2019-12-05T05:01:29.897 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../083-ceph-mgr-ssh_15.0.0-8014-gc529080-1bionic_all.deb ... 2019-12-05T05:01:29.940 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking ceph-mgr-ssh (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T05:01:30.058 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python-yaml. 2019-12-05T05:01:30.074 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../079-python-yaml_3.12-1build2_amd64.deb ... 2019-12-05T05:01:30.149 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python-yaml (3.12-1build2) ... 2019-12-05T05:01:30.302 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package libonig4:amd64. 2019-12-05T05:01:30.323 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../084-libonig4_6.7.0-1_amd64.deb ... 2019-12-05T05:01:30.371 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking libonig4:amd64 (6.7.0-1) ... 2019-12-05T05:01:30.527 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python-kubernetes. 2019-12-05T05:01:30.544 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../080-python-kubernetes_2.0.0-2ubuntu1_all.deb ... 2019-12-05T05:01:30.701 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python-kubernetes (2.0.0-2ubuntu1) ... 2019-12-05T05:01:30.748 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package libjq1:amd64. 2019-12-05T05:01:30.764 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../085-libjq1_1.5+dfsg-2_amd64.deb ... 2019-12-05T05:01:30.806 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking libjq1:amd64 (1.5+dfsg-2) ... 2019-12-05T05:01:31.085 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package jq. 2019-12-05T05:01:31.100 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../086-jq_1.5+dfsg-2_amd64.deb ... 2019-12-05T05:01:31.115 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2019-12-05T05:01:31.129 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../081-ceph-mgr-k8sevents_15.0.0-8014-gc529080-1bionic_all.deb ... 2019-12-05T05:01:31.143 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking jq (1.5+dfsg-2) ... 2019-12-05T05:01:31.200 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking ceph-mgr-k8sevents (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T05:01:31.393 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package socat. 2019-12-05T05:01:31.406 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../087-socat_1.7.3.2-2ubuntu2_amd64.deb ... 2019-12-05T05:01:31.455 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking socat (1.7.3.2-2ubuntu2) ... 2019-12-05T05:01:31.520 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package ceph-mgr-rook. 2019-12-05T05:01:31.535 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../082-ceph-mgr-rook_15.0.0-8014-gc529080-1bionic_all.deb ... 2019-12-05T05:01:31.580 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking ceph-mgr-rook (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T05:01:31.873 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package libxslt1.1:amd64. 2019-12-05T05:01:31.901 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../088-libxslt1.1_1.1.29-5ubuntu0.2_amd64.deb ... 2019-12-05T05:01:31.913 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package ceph-mgr-ssh. 2019-12-05T05:01:31.930 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../083-ceph-mgr-ssh_15.0.0-8014-gc529080-1bionic_all.deb ... 2019-12-05T05:01:31.936 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking libxslt1.1:amd64 (1.1.29-5ubuntu0.2) ... 2019-12-05T05:01:31.973 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking ceph-mgr-ssh (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T05:01:32.266 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package xmlstarlet. 2019-12-05T05:01:32.281 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../089-xmlstarlet_1.6.1-2_amd64.deb ... 2019-12-05T05:01:32.324 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking xmlstarlet (1.6.1-2) ... 2019-12-05T05:01:32.326 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package libonig4:amd64. 2019-12-05T05:01:32.340 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../084-libonig4_6.7.0-1_amd64.deb ... 2019-12-05T05:01:32.393 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking libonig4:amd64 (6.7.0-1) ... 2019-12-05T05:01:32.698 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package ceph-test. 2019-12-05T05:01:32.713 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../090-ceph-test_15.0.0-8014-gc529080-1bionic_amd64.deb ... 2019-12-05T05:01:32.757 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking ceph-test (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T05:01:32.787 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package libjq1:amd64. 2019-12-05T05:01:32.802 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../085-libjq1_1.5+dfsg-2_amd64.deb ... 2019-12-05T05:01:32.851 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking libjq1:amd64 (1.5+dfsg-2) ... 2019-12-05T05:01:33.133 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package jq. 2019-12-05T05:01:33.150 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../086-jq_1.5+dfsg-2_amd64.deb ... 2019-12-05T05:01:33.210 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking jq (1.5+dfsg-2) ... 2019-12-05T05:01:33.492 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package socat. 2019-12-05T05:01:33.506 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../087-socat_1.7.3.2-2ubuntu2_amd64.deb ... 2019-12-05T05:01:33.552 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking socat (1.7.3.2-2ubuntu2) ... 2019-12-05T05:01:33.957 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package libxslt1.1:amd64. 2019-12-05T05:01:33.973 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../088-libxslt1.1_1.1.29-5ubuntu0.2_amd64.deb ... 2019-12-05T05:01:34.018 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking libxslt1.1:amd64 (1.1.29-5ubuntu0.2) ... 2019-12-05T05:01:34.353 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package xmlstarlet. 2019-12-05T05:01:34.367 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../089-xmlstarlet_1.6.1-2_amd64.deb ... 2019-12-05T05:01:34.420 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking xmlstarlet (1.6.1-2) ... 2019-12-05T05:01:34.814 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package ceph-test. 2019-12-05T05:01:34.831 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../090-ceph-test_15.0.0-8014-gc529080-1bionic_amd64.deb ... 2019-12-05T05:01:34.873 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking ceph-test (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T05:01:35.623 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-rados. 2019-12-05T05:01:35.637 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../091-python3-rados_15.0.0-8014-gc529080-1bionic_amd64.deb ... 2019-12-05T05:01:35.675 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-rados (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T05:01:36.042 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-ceph-argparse. 2019-12-05T05:01:36.057 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../092-python3-ceph-argparse_15.0.0-8014-gc529080-1bionic_all.deb ... 2019-12-05T05:01:36.104 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-ceph-argparse (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T05:01:36.444 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-cephfs. 2019-12-05T05:01:36.460 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../093-python3-cephfs_15.0.0-8014-gc529080-1bionic_amd64.deb ... 2019-12-05T05:01:36.512 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-cephfs (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T05:01:36.899 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-colorama. 2019-12-05T05:01:36.913 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../094-python3-colorama_0.3.7-1_all.deb ... 2019-12-05T05:01:36.951 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-colorama (0.3.7-1) ... 2019-12-05T05:01:37.211 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package cephfs-shell. 2019-12-05T05:01:37.227 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../095-cephfs-shell_15.0.0-8014-gc529080-1bionic_all.deb ... 2019-12-05T05:01:37.329 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking cephfs-shell (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T05:01:37.588 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package libcephfs-dev. 2019-12-05T05:01:37.605 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../096-libcephfs-dev_15.0.0-8014-gc529080-1bionic_amd64.deb ... 2019-12-05T05:01:37.636 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-rados. 2019-12-05T05:01:37.646 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking libcephfs-dev (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T05:01:37.650 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../091-python3-rados_15.0.0-8014-gc529080-1bionic_amd64.deb ... 2019-12-05T05:01:37.688 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-rados (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T05:01:37.980 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python-attr. 2019-12-05T05:01:37.996 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../097-python-attr_17.4.0-2_all.deb ... 2019-12-05T05:01:38.048 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python-attr (17.4.0-2) ... 2019-12-05T05:01:38.082 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-ceph-argparse. 2019-12-05T05:01:38.097 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../092-python3-ceph-argparse_15.0.0-8014-gc529080-1bionic_all.deb ... 2019-12-05T05:01:38.141 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-ceph-argparse (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T05:01:38.346 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python-ceph. 2019-12-05T05:01:38.359 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../098-python-ceph_15.0.0-8014-gc529080-1bionic_amd64.deb ... 2019-12-05T05:01:38.400 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python-ceph (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T05:01:38.482 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-cephfs. 2019-12-05T05:01:38.498 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../093-python3-cephfs_15.0.0-8014-gc529080-1bionic_amd64.deb ... 2019-12-05T05:01:38.541 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-cephfs (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T05:01:38.702 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python-funcsigs. 2019-12-05T05:01:38.720 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../099-python-funcsigs_1.0.2-4_all.deb ... 2019-12-05T05:01:38.761 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python-funcsigs (1.0.2-4) ... 2019-12-05T05:01:38.893 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-colorama. 2019-12-05T05:01:38.907 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../094-python3-colorama_0.3.7-1_all.deb ... 2019-12-05T05:01:38.945 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-colorama (0.3.7-1) ... 2019-12-05T05:01:39.084 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python-webencodings. 2019-12-05T05:01:39.099 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../100-python-webencodings_0.5-2_all.deb ... 2019-12-05T05:01:39.141 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python-webencodings (0.5-2) ... 2019-12-05T05:01:39.240 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package cephfs-shell. 2019-12-05T05:01:39.256 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../095-cephfs-shell_15.0.0-8014-gc529080-1bionic_all.deb ... 2019-12-05T05:01:39.298 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking cephfs-shell (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T05:01:39.467 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python-html5lib. 2019-12-05T05:01:39.481 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../101-python-html5lib_0.999999999-1_all.deb ... 2019-12-05T05:01:39.526 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python-html5lib (0.999999999-1) ... 2019-12-05T05:01:39.582 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package libcephfs-dev. 2019-12-05T05:01:39.598 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../096-libcephfs-dev_15.0.0-8014-gc529080-1bionic_amd64.deb ... 2019-12-05T05:01:39.641 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking libcephfs-dev (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T05:01:39.862 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python-lxml:amd64. 2019-12-05T05:01:39.878 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../102-python-lxml_4.2.1-1ubuntu0.1_amd64.deb ... 2019-12-05T05:01:39.923 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python-lxml:amd64 (4.2.1-1ubuntu0.1) ... 2019-12-05T05:01:39.986 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python-attr. 2019-12-05T05:01:40.005 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../097-python-attr_17.4.0-2_all.deb ... 2019-12-05T05:01:40.053 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python-attr (17.4.0-2) ... 2019-12-05T05:01:40.302 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python-ceph. 2019-12-05T05:01:40.317 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../098-python-ceph_15.0.0-8014-gc529080-1bionic_amd64.deb ... 2019-12-05T05:01:40.395 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python-ceph (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T05:01:40.401 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python-openid. 2019-12-05T05:01:40.416 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../103-python-openid_2.2.5-6_all.deb ... 2019-12-05T05:01:40.459 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python-openid (2.2.5-6) ... 2019-12-05T05:01:40.679 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python-funcsigs. 2019-12-05T05:01:40.692 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../099-python-funcsigs_1.0.2-4_all.deb ... 2019-12-05T05:01:40.737 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python-funcsigs (1.0.2-4) ... 2019-12-05T05:01:40.788 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python-pastescript. 2019-12-05T05:01:40.805 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../104-python-pastescript_2.0.2-2_all.deb ... 2019-12-05T05:01:40.846 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python-pastescript (2.0.2-2) ... 2019-12-05T05:01:41.090 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python-webencodings. 2019-12-05T05:01:41.106 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../100-python-webencodings_0.5-2_all.deb ... 2019-12-05T05:01:41.149 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python-webencodings (0.5-2) ... 2019-12-05T05:01:41.214 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python-pluggy. 2019-12-05T05:01:41.231 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../105-python-pluggy_0.6.0-1_all.deb ... 2019-12-05T05:01:41.267 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python-pluggy (0.6.0-1) ... 2019-12-05T05:01:41.477 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python-html5lib. 2019-12-05T05:01:41.491 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../101-python-html5lib_0.999999999-1_all.deb ... 2019-12-05T05:01:41.530 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python-html5lib (0.999999999-1) ... 2019-12-05T05:01:41.575 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python-py. 2019-12-05T05:01:41.591 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../106-python-py_1.5.2-1_all.deb ... 2019-12-05T05:01:41.633 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python-py (1.5.2-1) ... 2019-12-05T05:01:41.874 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python-lxml:amd64. 2019-12-05T05:01:41.887 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../102-python-lxml_4.2.1-1ubuntu0.1_amd64.deb ... 2019-12-05T05:01:41.957 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python-lxml:amd64 (4.2.1-1ubuntu0.1) ... 2019-12-05T05:01:41.982 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python-pyinotify. 2019-12-05T05:01:41.997 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../107-python-pyinotify_0.9.6-1_all.deb ... 2019-12-05T05:01:42.042 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python-pyinotify (0.9.6-1) ... 2019-12-05T05:01:42.377 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python-pytest. 2019-12-05T05:01:42.392 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../108-python-pytest_3.3.2-2_all.deb ... 2019-12-05T05:01:42.402 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python-openid. 2019-12-05T05:01:42.419 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../103-python-openid_2.2.5-6_all.deb ... 2019-12-05T05:01:42.437 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python-pytest (3.3.2-2) ... 2019-12-05T05:01:42.472 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python-openid (2.2.5-6) ... 2019-12-05T05:01:42.790 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python-scgi. 2019-12-05T05:01:42.808 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python-pastescript. 2019-12-05T05:01:42.808 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../109-python-scgi_1.13-1.1build1_amd64.deb ... 2019-12-05T05:01:42.819 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../104-python-pastescript_2.0.2-2_all.deb ... 2019-12-05T05:01:42.856 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python-scgi (1.13-1.1build1) ... 2019-12-05T05:01:42.865 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python-pastescript (2.0.2-2) ... 2019-12-05T05:01:43.133 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python-sqlalchemy-ext. 2019-12-05T05:01:43.150 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../110-python-sqlalchemy-ext_1.1.11+ds1-1ubuntu1_amd64.deb ... 2019-12-05T05:01:43.197 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python-sqlalchemy-ext (1.1.11+ds1-1ubuntu1) ... 2019-12-05T05:01:43.269 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python-pluggy. 2019-12-05T05:01:43.283 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../105-python-pluggy_0.6.0-1_all.deb ... 2019-12-05T05:01:43.327 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python-pluggy (0.6.0-1) ... 2019-12-05T05:01:43.558 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package radosgw. 2019-12-05T05:01:43.575 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../111-radosgw_15.0.0-8014-gc529080-1bionic_amd64.deb ... 2019-12-05T05:01:43.618 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking radosgw (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T05:01:43.660 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python-py. 2019-12-05T05:01:43.674 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../106-python-py_1.5.2-1_all.deb ... 2019-12-05T05:01:43.720 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python-py (1.5.2-1) ... 2019-12-05T05:01:44.079 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python-pyinotify. 2019-12-05T05:01:44.093 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../107-python-pyinotify_0.9.6-1_all.deb ... 2019-12-05T05:01:44.137 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python-pyinotify (0.9.6-1) ... 2019-12-05T05:01:44.354 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package rbd-fuse. 2019-12-05T05:01:44.369 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../112-rbd-fuse_15.0.0-8014-gc529080-1bionic_amd64.deb ... 2019-12-05T05:01:44.413 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking rbd-fuse (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T05:01:44.493 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python-pytest. 2019-12-05T05:01:44.503 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../108-python-pytest_3.3.2-2_all.deb ... 2019-12-05T05:01:44.544 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python-pytest (3.3.2-2) ... 2019-12-05T05:01:44.743 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python-dnspython (1.15.0-1) ... 2019-12-05T05:01:44.920 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python-scgi. 2019-12-05T05:01:44.935 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../109-python-scgi_1.13-1.1build1_amd64.deb ... 2019-12-05T05:01:45.005 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python-scgi (1.13-1.1build1) ... 2019-12-05T05:01:45.050 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python-simplejson (3.13.2-1) ... 2019-12-05T05:01:45.245 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python-sqlalchemy-ext. 2019-12-05T05:01:45.259 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../110-python-sqlalchemy-ext_1.1.11+ds1-1ubuntu1_amd64.deb ... 2019-12-05T05:01:45.263 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python-repoze.lru (0.7-1) ... 2019-12-05T05:01:45.312 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python-sqlalchemy-ext (1.1.11+ds1-1ubuntu1) ... 2019-12-05T05:01:45.484 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up librdkafka1:amd64 (0.11.3-1build1) ... 2019-12-05T05:01:45.618 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python-sqlalchemy (1.1.11+ds1-1ubuntu1) ... 2019-12-05T05:01:45.721 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package radosgw. 2019-12-05T05:01:45.735 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../111-radosgw_15.0.0-8014-gc529080-1bionic_amd64.deb ... 2019-12-05T05:01:45.785 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking radosgw (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T05:01:46.107 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python-openid (2.2.5-6) ... 2019-12-05T05:01:46.346 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python-prettytable (0.7.2-3) ... 2019-12-05T05:01:46.481 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package rbd-fuse. 2019-12-05T05:01:46.496 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../112-rbd-fuse_15.0.0-8014-gc529080-1bionic_amd64.deb ... 2019-12-05T05:01:46.568 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python-urllib3 (1.22-1ubuntu0.18.04.1) ... 2019-12-05T05:01:46.780 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking rbd-fuse (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T05:01:46.800 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python-yaml (3.12-1build2) ... 2019-12-05T05:01:47.010 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python-simplegeneric (0.8.1-1) ... 2019-12-05T05:01:47.210 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python-chardet (3.0.4-1) ... 2019-12-05T05:01:47.333 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python-dnspython (1.15.0-1) ... 2019-12-05T05:01:47.483 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python-attr (17.4.0-2) ... 2019-12-05T05:01:47.615 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python-simplejson (3.13.2-1) ... 2019-12-05T05:01:47.697 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up libjs-underscore (1.8.3~dfsg-1) ... 2019-12-05T05:01:47.834 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python-openssl (17.5.0-1ubuntu1) ... 2019-12-05T05:01:47.840 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python-repoze.lru (0.7-1) ... 2019-12-05T05:01:48.046 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up librdkafka1:amd64 (0.11.3-1build1) ... 2019-12-05T05:01:48.047 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-ceph-argparse (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T05:01:48.210 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python-sqlalchemy (1.1.11+ds1-1ubuntu1) ... 2019-12-05T05:01:48.343 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up socat (1.7.3.2-2ubuntu2) ... 2019-12-05T05:01:48.470 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python-pyasn1 (0.4.2-3) ... 2019-12-05T05:01:48.698 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python-openid (2.2.5-6) ... 2019-12-05T05:01:48.703 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python-py (1.5.2-1) ... 2019-12-05T05:01:48.919 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python-prettytable (0.7.2-3) ... 2019-12-05T05:01:48.951 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python-monotonic (1.1-2) ... 2019-12-05T05:01:49.139 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python-urllib3 (1.22-1ubuntu0.18.04.1) ... 2019-12-05T05:01:49.147 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python-joblib (0.11-1) ... 2019-12-05T05:01:49.369 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python-yaml (3.12-1build2) ... 2019-12-05T05:01:49.403 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up libonig4:amd64 (6.7.0-1) ... 2019-12-05T05:01:49.530 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python-pluggy (0.6.0-1) ... 2019-12-05T05:01:49.582 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python-simplegeneric (0.8.1-1) ... 2019-12-05T05:01:49.743 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python-certifi (2018.1.18-2) ... 2019-12-05T05:01:49.802 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python-chardet (3.0.4-1) ... 2019-12-05T05:01:49.938 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-colorama (0.3.7-1) ... 2019-12-05T05:01:50.098 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python-attr (17.4.0-2) ... 2019-12-05T05:01:50.236 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-pyparsing (2.2.0+dfsg1-2) ... 2019-12-05T05:01:50.318 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up libjs-underscore (1.8.3~dfsg-1) ... 2019-12-05T05:01:50.449 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python-openssl (17.5.0-1ubuntu1) ... 2019-12-05T05:01:50.559 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up formencode-i18n (1.3.0-0ubuntu5) ... 2019-12-05T05:01:50.691 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-ceph-argparse (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T05:01:50.711 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up libjs-sphinxdoc (1.6.7-1ubuntu1) ... 2019-12-05T05:01:50.840 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python-webob (1:1.7.3-2fakesync1) ... 2019-12-05T05:01:50.988 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up socat (1.7.3.2-2ubuntu2) ... 2019-12-05T05:01:51.087 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python-ceph-argparse (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T05:01:51.132 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python-pyasn1 (0.4.2-3) ... 2019-12-05T05:01:51.217 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python-waitress (1.0.1-1) ... 2019-12-05T05:01:51.382 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python-py (1.5.2-1) ... 2019-12-05T05:01:51.537 INFO:teuthology.orchestra.run.smithi042.stdout:update-alternatives: using /usr/bin/waitress-serve-python2 to provide /usr/bin/waitress-serve (waitress-serve) in auto mode 2019-12-05T05:01:51.617 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python-monotonic (1.1-2) ... 2019-12-05T05:01:51.719 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python-pyasn1-modules (0.2.1-0.2) ... 2019-12-05T05:01:51.839 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python-joblib (0.11-1) ... 2019-12-05T05:01:51.973 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python-fasteners (0.12.0-3) ... 2019-12-05T05:01:52.112 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up libonig4:amd64 (6.7.0-1) ... 2019-12-05T05:01:52.184 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python-jwt (1.5.3+ds1-1) ... 2019-12-05T05:01:52.266 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python-pluggy (0.6.0-1) ... 2019-12-05T05:01:52.386 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python-markupsafe (1.0-1build1) ... 2019-12-05T05:01:52.480 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python-certifi (2018.1.18-2) ... 2019-12-05T05:01:52.589 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python-werkzeug (0.14.1+dfsg1-1) ... 2019-12-05T05:01:52.702 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-colorama (0.3.7-1) ... 2019-12-05T05:01:52.870 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python-pyinotify (0.9.6-1) ... 2019-12-05T05:01:52.991 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-pyparsing (2.2.0+dfsg1-2) ... 2019-12-05T05:01:53.055 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python-scgi (1.13-1.1build1) ... 2019-12-05T05:01:53.249 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python-websocket (0.44.0-0ubuntu2) ... 2019-12-05T05:01:53.297 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up formencode-i18n (1.3.0-0ubuntu5) ... 2019-12-05T05:01:53.337 INFO:teuthology.orchestra.run.smithi042.stdout:update-alternatives: using /usr/bin/python2-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2019-12-05T05:01:53.426 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up libjs-sphinxdoc (1.6.7-1ubuntu1) ... 2019-12-05T05:01:53.500 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up libxslt1.1:amd64 (1.1.29-5ubuntu0.2) ... 2019-12-05T05:01:53.566 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python-webob (1:1.7.3-2fakesync1) ... 2019-12-05T05:01:53.636 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python-httplib2 (0.9.2+dfsg-1ubuntu0.1) ... 2019-12-05T05:01:53.803 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python-ceph-argparse (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T05:01:53.849 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up libjq1:amd64 (1.5+dfsg-2) ... 2019-12-05T05:01:53.942 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python-waitress (1.0.1-1) ... 2019-12-05T05:01:53.986 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python-ceph-common (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T05:01:54.103 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python-bs4 (4.6.0-1) ... 2019-12-05T05:01:54.321 INFO:teuthology.orchestra.run.smithi040.stdout:update-alternatives: using /usr/bin/waitress-serve-python2 to provide /usr/bin/waitress-serve (waitress-serve) in auto mode 2019-12-05T05:01:54.363 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python-mako (1.0.7+ds1-1) ... 2019-12-05T05:01:54.535 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python-pyasn1-modules (0.2.1-0.2) ... 2019-12-05T05:01:54.618 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python-sklearn-lib (0.19.1-3) ... 2019-12-05T05:01:54.744 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up libsnappy1v5:amd64 (1.1.7-1) ... 2019-12-05T05:01:54.789 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python-fasteners (0.12.0-3) ... 2019-12-05T05:01:54.870 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python-lxml:amd64 (4.2.1-1ubuntu0.1) ... 2019-12-05T05:01:55.002 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python-jwt (1.5.3+ds1-1) ... 2019-12-05T05:01:55.092 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python-sqlalchemy-ext (1.1.11+ds1-1ubuntu1) ... 2019-12-05T05:01:55.222 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up ceph-daemon (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T05:01:55.223 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python-markupsafe (1.0-1build1) ... 2019-12-05T05:01:55.461 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python-werkzeug (0.14.1+dfsg1-1) ... 2019-12-05T05:01:55.741 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python-pyinotify (0.9.6-1) ... 2019-12-05T05:01:55.907 INFO:teuthology.orchestra.run.smithi042.stdout:Adding system user cephdaemon....done 2019-12-05T05:01:55.962 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python-scgi (1.13-1.1build1) ... 2019-12-05T05:01:56.084 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python-funcsigs (1.0.2-4) ... 2019-12-05T05:01:56.182 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python-websocket (0.44.0-0ubuntu2) ... 2019-12-05T05:01:56.287 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python-requests (2.18.4-2ubuntu0.1) ... 2019-12-05T05:01:56.306 INFO:teuthology.orchestra.run.smithi040.stdout:update-alternatives: using /usr/bin/python2-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2019-12-05T05:01:56.465 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up libxslt1.1:amd64 (1.1.29-5ubuntu0.2) ... 2019-12-05T05:01:56.513 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python-uritemplate (0.6-1ubuntu1) ... 2019-12-05T05:01:56.603 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python-httplib2 (0.9.2+dfsg-1ubuntu0.1) ... 2019-12-05T05:01:56.715 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python-decorator (4.1.2-1) ... 2019-12-05T05:01:56.823 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up libjq1:amd64 (1.5+dfsg-2) ... 2019-12-05T05:01:56.914 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up liboath0 (2.6.1-1) ... 2019-12-05T05:01:56.950 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python-ceph-common (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T05:01:57.085 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python-singledispatch (3.4.0.3-2) ... 2019-12-05T05:01:57.089 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python-bs4 (4.6.0-1) ... 2019-12-05T05:01:57.309 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up librabbitmq4:amd64 (0.8.0-1ubuntu0.18.04.1) ... 2019-12-05T05:01:57.341 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python-mako (1.0.7+ds1-1) ... 2019-12-05T05:01:57.428 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python-pastedeploy-tpl (1.5.2-4) ... 2019-12-05T05:01:57.554 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python-tempita (0.5.2-2) ... 2019-12-05T05:01:57.573 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python-sklearn-lib (0.19.1-3) ... 2019-12-05T05:01:57.691 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up libsnappy1v5:amd64 (1.1.7-1) ... 2019-12-05T05:01:57.756 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up liblttng-ust0:amd64 (2.10.1-1) ... 2019-12-05T05:01:57.857 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python-lxml:amd64 (4.2.1-1ubuntu0.1) ... 2019-12-05T05:01:57.901 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python-webencodings (0.5-2) ... 2019-12-05T05:01:58.075 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python-sqlalchemy-ext (1.1.11+ds1-1ubuntu1) ... 2019-12-05T05:01:58.111 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python-cherrypy3 (8.9.1-2) ... 2019-12-05T05:01:58.200 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up ceph-daemon (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T05:01:58.399 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python-bcrypt (3.1.4-2) ... 2019-12-05T05:01:58.599 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python-logutils (0.3.3-5) ... 2019-12-05T05:01:58.787 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-pyperclip (1.6.0-1) ... 2019-12-05T05:01:58.944 INFO:teuthology.orchestra.run.smithi040.stdout:Adding system user cephdaemon....done 2019-12-05T05:01:59.085 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python-routes (2.4.1-1) ... 2019-12-05T05:01:59.116 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python-funcsigs (1.0.2-4) ... 2019-12-05T05:01:59.283 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python-formencode (1.3.0-0ubuntu5) ... 2019-12-05T05:01:59.311 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python-requests (2.18.4-2ubuntu0.1) ... 2019-12-05T05:01:59.538 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up jq (1.5+dfsg-2) ... 2019-12-05T05:01:59.548 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python-uritemplate (0.6-1ubuntu1) ... 2019-12-05T05:01:59.667 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python-scipy (0.19.1-2ubuntu1) ... 2019-12-05T05:01:59.785 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python-decorator (4.1.2-1) ... 2019-12-05T05:02:00.002 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up liboath0 (2.6.1-1) ... 2019-12-05T05:02:00.188 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python-singledispatch (3.4.0.3-2) ... 2019-12-05T05:02:00.430 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up librabbitmq4:amd64 (0.8.0-1ubuntu0.18.04.1) ... 2019-12-05T05:02:00.573 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python-pastedeploy-tpl (1.5.2-4) ... 2019-12-05T05:02:00.699 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python-tempita (0.5.2-2) ... 2019-12-05T05:02:00.724 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python-rsa (3.4.2-1) ... 2019-12-05T05:02:00.923 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up liblttng-ust0:amd64 (2.10.1-1) ... 2019-12-05T05:02:00.928 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-cmd2 (0.7.9-0ubuntu1) ... 2019-12-05T05:02:01.057 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python-webencodings (0.5-2) ... 2019-12-05T05:02:01.243 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python-paste (2.0.3+dfsg-4ubuntu1) ... 2019-12-05T05:02:01.253 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python-cherrypy3 (8.9.1-2) ... 2019-12-05T05:02:01.540 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python-bcrypt (3.1.4-2) ... 2019-12-05T05:02:01.561 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python-pytest (3.3.2-2) ... 2019-12-05T05:02:01.753 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python-logutils (0.3.3-5) ... 2019-12-05T05:02:01.815 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python-oauth2client (4.1.2-3) ... 2019-12-05T05:02:01.977 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-pyperclip (1.6.0-1) ... 2019-12-05T05:02:02.025 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up libleveldb1v5:amd64 (1.20-2) ... 2019-12-05T05:02:02.151 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up xmlstarlet (1.6.1-2) ... 2019-12-05T05:02:02.296 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python-routes (2.4.1-1) ... 2019-12-05T05:02:02.296 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up librados2 (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T05:02:02.438 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up libcephfs2 (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T05:02:02.493 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python-formencode (1.3.0-0ubuntu5) ... 2019-12-05T05:02:02.564 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python-sklearn (0.19.1-3) ... 2019-12-05T05:02:02.727 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up jq (1.5+dfsg-2) ... 2019-12-05T05:02:02.863 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python-scipy (0.19.1-2ubuntu1) ... 2019-12-05T05:02:03.281 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up libcephfs-dev (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T05:02:03.412 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-rados (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T05:02:03.538 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python-kubernetes (2.0.0-2ubuntu1) ... 2019-12-05T05:02:03.949 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python-rsa (3.4.2-1) ... 2019-12-05T05:02:04.085 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python-html5lib (0.999999999-1) ... 2019-12-05T05:02:04.195 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-cmd2 (0.7.9-0ubuntu1) ... 2019-12-05T05:02:04.350 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python-rados (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T05:02:04.514 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python-cephfs (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T05:02:04.522 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python-paste (2.0.3+dfsg-4ubuntu1) ... 2019-12-05T05:02:04.824 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up libradosstriper1 (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T05:02:04.830 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python-pytest (3.3.2-2) ... 2019-12-05T05:02:04.950 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-cephfs (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T05:02:05.075 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python-oauth2client (4.1.2-3) ... 2019-12-05T05:02:05.235 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up librgw2 (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T05:02:05.296 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up libleveldb1v5:amd64 (1.20-2) ... 2019-12-05T05:02:05.394 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python-rgw (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T05:02:05.421 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up xmlstarlet (1.6.1-2) ... 2019-12-05T05:02:05.529 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python-pastedeploy (1.5.2-4) ... 2019-12-05T05:02:05.567 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up librados2 (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T05:02:05.722 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up libcephfs2 (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T05:02:05.722 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up ceph-fuse (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T05:02:05.843 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python-sklearn (0.19.1-3) ... 2019-12-05T05:02:05.944 INFO:teuthology.orchestra.run.smithi042.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2019-12-05T05:02:05.944 INFO:teuthology.orchestra.run.smithi042.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2019-12-05T05:02:06.219 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up librbd1 (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T05:02:06.341 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up cephfs-shell (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T05:02:06.467 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python-pastescript (2.0.2-2) ... 2019-12-05T05:02:06.575 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up libcephfs-dev (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T05:02:06.706 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-rados (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T05:02:06.712 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python-rbd (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T05:02:06.825 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python-kubernetes (2.0.0-2ubuntu1) ... 2019-12-05T05:02:06.853 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up ceph-common (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T05:02:07.370 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python-html5lib (0.999999999-1) ... 2019-12-05T05:02:07.395 INFO:teuthology.orchestra.run.smithi042.stdout:Adding group ceph....done 2019-12-05T05:02:07.632 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python-rados (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T05:02:07.766 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python-cephfs (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T05:02:07.812 INFO:teuthology.orchestra.run.smithi042.stdout:Adding system user ceph....done 2019-12-05T05:02:08.005 INFO:teuthology.orchestra.run.smithi042.stdout:Setting system user ceph properties....done 2019-12-05T05:02:08.049 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up libradosstriper1 (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T05:02:08.173 INFO:teuthology.orchestra.run.smithi042.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2019-12-05T05:02:08.226 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-cephfs (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T05:02:08.394 INFO:teuthology.orchestra.run.smithi042.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2019-12-05T05:02:08.557 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up librgw2 (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T05:02:08.661 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up radosgw (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T05:02:08.717 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python-rgw (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T05:02:08.843 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python-pastedeploy (1.5.2-4) ... 2019-12-05T05:02:08.940 INFO:teuthology.orchestra.run.smithi042.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2019-12-05T05:02:08.941 INFO:teuthology.orchestra.run.smithi042.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2019-12-05T05:02:09.044 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up ceph-fuse (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T05:02:09.292 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2019-12-05T05:02:09.293 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2019-12-05T05:02:09.321 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up rbd-fuse (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T05:02:09.447 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python-ceph (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T05:02:09.573 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python-webtest (2.0.28-1ubuntu1) ... 2019-12-05T05:02:09.587 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up librbd1 (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T05:02:09.741 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up cephfs-shell (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T05:02:09.865 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python-pastescript (2.0.2-2) ... 2019-12-05T05:02:10.091 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python-rbd (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T05:02:10.252 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up ceph-common (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T05:02:10.283 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up ceph-test (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T05:02:10.409 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up ceph-base (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T05:02:10.776 INFO:teuthology.orchestra.run.smithi042.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2019-12-05T05:02:10.818 INFO:teuthology.orchestra.run.smithi040.stdout:Adding group ceph....done 2019-12-05T05:02:11.061 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python-pecan (1.2.1-2) ... 2019-12-05T05:02:11.159 INFO:teuthology.orchestra.run.smithi042.stdout:update-alternatives: using /usr/bin/python2-pecan to provide /usr/bin/pecan (pecan) in auto mode 2019-12-05T05:02:11.182 INFO:teuthology.orchestra.run.smithi042.stdout:update-alternatives: using /usr/bin/python2-gunicorn_pecan to provide /usr/bin/gunicorn_pecan (gunicorn_pecan) in auto mode 2019-12-05T05:02:11.260 INFO:teuthology.orchestra.run.smithi040.stdout:Adding system user ceph....done 2019-12-05T05:02:11.382 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up ceph-osd (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T05:02:11.454 INFO:teuthology.orchestra.run.smithi040.stdout:Setting system user ceph properties....done 2019-12-05T05:02:11.621 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2019-12-05T05:02:11.859 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2019-12-05T05:02:11.936 INFO:teuthology.orchestra.run.smithi042.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2019-12-05T05:02:11.936 INFO:teuthology.orchestra.run.smithi042.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2019-12-05T05:02:12.135 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up radosgw (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T05:02:12.191 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up ceph-mds (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T05:02:12.415 INFO:teuthology.orchestra.run.smithi042.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2019-12-05T05:02:12.415 INFO:teuthology.orchestra.run.smithi042.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2019-12-05T05:02:12.423 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2019-12-05T05:02:12.423 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2019-12-05T05:02:12.692 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up ceph-mon (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T05:02:12.838 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up rbd-fuse (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T05:02:12.916 INFO:teuthology.orchestra.run.smithi042.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2019-12-05T05:02:12.916 INFO:teuthology.orchestra.run.smithi042.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2019-12-05T05:02:12.963 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python-ceph (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T05:02:13.101 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python-webtest (2.0.28-1ubuntu1) ... 2019-12-05T05:02:13.169 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up ceph-mgr (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T05:02:13.299 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up ceph-test (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T05:02:13.397 INFO:teuthology.orchestra.run.smithi042.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2019-12-05T05:02:13.397 INFO:teuthology.orchestra.run.smithi042.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2019-12-05T05:02:13.430 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up ceph-base (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T05:02:13.679 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up ceph-mgr-diskprediction-local (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T05:02:13.758 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2019-12-05T05:02:13.821 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up ceph-mgr-diskprediction-cloud (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T05:02:13.966 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up ceph-mgr-rook (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T05:02:14.069 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python-pecan (1.2.1-2) ... 2019-12-05T05:02:14.115 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up ceph (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T05:02:14.159 INFO:teuthology.orchestra.run.smithi040.stdout:update-alternatives: using /usr/bin/python2-pecan to provide /usr/bin/pecan (pecan) in auto mode 2019-12-05T05:02:14.199 INFO:teuthology.orchestra.run.smithi040.stdout:update-alternatives: using /usr/bin/python2-gunicorn_pecan to provide /usr/bin/gunicorn_pecan (gunicorn_pecan) in auto mode 2019-12-05T05:02:14.243 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up ceph-mgr-k8sevents (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T05:02:14.368 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up ceph-mgr-ssh (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T05:02:14.389 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up ceph-osd (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T05:02:14.513 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up ceph-mgr-dashboard (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T05:02:14.670 INFO:teuthology.orchestra.run.smithi042.stdout:Processing triggers for libc-bin (2.27-3ubuntu1) ... 2019-12-05T05:02:14.806 INFO:teuthology.orchestra.run.smithi042.stdout:Processing triggers for systemd (237-3ubuntu10.31) ... 2019-12-05T05:02:14.969 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2019-12-05T05:02:14.969 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2019-12-05T05:02:15.005 INFO:teuthology.orchestra.run.smithi042.stdout:Processing triggers for man-db (2.8.3-2ubuntu0.1) ... 2019-12-05T05:02:15.291 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up ceph-mds (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T05:02:15.521 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2019-12-05T05:02:15.521 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2019-12-05T05:02:15.820 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up ceph-mon (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T05:02:16.037 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2019-12-05T05:02:16.037 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2019-12-05T05:02:16.344 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up ceph-mgr (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T05:02:16.566 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2019-12-05T05:02:16.566 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2019-12-05T05:02:16.684 INFO:teuthology.orchestra.run.smithi042.stdout:Processing triggers for ureadahead (0.100.0-21) ... 2019-12-05T05:02:16.728 INFO:teuthology.orchestra.run.smithi042.stdout:ureadahead will be reprofiled on next reboot 2019-12-05T05:02:16.871 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up ceph-mgr-diskprediction-local (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T05:02:17.014 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up ceph-mgr-diskprediction-cloud (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T05:02:17.103 INFO:teuthology.orchestra.run.smithi042.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2019-12-05T05:02:17.103 INFO:teuthology.orchestra.run.smithi042:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install bison flex libelf-dev libssl-dev 2019-12-05T05:02:17.166 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up ceph-mgr-rook (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T05:02:17.197 INFO:teuthology.orchestra.run.smithi042.stdout:Reading package lists... 2019-12-05T05:02:17.345 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up ceph (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T05:02:17.454 INFO:teuthology.orchestra.run.smithi042.stdout:Building dependency tree... 2019-12-05T05:02:17.454 INFO:teuthology.orchestra.run.smithi042.stdout:Reading state information... 2019-12-05T05:02:17.467 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up ceph-mgr-k8sevents (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T05:02:17.639 INFO:teuthology.orchestra.run.smithi042.stdout:The following additional packages will be installed: 2019-12-05T05:02:17.640 INFO:teuthology.orchestra.run.smithi042.stdout: libbison-dev libfl-dev libfl2 libssl1.1 zlib1g-dev 2019-12-05T05:02:17.642 INFO:teuthology.orchestra.run.smithi042.stdout:Suggested packages: 2019-12-05T05:02:17.642 INFO:teuthology.orchestra.run.smithi042.stdout: bison-doc flex-doc libssl-doc 2019-12-05T05:02:17.669 INFO:teuthology.orchestra.run.smithi042.stdout:The following NEW packages will be installed: 2019-12-05T05:02:17.670 INFO:teuthology.orchestra.run.smithi042.stdout: bison flex libbison-dev libelf-dev libfl-dev libfl2 libssl-dev zlib1g-dev 2019-12-05T05:02:17.672 INFO:teuthology.orchestra.run.smithi042.stdout:The following packages will be upgraded: 2019-12-05T05:02:17.672 INFO:teuthology.orchestra.run.smithi042.stdout: libssl1.1 2019-12-05T05:02:17.855 INFO:teuthology.orchestra.run.smithi042.stdout:1 upgraded, 8 newly installed, 0 to remove and 51 not upgraded. 2019-12-05T05:02:17.856 INFO:teuthology.orchestra.run.smithi042.stdout:Need to get 4,037 kB of archives. 2019-12-05T05:02:17.856 INFO:teuthology.orchestra.run.smithi042.stdout:After this operation, 11.6 MB of additional disk space will be used. 2019-12-05T05:02:17.856 INFO:teuthology.orchestra.run.smithi042.stdout:Get:1 http://archive.ubuntu.com/ubuntu bionic/main amd64 flex amd64 2.6.4-6 [316 kB] 2019-12-05T05:02:18.251 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up ceph-mgr-ssh (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T05:02:18.326 INFO:teuthology.orchestra.run.smithi042.stdout:Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libssl1.1 amd64 1.1.1-1ubuntu2.1~18.04.5 [1,300 kB] 2019-12-05T05:02:18.428 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up ceph-mgr-dashboard (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T05:02:18.533 INFO:teuthology.orchestra.run.smithi042.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic/main amd64 libbison-dev amd64 2:3.0.4.dfsg-1build1 [339 kB] 2019-12-05T05:02:18.560 INFO:teuthology.orchestra.run.smithi042.stdout:Get:4 http://archive.ubuntu.com/ubuntu bionic/main amd64 bison amd64 2:3.0.4.dfsg-1build1 [266 kB] 2019-12-05T05:02:18.571 INFO:teuthology.orchestra.run.smithi040.stdout:Processing triggers for libc-bin (2.27-3ubuntu1) ... 2019-12-05T05:02:18.576 INFO:teuthology.orchestra.run.smithi042.stdout:Get:5 http://archive.ubuntu.com/ubuntu bionic/main amd64 zlib1g-dev amd64 1:1.2.11.dfsg-0ubuntu2 [176 kB] 2019-12-05T05:02:18.585 INFO:teuthology.orchestra.run.smithi042.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libelf-dev amd64 0.170-0.4ubuntu0.1 [57.3 kB] 2019-12-05T05:02:18.588 INFO:teuthology.orchestra.run.smithi042.stdout:Get:7 http://archive.ubuntu.com/ubuntu bionic/main amd64 libfl2 amd64 2.6.4-6 [11.4 kB] 2019-12-05T05:02:18.589 INFO:teuthology.orchestra.run.smithi042.stdout:Get:8 http://archive.ubuntu.com/ubuntu bionic/main amd64 libfl-dev amd64 2.6.4-6 [6,320 B] 2019-12-05T05:02:18.589 INFO:teuthology.orchestra.run.smithi042.stdout:Get:9 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libssl-dev amd64 1.1.1-1ubuntu2.1~18.04.5 [1,566 kB] 2019-12-05T05:02:18.699 INFO:teuthology.orchestra.run.smithi040.stdout:Processing triggers for systemd (237-3ubuntu10.31) ... 2019-12-05T05:02:18.916 INFO:teuthology.orchestra.run.smithi040.stdout:Processing triggers for man-db (2.8.3-2ubuntu0.1) ... 2019-12-05T05:02:18.993 INFO:teuthology.orchestra.run.smithi042.stdout:Preconfiguring packages ... 2019-12-05T05:02:19.131 INFO:teuthology.orchestra.run.smithi042.stdout:Fetched 4,037 kB in 1s (4,157 kB/s) 2019-12-05T05:02:19.275 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package flex. 2019-12-05T05:02:19.324 INFO:teuthology.orchestra.run.smithi042.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 143715 files and directories currently installed.) 2019-12-05T05:02:19.329 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../0-flex_2.6.4-6_amd64.deb ... 2019-12-05T05:02:19.371 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking flex (2.6.4-6) ... 2019-12-05T05:02:19.943 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../1-libssl1.1_1.1.1-1ubuntu2.1~18.04.5_amd64.deb ... 2019-12-05T05:02:20.127 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking libssl1.1:amd64 (1.1.1-1ubuntu2.1~18.04.5) over (1.1.1-1ubuntu2.1~18.04.4) ... 2019-12-05T05:02:20.552 INFO:teuthology.orchestra.run.smithi040.stdout:Processing triggers for ureadahead (0.100.0-21) ... 2019-12-05T05:02:20.595 INFO:teuthology.orchestra.run.smithi040.stdout:ureadahead will be reprofiled on next reboot 2019-12-05T05:02:20.765 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package libbison-dev:amd64. 2019-12-05T05:02:20.780 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../2-libbison-dev_2%3a3.0.4.dfsg-1build1_amd64.deb ... 2019-12-05T05:02:20.823 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking libbison-dev:amd64 (2:3.0.4.dfsg-1build1) ... 2019-12-05T05:02:20.961 INFO:teuthology.orchestra.run.smithi040.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2019-12-05T05:02:20.963 INFO:teuthology.orchestra.run.smithi040:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install bison flex libelf-dev libssl-dev 2019-12-05T05:02:21.049 INFO:teuthology.orchestra.run.smithi040.stdout:Reading package lists... 2019-12-05T05:02:21.176 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package bison. 2019-12-05T05:02:21.190 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../3-bison_2%3a3.0.4.dfsg-1build1_amd64.deb ... 2019-12-05T05:02:21.243 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking bison (2:3.0.4.dfsg-1build1) ... 2019-12-05T05:02:21.301 INFO:teuthology.orchestra.run.smithi040.stdout:Building dependency tree... 2019-12-05T05:02:21.301 INFO:teuthology.orchestra.run.smithi040.stdout:Reading state information... 2019-12-05T05:02:21.484 INFO:teuthology.orchestra.run.smithi040.stdout:The following additional packages will be installed: 2019-12-05T05:02:21.485 INFO:teuthology.orchestra.run.smithi040.stdout: libbison-dev libfl-dev libfl2 libssl1.1 zlib1g-dev 2019-12-05T05:02:21.486 INFO:teuthology.orchestra.run.smithi040.stdout:Suggested packages: 2019-12-05T05:02:21.486 INFO:teuthology.orchestra.run.smithi040.stdout: bison-doc flex-doc libssl-doc 2019-12-05T05:02:21.513 INFO:teuthology.orchestra.run.smithi040.stdout:The following NEW packages will be installed: 2019-12-05T05:02:21.514 INFO:teuthology.orchestra.run.smithi040.stdout: bison flex libbison-dev libelf-dev libfl-dev libfl2 libssl-dev zlib1g-dev 2019-12-05T05:02:21.516 INFO:teuthology.orchestra.run.smithi040.stdout:The following packages will be upgraded: 2019-12-05T05:02:21.517 INFO:teuthology.orchestra.run.smithi040.stdout: libssl1.1 2019-12-05T05:02:21.538 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package zlib1g-dev:amd64. 2019-12-05T05:02:21.553 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../4-zlib1g-dev_1%3a1.2.11.dfsg-0ubuntu2_amd64.deb ... 2019-12-05T05:02:21.596 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking zlib1g-dev:amd64 (1:1.2.11.dfsg-0ubuntu2) ... 2019-12-05T05:02:21.698 INFO:teuthology.orchestra.run.smithi040.stdout:1 upgraded, 8 newly installed, 0 to remove and 51 not upgraded. 2019-12-05T05:02:21.698 INFO:teuthology.orchestra.run.smithi040.stdout:Need to get 4,037 kB of archives. 2019-12-05T05:02:21.699 INFO:teuthology.orchestra.run.smithi040.stdout:After this operation, 11.6 MB of additional disk space will be used. 2019-12-05T05:02:21.699 INFO:teuthology.orchestra.run.smithi040.stdout:Get:1 http://archive.ubuntu.com/ubuntu bionic/main amd64 flex amd64 2.6.4-6 [316 kB] 2019-12-05T05:02:21.908 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package libelf-dev:amd64. 2019-12-05T05:02:21.931 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../5-libelf-dev_0.170-0.4ubuntu0.1_amd64.deb ... 2019-12-05T05:02:21.974 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking libelf-dev:amd64 (0.170-0.4ubuntu0.1) ... 2019-12-05T05:02:22.177 INFO:teuthology.orchestra.run.smithi040.stdout:Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libssl1.1 amd64 1.1.1-1ubuntu2.1~18.04.5 [1,300 kB] 2019-12-05T05:02:22.327 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package libfl2:amd64. 2019-12-05T05:02:22.341 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../6-libfl2_2.6.4-6_amd64.deb ... 2019-12-05T05:02:22.379 INFO:teuthology.orchestra.run.smithi040.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic/main amd64 libbison-dev amd64 2:3.0.4.dfsg-1build1 [339 kB] 2019-12-05T05:02:22.386 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking libfl2:amd64 (2.6.4-6) ... 2019-12-05T05:02:22.406 INFO:teuthology.orchestra.run.smithi040.stdout:Get:4 http://archive.ubuntu.com/ubuntu bionic/main amd64 bison amd64 2:3.0.4.dfsg-1build1 [266 kB] 2019-12-05T05:02:22.424 INFO:teuthology.orchestra.run.smithi040.stdout:Get:5 http://archive.ubuntu.com/ubuntu bionic/main amd64 zlib1g-dev amd64 1:1.2.11.dfsg-0ubuntu2 [176 kB] 2019-12-05T05:02:22.434 INFO:teuthology.orchestra.run.smithi040.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libelf-dev amd64 0.170-0.4ubuntu0.1 [57.3 kB] 2019-12-05T05:02:22.436 INFO:teuthology.orchestra.run.smithi040.stdout:Get:7 http://archive.ubuntu.com/ubuntu bionic/main amd64 libfl2 amd64 2.6.4-6 [11.4 kB] 2019-12-05T05:02:22.437 INFO:teuthology.orchestra.run.smithi040.stdout:Get:8 http://archive.ubuntu.com/ubuntu bionic/main amd64 libfl-dev amd64 2.6.4-6 [6,320 B] 2019-12-05T05:02:22.438 INFO:teuthology.orchestra.run.smithi040.stdout:Get:9 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libssl-dev amd64 1.1.1-1ubuntu2.1~18.04.5 [1,566 kB] 2019-12-05T05:02:22.646 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package libfl-dev:amd64. 2019-12-05T05:02:22.660 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../7-libfl-dev_2.6.4-6_amd64.deb ... 2019-12-05T05:02:22.705 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking libfl-dev:amd64 (2.6.4-6) ... 2019-12-05T05:02:22.844 INFO:teuthology.orchestra.run.smithi040.stdout:Preconfiguring packages ... 2019-12-05T05:02:22.968 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package libssl-dev:amd64. 2019-12-05T05:02:22.980 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../8-libssl-dev_1.1.1-1ubuntu2.1~18.04.5_amd64.deb ... 2019-12-05T05:02:23.041 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking libssl-dev:amd64 (1.1.1-1ubuntu2.1~18.04.5) ... 2019-12-05T05:02:23.066 INFO:teuthology.orchestra.run.smithi040.stdout:Fetched 4,037 kB in 1s (4,143 kB/s) 2019-12-05T05:02:23.300 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package flex. 2019-12-05T05:02:23.353 INFO:teuthology.orchestra.run.smithi040.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 143715 files and directories currently installed.) 2019-12-05T05:02:23.357 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../0-flex_2.6.4-6_amd64.deb ... 2019-12-05T05:02:23.407 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking flex (2.6.4-6) ... 2019-12-05T05:02:23.510 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up flex (2.6.4-6) ... 2019-12-05T05:02:23.629 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up libssl1.1:amd64 (1.1.1-1ubuntu2.1~18.04.5) ... 2019-12-05T05:02:23.922 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up libbison-dev:amd64 (2:3.0.4.dfsg-1build1) ... 2019-12-05T05:02:24.001 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../1-libssl1.1_1.1.1-1ubuntu2.1~18.04.5_amd64.deb ... 2019-12-05T05:02:24.065 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up libfl2:amd64 (2.6.4-6) ... 2019-12-05T05:02:24.200 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up bison (2:3.0.4.dfsg-1build1) ... 2019-12-05T05:02:24.213 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking libssl1.1:amd64 (1.1.1-1ubuntu2.1~18.04.5) over (1.1.1-1ubuntu2.1~18.04.4) ... 2019-12-05T05:02:24.289 INFO:teuthology.orchestra.run.smithi042.stdout:update-alternatives: using /usr/bin/bison.yacc to provide /usr/bin/yacc (yacc) in auto mode 2019-12-05T05:02:24.359 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up zlib1g-dev:amd64 (1:1.2.11.dfsg-0ubuntu2) ... 2019-12-05T05:02:24.494 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up libssl-dev:amd64 (1.1.1-1ubuntu2.1~18.04.5) ... 2019-12-05T05:02:24.619 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up libelf-dev:amd64 (0.170-0.4ubuntu0.1) ... 2019-12-05T05:02:24.667 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package libbison-dev:amd64. 2019-12-05T05:02:24.682 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../2-libbison-dev_2%3a3.0.4.dfsg-1build1_amd64.deb ... 2019-12-05T05:02:24.725 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking libbison-dev:amd64 (2:3.0.4.dfsg-1build1) ... 2019-12-05T05:02:24.762 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up libfl-dev:amd64 (2.6.4-6) ... 2019-12-05T05:02:24.889 INFO:teuthology.orchestra.run.smithi042.stdout:Processing triggers for man-db (2.8.3-2ubuntu0.1) ... 2019-12-05T05:02:25.111 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package bison. 2019-12-05T05:02:25.126 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../3-bison_2%3a3.0.4.dfsg-1build1_amd64.deb ... 2019-12-05T05:02:25.174 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking bison (2:3.0.4.dfsg-1build1) ... 2019-12-05T05:02:25.515 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package zlib1g-dev:amd64. 2019-12-05T05:02:25.530 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../4-zlib1g-dev_1%3a1.2.11.dfsg-0ubuntu2_amd64.deb ... 2019-12-05T05:02:25.600 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking zlib1g-dev:amd64 (1:1.2.11.dfsg-0ubuntu2) ... 2019-12-05T05:02:25.901 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package libelf-dev:amd64. 2019-12-05T05:02:25.923 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../5-libelf-dev_0.170-0.4ubuntu0.1_amd64.deb ... 2019-12-05T05:02:25.959 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking libelf-dev:amd64 (0.170-0.4ubuntu0.1) ... 2019-12-05T05:02:26.346 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package libfl2:amd64. 2019-12-05T05:02:26.361 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../6-libfl2_2.6.4-6_amd64.deb ... 2019-12-05T05:02:26.404 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking libfl2:amd64 (2.6.4-6) ... 2019-12-05T05:02:26.684 INFO:teuthology.orchestra.run.smithi042.stdout:Processing triggers for install-info (6.5.0.dfsg.1-2) ... 2019-12-05T05:02:26.724 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package libfl-dev:amd64. 2019-12-05T05:02:26.738 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../7-libfl-dev_2.6.4-6_amd64.deb ... 2019-12-05T05:02:26.782 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking libfl-dev:amd64 (2.6.4-6) ... 2019-12-05T05:02:27.002 INFO:teuthology.orchestra.run.smithi042.stdout:Processing triggers for libc-bin (2.27-3ubuntu1) ... 2019-12-05T05:02:27.035 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package libssl-dev:amd64. 2019-12-05T05:02:27.057 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../8-libssl-dev_1.1.1-1ubuntu2.1~18.04.5_amd64.deb ... 2019-12-05T05:02:27.094 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking libssl-dev:amd64 (1.1.1-1ubuntu2.1~18.04.5) ... 2019-12-05T05:02:27.444 INFO:teuthology.orchestra.run.smithi042.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2019-12-05T05:02:27.446 DEBUG:teuthology.parallel:result is None 2019-12-05T05:02:27.586 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up flex (2.6.4-6) ... 2019-12-05T05:02:27.734 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up libssl1.1:amd64 (1.1.1-1ubuntu2.1~18.04.5) ... 2019-12-05T05:02:28.165 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up libbison-dev:amd64 (2:3.0.4.dfsg-1build1) ... 2019-12-05T05:02:28.320 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up libfl2:amd64 (2.6.4-6) ... 2019-12-05T05:02:28.463 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up bison (2:3.0.4.dfsg-1build1) ... 2019-12-05T05:02:28.552 INFO:teuthology.orchestra.run.smithi040.stdout:update-alternatives: using /usr/bin/bison.yacc to provide /usr/bin/yacc (yacc) in auto mode 2019-12-05T05:02:28.626 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up zlib1g-dev:amd64 (1:1.2.11.dfsg-0ubuntu2) ... 2019-12-05T05:02:28.786 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up libssl-dev:amd64 (1.1.1-1ubuntu2.1~18.04.5) ... 2019-12-05T05:02:28.917 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up libelf-dev:amd64 (0.170-0.4ubuntu0.1) ... 2019-12-05T05:02:29.043 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up libfl-dev:amd64 (2.6.4-6) ... 2019-12-05T05:02:29.169 INFO:teuthology.orchestra.run.smithi040.stdout:Processing triggers for man-db (2.8.3-2ubuntu0.1) ... 2019-12-05T05:02:30.938 INFO:teuthology.orchestra.run.smithi040.stdout:Processing triggers for install-info (6.5.0.dfsg.1-2) ... 2019-12-05T05:02:31.281 INFO:teuthology.orchestra.run.smithi040.stdout:Processing triggers for libc-bin (2.27-3ubuntu1) ... 2019-12-05T05:02:31.715 INFO:teuthology.orchestra.run.smithi040.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2019-12-05T05:02:31.718 DEBUG:teuthology.parallel:result is None 2019-12-05T05:02:31.718 INFO:teuthology.orchestra.run.smithi042:> sudo lsb_release -is 2019-12-05T05:02:31.774 INFO:teuthology.orchestra.run.smithi042.stdout:Ubuntu 2019-12-05T05:02:31.780 DEBUG:teuthology.misc:System to be installed: Ubuntu 2019-12-05T05:02:31.780 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1=c5290805033483ba0cec165c06bde07dae5f19ec 2019-12-05T05:02:31.977 INFO:teuthology.orchestra.run.smithi042:> dpkg-query -W -f '${Version}' ceph 2019-12-05T05:02:32.004 INFO:teuthology.orchestra.run.smithi042.stdout:15.0.0-8014-gc529080-1bionic 2019-12-05T05:02:32.004 INFO:teuthology.packaging:The installed version of ceph is 15.0.0-8014-gc529080-1bionic 2019-12-05T05:02:32.005 INFO:teuthology.task.install:The correct ceph version 15.0.0-8014-gc529080-1bionic is installed. 2019-12-05T05:02:32.005 INFO:teuthology.orchestra.run.smithi040:> sudo lsb_release -is 2019-12-05T05:02:32.060 INFO:teuthology.orchestra.run.smithi040.stdout:Ubuntu 2019-12-05T05:02:32.066 DEBUG:teuthology.misc:System to be installed: Ubuntu 2019-12-05T05:02:32.066 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1=c5290805033483ba0cec165c06bde07dae5f19ec 2019-12-05T05:02:32.238 INFO:teuthology.orchestra.run.smithi040:> dpkg-query -W -f '${Version}' ceph 2019-12-05T05:02:32.262 INFO:teuthology.orchestra.run.smithi040.stdout:15.0.0-8014-gc529080-1bionic 2019-12-05T05:02:32.263 INFO:teuthology.packaging:The installed version of ceph is 15.0.0-8014-gc529080-1bionic 2019-12-05T05:02:32.263 INFO:teuthology.task.install:The correct ceph version 15.0.0-8014-gc529080-1bionic is installed. 2019-12-05T05:02:32.263 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2019-12-05T05:02:32.264 INFO:teuthology.orchestra.run.smithi042:> sudo sh -c 'cat > /home/ubuntu/cephtest/valgrind.supp' 2019-12-05T05:02:32.282 INFO:teuthology.orchestra.run.smithi040:> sudo sh -c 'cat > /home/ubuntu/cephtest/valgrind.supp' 2019-12-05T05:02:32.331 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2019-12-05T05:02:32.331 INFO:teuthology.orchestra.run.smithi042:> sudo sh -c 'cat > /usr/bin/daemon-helper' 2019-12-05T05:02:32.349 INFO:teuthology.orchestra.run.smithi042:> sudo chmod a=rx -- /usr/bin/daemon-helper 2019-12-05T05:02:32.433 INFO:teuthology.orchestra.run.smithi040:> sudo sh -c 'cat > /usr/bin/daemon-helper' 2019-12-05T05:02:32.449 INFO:teuthology.orchestra.run.smithi040:> sudo chmod a=rx -- /usr/bin/daemon-helper 2019-12-05T05:02:32.526 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2019-12-05T05:02:32.526 INFO:teuthology.orchestra.run.smithi042:> sudo sh -c 'cat > /usr/bin/adjust-ulimits' 2019-12-05T05:02:32.572 INFO:teuthology.orchestra.run.smithi042:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2019-12-05T05:02:32.652 INFO:teuthology.orchestra.run.smithi040:> sudo sh -c 'cat > /usr/bin/adjust-ulimits' 2019-12-05T05:02:32.702 INFO:teuthology.orchestra.run.smithi040:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2019-12-05T05:02:32.781 INFO:teuthology.run_tasks:Running task ceph... 2019-12-05T05:02:32.845 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2019-12-05T05:02:32.845 INFO:teuthology.orchestra.run.smithi040:> sudo chmod 777 /var/log/ceph 2019-12-05T05:02:32.848 INFO:teuthology.orchestra.run.smithi042:> sudo chmod 777 /var/log/ceph 2019-12-05T05:02:32.864 INFO:tasks.ceph:Disabling ceph logrotate... 2019-12-05T05:02:32.865 INFO:teuthology.orchestra.run.smithi040:> sudo rm -f -- /etc/logrotate.d/ceph 2019-12-05T05:02:32.932 INFO:teuthology.orchestra.run.smithi042:> sudo rm -f -- /etc/logrotate.d/ceph 2019-12-05T05:02:32.950 INFO:tasks.ceph:Creating extra log directories... 2019-12-05T05:02:32.950 INFO:teuthology.orchestra.run.smithi040:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2019-12-05T05:02:33.020 INFO:teuthology.orchestra.run.smithi042:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2019-12-05T05:02:33.037 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2019-12-05T05:02:33.037 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2019-12-05T05:02:33.037 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2019-12-05T05:02:33.038 INFO:teuthology.orchestra.run.smithi042:> cat > /home/ubuntu/cephtest/logrotate.ceph-test.conf 2019-12-05T05:02:33.149 INFO:teuthology.orchestra.run.smithi042:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2019-12-05T05:02:33.253 INFO:teuthology.orchestra.run.smithi040:> cat > /home/ubuntu/cephtest/logrotate.ceph-test.conf 2019-12-05T05:02:33.315 INFO:teuthology.orchestra.run.smithi040:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2019-12-05T05:02:33.421 INFO:tasks.ceph:Creating ceph cluster ceph... 2019-12-05T05:02:33.421 INFO:tasks.ceph:config {'mon_bind_addrvec': True, 'block_journal': None, 'fs': 'xfs', 'log_whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'not responding, replacing', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'skip_mgr_daemons': False, 'cpu_profile': set([]), 'mkfs_options': None, 'tmpfs_journal': None, 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mount_options': None, 'conf': {'mds': {'mds debug frag': True, 'mds bal split bits': 3, 'mds bal split size': 100, 'debug mds': 20, 'mds bal merge size': 5, 'debug ms': 1, 'mds bal frag': True, 'mds bal fragment size max': 10000, 'mds op complaint time': 180, 'mds debug scatterstat': True, 'osd op complaint time': 180, 'mds verify scatter': True}, 'global': {'ms inject delay type': 'osd mds', 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'mon client directed command retry': 5, 'ms inject socket failures': 2500}, 'osd': {'mon osd full ratio': 0.9, 'debug ms': 1, 'debug journal': 20, 'debug osd': 25, 'debug bluestore': 20, 'debug bluefs': 20, 'osd objectstore': 'bluestore', 'mon osd backfillfull_ratio': 0.85, 'osd failsafe full ratio': 0.95, 'bluestore block size': 96636764160, 'debug filestore': 20, 'debug rocksdb': 10, 'mon osd nearfull ratio': 0.8, 'osd op complaint time': 180, 'bluestore fsck on mount': True}, 'mon': {'debug mon': 20, 'debug paxos': 20, 'mon op complaint time': 120, 'debug ms': 1}, 'client': {'debug ms': 1, 'debug client': 20, 'client mount timeout': 600}}} 2019-12-05T05:02:33.421 INFO:tasks.ceph:ctx.config {'kernel': {'sha1': 'distro', 'kdb': True}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.lock_machines': (2, 'smithi')}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'sha1': 'distro', 'kdb': True}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel']}, 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell'], 'rpm': ['python3-cephfs']}, 'sha1': 'c5290805033483ba0cec165c06bde07dae5f19ec'}}, {'ceph': None}, {'mds_thrash': None}, {'ceph-fuse': None}, {'workunit': {'clients': {'all': ['fs/snaps']}}}], 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_wip-pdonnell-testing-20191204.231200/qa', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/worker.smithi.9180', 'owner': 'scheduled_pdonnell@teuthology', 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'targets': {'smithi042.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDMZ4TtLKXWjwgZ56TLIpfpPIWboZCfXhbjBb/Qn2NiHen0ZAuEBJ8bNv4Slz4g63tJrnNOqlGQITIsdjbLZ6BdoOpeYWlGQDSZzSLvX6WxTtQpdl+F7OthfarPxPX8MZ1Lhd1wsPG5ZYnuuFgg9yrsPqbCr/1G5QbVU+cTl0FB1F7aMwEcpUiE6TV8y2/qTJZbCAR3NnSFjYUWUR6PVmYFg3fWT33wz0h3cZIHuEx1HTORW0f6XvJ543BNSlik9/7k67PuxHo4HIZkohNLUHHV3sYGS9yO7zP4tdD9B9MfjRG9lHpyoC0L1J55CFASYoZnxblDrh6ETrGW21tU83hd', 'smithi040.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCgyJ01L5AbcRj/1XTjdYFJIHzwMpnOxBZONLQTXqlhq7lTSfDfiDodGztGQJajHviERj+qqE3vO3E2x/P4ZU/5IT/LJv50GUX9BdxIIGuuynvuvR3O4HMkop0bQsDgw4a+v8zt37zDNjCytnTl6Y0aNBnnauTeyzzx8XCbIkVH57yJls5rMIZoie6zgVfE/UL9aQcrsK5nhq9TLY5i0RLGuE2QTYnNfqm1/t5qdnfBNjQSp6ISqMi46VLqiKOjU6TyYT8fz1zQHZ8TtMIPHTf63z2yJ/KNgMZdAMzNehRjmvpzMD0FKJQSSeop8QREHqaamxu76t1F6npDEIlKsBOF'}, 'job_id': '4567617', 'tube': 'smithi', 'suite_branch': 'wip-pdonnell-testing-20191204.231200', 'priority': 75, 'os_version': '18.04', 'first_in_suite': False, 'branch': 'wip-pdonnell-testing-20191204.231200', 'suite': 'fs', 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'email': 'pdonnell@redhat.com', 'archive_path': '/home/teuthworker/archive/pdonnell-2019-12-05_02:12:53-fs-wip-pdonnell-testing-20191204.231200-distro-basic-smithi/4567617', 'description': 'fs/thrash/{begin.yaml ceph-thrash/mds.yaml clusters/1-mds-1-client-coloc.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} mount/fuse.yaml msgr-failures/osd-mds-delay.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} supported-random-distros$/{ubuntu_latest.yaml} tasks/cfuse_workunit_snaptests.yaml}', 'nuke-on-error': True, 'repo': 'git://git.ceph.com/ceph-ci.git', 'last_in_suite': False, 'machine_type': 'smithi', 'teuthology_branch': 'master', 'sha1': 'c5290805033483ba0cec165c06bde07dae5f19ec', 'name': 'pdonnell-2019-12-05_02:12:53-fs-wip-pdonnell-testing-20191204.231200-distro-basic-smithi', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'osd.4', 'osd.5', 'osd.6', 'osd.7']], 'suite_relpath': 'qa', 'suite_repo': 'git://git.ceph.com/ceph-ci.git', 'overrides': {'ceph': {'cephfs_ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'not responding, replacing', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'conf': {'mds': {'mds debug frag': True, 'mds bal split bits': 3, 'mds bal split size': 100, 'debug mds': 20, 'mds bal merge size': 5, 'debug ms': 1, 'mds bal frag': True, 'mds bal fragment size max': 10000, 'mds op complaint time': 180, 'mds debug scatterstat': True, 'osd op complaint time': 180, 'mds verify scatter': True}, 'global': {'ms inject delay type': 'osd mds', 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'mon client directed command retry': 5, 'ms inject socket failures': 2500}, 'osd': {'mon osd full ratio': 0.9, 'debug ms': 1, 'debug journal': 20, 'debug osd': 25, 'debug bluestore': 20, 'debug bluefs': 20, 'osd objectstore': 'bluestore', 'mon osd backfillfull_ratio': 0.85, 'osd failsafe full ratio': 0.95, 'bluestore block size': 96636764160, 'debug filestore': 20, 'debug rocksdb': 10, 'mon osd nearfull ratio': 0.8, 'osd op complaint time': 180, 'bluestore fsck on mount': True}, 'mon': {'debug mon': 20, 'debug paxos': 20, 'mon op complaint time': 120, 'debug ms': 1}, 'client': {'debug ms': 1, 'debug client': 20, 'client mount timeout': 600}}, 'fs': 'xfs', 'sha1': 'c5290805033483ba0cec165c06bde07dae5f19ec'}, 'ceph-deploy': {'fs': 'xfs', 'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}, 'osd': {'mon osd full ratio': 0.9, 'mon osd backfillfull_ratio': 0.85, 'bluestore fsck on mount': True, 'mon osd nearfull ratio': 0.8, 'debug bluestore': 20, 'debug bluefs': 20, 'osd objectstore': 'bluestore', 'bluestore block size': 96636764160, 'debug rocksdb': 10, 'osd failsafe full ratio': 0.95}}}, 'install': {'ceph': {'sha1': 'c5290805033483ba0cec165c06bde07dae5f19ec'}}, 'admin_socket': {'branch': 'wip-pdonnell-testing-20191204.231200'}, 'thrashosds': {'bdev_inject_crash_probability': 0.5, 'bdev_inject_crash': 2}, 'workunit': {'sha1': 'c5290805033483ba0cec165c06bde07dae5f19ec', 'branch': 'wip-pdonnell-testing-20191204.231200'}}, 'os_type': 'ubuntu', 'suite_sha1': 'c5290805033483ba0cec165c06bde07dae5f19ec'} 2019-12-05T05:02:33.422 INFO:teuthology.orchestra.run.smithi040:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2019-12-05T05:02:33.499 INFO:teuthology.orchestra.run.smithi042:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2019-12-05T05:02:33.508 INFO:teuthology.orchestra.run.smithi040:> sudo install -d -m0777 -- /var/run/ceph 2019-12-05T05:02:33.582 INFO:teuthology.orchestra.run.smithi042:> sudo install -d -m0777 -- /var/run/ceph 2019-12-05T05:02:33.670 DEBUG:teuthology.orchestra.remote:smithi042:/scratch_devs is 72B 2019-12-05T05:02:33.728 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_4', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_1'] 2019-12-05T05:02:33.728 INFO:teuthology.orchestra.run.smithi042:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2019-12-05T05:02:33.737 INFO:teuthology.orchestra.run.smithi042.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-1 2019-12-05T05:02:33.737 INFO:teuthology.orchestra.run.smithi042.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2019-12-05T05:02:33.737 INFO:teuthology.orchestra.run.smithi042.stdout:Device: 6h/6d Inode: 443 Links: 1 2019-12-05T05:02:33.737 INFO:teuthology.orchestra.run.smithi042.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2019-12-05T05:02:33.738 INFO:teuthology.orchestra.run.smithi042.stdout:Access: 2019-12-05 04:59:38.696771176 +0000 2019-12-05T05:02:33.738 INFO:teuthology.orchestra.run.smithi042.stdout:Modify: 2019-12-05 04:59:38.584827161 +0000 2019-12-05T05:02:33.738 INFO:teuthology.orchestra.run.smithi042.stdout:Change: 2019-12-05 04:59:38.584827161 +0000 2019-12-05T05:02:33.738 INFO:teuthology.orchestra.run.smithi042.stdout: Birth: - 2019-12-05T05:02:33.746 INFO:teuthology.orchestra.run.smithi042.stderr:1+0 records in 2019-12-05T05:02:33.747 INFO:teuthology.orchestra.run.smithi042.stderr:1+0 records out 2019-12-05T05:02:33.747 INFO:teuthology.orchestra.run.smithi042.stderr:512 bytes copied, 0.000340534 s, 1.5 MB/s 2019-12-05T05:02:33.751 INFO:teuthology.orchestra.run.smithi042:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2019-12-05T05:02:33.823 INFO:teuthology.orchestra.run.smithi042.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2019-12-05T05:02:33.824 INFO:teuthology.orchestra.run.smithi042.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2019-12-05T05:02:33.824 INFO:teuthology.orchestra.run.smithi042.stdout:Device: 6h/6d Inode: 449 Links: 1 2019-12-05T05:02:33.824 INFO:teuthology.orchestra.run.smithi042.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2019-12-05T05:02:33.824 INFO:teuthology.orchestra.run.smithi042.stdout:Access: 2019-12-05 04:59:38.916661206 +0000 2019-12-05T05:02:33.824 INFO:teuthology.orchestra.run.smithi042.stdout:Modify: 2019-12-05 04:59:38.804717191 +0000 2019-12-05T05:02:33.824 INFO:teuthology.orchestra.run.smithi042.stdout:Change: 2019-12-05 04:59:38.804717191 +0000 2019-12-05T05:02:33.825 INFO:teuthology.orchestra.run.smithi042.stdout: Birth: - 2019-12-05T05:02:33.833 INFO:teuthology.orchestra.run.smithi042.stderr:1+0 records in 2019-12-05T05:02:33.833 INFO:teuthology.orchestra.run.smithi042.stderr:1+0 records out 2019-12-05T05:02:33.833 INFO:teuthology.orchestra.run.smithi042.stderr:512 bytes copied, 0.000286457 s, 1.8 MB/s 2019-12-05T05:02:33.837 INFO:teuthology.orchestra.run.smithi042:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2019-12-05T05:02:33.909 INFO:teuthology.orchestra.run.smithi042.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-3 2019-12-05T05:02:33.909 INFO:teuthology.orchestra.run.smithi042.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2019-12-05T05:02:33.909 INFO:teuthology.orchestra.run.smithi042.stdout:Device: 6h/6d Inode: 455 Links: 1 2019-12-05T05:02:33.909 INFO:teuthology.orchestra.run.smithi042.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2019-12-05T05:02:33.909 INFO:teuthology.orchestra.run.smithi042.stdout:Access: 2019-12-05 04:59:39.160539238 +0000 2019-12-05T05:02:33.909 INFO:teuthology.orchestra.run.smithi042.stdout:Modify: 2019-12-05 04:59:39.032603221 +0000 2019-12-05T05:02:33.909 INFO:teuthology.orchestra.run.smithi042.stdout:Change: 2019-12-05 04:59:39.032603221 +0000 2019-12-05T05:02:33.910 INFO:teuthology.orchestra.run.smithi042.stdout: Birth: - 2019-12-05T05:02:33.918 INFO:teuthology.orchestra.run.smithi042.stderr:1+0 records in 2019-12-05T05:02:33.918 INFO:teuthology.orchestra.run.smithi042.stderr:1+0 records out 2019-12-05T05:02:33.918 INFO:teuthology.orchestra.run.smithi042.stderr:512 bytes copied, 0.000215263 s, 2.4 MB/s 2019-12-05T05:02:33.923 INFO:teuthology.orchestra.run.smithi042:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2019-12-05T05:02:33.994 INFO:teuthology.orchestra.run.smithi042.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-4 2019-12-05T05:02:33.994 INFO:teuthology.orchestra.run.smithi042.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2019-12-05T05:02:33.994 INFO:teuthology.orchestra.run.smithi042.stdout:Device: 6h/6d Inode: 461 Links: 1 2019-12-05T05:02:33.996 INFO:teuthology.orchestra.run.smithi042.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2019-12-05T05:02:33.997 INFO:teuthology.orchestra.run.smithi042.stdout:Access: 2019-12-05 05:02:06.002331310 +0000 2019-12-05T05:02:33.997 INFO:teuthology.orchestra.run.smithi042.stdout:Modify: 2019-12-05 04:59:39.268485253 +0000 2019-12-05T05:02:33.997 INFO:teuthology.orchestra.run.smithi042.stdout:Change: 2019-12-05 04:59:39.268485253 +0000 2019-12-05T05:02:33.997 INFO:teuthology.orchestra.run.smithi042.stdout: Birth: - 2019-12-05T05:02:34.004 INFO:teuthology.orchestra.run.smithi042.stderr:1+0 records in 2019-12-05T05:02:34.004 INFO:teuthology.orchestra.run.smithi042.stderr:1+0 records out 2019-12-05T05:02:34.004 INFO:teuthology.orchestra.run.smithi042.stderr:512 bytes copied, 0.000212391 s, 2.4 MB/s 2019-12-05T05:02:34.008 INFO:tasks.ceph:fs option selected, checking for scratch devs 2019-12-05T05:02:34.008 INFO:tasks.ceph:found devs: ['/dev/vg_nvme/lv_4', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_1'] 2019-12-05T05:02:34.008 WARNING:teuthology.misc:Entering get_wwn_id_map, a deprecated function that will be removed 2019-12-05T05:02:34.008 INFO:tasks.ceph:dev map: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.0': '/dev/vg_nvme/lv_4', 'osd.3': '/dev/vg_nvme/lv_3', 'osd.2': '/dev/vg_nvme/lv_2'} 2019-12-05T05:02:34.018 DEBUG:teuthology.orchestra.remote:smithi040:/scratch_devs is 72B 2019-12-05T05:02:34.034 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_4', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_1'] 2019-12-05T05:02:34.035 INFO:teuthology.orchestra.run.smithi040:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2019-12-05T05:02:34.043 INFO:teuthology.orchestra.run.smithi040.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-1 2019-12-05T05:02:34.043 INFO:teuthology.orchestra.run.smithi040.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2019-12-05T05:02:34.043 INFO:teuthology.orchestra.run.smithi040.stdout:Device: 6h/6d Inode: 452 Links: 1 2019-12-05T05:02:34.043 INFO:teuthology.orchestra.run.smithi040.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2019-12-05T05:02:34.044 INFO:teuthology.orchestra.run.smithi040.stdout:Access: 2019-12-05 04:59:39.039409090 +0000 2019-12-05T05:02:34.044 INFO:teuthology.orchestra.run.smithi040.stdout:Modify: 2019-12-05 04:59:38.931355093 +0000 2019-12-05T05:02:34.044 INFO:teuthology.orchestra.run.smithi040.stdout:Change: 2019-12-05 04:59:38.931355093 +0000 2019-12-05T05:02:34.045 INFO:teuthology.orchestra.run.smithi040.stdout: Birth: - 2019-12-05T05:02:34.051 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records in 2019-12-05T05:02:34.052 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records out 2019-12-05T05:02:34.052 INFO:teuthology.orchestra.run.smithi040.stderr:512 bytes copied, 0.000256508 s, 2.0 MB/s 2019-12-05T05:02:34.055 INFO:teuthology.orchestra.run.smithi040:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2019-12-05T05:02:34.126 INFO:teuthology.orchestra.run.smithi040.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2019-12-05T05:02:34.127 INFO:teuthology.orchestra.run.smithi040.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2019-12-05T05:02:34.127 INFO:teuthology.orchestra.run.smithi040.stdout:Device: 6h/6d Inode: 458 Links: 1 2019-12-05T05:02:34.127 INFO:teuthology.orchestra.run.smithi040.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2019-12-05T05:02:34.127 INFO:teuthology.orchestra.run.smithi040.stdout:Access: 2019-12-05 04:59:39.295537085 +0000 2019-12-05T05:02:34.127 INFO:teuthology.orchestra.run.smithi040.stdout:Modify: 2019-12-05 04:59:39.171475088 +0000 2019-12-05T05:02:34.128 INFO:teuthology.orchestra.run.smithi040.stdout:Change: 2019-12-05 04:59:39.171475088 +0000 2019-12-05T05:02:34.128 INFO:teuthology.orchestra.run.smithi040.stdout: Birth: - 2019-12-05T05:02:34.135 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records in 2019-12-05T05:02:34.136 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records out 2019-12-05T05:02:34.136 INFO:teuthology.orchestra.run.smithi040.stderr:512 bytes copied, 0.000347073 s, 1.5 MB/s 2019-12-05T05:02:34.140 INFO:teuthology.orchestra.run.smithi040:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2019-12-05T05:02:34.188 INFO:teuthology.orchestra.run.smithi040.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-3 2019-12-05T05:02:34.188 INFO:teuthology.orchestra.run.smithi040.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2019-12-05T05:02:34.188 INFO:teuthology.orchestra.run.smithi040.stdout:Device: 6h/6d Inode: 464 Links: 1 2019-12-05T05:02:34.188 INFO:teuthology.orchestra.run.smithi040.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2019-12-05T05:02:34.189 INFO:teuthology.orchestra.run.smithi040.stdout:Access: 2019-12-05 04:59:39.543661081 +0000 2019-12-05T05:02:34.189 INFO:teuthology.orchestra.run.smithi040.stdout:Modify: 2019-12-05 04:59:39.447613082 +0000 2019-12-05T05:02:34.189 INFO:teuthology.orchestra.run.smithi040.stdout:Change: 2019-12-05 04:59:39.447613082 +0000 2019-12-05T05:02:34.189 INFO:teuthology.orchestra.run.smithi040.stdout: Birth: - 2019-12-05T05:02:34.198 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records in 2019-12-05T05:02:34.198 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records out 2019-12-05T05:02:34.198 INFO:teuthology.orchestra.run.smithi040.stderr:512 bytes copied, 0.000453494 s, 1.1 MB/s 2019-12-05T05:02:34.203 INFO:teuthology.orchestra.run.smithi040:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2019-12-05T05:02:34.280 INFO:teuthology.orchestra.run.smithi040.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-4 2019-12-05T05:02:34.280 INFO:teuthology.orchestra.run.smithi040.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2019-12-05T05:02:34.280 INFO:teuthology.orchestra.run.smithi040.stdout:Device: 6h/6d Inode: 470 Links: 1 2019-12-05T05:02:34.280 INFO:teuthology.orchestra.run.smithi040.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2019-12-05T05:02:34.280 INFO:teuthology.orchestra.run.smithi040.stdout:Access: 2019-12-05 05:02:09.352087442 +0000 2019-12-05T05:02:34.281 INFO:teuthology.orchestra.run.smithi040.stdout:Modify: 2019-12-05 04:59:39.675727076 +0000 2019-12-05T05:02:34.281 INFO:teuthology.orchestra.run.smithi040.stdout:Change: 2019-12-05 04:59:39.675727076 +0000 2019-12-05T05:02:34.281 INFO:teuthology.orchestra.run.smithi040.stdout: Birth: - 2019-12-05T05:02:34.289 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records in 2019-12-05T05:02:34.290 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records out 2019-12-05T05:02:34.290 INFO:teuthology.orchestra.run.smithi040.stderr:512 bytes copied, 0.000402557 s, 1.3 MB/s 2019-12-05T05:02:34.294 INFO:tasks.ceph:fs option selected, checking for scratch devs 2019-12-05T05:02:34.295 INFO:tasks.ceph:found devs: ['/dev/vg_nvme/lv_4', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_1'] 2019-12-05T05:02:34.295 WARNING:teuthology.misc:Entering get_wwn_id_map, a deprecated function that will be removed 2019-12-05T05:02:34.295 INFO:tasks.ceph:dev map: {'osd.5': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_4', 'osd.7': '/dev/vg_nvme/lv_3', 'osd.6': '/dev/vg_nvme/lv_2'} 2019-12-05T05:02:34.295 INFO:tasks.ceph:Generating config... 2019-12-05T05:02:34.303 INFO:tasks.ceph:[mds] mds debug frag = True 2019-12-05T05:02:34.303 INFO:tasks.ceph:[mds] mds bal split bits = 3 2019-12-05T05:02:34.303 INFO:tasks.ceph:[mds] mds bal split size = 100 2019-12-05T05:02:34.303 INFO:tasks.ceph:[mds] debug mds = 20 2019-12-05T05:02:34.304 INFO:tasks.ceph:[mds] mds bal merge size = 5 2019-12-05T05:02:34.304 INFO:tasks.ceph:[mds] debug ms = 1 2019-12-05T05:02:34.304 INFO:tasks.ceph:[mds] mds bal frag = True 2019-12-05T05:02:34.304 INFO:tasks.ceph:[mds] mds bal fragment size max = 10000 2019-12-05T05:02:34.304 INFO:tasks.ceph:[mds] mds op complaint time = 180 2019-12-05T05:02:34.304 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2019-12-05T05:02:34.304 INFO:tasks.ceph:[mds] osd op complaint time = 180 2019-12-05T05:02:34.305 INFO:tasks.ceph:[mds] mds verify scatter = True 2019-12-05T05:02:34.305 INFO:tasks.ceph:[global] ms inject delay type = osd mds 2019-12-05T05:02:34.305 INFO:tasks.ceph:[global] ms inject delay max = 1 2019-12-05T05:02:34.305 INFO:tasks.ceph:[global] ms inject delay probability = 0.005 2019-12-05T05:02:34.305 INFO:tasks.ceph:[global] mon client directed command retry = 5 2019-12-05T05:02:34.305 INFO:tasks.ceph:[global] ms inject socket failures = 2500 2019-12-05T05:02:34.305 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2019-12-05T05:02:34.306 INFO:tasks.ceph:[osd] debug ms = 1 2019-12-05T05:02:34.306 INFO:tasks.ceph:[osd] debug journal = 20 2019-12-05T05:02:34.306 INFO:tasks.ceph:[osd] debug osd = 25 2019-12-05T05:02:34.306 INFO:tasks.ceph:[osd] debug bluestore = 20 2019-12-05T05:02:34.306 INFO:tasks.ceph:[osd] debug bluefs = 20 2019-12-05T05:02:34.306 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2019-12-05T05:02:34.306 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2019-12-05T05:02:34.306 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2019-12-05T05:02:34.307 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2019-12-05T05:02:34.307 INFO:tasks.ceph:[osd] debug filestore = 20 2019-12-05T05:02:34.307 INFO:tasks.ceph:[osd] debug rocksdb = 10 2019-12-05T05:02:34.307 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2019-12-05T05:02:34.330 INFO:tasks.ceph:[osd] osd op complaint time = 180 2019-12-05T05:02:34.331 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2019-12-05T05:02:34.331 INFO:tasks.ceph:[mon] debug mon = 20 2019-12-05T05:02:34.331 INFO:tasks.ceph:[mon] debug paxos = 20 2019-12-05T05:02:34.331 INFO:tasks.ceph:[mon] mon op complaint time = 120 2019-12-05T05:02:34.331 INFO:tasks.ceph:[mon] debug ms = 1 2019-12-05T05:02:34.331 INFO:tasks.ceph:[client] debug ms = 1 2019-12-05T05:02:34.332 INFO:tasks.ceph:[client] debug client = 20 2019-12-05T05:02:34.332 INFO:tasks.ceph:[client] client mount timeout = 600 2019-12-05T05:02:34.348 INFO:tasks.ceph:Setting up mon.a... 2019-12-05T05:02:34.349 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2019-12-05T05:02:34.394 INFO:teuthology.orchestra.run.smithi042.stdout:creating /etc/ceph/ceph.keyring 2019-12-05T05:02:34.394 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2019-12-05T05:02:34.475 INFO:teuthology.orchestra.run.smithi042:> sudo chmod 0644 /etc/ceph/ceph.keyring 2019-12-05T05:02:34.564 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.42'), ('mon.c', '[v2:172.21.15.40:3301,v1:172.21.15.40:6790]'), ('mon.b', '172.21.15.40')] 2019-12-05T05:02:34.564 INFO:teuthology.orchestra.run.smithi042:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool --create --clobber --enable-all-features --add a 172.21.15.42 --addv c '[v2:172.21.15.40:3301,v1:172.21.15.40:6790]' --add b 172.21.15.40 --print /home/ubuntu/cephtest/ceph.monmap 2019-12-05T05:02:34.669 INFO:teuthology.orchestra.run.smithi042.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2019-12-05T05:02:34.669 INFO:teuthology.orchestra.run.smithi042.stdout:monmaptool: generated fsid 5d836cfc-a83b-4433-9b3e-b9b2f31c9050 2019-12-05T05:02:34.670 INFO:teuthology.orchestra.run.smithi042.stdout:epoch 0 2019-12-05T05:02:34.670 INFO:teuthology.orchestra.run.smithi042.stdout:fsid 5d836cfc-a83b-4433-9b3e-b9b2f31c9050 2019-12-05T05:02:34.670 INFO:teuthology.orchestra.run.smithi042.stdout:last_changed 2019-12-05T05:02:34.667737+0000 2019-12-05T05:02:34.670 INFO:teuthology.orchestra.run.smithi042.stdout:created 2019-12-05T05:02:34.667737+0000 2019-12-05T05:02:34.670 INFO:teuthology.orchestra.run.smithi042.stdout:min_mon_release 0 (unknown) 2019-12-05T05:02:34.670 INFO:teuthology.orchestra.run.smithi042.stdout:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2019-12-05T05:02:34.670 INFO:teuthology.orchestra.run.smithi042.stdout:1: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.b 2019-12-05T05:02:34.671 INFO:teuthology.orchestra.run.smithi042.stdout:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2019-12-05T05:02:34.671 INFO:teuthology.orchestra.run.smithi042.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2019-12-05T05:02:34.671 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 5d836cfc-a83b-4433-9b3e-b9b2f31c9050... 2019-12-05T05:02:34.672 INFO:teuthology.orchestra.run.smithi040:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo python -c 'import shutil, sys; shutil.copyfileobj(sys.stdin, file(sys.argv[1], "wb"))' /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf 2019-12-05T05:02:34.689 INFO:teuthology.orchestra.run.smithi042:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo python -c 'import shutil, sys; shutil.copyfileobj(sys.stdin, file(sys.argv[1], "wb"))' /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf 2019-12-05T05:02:34.780 INFO:tasks.ceph:Creating admin key on mon.a... 2019-12-05T05:02:34.781 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2019-12-05T05:02:34.824 INFO:tasks.ceph:Copying monmap to all nodes... 2019-12-05T05:02:34.904 DEBUG:teuthology.orchestra.remote:smithi042:/etc/ceph/ceph.keyring is 206B 2019-12-05T05:02:34.973 DEBUG:teuthology.orchestra.remote:smithi042:/home/ubuntu/cephtest/ceph.monmap is 379B 2019-12-05T05:02:35.052 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi042.front.sepia.ceph.com 2019-12-05T05:02:35.052 INFO:teuthology.orchestra.run.smithi042:> sudo sh -c 'cat > /etc/ceph/ceph.keyring' && sudo chmod 0644 /etc/ceph/ceph.keyring 2019-12-05T05:02:35.111 INFO:teuthology.orchestra.run.smithi042:> cat > /home/ubuntu/cephtest/ceph.monmap 2019-12-05T05:02:35.220 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi040.front.sepia.ceph.com 2019-12-05T05:02:35.220 INFO:teuthology.orchestra.run.smithi040:> sudo sh -c 'cat > /etc/ceph/ceph.keyring' && sudo chmod 0644 /etc/ceph/ceph.keyring 2019-12-05T05:02:35.288 INFO:teuthology.orchestra.run.smithi040:> cat > /home/ubuntu/cephtest/ceph.monmap 2019-12-05T05:02:35.402 INFO:tasks.ceph:Setting up mon nodes... 2019-12-05T05:02:35.402 INFO:tasks.ceph:Setting up mgr nodes... 2019-12-05T05:02:35.403 INFO:teuthology.orchestra.run.smithi042:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2019-12-05T05:02:35.450 INFO:teuthology.orchestra.run.smithi042.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2019-12-05T05:02:35.453 INFO:teuthology.orchestra.run.smithi040:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2019-12-05T05:02:35.499 INFO:teuthology.orchestra.run.smithi040.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2019-12-05T05:02:35.789 INFO:tasks.ceph:Setting up mds nodes... 2019-12-05T05:02:35.790 INFO:teuthology.orchestra.run.smithi042:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2019-12-05T05:02:35.842 INFO:teuthology.orchestra.run.smithi042.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2019-12-05T05:02:35.844 INFO:teuthology.orchestra.run.smithi042:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2019-12-05T05:02:35.928 INFO:teuthology.orchestra.run.smithi040:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2019-12-05T05:02:35.974 INFO:teuthology.orchestra.run.smithi040.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2019-12-05T05:02:35.976 INFO:teuthology.orchestra.run.smithi040:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2019-12-05T05:02:36.062 INFO:tasks.ceph_client:Setting up client nodes... 2019-12-05T05:02:36.062 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2019-12-05T05:02:36.105 INFO:teuthology.orchestra.run.smithi042.stdout:creating /etc/ceph/ceph.client.0.keyring 2019-12-05T05:02:36.118 INFO:tasks.ceph:Running mkfs on osd nodes... 2019-12-05T05:02:36.118 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi042.front.sepia.ceph.com'): {'osd.1': '/dev/vg_nvme/lv_1', 'osd.0': '/dev/vg_nvme/lv_4', 'osd.3': '/dev/vg_nvme/lv_3', 'osd.2': '/dev/vg_nvme/lv_2'}, Remote(name='ubuntu@smithi040.front.sepia.ceph.com'): {'osd.5': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_4', 'osd.7': '/dev/vg_nvme/lv_3', 'osd.6': '/dev/vg_nvme/lv_2'}} 2019-12-05T05:02:36.119 INFO:teuthology.orchestra.run.smithi042:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2019-12-05T05:02:36.203 INFO:tasks.ceph:{'osd.1': '/dev/vg_nvme/lv_1', 'osd.0': '/dev/vg_nvme/lv_4', 'osd.3': '/dev/vg_nvme/lv_3', 'osd.2': '/dev/vg_nvme/lv_2'} 2019-12-05T05:02:36.203 INFO:tasks.ceph:{} 2019-12-05T05:02:36.203 INFO:tasks.ceph:osd.0 2019-12-05T05:02:36.203 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi042.front.sepia.ceph.com 2019-12-05T05:02:36.203 INFO:teuthology.orchestra.run.smithi042:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2019-12-05T05:02:36.302 INFO:teuthology.orchestra.run.smithi042.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2019-12-05T05:02:36.303 INFO:teuthology.orchestra.run.smithi042.stdout: = sectsz=512 attr=2, projid32bit=1 2019-12-05T05:02:36.303 INFO:teuthology.orchestra.run.smithi042.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2019-12-05T05:02:36.303 INFO:teuthology.orchestra.run.smithi042.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2019-12-05T05:02:36.304 INFO:teuthology.orchestra.run.smithi042.stdout: = sunit=0 swidth=0 blks 2019-12-05T05:02:36.304 INFO:teuthology.orchestra.run.smithi042.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2019-12-05T05:02:36.304 INFO:teuthology.orchestra.run.smithi042.stdout:log =internal log bsize=4096 blocks=11443, version=2 2019-12-05T05:02:36.304 INFO:teuthology.orchestra.run.smithi042.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2019-12-05T05:02:36.305 INFO:teuthology.orchestra.run.smithi042.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2019-12-05T05:02:36.306 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi042.front.sepia.ceph.com -o noatime 2019-12-05T05:02:36.306 INFO:teuthology.orchestra.run.smithi042:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-0 2019-12-05T05:02:36.406 INFO:teuthology.orchestra.run.smithi042:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2019-12-05T05:02:36.492 INFO:teuthology.orchestra.run.smithi042.stderr:sudo: /sbin/restorecon: command not found 2019-12-05T05:02:36.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2019-12-05T05:02:36.493 INFO:teuthology.orchestra.run.smithi042:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2019-12-05T05:02:36.553 INFO:tasks.ceph:{'osd.1': '/dev/vg_nvme/lv_1', 'osd.0': '/dev/vg_nvme/lv_4', 'osd.3': '/dev/vg_nvme/lv_3', 'osd.2': '/dev/vg_nvme/lv_2'} 2019-12-05T05:02:36.553 INFO:tasks.ceph:{} 2019-12-05T05:02:36.554 INFO:tasks.ceph:osd.1 2019-12-05T05:02:36.554 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi042.front.sepia.ceph.com 2019-12-05T05:02:36.554 INFO:teuthology.orchestra.run.smithi042:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2019-12-05T05:02:36.667 INFO:teuthology.orchestra.run.smithi042.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2019-12-05T05:02:36.667 INFO:teuthology.orchestra.run.smithi042.stdout: = sectsz=512 attr=2, projid32bit=1 2019-12-05T05:02:36.667 INFO:teuthology.orchestra.run.smithi042.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2019-12-05T05:02:36.668 INFO:teuthology.orchestra.run.smithi042.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2019-12-05T05:02:36.668 INFO:teuthology.orchestra.run.smithi042.stdout: = sunit=0 swidth=0 blks 2019-12-05T05:02:36.668 INFO:teuthology.orchestra.run.smithi042.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2019-12-05T05:02:36.668 INFO:teuthology.orchestra.run.smithi042.stdout:log =internal log bsize=4096 blocks=11443, version=2 2019-12-05T05:02:36.668 INFO:teuthology.orchestra.run.smithi042.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2019-12-05T05:02:36.668 INFO:teuthology.orchestra.run.smithi042.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2019-12-05T05:02:36.669 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi042.front.sepia.ceph.com -o noatime 2019-12-05T05:02:36.669 INFO:teuthology.orchestra.run.smithi042:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-1 2019-12-05T05:02:36.757 INFO:teuthology.orchestra.run.smithi042:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2019-12-05T05:02:36.834 INFO:teuthology.orchestra.run.smithi042.stderr:sudo: /sbin/restorecon: command not found 2019-12-05T05:02:36.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2019-12-05T05:02:36.835 INFO:teuthology.orchestra.run.smithi042:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2019-12-05T05:02:36.913 INFO:tasks.ceph:{'osd.1': '/dev/vg_nvme/lv_1', 'osd.0': '/dev/vg_nvme/lv_4', 'osd.3': '/dev/vg_nvme/lv_3', 'osd.2': '/dev/vg_nvme/lv_2'} 2019-12-05T05:02:36.914 INFO:tasks.ceph:{} 2019-12-05T05:02:36.914 INFO:tasks.ceph:osd.2 2019-12-05T05:02:36.914 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi042.front.sepia.ceph.com 2019-12-05T05:02:36.914 INFO:teuthology.orchestra.run.smithi042:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2019-12-05T05:02:37.029 INFO:teuthology.orchestra.run.smithi042.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2019-12-05T05:02:37.029 INFO:teuthology.orchestra.run.smithi042.stdout: = sectsz=512 attr=2, projid32bit=1 2019-12-05T05:02:37.029 INFO:teuthology.orchestra.run.smithi042.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2019-12-05T05:02:37.029 INFO:teuthology.orchestra.run.smithi042.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2019-12-05T05:02:37.030 INFO:teuthology.orchestra.run.smithi042.stdout: = sunit=0 swidth=0 blks 2019-12-05T05:02:37.030 INFO:teuthology.orchestra.run.smithi042.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2019-12-05T05:02:37.030 INFO:teuthology.orchestra.run.smithi042.stdout:log =internal log bsize=4096 blocks=11443, version=2 2019-12-05T05:02:37.030 INFO:teuthology.orchestra.run.smithi042.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2019-12-05T05:02:37.030 INFO:teuthology.orchestra.run.smithi042.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2019-12-05T05:02:37.031 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi042.front.sepia.ceph.com -o noatime 2019-12-05T05:02:37.031 INFO:teuthology.orchestra.run.smithi042:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-2 2019-12-05T05:02:37.126 INFO:teuthology.orchestra.run.smithi042:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2019-12-05T05:02:37.207 INFO:teuthology.orchestra.run.smithi042.stderr:sudo: /sbin/restorecon: command not found 2019-12-05T05:02:37.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2019-12-05T05:02:37.208 INFO:teuthology.orchestra.run.smithi042:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2019-12-05T05:02:37.293 INFO:tasks.ceph:{'osd.1': '/dev/vg_nvme/lv_1', 'osd.0': '/dev/vg_nvme/lv_4', 'osd.3': '/dev/vg_nvme/lv_3', 'osd.2': '/dev/vg_nvme/lv_2'} 2019-12-05T05:02:37.293 INFO:tasks.ceph:{} 2019-12-05T05:02:37.293 INFO:tasks.ceph:osd.3 2019-12-05T05:02:37.294 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi042.front.sepia.ceph.com 2019-12-05T05:02:37.294 INFO:teuthology.orchestra.run.smithi042:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2019-12-05T05:02:37.405 INFO:teuthology.orchestra.run.smithi042.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2019-12-05T05:02:37.405 INFO:teuthology.orchestra.run.smithi042.stdout: = sectsz=512 attr=2, projid32bit=1 2019-12-05T05:02:37.405 INFO:teuthology.orchestra.run.smithi042.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2019-12-05T05:02:37.405 INFO:teuthology.orchestra.run.smithi042.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2019-12-05T05:02:37.405 INFO:teuthology.orchestra.run.smithi042.stdout: = sunit=0 swidth=0 blks 2019-12-05T05:02:37.406 INFO:teuthology.orchestra.run.smithi042.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2019-12-05T05:02:37.406 INFO:teuthology.orchestra.run.smithi042.stdout:log =internal log bsize=4096 blocks=11443, version=2 2019-12-05T05:02:37.406 INFO:teuthology.orchestra.run.smithi042.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2019-12-05T05:02:37.406 INFO:teuthology.orchestra.run.smithi042.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2019-12-05T05:02:37.407 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi042.front.sepia.ceph.com -o noatime 2019-12-05T05:02:37.407 INFO:teuthology.orchestra.run.smithi042:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-3 2019-12-05T05:02:37.492 INFO:teuthology.orchestra.run.smithi042:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2019-12-05T05:02:37.568 INFO:teuthology.orchestra.run.smithi042.stderr:sudo: /sbin/restorecon: command not found 2019-12-05T05:02:37.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2019-12-05T05:02:37.570 INFO:teuthology.orchestra.run.smithi042:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2019-12-05T05:02:37.675 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:02:37.670+0000 7f7a84533d80 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2019-12-05T05:02:37.675 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:02:37.674+0000 7f7a84533d80 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2019-12-05T05:02:37.675 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:02:37.674+0000 7f7a84533d80 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2019-12-05T05:02:37.675 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:02:37.674+0000 7f7a84533d80 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2019-12-05T05:02:37.676 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:02:37.674+0000 7f7a84533d80 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2019-12-05T05:02:37.679 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:02:37.674+0000 7f7a84533d80 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2019-12-05T05:02:40.108 INFO:teuthology.orchestra.run.smithi042:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2019-12-05T05:02:40.189 INFO:teuthology.orchestra.run.smithi042:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2019-12-05T05:02:40.296 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:02:40.294+0000 7fcdcf082d80 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2019-12-05T05:02:40.297 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:02:40.294+0000 7fcdcf082d80 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2019-12-05T05:02:40.297 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:02:40.294+0000 7fcdcf082d80 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2019-12-05T05:02:40.298 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:02:40.294+0000 7fcdcf082d80 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2019-12-05T05:02:40.298 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:02:40.294+0000 7fcdcf082d80 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2019-12-05T05:02:40.298 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:02:40.294+0000 7fcdcf082d80 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2019-12-05T05:02:42.730 INFO:teuthology.orchestra.run.smithi042:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2019-12-05T05:02:42.815 INFO:teuthology.orchestra.run.smithi042:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2019-12-05T05:02:42.924 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:02:42.922+0000 7fb274e14d80 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2019-12-05T05:02:42.924 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:02:42.922+0000 7fb274e14d80 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2019-12-05T05:02:42.924 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:02:42.922+0000 7fb274e14d80 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2019-12-05T05:02:42.924 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:02:42.922+0000 7fb274e14d80 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2019-12-05T05:02:42.925 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:02:42.922+0000 7fb274e14d80 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2019-12-05T05:02:42.925 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:02:42.922+0000 7fb274e14d80 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2019-12-05T05:02:45.372 INFO:teuthology.orchestra.run.smithi042:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2019-12-05T05:02:45.453 INFO:teuthology.orchestra.run.smithi042:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2019-12-05T05:02:45.564 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:02:45.562+0000 7f0690ba5d80 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2019-12-05T05:02:45.564 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:02:45.562+0000 7f0690ba5d80 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2019-12-05T05:02:45.565 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:02:45.562+0000 7f0690ba5d80 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2019-12-05T05:02:45.565 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:02:45.562+0000 7f0690ba5d80 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2019-12-05T05:02:45.565 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:02:45.562+0000 7f0690ba5d80 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2019-12-05T05:02:45.565 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:02:45.562+0000 7f0690ba5d80 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2019-12-05T05:02:48.011 INFO:teuthology.orchestra.run.smithi042:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2019-12-05T05:02:48.092 INFO:teuthology.orchestra.run.smithi040:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2019-12-05T05:02:48.115 INFO:tasks.ceph:{'osd.5': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_4', 'osd.7': '/dev/vg_nvme/lv_3', 'osd.6': '/dev/vg_nvme/lv_2'} 2019-12-05T05:02:48.115 INFO:tasks.ceph:{} 2019-12-05T05:02:48.116 INFO:tasks.ceph:osd.4 2019-12-05T05:02:48.116 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi040.front.sepia.ceph.com 2019-12-05T05:02:48.116 INFO:teuthology.orchestra.run.smithi040:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2019-12-05T05:02:48.666 INFO:teuthology.orchestra.run.smithi040.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2019-12-05T05:02:48.667 INFO:teuthology.orchestra.run.smithi040.stdout: = sectsz=512 attr=2, projid32bit=1 2019-12-05T05:02:48.667 INFO:teuthology.orchestra.run.smithi040.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2019-12-05T05:02:48.667 INFO:teuthology.orchestra.run.smithi040.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2019-12-05T05:02:48.667 INFO:teuthology.orchestra.run.smithi040.stdout: = sunit=0 swidth=0 blks 2019-12-05T05:02:48.667 INFO:teuthology.orchestra.run.smithi040.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2019-12-05T05:02:48.668 INFO:teuthology.orchestra.run.smithi040.stdout:log =internal log bsize=4096 blocks=11443, version=2 2019-12-05T05:02:48.668 INFO:teuthology.orchestra.run.smithi040.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2019-12-05T05:02:48.668 INFO:teuthology.orchestra.run.smithi040.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2019-12-05T05:02:48.669 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi040.front.sepia.ceph.com -o noatime 2019-12-05T05:02:48.669 INFO:teuthology.orchestra.run.smithi040:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-4 2019-12-05T05:02:48.754 INFO:teuthology.orchestra.run.smithi040:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2019-12-05T05:02:48.766 INFO:teuthology.orchestra.run.smithi040.stderr:sudo: /sbin/restorecon: command not found 2019-12-05T05:02:48.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2019-12-05T05:02:48.773 INFO:teuthology.orchestra.run.smithi040:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2019-12-05T05:02:48.864 INFO:tasks.ceph:{'osd.5': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_4', 'osd.7': '/dev/vg_nvme/lv_3', 'osd.6': '/dev/vg_nvme/lv_2'} 2019-12-05T05:02:48.865 INFO:tasks.ceph:{} 2019-12-05T05:02:48.865 INFO:tasks.ceph:osd.5 2019-12-05T05:02:48.865 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi040.front.sepia.ceph.com 2019-12-05T05:02:48.865 INFO:teuthology.orchestra.run.smithi040:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2019-12-05T05:02:49.391 INFO:teuthology.orchestra.run.smithi040.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2019-12-05T05:02:49.391 INFO:teuthology.orchestra.run.smithi040.stdout: = sectsz=512 attr=2, projid32bit=1 2019-12-05T05:02:49.391 INFO:teuthology.orchestra.run.smithi040.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2019-12-05T05:02:49.392 INFO:teuthology.orchestra.run.smithi040.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2019-12-05T05:02:49.392 INFO:teuthology.orchestra.run.smithi040.stdout: = sunit=0 swidth=0 blks 2019-12-05T05:02:49.392 INFO:teuthology.orchestra.run.smithi040.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2019-12-05T05:02:49.392 INFO:teuthology.orchestra.run.smithi040.stdout:log =internal log bsize=4096 blocks=11443, version=2 2019-12-05T05:02:49.392 INFO:teuthology.orchestra.run.smithi040.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2019-12-05T05:02:49.392 INFO:teuthology.orchestra.run.smithi040.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2019-12-05T05:02:49.394 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi040.front.sepia.ceph.com -o noatime 2019-12-05T05:02:49.394 INFO:teuthology.orchestra.run.smithi040:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-5 2019-12-05T05:02:49.425 INFO:teuthology.orchestra.run.smithi040:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2019-12-05T05:02:49.502 INFO:teuthology.orchestra.run.smithi040.stderr:sudo: /sbin/restorecon: command not found 2019-12-05T05:02:49.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2019-12-05T05:02:49.503 INFO:teuthology.orchestra.run.smithi040:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2019-12-05T05:02:49.589 INFO:tasks.ceph:{'osd.5': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_4', 'osd.7': '/dev/vg_nvme/lv_3', 'osd.6': '/dev/vg_nvme/lv_2'} 2019-12-05T05:02:49.589 INFO:tasks.ceph:{} 2019-12-05T05:02:49.589 INFO:tasks.ceph:osd.6 2019-12-05T05:02:49.589 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi040.front.sepia.ceph.com 2019-12-05T05:02:49.590 INFO:teuthology.orchestra.run.smithi040:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2019-12-05T05:02:50.143 INFO:teuthology.orchestra.run.smithi040.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2019-12-05T05:02:50.143 INFO:teuthology.orchestra.run.smithi040.stdout: = sectsz=512 attr=2, projid32bit=1 2019-12-05T05:02:50.143 INFO:teuthology.orchestra.run.smithi040.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2019-12-05T05:02:50.143 INFO:teuthology.orchestra.run.smithi040.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2019-12-05T05:02:50.143 INFO:teuthology.orchestra.run.smithi040.stdout: = sunit=0 swidth=0 blks 2019-12-05T05:02:50.144 INFO:teuthology.orchestra.run.smithi040.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2019-12-05T05:02:50.144 INFO:teuthology.orchestra.run.smithi040.stdout:log =internal log bsize=4096 blocks=11443, version=2 2019-12-05T05:02:50.144 INFO:teuthology.orchestra.run.smithi040.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2019-12-05T05:02:50.144 INFO:teuthology.orchestra.run.smithi040.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2019-12-05T05:02:50.145 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi040.front.sepia.ceph.com -o noatime 2019-12-05T05:02:50.145 INFO:teuthology.orchestra.run.smithi040:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-6 2019-12-05T05:02:50.193 INFO:teuthology.orchestra.run.smithi040:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2019-12-05T05:02:50.276 INFO:teuthology.orchestra.run.smithi040.stderr:sudo: /sbin/restorecon: command not found 2019-12-05T05:02:50.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2019-12-05T05:02:50.283 INFO:teuthology.orchestra.run.smithi040:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2019-12-05T05:02:50.363 INFO:tasks.ceph:{'osd.5': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_4', 'osd.7': '/dev/vg_nvme/lv_3', 'osd.6': '/dev/vg_nvme/lv_2'} 2019-12-05T05:02:50.363 INFO:tasks.ceph:{} 2019-12-05T05:02:50.363 INFO:tasks.ceph:osd.7 2019-12-05T05:02:50.363 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi040.front.sepia.ceph.com 2019-12-05T05:02:50.364 INFO:teuthology.orchestra.run.smithi040:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2019-12-05T05:02:50.908 INFO:teuthology.orchestra.run.smithi040.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2019-12-05T05:02:50.909 INFO:teuthology.orchestra.run.smithi040.stdout: = sectsz=512 attr=2, projid32bit=1 2019-12-05T05:02:50.909 INFO:teuthology.orchestra.run.smithi040.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2019-12-05T05:02:50.909 INFO:teuthology.orchestra.run.smithi040.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2019-12-05T05:02:50.909 INFO:teuthology.orchestra.run.smithi040.stdout: = sunit=0 swidth=0 blks 2019-12-05T05:02:50.909 INFO:teuthology.orchestra.run.smithi040.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2019-12-05T05:02:50.910 INFO:teuthology.orchestra.run.smithi040.stdout:log =internal log bsize=4096 blocks=11443, version=2 2019-12-05T05:02:50.910 INFO:teuthology.orchestra.run.smithi040.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2019-12-05T05:02:50.910 INFO:teuthology.orchestra.run.smithi040.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2019-12-05T05:02:50.910 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi040.front.sepia.ceph.com -o noatime 2019-12-05T05:02:50.911 INFO:teuthology.orchestra.run.smithi040:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-7 2019-12-05T05:02:50.990 INFO:teuthology.orchestra.run.smithi040:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2019-12-05T05:02:51.003 INFO:teuthology.orchestra.run.smithi040.stderr:sudo: /sbin/restorecon: command not found 2019-12-05T05:02:51.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2019-12-05T05:02:51.005 INFO:teuthology.orchestra.run.smithi040:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2019-12-05T05:02:51.117 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:02:51.116+0000 7f74b4f05d80 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2019-12-05T05:02:51.120 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:02:51.116+0000 7f74b4f05d80 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2019-12-05T05:02:51.121 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:02:51.116+0000 7f74b4f05d80 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2019-12-05T05:02:51.121 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:02:51.116+0000 7f74b4f05d80 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2019-12-05T05:02:51.121 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:02:51.116+0000 7f74b4f05d80 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2019-12-05T05:02:51.128 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:02:51.128+0000 7f74b4f05d80 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2019-12-05T05:02:53.587 INFO:teuthology.orchestra.run.smithi040:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2019-12-05T05:02:53.668 INFO:teuthology.orchestra.run.smithi040:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2019-12-05T05:02:53.773 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:02:53.772+0000 7faebad68d80 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2019-12-05T05:02:53.773 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:02:53.772+0000 7faebad68d80 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2019-12-05T05:02:53.773 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:02:53.772+0000 7faebad68d80 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2019-12-05T05:02:53.773 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:02:53.772+0000 7faebad68d80 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2019-12-05T05:02:53.774 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:02:53.772+0000 7faebad68d80 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2019-12-05T05:02:53.774 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:02:53.772+0000 7faebad68d80 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2019-12-05T05:02:56.232 INFO:teuthology.orchestra.run.smithi040:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2019-12-05T05:02:56.320 INFO:teuthology.orchestra.run.smithi040:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2019-12-05T05:02:56.424 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:02:56.420+0000 7fc34bb01d80 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2019-12-05T05:02:56.425 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:02:56.424+0000 7fc34bb01d80 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2019-12-05T05:02:56.425 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:02:56.424+0000 7fc34bb01d80 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2019-12-05T05:02:56.425 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:02:56.424+0000 7fc34bb01d80 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2019-12-05T05:02:56.425 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:02:56.424+0000 7fc34bb01d80 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2019-12-05T05:02:56.426 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:02:56.424+0000 7fc34bb01d80 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2019-12-05T05:02:58.864 INFO:teuthology.orchestra.run.smithi040:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2019-12-05T05:02:58.946 INFO:teuthology.orchestra.run.smithi040:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2019-12-05T05:02:59.057 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:02:59.052+0000 7f6f2e348d80 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2019-12-05T05:02:59.057 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:02:59.056+0000 7f6f2e348d80 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2019-12-05T05:02:59.057 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:02:59.056+0000 7f6f2e348d80 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2019-12-05T05:02:59.057 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:02:59.056+0000 7f6f2e348d80 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2019-12-05T05:02:59.058 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:02:59.056+0000 7f6f2e348d80 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2019-12-05T05:02:59.058 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:02:59.056+0000 7f6f2e348d80 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2019-12-05T05:03:01.510 INFO:teuthology.orchestra.run.smithi040:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2019-12-05T05:03:01.600 INFO:tasks.ceph:Reading keys from all nodes... 2019-12-05T05:03:01.600 INFO:teuthology.orchestra.run.smithi042:> mktemp 2019-12-05T05:03:01.620 INFO:teuthology.orchestra.run.smithi042.stdout:/tmp/tmp.q6uRLx4eB0 2019-12-05T05:03:01.620 INFO:teuthology.orchestra.run.smithi042:> sudo cp /var/lib/ceph/mgr/ceph-y/keyring /tmp/tmp.q6uRLx4eB0 2019-12-05T05:03:01.703 INFO:teuthology.orchestra.run.smithi042:> sudo chmod 0666 /tmp/tmp.q6uRLx4eB0 2019-12-05T05:03:01.841 DEBUG:teuthology.orchestra.remote:smithi042:/tmp/tmp.q6uRLx4eB0 is 56B 2019-12-05T05:03:01.853 INFO:teuthology.orchestra.run.smithi042:> rm -fr /tmp/tmp.q6uRLx4eB0 2019-12-05T05:03:01.865 INFO:teuthology.orchestra.run.smithi042:> mktemp 2019-12-05T05:03:01.944 INFO:teuthology.orchestra.run.smithi042.stdout:/tmp/tmp.MDYnNcBe9N 2019-12-05T05:03:01.944 INFO:teuthology.orchestra.run.smithi042:> sudo cp /var/lib/ceph/mds/ceph-a/keyring /tmp/tmp.MDYnNcBe9N 2019-12-05T05:03:02.025 INFO:teuthology.orchestra.run.smithi042:> sudo chmod 0666 /tmp/tmp.MDYnNcBe9N 2019-12-05T05:03:02.189 DEBUG:teuthology.orchestra.remote:smithi042:/tmp/tmp.MDYnNcBe9N is 56B 2019-12-05T05:03:02.201 INFO:teuthology.orchestra.run.smithi042:> rm -fr /tmp/tmp.MDYnNcBe9N 2019-12-05T05:03:02.210 INFO:teuthology.orchestra.run.smithi042:> mktemp 2019-12-05T05:03:02.281 INFO:teuthology.orchestra.run.smithi042.stdout:/tmp/tmp.RPricde0Rt 2019-12-05T05:03:02.281 INFO:teuthology.orchestra.run.smithi042:> sudo cp /var/lib/ceph/osd/ceph-0/keyring /tmp/tmp.RPricde0Rt 2019-12-05T05:03:02.364 INFO:teuthology.orchestra.run.smithi042:> sudo chmod 0666 /tmp/tmp.RPricde0Rt 2019-12-05T05:03:02.518 DEBUG:teuthology.orchestra.remote:smithi042:/tmp/tmp.RPricde0Rt is 56B 2019-12-05T05:03:02.530 INFO:teuthology.orchestra.run.smithi042:> rm -fr /tmp/tmp.RPricde0Rt 2019-12-05T05:03:02.538 INFO:teuthology.orchestra.run.smithi042:> mktemp 2019-12-05T05:03:02.610 INFO:teuthology.orchestra.run.smithi042.stdout:/tmp/tmp.MK2LexiDMi 2019-12-05T05:03:02.610 INFO:teuthology.orchestra.run.smithi042:> sudo cp /var/lib/ceph/osd/ceph-1/keyring /tmp/tmp.MK2LexiDMi 2019-12-05T05:03:02.691 INFO:teuthology.orchestra.run.smithi042:> sudo chmod 0666 /tmp/tmp.MK2LexiDMi 2019-12-05T05:03:02.853 DEBUG:teuthology.orchestra.remote:smithi042:/tmp/tmp.MK2LexiDMi is 56B 2019-12-05T05:03:02.865 INFO:teuthology.orchestra.run.smithi042:> rm -fr /tmp/tmp.MK2LexiDMi 2019-12-05T05:03:02.872 INFO:teuthology.orchestra.run.smithi042:> mktemp 2019-12-05T05:03:02.946 INFO:teuthology.orchestra.run.smithi042.stdout:/tmp/tmp.kbcsBVjEBm 2019-12-05T05:03:02.947 INFO:teuthology.orchestra.run.smithi042:> sudo cp /var/lib/ceph/osd/ceph-2/keyring /tmp/tmp.kbcsBVjEBm 2019-12-05T05:03:03.027 INFO:teuthology.orchestra.run.smithi042:> sudo chmod 0666 /tmp/tmp.kbcsBVjEBm 2019-12-05T05:03:03.187 DEBUG:teuthology.orchestra.remote:smithi042:/tmp/tmp.kbcsBVjEBm is 56B 2019-12-05T05:03:03.199 INFO:teuthology.orchestra.run.smithi042:> rm -fr /tmp/tmp.kbcsBVjEBm 2019-12-05T05:03:03.206 INFO:teuthology.orchestra.run.smithi042:> mktemp 2019-12-05T05:03:03.277 INFO:teuthology.orchestra.run.smithi042.stdout:/tmp/tmp.eB6mfFi7vu 2019-12-05T05:03:03.277 INFO:teuthology.orchestra.run.smithi042:> sudo cp /var/lib/ceph/osd/ceph-3/keyring /tmp/tmp.eB6mfFi7vu 2019-12-05T05:03:03.368 INFO:teuthology.orchestra.run.smithi042:> sudo chmod 0666 /tmp/tmp.eB6mfFi7vu 2019-12-05T05:03:03.423 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:03:03.434 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:03:03.588 DEBUG:teuthology.orchestra.remote:smithi042:/tmp/tmp.eB6mfFi7vu is 56B 2019-12-05T05:03:03.741 INFO:teuthology.orchestra.run.smithi042:> rm -fr /tmp/tmp.eB6mfFi7vu 2019-12-05T05:03:03.748 INFO:teuthology.orchestra.run.smithi040:> mktemp 2019-12-05T05:03:03.758 INFO:teuthology.orchestra.run.smithi040.stdout:/tmp/tmp.8BZqGrSODo 2019-12-05T05:03:03.758 INFO:teuthology.orchestra.run.smithi040:> sudo cp /var/lib/ceph/mgr/ceph-x/keyring /tmp/tmp.8BZqGrSODo 2019-12-05T05:03:03.840 INFO:teuthology.orchestra.run.smithi040:> sudo chmod 0666 /tmp/tmp.8BZqGrSODo 2019-12-05T05:03:03.995 DEBUG:teuthology.orchestra.remote:smithi040:/tmp/tmp.8BZqGrSODo is 56B 2019-12-05T05:03:04.007 INFO:teuthology.orchestra.run.smithi040:> rm -fr /tmp/tmp.8BZqGrSODo 2019-12-05T05:03:04.014 INFO:teuthology.orchestra.run.smithi040:> mktemp 2019-12-05T05:03:04.088 INFO:teuthology.orchestra.run.smithi040.stdout:/tmp/tmp.Mzs7MOcZ6N 2019-12-05T05:03:04.088 INFO:teuthology.orchestra.run.smithi040:> sudo cp /var/lib/ceph/mds/ceph-b/keyring /tmp/tmp.Mzs7MOcZ6N 2019-12-05T05:03:04.171 INFO:teuthology.orchestra.run.smithi040:> sudo chmod 0666 /tmp/tmp.Mzs7MOcZ6N 2019-12-05T05:03:04.332 DEBUG:teuthology.orchestra.remote:smithi040:/tmp/tmp.Mzs7MOcZ6N is 56B 2019-12-05T05:03:04.356 INFO:teuthology.orchestra.run.smithi040:> rm -fr /tmp/tmp.Mzs7MOcZ6N 2019-12-05T05:03:04.430 INFO:teuthology.orchestra.run.smithi040:> mktemp 2019-12-05T05:03:04.503 INFO:teuthology.orchestra.run.smithi040.stdout:/tmp/tmp.483B0MoqtT 2019-12-05T05:03:04.503 INFO:teuthology.orchestra.run.smithi040:> sudo cp /var/lib/ceph/osd/ceph-4/keyring /tmp/tmp.483B0MoqtT 2019-12-05T05:03:04.587 INFO:teuthology.orchestra.run.smithi040:> sudo chmod 0666 /tmp/tmp.483B0MoqtT 2019-12-05T05:03:04.723 DEBUG:teuthology.orchestra.remote:smithi040:/tmp/tmp.483B0MoqtT is 56B 2019-12-05T05:03:04.734 INFO:teuthology.orchestra.run.smithi040:> rm -fr /tmp/tmp.483B0MoqtT 2019-12-05T05:03:04.748 INFO:teuthology.orchestra.run.smithi040:> mktemp 2019-12-05T05:03:04.820 INFO:teuthology.orchestra.run.smithi040.stdout:/tmp/tmp.f2EtFWaFZ2 2019-12-05T05:03:04.820 INFO:teuthology.orchestra.run.smithi040:> sudo cp /var/lib/ceph/osd/ceph-5/keyring /tmp/tmp.f2EtFWaFZ2 2019-12-05T05:03:04.910 INFO:teuthology.orchestra.run.smithi040:> sudo chmod 0666 /tmp/tmp.f2EtFWaFZ2 2019-12-05T05:03:05.064 DEBUG:teuthology.orchestra.remote:smithi040:/tmp/tmp.f2EtFWaFZ2 is 56B 2019-12-05T05:03:05.077 INFO:teuthology.orchestra.run.smithi040:> rm -fr /tmp/tmp.f2EtFWaFZ2 2019-12-05T05:03:05.084 INFO:teuthology.orchestra.run.smithi040:> mktemp 2019-12-05T05:03:05.156 INFO:teuthology.orchestra.run.smithi040.stdout:/tmp/tmp.hSCnVAPLEf 2019-12-05T05:03:05.156 INFO:teuthology.orchestra.run.smithi040:> sudo cp /var/lib/ceph/osd/ceph-6/keyring /tmp/tmp.hSCnVAPLEf 2019-12-05T05:03:05.239 INFO:teuthology.orchestra.run.smithi040:> sudo chmod 0666 /tmp/tmp.hSCnVAPLEf 2019-12-05T05:03:05.399 DEBUG:teuthology.orchestra.remote:smithi040:/tmp/tmp.hSCnVAPLEf is 56B 2019-12-05T05:03:05.411 INFO:teuthology.orchestra.run.smithi040:> rm -fr /tmp/tmp.hSCnVAPLEf 2019-12-05T05:03:05.419 INFO:teuthology.orchestra.run.smithi040:> mktemp 2019-12-05T05:03:05.492 INFO:teuthology.orchestra.run.smithi040.stdout:/tmp/tmp.LClnziWNwY 2019-12-05T05:03:05.493 INFO:teuthology.orchestra.run.smithi040:> sudo cp /var/lib/ceph/osd/ceph-7/keyring /tmp/tmp.LClnziWNwY 2019-12-05T05:03:05.584 INFO:teuthology.orchestra.run.smithi040:> sudo chmod 0666 /tmp/tmp.LClnziWNwY 2019-12-05T05:03:05.740 DEBUG:teuthology.orchestra.remote:smithi040:/tmp/tmp.LClnziWNwY is 56B 2019-12-05T05:03:05.754 INFO:teuthology.orchestra.run.smithi040:> rm -fr /tmp/tmp.LClnziWNwY 2019-12-05T05:03:05.776 DEBUG:teuthology.orchestra.remote:smithi042:/etc/ceph/ceph.client.0.keyring is 59B 2019-12-05T05:03:05.789 INFO:tasks.ceph:Adding keys to all mons... 2019-12-05T05:03:05.790 INFO:teuthology.orchestra.run.smithi040:> sudo tee -a /etc/ceph/ceph.keyring 2019-12-05T05:03:05.824 INFO:teuthology.orchestra.run.smithi042:> sudo tee -a /etc/ceph/ceph.keyring 2019-12-05T05:03:05.835 INFO:teuthology.orchestra.run.smithi040.stdout:[mgr.y] 2019-12-05T05:03:05.835 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQBrj+hdsFbUGhAA9O0a1gRTog4zYQyuxyDJZg== 2019-12-05T05:03:05.835 INFO:teuthology.orchestra.run.smithi040.stdout:[mds.a] 2019-12-05T05:03:05.836 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQBrj+hd1GEwMhAATX2v7fihrkjL27EUsy5jIw== 2019-12-05T05:03:05.836 INFO:teuthology.orchestra.run.smithi040.stdout:[osd.0] 2019-12-05T05:03:05.836 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQBtj+hdx3o4KBAALJZPuwXojuZCfJzAAJkD2w== 2019-12-05T05:03:05.836 INFO:teuthology.orchestra.run.smithi040.stdout:[osd.1] 2019-12-05T05:03:05.836 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQBwj+hdXk2iERAAmizYtjsV/tNHMjGTdyv3rQ== 2019-12-05T05:03:05.836 INFO:teuthology.orchestra.run.smithi040.stdout:[osd.2] 2019-12-05T05:03:05.836 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQByj+hdQK8MNxAA5x6p8EHqfWVcL2kVlU3yaA== 2019-12-05T05:03:05.837 INFO:teuthology.orchestra.run.smithi040.stdout:[osd.3] 2019-12-05T05:03:05.837 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQB1j+hd1bCgIRAAO8J2gtl7TRrIdmMFX/h36w== 2019-12-05T05:03:05.837 INFO:teuthology.orchestra.run.smithi040.stdout:[mgr.x] 2019-12-05T05:03:05.837 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQBrj+hdh+nSHRAAvaGzOnExeWLKppCoEJ4tCQ== 2019-12-05T05:03:05.837 INFO:teuthology.orchestra.run.smithi040.stdout:[mds.b] 2019-12-05T05:03:05.838 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQBrj+hdq4UOOhAAlVjUSheosNK1a6u9yk5bIg== 2019-12-05T05:03:05.838 INFO:teuthology.orchestra.run.smithi040.stdout:[osd.4] 2019-12-05T05:03:05.838 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQB7j+hdtksPBxAAPAGs4oSFqwFTbkq/Jg848Q== 2019-12-05T05:03:05.838 INFO:teuthology.orchestra.run.smithi040.stdout:[osd.5] 2019-12-05T05:03:05.838 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQB9j+hdZ5UdLhAA637BibaxiO+TvobQMOep9Q== 2019-12-05T05:03:05.838 INFO:teuthology.orchestra.run.smithi040.stdout:[osd.6] 2019-12-05T05:03:05.838 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQCAj+hdN29WGRAAKyilk6halM3mhTPIboAc6w== 2019-12-05T05:03:05.839 INFO:teuthology.orchestra.run.smithi040.stdout:[osd.7] 2019-12-05T05:03:05.839 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQCDj+hdl/BlAxAA0dT/vUxqwdkvmXEJXe0XSQ== 2019-12-05T05:03:05.839 INFO:teuthology.orchestra.run.smithi040.stdout:[client.0] 2019-12-05T05:03:05.839 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQBsj+hdWnhNBhAA2lLq413iBZlBYN0z8Uvh6g== 2019-12-05T05:03:05.878 INFO:teuthology.orchestra.run.smithi042.stdout:[mgr.y] 2019-12-05T05:03:05.878 INFO:teuthology.orchestra.run.smithi042.stdout: key = AQBrj+hdsFbUGhAA9O0a1gRTog4zYQyuxyDJZg== 2019-12-05T05:03:05.878 INFO:teuthology.orchestra.run.smithi042.stdout:[mds.a] 2019-12-05T05:03:05.878 INFO:teuthology.orchestra.run.smithi042.stdout: key = AQBrj+hd1GEwMhAATX2v7fihrkjL27EUsy5jIw== 2019-12-05T05:03:05.879 INFO:teuthology.orchestra.run.smithi042.stdout:[osd.0] 2019-12-05T05:03:05.879 INFO:teuthology.orchestra.run.smithi042.stdout: key = AQBtj+hdx3o4KBAALJZPuwXojuZCfJzAAJkD2w== 2019-12-05T05:03:05.879 INFO:teuthology.orchestra.run.smithi042.stdout:[osd.1] 2019-12-05T05:03:05.879 INFO:teuthology.orchestra.run.smithi042.stdout: key = AQBwj+hdXk2iERAAmizYtjsV/tNHMjGTdyv3rQ== 2019-12-05T05:03:05.879 INFO:teuthology.orchestra.run.smithi042.stdout:[osd.2] 2019-12-05T05:03:05.879 INFO:teuthology.orchestra.run.smithi042.stdout: key = AQByj+hdQK8MNxAA5x6p8EHqfWVcL2kVlU3yaA== 2019-12-05T05:03:05.879 INFO:teuthology.orchestra.run.smithi042.stdout:[osd.3] 2019-12-05T05:03:05.880 INFO:teuthology.orchestra.run.smithi042.stdout: key = AQB1j+hd1bCgIRAAO8J2gtl7TRrIdmMFX/h36w== 2019-12-05T05:03:05.880 INFO:teuthology.orchestra.run.smithi042.stdout:[mgr.x] 2019-12-05T05:03:05.880 INFO:teuthology.orchestra.run.smithi042.stdout: key = AQBrj+hdh+nSHRAAvaGzOnExeWLKppCoEJ4tCQ== 2019-12-05T05:03:05.880 INFO:teuthology.orchestra.run.smithi042.stdout:[mds.b] 2019-12-05T05:03:05.880 INFO:teuthology.orchestra.run.smithi042.stdout: key = AQBrj+hdq4UOOhAAlVjUSheosNK1a6u9yk5bIg== 2019-12-05T05:03:05.880 INFO:teuthology.orchestra.run.smithi042.stdout:[osd.4] 2019-12-05T05:03:05.881 INFO:teuthology.orchestra.run.smithi042.stdout: key = AQB7j+hdtksPBxAAPAGs4oSFqwFTbkq/Jg848Q== 2019-12-05T05:03:05.881 INFO:teuthology.orchestra.run.smithi042.stdout:[osd.5] 2019-12-05T05:03:05.881 INFO:teuthology.orchestra.run.smithi042.stdout: key = AQB9j+hdZ5UdLhAA637BibaxiO+TvobQMOep9Q== 2019-12-05T05:03:05.881 INFO:teuthology.orchestra.run.smithi042.stdout:[osd.6] 2019-12-05T05:03:05.881 INFO:teuthology.orchestra.run.smithi042.stdout: key = AQCAj+hdN29WGRAAKyilk6halM3mhTPIboAc6w== 2019-12-05T05:03:05.881 INFO:teuthology.orchestra.run.smithi042.stdout:[osd.7] 2019-12-05T05:03:05.881 INFO:teuthology.orchestra.run.smithi042.stdout: key = AQCDj+hdl/BlAxAA0dT/vUxqwdkvmXEJXe0XSQ== 2019-12-05T05:03:05.882 INFO:teuthology.orchestra.run.smithi042.stdout:[client.0] 2019-12-05T05:03:05.882 INFO:teuthology.orchestra.run.smithi042.stdout: key = AQBsj+hdWnhNBhAA2lLq413iBZlBYN0z8Uvh6g== 2019-12-05T05:03:05.882 INFO:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mds 'allow *' --cap osd 'allow *' --cap mon 'allow profile mgr' 2019-12-05T05:03:05.886 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mds 'allow *' --cap osd 'allow *' --cap mon 'allow profile mgr' 2019-12-05T05:03:05.996 INFO:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mds allow --cap osd 'allow *' --cap mgr 'allow *' --cap mon 'allow *' 2019-12-05T05:03:06.002 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mds allow --cap osd 'allow *' --cap mgr 'allow *' --cap mon 'allow *' 2019-12-05T05:03:06.126 INFO:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2019-12-05T05:03:06.129 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2019-12-05T05:03:06.239 INFO:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2019-12-05T05:03:06.242 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2019-12-05T05:03:06.282 INFO:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2019-12-05T05:03:06.330 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2019-12-05T05:03:06.374 INFO:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2019-12-05T05:03:06.441 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2019-12-05T05:03:06.486 INFO:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mds 'allow *' --cap osd 'allow *' --cap mon 'allow profile mgr' 2019-12-05T05:03:06.554 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mds 'allow *' --cap osd 'allow *' --cap mon 'allow profile mgr' 2019-12-05T05:03:06.599 INFO:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mds allow --cap osd 'allow *' --cap mgr 'allow *' --cap mon 'allow *' 2019-12-05T05:03:06.684 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mds allow --cap osd 'allow *' --cap mgr 'allow *' --cap mon 'allow *' 2019-12-05T05:03:06.733 INFO:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2019-12-05T05:03:06.818 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2019-12-05T05:03:06.869 INFO:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2019-12-05T05:03:06.940 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2019-12-05T05:03:06.993 INFO:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2019-12-05T05:03:07.031 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2019-12-05T05:03:07.102 INFO:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2019-12-05T05:03:07.138 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2019-12-05T05:03:07.210 INFO:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mds allow --cap osd 'allow rwx' --cap mgr 'allow r' --cap mon 'allow rw' 2019-12-05T05:03:07.234 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mds allow --cap osd 'allow rwx' --cap mgr 'allow r' --cap mon 'allow rw' 2019-12-05T05:03:07.314 INFO:tasks.ceph:Running mkfs on mon nodes... 2019-12-05T05:03:07.314 INFO:teuthology.orchestra.run.smithi042:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2019-12-05T05:03:07.330 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2019-12-05T05:03:07.455 INFO:teuthology.orchestra.run.smithi042:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2019-12-05T05:03:07.476 INFO:teuthology.orchestra.run.smithi040:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2019-12-05T05:03:07.498 INFO:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2019-12-05T05:03:07.646 INFO:teuthology.orchestra.run.smithi040:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2019-12-05T05:03:07.665 INFO:teuthology.orchestra.run.smithi040:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2019-12-05T05:03:07.748 INFO:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2019-12-05T05:03:07.892 INFO:teuthology.orchestra.run.smithi040:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2019-12-05T05:03:07.909 INFO:teuthology.orchestra.run.smithi040:> rm -- /home/ubuntu/cephtest/ceph.monmap 2019-12-05T05:03:07.976 INFO:teuthology.orchestra.run.smithi042:> rm -- /home/ubuntu/cephtest/ceph.monmap 2019-12-05T05:03:07.983 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2019-12-05T05:03:07.984 INFO:tasks.ceph.mon.a:Restarting daemon 2019-12-05T05:03:07.984 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2019-12-05T05:03:08.054 INFO:tasks.ceph.mon.a:Started 2019-12-05T05:03:08.054 INFO:tasks.ceph.mon.b:Restarting daemon 2019-12-05T05:03:08.054 INFO:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2019-12-05T05:03:08.058 INFO:tasks.ceph.mon.b:Started 2019-12-05T05:03:08.058 INFO:tasks.ceph.mon.c:Restarting daemon 2019-12-05T05:03:08.058 INFO:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2019-12-05T05:03:08.061 INFO:tasks.ceph.mon.c:Started 2019-12-05T05:03:08.061 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2019-12-05T05:03:08.062 INFO:tasks.ceph.mgr.y:Restarting daemon 2019-12-05T05:03:08.062 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2019-12-05T05:03:08.067 INFO:tasks.ceph.mgr.y:Started 2019-12-05T05:03:08.068 INFO:tasks.ceph.mgr.x:Restarting daemon 2019-12-05T05:03:08.068 INFO:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2019-12-05T05:03:08.074 INFO:tasks.ceph.mgr.x:Started 2019-12-05T05:03:08.074 INFO:tasks.ceph:Setting crush tunables to default 2019-12-05T05:03:08.075 INFO:teuthology.orchestra.run.smithi042:> sudo ceph --cluster ceph osd crush tunables default 2019-12-05T05:03:13.263 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:08.158+0000 7f5b5fe6f700 1 Processor -- start 2019-12-05T05:03:13.263 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:08.158+0000 7f5b5fe6f700 1 -- start start 2019-12-05T05:03:13.263 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:08.158+0000 7f5b5fe6f700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5b5808df70 0x7f5b5809f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:13.264 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:08.158+0000 7f5b5fe6f700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5b5809f820 0x7f5b5814efd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:13.264 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:08.158+0000 7f5b5fe6f700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b5809fc50 con 0x7f5b580a7730 2019-12-05T05:03:13.264 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:08.158+0000 7f5b5fe6f700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b580a0af0 con 0x7f5b5808df70 2019-12-05T05:03:13.264 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:08.158+0000 7f5b5e40c700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f5b580a7730 0x7f5b5807e710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:52614/0 (socket says 172.21.15.42:52614) 2019-12-05T05:03:13.264 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:08.158+0000 7f5b5fe6f700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b5808e5a0 con 0x7f5b5809f820 2019-12-05T05:03:13.264 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:08.158+0000 7f5b5e40c700 1 -- 172.21.15.42:0/2029094007 learned_addr learned my addr 172.21.15.42:0/2029094007 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:03:13.264 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:08.158+0000 7f5b5dc0b700 1 --2- 172.21.15.42:0/2029094007 >> v2:172.21.15.42:3300/0 conn(0x7f5b5808df70 0x7f5b5809f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:13.265 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:08.162+0000 7f5b5ec0d700 1 --2- 172.21.15.42:0/2029094007 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5b5809f820 0x7f5b5814efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:13.265 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:08.162+0000 7f5b5dc0b700 1 -- 172.21.15.42:0/2029094007 >> v2:172.21.15.42:3300/0 conn(0x7f5b5808df70 msgr2=0x7f5b5809f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2019-12-05T05:03:13.265 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:08.162+0000 7f5b5dc0b700 1 -- 172.21.15.42:0/2029094007 >> v2:172.21.15.42:3300/0 conn(0x7f5b5808df70 msgr2=0x7f5b5809f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2019-12-05T05:03:13.265 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:08.162+0000 7f5b5dc0b700 1 --2- 172.21.15.42:0/2029094007 >> v2:172.21.15.42:3300/0 conn(0x7f5b5808df70 0x7f5b5809f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-12-05T05:03:13.265 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:08.162+0000 7f5b5dc0b700 1 --2- 172.21.15.42:0/2029094007 >> v2:172.21.15.42:3300/0 conn(0x7f5b5808df70 0x7f5b5809f2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rx=0 tx=0)._fault waiting 0.200000 2019-12-05T05:03:13.265 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:08.162+0000 7f5b5ec0d700 1 -- 172.21.15.42:0/2029094007 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5b5809f820 msgr2=0x7f5b5814efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2019-12-05T05:03:13.266 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:08.162+0000 7f5b5ec0d700 1 -- 172.21.15.42:0/2029094007 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5b5809f820 msgr2=0x7f5b5814efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2019-12-05T05:03:13.266 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:08.162+0000 7f5b5ec0d700 1 --2- 172.21.15.42:0/2029094007 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5b5809f820 0x7f5b5814efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-12-05T05:03:13.266 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:08.162+0000 7f5b5ec0d700 1 --2- 172.21.15.42:0/2029094007 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5b5809f820 0x7f5b5814efd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rx=0 tx=0)._fault waiting 0.200000 2019-12-05T05:03:13.266 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:08.366+0000 7f5b5dc0b700 1 --2- 172.21.15.42:0/2029094007 >> v2:172.21.15.42:3300/0 conn(0x7f5b5808df70 0x7f5b5809f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:13.266 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:08.366+0000 7f5b5ec0d700 1 --2- 172.21.15.42:0/2029094007 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5b5809f820 0x7f5b5814efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:13.267 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:08.366+0000 7f5b5dc0b700 1 -- 172.21.15.42:0/2029094007 >> v2:172.21.15.42:3300/0 conn(0x7f5b5808df70 msgr2=0x7f5b5809f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2019-12-05T05:03:13.267 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:08.366+0000 7f5b5dc0b700 1 -- 172.21.15.42:0/2029094007 >> v2:172.21.15.42:3300/0 conn(0x7f5b5808df70 msgr2=0x7f5b5809f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2019-12-05T05:03:13.267 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:08.366+0000 7f5b5dc0b700 1 --2- 172.21.15.42:0/2029094007 >> v2:172.21.15.42:3300/0 conn(0x7f5b5808df70 0x7f5b5809f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-12-05T05:03:13.267 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:08.366+0000 7f5b5dc0b700 1 --2- 172.21.15.42:0/2029094007 >> v2:172.21.15.42:3300/0 conn(0x7f5b5808df70 0x7f5b5809f2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rx=0 tx=0)._fault waiting 0.400000 2019-12-05T05:03:13.267 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:08.366+0000 7f5b5ec0d700 1 -- 172.21.15.42:0/2029094007 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5b5809f820 msgr2=0x7f5b5814efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2019-12-05T05:03:13.267 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:08.366+0000 7f5b5ec0d700 1 -- 172.21.15.42:0/2029094007 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5b5809f820 msgr2=0x7f5b5814efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2019-12-05T05:03:13.267 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:08.366+0000 7f5b5ec0d700 1 --2- 172.21.15.42:0/2029094007 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5b5809f820 0x7f5b5814efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-12-05T05:03:13.268 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:08.366+0000 7f5b5ec0d700 1 --2- 172.21.15.42:0/2029094007 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5b5809f820 0x7f5b5814efd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rx=0 tx=0)._fault waiting 0.400000 2019-12-05T05:03:13.268 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:08.770+0000 7f5b5dc0b700 1 --2- 172.21.15.42:0/2029094007 >> v2:172.21.15.42:3300/0 conn(0x7f5b5808df70 0x7f5b5809f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:13.268 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:08.770+0000 7f5b5ec0d700 1 --2- 172.21.15.42:0/2029094007 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5b5809f820 0x7f5b5814efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:13.268 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:08.770+0000 7f5b5dc0b700 1 -- 172.21.15.42:0/2029094007 >> v2:172.21.15.42:3300/0 conn(0x7f5b5808df70 msgr2=0x7f5b5809f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2019-12-05T05:03:13.268 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:08.770+0000 7f5b5dc0b700 1 -- 172.21.15.42:0/2029094007 >> v2:172.21.15.42:3300/0 conn(0x7f5b5808df70 msgr2=0x7f5b5809f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2019-12-05T05:03:13.268 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:08.770+0000 7f5b5dc0b700 1 --2- 172.21.15.42:0/2029094007 >> v2:172.21.15.42:3300/0 conn(0x7f5b5808df70 0x7f5b5809f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-12-05T05:03:13.269 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:08.770+0000 7f5b5ec0d700 1 -- 172.21.15.42:0/2029094007 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5b5809f820 msgr2=0x7f5b5814efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2019-12-05T05:03:13.269 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:08.770+0000 7f5b5ec0d700 1 -- 172.21.15.42:0/2029094007 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5b5809f820 msgr2=0x7f5b5814efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2019-12-05T05:03:13.269 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:08.770+0000 7f5b5dc0b700 1 --2- 172.21.15.42:0/2029094007 >> v2:172.21.15.42:3300/0 conn(0x7f5b5808df70 0x7f5b5809f2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rx=0 tx=0)._fault waiting 0.800000 2019-12-05T05:03:13.269 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:08.770+0000 7f5b5ec0d700 1 --2- 172.21.15.42:0/2029094007 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5b5809f820 0x7f5b5814efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-12-05T05:03:13.269 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:08.770+0000 7f5b5ec0d700 1 --2- 172.21.15.42:0/2029094007 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5b5809f820 0x7f5b5814efd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rx=0 tx=0)._fault waiting 0.800000 2019-12-05T05:03:13.269 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:09.574+0000 7f5b5dc0b700 1 --2- 172.21.15.42:0/2029094007 >> v2:172.21.15.42:3300/0 conn(0x7f5b5808df70 0x7f5b5809f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:13.270 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:09.574+0000 7f5b5ec0d700 1 --2- 172.21.15.42:0/2029094007 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5b5809f820 0x7f5b5814efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:13.270 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:09.574+0000 7f5b5dc0b700 1 -- 172.21.15.42:0/2029094007 >> v2:172.21.15.42:3300/0 conn(0x7f5b5808df70 msgr2=0x7f5b5809f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2019-12-05T05:03:13.270 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:09.574+0000 7f5b5dc0b700 1 -- 172.21.15.42:0/2029094007 >> v2:172.21.15.42:3300/0 conn(0x7f5b5808df70 msgr2=0x7f5b5809f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2019-12-05T05:03:13.270 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:09.574+0000 7f5b5dc0b700 1 --2- 172.21.15.42:0/2029094007 >> v2:172.21.15.42:3300/0 conn(0x7f5b5808df70 0x7f5b5809f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-12-05T05:03:13.271 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:09.574+0000 7f5b5ec0d700 1 -- 172.21.15.42:0/2029094007 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5b5809f820 msgr2=0x7f5b5814efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2019-12-05T05:03:13.271 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:09.574+0000 7f5b5ec0d700 1 -- 172.21.15.42:0/2029094007 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5b5809f820 msgr2=0x7f5b5814efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2019-12-05T05:03:13.271 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:09.574+0000 7f5b5dc0b700 1 --2- 172.21.15.42:0/2029094007 >> v2:172.21.15.42:3300/0 conn(0x7f5b5808df70 0x7f5b5809f2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rx=0 tx=0)._fault waiting 1.600000 2019-12-05T05:03:13.271 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:09.574+0000 7f5b5ec0d700 1 --2- 172.21.15.42:0/2029094007 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5b5809f820 0x7f5b5814efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-12-05T05:03:13.272 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:09.574+0000 7f5b5ec0d700 1 --2- 172.21.15.42:0/2029094007 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5b5809f820 0x7f5b5814efd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rx=0 tx=0)._fault waiting 1.600000 2019-12-05T05:03:13.272 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:11.178+0000 7f5b5dc0b700 1 --2- 172.21.15.42:0/2029094007 >> v2:172.21.15.42:3300/0 conn(0x7f5b5808df70 0x7f5b5809f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:13.274 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:11.178+0000 7f5b5ec0d700 1 --2- 172.21.15.42:0/2029094007 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5b5809f820 0x7f5b5814efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:13.274 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:11.178+0000 7f5b5dc0b700 1 -- 172.21.15.42:0/2029094007 >> v2:172.21.15.42:3300/0 conn(0x7f5b5808df70 msgr2=0x7f5b5809f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2019-12-05T05:03:13.274 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:11.178+0000 7f5b5dc0b700 1 -- 172.21.15.42:0/2029094007 >> v2:172.21.15.42:3300/0 conn(0x7f5b5808df70 msgr2=0x7f5b5809f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2019-12-05T05:03:13.274 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:11.178+0000 7f5b5dc0b700 1 --2- 172.21.15.42:0/2029094007 >> v2:172.21.15.42:3300/0 conn(0x7f5b5808df70 0x7f5b5809f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-12-05T05:03:13.275 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:11.178+0000 7f5b5ec0d700 1 -- 172.21.15.42:0/2029094007 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5b5809f820 msgr2=0x7f5b5814efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2019-12-05T05:03:13.275 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:11.178+0000 7f5b5ec0d700 1 -- 172.21.15.42:0/2029094007 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5b5809f820 msgr2=0x7f5b5814efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2019-12-05T05:03:13.275 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:11.178+0000 7f5b5dc0b700 1 --2- 172.21.15.42:0/2029094007 >> v2:172.21.15.42:3300/0 conn(0x7f5b5808df70 0x7f5b5809f2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rx=0 tx=0)._fault waiting 3.200000 2019-12-05T05:03:13.276 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:11.178+0000 7f5b5ec0d700 1 --2- 172.21.15.42:0/2029094007 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5b5809f820 0x7f5b5814efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-12-05T05:03:13.276 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:11.178+0000 7f5b5ec0d700 1 --2- 172.21.15.42:0/2029094007 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5b5809f820 0x7f5b5814efd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rx=0 tx=0)._fault waiting 3.200000 2019-12-05T05:03:13.277 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:13.242+0000 7f5b5d40a700 1 -- 172.21.15.42:0/2029094007 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2191013988 0 0) 0x7f5b5809fc50 con 0x7f5b580a7730 2019-12-05T05:03:13.277 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:13.242+0000 7f5b5d40a700 1 -- 172.21.15.42:0/2029094007 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b440030b0 con 0x7f5b580a7730 2019-12-05T05:03:13.277 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:13.246+0000 7f5b5d40a700 1 -- 172.21.15.42:0/2029094007 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 685343735 0 0) 0x7f5b440030b0 con 0x7f5b580a7730 2019-12-05T05:03:13.278 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:13.246+0000 7f5b5d40a700 1 -- 172.21.15.42:0/2029094007 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5b5809f820 msgr2=0x7f5b5814efd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2019-12-05T05:03:13.278 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:13.246+0000 7f5b5d40a700 1 --2- 172.21.15.42:0/2029094007 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5b5809f820 0x7f5b5814efd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:03:13.278 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:13.246+0000 7f5b5d40a700 1 -- 172.21.15.42:0/2029094007 >> v2:172.21.15.42:3300/0 conn(0x7f5b5808df70 msgr2=0x7f5b5809f2d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2019-12-05T05:03:13.278 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:13.246+0000 7f5b5d40a700 1 --2- 172.21.15.42:0/2029094007 >> v2:172.21.15.42:3300/0 conn(0x7f5b5808df70 0x7f5b5809f2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:03:13.278 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:13.246+0000 7f5b5d40a700 1 -- 172.21.15.42:0/2029094007 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b580a0700 con 0x7f5b580a7730 2019-12-05T05:03:13.279 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:13.246+0000 7f5b5d40a700 1 -- 172.21.15.42:0/2029094007 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7f5b50001ad0 con 0x7f5b580a7730 2019-12-05T05:03:13.279 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:13.246+0000 7f5b5d40a700 1 -- 172.21.15.42:0/2029094007 >> v1:172.21.15.42:6789/0 conn(0x7f5b580a7730 legacy=0x7f5b5807e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:13.279 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:13.246+0000 7f5b5d40a700 1 --2- 172.21.15.42:0/2029094007 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b440036e0 0x7f5b44003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:13.279 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:13.246+0000 7f5b5d40a700 1 --2- 172.21.15.42:0/2029094007 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5b44004420 0x7f5b44004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:13.280 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:13.246+0000 7f5b5d40a700 1 --2- 172.21.15.42:0/2029094007 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5b44004da0 0x7f5b4400b670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:13.280 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:13.246+0000 7f5b5d40a700 1 -- 172.21.15.42:0/2029094007 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b580a0af0 con 0x7f5b440036e0 2019-12-05T05:03:13.281 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:13.246+0000 7f5b5d40a700 1 -- 172.21.15.42:0/2029094007 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b5808e5a0 con 0x7f5b44004420 2019-12-05T05:03:13.281 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:13.246+0000 7f5b5d40a700 1 -- 172.21.15.42:0/2029094007 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b44005250 con 0x7f5b44004da0 2019-12-05T05:03:13.281 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:13.246+0000 7f5b5e40c700 1 --2- 172.21.15.42:0/2029094007 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b440036e0 0x7f5b44003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:13.281 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:13.246+0000 7f5b5ec0d700 1 --2- 172.21.15.42:0/2029094007 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5b44004da0 0x7f5b4400b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:13.282 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:13.246+0000 7f5b5dc0b700 1 --2- 172.21.15.42:0/2029094007 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5b44004420 0x7f5b44004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:13.282 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:13.246+0000 7f5b5e40c700 1 -- 172.21.15.42:0/2029094007 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b440036e0 msgr2=0x7f5b44003b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2019-12-05T05:03:13.282 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:13.246+0000 7f5b5e40c700 1 -- 172.21.15.42:0/2029094007 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b440036e0 msgr2=0x7f5b44003b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2019-12-05T05:03:13.282 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:13.246+0000 7f5b5e40c700 1 --2- 172.21.15.42:0/2029094007 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b440036e0 0x7f5b44003b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-12-05T05:03:13.283 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:13.246+0000 7f5b5e40c700 1 --2- 172.21.15.42:0/2029094007 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b440036e0 0x7f5b44003b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rx=0 tx=0)._fault waiting 0.200000 2019-12-05T05:03:13.283 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:13.250+0000 7f5b5ec0d700 1 -- 172.21.15.42:0/2029094007 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5b44004420 msgr2=0x7f5b44004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:03:13.284 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:13.250+0000 7f5b5ec0d700 1 --2- 172.21.15.42:0/2029094007 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5b44004420 0x7f5b44004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:03:13.284 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:13.250+0000 7f5b5ec0d700 1 -- 172.21.15.42:0/2029094007 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b440036e0 msgr2=0x7f5b44003b50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2019-12-05T05:03:13.285 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:13.250+0000 7f5b5ec0d700 1 --2- 172.21.15.42:0/2029094007 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b440036e0 0x7f5b44003b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:03:13.285 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:13.250+0000 7f5b5ec0d700 1 -- 172.21.15.42:0/2029094007 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b4400bd00 con 0x7f5b44004da0 2019-12-05T05:03:13.285 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:13.250+0000 7f5b5ec0d700 1 --2- 172.21.15.42:0/2029094007 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5b44004da0 0x7f5b4400b670 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f5b54002f30 tx=0x7f5b5400bad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:13.285 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:13.250+0000 7f5b5ec0d700 1 --2- 172.21.15.42:0/2029094007 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5b44004da0 0x7f5b4400b670 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f5b54002f30 tx=0x7f5b5400bad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:13.286 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:13.250+0000 7f5b5ec0d700 1 --2- 172.21.15.42:0/2029094007 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5b44004da0 0x7f5b4400b670 secure :-1 s=READY pgs=6 cs=0 l=1 rx=0x7f5b54002f30 tx=0x7f5b5400bad0).ready entity=mon.2 client_cookie=a23a44f8b625cced server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:13.286 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:13.250+0000 7f5b5ec0d700 1 --2- 172.21.15.42:0/2029094007 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5b44004da0 0x7f5b4400b670 secure :-1 s=THROTTLE_DONE pgs=6 cs=0 l=1 rx=0x7f5b54002f30 tx=0x7f5b5400bad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:13.286 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:13.250+0000 7f5b5ec0d700 1 --2- 172.21.15.42:0/2029094007 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5b44004da0 0x7f5b4400b670 secure :-1 s=THROTTLE_DONE pgs=6 cs=0 l=1 rx=0x7f5b54002f30 tx=0x7f5b5400bad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:13.288 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:13.250+0000 7f5b5d40a700 1 -- 172.21.15.42:0/2029094007 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5b54010a70 con 0x7f5b44004da0 2019-12-05T05:03:13.288 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:13.250+0000 7f5b5ec0d700 1 --2- 172.21.15.42:0/2029094007 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5b44004da0 0x7f5b4400b670 secure :-1 s=THROTTLE_DONE pgs=6 cs=0 l=1 rx=0x7f5b54002f30 tx=0x7f5b5400bad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:13.288 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:13.250+0000 7f5b5d40a700 1 -- 172.21.15.42:0/2029094007 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b54010690 con 0x7f5b44004da0 2019-12-05T05:03:13.288 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:13.250+0000 7f5b5d40a700 1 -- 172.21.15.42:0/2029094007 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5b5400e720 con 0x7f5b44004da0 2019-12-05T05:03:13.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:13.250+0000 7f5b5fe6f700 1 -- 172.21.15.42:0/2029094007 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5b44004da0 msgr2=0x7f5b4400b670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:13.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:13.250+0000 7f5b5fe6f700 1 --2- 172.21.15.42:0/2029094007 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5b44004da0 0x7f5b4400b670 secure :-1 s=READY pgs=6 cs=0 l=1 rx=0x7f5b54002f30 tx=0x7f5b5400bad0).stop 2019-12-05T05:03:13.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:13.250+0000 7f5b5ec0d700 1 -- 172.21.15.42:0/2029094007 reap_dead start 2019-12-05T05:03:13.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:13.250+0000 7f5b5fe6f700 1 -- 172.21.15.42:0/2029094007 shutdown_connections 2019-12-05T05:03:13.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:13.250+0000 7f5b5fe6f700 1 -- 172.21.15.42:0/2029094007 shutdown_connections 2019-12-05T05:03:13.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:13.250+0000 7f5b5fe6f700 1 -- 172.21.15.42:0/2029094007 wait complete. 2019-12-05T05:03:13.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:13.250+0000 7f5b5fe6f700 1 -- 172.21.15.42:0/2029094007 >> 172.21.15.42:0/2029094007 conn(0x7f5b580aabd0 msgr2=0x7f5b580a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:13.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:13.254+0000 7f5b5fe6f700 1 Processor -- start 2019-12-05T05:03:13.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:13.254+0000 7f5b5fe6f700 1 -- start start 2019-12-05T05:03:13.292 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:13.258+0000 7f5b5fe6f700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5b58142ea0 0x7f5b580a3d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:13.292 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:13.258+0000 7f5b5fe6f700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5b580a1c40 0x7f5b580a2070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:13.292 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:13.258+0000 7f5b5fe6f700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5b5807d380 0x7f5b5807d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:13.292 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:13.258+0000 7f5b5fe6f700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b581447f0 con 0x7f5b58142ea0 2019-12-05T05:03:13.293 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:13.258+0000 7f5b5fe6f700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b58144a70 con 0x7f5b5807d380 2019-12-05T05:03:13.293 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:13.258+0000 7f5b5fe6f700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b58144d10 con 0x7f5b580a1c40 2019-12-05T05:03:13.294 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:13.258+0000 7f5b5ec0d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5b5807d380 0x7f5b5807d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:13.295 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:13.258+0000 7f5b5e40c700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5b58142ea0 0x7f5b580a3d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:13.295 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:13.258+0000 7f5b5e40c700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5b58142ea0 0x7f5b580a3d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.42:40452/0 (socket says 172.21.15.42:40452) 2019-12-05T05:03:13.295 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:13.258+0000 7f5b5e40c700 1 -- 172.21.15.42:0/1313374076 learned_addr learned my addr 172.21.15.42:0/1313374076 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:03:13.295 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:13.258+0000 7f5b5dc0b700 1 --2- 172.21.15.42:0/1313374076 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5b580a1c40 0x7f5b580a2070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:13.296 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:13.258+0000 7f5b5ec0d700 1 -- 172.21.15.42:0/1313374076 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5b580a1c40 msgr2=0x7f5b580a2070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:13.296 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:13.258+0000 7f5b5ec0d700 1 --2- 172.21.15.42:0/1313374076 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5b580a1c40 0x7f5b580a2070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:13.296 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:13.258+0000 7f5b5ec0d700 1 -- 172.21.15.42:0/1313374076 >> v2:172.21.15.40:3300/0 conn(0x7f5b58142ea0 msgr2=0x7f5b580a3d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:13.296 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:13.258+0000 7f5b5ec0d700 1 --2- 172.21.15.42:0/1313374076 >> v2:172.21.15.40:3300/0 conn(0x7f5b58142ea0 0x7f5b580a3d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:13.296 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:13.258+0000 7f5b5ec0d700 1 -- 172.21.15.42:0/1313374076 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b58143d80 con 0x7f5b5807d380 2019-12-05T05:03:13.297 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:13.258+0000 7f5b5ec0d700 1 --2- 172.21.15.42:0/1313374076 >> v2:172.21.15.42:3300/0 conn(0x7f5b5807d380 0x7f5b5807d7b0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f5b58144ed0 tx=0x7f5b540109c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:13.297 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:13.258+0000 7f5b5dc0b700 1 --2- 172.21.15.42:0/1313374076 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5b580a1c40 0x7f5b580a2070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:03:13.297 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:13.258+0000 7f5b5ec0d700 1 --2- 172.21.15.42:0/1313374076 >> v2:172.21.15.42:3300/0 conn(0x7f5b5807d380 0x7f5b5807d7b0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f5b58144ed0 tx=0x7f5b540109c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:13.297 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:13.258+0000 7f5b5ec0d700 1 --2- 172.21.15.42:0/1313374076 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5b5807d380 0x7f5b5807d7b0 secure :-1 s=READY pgs=10 cs=0 l=1 rx=0x7f5b58144ed0 tx=0x7f5b540109c0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:13.298 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:13.258+0000 7f5b5ec0d700 1 --2- 172.21.15.42:0/1313374076 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5b5807d380 0x7f5b5807d7b0 secure :-1 s=THROTTLE_DONE pgs=10 cs=0 l=1 rx=0x7f5b58144ed0 tx=0x7f5b540109c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:13.298 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:13.258+0000 7f5b4effd700 1 -- 172.21.15.42:0/1313374076 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5b54007350 con 0x7f5b5807d380 2019-12-05T05:03:13.298 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:13.258+0000 7f5b5ec0d700 1 --2- 172.21.15.42:0/1313374076 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5b5807d380 0x7f5b5807d7b0 secure :-1 s=THROTTLE_DONE pgs=10 cs=0 l=1 rx=0x7f5b58144ed0 tx=0x7f5b540109c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:13.299 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:13.258+0000 7f5b5ec0d700 1 --2- 172.21.15.42:0/1313374076 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5b5807d380 0x7f5b5807d7b0 secure :-1 s=THROTTLE_DONE pgs=10 cs=0 l=1 rx=0x7f5b58144ed0 tx=0x7f5b540109c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:13.299 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:13.258+0000 7f5b4effd700 1 -- 172.21.15.42:0/1313374076 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b54010c70 con 0x7f5b5807d380 2019-12-05T05:03:13.299 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:13.258+0000 7f5b5fe6f700 1 -- 172.21.15.42:0/1313374076 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b58145480 con 0x7f5b5807d380 2019-12-05T05:03:13.299 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:13.258+0000 7f5b4effd700 1 -- 172.21.15.42:0/1313374076 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5b54005b30 con 0x7f5b5807d380 2019-12-05T05:03:13.299 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:13.258+0000 7f5b5fe6f700 1 -- 172.21.15.42:0/1313374076 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b580a42d0 con 0x7f5b5807d380 2019-12-05T05:03:13.300 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:13.258+0000 7f5b5ec0d700 1 --2- 172.21.15.42:0/1313374076 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5b5807d380 0x7f5b5807d7b0 secure :-1 s=THROTTLE_DONE pgs=10 cs=0 l=1 rx=0x7f5b58144ed0 tx=0x7f5b540109c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:13.303 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:13.258+0000 7f5b4effd700 1 -- 172.21.15.42:0/1313374076 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 1) v1 ==== 349+0+0 (secure 0 0 0) 0x7f5b5400e700 con 0x7f5b5807d380 2019-12-05T05:03:13.303 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:13.258+0000 7f5b5ec0d700 1 --2- 172.21.15.42:0/1313374076 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5b5807d380 0x7f5b5807d7b0 secure :-1 s=THROTTLE_DONE pgs=10 cs=0 l=1 rx=0x7f5b58144ed0 tx=0x7f5b540109c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:13.303 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:13.258+0000 7f5b4effd700 1 -- 172.21.15.42:0/1313374076 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f5b5400a930 con 0x7f5b5807d380 2019-12-05T05:03:13.307 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:13.286+0000 7f5b5fe6f700 1 -- 172.21.15.42:0/1313374076 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b5805d100 con 0x7f5b5807d380 2019-12-05T05:03:13.308 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:13.290+0000 7f5b5ec0d700 1 --2- 172.21.15.42:0/1313374076 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5b5807d380 0x7f5b5807d7b0 secure :-1 s=THROTTLE_DONE pgs=10 cs=0 l=1 rx=0x7f5b58144ed0 tx=0x7f5b540109c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:13.308 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:13.290+0000 7f5b4effd700 1 -- 172.21.15.42:0/1313374076 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+66539 (secure 0 0 0) 0x7f5b5400f230 con 0x7f5b5807d380 2019-12-05T05:03:13.387 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:13.386+0000 7f5b5fe6f700 1 -- 172.21.15.42:0/1313374076 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"profile": "default", "prefix": "osd crush tunables"} v 0) v1 -- 0x7f5b5805b8b0 con 0x7f5b5807d380 2019-12-05T05:03:13.394 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:13.394+0000 7f5b5ec0d700 1 --2- 172.21.15.42:0/1313374076 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5b5807d380 0x7f5b5807d7b0 secure :-1 s=THROTTLE_DONE pgs=10 cs=0 l=1 rx=0x7f5b58144ed0 tx=0x7f5b540109c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:13.395 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:13.394+0000 7f5b4effd700 1 -- 172.21.15.42:0/1313374076 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"profile": "default", "prefix": "osd crush tunables"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7f5b5400f450 con 0x7f5b5807d380 2019-12-05T05:03:13.401 INFO:teuthology.orchestra.run.smithi042.stderr:adjusted tunables profile to default 2019-12-05T05:03:13.404 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:13.402+0000 7f5b5fe6f700 1 -- 172.21.15.42:0/1313374076 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5b5807d380 msgr2=0x7f5b5807d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:13.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:13.402+0000 7f5b5fe6f700 1 --2- 172.21.15.42:0/1313374076 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5b5807d380 0x7f5b5807d7b0 secure :-1 s=READY pgs=10 cs=0 l=1 rx=0x7f5b58144ed0 tx=0x7f5b540109c0).stop 2019-12-05T05:03:13.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:13.402+0000 7f5b5fe6f700 1 -- 172.21.15.42:0/1313374076 shutdown_connections 2019-12-05T05:03:13.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:13.402+0000 7f5b5fe6f700 1 --2- 172.21.15.42:0/1313374076 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5b5807d380 0x7f5b5807d7b0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:13.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:13.402+0000 7f5b5fe6f700 1 --2- 172.21.15.42:0/1313374076 >> v2:172.21.15.40:3300/0 conn(0x7f5b58142ea0 0x7f5b580a3d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:13.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:13.402+0000 7f5b5fe6f700 1 --2- 172.21.15.42:0/1313374076 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5b580a1c40 0x7f5b580a2070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:13.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:13.402+0000 7f5b5fe6f700 1 -- 172.21.15.42:0/1313374076 shutdown_connections 2019-12-05T05:03:13.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:13.402+0000 7f5b5fe6f700 1 -- 172.21.15.42:0/1313374076 wait complete. 2019-12-05T05:03:13.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:13.402+0000 7f5b5fe6f700 1 -- 172.21.15.42:0/1313374076 >> 172.21.15.42:0/1313374076 conn(0x7f5b580a8290 msgr2=0x7f5b5808eb60 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:13.418 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2019-12-05T05:03:13.418 INFO:teuthology.orchestra.run.smithi042:> mktemp 2019-12-05T05:03:13.425 INFO:teuthology.orchestra.run.smithi042.stdout:/tmp/tmp.GXEyUiDXNp 2019-12-05T05:03:13.425 INFO:teuthology.orchestra.run.smithi042:> sudo cp /var/lib/ceph/osd/ceph-0/fsid /tmp/tmp.GXEyUiDXNp 2019-12-05T05:03:13.511 INFO:teuthology.orchestra.run.smithi042:> sudo chmod 0666 /tmp/tmp.GXEyUiDXNp 2019-12-05T05:03:13.670 DEBUG:teuthology.orchestra.remote:smithi042:/tmp/tmp.GXEyUiDXNp is 37B 2019-12-05T05:03:13.681 INFO:teuthology.orchestra.run.smithi042:> rm -fr /tmp/tmp.GXEyUiDXNp 2019-12-05T05:03:13.689 INFO:teuthology.orchestra.run.smithi042:> mktemp 2019-12-05T05:03:13.760 INFO:teuthology.orchestra.run.smithi042.stdout:/tmp/tmp.Sz0lNNcKv1 2019-12-05T05:03:13.761 INFO:teuthology.orchestra.run.smithi042:> sudo cp /var/lib/ceph/osd/ceph-1/fsid /tmp/tmp.Sz0lNNcKv1 2019-12-05T05:03:13.842 INFO:teuthology.orchestra.run.smithi042:> sudo chmod 0666 /tmp/tmp.Sz0lNNcKv1 2019-12-05T05:03:13.992 DEBUG:teuthology.orchestra.remote:smithi042:/tmp/tmp.Sz0lNNcKv1 is 37B 2019-12-05T05:03:14.003 INFO:teuthology.orchestra.run.smithi042:> rm -fr /tmp/tmp.Sz0lNNcKv1 2019-12-05T05:03:14.010 INFO:teuthology.orchestra.run.smithi042:> mktemp 2019-12-05T05:03:14.081 INFO:teuthology.orchestra.run.smithi042.stdout:/tmp/tmp.5MqMY9P5WR 2019-12-05T05:03:14.081 INFO:teuthology.orchestra.run.smithi042:> sudo cp /var/lib/ceph/osd/ceph-2/fsid /tmp/tmp.5MqMY9P5WR 2019-12-05T05:03:14.163 INFO:teuthology.orchestra.run.smithi042:> sudo chmod 0666 /tmp/tmp.5MqMY9P5WR 2019-12-05T05:03:14.322 DEBUG:teuthology.orchestra.remote:smithi042:/tmp/tmp.5MqMY9P5WR is 37B 2019-12-05T05:03:14.375 INFO:teuthology.orchestra.run.smithi042:> rm -fr /tmp/tmp.5MqMY9P5WR 2019-12-05T05:03:14.382 INFO:teuthology.orchestra.run.smithi042:> mktemp 2019-12-05T05:03:14.459 INFO:teuthology.orchestra.run.smithi042.stdout:/tmp/tmp.8MVqPzugTH 2019-12-05T05:03:14.460 INFO:teuthology.orchestra.run.smithi042:> sudo cp /var/lib/ceph/osd/ceph-3/fsid /tmp/tmp.8MVqPzugTH 2019-12-05T05:03:14.523 INFO:teuthology.orchestra.run.smithi042:> sudo chmod 0666 /tmp/tmp.8MVqPzugTH 2019-12-05T05:03:14.679 DEBUG:teuthology.orchestra.remote:smithi042:/tmp/tmp.8MVqPzugTH is 37B 2019-12-05T05:03:14.691 INFO:teuthology.orchestra.run.smithi042:> rm -fr /tmp/tmp.8MVqPzugTH 2019-12-05T05:03:14.698 INFO:teuthology.orchestra.run.smithi040:> mktemp 2019-12-05T05:03:14.706 INFO:teuthology.orchestra.run.smithi040.stdout:/tmp/tmp.i8kf60xcok 2019-12-05T05:03:14.706 INFO:teuthology.orchestra.run.smithi040:> sudo cp /var/lib/ceph/osd/ceph-4/fsid /tmp/tmp.i8kf60xcok 2019-12-05T05:03:14.786 INFO:teuthology.orchestra.run.smithi040:> sudo chmod 0666 /tmp/tmp.i8kf60xcok 2019-12-05T05:03:14.918 DEBUG:teuthology.orchestra.remote:smithi040:/tmp/tmp.i8kf60xcok is 37B 2019-12-05T05:03:14.968 INFO:teuthology.orchestra.run.smithi040:> rm -fr /tmp/tmp.i8kf60xcok 2019-12-05T05:03:14.976 INFO:teuthology.orchestra.run.smithi040:> mktemp 2019-12-05T05:03:15.049 INFO:teuthology.orchestra.run.smithi040.stdout:/tmp/tmp.Eg6Ifbl6rY 2019-12-05T05:03:15.049 INFO:teuthology.orchestra.run.smithi040:> sudo cp /var/lib/ceph/osd/ceph-5/fsid /tmp/tmp.Eg6Ifbl6rY 2019-12-05T05:03:15.140 INFO:teuthology.orchestra.run.smithi040:> sudo chmod 0666 /tmp/tmp.Eg6Ifbl6rY 2019-12-05T05:03:15.294 DEBUG:teuthology.orchestra.remote:smithi040:/tmp/tmp.Eg6Ifbl6rY is 37B 2019-12-05T05:03:15.341 INFO:teuthology.orchestra.run.smithi040:> rm -fr /tmp/tmp.Eg6Ifbl6rY 2019-12-05T05:03:15.351 INFO:teuthology.orchestra.run.smithi040:> mktemp 2019-12-05T05:03:15.408 INFO:teuthology.orchestra.run.smithi040.stdout:/tmp/tmp.Hwi2SpcnBC 2019-12-05T05:03:15.409 INFO:teuthology.orchestra.run.smithi040:> sudo cp /var/lib/ceph/osd/ceph-6/fsid /tmp/tmp.Hwi2SpcnBC 2019-12-05T05:03:15.489 INFO:teuthology.orchestra.run.smithi040:> sudo chmod 0666 /tmp/tmp.Hwi2SpcnBC 2019-12-05T05:03:15.639 DEBUG:teuthology.orchestra.remote:smithi040:/tmp/tmp.Hwi2SpcnBC is 37B 2019-12-05T05:03:15.650 INFO:teuthology.orchestra.run.smithi040:> rm -fr /tmp/tmp.Hwi2SpcnBC 2019-12-05T05:03:15.656 INFO:teuthology.orchestra.run.smithi040:> mktemp 2019-12-05T05:03:15.742 INFO:teuthology.orchestra.run.smithi040.stdout:/tmp/tmp.Rp5LRowOET 2019-12-05T05:03:15.743 INFO:teuthology.orchestra.run.smithi040:> sudo cp /var/lib/ceph/osd/ceph-7/fsid /tmp/tmp.Rp5LRowOET 2019-12-05T05:03:15.824 INFO:teuthology.orchestra.run.smithi040:> sudo chmod 0666 /tmp/tmp.Rp5LRowOET 2019-12-05T05:03:16.007 DEBUG:teuthology.orchestra.remote:smithi040:/tmp/tmp.Rp5LRowOET is 37B 2019-12-05T05:03:16.066 INFO:teuthology.orchestra.run.smithi040:> rm -fr /tmp/tmp.Rp5LRowOET 2019-12-05T05:03:16.074 INFO:teuthology.orchestra.run.smithi040:> sudo ceph --cluster ceph osd new e3fc49bb-dcbc-43ff-ac7c-9eb0ea0bd257 0 2019-12-05T05:03:16.407 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.392+0000 7ffaed9e6700 1 Processor -- start 2019-12-05T05:03:16.408 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.392+0000 7ffaed9e6700 1 -- start start 2019-12-05T05:03:16.408 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.392+0000 7ffaed9e6700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ffae80a7730 0x7ffae808df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:16.409 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.392+0000 7ffaed9e6700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ffae809fad0 0x7ffae807eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:16.409 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.392+0000 7ffaed9e6700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffae80a0700 con 0x7ffae807e710 2019-12-05T05:03:16.409 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.392+0000 7ffaed9e6700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffae80a0af0 con 0x7ffae809fad0 2019-12-05T05:03:16.410 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.392+0000 7ffaed9e6700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffae808e5a0 con 0x7ffae80a7730 2019-12-05T05:03:16.410 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.392+0000 7ffae77fe700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ffae80a7730 0x7ffae808df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:16.410 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.396+0000 7ffae6ffd700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7ffae807e710 0x7ffae809f2d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:59668/0 (socket says 172.21.15.40:59668) 2019-12-05T05:03:16.410 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.396+0000 7ffae6ffd700 1 -- 172.21.15.40:0/5409389 learned_addr learned my addr 172.21.15.40:0/5409389 (peer_addr_for_me v1:172.21.15.40:0/0) 2019-12-05T05:03:16.411 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.396+0000 7ffae7fff700 1 --2- 172.21.15.40:0/5409389 >> v2:172.21.15.40:3300/0 conn(0x7ffae809fad0 0x7ffae807eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:16.411 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.396+0000 7ffae67fc700 1 -- 172.21.15.40:0/5409389 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1569397413 0 0) 0x7ffae80a0700 con 0x7ffae807e710 2019-12-05T05:03:16.411 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.396+0000 7ffae67fc700 1 -- 172.21.15.40:0/5409389 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffac80030b0 con 0x7ffae807e710 2019-12-05T05:03:16.411 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.396+0000 7ffae7fff700 1 -- 172.21.15.40:0/5409389 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ffae80a7730 msgr2=0x7ffae808df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:03:16.411 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.396+0000 7ffae7fff700 1 --2- 172.21.15.40:0/5409389 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ffae80a7730 0x7ffae808df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:03:16.412 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.396+0000 7ffae7fff700 1 -- 172.21.15.40:0/5409389 >> v1:172.21.15.40:6789/0 conn(0x7ffae807e710 legacy=0x7ffae809f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:16.412 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.396+0000 7ffae7fff700 1 -- 172.21.15.40:0/5409389 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffae807f090 con 0x7ffae809fad0 2019-12-05T05:03:16.412 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.396+0000 7ffae7fff700 1 --2- 172.21.15.40:0/5409389 >> v2:172.21.15.40:3300/0 conn(0x7ffae809fad0 0x7ffae807eb40 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7ffae0008790 tx=0x7ffae000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:16.412 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.396+0000 7ffae7fff700 1 --2- 172.21.15.40:0/5409389 >> v2:172.21.15.40:3300/0 conn(0x7ffae809fad0 0x7ffae807eb40 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7ffae0008790 tx=0x7ffae000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:16.413 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.396+0000 7ffae7fff700 1 --2- 172.21.15.40:0/5409389 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffae809fad0 0x7ffae807eb40 secure :-1 s=READY pgs=14 cs=0 l=1 rx=0x7ffae0008790 tx=0x7ffae000fc70).ready entity=mon.0 client_cookie=261bf359519b8d1d server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:16.413 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.396+0000 7ffae7fff700 1 --2- 172.21.15.40:0/5409389 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffae809fad0 0x7ffae807eb40 secure :-1 s=THROTTLE_DONE pgs=14 cs=0 l=1 rx=0x7ffae0008790 tx=0x7ffae000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:16.413 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.396+0000 7ffae67fc700 1 -- 172.21.15.40:0/5409389 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ffae0008bb0 con 0x7ffae809fad0 2019-12-05T05:03:16.413 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.396+0000 7ffae7fff700 1 --2- 172.21.15.40:0/5409389 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffae809fad0 0x7ffae807eb40 secure :-1 s=THROTTLE_DONE pgs=14 cs=0 l=1 rx=0x7ffae0008790 tx=0x7ffae000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:16.414 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.396+0000 7ffae67fc700 1 -- 172.21.15.40:0/5409389 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffae00087d0 con 0x7ffae809fad0 2019-12-05T05:03:16.414 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.396+0000 7ffae7fff700 1 --2- 172.21.15.40:0/5409389 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffae809fad0 0x7ffae807eb40 secure :-1 s=THROTTLE_DONE pgs=14 cs=0 l=1 rx=0x7ffae0008790 tx=0x7ffae000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:16.414 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.396+0000 7ffae67fc700 1 -- 172.21.15.40:0/5409389 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ffae0011860 con 0x7ffae809fad0 2019-12-05T05:03:16.414 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.396+0000 7ffaed9e6700 1 -- 172.21.15.40:0/5409389 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffae809fad0 msgr2=0x7ffae807eb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:16.414 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.396+0000 7ffaed9e6700 1 --2- 172.21.15.40:0/5409389 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffae809fad0 0x7ffae807eb40 secure :-1 s=READY pgs=14 cs=0 l=1 rx=0x7ffae0008790 tx=0x7ffae000fc70).stop 2019-12-05T05:03:16.415 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.396+0000 7ffaed9e6700 1 -- 172.21.15.40:0/5409389 shutdown_connections 2019-12-05T05:03:16.415 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.396+0000 7ffaed9e6700 1 --2- 172.21.15.40:0/5409389 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffae809fad0 0x7ffae807eb40 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:16.415 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.396+0000 7ffaed9e6700 1 --2- 172.21.15.40:0/5409389 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ffae80a7730 0x7ffae808df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:03:16.415 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.396+0000 7ffaed9e6700 1 -- 172.21.15.40:0/5409389 shutdown_connections 2019-12-05T05:03:16.416 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.396+0000 7ffaed9e6700 1 -- 172.21.15.40:0/5409389 wait complete. 2019-12-05T05:03:16.416 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.396+0000 7ffaed9e6700 1 -- 172.21.15.40:0/5409389 >> 172.21.15.40:0/5409389 conn(0x7ffae80aabd0 msgr2=0x7ffae80a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:16.416 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.400+0000 7ffaed9e6700 1 Processor -- start 2019-12-05T05:03:16.416 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.400+0000 7ffaed9e6700 1 -- start start 2019-12-05T05:03:16.417 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.400+0000 7ffaed9e6700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ffae8130c30 0x7ffae8121ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:16.417 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.400+0000 7ffaed9e6700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ffae80742d0 0x7ffae8076710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:16.417 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.400+0000 7ffaed9e6700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffae8122540 con 0x7ffae81164f0 2019-12-05T05:03:16.417 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.400+0000 7ffaed9e6700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffae8122d60 con 0x7ffae8130c30 2019-12-05T05:03:16.418 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.400+0000 7ffaed9e6700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffae8122820 con 0x7ffae80742d0 2019-12-05T05:03:16.418 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.400+0000 7ffae6ffd700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ffae8130c30 0x7ffae8121ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:16.418 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.400+0000 7ffae6ffd700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ffae8130c30 0x7ffae8121ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36012/0 (socket says 172.21.15.40:36012) 2019-12-05T05:03:16.419 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.400+0000 7ffae6ffd700 1 -- 172.21.15.40:0/2781751707 learned_addr learned my addr 172.21.15.40:0/2781751707 (peer_addr_for_me v2:172.21.15.40:0/0) 2019-12-05T05:03:16.420 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.400+0000 7ffae7fff700 1 --2- 172.21.15.40:0/2781751707 >> v2:172.21.15.42:3300/0 conn(0x7ffae80742d0 0x7ffae8076710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:16.420 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.400+0000 7ffae6ffd700 1 -- 172.21.15.40:0/2781751707 >> v2:172.21.15.42:3300/0 conn(0x7ffae80742d0 msgr2=0x7ffae8076710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:16.420 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.400+0000 7ffae6ffd700 1 --2- 172.21.15.40:0/2781751707 >> v2:172.21.15.42:3300/0 conn(0x7ffae80742d0 0x7ffae8076710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:16.420 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.400+0000 7ffae6ffd700 1 -- 172.21.15.40:0/2781751707 >> v1:172.21.15.40:6789/0 conn(0x7ffae81164f0 legacy=0x7ffae8130430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:16.421 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.400+0000 7ffae6ffd700 1 -- 172.21.15.40:0/2781751707 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffae8121920 con 0x7ffae8130c30 2019-12-05T05:03:16.421 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.404+0000 7ffae6ffd700 1 --2- 172.21.15.40:0/2781751707 >> v2:172.21.15.40:3300/0 conn(0x7ffae8130c30 0x7ffae8121ff0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7ffae81229e0 tx=0x7ffad0010a00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:16.421 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.404+0000 7ffae6ffd700 1 --2- 172.21.15.40:0/2781751707 >> v2:172.21.15.40:3300/0 conn(0x7ffae8130c30 0x7ffae8121ff0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7ffae81229e0 tx=0x7ffad0010a00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:16.422 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.404+0000 7ffae6ffd700 1 --2- 172.21.15.40:0/2781751707 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffae8130c30 0x7ffae8121ff0 secure :-1 s=READY pgs=16 cs=0 l=1 rx=0x7ffae81229e0 tx=0x7ffad0010a00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:16.422 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.404+0000 7ffae6ffd700 1 --2- 172.21.15.40:0/2781751707 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffae8130c30 0x7ffae8121ff0 secure :-1 s=THROTTLE_DONE pgs=16 cs=0 l=1 rx=0x7ffae81229e0 tx=0x7ffad0010a00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:16.422 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.404+0000 7ffae6ffd700 1 --2- 172.21.15.40:0/2781751707 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffae8130c30 0x7ffae8121ff0 secure :-1 s=THROTTLE_DONE pgs=16 cs=0 l=1 rx=0x7ffae81229e0 tx=0x7ffad0010a00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:16.423 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.404+0000 7ffae6ffd700 1 --2- 172.21.15.40:0/2781751707 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffae8130c30 0x7ffae8121ff0 secure :-1 s=THROTTLE_DONE pgs=16 cs=0 l=1 rx=0x7ffae81229e0 tx=0x7ffad0010a00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:16.423 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.404+0000 7ffadffff700 1 -- 172.21.15.40:0/2781751707 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ffad000c9c0 con 0x7ffae8130c30 2019-12-05T05:03:16.423 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.404+0000 7ffadffff700 1 -- 172.21.15.40:0/2781751707 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffad0010250 con 0x7ffae8130c30 2019-12-05T05:03:16.423 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.404+0000 7ffaed9e6700 1 -- 172.21.15.40:0/2781751707 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffae8076c60 con 0x7ffae8130c30 2019-12-05T05:03:16.424 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.404+0000 7ffadffff700 1 -- 172.21.15.40:0/2781751707 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ffad000d0c0 con 0x7ffae8130c30 2019-12-05T05:03:16.424 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.404+0000 7ffaed9e6700 1 -- 172.21.15.40:0/2781751707 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffae80771e0 con 0x7ffae8130c30 2019-12-05T05:03:16.424 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.404+0000 7ffae6ffd700 1 --2- 172.21.15.40:0/2781751707 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffae8130c30 0x7ffae8121ff0 secure :-1 s=THROTTLE_DONE pgs=16 cs=0 l=1 rx=0x7ffae81229e0 tx=0x7ffad0010a00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:16.425 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.404+0000 7ffadffff700 1 -- 172.21.15.40:0/2781751707 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 2) v1 ==== 22500+0+0 (secure 0 0 0) 0x7ffad000d2c0 con 0x7ffae8130c30 2019-12-05T05:03:16.425 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.404+0000 7ffae6ffd700 1 --2- 172.21.15.40:0/2781751707 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffae8130c30 0x7ffae8121ff0 secure :-1 s=THROTTLE_DONE pgs=16 cs=0 l=1 rx=0x7ffae81229e0 tx=0x7ffad0010a00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:16.425 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.404+0000 7ffadffff700 1 -- 172.21.15.40:0/2781751707 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 754+0+0 (secure 0 0 0) 0x7ffad00299f0 con 0x7ffae8130c30 2019-12-05T05:03:16.426 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.408+0000 7ffaed9e6700 1 -- 172.21.15.40:0/2781751707 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffae805d100 con 0x7ffae8130c30 2019-12-05T05:03:16.427 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.416+0000 7ffae6ffd700 1 --2- 172.21.15.40:0/2781751707 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffae8130c30 0x7ffae8121ff0 secure :-1 s=THROTTLE_DONE pgs=16 cs=0 l=1 rx=0x7ffae81229e0 tx=0x7ffad0010a00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:16.427 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.416+0000 7ffadffff700 1 -- 172.21.15.40:0/2781751707 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+66539 (secure 0 0 0) 0x7ffad000d6c0 con 0x7ffae8130c30 2019-12-05T05:03:16.530 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.528+0000 7ffaed9e6700 1 -- 172.21.15.40:0/2781751707 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd new", "uuid": "e3fc49bb-dcbc-43ff-ac7c-9eb0ea0bd257", "id": 0} v 0) v1 -- 0x7ffae811d4a0 con 0x7ffae8130c30 2019-12-05T05:03:16.536 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.536+0000 7ffae6ffd700 1 --2- 172.21.15.40:0/2781751707 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffae8130c30 0x7ffae8121ff0 secure :-1 s=THROTTLE_DONE pgs=16 cs=0 l=1 rx=0x7ffae81229e0 tx=0x7ffad0010a00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:16.537 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.536+0000 7ffadffff700 1 -- 172.21.15.40:0/2781751707 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "e3fc49bb-dcbc-43ff-ac7c-9eb0ea0bd257", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7ffad0011b60 con 0x7ffae8130c30 2019-12-05T05:03:16.537 INFO:teuthology.orchestra.run.smithi040.stdout:0 2019-12-05T05:03:16.540 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.536+0000 7ffaed9e6700 1 -- 172.21.15.40:0/2781751707 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffae8130c30 msgr2=0x7ffae8121ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:16.540 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.536+0000 7ffaed9e6700 1 --2- 172.21.15.40:0/2781751707 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffae8130c30 0x7ffae8121ff0 secure :-1 s=READY pgs=16 cs=0 l=1 rx=0x7ffae81229e0 tx=0x7ffad0010a00).stop 2019-12-05T05:03:16.541 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.540+0000 7ffaed9e6700 1 -- 172.21.15.40:0/2781751707 shutdown_connections 2019-12-05T05:03:16.541 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.540+0000 7ffaed9e6700 1 --2- 172.21.15.40:0/2781751707 >> v2:172.21.15.42:3300/0 conn(0x7ffae80742d0 0x7ffae8076710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:16.541 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.540+0000 7ffaed9e6700 1 --2- 172.21.15.40:0/2781751707 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffae8130c30 0x7ffae8121ff0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:16.541 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.540+0000 7ffaed9e6700 1 -- 172.21.15.40:0/2781751707 shutdown_connections 2019-12-05T05:03:16.541 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.540+0000 7ffaed9e6700 1 -- 172.21.15.40:0/2781751707 wait complete. 2019-12-05T05:03:16.541 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.540+0000 7ffaed9e6700 1 -- 172.21.15.40:0/2781751707 >> 172.21.15.40:0/2781751707 conn(0x7ffae807e5c0 msgr2=0x7ffae811c600 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:16.551 INFO:teuthology.orchestra.run.smithi040:> sudo ceph --cluster ceph osd new 902543fb-f5a2-4321-9c9c-26f9b9ca8629 1 2019-12-05T05:03:16.654 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.648+0000 7f8cb7772700 1 Processor -- start 2019-12-05T05:03:16.654 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.648+0000 7f8cb7772700 1 -- start start 2019-12-05T05:03:16.654 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.648+0000 7f8cb7772700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f8cb00a7740 0x7f8cb008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:16.654 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.648+0000 7f8cb7772700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8cb009fae0 0x7f8cb007eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:16.654 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.648+0000 7f8cb7772700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cb00a0710 con 0x7f8cb007e720 2019-12-05T05:03:16.655 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.648+0000 7f8cb7772700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8cb00a0b00 con 0x7f8cb00a7740 2019-12-05T05:03:16.655 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.648+0000 7f8cb7772700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cb008e5b0 con 0x7f8cb009fae0 2019-12-05T05:03:16.655 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.648+0000 7f8cb6510700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8cb009fae0 0x7f8cb007eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:16.655 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.648+0000 7f8cb550e700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f8cb007e720 0x7f8cb009f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:59688/0 (socket says 172.21.15.40:59688) 2019-12-05T05:03:16.655 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.648+0000 7f8cb550e700 1 -- 172.21.15.40:0/3156609846 learned_addr learned my addr 172.21.15.40:0/3156609846 (peer_addr_for_me v1:172.21.15.40:0/0) 2019-12-05T05:03:16.655 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.648+0000 7f8cb5d0f700 1 --2- 172.21.15.40:0/3156609846 >> v2:172.21.15.42:3300/0 conn(0x7f8cb00a7740 0x7f8cb008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:16.655 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.648+0000 7f8cb6510700 1 -- 172.21.15.40:0/3156609846 >> v2:172.21.15.42:3300/0 conn(0x7f8cb00a7740 msgr2=0x7f8cb008df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:03:16.656 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.648+0000 7f8cb6510700 1 --2- 172.21.15.40:0/3156609846 >> v2:172.21.15.42:3300/0 conn(0x7f8cb00a7740 0x7f8cb008df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:03:16.656 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.648+0000 7f8cb6510700 1 -- 172.21.15.40:0/3156609846 >> v1:172.21.15.40:6789/0 conn(0x7f8cb007e720 legacy=0x7f8cb009f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:16.656 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.648+0000 7f8cb6510700 1 -- 172.21.15.40:0/3156609846 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cb007f0a0 con 0x7f8cb009fae0 2019-12-05T05:03:16.656 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.648+0000 7f8cb4d0d700 1 -- 172.21.15.40:0/3156609846 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1463042918 0 0) 0x7f8cb00a0710 con 0x7f8cb007e720 2019-12-05T05:03:16.656 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.648+0000 7f8cb6510700 1 --2- 172.21.15.40:0/3156609846 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8cb009fae0 0x7f8cb007eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f8ca8008b10 tx=0x7f8ca800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:16.656 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.648+0000 7f8cb6510700 1 --2- 172.21.15.40:0/3156609846 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8cb009fae0 0x7f8cb007eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f8ca8008b10 tx=0x7f8ca800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:16.656 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.648+0000 7f8cb6510700 1 --2- 172.21.15.40:0/3156609846 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8cb009fae0 0x7f8cb007eb50 secure :-1 s=READY pgs=8 cs=0 l=1 rx=0x7f8ca8008b10 tx=0x7f8ca800fff0).ready entity=mon.2 client_cookie=8177e31745deb9fe server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:16.657 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.648+0000 7f8cb6510700 1 --2- 172.21.15.40:0/3156609846 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8cb009fae0 0x7f8cb007eb50 secure :-1 s=THROTTLE_DONE pgs=8 cs=0 l=1 rx=0x7f8ca8008b10 tx=0x7f8ca800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:16.657 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.648+0000 7f8cb4d0d700 1 -- 172.21.15.40:0/3156609846 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8ca8010900 con 0x7f8cb009fae0 2019-12-05T05:03:16.657 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.648+0000 7f8cb6510700 1 --2- 172.21.15.40:0/3156609846 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8cb009fae0 0x7f8cb007eb50 secure :-1 s=THROTTLE_DONE pgs=8 cs=0 l=1 rx=0x7f8ca8008b10 tx=0x7f8ca800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:16.657 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.648+0000 7f8cb6510700 1 --2- 172.21.15.40:0/3156609846 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8cb009fae0 0x7f8cb007eb50 secure :-1 s=THROTTLE_DONE pgs=8 cs=0 l=1 rx=0x7f8ca8008b10 tx=0x7f8ca800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:16.657 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.648+0000 7f8cb4d0d700 1 -- 172.21.15.40:0/3156609846 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ca8008b50 con 0x7f8cb009fae0 2019-12-05T05:03:16.657 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.648+0000 7f8cb4d0d700 1 -- 172.21.15.40:0/3156609846 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8ca80110c0 con 0x7f8cb009fae0 2019-12-05T05:03:16.657 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.648+0000 7f8cb7772700 1 -- 172.21.15.40:0/3156609846 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8cb009fae0 msgr2=0x7f8cb007eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:16.658 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.648+0000 7f8cb7772700 1 --2- 172.21.15.40:0/3156609846 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8cb009fae0 0x7f8cb007eb50 secure :-1 s=READY pgs=8 cs=0 l=1 rx=0x7f8ca8008b10 tx=0x7f8ca800fff0).stop 2019-12-05T05:03:16.658 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.652+0000 7f8cb7772700 1 -- 172.21.15.40:0/3156609846 shutdown_connections 2019-12-05T05:03:16.658 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.652+0000 7f8cb7772700 1 --2- 172.21.15.40:0/3156609846 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8cb009fae0 0x7f8cb007eb50 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:16.658 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.652+0000 7f8cb7772700 1 --2- 172.21.15.40:0/3156609846 >> v2:172.21.15.42:3300/0 conn(0x7f8cb00a7740 0x7f8cb008df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:03:16.658 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.652+0000 7f8cb7772700 1 -- 172.21.15.40:0/3156609846 shutdown_connections 2019-12-05T05:03:16.658 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.652+0000 7f8cb7772700 1 -- 172.21.15.40:0/3156609846 wait complete. 2019-12-05T05:03:16.658 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.652+0000 7f8cb7772700 1 -- 172.21.15.40:0/3156609846 >> 172.21.15.40:0/3156609846 conn(0x7f8cb00aabe0 msgr2=0x7f8cb00a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:16.663 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.652+0000 7f8cb7772700 1 Processor -- start 2019-12-05T05:03:16.663 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.652+0000 7f8cb7772700 1 -- start start 2019-12-05T05:03:16.663 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.652+0000 7f8cb7772700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8cb011bf30 0x7f8cb00a2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:16.663 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.652+0000 7f8cb7772700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f8cb007ad20 0x7f8cb007b150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:16.663 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.652+0000 7f8cb7772700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8cb0130450 0x7f8cb0130880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:16.664 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.652+0000 7f8cb7772700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8cb00a6830 con 0x7f8cb0130450 2019-12-05T05:03:16.664 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.652+0000 7f8cb7772700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8cb00a6ad0 con 0x7f8cb007ad20 2019-12-05T05:03:16.664 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.652+0000 7f8cb7772700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cb00a6d70 con 0x7f8cb011bf30 2019-12-05T05:03:16.664 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.656+0000 7f8cb5d0f700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8cb011bf30 0x7f8cb00a2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:16.664 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.656+0000 7f8cb5d0f700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8cb011bf30 0x7f8cb00a2170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:33090/0 (socket says 172.21.15.40:33090) 2019-12-05T05:03:16.664 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.656+0000 7f8cb5d0f700 1 -- 172.21.15.40:0/3828386966 learned_addr learned my addr 172.21.15.40:0/3828386966 (peer_addr_for_me v2:172.21.15.40:0/0) 2019-12-05T05:03:16.665 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.656+0000 7f8cb6510700 1 --2- 172.21.15.40:0/3828386966 >> v2:172.21.15.40:3300/0 conn(0x7f8cb0130450 0x7f8cb0130880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:16.665 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.656+0000 7f8cb550e700 1 --2- 172.21.15.40:0/3828386966 >> v2:172.21.15.42:3300/0 conn(0x7f8cb007ad20 0x7f8cb007b150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:16.665 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.656+0000 7f8cb5d0f700 1 -- 172.21.15.40:0/3828386966 >> v2:172.21.15.42:3300/0 conn(0x7f8cb007ad20 msgr2=0x7f8cb007b150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:16.665 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.656+0000 7f8cb5d0f700 1 --2- 172.21.15.40:0/3828386966 >> v2:172.21.15.42:3300/0 conn(0x7f8cb007ad20 0x7f8cb007b150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:16.665 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.656+0000 7f8cb5d0f700 1 -- 172.21.15.40:0/3828386966 >> v2:172.21.15.40:3300/0 conn(0x7f8cb0130450 msgr2=0x7f8cb0130880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:16.665 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.656+0000 7f8cb5d0f700 1 --2- 172.21.15.40:0/3828386966 >> v2:172.21.15.40:3300/0 conn(0x7f8cb0130450 0x7f8cb0130880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:16.665 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.656+0000 7f8cb5d0f700 1 -- 172.21.15.40:0/3828386966 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cb0130dd0 con 0x7f8cb011bf30 2019-12-05T05:03:16.665 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.656+0000 7f8cb5d0f700 1 --2- 172.21.15.40:0/3828386966 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8cb011bf30 0x7f8cb00a2170 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f8cb00a1cb0 tx=0x7f8ca0010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:16.666 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.656+0000 7f8cb5d0f700 1 --2- 172.21.15.40:0/3828386966 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8cb011bf30 0x7f8cb00a2170 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f8cb00a1cb0 tx=0x7f8ca0010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:16.666 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.656+0000 7f8cb5d0f700 1 --2- 172.21.15.40:0/3828386966 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8cb011bf30 0x7f8cb00a2170 secure :-1 s=READY pgs=9 cs=0 l=1 rx=0x7f8cb00a1cb0 tx=0x7f8ca0010b60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:16.666 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.656+0000 7f8cb5d0f700 1 --2- 172.21.15.40:0/3828386966 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8cb011bf30 0x7f8cb00a2170 secure :-1 s=THROTTLE_DONE pgs=9 cs=0 l=1 rx=0x7f8cb00a1cb0 tx=0x7f8ca0010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:16.666 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.656+0000 7f8cb5d0f700 1 --2- 172.21.15.40:0/3828386966 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8cb011bf30 0x7f8cb00a2170 secure :-1 s=THROTTLE_DONE pgs=9 cs=0 l=1 rx=0x7f8cb00a1cb0 tx=0x7f8ca0010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:16.666 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.656+0000 7f8cb5d0f700 1 --2- 172.21.15.40:0/3828386966 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8cb011bf30 0x7f8cb00a2170 secure :-1 s=THROTTLE_DONE pgs=9 cs=0 l=1 rx=0x7f8cb00a1cb0 tx=0x7f8ca0010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:16.666 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.656+0000 7f8cae7fc700 1 -- 172.21.15.40:0/3828386966 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8ca0005930 con 0x7f8cb011bf30 2019-12-05T05:03:16.666 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.656+0000 7f8cae7fc700 1 -- 172.21.15.40:0/3828386966 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ca0007160 con 0x7f8cb011bf30 2019-12-05T05:03:16.667 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.656+0000 7f8cb7772700 1 -- 172.21.15.40:0/3828386966 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cb00a7010 con 0x7f8cb011bf30 2019-12-05T05:03:16.667 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.656+0000 7f8cae7fc700 1 -- 172.21.15.40:0/3828386966 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8ca00121a0 con 0x7f8cb011bf30 2019-12-05T05:03:16.667 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.656+0000 7f8cb7772700 1 -- 172.21.15.40:0/3828386966 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cb00a26c0 con 0x7f8cb011bf30 2019-12-05T05:03:16.667 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.656+0000 7f8cb5d0f700 1 --2- 172.21.15.40:0/3828386966 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8cb011bf30 0x7f8cb00a2170 secure :-1 s=THROTTLE_DONE pgs=9 cs=0 l=1 rx=0x7f8cb00a1cb0 tx=0x7f8ca0010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:16.667 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.656+0000 7f8cae7fc700 1 -- 172.21.15.40:0/3828386966 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f8ca00124b0 con 0x7f8cb011bf30 2019-12-05T05:03:16.667 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.656+0000 7f8cb5d0f700 1 --2- 172.21.15.40:0/3828386966 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8cb011bf30 0x7f8cb00a2170 secure :-1 s=THROTTLE_DONE pgs=9 cs=0 l=1 rx=0x7f8cb00a1cb0 tx=0x7f8ca0010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:16.667 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.656+0000 7f8cae7fc700 1 --2- 172.21.15.40:0/3828386966 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8c9003ae60 0x7f8c9003d370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:16.668 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.656+0000 7f8cae7fc700 1 -- 172.21.15.40:0/3828386966 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 869+0+0 (secure 0 0 0) 0x7f8ca004b680 con 0x7f8cb011bf30 2019-12-05T05:03:16.668 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.656+0000 7f8cb550e700 1 --2- 172.21.15.40:0/3828386966 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8c9003ae60 0x7f8c9003d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:16.668 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.660+0000 7f8cb550e700 1 --2- 172.21.15.40:0/3828386966 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8c9003ae60 0x7f8c9003d370 crc :-1 s=READY pgs=6 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:16.669 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.664+0000 7f8cb7772700 1 -- 172.21.15.40:0/3828386966 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cb00a6630 con 0x7f8cb011bf30 2019-12-05T05:03:16.674 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.668+0000 7f8cb5d0f700 1 --2- 172.21.15.40:0/3828386966 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8cb011bf30 0x7f8cb00a2170 secure :-1 s=THROTTLE_DONE pgs=9 cs=0 l=1 rx=0x7f8cb00a1cb0 tx=0x7f8ca0010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:16.674 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.668+0000 7f8cae7fc700 1 -- 172.21.15.40:0/3828386966 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f8ca0031f40 con 0x7f8cb011bf30 2019-12-05T05:03:16.840 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.836+0000 7f8cb7772700 1 -- 172.21.15.40:0/3828386966 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "osd new", "uuid": "902543fb-f5a2-4321-9c9c-26f9b9ca8629", "id": 1} v 0) v1 -- 0x7f8cb005b8c0 con 0x7f8cb011bf30 2019-12-05T05:03:16.851 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.848+0000 7f8cb5d0f700 1 --2- 172.21.15.40:0/3828386966 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8cb011bf30 0x7f8cb00a2170 secure :-1 s=THROTTLE_DONE pgs=9 cs=0 l=1 rx=0x7f8cb00a1cb0 tx=0x7f8ca0010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:16.851 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.848+0000 7f8cae7fc700 1 -- 172.21.15.40:0/3828386966 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "902543fb-f5a2-4321-9c9c-26f9b9ca8629", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7f8ca00321a0 con 0x7f8cb011bf30 2019-12-05T05:03:16.855 INFO:teuthology.orchestra.run.smithi040.stdout:1 2019-12-05T05:03:16.858 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.856+0000 7f8cb7772700 1 -- 172.21.15.40:0/3828386966 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8c9003ae60 msgr2=0x7f8c9003d370 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:16.858 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.856+0000 7f8cb7772700 1 --2- 172.21.15.40:0/3828386966 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8c9003ae60 0x7f8c9003d370 crc :-1 s=READY pgs=6 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:16.858 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.856+0000 7f8cb7772700 1 -- 172.21.15.40:0/3828386966 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8cb011bf30 msgr2=0x7f8cb00a2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:16.858 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.856+0000 7f8cb7772700 1 --2- 172.21.15.40:0/3828386966 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8cb011bf30 0x7f8cb00a2170 secure :-1 s=READY pgs=9 cs=0 l=1 rx=0x7f8cb00a1cb0 tx=0x7f8ca0010b60).stop 2019-12-05T05:03:16.859 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.856+0000 7f8cb7772700 1 -- 172.21.15.40:0/3828386966 shutdown_connections 2019-12-05T05:03:16.859 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.856+0000 7f8cb7772700 1 --2- 172.21.15.40:0/3828386966 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8c9003ae60 0x7f8c9003d370 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:16.859 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.856+0000 7f8cb7772700 1 --2- 172.21.15.40:0/3828386966 >> v2:172.21.15.40:3300/0 conn(0x7f8cb0130450 0x7f8cb0130880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:16.859 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.856+0000 7f8cb7772700 1 --2- 172.21.15.40:0/3828386966 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8cb011bf30 0x7f8cb00a2170 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:16.859 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.856+0000 7f8cb7772700 1 --2- 172.21.15.40:0/3828386966 >> v2:172.21.15.42:3300/0 conn(0x7f8cb007ad20 0x7f8cb007b150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:16.859 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.856+0000 7f8cb7772700 1 -- 172.21.15.40:0/3828386966 shutdown_connections 2019-12-05T05:03:16.860 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.856+0000 7f8cb7772700 1 -- 172.21.15.40:0/3828386966 wait complete. 2019-12-05T05:03:16.860 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.856+0000 7f8cb7772700 1 -- 172.21.15.40:0/3828386966 >> 172.21.15.40:0/3828386966 conn(0x7f8cb011c860 msgr2=0x7f8cb00a9790 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:16.867 INFO:teuthology.orchestra.run.smithi040:> sudo ceph --cluster ceph osd new 28739ff7-bed7-4b75-8ea5-b57d088f1250 2 2019-12-05T05:03:16.962 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.956+0000 7fd48cbd7700 1 Processor -- start 2019-12-05T05:03:16.962 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.956+0000 7fd48cbd7700 1 -- start start 2019-12-05T05:03:16.962 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.956+0000 7fd48cbd7700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd48808df80 0x7fd48809f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:16.962 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.956+0000 7fd48cbd7700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd48809fc60 con 0x7fd4880a7740 2019-12-05T05:03:16.962 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.956+0000 7fd48cbd7700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4880a0710 con 0x7fd48809f830 2019-12-05T05:03:16.963 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.956+0000 7fd48cbd7700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4880a0b00 con 0x7fd48808df80 2019-12-05T05:03:16.963 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.956+0000 7fd48659c700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd48808df80 0x7fd48809f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:16.963 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.956+0000 7fd486d9d700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fd4880a7740 0x7fd48807e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:59700/0 (socket says 172.21.15.40:59700) 2019-12-05T05:03:16.963 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.956+0000 7fd486d9d700 1 -- 172.21.15.40:0/1486730228 learned_addr learned my addr 172.21.15.40:0/1486730228 (peer_addr_for_me v1:172.21.15.40:0/0) 2019-12-05T05:03:16.963 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.956+0000 7fd485d9b700 1 -- 172.21.15.40:0/1486730228 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2940152868 0 0) 0x7fd4880a0710 con 0x7fd48809f830 2019-12-05T05:03:16.963 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.956+0000 7fd485d9b700 1 -- 172.21.15.40:0/1486730228 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4700030b0 con 0x7fd48809f830 2019-12-05T05:03:16.963 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.956+0000 7fd48659c700 1 -- 172.21.15.40:0/1486730228 >> v1:172.21.15.42:6789/0 conn(0x7fd48809f830 legacy=0x7fd48814efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:16.964 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.956+0000 7fd48659c700 1 -- 172.21.15.40:0/1486730228 >> v1:172.21.15.40:6789/0 conn(0x7fd4880a7740 legacy=0x7fd48807e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:16.964 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.956+0000 7fd485d9b700 1 -- 172.21.15.40:0/1486730228 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2905195751 0 0) 0x7fd48809fc60 con 0x7fd4880a7740 2019-12-05T05:03:16.964 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.956+0000 7fd48659c700 1 -- 172.21.15.40:0/1486730228 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd48807ef20 con 0x7fd48808df80 2019-12-05T05:03:16.964 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.956+0000 7fd48659c700 1 --2- 172.21.15.40:0/1486730228 >> v2:172.21.15.40:3300/0 conn(0x7fd48808df80 0x7fd48809f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fd47c008790 tx=0x7fd47c00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:16.964 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.956+0000 7fd48659c700 1 --2- 172.21.15.40:0/1486730228 >> v2:172.21.15.40:3300/0 conn(0x7fd48808df80 0x7fd48809f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fd47c008790 tx=0x7fd47c00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:16.964 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.956+0000 7fd48659c700 1 --2- 172.21.15.40:0/1486730228 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd48808df80 0x7fd48809f2e0 secure :-1 s=READY pgs=19 cs=0 l=1 rx=0x7fd47c008790 tx=0x7fd47c00fbf0).ready entity=mon.0 client_cookie=aa9a896464fce0bd server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:16.965 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.960+0000 7fd48659c700 1 --2- 172.21.15.40:0/1486730228 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd48808df80 0x7fd48809f2e0 secure :-1 s=THROTTLE_DONE pgs=19 cs=0 l=1 rx=0x7fd47c008790 tx=0x7fd47c00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:16.965 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.960+0000 7fd485d9b700 1 -- 172.21.15.40:0/1486730228 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd47c011270 con 0x7fd48808df80 2019-12-05T05:03:16.965 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.960+0000 7fd48659c700 1 --2- 172.21.15.40:0/1486730228 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd48808df80 0x7fd48809f2e0 secure :-1 s=THROTTLE_DONE pgs=19 cs=0 l=1 rx=0x7fd47c008790 tx=0x7fd47c00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:16.965 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.960+0000 7fd485d9b700 1 -- 172.21.15.40:0/1486730228 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd47c008bc0 con 0x7fd48808df80 2019-12-05T05:03:16.966 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.960+0000 7fd48659c700 1 --2- 172.21.15.40:0/1486730228 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd48808df80 0x7fd48809f2e0 secure :-1 s=THROTTLE_DONE pgs=19 cs=0 l=1 rx=0x7fd47c008790 tx=0x7fd47c00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:16.966 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.960+0000 7fd485d9b700 1 -- 172.21.15.40:0/1486730228 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd47c011850 con 0x7fd48808df80 2019-12-05T05:03:16.966 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.960+0000 7fd48cbd7700 1 -- 172.21.15.40:0/1486730228 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd48808df80 msgr2=0x7fd48809f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:16.966 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.960+0000 7fd48cbd7700 1 --2- 172.21.15.40:0/1486730228 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd48808df80 0x7fd48809f2e0 secure :-1 s=READY pgs=19 cs=0 l=1 rx=0x7fd47c008790 tx=0x7fd47c00fbf0).stop 2019-12-05T05:03:16.966 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.960+0000 7fd48cbd7700 1 -- 172.21.15.40:0/1486730228 shutdown_connections 2019-12-05T05:03:16.966 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.960+0000 7fd48cbd7700 1 --2- 172.21.15.40:0/1486730228 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd48808df80 0x7fd48809f2e0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:16.966 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.960+0000 7fd48cbd7700 1 -- 172.21.15.40:0/1486730228 shutdown_connections 2019-12-05T05:03:16.967 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.960+0000 7fd48cbd7700 1 -- 172.21.15.40:0/1486730228 wait complete. 2019-12-05T05:03:16.967 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.960+0000 7fd48cbd7700 1 -- 172.21.15.40:0/1486730228 >> 172.21.15.40:0/1486730228 conn(0x7fd4880aabe0 msgr2=0x7fd4880a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:16.970 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.960+0000 7fd48cbd7700 1 Processor -- start 2019-12-05T05:03:16.970 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.964+0000 7fd48cbd7700 1 -- start start 2019-12-05T05:03:16.971 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.964+0000 7fd48cbd7700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd4881247f0 0x7fd488144810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:16.971 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.964+0000 7fd48cbd7700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd4880722c0 0x7fd488074700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:16.971 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.964+0000 7fd48cbd7700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd48808f1b0 con 0x7fd488120a60 2019-12-05T05:03:16.971 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.964+0000 7fd48cbd7700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd48808f4c0 con 0x7fd4880722c0 2019-12-05T05:03:16.971 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.964+0000 7fd48cbd7700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd48808f760 con 0x7fd4881247f0 2019-12-05T05:03:16.971 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.964+0000 7fd48759e700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd4880722c0 0x7fd488074700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:16.972 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.964+0000 7fd48659c700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd4881247f0 0x7fd488144810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:16.972 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.964+0000 7fd48759e700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd4880722c0 0x7fd488074700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36042/0 (socket says 172.21.15.40:36042) 2019-12-05T05:03:16.972 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.964+0000 7fd48759e700 1 -- 172.21.15.40:0/1048583769 learned_addr learned my addr 172.21.15.40:0/1048583769 (peer_addr_for_me v2:172.21.15.40:0/0) 2019-12-05T05:03:16.972 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.964+0000 7fd4777fe700 1 -- 172.21.15.40:0/1048583769 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1243345050 0 0) 0x7fd48808f1b0 con 0x7fd488120a60 2019-12-05T05:03:16.972 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.964+0000 7fd48659c700 1 -- 172.21.15.40:0/1048583769 >> v2:172.21.15.40:3300/0 conn(0x7fd4880722c0 msgr2=0x7fd488074700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:16.972 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.964+0000 7fd48659c700 1 --2- 172.21.15.40:0/1048583769 >> v2:172.21.15.40:3300/0 conn(0x7fd4880722c0 0x7fd488074700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:16.972 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.964+0000 7fd48659c700 1 -- 172.21.15.40:0/1048583769 >> v1:172.21.15.42:6789/0 conn(0x7fd488120a60 legacy=0x7fd488123ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:16.973 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.964+0000 7fd48659c700 1 -- 172.21.15.40:0/1048583769 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd488144d60 con 0x7fd4881247f0 2019-12-05T05:03:16.973 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.964+0000 7fd48759e700 1 --2- 172.21.15.40:0/1048583769 >> v2:172.21.15.40:3300/0 conn(0x7fd4880722c0 0x7fd488074700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:03:16.973 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.964+0000 7fd48659c700 1 --2- 172.21.15.40:0/1048583769 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd4881247f0 0x7fd488144810 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fd47c008790 tx=0x7fd47c011910).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:16.973 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.964+0000 7fd48659c700 1 --2- 172.21.15.40:0/1048583769 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd4881247f0 0x7fd488144810 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fd47c008790 tx=0x7fd47c011910).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:16.973 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.964+0000 7fd48659c700 1 --2- 172.21.15.40:0/1048583769 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd4881247f0 0x7fd488144810 secure :-1 s=READY pgs=10 cs=0 l=1 rx=0x7fd47c008790 tx=0x7fd47c011910).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:16.973 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.964+0000 7fd48659c700 1 --2- 172.21.15.40:0/1048583769 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd4881247f0 0x7fd488144810 secure :-1 s=THROTTLE_DONE pgs=10 cs=0 l=1 rx=0x7fd47c008790 tx=0x7fd47c011910).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:16.973 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.964+0000 7fd4777fe700 1 -- 172.21.15.40:0/1048583769 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd47c00f550 con 0x7fd4881247f0 2019-12-05T05:03:16.974 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.964+0000 7fd48659c700 1 --2- 172.21.15.40:0/1048583769 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd4881247f0 0x7fd488144810 secure :-1 s=THROTTLE_DONE pgs=10 cs=0 l=1 rx=0x7fd47c008790 tx=0x7fd47c011910).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:16.974 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.964+0000 7fd48659c700 1 --2- 172.21.15.40:0/1048583769 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd4881247f0 0x7fd488144810 secure :-1 s=THROTTLE_DONE pgs=10 cs=0 l=1 rx=0x7fd47c008790 tx=0x7fd47c011910).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:16.974 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.964+0000 7fd4777fe700 1 -- 172.21.15.40:0/1048583769 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd47c011470 con 0x7fd4881247f0 2019-12-05T05:03:16.974 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.964+0000 7fd48cbd7700 1 -- 172.21.15.40:0/1048583769 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd488141fe0 con 0x7fd4881247f0 2019-12-05T05:03:16.974 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.964+0000 7fd4777fe700 1 -- 172.21.15.40:0/1048583769 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd47c005540 con 0x7fd4881247f0 2019-12-05T05:03:16.974 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.964+0000 7fd48cbd7700 1 -- 172.21.15.40:0/1048583769 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd488118f80 con 0x7fd4881247f0 2019-12-05T05:03:16.974 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.968+0000 7fd48659c700 1 --2- 172.21.15.40:0/1048583769 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd4881247f0 0x7fd488144810 secure :-1 s=THROTTLE_DONE pgs=10 cs=0 l=1 rx=0x7fd47c008790 tx=0x7fd47c011910).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:16.974 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.968+0000 7fd4777fe700 1 -- 172.21.15.40:0/1048583769 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fd47c005810 con 0x7fd4881247f0 2019-12-05T05:03:16.975 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.968+0000 7fd48659c700 1 --2- 172.21.15.40:0/1048583769 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd4881247f0 0x7fd488144810 secure :-1 s=THROTTLE_DONE pgs=10 cs=0 l=1 rx=0x7fd47c008790 tx=0x7fd47c011910).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:16.975 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.968+0000 7fd4777fe700 1 --2- 172.21.15.40:0/1048583769 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd46c03ace0 0x7fd46c03d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:16.975 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.968+0000 7fd4777fe700 1 -- 172.21.15.40:0/1048583769 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 984+0+0 (secure 0 0 0) 0x7fd47c04bb30 con 0x7fd4881247f0 2019-12-05T05:03:16.975 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.968+0000 7fd486d9d700 1 --2- 172.21.15.40:0/1048583769 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd46c03ace0 0x7fd46c03d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:16.976 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.968+0000 7fd486d9d700 1 --2- 172.21.15.40:0/1048583769 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd46c03ace0 0x7fd46c03d1f0 crc :-1 s=READY pgs=7 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:16.976 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.972+0000 7fd48cbd7700 1 -- 172.21.15.40:0/1048583769 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd48805d110 con 0x7fd4881247f0 2019-12-05T05:03:16.979 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.976+0000 7fd48659c700 1 --2- 172.21.15.40:0/1048583769 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd4881247f0 0x7fd488144810 secure :-1 s=THROTTLE_DONE pgs=10 cs=0 l=1 rx=0x7fd47c008790 tx=0x7fd47c011910).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:16.979 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:16.976+0000 7fd4777fe700 1 -- 172.21.15.40:0/1048583769 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fd47c032bb0 con 0x7fd4881247f0 2019-12-05T05:03:17.152 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.148+0000 7fd48cbd7700 1 -- 172.21.15.40:0/1048583769 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "osd new", "uuid": "28739ff7-bed7-4b75-8ea5-b57d088f1250", "id": 2} v 0) v1 -- 0x7fd48805b8c0 con 0x7fd4881247f0 2019-12-05T05:03:17.169 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.160+0000 7fd48659c700 1 --2- 172.21.15.40:0/1048583769 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd4881247f0 0x7fd488144810 secure :-1 s=THROTTLE_DONE pgs=10 cs=0 l=1 rx=0x7fd47c008790 tx=0x7fd47c011910).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:17.169 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.160+0000 7fd4777fe700 1 -- 172.21.15.40:0/1048583769 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "28739ff7-bed7-4b75-8ea5-b57d088f1250", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7fd47c032e10 con 0x7fd4881247f0 2019-12-05T05:03:17.169 INFO:teuthology.orchestra.run.smithi040.stdout:2 2019-12-05T05:03:17.171 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.168+0000 7fd48cbd7700 1 -- 172.21.15.40:0/1048583769 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd46c03ace0 msgr2=0x7fd46c03d1f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:17.171 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.168+0000 7fd48cbd7700 1 --2- 172.21.15.40:0/1048583769 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd46c03ace0 0x7fd46c03d1f0 crc :-1 s=READY pgs=7 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:17.171 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.168+0000 7fd48cbd7700 1 -- 172.21.15.40:0/1048583769 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd4881247f0 msgr2=0x7fd488144810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:17.171 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.168+0000 7fd48cbd7700 1 --2- 172.21.15.40:0/1048583769 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd4881247f0 0x7fd488144810 secure :-1 s=READY pgs=10 cs=0 l=1 rx=0x7fd47c008790 tx=0x7fd47c011910).stop 2019-12-05T05:03:17.171 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.168+0000 7fd48cbd7700 1 -- 172.21.15.40:0/1048583769 shutdown_connections 2019-12-05T05:03:17.171 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.168+0000 7fd48cbd7700 1 --2- 172.21.15.40:0/1048583769 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd46c03ace0 0x7fd46c03d1f0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:17.171 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.168+0000 7fd48cbd7700 1 --2- 172.21.15.40:0/1048583769 >> v2:172.21.15.40:3300/0 conn(0x7fd4880722c0 0x7fd488074700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:17.172 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.168+0000 7fd48cbd7700 1 --2- 172.21.15.40:0/1048583769 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd4881247f0 0x7fd488144810 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:17.172 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.168+0000 7fd48cbd7700 1 -- 172.21.15.40:0/1048583769 shutdown_connections 2019-12-05T05:03:17.172 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.168+0000 7fd48cbd7700 1 -- 172.21.15.40:0/1048583769 wait complete. 2019-12-05T05:03:17.172 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.168+0000 7fd48cbd7700 1 -- 172.21.15.40:0/1048583769 >> 172.21.15.40:0/1048583769 conn(0x7fd48807e790 msgr2=0x7fd4880a9770 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:17.183 INFO:teuthology.orchestra.run.smithi040:> sudo ceph --cluster ceph osd new 1960f66b-2d40-4bb9-a289-ce8cd0c1c142 3 2019-12-05T05:03:17.304 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.272+0000 7f6ce4777700 1 Processor -- start 2019-12-05T05:03:17.308 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.272+0000 7f6ce4777700 1 -- start start 2019-12-05T05:03:17.308 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.272+0000 7f6ce4777700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6cdc08df80 0x7f6cdc09f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:17.308 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.272+0000 7f6ce4777700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cdc09fc60 con 0x7f6cdc09f830 2019-12-05T05:03:17.309 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.272+0000 7f6ce4777700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cdc0a0710 con 0x7f6cdc0a7740 2019-12-05T05:03:17.309 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.272+0000 7f6ce4777700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cdc0a0b00 con 0x7f6cdc08df80 2019-12-05T05:03:17.309 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.272+0000 7f6ce2513700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6cdc08df80 0x7f6cdc09f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:17.309 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.272+0000 7f6ce3515700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f6cdc09f830 0x7f6cdc14efe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:59718/0 (socket says 172.21.15.40:59718) 2019-12-05T05:03:17.309 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.272+0000 7f6ce3515700 1 -- 172.21.15.40:0/2064173280 learned_addr learned my addr 172.21.15.40:0/2064173280 (peer_addr_for_me v1:172.21.15.40:0/0) 2019-12-05T05:03:17.310 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.276+0000 7f6ce1d12700 1 -- 172.21.15.40:0/2064173280 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3340127459 0 0) 0x7f6cdc09fc60 con 0x7f6cdc09f830 2019-12-05T05:03:17.310 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.276+0000 7f6ce1d12700 1 -- 172.21.15.40:0/2064173280 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6cc80030b0 con 0x7f6cdc09f830 2019-12-05T05:03:17.310 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.276+0000 7f6ce1d12700 1 -- 172.21.15.40:0/2064173280 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3756399127 0 0) 0x7f6cdc0a0710 con 0x7f6cdc0a7740 2019-12-05T05:03:17.310 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.276+0000 7f6ce1d12700 1 -- 172.21.15.40:0/2064173280 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6cdc09fc60 con 0x7f6cdc0a7740 2019-12-05T05:03:17.310 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.276+0000 7f6ce2513700 1 -- 172.21.15.40:0/2064173280 >> v1:172.21.15.42:6789/0 conn(0x7f6cdc0a7740 legacy=0x7f6cdc07e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:17.310 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.276+0000 7f6ce2513700 1 -- 172.21.15.40:0/2064173280 >> v1:172.21.15.40:6789/0 conn(0x7f6cdc09f830 legacy=0x7f6cdc14efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:17.310 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.276+0000 7f6ce2513700 1 -- 172.21.15.40:0/2064173280 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cdc07ef20 con 0x7f6cdc08df80 2019-12-05T05:03:17.311 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.276+0000 7f6ce2513700 1 --2- 172.21.15.40:0/2064173280 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6cdc08df80 0x7f6cdc09f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f6ccc002f30 tx=0x7f6ccc00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:17.311 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.276+0000 7f6ce2513700 1 --2- 172.21.15.40:0/2064173280 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6cdc08df80 0x7f6cdc09f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f6ccc002f30 tx=0x7f6ccc00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:17.311 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.276+0000 7f6ce2513700 1 --2- 172.21.15.40:0/2064173280 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6cdc08df80 0x7f6cdc09f2e0 secure :-1 s=READY pgs=11 cs=0 l=1 rx=0x7f6ccc002f30 tx=0x7f6ccc00fbf0).ready entity=mon.2 client_cookie=70a5e299f2ea0a4f server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:17.311 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.276+0000 7f6ce2513700 1 --2- 172.21.15.40:0/2064173280 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6cdc08df80 0x7f6cdc09f2e0 secure :-1 s=THROTTLE_DONE pgs=11 cs=0 l=1 rx=0x7f6ccc002f30 tx=0x7f6ccc00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:17.312 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.276+0000 7f6ce2513700 1 --2- 172.21.15.40:0/2064173280 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6cdc08df80 0x7f6cdc09f2e0 secure :-1 s=THROTTLE_DONE pgs=11 cs=0 l=1 rx=0x7f6ccc002f30 tx=0x7f6ccc00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:17.312 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.276+0000 7f6ce2513700 1 --2- 172.21.15.40:0/2064173280 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6cdc08df80 0x7f6cdc09f2e0 secure :-1 s=THROTTLE_DONE pgs=11 cs=0 l=1 rx=0x7f6ccc002f30 tx=0x7f6ccc00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:17.312 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.276+0000 7f6ce1d12700 1 -- 172.21.15.40:0/2064173280 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6ccc0112b0 con 0x7f6cdc08df80 2019-12-05T05:03:17.312 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.276+0000 7f6ce1d12700 1 -- 172.21.15.40:0/2064173280 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ccc008bc0 con 0x7f6cdc08df80 2019-12-05T05:03:17.312 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.276+0000 7f6ce1d12700 1 -- 172.21.15.40:0/2064173280 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6ccc011890 con 0x7f6cdc08df80 2019-12-05T05:03:17.312 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.276+0000 7f6ce4777700 1 -- 172.21.15.40:0/2064173280 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6cdc08df80 msgr2=0x7f6cdc09f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:17.312 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.276+0000 7f6ce4777700 1 --2- 172.21.15.40:0/2064173280 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6cdc08df80 0x7f6cdc09f2e0 secure :-1 s=READY pgs=11 cs=0 l=1 rx=0x7f6ccc002f30 tx=0x7f6ccc00fbf0).stop 2019-12-05T05:03:17.313 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.276+0000 7f6ce4777700 1 -- 172.21.15.40:0/2064173280 shutdown_connections 2019-12-05T05:03:17.313 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.276+0000 7f6ce4777700 1 --2- 172.21.15.40:0/2064173280 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6cdc08df80 0x7f6cdc09f2e0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:17.313 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.276+0000 7f6ce4777700 1 -- 172.21.15.40:0/2064173280 shutdown_connections 2019-12-05T05:03:17.313 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.276+0000 7f6ce4777700 1 -- 172.21.15.40:0/2064173280 wait complete. 2019-12-05T05:03:17.313 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.276+0000 7f6ce4777700 1 -- 172.21.15.40:0/2064173280 >> 172.21.15.40:0/2064173280 conn(0x7f6cdc0aabe0 msgr2=0x7f6cdc0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:17.313 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.280+0000 7f6ce4777700 1 Processor -- start 2019-12-05T05:03:17.313 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.280+0000 7f6ce4777700 1 -- start start 2019-12-05T05:03:17.314 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.280+0000 7f6ce4777700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6cdc08eeb0 0x7f6cdc0647d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:17.314 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.280+0000 7f6ce4777700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6cdc128960 0x7f6cdc065540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:17.314 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.280+0000 7f6ce4777700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cdc142530 con 0x7f6cdc07ba40 2019-12-05T05:03:17.314 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.280+0000 7f6ce4777700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6cdc143210 con 0x7f6cdc08eeb0 2019-12-05T05:03:17.314 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.280+0000 7f6ce4777700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cdc1428b0 con 0x7f6cdc128960 2019-12-05T05:03:17.314 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.280+0000 7f6ce2513700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6cdc128960 0x7f6cdc065540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:17.314 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.280+0000 7f6ce2513700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6cdc128960 0x7f6cdc065540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:33120/0 (socket says 172.21.15.40:33120) 2019-12-05T05:03:17.315 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.280+0000 7f6ce2513700 1 -- 172.21.15.40:0/2220417725 learned_addr learned my addr 172.21.15.40:0/2220417725 (peer_addr_for_me v2:172.21.15.40:0/0) 2019-12-05T05:03:17.315 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.280+0000 7f6ce2d14700 1 --2- 172.21.15.40:0/2220417725 >> v2:172.21.15.42:3300/0 conn(0x7f6cdc08eeb0 0x7f6cdc0647d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:17.315 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.284+0000 7f6ce2513700 1 -- 172.21.15.40:0/2220417725 >> v2:172.21.15.42:3300/0 conn(0x7f6cdc08eeb0 msgr2=0x7f6cdc0647d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:17.315 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.284+0000 7f6ce2513700 1 --2- 172.21.15.40:0/2220417725 >> v2:172.21.15.42:3300/0 conn(0x7f6cdc08eeb0 0x7f6cdc0647d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:17.315 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.284+0000 7f6ce2513700 1 -- 172.21.15.40:0/2220417725 >> v1:172.21.15.42:6789/0 conn(0x7f6cdc07ba40 legacy=0x7f6cdc063190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:17.315 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.284+0000 7f6ce2513700 1 -- 172.21.15.40:0/2220417725 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cdc128d90 con 0x7f6cdc128960 2019-12-05T05:03:17.315 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.284+0000 7f6ce2513700 1 --2- 172.21.15.40:0/2220417725 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6cdc128960 0x7f6cdc065540 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f6ccc002f30 tx=0x7f6ccc008e70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:17.316 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.284+0000 7f6ce2513700 1 --2- 172.21.15.40:0/2220417725 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6cdc128960 0x7f6cdc065540 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f6ccc002f30 tx=0x7f6ccc008e70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:17.316 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.284+0000 7f6ce2513700 1 --2- 172.21.15.40:0/2220417725 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6cdc128960 0x7f6cdc065540 secure :-1 s=READY pgs=12 cs=0 l=1 rx=0x7f6ccc002f30 tx=0x7f6ccc008e70).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:17.317 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.284+0000 7f6ce2513700 1 --2- 172.21.15.40:0/2220417725 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6cdc128960 0x7f6cdc065540 secure :-1 s=THROTTLE_DONE pgs=12 cs=0 l=1 rx=0x7f6ccc002f30 tx=0x7f6ccc008e70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:17.317 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.284+0000 7f6cd37fe700 1 -- 172.21.15.40:0/2220417725 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6ccc0118b0 con 0x7f6cdc128960 2019-12-05T05:03:17.317 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.284+0000 7f6ce2513700 1 --2- 172.21.15.40:0/2220417725 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6cdc128960 0x7f6cdc065540 secure :-1 s=THROTTLE_DONE pgs=12 cs=0 l=1 rx=0x7f6ccc002f30 tx=0x7f6ccc008e70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:17.317 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.284+0000 7f6ce2513700 1 --2- 172.21.15.40:0/2220417725 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6cdc128960 0x7f6cdc065540 secure :-1 s=THROTTLE_DONE pgs=12 cs=0 l=1 rx=0x7f6ccc002f30 tx=0x7f6ccc008e70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:17.317 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.284+0000 7f6cd37fe700 1 -- 172.21.15.40:0/2220417725 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ccc0114b0 con 0x7f6cdc128960 2019-12-05T05:03:17.318 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.284+0000 7f6cd37fe700 1 -- 172.21.15.40:0/2220417725 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6ccc005540 con 0x7f6cdc128960 2019-12-05T05:03:17.318 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.284+0000 7f6ce4777700 1 -- 172.21.15.40:0/2220417725 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cdc065a90 con 0x7f6cdc128960 2019-12-05T05:03:17.318 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.284+0000 7f6ce4777700 1 -- 172.21.15.40:0/2220417725 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cdc07be70 con 0x7f6cdc128960 2019-12-05T05:03:17.318 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.284+0000 7f6ce2513700 1 --2- 172.21.15.40:0/2220417725 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6cdc128960 0x7f6cdc065540 secure :-1 s=THROTTLE_DONE pgs=12 cs=0 l=1 rx=0x7f6ccc002f30 tx=0x7f6ccc008e70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:17.319 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.284+0000 7f6ce2513700 1 --2- 172.21.15.40:0/2220417725 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6cdc128960 0x7f6cdc065540 secure :-1 s=THROTTLE_DONE pgs=12 cs=0 l=1 rx=0x7f6ccc002f30 tx=0x7f6ccc008e70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:17.319 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.284+0000 7f6cd37fe700 1 -- 172.21.15.40:0/2220417725 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f6ccc005810 con 0x7f6cdc128960 2019-12-05T05:03:17.319 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.284+0000 7f6cd37fe700 1 --2- 172.21.15.40:0/2220417725 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6cc403ae10 0x7f6cc403d320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:17.319 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.284+0000 7f6cd37fe700 1 -- 172.21.15.40:0/2220417725 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1099+0+0 (secure 0 0 0) 0x7f6ccc04b580 con 0x7f6cdc128960 2019-12-05T05:03:17.320 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.284+0000 7f6ce2d14700 1 --2- 172.21.15.40:0/2220417725 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6cc403ae10 0x7f6cc403d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:17.320 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.284+0000 7f6ce2d14700 1 --2- 172.21.15.40:0/2220417725 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6cc403ae10 0x7f6cc403d320 crc :-1 s=READY pgs=8 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:17.320 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.288+0000 7f6ce4777700 1 -- 172.21.15.40:0/2220417725 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6cdc05d110 con 0x7f6cdc128960 2019-12-05T05:03:17.320 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.292+0000 7f6ce2513700 1 --2- 172.21.15.40:0/2220417725 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6cdc128960 0x7f6cdc065540 secure :-1 s=THROTTLE_DONE pgs=12 cs=0 l=1 rx=0x7f6ccc002f30 tx=0x7f6ccc008e70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:17.320 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.292+0000 7f6cd37fe700 1 -- 172.21.15.40:0/2220417725 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f6ccc0320c0 con 0x7f6cdc128960 2019-12-05T05:03:17.472 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.468+0000 7f6ce4777700 1 -- 172.21.15.40:0/2220417725 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "osd new", "uuid": "1960f66b-2d40-4bb9-a289-ce8cd0c1c142", "id": 3} v 0) v1 -- 0x7f6cdc05b8c0 con 0x7f6cdc128960 2019-12-05T05:03:17.485 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.480+0000 7f6ce2513700 1 --2- 172.21.15.40:0/2220417725 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6cdc128960 0x7f6cdc065540 secure :-1 s=THROTTLE_DONE pgs=12 cs=0 l=1 rx=0x7f6ccc002f30 tx=0x7f6ccc008e70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:17.485 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.480+0000 7f6cd37fe700 1 -- 172.21.15.40:0/2220417725 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "1960f66b-2d40-4bb9-a289-ce8cd0c1c142", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7f6ccc033410 con 0x7f6cdc128960 2019-12-05T05:03:17.487 INFO:teuthology.orchestra.run.smithi040.stdout:3 2019-12-05T05:03:17.491 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.488+0000 7f6ce4777700 1 -- 172.21.15.40:0/2220417725 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6cc403ae10 msgr2=0x7f6cc403d320 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:17.491 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.488+0000 7f6ce4777700 1 --2- 172.21.15.40:0/2220417725 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6cc403ae10 0x7f6cc403d320 crc :-1 s=READY pgs=8 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:17.491 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.488+0000 7f6ce4777700 1 -- 172.21.15.40:0/2220417725 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6cdc128960 msgr2=0x7f6cdc065540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:17.492 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.488+0000 7f6ce4777700 1 --2- 172.21.15.40:0/2220417725 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6cdc128960 0x7f6cdc065540 secure :-1 s=READY pgs=12 cs=0 l=1 rx=0x7f6ccc002f30 tx=0x7f6ccc008e70).stop 2019-12-05T05:03:17.492 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.488+0000 7f6ce4777700 1 -- 172.21.15.40:0/2220417725 shutdown_connections 2019-12-05T05:03:17.492 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.488+0000 7f6ce4777700 1 --2- 172.21.15.40:0/2220417725 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6cc403ae10 0x7f6cc403d320 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:17.492 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.488+0000 7f6ce4777700 1 --2- 172.21.15.40:0/2220417725 >> v2:172.21.15.42:3300/0 conn(0x7f6cdc08eeb0 0x7f6cdc0647d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:17.493 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.488+0000 7f6ce4777700 1 --2- 172.21.15.40:0/2220417725 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6cdc128960 0x7f6cdc065540 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:17.493 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.488+0000 7f6ce4777700 1 -- 172.21.15.40:0/2220417725 shutdown_connections 2019-12-05T05:03:17.493 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.488+0000 7f6ce4777700 1 -- 172.21.15.40:0/2220417725 wait complete. 2019-12-05T05:03:17.493 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.488+0000 7f6ce4777700 1 -- 172.21.15.40:0/2220417725 >> 172.21.15.40:0/2220417725 conn(0x7f6cdc0a7740 msgr2=0x7f6cdc0a6b00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:17.503 INFO:teuthology.orchestra.run.smithi040:> sudo ceph --cluster ceph osd new 45bbf1bd-7c8a-4a0d-b76b-afa6148a855c 4 2019-12-05T05:03:17.603 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.596+0000 7f3ebab75700 1 Processor -- start 2019-12-05T05:03:17.604 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.596+0000 7f3ebab75700 1 -- start start 2019-12-05T05:03:17.604 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.596+0000 7f3ebab75700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3eb40a7740 0x7f3eb408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:17.604 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.596+0000 7f3ebab75700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f3eb407e720 0x7f3eb407eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:17.604 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.596+0000 7f3ebab75700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3eb40a0710 con 0x7f3eb409f2e0 2019-12-05T05:03:17.604 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.596+0000 7f3ebab75700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3eb40a0b00 con 0x7f3eb407e720 2019-12-05T05:03:17.605 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.596+0000 7f3ebab75700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3eb408e5b0 con 0x7f3eb40a7740 2019-12-05T05:03:17.605 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.596+0000 7f3eb8911700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f3eb407e720 0x7f3eb407eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:17.605 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.596+0000 7f3eb9112700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3eb40a7740 0x7f3eb408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:17.605 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.600+0000 7f3eb9112700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3eb40a7740 0x7f3eb408df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:33132/0 (socket says 172.21.15.40:33132) 2019-12-05T05:03:17.606 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.600+0000 7f3eb8911700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f3eb407e720 0x7f3eb407eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36064/0 (socket says 172.21.15.40:36064) 2019-12-05T05:03:17.606 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.600+0000 7f3eb9112700 1 -- 172.21.15.40:0/3026818052 learned_addr learned my addr 172.21.15.40:0/3026818052 (peer_addr_for_me v2:172.21.15.40:0/0) 2019-12-05T05:03:17.606 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.600+0000 7f3eb8911700 1 -- 172.21.15.40:0/3026818052 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3eb40a7740 msgr2=0x7f3eb408df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:03:17.606 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.600+0000 7f3eb8911700 1 --2- 172.21.15.40:0/3026818052 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3eb40a7740 0x7f3eb408df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:03:17.606 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.600+0000 7f3eb8911700 1 -- 172.21.15.40:0/3026818052 >> v1:172.21.15.42:6789/0 conn(0x7f3eb409f2e0 legacy=0x7f3eb409f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:17.607 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.600+0000 7f3eb8911700 1 -- 172.21.15.40:0/3026818052 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3eb407f0a0 con 0x7f3eb407e720 2019-12-05T05:03:17.607 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.600+0000 7f3eb8911700 1 --2- 172.21.15.40:0/3026818052 >> v2:172.21.15.40:3300/0 conn(0x7f3eb407e720 0x7f3eb407eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f3ea4008790 tx=0x7f3ea400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:17.607 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.600+0000 7f3eb8911700 1 --2- 172.21.15.40:0/3026818052 >> v2:172.21.15.40:3300/0 conn(0x7f3eb407e720 0x7f3eb407eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f3ea4008790 tx=0x7f3ea400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:17.607 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.600+0000 7f3eb8911700 1 --2- 172.21.15.40:0/3026818052 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3eb407e720 0x7f3eb407eb50 secure :-1 s=READY pgs=21 cs=0 l=1 rx=0x7f3ea4008790 tx=0x7f3ea400fc70).ready entity=mon.0 client_cookie=306cdf4735ba947e server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:17.608 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.600+0000 7f3eb8911700 1 --2- 172.21.15.40:0/3026818052 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3eb407e720 0x7f3eb407eb50 secure :-1 s=THROTTLE_DONE pgs=21 cs=0 l=1 rx=0x7f3ea4008790 tx=0x7f3ea400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:17.608 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.600+0000 7f3eabfff700 1 -- 172.21.15.40:0/3026818052 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3ea40112f0 con 0x7f3eb407e720 2019-12-05T05:03:17.608 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.600+0000 7f3eb8911700 1 --2- 172.21.15.40:0/3026818052 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3eb407e720 0x7f3eb407eb50 secure :-1 s=THROTTLE_DONE pgs=21 cs=0 l=1 rx=0x7f3ea4008790 tx=0x7f3ea400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:17.609 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.600+0000 7f3eb8911700 1 --2- 172.21.15.40:0/3026818052 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3eb407e720 0x7f3eb407eb50 secure :-1 s=THROTTLE_DONE pgs=21 cs=0 l=1 rx=0x7f3ea4008790 tx=0x7f3ea400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:17.610 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.600+0000 7f3eabfff700 1 -- 172.21.15.40:0/3026818052 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ea4008ad0 con 0x7f3eb407e720 2019-12-05T05:03:17.610 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.600+0000 7f3eabfff700 1 -- 172.21.15.40:0/3026818052 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3ea40118d0 con 0x7f3eb407e720 2019-12-05T05:03:17.610 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.600+0000 7f3ebab75700 1 -- 172.21.15.40:0/3026818052 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3eb407e720 msgr2=0x7f3eb407eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:17.610 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.600+0000 7f3ebab75700 1 --2- 172.21.15.40:0/3026818052 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3eb407e720 0x7f3eb407eb50 secure :-1 s=READY pgs=21 cs=0 l=1 rx=0x7f3ea4008790 tx=0x7f3ea400fc70).stop 2019-12-05T05:03:17.611 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.600+0000 7f3ebab75700 1 -- 172.21.15.40:0/3026818052 shutdown_connections 2019-12-05T05:03:17.611 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.600+0000 7f3ebab75700 1 --2- 172.21.15.40:0/3026818052 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3eb40a7740 0x7f3eb408df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:03:17.611 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.600+0000 7f3ebab75700 1 --2- 172.21.15.40:0/3026818052 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3eb407e720 0x7f3eb407eb50 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:17.611 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.600+0000 7f3ebab75700 1 -- 172.21.15.40:0/3026818052 shutdown_connections 2019-12-05T05:03:17.612 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.600+0000 7f3ebab75700 1 -- 172.21.15.40:0/3026818052 wait complete. 2019-12-05T05:03:17.612 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.600+0000 7f3ebab75700 1 -- 172.21.15.40:0/3026818052 >> 172.21.15.40:0/3026818052 conn(0x7f3eb40aabe0 msgr2=0x7f3eb40a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:17.617 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.604+0000 7f3ebab75700 1 Processor -- start 2019-12-05T05:03:17.617 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.604+0000 7f3ebab75700 1 -- start start 2019-12-05T05:03:17.617 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.604+0000 7f3ebab75700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f3eb40a9ad0 0x7f3eb411af60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:17.618 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.604+0000 7f3ebab75700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3eb4118410 0x7f3eb411b4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:17.618 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.604+0000 7f3ebab75700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3eb4130580 con 0x7f3eb4117a50 2019-12-05T05:03:17.618 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.604+0000 7f3ebab75700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3eb4119fb0 con 0x7f3eb40a9ad0 2019-12-05T05:03:17.618 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.604+0000 7f3ebab75700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3eb4116bb0 con 0x7f3eb4118410 2019-12-05T05:03:17.619 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.604+0000 7f3eb8911700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f3eb40a9ad0 0x7f3eb411af60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:17.619 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.604+0000 7f3eb8911700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f3eb40a9ad0 0x7f3eb411af60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36070/0 (socket says 172.21.15.40:36070) 2019-12-05T05:03:17.619 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.604+0000 7f3eb8911700 1 -- 172.21.15.40:0/546711194 learned_addr learned my addr 172.21.15.40:0/546711194 (peer_addr_for_me v2:172.21.15.40:0/0) 2019-12-05T05:03:17.619 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.604+0000 7f3eb9913700 1 --2- 172.21.15.40:0/546711194 >> v2:172.21.15.42:3300/0 conn(0x7f3eb4118410 0x7f3eb411b4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:17.620 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.604+0000 7f3eb8911700 1 -- 172.21.15.40:0/546711194 >> v2:172.21.15.42:3300/0 conn(0x7f3eb4118410 msgr2=0x7f3eb411b4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:17.620 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.604+0000 7f3eb8911700 1 --2- 172.21.15.40:0/546711194 >> v2:172.21.15.42:3300/0 conn(0x7f3eb4118410 0x7f3eb411b4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:17.621 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.604+0000 7f3eb8911700 1 -- 172.21.15.40:0/546711194 >> v1:172.21.15.42:6789/0 conn(0x7f3eb4117a50 legacy=0x7f3eb4130be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:17.621 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.604+0000 7f3eb8911700 1 -- 172.21.15.40:0/546711194 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3eb41313e0 con 0x7f3eb40a9ad0 2019-12-05T05:03:17.621 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.604+0000 7f3eb8911700 1 --2- 172.21.15.40:0/546711194 >> v2:172.21.15.40:3300/0 conn(0x7f3eb40a9ad0 0x7f3eb411af60 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f3ea4008790 tx=0x7f3ea4011f10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:17.621 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.604+0000 7f3eb8911700 1 --2- 172.21.15.40:0/546711194 >> v2:172.21.15.40:3300/0 conn(0x7f3eb40a9ad0 0x7f3eb411af60 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f3ea4008790 tx=0x7f3ea4011f10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:17.621 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.604+0000 7f3eb8911700 1 --2- 172.21.15.40:0/546711194 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3eb40a9ad0 0x7f3eb411af60 secure :-1 s=READY pgs=22 cs=0 l=1 rx=0x7f3ea4008790 tx=0x7f3ea4011f10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:17.622 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.604+0000 7f3eb8911700 1 --2- 172.21.15.40:0/546711194 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3eb40a9ad0 0x7f3eb411af60 secure :-1 s=THROTTLE_DONE pgs=22 cs=0 l=1 rx=0x7f3ea4008790 tx=0x7f3ea4011f10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:17.622 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.604+0000 7f3ea9ffb700 1 -- 172.21.15.40:0/546711194 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3ea4007340 con 0x7f3eb40a9ad0 2019-12-05T05:03:17.622 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.604+0000 7f3eb8911700 1 --2- 172.21.15.40:0/546711194 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3eb40a9ad0 0x7f3eb411af60 secure :-1 s=THROTTLE_DONE pgs=22 cs=0 l=1 rx=0x7f3ea4008790 tx=0x7f3ea4011f10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:17.622 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.604+0000 7f3ea9ffb700 1 -- 172.21.15.40:0/546711194 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ea40114f0 con 0x7f3eb40a9ad0 2019-12-05T05:03:17.623 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.604+0000 7f3eb8911700 1 --2- 172.21.15.40:0/546711194 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3eb40a9ad0 0x7f3eb411af60 secure :-1 s=THROTTLE_DONE pgs=22 cs=0 l=1 rx=0x7f3ea4008790 tx=0x7f3ea4011f10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:17.623 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.604+0000 7f3ebab75700 1 -- 172.21.15.40:0/546711194 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3eb41379d0 con 0x7f3eb40a9ad0 2019-12-05T05:03:17.623 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.604+0000 7f3ea9ffb700 1 -- 172.21.15.40:0/546711194 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3ea4005540 con 0x7f3eb40a9ad0 2019-12-05T05:03:17.623 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.604+0000 7f3ebab75700 1 -- 172.21.15.40:0/546711194 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3eb4137f50 con 0x7f3eb40a9ad0 2019-12-05T05:03:17.623 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.604+0000 7f3eb8911700 1 --2- 172.21.15.40:0/546711194 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3eb40a9ad0 0x7f3eb411af60 secure :-1 s=THROTTLE_DONE pgs=22 cs=0 l=1 rx=0x7f3ea4008790 tx=0x7f3ea4011f10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:17.624 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.604+0000 7f3ea9ffb700 1 -- 172.21.15.40:0/546711194 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f3ea4005820 con 0x7f3eb40a9ad0 2019-12-05T05:03:17.624 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.604+0000 7f3eb8911700 1 --2- 172.21.15.40:0/546711194 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3eb40a9ad0 0x7f3eb411af60 secure :-1 s=THROTTLE_DONE pgs=22 cs=0 l=1 rx=0x7f3ea4008790 tx=0x7f3ea4011f10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:17.624 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.608+0000 7f3ea9ffb700 1 --2- 172.21.15.40:0/546711194 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3e9403ad40 0x7f3e9403d250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:17.624 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.608+0000 7f3ea9ffb700 1 -- 172.21.15.40:0/546711194 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1214+0+0 (secure 0 0 0) 0x7f3ea404b9e0 con 0x7f3eb40a9ad0 2019-12-05T05:03:17.625 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.608+0000 7f3eb9112700 1 --2- 172.21.15.40:0/546711194 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3e9403ad40 0x7f3e9403d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:17.625 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.608+0000 7f3eb9112700 1 --2- 172.21.15.40:0/546711194 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3e9403ad40 0x7f3e9403d250 crc :-1 s=READY pgs=10 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:17.625 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.612+0000 7f3ebab75700 1 -- 172.21.15.40:0/546711194 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3eb405d110 con 0x7f3eb40a9ad0 2019-12-05T05:03:17.626 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.616+0000 7f3eb8911700 1 --2- 172.21.15.40:0/546711194 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3eb40a9ad0 0x7f3eb411af60 secure :-1 s=THROTTLE_DONE pgs=22 cs=0 l=1 rx=0x7f3ea4008790 tx=0x7f3ea4011f10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:17.626 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.616+0000 7f3ea9ffb700 1 -- 172.21.15.40:0/546711194 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f3ea4032ca0 con 0x7f3eb40a9ad0 2019-12-05T05:03:17.803 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.800+0000 7f3ebab75700 1 -- 172.21.15.40:0/546711194 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd new", "uuid": "45bbf1bd-7c8a-4a0d-b76b-afa6148a855c", "id": 4} v 0) v1 -- 0x7f3eb405b8c0 con 0x7f3eb40a9ad0 2019-12-05T05:03:17.814 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.812+0000 7f3eb8911700 1 --2- 172.21.15.40:0/546711194 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3eb40a9ad0 0x7f3eb411af60 secure :-1 s=THROTTLE_DONE pgs=22 cs=0 l=1 rx=0x7f3ea4008790 tx=0x7f3ea4011f10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:17.814 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.812+0000 7f3ea9ffb700 1 -- 172.21.15.40:0/546711194 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "45bbf1bd-7c8a-4a0d-b76b-afa6148a855c", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7f3ea4033850 con 0x7f3eb40a9ad0 2019-12-05T05:03:17.818 INFO:teuthology.orchestra.run.smithi040.stdout:4 2019-12-05T05:03:17.822 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.820+0000 7f3ebab75700 1 -- 172.21.15.40:0/546711194 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3e9403ad40 msgr2=0x7f3e9403d250 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:17.822 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.820+0000 7f3ebab75700 1 --2- 172.21.15.40:0/546711194 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3e9403ad40 0x7f3e9403d250 crc :-1 s=READY pgs=10 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:17.823 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.820+0000 7f3ebab75700 1 -- 172.21.15.40:0/546711194 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3eb40a9ad0 msgr2=0x7f3eb411af60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:17.823 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.820+0000 7f3ebab75700 1 --2- 172.21.15.40:0/546711194 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3eb40a9ad0 0x7f3eb411af60 secure :-1 s=READY pgs=22 cs=0 l=1 rx=0x7f3ea4008790 tx=0x7f3ea4011f10).stop 2019-12-05T05:03:17.823 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.820+0000 7f3ebab75700 1 -- 172.21.15.40:0/546711194 shutdown_connections 2019-12-05T05:03:17.823 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.820+0000 7f3ebab75700 1 --2- 172.21.15.40:0/546711194 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3e9403ad40 0x7f3e9403d250 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:17.823 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.820+0000 7f3ebab75700 1 --2- 172.21.15.40:0/546711194 >> v2:172.21.15.42:3300/0 conn(0x7f3eb4118410 0x7f3eb411b4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:17.824 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.820+0000 7f3ebab75700 1 --2- 172.21.15.40:0/546711194 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3eb40a9ad0 0x7f3eb411af60 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:17.824 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.820+0000 7f3ebab75700 1 -- 172.21.15.40:0/546711194 shutdown_connections 2019-12-05T05:03:17.824 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.820+0000 7f3ebab75700 1 -- 172.21.15.40:0/546711194 wait complete. 2019-12-05T05:03:17.824 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.820+0000 7f3ebab75700 1 -- 172.21.15.40:0/546711194 >> 172.21.15.40:0/546711194 conn(0x7f3eb40a7740 msgr2=0x7f3eb4059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:17.834 INFO:teuthology.orchestra.run.smithi040:> sudo ceph --cluster ceph osd new 8a5511cd-925f-4869-a49e-924383e4ce53 5 2019-12-05T05:03:17.930 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.924+0000 7f4119244700 1 Processor -- start 2019-12-05T05:03:17.930 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.924+0000 7f4119244700 1 -- start start 2019-12-05T05:03:17.930 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.924+0000 7f4119244700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f41140a7740 0x7f411408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:17.930 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.924+0000 7f4119244700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41140a0710 con 0x7f411409fae0 2019-12-05T05:03:17.930 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.924+0000 7f4119244700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41140a9ba0 con 0x7f411407e720 2019-12-05T05:03:17.930 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.924+0000 7f4119244700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41140a0b00 con 0x7f41140a7740 2019-12-05T05:03:17.931 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.924+0000 7f411359e700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f41140a7740 0x7f411408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:17.931 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.924+0000 7f4118a43700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f411409fae0 0x7f411414efe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:59748/0 (socket says 172.21.15.40:59748) 2019-12-05T05:03:17.931 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.924+0000 7f4118a43700 1 -- 172.21.15.40:0/2650327509 learned_addr learned my addr 172.21.15.40:0/2650327509 (peer_addr_for_me v1:172.21.15.40:0/0) 2019-12-05T05:03:17.931 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.924+0000 7f411359e700 1 -- 172.21.15.40:0/2650327509 >> v1:172.21.15.42:6789/0 conn(0x7f411407e720 legacy=0x7f411409f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:17.931 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.924+0000 7f411359e700 1 -- 172.21.15.40:0/2650327509 >> v1:172.21.15.40:6789/0 conn(0x7f411409fae0 legacy=0x7f411414efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:17.931 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.924+0000 7f411359e700 1 -- 172.21.15.40:0/2650327509 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f411407eb50 con 0x7f41140a7740 2019-12-05T05:03:17.932 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.924+0000 7f411359e700 1 --2- 172.21.15.40:0/2650327509 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f41140a7740 0x7f411408df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f4104002f30 tx=0x7f410400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:17.932 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.924+0000 7f411359e700 1 --2- 172.21.15.40:0/2650327509 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f41140a7740 0x7f411408df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f4104002f30 tx=0x7f410400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:17.932 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.924+0000 7f411359e700 1 --2- 172.21.15.40:0/2650327509 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f41140a7740 0x7f411408df80 secure :-1 s=READY pgs=13 cs=0 l=1 rx=0x7f4104002f30 tx=0x7f410400fbf0).ready entity=mon.2 client_cookie=4921ea68fec2cef7 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:17.932 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.928+0000 7f411359e700 1 --2- 172.21.15.40:0/2650327509 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f41140a7740 0x7f411408df80 secure :-1 s=THROTTLE_DONE pgs=13 cs=0 l=1 rx=0x7f4104002f30 tx=0x7f410400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:17.933 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.928+0000 7f411259c700 1 -- 172.21.15.40:0/2650327509 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f41040112b0 con 0x7f41140a7740 2019-12-05T05:03:17.933 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.928+0000 7f411359e700 1 --2- 172.21.15.40:0/2650327509 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f41140a7740 0x7f411408df80 secure :-1 s=THROTTLE_DONE pgs=13 cs=0 l=1 rx=0x7f4104002f30 tx=0x7f410400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:17.933 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.928+0000 7f411259c700 1 -- 172.21.15.40:0/2650327509 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4104008bc0 con 0x7f41140a7740 2019-12-05T05:03:17.933 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.928+0000 7f411359e700 1 --2- 172.21.15.40:0/2650327509 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f41140a7740 0x7f411408df80 secure :-1 s=THROTTLE_DONE pgs=13 cs=0 l=1 rx=0x7f4104002f30 tx=0x7f410400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:17.934 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.928+0000 7f411259c700 1 -- 172.21.15.40:0/2650327509 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4104011890 con 0x7f41140a7740 2019-12-05T05:03:17.934 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.928+0000 7f4119244700 1 -- 172.21.15.40:0/2650327509 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f41140a7740 msgr2=0x7f411408df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:17.934 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.928+0000 7f4119244700 1 --2- 172.21.15.40:0/2650327509 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f41140a7740 0x7f411408df80 secure :-1 s=READY pgs=13 cs=0 l=1 rx=0x7f4104002f30 tx=0x7f410400fbf0).stop 2019-12-05T05:03:17.934 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.928+0000 7f4119244700 1 -- 172.21.15.40:0/2650327509 shutdown_connections 2019-12-05T05:03:17.934 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.928+0000 7f4119244700 1 --2- 172.21.15.40:0/2650327509 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f41140a7740 0x7f411408df80 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:17.935 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.928+0000 7f4119244700 1 -- 172.21.15.40:0/2650327509 shutdown_connections 2019-12-05T05:03:17.935 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.928+0000 7f4119244700 1 -- 172.21.15.40:0/2650327509 wait complete. 2019-12-05T05:03:17.935 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.928+0000 7f4119244700 1 -- 172.21.15.40:0/2650327509 >> 172.21.15.40:0/2650327509 conn(0x7f41140aabe0 msgr2=0x7f41140a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:17.939 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.928+0000 7f4119244700 1 Processor -- start 2019-12-05T05:03:17.940 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.928+0000 7f4119244700 1 -- start start 2019-12-05T05:03:17.940 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.932+0000 7f4119244700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f4114064370 0x7f41140647a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:17.940 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.932+0000 7f4119244700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4114063190 con 0x7f41140a1d70 2019-12-05T05:03:17.940 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.932+0000 7f4119244700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41140a2a70 con 0x7f4114144e60 2019-12-05T05:03:17.940 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.932+0000 7f4119244700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41141235a0 con 0x7f4114064370 2019-12-05T05:03:17.940 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.932+0000 7f4118a43700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f41140a1d70 0x7f41141305c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:59754/0 (socket says 172.21.15.40:59754) 2019-12-05T05:03:17.941 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.932+0000 7f4118a43700 1 -- 172.21.15.40:0/2170682134 learned_addr learned my addr 172.21.15.40:0/2170682134 (peer_addr_for_me v1:172.21.15.40:0/0) 2019-12-05T05:03:17.941 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.932+0000 7f4112d9d700 1 --2- 172.21.15.40:0/2170682134 >> v2:172.21.15.42:3300/0 conn(0x7f4114064370 0x7f41140647a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:17.941 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.932+0000 7f40fbfff700 1 -- 172.21.15.40:0/2170682134 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2798510503 0 0) 0x7f4114063190 con 0x7f41140a1d70 2019-12-05T05:03:17.941 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.932+0000 7f40fbfff700 1 -- 172.21.15.40:0/2170682134 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40f40030b0 con 0x7f41140a1d70 2019-12-05T05:03:17.941 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.932+0000 7f40fbfff700 1 -- 172.21.15.40:0/2170682134 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 3321814373 0 0) 0x7f40f40030b0 con 0x7f41140a1d70 2019-12-05T05:03:17.941 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.932+0000 7f40fbfff700 1 -- 172.21.15.40:0/2170682134 --> v1:172.21.15.40:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4114063190 con 0x7f41140a1d70 2019-12-05T05:03:17.941 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.932+0000 7f40fbfff700 1 -- 172.21.15.40:0/2170682134 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7f410c001ba0 con 0x7f41140a1d70 2019-12-05T05:03:17.942 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.932+0000 7f4112d9d700 1 -- 172.21.15.40:0/2170682134 >> v1:172.21.15.42:6789/0 conn(0x7f4114144e60 legacy=0x7f4114122da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:17.942 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.932+0000 7f4112d9d700 1 -- 172.21.15.40:0/2170682134 >> v1:172.21.15.40:6789/0 conn(0x7f41140a1d70 legacy=0x7f41141305c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:17.942 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.932+0000 7f4112d9d700 1 -- 172.21.15.40:0/2170682134 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41140a21a0 con 0x7f4114064370 2019-12-05T05:03:17.942 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.932+0000 7f4112d9d700 1 --2- 172.21.15.40:0/2170682134 >> v2:172.21.15.42:3300/0 conn(0x7f4114064370 0x7f41140647a0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f4108001430 tx=0x7f410c0018f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:17.942 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.932+0000 7f4112d9d700 1 --2- 172.21.15.40:0/2170682134 >> v2:172.21.15.42:3300/0 conn(0x7f4114064370 0x7f41140647a0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f4108001430 tx=0x7f410c0018f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:17.942 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.932+0000 7f4112d9d700 1 --2- 172.21.15.40:0/2170682134 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4114064370 0x7f41140647a0 secure :-1 s=READY pgs=21 cs=0 l=1 rx=0x7f4108001430 tx=0x7f410c0018f0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:17.942 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.932+0000 7f4112d9d700 1 --2- 172.21.15.40:0/2170682134 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4114064370 0x7f41140647a0 secure :-1 s=THROTTLE_DONE pgs=21 cs=0 l=1 rx=0x7f4108001430 tx=0x7f410c0018f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:17.943 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.932+0000 7f40fbfff700 1 -- 172.21.15.40:0/2170682134 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f410800c960 con 0x7f4114064370 2019-12-05T05:03:17.943 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.932+0000 7f4112d9d700 1 --2- 172.21.15.40:0/2170682134 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4114064370 0x7f41140647a0 secure :-1 s=THROTTLE_DONE pgs=21 cs=0 l=1 rx=0x7f4108001430 tx=0x7f410c0018f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:17.943 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.932+0000 7f4112d9d700 1 --2- 172.21.15.40:0/2170682134 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4114064370 0x7f41140647a0 secure :-1 s=THROTTLE_DONE pgs=21 cs=0 l=1 rx=0x7f4108001430 tx=0x7f410c0018f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:17.943 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.932+0000 7f40fbfff700 1 -- 172.21.15.40:0/2170682134 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f410800c580 con 0x7f4114064370 2019-12-05T05:03:17.943 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.932+0000 7f40fbfff700 1 -- 172.21.15.40:0/2170682134 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4108010d00 con 0x7f4114064370 2019-12-05T05:03:17.943 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.932+0000 7f4119244700 1 -- 172.21.15.40:0/2170682134 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41140a23b0 con 0x7f4114064370 2019-12-05T05:03:17.943 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.932+0000 7f4119244700 1 -- 172.21.15.40:0/2170682134 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4114131dd0 con 0x7f4114064370 2019-12-05T05:03:17.944 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.932+0000 7f4112d9d700 1 --2- 172.21.15.40:0/2170682134 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4114064370 0x7f41140647a0 secure :-1 s=THROTTLE_DONE pgs=21 cs=0 l=1 rx=0x7f4108001430 tx=0x7f410c0018f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:17.944 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.936+0000 7f40fbfff700 1 -- 172.21.15.40:0/2170682134 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f4108007430 con 0x7f4114064370 2019-12-05T05:03:17.944 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.936+0000 7f4112d9d700 1 --2- 172.21.15.40:0/2170682134 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4114064370 0x7f41140647a0 secure :-1 s=THROTTLE_DONE pgs=21 cs=0 l=1 rx=0x7f4108001430 tx=0x7f410c0018f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:17.944 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.936+0000 7f40fbfff700 1 --2- 172.21.15.40:0/2170682134 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f40f403b650 0x7f40f403db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:17.944 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.936+0000 7f40fbfff700 1 -- 172.21.15.40:0/2170682134 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1329+0+0 (secure 0 0 0) 0x7f410804af60 con 0x7f4114064370 2019-12-05T05:03:17.944 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.936+0000 7f411359e700 1 --2- 172.21.15.40:0/2170682134 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f40f403b650 0x7f40f403db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:17.945 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.936+0000 7f411359e700 1 --2- 172.21.15.40:0/2170682134 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f40f403b650 0x7f40f403db60 crc :-1 s=READY pgs=11 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:17.945 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.940+0000 7f4119244700 1 -- 172.21.15.40:0/2170682134 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f411405d110 con 0x7f4114064370 2019-12-05T05:03:17.953 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.952+0000 7f4112d9d700 1 --2- 172.21.15.40:0/2170682134 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4114064370 0x7f41140647a0 secure :-1 s=THROTTLE_DONE pgs=21 cs=0 l=1 rx=0x7f4108001430 tx=0x7f410c0018f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:17.953 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:17.952+0000 7f40fbfff700 1 -- 172.21.15.40:0/2170682134 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f4108031fa0 con 0x7f4114064370 2019-12-05T05:03:18.129 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.128+0000 7f4119244700 1 -- 172.21.15.40:0/2170682134 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "osd new", "uuid": "8a5511cd-925f-4869-a49e-924383e4ce53", "id": 5} v 0) v1 -- 0x7f411405b8c0 con 0x7f4114064370 2019-12-05T05:03:18.140 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.140+0000 7f4112d9d700 1 --2- 172.21.15.40:0/2170682134 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4114064370 0x7f41140647a0 secure :-1 s=THROTTLE_DONE pgs=21 cs=0 l=1 rx=0x7f4108001430 tx=0x7f410c0018f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:18.141 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.140+0000 7f40fbfff700 1 -- 172.21.15.40:0/2170682134 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "8a5511cd-925f-4869-a49e-924383e4ce53", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7f4108032200 con 0x7f4114064370 2019-12-05T05:03:18.143 INFO:teuthology.orchestra.run.smithi040.stdout:5 2019-12-05T05:03:18.147 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.144+0000 7f4119244700 1 -- 172.21.15.40:0/2170682134 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f40f403b650 msgr2=0x7f40f403db60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:18.147 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.144+0000 7f4119244700 1 --2- 172.21.15.40:0/2170682134 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f40f403b650 0x7f40f403db60 crc :-1 s=READY pgs=11 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:18.147 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.144+0000 7f4119244700 1 -- 172.21.15.40:0/2170682134 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4114064370 msgr2=0x7f41140647a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:18.148 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.144+0000 7f4119244700 1 --2- 172.21.15.40:0/2170682134 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4114064370 0x7f41140647a0 secure :-1 s=READY pgs=21 cs=0 l=1 rx=0x7f4108001430 tx=0x7f410c0018f0).stop 2019-12-05T05:03:18.148 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.144+0000 7f4119244700 1 -- 172.21.15.40:0/2170682134 shutdown_connections 2019-12-05T05:03:18.148 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.144+0000 7f4119244700 1 --2- 172.21.15.40:0/2170682134 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f40f403b650 0x7f40f403db60 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:18.148 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.144+0000 7f4119244700 1 --2- 172.21.15.40:0/2170682134 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4114064370 0x7f41140647a0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:18.148 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.144+0000 7f4119244700 1 -- 172.21.15.40:0/2170682134 shutdown_connections 2019-12-05T05:03:18.148 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.144+0000 7f4119244700 1 -- 172.21.15.40:0/2170682134 wait complete. 2019-12-05T05:03:18.149 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.144+0000 7f4119244700 1 -- 172.21.15.40:0/2170682134 >> 172.21.15.40:0/2170682134 conn(0x7f411409ffb0 msgr2=0x7f41140a6bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:18.157 INFO:teuthology.orchestra.run.smithi040:> sudo ceph --cluster ceph osd new d692c179-379e-446c-8484-a219a9241e04 6 2019-12-05T05:03:18.257 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.248+0000 7ff0c98a0700 1 Processor -- start 2019-12-05T05:03:18.258 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.248+0000 7ff0c98a0700 1 -- start start 2019-12-05T05:03:18.258 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.248+0000 7ff0c98a0700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff0c40a7740 0x7ff0c408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:18.258 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.248+0000 7ff0c98a0700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff0c409fae0 0x7ff0c407eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:18.258 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.248+0000 7ff0c98a0700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0c40a0710 con 0x7ff0c407e720 2019-12-05T05:03:18.259 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.248+0000 7ff0c98a0700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0c40a0b00 con 0x7ff0c409fae0 2019-12-05T05:03:18.259 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.248+0000 7ff0c98a0700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0c408e5b0 con 0x7ff0c40a7740 2019-12-05T05:03:18.259 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.248+0000 7ff0c3fff700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff0c409fae0 0x7ff0c407eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:18.259 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.248+0000 7ff0c37fe700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff0c40a7740 0x7ff0c408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:18.259 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.248+0000 7ff0c3fff700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff0c409fae0 0x7ff0c407eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36096/0 (socket says 172.21.15.40:36096) 2019-12-05T05:03:18.260 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.248+0000 7ff0c37fe700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff0c40a7740 0x7ff0c408df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:33156/0 (socket says 172.21.15.40:33156) 2019-12-05T05:03:18.260 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.248+0000 7ff0c3fff700 1 -- 172.21.15.40:0/718478650 learned_addr learned my addr 172.21.15.40:0/718478650 (peer_addr_for_me v2:172.21.15.40:0/0) 2019-12-05T05:03:18.260 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.252+0000 7ff0c3fff700 1 -- 172.21.15.40:0/718478650 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff0c40a7740 msgr2=0x7ff0c408df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:03:18.260 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.252+0000 7ff0c3fff700 1 --2- 172.21.15.40:0/718478650 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff0c40a7740 0x7ff0c408df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:03:18.260 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.252+0000 7ff0c3fff700 1 -- 172.21.15.40:0/718478650 >> v1:172.21.15.40:6789/0 conn(0x7ff0c407e720 legacy=0x7ff0c409f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:18.261 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.252+0000 7ff0c3fff700 1 -- 172.21.15.40:0/718478650 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0c407f0a0 con 0x7ff0c409fae0 2019-12-05T05:03:18.261 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.252+0000 7ff0c3fff700 1 --2- 172.21.15.40:0/718478650 >> v2:172.21.15.40:3300/0 conn(0x7ff0c409fae0 0x7ff0c407eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7ff0bc008b10 tx=0x7ff0bc0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:18.261 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.252+0000 7ff0c3fff700 1 --2- 172.21.15.40:0/718478650 >> v2:172.21.15.40:3300/0 conn(0x7ff0c409fae0 0x7ff0c407eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7ff0bc008b10 tx=0x7ff0bc0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:18.261 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.252+0000 7ff0c3fff700 1 --2- 172.21.15.40:0/718478650 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff0c409fae0 0x7ff0c407eb50 secure :-1 s=READY pgs=27 cs=0 l=1 rx=0x7ff0bc008b10 tx=0x7ff0bc0100b0).ready entity=mon.0 client_cookie=4528394792f4c83 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:18.264 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.252+0000 7ff0c3fff700 1 --2- 172.21.15.40:0/718478650 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff0c409fae0 0x7ff0c407eb50 secure :-1 s=THROTTLE_DONE pgs=27 cs=0 l=1 rx=0x7ff0bc008b10 tx=0x7ff0bc0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:18.264 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.252+0000 7ff0c2ffd700 1 -- 172.21.15.40:0/718478650 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff0bc010da0 con 0x7ff0c409fae0 2019-12-05T05:03:18.265 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.252+0000 7ff0c3fff700 1 --2- 172.21.15.40:0/718478650 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff0c409fae0 0x7ff0c407eb50 secure :-1 s=THROTTLE_DONE pgs=27 cs=0 l=1 rx=0x7ff0bc008b10 tx=0x7ff0bc0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:18.265 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.252+0000 7ff0c3fff700 1 --2- 172.21.15.40:0/718478650 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff0c409fae0 0x7ff0c407eb50 secure :-1 s=THROTTLE_DONE pgs=27 cs=0 l=1 rx=0x7ff0bc008b10 tx=0x7ff0bc0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:18.265 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.252+0000 7ff0c2ffd700 1 -- 172.21.15.40:0/718478650 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0bc0109c0 con 0x7ff0c409fae0 2019-12-05T05:03:18.265 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.252+0000 7ff0c2ffd700 1 -- 172.21.15.40:0/718478650 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff0bc0114a0 con 0x7ff0c409fae0 2019-12-05T05:03:18.265 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.252+0000 7ff0c98a0700 1 -- 172.21.15.40:0/718478650 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff0c409fae0 msgr2=0x7ff0c407eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:18.266 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.252+0000 7ff0c98a0700 1 --2- 172.21.15.40:0/718478650 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff0c409fae0 0x7ff0c407eb50 secure :-1 s=READY pgs=27 cs=0 l=1 rx=0x7ff0bc008b10 tx=0x7ff0bc0100b0).stop 2019-12-05T05:03:18.266 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.252+0000 7ff0c98a0700 1 -- 172.21.15.40:0/718478650 shutdown_connections 2019-12-05T05:03:18.266 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.252+0000 7ff0c98a0700 1 --2- 172.21.15.40:0/718478650 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff0c409fae0 0x7ff0c407eb50 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:18.266 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.252+0000 7ff0c98a0700 1 --2- 172.21.15.40:0/718478650 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff0c40a7740 0x7ff0c408df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:03:18.266 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.252+0000 7ff0c98a0700 1 -- 172.21.15.40:0/718478650 shutdown_connections 2019-12-05T05:03:18.266 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.252+0000 7ff0c98a0700 1 -- 172.21.15.40:0/718478650 wait complete. 2019-12-05T05:03:18.267 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.252+0000 7ff0c98a0700 1 -- 172.21.15.40:0/718478650 >> 172.21.15.40:0/718478650 conn(0x7ff0c40aabe0 msgr2=0x7ff0c40a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:18.267 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.256+0000 7ff0c98a0700 1 Processor -- start 2019-12-05T05:03:18.267 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.256+0000 7ff0c98a0700 1 -- start start 2019-12-05T05:03:18.272 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.256+0000 7ff0c98a0700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff0c4146180 0x7ff0c412d000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:18.272 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.256+0000 7ff0c98a0700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff0c412d550 0x7ff0c4137a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:18.272 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.256+0000 7ff0c98a0700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0c4121ce0 con 0x7ff0c40a1fa0 2019-12-05T05:03:18.272 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.256+0000 7ff0c98a0700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0c40a3190 con 0x7ff0c4146180 2019-12-05T05:03:18.273 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.256+0000 7ff0c98a0700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0c40a3430 con 0x7ff0c412d550 2019-12-05T05:03:18.273 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.256+0000 7ff0c37fe700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7ff0c40a1fa0 0x7ff0c41214e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:59764/0 (socket says 172.21.15.40:59764) 2019-12-05T05:03:18.273 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.256+0000 7ff0c37fe700 1 -- 172.21.15.40:0/2673677279 learned_addr learned my addr 172.21.15.40:0/2673677279 (peer_addr_for_me v1:172.21.15.40:0/0) 2019-12-05T05:03:18.273 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.256+0000 7ff0c3fff700 1 --2- 172.21.15.40:0/2673677279 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff0c412d550 0x7ff0c4137a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:18.273 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.256+0000 7ff0baffd700 1 --2- 172.21.15.40:0/2673677279 >> v2:172.21.15.40:3300/0 conn(0x7ff0c4146180 0x7ff0c412d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:18.281 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.256+0000 7ff0c0ff9700 1 -- 172.21.15.40:0/2673677279 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2516142705 0 0) 0x7ff0c4121ce0 con 0x7ff0c40a1fa0 2019-12-05T05:03:18.281 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.256+0000 7ff0c0ff9700 1 -- 172.21.15.40:0/2673677279 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0a80030b0 con 0x7ff0c40a1fa0 2019-12-05T05:03:18.281 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.256+0000 7ff0c3fff700 1 -- 172.21.15.40:0/2673677279 >> v2:172.21.15.40:3300/0 conn(0x7ff0c4146180 msgr2=0x7ff0c412d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:18.281 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.256+0000 7ff0c3fff700 1 --2- 172.21.15.40:0/2673677279 >> v2:172.21.15.40:3300/0 conn(0x7ff0c4146180 0x7ff0c412d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:18.282 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.256+0000 7ff0c3fff700 1 -- 172.21.15.40:0/2673677279 >> v1:172.21.15.40:6789/0 conn(0x7ff0c40a1fa0 legacy=0x7ff0c41214e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:18.282 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.256+0000 7ff0c3fff700 1 -- 172.21.15.40:0/2673677279 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0c40a2580 con 0x7ff0c412d550 2019-12-05T05:03:18.282 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.256+0000 7ff0c3fff700 1 --2- 172.21.15.40:0/2673677279 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff0c412d550 0x7ff0c4137a10 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7ff0c41221b0 tx=0x7ff0bc010cf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:18.282 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.256+0000 7ff0c3fff700 1 --2- 172.21.15.40:0/2673677279 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff0c412d550 0x7ff0c4137a10 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7ff0c41221b0 tx=0x7ff0bc010cf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:18.282 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.256+0000 7ff0c3fff700 1 --2- 172.21.15.40:0/2673677279 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff0c412d550 0x7ff0c4137a10 secure :-1 s=READY pgs=14 cs=0 l=1 rx=0x7ff0c41221b0 tx=0x7ff0bc010cf0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:18.283 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.260+0000 7ff0c3fff700 1 --2- 172.21.15.40:0/2673677279 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff0c412d550 0x7ff0c4137a10 secure :-1 s=THROTTLE_DONE pgs=14 cs=0 l=1 rx=0x7ff0c41221b0 tx=0x7ff0bc010cf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:18.283 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.260+0000 7ff0c0ff9700 1 -- 172.21.15.40:0/2673677279 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff0bc011bb0 con 0x7ff0c412d550 2019-12-05T05:03:18.283 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.260+0000 7ff0c3fff700 1 --2- 172.21.15.40:0/2673677279 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff0c412d550 0x7ff0c4137a10 secure :-1 s=THROTTLE_DONE pgs=14 cs=0 l=1 rx=0x7ff0c41221b0 tx=0x7ff0bc010cf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:18.283 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.260+0000 7ff0c3fff700 1 --2- 172.21.15.40:0/2673677279 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff0c412d550 0x7ff0c4137a10 secure :-1 s=THROTTLE_DONE pgs=14 cs=0 l=1 rx=0x7ff0c41221b0 tx=0x7ff0bc010cf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:18.284 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.260+0000 7ff0c0ff9700 1 -- 172.21.15.40:0/2673677279 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0bc0110c0 con 0x7ff0c412d550 2019-12-05T05:03:18.284 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.260+0000 7ff0c98a0700 1 -- 172.21.15.40:0/2673677279 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0c40a2e10 con 0x7ff0c412d550 2019-12-05T05:03:18.284 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.260+0000 7ff0c0ff9700 1 -- 172.21.15.40:0/2673677279 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff0bc005ab0 con 0x7ff0c412d550 2019-12-05T05:03:18.284 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.260+0000 7ff0c98a0700 1 -- 172.21.15.40:0/2673677279 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0c41465b0 con 0x7ff0c412d550 2019-12-05T05:03:18.285 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.260+0000 7ff0c3fff700 1 --2- 172.21.15.40:0/2673677279 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff0c412d550 0x7ff0c4137a10 secure :-1 s=THROTTLE_DONE pgs=14 cs=0 l=1 rx=0x7ff0c41221b0 tx=0x7ff0bc010cf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:18.285 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.260+0000 7ff0c0ff9700 1 -- 172.21.15.40:0/2673677279 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7ff0bc00f280 con 0x7ff0c412d550 2019-12-05T05:03:18.285 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.260+0000 7ff0c3fff700 1 --2- 172.21.15.40:0/2673677279 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff0c412d550 0x7ff0c4137a10 secure :-1 s=THROTTLE_DONE pgs=14 cs=0 l=1 rx=0x7ff0c41221b0 tx=0x7ff0bc010cf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:18.285 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.260+0000 7ff0c0ff9700 1 --2- 172.21.15.40:0/2673677279 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff0a803b2f0 0x7ff0a803d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:18.286 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.260+0000 7ff0c0ff9700 1 -- 172.21.15.40:0/2673677279 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1444+0+0 (secure 0 0 0) 0x7ff0bc04b470 con 0x7ff0c412d550 2019-12-05T05:03:18.286 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.260+0000 7ff0c37fe700 1 --2- 172.21.15.40:0/2673677279 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff0a803b2f0 0x7ff0a803d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:18.286 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.260+0000 7ff0c37fe700 1 --2- 172.21.15.40:0/2673677279 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff0a803b2f0 0x7ff0a803d800 crc :-1 s=READY pgs=12 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:18.286 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.264+0000 7ff0c98a0700 1 -- 172.21.15.40:0/2673677279 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0c405d110 con 0x7ff0c412d550 2019-12-05T05:03:18.287 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.272+0000 7ff0c3fff700 1 --2- 172.21.15.40:0/2673677279 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff0c412d550 0x7ff0c4137a10 secure :-1 s=THROTTLE_DONE pgs=14 cs=0 l=1 rx=0x7ff0c41221b0 tx=0x7ff0bc010cf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:18.288 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.272+0000 7ff0c0ff9700 1 -- 172.21.15.40:0/2673677279 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7ff0c40a2580 con 0x7ff0c412d550 2019-12-05T05:03:18.454 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.452+0000 7ff0c98a0700 1 -- 172.21.15.40:0/2673677279 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "osd new", "uuid": "d692c179-379e-446c-8484-a219a9241e04", "id": 6} v 0) v1 -- 0x7ff0c405b8c0 con 0x7ff0c412d550 2019-12-05T05:03:18.461 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.460+0000 7ff0c3fff700 1 --2- 172.21.15.40:0/2673677279 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff0c412d550 0x7ff0c4137a10 secure :-1 s=THROTTLE_DONE pgs=14 cs=0 l=1 rx=0x7ff0c41221b0 tx=0x7ff0bc010cf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:18.461 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.460+0000 7ff0c0ff9700 1 -- 172.21.15.40:0/2673677279 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "d692c179-379e-446c-8484-a219a9241e04", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7ff0bc032750 con 0x7ff0c412d550 2019-12-05T05:03:18.469 INFO:teuthology.orchestra.run.smithi040.stdout:6 2019-12-05T05:03:18.474 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.472+0000 7ff0c98a0700 1 -- 172.21.15.40:0/2673677279 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff0a803b2f0 msgr2=0x7ff0a803d800 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:18.474 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.472+0000 7ff0c98a0700 1 --2- 172.21.15.40:0/2673677279 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff0a803b2f0 0x7ff0a803d800 crc :-1 s=READY pgs=12 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:18.474 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.472+0000 7ff0c98a0700 1 -- 172.21.15.40:0/2673677279 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff0c412d550 msgr2=0x7ff0c4137a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:18.475 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.472+0000 7ff0c98a0700 1 --2- 172.21.15.40:0/2673677279 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff0c412d550 0x7ff0c4137a10 secure :-1 s=READY pgs=14 cs=0 l=1 rx=0x7ff0c41221b0 tx=0x7ff0bc010cf0).stop 2019-12-05T05:03:18.475 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.472+0000 7ff0c98a0700 1 -- 172.21.15.40:0/2673677279 shutdown_connections 2019-12-05T05:03:18.475 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.472+0000 7ff0c98a0700 1 --2- 172.21.15.40:0/2673677279 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff0a803b2f0 0x7ff0a803d800 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:18.475 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.472+0000 7ff0c98a0700 1 --2- 172.21.15.40:0/2673677279 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff0c412d550 0x7ff0c4137a10 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:18.476 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.472+0000 7ff0c98a0700 1 --2- 172.21.15.40:0/2673677279 >> v2:172.21.15.40:3300/0 conn(0x7ff0c4146180 0x7ff0c412d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:18.476 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.472+0000 7ff0c98a0700 1 -- 172.21.15.40:0/2673677279 shutdown_connections 2019-12-05T05:03:18.476 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.472+0000 7ff0c98a0700 1 -- 172.21.15.40:0/2673677279 wait complete. 2019-12-05T05:03:18.476 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.472+0000 7ff0c98a0700 1 -- 172.21.15.40:0/2673677279 >> 172.21.15.40:0/2673677279 conn(0x7ff0c40a82a0 msgr2=0x7ff0c4059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:18.485 INFO:teuthology.orchestra.run.smithi040:> sudo ceph --cluster ceph osd new 79c3c243-190e-4e35-b769-319a2077e7cd 7 2019-12-05T05:03:18.581 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.572+0000 7f5f113dd700 1 Processor -- start 2019-12-05T05:03:18.581 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.572+0000 7f5f113dd700 1 -- start start 2019-12-05T05:03:18.581 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.572+0000 7f5f113dd700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5f0c08df80 0x7f5f0c09f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:18.581 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.572+0000 7f5f113dd700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5f0c09f830 0x7f5f0c14efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:18.582 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.572+0000 7f5f113dd700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f0c09fc60 con 0x7f5f0c0a7740 2019-12-05T05:03:18.582 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.572+0000 7f5f113dd700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f0c0a0b00 con 0x7f5f0c09f830 2019-12-05T05:03:18.582 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.572+0000 7f5f113dd700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f0c08e5b0 con 0x7f5f0c08df80 2019-12-05T05:03:18.582 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.572+0000 7f5f0affd700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5f0c08df80 0x7f5f0c09f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:18.582 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.572+0000 7f5f0bfff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5f0c09f830 0x7f5f0c14efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:18.583 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.572+0000 7f5f0bfff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5f0c09f830 0x7f5f0c14efe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.40:53372/0 (socket says 172.21.15.40:53372) 2019-12-05T05:03:18.583 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.572+0000 7f5f0affd700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5f0c08df80 0x7f5f0c09f2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:33172/0 (socket says 172.21.15.40:33172) 2019-12-05T05:03:18.583 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.572+0000 7f5f0b7fe700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f5f0c0a7740 0x7f5f0c07e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.40:51318/0 (socket says 172.21.15.40:51318) 2019-12-05T05:03:18.583 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.572+0000 7f5f0bfff700 1 -- 172.21.15.40:0/3286940669 learned_addr learned my addr 172.21.15.40:0/3286940669 (peer_addr_for_me v2:172.21.15.40:0/0) 2019-12-05T05:03:18.583 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.576+0000 7f5f0affd700 1 -- 172.21.15.40:0/3286940669 >> v2:172.21.15.42:3300/0 conn(0x7f5f0c09f830 msgr2=0x7f5f0c14efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:03:18.583 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.576+0000 7f5f0affd700 1 --2- 172.21.15.40:0/3286940669 >> v2:172.21.15.42:3300/0 conn(0x7f5f0c09f830 0x7f5f0c14efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:03:18.584 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.576+0000 7f5f0affd700 1 -- 172.21.15.40:0/3286940669 >> v1:172.21.15.42:6789/0 conn(0x7f5f0c0a7740 legacy=0x7f5f0c07e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:18.584 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.576+0000 7f5f0affd700 1 -- 172.21.15.40:0/3286940669 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f0c0a0710 con 0x7f5f0c08df80 2019-12-05T05:03:18.584 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.576+0000 7f5f0bfff700 1 --2- 172.21.15.40:0/3286940669 >> v2:172.21.15.42:3300/0 conn(0x7f5f0c09f830 0x7f5f0c14efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:03:18.584 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.576+0000 7f5f0affd700 1 --2- 172.21.15.40:0/3286940669 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5f0c08df80 0x7f5f0c09f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f5f00008790 tx=0x7f5f0000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:18.584 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.576+0000 7f5f0affd700 1 --2- 172.21.15.40:0/3286940669 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5f0c08df80 0x7f5f0c09f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f5f00008790 tx=0x7f5f0000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:18.585 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.576+0000 7f5f0affd700 1 --2- 172.21.15.40:0/3286940669 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5f0c08df80 0x7f5f0c09f2e0 secure :-1 s=READY pgs=15 cs=0 l=1 rx=0x7f5f00008790 tx=0x7f5f0000fc70).ready entity=mon.2 client_cookie=f94c07eb780b9e6d server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:18.585 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.576+0000 7f5f0affd700 1 --2- 172.21.15.40:0/3286940669 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5f0c08df80 0x7f5f0c09f2e0 secure :-1 s=THROTTLE_DONE pgs=15 cs=0 l=1 rx=0x7f5f00008790 tx=0x7f5f0000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:18.585 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.576+0000 7f5f0affd700 1 --2- 172.21.15.40:0/3286940669 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5f0c08df80 0x7f5f0c09f2e0 secure :-1 s=THROTTLE_DONE pgs=15 cs=0 l=1 rx=0x7f5f00008790 tx=0x7f5f0000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:18.585 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.576+0000 7f5f0affd700 1 --2- 172.21.15.40:0/3286940669 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5f0c08df80 0x7f5f0c09f2e0 secure :-1 s=THROTTLE_DONE pgs=15 cs=0 l=1 rx=0x7f5f00008790 tx=0x7f5f0000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:18.585 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.576+0000 7f5f0a7fc700 1 -- 172.21.15.40:0/3286940669 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5f00008bb0 con 0x7f5f0c08df80 2019-12-05T05:03:18.586 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.576+0000 7f5f0a7fc700 1 -- 172.21.15.40:0/3286940669 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f000087d0 con 0x7f5f0c08df80 2019-12-05T05:03:18.586 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.576+0000 7f5f0a7fc700 1 -- 172.21.15.40:0/3286940669 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5f00011860 con 0x7f5f0c08df80 2019-12-05T05:03:18.586 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.576+0000 7f5f113dd700 1 -- 172.21.15.40:0/3286940669 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5f0c08df80 msgr2=0x7f5f0c09f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:18.586 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.576+0000 7f5f113dd700 1 --2- 172.21.15.40:0/3286940669 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5f0c08df80 0x7f5f0c09f2e0 secure :-1 s=READY pgs=15 cs=0 l=1 rx=0x7f5f00008790 tx=0x7f5f0000fc70).stop 2019-12-05T05:03:18.586 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.576+0000 7f5f113dd700 1 -- 172.21.15.40:0/3286940669 shutdown_connections 2019-12-05T05:03:18.587 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.576+0000 7f5f113dd700 1 --2- 172.21.15.40:0/3286940669 >> v2:172.21.15.42:3300/0 conn(0x7f5f0c09f830 0x7f5f0c14efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:03:18.587 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.576+0000 7f5f113dd700 1 --2- 172.21.15.40:0/3286940669 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5f0c08df80 0x7f5f0c09f2e0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:18.587 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.576+0000 7f5f113dd700 1 -- 172.21.15.40:0/3286940669 shutdown_connections 2019-12-05T05:03:18.587 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.576+0000 7f5f113dd700 1 -- 172.21.15.40:0/3286940669 wait complete. 2019-12-05T05:03:18.587 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.576+0000 7f5f113dd700 1 -- 172.21.15.40:0/3286940669 >> 172.21.15.40:0/3286940669 conn(0x7f5f0c0aabe0 msgr2=0x7f5f0c0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:18.588 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.580+0000 7f5f113dd700 1 Processor -- start 2019-12-05T05:03:18.588 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.580+0000 7f5f113dd700 1 -- start start 2019-12-05T05:03:18.593 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.580+0000 7f5f113dd700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5f0c11ab60 0x7f5f0c138a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:18.593 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.580+0000 7f5f113dd700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f0c11af90 con 0x7f5f0c11bc90 2019-12-05T05:03:18.593 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.580+0000 7f5f113dd700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f0c07b6d0 con 0x7f5f0c124b50 2019-12-05T05:03:18.593 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.580+0000 7f5f113dd700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f0c11b750 con 0x7f5f0c11ab60 2019-12-05T05:03:18.594 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.580+0000 7f5f0bfff700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5f0c11ab60 0x7f5f0c138a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:18.594 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.580+0000 7f5f0affd700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f5f0c11bc90 0x7f5f0c07aed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:59780/0 (socket says 172.21.15.40:59780) 2019-12-05T05:03:18.594 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.580+0000 7f5f0b7fe700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f5f0c124b50 0x7f5f0c07c070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.40:51324/0 (socket says 172.21.15.40:51324) 2019-12-05T05:03:18.594 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.580+0000 7f5f0affd700 1 -- 172.21.15.40:0/4180003501 learned_addr learned my addr 172.21.15.40:0/4180003501 (peer_addr_for_me v1:172.21.15.40:0/0) 2019-12-05T05:03:18.595 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.580+0000 7f5ef3fff700 1 -- 172.21.15.40:0/4180003501 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1314345737 0 0) 0x7f5f0c07b6d0 con 0x7f5f0c124b50 2019-12-05T05:03:18.595 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.580+0000 7f5ef3fff700 1 -- 172.21.15.40:0/4180003501 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5eec0030b0 con 0x7f5f0c124b50 2019-12-05T05:03:18.595 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.580+0000 7f5ef3fff700 1 -- 172.21.15.40:0/4180003501 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2741095298 0 0) 0x7f5f00008470 con 0x7f5f0c11bc90 2019-12-05T05:03:18.595 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.580+0000 7f5f0bfff700 1 -- 172.21.15.40:0/4180003501 >> v1:172.21.15.42:6789/0 conn(0x7f5f0c124b50 legacy=0x7f5f0c07c070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:18.595 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.580+0000 7f5f0bfff700 1 -- 172.21.15.40:0/4180003501 >> v1:172.21.15.40:6789/0 conn(0x7f5f0c11bc90 legacy=0x7f5f0c07aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:18.596 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.580+0000 7f5f0bfff700 1 -- 172.21.15.40:0/4180003501 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f0c07cd20 con 0x7f5f0c11ab60 2019-12-05T05:03:18.596 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.580+0000 7f5f0bfff700 1 --2- 172.21.15.40:0/4180003501 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5f0c11ab60 0x7f5f0c138a30 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f5f040032b0 tx=0x7f5f04010ea0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:18.596 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.580+0000 7f5f0bfff700 1 --2- 172.21.15.40:0/4180003501 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5f0c11ab60 0x7f5f0c138a30 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f5f040032b0 tx=0x7f5f04010ea0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:18.596 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.580+0000 7f5f0bfff700 1 --2- 172.21.15.40:0/4180003501 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5f0c11ab60 0x7f5f0c138a30 secure :-1 s=READY pgs=16 cs=0 l=1 rx=0x7f5f040032b0 tx=0x7f5f04010ea0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:18.597 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.584+0000 7f5f0bfff700 1 --2- 172.21.15.40:0/4180003501 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5f0c11ab60 0x7f5f0c138a30 secure :-1 s=THROTTLE_DONE pgs=16 cs=0 l=1 rx=0x7f5f040032b0 tx=0x7f5f04010ea0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:18.597 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.584+0000 7f5f0bfff700 1 --2- 172.21.15.40:0/4180003501 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5f0c11ab60 0x7f5f0c138a30 secure :-1 s=THROTTLE_DONE pgs=16 cs=0 l=1 rx=0x7f5f040032b0 tx=0x7f5f04010ea0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:18.597 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.584+0000 7f5ef3fff700 1 -- 172.21.15.40:0/4180003501 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5f04007160 con 0x7f5f0c11ab60 2019-12-05T05:03:18.597 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.584+0000 7f5f0bfff700 1 --2- 172.21.15.40:0/4180003501 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5f0c11ab60 0x7f5f0c138a30 secure :-1 s=THROTTLE_DONE pgs=16 cs=0 l=1 rx=0x7f5f040032b0 tx=0x7f5f04010ea0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:18.598 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.584+0000 7f5ef3fff700 1 -- 172.21.15.40:0/4180003501 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f04005bc0 con 0x7f5f0c11ab60 2019-12-05T05:03:18.598 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.584+0000 7f5ef3fff700 1 -- 172.21.15.40:0/4180003501 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5f04007740 con 0x7f5f0c11ab60 2019-12-05T05:03:18.598 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.584+0000 7f5f113dd700 1 -- 172.21.15.40:0/4180003501 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f0c138f80 con 0x7f5f0c11ab60 2019-12-05T05:03:18.598 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.584+0000 7f5f113dd700 1 -- 172.21.15.40:0/4180003501 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f0c139480 con 0x7f5f0c11ab60 2019-12-05T05:03:18.599 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.584+0000 7f5f0bfff700 1 --2- 172.21.15.40:0/4180003501 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5f0c11ab60 0x7f5f0c138a30 secure :-1 s=THROTTLE_DONE pgs=16 cs=0 l=1 rx=0x7f5f040032b0 tx=0x7f5f04010ea0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:18.599 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.584+0000 7f5ef3fff700 1 -- 172.21.15.40:0/4180003501 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f5f040125f0 con 0x7f5f0c11ab60 2019-12-05T05:03:18.599 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.584+0000 7f5f0bfff700 1 --2- 172.21.15.40:0/4180003501 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5f0c11ab60 0x7f5f0c138a30 secure :-1 s=THROTTLE_DONE pgs=16 cs=0 l=1 rx=0x7f5f040032b0 tx=0x7f5f04010ea0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:18.599 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.584+0000 7f5ef3fff700 1 --2- 172.21.15.40:0/4180003501 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5eec03b1c0 0x7f5eec03d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:18.599 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.584+0000 7f5ef3fff700 1 -- 172.21.15.40:0/4180003501 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1559+0+0 (secure 0 0 0) 0x7f5f0404b400 con 0x7f5f0c11ab60 2019-12-05T05:03:18.600 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.584+0000 7f5f0b7fe700 1 --2- 172.21.15.40:0/4180003501 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5eec03b1c0 0x7f5eec03d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:18.600 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.584+0000 7f5f0b7fe700 1 --2- 172.21.15.40:0/4180003501 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5eec03b1c0 0x7f5eec03d6d0 crc :-1 s=READY pgs=13 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:18.600 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.588+0000 7f5f113dd700 1 -- 172.21.15.40:0/4180003501 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f0c05d110 con 0x7f5f0c11ab60 2019-12-05T05:03:18.601 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.596+0000 7f5f0bfff700 1 --2- 172.21.15.40:0/4180003501 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5f0c11ab60 0x7f5f0c138a30 secure :-1 s=THROTTLE_DONE pgs=16 cs=0 l=1 rx=0x7f5f040032b0 tx=0x7f5f04010ea0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:18.601 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.596+0000 7f5ef3fff700 1 -- 172.21.15.40:0/4180003501 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f5f04032480 con 0x7f5f0c11ab60 2019-12-05T05:03:18.781 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.780+0000 7f5f113dd700 1 -- 172.21.15.40:0/4180003501 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "osd new", "uuid": "79c3c243-190e-4e35-b769-319a2077e7cd", "id": 7} v 0) v1 -- 0x7f5f0c05b8c0 con 0x7f5f0c11ab60 2019-12-05T05:03:18.793 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.788+0000 7f5f0bfff700 1 --2- 172.21.15.40:0/4180003501 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5f0c11ab60 0x7f5f0c138a30 secure :-1 s=THROTTLE_DONE pgs=16 cs=0 l=1 rx=0x7f5f040032b0 tx=0x7f5f04010ea0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:18.793 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.788+0000 7f5ef3fff700 1 -- 172.21.15.40:0/4180003501 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "79c3c243-190e-4e35-b769-319a2077e7cd", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7f5f040326e0 con 0x7f5f0c11ab60 2019-12-05T05:03:18.795 INFO:teuthology.orchestra.run.smithi040.stdout:7 2019-12-05T05:03:18.800 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.796+0000 7f5f113dd700 1 -- 172.21.15.40:0/4180003501 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5eec03b1c0 msgr2=0x7f5eec03d6d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:18.800 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.796+0000 7f5f113dd700 1 --2- 172.21.15.40:0/4180003501 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5eec03b1c0 0x7f5eec03d6d0 crc :-1 s=READY pgs=13 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:18.800 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.796+0000 7f5f113dd700 1 -- 172.21.15.40:0/4180003501 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5f0c11ab60 msgr2=0x7f5f0c138a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:18.800 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.796+0000 7f5f113dd700 1 --2- 172.21.15.40:0/4180003501 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5f0c11ab60 0x7f5f0c138a30 secure :-1 s=READY pgs=16 cs=0 l=1 rx=0x7f5f040032b0 tx=0x7f5f04010ea0).stop 2019-12-05T05:03:18.800 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.796+0000 7f5f113dd700 1 -- 172.21.15.40:0/4180003501 shutdown_connections 2019-12-05T05:03:18.800 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.796+0000 7f5f113dd700 1 --2- 172.21.15.40:0/4180003501 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5eec03b1c0 0x7f5eec03d6d0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:18.801 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.796+0000 7f5f113dd700 1 --2- 172.21.15.40:0/4180003501 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5f0c11ab60 0x7f5f0c138a30 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:18.801 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.796+0000 7f5f113dd700 1 -- 172.21.15.40:0/4180003501 shutdown_connections 2019-12-05T05:03:18.801 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.800+0000 7f5f113dd700 1 -- 172.21.15.40:0/4180003501 wait complete. 2019-12-05T05:03:18.801 INFO:teuthology.orchestra.run.smithi040.stderr:2019-12-05T05:03:18.800+0000 7f5f113dd700 1 -- 172.21.15.40:0/4180003501 >> 172.21.15.40:0/4180003501 conn(0x7f5f0c066fe0 msgr2=0x7f5f0c09f350 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:18.812 INFO:tasks.ceph.osd.0:Restarting daemon 2019-12-05T05:03:18.812 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2019-12-05T05:03:18.816 INFO:tasks.ceph.osd.0:Started 2019-12-05T05:03:18.816 INFO:tasks.ceph.osd.1:Restarting daemon 2019-12-05T05:03:18.816 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2019-12-05T05:03:18.819 INFO:tasks.ceph.osd.1:Started 2019-12-05T05:03:18.819 INFO:tasks.ceph.osd.2:Restarting daemon 2019-12-05T05:03:18.820 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2019-12-05T05:03:18.823 INFO:tasks.ceph.osd.2:Started 2019-12-05T05:03:18.824 INFO:tasks.ceph.osd.3:Restarting daemon 2019-12-05T05:03:18.824 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2019-12-05T05:03:18.827 INFO:tasks.ceph.osd.3:Started 2019-12-05T05:03:18.828 INFO:tasks.ceph.osd.4:Restarting daemon 2019-12-05T05:03:18.828 INFO:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2019-12-05T05:03:18.831 INFO:tasks.ceph.osd.4:Started 2019-12-05T05:03:18.832 INFO:tasks.ceph.osd.5:Restarting daemon 2019-12-05T05:03:18.832 INFO:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2019-12-05T05:03:18.835 INFO:tasks.ceph.osd.5:Started 2019-12-05T05:03:18.835 INFO:tasks.ceph.osd.6:Restarting daemon 2019-12-05T05:03:18.835 INFO:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2019-12-05T05:03:18.838 INFO:tasks.ceph.osd.6:Started 2019-12-05T05:03:18.839 INFO:tasks.ceph.osd.7:Restarting daemon 2019-12-05T05:03:18.839 INFO:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2019-12-05T05:03:18.843 INFO:tasks.ceph.osd.7:Started 2019-12-05T05:03:18.843 INFO:tasks.ceph:Waiting for OSDs to come up 2019-12-05T05:03:19.183 INFO:tasks.ceph.osd.2.smithi042.stderr:2019-12-05T05:03:19.182+0000 7fc20b11cd80 -1 Falling back to public interface 2019-12-05T05:03:19.192 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T05:03:19.190+0000 7fbfab286d80 -1 Falling back to public interface 2019-12-05T05:03:19.370 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T05:03:19.368+0000 7fc735e1fd80 -1 Falling back to public interface 2019-12-05T05:03:19.438 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T05:03:19.434+0000 7fc9e6231d80 -1 Falling back to public interface 2019-12-05T05:03:19.443 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T05:03:19.440+0000 7f4295101d80 -1 Falling back to public interface 2019-12-05T05:03:19.444 INFO:tasks.ceph.osd.6.smithi040.stderr:2019-12-05T05:03:19.440+0000 7fcee8a0ad80 -1 Falling back to public interface 2019-12-05T05:03:19.444 INFO:tasks.ceph.osd.7.smithi040.stderr:2019-12-05T05:03:19.440+0000 7f1c830fed80 -1 Falling back to public interface 2019-12-05T05:03:19.448 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T05:03:19.446+0000 7fd885a32d80 -1 Falling back to public interface 2019-12-05T05:03:19.651 INFO:teuthology.orchestra.run.smithi042:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2019-12-05T05:03:19.741 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:19.730+0000 7fa5876bb700 1 Processor -- start 2019-12-05T05:03:19.741 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:19.730+0000 7fa5876bb700 1 -- start start 2019-12-05T05:03:19.741 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:19.734+0000 7fa5876bb700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa58008df70 0x7fa58009f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:19.741 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:19.734+0000 7fa5876bb700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa58009fc50 con 0x7fa58009f820 2019-12-05T05:03:19.742 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:19.734+0000 7fa5876bb700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5800a0700 con 0x7fa5800a7730 2019-12-05T05:03:19.742 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:19.734+0000 7fa5876bb700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5800a0af0 con 0x7fa58008df70 2019-12-05T05:03:19.742 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:19.734+0000 7fa585c58700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fa5800a7730 0x7fa58007e710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:52728/0 (socket says 172.21.15.42:52728) 2019-12-05T05:03:19.742 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:19.734+0000 7fa585c58700 1 -- 172.21.15.42:0/1787319592 learned_addr learned my addr 172.21.15.42:0/1787319592 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:03:19.742 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:19.734+0000 7fa585457700 1 --2- 172.21.15.42:0/1787319592 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa58008df70 0x7fa58009f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:19.742 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:19.734+0000 7fa584c56700 1 -- 172.21.15.42:0/1787319592 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 65890821 0 0) 0x7fa5800a0700 con 0x7fa5800a7730 2019-12-05T05:03:19.742 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:19.734+0000 7fa584c56700 1 -- 172.21.15.42:0/1787319592 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa56c0030b0 con 0x7fa5800a7730 2019-12-05T05:03:19.743 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:19.734+0000 7fa584c56700 1 -- 172.21.15.42:0/1787319592 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3477227207 0 0) 0x7fa58009fc50 con 0x7fa58009f820 2019-12-05T05:03:19.743 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:19.734+0000 7fa584c56700 1 -- 172.21.15.42:0/1787319592 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5800a0700 con 0x7fa58009f820 2019-12-05T05:03:19.743 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:19.734+0000 7fa585457700 1 -- 172.21.15.42:0/1787319592 >> v1:172.21.15.42:6789/0 conn(0x7fa5800a7730 legacy=0x7fa58007e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:19.743 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:19.734+0000 7fa585457700 1 -- 172.21.15.42:0/1787319592 >> v1:172.21.15.40:6789/0 conn(0x7fa58009f820 legacy=0x7fa58014efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:19.743 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:19.734+0000 7fa585457700 1 -- 172.21.15.42:0/1787319592 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa58007ef10 con 0x7fa58008df70 2019-12-05T05:03:19.743 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:19.734+0000 7fa585457700 1 --2- 172.21.15.42:0/1787319592 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa58008df70 0x7fa58009f2d0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fa570002f30 tx=0x7fa57000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:19.743 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:19.734+0000 7fa585457700 1 --2- 172.21.15.42:0/1787319592 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa58008df70 0x7fa58009f2d0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fa570002f30 tx=0x7fa57000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:19.744 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:19.734+0000 7fa585457700 1 --2- 172.21.15.42:0/1787319592 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa58008df70 0x7fa58009f2d0 secure :-1 s=READY pgs=20 cs=0 l=1 rx=0x7fa570002f30 tx=0x7fa57000fbf0).ready entity=mon.2 client_cookie=14f8f2c1742f8799 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:19.744 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:19.734+0000 7fa585457700 1 --2- 172.21.15.42:0/1787319592 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa58008df70 0x7fa58009f2d0 secure :-1 s=THROTTLE_DONE pgs=20 cs=0 l=1 rx=0x7fa570002f30 tx=0x7fa57000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:19.744 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:19.734+0000 7fa584c56700 1 -- 172.21.15.42:0/1787319592 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa5700112b0 con 0x7fa58008df70 2019-12-05T05:03:19.744 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:19.734+0000 7fa585457700 1 --2- 172.21.15.42:0/1787319592 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa58008df70 0x7fa58009f2d0 secure :-1 s=THROTTLE_DONE pgs=20 cs=0 l=1 rx=0x7fa570002f30 tx=0x7fa57000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:19.744 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:19.734+0000 7fa584c56700 1 -- 172.21.15.42:0/1787319592 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa570008bc0 con 0x7fa58008df70 2019-12-05T05:03:19.745 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:19.734+0000 7fa585457700 1 --2- 172.21.15.42:0/1787319592 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa58008df70 0x7fa58009f2d0 secure :-1 s=THROTTLE_DONE pgs=20 cs=0 l=1 rx=0x7fa570002f30 tx=0x7fa57000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:19.745 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:19.734+0000 7fa584c56700 1 -- 172.21.15.42:0/1787319592 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa570011890 con 0x7fa58008df70 2019-12-05T05:03:19.745 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:19.738+0000 7fa5876bb700 1 -- 172.21.15.42:0/1787319592 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa58008df70 msgr2=0x7fa58009f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:19.745 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:19.738+0000 7fa5876bb700 1 --2- 172.21.15.42:0/1787319592 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa58008df70 0x7fa58009f2d0 secure :-1 s=READY pgs=20 cs=0 l=1 rx=0x7fa570002f30 tx=0x7fa57000fbf0).stop 2019-12-05T05:03:19.745 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:19.738+0000 7fa5876bb700 1 -- 172.21.15.42:0/1787319592 shutdown_connections 2019-12-05T05:03:19.745 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:19.738+0000 7fa5876bb700 1 --2- 172.21.15.42:0/1787319592 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa58008df70 0x7fa58009f2d0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:19.745 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:19.738+0000 7fa5876bb700 1 -- 172.21.15.42:0/1787319592 shutdown_connections 2019-12-05T05:03:19.746 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:19.738+0000 7fa5876bb700 1 -- 172.21.15.42:0/1787319592 wait complete. 2019-12-05T05:03:19.746 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:19.738+0000 7fa5876bb700 1 -- 172.21.15.42:0/1787319592 >> 172.21.15.42:0/1787319592 conn(0x7fa5800aabd0 msgr2=0x7fa5800a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:19.749 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:19.738+0000 7fa5876bb700 1 Processor -- start 2019-12-05T05:03:19.749 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:19.742+0000 7fa5876bb700 1 -- start start 2019-12-05T05:03:19.750 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:19.742+0000 7fa5876bb700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa580143800 0x7fa580066fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:19.750 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:19.742+0000 7fa5876bb700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa580143c30 con 0x7fa580130ee0 2019-12-05T05:03:19.750 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:19.742+0000 7fa5876bb700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa580067520 con 0x7fa5800a1a50 2019-12-05T05:03:19.751 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:19.742+0000 7fa5876bb700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa580143f10 con 0x7fa580143800 2019-12-05T05:03:19.751 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:19.742+0000 7fa585c58700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fa5800a1a50 0x7fa5801306e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:52734/0 (socket says 172.21.15.42:52734) 2019-12-05T05:03:19.751 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:19.742+0000 7fa585c58700 1 -- 172.21.15.42:0/1385177208 learned_addr learned my addr 172.21.15.42:0/1385177208 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:03:19.751 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:19.742+0000 7fa586459700 1 --2- 172.21.15.42:0/1385177208 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa580143800 0x7fa580066fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:19.751 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:19.742+0000 7fa5767fc700 1 -- 172.21.15.42:0/1385177208 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 380446349 0 0) 0x7fa580067520 con 0x7fa5800a1a50 2019-12-05T05:03:19.752 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:19.742+0000 7fa5767fc700 1 -- 172.21.15.42:0/1385177208 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5680030b0 con 0x7fa5800a1a50 2019-12-05T05:03:19.752 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:19.742+0000 7fa5767fc700 1 -- 172.21.15.42:0/1385177208 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1619936869 0 0) 0x7fa5700078c0 con 0x7fa580130ee0 2019-12-05T05:03:19.752 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:19.742+0000 7fa586459700 1 -- 172.21.15.42:0/1385177208 >> v1:172.21.15.42:6789/0 conn(0x7fa5800a1a50 legacy=0x7fa5801306e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:19.752 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:19.742+0000 7fa586459700 1 -- 172.21.15.42:0/1385177208 >> v1:172.21.15.40:6789/0 conn(0x7fa580130ee0 legacy=0x7fa5801359a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:19.752 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:19.742+0000 7fa586459700 1 -- 172.21.15.42:0/1385177208 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa58007d100 con 0x7fa580143800 2019-12-05T05:03:19.752 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:19.742+0000 7fa586459700 1 --2- 172.21.15.42:0/1385177208 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa580143800 0x7fa580066fd0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fa578001210 tx=0x7fa578010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:19.753 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:19.742+0000 7fa586459700 1 --2- 172.21.15.42:0/1385177208 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa580143800 0x7fa580066fd0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fa578001210 tx=0x7fa578010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:19.753 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:19.742+0000 7fa586459700 1 --2- 172.21.15.42:0/1385177208 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa580143800 0x7fa580066fd0 secure :-1 s=READY pgs=21 cs=0 l=1 rx=0x7fa578001210 tx=0x7fa578010aa0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:19.753 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:19.742+0000 7fa586459700 1 --2- 172.21.15.42:0/1385177208 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa580143800 0x7fa580066fd0 secure :-1 s=THROTTLE_DONE pgs=21 cs=0 l=1 rx=0x7fa578001210 tx=0x7fa578010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:19.753 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:19.742+0000 7fa5767fc700 1 -- 172.21.15.42:0/1385177208 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa578006ac0 con 0x7fa580143800 2019-12-05T05:03:19.753 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:19.742+0000 7fa586459700 1 --2- 172.21.15.42:0/1385177208 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa580143800 0x7fa580066fd0 secure :-1 s=THROTTLE_DONE pgs=21 cs=0 l=1 rx=0x7fa578001210 tx=0x7fa578010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:19.753 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:19.742+0000 7fa5767fc700 1 -- 172.21.15.42:0/1385177208 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5780066e0 con 0x7fa580143800 2019-12-05T05:03:19.753 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:19.742+0000 7fa586459700 1 --2- 172.21.15.42:0/1385177208 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa580143800 0x7fa580066fd0 secure :-1 s=THROTTLE_DONE pgs=21 cs=0 l=1 rx=0x7fa578001210 tx=0x7fa578010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:19.754 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:19.742+0000 7fa5876bb700 1 -- 172.21.15.42:0/1385177208 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5801391c0 con 0x7fa580143800 2019-12-05T05:03:19.754 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:19.742+0000 7fa5767fc700 1 -- 172.21.15.42:0/1385177208 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa5780127c0 con 0x7fa580143800 2019-12-05T05:03:19.754 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:19.742+0000 7fa5876bb700 1 -- 172.21.15.42:0/1385177208 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa58008eb60 con 0x7fa580143800 2019-12-05T05:03:19.754 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:19.746+0000 7fa586459700 1 --2- 172.21.15.42:0/1385177208 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa580143800 0x7fa580066fd0 secure :-1 s=THROTTLE_DONE pgs=21 cs=0 l=1 rx=0x7fa578001210 tx=0x7fa578010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:19.754 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:19.746+0000 7fa5767fc700 1 -- 172.21.15.42:0/1385177208 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fa578012b20 con 0x7fa580143800 2019-12-05T05:03:19.754 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:19.746+0000 7fa586459700 1 --2- 172.21.15.42:0/1385177208 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa580143800 0x7fa580066fd0 secure :-1 s=THROTTLE_DONE pgs=21 cs=0 l=1 rx=0x7fa578001210 tx=0x7fa578010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:19.754 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:19.746+0000 7fa5767fc700 1 --2- 172.21.15.42:0/1385177208 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa56803b1e0 0x7fa56803d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:19.755 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:19.746+0000 7fa5767fc700 1 -- 172.21.15.42:0/1385177208 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1674+0+0 (secure 0 0 0) 0x7fa57804d290 con 0x7fa580143800 2019-12-05T05:03:19.755 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:19.746+0000 7fa585c58700 1 --2- 172.21.15.42:0/1385177208 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa56803b1e0 0x7fa56803d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:19.755 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:19.746+0000 7fa585c58700 1 --2- 172.21.15.42:0/1385177208 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa56803b1e0 0x7fa56803d6f0 crc :-1 s=READY pgs=14 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:19.755 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:19.746+0000 7fa5876bb700 1 -- 172.21.15.42:0/1385177208 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa58005d100 con 0x7fa580143800 2019-12-05T05:03:19.759 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:19.758+0000 7fa586459700 1 --2- 172.21.15.42:0/1385177208 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa580143800 0x7fa580066fd0 secure :-1 s=THROTTLE_DONE pgs=21 cs=0 l=1 rx=0x7fa578001210 tx=0x7fa578010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:19.760 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:19.758+0000 7fa5767fc700 1 -- 172.21.15.42:0/1385177208 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fa5780342f0 con 0x7fa580143800 2019-12-05T05:03:19.930 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:19.926+0000 7fa5876bb700 1 -- 172.21.15.42:0/1385177208 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa58005b8b0 con 0x7fa580143800 2019-12-05T05:03:19.930 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:19.930+0000 7fa586459700 1 --2- 172.21.15.42:0/1385177208 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa580143800 0x7fa580066fd0 secure :-1 s=THROTTLE_DONE pgs=21 cs=0 l=1 rx=0x7fa578001210 tx=0x7fa578010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:19.930 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:19.930+0000 7fa5767fc700 1 -- 172.21.15.42:0/1385177208 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+6849 (secure 0 0 0) 0x7fa578012e30 con 0x7fa580143800 2019-12-05T05:03:19.932 INFO:teuthology.misc.health.smithi042.stdout: 2019-12-05T05:03:19.933 INFO:teuthology.misc.health.smithi042.stdout:{"epoch":10,"fsid":"5d836cfc-a83b-4433-9b3e-b9b2f31c9050","created":"2019-12-05T05:03:13.215602+0000","modified":"2019-12-05T05:03:18.783331+0000","last_up_change":"0.000000","last_in_change":"0.000000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[],"osds":[{"osd":0,"uuid":"e3fc49bb-dcbc-43ff-ac7c-9eb0ea0bd257","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"902543fb-f5a2-4321-9c9c-26f9b9ca8629","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"28739ff7-bed7-4b75-8ea5-b57d088f1250","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1960f66b-2d40-4bb9-a289-ce8cd0c1c142","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"45bbf1bd-7c8a-4a0d-b76b-afa6148a855c","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"8a5511cd-925f-4869-a49e-924383e4ce53","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d692c179-379e-446c-8484-a219a9241e04","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"79c3c243-190e-4e35-b769-319a2077e7cd","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2019-12-05T05:03:19.935 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:19.934+0000 7fa5876bb700 1 -- 172.21.15.42:0/1385177208 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa56803b1e0 msgr2=0x7fa56803d6f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:19.936 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:19.934+0000 7fa5876bb700 1 --2- 172.21.15.42:0/1385177208 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa56803b1e0 0x7fa56803d6f0 crc :-1 s=READY pgs=14 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:19.936 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:19.934+0000 7fa5876bb700 1 -- 172.21.15.42:0/1385177208 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa580143800 msgr2=0x7fa580066fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:19.936 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:19.934+0000 7fa5876bb700 1 --2- 172.21.15.42:0/1385177208 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa580143800 0x7fa580066fd0 secure :-1 s=READY pgs=21 cs=0 l=1 rx=0x7fa578001210 tx=0x7fa578010aa0).stop 2019-12-05T05:03:19.936 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:19.934+0000 7fa5876bb700 1 -- 172.21.15.42:0/1385177208 shutdown_connections 2019-12-05T05:03:19.936 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:19.934+0000 7fa5876bb700 1 --2- 172.21.15.42:0/1385177208 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa56803b1e0 0x7fa56803d6f0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:19.936 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:19.934+0000 7fa5876bb700 1 --2- 172.21.15.42:0/1385177208 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa580143800 0x7fa580066fd0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:19.937 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:19.934+0000 7fa5876bb700 1 -- 172.21.15.42:0/1385177208 shutdown_connections 2019-12-05T05:03:19.937 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:19.934+0000 7fa5876bb700 1 -- 172.21.15.42:0/1385177208 wait complete. 2019-12-05T05:03:19.937 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:19.934+0000 7fa5876bb700 1 -- 172.21.15.42:0/1385177208 >> 172.21.15.42:0/1385177208 conn(0x7fa58007dd00 msgr2=0x7fa58011ce40 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:19.944 DEBUG:teuthology.misc:0 of 8 OSDs are up 2019-12-05T05:03:20.354 INFO:tasks.ceph.osd.2.smithi042.stderr:2019-12-05T05:03:20.354+0000 7fc20b11cd80 -1 osd.2 0 log_to_monitors {default=true} 2019-12-05T05:03:20.382 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T05:03:20.378+0000 7fbfab286d80 -1 osd.0 0 log_to_monitors {default=true} 2019-12-05T05:03:20.520 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T05:03:20.516+0000 7fc735e1fd80 -1 osd.5 0 log_to_monitors {default=true} 2019-12-05T05:03:20.604 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T05:03:20.598+0000 7fc9e6231d80 -1 osd.3 0 log_to_monitors {default=true} 2019-12-05T05:03:20.614 INFO:tasks.ceph.osd.6.smithi040.stderr:2019-12-05T05:03:20.612+0000 7fcee8a0ad80 -1 osd.6 0 log_to_monitors {default=true} 2019-12-05T05:03:20.622 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T05:03:20.620+0000 7f4295101d80 -1 osd.4 0 log_to_monitors {default=true} 2019-12-05T05:03:20.638 INFO:tasks.ceph.osd.7.smithi040.stderr:2019-12-05T05:03:20.636+0000 7f1c830fed80 -1 osd.7 0 log_to_monitors {default=true} 2019-12-05T05:03:20.643 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T05:03:20.642+0000 7fd885a32d80 -1 osd.1 0 log_to_monitors {default=true} 2019-12-05T05:03:21.665 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T05:03:21.662+0000 7fd86ec64700 -1 osd.1 0 waiting for initial osdmap 2019-12-05T05:03:21.669 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T05:03:21.666+0000 7fbf944b8700 -1 osd.0 0 waiting for initial osdmap 2019-12-05T05:03:21.670 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T05:03:21.668+0000 7fc71f051700 -1 osd.5 0 waiting for initial osdmap 2019-12-05T05:03:21.670 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T05:03:21.668+0000 7f427e333700 -1 osd.4 0 waiting for initial osdmap 2019-12-05T05:03:21.670 INFO:tasks.ceph.osd.6.smithi040.stderr:2019-12-05T05:03:21.668+0000 7fced1c3c700 -1 osd.6 0 waiting for initial osdmap 2019-12-05T05:03:21.670 INFO:tasks.ceph.osd.7.smithi040.stderr:2019-12-05T05:03:21.668+0000 7f1c6c330700 -1 osd.7 0 waiting for initial osdmap 2019-12-05T05:03:21.671 INFO:tasks.ceph.osd.2.smithi042.stderr:2019-12-05T05:03:21.670+0000 7fc1f434e700 -1 osd.2 0 waiting for initial osdmap 2019-12-05T05:03:21.671 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T05:03:21.670+0000 7fc9cf463700 -1 osd.3 0 waiting for initial osdmap 2019-12-05T05:03:26.755 INFO:teuthology.orchestra.run.smithi042:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2019-12-05T05:03:26.869 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:26.848+0000 7f014670f700 1 Processor -- start 2019-12-05T05:03:26.872 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:26.848+0000 7f014670f700 1 -- start start 2019-12-05T05:03:26.872 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:26.848+0000 7f014670f700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f01400a7740 0x7f014008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:26.872 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:26.848+0000 7f014670f700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f014007e720 0x7f014007eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:26.872 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:26.848+0000 7f014670f700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01400a0710 con 0x7f014009f2e0 2019-12-05T05:03:26.876 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:26.848+0000 7f014670f700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01400a0b00 con 0x7f01400a7740 2019-12-05T05:03:26.876 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:26.848+0000 7f014670f700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f014008e5b0 con 0x7f014007e720 2019-12-05T05:03:26.877 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:26.848+0000 7f013ffff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f014007e720 0x7f014007eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:26.877 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:26.848+0000 7f013ffff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f014007e720 0x7f014007eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:52928/0 (socket says 172.21.15.42:52928) 2019-12-05T05:03:26.877 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:26.848+0000 7f013ffff700 1 -- 172.21.15.42:0/3597763243 learned_addr learned my addr 172.21.15.42:0/3597763243 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:03:26.877 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:26.848+0000 7f0144cac700 1 --2- 172.21.15.42:0/3597763243 >> v2:172.21.15.40:3300/0 conn(0x7f01400a7740 0x7f014008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:26.877 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:26.848+0000 7f013ffff700 1 -- 172.21.15.42:0/3597763243 >> v2:172.21.15.40:3300/0 conn(0x7f01400a7740 msgr2=0x7f014008df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:03:26.877 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:26.848+0000 7f013ffff700 1 --2- 172.21.15.42:0/3597763243 >> v2:172.21.15.40:3300/0 conn(0x7f01400a7740 0x7f014008df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:03:26.878 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:26.848+0000 7f013ffff700 1 -- 172.21.15.42:0/3597763243 >> v1:172.21.15.40:6789/0 conn(0x7f014009f2e0 legacy=0x7f014009f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:26.878 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:26.848+0000 7f013ffff700 1 -- 172.21.15.42:0/3597763243 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f014007f0a0 con 0x7f014007e720 2019-12-05T05:03:26.878 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:26.848+0000 7f013ffff700 1 --2- 172.21.15.42:0/3597763243 >> v2:172.21.15.42:3300/0 conn(0x7f014007e720 0x7f014007eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f0128002f30 tx=0x7f012800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:26.878 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:26.848+0000 7f013ffff700 1 --2- 172.21.15.42:0/3597763243 >> v2:172.21.15.42:3300/0 conn(0x7f014007e720 0x7f014007eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f0128002f30 tx=0x7f012800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:26.878 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:26.848+0000 7f013ffff700 1 --2- 172.21.15.42:0/3597763243 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f014007e720 0x7f014007eb50 secure :-1 s=READY pgs=39 cs=0 l=1 rx=0x7f0128002f30 tx=0x7f012800fd30).ready entity=mon.1 client_cookie=17400fa8808819b5 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:26.878 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:26.848+0000 7f013ffff700 1 --2- 172.21.15.42:0/3597763243 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f014007e720 0x7f014007eb50 secure :-1 s=THROTTLE_DONE pgs=39 cs=0 l=1 rx=0x7f0128002f30 tx=0x7f012800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:26.878 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:26.848+0000 7f013f7fe700 1 -- 172.21.15.42:0/3597763243 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0128011370 con 0x7f014007e720 2019-12-05T05:03:26.879 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:26.848+0000 7f013ffff700 1 --2- 172.21.15.42:0/3597763243 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f014007e720 0x7f014007eb50 secure :-1 s=THROTTLE_DONE pgs=39 cs=0 l=1 rx=0x7f0128002f30 tx=0x7f012800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:26.879 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:26.848+0000 7f013ffff700 1 --2- 172.21.15.42:0/3597763243 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f014007e720 0x7f014007eb50 secure :-1 s=THROTTLE_DONE pgs=39 cs=0 l=1 rx=0x7f0128002f30 tx=0x7f012800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:26.879 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:26.848+0000 7f013f7fe700 1 -- 172.21.15.42:0/3597763243 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0128008bb0 con 0x7f014007e720 2019-12-05T05:03:26.879 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:26.848+0000 7f013f7fe700 1 -- 172.21.15.42:0/3597763243 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0128011950 con 0x7f014007e720 2019-12-05T05:03:26.879 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:26.848+0000 7f014670f700 1 -- 172.21.15.42:0/3597763243 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f014007e720 msgr2=0x7f014007eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:26.879 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:26.848+0000 7f014670f700 1 --2- 172.21.15.42:0/3597763243 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f014007e720 0x7f014007eb50 secure :-1 s=READY pgs=39 cs=0 l=1 rx=0x7f0128002f30 tx=0x7f012800fd30).stop 2019-12-05T05:03:26.879 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:26.852+0000 7f014670f700 1 -- 172.21.15.42:0/3597763243 shutdown_connections 2019-12-05T05:03:26.880 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:26.852+0000 7f014670f700 1 --2- 172.21.15.42:0/3597763243 >> v2:172.21.15.40:3300/0 conn(0x7f01400a7740 0x7f014008df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:03:26.880 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:26.852+0000 7f014670f700 1 --2- 172.21.15.42:0/3597763243 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f014007e720 0x7f014007eb50 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:26.880 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:26.852+0000 7f014670f700 1 -- 172.21.15.42:0/3597763243 shutdown_connections 2019-12-05T05:03:26.880 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:26.852+0000 7f014670f700 1 -- 172.21.15.42:0/3597763243 wait complete. 2019-12-05T05:03:26.880 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:26.852+0000 7f014670f700 1 -- 172.21.15.42:0/3597763243 >> 172.21.15.42:0/3597763243 conn(0x7f01400aabe0 msgr2=0x7f01400a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:26.880 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:26.852+0000 7f014670f700 1 Processor -- start 2019-12-05T05:03:26.880 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:26.852+0000 7f014670f700 1 -- start start 2019-12-05T05:03:26.881 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:26.852+0000 7f014670f700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f014012e290 0x7f0140113070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:26.881 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:26.852+0000 7f014670f700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f014012e6c0 0x7f01401135c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:26.881 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:26.852+0000 7f014670f700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f014012da70 con 0x7f014005a070 2019-12-05T05:03:26.881 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:26.852+0000 7f014670f700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f014005a4a0 con 0x7f014012e6c0 2019-12-05T05:03:26.881 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:26.852+0000 7f014670f700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0140112650 con 0x7f014012e290 2019-12-05T05:03:26.881 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:26.852+0000 7f01454ad700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f014012e6c0 0x7f01401135c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:26.881 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:26.852+0000 7f01454ad700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f014012e6c0 0x7f01401135c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:52938/0 (socket says 172.21.15.42:52938) 2019-12-05T05:03:26.882 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:26.852+0000 7f01454ad700 1 -- 172.21.15.42:0/2880202399 learned_addr learned my addr 172.21.15.42:0/2880202399 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:03:26.882 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:26.852+0000 7f013ffff700 1 --2- 172.21.15.42:0/2880202399 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f014012e290 0x7f0140113070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:26.882 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:26.856+0000 7f01454ad700 1 -- 172.21.15.42:0/2880202399 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f014012e290 msgr2=0x7f0140113070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:26.882 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:26.856+0000 7f01454ad700 1 --2- 172.21.15.42:0/2880202399 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f014012e290 0x7f0140113070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:26.882 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:26.856+0000 7f01454ad700 1 -- 172.21.15.42:0/2880202399 >> v1:172.21.15.40:6789/0 conn(0x7f014005a070 legacy=0x7f0140111db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:26.882 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:26.856+0000 7f01454ad700 1 -- 172.21.15.42:0/2880202399 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01401425e0 con 0x7f014012e6c0 2019-12-05T05:03:26.882 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:26.856+0000 7f013ffff700 1 --2- 172.21.15.42:0/2880202399 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f014012e290 0x7f0140113070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T05:03:26.883 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:26.856+0000 7f01454ad700 1 --2- 172.21.15.42:0/2880202399 >> v2:172.21.15.42:3300/0 conn(0x7f014012e6c0 0x7f01401135c0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f014014e440 tx=0x7f01380107f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:26.883 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:26.856+0000 7f01454ad700 1 --2- 172.21.15.42:0/2880202399 >> v2:172.21.15.42:3300/0 conn(0x7f014012e6c0 0x7f01401135c0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f014014e440 tx=0x7f01380107f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:26.883 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:26.856+0000 7f01454ad700 1 --2- 172.21.15.42:0/2880202399 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f014012e6c0 0x7f01401135c0 secure :-1 s=READY pgs=40 cs=0 l=1 rx=0x7f014014e440 tx=0x7f01380107f0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:26.883 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:26.856+0000 7f01454ad700 1 --2- 172.21.15.42:0/2880202399 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f014012e6c0 0x7f01401135c0 secure :-1 s=THROTTLE_DONE pgs=40 cs=0 l=1 rx=0x7f014014e440 tx=0x7f01380107f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:26.883 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:26.856+0000 7f013d7fa700 1 -- 172.21.15.42:0/2880202399 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0138006a40 con 0x7f014012e6c0 2019-12-05T05:03:26.883 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:26.856+0000 7f01454ad700 1 --2- 172.21.15.42:0/2880202399 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f014012e6c0 0x7f01401135c0 secure :-1 s=THROTTLE_DONE pgs=40 cs=0 l=1 rx=0x7f014014e440 tx=0x7f01380107f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:26.884 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:26.856+0000 7f013d7fa700 1 -- 172.21.15.42:0/2880202399 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0138006660 con 0x7f014012e6c0 2019-12-05T05:03:26.884 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:26.856+0000 7f01454ad700 1 --2- 172.21.15.42:0/2880202399 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f014012e6c0 0x7f01401135c0 secure :-1 s=THROTTLE_DONE pgs=40 cs=0 l=1 rx=0x7f014014e440 tx=0x7f01380107f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:26.884 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:26.856+0000 7f014670f700 1 -- 172.21.15.42:0/2880202399 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0140134ec0 con 0x7f014012e6c0 2019-12-05T05:03:26.884 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:26.856+0000 7f013d7fa700 1 -- 172.21.15.42:0/2880202399 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0138010ff0 con 0x7f014012e6c0 2019-12-05T05:03:26.884 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:26.856+0000 7f014670f700 1 -- 172.21.15.42:0/2880202399 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0140135470 con 0x7f014012e6c0 2019-12-05T05:03:26.885 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:26.856+0000 7f01454ad700 1 --2- 172.21.15.42:0/2880202399 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f014012e6c0 0x7f01401135c0 secure :-1 s=THROTTLE_DONE pgs=40 cs=0 l=1 rx=0x7f014014e440 tx=0x7f01380107f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:26.885 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:26.856+0000 7f013d7fa700 1 -- 172.21.15.42:0/2880202399 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f0138011310 con 0x7f014012e6c0 2019-12-05T05:03:26.885 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:26.856+0000 7f01454ad700 1 --2- 172.21.15.42:0/2880202399 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f014012e6c0 0x7f01401135c0 secure :-1 s=THROTTLE_DONE pgs=40 cs=0 l=1 rx=0x7f014014e440 tx=0x7f01380107f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:26.885 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:26.856+0000 7f013d7fa700 1 --2- 172.21.15.42:0/2880202399 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f012403ade0 0x7f012403d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:26.885 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:26.856+0000 7f013d7fa700 1 -- 172.21.15.42:0/2880202399 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 4500+0+0 (secure 0 0 0) 0x7f013804b0d0 con 0x7f014012e6c0 2019-12-05T05:03:26.885 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:26.856+0000 7f0144cac700 1 --2- 172.21.15.42:0/2880202399 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f012403ade0 0x7f012403d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:26.885 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:26.856+0000 7f0144cac700 1 --2- 172.21.15.42:0/2880202399 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f012403ade0 0x7f012403d2f0 crc :-1 s=READY pgs=39 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:26.886 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:26.864+0000 7f014670f700 1 -- 172.21.15.42:0/2880202399 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f014005d110 con 0x7f014012e6c0 2019-12-05T05:03:26.886 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:26.872+0000 7f01454ad700 1 --2- 172.21.15.42:0/2880202399 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f014012e6c0 0x7f01401135c0 secure :-1 s=THROTTLE_DONE pgs=40 cs=0 l=1 rx=0x7f014014e440 tx=0x7f01380107f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:26.886 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:26.872+0000 7f013d7fa700 1 -- 172.21.15.42:0/2880202399 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f0138032130 con 0x7f014012e6c0 2019-12-05T05:03:27.053 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:27.044+0000 7f014670f700 1 -- 172.21.15.42:0/2880202399 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f01400a71e0 con 0x7f014012e6c0 2019-12-05T05:03:27.053 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:27.044+0000 7f01454ad700 1 --2- 172.21.15.42:0/2880202399 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f014012e6c0 0x7f01401135c0 secure :-1 s=THROTTLE_DONE pgs=40 cs=0 l=1 rx=0x7f014014e440 tx=0x7f01380107f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:27.053 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:27.044+0000 7f013d7fa700 1 -- 172.21.15.42:0/2880202399 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v14) v1 ==== 74+0+10391 (secure 0 0 0) 0x7f0138032390 con 0x7f014012e6c0 2019-12-05T05:03:27.054 INFO:teuthology.misc.health.smithi042.stdout: 2019-12-05T05:03:27.054 INFO:teuthology.misc.health.smithi042.stdout:{"epoch":14,"fsid":"5d836cfc-a83b-4433-9b3e-b9b2f31c9050","created":"2019-12-05T05:03:13.215602+0000","modified":"2019-12-05T05:03:24.147744+0000","last_up_change":"2019-12-05T05:03:22.664783+0000","last_in_change":"2019-12-05T05:03:22.664783+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[],"osds":[{"osd":0,"uuid":"e3fc49bb-dcbc-43ff-ac7c-9eb0ea0bd257","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6810","nonce":12118},{"type":"v1","addr":"172.21.15.42:6811","nonce":12118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6812","nonce":12118},{"type":"v1","addr":"172.21.15.42:6813","nonce":12118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6816","nonce":12118},{"type":"v1","addr":"172.21.15.42:6817","nonce":12118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6814","nonce":12118},{"type":"v1","addr":"172.21.15.42:6815","nonce":12118}]},"public_addr":"172.21.15.42:6811/12118","cluster_addr":"172.21.15.42:6813/12118","heartbeat_back_addr":"172.21.15.42:6817/12118","heartbeat_front_addr":"172.21.15.42:6815/12118","state":["exists","up"]},{"osd":1,"uuid":"902543fb-f5a2-4321-9c9c-26f9b9ca8629","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6826","nonce":12119},{"type":"v1","addr":"172.21.15.42:6827","nonce":12119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6828","nonce":12119},{"type":"v1","addr":"172.21.15.42:6829","nonce":12119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6832","nonce":12119},{"type":"v1","addr":"172.21.15.42:6833","nonce":12119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6830","nonce":12119},{"type":"v1","addr":"172.21.15.42:6831","nonce":12119}]},"public_addr":"172.21.15.42:6827/12119","cluster_addr":"172.21.15.42:6829/12119","heartbeat_back_addr":"172.21.15.42:6833/12119","heartbeat_front_addr":"172.21.15.42:6831/12119","state":["exists","up"]},{"osd":2,"uuid":"28739ff7-bed7-4b75-8ea5-b57d088f1250","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6802","nonce":12117},{"type":"v1","addr":"172.21.15.42:6803","nonce":12117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6804","nonce":12117},{"type":"v1","addr":"172.21.15.42:6805","nonce":12117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6808","nonce":12117},{"type":"v1","addr":"172.21.15.42:6809","nonce":12117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6806","nonce":12117},{"type":"v1","addr":"172.21.15.42:6807","nonce":12117}]},"public_addr":"172.21.15.42:6803/12117","cluster_addr":"172.21.15.42:6805/12117","heartbeat_back_addr":"172.21.15.42:6809/12117","heartbeat_front_addr":"172.21.15.42:6807/12117","state":["exists","up"]},{"osd":3,"uuid":"1960f66b-2d40-4bb9-a289-ce8cd0c1c142","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6818","nonce":12116},{"type":"v1","addr":"172.21.15.42:6819","nonce":12116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6820","nonce":12116},{"type":"v1","addr":"172.21.15.42:6821","nonce":12116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6824","nonce":12116},{"type":"v1","addr":"172.21.15.42:6825","nonce":12116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6822","nonce":12116},{"type":"v1","addr":"172.21.15.42:6823","nonce":12116}]},"public_addr":"172.21.15.42:6819/12116","cluster_addr":"172.21.15.42:6821/12116","heartbeat_back_addr":"172.21.15.42:6825/12116","heartbeat_front_addr":"172.21.15.42:6823/12116","state":["exists","up"]},{"osd":4,"uuid":"45bbf1bd-7c8a-4a0d-b76b-afa6148a855c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":12488},{"type":"v1","addr":"172.21.15.40:6809","nonce":12488}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":12488},{"type":"v1","addr":"172.21.15.40:6812","nonce":12488}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":12488},{"type":"v1","addr":"172.21.15.40:6825","nonce":12488}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":12488},{"type":"v1","addr":"172.21.15.40:6819","nonce":12488}]},"public_addr":"172.21.15.40:6809/12488","cluster_addr":"172.21.15.40:6812/12488","heartbeat_back_addr":"172.21.15.40:6825/12488","heartbeat_front_addr":"172.21.15.40:6819/12488","state":["exists","up"]},{"osd":5,"uuid":"8a5511cd-925f-4869-a49e-924383e4ce53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":12489},{"type":"v1","addr":"172.21.15.40:6801","nonce":12489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":12489},{"type":"v1","addr":"172.21.15.40:6803","nonce":12489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":12489},{"type":"v1","addr":"172.21.15.40:6807","nonce":12489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":12489},{"type":"v1","addr":"172.21.15.40:6805","nonce":12489}]},"public_addr":"172.21.15.40:6801/12489","cluster_addr":"172.21.15.40:6803/12489","heartbeat_back_addr":"172.21.15.40:6807/12489","heartbeat_front_addr":"172.21.15.40:6805/12489","state":["exists","up"]},{"osd":6,"uuid":"d692c179-379e-446c-8484-a219a9241e04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6811","nonce":12491},{"type":"v1","addr":"172.21.15.40:6814","nonce":12491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6817","nonce":12491},{"type":"v1","addr":"172.21.15.40:6818","nonce":12491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":12491},{"type":"v1","addr":"172.21.15.40:6828","nonce":12491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":12491},{"type":"v1","addr":"172.21.15.40:6824","nonce":12491}]},"public_addr":"172.21.15.40:6814/12491","cluster_addr":"172.21.15.40:6818/12491","heartbeat_back_addr":"172.21.15.40:6828/12491","heartbeat_front_addr":"172.21.15.40:6824/12491","state":["exists","up"]},{"osd":7,"uuid":"79c3c243-190e-4e35-b769-319a2077e7cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6813","nonce":12490},{"type":"v1","addr":"172.21.15.40:6816","nonce":12490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":12490},{"type":"v1","addr":"172.21.15.40:6823","nonce":12490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":12490},{"type":"v1","addr":"172.21.15.40:6831","nonce":12490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6827","nonce":12490},{"type":"v1","addr":"172.21.15.40:6829","nonce":12490}]},"public_addr":"172.21.15.40:6816/12490","cluster_addr":"172.21.15.40:6823/12490","heartbeat_back_addr":"172.21.15.40:6831/12490","heartbeat_front_addr":"172.21.15.40:6829/12490","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T05:03:21.349150+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T05:03:21.644281+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T05:03:21.394040+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T05:03:21.587096+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T05:03:21.656497+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T05:03:21.510547+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T05:03:21.659358+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T05:03:21.628063+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2019-12-05T05:03:27.055 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:27.048+0000 7f014670f700 1 -- 172.21.15.42:0/2880202399 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f012403ade0 msgr2=0x7f012403d2f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:27.055 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:27.048+0000 7f014670f700 1 --2- 172.21.15.42:0/2880202399 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f012403ade0 0x7f012403d2f0 crc :-1 s=READY pgs=39 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:27.055 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:27.048+0000 7f014670f700 1 -- 172.21.15.42:0/2880202399 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f014012e6c0 msgr2=0x7f01401135c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:27.055 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:27.048+0000 7f014670f700 1 --2- 172.21.15.42:0/2880202399 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f014012e6c0 0x7f01401135c0 secure :-1 s=READY pgs=40 cs=0 l=1 rx=0x7f014014e440 tx=0x7f01380107f0).stop 2019-12-05T05:03:27.055 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:27.048+0000 7f014670f700 1 -- 172.21.15.42:0/2880202399 shutdown_connections 2019-12-05T05:03:27.056 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:27.048+0000 7f014670f700 1 --2- 172.21.15.42:0/2880202399 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f012403ade0 0x7f012403d2f0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:27.056 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:27.048+0000 7f014670f700 1 --2- 172.21.15.42:0/2880202399 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f014012e6c0 0x7f01401135c0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:27.056 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:27.048+0000 7f014670f700 1 --2- 172.21.15.42:0/2880202399 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f014012e290 0x7f0140113070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:27.056 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:27.048+0000 7f014670f700 1 -- 172.21.15.42:0/2880202399 shutdown_connections 2019-12-05T05:03:27.056 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:27.048+0000 7f014670f700 1 -- 172.21.15.42:0/2880202399 wait complete. 2019-12-05T05:03:27.056 INFO:teuthology.misc.health.smithi042.stderr:2019-12-05T05:03:27.048+0000 7f014670f700 1 -- 172.21.15.42:0/2880202399 >> 172.21.15.42:0/2880202399 conn(0x7f01400aabe0 msgr2=0x7f0140117230 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:27.066 DEBUG:teuthology.misc:8 of 8 OSDs are up 2019-12-05T05:03:27.066 INFO:tasks.ceph:Creating RBD pool 2019-12-05T05:03:27.066 INFO:teuthology.orchestra.run.smithi042:> sudo ceph --cluster ceph osd pool create rbd 8 2019-12-05T05:03:27.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:27.172+0000 7f44948df700 1 Processor -- start 2019-12-05T05:03:27.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:27.172+0000 7f44948df700 1 -- start start 2019-12-05T05:03:27.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:27.172+0000 7f44948df700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f449009f550 0x7f4490144630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:27.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:27.172+0000 7f44948df700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f449014cf50 0x7f449014f390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:27.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:27.172+0000 7f44948df700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f449007f080 con 0x7f44900a9a00 2019-12-05T05:03:27.184 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:27.176+0000 7f44948df700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44900a8f80 con 0x7f449014cf50 2019-12-05T05:03:27.184 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:27.176+0000 7f44948df700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44900a9220 con 0x7f449009f550 2019-12-05T05:03:27.184 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:27.176+0000 7f448f59e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f449014cf50 0x7f449014f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:27.184 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:27.176+0000 7f448ed9d700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f449009f550 0x7f4490144630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:27.184 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:27.176+0000 7f448e59c700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f44900a9a00 0x7f449007e880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:34190/0 (socket says 172.21.15.42:34190) 2019-12-05T05:03:27.184 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:27.176+0000 7f448f59e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f449014cf50 0x7f449014f390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:52946/0 (socket says 172.21.15.42:52946) 2019-12-05T05:03:27.184 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:27.176+0000 7f448e59c700 1 -- 172.21.15.42:0/1746094940 learned_addr learned my addr 172.21.15.42:0/1746094940 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:03:27.185 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:27.176+0000 7f448dd9b700 1 -- 172.21.15.42:0/1746094940 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2345181541 0 0) 0x7f449007f080 con 0x7f44900a9a00 2019-12-05T05:03:27.185 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:27.176+0000 7f448f59e700 1 -- 172.21.15.42:0/1746094940 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f449009f550 msgr2=0x7f4490144630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:03:27.185 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:27.176+0000 7f448f59e700 1 --2- 172.21.15.42:0/1746094940 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f449009f550 0x7f4490144630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:03:27.185 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:27.176+0000 7f448f59e700 1 -- 172.21.15.42:0/1746094940 >> v1:172.21.15.40:6789/0 conn(0x7f44900a9a00 legacy=0x7f449007e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:27.185 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:27.176+0000 7f448f59e700 1 -- 172.21.15.42:0/1746094940 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44900a1000 con 0x7f449014cf50 2019-12-05T05:03:27.186 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:27.176+0000 7f448f59e700 1 --2- 172.21.15.42:0/1746094940 >> v2:172.21.15.42:3300/0 conn(0x7f449014cf50 0x7f449014f390 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f44900a9140 tx=0x7f448000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:27.186 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:27.176+0000 7f448f59e700 1 --2- 172.21.15.42:0/1746094940 >> v2:172.21.15.42:3300/0 conn(0x7f449014cf50 0x7f449014f390 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f44900a9140 tx=0x7f448000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:27.186 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:27.176+0000 7f448f59e700 1 --2- 172.21.15.42:0/1746094940 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f449014cf50 0x7f449014f390 secure :-1 s=READY pgs=41 cs=0 l=1 rx=0x7f44900a9140 tx=0x7f448000fc70).ready entity=mon.1 client_cookie=ba07e53950977b22 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:27.186 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:27.176+0000 7f448f59e700 1 --2- 172.21.15.42:0/1746094940 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f449014cf50 0x7f449014f390 secure :-1 s=THROTTLE_DONE pgs=41 cs=0 l=1 rx=0x7f44900a9140 tx=0x7f448000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:27.186 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:27.176+0000 7f448f59e700 1 --2- 172.21.15.42:0/1746094940 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f449014cf50 0x7f449014f390 secure :-1 s=THROTTLE_DONE pgs=41 cs=0 l=1 rx=0x7f44900a9140 tx=0x7f448000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:27.187 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:27.176+0000 7f448f59e700 1 --2- 172.21.15.42:0/1746094940 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f449014cf50 0x7f449014f390 secure :-1 s=THROTTLE_DONE pgs=41 cs=0 l=1 rx=0x7f44900a9140 tx=0x7f448000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:27.187 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:27.176+0000 7f448dd9b700 1 -- 172.21.15.42:0/1746094940 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f44800112b0 con 0x7f449014cf50 2019-12-05T05:03:27.187 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:27.176+0000 7f448dd9b700 1 -- 172.21.15.42:0/1746094940 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4480008ad0 con 0x7f449014cf50 2019-12-05T05:03:27.187 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:27.176+0000 7f448dd9b700 1 -- 172.21.15.42:0/1746094940 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4480011890 con 0x7f449014cf50 2019-12-05T05:03:27.187 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:27.176+0000 7f44948df700 1 -- 172.21.15.42:0/1746094940 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f449014cf50 msgr2=0x7f449014f390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:27.187 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:27.176+0000 7f44948df700 1 --2- 172.21.15.42:0/1746094940 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f449014cf50 0x7f449014f390 secure :-1 s=READY pgs=41 cs=0 l=1 rx=0x7f44900a9140 tx=0x7f448000fc70).stop 2019-12-05T05:03:27.188 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:27.176+0000 7f44948df700 1 -- 172.21.15.42:0/1746094940 shutdown_connections 2019-12-05T05:03:27.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:27.176+0000 7f44948df700 1 --2- 172.21.15.42:0/1746094940 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f449014cf50 0x7f449014f390 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:27.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:27.176+0000 7f44948df700 1 --2- 172.21.15.42:0/1746094940 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f449009f550 0x7f4490144630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:03:27.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:27.176+0000 7f44948df700 1 -- 172.21.15.42:0/1746094940 shutdown_connections 2019-12-05T05:03:27.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:27.176+0000 7f44948df700 1 -- 172.21.15.42:0/1746094940 wait complete. 2019-12-05T05:03:27.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:27.176+0000 7f44948df700 1 -- 172.21.15.42:0/1746094940 >> 172.21.15.42:0/1746094940 conn(0x7f44900a9ef0 msgr2=0x7f44900aa320 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:27.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:27.180+0000 7f44948df700 1 Processor -- start 2019-12-05T05:03:27.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:27.180+0000 7f44948df700 1 -- start start 2019-12-05T05:03:27.193 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:27.180+0000 7f44948df700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f449008fd60 0x7f44900a3d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:27.193 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:27.180+0000 7f44948df700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4490138990 con 0x7f44900a2320 2019-12-05T05:03:27.193 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:27.180+0000 7f44948df700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4490139bd0 con 0x7f44900a14c0 2019-12-05T05:03:27.193 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:27.180+0000 7f44948df700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44900a42e0 con 0x7f449008fd60 2019-12-05T05:03:27.193 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:27.180+0000 7f448e59c700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f44900a14c0 0x7f449007dbf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:52988/0 (socket says 172.21.15.42:52988) 2019-12-05T05:03:27.193 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:27.180+0000 7f448e59c700 1 -- 172.21.15.42:0/2551608457 learned_addr learned my addr 172.21.15.42:0/2551608457 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:03:27.194 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:27.180+0000 7f448f59e700 1 --2- 172.21.15.42:0/2551608457 >> v2:172.21.15.40:3300/0 conn(0x7f449008fd60 0x7f44900a3d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:27.194 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:27.180+0000 7f44877fe700 1 -- 172.21.15.42:0/2551608457 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3848688531 0 0) 0x7f4490139bd0 con 0x7f44900a14c0 2019-12-05T05:03:27.194 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:27.180+0000 7f44877fe700 1 -- 172.21.15.42:0/2551608457 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44740030b0 con 0x7f44900a14c0 2019-12-05T05:03:27.194 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:27.180+0000 7f448f59e700 1 -- 172.21.15.42:0/2551608457 >> v1:172.21.15.42:6789/0 conn(0x7f44900a14c0 legacy=0x7f449007dbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:27.194 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:27.180+0000 7f448f59e700 1 -- 172.21.15.42:0/2551608457 >> v1:172.21.15.40:6789/0 conn(0x7f44900a2320 legacy=0x7f449012cf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:27.194 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:27.180+0000 7f448f59e700 1 -- 172.21.15.42:0/2551608457 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f449013ae10 con 0x7f449008fd60 2019-12-05T05:03:27.194 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:27.180+0000 7f448f59e700 1 --2- 172.21.15.42:0/2551608457 >> v2:172.21.15.40:3300/0 conn(0x7f449008fd60 0x7f44900a3d90 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f44880033f0 tx=0x7f4488007aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:27.194 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:27.180+0000 7f448f59e700 1 --2- 172.21.15.42:0/2551608457 >> v2:172.21.15.40:3300/0 conn(0x7f449008fd60 0x7f44900a3d90 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f44880033f0 tx=0x7f4488007aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:27.195 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:27.180+0000 7f448f59e700 1 --2- 172.21.15.42:0/2551608457 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f449008fd60 0x7f44900a3d90 secure :-1 s=READY pgs=52 cs=0 l=1 rx=0x7f44880033f0 tx=0x7f4488007aa0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:27.195 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:27.180+0000 7f448f59e700 1 --2- 172.21.15.42:0/2551608457 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f449008fd60 0x7f44900a3d90 secure :-1 s=THROTTLE_DONE pgs=52 cs=0 l=1 rx=0x7f44880033f0 tx=0x7f4488007aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:27.195 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:27.180+0000 7f44877fe700 1 -- 172.21.15.42:0/2551608457 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4480007150 con 0x7f449008fd60 2019-12-05T05:03:27.195 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:27.180+0000 7f448f59e700 1 --2- 172.21.15.42:0/2551608457 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f449008fd60 0x7f44900a3d90 secure :-1 s=THROTTLE_DONE pgs=52 cs=0 l=1 rx=0x7f44880033f0 tx=0x7f4488007aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:27.195 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:27.180+0000 7f44877fe700 1 -- 172.21.15.42:0/2551608457 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44800114b0 con 0x7f449008fd60 2019-12-05T05:03:27.195 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:27.180+0000 7f448f59e700 1 --2- 172.21.15.42:0/2551608457 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f449008fd60 0x7f44900a3d90 secure :-1 s=THROTTLE_DONE pgs=52 cs=0 l=1 rx=0x7f44880033f0 tx=0x7f4488007aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:27.196 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:27.180+0000 7f44948df700 1 -- 172.21.15.42:0/2551608457 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4490138bc0 con 0x7f449008fd60 2019-12-05T05:03:27.196 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:27.184+0000 7f44877fe700 1 -- 172.21.15.42:0/2551608457 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4480008cc0 con 0x7f449008fd60 2019-12-05T05:03:27.196 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:27.184+0000 7f44948df700 1 -- 172.21.15.42:0/2551608457 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4490139090 con 0x7f449008fd60 2019-12-05T05:03:27.196 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:27.184+0000 7f448f59e700 1 --2- 172.21.15.42:0/2551608457 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f449008fd60 0x7f44900a3d90 secure :-1 s=THROTTLE_DONE pgs=52 cs=0 l=1 rx=0x7f44880033f0 tx=0x7f4488007aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:27.196 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:27.184+0000 7f44877fe700 1 -- 172.21.15.42:0/2551608457 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f4480005540 con 0x7f449008fd60 2019-12-05T05:03:27.196 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:27.184+0000 7f448f59e700 1 --2- 172.21.15.42:0/2551608457 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f449008fd60 0x7f44900a3d90 secure :-1 s=THROTTLE_DONE pgs=52 cs=0 l=1 rx=0x7f44880033f0 tx=0x7f4488007aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:27.196 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:27.184+0000 7f44877fe700 1 --2- 172.21.15.42:0/2551608457 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f447403b230 0x7f447403d740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:27.197 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:27.184+0000 7f44877fe700 1 -- 172.21.15.42:0/2551608457 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 4500+0+0 (secure 0 0 0) 0x7f448004b550 con 0x7f449008fd60 2019-12-05T05:03:27.197 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:27.184+0000 7f448ed9d700 1 --2- 172.21.15.42:0/2551608457 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f447403b230 0x7f447403d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:27.197 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:27.184+0000 7f448ed9d700 1 --2- 172.21.15.42:0/2551608457 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f447403b230 0x7f447403d740 crc :-1 s=READY pgs=40 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:27.198 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:27.188+0000 7f44948df700 1 -- 172.21.15.42:0/2551608457 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f449005d110 con 0x7f449008fd60 2019-12-05T05:03:27.204 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:27.200+0000 7f448f59e700 1 --2- 172.21.15.42:0/2551608457 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f449008fd60 0x7f44900a3d90 secure :-1 s=THROTTLE_DONE pgs=52 cs=0 l=1 rx=0x7f44880033f0 tx=0x7f4488007aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:27.204 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:27.200+0000 7f44877fe700 1 -- 172.21.15.42:0/2551608457 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f44800333a0 con 0x7f449008fd60 2019-12-05T05:03:27.368 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:27.360+0000 7f44948df700 1 -- 172.21.15.42:0/2551608457 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "rbd"} v 0) v1 -- 0x7f4490059a60 con 0x7f449008fd60 2019-12-05T05:03:28.173 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:28.168+0000 7f448f59e700 1 --2- 172.21.15.42:0/2551608457 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f449008fd60 0x7f44900a3d90 secure :-1 s=THROTTLE_DONE pgs=52 cs=0 l=1 rx=0x7f44880033f0 tx=0x7f4488007aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:28.173 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:28.168+0000 7f44877fe700 1 -- 172.21.15.42:0/2551608457 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "rbd"}]=0 pool 'rbd' created v15) v1 ==== 109+0+0 (secure 0 0 0) 0x7f4480005850 con 0x7f449008fd60 2019-12-05T05:03:28.183 INFO:teuthology.orchestra.run.smithi042.stderr:pool 'rbd' created 2019-12-05T05:03:28.187 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:28.180+0000 7f44948df700 1 -- 172.21.15.42:0/2551608457 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f447403b230 msgr2=0x7f447403d740 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:28.187 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:28.180+0000 7f44948df700 1 --2- 172.21.15.42:0/2551608457 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f447403b230 0x7f447403d740 crc :-1 s=READY pgs=40 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:28.188 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:28.180+0000 7f44948df700 1 -- 172.21.15.42:0/2551608457 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f449008fd60 msgr2=0x7f44900a3d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:28.188 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:28.180+0000 7f44948df700 1 --2- 172.21.15.42:0/2551608457 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f449008fd60 0x7f44900a3d90 secure :-1 s=READY pgs=52 cs=0 l=1 rx=0x7f44880033f0 tx=0x7f4488007aa0).stop 2019-12-05T05:03:28.188 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:28.180+0000 7f44948df700 1 -- 172.21.15.42:0/2551608457 shutdown_connections 2019-12-05T05:03:28.188 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:28.180+0000 7f44948df700 1 --2- 172.21.15.42:0/2551608457 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f447403b230 0x7f447403d740 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:28.188 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:28.180+0000 7f44948df700 1 --2- 172.21.15.42:0/2551608457 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f449008fd60 0x7f44900a3d90 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:28.188 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:28.180+0000 7f44948df700 1 -- 172.21.15.42:0/2551608457 shutdown_connections 2019-12-05T05:03:28.188 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:28.180+0000 7f44948df700 1 -- 172.21.15.42:0/2551608457 wait complete. 2019-12-05T05:03:28.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:28.180+0000 7f44948df700 1 -- 172.21.15.42:0/2551608457 >> 172.21.15.42:0/2551608457 conn(0x7f44900a57b0 msgr2=0x7f4490059c80 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:28.198 INFO:teuthology.orchestra.run.smithi042:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2019-12-05T05:03:28.306 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:28.296+0000 7fd41636a700 1 Processor -- start 2019-12-05T05:03:28.306 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:28.296+0000 7fd41636a700 1 -- start start 2019-12-05T05:03:28.307 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:28.296+0000 7fd41636a700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd41009f550 0x7fd410144630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:28.307 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:28.296+0000 7fd41636a700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd41007f080 con 0x7fd4100a9a00 2019-12-05T05:03:28.307 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:28.296+0000 7fd41636a700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4100a1000 con 0x7fd41014cf50 2019-12-05T05:03:28.307 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:28.296+0000 7fd41636a700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4100a8f80 con 0x7fd41009f550 2019-12-05T05:03:28.307 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:28.296+0000 7fd415108700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fd41014cf50 0x7fd41014f390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:52998/0 (socket says 172.21.15.42:52998) 2019-12-05T05:03:28.307 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:28.296+0000 7fd415108700 1 -- 172.21.15.42:0/1164348332 learned_addr learned my addr 172.21.15.42:0/1164348332 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:03:28.308 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:28.296+0000 7fd414907700 1 --2- 172.21.15.42:0/1164348332 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd41009f550 0x7fd410144630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:28.308 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:28.296+0000 7fd40f7fe700 1 -- 172.21.15.42:0/1164348332 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1187474328 0 0) 0x7fd4100a1000 con 0x7fd41014cf50 2019-12-05T05:03:28.308 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:28.296+0000 7fd40f7fe700 1 -- 172.21.15.42:0/1164348332 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3fc0030b0 con 0x7fd41014cf50 2019-12-05T05:03:28.308 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:28.296+0000 7fd414907700 1 -- 172.21.15.42:0/1164348332 >> v1:172.21.15.42:6789/0 conn(0x7fd41014cf50 legacy=0x7fd41014f390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:28.308 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:28.296+0000 7fd414907700 1 -- 172.21.15.42:0/1164348332 >> v1:172.21.15.40:6789/0 conn(0x7fd4100a9a00 legacy=0x7fd41007e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:28.308 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:28.296+0000 7fd414907700 1 -- 172.21.15.42:0/1164348332 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd410144b80 con 0x7fd41009f550 2019-12-05T05:03:28.308 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:28.296+0000 7fd414907700 1 --2- 172.21.15.42:0/1164348332 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd41009f550 0x7fd410144630 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fd3f8001110 tx=0x7fd3f8001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:28.309 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:28.296+0000 7fd414907700 1 --2- 172.21.15.42:0/1164348332 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd41009f550 0x7fd410144630 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fd3f8001110 tx=0x7fd3f8001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:28.309 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:28.296+0000 7fd414907700 1 --2- 172.21.15.42:0/1164348332 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd41009f550 0x7fd410144630 secure :-1 s=READY pgs=25 cs=0 l=1 rx=0x7fd3f8001110 tx=0x7fd3f8001150).ready entity=mon.2 client_cookie=a7fc5ab831611ab0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:28.309 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:28.296+0000 7fd414907700 1 --2- 172.21.15.42:0/1164348332 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd41009f550 0x7fd410144630 secure :-1 s=THROTTLE_DONE pgs=25 cs=0 l=1 rx=0x7fd3f8001110 tx=0x7fd3f8001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:28.309 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:28.296+0000 7fd414907700 1 --2- 172.21.15.42:0/1164348332 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd41009f550 0x7fd410144630 secure :-1 s=THROTTLE_DONE pgs=25 cs=0 l=1 rx=0x7fd3f8001110 tx=0x7fd3f8001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:28.309 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:28.296+0000 7fd40f7fe700 1 -- 172.21.15.42:0/1164348332 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd400010870 con 0x7fd41009f550 2019-12-05T05:03:28.309 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:28.296+0000 7fd414907700 1 --2- 172.21.15.42:0/1164348332 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd41009f550 0x7fd410144630 secure :-1 s=THROTTLE_DONE pgs=25 cs=0 l=1 rx=0x7fd3f8001110 tx=0x7fd3f8001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:28.309 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:28.300+0000 7fd40f7fe700 1 -- 172.21.15.42:0/1164348332 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd400008b50 con 0x7fd41009f550 2019-12-05T05:03:28.310 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:28.300+0000 7fd40f7fe700 1 -- 172.21.15.42:0/1164348332 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd4000110c0 con 0x7fd41009f550 2019-12-05T05:03:28.310 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:28.300+0000 7fd41636a700 1 -- 172.21.15.42:0/1164348332 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd41009f550 msgr2=0x7fd410144630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:28.310 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:28.300+0000 7fd41636a700 1 --2- 172.21.15.42:0/1164348332 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd41009f550 0x7fd410144630 secure :-1 s=READY pgs=25 cs=0 l=1 rx=0x7fd3f8001110 tx=0x7fd3f8001150).stop 2019-12-05T05:03:28.310 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:28.300+0000 7fd41636a700 1 -- 172.21.15.42:0/1164348332 shutdown_connections 2019-12-05T05:03:28.310 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:28.300+0000 7fd41636a700 1 --2- 172.21.15.42:0/1164348332 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd41009f550 0x7fd410144630 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:28.310 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:28.300+0000 7fd41636a700 1 -- 172.21.15.42:0/1164348332 shutdown_connections 2019-12-05T05:03:28.310 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:28.300+0000 7fd41636a700 1 -- 172.21.15.42:0/1164348332 wait complete. 2019-12-05T05:03:28.311 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:28.300+0000 7fd41636a700 1 -- 172.21.15.42:0/1164348332 >> 172.21.15.42:0/1164348332 conn(0x7fd4100a9ef0 msgr2=0x7fd4100aa320 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:28.314 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:28.304+0000 7fd41636a700 1 Processor -- start 2019-12-05T05:03:28.314 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:28.304+0000 7fd41636a700 1 -- start start 2019-12-05T05:03:28.314 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:28.304+0000 7fd41636a700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd41008eff0 0x7fd4101457e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:28.314 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:28.304+0000 7fd41636a700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd410147790 con 0x7fd410147360 2019-12-05T05:03:28.315 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:28.304+0000 7fd41636a700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd41007e170 con 0x7fd410125290 2019-12-05T05:03:28.315 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:28.304+0000 7fd41636a700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd410145f20 con 0x7fd41008eff0 2019-12-05T05:03:28.315 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:28.304+0000 7fd414907700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd41008eff0 0x7fd4101457e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:28.315 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:28.304+0000 7fd414907700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd41008eff0 0x7fd4101457e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:52964/0 (socket says 172.21.15.42:52964) 2019-12-05T05:03:28.321 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:28.304+0000 7fd414907700 1 -- 172.21.15.42:0/2130424680 learned_addr learned my addr 172.21.15.42:0/2130424680 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:03:28.321 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:28.304+0000 7fd414907700 1 -- 172.21.15.42:0/2130424680 >> v1:172.21.15.42:6789/0 conn(0x7fd410125290 legacy=0x7fd41012c6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:28.321 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:28.304+0000 7fd414907700 1 -- 172.21.15.42:0/2130424680 >> v1:172.21.15.40:6789/0 conn(0x7fd410147360 legacy=0x7fd41007d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:28.322 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:28.304+0000 7fd414907700 1 -- 172.21.15.42:0/2130424680 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4101256c0 con 0x7fd41008eff0 2019-12-05T05:03:28.322 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:28.304+0000 7fd414907700 1 --2- 172.21.15.42:0/2130424680 >> v2:172.21.15.42:3300/0 conn(0x7fd41008eff0 0x7fd4101457e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fd3f8001110 tx=0x7fd400010c60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:28.322 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:28.304+0000 7fd414907700 1 --2- 172.21.15.42:0/2130424680 >> v2:172.21.15.42:3300/0 conn(0x7fd41008eff0 0x7fd4101457e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fd3f8001110 tx=0x7fd400010c60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:28.322 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:28.304+0000 7fd414907700 1 --2- 172.21.15.42:0/2130424680 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd41008eff0 0x7fd4101457e0 secure :-1 s=READY pgs=45 cs=0 l=1 rx=0x7fd3f8001110 tx=0x7fd400010c60).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:28.322 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:28.304+0000 7fd414907700 1 --2- 172.21.15.42:0/2130424680 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd41008eff0 0x7fd4101457e0 secure :-1 s=THROTTLE_DONE pgs=45 cs=0 l=1 rx=0x7fd3f8001110 tx=0x7fd400010c60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:28.322 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:28.304+0000 7fd40d7fa700 1 -- 172.21.15.42:0/2130424680 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd400005ab0 con 0x7fd41008eff0 2019-12-05T05:03:28.322 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:28.304+0000 7fd414907700 1 --2- 172.21.15.42:0/2130424680 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd41008eff0 0x7fd4101457e0 secure :-1 s=THROTTLE_DONE pgs=45 cs=0 l=1 rx=0x7fd3f8001110 tx=0x7fd400010c60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:28.323 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:28.304+0000 7fd40d7fa700 1 -- 172.21.15.42:0/2130424680 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd400010a70 con 0x7fd41008eff0 2019-12-05T05:03:28.323 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:28.304+0000 7fd414907700 1 --2- 172.21.15.42:0/2130424680 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd41008eff0 0x7fd4101457e0 secure :-1 s=THROTTLE_DONE pgs=45 cs=0 l=1 rx=0x7fd3f8001110 tx=0x7fd400010c60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:28.323 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:28.304+0000 7fd41636a700 1 -- 172.21.15.42:0/2130424680 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd410125930 con 0x7fd41008eff0 2019-12-05T05:03:28.323 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:28.304+0000 7fd40d7fa700 1 -- 172.21.15.42:0/2130424680 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd40000f680 con 0x7fd41008eff0 2019-12-05T05:03:28.323 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:28.304+0000 7fd41636a700 1 -- 172.21.15.42:0/2130424680 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd410059700 con 0x7fd41008eff0 2019-12-05T05:03:28.323 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:28.308+0000 7fd414907700 1 --2- 172.21.15.42:0/2130424680 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd41008eff0 0x7fd4101457e0 secure :-1 s=THROTTLE_DONE pgs=45 cs=0 l=1 rx=0x7fd3f8001110 tx=0x7fd400010c60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:28.323 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:28.308+0000 7fd40d7fa700 1 -- 172.21.15.42:0/2130424680 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fd400011390 con 0x7fd41008eff0 2019-12-05T05:03:28.324 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:28.308+0000 7fd414907700 1 --2- 172.21.15.42:0/2130424680 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd41008eff0 0x7fd4101457e0 secure :-1 s=THROTTLE_DONE pgs=45 cs=0 l=1 rx=0x7fd3f8001110 tx=0x7fd400010c60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:28.324 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:28.308+0000 7fd40d7fa700 1 --2- 172.21.15.42:0/2130424680 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd3f403acc0 0x7fd3f403d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:28.324 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:28.308+0000 7fd40d7fa700 1 -- 172.21.15.42:0/2130424680 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4842+0+0 (secure 0 0 0) 0x7fd40004b1e0 con 0x7fd41008eff0 2019-12-05T05:03:28.324 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:28.308+0000 7fd40ffff700 1 --2- 172.21.15.42:0/2130424680 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd3f403acc0 0x7fd3f403d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:28.324 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:28.308+0000 7fd40ffff700 1 --2- 172.21.15.42:0/2130424680 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd3f403acc0 0x7fd3f403d1d0 crc :-1 s=READY pgs=41 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:28.325 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:28.308+0000 7fd41636a700 1 -- 172.21.15.42:0/2130424680 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd41005d110 con 0x7fd41008eff0 2019-12-05T05:03:28.325 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:28.316+0000 7fd414907700 1 --2- 172.21.15.42:0/2130424680 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd41008eff0 0x7fd4101457e0 secure :-1 s=THROTTLE_DONE pgs=45 cs=0 l=1 rx=0x7fd3f8001110 tx=0x7fd400010c60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:28.325 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:28.316+0000 7fd40d7fa700 1 -- 172.21.15.42:0/2130424680 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fd40000f9a0 con 0x7fd41008eff0 2019-12-05T05:03:28.486 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:28.480+0000 7fd41636a700 1 -- 172.21.15.42:0/2130424680 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "osd pool application enable", "app": "rbd", "pool": "rbd", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fd4100a77a0 con 0x7fd41008eff0 2019-12-05T05:03:29.178 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.172+0000 7fd414907700 1 --2- 172.21.15.42:0/2130424680 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd41008eff0 0x7fd4101457e0 secure :-1 s=THROTTLE_DONE pgs=45 cs=0 l=1 rx=0x7fd3f8001110 tx=0x7fd400010c60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:29.178 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.172+0000 7fd40d7fa700 1 -- 172.21.15.42:0/2130424680 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool application enable", "app": "rbd", "pool": "rbd", "yes_i_really_mean_it": true}]=0 enabled application 'rbd' on pool 'rbd' v16) v1 ==== 173+0+0 (secure 0 0 0) 0x7fd4000324a0 con 0x7fd41008eff0 2019-12-05T05:03:29.202 INFO:teuthology.orchestra.run.smithi042.stderr:enabled application 'rbd' on pool 'rbd' 2019-12-05T05:03:29.206 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.200+0000 7fd41636a700 1 -- 172.21.15.42:0/2130424680 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd3f403acc0 msgr2=0x7fd3f403d1d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:29.206 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.200+0000 7fd41636a700 1 --2- 172.21.15.42:0/2130424680 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd3f403acc0 0x7fd3f403d1d0 crc :-1 s=READY pgs=41 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:29.206 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.200+0000 7fd41636a700 1 -- 172.21.15.42:0/2130424680 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd41008eff0 msgr2=0x7fd4101457e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:29.206 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.200+0000 7fd41636a700 1 --2- 172.21.15.42:0/2130424680 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd41008eff0 0x7fd4101457e0 secure :-1 s=READY pgs=45 cs=0 l=1 rx=0x7fd3f8001110 tx=0x7fd400010c60).stop 2019-12-05T05:03:29.206 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.200+0000 7fd41636a700 1 -- 172.21.15.42:0/2130424680 shutdown_connections 2019-12-05T05:03:29.207 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.200+0000 7fd41636a700 1 --2- 172.21.15.42:0/2130424680 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd3f403acc0 0x7fd3f403d1d0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:29.207 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.200+0000 7fd41636a700 1 --2- 172.21.15.42:0/2130424680 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd41008eff0 0x7fd4101457e0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:29.207 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.200+0000 7fd41636a700 1 -- 172.21.15.42:0/2130424680 shutdown_connections 2019-12-05T05:03:29.207 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.200+0000 7fd41636a700 1 -- 172.21.15.42:0/2130424680 wait complete. 2019-12-05T05:03:29.207 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.200+0000 7fd41636a700 1 -- 172.21.15.42:0/2130424680 >> 172.21.15.42:0/2130424680 conn(0x7fd4100a8b30 msgr2=0x7fd4101501f0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:29.216 INFO:tasks.ceph:Setting up CephFS filesystem... 2019-12-05T05:03:29.216 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2019-12-05T05:03:29.330 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.304+0000 7fd5eaa68700 1 Processor -- start 2019-12-05T05:03:29.331 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.304+0000 7fd5eaa68700 1 -- start start 2019-12-05T05:03:29.331 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.304+0000 7fd5eaa68700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd5e40a7740 0x7fd5e408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:29.331 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.304+0000 7fd5eaa68700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd5e407e720 0x7fd5e407eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:29.331 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.304+0000 7fd5eaa68700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd5e409f2e0 0x7fd5e409f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:29.331 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.304+0000 7fd5eaa68700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5e40a0b00 con 0x7fd5e409f2e0 2019-12-05T05:03:29.331 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.304+0000 7fd5eaa68700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5e408e5b0 con 0x7fd5e407e720 2019-12-05T05:03:29.332 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.304+0000 7fd5eaa68700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5e408e850 con 0x7fd5e40a7740 2019-12-05T05:03:29.332 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.304+0000 7fd5e8804700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd5e407e720 0x7fd5e407eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:29.332 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.308+0000 7fd5e8804700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd5e407e720 0x7fd5e407eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:52972/0 (socket says 172.21.15.42:52972) 2019-12-05T05:03:29.332 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.308+0000 7fd5e8804700 1 -- 172.21.15.42:0/2719060087 learned_addr learned my addr 172.21.15.42:0/2719060087 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:03:29.332 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.308+0000 7fd5e9806700 1 --2- 172.21.15.42:0/2719060087 >> v2:172.21.15.40:3300/0 conn(0x7fd5e409f2e0 0x7fd5e409f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:29.333 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.308+0000 7fd5e9005700 1 --2- 172.21.15.42:0/2719060087 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd5e40a7740 0x7fd5e408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:29.333 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.308+0000 7fd5e8804700 1 -- 172.21.15.42:0/2719060087 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd5e40a7740 msgr2=0x7fd5e408df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:03:29.333 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.308+0000 7fd5e8804700 1 --2- 172.21.15.42:0/2719060087 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd5e40a7740 0x7fd5e408df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:03:29.333 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.308+0000 7fd5e8804700 1 -- 172.21.15.42:0/2719060087 >> v2:172.21.15.40:3300/0 conn(0x7fd5e409f2e0 msgr2=0x7fd5e409f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:03:29.333 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.308+0000 7fd5e8804700 1 --2- 172.21.15.42:0/2719060087 >> v2:172.21.15.40:3300/0 conn(0x7fd5e409f2e0 0x7fd5e409f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:03:29.333 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.308+0000 7fd5e8804700 1 -- 172.21.15.42:0/2719060087 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5e407f0a0 con 0x7fd5e407e720 2019-12-05T05:03:29.333 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.308+0000 7fd5e9005700 1 --2- 172.21.15.42:0/2719060087 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd5e40a7740 0x7fd5e408df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T05:03:29.334 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.308+0000 7fd5e8804700 1 --2- 172.21.15.42:0/2719060087 >> v2:172.21.15.42:3300/0 conn(0x7fd5e407e720 0x7fd5e407eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fd5e408e770 tx=0x7fd5d400fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:29.334 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.308+0000 7fd5e8804700 1 --2- 172.21.15.42:0/2719060087 >> v2:172.21.15.42:3300/0 conn(0x7fd5e407e720 0x7fd5e407eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fd5e408e770 tx=0x7fd5d400fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:29.334 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.308+0000 7fd5e8804700 1 --2- 172.21.15.42:0/2719060087 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd5e407e720 0x7fd5e407eb50 secure :-1 s=READY pgs=46 cs=0 l=1 rx=0x7fd5e408e770 tx=0x7fd5d400fdb0).ready entity=mon.1 client_cookie=117c4c7ea9132ba7 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:29.334 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.308+0000 7fd5e8804700 1 --2- 172.21.15.42:0/2719060087 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd5e407e720 0x7fd5e407eb50 secure :-1 s=THROTTLE_DONE pgs=46 cs=0 l=1 rx=0x7fd5e408e770 tx=0x7fd5d400fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:29.334 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.308+0000 7fd5dbfff700 1 -- 172.21.15.42:0/2719060087 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd5d4008c90 con 0x7fd5e407e720 2019-12-05T05:03:29.334 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.308+0000 7fd5e8804700 1 --2- 172.21.15.42:0/2719060087 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd5e407e720 0x7fd5e407eb50 secure :-1 s=THROTTLE_DONE pgs=46 cs=0 l=1 rx=0x7fd5e408e770 tx=0x7fd5d400fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:29.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.308+0000 7fd5e8804700 1 --2- 172.21.15.42:0/2719060087 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd5e407e720 0x7fd5e407eb50 secure :-1 s=THROTTLE_DONE pgs=46 cs=0 l=1 rx=0x7fd5e408e770 tx=0x7fd5d400fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:29.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.308+0000 7fd5dbfff700 1 -- 172.21.15.42:0/2719060087 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5d40088b0 con 0x7fd5e407e720 2019-12-05T05:03:29.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.308+0000 7fd5dbfff700 1 -- 172.21.15.42:0/2719060087 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd5d4011b50 con 0x7fd5e407e720 2019-12-05T05:03:29.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.308+0000 7fd5eaa68700 1 -- 172.21.15.42:0/2719060087 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd5e407e720 msgr2=0x7fd5e407eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:29.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.308+0000 7fd5eaa68700 1 --2- 172.21.15.42:0/2719060087 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd5e407e720 0x7fd5e407eb50 secure :-1 s=READY pgs=46 cs=0 l=1 rx=0x7fd5e408e770 tx=0x7fd5d400fdb0).stop 2019-12-05T05:03:29.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.308+0000 7fd5eaa68700 1 -- 172.21.15.42:0/2719060087 shutdown_connections 2019-12-05T05:03:29.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.308+0000 7fd5eaa68700 1 --2- 172.21.15.42:0/2719060087 >> v2:172.21.15.40:3300/0 conn(0x7fd5e409f2e0 0x7fd5e409f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:03:29.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.308+0000 7fd5eaa68700 1 --2- 172.21.15.42:0/2719060087 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd5e40a7740 0x7fd5e408df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:03:29.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.308+0000 7fd5eaa68700 1 --2- 172.21.15.42:0/2719060087 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd5e407e720 0x7fd5e407eb50 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:29.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.308+0000 7fd5eaa68700 1 -- 172.21.15.42:0/2719060087 shutdown_connections 2019-12-05T05:03:29.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.308+0000 7fd5eaa68700 1 -- 172.21.15.42:0/2719060087 wait complete. 2019-12-05T05:03:29.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.308+0000 7fd5eaa68700 1 -- 172.21.15.42:0/2719060087 >> 172.21.15.42:0/2719060087 conn(0x7fd5e40aabe0 msgr2=0x7fd5e40a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:29.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.312+0000 7fd5eaa68700 1 Processor -- start 2019-12-05T05:03:29.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.312+0000 7fd5eaa68700 1 -- start start 2019-12-05T05:03:29.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.312+0000 7fd5eaa68700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd5e40a9ad0 0x7fd5e4119df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:29.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.312+0000 7fd5eaa68700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5e41195f0 con 0x7fd5e40a53d0 2019-12-05T05:03:29.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.312+0000 7fd5eaa68700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5e40a5800 con 0x7fd5e4134020 2019-12-05T05:03:29.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.312+0000 7fd5eaa68700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5e4133ae0 con 0x7fd5e40a9ad0 2019-12-05T05:03:29.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.312+0000 7fd5e9806700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fd5e4134020 0x7fd5e41240c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:53018/0 (socket says 172.21.15.42:53018) 2019-12-05T05:03:29.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.312+0000 7fd5e9806700 1 -- 172.21.15.42:0/1275775870 learned_addr learned my addr 172.21.15.42:0/1275775870 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:03:29.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.312+0000 7fd5e9005700 1 --2- 172.21.15.42:0/1275775870 >> v2:172.21.15.40:3300/0 conn(0x7fd5e40a9ad0 0x7fd5e4119df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:29.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.312+0000 7fd5d9ffb700 1 -- 172.21.15.42:0/1275775870 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 324261670 0 0) 0x7fd5e40a5800 con 0x7fd5e4134020 2019-12-05T05:03:29.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.312+0000 7fd5d9ffb700 1 -- 172.21.15.42:0/1275775870 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5c40030b0 con 0x7fd5e4134020 2019-12-05T05:03:29.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.316+0000 7fd5d9ffb700 1 -- 172.21.15.42:0/1275775870 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3390121689 0 0) 0x7fd5e41195f0 con 0x7fd5e40a53d0 2019-12-05T05:03:29.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.316+0000 7fd5d9ffb700 1 -- 172.21.15.42:0/1275775870 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5e40a5800 con 0x7fd5e40a53d0 2019-12-05T05:03:29.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.316+0000 7fd5d9ffb700 1 -- 172.21.15.42:0/1275775870 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 3017156331 0 0) 0x7fd5c40030b0 con 0x7fd5e4134020 2019-12-05T05:03:29.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.316+0000 7fd5e9005700 1 -- 172.21.15.42:0/1275775870 >> v1:172.21.15.42:6789/0 conn(0x7fd5e4134020 legacy=0x7fd5e41240c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:29.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.316+0000 7fd5e9005700 1 -- 172.21.15.42:0/1275775870 >> v1:172.21.15.40:6789/0 conn(0x7fd5e40a53d0 legacy=0x7fd5e41238c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:29.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.316+0000 7fd5e9005700 1 -- 172.21.15.42:0/1275775870 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5e4133730 con 0x7fd5e40a9ad0 2019-12-05T05:03:29.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.316+0000 7fd5e9005700 1 --2- 172.21.15.42:0/1275775870 >> v2:172.21.15.40:3300/0 conn(0x7fd5e40a9ad0 0x7fd5e4119df0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fd5dc007fe0 tx=0x7fd5dc0032f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:29.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.316+0000 7fd5e9005700 1 --2- 172.21.15.42:0/1275775870 >> v2:172.21.15.40:3300/0 conn(0x7fd5e40a9ad0 0x7fd5e4119df0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fd5dc007fe0 tx=0x7fd5dc0032f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:29.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.316+0000 7fd5e9005700 1 --2- 172.21.15.42:0/1275775870 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd5e40a9ad0 0x7fd5e4119df0 secure :-1 s=READY pgs=57 cs=0 l=1 rx=0x7fd5dc007fe0 tx=0x7fd5dc0032f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:29.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.316+0000 7fd5e9005700 1 --2- 172.21.15.42:0/1275775870 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd5e40a9ad0 0x7fd5e4119df0 secure :-1 s=THROTTLE_DONE pgs=57 cs=0 l=1 rx=0x7fd5dc007fe0 tx=0x7fd5dc0032f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:29.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.316+0000 7fd5d9ffb700 1 -- 172.21.15.42:0/1275775870 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd5dc001810 con 0x7fd5e40a9ad0 2019-12-05T05:03:29.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.316+0000 7fd5e9005700 1 --2- 172.21.15.42:0/1275775870 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd5e40a9ad0 0x7fd5e4119df0 secure :-1 s=THROTTLE_DONE pgs=57 cs=0 l=1 rx=0x7fd5dc007fe0 tx=0x7fd5dc0032f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:29.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.316+0000 7fd5d9ffb700 1 -- 172.21.15.42:0/1275775870 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5dc001620 con 0x7fd5e40a9ad0 2019-12-05T05:03:29.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.316+0000 7fd5e9005700 1 --2- 172.21.15.42:0/1275775870 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd5e40a9ad0 0x7fd5e4119df0 secure :-1 s=THROTTLE_DONE pgs=57 cs=0 l=1 rx=0x7fd5dc007fe0 tx=0x7fd5dc0032f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:29.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.316+0000 7fd5d9ffb700 1 -- 172.21.15.42:0/1275775870 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd5e0005b20 con 0x7fd5e40a9ad0 2019-12-05T05:03:29.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.316+0000 7fd5eaa68700 1 -- 172.21.15.42:0/1275775870 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5e4119aa0 con 0x7fd5e40a9ad0 2019-12-05T05:03:29.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.316+0000 7fd5eaa68700 1 -- 172.21.15.42:0/1275775870 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5e4134450 con 0x7fd5e40a9ad0 2019-12-05T05:03:29.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.316+0000 7fd5e9005700 1 --2- 172.21.15.42:0/1275775870 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd5e40a9ad0 0x7fd5e4119df0 secure :-1 s=THROTTLE_DONE pgs=57 cs=0 l=1 rx=0x7fd5dc007fe0 tx=0x7fd5dc0032f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:29.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.316+0000 7fd5d9ffb700 1 -- 172.21.15.42:0/1275775870 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fd5e0007bb0 con 0x7fd5e40a9ad0 2019-12-05T05:03:29.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.316+0000 7fd5e9005700 1 --2- 172.21.15.42:0/1275775870 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd5e40a9ad0 0x7fd5e4119df0 secure :-1 s=THROTTLE_DONE pgs=57 cs=0 l=1 rx=0x7fd5dc007fe0 tx=0x7fd5dc0032f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:29.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.316+0000 7fd5d9ffb700 1 --2- 172.21.15.42:0/1275775870 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd5c403b400 0x7fd5c403d910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:29.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.316+0000 7fd5d9ffb700 1 -- 172.21.15.42:0/1275775870 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4853+0+0 (secure 0 0 0) 0x7fd5e004a730 con 0x7fd5e40a9ad0 2019-12-05T05:03:29.365 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.320+0000 7fd5e8804700 1 --2- 172.21.15.42:0/1275775870 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd5c403b400 0x7fd5c403d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:29.365 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.320+0000 7fd5e8804700 1 --2- 172.21.15.42:0/1275775870 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd5c403b400 0x7fd5c403d910 crc :-1 s=READY pgs=43 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:29.366 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.320+0000 7fd5eaa68700 1 -- 172.21.15.42:0/1275775870 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5e405d110 con 0x7fd5e40a9ad0 2019-12-05T05:03:29.366 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.328+0000 7fd5e9005700 1 --2- 172.21.15.42:0/1275775870 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd5e40a9ad0 0x7fd5e4119df0 secure :-1 s=THROTTLE_DONE pgs=57 cs=0 l=1 rx=0x7fd5dc007fe0 tx=0x7fd5dc0032f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:29.366 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.328+0000 7fd5d9ffb700 1 -- 172.21.15.42:0/1275775870 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fd5e0005d70 con 0x7fd5e40a9ad0 2019-12-05T05:03:29.510 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.504+0000 7fd5eaa68700 1 -- 172.21.15.42:0/1275775870 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd5e405b8c0 con 0x7fd5e40a9ad0 2019-12-05T05:03:29.511 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.504+0000 7fd5e9005700 1 --2- 172.21.15.42:0/1275775870 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd5e40a9ad0 0x7fd5e4119df0 secure :-1 s=THROTTLE_DONE pgs=57 cs=0 l=1 rx=0x7fd5dc007fe0 tx=0x7fd5dc0032f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:29.511 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.504+0000 7fd5d9ffb700 1 -- 172.21.15.42:0/1275775870 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+11778 (secure 0 0 0) 0x7fd5e0031790 con 0x7fd5e40a9ad0 2019-12-05T05:03:29.513 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:03:29.513 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":16,"fsid":"5d836cfc-a83b-4433-9b3e-b9b2f31c9050","created":"2019-12-05T05:03:13.215602+0000","modified":"2019-12-05T05:03:29.169324+0000","last_up_change":"2019-12-05T05:03:22.664783+0000","last_in_change":"2019-12-05T05:03:22.664783+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2019-12-05T05:03:27.394605+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"e3fc49bb-dcbc-43ff-ac7c-9eb0ea0bd257","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6810","nonce":12118},{"type":"v1","addr":"172.21.15.42:6811","nonce":12118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6812","nonce":12118},{"type":"v1","addr":"172.21.15.42:6813","nonce":12118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6816","nonce":12118},{"type":"v1","addr":"172.21.15.42:6817","nonce":12118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6814","nonce":12118},{"type":"v1","addr":"172.21.15.42:6815","nonce":12118}]},"public_addr":"172.21.15.42:6811/12118","cluster_addr":"172.21.15.42:6813/12118","heartbeat_back_addr":"172.21.15.42:6817/12118","heartbeat_front_addr":"172.21.15.42:6815/12118","state":["exists","up"]},{"osd":1,"uuid":"902543fb-f5a2-4321-9c9c-26f9b9ca8629","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6826","nonce":12119},{"type":"v1","addr":"172.21.15.42:6827","nonce":12119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6828","nonce":12119},{"type":"v1","addr":"172.21.15.42:6829","nonce":12119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6832","nonce":12119},{"type":"v1","addr":"172.21.15.42:6833","nonce":12119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6830","nonce":12119},{"type":"v1","addr":"172.21.15.42:6831","nonce":12119}]},"public_addr":"172.21.15.42:6827/12119","cluster_addr":"172.21.15.42:6829/12119","heartbeat_back_addr":"172.21.15.42:6833/12119","heartbeat_front_addr":"172.21.15.42:6831/12119","state":["exists","up"]},{"osd":2,"uuid":"28739ff7-bed7-4b75-8ea5-b57d088f1250","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6802","nonce":12117},{"type":"v1","addr":"172.21.15.42:6803","nonce":12117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6804","nonce":12117},{"type":"v1","addr":"172.21.15.42:6805","nonce":12117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6808","nonce":12117},{"type":"v1","addr":"172.21.15.42:6809","nonce":12117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6806","nonce":12117},{"type":"v1","addr":"172.21.15.42:6807","nonce":12117}]},"public_addr":"172.21.15.42:6803/12117","cluster_addr":"172.21.15.42:6805/12117","heartbeat_back_addr":"172.21.15.42:6809/12117","heartbeat_front_addr":"172.21.15.42:6807/12117","state":["exists","up"]},{"osd":3,"uuid":"1960f66b-2d40-4bb9-a289-ce8cd0c1c142","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6818","nonce":12116},{"type":"v1","addr":"172.21.15.42:6819","nonce":12116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6820","nonce":12116},{"type":"v1","addr":"172.21.15.42:6821","nonce":12116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6824","nonce":12116},{"type":"v1","addr":"172.21.15.42:6825","nonce":12116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6822","nonce":12116},{"type":"v1","addr":"172.21.15.42:6823","nonce":12116}]},"public_addr":"172.21.15.42:6819/12116","cluster_addr":"172.21.15.42:6821/12116","heartbeat_back_addr":"172.21.15.42:6825/12116","heartbeat_front_addr":"172.21.15.42:6823/12116","state":["exists","up"]},{"osd":4,"uuid":"45bbf1bd-7c8a-4a0d-b76b-afa6148a855c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":12488},{"type":"v1","addr":"172.21.15.40:6809","nonce":12488}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":12488},{"type":"v1","addr":"172.21.15.40:6812","nonce":12488}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":12488},{"type":"v1","addr":"172.21.15.40:6825","nonce":12488}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":12488},{"type":"v1","addr":"172.21.15.40:6819","nonce":12488}]},"public_addr":"172.21.15.40:6809/12488","cluster_addr":"172.21.15.40:6812/12488","heartbeat_back_addr":"172.21.15.40:6825/12488","heartbeat_front_addr":"172.21.15.40:6819/12488","state":["exists","up"]},{"osd":5,"uuid":"8a5511cd-925f-4869-a49e-924383e4ce53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":12489},{"type":"v1","addr":"172.21.15.40:6801","nonce":12489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":12489},{"type":"v1","addr":"172.21.15.40:6803","nonce":12489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":12489},{"type":"v1","addr":"172.21.15.40:6807","nonce":12489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":12489},{"type":"v1","addr":"172.21.15.40:6805","nonce":12489}]},"public_addr":"172.21.15.40:6801/12489","cluster_addr":"172.21.15.40:6803/12489","heartbeat_back_addr":"172.21.15.40:6807/12489","heartbeat_front_addr":"172.21.15.40:6805/12489","state":["exists","up"]},{"osd":6,"uuid":"d692c179-379e-446c-8484-a219a9241e04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6811","nonce":12491},{"type":"v1","addr":"172.21.15.40:6814","nonce":12491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6817","nonce":12491},{"type":"v1","addr":"172.21.15.40:6818","nonce":12491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":12491},{"type":"v1","addr":"172.21.15.40:6828","nonce":12491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":12491},{"type":"v1","addr":"172.21.15.40:6824","nonce":12491}]},"public_addr":"172.21.15.40:6814/12491","cluster_addr":"172.21.15.40:6818/12491","heartbeat_back_addr":"172.21.15.40:6828/12491","heartbeat_front_addr":"172.21.15.40:6824/12491","state":["exists","up"]},{"osd":7,"uuid":"79c3c243-190e-4e35-b769-319a2077e7cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6813","nonce":12490},{"type":"v1","addr":"172.21.15.40:6816","nonce":12490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":12490},{"type":"v1","addr":"172.21.15.40:6823","nonce":12490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":12490},{"type":"v1","addr":"172.21.15.40:6831","nonce":12490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6827","nonce":12490},{"type":"v1","addr":"172.21.15.40:6829","nonce":12490}]},"public_addr":"172.21.15.40:6816/12490","cluster_addr":"172.21.15.40:6823/12490","heartbeat_back_addr":"172.21.15.40:6831/12490","heartbeat_front_addr":"172.21.15.40:6829/12490","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T05:03:21.349150+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T05:03:21.644281+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T05:03:21.394040+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T05:03:21.587096+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T05:03:21.656497+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T05:03:21.510547+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T05:03:21.659358+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T05:03:21.628063+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2019-12-05T05:03:29.515 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.508+0000 7fd5eaa68700 1 -- 172.21.15.42:0/1275775870 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd5c403b400 msgr2=0x7fd5c403d910 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:29.516 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.508+0000 7fd5eaa68700 1 --2- 172.21.15.42:0/1275775870 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd5c403b400 0x7fd5c403d910 crc :-1 s=READY pgs=43 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:29.516 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.508+0000 7fd5eaa68700 1 -- 172.21.15.42:0/1275775870 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd5e40a9ad0 msgr2=0x7fd5e4119df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:29.516 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.508+0000 7fd5eaa68700 1 --2- 172.21.15.42:0/1275775870 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd5e40a9ad0 0x7fd5e4119df0 secure :-1 s=READY pgs=57 cs=0 l=1 rx=0x7fd5dc007fe0 tx=0x7fd5dc0032f0).stop 2019-12-05T05:03:29.516 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.508+0000 7fd5eaa68700 1 -- 172.21.15.42:0/1275775870 shutdown_connections 2019-12-05T05:03:29.516 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.508+0000 7fd5eaa68700 1 --2- 172.21.15.42:0/1275775870 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd5c403b400 0x7fd5c403d910 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:29.516 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.508+0000 7fd5eaa68700 1 --2- 172.21.15.42:0/1275775870 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd5e40a9ad0 0x7fd5e4119df0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:29.516 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.508+0000 7fd5eaa68700 1 -- 172.21.15.42:0/1275775870 shutdown_connections 2019-12-05T05:03:29.517 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.508+0000 7fd5eaa68700 1 -- 172.21.15.42:0/1275775870 wait complete. 2019-12-05T05:03:29.517 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.508+0000 7fd5eaa68700 1 -- 172.21.15.42:0/1275775870 >> 172.21.15.42:0/1275775870 conn(0x7fd5e40a9010 msgr2=0x7fd5e407e4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:29.526 INFO:tasks.cephfs.filesystem.ceph_manager:[{u'cache_target_full_ratio_micro': 800000, u'fast_read': False, u'stripe_width': 0, u'flags_names': u'hashpspool,creating', u'tier_of': -1, u'hit_set_grade_decay_rate': 0, u'pg_placement_num': 8, u'use_gmt_hitset': True, u'cache_target_dirty_ratio_micro': 400000, u'create_time': u'2019-12-05T05:03:27.394605+0000', u'snap_epoch': 0, u'erasure_code_profile': u'', u'pg_autoscale_mode': u'off', u'snap_seq': 0, u'expected_num_objects': 0, u'size': 2, u'pg_num_pending': 8, u'auid': 0, u'cache_min_flush_age': 0, u'hit_set_period': 0, u'min_read_recency_for_promote': 0, u'target_max_objects': 0, u'pg_placement_num_target': 8, u'pg_num': 8, u'type': 1, u'grade_table': [], u'pool_name': u'rbd', u'cache_min_evict_age': 0, u'snap_mode': u'selfmanaged', u'pg_num_target': 8, u'cache_mode': u'none', u'min_size': 1, u'cache_target_dirty_high_ratio_micro': 600000, u'object_hash': 2, u'last_pg_merge_meta': {u'ready_epoch': 0, u'source_version': u"0'0", u'source_pgid': u'0.0', u'last_epoch_clean': 0, u'target_version': u"0'0", u'last_epoch_started': 0}, u'write_tier': -1, u'last_force_op_resend_preluminous': u'0', u'pool': 1, u'removed_snaps': u'[]', u'crush_rule': 0, u'tiers': [], u'hit_set_params': {u'type': u'none'}, u'last_force_op_resend': u'0', u'pool_snaps': [], u'quota_max_objects': 0, u'last_force_op_resend_prenautilus': u'0', u'application_metadata': {u'rbd': {}}, u'options': {}, u'hit_set_count': 0, u'flags': 32769, u'target_max_bytes': 0, u'quota_max_bytes': 0, u'hit_set_search_last_n': 0, u'last_change': u'16', u'min_write_recency_for_promote': 0, u'read_tier': -1}] 2019-12-05T05:03:29.526 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2019-12-05T05:03:29.625 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.612+0000 7f318fd2a700 1 Processor -- start 2019-12-05T05:03:29.625 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.616+0000 7f318fd2a700 1 -- start start 2019-12-05T05:03:29.625 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.616+0000 7f318fd2a700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f318809f550 0x7f3188144630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:29.625 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.616+0000 7f318fd2a700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f31880a9a00 0x7f318807e880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:29.626 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.616+0000 7f318fd2a700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f318807edd0 0x7f3188147b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:29.626 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.616+0000 7f318fd2a700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31880a8f80 con 0x7f318807edd0 2019-12-05T05:03:29.626 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.616+0000 7f318fd2a700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31880a9220 con 0x7f318809f550 2019-12-05T05:03:29.626 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.616+0000 7f318fd2a700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31880a94c0 con 0x7f31880a9a00 2019-12-05T05:03:29.626 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.616+0000 7f318e2c7700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f318809f550 0x7f3188144630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:29.626 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.616+0000 7f318e2c7700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f318809f550 0x7f3188144630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:52984/0 (socket says 172.21.15.42:52984) 2019-12-05T05:03:29.626 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.616+0000 7f318e2c7700 1 -- 172.21.15.42:0/2213593455 learned_addr learned my addr 172.21.15.42:0/2213593455 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:03:29.627 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.616+0000 7f318eac8700 1 --2- 172.21.15.42:0/2213593455 >> v2:172.21.15.40:3300/0 conn(0x7f318807edd0 0x7f3188147b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:29.627 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.616+0000 7f318dac6700 1 --2- 172.21.15.42:0/2213593455 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f31880a9a00 0x7f318807e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:29.627 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.616+0000 7f318e2c7700 1 -- 172.21.15.42:0/2213593455 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f31880a9a00 msgr2=0x7f318807e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:03:29.627 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.616+0000 7f318e2c7700 1 --2- 172.21.15.42:0/2213593455 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f31880a9a00 0x7f318807e880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:03:29.627 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.616+0000 7f318e2c7700 1 -- 172.21.15.42:0/2213593455 >> v2:172.21.15.40:3300/0 conn(0x7f318807edd0 msgr2=0x7f3188147b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:03:29.627 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.616+0000 7f318e2c7700 1 --2- 172.21.15.42:0/2213593455 >> v2:172.21.15.40:3300/0 conn(0x7f318807edd0 0x7f3188147b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:03:29.627 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.616+0000 7f318e2c7700 1 -- 172.21.15.42:0/2213593455 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31880a1000 con 0x7f318809f550 2019-12-05T05:03:29.628 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.616+0000 7f318eac8700 1 --2- 172.21.15.42:0/2213593455 >> v2:172.21.15.40:3300/0 conn(0x7f318807edd0 0x7f3188147b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).send_auth_request state changed! 2019-12-05T05:03:29.628 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.616+0000 7f318dac6700 1 --2- 172.21.15.42:0/2213593455 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f31880a9a00 0x7f318807e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).send_auth_request state changed! 2019-12-05T05:03:29.628 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.616+0000 7f318e2c7700 1 --2- 172.21.15.42:0/2213593455 >> v2:172.21.15.42:3300/0 conn(0x7f318809f550 0x7f3188144630 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f31880a93e0 tx=0x7f3178010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:29.628 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.616+0000 7f318e2c7700 1 --2- 172.21.15.42:0/2213593455 >> v2:172.21.15.42:3300/0 conn(0x7f318809f550 0x7f3188144630 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f31880a93e0 tx=0x7f3178010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:29.628 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.616+0000 7f318e2c7700 1 --2- 172.21.15.42:0/2213593455 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f318809f550 0x7f3188144630 secure :-1 s=READY pgs=48 cs=0 l=1 rx=0x7f31880a93e0 tx=0x7f3178010130).ready entity=mon.1 client_cookie=26ef8eaf574376d2 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:29.629 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.616+0000 7f318e2c7700 1 --2- 172.21.15.42:0/2213593455 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f318809f550 0x7f3188144630 secure :-1 s=THROTTLE_DONE pgs=48 cs=0 l=1 rx=0x7f31880a93e0 tx=0x7f3178010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:29.629 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.616+0000 7f318d2c5700 1 -- 172.21.15.42:0/2213593455 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f317800a160 con 0x7f318809f550 2019-12-05T05:03:29.629 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.616+0000 7f318e2c7700 1 --2- 172.21.15.42:0/2213593455 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f318809f550 0x7f3188144630 secure :-1 s=THROTTLE_DONE pgs=48 cs=0 l=1 rx=0x7f31880a93e0 tx=0x7f3178010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:29.629 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.616+0000 7f318e2c7700 1 --2- 172.21.15.42:0/2213593455 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f318809f550 0x7f3188144630 secure :-1 s=THROTTLE_DONE pgs=48 cs=0 l=1 rx=0x7f31880a93e0 tx=0x7f3178010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:29.629 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.616+0000 7f318d2c5700 1 -- 172.21.15.42:0/2213593455 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3178011a70 con 0x7f318809f550 2019-12-05T05:03:29.630 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.616+0000 7f318d2c5700 1 -- 172.21.15.42:0/2213593455 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f317800a740 con 0x7f318809f550 2019-12-05T05:03:29.630 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.616+0000 7f318fd2a700 1 -- 172.21.15.42:0/2213593455 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f318809f550 msgr2=0x7f3188144630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:29.630 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.616+0000 7f318fd2a700 1 --2- 172.21.15.42:0/2213593455 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f318809f550 0x7f3188144630 secure :-1 s=READY pgs=48 cs=0 l=1 rx=0x7f31880a93e0 tx=0x7f3178010130).stop 2019-12-05T05:03:29.630 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.620+0000 7f318fd2a700 1 -- 172.21.15.42:0/2213593455 shutdown_connections 2019-12-05T05:03:29.630 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.620+0000 7f318fd2a700 1 --2- 172.21.15.42:0/2213593455 >> v2:172.21.15.40:3300/0 conn(0x7f318807edd0 0x7f3188147b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:03:29.630 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.620+0000 7f318fd2a700 1 --2- 172.21.15.42:0/2213593455 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f318809f550 0x7f3188144630 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:29.630 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.620+0000 7f318fd2a700 1 --2- 172.21.15.42:0/2213593455 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f31880a9a00 0x7f318807e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:03:29.631 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.620+0000 7f318fd2a700 1 -- 172.21.15.42:0/2213593455 shutdown_connections 2019-12-05T05:03:29.631 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.620+0000 7f318fd2a700 1 -- 172.21.15.42:0/2213593455 wait complete. 2019-12-05T05:03:29.631 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.620+0000 7f318fd2a700 1 -- 172.21.15.42:0/2213593455 >> 172.21.15.42:0/2213593455 conn(0x7f31880a9ef0 msgr2=0x7f31880aa320 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:29.634 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.620+0000 7f318fd2a700 1 Processor -- start 2019-12-05T05:03:29.634 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.620+0000 7f318fd2a700 1 -- start start 2019-12-05T05:03:29.634 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.624+0000 7f318fd2a700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3188128060 0x7f3188128490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:29.635 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.624+0000 7f318fd2a700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31881472a0 con 0x7f318807d920 2019-12-05T05:03:29.635 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.624+0000 7f318fd2a700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f318808ff80 con 0x7f3188119d00 2019-12-05T05:03:29.635 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.624+0000 7f318fd2a700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3188146ac0 con 0x7f3188128060 2019-12-05T05:03:29.635 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.624+0000 7f318e2c7700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f3188119d00 0x7f318808f780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:53026/0 (socket says 172.21.15.42:53026) 2019-12-05T05:03:29.635 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.624+0000 7f318dac6700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3188128060 0x7f3188128490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:29.635 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.624+0000 7f318e2c7700 1 -- 172.21.15.42:0/2567568547 learned_addr learned my addr 172.21.15.42:0/2567568547 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:03:29.635 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.624+0000 7f3186ffd700 1 -- 172.21.15.42:0/2567568547 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2541238996 0 0) 0x7f318808ff80 con 0x7f3188119d00 2019-12-05T05:03:29.636 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.624+0000 7f3186ffd700 1 -- 172.21.15.42:0/2567568547 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31680030b0 con 0x7f3188119d00 2019-12-05T05:03:29.636 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.624+0000 7f318dac6700 1 -- 172.21.15.42:0/2567568547 >> v1:172.21.15.42:6789/0 conn(0x7f3188119d00 legacy=0x7f318808f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:29.636 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.624+0000 7f318dac6700 1 -- 172.21.15.42:0/2567568547 >> v1:172.21.15.40:6789/0 conn(0x7f318807d920 legacy=0x7f318812f4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:29.636 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.624+0000 7f318dac6700 1 -- 172.21.15.42:0/2567568547 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f318807dd50 con 0x7f3188128060 2019-12-05T05:03:29.636 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.624+0000 7f318dac6700 1 --2- 172.21.15.42:0/2567568547 >> v2:172.21.15.42:3300/0 conn(0x7f3188128060 0x7f3188128490 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f317c002f30 tx=0x7f317c010790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:29.636 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.624+0000 7f318dac6700 1 --2- 172.21.15.42:0/2567568547 >> v2:172.21.15.42:3300/0 conn(0x7f3188128060 0x7f3188128490 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f317c002f30 tx=0x7f317c010790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:29.637 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.624+0000 7f318dac6700 1 --2- 172.21.15.42:0/2567568547 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3188128060 0x7f3188128490 secure :-1 s=READY pgs=50 cs=0 l=1 rx=0x7f317c002f30 tx=0x7f317c010790).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:29.637 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.624+0000 7f318dac6700 1 --2- 172.21.15.42:0/2567568547 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3188128060 0x7f3188128490 secure :-1 s=THROTTLE_DONE pgs=50 cs=0 l=1 rx=0x7f317c002f30 tx=0x7f317c010790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:29.637 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.624+0000 7f3186ffd700 1 -- 172.21.15.42:0/2567568547 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f317c005980 con 0x7f3188128060 2019-12-05T05:03:29.637 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.624+0000 7f318dac6700 1 --2- 172.21.15.42:0/2567568547 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3188128060 0x7f3188128490 secure :-1 s=THROTTLE_DONE pgs=50 cs=0 l=1 rx=0x7f317c002f30 tx=0x7f317c010790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:29.637 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.624+0000 7f318dac6700 1 --2- 172.21.15.42:0/2567568547 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3188128060 0x7f3188128490 secure :-1 s=THROTTLE_DONE pgs=50 cs=0 l=1 rx=0x7f317c002f30 tx=0x7f317c010790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:29.637 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.624+0000 7f3186ffd700 1 -- 172.21.15.42:0/2567568547 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f317c007160 con 0x7f3188128060 2019-12-05T05:03:29.637 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.624+0000 7f3186ffd700 1 -- 172.21.15.42:0/2567568547 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f317c0060c0 con 0x7f3188128060 2019-12-05T05:03:29.637 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.624+0000 7f318fd2a700 1 -- 172.21.15.42:0/2567568547 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f318807e020 con 0x7f3188128060 2019-12-05T05:03:29.638 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.624+0000 7f318fd2a700 1 -- 172.21.15.42:0/2567568547 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3188130cc0 con 0x7f3188128060 2019-12-05T05:03:29.638 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.624+0000 7f318dac6700 1 --2- 172.21.15.42:0/2567568547 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3188128060 0x7f3188128490 secure :-1 s=THROTTLE_DONE pgs=50 cs=0 l=1 rx=0x7f317c002f30 tx=0x7f317c010790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:29.638 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.624+0000 7f3186ffd700 1 -- 172.21.15.42:0/2567568547 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f317c0078a0 con 0x7f3188128060 2019-12-05T05:03:29.638 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.624+0000 7f318dac6700 1 --2- 172.21.15.42:0/2567568547 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3188128060 0x7f3188128490 secure :-1 s=THROTTLE_DONE pgs=50 cs=0 l=1 rx=0x7f317c002f30 tx=0x7f317c010790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:29.638 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.628+0000 7f3186ffd700 1 --2- 172.21.15.42:0/2567568547 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f316803b260 0x7f316803d770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:29.638 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.628+0000 7f3186ffd700 1 -- 172.21.15.42:0/2567568547 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4853+0+0 (secure 0 0 0) 0x7f317c04ac60 con 0x7f3188128060 2019-12-05T05:03:29.638 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.628+0000 7f318e2c7700 1 --2- 172.21.15.42:0/2567568547 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f316803b260 0x7f316803d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:29.639 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.628+0000 7f318e2c7700 1 --2- 172.21.15.42:0/2567568547 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f316803b260 0x7f316803d770 crc :-1 s=READY pgs=44 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:29.642 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.628+0000 7f318fd2a700 1 -- 172.21.15.42:0/2567568547 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f318805d110 con 0x7f3188128060 2019-12-05T05:03:29.642 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.636+0000 7f318dac6700 1 --2- 172.21.15.42:0/2567568547 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3188128060 0x7f3188128490 secure :-1 s=THROTTLE_DONE pgs=50 cs=0 l=1 rx=0x7f317c002f30 tx=0x7f317c010790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:29.642 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.636+0000 7f3186ffd700 1 -- 172.21.15.42:0/2567568547 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f317c031ce0 con 0x7f3188128060 2019-12-05T05:03:29.798 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.792+0000 7f318fd2a700 1 -- 172.21.15.42:0/2567568547 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f318805b8c0 con 0x7f3188128060 2019-12-05T05:03:29.799 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.792+0000 7f318dac6700 1 --2- 172.21.15.42:0/2567568547 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3188128060 0x7f3188128490 secure :-1 s=THROTTLE_DONE pgs=50 cs=0 l=1 rx=0x7f317c002f30 tx=0x7f317c010790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:29.799 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.792+0000 7f3186ffd700 1 -- 172.21.15.42:0/2567568547 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v16) v1 ==== 92+0+10 (secure 0 0 0) 0x7f317c031f40 con 0x7f3188128060 2019-12-05T05:03:29.800 INFO:teuthology.orchestra.run.smithi042.stdout:pg_num: 8 2019-12-05T05:03:29.802 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.796+0000 7f318fd2a700 1 -- 172.21.15.42:0/2567568547 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f316803b260 msgr2=0x7f316803d770 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:29.802 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.796+0000 7f318fd2a700 1 --2- 172.21.15.42:0/2567568547 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f316803b260 0x7f316803d770 crc :-1 s=READY pgs=44 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:29.802 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.796+0000 7f318fd2a700 1 -- 172.21.15.42:0/2567568547 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3188128060 msgr2=0x7f3188128490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:29.802 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.796+0000 7f318fd2a700 1 --2- 172.21.15.42:0/2567568547 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3188128060 0x7f3188128490 secure :-1 s=READY pgs=50 cs=0 l=1 rx=0x7f317c002f30 tx=0x7f317c010790).stop 2019-12-05T05:03:29.802 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.796+0000 7f318fd2a700 1 -- 172.21.15.42:0/2567568547 shutdown_connections 2019-12-05T05:03:29.803 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.796+0000 7f318fd2a700 1 --2- 172.21.15.42:0/2567568547 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f316803b260 0x7f316803d770 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:29.803 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.796+0000 7f318fd2a700 1 --2- 172.21.15.42:0/2567568547 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3188128060 0x7f3188128490 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:29.803 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.796+0000 7f318fd2a700 1 -- 172.21.15.42:0/2567568547 shutdown_connections 2019-12-05T05:03:29.803 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.796+0000 7f318fd2a700 1 -- 172.21.15.42:0/2567568547 wait complete. 2019-12-05T05:03:29.803 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.796+0000 7f318fd2a700 1 -- 172.21.15.42:0/2567568547 >> 172.21.15.42:0/2567568547 conn(0x7f318805a000 msgr2=0x7f31880a4ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:29.813 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2019-12-05T05:03:29.916 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.900+0000 7fa486282700 1 Processor -- start 2019-12-05T05:03:29.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.900+0000 7fa486282700 1 -- start start 2019-12-05T05:03:29.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.900+0000 7fa486282700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa4800a7740 0x7fa48008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:29.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.900+0000 7fa486282700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fa48007e720 0x7fa48007eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:29.920 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.900+0000 7fa486282700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4800a0710 con 0x7fa48009f2e0 2019-12-05T05:03:29.920 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.900+0000 7fa486282700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4800a0b00 con 0x7fa4800a7740 2019-12-05T05:03:29.920 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.900+0000 7fa486282700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa48008e5b0 con 0x7fa48007e720 2019-12-05T05:03:29.920 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.900+0000 7fa477fff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fa48007e720 0x7fa48007eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:29.920 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.900+0000 7fa477fff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fa48007e720 0x7fa48007eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:52998/0 (socket says 172.21.15.42:52998) 2019-12-05T05:03:29.920 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.900+0000 7fa477fff700 1 -- 172.21.15.42:0/1233668384 learned_addr learned my addr 172.21.15.42:0/1233668384 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:03:29.921 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.900+0000 7fa48481f700 1 --2- 172.21.15.42:0/1233668384 >> v2:172.21.15.40:3300/0 conn(0x7fa4800a7740 0x7fa48008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:29.921 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.904+0000 7fa477fff700 1 -- 172.21.15.42:0/1233668384 >> v2:172.21.15.40:3300/0 conn(0x7fa4800a7740 msgr2=0x7fa48008df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:03:29.921 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.904+0000 7fa477fff700 1 --2- 172.21.15.42:0/1233668384 >> v2:172.21.15.40:3300/0 conn(0x7fa4800a7740 0x7fa48008df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:03:29.921 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.904+0000 7fa477fff700 1 -- 172.21.15.42:0/1233668384 >> v1:172.21.15.40:6789/0 conn(0x7fa48009f2e0 legacy=0x7fa48009f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:29.921 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.904+0000 7fa477fff700 1 -- 172.21.15.42:0/1233668384 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa48007f0a0 con 0x7fa48007e720 2019-12-05T05:03:29.921 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.904+0000 7fa477fff700 1 --2- 172.21.15.42:0/1233668384 >> v2:172.21.15.42:3300/0 conn(0x7fa48007e720 0x7fa48007eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fa470002f30 tx=0x7fa47000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:29.921 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.904+0000 7fa477fff700 1 --2- 172.21.15.42:0/1233668384 >> v2:172.21.15.42:3300/0 conn(0x7fa48007e720 0x7fa48007eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fa470002f30 tx=0x7fa47000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:29.922 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.904+0000 7fa477fff700 1 --2- 172.21.15.42:0/1233668384 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa48007e720 0x7fa48007eb50 secure :-1 s=READY pgs=51 cs=0 l=1 rx=0x7fa470002f30 tx=0x7fa47000fd30).ready entity=mon.1 client_cookie=c37add9060e814a0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:29.922 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.904+0000 7fa477fff700 1 --2- 172.21.15.42:0/1233668384 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa48007e720 0x7fa48007eb50 secure :-1 s=THROTTLE_DONE pgs=51 cs=0 l=1 rx=0x7fa470002f30 tx=0x7fa47000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:29.922 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.904+0000 7fa4777fe700 1 -- 172.21.15.42:0/1233668384 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa470011370 con 0x7fa48007e720 2019-12-05T05:03:29.922 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.904+0000 7fa477fff700 1 --2- 172.21.15.42:0/1233668384 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa48007e720 0x7fa48007eb50 secure :-1 s=THROTTLE_DONE pgs=51 cs=0 l=1 rx=0x7fa470002f30 tx=0x7fa47000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:29.922 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.904+0000 7fa4777fe700 1 -- 172.21.15.42:0/1233668384 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa470008bb0 con 0x7fa48007e720 2019-12-05T05:03:29.922 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.904+0000 7fa477fff700 1 --2- 172.21.15.42:0/1233668384 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa48007e720 0x7fa48007eb50 secure :-1 s=THROTTLE_DONE pgs=51 cs=0 l=1 rx=0x7fa470002f30 tx=0x7fa47000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:29.923 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.904+0000 7fa4777fe700 1 -- 172.21.15.42:0/1233668384 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa470011950 con 0x7fa48007e720 2019-12-05T05:03:29.923 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.904+0000 7fa486282700 1 -- 172.21.15.42:0/1233668384 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa48007e720 msgr2=0x7fa48007eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:29.924 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.904+0000 7fa486282700 1 --2- 172.21.15.42:0/1233668384 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa48007e720 0x7fa48007eb50 secure :-1 s=READY pgs=51 cs=0 l=1 rx=0x7fa470002f30 tx=0x7fa47000fd30).stop 2019-12-05T05:03:29.924 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.904+0000 7fa486282700 1 -- 172.21.15.42:0/1233668384 shutdown_connections 2019-12-05T05:03:29.924 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.904+0000 7fa486282700 1 --2- 172.21.15.42:0/1233668384 >> v2:172.21.15.40:3300/0 conn(0x7fa4800a7740 0x7fa48008df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:03:29.924 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.904+0000 7fa486282700 1 --2- 172.21.15.42:0/1233668384 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa48007e720 0x7fa48007eb50 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:29.924 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.904+0000 7fa486282700 1 -- 172.21.15.42:0/1233668384 shutdown_connections 2019-12-05T05:03:29.924 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.904+0000 7fa486282700 1 -- 172.21.15.42:0/1233668384 wait complete. 2019-12-05T05:03:29.925 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.904+0000 7fa486282700 1 -- 172.21.15.42:0/1233668384 >> 172.21.15.42:0/1233668384 conn(0x7fa4800aabe0 msgr2=0x7fa4800a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:29.925 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.908+0000 7fa486282700 1 Processor -- start 2019-12-05T05:03:29.925 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.908+0000 7fa486282700 1 -- start start 2019-12-05T05:03:29.925 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.908+0000 7fa486282700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa480131940 0x7fa48013c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:29.925 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.908+0000 7fa486282700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa48007b520 con 0x7fa480131510 2019-12-05T05:03:29.925 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.908+0000 7fa486282700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa48013df00 con 0x7fa4800a5250 2019-12-05T05:03:29.925 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.908+0000 7fa486282700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4800a4a70 con 0x7fa480131940 2019-12-05T05:03:29.926 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.908+0000 7fa48481f700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fa4800a5250 0x7fa480118450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:53040/0 (socket says 172.21.15.42:53040) 2019-12-05T05:03:29.926 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.908+0000 7fa48481f700 1 -- 172.21.15.42:0/2139150057 learned_addr learned my addr 172.21.15.42:0/2139150057 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:03:29.926 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.908+0000 7fa485020700 1 --2- 172.21.15.42:0/2139150057 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa480131940 0x7fa48013c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:29.926 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.908+0000 7fa4757fa700 1 -- 172.21.15.42:0/2139150057 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1237638725 0 0) 0x7fa48013df00 con 0x7fa4800a5250 2019-12-05T05:03:29.926 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.908+0000 7fa4757fa700 1 -- 172.21.15.42:0/2139150057 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4600030b0 con 0x7fa4800a5250 2019-12-05T05:03:29.926 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.908+0000 7fa4757fa700 1 -- 172.21.15.42:0/2139150057 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3046454183 0 0) 0x7fa48007b520 con 0x7fa480131510 2019-12-05T05:03:29.926 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.908+0000 7fa4757fa700 1 -- 172.21.15.42:0/2139150057 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa48013df00 con 0x7fa480131510 2019-12-05T05:03:29.927 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.908+0000 7fa485020700 1 -- 172.21.15.42:0/2139150057 >> v1:172.21.15.42:6789/0 conn(0x7fa4800a5250 legacy=0x7fa480118450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:29.927 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.908+0000 7fa485020700 1 -- 172.21.15.42:0/2139150057 >> v1:172.21.15.40:6789/0 conn(0x7fa480131510 legacy=0x7fa4800a6100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:29.927 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.908+0000 7fa485020700 1 -- 172.21.15.42:0/2139150057 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa48013f140 con 0x7fa480131940 2019-12-05T05:03:29.927 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.908+0000 7fa485020700 1 --2- 172.21.15.42:0/2139150057 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa480131940 0x7fa48013c9a0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fa478001210 tx=0x7fa478010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:29.927 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.908+0000 7fa485020700 1 --2- 172.21.15.42:0/2139150057 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa480131940 0x7fa48013c9a0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fa478001210 tx=0x7fa478010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:29.927 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.908+0000 7fa485020700 1 --2- 172.21.15.42:0/2139150057 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa480131940 0x7fa48013c9a0 secure :-1 s=READY pgs=26 cs=0 l=1 rx=0x7fa478001210 tx=0x7fa478010ac0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:29.927 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.908+0000 7fa485020700 1 --2- 172.21.15.42:0/2139150057 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa480131940 0x7fa48013c9a0 secure :-1 s=THROTTLE_DONE pgs=26 cs=0 l=1 rx=0x7fa478001210 tx=0x7fa478010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:29.928 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.912+0000 7fa4757fa700 1 -- 172.21.15.42:0/2139150057 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa478002240 con 0x7fa480131940 2019-12-05T05:03:29.928 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.912+0000 7fa486282700 1 -- 172.21.15.42:0/2139150057 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa48013cef0 con 0x7fa480131940 2019-12-05T05:03:29.928 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.912+0000 7fa485020700 1 --2- 172.21.15.42:0/2139150057 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa480131940 0x7fa48013c9a0 secure :-1 s=THROTTLE_DONE pgs=26 cs=0 l=1 rx=0x7fa478001210 tx=0x7fa478010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:29.928 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.912+0000 7fa485020700 1 --2- 172.21.15.42:0/2139150057 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa480131940 0x7fa48013c9a0 secure :-1 s=THROTTLE_DONE pgs=26 cs=0 l=1 rx=0x7fa478001210 tx=0x7fa478010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:29.928 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.912+0000 7fa4757fa700 1 -- 172.21.15.42:0/2139150057 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa478006ad0 con 0x7fa480131940 2019-12-05T05:03:29.928 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.912+0000 7fa486282700 1 -- 172.21.15.42:0/2139150057 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa48013d3f0 con 0x7fa480131940 2019-12-05T05:03:29.929 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.912+0000 7fa4757fa700 1 -- 172.21.15.42:0/2139150057 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa478002820 con 0x7fa480131940 2019-12-05T05:03:29.930 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.912+0000 7fa485020700 1 --2- 172.21.15.42:0/2139150057 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa480131940 0x7fa48013c9a0 secure :-1 s=THROTTLE_DONE pgs=26 cs=0 l=1 rx=0x7fa478001210 tx=0x7fa478010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:29.930 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.912+0000 7fa485020700 1 --2- 172.21.15.42:0/2139150057 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa480131940 0x7fa48013c9a0 secure :-1 s=THROTTLE_DONE pgs=26 cs=0 l=1 rx=0x7fa478001210 tx=0x7fa478010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:29.930 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.912+0000 7fa4757fa700 1 -- 172.21.15.42:0/2139150057 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fa478002b80 con 0x7fa480131940 2019-12-05T05:03:29.931 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.912+0000 7fa4757fa700 1 --2- 172.21.15.42:0/2139150057 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa46003b490 0x7fa46003d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:29.931 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.912+0000 7fa4757fa700 1 -- 172.21.15.42:0/2139150057 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4853+0+0 (secure 0 0 0) 0x7fa47804b340 con 0x7fa480131940 2019-12-05T05:03:29.931 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.912+0000 7fa48481f700 1 --2- 172.21.15.42:0/2139150057 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa46003b490 0x7fa46003d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:29.931 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.912+0000 7fa48481f700 1 --2- 172.21.15.42:0/2139150057 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa46003b490 0x7fa46003d9a0 crc :-1 s=READY pgs=45 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:29.931 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.916+0000 7fa486282700 1 -- 172.21.15.42:0/2139150057 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa48005d110 con 0x7fa480131940 2019-12-05T05:03:29.931 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.924+0000 7fa485020700 1 --2- 172.21.15.42:0/2139150057 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa480131940 0x7fa48013c9a0 secure :-1 s=THROTTLE_DONE pgs=26 cs=0 l=1 rx=0x7fa478001210 tx=0x7fa478010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:29.932 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:29.924+0000 7fa4757fa700 1 -- 172.21.15.42:0/2139150057 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fa478032380 con 0x7fa480131940 2019-12-05T05:03:30.092 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:30.088+0000 7fa486282700 1 -- 172.21.15.42:0/2139150057 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fa48005b8c0 con 0x7fa480131940 2019-12-05T05:03:30.093 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:30.088+0000 7fa485020700 1 --2- 172.21.15.42:0/2139150057 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa480131940 0x7fa48013c9a0 secure :-1 s=THROTTLE_DONE pgs=26 cs=0 l=1 rx=0x7fa478001210 tx=0x7fa478010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:30.093 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:30.088+0000 7fa4757fa700 1 -- 172.21.15.42:0/2139150057 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v16) v1 ==== 84+0+64 (secure 0 0 0) 0x7fa4780325e0 con 0x7fa480131940 2019-12-05T05:03:30.095 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:03:30.095 INFO:teuthology.orchestra.run.smithi042.stdout:[ 2019-12-05T05:03:30.095 INFO:teuthology.orchestra.run.smithi042.stdout: { 2019-12-05T05:03:30.095 INFO:teuthology.orchestra.run.smithi042.stdout: "poolnum": 1, 2019-12-05T05:03:30.095 INFO:teuthology.orchestra.run.smithi042.stdout: "poolname": "rbd" 2019-12-05T05:03:30.096 INFO:teuthology.orchestra.run.smithi042.stdout: } 2019-12-05T05:03:30.096 INFO:teuthology.orchestra.run.smithi042.stdout:] 2019-12-05T05:03:30.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:30.092+0000 7fa486282700 1 -- 172.21.15.42:0/2139150057 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa46003b490 msgr2=0x7fa46003d9a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:30.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:30.092+0000 7fa486282700 1 --2- 172.21.15.42:0/2139150057 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa46003b490 0x7fa46003d9a0 crc :-1 s=READY pgs=45 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:30.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:30.092+0000 7fa486282700 1 -- 172.21.15.42:0/2139150057 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa480131940 msgr2=0x7fa48013c9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:30.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:30.092+0000 7fa486282700 1 --2- 172.21.15.42:0/2139150057 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa480131940 0x7fa48013c9a0 secure :-1 s=READY pgs=26 cs=0 l=1 rx=0x7fa478001210 tx=0x7fa478010ac0).stop 2019-12-05T05:03:30.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:30.092+0000 7fa486282700 1 -- 172.21.15.42:0/2139150057 shutdown_connections 2019-12-05T05:03:30.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:30.092+0000 7fa486282700 1 --2- 172.21.15.42:0/2139150057 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa46003b490 0x7fa46003d9a0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:30.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:30.092+0000 7fa486282700 1 --2- 172.21.15.42:0/2139150057 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa480131940 0x7fa48013c9a0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:30.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:30.092+0000 7fa486282700 1 -- 172.21.15.42:0/2139150057 shutdown_connections 2019-12-05T05:03:30.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:30.092+0000 7fa486282700 1 -- 172.21.15.42:0/2139150057 wait complete. 2019-12-05T05:03:30.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:30.092+0000 7fa486282700 1 -- 172.21.15.42:0/2139150057 >> 172.21.15.42:0/2139150057 conn(0x7fa4800a7f90 msgr2=0x7fa480059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:30.108 INFO:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2019-12-05T05:03:30.108 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata 8 2019-12-05T05:03:30.215 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:30.196+0000 7f91aabae700 1 Processor -- start 2019-12-05T05:03:30.215 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:30.196+0000 7f91aabae700 1 -- start start 2019-12-05T05:03:30.216 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:30.200+0000 7f91aabae700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f91a409f550 0x7f91a4144630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:30.216 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:30.200+0000 7f91aabae700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91a407f080 con 0x7f91a414cf50 2019-12-05T05:03:30.216 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:30.200+0000 7f91aabae700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91a40a1000 con 0x7f91a40a9a00 2019-12-05T05:03:30.217 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:30.200+0000 7f91aabae700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91a40a8f80 con 0x7f91a409f550 2019-12-05T05:03:30.217 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:30.200+0000 7f91a914b700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f91a409f550 0x7f91a4144630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:30.217 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:30.200+0000 7f91a914b700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f91a409f550 0x7f91a4144630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:53010/0 (socket says 172.21.15.42:53010) 2019-12-05T05:03:30.218 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:30.200+0000 7f91a914b700 1 -- 172.21.15.42:0/212433448 learned_addr learned my addr 172.21.15.42:0/212433448 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:03:30.218 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:30.200+0000 7f91a914b700 1 -- 172.21.15.42:0/212433448 >> v1:172.21.15.42:6789/0 conn(0x7f91a40a9a00 legacy=0x7f91a407e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:30.218 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:30.200+0000 7f91a914b700 1 -- 172.21.15.42:0/212433448 >> v1:172.21.15.40:6789/0 conn(0x7f91a414cf50 legacy=0x7f91a414f390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:30.218 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:30.200+0000 7f91a914b700 1 -- 172.21.15.42:0/212433448 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91a4144b80 con 0x7f91a409f550 2019-12-05T05:03:30.218 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:30.200+0000 7f91a914b700 1 --2- 172.21.15.42:0/212433448 >> v2:172.21.15.42:3300/0 conn(0x7f91a409f550 0x7f91a4144630 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f91a0002f30 tx=0x7f91a000fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:30.218 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:30.200+0000 7f91a914b700 1 --2- 172.21.15.42:0/212433448 >> v2:172.21.15.42:3300/0 conn(0x7f91a409f550 0x7f91a4144630 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f91a0002f30 tx=0x7f91a000fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:30.219 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:30.200+0000 7f91a914b700 1 --2- 172.21.15.42:0/212433448 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f91a409f550 0x7f91a4144630 secure :-1 s=READY pgs=54 cs=0 l=1 rx=0x7f91a0002f30 tx=0x7f91a000fcb0).ready entity=mon.1 client_cookie=1ad113cf3d9345a4 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:30.219 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:30.200+0000 7f91a914b700 1 --2- 172.21.15.42:0/212433448 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f91a409f550 0x7f91a4144630 secure :-1 s=THROTTLE_DONE pgs=54 cs=0 l=1 rx=0x7f91a0002f30 tx=0x7f91a000fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:30.219 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:30.200+0000 7f919bfff700 1 -- 172.21.15.42:0/212433448 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f91a00114e0 con 0x7f91a409f550 2019-12-05T05:03:30.220 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:30.200+0000 7f91a914b700 1 --2- 172.21.15.42:0/212433448 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f91a409f550 0x7f91a4144630 secure :-1 s=THROTTLE_DONE pgs=54 cs=0 l=1 rx=0x7f91a0002f30 tx=0x7f91a000fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:30.220 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:30.200+0000 7f91a914b700 1 --2- 172.21.15.42:0/212433448 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f91a409f550 0x7f91a4144630 secure :-1 s=THROTTLE_DONE pgs=54 cs=0 l=1 rx=0x7f91a0002f30 tx=0x7f91a000fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:30.220 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:30.200+0000 7f919bfff700 1 -- 172.21.15.42:0/212433448 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91a0008ca0 con 0x7f91a409f550 2019-12-05T05:03:30.220 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:30.200+0000 7f919bfff700 1 -- 172.21.15.42:0/212433448 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f91a0011ac0 con 0x7f91a409f550 2019-12-05T05:03:30.221 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:30.200+0000 7f91aabae700 1 -- 172.21.15.42:0/212433448 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f91a409f550 msgr2=0x7f91a4144630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:30.221 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:30.200+0000 7f91aabae700 1 --2- 172.21.15.42:0/212433448 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f91a409f550 0x7f91a4144630 secure :-1 s=READY pgs=54 cs=0 l=1 rx=0x7f91a0002f30 tx=0x7f91a000fcb0).stop 2019-12-05T05:03:30.221 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:30.200+0000 7f91aabae700 1 -- 172.21.15.42:0/212433448 shutdown_connections 2019-12-05T05:03:30.221 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:30.200+0000 7f91aabae700 1 --2- 172.21.15.42:0/212433448 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f91a409f550 0x7f91a4144630 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:30.221 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:30.200+0000 7f91aabae700 1 -- 172.21.15.42:0/212433448 shutdown_connections 2019-12-05T05:03:30.221 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:30.200+0000 7f91aabae700 1 -- 172.21.15.42:0/212433448 wait complete. 2019-12-05T05:03:30.221 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:30.200+0000 7f91aabae700 1 -- 172.21.15.42:0/212433448 >> 172.21.15.42:0/212433448 conn(0x7f91a40a9ef0 msgr2=0x7f91a40aa320 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:30.222 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:30.204+0000 7f91aabae700 1 Processor -- start 2019-12-05T05:03:30.222 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:30.204+0000 7f91aabae700 1 -- start start 2019-12-05T05:03:30.222 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:30.204+0000 7f91aabae700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f91a41250a0 0x7f91a41254d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:30.222 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:30.204+0000 7f91aabae700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f91a4121aa0 0x7f91a4121ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:30.222 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:30.204+0000 7f91aabae700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91a4142df0 con 0x7f91a41429c0 2019-12-05T05:03:30.222 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:30.204+0000 7f91aabae700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91a41430d0 con 0x7f91a41250a0 2019-12-05T05:03:30.222 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:30.204+0000 7f91aabae700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91a4124e00 con 0x7f91a4121aa0 2019-12-05T05:03:30.223 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:30.204+0000 7f91a994c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f91a4121aa0 0x7f91a4121ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:30.223 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:30.204+0000 7f91a914b700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f91a41429c0 0x7f91a41465f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:53056/0 (socket says 172.21.15.42:53056) 2019-12-05T05:03:30.223 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:30.204+0000 7f91a914b700 1 -- 172.21.15.42:0/735657737 learned_addr learned my addr 172.21.15.42:0/735657737 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:03:30.223 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:30.204+0000 7f91a894a700 1 --2- 172.21.15.42:0/735657737 >> v2:172.21.15.40:3300/0 conn(0x7f91a41250a0 0x7f91a41254d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:30.223 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:30.204+0000 7f91a994c700 1 -- 172.21.15.42:0/735657737 >> v2:172.21.15.40:3300/0 conn(0x7f91a41250a0 msgr2=0x7f91a41254d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:30.223 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:30.204+0000 7f91a994c700 1 --2- 172.21.15.42:0/735657737 >> v2:172.21.15.40:3300/0 conn(0x7f91a41250a0 0x7f91a41254d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:30.223 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:30.204+0000 7f91a994c700 1 -- 172.21.15.42:0/735657737 >> v1:172.21.15.42:6789/0 conn(0x7f91a41429c0 legacy=0x7f91a41465f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:30.223 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:30.204+0000 7f9199ffb700 1 -- 172.21.15.42:0/735657737 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3382586455 0 0) 0x7f91a0011dd0 con 0x7f91a41429c0 2019-12-05T05:03:30.224 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:30.204+0000 7f91a994c700 1 -- 172.21.15.42:0/735657737 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91a4125a20 con 0x7f91a4121aa0 2019-12-05T05:03:30.224 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:30.204+0000 7f91a994c700 1 --2- 172.21.15.42:0/735657737 >> v2:172.21.15.42:3300/0 conn(0x7f91a4121aa0 0x7f91a4121ed0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f919c001210 tx=0x7f919c012d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:30.224 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:30.204+0000 7f91a994c700 1 --2- 172.21.15.42:0/735657737 >> v2:172.21.15.42:3300/0 conn(0x7f91a4121aa0 0x7f91a4121ed0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f919c001210 tx=0x7f919c012d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:30.224 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:30.204+0000 7f91a994c700 1 --2- 172.21.15.42:0/735657737 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f91a4121aa0 0x7f91a4121ed0 secure :-1 s=READY pgs=56 cs=0 l=1 rx=0x7f919c001210 tx=0x7f919c012d50).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:30.224 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:30.204+0000 7f91a994c700 1 --2- 172.21.15.42:0/735657737 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f91a4121aa0 0x7f91a4121ed0 secure :-1 s=THROTTLE_DONE pgs=56 cs=0 l=1 rx=0x7f919c001210 tx=0x7f919c012d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:30.224 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:30.204+0000 7f9199ffb700 1 -- 172.21.15.42:0/735657737 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f919c014af0 con 0x7f91a4121aa0 2019-12-05T05:03:30.225 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:30.204+0000 7f91a994c700 1 --2- 172.21.15.42:0/735657737 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f91a4121aa0 0x7f91a4121ed0 secure :-1 s=THROTTLE_DONE pgs=56 cs=0 l=1 rx=0x7f919c001210 tx=0x7f919c012d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:30.225 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:30.204+0000 7f91a994c700 1 --2- 172.21.15.42:0/735657737 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f91a4121aa0 0x7f91a4121ed0 secure :-1 s=THROTTLE_DONE pgs=56 cs=0 l=1 rx=0x7f919c001210 tx=0x7f919c012d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:30.225 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:30.204+0000 7f9199ffb700 1 -- 172.21.15.42:0/735657737 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f919c014710 con 0x7f91a4121aa0 2019-12-05T05:03:30.225 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:30.204+0000 7f91aabae700 1 -- 172.21.15.42:0/735657737 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91a4122420 con 0x7f91a4121aa0 2019-12-05T05:03:30.225 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:30.204+0000 7f9199ffb700 1 -- 172.21.15.42:0/735657737 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f919c0068d0 con 0x7f91a4121aa0 2019-12-05T05:03:30.225 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:30.204+0000 7f91aabae700 1 -- 172.21.15.42:0/735657737 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91a407e7f0 con 0x7f91a4121aa0 2019-12-05T05:03:30.225 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:30.208+0000 7f91a994c700 1 --2- 172.21.15.42:0/735657737 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f91a4121aa0 0x7f91a4121ed0 secure :-1 s=THROTTLE_DONE pgs=56 cs=0 l=1 rx=0x7f919c001210 tx=0x7f919c012d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:30.226 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:30.208+0000 7f9199ffb700 1 -- 172.21.15.42:0/735657737 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f919c006be0 con 0x7f91a4121aa0 2019-12-05T05:03:30.226 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:30.208+0000 7f91a994c700 1 --2- 172.21.15.42:0/735657737 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f91a4121aa0 0x7f91a4121ed0 secure :-1 s=THROTTLE_DONE pgs=56 cs=0 l=1 rx=0x7f919c001210 tx=0x7f919c012d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:30.226 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:30.208+0000 7f9199ffb700 1 --2- 172.21.15.42:0/735657737 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f918403ace0 0x7f918403d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:30.226 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:30.208+0000 7f9199ffb700 1 -- 172.21.15.42:0/735657737 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4853+0+0 (secure 0 0 0) 0x7f919c04d130 con 0x7f91a4121aa0 2019-12-05T05:03:30.226 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:30.208+0000 7f91a914b700 1 --2- 172.21.15.42:0/735657737 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f918403ace0 0x7f918403d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:30.226 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:30.208+0000 7f91a914b700 1 --2- 172.21.15.42:0/735657737 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f918403ace0 0x7f918403d1f0 crc :-1 s=READY pgs=46 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:30.227 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:30.212+0000 7f91aabae700 1 -- 172.21.15.42:0/735657737 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91a405d110 con 0x7f91a4121aa0 2019-12-05T05:03:30.227 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:30.220+0000 7f91a994c700 1 --2- 172.21.15.42:0/735657737 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f91a4121aa0 0x7f91a4121ed0 secure :-1 s=THROTTLE_DONE pgs=56 cs=0 l=1 rx=0x7f919c001210 tx=0x7f919c012d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:30.227 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:30.220+0000 7f9199ffb700 1 -- 172.21.15.42:0/735657737 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f919c0341b0 con 0x7f91a4121aa0 2019-12-05T05:03:30.384 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:30.380+0000 7f91aabae700 1 -- 172.21.15.42:0/735657737 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"} v 0) v1 -- 0x7f91a405b8c0 con 0x7f91a4121aa0 2019-12-05T05:03:31.215 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:31.208+0000 7f91a994c700 1 --2- 172.21.15.42:0/735657737 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f91a4121aa0 0x7f91a4121ed0 secure :-1 s=THROTTLE_DONE pgs=56 cs=0 l=1 rx=0x7f919c001210 tx=0x7f919c012d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:31.216 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:31.208+0000 7f9199ffb700 1 -- 172.21.15.42:0/735657737 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"}]=0 pool 'cephfs_metadata' created v18) v1 ==== 133+0+0 (secure 0 0 0) 0x7f919c034410 con 0x7f91a4121aa0 2019-12-05T05:03:31.253 INFO:teuthology.orchestra.run.smithi042.stderr:pool 'cephfs_metadata' created 2019-12-05T05:03:31.254 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:31.248+0000 7f91aabae700 1 -- 172.21.15.42:0/735657737 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f918403ace0 msgr2=0x7f918403d1f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:31.255 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:31.248+0000 7f91aabae700 1 --2- 172.21.15.42:0/735657737 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f918403ace0 0x7f918403d1f0 crc :-1 s=READY pgs=46 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:31.255 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:31.248+0000 7f91aabae700 1 -- 172.21.15.42:0/735657737 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f91a4121aa0 msgr2=0x7f91a4121ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:31.255 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:31.248+0000 7f91aabae700 1 --2- 172.21.15.42:0/735657737 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f91a4121aa0 0x7f91a4121ed0 secure :-1 s=READY pgs=56 cs=0 l=1 rx=0x7f919c001210 tx=0x7f919c012d50).stop 2019-12-05T05:03:31.255 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:31.248+0000 7f91aabae700 1 -- 172.21.15.42:0/735657737 shutdown_connections 2019-12-05T05:03:31.256 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:31.248+0000 7f91aabae700 1 --2- 172.21.15.42:0/735657737 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f918403ace0 0x7f918403d1f0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:31.256 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:31.248+0000 7f91aabae700 1 --2- 172.21.15.42:0/735657737 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f91a4121aa0 0x7f91a4121ed0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:31.256 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:31.248+0000 7f91aabae700 1 --2- 172.21.15.42:0/735657737 >> v2:172.21.15.40:3300/0 conn(0x7f91a41250a0 0x7f91a41254d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:31.256 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:31.248+0000 7f91aabae700 1 -- 172.21.15.42:0/735657737 shutdown_connections 2019-12-05T05:03:31.257 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:31.248+0000 7f91aabae700 1 -- 172.21.15.42:0/735657737 wait complete. 2019-12-05T05:03:31.257 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:31.248+0000 7f91aabae700 1 -- 172.21.15.42:0/735657737 >> 172.21.15.42:0/735657737 conn(0x7f91a40a7420 msgr2=0x7f91a40a98f0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:31.269 INFO:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2019-12-05T05:03:31.269 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data m=2 k=2 crush-failure-domain=osd 2019-12-05T05:03:31.393 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:31.380+0000 7f083f6b9700 1 Processor -- start 2019-12-05T05:03:31.394 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:31.380+0000 7f083f6b9700 1 -- start start 2019-12-05T05:03:31.394 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:31.384+0000 7f083f6b9700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f083809f550 0x7f0838144630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:31.394 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:31.384+0000 7f083f6b9700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f083814cf50 0x7f083814f390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:31.394 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:31.384+0000 7f083f6b9700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f083807f080 con 0x7f08380a9a00 2019-12-05T05:03:31.395 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:31.384+0000 7f083f6b9700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08380a8f80 con 0x7f083814cf50 2019-12-05T05:03:31.395 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:31.384+0000 7f083f6b9700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08380a9220 con 0x7f083809f550 2019-12-05T05:03:31.395 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:31.384+0000 7f083e457700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f083814cf50 0x7f083814f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:31.395 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:31.384+0000 7f083e457700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f083814cf50 0x7f083814f390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:53030/0 (socket says 172.21.15.42:53030) 2019-12-05T05:03:31.395 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:31.384+0000 7f083e457700 1 -- 172.21.15.42:0/891846901 learned_addr learned my addr 172.21.15.42:0/891846901 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:03:31.395 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:31.384+0000 7f083d455700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f08380a9a00 0x7f083807e880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:53064/0 (socket says 172.21.15.42:53064) 2019-12-05T05:03:31.396 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:31.384+0000 7f083dc56700 1 --2- 172.21.15.42:0/891846901 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f083809f550 0x7f0838144630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:31.396 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:31.384+0000 7f083e457700 1 -- 172.21.15.42:0/891846901 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f083809f550 msgr2=0x7f0838144630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:03:31.396 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:31.384+0000 7f083e457700 1 --2- 172.21.15.42:0/891846901 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f083809f550 0x7f0838144630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:03:31.396 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:31.384+0000 7f083e457700 1 -- 172.21.15.42:0/891846901 >> v1:172.21.15.42:6789/0 conn(0x7f08380a9a00 legacy=0x7f083807e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:31.396 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:31.384+0000 7f083e457700 1 -- 172.21.15.42:0/891846901 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08380a1000 con 0x7f083814cf50 2019-12-05T05:03:31.396 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:31.384+0000 7f083cc54700 1 -- 172.21.15.42:0/891846901 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2279729209 0 0) 0x7f083807f080 con 0x7f08380a9a00 2019-12-05T05:03:31.396 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:31.384+0000 7f083e457700 1 --2- 172.21.15.42:0/891846901 >> v2:172.21.15.42:3300/0 conn(0x7f083814cf50 0x7f083814f390 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f08380a9140 tx=0x7f083000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:31.397 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:31.384+0000 7f083e457700 1 --2- 172.21.15.42:0/891846901 >> v2:172.21.15.42:3300/0 conn(0x7f083814cf50 0x7f083814f390 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f08380a9140 tx=0x7f083000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:31.397 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:31.384+0000 7f083e457700 1 --2- 172.21.15.42:0/891846901 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f083814cf50 0x7f083814f390 secure :-1 s=READY pgs=58 cs=0 l=1 rx=0x7f08380a9140 tx=0x7f083000fd30).ready entity=mon.1 client_cookie=2196b9a3d690f983 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:31.397 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:31.384+0000 7f083e457700 1 --2- 172.21.15.42:0/891846901 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f083814cf50 0x7f083814f390 secure :-1 s=THROTTLE_DONE pgs=58 cs=0 l=1 rx=0x7f08380a9140 tx=0x7f083000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:31.397 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:31.384+0000 7f083cc54700 1 -- 172.21.15.42:0/891846901 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0830011370 con 0x7f083814cf50 2019-12-05T05:03:31.397 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:31.384+0000 7f083e457700 1 --2- 172.21.15.42:0/891846901 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f083814cf50 0x7f083814f390 secure :-1 s=THROTTLE_DONE pgs=58 cs=0 l=1 rx=0x7f08380a9140 tx=0x7f083000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:31.397 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:31.384+0000 7f083e457700 1 --2- 172.21.15.42:0/891846901 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f083814cf50 0x7f083814f390 secure :-1 s=THROTTLE_DONE pgs=58 cs=0 l=1 rx=0x7f08380a9140 tx=0x7f083000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:31.397 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:31.384+0000 7f083cc54700 1 -- 172.21.15.42:0/891846901 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0830008bb0 con 0x7f083814cf50 2019-12-05T05:03:31.398 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:31.384+0000 7f083cc54700 1 -- 172.21.15.42:0/891846901 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0830011950 con 0x7f083814cf50 2019-12-05T05:03:31.398 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:31.384+0000 7f083f6b9700 1 -- 172.21.15.42:0/891846901 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f083814cf50 msgr2=0x7f083814f390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:31.398 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:31.384+0000 7f083f6b9700 1 --2- 172.21.15.42:0/891846901 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f083814cf50 0x7f083814f390 secure :-1 s=READY pgs=58 cs=0 l=1 rx=0x7f08380a9140 tx=0x7f083000fd30).stop 2019-12-05T05:03:31.398 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:31.384+0000 7f083f6b9700 1 -- 172.21.15.42:0/891846901 shutdown_connections 2019-12-05T05:03:31.398 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:31.384+0000 7f083f6b9700 1 --2- 172.21.15.42:0/891846901 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f083814cf50 0x7f083814f390 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:31.398 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:31.384+0000 7f083f6b9700 1 --2- 172.21.15.42:0/891846901 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f083809f550 0x7f0838144630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:03:31.398 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:31.384+0000 7f083f6b9700 1 -- 172.21.15.42:0/891846901 shutdown_connections 2019-12-05T05:03:31.399 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:31.384+0000 7f083f6b9700 1 -- 172.21.15.42:0/891846901 wait complete. 2019-12-05T05:03:31.399 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:31.384+0000 7f083f6b9700 1 -- 172.21.15.42:0/891846901 >> 172.21.15.42:0/891846901 conn(0x7f08380a9ef0 msgr2=0x7f08380aa320 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:31.399 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:31.388+0000 7f083f6b9700 1 Processor -- start 2019-12-05T05:03:31.399 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:31.388+0000 7f083f6b9700 1 -- start start 2019-12-05T05:03:31.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:31.388+0000 7f083f6b9700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0838120dd0 0x7f08380a58b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:31.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:31.388+0000 7f083f6b9700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f08380a6b90 0x7f08380a6fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:31.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:31.388+0000 7f083f6b9700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08380a7510 con 0x7f0838144230 2019-12-05T05:03:31.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:31.388+0000 7f083f6b9700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0838144cf0 con 0x7f0838120dd0 2019-12-05T05:03:31.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:31.388+0000 7f083f6b9700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08381447b0 con 0x7f08380a6b90 2019-12-05T05:03:31.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:31.388+0000 7f083dc56700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0838120dd0 0x7f08380a58b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:31.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:31.388+0000 7f083dc56700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0838120dd0 0x7f08380a58b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:53032/0 (socket says 172.21.15.42:53032) 2019-12-05T05:03:31.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:31.388+0000 7f083d455700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f0838144230 0x7f083807bbd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:53072/0 (socket says 172.21.15.42:53072) 2019-12-05T05:03:31.404 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:31.388+0000 7f083dc56700 1 -- 172.21.15.42:0/431043543 learned_addr learned my addr 172.21.15.42:0/431043543 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:03:31.404 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:31.388+0000 7f083e457700 1 --2- 172.21.15.42:0/431043543 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f08380a6b90 0x7f08380a6fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:31.404 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:31.388+0000 7f082a7fc700 1 -- 172.21.15.42:0/431043543 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 662608365 0 0) 0x7f08380a7510 con 0x7f0838144230 2019-12-05T05:03:31.404 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:31.388+0000 7f083dc56700 1 -- 172.21.15.42:0/431043543 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f08380a6b90 msgr2=0x7f08380a6fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:31.404 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:31.388+0000 7f083dc56700 1 --2- 172.21.15.42:0/431043543 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f08380a6b90 0x7f08380a6fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:31.404 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:31.388+0000 7f083dc56700 1 -- 172.21.15.42:0/431043543 >> v1:172.21.15.42:6789/0 conn(0x7f0838144230 legacy=0x7f083807bbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:31.404 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:31.388+0000 7f083dc56700 1 -- 172.21.15.42:0/431043543 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f083813aad0 con 0x7f0838120dd0 2019-12-05T05:03:31.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:31.388+0000 7f083dc56700 1 --2- 172.21.15.42:0/431043543 >> v2:172.21.15.42:3300/0 conn(0x7f0838120dd0 0x7f08380a58b0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f0838144970 tx=0x7f0834010c70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:31.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:31.388+0000 7f083dc56700 1 --2- 172.21.15.42:0/431043543 >> v2:172.21.15.42:3300/0 conn(0x7f0838120dd0 0x7f08380a58b0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f0838144970 tx=0x7f0834010c70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:31.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:31.388+0000 7f083dc56700 1 --2- 172.21.15.42:0/431043543 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0838120dd0 0x7f08380a58b0 secure :-1 s=READY pgs=60 cs=0 l=1 rx=0x7f0838144970 tx=0x7f0834010c70).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:31.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:31.392+0000 7f083dc56700 1 --2- 172.21.15.42:0/431043543 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0838120dd0 0x7f08380a58b0 secure :-1 s=THROTTLE_DONE pgs=60 cs=0 l=1 rx=0x7f0838144970 tx=0x7f0834010c70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:31.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:31.392+0000 7f082a7fc700 1 -- 172.21.15.42:0/431043543 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0834005930 con 0x7f0838120dd0 2019-12-05T05:03:31.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:31.392+0000 7f083dc56700 1 --2- 172.21.15.42:0/431043543 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0838120dd0 0x7f08380a58b0 secure :-1 s=THROTTLE_DONE pgs=60 cs=0 l=1 rx=0x7f0838144970 tx=0x7f0834010c70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:31.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:31.392+0000 7f083dc56700 1 --2- 172.21.15.42:0/431043543 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0838120dd0 0x7f08380a58b0 secure :-1 s=THROTTLE_DONE pgs=60 cs=0 l=1 rx=0x7f0838144970 tx=0x7f0834010c70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:31.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:31.392+0000 7f082a7fc700 1 -- 172.21.15.42:0/431043543 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0834007160 con 0x7f0838120dd0 2019-12-05T05:03:31.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:31.392+0000 7f083f6b9700 1 -- 172.21.15.42:0/431043543 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0838139ac0 con 0x7f0838120dd0 2019-12-05T05:03:31.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:31.392+0000 7f082a7fc700 1 -- 172.21.15.42:0/431043543 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f08340122b0 con 0x7f0838120dd0 2019-12-05T05:03:31.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:31.392+0000 7f083f6b9700 1 -- 172.21.15.42:0/431043543 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f083813a070 con 0x7f0838120dd0 2019-12-05T05:03:31.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:31.392+0000 7f083dc56700 1 --2- 172.21.15.42:0/431043543 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0838120dd0 0x7f08380a58b0 secure :-1 s=THROTTLE_DONE pgs=60 cs=0 l=1 rx=0x7f0838144970 tx=0x7f0834010c70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:31.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:31.392+0000 7f082a7fc700 1 -- 172.21.15.42:0/431043543 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f08340125c0 con 0x7f0838120dd0 2019-12-05T05:03:31.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:31.392+0000 7f083dc56700 1 --2- 172.21.15.42:0/431043543 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0838120dd0 0x7f08380a58b0 secure :-1 s=THROTTLE_DONE pgs=60 cs=0 l=1 rx=0x7f0838144970 tx=0x7f0834010c70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:31.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:31.392+0000 7f082a7fc700 1 --2- 172.21.15.42:0/431043543 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f082003ad20 0x7f082003d230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:31.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:31.392+0000 7f082a7fc700 1 -- 172.21.15.42:0/431043543 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5207+0+0 (secure 0 0 0) 0x7f083404b390 con 0x7f0838120dd0 2019-12-05T05:03:31.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:31.392+0000 7f083d455700 1 --2- 172.21.15.42:0/431043543 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f082003ad20 0x7f082003d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:31.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:31.392+0000 7f083d455700 1 --2- 172.21.15.42:0/431043543 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f082003ad20 0x7f082003d230 crc :-1 s=READY pgs=47 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:31.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:31.400+0000 7f083f6b9700 1 -- 172.21.15.42:0/431043543 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08380a5e00 con 0x7f0838120dd0 2019-12-05T05:03:31.412 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:31.408+0000 7f083dc56700 1 --2- 172.21.15.42:0/431043543 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0838120dd0 0x7f08380a58b0 secure :-1 s=THROTTLE_DONE pgs=60 cs=0 l=1 rx=0x7f0838144970 tx=0x7f0834010c70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:31.412 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:31.408+0000 7f082a7fc700 1 -- 172.21.15.42:0/431043543 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f0834032410 con 0x7f0838120dd0 2019-12-05T05:03:31.578 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:31.572+0000 7f083f6b9700 1 -- 172.21.15.42:0/431043543 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"profile": ["m=2", "k=2", "crush-failure-domain=osd"], "prefix": "osd erasure-code-profile set", "name": "cephfs_data"} v 0) v1 -- 0x7f083805d110 con 0x7f0838120dd0 2019-12-05T05:03:32.218 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:32.212+0000 7f083dc56700 1 --2- 172.21.15.42:0/431043543 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0838120dd0 0x7f08380a58b0 secure :-1 s=THROTTLE_DONE pgs=60 cs=0 l=1 rx=0x7f0838144970 tx=0x7f0834010c70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:32.218 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:32.212+0000 7f082a7fc700 1 -- 172.21.15.42:0/431043543 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"profile": ["m=2", "k=2", "crush-failure-domain=osd"], "prefix": "osd erasure-code-profile set", "name": "cephfs_data"}]=0 v19) v1 ==== 154+0+0 (secure 0 0 0) 0x7f0834032670 con 0x7f0838120dd0 2019-12-05T05:03:32.246 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:32.240+0000 7f083f6b9700 1 -- 172.21.15.42:0/431043543 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f082003ad20 msgr2=0x7f082003d230 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:32.247 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:32.240+0000 7f083f6b9700 1 --2- 172.21.15.42:0/431043543 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f082003ad20 0x7f082003d230 crc :-1 s=READY pgs=47 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:32.247 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:32.240+0000 7f083f6b9700 1 -- 172.21.15.42:0/431043543 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0838120dd0 msgr2=0x7f08380a58b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:32.247 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:32.240+0000 7f083f6b9700 1 --2- 172.21.15.42:0/431043543 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0838120dd0 0x7f08380a58b0 secure :-1 s=READY pgs=60 cs=0 l=1 rx=0x7f0838144970 tx=0x7f0834010c70).stop 2019-12-05T05:03:32.247 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:32.240+0000 7f083f6b9700 1 -- 172.21.15.42:0/431043543 shutdown_connections 2019-12-05T05:03:32.247 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:32.240+0000 7f083f6b9700 1 --2- 172.21.15.42:0/431043543 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f082003ad20 0x7f082003d230 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:32.247 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:32.240+0000 7f083f6b9700 1 --2- 172.21.15.42:0/431043543 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f08380a6b90 0x7f08380a6fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:32.248 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:32.240+0000 7f083f6b9700 1 --2- 172.21.15.42:0/431043543 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0838120dd0 0x7f08380a58b0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:32.248 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:32.240+0000 7f083f6b9700 1 -- 172.21.15.42:0/431043543 shutdown_connections 2019-12-05T05:03:32.248 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:32.240+0000 7f083f6b9700 1 -- 172.21.15.42:0/431043543 wait complete. 2019-12-05T05:03:32.248 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:32.240+0000 7f083f6b9700 1 -- 172.21.15.42:0/431043543 >> 172.21.15.42:0/431043543 conn(0x7f0838059ce0 msgr2=0x7f083805a300 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:32.258 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 8 erasure cephfs_data 2019-12-05T05:03:32.385 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:32.372+0000 7f9ac2c40700 1 Processor -- start 2019-12-05T05:03:32.385 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:32.372+0000 7f9ac2c40700 1 -- start start 2019-12-05T05:03:32.385 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:32.376+0000 7f9ac2c40700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9abc09f550 0x7f9abc144630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:32.385 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:32.376+0000 7f9ac2c40700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f9abc0a9a00 0x7f9abc07e880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:32.385 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:32.376+0000 7f9ac2c40700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9abc0a1000 con 0x7f9abc07edd0 2019-12-05T05:03:32.385 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:32.376+0000 7f9ac2c40700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9abc0a8f80 con 0x7f9abc0a9a00 2019-12-05T05:03:32.386 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:32.376+0000 7f9ac2c40700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9abc0a9220 con 0x7f9abc09f550 2019-12-05T05:03:32.386 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:32.376+0000 7f9ac09dc700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f9abc0a9a00 0x7f9abc07e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:32.386 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:32.376+0000 7f9ac09dc700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f9abc0a9a00 0x7f9abc07e880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:53040/0 (socket says 172.21.15.42:53040) 2019-12-05T05:03:32.387 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:32.376+0000 7f9ac09dc700 1 -- 172.21.15.42:0/3712154837 learned_addr learned my addr 172.21.15.42:0/3712154837 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:03:32.387 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:32.376+0000 7f9ac11dd700 1 --2- 172.21.15.42:0/3712154837 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9abc09f550 0x7f9abc144630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:32.387 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:32.376+0000 7f9ac09dc700 1 -- 172.21.15.42:0/3712154837 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9abc09f550 msgr2=0x7f9abc144630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:03:32.387 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:32.376+0000 7f9ac09dc700 1 --2- 172.21.15.42:0/3712154837 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9abc09f550 0x7f9abc144630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:03:32.387 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:32.376+0000 7f9ac09dc700 1 -- 172.21.15.42:0/3712154837 >> v1:172.21.15.42:6789/0 conn(0x7f9abc07edd0 legacy=0x7f9abc147b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:32.387 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:32.376+0000 7f9ac09dc700 1 -- 172.21.15.42:0/3712154837 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9abc144b80 con 0x7f9abc0a9a00 2019-12-05T05:03:32.387 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:32.376+0000 7f9ac09dc700 1 --2- 172.21.15.42:0/3712154837 >> v2:172.21.15.42:3300/0 conn(0x7f9abc0a9a00 0x7f9abc07e880 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f9abc0a9140 tx=0x7f9aac00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:32.388 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:32.376+0000 7f9ac09dc700 1 --2- 172.21.15.42:0/3712154837 >> v2:172.21.15.42:3300/0 conn(0x7f9abc0a9a00 0x7f9abc07e880 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f9abc0a9140 tx=0x7f9aac00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:32.388 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:32.376+0000 7f9ac09dc700 1 --2- 172.21.15.42:0/3712154837 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9abc0a9a00 0x7f9abc07e880 secure :-1 s=READY pgs=62 cs=0 l=1 rx=0x7f9abc0a9140 tx=0x7f9aac00fd30).ready entity=mon.1 client_cookie=f475acb5e168db80 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:32.388 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:32.376+0000 7f9ac09dc700 1 --2- 172.21.15.42:0/3712154837 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9abc0a9a00 0x7f9abc07e880 secure :-1 s=THROTTLE_DONE pgs=62 cs=0 l=1 rx=0x7f9abc0a9140 tx=0x7f9aac00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:32.388 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:32.376+0000 7f9ab3fff700 1 -- 172.21.15.42:0/3712154837 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9aac0113b0 con 0x7f9abc0a9a00 2019-12-05T05:03:32.388 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:32.376+0000 7f9ac09dc700 1 --2- 172.21.15.42:0/3712154837 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9abc0a9a00 0x7f9abc07e880 secure :-1 s=THROTTLE_DONE pgs=62 cs=0 l=1 rx=0x7f9abc0a9140 tx=0x7f9aac00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:32.388 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:32.376+0000 7f9ac09dc700 1 --2- 172.21.15.42:0/3712154837 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9abc0a9a00 0x7f9abc07e880 secure :-1 s=THROTTLE_DONE pgs=62 cs=0 l=1 rx=0x7f9abc0a9140 tx=0x7f9aac00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:32.388 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:32.376+0000 7f9ab3fff700 1 -- 172.21.15.42:0/3712154837 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9aac008bb0 con 0x7f9abc0a9a00 2019-12-05T05:03:32.389 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:32.376+0000 7f9ab3fff700 1 -- 172.21.15.42:0/3712154837 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9aac011990 con 0x7f9abc0a9a00 2019-12-05T05:03:32.389 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:32.376+0000 7f9ac2c40700 1 -- 172.21.15.42:0/3712154837 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9abc0a9a00 msgr2=0x7f9abc07e880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:32.389 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:32.376+0000 7f9ac2c40700 1 --2- 172.21.15.42:0/3712154837 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9abc0a9a00 0x7f9abc07e880 secure :-1 s=READY pgs=62 cs=0 l=1 rx=0x7f9abc0a9140 tx=0x7f9aac00fd30).stop 2019-12-05T05:03:32.389 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:32.376+0000 7f9ac2c40700 1 -- 172.21.15.42:0/3712154837 shutdown_connections 2019-12-05T05:03:32.389 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:32.376+0000 7f9ac2c40700 1 --2- 172.21.15.42:0/3712154837 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9abc09f550 0x7f9abc144630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:03:32.389 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:32.376+0000 7f9ac2c40700 1 --2- 172.21.15.42:0/3712154837 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9abc0a9a00 0x7f9abc07e880 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:32.389 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:32.376+0000 7f9ac2c40700 1 -- 172.21.15.42:0/3712154837 shutdown_connections 2019-12-05T05:03:32.390 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:32.376+0000 7f9ac2c40700 1 -- 172.21.15.42:0/3712154837 wait complete. 2019-12-05T05:03:32.390 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:32.376+0000 7f9ac2c40700 1 -- 172.21.15.42:0/3712154837 >> 172.21.15.42:0/3712154837 conn(0x7f9abc0a9ef0 msgr2=0x7f9abc0aa320 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:32.393 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:32.380+0000 7f9ac2c40700 1 Processor -- start 2019-12-05T05:03:32.393 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:32.380+0000 7f9ac2c40700 1 -- start start 2019-12-05T05:03:32.394 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:32.380+0000 7f9ac2c40700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9abc08f6f0 0x7f9abc0a2310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:32.394 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:32.380+0000 7f9ac2c40700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f9abc065490 0x7f9abc0a1880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:32.394 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:32.380+0000 7f9ac2c40700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9abc0a1dd0 con 0x7f9abc143510 2019-12-05T05:03:32.394 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:32.380+0000 7f9ac2c40700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9abc0a36a0 con 0x7f9abc065490 2019-12-05T05:03:32.394 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:32.380+0000 7f9ac2c40700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9abc0a4070 con 0x7f9abc08f6f0 2019-12-05T05:03:32.394 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:32.380+0000 7f9ac09dc700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f9abc065490 0x7f9abc0a1880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:32.394 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:32.384+0000 7f9ac09dc700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f9abc065490 0x7f9abc0a1880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:53046/0 (socket says 172.21.15.42:53046) 2019-12-05T05:03:32.395 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:32.384+0000 7f9ac09dc700 1 -- 172.21.15.42:0/3397259410 learned_addr learned my addr 172.21.15.42:0/3397259410 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:03:32.395 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:32.384+0000 7f9ac11dd700 1 --2- 172.21.15.42:0/3397259410 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9abc08f6f0 0x7f9abc0a2310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:32.395 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:32.384+0000 7f9ab1ffb700 1 -- 172.21.15.42:0/3397259410 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3496530943 0 0) 0x7f9abc0a1dd0 con 0x7f9abc143510 2019-12-05T05:03:32.395 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:32.384+0000 7f9ac09dc700 1 -- 172.21.15.42:0/3397259410 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9abc08f6f0 msgr2=0x7f9abc0a2310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:32.395 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:32.384+0000 7f9ac09dc700 1 --2- 172.21.15.42:0/3397259410 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9abc08f6f0 0x7f9abc0a2310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:32.395 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:32.384+0000 7f9ac09dc700 1 -- 172.21.15.42:0/3397259410 >> v1:172.21.15.40:6789/0 conn(0x7f9abc143510 legacy=0x7f9abc143940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:32.395 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:32.384+0000 7f9ac09dc700 1 -- 172.21.15.42:0/3397259410 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9abc0a3d30 con 0x7f9abc065490 2019-12-05T05:03:32.396 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:32.384+0000 7f9ac09dc700 1 --2- 172.21.15.42:0/3397259410 >> v2:172.21.15.42:3300/0 conn(0x7f9abc065490 0x7f9abc0a1880 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f9abc0a9140 tx=0x7f9aac011f10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:32.396 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:32.384+0000 7f9ac09dc700 1 --2- 172.21.15.42:0/3397259410 >> v2:172.21.15.42:3300/0 conn(0x7f9abc065490 0x7f9abc0a1880 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f9abc0a9140 tx=0x7f9aac011f10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:32.396 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:32.384+0000 7f9ac09dc700 1 --2- 172.21.15.42:0/3397259410 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9abc065490 0x7f9abc0a1880 secure :-1 s=READY pgs=63 cs=0 l=1 rx=0x7f9abc0a9140 tx=0x7f9aac011f10).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:32.396 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:32.384+0000 7f9ac09dc700 1 --2- 172.21.15.42:0/3397259410 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9abc065490 0x7f9abc0a1880 secure :-1 s=THROTTLE_DONE pgs=63 cs=0 l=1 rx=0x7f9abc0a9140 tx=0x7f9aac011f10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:32.396 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:32.384+0000 7f9ab1ffb700 1 -- 172.21.15.42:0/3397259410 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9aac007340 con 0x7f9abc065490 2019-12-05T05:03:32.396 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:32.384+0000 7f9ac09dc700 1 --2- 172.21.15.42:0/3397259410 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9abc065490 0x7f9abc0a1880 secure :-1 s=THROTTLE_DONE pgs=63 cs=0 l=1 rx=0x7f9abc0a9140 tx=0x7f9aac011f10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:32.396 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:32.384+0000 7f9ac09dc700 1 --2- 172.21.15.42:0/3397259410 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9abc065490 0x7f9abc0a1880 secure :-1 s=THROTTLE_DONE pgs=63 cs=0 l=1 rx=0x7f9abc0a9140 tx=0x7f9aac011f10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:32.397 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:32.384+0000 7f9ab1ffb700 1 -- 172.21.15.42:0/3397259410 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9aac0115b0 con 0x7f9abc065490 2019-12-05T05:03:32.397 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:32.384+0000 7f9ab1ffb700 1 -- 172.21.15.42:0/3397259410 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9aac005540 con 0x7f9abc065490 2019-12-05T05:03:32.397 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:32.384+0000 7f9ac2c40700 1 -- 172.21.15.42:0/3397259410 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9abc1448d0 con 0x7f9abc065490 2019-12-05T05:03:32.397 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:32.384+0000 7f9ac2c40700 1 -- 172.21.15.42:0/3397259410 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9abc05fdc0 con 0x7f9abc065490 2019-12-05T05:03:32.397 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:32.384+0000 7f9ac09dc700 1 --2- 172.21.15.42:0/3397259410 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9abc065490 0x7f9abc0a1880 secure :-1 s=THROTTLE_DONE pgs=63 cs=0 l=1 rx=0x7f9abc0a9140 tx=0x7f9aac011f10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:32.397 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:32.384+0000 7f9ab1ffb700 1 -- 172.21.15.42:0/3397259410 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f9aac005810 con 0x7f9abc065490 2019-12-05T05:03:32.398 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:32.384+0000 7f9ac09dc700 1 --2- 172.21.15.42:0/3397259410 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9abc065490 0x7f9abc0a1880 secure :-1 s=THROTTLE_DONE pgs=63 cs=0 l=1 rx=0x7f9abc0a9140 tx=0x7f9aac011f10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:32.398 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:32.384+0000 7f9ab1ffb700 1 --2- 172.21.15.42:0/3397259410 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9a9c03cef0 0x7f9a9c03f400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:32.398 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:32.384+0000 7f9ab1ffb700 1 -- 172.21.15.42:0/3397259410 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5463+0+0 (secure 0 0 0) 0x7f9aac04b330 con 0x7f9abc065490 2019-12-05T05:03:32.398 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:32.388+0000 7f9ac11dd700 1 --2- 172.21.15.42:0/3397259410 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9a9c03cef0 0x7f9a9c03f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:32.398 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:32.388+0000 7f9ac11dd700 1 --2- 172.21.15.42:0/3397259410 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9a9c03cef0 0x7f9a9c03f400 crc :-1 s=READY pgs=48 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:32.398 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:32.388+0000 7f9ac2c40700 1 -- 172.21.15.42:0/3397259410 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9abc05d110 con 0x7f9abc065490 2019-12-05T05:03:32.399 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:32.392+0000 7f9ac09dc700 1 --2- 172.21.15.42:0/3397259410 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9abc065490 0x7f9abc0a1880 secure :-1 s=THROTTLE_DONE pgs=63 cs=0 l=1 rx=0x7f9abc0a9140 tx=0x7f9aac011f10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:32.399 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:32.392+0000 7f9ab1ffb700 1 -- 172.21.15.42:0/3397259410 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f9aac0323b0 con 0x7f9abc065490 2019-12-05T05:03:32.561 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:32.556+0000 7f9ac2c40700 1 -- 172.21.15.42:0/3397259410 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"pool_type": "erasure", "prefix": "osd pool create", "pg_num": 8, "erasure_code_profile": "cephfs_data", "pool": "cephfs_data"} v 0) v1 -- 0x7f9abc07e2e0 con 0x7f9abc065490 2019-12-05T05:03:33.490 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:03:33.494 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:03:34.250 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:34.244+0000 7f9ac09dc700 1 --2- 172.21.15.42:0/3397259410 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9abc065490 0x7f9abc0a1880 secure :-1 s=THROTTLE_DONE pgs=63 cs=0 l=1 rx=0x7f9abc0a9140 tx=0x7f9aac011f10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:34.250 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:34.244+0000 7f9ab1ffb700 1 -- 172.21.15.42:0/3397259410 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"pool_type": "erasure", "prefix": "osd pool create", "pg_num": 8, "erasure_code_profile": "cephfs_data", "pool": "cephfs_data"}]=0 pool 'cephfs_data' created v21) v1 ==== 188+0+0 (secure 0 0 0) 0x7f9aac032610 con 0x7f9abc065490 2019-12-05T05:03:34.279 INFO:teuthology.orchestra.run.smithi042.stderr:pool 'cephfs_data' created 2019-12-05T05:03:34.283 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:34.276+0000 7f9ac2c40700 1 -- 172.21.15.42:0/3397259410 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9a9c03cef0 msgr2=0x7f9a9c03f400 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:34.283 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:34.276+0000 7f9ac2c40700 1 --2- 172.21.15.42:0/3397259410 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9a9c03cef0 0x7f9a9c03f400 crc :-1 s=READY pgs=48 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:34.283 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:34.276+0000 7f9ac2c40700 1 -- 172.21.15.42:0/3397259410 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9abc065490 msgr2=0x7f9abc0a1880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:34.284 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:34.276+0000 7f9ac2c40700 1 --2- 172.21.15.42:0/3397259410 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9abc065490 0x7f9abc0a1880 secure :-1 s=READY pgs=63 cs=0 l=1 rx=0x7f9abc0a9140 tx=0x7f9aac011f10).stop 2019-12-05T05:03:34.285 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:34.276+0000 7f9ac2c40700 1 -- 172.21.15.42:0/3397259410 shutdown_connections 2019-12-05T05:03:34.285 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:34.276+0000 7f9ac2c40700 1 --2- 172.21.15.42:0/3397259410 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9a9c03cef0 0x7f9a9c03f400 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:34.285 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:34.276+0000 7f9ac2c40700 1 --2- 172.21.15.42:0/3397259410 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9abc08f6f0 0x7f9abc0a2310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:34.285 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:34.276+0000 7f9ac2c40700 1 --2- 172.21.15.42:0/3397259410 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9abc065490 0x7f9abc0a1880 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:34.285 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:34.280+0000 7f9ac2c40700 1 -- 172.21.15.42:0/3397259410 shutdown_connections 2019-12-05T05:03:34.285 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:34.280+0000 7f9ac2c40700 1 -- 172.21.15.42:0/3397259410 wait complete. 2019-12-05T05:03:34.285 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:34.280+0000 7f9ac2c40700 1 -- 172.21.15.42:0/3397259410 >> 172.21.15.42:0/3397259410 conn(0x7f9abc0a6440 msgr2=0x7f9abc0a69b0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:34.297 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data allow_ec_overwrites true 2019-12-05T05:03:34.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:34.396+0000 7fc132543700 1 Processor -- start 2019-12-05T05:03:34.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:34.396+0000 7fc132543700 1 -- start start 2019-12-05T05:03:34.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:34.396+0000 7fc132543700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc12c0a9a00 0x7fc12c07e880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:34.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:34.396+0000 7fc132543700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc12c07edd0 0x7fc12c14ef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:34.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:34.396+0000 7fc132543700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc12c0a1000 con 0x7fc12c09f550 2019-12-05T05:03:34.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:34.396+0000 7fc132543700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc12c0a8f80 con 0x7fc12c0a9a00 2019-12-05T05:03:34.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:34.396+0000 7fc132543700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc12c0a9220 con 0x7fc12c07edd0 2019-12-05T05:03:34.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:34.396+0000 7fc1312e1700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc12c07edd0 0x7fc12c14ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:34.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:34.400+0000 7fc12bfff700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc12c0a9a00 0x7fc12c07e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:34.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:34.400+0000 7fc1312e1700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc12c07edd0 0x7fc12c14ef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:53064/0 (socket says 172.21.15.42:53064) 2019-12-05T05:03:34.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:34.400+0000 7fc1312e1700 1 -- 172.21.15.42:0/3139809406 learned_addr learned my addr 172.21.15.42:0/3139809406 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:03:34.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:34.400+0000 7fc12bfff700 1 -- 172.21.15.42:0/3139809406 >> v2:172.21.15.42:3300/0 conn(0x7fc12c07edd0 msgr2=0x7fc12c14ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:03:34.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:34.400+0000 7fc12bfff700 1 --2- 172.21.15.42:0/3139809406 >> v2:172.21.15.42:3300/0 conn(0x7fc12c07edd0 0x7fc12c14ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:03:34.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:34.400+0000 7fc12bfff700 1 -- 172.21.15.42:0/3139809406 >> v1:172.21.15.40:6789/0 conn(0x7fc12c09f550 legacy=0x7fc12c144630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:34.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:34.400+0000 7fc12bfff700 1 -- 172.21.15.42:0/3139809406 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc12c09ee30 con 0x7fc12c0a9a00 2019-12-05T05:03:34.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:34.400+0000 7fc1312e1700 1 --2- 172.21.15.42:0/3139809406 >> v2:172.21.15.42:3300/0 conn(0x7fc12c07edd0 0x7fc12c14ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:03:34.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:34.400+0000 7fc12bfff700 1 --2- 172.21.15.42:0/3139809406 >> v2:172.21.15.40:3300/0 conn(0x7fc12c0a9a00 0x7fc12c07e880 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fc12c0a9140 tx=0x7fc12400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:34.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:34.400+0000 7fc12bfff700 1 --2- 172.21.15.42:0/3139809406 >> v2:172.21.15.40:3300/0 conn(0x7fc12c0a9a00 0x7fc12c07e880 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fc12c0a9140 tx=0x7fc12400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:34.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:34.400+0000 7fc12bfff700 1 --2- 172.21.15.42:0/3139809406 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc12c0a9a00 0x7fc12c07e880 secure :-1 s=READY pgs=64 cs=0 l=1 rx=0x7fc12c0a9140 tx=0x7fc12400fc70).ready entity=mon.0 client_cookie=caaa558d3f074221 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:34.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:34.400+0000 7fc12bfff700 1 --2- 172.21.15.42:0/3139809406 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc12c0a9a00 0x7fc12c07e880 secure :-1 s=THROTTLE_DONE pgs=64 cs=0 l=1 rx=0x7fc12c0a9140 tx=0x7fc12400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:34.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:34.400+0000 7fc12bfff700 1 --2- 172.21.15.42:0/3139809406 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc12c0a9a00 0x7fc12c07e880 secure :-1 s=THROTTLE_DONE pgs=64 cs=0 l=1 rx=0x7fc12c0a9140 tx=0x7fc12400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:34.411 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:34.400+0000 7fc12b7fe700 1 -- 172.21.15.42:0/3139809406 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc124008bb0 con 0x7fc12c0a9a00 2019-12-05T05:03:34.411 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:34.400+0000 7fc12bfff700 1 --2- 172.21.15.42:0/3139809406 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc12c0a9a00 0x7fc12c07e880 secure :-1 s=THROTTLE_DONE pgs=64 cs=0 l=1 rx=0x7fc12c0a9140 tx=0x7fc12400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:34.411 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:34.400+0000 7fc12b7fe700 1 -- 172.21.15.42:0/3139809406 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1240087d0 con 0x7fc12c0a9a00 2019-12-05T05:03:34.411 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:34.400+0000 7fc12b7fe700 1 -- 172.21.15.42:0/3139809406 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc124011820 con 0x7fc12c0a9a00 2019-12-05T05:03:34.411 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:34.400+0000 7fc132543700 1 -- 172.21.15.42:0/3139809406 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc12c0a9a00 msgr2=0x7fc12c07e880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:34.411 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:34.400+0000 7fc132543700 1 --2- 172.21.15.42:0/3139809406 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc12c0a9a00 0x7fc12c07e880 secure :-1 s=READY pgs=64 cs=0 l=1 rx=0x7fc12c0a9140 tx=0x7fc12400fc70).stop 2019-12-05T05:03:34.411 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:34.400+0000 7fc132543700 1 -- 172.21.15.42:0/3139809406 shutdown_connections 2019-12-05T05:03:34.412 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:34.400+0000 7fc132543700 1 --2- 172.21.15.42:0/3139809406 >> v2:172.21.15.42:3300/0 conn(0x7fc12c07edd0 0x7fc12c14ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:03:34.412 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:34.400+0000 7fc132543700 1 --2- 172.21.15.42:0/3139809406 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc12c0a9a00 0x7fc12c07e880 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:34.412 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:34.400+0000 7fc132543700 1 -- 172.21.15.42:0/3139809406 shutdown_connections 2019-12-05T05:03:34.412 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:34.400+0000 7fc132543700 1 -- 172.21.15.42:0/3139809406 wait complete. 2019-12-05T05:03:34.412 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:34.400+0000 7fc132543700 1 -- 172.21.15.42:0/3139809406 >> 172.21.15.42:0/3139809406 conn(0x7fc12c0a9ef0 msgr2=0x7fc12c0aa320 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:34.419 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:34.404+0000 7fc132543700 1 Processor -- start 2019-12-05T05:03:34.419 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:34.404+0000 7fc132543700 1 -- start start 2019-12-05T05:03:34.419 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:34.404+0000 7fc132543700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc12c0a4ea0 0x7fc12c12a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:34.420 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:34.404+0000 7fc132543700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc12c143af0 0x7fc12c1267a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:34.420 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:34.404+0000 7fc132543700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc12c143f20 con 0x7fc12c143130 2019-12-05T05:03:34.420 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:34.404+0000 7fc132543700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc12c126cf0 con 0x7fc12c0a4ea0 2019-12-05T05:03:34.420 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:34.404+0000 7fc132543700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc12c129d60 con 0x7fc12c143af0 2019-12-05T05:03:34.420 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:34.404+0000 7fc130ae0700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc12c0a4ea0 0x7fc12c12a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:34.420 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:34.404+0000 7fc130ae0700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc12c0a4ea0 0x7fc12c12a7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:53066/0 (socket says 172.21.15.42:53066) 2019-12-05T05:03:34.420 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:34.404+0000 7fc130ae0700 1 -- 172.21.15.42:0/1159571054 learned_addr learned my addr 172.21.15.42:0/1159571054 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:03:34.421 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:34.404+0000 7fc1312e1700 1 --2- 172.21.15.42:0/1159571054 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc12c143af0 0x7fc12c1267a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:34.421 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:34.404+0000 7fc130ae0700 1 -- 172.21.15.42:0/1159571054 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc12c143af0 msgr2=0x7fc12c1267a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:34.421 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:34.404+0000 7fc130ae0700 1 --2- 172.21.15.42:0/1159571054 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc12c143af0 0x7fc12c1267a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:34.421 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:34.404+0000 7fc130ae0700 1 -- 172.21.15.42:0/1159571054 >> v1:172.21.15.40:6789/0 conn(0x7fc12c143130 legacy=0x7fc12c064f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:34.421 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:34.404+0000 7fc1297fa700 1 -- 172.21.15.42:0/1159571054 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1298153780 0 0) 0x7fc12c143f20 con 0x7fc12c143130 2019-12-05T05:03:34.421 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:34.404+0000 7fc130ae0700 1 -- 172.21.15.42:0/1159571054 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc12c065730 con 0x7fc12c0a4ea0 2019-12-05T05:03:34.421 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:34.408+0000 7fc130ae0700 1 --2- 172.21.15.42:0/1159571054 >> v2:172.21.15.42:3300/0 conn(0x7fc12c0a4ea0 0x7fc12c12a7e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fc114001210 tx=0x7fc114010d20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:34.422 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:34.408+0000 7fc130ae0700 1 --2- 172.21.15.42:0/1159571054 >> v2:172.21.15.42:3300/0 conn(0x7fc12c0a4ea0 0x7fc12c12a7e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fc114001210 tx=0x7fc114010d20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:34.422 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:34.408+0000 7fc130ae0700 1 --2- 172.21.15.42:0/1159571054 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc12c0a4ea0 0x7fc12c12a7e0 secure :-1 s=READY pgs=64 cs=0 l=1 rx=0x7fc114001210 tx=0x7fc114010d20).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:34.422 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:34.408+0000 7fc130ae0700 1 --2- 172.21.15.42:0/1159571054 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc12c0a4ea0 0x7fc12c12a7e0 secure :-1 s=THROTTLE_DONE pgs=64 cs=0 l=1 rx=0x7fc114001210 tx=0x7fc114010d20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:34.422 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:34.408+0000 7fc1297fa700 1 -- 172.21.15.42:0/1159571054 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc114010550 con 0x7fc12c0a4ea0 2019-12-05T05:03:34.422 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:34.408+0000 7fc130ae0700 1 --2- 172.21.15.42:0/1159571054 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc12c0a4ea0 0x7fc12c12a7e0 secure :-1 s=THROTTLE_DONE pgs=64 cs=0 l=1 rx=0x7fc114001210 tx=0x7fc114010d20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:34.422 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:34.408+0000 7fc130ae0700 1 --2- 172.21.15.42:0/1159571054 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc12c0a4ea0 0x7fc12c12a7e0 secure :-1 s=THROTTLE_DONE pgs=64 cs=0 l=1 rx=0x7fc114001210 tx=0x7fc114010d20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:34.423 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:34.408+0000 7fc1297fa700 1 -- 172.21.15.42:0/1159571054 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc114010170 con 0x7fc12c0a4ea0 2019-12-05T05:03:34.423 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:34.408+0000 7fc1297fa700 1 -- 172.21.15.42:0/1159571054 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc11400cb20 con 0x7fc12c0a4ea0 2019-12-05T05:03:34.423 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:34.408+0000 7fc132543700 1 -- 172.21.15.42:0/1159571054 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc12c1311b0 con 0x7fc12c0a4ea0 2019-12-05T05:03:34.423 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:34.408+0000 7fc132543700 1 -- 172.21.15.42:0/1159571054 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc12c1316e0 con 0x7fc12c0a4ea0 2019-12-05T05:03:34.423 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:34.408+0000 7fc130ae0700 1 --2- 172.21.15.42:0/1159571054 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc12c0a4ea0 0x7fc12c12a7e0 secure :-1 s=THROTTLE_DONE pgs=64 cs=0 l=1 rx=0x7fc114001210 tx=0x7fc114010d20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:34.423 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:34.408+0000 7fc1297fa700 1 -- 172.21.15.42:0/1159571054 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fc1140066e0 con 0x7fc12c0a4ea0 2019-12-05T05:03:34.423 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:34.408+0000 7fc130ae0700 1 --2- 172.21.15.42:0/1159571054 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc12c0a4ea0 0x7fc12c12a7e0 secure :-1 s=THROTTLE_DONE pgs=64 cs=0 l=1 rx=0x7fc114001210 tx=0x7fc114010d20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:34.424 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:34.408+0000 7fc1297fa700 1 --2- 172.21.15.42:0/1159571054 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc10403acf0 0x7fc10403d200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:34.424 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:34.408+0000 7fc1297fa700 1 -- 172.21.15.42:0/1159571054 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5915+0+0 (secure 0 0 0) 0x7fc11404b440 con 0x7fc12c0a4ea0 2019-12-05T05:03:34.424 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:34.408+0000 7fc12bfff700 1 --2- 172.21.15.42:0/1159571054 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc10403acf0 0x7fc10403d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:34.424 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:34.408+0000 7fc12bfff700 1 --2- 172.21.15.42:0/1159571054 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc10403acf0 0x7fc10403d200 crc :-1 s=READY pgs=49 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:34.424 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:34.412+0000 7fc132543700 1 -- 172.21.15.42:0/1159571054 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc12c05d110 con 0x7fc12c0a4ea0 2019-12-05T05:03:34.427 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:34.420+0000 7fc130ae0700 1 --2- 172.21.15.42:0/1159571054 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc12c0a4ea0 0x7fc12c12a7e0 secure :-1 s=THROTTLE_DONE pgs=64 cs=0 l=1 rx=0x7fc114001210 tx=0x7fc114010d20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:34.427 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:34.420+0000 7fc1297fa700 1 -- 172.21.15.42:0/1159571054 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fc114032440 con 0x7fc12c0a4ea0 2019-12-05T05:03:34.592 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:34.588+0000 7fc132543700 1 -- 172.21.15.42:0/1159571054 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"var": "allow_ec_overwrites", "prefix": "osd pool set", "pool": "cephfs_data", "val": "true"} v 0) v1 -- 0x7fc12c05b8c0 con 0x7fc12c0a4ea0 2019-12-05T05:03:35.261 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.256+0000 7fc130ae0700 1 --2- 172.21.15.42:0/1159571054 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc12c0a4ea0 0x7fc12c12a7e0 secure :-1 s=THROTTLE_DONE pgs=64 cs=0 l=1 rx=0x7fc114001210 tx=0x7fc114010d20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:35.262 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.256+0000 7fc1297fa700 1 -- 172.21.15.42:0/1159571054 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"var": "allow_ec_overwrites", "prefix": "osd pool set", "pool": "cephfs_data", "val": "true"}]=0 set pool 3 allow_ec_overwrites to true v22) v1 ==== 166+0+0 (secure 0 0 0) 0x7fc1140326a0 con 0x7fc12c0a4ea0 2019-12-05T05:03:35.308 INFO:teuthology.orchestra.run.smithi042.stderr:set pool 3 allow_ec_overwrites to true 2019-12-05T05:03:35.311 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.304+0000 7fc132543700 1 -- 172.21.15.42:0/1159571054 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc10403acf0 msgr2=0x7fc10403d200 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:35.312 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.304+0000 7fc132543700 1 --2- 172.21.15.42:0/1159571054 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc10403acf0 0x7fc10403d200 crc :-1 s=READY pgs=49 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:35.312 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.304+0000 7fc132543700 1 -- 172.21.15.42:0/1159571054 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc12c0a4ea0 msgr2=0x7fc12c12a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:35.312 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.304+0000 7fc132543700 1 --2- 172.21.15.42:0/1159571054 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc12c0a4ea0 0x7fc12c12a7e0 secure :-1 s=READY pgs=64 cs=0 l=1 rx=0x7fc114001210 tx=0x7fc114010d20).stop 2019-12-05T05:03:35.312 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.304+0000 7fc132543700 1 -- 172.21.15.42:0/1159571054 shutdown_connections 2019-12-05T05:03:35.312 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.304+0000 7fc132543700 1 --2- 172.21.15.42:0/1159571054 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc10403acf0 0x7fc10403d200 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:35.312 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.304+0000 7fc132543700 1 --2- 172.21.15.42:0/1159571054 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc12c143af0 0x7fc12c1267a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:35.313 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.304+0000 7fc132543700 1 --2- 172.21.15.42:0/1159571054 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc12c0a4ea0 0x7fc12c12a7e0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:35.313 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.304+0000 7fc132543700 1 -- 172.21.15.42:0/1159571054 shutdown_connections 2019-12-05T05:03:35.313 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.308+0000 7fc132543700 1 -- 172.21.15.42:0/1159571054 wait complete. 2019-12-05T05:03:35.313 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.308+0000 7fc132543700 1 -- 172.21.15.42:0/1159571054 >> 172.21.15.42:0/1159571054 conn(0x7fc12c05a4f0 msgr2=0x7fc12c0a5fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:35.325 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2019-12-05T05:03:35.428 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.416+0000 7f61b7af9700 1 Processor -- start 2019-12-05T05:03:35.428 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.416+0000 7f61b7af9700 1 -- start start 2019-12-05T05:03:35.428 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.420+0000 7f61b7af9700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f61b00a9a00 0x7f61b007e880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:35.428 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.420+0000 7f61b7af9700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61b00a1000 con 0x7f61b007edd0 2019-12-05T05:03:35.428 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.420+0000 7f61b7af9700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61b009ee30 con 0x7f61b009f550 2019-12-05T05:03:35.429 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.420+0000 7f61b7af9700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61b00a8f80 con 0x7f61b00a9a00 2019-12-05T05:03:35.429 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.420+0000 7f61b6096700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f61b009f550 0x7f61b0144630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:53120/0 (socket says 172.21.15.42:53120) 2019-12-05T05:03:35.429 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.420+0000 7f61b6096700 1 -- 172.21.15.42:0/3145911990 learned_addr learned my addr 172.21.15.42:0/3145911990 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:03:35.429 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.420+0000 7f61b5895700 1 --2- 172.21.15.42:0/3145911990 >> v2:172.21.15.40:3300/0 conn(0x7f61b00a9a00 0x7f61b007e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:35.429 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.420+0000 7f61b5094700 1 -- 172.21.15.42:0/3145911990 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 236502255 0 0) 0x7f61b009ee30 con 0x7f61b009f550 2019-12-05T05:03:35.429 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.420+0000 7f61b5094700 1 -- 172.21.15.42:0/3145911990 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f619c0030b0 con 0x7f61b009f550 2019-12-05T05:03:35.429 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.420+0000 7f61b5895700 1 -- 172.21.15.42:0/3145911990 >> v1:172.21.15.42:6789/0 conn(0x7f61b009f550 legacy=0x7f61b0144630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:35.430 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.420+0000 7f61b5895700 1 -- 172.21.15.42:0/3145911990 >> v1:172.21.15.40:6789/0 conn(0x7f61b007edd0 legacy=0x7f61b014ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:35.430 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.420+0000 7f61b5895700 1 -- 172.21.15.42:0/3145911990 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61b008dd20 con 0x7f61b00a9a00 2019-12-05T05:03:35.430 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.420+0000 7f61b5895700 1 --2- 172.21.15.42:0/3145911990 >> v2:172.21.15.40:3300/0 conn(0x7f61b00a9a00 0x7f61b007e880 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f61a4002f30 tx=0x7f61a400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:35.430 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.420+0000 7f61b5895700 1 --2- 172.21.15.42:0/3145911990 >> v2:172.21.15.40:3300/0 conn(0x7f61b00a9a00 0x7f61b007e880 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f61a4002f30 tx=0x7f61a400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:35.430 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.420+0000 7f61b5895700 1 --2- 172.21.15.42:0/3145911990 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f61b00a9a00 0x7f61b007e880 secure :-1 s=READY pgs=68 cs=0 l=1 rx=0x7f61a4002f30 tx=0x7f61a400fbf0).ready entity=mon.0 client_cookie=77908da708472a29 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:35.430 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.420+0000 7f61b5895700 1 --2- 172.21.15.42:0/3145911990 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f61b00a9a00 0x7f61b007e880 secure :-1 s=THROTTLE_DONE pgs=68 cs=0 l=1 rx=0x7f61a4002f30 tx=0x7f61a400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:35.430 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.420+0000 7f61b5094700 1 -- 172.21.15.42:0/3145911990 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f61a4011230 con 0x7f61b00a9a00 2019-12-05T05:03:35.431 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.420+0000 7f61b5895700 1 --2- 172.21.15.42:0/3145911990 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f61b00a9a00 0x7f61b007e880 secure :-1 s=THROTTLE_DONE pgs=68 cs=0 l=1 rx=0x7f61a4002f30 tx=0x7f61a400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:35.431 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.420+0000 7f61b5895700 1 --2- 172.21.15.42:0/3145911990 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f61b00a9a00 0x7f61b007e880 secure :-1 s=THROTTLE_DONE pgs=68 cs=0 l=1 rx=0x7f61a4002f30 tx=0x7f61a400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:35.431 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.420+0000 7f61b5094700 1 -- 172.21.15.42:0/3145911990 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61a4008bc0 con 0x7f61b00a9a00 2019-12-05T05:03:35.431 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.420+0000 7f61b5094700 1 -- 172.21.15.42:0/3145911990 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f61a4011810 con 0x7f61b00a9a00 2019-12-05T05:03:35.431 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.420+0000 7f61b7af9700 1 -- 172.21.15.42:0/3145911990 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f61b00a9a00 msgr2=0x7f61b007e880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:35.431 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.420+0000 7f61b7af9700 1 --2- 172.21.15.42:0/3145911990 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f61b00a9a00 0x7f61b007e880 secure :-1 s=READY pgs=68 cs=0 l=1 rx=0x7f61a4002f30 tx=0x7f61a400fbf0).stop 2019-12-05T05:03:35.431 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.420+0000 7f61b7af9700 1 -- 172.21.15.42:0/3145911990 shutdown_connections 2019-12-05T05:03:35.432 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.420+0000 7f61b7af9700 1 --2- 172.21.15.42:0/3145911990 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f61b00a9a00 0x7f61b007e880 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:35.432 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.420+0000 7f61b7af9700 1 -- 172.21.15.42:0/3145911990 shutdown_connections 2019-12-05T05:03:35.432 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.420+0000 7f61b7af9700 1 -- 172.21.15.42:0/3145911990 wait complete. 2019-12-05T05:03:35.432 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.420+0000 7f61b7af9700 1 -- 172.21.15.42:0/3145911990 >> 172.21.15.42:0/3145911990 conn(0x7f61b00a9ef0 msgr2=0x7f61b00aa320 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:35.435 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.424+0000 7f61b7af9700 1 Processor -- start 2019-12-05T05:03:35.435 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.424+0000 7f61b7af9700 1 -- start start 2019-12-05T05:03:35.435 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.424+0000 7f61b7af9700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f61b014fe70 0x7f61b0143890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:35.436 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.424+0000 7f61b7af9700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61b01467b0 con 0x7f61b01429e0 2019-12-05T05:03:35.436 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.424+0000 7f61b7af9700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61b0144210 con 0x7f61b0143de0 2019-12-05T05:03:35.436 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.424+0000 7f61b7af9700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61b0147260 con 0x7f61b014fe70 2019-12-05T05:03:35.436 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.424+0000 7f61b5895700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f61b014fe70 0x7f61b0143890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:35.436 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.424+0000 7f61b5895700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f61b014fe70 0x7f61b0143890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:53090/0 (socket says 172.21.15.42:53090) 2019-12-05T05:03:35.436 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.424+0000 7f61b5895700 1 -- 172.21.15.42:0/3198697527 learned_addr learned my addr 172.21.15.42:0/3198697527 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:03:35.437 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.424+0000 7f61a2ffd700 1 -- 172.21.15.42:0/3198697527 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 430856534 0 0) 0x7f61b01467b0 con 0x7f61b01429e0 2019-12-05T05:03:35.437 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.424+0000 7f61a2ffd700 1 -- 172.21.15.42:0/3198697527 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61980030b0 con 0x7f61b01429e0 2019-12-05T05:03:35.437 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.424+0000 7f61b5895700 1 -- 172.21.15.42:0/3198697527 >> v1:172.21.15.42:6789/0 conn(0x7f61b0143de0 legacy=0x7f61b012de20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:35.437 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.424+0000 7f61b5895700 1 -- 172.21.15.42:0/3198697527 >> v1:172.21.15.40:6789/0 conn(0x7f61b01429e0 legacy=0x7f61b007dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:35.437 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.424+0000 7f61b5895700 1 -- 172.21.15.42:0/3198697527 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61b0142e10 con 0x7f61b014fe70 2019-12-05T05:03:35.437 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.424+0000 7f61b5895700 1 --2- 172.21.15.42:0/3198697527 >> v2:172.21.15.42:3300/0 conn(0x7f61b014fe70 0x7f61b0143890 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f61b0146030 tx=0x7f61a4008e50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:35.437 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.424+0000 7f61b5895700 1 --2- 172.21.15.42:0/3198697527 >> v2:172.21.15.42:3300/0 conn(0x7f61b014fe70 0x7f61b0143890 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f61b0146030 tx=0x7f61a4008e50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:35.438 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.424+0000 7f61b5895700 1 --2- 172.21.15.42:0/3198697527 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f61b014fe70 0x7f61b0143890 secure :-1 s=READY pgs=69 cs=0 l=1 rx=0x7f61b0146030 tx=0x7f61a4008e50).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:35.438 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.424+0000 7f61b5895700 1 --2- 172.21.15.42:0/3198697527 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f61b014fe70 0x7f61b0143890 secure :-1 s=THROTTLE_DONE pgs=69 cs=0 l=1 rx=0x7f61b0146030 tx=0x7f61a4008e50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:35.438 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.424+0000 7f61a2ffd700 1 -- 172.21.15.42:0/3198697527 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f61a4007150 con 0x7f61b014fe70 2019-12-05T05:03:35.438 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.424+0000 7f61b5895700 1 --2- 172.21.15.42:0/3198697527 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f61b014fe70 0x7f61b0143890 secure :-1 s=THROTTLE_DONE pgs=69 cs=0 l=1 rx=0x7f61b0146030 tx=0x7f61a4008e50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:35.438 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.424+0000 7f61b5895700 1 --2- 172.21.15.42:0/3198697527 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f61b014fe70 0x7f61b0143890 secure :-1 s=THROTTLE_DONE pgs=69 cs=0 l=1 rx=0x7f61b0146030 tx=0x7f61a4008e50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:35.438 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.424+0000 7f61a2ffd700 1 -- 172.21.15.42:0/3198697527 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61a4011430 con 0x7f61b014fe70 2019-12-05T05:03:35.438 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.424+0000 7f61b7af9700 1 -- 172.21.15.42:0/3198697527 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61b01502a0 con 0x7f61b014fe70 2019-12-05T05:03:35.439 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.428+0000 7f61a2ffd700 1 -- 172.21.15.42:0/3198697527 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f61a400f550 con 0x7f61b014fe70 2019-12-05T05:03:35.439 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.428+0000 7f61b7af9700 1 -- 172.21.15.42:0/3198697527 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61b012f630 con 0x7f61b014fe70 2019-12-05T05:03:35.439 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.428+0000 7f61b5895700 1 --2- 172.21.15.42:0/3198697527 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f61b014fe70 0x7f61b0143890 secure :-1 s=THROTTLE_DONE pgs=69 cs=0 l=1 rx=0x7f61b0146030 tx=0x7f61a4008e50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:35.439 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.428+0000 7f61a2ffd700 1 -- 172.21.15.42:0/3198697527 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f61a4005540 con 0x7f61b014fe70 2019-12-05T05:03:35.439 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.428+0000 7f61b5895700 1 --2- 172.21.15.42:0/3198697527 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f61b014fe70 0x7f61b0143890 secure :-1 s=THROTTLE_DONE pgs=69 cs=0 l=1 rx=0x7f61b0146030 tx=0x7f61a4008e50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:35.439 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.428+0000 7f61a2ffd700 1 --2- 172.21.15.42:0/3198697527 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f619803b260 0x7f619803d770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:35.439 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.428+0000 7f61a2ffd700 1 -- 172.21.15.42:0/3198697527 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5915+0+0 (secure 0 0 0) 0x7f61a404baa0 con 0x7f61b014fe70 2019-12-05T05:03:35.440 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.428+0000 7f61b6096700 1 --2- 172.21.15.42:0/3198697527 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f619803b260 0x7f619803d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:35.440 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.428+0000 7f61b6096700 1 --2- 172.21.15.42:0/3198697527 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f619803b260 0x7f619803d770 crc :-1 s=READY pgs=50 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:35.441 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.436+0000 7f61b7af9700 1 -- 172.21.15.42:0/3198697527 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61b005d110 con 0x7f61b014fe70 2019-12-05T05:03:35.446 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.440+0000 7f61b5895700 1 --2- 172.21.15.42:0/3198697527 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f61b014fe70 0x7f61b0143890 secure :-1 s=THROTTLE_DONE pgs=69 cs=0 l=1 rx=0x7f61b0146030 tx=0x7f61a4008e50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:35.447 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.440+0000 7f61a2ffd700 1 -- 172.21.15.42:0/3198697527 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f61a4032b20 con 0x7f61b014fe70 2019-12-05T05:03:35.601 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.596+0000 7f61b7af9700 1 -- 172.21.15.42:0/3198697527 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs new", "data": "cephfs_data", "fs_name": "cephfs", "metadata": "cephfs_metadata"} v 0) v1 -- 0x7f61b005b8c0 con 0x7f61b014fe70 2019-12-05T05:03:35.603 INFO:tasks.ceph.mon.b.smithi040.stderr:2019-12-05T05:03:35.600+0000 7faeb9919700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2019-12-05T05:03:35.612 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.608+0000 7f61b5895700 1 --2- 172.21.15.42:0/3198697527 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f61b014fe70 0x7f61b0143890 secure :-1 s=THROTTLE_DONE pgs=69 cs=0 l=1 rx=0x7f61b0146030 tx=0x7f61a4008e50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:35.612 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.608+0000 7f61a2ffd700 1 -- 172.21.15.42:0/3198697527 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "data": "cephfs_data", "fs_name": "cephfs", "metadata": "cephfs_metadata"}]=0 new fs with metadata pool 2 and data pool 3 v2) v1 ==== 172+0+0 (secure 0 0 0) 0x7f61a4030750 con 0x7f61b014fe70 2019-12-05T05:03:35.617 INFO:teuthology.orchestra.run.smithi042.stderr:new fs with metadata pool 2 and data pool 3 2019-12-05T05:03:35.620 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.612+0000 7f61b7af9700 1 -- 172.21.15.42:0/3198697527 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f619803b260 msgr2=0x7f619803d770 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:35.620 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.612+0000 7f61b7af9700 1 --2- 172.21.15.42:0/3198697527 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f619803b260 0x7f619803d770 crc :-1 s=READY pgs=50 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:35.620 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.612+0000 7f61b7af9700 1 -- 172.21.15.42:0/3198697527 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f61b014fe70 msgr2=0x7f61b0143890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:35.620 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.612+0000 7f61b7af9700 1 --2- 172.21.15.42:0/3198697527 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f61b014fe70 0x7f61b0143890 secure :-1 s=READY pgs=69 cs=0 l=1 rx=0x7f61b0146030 tx=0x7f61a4008e50).stop 2019-12-05T05:03:35.620 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.612+0000 7f61b7af9700 1 -- 172.21.15.42:0/3198697527 shutdown_connections 2019-12-05T05:03:35.620 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.612+0000 7f61b7af9700 1 --2- 172.21.15.42:0/3198697527 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f619803b260 0x7f619803d770 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:35.621 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.612+0000 7f61b7af9700 1 --2- 172.21.15.42:0/3198697527 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f61b014fe70 0x7f61b0143890 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:35.621 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.616+0000 7f61b7af9700 1 -- 172.21.15.42:0/3198697527 shutdown_connections 2019-12-05T05:03:35.621 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.616+0000 7f61b7af9700 1 -- 172.21.15.42:0/3198697527 wait complete. 2019-12-05T05:03:35.621 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.616+0000 7f61b7af9700 1 -- 172.21.15.42:0/3198697527 >> 172.21.15.42:0/3198697527 conn(0x7f61b00aa450 msgr2=0x7f61b00a98e0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:35.632 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2019-12-05T05:03:35.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.708+0000 7f99ca9df700 1 Processor -- start 2019-12-05T05:03:35.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.708+0000 7f99ca9df700 1 -- start start 2019-12-05T05:03:35.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.712+0000 7f99ca9df700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f99c40a7740 0x7f99c408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:35.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.712+0000 7f99ca9df700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f99c407e720 0x7f99c407eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:35.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.712+0000 7f99ca9df700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f99c409f2e0 0x7f99c409f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:35.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.712+0000 7f99ca9df700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99c40a0b00 con 0x7f99c407e720 2019-12-05T05:03:35.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.712+0000 7f99ca9df700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99c408e5b0 con 0x7f99c40a7740 2019-12-05T05:03:35.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.712+0000 7f99ca9df700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99c408e850 con 0x7f99c409f2e0 2019-12-05T05:03:35.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.712+0000 7f99c8f7c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f99c40a7740 0x7f99c408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:35.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.712+0000 7f99c8f7c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f99c40a7740 0x7f99c408df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:53096/0 (socket says 172.21.15.42:53096) 2019-12-05T05:03:35.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.712+0000 7f99c8f7c700 1 -- 172.21.15.42:0/632125250 learned_addr learned my addr 172.21.15.42:0/632125250 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:03:35.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.712+0000 7f99c3fff700 1 --2- 172.21.15.42:0/632125250 >> v2:172.21.15.40:3300/0 conn(0x7f99c407e720 0x7f99c407eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:35.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.712+0000 7f99c977d700 1 --2- 172.21.15.42:0/632125250 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f99c409f2e0 0x7f99c409f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:35.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.712+0000 7f99c8f7c700 1 -- 172.21.15.42:0/632125250 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f99c409f2e0 msgr2=0x7f99c409f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:03:35.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.712+0000 7f99c8f7c700 1 --2- 172.21.15.42:0/632125250 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f99c409f2e0 0x7f99c409f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:03:35.724 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.712+0000 7f99c8f7c700 1 -- 172.21.15.42:0/632125250 >> v2:172.21.15.40:3300/0 conn(0x7f99c407e720 msgr2=0x7f99c407eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:03:35.724 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.712+0000 7f99c8f7c700 1 --2- 172.21.15.42:0/632125250 >> v2:172.21.15.40:3300/0 conn(0x7f99c407e720 0x7f99c407eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:03:35.724 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.712+0000 7f99c8f7c700 1 -- 172.21.15.42:0/632125250 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99c407f0a0 con 0x7f99c40a7740 2019-12-05T05:03:35.724 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.712+0000 7f99c8f7c700 1 --2- 172.21.15.42:0/632125250 >> v2:172.21.15.42:3300/0 conn(0x7f99c40a7740 0x7f99c408df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f99c40a0f50 tx=0x7f99b400fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:35.725 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.712+0000 7f99c8f7c700 1 --2- 172.21.15.42:0/632125250 >> v2:172.21.15.42:3300/0 conn(0x7f99c40a7740 0x7f99c408df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f99c40a0f50 tx=0x7f99b400fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:35.725 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.712+0000 7f99c8f7c700 1 --2- 172.21.15.42:0/632125250 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f99c40a7740 0x7f99c408df80 secure :-1 s=READY pgs=70 cs=0 l=1 rx=0x7f99c40a0f50 tx=0x7f99b400fdb0).ready entity=mon.1 client_cookie=35c02691dd4896a3 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:35.725 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.712+0000 7f99c8f7c700 1 --2- 172.21.15.42:0/632125250 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f99c40a7740 0x7f99c408df80 secure :-1 s=THROTTLE_DONE pgs=70 cs=0 l=1 rx=0x7f99c40a0f50 tx=0x7f99b400fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:35.725 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.712+0000 7f99c8f7c700 1 --2- 172.21.15.42:0/632125250 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f99c40a7740 0x7f99c408df80 secure :-1 s=THROTTLE_DONE pgs=70 cs=0 l=1 rx=0x7f99c40a0f50 tx=0x7f99b400fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:35.726 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.712+0000 7f99c37fe700 1 -- 172.21.15.42:0/632125250 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f99b4008c90 con 0x7f99c40a7740 2019-12-05T05:03:35.726 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.712+0000 7f99c8f7c700 1 --2- 172.21.15.42:0/632125250 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f99c40a7740 0x7f99c408df80 secure :-1 s=THROTTLE_DONE pgs=70 cs=0 l=1 rx=0x7f99c40a0f50 tx=0x7f99b400fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:35.727 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.712+0000 7f99c37fe700 1 -- 172.21.15.42:0/632125250 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99b40088b0 con 0x7f99c40a7740 2019-12-05T05:03:35.727 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.712+0000 7f99c37fe700 1 -- 172.21.15.42:0/632125250 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f99b4011b50 con 0x7f99c40a7740 2019-12-05T05:03:35.728 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.712+0000 7f99ca9df700 1 -- 172.21.15.42:0/632125250 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f99c40a7740 msgr2=0x7f99c408df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:35.728 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.712+0000 7f99ca9df700 1 --2- 172.21.15.42:0/632125250 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f99c40a7740 0x7f99c408df80 secure :-1 s=READY pgs=70 cs=0 l=1 rx=0x7f99c40a0f50 tx=0x7f99b400fdb0).stop 2019-12-05T05:03:35.728 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.712+0000 7f99ca9df700 1 -- 172.21.15.42:0/632125250 shutdown_connections 2019-12-05T05:03:35.728 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.712+0000 7f99ca9df700 1 --2- 172.21.15.42:0/632125250 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f99c409f2e0 0x7f99c409f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:03:35.729 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.712+0000 7f99ca9df700 1 --2- 172.21.15.42:0/632125250 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f99c40a7740 0x7f99c408df80 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:35.729 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.712+0000 7f99ca9df700 1 --2- 172.21.15.42:0/632125250 >> v2:172.21.15.40:3300/0 conn(0x7f99c407e720 0x7f99c407eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:03:35.729 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.712+0000 7f99ca9df700 1 -- 172.21.15.42:0/632125250 shutdown_connections 2019-12-05T05:03:35.729 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.712+0000 7f99ca9df700 1 -- 172.21.15.42:0/632125250 wait complete. 2019-12-05T05:03:35.730 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.712+0000 7f99ca9df700 1 -- 172.21.15.42:0/632125250 >> 172.21.15.42:0/632125250 conn(0x7f99c40aabe0 msgr2=0x7f99c40a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:35.736 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.716+0000 7f99ca9df700 1 Processor -- start 2019-12-05T05:03:35.736 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.716+0000 7f99ca9df700 1 -- start start 2019-12-05T05:03:35.736 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.716+0000 7f99ca9df700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f99c41191f0 0x7f99c411b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:35.736 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.716+0000 7f99ca9df700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f99c411bb00 0x7f99c4144b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:35.737 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.716+0000 7f99ca9df700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99c4142b90 con 0x7f99c4145060 2019-12-05T05:03:35.737 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.716+0000 7f99ca9df700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99c4142fb0 con 0x7f99c41191f0 2019-12-05T05:03:35.737 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.716+0000 7f99ca9df700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99c4143250 con 0x7f99c411bb00 2019-12-05T05:03:35.737 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.716+0000 7f99c8f7c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f99c41191f0 0x7f99c411b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:35.737 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.716+0000 7f99c8f7c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f99c41191f0 0x7f99c411b5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:53104/0 (socket says 172.21.15.42:53104) 2019-12-05T05:03:35.738 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.716+0000 7f99c8f7c700 1 -- 172.21.15.42:0/1417841711 learned_addr learned my addr 172.21.15.42:0/1417841711 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:03:35.738 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.716+0000 7f99c977d700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f99c4145060 0x7f99c411a2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:53144/0 (socket says 172.21.15.42:53144) 2019-12-05T05:03:35.738 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.716+0000 7f99c3fff700 1 --2- 172.21.15.42:0/1417841711 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f99c411bb00 0x7f99c4144b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:35.738 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.716+0000 7f99c8f7c700 1 -- 172.21.15.42:0/1417841711 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f99c411bb00 msgr2=0x7f99c4144b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:35.739 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.716+0000 7f99c8f7c700 1 --2- 172.21.15.42:0/1417841711 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f99c411bb00 0x7f99c4144b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:35.739 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.716+0000 7f99c8f7c700 1 -- 172.21.15.42:0/1417841711 >> v1:172.21.15.42:6789/0 conn(0x7f99c4145060 legacy=0x7f99c411a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:35.739 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.716+0000 7f99c8f7c700 1 -- 172.21.15.42:0/1417841711 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99c41434f0 con 0x7f99c41191f0 2019-12-05T05:03:35.739 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.716+0000 7f99c17fa700 1 -- 172.21.15.42:0/1417841711 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3258303279 0 0) 0x7f99c4142b90 con 0x7f99c4145060 2019-12-05T05:03:35.739 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.716+0000 7f99c8f7c700 1 --2- 172.21.15.42:0/1417841711 >> v2:172.21.15.42:3300/0 conn(0x7f99c41191f0 0x7f99c411b5b0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f99c4119b80 tx=0x7f99b4008bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:35.739 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.716+0000 7f99c8f7c700 1 --2- 172.21.15.42:0/1417841711 >> v2:172.21.15.42:3300/0 conn(0x7f99c41191f0 0x7f99c411b5b0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f99c4119b80 tx=0x7f99b4008bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:35.740 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.716+0000 7f99c8f7c700 1 --2- 172.21.15.42:0/1417841711 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f99c41191f0 0x7f99c411b5b0 secure :-1 s=READY pgs=72 cs=0 l=1 rx=0x7f99c4119b80 tx=0x7f99b4008bc0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:35.740 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.716+0000 7f99c8f7c700 1 --2- 172.21.15.42:0/1417841711 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f99c41191f0 0x7f99c411b5b0 secure :-1 s=THROTTLE_DONE pgs=72 cs=0 l=1 rx=0x7f99c4119b80 tx=0x7f99b4008bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:35.740 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.716+0000 7f99c17fa700 1 -- 172.21.15.42:0/1417841711 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f99b400f490 con 0x7f99c41191f0 2019-12-05T05:03:35.740 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.716+0000 7f99c8f7c700 1 --2- 172.21.15.42:0/1417841711 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f99c41191f0 0x7f99c411b5b0 secure :-1 s=THROTTLE_DONE pgs=72 cs=0 l=1 rx=0x7f99c4119b80 tx=0x7f99b4008bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:35.741 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.716+0000 7f99c8f7c700 1 --2- 172.21.15.42:0/1417841711 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f99c41191f0 0x7f99c411b5b0 secure :-1 s=THROTTLE_DONE pgs=72 cs=0 l=1 rx=0x7f99c4119b80 tx=0x7f99b4008bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:35.741 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.716+0000 7f99c17fa700 1 -- 172.21.15.42:0/1417841711 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99b4011770 con 0x7f99c41191f0 2019-12-05T05:03:35.741 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.716+0000 7f99ca9df700 1 -- 172.21.15.42:0/1417841711 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99c411aaf0 con 0x7f99c41191f0 2019-12-05T05:03:35.741 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.716+0000 7f99c17fa700 1 -- 172.21.15.42:0/1417841711 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f99b4005730 con 0x7f99c41191f0 2019-12-05T05:03:35.742 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.716+0000 7f99ca9df700 1 -- 172.21.15.42:0/1417841711 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99c411af00 con 0x7f99c41191f0 2019-12-05T05:03:35.742 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.720+0000 7f99c8f7c700 1 --2- 172.21.15.42:0/1417841711 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f99c41191f0 0x7f99c411b5b0 secure :-1 s=THROTTLE_DONE pgs=72 cs=0 l=1 rx=0x7f99c4119b80 tx=0x7f99b4008bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:35.742 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.720+0000 7f99c17fa700 1 -- 172.21.15.42:0/1417841711 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f99b40059d0 con 0x7f99c41191f0 2019-12-05T05:03:35.742 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.720+0000 7f99c8f7c700 1 --2- 172.21.15.42:0/1417841711 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f99c41191f0 0x7f99c411b5b0 secure :-1 s=THROTTLE_DONE pgs=72 cs=0 l=1 rx=0x7f99c4119b80 tx=0x7f99b4008bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:35.742 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.720+0000 7f99c17fa700 1 --2- 172.21.15.42:0/1417841711 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f99a403acc0 0x7f99a403d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:35.742 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.720+0000 7f99c17fa700 1 -- 172.21.15.42:0/1417841711 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f99b408f590 con 0x7f99c41191f0 2019-12-05T05:03:35.743 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.720+0000 7f99c3fff700 1 --2- 172.21.15.42:0/1417841711 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f99a403acc0 0x7f99a403d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:35.743 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.720+0000 7f99c3fff700 1 --2- 172.21.15.42:0/1417841711 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f99a403acc0 0x7f99a403d1d0 crc :-1 s=READY pgs=51 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:35.743 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.724+0000 7f99ca9df700 1 -- 172.21.15.42:0/1417841711 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99c405d110 con 0x7f99c41191f0 2019-12-05T05:03:35.744 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.732+0000 7f99c8f7c700 1 --2- 172.21.15.42:0/1417841711 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f99c41191f0 0x7f99c411b5b0 secure :-1 s=THROTTLE_DONE pgs=72 cs=0 l=1 rx=0x7f99c4119b80 tx=0x7f99b4008bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:35.744 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.732+0000 7f99c17fa700 1 -- 172.21.15.42:0/1417841711 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f99b4076850 con 0x7f99c41191f0 2019-12-05T05:03:35.909 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.900+0000 7f99ca9df700 1 -- 172.21.15.42:0/1417841711 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f99c40aadf0 con 0x7f99c41191f0 2019-12-05T05:03:35.910 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.904+0000 7f99c8f7c700 1 --2- 172.21.15.42:0/1417841711 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f99c41191f0 0x7f99c411b5b0 secure :-1 s=THROTTLE_DONE pgs=72 cs=0 l=1 rx=0x7f99c4119b80 tx=0x7f99b4008bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:35.910 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.904+0000 7f99c17fa700 1 -- 172.21.15.42:0/1417841711 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+14898 (secure 0 0 0) 0x7f99b400ebd0 con 0x7f99c41191f0 2019-12-05T05:03:35.911 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:03:35.911 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":23,"fsid":"5d836cfc-a83b-4433-9b3e-b9b2f31c9050","created":"2019-12-05T05:03:13.215602+0000","modified":"2019-12-05T05:03:35.604379+0000","last_up_change":"2019-12-05T05:03:22.664783+0000","last_in_change":"2019-12-05T05:03:22.664783+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2019-12-05T05:03:27.394605+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2019-12-05T05:03:30.419845+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2019-12-05T05:03:33.249075+0000","flags":32773,"flags_names":"hashpspool,ec_overwrites,creating","type":3,"size":4,"min_size":3,"crush_rule":1,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e3fc49bb-dcbc-43ff-ac7c-9eb0ea0bd257","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6810","nonce":12118},{"type":"v1","addr":"172.21.15.42:6811","nonce":12118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6812","nonce":12118},{"type":"v1","addr":"172.21.15.42:6813","nonce":12118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6816","nonce":12118},{"type":"v1","addr":"172.21.15.42:6817","nonce":12118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6814","nonce":12118},{"type":"v1","addr":"172.21.15.42:6815","nonce":12118}]},"public_addr":"172.21.15.42:6811/12118","cluster_addr":"172.21.15.42:6813/12118","heartbeat_back_addr":"172.21.15.42:6817/12118","heartbeat_front_addr":"172.21.15.42:6815/12118","state":["exists","up"]},{"osd":1,"uuid":"902543fb-f5a2-4321-9c9c-26f9b9ca8629","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6826","nonce":12119},{"type":"v1","addr":"172.21.15.42:6827","nonce":12119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6828","nonce":12119},{"type":"v1","addr":"172.21.15.42:6829","nonce":12119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6832","nonce":12119},{"type":"v1","addr":"172.21.15.42:6833","nonce":12119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6830","nonce":12119},{"type":"v1","addr":"172.21.15.42:6831","nonce":12119}]},"public_addr":"172.21.15.42:6827/12119","cluster_addr":"172.21.15.42:6829/12119","heartbeat_back_addr":"172.21.15.42:6833/12119","heartbeat_front_addr":"172.21.15.42:6831/12119","state":["exists","up"]},{"osd":2,"uuid":"28739ff7-bed7-4b75-8ea5-b57d088f1250","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6802","nonce":12117},{"type":"v1","addr":"172.21.15.42:6803","nonce":12117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6804","nonce":12117},{"type":"v1","addr":"172.21.15.42:6805","nonce":12117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6808","nonce":12117},{"type":"v1","addr":"172.21.15.42:6809","nonce":12117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6806","nonce":12117},{"type":"v1","addr":"172.21.15.42:6807","nonce":12117}]},"public_addr":"172.21.15.42:6803/12117","cluster_addr":"172.21.15.42:6805/12117","heartbeat_back_addr":"172.21.15.42:6809/12117","heartbeat_front_addr":"172.21.15.42:6807/12117","state":["exists","up"]},{"osd":3,"uuid":"1960f66b-2d40-4bb9-a289-ce8cd0c1c142","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6818","nonce":12116},{"type":"v1","addr":"172.21.15.42:6819","nonce":12116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6820","nonce":12116},{"type":"v1","addr":"172.21.15.42:6821","nonce":12116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6824","nonce":12116},{"type":"v1","addr":"172.21.15.42:6825","nonce":12116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6822","nonce":12116},{"type":"v1","addr":"172.21.15.42:6823","nonce":12116}]},"public_addr":"172.21.15.42:6819/12116","cluster_addr":"172.21.15.42:6821/12116","heartbeat_back_addr":"172.21.15.42:6825/12116","heartbeat_front_addr":"172.21.15.42:6823/12116","state":["exists","up"]},{"osd":4,"uuid":"45bbf1bd-7c8a-4a0d-b76b-afa6148a855c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":12488},{"type":"v1","addr":"172.21.15.40:6809","nonce":12488}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":12488},{"type":"v1","addr":"172.21.15.40:6812","nonce":12488}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":12488},{"type":"v1","addr":"172.21.15.40:6825","nonce":12488}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":12488},{"type":"v1","addr":"172.21.15.40:6819","nonce":12488}]},"public_addr":"172.21.15.40:6809/12488","cluster_addr":"172.21.15.40:6812/12488","heartbeat_back_addr":"172.21.15.40:6825/12488","heartbeat_front_addr":"172.21.15.40:6819/12488","state":["exists","up"]},{"osd":5,"uuid":"8a5511cd-925f-4869-a49e-924383e4ce53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":12489},{"type":"v1","addr":"172.21.15.40:6801","nonce":12489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":12489},{"type":"v1","addr":"172.21.15.40:6803","nonce":12489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":12489},{"type":"v1","addr":"172.21.15.40:6807","nonce":12489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":12489},{"type":"v1","addr":"172.21.15.40:6805","nonce":12489}]},"public_addr":"172.21.15.40:6801/12489","cluster_addr":"172.21.15.40:6803/12489","heartbeat_back_addr":"172.21.15.40:6807/12489","heartbeat_front_addr":"172.21.15.40:6805/12489","state":["exists","up"]},{"osd":6,"uuid":"d692c179-379e-446c-8484-a219a9241e04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6811","nonce":12491},{"type":"v1","addr":"172.21.15.40:6814","nonce":12491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6817","nonce":12491},{"type":"v1","addr":"172.21.15.40:6818","nonce":12491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":12491},{"type":"v1","addr":"172.21.15.40:6828","nonce":12491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":12491},{"type":"v1","addr":"172.21.15.40:6824","nonce":12491}]},"public_addr":"172.21.15.40:6814/12491","cluster_addr":"172.21.15.40:6818/12491","heartbeat_back_addr":"172.21.15.40:6828/12491","heartbeat_front_addr":"172.21.15.40:6824/12491","state":["exists","up"]},{"osd":7,"uuid":"79c3c243-190e-4e35-b769-319a2077e7cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6813","nonce":12490},{"type":"v1","addr":"172.21.15.40:6816","nonce":12490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":12490},{"type":"v1","addr":"172.21.15.40:6823","nonce":12490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":12490},{"type":"v1","addr":"172.21.15.40:6831","nonce":12490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6827","nonce":12490},{"type":"v1","addr":"172.21.15.40:6829","nonce":12490}]},"public_addr":"172.21.15.40:6816/12490","cluster_addr":"172.21.15.40:6823/12490","heartbeat_back_addr":"172.21.15.40:6831/12490","heartbeat_front_addr":"172.21.15.40:6829/12490","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T05:03:21.349150+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T05:03:21.644281+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T05:03:21.394040+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T05:03:21.587096+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T05:03:21.656497+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T05:03:21.510547+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T05:03:21.659358+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T05:03:21.628063+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2019-12-05T05:03:35.913 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.908+0000 7f99ca9df700 1 -- 172.21.15.42:0/1417841711 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f99a403acc0 msgr2=0x7f99a403d1d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:35.913 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.908+0000 7f99ca9df700 1 --2- 172.21.15.42:0/1417841711 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f99a403acc0 0x7f99a403d1d0 crc :-1 s=READY pgs=51 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:35.914 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.908+0000 7f99ca9df700 1 -- 172.21.15.42:0/1417841711 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f99c41191f0 msgr2=0x7f99c411b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:35.914 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.908+0000 7f99ca9df700 1 --2- 172.21.15.42:0/1417841711 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f99c41191f0 0x7f99c411b5b0 secure :-1 s=READY pgs=72 cs=0 l=1 rx=0x7f99c4119b80 tx=0x7f99b4008bc0).stop 2019-12-05T05:03:35.914 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.908+0000 7f99ca9df700 1 -- 172.21.15.42:0/1417841711 shutdown_connections 2019-12-05T05:03:35.914 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.908+0000 7f99ca9df700 1 --2- 172.21.15.42:0/1417841711 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f99a403acc0 0x7f99a403d1d0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:35.914 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.908+0000 7f99ca9df700 1 --2- 172.21.15.42:0/1417841711 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f99c41191f0 0x7f99c411b5b0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:35.915 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.908+0000 7f99ca9df700 1 --2- 172.21.15.42:0/1417841711 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f99c411bb00 0x7f99c4144b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:35.915 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.908+0000 7f99ca9df700 1 -- 172.21.15.42:0/1417841711 shutdown_connections 2019-12-05T05:03:35.915 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.908+0000 7f99ca9df700 1 -- 172.21.15.42:0/1417841711 wait complete. 2019-12-05T05:03:35.915 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:35.908+0000 7f99ca9df700 1 -- 172.21.15.42:0/1417841711 >> 172.21.15.42:0/1417841711 conn(0x7f99c40a7740 msgr2=0x7f99c4059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:35.925 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2019-12-05T05:03:36.029 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.016+0000 7fa15c934700 1 Processor -- start 2019-12-05T05:03:36.029 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.016+0000 7fa15c934700 1 -- start start 2019-12-05T05:03:36.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.020+0000 7fa15c934700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa15809fae0 0x7fa15814fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:36.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.020+0000 7fa15c934700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1580a0710 con 0x7fa1580a7740 2019-12-05T05:03:36.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.020+0000 7fa15c934700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa15807ef20 con 0x7fa15808df80 2019-12-05T05:03:36.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.020+0000 7fa15c934700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1580a0b00 con 0x7fa15809fae0 2019-12-05T05:03:36.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.020+0000 7fa15659c700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fa15808df80 0x7fa15809f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:53150/0 (socket says 172.21.15.42:53150) 2019-12-05T05:03:36.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.020+0000 7fa15659c700 1 -- 172.21.15.42:0/191872813 learned_addr learned my addr 172.21.15.42:0/191872813 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:03:36.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.020+0000 7fa15759e700 1 --2- 172.21.15.42:0/191872813 >> v2:172.21.15.40:3300/0 conn(0x7fa15809fae0 0x7fa15814fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:36.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.020+0000 7fa155d9b700 1 -- 172.21.15.42:0/191872813 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1966519439 0 0) 0x7fa15807ef20 con 0x7fa15808df80 2019-12-05T05:03:36.032 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.020+0000 7fa155d9b700 1 -- 172.21.15.42:0/191872813 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1440030b0 con 0x7fa15808df80 2019-12-05T05:03:36.032 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.020+0000 7fa15759e700 1 -- 172.21.15.42:0/191872813 >> v1:172.21.15.42:6789/0 conn(0x7fa15808df80 legacy=0x7fa15809f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:36.032 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.020+0000 7fa155d9b700 1 -- 172.21.15.42:0/191872813 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2864692445 0 0) 0x7fa1580a0710 con 0x7fa1580a7740 2019-12-05T05:03:36.032 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.020+0000 7fa15759e700 1 -- 172.21.15.42:0/191872813 >> v1:172.21.15.40:6789/0 conn(0x7fa1580a7740 legacy=0x7fa15807e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:36.032 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.020+0000 7fa15759e700 1 -- 172.21.15.42:0/191872813 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1580a9ba0 con 0x7fa15809fae0 2019-12-05T05:03:36.033 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.020+0000 7fa15759e700 1 --2- 172.21.15.42:0/191872813 >> v2:172.21.15.40:3300/0 conn(0x7fa15809fae0 0x7fa15814fff0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fa150008b10 tx=0x7fa15000ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:36.033 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.020+0000 7fa15759e700 1 --2- 172.21.15.42:0/191872813 >> v2:172.21.15.40:3300/0 conn(0x7fa15809fae0 0x7fa15814fff0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fa150008b10 tx=0x7fa15000ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:36.033 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.020+0000 7fa15759e700 1 --2- 172.21.15.42:0/191872813 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa15809fae0 0x7fa15814fff0 secure :-1 s=READY pgs=71 cs=0 l=1 rx=0x7fa150008b10 tx=0x7fa15000ff70).ready entity=mon.0 client_cookie=fd0e15fbdb56248d server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:36.033 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.020+0000 7fa15759e700 1 --2- 172.21.15.42:0/191872813 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa15809fae0 0x7fa15814fff0 secure :-1 s=THROTTLE_DONE pgs=71 cs=0 l=1 rx=0x7fa150008b10 tx=0x7fa15000ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:36.033 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.020+0000 7fa155d9b700 1 -- 172.21.15.42:0/191872813 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa150010970 con 0x7fa15809fae0 2019-12-05T05:03:36.033 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.020+0000 7fa15759e700 1 --2- 172.21.15.42:0/191872813 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa15809fae0 0x7fa15814fff0 secure :-1 s=THROTTLE_DONE pgs=71 cs=0 l=1 rx=0x7fa150008b10 tx=0x7fa15000ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:36.033 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.020+0000 7fa15759e700 1 --2- 172.21.15.42:0/191872813 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa15809fae0 0x7fa15814fff0 secure :-1 s=THROTTLE_DONE pgs=71 cs=0 l=1 rx=0x7fa150008b10 tx=0x7fa15000ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:36.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.020+0000 7fa155d9b700 1 -- 172.21.15.42:0/191872813 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa150008b50 con 0x7fa15809fae0 2019-12-05T05:03:36.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.020+0000 7fa155d9b700 1 -- 172.21.15.42:0/191872813 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa1500110c0 con 0x7fa15809fae0 2019-12-05T05:03:36.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.020+0000 7fa15c934700 1 -- 172.21.15.42:0/191872813 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa15809fae0 msgr2=0x7fa15814fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:36.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.020+0000 7fa15c934700 1 --2- 172.21.15.42:0/191872813 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa15809fae0 0x7fa15814fff0 secure :-1 s=READY pgs=71 cs=0 l=1 rx=0x7fa150008b10 tx=0x7fa15000ff70).stop 2019-12-05T05:03:36.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.024+0000 7fa15c934700 1 -- 172.21.15.42:0/191872813 shutdown_connections 2019-12-05T05:03:36.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.024+0000 7fa15c934700 1 --2- 172.21.15.42:0/191872813 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa15809fae0 0x7fa15814fff0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:36.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.024+0000 7fa15c934700 1 -- 172.21.15.42:0/191872813 shutdown_connections 2019-12-05T05:03:36.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.024+0000 7fa15c934700 1 -- 172.21.15.42:0/191872813 wait complete. 2019-12-05T05:03:36.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.024+0000 7fa15c934700 1 -- 172.21.15.42:0/191872813 >> 172.21.15.42:0/191872813 conn(0x7fa1580aabe0 msgr2=0x7fa1580a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:36.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.024+0000 7fa15c934700 1 Processor -- start 2019-12-05T05:03:36.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.024+0000 7fa15c934700 1 -- start start 2019-12-05T05:03:36.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.024+0000 7fa15c934700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fa158117940 0x7fa1581431b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:36.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.024+0000 7fa15c934700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa158117d70 con 0x7fa15812cf70 2019-12-05T05:03:36.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.024+0000 7fa15c934700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa158143700 con 0x7fa15812cb40 2019-12-05T05:03:36.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.024+0000 7fa15c934700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1581440a0 con 0x7fa158117940 2019-12-05T05:03:36.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.028+0000 7fa15659c700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fa15812cb40 0x7fa15807d850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:53156/0 (socket says 172.21.15.42:53156) 2019-12-05T05:03:36.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.028+0000 7fa15659c700 1 -- 172.21.15.42:0/2381254106 learned_addr learned my addr 172.21.15.42:0/2381254106 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:03:36.039 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.028+0000 7fa156d9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fa158117940 0x7fa1581431b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:36.039 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.028+0000 7fa14f7fe700 1 -- 172.21.15.42:0/2381254106 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1524482822 0 0) 0x7fa158143700 con 0x7fa15812cb40 2019-12-05T05:03:36.039 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.028+0000 7fa14f7fe700 1 -- 172.21.15.42:0/2381254106 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa13c0030b0 con 0x7fa15812cb40 2019-12-05T05:03:36.039 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.028+0000 7fa14f7fe700 1 -- 172.21.15.42:0/2381254106 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4082891137 0 0) 0x7fa158117d70 con 0x7fa15812cf70 2019-12-05T05:03:36.039 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.028+0000 7fa156d9d700 1 -- 172.21.15.42:0/2381254106 >> v1:172.21.15.42:6789/0 conn(0x7fa15812cb40 legacy=0x7fa15807d850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:36.040 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.028+0000 7fa156d9d700 1 -- 172.21.15.42:0/2381254106 >> v1:172.21.15.40:6789/0 conn(0x7fa15812cf70 legacy=0x7fa158137550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:36.040 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.028+0000 7fa156d9d700 1 -- 172.21.15.42:0/2381254106 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa158144880 con 0x7fa158117940 2019-12-05T05:03:36.040 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.028+0000 7fa156d9d700 1 --2- 172.21.15.42:0/2381254106 >> v2:172.21.15.42:3300/0 conn(0x7fa158117940 0x7fa1581431b0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fa140000e90 tx=0x7fa14000c2f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:36.040 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.028+0000 7fa156d9d700 1 --2- 172.21.15.42:0/2381254106 >> v2:172.21.15.42:3300/0 conn(0x7fa158117940 0x7fa1581431b0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fa140000e90 tx=0x7fa14000c2f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:36.040 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.028+0000 7fa156d9d700 1 --2- 172.21.15.42:0/2381254106 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa158117940 0x7fa1581431b0 secure :-1 s=READY pgs=75 cs=0 l=1 rx=0x7fa140000e90 tx=0x7fa14000c2f0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:36.041 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.028+0000 7fa156d9d700 1 --2- 172.21.15.42:0/2381254106 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa158117940 0x7fa1581431b0 secure :-1 s=THROTTLE_DONE pgs=75 cs=0 l=1 rx=0x7fa140000e90 tx=0x7fa14000c2f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:36.041 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.028+0000 7fa14f7fe700 1 -- 172.21.15.42:0/2381254106 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa14000cb90 con 0x7fa158117940 2019-12-05T05:03:36.041 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.028+0000 7fa156d9d700 1 --2- 172.21.15.42:0/2381254106 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa158117940 0x7fa1581431b0 secure :-1 s=THROTTLE_DONE pgs=75 cs=0 l=1 rx=0x7fa140000e90 tx=0x7fa14000c2f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:36.041 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.028+0000 7fa156d9d700 1 --2- 172.21.15.42:0/2381254106 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa158117940 0x7fa1581431b0 secure :-1 s=THROTTLE_DONE pgs=75 cs=0 l=1 rx=0x7fa140000e90 tx=0x7fa14000c2f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:36.041 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.028+0000 7fa14f7fe700 1 -- 172.21.15.42:0/2381254106 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa14000c7b0 con 0x7fa158117940 2019-12-05T05:03:36.041 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.028+0000 7fa15c934700 1 -- 172.21.15.42:0/2381254106 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa15807e050 con 0x7fa158117940 2019-12-05T05:03:36.042 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.028+0000 7fa14f7fe700 1 -- 172.21.15.42:0/2381254106 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa140011070 con 0x7fa158117940 2019-12-05T05:03:36.042 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.028+0000 7fa15c934700 1 -- 172.21.15.42:0/2381254106 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa15808eb70 con 0x7fa158117940 2019-12-05T05:03:36.042 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.028+0000 7fa156d9d700 1 --2- 172.21.15.42:0/2381254106 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa158117940 0x7fa1581431b0 secure :-1 s=THROTTLE_DONE pgs=75 cs=0 l=1 rx=0x7fa140000e90 tx=0x7fa14000c2f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:36.044 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.028+0000 7fa14f7fe700 1 -- 172.21.15.42:0/2381254106 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fa140011340 con 0x7fa158117940 2019-12-05T05:03:36.044 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.028+0000 7fa156d9d700 1 --2- 172.21.15.42:0/2381254106 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa158117940 0x7fa1581431b0 secure :-1 s=THROTTLE_DONE pgs=75 cs=0 l=1 rx=0x7fa140000e90 tx=0x7fa14000c2f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:36.044 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.032+0000 7fa14f7fe700 1 --2- 172.21.15.42:0/2381254106 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa13c03b200 0x7fa13c03d710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:36.044 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.032+0000 7fa14f7fe700 1 -- 172.21.15.42:0/2381254106 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6031+0+0 (secure 0 0 0) 0x7fa14004b190 con 0x7fa158117940 2019-12-05T05:03:36.044 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.032+0000 7fa15659c700 1 --2- 172.21.15.42:0/2381254106 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa13c03b200 0x7fa13c03d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:36.044 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.032+0000 7fa15659c700 1 --2- 172.21.15.42:0/2381254106 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa13c03b200 0x7fa13c03d710 crc :-1 s=READY pgs=52 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:36.044 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.032+0000 7fa15c934700 1 -- 172.21.15.42:0/2381254106 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa15805d110 con 0x7fa158117940 2019-12-05T05:03:36.045 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.036+0000 7fa156d9d700 1 --2- 172.21.15.42:0/2381254106 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa158117940 0x7fa1581431b0 secure :-1 s=THROTTLE_DONE pgs=75 cs=0 l=1 rx=0x7fa140000e90 tx=0x7fa14000c2f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:36.045 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.036+0000 7fa14f7fe700 1 -- 172.21.15.42:0/2381254106 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fa140032210 con 0x7fa158117940 2019-12-05T05:03:36.210 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.204+0000 7fa15c934700 1 -- 172.21.15.42:0/2381254106 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa15805b8c0 con 0x7fa158117940 2019-12-05T05:03:36.212 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.208+0000 7fa156d9d700 1 --2- 172.21.15.42:0/2381254106 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa158117940 0x7fa1581431b0 secure :-1 s=THROTTLE_DONE pgs=75 cs=0 l=1 rx=0x7fa140000e90 tx=0x7fa14000c2f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:36.212 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.208+0000 7fa14f7fe700 1 -- 172.21.15.42:0/2381254106 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+14898 (secure 0 0 0) 0x7fa140032470 con 0x7fa158117940 2019-12-05T05:03:36.213 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:03:36.213 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":23,"fsid":"5d836cfc-a83b-4433-9b3e-b9b2f31c9050","created":"2019-12-05T05:03:13.215602+0000","modified":"2019-12-05T05:03:35.604379+0000","last_up_change":"2019-12-05T05:03:22.664783+0000","last_in_change":"2019-12-05T05:03:22.664783+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2019-12-05T05:03:27.394605+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2019-12-05T05:03:30.419845+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2019-12-05T05:03:33.249075+0000","flags":32773,"flags_names":"hashpspool,ec_overwrites,creating","type":3,"size":4,"min_size":3,"crush_rule":1,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e3fc49bb-dcbc-43ff-ac7c-9eb0ea0bd257","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6810","nonce":12118},{"type":"v1","addr":"172.21.15.42:6811","nonce":12118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6812","nonce":12118},{"type":"v1","addr":"172.21.15.42:6813","nonce":12118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6816","nonce":12118},{"type":"v1","addr":"172.21.15.42:6817","nonce":12118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6814","nonce":12118},{"type":"v1","addr":"172.21.15.42:6815","nonce":12118}]},"public_addr":"172.21.15.42:6811/12118","cluster_addr":"172.21.15.42:6813/12118","heartbeat_back_addr":"172.21.15.42:6817/12118","heartbeat_front_addr":"172.21.15.42:6815/12118","state":["exists","up"]},{"osd":1,"uuid":"902543fb-f5a2-4321-9c9c-26f9b9ca8629","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6826","nonce":12119},{"type":"v1","addr":"172.21.15.42:6827","nonce":12119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6828","nonce":12119},{"type":"v1","addr":"172.21.15.42:6829","nonce":12119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6832","nonce":12119},{"type":"v1","addr":"172.21.15.42:6833","nonce":12119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6830","nonce":12119},{"type":"v1","addr":"172.21.15.42:6831","nonce":12119}]},"public_addr":"172.21.15.42:6827/12119","cluster_addr":"172.21.15.42:6829/12119","heartbeat_back_addr":"172.21.15.42:6833/12119","heartbeat_front_addr":"172.21.15.42:6831/12119","state":["exists","up"]},{"osd":2,"uuid":"28739ff7-bed7-4b75-8ea5-b57d088f1250","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6802","nonce":12117},{"type":"v1","addr":"172.21.15.42:6803","nonce":12117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6804","nonce":12117},{"type":"v1","addr":"172.21.15.42:6805","nonce":12117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6808","nonce":12117},{"type":"v1","addr":"172.21.15.42:6809","nonce":12117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6806","nonce":12117},{"type":"v1","addr":"172.21.15.42:6807","nonce":12117}]},"public_addr":"172.21.15.42:6803/12117","cluster_addr":"172.21.15.42:6805/12117","heartbeat_back_addr":"172.21.15.42:6809/12117","heartbeat_front_addr":"172.21.15.42:6807/12117","state":["exists","up"]},{"osd":3,"uuid":"1960f66b-2d40-4bb9-a289-ce8cd0c1c142","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6818","nonce":12116},{"type":"v1","addr":"172.21.15.42:6819","nonce":12116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6820","nonce":12116},{"type":"v1","addr":"172.21.15.42:6821","nonce":12116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6824","nonce":12116},{"type":"v1","addr":"172.21.15.42:6825","nonce":12116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6822","nonce":12116},{"type":"v1","addr":"172.21.15.42:6823","nonce":12116}]},"public_addr":"172.21.15.42:6819/12116","cluster_addr":"172.21.15.42:6821/12116","heartbeat_back_addr":"172.21.15.42:6825/12116","heartbeat_front_addr":"172.21.15.42:6823/12116","state":["exists","up"]},{"osd":4,"uuid":"45bbf1bd-7c8a-4a0d-b76b-afa6148a855c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":12488},{"type":"v1","addr":"172.21.15.40:6809","nonce":12488}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":12488},{"type":"v1","addr":"172.21.15.40:6812","nonce":12488}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":12488},{"type":"v1","addr":"172.21.15.40:6825","nonce":12488}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":12488},{"type":"v1","addr":"172.21.15.40:6819","nonce":12488}]},"public_addr":"172.21.15.40:6809/12488","cluster_addr":"172.21.15.40:6812/12488","heartbeat_back_addr":"172.21.15.40:6825/12488","heartbeat_front_addr":"172.21.15.40:6819/12488","state":["exists","up"]},{"osd":5,"uuid":"8a5511cd-925f-4869-a49e-924383e4ce53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":12489},{"type":"v1","addr":"172.21.15.40:6801","nonce":12489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":12489},{"type":"v1","addr":"172.21.15.40:6803","nonce":12489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":12489},{"type":"v1","addr":"172.21.15.40:6807","nonce":12489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":12489},{"type":"v1","addr":"172.21.15.40:6805","nonce":12489}]},"public_addr":"172.21.15.40:6801/12489","cluster_addr":"172.21.15.40:6803/12489","heartbeat_back_addr":"172.21.15.40:6807/12489","heartbeat_front_addr":"172.21.15.40:6805/12489","state":["exists","up"]},{"osd":6,"uuid":"d692c179-379e-446c-8484-a219a9241e04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6811","nonce":12491},{"type":"v1","addr":"172.21.15.40:6814","nonce":12491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6817","nonce":12491},{"type":"v1","addr":"172.21.15.40:6818","nonce":12491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":12491},{"type":"v1","addr":"172.21.15.40:6828","nonce":12491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":12491},{"type":"v1","addr":"172.21.15.40:6824","nonce":12491}]},"public_addr":"172.21.15.40:6814/12491","cluster_addr":"172.21.15.40:6818/12491","heartbeat_back_addr":"172.21.15.40:6828/12491","heartbeat_front_addr":"172.21.15.40:6824/12491","state":["exists","up"]},{"osd":7,"uuid":"79c3c243-190e-4e35-b769-319a2077e7cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6813","nonce":12490},{"type":"v1","addr":"172.21.15.40:6816","nonce":12490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":12490},{"type":"v1","addr":"172.21.15.40:6823","nonce":12490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":12490},{"type":"v1","addr":"172.21.15.40:6831","nonce":12490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6827","nonce":12490},{"type":"v1","addr":"172.21.15.40:6829","nonce":12490}]},"public_addr":"172.21.15.40:6816/12490","cluster_addr":"172.21.15.40:6823/12490","heartbeat_back_addr":"172.21.15.40:6831/12490","heartbeat_front_addr":"172.21.15.40:6829/12490","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T05:03:21.349150+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T05:03:21.644281+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T05:03:21.394040+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T05:03:21.587096+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T05:03:21.656497+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T05:03:21.510547+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T05:03:21.659358+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T05:03:21.628063+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2019-12-05T05:03:36.215 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.208+0000 7fa15c934700 1 -- 172.21.15.42:0/2381254106 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa13c03b200 msgr2=0x7fa13c03d710 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:36.216 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.208+0000 7fa15c934700 1 --2- 172.21.15.42:0/2381254106 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa13c03b200 0x7fa13c03d710 crc :-1 s=READY pgs=52 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:36.216 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.208+0000 7fa15c934700 1 -- 172.21.15.42:0/2381254106 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa158117940 msgr2=0x7fa1581431b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:36.216 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.208+0000 7fa15c934700 1 --2- 172.21.15.42:0/2381254106 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa158117940 0x7fa1581431b0 secure :-1 s=READY pgs=75 cs=0 l=1 rx=0x7fa140000e90 tx=0x7fa14000c2f0).stop 2019-12-05T05:03:36.216 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.208+0000 7fa15c934700 1 -- 172.21.15.42:0/2381254106 shutdown_connections 2019-12-05T05:03:36.216 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.208+0000 7fa15c934700 1 --2- 172.21.15.42:0/2381254106 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa13c03b200 0x7fa13c03d710 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:36.216 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.208+0000 7fa15c934700 1 --2- 172.21.15.42:0/2381254106 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa158117940 0x7fa1581431b0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:36.217 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.208+0000 7fa15c934700 1 -- 172.21.15.42:0/2381254106 shutdown_connections 2019-12-05T05:03:36.217 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.208+0000 7fa15c934700 1 -- 172.21.15.42:0/2381254106 wait complete. 2019-12-05T05:03:36.217 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.208+0000 7fa15c934700 1 -- 172.21.15.42:0/2381254106 >> 172.21.15.42:0/2381254106 conn(0x7fa1580a7740 msgr2=0x7fa15811c560 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:36.226 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2019-12-05T05:03:36.330 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.320+0000 7ff4a3a9a700 1 Processor -- start 2019-12-05T05:03:36.331 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.320+0000 7ff4a3a9a700 1 -- start start 2019-12-05T05:03:36.331 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.320+0000 7ff4a3a9a700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff49c09f550 0x7ff49c144630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:36.331 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.320+0000 7ff4a3a9a700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff49c14cf50 0x7ff49c14f390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:36.331 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.320+0000 7ff4a3a9a700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff49c07f080 con 0x7ff49c0a9a00 2019-12-05T05:03:36.331 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.320+0000 7ff4a3a9a700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff49c0a8f80 con 0x7ff49c09f550 2019-12-05T05:03:36.331 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.320+0000 7ff4a3a9a700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff49c0a9220 con 0x7ff49c14cf50 2019-12-05T05:03:36.332 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.320+0000 7ff4a1836700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7ff49c0a9a00 0x7ff49c07e880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:34372/0 (socket says 172.21.15.42:34372) 2019-12-05T05:03:36.332 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.320+0000 7ff4a2037700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff49c09f550 0x7ff49c144630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:36.332 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.320+0000 7ff4a2838700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff49c14cf50 0x7ff49c14f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:36.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.320+0000 7ff4a1836700 1 -- 172.21.15.42:0/371371725 learned_addr learned my addr 172.21.15.42:0/371371725 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:03:36.341 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.320+0000 7ff4a1035700 1 -- 172.21.15.42:0/371371725 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3190738686 0 0) 0x7ff49c07f080 con 0x7ff49c0a9a00 2019-12-05T05:03:36.341 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.320+0000 7ff4a2838700 1 -- 172.21.15.42:0/371371725 >> v2:172.21.15.42:3300/0 conn(0x7ff49c09f550 msgr2=0x7ff49c144630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:03:36.341 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.320+0000 7ff4a2838700 1 --2- 172.21.15.42:0/371371725 >> v2:172.21.15.42:3300/0 conn(0x7ff49c09f550 0x7ff49c144630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:03:36.341 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.320+0000 7ff4a2838700 1 -- 172.21.15.42:0/371371725 >> v1:172.21.15.40:6789/0 conn(0x7ff49c0a9a00 legacy=0x7ff49c07e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:36.341 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.320+0000 7ff4a2838700 1 -- 172.21.15.42:0/371371725 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff49c0a1000 con 0x7ff49c14cf50 2019-12-05T05:03:36.342 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.324+0000 7ff4a2838700 1 --2- 172.21.15.42:0/371371725 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff49c14cf50 0x7ff49c14f390 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7ff4940032b0 tx=0x7ff49400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:36.342 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.324+0000 7ff4a2838700 1 --2- 172.21.15.42:0/371371725 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff49c14cf50 0x7ff49c14f390 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7ff4940032b0 tx=0x7ff49400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:36.342 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.324+0000 7ff4a2838700 1 --2- 172.21.15.42:0/371371725 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff49c14cf50 0x7ff49c14f390 secure :-1 s=READY pgs=27 cs=0 l=1 rx=0x7ff4940032b0 tx=0x7ff49400fff0).ready entity=mon.2 client_cookie=70614b224595a4f server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:36.342 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.324+0000 7ff4a2838700 1 --2- 172.21.15.42:0/371371725 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff49c14cf50 0x7ff49c14f390 secure :-1 s=THROTTLE_DONE pgs=27 cs=0 l=1 rx=0x7ff4940032b0 tx=0x7ff49400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:36.342 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.324+0000 7ff4a2838700 1 --2- 172.21.15.42:0/371371725 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff49c14cf50 0x7ff49c14f390 secure :-1 s=THROTTLE_DONE pgs=27 cs=0 l=1 rx=0x7ff4940032b0 tx=0x7ff49400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:36.342 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.324+0000 7ff4a1035700 1 -- 172.21.15.42:0/371371725 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff494011a10 con 0x7ff49c14cf50 2019-12-05T05:03:36.342 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.324+0000 7ff4a2838700 1 --2- 172.21.15.42:0/371371725 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff49c14cf50 0x7ff49c14f390 secure :-1 s=THROTTLE_DONE pgs=27 cs=0 l=1 rx=0x7ff4940032b0 tx=0x7ff49400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:36.343 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.324+0000 7ff4a1035700 1 -- 172.21.15.42:0/371371725 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff494011630 con 0x7ff49c14cf50 2019-12-05T05:03:36.343 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.324+0000 7ff4a1035700 1 -- 172.21.15.42:0/371371725 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff49400a160 con 0x7ff49c14cf50 2019-12-05T05:03:36.343 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.324+0000 7ff4a3a9a700 1 -- 172.21.15.42:0/371371725 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff49c14cf50 msgr2=0x7ff49c14f390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:36.343 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.324+0000 7ff4a3a9a700 1 --2- 172.21.15.42:0/371371725 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff49c14cf50 0x7ff49c14f390 secure :-1 s=READY pgs=27 cs=0 l=1 rx=0x7ff4940032b0 tx=0x7ff49400fff0).stop 2019-12-05T05:03:36.343 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.324+0000 7ff4a3a9a700 1 -- 172.21.15.42:0/371371725 shutdown_connections 2019-12-05T05:03:36.343 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.324+0000 7ff4a3a9a700 1 --2- 172.21.15.42:0/371371725 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff49c14cf50 0x7ff49c14f390 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:36.343 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.324+0000 7ff4a3a9a700 1 --2- 172.21.15.42:0/371371725 >> v2:172.21.15.42:3300/0 conn(0x7ff49c09f550 0x7ff49c144630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:03:36.344 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.324+0000 7ff4a3a9a700 1 -- 172.21.15.42:0/371371725 shutdown_connections 2019-12-05T05:03:36.344 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.324+0000 7ff4a3a9a700 1 -- 172.21.15.42:0/371371725 wait complete. 2019-12-05T05:03:36.344 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.324+0000 7ff4a3a9a700 1 -- 172.21.15.42:0/371371725 >> 172.21.15.42:0/371371725 conn(0x7ff49c0a9ef0 msgr2=0x7ff49c0aa320 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:36.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.328+0000 7ff4a3a9a700 1 Processor -- start 2019-12-05T05:03:36.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.328+0000 7ff4a3a9a700 1 -- start start 2019-12-05T05:03:36.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.328+0000 7ff4a3a9a700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff49c123250 0x7ff49c07dac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:36.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.328+0000 7ff4a3a9a700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff49c07e010 0x7ff49c08e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:36.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.328+0000 7ff4a3a9a700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff49c08ffd0 con 0x7ff49c11ae30 2019-12-05T05:03:36.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.328+0000 7ff4a3a9a700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff49c143fc0 con 0x7ff49c123250 2019-12-05T05:03:36.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.328+0000 7ff4a3a9a700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff49c144260 con 0x7ff49c07e010 2019-12-05T05:03:36.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.328+0000 7ff4a2037700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff49c123250 0x7ff49c07dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:36.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.328+0000 7ff4a2037700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff49c123250 0x7ff49c07dac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:53130/0 (socket says 172.21.15.42:53130) 2019-12-05T05:03:36.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.328+0000 7ff4a2037700 1 -- 172.21.15.42:0/310791709 learned_addr learned my addr 172.21.15.42:0/310791709 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:03:36.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.328+0000 7ff4a1836700 1 --2- 172.21.15.42:0/310791709 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff49c07e010 0x7ff49c08e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:36.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.328+0000 7ff4a2037700 1 -- 172.21.15.42:0/310791709 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff49c07e010 msgr2=0x7ff49c08e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:36.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.328+0000 7ff4a2037700 1 --2- 172.21.15.42:0/310791709 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff49c07e010 0x7ff49c08e6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:36.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.328+0000 7ff4a2037700 1 -- 172.21.15.42:0/310791709 >> v1:172.21.15.42:6789/0 conn(0x7ff49c11ae30 legacy=0x7ff49c1303c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:36.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.328+0000 7ff492ffd700 1 -- 172.21.15.42:0/310791709 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1103200019 0 0) 0x7ff49c08ffd0 con 0x7ff49c11ae30 2019-12-05T05:03:36.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.328+0000 7ff4a2037700 1 -- 172.21.15.42:0/310791709 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff49c1447a0 con 0x7ff49c123250 2019-12-05T05:03:36.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.328+0000 7ff4a1836700 1 --2- 172.21.15.42:0/310791709 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff49c07e010 0x7ff49c08e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:03:36.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.328+0000 7ff4a2037700 1 --2- 172.21.15.42:0/310791709 >> v2:172.21.15.42:3300/0 conn(0x7ff49c123250 0x7ff49c07dac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7ff49c144180 tx=0x7ff498010c70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:36.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.328+0000 7ff4a2037700 1 --2- 172.21.15.42:0/310791709 >> v2:172.21.15.42:3300/0 conn(0x7ff49c123250 0x7ff49c07dac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7ff49c144180 tx=0x7ff498010c70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:36.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.328+0000 7ff4a2037700 1 --2- 172.21.15.42:0/310791709 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff49c123250 0x7ff49c07dac0 secure :-1 s=READY pgs=77 cs=0 l=1 rx=0x7ff49c144180 tx=0x7ff498010c70).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:36.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.328+0000 7ff4a2037700 1 --2- 172.21.15.42:0/310791709 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff49c123250 0x7ff49c07dac0 secure :-1 s=THROTTLE_DONE pgs=77 cs=0 l=1 rx=0x7ff49c144180 tx=0x7ff498010c70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:36.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.328+0000 7ff492ffd700 1 -- 172.21.15.42:0/310791709 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff498005930 con 0x7ff49c123250 2019-12-05T05:03:36.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.328+0000 7ff4a2037700 1 --2- 172.21.15.42:0/310791709 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff49c123250 0x7ff49c07dac0 secure :-1 s=THROTTLE_DONE pgs=77 cs=0 l=1 rx=0x7ff49c144180 tx=0x7ff498010c70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:36.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.328+0000 7ff4a2037700 1 --2- 172.21.15.42:0/310791709 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff49c123250 0x7ff49c07dac0 secure :-1 s=THROTTLE_DONE pgs=77 cs=0 l=1 rx=0x7ff49c144180 tx=0x7ff498010c70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:36.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.328+0000 7ff492ffd700 1 -- 172.21.15.42:0/310791709 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff498007160 con 0x7ff49c123250 2019-12-05T05:03:36.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.328+0000 7ff4a3a9a700 1 -- 172.21.15.42:0/310791709 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff49c08dd20 con 0x7ff49c123250 2019-12-05T05:03:36.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.328+0000 7ff492ffd700 1 -- 172.21.15.42:0/310791709 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff4980122b0 con 0x7ff49c123250 2019-12-05T05:03:36.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.328+0000 7ff4a3a9a700 1 -- 172.21.15.42:0/310791709 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff49c08ec20 con 0x7ff49c123250 2019-12-05T05:03:36.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.332+0000 7ff4a2037700 1 --2- 172.21.15.42:0/310791709 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff49c123250 0x7ff49c07dac0 secure :-1 s=THROTTLE_DONE pgs=77 cs=0 l=1 rx=0x7ff49c144180 tx=0x7ff498010c70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:36.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.332+0000 7ff492ffd700 1 -- 172.21.15.42:0/310791709 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7ff498012550 con 0x7ff49c123250 2019-12-05T05:03:36.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.332+0000 7ff4a2037700 1 --2- 172.21.15.42:0/310791709 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff49c123250 0x7ff49c07dac0 secure :-1 s=THROTTLE_DONE pgs=77 cs=0 l=1 rx=0x7ff49c144180 tx=0x7ff498010c70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:36.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.332+0000 7ff492ffd700 1 --2- 172.21.15.42:0/310791709 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff48403aca0 0x7ff48403d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:36.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.332+0000 7ff492ffd700 1 -- 172.21.15.42:0/310791709 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6031+0+0 (secure 0 0 0) 0x7ff49804b3f0 con 0x7ff49c123250 2019-12-05T05:03:36.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.332+0000 7ff4a1836700 1 --2- 172.21.15.42:0/310791709 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff48403aca0 0x7ff48403d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:36.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.332+0000 7ff4a1836700 1 --2- 172.21.15.42:0/310791709 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff48403aca0 0x7ff48403d1b0 crc :-1 s=READY pgs=53 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:36.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.336+0000 7ff4a3a9a700 1 -- 172.21.15.42:0/310791709 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff49c05d110 con 0x7ff49c123250 2019-12-05T05:03:36.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.340+0000 7ff4a2037700 1 --2- 172.21.15.42:0/310791709 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff49c123250 0x7ff49c07dac0 secure :-1 s=THROTTLE_DONE pgs=77 cs=0 l=1 rx=0x7ff49c144180 tx=0x7ff498010c70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:36.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.340+0000 7ff492ffd700 1 -- 172.21.15.42:0/310791709 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7ff498032450 con 0x7ff49c123250 2019-12-05T05:03:36.513 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:36.508+0000 7ff4a3a9a700 1 -- 172.21.15.42:0/310791709 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"} v 0) v1 -- 0x7ff49c05b8c0 con 0x7ff49c123250 2019-12-05T05:03:37.281 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.272+0000 7ff4a2037700 1 --2- 172.21.15.42:0/310791709 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff49c123250 0x7ff49c07dac0 secure :-1 s=THROTTLE_DONE pgs=77 cs=0 l=1 rx=0x7ff49c144180 tx=0x7ff498010c70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:37.281 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.272+0000 7ff492ffd700 1 -- 172.21.15.42:0/310791709 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"}]=0 v3) v1 ==== 118+0+0 (secure 0 0 0) 0x7ff4980326b0 con 0x7ff49c123250 2019-12-05T05:03:37.333 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.328+0000 7ff4a3a9a700 1 -- 172.21.15.42:0/310791709 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff48403aca0 msgr2=0x7ff48403d1b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:37.333 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.328+0000 7ff4a3a9a700 1 --2- 172.21.15.42:0/310791709 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff48403aca0 0x7ff48403d1b0 crc :-1 s=READY pgs=53 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:37.334 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.328+0000 7ff4a3a9a700 1 -- 172.21.15.42:0/310791709 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff49c123250 msgr2=0x7ff49c07dac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:37.334 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.328+0000 7ff4a3a9a700 1 --2- 172.21.15.42:0/310791709 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff49c123250 0x7ff49c07dac0 secure :-1 s=READY pgs=77 cs=0 l=1 rx=0x7ff49c144180 tx=0x7ff498010c70).stop 2019-12-05T05:03:37.334 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.328+0000 7ff4a3a9a700 1 -- 172.21.15.42:0/310791709 shutdown_connections 2019-12-05T05:03:37.334 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.328+0000 7ff4a3a9a700 1 --2- 172.21.15.42:0/310791709 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff48403aca0 0x7ff48403d1b0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:37.335 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.328+0000 7ff4a3a9a700 1 --2- 172.21.15.42:0/310791709 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff49c123250 0x7ff49c07dac0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:37.335 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.328+0000 7ff4a3a9a700 1 --2- 172.21.15.42:0/310791709 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff49c07e010 0x7ff49c08e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:37.335 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.328+0000 7ff4a3a9a700 1 -- 172.21.15.42:0/310791709 shutdown_connections 2019-12-05T05:03:37.335 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.328+0000 7ff4a3a9a700 1 -- 172.21.15.42:0/310791709 wait complete. 2019-12-05T05:03:37.335 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.328+0000 7ff4a3a9a700 1 -- 172.21.15.42:0/310791709 >> 172.21.15.42:0/310791709 conn(0x7ff49c059ef0 msgr2=0x7ff49c0a5060 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:37.348 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:03:37.448 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.432+0000 7f6cdb1b6700 1 Processor -- start 2019-12-05T05:03:37.448 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.436+0000 7f6cdb1b6700 1 -- start start 2019-12-05T05:03:37.449 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.436+0000 7f6cdb1b6700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f6cd40a7740 0x7f6cd408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:37.449 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.436+0000 7f6cdb1b6700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6cd407e720 0x7f6cd407eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:37.449 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.436+0000 7f6cdb1b6700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cd40a0710 con 0x7f6cd409f2e0 2019-12-05T05:03:37.449 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.436+0000 7f6cdb1b6700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6cd40a0b00 con 0x7f6cd40a7740 2019-12-05T05:03:37.449 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.436+0000 7f6cdb1b6700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cd408e5b0 con 0x7f6cd407e720 2019-12-05T05:03:37.449 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.436+0000 7f6cd9f54700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f6cd409f2e0 0x7f6cd409f710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:53180/0 (socket says 172.21.15.42:53180) 2019-12-05T05:03:37.449 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.436+0000 7f6cd9f54700 1 -- 172.21.15.42:0/4063620892 learned_addr learned my addr 172.21.15.42:0/4063620892 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:03:37.450 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.436+0000 7f6cd9753700 1 --2- 172.21.15.42:0/4063620892 >> v2:172.21.15.40:3300/0 conn(0x7f6cd40a7740 0x7f6cd408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:37.450 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.436+0000 7f6cd8f52700 1 --2- 172.21.15.42:0/4063620892 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6cd407e720 0x7f6cd407eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:37.450 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.436+0000 7f6cc7fff700 1 -- 172.21.15.42:0/4063620892 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1101633593 0 0) 0x7f6cd40a0710 con 0x7f6cd409f2e0 2019-12-05T05:03:37.450 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.436+0000 7f6cc7fff700 1 -- 172.21.15.42:0/4063620892 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6cc00030b0 con 0x7f6cd409f2e0 2019-12-05T05:03:37.450 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.436+0000 7f6cc7fff700 1 -- 172.21.15.42:0/4063620892 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1097068502 0 0) 0x7f6cc00030b0 con 0x7f6cd409f2e0 2019-12-05T05:03:37.450 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.436+0000 7f6cc7fff700 1 -- 172.21.15.42:0/4063620892 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6cd407e720 msgr2=0x7f6cd407eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:03:37.450 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.436+0000 7f6cc7fff700 1 --2- 172.21.15.42:0/4063620892 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6cd407e720 0x7f6cd407eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:03:37.451 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.436+0000 7f6cc7fff700 1 -- 172.21.15.42:0/4063620892 >> v2:172.21.15.40:3300/0 conn(0x7f6cd40a7740 msgr2=0x7f6cd408df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:03:37.451 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.436+0000 7f6cc7fff700 1 --2- 172.21.15.42:0/4063620892 >> v2:172.21.15.40:3300/0 conn(0x7f6cd40a7740 0x7f6cd408df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:03:37.451 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.436+0000 7f6cc7fff700 1 -- 172.21.15.42:0/4063620892 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cd407f0a0 con 0x7f6cd409f2e0 2019-12-05T05:03:37.451 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.436+0000 7f6cc7fff700 1 -- 172.21.15.42:0/4063620892 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7f6ccc001ad0 con 0x7f6cd409f2e0 2019-12-05T05:03:37.451 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.436+0000 7f6cc7fff700 1 -- 172.21.15.42:0/4063620892 >> v1:172.21.15.42:6789/0 conn(0x7f6cd409f2e0 legacy=0x7f6cd409f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:37.451 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.436+0000 7f6cc7fff700 1 --2- 172.21.15.42:0/4063620892 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6cc00036e0 0x7f6cc0003b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:37.451 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.436+0000 7f6cc7fff700 1 --2- 172.21.15.42:0/4063620892 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6cc0004420 0x7f6cc0004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:37.452 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.436+0000 7f6cc7fff700 1 --2- 172.21.15.42:0/4063620892 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6cc0004da0 0x7f6cc004f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:37.453 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.436+0000 7f6cc7fff700 1 -- 172.21.15.42:0/4063620892 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cd40a0b00 con 0x7f6cc0004da0 2019-12-05T05:03:37.453 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.436+0000 7f6cc7fff700 1 -- 172.21.15.42:0/4063620892 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cd408e5b0 con 0x7f6cc00036e0 2019-12-05T05:03:37.453 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.436+0000 7f6cc7fff700 1 -- 172.21.15.42:0/4063620892 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cc00041e0 con 0x7f6cc0004420 2019-12-05T05:03:37.453 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.436+0000 7f6cd9753700 1 --2- 172.21.15.42:0/4063620892 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6cc00036e0 0x7f6cc0003b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:37.453 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.436+0000 7f6cd9f54700 1 --2- 172.21.15.42:0/4063620892 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6cc0004da0 0x7f6cc004f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:37.453 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.436+0000 7f6cd8f52700 1 --2- 172.21.15.42:0/4063620892 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6cc0004420 0x7f6cc0004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:37.453 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.436+0000 7f6cd9753700 1 -- 172.21.15.42:0/4063620892 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6cc0004420 msgr2=0x7f6cc0004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:03:37.454 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.436+0000 7f6cd9753700 1 --2- 172.21.15.42:0/4063620892 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6cc0004420 0x7f6cc0004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:03:37.454 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.436+0000 7f6cd9753700 1 -- 172.21.15.42:0/4063620892 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6cc0004da0 msgr2=0x7f6cc004f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:03:37.454 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.436+0000 7f6cd9753700 1 --2- 172.21.15.42:0/4063620892 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6cc0004da0 0x7f6cc004f680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:03:37.454 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.436+0000 7f6cd9753700 1 -- 172.21.15.42:0/4063620892 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cc004fce0 con 0x7f6cc00036e0 2019-12-05T05:03:37.454 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.436+0000 7f6cd9753700 1 --2- 172.21.15.42:0/4063620892 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6cc00036e0 0x7f6cc0003b10 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f6cc00053f0 tx=0x7f6cc8010010).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:37.454 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.436+0000 7f6cd9753700 1 --2- 172.21.15.42:0/4063620892 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6cc00036e0 0x7f6cc0003b10 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f6cc00053f0 tx=0x7f6cc8010010).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:37.454 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.436+0000 7f6cd9753700 1 --2- 172.21.15.42:0/4063620892 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6cc00036e0 0x7f6cc0003b10 secure :-1 s=READY pgs=79 cs=0 l=1 rx=0x7f6cc00053f0 tx=0x7f6cc8010010).ready entity=mon.1 client_cookie=f5604a37b3d3ea05 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:37.455 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.436+0000 7f6cd9753700 1 --2- 172.21.15.42:0/4063620892 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6cc00036e0 0x7f6cc0003b10 secure :-1 s=THROTTLE_DONE pgs=79 cs=0 l=1 rx=0x7f6cc00053f0 tx=0x7f6cc8010010).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:37.455 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.436+0000 7f6cd9753700 1 --2- 172.21.15.42:0/4063620892 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6cc00036e0 0x7f6cc0003b10 secure :-1 s=THROTTLE_DONE pgs=79 cs=0 l=1 rx=0x7f6cc00053f0 tx=0x7f6cc8010010).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:37.455 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.436+0000 7f6cc7fff700 1 -- 172.21.15.42:0/4063620892 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6cc8010d00 con 0x7f6cc00036e0 2019-12-05T05:03:37.455 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.436+0000 7f6cd9753700 1 --2- 172.21.15.42:0/4063620892 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6cc00036e0 0x7f6cc0003b10 secure :-1 s=THROTTLE_DONE pgs=79 cs=0 l=1 rx=0x7f6cc00053f0 tx=0x7f6cc8010010).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:37.455 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.436+0000 7f6cc7fff700 1 -- 172.21.15.42:0/4063620892 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cc8010920 con 0x7f6cc00036e0 2019-12-05T05:03:37.455 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.436+0000 7f6cc7fff700 1 -- 172.21.15.42:0/4063620892 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6cc8008bd0 con 0x7f6cc00036e0 2019-12-05T05:03:37.455 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.436+0000 7f6cdb1b6700 1 -- 172.21.15.42:0/4063620892 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6cc00036e0 msgr2=0x7f6cc0003b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:37.456 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.436+0000 7f6cdb1b6700 1 --2- 172.21.15.42:0/4063620892 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6cc00036e0 0x7f6cc0003b10 secure :-1 s=READY pgs=79 cs=0 l=1 rx=0x7f6cc00053f0 tx=0x7f6cc8010010).stop 2019-12-05T05:03:37.456 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.436+0000 7f6cd9f54700 1 -- 172.21.15.42:0/4063620892 reap_dead start 2019-12-05T05:03:37.456 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.440+0000 7f6cdb1b6700 1 -- 172.21.15.42:0/4063620892 shutdown_connections 2019-12-05T05:03:37.456 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.440+0000 7f6cdb1b6700 1 -- 172.21.15.42:0/4063620892 shutdown_connections 2019-12-05T05:03:37.456 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.440+0000 7f6cdb1b6700 1 -- 172.21.15.42:0/4063620892 wait complete. 2019-12-05T05:03:37.456 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.440+0000 7f6cdb1b6700 1 -- 172.21.15.42:0/4063620892 >> 172.21.15.42:0/4063620892 conn(0x7f6cd40aabe0 msgr2=0x7f6cd40a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:37.456 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.440+0000 7f6cdb1b6700 1 Processor -- start 2019-12-05T05:03:37.457 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.440+0000 7f6cdb1b6700 1 -- start start 2019-12-05T05:03:37.460 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.444+0000 7f6cdb1b6700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6cd40655b0 0x7f6cd407aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:37.460 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.444+0000 7f6cdb1b6700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f6cd407afa0 0x7f6cd413ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:37.460 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.444+0000 7f6cdb1b6700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cd4065c10 con 0x7f6cd407c220 2019-12-05T05:03:37.461 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.444+0000 7f6cdb1b6700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6cd408f5d0 con 0x7f6cd407afa0 2019-12-05T05:03:37.461 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.444+0000 7f6cdb1b6700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cd407b3d0 con 0x7f6cd40655b0 2019-12-05T05:03:37.461 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.444+0000 7f6cd9753700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f6cd407c220 0x7f6cd4063e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:34400/0 (socket says 172.21.15.42:34400) 2019-12-05T05:03:37.461 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.444+0000 7f6cd8f52700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6cd40655b0 0x7f6cd407aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:37.461 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.444+0000 7f6cd9753700 1 -- 172.21.15.42:0/239041465 learned_addr learned my addr 172.21.15.42:0/239041465 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:03:37.461 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.444+0000 7f6cd9f54700 1 --2- 172.21.15.42:0/239041465 >> v2:172.21.15.40:3300/0 conn(0x7f6cd407afa0 0x7f6cd413ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:37.461 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.444+0000 7f6cc5ffb700 1 -- 172.21.15.42:0/239041465 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2756526806 0 0) 0x7f6cd4065c10 con 0x7f6cd407c220 2019-12-05T05:03:37.462 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.444+0000 7f6cc5ffb700 1 -- 172.21.15.42:0/239041465 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6cb40030b0 con 0x7f6cd407c220 2019-12-05T05:03:37.462 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.444+0000 7f6cd8f52700 1 -- 172.21.15.42:0/239041465 >> v2:172.21.15.40:3300/0 conn(0x7f6cd407afa0 msgr2=0x7f6cd413ac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:37.462 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.444+0000 7f6cd8f52700 1 --2- 172.21.15.42:0/239041465 >> v2:172.21.15.40:3300/0 conn(0x7f6cd407afa0 0x7f6cd413ac30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:37.462 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.444+0000 7f6cd8f52700 1 -- 172.21.15.42:0/239041465 >> v1:172.21.15.40:6789/0 conn(0x7f6cd407c220 legacy=0x7f6cd4063e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:37.463 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.444+0000 7f6cd8f52700 1 -- 172.21.15.42:0/239041465 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cd408f870 con 0x7f6cd40655b0 2019-12-05T05:03:37.463 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.444+0000 7f6cd9f54700 1 --2- 172.21.15.42:0/239041465 >> v2:172.21.15.40:3300/0 conn(0x7f6cd407afa0 0x7f6cd413ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:03:37.463 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.444+0000 7f6cd8f52700 1 --2- 172.21.15.42:0/239041465 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6cd40655b0 0x7f6cd407aa50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f6cd0002f30 tx=0x7f6cd0010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:37.463 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.444+0000 7f6cd8f52700 1 --2- 172.21.15.42:0/239041465 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6cd40655b0 0x7f6cd407aa50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f6cd0002f30 tx=0x7f6cd0010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:37.464 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.444+0000 7f6cd8f52700 1 --2- 172.21.15.42:0/239041465 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6cd40655b0 0x7f6cd407aa50 secure :-1 s=READY pgs=28 cs=0 l=1 rx=0x7f6cd0002f30 tx=0x7f6cd0010ab0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:37.464 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.444+0000 7f6cd8f52700 1 --2- 172.21.15.42:0/239041465 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6cd40655b0 0x7f6cd407aa50 secure :-1 s=THROTTLE_DONE pgs=28 cs=0 l=1 rx=0x7f6cd0002f30 tx=0x7f6cd0010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:37.464 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.444+0000 7f6cc5ffb700 1 -- 172.21.15.42:0/239041465 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6cd0005930 con 0x7f6cd40655b0 2019-12-05T05:03:37.464 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.444+0000 7f6cd8f52700 1 --2- 172.21.15.42:0/239041465 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6cd40655b0 0x7f6cd407aa50 secure :-1 s=THROTTLE_DONE pgs=28 cs=0 l=1 rx=0x7f6cd0002f30 tx=0x7f6cd0010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:37.464 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.444+0000 7f6cd8f52700 1 --2- 172.21.15.42:0/239041465 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6cd40655b0 0x7f6cd407aa50 secure :-1 s=THROTTLE_DONE pgs=28 cs=0 l=1 rx=0x7f6cd0002f30 tx=0x7f6cd0010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:37.464 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.444+0000 7f6cc5ffb700 1 -- 172.21.15.42:0/239041465 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cd0007160 con 0x7f6cd40655b0 2019-12-05T05:03:37.464 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.444+0000 7f6cdb1b6700 1 -- 172.21.15.42:0/239041465 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cd40a1510 con 0x7f6cd40655b0 2019-12-05T05:03:37.465 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.444+0000 7f6cc5ffb700 1 -- 172.21.15.42:0/239041465 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6cd0008260 con 0x7f6cd40655b0 2019-12-05T05:03:37.465 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.444+0000 7f6cdb1b6700 1 -- 172.21.15.42:0/239041465 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cd413b320 con 0x7f6cd40655b0 2019-12-05T05:03:37.465 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.448+0000 7f6cd8f52700 1 --2- 172.21.15.42:0/239041465 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6cd40655b0 0x7f6cd407aa50 secure :-1 s=THROTTLE_DONE pgs=28 cs=0 l=1 rx=0x7f6cd0002f30 tx=0x7f6cd0010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:37.465 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.448+0000 7f6cc5ffb700 1 -- 172.21.15.42:0/239041465 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f6cd0008580 con 0x7f6cd40655b0 2019-12-05T05:03:37.465 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.448+0000 7f6cd8f52700 1 --2- 172.21.15.42:0/239041465 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6cd40655b0 0x7f6cd407aa50 secure :-1 s=THROTTLE_DONE pgs=28 cs=0 l=1 rx=0x7f6cd0002f30 tx=0x7f6cd0010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:37.465 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.448+0000 7f6cc5ffb700 1 --2- 172.21.15.42:0/239041465 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6cb403b260 0x7f6cb403d770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:37.465 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.448+0000 7f6cc5ffb700 1 -- 172.21.15.42:0/239041465 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f6cd004b6a0 con 0x7f6cd40655b0 2019-12-05T05:03:37.466 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.448+0000 7f6cd9753700 1 --2- 172.21.15.42:0/239041465 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6cb403b260 0x7f6cb403d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:37.466 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.448+0000 7f6cd9753700 1 --2- 172.21.15.42:0/239041465 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6cb403b260 0x7f6cb403d770 crc :-1 s=READY pgs=54 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:37.466 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.448+0000 7f6cdb1b6700 1 -- 172.21.15.42:0/239041465 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6cd405d110 con 0x7f6cd40655b0 2019-12-05T05:03:37.467 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.460+0000 7f6cd8f52700 1 --2- 172.21.15.42:0/239041465 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6cd40655b0 0x7f6cd407aa50 secure :-1 s=THROTTLE_DONE pgs=28 cs=0 l=1 rx=0x7f6cd0002f30 tx=0x7f6cd0010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:37.467 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.460+0000 7f6cc5ffb700 1 -- 172.21.15.42:0/239041465 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f6cd0032660 con 0x7f6cd40655b0 2019-12-05T05:03:37.633 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.628+0000 7f6cdb1b6700 1 -- 172.21.15.42:0/239041465 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6cd405b8c0 con 0x7f6cd40655b0 2019-12-05T05:03:37.634 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.628+0000 7f6cd8f52700 1 --2- 172.21.15.42:0/239041465 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6cd40655b0 0x7f6cd407aa50 secure :-1 s=THROTTLE_DONE pgs=28 cs=0 l=1 rx=0x7f6cd0002f30 tx=0x7f6cd0010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:37.634 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.628+0000 7f6cc5ffb700 1 -- 172.21.15.42:0/239041465 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+1420 (secure 0 0 0) 0x7f6cd00328c0 con 0x7f6cd40655b0 2019-12-05T05:03:37.637 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:03:37.637 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":3,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":3,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:03:37.267417+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[],"up":{},"failed":[],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:03:37.637 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 3 2019-12-05T05:03:37.639 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.632+0000 7f6cdb1b6700 1 -- 172.21.15.42:0/239041465 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6cb403b260 msgr2=0x7f6cb403d770 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:37.639 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.632+0000 7f6cdb1b6700 1 --2- 172.21.15.42:0/239041465 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6cb403b260 0x7f6cb403d770 crc :-1 s=READY pgs=54 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:37.639 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.632+0000 7f6cdb1b6700 1 -- 172.21.15.42:0/239041465 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6cd40655b0 msgr2=0x7f6cd407aa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:37.639 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.632+0000 7f6cdb1b6700 1 --2- 172.21.15.42:0/239041465 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6cd40655b0 0x7f6cd407aa50 secure :-1 s=READY pgs=28 cs=0 l=1 rx=0x7f6cd0002f30 tx=0x7f6cd0010ab0).stop 2019-12-05T05:03:37.640 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.632+0000 7f6cdb1b6700 1 -- 172.21.15.42:0/239041465 shutdown_connections 2019-12-05T05:03:37.640 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.632+0000 7f6cdb1b6700 1 --2- 172.21.15.42:0/239041465 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6cb403b260 0x7f6cb403d770 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:37.640 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.632+0000 7f6cdb1b6700 1 --2- 172.21.15.42:0/239041465 >> v2:172.21.15.40:3300/0 conn(0x7f6cd407afa0 0x7f6cd413ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:37.640 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.632+0000 7f6cdb1b6700 1 --2- 172.21.15.42:0/239041465 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6cd40655b0 0x7f6cd407aa50 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:37.640 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.632+0000 7f6cdb1b6700 1 -- 172.21.15.42:0/239041465 shutdown_connections 2019-12-05T05:03:37.640 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.632+0000 7f6cdb1b6700 1 -- 172.21.15.42:0/239041465 wait complete. 2019-12-05T05:03:37.641 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.632+0000 7f6cdb1b6700 1 -- 172.21.15.42:0/239041465 >> 172.21.15.42:0/239041465 conn(0x7f6cd40a9c20 msgr2=0x7f6cd40a6a60 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:37.649 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2019-12-05T05:03:37.747 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.736+0000 7f78c8dcd700 1 Processor -- start 2019-12-05T05:03:37.747 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.736+0000 7f78c8dcd700 1 -- start start 2019-12-05T05:03:37.747 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.736+0000 7f78c8dcd700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f78c40a7740 0x7f78c408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:37.747 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.736+0000 7f78c8dcd700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78c40a0710 con 0x7f78c407e720 2019-12-05T05:03:37.747 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.736+0000 7f78c8dcd700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78c40a9ba0 con 0x7f78c409fae0 2019-12-05T05:03:37.748 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.736+0000 7f78c8dcd700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78c40a0b00 con 0x7f78c40a7740 2019-12-05T05:03:37.748 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.736+0000 7f78c359e700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f78c409fae0 0x7f78c414efe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:53204/0 (socket says 172.21.15.42:53204) 2019-12-05T05:03:37.748 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.736+0000 7f78c359e700 1 -- 172.21.15.42:0/4169499665 learned_addr learned my addr 172.21.15.42:0/4169499665 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:03:37.748 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.736+0000 7f78c2d9d700 1 --2- 172.21.15.42:0/4169499665 >> v2:172.21.15.40:3300/0 conn(0x7f78c40a7740 0x7f78c408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:37.748 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.736+0000 7f78c1d9b700 1 -- 172.21.15.42:0/4169499665 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1627321699 0 0) 0x7f78c40a9ba0 con 0x7f78c409fae0 2019-12-05T05:03:37.749 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.736+0000 7f78c1d9b700 1 -- 172.21.15.42:0/4169499665 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78ac0030b0 con 0x7f78c409fae0 2019-12-05T05:03:37.749 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.736+0000 7f78c1d9b700 1 -- 172.21.15.42:0/4169499665 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2375952530 0 0) 0x7f78c40a0710 con 0x7f78c407e720 2019-12-05T05:03:37.749 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.736+0000 7f78c2d9d700 1 -- 172.21.15.42:0/4169499665 >> v1:172.21.15.42:6789/0 conn(0x7f78c409fae0 legacy=0x7f78c414efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:37.750 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.736+0000 7f78c2d9d700 1 -- 172.21.15.42:0/4169499665 >> v1:172.21.15.40:6789/0 conn(0x7f78c407e720 legacy=0x7f78c409f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:37.750 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.736+0000 7f78c2d9d700 1 -- 172.21.15.42:0/4169499665 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78c407eb50 con 0x7f78c40a7740 2019-12-05T05:03:37.750 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.736+0000 7f78c2d9d700 1 --2- 172.21.15.42:0/4169499665 >> v2:172.21.15.40:3300/0 conn(0x7f78c40a7740 0x7f78c408df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f78b4008790 tx=0x7f78b400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:37.750 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.736+0000 7f78c2d9d700 1 --2- 172.21.15.42:0/4169499665 >> v2:172.21.15.40:3300/0 conn(0x7f78c40a7740 0x7f78c408df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f78b4008790 tx=0x7f78b400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:37.750 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.736+0000 7f78c2d9d700 1 --2- 172.21.15.42:0/4169499665 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f78c40a7740 0x7f78c408df80 secure :-1 s=READY pgs=76 cs=0 l=1 rx=0x7f78b4008790 tx=0x7f78b400fbf0).ready entity=mon.0 client_cookie=6479d67c63fb2326 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:37.751 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.740+0000 7f78c2d9d700 1 --2- 172.21.15.42:0/4169499665 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f78c40a7740 0x7f78c408df80 secure :-1 s=THROTTLE_DONE pgs=76 cs=0 l=1 rx=0x7f78b4008790 tx=0x7f78b400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:37.752 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.740+0000 7f78c1d9b700 1 -- 172.21.15.42:0/4169499665 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f78b4011270 con 0x7f78c40a7740 2019-12-05T05:03:37.753 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.740+0000 7f78c2d9d700 1 --2- 172.21.15.42:0/4169499665 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f78c40a7740 0x7f78c408df80 secure :-1 s=THROTTLE_DONE pgs=76 cs=0 l=1 rx=0x7f78b4008790 tx=0x7f78b400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:37.753 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.740+0000 7f78c2d9d700 1 --2- 172.21.15.42:0/4169499665 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f78c40a7740 0x7f78c408df80 secure :-1 s=THROTTLE_DONE pgs=76 cs=0 l=1 rx=0x7f78b4008790 tx=0x7f78b400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:37.753 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.740+0000 7f78c1d9b700 1 -- 172.21.15.42:0/4169499665 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78b4008bc0 con 0x7f78c40a7740 2019-12-05T05:03:37.753 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.740+0000 7f78c1d9b700 1 -- 172.21.15.42:0/4169499665 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f78b4011850 con 0x7f78c40a7740 2019-12-05T05:03:37.753 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.740+0000 7f78c8dcd700 1 -- 172.21.15.42:0/4169499665 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f78c40a7740 msgr2=0x7f78c408df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:37.753 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.740+0000 7f78c8dcd700 1 --2- 172.21.15.42:0/4169499665 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f78c40a7740 0x7f78c408df80 secure :-1 s=READY pgs=76 cs=0 l=1 rx=0x7f78b4008790 tx=0x7f78b400fbf0).stop 2019-12-05T05:03:37.753 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.740+0000 7f78c8dcd700 1 -- 172.21.15.42:0/4169499665 shutdown_connections 2019-12-05T05:03:37.754 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.740+0000 7f78c8dcd700 1 --2- 172.21.15.42:0/4169499665 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f78c40a7740 0x7f78c408df80 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:37.754 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.740+0000 7f78c8dcd700 1 -- 172.21.15.42:0/4169499665 shutdown_connections 2019-12-05T05:03:37.754 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.740+0000 7f78c8dcd700 1 -- 172.21.15.42:0/4169499665 wait complete. 2019-12-05T05:03:37.754 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.740+0000 7f78c8dcd700 1 -- 172.21.15.42:0/4169499665 >> 172.21.15.42:0/4169499665 conn(0x7f78c40aabe0 msgr2=0x7f78c40a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:37.758 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.740+0000 7f78c8dcd700 1 Processor -- start 2019-12-05T05:03:37.759 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.744+0000 7f78c8dcd700 1 -- start start 2019-12-05T05:03:37.759 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.744+0000 7f78c8dcd700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f78c4116500 0x7f78c4142a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:37.759 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.744+0000 7f78c8dcd700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f78c407a9c0 0x7f78c407adf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:37.759 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.744+0000 7f78c8dcd700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78c40a5c10 con 0x7f78c4128570 2019-12-05T05:03:37.759 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.744+0000 7f78c8dcd700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78c40a6770 con 0x7f78c4116500 2019-12-05T05:03:37.759 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.744+0000 7f78c8dcd700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78c40a5f90 con 0x7f78c407a9c0 2019-12-05T05:03:37.760 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.744+0000 7f78c259c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f78c407a9c0 0x7f78c407adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:37.760 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.744+0000 7f78c259c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f78c407a9c0 0x7f78c407adf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:53170/0 (socket says 172.21.15.42:53170) 2019-12-05T05:03:37.760 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.744+0000 7f78c259c700 1 -- 172.21.15.42:0/1545255214 learned_addr learned my addr 172.21.15.42:0/1545255214 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:03:37.760 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.744+0000 7f78c2d9d700 1 --2- 172.21.15.42:0/1545255214 >> v2:172.21.15.40:3300/0 conn(0x7f78c4116500 0x7f78c4142a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:37.760 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.744+0000 7f78c259c700 1 -- 172.21.15.42:0/1545255214 >> v2:172.21.15.40:3300/0 conn(0x7f78c4116500 msgr2=0x7f78c4142a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:37.761 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.744+0000 7f78c259c700 1 --2- 172.21.15.42:0/1545255214 >> v2:172.21.15.40:3300/0 conn(0x7f78c4116500 0x7f78c4142a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:37.761 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.744+0000 7f78c259c700 1 -- 172.21.15.42:0/1545255214 >> v1:172.21.15.40:6789/0 conn(0x7f78c4128570 legacy=0x7f78c408eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:37.761 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.744+0000 7f78c259c700 1 -- 172.21.15.42:0/1545255214 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78c41289a0 con 0x7f78c407a9c0 2019-12-05T05:03:37.761 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.744+0000 7f78b37fe700 1 -- 172.21.15.42:0/1545255214 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 172015470 0 0) 0x7f78c40a5c10 con 0x7f78c4128570 2019-12-05T05:03:37.761 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.744+0000 7f78c259c700 1 --2- 172.21.15.42:0/1545255214 >> v2:172.21.15.42:3300/0 conn(0x7f78c407a9c0 0x7f78c407adf0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f78b8000b80 tx=0x7f78b800c340).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:37.761 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.744+0000 7f78c259c700 1 --2- 172.21.15.42:0/1545255214 >> v2:172.21.15.42:3300/0 conn(0x7f78c407a9c0 0x7f78c407adf0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f78b8000b80 tx=0x7f78b800c340).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:37.761 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.744+0000 7f78c259c700 1 --2- 172.21.15.42:0/1545255214 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f78c407a9c0 0x7f78c407adf0 secure :-1 s=READY pgs=81 cs=0 l=1 rx=0x7f78b8000b80 tx=0x7f78b800c340).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:37.762 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.744+0000 7f78c259c700 1 --2- 172.21.15.42:0/1545255214 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f78c407a9c0 0x7f78c407adf0 secure :-1 s=THROTTLE_DONE pgs=81 cs=0 l=1 rx=0x7f78b8000b80 tx=0x7f78b800c340).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:37.762 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.744+0000 7f78b37fe700 1 -- 172.21.15.42:0/1545255214 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f78b800caa0 con 0x7f78c407a9c0 2019-12-05T05:03:37.762 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.744+0000 7f78c259c700 1 --2- 172.21.15.42:0/1545255214 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f78c407a9c0 0x7f78c407adf0 secure :-1 s=THROTTLE_DONE pgs=81 cs=0 l=1 rx=0x7f78b8000b80 tx=0x7f78b800c340).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:37.762 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.744+0000 7f78b37fe700 1 -- 172.21.15.42:0/1545255214 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78b800c6c0 con 0x7f78c407a9c0 2019-12-05T05:03:37.762 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.744+0000 7f78c259c700 1 --2- 172.21.15.42:0/1545255214 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f78c407a9c0 0x7f78c407adf0 secure :-1 s=THROTTLE_DONE pgs=81 cs=0 l=1 rx=0x7f78b8000b80 tx=0x7f78b800c340).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:37.762 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.744+0000 7f78c8dcd700 1 -- 172.21.15.42:0/1545255214 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78c4128c10 con 0x7f78c407a9c0 2019-12-05T05:03:37.762 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.744+0000 7f78b37fe700 1 -- 172.21.15.42:0/1545255214 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f78b8011120 con 0x7f78c407a9c0 2019-12-05T05:03:37.763 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.744+0000 7f78c8dcd700 1 -- 172.21.15.42:0/1545255214 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78c408f370 con 0x7f78c407a9c0 2019-12-05T05:03:37.763 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.748+0000 7f78c259c700 1 --2- 172.21.15.42:0/1545255214 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f78c407a9c0 0x7f78c407adf0 secure :-1 s=THROTTLE_DONE pgs=81 cs=0 l=1 rx=0x7f78b8000b80 tx=0x7f78b800c340).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:37.763 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.748+0000 7f78b37fe700 1 -- 172.21.15.42:0/1545255214 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f78b80113c0 con 0x7f78c407a9c0 2019-12-05T05:03:37.763 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.748+0000 7f78c259c700 1 --2- 172.21.15.42:0/1545255214 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f78c407a9c0 0x7f78c407adf0 secure :-1 s=THROTTLE_DONE pgs=81 cs=0 l=1 rx=0x7f78b8000b80 tx=0x7f78b800c340).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:37.763 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.748+0000 7f78b37fe700 1 --2- 172.21.15.42:0/1545255214 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f78a803acd0 0x7f78a803d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:37.763 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.748+0000 7f78b37fe700 1 -- 172.21.15.42:0/1545255214 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f78b804b1f0 con 0x7f78c407a9c0 2019-12-05T05:03:37.763 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.748+0000 7f78c2d9d700 1 --2- 172.21.15.42:0/1545255214 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f78a803acd0 0x7f78a803d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:37.764 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.748+0000 7f78c2d9d700 1 --2- 172.21.15.42:0/1545255214 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f78a803acd0 0x7f78a803d1e0 crc :-1 s=READY pgs=55 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:37.764 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.752+0000 7f78c8dcd700 1 -- 172.21.15.42:0/1545255214 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78c405d110 con 0x7f78c407a9c0 2019-12-05T05:03:37.764 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.756+0000 7f78c259c700 1 --2- 172.21.15.42:0/1545255214 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f78c407a9c0 0x7f78c407adf0 secure :-1 s=THROTTLE_DONE pgs=81 cs=0 l=1 rx=0x7f78b8000b80 tx=0x7f78b800c340).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:37.764 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.756+0000 7f78b37fe700 1 -- 172.21.15.42:0/1545255214 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f78b8032250 con 0x7f78c407a9c0 2019-12-05T05:03:37.926 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.920+0000 7f78c8dcd700 1 -- 172.21.15.42:0/1545255214 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f78c405b8c0 con 0x7f78c407a9c0 2019-12-05T05:03:37.928 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.924+0000 7f78c259c700 1 --2- 172.21.15.42:0/1545255214 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f78c407a9c0 0x7f78c407adf0 secure :-1 s=THROTTLE_DONE pgs=81 cs=0 l=1 rx=0x7f78b8000b80 tx=0x7f78b800c340).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:37.928 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.924+0000 7f78b37fe700 1 -- 172.21.15.42:0/1545255214 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+14885 (secure 0 0 0) 0x7f78b80324b0 con 0x7f78c407a9c0 2019-12-05T05:03:37.929 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:03:37.930 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":24,"fsid":"5d836cfc-a83b-4433-9b3e-b9b2f31c9050","created":"2019-12-05T05:03:13.215602+0000","modified":"2019-12-05T05:03:36.607704+0000","last_up_change":"2019-12-05T05:03:22.664783+0000","last_in_change":"2019-12-05T05:03:22.664783+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2019-12-05T05:03:27.394605+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2019-12-05T05:03:30.419845+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2019-12-05T05:03:33.249075+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e3fc49bb-dcbc-43ff-ac7c-9eb0ea0bd257","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6810","nonce":12118},{"type":"v1","addr":"172.21.15.42:6811","nonce":12118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6812","nonce":12118},{"type":"v1","addr":"172.21.15.42:6813","nonce":12118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6816","nonce":12118},{"type":"v1","addr":"172.21.15.42:6817","nonce":12118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6814","nonce":12118},{"type":"v1","addr":"172.21.15.42:6815","nonce":12118}]},"public_addr":"172.21.15.42:6811/12118","cluster_addr":"172.21.15.42:6813/12118","heartbeat_back_addr":"172.21.15.42:6817/12118","heartbeat_front_addr":"172.21.15.42:6815/12118","state":["exists","up"]},{"osd":1,"uuid":"902543fb-f5a2-4321-9c9c-26f9b9ca8629","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6826","nonce":12119},{"type":"v1","addr":"172.21.15.42:6827","nonce":12119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6828","nonce":12119},{"type":"v1","addr":"172.21.15.42:6829","nonce":12119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6832","nonce":12119},{"type":"v1","addr":"172.21.15.42:6833","nonce":12119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6830","nonce":12119},{"type":"v1","addr":"172.21.15.42:6831","nonce":12119}]},"public_addr":"172.21.15.42:6827/12119","cluster_addr":"172.21.15.42:6829/12119","heartbeat_back_addr":"172.21.15.42:6833/12119","heartbeat_front_addr":"172.21.15.42:6831/12119","state":["exists","up"]},{"osd":2,"uuid":"28739ff7-bed7-4b75-8ea5-b57d088f1250","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6802","nonce":12117},{"type":"v1","addr":"172.21.15.42:6803","nonce":12117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6804","nonce":12117},{"type":"v1","addr":"172.21.15.42:6805","nonce":12117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6808","nonce":12117},{"type":"v1","addr":"172.21.15.42:6809","nonce":12117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6806","nonce":12117},{"type":"v1","addr":"172.21.15.42:6807","nonce":12117}]},"public_addr":"172.21.15.42:6803/12117","cluster_addr":"172.21.15.42:6805/12117","heartbeat_back_addr":"172.21.15.42:6809/12117","heartbeat_front_addr":"172.21.15.42:6807/12117","state":["exists","up"]},{"osd":3,"uuid":"1960f66b-2d40-4bb9-a289-ce8cd0c1c142","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6818","nonce":12116},{"type":"v1","addr":"172.21.15.42:6819","nonce":12116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6820","nonce":12116},{"type":"v1","addr":"172.21.15.42:6821","nonce":12116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6824","nonce":12116},{"type":"v1","addr":"172.21.15.42:6825","nonce":12116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6822","nonce":12116},{"type":"v1","addr":"172.21.15.42:6823","nonce":12116}]},"public_addr":"172.21.15.42:6819/12116","cluster_addr":"172.21.15.42:6821/12116","heartbeat_back_addr":"172.21.15.42:6825/12116","heartbeat_front_addr":"172.21.15.42:6823/12116","state":["exists","up"]},{"osd":4,"uuid":"45bbf1bd-7c8a-4a0d-b76b-afa6148a855c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":12488},{"type":"v1","addr":"172.21.15.40:6809","nonce":12488}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":12488},{"type":"v1","addr":"172.21.15.40:6812","nonce":12488}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":12488},{"type":"v1","addr":"172.21.15.40:6825","nonce":12488}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":12488},{"type":"v1","addr":"172.21.15.40:6819","nonce":12488}]},"public_addr":"172.21.15.40:6809/12488","cluster_addr":"172.21.15.40:6812/12488","heartbeat_back_addr":"172.21.15.40:6825/12488","heartbeat_front_addr":"172.21.15.40:6819/12488","state":["exists","up"]},{"osd":5,"uuid":"8a5511cd-925f-4869-a49e-924383e4ce53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":12489},{"type":"v1","addr":"172.21.15.40:6801","nonce":12489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":12489},{"type":"v1","addr":"172.21.15.40:6803","nonce":12489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":12489},{"type":"v1","addr":"172.21.15.40:6807","nonce":12489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":12489},{"type":"v1","addr":"172.21.15.40:6805","nonce":12489}]},"public_addr":"172.21.15.40:6801/12489","cluster_addr":"172.21.15.40:6803/12489","heartbeat_back_addr":"172.21.15.40:6807/12489","heartbeat_front_addr":"172.21.15.40:6805/12489","state":["exists","up"]},{"osd":6,"uuid":"d692c179-379e-446c-8484-a219a9241e04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6811","nonce":12491},{"type":"v1","addr":"172.21.15.40:6814","nonce":12491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6817","nonce":12491},{"type":"v1","addr":"172.21.15.40:6818","nonce":12491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":12491},{"type":"v1","addr":"172.21.15.40:6828","nonce":12491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":12491},{"type":"v1","addr":"172.21.15.40:6824","nonce":12491}]},"public_addr":"172.21.15.40:6814/12491","cluster_addr":"172.21.15.40:6818/12491","heartbeat_back_addr":"172.21.15.40:6828/12491","heartbeat_front_addr":"172.21.15.40:6824/12491","state":["exists","up"]},{"osd":7,"uuid":"79c3c243-190e-4e35-b769-319a2077e7cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6813","nonce":12490},{"type":"v1","addr":"172.21.15.40:6816","nonce":12490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":12490},{"type":"v1","addr":"172.21.15.40:6823","nonce":12490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":12490},{"type":"v1","addr":"172.21.15.40:6831","nonce":12490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6827","nonce":12490},{"type":"v1","addr":"172.21.15.40:6829","nonce":12490}]},"public_addr":"172.21.15.40:6816/12490","cluster_addr":"172.21.15.40:6823/12490","heartbeat_back_addr":"172.21.15.40:6831/12490","heartbeat_front_addr":"172.21.15.40:6829/12490","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T05:03:21.349150+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T05:03:21.644281+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T05:03:21.394040+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T05:03:21.587096+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T05:03:21.656497+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T05:03:21.510547+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T05:03:21.659358+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T05:03:21.628063+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2019-12-05T05:03:37.932 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.924+0000 7f78c8dcd700 1 -- 172.21.15.42:0/1545255214 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f78a803acd0 msgr2=0x7f78a803d1e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:37.932 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.924+0000 7f78c8dcd700 1 --2- 172.21.15.42:0/1545255214 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f78a803acd0 0x7f78a803d1e0 crc :-1 s=READY pgs=55 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:37.932 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.924+0000 7f78c8dcd700 1 -- 172.21.15.42:0/1545255214 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f78c407a9c0 msgr2=0x7f78c407adf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:37.932 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.924+0000 7f78c8dcd700 1 --2- 172.21.15.42:0/1545255214 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f78c407a9c0 0x7f78c407adf0 secure :-1 s=READY pgs=81 cs=0 l=1 rx=0x7f78b8000b80 tx=0x7f78b800c340).stop 2019-12-05T05:03:37.932 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.924+0000 7f78c8dcd700 1 -- 172.21.15.42:0/1545255214 shutdown_connections 2019-12-05T05:03:37.932 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.924+0000 7f78c8dcd700 1 --2- 172.21.15.42:0/1545255214 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f78a803acd0 0x7f78a803d1e0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:37.933 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.924+0000 7f78c8dcd700 1 --2- 172.21.15.42:0/1545255214 >> v2:172.21.15.40:3300/0 conn(0x7f78c4116500 0x7f78c4142a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:37.933 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.924+0000 7f78c8dcd700 1 --2- 172.21.15.42:0/1545255214 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f78c407a9c0 0x7f78c407adf0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:37.933 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.928+0000 7f78c8dcd700 1 -- 172.21.15.42:0/1545255214 shutdown_connections 2019-12-05T05:03:37.933 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.928+0000 7f78c8dcd700 1 -- 172.21.15.42:0/1545255214 wait complete. 2019-12-05T05:03:37.933 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:37.928+0000 7f78c8dcd700 1 -- 172.21.15.42:0/1545255214 >> 172.21.15.42:0/1545255214 conn(0x7f78c40aabe0 msgr2=0x7f78c409f3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:37.942 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2019-12-05T05:03:37.943 INFO:tasks.ceph.mds.a:Restarting daemon 2019-12-05T05:03:37.943 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2019-12-05T05:03:37.947 INFO:tasks.ceph.mds.a:Started 2019-12-05T05:03:37.947 INFO:tasks.ceph.mds.b:Restarting daemon 2019-12-05T05:03:37.947 INFO:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2019-12-05T05:03:37.951 INFO:tasks.ceph.mds.b:Started 2019-12-05T05:03:37.951 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2019-12-05T05:03:37.952 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2019-12-05T05:03:38.011 INFO:tasks.ceph.mds.a.smithi042.stdout:starting mds.a at 2019-12-05T05:03:38.014 INFO:tasks.ceph.mds.b.smithi040.stdout:starting mds.b at 2019-12-05T05:03:38.045 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.036+0000 7f831c31b700 1 Processor -- start 2019-12-05T05:03:38.045 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.036+0000 7f831c31b700 1 -- start start 2019-12-05T05:03:38.045 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.036+0000 7f831c31b700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f83140a7740 0x7f831408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:38.045 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.036+0000 7f831c31b700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f831407e720 0x7f831407eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:38.045 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.036+0000 7f831c31b700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f831409f2e0 0x7f831409f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:38.046 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.036+0000 7f831c31b700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83140a0b00 con 0x7f83140a7740 2019-12-05T05:03:38.046 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.036+0000 7f831c31b700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f831408e5b0 con 0x7f831409f2e0 2019-12-05T05:03:38.046 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.036+0000 7f831c31b700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f831408e850 con 0x7f831407e720 2019-12-05T05:03:38.046 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.036+0000 7f831a0b7700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f831407e720 0x7f831407eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:38.046 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.036+0000 7f831a0b7700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f831407e720 0x7f831407eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:53654/0 (socket says 172.21.15.42:53654) 2019-12-05T05:03:38.046 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.036+0000 7f831a0b7700 1 -- 172.21.15.42:0/2881371466 learned_addr learned my addr 172.21.15.42:0/2881371466 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:03:38.047 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.036+0000 7f831a8b8700 1 --2- 172.21.15.42:0/2881371466 >> v2:172.21.15.40:3300/0 conn(0x7f83140a7740 0x7f831408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:38.047 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.036+0000 7f831b0b9700 1 --2- 172.21.15.42:0/2881371466 >> v2:172.21.15.42:3300/0 conn(0x7f831409f2e0 0x7f831409f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:38.047 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.036+0000 7f831a0b7700 1 -- 172.21.15.42:0/2881371466 >> v2:172.21.15.42:3300/0 conn(0x7f831409f2e0 msgr2=0x7f831409f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:03:38.047 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.036+0000 7f831a0b7700 1 --2- 172.21.15.42:0/2881371466 >> v2:172.21.15.42:3300/0 conn(0x7f831409f2e0 0x7f831409f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:03:38.047 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.036+0000 7f831a0b7700 1 -- 172.21.15.42:0/2881371466 >> v2:172.21.15.40:3300/0 conn(0x7f83140a7740 msgr2=0x7f831408df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:03:38.047 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.036+0000 7f831a0b7700 1 --2- 172.21.15.42:0/2881371466 >> v2:172.21.15.40:3300/0 conn(0x7f83140a7740 0x7f831408df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:03:38.047 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.036+0000 7f831a0b7700 1 -- 172.21.15.42:0/2881371466 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f831407f0a0 con 0x7f831407e720 2019-12-05T05:03:38.048 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.036+0000 7f831b0b9700 1 --2- 172.21.15.42:0/2881371466 >> v2:172.21.15.42:3300/0 conn(0x7f831409f2e0 0x7f831409f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T05:03:38.048 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.036+0000 7f831a0b7700 1 --2- 172.21.15.42:0/2881371466 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f831407e720 0x7f831407eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f830c0032b0 tx=0x7f830c010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:38.048 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.036+0000 7f831a0b7700 1 --2- 172.21.15.42:0/2881371466 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f831407e720 0x7f831407eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f830c0032b0 tx=0x7f830c010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:38.048 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.036+0000 7f831a0b7700 1 --2- 172.21.15.42:0/2881371466 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f831407e720 0x7f831407eb50 secure :-1 s=READY pgs=29 cs=0 l=1 rx=0x7f830c0032b0 tx=0x7f830c010130).ready entity=mon.2 client_cookie=fb8baf0c30f7d346 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:38.048 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.036+0000 7f831a0b7700 1 --2- 172.21.15.42:0/2881371466 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f831407e720 0x7f831407eb50 secure :-1 s=THROTTLE_DONE pgs=29 cs=0 l=1 rx=0x7f830c0032b0 tx=0x7f830c010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:38.048 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.036+0000 7f831a0b7700 1 --2- 172.21.15.42:0/2881371466 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f831407e720 0x7f831407eb50 secure :-1 s=THROTTLE_DONE pgs=29 cs=0 l=1 rx=0x7f830c0032b0 tx=0x7f830c010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:38.049 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.036+0000 7f83198b6700 1 -- 172.21.15.42:0/2881371466 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f830c010de0 con 0x7f831407e720 2019-12-05T05:03:38.049 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.036+0000 7f831a0b7700 1 --2- 172.21.15.42:0/2881371466 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f831407e720 0x7f831407eb50 secure :-1 s=THROTTLE_DONE pgs=29 cs=0 l=1 rx=0x7f830c0032b0 tx=0x7f830c010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:38.049 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.036+0000 7f83198b6700 1 -- 172.21.15.42:0/2881371466 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f830c010a00 con 0x7f831407e720 2019-12-05T05:03:38.049 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.036+0000 7f83198b6700 1 -- 172.21.15.42:0/2881371466 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f830c0114a0 con 0x7f831407e720 2019-12-05T05:03:38.049 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.036+0000 7f831c31b700 1 -- 172.21.15.42:0/2881371466 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f831407e720 msgr2=0x7f831407eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:38.049 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.036+0000 7f831c31b700 1 --2- 172.21.15.42:0/2881371466 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f831407e720 0x7f831407eb50 secure :-1 s=READY pgs=29 cs=0 l=1 rx=0x7f830c0032b0 tx=0x7f830c010130).stop 2019-12-05T05:03:38.051 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.040+0000 7f831c31b700 1 -- 172.21.15.42:0/2881371466 shutdown_connections 2019-12-05T05:03:38.051 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.040+0000 7f831c31b700 1 --2- 172.21.15.42:0/2881371466 >> v2:172.21.15.42:3300/0 conn(0x7f831409f2e0 0x7f831409f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:03:38.051 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.040+0000 7f831c31b700 1 --2- 172.21.15.42:0/2881371466 >> v2:172.21.15.40:3300/0 conn(0x7f83140a7740 0x7f831408df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:03:38.051 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.040+0000 7f831c31b700 1 --2- 172.21.15.42:0/2881371466 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f831407e720 0x7f831407eb50 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:38.052 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.040+0000 7f831c31b700 1 -- 172.21.15.42:0/2881371466 shutdown_connections 2019-12-05T05:03:38.052 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.040+0000 7f831c31b700 1 -- 172.21.15.42:0/2881371466 wait complete. 2019-12-05T05:03:38.052 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.040+0000 7f831c31b700 1 -- 172.21.15.42:0/2881371466 >> 172.21.15.42:0/2881371466 conn(0x7f83140aabe0 msgr2=0x7f83140a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:38.055 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.040+0000 7f831c31b700 1 Processor -- start 2019-12-05T05:03:38.055 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.040+0000 7f831c31b700 1 -- start start 2019-12-05T05:03:38.055 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.040+0000 7f831c31b700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f831408eeb0 0x7f8314147a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:38.056 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.040+0000 7f831c31b700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8314147f70 con 0x7f83141302e0 2019-12-05T05:03:38.056 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.040+0000 7f831c31b700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8314147640 con 0x7f831412ce30 2019-12-05T05:03:38.056 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.040+0000 7f831c31b700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8314148820 con 0x7f831408eeb0 2019-12-05T05:03:38.056 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.040+0000 7f831a0b7700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f831412ce30 0x7f831412d260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:53234/0 (socket says 172.21.15.42:53234) 2019-12-05T05:03:38.056 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.040+0000 7f831a0b7700 1 -- 172.21.15.42:0/3476446783 learned_addr learned my addr 172.21.15.42:0/3476446783 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:03:38.056 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.040+0000 7f831a8b8700 1 --2- 172.21.15.42:0/3476446783 >> v2:172.21.15.40:3300/0 conn(0x7f831408eeb0 0x7f8314147a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:38.057 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.044+0000 7f831a8b8700 1 -- 172.21.15.42:0/3476446783 >> v1:172.21.15.42:6789/0 conn(0x7f831412ce30 legacy=0x7f831412d260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:38.057 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.044+0000 7f831a8b8700 1 -- 172.21.15.42:0/3476446783 >> v1:172.21.15.40:6789/0 conn(0x7f83141302e0 legacy=0x7f8314130710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:38.057 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.044+0000 7f831a8b8700 1 -- 172.21.15.42:0/3476446783 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8314130f10 con 0x7f831408eeb0 2019-12-05T05:03:38.057 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.044+0000 7f831a8b8700 1 --2- 172.21.15.42:0/3476446783 >> v2:172.21.15.40:3300/0 conn(0x7f831408eeb0 0x7f8314147a20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f8310002f30 tx=0x7f83100109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:38.057 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.044+0000 7f831a8b8700 1 --2- 172.21.15.42:0/3476446783 >> v2:172.21.15.40:3300/0 conn(0x7f831408eeb0 0x7f8314147a20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f8310002f30 tx=0x7f83100109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:38.057 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.044+0000 7f831a8b8700 1 --2- 172.21.15.42:0/3476446783 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f831408eeb0 0x7f8314147a20 secure :-1 s=READY pgs=83 cs=0 l=1 rx=0x7f8310002f30 tx=0x7f83100109e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:38.057 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.044+0000 7f831a8b8700 1 --2- 172.21.15.42:0/3476446783 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f831408eeb0 0x7f8314147a20 secure :-1 s=THROTTLE_DONE pgs=83 cs=0 l=1 rx=0x7f8310002f30 tx=0x7f83100109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:38.058 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.044+0000 7f830b7fe700 1 -- 172.21.15.42:0/3476446783 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8310005920 con 0x7f831408eeb0 2019-12-05T05:03:38.058 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.044+0000 7f831a8b8700 1 --2- 172.21.15.42:0/3476446783 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f831408eeb0 0x7f8314147a20 secure :-1 s=THROTTLE_DONE pgs=83 cs=0 l=1 rx=0x7f8310002f30 tx=0x7f83100109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:38.058 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.044+0000 7f831a8b8700 1 --2- 172.21.15.42:0/3476446783 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f831408eeb0 0x7f8314147a20 secure :-1 s=THROTTLE_DONE pgs=83 cs=0 l=1 rx=0x7f8310002f30 tx=0x7f83100109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:38.058 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.044+0000 7f830b7fe700 1 -- 172.21.15.42:0/3476446783 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8310005540 con 0x7f831408eeb0 2019-12-05T05:03:38.058 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.044+0000 7f830b7fe700 1 -- 172.21.15.42:0/3476446783 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8310012700 con 0x7f831408eeb0 2019-12-05T05:03:38.059 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.044+0000 7f831c31b700 1 -- 172.21.15.42:0/3476446783 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8314138ac0 con 0x7f831408eeb0 2019-12-05T05:03:38.059 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.044+0000 7f831c31b700 1 -- 172.21.15.42:0/3476446783 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8314139070 con 0x7f831408eeb0 2019-12-05T05:03:38.059 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.044+0000 7f831a8b8700 1 --2- 172.21.15.42:0/3476446783 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f831408eeb0 0x7f8314147a20 secure :-1 s=THROTTLE_DONE pgs=83 cs=0 l=1 rx=0x7f8310002f30 tx=0x7f83100109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:38.059 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.044+0000 7f830b7fe700 1 -- 172.21.15.42:0/3476446783 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f8310012a10 con 0x7f831408eeb0 2019-12-05T05:03:38.059 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.044+0000 7f831a8b8700 1 --2- 172.21.15.42:0/3476446783 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f831408eeb0 0x7f8314147a20 secure :-1 s=THROTTLE_DONE pgs=83 cs=0 l=1 rx=0x7f8310002f30 tx=0x7f83100109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:38.059 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.044+0000 7f830b7fe700 1 --2- 172.21.15.42:0/3476446783 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f82f403ac80 0x7f82f403d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:38.059 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.044+0000 7f830b7fe700 1 -- 172.21.15.42:0/3476446783 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f831004af40 con 0x7f831408eeb0 2019-12-05T05:03:38.060 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.044+0000 7f831a0b7700 1 --2- 172.21.15.42:0/3476446783 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f82f403ac80 0x7f82f403d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:38.060 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.048+0000 7f831a0b7700 1 --2- 172.21.15.42:0/3476446783 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f82f403ac80 0x7f82f403d190 crc :-1 s=READY pgs=56 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:38.061 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.052+0000 7f831c31b700 1 -- 172.21.15.42:0/3476446783 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f831405d110 con 0x7f831408eeb0 2019-12-05T05:03:38.067 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.060+0000 7f831a8b8700 1 --2- 172.21.15.42:0/3476446783 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f831408eeb0 0x7f8314147a20 secure :-1 s=THROTTLE_DONE pgs=83 cs=0 l=1 rx=0x7f8310002f30 tx=0x7f83100109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:38.067 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.060+0000 7f830b7fe700 1 -- 172.21.15.42:0/3476446783 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f8310031fc0 con 0x7f831408eeb0 2019-12-05T05:03:38.236 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.232+0000 7f831c31b700 1 -- 172.21.15.42:0/3476446783 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f831405b8c0 con 0x7f831408eeb0 2019-12-05T05:03:38.238 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.232+0000 7f831a8b8700 1 --2- 172.21.15.42:0/3476446783 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f831408eeb0 0x7f8314147a20 secure :-1 s=THROTTLE_DONE pgs=83 cs=0 l=1 rx=0x7f8310002f30 tx=0x7f83100109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:38.238 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.232+0000 7f830b7fe700 1 -- 172.21.15.42:0/3476446783 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+14885 (secure 0 0 0) 0x7f8310032220 con 0x7f831408eeb0 2019-12-05T05:03:38.240 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:03:38.240 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":24,"fsid":"5d836cfc-a83b-4433-9b3e-b9b2f31c9050","created":"2019-12-05T05:03:13.215602+0000","modified":"2019-12-05T05:03:36.607704+0000","last_up_change":"2019-12-05T05:03:22.664783+0000","last_in_change":"2019-12-05T05:03:22.664783+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2019-12-05T05:03:27.394605+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2019-12-05T05:03:30.419845+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2019-12-05T05:03:33.249075+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e3fc49bb-dcbc-43ff-ac7c-9eb0ea0bd257","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6810","nonce":12118},{"type":"v1","addr":"172.21.15.42:6811","nonce":12118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6812","nonce":12118},{"type":"v1","addr":"172.21.15.42:6813","nonce":12118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6816","nonce":12118},{"type":"v1","addr":"172.21.15.42:6817","nonce":12118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6814","nonce":12118},{"type":"v1","addr":"172.21.15.42:6815","nonce":12118}]},"public_addr":"172.21.15.42:6811/12118","cluster_addr":"172.21.15.42:6813/12118","heartbeat_back_addr":"172.21.15.42:6817/12118","heartbeat_front_addr":"172.21.15.42:6815/12118","state":["exists","up"]},{"osd":1,"uuid":"902543fb-f5a2-4321-9c9c-26f9b9ca8629","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6826","nonce":12119},{"type":"v1","addr":"172.21.15.42:6827","nonce":12119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6828","nonce":12119},{"type":"v1","addr":"172.21.15.42:6829","nonce":12119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6832","nonce":12119},{"type":"v1","addr":"172.21.15.42:6833","nonce":12119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6830","nonce":12119},{"type":"v1","addr":"172.21.15.42:6831","nonce":12119}]},"public_addr":"172.21.15.42:6827/12119","cluster_addr":"172.21.15.42:6829/12119","heartbeat_back_addr":"172.21.15.42:6833/12119","heartbeat_front_addr":"172.21.15.42:6831/12119","state":["exists","up"]},{"osd":2,"uuid":"28739ff7-bed7-4b75-8ea5-b57d088f1250","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6802","nonce":12117},{"type":"v1","addr":"172.21.15.42:6803","nonce":12117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6804","nonce":12117},{"type":"v1","addr":"172.21.15.42:6805","nonce":12117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6808","nonce":12117},{"type":"v1","addr":"172.21.15.42:6809","nonce":12117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6806","nonce":12117},{"type":"v1","addr":"172.21.15.42:6807","nonce":12117}]},"public_addr":"172.21.15.42:6803/12117","cluster_addr":"172.21.15.42:6805/12117","heartbeat_back_addr":"172.21.15.42:6809/12117","heartbeat_front_addr":"172.21.15.42:6807/12117","state":["exists","up"]},{"osd":3,"uuid":"1960f66b-2d40-4bb9-a289-ce8cd0c1c142","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6818","nonce":12116},{"type":"v1","addr":"172.21.15.42:6819","nonce":12116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6820","nonce":12116},{"type":"v1","addr":"172.21.15.42:6821","nonce":12116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6824","nonce":12116},{"type":"v1","addr":"172.21.15.42:6825","nonce":12116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6822","nonce":12116},{"type":"v1","addr":"172.21.15.42:6823","nonce":12116}]},"public_addr":"172.21.15.42:6819/12116","cluster_addr":"172.21.15.42:6821/12116","heartbeat_back_addr":"172.21.15.42:6825/12116","heartbeat_front_addr":"172.21.15.42:6823/12116","state":["exists","up"]},{"osd":4,"uuid":"45bbf1bd-7c8a-4a0d-b76b-afa6148a855c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":12488},{"type":"v1","addr":"172.21.15.40:6809","nonce":12488}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":12488},{"type":"v1","addr":"172.21.15.40:6812","nonce":12488}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":12488},{"type":"v1","addr":"172.21.15.40:6825","nonce":12488}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":12488},{"type":"v1","addr":"172.21.15.40:6819","nonce":12488}]},"public_addr":"172.21.15.40:6809/12488","cluster_addr":"172.21.15.40:6812/12488","heartbeat_back_addr":"172.21.15.40:6825/12488","heartbeat_front_addr":"172.21.15.40:6819/12488","state":["exists","up"]},{"osd":5,"uuid":"8a5511cd-925f-4869-a49e-924383e4ce53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":12489},{"type":"v1","addr":"172.21.15.40:6801","nonce":12489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":12489},{"type":"v1","addr":"172.21.15.40:6803","nonce":12489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":12489},{"type":"v1","addr":"172.21.15.40:6807","nonce":12489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":12489},{"type":"v1","addr":"172.21.15.40:6805","nonce":12489}]},"public_addr":"172.21.15.40:6801/12489","cluster_addr":"172.21.15.40:6803/12489","heartbeat_back_addr":"172.21.15.40:6807/12489","heartbeat_front_addr":"172.21.15.40:6805/12489","state":["exists","up"]},{"osd":6,"uuid":"d692c179-379e-446c-8484-a219a9241e04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6811","nonce":12491},{"type":"v1","addr":"172.21.15.40:6814","nonce":12491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6817","nonce":12491},{"type":"v1","addr":"172.21.15.40:6818","nonce":12491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":12491},{"type":"v1","addr":"172.21.15.40:6828","nonce":12491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":12491},{"type":"v1","addr":"172.21.15.40:6824","nonce":12491}]},"public_addr":"172.21.15.40:6814/12491","cluster_addr":"172.21.15.40:6818/12491","heartbeat_back_addr":"172.21.15.40:6828/12491","heartbeat_front_addr":"172.21.15.40:6824/12491","state":["exists","up"]},{"osd":7,"uuid":"79c3c243-190e-4e35-b769-319a2077e7cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6813","nonce":12490},{"type":"v1","addr":"172.21.15.40:6816","nonce":12490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":12490},{"type":"v1","addr":"172.21.15.40:6823","nonce":12490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":12490},{"type":"v1","addr":"172.21.15.40:6831","nonce":12490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6827","nonce":12490},{"type":"v1","addr":"172.21.15.40:6829","nonce":12490}]},"public_addr":"172.21.15.40:6816/12490","cluster_addr":"172.21.15.40:6823/12490","heartbeat_back_addr":"172.21.15.40:6831/12490","heartbeat_front_addr":"172.21.15.40:6829/12490","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T05:03:21.349150+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T05:03:21.644281+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T05:03:21.394040+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T05:03:21.587096+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T05:03:21.656497+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T05:03:21.510547+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T05:03:21.659358+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T05:03:21.628063+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2019-12-05T05:03:38.242 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.236+0000 7f831c31b700 1 -- 172.21.15.42:0/3476446783 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f82f403ac80 msgr2=0x7f82f403d190 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:38.243 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.236+0000 7f831c31b700 1 --2- 172.21.15.42:0/3476446783 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f82f403ac80 0x7f82f403d190 crc :-1 s=READY pgs=56 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:38.243 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.236+0000 7f831c31b700 1 -- 172.21.15.42:0/3476446783 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f831408eeb0 msgr2=0x7f8314147a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:38.243 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.236+0000 7f831c31b700 1 --2- 172.21.15.42:0/3476446783 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f831408eeb0 0x7f8314147a20 secure :-1 s=READY pgs=83 cs=0 l=1 rx=0x7f8310002f30 tx=0x7f83100109e0).stop 2019-12-05T05:03:38.243 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.236+0000 7f831c31b700 1 -- 172.21.15.42:0/3476446783 shutdown_connections 2019-12-05T05:03:38.243 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.236+0000 7f831c31b700 1 --2- 172.21.15.42:0/3476446783 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f82f403ac80 0x7f82f403d190 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:38.243 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.236+0000 7f831c31b700 1 --2- 172.21.15.42:0/3476446783 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f831408eeb0 0x7f8314147a20 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:38.244 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.236+0000 7f831c31b700 1 -- 172.21.15.42:0/3476446783 shutdown_connections 2019-12-05T05:03:38.244 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.236+0000 7f831c31b700 1 -- 172.21.15.42:0/3476446783 wait complete. 2019-12-05T05:03:38.244 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.236+0000 7f831c31b700 1 -- 172.21.15.42:0/3476446783 >> 172.21.15.42:0/3476446783 conn(0x7f83140aabe0 msgr2=0x7f831407e4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:38.253 INFO:tasks.ceph.ceph_manager.ceph:[{u'cache_target_full_ratio_micro': 800000, u'fast_read': False, u'stripe_width': 0, u'flags_names': u'hashpspool', u'tier_of': -1, u'hit_set_grade_decay_rate': 0, u'pg_placement_num': 8, u'use_gmt_hitset': True, u'cache_target_dirty_ratio_micro': 400000, u'create_time': u'2019-12-05T05:03:27.394605+0000', u'snap_epoch': 0, u'erasure_code_profile': u'', u'pg_autoscale_mode': u'off', u'snap_seq': 0, u'expected_num_objects': 0, u'size': 2, u'pg_num_pending': 8, u'auid': 0, u'cache_min_flush_age': 0, u'hit_set_period': 0, u'min_read_recency_for_promote': 0, u'target_max_objects': 0, u'pg_placement_num_target': 8, u'pg_num': 8, u'type': 1, u'grade_table': [], u'pool_name': u'rbd', u'cache_min_evict_age': 0, u'snap_mode': u'selfmanaged', u'pg_num_target': 8, u'cache_mode': u'none', u'min_size': 1, u'cache_target_dirty_high_ratio_micro': 600000, u'object_hash': 2, u'last_pg_merge_meta': {u'ready_epoch': 0, u'source_version': u"0'0", u'source_pgid': u'0.0', u'last_epoch_clean': 0, u'target_version': u"0'0", u'last_epoch_started': 0}, u'write_tier': -1, u'last_force_op_resend_preluminous': u'0', u'pool': 1, u'removed_snaps': u'[]', u'crush_rule': 0, u'tiers': [], u'hit_set_params': {u'type': u'none'}, u'last_force_op_resend': u'0', u'pool_snaps': [], u'quota_max_objects': 0, u'last_force_op_resend_prenautilus': u'0', u'application_metadata': {u'rbd': {}}, u'options': {}, u'hit_set_count': 0, u'flags': 1, u'target_max_bytes': 0, u'quota_max_bytes': 0, u'hit_set_search_last_n': 0, u'last_change': u'17', u'min_write_recency_for_promote': 0, u'read_tier': -1}, {u'cache_target_full_ratio_micro': 800000, u'fast_read': False, u'stripe_width': 0, u'flags_names': u'hashpspool', u'tier_of': -1, u'hit_set_grade_decay_rate': 0, u'pg_placement_num': 8, u'use_gmt_hitset': True, u'cache_target_dirty_ratio_micro': 400000, u'create_time': u'2019-12-05T05:03:30.419845+0000', u'snap_epoch': 0, u'erasure_code_profile': u'', u'pg_autoscale_mode': u'off', u'snap_seq': 0, u'expected_num_objects': 0, u'size': 2, u'pg_num_pending': 8, u'auid': 0, u'cache_min_flush_age': 0, u'hit_set_period': 0, u'min_read_recency_for_promote': 0, u'target_max_objects': 0, u'pg_placement_num_target': 8, u'pg_num': 8, u'type': 1, u'grade_table': [], u'pool_name': u'cephfs_metadata', u'cache_min_evict_age': 0, u'snap_mode': u'selfmanaged', u'pg_num_target': 8, u'cache_mode': u'none', u'min_size': 1, u'cache_target_dirty_high_ratio_micro': 600000, u'object_hash': 2, u'last_pg_merge_meta': {u'ready_epoch': 0, u'source_version': u"0'0", u'source_pgid': u'0.0', u'last_epoch_clean': 0, u'target_version': u"0'0", u'last_epoch_started': 0}, u'write_tier': -1, u'last_force_op_resend_preluminous': u'0', u'pool': 2, u'removed_snaps': u'[]', u'crush_rule': 0, u'tiers': [], u'hit_set_params': {u'type': u'none'}, u'last_force_op_resend': u'0', u'pool_snaps': [], u'quota_max_objects': 0, u'last_force_op_resend_prenautilus': u'0', u'application_metadata': {u'cephfs': {u'metadata': u'cephfs'}}, u'options': {u'pg_autoscale_bias': 4, u'recovery_priority': 5, u'pg_num_min': 16}, u'hit_set_count': 0, u'flags': 1, u'target_max_bytes': 0, u'quota_max_bytes': 0, u'hit_set_search_last_n': 0, u'last_change': u'23', u'min_write_recency_for_promote': 0, u'read_tier': -1}, {u'cache_target_full_ratio_micro': 800000, u'fast_read': False, u'stripe_width': 8192, u'flags_names': u'hashpspool,ec_overwrites', u'tier_of': -1, u'hit_set_grade_decay_rate': 0, u'pg_placement_num': 8, u'use_gmt_hitset': True, u'cache_target_dirty_ratio_micro': 400000, u'create_time': u'2019-12-05T05:03:33.249075+0000', u'snap_epoch': 0, u'erasure_code_profile': u'cephfs_data', u'pg_autoscale_mode': u'off', u'snap_seq': 0, u'expected_num_objects': 0, u'size': 4, u'pg_num_pending': 8, u'auid': 0, u'cache_min_flush_age': 0, u'hit_set_period': 0, u'min_read_recency_for_promote': 0, u'target_max_objects': 0, u'pg_placement_num_target': 8, u'pg_num': 8, u'type': 3, u'grade_table': [], u'pool_name': u'cephfs_data', u'cache_min_evict_age': 0, u'snap_mode': u'selfmanaged', u'pg_num_target': 8, u'cache_mode': u'none', u'min_size': 3, u'cache_target_dirty_high_ratio_micro': 600000, u'object_hash': 2, u'last_pg_merge_meta': {u'ready_epoch': 0, u'source_version': u"0'0", u'source_pgid': u'0.0', u'last_epoch_clean': 0, u'target_version': u"0'0", u'last_epoch_started': 0}, u'write_tier': -1, u'last_force_op_resend_preluminous': u'0', u'pool': 3, u'removed_snaps': u'[]', u'crush_rule': 1, u'tiers': [], u'hit_set_params': {u'type': u'none'}, u'last_force_op_resend': u'0', u'pool_snaps': [], u'quota_max_objects': 0, u'last_force_op_resend_prenautilus': u'0', u'application_metadata': {u'cephfs': {u'data': u'cephfs'}}, u'options': {}, u'hit_set_count': 0, u'flags': 5, u'target_max_bytes': 0, u'quota_max_bytes': 0, u'hit_set_search_last_n': 0, u'last_change': u'24', u'min_write_recency_for_promote': 0, u'read_tier': -1}] 2019-12-05T05:03:38.253 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2019-12-05T05:03:38.366 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.356+0000 7f8ebd4a7700 1 Processor -- start 2019-12-05T05:03:38.367 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.356+0000 7f8ebd4a7700 1 -- start start 2019-12-05T05:03:38.367 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.356+0000 7f8ebd4a7700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f8eb814df60 0x7f8eb81503a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:38.367 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.356+0000 7f8ebd4a7700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8eb807f080 con 0x7f8eb80a9a00 2019-12-05T05:03:38.367 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.356+0000 7f8ebd4a7700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8eb80a1000 con 0x7f8eb809f550 2019-12-05T05:03:38.367 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.356+0000 7f8ebd4a7700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8eb80a8f80 con 0x7f8eb814df60 2019-12-05T05:03:38.367 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.356+0000 7f8eb7fff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f8eb814df60 0x7f8eb81503a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:38.368 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.356+0000 7f8eb7fff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f8eb814df60 0x7f8eb81503a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:53206/0 (socket says 172.21.15.42:53206) 2019-12-05T05:03:38.368 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.356+0000 7f8eb7fff700 1 -- 172.21.15.42:0/781847721 learned_addr learned my addr 172.21.15.42:0/781847721 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:03:38.369 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.356+0000 7f8eb77fe700 1 --1- 172.21.15.42:0/781847721 >> v1:172.21.15.42:6789/0 conn(0x7f8eb809f550 0x7f8eb8144630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:53240/0 (socket says 172.21.15.42:53240) 2019-12-05T05:03:38.369 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.356+0000 7f8eb7fff700 1 -- 172.21.15.42:0/781847721 >> v1:172.21.15.42:6789/0 conn(0x7f8eb809f550 legacy=0x7f8eb8144630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:38.369 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.356+0000 7f8eb7fff700 1 -- 172.21.15.42:0/781847721 >> v1:172.21.15.40:6789/0 conn(0x7f8eb80a9a00 legacy=0x7f8eb807e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:38.369 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.356+0000 7f8eb7fff700 1 -- 172.21.15.42:0/781847721 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8eb809ee30 con 0x7f8eb814df60 2019-12-05T05:03:38.369 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.356+0000 7f8eb7fff700 1 --2- 172.21.15.42:0/781847721 >> v2:172.21.15.42:3300/0 conn(0x7f8eb814df60 0x7f8eb81503a0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f8eb00032b0 tx=0x7f8eb0010030).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:38.369 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.356+0000 7f8eb7fff700 1 --2- 172.21.15.42:0/781847721 >> v2:172.21.15.42:3300/0 conn(0x7f8eb814df60 0x7f8eb81503a0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f8eb00032b0 tx=0x7f8eb0010030).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:38.369 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.356+0000 7f8eb7fff700 1 --2- 172.21.15.42:0/781847721 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8eb814df60 0x7f8eb81503a0 secure :-1 s=READY pgs=88 cs=0 l=1 rx=0x7f8eb00032b0 tx=0x7f8eb0010030).ready entity=mon.1 client_cookie=eff97f2f8a37da85 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:38.370 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.356+0000 7f8eb7fff700 1 --2- 172.21.15.42:0/781847721 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8eb814df60 0x7f8eb81503a0 secure :-1 s=THROTTLE_DONE pgs=88 cs=0 l=1 rx=0x7f8eb00032b0 tx=0x7f8eb0010030).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:38.370 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.356+0000 7f8eb67fc700 1 -- 172.21.15.42:0/781847721 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8eb0010dd0 con 0x7f8eb814df60 2019-12-05T05:03:38.370 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.356+0000 7f8eb7fff700 1 --2- 172.21.15.42:0/781847721 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8eb814df60 0x7f8eb81503a0 secure :-1 s=THROTTLE_DONE pgs=88 cs=0 l=1 rx=0x7f8eb00032b0 tx=0x7f8eb0010030).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:38.370 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.356+0000 7f8eb67fc700 1 -- 172.21.15.42:0/781847721 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8eb00109f0 con 0x7f8eb814df60 2019-12-05T05:03:38.370 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.356+0000 7f8eb7fff700 1 --2- 172.21.15.42:0/781847721 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8eb814df60 0x7f8eb81503a0 secure :-1 s=THROTTLE_DONE pgs=88 cs=0 l=1 rx=0x7f8eb00032b0 tx=0x7f8eb0010030).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:38.370 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.356+0000 7f8eb67fc700 1 -- 172.21.15.42:0/781847721 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8eb00114a0 con 0x7f8eb814df60 2019-12-05T05:03:38.370 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.356+0000 7f8ebd4a7700 1 -- 172.21.15.42:0/781847721 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8eb814df60 msgr2=0x7f8eb81503a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:38.371 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.356+0000 7f8ebd4a7700 1 --2- 172.21.15.42:0/781847721 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8eb814df60 0x7f8eb81503a0 secure :-1 s=READY pgs=88 cs=0 l=1 rx=0x7f8eb00032b0 tx=0x7f8eb0010030).stop 2019-12-05T05:03:38.371 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.356+0000 7f8ebd4a7700 1 -- 172.21.15.42:0/781847721 shutdown_connections 2019-12-05T05:03:38.371 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.356+0000 7f8ebd4a7700 1 --2- 172.21.15.42:0/781847721 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8eb814df60 0x7f8eb81503a0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:38.371 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.356+0000 7f8ebd4a7700 1 -- 172.21.15.42:0/781847721 shutdown_connections 2019-12-05T05:03:38.371 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.360+0000 7f8ebd4a7700 1 -- 172.21.15.42:0/781847721 wait complete. 2019-12-05T05:03:38.371 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.360+0000 7f8ebd4a7700 1 -- 172.21.15.42:0/781847721 >> 172.21.15.42:0/781847721 conn(0x7f8eb80a9ef0 msgr2=0x7f8eb80aa320 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:38.371 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.360+0000 7f8ebd4a7700 1 Processor -- start 2019-12-05T05:03:38.372 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.360+0000 7f8ebd4a7700 1 -- start start 2019-12-05T05:03:38.378 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.360+0000 7f8ebd4a7700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8eb8119fd0 0x7f8eb807dc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:38.379 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.360+0000 7f8ebd4a7700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8eb8117850 0x7f8eb808e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:38.379 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.360+0000 7f8ebd4a7700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8eb807e160 con 0x7f8eb808ed10 2019-12-05T05:03:38.379 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.360+0000 7f8ebd4a7700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8eb8118e70 con 0x7f8eb8117850 2019-12-05T05:03:38.379 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.360+0000 7f8ebd4a7700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8eb8118930 con 0x7f8eb8119fd0 2019-12-05T05:03:38.379 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.360+0000 7f8eb7fff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f8eb808ed10 0x7f8eb8114e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:53246/0 (socket says 172.21.15.42:53246) 2019-12-05T05:03:38.379 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.360+0000 7f8eb7fff700 1 -- 172.21.15.42:0/1240395636 learned_addr learned my addr 172.21.15.42:0/1240395636 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:03:38.380 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.360+0000 7f8eb6ffd700 1 --2- 172.21.15.42:0/1240395636 >> v2:172.21.15.40:3300/0 conn(0x7f8eb8117850 0x7f8eb808e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:38.380 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.364+0000 7f8eb77fe700 1 --2- 172.21.15.42:0/1240395636 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8eb8119fd0 0x7f8eb807dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:38.380 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.364+0000 7f8e9ffff700 1 -- 172.21.15.42:0/1240395636 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3232421662 0 0) 0x7f8eb807e160 con 0x7f8eb808ed10 2019-12-05T05:03:38.380 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.364+0000 7f8e9ffff700 1 -- 172.21.15.42:0/1240395636 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e980030b0 con 0x7f8eb808ed10 2019-12-05T05:03:38.380 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.364+0000 7f8e9ffff700 1 -- 172.21.15.42:0/1240395636 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 3820462271 0 0) 0x7f8e980030b0 con 0x7f8eb808ed10 2019-12-05T05:03:38.380 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.364+0000 7f8e9ffff700 1 -- 172.21.15.42:0/1240395636 --> v1:172.21.15.42:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8eb807e160 con 0x7f8eb808ed10 2019-12-05T05:03:38.381 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.364+0000 7f8e9ffff700 1 -- 172.21.15.42:0/1240395636 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7f8eb00087f0 con 0x7f8eb808ed10 2019-12-05T05:03:38.381 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.364+0000 7f8e9ffff700 1 -- 172.21.15.42:0/1240395636 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8eb8119fd0 msgr2=0x7f8eb807dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:38.381 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.364+0000 7f8e9ffff700 1 --2- 172.21.15.42:0/1240395636 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8eb8119fd0 0x7f8eb807dc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:38.381 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.364+0000 7f8e9ffff700 1 -- 172.21.15.42:0/1240395636 >> v2:172.21.15.40:3300/0 conn(0x7f8eb8117850 msgr2=0x7f8eb808e7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:38.381 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.364+0000 7f8e9ffff700 1 --2- 172.21.15.42:0/1240395636 >> v2:172.21.15.40:3300/0 conn(0x7f8eb8117850 0x7f8eb808e7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:38.381 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.364+0000 7f8e9ffff700 1 -- 172.21.15.42:0/1240395636 >> v1:172.21.15.42:6789/0 conn(0x7f8eb808ed10 legacy=0x7f8eb8114e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:38.381 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.364+0000 7f8e9ffff700 1 --2- 172.21.15.42:0/1240395636 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e980038a0 0x7f8e98003d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:38.382 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.364+0000 7f8e9ffff700 1 --2- 172.21.15.42:0/1240395636 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8e98004420 0x7f8e98004850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:38.382 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.364+0000 7f8e9ffff700 1 --2- 172.21.15.42:0/1240395636 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8e98004e40 0x7f8e9800b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:38.382 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.364+0000 7f8e9ffff700 1 -- 172.21.15.42:0/1240395636 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8eb8118e70 con 0x7f8e980038a0 2019-12-05T05:03:38.382 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.364+0000 7f8e9ffff700 1 -- 172.21.15.42:0/1240395636 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8eb8118930 con 0x7f8e98004420 2019-12-05T05:03:38.382 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.364+0000 7f8e9ffff700 1 -- 172.21.15.42:0/1240395636 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e9800bc70 con 0x7f8e98004e40 2019-12-05T05:03:38.382 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.364+0000 7f8eb77fe700 1 --2- 172.21.15.42:0/1240395636 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8eb8119fd0 0x7f8eb807dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:03:38.383 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.364+0000 7f8eb6ffd700 1 --2- 172.21.15.42:0/1240395636 >> v2:172.21.15.40:3300/0 conn(0x7f8eb8117850 0x7f8eb808e7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:03:38.383 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.364+0000 7f8eb6ffd700 1 --2- 172.21.15.42:0/1240395636 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8e98004420 0x7f8e98004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:38.383 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.364+0000 7f8eb77fe700 1 --2- 172.21.15.42:0/1240395636 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e980038a0 0x7f8e98003d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:38.383 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.364+0000 7f8eb7fff700 1 --2- 172.21.15.42:0/1240395636 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8e98004e40 0x7f8e9800b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:38.384 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.364+0000 7f8eb6ffd700 1 -- 172.21.15.42:0/1240395636 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8e98004e40 msgr2=0x7f8e9800b670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:38.384 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.364+0000 7f8eb6ffd700 1 --2- 172.21.15.42:0/1240395636 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8e98004e40 0x7f8e9800b670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:38.385 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.364+0000 7f8eb6ffd700 1 -- 172.21.15.42:0/1240395636 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e980038a0 msgr2=0x7f8e98003d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:38.385 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.364+0000 7f8eb6ffd700 1 --2- 172.21.15.42:0/1240395636 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e980038a0 0x7f8e98003d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:38.385 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.364+0000 7f8eb6ffd700 1 -- 172.21.15.42:0/1240395636 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8eb81184e0 con 0x7f8e98004420 2019-12-05T05:03:38.385 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.364+0000 7f8ebd4a7700 1 -- 172.21.15.42:0/1240395636 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8eb80aa690 con 0x7f8e98004420 2019-12-05T05:03:38.385 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.364+0000 7f8eb6ffd700 1 --2- 172.21.15.42:0/1240395636 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8e98004420 0x7f8e98004850 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f8eac000b80 tx=0x7f8eac010f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:38.385 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.364+0000 7f8eb7fff700 1 --2- 172.21.15.42:0/1240395636 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8e98004e40 0x7f8e9800b670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T05:03:38.385 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.364+0000 7f8ebd4a7700 1 -- 172.21.15.42:0/1240395636 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8eb808f140 con 0x7f8e98004420 2019-12-05T05:03:38.386 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.364+0000 7f8eb6ffd700 1 --2- 172.21.15.42:0/1240395636 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8e98004420 0x7f8e98004850 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f8eac000b80 tx=0x7f8eac010f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:38.386 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.364+0000 7f8eb6ffd700 1 --2- 172.21.15.42:0/1240395636 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8e98004420 0x7f8e98004850 secure :-1 s=READY pgs=90 cs=0 l=1 rx=0x7f8eac000b80 tx=0x7f8eac010f70).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:38.386 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.364+0000 7f8eb6ffd700 1 --2- 172.21.15.42:0/1240395636 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8e98004420 0x7f8e98004850 secure :-1 s=THROTTLE_DONE pgs=90 cs=0 l=1 rx=0x7f8eac000b80 tx=0x7f8eac010f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:38.386 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.364+0000 7f8eb6ffd700 1 --2- 172.21.15.42:0/1240395636 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8e98004420 0x7f8e98004850 secure :-1 s=THROTTLE_DONE pgs=90 cs=0 l=1 rx=0x7f8eac000b80 tx=0x7f8eac010f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:38.386 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.364+0000 7f8e9ffff700 1 -- 172.21.15.42:0/1240395636 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8eac012e00 con 0x7f8e98004420 2019-12-05T05:03:38.386 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.364+0000 7f8eb6ffd700 1 --2- 172.21.15.42:0/1240395636 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8e98004420 0x7f8e98004850 secure :-1 s=THROTTLE_DONE pgs=90 cs=0 l=1 rx=0x7f8eac000b80 tx=0x7f8eac010f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:38.386 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.364+0000 7f8e9ffff700 1 -- 172.21.15.42:0/1240395636 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8eac012a20 con 0x7f8e98004420 2019-12-05T05:03:38.387 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.364+0000 7f8e9ffff700 1 -- 172.21.15.42:0/1240395636 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8eac00c900 con 0x7f8e98004420 2019-12-05T05:03:38.387 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.364+0000 7f8eb6ffd700 1 --2- 172.21.15.42:0/1240395636 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8e98004420 0x7f8e98004850 secure :-1 s=THROTTLE_DONE pgs=90 cs=0 l=1 rx=0x7f8eac000b80 tx=0x7f8eac010f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:38.387 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.364+0000 7f8e9ffff700 1 -- 172.21.15.42:0/1240395636 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f8eac00cb00 con 0x7f8e98004420 2019-12-05T05:03:38.387 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.364+0000 7f8eb6ffd700 1 --2- 172.21.15.42:0/1240395636 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8e98004420 0x7f8e98004850 secure :-1 s=THROTTLE_DONE pgs=90 cs=0 l=1 rx=0x7f8eac000b80 tx=0x7f8eac010f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:38.387 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.368+0000 7f8e9ffff700 1 --2- 172.21.15.42:0/1240395636 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8e98043b90 0x7f8e980460a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:38.387 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.368+0000 7f8e9ffff700 1 -- 172.21.15.42:0/1240395636 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f8eac04b480 con 0x7f8e98004420 2019-12-05T05:03:38.387 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.368+0000 7f8eb77fe700 1 --2- 172.21.15.42:0/1240395636 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8e98043b90 0x7f8e980460a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:38.388 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.368+0000 7f8eb77fe700 1 --2- 172.21.15.42:0/1240395636 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8e98043b90 0x7f8e980460a0 crc :-1 s=READY pgs=57 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:38.388 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.368+0000 7f8ebd4a7700 1 -- 172.21.15.42:0/1240395636 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e90003200 con 0x7f8e98004420 2019-12-05T05:03:38.388 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.372+0000 7f8eb6ffd700 1 --2- 172.21.15.42:0/1240395636 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8e98004420 0x7f8e98004850 secure :-1 s=THROTTLE_DONE pgs=90 cs=0 l=1 rx=0x7f8eac000b80 tx=0x7f8eac010f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:38.389 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.372+0000 7f8e9ffff700 1 -- 172.21.15.42:0/1240395636 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f8eac032540 con 0x7f8e98004420 2019-12-05T05:03:38.542 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.536+0000 7f8ebd4a7700 1 -- 172.21.15.42:0/1240395636 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f8e90003650 con 0x7f8e98004420 2019-12-05T05:03:38.542 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.536+0000 7f8eb6ffd700 1 --2- 172.21.15.42:0/1240395636 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8e98004420 0x7f8e98004850 secure :-1 s=THROTTLE_DONE pgs=90 cs=0 l=1 rx=0x7f8eac000b80 tx=0x7f8eac010f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:38.543 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.536+0000 7f8e9ffff700 1 -- 172.21.15.42:0/1240395636 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v24) v1 ==== 92+0+10 (secure 0 0 0) 0x7f8eac0327a0 con 0x7f8e98004420 2019-12-05T05:03:38.543 INFO:teuthology.orchestra.run.smithi042.stdout:pg_num: 8 2019-12-05T05:03:38.545 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.540+0000 7f8ebd4a7700 1 -- 172.21.15.42:0/1240395636 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8e98043b90 msgr2=0x7f8e980460a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:38.545 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.540+0000 7f8ebd4a7700 1 --2- 172.21.15.42:0/1240395636 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8e98043b90 0x7f8e980460a0 crc :-1 s=READY pgs=57 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:38.545 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.540+0000 7f8ebd4a7700 1 -- 172.21.15.42:0/1240395636 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8e98004420 msgr2=0x7f8e98004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:38.545 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.540+0000 7f8eb7fff700 1 -- 172.21.15.42:0/1240395636 reap_dead start 2019-12-05T05:03:38.546 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.540+0000 7f8ebd4a7700 1 --2- 172.21.15.42:0/1240395636 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8e98004420 0x7f8e98004850 secure :-1 s=READY pgs=90 cs=0 l=1 rx=0x7f8eac000b80 tx=0x7f8eac010f70).stop 2019-12-05T05:03:38.546 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.540+0000 7f8ebd4a7700 1 -- 172.21.15.42:0/1240395636 shutdown_connections 2019-12-05T05:03:38.546 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.540+0000 7f8ebd4a7700 1 --2- 172.21.15.42:0/1240395636 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8e98004420 0x7f8e98004850 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:38.546 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.540+0000 7f8ebd4a7700 1 -- 172.21.15.42:0/1240395636 shutdown_connections 2019-12-05T05:03:38.546 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.540+0000 7f8ebd4a7700 1 -- 172.21.15.42:0/1240395636 wait complete. 2019-12-05T05:03:38.546 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.540+0000 7f8ebd4a7700 1 -- 172.21.15.42:0/1240395636 >> 172.21.15.42:0/1240395636 conn(0x7f8eb80a8b30 msgr2=0x7f8eb80a5070 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:38.554 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2019-12-05T05:03:38.660 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.648+0000 7f3f99f1c700 1 Processor -- start 2019-12-05T05:03:38.660 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.648+0000 7f3f99f1c700 1 -- start start 2019-12-05T05:03:38.660 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.648+0000 7f3f99f1c700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3f9409f550 0x7f3f94144630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:38.660 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.648+0000 7f3f99f1c700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f3f940a9a00 0x7f3f9407e880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:38.660 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.648+0000 7f3f99f1c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3f9407edd0 0x7f3f94147b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:38.660 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.648+0000 7f3f99f1c700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f940a8f80 con 0x7f3f940a9a00 2019-12-05T05:03:38.661 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.648+0000 7f3f99f1c700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f940a9220 con 0x7f3f9407edd0 2019-12-05T05:03:38.661 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.648+0000 7f3f99f1c700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f940a94c0 con 0x7f3f9409f550 2019-12-05T05:03:38.661 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.648+0000 7f3f98cba700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3f9407edd0 0x7f3f94147b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:38.661 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.648+0000 7f3f98cba700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3f9407edd0 0x7f3f94147b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:53226/0 (socket says 172.21.15.42:53226) 2019-12-05T05:03:38.661 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.648+0000 7f3f8bfff700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f3f940a9a00 0x7f3f9407e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:38.661 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.648+0000 7f3f93fff700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3f9409f550 0x7f3f94144630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:38.661 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.648+0000 7f3f98cba700 1 -- 172.21.15.42:0/2490603654 learned_addr learned my addr 172.21.15.42:0/2490603654 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:03:38.662 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.648+0000 7f3f98cba700 1 -- 172.21.15.42:0/2490603654 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3f9409f550 msgr2=0x7f3f94144630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:03:38.662 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.648+0000 7f3f98cba700 1 --2- 172.21.15.42:0/2490603654 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3f9409f550 0x7f3f94144630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:03:38.662 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.648+0000 7f3f98cba700 1 -- 172.21.15.42:0/2490603654 >> v2:172.21.15.40:3300/0 conn(0x7f3f940a9a00 msgr2=0x7f3f9407e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:03:38.662 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.648+0000 7f3f98cba700 1 --2- 172.21.15.42:0/2490603654 >> v2:172.21.15.40:3300/0 conn(0x7f3f940a9a00 0x7f3f9407e880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:03:38.662 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.648+0000 7f3f98cba700 1 -- 172.21.15.42:0/2490603654 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f940a1000 con 0x7f3f9407edd0 2019-12-05T05:03:38.662 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.648+0000 7f3f98cba700 1 --2- 172.21.15.42:0/2490603654 >> v2:172.21.15.42:3300/0 conn(0x7f3f9407edd0 0x7f3f94147b70 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f3f940a93e0 tx=0x7f3f8c010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:38.662 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.648+0000 7f3f98cba700 1 --2- 172.21.15.42:0/2490603654 >> v2:172.21.15.42:3300/0 conn(0x7f3f9407edd0 0x7f3f94147b70 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f3f940a93e0 tx=0x7f3f8c010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:38.663 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.648+0000 7f3f98cba700 1 --2- 172.21.15.42:0/2490603654 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3f9407edd0 0x7f3f94147b70 secure :-1 s=READY pgs=91 cs=0 l=1 rx=0x7f3f940a93e0 tx=0x7f3f8c010130).ready entity=mon.1 client_cookie=5b1ec40f953a40b1 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:38.663 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.648+0000 7f3f98cba700 1 --2- 172.21.15.42:0/2490603654 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3f9407edd0 0x7f3f94147b70 secure :-1 s=THROTTLE_DONE pgs=91 cs=0 l=1 rx=0x7f3f940a93e0 tx=0x7f3f8c010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:38.663 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.648+0000 7f3f937fe700 1 -- 172.21.15.42:0/2490603654 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3f8c00a160 con 0x7f3f9407edd0 2019-12-05T05:03:38.663 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.648+0000 7f3f98cba700 1 --2- 172.21.15.42:0/2490603654 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3f9407edd0 0x7f3f94147b70 secure :-1 s=THROTTLE_DONE pgs=91 cs=0 l=1 rx=0x7f3f940a93e0 tx=0x7f3f8c010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:38.663 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.648+0000 7f3f98cba700 1 --2- 172.21.15.42:0/2490603654 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3f9407edd0 0x7f3f94147b70 secure :-1 s=THROTTLE_DONE pgs=91 cs=0 l=1 rx=0x7f3f940a93e0 tx=0x7f3f8c010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:38.663 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.648+0000 7f3f937fe700 1 -- 172.21.15.42:0/2490603654 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f8c011a70 con 0x7f3f9407edd0 2019-12-05T05:03:38.663 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.648+0000 7f3f937fe700 1 -- 172.21.15.42:0/2490603654 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3f8c00a740 con 0x7f3f9407edd0 2019-12-05T05:03:38.664 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.648+0000 7f3f99f1c700 1 -- 172.21.15.42:0/2490603654 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3f9407edd0 msgr2=0x7f3f94147b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:38.664 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.648+0000 7f3f99f1c700 1 --2- 172.21.15.42:0/2490603654 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3f9407edd0 0x7f3f94147b70 secure :-1 s=READY pgs=91 cs=0 l=1 rx=0x7f3f940a93e0 tx=0x7f3f8c010130).stop 2019-12-05T05:03:38.664 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.652+0000 7f3f99f1c700 1 -- 172.21.15.42:0/2490603654 shutdown_connections 2019-12-05T05:03:38.664 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.652+0000 7f3f99f1c700 1 --2- 172.21.15.42:0/2490603654 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3f9407edd0 0x7f3f94147b70 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:38.664 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.652+0000 7f3f99f1c700 1 --2- 172.21.15.42:0/2490603654 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3f9409f550 0x7f3f94144630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:03:38.664 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.652+0000 7f3f99f1c700 1 --2- 172.21.15.42:0/2490603654 >> v2:172.21.15.40:3300/0 conn(0x7f3f940a9a00 0x7f3f9407e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:03:38.664 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.652+0000 7f3f99f1c700 1 -- 172.21.15.42:0/2490603654 shutdown_connections 2019-12-05T05:03:38.664 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.652+0000 7f3f99f1c700 1 -- 172.21.15.42:0/2490603654 wait complete. 2019-12-05T05:03:38.665 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.652+0000 7f3f99f1c700 1 -- 172.21.15.42:0/2490603654 >> 172.21.15.42:0/2490603654 conn(0x7f3f940a9ef0 msgr2=0x7f3f940aa320 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:38.665 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.652+0000 7f3f99f1c700 1 Processor -- start 2019-12-05T05:03:38.665 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.656+0000 7f3f99f1c700 1 -- start start 2019-12-05T05:03:38.669 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.656+0000 7f3f99f1c700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3f941457e0 0x7f3f941353e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:38.670 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.656+0000 7f3f99f1c700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f94145c10 con 0x7f3f94120270 2019-12-05T05:03:38.670 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.656+0000 7f3f99f1c700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f9408e3e0 con 0x7f3f9408fe70 2019-12-05T05:03:38.670 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.656+0000 7f3f99f1c700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f94144470 con 0x7f3f941457e0 2019-12-05T05:03:38.670 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.656+0000 7f3f93fff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f3f9408fe70 0x7f3f9407dca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:53266/0 (socket says 172.21.15.42:53266) 2019-12-05T05:03:38.670 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.656+0000 7f3f93fff700 1 -- 172.21.15.42:0/373621560 learned_addr learned my addr 172.21.15.42:0/373621560 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:03:38.670 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.656+0000 7f3f98cba700 1 --2- 172.21.15.42:0/373621560 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3f941457e0 0x7f3f941353e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:38.671 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.656+0000 7f3f917fa700 1 -- 172.21.15.42:0/373621560 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1094074955 0 0) 0x7f3f9408e3e0 con 0x7f3f9408fe70 2019-12-05T05:03:38.671 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.656+0000 7f3f917fa700 1 -- 172.21.15.42:0/373621560 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f780030b0 con 0x7f3f9408fe70 2019-12-05T05:03:38.671 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.656+0000 7f3f917fa700 1 -- 172.21.15.42:0/373621560 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 208933806 0 0) 0x7f3f94145c10 con 0x7f3f94120270 2019-12-05T05:03:38.671 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.656+0000 7f3f917fa700 1 -- 172.21.15.42:0/373621560 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f9408e3e0 con 0x7f3f94120270 2019-12-05T05:03:38.671 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.656+0000 7f3f98cba700 1 -- 172.21.15.42:0/373621560 >> v1:172.21.15.42:6789/0 conn(0x7f3f9408fe70 legacy=0x7f3f9407dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:38.671 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.656+0000 7f3f98cba700 1 -- 172.21.15.42:0/373621560 >> v1:172.21.15.40:6789/0 conn(0x7f3f94120270 legacy=0x7f3f94143c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:38.671 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.656+0000 7f3f98cba700 1 -- 172.21.15.42:0/373621560 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f94137950 con 0x7f3f941457e0 2019-12-05T05:03:38.671 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.656+0000 7f3f98cba700 1 --2- 172.21.15.42:0/373621560 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3f941457e0 0x7f3f941353e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f3f940a93e0 tx=0x7f3f8c011ca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:38.672 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.656+0000 7f3f98cba700 1 --2- 172.21.15.42:0/373621560 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3f941457e0 0x7f3f941353e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f3f940a93e0 tx=0x7f3f8c011ca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:38.672 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.656+0000 7f3f98cba700 1 --2- 172.21.15.42:0/373621560 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3f941457e0 0x7f3f941353e0 secure :-1 s=READY pgs=30 cs=0 l=1 rx=0x7f3f940a93e0 tx=0x7f3f8c011ca0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:38.672 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.656+0000 7f3f98cba700 1 --2- 172.21.15.42:0/373621560 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3f941457e0 0x7f3f941353e0 secure :-1 s=THROTTLE_DONE pgs=30 cs=0 l=1 rx=0x7f3f940a93e0 tx=0x7f3f8c011ca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:38.672 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.656+0000 7f3f917fa700 1 -- 172.21.15.42:0/373621560 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3f8c0058c0 con 0x7f3f941457e0 2019-12-05T05:03:38.672 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.656+0000 7f3f99f1c700 1 -- 172.21.15.42:0/373621560 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f94135930 con 0x7f3f941457e0 2019-12-05T05:03:38.672 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.656+0000 7f3f98cba700 1 --2- 172.21.15.42:0/373621560 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3f941457e0 0x7f3f941353e0 secure :-1 s=THROTTLE_DONE pgs=30 cs=0 l=1 rx=0x7f3f940a93e0 tx=0x7f3f8c011ca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:38.673 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.656+0000 7f3f917fa700 1 -- 172.21.15.42:0/373621560 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f8c00a360 con 0x7f3f941457e0 2019-12-05T05:03:38.673 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.656+0000 7f3f98cba700 1 --2- 172.21.15.42:0/373621560 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3f941457e0 0x7f3f941353e0 secure :-1 s=THROTTLE_DONE pgs=30 cs=0 l=1 rx=0x7f3f940a93e0 tx=0x7f3f8c011ca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:38.673 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.656+0000 7f3f99f1c700 1 -- 172.21.15.42:0/373621560 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f94135e80 con 0x7f3f941457e0 2019-12-05T05:03:38.673 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.656+0000 7f3f917fa700 1 -- 172.21.15.42:0/373621560 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3f8c007150 con 0x7f3f941457e0 2019-12-05T05:03:38.673 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.660+0000 7f3f98cba700 1 --2- 172.21.15.42:0/373621560 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3f941457e0 0x7f3f941353e0 secure :-1 s=THROTTLE_DONE pgs=30 cs=0 l=1 rx=0x7f3f940a93e0 tx=0x7f3f8c011ca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:38.673 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.660+0000 7f3f917fa700 1 -- 172.21.15.42:0/373621560 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f3f8c0073f0 con 0x7f3f941457e0 2019-12-05T05:03:38.673 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.660+0000 7f3f98cba700 1 --2- 172.21.15.42:0/373621560 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3f941457e0 0x7f3f941353e0 secure :-1 s=THROTTLE_DONE pgs=30 cs=0 l=1 rx=0x7f3f940a93e0 tx=0x7f3f8c011ca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:38.674 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.660+0000 7f3f917fa700 1 --2- 172.21.15.42:0/373621560 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3f7803b410 0x7f3f7803d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:38.674 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.660+0000 7f3f917fa700 1 -- 172.21.15.42:0/373621560 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f3f8c08fa30 con 0x7f3f941457e0 2019-12-05T05:03:38.674 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.660+0000 7f3f93fff700 1 --2- 172.21.15.42:0/373621560 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3f7803b410 0x7f3f7803d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:38.674 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.660+0000 7f3f93fff700 1 --2- 172.21.15.42:0/373621560 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3f7803b410 0x7f3f7803d920 crc :-1 s=READY pgs=58 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:38.674 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.660+0000 7f3f99f1c700 1 -- 172.21.15.42:0/373621560 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f9405d110 con 0x7f3f941457e0 2019-12-05T05:03:38.678 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.672+0000 7f3f98cba700 1 --2- 172.21.15.42:0/373621560 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3f941457e0 0x7f3f941353e0 secure :-1 s=THROTTLE_DONE pgs=30 cs=0 l=1 rx=0x7f3f940a93e0 tx=0x7f3f8c011ca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:38.678 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.672+0000 7f3f917fa700 1 -- 172.21.15.42:0/373621560 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f3f8c007700 con 0x7f3f941457e0 2019-12-05T05:03:38.840 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.836+0000 7f3f99f1c700 1 -- 172.21.15.42:0/373621560 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7f3f9405b8c0 con 0x7f3f941457e0 2019-12-05T05:03:38.841 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.836+0000 7f3f98cba700 1 --2- 172.21.15.42:0/373621560 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3f941457e0 0x7f3f941353e0 secure :-1 s=THROTTLE_DONE pgs=30 cs=0 l=1 rx=0x7f3f940a93e0 tx=0x7f3f8c011ca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:38.841 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.836+0000 7f3f917fa700 1 -- 172.21.15.42:0/373621560 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v24) v1 ==== 104+0+10 (secure 0 0 0) 0x7f3f8c076cf0 con 0x7f3f941457e0 2019-12-05T05:03:38.843 INFO:teuthology.orchestra.run.smithi042.stdout:pg_num: 8 2019-12-05T05:03:38.854 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.840+0000 7f3f99f1c700 1 -- 172.21.15.42:0/373621560 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3f7803b410 msgr2=0x7f3f7803d920 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:38.854 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.840+0000 7f3f99f1c700 1 --2- 172.21.15.42:0/373621560 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3f7803b410 0x7f3f7803d920 crc :-1 s=READY pgs=58 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:38.854 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.840+0000 7f3f99f1c700 1 -- 172.21.15.42:0/373621560 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3f941457e0 msgr2=0x7f3f941353e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:38.854 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.840+0000 7f3f99f1c700 1 --2- 172.21.15.42:0/373621560 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3f941457e0 0x7f3f941353e0 secure :-1 s=READY pgs=30 cs=0 l=1 rx=0x7f3f940a93e0 tx=0x7f3f8c011ca0).stop 2019-12-05T05:03:38.854 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.840+0000 7f3f99f1c700 1 -- 172.21.15.42:0/373621560 shutdown_connections 2019-12-05T05:03:38.855 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.840+0000 7f3f99f1c700 1 --2- 172.21.15.42:0/373621560 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3f7803b410 0x7f3f7803d920 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:38.855 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.840+0000 7f3f99f1c700 1 --2- 172.21.15.42:0/373621560 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3f941457e0 0x7f3f941353e0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:38.855 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.840+0000 7f3f99f1c700 1 -- 172.21.15.42:0/373621560 shutdown_connections 2019-12-05T05:03:38.855 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.840+0000 7f3f99f1c700 1 -- 172.21.15.42:0/373621560 wait complete. 2019-12-05T05:03:38.855 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.840+0000 7f3f99f1c700 1 -- 172.21.15.42:0/373621560 >> 172.21.15.42:0/373621560 conn(0x7f3f940a6420 msgr2=0x7f3f940a4d20 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:38.856 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2019-12-05T05:03:38.968 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.956+0000 7fae00cff700 1 Processor -- start 2019-12-05T05:03:38.968 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.956+0000 7fae00cff700 1 -- start start 2019-12-05T05:03:38.968 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.956+0000 7fae00cff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fadfc09f550 0x7fadfc144630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:38.968 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.956+0000 7fae00cff700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fadfc0a9a00 0x7fadfc07e880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:38.968 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.956+0000 7fae00cff700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadfc0a1000 con 0x7fadfc07edd0 2019-12-05T05:03:38.969 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.956+0000 7fae00cff700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fadfc0a8f80 con 0x7fadfc0a9a00 2019-12-05T05:03:38.969 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.956+0000 7fae00cff700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fadfc0a9220 con 0x7fadfc09f550 2019-12-05T05:03:38.969 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.956+0000 7fadfad9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fadfc09f550 0x7fadfc144630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:38.969 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.960+0000 7fadfa59c700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fadfc0a9a00 0x7fadfc07e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:38.969 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.960+0000 7fadfad9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fadfc09f550 0x7fadfc144630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:53236/0 (socket says 172.21.15.42:53236) 2019-12-05T05:03:38.969 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.960+0000 7fadfad9d700 1 -- 172.21.15.42:0/403838468 learned_addr learned my addr 172.21.15.42:0/403838468 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:03:38.969 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.960+0000 7fadfad9d700 1 -- 172.21.15.42:0/403838468 >> v2:172.21.15.40:3300/0 conn(0x7fadfc0a9a00 msgr2=0x7fadfc07e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:03:38.970 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.960+0000 7fadfad9d700 1 --2- 172.21.15.42:0/403838468 >> v2:172.21.15.40:3300/0 conn(0x7fadfc0a9a00 0x7fadfc07e880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:03:38.970 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.960+0000 7fadfad9d700 1 -- 172.21.15.42:0/403838468 >> v1:172.21.15.40:6789/0 conn(0x7fadfc07edd0 legacy=0x7fadfc147b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:38.970 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.960+0000 7fadfad9d700 1 -- 172.21.15.42:0/403838468 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadfc144b80 con 0x7fadfc09f550 2019-12-05T05:03:38.970 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.960+0000 7fadfad9d700 1 --2- 172.21.15.42:0/403838468 >> v2:172.21.15.42:3300/0 conn(0x7fadfc09f550 0x7fadfc144630 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fadec002f30 tx=0x7fadec00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:38.970 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.960+0000 7fadfad9d700 1 --2- 172.21.15.42:0/403838468 >> v2:172.21.15.42:3300/0 conn(0x7fadfc09f550 0x7fadfc144630 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fadec002f30 tx=0x7fadec00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:38.970 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.960+0000 7fadfad9d700 1 --2- 172.21.15.42:0/403838468 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fadfc09f550 0x7fadfc144630 secure :-1 s=READY pgs=93 cs=0 l=1 rx=0x7fadec002f30 tx=0x7fadec00fd30).ready entity=mon.1 client_cookie=c8d957b746b95e68 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:38.970 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.960+0000 7fadfad9d700 1 --2- 172.21.15.42:0/403838468 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fadfc09f550 0x7fadfc144630 secure :-1 s=THROTTLE_DONE pgs=93 cs=0 l=1 rx=0x7fadec002f30 tx=0x7fadec00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:38.971 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.960+0000 7fadfad9d700 1 --2- 172.21.15.42:0/403838468 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fadfc09f550 0x7fadfc144630 secure :-1 s=THROTTLE_DONE pgs=93 cs=0 l=1 rx=0x7fadec002f30 tx=0x7fadec00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:38.971 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.960+0000 7fadf9d9b700 1 -- 172.21.15.42:0/403838468 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fadec011370 con 0x7fadfc09f550 2019-12-05T05:03:38.971 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.960+0000 7fadfad9d700 1 --2- 172.21.15.42:0/403838468 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fadfc09f550 0x7fadfc144630 secure :-1 s=THROTTLE_DONE pgs=93 cs=0 l=1 rx=0x7fadec002f30 tx=0x7fadec00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:38.971 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.960+0000 7fadf9d9b700 1 -- 172.21.15.42:0/403838468 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadec008bb0 con 0x7fadfc09f550 2019-12-05T05:03:38.971 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.960+0000 7fadf9d9b700 1 -- 172.21.15.42:0/403838468 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fadec011950 con 0x7fadfc09f550 2019-12-05T05:03:38.971 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.960+0000 7fae00cff700 1 -- 172.21.15.42:0/403838468 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fadfc09f550 msgr2=0x7fadfc144630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:38.971 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.960+0000 7fae00cff700 1 --2- 172.21.15.42:0/403838468 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fadfc09f550 0x7fadfc144630 secure :-1 s=READY pgs=93 cs=0 l=1 rx=0x7fadec002f30 tx=0x7fadec00fd30).stop 2019-12-05T05:03:38.972 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.960+0000 7fae00cff700 1 -- 172.21.15.42:0/403838468 shutdown_connections 2019-12-05T05:03:38.972 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.960+0000 7fae00cff700 1 --2- 172.21.15.42:0/403838468 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fadfc09f550 0x7fadfc144630 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:38.972 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.960+0000 7fae00cff700 1 --2- 172.21.15.42:0/403838468 >> v2:172.21.15.40:3300/0 conn(0x7fadfc0a9a00 0x7fadfc07e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:03:38.972 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.960+0000 7fae00cff700 1 -- 172.21.15.42:0/403838468 shutdown_connections 2019-12-05T05:03:38.972 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.960+0000 7fae00cff700 1 -- 172.21.15.42:0/403838468 wait complete. 2019-12-05T05:03:38.972 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.960+0000 7fae00cff700 1 -- 172.21.15.42:0/403838468 >> 172.21.15.42:0/403838468 conn(0x7fadfc0a9ef0 msgr2=0x7fadfc0aa320 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:38.976 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.964+0000 7fae00cff700 1 Processor -- start 2019-12-05T05:03:38.976 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.964+0000 7fae00cff700 1 -- start start 2019-12-05T05:03:38.976 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.964+0000 7fae00cff700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fadfc143a70 0x7fadfc120f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:38.976 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.964+0000 7fae00cff700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fadfc08f8f0 0x7fadfc130600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:38.976 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.964+0000 7fae00cff700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadfc121490 con 0x7fadfc08fd20 2019-12-05T05:03:38.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.964+0000 7fae00cff700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fadfc143290 con 0x7fadfc143a70 2019-12-05T05:03:38.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.964+0000 7fae00cff700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadfc143530 con 0x7fadfc08f8f0 2019-12-05T05:03:38.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.964+0000 7fadfb59e700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fadfc08fd20 0x7fadfc08dd20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:53282/0 (socket says 172.21.15.42:53282) 2019-12-05T05:03:38.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.964+0000 7fadfb59e700 1 -- 172.21.15.42:0/941996637 learned_addr learned my addr 172.21.15.42:0/941996637 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:03:38.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.964+0000 7fadfad9d700 1 --2- 172.21.15.42:0/941996637 >> v2:172.21.15.40:3300/0 conn(0x7fadfc143a70 0x7fadfc120f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:38.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.964+0000 7fadfa59c700 1 --2- 172.21.15.42:0/941996637 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fadfc08f8f0 0x7fadfc130600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:38.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.964+0000 7fadeb7fe700 1 -- 172.21.15.42:0/941996637 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3713166269 0 0) 0x7fadfc121490 con 0x7fadfc08fd20 2019-12-05T05:03:38.978 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.964+0000 7fadeb7fe700 1 -- 172.21.15.42:0/941996637 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fade00030b0 con 0x7fadfc08fd20 2019-12-05T05:03:38.978 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.964+0000 7fadeb7fe700 1 -- 172.21.15.42:0/941996637 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 172797054 0 0) 0x7fade00030b0 con 0x7fadfc08fd20 2019-12-05T05:03:38.978 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.964+0000 7fadfad9d700 1 -- 172.21.15.42:0/941996637 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fadfc08f8f0 msgr2=0x7fadfc130600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:38.978 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.964+0000 7fadfad9d700 1 --2- 172.21.15.42:0/941996637 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fadfc08f8f0 0x7fadfc130600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:38.978 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.964+0000 7fadfad9d700 1 -- 172.21.15.42:0/941996637 >> v1:172.21.15.42:6789/0 conn(0x7fadfc08fd20 legacy=0x7fadfc08dd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:38.979 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.964+0000 7fadfad9d700 1 -- 172.21.15.42:0/941996637 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadfc142f10 con 0x7fadfc143a70 2019-12-05T05:03:38.979 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.964+0000 7fadfad9d700 1 --2- 172.21.15.42:0/941996637 >> v2:172.21.15.40:3300/0 conn(0x7fadfc143a70 0x7fadfc120f40 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fadf4001ee0 tx=0x7fadf4001f20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:38.979 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.964+0000 7fadfad9d700 1 --2- 172.21.15.42:0/941996637 >> v2:172.21.15.40:3300/0 conn(0x7fadfc143a70 0x7fadfc120f40 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fadf4001ee0 tx=0x7fadf4001f20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:38.979 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.964+0000 7fadfad9d700 1 --2- 172.21.15.42:0/941996637 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fadfc143a70 0x7fadfc120f40 secure :-1 s=READY pgs=87 cs=0 l=1 rx=0x7fadf4001ee0 tx=0x7fadf4001f20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:38.979 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.964+0000 7fadfad9d700 1 --2- 172.21.15.42:0/941996637 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fadfc143a70 0x7fadfc120f40 secure :-1 s=THROTTLE_DONE pgs=87 cs=0 l=1 rx=0x7fadf4001ee0 tx=0x7fadf4001f20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:38.979 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.964+0000 7fadeb7fe700 1 -- 172.21.15.42:0/941996637 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fadf4001ca0 con 0x7fadfc143a70 2019-12-05T05:03:38.979 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.964+0000 7fadfad9d700 1 --2- 172.21.15.42:0/941996637 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fadfc143a70 0x7fadfc120f40 secure :-1 s=THROTTLE_DONE pgs=87 cs=0 l=1 rx=0x7fadf4001ee0 tx=0x7fadf4001f20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:38.980 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.964+0000 7fadfad9d700 1 --2- 172.21.15.42:0/941996637 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fadfc143a70 0x7fadfc120f40 secure :-1 s=THROTTLE_DONE pgs=87 cs=0 l=1 rx=0x7fadf4001ee0 tx=0x7fadf4001f20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:38.980 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.968+0000 7fadeb7fe700 1 -- 172.21.15.42:0/941996637 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadf4001ab0 con 0x7fadfc143a70 2019-12-05T05:03:38.980 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.968+0000 7fae00cff700 1 -- 172.21.15.42:0/941996637 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadfc130b50 con 0x7fadfc143a70 2019-12-05T05:03:38.980 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.968+0000 7fadeb7fe700 1 -- 172.21.15.42:0/941996637 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fadec00f360 con 0x7fadfc143a70 2019-12-05T05:03:38.980 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.968+0000 7fae00cff700 1 -- 172.21.15.42:0/941996637 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fadfc0a3c10 con 0x7fadfc143a70 2019-12-05T05:03:38.980 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.968+0000 7fadfad9d700 1 --2- 172.21.15.42:0/941996637 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fadfc143a70 0x7fadfc120f40 secure :-1 s=THROTTLE_DONE pgs=87 cs=0 l=1 rx=0x7fadf4001ee0 tx=0x7fadf4001f20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:38.981 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.968+0000 7fadeb7fe700 1 -- 172.21.15.42:0/941996637 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fadec00f5f0 con 0x7fadfc143a70 2019-12-05T05:03:38.981 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.968+0000 7fadfad9d700 1 --2- 172.21.15.42:0/941996637 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fadfc143a70 0x7fadfc120f40 secure :-1 s=THROTTLE_DONE pgs=87 cs=0 l=1 rx=0x7fadf4001ee0 tx=0x7fadf4001f20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:38.981 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.968+0000 7fadeb7fe700 1 --2- 172.21.15.42:0/941996637 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fade003b250 0x7fade003d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:38.981 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.968+0000 7fadeb7fe700 1 -- 172.21.15.42:0/941996637 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6031+0+0 (secure 0 0 0) 0x7fadec04acd0 con 0x7fadfc143a70 2019-12-05T05:03:38.981 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.968+0000 7fadfa59c700 1 --2- 172.21.15.42:0/941996637 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fade003b250 0x7fade003d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:38.981 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.968+0000 7fadfa59c700 1 --2- 172.21.15.42:0/941996637 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fade003b250 0x7fade003d760 crc :-1 s=READY pgs=59 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:38.982 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.972+0000 7fae00cff700 1 -- 172.21.15.42:0/941996637 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fadfc05d110 con 0x7fadfc143a70 2019-12-05T05:03:38.988 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.984+0000 7fadfad9d700 1 --2- 172.21.15.42:0/941996637 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fadfc143a70 0x7fadfc120f40 secure :-1 s=THROTTLE_DONE pgs=87 cs=0 l=1 rx=0x7fadf4001ee0 tx=0x7fadf4001f20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:38.989 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:38.984+0000 7fadeb7fe700 1 -- 172.21.15.42:0/941996637 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fadec032c00 con 0x7fadfc143a70 2019-12-05T05:03:39.151 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.144+0000 7fae00cff700 1 -- 172.21.15.42:0/941996637 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7fadfc05b8c0 con 0x7fadfc143a70 2019-12-05T05:03:39.152 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.148+0000 7fadfad9d700 1 --2- 172.21.15.42:0/941996637 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fadfc143a70 0x7fadfc120f40 secure :-1 s=THROTTLE_DONE pgs=87 cs=0 l=1 rx=0x7fadf4001ee0 tx=0x7fadf4001f20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:39.152 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.148+0000 7fadeb7fe700 1 -- 172.21.15.42:0/941996637 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v24) v1 ==== 100+0+10 (secure 0 0 0) 0x7fadec032310 con 0x7fadfc143a70 2019-12-05T05:03:39.152 INFO:teuthology.orchestra.run.smithi042.stdout:pg_num: 8 2019-12-05T05:03:39.155 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.148+0000 7fae00cff700 1 -- 172.21.15.42:0/941996637 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fade003b250 msgr2=0x7fade003d760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:39.155 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.148+0000 7fae00cff700 1 --2- 172.21.15.42:0/941996637 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fade003b250 0x7fade003d760 crc :-1 s=READY pgs=59 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:39.155 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.148+0000 7fae00cff700 1 -- 172.21.15.42:0/941996637 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fadfc143a70 msgr2=0x7fadfc120f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:39.155 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.148+0000 7fae00cff700 1 --2- 172.21.15.42:0/941996637 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fadfc143a70 0x7fadfc120f40 secure :-1 s=READY pgs=87 cs=0 l=1 rx=0x7fadf4001ee0 tx=0x7fadf4001f20).stop 2019-12-05T05:03:39.155 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.148+0000 7fae00cff700 1 -- 172.21.15.42:0/941996637 shutdown_connections 2019-12-05T05:03:39.156 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.148+0000 7fae00cff700 1 --2- 172.21.15.42:0/941996637 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fade003b250 0x7fade003d760 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:39.156 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.148+0000 7fae00cff700 1 --2- 172.21.15.42:0/941996637 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fadfc143a70 0x7fadfc120f40 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:39.156 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.148+0000 7fae00cff700 1 --2- 172.21.15.42:0/941996637 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fadfc08f8f0 0x7fadfc130600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:39.156 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.148+0000 7fae00cff700 1 -- 172.21.15.42:0/941996637 shutdown_connections 2019-12-05T05:03:39.156 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.148+0000 7fae00cff700 1 -- 172.21.15.42:0/941996637 wait complete. 2019-12-05T05:03:39.156 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.148+0000 7fae00cff700 1 -- 172.21.15.42:0/941996637 >> 172.21.15.42:0/941996637 conn(0x7fadfc05a900 msgr2=0x7fadfc059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:39.164 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2019-12-05T05:03:39.164 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2019-12-05T05:03:39.165 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2019-12-05T05:03:39.262 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.252+0000 7f553d45a700 1 Processor -- start 2019-12-05T05:03:39.262 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.252+0000 7f553d45a700 1 -- start start 2019-12-05T05:03:39.262 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.252+0000 7f553d45a700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f553809fae0 0x7f553814fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:39.263 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.252+0000 7f553d45a700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55380a0710 con 0x7f553808df80 2019-12-05T05:03:39.263 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.252+0000 7f553d45a700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f553807ef20 con 0x7f55380a7740 2019-12-05T05:03:39.263 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.252+0000 7f553d45a700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55380a0b00 con 0x7f553809fae0 2019-12-05T05:03:39.263 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.252+0000 7f55377fe700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f55380a7740 0x7f553807e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:53288/0 (socket says 172.21.15.42:53288) 2019-12-05T05:03:39.263 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.252+0000 7f55377fe700 1 -- 172.21.15.42:0/1086298188 learned_addr learned my addr 172.21.15.42:0/1086298188 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:03:39.263 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.252+0000 7f5537fff700 1 --2- 172.21.15.42:0/1086298188 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f553809fae0 0x7f553814fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:39.263 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.252+0000 7f55367fc700 1 -- 172.21.15.42:0/1086298188 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1231103846 0 0) 0x7f553807ef20 con 0x7f55380a7740 2019-12-05T05:03:39.264 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.252+0000 7f55367fc700 1 -- 172.21.15.42:0/1086298188 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55200030b0 con 0x7f55380a7740 2019-12-05T05:03:39.264 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.252+0000 7f55367fc700 1 -- 172.21.15.42:0/1086298188 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2269238814 0 0) 0x7f55380a0710 con 0x7f553808df80 2019-12-05T05:03:39.264 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.252+0000 7f55367fc700 1 -- 172.21.15.42:0/1086298188 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f553807ef20 con 0x7f553808df80 2019-12-05T05:03:39.264 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.252+0000 7f5537fff700 1 -- 172.21.15.42:0/1086298188 >> v1:172.21.15.42:6789/0 conn(0x7f55380a7740 legacy=0x7f553807e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:39.264 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.252+0000 7f5537fff700 1 -- 172.21.15.42:0/1086298188 >> v1:172.21.15.40:6789/0 conn(0x7f553808df80 legacy=0x7f553809f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:39.264 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.252+0000 7f5537fff700 1 -- 172.21.15.42:0/1086298188 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55380a9ba0 con 0x7f553809fae0 2019-12-05T05:03:39.265 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.252+0000 7f5537fff700 1 --2- 172.21.15.42:0/1086298188 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f553809fae0 0x7f553814fff0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f55300032b0 tx=0x7f553000ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:39.265 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.252+0000 7f5537fff700 1 --2- 172.21.15.42:0/1086298188 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f553809fae0 0x7f553814fff0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f55300032b0 tx=0x7f553000ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:39.265 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.252+0000 7f5537fff700 1 --2- 172.21.15.42:0/1086298188 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f553809fae0 0x7f553814fff0 secure :-1 s=READY pgs=31 cs=0 l=1 rx=0x7f55300032b0 tx=0x7f553000ff70).ready entity=mon.2 client_cookie=1fc6e37475e2bf6c server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:39.265 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.256+0000 7f5537fff700 1 --2- 172.21.15.42:0/1086298188 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f553809fae0 0x7f553814fff0 secure :-1 s=THROTTLE_DONE pgs=31 cs=0 l=1 rx=0x7f55300032b0 tx=0x7f553000ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:39.265 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.256+0000 7f55367fc700 1 -- 172.21.15.42:0/1086298188 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f55300109b0 con 0x7f553809fae0 2019-12-05T05:03:39.265 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.256+0000 7f5537fff700 1 --2- 172.21.15.42:0/1086298188 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f553809fae0 0x7f553814fff0 secure :-1 s=THROTTLE_DONE pgs=31 cs=0 l=1 rx=0x7f55300032b0 tx=0x7f553000ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:39.265 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.256+0000 7f5537fff700 1 --2- 172.21.15.42:0/1086298188 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f553809fae0 0x7f553814fff0 secure :-1 s=THROTTLE_DONE pgs=31 cs=0 l=1 rx=0x7f55300032b0 tx=0x7f553000ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:39.265 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.256+0000 7f55367fc700 1 -- 172.21.15.42:0/1086298188 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5530008b50 con 0x7f553809fae0 2019-12-05T05:03:39.266 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.256+0000 7f55367fc700 1 -- 172.21.15.42:0/1086298188 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f55300110c0 con 0x7f553809fae0 2019-12-05T05:03:39.266 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.256+0000 7f553d45a700 1 -- 172.21.15.42:0/1086298188 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f553809fae0 msgr2=0x7f553814fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:39.266 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.256+0000 7f553d45a700 1 --2- 172.21.15.42:0/1086298188 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f553809fae0 0x7f553814fff0 secure :-1 s=READY pgs=31 cs=0 l=1 rx=0x7f55300032b0 tx=0x7f553000ff70).stop 2019-12-05T05:03:39.266 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.256+0000 7f553d45a700 1 -- 172.21.15.42:0/1086298188 shutdown_connections 2019-12-05T05:03:39.266 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.256+0000 7f553d45a700 1 --2- 172.21.15.42:0/1086298188 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f553809fae0 0x7f553814fff0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:39.266 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.256+0000 7f553d45a700 1 -- 172.21.15.42:0/1086298188 shutdown_connections 2019-12-05T05:03:39.266 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.256+0000 7f553d45a700 1 -- 172.21.15.42:0/1086298188 wait complete. 2019-12-05T05:03:39.267 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.256+0000 7f553d45a700 1 -- 172.21.15.42:0/1086298188 >> 172.21.15.42:0/1086298188 conn(0x7f55380aabe0 msgr2=0x7f55380a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:39.270 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.260+0000 7f553d45a700 1 Processor -- start 2019-12-05T05:03:39.270 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.260+0000 7f553d45a700 1 -- start start 2019-12-05T05:03:39.270 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.260+0000 7f553d45a700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f553807a400 0x7f55380a25c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:39.270 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.260+0000 7f553d45a700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f55380a2b10 0x7f55381224f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:39.270 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.260+0000 7f553d45a700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f553807a830 con 0x7f5538144d20 2019-12-05T05:03:39.270 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.260+0000 7f553d45a700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55380a3110 con 0x7f55380a2b10 2019-12-05T05:03:39.271 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.260+0000 7f553d45a700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5538119e30 con 0x7f553807a400 2019-12-05T05:03:39.271 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.260+0000 7f5536ffd700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f553807a400 0x7f55380a25c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:39.271 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.260+0000 7f55377fe700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f5538144d20 0x7f5538119630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:34502/0 (socket says 172.21.15.42:34502) 2019-12-05T05:03:39.271 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.260+0000 7f55377fe700 1 -- 172.21.15.42:0/3935070799 learned_addr learned my addr 172.21.15.42:0/3935070799 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:03:39.271 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.260+0000 7f5537fff700 1 --2- 172.21.15.42:0/3935070799 >> v2:172.21.15.40:3300/0 conn(0x7f55380a2b10 0x7f55381224f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:39.271 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.260+0000 7f5536ffd700 1 -- 172.21.15.42:0/3935070799 >> v2:172.21.15.40:3300/0 conn(0x7f55380a2b10 msgr2=0x7f55381224f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:39.271 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.260+0000 7f551ffff700 1 -- 172.21.15.42:0/3935070799 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 187699338 0 0) 0x7f553807a830 con 0x7f5538144d20 2019-12-05T05:03:39.272 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.260+0000 7f5536ffd700 1 --2- 172.21.15.42:0/3935070799 >> v2:172.21.15.40:3300/0 conn(0x7f55380a2b10 0x7f55381224f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:39.272 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.260+0000 7f5536ffd700 1 -- 172.21.15.42:0/3935070799 >> v1:172.21.15.40:6789/0 conn(0x7f5538144d20 legacy=0x7f5538119630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:39.272 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.260+0000 7f5536ffd700 1 -- 172.21.15.42:0/3935070799 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5538122a40 con 0x7f553807a400 2019-12-05T05:03:39.272 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.260+0000 7f5536ffd700 1 --2- 172.21.15.42:0/3935070799 >> v2:172.21.15.42:3300/0 conn(0x7f553807a400 0x7f55380a25c0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f5538126af0 tx=0x7f552c0108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:39.272 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.260+0000 7f5536ffd700 1 --2- 172.21.15.42:0/3935070799 >> v2:172.21.15.42:3300/0 conn(0x7f553807a400 0x7f55380a25c0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f5538126af0 tx=0x7f552c0108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:39.272 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.260+0000 7f5536ffd700 1 --2- 172.21.15.42:0/3935070799 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f553807a400 0x7f55380a25c0 secure :-1 s=READY pgs=96 cs=0 l=1 rx=0x7f5538126af0 tx=0x7f552c0108f0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:39.272 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.260+0000 7f5536ffd700 1 --2- 172.21.15.42:0/3935070799 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f553807a400 0x7f55380a25c0 secure :-1 s=THROTTLE_DONE pgs=96 cs=0 l=1 rx=0x7f5538126af0 tx=0x7f552c0108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:39.273 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.260+0000 7f551ffff700 1 -- 172.21.15.42:0/3935070799 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f552c00ca80 con 0x7f553807a400 2019-12-05T05:03:39.273 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.260+0000 7f5536ffd700 1 --2- 172.21.15.42:0/3935070799 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f553807a400 0x7f55380a25c0 secure :-1 s=THROTTLE_DONE pgs=96 cs=0 l=1 rx=0x7f5538126af0 tx=0x7f552c0108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:39.273 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.260+0000 7f5536ffd700 1 --2- 172.21.15.42:0/3935070799 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f553807a400 0x7f55380a25c0 secure :-1 s=THROTTLE_DONE pgs=96 cs=0 l=1 rx=0x7f5538126af0 tx=0x7f552c0108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:39.273 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.260+0000 7f551ffff700 1 -- 172.21.15.42:0/3935070799 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f552c00c6a0 con 0x7f553807a400 2019-12-05T05:03:39.273 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.260+0000 7f553d45a700 1 -- 172.21.15.42:0/3935070799 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5538122d10 con 0x7f553807a400 2019-12-05T05:03:39.273 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.260+0000 7f551ffff700 1 -- 172.21.15.42:0/3935070799 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f552c00d680 con 0x7f553807a400 2019-12-05T05:03:39.273 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.260+0000 7f553d45a700 1 -- 172.21.15.42:0/3935070799 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f553812d120 con 0x7f553807a400 2019-12-05T05:03:39.274 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.264+0000 7f5536ffd700 1 --2- 172.21.15.42:0/3935070799 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f553807a400 0x7f55380a25c0 secure :-1 s=THROTTLE_DONE pgs=96 cs=0 l=1 rx=0x7f5538126af0 tx=0x7f552c0108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:39.274 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.264+0000 7f551ffff700 1 -- 172.21.15.42:0/3935070799 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f552c00d990 con 0x7f553807a400 2019-12-05T05:03:39.274 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.264+0000 7f5536ffd700 1 --2- 172.21.15.42:0/3935070799 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f553807a400 0x7f55380a25c0 secure :-1 s=THROTTLE_DONE pgs=96 cs=0 l=1 rx=0x7f5538126af0 tx=0x7f552c0108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:39.274 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.264+0000 7f551ffff700 1 --2- 172.21.15.42:0/3935070799 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f551803ac40 0x7f551803d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:39.274 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.264+0000 7f551ffff700 1 -- 172.21.15.42:0/3935070799 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f552c04b2e0 con 0x7f553807a400 2019-12-05T05:03:39.274 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.264+0000 7f55377fe700 1 --2- 172.21.15.42:0/3935070799 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f551803ac40 0x7f551803d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:39.275 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.264+0000 7f55377fe700 1 --2- 172.21.15.42:0/3935070799 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f551803ac40 0x7f551803d150 crc :-1 s=READY pgs=60 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:39.275 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.268+0000 7f553d45a700 1 -- 172.21.15.42:0/3935070799 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f553805d110 con 0x7f553807a400 2019-12-05T05:03:39.278 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.272+0000 7f5536ffd700 1 --2- 172.21.15.42:0/3935070799 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f553807a400 0x7f55380a25c0 secure :-1 s=THROTTLE_DONE pgs=96 cs=0 l=1 rx=0x7f5538126af0 tx=0x7f552c0108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:39.279 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.272+0000 7f551ffff700 1 -- 172.21.15.42:0/3935070799 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f552c032360 con 0x7f553807a400 2019-12-05T05:03:39.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.488+0000 7f553d45a700 1 -- 172.21.15.42:0/3935070799 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f553805b8c0 con 0x7f553807a400 2019-12-05T05:03:39.499 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.492+0000 7f5536ffd700 1 --2- 172.21.15.42:0/3935070799 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f553807a400 0x7f55380a25c0 secure :-1 s=THROTTLE_DONE pgs=96 cs=0 l=1 rx=0x7f5538126af0 tx=0x7f552c0108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:39.499 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.492+0000 7f551ffff700 1 -- 172.21.15.42:0/3935070799 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v3) v1 ==== 74+0+94024 (secure 0 0 0) 0x7f552c0325c0 con 0x7f553807a400 2019-12-05T05:03:39.502 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:03:39.504 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.500+0000 7f553d45a700 1 -- 172.21.15.42:0/3935070799 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f551803ac40 msgr2=0x7f551803d150 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:39.504 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.500+0000 7f553d45a700 1 --2- 172.21.15.42:0/3935070799 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f551803ac40 0x7f551803d150 crc :-1 s=READY pgs=60 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:39.504 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.500+0000 7f553d45a700 1 -- 172.21.15.42:0/3935070799 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f553807a400 msgr2=0x7f55380a25c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:39.505 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.500+0000 7f553d45a700 1 --2- 172.21.15.42:0/3935070799 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f553807a400 0x7f55380a25c0 secure :-1 s=READY pgs=96 cs=0 l=1 rx=0x7f5538126af0 tx=0x7f552c0108f0).stop 2019-12-05T05:03:39.505 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.500+0000 7f553d45a700 1 -- 172.21.15.42:0/3935070799 shutdown_connections 2019-12-05T05:03:39.505 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.500+0000 7f553d45a700 1 --2- 172.21.15.42:0/3935070799 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f551803ac40 0x7f551803d150 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:39.505 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.500+0000 7f553d45a700 1 --2- 172.21.15.42:0/3935070799 >> v2:172.21.15.40:3300/0 conn(0x7f55380a2b10 0x7f55381224f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:39.505 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.500+0000 7f553d45a700 1 --2- 172.21.15.42:0/3935070799 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f553807a400 0x7f55380a25c0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:39.505 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.500+0000 7f553d45a700 1 -- 172.21.15.42:0/3935070799 shutdown_connections 2019-12-05T05:03:39.505 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.500+0000 7f553d45a700 1 -- 172.21.15.42:0/3935070799 wait complete. 2019-12-05T05:03:39.506 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.500+0000 7f553d45a700 1 -- 172.21.15.42:0/3935070799 >> 172.21.15.42:0/3935070799 conn(0x7f553809f240 msgr2=0x7f5538059ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:39.516 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":3,"active_gid":4108,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6800","nonce":11930},{"type":"v1","addr":"172.21.15.42:6801","nonce":11930}]},"active_addr":"172.21.15.42:6801/11930","active_change":"2019-12-05T05:03:15.585660+0000","active_mgr_features":4540138292836696063,"available":true,"standbys":[{"gid":4101,"name":"x","mgr_features":4540138292836696063,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"str","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"ansible","can_run":true,"error_string":"","module_options":{"ca_bundle":{"name":"ca_bundle","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_location":{"name":"server_location","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_server":{"name":"verify_server","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"none","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"1","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"str","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"str","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"str","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_cephfs":{"name":"FEATURE_TOGGLE_cephfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_iscsi":{"name":"FEATURE_TOGGLE_iscsi","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_mirroring":{"name":"FEATURE_TOGGLE_mirroring","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rbd":{"name":"FEATURE_TOGGLE_rbd","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rgw":{"name":"FEATURE_TOGGLE_rgw","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"str","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"str","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"str","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"str","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"str","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"str","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"deepsea","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"salt_api_eauth":{"name":"salt_api_eauth","type":"str","level":"advanced","flags":0,"default_value":"sharedsecret","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"salt_api_password":{"name":"salt_api_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"salt_api_url":{"name":"salt_api_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"salt_api_username":{"name":"salt_api_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_cloud","can_run":true,"error_string":"","module_options":{"diskprediction_cert_context":{"name":"diskprediction_cert_context","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_default_authority":{"name":"diskprediction_default_authority","type":"str","level":"advanced","flags":0,"default_value":"localhost","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_password":{"name":"diskprediction_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_port":{"name":"diskprediction_port","type":"str","level":"advanced","flags":0,"default_value":"31400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_retrieve_prediction_interval":{"name":"diskprediction_retrieve_prediction_interval","type":"str","level":"advanced","flags":0,"default_value":"43200","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_server":{"name":"diskprediction_server","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_ssl_target_name_override":{"name":"diskprediction_ssl_target_name_override","type":"str","level":"advanced","flags":0,"default_value":"localhost","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_upload_metrics_interval":{"name":"diskprediction_upload_metrics_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_upload_smart_interval":{"name":"diskprediction_upload_smart_interval","type":"str","level":"advanced","flags":0,"default_value":"43200","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_user":{"name":"diskprediction_user","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator_cli","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"ssh","can_run":true,"error_string":"","module_options":{"inventory_cache_timeout":{"name":"inventory_cache_timeout","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["ceph-daemon-package","root"],"desc":"mode for remote execution of ceph-daemon","long_desc":"","tags":[],"see_also":[]},"service_cache_timeout":{"name":"service_cache_timeout","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"str","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"ansible","can_run":true,"error_string":"","module_options":{"ca_bundle":{"name":"ca_bundle","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_location":{"name":"server_location","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_server":{"name":"verify_server","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"none","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"1","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"str","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"str","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"str","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_cephfs":{"name":"FEATURE_TOGGLE_cephfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_iscsi":{"name":"FEATURE_TOGGLE_iscsi","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_mirroring":{"name":"FEATURE_TOGGLE_mirroring","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rbd":{"name":"FEATURE_TOGGLE_rbd","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rgw":{"name":"FEATURE_TOGGLE_rgw","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"str","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"str","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"str","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"str","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"str","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"str","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"deepsea","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"salt_api_eauth":{"name":"salt_api_eauth","type":"str","level":"advanced","flags":0,"default_value":"sharedsecret","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"salt_api_password":{"name":"salt_api_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"salt_api_url":{"name":"salt_api_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"salt_api_username":{"name":"salt_api_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_cloud","can_run":true,"error_string":"","module_options":{"diskprediction_cert_context":{"name":"diskprediction_cert_context","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_default_authority":{"name":"diskprediction_default_authority","type":"str","level":"advanced","flags":0,"default_value":"localhost","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_password":{"name":"diskprediction_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_port":{"name":"diskprediction_port","type":"str","level":"advanced","flags":0,"default_value":"31400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_retrieve_prediction_interval":{"name":"diskprediction_retrieve_prediction_interval","type":"str","level":"advanced","flags":0,"default_value":"43200","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_server":{"name":"diskprediction_server","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_ssl_target_name_override":{"name":"diskprediction_ssl_target_name_override","type":"str","level":"advanced","flags":0,"default_value":"localhost","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_upload_metrics_interval":{"name":"diskprediction_upload_metrics_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_upload_smart_interval":{"name":"diskprediction_upload_smart_interval","type":"str","level":"advanced","flags":0,"default_value":"43200","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_user":{"name":"diskprediction_user","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator_cli","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"ssh","can_run":true,"error_string":"","module_options":{"inventory_cache_timeout":{"name":"inventory_cache_timeout","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["ceph-daemon-package","root"],"desc":"mode for remote execution of ceph-daemon","long_desc":"","tags":[],"see_also":[]},"service_cache_timeout":{"name":"service_cache_timeout","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"nautilus":["balancer","crash","devicehealth","orchestrator_cli","progress","rbd_support","status","volumes"],"octopus":["balancer","crash","devicehealth","orchestrator_cli","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":0,"active_clients":[]}} 2019-12-05T05:03:39.524 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2019-12-05T05:03:39.524 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2019-12-05T05:03:39.524 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2019-12-05T05:03:39.626 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.616+0000 7f85253d9700 1 Processor -- start 2019-12-05T05:03:39.627 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.616+0000 7f85253d9700 1 -- start start 2019-12-05T05:03:39.627 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.616+0000 7f85253d9700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f852008df80 0x7f852009f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:39.627 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.616+0000 7f85253d9700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f852009f830 0x7f852014efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:39.627 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.616+0000 7f85253d9700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f852009fc60 con 0x7f85200a7740 2019-12-05T05:03:39.627 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.616+0000 7f85253d9700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85200a0b00 con 0x7f852009f830 2019-12-05T05:03:39.628 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.616+0000 7f85253d9700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f852008e5b0 con 0x7f852008df80 2019-12-05T05:03:39.628 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.616+0000 7f851ffff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f852009f830 0x7f852014efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:39.628 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.616+0000 7f851f7fe700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f85200a7740 0x7f852007e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:53302/0 (socket says 172.21.15.42:53302) 2019-12-05T05:03:39.628 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.616+0000 7f851f7fe700 1 -- 172.21.15.42:0/2532417725 learned_addr learned my addr 172.21.15.42:0/2532417725 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:03:39.628 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.616+0000 7f851effd700 1 --2- 172.21.15.42:0/2532417725 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f852008df80 0x7f852009f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:39.628 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.616+0000 7f851ffff700 1 -- 172.21.15.42:0/2532417725 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f852008df80 msgr2=0x7f852009f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:03:39.628 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.616+0000 7f851ffff700 1 --2- 172.21.15.42:0/2532417725 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f852008df80 0x7f852009f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:03:39.629 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.616+0000 7f851ffff700 1 -- 172.21.15.42:0/2532417725 >> v1:172.21.15.42:6789/0 conn(0x7f85200a7740 legacy=0x7f852007e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:39.629 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.616+0000 7f851e7fc700 1 -- 172.21.15.42:0/2532417725 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2178849093 0 0) 0x7f852009fc60 con 0x7f85200a7740 2019-12-05T05:03:39.629 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.616+0000 7f851ffff700 1 -- 172.21.15.42:0/2532417725 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85200a0710 con 0x7f852009f830 2019-12-05T05:03:39.629 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.616+0000 7f851ffff700 1 --2- 172.21.15.42:0/2532417725 >> v2:172.21.15.42:3300/0 conn(0x7f852009f830 0x7f852014efe0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f8518008790 tx=0x7f851800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:39.629 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.616+0000 7f851ffff700 1 --2- 172.21.15.42:0/2532417725 >> v2:172.21.15.42:3300/0 conn(0x7f852009f830 0x7f852014efe0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f8518008790 tx=0x7f851800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:39.629 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.620+0000 7f851ffff700 1 --2- 172.21.15.42:0/2532417725 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f852009f830 0x7f852014efe0 secure :-1 s=READY pgs=98 cs=0 l=1 rx=0x7f8518008790 tx=0x7f851800fc70).ready entity=mon.1 client_cookie=96810a6409a092d4 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:39.629 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.620+0000 7f851ffff700 1 --2- 172.21.15.42:0/2532417725 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f852009f830 0x7f852014efe0 secure :-1 s=THROTTLE_DONE pgs=98 cs=0 l=1 rx=0x7f8518008790 tx=0x7f851800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:39.630 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.620+0000 7f851e7fc700 1 -- 172.21.15.42:0/2532417725 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8518008bb0 con 0x7f852009f830 2019-12-05T05:03:39.630 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.620+0000 7f851ffff700 1 --2- 172.21.15.42:0/2532417725 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f852009f830 0x7f852014efe0 secure :-1 s=THROTTLE_DONE pgs=98 cs=0 l=1 rx=0x7f8518008790 tx=0x7f851800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:39.630 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.620+0000 7f851e7fc700 1 -- 172.21.15.42:0/2532417725 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85180087d0 con 0x7f852009f830 2019-12-05T05:03:39.630 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.620+0000 7f851ffff700 1 --2- 172.21.15.42:0/2532417725 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f852009f830 0x7f852014efe0 secure :-1 s=THROTTLE_DONE pgs=98 cs=0 l=1 rx=0x7f8518008790 tx=0x7f851800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:39.630 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.620+0000 7f85253d9700 1 -- 172.21.15.42:0/2532417725 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f852009f830 msgr2=0x7f852014efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:39.630 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.620+0000 7f851e7fc700 1 -- 172.21.15.42:0/2532417725 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8518011860 con 0x7f852009f830 2019-12-05T05:03:39.631 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.620+0000 7f85253d9700 1 --2- 172.21.15.42:0/2532417725 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f852009f830 0x7f852014efe0 secure :-1 s=READY pgs=98 cs=0 l=1 rx=0x7f8518008790 tx=0x7f851800fc70).stop 2019-12-05T05:03:39.631 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.620+0000 7f85253d9700 1 -- 172.21.15.42:0/2532417725 shutdown_connections 2019-12-05T05:03:39.631 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.620+0000 7f85253d9700 1 --2- 172.21.15.42:0/2532417725 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f852009f830 0x7f852014efe0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:39.631 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.620+0000 7f85253d9700 1 --2- 172.21.15.42:0/2532417725 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f852008df80 0x7f852009f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:03:39.631 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.620+0000 7f85253d9700 1 -- 172.21.15.42:0/2532417725 shutdown_connections 2019-12-05T05:03:39.631 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.620+0000 7f85253d9700 1 -- 172.21.15.42:0/2532417725 wait complete. 2019-12-05T05:03:39.632 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.620+0000 7f85253d9700 1 -- 172.21.15.42:0/2532417725 >> 172.21.15.42:0/2532417725 conn(0x7f85200aabe0 msgr2=0x7f85200a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:39.635 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.624+0000 7f85253d9700 1 Processor -- start 2019-12-05T05:03:39.635 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.624+0000 7f85253d9700 1 -- start start 2019-12-05T05:03:39.635 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.624+0000 7f85253d9700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8520117dd0 0x7f8520145b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:39.635 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.624+0000 7f85253d9700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8520127550 con 0x7f8520116700 2019-12-05T05:03:39.635 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.624+0000 7f85253d9700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85201304c0 con 0x7f852008eeb0 2019-12-05T05:03:39.636 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.624+0000 7f85253d9700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8520146450 con 0x7f8520117dd0 2019-12-05T05:03:39.636 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.624+0000 7f851effd700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f852008eeb0 0x7f8520126d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:53310/0 (socket says 172.21.15.42:53310) 2019-12-05T05:03:39.636 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.624+0000 7f851effd700 1 -- 172.21.15.42:0/1020022160 learned_addr learned my addr 172.21.15.42:0/1020022160 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:03:39.636 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.624+0000 7f851f7fe700 1 --2- 172.21.15.42:0/1020022160 >> v2:172.21.15.40:3300/0 conn(0x7f8520117dd0 0x7f8520145b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:39.636 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.624+0000 7f8507fff700 1 -- 172.21.15.42:0/1020022160 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 343652089 0 0) 0x7f85201304c0 con 0x7f852008eeb0 2019-12-05T05:03:39.636 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.624+0000 7f8507fff700 1 -- 172.21.15.42:0/1020022160 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85000030b0 con 0x7f852008eeb0 2019-12-05T05:03:39.636 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.624+0000 7f8507fff700 1 -- 172.21.15.42:0/1020022160 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 734987504 0 0) 0x7f85000030b0 con 0x7f852008eeb0 2019-12-05T05:03:39.637 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.624+0000 7f851f7fe700 1 -- 172.21.15.42:0/1020022160 >> v1:172.21.15.42:6789/0 conn(0x7f852008eeb0 legacy=0x7f8520126d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:39.637 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.624+0000 7f851f7fe700 1 -- 172.21.15.42:0/1020022160 >> v1:172.21.15.40:6789/0 conn(0x7f8520116700 legacy=0x7f8520137910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:39.637 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.624+0000 7f851f7fe700 1 -- 172.21.15.42:0/1020022160 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85201460d0 con 0x7f8520117dd0 2019-12-05T05:03:39.637 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.624+0000 7f851f7fe700 1 --2- 172.21.15.42:0/1020022160 >> v2:172.21.15.40:3300/0 conn(0x7f8520117dd0 0x7f8520145b80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f8518008370 tx=0x7f8514003370).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:39.637 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.628+0000 7f851f7fe700 1 --2- 172.21.15.42:0/1020022160 >> v2:172.21.15.40:3300/0 conn(0x7f8520117dd0 0x7f8520145b80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f8518008370 tx=0x7f8514003370).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:39.637 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.628+0000 7f851f7fe700 1 --2- 172.21.15.42:0/1020022160 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8520117dd0 0x7f8520145b80 secure :-1 s=READY pgs=91 cs=0 l=1 rx=0x7f8518008370 tx=0x7f8514003370).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:39.637 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.628+0000 7f851f7fe700 1 --2- 172.21.15.42:0/1020022160 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8520117dd0 0x7f8520145b80 secure :-1 s=THROTTLE_DONE pgs=91 cs=0 l=1 rx=0x7f8518008370 tx=0x7f8514003370).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:39.638 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.628+0000 7f8507fff700 1 -- 172.21.15.42:0/1020022160 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8514001810 con 0x7f8520117dd0 2019-12-05T05:03:39.638 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.628+0000 7f851f7fe700 1 --2- 172.21.15.42:0/1020022160 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8520117dd0 0x7f8520145b80 secure :-1 s=THROTTLE_DONE pgs=91 cs=0 l=1 rx=0x7f8518008370 tx=0x7f8514003370).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:39.638 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.628+0000 7f851f7fe700 1 --2- 172.21.15.42:0/1020022160 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8520117dd0 0x7f8520145b80 secure :-1 s=THROTTLE_DONE pgs=91 cs=0 l=1 rx=0x7f8518008370 tx=0x7f8514003370).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:39.638 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.628+0000 7f8507fff700 1 -- 172.21.15.42:0/1020022160 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8514001620 con 0x7f8520117dd0 2019-12-05T05:03:39.639 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.628+0000 7f8507fff700 1 -- 172.21.15.42:0/1020022160 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f85100116f0 con 0x7f8520117dd0 2019-12-05T05:03:39.639 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.628+0000 7f85253d9700 1 -- 172.21.15.42:0/1020022160 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8520116b30 con 0x7f8520117dd0 2019-12-05T05:03:39.639 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.628+0000 7f85253d9700 1 -- 172.21.15.42:0/1020022160 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8520139120 con 0x7f8520117dd0 2019-12-05T05:03:39.639 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.628+0000 7f851f7fe700 1 --2- 172.21.15.42:0/1020022160 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8520117dd0 0x7f8520145b80 secure :-1 s=THROTTLE_DONE pgs=91 cs=0 l=1 rx=0x7f8518008370 tx=0x7f8514003370).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:39.639 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.628+0000 7f8507fff700 1 -- 172.21.15.42:0/1020022160 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f85100119c0 con 0x7f8520117dd0 2019-12-05T05:03:39.639 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.628+0000 7f851f7fe700 1 --2- 172.21.15.42:0/1020022160 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8520117dd0 0x7f8520145b80 secure :-1 s=THROTTLE_DONE pgs=91 cs=0 l=1 rx=0x7f8518008370 tx=0x7f8514003370).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:39.639 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.628+0000 7f8507fff700 1 --2- 172.21.15.42:0/1020022160 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f850003b260 0x7f850003d770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:39.640 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.628+0000 7f8507fff700 1 -- 172.21.15.42:0/1020022160 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f851004ad00 con 0x7f8520117dd0 2019-12-05T05:03:39.640 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.628+0000 7f851effd700 1 --2- 172.21.15.42:0/1020022160 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f850003b260 0x7f850003d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:39.640 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.628+0000 7f851effd700 1 --2- 172.21.15.42:0/1020022160 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f850003b260 0x7f850003d770 crc :-1 s=READY pgs=61 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:39.641 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.632+0000 7f85253d9700 1 -- 172.21.15.42:0/1020022160 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f852005d110 con 0x7f8520117dd0 2019-12-05T05:03:39.647 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.640+0000 7f851f7fe700 1 --2- 172.21.15.42:0/1020022160 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8520117dd0 0x7f8520145b80 secure :-1 s=THROTTLE_DONE pgs=91 cs=0 l=1 rx=0x7f8518008370 tx=0x7f8514003370).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:39.647 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.640+0000 7f8507fff700 1 -- 172.21.15.42:0/1020022160 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f8510031fe0 con 0x7f8520117dd0 2019-12-05T05:03:39.815 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.808+0000 7f85253d9700 1 -- 172.21.15.42:0/1020022160 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f852005b8c0 con 0x7f8520117dd0 2019-12-05T05:03:39.817 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.812+0000 7f851f7fe700 1 --2- 172.21.15.42:0/1020022160 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8520117dd0 0x7f8520145b80 secure :-1 s=THROTTLE_DONE pgs=91 cs=0 l=1 rx=0x7f8518008370 tx=0x7f8514003370).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:39.818 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.812+0000 7f8507fff700 1 -- 172.21.15.42:0/1020022160 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+14885 (secure 0 0 0) 0x7f8510032b90 con 0x7f8520117dd0 2019-12-05T05:03:39.819 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:03:39.819 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":24,"fsid":"5d836cfc-a83b-4433-9b3e-b9b2f31c9050","created":"2019-12-05T05:03:13.215602+0000","modified":"2019-12-05T05:03:36.607704+0000","last_up_change":"2019-12-05T05:03:22.664783+0000","last_in_change":"2019-12-05T05:03:22.664783+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2019-12-05T05:03:27.394605+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2019-12-05T05:03:30.419845+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2019-12-05T05:03:33.249075+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e3fc49bb-dcbc-43ff-ac7c-9eb0ea0bd257","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6810","nonce":12118},{"type":"v1","addr":"172.21.15.42:6811","nonce":12118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6812","nonce":12118},{"type":"v1","addr":"172.21.15.42:6813","nonce":12118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6816","nonce":12118},{"type":"v1","addr":"172.21.15.42:6817","nonce":12118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6814","nonce":12118},{"type":"v1","addr":"172.21.15.42:6815","nonce":12118}]},"public_addr":"172.21.15.42:6811/12118","cluster_addr":"172.21.15.42:6813/12118","heartbeat_back_addr":"172.21.15.42:6817/12118","heartbeat_front_addr":"172.21.15.42:6815/12118","state":["exists","up"]},{"osd":1,"uuid":"902543fb-f5a2-4321-9c9c-26f9b9ca8629","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6826","nonce":12119},{"type":"v1","addr":"172.21.15.42:6827","nonce":12119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6828","nonce":12119},{"type":"v1","addr":"172.21.15.42:6829","nonce":12119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6832","nonce":12119},{"type":"v1","addr":"172.21.15.42:6833","nonce":12119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6830","nonce":12119},{"type":"v1","addr":"172.21.15.42:6831","nonce":12119}]},"public_addr":"172.21.15.42:6827/12119","cluster_addr":"172.21.15.42:6829/12119","heartbeat_back_addr":"172.21.15.42:6833/12119","heartbeat_front_addr":"172.21.15.42:6831/12119","state":["exists","up"]},{"osd":2,"uuid":"28739ff7-bed7-4b75-8ea5-b57d088f1250","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6802","nonce":12117},{"type":"v1","addr":"172.21.15.42:6803","nonce":12117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6804","nonce":12117},{"type":"v1","addr":"172.21.15.42:6805","nonce":12117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6808","nonce":12117},{"type":"v1","addr":"172.21.15.42:6809","nonce":12117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6806","nonce":12117},{"type":"v1","addr":"172.21.15.42:6807","nonce":12117}]},"public_addr":"172.21.15.42:6803/12117","cluster_addr":"172.21.15.42:6805/12117","heartbeat_back_addr":"172.21.15.42:6809/12117","heartbeat_front_addr":"172.21.15.42:6807/12117","state":["exists","up"]},{"osd":3,"uuid":"1960f66b-2d40-4bb9-a289-ce8cd0c1c142","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6818","nonce":12116},{"type":"v1","addr":"172.21.15.42:6819","nonce":12116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6820","nonce":12116},{"type":"v1","addr":"172.21.15.42:6821","nonce":12116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6824","nonce":12116},{"type":"v1","addr":"172.21.15.42:6825","nonce":12116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6822","nonce":12116},{"type":"v1","addr":"172.21.15.42:6823","nonce":12116}]},"public_addr":"172.21.15.42:6819/12116","cluster_addr":"172.21.15.42:6821/12116","heartbeat_back_addr":"172.21.15.42:6825/12116","heartbeat_front_addr":"172.21.15.42:6823/12116","state":["exists","up"]},{"osd":4,"uuid":"45bbf1bd-7c8a-4a0d-b76b-afa6148a855c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":12488},{"type":"v1","addr":"172.21.15.40:6809","nonce":12488}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":12488},{"type":"v1","addr":"172.21.15.40:6812","nonce":12488}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":12488},{"type":"v1","addr":"172.21.15.40:6825","nonce":12488}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":12488},{"type":"v1","addr":"172.21.15.40:6819","nonce":12488}]},"public_addr":"172.21.15.40:6809/12488","cluster_addr":"172.21.15.40:6812/12488","heartbeat_back_addr":"172.21.15.40:6825/12488","heartbeat_front_addr":"172.21.15.40:6819/12488","state":["exists","up"]},{"osd":5,"uuid":"8a5511cd-925f-4869-a49e-924383e4ce53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":12489},{"type":"v1","addr":"172.21.15.40:6801","nonce":12489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":12489},{"type":"v1","addr":"172.21.15.40:6803","nonce":12489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":12489},{"type":"v1","addr":"172.21.15.40:6807","nonce":12489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":12489},{"type":"v1","addr":"172.21.15.40:6805","nonce":12489}]},"public_addr":"172.21.15.40:6801/12489","cluster_addr":"172.21.15.40:6803/12489","heartbeat_back_addr":"172.21.15.40:6807/12489","heartbeat_front_addr":"172.21.15.40:6805/12489","state":["exists","up"]},{"osd":6,"uuid":"d692c179-379e-446c-8484-a219a9241e04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6811","nonce":12491},{"type":"v1","addr":"172.21.15.40:6814","nonce":12491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6817","nonce":12491},{"type":"v1","addr":"172.21.15.40:6818","nonce":12491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":12491},{"type":"v1","addr":"172.21.15.40:6828","nonce":12491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":12491},{"type":"v1","addr":"172.21.15.40:6824","nonce":12491}]},"public_addr":"172.21.15.40:6814/12491","cluster_addr":"172.21.15.40:6818/12491","heartbeat_back_addr":"172.21.15.40:6828/12491","heartbeat_front_addr":"172.21.15.40:6824/12491","state":["exists","up"]},{"osd":7,"uuid":"79c3c243-190e-4e35-b769-319a2077e7cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6813","nonce":12490},{"type":"v1","addr":"172.21.15.40:6816","nonce":12490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":12490},{"type":"v1","addr":"172.21.15.40:6823","nonce":12490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":12490},{"type":"v1","addr":"172.21.15.40:6831","nonce":12490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6827","nonce":12490},{"type":"v1","addr":"172.21.15.40:6829","nonce":12490}]},"public_addr":"172.21.15.40:6816/12490","cluster_addr":"172.21.15.40:6823/12490","heartbeat_back_addr":"172.21.15.40:6831/12490","heartbeat_front_addr":"172.21.15.40:6829/12490","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T05:03:21.349150+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T05:03:21.644281+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T05:03:21.394040+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T05:03:21.587096+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T05:03:21.656497+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T05:03:21.510547+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T05:03:21.659358+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T05:03:21.628063+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2019-12-05T05:03:39.821 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.816+0000 7f85253d9700 1 -- 172.21.15.42:0/1020022160 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f850003b260 msgr2=0x7f850003d770 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:39.821 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.816+0000 7f85253d9700 1 --2- 172.21.15.42:0/1020022160 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f850003b260 0x7f850003d770 crc :-1 s=READY pgs=61 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:39.821 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.816+0000 7f85253d9700 1 -- 172.21.15.42:0/1020022160 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8520117dd0 msgr2=0x7f8520145b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:39.822 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.816+0000 7f85253d9700 1 --2- 172.21.15.42:0/1020022160 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8520117dd0 0x7f8520145b80 secure :-1 s=READY pgs=91 cs=0 l=1 rx=0x7f8518008370 tx=0x7f8514003370).stop 2019-12-05T05:03:39.822 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.816+0000 7f85253d9700 1 -- 172.21.15.42:0/1020022160 shutdown_connections 2019-12-05T05:03:39.822 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.816+0000 7f85253d9700 1 --2- 172.21.15.42:0/1020022160 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f850003b260 0x7f850003d770 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:39.822 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.816+0000 7f85253d9700 1 --2- 172.21.15.42:0/1020022160 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8520117dd0 0x7f8520145b80 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:39.822 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.816+0000 7f85253d9700 1 -- 172.21.15.42:0/1020022160 shutdown_connections 2019-12-05T05:03:39.822 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.816+0000 7f85253d9700 1 -- 172.21.15.42:0/1020022160 wait complete. 2019-12-05T05:03:39.823 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.816+0000 7f85253d9700 1 -- 172.21.15.42:0/1020022160 >> 172.21.15.42:0/1020022160 conn(0x7f85200aabe0 msgr2=0x7f852011c470 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:39.832 INFO:tasks.ceph.ceph_manager.ceph:all up! 2019-12-05T05:03:39.832 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2019-12-05T05:03:39.929 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.916+0000 7f5d42785700 1 Processor -- start 2019-12-05T05:03:39.929 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.916+0000 7f5d42785700 1 -- start start 2019-12-05T05:03:39.929 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.920+0000 7f5d42785700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5d3c0a7740 0x7f5d3c08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:39.930 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.920+0000 7f5d42785700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d3c0a0710 con 0x7f5d3c09fae0 2019-12-05T05:03:39.930 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.920+0000 7f5d42785700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d3c0a9ba0 con 0x7f5d3c07e720 2019-12-05T05:03:39.930 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.920+0000 7f5d42785700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d3c0a0b00 con 0x7f5d3c0a7740 2019-12-05T05:03:39.930 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.920+0000 7f5d3bfff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f5d3c07e720 0x7f5d3c09f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:53318/0 (socket says 172.21.15.42:53318) 2019-12-05T05:03:39.930 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.920+0000 7f5d3bfff700 1 -- 172.21.15.42:0/1628316344 learned_addr learned my addr 172.21.15.42:0/1628316344 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:03:39.931 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.920+0000 7f5d40d22700 1 --2- 172.21.15.42:0/1628316344 >> v2:172.21.15.40:3300/0 conn(0x7f5d3c0a7740 0x7f5d3c08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:39.931 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.920+0000 7f5d3b7fe700 1 -- 172.21.15.42:0/1628316344 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1375281194 0 0) 0x7f5d3c0a9ba0 con 0x7f5d3c07e720 2019-12-05T05:03:39.931 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.920+0000 7f5d3b7fe700 1 -- 172.21.15.42:0/1628316344 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d280030b0 con 0x7f5d3c07e720 2019-12-05T05:03:39.931 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.920+0000 7f5d3b7fe700 1 -- 172.21.15.42:0/1628316344 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3080011855 0 0) 0x7f5d280030b0 con 0x7f5d3c07e720 2019-12-05T05:03:39.931 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.920+0000 7f5d3b7fe700 1 -- 172.21.15.42:0/1628316344 >> v2:172.21.15.40:3300/0 conn(0x7f5d3c0a7740 msgr2=0x7f5d3c08df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:03:39.931 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.920+0000 7f5d3b7fe700 1 --2- 172.21.15.42:0/1628316344 >> v2:172.21.15.40:3300/0 conn(0x7f5d3c0a7740 0x7f5d3c08df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:03:39.932 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.920+0000 7f5d3b7fe700 1 -- 172.21.15.42:0/1628316344 >> v1:172.21.15.40:6789/0 conn(0x7f5d3c09fae0 legacy=0x7f5d3c14efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:39.932 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.920+0000 7f5d3b7fe700 1 -- 172.21.15.42:0/1628316344 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d3c07eb50 con 0x7f5d3c07e720 2019-12-05T05:03:39.932 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.920+0000 7f5d3b7fe700 1 -- 172.21.15.42:0/1628316344 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7f5d24001750 con 0x7f5d3c07e720 2019-12-05T05:03:39.932 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.920+0000 7f5d3b7fe700 1 -- 172.21.15.42:0/1628316344 >> v1:172.21.15.42:6789/0 conn(0x7f5d3c07e720 legacy=0x7f5d3c09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:39.932 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.920+0000 7f5d3b7fe700 1 --2- 172.21.15.42:0/1628316344 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5d28003730 0x7f5d28003ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:39.932 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.920+0000 7f5d3b7fe700 1 --2- 172.21.15.42:0/1628316344 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5d28004420 0x7f5d28004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:39.932 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.920+0000 7f5d3b7fe700 1 --2- 172.21.15.42:0/1628316344 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5d28004da0 0x7f5d2800b670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:39.933 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.920+0000 7f5d3b7fe700 1 -- 172.21.15.42:0/1628316344 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d3c0a0b00 con 0x7f5d28004420 2019-12-05T05:03:39.933 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.920+0000 7f5d3b7fe700 1 -- 172.21.15.42:0/1628316344 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d2804fc90 con 0x7f5d28003730 2019-12-05T05:03:39.933 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.920+0000 7f5d40d22700 1 --2- 172.21.15.42:0/1628316344 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5d28003730 0x7f5d28003ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:39.933 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.920+0000 7f5d3b7fe700 1 -- 172.21.15.42:0/1628316344 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d2804fed0 con 0x7f5d28004da0 2019-12-05T05:03:39.933 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.920+0000 7f5d3bfff700 1 --2- 172.21.15.42:0/1628316344 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5d28004420 0x7f5d28004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:39.933 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.920+0000 7f5d41523700 1 --2- 172.21.15.42:0/1628316344 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5d28004da0 0x7f5d2800b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:39.933 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.920+0000 7f5d40d22700 1 -- 172.21.15.42:0/1628316344 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5d28004da0 msgr2=0x7f5d2800b670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:03:39.934 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.920+0000 7f5d40d22700 1 --2- 172.21.15.42:0/1628316344 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5d28004da0 0x7f5d2800b670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:03:39.934 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.920+0000 7f5d40d22700 1 -- 172.21.15.42:0/1628316344 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5d28004420 msgr2=0x7f5d28004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:03:39.934 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.920+0000 7f5d40d22700 1 --2- 172.21.15.42:0/1628316344 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5d28004420 0x7f5d28004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:03:39.934 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.920+0000 7f5d40d22700 1 -- 172.21.15.42:0/1628316344 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d280501d0 con 0x7f5d28003730 2019-12-05T05:03:39.934 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.920+0000 7f5d41523700 1 -- 172.21.15.42:0/1628316344 reap_dead start 2019-12-05T05:03:39.934 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.920+0000 7f5d40d22700 1 --2- 172.21.15.42:0/1628316344 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5d28003730 0x7f5d28003ba0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f5d340032b0 tx=0x7f5d34010360).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:39.934 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.920+0000 7f5d40d22700 1 --2- 172.21.15.42:0/1628316344 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5d28003730 0x7f5d28003ba0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f5d340032b0 tx=0x7f5d34010360).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:39.935 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.920+0000 7f5d40d22700 1 --2- 172.21.15.42:0/1628316344 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5d28003730 0x7f5d28003ba0 secure :-1 s=READY pgs=101 cs=0 l=1 rx=0x7f5d340032b0 tx=0x7f5d34010360).ready entity=mon.1 client_cookie=55409dfab586799a server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:39.935 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.920+0000 7f5d40d22700 1 --2- 172.21.15.42:0/1628316344 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5d28003730 0x7f5d28003ba0 secure :-1 s=THROTTLE_DONE pgs=101 cs=0 l=1 rx=0x7f5d340032b0 tx=0x7f5d34010360).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:39.935 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.920+0000 7f5d3b7fe700 1 -- 172.21.15.42:0/1628316344 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5d34008d40 con 0x7f5d28003730 2019-12-05T05:03:39.935 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.920+0000 7f5d40d22700 1 --2- 172.21.15.42:0/1628316344 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5d28003730 0x7f5d28003ba0 secure :-1 s=THROTTLE_DONE pgs=101 cs=0 l=1 rx=0x7f5d340032b0 tx=0x7f5d34010360).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:39.935 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.920+0000 7f5d40d22700 1 --2- 172.21.15.42:0/1628316344 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5d28003730 0x7f5d28003ba0 secure :-1 s=THROTTLE_DONE pgs=101 cs=0 l=1 rx=0x7f5d340032b0 tx=0x7f5d34010360).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:39.935 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.920+0000 7f5d3b7fe700 1 -- 172.21.15.42:0/1628316344 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d34010cb0 con 0x7f5d28003730 2019-12-05T05:03:39.935 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.920+0000 7f5d3b7fe700 1 -- 172.21.15.42:0/1628316344 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5d34005ca0 con 0x7f5d28003730 2019-12-05T05:03:39.936 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.920+0000 7f5d42785700 1 -- 172.21.15.42:0/1628316344 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5d28003730 msgr2=0x7f5d28003ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:39.936 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.920+0000 7f5d42785700 1 --2- 172.21.15.42:0/1628316344 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5d28003730 0x7f5d28003ba0 secure :-1 s=READY pgs=101 cs=0 l=1 rx=0x7f5d340032b0 tx=0x7f5d34010360).stop 2019-12-05T05:03:39.936 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.920+0000 7f5d42785700 1 -- 172.21.15.42:0/1628316344 shutdown_connections 2019-12-05T05:03:39.936 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.920+0000 7f5d42785700 1 --2- 172.21.15.42:0/1628316344 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5d28003730 0x7f5d28003ba0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:39.936 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.920+0000 7f5d42785700 1 -- 172.21.15.42:0/1628316344 shutdown_connections 2019-12-05T05:03:39.936 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.920+0000 7f5d42785700 1 -- 172.21.15.42:0/1628316344 wait complete. 2019-12-05T05:03:39.937 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.920+0000 7f5d42785700 1 -- 172.21.15.42:0/1628316344 >> 172.21.15.42:0/1628316344 conn(0x7f5d3c0aabe0 msgr2=0x7f5d3c0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:39.937 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.924+0000 7f5d42785700 1 Processor -- start 2019-12-05T05:03:39.937 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.924+0000 7f5d42785700 1 -- start start 2019-12-05T05:03:39.942 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.924+0000 7f5d42785700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5d3c0a4b20 0x7f5d3c0a2930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:39.942 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.924+0000 7f5d42785700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5d3c133820 0x7f5d3c135c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:39.942 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.924+0000 7f5d42785700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d3c128820 con 0x7f5d3c07cf80 2019-12-05T05:03:39.942 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.924+0000 7f5d42785700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d3c142960 con 0x7f5d3c133820 2019-12-05T05:03:39.942 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.924+0000 7f5d42785700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d3c142c00 con 0x7f5d3c0a4b20 2019-12-05T05:03:39.943 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.924+0000 7f5d3bfff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f5d3c07cf80 0x7f5d3c128020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:53330/0 (socket says 172.21.15.42:53330) 2019-12-05T05:03:39.943 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.924+0000 7f5d3bfff700 1 -- 172.21.15.42:0/450814007 learned_addr learned my addr 172.21.15.42:0/450814007 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:03:39.943 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.924+0000 7f5d40d22700 1 --2- 172.21.15.42:0/450814007 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5d3c0a4b20 0x7f5d3c0a2930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:39.943 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.924+0000 7f5d41523700 1 --2- 172.21.15.42:0/450814007 >> v2:172.21.15.40:3300/0 conn(0x7f5d3c133820 0x7f5d3c135c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:39.943 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.924+0000 7f5d397fa700 1 -- 172.21.15.42:0/450814007 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2560306171 0 0) 0x7f5d3c128820 con 0x7f5d3c07cf80 2019-12-05T05:03:39.943 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.924+0000 7f5d397fa700 1 -- 172.21.15.42:0/450814007 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d200030b0 con 0x7f5d3c07cf80 2019-12-05T05:03:39.943 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.924+0000 7f5d397fa700 1 -- 172.21.15.42:0/450814007 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 3395635993 0 0) 0x7f5d200030b0 con 0x7f5d3c07cf80 2019-12-05T05:03:39.944 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.924+0000 7f5d397fa700 1 -- 172.21.15.42:0/450814007 --> v1:172.21.15.42:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5d3c128820 con 0x7f5d3c07cf80 2019-12-05T05:03:39.944 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.924+0000 7f5d397fa700 1 -- 172.21.15.42:0/450814007 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7f5d24001750 con 0x7f5d3c07cf80 2019-12-05T05:03:39.944 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.924+0000 7f5d40d22700 1 -- 172.21.15.42:0/450814007 >> v2:172.21.15.40:3300/0 conn(0x7f5d3c133820 msgr2=0x7f5d3c135c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:39.944 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.924+0000 7f5d40d22700 1 --2- 172.21.15.42:0/450814007 >> v2:172.21.15.40:3300/0 conn(0x7f5d3c133820 0x7f5d3c135c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:39.944 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.924+0000 7f5d40d22700 1 -- 172.21.15.42:0/450814007 >> v1:172.21.15.42:6789/0 conn(0x7f5d3c07cf80 legacy=0x7f5d3c128020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:39.944 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.924+0000 7f5d40d22700 1 -- 172.21.15.42:0/450814007 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d3c0a4840 con 0x7f5d3c0a4b20 2019-12-05T05:03:39.945 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.924+0000 7f5d41523700 1 --2- 172.21.15.42:0/450814007 >> v2:172.21.15.40:3300/0 conn(0x7f5d3c133820 0x7f5d3c135c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:03:39.945 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.924+0000 7f5d40d22700 1 --2- 172.21.15.42:0/450814007 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5d3c0a4b20 0x7f5d3c0a2930 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f5d340032b0 tx=0x7f5d24001950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:39.945 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.924+0000 7f5d40d22700 1 --2- 172.21.15.42:0/450814007 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5d3c0a4b20 0x7f5d3c0a2930 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f5d340032b0 tx=0x7f5d24001950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:39.945 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.924+0000 7f5d40d22700 1 --2- 172.21.15.42:0/450814007 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5d3c0a4b20 0x7f5d3c0a2930 secure :-1 s=READY pgs=32 cs=0 l=1 rx=0x7f5d340032b0 tx=0x7f5d24001950).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:39.945 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.928+0000 7f5d40d22700 1 --2- 172.21.15.42:0/450814007 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5d3c0a4b20 0x7f5d3c0a2930 secure :-1 s=THROTTLE_DONE pgs=32 cs=0 l=1 rx=0x7f5d340032b0 tx=0x7f5d24001950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:39.945 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.928+0000 7f5d397fa700 1 -- 172.21.15.42:0/450814007 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5d34007350 con 0x7f5d3c0a4b20 2019-12-05T05:03:39.945 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.928+0000 7f5d40d22700 1 --2- 172.21.15.42:0/450814007 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5d3c0a4b20 0x7f5d3c0a2930 secure :-1 s=THROTTLE_DONE pgs=32 cs=0 l=1 rx=0x7f5d340032b0 tx=0x7f5d24001950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:39.946 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.928+0000 7f5d40d22700 1 --2- 172.21.15.42:0/450814007 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5d3c0a4b20 0x7f5d3c0a2930 secure :-1 s=THROTTLE_DONE pgs=32 cs=0 l=1 rx=0x7f5d340032b0 tx=0x7f5d24001950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:39.946 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.928+0000 7f5d397fa700 1 -- 172.21.15.42:0/450814007 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d340058c0 con 0x7f5d3c0a4b20 2019-12-05T05:03:39.946 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.928+0000 7f5d42785700 1 -- 172.21.15.42:0/450814007 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d3c05ae30 con 0x7f5d3c0a4b20 2019-12-05T05:03:39.946 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.928+0000 7f5d397fa700 1 -- 172.21.15.42:0/450814007 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5d340151f0 con 0x7f5d3c0a4b20 2019-12-05T05:03:39.946 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.928+0000 7f5d42785700 1 -- 172.21.15.42:0/450814007 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d3c0a4220 con 0x7f5d3c0a4b20 2019-12-05T05:03:39.946 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.928+0000 7f5d40d22700 1 --2- 172.21.15.42:0/450814007 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5d3c0a4b20 0x7f5d3c0a2930 secure :-1 s=THROTTLE_DONE pgs=32 cs=0 l=1 rx=0x7f5d340032b0 tx=0x7f5d24001950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:39.946 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.928+0000 7f5d397fa700 1 -- 172.21.15.42:0/450814007 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f5d340153f0 con 0x7f5d3c0a4b20 2019-12-05T05:03:39.947 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.928+0000 7f5d40d22700 1 --2- 172.21.15.42:0/450814007 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5d3c0a4b20 0x7f5d3c0a2930 secure :-1 s=THROTTLE_DONE pgs=32 cs=0 l=1 rx=0x7f5d340032b0 tx=0x7f5d24001950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:39.947 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.928+0000 7f5d397fa700 1 --2- 172.21.15.42:0/450814007 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5d2003b5e0 0x7f5d2003daf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:39.947 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.928+0000 7f5d397fa700 1 -- 172.21.15.42:0/450814007 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f5d3404aeb0 con 0x7f5d3c0a4b20 2019-12-05T05:03:39.947 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.928+0000 7f5d3bfff700 1 --2- 172.21.15.42:0/450814007 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5d2003b5e0 0x7f5d2003daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:39.947 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.928+0000 7f5d3bfff700 1 --2- 172.21.15.42:0/450814007 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5d2003b5e0 0x7f5d2003daf0 crc :-1 s=READY pgs=62 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:39.947 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.932+0000 7f5d42785700 1 -- 172.21.15.42:0/450814007 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d3c05d110 con 0x7f5d3c0a4b20 2019-12-05T05:03:39.948 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.944+0000 7f5d40d22700 1 --2- 172.21.15.42:0/450814007 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5d3c0a4b20 0x7f5d3c0a2930 secure :-1 s=THROTTLE_DONE pgs=32 cs=0 l=1 rx=0x7f5d340032b0 tx=0x7f5d24001950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:39.948 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:39.944+0000 7f5d397fa700 1 -- 172.21.15.42:0/450814007 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f5d34031f30 con 0x7f5d3c0a4b20 2019-12-05T05:03:40.119 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.112+0000 7f5d42785700 1 -- 172.21.15.42:0/450814007 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5d3c0a4000 con 0x7f5d3c0a4b20 2019-12-05T05:03:40.121 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.116+0000 7f5d40d22700 1 --2- 172.21.15.42:0/450814007 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5d3c0a4b20 0x7f5d3c0a2930 secure :-1 s=THROTTLE_DONE pgs=32 cs=0 l=1 rx=0x7f5d340032b0 tx=0x7f5d24001950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:40.121 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.116+0000 7f5d397fa700 1 -- 172.21.15.42:0/450814007 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+14885 (secure 0 0 0) 0x7f5d34032190 con 0x7f5d3c0a4b20 2019-12-05T05:03:40.122 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:03:40.123 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":24,"fsid":"5d836cfc-a83b-4433-9b3e-b9b2f31c9050","created":"2019-12-05T05:03:13.215602+0000","modified":"2019-12-05T05:03:36.607704+0000","last_up_change":"2019-12-05T05:03:22.664783+0000","last_in_change":"2019-12-05T05:03:22.664783+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2019-12-05T05:03:27.394605+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2019-12-05T05:03:30.419845+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2019-12-05T05:03:33.249075+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e3fc49bb-dcbc-43ff-ac7c-9eb0ea0bd257","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6810","nonce":12118},{"type":"v1","addr":"172.21.15.42:6811","nonce":12118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6812","nonce":12118},{"type":"v1","addr":"172.21.15.42:6813","nonce":12118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6816","nonce":12118},{"type":"v1","addr":"172.21.15.42:6817","nonce":12118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6814","nonce":12118},{"type":"v1","addr":"172.21.15.42:6815","nonce":12118}]},"public_addr":"172.21.15.42:6811/12118","cluster_addr":"172.21.15.42:6813/12118","heartbeat_back_addr":"172.21.15.42:6817/12118","heartbeat_front_addr":"172.21.15.42:6815/12118","state":["exists","up"]},{"osd":1,"uuid":"902543fb-f5a2-4321-9c9c-26f9b9ca8629","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6826","nonce":12119},{"type":"v1","addr":"172.21.15.42:6827","nonce":12119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6828","nonce":12119},{"type":"v1","addr":"172.21.15.42:6829","nonce":12119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6832","nonce":12119},{"type":"v1","addr":"172.21.15.42:6833","nonce":12119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6830","nonce":12119},{"type":"v1","addr":"172.21.15.42:6831","nonce":12119}]},"public_addr":"172.21.15.42:6827/12119","cluster_addr":"172.21.15.42:6829/12119","heartbeat_back_addr":"172.21.15.42:6833/12119","heartbeat_front_addr":"172.21.15.42:6831/12119","state":["exists","up"]},{"osd":2,"uuid":"28739ff7-bed7-4b75-8ea5-b57d088f1250","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6802","nonce":12117},{"type":"v1","addr":"172.21.15.42:6803","nonce":12117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6804","nonce":12117},{"type":"v1","addr":"172.21.15.42:6805","nonce":12117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6808","nonce":12117},{"type":"v1","addr":"172.21.15.42:6809","nonce":12117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6806","nonce":12117},{"type":"v1","addr":"172.21.15.42:6807","nonce":12117}]},"public_addr":"172.21.15.42:6803/12117","cluster_addr":"172.21.15.42:6805/12117","heartbeat_back_addr":"172.21.15.42:6809/12117","heartbeat_front_addr":"172.21.15.42:6807/12117","state":["exists","up"]},{"osd":3,"uuid":"1960f66b-2d40-4bb9-a289-ce8cd0c1c142","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6818","nonce":12116},{"type":"v1","addr":"172.21.15.42:6819","nonce":12116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6820","nonce":12116},{"type":"v1","addr":"172.21.15.42:6821","nonce":12116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6824","nonce":12116},{"type":"v1","addr":"172.21.15.42:6825","nonce":12116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6822","nonce":12116},{"type":"v1","addr":"172.21.15.42:6823","nonce":12116}]},"public_addr":"172.21.15.42:6819/12116","cluster_addr":"172.21.15.42:6821/12116","heartbeat_back_addr":"172.21.15.42:6825/12116","heartbeat_front_addr":"172.21.15.42:6823/12116","state":["exists","up"]},{"osd":4,"uuid":"45bbf1bd-7c8a-4a0d-b76b-afa6148a855c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":12488},{"type":"v1","addr":"172.21.15.40:6809","nonce":12488}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":12488},{"type":"v1","addr":"172.21.15.40:6812","nonce":12488}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":12488},{"type":"v1","addr":"172.21.15.40:6825","nonce":12488}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":12488},{"type":"v1","addr":"172.21.15.40:6819","nonce":12488}]},"public_addr":"172.21.15.40:6809/12488","cluster_addr":"172.21.15.40:6812/12488","heartbeat_back_addr":"172.21.15.40:6825/12488","heartbeat_front_addr":"172.21.15.40:6819/12488","state":["exists","up"]},{"osd":5,"uuid":"8a5511cd-925f-4869-a49e-924383e4ce53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":12489},{"type":"v1","addr":"172.21.15.40:6801","nonce":12489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":12489},{"type":"v1","addr":"172.21.15.40:6803","nonce":12489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":12489},{"type":"v1","addr":"172.21.15.40:6807","nonce":12489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":12489},{"type":"v1","addr":"172.21.15.40:6805","nonce":12489}]},"public_addr":"172.21.15.40:6801/12489","cluster_addr":"172.21.15.40:6803/12489","heartbeat_back_addr":"172.21.15.40:6807/12489","heartbeat_front_addr":"172.21.15.40:6805/12489","state":["exists","up"]},{"osd":6,"uuid":"d692c179-379e-446c-8484-a219a9241e04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6811","nonce":12491},{"type":"v1","addr":"172.21.15.40:6814","nonce":12491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6817","nonce":12491},{"type":"v1","addr":"172.21.15.40:6818","nonce":12491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":12491},{"type":"v1","addr":"172.21.15.40:6828","nonce":12491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":12491},{"type":"v1","addr":"172.21.15.40:6824","nonce":12491}]},"public_addr":"172.21.15.40:6814/12491","cluster_addr":"172.21.15.40:6818/12491","heartbeat_back_addr":"172.21.15.40:6828/12491","heartbeat_front_addr":"172.21.15.40:6824/12491","state":["exists","up"]},{"osd":7,"uuid":"79c3c243-190e-4e35-b769-319a2077e7cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6813","nonce":12490},{"type":"v1","addr":"172.21.15.40:6816","nonce":12490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":12490},{"type":"v1","addr":"172.21.15.40:6823","nonce":12490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":12490},{"type":"v1","addr":"172.21.15.40:6831","nonce":12490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6827","nonce":12490},{"type":"v1","addr":"172.21.15.40:6829","nonce":12490}]},"public_addr":"172.21.15.40:6816/12490","cluster_addr":"172.21.15.40:6823/12490","heartbeat_back_addr":"172.21.15.40:6831/12490","heartbeat_front_addr":"172.21.15.40:6829/12490","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T05:03:21.349150+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T05:03:21.644281+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T05:03:21.394040+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T05:03:21.587096+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T05:03:21.656497+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T05:03:21.510547+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T05:03:21.659358+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T05:03:21.628063+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2019-12-05T05:03:40.125 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.120+0000 7f5d42785700 1 -- 172.21.15.42:0/450814007 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5d2003b5e0 msgr2=0x7f5d2003daf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:40.125 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.120+0000 7f5d42785700 1 --2- 172.21.15.42:0/450814007 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5d2003b5e0 0x7f5d2003daf0 crc :-1 s=READY pgs=62 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:40.125 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.120+0000 7f5d42785700 1 -- 172.21.15.42:0/450814007 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5d3c0a4b20 msgr2=0x7f5d3c0a2930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:40.125 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.120+0000 7f5d42785700 1 --2- 172.21.15.42:0/450814007 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5d3c0a4b20 0x7f5d3c0a2930 secure :-1 s=READY pgs=32 cs=0 l=1 rx=0x7f5d340032b0 tx=0x7f5d24001950).stop 2019-12-05T05:03:40.125 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.120+0000 7f5d42785700 1 -- 172.21.15.42:0/450814007 shutdown_connections 2019-12-05T05:03:40.126 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.120+0000 7f5d42785700 1 --2- 172.21.15.42:0/450814007 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5d2003b5e0 0x7f5d2003daf0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:40.126 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.120+0000 7f5d42785700 1 --2- 172.21.15.42:0/450814007 >> v2:172.21.15.40:3300/0 conn(0x7f5d3c133820 0x7f5d3c135c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:40.126 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.120+0000 7f5d42785700 1 --2- 172.21.15.42:0/450814007 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5d3c0a4b20 0x7f5d3c0a2930 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:40.126 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.120+0000 7f5d42785700 1 -- 172.21.15.42:0/450814007 shutdown_connections 2019-12-05T05:03:40.126 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.120+0000 7f5d42785700 1 -- 172.21.15.42:0/450814007 wait complete. 2019-12-05T05:03:40.127 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.120+0000 7f5d42785700 1 -- 172.21.15.42:0/450814007 >> 172.21.15.42:0/450814007 conn(0x7f5d3c0aabe0 msgr2=0x7f5d3c07c440 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:40.135 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2019-12-05T05:03:40.229 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.216+0000 7f4ee02a7700 1 Processor -- start 2019-12-05T05:03:40.229 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.216+0000 7f4ee02a7700 1 -- start start 2019-12-05T05:03:40.229 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.220+0000 7f4ee02a7700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4ed80a7740 0x7f4ed808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:40.229 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.220+0000 7f4ee02a7700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f4ed807e720 0x7f4ed807eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:40.230 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.220+0000 7f4ee02a7700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ed80a0710 con 0x7f4ed809f2e0 2019-12-05T05:03:40.230 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.220+0000 7f4ee02a7700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ed80a0b00 con 0x7f4ed807e720 2019-12-05T05:03:40.230 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.220+0000 7f4ee02a7700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ed808e5b0 con 0x7f4ed80a7740 2019-12-05T05:03:40.230 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.220+0000 7f4ede043700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f4ed807e720 0x7f4ed807eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:40.230 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.220+0000 7f4edf045700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f4ed809f2e0 0x7f4ed809f710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:53340/0 (socket says 172.21.15.42:53340) 2019-12-05T05:03:40.231 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.220+0000 7f4edf045700 1 -- 172.21.15.42:0/3719453718 learned_addr learned my addr 172.21.15.42:0/3719453718 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:03:40.231 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.220+0000 7f4ede844700 1 --2- 172.21.15.42:0/3719453718 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4ed80a7740 0x7f4ed808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:40.231 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.220+0000 7f4ede043700 1 -- 172.21.15.42:0/3719453718 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4ed80a7740 msgr2=0x7f4ed808df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:03:40.231 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.220+0000 7f4ede043700 1 --2- 172.21.15.42:0/3719453718 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4ed80a7740 0x7f4ed808df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:03:40.231 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.220+0000 7f4ede043700 1 -- 172.21.15.42:0/3719453718 >> v1:172.21.15.42:6789/0 conn(0x7f4ed809f2e0 legacy=0x7f4ed809f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:40.232 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.220+0000 7f4ede043700 1 -- 172.21.15.42:0/3719453718 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ed807f0a0 con 0x7f4ed807e720 2019-12-05T05:03:40.232 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.220+0000 7f4ede043700 1 --2- 172.21.15.42:0/3719453718 >> v2:172.21.15.42:3300/0 conn(0x7f4ed807e720 0x7f4ed807eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f4ec8008790 tx=0x7f4ed00014d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:40.232 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.220+0000 7f4ede043700 1 --2- 172.21.15.42:0/3719453718 >> v2:172.21.15.42:3300/0 conn(0x7f4ed807e720 0x7f4ed807eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f4ec8008790 tx=0x7f4ed00014d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:40.232 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.220+0000 7f4ede043700 1 --2- 172.21.15.42:0/3719453718 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4ed807e720 0x7f4ed807eb50 secure :-1 s=READY pgs=104 cs=0 l=1 rx=0x7f4ec8008790 tx=0x7f4ed00014d0).ready entity=mon.1 client_cookie=5822736f9b08cdbb server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:40.232 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.220+0000 7f4ede043700 1 --2- 172.21.15.42:0/3719453718 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4ed807e720 0x7f4ed807eb50 secure :-1 s=THROTTLE_DONE pgs=104 cs=0 l=1 rx=0x7f4ec8008790 tx=0x7f4ed00014d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:40.232 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.220+0000 7f4ede043700 1 --2- 172.21.15.42:0/3719453718 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4ed807e720 0x7f4ed807eb50 secure :-1 s=THROTTLE_DONE pgs=104 cs=0 l=1 rx=0x7f4ec8008790 tx=0x7f4ed00014d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:40.233 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.220+0000 7f4edd842700 1 -- 172.21.15.42:0/3719453718 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4ec8008bb0 con 0x7f4ed807e720 2019-12-05T05:03:40.233 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.220+0000 7f4ede043700 1 --2- 172.21.15.42:0/3719453718 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4ed807e720 0x7f4ed807eb50 secure :-1 s=THROTTLE_DONE pgs=104 cs=0 l=1 rx=0x7f4ec8008790 tx=0x7f4ed00014d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:40.233 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.220+0000 7f4edd842700 1 -- 172.21.15.42:0/3719453718 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ec80087d0 con 0x7f4ed807e720 2019-12-05T05:03:40.233 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.220+0000 7f4edd842700 1 -- 172.21.15.42:0/3719453718 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4ec8011760 con 0x7f4ed807e720 2019-12-05T05:03:40.233 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.220+0000 7f4ee02a7700 1 -- 172.21.15.42:0/3719453718 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4ed807e720 msgr2=0x7f4ed807eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:40.233 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.220+0000 7f4ee02a7700 1 --2- 172.21.15.42:0/3719453718 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4ed807e720 0x7f4ed807eb50 secure :-1 s=READY pgs=104 cs=0 l=1 rx=0x7f4ec8008790 tx=0x7f4ed00014d0).stop 2019-12-05T05:03:40.233 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.224+0000 7f4ee02a7700 1 -- 172.21.15.42:0/3719453718 shutdown_connections 2019-12-05T05:03:40.234 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.224+0000 7f4ee02a7700 1 --2- 172.21.15.42:0/3719453718 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4ed80a7740 0x7f4ed808df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:03:40.234 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.224+0000 7f4ee02a7700 1 --2- 172.21.15.42:0/3719453718 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4ed807e720 0x7f4ed807eb50 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:40.234 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.224+0000 7f4ee02a7700 1 -- 172.21.15.42:0/3719453718 shutdown_connections 2019-12-05T05:03:40.234 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.224+0000 7f4ee02a7700 1 -- 172.21.15.42:0/3719453718 wait complete. 2019-12-05T05:03:40.234 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.224+0000 7f4ee02a7700 1 -- 172.21.15.42:0/3719453718 >> 172.21.15.42:0/3719453718 conn(0x7f4ed80aabe0 msgr2=0x7f4ed80a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:40.237 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.224+0000 7f4ee02a7700 1 Processor -- start 2019-12-05T05:03:40.237 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.224+0000 7f4ee02a7700 1 -- start start 2019-12-05T05:03:40.238 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.224+0000 7f4ee02a7700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f4ed807c450 0x7f4ed80a4bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:40.238 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.224+0000 7f4ee02a7700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ed81312a0 con 0x7f4ed80a9ad0 2019-12-05T05:03:40.238 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.224+0000 7f4ee02a7700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ed80a5550 con 0x7f4ed80a5120 2019-12-05T05:03:40.238 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.224+0000 7f4ee02a7700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ed807c880 con 0x7f4ed807c450 2019-12-05T05:03:40.238 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.224+0000 7f4edf045700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f4ed80a5120 0x7f4ed813ad90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:53344/0 (socket says 172.21.15.42:53344) 2019-12-05T05:03:40.238 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.224+0000 7f4edf045700 1 -- 172.21.15.42:0/2054031546 learned_addr learned my addr 172.21.15.42:0/2054031546 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:03:40.238 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.224+0000 7f4ede043700 1 --2- 172.21.15.42:0/2054031546 >> v2:172.21.15.42:3300/0 conn(0x7f4ed807c450 0x7f4ed80a4bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:40.239 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.228+0000 7f4ecf7fe700 1 -- 172.21.15.42:0/2054031546 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1168152682 0 0) 0x7f4ed80a5550 con 0x7f4ed80a5120 2019-12-05T05:03:40.239 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.228+0000 7f4ecf7fe700 1 -- 172.21.15.42:0/2054031546 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ec00030b0 con 0x7f4ed80a5120 2019-12-05T05:03:40.239 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.228+0000 7f4ede043700 1 -- 172.21.15.42:0/2054031546 >> v1:172.21.15.42:6789/0 conn(0x7f4ed80a5120 legacy=0x7f4ed813ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:40.239 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.228+0000 7f4ecf7fe700 1 -- 172.21.15.42:0/2054031546 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 687025192 0 0) 0x7f4ed81312a0 con 0x7f4ed80a9ad0 2019-12-05T05:03:40.239 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.228+0000 7f4ede043700 1 -- 172.21.15.42:0/2054031546 >> v1:172.21.15.40:6789/0 conn(0x7f4ed80a9ad0 legacy=0x7f4ed8130aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:40.239 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.228+0000 7f4ede043700 1 -- 172.21.15.42:0/2054031546 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ed8122fa0 con 0x7f4ed807c450 2019-12-05T05:03:40.239 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.228+0000 7f4ede043700 1 --2- 172.21.15.42:0/2054031546 >> v2:172.21.15.42:3300/0 conn(0x7f4ed807c450 0x7f4ed80a4bd0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f4ec8008790 tx=0x7f4ec8011f30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:40.240 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.228+0000 7f4ede043700 1 --2- 172.21.15.42:0/2054031546 >> v2:172.21.15.42:3300/0 conn(0x7f4ed807c450 0x7f4ed80a4bd0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f4ec8008790 tx=0x7f4ec8011f30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:40.240 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.228+0000 7f4ede043700 1 --2- 172.21.15.42:0/2054031546 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4ed807c450 0x7f4ed80a4bd0 secure :-1 s=READY pgs=106 cs=0 l=1 rx=0x7f4ec8008790 tx=0x7f4ec8011f30).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:40.240 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.228+0000 7f4ede043700 1 --2- 172.21.15.42:0/2054031546 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4ed807c450 0x7f4ed80a4bd0 secure :-1 s=THROTTLE_DONE pgs=106 cs=0 l=1 rx=0x7f4ec8008790 tx=0x7f4ec8011f30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:40.240 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.228+0000 7f4ecf7fe700 1 -- 172.21.15.42:0/2054031546 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4ec800f280 con 0x7f4ed807c450 2019-12-05T05:03:40.241 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.228+0000 7f4ede043700 1 --2- 172.21.15.42:0/2054031546 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4ed807c450 0x7f4ed80a4bd0 secure :-1 s=THROTTLE_DONE pgs=106 cs=0 l=1 rx=0x7f4ec8008790 tx=0x7f4ec8011f30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:40.241 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.228+0000 7f4ee02a7700 1 -- 172.21.15.42:0/2054031546 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ed807b610 con 0x7f4ed807c450 2019-12-05T05:03:40.241 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.228+0000 7f4ecf7fe700 1 -- 172.21.15.42:0/2054031546 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ec8008db0 con 0x7f4ed807c450 2019-12-05T05:03:40.241 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.228+0000 7f4ede043700 1 --2- 172.21.15.42:0/2054031546 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4ed807c450 0x7f4ed80a4bd0 secure :-1 s=THROTTLE_DONE pgs=106 cs=0 l=1 rx=0x7f4ec8008790 tx=0x7f4ec8011f30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:40.241 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.228+0000 7f4ecf7fe700 1 -- 172.21.15.42:0/2054031546 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4ec8005540 con 0x7f4ed807c450 2019-12-05T05:03:40.241 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.228+0000 7f4ee02a7700 1 -- 172.21.15.42:0/2054031546 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ed813c5a0 con 0x7f4ed807c450 2019-12-05T05:03:40.242 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.228+0000 7f4ede043700 1 --2- 172.21.15.42:0/2054031546 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4ed807c450 0x7f4ed80a4bd0 secure :-1 s=THROTTLE_DONE pgs=106 cs=0 l=1 rx=0x7f4ec8008790 tx=0x7f4ec8011f30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:40.242 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.228+0000 7f4ecf7fe700 1 -- 172.21.15.42:0/2054031546 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f4ec80057e0 con 0x7f4ed807c450 2019-12-05T05:03:40.242 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.228+0000 7f4ede043700 1 --2- 172.21.15.42:0/2054031546 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4ed807c450 0x7f4ed80a4bd0 secure :-1 s=THROTTLE_DONE pgs=106 cs=0 l=1 rx=0x7f4ec8008790 tx=0x7f4ec8011f30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:40.243 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.228+0000 7f4ecf7fe700 1 --2- 172.21.15.42:0/2054031546 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4ec003b190 0x7f4ec003d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:40.243 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.228+0000 7f4ecf7fe700 1 -- 172.21.15.42:0/2054031546 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f4ec804b900 con 0x7f4ed807c450 2019-12-05T05:03:40.243 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.228+0000 7f4ede844700 1 --2- 172.21.15.42:0/2054031546 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4ec003b190 0x7f4ec003d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:40.243 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.228+0000 7f4ede844700 1 --2- 172.21.15.42:0/2054031546 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4ec003b190 0x7f4ec003d6a0 crc :-1 s=READY pgs=63 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:40.244 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.232+0000 7f4ee02a7700 1 --2- 172.21.15.42:0/2054031546 >> [v2:172.21.15.42:6810/12118,v1:172.21.15.42:6811/12118] conn(0x7f4ed808f160 0x7f4ed8054260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:40.244 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.232+0000 7f4ee02a7700 1 -- 172.21.15.42:0/2054031546 --> [v2:172.21.15.42:6810/12118,v1:172.21.15.42:6811/12118] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4ed805d110 con 0x7f4ed808f160 2019-12-05T05:03:40.244 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.232+0000 7f4edf045700 1 --2- 172.21.15.42:0/2054031546 >> [v2:172.21.15.42:6810/12118,v1:172.21.15.42:6811/12118] conn(0x7f4ed808f160 0x7f4ed8054260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:40.245 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.232+0000 7f4edf045700 1 -- 172.21.15.42:0/2054031546 >> [v2:172.21.15.42:6810/12118,v1:172.21.15.42:6811/12118] conn(0x7f4ed808f160 msgr2=0x7f4ed8054260 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2019-12-05T05:03:40.245 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.232+0000 7f4edf045700 1 --2- 172.21.15.42:0/2054031546 >> [v2:172.21.15.42:6810/12118,v1:172.21.15.42:6811/12118] conn(0x7f4ed808f160 0x7f4ed8054260 crc :-1 s=READY pgs=10 cs=0 l=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:40.245 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.236+0000 7f4ecf7fe700 1 -- 172.21.15.42:0/2054031546 <== osd.0 v2:172.21.15.42:6810/12118 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23014 (crc 0 0 0) 0x7f4ed805d110 con 0x7f4ed808f160 2019-12-05T05:03:40.264 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.260+0000 7f4ee02a7700 1 -- 172.21.15.42:0/2054031546 --> [v2:172.21.15.42:6810/12118,v1:172.21.15.42:6811/12118] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f4ed805b8c0 con 0x7f4ed808f160 2019-12-05T05:03:40.265 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.260+0000 7f4ecf7fe700 1 -- 172.21.15.42:0/2054031546 <== osd.0 v2:172.21.15.42:6810/12118 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f4ed805b8c0 con 0x7f4ed808f160 2019-12-05T05:03:40.267 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.260+0000 7f4ee02a7700 1 -- 172.21.15.42:0/2054031546 >> [v2:172.21.15.42:6810/12118,v1:172.21.15.42:6811/12118] conn(0x7f4ed808f160 msgr2=0x7f4ed8054260 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:40.267 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.260+0000 7f4ee02a7700 1 --2- 172.21.15.42:0/2054031546 >> [v2:172.21.15.42:6810/12118,v1:172.21.15.42:6811/12118] conn(0x7f4ed808f160 0x7f4ed8054260 crc :-1 s=READY pgs=10 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:40.267 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.260+0000 7f4ee02a7700 1 -- 172.21.15.42:0/2054031546 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4ec003b190 msgr2=0x7f4ec003d6a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:40.267 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.260+0000 7f4ee02a7700 1 --2- 172.21.15.42:0/2054031546 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4ec003b190 0x7f4ec003d6a0 crc :-1 s=READY pgs=63 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:40.267 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.260+0000 7f4ee02a7700 1 -- 172.21.15.42:0/2054031546 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4ed807c450 msgr2=0x7f4ed80a4bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:40.267 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.260+0000 7f4ee02a7700 1 --2- 172.21.15.42:0/2054031546 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4ed807c450 0x7f4ed80a4bd0 secure :-1 s=READY pgs=106 cs=0 l=1 rx=0x7f4ec8008790 tx=0x7f4ec8011f30).stop 2019-12-05T05:03:40.268 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.260+0000 7f4edf045700 1 -- 172.21.15.42:0/2054031546 reap_dead start 2019-12-05T05:03:40.268 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.260+0000 7f4ee02a7700 1 -- 172.21.15.42:0/2054031546 shutdown_connections 2019-12-05T05:03:40.268 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.260+0000 7f4ee02a7700 1 --2- 172.21.15.42:0/2054031546 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4ed807c450 0x7f4ed80a4bd0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:40.268 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.260+0000 7f4ee02a7700 1 -- 172.21.15.42:0/2054031546 shutdown_connections 2019-12-05T05:03:40.268 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.260+0000 7f4ee02a7700 1 -- 172.21.15.42:0/2054031546 wait complete. 2019-12-05T05:03:40.268 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.260+0000 7f4ee02a7700 1 -- 172.21.15.42:0/2054031546 >> 172.21.15.42:0/2054031546 conn(0x7f4ed8066fe0 msgr2=0x7f4ed809f0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:40.276 INFO:teuthology.orchestra.run.smithi042.stdout:55834574855 2019-12-05T05:03:40.276 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2019-12-05T05:03:40.383 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.368+0000 7fd8982e8700 1 Processor -- start 2019-12-05T05:03:40.383 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.368+0000 7fd8982e8700 1 -- start start 2019-12-05T05:03:40.383 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.368+0000 7fd8982e8700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd8900a7740 0x7fd89008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:40.383 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.368+0000 7fd8982e8700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd89007e720 0x7fd89007eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:40.384 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.368+0000 7fd8982e8700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd89009f2e0 0x7fd89009f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:40.384 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.368+0000 7fd8982e8700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8900a0b00 con 0x7fd89007e720 2019-12-05T05:03:40.384 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.368+0000 7fd8982e8700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd89008e5b0 con 0x7fd8900a7740 2019-12-05T05:03:40.384 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.368+0000 7fd8982e8700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd89008e850 con 0x7fd89009f2e0 2019-12-05T05:03:40.384 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.368+0000 7fd896885700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd8900a7740 0x7fd89008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:40.384 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.368+0000 7fd896885700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd8900a7740 0x7fd89008df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:53316/0 (socket says 172.21.15.42:53316) 2019-12-05T05:03:40.384 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.368+0000 7fd896885700 1 -- 172.21.15.42:0/1487551308 learned_addr learned my addr 172.21.15.42:0/1487551308 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:03:40.385 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.368+0000 7fd896084700 1 --2- 172.21.15.42:0/1487551308 >> v2:172.21.15.40:3300/0 conn(0x7fd89007e720 0x7fd89007eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:40.385 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.368+0000 7fd897086700 1 --2- 172.21.15.42:0/1487551308 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd89009f2e0 0x7fd89009f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:40.385 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.372+0000 7fd896885700 1 -- 172.21.15.42:0/1487551308 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd89009f2e0 msgr2=0x7fd89009f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:03:40.385 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.372+0000 7fd896885700 1 --2- 172.21.15.42:0/1487551308 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd89009f2e0 0x7fd89009f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:03:40.385 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.372+0000 7fd896885700 1 -- 172.21.15.42:0/1487551308 >> v2:172.21.15.40:3300/0 conn(0x7fd89007e720 msgr2=0x7fd89007eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:03:40.385 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.372+0000 7fd896885700 1 --2- 172.21.15.42:0/1487551308 >> v2:172.21.15.40:3300/0 conn(0x7fd89007e720 0x7fd89007eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:03:40.385 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.372+0000 7fd896885700 1 -- 172.21.15.42:0/1487551308 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd89007f0a0 con 0x7fd8900a7740 2019-12-05T05:03:40.386 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.372+0000 7fd896885700 1 --2- 172.21.15.42:0/1487551308 >> v2:172.21.15.42:3300/0 conn(0x7fd8900a7740 0x7fd89008df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fd8900a0f50 tx=0x7fd88c00fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:40.386 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.372+0000 7fd896885700 1 --2- 172.21.15.42:0/1487551308 >> v2:172.21.15.42:3300/0 conn(0x7fd8900a7740 0x7fd89008df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fd8900a0f50 tx=0x7fd88c00fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:40.386 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.372+0000 7fd896885700 1 --2- 172.21.15.42:0/1487551308 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd8900a7740 0x7fd89008df80 secure :-1 s=READY pgs=107 cs=0 l=1 rx=0x7fd8900a0f50 tx=0x7fd88c00fdb0).ready entity=mon.1 client_cookie=488129ef0dccbd0e server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:40.387 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.372+0000 7fd896885700 1 --2- 172.21.15.42:0/1487551308 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd8900a7740 0x7fd89008df80 secure :-1 s=THROTTLE_DONE pgs=107 cs=0 l=1 rx=0x7fd8900a0f50 tx=0x7fd88c00fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:40.387 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.372+0000 7fd895883700 1 -- 172.21.15.42:0/1487551308 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd88c008c90 con 0x7fd8900a7740 2019-12-05T05:03:40.387 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.372+0000 7fd896885700 1 --2- 172.21.15.42:0/1487551308 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd8900a7740 0x7fd89008df80 secure :-1 s=THROTTLE_DONE pgs=107 cs=0 l=1 rx=0x7fd8900a0f50 tx=0x7fd88c00fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:40.387 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.372+0000 7fd896885700 1 --2- 172.21.15.42:0/1487551308 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd8900a7740 0x7fd89008df80 secure :-1 s=THROTTLE_DONE pgs=107 cs=0 l=1 rx=0x7fd8900a0f50 tx=0x7fd88c00fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:40.387 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.372+0000 7fd895883700 1 -- 172.21.15.42:0/1487551308 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd88c0088b0 con 0x7fd8900a7740 2019-12-05T05:03:40.387 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.372+0000 7fd895883700 1 -- 172.21.15.42:0/1487551308 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd88c011b50 con 0x7fd8900a7740 2019-12-05T05:03:40.387 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.372+0000 7fd8982e8700 1 -- 172.21.15.42:0/1487551308 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd8900a7740 msgr2=0x7fd89008df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:40.388 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.372+0000 7fd8982e8700 1 --2- 172.21.15.42:0/1487551308 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd8900a7740 0x7fd89008df80 secure :-1 s=READY pgs=107 cs=0 l=1 rx=0x7fd8900a0f50 tx=0x7fd88c00fdb0).stop 2019-12-05T05:03:40.388 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.372+0000 7fd8982e8700 1 -- 172.21.15.42:0/1487551308 shutdown_connections 2019-12-05T05:03:40.388 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.372+0000 7fd8982e8700 1 --2- 172.21.15.42:0/1487551308 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd89009f2e0 0x7fd89009f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:03:40.388 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.372+0000 7fd8982e8700 1 --2- 172.21.15.42:0/1487551308 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd8900a7740 0x7fd89008df80 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:40.389 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.372+0000 7fd8982e8700 1 --2- 172.21.15.42:0/1487551308 >> v2:172.21.15.40:3300/0 conn(0x7fd89007e720 0x7fd89007eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:03:40.389 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.372+0000 7fd8982e8700 1 -- 172.21.15.42:0/1487551308 shutdown_connections 2019-12-05T05:03:40.389 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.372+0000 7fd8982e8700 1 -- 172.21.15.42:0/1487551308 wait complete. 2019-12-05T05:03:40.389 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.372+0000 7fd8982e8700 1 -- 172.21.15.42:0/1487551308 >> 172.21.15.42:0/1487551308 conn(0x7fd8900aabe0 msgr2=0x7fd8900a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:40.389 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.376+0000 7fd8982e8700 1 Processor -- start 2019-12-05T05:03:40.389 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.376+0000 7fd8982e8700 1 -- start start 2019-12-05T05:03:40.390 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.376+0000 7fd8982e8700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd89011c020 0x7fd8900a6ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:40.390 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.376+0000 7fd8982e8700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd8900742e0 0x7fd890076720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:40.390 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.376+0000 7fd8982e8700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd89011c7d0 con 0x7fd8900a5b90 2019-12-05T05:03:40.390 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.376+0000 7fd8982e8700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd89011cba0 con 0x7fd89011c020 2019-12-05T05:03:40.390 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.376+0000 7fd8982e8700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8900a6180 con 0x7fd8900742e0 2019-12-05T05:03:40.390 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.376+0000 7fd896885700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fd8900a5b90 0x7fd89011b820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:53358/0 (socket says 172.21.15.42:53358) 2019-12-05T05:03:40.391 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.376+0000 7fd896885700 1 -- 172.21.15.42:0/3880352398 learned_addr learned my addr 172.21.15.42:0/3880352398 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:03:40.391 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.376+0000 7fd896084700 1 --2- 172.21.15.42:0/3880352398 >> v2:172.21.15.42:3300/0 conn(0x7fd89011c020 0x7fd8900a6ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:40.391 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.376+0000 7fd897086700 1 --2- 172.21.15.42:0/3880352398 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd8900742e0 0x7fd890076720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:40.391 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.376+0000 7fd8837fe700 1 -- 172.21.15.42:0/3880352398 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2818740452 0 0) 0x7fd89011c7d0 con 0x7fd8900a5b90 2019-12-05T05:03:40.391 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.376+0000 7fd8837fe700 1 -- 172.21.15.42:0/3880352398 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8780030b0 con 0x7fd8900a5b90 2019-12-05T05:03:40.391 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.376+0000 7fd896084700 1 -- 172.21.15.42:0/3880352398 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd8900742e0 msgr2=0x7fd890076720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:40.391 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.376+0000 7fd896084700 1 --2- 172.21.15.42:0/3880352398 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd8900742e0 0x7fd890076720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:40.392 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.376+0000 7fd896084700 1 -- 172.21.15.42:0/3880352398 >> v1:172.21.15.42:6789/0 conn(0x7fd8900a5b90 legacy=0x7fd89011b820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:40.392 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.376+0000 7fd896084700 1 -- 172.21.15.42:0/3880352398 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd890077c80 con 0x7fd89011c020 2019-12-05T05:03:40.392 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.376+0000 7fd896084700 1 --2- 172.21.15.42:0/3880352398 >> v2:172.21.15.42:3300/0 conn(0x7fd89011c020 0x7fd8900a6ca0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fd890117a40 tx=0x7fd884010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:40.392 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.376+0000 7fd896084700 1 --2- 172.21.15.42:0/3880352398 >> v2:172.21.15.42:3300/0 conn(0x7fd89011c020 0x7fd8900a6ca0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fd890117a40 tx=0x7fd884010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:40.392 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.376+0000 7fd896084700 1 --2- 172.21.15.42:0/3880352398 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd89011c020 0x7fd8900a6ca0 secure :-1 s=READY pgs=109 cs=0 l=1 rx=0x7fd890117a40 tx=0x7fd884010b50).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:40.392 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.376+0000 7fd896084700 1 --2- 172.21.15.42:0/3880352398 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd89011c020 0x7fd8900a6ca0 secure :-1 s=THROTTLE_DONE pgs=109 cs=0 l=1 rx=0x7fd890117a40 tx=0x7fd884010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:40.392 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.376+0000 7fd8837fe700 1 -- 172.21.15.42:0/3880352398 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd88400e820 con 0x7fd89011c020 2019-12-05T05:03:40.393 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.376+0000 7fd896084700 1 --2- 172.21.15.42:0/3880352398 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd89011c020 0x7fd8900a6ca0 secure :-1 s=THROTTLE_DONE pgs=109 cs=0 l=1 rx=0x7fd890117a40 tx=0x7fd884010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:40.393 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.376+0000 7fd896084700 1 --2- 172.21.15.42:0/3880352398 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd89011c020 0x7fd8900a6ca0 secure :-1 s=THROTTLE_DONE pgs=109 cs=0 l=1 rx=0x7fd890117a40 tx=0x7fd884010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:40.393 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.376+0000 7fd8837fe700 1 -- 172.21.15.42:0/3880352398 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd88400e440 con 0x7fd89011c020 2019-12-05T05:03:40.393 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.376+0000 7fd8982e8700 1 -- 172.21.15.42:0/3880352398 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd890076c70 con 0x7fd89011c020 2019-12-05T05:03:40.393 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.376+0000 7fd8837fe700 1 -- 172.21.15.42:0/3880352398 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd8840121c0 con 0x7fd89011c020 2019-12-05T05:03:40.394 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.376+0000 7fd8982e8700 1 -- 172.21.15.42:0/3880352398 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd890077220 con 0x7fd89011c020 2019-12-05T05:03:40.395 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.380+0000 7fd896084700 1 --2- 172.21.15.42:0/3880352398 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd89011c020 0x7fd8900a6ca0 secure :-1 s=THROTTLE_DONE pgs=109 cs=0 l=1 rx=0x7fd890117a40 tx=0x7fd884010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:40.395 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.380+0000 7fd8837fe700 1 -- 172.21.15.42:0/3880352398 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fd884005540 con 0x7fd89011c020 2019-12-05T05:03:40.396 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.380+0000 7fd896084700 1 --2- 172.21.15.42:0/3880352398 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd89011c020 0x7fd8900a6ca0 secure :-1 s=THROTTLE_DONE pgs=109 cs=0 l=1 rx=0x7fd890117a40 tx=0x7fd884010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:40.396 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.380+0000 7fd8837fe700 1 --2- 172.21.15.42:0/3880352398 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd87803b260 0x7fd87803d770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:40.396 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.380+0000 7fd8837fe700 1 -- 172.21.15.42:0/3880352398 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6031+0+0 (secure 0 0 0) 0x7fd88404ae00 con 0x7fd89011c020 2019-12-05T05:03:40.396 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.380+0000 7fd896885700 1 --2- 172.21.15.42:0/3880352398 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd87803b260 0x7fd87803d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:40.397 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.380+0000 7fd896885700 1 -- 172.21.15.42:0/3880352398 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd87803b260 msgr2=0x7fd87803d770 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2019-12-05T05:03:40.397 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.380+0000 7fd896885700 1 -- 172.21.15.42:0/3880352398 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd87803b260 msgr2=0x7fd87803d770 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2019-12-05T05:03:40.397 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.380+0000 7fd896885700 1 --2- 172.21.15.42:0/3880352398 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd87803b260 0x7fd87803d770 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-12-05T05:03:40.397 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.380+0000 7fd896885700 1 --2- 172.21.15.42:0/3880352398 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd87803b260 0x7fd87803d770 crc :-1 s=START_CONNECT pgs=0 cs=0 l=1 rx=0 tx=0)._fault waiting 0.200000 2019-12-05T05:03:40.397 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.384+0000 7fd8982e8700 1 --2- 172.21.15.42:0/3880352398 >> [v2:172.21.15.42:6826/12119,v1:172.21.15.42:6827/12119] conn(0x7fd89009f3f0 0x7fd890054260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:40.397 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.384+0000 7fd8982e8700 1 -- 172.21.15.42:0/3880352398 --> [v2:172.21.15.42:6826/12119,v1:172.21.15.42:6827/12119] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd8900a59a0 con 0x7fd89009f3f0 2019-12-05T05:03:40.398 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.384+0000 7fd897086700 1 --2- 172.21.15.42:0/3880352398 >> [v2:172.21.15.42:6826/12119,v1:172.21.15.42:6827/12119] conn(0x7fd89009f3f0 0x7fd890054260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:40.398 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.384+0000 7fd897086700 1 -- 172.21.15.42:0/3880352398 >> [v2:172.21.15.42:6826/12119,v1:172.21.15.42:6827/12119] conn(0x7fd89009f3f0 msgr2=0x7fd890054260 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2019-12-05T05:03:40.398 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.384+0000 7fd897086700 1 --2- 172.21.15.42:0/3880352398 >> [v2:172.21.15.42:6826/12119,v1:172.21.15.42:6827/12119] conn(0x7fd89009f3f0 0x7fd890054260 crc :-1 s=READY pgs=7 cs=0 l=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:40.399 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.388+0000 7fd8837fe700 1 -- 172.21.15.42:0/3880352398 <== osd.1 v2:172.21.15.42:6826/12119 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23014 (crc 0 0 0) 0x7fd888007210 con 0x7fd89009f3f0 2019-12-05T05:03:40.418 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.412+0000 7fd8982e8700 1 -- 172.21.15.42:0/3880352398 --> [v2:172.21.15.42:6826/12119,v1:172.21.15.42:6827/12119] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fd89005d110 con 0x7fd89009f3f0 2019-12-05T05:03:40.418 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.412+0000 7fd8837fe700 1 -- 172.21.15.42:0/3880352398 <== osd.1 v2:172.21.15.42:6826/12119 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fd89005d110 con 0x7fd89009f3f0 2019-12-05T05:03:40.420 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.412+0000 7fd8982e8700 1 -- 172.21.15.42:0/3880352398 >> [v2:172.21.15.42:6826/12119,v1:172.21.15.42:6827/12119] conn(0x7fd89009f3f0 msgr2=0x7fd890054260 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:40.421 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.412+0000 7fd8982e8700 1 --2- 172.21.15.42:0/3880352398 >> [v2:172.21.15.42:6826/12119,v1:172.21.15.42:6827/12119] conn(0x7fd89009f3f0 0x7fd890054260 crc :-1 s=READY pgs=7 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:40.421 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.412+0000 7fd8982e8700 1 -- 172.21.15.42:0/3880352398 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd87803b260 msgr2=0x7fd87803d770 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2019-12-05T05:03:40.421 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.412+0000 7fd8982e8700 1 --2- 172.21.15.42:0/3880352398 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd87803b260 0x7fd87803d770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:40.421 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.412+0000 7fd8982e8700 1 -- 172.21.15.42:0/3880352398 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd89011c020 msgr2=0x7fd8900a6ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:40.421 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.412+0000 7fd8982e8700 1 --2- 172.21.15.42:0/3880352398 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd89011c020 0x7fd8900a6ca0 secure :-1 s=READY pgs=109 cs=0 l=1 rx=0x7fd890117a40 tx=0x7fd884010b50).stop 2019-12-05T05:03:40.421 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.412+0000 7fd897086700 1 -- 172.21.15.42:0/3880352398 reap_dead start 2019-12-05T05:03:40.422 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.412+0000 7fd8982e8700 1 -- 172.21.15.42:0/3880352398 shutdown_connections 2019-12-05T05:03:40.422 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.412+0000 7fd8982e8700 1 --2- 172.21.15.42:0/3880352398 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd89011c020 0x7fd8900a6ca0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:40.422 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.412+0000 7fd8982e8700 1 -- 172.21.15.42:0/3880352398 shutdown_connections 2019-12-05T05:03:40.422 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.416+0000 7fd8982e8700 1 -- 172.21.15.42:0/3880352398 wait complete. 2019-12-05T05:03:40.422 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.416+0000 7fd8982e8700 1 -- 172.21.15.42:0/3880352398 >> 172.21.15.42:0/3880352398 conn(0x7fd8900aabe0 msgr2=0x7fd89011d4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:40.431 INFO:teuthology.orchestra.run.smithi042.stdout:55834574853 2019-12-05T05:03:40.431 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2019-12-05T05:03:40.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.524+0000 7f08af1ed700 1 Processor -- start 2019-12-05T05:03:40.541 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.524+0000 7f08af1ed700 1 -- start start 2019-12-05T05:03:40.541 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.524+0000 7f08af1ed700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f08a808df80 0x7f08a809f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:40.541 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.524+0000 7f08af1ed700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f08a809f830 0x7f08a814efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:40.541 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.524+0000 7f08af1ed700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08a809fc60 con 0x7f08a80a7740 2019-12-05T05:03:40.541 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.524+0000 7f08af1ed700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08a80a0b00 con 0x7f08a808df80 2019-12-05T05:03:40.541 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.524+0000 7f08af1ed700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08a808e5b0 con 0x7f08a809f830 2019-12-05T05:03:40.542 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.524+0000 7f08adf8b700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f08a809f830 0x7f08a814efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:40.542 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.524+0000 7f08acf89700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f08a808df80 0x7f08a809f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:40.542 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.524+0000 7f08adf8b700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f08a809f830 0x7f08a814efe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:53800/0 (socket says 172.21.15.42:53800) 2019-12-05T05:03:40.542 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.524+0000 7f08adf8b700 1 -- 172.21.15.42:0/2421277783 learned_addr learned my addr 172.21.15.42:0/2421277783 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:03:40.542 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.528+0000 7f08acf89700 1 -- 172.21.15.42:0/2421277783 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f08a809f830 msgr2=0x7f08a814efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:03:40.542 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.528+0000 7f08acf89700 1 --2- 172.21.15.42:0/2421277783 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f08a809f830 0x7f08a814efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:03:40.543 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.528+0000 7f08acf89700 1 -- 172.21.15.42:0/2421277783 >> v1:172.21.15.40:6789/0 conn(0x7f08a80a7740 legacy=0x7f08a807e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:40.543 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.528+0000 7f08acf89700 1 -- 172.21.15.42:0/2421277783 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08a80a0710 con 0x7f08a808df80 2019-12-05T05:03:40.543 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.528+0000 7f08adf8b700 1 --2- 172.21.15.42:0/2421277783 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f08a809f830 0x7f08a814efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:03:40.543 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.528+0000 7f08acf89700 1 --2- 172.21.15.42:0/2421277783 >> v2:172.21.15.40:3300/0 conn(0x7f08a808df80 0x7f08a809f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f0898008790 tx=0x7f089800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:40.544 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.528+0000 7f08acf89700 1 --2- 172.21.15.42:0/2421277783 >> v2:172.21.15.40:3300/0 conn(0x7f08a808df80 0x7f08a809f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f0898008790 tx=0x7f089800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:40.545 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.528+0000 7f08acf89700 1 --2- 172.21.15.42:0/2421277783 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f08a808df80 0x7f08a809f2e0 secure :-1 s=READY pgs=95 cs=0 l=1 rx=0x7f0898008790 tx=0x7f089800fc70).ready entity=mon.0 client_cookie=63077271efb8d721 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:40.545 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.528+0000 7f08acf89700 1 --2- 172.21.15.42:0/2421277783 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f08a808df80 0x7f08a809f2e0 secure :-1 s=THROTTLE_DONE pgs=95 cs=0 l=1 rx=0x7f0898008790 tx=0x7f089800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:40.545 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.528+0000 7f08acf89700 1 --2- 172.21.15.42:0/2421277783 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f08a808df80 0x7f08a809f2e0 secure :-1 s=THROTTLE_DONE pgs=95 cs=0 l=1 rx=0x7f0898008790 tx=0x7f089800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:40.545 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.528+0000 7f08acf89700 1 --2- 172.21.15.42:0/2421277783 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f08a808df80 0x7f08a809f2e0 secure :-1 s=THROTTLE_DONE pgs=95 cs=0 l=1 rx=0x7f0898008790 tx=0x7f089800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:40.546 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.528+0000 7f089ffff700 1 -- 172.21.15.42:0/2421277783 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0898008bb0 con 0x7f08a808df80 2019-12-05T05:03:40.546 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.528+0000 7f089ffff700 1 -- 172.21.15.42:0/2421277783 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08980087d0 con 0x7f08a808df80 2019-12-05T05:03:40.546 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.528+0000 7f089ffff700 1 -- 172.21.15.42:0/2421277783 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0898011860 con 0x7f08a808df80 2019-12-05T05:03:40.546 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.528+0000 7f08af1ed700 1 -- 172.21.15.42:0/2421277783 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f08a808df80 msgr2=0x7f08a809f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:40.546 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.528+0000 7f08af1ed700 1 --2- 172.21.15.42:0/2421277783 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f08a808df80 0x7f08a809f2e0 secure :-1 s=READY pgs=95 cs=0 l=1 rx=0x7f0898008790 tx=0x7f089800fc70).stop 2019-12-05T05:03:40.546 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.528+0000 7f08af1ed700 1 -- 172.21.15.42:0/2421277783 shutdown_connections 2019-12-05T05:03:40.547 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.528+0000 7f08af1ed700 1 --2- 172.21.15.42:0/2421277783 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f08a809f830 0x7f08a814efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:03:40.547 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.528+0000 7f08af1ed700 1 --2- 172.21.15.42:0/2421277783 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f08a808df80 0x7f08a809f2e0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:40.547 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.528+0000 7f08af1ed700 1 -- 172.21.15.42:0/2421277783 shutdown_connections 2019-12-05T05:03:40.547 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.528+0000 7f08af1ed700 1 -- 172.21.15.42:0/2421277783 wait complete. 2019-12-05T05:03:40.547 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.528+0000 7f08af1ed700 1 -- 172.21.15.42:0/2421277783 >> 172.21.15.42:0/2421277783 conn(0x7f08a80aabe0 msgr2=0x7f08a80a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:40.547 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.528+0000 7f08af1ed700 1 Processor -- start 2019-12-05T05:03:40.547 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.528+0000 7f08af1ed700 1 -- start start 2019-12-05T05:03:40.548 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.532+0000 7f08af1ed700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f08a80a1a20 0x7f08a80a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:40.548 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.532+0000 7f08af1ed700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f08a81128c0 0x7f08a80a3d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:40.548 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.532+0000 7f08af1ed700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08a8144e80 con 0x7f08a8078800 2019-12-05T05:03:40.548 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.532+0000 7f08af1ed700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08a8144940 con 0x7f08a80a1a20 2019-12-05T05:03:40.548 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.532+0000 7f08af1ed700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08a8144be0 con 0x7f08a81128c0 2019-12-05T05:03:40.548 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.532+0000 7f08acf89700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f08a81128c0 0x7f08a80a3d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:40.548 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.532+0000 7f08acf89700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f08a81128c0 0x7f08a80a3d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:53338/0 (socket says 172.21.15.42:53338) 2019-12-05T05:03:40.549 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.532+0000 7f08acf89700 1 -- 172.21.15.42:0/1795683095 learned_addr learned my addr 172.21.15.42:0/1795683095 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:03:40.549 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.532+0000 7f08adf8b700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f08a8078800 0x7f08a808eb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:53378/0 (socket says 172.21.15.42:53378) 2019-12-05T05:03:40.549 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.532+0000 7f08ad78a700 1 --2- 172.21.15.42:0/1795683095 >> v2:172.21.15.40:3300/0 conn(0x7f08a80a1a20 0x7f08a80a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:40.549 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.532+0000 7f08acf89700 1 -- 172.21.15.42:0/1795683095 >> v2:172.21.15.40:3300/0 conn(0x7f08a80a1a20 msgr2=0x7f08a80a37f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:40.549 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.532+0000 7f08acf89700 1 --2- 172.21.15.42:0/1795683095 >> v2:172.21.15.40:3300/0 conn(0x7f08a80a1a20 0x7f08a80a37f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:40.549 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.532+0000 7f08acf89700 1 -- 172.21.15.42:0/1795683095 >> v1:172.21.15.42:6789/0 conn(0x7f08a8078800 legacy=0x7f08a808eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:40.550 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.532+0000 7f08acf89700 1 -- 172.21.15.42:0/1795683095 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08a8145220 con 0x7f08a81128c0 2019-12-05T05:03:40.550 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.532+0000 7f08ad78a700 1 --2- 172.21.15.42:0/1795683095 >> v2:172.21.15.40:3300/0 conn(0x7f08a80a1a20 0x7f08a80a37f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T05:03:40.550 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.532+0000 7f08acf89700 1 --2- 172.21.15.42:0/1795683095 >> v2:172.21.15.42:3300/0 conn(0x7f08a81128c0 0x7f08a80a3d40 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f0898008790 tx=0x7f089800f3d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:40.550 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.532+0000 7f08acf89700 1 --2- 172.21.15.42:0/1795683095 >> v2:172.21.15.42:3300/0 conn(0x7f08a81128c0 0x7f08a80a3d40 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f0898008790 tx=0x7f089800f3d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:40.550 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.532+0000 7f08acf89700 1 --2- 172.21.15.42:0/1795683095 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f08a81128c0 0x7f08a80a3d40 secure :-1 s=READY pgs=111 cs=0 l=1 rx=0x7f0898008790 tx=0x7f089800f3d0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:40.550 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.532+0000 7f08acf89700 1 --2- 172.21.15.42:0/1795683095 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f08a81128c0 0x7f08a80a3d40 secure :-1 s=THROTTLE_DONE pgs=111 cs=0 l=1 rx=0x7f0898008790 tx=0x7f089800f3d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:40.550 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.532+0000 7f08acf89700 1 --2- 172.21.15.42:0/1795683095 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f08a81128c0 0x7f08a80a3d40 secure :-1 s=THROTTLE_DONE pgs=111 cs=0 l=1 rx=0x7f0898008790 tx=0x7f089800f3d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:40.551 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.532+0000 7f089dffb700 1 -- 172.21.15.42:0/1795683095 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0898007350 con 0x7f08a81128c0 2019-12-05T05:03:40.551 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.532+0000 7f08acf89700 1 --2- 172.21.15.42:0/1795683095 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f08a81128c0 0x7f08a80a3d40 secure :-1 s=THROTTLE_DONE pgs=111 cs=0 l=1 rx=0x7f0898008790 tx=0x7f089800f3d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:40.551 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.532+0000 7f089dffb700 1 -- 172.21.15.42:0/1795683095 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0898008db0 con 0x7f08a81128c0 2019-12-05T05:03:40.551 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.532+0000 7f08af1ed700 1 -- 172.21.15.42:0/1795683095 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08a8116cd0 con 0x7f08a81128c0 2019-12-05T05:03:40.551 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.532+0000 7f089dffb700 1 -- 172.21.15.42:0/1795683095 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0898011c60 con 0x7f08a81128c0 2019-12-05T05:03:40.551 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.532+0000 7f08af1ed700 1 -- 172.21.15.42:0/1795683095 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08a8078dd0 con 0x7f08a81128c0 2019-12-05T05:03:40.551 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.532+0000 7f08acf89700 1 --2- 172.21.15.42:0/1795683095 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f08a81128c0 0x7f08a80a3d40 secure :-1 s=THROTTLE_DONE pgs=111 cs=0 l=1 rx=0x7f0898008790 tx=0x7f089800f3d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:40.552 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.536+0000 7f089dffb700 1 -- 172.21.15.42:0/1795683095 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f0898005540 con 0x7f08a81128c0 2019-12-05T05:03:40.552 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.536+0000 7f08acf89700 1 --2- 172.21.15.42:0/1795683095 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f08a81128c0 0x7f08a80a3d40 secure :-1 s=THROTTLE_DONE pgs=111 cs=0 l=1 rx=0x7f0898008790 tx=0x7f089800f3d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:40.552 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.536+0000 7f089dffb700 1 --2- 172.21.15.42:0/1795683095 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f088803ade0 0x7f088803d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:40.552 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.536+0000 7f089dffb700 1 -- 172.21.15.42:0/1795683095 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f089804b940 con 0x7f08a81128c0 2019-12-05T05:03:40.552 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.536+0000 7f08ad78a700 1 --2- 172.21.15.42:0/1795683095 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f088803ade0 0x7f088803d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:40.552 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.536+0000 7f08ad78a700 1 --2- 172.21.15.42:0/1795683095 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f088803ade0 0x7f088803d2f0 crc :-1 s=READY pgs=65 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:40.553 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.540+0000 7f08af1ed700 1 --2- 172.21.15.42:0/1795683095 >> [v2:172.21.15.42:6802/12117,v1:172.21.15.42:6803/12117] conn(0x7f08a8111db0 0x7f08a8054260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:40.553 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.540+0000 7f08af1ed700 1 -- 172.21.15.42:0/1795683095 --> [v2:172.21.15.42:6802/12117,v1:172.21.15.42:6803/12117] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f08a805d110 con 0x7f08a8111db0 2019-12-05T05:03:40.554 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.540+0000 7f08adf8b700 1 --2- 172.21.15.42:0/1795683095 >> [v2:172.21.15.42:6802/12117,v1:172.21.15.42:6803/12117] conn(0x7f08a8111db0 0x7f08a8054260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:40.554 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.540+0000 7f08adf8b700 1 -- 172.21.15.42:0/1795683095 >> [v2:172.21.15.42:6802/12117,v1:172.21.15.42:6803/12117] conn(0x7f08a8111db0 msgr2=0x7f08a8054260 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2019-12-05T05:03:40.554 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.540+0000 7f08adf8b700 1 --2- 172.21.15.42:0/1795683095 >> [v2:172.21.15.42:6802/12117,v1:172.21.15.42:6803/12117] conn(0x7f08a8111db0 0x7f08a8054260 crc :-1 s=READY pgs=7 cs=0 l=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:40.554 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.544+0000 7f089dffb700 1 -- 172.21.15.42:0/1795683095 <== osd.2 v2:172.21.15.42:6802/12117 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23014 (crc 0 0 0) 0x7f08a805d110 con 0x7f08a8111db0 2019-12-05T05:03:40.571 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.564+0000 7f08af1ed700 1 -- 172.21.15.42:0/1795683095 --> [v2:172.21.15.42:6802/12117,v1:172.21.15.42:6803/12117] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f08a80a71c0 con 0x7f08a8111db0 2019-12-05T05:03:40.572 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.568+0000 7f089dffb700 1 -- 172.21.15.42:0/1795683095 <== osd.2 v2:172.21.15.42:6802/12117 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f08a80a71c0 con 0x7f08a8111db0 2019-12-05T05:03:40.574 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.568+0000 7f08af1ed700 1 -- 172.21.15.42:0/1795683095 >> [v2:172.21.15.42:6802/12117,v1:172.21.15.42:6803/12117] conn(0x7f08a8111db0 msgr2=0x7f08a8054260 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:40.574 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.568+0000 7f08af1ed700 1 --2- 172.21.15.42:0/1795683095 >> [v2:172.21.15.42:6802/12117,v1:172.21.15.42:6803/12117] conn(0x7f08a8111db0 0x7f08a8054260 crc :-1 s=READY pgs=7 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:40.574 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.568+0000 7f08af1ed700 1 -- 172.21.15.42:0/1795683095 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f088803ade0 msgr2=0x7f088803d2f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:40.574 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.568+0000 7f08af1ed700 1 --2- 172.21.15.42:0/1795683095 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f088803ade0 0x7f088803d2f0 crc :-1 s=READY pgs=65 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:40.574 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.568+0000 7f08af1ed700 1 -- 172.21.15.42:0/1795683095 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f08a81128c0 msgr2=0x7f08a80a3d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:40.574 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.568+0000 7f08af1ed700 1 --2- 172.21.15.42:0/1795683095 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f08a81128c0 0x7f08a80a3d40 secure :-1 s=READY pgs=111 cs=0 l=1 rx=0x7f0898008790 tx=0x7f089800f3d0).stop 2019-12-05T05:03:40.575 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.568+0000 7f08adf8b700 1 -- 172.21.15.42:0/1795683095 reap_dead start 2019-12-05T05:03:40.575 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.568+0000 7f08af1ed700 1 -- 172.21.15.42:0/1795683095 shutdown_connections 2019-12-05T05:03:40.575 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.568+0000 7f08af1ed700 1 --2- 172.21.15.42:0/1795683095 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f08a81128c0 0x7f08a80a3d40 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:40.575 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.568+0000 7f08af1ed700 1 -- 172.21.15.42:0/1795683095 shutdown_connections 2019-12-05T05:03:40.575 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.568+0000 7f08af1ed700 1 -- 172.21.15.42:0/1795683095 wait complete. 2019-12-05T05:03:40.575 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.568+0000 7f08af1ed700 1 -- 172.21.15.42:0/1795683095 >> 172.21.15.42:0/1795683095 conn(0x7f08a808fda0 msgr2=0x7f08a807d760 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:40.589 INFO:teuthology.orchestra.run.smithi042.stdout:55834574853 2019-12-05T05:03:40.590 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2019-12-05T05:03:40.693 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.684+0000 7f896d748700 1 Processor -- start 2019-12-05T05:03:40.694 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.684+0000 7f896d748700 1 -- start start 2019-12-05T05:03:40.694 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.684+0000 7f896d748700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f89680a7740 0x7f896808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:40.694 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.684+0000 7f896d748700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89680a0710 con 0x7f896807e720 2019-12-05T05:03:40.694 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.684+0000 7f896d748700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89680a9ba0 con 0x7f896809fae0 2019-12-05T05:03:40.694 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.684+0000 7f896d748700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89680a0b00 con 0x7f89680a7740 2019-12-05T05:03:40.695 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.684+0000 7f89677fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f89680a7740 0x7f896808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:40.695 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.684+0000 7f89677fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f89680a7740 0x7f896808df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:53346/0 (socket says 172.21.15.42:53346) 2019-12-05T05:03:40.695 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.684+0000 7f89677fe700 1 -- 172.21.15.42:0/283917952 learned_addr learned my addr 172.21.15.42:0/283917952 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:03:40.695 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.684+0000 7f89677fe700 1 -- 172.21.15.42:0/283917952 >> v1:172.21.15.42:6789/0 conn(0x7f896809fae0 legacy=0x7f896814efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:03:40.695 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.684+0000 7f89677fe700 1 -- 172.21.15.42:0/283917952 >> v1:172.21.15.40:6789/0 conn(0x7f896807e720 legacy=0x7f896809f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:40.695 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.684+0000 7f89677fe700 1 -- 172.21.15.42:0/283917952 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f896807eb50 con 0x7f89680a7740 2019-12-05T05:03:40.696 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.684+0000 7f89677fe700 1 --2- 172.21.15.42:0/283917952 >> v2:172.21.15.42:3300/0 conn(0x7f89680a7740 0x7f896808df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f8958008790 tx=0x7f895800fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:40.696 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.684+0000 7f89677fe700 1 --2- 172.21.15.42:0/283917952 >> v2:172.21.15.42:3300/0 conn(0x7f89680a7740 0x7f896808df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f8958008790 tx=0x7f895800fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:40.696 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.684+0000 7f89677fe700 1 --2- 172.21.15.42:0/283917952 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f89680a7740 0x7f896808df80 secure :-1 s=READY pgs=113 cs=0 l=1 rx=0x7f8958008790 tx=0x7f895800fcb0).ready entity=mon.1 client_cookie=fb0ec3dba0820541 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:40.696 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.684+0000 7f89677fe700 1 --2- 172.21.15.42:0/283917952 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f89680a7740 0x7f896808df80 secure :-1 s=THROTTLE_DONE pgs=113 cs=0 l=1 rx=0x7f8958008790 tx=0x7f895800fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:40.697 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.684+0000 7f89677fe700 1 --2- 172.21.15.42:0/283917952 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f89680a7740 0x7f896808df80 secure :-1 s=THROTTLE_DONE pgs=113 cs=0 l=1 rx=0x7f8958008790 tx=0x7f895800fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:40.697 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.684+0000 7f8966ffd700 1 -- 172.21.15.42:0/283917952 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8958011520 con 0x7f89680a7740 2019-12-05T05:03:40.697 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.684+0000 7f89677fe700 1 --2- 172.21.15.42:0/283917952 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f89680a7740 0x7f896808df80 secure :-1 s=THROTTLE_DONE pgs=113 cs=0 l=1 rx=0x7f8958008790 tx=0x7f895800fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:40.697 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.684+0000 7f8966ffd700 1 -- 172.21.15.42:0/283917952 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8958008ca0 con 0x7f89680a7740 2019-12-05T05:03:40.698 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.684+0000 7f8966ffd700 1 -- 172.21.15.42:0/283917952 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8958011b00 con 0x7f89680a7740 2019-12-05T05:03:40.698 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.684+0000 7f896d748700 1 -- 172.21.15.42:0/283917952 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f89680a7740 msgr2=0x7f896808df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:40.698 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.684+0000 7f896d748700 1 --2- 172.21.15.42:0/283917952 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f89680a7740 0x7f896808df80 secure :-1 s=READY pgs=113 cs=0 l=1 rx=0x7f8958008790 tx=0x7f895800fcb0).stop 2019-12-05T05:03:40.698 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.688+0000 7f896d748700 1 -- 172.21.15.42:0/283917952 shutdown_connections 2019-12-05T05:03:40.698 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.688+0000 7f896d748700 1 --2- 172.21.15.42:0/283917952 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f89680a7740 0x7f896808df80 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:40.699 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.688+0000 7f896d748700 1 -- 172.21.15.42:0/283917952 shutdown_connections 2019-12-05T05:03:40.700 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.688+0000 7f896d748700 1 -- 172.21.15.42:0/283917952 wait complete. 2019-12-05T05:03:40.700 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.688+0000 7f896d748700 1 -- 172.21.15.42:0/283917952 >> 172.21.15.42:0/283917952 conn(0x7f89680aabe0 msgr2=0x7f89680a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:40.708 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.688+0000 7f896d748700 1 Processor -- start 2019-12-05T05:03:40.708 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.688+0000 7f896d748700 1 -- start start 2019-12-05T05:03:40.708 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.692+0000 7f896d748700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f896812bf90 0x7f896812e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:40.708 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.692+0000 7f896d748700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89680a58f0 con 0x7f89680a3a60 2019-12-05T05:03:40.708 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.692+0000 7f896d748700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8968147240 con 0x7f8968119040 2019-12-05T05:03:40.709 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.692+0000 7f896d748700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8968119470 con 0x7f896812bf90 2019-12-05T05:03:40.709 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.692+0000 7f895ffff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f8968119040 0x7f8968146a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:53392/0 (socket says 172.21.15.42:53392) 2019-12-05T05:03:40.709 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.692+0000 7f895ffff700 1 -- 172.21.15.42:0/2333677649 learned_addr learned my addr 172.21.15.42:0/2333677649 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:03:40.709 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.692+0000 7f8967fff700 1 --2- 172.21.15.42:0/2333677649 >> v2:172.21.15.40:3300/0 conn(0x7f896812bf90 0x7f896812e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:40.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.692+0000 7f8964ff9700 1 -- 172.21.15.42:0/2333677649 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2646651116 0 0) 0x7f8968147240 con 0x7f8968119040 2019-12-05T05:03:40.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.692+0000 7f8964ff9700 1 -- 172.21.15.42:0/2333677649 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f894c0030b0 con 0x7f8968119040 2019-12-05T05:03:40.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.692+0000 7f8964ff9700 1 -- 172.21.15.42:0/2333677649 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3262638049 0 0) 0x7f89680a58f0 con 0x7f89680a3a60 2019-12-05T05:03:40.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.692+0000 7f8964ff9700 1 -- 172.21.15.42:0/2333677649 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8968147240 con 0x7f89680a3a60 2019-12-05T05:03:40.711 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.692+0000 7f8967fff700 1 -- 172.21.15.42:0/2333677649 >> v1:172.21.15.42:6789/0 conn(0x7f8968119040 legacy=0x7f8968146a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:40.711 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.692+0000 7f8967fff700 1 -- 172.21.15.42:0/2333677649 >> v1:172.21.15.40:6789/0 conn(0x7f89680a3a60 legacy=0x7f89680a5020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:40.711 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.692+0000 7f8967fff700 1 -- 172.21.15.42:0/2333677649 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8968130940 con 0x7f896812bf90 2019-12-05T05:03:40.711 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.692+0000 7f8967fff700 1 --2- 172.21.15.42:0/2333677649 >> v2:172.21.15.40:3300/0 conn(0x7f896812bf90 0x7f896812e3d0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f8960001210 tx=0x7f8960010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:40.712 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.692+0000 7f8967fff700 1 --2- 172.21.15.42:0/2333677649 >> v2:172.21.15.40:3300/0 conn(0x7f896812bf90 0x7f896812e3d0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f8960001210 tx=0x7f8960010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:40.713 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.692+0000 7f8967fff700 1 --2- 172.21.15.42:0/2333677649 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f896812bf90 0x7f896812e3d0 secure :-1 s=READY pgs=98 cs=0 l=1 rx=0x7f8960001210 tx=0x7f8960010ac0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:40.713 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.692+0000 7f8967fff700 1 --2- 172.21.15.42:0/2333677649 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f896812bf90 0x7f896812e3d0 secure :-1 s=THROTTLE_DONE pgs=98 cs=0 l=1 rx=0x7f8960001210 tx=0x7f8960010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:40.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.692+0000 7f8964ff9700 1 -- 172.21.15.42:0/2333677649 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8960002240 con 0x7f896812bf90 2019-12-05T05:03:40.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.692+0000 7f8967fff700 1 --2- 172.21.15.42:0/2333677649 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f896812bf90 0x7f896812e3d0 secure :-1 s=THROTTLE_DONE pgs=98 cs=0 l=1 rx=0x7f8960001210 tx=0x7f8960010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:40.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.692+0000 7f8964ff9700 1 -- 172.21.15.42:0/2333677649 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8960006ad0 con 0x7f896812bf90 2019-12-05T05:03:40.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.692+0000 7f8967fff700 1 --2- 172.21.15.42:0/2333677649 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f896812bf90 0x7f896812e3d0 secure :-1 s=THROTTLE_DONE pgs=98 cs=0 l=1 rx=0x7f8960001210 tx=0x7f8960010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:40.715 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.692+0000 7f8964ff9700 1 -- 172.21.15.42:0/2333677649 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8960002820 con 0x7f896812bf90 2019-12-05T05:03:40.715 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.692+0000 7f896d748700 1 -- 172.21.15.42:0/2333677649 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f896812e920 con 0x7f896812bf90 2019-12-05T05:03:40.715 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.692+0000 7f896d748700 1 -- 172.21.15.42:0/2333677649 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f896812eea0 con 0x7f896812bf90 2019-12-05T05:03:40.715 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.692+0000 7f8967fff700 1 --2- 172.21.15.42:0/2333677649 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f896812bf90 0x7f896812e3d0 secure :-1 s=THROTTLE_DONE pgs=98 cs=0 l=1 rx=0x7f8960001210 tx=0x7f8960010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:40.716 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.696+0000 7f8964ff9700 1 -- 172.21.15.42:0/2333677649 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f8960002b80 con 0x7f896812bf90 2019-12-05T05:03:40.716 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.696+0000 7f8967fff700 1 --2- 172.21.15.42:0/2333677649 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f896812bf90 0x7f896812e3d0 secure :-1 s=THROTTLE_DONE pgs=98 cs=0 l=1 rx=0x7f8960001210 tx=0x7f8960010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:40.716 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.696+0000 7f8964ff9700 1 --2- 172.21.15.42:0/2333677649 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f894c03b420 0x7f894c03d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:40.716 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.696+0000 7f8964ff9700 1 -- 172.21.15.42:0/2333677649 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f896004b3d0 con 0x7f896812bf90 2019-12-05T05:03:40.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.696+0000 7f89677fe700 1 --2- 172.21.15.42:0/2333677649 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f894c03b420 0x7f894c03d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:40.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.696+0000 7f89677fe700 1 --2- 172.21.15.42:0/2333677649 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f894c03b420 0x7f894c03d930 crc :-1 s=READY pgs=66 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:40.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.696+0000 7f896d748700 1 --2- 172.21.15.42:0/2333677649 >> [v2:172.21.15.42:6818/12116,v1:172.21.15.42:6819/12116] conn(0x7f8968145860 0x7f8968054260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:40.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.696+0000 7f896d748700 1 -- 172.21.15.42:0/2333677649 --> [v2:172.21.15.42:6818/12116,v1:172.21.15.42:6819/12116] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f896805d110 con 0x7f8968145860 2019-12-05T05:03:40.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.696+0000 7f895ffff700 1 --2- 172.21.15.42:0/2333677649 >> [v2:172.21.15.42:6818/12116,v1:172.21.15.42:6819/12116] conn(0x7f8968145860 0x7f8968054260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:40.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.700+0000 7f895ffff700 1 -- 172.21.15.42:0/2333677649 >> [v2:172.21.15.42:6818/12116,v1:172.21.15.42:6819/12116] conn(0x7f8968145860 msgr2=0x7f8968054260 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2019-12-05T05:03:40.718 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.700+0000 7f895ffff700 1 --2- 172.21.15.42:0/2333677649 >> [v2:172.21.15.42:6818/12116,v1:172.21.15.42:6819/12116] conn(0x7f8968145860 0x7f8968054260 crc :-1 s=READY pgs=7 cs=0 l=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:40.718 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.700+0000 7f8964ff9700 1 -- 172.21.15.42:0/2333677649 <== osd.3 v2:172.21.15.42:6818/12116 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23014 (crc 0 0 0) 0x7f896805d110 con 0x7f8968145860 2019-12-05T05:03:40.732 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.728+0000 7f896d748700 1 -- 172.21.15.42:0/2333677649 --> [v2:172.21.15.42:6818/12116,v1:172.21.15.42:6819/12116] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f896805b8c0 con 0x7f8968145860 2019-12-05T05:03:40.732 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.728+0000 7f8964ff9700 1 -- 172.21.15.42:0/2333677649 <== osd.3 v2:172.21.15.42:6818/12116 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f896805b8c0 con 0x7f8968145860 2019-12-05T05:03:40.735 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.728+0000 7f896d748700 1 -- 172.21.15.42:0/2333677649 >> [v2:172.21.15.42:6818/12116,v1:172.21.15.42:6819/12116] conn(0x7f8968145860 msgr2=0x7f8968054260 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:40.735 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.728+0000 7f896d748700 1 --2- 172.21.15.42:0/2333677649 >> [v2:172.21.15.42:6818/12116,v1:172.21.15.42:6819/12116] conn(0x7f8968145860 0x7f8968054260 crc :-1 s=READY pgs=7 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:40.735 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.728+0000 7f896d748700 1 -- 172.21.15.42:0/2333677649 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f894c03b420 msgr2=0x7f894c03d930 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:40.735 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.728+0000 7f896d748700 1 --2- 172.21.15.42:0/2333677649 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f894c03b420 0x7f894c03d930 crc :-1 s=READY pgs=66 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:40.735 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.728+0000 7f896d748700 1 -- 172.21.15.42:0/2333677649 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f896812bf90 msgr2=0x7f896812e3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:40.736 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.728+0000 7f896d748700 1 --2- 172.21.15.42:0/2333677649 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f896812bf90 0x7f896812e3d0 secure :-1 s=READY pgs=98 cs=0 l=1 rx=0x7f8960001210 tx=0x7f8960010ac0).stop 2019-12-05T05:03:40.736 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.728+0000 7f8967fff700 1 -- 172.21.15.42:0/2333677649 reap_dead start 2019-12-05T05:03:40.736 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.728+0000 7f896d748700 1 -- 172.21.15.42:0/2333677649 shutdown_connections 2019-12-05T05:03:40.736 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.728+0000 7f896d748700 1 --2- 172.21.15.42:0/2333677649 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f896812bf90 0x7f896812e3d0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:40.736 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.728+0000 7f896d748700 1 -- 172.21.15.42:0/2333677649 shutdown_connections 2019-12-05T05:03:40.736 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.728+0000 7f896d748700 1 -- 172.21.15.42:0/2333677649 wait complete. 2019-12-05T05:03:40.736 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.728+0000 7f896d748700 1 -- 172.21.15.42:0/2333677649 >> 172.21.15.42:0/2333677649 conn(0x7f896807ee20 msgr2=0x7f8968059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:40.745 INFO:teuthology.orchestra.run.smithi042.stdout:55834574854 2019-12-05T05:03:40.746 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2019-12-05T05:03:40.851 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.836+0000 7fe2466e1700 1 Processor -- start 2019-12-05T05:03:40.851 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.836+0000 7fe2466e1700 1 -- start start 2019-12-05T05:03:40.851 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.840+0000 7fe2466e1700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe2400a7740 0x7fe24008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:40.851 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.840+0000 7fe2466e1700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe24007e720 0x7fe24007eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:40.851 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.840+0000 7fe2466e1700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2400a0710 con 0x7fe24009f2e0 2019-12-05T05:03:40.852 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.840+0000 7fe2466e1700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2400a0b00 con 0x7fe24007e720 2019-12-05T05:03:40.852 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.840+0000 7fe2466e1700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe24008e5b0 con 0x7fe2400a7740 2019-12-05T05:03:40.852 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.840+0000 7fe244c7e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe2400a7740 0x7fe24008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:40.852 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.840+0000 7fe244c7e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe2400a7740 0x7fe24008df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:53362/0 (socket says 172.21.15.42:53362) 2019-12-05T05:03:40.852 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.840+0000 7fe244c7e700 1 -- 172.21.15.42:0/3474605190 learned_addr learned my addr 172.21.15.42:0/3474605190 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:03:40.852 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.840+0000 7fe23ffff700 1 --2- 172.21.15.42:0/3474605190 >> v2:172.21.15.40:3300/0 conn(0x7fe24007e720 0x7fe24007eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:40.852 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.840+0000 7fe244c7e700 1 -- 172.21.15.42:0/3474605190 >> v2:172.21.15.40:3300/0 conn(0x7fe24007e720 msgr2=0x7fe24007eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:03:40.853 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.840+0000 7fe244c7e700 1 --2- 172.21.15.42:0/3474605190 >> v2:172.21.15.40:3300/0 conn(0x7fe24007e720 0x7fe24007eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:03:40.853 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.840+0000 7fe244c7e700 1 -- 172.21.15.42:0/3474605190 >> v1:172.21.15.40:6789/0 conn(0x7fe24009f2e0 legacy=0x7fe24009f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:40.853 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.840+0000 7fe244c7e700 1 -- 172.21.15.42:0/3474605190 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe24007f0a0 con 0x7fe2400a7740 2019-12-05T05:03:40.853 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.840+0000 7fe244c7e700 1 --2- 172.21.15.42:0/3474605190 >> v2:172.21.15.42:3300/0 conn(0x7fe2400a7740 0x7fe24008df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fe2300032b0 tx=0x7fe2300100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:40.853 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.840+0000 7fe244c7e700 1 --2- 172.21.15.42:0/3474605190 >> v2:172.21.15.42:3300/0 conn(0x7fe2400a7740 0x7fe24008df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fe2300032b0 tx=0x7fe2300100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:40.853 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.840+0000 7fe244c7e700 1 --2- 172.21.15.42:0/3474605190 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe2400a7740 0x7fe24008df80 secure :-1 s=READY pgs=115 cs=0 l=1 rx=0x7fe2300032b0 tx=0x7fe2300100b0).ready entity=mon.1 client_cookie=2b7abaf75f2761f1 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:40.853 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.840+0000 7fe244c7e700 1 --2- 172.21.15.42:0/3474605190 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe2400a7740 0x7fe24008df80 secure :-1 s=THROTTLE_DONE pgs=115 cs=0 l=1 rx=0x7fe2300032b0 tx=0x7fe2300100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:40.854 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.840+0000 7fe23f7fe700 1 -- 172.21.15.42:0/3474605190 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe230011dd0 con 0x7fe2400a7740 2019-12-05T05:03:40.854 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.840+0000 7fe244c7e700 1 --2- 172.21.15.42:0/3474605190 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe2400a7740 0x7fe24008df80 secure :-1 s=THROTTLE_DONE pgs=115 cs=0 l=1 rx=0x7fe2300032b0 tx=0x7fe2300100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:40.854 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.840+0000 7fe244c7e700 1 --2- 172.21.15.42:0/3474605190 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe2400a7740 0x7fe24008df80 secure :-1 s=THROTTLE_DONE pgs=115 cs=0 l=1 rx=0x7fe2300032b0 tx=0x7fe2300100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:40.854 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.840+0000 7fe23f7fe700 1 -- 172.21.15.42:0/3474605190 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2300119f0 con 0x7fe2400a7740 2019-12-05T05:03:40.854 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.840+0000 7fe23f7fe700 1 -- 172.21.15.42:0/3474605190 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe23000a540 con 0x7fe2400a7740 2019-12-05T05:03:40.854 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.840+0000 7fe2466e1700 1 -- 172.21.15.42:0/3474605190 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe2400a7740 msgr2=0x7fe24008df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:40.854 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.840+0000 7fe2466e1700 1 --2- 172.21.15.42:0/3474605190 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe2400a7740 0x7fe24008df80 secure :-1 s=READY pgs=115 cs=0 l=1 rx=0x7fe2300032b0 tx=0x7fe2300100b0).stop 2019-12-05T05:03:40.855 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.840+0000 7fe2466e1700 1 -- 172.21.15.42:0/3474605190 shutdown_connections 2019-12-05T05:03:40.855 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.840+0000 7fe2466e1700 1 --2- 172.21.15.42:0/3474605190 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe2400a7740 0x7fe24008df80 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:40.855 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.840+0000 7fe2466e1700 1 --2- 172.21.15.42:0/3474605190 >> v2:172.21.15.40:3300/0 conn(0x7fe24007e720 0x7fe24007eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:03:40.855 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.840+0000 7fe2466e1700 1 -- 172.21.15.42:0/3474605190 shutdown_connections 2019-12-05T05:03:40.855 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.840+0000 7fe2466e1700 1 -- 172.21.15.42:0/3474605190 wait complete. 2019-12-05T05:03:40.855 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.840+0000 7fe2466e1700 1 -- 172.21.15.42:0/3474605190 >> 172.21.15.42:0/3474605190 conn(0x7fe2400aabe0 msgr2=0x7fe2400a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:40.855 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.844+0000 7fe2466e1700 1 Processor -- start 2019-12-05T05:03:40.856 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.844+0000 7fe2466e1700 1 -- start start 2019-12-05T05:03:40.856 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.844+0000 7fe2466e1700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe240121260 0x7fe24013bc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:40.856 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.844+0000 7fe2466e1700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe24007b600 con 0x7fe240118930 2019-12-05T05:03:40.856 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.844+0000 7fe2466e1700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe240119ed0 con 0x7fe240123690 2019-12-05T05:03:40.856 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.844+0000 7fe2466e1700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe240117b40 con 0x7fe240121260 2019-12-05T05:03:40.856 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.844+0000 7fe244c7e700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fe240123690 0x7fe24007ae00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:53406/0 (socket says 172.21.15.42:53406) 2019-12-05T05:03:40.856 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.844+0000 7fe244c7e700 1 -- 172.21.15.42:0/3295166387 learned_addr learned my addr 172.21.15.42:0/3295166387 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:03:40.857 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.844+0000 7fe24547f700 1 --2- 172.21.15.42:0/3295166387 >> v2:172.21.15.40:3300/0 conn(0x7fe240121260 0x7fe24013bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:40.857 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.844+0000 7fe23d7fa700 1 -- 172.21.15.42:0/3295166387 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 661903209 0 0) 0x7fe240119ed0 con 0x7fe240123690 2019-12-05T05:03:40.857 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.844+0000 7fe23d7fa700 1 -- 172.21.15.42:0/3295166387 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2240030b0 con 0x7fe240123690 2019-12-05T05:03:40.857 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.844+0000 7fe23d7fa700 1 -- 172.21.15.42:0/3295166387 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3509604834 0 0) 0x7fe24007b600 con 0x7fe240118930 2019-12-05T05:03:40.857 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.844+0000 7fe23d7fa700 1 -- 172.21.15.42:0/3295166387 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe240119ed0 con 0x7fe240118930 2019-12-05T05:03:40.862 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.844+0000 7fe23d7fa700 1 -- 172.21.15.42:0/3295166387 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 366133077 0 0) 0x7fe2240030b0 con 0x7fe240123690 2019-12-05T05:03:40.862 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.844+0000 7fe23d7fa700 1 -- 172.21.15.42:0/3295166387 --> v1:172.21.15.42:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe24007b600 con 0x7fe240123690 2019-12-05T05:03:40.862 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.844+0000 7fe23d7fa700 1 -- 172.21.15.42:0/3295166387 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7fe230007c40 con 0x7fe240123690 2019-12-05T05:03:40.862 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.844+0000 7fe23d7fa700 1 -- 172.21.15.42:0/3295166387 >> v2:172.21.15.40:3300/0 conn(0x7fe240121260 msgr2=0x7fe24013bc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:40.862 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.844+0000 7fe23d7fa700 1 --2- 172.21.15.42:0/3295166387 >> v2:172.21.15.40:3300/0 conn(0x7fe240121260 0x7fe24013bc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:40.863 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.844+0000 7fe23d7fa700 1 -- 172.21.15.42:0/3295166387 >> v1:172.21.15.42:6789/0 conn(0x7fe240123690 legacy=0x7fe24007ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:40.863 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.844+0000 7fe23d7fa700 1 -- 172.21.15.42:0/3295166387 >> v1:172.21.15.40:6789/0 conn(0x7fe240118930 legacy=0x7fe240120a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:40.863 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.844+0000 7fe23d7fa700 1 --2- 172.21.15.42:0/3295166387 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe224003ab0 0x7fe224004420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:40.863 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.844+0000 7fe23d7fa700 1 --2- 172.21.15.42:0/3295166387 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe224003fc0 0x7fe224004a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:40.863 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.844+0000 7fe23d7fa700 1 --2- 172.21.15.42:0/3295166387 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe224009660 0x7fe22400baa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:40.863 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.844+0000 7fe23d7fa700 1 -- 172.21.15.42:0/3295166387 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe240117b40 con 0x7fe224009660 2019-12-05T05:03:40.863 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.844+0000 7fe23d7fa700 1 -- 172.21.15.42:0/3295166387 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe22400bff0 con 0x7fe224003fc0 2019-12-05T05:03:40.864 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.844+0000 7fe23d7fa700 1 -- 172.21.15.42:0/3295166387 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe22400c1b0 con 0x7fe224003ab0 2019-12-05T05:03:40.864 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.848+0000 7fe23ffff700 1 --2- 172.21.15.42:0/3295166387 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe224003fc0 0x7fe224004a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:40.864 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.848+0000 7fe244c7e700 1 --2- 172.21.15.42:0/3295166387 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe224003ab0 0x7fe224004420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:40.864 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.848+0000 7fe24547f700 1 --2- 172.21.15.42:0/3295166387 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe224009660 0x7fe22400baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:40.864 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.848+0000 7fe23ffff700 1 -- 172.21.15.42:0/3295166387 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe224003ab0 msgr2=0x7fe224004420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:40.864 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.848+0000 7fe23ffff700 1 --2- 172.21.15.42:0/3295166387 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe224003ab0 0x7fe224004420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:40.865 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.848+0000 7fe23ffff700 1 -- 172.21.15.42:0/3295166387 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe224009660 msgr2=0x7fe22400baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:40.865 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.848+0000 7fe23ffff700 1 --2- 172.21.15.42:0/3295166387 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe224009660 0x7fe22400baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:40.865 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.848+0000 7fe23ffff700 1 -- 172.21.15.42:0/3295166387 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe240117710 con 0x7fe224003fc0 2019-12-05T05:03:40.865 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.848+0000 7fe24547f700 1 -- 172.21.15.42:0/3295166387 reap_dead start 2019-12-05T05:03:40.865 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.848+0000 7fe2466e1700 1 -- 172.21.15.42:0/3295166387 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe240123690 con 0x7fe224003fc0 2019-12-05T05:03:40.865 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.848+0000 7fe23ffff700 1 --2- 172.21.15.42:0/3295166387 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe224003fc0 0x7fe224004a10 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fe228007e90 tx=0x7fe228010e70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:40.865 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.848+0000 7fe2466e1700 1 -- 172.21.15.42:0/3295166387 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe240118930 con 0x7fe224003fc0 2019-12-05T05:03:40.866 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.848+0000 7fe23ffff700 1 --2- 172.21.15.42:0/3295166387 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe224003fc0 0x7fe224004a10 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fe228007e90 tx=0x7fe228010e70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:40.866 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.848+0000 7fe23ffff700 1 --2- 172.21.15.42:0/3295166387 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe224003fc0 0x7fe224004a10 secure :-1 s=READY pgs=117 cs=0 l=1 rx=0x7fe228007e90 tx=0x7fe228010e70).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:40.866 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.848+0000 7fe23ffff700 1 --2- 172.21.15.42:0/3295166387 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe224003fc0 0x7fe224004a10 secure :-1 s=THROTTLE_DONE pgs=117 cs=0 l=1 rx=0x7fe228007e90 tx=0x7fe228010e70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:40.866 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.848+0000 7fe23d7fa700 1 -- 172.21.15.42:0/3295166387 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe22800e710 con 0x7fe224003fc0 2019-12-05T05:03:40.866 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.848+0000 7fe23ffff700 1 --2- 172.21.15.42:0/3295166387 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe224003fc0 0x7fe224004a10 secure :-1 s=THROTTLE_DONE pgs=117 cs=0 l=1 rx=0x7fe228007e90 tx=0x7fe228010e70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:40.866 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.848+0000 7fe23ffff700 1 --2- 172.21.15.42:0/3295166387 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe224003fc0 0x7fe224004a10 secure :-1 s=THROTTLE_DONE pgs=117 cs=0 l=1 rx=0x7fe228007e90 tx=0x7fe228010e70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:40.866 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.848+0000 7fe23d7fa700 1 -- 172.21.15.42:0/3295166387 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe228007170 con 0x7fe224003fc0 2019-12-05T05:03:40.867 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.848+0000 7fe23d7fa700 1 -- 172.21.15.42:0/3295166387 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe22800ecf0 con 0x7fe224003fc0 2019-12-05T05:03:40.867 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.848+0000 7fe23ffff700 1 --2- 172.21.15.42:0/3295166387 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe224003fc0 0x7fe224004a10 secure :-1 s=THROTTLE_DONE pgs=117 cs=0 l=1 rx=0x7fe228007e90 tx=0x7fe228010e70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:40.867 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.848+0000 7fe23d7fa700 1 -- 172.21.15.42:0/3295166387 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fe228005540 con 0x7fe224003fc0 2019-12-05T05:03:40.867 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.848+0000 7fe23ffff700 1 --2- 172.21.15.42:0/3295166387 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe224003fc0 0x7fe224004a10 secure :-1 s=THROTTLE_DONE pgs=117 cs=0 l=1 rx=0x7fe228007e90 tx=0x7fe228010e70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:40.867 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.848+0000 7fe23d7fa700 1 --2- 172.21.15.42:0/3295166387 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe22403ecd0 0x7fe2240411e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:40.867 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.848+0000 7fe23d7fa700 1 -- 172.21.15.42:0/3295166387 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6031+0+0 (secure 0 0 0) 0x7fe22804b010 con 0x7fe224003fc0 2019-12-05T05:03:40.867 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.848+0000 7fe244c7e700 1 --2- 172.21.15.42:0/3295166387 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe22403ecd0 0x7fe2240411e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:40.868 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.852+0000 7fe244c7e700 1 --2- 172.21.15.42:0/3295166387 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe22403ecd0 0x7fe2240411e0 crc :-1 s=READY pgs=67 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:40.868 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.852+0000 7fe2466e1700 1 --2- 172.21.15.42:0/3295166387 >> [v2:172.21.15.40:6808/12488,v1:172.21.15.40:6809/12488] conn(0x7fe218001620 0x7fe218003b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:40.868 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.852+0000 7fe2466e1700 1 -- 172.21.15.42:0/3295166387 --> [v2:172.21.15.40:6808/12488,v1:172.21.15.40:6809/12488] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe2180066f0 con 0x7fe218001620 2019-12-05T05:03:40.868 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.852+0000 7fe24547f700 1 --2- 172.21.15.42:0/3295166387 >> [v2:172.21.15.40:6808/12488,v1:172.21.15.40:6809/12488] conn(0x7fe218001620 0x7fe218003b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:40.868 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.852+0000 7fe24547f700 1 -- 172.21.15.42:0/3295166387 >> [v2:172.21.15.40:6808/12488,v1:172.21.15.40:6809/12488] conn(0x7fe218001620 msgr2=0x7fe218003b30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2019-12-05T05:03:40.868 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.852+0000 7fe24547f700 1 --2- 172.21.15.42:0/3295166387 >> [v2:172.21.15.40:6808/12488,v1:172.21.15.40:6809/12488] conn(0x7fe218001620 0x7fe218003b30 crc :-1 s=READY pgs=7 cs=0 l=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:40.868 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.856+0000 7fe23d7fa700 1 -- 172.21.15.42:0/3295166387 <== osd.4 v2:172.21.15.40:6808/12488 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23014 (crc 0 0 0) 0x7fe2180066f0 con 0x7fe218001620 2019-12-05T05:03:40.883 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.876+0000 7fe2466e1700 1 -- 172.21.15.42:0/3295166387 --> [v2:172.21.15.40:6808/12488,v1:172.21.15.40:6809/12488] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fe218005790 con 0x7fe218001620 2019-12-05T05:03:40.884 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.880+0000 7fe23d7fa700 1 -- 172.21.15.42:0/3295166387 <== osd.4 v2:172.21.15.40:6808/12488 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fe218005790 con 0x7fe218001620 2019-12-05T05:03:40.885 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.880+0000 7fe2466e1700 1 -- 172.21.15.42:0/3295166387 >> [v2:172.21.15.40:6808/12488,v1:172.21.15.40:6809/12488] conn(0x7fe218001620 msgr2=0x7fe218003b30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:40.886 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.880+0000 7fe2466e1700 1 --2- 172.21.15.42:0/3295166387 >> [v2:172.21.15.40:6808/12488,v1:172.21.15.40:6809/12488] conn(0x7fe218001620 0x7fe218003b30 crc :-1 s=READY pgs=7 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:40.886 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.880+0000 7fe2466e1700 1 -- 172.21.15.42:0/3295166387 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe22403ecd0 msgr2=0x7fe2240411e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:40.886 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.880+0000 7fe2466e1700 1 --2- 172.21.15.42:0/3295166387 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe22403ecd0 0x7fe2240411e0 crc :-1 s=READY pgs=67 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:40.886 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.880+0000 7fe2466e1700 1 -- 172.21.15.42:0/3295166387 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe224003fc0 msgr2=0x7fe224004a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:40.886 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.880+0000 7fe2466e1700 1 --2- 172.21.15.42:0/3295166387 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe224003fc0 0x7fe224004a10 secure :-1 s=READY pgs=117 cs=0 l=1 rx=0x7fe228007e90 tx=0x7fe228010e70).stop 2019-12-05T05:03:40.886 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.880+0000 7fe2466e1700 1 -- 172.21.15.42:0/3295166387 shutdown_connections 2019-12-05T05:03:40.886 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.880+0000 7fe2466e1700 1 --2- 172.21.15.42:0/3295166387 >> [v2:172.21.15.40:6808/12488,v1:172.21.15.40:6809/12488] conn(0x7fe218001620 0x7fe218003b30 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:40.887 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.880+0000 7fe2466e1700 1 --2- 172.21.15.42:0/3295166387 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe22403ecd0 0x7fe2240411e0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:40.887 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.880+0000 7fe2466e1700 1 --2- 172.21.15.42:0/3295166387 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe224003fc0 0x7fe224004a10 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:40.887 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.880+0000 7fe2466e1700 1 -- 172.21.15.42:0/3295166387 shutdown_connections 2019-12-05T05:03:40.887 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.880+0000 7fe2466e1700 1 -- 172.21.15.42:0/3295166387 wait complete. 2019-12-05T05:03:40.887 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.880+0000 7fe2466e1700 1 -- 172.21.15.42:0/3295166387 >> 172.21.15.42:0/3295166387 conn(0x7fe2400a97e0 msgr2=0x7fe24009f580 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:40.896 INFO:teuthology.orchestra.run.smithi042.stdout:55834574853 2019-12-05T05:03:40.896 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2019-12-05T05:03:41.003 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.988+0000 7f47548c4700 1 Processor -- start 2019-12-05T05:03:41.003 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.988+0000 7f47548c4700 1 -- start start 2019-12-05T05:03:41.003 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.992+0000 7f47548c4700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f47500a7740 0x7f475008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:41.003 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.992+0000 7f47548c4700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f475009fae0 0x7f475007eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:41.003 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.992+0000 7f47548c4700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47500a0710 con 0x7f475007e720 2019-12-05T05:03:41.003 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.992+0000 7f47548c4700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47500a0b00 con 0x7f47500a7740 2019-12-05T05:03:41.004 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.992+0000 7f47548c4700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f475008e5b0 con 0x7f475009fae0 2019-12-05T05:03:41.004 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.992+0000 7f474f59e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f475009fae0 0x7f475007eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:41.004 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.992+0000 7f474f59e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f475009fae0 0x7f475007eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:53388/0 (socket says 172.21.15.42:53388) 2019-12-05T05:03:41.004 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.992+0000 7f474f59e700 1 -- 172.21.15.42:0/194884303 learned_addr learned my addr 172.21.15.42:0/194884303 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:03:41.004 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.992+0000 7f474ed9d700 1 --2- 172.21.15.42:0/194884303 >> v2:172.21.15.40:3300/0 conn(0x7f47500a7740 0x7f475008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:41.005 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.992+0000 7f474f59e700 1 -- 172.21.15.42:0/194884303 >> v2:172.21.15.40:3300/0 conn(0x7f47500a7740 msgr2=0x7f475008df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:03:41.005 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.992+0000 7f474f59e700 1 --2- 172.21.15.42:0/194884303 >> v2:172.21.15.40:3300/0 conn(0x7f47500a7740 0x7f475008df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:03:41.005 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.992+0000 7f474f59e700 1 -- 172.21.15.42:0/194884303 >> v1:172.21.15.40:6789/0 conn(0x7f475007e720 legacy=0x7f475009f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:41.005 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.992+0000 7f474f59e700 1 -- 172.21.15.42:0/194884303 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f475007f0a0 con 0x7f475009fae0 2019-12-05T05:03:41.005 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.992+0000 7f474f59e700 1 --2- 172.21.15.42:0/194884303 >> v2:172.21.15.42:3300/0 conn(0x7f475009fae0 0x7f475007eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f47480032b0 tx=0x7f47480100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:41.005 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.992+0000 7f474f59e700 1 --2- 172.21.15.42:0/194884303 >> v2:172.21.15.42:3300/0 conn(0x7f475009fae0 0x7f475007eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f47480032b0 tx=0x7f47480100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:41.005 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.992+0000 7f474f59e700 1 --2- 172.21.15.42:0/194884303 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f475009fae0 0x7f475007eb50 secure :-1 s=READY pgs=118 cs=0 l=1 rx=0x7f47480032b0 tx=0x7f47480100b0).ready entity=mon.1 client_cookie=ea07620e6d5385e2 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:41.006 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.992+0000 7f474f59e700 1 --2- 172.21.15.42:0/194884303 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f475009fae0 0x7f475007eb50 secure :-1 s=THROTTLE_DONE pgs=118 cs=0 l=1 rx=0x7f47480032b0 tx=0x7f47480100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:41.006 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.992+0000 7f474f59e700 1 --2- 172.21.15.42:0/194884303 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f475009fae0 0x7f475007eb50 secure :-1 s=THROTTLE_DONE pgs=118 cs=0 l=1 rx=0x7f47480032b0 tx=0x7f47480100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:41.006 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.992+0000 7f474f59e700 1 --2- 172.21.15.42:0/194884303 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f475009fae0 0x7f475007eb50 secure :-1 s=THROTTLE_DONE pgs=118 cs=0 l=1 rx=0x7f47480032b0 tx=0x7f47480100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:41.006 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.992+0000 7f474dd9b700 1 -- 172.21.15.42:0/194884303 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4748010d60 con 0x7f475009fae0 2019-12-05T05:03:41.007 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.992+0000 7f474dd9b700 1 -- 172.21.15.42:0/194884303 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4748010980 con 0x7f475009fae0 2019-12-05T05:03:41.007 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.992+0000 7f474dd9b700 1 -- 172.21.15.42:0/194884303 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f47480114a0 con 0x7f475009fae0 2019-12-05T05:03:41.007 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.992+0000 7f47548c4700 1 -- 172.21.15.42:0/194884303 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f475009fae0 msgr2=0x7f475007eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:41.007 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.992+0000 7f47548c4700 1 --2- 172.21.15.42:0/194884303 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f475009fae0 0x7f475007eb50 secure :-1 s=READY pgs=118 cs=0 l=1 rx=0x7f47480032b0 tx=0x7f47480100b0).stop 2019-12-05T05:03:41.008 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.992+0000 7f47548c4700 1 -- 172.21.15.42:0/194884303 shutdown_connections 2019-12-05T05:03:41.008 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.992+0000 7f47548c4700 1 --2- 172.21.15.42:0/194884303 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f475009fae0 0x7f475007eb50 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:41.008 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.992+0000 7f47548c4700 1 --2- 172.21.15.42:0/194884303 >> v2:172.21.15.40:3300/0 conn(0x7f47500a7740 0x7f475008df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:03:41.008 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.992+0000 7f47548c4700 1 -- 172.21.15.42:0/194884303 shutdown_connections 2019-12-05T05:03:41.009 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.992+0000 7f47548c4700 1 -- 172.21.15.42:0/194884303 wait complete. 2019-12-05T05:03:41.009 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.992+0000 7f47548c4700 1 -- 172.21.15.42:0/194884303 >> 172.21.15.42:0/194884303 conn(0x7f47500aabe0 msgr2=0x7f47500a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:41.009 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.996+0000 7f47548c4700 1 Processor -- start 2019-12-05T05:03:41.009 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.996+0000 7f47548c4700 1 -- start start 2019-12-05T05:03:41.013 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.996+0000 7f47548c4700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4750144b50 0x7f4750130490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:41.013 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.996+0000 7f47548c4700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47500a58e0 con 0x7f4750113b00 2019-12-05T05:03:41.014 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.996+0000 7f47548c4700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4750128ea0 con 0x7f47500a4240 2019-12-05T05:03:41.014 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:40.996+0000 7f47548c4700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47500a4e40 con 0x7f4750144b50 2019-12-05T05:03:41.014 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.000+0000 7f474ed9d700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f47500a4240 0x7f47500a50e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:53428/0 (socket says 172.21.15.42:53428) 2019-12-05T05:03:41.014 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.000+0000 7f474ed9d700 1 -- 172.21.15.42:0/411326628 learned_addr learned my addr 172.21.15.42:0/411326628 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:03:41.014 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.000+0000 7f474f59e700 1 --2- 172.21.15.42:0/411326628 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4750144b50 0x7f4750130490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:41.015 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.000+0000 7f473f7fe700 1 -- 172.21.15.42:0/411326628 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2242746034 0 0) 0x7f4750128ea0 con 0x7f47500a4240 2019-12-05T05:03:41.015 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.000+0000 7f473f7fe700 1 -- 172.21.15.42:0/411326628 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47340030b0 con 0x7f47500a4240 2019-12-05T05:03:41.015 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.000+0000 7f473f7fe700 1 -- 172.21.15.42:0/411326628 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3811042473 0 0) 0x7f47500a58e0 con 0x7f4750113b00 2019-12-05T05:03:41.015 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.000+0000 7f473f7fe700 1 -- 172.21.15.42:0/411326628 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4750128ea0 con 0x7f4750113b00 2019-12-05T05:03:41.015 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.000+0000 7f474f59e700 1 -- 172.21.15.42:0/411326628 >> v1:172.21.15.42:6789/0 conn(0x7f47500a4240 legacy=0x7f47500a50e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:41.016 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.000+0000 7f474f59e700 1 -- 172.21.15.42:0/411326628 >> v1:172.21.15.40:6789/0 conn(0x7f4750113b00 legacy=0x7f47501286a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:41.016 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.000+0000 7f474f59e700 1 -- 172.21.15.42:0/411326628 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f475008f5d0 con 0x7f4750144b50 2019-12-05T05:03:41.016 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.000+0000 7f474f59e700 1 --2- 172.21.15.42:0/411326628 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4750144b50 0x7f4750130490 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f47480032b0 tx=0x7f4748010e80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:41.016 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.000+0000 7f474f59e700 1 --2- 172.21.15.42:0/411326628 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4750144b50 0x7f4750130490 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f47480032b0 tx=0x7f4748010e80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:41.016 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.000+0000 7f474f59e700 1 --2- 172.21.15.42:0/411326628 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4750144b50 0x7f4750130490 secure :-1 s=READY pgs=33 cs=0 l=1 rx=0x7f47480032b0 tx=0x7f4748010e80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:41.016 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.000+0000 7f474f59e700 1 --2- 172.21.15.42:0/411326628 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4750144b50 0x7f4750130490 secure :-1 s=THROTTLE_DONE pgs=33 cs=0 l=1 rx=0x7f47480032b0 tx=0x7f4748010e80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:41.018 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.000+0000 7f474f59e700 1 --2- 172.21.15.42:0/411326628 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4750144b50 0x7f4750130490 secure :-1 s=THROTTLE_DONE pgs=33 cs=0 l=1 rx=0x7f47480032b0 tx=0x7f4748010e80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:41.018 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.000+0000 7f473f7fe700 1 -- 172.21.15.42:0/411326628 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4748011bb0 con 0x7f4750144b50 2019-12-05T05:03:41.018 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.000+0000 7f474f59e700 1 --2- 172.21.15.42:0/411326628 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4750144b50 0x7f4750130490 secure :-1 s=THROTTLE_DONE pgs=33 cs=0 l=1 rx=0x7f47480032b0 tx=0x7f4748010e80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:41.018 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.000+0000 7f473f7fe700 1 -- 172.21.15.42:0/411326628 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47480110c0 con 0x7f4750144b50 2019-12-05T05:03:41.018 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.000+0000 7f47548c4700 1 -- 172.21.15.42:0/411326628 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4750128030 con 0x7f4750144b50 2019-12-05T05:03:41.018 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.000+0000 7f473f7fe700 1 -- 172.21.15.42:0/411326628 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4748005ab0 con 0x7f4750144b50 2019-12-05T05:03:41.019 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.000+0000 7f47548c4700 1 -- 172.21.15.42:0/411326628 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47501309e0 con 0x7f4750144b50 2019-12-05T05:03:41.019 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.000+0000 7f474f59e700 1 --2- 172.21.15.42:0/411326628 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4750144b50 0x7f4750130490 secure :-1 s=THROTTLE_DONE pgs=33 cs=0 l=1 rx=0x7f47480032b0 tx=0x7f4748010e80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:41.019 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.004+0000 7f473f7fe700 1 -- 172.21.15.42:0/411326628 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f474800f280 con 0x7f4750144b50 2019-12-05T05:03:41.019 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.004+0000 7f474f59e700 1 --2- 172.21.15.42:0/411326628 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4750144b50 0x7f4750130490 secure :-1 s=THROTTLE_DONE pgs=33 cs=0 l=1 rx=0x7f47480032b0 tx=0x7f4748010e80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:41.019 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.004+0000 7f473f7fe700 1 --2- 172.21.15.42:0/411326628 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f473403b4a0 0x7f473403d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:41.019 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.004+0000 7f473f7fe700 1 -- 172.21.15.42:0/411326628 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f474804b420 con 0x7f4750144b50 2019-12-05T05:03:41.019 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.004+0000 7f474ed9d700 1 --2- 172.21.15.42:0/411326628 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f473403b4a0 0x7f473403d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:41.020 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.004+0000 7f474ed9d700 1 --2- 172.21.15.42:0/411326628 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f473403b4a0 0x7f473403d9b0 crc :-1 s=READY pgs=68 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:41.020 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.004+0000 7f47548c4700 1 --2- 172.21.15.42:0/411326628 >> [v2:172.21.15.40:6800/12489,v1:172.21.15.40:6801/12489] conn(0x7f4750054260 0x7f4750054690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:41.020 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.004+0000 7f47548c4700 1 -- 172.21.15.42:0/411326628 --> [v2:172.21.15.40:6800/12489,v1:172.21.15.40:6801/12489] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f475005b8c0 con 0x7f4750054260 2019-12-05T05:03:41.020 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.004+0000 7f474e59c700 1 --2- 172.21.15.42:0/411326628 >> [v2:172.21.15.40:6800/12489,v1:172.21.15.40:6801/12489] conn(0x7f4750054260 0x7f4750054690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:41.020 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.004+0000 7f474e59c700 1 -- 172.21.15.42:0/411326628 >> [v2:172.21.15.40:6800/12489,v1:172.21.15.40:6801/12489] conn(0x7f4750054260 msgr2=0x7f4750054690 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2019-12-05T05:03:41.020 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.004+0000 7f474e59c700 1 --2- 172.21.15.42:0/411326628 >> [v2:172.21.15.40:6800/12489,v1:172.21.15.40:6801/12489] conn(0x7f4750054260 0x7f4750054690 crc :-1 s=READY pgs=7 cs=0 l=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:41.021 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.008+0000 7f473f7fe700 1 -- 172.21.15.42:0/411326628 <== osd.5 v2:172.21.15.40:6800/12489 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23014 (crc 0 0 0) 0x7f475005b8c0 con 0x7f4750054260 2019-12-05T05:03:41.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.032+0000 7f47548c4700 1 -- 172.21.15.42:0/411326628 --> [v2:172.21.15.40:6800/12489,v1:172.21.15.40:6801/12489] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f47500a4b70 con 0x7f4750054260 2019-12-05T05:03:41.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.032+0000 7f473f7fe700 1 -- 172.21.15.42:0/411326628 <== osd.5 v2:172.21.15.40:6800/12489 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f47500a4b70 con 0x7f4750054260 2019-12-05T05:03:41.039 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.032+0000 7f47548c4700 1 -- 172.21.15.42:0/411326628 >> [v2:172.21.15.40:6800/12489,v1:172.21.15.40:6801/12489] conn(0x7f4750054260 msgr2=0x7f4750054690 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:41.039 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.032+0000 7f47548c4700 1 --2- 172.21.15.42:0/411326628 >> [v2:172.21.15.40:6800/12489,v1:172.21.15.40:6801/12489] conn(0x7f4750054260 0x7f4750054690 crc :-1 s=READY pgs=7 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:41.039 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.032+0000 7f47548c4700 1 -- 172.21.15.42:0/411326628 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f473403b4a0 msgr2=0x7f473403d9b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:41.039 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.032+0000 7f47548c4700 1 --2- 172.21.15.42:0/411326628 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f473403b4a0 0x7f473403d9b0 crc :-1 s=READY pgs=68 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:41.040 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.032+0000 7f47548c4700 1 -- 172.21.15.42:0/411326628 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4750144b50 msgr2=0x7f4750130490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:41.040 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.032+0000 7f47548c4700 1 --2- 172.21.15.42:0/411326628 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4750144b50 0x7f4750130490 secure :-1 s=READY pgs=33 cs=0 l=1 rx=0x7f47480032b0 tx=0x7f4748010e80).stop 2019-12-05T05:03:41.040 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.032+0000 7f474f59e700 1 -- 172.21.15.42:0/411326628 reap_dead start 2019-12-05T05:03:41.040 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.032+0000 7f47548c4700 1 -- 172.21.15.42:0/411326628 shutdown_connections 2019-12-05T05:03:41.040 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.032+0000 7f47548c4700 1 -- 172.21.15.42:0/411326628 shutdown_connections 2019-12-05T05:03:41.040 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.032+0000 7f47548c4700 1 -- 172.21.15.42:0/411326628 wait complete. 2019-12-05T05:03:41.040 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.032+0000 7f47548c4700 1 -- 172.21.15.42:0/411326628 >> 172.21.15.42:0/411326628 conn(0x7f47500a9990 msgr2=0x7f4750059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:41.052 INFO:teuthology.orchestra.run.smithi042.stdout:55834574854 2019-12-05T05:03:41.053 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2019-12-05T05:03:41.149 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.140+0000 7f1235ee3700 1 Processor -- start 2019-12-05T05:03:41.149 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.140+0000 7f1235ee3700 1 -- start start 2019-12-05T05:03:41.149 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.140+0000 7f1235ee3700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f12300a7730 0x7f123008df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:41.149 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.140+0000 7f1235ee3700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f123009fad0 0x7f123007eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:41.150 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.140+0000 7f1235ee3700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12300a0700 con 0x7f123007e710 2019-12-05T05:03:41.150 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.140+0000 7f1235ee3700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12300a0af0 con 0x7f123009fad0 2019-12-05T05:03:41.150 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.140+0000 7f1235ee3700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f123008e5a0 con 0x7f12300a7730 2019-12-05T05:03:41.150 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.140+0000 7f122f7fe700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f123007e710 0x7f123009f2d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:53438/0 (socket says 172.21.15.42:53438) 2019-12-05T05:03:41.150 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.140+0000 7f122f7fe700 1 -- 172.21.15.42:0/687333869 learned_addr learned my addr 172.21.15.42:0/687333869 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:03:41.150 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.140+0000 7f1234c81700 1 --2- 172.21.15.42:0/687333869 >> v2:172.21.15.40:3300/0 conn(0x7f123009fad0 0x7f123007eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:41.151 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.140+0000 7f122ffff700 1 --2- 172.21.15.42:0/687333869 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f12300a7730 0x7f123008df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:41.151 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.140+0000 7f122effd700 1 -- 172.21.15.42:0/687333869 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4117792379 0 0) 0x7f12300a0700 con 0x7f123007e710 2019-12-05T05:03:41.151 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.140+0000 7f122effd700 1 -- 172.21.15.42:0/687333869 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f121c0030b0 con 0x7f123007e710 2019-12-05T05:03:41.151 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.140+0000 7f1234c81700 1 -- 172.21.15.42:0/687333869 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f12300a7730 msgr2=0x7f123008df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:03:41.151 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.140+0000 7f1234c81700 1 --2- 172.21.15.42:0/687333869 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f12300a7730 0x7f123008df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:03:41.151 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.140+0000 7f1234c81700 1 -- 172.21.15.42:0/687333869 >> v1:172.21.15.42:6789/0 conn(0x7f123007e710 legacy=0x7f123009f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:41.152 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.140+0000 7f1234c81700 1 -- 172.21.15.42:0/687333869 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f123007f090 con 0x7f123009fad0 2019-12-05T05:03:41.152 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.140+0000 7f1234c81700 1 --2- 172.21.15.42:0/687333869 >> v2:172.21.15.40:3300/0 conn(0x7f123009fad0 0x7f123007eb40 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f1220008790 tx=0x7f122000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:41.152 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.140+0000 7f1234c81700 1 --2- 172.21.15.42:0/687333869 >> v2:172.21.15.40:3300/0 conn(0x7f123009fad0 0x7f123007eb40 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f1220008790 tx=0x7f122000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:41.152 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.140+0000 7f1234c81700 1 --2- 172.21.15.42:0/687333869 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f123009fad0 0x7f123007eb40 secure :-1 s=READY pgs=103 cs=0 l=1 rx=0x7f1220008790 tx=0x7f122000fc70).ready entity=mon.0 client_cookie=a966509752d34b99 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:41.152 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.140+0000 7f1234c81700 1 --2- 172.21.15.42:0/687333869 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f123009fad0 0x7f123007eb40 secure :-1 s=THROTTLE_DONE pgs=103 cs=0 l=1 rx=0x7f1220008790 tx=0x7f122000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:41.152 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.140+0000 7f122effd700 1 -- 172.21.15.42:0/687333869 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1220008bb0 con 0x7f123009fad0 2019-12-05T05:03:41.153 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.140+0000 7f1234c81700 1 --2- 172.21.15.42:0/687333869 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f123009fad0 0x7f123007eb40 secure :-1 s=THROTTLE_DONE pgs=103 cs=0 l=1 rx=0x7f1220008790 tx=0x7f122000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:41.153 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.140+0000 7f1234c81700 1 --2- 172.21.15.42:0/687333869 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f123009fad0 0x7f123007eb40 secure :-1 s=THROTTLE_DONE pgs=103 cs=0 l=1 rx=0x7f1220008790 tx=0x7f122000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:41.153 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.140+0000 7f122effd700 1 -- 172.21.15.42:0/687333869 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12200087d0 con 0x7f123009fad0 2019-12-05T05:03:41.153 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.140+0000 7f122effd700 1 -- 172.21.15.42:0/687333869 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1220011860 con 0x7f123009fad0 2019-12-05T05:03:41.153 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.140+0000 7f1235ee3700 1 -- 172.21.15.42:0/687333869 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f123009fad0 msgr2=0x7f123007eb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:41.154 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.140+0000 7f1235ee3700 1 --2- 172.21.15.42:0/687333869 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f123009fad0 0x7f123007eb40 secure :-1 s=READY pgs=103 cs=0 l=1 rx=0x7f1220008790 tx=0x7f122000fc70).stop 2019-12-05T05:03:41.154 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.144+0000 7f1235ee3700 1 -- 172.21.15.42:0/687333869 shutdown_connections 2019-12-05T05:03:41.154 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.144+0000 7f1235ee3700 1 --2- 172.21.15.42:0/687333869 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f123009fad0 0x7f123007eb40 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:41.155 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.144+0000 7f1235ee3700 1 --2- 172.21.15.42:0/687333869 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f12300a7730 0x7f123008df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:03:41.155 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.144+0000 7f1235ee3700 1 -- 172.21.15.42:0/687333869 shutdown_connections 2019-12-05T05:03:41.155 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.144+0000 7f1235ee3700 1 -- 172.21.15.42:0/687333869 wait complete. 2019-12-05T05:03:41.156 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.144+0000 7f1235ee3700 1 -- 172.21.15.42:0/687333869 >> 172.21.15.42:0/687333869 conn(0x7f12300aabd0 msgr2=0x7f12300a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:41.159 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.144+0000 7f1235ee3700 1 Processor -- start 2019-12-05T05:03:41.159 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.144+0000 7f1235ee3700 1 -- start start 2019-12-05T05:03:41.159 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.144+0000 7f1235ee3700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f12300a6ec0 0x7f12300a6080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:41.159 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.144+0000 7f1235ee3700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f12301161e0 0x7f1230144ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:41.159 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.144+0000 7f1235ee3700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f123012cea0 con 0x7f12301451f0 2019-12-05T05:03:41.159 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.144+0000 7f1235ee3700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12300a58a0 con 0x7f12301161e0 2019-12-05T05:03:41.160 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.144+0000 7f1235ee3700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12300a5b40 con 0x7f12300a6ec0 2019-12-05T05:03:41.160 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.144+0000 7f122f7fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f12301161e0 0x7f1230144ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:41.160 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.144+0000 7f1234c81700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f12301451f0 0x7f123012c6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:53448/0 (socket says 172.21.15.42:53448) 2019-12-05T05:03:41.160 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.144+0000 7f122f7fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f12301161e0 0x7f1230144ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:53408/0 (socket says 172.21.15.42:53408) 2019-12-05T05:03:41.160 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.144+0000 7f1234c81700 1 -- 172.21.15.42:0/3834758206 learned_addr learned my addr 172.21.15.42:0/3834758206 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:03:41.160 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.144+0000 7f122ffff700 1 --2- 172.21.15.42:0/3834758206 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f12300a6ec0 0x7f12300a6080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:41.161 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.144+0000 7f122f7fe700 1 -- 172.21.15.42:0/3834758206 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f12300a6ec0 msgr2=0x7f12300a6080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:41.161 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.144+0000 7f122f7fe700 1 --2- 172.21.15.42:0/3834758206 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f12300a6ec0 0x7f12300a6080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:41.161 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.144+0000 7f122f7fe700 1 -- 172.21.15.42:0/3834758206 >> v1:172.21.15.42:6789/0 conn(0x7f12301451f0 legacy=0x7f123012c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:41.161 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.144+0000 7f122f7fe700 1 -- 172.21.15.42:0/3834758206 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12300a65d0 con 0x7f12301161e0 2019-12-05T05:03:41.161 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.144+0000 7f122f7fe700 1 --2- 172.21.15.42:0/3834758206 >> v2:172.21.15.42:3300/0 conn(0x7f12301161e0 0x7f1230144ca0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f12300a5a60 tx=0x7f1228010c60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:41.161 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.144+0000 7f122f7fe700 1 --2- 172.21.15.42:0/3834758206 >> v2:172.21.15.42:3300/0 conn(0x7f12301161e0 0x7f1230144ca0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f12300a5a60 tx=0x7f1228010c60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:41.161 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.144+0000 7f122f7fe700 1 --2- 172.21.15.42:0/3834758206 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f12301161e0 0x7f1230144ca0 secure :-1 s=READY pgs=122 cs=0 l=1 rx=0x7f12300a5a60 tx=0x7f1228010c60).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:41.161 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.144+0000 7f122f7fe700 1 --2- 172.21.15.42:0/3834758206 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f12301161e0 0x7f1230144ca0 secure :-1 s=THROTTLE_DONE pgs=122 cs=0 l=1 rx=0x7f12300a5a60 tx=0x7f1228010c60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:41.162 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.144+0000 7f122f7fe700 1 --2- 172.21.15.42:0/3834758206 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f12301161e0 0x7f1230144ca0 secure :-1 s=THROTTLE_DONE pgs=122 cs=0 l=1 rx=0x7f12300a5a60 tx=0x7f1228010c60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:41.162 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.144+0000 7f122cff9700 1 -- 172.21.15.42:0/3834758206 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f12280122e0 con 0x7f12301161e0 2019-12-05T05:03:41.163 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.144+0000 7f122f7fe700 1 --2- 172.21.15.42:0/3834758206 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f12301161e0 0x7f1230144ca0 secure :-1 s=THROTTLE_DONE pgs=122 cs=0 l=1 rx=0x7f12300a5a60 tx=0x7f1228010c60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:41.163 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.144+0000 7f122cff9700 1 -- 172.21.15.42:0/3834758206 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1228005ae0 con 0x7f12301161e0 2019-12-05T05:03:41.163 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.144+0000 7f122cff9700 1 -- 172.21.15.42:0/3834758206 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f12280128c0 con 0x7f12301161e0 2019-12-05T05:03:41.163 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.148+0000 7f1235ee3700 1 -- 172.21.15.42:0/3834758206 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12301369e0 con 0x7f12301161e0 2019-12-05T05:03:41.163 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.148+0000 7f1235ee3700 1 -- 172.21.15.42:0/3834758206 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1230136f10 con 0x7f12301161e0 2019-12-05T05:03:41.164 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.148+0000 7f122f7fe700 1 --2- 172.21.15.42:0/3834758206 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f12301161e0 0x7f1230144ca0 secure :-1 s=THROTTLE_DONE pgs=122 cs=0 l=1 rx=0x7f12300a5a60 tx=0x7f1228010c60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:41.164 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.148+0000 7f122cff9700 1 -- 172.21.15.42:0/3834758206 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f1228013060 con 0x7f12301161e0 2019-12-05T05:03:41.164 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.148+0000 7f122f7fe700 1 --2- 172.21.15.42:0/3834758206 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f12301161e0 0x7f1230144ca0 secure :-1 s=THROTTLE_DONE pgs=122 cs=0 l=1 rx=0x7f12300a5a60 tx=0x7f1228010c60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:41.165 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.148+0000 7f122cff9700 1 --2- 172.21.15.42:0/3834758206 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f121403acf0 0x7f121403d200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:41.166 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.148+0000 7f122cff9700 1 -- 172.21.15.42:0/3834758206 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f122804b380 con 0x7f12301161e0 2019-12-05T05:03:41.166 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.148+0000 7f122ffff700 1 --2- 172.21.15.42:0/3834758206 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f121403acf0 0x7f121403d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:41.166 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.148+0000 7f122ffff700 1 --2- 172.21.15.42:0/3834758206 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f121403acf0 0x7f121403d200 crc :-1 s=READY pgs=69 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:41.167 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.156+0000 7f1235ee3700 1 --2- 172.21.15.42:0/3834758206 >> [v2:172.21.15.40:6811/12491,v1:172.21.15.40:6814/12491] conn(0x7f1230054250 0x7f1230054680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:41.168 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.156+0000 7f1235ee3700 1 -- 172.21.15.42:0/3834758206 --> [v2:172.21.15.40:6811/12491,v1:172.21.15.40:6814/12491] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f123005d100 con 0x7f1230054250 2019-12-05T05:03:41.168 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.156+0000 7f1234c81700 1 --2- 172.21.15.42:0/3834758206 >> [v2:172.21.15.40:6811/12491,v1:172.21.15.40:6814/12491] conn(0x7f1230054250 0x7f1230054680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:41.168 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.156+0000 7f1234c81700 1 -- 172.21.15.42:0/3834758206 >> [v2:172.21.15.40:6811/12491,v1:172.21.15.40:6814/12491] conn(0x7f1230054250 msgr2=0x7f1230054680 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2019-12-05T05:03:41.168 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.156+0000 7f1234c81700 1 --2- 172.21.15.42:0/3834758206 >> [v2:172.21.15.40:6811/12491,v1:172.21.15.40:6814/12491] conn(0x7f1230054250 0x7f1230054680 crc :-1 s=READY pgs=12 cs=0 l=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:41.169 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.156+0000 7f122cff9700 1 -- 172.21.15.42:0/3834758206 <== osd.6 v2:172.21.15.40:6811/12491 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23014 (crc 0 0 0) 0x7f123005d100 con 0x7f1230054250 2019-12-05T05:03:41.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.184+0000 7f1235ee3700 1 -- 172.21.15.42:0/3834758206 --> [v2:172.21.15.40:6811/12491,v1:172.21.15.40:6814/12491] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f12300a6960 con 0x7f1230054250 2019-12-05T05:03:41.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.184+0000 7f122cff9700 1 -- 172.21.15.42:0/3834758206 <== osd.6 v2:172.21.15.40:6811/12491 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f12300a6960 con 0x7f1230054250 2019-12-05T05:03:41.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.184+0000 7f1235ee3700 1 -- 172.21.15.42:0/3834758206 >> [v2:172.21.15.40:6811/12491,v1:172.21.15.40:6814/12491] conn(0x7f1230054250 msgr2=0x7f1230054680 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:41.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.184+0000 7f1235ee3700 1 --2- 172.21.15.42:0/3834758206 >> [v2:172.21.15.40:6811/12491,v1:172.21.15.40:6814/12491] conn(0x7f1230054250 0x7f1230054680 crc :-1 s=READY pgs=12 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:41.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.184+0000 7f1235ee3700 1 -- 172.21.15.42:0/3834758206 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f121403acf0 msgr2=0x7f121403d200 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:41.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.184+0000 7f1235ee3700 1 --2- 172.21.15.42:0/3834758206 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f121403acf0 0x7f121403d200 crc :-1 s=READY pgs=69 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:41.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.184+0000 7f1235ee3700 1 -- 172.21.15.42:0/3834758206 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f12301161e0 msgr2=0x7f1230144ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:41.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.184+0000 7f1235ee3700 1 --2- 172.21.15.42:0/3834758206 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f12301161e0 0x7f1230144ca0 secure :-1 s=READY pgs=122 cs=0 l=1 rx=0x7f12300a5a60 tx=0x7f1228010c60).stop 2019-12-05T05:03:41.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.184+0000 7f1234c81700 1 -- 172.21.15.42:0/3834758206 reap_dead start 2019-12-05T05:03:41.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.184+0000 7f1235ee3700 1 -- 172.21.15.42:0/3834758206 shutdown_connections 2019-12-05T05:03:41.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.184+0000 7f1235ee3700 1 --2- 172.21.15.42:0/3834758206 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f12301161e0 0x7f1230144ca0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:41.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.184+0000 7f1235ee3700 1 -- 172.21.15.42:0/3834758206 shutdown_connections 2019-12-05T05:03:41.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.184+0000 7f1235ee3700 1 -- 172.21.15.42:0/3834758206 wait complete. 2019-12-05T05:03:41.193 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.184+0000 7f1235ee3700 1 -- 172.21.15.42:0/3834758206 >> 172.21.15.42:0/3834758206 conn(0x7f12300a7f80 msgr2=0x7f1230141350 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:41.206 INFO:teuthology.orchestra.run.smithi042.stdout:55834574855 2019-12-05T05:03:41.206 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2019-12-05T05:03:41.312 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.300+0000 7f8341566700 1 Processor -- start 2019-12-05T05:03:41.312 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.300+0000 7f8341566700 1 -- start start 2019-12-05T05:03:41.312 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.300+0000 7f8341566700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f833c0a7730 0x7f833c08df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:41.312 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.300+0000 7f8341566700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f833c09fad0 0x7f833c07eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:41.312 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.300+0000 7f8341566700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f833c0a0700 con 0x7f833c07e710 2019-12-05T05:03:41.312 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.300+0000 7f8341566700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f833c0a0af0 con 0x7f833c09fad0 2019-12-05T05:03:41.313 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.300+0000 7f8341566700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833c08e5a0 con 0x7f833c0a7730 2019-12-05T05:03:41.313 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.300+0000 7f833bfff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f833c09fad0 0x7f833c07eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:41.313 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.300+0000 7f833bfff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f833c09fad0 0x7f833c07eb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:53420/0 (socket says 172.21.15.42:53420) 2019-12-05T05:03:41.313 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.300+0000 7f833bfff700 1 -- 172.21.15.42:0/2323769901 learned_addr learned my addr 172.21.15.42:0/2323769901 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:03:41.313 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.304+0000 7f833b7fe700 1 --2- 172.21.15.42:0/2323769901 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f833c0a7730 0x7f833c08df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:41.314 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.304+0000 7f833a7fc700 1 -- 172.21.15.42:0/2323769901 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 976514950 0 0) 0x7f833c0a0700 con 0x7f833c07e710 2019-12-05T05:03:41.314 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.304+0000 7f833b7fe700 1 -- 172.21.15.42:0/2323769901 >> v2:172.21.15.42:3300/0 conn(0x7f833c09fad0 msgr2=0x7f833c07eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:03:41.314 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.304+0000 7f833b7fe700 1 --2- 172.21.15.42:0/2323769901 >> v2:172.21.15.42:3300/0 conn(0x7f833c09fad0 0x7f833c07eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:03:41.314 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.304+0000 7f833b7fe700 1 -- 172.21.15.42:0/2323769901 >> v1:172.21.15.40:6789/0 conn(0x7f833c07e710 legacy=0x7f833c09f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:41.314 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.304+0000 7f833b7fe700 1 -- 172.21.15.42:0/2323769901 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f833c07f090 con 0x7f833c0a7730 2019-12-05T05:03:41.314 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.304+0000 7f833bfff700 1 --2- 172.21.15.42:0/2323769901 >> v2:172.21.15.42:3300/0 conn(0x7f833c09fad0 0x7f833c07eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:03:41.315 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.304+0000 7f833b7fe700 1 --2- 172.21.15.42:0/2323769901 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f833c0a7730 0x7f833c08df70 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f832c008790 tx=0x7f832c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:41.315 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.304+0000 7f833b7fe700 1 --2- 172.21.15.42:0/2323769901 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f833c0a7730 0x7f833c08df70 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f832c008790 tx=0x7f832c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:41.315 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.304+0000 7f833b7fe700 1 --2- 172.21.15.42:0/2323769901 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f833c0a7730 0x7f833c08df70 secure :-1 s=READY pgs=34 cs=0 l=1 rx=0x7f832c008790 tx=0x7f832c00fc70).ready entity=mon.2 client_cookie=bc58fd72fb6deaa1 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:41.315 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.304+0000 7f833b7fe700 1 --2- 172.21.15.42:0/2323769901 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f833c0a7730 0x7f833c08df70 secure :-1 s=THROTTLE_DONE pgs=34 cs=0 l=1 rx=0x7f832c008790 tx=0x7f832c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:41.315 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.304+0000 7f833a7fc700 1 -- 172.21.15.42:0/2323769901 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f832c008bb0 con 0x7f833c0a7730 2019-12-05T05:03:41.315 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.304+0000 7f833b7fe700 1 --2- 172.21.15.42:0/2323769901 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f833c0a7730 0x7f833c08df70 secure :-1 s=THROTTLE_DONE pgs=34 cs=0 l=1 rx=0x7f832c008790 tx=0x7f832c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:41.315 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.304+0000 7f833b7fe700 1 --2- 172.21.15.42:0/2323769901 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f833c0a7730 0x7f833c08df70 secure :-1 s=THROTTLE_DONE pgs=34 cs=0 l=1 rx=0x7f832c008790 tx=0x7f832c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:41.316 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.304+0000 7f833a7fc700 1 -- 172.21.15.42:0/2323769901 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f832c0087d0 con 0x7f833c0a7730 2019-12-05T05:03:41.316 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.304+0000 7f833a7fc700 1 -- 172.21.15.42:0/2323769901 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f832c011860 con 0x7f833c0a7730 2019-12-05T05:03:41.316 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.304+0000 7f8341566700 1 -- 172.21.15.42:0/2323769901 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f833c0a7730 msgr2=0x7f833c08df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:41.316 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.304+0000 7f8341566700 1 --2- 172.21.15.42:0/2323769901 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f833c0a7730 0x7f833c08df70 secure :-1 s=READY pgs=34 cs=0 l=1 rx=0x7f832c008790 tx=0x7f832c00fc70).stop 2019-12-05T05:03:41.316 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.304+0000 7f8341566700 1 -- 172.21.15.42:0/2323769901 shutdown_connections 2019-12-05T05:03:41.316 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.304+0000 7f8341566700 1 --2- 172.21.15.42:0/2323769901 >> v2:172.21.15.42:3300/0 conn(0x7f833c09fad0 0x7f833c07eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:03:41.316 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.304+0000 7f8341566700 1 --2- 172.21.15.42:0/2323769901 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f833c0a7730 0x7f833c08df70 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:41.317 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.304+0000 7f8341566700 1 -- 172.21.15.42:0/2323769901 shutdown_connections 2019-12-05T05:03:41.317 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.304+0000 7f8341566700 1 -- 172.21.15.42:0/2323769901 wait complete. 2019-12-05T05:03:41.317 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.304+0000 7f8341566700 1 -- 172.21.15.42:0/2323769901 >> 172.21.15.42:0/2323769901 conn(0x7f833c0aabd0 msgr2=0x7f833c0a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:41.320 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.308+0000 7f8341566700 1 Processor -- start 2019-12-05T05:03:41.320 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.308+0000 7f8341566700 1 -- start start 2019-12-05T05:03:41.320 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.308+0000 7f8341566700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f833c11c0b0 0x7f833c0a20a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:41.321 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.308+0000 7f8341566700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f833c11a480 0x7f833c0a3bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:41.321 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.308+0000 7f8341566700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f833c11bb40 con 0x7f833c07da70 2019-12-05T05:03:41.321 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.308+0000 7f8341566700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f833c0a25f0 con 0x7f833c11a480 2019-12-05T05:03:41.321 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.308+0000 7f8341566700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f833c0a2f40 con 0x7f833c11c0b0 2019-12-05T05:03:41.321 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.308+0000 7f833b7fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f833c11c0b0 0x7f833c0a20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:41.321 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.308+0000 7f833b7fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f833c11c0b0 0x7f833c0a20a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:53424/0 (socket says 172.21.15.42:53424) 2019-12-05T05:03:41.321 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.308+0000 7f833b7fe700 1 -- 172.21.15.42:0/897960908 learned_addr learned my addr 172.21.15.42:0/897960908 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:03:41.322 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.308+0000 7f833affd700 1 --2- 172.21.15.42:0/897960908 >> v2:172.21.15.40:3300/0 conn(0x7f833c11a480 0x7f833c0a3bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:41.322 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.308+0000 7f8333fff700 1 -- 172.21.15.42:0/897960908 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 999467684 0 0) 0x7f833c11bb40 con 0x7f833c07da70 2019-12-05T05:03:41.322 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.308+0000 7f8333fff700 1 -- 172.21.15.42:0/897960908 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83200030b0 con 0x7f833c07da70 2019-12-05T05:03:41.322 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.308+0000 7f833b7fe700 1 -- 172.21.15.42:0/897960908 >> v2:172.21.15.40:3300/0 conn(0x7f833c11a480 msgr2=0x7f833c0a3bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:41.322 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.308+0000 7f833b7fe700 1 --2- 172.21.15.42:0/897960908 >> v2:172.21.15.40:3300/0 conn(0x7f833c11a480 0x7f833c0a3bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:41.322 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.308+0000 7f833b7fe700 1 -- 172.21.15.42:0/897960908 >> v1:172.21.15.40:6789/0 conn(0x7f833c07da70 legacy=0x7f833c130450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:41.322 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.308+0000 7f833b7fe700 1 -- 172.21.15.42:0/897960908 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f833c0a3870 con 0x7f833c11c0b0 2019-12-05T05:03:41.323 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.312+0000 7f833b7fe700 1 --2- 172.21.15.42:0/897960908 >> v2:172.21.15.42:3300/0 conn(0x7f833c11c0b0 0x7f833c0a20a0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f832c008790 tx=0x7f832c0119e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:41.323 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.312+0000 7f833b7fe700 1 --2- 172.21.15.42:0/897960908 >> v2:172.21.15.42:3300/0 conn(0x7f833c11c0b0 0x7f833c0a20a0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f832c008790 tx=0x7f832c0119e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:41.323 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.312+0000 7f833b7fe700 1 --2- 172.21.15.42:0/897960908 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f833c11c0b0 0x7f833c0a20a0 secure :-1 s=READY pgs=123 cs=0 l=1 rx=0x7f832c008790 tx=0x7f832c0119e0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:41.323 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.312+0000 7f833b7fe700 1 --2- 172.21.15.42:0/897960908 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f833c11c0b0 0x7f833c0a20a0 secure :-1 s=THROTTLE_DONE pgs=123 cs=0 l=1 rx=0x7f832c008790 tx=0x7f832c0119e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:41.323 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.312+0000 7f8333fff700 1 -- 172.21.15.42:0/897960908 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f832c00f280 con 0x7f833c11c0b0 2019-12-05T05:03:41.323 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.312+0000 7f833b7fe700 1 --2- 172.21.15.42:0/897960908 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f833c11c0b0 0x7f833c0a20a0 secure :-1 s=THROTTLE_DONE pgs=123 cs=0 l=1 rx=0x7f832c008790 tx=0x7f832c0119e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:41.323 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.312+0000 7f8333fff700 1 -- 172.21.15.42:0/897960908 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f832c008db0 con 0x7f833c11c0b0 2019-12-05T05:03:41.324 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.312+0000 7f833b7fe700 1 --2- 172.21.15.42:0/897960908 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f833c11c0b0 0x7f833c0a20a0 secure :-1 s=THROTTLE_DONE pgs=123 cs=0 l=1 rx=0x7f832c008790 tx=0x7f832c0119e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:41.324 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.312+0000 7f8341566700 1 -- 172.21.15.42:0/897960908 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f833c124560 con 0x7f833c11c0b0 2019-12-05T05:03:41.324 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.312+0000 7f8333fff700 1 -- 172.21.15.42:0/897960908 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f832c005540 con 0x7f833c11c0b0 2019-12-05T05:03:41.324 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.312+0000 7f8341566700 1 -- 172.21.15.42:0/897960908 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f833c0a4100 con 0x7f833c11c0b0 2019-12-05T05:03:41.324 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.312+0000 7f833b7fe700 1 --2- 172.21.15.42:0/897960908 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f833c11c0b0 0x7f833c0a20a0 secure :-1 s=THROTTLE_DONE pgs=123 cs=0 l=1 rx=0x7f832c008790 tx=0x7f832c0119e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:41.325 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.312+0000 7f8333fff700 1 -- 172.21.15.42:0/897960908 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f832c0057e0 con 0x7f833c11c0b0 2019-12-05T05:03:41.325 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.312+0000 7f833b7fe700 1 --2- 172.21.15.42:0/897960908 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f833c11c0b0 0x7f833c0a20a0 secure :-1 s=THROTTLE_DONE pgs=123 cs=0 l=1 rx=0x7f832c008790 tx=0x7f832c0119e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:41.325 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.312+0000 7f8333fff700 1 --2- 172.21.15.42:0/897960908 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f832003b280 0x7f832003d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:41.325 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.312+0000 7f8333fff700 1 -- 172.21.15.42:0/897960908 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f832c04b5d0 con 0x7f833c11c0b0 2019-12-05T05:03:41.325 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.312+0000 7f833affd700 1 --2- 172.21.15.42:0/897960908 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f832003b280 0x7f832003d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:41.325 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.312+0000 7f833affd700 1 --2- 172.21.15.42:0/897960908 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f832003b280 0x7f832003d790 crc :-1 s=READY pgs=70 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:41.326 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.316+0000 7f8341566700 1 --2- 172.21.15.42:0/897960908 >> [v2:172.21.15.40:6813/12490,v1:172.21.15.40:6816/12490] conn(0x7f833c130c50 0x7f833c1388b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:41.326 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.316+0000 7f8341566700 1 -- 172.21.15.42:0/897960908 --> [v2:172.21.15.40:6813/12490,v1:172.21.15.40:6816/12490] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f833c05b8b0 con 0x7f833c130c50 2019-12-05T05:03:41.326 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.316+0000 7f833bfff700 1 --2- 172.21.15.42:0/897960908 >> [v2:172.21.15.40:6813/12490,v1:172.21.15.40:6816/12490] conn(0x7f833c130c50 0x7f833c1388b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:41.327 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.316+0000 7f833bfff700 1 -- 172.21.15.42:0/897960908 >> [v2:172.21.15.40:6813/12490,v1:172.21.15.40:6816/12490] conn(0x7f833c130c50 msgr2=0x7f833c1388b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2019-12-05T05:03:41.327 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.316+0000 7f833bfff700 1 --2- 172.21.15.42:0/897960908 >> [v2:172.21.15.40:6813/12490,v1:172.21.15.40:6816/12490] conn(0x7f833c130c50 0x7f833c1388b0 crc :-1 s=READY pgs=7 cs=0 l=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:41.327 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.316+0000 7f8333fff700 1 -- 172.21.15.42:0/897960908 <== osd.7 v2:172.21.15.40:6813/12490 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23014 (crc 0 0 0) 0x7f833c05b8b0 con 0x7f833c130c50 2019-12-05T05:03:41.347 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.340+0000 7f8341566700 1 -- 172.21.15.42:0/897960908 --> [v2:172.21.15.40:6813/12490,v1:172.21.15.40:6816/12490] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f833c07e040 con 0x7f833c130c50 2019-12-05T05:03:41.347 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.344+0000 7f8333fff700 1 -- 172.21.15.42:0/897960908 <== osd.7 v2:172.21.15.40:6813/12490 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f833c07e040 con 0x7f833c130c50 2019-12-05T05:03:41.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.344+0000 7f8341566700 1 -- 172.21.15.42:0/897960908 >> [v2:172.21.15.40:6813/12490,v1:172.21.15.40:6816/12490] conn(0x7f833c130c50 msgr2=0x7f833c1388b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:41.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.344+0000 7f8341566700 1 --2- 172.21.15.42:0/897960908 >> [v2:172.21.15.40:6813/12490,v1:172.21.15.40:6816/12490] conn(0x7f833c130c50 0x7f833c1388b0 crc :-1 s=READY pgs=7 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:41.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.344+0000 7f8341566700 1 -- 172.21.15.42:0/897960908 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f832003b280 msgr2=0x7f832003d790 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:41.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.344+0000 7f8341566700 1 --2- 172.21.15.42:0/897960908 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f832003b280 0x7f832003d790 crc :-1 s=READY pgs=70 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:41.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.344+0000 7f8341566700 1 -- 172.21.15.42:0/897960908 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f833c11c0b0 msgr2=0x7f833c0a20a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:41.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.344+0000 7f8341566700 1 --2- 172.21.15.42:0/897960908 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f833c11c0b0 0x7f833c0a20a0 secure :-1 s=READY pgs=123 cs=0 l=1 rx=0x7f832c008790 tx=0x7f832c0119e0).stop 2019-12-05T05:03:41.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.344+0000 7f833bfff700 1 -- 172.21.15.42:0/897960908 reap_dead start 2019-12-05T05:03:41.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.344+0000 7f8341566700 1 -- 172.21.15.42:0/897960908 shutdown_connections 2019-12-05T05:03:41.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.344+0000 7f8341566700 1 --2- 172.21.15.42:0/897960908 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f833c11c0b0 0x7f833c0a20a0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:41.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.344+0000 7f8341566700 1 -- 172.21.15.42:0/897960908 shutdown_connections 2019-12-05T05:03:41.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.344+0000 7f8341566700 1 -- 172.21.15.42:0/897960908 wait complete. 2019-12-05T05:03:41.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.344+0000 7f8341566700 1 -- 172.21.15.42:0/897960908 >> 172.21.15.42:0/897960908 conn(0x7f833c0aabd0 msgr2=0x7f833c0a7f80 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:41.359 INFO:teuthology.orchestra.run.smithi042.stdout:55834574854 2019-12-05T05:03:41.359 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2019-12-05T05:03:41.468 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.456+0000 7f4fea481700 1 Processor -- start 2019-12-05T05:03:41.468 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.456+0000 7f4fea481700 1 -- start start 2019-12-05T05:03:41.468 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.456+0000 7f4fea481700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f4fe409fae0 0x7f4fe414fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:41.469 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.456+0000 7f4fea481700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fe40a0710 con 0x7f4fe408df80 2019-12-05T05:03:41.469 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.456+0000 7f4fea481700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fe407ef20 con 0x7f4fe40a7740 2019-12-05T05:03:41.469 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.456+0000 7f4fea481700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4fe40a0b00 con 0x7f4fe409fae0 2019-12-05T05:03:41.469 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.456+0000 7f4fe8a1e700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f4fe40a7740 0x7f4fe407e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:53470/0 (socket says 172.21.15.42:53470) 2019-12-05T05:03:41.469 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.460+0000 7f4fe8a1e700 1 -- 172.21.15.42:0/215122140 learned_addr learned my addr 172.21.15.42:0/215122140 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:03:41.469 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.460+0000 7f4fe921f700 1 --2- 172.21.15.42:0/215122140 >> v2:172.21.15.40:3300/0 conn(0x7f4fe409fae0 0x7f4fe414fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:41.470 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.460+0000 7f4fe37fe700 1 -- 172.21.15.42:0/215122140 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 310823619 0 0) 0x7f4fe407ef20 con 0x7f4fe40a7740 2019-12-05T05:03:41.470 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.460+0000 7f4fe37fe700 1 -- 172.21.15.42:0/215122140 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4fcc0030b0 con 0x7f4fe40a7740 2019-12-05T05:03:41.470 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.460+0000 7f4fe921f700 1 -- 172.21.15.42:0/215122140 >> v1:172.21.15.42:6789/0 conn(0x7f4fe40a7740 legacy=0x7f4fe407e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:41.471 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.460+0000 7f4fe921f700 1 -- 172.21.15.42:0/215122140 >> v1:172.21.15.40:6789/0 conn(0x7f4fe408df80 legacy=0x7f4fe409f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:41.471 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.460+0000 7f4fe37fe700 1 -- 172.21.15.42:0/215122140 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3710940610 0 0) 0x7f4fe40a0710 con 0x7f4fe408df80 2019-12-05T05:03:41.471 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.460+0000 7f4fe921f700 1 -- 172.21.15.42:0/215122140 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fe40a9ba0 con 0x7f4fe409fae0 2019-12-05T05:03:41.471 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.460+0000 7f4fe921f700 1 --2- 172.21.15.42:0/215122140 >> v2:172.21.15.40:3300/0 conn(0x7f4fe409fae0 0x7f4fe414fff0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f4fdc008b10 tx=0x7f4fdc00ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:41.471 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.460+0000 7f4fe921f700 1 --2- 172.21.15.42:0/215122140 >> v2:172.21.15.40:3300/0 conn(0x7f4fe409fae0 0x7f4fe414fff0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f4fdc008b10 tx=0x7f4fdc00ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:41.472 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.460+0000 7f4fe921f700 1 --2- 172.21.15.42:0/215122140 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4fe409fae0 0x7f4fe414fff0 secure :-1 s=READY pgs=107 cs=0 l=1 rx=0x7f4fdc008b10 tx=0x7f4fdc00ff70).ready entity=mon.0 client_cookie=7e0035e207a402d1 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:41.472 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.460+0000 7f4fe921f700 1 --2- 172.21.15.42:0/215122140 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4fe409fae0 0x7f4fe414fff0 secure :-1 s=THROTTLE_DONE pgs=107 cs=0 l=1 rx=0x7f4fdc008b10 tx=0x7f4fdc00ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:41.472 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.460+0000 7f4fe37fe700 1 -- 172.21.15.42:0/215122140 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4fdc010970 con 0x7f4fe409fae0 2019-12-05T05:03:41.472 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.460+0000 7f4fe921f700 1 --2- 172.21.15.42:0/215122140 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4fe409fae0 0x7f4fe414fff0 secure :-1 s=THROTTLE_DONE pgs=107 cs=0 l=1 rx=0x7f4fdc008b10 tx=0x7f4fdc00ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:41.472 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.460+0000 7f4fe37fe700 1 -- 172.21.15.42:0/215122140 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fdc008b50 con 0x7f4fe409fae0 2019-12-05T05:03:41.472 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.460+0000 7f4fe921f700 1 --2- 172.21.15.42:0/215122140 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4fe409fae0 0x7f4fe414fff0 secure :-1 s=THROTTLE_DONE pgs=107 cs=0 l=1 rx=0x7f4fdc008b10 tx=0x7f4fdc00ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:41.473 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.460+0000 7f4fe37fe700 1 -- 172.21.15.42:0/215122140 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4fdc0110c0 con 0x7f4fe409fae0 2019-12-05T05:03:41.473 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.460+0000 7f4fea481700 1 -- 172.21.15.42:0/215122140 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4fe409fae0 msgr2=0x7f4fe414fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:41.473 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.460+0000 7f4fea481700 1 --2- 172.21.15.42:0/215122140 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4fe409fae0 0x7f4fe414fff0 secure :-1 s=READY pgs=107 cs=0 l=1 rx=0x7f4fdc008b10 tx=0x7f4fdc00ff70).stop 2019-12-05T05:03:41.473 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.460+0000 7f4fea481700 1 -- 172.21.15.42:0/215122140 shutdown_connections 2019-12-05T05:03:41.473 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.460+0000 7f4fea481700 1 --2- 172.21.15.42:0/215122140 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4fe409fae0 0x7f4fe414fff0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:41.473 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.460+0000 7f4fea481700 1 -- 172.21.15.42:0/215122140 shutdown_connections 2019-12-05T05:03:41.474 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.460+0000 7f4fea481700 1 -- 172.21.15.42:0/215122140 wait complete. 2019-12-05T05:03:41.474 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.460+0000 7f4fea481700 1 -- 172.21.15.42:0/215122140 >> 172.21.15.42:0/215122140 conn(0x7f4fe40aabe0 msgr2=0x7f4fe40a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:41.478 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.464+0000 7f4fea481700 1 Processor -- start 2019-12-05T05:03:41.478 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.464+0000 7f4fea481700 1 -- start start 2019-12-05T05:03:41.479 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.464+0000 7f4fea481700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f4fe41206a0 0x7f4fe41175d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:41.479 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.464+0000 7f4fea481700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f4fe41182b0 0x7f4fe407d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:41.479 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.464+0000 7f4fea481700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fe4123720 con 0x7f4fe407ddf0 2019-12-05T05:03:41.479 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.464+0000 7f4fea481700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4fe4117c60 con 0x7f4fe41182b0 2019-12-05T05:03:41.479 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.464+0000 7f4fea481700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4fe4117f00 con 0x7f4fe41206a0 2019-12-05T05:03:41.479 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.464+0000 7f4fe8a1e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f4fe41206a0 0x7f4fe41175d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:41.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.464+0000 7f4fe8a1e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f4fe41206a0 0x7f4fe41175d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:53438/0 (socket says 172.21.15.42:53438) 2019-12-05T05:03:41.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.464+0000 7f4fe8a1e700 1 -- 172.21.15.42:0/2227779099 learned_addr learned my addr 172.21.15.42:0/2227779099 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:03:41.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.464+0000 7f4fe3fff700 1 --2- 172.21.15.42:0/2227779099 >> v2:172.21.15.40:3300/0 conn(0x7f4fe41182b0 0x7f4fe407d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:41.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.468+0000 7f4fe3fff700 1 -- 172.21.15.42:0/2227779099 >> v2:172.21.15.42:3300/0 conn(0x7f4fe41206a0 msgr2=0x7f4fe41175d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:41.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.468+0000 7f4fe3fff700 1 --2- 172.21.15.42:0/2227779099 >> v2:172.21.15.42:3300/0 conn(0x7f4fe41206a0 0x7f4fe41175d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:41.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.468+0000 7f4fe3fff700 1 -- 172.21.15.42:0/2227779099 >> v1:172.21.15.40:6789/0 conn(0x7f4fe407ddf0 legacy=0x7f4fe4121b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:41.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.468+0000 7f4fe3fff700 1 -- 172.21.15.42:0/2227779099 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fe4122330 con 0x7f4fe41182b0 2019-12-05T05:03:41.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.468+0000 7f4fe17fa700 1 -- 172.21.15.42:0/2227779099 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2617451116 0 0) 0x7f4fdc0110c0 con 0x7f4fe407ddf0 2019-12-05T05:03:41.482 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.468+0000 7f4fe8a1e700 1 --2- 172.21.15.42:0/2227779099 >> v2:172.21.15.42:3300/0 conn(0x7f4fe41206a0 0x7f4fe41175d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:03:41.482 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.468+0000 7f4fe3fff700 1 --2- 172.21.15.42:0/2227779099 >> v2:172.21.15.40:3300/0 conn(0x7f4fe41182b0 0x7f4fe407d8a0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f4fe4117e20 tx=0x7f4fd8010840).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:41.482 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.468+0000 7f4fe3fff700 1 --2- 172.21.15.42:0/2227779099 >> v2:172.21.15.40:3300/0 conn(0x7f4fe41182b0 0x7f4fe407d8a0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f4fe4117e20 tx=0x7f4fd8010840).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:41.482 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.468+0000 7f4fe3fff700 1 --2- 172.21.15.42:0/2227779099 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4fe41182b0 0x7f4fe407d8a0 secure :-1 s=READY pgs=109 cs=0 l=1 rx=0x7f4fe4117e20 tx=0x7f4fd8010840).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:41.482 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.468+0000 7f4fe3fff700 1 --2- 172.21.15.42:0/2227779099 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4fe41182b0 0x7f4fe407d8a0 secure :-1 s=THROTTLE_DONE pgs=109 cs=0 l=1 rx=0x7f4fe4117e20 tx=0x7f4fd8010840).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:41.482 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.468+0000 7f4fe17fa700 1 -- 172.21.15.42:0/2227779099 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4fd8006b30 con 0x7f4fe41182b0 2019-12-05T05:03:41.483 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.468+0000 7f4fe3fff700 1 --2- 172.21.15.42:0/2227779099 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4fe41182b0 0x7f4fe407d8a0 secure :-1 s=THROTTLE_DONE pgs=109 cs=0 l=1 rx=0x7f4fe4117e20 tx=0x7f4fd8010840).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:41.483 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.468+0000 7f4fe17fa700 1 -- 172.21.15.42:0/2227779099 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fd8006750 con 0x7f4fe41182b0 2019-12-05T05:03:41.483 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.468+0000 7f4fe3fff700 1 --2- 172.21.15.42:0/2227779099 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4fe41182b0 0x7f4fe407d8a0 secure :-1 s=THROTTLE_DONE pgs=109 cs=0 l=1 rx=0x7f4fe4117e20 tx=0x7f4fd8010840).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:41.483 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.468+0000 7f4fea481700 1 -- 172.21.15.42:0/2227779099 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fe4122600 con 0x7f4fe41182b0 2019-12-05T05:03:41.483 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.468+0000 7f4fe17fa700 1 -- 172.21.15.42:0/2227779099 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4fd8011040 con 0x7f4fe41182b0 2019-12-05T05:03:41.483 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.468+0000 7f4fea481700 1 -- 172.21.15.42:0/2227779099 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fe409f500 con 0x7f4fe41182b0 2019-12-05T05:03:41.484 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.468+0000 7f4fe3fff700 1 --2- 172.21.15.42:0/2227779099 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4fe41182b0 0x7f4fe407d8a0 secure :-1 s=THROTTLE_DONE pgs=109 cs=0 l=1 rx=0x7f4fe4117e20 tx=0x7f4fd8010840).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:41.484 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.468+0000 7f4fe17fa700 1 -- 172.21.15.42:0/2227779099 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f4fd8011360 con 0x7f4fe41182b0 2019-12-05T05:03:41.484 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.468+0000 7f4fe3fff700 1 --2- 172.21.15.42:0/2227779099 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4fe41182b0 0x7f4fe407d8a0 secure :-1 s=THROTTLE_DONE pgs=109 cs=0 l=1 rx=0x7f4fe4117e20 tx=0x7f4fd8010840).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:41.484 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.468+0000 7f4fe17fa700 1 --2- 172.21.15.42:0/2227779099 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4fc403aca0 0x7f4fc403d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:41.484 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.468+0000 7f4fe17fa700 1 -- 172.21.15.42:0/2227779099 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f4fd804b110 con 0x7f4fe41182b0 2019-12-05T05:03:41.484 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.468+0000 7f4fe8a1e700 1 --2- 172.21.15.42:0/2227779099 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4fc403aca0 0x7f4fc403d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:41.485 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.468+0000 7f4fe8a1e700 1 --2- 172.21.15.42:0/2227779099 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4fc403aca0 0x7f4fc403d1b0 crc :-1 s=READY pgs=71 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:41.485 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.472+0000 7f4fea481700 1 -- 172.21.15.42:0/2227779099 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4fe405d110 con 0x7f4fe41182b0 2019-12-05T05:03:41.486 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.480+0000 7f4fe3fff700 1 --2- 172.21.15.42:0/2227779099 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4fe41182b0 0x7f4fe407d8a0 secure :-1 s=THROTTLE_DONE pgs=109 cs=0 l=1 rx=0x7f4fe4117e20 tx=0x7f4fd8010840).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:41.486 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.480+0000 7f4fe17fa700 1 -- 172.21.15.42:0/2227779099 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f4fd8032170 con 0x7f4fe41182b0 2019-12-05T05:03:41.671 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.664+0000 7f4fea481700 1 -- 172.21.15.42:0/2227779099 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f4fe41197d0 con 0x7f4fe41182b0 2019-12-05T05:03:41.672 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.664+0000 7f4fe3fff700 1 --2- 172.21.15.42:0/2227779099 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4fe41182b0 0x7f4fe407d8a0 secure :-1 s=THROTTLE_DONE pgs=109 cs=0 l=1 rx=0x7f4fe4117e20 tx=0x7f4fd8010840).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:41.672 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.668+0000 7f4fe17fa700 1 -- 172.21.15.42:0/2227779099 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f4fd80323d0 con 0x7f4fe41182b0 2019-12-05T05:03:41.672 INFO:teuthology.orchestra.run.smithi042.stdout:55834574855 2019-12-05T05:03:41.675 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.668+0000 7f4fea481700 1 -- 172.21.15.42:0/2227779099 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4fc403aca0 msgr2=0x7f4fc403d1b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:41.675 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.668+0000 7f4fea481700 1 --2- 172.21.15.42:0/2227779099 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4fc403aca0 0x7f4fc403d1b0 crc :-1 s=READY pgs=71 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:41.675 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.668+0000 7f4fea481700 1 -- 172.21.15.42:0/2227779099 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4fe41182b0 msgr2=0x7f4fe407d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:41.675 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.668+0000 7f4fea481700 1 --2- 172.21.15.42:0/2227779099 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4fe41182b0 0x7f4fe407d8a0 secure :-1 s=READY pgs=109 cs=0 l=1 rx=0x7f4fe4117e20 tx=0x7f4fd8010840).stop 2019-12-05T05:03:41.676 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.668+0000 7f4fea481700 1 -- 172.21.15.42:0/2227779099 shutdown_connections 2019-12-05T05:03:41.676 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.668+0000 7f4fea481700 1 --2- 172.21.15.42:0/2227779099 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4fc403aca0 0x7f4fc403d1b0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:41.676 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.668+0000 7f4fea481700 1 --2- 172.21.15.42:0/2227779099 >> v2:172.21.15.42:3300/0 conn(0x7f4fe41206a0 0x7f4fe41175d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:41.676 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.668+0000 7f4fea481700 1 --2- 172.21.15.42:0/2227779099 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4fe41182b0 0x7f4fe407d8a0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:41.676 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.668+0000 7f4fea481700 1 -- 172.21.15.42:0/2227779099 shutdown_connections 2019-12-05T05:03:41.676 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.668+0000 7f4fea481700 1 -- 172.21.15.42:0/2227779099 wait complete. 2019-12-05T05:03:41.676 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.668+0000 7f4fea481700 1 -- 172.21.15.42:0/2227779099 >> 172.21.15.42:0/2227779099 conn(0x7f4fe408fef0 msgr2=0x7f4fe4059ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:41.684 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.0 2019-12-05T05:03:41.685 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2019-12-05T05:03:41.785 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.776+0000 7f5bc1311700 1 Processor -- start 2019-12-05T05:03:41.786 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.776+0000 7f5bc1311700 1 -- start start 2019-12-05T05:03:41.786 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.776+0000 7f5bc1311700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5bbc0a7740 0x7f5bbc08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:41.786 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.776+0000 7f5bc1311700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5bbc09fae0 0x7f5bbc07eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:41.786 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.776+0000 7f5bc1311700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bbc0a0710 con 0x7f5bbc07e720 2019-12-05T05:03:41.786 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.776+0000 7f5bc1311700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5bbc0a0b00 con 0x7f5bbc0a7740 2019-12-05T05:03:41.786 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.776+0000 7f5bc1311700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bbc08e5b0 con 0x7f5bbc09fae0 2019-12-05T05:03:41.787 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.776+0000 7f5bbaffd700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f5bbc07e720 0x7f5bbc09f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:53484/0 (socket says 172.21.15.42:53484) 2019-12-05T05:03:41.787 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.776+0000 7f5bbaffd700 1 -- 172.21.15.42:0/1062911901 learned_addr learned my addr 172.21.15.42:0/1062911901 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:03:41.787 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.776+0000 7f5bbbfff700 1 --2- 172.21.15.42:0/1062911901 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5bbc09fae0 0x7f5bbc07eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:41.787 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.776+0000 7f5bbb7fe700 1 --2- 172.21.15.42:0/1062911901 >> v2:172.21.15.40:3300/0 conn(0x7f5bbc0a7740 0x7f5bbc08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:41.787 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.776+0000 7f5bba7fc700 1 -- 172.21.15.42:0/1062911901 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4125757740 0 0) 0x7f5bbc0a0710 con 0x7f5bbc07e720 2019-12-05T05:03:41.787 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.776+0000 7f5bba7fc700 1 -- 172.21.15.42:0/1062911901 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ba40030b0 con 0x7f5bbc07e720 2019-12-05T05:03:41.788 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.776+0000 7f5bbb7fe700 1 -- 172.21.15.42:0/1062911901 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5bbc09fae0 msgr2=0x7f5bbc07eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:03:41.788 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.776+0000 7f5bbb7fe700 1 --2- 172.21.15.42:0/1062911901 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5bbc09fae0 0x7f5bbc07eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:03:41.788 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.776+0000 7f5bbb7fe700 1 -- 172.21.15.42:0/1062911901 >> v1:172.21.15.42:6789/0 conn(0x7f5bbc07e720 legacy=0x7f5bbc09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:41.788 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.776+0000 7f5bbb7fe700 1 -- 172.21.15.42:0/1062911901 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bbc07f0a0 con 0x7f5bbc0a7740 2019-12-05T05:03:41.788 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.776+0000 7f5bbbfff700 1 --2- 172.21.15.42:0/1062911901 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5bbc09fae0 0x7f5bbc07eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:03:41.788 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.776+0000 7f5bbb7fe700 1 --2- 172.21.15.42:0/1062911901 >> v2:172.21.15.40:3300/0 conn(0x7f5bbc0a7740 0x7f5bbc08df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f5bb0008790 tx=0x7f5bb000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:41.789 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.776+0000 7f5bbb7fe700 1 --2- 172.21.15.42:0/1062911901 >> v2:172.21.15.40:3300/0 conn(0x7f5bbc0a7740 0x7f5bbc08df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f5bb0008790 tx=0x7f5bb000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:41.789 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.776+0000 7f5bbb7fe700 1 --2- 172.21.15.42:0/1062911901 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5bbc0a7740 0x7f5bbc08df80 secure :-1 s=READY pgs=110 cs=0 l=1 rx=0x7f5bb0008790 tx=0x7f5bb000fc70).ready entity=mon.0 client_cookie=76571faa4a65286b server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:41.789 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.776+0000 7f5bbb7fe700 1 --2- 172.21.15.42:0/1062911901 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5bbc0a7740 0x7f5bbc08df80 secure :-1 s=THROTTLE_DONE pgs=110 cs=0 l=1 rx=0x7f5bb0008790 tx=0x7f5bb000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:41.789 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.776+0000 7f5bba7fc700 1 -- 172.21.15.42:0/1062911901 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5bb0008bb0 con 0x7f5bbc0a7740 2019-12-05T05:03:41.789 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.776+0000 7f5bbb7fe700 1 --2- 172.21.15.42:0/1062911901 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5bbc0a7740 0x7f5bbc08df80 secure :-1 s=THROTTLE_DONE pgs=110 cs=0 l=1 rx=0x7f5bb0008790 tx=0x7f5bb000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:41.789 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.776+0000 7f5bbb7fe700 1 --2- 172.21.15.42:0/1062911901 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5bbc0a7740 0x7f5bbc08df80 secure :-1 s=THROTTLE_DONE pgs=110 cs=0 l=1 rx=0x7f5bb0008790 tx=0x7f5bb000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:41.789 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.776+0000 7f5bba7fc700 1 -- 172.21.15.42:0/1062911901 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bb00087d0 con 0x7f5bbc0a7740 2019-12-05T05:03:41.790 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.780+0000 7f5bba7fc700 1 -- 172.21.15.42:0/1062911901 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5bb0011860 con 0x7f5bbc0a7740 2019-12-05T05:03:41.790 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.780+0000 7f5bc1311700 1 -- 172.21.15.42:0/1062911901 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5bbc0a7740 msgr2=0x7f5bbc08df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:41.790 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.780+0000 7f5bc1311700 1 --2- 172.21.15.42:0/1062911901 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5bbc0a7740 0x7f5bbc08df80 secure :-1 s=READY pgs=110 cs=0 l=1 rx=0x7f5bb0008790 tx=0x7f5bb000fc70).stop 2019-12-05T05:03:41.790 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.780+0000 7f5bc1311700 1 -- 172.21.15.42:0/1062911901 shutdown_connections 2019-12-05T05:03:41.790 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.780+0000 7f5bc1311700 1 --2- 172.21.15.42:0/1062911901 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5bbc09fae0 0x7f5bbc07eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:03:41.790 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.780+0000 7f5bc1311700 1 --2- 172.21.15.42:0/1062911901 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5bbc0a7740 0x7f5bbc08df80 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:41.790 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.780+0000 7f5bc1311700 1 -- 172.21.15.42:0/1062911901 shutdown_connections 2019-12-05T05:03:41.791 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.780+0000 7f5bc1311700 1 -- 172.21.15.42:0/1062911901 wait complete. 2019-12-05T05:03:41.791 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.780+0000 7f5bc1311700 1 -- 172.21.15.42:0/1062911901 >> 172.21.15.42:0/1062911901 conn(0x7f5bbc0aabe0 msgr2=0x7f5bbc0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:41.794 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.780+0000 7f5bc1311700 1 Processor -- start 2019-12-05T05:03:41.794 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.780+0000 7f5bc1311700 1 -- start start 2019-12-05T05:03:41.794 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.784+0000 7f5bc1311700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5bbc0a3ae0 0x7f5bbc11bed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:41.795 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.784+0000 7f5bc1311700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bbc0a4440 con 0x7f5bbc123c90 2019-12-05T05:03:41.795 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.784+0000 7f5bc1311700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bbc130440 con 0x7f5bbc124850 2019-12-05T05:03:41.795 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.784+0000 7f5bc1311700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bbc0a3f10 con 0x7f5bbc0a3ae0 2019-12-05T05:03:41.795 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.784+0000 7f5bbb7fe700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f5bbc124850 0x7f5bbc07daa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:53490/0 (socket says 172.21.15.42:53490) 2019-12-05T05:03:41.795 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.784+0000 7f5bbb7fe700 1 -- 172.21.15.42:0/3954732347 learned_addr learned my addr 172.21.15.42:0/3954732347 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:03:41.795 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.784+0000 7f5bbbfff700 1 --2- 172.21.15.42:0/3954732347 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5bbc0a3ae0 0x7f5bbc11bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:41.795 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.784+0000 7f5ba3fff700 1 -- 172.21.15.42:0/3954732347 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3120694701 0 0) 0x7f5bbc130440 con 0x7f5bbc124850 2019-12-05T05:03:41.796 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.784+0000 7f5ba3fff700 1 -- 172.21.15.42:0/3954732347 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b9c0030b0 con 0x7f5bbc124850 2019-12-05T05:03:41.796 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.784+0000 7f5ba3fff700 1 -- 172.21.15.42:0/3954732347 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1644583486 0 0) 0x7f5bbc0a4440 con 0x7f5bbc123c90 2019-12-05T05:03:41.796 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.784+0000 7f5ba3fff700 1 -- 172.21.15.42:0/3954732347 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5bbc130440 con 0x7f5bbc123c90 2019-12-05T05:03:41.796 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.784+0000 7f5bbbfff700 1 -- 172.21.15.42:0/3954732347 >> v1:172.21.15.42:6789/0 conn(0x7f5bbc124850 legacy=0x7f5bbc07daa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:41.796 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.784+0000 7f5bbbfff700 1 -- 172.21.15.42:0/3954732347 >> v1:172.21.15.40:6789/0 conn(0x7f5bbc123c90 legacy=0x7f5bbc135880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:41.796 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.784+0000 7f5bbbfff700 1 -- 172.21.15.42:0/3954732347 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bbc130670 con 0x7f5bbc0a3ae0 2019-12-05T05:03:41.796 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.784+0000 7f5bbbfff700 1 --2- 172.21.15.42:0/3954732347 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5bbc0a3ae0 0x7f5bbc11bed0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f5bb40032b0 tx=0x7f5bb4010e80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:41.797 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.784+0000 7f5bbbfff700 1 --2- 172.21.15.42:0/3954732347 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5bbc0a3ae0 0x7f5bbc11bed0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f5bb40032b0 tx=0x7f5bb4010e80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:41.797 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.784+0000 7f5bbbfff700 1 --2- 172.21.15.42:0/3954732347 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5bbc0a3ae0 0x7f5bbc11bed0 secure :-1 s=READY pgs=35 cs=0 l=1 rx=0x7f5bb40032b0 tx=0x7f5bb4010e80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:41.797 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.784+0000 7f5bbbfff700 1 --2- 172.21.15.42:0/3954732347 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5bbc0a3ae0 0x7f5bbc11bed0 secure :-1 s=THROTTLE_DONE pgs=35 cs=0 l=1 rx=0x7f5bb40032b0 tx=0x7f5bb4010e80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:41.797 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.784+0000 7f5ba3fff700 1 -- 172.21.15.42:0/3954732347 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5bb40058c0 con 0x7f5bbc0a3ae0 2019-12-05T05:03:41.797 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.784+0000 7f5bbbfff700 1 --2- 172.21.15.42:0/3954732347 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5bbc0a3ae0 0x7f5bbc11bed0 secure :-1 s=THROTTLE_DONE pgs=35 cs=0 l=1 rx=0x7f5bb40032b0 tx=0x7f5bb4010e80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:41.797 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.784+0000 7f5ba3fff700 1 -- 172.21.15.42:0/3954732347 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bb400e160 con 0x7f5bbc0a3ae0 2019-12-05T05:03:41.797 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.784+0000 7f5bbbfff700 1 --2- 172.21.15.42:0/3954732347 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5bbc0a3ae0 0x7f5bbc11bed0 secure :-1 s=THROTTLE_DONE pgs=35 cs=0 l=1 rx=0x7f5bb40032b0 tx=0x7f5bb4010e80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:41.798 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.784+0000 7f5bc1311700 1 -- 172.21.15.42:0/3954732347 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bbc08eb70 con 0x7f5bbc0a3ae0 2019-12-05T05:03:41.798 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.784+0000 7f5ba3fff700 1 -- 172.21.15.42:0/3954732347 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5bb400c8a0 con 0x7f5bbc0a3ae0 2019-12-05T05:03:41.798 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.784+0000 7f5bc1311700 1 -- 172.21.15.42:0/3954732347 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bbc08f0c0 con 0x7f5bbc0a3ae0 2019-12-05T05:03:41.798 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.784+0000 7f5bbbfff700 1 --2- 172.21.15.42:0/3954732347 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5bbc0a3ae0 0x7f5bbc11bed0 secure :-1 s=THROTTLE_DONE pgs=35 cs=0 l=1 rx=0x7f5bb40032b0 tx=0x7f5bb4010e80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:41.798 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.788+0000 7f5ba3fff700 1 -- 172.21.15.42:0/3954732347 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f5bb400cbc0 con 0x7f5bbc0a3ae0 2019-12-05T05:03:41.798 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.788+0000 7f5bbbfff700 1 --2- 172.21.15.42:0/3954732347 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5bbc0a3ae0 0x7f5bbc11bed0 secure :-1 s=THROTTLE_DONE pgs=35 cs=0 l=1 rx=0x7f5bb40032b0 tx=0x7f5bb4010e80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:41.798 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.788+0000 7f5ba3fff700 1 --2- 172.21.15.42:0/3954732347 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5b9c03b400 0x7f5b9c03d910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:41.799 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.788+0000 7f5ba3fff700 1 -- 172.21.15.42:0/3954732347 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f5bb404ae70 con 0x7f5bbc0a3ae0 2019-12-05T05:03:41.799 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.788+0000 7f5bbb7fe700 1 --2- 172.21.15.42:0/3954732347 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5b9c03b400 0x7f5b9c03d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:41.799 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.788+0000 7f5bbb7fe700 1 --2- 172.21.15.42:0/3954732347 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5b9c03b400 0x7f5b9c03d910 crc :-1 s=READY pgs=72 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:41.799 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.788+0000 7f5bc1311700 1 -- 172.21.15.42:0/3954732347 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5bbc05d110 con 0x7f5bbc0a3ae0 2019-12-05T05:03:41.805 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.800+0000 7f5bbbfff700 1 --2- 172.21.15.42:0/3954732347 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5bbc0a3ae0 0x7f5bbc11bed0 secure :-1 s=THROTTLE_DONE pgs=35 cs=0 l=1 rx=0x7f5bb40032b0 tx=0x7f5bb4010e80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:41.805 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.800+0000 7f5ba3fff700 1 -- 172.21.15.42:0/3954732347 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f5bb4031ed0 con 0x7f5bbc0a3ae0 2019-12-05T05:03:41.988 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.984+0000 7f5bc1311700 1 -- 172.21.15.42:0/3954732347 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f5bbc05b8c0 con 0x7f5bbc0a3ae0 2019-12-05T05:03:41.989 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.984+0000 7f5bbbfff700 1 --2- 172.21.15.42:0/3954732347 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5bbc0a3ae0 0x7f5bbc11bed0 secure :-1 s=THROTTLE_DONE pgs=35 cs=0 l=1 rx=0x7f5bb40032b0 tx=0x7f5bb4010e80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:41.989 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.984+0000 7f5ba3fff700 1 -- 172.21.15.42:0/3954732347 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f5bb4032130 con 0x7f5bbc0a3ae0 2019-12-05T05:03:41.990 INFO:teuthology.orchestra.run.smithi042.stdout:55834574853 2019-12-05T05:03:41.992 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.984+0000 7f5bc1311700 1 -- 172.21.15.42:0/3954732347 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5b9c03b400 msgr2=0x7f5b9c03d910 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:41.992 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.984+0000 7f5bc1311700 1 --2- 172.21.15.42:0/3954732347 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5b9c03b400 0x7f5b9c03d910 crc :-1 s=READY pgs=72 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:41.992 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.984+0000 7f5bc1311700 1 -- 172.21.15.42:0/3954732347 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5bbc0a3ae0 msgr2=0x7f5bbc11bed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:41.993 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.984+0000 7f5bc1311700 1 --2- 172.21.15.42:0/3954732347 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5bbc0a3ae0 0x7f5bbc11bed0 secure :-1 s=READY pgs=35 cs=0 l=1 rx=0x7f5bb40032b0 tx=0x7f5bb4010e80).stop 2019-12-05T05:03:41.993 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.988+0000 7f5bc1311700 1 -- 172.21.15.42:0/3954732347 shutdown_connections 2019-12-05T05:03:41.993 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.988+0000 7f5bc1311700 1 --2- 172.21.15.42:0/3954732347 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5b9c03b400 0x7f5b9c03d910 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:41.993 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.988+0000 7f5bc1311700 1 --2- 172.21.15.42:0/3954732347 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5bbc0a3ae0 0x7f5bbc11bed0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:41.993 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.988+0000 7f5bc1311700 1 -- 172.21.15.42:0/3954732347 shutdown_connections 2019-12-05T05:03:41.993 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.988+0000 7f5bc1311700 1 -- 172.21.15.42:0/3954732347 wait complete. 2019-12-05T05:03:41.993 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:41.988+0000 7f5bc1311700 1 -- 172.21.15.42:0/3954732347 >> 172.21.15.42:0/3954732347 conn(0x7f5bbc0aabe0 msgr2=0x7f5bbc0a9640 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:42.002 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574853 for osd.1 2019-12-05T05:03:42.002 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2019-12-05T05:03:42.105 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.092+0000 7f73b32fe700 1 Processor -- start 2019-12-05T05:03:42.105 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.092+0000 7f73b32fe700 1 -- start start 2019-12-05T05:03:42.105 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.092+0000 7f73b32fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f73ac08df70 0x7f73ac09f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:42.106 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.092+0000 7f73b32fe700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73ac09fc50 con 0x7f73ac09f820 2019-12-05T05:03:42.106 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.092+0000 7f73b32fe700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73ac0a0700 con 0x7f73ac0a7730 2019-12-05T05:03:42.106 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.092+0000 7f73b32fe700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73ac0a0af0 con 0x7f73ac08df70 2019-12-05T05:03:42.106 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.092+0000 7f73b109a700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f73ac08df70 0x7f73ac09f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:42.107 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.092+0000 7f73b109a700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f73ac08df70 0x7f73ac09f2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:53462/0 (socket says 172.21.15.42:53462) 2019-12-05T05:03:42.107 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.092+0000 7f73b109a700 1 -- 172.21.15.42:0/3606848076 learned_addr learned my addr 172.21.15.42:0/3606848076 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:03:42.107 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.096+0000 7f73b0899700 1 -- 172.21.15.42:0/3606848076 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3458750275 0 0) 0x7f73ac09fc50 con 0x7f73ac09f820 2019-12-05T05:03:42.107 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.096+0000 7f73b0899700 1 -- 172.21.15.42:0/3606848076 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73940030b0 con 0x7f73ac09f820 2019-12-05T05:03:42.107 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.096+0000 7f73b109a700 1 -- 172.21.15.42:0/3606848076 >> v1:172.21.15.42:6789/0 conn(0x7f73ac0a7730 legacy=0x7f73ac07e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:42.107 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.096+0000 7f73b109a700 1 -- 172.21.15.42:0/3606848076 >> v1:172.21.15.40:6789/0 conn(0x7f73ac09f820 legacy=0x7f73ac14efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:42.107 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.096+0000 7f73b109a700 1 -- 172.21.15.42:0/3606848076 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73ac07ef10 con 0x7f73ac08df70 2019-12-05T05:03:42.108 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.096+0000 7f73b109a700 1 --2- 172.21.15.42:0/3606848076 >> v2:172.21.15.42:3300/0 conn(0x7f73ac08df70 0x7f73ac09f2d0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f73a0008790 tx=0x7f73a000fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:42.108 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.096+0000 7f73b109a700 1 --2- 172.21.15.42:0/3606848076 >> v2:172.21.15.42:3300/0 conn(0x7f73ac08df70 0x7f73ac09f2d0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f73a0008790 tx=0x7f73a000fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:42.108 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.096+0000 7f73b109a700 1 --2- 172.21.15.42:0/3606848076 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f73ac08df70 0x7f73ac09f2d0 secure :-1 s=READY pgs=128 cs=0 l=1 rx=0x7f73a0008790 tx=0x7f73a000fcb0).ready entity=mon.1 client_cookie=3ef47dc14f09f48 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:42.108 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.096+0000 7f73b109a700 1 -- 172.21.15.42:0/3606848076 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f73ac08df70 msgr2=0x7f73ac09f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2019-12-05T05:03:42.108 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.096+0000 7f73b109a700 1 -- 172.21.15.42:0/3606848076 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f73ac08df70 msgr2=0x7f73ac09f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2019-12-05T05:03:42.108 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.096+0000 7f73b109a700 1 --2- 172.21.15.42:0/3606848076 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f73ac08df70 0x7f73ac09f2d0 secure :-1 s=READY pgs=128 cs=0 l=1 rx=0x7f73a0008790 tx=0x7f73a000fcb0).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-12-05T05:03:42.108 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.096+0000 7f73b109a700 1 --2- 172.21.15.42:0/3606848076 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f73ac08df70 0x7f73ac09f2d0 secure :-1 s=READY pgs=128 cs=0 l=1 rx=0x7f73a0008790 tx=0x7f73a000fcb0).stop 2019-12-05T05:03:42.109 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.096+0000 7f73b0899700 1 -- 172.21.15.42:0/3606848076 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f73ac08df70 msgr2=0x7f73ac09f2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2019-12-05T05:03:42.109 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.096+0000 7f73b0899700 1 --2- 172.21.15.42:0/3606848076 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f73ac08df70 0x7f73ac09f2d0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:42.109 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.096+0000 7f73b0899700 1 --2- 172.21.15.42:0/3606848076 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7394003420 0x7f7394003850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:42.109 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.096+0000 7f73b0899700 1 --2- 172.21.15.42:0/3606848076 >> v2:172.21.15.40:3300/0 conn(0x7f7394003e50 0x7f7394004420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:42.109 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.096+0000 7f73b0899700 1 -- 172.21.15.42:0/3606848076 --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7394004a30 con 0x7f7394003e50 2019-12-05T05:03:42.109 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.096+0000 7f73b0899700 1 -- 172.21.15.42:0/3606848076 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7394004c20 con 0x7f7394003420 2019-12-05T05:03:42.109 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.096+0000 7f73b109a700 1 --2- 172.21.15.42:0/3606848076 >> v2:172.21.15.40:3300/0 conn(0x7f7394003e50 0x7f7394004420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:42.110 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.096+0000 7f73b189b700 1 --2- 172.21.15.42:0/3606848076 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7394003420 0x7f7394003850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:42.110 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.096+0000 7f73b189b700 1 -- 172.21.15.42:0/3606848076 >> v2:172.21.15.40:3300/0 conn(0x7f7394003e50 msgr2=0x7f7394004420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:03:42.110 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.096+0000 7f73b189b700 1 --2- 172.21.15.42:0/3606848076 >> v2:172.21.15.40:3300/0 conn(0x7f7394003e50 0x7f7394004420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:03:42.110 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.096+0000 7f73b189b700 1 -- 172.21.15.42:0/3606848076 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7394004ed0 con 0x7f7394003420 2019-12-05T05:03:42.110 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.096+0000 7f73b109a700 1 --2- 172.21.15.42:0/3606848076 >> v2:172.21.15.40:3300/0 conn(0x7f7394003e50 0x7f7394004420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:03:42.110 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.096+0000 7f73b189b700 1 --2- 172.21.15.42:0/3606848076 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7394003420 0x7f7394003850 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f739c001000 tx=0x7f739c00fc20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:42.110 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.096+0000 7f73b189b700 1 --2- 172.21.15.42:0/3606848076 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7394003420 0x7f7394003850 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f739c001000 tx=0x7f739c00fc20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:42.111 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.096+0000 7f73b189b700 1 --2- 172.21.15.42:0/3606848076 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7394003420 0x7f7394003850 secure :-1 s=READY pgs=36 cs=0 l=1 rx=0x7f739c001000 tx=0x7f739c00fc20).ready entity=mon.2 client_cookie=4e01d96d7ecd38c4 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:42.111 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.096+0000 7f73b189b700 1 --2- 172.21.15.42:0/3606848076 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7394003420 0x7f7394003850 secure :-1 s=THROTTLE_DONE pgs=36 cs=0 l=1 rx=0x7f739c001000 tx=0x7f739c00fc20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:42.111 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.096+0000 7f73b0899700 1 -- 172.21.15.42:0/3606848076 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f739c006a40 con 0x7f7394003420 2019-12-05T05:03:42.111 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.096+0000 7f73b189b700 1 --2- 172.21.15.42:0/3606848076 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7394003420 0x7f7394003850 secure :-1 s=THROTTLE_DONE pgs=36 cs=0 l=1 rx=0x7f739c001000 tx=0x7f739c00fc20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:42.111 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.096+0000 7f73b0899700 1 -- 172.21.15.42:0/3606848076 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f739c006660 con 0x7f7394003420 2019-12-05T05:03:42.111 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.096+0000 7f73b32fe700 1 -- 172.21.15.42:0/3606848076 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7394003420 msgr2=0x7f7394003850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:42.111 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.096+0000 7f73b32fe700 1 --2- 172.21.15.42:0/3606848076 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7394003420 0x7f7394003850 secure :-1 s=READY pgs=36 cs=0 l=1 rx=0x7f739c001000 tx=0x7f739c00fc20).stop 2019-12-05T05:03:42.112 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.096+0000 7f73b209c700 1 -- 172.21.15.42:0/3606848076 reap_dead start 2019-12-05T05:03:42.112 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.096+0000 7f73b32fe700 1 -- 172.21.15.42:0/3606848076 shutdown_connections 2019-12-05T05:03:42.112 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.096+0000 7f73b32fe700 1 --2- 172.21.15.42:0/3606848076 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f73ac08df70 0x7f73ac09f2d0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:42.112 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.096+0000 7f73b32fe700 1 -- 172.21.15.42:0/3606848076 shutdown_connections 2019-12-05T05:03:42.112 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.096+0000 7f73b32fe700 1 -- 172.21.15.42:0/3606848076 wait complete. 2019-12-05T05:03:42.112 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.096+0000 7f73b32fe700 1 -- 172.21.15.42:0/3606848076 >> 172.21.15.42:0/3606848076 conn(0x7f73ac0aabd0 msgr2=0x7f73ac0a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:42.112 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.100+0000 7f73b32fe700 1 Processor -- start 2019-12-05T05:03:42.113 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.100+0000 7f73b32fe700 1 -- start start 2019-12-05T05:03:42.116 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.100+0000 7f73b32fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f73ac1179c0 0x7f73ac11bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:42.116 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.100+0000 7f73b32fe700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f73ac132710 0x7f73ac132b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:42.116 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.100+0000 7f73b32fe700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73ac133090 con 0x7f73ac1222b0 2019-12-05T05:03:42.116 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.100+0000 7f73b32fe700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73ac1237d0 con 0x7f73ac1179c0 2019-12-05T05:03:42.117 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.100+0000 7f73b32fe700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73ac124080 con 0x7f73ac132710 2019-12-05T05:03:42.117 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.100+0000 7f73b109a700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f73ac1179c0 0x7f73ac11bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:42.117 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.100+0000 7f73b109a700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f73ac1179c0 0x7f73ac11bb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:53472/0 (socket says 172.21.15.42:53472) 2019-12-05T05:03:42.117 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.100+0000 7f73b109a700 1 -- 172.21.15.42:0/564481437 learned_addr learned my addr 172.21.15.42:0/564481437 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:03:42.117 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.100+0000 7f73b189b700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f73ac1222b0 0x7f73ac1171c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:53508/0 (socket says 172.21.15.42:53508) 2019-12-05T05:03:42.118 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.100+0000 7f73b209c700 1 --2- 172.21.15.42:0/564481437 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f73ac132710 0x7f73ac132b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:42.119 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.100+0000 7f73b109a700 1 -- 172.21.15.42:0/564481437 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f73ac132710 msgr2=0x7f73ac132b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:42.119 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.100+0000 7f73b109a700 1 --2- 172.21.15.42:0/564481437 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f73ac132710 0x7f73ac132b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:42.119 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.100+0000 7f73b109a700 1 -- 172.21.15.42:0/564481437 >> v1:172.21.15.42:6789/0 conn(0x7f73ac1222b0 legacy=0x7f73ac1171c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:42.120 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.100+0000 7f73aa7fc700 1 -- 172.21.15.42:0/564481437 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4157891166 0 0) 0x7f73ac133090 con 0x7f73ac1222b0 2019-12-05T05:03:42.120 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.100+0000 7f73b109a700 1 -- 172.21.15.42:0/564481437 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73ac123be0 con 0x7f73ac1179c0 2019-12-05T05:03:42.120 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.100+0000 7f73b109a700 1 --2- 172.21.15.42:0/564481437 >> v2:172.21.15.42:3300/0 conn(0x7f73ac1179c0 0x7f73ac11bb20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f73ac130280 tx=0x7f73a0011a30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:42.120 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.100+0000 7f73b109a700 1 --2- 172.21.15.42:0/564481437 >> v2:172.21.15.42:3300/0 conn(0x7f73ac1179c0 0x7f73ac11bb20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f73ac130280 tx=0x7f73a0011a30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:42.120 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.100+0000 7f73b109a700 1 --2- 172.21.15.42:0/564481437 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f73ac1179c0 0x7f73ac11bb20 secure :-1 s=READY pgs=130 cs=0 l=1 rx=0x7f73ac130280 tx=0x7f73a0011a30).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:42.120 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.104+0000 7f73b109a700 1 --2- 172.21.15.42:0/564481437 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f73ac1179c0 0x7f73ac11bb20 secure :-1 s=THROTTLE_DONE pgs=130 cs=0 l=1 rx=0x7f73ac130280 tx=0x7f73a0011a30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:42.121 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.104+0000 7f73aa7fc700 1 -- 172.21.15.42:0/564481437 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f73a0005540 con 0x7f73ac1179c0 2019-12-05T05:03:42.121 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.104+0000 7f73b109a700 1 --2- 172.21.15.42:0/564481437 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f73ac1179c0 0x7f73ac11bb20 secure :-1 s=THROTTLE_DONE pgs=130 cs=0 l=1 rx=0x7f73ac130280 tx=0x7f73a0011a30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:42.121 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.104+0000 7f73b109a700 1 --2- 172.21.15.42:0/564481437 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f73ac1179c0 0x7f73ac11bb20 secure :-1 s=THROTTLE_DONE pgs=130 cs=0 l=1 rx=0x7f73ac130280 tx=0x7f73a0011a30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:42.121 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.104+0000 7f73aa7fc700 1 -- 172.21.15.42:0/564481437 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73a0007150 con 0x7f73ac1179c0 2019-12-05T05:03:42.121 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.104+0000 7f73aa7fc700 1 -- 172.21.15.42:0/564481437 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f73a0005b20 con 0x7f73ac1179c0 2019-12-05T05:03:42.122 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.104+0000 7f73b32fe700 1 -- 172.21.15.42:0/564481437 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73ac07a320 con 0x7f73ac1179c0 2019-12-05T05:03:42.122 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.104+0000 7f73b32fe700 1 -- 172.21.15.42:0/564481437 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73ac07a7f0 con 0x7f73ac1179c0 2019-12-05T05:03:42.122 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.104+0000 7f73b109a700 1 --2- 172.21.15.42:0/564481437 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f73ac1179c0 0x7f73ac11bb20 secure :-1 s=THROTTLE_DONE pgs=130 cs=0 l=1 rx=0x7f73ac130280 tx=0x7f73a0011a30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:42.122 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.104+0000 7f73aa7fc700 1 -- 172.21.15.42:0/564481437 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f73a000d400 con 0x7f73ac1179c0 2019-12-05T05:03:42.122 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.104+0000 7f73b109a700 1 --2- 172.21.15.42:0/564481437 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f73ac1179c0 0x7f73ac11bb20 secure :-1 s=THROTTLE_DONE pgs=130 cs=0 l=1 rx=0x7f73ac130280 tx=0x7f73a0011a30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:42.122 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.104+0000 7f73aa7fc700 1 --2- 172.21.15.42:0/564481437 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f738c03acf0 0x7f738c03d200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:42.123 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.104+0000 7f73aa7fc700 1 -- 172.21.15.42:0/564481437 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f73a004b700 con 0x7f73ac1179c0 2019-12-05T05:03:42.123 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.104+0000 7f73b189b700 1 --2- 172.21.15.42:0/564481437 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f738c03acf0 0x7f738c03d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:42.123 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.104+0000 7f73b189b700 1 --2- 172.21.15.42:0/564481437 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f738c03acf0 0x7f738c03d200 crc :-1 s=READY pgs=73 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:42.123 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.108+0000 7f73b32fe700 1 -- 172.21.15.42:0/564481437 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73ac05d100 con 0x7f73ac1179c0 2019-12-05T05:03:42.124 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.116+0000 7f73b109a700 1 --2- 172.21.15.42:0/564481437 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f73ac1179c0 0x7f73ac11bb20 secure :-1 s=THROTTLE_DONE pgs=130 cs=0 l=1 rx=0x7f73ac130280 tx=0x7f73a0011a30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:42.124 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.116+0000 7f73aa7fc700 1 -- 172.21.15.42:0/564481437 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f73a0007f50 con 0x7f73ac1179c0 2019-12-05T05:03:42.313 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.308+0000 7f73b32fe700 1 -- 172.21.15.42:0/564481437 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f73ac05b8b0 con 0x7f73ac1179c0 2019-12-05T05:03:42.313 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.308+0000 7f73b109a700 1 --2- 172.21.15.42:0/564481437 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f73ac1179c0 0x7f73ac11bb20 secure :-1 s=THROTTLE_DONE pgs=130 cs=0 l=1 rx=0x7f73ac130280 tx=0x7f73a0011a30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:42.314 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.308+0000 7f73aa7fc700 1 -- 172.21.15.42:0/564481437 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f73a000dd00 con 0x7f73ac1179c0 2019-12-05T05:03:42.314 INFO:teuthology.orchestra.run.smithi042.stdout:55834574853 2019-12-05T05:03:42.316 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.312+0000 7f73b32fe700 1 -- 172.21.15.42:0/564481437 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f738c03acf0 msgr2=0x7f738c03d200 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:42.317 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.312+0000 7f73b32fe700 1 --2- 172.21.15.42:0/564481437 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f738c03acf0 0x7f738c03d200 crc :-1 s=READY pgs=73 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:42.317 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.312+0000 7f73b32fe700 1 -- 172.21.15.42:0/564481437 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f73ac1179c0 msgr2=0x7f73ac11bb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:42.317 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.312+0000 7f73b32fe700 1 --2- 172.21.15.42:0/564481437 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f73ac1179c0 0x7f73ac11bb20 secure :-1 s=READY pgs=130 cs=0 l=1 rx=0x7f73ac130280 tx=0x7f73a0011a30).stop 2019-12-05T05:03:42.317 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.312+0000 7f73b32fe700 1 -- 172.21.15.42:0/564481437 shutdown_connections 2019-12-05T05:03:42.317 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.312+0000 7f73b32fe700 1 --2- 172.21.15.42:0/564481437 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f738c03acf0 0x7f738c03d200 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:42.317 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.312+0000 7f73b32fe700 1 --2- 172.21.15.42:0/564481437 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f73ac132710 0x7f73ac132b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:42.317 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.312+0000 7f73b32fe700 1 --2- 172.21.15.42:0/564481437 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f73ac1179c0 0x7f73ac11bb20 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:42.318 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.312+0000 7f73b32fe700 1 -- 172.21.15.42:0/564481437 shutdown_connections 2019-12-05T05:03:42.318 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.312+0000 7f73b32fe700 1 -- 172.21.15.42:0/564481437 wait complete. 2019-12-05T05:03:42.318 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.312+0000 7f73b32fe700 1 -- 172.21.15.42:0/564481437 >> 172.21.15.42:0/564481437 conn(0x7f73ac0a96c0 msgr2=0x7f73ac141350 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:42.326 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574853 for osd.2 2019-12-05T05:03:42.327 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2019-12-05T05:03:42.428 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.416+0000 7efca70df700 1 Processor -- start 2019-12-05T05:03:42.428 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.416+0000 7efca70df700 1 -- start start 2019-12-05T05:03:42.429 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.416+0000 7efca70df700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7efca008df70 0x7efca009f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:42.429 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.416+0000 7efca70df700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7efca009f820 0x7efca014efd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:42.429 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.416+0000 7efca70df700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efca009fc50 con 0x7efca00a7730 2019-12-05T05:03:42.429 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.416+0000 7efca70df700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efca00a0af0 con 0x7efca009f820 2019-12-05T05:03:42.429 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.416+0000 7efca70df700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efca008e5a0 con 0x7efca008df70 2019-12-05T05:03:42.430 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.416+0000 7efca4e7b700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7efca008df70 0x7efca009f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:42.430 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.416+0000 7efca4e7b700 1 -- >> v2:172.21.15.42:3300/0 conn(0x7efca008df70 msgr2=0x7efca009f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2019-12-05T05:03:42.431 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.416+0000 7efca4e7b700 1 -- >> v2:172.21.15.42:3300/0 conn(0x7efca008df70 msgr2=0x7efca009f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2019-12-05T05:03:42.431 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.416+0000 7efca4e7b700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7efca008df70 0x7efca009f2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-12-05T05:03:42.431 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.416+0000 7efca4e7b700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7efca008df70 0x7efca009f2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rx=0 tx=0)._fault waiting 0.200000 2019-12-05T05:03:42.431 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.416+0000 7efca5e7d700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7efca009f820 0x7efca014efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:42.431 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.416+0000 7efca567c700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7efca00a7730 0x7efca007e710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:34724/0 (socket says 172.21.15.42:34724) 2019-12-05T05:03:42.432 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.416+0000 7efca567c700 1 -- 172.21.15.42:0/336812719 learned_addr learned my addr 172.21.15.42:0/336812719 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:03:42.432 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.416+0000 7efca5e7d700 1 -- 172.21.15.42:0/336812719 >> v2:172.21.15.42:3300/0 conn(0x7efca008df70 msgr2=0x7efca009f2d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2019-12-05T05:03:42.432 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.416+0000 7efca5e7d700 1 --2- 172.21.15.42:0/336812719 >> v2:172.21.15.42:3300/0 conn(0x7efca008df70 0x7efca009f2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:03:42.432 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.420+0000 7efca5e7d700 1 -- 172.21.15.42:0/336812719 >> v1:172.21.15.40:6789/0 conn(0x7efca00a7730 legacy=0x7efca007e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:42.432 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.420+0000 7efc97fff700 1 -- 172.21.15.42:0/336812719 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2219232423 0 0) 0x7efca009fc50 con 0x7efca00a7730 2019-12-05T05:03:42.433 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.420+0000 7efca5e7d700 1 -- 172.21.15.42:0/336812719 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efca00a0700 con 0x7efca009f820 2019-12-05T05:03:42.433 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.420+0000 7efca5e7d700 1 --2- 172.21.15.42:0/336812719 >> v2:172.21.15.40:3300/0 conn(0x7efca009f820 0x7efca014efd0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7efc980032b0 tx=0x7efc9800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:42.433 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.420+0000 7efca5e7d700 1 --2- 172.21.15.42:0/336812719 >> v2:172.21.15.40:3300/0 conn(0x7efca009f820 0x7efca014efd0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7efc980032b0 tx=0x7efc9800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:42.433 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.420+0000 7efca5e7d700 1 --2- 172.21.15.42:0/336812719 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efca009f820 0x7efca014efd0 secure :-1 s=READY pgs=114 cs=0 l=1 rx=0x7efc980032b0 tx=0x7efc9800fff0).ready entity=mon.0 client_cookie=5a5e6034ea84519d server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:42.433 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.420+0000 7efca5e7d700 1 --2- 172.21.15.42:0/336812719 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efca009f820 0x7efca014efd0 secure :-1 s=THROTTLE_DONE pgs=114 cs=0 l=1 rx=0x7efc980032b0 tx=0x7efc9800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:42.433 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.420+0000 7efca5e7d700 1 --2- 172.21.15.42:0/336812719 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efca009f820 0x7efca014efd0 secure :-1 s=THROTTLE_DONE pgs=114 cs=0 l=1 rx=0x7efc980032b0 tx=0x7efc9800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:42.434 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.420+0000 7efc97fff700 1 -- 172.21.15.42:0/336812719 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efc980108c0 con 0x7efca009f820 2019-12-05T05:03:42.434 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.420+0000 7efca5e7d700 1 --2- 172.21.15.42:0/336812719 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efca009f820 0x7efca014efd0 secure :-1 s=THROTTLE_DONE pgs=114 cs=0 l=1 rx=0x7efc980032b0 tx=0x7efc9800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:42.434 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.420+0000 7efc97fff700 1 -- 172.21.15.42:0/336812719 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc98008b50 con 0x7efca009f820 2019-12-05T05:03:42.434 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.420+0000 7efc97fff700 1 -- 172.21.15.42:0/336812719 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efc980110c0 con 0x7efca009f820 2019-12-05T05:03:42.434 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.420+0000 7efca70df700 1 -- 172.21.15.42:0/336812719 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efca009f820 msgr2=0x7efca014efd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:42.434 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.420+0000 7efca70df700 1 --2- 172.21.15.42:0/336812719 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efca009f820 0x7efca014efd0 secure :-1 s=READY pgs=114 cs=0 l=1 rx=0x7efc980032b0 tx=0x7efc9800fff0).stop 2019-12-05T05:03:42.435 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.420+0000 7efca70df700 1 -- 172.21.15.42:0/336812719 shutdown_connections 2019-12-05T05:03:42.435 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.420+0000 7efca70df700 1 --2- 172.21.15.42:0/336812719 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efca009f820 0x7efca014efd0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:42.435 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.420+0000 7efca70df700 1 --2- 172.21.15.42:0/336812719 >> v2:172.21.15.42:3300/0 conn(0x7efca008df70 0x7efca009f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:03:42.435 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.420+0000 7efca70df700 1 -- 172.21.15.42:0/336812719 shutdown_connections 2019-12-05T05:03:42.435 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.420+0000 7efca70df700 1 -- 172.21.15.42:0/336812719 wait complete. 2019-12-05T05:03:42.435 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.420+0000 7efca70df700 1 -- 172.21.15.42:0/336812719 >> 172.21.15.42:0/336812719 conn(0x7efca00aabd0 msgr2=0x7efca00a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:42.439 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.424+0000 7efca70df700 1 Processor -- start 2019-12-05T05:03:42.440 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.424+0000 7efca70df700 1 -- start start 2019-12-05T05:03:42.440 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.424+0000 7efca70df700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7efca01232d0 0x7efca0123700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:42.440 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.424+0000 7efca70df700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7efca014dd10 0x7efca014e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:42.440 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.424+0000 7efca70df700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efca00a6630 con 0x7efca0067310 2019-12-05T05:03:42.441 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.424+0000 7efca70df700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efca00a51c0 con 0x7efca014dd10 2019-12-05T05:03:42.441 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.424+0000 7efca70df700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efca00a5420 con 0x7efca01232d0 2019-12-05T05:03:42.441 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.424+0000 7efca4e7b700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7efca01232d0 0x7efca0123700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:42.442 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.424+0000 7efca4e7b700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7efca01232d0 0x7efca0123700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:53486/0 (socket says 172.21.15.42:53486) 2019-12-05T05:03:42.442 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.424+0000 7efca4e7b700 1 -- 172.21.15.42:0/3324668795 learned_addr learned my addr 172.21.15.42:0/3324668795 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:03:42.442 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.424+0000 7efca5e7d700 1 --2- 172.21.15.42:0/3324668795 >> v2:172.21.15.40:3300/0 conn(0x7efca014dd10 0x7efca014e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:42.442 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.424+0000 7efca4e7b700 1 -- 172.21.15.42:0/3324668795 >> v2:172.21.15.40:3300/0 conn(0x7efca014dd10 msgr2=0x7efca014e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:42.442 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.424+0000 7efca4e7b700 1 --2- 172.21.15.42:0/3324668795 >> v2:172.21.15.40:3300/0 conn(0x7efca014dd10 0x7efca014e140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:42.442 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.424+0000 7efca4e7b700 1 -- 172.21.15.42:0/3324668795 >> v1:172.21.15.40:6789/0 conn(0x7efca0067310 legacy=0x7efca00a49c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:42.443 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.424+0000 7efca4e7b700 1 -- 172.21.15.42:0/3324668795 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efca0123c50 con 0x7efca01232d0 2019-12-05T05:03:42.443 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.424+0000 7efca5e7d700 1 --2- 172.21.15.42:0/3324668795 >> v2:172.21.15.40:3300/0 conn(0x7efca014dd10 0x7efca014e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:03:42.443 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.428+0000 7efca4e7b700 1 --2- 172.21.15.42:0/3324668795 >> v2:172.21.15.42:3300/0 conn(0x7efca01232d0 0x7efca0123700 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7efc90002f30 tx=0x7efc90010990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:42.444 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.428+0000 7efca4e7b700 1 --2- 172.21.15.42:0/3324668795 >> v2:172.21.15.42:3300/0 conn(0x7efca01232d0 0x7efca0123700 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7efc90002f30 tx=0x7efc90010990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:42.444 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.428+0000 7efca4e7b700 1 --2- 172.21.15.42:0/3324668795 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efca01232d0 0x7efca0123700 secure :-1 s=READY pgs=131 cs=0 l=1 rx=0x7efc90002f30 tx=0x7efc90010990).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:42.444 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.428+0000 7efca4e7b700 1 --2- 172.21.15.42:0/3324668795 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efca01232d0 0x7efca0123700 secure :-1 s=THROTTLE_DONE pgs=131 cs=0 l=1 rx=0x7efc90002f30 tx=0x7efc90010990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:42.444 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.428+0000 7efc95ffb700 1 -- 172.21.15.42:0/3324668795 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efc9000f620 con 0x7efca01232d0 2019-12-05T05:03:42.444 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.428+0000 7efca70df700 1 -- 172.21.15.42:0/3324668795 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efca014e690 con 0x7efca01232d0 2019-12-05T05:03:42.444 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.428+0000 7efca4e7b700 1 --2- 172.21.15.42:0/3324668795 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efca01232d0 0x7efca0123700 secure :-1 s=THROTTLE_DONE pgs=131 cs=0 l=1 rx=0x7efc90002f30 tx=0x7efc90010990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:42.445 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.428+0000 7efca4e7b700 1 --2- 172.21.15.42:0/3324668795 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efca01232d0 0x7efca0123700 secure :-1 s=THROTTLE_DONE pgs=131 cs=0 l=1 rx=0x7efc90002f30 tx=0x7efc90010990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:42.445 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.428+0000 7efc95ffb700 1 -- 172.21.15.42:0/3324668795 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc9000f240 con 0x7efca01232d0 2019-12-05T05:03:42.445 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.428+0000 7efca70df700 1 -- 172.21.15.42:0/3324668795 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efca012dd90 con 0x7efca01232d0 2019-12-05T05:03:42.445 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.428+0000 7efc95ffb700 1 -- 172.21.15.42:0/3324668795 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efc90006c50 con 0x7efca01232d0 2019-12-05T05:03:42.445 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.428+0000 7efca4e7b700 1 --2- 172.21.15.42:0/3324668795 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efca01232d0 0x7efca0123700 secure :-1 s=THROTTLE_DONE pgs=131 cs=0 l=1 rx=0x7efc90002f30 tx=0x7efc90010990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:42.445 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.428+0000 7efc95ffb700 1 -- 172.21.15.42:0/3324668795 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7efc90006f30 con 0x7efca01232d0 2019-12-05T05:03:42.446 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.428+0000 7efca4e7b700 1 --2- 172.21.15.42:0/3324668795 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efca01232d0 0x7efca0123700 secure :-1 s=THROTTLE_DONE pgs=131 cs=0 l=1 rx=0x7efc90002f30 tx=0x7efc90010990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:42.446 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.428+0000 7efc95ffb700 1 --2- 172.21.15.42:0/3324668795 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7efc8003ad30 0x7efc8003d240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:42.446 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.428+0000 7efc95ffb700 1 -- 172.21.15.42:0/3324668795 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6031+0+0 (secure 0 0 0) 0x7efc9004acf0 con 0x7efca01232d0 2019-12-05T05:03:42.446 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.428+0000 7efca567c700 1 --2- 172.21.15.42:0/3324668795 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7efc8003ad30 0x7efc8003d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:42.446 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.432+0000 7efca567c700 1 --2- 172.21.15.42:0/3324668795 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7efc8003ad30 0x7efc8003d240 crc :-1 s=READY pgs=74 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:42.447 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.432+0000 7efca70df700 1 -- 172.21.15.42:0/3324668795 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efca005d100 con 0x7efca01232d0 2019-12-05T05:03:42.447 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.440+0000 7efca4e7b700 1 --2- 172.21.15.42:0/3324668795 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efca01232d0 0x7efca0123700 secure :-1 s=THROTTLE_DONE pgs=131 cs=0 l=1 rx=0x7efc90002f30 tx=0x7efc90010990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:42.448 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.440+0000 7efc95ffb700 1 -- 172.21.15.42:0/3324668795 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7efc90031d50 con 0x7efca01232d0 2019-12-05T05:03:42.628 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.624+0000 7efca70df700 1 -- 172.21.15.42:0/3324668795 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7efca005b8b0 con 0x7efca01232d0 2019-12-05T05:03:42.629 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.624+0000 7efca4e7b700 1 --2- 172.21.15.42:0/3324668795 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efca01232d0 0x7efca0123700 secure :-1 s=THROTTLE_DONE pgs=131 cs=0 l=1 rx=0x7efc90002f30 tx=0x7efc90010990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:42.629 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.624+0000 7efc95ffb700 1 -- 172.21.15.42:0/3324668795 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7efc90031fb0 con 0x7efca01232d0 2019-12-05T05:03:42.629 INFO:teuthology.orchestra.run.smithi042.stdout:55834574854 2019-12-05T05:03:42.632 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.624+0000 7efca70df700 1 -- 172.21.15.42:0/3324668795 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7efc8003ad30 msgr2=0x7efc8003d240 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:42.632 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.624+0000 7efca70df700 1 --2- 172.21.15.42:0/3324668795 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7efc8003ad30 0x7efc8003d240 crc :-1 s=READY pgs=74 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:42.633 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.624+0000 7efca70df700 1 -- 172.21.15.42:0/3324668795 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efca01232d0 msgr2=0x7efca0123700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:42.633 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.624+0000 7efca70df700 1 --2- 172.21.15.42:0/3324668795 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efca01232d0 0x7efca0123700 secure :-1 s=READY pgs=131 cs=0 l=1 rx=0x7efc90002f30 tx=0x7efc90010990).stop 2019-12-05T05:03:42.633 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.624+0000 7efca70df700 1 -- 172.21.15.42:0/3324668795 shutdown_connections 2019-12-05T05:03:42.633 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.624+0000 7efca70df700 1 --2- 172.21.15.42:0/3324668795 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7efc8003ad30 0x7efc8003d240 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:42.633 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.624+0000 7efca70df700 1 --2- 172.21.15.42:0/3324668795 >> v2:172.21.15.40:3300/0 conn(0x7efca014dd10 0x7efca014e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:42.633 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.624+0000 7efca70df700 1 --2- 172.21.15.42:0/3324668795 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efca01232d0 0x7efca0123700 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:42.634 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.624+0000 7efca70df700 1 -- 172.21.15.42:0/3324668795 shutdown_connections 2019-12-05T05:03:42.634 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.624+0000 7efca70df700 1 -- 172.21.15.42:0/3324668795 wait complete. 2019-12-05T05:03:42.634 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.624+0000 7efca70df700 1 -- 172.21.15.42:0/3324668795 >> 172.21.15.42:0/3324668795 conn(0x7efca00a6ec0 msgr2=0x7efca007ce90 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:42.643 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.3 2019-12-05T05:03:42.643 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2019-12-05T05:03:42.758 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.748+0000 7f45c04ac700 1 Processor -- start 2019-12-05T05:03:42.759 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.748+0000 7f45c04ac700 1 -- start start 2019-12-05T05:03:42.759 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.748+0000 7f45c04ac700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f45b80a7730 0x7f45b808df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:42.759 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.748+0000 7f45c04ac700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f45b809fad0 0x7f45b807eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:42.759 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.748+0000 7f45c04ac700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45b80a0700 con 0x7f45b807e710 2019-12-05T05:03:42.759 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.748+0000 7f45c04ac700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45b80a0af0 con 0x7f45b80a7730 2019-12-05T05:03:42.759 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.748+0000 7f45c04ac700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45b808e5a0 con 0x7f45b809fad0 2019-12-05T05:03:42.759 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.748+0000 7f45be248700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f45b807e710 0x7f45b809f2d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:53534/0 (socket says 172.21.15.42:53534) 2019-12-05T05:03:42.760 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.748+0000 7f45be248700 1 -- 172.21.15.42:0/938792973 learned_addr learned my addr 172.21.15.42:0/938792973 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:03:42.760 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.748+0000 7f45bf24a700 1 --2- 172.21.15.42:0/938792973 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f45b809fad0 0x7f45b807eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:42.760 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.748+0000 7f45bea49700 1 --2- 172.21.15.42:0/938792973 >> v2:172.21.15.40:3300/0 conn(0x7f45b80a7730 0x7f45b808df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:42.760 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.748+0000 7f45bda47700 1 -- 172.21.15.42:0/938792973 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3261063388 0 0) 0x7f45b80a0700 con 0x7f45b807e710 2019-12-05T05:03:42.760 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.748+0000 7f45bf24a700 1 -- 172.21.15.42:0/938792973 >> v2:172.21.15.40:3300/0 conn(0x7f45b80a7730 msgr2=0x7f45b808df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:03:42.760 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.748+0000 7f45bf24a700 1 --2- 172.21.15.42:0/938792973 >> v2:172.21.15.40:3300/0 conn(0x7f45b80a7730 0x7f45b808df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:03:42.761 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.748+0000 7f45bf24a700 1 -- 172.21.15.42:0/938792973 >> v1:172.21.15.42:6789/0 conn(0x7f45b807e710 legacy=0x7f45b809f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:42.761 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.748+0000 7f45bf24a700 1 -- 172.21.15.42:0/938792973 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45b807f090 con 0x7f45b809fad0 2019-12-05T05:03:42.761 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.748+0000 7f45bea49700 1 --2- 172.21.15.42:0/938792973 >> v2:172.21.15.40:3300/0 conn(0x7f45b80a7730 0x7f45b808df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:03:42.761 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.748+0000 7f45bf24a700 1 --2- 172.21.15.42:0/938792973 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f45b809fad0 0x7f45b807eb40 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f45b0008b10 tx=0x7f45b000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:42.761 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.748+0000 7f45bf24a700 1 --2- 172.21.15.42:0/938792973 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f45b809fad0 0x7f45b807eb40 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f45b0008b10 tx=0x7f45b000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:42.761 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.752+0000 7f45bf24a700 1 --2- 172.21.15.42:0/938792973 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f45b809fad0 0x7f45b807eb40 secure :-1 s=READY pgs=37 cs=0 l=1 rx=0x7f45b0008b10 tx=0x7f45b000fff0).ready entity=mon.2 client_cookie=4c0c29ed34f9ba1c server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:42.761 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.752+0000 7f45bf24a700 1 --2- 172.21.15.42:0/938792973 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f45b809fad0 0x7f45b807eb40 secure :-1 s=THROTTLE_DONE pgs=37 cs=0 l=1 rx=0x7f45b0008b10 tx=0x7f45b000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:42.762 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.752+0000 7f45bda47700 1 -- 172.21.15.42:0/938792973 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f45b0010900 con 0x7f45b809fad0 2019-12-05T05:03:42.762 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.752+0000 7f45bf24a700 1 --2- 172.21.15.42:0/938792973 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f45b809fad0 0x7f45b807eb40 secure :-1 s=THROTTLE_DONE pgs=37 cs=0 l=1 rx=0x7f45b0008b10 tx=0x7f45b000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:42.762 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.752+0000 7f45bf24a700 1 --2- 172.21.15.42:0/938792973 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f45b809fad0 0x7f45b807eb40 secure :-1 s=THROTTLE_DONE pgs=37 cs=0 l=1 rx=0x7f45b0008b10 tx=0x7f45b000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:42.762 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.752+0000 7f45bda47700 1 -- 172.21.15.42:0/938792973 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45b0008b50 con 0x7f45b809fad0 2019-12-05T05:03:42.762 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.752+0000 7f45bda47700 1 -- 172.21.15.42:0/938792973 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f45b00110c0 con 0x7f45b809fad0 2019-12-05T05:03:42.762 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.752+0000 7f45c04ac700 1 -- 172.21.15.42:0/938792973 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f45b809fad0 msgr2=0x7f45b807eb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:42.762 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.752+0000 7f45c04ac700 1 --2- 172.21.15.42:0/938792973 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f45b809fad0 0x7f45b807eb40 secure :-1 s=READY pgs=37 cs=0 l=1 rx=0x7f45b0008b10 tx=0x7f45b000fff0).stop 2019-12-05T05:03:42.763 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.752+0000 7f45c04ac700 1 -- 172.21.15.42:0/938792973 shutdown_connections 2019-12-05T05:03:42.763 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.752+0000 7f45c04ac700 1 --2- 172.21.15.42:0/938792973 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f45b809fad0 0x7f45b807eb40 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:42.763 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.752+0000 7f45c04ac700 1 --2- 172.21.15.42:0/938792973 >> v2:172.21.15.40:3300/0 conn(0x7f45b80a7730 0x7f45b808df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:03:42.763 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.752+0000 7f45c04ac700 1 -- 172.21.15.42:0/938792973 shutdown_connections 2019-12-05T05:03:42.763 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.752+0000 7f45c04ac700 1 -- 172.21.15.42:0/938792973 wait complete. 2019-12-05T05:03:42.763 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.752+0000 7f45c04ac700 1 -- 172.21.15.42:0/938792973 >> 172.21.15.42:0/938792973 conn(0x7f45b80aabd0 msgr2=0x7f45b80a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:42.767 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.752+0000 7f45c04ac700 1 Processor -- start 2019-12-05T05:03:42.767 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.756+0000 7f45c04ac700 1 -- start start 2019-12-05T05:03:42.767 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.756+0000 7f45c04ac700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f45b80a1f80 0x7f45b80a5540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:42.768 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.756+0000 7f45c04ac700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f45b80a23b0 0x7f45b80a5a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:42.768 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.756+0000 7f45c04ac700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45b8124950 con 0x7f45b80a67b0 2019-12-05T05:03:42.768 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.756+0000 7f45c04ac700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45b80a6be0 con 0x7f45b80a1f80 2019-12-05T05:03:42.768 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.756+0000 7f45c04ac700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45b8131350 con 0x7f45b80a23b0 2019-12-05T05:03:42.768 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.756+0000 7f45be248700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f45b80a1f80 0x7f45b80a5540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:42.769 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.756+0000 7f45be248700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f45b80a1f80 0x7f45b80a5540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:53502/0 (socket says 172.21.15.42:53502) 2019-12-05T05:03:42.769 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.756+0000 7f45bea49700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f45b80a67b0 0x7f45b8131c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:53538/0 (socket says 172.21.15.42:53538) 2019-12-05T05:03:42.769 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.756+0000 7f45be248700 1 -- 172.21.15.42:0/2705021133 learned_addr learned my addr 172.21.15.42:0/2705021133 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:03:42.769 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.756+0000 7f45bf24a700 1 --2- 172.21.15.42:0/2705021133 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f45b80a23b0 0x7f45b80a5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:42.769 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.756+0000 7f45bf24a700 1 -- 172.21.15.42:0/2705021133 >> v2:172.21.15.42:3300/0 conn(0x7f45b80a1f80 msgr2=0x7f45b80a5540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:42.769 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.756+0000 7f45bf24a700 1 --2- 172.21.15.42:0/2705021133 >> v2:172.21.15.42:3300/0 conn(0x7f45b80a1f80 0x7f45b80a5540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:42.769 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.756+0000 7f45bf24a700 1 -- 172.21.15.42:0/2705021133 >> v1:172.21.15.42:6789/0 conn(0x7f45b80a67b0 legacy=0x7f45b8131c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:42.770 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.756+0000 7f45bf24a700 1 -- 172.21.15.42:0/2705021133 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45b807ba90 con 0x7f45b80a23b0 2019-12-05T05:03:42.770 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.756+0000 7f45bf24a700 1 --2- 172.21.15.42:0/2705021133 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f45b80a23b0 0x7f45b80a5a90 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f45b0008b10 tx=0x7f45b0008a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:42.770 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.756+0000 7f45bf24a700 1 --2- 172.21.15.42:0/2705021133 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f45b80a23b0 0x7f45b80a5a90 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f45b0008b10 tx=0x7f45b0008a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:42.770 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.756+0000 7f45bf24a700 1 --2- 172.21.15.42:0/2705021133 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f45b80a23b0 0x7f45b80a5a90 secure :-1 s=READY pgs=38 cs=0 l=1 rx=0x7f45b0008b10 tx=0x7f45b0008a20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:42.770 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.756+0000 7f45bf24a700 1 --2- 172.21.15.42:0/2705021133 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f45b80a23b0 0x7f45b80a5a90 secure :-1 s=THROTTLE_DONE pgs=38 cs=0 l=1 rx=0x7f45b0008b10 tx=0x7f45b0008a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:42.770 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.756+0000 7f45bf24a700 1 --2- 172.21.15.42:0/2705021133 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f45b80a23b0 0x7f45b80a5a90 secure :-1 s=THROTTLE_DONE pgs=38 cs=0 l=1 rx=0x7f45b0008b10 tx=0x7f45b0008a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:42.770 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.756+0000 7f45ab7fe700 1 -- 172.21.15.42:0/2705021133 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f45b0005ab0 con 0x7f45b80a23b0 2019-12-05T05:03:42.771 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.756+0000 7f45bf24a700 1 --2- 172.21.15.42:0/2705021133 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f45b80a23b0 0x7f45b80a5a90 secure :-1 s=THROTTLE_DONE pgs=38 cs=0 l=1 rx=0x7f45b0008b10 tx=0x7f45b0008a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:42.771 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.756+0000 7f45ab7fe700 1 -- 172.21.15.42:0/2705021133 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45b0010b00 con 0x7f45b80a23b0 2019-12-05T05:03:42.771 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.756+0000 7f45c04ac700 1 -- 172.21.15.42:0/2705021133 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45b81387a0 con 0x7f45b80a23b0 2019-12-05T05:03:42.771 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.756+0000 7f45ab7fe700 1 -- 172.21.15.42:0/2705021133 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f45b000f680 con 0x7f45b80a23b0 2019-12-05T05:03:42.771 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.756+0000 7f45c04ac700 1 -- 172.21.15.42:0/2705021133 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45b8138d20 con 0x7f45b80a23b0 2019-12-05T05:03:42.771 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.760+0000 7f45bf24a700 1 --2- 172.21.15.42:0/2705021133 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f45b80a23b0 0x7f45b80a5a90 secure :-1 s=THROTTLE_DONE pgs=38 cs=0 l=1 rx=0x7f45b0008b10 tx=0x7f45b0008a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:42.771 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.760+0000 7f45ab7fe700 1 -- 172.21.15.42:0/2705021133 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f45b00113d0 con 0x7f45b80a23b0 2019-12-05T05:03:42.772 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.760+0000 7f45bf24a700 1 --2- 172.21.15.42:0/2705021133 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f45b80a23b0 0x7f45b80a5a90 secure :-1 s=THROTTLE_DONE pgs=38 cs=0 l=1 rx=0x7f45b0008b10 tx=0x7f45b0008a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:42.772 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.760+0000 7f45ab7fe700 1 --2- 172.21.15.42:0/2705021133 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f45a003ae40 0x7f45a003d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:42.772 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.760+0000 7f45ab7fe700 1 -- 172.21.15.42:0/2705021133 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f45b008f040 con 0x7f45b80a23b0 2019-12-05T05:03:42.772 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.760+0000 7f45bea49700 1 --2- 172.21.15.42:0/2705021133 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f45a003ae40 0x7f45a003d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:42.772 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.760+0000 7f45bea49700 1 --2- 172.21.15.42:0/2705021133 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f45a003ae40 0x7f45a003d350 crc :-1 s=READY pgs=77 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:42.773 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.764+0000 7f45c04ac700 1 -- 172.21.15.42:0/2705021133 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45b805d100 con 0x7f45b80a23b0 2019-12-05T05:03:42.777 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.772+0000 7f45bf24a700 1 --2- 172.21.15.42:0/2705021133 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f45b80a23b0 0x7f45b80a5a90 secure :-1 s=THROTTLE_DONE pgs=38 cs=0 l=1 rx=0x7f45b0008b10 tx=0x7f45b0008a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:42.778 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.772+0000 7f45ab7fe700 1 -- 172.21.15.42:0/2705021133 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f45b0076080 con 0x7f45b80a23b0 2019-12-05T05:03:42.964 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.960+0000 7f45c04ac700 1 -- 172.21.15.42:0/2705021133 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f45b805b8b0 con 0x7f45b80a23b0 2019-12-05T05:03:42.965 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.960+0000 7f45bf24a700 1 --2- 172.21.15.42:0/2705021133 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f45b80a23b0 0x7f45b80a5a90 secure :-1 s=THROTTLE_DONE pgs=38 cs=0 l=1 rx=0x7f45b0008b10 tx=0x7f45b0008a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:42.965 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.960+0000 7f45ab7fe700 1 -- 172.21.15.42:0/2705021133 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f45b00762e0 con 0x7f45b80a23b0 2019-12-05T05:03:42.965 INFO:teuthology.orchestra.run.smithi042.stdout:55834574853 2019-12-05T05:03:42.968 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.960+0000 7f45c04ac700 1 -- 172.21.15.42:0/2705021133 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f45a003ae40 msgr2=0x7f45a003d350 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:42.968 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.960+0000 7f45c04ac700 1 --2- 172.21.15.42:0/2705021133 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f45a003ae40 0x7f45a003d350 crc :-1 s=READY pgs=77 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:42.969 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.960+0000 7f45c04ac700 1 -- 172.21.15.42:0/2705021133 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f45b80a23b0 msgr2=0x7f45b80a5a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:42.969 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.960+0000 7f45c04ac700 1 --2- 172.21.15.42:0/2705021133 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f45b80a23b0 0x7f45b80a5a90 secure :-1 s=READY pgs=38 cs=0 l=1 rx=0x7f45b0008b10 tx=0x7f45b0008a20).stop 2019-12-05T05:03:42.969 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.960+0000 7f45c04ac700 1 -- 172.21.15.42:0/2705021133 shutdown_connections 2019-12-05T05:03:42.969 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.960+0000 7f45c04ac700 1 --2- 172.21.15.42:0/2705021133 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f45a003ae40 0x7f45a003d350 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:42.969 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.960+0000 7f45c04ac700 1 --2- 172.21.15.42:0/2705021133 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f45b80a23b0 0x7f45b80a5a90 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:42.969 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.960+0000 7f45c04ac700 1 --2- 172.21.15.42:0/2705021133 >> v2:172.21.15.42:3300/0 conn(0x7f45b80a1f80 0x7f45b80a5540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:42.969 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.964+0000 7f45c04ac700 1 -- 172.21.15.42:0/2705021133 shutdown_connections 2019-12-05T05:03:42.970 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.964+0000 7f45c04ac700 1 -- 172.21.15.42:0/2705021133 wait complete. 2019-12-05T05:03:42.970 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:42.964+0000 7f45c04ac700 1 -- 172.21.15.42:0/2705021133 >> 172.21.15.42:0/2705021133 conn(0x7f45b80a8110 msgr2=0x7f45b80596f0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:42.977 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574853 for osd.4 2019-12-05T05:03:42.977 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2019-12-05T05:03:43.100 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.068+0000 7fbfeb056700 1 Processor -- start 2019-12-05T05:03:43.100 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.068+0000 7fbfeb056700 1 -- start start 2019-12-05T05:03:43.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.068+0000 7fbfeb056700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fbfe40a7740 0x7fbfe408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:43.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.068+0000 7fbfeb056700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbfe409fae0 0x7fbfe407eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:43.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.068+0000 7fbfeb056700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfe40a0710 con 0x7fbfe407e720 2019-12-05T05:03:43.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.068+0000 7fbfeb056700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfe40a0b00 con 0x7fbfe40a7740 2019-12-05T05:03:43.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.068+0000 7fbfeb056700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfe408e5b0 con 0x7fbfe409fae0 2019-12-05T05:03:43.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.068+0000 7fbfe8df2700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fbfe407e720 0x7fbfe409f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:53546/0 (socket says 172.21.15.42:53546) 2019-12-05T05:03:43.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.068+0000 7fbfe8df2700 1 -- 172.21.15.42:0/2816724818 learned_addr learned my addr 172.21.15.42:0/2816724818 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:03:43.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.068+0000 7fbfe9df4700 1 --2- 172.21.15.42:0/2816724818 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbfe409fae0 0x7fbfe407eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:43.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.068+0000 7fbfe95f3700 1 --2- 172.21.15.42:0/2816724818 >> v2:172.21.15.40:3300/0 conn(0x7fbfe40a7740 0x7fbfe408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:43.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.068+0000 7fbfd7fff700 1 -- 172.21.15.42:0/2816724818 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3898317439 0 0) 0x7fbfe40a0710 con 0x7fbfe407e720 2019-12-05T05:03:43.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.068+0000 7fbfd7fff700 1 -- 172.21.15.42:0/2816724818 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbfd00030b0 con 0x7fbfe407e720 2019-12-05T05:03:43.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.068+0000 7fbfd7fff700 1 -- 172.21.15.42:0/2816724818 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 322717505 0 0) 0x7fbfd00030b0 con 0x7fbfe407e720 2019-12-05T05:03:43.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.068+0000 7fbfd7fff700 1 -- 172.21.15.42:0/2816724818 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbfe409fae0 msgr2=0x7fbfe407eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:03:43.103 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.068+0000 7fbfd7fff700 1 --2- 172.21.15.42:0/2816724818 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbfe409fae0 0x7fbfe407eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:03:43.103 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.068+0000 7fbfd7fff700 1 -- 172.21.15.42:0/2816724818 >> v2:172.21.15.40:3300/0 conn(0x7fbfe40a7740 msgr2=0x7fbfe408df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:03:43.103 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.068+0000 7fbfd7fff700 1 --2- 172.21.15.42:0/2816724818 >> v2:172.21.15.40:3300/0 conn(0x7fbfe40a7740 0x7fbfe408df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:03:43.103 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.068+0000 7fbfd7fff700 1 -- 172.21.15.42:0/2816724818 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfe407f0a0 con 0x7fbfe407e720 2019-12-05T05:03:43.103 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.068+0000 7fbfd7fff700 1 -- 172.21.15.42:0/2816724818 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7fbfe0001ad0 con 0x7fbfe407e720 2019-12-05T05:03:43.103 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.068+0000 7fbfd7fff700 1 -- 172.21.15.42:0/2816724818 >> v1:172.21.15.42:6789/0 conn(0x7fbfe407e720 legacy=0x7fbfe409f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:43.103 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.068+0000 7fbfd7fff700 1 --2- 172.21.15.42:0/2816724818 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbfd00036e0 0x7fbfd0003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:43.104 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.068+0000 7fbfd7fff700 1 --2- 172.21.15.42:0/2816724818 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbfd0004420 0x7fbfd0004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:43.104 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.068+0000 7fbfd7fff700 1 --2- 172.21.15.42:0/2816724818 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbfd0004da0 0x7fbfd004f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:43.104 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.068+0000 7fbfd7fff700 1 -- 172.21.15.42:0/2816724818 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfe40a0b00 con 0x7fbfd00036e0 2019-12-05T05:03:43.104 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.068+0000 7fbfd7fff700 1 -- 172.21.15.42:0/2816724818 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfe408e5b0 con 0x7fbfd0004da0 2019-12-05T05:03:43.104 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.068+0000 7fbfd7fff700 1 -- 172.21.15.42:0/2816724818 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfd004fc20 con 0x7fbfd0004420 2019-12-05T05:03:43.105 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.068+0000 7fbfe9df4700 1 --2- 172.21.15.42:0/2816724818 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbfe409fae0 0x7fbfe407eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:03:43.105 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.068+0000 7fbfe9df4700 1 --2- 172.21.15.42:0/2816724818 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbfd0004da0 0x7fbfd004f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:43.105 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.068+0000 7fbfe95f3700 1 --2- 172.21.15.42:0/2816724818 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbfd00036e0 0x7fbfd0003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:43.105 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.068+0000 7fbfe8df2700 1 --2- 172.21.15.42:0/2816724818 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbfd0004420 0x7fbfd0004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:43.106 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.068+0000 7fbfe9df4700 1 -- 172.21.15.42:0/2816724818 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbfd0004420 msgr2=0x7fbfd0004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:03:43.106 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.068+0000 7fbfe9df4700 1 --2- 172.21.15.42:0/2816724818 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbfd0004420 0x7fbfd0004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:03:43.106 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.068+0000 7fbfe9df4700 1 -- 172.21.15.42:0/2816724818 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbfd00036e0 msgr2=0x7fbfd0003b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:03:43.106 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.068+0000 7fbfe9df4700 1 --2- 172.21.15.42:0/2816724818 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbfd00036e0 0x7fbfd0003b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:03:43.106 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.068+0000 7fbfe9df4700 1 -- 172.21.15.42:0/2816724818 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfd004ff20 con 0x7fbfd0004da0 2019-12-05T05:03:43.106 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.068+0000 7fbfe95f3700 1 --2- 172.21.15.42:0/2816724818 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbfd00036e0 0x7fbfd0003b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:03:43.107 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.068+0000 7fbfe9df4700 1 --2- 172.21.15.42:0/2816724818 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbfd0004da0 0x7fbfd004f680 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fbfdc002f30 tx=0x7fbfdc00fc80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:43.107 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.072+0000 7fbfe9df4700 1 --2- 172.21.15.42:0/2816724818 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbfd0004da0 0x7fbfd004f680 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fbfdc002f30 tx=0x7fbfdc00fc80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:43.107 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.072+0000 7fbfe9df4700 1 --2- 172.21.15.42:0/2816724818 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbfd0004da0 0x7fbfd004f680 secure :-1 s=READY pgs=135 cs=0 l=1 rx=0x7fbfdc002f30 tx=0x7fbfdc00fc80).ready entity=mon.1 client_cookie=2eed6d8b0fb07e9d server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:43.107 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.072+0000 7fbfe9df4700 1 --2- 172.21.15.42:0/2816724818 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbfd0004da0 0x7fbfd004f680 secure :-1 s=THROTTLE_DONE pgs=135 cs=0 l=1 rx=0x7fbfdc002f30 tx=0x7fbfdc00fc80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:43.107 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.072+0000 7fbfd7fff700 1 -- 172.21.15.42:0/2816724818 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbfdc011b90 con 0x7fbfd0004da0 2019-12-05T05:03:43.108 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.072+0000 7fbfe9df4700 1 --2- 172.21.15.42:0/2816724818 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbfd0004da0 0x7fbfd004f680 secure :-1 s=THROTTLE_DONE pgs=135 cs=0 l=1 rx=0x7fbfdc002f30 tx=0x7fbfdc00fc80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:43.108 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.072+0000 7fbfd7fff700 1 -- 172.21.15.42:0/2816724818 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfdc0117b0 con 0x7fbfd0004da0 2019-12-05T05:03:43.108 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.072+0000 7fbfe9df4700 1 --2- 172.21.15.42:0/2816724818 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbfd0004da0 0x7fbfd004f680 secure :-1 s=THROTTLE_DONE pgs=135 cs=0 l=1 rx=0x7fbfdc002f30 tx=0x7fbfdc00fc80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:43.108 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.072+0000 7fbfeb056700 1 -- 172.21.15.42:0/2816724818 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbfd0004da0 msgr2=0x7fbfd004f680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:43.108 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.072+0000 7fbfd7fff700 1 -- 172.21.15.42:0/2816724818 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbfdc005730 con 0x7fbfd0004da0 2019-12-05T05:03:43.108 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.072+0000 7fbfeb056700 1 --2- 172.21.15.42:0/2816724818 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbfd0004da0 0x7fbfd004f680 secure :-1 s=READY pgs=135 cs=0 l=1 rx=0x7fbfdc002f30 tx=0x7fbfdc00fc80).stop 2019-12-05T05:03:43.109 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.072+0000 7fbfe9df4700 1 -- 172.21.15.42:0/2816724818 reap_dead start 2019-12-05T05:03:43.109 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.072+0000 7fbfeb056700 1 -- 172.21.15.42:0/2816724818 shutdown_connections 2019-12-05T05:03:43.109 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.072+0000 7fbfeb056700 1 -- 172.21.15.42:0/2816724818 shutdown_connections 2019-12-05T05:03:43.109 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.072+0000 7fbfeb056700 1 -- 172.21.15.42:0/2816724818 wait complete. 2019-12-05T05:03:43.109 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.072+0000 7fbfeb056700 1 -- 172.21.15.42:0/2816724818 >> 172.21.15.42:0/2816724818 conn(0x7fbfe40aabe0 msgr2=0x7fbfe40a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:43.109 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.076+0000 7fbfeb056700 1 Processor -- start 2019-12-05T05:03:43.109 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.076+0000 7fbfeb056700 1 -- start start 2019-12-05T05:03:43.110 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.076+0000 7fbfeb056700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbfe40a1510 0x7fbfe4122b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:43.110 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.076+0000 7fbfeb056700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fbfe4123050 0x7fbfe4123480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:43.110 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.076+0000 7fbfeb056700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfe4065200 con 0x7fbfe408eeb0 2019-12-05T05:03:43.110 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.076+0000 7fbfeb056700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfe40a42f0 con 0x7fbfe4123050 2019-12-05T05:03:43.110 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.076+0000 7fbfeb056700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfe40a1b70 con 0x7fbfe40a1510 2019-12-05T05:03:43.110 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.076+0000 7fbfe9df4700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fbfe4123050 0x7fbfe4123480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:43.110 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.076+0000 7fbfe9df4700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fbfe4123050 0x7fbfe4123480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:53522/0 (socket says 172.21.15.42:53522) 2019-12-05T05:03:43.111 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.076+0000 7fbfe9df4700 1 -- 172.21.15.42:0/2899959935 learned_addr learned my addr 172.21.15.42:0/2899959935 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:03:43.111 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.076+0000 7fbfe8df2700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbfe40a1510 0x7fbfe4122b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:43.111 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.076+0000 7fbfe95f3700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fbfe408eeb0 0x7fbfe4064a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:34766/0 (socket says 172.21.15.42:34766) 2019-12-05T05:03:43.111 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.076+0000 7fbfd5ffb700 1 -- 172.21.15.42:0/2899959935 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2861917300 0 0) 0x7fbfe4065200 con 0x7fbfe408eeb0 2019-12-05T05:03:43.111 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.076+0000 7fbfd5ffb700 1 -- 172.21.15.42:0/2899959935 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbfc40030b0 con 0x7fbfe408eeb0 2019-12-05T05:03:43.111 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.076+0000 7fbfe9df4700 1 -- 172.21.15.42:0/2899959935 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbfe40a1510 msgr2=0x7fbfe4122b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:43.111 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.076+0000 7fbfe9df4700 1 --2- 172.21.15.42:0/2899959935 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbfe40a1510 0x7fbfe4122b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:43.112 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.076+0000 7fbfe9df4700 1 -- 172.21.15.42:0/2899959935 >> v1:172.21.15.40:6789/0 conn(0x7fbfe408eeb0 legacy=0x7fbfe4064a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:43.112 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.076+0000 7fbfe9df4700 1 -- 172.21.15.42:0/2899959935 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfe4065770 con 0x7fbfe4123050 2019-12-05T05:03:43.112 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.076+0000 7fbfe8df2700 1 --2- 172.21.15.42:0/2899959935 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbfe40a1510 0x7fbfe4122b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:03:43.112 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.076+0000 7fbfe9df4700 1 --2- 172.21.15.42:0/2899959935 >> v2:172.21.15.42:3300/0 conn(0x7fbfe4123050 0x7fbfe4123480 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fbfe40a44b0 tx=0x7fbfdc011a60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:43.112 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.076+0000 7fbfe9df4700 1 --2- 172.21.15.42:0/2899959935 >> v2:172.21.15.42:3300/0 conn(0x7fbfe4123050 0x7fbfe4123480 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fbfe40a44b0 tx=0x7fbfdc011a60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:43.112 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.076+0000 7fbfe9df4700 1 --2- 172.21.15.42:0/2899959935 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbfe4123050 0x7fbfe4123480 secure :-1 s=READY pgs=136 cs=0 l=1 rx=0x7fbfe40a44b0 tx=0x7fbfdc011a60).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:43.112 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.076+0000 7fbfe9df4700 1 --2- 172.21.15.42:0/2899959935 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbfe4123050 0x7fbfe4123480 secure :-1 s=THROTTLE_DONE pgs=136 cs=0 l=1 rx=0x7fbfe40a44b0 tx=0x7fbfdc011a60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:43.113 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.076+0000 7fbfd5ffb700 1 -- 172.21.15.42:0/2899959935 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbfdc005750 con 0x7fbfe4123050 2019-12-05T05:03:43.113 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.076+0000 7fbfe9df4700 1 --2- 172.21.15.42:0/2899959935 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbfe4123050 0x7fbfe4123480 secure :-1 s=THROTTLE_DONE pgs=136 cs=0 l=1 rx=0x7fbfe40a44b0 tx=0x7fbfdc011a60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:43.113 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.076+0000 7fbfd5ffb700 1 -- 172.21.15.42:0/2899959935 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfdc011d90 con 0x7fbfe4123050 2019-12-05T05:03:43.113 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.076+0000 7fbfe9df4700 1 --2- 172.21.15.42:0/2899959935 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbfe4123050 0x7fbfe4123480 secure :-1 s=THROTTLE_DONE pgs=136 cs=0 l=1 rx=0x7fbfe40a44b0 tx=0x7fbfdc011a60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:43.113 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.076+0000 7fbfeb056700 1 -- 172.21.15.42:0/2899959935 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfe40a29f0 con 0x7fbfe4123050 2019-12-05T05:03:43.113 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.076+0000 7fbfd5ffb700 1 -- 172.21.15.42:0/2899959935 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbfdc005d30 con 0x7fbfe4123050 2019-12-05T05:03:43.113 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.076+0000 7fbfeb056700 1 -- 172.21.15.42:0/2899959935 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfe4129ec0 con 0x7fbfe4123050 2019-12-05T05:03:43.114 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.080+0000 7fbfe9df4700 1 --2- 172.21.15.42:0/2899959935 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbfe4123050 0x7fbfe4123480 secure :-1 s=THROTTLE_DONE pgs=136 cs=0 l=1 rx=0x7fbfe40a44b0 tx=0x7fbfdc011a60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:43.114 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.080+0000 7fbfd5ffb700 1 -- 172.21.15.42:0/2899959935 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fbfdc012060 con 0x7fbfe4123050 2019-12-05T05:03:43.114 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.080+0000 7fbfe9df4700 1 --2- 172.21.15.42:0/2899959935 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbfe4123050 0x7fbfe4123480 secure :-1 s=THROTTLE_DONE pgs=136 cs=0 l=1 rx=0x7fbfe40a44b0 tx=0x7fbfdc011a60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:43.114 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.080+0000 7fbfd5ffb700 1 --2- 172.21.15.42:0/2899959935 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbfc403b1c0 0x7fbfc403d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:43.114 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.080+0000 7fbfd5ffb700 1 -- 172.21.15.42:0/2899959935 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6031+0+0 (secure 0 0 0) 0x7fbfd8008280 con 0x7fbfe4123050 2019-12-05T05:03:43.114 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.080+0000 7fbfe95f3700 1 --2- 172.21.15.42:0/2899959935 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbfc403b1c0 0x7fbfc403d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:43.114 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.080+0000 7fbfe95f3700 1 --2- 172.21.15.42:0/2899959935 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbfc403b1c0 0x7fbfc403d6d0 crc :-1 s=READY pgs=78 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:43.115 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.084+0000 7fbfeb056700 1 -- 172.21.15.42:0/2899959935 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfe405d110 con 0x7fbfe4123050 2019-12-05T05:03:43.115 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.092+0000 7fbfe9df4700 1 --2- 172.21.15.42:0/2899959935 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbfe4123050 0x7fbfe4123480 secure :-1 s=THROTTLE_DONE pgs=136 cs=0 l=1 rx=0x7fbfe40a44b0 tx=0x7fbfdc011a60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:43.115 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.092+0000 7fbfd5ffb700 1 -- 172.21.15.42:0/2899959935 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fbfdc0325f0 con 0x7fbfe4123050 2019-12-05T05:03:43.283 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.276+0000 7fbfeb056700 1 -- 172.21.15.42:0/2899959935 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fbfe40a8650 con 0x7fbfe4123050 2019-12-05T05:03:43.284 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.280+0000 7fbfe9df4700 1 --2- 172.21.15.42:0/2899959935 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbfe4123050 0x7fbfe4123480 secure :-1 s=THROTTLE_DONE pgs=136 cs=0 l=1 rx=0x7fbfe40a44b0 tx=0x7fbfdc011a60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:43.284 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.280+0000 7fbfd5ffb700 1 -- 172.21.15.42:0/2899959935 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fbfdc032850 con 0x7fbfe4123050 2019-12-05T05:03:43.284 INFO:teuthology.orchestra.run.smithi042.stdout:55834574854 2019-12-05T05:03:43.287 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.280+0000 7fbfeb056700 1 -- 172.21.15.42:0/2899959935 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbfc403b1c0 msgr2=0x7fbfc403d6d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:43.288 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.280+0000 7fbfeb056700 1 --2- 172.21.15.42:0/2899959935 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbfc403b1c0 0x7fbfc403d6d0 crc :-1 s=READY pgs=78 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:43.288 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.280+0000 7fbfeb056700 1 -- 172.21.15.42:0/2899959935 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbfe4123050 msgr2=0x7fbfe4123480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:43.288 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.280+0000 7fbfeb056700 1 --2- 172.21.15.42:0/2899959935 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbfe4123050 0x7fbfe4123480 secure :-1 s=READY pgs=136 cs=0 l=1 rx=0x7fbfe40a44b0 tx=0x7fbfdc011a60).stop 2019-12-05T05:03:43.288 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.280+0000 7fbfeb056700 1 -- 172.21.15.42:0/2899959935 shutdown_connections 2019-12-05T05:03:43.288 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.280+0000 7fbfeb056700 1 --2- 172.21.15.42:0/2899959935 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbfc403b1c0 0x7fbfc403d6d0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:43.288 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.280+0000 7fbfeb056700 1 --2- 172.21.15.42:0/2899959935 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbfe4123050 0x7fbfe4123480 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:43.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.280+0000 7fbfeb056700 1 --2- 172.21.15.42:0/2899959935 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbfe40a1510 0x7fbfe4122b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:43.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.280+0000 7fbfeb056700 1 -- 172.21.15.42:0/2899959935 shutdown_connections 2019-12-05T05:03:43.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.280+0000 7fbfeb056700 1 -- 172.21.15.42:0/2899959935 wait complete. 2019-12-05T05:03:43.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.280+0000 7fbfeb056700 1 -- 172.21.15.42:0/2899959935 >> 172.21.15.42:0/2899959935 conn(0x7fbfe40aabe0 msgr2=0x7fbfe411c310 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:43.297 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.5 2019-12-05T05:03:43.297 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2019-12-05T05:03:43.397 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.388+0000 7f6a7427a700 1 Processor -- start 2019-12-05T05:03:43.398 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.388+0000 7f6a7427a700 1 -- start start 2019-12-05T05:03:43.398 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.388+0000 7f6a7427a700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6a6c0a7730 0x7f6a6c08df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:43.398 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.388+0000 7f6a7427a700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f6a6c09fad0 0x7f6a6c07eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:43.398 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.388+0000 7f6a7427a700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a6c0a0700 con 0x7f6a6c07e710 2019-12-05T05:03:43.398 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.388+0000 7f6a7427a700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a6c0a0af0 con 0x7f6a6c09fad0 2019-12-05T05:03:43.399 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.388+0000 7f6a7427a700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a6c08e5a0 con 0x7f6a6c0a7730 2019-12-05T05:03:43.399 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.388+0000 7f6a72016700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f6a6c07e710 0x7f6a6c09f2d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:53566/0 (socket says 172.21.15.42:53566) 2019-12-05T05:03:43.399 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.388+0000 7f6a72016700 1 -- 172.21.15.42:0/3211166594 learned_addr learned my addr 172.21.15.42:0/3211166594 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:03:43.399 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.388+0000 7f6a73018700 1 --2- 172.21.15.42:0/3211166594 >> v2:172.21.15.40:3300/0 conn(0x7f6a6c09fad0 0x7f6a6c07eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:43.399 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.388+0000 7f6a72817700 1 --2- 172.21.15.42:0/3211166594 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6a6c0a7730 0x7f6a6c08df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:43.399 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.388+0000 7f6a71815700 1 -- 172.21.15.42:0/3211166594 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1732572530 0 0) 0x7f6a6c0a0700 con 0x7f6a6c07e710 2019-12-05T05:03:43.399 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.388+0000 7f6a71815700 1 -- 172.21.15.42:0/3211166594 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a540030b0 con 0x7f6a6c07e710 2019-12-05T05:03:43.400 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.388+0000 7f6a71815700 1 -- 172.21.15.42:0/3211166594 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1521547240 0 0) 0x7f6a540030b0 con 0x7f6a6c07e710 2019-12-05T05:03:43.400 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.388+0000 7f6a73018700 1 -- 172.21.15.42:0/3211166594 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6a6c0a7730 msgr2=0x7f6a6c08df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:03:43.400 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.388+0000 7f6a73018700 1 --2- 172.21.15.42:0/3211166594 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6a6c0a7730 0x7f6a6c08df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:03:43.400 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.388+0000 7f6a73018700 1 -- 172.21.15.42:0/3211166594 >> v1:172.21.15.42:6789/0 conn(0x7f6a6c07e710 legacy=0x7f6a6c09f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:43.400 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.388+0000 7f6a73018700 1 -- 172.21.15.42:0/3211166594 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a6c07f090 con 0x7f6a6c09fad0 2019-12-05T05:03:43.400 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.388+0000 7f6a72817700 1 --2- 172.21.15.42:0/3211166594 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6a6c0a7730 0x7f6a6c08df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:03:43.400 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.388+0000 7f6a73018700 1 --2- 172.21.15.42:0/3211166594 >> v2:172.21.15.40:3300/0 conn(0x7f6a6c09fad0 0x7f6a6c07eb40 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f6a5c008790 tx=0x7f6a64001820).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:43.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.388+0000 7f6a73018700 1 --2- 172.21.15.42:0/3211166594 >> v2:172.21.15.40:3300/0 conn(0x7f6a6c09fad0 0x7f6a6c07eb40 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f6a5c008790 tx=0x7f6a64001820).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:43.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.388+0000 7f6a73018700 1 --2- 172.21.15.42:0/3211166594 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6a6c09fad0 0x7f6a6c07eb40 secure :-1 s=READY pgs=118 cs=0 l=1 rx=0x7f6a5c008790 tx=0x7f6a64001820).ready entity=mon.0 client_cookie=65239e73766297e0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:43.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.388+0000 7f6a73018700 1 --2- 172.21.15.42:0/3211166594 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6a6c09fad0 0x7f6a6c07eb40 secure :-1 s=THROTTLE_DONE pgs=118 cs=0 l=1 rx=0x7f6a5c008790 tx=0x7f6a64001820).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:43.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.388+0000 7f6a73018700 1 --2- 172.21.15.42:0/3211166594 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6a6c09fad0 0x7f6a6c07eb40 secure :-1 s=THROTTLE_DONE pgs=118 cs=0 l=1 rx=0x7f6a5c008790 tx=0x7f6a64001820).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:43.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.388+0000 7f6a71815700 1 -- 172.21.15.42:0/3211166594 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6a64001ad0 con 0x7f6a6c09fad0 2019-12-05T05:03:43.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.388+0000 7f6a73018700 1 --2- 172.21.15.42:0/3211166594 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6a6c09fad0 0x7f6a6c07eb40 secure :-1 s=THROTTLE_DONE pgs=118 cs=0 l=1 rx=0x7f6a5c008790 tx=0x7f6a64001820).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:43.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.388+0000 7f6a71815700 1 -- 172.21.15.42:0/3211166594 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a640018e0 con 0x7f6a6c09fad0 2019-12-05T05:03:43.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.388+0000 7f6a71815700 1 -- 172.21.15.42:0/3211166594 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6a5c008da0 con 0x7f6a6c09fad0 2019-12-05T05:03:43.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.388+0000 7f6a7427a700 1 -- 172.21.15.42:0/3211166594 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6a6c09fad0 msgr2=0x7f6a6c07eb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:43.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.388+0000 7f6a7427a700 1 --2- 172.21.15.42:0/3211166594 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6a6c09fad0 0x7f6a6c07eb40 secure :-1 s=READY pgs=118 cs=0 l=1 rx=0x7f6a5c008790 tx=0x7f6a64001820).stop 2019-12-05T05:03:43.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.392+0000 7f6a7427a700 1 -- 172.21.15.42:0/3211166594 shutdown_connections 2019-12-05T05:03:43.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.392+0000 7f6a7427a700 1 --2- 172.21.15.42:0/3211166594 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6a6c09fad0 0x7f6a6c07eb40 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:43.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.392+0000 7f6a7427a700 1 --2- 172.21.15.42:0/3211166594 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6a6c0a7730 0x7f6a6c08df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:03:43.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.392+0000 7f6a7427a700 1 -- 172.21.15.42:0/3211166594 shutdown_connections 2019-12-05T05:03:43.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.392+0000 7f6a7427a700 1 -- 172.21.15.42:0/3211166594 wait complete. 2019-12-05T05:03:43.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.392+0000 7f6a7427a700 1 -- 172.21.15.42:0/3211166594 >> 172.21.15.42:0/3211166594 conn(0x7f6a6c0aabd0 msgr2=0x7f6a6c0a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:43.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.392+0000 7f6a7427a700 1 Processor -- start 2019-12-05T05:03:43.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.392+0000 7f6a7427a700 1 -- start start 2019-12-05T05:03:43.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.396+0000 7f6a7427a700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f6a6c143410 0x7f6a6c11ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:43.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.396+0000 7f6a7427a700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6a6c11b360 0x7f6a6c11d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:43.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.396+0000 7f6a7427a700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a6c0a21b0 con 0x7f6a6c0a1d80 2019-12-05T05:03:43.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.396+0000 7f6a7427a700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a6c143d90 con 0x7f6a6c143410 2019-12-05T05:03:43.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.396+0000 7f6a7427a700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a6c144030 con 0x7f6a6c11b360 2019-12-05T05:03:43.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.396+0000 7f6a72817700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f6a6c0a1d80 0x7f6a6c130180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:53570/0 (socket says 172.21.15.42:53570) 2019-12-05T05:03:43.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.396+0000 7f6a72817700 1 -- 172.21.15.42:0/1319588954 learned_addr learned my addr 172.21.15.42:0/1319588954 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:03:43.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.396+0000 7f6a72016700 1 --2- 172.21.15.42:0/1319588954 >> v2:172.21.15.40:3300/0 conn(0x7f6a6c143410 0x7f6a6c11ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:43.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.396+0000 7f6a73018700 1 --2- 172.21.15.42:0/1319588954 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6a6c11b360 0x7f6a6c11d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:43.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.396+0000 7f6a6b7fe700 1 -- 172.21.15.42:0/1319588954 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 507264702 0 0) 0x7f6a6c0a21b0 con 0x7f6a6c0a1d80 2019-12-05T05:03:43.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.396+0000 7f6a6b7fe700 1 -- 172.21.15.42:0/1319588954 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a4c0030b0 con 0x7f6a6c0a1d80 2019-12-05T05:03:43.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.396+0000 7f6a72016700 1 -- 172.21.15.42:0/1319588954 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6a6c11b360 msgr2=0x7f6a6c11d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:43.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.396+0000 7f6a72016700 1 --2- 172.21.15.42:0/1319588954 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6a6c11b360 0x7f6a6c11d7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:43.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.396+0000 7f6a72016700 1 -- 172.21.15.42:0/1319588954 >> v1:172.21.15.42:6789/0 conn(0x7f6a6c0a1d80 legacy=0x7f6a6c130180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:43.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.396+0000 7f6a72016700 1 -- 172.21.15.42:0/1319588954 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a6c143840 con 0x7f6a6c143410 2019-12-05T05:03:43.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.396+0000 7f6a72016700 1 --2- 172.21.15.42:0/1319588954 >> v2:172.21.15.40:3300/0 conn(0x7f6a6c143410 0x7f6a6c11ae10 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f6a6c0a23e0 tx=0x7f6a64010cd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:43.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.396+0000 7f6a72016700 1 --2- 172.21.15.42:0/1319588954 >> v2:172.21.15.40:3300/0 conn(0x7f6a6c143410 0x7f6a6c11ae10 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f6a6c0a23e0 tx=0x7f6a64010cd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:43.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.396+0000 7f6a72016700 1 --2- 172.21.15.42:0/1319588954 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6a6c143410 0x7f6a6c11ae10 secure :-1 s=READY pgs=119 cs=0 l=1 rx=0x7f6a6c0a23e0 tx=0x7f6a64010cd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:43.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.396+0000 7f6a72016700 1 --2- 172.21.15.42:0/1319588954 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6a6c143410 0x7f6a6c11ae10 secure :-1 s=THROTTLE_DONE pgs=119 cs=0 l=1 rx=0x7f6a6c0a23e0 tx=0x7f6a64010cd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:43.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.396+0000 7f6a6b7fe700 1 -- 172.21.15.42:0/1319588954 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6a64004b90 con 0x7f6a6c143410 2019-12-05T05:03:43.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.396+0000 7f6a72016700 1 --2- 172.21.15.42:0/1319588954 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6a6c143410 0x7f6a6c11ae10 secure :-1 s=THROTTLE_DONE pgs=119 cs=0 l=1 rx=0x7f6a6c0a23e0 tx=0x7f6a64010cd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:43.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.396+0000 7f6a7427a700 1 -- 172.21.15.42:0/1319588954 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a6c144570 con 0x7f6a6c143410 2019-12-05T05:03:43.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.396+0000 7f6a6b7fe700 1 -- 172.21.15.42:0/1319588954 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a640047b0 con 0x7f6a6c143410 2019-12-05T05:03:43.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.396+0000 7f6a72016700 1 --2- 172.21.15.42:0/1319588954 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6a6c143410 0x7f6a6c11ae10 secure :-1 s=THROTTLE_DONE pgs=119 cs=0 l=1 rx=0x7f6a6c0a23e0 tx=0x7f6a64010cd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:43.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.396+0000 7f6a6b7fe700 1 -- 172.21.15.42:0/1319588954 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6a64012840 con 0x7f6a6c143410 2019-12-05T05:03:43.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.396+0000 7f6a7427a700 1 -- 172.21.15.42:0/1319588954 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a6c11dcf0 con 0x7f6a6c143410 2019-12-05T05:03:43.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.396+0000 7f6a72016700 1 --2- 172.21.15.42:0/1319588954 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6a6c143410 0x7f6a6c11ae10 secure :-1 s=THROTTLE_DONE pgs=119 cs=0 l=1 rx=0x7f6a6c0a23e0 tx=0x7f6a64010cd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:43.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.396+0000 7f6a6b7fe700 1 -- 172.21.15.42:0/1319588954 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f6a64012b60 con 0x7f6a6c143410 2019-12-05T05:03:43.411 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.396+0000 7f6a72016700 1 --2- 172.21.15.42:0/1319588954 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6a6c143410 0x7f6a6c11ae10 secure :-1 s=THROTTLE_DONE pgs=119 cs=0 l=1 rx=0x7f6a6c0a23e0 tx=0x7f6a64010cd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:43.411 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.400+0000 7f6a6b7fe700 1 --2- 172.21.15.42:0/1319588954 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6a4c03b1c0 0x7f6a4c03d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:43.411 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.400+0000 7f6a6b7fe700 1 -- 172.21.15.42:0/1319588954 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f6a6404b620 con 0x7f6a6c143410 2019-12-05T05:03:43.411 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.400+0000 7f6a72817700 1 --2- 172.21.15.42:0/1319588954 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6a4c03b1c0 0x7f6a4c03d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:43.411 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.400+0000 7f6a72817700 1 --2- 172.21.15.42:0/1319588954 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6a4c03b1c0 0x7f6a4c03d6d0 crc :-1 s=READY pgs=79 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:43.412 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.404+0000 7f6a7427a700 1 -- 172.21.15.42:0/1319588954 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a6c05d100 con 0x7f6a6c143410 2019-12-05T05:03:43.418 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.412+0000 7f6a72016700 1 --2- 172.21.15.42:0/1319588954 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6a6c143410 0x7f6a6c11ae10 secure :-1 s=THROTTLE_DONE pgs=119 cs=0 l=1 rx=0x7f6a6c0a23e0 tx=0x7f6a64010cd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:43.418 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.412+0000 7f6a6b7fe700 1 -- 172.21.15.42:0/1319588954 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f6a64032680 con 0x7f6a6c143410 2019-12-05T05:03:43.599 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.592+0000 7f6a7427a700 1 -- 172.21.15.42:0/1319588954 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f6a6c05b8b0 con 0x7f6a6c143410 2019-12-05T05:03:43.599 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.592+0000 7f6a72016700 1 --2- 172.21.15.42:0/1319588954 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6a6c143410 0x7f6a6c11ae10 secure :-1 s=THROTTLE_DONE pgs=119 cs=0 l=1 rx=0x7f6a6c0a23e0 tx=0x7f6a64010cd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:43.600 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.592+0000 7f6a6b7fe700 1 -- 172.21.15.42:0/1319588954 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f6a640328e0 con 0x7f6a6c143410 2019-12-05T05:03:43.600 INFO:teuthology.orchestra.run.smithi042.stdout:55834574855 2019-12-05T05:03:43.602 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.596+0000 7f6a7427a700 1 -- 172.21.15.42:0/1319588954 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6a4c03b1c0 msgr2=0x7f6a4c03d6d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:43.602 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.596+0000 7f6a7427a700 1 --2- 172.21.15.42:0/1319588954 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6a4c03b1c0 0x7f6a4c03d6d0 crc :-1 s=READY pgs=79 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:43.602 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.596+0000 7f6a7427a700 1 -- 172.21.15.42:0/1319588954 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6a6c143410 msgr2=0x7f6a6c11ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:43.603 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.596+0000 7f6a7427a700 1 --2- 172.21.15.42:0/1319588954 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6a6c143410 0x7f6a6c11ae10 secure :-1 s=READY pgs=119 cs=0 l=1 rx=0x7f6a6c0a23e0 tx=0x7f6a64010cd0).stop 2019-12-05T05:03:43.603 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.596+0000 7f6a7427a700 1 -- 172.21.15.42:0/1319588954 shutdown_connections 2019-12-05T05:03:43.603 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.596+0000 7f6a7427a700 1 --2- 172.21.15.42:0/1319588954 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6a4c03b1c0 0x7f6a4c03d6d0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:43.603 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.596+0000 7f6a7427a700 1 --2- 172.21.15.42:0/1319588954 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6a6c11b360 0x7f6a6c11d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:43.603 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.596+0000 7f6a7427a700 1 --2- 172.21.15.42:0/1319588954 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6a6c143410 0x7f6a6c11ae10 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:43.603 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.596+0000 7f6a7427a700 1 -- 172.21.15.42:0/1319588954 shutdown_connections 2019-12-05T05:03:43.603 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.596+0000 7f6a7427a700 1 -- 172.21.15.42:0/1319588954 wait complete. 2019-12-05T05:03:43.604 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.596+0000 7f6a7427a700 1 -- 172.21.15.42:0/1319588954 >> 172.21.15.42:0/1319588954 conn(0x7f6a6c0a7730 msgr2=0x7f6a6c09f370 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:43.612 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.6 2019-12-05T05:03:43.612 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2019-12-05T05:03:43.718 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.708+0000 7f41ae18a700 1 Processor -- start 2019-12-05T05:03:43.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.708+0000 7f41ae18a700 1 -- start start 2019-12-05T05:03:43.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.708+0000 7f41ae18a700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f41a809fad0 0x7f41a814ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:43.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.708+0000 7f41ae18a700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41a80a0700 con 0x7f41a808df70 2019-12-05T05:03:43.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.708+0000 7f41ae18a700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41a807ef10 con 0x7f41a80a7730 2019-12-05T05:03:43.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.708+0000 7f41ae18a700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41a80a0af0 con 0x7f41a809fad0 2019-12-05T05:03:43.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.708+0000 7f41a7fff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f41a80a7730 0x7f41a807e710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:53578/0 (socket says 172.21.15.42:53578) 2019-12-05T05:03:43.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.708+0000 7f41a7fff700 1 -- 172.21.15.42:0/1944506297 learned_addr learned my addr 172.21.15.42:0/1944506297 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:03:43.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.708+0000 7f41acf28700 1 --2- 172.21.15.42:0/1944506297 >> v2:172.21.15.42:3300/0 conn(0x7f41a809fad0 0x7f41a814ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:43.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.708+0000 7f41a6ffd700 1 -- 172.21.15.42:0/1944506297 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1171746011 0 0) 0x7f41a807ef10 con 0x7f41a80a7730 2019-12-05T05:03:43.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.708+0000 7f41a6ffd700 1 -- 172.21.15.42:0/1944506297 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41940030b0 con 0x7f41a80a7730 2019-12-05T05:03:43.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.708+0000 7f41a6ffd700 1 -- 172.21.15.42:0/1944506297 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3027759436 0 0) 0x7f41a80a0700 con 0x7f41a808df70 2019-12-05T05:03:43.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.708+0000 7f41acf28700 1 -- 172.21.15.42:0/1944506297 >> v1:172.21.15.42:6789/0 conn(0x7f41a80a7730 legacy=0x7f41a807e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:43.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.708+0000 7f41acf28700 1 -- 172.21.15.42:0/1944506297 >> v1:172.21.15.40:6789/0 conn(0x7f41a808df70 legacy=0x7f41a809f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:43.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.708+0000 7f41acf28700 1 -- 172.21.15.42:0/1944506297 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41a80a9b90 con 0x7f41a809fad0 2019-12-05T05:03:43.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.708+0000 7f41acf28700 1 --2- 172.21.15.42:0/1944506297 >> v2:172.21.15.42:3300/0 conn(0x7f41a809fad0 0x7f41a814ffe0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f41a0008b10 tx=0x7f41a000ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:43.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.708+0000 7f41acf28700 1 --2- 172.21.15.42:0/1944506297 >> v2:172.21.15.42:3300/0 conn(0x7f41a809fad0 0x7f41a814ffe0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f41a0008b10 tx=0x7f41a000ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:43.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.708+0000 7f41acf28700 1 --2- 172.21.15.42:0/1944506297 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f41a809fad0 0x7f41a814ffe0 secure :-1 s=READY pgs=140 cs=0 l=1 rx=0x7f41a0008b10 tx=0x7f41a000ff70).ready entity=mon.1 client_cookie=c17f27227fcb4e71 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:43.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.708+0000 7f41acf28700 1 --2- 172.21.15.42:0/1944506297 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f41a809fad0 0x7f41a814ffe0 secure :-1 s=THROTTLE_DONE pgs=140 cs=0 l=1 rx=0x7f41a0008b10 tx=0x7f41a000ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:43.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.708+0000 7f41a6ffd700 1 -- 172.21.15.42:0/1944506297 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f41a0010970 con 0x7f41a809fad0 2019-12-05T05:03:43.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.708+0000 7f41acf28700 1 --2- 172.21.15.42:0/1944506297 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f41a809fad0 0x7f41a814ffe0 secure :-1 s=THROTTLE_DONE pgs=140 cs=0 l=1 rx=0x7f41a0008b10 tx=0x7f41a000ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:43.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.708+0000 7f41acf28700 1 --2- 172.21.15.42:0/1944506297 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f41a809fad0 0x7f41a814ffe0 secure :-1 s=THROTTLE_DONE pgs=140 cs=0 l=1 rx=0x7f41a0008b10 tx=0x7f41a000ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:43.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.708+0000 7f41a6ffd700 1 -- 172.21.15.42:0/1944506297 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41a0008b50 con 0x7f41a809fad0 2019-12-05T05:03:43.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.708+0000 7f41a6ffd700 1 -- 172.21.15.42:0/1944506297 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f41a00110c0 con 0x7f41a809fad0 2019-12-05T05:03:43.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.712+0000 7f41ae18a700 1 -- 172.21.15.42:0/1944506297 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f41a809fad0 msgr2=0x7f41a814ffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:43.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.712+0000 7f41ae18a700 1 --2- 172.21.15.42:0/1944506297 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f41a809fad0 0x7f41a814ffe0 secure :-1 s=READY pgs=140 cs=0 l=1 rx=0x7f41a0008b10 tx=0x7f41a000ff70).stop 2019-12-05T05:03:43.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.712+0000 7f41ae18a700 1 -- 172.21.15.42:0/1944506297 shutdown_connections 2019-12-05T05:03:43.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.712+0000 7f41ae18a700 1 --2- 172.21.15.42:0/1944506297 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f41a809fad0 0x7f41a814ffe0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:43.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.712+0000 7f41ae18a700 1 -- 172.21.15.42:0/1944506297 shutdown_connections 2019-12-05T05:03:43.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.712+0000 7f41ae18a700 1 -- 172.21.15.42:0/1944506297 wait complete. 2019-12-05T05:03:43.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.712+0000 7f41ae18a700 1 -- 172.21.15.42:0/1944506297 >> 172.21.15.42:0/1944506297 conn(0x7f41a80aabd0 msgr2=0x7f41a80a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:43.726 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.712+0000 7f41ae18a700 1 Processor -- start 2019-12-05T05:03:43.727 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.712+0000 7f41ae18a700 1 -- start start 2019-12-05T05:03:43.727 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.716+0000 7f41ae18a700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f41a807acd0 0x7f41a80a60d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:43.727 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.716+0000 7f41ae18a700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f41a80a6620 0x7f41a8077ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:43.727 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.716+0000 7f41ae18a700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f41a8078230 0x7f41a8117620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:43.727 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.716+0000 7f41ae18a700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41a8119270 con 0x7f41a8078230 2019-12-05T05:03:43.727 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.716+0000 7f41ae18a700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41a8119510 con 0x7f41a807acd0 2019-12-05T05:03:43.727 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.716+0000 7f41ae18a700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41a81197b0 con 0x7f41a80a6620 2019-12-05T05:03:43.728 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.716+0000 7f41a7fff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f41a807acd0 0x7f41a80a60d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:43.728 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.716+0000 7f41acf28700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f41a8078230 0x7f41a8117620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:43.728 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.716+0000 7f41a77fe700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f41a80a6620 0x7f41a8077ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:43.728 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.716+0000 7f41a7fff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f41a807acd0 0x7f41a80a60d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:53548/0 (socket says 172.21.15.42:53548) 2019-12-05T05:03:43.728 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.716+0000 7f41a7fff700 1 -- 172.21.15.42:0/1639275921 learned_addr learned my addr 172.21.15.42:0/1639275921 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:03:43.728 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.716+0000 7f41a7fff700 1 -- 172.21.15.42:0/1639275921 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f41a80a6620 msgr2=0x7f41a8077ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:43.729 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.716+0000 7f41a7fff700 1 --2- 172.21.15.42:0/1639275921 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f41a80a6620 0x7f41a8077ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:43.729 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.716+0000 7f41a7fff700 1 -- 172.21.15.42:0/1639275921 >> v2:172.21.15.40:3300/0 conn(0x7f41a8078230 msgr2=0x7f41a8117620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:43.729 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.716+0000 7f41a7fff700 1 --2- 172.21.15.42:0/1639275921 >> v2:172.21.15.40:3300/0 conn(0x7f41a8078230 0x7f41a8117620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:43.729 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.716+0000 7f41a7fff700 1 -- 172.21.15.42:0/1639275921 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41a8117b70 con 0x7f41a807acd0 2019-12-05T05:03:43.729 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.716+0000 7f41a77fe700 1 --2- 172.21.15.42:0/1639275921 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f41a80a6620 0x7f41a8077ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:03:43.729 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.716+0000 7f41a7fff700 1 --2- 172.21.15.42:0/1639275921 >> v2:172.21.15.42:3300/0 conn(0x7f41a807acd0 0x7f41a80a60d0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f4198000b80 tx=0x7f4198010970).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:43.729 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.716+0000 7f41a7fff700 1 --2- 172.21.15.42:0/1639275921 >> v2:172.21.15.42:3300/0 conn(0x7f41a807acd0 0x7f41a80a60d0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f4198000b80 tx=0x7f4198010970).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:43.729 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.716+0000 7f41a7fff700 1 --2- 172.21.15.42:0/1639275921 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f41a807acd0 0x7f41a80a60d0 secure :-1 s=READY pgs=141 cs=0 l=1 rx=0x7f4198000b80 tx=0x7f4198010970).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:43.730 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.716+0000 7f41a7fff700 1 --2- 172.21.15.42:0/1639275921 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f41a807acd0 0x7f41a80a60d0 secure :-1 s=THROTTLE_DONE pgs=141 cs=0 l=1 rx=0x7f4198000b80 tx=0x7f4198010970).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:43.730 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.716+0000 7f41a4ff9700 1 -- 172.21.15.42:0/1639275921 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4198005b40 con 0x7f41a807acd0 2019-12-05T05:03:43.730 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.716+0000 7f41a7fff700 1 --2- 172.21.15.42:0/1639275921 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f41a807acd0 0x7f41a80a60d0 secure :-1 s=THROTTLE_DONE pgs=141 cs=0 l=1 rx=0x7f4198000b80 tx=0x7f4198010970).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:43.730 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.716+0000 7f41a7fff700 1 --2- 172.21.15.42:0/1639275921 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f41a807acd0 0x7f41a80a60d0 secure :-1 s=THROTTLE_DONE pgs=141 cs=0 l=1 rx=0x7f4198000b80 tx=0x7f4198010970).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:43.730 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.716+0000 7f41a4ff9700 1 -- 172.21.15.42:0/1639275921 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4198005760 con 0x7f41a807acd0 2019-12-05T05:03:43.730 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.716+0000 7f41a4ff9700 1 -- 172.21.15.42:0/1639275921 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f41980062b0 con 0x7f41a807acd0 2019-12-05T05:03:43.730 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.716+0000 7f41ae18a700 1 -- 172.21.15.42:0/1639275921 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41a8117e40 con 0x7f41a807acd0 2019-12-05T05:03:43.731 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.716+0000 7f41ae18a700 1 -- 172.21.15.42:0/1639275921 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41a808eb60 con 0x7f41a807acd0 2019-12-05T05:03:43.731 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.716+0000 7f41a7fff700 1 --2- 172.21.15.42:0/1639275921 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f41a807acd0 0x7f41a80a60d0 secure :-1 s=THROTTLE_DONE pgs=141 cs=0 l=1 rx=0x7f4198000b80 tx=0x7f4198010970).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:43.731 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.720+0000 7f41a4ff9700 1 -- 172.21.15.42:0/1639275921 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f41980065c0 con 0x7f41a807acd0 2019-12-05T05:03:43.731 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.720+0000 7f41a7fff700 1 --2- 172.21.15.42:0/1639275921 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f41a807acd0 0x7f41a80a60d0 secure :-1 s=THROTTLE_DONE pgs=141 cs=0 l=1 rx=0x7f4198000b80 tx=0x7f4198010970).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:43.731 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.720+0000 7f41a4ff9700 1 --2- 172.21.15.42:0/1639275921 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f418c03ae60 0x7f418c03d370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:43.731 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.720+0000 7f41a4ff9700 1 -- 172.21.15.42:0/1639275921 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f419804afb0 con 0x7f41a807acd0 2019-12-05T05:03:43.731 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.720+0000 7f41a77fe700 1 --2- 172.21.15.42:0/1639275921 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f418c03ae60 0x7f418c03d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:43.732 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.720+0000 7f41a77fe700 1 --2- 172.21.15.42:0/1639275921 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f418c03ae60 0x7f418c03d370 crc :-1 s=READY pgs=80 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:43.732 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.724+0000 7f41ae18a700 1 -- 172.21.15.42:0/1639275921 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41a805d100 con 0x7f41a807acd0 2019-12-05T05:03:43.737 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.732+0000 7f41a7fff700 1 --2- 172.21.15.42:0/1639275921 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f41a807acd0 0x7f41a80a60d0 secure :-1 s=THROTTLE_DONE pgs=141 cs=0 l=1 rx=0x7f4198000b80 tx=0x7f4198010970).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:43.738 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.732+0000 7f41a4ff9700 1 -- 172.21.15.42:0/1639275921 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f4198032030 con 0x7f41a807acd0 2019-12-05T05:03:43.923 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.916+0000 7f41ae18a700 1 -- 172.21.15.42:0/1639275921 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f41a80a71d0 con 0x7f41a807acd0 2019-12-05T05:03:43.924 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.920+0000 7f41a7fff700 1 --2- 172.21.15.42:0/1639275921 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f41a807acd0 0x7f41a80a60d0 secure :-1 s=THROTTLE_DONE pgs=141 cs=0 l=1 rx=0x7f4198000b80 tx=0x7f4198010970).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:43.924 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.920+0000 7f41a4ff9700 1 -- 172.21.15.42:0/1639275921 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f4198032290 con 0x7f41a807acd0 2019-12-05T05:03:43.924 INFO:teuthology.orchestra.run.smithi042.stdout:55834574854 2019-12-05T05:03:43.927 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.920+0000 7f41ae18a700 1 -- 172.21.15.42:0/1639275921 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f418c03ae60 msgr2=0x7f418c03d370 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:43.927 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.920+0000 7f41ae18a700 1 --2- 172.21.15.42:0/1639275921 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f418c03ae60 0x7f418c03d370 crc :-1 s=READY pgs=80 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:43.927 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.920+0000 7f41ae18a700 1 -- 172.21.15.42:0/1639275921 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f41a807acd0 msgr2=0x7f41a80a60d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:43.927 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.920+0000 7f41ae18a700 1 --2- 172.21.15.42:0/1639275921 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f41a807acd0 0x7f41a80a60d0 secure :-1 s=READY pgs=141 cs=0 l=1 rx=0x7f4198000b80 tx=0x7f4198010970).stop 2019-12-05T05:03:43.927 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.920+0000 7f41ae18a700 1 -- 172.21.15.42:0/1639275921 shutdown_connections 2019-12-05T05:03:43.927 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.920+0000 7f41ae18a700 1 --2- 172.21.15.42:0/1639275921 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f418c03ae60 0x7f418c03d370 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:43.928 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.920+0000 7f41ae18a700 1 --2- 172.21.15.42:0/1639275921 >> v2:172.21.15.40:3300/0 conn(0x7f41a8078230 0x7f41a8117620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:43.928 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.920+0000 7f41ae18a700 1 --2- 172.21.15.42:0/1639275921 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f41a807acd0 0x7f41a80a60d0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:43.928 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.920+0000 7f41ae18a700 1 --2- 172.21.15.42:0/1639275921 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f41a80a6620 0x7f41a8077ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:43.928 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.920+0000 7f41ae18a700 1 -- 172.21.15.42:0/1639275921 shutdown_connections 2019-12-05T05:03:43.928 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.920+0000 7f41ae18a700 1 -- 172.21.15.42:0/1639275921 wait complete. 2019-12-05T05:03:43.928 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:43.920+0000 7f41ae18a700 1 -- 172.21.15.42:0/1639275921 >> 172.21.15.42:0/1639275921 conn(0x7f41a80a7730 msgr2=0x7f41a80596f0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:43.936 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.7 2019-12-05T05:03:43.936 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2019-12-05T05:03:43.936 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2019-12-05T05:03:44.032 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.020+0000 7efc1917b700 1 Processor -- start 2019-12-05T05:03:44.032 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.020+0000 7efc1917b700 1 -- start start 2019-12-05T05:03:44.032 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.024+0000 7efc1917b700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efc1408df80 0x7efc1409f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:44.032 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.024+0000 7efc1917b700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7efc1409f830 0x7efc1414efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:44.032 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.024+0000 7efc1917b700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc1409fc60 con 0x7efc140a7740 2019-12-05T05:03:44.033 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.024+0000 7efc1917b700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc140a0b00 con 0x7efc1409f830 2019-12-05T05:03:44.033 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.024+0000 7efc1917b700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc1408e5b0 con 0x7efc1408df80 2019-12-05T05:03:44.033 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.024+0000 7efc1897a700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7efc1409f830 0x7efc1414efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:44.033 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.024+0000 7efc12d9d700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efc1408df80 0x7efc1409f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:44.033 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.024+0000 7efc1897a700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7efc1409f830 0x7efc1414efe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:53558/0 (socket says 172.21.15.42:53558) 2019-12-05T05:03:44.033 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.024+0000 7efc1897a700 1 -- 172.21.15.42:0/3186311733 learned_addr learned my addr 172.21.15.42:0/3186311733 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:03:44.033 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.024+0000 7efc1897a700 1 -- 172.21.15.42:0/3186311733 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efc1408df80 msgr2=0x7efc1409f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:03:44.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.024+0000 7efc1897a700 1 --2- 172.21.15.42:0/3186311733 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efc1408df80 0x7efc1409f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:03:44.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.024+0000 7efc1897a700 1 -- 172.21.15.42:0/3186311733 >> v1:172.21.15.40:6789/0 conn(0x7efc140a7740 legacy=0x7efc1407e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:44.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.024+0000 7efc1897a700 1 -- 172.21.15.42:0/3186311733 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc140a0710 con 0x7efc1409f830 2019-12-05T05:03:44.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.024+0000 7efc1897a700 1 --2- 172.21.15.42:0/3186311733 >> v2:172.21.15.42:3300/0 conn(0x7efc1409f830 0x7efc1414efe0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7efc04008790 tx=0x7efc0400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:44.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.024+0000 7efc1897a700 1 --2- 172.21.15.42:0/3186311733 >> v2:172.21.15.42:3300/0 conn(0x7efc1409f830 0x7efc1414efe0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7efc04008790 tx=0x7efc0400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:44.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.024+0000 7efc1897a700 1 --2- 172.21.15.42:0/3186311733 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efc1409f830 0x7efc1414efe0 secure :-1 s=READY pgs=142 cs=0 l=1 rx=0x7efc04008790 tx=0x7efc0400fd30).ready entity=mon.1 client_cookie=23c0d273016e5637 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:44.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.024+0000 7efc1897a700 1 --2- 172.21.15.42:0/3186311733 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efc1409f830 0x7efc1414efe0 secure :-1 s=THROTTLE_DONE pgs=142 cs=0 l=1 rx=0x7efc04008790 tx=0x7efc0400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:44.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.024+0000 7efc1897a700 1 --2- 172.21.15.42:0/3186311733 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efc1409f830 0x7efc1414efe0 secure :-1 s=THROTTLE_DONE pgs=142 cs=0 l=1 rx=0x7efc04008790 tx=0x7efc0400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:44.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.024+0000 7efc1259c700 1 -- 172.21.15.42:0/3186311733 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efc04008c90 con 0x7efc1409f830 2019-12-05T05:03:44.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.024+0000 7efc1897a700 1 --2- 172.21.15.42:0/3186311733 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efc1409f830 0x7efc1414efe0 secure :-1 s=THROTTLE_DONE pgs=142 cs=0 l=1 rx=0x7efc04008790 tx=0x7efc0400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:44.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.024+0000 7efc1259c700 1 -- 172.21.15.42:0/3186311733 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc040088b0 con 0x7efc1409f830 2019-12-05T05:03:44.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.024+0000 7efc1259c700 1 -- 172.21.15.42:0/3186311733 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efc04011b10 con 0x7efc1409f830 2019-12-05T05:03:44.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.024+0000 7efc1917b700 1 -- 172.21.15.42:0/3186311733 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efc1409f830 msgr2=0x7efc1414efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:44.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.024+0000 7efc1917b700 1 --2- 172.21.15.42:0/3186311733 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efc1409f830 0x7efc1414efe0 secure :-1 s=READY pgs=142 cs=0 l=1 rx=0x7efc04008790 tx=0x7efc0400fd30).stop 2019-12-05T05:03:44.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.024+0000 7efc1917b700 1 -- 172.21.15.42:0/3186311733 shutdown_connections 2019-12-05T05:03:44.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.024+0000 7efc1917b700 1 --2- 172.21.15.42:0/3186311733 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efc1409f830 0x7efc1414efe0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:44.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.024+0000 7efc1917b700 1 --2- 172.21.15.42:0/3186311733 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efc1408df80 0x7efc1409f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:03:44.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.024+0000 7efc1917b700 1 -- 172.21.15.42:0/3186311733 shutdown_connections 2019-12-05T05:03:44.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.024+0000 7efc1917b700 1 -- 172.21.15.42:0/3186311733 wait complete. 2019-12-05T05:03:44.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.024+0000 7efc1917b700 1 -- 172.21.15.42:0/3186311733 >> 172.21.15.42:0/3186311733 conn(0x7efc140aabe0 msgr2=0x7efc140a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:44.040 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.028+0000 7efc1917b700 1 Processor -- start 2019-12-05T05:03:44.040 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.028+0000 7efc1917b700 1 -- start start 2019-12-05T05:03:44.040 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.028+0000 7efc1917b700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efc14117b00 0x7efc141449b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:44.041 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.028+0000 7efc1917b700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7efc14119e90 0x7efc1411b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:44.041 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.028+0000 7efc1917b700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc140742d0 con 0x7efc14119a60 2019-12-05T05:03:44.041 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.028+0000 7efc1917b700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc14118120 con 0x7efc14119e90 2019-12-05T05:03:44.041 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.028+0000 7efc1917b700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc14075510 con 0x7efc14117b00 2019-12-05T05:03:44.041 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.028+0000 7efc12d9d700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7efc14119a60 0x7efc1411abb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:53600/0 (socket says 172.21.15.42:53600) 2019-12-05T05:03:44.043 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.028+0000 7efc12d9d700 1 -- 172.21.15.42:0/774089950 learned_addr learned my addr 172.21.15.42:0/774089950 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:03:44.043 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.028+0000 7efc1359e700 1 --2- 172.21.15.42:0/774089950 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efc14117b00 0x7efc141449b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:44.044 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.028+0000 7efc1897a700 1 --2- 172.21.15.42:0/774089950 >> v2:172.21.15.40:3300/0 conn(0x7efc14119e90 0x7efc1411b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:44.044 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.032+0000 7efc0bfff700 1 -- 172.21.15.42:0/774089950 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3823009411 0 0) 0x7efc140742d0 con 0x7efc14119a60 2019-12-05T05:03:44.044 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.032+0000 7efc0bfff700 1 -- 172.21.15.42:0/774089950 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efbf80030b0 con 0x7efc14119a60 2019-12-05T05:03:44.044 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.032+0000 7efc1897a700 1 -- 172.21.15.42:0/774089950 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efc14117b00 msgr2=0x7efc141449b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:44.044 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.032+0000 7efc1897a700 1 --2- 172.21.15.42:0/774089950 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efc14117b00 0x7efc141449b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:44.044 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.032+0000 7efc1897a700 1 -- 172.21.15.42:0/774089950 >> v1:172.21.15.42:6789/0 conn(0x7efc14119a60 legacy=0x7efc1411abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:44.044 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.032+0000 7efc1897a700 1 -- 172.21.15.42:0/774089950 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc14075700 con 0x7efc14119e90 2019-12-05T05:03:44.045 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.032+0000 7efc1897a700 1 --2- 172.21.15.42:0/774089950 >> v2:172.21.15.40:3300/0 conn(0x7efc14119e90 0x7efc1411b3b0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7efc04008790 tx=0x7efc04008b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:44.045 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.032+0000 7efc1897a700 1 --2- 172.21.15.42:0/774089950 >> v2:172.21.15.40:3300/0 conn(0x7efc14119e90 0x7efc1411b3b0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7efc04008790 tx=0x7efc04008b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:44.045 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.032+0000 7efc1897a700 1 --2- 172.21.15.42:0/774089950 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efc14119e90 0x7efc1411b3b0 secure :-1 s=READY pgs=122 cs=0 l=1 rx=0x7efc04008790 tx=0x7efc04008b60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:44.045 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.032+0000 7efc1897a700 1 --2- 172.21.15.42:0/774089950 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efc14119e90 0x7efc1411b3b0 secure :-1 s=THROTTLE_DONE pgs=122 cs=0 l=1 rx=0x7efc04008790 tx=0x7efc04008b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:44.045 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.032+0000 7efc0bfff700 1 -- 172.21.15.42:0/774089950 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efc0400f470 con 0x7efc14119e90 2019-12-05T05:03:44.045 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.032+0000 7efc1917b700 1 -- 172.21.15.42:0/774089950 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc14075a30 con 0x7efc14119e90 2019-12-05T05:03:44.045 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.032+0000 7efc1917b700 1 -- 172.21.15.42:0/774089950 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc14075f30 con 0x7efc14119e90 2019-12-05T05:03:44.046 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.032+0000 7efc1897a700 1 --2- 172.21.15.42:0/774089950 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efc14119e90 0x7efc1411b3b0 secure :-1 s=THROTTLE_DONE pgs=122 cs=0 l=1 rx=0x7efc04008790 tx=0x7efc04008b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:44.046 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.032+0000 7efc0bfff700 1 -- 172.21.15.42:0/774089950 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc04011730 con 0x7efc14119e90 2019-12-05T05:03:44.046 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.032+0000 7efc1897a700 1 --2- 172.21.15.42:0/774089950 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efc14119e90 0x7efc1411b3b0 secure :-1 s=THROTTLE_DONE pgs=122 cs=0 l=1 rx=0x7efc04008790 tx=0x7efc04008b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:44.046 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.032+0000 7efc0bfff700 1 -- 172.21.15.42:0/774089950 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efc04005730 con 0x7efc14119e90 2019-12-05T05:03:44.046 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.032+0000 7efc1897a700 1 --2- 172.21.15.42:0/774089950 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efc14119e90 0x7efc1411b3b0 secure :-1 s=THROTTLE_DONE pgs=122 cs=0 l=1 rx=0x7efc04008790 tx=0x7efc04008b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:44.046 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.032+0000 7efc0bfff700 1 -- 172.21.15.42:0/774089950 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7efc04007660 con 0x7efc14119e90 2019-12-05T05:03:44.046 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.032+0000 7efc1897a700 1 --2- 172.21.15.42:0/774089950 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efc14119e90 0x7efc1411b3b0 secure :-1 s=THROTTLE_DONE pgs=122 cs=0 l=1 rx=0x7efc04008790 tx=0x7efc04008b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:44.047 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.032+0000 7efc0bfff700 1 --2- 172.21.15.42:0/774089950 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7efbf803b230 0x7efbf803d740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:44.047 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.032+0000 7efc0bfff700 1 -- 172.21.15.42:0/774089950 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6031+0+0 (secure 0 0 0) 0x7efc0404b640 con 0x7efc14119e90 2019-12-05T05:03:44.047 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.036+0000 7efc1359e700 1 --2- 172.21.15.42:0/774089950 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7efbf803b230 0x7efbf803d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:44.047 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.036+0000 7efc1359e700 1 --2- 172.21.15.42:0/774089950 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7efbf803b230 0x7efbf803d740 crc :-1 s=READY pgs=81 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:44.048 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.036+0000 7efc1917b700 1 -- 172.21.15.42:0/774089950 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc141182e0 con 0x7efc14119e90 2019-12-05T05:03:44.051 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.044+0000 7efc1897a700 1 --2- 172.21.15.42:0/774089950 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efc14119e90 0x7efc1411b3b0 secure :-1 s=THROTTLE_DONE pgs=122 cs=0 l=1 rx=0x7efc04008790 tx=0x7efc04008b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:44.051 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.044+0000 7efc0bfff700 1 -- 172.21.15.42:0/774089950 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7efc04008de0 con 0x7efc14119e90 2019-12-05T05:03:44.223 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.216+0000 7efc1917b700 1 -- 172.21.15.42:0/774089950 --> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7efc1411a9c0 con 0x7efbf803b230 2019-12-05T05:03:44.231 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.224+0000 7efc0bfff700 1 -- 172.21.15.42:0/774089950 <== mgr.4108 v2:172.21.15.42:6800/11930 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+80367 (crc 0 0 0) 0x7efc0c00c8c0 con 0x7efbf803b230 2019-12-05T05:03:44.240 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:03:44.240 INFO:teuthology.orchestra.run.smithi042.stderr:dumped all 2019-12-05T05:03:44.242 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.236+0000 7efc1917b700 1 -- 172.21.15.42:0/774089950 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7efbf803b230 msgr2=0x7efbf803d740 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:44.242 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.236+0000 7efc1917b700 1 --2- 172.21.15.42:0/774089950 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7efbf803b230 0x7efbf803d740 crc :-1 s=READY pgs=81 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:44.243 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.236+0000 7efc1917b700 1 -- 172.21.15.42:0/774089950 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efc14119e90 msgr2=0x7efc1411b3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:44.243 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.236+0000 7efc1917b700 1 --2- 172.21.15.42:0/774089950 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efc14119e90 0x7efc1411b3b0 secure :-1 s=READY pgs=122 cs=0 l=1 rx=0x7efc04008790 tx=0x7efc04008b60).stop 2019-12-05T05:03:44.243 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.236+0000 7efc1917b700 1 -- 172.21.15.42:0/774089950 shutdown_connections 2019-12-05T05:03:44.243 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.236+0000 7efc1917b700 1 --2- 172.21.15.42:0/774089950 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7efbf803b230 0x7efbf803d740 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:44.243 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.236+0000 7efc1917b700 1 --2- 172.21.15.42:0/774089950 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efc14119e90 0x7efc1411b3b0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:44.243 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.236+0000 7efc1917b700 1 --2- 172.21.15.42:0/774089950 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efc14117b00 0x7efc141449b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:44.243 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.236+0000 7efc1917b700 1 -- 172.21.15.42:0/774089950 shutdown_connections 2019-12-05T05:03:44.244 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.236+0000 7efc1917b700 1 -- 172.21.15.42:0/774089950 wait complete. 2019-12-05T05:03:44.244 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.236+0000 7efc1917b700 1 -- 172.21.15.42:0/774089950 >> 172.21.15.42:0/774089950 conn(0x7efc140a7f90 msgr2=0x7efc14059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:44.254 INFO:teuthology.orchestra.run.smithi042.stdout:{"pg_ready":true,"pg_map":{"version":35,"stamp":"2019-12-05T05:03:43.598534+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":8390656,"kb_used_data":1984,"kb_used_omap":0,"kb_used_meta":8388608,"kb_avail":746584064,"statfs":{"total":773094113280,"available":764502081536,"internally_reserved":8589934592,"allocated":2031616,"data_stored":767680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8589934592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"3.7","version":"0'0","reported_seq":"12","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T05:03:35.610591+0000","last_change":"2019-12-05T05:03:35.333820+0000","last_active":"2019-12-05T05:03:35.610591+0000","last_peered":"2019-12-05T05:03:35.610591+0000","last_clean":"2019-12-05T05:03:35.610591+0000","last_became_active":"2019-12-05T05:03:35.332717+0000","last_became_peered":"2019-12-05T05:03:35.332717+0000","last_unstale":"2019-12-05T05:03:35.610591+0000","last_undegraded":"2019-12-05T05:03:35.610591+0000","last_fullsized":"2019-12-05T05:03:35.610591+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:34.235090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:34.235090+0000","last_clean_scrub_stamp":"2019-12-05T05:03:34.235090+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7,0],"acting":[3,5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":"16","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T05:03:35.731480+0000","last_change":"2019-12-05T05:03:29.184885+0000","last_active":"2019-12-05T05:03:35.731480+0000","last_peered":"2019-12-05T05:03:35.731480+0000","last_clean":"2019-12-05T05:03:35.731480+0000","last_became_active":"2019-12-05T05:03:29.184431+0000","last_became_peered":"2019-12-05T05:03:29.184431+0000","last_unstale":"2019-12-05T05:03:35.731480+0000","last_undegraded":"2019-12-05T05:03:35.731480+0000","last_fullsized":"2019-12-05T05:03:35.731480+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:28.161818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:28.161818+0000","last_clean_scrub_stamp":"2019-12-05T05:03:28.161818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"13","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T05:03:35.626956+0000","last_change":"2019-12-05T05:03:32.225175+0000","last_active":"2019-12-05T05:03:35.626956+0000","last_peered":"2019-12-05T05:03:35.626956+0000","last_clean":"2019-12-05T05:03:35.626956+0000","last_became_active":"2019-12-05T05:03:32.224827+0000","last_became_peered":"2019-12-05T05:03:32.224827+0000","last_unstale":"2019-12-05T05:03:35.626956+0000","last_undegraded":"2019-12-05T05:03:35.626956+0000","last_fullsized":"2019-12-05T05:03:35.626956+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:31.200243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:31.200243+0000","last_clean_scrub_stamp":"2019-12-05T05:03:31.200243+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":"12","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T05:03:36.529724+0000","last_change":"2019-12-05T05:03:35.333820+0000","last_active":"2019-12-05T05:03:36.529724+0000","last_peered":"2019-12-05T05:03:36.529724+0000","last_clean":"2019-12-05T05:03:36.529724+0000","last_became_active":"2019-12-05T05:03:35.332648+0000","last_became_peered":"2019-12-05T05:03:35.332648+0000","last_unstale":"2019-12-05T05:03:36.529724+0000","last_undegraded":"2019-12-05T05:03:36.529724+0000","last_fullsized":"2019-12-05T05:03:36.529724+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:34.235090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:34.235090+0000","last_clean_scrub_stamp":"2019-12-05T05:03:34.235090+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,7],"acting":[0,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":"16","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T05:03:35.610218+0000","last_change":"2019-12-05T05:03:29.182769+0000","last_active":"2019-12-05T05:03:35.610218+0000","last_peered":"2019-12-05T05:03:35.610218+0000","last_clean":"2019-12-05T05:03:35.610218+0000","last_became_active":"2019-12-05T05:03:29.182409+0000","last_became_peered":"2019-12-05T05:03:29.182409+0000","last_unstale":"2019-12-05T05:03:35.610218+0000","last_undegraded":"2019-12-05T05:03:35.610218+0000","last_fullsized":"2019-12-05T05:03:35.610218+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:28.161818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:28.161818+0000","last_clean_scrub_stamp":"2019-12-05T05:03:28.161818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":"12","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T05:03:35.624389+0000","last_change":"2019-12-05T05:03:32.228176+0000","last_active":"2019-12-05T05:03:35.624389+0000","last_peered":"2019-12-05T05:03:35.624389+0000","last_clean":"2019-12-05T05:03:35.624389+0000","last_became_active":"2019-12-05T05:03:32.227690+0000","last_became_peered":"2019-12-05T05:03:32.227690+0000","last_unstale":"2019-12-05T05:03:35.624389+0000","last_undegraded":"2019-12-05T05:03:35.624389+0000","last_fullsized":"2019-12-05T05:03:35.624389+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:31.200243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:31.200243+0000","last_clean_scrub_stamp":"2019-12-05T05:03:31.200243+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":"12","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T05:03:36.529831+0000","last_change":"2019-12-05T05:03:35.267212+0000","last_active":"2019-12-05T05:03:36.529831+0000","last_peered":"2019-12-05T05:03:36.529831+0000","last_clean":"2019-12-05T05:03:36.529831+0000","last_became_active":"2019-12-05T05:03:35.266235+0000","last_became_peered":"2019-12-05T05:03:35.266235+0000","last_unstale":"2019-12-05T05:03:36.529831+0000","last_undegraded":"2019-12-05T05:03:36.529831+0000","last_fullsized":"2019-12-05T05:03:36.529831+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:34.235090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:34.235090+0000","last_clean_scrub_stamp":"2019-12-05T05:03:34.235090+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3,5],"acting":[0,4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":"16","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T05:03:35.626677+0000","last_change":"2019-12-05T05:03:29.181115+0000","last_active":"2019-12-05T05:03:35.626677+0000","last_peered":"2019-12-05T05:03:35.626677+0000","last_clean":"2019-12-05T05:03:35.626677+0000","last_became_active":"2019-12-05T05:03:29.180832+0000","last_became_peered":"2019-12-05T05:03:29.180832+0000","last_unstale":"2019-12-05T05:03:35.626677+0000","last_undegraded":"2019-12-05T05:03:35.626677+0000","last_fullsized":"2019-12-05T05:03:35.626677+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:28.161818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:28.161818+0000","last_clean_scrub_stamp":"2019-12-05T05:03:28.161818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"13","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T05:03:36.847411+0000","last_change":"2019-12-05T05:03:32.227259+0000","last_active":"2019-12-05T05:03:36.847411+0000","last_peered":"2019-12-05T05:03:36.847411+0000","last_clean":"2019-12-05T05:03:36.847411+0000","last_became_active":"2019-12-05T05:03:32.226895+0000","last_became_peered":"2019-12-05T05:03:32.226895+0000","last_unstale":"2019-12-05T05:03:36.847411+0000","last_undegraded":"2019-12-05T05:03:36.847411+0000","last_fullsized":"2019-12-05T05:03:36.847411+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:31.200243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:31.200243+0000","last_clean_scrub_stamp":"2019-12-05T05:03:31.200243+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":"12","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T05:03:35.628786+0000","last_change":"2019-12-05T05:03:35.628207+0000","last_active":"2019-12-05T05:03:35.628786+0000","last_peered":"2019-12-05T05:03:35.628786+0000","last_clean":"2019-12-05T05:03:35.628786+0000","last_became_active":"2019-12-05T05:03:35.627352+0000","last_became_peered":"2019-12-05T05:03:35.627352+0000","last_unstale":"2019-12-05T05:03:35.628786+0000","last_undegraded":"2019-12-05T05:03:35.628786+0000","last_fullsized":"2019-12-05T05:03:35.628786+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:34.235090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:34.235090+0000","last_clean_scrub_stamp":"2019-12-05T05:03:34.235090+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6,5],"acting":[1,2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":"16","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T05:03:36.847322+0000","last_change":"2019-12-05T05:03:29.408129+0000","last_active":"2019-12-05T05:03:36.847322+0000","last_peered":"2019-12-05T05:03:36.847322+0000","last_clean":"2019-12-05T05:03:36.847322+0000","last_became_active":"2019-12-05T05:03:29.407108+0000","last_became_peered":"2019-12-05T05:03:29.407108+0000","last_unstale":"2019-12-05T05:03:36.847322+0000","last_undegraded":"2019-12-05T05:03:36.847322+0000","last_fullsized":"2019-12-05T05:03:36.847322+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:28.161818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:28.161818+0000","last_clean_scrub_stamp":"2019-12-05T05:03:28.161818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"13","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T05:03:35.610338+0000","last_change":"2019-12-05T05:03:32.636592+0000","last_active":"2019-12-05T05:03:35.610338+0000","last_peered":"2019-12-05T05:03:35.610338+0000","last_clean":"2019-12-05T05:03:35.610338+0000","last_became_active":"2019-12-05T05:03:32.635937+0000","last_became_peered":"2019-12-05T05:03:32.635937+0000","last_unstale":"2019-12-05T05:03:35.610338+0000","last_undegraded":"2019-12-05T05:03:35.610338+0000","last_fullsized":"2019-12-05T05:03:35.610338+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:31.200243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:31.200243+0000","last_clean_scrub_stamp":"2019-12-05T05:03:31.200243+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":"12","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T05:03:36.315636+0000","last_change":"2019-12-05T05:03:36.315160+0000","last_active":"2019-12-05T05:03:36.315636+0000","last_peered":"2019-12-05T05:03:36.315636+0000","last_clean":"2019-12-05T05:03:36.315636+0000","last_became_active":"2019-12-05T05:03:36.314006+0000","last_became_peered":"2019-12-05T05:03:36.314006+0000","last_unstale":"2019-12-05T05:03:36.315636+0000","last_undegraded":"2019-12-05T05:03:36.315636+0000","last_fullsized":"2019-12-05T05:03:36.315636+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:34.235090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:34.235090+0000","last_clean_scrub_stamp":"2019-12-05T05:03:34.235090+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6,4],"acting":[3,5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"16","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T05:03:36.847644+0000","last_change":"2019-12-05T05:03:29.407882+0000","last_active":"2019-12-05T05:03:36.847644+0000","last_peered":"2019-12-05T05:03:36.847644+0000","last_clean":"2019-12-05T05:03:36.847644+0000","last_became_active":"2019-12-05T05:03:29.406770+0000","last_became_peered":"2019-12-05T05:03:29.406770+0000","last_unstale":"2019-12-05T05:03:36.847644+0000","last_undegraded":"2019-12-05T05:03:36.847644+0000","last_fullsized":"2019-12-05T05:03:36.847644+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:28.161818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:28.161818+0000","last_clean_scrub_stamp":"2019-12-05T05:03:28.161818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"13","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T05:03:35.629132+0000","last_change":"2019-12-05T05:03:32.227654+0000","last_active":"2019-12-05T05:03:35.629132+0000","last_peered":"2019-12-05T05:03:35.629132+0000","last_clean":"2019-12-05T05:03:35.629132+0000","last_became_active":"2019-12-05T05:03:32.227052+0000","last_became_peered":"2019-12-05T05:03:32.227052+0000","last_unstale":"2019-12-05T05:03:35.629132+0000","last_undegraded":"2019-12-05T05:03:35.629132+0000","last_fullsized":"2019-12-05T05:03:35.629132+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:31.200243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:31.200243+0000","last_clean_scrub_stamp":"2019-12-05T05:03:31.200243+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":"12","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T05:03:35.734103+0000","last_change":"2019-12-05T05:03:35.733650+0000","last_active":"2019-12-05T05:03:35.734103+0000","last_peered":"2019-12-05T05:03:35.734103+0000","last_clean":"2019-12-05T05:03:35.734103+0000","last_became_active":"2019-12-05T05:03:35.732405+0000","last_became_peered":"2019-12-05T05:03:35.732405+0000","last_unstale":"2019-12-05T05:03:35.734103+0000","last_undegraded":"2019-12-05T05:03:35.734103+0000","last_fullsized":"2019-12-05T05:03:35.734103+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:34.235090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:34.235090+0000","last_clean_scrub_stamp":"2019-12-05T05:03:34.235090+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0,6],"acting":[4,5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":"16","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T05:03:35.629127+0000","last_change":"2019-12-05T05:03:29.407539+0000","last_active":"2019-12-05T05:03:35.629127+0000","last_peered":"2019-12-05T05:03:35.629127+0000","last_clean":"2019-12-05T05:03:35.629127+0000","last_became_active":"2019-12-05T05:03:29.406903+0000","last_became_peered":"2019-12-05T05:03:29.406903+0000","last_unstale":"2019-12-05T05:03:35.629127+0000","last_undegraded":"2019-12-05T05:03:35.629127+0000","last_fullsized":"2019-12-05T05:03:35.629127+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:28.161818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:28.161818+0000","last_clean_scrub_stamp":"2019-12-05T05:03:28.161818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"13","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T05:03:35.629133+0000","last_change":"2019-12-05T05:03:32.227838+0000","last_active":"2019-12-05T05:03:35.629133+0000","last_peered":"2019-12-05T05:03:35.629133+0000","last_clean":"2019-12-05T05:03:35.629133+0000","last_became_active":"2019-12-05T05:03:32.227050+0000","last_became_peered":"2019-12-05T05:03:32.227050+0000","last_unstale":"2019-12-05T05:03:35.629133+0000","last_undegraded":"2019-12-05T05:03:35.629133+0000","last_fullsized":"2019-12-05T05:03:35.629133+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:31.200243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:31.200243+0000","last_clean_scrub_stamp":"2019-12-05T05:03:31.200243+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":"12","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T05:03:35.626774+0000","last_change":"2019-12-05T05:03:35.267419+0000","last_active":"2019-12-05T05:03:35.626774+0000","last_peered":"2019-12-05T05:03:35.626774+0000","last_clean":"2019-12-05T05:03:35.626774+0000","last_became_active":"2019-12-05T05:03:35.266868+0000","last_became_peered":"2019-12-05T05:03:35.266868+0000","last_unstale":"2019-12-05T05:03:35.626774+0000","last_undegraded":"2019-12-05T05:03:35.626774+0000","last_fullsized":"2019-12-05T05:03:35.626774+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:34.235090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:34.235090+0000","last_clean_scrub_stamp":"2019-12-05T05:03:34.235090+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0,5],"acting":[1,2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":"15","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T05:03:35.624631+0000","last_change":"2019-12-05T05:03:29.407541+0000","last_active":"2019-12-05T05:03:35.624631+0000","last_peered":"2019-12-05T05:03:35.624631+0000","last_clean":"2019-12-05T05:03:35.624631+0000","last_became_active":"2019-12-05T05:03:29.406857+0000","last_became_peered":"2019-12-05T05:03:29.406857+0000","last_unstale":"2019-12-05T05:03:35.624631+0000","last_undegraded":"2019-12-05T05:03:35.624631+0000","last_fullsized":"2019-12-05T05:03:35.624631+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:28.161818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:28.161818+0000","last_clean_scrub_stamp":"2019-12-05T05:03:28.161818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"13","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T05:03:36.847262+0000","last_change":"2019-12-05T05:03:32.480464+0000","last_active":"2019-12-05T05:03:36.847262+0000","last_peered":"2019-12-05T05:03:36.847262+0000","last_clean":"2019-12-05T05:03:36.847262+0000","last_became_active":"2019-12-05T05:03:32.479565+0000","last_became_peered":"2019-12-05T05:03:32.479565+0000","last_unstale":"2019-12-05T05:03:36.847262+0000","last_undegraded":"2019-12-05T05:03:36.847262+0000","last_fullsized":"2019-12-05T05:03:36.847262+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:31.200243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:31.200243+0000","last_clean_scrub_stamp":"2019-12-05T05:03:31.200243+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":"12","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T05:03:35.630204+0000","last_change":"2019-12-05T05:03:35.629791+0000","last_active":"2019-12-05T05:03:35.630204+0000","last_peered":"2019-12-05T05:03:35.630204+0000","last_clean":"2019-12-05T05:03:35.630204+0000","last_became_active":"2019-12-05T05:03:35.629300+0000","last_became_peered":"2019-12-05T05:03:35.629300+0000","last_unstale":"2019-12-05T05:03:35.630204+0000","last_undegraded":"2019-12-05T05:03:35.630204+0000","last_fullsized":"2019-12-05T05:03:35.630204+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:34.235090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:34.235090+0000","last_clean_scrub_stamp":"2019-12-05T05:03:34.235090+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2,6],"acting":[5,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":"15","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T05:03:35.624610+0000","last_change":"2019-12-05T05:03:29.185245+0000","last_active":"2019-12-05T05:03:35.624610+0000","last_peered":"2019-12-05T05:03:35.624610+0000","last_clean":"2019-12-05T05:03:35.624610+0000","last_became_active":"2019-12-05T05:03:29.184716+0000","last_became_peered":"2019-12-05T05:03:29.184716+0000","last_unstale":"2019-12-05T05:03:35.624610+0000","last_undegraded":"2019-12-05T05:03:35.624610+0000","last_fullsized":"2019-12-05T05:03:35.624610+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:28.161818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:28.161818+0000","last_clean_scrub_stamp":"2019-12-05T05:03:28.161818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"13","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T05:03:35.626773+0000","last_change":"2019-12-05T05:03:32.476846+0000","last_active":"2019-12-05T05:03:35.626773+0000","last_peered":"2019-12-05T05:03:35.626773+0000","last_clean":"2019-12-05T05:03:35.626773+0000","last_became_active":"2019-12-05T05:03:32.476140+0000","last_became_peered":"2019-12-05T05:03:32.476140+0000","last_unstale":"2019-12-05T05:03:35.626773+0000","last_undegraded":"2019-12-05T05:03:35.626773+0000","last_fullsized":"2019-12-05T05:03:35.626773+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:31.200243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:31.200243+0000","last_clean_scrub_stamp":"2019-12-05T05:03:31.200243+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":32,"acting":32,"num_store_stats":8},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574854,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048832,"kb_used_data":248,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323008,"statfs":{"total":96636764160,"available":95562760192,"internally_reserved":1073741824,"allocated":253952,"data_stored":95960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.792}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.045}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.900}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.190}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.855}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.244}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.335}]}]},{"osd":6,"up_from":13,"seq":55834574855,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048832,"kb_used_data":248,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323008,"statfs":{"total":96636764160,"available":95562760192,"internally_reserved":1073741824,"allocated":253952,"data_stored":95960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.632}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.811}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.255}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.346}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.920}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.396}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.604}]}]},{"osd":1,"up_from":13,"seq":55834574853,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048832,"kb_used_data":248,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323008,"statfs":{"total":96636764160,"available":95562760192,"internally_reserved":1073741824,"allocated":253952,"data_stored":95960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.613}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.922}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.490}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.054}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.753}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.826}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.790}]}]},{"osd":0,"up_from":13,"seq":55834574855,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048832,"kb_used_data":248,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323008,"statfs":{"total":96636764160,"available":95562760192,"internally_reserved":1073741824,"allocated":253952,"data_stored":95960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.693}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.790}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.317}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.095}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.174}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.542}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.500}]}]},{"osd":2,"up_from":13,"seq":55834574853,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048832,"kb_used_data":248,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323008,"statfs":{"total":96636764160,"available":95562760192,"internally_reserved":1073741824,"allocated":253952,"data_stored":95960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.979}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.842}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.871}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.483}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.376}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.273}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.439}]}]},{"osd":3,"up_from":13,"seq":55834574854,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048832,"kb_used_data":248,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323008,"statfs":{"total":96636764160,"available":95562760192,"internally_reserved":1073741824,"allocated":253952,"data_stored":95960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.838}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.555}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.812}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.336}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.374}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.300}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.264}]}]},{"osd":4,"up_from":13,"seq":55834574853,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048832,"kb_used_data":248,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323008,"statfs":{"total":96636764160,"available":95562760192,"internally_reserved":1073741824,"allocated":253952,"data_stored":95960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.098}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.865}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.749}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.633}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.264}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.566}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.354}]}]},{"osd":5,"up_from":13,"seq":55834574854,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048832,"kb_used_data":248,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323008,"statfs":{"total":96636764160,"available":95562760192,"internally_reserved":1073741824,"allocated":253952,"data_stored":95960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.721}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.964}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.894}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.105}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.376}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.323}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.249}]}]}]}} 2019-12-05T05:03:44.258 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2019-12-05T05:03:44.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.344+0000 7f97937a0700 1 Processor -- start 2019-12-05T05:03:44.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.344+0000 7f97937a0700 1 -- start start 2019-12-05T05:03:44.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.344+0000 7f97937a0700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f978c0a7740 0x7f978c08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:44.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.344+0000 7f97937a0700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f978c07e720 0x7f978c07eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:44.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.344+0000 7f97937a0700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f978c0a0710 con 0x7f978c09f2e0 2019-12-05T05:03:44.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.344+0000 7f97937a0700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f978c0a0b00 con 0x7f978c0a7740 2019-12-05T05:03:44.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.344+0000 7f97937a0700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f978c08e5b0 con 0x7f978c07e720 2019-12-05T05:03:44.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.344+0000 7f979153c700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f978c07e720 0x7f978c07eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:44.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.344+0000 7f979153c700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f978c07e720 0x7f978c07eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:54038/0 (socket says 172.21.15.42:54038) 2019-12-05T05:03:44.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.344+0000 7f9791d3d700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f978c0a7740 0x7f978c08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:44.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.344+0000 7f979153c700 1 -- 172.21.15.42:0/367269298 learned_addr learned my addr 172.21.15.42:0/367269298 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:03:44.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.344+0000 7f979153c700 1 -- 172.21.15.42:0/367269298 >> v2:172.21.15.40:3300/0 conn(0x7f978c0a7740 msgr2=0x7f978c08df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:03:44.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.344+0000 7f979153c700 1 --2- 172.21.15.42:0/367269298 >> v2:172.21.15.40:3300/0 conn(0x7f978c0a7740 0x7f978c08df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:03:44.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.344+0000 7f979153c700 1 -- 172.21.15.42:0/367269298 >> v1:172.21.15.40:6789/0 conn(0x7f978c09f2e0 legacy=0x7f978c09f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:44.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.344+0000 7f9790d3b700 1 -- 172.21.15.42:0/367269298 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2460727032 0 0) 0x7f978c0a0710 con 0x7f978c09f2e0 2019-12-05T05:03:44.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.344+0000 7f979153c700 1 -- 172.21.15.42:0/367269298 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f978c07f0a0 con 0x7f978c07e720 2019-12-05T05:03:44.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.344+0000 7f979153c700 1 --2- 172.21.15.42:0/367269298 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f978c07e720 0x7f978c07eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f977c008790 tx=0x7f977c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:44.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.348+0000 7f979153c700 1 --2- 172.21.15.42:0/367269298 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f978c07e720 0x7f978c07eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f977c008790 tx=0x7f977c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:44.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.348+0000 7f979153c700 1 --2- 172.21.15.42:0/367269298 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f978c07e720 0x7f978c07eb50 secure :-1 s=READY pgs=39 cs=0 l=1 rx=0x7f977c008790 tx=0x7f977c00fd30).ready entity=mon.2 client_cookie=dba09e0c2d5186e3 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:44.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.348+0000 7f979153c700 1 --2- 172.21.15.42:0/367269298 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f978c07e720 0x7f978c07eb50 secure :-1 s=THROTTLE_DONE pgs=39 cs=0 l=1 rx=0x7f977c008790 tx=0x7f977c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:44.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.348+0000 7f9790d3b700 1 -- 172.21.15.42:0/367269298 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f977c0113b0 con 0x7f978c07e720 2019-12-05T05:03:44.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.348+0000 7f979153c700 1 --2- 172.21.15.42:0/367269298 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f978c07e720 0x7f978c07eb50 secure :-1 s=THROTTLE_DONE pgs=39 cs=0 l=1 rx=0x7f977c008790 tx=0x7f977c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:44.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.348+0000 7f979153c700 1 --2- 172.21.15.42:0/367269298 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f978c07e720 0x7f978c07eb50 secure :-1 s=THROTTLE_DONE pgs=39 cs=0 l=1 rx=0x7f977c008790 tx=0x7f977c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:44.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.348+0000 7f9790d3b700 1 -- 172.21.15.42:0/367269298 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f977c008bb0 con 0x7f978c07e720 2019-12-05T05:03:44.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.348+0000 7f9790d3b700 1 -- 172.21.15.42:0/367269298 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f977c011990 con 0x7f978c07e720 2019-12-05T05:03:44.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.348+0000 7f97937a0700 1 -- 172.21.15.42:0/367269298 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f978c07e720 msgr2=0x7f978c07eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:44.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.348+0000 7f97937a0700 1 --2- 172.21.15.42:0/367269298 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f978c07e720 0x7f978c07eb50 secure :-1 s=READY pgs=39 cs=0 l=1 rx=0x7f977c008790 tx=0x7f977c00fd30).stop 2019-12-05T05:03:44.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.348+0000 7f97937a0700 1 -- 172.21.15.42:0/367269298 shutdown_connections 2019-12-05T05:03:44.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.348+0000 7f97937a0700 1 --2- 172.21.15.42:0/367269298 >> v2:172.21.15.40:3300/0 conn(0x7f978c0a7740 0x7f978c08df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:03:44.358 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.348+0000 7f97937a0700 1 --2- 172.21.15.42:0/367269298 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f978c07e720 0x7f978c07eb50 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:44.358 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.348+0000 7f97937a0700 1 -- 172.21.15.42:0/367269298 shutdown_connections 2019-12-05T05:03:44.358 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.348+0000 7f97937a0700 1 -- 172.21.15.42:0/367269298 wait complete. 2019-12-05T05:03:44.358 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.348+0000 7f97937a0700 1 -- 172.21.15.42:0/367269298 >> 172.21.15.42:0/367269298 conn(0x7f978c0aabe0 msgr2=0x7f978c0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:44.361 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.348+0000 7f97937a0700 1 Processor -- start 2019-12-05T05:03:44.361 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.348+0000 7f97937a0700 1 -- start start 2019-12-05T05:03:44.362 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.352+0000 7f97937a0700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f978c11aaf0 0x7f978c123f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:44.362 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.352+0000 7f97937a0700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f978c135650 0x7f978c137a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:44.362 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.352+0000 7f97937a0700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f978c1244c0 con 0x7f978c130210 2019-12-05T05:03:44.362 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.352+0000 7f97937a0700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f978c11b030 con 0x7f978c135650 2019-12-05T05:03:44.362 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.352+0000 7f97937a0700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f978c11b2d0 con 0x7f978c11aaf0 2019-12-05T05:03:44.362 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.352+0000 7f9791d3d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f978c11aaf0 0x7f978c123f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:44.362 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.352+0000 7f9791d3d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f978c11aaf0 0x7f978c123f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:53574/0 (socket says 172.21.15.42:53574) 2019-12-05T05:03:44.363 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.352+0000 7f9791d3d700 1 -- 172.21.15.42:0/3680790717 learned_addr learned my addr 172.21.15.42:0/3680790717 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:03:44.363 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.352+0000 7f979153c700 1 --1- 172.21.15.42:0/3680790717 >> v1:172.21.15.42:6789/0 conn(0x7f978c130210 0x7f978c130640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:53614/0 (socket says 172.21.15.42:53614) 2019-12-05T05:03:44.363 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.352+0000 7f979253e700 1 --2- 172.21.15.42:0/3680790717 >> v2:172.21.15.40:3300/0 conn(0x7f978c135650 0x7f978c137a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:44.363 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.352+0000 7f9791d3d700 1 -- 172.21.15.42:0/3680790717 >> v2:172.21.15.40:3300/0 conn(0x7f978c135650 msgr2=0x7f978c137a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:44.363 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.352+0000 7f9791d3d700 1 --2- 172.21.15.42:0/3680790717 >> v2:172.21.15.40:3300/0 conn(0x7f978c135650 0x7f978c137a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:44.363 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.352+0000 7f9791d3d700 1 -- 172.21.15.42:0/3680790717 >> v1:172.21.15.42:6789/0 conn(0x7f978c130210 legacy=0x7f978c130640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:44.363 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.352+0000 7f9791d3d700 1 -- 172.21.15.42:0/3680790717 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f978c142530 con 0x7f978c11aaf0 2019-12-05T05:03:44.364 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.352+0000 7f9791d3d700 1 --2- 172.21.15.42:0/3680790717 >> v2:172.21.15.42:3300/0 conn(0x7f978c11aaf0 0x7f978c123f70 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f978c11af50 tx=0x7f9788010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:44.364 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.352+0000 7f9791d3d700 1 --2- 172.21.15.42:0/3680790717 >> v2:172.21.15.42:3300/0 conn(0x7f978c11aaf0 0x7f978c123f70 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f978c11af50 tx=0x7f9788010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:44.364 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.352+0000 7f9791d3d700 1 --2- 172.21.15.42:0/3680790717 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f978c11aaf0 0x7f978c123f70 secure :-1 s=READY pgs=145 cs=0 l=1 rx=0x7f978c11af50 tx=0x7f9788010b60).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:44.364 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.352+0000 7f9791d3d700 1 --2- 172.21.15.42:0/3680790717 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f978c11aaf0 0x7f978c123f70 secure :-1 s=THROTTLE_DONE pgs=145 cs=0 l=1 rx=0x7f978c11af50 tx=0x7f9788010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:44.364 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.352+0000 7f97827fc700 1 -- 172.21.15.42:0/3680790717 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9788012590 con 0x7f978c11aaf0 2019-12-05T05:03:44.364 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.352+0000 7f9791d3d700 1 --2- 172.21.15.42:0/3680790717 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f978c11aaf0 0x7f978c123f70 secure :-1 s=THROTTLE_DONE pgs=145 cs=0 l=1 rx=0x7f978c11af50 tx=0x7f9788010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:44.364 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.352+0000 7f9791d3d700 1 --2- 172.21.15.42:0/3680790717 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f978c11aaf0 0x7f978c123f70 secure :-1 s=THROTTLE_DONE pgs=145 cs=0 l=1 rx=0x7f978c11af50 tx=0x7f9788010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:44.365 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.352+0000 7f97827fc700 1 -- 172.21.15.42:0/3680790717 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9788007150 con 0x7f978c11aaf0 2019-12-05T05:03:44.365 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.352+0000 7f97937a0700 1 -- 172.21.15.42:0/3680790717 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f978c11b9a0 con 0x7f978c11aaf0 2019-12-05T05:03:44.365 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.352+0000 7f97827fc700 1 -- 172.21.15.42:0/3680790717 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9788012b70 con 0x7f978c11aaf0 2019-12-05T05:03:44.365 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.352+0000 7f97937a0700 1 -- 172.21.15.42:0/3680790717 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f978c07df90 con 0x7f978c11aaf0 2019-12-05T05:03:44.365 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.352+0000 7f9791d3d700 1 --2- 172.21.15.42:0/3680790717 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f978c11aaf0 0x7f978c123f70 secure :-1 s=THROTTLE_DONE pgs=145 cs=0 l=1 rx=0x7f978c11af50 tx=0x7f9788010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:44.365 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.356+0000 7f97827fc700 1 -- 172.21.15.42:0/3680790717 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f9788005540 con 0x7f978c11aaf0 2019-12-05T05:03:44.365 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.356+0000 7f9791d3d700 1 --2- 172.21.15.42:0/3680790717 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f978c11aaf0 0x7f978c123f70 secure :-1 s=THROTTLE_DONE pgs=145 cs=0 l=1 rx=0x7f978c11af50 tx=0x7f9788010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:44.366 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.356+0000 7f97827fc700 1 --2- 172.21.15.42:0/3680790717 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f977403adb0 0x7f977403d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:44.366 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.356+0000 7f97827fc700 1 -- 172.21.15.42:0/3680790717 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f978804af80 con 0x7f978c11aaf0 2019-12-05T05:03:44.366 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.356+0000 7f979153c700 1 --2- 172.21.15.42:0/3680790717 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f977403adb0 0x7f977403d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:44.366 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.356+0000 7f979153c700 1 --2- 172.21.15.42:0/3680790717 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f977403adb0 0x7f977403d2c0 crc :-1 s=READY pgs=82 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:44.366 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.360+0000 7f97937a0700 1 -- 172.21.15.42:0/3680790717 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f978c05d110 con 0x7f978c11aaf0 2019-12-05T05:03:44.370 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.364+0000 7f9791d3d700 1 --2- 172.21.15.42:0/3680790717 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f978c11aaf0 0x7f978c123f70 secure :-1 s=THROTTLE_DONE pgs=145 cs=0 l=1 rx=0x7f978c11af50 tx=0x7f9788010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:44.370 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.364+0000 7f97827fc700 1 -- 172.21.15.42:0/3680790717 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f9788032000 con 0x7f978c11aaf0 2019-12-05T05:03:44.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.532+0000 7f97937a0700 1 -- 172.21.15.42:0/3680790717 --> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f978c0a9ed0 con 0x7f977403adb0 2019-12-05T05:03:44.545 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.540+0000 7f97827fc700 1 -- 172.21.15.42:0/3680790717 <== mgr.4108 v2:172.21.15.42:6800/11930 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+80367 (crc 0 0 0) 0x7f978c0a9ed0 con 0x7f977403adb0 2019-12-05T05:03:44.546 INFO:teuthology.orchestra.run.smithi042.stderr:dumped all 2019-12-05T05:03:44.546 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:03:44.549 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.544+0000 7f97937a0700 1 -- 172.21.15.42:0/3680790717 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f977403adb0 msgr2=0x7f977403d2c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:44.549 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.544+0000 7f97937a0700 1 --2- 172.21.15.42:0/3680790717 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f977403adb0 0x7f977403d2c0 crc :-1 s=READY pgs=82 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:44.549 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.544+0000 7f97937a0700 1 -- 172.21.15.42:0/3680790717 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f978c11aaf0 msgr2=0x7f978c123f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:44.549 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.544+0000 7f97937a0700 1 --2- 172.21.15.42:0/3680790717 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f978c11aaf0 0x7f978c123f70 secure :-1 s=READY pgs=145 cs=0 l=1 rx=0x7f978c11af50 tx=0x7f9788010b60).stop 2019-12-05T05:03:44.549 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.544+0000 7f97937a0700 1 -- 172.21.15.42:0/3680790717 shutdown_connections 2019-12-05T05:03:44.549 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.544+0000 7f97937a0700 1 --2- 172.21.15.42:0/3680790717 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f977403adb0 0x7f977403d2c0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:44.549 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.544+0000 7f97937a0700 1 --2- 172.21.15.42:0/3680790717 >> v2:172.21.15.40:3300/0 conn(0x7f978c135650 0x7f978c137a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:44.550 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.544+0000 7f97937a0700 1 --2- 172.21.15.42:0/3680790717 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f978c11aaf0 0x7f978c123f70 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:44.550 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.544+0000 7f97937a0700 1 -- 172.21.15.42:0/3680790717 shutdown_connections 2019-12-05T05:03:44.550 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.544+0000 7f97937a0700 1 -- 172.21.15.42:0/3680790717 wait complete. 2019-12-05T05:03:44.550 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.544+0000 7f97937a0700 1 -- 172.21.15.42:0/3680790717 >> 172.21.15.42:0/3680790717 conn(0x7f978c0a96e0 msgr2=0x7f978c0a7f90 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:44.560 INFO:teuthology.orchestra.run.smithi042.stdout:{"pg_ready":true,"pg_map":{"version":35,"stamp":"2019-12-05T05:03:43.598534+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":8390656,"kb_used_data":1984,"kb_used_omap":0,"kb_used_meta":8388608,"kb_avail":746584064,"statfs":{"total":773094113280,"available":764502081536,"internally_reserved":8589934592,"allocated":2031616,"data_stored":767680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8589934592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"3.7","version":"0'0","reported_seq":"12","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T05:03:35.610591+0000","last_change":"2019-12-05T05:03:35.333820+0000","last_active":"2019-12-05T05:03:35.610591+0000","last_peered":"2019-12-05T05:03:35.610591+0000","last_clean":"2019-12-05T05:03:35.610591+0000","last_became_active":"2019-12-05T05:03:35.332717+0000","last_became_peered":"2019-12-05T05:03:35.332717+0000","last_unstale":"2019-12-05T05:03:35.610591+0000","last_undegraded":"2019-12-05T05:03:35.610591+0000","last_fullsized":"2019-12-05T05:03:35.610591+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:34.235090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:34.235090+0000","last_clean_scrub_stamp":"2019-12-05T05:03:34.235090+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7,0],"acting":[3,5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":"16","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T05:03:35.731480+0000","last_change":"2019-12-05T05:03:29.184885+0000","last_active":"2019-12-05T05:03:35.731480+0000","last_peered":"2019-12-05T05:03:35.731480+0000","last_clean":"2019-12-05T05:03:35.731480+0000","last_became_active":"2019-12-05T05:03:29.184431+0000","last_became_peered":"2019-12-05T05:03:29.184431+0000","last_unstale":"2019-12-05T05:03:35.731480+0000","last_undegraded":"2019-12-05T05:03:35.731480+0000","last_fullsized":"2019-12-05T05:03:35.731480+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:28.161818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:28.161818+0000","last_clean_scrub_stamp":"2019-12-05T05:03:28.161818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"13","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T05:03:35.626956+0000","last_change":"2019-12-05T05:03:32.225175+0000","last_active":"2019-12-05T05:03:35.626956+0000","last_peered":"2019-12-05T05:03:35.626956+0000","last_clean":"2019-12-05T05:03:35.626956+0000","last_became_active":"2019-12-05T05:03:32.224827+0000","last_became_peered":"2019-12-05T05:03:32.224827+0000","last_unstale":"2019-12-05T05:03:35.626956+0000","last_undegraded":"2019-12-05T05:03:35.626956+0000","last_fullsized":"2019-12-05T05:03:35.626956+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:31.200243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:31.200243+0000","last_clean_scrub_stamp":"2019-12-05T05:03:31.200243+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":"12","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T05:03:36.529724+0000","last_change":"2019-12-05T05:03:35.333820+0000","last_active":"2019-12-05T05:03:36.529724+0000","last_peered":"2019-12-05T05:03:36.529724+0000","last_clean":"2019-12-05T05:03:36.529724+0000","last_became_active":"2019-12-05T05:03:35.332648+0000","last_became_peered":"2019-12-05T05:03:35.332648+0000","last_unstale":"2019-12-05T05:03:36.529724+0000","last_undegraded":"2019-12-05T05:03:36.529724+0000","last_fullsized":"2019-12-05T05:03:36.529724+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:34.235090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:34.235090+0000","last_clean_scrub_stamp":"2019-12-05T05:03:34.235090+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,7],"acting":[0,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":"16","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T05:03:35.610218+0000","last_change":"2019-12-05T05:03:29.182769+0000","last_active":"2019-12-05T05:03:35.610218+0000","last_peered":"2019-12-05T05:03:35.610218+0000","last_clean":"2019-12-05T05:03:35.610218+0000","last_became_active":"2019-12-05T05:03:29.182409+0000","last_became_peered":"2019-12-05T05:03:29.182409+0000","last_unstale":"2019-12-05T05:03:35.610218+0000","last_undegraded":"2019-12-05T05:03:35.610218+0000","last_fullsized":"2019-12-05T05:03:35.610218+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:28.161818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:28.161818+0000","last_clean_scrub_stamp":"2019-12-05T05:03:28.161818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":"12","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T05:03:35.624389+0000","last_change":"2019-12-05T05:03:32.228176+0000","last_active":"2019-12-05T05:03:35.624389+0000","last_peered":"2019-12-05T05:03:35.624389+0000","last_clean":"2019-12-05T05:03:35.624389+0000","last_became_active":"2019-12-05T05:03:32.227690+0000","last_became_peered":"2019-12-05T05:03:32.227690+0000","last_unstale":"2019-12-05T05:03:35.624389+0000","last_undegraded":"2019-12-05T05:03:35.624389+0000","last_fullsized":"2019-12-05T05:03:35.624389+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:31.200243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:31.200243+0000","last_clean_scrub_stamp":"2019-12-05T05:03:31.200243+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":"12","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T05:03:36.529831+0000","last_change":"2019-12-05T05:03:35.267212+0000","last_active":"2019-12-05T05:03:36.529831+0000","last_peered":"2019-12-05T05:03:36.529831+0000","last_clean":"2019-12-05T05:03:36.529831+0000","last_became_active":"2019-12-05T05:03:35.266235+0000","last_became_peered":"2019-12-05T05:03:35.266235+0000","last_unstale":"2019-12-05T05:03:36.529831+0000","last_undegraded":"2019-12-05T05:03:36.529831+0000","last_fullsized":"2019-12-05T05:03:36.529831+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:34.235090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:34.235090+0000","last_clean_scrub_stamp":"2019-12-05T05:03:34.235090+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3,5],"acting":[0,4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":"16","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T05:03:35.626677+0000","last_change":"2019-12-05T05:03:29.181115+0000","last_active":"2019-12-05T05:03:35.626677+0000","last_peered":"2019-12-05T05:03:35.626677+0000","last_clean":"2019-12-05T05:03:35.626677+0000","last_became_active":"2019-12-05T05:03:29.180832+0000","last_became_peered":"2019-12-05T05:03:29.180832+0000","last_unstale":"2019-12-05T05:03:35.626677+0000","last_undegraded":"2019-12-05T05:03:35.626677+0000","last_fullsized":"2019-12-05T05:03:35.626677+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:28.161818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:28.161818+0000","last_clean_scrub_stamp":"2019-12-05T05:03:28.161818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"13","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T05:03:36.847411+0000","last_change":"2019-12-05T05:03:32.227259+0000","last_active":"2019-12-05T05:03:36.847411+0000","last_peered":"2019-12-05T05:03:36.847411+0000","last_clean":"2019-12-05T05:03:36.847411+0000","last_became_active":"2019-12-05T05:03:32.226895+0000","last_became_peered":"2019-12-05T05:03:32.226895+0000","last_unstale":"2019-12-05T05:03:36.847411+0000","last_undegraded":"2019-12-05T05:03:36.847411+0000","last_fullsized":"2019-12-05T05:03:36.847411+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:31.200243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:31.200243+0000","last_clean_scrub_stamp":"2019-12-05T05:03:31.200243+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":"12","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T05:03:35.628786+0000","last_change":"2019-12-05T05:03:35.628207+0000","last_active":"2019-12-05T05:03:35.628786+0000","last_peered":"2019-12-05T05:03:35.628786+0000","last_clean":"2019-12-05T05:03:35.628786+0000","last_became_active":"2019-12-05T05:03:35.627352+0000","last_became_peered":"2019-12-05T05:03:35.627352+0000","last_unstale":"2019-12-05T05:03:35.628786+0000","last_undegraded":"2019-12-05T05:03:35.628786+0000","last_fullsized":"2019-12-05T05:03:35.628786+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:34.235090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:34.235090+0000","last_clean_scrub_stamp":"2019-12-05T05:03:34.235090+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6,5],"acting":[1,2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":"16","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T05:03:36.847322+0000","last_change":"2019-12-05T05:03:29.408129+0000","last_active":"2019-12-05T05:03:36.847322+0000","last_peered":"2019-12-05T05:03:36.847322+0000","last_clean":"2019-12-05T05:03:36.847322+0000","last_became_active":"2019-12-05T05:03:29.407108+0000","last_became_peered":"2019-12-05T05:03:29.407108+0000","last_unstale":"2019-12-05T05:03:36.847322+0000","last_undegraded":"2019-12-05T05:03:36.847322+0000","last_fullsized":"2019-12-05T05:03:36.847322+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:28.161818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:28.161818+0000","last_clean_scrub_stamp":"2019-12-05T05:03:28.161818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"13","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T05:03:35.610338+0000","last_change":"2019-12-05T05:03:32.636592+0000","last_active":"2019-12-05T05:03:35.610338+0000","last_peered":"2019-12-05T05:03:35.610338+0000","last_clean":"2019-12-05T05:03:35.610338+0000","last_became_active":"2019-12-05T05:03:32.635937+0000","last_became_peered":"2019-12-05T05:03:32.635937+0000","last_unstale":"2019-12-05T05:03:35.610338+0000","last_undegraded":"2019-12-05T05:03:35.610338+0000","last_fullsized":"2019-12-05T05:03:35.610338+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:31.200243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:31.200243+0000","last_clean_scrub_stamp":"2019-12-05T05:03:31.200243+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":"12","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T05:03:36.315636+0000","last_change":"2019-12-05T05:03:36.315160+0000","last_active":"2019-12-05T05:03:36.315636+0000","last_peered":"2019-12-05T05:03:36.315636+0000","last_clean":"2019-12-05T05:03:36.315636+0000","last_became_active":"2019-12-05T05:03:36.314006+0000","last_became_peered":"2019-12-05T05:03:36.314006+0000","last_unstale":"2019-12-05T05:03:36.315636+0000","last_undegraded":"2019-12-05T05:03:36.315636+0000","last_fullsized":"2019-12-05T05:03:36.315636+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:34.235090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:34.235090+0000","last_clean_scrub_stamp":"2019-12-05T05:03:34.235090+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6,4],"acting":[3,5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"16","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T05:03:36.847644+0000","last_change":"2019-12-05T05:03:29.407882+0000","last_active":"2019-12-05T05:03:36.847644+0000","last_peered":"2019-12-05T05:03:36.847644+0000","last_clean":"2019-12-05T05:03:36.847644+0000","last_became_active":"2019-12-05T05:03:29.406770+0000","last_became_peered":"2019-12-05T05:03:29.406770+0000","last_unstale":"2019-12-05T05:03:36.847644+0000","last_undegraded":"2019-12-05T05:03:36.847644+0000","last_fullsized":"2019-12-05T05:03:36.847644+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:28.161818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:28.161818+0000","last_clean_scrub_stamp":"2019-12-05T05:03:28.161818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"13","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T05:03:35.629132+0000","last_change":"2019-12-05T05:03:32.227654+0000","last_active":"2019-12-05T05:03:35.629132+0000","last_peered":"2019-12-05T05:03:35.629132+0000","last_clean":"2019-12-05T05:03:35.629132+0000","last_became_active":"2019-12-05T05:03:32.227052+0000","last_became_peered":"2019-12-05T05:03:32.227052+0000","last_unstale":"2019-12-05T05:03:35.629132+0000","last_undegraded":"2019-12-05T05:03:35.629132+0000","last_fullsized":"2019-12-05T05:03:35.629132+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:31.200243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:31.200243+0000","last_clean_scrub_stamp":"2019-12-05T05:03:31.200243+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":"12","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T05:03:35.734103+0000","last_change":"2019-12-05T05:03:35.733650+0000","last_active":"2019-12-05T05:03:35.734103+0000","last_peered":"2019-12-05T05:03:35.734103+0000","last_clean":"2019-12-05T05:03:35.734103+0000","last_became_active":"2019-12-05T05:03:35.732405+0000","last_became_peered":"2019-12-05T05:03:35.732405+0000","last_unstale":"2019-12-05T05:03:35.734103+0000","last_undegraded":"2019-12-05T05:03:35.734103+0000","last_fullsized":"2019-12-05T05:03:35.734103+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:34.235090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:34.235090+0000","last_clean_scrub_stamp":"2019-12-05T05:03:34.235090+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0,6],"acting":[4,5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":"16","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T05:03:35.629127+0000","last_change":"2019-12-05T05:03:29.407539+0000","last_active":"2019-12-05T05:03:35.629127+0000","last_peered":"2019-12-05T05:03:35.629127+0000","last_clean":"2019-12-05T05:03:35.629127+0000","last_became_active":"2019-12-05T05:03:29.406903+0000","last_became_peered":"2019-12-05T05:03:29.406903+0000","last_unstale":"2019-12-05T05:03:35.629127+0000","last_undegraded":"2019-12-05T05:03:35.629127+0000","last_fullsized":"2019-12-05T05:03:35.629127+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:28.161818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:28.161818+0000","last_clean_scrub_stamp":"2019-12-05T05:03:28.161818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"13","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T05:03:35.629133+0000","last_change":"2019-12-05T05:03:32.227838+0000","last_active":"2019-12-05T05:03:35.629133+0000","last_peered":"2019-12-05T05:03:35.629133+0000","last_clean":"2019-12-05T05:03:35.629133+0000","last_became_active":"2019-12-05T05:03:32.227050+0000","last_became_peered":"2019-12-05T05:03:32.227050+0000","last_unstale":"2019-12-05T05:03:35.629133+0000","last_undegraded":"2019-12-05T05:03:35.629133+0000","last_fullsized":"2019-12-05T05:03:35.629133+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:31.200243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:31.200243+0000","last_clean_scrub_stamp":"2019-12-05T05:03:31.200243+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":"12","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T05:03:35.626774+0000","last_change":"2019-12-05T05:03:35.267419+0000","last_active":"2019-12-05T05:03:35.626774+0000","last_peered":"2019-12-05T05:03:35.626774+0000","last_clean":"2019-12-05T05:03:35.626774+0000","last_became_active":"2019-12-05T05:03:35.266868+0000","last_became_peered":"2019-12-05T05:03:35.266868+0000","last_unstale":"2019-12-05T05:03:35.626774+0000","last_undegraded":"2019-12-05T05:03:35.626774+0000","last_fullsized":"2019-12-05T05:03:35.626774+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:34.235090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:34.235090+0000","last_clean_scrub_stamp":"2019-12-05T05:03:34.235090+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0,5],"acting":[1,2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":"15","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T05:03:35.624631+0000","last_change":"2019-12-05T05:03:29.407541+0000","last_active":"2019-12-05T05:03:35.624631+0000","last_peered":"2019-12-05T05:03:35.624631+0000","last_clean":"2019-12-05T05:03:35.624631+0000","last_became_active":"2019-12-05T05:03:29.406857+0000","last_became_peered":"2019-12-05T05:03:29.406857+0000","last_unstale":"2019-12-05T05:03:35.624631+0000","last_undegraded":"2019-12-05T05:03:35.624631+0000","last_fullsized":"2019-12-05T05:03:35.624631+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:28.161818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:28.161818+0000","last_clean_scrub_stamp":"2019-12-05T05:03:28.161818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"13","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T05:03:36.847262+0000","last_change":"2019-12-05T05:03:32.480464+0000","last_active":"2019-12-05T05:03:36.847262+0000","last_peered":"2019-12-05T05:03:36.847262+0000","last_clean":"2019-12-05T05:03:36.847262+0000","last_became_active":"2019-12-05T05:03:32.479565+0000","last_became_peered":"2019-12-05T05:03:32.479565+0000","last_unstale":"2019-12-05T05:03:36.847262+0000","last_undegraded":"2019-12-05T05:03:36.847262+0000","last_fullsized":"2019-12-05T05:03:36.847262+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:31.200243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:31.200243+0000","last_clean_scrub_stamp":"2019-12-05T05:03:31.200243+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":"12","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T05:03:35.630204+0000","last_change":"2019-12-05T05:03:35.629791+0000","last_active":"2019-12-05T05:03:35.630204+0000","last_peered":"2019-12-05T05:03:35.630204+0000","last_clean":"2019-12-05T05:03:35.630204+0000","last_became_active":"2019-12-05T05:03:35.629300+0000","last_became_peered":"2019-12-05T05:03:35.629300+0000","last_unstale":"2019-12-05T05:03:35.630204+0000","last_undegraded":"2019-12-05T05:03:35.630204+0000","last_fullsized":"2019-12-05T05:03:35.630204+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:34.235090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:34.235090+0000","last_clean_scrub_stamp":"2019-12-05T05:03:34.235090+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2,6],"acting":[5,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":"15","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T05:03:35.624610+0000","last_change":"2019-12-05T05:03:29.185245+0000","last_active":"2019-12-05T05:03:35.624610+0000","last_peered":"2019-12-05T05:03:35.624610+0000","last_clean":"2019-12-05T05:03:35.624610+0000","last_became_active":"2019-12-05T05:03:29.184716+0000","last_became_peered":"2019-12-05T05:03:29.184716+0000","last_unstale":"2019-12-05T05:03:35.624610+0000","last_undegraded":"2019-12-05T05:03:35.624610+0000","last_fullsized":"2019-12-05T05:03:35.624610+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:28.161818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:28.161818+0000","last_clean_scrub_stamp":"2019-12-05T05:03:28.161818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"13","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T05:03:35.626773+0000","last_change":"2019-12-05T05:03:32.476846+0000","last_active":"2019-12-05T05:03:35.626773+0000","last_peered":"2019-12-05T05:03:35.626773+0000","last_clean":"2019-12-05T05:03:35.626773+0000","last_became_active":"2019-12-05T05:03:32.476140+0000","last_became_peered":"2019-12-05T05:03:32.476140+0000","last_unstale":"2019-12-05T05:03:35.626773+0000","last_undegraded":"2019-12-05T05:03:35.626773+0000","last_fullsized":"2019-12-05T05:03:35.626773+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:31.200243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:31.200243+0000","last_clean_scrub_stamp":"2019-12-05T05:03:31.200243+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":32,"acting":32,"num_store_stats":8},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574854,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048832,"kb_used_data":248,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323008,"statfs":{"total":96636764160,"available":95562760192,"internally_reserved":1073741824,"allocated":253952,"data_stored":95960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.792}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.045}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.900}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.190}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.855}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.244}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.335}]}]},{"osd":6,"up_from":13,"seq":55834574855,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048832,"kb_used_data":248,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323008,"statfs":{"total":96636764160,"available":95562760192,"internally_reserved":1073741824,"allocated":253952,"data_stored":95960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.632}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.811}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.255}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.346}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.920}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.396}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.604}]}]},{"osd":1,"up_from":13,"seq":55834574853,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048832,"kb_used_data":248,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323008,"statfs":{"total":96636764160,"available":95562760192,"internally_reserved":1073741824,"allocated":253952,"data_stored":95960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.613}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.922}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.490}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.054}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.753}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.826}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.790}]}]},{"osd":0,"up_from":13,"seq":55834574855,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048832,"kb_used_data":248,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323008,"statfs":{"total":96636764160,"available":95562760192,"internally_reserved":1073741824,"allocated":253952,"data_stored":95960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.693}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.790}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.317}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.095}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.174}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.542}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.500}]}]},{"osd":2,"up_from":13,"seq":55834574853,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048832,"kb_used_data":248,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323008,"statfs":{"total":96636764160,"available":95562760192,"internally_reserved":1073741824,"allocated":253952,"data_stored":95960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.979}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.842}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.871}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.483}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.376}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.273}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.439}]}]},{"osd":3,"up_from":13,"seq":55834574854,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048832,"kb_used_data":248,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323008,"statfs":{"total":96636764160,"available":95562760192,"internally_reserved":1073741824,"allocated":253952,"data_stored":95960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.838}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.555}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.812}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.336}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.374}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.300}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.264}]}]},{"osd":4,"up_from":13,"seq":55834574853,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048832,"kb_used_data":248,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323008,"statfs":{"total":96636764160,"available":95562760192,"internally_reserved":1073741824,"allocated":253952,"data_stored":95960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.098}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.865}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.749}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.633}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.264}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.566}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.354}]}]},{"osd":5,"up_from":13,"seq":55834574854,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048832,"kb_used_data":248,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323008,"statfs":{"total":96636764160,"available":95562760192,"internally_reserved":1073741824,"allocated":253952,"data_stored":95960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.721}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.964}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.894}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.105}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.376}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.323}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.249}]}]}]}} 2019-12-05T05:03:44.563 INFO:tasks.ceph.ceph_manager.ceph:clean! 2019-12-05T05:03:44.563 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2019-12-05T05:03:44.564 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2019-12-05T05:03:44.564 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2019-12-05T05:03:44.662 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.652+0000 7f16b5770700 1 Processor -- start 2019-12-05T05:03:44.662 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.652+0000 7f16b5770700 1 -- start start 2019-12-05T05:03:44.662 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.652+0000 7f16b5770700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f16b00a7740 0x7f16b008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:44.663 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.652+0000 7f16b5770700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f16b007e720 0x7f16b007eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:44.663 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.652+0000 7f16b5770700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16b00a0710 con 0x7f16b009f2e0 2019-12-05T05:03:44.663 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.652+0000 7f16b5770700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16b00a0b00 con 0x7f16b007e720 2019-12-05T05:03:44.663 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.652+0000 7f16b5770700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16b008e5b0 con 0x7f16b00a7740 2019-12-05T05:03:44.663 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.652+0000 7f16af7fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f16b00a7740 0x7f16b008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:44.663 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.652+0000 7f16affff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f16b009f2e0 0x7f16b009f710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:53624/0 (socket says 172.21.15.42:53624) 2019-12-05T05:03:44.663 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.652+0000 7f16affff700 1 -- 172.21.15.42:0/944376286 learned_addr learned my addr 172.21.15.42:0/944376286 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:03:44.664 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.652+0000 7f16a6ffd700 1 --2- 172.21.15.42:0/944376286 >> v2:172.21.15.40:3300/0 conn(0x7f16b007e720 0x7f16b007eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:44.664 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.652+0000 7f16af7fe700 1 -- 172.21.15.42:0/944376286 >> v2:172.21.15.40:3300/0 conn(0x7f16b007e720 msgr2=0x7f16b007eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:03:44.664 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.652+0000 7f16af7fe700 1 --2- 172.21.15.42:0/944376286 >> v2:172.21.15.40:3300/0 conn(0x7f16b007e720 0x7f16b007eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:03:44.664 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.652+0000 7f16af7fe700 1 -- 172.21.15.42:0/944376286 >> v1:172.21.15.42:6789/0 conn(0x7f16b009f2e0 legacy=0x7f16b009f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:44.664 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.652+0000 7f16aeffd700 1 -- 172.21.15.42:0/944376286 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 763053600 0 0) 0x7f16b00a0710 con 0x7f16b009f2e0 2019-12-05T05:03:44.664 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.652+0000 7f16af7fe700 1 -- 172.21.15.42:0/944376286 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16b007f0a0 con 0x7f16b00a7740 2019-12-05T05:03:44.665 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.652+0000 7f16a6ffd700 1 --2- 172.21.15.42:0/944376286 >> v2:172.21.15.40:3300/0 conn(0x7f16b007e720 0x7f16b007eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:03:44.665 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.652+0000 7f16af7fe700 1 --2- 172.21.15.42:0/944376286 >> v2:172.21.15.42:3300/0 conn(0x7f16b00a7740 0x7f16b008df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f16a0002f30 tx=0x7f16a000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:44.665 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.652+0000 7f16af7fe700 1 --2- 172.21.15.42:0/944376286 >> v2:172.21.15.42:3300/0 conn(0x7f16b00a7740 0x7f16b008df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f16a0002f30 tx=0x7f16a000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:44.665 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.656+0000 7f16af7fe700 1 --2- 172.21.15.42:0/944376286 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f16b00a7740 0x7f16b008df80 secure :-1 s=READY pgs=147 cs=0 l=1 rx=0x7f16a0002f30 tx=0x7f16a000fc70).ready entity=mon.1 client_cookie=b6045c07585d3890 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:44.665 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.656+0000 7f16af7fe700 1 --2- 172.21.15.42:0/944376286 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f16b00a7740 0x7f16b008df80 secure :-1 s=THROTTLE_DONE pgs=147 cs=0 l=1 rx=0x7f16a0002f30 tx=0x7f16a000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:44.665 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.656+0000 7f16af7fe700 1 --2- 172.21.15.42:0/944376286 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f16b00a7740 0x7f16b008df80 secure :-1 s=THROTTLE_DONE pgs=147 cs=0 l=1 rx=0x7f16a0002f30 tx=0x7f16a000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:44.665 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.656+0000 7f16aeffd700 1 -- 172.21.15.42:0/944376286 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f16a00112b0 con 0x7f16b00a7740 2019-12-05T05:03:44.666 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.656+0000 7f16af7fe700 1 --2- 172.21.15.42:0/944376286 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f16b00a7740 0x7f16b008df80 secure :-1 s=THROTTLE_DONE pgs=147 cs=0 l=1 rx=0x7f16a0002f30 tx=0x7f16a000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:44.666 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.656+0000 7f16aeffd700 1 -- 172.21.15.42:0/944376286 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16a0008ad0 con 0x7f16b00a7740 2019-12-05T05:03:44.666 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.656+0000 7f16aeffd700 1 -- 172.21.15.42:0/944376286 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f16a0011890 con 0x7f16b00a7740 2019-12-05T05:03:44.666 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.656+0000 7f16b5770700 1 -- 172.21.15.42:0/944376286 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f16b00a7740 msgr2=0x7f16b008df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:44.666 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.656+0000 7f16b5770700 1 --2- 172.21.15.42:0/944376286 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f16b00a7740 0x7f16b008df80 secure :-1 s=READY pgs=147 cs=0 l=1 rx=0x7f16a0002f30 tx=0x7f16a000fc70).stop 2019-12-05T05:03:44.666 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.656+0000 7f16b5770700 1 -- 172.21.15.42:0/944376286 shutdown_connections 2019-12-05T05:03:44.666 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.656+0000 7f16b5770700 1 --2- 172.21.15.42:0/944376286 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f16b00a7740 0x7f16b008df80 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:44.667 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.656+0000 7f16b5770700 1 --2- 172.21.15.42:0/944376286 >> v2:172.21.15.40:3300/0 conn(0x7f16b007e720 0x7f16b007eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:03:44.667 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.656+0000 7f16b5770700 1 -- 172.21.15.42:0/944376286 shutdown_connections 2019-12-05T05:03:44.667 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.656+0000 7f16b5770700 1 -- 172.21.15.42:0/944376286 wait complete. 2019-12-05T05:03:44.667 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.656+0000 7f16b5770700 1 -- 172.21.15.42:0/944376286 >> 172.21.15.42:0/944376286 conn(0x7f16b00aabe0 msgr2=0x7f16b00a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:44.670 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.660+0000 7f16b5770700 1 Processor -- start 2019-12-05T05:03:44.670 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.660+0000 7f16b5770700 1 -- start start 2019-12-05T05:03:44.670 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.660+0000 7f16b5770700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f16b01231d0 0x7f16b0137e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:44.670 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.660+0000 7f16b5770700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16b01238a0 con 0x7f16b00a4a50 2019-12-05T05:03:44.670 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.660+0000 7f16b5770700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16b0130700 con 0x7f16b0119210 2019-12-05T05:03:44.671 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.660+0000 7f16b5770700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16b011a190 con 0x7f16b01231d0 2019-12-05T05:03:44.671 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.660+0000 7f16a6ffd700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f16b0119210 0x7f16b01229d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:53628/0 (socket says 172.21.15.42:53628) 2019-12-05T05:03:44.671 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.660+0000 7f16a6ffd700 1 -- 172.21.15.42:0/741243080 learned_addr learned my addr 172.21.15.42:0/741243080 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:03:44.671 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.660+0000 7f16affff700 1 --2- 172.21.15.42:0/741243080 >> v2:172.21.15.42:3300/0 conn(0x7f16b01231d0 0x7f16b0137e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:44.671 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.660+0000 7f16acff9700 1 -- 172.21.15.42:0/741243080 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3084230682 0 0) 0x7f16b0130700 con 0x7f16b0119210 2019-12-05T05:03:44.671 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.660+0000 7f16acff9700 1 -- 172.21.15.42:0/741243080 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16940030b0 con 0x7f16b0119210 2019-12-05T05:03:44.671 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.660+0000 7f16affff700 1 -- 172.21.15.42:0/741243080 >> v1:172.21.15.42:6789/0 conn(0x7f16b0119210 legacy=0x7f16b01229d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:44.672 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.660+0000 7f16affff700 1 -- 172.21.15.42:0/741243080 >> v1:172.21.15.40:6789/0 conn(0x7f16b00a4a50 legacy=0x7f16b012ff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:44.672 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.660+0000 7f16affff700 1 -- 172.21.15.42:0/741243080 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16b0119c10 con 0x7f16b01231d0 2019-12-05T05:03:44.672 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.660+0000 7f16affff700 1 --2- 172.21.15.42:0/741243080 >> v2:172.21.15.42:3300/0 conn(0x7f16b01231d0 0x7f16b0137e30 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f16b011aae0 tx=0x7f16a8010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:44.672 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.660+0000 7f16affff700 1 --2- 172.21.15.42:0/741243080 >> v2:172.21.15.42:3300/0 conn(0x7f16b01231d0 0x7f16b0137e30 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f16b011aae0 tx=0x7f16a8010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:44.673 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.660+0000 7f16affff700 1 --2- 172.21.15.42:0/741243080 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f16b01231d0 0x7f16b0137e30 secure :-1 s=READY pgs=149 cs=0 l=1 rx=0x7f16b011aae0 tx=0x7f16a8010b60).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:44.673 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.660+0000 7f16affff700 1 --2- 172.21.15.42:0/741243080 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f16b01231d0 0x7f16b0137e30 secure :-1 s=THROTTLE_DONE pgs=149 cs=0 l=1 rx=0x7f16b011aae0 tx=0x7f16a8010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:44.673 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.660+0000 7f16acff9700 1 -- 172.21.15.42:0/741243080 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f16a8004b90 con 0x7f16b01231d0 2019-12-05T05:03:44.673 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.660+0000 7f16affff700 1 --2- 172.21.15.42:0/741243080 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f16b01231d0 0x7f16b0137e30 secure :-1 s=THROTTLE_DONE pgs=149 cs=0 l=1 rx=0x7f16b011aae0 tx=0x7f16a8010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:44.673 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.660+0000 7f16affff700 1 --2- 172.21.15.42:0/741243080 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f16b01231d0 0x7f16b0137e30 secure :-1 s=THROTTLE_DONE pgs=149 cs=0 l=1 rx=0x7f16b011aae0 tx=0x7f16a8010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:44.673 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.660+0000 7f16acff9700 1 -- 172.21.15.42:0/741243080 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16a80047b0 con 0x7f16b01231d0 2019-12-05T05:03:44.673 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.660+0000 7f16b5770700 1 -- 172.21.15.42:0/741243080 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16b0119ee0 con 0x7f16b01231d0 2019-12-05T05:03:44.674 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.660+0000 7f16acff9700 1 -- 172.21.15.42:0/741243080 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f16a80125b0 con 0x7f16b01231d0 2019-12-05T05:03:44.674 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.660+0000 7f16b5770700 1 -- 172.21.15.42:0/741243080 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16b00a9790 con 0x7f16b01231d0 2019-12-05T05:03:44.674 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.664+0000 7f16affff700 1 --2- 172.21.15.42:0/741243080 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f16b01231d0 0x7f16b0137e30 secure :-1 s=THROTTLE_DONE pgs=149 cs=0 l=1 rx=0x7f16b011aae0 tx=0x7f16a8010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:44.674 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.664+0000 7f16acff9700 1 -- 172.21.15.42:0/741243080 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f16a80128c0 con 0x7f16b01231d0 2019-12-05T05:03:44.674 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.664+0000 7f16affff700 1 --2- 172.21.15.42:0/741243080 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f16b01231d0 0x7f16b0137e30 secure :-1 s=THROTTLE_DONE pgs=149 cs=0 l=1 rx=0x7f16b011aae0 tx=0x7f16a8010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:44.674 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.664+0000 7f16acff9700 1 --2- 172.21.15.42:0/741243080 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f169403b210 0x7f169403d720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:44.674 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.664+0000 7f16acff9700 1 -- 172.21.15.42:0/741243080 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f16a804b090 con 0x7f16b01231d0 2019-12-05T05:03:44.675 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.664+0000 7f16af7fe700 1 --2- 172.21.15.42:0/741243080 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f169403b210 0x7f169403d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:44.675 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.664+0000 7f16af7fe700 1 --2- 172.21.15.42:0/741243080 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f169403b210 0x7f169403d720 crc :-1 s=READY pgs=83 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:44.675 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.668+0000 7f16b5770700 1 -- 172.21.15.42:0/741243080 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16b005d110 con 0x7f16b01231d0 2019-12-05T05:03:44.678 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.672+0000 7f16affff700 1 --2- 172.21.15.42:0/741243080 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f16b01231d0 0x7f16b0137e30 secure :-1 s=THROTTLE_DONE pgs=149 cs=0 l=1 rx=0x7f16b011aae0 tx=0x7f16a8010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:44.678 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.672+0000 7f16acff9700 1 -- 172.21.15.42:0/741243080 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f16a8032370 con 0x7f16b01231d0 2019-12-05T05:03:44.832 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.828+0000 7f16b5770700 1 -- 172.21.15.42:0/741243080 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f16b005b8c0 con 0x7f16b01231d0 2019-12-05T05:03:44.833 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.828+0000 7f16affff700 1 --2- 172.21.15.42:0/741243080 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f16b01231d0 0x7f16b0137e30 secure :-1 s=THROTTLE_DONE pgs=149 cs=0 l=1 rx=0x7f16b011aae0 tx=0x7f16a8010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:44.833 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.828+0000 7f16acff9700 1 -- 172.21.15.42:0/741243080 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+45 (secure 0 0 0) 0x7f16a8032f20 con 0x7f16b01231d0 2019-12-05T05:03:44.833 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:03:44.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.828+0000 7f16b5770700 1 -- 172.21.15.42:0/741243080 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f169403b210 msgr2=0x7f169403d720 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:44.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.828+0000 7f16b5770700 1 --2- 172.21.15.42:0/741243080 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f169403b210 0x7f169403d720 crc :-1 s=READY pgs=83 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:44.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.828+0000 7f16b5770700 1 -- 172.21.15.42:0/741243080 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f16b01231d0 msgr2=0x7f16b0137e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:44.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.828+0000 7f16b5770700 1 --2- 172.21.15.42:0/741243080 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f16b01231d0 0x7f16b0137e30 secure :-1 s=READY pgs=149 cs=0 l=1 rx=0x7f16b011aae0 tx=0x7f16a8010b60).stop 2019-12-05T05:03:44.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.828+0000 7f16b5770700 1 -- 172.21.15.42:0/741243080 shutdown_connections 2019-12-05T05:03:44.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.828+0000 7f16b5770700 1 --2- 172.21.15.42:0/741243080 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f169403b210 0x7f169403d720 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:44.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.828+0000 7f16b5770700 1 --2- 172.21.15.42:0/741243080 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f16b01231d0 0x7f16b0137e30 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:44.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.828+0000 7f16b5770700 1 -- 172.21.15.42:0/741243080 shutdown_connections 2019-12-05T05:03:44.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.828+0000 7f16b5770700 1 -- 172.21.15.42:0/741243080 wait complete. 2019-12-05T05:03:44.837 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.828+0000 7f16b5770700 1 -- 172.21.15.42:0/741243080 >> 172.21.15.42:0/741243080 conn(0x7f16b00aabe0 msgr2=0x7f16b0059ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:44.845 INFO:teuthology.orchestra.run.smithi042.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2019-12-05T05:03:44.845 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2019-12-05T05:03:44.846 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2019-12-05T05:03:44.938 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.928+0000 7ff9cff0c700 1 Processor -- start 2019-12-05T05:03:44.939 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.928+0000 7ff9cff0c700 1 -- start start 2019-12-05T05:03:44.939 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.932+0000 7ff9cff0c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff9c80a7730 0x7ff9c808df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:44.939 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.932+0000 7ff9cff0c700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9c80a0700 con 0x7ff9c807e710 2019-12-05T05:03:44.939 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.932+0000 7ff9cff0c700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9c80a9b90 con 0x7ff9c809fad0 2019-12-05T05:03:44.939 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.932+0000 7ff9cff0c700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9c80a0af0 con 0x7ff9c80a7730 2019-12-05T05:03:44.939 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.932+0000 7ff9ce4a9700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff9c80a7730 0x7ff9c808df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:44.940 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.932+0000 7ff9ce4a9700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff9c80a7730 0x7ff9c808df70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:53596/0 (socket says 172.21.15.42:53596) 2019-12-05T05:03:44.940 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.932+0000 7ff9ce4a9700 1 -- 172.21.15.42:0/652878483 learned_addr learned my addr 172.21.15.42:0/652878483 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:03:44.940 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.932+0000 7ff9ce4a9700 1 -- 172.21.15.42:0/652878483 >> v1:172.21.15.42:6789/0 conn(0x7ff9c809fad0 legacy=0x7ff9c814efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:44.940 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.932+0000 7ff9ce4a9700 1 -- 172.21.15.42:0/652878483 >> v1:172.21.15.40:6789/0 conn(0x7ff9c807e710 legacy=0x7ff9c809f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:44.940 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.932+0000 7ff9ce4a9700 1 -- 172.21.15.42:0/652878483 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9c807eb40 con 0x7ff9c80a7730 2019-12-05T05:03:44.940 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.932+0000 7ff9ce4a9700 1 --2- 172.21.15.42:0/652878483 >> v2:172.21.15.42:3300/0 conn(0x7ff9c80a7730 0x7ff9c808df70 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7ff9b8008790 tx=0x7ff9b800fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:44.941 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.932+0000 7ff9ce4a9700 1 --2- 172.21.15.42:0/652878483 >> v2:172.21.15.42:3300/0 conn(0x7ff9c80a7730 0x7ff9c808df70 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7ff9b8008790 tx=0x7ff9b800fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:44.941 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.932+0000 7ff9ce4a9700 1 --2- 172.21.15.42:0/652878483 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff9c80a7730 0x7ff9c808df70 secure :-1 s=READY pgs=151 cs=0 l=1 rx=0x7ff9b8008790 tx=0x7ff9b800fcb0).ready entity=mon.1 client_cookie=ac71d1e91753a66d server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:44.941 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.932+0000 7ff9ce4a9700 1 --2- 172.21.15.42:0/652878483 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff9c80a7730 0x7ff9c808df70 secure :-1 s=THROTTLE_DONE pgs=151 cs=0 l=1 rx=0x7ff9b8008790 tx=0x7ff9b800fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:44.941 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.932+0000 7ff9cd4a7700 1 -- 172.21.15.42:0/652878483 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff9b8011520 con 0x7ff9c80a7730 2019-12-05T05:03:44.941 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.932+0000 7ff9ce4a9700 1 --2- 172.21.15.42:0/652878483 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff9c80a7730 0x7ff9c808df70 secure :-1 s=THROTTLE_DONE pgs=151 cs=0 l=1 rx=0x7ff9b8008790 tx=0x7ff9b800fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:44.941 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.932+0000 7ff9cd4a7700 1 -- 172.21.15.42:0/652878483 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9b8008ca0 con 0x7ff9c80a7730 2019-12-05T05:03:44.942 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.932+0000 7ff9ce4a9700 1 --2- 172.21.15.42:0/652878483 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff9c80a7730 0x7ff9c808df70 secure :-1 s=THROTTLE_DONE pgs=151 cs=0 l=1 rx=0x7ff9b8008790 tx=0x7ff9b800fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:44.942 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.932+0000 7ff9cd4a7700 1 -- 172.21.15.42:0/652878483 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff9b8011b00 con 0x7ff9c80a7730 2019-12-05T05:03:44.942 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.932+0000 7ff9cff0c700 1 -- 172.21.15.42:0/652878483 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff9c80a7730 msgr2=0x7ff9c808df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:44.942 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.932+0000 7ff9cff0c700 1 --2- 172.21.15.42:0/652878483 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff9c80a7730 0x7ff9c808df70 secure :-1 s=READY pgs=151 cs=0 l=1 rx=0x7ff9b8008790 tx=0x7ff9b800fcb0).stop 2019-12-05T05:03:44.942 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.932+0000 7ff9cff0c700 1 -- 172.21.15.42:0/652878483 shutdown_connections 2019-12-05T05:03:44.943 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.932+0000 7ff9cff0c700 1 --2- 172.21.15.42:0/652878483 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff9c80a7730 0x7ff9c808df70 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:44.943 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.932+0000 7ff9cff0c700 1 -- 172.21.15.42:0/652878483 shutdown_connections 2019-12-05T05:03:44.943 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.932+0000 7ff9cff0c700 1 -- 172.21.15.42:0/652878483 wait complete. 2019-12-05T05:03:44.943 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.932+0000 7ff9cff0c700 1 -- 172.21.15.42:0/652878483 >> 172.21.15.42:0/652878483 conn(0x7ff9c80aabd0 msgr2=0x7ff9c80a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:44.947 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.936+0000 7ff9cff0c700 1 Processor -- start 2019-12-05T05:03:44.947 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.936+0000 7ff9cff0c700 1 -- start start 2019-12-05T05:03:44.947 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.936+0000 7ff9cff0c700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff9c81259d0 0x7ff9c8125e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:44.947 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.936+0000 7ff9cff0c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff9c807d380 0x7ff9c807d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:44.947 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.936+0000 7ff9cff0c700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9c8144450 con 0x7ff9c81120e0 2019-12-05T05:03:44.948 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.936+0000 7ff9cff0c700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9c81441b0 con 0x7ff9c81259d0 2019-12-05T05:03:44.948 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.936+0000 7ff9cff0c700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9c8150380 con 0x7ff9c807d380 2019-12-05T05:03:44.948 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.936+0000 7ff9ce4a9700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7ff9c81120e0 0x7ff9c8143600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:53640/0 (socket says 172.21.15.42:53640) 2019-12-05T05:03:44.948 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.936+0000 7ff9ce4a9700 1 -- 172.21.15.42:0/4129492973 learned_addr learned my addr 172.21.15.42:0/4129492973 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:03:44.948 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.936+0000 7ff9cecaa700 1 --2- 172.21.15.42:0/4129492973 >> v2:172.21.15.42:3300/0 conn(0x7ff9c807d380 0x7ff9c807d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:44.948 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.936+0000 7ff9cdca8700 1 --2- 172.21.15.42:0/4129492973 >> v2:172.21.15.40:3300/0 conn(0x7ff9c81259d0 0x7ff9c8125e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:44.948 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.936+0000 7ff9c6ffd700 1 -- 172.21.15.42:0/4129492973 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2204783495 0 0) 0x7ff9c8144450 con 0x7ff9c81120e0 2019-12-05T05:03:44.949 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.936+0000 7ff9c6ffd700 1 -- 172.21.15.42:0/4129492973 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9a80030b0 con 0x7ff9c81120e0 2019-12-05T05:03:44.949 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.936+0000 7ff9cecaa700 1 -- 172.21.15.42:0/4129492973 >> v2:172.21.15.40:3300/0 conn(0x7ff9c81259d0 msgr2=0x7ff9c8125e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:44.949 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.936+0000 7ff9cecaa700 1 --2- 172.21.15.42:0/4129492973 >> v2:172.21.15.40:3300/0 conn(0x7ff9c81259d0 0x7ff9c8125e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:44.949 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.936+0000 7ff9cecaa700 1 -- 172.21.15.42:0/4129492973 >> v1:172.21.15.42:6789/0 conn(0x7ff9c81120e0 legacy=0x7ff9c8143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:44.949 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.936+0000 7ff9cecaa700 1 -- 172.21.15.42:0/4129492973 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9c8143e00 con 0x7ff9c807d380 2019-12-05T05:03:44.949 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.936+0000 7ff9cecaa700 1 --2- 172.21.15.42:0/4129492973 >> v2:172.21.15.42:3300/0 conn(0x7ff9c807d380 0x7ff9c807d7b0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7ff9c0000f00 tx=0x7ff9c0010b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:44.949 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.936+0000 7ff9cecaa700 1 --2- 172.21.15.42:0/4129492973 >> v2:172.21.15.42:3300/0 conn(0x7ff9c807d380 0x7ff9c807d7b0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7ff9c0000f00 tx=0x7ff9c0010b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:44.950 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.936+0000 7ff9cecaa700 1 --2- 172.21.15.42:0/4129492973 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff9c807d380 0x7ff9c807d7b0 secure :-1 s=READY pgs=153 cs=0 l=1 rx=0x7ff9c0000f00 tx=0x7ff9c0010b00).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:44.950 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.936+0000 7ff9cecaa700 1 --2- 172.21.15.42:0/4129492973 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff9c807d380 0x7ff9c807d7b0 secure :-1 s=THROTTLE_DONE pgs=153 cs=0 l=1 rx=0x7ff9c0000f00 tx=0x7ff9c0010b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:44.950 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.936+0000 7ff9cecaa700 1 --2- 172.21.15.42:0/4129492973 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff9c807d380 0x7ff9c807d7b0 secure :-1 s=THROTTLE_DONE pgs=153 cs=0 l=1 rx=0x7ff9c0000f00 tx=0x7ff9c0010b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:44.950 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.936+0000 7ff9c6ffd700 1 -- 172.21.15.42:0/4129492973 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff9c0006bd0 con 0x7ff9c807d380 2019-12-05T05:03:44.950 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.936+0000 7ff9cecaa700 1 --2- 172.21.15.42:0/4129492973 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff9c807d380 0x7ff9c807d7b0 secure :-1 s=THROTTLE_DONE pgs=153 cs=0 l=1 rx=0x7ff9c0000f00 tx=0x7ff9c0010b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:44.950 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.936+0000 7ff9c6ffd700 1 -- 172.21.15.42:0/4129492973 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9c0013060 con 0x7ff9c807d380 2019-12-05T05:03:44.950 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.936+0000 7ff9cff0c700 1 -- 172.21.15.42:0/4129492973 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9c8126350 con 0x7ff9c807d380 2019-12-05T05:03:44.951 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.936+0000 7ff9c6ffd700 1 -- 172.21.15.42:0/4129492973 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff9c0002410 con 0x7ff9c807d380 2019-12-05T05:03:44.951 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.936+0000 7ff9cff0c700 1 -- 172.21.15.42:0/4129492973 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9c807dea0 con 0x7ff9c807d380 2019-12-05T05:03:44.951 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.936+0000 7ff9cecaa700 1 --2- 172.21.15.42:0/4129492973 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff9c807d380 0x7ff9c807d7b0 secure :-1 s=THROTTLE_DONE pgs=153 cs=0 l=1 rx=0x7ff9c0000f00 tx=0x7ff9c0010b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:44.951 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.940+0000 7ff9c6ffd700 1 -- 172.21.15.42:0/4129492973 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7ff9c00026e0 con 0x7ff9c807d380 2019-12-05T05:03:44.951 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.940+0000 7ff9cecaa700 1 --2- 172.21.15.42:0/4129492973 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff9c807d380 0x7ff9c807d7b0 secure :-1 s=THROTTLE_DONE pgs=153 cs=0 l=1 rx=0x7ff9c0000f00 tx=0x7ff9c0010b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:44.952 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.940+0000 7ff9c6ffd700 1 --2- 172.21.15.42:0/4129492973 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff9a803b280 0x7ff9a803d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:44.952 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.940+0000 7ff9c6ffd700 1 -- 172.21.15.42:0/4129492973 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6031+0+0 (secure 0 0 0) 0x7ff9c004b240 con 0x7ff9c807d380 2019-12-05T05:03:44.952 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.940+0000 7ff9ce4a9700 1 --2- 172.21.15.42:0/4129492973 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff9a803b280 0x7ff9a803d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:44.952 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.940+0000 7ff9ce4a9700 1 --2- 172.21.15.42:0/4129492973 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff9a803b280 0x7ff9a803d790 crc :-1 s=READY pgs=84 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:44.953 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.944+0000 7ff9cff0c700 1 -- 172.21.15.42:0/4129492973 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9c805d100 con 0x7ff9c807d380 2019-12-05T05:03:44.956 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.952+0000 7ff9cecaa700 1 --2- 172.21.15.42:0/4129492973 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff9c807d380 0x7ff9c807d7b0 secure :-1 s=THROTTLE_DONE pgs=153 cs=0 l=1 rx=0x7ff9c0000f00 tx=0x7ff9c0010b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:44.956 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:44.952+0000 7ff9c6ffd700 1 -- 172.21.15.42:0/4129492973 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7ff9c0032520 con 0x7ff9c807d380 2019-12-05T05:03:45.129 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.116+0000 7ff9cff0c700 1 -- 172.21.15.42:0/4129492973 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9c805b8b0 con 0x7ff9c807d380 2019-12-05T05:03:45.129 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.120+0000 7ff9cecaa700 1 --2- 172.21.15.42:0/4129492973 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff9c807d380 0x7ff9c807d7b0 secure :-1 s=THROTTLE_DONE pgs=153 cs=0 l=1 rx=0x7ff9c0000f00 tx=0x7ff9c0010b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:45.129 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.120+0000 7ff9c6ffd700 1 -- 172.21.15.42:0/4129492973 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+14885 (secure 0 0 0) 0x7ff9c00330d0 con 0x7ff9c807d380 2019-12-05T05:03:45.130 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.124+0000 7ff9cff0c700 1 -- 172.21.15.42:0/4129492973 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff9a803b280 msgr2=0x7ff9a803d790 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:45.130 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.124+0000 7ff9cff0c700 1 --2- 172.21.15.42:0/4129492973 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff9a803b280 0x7ff9a803d790 crc :-1 s=READY pgs=84 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:45.130 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.124+0000 7ff9cff0c700 1 -- 172.21.15.42:0/4129492973 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff9c807d380 msgr2=0x7ff9c807d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:45.130 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.124+0000 7ff9cff0c700 1 --2- 172.21.15.42:0/4129492973 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff9c807d380 0x7ff9c807d7b0 secure :-1 s=READY pgs=153 cs=0 l=1 rx=0x7ff9c0000f00 tx=0x7ff9c0010b00).stop 2019-12-05T05:03:45.130 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.124+0000 7ff9cff0c700 1 -- 172.21.15.42:0/4129492973 shutdown_connections 2019-12-05T05:03:45.130 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.124+0000 7ff9cff0c700 1 --2- 172.21.15.42:0/4129492973 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff9a803b280 0x7ff9a803d790 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:45.130 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.124+0000 7ff9cff0c700 1 --2- 172.21.15.42:0/4129492973 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff9c807d380 0x7ff9c807d7b0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:45.131 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.124+0000 7ff9cff0c700 1 --2- 172.21.15.42:0/4129492973 >> v2:172.21.15.40:3300/0 conn(0x7ff9c81259d0 0x7ff9c8125e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:45.131 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.124+0000 7ff9cff0c700 1 -- 172.21.15.42:0/4129492973 shutdown_connections 2019-12-05T05:03:45.131 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.124+0000 7ff9cff0c700 1 -- 172.21.15.42:0/4129492973 wait complete. 2019-12-05T05:03:45.131 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.124+0000 7ff9cff0c700 1 -- 172.21.15.42:0/4129492973 >> 172.21.15.42:0/4129492973 conn(0x7ff9c80a80c0 msgr2=0x7ff9c80a4a40 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:45.131 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:03:45.132 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":24,"fsid":"5d836cfc-a83b-4433-9b3e-b9b2f31c9050","created":"2019-12-05T05:03:13.215602+0000","modified":"2019-12-05T05:03:36.607704+0000","last_up_change":"2019-12-05T05:03:22.664783+0000","last_in_change":"2019-12-05T05:03:22.664783+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2019-12-05T05:03:27.394605+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2019-12-05T05:03:30.419845+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2019-12-05T05:03:33.249075+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e3fc49bb-dcbc-43ff-ac7c-9eb0ea0bd257","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6810","nonce":12118},{"type":"v1","addr":"172.21.15.42:6811","nonce":12118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6812","nonce":12118},{"type":"v1","addr":"172.21.15.42:6813","nonce":12118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6816","nonce":12118},{"type":"v1","addr":"172.21.15.42:6817","nonce":12118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6814","nonce":12118},{"type":"v1","addr":"172.21.15.42:6815","nonce":12118}]},"public_addr":"172.21.15.42:6811/12118","cluster_addr":"172.21.15.42:6813/12118","heartbeat_back_addr":"172.21.15.42:6817/12118","heartbeat_front_addr":"172.21.15.42:6815/12118","state":["exists","up"]},{"osd":1,"uuid":"902543fb-f5a2-4321-9c9c-26f9b9ca8629","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6826","nonce":12119},{"type":"v1","addr":"172.21.15.42:6827","nonce":12119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6828","nonce":12119},{"type":"v1","addr":"172.21.15.42:6829","nonce":12119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6832","nonce":12119},{"type":"v1","addr":"172.21.15.42:6833","nonce":12119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6830","nonce":12119},{"type":"v1","addr":"172.21.15.42:6831","nonce":12119}]},"public_addr":"172.21.15.42:6827/12119","cluster_addr":"172.21.15.42:6829/12119","heartbeat_back_addr":"172.21.15.42:6833/12119","heartbeat_front_addr":"172.21.15.42:6831/12119","state":["exists","up"]},{"osd":2,"uuid":"28739ff7-bed7-4b75-8ea5-b57d088f1250","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6802","nonce":12117},{"type":"v1","addr":"172.21.15.42:6803","nonce":12117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6804","nonce":12117},{"type":"v1","addr":"172.21.15.42:6805","nonce":12117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6808","nonce":12117},{"type":"v1","addr":"172.21.15.42:6809","nonce":12117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6806","nonce":12117},{"type":"v1","addr":"172.21.15.42:6807","nonce":12117}]},"public_addr":"172.21.15.42:6803/12117","cluster_addr":"172.21.15.42:6805/12117","heartbeat_back_addr":"172.21.15.42:6809/12117","heartbeat_front_addr":"172.21.15.42:6807/12117","state":["exists","up"]},{"osd":3,"uuid":"1960f66b-2d40-4bb9-a289-ce8cd0c1c142","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6818","nonce":12116},{"type":"v1","addr":"172.21.15.42:6819","nonce":12116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6820","nonce":12116},{"type":"v1","addr":"172.21.15.42:6821","nonce":12116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6824","nonce":12116},{"type":"v1","addr":"172.21.15.42:6825","nonce":12116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6822","nonce":12116},{"type":"v1","addr":"172.21.15.42:6823","nonce":12116}]},"public_addr":"172.21.15.42:6819/12116","cluster_addr":"172.21.15.42:6821/12116","heartbeat_back_addr":"172.21.15.42:6825/12116","heartbeat_front_addr":"172.21.15.42:6823/12116","state":["exists","up"]},{"osd":4,"uuid":"45bbf1bd-7c8a-4a0d-b76b-afa6148a855c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":12488},{"type":"v1","addr":"172.21.15.40:6809","nonce":12488}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":12488},{"type":"v1","addr":"172.21.15.40:6812","nonce":12488}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":12488},{"type":"v1","addr":"172.21.15.40:6825","nonce":12488}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":12488},{"type":"v1","addr":"172.21.15.40:6819","nonce":12488}]},"public_addr":"172.21.15.40:6809/12488","cluster_addr":"172.21.15.40:6812/12488","heartbeat_back_addr":"172.21.15.40:6825/12488","heartbeat_front_addr":"172.21.15.40:6819/12488","state":["exists","up"]},{"osd":5,"uuid":"8a5511cd-925f-4869-a49e-924383e4ce53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":12489},{"type":"v1","addr":"172.21.15.40:6801","nonce":12489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":12489},{"type":"v1","addr":"172.21.15.40:6803","nonce":12489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":12489},{"type":"v1","addr":"172.21.15.40:6807","nonce":12489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":12489},{"type":"v1","addr":"172.21.15.40:6805","nonce":12489}]},"public_addr":"172.21.15.40:6801/12489","cluster_addr":"172.21.15.40:6803/12489","heartbeat_back_addr":"172.21.15.40:6807/12489","heartbeat_front_addr":"172.21.15.40:6805/12489","state":["exists","up"]},{"osd":6,"uuid":"d692c179-379e-446c-8484-a219a9241e04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6811","nonce":12491},{"type":"v1","addr":"172.21.15.40:6814","nonce":12491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6817","nonce":12491},{"type":"v1","addr":"172.21.15.40:6818","nonce":12491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":12491},{"type":"v1","addr":"172.21.15.40:6828","nonce":12491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":12491},{"type":"v1","addr":"172.21.15.40:6824","nonce":12491}]},"public_addr":"172.21.15.40:6814/12491","cluster_addr":"172.21.15.40:6818/12491","heartbeat_back_addr":"172.21.15.40:6828/12491","heartbeat_front_addr":"172.21.15.40:6824/12491","state":["exists","up"]},{"osd":7,"uuid":"79c3c243-190e-4e35-b769-319a2077e7cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6813","nonce":12490},{"type":"v1","addr":"172.21.15.40:6816","nonce":12490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":12490},{"type":"v1","addr":"172.21.15.40:6823","nonce":12490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":12490},{"type":"v1","addr":"172.21.15.40:6831","nonce":12490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6827","nonce":12490},{"type":"v1","addr":"172.21.15.40:6829","nonce":12490}]},"public_addr":"172.21.15.40:6816/12490","cluster_addr":"172.21.15.40:6823/12490","heartbeat_back_addr":"172.21.15.40:6831/12490","heartbeat_front_addr":"172.21.15.40:6829/12490","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T05:03:21.349150+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T05:03:21.644281+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T05:03:21.394040+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T05:03:21.587096+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T05:03:21.656497+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T05:03:21.510547+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T05:03:21.659358+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T05:03:21.628063+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2019-12-05T05:03:45.142 INFO:tasks.cephfs.filesystem.ceph_manager:[{u'cache_target_full_ratio_micro': 800000, u'fast_read': False, u'stripe_width': 0, u'flags_names': u'hashpspool', u'tier_of': -1, u'hit_set_grade_decay_rate': 0, u'pg_placement_num': 8, u'use_gmt_hitset': True, u'cache_target_dirty_ratio_micro': 400000, u'create_time': u'2019-12-05T05:03:27.394605+0000', u'snap_epoch': 0, u'erasure_code_profile': u'', u'pg_autoscale_mode': u'off', u'snap_seq': 0, u'expected_num_objects': 0, u'size': 2, u'pg_num_pending': 8, u'auid': 0, u'cache_min_flush_age': 0, u'hit_set_period': 0, u'min_read_recency_for_promote': 0, u'target_max_objects': 0, u'pg_placement_num_target': 8, u'pg_num': 8, u'type': 1, u'grade_table': [], u'pool_name': u'rbd', u'cache_min_evict_age': 0, u'snap_mode': u'selfmanaged', u'pg_num_target': 8, u'cache_mode': u'none', u'min_size': 1, u'cache_target_dirty_high_ratio_micro': 600000, u'object_hash': 2, u'last_pg_merge_meta': {u'ready_epoch': 0, u'source_version': u"0'0", u'source_pgid': u'0.0', u'last_epoch_clean': 0, u'target_version': u"0'0", u'last_epoch_started': 0}, u'write_tier': -1, u'last_force_op_resend_preluminous': u'0', u'pool': 1, u'removed_snaps': u'[]', u'crush_rule': 0, u'tiers': [], u'hit_set_params': {u'type': u'none'}, u'last_force_op_resend': u'0', u'pool_snaps': [], u'quota_max_objects': 0, u'last_force_op_resend_prenautilus': u'0', u'application_metadata': {u'rbd': {}}, u'options': {}, u'hit_set_count': 0, u'flags': 1, u'target_max_bytes': 0, u'quota_max_bytes': 0, u'hit_set_search_last_n': 0, u'last_change': u'17', u'min_write_recency_for_promote': 0, u'read_tier': -1}, {u'cache_target_full_ratio_micro': 800000, u'fast_read': False, u'stripe_width': 0, u'flags_names': u'hashpspool', u'tier_of': -1, u'hit_set_grade_decay_rate': 0, u'pg_placement_num': 8, u'use_gmt_hitset': True, u'cache_target_dirty_ratio_micro': 400000, u'create_time': u'2019-12-05T05:03:30.419845+0000', u'snap_epoch': 0, u'erasure_code_profile': u'', u'pg_autoscale_mode': u'off', u'snap_seq': 0, u'expected_num_objects': 0, u'size': 2, u'pg_num_pending': 8, u'auid': 0, u'cache_min_flush_age': 0, u'hit_set_period': 0, u'min_read_recency_for_promote': 0, u'target_max_objects': 0, u'pg_placement_num_target': 8, u'pg_num': 8, u'type': 1, u'grade_table': [], u'pool_name': u'cephfs_metadata', u'cache_min_evict_age': 0, u'snap_mode': u'selfmanaged', u'pg_num_target': 8, u'cache_mode': u'none', u'min_size': 1, u'cache_target_dirty_high_ratio_micro': 600000, u'object_hash': 2, u'last_pg_merge_meta': {u'ready_epoch': 0, u'source_version': u"0'0", u'source_pgid': u'0.0', u'last_epoch_clean': 0, u'target_version': u"0'0", u'last_epoch_started': 0}, u'write_tier': -1, u'last_force_op_resend_preluminous': u'0', u'pool': 2, u'removed_snaps': u'[]', u'crush_rule': 0, u'tiers': [], u'hit_set_params': {u'type': u'none'}, u'last_force_op_resend': u'0', u'pool_snaps': [], u'quota_max_objects': 0, u'last_force_op_resend_prenautilus': u'0', u'application_metadata': {u'cephfs': {u'metadata': u'cephfs'}}, u'options': {u'pg_autoscale_bias': 4, u'recovery_priority': 5, u'pg_num_min': 16}, u'hit_set_count': 0, u'flags': 1, u'target_max_bytes': 0, u'quota_max_bytes': 0, u'hit_set_search_last_n': 0, u'last_change': u'23', u'min_write_recency_for_promote': 0, u'read_tier': -1}, {u'cache_target_full_ratio_micro': 800000, u'fast_read': False, u'stripe_width': 8192, u'flags_names': u'hashpspool,ec_overwrites', u'tier_of': -1, u'hit_set_grade_decay_rate': 0, u'pg_placement_num': 8, u'use_gmt_hitset': True, u'cache_target_dirty_ratio_micro': 400000, u'create_time': u'2019-12-05T05:03:33.249075+0000', u'snap_epoch': 0, u'erasure_code_profile': u'cephfs_data', u'pg_autoscale_mode': u'off', u'snap_seq': 0, u'expected_num_objects': 0, u'size': 4, u'pg_num_pending': 8, u'auid': 0, u'cache_min_flush_age': 0, u'hit_set_period': 0, u'min_read_recency_for_promote': 0, u'target_max_objects': 0, u'pg_placement_num_target': 8, u'pg_num': 8, u'type': 3, u'grade_table': [], u'pool_name': u'cephfs_data', u'cache_min_evict_age': 0, u'snap_mode': u'selfmanaged', u'pg_num_target': 8, u'cache_mode': u'none', u'min_size': 3, u'cache_target_dirty_high_ratio_micro': 600000, u'object_hash': 2, u'last_pg_merge_meta': {u'ready_epoch': 0, u'source_version': u"0'0", u'source_pgid': u'0.0', u'last_epoch_clean': 0, u'target_version': u"0'0", u'last_epoch_started': 0}, u'write_tier': -1, u'last_force_op_resend_preluminous': u'0', u'pool': 3, u'removed_snaps': u'[]', u'crush_rule': 1, u'tiers': [], u'hit_set_params': {u'type': u'none'}, u'last_force_op_resend': u'0', u'pool_snaps': [], u'quota_max_objects': 0, u'last_force_op_resend_prenautilus': u'0', u'application_metadata': {u'cephfs': {u'data': u'cephfs'}}, u'options': {}, u'hit_set_count': 0, u'flags': 5, u'target_max_bytes': 0, u'quota_max_bytes': 0, u'hit_set_search_last_n': 0, u'last_change': u'24', u'min_write_recency_for_promote': 0, u'read_tier': -1}] 2019-12-05T05:03:45.143 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2019-12-05T05:03:45.256 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.244+0000 7f338f7f9700 1 Processor -- start 2019-12-05T05:03:45.257 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.244+0000 7f338f7f9700 1 -- start start 2019-12-05T05:03:45.257 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.248+0000 7f338f7f9700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f338814df60 0x7f33881503a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:45.257 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.248+0000 7f338f7f9700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f338807f080 con 0x7f33880a9a00 2019-12-05T05:03:45.257 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.248+0000 7f338f7f9700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33880a1000 con 0x7f338809f550 2019-12-05T05:03:45.257 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.248+0000 7f338f7f9700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33880a8f80 con 0x7f338814df60 2019-12-05T05:03:45.257 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.248+0000 7f338dd96700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f338809f550 0x7f3388144630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:53650/0 (socket says 172.21.15.42:53650) 2019-12-05T05:03:45.258 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.248+0000 7f338dd96700 1 -- 172.21.15.42:0/3727892895 learned_addr learned my addr 172.21.15.42:0/3727892895 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:03:45.258 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.248+0000 7f338e597700 1 --2- 172.21.15.42:0/3727892895 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f338814df60 0x7f33881503a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:45.258 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.248+0000 7f338cd94700 1 -- 172.21.15.42:0/3727892895 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1792266748 0 0) 0x7f33880a1000 con 0x7f338809f550 2019-12-05T05:03:45.258 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.248+0000 7f338cd94700 1 -- 172.21.15.42:0/3727892895 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33740030b0 con 0x7f338809f550 2019-12-05T05:03:45.258 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.248+0000 7f338cd94700 1 -- 172.21.15.42:0/3727892895 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4204830495 0 0) 0x7f338807f080 con 0x7f33880a9a00 2019-12-05T05:03:45.258 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.248+0000 7f338cd94700 1 -- 172.21.15.42:0/3727892895 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33880a1000 con 0x7f33880a9a00 2019-12-05T05:03:45.258 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.248+0000 7f338cd94700 1 -- 172.21.15.42:0/3727892895 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1489585940 0 0) 0x7f33740030b0 con 0x7f338809f550 2019-12-05T05:03:45.258 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.248+0000 7f338e597700 1 -- 172.21.15.42:0/3727892895 >> v1:172.21.15.42:6789/0 conn(0x7f338809f550 legacy=0x7f3388144630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:45.259 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.248+0000 7f338e597700 1 -- 172.21.15.42:0/3727892895 >> v1:172.21.15.40:6789/0 conn(0x7f33880a9a00 legacy=0x7f338807e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:45.259 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.248+0000 7f338e597700 1 -- 172.21.15.42:0/3727892895 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f338809ee30 con 0x7f338814df60 2019-12-05T05:03:45.259 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.248+0000 7f338e597700 1 --2- 172.21.15.42:0/3727892895 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f338814df60 0x7f33881503a0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f3380008b10 tx=0x7f3384001950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:45.259 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.248+0000 7f338e597700 1 --2- 172.21.15.42:0/3727892895 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f338814df60 0x7f33881503a0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f3380008b10 tx=0x7f3384001950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:45.259 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.248+0000 7f338e597700 1 --2- 172.21.15.42:0/3727892895 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f338814df60 0x7f33881503a0 secure :-1 s=READY pgs=40 cs=0 l=1 rx=0x7f3380008b10 tx=0x7f3384001950).ready entity=mon.2 client_cookie=bbd2f306bd59a2e0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:45.259 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.248+0000 7f338e597700 1 --2- 172.21.15.42:0/3727892895 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f338814df60 0x7f33881503a0 secure :-1 s=THROTTLE_DONE pgs=40 cs=0 l=1 rx=0x7f3380008b10 tx=0x7f3384001950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:45.259 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.248+0000 7f338cd94700 1 -- 172.21.15.42:0/3727892895 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3384001750 con 0x7f338814df60 2019-12-05T05:03:45.260 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.248+0000 7f338e597700 1 --2- 172.21.15.42:0/3727892895 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f338814df60 0x7f33881503a0 secure :-1 s=THROTTLE_DONE pgs=40 cs=0 l=1 rx=0x7f3380008b10 tx=0x7f3384001950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:45.260 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.248+0000 7f338cd94700 1 -- 172.21.15.42:0/3727892895 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3384001560 con 0x7f338814df60 2019-12-05T05:03:45.260 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.248+0000 7f338e597700 1 --2- 172.21.15.42:0/3727892895 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f338814df60 0x7f33881503a0 secure :-1 s=THROTTLE_DONE pgs=40 cs=0 l=1 rx=0x7f3380008b10 tx=0x7f3384001950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:45.260 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.248+0000 7f338cd94700 1 -- 172.21.15.42:0/3727892895 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3380011ac0 con 0x7f338814df60 2019-12-05T05:03:45.260 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.248+0000 7f338f7f9700 1 -- 172.21.15.42:0/3727892895 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f338814df60 msgr2=0x7f33881503a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:45.260 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.248+0000 7f338f7f9700 1 --2- 172.21.15.42:0/3727892895 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f338814df60 0x7f33881503a0 secure :-1 s=READY pgs=40 cs=0 l=1 rx=0x7f3380008b10 tx=0x7f3384001950).stop 2019-12-05T05:03:45.260 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.248+0000 7f338f7f9700 1 -- 172.21.15.42:0/3727892895 shutdown_connections 2019-12-05T05:03:45.261 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.248+0000 7f338f7f9700 1 --2- 172.21.15.42:0/3727892895 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f338814df60 0x7f33881503a0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:45.261 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.248+0000 7f338f7f9700 1 -- 172.21.15.42:0/3727892895 shutdown_connections 2019-12-05T05:03:45.261 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.248+0000 7f338f7f9700 1 -- 172.21.15.42:0/3727892895 wait complete. 2019-12-05T05:03:45.261 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.248+0000 7f338f7f9700 1 -- 172.21.15.42:0/3727892895 >> 172.21.15.42:0/3727892895 conn(0x7f33880a9ef0 msgr2=0x7f33880aa320 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:45.264 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.252+0000 7f338f7f9700 1 Processor -- start 2019-12-05T05:03:45.264 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.252+0000 7f338f7f9700 1 -- start start 2019-12-05T05:03:45.265 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.252+0000 7f338f7f9700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f33881214e0 0x7f33881440e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:45.265 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.252+0000 7f338f7f9700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f3388121d20 0x7f338808dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:45.265 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.252+0000 7f338f7f9700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3388119240 con 0x7f3388118e10 2019-12-05T05:03:45.265 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.252+0000 7f338f7f9700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3388120ac0 con 0x7f3388121d20 2019-12-05T05:03:45.265 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.252+0000 7f338f7f9700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3388120c80 con 0x7f33881214e0 2019-12-05T05:03:45.265 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.252+0000 7f338dd96700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f33881214e0 0x7f33881440e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:45.267 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.252+0000 7f338d595700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f3388121d20 0x7f338808dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:45.267 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.252+0000 7f338dd96700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f33881214e0 0x7f33881440e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:54082/0 (socket says 172.21.15.42:54082) 2019-12-05T05:03:45.267 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.252+0000 7f338dd96700 1 -- 172.21.15.42:0/1553471706 learned_addr learned my addr 172.21.15.42:0/1553471706 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:03:45.267 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.256+0000 7f338dd96700 1 -- 172.21.15.42:0/1553471706 >> v2:172.21.15.40:3300/0 conn(0x7f3388121d20 msgr2=0x7f338808dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:45.268 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.256+0000 7f338dd96700 1 --2- 172.21.15.42:0/1553471706 >> v2:172.21.15.40:3300/0 conn(0x7f3388121d20 0x7f338808dea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:45.268 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.256+0000 7f338dd96700 1 -- 172.21.15.42:0/1553471706 >> v1:172.21.15.40:6789/0 conn(0x7f3388118e10 legacy=0x7f3388131de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:45.268 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.256+0000 7f338dd96700 1 -- 172.21.15.42:0/1553471706 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3388120e70 con 0x7f33881214e0 2019-12-05T05:03:45.268 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.256+0000 7f338dd96700 1 --2- 172.21.15.42:0/1553471706 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f33881214e0 0x7f33881440e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f3384000fa0 tx=0x7f33840109d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:45.268 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.256+0000 7f338dd96700 1 --2- 172.21.15.42:0/1553471706 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f33881214e0 0x7f33881440e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f3384000fa0 tx=0x7f33840109d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:45.268 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.256+0000 7f338dd96700 1 --2- 172.21.15.42:0/1553471706 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f33881214e0 0x7f33881440e0 secure :-1 s=READY pgs=41 cs=0 l=1 rx=0x7f3384000fa0 tx=0x7f33840109d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:45.269 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.256+0000 7f338dd96700 1 --2- 172.21.15.42:0/1553471706 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f33881214e0 0x7f33881440e0 secure :-1 s=THROTTLE_DONE pgs=41 cs=0 l=1 rx=0x7f3384000fa0 tx=0x7f33840109d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:45.269 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.256+0000 7f338dd96700 1 --2- 172.21.15.42:0/1553471706 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f33881214e0 0x7f33881440e0 secure :-1 s=THROTTLE_DONE pgs=41 cs=0 l=1 rx=0x7f3384000fa0 tx=0x7f33840109d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:45.269 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.256+0000 7f337e7fc700 1 -- 172.21.15.42:0/1553471706 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f338400c7d0 con 0x7f33881214e0 2019-12-05T05:03:45.269 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.256+0000 7f338dd96700 1 --2- 172.21.15.42:0/1553471706 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f33881214e0 0x7f33881440e0 secure :-1 s=THROTTLE_DONE pgs=41 cs=0 l=1 rx=0x7f3384000fa0 tx=0x7f33840109d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:45.269 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.256+0000 7f337e7fc700 1 -- 172.21.15.42:0/1553471706 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f338400c3f0 con 0x7f33881214e0 2019-12-05T05:03:45.269 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.256+0000 7f337e7fc700 1 -- 172.21.15.42:0/1553471706 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f338400cdb0 con 0x7f33881214e0 2019-12-05T05:03:45.269 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.256+0000 7f338f7f9700 1 -- 172.21.15.42:0/1553471706 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33881325e0 con 0x7f33881214e0 2019-12-05T05:03:45.270 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.256+0000 7f338f7f9700 1 -- 172.21.15.42:0/1553471706 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3388132940 con 0x7f33881214e0 2019-12-05T05:03:45.270 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.256+0000 7f338dd96700 1 --2- 172.21.15.42:0/1553471706 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f33881214e0 0x7f33881440e0 secure :-1 s=THROTTLE_DONE pgs=41 cs=0 l=1 rx=0x7f3384000fa0 tx=0x7f33840109d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:45.270 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.256+0000 7f337e7fc700 1 -- 172.21.15.42:0/1553471706 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f3384005450 con 0x7f33881214e0 2019-12-05T05:03:45.270 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.256+0000 7f338dd96700 1 --2- 172.21.15.42:0/1553471706 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f33881214e0 0x7f33881440e0 secure :-1 s=THROTTLE_DONE pgs=41 cs=0 l=1 rx=0x7f3384000fa0 tx=0x7f33840109d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:45.270 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.256+0000 7f337e7fc700 1 --2- 172.21.15.42:0/1553471706 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f337003ad70 0x7f337003d280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:45.270 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.256+0000 7f337e7fc700 1 -- 172.21.15.42:0/1553471706 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f338404ae90 con 0x7f33881214e0 2019-12-05T05:03:45.270 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.260+0000 7f338d595700 1 --2- 172.21.15.42:0/1553471706 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f337003ad70 0x7f337003d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:45.271 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.260+0000 7f338d595700 1 --2- 172.21.15.42:0/1553471706 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f337003ad70 0x7f337003d280 crc :-1 s=READY pgs=85 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:45.271 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.260+0000 7f338f7f9700 1 -- 172.21.15.42:0/1553471706 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3388121140 con 0x7f33881214e0 2019-12-05T05:03:45.276 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.272+0000 7f338dd96700 1 --2- 172.21.15.42:0/1553471706 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f33881214e0 0x7f33881440e0 secure :-1 s=THROTTLE_DONE pgs=41 cs=0 l=1 rx=0x7f3384000fa0 tx=0x7f33840109d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:45.276 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.272+0000 7f337e7fc700 1 -- 172.21.15.42:0/1553471706 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f3384031f10 con 0x7f33881214e0 2019-12-05T05:03:45.442 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.436+0000 7f338f7f9700 1 -- 172.21.15.42:0/1553471706 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f338805d110 con 0x7f33881214e0 2019-12-05T05:03:45.443 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.436+0000 7f338dd96700 1 --2- 172.21.15.42:0/1553471706 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f33881214e0 0x7f33881440e0 secure :-1 s=THROTTLE_DONE pgs=41 cs=0 l=1 rx=0x7f3384000fa0 tx=0x7f33840109d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:45.443 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.436+0000 7f337e7fc700 1 -- 172.21.15.42:0/1553471706 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v24) v1 ==== 92+0+10 (secure 0 0 0) 0x7f33840101e0 con 0x7f33881214e0 2019-12-05T05:03:45.445 INFO:teuthology.orchestra.run.smithi042.stdout:pg_num: 8 2019-12-05T05:03:45.450 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.444+0000 7f338f7f9700 1 -- 172.21.15.42:0/1553471706 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f337003ad70 msgr2=0x7f337003d280 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:45.450 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.444+0000 7f338f7f9700 1 --2- 172.21.15.42:0/1553471706 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f337003ad70 0x7f337003d280 crc :-1 s=READY pgs=85 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:45.450 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.444+0000 7f338f7f9700 1 -- 172.21.15.42:0/1553471706 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f33881214e0 msgr2=0x7f33881440e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:45.450 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.444+0000 7f338f7f9700 1 --2- 172.21.15.42:0/1553471706 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f33881214e0 0x7f33881440e0 secure :-1 s=READY pgs=41 cs=0 l=1 rx=0x7f3384000fa0 tx=0x7f33840109d0).stop 2019-12-05T05:03:45.450 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.444+0000 7f338f7f9700 1 -- 172.21.15.42:0/1553471706 shutdown_connections 2019-12-05T05:03:45.450 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.444+0000 7f338f7f9700 1 --2- 172.21.15.42:0/1553471706 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f337003ad70 0x7f337003d280 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:45.451 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.444+0000 7f338f7f9700 1 --2- 172.21.15.42:0/1553471706 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f33881214e0 0x7f33881440e0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:45.451 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.444+0000 7f338f7f9700 1 --2- 172.21.15.42:0/1553471706 >> v2:172.21.15.40:3300/0 conn(0x7f3388121d20 0x7f338808dea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:45.451 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.444+0000 7f338f7f9700 1 -- 172.21.15.42:0/1553471706 shutdown_connections 2019-12-05T05:03:45.451 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.444+0000 7f338f7f9700 1 -- 172.21.15.42:0/1553471706 wait complete. 2019-12-05T05:03:45.451 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.444+0000 7f338f7f9700 1 -- 172.21.15.42:0/1553471706 >> 172.21.15.42:0/1553471706 conn(0x7f33880a4970 msgr2=0x7f33880a51f0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:45.463 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2019-12-05T05:03:45.563 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.552+0000 7fcabbf47700 1 Processor -- start 2019-12-05T05:03:45.563 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.552+0000 7fcabbf47700 1 -- start start 2019-12-05T05:03:45.564 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.552+0000 7fcabbf47700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fcab40a9a00 0x7fcab407e880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:45.564 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.552+0000 7fcabbf47700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fcab407edd0 0x7fcab414ef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:45.564 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.552+0000 7fcabbf47700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcab40a1000 con 0x7fcab409f550 2019-12-05T05:03:45.564 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.552+0000 7fcabbf47700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcab40a8f80 con 0x7fcab40a9a00 2019-12-05T05:03:45.564 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.552+0000 7fcabbf47700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcab40a9220 con 0x7fcab407edd0 2019-12-05T05:03:45.565 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.552+0000 7fcabace5700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fcab407edd0 0x7fcab414ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:45.565 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.552+0000 7fcabace5700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fcab407edd0 0x7fcab414ef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:53628/0 (socket says 172.21.15.42:53628) 2019-12-05T05:03:45.565 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.552+0000 7fcaba4e4700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fcab409f550 0x7fcab4144630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:53662/0 (socket says 172.21.15.42:53662) 2019-12-05T05:03:45.565 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.552+0000 7fcabace5700 1 -- 172.21.15.42:0/1356704116 learned_addr learned my addr 172.21.15.42:0/1356704116 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:03:45.565 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.552+0000 7fcab9ce3700 1 --2- 172.21.15.42:0/1356704116 >> v2:172.21.15.40:3300/0 conn(0x7fcab40a9a00 0x7fcab407e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:45.565 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.556+0000 7fcabace5700 1 -- 172.21.15.42:0/1356704116 >> v2:172.21.15.40:3300/0 conn(0x7fcab40a9a00 msgr2=0x7fcab407e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:03:45.565 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.556+0000 7fcabace5700 1 --2- 172.21.15.42:0/1356704116 >> v2:172.21.15.40:3300/0 conn(0x7fcab40a9a00 0x7fcab407e880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:03:45.566 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.556+0000 7fcabace5700 1 -- 172.21.15.42:0/1356704116 >> v1:172.21.15.42:6789/0 conn(0x7fcab409f550 legacy=0x7fcab4144630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:45.566 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.556+0000 7fcab94e2700 1 -- 172.21.15.42:0/1356704116 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3537268645 0 0) 0x7fcab40a1000 con 0x7fcab409f550 2019-12-05T05:03:45.566 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.556+0000 7fcabace5700 1 -- 172.21.15.42:0/1356704116 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcab409ee30 con 0x7fcab407edd0 2019-12-05T05:03:45.566 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.556+0000 7fcabace5700 1 --2- 172.21.15.42:0/1356704116 >> v2:172.21.15.42:3300/0 conn(0x7fcab407edd0 0x7fcab414ef60 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fcaac002f30 tx=0x7fcaac00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:45.566 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.556+0000 7fcabace5700 1 --2- 172.21.15.42:0/1356704116 >> v2:172.21.15.42:3300/0 conn(0x7fcab407edd0 0x7fcab414ef60 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fcaac002f30 tx=0x7fcaac00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:45.566 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.556+0000 7fcabace5700 1 --2- 172.21.15.42:0/1356704116 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcab407edd0 0x7fcab414ef60 secure :-1 s=READY pgs=156 cs=0 l=1 rx=0x7fcaac002f30 tx=0x7fcaac00fd30).ready entity=mon.1 client_cookie=d420f4402e86c494 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:45.566 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.556+0000 7fcabace5700 1 --2- 172.21.15.42:0/1356704116 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcab407edd0 0x7fcab414ef60 secure :-1 s=THROTTLE_DONE pgs=156 cs=0 l=1 rx=0x7fcaac002f30 tx=0x7fcaac00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:45.567 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.556+0000 7fcab94e2700 1 -- 172.21.15.42:0/1356704116 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcaac008c90 con 0x7fcab407edd0 2019-12-05T05:03:45.567 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.556+0000 7fcabace5700 1 --2- 172.21.15.42:0/1356704116 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcab407edd0 0x7fcab414ef60 secure :-1 s=THROTTLE_DONE pgs=156 cs=0 l=1 rx=0x7fcaac002f30 tx=0x7fcaac00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:45.567 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.556+0000 7fcabace5700 1 --2- 172.21.15.42:0/1356704116 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcab407edd0 0x7fcab414ef60 secure :-1 s=THROTTLE_DONE pgs=156 cs=0 l=1 rx=0x7fcaac002f30 tx=0x7fcaac00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:45.567 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.556+0000 7fcab94e2700 1 -- 172.21.15.42:0/1356704116 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcaac0088b0 con 0x7fcab407edd0 2019-12-05T05:03:45.567 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.556+0000 7fcab94e2700 1 -- 172.21.15.42:0/1356704116 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcaac011ad0 con 0x7fcab407edd0 2019-12-05T05:03:45.567 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.556+0000 7fcabbf47700 1 -- 172.21.15.42:0/1356704116 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcab407edd0 msgr2=0x7fcab414ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:45.568 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.556+0000 7fcabbf47700 1 --2- 172.21.15.42:0/1356704116 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcab407edd0 0x7fcab414ef60 secure :-1 s=READY pgs=156 cs=0 l=1 rx=0x7fcaac002f30 tx=0x7fcaac00fd30).stop 2019-12-05T05:03:45.568 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.556+0000 7fcabbf47700 1 -- 172.21.15.42:0/1356704116 shutdown_connections 2019-12-05T05:03:45.568 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.556+0000 7fcabbf47700 1 --2- 172.21.15.42:0/1356704116 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcab407edd0 0x7fcab414ef60 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:45.568 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.556+0000 7fcabbf47700 1 --2- 172.21.15.42:0/1356704116 >> v2:172.21.15.40:3300/0 conn(0x7fcab40a9a00 0x7fcab407e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:03:45.568 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.556+0000 7fcabbf47700 1 -- 172.21.15.42:0/1356704116 shutdown_connections 2019-12-05T05:03:45.568 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.556+0000 7fcabbf47700 1 -- 172.21.15.42:0/1356704116 wait complete. 2019-12-05T05:03:45.568 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.556+0000 7fcabbf47700 1 -- 172.21.15.42:0/1356704116 >> 172.21.15.42:0/1356704116 conn(0x7fcab40a9ef0 msgr2=0x7fcab40aa320 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:45.569 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.556+0000 7fcabbf47700 1 Processor -- start 2019-12-05T05:03:45.569 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.556+0000 7fcabbf47700 1 -- start start 2019-12-05T05:03:45.573 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.560+0000 7fcabbf47700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fcab40a57c0 0x7fcab4133530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:45.573 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.560+0000 7fcabbf47700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fcab4133a80 0x7fcab4130210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:45.573 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.560+0000 7fcabbf47700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcab40a5bf0 con 0x7fcab4130760 2019-12-05T05:03:45.573 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.560+0000 7fcabbf47700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcab41340b0 con 0x7fcab4133a80 2019-12-05T05:03:45.574 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.560+0000 7fcabbf47700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcab4134350 con 0x7fcab40a57c0 2019-12-05T05:03:45.574 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.560+0000 7fcaba4e4700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fcab40a57c0 0x7fcab4133530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:45.574 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.560+0000 7fcaba4e4700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fcab40a57c0 0x7fcab4133530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:53632/0 (socket says 172.21.15.42:53632) 2019-12-05T05:03:45.574 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.560+0000 7fcaba4e4700 1 -- 172.21.15.42:0/820014893 learned_addr learned my addr 172.21.15.42:0/820014893 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:03:45.574 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.560+0000 7fcab9ce3700 1 --2- 172.21.15.42:0/820014893 >> v2:172.21.15.40:3300/0 conn(0x7fcab4133a80 0x7fcab4130210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:45.574 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.560+0000 7fcaba4e4700 1 -- 172.21.15.42:0/820014893 >> v2:172.21.15.40:3300/0 conn(0x7fcab4133a80 msgr2=0x7fcab4130210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:45.574 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.560+0000 7fcaba4e4700 1 --2- 172.21.15.42:0/820014893 >> v2:172.21.15.40:3300/0 conn(0x7fcab4133a80 0x7fcab4130210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:45.575 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.560+0000 7fcaba4e4700 1 -- 172.21.15.42:0/820014893 >> v1:172.21.15.40:6789/0 conn(0x7fcab4130760 legacy=0x7fcab4125260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:45.575 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.560+0000 7fcaba4e4700 1 -- 172.21.15.42:0/820014893 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcab4130b90 con 0x7fcab40a57c0 2019-12-05T05:03:45.575 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.560+0000 7fcab9ce3700 1 --2- 172.21.15.42:0/820014893 >> v2:172.21.15.40:3300/0 conn(0x7fcab4133a80 0x7fcab4130210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:03:45.575 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.560+0000 7fcaba4e4700 1 --2- 172.21.15.42:0/820014893 >> v2:172.21.15.42:3300/0 conn(0x7fcab40a57c0 0x7fcab4133530 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fcaa4000b80 tx=0x7fcaa4010940).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:45.575 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.560+0000 7fcaba4e4700 1 --2- 172.21.15.42:0/820014893 >> v2:172.21.15.42:3300/0 conn(0x7fcab40a57c0 0x7fcab4133530 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fcaa4000b80 tx=0x7fcaa4010940).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:45.575 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.560+0000 7fcaba4e4700 1 --2- 172.21.15.42:0/820014893 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcab40a57c0 0x7fcab4133530 secure :-1 s=READY pgs=157 cs=0 l=1 rx=0x7fcaa4000b80 tx=0x7fcaa4010940).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:45.575 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.560+0000 7fcaba4e4700 1 --2- 172.21.15.42:0/820014893 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcab40a57c0 0x7fcab4133530 secure :-1 s=THROTTLE_DONE pgs=157 cs=0 l=1 rx=0x7fcaa4000b80 tx=0x7fcaa4010940).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:45.576 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.560+0000 7fcaba4e4700 1 --2- 172.21.15.42:0/820014893 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcab40a57c0 0x7fcab4133530 secure :-1 s=THROTTLE_DONE pgs=157 cs=0 l=1 rx=0x7fcaa4000b80 tx=0x7fcaa4010940).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:45.576 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.560+0000 7fcaba4e4700 1 --2- 172.21.15.42:0/820014893 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcab40a57c0 0x7fcab4133530 secure :-1 s=THROTTLE_DONE pgs=157 cs=0 l=1 rx=0x7fcaa4000b80 tx=0x7fcaa4010940).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:45.576 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.560+0000 7fcaaaffd700 1 -- 172.21.15.42:0/820014893 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcaa4011630 con 0x7fcab40a57c0 2019-12-05T05:03:45.576 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.560+0000 7fcaaaffd700 1 -- 172.21.15.42:0/820014893 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcaa4011250 con 0x7fcab40a57c0 2019-12-05T05:03:45.576 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.560+0000 7fcabbf47700 1 -- 172.21.15.42:0/820014893 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcab4125a60 con 0x7fcab40a57c0 2019-12-05T05:03:45.576 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.560+0000 7fcaaaffd700 1 -- 172.21.15.42:0/820014893 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcaa4011c10 con 0x7fcab40a57c0 2019-12-05T05:03:45.577 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.560+0000 7fcabbf47700 1 -- 172.21.15.42:0/820014893 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcab4125f90 con 0x7fcab40a57c0 2019-12-05T05:03:45.577 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.560+0000 7fcaba4e4700 1 --2- 172.21.15.42:0/820014893 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcab40a57c0 0x7fcab4133530 secure :-1 s=THROTTLE_DONE pgs=157 cs=0 l=1 rx=0x7fcaa4000b80 tx=0x7fcaa4010940).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:45.577 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.560+0000 7fcaaaffd700 1 -- 172.21.15.42:0/820014893 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fcaa4006460 con 0x7fcab40a57c0 2019-12-05T05:03:45.577 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.564+0000 7fcaba4e4700 1 --2- 172.21.15.42:0/820014893 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcab40a57c0 0x7fcab4133530 secure :-1 s=THROTTLE_DONE pgs=157 cs=0 l=1 rx=0x7fcaa4000b80 tx=0x7fcaa4010940).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:45.577 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.564+0000 7fcaaaffd700 1 --2- 172.21.15.42:0/820014893 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fca9403ae70 0x7fca9403d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:45.577 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.564+0000 7fcaaaffd700 1 -- 172.21.15.42:0/820014893 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6031+0+0 (secure 0 0 0) 0x7fcaa404ae70 con 0x7fcab40a57c0 2019-12-05T05:03:45.578 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.564+0000 7fcab9ce3700 1 --2- 172.21.15.42:0/820014893 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fca9403ae70 0x7fca9403d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:45.578 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.564+0000 7fcab9ce3700 1 --2- 172.21.15.42:0/820014893 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fca9403ae70 0x7fca9403d380 crc :-1 s=READY pgs=86 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:45.578 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.568+0000 7fcabbf47700 1 -- 172.21.15.42:0/820014893 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcab4133eb0 con 0x7fcab40a57c0 2019-12-05T05:03:45.584 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.580+0000 7fcaba4e4700 1 --2- 172.21.15.42:0/820014893 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcab40a57c0 0x7fcab4133530 secure :-1 s=THROTTLE_DONE pgs=157 cs=0 l=1 rx=0x7fcaa4000b80 tx=0x7fcaa4010940).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:45.584 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.580+0000 7fcaaaffd700 1 -- 172.21.15.42:0/820014893 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fcaa4031ef0 con 0x7fcab40a57c0 2019-12-05T05:03:45.744 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.740+0000 7fcabbf47700 1 -- 172.21.15.42:0/820014893 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7fcab405d110 con 0x7fcab40a57c0 2019-12-05T05:03:45.745 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.740+0000 7fcaba4e4700 1 --2- 172.21.15.42:0/820014893 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcab40a57c0 0x7fcab4133530 secure :-1 s=THROTTLE_DONE pgs=157 cs=0 l=1 rx=0x7fcaa4000b80 tx=0x7fcaa4010940).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:45.745 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.740+0000 7fcaaaffd700 1 -- 172.21.15.42:0/820014893 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v24) v1 ==== 104+0+10 (secure 0 0 0) 0x7fcaa4032150 con 0x7fcab40a57c0 2019-12-05T05:03:45.746 INFO:teuthology.orchestra.run.smithi042.stdout:pg_num: 8 2019-12-05T05:03:45.748 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.744+0000 7fcabbf47700 1 -- 172.21.15.42:0/820014893 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fca9403ae70 msgr2=0x7fca9403d380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:45.748 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.744+0000 7fcabbf47700 1 --2- 172.21.15.42:0/820014893 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fca9403ae70 0x7fca9403d380 crc :-1 s=READY pgs=86 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:45.749 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.744+0000 7fcabbf47700 1 -- 172.21.15.42:0/820014893 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcab40a57c0 msgr2=0x7fcab4133530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:45.749 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.744+0000 7fcabbf47700 1 --2- 172.21.15.42:0/820014893 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcab40a57c0 0x7fcab4133530 secure :-1 s=READY pgs=157 cs=0 l=1 rx=0x7fcaa4000b80 tx=0x7fcaa4010940).stop 2019-12-05T05:03:45.749 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.744+0000 7fcabbf47700 1 -- 172.21.15.42:0/820014893 shutdown_connections 2019-12-05T05:03:45.749 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.744+0000 7fcabbf47700 1 --2- 172.21.15.42:0/820014893 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fca9403ae70 0x7fca9403d380 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:45.749 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.744+0000 7fcabbf47700 1 --2- 172.21.15.42:0/820014893 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcab40a57c0 0x7fcab4133530 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:45.749 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.744+0000 7fcabbf47700 1 --2- 172.21.15.42:0/820014893 >> v2:172.21.15.40:3300/0 conn(0x7fcab4133a80 0x7fcab4130210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:45.749 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.744+0000 7fcabbf47700 1 -- 172.21.15.42:0/820014893 shutdown_connections 2019-12-05T05:03:45.750 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.744+0000 7fcabbf47700 1 -- 172.21.15.42:0/820014893 wait complete. 2019-12-05T05:03:45.750 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.744+0000 7fcabbf47700 1 -- 172.21.15.42:0/820014893 >> 172.21.15.42:0/820014893 conn(0x7fcab4059b30 msgr2=0x7fcab40a71a0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:45.758 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2019-12-05T05:03:45.859 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.844+0000 7f34c08a1700 1 Processor -- start 2019-12-05T05:03:45.859 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.844+0000 7f34c08a1700 1 -- start start 2019-12-05T05:03:45.860 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.848+0000 7f34c08a1700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f34bc0a9a00 0x7f34bc07e880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:45.860 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.848+0000 7f34c08a1700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f34bc07edd0 0x7f34bc14ef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:45.860 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.848+0000 7f34c08a1700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34bc0a1000 con 0x7f34bc09f550 2019-12-05T05:03:45.860 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.848+0000 7f34c08a1700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34bc0a8f80 con 0x7f34bc07edd0 2019-12-05T05:03:45.860 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.848+0000 7f34c08a1700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34bc0a9220 con 0x7f34bc0a9a00 2019-12-05T05:03:45.860 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.848+0000 7f34bad9d700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f34bc09f550 0x7f34bc144630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:53676/0 (socket says 172.21.15.42:53676) 2019-12-05T05:03:45.861 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.848+0000 7f34bad9d700 1 -- 172.21.15.42:0/1251749110 learned_addr learned my addr 172.21.15.42:0/1251749110 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:03:45.861 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.848+0000 7f34ba59c700 1 --2- 172.21.15.42:0/1251749110 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f34bc0a9a00 0x7f34bc07e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:45.861 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.848+0000 7f34bb59e700 1 --2- 172.21.15.42:0/1251749110 >> v2:172.21.15.40:3300/0 conn(0x7f34bc07edd0 0x7f34bc14ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:45.861 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.848+0000 7f34b9d9b700 1 -- 172.21.15.42:0/1251749110 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 452873154 0 0) 0x7f34bc0a1000 con 0x7f34bc09f550 2019-12-05T05:03:45.861 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.848+0000 7f34b9d9b700 1 -- 172.21.15.42:0/1251749110 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34a40030b0 con 0x7f34bc09f550 2019-12-05T05:03:45.861 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.848+0000 7f34b9d9b700 1 -- 172.21.15.42:0/1251749110 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1004542751 0 0) 0x7f34a40030b0 con 0x7f34bc09f550 2019-12-05T05:03:45.862 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.848+0000 7f34b9d9b700 1 -- 172.21.15.42:0/1251749110 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f34bc0a9a00 msgr2=0x7f34bc07e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:03:45.862 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.848+0000 7f34b9d9b700 1 --2- 172.21.15.42:0/1251749110 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f34bc0a9a00 0x7f34bc07e880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:03:45.862 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.848+0000 7f34b9d9b700 1 -- 172.21.15.42:0/1251749110 >> v2:172.21.15.40:3300/0 conn(0x7f34bc07edd0 msgr2=0x7f34bc14ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:03:45.862 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.848+0000 7f34b9d9b700 1 --2- 172.21.15.42:0/1251749110 >> v2:172.21.15.40:3300/0 conn(0x7f34bc07edd0 0x7f34bc14ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:03:45.862 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.848+0000 7f34b9d9b700 1 -- 172.21.15.42:0/1251749110 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34bc09ee30 con 0x7f34bc09f550 2019-12-05T05:03:45.863 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.848+0000 7f34b9d9b700 1 -- 172.21.15.42:0/1251749110 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7f34ac001750 con 0x7f34bc09f550 2019-12-05T05:03:45.863 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.848+0000 7f34b9d9b700 1 -- 172.21.15.42:0/1251749110 >> v1:172.21.15.42:6789/0 conn(0x7f34bc09f550 legacy=0x7f34bc144630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:45.863 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.848+0000 7f34b9d9b700 1 --2- 172.21.15.42:0/1251749110 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f34a4003710 0x7f34a4003b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:45.863 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.848+0000 7f34b9d9b700 1 --2- 172.21.15.42:0/1251749110 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f34a4004420 0x7f34a4004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:45.863 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.848+0000 7f34b9d9b700 1 --2- 172.21.15.42:0/1251749110 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f34a4004da0 0x7f34a404f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:45.863 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.848+0000 7f34b9d9b700 1 -- 172.21.15.42:0/1251749110 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34bc0a8f80 con 0x7f34a4003710 2019-12-05T05:03:45.863 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.848+0000 7f34b9d9b700 1 -- 172.21.15.42:0/1251749110 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34bc0a9220 con 0x7f34a4004420 2019-12-05T05:03:45.864 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.848+0000 7f34b9d9b700 1 -- 172.21.15.42:0/1251749110 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34a40051d0 con 0x7f34a4004da0 2019-12-05T05:03:45.864 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.848+0000 7f34ba59c700 1 --2- 172.21.15.42:0/1251749110 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f34a4004420 0x7f34a4004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:45.865 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.848+0000 7f34bad9d700 1 --2- 172.21.15.42:0/1251749110 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f34a4003710 0x7f34a4003b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:45.865 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.848+0000 7f34bb59e700 1 --2- 172.21.15.42:0/1251749110 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f34a4004da0 0x7f34a404f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:45.865 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.848+0000 7f34ba59c700 1 -- 172.21.15.42:0/1251749110 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f34a4004da0 msgr2=0x7f34a404f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:03:45.865 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.848+0000 7f34ba59c700 1 --2- 172.21.15.42:0/1251749110 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f34a4004da0 0x7f34a404f680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:03:45.865 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.848+0000 7f34ba59c700 1 -- 172.21.15.42:0/1251749110 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f34a4003710 msgr2=0x7f34a4003b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:03:45.865 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.848+0000 7f34ba59c700 1 --2- 172.21.15.42:0/1251749110 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f34a4003710 0x7f34a4003b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:03:45.865 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.848+0000 7f34ba59c700 1 -- 172.21.15.42:0/1251749110 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34a404fc90 con 0x7f34a4004420 2019-12-05T05:03:45.866 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.848+0000 7f34ba59c700 1 --2- 172.21.15.42:0/1251749110 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f34a4004420 0x7f34a4004850 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f34b0002f30 tx=0x7f34b000ffd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:45.866 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.848+0000 7f34ba59c700 1 --2- 172.21.15.42:0/1251749110 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f34a4004420 0x7f34a4004850 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f34b0002f30 tx=0x7f34b000ffd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:45.866 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.848+0000 7f34ba59c700 1 --2- 172.21.15.42:0/1251749110 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f34a4004420 0x7f34a4004850 secure :-1 s=READY pgs=159 cs=0 l=1 rx=0x7f34b0002f30 tx=0x7f34b000ffd0).ready entity=mon.1 client_cookie=a5704573c600e4c6 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:45.866 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.848+0000 7f34ba59c700 1 --2- 172.21.15.42:0/1251749110 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f34a4004420 0x7f34a4004850 secure :-1 s=THROTTLE_DONE pgs=159 cs=0 l=1 rx=0x7f34b0002f30 tx=0x7f34b000ffd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:45.866 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.848+0000 7f34b9d9b700 1 -- 172.21.15.42:0/1251749110 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f34b0010d00 con 0x7f34a4004420 2019-12-05T05:03:45.866 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.848+0000 7f34ba59c700 1 --2- 172.21.15.42:0/1251749110 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f34a4004420 0x7f34a4004850 secure :-1 s=THROTTLE_DONE pgs=159 cs=0 l=1 rx=0x7f34b0002f30 tx=0x7f34b000ffd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:45.867 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.848+0000 7f34b9d9b700 1 -- 172.21.15.42:0/1251749110 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34b0010920 con 0x7f34a4004420 2019-12-05T05:03:45.867 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.848+0000 7f34ba59c700 1 --2- 172.21.15.42:0/1251749110 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f34a4004420 0x7f34a4004850 secure :-1 s=THROTTLE_DONE pgs=159 cs=0 l=1 rx=0x7f34b0002f30 tx=0x7f34b000ffd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:45.867 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.848+0000 7f34c08a1700 1 -- 172.21.15.42:0/1251749110 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f34a4004420 msgr2=0x7f34a4004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:45.868 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.848+0000 7f34b9d9b700 1 -- 172.21.15.42:0/1251749110 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f34b0008bb0 con 0x7f34a4004420 2019-12-05T05:03:45.868 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.848+0000 7f34c08a1700 1 --2- 172.21.15.42:0/1251749110 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f34a4004420 0x7f34a4004850 secure :-1 s=READY pgs=159 cs=0 l=1 rx=0x7f34b0002f30 tx=0x7f34b000ffd0).stop 2019-12-05T05:03:45.868 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.848+0000 7f34bb59e700 1 -- 172.21.15.42:0/1251749110 reap_dead start 2019-12-05T05:03:45.868 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.848+0000 7f34c08a1700 1 -- 172.21.15.42:0/1251749110 shutdown_connections 2019-12-05T05:03:45.868 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.848+0000 7f34c08a1700 1 -- 172.21.15.42:0/1251749110 shutdown_connections 2019-12-05T05:03:45.868 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.852+0000 7f34c08a1700 1 -- 172.21.15.42:0/1251749110 wait complete. 2019-12-05T05:03:45.869 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.852+0000 7f34c08a1700 1 -- 172.21.15.42:0/1251749110 >> 172.21.15.42:0/1251749110 conn(0x7f34bc0a9ef0 msgr2=0x7f34bc0aa320 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:45.869 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.852+0000 7f34c08a1700 1 Processor -- start 2019-12-05T05:03:45.869 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.852+0000 7f34c08a1700 1 -- start start 2019-12-05T05:03:45.869 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.852+0000 7f34c08a1700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f34bc123a80 0x7f34bc08e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:45.869 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.852+0000 7f34c08a1700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34bc11b9e0 con 0x7f34bc07dcd0 2019-12-05T05:03:45.869 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.852+0000 7f34c08a1700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34bc11c380 con 0x7f34bc11bf50 2019-12-05T05:03:45.869 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.852+0000 7f34c08a1700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34bc08e460 con 0x7f34bc123a80 2019-12-05T05:03:45.870 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.852+0000 7f34ba59c700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f34bc11bf50 0x7f34bc07d4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:53690/0 (socket says 172.21.15.42:53690) 2019-12-05T05:03:45.870 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.852+0000 7f34ba59c700 1 -- 172.21.15.42:0/27579852 learned_addr learned my addr 172.21.15.42:0/27579852 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:03:45.870 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.852+0000 7f34bad9d700 1 --2- 172.21.15.42:0/27579852 >> v2:172.21.15.40:3300/0 conn(0x7f34bc123a80 0x7f34bc08e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:45.870 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.852+0000 7f34ab7fe700 1 -- 172.21.15.42:0/27579852 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1488597036 0 0) 0x7f34bc11c380 con 0x7f34bc11bf50 2019-12-05T05:03:45.870 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.852+0000 7f34ab7fe700 1 -- 172.21.15.42:0/27579852 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34a00030b0 con 0x7f34bc11bf50 2019-12-05T05:03:45.875 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.852+0000 7f34bad9d700 1 -- 172.21.15.42:0/27579852 >> v1:172.21.15.42:6789/0 conn(0x7f34bc11bf50 legacy=0x7f34bc07d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:45.876 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.852+0000 7f34bad9d700 1 -- 172.21.15.42:0/27579852 >> v1:172.21.15.40:6789/0 conn(0x7f34bc07dcd0 legacy=0x7f34bc137dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:45.876 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.852+0000 7f34bad9d700 1 -- 172.21.15.42:0/27579852 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34bc0a3c20 con 0x7f34bc123a80 2019-12-05T05:03:45.877 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.852+0000 7f34bad9d700 1 --2- 172.21.15.42:0/27579852 >> v2:172.21.15.40:3300/0 conn(0x7f34bc123a80 0x7f34bc08e9d0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f34ac001cd0 tx=0x7f34ac010ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:45.877 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.856+0000 7f34bad9d700 1 --2- 172.21.15.42:0/27579852 >> v2:172.21.15.40:3300/0 conn(0x7f34bc123a80 0x7f34bc08e9d0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f34ac001cd0 tx=0x7f34ac010ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:45.877 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.856+0000 7f34bad9d700 1 --2- 172.21.15.42:0/27579852 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f34bc123a80 0x7f34bc08e9d0 secure :-1 s=READY pgs=130 cs=0 l=1 rx=0x7f34ac001cd0 tx=0x7f34ac010ae0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:45.877 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.856+0000 7f34bad9d700 1 --2- 172.21.15.42:0/27579852 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f34bc123a80 0x7f34bc08e9d0 secure :-1 s=THROTTLE_DONE pgs=130 cs=0 l=1 rx=0x7f34ac001cd0 tx=0x7f34ac010ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:45.877 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.856+0000 7f34ab7fe700 1 -- 172.21.15.42:0/27579852 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f34ac00e3b0 con 0x7f34bc123a80 2019-12-05T05:03:45.877 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.856+0000 7f34bad9d700 1 --2- 172.21.15.42:0/27579852 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f34bc123a80 0x7f34bc08e9d0 secure :-1 s=THROTTLE_DONE pgs=130 cs=0 l=1 rx=0x7f34ac001cd0 tx=0x7f34ac010ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:45.877 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.856+0000 7f34bad9d700 1 --2- 172.21.15.42:0/27579852 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f34bc123a80 0x7f34bc08e9d0 secure :-1 s=THROTTLE_DONE pgs=130 cs=0 l=1 rx=0x7f34ac001cd0 tx=0x7f34ac010ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:45.878 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.856+0000 7f34ab7fe700 1 -- 172.21.15.42:0/27579852 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34ac00f070 con 0x7f34bc123a80 2019-12-05T05:03:45.878 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.856+0000 7f34c08a1700 1 -- 172.21.15.42:0/27579852 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34bc07e100 con 0x7f34bc123a80 2019-12-05T05:03:45.878 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.856+0000 7f34ab7fe700 1 -- 172.21.15.42:0/27579852 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f34ac00e9d0 con 0x7f34bc123a80 2019-12-05T05:03:45.879 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.856+0000 7f34c08a1700 1 -- 172.21.15.42:0/27579852 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34bc144170 con 0x7f34bc123a80 2019-12-05T05:03:45.879 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.856+0000 7f34bad9d700 1 --2- 172.21.15.42:0/27579852 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f34bc123a80 0x7f34bc08e9d0 secure :-1 s=THROTTLE_DONE pgs=130 cs=0 l=1 rx=0x7f34ac001cd0 tx=0x7f34ac010ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:45.879 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.856+0000 7f34bad9d700 1 --2- 172.21.15.42:0/27579852 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f34bc123a80 0x7f34bc08e9d0 secure :-1 s=THROTTLE_DONE pgs=130 cs=0 l=1 rx=0x7f34ac001cd0 tx=0x7f34ac010ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:45.879 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.856+0000 7f34ab7fe700 1 -- 172.21.15.42:0/27579852 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f34ac00ece0 con 0x7f34bc123a80 2019-12-05T05:03:45.880 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.856+0000 7f34ab7fe700 1 --2- 172.21.15.42:0/27579852 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f34a003b250 0x7f34a003d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:45.880 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.856+0000 7f34ab7fe700 1 -- 172.21.15.42:0/27579852 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f34ac04b120 con 0x7f34bc123a80 2019-12-05T05:03:45.884 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.856+0000 7f34ba59c700 1 --2- 172.21.15.42:0/27579852 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f34a003b250 0x7f34a003d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:45.884 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.856+0000 7f34ba59c700 1 --2- 172.21.15.42:0/27579852 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f34a003b250 0x7f34a003d760 crc :-1 s=READY pgs=87 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:45.885 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.860+0000 7f34c08a1700 1 -- 172.21.15.42:0/27579852 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34bc05d110 con 0x7f34bc123a80 2019-12-05T05:03:45.885 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.868+0000 7f34bad9d700 1 --2- 172.21.15.42:0/27579852 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f34bc123a80 0x7f34bc08e9d0 secure :-1 s=THROTTLE_DONE pgs=130 cs=0 l=1 rx=0x7f34ac001cd0 tx=0x7f34ac010ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:45.885 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:45.868+0000 7f34ab7fe700 1 -- 172.21.15.42:0/27579852 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f34ac032160 con 0x7f34bc123a80 2019-12-05T05:03:46.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.028+0000 7f34c08a1700 1 -- 172.21.15.42:0/27579852 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7f34bc0a4620 con 0x7f34bc123a80 2019-12-05T05:03:46.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.032+0000 7f34bad9d700 1 --2- 172.21.15.42:0/27579852 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f34bc123a80 0x7f34bc08e9d0 secure :-1 s=THROTTLE_DONE pgs=130 cs=0 l=1 rx=0x7f34ac001cd0 tx=0x7f34ac010ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:46.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.032+0000 7f34ab7fe700 1 -- 172.21.15.42:0/27579852 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v24) v1 ==== 100+0+10 (secure 0 0 0) 0x7f34ac0323c0 con 0x7f34bc123a80 2019-12-05T05:03:46.038 INFO:teuthology.orchestra.run.smithi042.stdout:pg_num: 8 2019-12-05T05:03:46.043 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.036+0000 7f34c08a1700 1 -- 172.21.15.42:0/27579852 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f34a003b250 msgr2=0x7f34a003d760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:46.043 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.036+0000 7f34c08a1700 1 --2- 172.21.15.42:0/27579852 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f34a003b250 0x7f34a003d760 crc :-1 s=READY pgs=87 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:46.043 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.036+0000 7f34c08a1700 1 -- 172.21.15.42:0/27579852 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f34bc123a80 msgr2=0x7f34bc08e9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:46.043 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.036+0000 7f34c08a1700 1 --2- 172.21.15.42:0/27579852 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f34bc123a80 0x7f34bc08e9d0 secure :-1 s=READY pgs=130 cs=0 l=1 rx=0x7f34ac001cd0 tx=0x7f34ac010ae0).stop 2019-12-05T05:03:46.043 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.036+0000 7f34c08a1700 1 -- 172.21.15.42:0/27579852 shutdown_connections 2019-12-05T05:03:46.043 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.036+0000 7f34c08a1700 1 --2- 172.21.15.42:0/27579852 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f34a003b250 0x7f34a003d760 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:46.044 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.036+0000 7f34c08a1700 1 --2- 172.21.15.42:0/27579852 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f34bc123a80 0x7f34bc08e9d0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:46.044 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.036+0000 7f34c08a1700 1 -- 172.21.15.42:0/27579852 shutdown_connections 2019-12-05T05:03:46.044 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.036+0000 7f34c08a1700 1 -- 172.21.15.42:0/27579852 wait complete. 2019-12-05T05:03:46.044 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.036+0000 7f34c08a1700 1 -- 172.21.15.42:0/27579852 >> 172.21.15.42:0/27579852 conn(0x7f34bc0a4d70 msgr2=0x7f34bc059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:46.056 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:03:46.154 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.144+0000 7fb89a548700 1 Processor -- start 2019-12-05T05:03:46.154 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.144+0000 7fb89a548700 1 -- start start 2019-12-05T05:03:46.155 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.144+0000 7fb89a548700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb8940a7740 0x7fb89408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:46.155 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.144+0000 7fb89a548700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb89409fae0 0x7fb89407eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:46.155 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.144+0000 7fb89a548700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8940a0710 con 0x7fb89407e720 2019-12-05T05:03:46.155 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.144+0000 7fb89a548700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8940a0b00 con 0x7fb8940a7740 2019-12-05T05:03:46.155 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.144+0000 7fb89a548700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb89408e5b0 con 0x7fb89409fae0 2019-12-05T05:03:46.156 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.144+0000 7fb893fff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fb89407e720 0x7fb89409f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:53696/0 (socket says 172.21.15.42:53696) 2019-12-05T05:03:46.156 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.144+0000 7fb893fff700 1 -- 172.21.15.42:0/1976315661 learned_addr learned my addr 172.21.15.42:0/1976315661 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:03:46.156 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.144+0000 7fb898ae5700 1 --2- 172.21.15.42:0/1976315661 >> v2:172.21.15.40:3300/0 conn(0x7fb8940a7740 0x7fb89408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:46.156 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.144+0000 7fb8992e6700 1 --2- 172.21.15.42:0/1976315661 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb89409fae0 0x7fb89407eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:46.156 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.144+0000 7fb8937fe700 1 -- 172.21.15.42:0/1976315661 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1632786065 0 0) 0x7fb8940a0710 con 0x7fb89407e720 2019-12-05T05:03:46.156 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.144+0000 7fb8937fe700 1 -- 172.21.15.42:0/1976315661 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8840030b0 con 0x7fb89407e720 2019-12-05T05:03:46.157 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.144+0000 7fb898ae5700 1 -- 172.21.15.42:0/1976315661 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb89409fae0 msgr2=0x7fb89407eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:03:46.157 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.144+0000 7fb898ae5700 1 --2- 172.21.15.42:0/1976315661 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb89409fae0 0x7fb89407eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:03:46.157 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.144+0000 7fb898ae5700 1 -- 172.21.15.42:0/1976315661 >> v1:172.21.15.42:6789/0 conn(0x7fb89407e720 legacy=0x7fb89409f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:46.157 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.144+0000 7fb898ae5700 1 -- 172.21.15.42:0/1976315661 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb89407f0a0 con 0x7fb8940a7740 2019-12-05T05:03:46.157 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.144+0000 7fb898ae5700 1 --2- 172.21.15.42:0/1976315661 >> v2:172.21.15.40:3300/0 conn(0x7fb8940a7740 0x7fb89408df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fb888008790 tx=0x7fb88800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:46.157 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.144+0000 7fb898ae5700 1 --2- 172.21.15.42:0/1976315661 >> v2:172.21.15.40:3300/0 conn(0x7fb8940a7740 0x7fb89408df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fb888008790 tx=0x7fb88800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:46.157 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.144+0000 7fb898ae5700 1 --2- 172.21.15.42:0/1976315661 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb8940a7740 0x7fb89408df80 secure :-1 s=READY pgs=131 cs=0 l=1 rx=0x7fb888008790 tx=0x7fb88800fc70).ready entity=mon.0 client_cookie=36e3c8c9ed9823fe server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:46.158 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.144+0000 7fb898ae5700 1 --2- 172.21.15.42:0/1976315661 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb8940a7740 0x7fb89408df80 secure :-1 s=THROTTLE_DONE pgs=131 cs=0 l=1 rx=0x7fb888008790 tx=0x7fb88800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:46.158 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.144+0000 7fb8937fe700 1 -- 172.21.15.42:0/1976315661 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb888008bb0 con 0x7fb8940a7740 2019-12-05T05:03:46.158 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.144+0000 7fb898ae5700 1 --2- 172.21.15.42:0/1976315661 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb8940a7740 0x7fb89408df80 secure :-1 s=THROTTLE_DONE pgs=131 cs=0 l=1 rx=0x7fb888008790 tx=0x7fb88800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:46.158 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.144+0000 7fb898ae5700 1 --2- 172.21.15.42:0/1976315661 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb8940a7740 0x7fb89408df80 secure :-1 s=THROTTLE_DONE pgs=131 cs=0 l=1 rx=0x7fb888008790 tx=0x7fb88800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:46.158 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.144+0000 7fb8937fe700 1 -- 172.21.15.42:0/1976315661 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8880087d0 con 0x7fb8940a7740 2019-12-05T05:03:46.158 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.144+0000 7fb8937fe700 1 -- 172.21.15.42:0/1976315661 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb888011860 con 0x7fb8940a7740 2019-12-05T05:03:46.158 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.144+0000 7fb89a548700 1 -- 172.21.15.42:0/1976315661 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb8940a7740 msgr2=0x7fb89408df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:46.159 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.144+0000 7fb89a548700 1 --2- 172.21.15.42:0/1976315661 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb8940a7740 0x7fb89408df80 secure :-1 s=READY pgs=131 cs=0 l=1 rx=0x7fb888008790 tx=0x7fb88800fc70).stop 2019-12-05T05:03:46.159 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.148+0000 7fb89a548700 1 -- 172.21.15.42:0/1976315661 shutdown_connections 2019-12-05T05:03:46.159 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.148+0000 7fb89a548700 1 --2- 172.21.15.42:0/1976315661 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb89409fae0 0x7fb89407eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:03:46.159 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.148+0000 7fb89a548700 1 --2- 172.21.15.42:0/1976315661 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb8940a7740 0x7fb89408df80 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:46.159 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.148+0000 7fb89a548700 1 -- 172.21.15.42:0/1976315661 shutdown_connections 2019-12-05T05:03:46.159 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.148+0000 7fb89a548700 1 -- 172.21.15.42:0/1976315661 wait complete. 2019-12-05T05:03:46.159 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.148+0000 7fb89a548700 1 -- 172.21.15.42:0/1976315661 >> 172.21.15.42:0/1976315661 conn(0x7fb8940aabe0 msgr2=0x7fb8940a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:46.166 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.148+0000 7fb89a548700 1 Processor -- start 2019-12-05T05:03:46.166 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.148+0000 7fb89a548700 1 -- start start 2019-12-05T05:03:46.166 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.152+0000 7fb89a548700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb894124c30 0x7fb894116f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:46.167 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.152+0000 7fb89a548700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb894122ff0 0x7fb89407d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:46.167 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.152+0000 7fb89a548700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb8940a5660 0x7fb8940a5a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:46.167 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.152+0000 7fb89a548700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb894123420 con 0x7fb894122ff0 2019-12-05T05:03:46.168 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.152+0000 7fb89a548700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb894117560 con 0x7fb8940a5660 2019-12-05T05:03:46.169 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.152+0000 7fb89a548700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb894117800 con 0x7fb894124c30 2019-12-05T05:03:46.169 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.152+0000 7fb8992e6700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb8940a5660 0x7fb8940a5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:46.169 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.152+0000 7fb8992e6700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb8940a5660 0x7fb8940a5a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:53668/0 (socket says 172.21.15.42:53668) 2019-12-05T05:03:46.170 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.152+0000 7fb8992e6700 1 -- 172.21.15.42:0/2733602144 learned_addr learned my addr 172.21.15.42:0/2733602144 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:03:46.170 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.152+0000 7fb898ae5700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb894124c30 0x7fb894116f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:46.170 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.152+0000 7fb893fff700 1 --2- 172.21.15.42:0/2733602144 >> v2:172.21.15.40:3300/0 conn(0x7fb894122ff0 0x7fb89407d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:46.170 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.152+0000 7fb8992e6700 1 -- 172.21.15.42:0/2733602144 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb894124c30 msgr2=0x7fb894116f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:46.170 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.152+0000 7fb8992e6700 1 --2- 172.21.15.42:0/2733602144 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb894124c30 0x7fb894116f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:46.170 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.152+0000 7fb8992e6700 1 -- 172.21.15.42:0/2733602144 >> v2:172.21.15.40:3300/0 conn(0x7fb894122ff0 msgr2=0x7fb89407d920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:46.170 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.152+0000 7fb8992e6700 1 --2- 172.21.15.42:0/2733602144 >> v2:172.21.15.40:3300/0 conn(0x7fb894122ff0 0x7fb89407d920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:46.171 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.152+0000 7fb8992e6700 1 -- 172.21.15.42:0/2733602144 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb894122210 con 0x7fb8940a5660 2019-12-05T05:03:46.171 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.152+0000 7fb8992e6700 1 --2- 172.21.15.42:0/2733602144 >> v2:172.21.15.42:3300/0 conn(0x7fb8940a5660 0x7fb8940a5a90 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fb88c0032b0 tx=0x7fb88c013090).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:46.171 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.152+0000 7fb8992e6700 1 --2- 172.21.15.42:0/2733602144 >> v2:172.21.15.42:3300/0 conn(0x7fb8940a5660 0x7fb8940a5a90 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fb88c0032b0 tx=0x7fb88c013090).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:46.171 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.152+0000 7fb8992e6700 1 --2- 172.21.15.42:0/2733602144 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb8940a5660 0x7fb8940a5a90 secure :-1 s=READY pgs=162 cs=0 l=1 rx=0x7fb88c0032b0 tx=0x7fb88c013090).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:46.171 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.152+0000 7fb8992e6700 1 --2- 172.21.15.42:0/2733602144 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb8940a5660 0x7fb8940a5a90 secure :-1 s=THROTTLE_DONE pgs=162 cs=0 l=1 rx=0x7fb88c0032b0 tx=0x7fb88c013090).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:46.171 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.152+0000 7fb8917fa700 1 -- 172.21.15.42:0/2733602144 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb88c0058c0 con 0x7fb8940a5660 2019-12-05T05:03:46.171 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.152+0000 7fb8992e6700 1 --2- 172.21.15.42:0/2733602144 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb8940a5660 0x7fb8940a5a90 secure :-1 s=THROTTLE_DONE pgs=162 cs=0 l=1 rx=0x7fb88c0032b0 tx=0x7fb88c013090).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:46.172 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.152+0000 7fb8917fa700 1 -- 172.21.15.42:0/2733602144 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb88c013b70 con 0x7fb8940a5660 2019-12-05T05:03:46.172 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.152+0000 7fb8992e6700 1 --2- 172.21.15.42:0/2733602144 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb8940a5660 0x7fb8940a5a90 secure :-1 s=THROTTLE_DONE pgs=162 cs=0 l=1 rx=0x7fb88c0032b0 tx=0x7fb88c013090).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:46.172 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.152+0000 7fb89a548700 1 -- 172.21.15.42:0/2733602144 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8940a5fe0 con 0x7fb8940a5660 2019-12-05T05:03:46.172 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.152+0000 7fb8917fa700 1 -- 172.21.15.42:0/2733602144 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb88c00a4e0 con 0x7fb8940a5660 2019-12-05T05:03:46.172 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.152+0000 7fb89a548700 1 -- 172.21.15.42:0/2733602144 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb894118f80 con 0x7fb8940a5660 2019-12-05T05:03:46.172 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.152+0000 7fb8992e6700 1 --2- 172.21.15.42:0/2733602144 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb8940a5660 0x7fb8940a5a90 secure :-1 s=THROTTLE_DONE pgs=162 cs=0 l=1 rx=0x7fb88c0032b0 tx=0x7fb88c013090).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:46.172 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.156+0000 7fb8917fa700 1 -- 172.21.15.42:0/2733602144 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fb88c00a800 con 0x7fb8940a5660 2019-12-05T05:03:46.173 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.156+0000 7fb8992e6700 1 --2- 172.21.15.42:0/2733602144 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb8940a5660 0x7fb8940a5a90 secure :-1 s=THROTTLE_DONE pgs=162 cs=0 l=1 rx=0x7fb88c0032b0 tx=0x7fb88c013090).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:46.173 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.156+0000 7fb8917fa700 1 --2- 172.21.15.42:0/2733602144 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb87403adc0 0x7fb87403d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:46.173 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.156+0000 7fb8917fa700 1 -- 172.21.15.42:0/2733602144 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6031+0+0 (secure 0 0 0) 0x7fb88c04d290 con 0x7fb8940a5660 2019-12-05T05:03:46.173 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.156+0000 7fb898ae5700 1 --2- 172.21.15.42:0/2733602144 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb87403adc0 0x7fb87403d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:46.173 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.156+0000 7fb898ae5700 1 --2- 172.21.15.42:0/2733602144 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb87403adc0 0x7fb87403d2d0 crc :-1 s=READY pgs=88 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:46.173 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.156+0000 7fb89a548700 1 -- 172.21.15.42:0/2733602144 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb89405d110 con 0x7fb8940a5660 2019-12-05T05:03:46.175 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.168+0000 7fb8992e6700 1 --2- 172.21.15.42:0/2733602144 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb8940a5660 0x7fb8940a5a90 secure :-1 s=THROTTLE_DONE pgs=162 cs=0 l=1 rx=0x7fb88c0032b0 tx=0x7fb88c013090).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:46.176 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.168+0000 7fb8917fa700 1 -- 172.21.15.42:0/2733602144 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fb88c0342f0 con 0x7fb8940a5660 2019-12-05T05:03:46.346 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.340+0000 7fb89a548700 1 -- 172.21.15.42:0/2733602144 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb89405b8c0 con 0x7fb8940a5660 2019-12-05T05:03:46.347 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.340+0000 7fb8992e6700 1 --2- 172.21.15.42:0/2733602144 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb8940a5660 0x7fb8940a5a90 secure :-1 s=THROTTLE_DONE pgs=162 cs=0 l=1 rx=0x7fb88c0032b0 tx=0x7fb88c013090).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:46.347 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.340+0000 7fb8917fa700 1 -- 172.21.15.42:0/2733602144 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+2102 (secure 0 0 0) 0x7fb88c034550 con 0x7fb8940a5660 2019-12-05T05:03:46.349 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 6 2019-12-05T05:03:46.349 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:03:46.350 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4354,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/2019200199","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2019200199},{"type":"v1","addr":"172.21.15.42:6835","nonce":2019200199}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":4}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:03:43.669739+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4341},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4341":{"gid":4341,"name":"b","rank":0,"incarnation":5,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6833/3296198322","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3296198322},{"type":"v1","addr":"172.21.15.40:6833","nonce":3296198322}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:03:46.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.348+0000 7fb89a548700 1 -- 172.21.15.42:0/2733602144 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb87403adc0 msgr2=0x7fb87403d2d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:46.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.348+0000 7fb89a548700 1 --2- 172.21.15.42:0/2733602144 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb87403adc0 0x7fb87403d2d0 crc :-1 s=READY pgs=88 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:46.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.348+0000 7fb89a548700 1 -- 172.21.15.42:0/2733602144 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb8940a5660 msgr2=0x7fb8940a5a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:46.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.348+0000 7fb89a548700 1 --2- 172.21.15.42:0/2733602144 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb8940a5660 0x7fb8940a5a90 secure :-1 s=READY pgs=162 cs=0 l=1 rx=0x7fb88c0032b0 tx=0x7fb88c013090).stop 2019-12-05T05:03:46.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.348+0000 7fb89a548700 1 -- 172.21.15.42:0/2733602144 shutdown_connections 2019-12-05T05:03:46.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.348+0000 7fb89a548700 1 --2- 172.21.15.42:0/2733602144 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb87403adc0 0x7fb87403d2d0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:46.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.348+0000 7fb89a548700 1 --2- 172.21.15.42:0/2733602144 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb8940a5660 0x7fb8940a5a90 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:46.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.348+0000 7fb89a548700 1 --2- 172.21.15.42:0/2733602144 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb894124c30 0x7fb894116f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:46.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.348+0000 7fb89a548700 1 --2- 172.21.15.42:0/2733602144 >> v2:172.21.15.40:3300/0 conn(0x7fb894122ff0 0x7fb89407d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:46.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.348+0000 7fb89a548700 1 -- 172.21.15.42:0/2733602144 shutdown_connections 2019-12-05T05:03:46.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.348+0000 7fb89a548700 1 -- 172.21.15.42:0/2733602144 wait complete. 2019-12-05T05:03:46.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.348+0000 7fb89a548700 1 -- 172.21.15.42:0/2733602144 >> 172.21.15.42:0/2733602144 conn(0x7fb8940a7740 msgr2=0x7fb8940a9640 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:46.564 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {u'session_autoclose': 300, u'balancer': u'', u'up': {u'mds_0': 4341}, u'last_failure_osd_epoch': 0, u'in': [0], u'last_failure': 0, u'max_file_size': 1099511627776, u'explicitly_allowed_features': 0, u'damaged': [], u'tableserver': 0, u'metadata_pool': 2, u'failed': [], u'epoch': 6, u'flags': 18, u'max_mds': 1, u'compat': {u'compat': {}, u'ro_compat': {}, u'incompat': {u'feature_10': u'snaprealm v2', u'feature_8': u'no anchor table', u'feature_9': u'file layout v2', u'feature_2': u'client writeable ranges', u'feature_3': u'default file layouts on dirs', u'feature_1': u'base v0.20', u'feature_6': u'dirfrag is stored in omap', u'feature_4': u'dir inode in separate object', u'feature_5': u'mds uses versioned encoding'}}, u'min_compat_client': u'0 (unknown)', u'data_pools': [3], u'info': {u'gid_4341': {u'export_targets': [], u'name': u'b', u'incarnation': 5, u'state_seq': 3, u'state': u'up:active', u'gid': 4341, u'features': 4540138292836696063, u'rank': 0, u'flags': 0, u'addrs': {u'addrvec': [{u'nonce': 3296198322, u'type': u'v2', u'addr': u'172.21.15.40:6832'}, {u'nonce': 3296198322, u'type': u'v1', u'addr': u'172.21.15.40:6833'}]}, u'addr': u'172.21.15.40:6833/3296198322'}}, u'fs_name': u'cephfs', u'created': u'2019-12-05T05:03:35.604738+0000', u'standby_count_wanted': 0, u'enabled': True, u'modified': u'2019-12-05T05:03:43.669739+0000', u'session_timeout': 60, u'stopped': [], u'ever_allowed_features': 0, u'root': 0} 2019-12-05T05:03:46.564 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2019-12-05T05:03:46.564 INFO:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2019-12-05T05:03:46.658 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2019-12-05T05:03:46.658 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_fsid": "5d836cfc-a83b-4433-9b3e-b9b2f31c9050", 2019-12-05T05:03:46.658 INFO:teuthology.orchestra.run.smithi040.stdout: "whoami": 0, 2019-12-05T05:03:46.658 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 4341, 2019-12-05T05:03:46.659 INFO:teuthology.orchestra.run.smithi040.stdout: "want_state": "up:active", 2019-12-05T05:03:46.659 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "up:active", 2019-12-05T05:03:46.659 INFO:teuthology.orchestra.run.smithi040.stdout: "rank_uptime": 3.989027536, 2019-12-05T05:03:46.659 INFO:teuthology.orchestra.run.smithi040.stdout: "mdsmap_epoch": 6, 2019-12-05T05:03:46.659 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch": 24, 2019-12-05T05:03:46.659 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch_barrier": 24, 2019-12-05T05:03:46.659 INFO:teuthology.orchestra.run.smithi040.stdout: "uptime": 8.6394791850000008 2019-12-05T05:03:46.660 INFO:teuthology.orchestra.run.smithi040.stdout:} 2019-12-05T05:03:46.664 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "5d836cfc-a83b-4433-9b3e-b9b2f31c9050", "whoami": 0, "id": 4341, "want_state": "up:active", "state": "up:active", "rank_uptime": 3.989027536, "mdsmap_epoch": 6, "osdmap_epoch": 24, "osdmap_epoch_barrier": 24, "uptime": 8.6394791850000008 } 2019-12-05T05:03:46.665 INFO:teuthology.run_tasks:Running task mds_thrash... 2019-12-05T05:03:46.710 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2019-12-05T05:03:46.818 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.808+0000 7eff0e12e700 1 Processor -- start 2019-12-05T05:03:46.818 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.808+0000 7eff0e12e700 1 -- start start 2019-12-05T05:03:46.818 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.808+0000 7eff0e12e700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7eff0808df80 0x7eff0809f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:46.819 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.808+0000 7eff0e12e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7eff0809f830 0x7eff0814efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:46.819 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.808+0000 7eff0e12e700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff0809fc60 con 0x7eff080a7740 2019-12-05T05:03:46.819 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.808+0000 7eff0e12e700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff080a0b00 con 0x7eff0808df80 2019-12-05T05:03:46.819 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.808+0000 7eff0e12e700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff0808e5b0 con 0x7eff0809f830 2019-12-05T05:03:46.819 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.808+0000 7eff0cecc700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7eff0809f830 0x7eff0814efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:46.819 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.808+0000 7eff0cecc700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7eff0809f830 0x7eff0814efe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:53676/0 (socket says 172.21.15.42:53676) 2019-12-05T05:03:46.819 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.808+0000 7eff077fe700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7eff0808df80 0x7eff0809f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:46.820 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.808+0000 7eff0cecc700 1 -- 172.21.15.42:0/1127467707 learned_addr learned my addr 172.21.15.42:0/1127467707 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:03:46.820 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.808+0000 7eff0cecc700 1 -- 172.21.15.42:0/1127467707 >> v2:172.21.15.40:3300/0 conn(0x7eff0808df80 msgr2=0x7eff0809f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:03:46.820 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.808+0000 7eff0cecc700 1 --2- 172.21.15.42:0/1127467707 >> v2:172.21.15.40:3300/0 conn(0x7eff0808df80 0x7eff0809f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:03:46.820 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.808+0000 7eff0cecc700 1 -- 172.21.15.42:0/1127467707 >> v1:172.21.15.40:6789/0 conn(0x7eff080a7740 legacy=0x7eff0807e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:46.820 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.808+0000 7eff0cecc700 1 -- 172.21.15.42:0/1127467707 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff080a0710 con 0x7eff0809f830 2019-12-05T05:03:46.820 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.808+0000 7eff0cecc700 1 --2- 172.21.15.42:0/1127467707 >> v2:172.21.15.42:3300/0 conn(0x7eff0809f830 0x7eff0814efe0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7efef0002f30 tx=0x7efef000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:46.821 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.808+0000 7eff0cecc700 1 --2- 172.21.15.42:0/1127467707 >> v2:172.21.15.42:3300/0 conn(0x7eff0809f830 0x7eff0814efe0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7efef0002f30 tx=0x7efef000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:46.821 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.808+0000 7eff0cecc700 1 --2- 172.21.15.42:0/1127467707 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7eff0809f830 0x7eff0814efe0 secure :-1 s=READY pgs=163 cs=0 l=1 rx=0x7efef0002f30 tx=0x7efef000fd30).ready entity=mon.1 client_cookie=f31fc418088ac275 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:46.821 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.808+0000 7eff0cecc700 1 --2- 172.21.15.42:0/1127467707 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7eff0809f830 0x7eff0814efe0 secure :-1 s=THROTTLE_DONE pgs=163 cs=0 l=1 rx=0x7efef0002f30 tx=0x7efef000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:46.821 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.808+0000 7eff06ffd700 1 -- 172.21.15.42:0/1127467707 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efef0008c90 con 0x7eff0809f830 2019-12-05T05:03:46.821 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.808+0000 7eff0cecc700 1 --2- 172.21.15.42:0/1127467707 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7eff0809f830 0x7eff0814efe0 secure :-1 s=THROTTLE_DONE pgs=163 cs=0 l=1 rx=0x7efef0002f30 tx=0x7efef000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:46.821 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.808+0000 7eff06ffd700 1 -- 172.21.15.42:0/1127467707 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efef00088b0 con 0x7eff0809f830 2019-12-05T05:03:46.822 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.808+0000 7eff0cecc700 1 --2- 172.21.15.42:0/1127467707 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7eff0809f830 0x7eff0814efe0 secure :-1 s=THROTTLE_DONE pgs=163 cs=0 l=1 rx=0x7efef0002f30 tx=0x7efef000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:46.822 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.808+0000 7eff06ffd700 1 -- 172.21.15.42:0/1127467707 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efef0011ad0 con 0x7eff0809f830 2019-12-05T05:03:46.822 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.808+0000 7eff0e12e700 1 -- 172.21.15.42:0/1127467707 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7eff0809f830 msgr2=0x7eff0814efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:46.822 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.808+0000 7eff0e12e700 1 --2- 172.21.15.42:0/1127467707 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7eff0809f830 0x7eff0814efe0 secure :-1 s=READY pgs=163 cs=0 l=1 rx=0x7efef0002f30 tx=0x7efef000fd30).stop 2019-12-05T05:03:46.822 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.812+0000 7eff0e12e700 1 -- 172.21.15.42:0/1127467707 shutdown_connections 2019-12-05T05:03:46.822 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.812+0000 7eff0e12e700 1 --2- 172.21.15.42:0/1127467707 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7eff0809f830 0x7eff0814efe0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:46.822 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.812+0000 7eff0e12e700 1 --2- 172.21.15.42:0/1127467707 >> v2:172.21.15.40:3300/0 conn(0x7eff0808df80 0x7eff0809f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:03:46.822 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.812+0000 7eff0e12e700 1 -- 172.21.15.42:0/1127467707 shutdown_connections 2019-12-05T05:03:46.823 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.812+0000 7eff0e12e700 1 -- 172.21.15.42:0/1127467707 wait complete. 2019-12-05T05:03:46.823 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.812+0000 7eff0e12e700 1 -- 172.21.15.42:0/1127467707 >> 172.21.15.42:0/1127467707 conn(0x7eff080aabe0 msgr2=0x7eff080a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:46.826 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.812+0000 7eff0e12e700 1 Processor -- start 2019-12-05T05:03:46.826 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.812+0000 7eff0e12e700 1 -- start start 2019-12-05T05:03:46.826 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.816+0000 7eff0e12e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7eff080a1850 0x7eff08123da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:46.826 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.816+0000 7eff0e12e700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7eff08114100 0x7eff08115920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:46.827 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.816+0000 7eff0e12e700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff08123240 con 0x7eff08114ae0 2019-12-05T05:03:46.827 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.816+0000 7eff0e12e700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff081235c0 con 0x7eff080a1850 2019-12-05T05:03:46.827 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.816+0000 7eff0e12e700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff08123860 con 0x7eff08114100 2019-12-05T05:03:46.827 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.816+0000 7eff07fff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7eff080a1850 0x7eff08123da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:46.827 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.816+0000 7eff07fff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7eff080a1850 0x7eff08123da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:53678/0 (socket says 172.21.15.42:53678) 2019-12-05T05:03:46.827 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.816+0000 7eff07fff700 1 -- 172.21.15.42:0/1600765135 learned_addr learned my addr 172.21.15.42:0/1600765135 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:03:46.827 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.816+0000 7eff077fe700 1 --2- 172.21.15.42:0/1600765135 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7eff08114100 0x7eff08115920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:46.827 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.816+0000 7eff07fff700 1 -- 172.21.15.42:0/1600765135 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7eff08114100 msgr2=0x7eff08115920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:46.828 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.816+0000 7eff07fff700 1 --2- 172.21.15.42:0/1600765135 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7eff08114100 0x7eff08115920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:46.828 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.816+0000 7eff04ff9700 1 -- 172.21.15.42:0/1600765135 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3706487831 0 0) 0x7eff08123240 con 0x7eff08114ae0 2019-12-05T05:03:46.828 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.816+0000 7eff07fff700 1 -- 172.21.15.42:0/1600765135 >> v1:172.21.15.40:6789/0 conn(0x7eff08114ae0 legacy=0x7eff08126800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:46.828 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.816+0000 7eff07fff700 1 -- 172.21.15.42:0/1600765135 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff08115e70 con 0x7eff080a1850 2019-12-05T05:03:46.828 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.816+0000 7eff07fff700 1 --2- 172.21.15.42:0/1600765135 >> v2:172.21.15.42:3300/0 conn(0x7eff080a1850 0x7eff08123da0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7eff08123780 tx=0x7eff00010d20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:46.828 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.816+0000 7eff07fff700 1 --2- 172.21.15.42:0/1600765135 >> v2:172.21.15.42:3300/0 conn(0x7eff080a1850 0x7eff08123da0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7eff08123780 tx=0x7eff00010d20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:46.828 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.816+0000 7eff07fff700 1 --2- 172.21.15.42:0/1600765135 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7eff080a1850 0x7eff08123da0 secure :-1 s=READY pgs=164 cs=0 l=1 rx=0x7eff08123780 tx=0x7eff00010d20).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:46.829 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.816+0000 7eff07fff700 1 --2- 172.21.15.42:0/1600765135 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7eff080a1850 0x7eff08123da0 secure :-1 s=THROTTLE_DONE pgs=164 cs=0 l=1 rx=0x7eff08123780 tx=0x7eff00010d20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:46.829 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.816+0000 7eff04ff9700 1 -- 172.21.15.42:0/1600765135 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7eff0000ca40 con 0x7eff080a1850 2019-12-05T05:03:46.829 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.816+0000 7eff07fff700 1 --2- 172.21.15.42:0/1600765135 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7eff080a1850 0x7eff08123da0 secure :-1 s=THROTTLE_DONE pgs=164 cs=0 l=1 rx=0x7eff08123780 tx=0x7eff00010d20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:46.829 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.816+0000 7eff07fff700 1 --2- 172.21.15.42:0/1600765135 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7eff080a1850 0x7eff08123da0 secure :-1 s=THROTTLE_DONE pgs=164 cs=0 l=1 rx=0x7eff08123780 tx=0x7eff00010d20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:46.829 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.816+0000 7eff04ff9700 1 -- 172.21.15.42:0/1600765135 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff000101c0 con 0x7eff080a1850 2019-12-05T05:03:46.829 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.816+0000 7eff0e12e700 1 -- 172.21.15.42:0/1600765135 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff08114530 con 0x7eff080a1850 2019-12-05T05:03:46.829 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.816+0000 7eff04ff9700 1 -- 172.21.15.42:0/1600765135 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7eff00012590 con 0x7eff080a1850 2019-12-05T05:03:46.830 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.816+0000 7eff0e12e700 1 -- 172.21.15.42:0/1600765135 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff08114f10 con 0x7eff080a1850 2019-12-05T05:03:46.830 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.816+0000 7eff07fff700 1 --2- 172.21.15.42:0/1600765135 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7eff080a1850 0x7eff08123da0 secure :-1 s=THROTTLE_DONE pgs=164 cs=0 l=1 rx=0x7eff08123780 tx=0x7eff00010d20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:46.830 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.816+0000 7eff04ff9700 1 -- 172.21.15.42:0/1600765135 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7eff000128a0 con 0x7eff080a1850 2019-12-05T05:03:46.830 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.816+0000 7eff07fff700 1 --2- 172.21.15.42:0/1600765135 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7eff080a1850 0x7eff08123da0 secure :-1 s=THROTTLE_DONE pgs=164 cs=0 l=1 rx=0x7eff08123780 tx=0x7eff00010d20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:46.830 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.816+0000 7eff04ff9700 1 --2- 172.21.15.42:0/1600765135 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7efeec03ce30 0x7efeec03f340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:46.830 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.820+0000 7eff04ff9700 1 -- 172.21.15.42:0/1600765135 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6031+0+0 (secure 0 0 0) 0x7eff0004b430 con 0x7eff080a1850 2019-12-05T05:03:46.830 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.820+0000 7eff077fe700 1 --2- 172.21.15.42:0/1600765135 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7efeec03ce30 0x7efeec03f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:46.831 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.820+0000 7eff077fe700 1 --2- 172.21.15.42:0/1600765135 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7efeec03ce30 0x7efeec03f340 crc :-1 s=READY pgs=89 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:46.831 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.820+0000 7eff0e12e700 1 -- 172.21.15.42:0/1600765135 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff0805d110 con 0x7eff080a1850 2019-12-05T05:03:46.831 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.824+0000 7eff07fff700 1 --2- 172.21.15.42:0/1600765135 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7eff080a1850 0x7eff08123da0 secure :-1 s=THROTTLE_DONE pgs=164 cs=0 l=1 rx=0x7eff08123780 tx=0x7eff00010d20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:46.832 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.824+0000 7eff04ff9700 1 -- 172.21.15.42:0/1600765135 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7eff000324b0 con 0x7eff080a1850 2019-12-05T05:03:47.002 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:46.996+0000 7eff0e12e700 1 -- 172.21.15.42:0/1600765135 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7eff0805b8c0 con 0x7eff080a1850 2019-12-05T05:03:47.004 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.000+0000 7eff07fff700 1 --2- 172.21.15.42:0/1600765135 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7eff080a1850 0x7eff08123da0 secure :-1 s=THROTTLE_DONE pgs=164 cs=0 l=1 rx=0x7eff08123780 tx=0x7eff00010d20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:47.004 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.000+0000 7eff04ff9700 1 -- 172.21.15.42:0/1600765135 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+14885 (secure 0 0 0) 0x7eff00032710 con 0x7eff080a1850 2019-12-05T05:03:47.005 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:03:47.005 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":24,"fsid":"5d836cfc-a83b-4433-9b3e-b9b2f31c9050","created":"2019-12-05T05:03:13.215602+0000","modified":"2019-12-05T05:03:36.607704+0000","last_up_change":"2019-12-05T05:03:22.664783+0000","last_in_change":"2019-12-05T05:03:22.664783+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2019-12-05T05:03:27.394605+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2019-12-05T05:03:30.419845+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2019-12-05T05:03:33.249075+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e3fc49bb-dcbc-43ff-ac7c-9eb0ea0bd257","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6810","nonce":12118},{"type":"v1","addr":"172.21.15.42:6811","nonce":12118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6812","nonce":12118},{"type":"v1","addr":"172.21.15.42:6813","nonce":12118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6816","nonce":12118},{"type":"v1","addr":"172.21.15.42:6817","nonce":12118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6814","nonce":12118},{"type":"v1","addr":"172.21.15.42:6815","nonce":12118}]},"public_addr":"172.21.15.42:6811/12118","cluster_addr":"172.21.15.42:6813/12118","heartbeat_back_addr":"172.21.15.42:6817/12118","heartbeat_front_addr":"172.21.15.42:6815/12118","state":["exists","up"]},{"osd":1,"uuid":"902543fb-f5a2-4321-9c9c-26f9b9ca8629","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6826","nonce":12119},{"type":"v1","addr":"172.21.15.42:6827","nonce":12119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6828","nonce":12119},{"type":"v1","addr":"172.21.15.42:6829","nonce":12119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6832","nonce":12119},{"type":"v1","addr":"172.21.15.42:6833","nonce":12119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6830","nonce":12119},{"type":"v1","addr":"172.21.15.42:6831","nonce":12119}]},"public_addr":"172.21.15.42:6827/12119","cluster_addr":"172.21.15.42:6829/12119","heartbeat_back_addr":"172.21.15.42:6833/12119","heartbeat_front_addr":"172.21.15.42:6831/12119","state":["exists","up"]},{"osd":2,"uuid":"28739ff7-bed7-4b75-8ea5-b57d088f1250","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6802","nonce":12117},{"type":"v1","addr":"172.21.15.42:6803","nonce":12117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6804","nonce":12117},{"type":"v1","addr":"172.21.15.42:6805","nonce":12117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6808","nonce":12117},{"type":"v1","addr":"172.21.15.42:6809","nonce":12117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6806","nonce":12117},{"type":"v1","addr":"172.21.15.42:6807","nonce":12117}]},"public_addr":"172.21.15.42:6803/12117","cluster_addr":"172.21.15.42:6805/12117","heartbeat_back_addr":"172.21.15.42:6809/12117","heartbeat_front_addr":"172.21.15.42:6807/12117","state":["exists","up"]},{"osd":3,"uuid":"1960f66b-2d40-4bb9-a289-ce8cd0c1c142","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6818","nonce":12116},{"type":"v1","addr":"172.21.15.42:6819","nonce":12116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6820","nonce":12116},{"type":"v1","addr":"172.21.15.42:6821","nonce":12116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6824","nonce":12116},{"type":"v1","addr":"172.21.15.42:6825","nonce":12116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6822","nonce":12116},{"type":"v1","addr":"172.21.15.42:6823","nonce":12116}]},"public_addr":"172.21.15.42:6819/12116","cluster_addr":"172.21.15.42:6821/12116","heartbeat_back_addr":"172.21.15.42:6825/12116","heartbeat_front_addr":"172.21.15.42:6823/12116","state":["exists","up"]},{"osd":4,"uuid":"45bbf1bd-7c8a-4a0d-b76b-afa6148a855c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":12488},{"type":"v1","addr":"172.21.15.40:6809","nonce":12488}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":12488},{"type":"v1","addr":"172.21.15.40:6812","nonce":12488}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":12488},{"type":"v1","addr":"172.21.15.40:6825","nonce":12488}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":12488},{"type":"v1","addr":"172.21.15.40:6819","nonce":12488}]},"public_addr":"172.21.15.40:6809/12488","cluster_addr":"172.21.15.40:6812/12488","heartbeat_back_addr":"172.21.15.40:6825/12488","heartbeat_front_addr":"172.21.15.40:6819/12488","state":["exists","up"]},{"osd":5,"uuid":"8a5511cd-925f-4869-a49e-924383e4ce53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":12489},{"type":"v1","addr":"172.21.15.40:6801","nonce":12489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":12489},{"type":"v1","addr":"172.21.15.40:6803","nonce":12489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":12489},{"type":"v1","addr":"172.21.15.40:6807","nonce":12489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":12489},{"type":"v1","addr":"172.21.15.40:6805","nonce":12489}]},"public_addr":"172.21.15.40:6801/12489","cluster_addr":"172.21.15.40:6803/12489","heartbeat_back_addr":"172.21.15.40:6807/12489","heartbeat_front_addr":"172.21.15.40:6805/12489","state":["exists","up"]},{"osd":6,"uuid":"d692c179-379e-446c-8484-a219a9241e04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6811","nonce":12491},{"type":"v1","addr":"172.21.15.40:6814","nonce":12491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6817","nonce":12491},{"type":"v1","addr":"172.21.15.40:6818","nonce":12491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":12491},{"type":"v1","addr":"172.21.15.40:6828","nonce":12491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":12491},{"type":"v1","addr":"172.21.15.40:6824","nonce":12491}]},"public_addr":"172.21.15.40:6814/12491","cluster_addr":"172.21.15.40:6818/12491","heartbeat_back_addr":"172.21.15.40:6828/12491","heartbeat_front_addr":"172.21.15.40:6824/12491","state":["exists","up"]},{"osd":7,"uuid":"79c3c243-190e-4e35-b769-319a2077e7cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6813","nonce":12490},{"type":"v1","addr":"172.21.15.40:6816","nonce":12490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":12490},{"type":"v1","addr":"172.21.15.40:6823","nonce":12490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":12490},{"type":"v1","addr":"172.21.15.40:6831","nonce":12490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6827","nonce":12490},{"type":"v1","addr":"172.21.15.40:6829","nonce":12490}]},"public_addr":"172.21.15.40:6816/12490","cluster_addr":"172.21.15.40:6823/12490","heartbeat_back_addr":"172.21.15.40:6831/12490","heartbeat_front_addr":"172.21.15.40:6829/12490","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T05:03:21.349150+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T05:03:21.644281+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T05:03:21.394040+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T05:03:21.587096+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T05:03:21.656497+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T05:03:21.510547+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T05:03:21.659358+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T05:03:21.628063+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2019-12-05T05:03:47.008 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.000+0000 7eff0e12e700 1 -- 172.21.15.42:0/1600765135 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7efeec03ce30 msgr2=0x7efeec03f340 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:47.008 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.000+0000 7eff0e12e700 1 --2- 172.21.15.42:0/1600765135 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7efeec03ce30 0x7efeec03f340 crc :-1 s=READY pgs=89 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:47.008 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.000+0000 7eff0e12e700 1 -- 172.21.15.42:0/1600765135 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7eff080a1850 msgr2=0x7eff08123da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:47.009 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.000+0000 7eff0e12e700 1 --2- 172.21.15.42:0/1600765135 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7eff080a1850 0x7eff08123da0 secure :-1 s=READY pgs=164 cs=0 l=1 rx=0x7eff08123780 tx=0x7eff00010d20).stop 2019-12-05T05:03:47.009 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.004+0000 7eff0e12e700 1 -- 172.21.15.42:0/1600765135 shutdown_connections 2019-12-05T05:03:47.009 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.004+0000 7eff0e12e700 1 --2- 172.21.15.42:0/1600765135 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7efeec03ce30 0x7efeec03f340 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:47.009 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.004+0000 7eff0e12e700 1 --2- 172.21.15.42:0/1600765135 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7eff080a1850 0x7eff08123da0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:47.009 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.004+0000 7eff0e12e700 1 --2- 172.21.15.42:0/1600765135 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7eff08114100 0x7eff08115920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:47.009 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.004+0000 7eff0e12e700 1 -- 172.21.15.42:0/1600765135 shutdown_connections 2019-12-05T05:03:47.010 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.004+0000 7eff0e12e700 1 -- 172.21.15.42:0/1600765135 wait complete. 2019-12-05T05:03:47.010 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.004+0000 7eff0e12e700 1 -- 172.21.15.42:0/1600765135 >> 172.21.15.42:0/1600765135 conn(0x7eff080a7f90 msgr2=0x7eff08113430 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:47.018 INFO:tasks.cephfs.filesystem.ceph_manager:[{u'cache_target_full_ratio_micro': 800000, u'fast_read': False, u'stripe_width': 0, u'flags_names': u'hashpspool', u'tier_of': -1, u'hit_set_grade_decay_rate': 0, u'pg_placement_num': 8, u'use_gmt_hitset': True, u'cache_target_dirty_ratio_micro': 400000, u'create_time': u'2019-12-05T05:03:27.394605+0000', u'snap_epoch': 0, u'erasure_code_profile': u'', u'pg_autoscale_mode': u'off', u'snap_seq': 0, u'expected_num_objects': 0, u'size': 2, u'pg_num_pending': 8, u'auid': 0, u'cache_min_flush_age': 0, u'hit_set_period': 0, u'min_read_recency_for_promote': 0, u'target_max_objects': 0, u'pg_placement_num_target': 8, u'pg_num': 8, u'type': 1, u'grade_table': [], u'pool_name': u'rbd', u'cache_min_evict_age': 0, u'snap_mode': u'selfmanaged', u'pg_num_target': 8, u'cache_mode': u'none', u'min_size': 1, u'cache_target_dirty_high_ratio_micro': 600000, u'object_hash': 2, u'last_pg_merge_meta': {u'ready_epoch': 0, u'source_version': u"0'0", u'source_pgid': u'0.0', u'last_epoch_clean': 0, u'target_version': u"0'0", u'last_epoch_started': 0}, u'write_tier': -1, u'last_force_op_resend_preluminous': u'0', u'pool': 1, u'removed_snaps': u'[]', u'crush_rule': 0, u'tiers': [], u'hit_set_params': {u'type': u'none'}, u'last_force_op_resend': u'0', u'pool_snaps': [], u'quota_max_objects': 0, u'last_force_op_resend_prenautilus': u'0', u'application_metadata': {u'rbd': {}}, u'options': {}, u'hit_set_count': 0, u'flags': 1, u'target_max_bytes': 0, u'quota_max_bytes': 0, u'hit_set_search_last_n': 0, u'last_change': u'17', u'min_write_recency_for_promote': 0, u'read_tier': -1}, {u'cache_target_full_ratio_micro': 800000, u'fast_read': False, u'stripe_width': 0, u'flags_names': u'hashpspool', u'tier_of': -1, u'hit_set_grade_decay_rate': 0, u'pg_placement_num': 8, u'use_gmt_hitset': True, u'cache_target_dirty_ratio_micro': 400000, u'create_time': u'2019-12-05T05:03:30.419845+0000', u'snap_epoch': 0, u'erasure_code_profile': u'', u'pg_autoscale_mode': u'off', u'snap_seq': 0, u'expected_num_objects': 0, u'size': 2, u'pg_num_pending': 8, u'auid': 0, u'cache_min_flush_age': 0, u'hit_set_period': 0, u'min_read_recency_for_promote': 0, u'target_max_objects': 0, u'pg_placement_num_target': 8, u'pg_num': 8, u'type': 1, u'grade_table': [], u'pool_name': u'cephfs_metadata', u'cache_min_evict_age': 0, u'snap_mode': u'selfmanaged', u'pg_num_target': 8, u'cache_mode': u'none', u'min_size': 1, u'cache_target_dirty_high_ratio_micro': 600000, u'object_hash': 2, u'last_pg_merge_meta': {u'ready_epoch': 0, u'source_version': u"0'0", u'source_pgid': u'0.0', u'last_epoch_clean': 0, u'target_version': u"0'0", u'last_epoch_started': 0}, u'write_tier': -1, u'last_force_op_resend_preluminous': u'0', u'pool': 2, u'removed_snaps': u'[]', u'crush_rule': 0, u'tiers': [], u'hit_set_params': {u'type': u'none'}, u'last_force_op_resend': u'0', u'pool_snaps': [], u'quota_max_objects': 0, u'last_force_op_resend_prenautilus': u'0', u'application_metadata': {u'cephfs': {u'metadata': u'cephfs'}}, u'options': {u'pg_autoscale_bias': 4, u'recovery_priority': 5, u'pg_num_min': 16}, u'hit_set_count': 0, u'flags': 1, u'target_max_bytes': 0, u'quota_max_bytes': 0, u'hit_set_search_last_n': 0, u'last_change': u'23', u'min_write_recency_for_promote': 0, u'read_tier': -1}, {u'cache_target_full_ratio_micro': 800000, u'fast_read': False, u'stripe_width': 8192, u'flags_names': u'hashpspool,ec_overwrites', u'tier_of': -1, u'hit_set_grade_decay_rate': 0, u'pg_placement_num': 8, u'use_gmt_hitset': True, u'cache_target_dirty_ratio_micro': 400000, u'create_time': u'2019-12-05T05:03:33.249075+0000', u'snap_epoch': 0, u'erasure_code_profile': u'cephfs_data', u'pg_autoscale_mode': u'off', u'snap_seq': 0, u'expected_num_objects': 0, u'size': 4, u'pg_num_pending': 8, u'auid': 0, u'cache_min_flush_age': 0, u'hit_set_period': 0, u'min_read_recency_for_promote': 0, u'target_max_objects': 0, u'pg_placement_num_target': 8, u'pg_num': 8, u'type': 3, u'grade_table': [], u'pool_name': u'cephfs_data', u'cache_min_evict_age': 0, u'snap_mode': u'selfmanaged', u'pg_num_target': 8, u'cache_mode': u'none', u'min_size': 3, u'cache_target_dirty_high_ratio_micro': 600000, u'object_hash': 2, u'last_pg_merge_meta': {u'ready_epoch': 0, u'source_version': u"0'0", u'source_pgid': u'0.0', u'last_epoch_clean': 0, u'target_version': u"0'0", u'last_epoch_started': 0}, u'write_tier': -1, u'last_force_op_resend_preluminous': u'0', u'pool': 3, u'removed_snaps': u'[]', u'crush_rule': 1, u'tiers': [], u'hit_set_params': {u'type': u'none'}, u'last_force_op_resend': u'0', u'pool_snaps': [], u'quota_max_objects': 0, u'last_force_op_resend_prenautilus': u'0', u'application_metadata': {u'cephfs': {u'data': u'cephfs'}}, u'options': {}, u'hit_set_count': 0, u'flags': 5, u'target_max_bytes': 0, u'quota_max_bytes': 0, u'hit_set_search_last_n': 0, u'last_change': u'24', u'min_write_recency_for_promote': 0, u'read_tier': -1}] 2019-12-05T05:03:47.019 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2019-12-05T05:03:47.123 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.112+0000 7f03d0583700 1 Processor -- start 2019-12-05T05:03:47.123 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.112+0000 7f03d0583700 1 -- start start 2019-12-05T05:03:47.123 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.112+0000 7f03d0583700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f03c809f550 0x7f03c8144630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:47.123 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.112+0000 7f03d0583700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f03c80a9a00 0x7f03c807e880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:47.123 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.112+0000 7f03d0583700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03c80a1000 con 0x7f03c807edd0 2019-12-05T05:03:47.124 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.112+0000 7f03d0583700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03c80a8f80 con 0x7f03c80a9a00 2019-12-05T05:03:47.124 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.112+0000 7f03d0583700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03c80a9220 con 0x7f03c809f550 2019-12-05T05:03:47.124 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.112+0000 7f03cf321700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f03c807edd0 0x7f03c8147b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:53728/0 (socket says 172.21.15.42:53728) 2019-12-05T05:03:47.124 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.112+0000 7f03cf321700 1 -- 172.21.15.42:0/2515355376 learned_addr learned my addr 172.21.15.42:0/2515355376 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:03:47.124 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.112+0000 7f03ce31f700 1 --2- 172.21.15.42:0/2515355376 >> v2:172.21.15.40:3300/0 conn(0x7f03c80a9a00 0x7f03c807e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:47.124 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.112+0000 7f03ceb20700 1 --2- 172.21.15.42:0/2515355376 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f03c809f550 0x7f03c8144630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:47.125 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.116+0000 7f03cdb1e700 1 -- 172.21.15.42:0/2515355376 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3758470917 0 0) 0x7f03c80a1000 con 0x7f03c807edd0 2019-12-05T05:03:47.125 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.116+0000 7f03cdb1e700 1 -- 172.21.15.42:0/2515355376 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03b00030b0 con 0x7f03c807edd0 2019-12-05T05:03:47.125 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.116+0000 7f03ce31f700 1 -- 172.21.15.42:0/2515355376 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f03c809f550 msgr2=0x7f03c8144630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:03:47.125 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.116+0000 7f03ce31f700 1 --2- 172.21.15.42:0/2515355376 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f03c809f550 0x7f03c8144630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:03:47.125 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.116+0000 7f03ce31f700 1 -- 172.21.15.42:0/2515355376 >> v1:172.21.15.42:6789/0 conn(0x7f03c807edd0 legacy=0x7f03c8147b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:47.125 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.116+0000 7f03ce31f700 1 -- 172.21.15.42:0/2515355376 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03c8144b80 con 0x7f03c80a9a00 2019-12-05T05:03:47.125 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.116+0000 7f03ce31f700 1 --2- 172.21.15.42:0/2515355376 >> v2:172.21.15.40:3300/0 conn(0x7f03c80a9a00 0x7f03c807e880 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f03c80a9140 tx=0x7f03c400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:47.126 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.116+0000 7f03ce31f700 1 --2- 172.21.15.42:0/2515355376 >> v2:172.21.15.40:3300/0 conn(0x7f03c80a9a00 0x7f03c807e880 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f03c80a9140 tx=0x7f03c400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:47.126 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.116+0000 7f03ce31f700 1 --2- 172.21.15.42:0/2515355376 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f03c80a9a00 0x7f03c807e880 secure :-1 s=READY pgs=134 cs=0 l=1 rx=0x7f03c80a9140 tx=0x7f03c400fc70).ready entity=mon.0 client_cookie=248e1806c75eee2c server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:47.126 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.116+0000 7f03ce31f700 1 --2- 172.21.15.42:0/2515355376 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f03c80a9a00 0x7f03c807e880 secure :-1 s=THROTTLE_DONE pgs=134 cs=0 l=1 rx=0x7f03c80a9140 tx=0x7f03c400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:47.126 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.116+0000 7f03cdb1e700 1 -- 172.21.15.42:0/2515355376 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f03c40112f0 con 0x7f03c80a9a00 2019-12-05T05:03:47.126 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.116+0000 7f03ce31f700 1 --2- 172.21.15.42:0/2515355376 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f03c80a9a00 0x7f03c807e880 secure :-1 s=THROTTLE_DONE pgs=134 cs=0 l=1 rx=0x7f03c80a9140 tx=0x7f03c400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:47.127 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.116+0000 7f03ce31f700 1 --2- 172.21.15.42:0/2515355376 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f03c80a9a00 0x7f03c807e880 secure :-1 s=THROTTLE_DONE pgs=134 cs=0 l=1 rx=0x7f03c80a9140 tx=0x7f03c400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:47.127 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.116+0000 7f03cdb1e700 1 -- 172.21.15.42:0/2515355376 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03c4008ad0 con 0x7f03c80a9a00 2019-12-05T05:03:47.127 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.116+0000 7f03cdb1e700 1 -- 172.21.15.42:0/2515355376 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f03c40118d0 con 0x7f03c80a9a00 2019-12-05T05:03:47.127 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.116+0000 7f03d0583700 1 -- 172.21.15.42:0/2515355376 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f03c80a9a00 msgr2=0x7f03c807e880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:47.127 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.116+0000 7f03d0583700 1 --2- 172.21.15.42:0/2515355376 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f03c80a9a00 0x7f03c807e880 secure :-1 s=READY pgs=134 cs=0 l=1 rx=0x7f03c80a9140 tx=0x7f03c400fc70).stop 2019-12-05T05:03:47.127 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.116+0000 7f03d0583700 1 -- 172.21.15.42:0/2515355376 shutdown_connections 2019-12-05T05:03:47.127 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.116+0000 7f03d0583700 1 --2- 172.21.15.42:0/2515355376 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f03c809f550 0x7f03c8144630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:03:47.128 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.116+0000 7f03d0583700 1 --2- 172.21.15.42:0/2515355376 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f03c80a9a00 0x7f03c807e880 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:47.128 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.116+0000 7f03d0583700 1 -- 172.21.15.42:0/2515355376 shutdown_connections 2019-12-05T05:03:47.128 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.116+0000 7f03d0583700 1 -- 172.21.15.42:0/2515355376 wait complete. 2019-12-05T05:03:47.128 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.116+0000 7f03d0583700 1 -- 172.21.15.42:0/2515355376 >> 172.21.15.42:0/2515355376 conn(0x7f03c80a9ef0 msgr2=0x7f03c80aa320 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:47.131 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.116+0000 7f03d0583700 1 Processor -- start 2019-12-05T05:03:47.131 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.116+0000 7f03d0583700 1 -- start start 2019-12-05T05:03:47.132 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.120+0000 7f03d0583700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f03c8059f50 0x7f03c81311b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:47.132 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.120+0000 7f03d0583700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03c8131700 con 0x7f03c808f330 2019-12-05T05:03:47.132 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.120+0000 7f03d0583700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03c8078830 con 0x7f03c808e490 2019-12-05T05:03:47.132 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.120+0000 7f03d0583700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03c808df50 con 0x7f03c8059f50 2019-12-05T05:03:47.132 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.120+0000 7f03ce31f700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f03c808e490 0x7f03c8120430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:53730/0 (socket says 172.21.15.42:53730) 2019-12-05T05:03:47.132 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.120+0000 7f03ce31f700 1 -- 172.21.15.42:0/1587021435 learned_addr learned my addr 172.21.15.42:0/1587021435 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:03:47.133 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.120+0000 7f03ceb20700 1 --2- 172.21.15.42:0/1587021435 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f03c8059f50 0x7f03c81311b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:47.133 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.120+0000 7f03bf7fe700 1 -- 172.21.15.42:0/1587021435 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1695489822 0 0) 0x7f03c8078830 con 0x7f03c808e490 2019-12-05T05:03:47.133 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.120+0000 7f03bf7fe700 1 -- 172.21.15.42:0/1587021435 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03a80030b0 con 0x7f03c808e490 2019-12-05T05:03:47.133 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.120+0000 7f03bf7fe700 1 -- 172.21.15.42:0/1587021435 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2274013697 0 0) 0x7f03c8131700 con 0x7f03c808f330 2019-12-05T05:03:47.133 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.120+0000 7f03bf7fe700 1 -- 172.21.15.42:0/1587021435 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03c8078830 con 0x7f03c808f330 2019-12-05T05:03:47.133 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.120+0000 7f03bf7fe700 1 -- 172.21.15.42:0/1587021435 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 2659067460 0 0) 0x7f03a80030b0 con 0x7f03c808e490 2019-12-05T05:03:47.133 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.120+0000 7f03ceb20700 1 -- 172.21.15.42:0/1587021435 >> v1:172.21.15.42:6789/0 conn(0x7f03c808e490 legacy=0x7f03c8120430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:47.134 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.120+0000 7f03ceb20700 1 -- 172.21.15.42:0/1587021435 >> v1:172.21.15.40:6789/0 conn(0x7f03c808f330 legacy=0x7f03c8138600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:47.134 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.120+0000 7f03ceb20700 1 -- 172.21.15.42:0/1587021435 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03c808fc80 con 0x7f03c8059f50 2019-12-05T05:03:47.134 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.120+0000 7f03ceb20700 1 --2- 172.21.15.42:0/1587021435 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f03c8059f50 0x7f03c81311b0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f03c4007670 tx=0x7f03c4010200).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:47.134 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.120+0000 7f03ceb20700 1 --2- 172.21.15.42:0/1587021435 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f03c8059f50 0x7f03c81311b0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f03c4007670 tx=0x7f03c4010200).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:47.134 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.120+0000 7f03ceb20700 1 --2- 172.21.15.42:0/1587021435 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f03c8059f50 0x7f03c81311b0 secure :-1 s=READY pgs=42 cs=0 l=1 rx=0x7f03c4007670 tx=0x7f03c4010200).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:47.134 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.120+0000 7f03ceb20700 1 --2- 172.21.15.42:0/1587021435 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f03c8059f50 0x7f03c81311b0 secure :-1 s=THROTTLE_DONE pgs=42 cs=0 l=1 rx=0x7f03c4007670 tx=0x7f03c4010200).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:47.134 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.120+0000 7f03bf7fe700 1 -- 172.21.15.42:0/1587021435 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f03c40112f0 con 0x7f03c8059f50 2019-12-05T05:03:47.135 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.120+0000 7f03d0583700 1 -- 172.21.15.42:0/1587021435 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03c808f760 con 0x7f03c8059f50 2019-12-05T05:03:47.135 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.120+0000 7f03ceb20700 1 --2- 172.21.15.42:0/1587021435 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f03c8059f50 0x7f03c81311b0 secure :-1 s=THROTTLE_DONE pgs=42 cs=0 l=1 rx=0x7f03c4007670 tx=0x7f03c4010200).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:47.135 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.120+0000 7f03d0583700 1 -- 172.21.15.42:0/1587021435 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03c8139e10 con 0x7f03c8059f50 2019-12-05T05:03:47.135 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.120+0000 7f03bf7fe700 1 -- 172.21.15.42:0/1587021435 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03c40114f0 con 0x7f03c8059f50 2019-12-05T05:03:47.135 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.120+0000 7f03ceb20700 1 --2- 172.21.15.42:0/1587021435 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f03c8059f50 0x7f03c81311b0 secure :-1 s=THROTTLE_DONE pgs=42 cs=0 l=1 rx=0x7f03c4007670 tx=0x7f03c4010200).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:47.135 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.120+0000 7f03bf7fe700 1 -- 172.21.15.42:0/1587021435 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f03c000ec40 con 0x7f03c8059f50 2019-12-05T05:03:47.135 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.124+0000 7f03ceb20700 1 --2- 172.21.15.42:0/1587021435 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f03c8059f50 0x7f03c81311b0 secure :-1 s=THROTTLE_DONE pgs=42 cs=0 l=1 rx=0x7f03c4007670 tx=0x7f03c4010200).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:47.135 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.124+0000 7f03bf7fe700 1 -- 172.21.15.42:0/1587021435 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f03c0013060 con 0x7f03c8059f50 2019-12-05T05:03:47.136 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.124+0000 7f03ceb20700 1 --2- 172.21.15.42:0/1587021435 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f03c8059f50 0x7f03c81311b0 secure :-1 s=THROTTLE_DONE pgs=42 cs=0 l=1 rx=0x7f03c4007670 tx=0x7f03c4010200).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:47.136 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.124+0000 7f03bf7fe700 1 --2- 172.21.15.42:0/1587021435 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f03a803b450 0x7f03a803d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:47.136 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.124+0000 7f03bf7fe700 1 -- 172.21.15.42:0/1587021435 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f03c004ab20 con 0x7f03c8059f50 2019-12-05T05:03:47.136 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.124+0000 7f03ce31f700 1 --2- 172.21.15.42:0/1587021435 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f03a803b450 0x7f03a803d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:47.149 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.124+0000 7f03ce31f700 1 --2- 172.21.15.42:0/1587021435 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f03a803b450 0x7f03a803d960 crc :-1 s=READY pgs=90 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:47.149 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.128+0000 7f03d0583700 1 -- 172.21.15.42:0/1587021435 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03c805d110 con 0x7f03c8059f50 2019-12-05T05:03:47.149 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.140+0000 7f03ceb20700 1 --2- 172.21.15.42:0/1587021435 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f03c8059f50 0x7f03c81311b0 secure :-1 s=THROTTLE_DONE pgs=42 cs=0 l=1 rx=0x7f03c4007670 tx=0x7f03c4010200).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:47.150 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.140+0000 7f03bf7fe700 1 -- 172.21.15.42:0/1587021435 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f03c808fc80 con 0x7f03c8059f50 2019-12-05T05:03:47.314 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.300+0000 7f03d0583700 1 -- 172.21.15.42:0/1587021435 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f03c805b8c0 con 0x7f03c8059f50 2019-12-05T05:03:47.314 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.304+0000 7f03ceb20700 1 --2- 172.21.15.42:0/1587021435 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f03c8059f50 0x7f03c81311b0 secure :-1 s=THROTTLE_DONE pgs=42 cs=0 l=1 rx=0x7f03c4007670 tx=0x7f03c4010200).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:47.314 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.304+0000 7f03bf7fe700 1 -- 172.21.15.42:0/1587021435 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v24) v1 ==== 92+0+10 (secure 0 0 0) 0x7f03c0031b20 con 0x7f03c8059f50 2019-12-05T05:03:47.314 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.308+0000 7f03d0583700 1 -- 172.21.15.42:0/1587021435 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f03a803b450 msgr2=0x7f03a803d960 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:47.315 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.308+0000 7f03d0583700 1 --2- 172.21.15.42:0/1587021435 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f03a803b450 0x7f03a803d960 crc :-1 s=READY pgs=90 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:47.315 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.308+0000 7f03d0583700 1 -- 172.21.15.42:0/1587021435 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f03c8059f50 msgr2=0x7f03c81311b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:47.315 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.308+0000 7f03d0583700 1 --2- 172.21.15.42:0/1587021435 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f03c8059f50 0x7f03c81311b0 secure :-1 s=READY pgs=42 cs=0 l=1 rx=0x7f03c4007670 tx=0x7f03c4010200).stop 2019-12-05T05:03:47.315 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.308+0000 7f03d0583700 1 -- 172.21.15.42:0/1587021435 shutdown_connections 2019-12-05T05:03:47.315 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.308+0000 7f03d0583700 1 --2- 172.21.15.42:0/1587021435 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f03a803b450 0x7f03a803d960 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:47.315 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.308+0000 7f03d0583700 1 --2- 172.21.15.42:0/1587021435 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f03c8059f50 0x7f03c81311b0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:47.315 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.308+0000 7f03d0583700 1 -- 172.21.15.42:0/1587021435 shutdown_connections 2019-12-05T05:03:47.316 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.308+0000 7f03d0583700 1 -- 172.21.15.42:0/1587021435 wait complete. 2019-12-05T05:03:47.316 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.308+0000 7f03d0583700 1 -- 172.21.15.42:0/1587021435 >> 172.21.15.42:0/1587021435 conn(0x7f03c80a5b00 msgr2=0x7f03c80a69b0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:47.316 INFO:teuthology.orchestra.run.smithi042.stdout:pg_num: 8 2019-12-05T05:03:47.323 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2019-12-05T05:03:47.429 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.420+0000 7f48e797c700 1 Processor -- start 2019-12-05T05:03:47.430 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.420+0000 7f48e797c700 1 -- start start 2019-12-05T05:03:47.430 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.420+0000 7f48e797c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f48e009f550 0x7f48e0144630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:47.430 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.420+0000 7f48e797c700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f48e00a9a00 0x7f48e007e880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:47.430 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.420+0000 7f48e797c700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48e00a1000 con 0x7f48e007edd0 2019-12-05T05:03:47.430 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.420+0000 7f48e797c700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48e00a8f80 con 0x7f48e00a9a00 2019-12-05T05:03:47.430 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.420+0000 7f48e797c700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48e00a9220 con 0x7f48e009f550 2019-12-05T05:03:47.430 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.420+0000 7f48e5f19700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f48e009f550 0x7f48e0144630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:47.431 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.420+0000 7f48e5f19700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f48e009f550 0x7f48e0144630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:53700/0 (socket says 172.21.15.42:53700) 2019-12-05T05:03:47.431 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.420+0000 7f48e5f19700 1 -- 172.21.15.42:0/84962612 learned_addr learned my addr 172.21.15.42:0/84962612 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:03:47.431 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.420+0000 7f48e5718700 1 --2- 172.21.15.42:0/84962612 >> v2:172.21.15.40:3300/0 conn(0x7f48e00a9a00 0x7f48e007e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:47.431 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.420+0000 7f48e5718700 1 -- 172.21.15.42:0/84962612 >> v2:172.21.15.42:3300/0 conn(0x7f48e009f550 msgr2=0x7f48e0144630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:03:47.431 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.420+0000 7f48e5718700 1 --2- 172.21.15.42:0/84962612 >> v2:172.21.15.42:3300/0 conn(0x7f48e009f550 0x7f48e0144630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:03:47.431 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.420+0000 7f48e5718700 1 -- 172.21.15.42:0/84962612 >> v1:172.21.15.42:6789/0 conn(0x7f48e007edd0 legacy=0x7f48e0147b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:47.431 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.420+0000 7f48e5718700 1 -- 172.21.15.42:0/84962612 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48e0144b80 con 0x7f48e00a9a00 2019-12-05T05:03:47.432 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.420+0000 7f48e5718700 1 --2- 172.21.15.42:0/84962612 >> v2:172.21.15.40:3300/0 conn(0x7f48e00a9a00 0x7f48e007e880 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f48e00a9140 tx=0x7f48d400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:47.432 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.420+0000 7f48e5718700 1 --2- 172.21.15.42:0/84962612 >> v2:172.21.15.40:3300/0 conn(0x7f48e00a9a00 0x7f48e007e880 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f48e00a9140 tx=0x7f48d400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:47.432 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.420+0000 7f48e5718700 1 --2- 172.21.15.42:0/84962612 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f48e00a9a00 0x7f48e007e880 secure :-1 s=READY pgs=136 cs=0 l=1 rx=0x7f48e00a9140 tx=0x7f48d400fff0).ready entity=mon.0 client_cookie=db725eddf59e77c2 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:47.432 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.420+0000 7f48e5718700 1 --2- 172.21.15.42:0/84962612 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f48e00a9a00 0x7f48e007e880 secure :-1 s=THROTTLE_DONE pgs=136 cs=0 l=1 rx=0x7f48e00a9140 tx=0x7f48d400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:47.432 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.420+0000 7f48e4f17700 1 -- 172.21.15.42:0/84962612 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f48d4011a10 con 0x7f48e00a9a00 2019-12-05T05:03:47.432 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.420+0000 7f48e5718700 1 --2- 172.21.15.42:0/84962612 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f48e00a9a00 0x7f48e007e880 secure :-1 s=THROTTLE_DONE pgs=136 cs=0 l=1 rx=0x7f48e00a9140 tx=0x7f48d400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:47.432 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.420+0000 7f48e4f17700 1 -- 172.21.15.42:0/84962612 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48d4011630 con 0x7f48e00a9a00 2019-12-05T05:03:47.433 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.420+0000 7f48e5718700 1 --2- 172.21.15.42:0/84962612 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f48e00a9a00 0x7f48e007e880 secure :-1 s=THROTTLE_DONE pgs=136 cs=0 l=1 rx=0x7f48e00a9140 tx=0x7f48d400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:47.433 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.420+0000 7f48e4f17700 1 -- 172.21.15.42:0/84962612 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f48d400a160 con 0x7f48e00a9a00 2019-12-05T05:03:47.433 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.420+0000 7f48e797c700 1 -- 172.21.15.42:0/84962612 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f48e00a9a00 msgr2=0x7f48e007e880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:47.433 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.420+0000 7f48e797c700 1 --2- 172.21.15.42:0/84962612 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f48e00a9a00 0x7f48e007e880 secure :-1 s=READY pgs=136 cs=0 l=1 rx=0x7f48e00a9140 tx=0x7f48d400fff0).stop 2019-12-05T05:03:47.433 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.424+0000 7f48e797c700 1 -- 172.21.15.42:0/84962612 shutdown_connections 2019-12-05T05:03:47.433 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.424+0000 7f48e797c700 1 --2- 172.21.15.42:0/84962612 >> v2:172.21.15.42:3300/0 conn(0x7f48e009f550 0x7f48e0144630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:03:47.433 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.424+0000 7f48e797c700 1 --2- 172.21.15.42:0/84962612 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f48e00a9a00 0x7f48e007e880 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:47.434 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.424+0000 7f48e797c700 1 -- 172.21.15.42:0/84962612 shutdown_connections 2019-12-05T05:03:47.434 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.424+0000 7f48e797c700 1 -- 172.21.15.42:0/84962612 wait complete. 2019-12-05T05:03:47.434 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.424+0000 7f48e797c700 1 -- 172.21.15.42:0/84962612 >> 172.21.15.42:0/84962612 conn(0x7f48e00a9ef0 msgr2=0x7f48e00aa320 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:47.437 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.424+0000 7f48e797c700 1 Processor -- start 2019-12-05T05:03:47.437 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.424+0000 7f48e797c700 1 -- start start 2019-12-05T05:03:47.437 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.428+0000 7f48e797c700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f48e0123c10 0x7f48e008fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:47.437 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.428+0000 7f48e797c700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f48e0120200 0x7f48e0120630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:47.437 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.428+0000 7f48e797c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f48e0131730 0x7f48e0131b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:47.438 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.428+0000 7f48e797c700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48e00a23a0 con 0x7f48e0123c10 2019-12-05T05:03:47.438 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.428+0000 7f48e797c700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48e00a2640 con 0x7f48e0131730 2019-12-05T05:03:47.438 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.428+0000 7f48e797c700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48e00a28e0 con 0x7f48e0120200 2019-12-05T05:03:47.438 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.428+0000 7f48e671a700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f48e0131730 0x7f48e0131b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:47.438 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.428+0000 7f48e5718700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f48e0120200 0x7f48e0120630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:47.438 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.428+0000 7f48e5718700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f48e0120200 0x7f48e0120630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:54172/0 (socket says 172.21.15.42:54172) 2019-12-05T05:03:47.438 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.428+0000 7f48e671a700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f48e0131730 0x7f48e0131b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:53710/0 (socket says 172.21.15.42:53710) 2019-12-05T05:03:47.439 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.428+0000 7f48e5718700 1 -- 172.21.15.42:0/3478538458 learned_addr learned my addr 172.21.15.42:0/3478538458 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:03:47.439 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.428+0000 7f48e5f19700 1 --2- 172.21.15.42:0/3478538458 >> v2:172.21.15.40:3300/0 conn(0x7f48e0123c10 0x7f48e008fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:47.439 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.428+0000 7f48e671a700 1 -- 172.21.15.42:0/3478538458 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f48e0120200 msgr2=0x7f48e0120630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:47.439 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.428+0000 7f48e671a700 1 --2- 172.21.15.42:0/3478538458 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f48e0120200 0x7f48e0120630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:47.439 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.428+0000 7f48e671a700 1 -- 172.21.15.42:0/3478538458 >> v2:172.21.15.40:3300/0 conn(0x7f48e0123c10 msgr2=0x7f48e008fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:47.439 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.428+0000 7f48e671a700 1 --2- 172.21.15.42:0/3478538458 >> v2:172.21.15.40:3300/0 conn(0x7f48e0123c10 0x7f48e008fad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:47.439 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.428+0000 7f48e671a700 1 -- 172.21.15.42:0/3478538458 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48e01320b0 con 0x7f48e0131730 2019-12-05T05:03:47.440 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.428+0000 7f48e5718700 1 --2- 172.21.15.42:0/3478538458 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f48e0120200 0x7f48e0120630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:03:47.440 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.428+0000 7f48e671a700 1 --2- 172.21.15.42:0/3478538458 >> v2:172.21.15.42:3300/0 conn(0x7f48e0131730 0x7f48e0131b60 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f48e00a2800 tx=0x7f48d8010840).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:47.440 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.428+0000 7f48e671a700 1 --2- 172.21.15.42:0/3478538458 >> v2:172.21.15.42:3300/0 conn(0x7f48e0131730 0x7f48e0131b60 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f48e00a2800 tx=0x7f48d8010840).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:47.440 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.428+0000 7f48e671a700 1 --2- 172.21.15.42:0/3478538458 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f48e0131730 0x7f48e0131b60 secure :-1 s=READY pgs=168 cs=0 l=1 rx=0x7f48e00a2800 tx=0x7f48d8010840).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:47.440 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.428+0000 7f48e671a700 1 --2- 172.21.15.42:0/3478538458 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f48e0131730 0x7f48e0131b60 secure :-1 s=THROTTLE_DONE pgs=168 cs=0 l=1 rx=0x7f48e00a2800 tx=0x7f48d8010840).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:47.440 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.428+0000 7f48d27fc700 1 -- 172.21.15.42:0/3478538458 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f48d8006a40 con 0x7f48e0131730 2019-12-05T05:03:47.440 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.428+0000 7f48e671a700 1 --2- 172.21.15.42:0/3478538458 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f48e0131730 0x7f48e0131b60 secure :-1 s=THROTTLE_DONE pgs=168 cs=0 l=1 rx=0x7f48e00a2800 tx=0x7f48d8010840).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:47.441 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.428+0000 7f48e797c700 1 -- 172.21.15.42:0/3478538458 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48e00a41d0 con 0x7f48e0131730 2019-12-05T05:03:47.441 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.428+0000 7f48d27fc700 1 -- 172.21.15.42:0/3478538458 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48d8006660 con 0x7f48e0131730 2019-12-05T05:03:47.441 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.428+0000 7f48e671a700 1 --2- 172.21.15.42:0/3478538458 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f48e0131730 0x7f48e0131b60 secure :-1 s=THROTTLE_DONE pgs=168 cs=0 l=1 rx=0x7f48e00a2800 tx=0x7f48d8010840).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:47.441 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.428+0000 7f48d27fc700 1 -- 172.21.15.42:0/3478538458 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f48d8011040 con 0x7f48e0131730 2019-12-05T05:03:47.441 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.428+0000 7f48e797c700 1 -- 172.21.15.42:0/3478538458 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48e00a1f90 con 0x7f48e0131730 2019-12-05T05:03:47.441 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.428+0000 7f48e671a700 1 --2- 172.21.15.42:0/3478538458 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f48e0131730 0x7f48e0131b60 secure :-1 s=THROTTLE_DONE pgs=168 cs=0 l=1 rx=0x7f48e00a2800 tx=0x7f48d8010840).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:47.441 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.428+0000 7f48d27fc700 1 -- 172.21.15.42:0/3478538458 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f48d8011360 con 0x7f48e0131730 2019-12-05T05:03:47.442 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.428+0000 7f48e671a700 1 --2- 172.21.15.42:0/3478538458 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f48e0131730 0x7f48e0131b60 secure :-1 s=THROTTLE_DONE pgs=168 cs=0 l=1 rx=0x7f48e00a2800 tx=0x7f48d8010840).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:47.442 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.428+0000 7f48d27fc700 1 --2- 172.21.15.42:0/3478538458 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f48c803ae10 0x7f48c803d320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:47.442 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.432+0000 7f48d27fc700 1 -- 172.21.15.42:0/3478538458 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f48d804af20 con 0x7f48e0131730 2019-12-05T05:03:47.442 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.432+0000 7f48e5f19700 1 --2- 172.21.15.42:0/3478538458 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f48c803ae10 0x7f48c803d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:47.442 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.432+0000 7f48e5f19700 1 --2- 172.21.15.42:0/3478538458 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f48c803ae10 0x7f48c803d320 crc :-1 s=READY pgs=91 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:47.443 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.436+0000 7f48e797c700 1 -- 172.21.15.42:0/3478538458 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48e005d110 con 0x7f48e0131730 2019-12-05T05:03:47.445 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.440+0000 7f48e671a700 1 --2- 172.21.15.42:0/3478538458 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f48e0131730 0x7f48e0131b60 secure :-1 s=THROTTLE_DONE pgs=168 cs=0 l=1 rx=0x7f48e00a2800 tx=0x7f48d8010840).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:47.445 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.440+0000 7f48d27fc700 1 -- 172.21.15.42:0/3478538458 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f48d8031f80 con 0x7f48e0131730 2019-12-05T05:03:47.603 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.596+0000 7f48e797c700 1 -- 172.21.15.42:0/3478538458 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7f48e005b8c0 con 0x7f48e0131730 2019-12-05T05:03:47.604 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.600+0000 7f48e671a700 1 --2- 172.21.15.42:0/3478538458 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f48e0131730 0x7f48e0131b60 secure :-1 s=THROTTLE_DONE pgs=168 cs=0 l=1 rx=0x7f48e00a2800 tx=0x7f48d8010840).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:47.604 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.600+0000 7f48d27fc700 1 -- 172.21.15.42:0/3478538458 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v24) v1 ==== 104+0+10 (secure 0 0 0) 0x7f48d80321e0 con 0x7f48e0131730 2019-12-05T05:03:47.605 INFO:teuthology.orchestra.run.smithi042.stdout:pg_num: 8 2019-12-05T05:03:47.607 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.600+0000 7f48e797c700 1 -- 172.21.15.42:0/3478538458 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f48c803ae10 msgr2=0x7f48c803d320 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:47.607 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.600+0000 7f48e797c700 1 --2- 172.21.15.42:0/3478538458 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f48c803ae10 0x7f48c803d320 crc :-1 s=READY pgs=91 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:47.607 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.600+0000 7f48e797c700 1 -- 172.21.15.42:0/3478538458 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f48e0131730 msgr2=0x7f48e0131b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:47.607 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.600+0000 7f48e797c700 1 --2- 172.21.15.42:0/3478538458 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f48e0131730 0x7f48e0131b60 secure :-1 s=READY pgs=168 cs=0 l=1 rx=0x7f48e00a2800 tx=0x7f48d8010840).stop 2019-12-05T05:03:47.608 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.600+0000 7f48e797c700 1 -- 172.21.15.42:0/3478538458 shutdown_connections 2019-12-05T05:03:47.608 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.600+0000 7f48e797c700 1 --2- 172.21.15.42:0/3478538458 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f48c803ae10 0x7f48c803d320 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:47.608 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.600+0000 7f48e797c700 1 --2- 172.21.15.42:0/3478538458 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f48e0131730 0x7f48e0131b60 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:47.608 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.600+0000 7f48e797c700 1 --2- 172.21.15.42:0/3478538458 >> v2:172.21.15.40:3300/0 conn(0x7f48e0123c10 0x7f48e008fad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:47.608 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.600+0000 7f48e797c700 1 --2- 172.21.15.42:0/3478538458 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f48e0120200 0x7f48e0120630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:47.608 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.600+0000 7f48e797c700 1 -- 172.21.15.42:0/3478538458 shutdown_connections 2019-12-05T05:03:47.608 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.600+0000 7f48e797c700 1 -- 172.21.15.42:0/3478538458 wait complete. 2019-12-05T05:03:47.609 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.600+0000 7f48e797c700 1 -- 172.21.15.42:0/3478538458 >> 172.21.15.42:0/3478538458 conn(0x7f48e005a210 msgr2=0x7f48e00a4d90 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:47.625 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2019-12-05T05:03:47.732 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.724+0000 7f6aad7a8700 1 Processor -- start 2019-12-05T05:03:47.733 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.724+0000 7f6aad7a8700 1 -- start start 2019-12-05T05:03:47.733 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.724+0000 7f6aad7a8700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6aa809f550 0x7f6aa8144630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:47.733 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.724+0000 7f6aad7a8700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6aa814cf50 0x7f6aa814f390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:47.733 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.724+0000 7f6aad7a8700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6aa807f080 con 0x7f6aa80a9a00 2019-12-05T05:03:47.733 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.724+0000 7f6aad7a8700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6aa80a8f80 con 0x7f6aa809f550 2019-12-05T05:03:47.733 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.724+0000 7f6aad7a8700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6aa80a9220 con 0x7f6aa814cf50 2019-12-05T05:03:47.734 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.724+0000 7f6aa77fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6aa809f550 0x7f6aa8144630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:47.734 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.724+0000 7f6aa77fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6aa809f550 0x7f6aa8144630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:53714/0 (socket says 172.21.15.42:53714) 2019-12-05T05:03:47.734 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.724+0000 7f6aa77fe700 1 -- 172.21.15.42:0/2751002626 learned_addr learned my addr 172.21.15.42:0/2751002626 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:03:47.734 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.724+0000 7f6aa7fff700 1 --2- 172.21.15.42:0/2751002626 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6aa814cf50 0x7f6aa814f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:47.734 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.724+0000 7f6aa77fe700 1 -- 172.21.15.42:0/2751002626 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6aa814cf50 msgr2=0x7f6aa814f390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:03:47.734 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.724+0000 7f6aa77fe700 1 --2- 172.21.15.42:0/2751002626 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6aa814cf50 0x7f6aa814f390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:03:47.734 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.724+0000 7f6aa77fe700 1 -- 172.21.15.42:0/2751002626 >> v1:172.21.15.40:6789/0 conn(0x7f6aa80a9a00 legacy=0x7f6aa807e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:47.735 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.724+0000 7f6aa77fe700 1 -- 172.21.15.42:0/2751002626 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6aa80a1000 con 0x7f6aa809f550 2019-12-05T05:03:47.735 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.724+0000 7f6aa7fff700 1 --2- 172.21.15.42:0/2751002626 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6aa814cf50 0x7f6aa814f390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T05:03:47.735 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.724+0000 7f6aa77fe700 1 --2- 172.21.15.42:0/2751002626 >> v2:172.21.15.42:3300/0 conn(0x7f6aa809f550 0x7f6aa8144630 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f6aa81421e0 tx=0x7f6a9800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:47.735 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.724+0000 7f6aa77fe700 1 --2- 172.21.15.42:0/2751002626 >> v2:172.21.15.42:3300/0 conn(0x7f6aa809f550 0x7f6aa8144630 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f6aa81421e0 tx=0x7f6a9800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:47.735 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.724+0000 7f6aa77fe700 1 --2- 172.21.15.42:0/2751002626 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6aa809f550 0x7f6aa8144630 secure :-1 s=READY pgs=169 cs=0 l=1 rx=0x7f6aa81421e0 tx=0x7f6a9800fd30).ready entity=mon.1 client_cookie=6c61f793f6040433 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:47.735 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.724+0000 7f6aa77fe700 1 --2- 172.21.15.42:0/2751002626 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6aa809f550 0x7f6aa8144630 secure :-1 s=THROTTLE_DONE pgs=169 cs=0 l=1 rx=0x7f6aa81421e0 tx=0x7f6a9800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:47.735 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.724+0000 7f6aa77fe700 1 --2- 172.21.15.42:0/2751002626 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6aa809f550 0x7f6aa8144630 secure :-1 s=THROTTLE_DONE pgs=169 cs=0 l=1 rx=0x7f6aa81421e0 tx=0x7f6a9800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:47.736 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.724+0000 7f6aa67fc700 1 -- 172.21.15.42:0/2751002626 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6a98011370 con 0x7f6aa809f550 2019-12-05T05:03:47.736 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.724+0000 7f6aa77fe700 1 --2- 172.21.15.42:0/2751002626 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6aa809f550 0x7f6aa8144630 secure :-1 s=THROTTLE_DONE pgs=169 cs=0 l=1 rx=0x7f6aa81421e0 tx=0x7f6a9800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:47.736 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.724+0000 7f6aa67fc700 1 -- 172.21.15.42:0/2751002626 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a98008bb0 con 0x7f6aa809f550 2019-12-05T05:03:47.736 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.724+0000 7f6aa67fc700 1 -- 172.21.15.42:0/2751002626 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6a98011950 con 0x7f6aa809f550 2019-12-05T05:03:47.736 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.724+0000 7f6aad7a8700 1 -- 172.21.15.42:0/2751002626 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6aa809f550 msgr2=0x7f6aa8144630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:47.737 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.724+0000 7f6aad7a8700 1 --2- 172.21.15.42:0/2751002626 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6aa809f550 0x7f6aa8144630 secure :-1 s=READY pgs=169 cs=0 l=1 rx=0x7f6aa81421e0 tx=0x7f6a9800fd30).stop 2019-12-05T05:03:47.737 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.724+0000 7f6aad7a8700 1 -- 172.21.15.42:0/2751002626 shutdown_connections 2019-12-05T05:03:47.737 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.724+0000 7f6aad7a8700 1 --2- 172.21.15.42:0/2751002626 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6aa814cf50 0x7f6aa814f390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:03:47.737 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.724+0000 7f6aad7a8700 1 --2- 172.21.15.42:0/2751002626 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6aa809f550 0x7f6aa8144630 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:47.737 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.728+0000 7f6aad7a8700 1 -- 172.21.15.42:0/2751002626 shutdown_connections 2019-12-05T05:03:47.737 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.728+0000 7f6aad7a8700 1 -- 172.21.15.42:0/2751002626 wait complete. 2019-12-05T05:03:47.737 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.728+0000 7f6aad7a8700 1 -- 172.21.15.42:0/2751002626 >> 172.21.15.42:0/2751002626 conn(0x7f6aa80a9ef0 msgr2=0x7f6aa80aa320 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:47.740 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.728+0000 7f6aad7a8700 1 Processor -- start 2019-12-05T05:03:47.741 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.728+0000 7f6aad7a8700 1 -- start start 2019-12-05T05:03:47.741 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.728+0000 7f6aad7a8700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f6aa812db10 0x7f6aa812ff50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:47.741 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.728+0000 7f6aad7a8700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6aa80a2000 con 0x7f6aa80a1bd0 2019-12-05T05:03:47.741 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.728+0000 7f6aad7a8700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6aa81170d0 con 0x7f6aa8116ca0 2019-12-05T05:03:47.741 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.728+0000 7f6aad7a8700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6aa8117740 con 0x7f6aa812db10 2019-12-05T05:03:47.742 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.728+0000 7f6aa6ffd700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f6aa8116ca0 0x7f6aa807d920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:53760/0 (socket says 172.21.15.42:53760) 2019-12-05T05:03:47.742 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.728+0000 7f6aa6ffd700 1 -- 172.21.15.42:0/572205505 learned_addr learned my addr 172.21.15.42:0/572205505 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:03:47.742 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.728+0000 7f6aa7fff700 1 --2- 172.21.15.42:0/572205505 >> v2:172.21.15.40:3300/0 conn(0x7f6aa812db10 0x7f6aa812ff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:47.742 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.728+0000 7f6a9ffff700 1 -- 172.21.15.42:0/572205505 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4065181712 0 0) 0x7f6aa81170d0 con 0x7f6aa8116ca0 2019-12-05T05:03:47.742 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.728+0000 7f6a9ffff700 1 -- 172.21.15.42:0/572205505 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a8c0030b0 con 0x7f6aa8116ca0 2019-12-05T05:03:47.742 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.732+0000 7f6aa7fff700 1 -- 172.21.15.42:0/572205505 >> v1:172.21.15.42:6789/0 conn(0x7f6aa8116ca0 legacy=0x7f6aa807d920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:47.743 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.732+0000 7f6aa7fff700 1 -- 172.21.15.42:0/572205505 >> v1:172.21.15.40:6789/0 conn(0x7f6aa80a1bd0 legacy=0x7f6aa8124100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:47.743 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.732+0000 7f6aa7fff700 1 -- 172.21.15.42:0/572205505 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6aa8117f20 con 0x7f6aa812db10 2019-12-05T05:03:47.743 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.732+0000 7f6aa7fff700 1 --2- 172.21.15.42:0/572205505 >> v2:172.21.15.40:3300/0 conn(0x7f6aa812db10 0x7f6aa812ff50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f6aa0002f30 tx=0x7f6aa0010800).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:47.743 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.732+0000 7f6aa7fff700 1 --2- 172.21.15.42:0/572205505 >> v2:172.21.15.40:3300/0 conn(0x7f6aa812db10 0x7f6aa812ff50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f6aa0002f30 tx=0x7f6aa0010800).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:47.743 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.732+0000 7f6aa7fff700 1 --2- 172.21.15.42:0/572205505 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6aa812db10 0x7f6aa812ff50 secure :-1 s=READY pgs=139 cs=0 l=1 rx=0x7f6aa0002f30 tx=0x7f6aa0010800).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:47.744 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.732+0000 7f6aa7fff700 1 --2- 172.21.15.42:0/572205505 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6aa812db10 0x7f6aa812ff50 secure :-1 s=THROTTLE_DONE pgs=139 cs=0 l=1 rx=0x7f6aa0002f30 tx=0x7f6aa0010800).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:47.744 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.732+0000 7f6a9ffff700 1 -- 172.21.15.42:0/572205505 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6aa0005b20 con 0x7f6aa812db10 2019-12-05T05:03:47.744 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.732+0000 7f6aa7fff700 1 --2- 172.21.15.42:0/572205505 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6aa812db10 0x7f6aa812ff50 secure :-1 s=THROTTLE_DONE pgs=139 cs=0 l=1 rx=0x7f6aa0002f30 tx=0x7f6aa0010800).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:47.744 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.732+0000 7f6aa7fff700 1 --2- 172.21.15.42:0/572205505 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6aa812db10 0x7f6aa812ff50 secure :-1 s=THROTTLE_DONE pgs=139 cs=0 l=1 rx=0x7f6aa0002f30 tx=0x7f6aa0010800).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:47.744 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.732+0000 7f6a9ffff700 1 -- 172.21.15.42:0/572205505 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6aa0007200 con 0x7f6aa812db10 2019-12-05T05:03:47.744 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.732+0000 7f6aad7a8700 1 -- 172.21.15.42:0/572205505 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6aa807e120 con 0x7f6aa812db10 2019-12-05T05:03:47.744 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.732+0000 7f6a9ffff700 1 -- 172.21.15.42:0/572205505 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6aa00062b0 con 0x7f6aa812db10 2019-12-05T05:03:47.745 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.732+0000 7f6aad7a8700 1 -- 172.21.15.42:0/572205505 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6aa81304a0 con 0x7f6aa812db10 2019-12-05T05:03:47.745 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.732+0000 7f6aa7fff700 1 --2- 172.21.15.42:0/572205505 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6aa812db10 0x7f6aa812ff50 secure :-1 s=THROTTLE_DONE pgs=139 cs=0 l=1 rx=0x7f6aa0002f30 tx=0x7f6aa0010800).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:47.745 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.732+0000 7f6a9ffff700 1 -- 172.21.15.42:0/572205505 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f6aa0007bb0 con 0x7f6aa812db10 2019-12-05T05:03:47.745 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.732+0000 7f6aa7fff700 1 --2- 172.21.15.42:0/572205505 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6aa812db10 0x7f6aa812ff50 secure :-1 s=THROTTLE_DONE pgs=139 cs=0 l=1 rx=0x7f6aa0002f30 tx=0x7f6aa0010800).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:47.745 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.732+0000 7f6a9ffff700 1 --2- 172.21.15.42:0/572205505 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6a8c03b280 0x7f6a8c03d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:47.745 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.732+0000 7f6a9ffff700 1 -- 172.21.15.42:0/572205505 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f6aa004a9e0 con 0x7f6aa812db10 2019-12-05T05:03:47.745 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.732+0000 7f6aa77fe700 1 --2- 172.21.15.42:0/572205505 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6a8c03b280 0x7f6a8c03d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:47.746 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.732+0000 7f6aa77fe700 1 --2- 172.21.15.42:0/572205505 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6a8c03b280 0x7f6a8c03d790 crc :-1 s=READY pgs=92 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:47.746 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.740+0000 7f6aad7a8700 1 -- 172.21.15.42:0/572205505 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6aa805d110 con 0x7f6aa812db10 2019-12-05T05:03:47.753 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.748+0000 7f6aa7fff700 1 --2- 172.21.15.42:0/572205505 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6aa812db10 0x7f6aa812ff50 secure :-1 s=THROTTLE_DONE pgs=139 cs=0 l=1 rx=0x7f6aa0002f30 tx=0x7f6aa0010800).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:47.753 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.748+0000 7f6a9ffff700 1 -- 172.21.15.42:0/572205505 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f6aa0031a60 con 0x7f6aa812db10 2019-12-05T05:03:47.919 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.912+0000 7f6aad7a8700 1 -- 172.21.15.42:0/572205505 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7f6aa805b8c0 con 0x7f6aa812db10 2019-12-05T05:03:47.919 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.912+0000 7f6aa7fff700 1 --2- 172.21.15.42:0/572205505 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6aa812db10 0x7f6aa812ff50 secure :-1 s=THROTTLE_DONE pgs=139 cs=0 l=1 rx=0x7f6aa0002f30 tx=0x7f6aa0010800).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:47.920 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.916+0000 7f6a9ffff700 1 -- 172.21.15.42:0/572205505 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v24) v1 ==== 100+0+10 (secure 0 0 0) 0x7f6aa0031cc0 con 0x7f6aa812db10 2019-12-05T05:03:47.922 INFO:teuthology.orchestra.run.smithi042.stdout:pg_num: 8 2019-12-05T05:03:47.925 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.920+0000 7f6aad7a8700 1 -- 172.21.15.42:0/572205505 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6a8c03b280 msgr2=0x7f6a8c03d790 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:47.925 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.920+0000 7f6aad7a8700 1 --2- 172.21.15.42:0/572205505 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6a8c03b280 0x7f6a8c03d790 crc :-1 s=READY pgs=92 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:47.925 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.920+0000 7f6aad7a8700 1 -- 172.21.15.42:0/572205505 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6aa812db10 msgr2=0x7f6aa812ff50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:47.925 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.920+0000 7f6aad7a8700 1 --2- 172.21.15.42:0/572205505 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6aa812db10 0x7f6aa812ff50 secure :-1 s=READY pgs=139 cs=0 l=1 rx=0x7f6aa0002f30 tx=0x7f6aa0010800).stop 2019-12-05T05:03:47.925 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.920+0000 7f6aad7a8700 1 -- 172.21.15.42:0/572205505 shutdown_connections 2019-12-05T05:03:47.926 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.920+0000 7f6aad7a8700 1 --2- 172.21.15.42:0/572205505 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6a8c03b280 0x7f6a8c03d790 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:47.927 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.920+0000 7f6aad7a8700 1 --2- 172.21.15.42:0/572205505 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6aa812db10 0x7f6aa812ff50 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:47.927 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.920+0000 7f6aad7a8700 1 -- 172.21.15.42:0/572205505 shutdown_connections 2019-12-05T05:03:47.927 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.920+0000 7f6aad7a8700 1 -- 172.21.15.42:0/572205505 wait complete. 2019-12-05T05:03:47.927 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:47.920+0000 7f6aad7a8700 1 -- 172.21.15.42:0/572205505 >> 172.21.15.42:0/572205505 conn(0x7f6aa80a7420 msgr2=0x7f6aa807e650 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:47.934 INFO:tasks.mds_thrash:mds thrasher using random seed: 1575522227 2019-12-05T05:03:47.935 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2019-12-05T05:03:48.028 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.016+0000 7fc9c6541700 1 Processor -- start 2019-12-05T05:03:48.028 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.020+0000 7fc9c6541700 1 -- start start 2019-12-05T05:03:48.028 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.020+0000 7fc9c6541700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc9c008df80 0x7fc9c009f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:48.028 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.020+0000 7fc9c6541700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9c009fc60 con 0x7fc9c00a7740 2019-12-05T05:03:48.028 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.020+0000 7fc9c6541700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9c00a0710 con 0x7fc9c009f830 2019-12-05T05:03:48.029 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.020+0000 7fc9c6541700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9c00a0b00 con 0x7fc9c008df80 2019-12-05T05:03:48.029 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.020+0000 7fc9bffff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc9c008df80 0x7fc9c009f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:48.029 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.020+0000 7fc9bffff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc9c008df80 0x7fc9c009f2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:53730/0 (socket says 172.21.15.42:53730) 2019-12-05T05:03:48.029 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.020+0000 7fc9bffff700 1 -- 172.21.15.42:0/4058885166 learned_addr learned my addr 172.21.15.42:0/4058885166 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:03:48.029 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.020+0000 7fc9c52df700 1 --1- 172.21.15.42:0/4058885166 >> v1:172.21.15.42:6789/0 conn(0x7fc9c009f830 0x7fc9c014efe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:53770/0 (socket says 172.21.15.42:53770) 2019-12-05T05:03:48.029 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.020+0000 7fc9bffff700 1 -- 172.21.15.42:0/4058885166 >> v1:172.21.15.42:6789/0 conn(0x7fc9c009f830 legacy=0x7fc9c014efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:48.029 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.020+0000 7fc9bffff700 1 -- 172.21.15.42:0/4058885166 >> v1:172.21.15.40:6789/0 conn(0x7fc9c00a7740 legacy=0x7fc9c007e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:48.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.020+0000 7fc9bffff700 1 -- 172.21.15.42:0/4058885166 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9c007ef20 con 0x7fc9c008df80 2019-12-05T05:03:48.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.020+0000 7fc9bffff700 1 --2- 172.21.15.42:0/4058885166 >> v2:172.21.15.42:3300/0 conn(0x7fc9c008df80 0x7fc9c009f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fc9ac008790 tx=0x7fc9ac00fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:48.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.020+0000 7fc9bffff700 1 --2- 172.21.15.42:0/4058885166 >> v2:172.21.15.42:3300/0 conn(0x7fc9c008df80 0x7fc9c009f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fc9ac008790 tx=0x7fc9ac00fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:48.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.020+0000 7fc9bffff700 1 --2- 172.21.15.42:0/4058885166 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc9c008df80 0x7fc9c009f2e0 secure :-1 s=READY pgs=172 cs=0 l=1 rx=0x7fc9ac008790 tx=0x7fc9ac00fcb0).ready entity=mon.1 client_cookie=e78f80e15e72e8b server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:48.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.020+0000 7fc9bffff700 1 --2- 172.21.15.42:0/4058885166 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc9c008df80 0x7fc9c009f2e0 secure :-1 s=THROTTLE_DONE pgs=172 cs=0 l=1 rx=0x7fc9ac008790 tx=0x7fc9ac00fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:48.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.020+0000 7fc9bf7fe700 1 -- 172.21.15.42:0/4058885166 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc9ac011520 con 0x7fc9c008df80 2019-12-05T05:03:48.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.020+0000 7fc9bffff700 1 --2- 172.21.15.42:0/4058885166 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc9c008df80 0x7fc9c009f2e0 secure :-1 s=THROTTLE_DONE pgs=172 cs=0 l=1 rx=0x7fc9ac008790 tx=0x7fc9ac00fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:48.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.020+0000 7fc9bffff700 1 --2- 172.21.15.42:0/4058885166 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc9c008df80 0x7fc9c009f2e0 secure :-1 s=THROTTLE_DONE pgs=172 cs=0 l=1 rx=0x7fc9ac008790 tx=0x7fc9ac00fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:48.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.020+0000 7fc9bf7fe700 1 -- 172.21.15.42:0/4058885166 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9ac008ca0 con 0x7fc9c008df80 2019-12-05T05:03:48.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.020+0000 7fc9bf7fe700 1 -- 172.21.15.42:0/4058885166 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc9ac011b00 con 0x7fc9c008df80 2019-12-05T05:03:48.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.020+0000 7fc9c6541700 1 -- 172.21.15.42:0/4058885166 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc9c008df80 msgr2=0x7fc9c009f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:48.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.020+0000 7fc9c6541700 1 --2- 172.21.15.42:0/4058885166 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc9c008df80 0x7fc9c009f2e0 secure :-1 s=READY pgs=172 cs=0 l=1 rx=0x7fc9ac008790 tx=0x7fc9ac00fcb0).stop 2019-12-05T05:03:48.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.020+0000 7fc9c6541700 1 -- 172.21.15.42:0/4058885166 shutdown_connections 2019-12-05T05:03:48.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.020+0000 7fc9c6541700 1 --2- 172.21.15.42:0/4058885166 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc9c008df80 0x7fc9c009f2e0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:48.032 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.020+0000 7fc9c6541700 1 -- 172.21.15.42:0/4058885166 shutdown_connections 2019-12-05T05:03:48.032 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.020+0000 7fc9c6541700 1 -- 172.21.15.42:0/4058885166 wait complete. 2019-12-05T05:03:48.032 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.020+0000 7fc9c6541700 1 -- 172.21.15.42:0/4058885166 >> 172.21.15.42:0/4058885166 conn(0x7fc9c00aabe0 msgr2=0x7fc9c00a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:48.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.024+0000 7fc9c6541700 1 Processor -- start 2019-12-05T05:03:48.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.024+0000 7fc9c6541700 1 -- start start 2019-12-05T05:03:48.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.024+0000 7fc9c6541700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc9c00a6af0 0x7fc9c0143f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:48.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.024+0000 7fc9c6541700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc9c0119a00 0x7fc9c014f340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:48.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.024+0000 7fc9c6541700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9c00a5c90 con 0x7fc9c014f890 2019-12-05T05:03:48.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.024+0000 7fc9c6541700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9c0143d30 con 0x7fc9c0119a00 2019-12-05T05:03:48.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.024+0000 7fc9c6541700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9c0142f10 con 0x7fc9c00a6af0 2019-12-05T05:03:48.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.028+0000 7fc9c4ade700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc9c00a6af0 0x7fc9c0143f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:48.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.028+0000 7fc9c4ade700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc9c00a6af0 0x7fc9c0143f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:53734/0 (socket says 172.21.15.42:53734) 2019-12-05T05:03:48.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.028+0000 7fc9c4ade700 1 -- 172.21.15.42:0/2784024391 learned_addr learned my addr 172.21.15.42:0/2784024391 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:03:48.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.028+0000 7fc9bffff700 1 --2- 172.21.15.42:0/2784024391 >> v2:172.21.15.40:3300/0 conn(0x7fc9c0119a00 0x7fc9c014f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:48.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.028+0000 7fc9c4ade700 1 -- 172.21.15.42:0/2784024391 >> v2:172.21.15.40:3300/0 conn(0x7fc9c0119a00 msgr2=0x7fc9c014f340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:48.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.028+0000 7fc9c4ade700 1 --2- 172.21.15.42:0/2784024391 >> v2:172.21.15.40:3300/0 conn(0x7fc9c0119a00 0x7fc9c014f340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:48.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.028+0000 7fc9c4ade700 1 -- 172.21.15.42:0/2784024391 >> v1:172.21.15.40:6789/0 conn(0x7fc9c014f890 legacy=0x7fc9c00a4840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:48.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.028+0000 7fc9c4ade700 1 -- 172.21.15.42:0/2784024391 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9c01452b0 con 0x7fc9c00a6af0 2019-12-05T05:03:48.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.028+0000 7fc9c4ade700 1 --2- 172.21.15.42:0/2784024391 >> v2:172.21.15.42:3300/0 conn(0x7fc9c00a6af0 0x7fc9c0143f90 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fc9b4000e90 tx=0x7fc9b40109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:48.039 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.028+0000 7fc9c4ade700 1 --2- 172.21.15.42:0/2784024391 >> v2:172.21.15.42:3300/0 conn(0x7fc9c00a6af0 0x7fc9c0143f90 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fc9b4000e90 tx=0x7fc9b40109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:48.039 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.028+0000 7fc9c4ade700 1 --2- 172.21.15.42:0/2784024391 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc9c00a6af0 0x7fc9c0143f90 secure :-1 s=READY pgs=173 cs=0 l=1 rx=0x7fc9b4000e90 tx=0x7fc9b40109a0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:48.039 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.028+0000 7fc9c4ade700 1 --2- 172.21.15.42:0/2784024391 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc9c00a6af0 0x7fc9c0143f90 secure :-1 s=THROTTLE_DONE pgs=173 cs=0 l=1 rx=0x7fc9b4000e90 tx=0x7fc9b40109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:48.039 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.028+0000 7fc9c4ade700 1 --2- 172.21.15.42:0/2784024391 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc9c00a6af0 0x7fc9c0143f90 secure :-1 s=THROTTLE_DONE pgs=173 cs=0 l=1 rx=0x7fc9b4000e90 tx=0x7fc9b40109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:48.039 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.028+0000 7fc9bd7fa700 1 -- 172.21.15.42:0/2784024391 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc9b400caa0 con 0x7fc9c00a6af0 2019-12-05T05:03:48.039 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.028+0000 7fc9c4ade700 1 --2- 172.21.15.42:0/2784024391 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc9c00a6af0 0x7fc9c0143f90 secure :-1 s=THROTTLE_DONE pgs=173 cs=0 l=1 rx=0x7fc9b4000e90 tx=0x7fc9b40109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:48.039 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.028+0000 7fc9bd7fa700 1 -- 172.21.15.42:0/2784024391 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9b400c6c0 con 0x7fc9c00a6af0 2019-12-05T05:03:48.040 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.028+0000 7fc9c6541700 1 -- 172.21.15.42:0/2784024391 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9c0142530 con 0x7fc9c00a6af0 2019-12-05T05:03:48.040 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.028+0000 7fc9bd7fa700 1 -- 172.21.15.42:0/2784024391 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc9b4012210 con 0x7fc9c00a6af0 2019-12-05T05:03:48.040 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.028+0000 7fc9c6541700 1 -- 172.21.15.42:0/2784024391 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9c01444e0 con 0x7fc9c00a6af0 2019-12-05T05:03:48.040 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.028+0000 7fc9c4ade700 1 --2- 172.21.15.42:0/2784024391 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc9c00a6af0 0x7fc9c0143f90 secure :-1 s=THROTTLE_DONE pgs=173 cs=0 l=1 rx=0x7fc9b4000e90 tx=0x7fc9b40109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:48.040 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.028+0000 7fc9bd7fa700 1 -- 172.21.15.42:0/2784024391 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fc9b4012520 con 0x7fc9c00a6af0 2019-12-05T05:03:48.040 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.028+0000 7fc9c4ade700 1 --2- 172.21.15.42:0/2784024391 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc9c00a6af0 0x7fc9c0143f90 secure :-1 s=THROTTLE_DONE pgs=173 cs=0 l=1 rx=0x7fc9b4000e90 tx=0x7fc9b40109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:48.040 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.028+0000 7fc9bd7fa700 1 --2- 172.21.15.42:0/2784024391 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc9a003ad20 0x7fc9a003d230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:48.041 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.028+0000 7fc9bd7fa700 1 -- 172.21.15.42:0/2784024391 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6031+0+0 (secure 0 0 0) 0x7fc9b404b2f0 con 0x7fc9c00a6af0 2019-12-05T05:03:48.041 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.028+0000 7fc9bffff700 1 --2- 172.21.15.42:0/2784024391 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc9a003ad20 0x7fc9a003d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:48.041 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.028+0000 7fc9bffff700 1 --2- 172.21.15.42:0/2784024391 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc9a003ad20 0x7fc9a003d230 crc :-1 s=READY pgs=93 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:48.041 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.032+0000 7fc9c6541700 1 -- 172.21.15.42:0/2784024391 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9c005d110 con 0x7fc9c00a6af0 2019-12-05T05:03:48.045 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.040+0000 7fc9c4ade700 1 --2- 172.21.15.42:0/2784024391 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc9c00a6af0 0x7fc9c0143f90 secure :-1 s=THROTTLE_DONE pgs=173 cs=0 l=1 rx=0x7fc9b4000e90 tx=0x7fc9b40109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:48.045 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.040+0000 7fc9bd7fa700 1 -- 172.21.15.42:0/2784024391 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fc9b4032370 con 0x7fc9c00a6af0 2019-12-05T05:03:48.207 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.200+0000 7fc9c6541700 1 -- 172.21.15.42:0/2784024391 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc9c005b8c0 con 0x7fc9c00a6af0 2019-12-05T05:03:48.209 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.204+0000 7fc9c4ade700 1 --2- 172.21.15.42:0/2784024391 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc9c00a6af0 0x7fc9c0143f90 secure :-1 s=THROTTLE_DONE pgs=173 cs=0 l=1 rx=0x7fc9b4000e90 tx=0x7fc9b40109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:48.209 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.204+0000 7fc9bd7fa700 1 -- 172.21.15.42:0/2784024391 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+14885 (secure 0 0 0) 0x7fc9b40325d0 con 0x7fc9c00a6af0 2019-12-05T05:03:48.210 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:03:48.211 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":24,"fsid":"5d836cfc-a83b-4433-9b3e-b9b2f31c9050","created":"2019-12-05T05:03:13.215602+0000","modified":"2019-12-05T05:03:36.607704+0000","last_up_change":"2019-12-05T05:03:22.664783+0000","last_in_change":"2019-12-05T05:03:22.664783+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2019-12-05T05:03:27.394605+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2019-12-05T05:03:30.419845+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2019-12-05T05:03:33.249075+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e3fc49bb-dcbc-43ff-ac7c-9eb0ea0bd257","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6810","nonce":12118},{"type":"v1","addr":"172.21.15.42:6811","nonce":12118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6812","nonce":12118},{"type":"v1","addr":"172.21.15.42:6813","nonce":12118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6816","nonce":12118},{"type":"v1","addr":"172.21.15.42:6817","nonce":12118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6814","nonce":12118},{"type":"v1","addr":"172.21.15.42:6815","nonce":12118}]},"public_addr":"172.21.15.42:6811/12118","cluster_addr":"172.21.15.42:6813/12118","heartbeat_back_addr":"172.21.15.42:6817/12118","heartbeat_front_addr":"172.21.15.42:6815/12118","state":["exists","up"]},{"osd":1,"uuid":"902543fb-f5a2-4321-9c9c-26f9b9ca8629","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6826","nonce":12119},{"type":"v1","addr":"172.21.15.42:6827","nonce":12119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6828","nonce":12119},{"type":"v1","addr":"172.21.15.42:6829","nonce":12119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6832","nonce":12119},{"type":"v1","addr":"172.21.15.42:6833","nonce":12119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6830","nonce":12119},{"type":"v1","addr":"172.21.15.42:6831","nonce":12119}]},"public_addr":"172.21.15.42:6827/12119","cluster_addr":"172.21.15.42:6829/12119","heartbeat_back_addr":"172.21.15.42:6833/12119","heartbeat_front_addr":"172.21.15.42:6831/12119","state":["exists","up"]},{"osd":2,"uuid":"28739ff7-bed7-4b75-8ea5-b57d088f1250","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6802","nonce":12117},{"type":"v1","addr":"172.21.15.42:6803","nonce":12117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6804","nonce":12117},{"type":"v1","addr":"172.21.15.42:6805","nonce":12117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6808","nonce":12117},{"type":"v1","addr":"172.21.15.42:6809","nonce":12117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6806","nonce":12117},{"type":"v1","addr":"172.21.15.42:6807","nonce":12117}]},"public_addr":"172.21.15.42:6803/12117","cluster_addr":"172.21.15.42:6805/12117","heartbeat_back_addr":"172.21.15.42:6809/12117","heartbeat_front_addr":"172.21.15.42:6807/12117","state":["exists","up"]},{"osd":3,"uuid":"1960f66b-2d40-4bb9-a289-ce8cd0c1c142","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6818","nonce":12116},{"type":"v1","addr":"172.21.15.42:6819","nonce":12116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6820","nonce":12116},{"type":"v1","addr":"172.21.15.42:6821","nonce":12116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6824","nonce":12116},{"type":"v1","addr":"172.21.15.42:6825","nonce":12116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6822","nonce":12116},{"type":"v1","addr":"172.21.15.42:6823","nonce":12116}]},"public_addr":"172.21.15.42:6819/12116","cluster_addr":"172.21.15.42:6821/12116","heartbeat_back_addr":"172.21.15.42:6825/12116","heartbeat_front_addr":"172.21.15.42:6823/12116","state":["exists","up"]},{"osd":4,"uuid":"45bbf1bd-7c8a-4a0d-b76b-afa6148a855c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":12488},{"type":"v1","addr":"172.21.15.40:6809","nonce":12488}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":12488},{"type":"v1","addr":"172.21.15.40:6812","nonce":12488}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":12488},{"type":"v1","addr":"172.21.15.40:6825","nonce":12488}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":12488},{"type":"v1","addr":"172.21.15.40:6819","nonce":12488}]},"public_addr":"172.21.15.40:6809/12488","cluster_addr":"172.21.15.40:6812/12488","heartbeat_back_addr":"172.21.15.40:6825/12488","heartbeat_front_addr":"172.21.15.40:6819/12488","state":["exists","up"]},{"osd":5,"uuid":"8a5511cd-925f-4869-a49e-924383e4ce53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":12489},{"type":"v1","addr":"172.21.15.40:6801","nonce":12489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":12489},{"type":"v1","addr":"172.21.15.40:6803","nonce":12489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":12489},{"type":"v1","addr":"172.21.15.40:6807","nonce":12489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":12489},{"type":"v1","addr":"172.21.15.40:6805","nonce":12489}]},"public_addr":"172.21.15.40:6801/12489","cluster_addr":"172.21.15.40:6803/12489","heartbeat_back_addr":"172.21.15.40:6807/12489","heartbeat_front_addr":"172.21.15.40:6805/12489","state":["exists","up"]},{"osd":6,"uuid":"d692c179-379e-446c-8484-a219a9241e04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6811","nonce":12491},{"type":"v1","addr":"172.21.15.40:6814","nonce":12491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6817","nonce":12491},{"type":"v1","addr":"172.21.15.40:6818","nonce":12491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":12491},{"type":"v1","addr":"172.21.15.40:6828","nonce":12491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":12491},{"type":"v1","addr":"172.21.15.40:6824","nonce":12491}]},"public_addr":"172.21.15.40:6814/12491","cluster_addr":"172.21.15.40:6818/12491","heartbeat_back_addr":"172.21.15.40:6828/12491","heartbeat_front_addr":"172.21.15.40:6824/12491","state":["exists","up"]},{"osd":7,"uuid":"79c3c243-190e-4e35-b769-319a2077e7cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6813","nonce":12490},{"type":"v1","addr":"172.21.15.40:6816","nonce":12490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":12490},{"type":"v1","addr":"172.21.15.40:6823","nonce":12490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":12490},{"type":"v1","addr":"172.21.15.40:6831","nonce":12490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6827","nonce":12490},{"type":"v1","addr":"172.21.15.40:6829","nonce":12490}]},"public_addr":"172.21.15.40:6816/12490","cluster_addr":"172.21.15.40:6823/12490","heartbeat_back_addr":"172.21.15.40:6831/12490","heartbeat_front_addr":"172.21.15.40:6829/12490","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T05:03:21.349150+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T05:03:21.644281+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T05:03:21.394040+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T05:03:21.587096+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T05:03:21.656497+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T05:03:21.510547+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T05:03:21.659358+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T05:03:21.628063+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2019-12-05T05:03:48.213 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.208+0000 7fc9c6541700 1 -- 172.21.15.42:0/2784024391 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc9a003ad20 msgr2=0x7fc9a003d230 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:48.214 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.208+0000 7fc9c6541700 1 --2- 172.21.15.42:0/2784024391 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc9a003ad20 0x7fc9a003d230 crc :-1 s=READY pgs=93 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:48.214 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.208+0000 7fc9c6541700 1 -- 172.21.15.42:0/2784024391 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc9c00a6af0 msgr2=0x7fc9c0143f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:48.214 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.208+0000 7fc9c6541700 1 --2- 172.21.15.42:0/2784024391 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc9c00a6af0 0x7fc9c0143f90 secure :-1 s=READY pgs=173 cs=0 l=1 rx=0x7fc9b4000e90 tx=0x7fc9b40109a0).stop 2019-12-05T05:03:48.214 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.208+0000 7fc9c6541700 1 -- 172.21.15.42:0/2784024391 shutdown_connections 2019-12-05T05:03:48.214 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.208+0000 7fc9c6541700 1 --2- 172.21.15.42:0/2784024391 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc9a003ad20 0x7fc9a003d230 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:48.214 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.208+0000 7fc9c6541700 1 --2- 172.21.15.42:0/2784024391 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc9c00a6af0 0x7fc9c0143f90 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:48.214 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.208+0000 7fc9c6541700 1 --2- 172.21.15.42:0/2784024391 >> v2:172.21.15.40:3300/0 conn(0x7fc9c0119a00 0x7fc9c014f340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:48.215 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.208+0000 7fc9c6541700 1 -- 172.21.15.42:0/2784024391 shutdown_connections 2019-12-05T05:03:48.215 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.208+0000 7fc9c6541700 1 -- 172.21.15.42:0/2784024391 wait complete. 2019-12-05T05:03:48.215 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.208+0000 7fc9c6541700 1 -- 172.21.15.42:0/2784024391 >> 172.21.15.42:0/2784024391 conn(0x7fc9c008fda0 msgr2=0x7fc9c007e720 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:48.224 INFO:tasks.mds_thrash.ceph_manager:[{u'cache_target_full_ratio_micro': 800000, u'fast_read': False, u'stripe_width': 0, u'flags_names': u'hashpspool', u'tier_of': -1, u'hit_set_grade_decay_rate': 0, u'pg_placement_num': 8, u'use_gmt_hitset': True, u'cache_target_dirty_ratio_micro': 400000, u'create_time': u'2019-12-05T05:03:27.394605+0000', u'snap_epoch': 0, u'erasure_code_profile': u'', u'pg_autoscale_mode': u'off', u'snap_seq': 0, u'expected_num_objects': 0, u'size': 2, u'pg_num_pending': 8, u'auid': 0, u'cache_min_flush_age': 0, u'hit_set_period': 0, u'min_read_recency_for_promote': 0, u'target_max_objects': 0, u'pg_placement_num_target': 8, u'pg_num': 8, u'type': 1, u'grade_table': [], u'pool_name': u'rbd', u'cache_min_evict_age': 0, u'snap_mode': u'selfmanaged', u'pg_num_target': 8, u'cache_mode': u'none', u'min_size': 1, u'cache_target_dirty_high_ratio_micro': 600000, u'object_hash': 2, u'last_pg_merge_meta': {u'ready_epoch': 0, u'source_version': u"0'0", u'source_pgid': u'0.0', u'last_epoch_clean': 0, u'target_version': u"0'0", u'last_epoch_started': 0}, u'write_tier': -1, u'last_force_op_resend_preluminous': u'0', u'pool': 1, u'removed_snaps': u'[]', u'crush_rule': 0, u'tiers': [], u'hit_set_params': {u'type': u'none'}, u'last_force_op_resend': u'0', u'pool_snaps': [], u'quota_max_objects': 0, u'last_force_op_resend_prenautilus': u'0', u'application_metadata': {u'rbd': {}}, u'options': {}, u'hit_set_count': 0, u'flags': 1, u'target_max_bytes': 0, u'quota_max_bytes': 0, u'hit_set_search_last_n': 0, u'last_change': u'17', u'min_write_recency_for_promote': 0, u'read_tier': -1}, {u'cache_target_full_ratio_micro': 800000, u'fast_read': False, u'stripe_width': 0, u'flags_names': u'hashpspool', u'tier_of': -1, u'hit_set_grade_decay_rate': 0, u'pg_placement_num': 8, u'use_gmt_hitset': True, u'cache_target_dirty_ratio_micro': 400000, u'create_time': u'2019-12-05T05:03:30.419845+0000', u'snap_epoch': 0, u'erasure_code_profile': u'', u'pg_autoscale_mode': u'off', u'snap_seq': 0, u'expected_num_objects': 0, u'size': 2, u'pg_num_pending': 8, u'auid': 0, u'cache_min_flush_age': 0, u'hit_set_period': 0, u'min_read_recency_for_promote': 0, u'target_max_objects': 0, u'pg_placement_num_target': 8, u'pg_num': 8, u'type': 1, u'grade_table': [], u'pool_name': u'cephfs_metadata', u'cache_min_evict_age': 0, u'snap_mode': u'selfmanaged', u'pg_num_target': 8, u'cache_mode': u'none', u'min_size': 1, u'cache_target_dirty_high_ratio_micro': 600000, u'object_hash': 2, u'last_pg_merge_meta': {u'ready_epoch': 0, u'source_version': u"0'0", u'source_pgid': u'0.0', u'last_epoch_clean': 0, u'target_version': u"0'0", u'last_epoch_started': 0}, u'write_tier': -1, u'last_force_op_resend_preluminous': u'0', u'pool': 2, u'removed_snaps': u'[]', u'crush_rule': 0, u'tiers': [], u'hit_set_params': {u'type': u'none'}, u'last_force_op_resend': u'0', u'pool_snaps': [], u'quota_max_objects': 0, u'last_force_op_resend_prenautilus': u'0', u'application_metadata': {u'cephfs': {u'metadata': u'cephfs'}}, u'options': {u'pg_autoscale_bias': 4, u'recovery_priority': 5, u'pg_num_min': 16}, u'hit_set_count': 0, u'flags': 1, u'target_max_bytes': 0, u'quota_max_bytes': 0, u'hit_set_search_last_n': 0, u'last_change': u'23', u'min_write_recency_for_promote': 0, u'read_tier': -1}, {u'cache_target_full_ratio_micro': 800000, u'fast_read': False, u'stripe_width': 8192, u'flags_names': u'hashpspool,ec_overwrites', u'tier_of': -1, u'hit_set_grade_decay_rate': 0, u'pg_placement_num': 8, u'use_gmt_hitset': True, u'cache_target_dirty_ratio_micro': 400000, u'create_time': u'2019-12-05T05:03:33.249075+0000', u'snap_epoch': 0, u'erasure_code_profile': u'cephfs_data', u'pg_autoscale_mode': u'off', u'snap_seq': 0, u'expected_num_objects': 0, u'size': 4, u'pg_num_pending': 8, u'auid': 0, u'cache_min_flush_age': 0, u'hit_set_period': 0, u'min_read_recency_for_promote': 0, u'target_max_objects': 0, u'pg_placement_num_target': 8, u'pg_num': 8, u'type': 3, u'grade_table': [], u'pool_name': u'cephfs_data', u'cache_min_evict_age': 0, u'snap_mode': u'selfmanaged', u'pg_num_target': 8, u'cache_mode': u'none', u'min_size': 3, u'cache_target_dirty_high_ratio_micro': 600000, u'object_hash': 2, u'last_pg_merge_meta': {u'ready_epoch': 0, u'source_version': u"0'0", u'source_pgid': u'0.0', u'last_epoch_clean': 0, u'target_version': u"0'0", u'last_epoch_started': 0}, u'write_tier': -1, u'last_force_op_resend_preluminous': u'0', u'pool': 3, u'removed_snaps': u'[]', u'crush_rule': 1, u'tiers': [], u'hit_set_params': {u'type': u'none'}, u'last_force_op_resend': u'0', u'pool_snaps': [], u'quota_max_objects': 0, u'last_force_op_resend_prenautilus': u'0', u'application_metadata': {u'cephfs': {u'data': u'cephfs'}}, u'options': {}, u'hit_set_count': 0, u'flags': 5, u'target_max_bytes': 0, u'quota_max_bytes': 0, u'hit_set_search_last_n': 0, u'last_change': u'24', u'min_write_recency_for_promote': 0, u'read_tier': -1}] 2019-12-05T05:03:48.224 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2019-12-05T05:03:48.343 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.324+0000 7fa9495bf700 1 Processor -- start 2019-12-05T05:03:48.343 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.324+0000 7fa9495bf700 1 -- start start 2019-12-05T05:03:48.343 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.324+0000 7fa9495bf700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fa94414df60 0x7fa9441503a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:48.343 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.324+0000 7fa9495bf700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa94407f080 con 0x7fa9440a9a00 2019-12-05T05:03:48.344 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.324+0000 7fa9495bf700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9440a1000 con 0x7fa94409f550 2019-12-05T05:03:48.344 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.324+0000 7fa9495bf700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9440a8f80 con 0x7fa94414df60 2019-12-05T05:03:48.344 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.324+0000 7fa9437fe700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fa94409f550 0x7fa944144630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:53780/0 (socket says 172.21.15.42:53780) 2019-12-05T05:03:48.344 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.324+0000 7fa9437fe700 1 -- 172.21.15.42:0/2487727181 learned_addr learned my addr 172.21.15.42:0/2487727181 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:03:48.344 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.324+0000 7fa943fff700 1 --2- 172.21.15.42:0/2487727181 >> v2:172.21.15.42:3300/0 conn(0x7fa94414df60 0x7fa9441503a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:48.344 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.328+0000 7fa9427fc700 1 -- 172.21.15.42:0/2487727181 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1164452534 0 0) 0x7fa9440a1000 con 0x7fa94409f550 2019-12-05T05:03:48.344 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.328+0000 7fa9427fc700 1 -- 172.21.15.42:0/2487727181 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9300030b0 con 0x7fa94409f550 2019-12-05T05:03:48.345 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.328+0000 7fa9427fc700 1 -- 172.21.15.42:0/2487727181 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3708585698 0 0) 0x7fa94407f080 con 0x7fa9440a9a00 2019-12-05T05:03:48.345 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.328+0000 7fa9427fc700 1 -- 172.21.15.42:0/2487727181 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9440a1000 con 0x7fa9440a9a00 2019-12-05T05:03:48.345 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.328+0000 7fa943fff700 1 -- 172.21.15.42:0/2487727181 >> v1:172.21.15.42:6789/0 conn(0x7fa94409f550 legacy=0x7fa944144630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:48.345 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.328+0000 7fa943fff700 1 -- 172.21.15.42:0/2487727181 >> v1:172.21.15.40:6789/0 conn(0x7fa9440a9a00 legacy=0x7fa94407e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:48.346 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.328+0000 7fa943fff700 1 -- 172.21.15.42:0/2487727181 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa94409ee30 con 0x7fa94414df60 2019-12-05T05:03:48.346 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.328+0000 7fa943fff700 1 --2- 172.21.15.42:0/2487727181 >> v2:172.21.15.42:3300/0 conn(0x7fa94414df60 0x7fa9441503a0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fa934002f30 tx=0x7fa93400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:48.346 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.328+0000 7fa943fff700 1 --2- 172.21.15.42:0/2487727181 >> v2:172.21.15.42:3300/0 conn(0x7fa94414df60 0x7fa9441503a0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fa934002f30 tx=0x7fa93400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:48.346 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.328+0000 7fa943fff700 1 --2- 172.21.15.42:0/2487727181 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa94414df60 0x7fa9441503a0 secure :-1 s=READY pgs=175 cs=0 l=1 rx=0x7fa934002f30 tx=0x7fa93400fbf0).ready entity=mon.1 client_cookie=72baf52ad5819d5c server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:48.347 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.328+0000 7fa943fff700 1 --2- 172.21.15.42:0/2487727181 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa94414df60 0x7fa9441503a0 secure :-1 s=THROTTLE_DONE pgs=175 cs=0 l=1 rx=0x7fa934002f30 tx=0x7fa93400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:48.347 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.328+0000 7fa9427fc700 1 -- 172.21.15.42:0/2487727181 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa934012060 con 0x7fa94414df60 2019-12-05T05:03:48.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.328+0000 7fa943fff700 1 --2- 172.21.15.42:0/2487727181 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa94414df60 0x7fa9441503a0 secure :-1 s=THROTTLE_DONE pgs=175 cs=0 l=1 rx=0x7fa934002f30 tx=0x7fa93400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:48.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.328+0000 7fa943fff700 1 --2- 172.21.15.42:0/2487727181 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa94414df60 0x7fa9441503a0 secure :-1 s=THROTTLE_DONE pgs=175 cs=0 l=1 rx=0x7fa934002f30 tx=0x7fa93400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:48.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.328+0000 7fa9427fc700 1 -- 172.21.15.42:0/2487727181 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9340110c0 con 0x7fa94414df60 2019-12-05T05:03:48.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.328+0000 7fa9427fc700 1 -- 172.21.15.42:0/2487727181 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa934008d60 con 0x7fa94414df60 2019-12-05T05:03:48.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.328+0000 7fa9495bf700 1 -- 172.21.15.42:0/2487727181 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa94414df60 msgr2=0x7fa9441503a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:48.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.328+0000 7fa9495bf700 1 --2- 172.21.15.42:0/2487727181 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa94414df60 0x7fa9441503a0 secure :-1 s=READY pgs=175 cs=0 l=1 rx=0x7fa934002f30 tx=0x7fa93400fbf0).stop 2019-12-05T05:03:48.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.328+0000 7fa9495bf700 1 -- 172.21.15.42:0/2487727181 shutdown_connections 2019-12-05T05:03:48.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.328+0000 7fa9495bf700 1 --2- 172.21.15.42:0/2487727181 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa94414df60 0x7fa9441503a0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:48.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.328+0000 7fa9495bf700 1 -- 172.21.15.42:0/2487727181 shutdown_connections 2019-12-05T05:03:48.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.328+0000 7fa9495bf700 1 -- 172.21.15.42:0/2487727181 wait complete. 2019-12-05T05:03:48.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.328+0000 7fa9495bf700 1 -- 172.21.15.42:0/2487727181 >> 172.21.15.42:0/2487727181 conn(0x7fa9440a9ef0 msgr2=0x7fa9440aa320 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:48.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.332+0000 7fa9495bf700 1 Processor -- start 2019-12-05T05:03:48.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.332+0000 7fa9495bf700 1 -- start start 2019-12-05T05:03:48.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.332+0000 7fa9495bf700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa9440aa690 0x7fa944143cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:48.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.332+0000 7fa9495bf700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa944126940 0x7fa944126d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:48.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.332+0000 7fa9495bf700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa944126390 con 0x7fa944112790 2019-12-05T05:03:48.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.332+0000 7fa9495bf700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa944112dc0 con 0x7fa944126940 2019-12-05T05:03:48.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.332+0000 7fa9495bf700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa944113060 con 0x7fa9440aa690 2019-12-05T05:03:48.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.332+0000 7fa942ffd700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fa944112790 0x7fa944142750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:53790/0 (socket says 172.21.15.42:53790) 2019-12-05T05:03:48.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.332+0000 7fa942ffd700 1 -- 172.21.15.42:0/1811637580 learned_addr learned my addr 172.21.15.42:0/1811637580 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:03:48.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.332+0000 7fa943fff700 1 --2- 172.21.15.42:0/1811637580 >> v2:172.21.15.40:3300/0 conn(0x7fa944126940 0x7fa944126d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:48.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.332+0000 7fa9437fe700 1 --2- 172.21.15.42:0/1811637580 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa9440aa690 0x7fa944143cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:48.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.332+0000 7fa93bfff700 1 -- 172.21.15.42:0/1811637580 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1596326644 0 0) 0x7fa944126390 con 0x7fa944112790 2019-12-05T05:03:48.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.332+0000 7fa93bfff700 1 -- 172.21.15.42:0/1811637580 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9280030b0 con 0x7fa944112790 2019-12-05T05:03:48.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.332+0000 7fa943fff700 1 -- 172.21.15.42:0/1811637580 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa9440aa690 msgr2=0x7fa944143cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:48.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.332+0000 7fa943fff700 1 --2- 172.21.15.42:0/1811637580 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa9440aa690 0x7fa944143cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:48.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.332+0000 7fa943fff700 1 -- 172.21.15.42:0/1811637580 >> v1:172.21.15.42:6789/0 conn(0x7fa944112790 legacy=0x7fa944142750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:48.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.332+0000 7fa943fff700 1 -- 172.21.15.42:0/1811637580 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa94412e7f0 con 0x7fa944126940 2019-12-05T05:03:48.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.332+0000 7fa9437fe700 1 --2- 172.21.15.42:0/1811637580 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa9440aa690 0x7fa944143cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:03:48.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.332+0000 7fa943fff700 1 --2- 172.21.15.42:0/1811637580 >> v2:172.21.15.40:3300/0 conn(0x7fa944126940 0x7fa944126d70 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fa944112f80 tx=0x7fa93400f450).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:48.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.332+0000 7fa943fff700 1 --2- 172.21.15.42:0/1811637580 >> v2:172.21.15.40:3300/0 conn(0x7fa944126940 0x7fa944126d70 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fa944112f80 tx=0x7fa93400f450).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:48.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.332+0000 7fa943fff700 1 --2- 172.21.15.42:0/1811637580 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa944126940 0x7fa944126d70 secure :-1 s=READY pgs=143 cs=0 l=1 rx=0x7fa944112f80 tx=0x7fa93400f450).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:48.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.332+0000 7fa943fff700 1 --2- 172.21.15.42:0/1811637580 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa944126940 0x7fa944126d70 secure :-1 s=THROTTLE_DONE pgs=143 cs=0 l=1 rx=0x7fa944112f80 tx=0x7fa93400f450).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:48.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.332+0000 7fa93bfff700 1 -- 172.21.15.42:0/1811637580 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa934007160 con 0x7fa944126940 2019-12-05T05:03:48.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.336+0000 7fa943fff700 1 --2- 172.21.15.42:0/1811637580 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa944126940 0x7fa944126d70 secure :-1 s=THROTTLE_DONE pgs=143 cs=0 l=1 rx=0x7fa944112f80 tx=0x7fa93400f450).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:48.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.336+0000 7fa943fff700 1 --2- 172.21.15.42:0/1811637580 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa944126940 0x7fa944126d70 secure :-1 s=THROTTLE_DONE pgs=143 cs=0 l=1 rx=0x7fa944112f80 tx=0x7fa93400f450).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:48.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.336+0000 7fa9495bf700 1 -- 172.21.15.42:0/1811637580 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa94412d7e0 con 0x7fa944126940 2019-12-05T05:03:48.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.336+0000 7fa93bfff700 1 -- 172.21.15.42:0/1811637580 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa934008980 con 0x7fa944126940 2019-12-05T05:03:48.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.336+0000 7fa93bfff700 1 -- 172.21.15.42:0/1811637580 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa934007360 con 0x7fa944126940 2019-12-05T05:03:48.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.336+0000 7fa9495bf700 1 -- 172.21.15.42:0/1811637580 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa94412db90 con 0x7fa944126940 2019-12-05T05:03:48.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.336+0000 7fa943fff700 1 --2- 172.21.15.42:0/1811637580 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa944126940 0x7fa944126d70 secure :-1 s=THROTTLE_DONE pgs=143 cs=0 l=1 rx=0x7fa944112f80 tx=0x7fa93400f450).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:48.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.336+0000 7fa93bfff700 1 -- 172.21.15.42:0/1811637580 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fa934011890 con 0x7fa944126940 2019-12-05T05:03:48.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.336+0000 7fa943fff700 1 --2- 172.21.15.42:0/1811637580 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa944126940 0x7fa944126d70 secure :-1 s=THROTTLE_DONE pgs=143 cs=0 l=1 rx=0x7fa944112f80 tx=0x7fa93400f450).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:48.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.336+0000 7fa93bfff700 1 --2- 172.21.15.42:0/1811637580 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa92803b250 0x7fa92803d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:48.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.336+0000 7fa93bfff700 1 -- 172.21.15.42:0/1811637580 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6031+0+0 (secure 0 0 0) 0x7fa93404a760 con 0x7fa944126940 2019-12-05T05:03:48.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.336+0000 7fa9437fe700 1 --2- 172.21.15.42:0/1811637580 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa92803b250 0x7fa92803d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:48.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.336+0000 7fa9437fe700 1 --2- 172.21.15.42:0/1811637580 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa92803b250 0x7fa92803d760 crc :-1 s=READY pgs=94 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:48.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.340+0000 7fa9495bf700 1 -- 172.21.15.42:0/1811637580 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa944112bc0 con 0x7fa944126940 2019-12-05T05:03:48.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.352+0000 7fa943fff700 1 --2- 172.21.15.42:0/1811637580 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa944126940 0x7fa944126d70 secure :-1 s=THROTTLE_DONE pgs=143 cs=0 l=1 rx=0x7fa944112f80 tx=0x7fa93400f450).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:48.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.352+0000 7fa93bfff700 1 -- 172.21.15.42:0/1811637580 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fa9340317c0 con 0x7fa944126940 2019-12-05T05:03:48.522 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.516+0000 7fa9495bf700 1 -- 172.21.15.42:0/1811637580 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7fa94405d110 con 0x7fa944126940 2019-12-05T05:03:48.523 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.516+0000 7fa943fff700 1 --2- 172.21.15.42:0/1811637580 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa944126940 0x7fa944126d70 secure :-1 s=THROTTLE_DONE pgs=143 cs=0 l=1 rx=0x7fa944112f80 tx=0x7fa93400f450).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:48.523 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.516+0000 7fa93bfff700 1 -- 172.21.15.42:0/1811637580 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v24) v1 ==== 92+0+10 (secure 0 0 0) 0x7fa934031a20 con 0x7fa944126940 2019-12-05T05:03:48.525 INFO:teuthology.orchestra.run.smithi042.stdout:pg_num: 8 2019-12-05T05:03:48.527 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.520+0000 7fa9495bf700 1 -- 172.21.15.42:0/1811637580 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa92803b250 msgr2=0x7fa92803d760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:48.527 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.520+0000 7fa9495bf700 1 --2- 172.21.15.42:0/1811637580 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa92803b250 0x7fa92803d760 crc :-1 s=READY pgs=94 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:48.528 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.520+0000 7fa9495bf700 1 -- 172.21.15.42:0/1811637580 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa944126940 msgr2=0x7fa944126d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:48.528 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.520+0000 7fa9495bf700 1 --2- 172.21.15.42:0/1811637580 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa944126940 0x7fa944126d70 secure :-1 s=READY pgs=143 cs=0 l=1 rx=0x7fa944112f80 tx=0x7fa93400f450).stop 2019-12-05T05:03:48.528 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.520+0000 7fa9495bf700 1 -- 172.21.15.42:0/1811637580 shutdown_connections 2019-12-05T05:03:48.528 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.520+0000 7fa9495bf700 1 --2- 172.21.15.42:0/1811637580 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa92803b250 0x7fa92803d760 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:48.528 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.520+0000 7fa9495bf700 1 --2- 172.21.15.42:0/1811637580 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa944126940 0x7fa944126d70 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:48.528 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.520+0000 7fa9495bf700 1 --2- 172.21.15.42:0/1811637580 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa9440aa690 0x7fa944143cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:48.528 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.520+0000 7fa9495bf700 1 -- 172.21.15.42:0/1811637580 shutdown_connections 2019-12-05T05:03:48.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.524+0000 7fa9495bf700 1 -- 172.21.15.42:0/1811637580 wait complete. 2019-12-05T05:03:48.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.524+0000 7fa9495bf700 1 -- 172.21.15.42:0/1811637580 >> 172.21.15.42:0/1811637580 conn(0x7fa9440a8b30 msgr2=0x7fa9440a69b0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:48.537 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2019-12-05T05:03:48.641 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.632+0000 7f1bf4dc6700 1 Processor -- start 2019-12-05T05:03:48.641 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.632+0000 7f1bf4dc6700 1 -- start start 2019-12-05T05:03:48.641 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.632+0000 7f1bf4dc6700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f1bf00a9a00 0x7f1bf007e880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:48.641 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.632+0000 7f1bf4dc6700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f1bf007edd0 0x7f1bf014ef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:48.642 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.632+0000 7f1bf4dc6700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bf00a1000 con 0x7f1bf009f550 2019-12-05T05:03:48.642 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.632+0000 7f1bf4dc6700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1bf00a8f80 con 0x7f1bf007edd0 2019-12-05T05:03:48.642 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.632+0000 7f1bf4dc6700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1bf00a9220 con 0x7f1bf00a9a00 2019-12-05T05:03:48.642 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.632+0000 7f1bee59c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f1bf00a9a00 0x7f1bf007e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:48.642 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.632+0000 7f1bee59c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f1bf00a9a00 0x7f1bf007e880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:53758/0 (socket says 172.21.15.42:53758) 2019-12-05T05:03:48.642 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.632+0000 7f1bee59c700 1 -- 172.21.15.42:0/153573609 learned_addr learned my addr 172.21.15.42:0/153573609 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:03:48.643 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.632+0000 7f1bef59e700 1 --2- 172.21.15.42:0/153573609 >> v2:172.21.15.40:3300/0 conn(0x7f1bf007edd0 0x7f1bf014ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:48.643 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.632+0000 7f1bee59c700 1 -- 172.21.15.42:0/153573609 >> v2:172.21.15.40:3300/0 conn(0x7f1bf007edd0 msgr2=0x7f1bf014ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:03:48.643 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.632+0000 7f1bee59c700 1 --2- 172.21.15.42:0/153573609 >> v2:172.21.15.40:3300/0 conn(0x7f1bf007edd0 0x7f1bf014ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:03:48.644 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.632+0000 7f1bee59c700 1 -- 172.21.15.42:0/153573609 >> v1:172.21.15.40:6789/0 conn(0x7f1bf009f550 legacy=0x7f1bf0144630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:48.644 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.632+0000 7f1bee59c700 1 -- 172.21.15.42:0/153573609 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bf009ee30 con 0x7f1bf00a9a00 2019-12-05T05:03:48.644 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.632+0000 7f1bee59c700 1 --2- 172.21.15.42:0/153573609 >> v2:172.21.15.42:3300/0 conn(0x7f1bf00a9a00 0x7f1bf007e880 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f1be4002f30 tx=0x7f1be400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:48.644 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.632+0000 7f1bee59c700 1 --2- 172.21.15.42:0/153573609 >> v2:172.21.15.42:3300/0 conn(0x7f1bf00a9a00 0x7f1bf007e880 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f1be4002f30 tx=0x7f1be400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:48.645 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.632+0000 7f1bee59c700 1 --2- 172.21.15.42:0/153573609 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1bf00a9a00 0x7f1bf007e880 secure :-1 s=READY pgs=177 cs=0 l=1 rx=0x7f1be4002f30 tx=0x7f1be400fd30).ready entity=mon.1 client_cookie=324fd43264a9e501 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:48.645 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.632+0000 7f1bee59c700 1 --2- 172.21.15.42:0/153573609 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1bf00a9a00 0x7f1bf007e880 secure :-1 s=THROTTLE_DONE pgs=177 cs=0 l=1 rx=0x7f1be4002f30 tx=0x7f1be400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:48.645 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.632+0000 7f1bee59c700 1 --2- 172.21.15.42:0/153573609 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1bf00a9a00 0x7f1bf007e880 secure :-1 s=THROTTLE_DONE pgs=177 cs=0 l=1 rx=0x7f1be4002f30 tx=0x7f1be400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:48.645 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.632+0000 7f1bee59c700 1 --2- 172.21.15.42:0/153573609 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1bf00a9a00 0x7f1bf007e880 secure :-1 s=THROTTLE_DONE pgs=177 cs=0 l=1 rx=0x7f1be4002f30 tx=0x7f1be400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:48.645 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.632+0000 7f1bedd9b700 1 -- 172.21.15.42:0/153573609 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1be4008c90 con 0x7f1bf00a9a00 2019-12-05T05:03:48.645 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.632+0000 7f1bedd9b700 1 -- 172.21.15.42:0/153573609 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1be40088b0 con 0x7f1bf00a9a00 2019-12-05T05:03:48.645 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.632+0000 7f1bedd9b700 1 -- 172.21.15.42:0/153573609 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1be4011ad0 con 0x7f1bf00a9a00 2019-12-05T05:03:48.646 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.632+0000 7f1bf4dc6700 1 -- 172.21.15.42:0/153573609 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1bf00a9a00 msgr2=0x7f1bf007e880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:48.646 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.632+0000 7f1bf4dc6700 1 --2- 172.21.15.42:0/153573609 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1bf00a9a00 0x7f1bf007e880 secure :-1 s=READY pgs=177 cs=0 l=1 rx=0x7f1be4002f30 tx=0x7f1be400fd30).stop 2019-12-05T05:03:48.646 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.636+0000 7f1bf4dc6700 1 -- 172.21.15.42:0/153573609 shutdown_connections 2019-12-05T05:03:48.646 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.636+0000 7f1bf4dc6700 1 --2- 172.21.15.42:0/153573609 >> v2:172.21.15.40:3300/0 conn(0x7f1bf007edd0 0x7f1bf014ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:03:48.646 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.636+0000 7f1bf4dc6700 1 --2- 172.21.15.42:0/153573609 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1bf00a9a00 0x7f1bf007e880 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:48.646 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.636+0000 7f1bf4dc6700 1 -- 172.21.15.42:0/153573609 shutdown_connections 2019-12-05T05:03:48.646 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.636+0000 7f1bf4dc6700 1 -- 172.21.15.42:0/153573609 wait complete. 2019-12-05T05:03:48.647 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.636+0000 7f1bf4dc6700 1 -- 172.21.15.42:0/153573609 >> 172.21.15.42:0/153573609 conn(0x7f1bf00a9ef0 msgr2=0x7f1bf00aa320 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:48.650 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.636+0000 7f1bf4dc6700 1 Processor -- start 2019-12-05T05:03:48.650 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.636+0000 7f1bf4dc6700 1 -- start start 2019-12-05T05:03:48.650 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.636+0000 7f1bf4dc6700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f1bf01443f0 0x7f1bf008f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:48.650 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.636+0000 7f1bf4dc6700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1bf0119ad0 0x7f1bf008fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:48.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.636+0000 7f1bf4dc6700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bf0142600 con 0x7f1bf0143200 2019-12-05T05:03:48.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.636+0000 7f1bf4dc6700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1bf01448c0 con 0x7f1bf01443f0 2019-12-05T05:03:48.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.636+0000 7f1bf4dc6700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bf0144b60 con 0x7f1bf0119ad0 2019-12-05T05:03:48.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.636+0000 7f1beed9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f1bf01443f0 0x7f1bf008f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:48.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.636+0000 7f1beed9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f1bf01443f0 0x7f1bf008f730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:53762/0 (socket says 172.21.15.42:53762) 2019-12-05T05:03:48.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.636+0000 7f1beed9d700 1 -- 172.21.15.42:0/1140469390 learned_addr learned my addr 172.21.15.42:0/1140469390 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:03:48.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.636+0000 7f1bee59c700 1 --2- 172.21.15.42:0/1140469390 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1bf0119ad0 0x7f1bf008fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:48.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.640+0000 7f1beed9d700 1 -- 172.21.15.42:0/1140469390 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1bf0119ad0 msgr2=0x7f1bf008fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:48.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.640+0000 7f1beed9d700 1 --2- 172.21.15.42:0/1140469390 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1bf0119ad0 0x7f1bf008fc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:48.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.640+0000 7f1beed9d700 1 -- 172.21.15.42:0/1140469390 >> v1:172.21.15.40:6789/0 conn(0x7f1bf0143200 legacy=0x7f1bf008dd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:48.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.640+0000 7f1beed9d700 1 -- 172.21.15.42:0/1140469390 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bf008e520 con 0x7f1bf01443f0 2019-12-05T05:03:48.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.640+0000 7f1bee59c700 1 --2- 172.21.15.42:0/1140469390 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1bf0119ad0 0x7f1bf008fc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T05:03:48.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.640+0000 7f1beed9d700 1 --2- 172.21.15.42:0/1140469390 >> v2:172.21.15.42:3300/0 conn(0x7f1bf01443f0 0x7f1bf008f730 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f1bf0143f10 tx=0x7f1be0012bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:48.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.640+0000 7f1beed9d700 1 --2- 172.21.15.42:0/1140469390 >> v2:172.21.15.42:3300/0 conn(0x7f1bf01443f0 0x7f1bf008f730 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f1bf0143f10 tx=0x7f1be0012bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:48.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.640+0000 7f1beed9d700 1 --2- 172.21.15.42:0/1140469390 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1bf01443f0 0x7f1bf008f730 secure :-1 s=READY pgs=178 cs=0 l=1 rx=0x7f1bf0143f10 tx=0x7f1be0012bb0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:48.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.640+0000 7f1beed9d700 1 --2- 172.21.15.42:0/1140469390 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1bf01443f0 0x7f1bf008f730 secure :-1 s=THROTTLE_DONE pgs=178 cs=0 l=1 rx=0x7f1bf0143f10 tx=0x7f1be0012bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:48.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.640+0000 7f1bdf7fe700 1 -- 172.21.15.42:0/1140469390 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1be000caa0 con 0x7f1bf01443f0 2019-12-05T05:03:48.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.640+0000 7f1beed9d700 1 --2- 172.21.15.42:0/1140469390 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1bf01443f0 0x7f1bf008f730 secure :-1 s=THROTTLE_DONE pgs=178 cs=0 l=1 rx=0x7f1bf0143f10 tx=0x7f1be0012bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:48.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.640+0000 7f1bf4dc6700 1 -- 172.21.15.42:0/1140469390 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bf008e7f0 con 0x7f1bf01443f0 2019-12-05T05:03:48.654 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.640+0000 7f1bdf7fe700 1 -- 172.21.15.42:0/1140469390 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1be000c6c0 con 0x7f1bf01443f0 2019-12-05T05:03:48.654 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.640+0000 7f1beed9d700 1 --2- 172.21.15.42:0/1140469390 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1bf01443f0 0x7f1bf008f730 secure :-1 s=THROTTLE_DONE pgs=178 cs=0 l=1 rx=0x7f1bf0143f10 tx=0x7f1be0012bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:48.655 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.640+0000 7f1bdf7fe700 1 -- 172.21.15.42:0/1140469390 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1be00143e0 con 0x7f1bf01443f0 2019-12-05T05:03:48.655 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.640+0000 7f1bf4dc6700 1 -- 172.21.15.42:0/1140469390 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bf00a9a00 con 0x7f1bf01443f0 2019-12-05T05:03:48.655 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.640+0000 7f1beed9d700 1 --2- 172.21.15.42:0/1140469390 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1bf01443f0 0x7f1bf008f730 secure :-1 s=THROTTLE_DONE pgs=178 cs=0 l=1 rx=0x7f1bf0143f10 tx=0x7f1be0012bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:48.655 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.640+0000 7f1bdf7fe700 1 -- 172.21.15.42:0/1140469390 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f1be0014740 con 0x7f1bf01443f0 2019-12-05T05:03:48.655 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.640+0000 7f1beed9d700 1 --2- 172.21.15.42:0/1140469390 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1bf01443f0 0x7f1bf008f730 secure :-1 s=THROTTLE_DONE pgs=178 cs=0 l=1 rx=0x7f1bf0143f10 tx=0x7f1be0012bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:48.655 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.640+0000 7f1bdf7fe700 1 --2- 172.21.15.42:0/1140469390 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f1bd403ade0 0x7f1bd403d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:48.655 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.640+0000 7f1bdf7fe700 1 -- 172.21.15.42:0/1140469390 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f1be004d710 con 0x7f1bf01443f0 2019-12-05T05:03:48.656 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.640+0000 7f1bee59c700 1 --2- 172.21.15.42:0/1140469390 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f1bd403ade0 0x7f1bd403d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:48.656 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.640+0000 7f1bee59c700 1 --2- 172.21.15.42:0/1140469390 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f1bd403ade0 0x7f1bd403d2f0 crc :-1 s=READY pgs=95 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:48.656 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.648+0000 7f1bf4dc6700 1 -- 172.21.15.42:0/1140469390 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1bf005d110 con 0x7f1bf01443f0 2019-12-05T05:03:48.660 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.656+0000 7f1beed9d700 1 --2- 172.21.15.42:0/1140469390 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1bf01443f0 0x7f1bf008f730 secure :-1 s=THROTTLE_DONE pgs=178 cs=0 l=1 rx=0x7f1bf0143f10 tx=0x7f1be0012bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:48.660 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.656+0000 7f1bdf7fe700 1 -- 172.21.15.42:0/1140469390 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f1be0015060 con 0x7f1bf01443f0 2019-12-05T05:03:48.823 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.816+0000 7f1bf4dc6700 1 -- 172.21.15.42:0/1140469390 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7f1bf005b8c0 con 0x7f1bf01443f0 2019-12-05T05:03:48.824 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.820+0000 7f1beed9d700 1 --2- 172.21.15.42:0/1140469390 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1bf01443f0 0x7f1bf008f730 secure :-1 s=THROTTLE_DONE pgs=178 cs=0 l=1 rx=0x7f1bf0143f10 tx=0x7f1be0012bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:48.824 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.820+0000 7f1bdf7fe700 1 -- 172.21.15.42:0/1140469390 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v24) v1 ==== 104+0+10 (secure 0 0 0) 0x7f1be0034770 con 0x7f1bf01443f0 2019-12-05T05:03:48.824 INFO:teuthology.orchestra.run.smithi042.stdout:pg_num: 8 2019-12-05T05:03:48.827 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.820+0000 7f1bf4dc6700 1 -- 172.21.15.42:0/1140469390 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f1bd403ade0 msgr2=0x7f1bd403d2f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:48.827 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.820+0000 7f1bf4dc6700 1 --2- 172.21.15.42:0/1140469390 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f1bd403ade0 0x7f1bd403d2f0 crc :-1 s=READY pgs=95 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:48.827 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.820+0000 7f1bf4dc6700 1 -- 172.21.15.42:0/1140469390 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1bf01443f0 msgr2=0x7f1bf008f730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:48.827 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.820+0000 7f1bf4dc6700 1 --2- 172.21.15.42:0/1140469390 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1bf01443f0 0x7f1bf008f730 secure :-1 s=READY pgs=178 cs=0 l=1 rx=0x7f1bf0143f10 tx=0x7f1be0012bb0).stop 2019-12-05T05:03:48.827 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.820+0000 7f1bf4dc6700 1 -- 172.21.15.42:0/1140469390 shutdown_connections 2019-12-05T05:03:48.827 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.820+0000 7f1bf4dc6700 1 --2- 172.21.15.42:0/1140469390 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f1bd403ade0 0x7f1bd403d2f0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:48.828 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.820+0000 7f1bf4dc6700 1 --2- 172.21.15.42:0/1140469390 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1bf01443f0 0x7f1bf008f730 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:48.828 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.820+0000 7f1bf4dc6700 1 --2- 172.21.15.42:0/1140469390 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1bf0119ad0 0x7f1bf008fc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:48.828 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.820+0000 7f1bf4dc6700 1 -- 172.21.15.42:0/1140469390 shutdown_connections 2019-12-05T05:03:48.828 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.820+0000 7f1bf4dc6700 1 -- 172.21.15.42:0/1140469390 wait complete. 2019-12-05T05:03:48.828 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.820+0000 7f1bf4dc6700 1 -- 172.21.15.42:0/1140469390 >> 172.21.15.42:0/1140469390 conn(0x7f1bf00a5370 msgr2=0x7f1bf00a57a0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:48.836 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2019-12-05T05:03:48.939 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.928+0000 7f152fc97700 1 Processor -- start 2019-12-05T05:03:48.939 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.928+0000 7f152fc97700 1 -- start start 2019-12-05T05:03:48.940 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.928+0000 7f152fc97700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f152809f550 0x7f1528144630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:48.940 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.928+0000 7f152fc97700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f152814cf50 0x7f152814f390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:48.940 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.928+0000 7f152fc97700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f152807f080 con 0x7f15280a9a00 2019-12-05T05:03:48.940 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.928+0000 7f152fc97700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15280a8f80 con 0x7f152809f550 2019-12-05T05:03:48.940 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.928+0000 7f152fc97700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15280a9220 con 0x7f152814cf50 2019-12-05T05:03:48.940 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.932+0000 7f152e234700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f152809f550 0x7f1528144630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:48.941 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.932+0000 7f152ea35700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f152814cf50 0x7f152814f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:48.941 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.932+0000 7f152e234700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f152809f550 0x7f1528144630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.42:41586/0 (socket says 172.21.15.42:41586) 2019-12-05T05:03:48.941 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.932+0000 7f152e234700 1 -- 172.21.15.42:0/1125440473 learned_addr learned my addr 172.21.15.42:0/1125440473 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:03:48.941 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.932+0000 7f152e234700 1 -- 172.21.15.42:0/1125440473 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f152814cf50 msgr2=0x7f152814f390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:03:48.941 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.932+0000 7f152e234700 1 --2- 172.21.15.42:0/1125440473 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f152814cf50 0x7f152814f390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:03:48.941 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.932+0000 7f152e234700 1 -- 172.21.15.42:0/1125440473 >> v1:172.21.15.40:6789/0 conn(0x7f15280a9a00 legacy=0x7f152807e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:48.942 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.932+0000 7f152e234700 1 -- 172.21.15.42:0/1125440473 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15280a1000 con 0x7f152809f550 2019-12-05T05:03:48.942 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.932+0000 7f152e234700 1 --2- 172.21.15.42:0/1125440473 >> v2:172.21.15.40:3300/0 conn(0x7f152809f550 0x7f1528144630 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f15281421e0 tx=0x7f15180100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:48.942 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.932+0000 7f152e234700 1 --2- 172.21.15.42:0/1125440473 >> v2:172.21.15.40:3300/0 conn(0x7f152809f550 0x7f1528144630 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f15281421e0 tx=0x7f15180100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:48.942 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.932+0000 7f152e234700 1 --2- 172.21.15.42:0/1125440473 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f152809f550 0x7f1528144630 secure :-1 s=READY pgs=147 cs=0 l=1 rx=0x7f15281421e0 tx=0x7f15180100b0).ready entity=mon.0 client_cookie=81cd8ecbb57fba03 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:48.942 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.932+0000 7f152e234700 1 --2- 172.21.15.42:0/1125440473 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f152809f550 0x7f1528144630 secure :-1 s=THROTTLE_DONE pgs=147 cs=0 l=1 rx=0x7f15281421e0 tx=0x7f15180100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:48.942 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.932+0000 7f152e234700 1 --2- 172.21.15.42:0/1125440473 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f152809f550 0x7f1528144630 secure :-1 s=THROTTLE_DONE pgs=147 cs=0 l=1 rx=0x7f15281421e0 tx=0x7f15180100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:48.943 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.932+0000 7f152d232700 1 -- 172.21.15.42:0/1125440473 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1518011dd0 con 0x7f152809f550 2019-12-05T05:03:48.944 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.932+0000 7f152e234700 1 --2- 172.21.15.42:0/1125440473 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f152809f550 0x7f1528144630 secure :-1 s=THROTTLE_DONE pgs=147 cs=0 l=1 rx=0x7f15281421e0 tx=0x7f15180100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:48.944 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.932+0000 7f152d232700 1 -- 172.21.15.42:0/1125440473 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15180119f0 con 0x7f152809f550 2019-12-05T05:03:48.944 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.932+0000 7f152d232700 1 -- 172.21.15.42:0/1125440473 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f151800a540 con 0x7f152809f550 2019-12-05T05:03:48.944 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.932+0000 7f152fc97700 1 -- 172.21.15.42:0/1125440473 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f152809f550 msgr2=0x7f1528144630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:48.944 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.932+0000 7f152fc97700 1 --2- 172.21.15.42:0/1125440473 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f152809f550 0x7f1528144630 secure :-1 s=READY pgs=147 cs=0 l=1 rx=0x7f15281421e0 tx=0x7f15180100b0).stop 2019-12-05T05:03:48.945 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.932+0000 7f152fc97700 1 -- 172.21.15.42:0/1125440473 shutdown_connections 2019-12-05T05:03:48.945 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.932+0000 7f152fc97700 1 --2- 172.21.15.42:0/1125440473 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f152814cf50 0x7f152814f390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:03:48.945 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.932+0000 7f152fc97700 1 --2- 172.21.15.42:0/1125440473 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f152809f550 0x7f1528144630 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:48.945 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.932+0000 7f152fc97700 1 -- 172.21.15.42:0/1125440473 shutdown_connections 2019-12-05T05:03:48.945 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.932+0000 7f152fc97700 1 -- 172.21.15.42:0/1125440473 wait complete. 2019-12-05T05:03:48.945 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.932+0000 7f152fc97700 1 -- 172.21.15.42:0/1125440473 >> 172.21.15.42:0/1125440473 conn(0x7f15280a9ef0 msgr2=0x7f15280aa320 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:48.949 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.936+0000 7f152fc97700 1 Processor -- start 2019-12-05T05:03:48.949 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.936+0000 7f152fc97700 1 -- start start 2019-12-05T05:03:48.949 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.936+0000 7f152fc97700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f1528143f50 0x7f1528064010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:48.949 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.936+0000 7f152fc97700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f152811ae30 0x7f15281214f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:48.951 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.936+0000 7f152fc97700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1528121a40 con 0x7f15280a9d40 2019-12-05T05:03:48.951 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.936+0000 7f152fc97700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1528144380 con 0x7f1528143f50 2019-12-05T05:03:48.952 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.936+0000 7f152fc97700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1528064560 con 0x7f152811ae30 2019-12-05T05:03:48.952 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.936+0000 7f152e234700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f15280a9d40 0x7f1528120cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:53816/0 (socket says 172.21.15.42:53816) 2019-12-05T05:03:48.952 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.936+0000 7f152ea35700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f152811ae30 0x7f15281214f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:48.952 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.936+0000 7f152e234700 1 -- 172.21.15.42:0/4033867132 learned_addr learned my addr 172.21.15.42:0/4033867132 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:03:48.952 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.936+0000 7f152da33700 1 --2- 172.21.15.42:0/4033867132 >> v2:172.21.15.40:3300/0 conn(0x7f1528143f50 0x7f1528064010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:48.952 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.936+0000 7f1526ffd700 1 -- 172.21.15.42:0/4033867132 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 830930716 0 0) 0x7f1528121a40 con 0x7f15280a9d40 2019-12-05T05:03:48.953 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.936+0000 7f1526ffd700 1 -- 172.21.15.42:0/4033867132 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15080030b0 con 0x7f15280a9d40 2019-12-05T05:03:48.953 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.936+0000 7f152ea35700 1 -- 172.21.15.42:0/4033867132 >> v2:172.21.15.40:3300/0 conn(0x7f1528143f50 msgr2=0x7f1528064010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:48.953 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.936+0000 7f152ea35700 1 --2- 172.21.15.42:0/4033867132 >> v2:172.21.15.40:3300/0 conn(0x7f1528143f50 0x7f1528064010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:48.953 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.936+0000 7f152ea35700 1 -- 172.21.15.42:0/4033867132 >> v1:172.21.15.42:6789/0 conn(0x7f15280a9d40 legacy=0x7f1528120cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:48.954 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.936+0000 7f152ea35700 1 -- 172.21.15.42:0/4033867132 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f152811b260 con 0x7f152811ae30 2019-12-05T05:03:48.954 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.936+0000 7f152ea35700 1 --2- 172.21.15.42:0/4033867132 >> v2:172.21.15.42:3300/0 conn(0x7f152811ae30 0x7f15281214f0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f1520002f30 tx=0x7f1520010990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:48.954 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.936+0000 7f152ea35700 1 --2- 172.21.15.42:0/4033867132 >> v2:172.21.15.42:3300/0 conn(0x7f152811ae30 0x7f15281214f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f1520002f30 tx=0x7f1520010990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:48.954 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.936+0000 7f152ea35700 1 --2- 172.21.15.42:0/4033867132 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f152811ae30 0x7f15281214f0 secure :-1 s=READY pgs=180 cs=0 l=1 rx=0x7f1520002f30 tx=0x7f1520010990).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:48.955 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.936+0000 7f152ea35700 1 --2- 172.21.15.42:0/4033867132 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f152811ae30 0x7f15281214f0 secure :-1 s=THROTTLE_DONE pgs=180 cs=0 l=1 rx=0x7f1520002f30 tx=0x7f1520010990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:48.955 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.936+0000 7f1526ffd700 1 -- 172.21.15.42:0/4033867132 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1520005930 con 0x7f152811ae30 2019-12-05T05:03:48.955 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.936+0000 7f152ea35700 1 --2- 172.21.15.42:0/4033867132 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f152811ae30 0x7f15281214f0 secure :-1 s=THROTTLE_DONE pgs=180 cs=0 l=1 rx=0x7f1520002f30 tx=0x7f1520010990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:48.955 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.936+0000 7f1526ffd700 1 -- 172.21.15.42:0/4033867132 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1520007160 con 0x7f152811ae30 2019-12-05T05:03:48.955 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.936+0000 7f152ea35700 1 --2- 172.21.15.42:0/4033867132 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f152811ae30 0x7f15281214f0 secure :-1 s=THROTTLE_DONE pgs=180 cs=0 l=1 rx=0x7f1520002f30 tx=0x7f1520010990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:48.955 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.936+0000 7f1526ffd700 1 -- 172.21.15.42:0/4033867132 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f15200060c0 con 0x7f152811ae30 2019-12-05T05:03:48.956 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.936+0000 7f152fc97700 1 -- 172.21.15.42:0/4033867132 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15281424b0 con 0x7f152811ae30 2019-12-05T05:03:48.956 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.936+0000 7f152fc97700 1 -- 172.21.15.42:0/4033867132 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1528077530 con 0x7f152811ae30 2019-12-05T05:03:48.956 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.940+0000 7f152ea35700 1 --2- 172.21.15.42:0/4033867132 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f152811ae30 0x7f15281214f0 secure :-1 s=THROTTLE_DONE pgs=180 cs=0 l=1 rx=0x7f1520002f30 tx=0x7f1520010990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:48.956 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.940+0000 7f1526ffd700 1 -- 172.21.15.42:0/4033867132 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f15200063e0 con 0x7f152811ae30 2019-12-05T05:03:48.956 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.940+0000 7f152ea35700 1 --2- 172.21.15.42:0/4033867132 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f152811ae30 0x7f15281214f0 secure :-1 s=THROTTLE_DONE pgs=180 cs=0 l=1 rx=0x7f1520002f30 tx=0x7f1520010990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:48.956 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.940+0000 7f1526ffd700 1 --2- 172.21.15.42:0/4033867132 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f150803b200 0x7f150803d710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:48.957 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.940+0000 7f1526ffd700 1 -- 172.21.15.42:0/4033867132 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f152004ab00 con 0x7f152811ae30 2019-12-05T05:03:48.957 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.940+0000 7f152e234700 1 --2- 172.21.15.42:0/4033867132 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f150803b200 0x7f150803d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:48.957 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.940+0000 7f152e234700 1 --2- 172.21.15.42:0/4033867132 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f150803b200 0x7f150803d710 crc :-1 s=READY pgs=96 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:48.958 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.944+0000 7f152fc97700 1 -- 172.21.15.42:0/4033867132 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1528142b50 con 0x7f152811ae30 2019-12-05T05:03:48.958 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.952+0000 7f152ea35700 1 --2- 172.21.15.42:0/4033867132 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f152811ae30 0x7f15281214f0 secure :-1 s=THROTTLE_DONE pgs=180 cs=0 l=1 rx=0x7f1520002f30 tx=0x7f1520010990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:48.958 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:48.952+0000 7f1526ffd700 1 -- 172.21.15.42:0/4033867132 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f1520031b60 con 0x7f152811ae30 2019-12-05T05:03:49.115 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.108+0000 7f152fc97700 1 -- 172.21.15.42:0/4033867132 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7f152805d110 con 0x7f152811ae30 2019-12-05T05:03:49.115 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.112+0000 7f152ea35700 1 --2- 172.21.15.42:0/4033867132 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f152811ae30 0x7f15281214f0 secure :-1 s=THROTTLE_DONE pgs=180 cs=0 l=1 rx=0x7f1520002f30 tx=0x7f1520010990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:49.116 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.112+0000 7f1526ffd700 1 -- 172.21.15.42:0/4033867132 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v24) v1 ==== 100+0+10 (secure 0 0 0) 0x7f1520031dc0 con 0x7f152811ae30 2019-12-05T05:03:49.116 INFO:teuthology.orchestra.run.smithi042.stdout:pg_num: 8 2019-12-05T05:03:49.119 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.112+0000 7f152fc97700 1 -- 172.21.15.42:0/4033867132 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f150803b200 msgr2=0x7f150803d710 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:49.119 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.112+0000 7f152fc97700 1 --2- 172.21.15.42:0/4033867132 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f150803b200 0x7f150803d710 crc :-1 s=READY pgs=96 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:49.119 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.112+0000 7f152fc97700 1 -- 172.21.15.42:0/4033867132 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f152811ae30 msgr2=0x7f15281214f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:49.120 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.112+0000 7f152fc97700 1 --2- 172.21.15.42:0/4033867132 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f152811ae30 0x7f15281214f0 secure :-1 s=READY pgs=180 cs=0 l=1 rx=0x7f1520002f30 tx=0x7f1520010990).stop 2019-12-05T05:03:49.120 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.112+0000 7f152fc97700 1 -- 172.21.15.42:0/4033867132 shutdown_connections 2019-12-05T05:03:49.120 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.112+0000 7f152fc97700 1 --2- 172.21.15.42:0/4033867132 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f150803b200 0x7f150803d710 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:49.120 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.112+0000 7f152fc97700 1 --2- 172.21.15.42:0/4033867132 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f152811ae30 0x7f15281214f0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:49.120 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.112+0000 7f152fc97700 1 --2- 172.21.15.42:0/4033867132 >> v2:172.21.15.40:3300/0 conn(0x7f1528143f50 0x7f1528064010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:49.120 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.112+0000 7f152fc97700 1 -- 172.21.15.42:0/4033867132 shutdown_connections 2019-12-05T05:03:49.120 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.112+0000 7f152fc97700 1 -- 172.21.15.42:0/4033867132 wait complete. 2019-12-05T05:03:49.121 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.112+0000 7f152fc97700 1 -- 172.21.15.42:0/4033867132 >> 172.21.15.42:0/4033867132 conn(0x7f15280a8b30 msgr2=0x7f15280a5970 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:49.128 INFO:tasks.mds_thrash:Wait for all MDSs to reach steady state... 2019-12-05T05:03:49.129 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:03:49.233 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.220+0000 7fab5897b700 1 Processor -- start 2019-12-05T05:03:49.234 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.220+0000 7fab5897b700 1 -- start start 2019-12-05T05:03:49.234 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.224+0000 7fab5897b700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fab5408df80 0x7fab5409f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:49.234 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.224+0000 7fab5897b700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab5409fc60 con 0x7fab5409f830 2019-12-05T05:03:49.234 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.224+0000 7fab5897b700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab540a0710 con 0x7fab540a7740 2019-12-05T05:03:49.234 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.224+0000 7fab5897b700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab540a0b00 con 0x7fab5408df80 2019-12-05T05:03:49.234 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.224+0000 7fab52d9d700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fab540a7740 0x7fab5407e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:53822/0 (socket says 172.21.15.42:53822) 2019-12-05T05:03:49.235 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.224+0000 7fab52d9d700 1 -- 172.21.15.42:0/2520487840 learned_addr learned my addr 172.21.15.42:0/2520487840 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:03:49.235 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.224+0000 7fab5259c700 1 --2- 172.21.15.42:0/2520487840 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fab5408df80 0x7fab5409f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:49.235 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.224+0000 7fab51d9b700 1 -- 172.21.15.42:0/2520487840 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1421449971 0 0) 0x7fab540a0710 con 0x7fab540a7740 2019-12-05T05:03:49.235 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.224+0000 7fab51d9b700 1 -- 172.21.15.42:0/2520487840 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab3c0030b0 con 0x7fab540a7740 2019-12-05T05:03:49.235 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.224+0000 7fab51d9b700 1 -- 172.21.15.42:0/2520487840 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3288295466 0 0) 0x7fab5409fc60 con 0x7fab5409f830 2019-12-05T05:03:49.235 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.224+0000 7fab51d9b700 1 -- 172.21.15.42:0/2520487840 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab540a0710 con 0x7fab5409f830 2019-12-05T05:03:49.235 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.224+0000 7fab5259c700 1 -- 172.21.15.42:0/2520487840 >> v1:172.21.15.42:6789/0 conn(0x7fab540a7740 legacy=0x7fab5407e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:49.235 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.224+0000 7fab5259c700 1 -- 172.21.15.42:0/2520487840 >> v1:172.21.15.40:6789/0 conn(0x7fab5409f830 legacy=0x7fab5414efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:49.236 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.224+0000 7fab5259c700 1 -- 172.21.15.42:0/2520487840 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab5407ef20 con 0x7fab5408df80 2019-12-05T05:03:49.236 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.224+0000 7fab5259c700 1 --2- 172.21.15.42:0/2520487840 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fab5408df80 0x7fab5409f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fab48002f30 tx=0x7fab4800fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:49.236 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.224+0000 7fab5259c700 1 --2- 172.21.15.42:0/2520487840 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fab5408df80 0x7fab5409f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fab48002f30 tx=0x7fab4800fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:49.236 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.224+0000 7fab5259c700 1 --2- 172.21.15.42:0/2520487840 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fab5408df80 0x7fab5409f2e0 secure :-1 s=READY pgs=43 cs=0 l=1 rx=0x7fab48002f30 tx=0x7fab4800fbf0).ready entity=mon.2 client_cookie=f82543d74c80b377 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:49.236 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.224+0000 7fab5259c700 1 --2- 172.21.15.42:0/2520487840 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fab5408df80 0x7fab5409f2e0 secure :-1 s=THROTTLE_DONE pgs=43 cs=0 l=1 rx=0x7fab48002f30 tx=0x7fab4800fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:49.236 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.224+0000 7fab51d9b700 1 -- 172.21.15.42:0/2520487840 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fab480112b0 con 0x7fab5408df80 2019-12-05T05:03:49.236 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.224+0000 7fab5259c700 1 --2- 172.21.15.42:0/2520487840 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fab5408df80 0x7fab5409f2e0 secure :-1 s=THROTTLE_DONE pgs=43 cs=0 l=1 rx=0x7fab48002f30 tx=0x7fab4800fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:49.237 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.224+0000 7fab51d9b700 1 -- 172.21.15.42:0/2520487840 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab48008bc0 con 0x7fab5408df80 2019-12-05T05:03:49.237 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.224+0000 7fab5259c700 1 --2- 172.21.15.42:0/2520487840 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fab5408df80 0x7fab5409f2e0 secure :-1 s=THROTTLE_DONE pgs=43 cs=0 l=1 rx=0x7fab48002f30 tx=0x7fab4800fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:49.237 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.224+0000 7fab51d9b700 1 -- 172.21.15.42:0/2520487840 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fab48011890 con 0x7fab5408df80 2019-12-05T05:03:49.237 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.224+0000 7fab5897b700 1 -- 172.21.15.42:0/2520487840 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fab5408df80 msgr2=0x7fab5409f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:49.237 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.224+0000 7fab5897b700 1 --2- 172.21.15.42:0/2520487840 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fab5408df80 0x7fab5409f2e0 secure :-1 s=READY pgs=43 cs=0 l=1 rx=0x7fab48002f30 tx=0x7fab4800fbf0).stop 2019-12-05T05:03:49.237 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.228+0000 7fab5897b700 1 -- 172.21.15.42:0/2520487840 shutdown_connections 2019-12-05T05:03:49.237 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.228+0000 7fab5897b700 1 --2- 172.21.15.42:0/2520487840 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fab5408df80 0x7fab5409f2e0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:49.238 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.228+0000 7fab5897b700 1 -- 172.21.15.42:0/2520487840 shutdown_connections 2019-12-05T05:03:49.238 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.228+0000 7fab5897b700 1 -- 172.21.15.42:0/2520487840 wait complete. 2019-12-05T05:03:49.238 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.228+0000 7fab5897b700 1 -- 172.21.15.42:0/2520487840 >> 172.21.15.42:0/2520487840 conn(0x7fab540aabe0 msgr2=0x7fab540a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:49.241 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.228+0000 7fab5897b700 1 Processor -- start 2019-12-05T05:03:49.241 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.228+0000 7fab5897b700 1 -- start start 2019-12-05T05:03:49.241 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.232+0000 7fab5897b700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fab54143810 0x7fab54143c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:49.241 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.232+0000 7fab5897b700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab54145280 con 0x7fab5407b090 2019-12-05T05:03:49.242 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.232+0000 7fab5897b700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab5412fba0 con 0x7fab54122160 2019-12-05T05:03:49.242 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.232+0000 7fab5897b700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab541443c0 con 0x7fab54143810 2019-12-05T05:03:49.242 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.232+0000 7fab5359e700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fab54122160 0x7fab54122590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:53832/0 (socket says 172.21.15.42:53832) 2019-12-05T05:03:49.242 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.232+0000 7fab5359e700 1 -- 172.21.15.42:0/782971455 learned_addr learned my addr 172.21.15.42:0/782971455 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:03:49.242 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.232+0000 7fab5259c700 1 --2- 172.21.15.42:0/782971455 >> v2:172.21.15.40:3300/0 conn(0x7fab54143810 0x7fab54143c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:49.242 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.232+0000 7fab437fe700 1 -- 172.21.15.42:0/782971455 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 258107364 0 0) 0x7fab5412fba0 con 0x7fab54122160 2019-12-05T05:03:49.242 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.232+0000 7fab437fe700 1 -- 172.21.15.42:0/782971455 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab380030b0 con 0x7fab54122160 2019-12-05T05:03:49.242 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.232+0000 7fab437fe700 1 -- 172.21.15.42:0/782971455 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 717550055 0 0) 0x7fab54145280 con 0x7fab5407b090 2019-12-05T05:03:49.243 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.232+0000 7fab5259c700 1 -- 172.21.15.42:0/782971455 >> v1:172.21.15.42:6789/0 conn(0x7fab54122160 legacy=0x7fab54122590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:49.243 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.232+0000 7fab5259c700 1 -- 172.21.15.42:0/782971455 >> v1:172.21.15.40:6789/0 conn(0x7fab5407b090 legacy=0x7fab540a30b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:49.244 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.232+0000 7fab5259c700 1 -- 172.21.15.42:0/782971455 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab54130de0 con 0x7fab54143810 2019-12-05T05:03:49.244 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.232+0000 7fab5259c700 1 --2- 172.21.15.42:0/782971455 >> v2:172.21.15.40:3300/0 conn(0x7fab54143810 0x7fab54143c40 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fab48002f30 tx=0x7fab48008680).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:49.244 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.232+0000 7fab5259c700 1 --2- 172.21.15.42:0/782971455 >> v2:172.21.15.40:3300/0 conn(0x7fab54143810 0x7fab54143c40 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fab48002f30 tx=0x7fab48008680).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:49.244 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.232+0000 7fab5259c700 1 --2- 172.21.15.42:0/782971455 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab54143810 0x7fab54143c40 secure :-1 s=READY pgs=150 cs=0 l=1 rx=0x7fab48002f30 tx=0x7fab48008680).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:49.244 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.232+0000 7fab5259c700 1 --2- 172.21.15.42:0/782971455 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab54143810 0x7fab54143c40 secure :-1 s=THROTTLE_DONE pgs=150 cs=0 l=1 rx=0x7fab48002f30 tx=0x7fab48008680).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:49.244 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.232+0000 7fab437fe700 1 -- 172.21.15.42:0/782971455 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fab48011d60 con 0x7fab54143810 2019-12-05T05:03:49.245 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.232+0000 7fab5259c700 1 --2- 172.21.15.42:0/782971455 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab54143810 0x7fab54143c40 secure :-1 s=THROTTLE_DONE pgs=150 cs=0 l=1 rx=0x7fab48002f30 tx=0x7fab48008680).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:49.245 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.232+0000 7fab5259c700 1 --2- 172.21.15.42:0/782971455 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab54143810 0x7fab54143c40 secure :-1 s=THROTTLE_DONE pgs=150 cs=0 l=1 rx=0x7fab48002f30 tx=0x7fab48008680).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:49.245 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.232+0000 7fab437fe700 1 -- 172.21.15.42:0/782971455 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab480114b0 con 0x7fab54143810 2019-12-05T05:03:49.245 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.232+0000 7fab5897b700 1 -- 172.21.15.42:0/782971455 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab5412eb90 con 0x7fab54143810 2019-12-05T05:03:49.245 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.232+0000 7fab437fe700 1 -- 172.21.15.42:0/782971455 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fab48016570 con 0x7fab54143810 2019-12-05T05:03:49.245 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.232+0000 7fab5897b700 1 -- 172.21.15.42:0/782971455 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab5412f140 con 0x7fab54143810 2019-12-05T05:03:49.245 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.232+0000 7fab5259c700 1 --2- 172.21.15.42:0/782971455 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab54143810 0x7fab54143c40 secure :-1 s=THROTTLE_DONE pgs=150 cs=0 l=1 rx=0x7fab48002f30 tx=0x7fab48008680).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:49.246 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.232+0000 7fab437fe700 1 -- 172.21.15.42:0/782971455 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fab48007170 con 0x7fab54143810 2019-12-05T05:03:49.246 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.232+0000 7fab5259c700 1 --2- 172.21.15.42:0/782971455 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab54143810 0x7fab54143c40 secure :-1 s=THROTTLE_DONE pgs=150 cs=0 l=1 rx=0x7fab48002f30 tx=0x7fab48008680).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:49.246 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.236+0000 7fab437fe700 1 --2- 172.21.15.42:0/782971455 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fab3803b130 0x7fab3803d640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:49.246 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.236+0000 7fab437fe700 1 -- 172.21.15.42:0/782971455 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6031+0+0 (secure 0 0 0) 0x7fab4804d4a0 con 0x7fab54143810 2019-12-05T05:03:49.246 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.236+0000 7fab52d9d700 1 --2- 172.21.15.42:0/782971455 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fab3803b130 0x7fab3803d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:49.246 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.236+0000 7fab52d9d700 1 --2- 172.21.15.42:0/782971455 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fab3803b130 0x7fab3803d640 crc :-1 s=READY pgs=97 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:49.247 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.240+0000 7fab5897b700 1 -- 172.21.15.42:0/782971455 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab5405d110 con 0x7fab54143810 2019-12-05T05:03:49.254 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.248+0000 7fab5259c700 1 --2- 172.21.15.42:0/782971455 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab54143810 0x7fab54143c40 secure :-1 s=THROTTLE_DONE pgs=150 cs=0 l=1 rx=0x7fab48002f30 tx=0x7fab48008680).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:49.254 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.248+0000 7fab437fe700 1 -- 172.21.15.42:0/782971455 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fab48034500 con 0x7fab54143810 2019-12-05T05:03:49.417 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.412+0000 7fab5897b700 1 -- 172.21.15.42:0/782971455 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fab5405b8c0 con 0x7fab54143810 2019-12-05T05:03:49.419 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.412+0000 7fab5259c700 1 --2- 172.21.15.42:0/782971455 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab54143810 0x7fab54143c40 secure :-1 s=THROTTLE_DONE pgs=150 cs=0 l=1 rx=0x7fab48002f30 tx=0x7fab48008680).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:49.419 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.412+0000 7fab437fe700 1 -- 172.21.15.42:0/782971455 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+2102 (secure 0 0 0) 0x7fab48034760 con 0x7fab54143810 2019-12-05T05:03:49.421 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:03:49.421 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4354,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/2019200199","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2019200199},{"type":"v1","addr":"172.21.15.42:6835","nonce":2019200199}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":4}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:03:43.669739+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4341},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4341":{"gid":4341,"name":"b","rank":0,"incarnation":5,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6833/3296198322","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3296198322},{"type":"v1","addr":"172.21.15.40:6833","nonce":3296198322}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:03:49.421 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 6 2019-12-05T05:03:49.423 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.416+0000 7fab5897b700 1 -- 172.21.15.42:0/782971455 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fab3803b130 msgr2=0x7fab3803d640 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:49.423 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.416+0000 7fab5897b700 1 --2- 172.21.15.42:0/782971455 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fab3803b130 0x7fab3803d640 crc :-1 s=READY pgs=97 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:49.423 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.416+0000 7fab5897b700 1 -- 172.21.15.42:0/782971455 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab54143810 msgr2=0x7fab54143c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:49.423 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.416+0000 7fab5897b700 1 --2- 172.21.15.42:0/782971455 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab54143810 0x7fab54143c40 secure :-1 s=READY pgs=150 cs=0 l=1 rx=0x7fab48002f30 tx=0x7fab48008680).stop 2019-12-05T05:03:49.424 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.416+0000 7fab5897b700 1 -- 172.21.15.42:0/782971455 shutdown_connections 2019-12-05T05:03:49.424 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.416+0000 7fab5897b700 1 --2- 172.21.15.42:0/782971455 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fab3803b130 0x7fab3803d640 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:49.424 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.416+0000 7fab5897b700 1 --2- 172.21.15.42:0/782971455 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab54143810 0x7fab54143c40 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:49.424 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.416+0000 7fab5897b700 1 -- 172.21.15.42:0/782971455 shutdown_connections 2019-12-05T05:03:49.424 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.416+0000 7fab5897b700 1 -- 172.21.15.42:0/782971455 wait complete. 2019-12-05T05:03:49.424 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.416+0000 7fab5897b700 1 -- 172.21.15.42:0/782971455 >> 172.21.15.42:0/782971455 conn(0x7fab540a96f0 msgr2=0x7fab5407e570 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:49.433 INFO:tasks.mds_thrash:Ready to start thrashing 2019-12-05T05:03:49.433 INFO:tasks.mds_thrash.ceph_manager:waiting for clean 2019-12-05T05:03:49.433 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2019-12-05T05:03:49.528 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.516+0000 7fce8c812700 1 Processor -- start 2019-12-05T05:03:49.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.516+0000 7fce8c812700 1 -- start start 2019-12-05T05:03:49.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.520+0000 7fce8c812700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fce880a7740 0x7fce8808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:49.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.520+0000 7fce8c812700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fce8807e720 0x7fce8807eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:49.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.520+0000 7fce8c812700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fce8809f2e0 0x7fce8809f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:49.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.520+0000 7fce8c812700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce880a0b00 con 0x7fce8809f2e0 2019-12-05T05:03:49.530 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.520+0000 7fce8c812700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce8808e5b0 con 0x7fce880a7740 2019-12-05T05:03:49.530 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.520+0000 7fce8c812700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce8808e850 con 0x7fce8807e720 2019-12-05T05:03:49.530 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.520+0000 7fce86d9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fce880a7740 0x7fce8808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:49.530 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.520+0000 7fce86d9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fce880a7740 0x7fce8808df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:53798/0 (socket says 172.21.15.42:53798) 2019-12-05T05:03:49.530 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.520+0000 7fce86d9d700 1 -- 172.21.15.42:0/3887984844 learned_addr learned my addr 172.21.15.42:0/3887984844 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:03:49.531 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.520+0000 7fce8759e700 1 --2- 172.21.15.42:0/3887984844 >> v2:172.21.15.40:3300/0 conn(0x7fce8809f2e0 0x7fce8809f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:49.531 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.520+0000 7fce8659c700 1 --2- 172.21.15.42:0/3887984844 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fce8807e720 0x7fce8807eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:49.531 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.520+0000 7fce86d9d700 1 -- 172.21.15.42:0/3887984844 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fce8807e720 msgr2=0x7fce8807eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:03:49.531 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.520+0000 7fce86d9d700 1 --2- 172.21.15.42:0/3887984844 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fce8807e720 0x7fce8807eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:03:49.531 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.520+0000 7fce86d9d700 1 -- 172.21.15.42:0/3887984844 >> v2:172.21.15.40:3300/0 conn(0x7fce8809f2e0 msgr2=0x7fce8809f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:03:49.531 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.520+0000 7fce86d9d700 1 --2- 172.21.15.42:0/3887984844 >> v2:172.21.15.40:3300/0 conn(0x7fce8809f2e0 0x7fce8809f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:03:49.531 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.520+0000 7fce86d9d700 1 -- 172.21.15.42:0/3887984844 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce8807f0a0 con 0x7fce880a7740 2019-12-05T05:03:49.531 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.520+0000 7fce8759e700 1 --2- 172.21.15.42:0/3887984844 >> v2:172.21.15.40:3300/0 conn(0x7fce8809f2e0 0x7fce8809f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:03:49.532 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.520+0000 7fce86d9d700 1 --2- 172.21.15.42:0/3887984844 >> v2:172.21.15.42:3300/0 conn(0x7fce880a7740 0x7fce8808df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fce8808e770 tx=0x7fce7800fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:49.532 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.520+0000 7fce86d9d700 1 --2- 172.21.15.42:0/3887984844 >> v2:172.21.15.42:3300/0 conn(0x7fce880a7740 0x7fce8808df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fce8808e770 tx=0x7fce7800fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:49.532 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.520+0000 7fce86d9d700 1 --2- 172.21.15.42:0/3887984844 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fce880a7740 0x7fce8808df80 secure :-1 s=READY pgs=183 cs=0 l=1 rx=0x7fce8808e770 tx=0x7fce7800fdb0).ready entity=mon.1 client_cookie=5610b8bf6f60c4bc server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:49.532 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.520+0000 7fce86d9d700 1 --2- 172.21.15.42:0/3887984844 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fce880a7740 0x7fce8808df80 secure :-1 s=THROTTLE_DONE pgs=183 cs=0 l=1 rx=0x7fce8808e770 tx=0x7fce7800fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:49.533 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.520+0000 7fce86d9d700 1 --2- 172.21.15.42:0/3887984844 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fce880a7740 0x7fce8808df80 secure :-1 s=THROTTLE_DONE pgs=183 cs=0 l=1 rx=0x7fce8808e770 tx=0x7fce7800fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:49.533 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.520+0000 7fce85d9b700 1 -- 172.21.15.42:0/3887984844 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fce78008c90 con 0x7fce880a7740 2019-12-05T05:03:49.533 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.520+0000 7fce86d9d700 1 --2- 172.21.15.42:0/3887984844 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fce880a7740 0x7fce8808df80 secure :-1 s=THROTTLE_DONE pgs=183 cs=0 l=1 rx=0x7fce8808e770 tx=0x7fce7800fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:49.533 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.520+0000 7fce85d9b700 1 -- 172.21.15.42:0/3887984844 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce780088b0 con 0x7fce880a7740 2019-12-05T05:03:49.534 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.520+0000 7fce85d9b700 1 -- 172.21.15.42:0/3887984844 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fce78011b50 con 0x7fce880a7740 2019-12-05T05:03:49.534 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.520+0000 7fce8c812700 1 -- 172.21.15.42:0/3887984844 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fce880a7740 msgr2=0x7fce8808df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:49.534 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.520+0000 7fce8c812700 1 --2- 172.21.15.42:0/3887984844 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fce880a7740 0x7fce8808df80 secure :-1 s=READY pgs=183 cs=0 l=1 rx=0x7fce8808e770 tx=0x7fce7800fdb0).stop 2019-12-05T05:03:49.534 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.524+0000 7fce8c812700 1 -- 172.21.15.42:0/3887984844 shutdown_connections 2019-12-05T05:03:49.534 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.524+0000 7fce8c812700 1 --2- 172.21.15.42:0/3887984844 >> v2:172.21.15.40:3300/0 conn(0x7fce8809f2e0 0x7fce8809f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:03:49.534 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.524+0000 7fce8c812700 1 --2- 172.21.15.42:0/3887984844 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fce880a7740 0x7fce8808df80 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:49.534 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.524+0000 7fce8c812700 1 --2- 172.21.15.42:0/3887984844 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fce8807e720 0x7fce8807eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:03:49.535 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.524+0000 7fce8c812700 1 -- 172.21.15.42:0/3887984844 shutdown_connections 2019-12-05T05:03:49.535 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.524+0000 7fce8c812700 1 -- 172.21.15.42:0/3887984844 wait complete. 2019-12-05T05:03:49.535 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.524+0000 7fce8c812700 1 -- 172.21.15.42:0/3887984844 >> 172.21.15.42:0/3887984844 conn(0x7fce880aabe0 msgr2=0x7fce880a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:49.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.524+0000 7fce8c812700 1 Processor -- start 2019-12-05T05:03:49.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.524+0000 7fce8c812700 1 -- start start 2019-12-05T05:03:49.539 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.528+0000 7fce8c812700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fce8811f070 0x7fce8811a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:49.539 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.528+0000 7fce8c812700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fce881448a0 0x7fce881318d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:49.539 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.528+0000 7fce8c812700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce88132250 con 0x7fce88131e20 2019-12-05T05:03:49.539 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.528+0000 7fce8c812700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce8811f4a0 con 0x7fce8811f070 2019-12-05T05:03:49.539 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.528+0000 7fce8c812700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce88124d70 con 0x7fce881448a0 2019-12-05T05:03:49.539 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.528+0000 7fce8659c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fce881448a0 0x7fce881318d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:49.539 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.528+0000 7fce8659c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fce881448a0 0x7fce881318d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:53804/0 (socket says 172.21.15.42:53804) 2019-12-05T05:03:49.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.528+0000 7fce8659c700 1 -- 172.21.15.42:0/443227891 learned_addr learned my addr 172.21.15.42:0/443227891 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:03:49.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.528+0000 7fce86d9d700 1 --2- 172.21.15.42:0/443227891 >> v2:172.21.15.40:3300/0 conn(0x7fce8811f070 0x7fce8811a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:49.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.528+0000 7fce8659c700 1 -- 172.21.15.42:0/443227891 >> v2:172.21.15.40:3300/0 conn(0x7fce8811f070 msgr2=0x7fce8811a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:49.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.528+0000 7fce8659c700 1 --2- 172.21.15.42:0/443227891 >> v2:172.21.15.40:3300/0 conn(0x7fce8811f070 0x7fce8811a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:49.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.528+0000 7fce8659c700 1 -- 172.21.15.42:0/443227891 >> v1:172.21.15.40:6789/0 conn(0x7fce88131e20 legacy=0x7fce8811b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:49.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.528+0000 7fce8659c700 1 -- 172.21.15.42:0/443227891 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce8811fc40 con 0x7fce881448a0 2019-12-05T05:03:49.541 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.528+0000 7fce8659c700 1 --2- 172.21.15.42:0/443227891 >> v2:172.21.15.42:3300/0 conn(0x7fce881448a0 0x7fce881318d0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fce70003070 tx=0x7fce70010bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:49.541 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.528+0000 7fce8659c700 1 --2- 172.21.15.42:0/443227891 >> v2:172.21.15.42:3300/0 conn(0x7fce881448a0 0x7fce881318d0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fce70003070 tx=0x7fce70010bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:49.541 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.528+0000 7fce8659c700 1 --2- 172.21.15.42:0/443227891 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fce881448a0 0x7fce881318d0 secure :-1 s=READY pgs=184 cs=0 l=1 rx=0x7fce70003070 tx=0x7fce70010bc0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:49.541 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.528+0000 7fce8659c700 1 --2- 172.21.15.42:0/443227891 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fce881448a0 0x7fce881318d0 secure :-1 s=THROTTLE_DONE pgs=184 cs=0 l=1 rx=0x7fce70003070 tx=0x7fce70010bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:49.541 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.528+0000 7fce7f7fe700 1 -- 172.21.15.42:0/443227891 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fce70005890 con 0x7fce881448a0 2019-12-05T05:03:49.541 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.528+0000 7fce8659c700 1 --2- 172.21.15.42:0/443227891 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fce881448a0 0x7fce881318d0 secure :-1 s=THROTTLE_DONE pgs=184 cs=0 l=1 rx=0x7fce70003070 tx=0x7fce70010bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:49.541 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.528+0000 7fce8659c700 1 --2- 172.21.15.42:0/443227891 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fce881448a0 0x7fce881318d0 secure :-1 s=THROTTLE_DONE pgs=184 cs=0 l=1 rx=0x7fce70003070 tx=0x7fce70010bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:49.542 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.528+0000 7fce7f7fe700 1 -- 172.21.15.42:0/443227891 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce70007160 con 0x7fce881448a0 2019-12-05T05:03:49.542 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.528+0000 7fce7f7fe700 1 -- 172.21.15.42:0/443227891 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fce70012240 con 0x7fce881448a0 2019-12-05T05:03:49.542 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.528+0000 7fce8c812700 1 -- 172.21.15.42:0/443227891 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce88144cd0 con 0x7fce881448a0 2019-12-05T05:03:49.542 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.528+0000 7fce8c812700 1 -- 172.21.15.42:0/443227891 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce8811bc50 con 0x7fce881448a0 2019-12-05T05:03:49.542 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.528+0000 7fce8659c700 1 --2- 172.21.15.42:0/443227891 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fce881448a0 0x7fce881318d0 secure :-1 s=THROTTLE_DONE pgs=184 cs=0 l=1 rx=0x7fce70003070 tx=0x7fce70010bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:49.542 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.532+0000 7fce7f7fe700 1 -- 172.21.15.42:0/443227891 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fce70012550 con 0x7fce881448a0 2019-12-05T05:03:49.542 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.532+0000 7fce8659c700 1 --2- 172.21.15.42:0/443227891 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fce881448a0 0x7fce881318d0 secure :-1 s=THROTTLE_DONE pgs=184 cs=0 l=1 rx=0x7fce70003070 tx=0x7fce70010bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:49.543 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.532+0000 7fce7f7fe700 1 --2- 172.21.15.42:0/443227891 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fce6c03ade0 0x7fce6c03d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:49.543 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.532+0000 7fce7f7fe700 1 -- 172.21.15.42:0/443227891 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6031+0+0 (secure 0 0 0) 0x7fce7004b200 con 0x7fce881448a0 2019-12-05T05:03:49.544 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.532+0000 7fce86d9d700 1 --2- 172.21.15.42:0/443227891 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fce6c03ade0 0x7fce6c03d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:49.544 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.532+0000 7fce86d9d700 1 --2- 172.21.15.42:0/443227891 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fce6c03ade0 0x7fce6c03d2f0 crc :-1 s=READY pgs=98 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:49.544 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.532+0000 7fce8c812700 1 -- 172.21.15.42:0/443227891 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce8805d110 con 0x7fce881448a0 2019-12-05T05:03:49.544 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.540+0000 7fce8659c700 1 --2- 172.21.15.42:0/443227891 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fce881448a0 0x7fce881318d0 secure :-1 s=THROTTLE_DONE pgs=184 cs=0 l=1 rx=0x7fce70003070 tx=0x7fce70010bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:49.545 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.540+0000 7fce7f7fe700 1 -- 172.21.15.42:0/443227891 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fce70032280 con 0x7fce881448a0 2019-12-05T05:03:49.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.700+0000 7fce8c812700 1 -- 172.21.15.42:0/443227891 --> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fce8805b8c0 con 0x7fce6c03ade0 2019-12-05T05:03:49.711 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.704+0000 7fce7f7fe700 1 -- 172.21.15.42:0/443227891 <== mgr.4108 v2:172.21.15.42:6800/11930 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+80437 (crc 0 0 0) 0x7fce8805b8c0 con 0x7fce6c03ade0 2019-12-05T05:03:49.711 INFO:teuthology.orchestra.run.smithi042.stderr:dumped all 2019-12-05T05:03:49.711 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:03:49.713 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.708+0000 7fce8c812700 1 -- 172.21.15.42:0/443227891 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fce6c03ade0 msgr2=0x7fce6c03d2f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:49.713 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.708+0000 7fce8c812700 1 --2- 172.21.15.42:0/443227891 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fce6c03ade0 0x7fce6c03d2f0 crc :-1 s=READY pgs=98 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:49.713 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.708+0000 7fce8c812700 1 -- 172.21.15.42:0/443227891 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fce881448a0 msgr2=0x7fce881318d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:49.713 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.708+0000 7fce8c812700 1 --2- 172.21.15.42:0/443227891 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fce881448a0 0x7fce881318d0 secure :-1 s=READY pgs=184 cs=0 l=1 rx=0x7fce70003070 tx=0x7fce70010bc0).stop 2019-12-05T05:03:49.713 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.708+0000 7fce8c812700 1 -- 172.21.15.42:0/443227891 shutdown_connections 2019-12-05T05:03:49.713 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.708+0000 7fce8c812700 1 --2- 172.21.15.42:0/443227891 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fce6c03ade0 0x7fce6c03d2f0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:49.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.708+0000 7fce8c812700 1 --2- 172.21.15.42:0/443227891 >> v2:172.21.15.40:3300/0 conn(0x7fce8811f070 0x7fce8811a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:49.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.708+0000 7fce8c812700 1 --2- 172.21.15.42:0/443227891 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fce881448a0 0x7fce881318d0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:49.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.708+0000 7fce8c812700 1 -- 172.21.15.42:0/443227891 shutdown_connections 2019-12-05T05:03:49.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.708+0000 7fce8c812700 1 -- 172.21.15.42:0/443227891 wait complete. 2019-12-05T05:03:49.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.708+0000 7fce8c812700 1 -- 172.21.15.42:0/443227891 >> 172.21.15.42:0/443227891 conn(0x7fce880a7740 msgr2=0x7fce8811c3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:49.723 INFO:teuthology.orchestra.run.smithi042.stdout:{"pg_ready":true,"pg_map":{"version":38,"stamp":"2019-12-05T05:03:49.600238+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2286,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":8390720,"kb_used_data":2048,"kb_used_omap":0,"kb_used_meta":8388608,"kb_avail":746584000,"statfs":{"total":773094113280,"available":764502016000,"internally_reserved":8589934592,"allocated":2097152,"data_stored":772252,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8589934592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001215"},"pg_stats":[{"pgid":"3.7","version":"0'0","reported_seq":"12","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T05:03:35.610591+0000","last_change":"2019-12-05T05:03:35.333820+0000","last_active":"2019-12-05T05:03:35.610591+0000","last_peered":"2019-12-05T05:03:35.610591+0000","last_clean":"2019-12-05T05:03:35.610591+0000","last_became_active":"2019-12-05T05:03:35.332717+0000","last_became_peered":"2019-12-05T05:03:35.332717+0000","last_unstale":"2019-12-05T05:03:35.610591+0000","last_undegraded":"2019-12-05T05:03:35.610591+0000","last_fullsized":"2019-12-05T05:03:35.610591+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:34.235090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:34.235090+0000","last_clean_scrub_stamp":"2019-12-05T05:03:34.235090+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7,0],"acting":[3,5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":"16","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T05:03:35.731480+0000","last_change":"2019-12-05T05:03:29.184885+0000","last_active":"2019-12-05T05:03:35.731480+0000","last_peered":"2019-12-05T05:03:35.731480+0000","last_clean":"2019-12-05T05:03:35.731480+0000","last_became_active":"2019-12-05T05:03:29.184431+0000","last_became_peered":"2019-12-05T05:03:29.184431+0000","last_unstale":"2019-12-05T05:03:35.731480+0000","last_undegraded":"2019-12-05T05:03:35.731480+0000","last_fullsized":"2019-12-05T05:03:35.731480+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:28.161818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:28.161818+0000","last_clean_scrub_stamp":"2019-12-05T05:03:28.161818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"24'4","reported_seq":"17","reported_epoch":"24","state":"active+clean","last_fresh":"2019-12-05T05:03:42.683004+0000","last_change":"2019-12-05T05:03:32.225175+0000","last_active":"2019-12-05T05:03:42.683004+0000","last_peered":"2019-12-05T05:03:42.683004+0000","last_clean":"2019-12-05T05:03:42.683004+0000","last_became_active":"2019-12-05T05:03:32.224827+0000","last_became_peered":"2019-12-05T05:03:32.224827+0000","last_unstale":"2019-12-05T05:03:42.683004+0000","last_undegraded":"2019-12-05T05:03:42.683004+0000","last_fullsized":"2019-12-05T05:03:42.683004+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:31.200243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:31.200243+0000","last_clean_scrub_stamp":"2019-12-05T05:03:31.200243+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":"12","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T05:03:36.529724+0000","last_change":"2019-12-05T05:03:35.333820+0000","last_active":"2019-12-05T05:03:36.529724+0000","last_peered":"2019-12-05T05:03:36.529724+0000","last_clean":"2019-12-05T05:03:36.529724+0000","last_became_active":"2019-12-05T05:03:35.332648+0000","last_became_peered":"2019-12-05T05:03:35.332648+0000","last_unstale":"2019-12-05T05:03:36.529724+0000","last_undegraded":"2019-12-05T05:03:36.529724+0000","last_fullsized":"2019-12-05T05:03:36.529724+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:34.235090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:34.235090+0000","last_clean_scrub_stamp":"2019-12-05T05:03:34.235090+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,7],"acting":[0,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":"16","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T05:03:35.610218+0000","last_change":"2019-12-05T05:03:29.182769+0000","last_active":"2019-12-05T05:03:35.610218+0000","last_peered":"2019-12-05T05:03:35.610218+0000","last_clean":"2019-12-05T05:03:35.610218+0000","last_became_active":"2019-12-05T05:03:29.182409+0000","last_became_peered":"2019-12-05T05:03:29.182409+0000","last_unstale":"2019-12-05T05:03:35.610218+0000","last_undegraded":"2019-12-05T05:03:35.610218+0000","last_fullsized":"2019-12-05T05:03:35.610218+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:28.161818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:28.161818+0000","last_clean_scrub_stamp":"2019-12-05T05:03:28.161818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"24'7","reported_seq":"19","reported_epoch":"24","state":"active+clean","last_fresh":"2019-12-05T05:03:42.685484+0000","last_change":"2019-12-05T05:03:32.228176+0000","last_active":"2019-12-05T05:03:42.685484+0000","last_peered":"2019-12-05T05:03:42.685484+0000","last_clean":"2019-12-05T05:03:42.685484+0000","last_became_active":"2019-12-05T05:03:32.227690+0000","last_became_peered":"2019-12-05T05:03:32.227690+0000","last_unstale":"2019-12-05T05:03:42.685484+0000","last_undegraded":"2019-12-05T05:03:42.685484+0000","last_fullsized":"2019-12-05T05:03:42.685484+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:31.200243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:31.200243+0000","last_clean_scrub_stamp":"2019-12-05T05:03:31.200243+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":566,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":"12","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T05:03:36.529831+0000","last_change":"2019-12-05T05:03:35.267212+0000","last_active":"2019-12-05T05:03:36.529831+0000","last_peered":"2019-12-05T05:03:36.529831+0000","last_clean":"2019-12-05T05:03:36.529831+0000","last_became_active":"2019-12-05T05:03:35.266235+0000","last_became_peered":"2019-12-05T05:03:35.266235+0000","last_unstale":"2019-12-05T05:03:36.529831+0000","last_undegraded":"2019-12-05T05:03:36.529831+0000","last_fullsized":"2019-12-05T05:03:36.529831+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:34.235090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:34.235090+0000","last_clean_scrub_stamp":"2019-12-05T05:03:34.235090+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3,5],"acting":[0,4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":"16","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T05:03:35.626677+0000","last_change":"2019-12-05T05:03:29.181115+0000","last_active":"2019-12-05T05:03:35.626677+0000","last_peered":"2019-12-05T05:03:35.626677+0000","last_clean":"2019-12-05T05:03:35.626677+0000","last_became_active":"2019-12-05T05:03:29.180832+0000","last_became_peered":"2019-12-05T05:03:29.180832+0000","last_unstale":"2019-12-05T05:03:35.626677+0000","last_undegraded":"2019-12-05T05:03:35.626677+0000","last_fullsized":"2019-12-05T05:03:35.626677+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:28.161818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:28.161818+0000","last_clean_scrub_stamp":"2019-12-05T05:03:28.161818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"24'8","reported_seq":"21","reported_epoch":"24","state":"active+clean","last_fresh":"2019-12-05T05:03:42.687799+0000","last_change":"2019-12-05T05:03:32.227259+0000","last_active":"2019-12-05T05:03:42.687799+0000","last_peered":"2019-12-05T05:03:42.687799+0000","last_clean":"2019-12-05T05:03:42.687799+0000","last_became_active":"2019-12-05T05:03:32.226895+0000","last_became_peered":"2019-12-05T05:03:32.226895+0000","last_unstale":"2019-12-05T05:03:42.687799+0000","last_undegraded":"2019-12-05T05:03:42.687799+0000","last_fullsized":"2019-12-05T05:03:42.687799+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:31.200243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:31.200243+0000","last_clean_scrub_stamp":"2019-12-05T05:03:31.200243+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":"12","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T05:03:35.628786+0000","last_change":"2019-12-05T05:03:35.628207+0000","last_active":"2019-12-05T05:03:35.628786+0000","last_peered":"2019-12-05T05:03:35.628786+0000","last_clean":"2019-12-05T05:03:35.628786+0000","last_became_active":"2019-12-05T05:03:35.627352+0000","last_became_peered":"2019-12-05T05:03:35.627352+0000","last_unstale":"2019-12-05T05:03:35.628786+0000","last_undegraded":"2019-12-05T05:03:35.628786+0000","last_fullsized":"2019-12-05T05:03:35.628786+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:34.235090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:34.235090+0000","last_clean_scrub_stamp":"2019-12-05T05:03:34.235090+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6,5],"acting":[1,2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":"16","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T05:03:36.847322+0000","last_change":"2019-12-05T05:03:29.408129+0000","last_active":"2019-12-05T05:03:36.847322+0000","last_peered":"2019-12-05T05:03:36.847322+0000","last_clean":"2019-12-05T05:03:36.847322+0000","last_became_active":"2019-12-05T05:03:29.407108+0000","last_became_peered":"2019-12-05T05:03:29.407108+0000","last_unstale":"2019-12-05T05:03:36.847322+0000","last_undegraded":"2019-12-05T05:03:36.847322+0000","last_fullsized":"2019-12-05T05:03:36.847322+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:28.161818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:28.161818+0000","last_clean_scrub_stamp":"2019-12-05T05:03:28.161818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"13","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T05:03:35.610338+0000","last_change":"2019-12-05T05:03:32.636592+0000","last_active":"2019-12-05T05:03:35.610338+0000","last_peered":"2019-12-05T05:03:35.610338+0000","last_clean":"2019-12-05T05:03:35.610338+0000","last_became_active":"2019-12-05T05:03:32.635937+0000","last_became_peered":"2019-12-05T05:03:32.635937+0000","last_unstale":"2019-12-05T05:03:35.610338+0000","last_undegraded":"2019-12-05T05:03:35.610338+0000","last_fullsized":"2019-12-05T05:03:35.610338+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:31.200243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:31.200243+0000","last_clean_scrub_stamp":"2019-12-05T05:03:31.200243+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":"12","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T05:03:36.315636+0000","last_change":"2019-12-05T05:03:36.315160+0000","last_active":"2019-12-05T05:03:36.315636+0000","last_peered":"2019-12-05T05:03:36.315636+0000","last_clean":"2019-12-05T05:03:36.315636+0000","last_became_active":"2019-12-05T05:03:36.314006+0000","last_became_peered":"2019-12-05T05:03:36.314006+0000","last_unstale":"2019-12-05T05:03:36.315636+0000","last_undegraded":"2019-12-05T05:03:36.315636+0000","last_fullsized":"2019-12-05T05:03:36.315636+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:34.235090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:34.235090+0000","last_clean_scrub_stamp":"2019-12-05T05:03:34.235090+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6,4],"acting":[3,5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"16","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T05:03:36.847644+0000","last_change":"2019-12-05T05:03:29.407882+0000","last_active":"2019-12-05T05:03:36.847644+0000","last_peered":"2019-12-05T05:03:36.847644+0000","last_clean":"2019-12-05T05:03:36.847644+0000","last_became_active":"2019-12-05T05:03:29.406770+0000","last_became_peered":"2019-12-05T05:03:29.406770+0000","last_unstale":"2019-12-05T05:03:36.847644+0000","last_undegraded":"2019-12-05T05:03:36.847644+0000","last_fullsized":"2019-12-05T05:03:36.847644+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:28.161818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:28.161818+0000","last_clean_scrub_stamp":"2019-12-05T05:03:28.161818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"24'5","reported_seq":"18","reported_epoch":"24","state":"active+clean","last_fresh":"2019-12-05T05:03:42.682907+0000","last_change":"2019-12-05T05:03:32.227654+0000","last_active":"2019-12-05T05:03:42.682907+0000","last_peered":"2019-12-05T05:03:42.682907+0000","last_clean":"2019-12-05T05:03:42.682907+0000","last_became_active":"2019-12-05T05:03:32.227052+0000","last_became_peered":"2019-12-05T05:03:32.227052+0000","last_unstale":"2019-12-05T05:03:42.682907+0000","last_undegraded":"2019-12-05T05:03:42.682907+0000","last_fullsized":"2019-12-05T05:03:42.682907+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:31.200243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:31.200243+0000","last_clean_scrub_stamp":"2019-12-05T05:03:31.200243+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":"12","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T05:03:35.734103+0000","last_change":"2019-12-05T05:03:35.733650+0000","last_active":"2019-12-05T05:03:35.734103+0000","last_peered":"2019-12-05T05:03:35.734103+0000","last_clean":"2019-12-05T05:03:35.734103+0000","last_became_active":"2019-12-05T05:03:35.732405+0000","last_became_peered":"2019-12-05T05:03:35.732405+0000","last_unstale":"2019-12-05T05:03:35.734103+0000","last_undegraded":"2019-12-05T05:03:35.734103+0000","last_fullsized":"2019-12-05T05:03:35.734103+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:34.235090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:34.235090+0000","last_clean_scrub_stamp":"2019-12-05T05:03:34.235090+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0,6],"acting":[4,5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":"16","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T05:03:35.629127+0000","last_change":"2019-12-05T05:03:29.407539+0000","last_active":"2019-12-05T05:03:35.629127+0000","last_peered":"2019-12-05T05:03:35.629127+0000","last_clean":"2019-12-05T05:03:35.629127+0000","last_became_active":"2019-12-05T05:03:29.406903+0000","last_became_peered":"2019-12-05T05:03:29.406903+0000","last_unstale":"2019-12-05T05:03:35.629127+0000","last_undegraded":"2019-12-05T05:03:35.629127+0000","last_fullsized":"2019-12-05T05:03:35.629127+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:28.161818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:28.161818+0000","last_clean_scrub_stamp":"2019-12-05T05:03:28.161818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"24'1","reported_seq":"14","reported_epoch":"24","state":"active+clean","last_fresh":"2019-12-05T05:03:37.365505+0000","last_change":"2019-12-05T05:03:32.227838+0000","last_active":"2019-12-05T05:03:37.365505+0000","last_peered":"2019-12-05T05:03:37.365505+0000","last_clean":"2019-12-05T05:03:37.365505+0000","last_became_active":"2019-12-05T05:03:32.227050+0000","last_became_peered":"2019-12-05T05:03:32.227050+0000","last_unstale":"2019-12-05T05:03:37.365505+0000","last_undegraded":"2019-12-05T05:03:37.365505+0000","last_fullsized":"2019-12-05T05:03:37.365505+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:31.200243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:31.200243+0000","last_clean_scrub_stamp":"2019-12-05T05:03:31.200243+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":"12","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T05:03:35.626774+0000","last_change":"2019-12-05T05:03:35.267419+0000","last_active":"2019-12-05T05:03:35.626774+0000","last_peered":"2019-12-05T05:03:35.626774+0000","last_clean":"2019-12-05T05:03:35.626774+0000","last_became_active":"2019-12-05T05:03:35.266868+0000","last_became_peered":"2019-12-05T05:03:35.266868+0000","last_unstale":"2019-12-05T05:03:35.626774+0000","last_undegraded":"2019-12-05T05:03:35.626774+0000","last_fullsized":"2019-12-05T05:03:35.626774+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:34.235090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:34.235090+0000","last_clean_scrub_stamp":"2019-12-05T05:03:34.235090+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0,5],"acting":[1,2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":"15","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T05:03:35.624631+0000","last_change":"2019-12-05T05:03:29.407541+0000","last_active":"2019-12-05T05:03:35.624631+0000","last_peered":"2019-12-05T05:03:35.624631+0000","last_clean":"2019-12-05T05:03:35.624631+0000","last_became_active":"2019-12-05T05:03:29.406857+0000","last_became_peered":"2019-12-05T05:03:29.406857+0000","last_unstale":"2019-12-05T05:03:35.624631+0000","last_undegraded":"2019-12-05T05:03:35.624631+0000","last_fullsized":"2019-12-05T05:03:35.624631+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:28.161818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:28.161818+0000","last_clean_scrub_stamp":"2019-12-05T05:03:28.161818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"24'6","reported_seq":"19","reported_epoch":"24","state":"active+clean","last_fresh":"2019-12-05T05:03:42.686730+0000","last_change":"2019-12-05T05:03:32.480464+0000","last_active":"2019-12-05T05:03:42.686730+0000","last_peered":"2019-12-05T05:03:42.686730+0000","last_clean":"2019-12-05T05:03:42.686730+0000","last_became_active":"2019-12-05T05:03:32.479565+0000","last_became_peered":"2019-12-05T05:03:32.479565+0000","last_unstale":"2019-12-05T05:03:42.686730+0000","last_undegraded":"2019-12-05T05:03:42.686730+0000","last_fullsized":"2019-12-05T05:03:42.686730+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:31.200243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:31.200243+0000","last_clean_scrub_stamp":"2019-12-05T05:03:31.200243+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":160,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":"12","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T05:03:35.630204+0000","last_change":"2019-12-05T05:03:35.629791+0000","last_active":"2019-12-05T05:03:35.630204+0000","last_peered":"2019-12-05T05:03:35.630204+0000","last_clean":"2019-12-05T05:03:35.630204+0000","last_became_active":"2019-12-05T05:03:35.629300+0000","last_became_peered":"2019-12-05T05:03:35.629300+0000","last_unstale":"2019-12-05T05:03:35.630204+0000","last_undegraded":"2019-12-05T05:03:35.630204+0000","last_fullsized":"2019-12-05T05:03:35.630204+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:34.235090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:34.235090+0000","last_clean_scrub_stamp":"2019-12-05T05:03:34.235090+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2,6],"acting":[5,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":"15","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T05:03:35.624610+0000","last_change":"2019-12-05T05:03:29.185245+0000","last_active":"2019-12-05T05:03:35.624610+0000","last_peered":"2019-12-05T05:03:35.624610+0000","last_clean":"2019-12-05T05:03:35.624610+0000","last_became_active":"2019-12-05T05:03:29.184716+0000","last_became_peered":"2019-12-05T05:03:29.184716+0000","last_unstale":"2019-12-05T05:03:35.624610+0000","last_undegraded":"2019-12-05T05:03:35.624610+0000","last_fullsized":"2019-12-05T05:03:35.624610+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:28.161818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:28.161818+0000","last_clean_scrub_stamp":"2019-12-05T05:03:28.161818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"24'8","reported_seq":"21","reported_epoch":"24","state":"active+clean","last_fresh":"2019-12-05T05:03:42.685962+0000","last_change":"2019-12-05T05:03:32.476846+0000","last_active":"2019-12-05T05:03:42.685962+0000","last_peered":"2019-12-05T05:03:42.685962+0000","last_clean":"2019-12-05T05:03:42.685962+0000","last_became_active":"2019-12-05T05:03:32.476140+0000","last_became_peered":"2019-12-05T05:03:32.476140+0000","last_unstale":"2019-12-05T05:03:42.685962+0000","last_undegraded":"2019-12-05T05:03:42.685962+0000","last_fullsized":"2019-12-05T05:03:42.685962+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:31.200243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:31.200243+0000","last_clean_scrub_stamp":"2019-12-05T05:03:31.200243+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1526,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":32,"acting":32,"num_store_stats":8},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2286,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574856,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048848,"kb_used_data":264,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322992,"statfs":{"total":96636764160,"available":95562743808,"internally_reserved":1073741824,"allocated":270336,"data_stored":96686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.801}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.102}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.038}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.982}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.732}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.176}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.348}]}]},{"osd":6,"up_from":13,"seq":55834574857,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048840,"kb_used_data":256,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323000,"statfs":{"total":96636764160,"available":95562752000,"internally_reserved":1073741824,"allocated":262144,"data_stored":96526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.790}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.931}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.325}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.134}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.086}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.210}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.383}]}]},{"osd":1,"up_from":13,"seq":55834574855,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048844,"kb_used_data":260,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322996,"statfs":{"total":96636764160,"available":95562747904,"internally_reserved":1073741824,"allocated":266240,"data_stored":97486,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.506}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.781}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.696}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.934}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.014}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.900}]}]},{"osd":0,"up_from":13,"seq":55834574857,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048852,"kb_used_data":268,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322988,"statfs":{"total":96636764160,"available":95562739712,"internally_reserved":1073741824,"allocated":274432,"data_stored":97646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.639}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.961}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.879}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.150}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.118}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.306}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.220}]}]},{"osd":2,"up_from":13,"seq":55834574855,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048836,"kb_used_data":252,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323004,"statfs":{"total":96636764160,"available":95562756096,"internally_reserved":1073741824,"allocated":258048,"data_stored":95994,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.925}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.738}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.648}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.430}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.231}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.200}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.159}]}]},{"osd":3,"up_from":13,"seq":55834574856,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048832,"kb_used_data":248,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323008,"statfs":{"total":96636764160,"available":95562760192,"internally_reserved":1073741824,"allocated":253952,"data_stored":95960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.858}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.668}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.712}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.812}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.026}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.092}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.733}]}]},{"osd":4,"up_from":13,"seq":55834574855,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048832,"kb_used_data":248,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323008,"statfs":{"total":96636764160,"available":95562760192,"internally_reserved":1073741824,"allocated":253952,"data_stored":95960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.828}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.744}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.916}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.036}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.356}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.295}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.398}]}]},{"osd":5,"up_from":13,"seq":55834574856,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048836,"kb_used_data":252,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323004,"statfs":{"total":96636764160,"available":95562756096,"internally_reserved":1073741824,"allocated":258048,"data_stored":95994,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.787}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.990}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.647}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.889}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.097}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.022}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.137}]}]}]}} 2019-12-05T05:03:49.727 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2019-12-05T05:03:49.830 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.812+0000 7efda9cdc700 1 Processor -- start 2019-12-05T05:03:49.830 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.812+0000 7efda9cdc700 1 -- start start 2019-12-05T05:03:49.830 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.816+0000 7efda9cdc700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7efda40a7730 0x7efda408df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:49.830 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.816+0000 7efda9cdc700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efda409fad0 0x7efda407eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:49.831 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.816+0000 7efda9cdc700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efda40a0700 con 0x7efda407e710 2019-12-05T05:03:49.831 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.816+0000 7efda9cdc700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efda40a0af0 con 0x7efda40a7730 2019-12-05T05:03:49.831 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.816+0000 7efda9cdc700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efda408e5a0 con 0x7efda409fad0 2019-12-05T05:03:49.831 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.816+0000 7efda8a7a700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efda409fad0 0x7efda407eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:49.831 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.816+0000 7efda8a7a700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efda409fad0 0x7efda407eb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:54282/0 (socket says 172.21.15.42:54282) 2019-12-05T05:03:49.831 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.816+0000 7efda8a7a700 1 -- 172.21.15.42:0/1754239039 learned_addr learned my addr 172.21.15.42:0/1754239039 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:03:49.831 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.816+0000 7efda3fff700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7efda40a7730 0x7efda408df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:49.832 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.816+0000 7efda8a7a700 1 -- 172.21.15.42:0/1754239039 >> v2:172.21.15.40:3300/0 conn(0x7efda40a7730 msgr2=0x7efda408df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:03:49.832 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.816+0000 7efda8a7a700 1 --2- 172.21.15.42:0/1754239039 >> v2:172.21.15.40:3300/0 conn(0x7efda40a7730 0x7efda408df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:03:49.832 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.816+0000 7efda8a7a700 1 -- 172.21.15.42:0/1754239039 >> v1:172.21.15.40:6789/0 conn(0x7efda407e710 legacy=0x7efda409f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:49.832 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.816+0000 7efda8a7a700 1 -- 172.21.15.42:0/1754239039 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efda407f090 con 0x7efda409fad0 2019-12-05T05:03:49.832 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.816+0000 7efda8a7a700 1 --2- 172.21.15.42:0/1754239039 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efda409fad0 0x7efda407eb40 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7efd9c008790 tx=0x7efd9c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:49.832 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.816+0000 7efda8a7a700 1 --2- 172.21.15.42:0/1754239039 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efda409fad0 0x7efda407eb40 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7efd9c008790 tx=0x7efd9c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:49.833 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.816+0000 7efda8a7a700 1 --2- 172.21.15.42:0/1754239039 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efda409fad0 0x7efda407eb40 secure :-1 s=READY pgs=44 cs=0 l=1 rx=0x7efd9c008790 tx=0x7efd9c00fd30).ready entity=mon.2 client_cookie=abdb57ba6b4e88e2 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:49.833 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.816+0000 7efda8a7a700 1 --2- 172.21.15.42:0/1754239039 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efda409fad0 0x7efda407eb40 secure :-1 s=THROTTLE_DONE pgs=44 cs=0 l=1 rx=0x7efd9c008790 tx=0x7efd9c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:49.833 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.816+0000 7efda2ffd700 1 -- 172.21.15.42:0/1754239039 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efd9c008c90 con 0x7efda409fad0 2019-12-05T05:03:49.833 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.816+0000 7efda8a7a700 1 --2- 172.21.15.42:0/1754239039 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efda409fad0 0x7efda407eb40 secure :-1 s=THROTTLE_DONE pgs=44 cs=0 l=1 rx=0x7efd9c008790 tx=0x7efd9c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:49.833 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.816+0000 7efda2ffd700 1 -- 172.21.15.42:0/1754239039 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd9c0088b0 con 0x7efda409fad0 2019-12-05T05:03:49.834 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.816+0000 7efda8a7a700 1 --2- 172.21.15.42:0/1754239039 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efda409fad0 0x7efda407eb40 secure :-1 s=THROTTLE_DONE pgs=44 cs=0 l=1 rx=0x7efd9c008790 tx=0x7efd9c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:49.834 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.816+0000 7efda2ffd700 1 -- 172.21.15.42:0/1754239039 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efd9c011b10 con 0x7efda409fad0 2019-12-05T05:03:49.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.816+0000 7efda9cdc700 1 -- 172.21.15.42:0/1754239039 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efda409fad0 msgr2=0x7efda407eb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:49.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.816+0000 7efda9cdc700 1 --2- 172.21.15.42:0/1754239039 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efda409fad0 0x7efda407eb40 secure :-1 s=READY pgs=44 cs=0 l=1 rx=0x7efd9c008790 tx=0x7efd9c00fd30).stop 2019-12-05T05:03:49.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.816+0000 7efda9cdc700 1 -- 172.21.15.42:0/1754239039 shutdown_connections 2019-12-05T05:03:49.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.816+0000 7efda9cdc700 1 --2- 172.21.15.42:0/1754239039 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efda409fad0 0x7efda407eb40 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:49.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.816+0000 7efda9cdc700 1 --2- 172.21.15.42:0/1754239039 >> v2:172.21.15.40:3300/0 conn(0x7efda40a7730 0x7efda408df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:03:49.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.816+0000 7efda9cdc700 1 -- 172.21.15.42:0/1754239039 shutdown_connections 2019-12-05T05:03:49.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.816+0000 7efda9cdc700 1 -- 172.21.15.42:0/1754239039 wait complete. 2019-12-05T05:03:49.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.816+0000 7efda9cdc700 1 -- 172.21.15.42:0/1754239039 >> 172.21.15.42:0/1754239039 conn(0x7efda40aabd0 msgr2=0x7efda40a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:49.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.820+0000 7efda9cdc700 1 Processor -- start 2019-12-05T05:03:49.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.820+0000 7efda9cdc700 1 -- start start 2019-12-05T05:03:49.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.820+0000 7efda9cdc700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7efda411bcf0 0x7efda411abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:49.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.820+0000 7efda9cdc700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efda4132dc0 0x7efda41331f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:49.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.820+0000 7efda9cdc700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efda4133740 con 0x7efda4118530 2019-12-05T05:03:49.837 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.820+0000 7efda9cdc700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efda4118cb0 con 0x7efda411bcf0 2019-12-05T05:03:49.837 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.820+0000 7efda9cdc700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efda4117050 con 0x7efda4132dc0 2019-12-05T05:03:49.837 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.820+0000 7efda37fe700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7efda4118530 0x7efda41172f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:53856/0 (socket says 172.21.15.42:53856) 2019-12-05T05:03:49.837 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.820+0000 7efda37fe700 1 -- 172.21.15.42:0/4158861128 learned_addr learned my addr 172.21.15.42:0/4158861128 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:03:49.837 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.820+0000 7efda3fff700 1 --2- 172.21.15.42:0/4158861128 >> v2:172.21.15.40:3300/0 conn(0x7efda411bcf0 0x7efda411abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:49.837 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.820+0000 7efda8a7a700 1 --2- 172.21.15.42:0/4158861128 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efda4132dc0 0x7efda41331f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:49.837 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.820+0000 7efda0ff9700 1 -- 172.21.15.42:0/4158861128 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2674146549 0 0) 0x7efda4133740 con 0x7efda4118530 2019-12-05T05:03:49.838 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.820+0000 7efda0ff9700 1 -- 172.21.15.42:0/4158861128 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd880030b0 con 0x7efda4118530 2019-12-05T05:03:49.838 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.824+0000 7efda0ff9700 1 -- 172.21.15.42:0/4158861128 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 1932904774 0 0) 0x7efd880030b0 con 0x7efda4118530 2019-12-05T05:03:49.838 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.824+0000 7efda0ff9700 1 -- 172.21.15.42:0/4158861128 --> v1:172.21.15.42:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efda4133740 con 0x7efda4118530 2019-12-05T05:03:49.838 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.824+0000 7efda0ff9700 1 -- 172.21.15.42:0/4158861128 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7efd8c001a60 con 0x7efda4118530 2019-12-05T05:03:49.838 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.824+0000 7efda0ff9700 1 -- 172.21.15.42:0/4158861128 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efda4132dc0 msgr2=0x7efda41331f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:49.838 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.824+0000 7efda0ff9700 1 --2- 172.21.15.42:0/4158861128 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efda4132dc0 0x7efda41331f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:49.838 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.824+0000 7efda0ff9700 1 -- 172.21.15.42:0/4158861128 >> v2:172.21.15.40:3300/0 conn(0x7efda411bcf0 msgr2=0x7efda411abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:49.839 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.824+0000 7efda0ff9700 1 --2- 172.21.15.42:0/4158861128 >> v2:172.21.15.40:3300/0 conn(0x7efda411bcf0 0x7efda411abe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:49.839 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.824+0000 7efda0ff9700 1 -- 172.21.15.42:0/4158861128 >> v1:172.21.15.42:6789/0 conn(0x7efda4118530 legacy=0x7efda41172f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:49.839 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.824+0000 7efda0ff9700 1 --2- 172.21.15.42:0/4158861128 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efd880038a0 0x7efd88003cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:49.839 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.824+0000 7efda0ff9700 1 --2- 172.21.15.42:0/4158861128 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efd88004420 0x7efd88004850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:49.839 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.824+0000 7efda0ff9700 1 --2- 172.21.15.42:0/4158861128 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efd88004da0 0x7efd8800b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:49.839 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.824+0000 7efda0ff9700 1 -- 172.21.15.42:0/4158861128 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efda4118cb0 con 0x7efd88004da0 2019-12-05T05:03:49.839 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.824+0000 7efda0ff9700 1 -- 172.21.15.42:0/4158861128 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efda4117050 con 0x7efd88004420 2019-12-05T05:03:49.840 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.824+0000 7efda0ff9700 1 -- 172.21.15.42:0/4158861128 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd8800bbc0 con 0x7efd880038a0 2019-12-05T05:03:49.840 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.824+0000 7efda37fe700 1 --2- 172.21.15.42:0/4158861128 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efd88004420 0x7efd88004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:49.840 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.824+0000 7efda3fff700 1 --2- 172.21.15.42:0/4158861128 >> v2:172.21.15.40:3300/0 conn(0x7efda411bcf0 0x7efda411abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:03:49.840 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.824+0000 7efda8a7a700 1 --2- 172.21.15.42:0/4158861128 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efd88004da0 0x7efd8800b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:49.840 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.824+0000 7efda37fe700 1 -- 172.21.15.42:0/4158861128 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efd880038a0 msgr2=0x7efd88003cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:49.840 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.824+0000 7efda3fff700 1 --2- 172.21.15.42:0/4158861128 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efd880038a0 0x7efd88003cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:49.840 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.824+0000 7efda37fe700 1 --2- 172.21.15.42:0/4158861128 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efd880038a0 0x7efd88003cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:49.841 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.824+0000 7efda37fe700 1 -- 172.21.15.42:0/4158861128 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efd88004da0 msgr2=0x7efd8800b670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:49.841 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.824+0000 7efda37fe700 1 --2- 172.21.15.42:0/4158861128 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efd88004da0 0x7efd8800b670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:49.841 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.824+0000 7efda37fe700 1 -- 172.21.15.42:0/4158861128 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd88003460 con 0x7efd88004420 2019-12-05T05:03:49.841 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.824+0000 7efda9cdc700 1 -- 172.21.15.42:0/4158861128 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efda41328f0 con 0x7efd88004420 2019-12-05T05:03:49.841 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.824+0000 7efda37fe700 1 --2- 172.21.15.42:0/4158861128 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efd88004420 0x7efd88004850 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7efd8c001d60 tx=0x7efd8c011600).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:49.841 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.824+0000 7efda3fff700 1 --2- 172.21.15.42:0/4158861128 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efd880038a0 0x7efd88003cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).send_auth_request state changed! 2019-12-05T05:03:49.841 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.824+0000 7efda37fe700 1 --2- 172.21.15.42:0/4158861128 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efd88004420 0x7efd88004850 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7efd8c001d60 tx=0x7efd8c011600).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:49.842 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.824+0000 7efda9cdc700 1 -- 172.21.15.42:0/4158861128 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efda4117af0 con 0x7efd88004420 2019-12-05T05:03:49.842 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.824+0000 7efda37fe700 1 --2- 172.21.15.42:0/4158861128 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efd88004420 0x7efd88004850 secure :-1 s=READY pgs=186 cs=0 l=1 rx=0x7efd8c001d60 tx=0x7efd8c011600).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:49.842 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.824+0000 7efda37fe700 1 --2- 172.21.15.42:0/4158861128 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efd88004420 0x7efd88004850 secure :-1 s=THROTTLE_DONE pgs=186 cs=0 l=1 rx=0x7efd8c001d60 tx=0x7efd8c011600).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:49.842 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.824+0000 7efda37fe700 1 --2- 172.21.15.42:0/4158861128 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efd88004420 0x7efd88004850 secure :-1 s=THROTTLE_DONE pgs=186 cs=0 l=1 rx=0x7efd8c001d60 tx=0x7efd8c011600).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:49.842 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.824+0000 7efda0ff9700 1 -- 172.21.15.42:0/4158861128 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efd8c006b70 con 0x7efd88004420 2019-12-05T05:03:49.842 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.824+0000 7efda37fe700 1 --2- 172.21.15.42:0/4158861128 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efd88004420 0x7efd88004850 secure :-1 s=THROTTLE_DONE pgs=186 cs=0 l=1 rx=0x7efd8c001d60 tx=0x7efd8c011600).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:49.842 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.824+0000 7efda0ff9700 1 -- 172.21.15.42:0/4158861128 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd8c006790 con 0x7efd88004420 2019-12-05T05:03:49.842 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.824+0000 7efda0ff9700 1 -- 172.21.15.42:0/4158861128 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efd8c0096f0 con 0x7efd88004420 2019-12-05T05:03:49.843 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.824+0000 7efda37fe700 1 --2- 172.21.15.42:0/4158861128 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efd88004420 0x7efd88004850 secure :-1 s=THROTTLE_DONE pgs=186 cs=0 l=1 rx=0x7efd8c001d60 tx=0x7efd8c011600).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:49.844 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.824+0000 7efda0ff9700 1 -- 172.21.15.42:0/4158861128 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7efd8c0098f0 con 0x7efd88004420 2019-12-05T05:03:49.844 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.824+0000 7efda37fe700 1 --2- 172.21.15.42:0/4158861128 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efd88004420 0x7efd88004850 secure :-1 s=THROTTLE_DONE pgs=186 cs=0 l=1 rx=0x7efd8c001d60 tx=0x7efd8c011600).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:49.844 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.824+0000 7efda0ff9700 1 --2- 172.21.15.42:0/4158861128 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7efd88043a40 0x7efd88045f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:49.844 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.824+0000 7efda0ff9700 1 -- 172.21.15.42:0/4158861128 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6031+0+0 (secure 0 0 0) 0x7efd8c04c050 con 0x7efd88004420 2019-12-05T05:03:49.845 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.824+0000 7efda3fff700 1 --2- 172.21.15.42:0/4158861128 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7efd88043a40 0x7efd88045f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:49.845 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.828+0000 7efda3fff700 1 --2- 172.21.15.42:0/4158861128 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7efd88043a40 0x7efd88045f50 crc :-1 s=READY pgs=99 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:49.845 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.828+0000 7efda9cdc700 1 -- 172.21.15.42:0/4158861128 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd7c003200 con 0x7efd88004420 2019-12-05T05:03:49.845 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.836+0000 7efda37fe700 1 --2- 172.21.15.42:0/4158861128 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efd88004420 0x7efd88004850 secure :-1 s=THROTTLE_DONE pgs=186 cs=0 l=1 rx=0x7efd8c001d60 tx=0x7efd8c011600).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:49.846 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:49.836+0000 7efda0ff9700 1 -- 172.21.15.42:0/4158861128 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7efd8c0330d0 con 0x7efd88004420 2019-12-05T05:03:50.010 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.004+0000 7efda9cdc700 1 -- 172.21.15.42:0/4158861128 --> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7efd7c002010 con 0x7efd88043a40 2019-12-05T05:03:50.015 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.008+0000 7efda0ff9700 1 -- 172.21.15.42:0/4158861128 <== mgr.4108 v2:172.21.15.42:6800/11930 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+80437 (crc 0 0 0) 0x7efd7c002010 con 0x7efd88043a40 2019-12-05T05:03:50.026 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:03:50.026 INFO:teuthology.orchestra.run.smithi042.stderr:dumped all 2019-12-05T05:03:50.028 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.020+0000 7efda9cdc700 1 -- 172.21.15.42:0/4158861128 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7efd88043a40 msgr2=0x7efd88045f50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:50.028 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.020+0000 7efda9cdc700 1 --2- 172.21.15.42:0/4158861128 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7efd88043a40 0x7efd88045f50 crc :-1 s=READY pgs=99 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:50.028 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.020+0000 7efda9cdc700 1 -- 172.21.15.42:0/4158861128 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efd88004420 msgr2=0x7efd88004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:50.029 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.020+0000 7efda8a7a700 1 -- 172.21.15.42:0/4158861128 reap_dead start 2019-12-05T05:03:50.029 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.020+0000 7efda9cdc700 1 --2- 172.21.15.42:0/4158861128 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efd88004420 0x7efd88004850 secure :-1 s=READY pgs=186 cs=0 l=1 rx=0x7efd8c001d60 tx=0x7efd8c011600).stop 2019-12-05T05:03:50.029 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.020+0000 7efda9cdc700 1 -- 172.21.15.42:0/4158861128 shutdown_connections 2019-12-05T05:03:50.029 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.020+0000 7efda9cdc700 1 --2- 172.21.15.42:0/4158861128 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efd88004420 0x7efd88004850 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:50.029 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.020+0000 7efda9cdc700 1 -- 172.21.15.42:0/4158861128 shutdown_connections 2019-12-05T05:03:50.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.020+0000 7efda9cdc700 1 -- 172.21.15.42:0/4158861128 wait complete. 2019-12-05T05:03:50.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.020+0000 7efda9cdc700 1 -- 172.21.15.42:0/4158861128 >> 172.21.15.42:0/4158861128 conn(0x7efda40a9780 msgr2=0x7efda411d8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:50.040 INFO:teuthology.orchestra.run.smithi042.stdout:{"pg_ready":true,"pg_map":{"version":38,"stamp":"2019-12-05T05:03:49.600238+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2286,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":8390720,"kb_used_data":2048,"kb_used_omap":0,"kb_used_meta":8388608,"kb_avail":746584000,"statfs":{"total":773094113280,"available":764502016000,"internally_reserved":8589934592,"allocated":2097152,"data_stored":772252,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8589934592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001215"},"pg_stats":[{"pgid":"3.7","version":"0'0","reported_seq":"12","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T05:03:35.610591+0000","last_change":"2019-12-05T05:03:35.333820+0000","last_active":"2019-12-05T05:03:35.610591+0000","last_peered":"2019-12-05T05:03:35.610591+0000","last_clean":"2019-12-05T05:03:35.610591+0000","last_became_active":"2019-12-05T05:03:35.332717+0000","last_became_peered":"2019-12-05T05:03:35.332717+0000","last_unstale":"2019-12-05T05:03:35.610591+0000","last_undegraded":"2019-12-05T05:03:35.610591+0000","last_fullsized":"2019-12-05T05:03:35.610591+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:34.235090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:34.235090+0000","last_clean_scrub_stamp":"2019-12-05T05:03:34.235090+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7,0],"acting":[3,5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":"16","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T05:03:35.731480+0000","last_change":"2019-12-05T05:03:29.184885+0000","last_active":"2019-12-05T05:03:35.731480+0000","last_peered":"2019-12-05T05:03:35.731480+0000","last_clean":"2019-12-05T05:03:35.731480+0000","last_became_active":"2019-12-05T05:03:29.184431+0000","last_became_peered":"2019-12-05T05:03:29.184431+0000","last_unstale":"2019-12-05T05:03:35.731480+0000","last_undegraded":"2019-12-05T05:03:35.731480+0000","last_fullsized":"2019-12-05T05:03:35.731480+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:28.161818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:28.161818+0000","last_clean_scrub_stamp":"2019-12-05T05:03:28.161818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"24'4","reported_seq":"17","reported_epoch":"24","state":"active+clean","last_fresh":"2019-12-05T05:03:42.683004+0000","last_change":"2019-12-05T05:03:32.225175+0000","last_active":"2019-12-05T05:03:42.683004+0000","last_peered":"2019-12-05T05:03:42.683004+0000","last_clean":"2019-12-05T05:03:42.683004+0000","last_became_active":"2019-12-05T05:03:32.224827+0000","last_became_peered":"2019-12-05T05:03:32.224827+0000","last_unstale":"2019-12-05T05:03:42.683004+0000","last_undegraded":"2019-12-05T05:03:42.683004+0000","last_fullsized":"2019-12-05T05:03:42.683004+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:31.200243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:31.200243+0000","last_clean_scrub_stamp":"2019-12-05T05:03:31.200243+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":"12","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T05:03:36.529724+0000","last_change":"2019-12-05T05:03:35.333820+0000","last_active":"2019-12-05T05:03:36.529724+0000","last_peered":"2019-12-05T05:03:36.529724+0000","last_clean":"2019-12-05T05:03:36.529724+0000","last_became_active":"2019-12-05T05:03:35.332648+0000","last_became_peered":"2019-12-05T05:03:35.332648+0000","last_unstale":"2019-12-05T05:03:36.529724+0000","last_undegraded":"2019-12-05T05:03:36.529724+0000","last_fullsized":"2019-12-05T05:03:36.529724+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:34.235090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:34.235090+0000","last_clean_scrub_stamp":"2019-12-05T05:03:34.235090+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,7],"acting":[0,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":"16","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T05:03:35.610218+0000","last_change":"2019-12-05T05:03:29.182769+0000","last_active":"2019-12-05T05:03:35.610218+0000","last_peered":"2019-12-05T05:03:35.610218+0000","last_clean":"2019-12-05T05:03:35.610218+0000","last_became_active":"2019-12-05T05:03:29.182409+0000","last_became_peered":"2019-12-05T05:03:29.182409+0000","last_unstale":"2019-12-05T05:03:35.610218+0000","last_undegraded":"2019-12-05T05:03:35.610218+0000","last_fullsized":"2019-12-05T05:03:35.610218+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:28.161818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:28.161818+0000","last_clean_scrub_stamp":"2019-12-05T05:03:28.161818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"24'7","reported_seq":"19","reported_epoch":"24","state":"active+clean","last_fresh":"2019-12-05T05:03:42.685484+0000","last_change":"2019-12-05T05:03:32.228176+0000","last_active":"2019-12-05T05:03:42.685484+0000","last_peered":"2019-12-05T05:03:42.685484+0000","last_clean":"2019-12-05T05:03:42.685484+0000","last_became_active":"2019-12-05T05:03:32.227690+0000","last_became_peered":"2019-12-05T05:03:32.227690+0000","last_unstale":"2019-12-05T05:03:42.685484+0000","last_undegraded":"2019-12-05T05:03:42.685484+0000","last_fullsized":"2019-12-05T05:03:42.685484+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:31.200243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:31.200243+0000","last_clean_scrub_stamp":"2019-12-05T05:03:31.200243+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":566,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":"12","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T05:03:36.529831+0000","last_change":"2019-12-05T05:03:35.267212+0000","last_active":"2019-12-05T05:03:36.529831+0000","last_peered":"2019-12-05T05:03:36.529831+0000","last_clean":"2019-12-05T05:03:36.529831+0000","last_became_active":"2019-12-05T05:03:35.266235+0000","last_became_peered":"2019-12-05T05:03:35.266235+0000","last_unstale":"2019-12-05T05:03:36.529831+0000","last_undegraded":"2019-12-05T05:03:36.529831+0000","last_fullsized":"2019-12-05T05:03:36.529831+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:34.235090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:34.235090+0000","last_clean_scrub_stamp":"2019-12-05T05:03:34.235090+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3,5],"acting":[0,4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":"16","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T05:03:35.626677+0000","last_change":"2019-12-05T05:03:29.181115+0000","last_active":"2019-12-05T05:03:35.626677+0000","last_peered":"2019-12-05T05:03:35.626677+0000","last_clean":"2019-12-05T05:03:35.626677+0000","last_became_active":"2019-12-05T05:03:29.180832+0000","last_became_peered":"2019-12-05T05:03:29.180832+0000","last_unstale":"2019-12-05T05:03:35.626677+0000","last_undegraded":"2019-12-05T05:03:35.626677+0000","last_fullsized":"2019-12-05T05:03:35.626677+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:28.161818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:28.161818+0000","last_clean_scrub_stamp":"2019-12-05T05:03:28.161818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"24'8","reported_seq":"21","reported_epoch":"24","state":"active+clean","last_fresh":"2019-12-05T05:03:42.687799+0000","last_change":"2019-12-05T05:03:32.227259+0000","last_active":"2019-12-05T05:03:42.687799+0000","last_peered":"2019-12-05T05:03:42.687799+0000","last_clean":"2019-12-05T05:03:42.687799+0000","last_became_active":"2019-12-05T05:03:32.226895+0000","last_became_peered":"2019-12-05T05:03:32.226895+0000","last_unstale":"2019-12-05T05:03:42.687799+0000","last_undegraded":"2019-12-05T05:03:42.687799+0000","last_fullsized":"2019-12-05T05:03:42.687799+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:31.200243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:31.200243+0000","last_clean_scrub_stamp":"2019-12-05T05:03:31.200243+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":"12","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T05:03:35.628786+0000","last_change":"2019-12-05T05:03:35.628207+0000","last_active":"2019-12-05T05:03:35.628786+0000","last_peered":"2019-12-05T05:03:35.628786+0000","last_clean":"2019-12-05T05:03:35.628786+0000","last_became_active":"2019-12-05T05:03:35.627352+0000","last_became_peered":"2019-12-05T05:03:35.627352+0000","last_unstale":"2019-12-05T05:03:35.628786+0000","last_undegraded":"2019-12-05T05:03:35.628786+0000","last_fullsized":"2019-12-05T05:03:35.628786+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:34.235090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:34.235090+0000","last_clean_scrub_stamp":"2019-12-05T05:03:34.235090+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6,5],"acting":[1,2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":"16","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T05:03:36.847322+0000","last_change":"2019-12-05T05:03:29.408129+0000","last_active":"2019-12-05T05:03:36.847322+0000","last_peered":"2019-12-05T05:03:36.847322+0000","last_clean":"2019-12-05T05:03:36.847322+0000","last_became_active":"2019-12-05T05:03:29.407108+0000","last_became_peered":"2019-12-05T05:03:29.407108+0000","last_unstale":"2019-12-05T05:03:36.847322+0000","last_undegraded":"2019-12-05T05:03:36.847322+0000","last_fullsized":"2019-12-05T05:03:36.847322+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:28.161818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:28.161818+0000","last_clean_scrub_stamp":"2019-12-05T05:03:28.161818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"13","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T05:03:35.610338+0000","last_change":"2019-12-05T05:03:32.636592+0000","last_active":"2019-12-05T05:03:35.610338+0000","last_peered":"2019-12-05T05:03:35.610338+0000","last_clean":"2019-12-05T05:03:35.610338+0000","last_became_active":"2019-12-05T05:03:32.635937+0000","last_became_peered":"2019-12-05T05:03:32.635937+0000","last_unstale":"2019-12-05T05:03:35.610338+0000","last_undegraded":"2019-12-05T05:03:35.610338+0000","last_fullsized":"2019-12-05T05:03:35.610338+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:31.200243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:31.200243+0000","last_clean_scrub_stamp":"2019-12-05T05:03:31.200243+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":"12","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T05:03:36.315636+0000","last_change":"2019-12-05T05:03:36.315160+0000","last_active":"2019-12-05T05:03:36.315636+0000","last_peered":"2019-12-05T05:03:36.315636+0000","last_clean":"2019-12-05T05:03:36.315636+0000","last_became_active":"2019-12-05T05:03:36.314006+0000","last_became_peered":"2019-12-05T05:03:36.314006+0000","last_unstale":"2019-12-05T05:03:36.315636+0000","last_undegraded":"2019-12-05T05:03:36.315636+0000","last_fullsized":"2019-12-05T05:03:36.315636+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:34.235090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:34.235090+0000","last_clean_scrub_stamp":"2019-12-05T05:03:34.235090+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6,4],"acting":[3,5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"16","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T05:03:36.847644+0000","last_change":"2019-12-05T05:03:29.407882+0000","last_active":"2019-12-05T05:03:36.847644+0000","last_peered":"2019-12-05T05:03:36.847644+0000","last_clean":"2019-12-05T05:03:36.847644+0000","last_became_active":"2019-12-05T05:03:29.406770+0000","last_became_peered":"2019-12-05T05:03:29.406770+0000","last_unstale":"2019-12-05T05:03:36.847644+0000","last_undegraded":"2019-12-05T05:03:36.847644+0000","last_fullsized":"2019-12-05T05:03:36.847644+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:28.161818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:28.161818+0000","last_clean_scrub_stamp":"2019-12-05T05:03:28.161818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"24'5","reported_seq":"18","reported_epoch":"24","state":"active+clean","last_fresh":"2019-12-05T05:03:42.682907+0000","last_change":"2019-12-05T05:03:32.227654+0000","last_active":"2019-12-05T05:03:42.682907+0000","last_peered":"2019-12-05T05:03:42.682907+0000","last_clean":"2019-12-05T05:03:42.682907+0000","last_became_active":"2019-12-05T05:03:32.227052+0000","last_became_peered":"2019-12-05T05:03:32.227052+0000","last_unstale":"2019-12-05T05:03:42.682907+0000","last_undegraded":"2019-12-05T05:03:42.682907+0000","last_fullsized":"2019-12-05T05:03:42.682907+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:31.200243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:31.200243+0000","last_clean_scrub_stamp":"2019-12-05T05:03:31.200243+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":"12","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T05:03:35.734103+0000","last_change":"2019-12-05T05:03:35.733650+0000","last_active":"2019-12-05T05:03:35.734103+0000","last_peered":"2019-12-05T05:03:35.734103+0000","last_clean":"2019-12-05T05:03:35.734103+0000","last_became_active":"2019-12-05T05:03:35.732405+0000","last_became_peered":"2019-12-05T05:03:35.732405+0000","last_unstale":"2019-12-05T05:03:35.734103+0000","last_undegraded":"2019-12-05T05:03:35.734103+0000","last_fullsized":"2019-12-05T05:03:35.734103+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:34.235090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:34.235090+0000","last_clean_scrub_stamp":"2019-12-05T05:03:34.235090+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0,6],"acting":[4,5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":"16","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T05:03:35.629127+0000","last_change":"2019-12-05T05:03:29.407539+0000","last_active":"2019-12-05T05:03:35.629127+0000","last_peered":"2019-12-05T05:03:35.629127+0000","last_clean":"2019-12-05T05:03:35.629127+0000","last_became_active":"2019-12-05T05:03:29.406903+0000","last_became_peered":"2019-12-05T05:03:29.406903+0000","last_unstale":"2019-12-05T05:03:35.629127+0000","last_undegraded":"2019-12-05T05:03:35.629127+0000","last_fullsized":"2019-12-05T05:03:35.629127+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:28.161818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:28.161818+0000","last_clean_scrub_stamp":"2019-12-05T05:03:28.161818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"24'1","reported_seq":"14","reported_epoch":"24","state":"active+clean","last_fresh":"2019-12-05T05:03:37.365505+0000","last_change":"2019-12-05T05:03:32.227838+0000","last_active":"2019-12-05T05:03:37.365505+0000","last_peered":"2019-12-05T05:03:37.365505+0000","last_clean":"2019-12-05T05:03:37.365505+0000","last_became_active":"2019-12-05T05:03:32.227050+0000","last_became_peered":"2019-12-05T05:03:32.227050+0000","last_unstale":"2019-12-05T05:03:37.365505+0000","last_undegraded":"2019-12-05T05:03:37.365505+0000","last_fullsized":"2019-12-05T05:03:37.365505+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:31.200243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:31.200243+0000","last_clean_scrub_stamp":"2019-12-05T05:03:31.200243+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":"12","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T05:03:35.626774+0000","last_change":"2019-12-05T05:03:35.267419+0000","last_active":"2019-12-05T05:03:35.626774+0000","last_peered":"2019-12-05T05:03:35.626774+0000","last_clean":"2019-12-05T05:03:35.626774+0000","last_became_active":"2019-12-05T05:03:35.266868+0000","last_became_peered":"2019-12-05T05:03:35.266868+0000","last_unstale":"2019-12-05T05:03:35.626774+0000","last_undegraded":"2019-12-05T05:03:35.626774+0000","last_fullsized":"2019-12-05T05:03:35.626774+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:34.235090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:34.235090+0000","last_clean_scrub_stamp":"2019-12-05T05:03:34.235090+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0,5],"acting":[1,2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":"15","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T05:03:35.624631+0000","last_change":"2019-12-05T05:03:29.407541+0000","last_active":"2019-12-05T05:03:35.624631+0000","last_peered":"2019-12-05T05:03:35.624631+0000","last_clean":"2019-12-05T05:03:35.624631+0000","last_became_active":"2019-12-05T05:03:29.406857+0000","last_became_peered":"2019-12-05T05:03:29.406857+0000","last_unstale":"2019-12-05T05:03:35.624631+0000","last_undegraded":"2019-12-05T05:03:35.624631+0000","last_fullsized":"2019-12-05T05:03:35.624631+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:28.161818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:28.161818+0000","last_clean_scrub_stamp":"2019-12-05T05:03:28.161818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"24'6","reported_seq":"19","reported_epoch":"24","state":"active+clean","last_fresh":"2019-12-05T05:03:42.686730+0000","last_change":"2019-12-05T05:03:32.480464+0000","last_active":"2019-12-05T05:03:42.686730+0000","last_peered":"2019-12-05T05:03:42.686730+0000","last_clean":"2019-12-05T05:03:42.686730+0000","last_became_active":"2019-12-05T05:03:32.479565+0000","last_became_peered":"2019-12-05T05:03:32.479565+0000","last_unstale":"2019-12-05T05:03:42.686730+0000","last_undegraded":"2019-12-05T05:03:42.686730+0000","last_fullsized":"2019-12-05T05:03:42.686730+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:31.200243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:31.200243+0000","last_clean_scrub_stamp":"2019-12-05T05:03:31.200243+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":160,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":"12","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T05:03:35.630204+0000","last_change":"2019-12-05T05:03:35.629791+0000","last_active":"2019-12-05T05:03:35.630204+0000","last_peered":"2019-12-05T05:03:35.630204+0000","last_clean":"2019-12-05T05:03:35.630204+0000","last_became_active":"2019-12-05T05:03:35.629300+0000","last_became_peered":"2019-12-05T05:03:35.629300+0000","last_unstale":"2019-12-05T05:03:35.630204+0000","last_undegraded":"2019-12-05T05:03:35.630204+0000","last_fullsized":"2019-12-05T05:03:35.630204+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:34.235090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:34.235090+0000","last_clean_scrub_stamp":"2019-12-05T05:03:34.235090+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2,6],"acting":[5,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":"15","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T05:03:35.624610+0000","last_change":"2019-12-05T05:03:29.185245+0000","last_active":"2019-12-05T05:03:35.624610+0000","last_peered":"2019-12-05T05:03:35.624610+0000","last_clean":"2019-12-05T05:03:35.624610+0000","last_became_active":"2019-12-05T05:03:29.184716+0000","last_became_peered":"2019-12-05T05:03:29.184716+0000","last_unstale":"2019-12-05T05:03:35.624610+0000","last_undegraded":"2019-12-05T05:03:35.624610+0000","last_fullsized":"2019-12-05T05:03:35.624610+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:28.161818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:28.161818+0000","last_clean_scrub_stamp":"2019-12-05T05:03:28.161818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"24'8","reported_seq":"21","reported_epoch":"24","state":"active+clean","last_fresh":"2019-12-05T05:03:42.685962+0000","last_change":"2019-12-05T05:03:32.476846+0000","last_active":"2019-12-05T05:03:42.685962+0000","last_peered":"2019-12-05T05:03:42.685962+0000","last_clean":"2019-12-05T05:03:42.685962+0000","last_became_active":"2019-12-05T05:03:32.476140+0000","last_became_peered":"2019-12-05T05:03:32.476140+0000","last_unstale":"2019-12-05T05:03:42.685962+0000","last_undegraded":"2019-12-05T05:03:42.685962+0000","last_fullsized":"2019-12-05T05:03:42.685962+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:31.200243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:31.200243+0000","last_clean_scrub_stamp":"2019-12-05T05:03:31.200243+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1526,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":32,"acting":32,"num_store_stats":8},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2286,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574856,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048848,"kb_used_data":264,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322992,"statfs":{"total":96636764160,"available":95562743808,"internally_reserved":1073741824,"allocated":270336,"data_stored":96686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.801}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.102}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.038}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.982}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.732}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.176}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.348}]}]},{"osd":6,"up_from":13,"seq":55834574857,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048840,"kb_used_data":256,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323000,"statfs":{"total":96636764160,"available":95562752000,"internally_reserved":1073741824,"allocated":262144,"data_stored":96526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.790}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.931}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.325}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.134}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.086}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.210}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.383}]}]},{"osd":1,"up_from":13,"seq":55834574855,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048844,"kb_used_data":260,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322996,"statfs":{"total":96636764160,"available":95562747904,"internally_reserved":1073741824,"allocated":266240,"data_stored":97486,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.506}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.781}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.696}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.934}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.014}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.900}]}]},{"osd":0,"up_from":13,"seq":55834574857,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048852,"kb_used_data":268,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322988,"statfs":{"total":96636764160,"available":95562739712,"internally_reserved":1073741824,"allocated":274432,"data_stored":97646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.639}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.961}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.879}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.150}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.118}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.306}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.220}]}]},{"osd":2,"up_from":13,"seq":55834574855,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048836,"kb_used_data":252,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323004,"statfs":{"total":96636764160,"available":95562756096,"internally_reserved":1073741824,"allocated":258048,"data_stored":95994,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.925}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.738}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.648}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.430}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.231}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.200}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.159}]}]},{"osd":3,"up_from":13,"seq":55834574856,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048832,"kb_used_data":248,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323008,"statfs":{"total":96636764160,"available":95562760192,"internally_reserved":1073741824,"allocated":253952,"data_stored":95960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.858}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.668}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.712}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.812}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.026}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.092}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.733}]}]},{"osd":4,"up_from":13,"seq":55834574855,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048832,"kb_used_data":248,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323008,"statfs":{"total":96636764160,"available":95562760192,"internally_reserved":1073741824,"allocated":253952,"data_stored":95960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.828}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.744}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.916}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.036}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.356}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.295}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.398}]}]},{"osd":5,"up_from":13,"seq":55834574856,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048836,"kb_used_data":252,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323004,"statfs":{"total":96636764160,"available":95562756096,"internally_reserved":1073741824,"allocated":258048,"data_stored":95994,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.787}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.990}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.647}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.889}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.097}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.022}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.137}]}]}]}} 2019-12-05T05:03:50.045 INFO:tasks.mds_thrash.ceph_manager:clean! 2019-12-05T05:03:50.045 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2019-12-05T05:03:50.150 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.136+0000 7fda65409700 1 Processor -- start 2019-12-05T05:03:50.151 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.136+0000 7fda65409700 1 -- start start 2019-12-05T05:03:50.151 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.136+0000 7fda65409700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fda600a7740 0x7fda6008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:50.151 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.136+0000 7fda65409700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda600a0710 con 0x7fda6007e720 2019-12-05T05:03:50.151 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.136+0000 7fda65409700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda600a9ba0 con 0x7fda6009fae0 2019-12-05T05:03:50.152 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.136+0000 7fda65409700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda600a0b00 con 0x7fda600a7740 2019-12-05T05:03:50.152 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.136+0000 7fda5ffff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fda6009fae0 0x7fda6014efe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:53874/0 (socket says 172.21.15.42:53874) 2019-12-05T05:03:50.152 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.136+0000 7fda5ffff700 1 -- 172.21.15.42:0/2226718157 learned_addr learned my addr 172.21.15.42:0/2226718157 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:03:50.152 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.136+0000 7fda5f7fe700 1 --2- 172.21.15.42:0/2226718157 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fda600a7740 0x7fda6008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:50.152 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.136+0000 7fda5e7fc700 1 -- 172.21.15.42:0/2226718157 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3304232879 0 0) 0x7fda600a9ba0 con 0x7fda6009fae0 2019-12-05T05:03:50.153 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.140+0000 7fda5e7fc700 1 -- 172.21.15.42:0/2226718157 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda4c0030b0 con 0x7fda6009fae0 2019-12-05T05:03:50.153 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.140+0000 7fda5e7fc700 1 -- 172.21.15.42:0/2226718157 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 673604968 0 0) 0x7fda600a0710 con 0x7fda6007e720 2019-12-05T05:03:50.153 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.140+0000 7fda5e7fc700 1 -- 172.21.15.42:0/2226718157 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda600a9ba0 con 0x7fda6007e720 2019-12-05T05:03:50.153 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.140+0000 7fda5f7fe700 1 -- 172.21.15.42:0/2226718157 >> v1:172.21.15.42:6789/0 conn(0x7fda6009fae0 legacy=0x7fda6014efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:50.154 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.140+0000 7fda5f7fe700 1 -- 172.21.15.42:0/2226718157 >> v1:172.21.15.40:6789/0 conn(0x7fda6007e720 legacy=0x7fda6009f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:50.154 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.140+0000 7fda5f7fe700 1 -- 172.21.15.42:0/2226718157 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda6007eb50 con 0x7fda600a7740 2019-12-05T05:03:50.154 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.140+0000 7fda5f7fe700 1 --2- 172.21.15.42:0/2226718157 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fda600a7740 0x7fda6008df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fda48002f30 tx=0x7fda4800fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:50.154 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.140+0000 7fda5f7fe700 1 --2- 172.21.15.42:0/2226718157 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fda600a7740 0x7fda6008df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fda48002f30 tx=0x7fda4800fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:50.154 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.140+0000 7fda5f7fe700 1 --2- 172.21.15.42:0/2226718157 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fda600a7740 0x7fda6008df80 secure :-1 s=READY pgs=45 cs=0 l=1 rx=0x7fda48002f30 tx=0x7fda4800fbf0).ready entity=mon.2 client_cookie=5c3eb3e3402ecf7b server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:50.155 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.140+0000 7fda5f7fe700 1 --2- 172.21.15.42:0/2226718157 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fda600a7740 0x7fda6008df80 secure :-1 s=THROTTLE_DONE pgs=45 cs=0 l=1 rx=0x7fda48002f30 tx=0x7fda4800fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:50.155 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.140+0000 7fda5e7fc700 1 -- 172.21.15.42:0/2226718157 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fda480112b0 con 0x7fda600a7740 2019-12-05T05:03:50.155 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.140+0000 7fda5f7fe700 1 --2- 172.21.15.42:0/2226718157 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fda600a7740 0x7fda6008df80 secure :-1 s=THROTTLE_DONE pgs=45 cs=0 l=1 rx=0x7fda48002f30 tx=0x7fda4800fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:50.155 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.140+0000 7fda5e7fc700 1 -- 172.21.15.42:0/2226718157 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda48008bc0 con 0x7fda600a7740 2019-12-05T05:03:50.156 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.140+0000 7fda5f7fe700 1 --2- 172.21.15.42:0/2226718157 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fda600a7740 0x7fda6008df80 secure :-1 s=THROTTLE_DONE pgs=45 cs=0 l=1 rx=0x7fda48002f30 tx=0x7fda4800fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:50.156 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.140+0000 7fda5e7fc700 1 -- 172.21.15.42:0/2226718157 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fda48011890 con 0x7fda600a7740 2019-12-05T05:03:50.156 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.140+0000 7fda65409700 1 -- 172.21.15.42:0/2226718157 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fda600a7740 msgr2=0x7fda6008df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:50.156 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.140+0000 7fda65409700 1 --2- 172.21.15.42:0/2226718157 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fda600a7740 0x7fda6008df80 secure :-1 s=READY pgs=45 cs=0 l=1 rx=0x7fda48002f30 tx=0x7fda4800fbf0).stop 2019-12-05T05:03:50.157 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.140+0000 7fda65409700 1 -- 172.21.15.42:0/2226718157 shutdown_connections 2019-12-05T05:03:50.157 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.140+0000 7fda65409700 1 --2- 172.21.15.42:0/2226718157 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fda600a7740 0x7fda6008df80 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:50.157 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.140+0000 7fda65409700 1 -- 172.21.15.42:0/2226718157 shutdown_connections 2019-12-05T05:03:50.157 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.140+0000 7fda65409700 1 -- 172.21.15.42:0/2226718157 wait complete. 2019-12-05T05:03:50.157 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.140+0000 7fda65409700 1 -- 172.21.15.42:0/2226718157 >> 172.21.15.42:0/2226718157 conn(0x7fda600aabe0 msgr2=0x7fda600a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:50.158 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.140+0000 7fda65409700 1 Processor -- start 2019-12-05T05:03:50.158 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.140+0000 7fda65409700 1 -- start start 2019-12-05T05:03:50.158 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.140+0000 7fda65409700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fda600a2ec0 0x7fda6007ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:50.158 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.140+0000 7fda65409700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fda6007bf90 0x7fda601201e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:50.159 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.140+0000 7fda65409700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda600a39e0 con 0x7fda600a6b90 2019-12-05T05:03:50.159 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.140+0000 7fda65409700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda600a34a0 con 0x7fda6007bf90 2019-12-05T05:03:50.159 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.140+0000 7fda65409700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda600a3740 con 0x7fda600a2ec0 2019-12-05T05:03:50.159 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.144+0000 7fda5effd700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fda600a2ec0 0x7fda6007ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:50.159 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.144+0000 7fda5ffff700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fda6007bf90 0x7fda601201e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:50.160 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.144+0000 7fda5effd700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fda600a2ec0 0x7fda6007ba40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:54308/0 (socket says 172.21.15.42:54308) 2019-12-05T05:03:50.160 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.144+0000 7fda5effd700 1 -- 172.21.15.42:0/2557007967 learned_addr learned my addr 172.21.15.42:0/2557007967 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:03:50.160 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.144+0000 7fda5ffff700 1 -- 172.21.15.42:0/2557007967 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fda600a2ec0 msgr2=0x7fda6007ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:50.160 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.144+0000 7fda5ffff700 1 --2- 172.21.15.42:0/2557007967 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fda600a2ec0 0x7fda6007ba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:50.161 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.144+0000 7fda5ffff700 1 -- 172.21.15.42:0/2557007967 >> v1:172.21.15.40:6789/0 conn(0x7fda600a6b90 legacy=0x7fda600a1a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:50.161 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.144+0000 7fda57fff700 1 -- 172.21.15.42:0/2557007967 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 692807734 0 0) 0x7fda600a39e0 con 0x7fda600a6b90 2019-12-05T05:03:50.161 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.144+0000 7fda5ffff700 1 -- 172.21.15.42:0/2557007967 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda60120730 con 0x7fda6007bf90 2019-12-05T05:03:50.161 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.144+0000 7fda5effd700 1 --2- 172.21.15.42:0/2557007967 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fda600a2ec0 0x7fda6007ba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:03:50.162 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.144+0000 7fda5ffff700 1 --2- 172.21.15.42:0/2557007967 >> v2:172.21.15.40:3300/0 conn(0x7fda6007bf90 0x7fda601201e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fda600a3660 tx=0x7fda50010830).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:50.162 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.144+0000 7fda5ffff700 1 --2- 172.21.15.42:0/2557007967 >> v2:172.21.15.40:3300/0 conn(0x7fda6007bf90 0x7fda601201e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fda600a3660 tx=0x7fda50010830).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:50.162 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.144+0000 7fda5ffff700 1 --2- 172.21.15.42:0/2557007967 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fda6007bf90 0x7fda601201e0 secure :-1 s=READY pgs=155 cs=0 l=1 rx=0x7fda600a3660 tx=0x7fda50010830).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:50.162 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.144+0000 7fda5ffff700 1 --2- 172.21.15.42:0/2557007967 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fda6007bf90 0x7fda601201e0 secure :-1 s=THROTTLE_DONE pgs=155 cs=0 l=1 rx=0x7fda600a3660 tx=0x7fda50010830).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:50.162 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.144+0000 7fda57fff700 1 -- 172.21.15.42:0/2557007967 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fda50005b40 con 0x7fda6007bf90 2019-12-05T05:03:50.163 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.144+0000 7fda5ffff700 1 --2- 172.21.15.42:0/2557007967 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fda6007bf90 0x7fda601201e0 secure :-1 s=THROTTLE_DONE pgs=155 cs=0 l=1 rx=0x7fda600a3660 tx=0x7fda50010830).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:50.163 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.144+0000 7fda57fff700 1 -- 172.21.15.42:0/2557007967 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda50005760 con 0x7fda6007bf90 2019-12-05T05:03:50.163 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.144+0000 7fda5ffff700 1 --2- 172.21.15.42:0/2557007967 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fda6007bf90 0x7fda601201e0 secure :-1 s=THROTTLE_DONE pgs=155 cs=0 l=1 rx=0x7fda600a3660 tx=0x7fda50010830).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:50.163 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.144+0000 7fda65409700 1 -- 172.21.15.42:0/2557007967 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda601209a0 con 0x7fda6007bf90 2019-12-05T05:03:50.164 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.144+0000 7fda57fff700 1 -- 172.21.15.42:0/2557007967 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fda500062b0 con 0x7fda6007bf90 2019-12-05T05:03:50.164 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.144+0000 7fda65409700 1 -- 172.21.15.42:0/2557007967 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda6012ae10 con 0x7fda6007bf90 2019-12-05T05:03:50.164 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.144+0000 7fda5ffff700 1 --2- 172.21.15.42:0/2557007967 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fda6007bf90 0x7fda601201e0 secure :-1 s=THROTTLE_DONE pgs=155 cs=0 l=1 rx=0x7fda600a3660 tx=0x7fda50010830).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:50.166 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.144+0000 7fda57fff700 1 -- 172.21.15.42:0/2557007967 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fda50006590 con 0x7fda6007bf90 2019-12-05T05:03:50.166 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.144+0000 7fda5ffff700 1 --2- 172.21.15.42:0/2557007967 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fda6007bf90 0x7fda601201e0 secure :-1 s=THROTTLE_DONE pgs=155 cs=0 l=1 rx=0x7fda600a3660 tx=0x7fda50010830).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:50.166 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.144+0000 7fda57fff700 1 --2- 172.21.15.42:0/2557007967 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fda4403ac30 0x7fda4403d140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:50.166 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.144+0000 7fda57fff700 1 -- 172.21.15.42:0/2557007967 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6031+0+0 (secure 0 0 0) 0x7fda5004af40 con 0x7fda6007bf90 2019-12-05T05:03:50.167 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.144+0000 7fda5f7fe700 1 --2- 172.21.15.42:0/2557007967 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fda4403ac30 0x7fda4403d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:50.167 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.148+0000 7fda5f7fe700 1 --2- 172.21.15.42:0/2557007967 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fda4403ac30 0x7fda4403d140 crc :-1 s=READY pgs=100 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:50.167 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.152+0000 7fda65409700 1 -- 172.21.15.42:0/2557007967 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda6005d110 con 0x7fda6007bf90 2019-12-05T05:03:50.168 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.160+0000 7fda5ffff700 1 --2- 172.21.15.42:0/2557007967 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fda6007bf90 0x7fda601201e0 secure :-1 s=THROTTLE_DONE pgs=155 cs=0 l=1 rx=0x7fda600a3660 tx=0x7fda50010830).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:50.168 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.160+0000 7fda57fff700 1 -- 172.21.15.42:0/2557007967 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fda50031fa0 con 0x7fda6007bf90 2019-12-05T05:03:50.337 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.332+0000 7fda65409700 1 -- 172.21.15.42:0/2557007967 --> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fda6005b8c0 con 0x7fda4403ac30 2019-12-05T05:03:50.344 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.340+0000 7fda57fff700 1 -- 172.21.15.42:0/2557007967 <== mgr.4108 v2:172.21.15.42:6800/11930 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+80437 (crc 0 0 0) 0x7fda6005b8c0 con 0x7fda4403ac30 2019-12-05T05:03:50.354 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:03:50.355 INFO:teuthology.orchestra.run.smithi042.stderr:dumped all 2019-12-05T05:03:50.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.348+0000 7fda65409700 1 -- 172.21.15.42:0/2557007967 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fda4403ac30 msgr2=0x7fda4403d140 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:50.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.348+0000 7fda65409700 1 --2- 172.21.15.42:0/2557007967 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fda4403ac30 0x7fda4403d140 crc :-1 s=READY pgs=100 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:50.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.348+0000 7fda65409700 1 -- 172.21.15.42:0/2557007967 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fda6007bf90 msgr2=0x7fda601201e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:50.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.348+0000 7fda65409700 1 --2- 172.21.15.42:0/2557007967 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fda6007bf90 0x7fda601201e0 secure :-1 s=READY pgs=155 cs=0 l=1 rx=0x7fda600a3660 tx=0x7fda50010830).stop 2019-12-05T05:03:50.358 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.348+0000 7fda65409700 1 -- 172.21.15.42:0/2557007967 shutdown_connections 2019-12-05T05:03:50.358 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.348+0000 7fda65409700 1 --2- 172.21.15.42:0/2557007967 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fda4403ac30 0x7fda4403d140 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:50.358 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.348+0000 7fda65409700 1 --2- 172.21.15.42:0/2557007967 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fda6007bf90 0x7fda601201e0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:50.358 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.348+0000 7fda65409700 1 --2- 172.21.15.42:0/2557007967 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fda600a2ec0 0x7fda6007ba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:50.359 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.348+0000 7fda65409700 1 -- 172.21.15.42:0/2557007967 shutdown_connections 2019-12-05T05:03:50.359 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.352+0000 7fda65409700 1 -- 172.21.15.42:0/2557007967 wait complete. 2019-12-05T05:03:50.359 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.352+0000 7fda65409700 1 -- 172.21.15.42:0/2557007967 >> 172.21.15.42:0/2557007967 conn(0x7fda600a7f90 msgr2=0x7fda60141600 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:50.367 INFO:teuthology.orchestra.run.smithi042.stdout:{"pg_ready":true,"pg_map":{"version":38,"stamp":"2019-12-05T05:03:49.600238+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2286,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":8390720,"kb_used_data":2048,"kb_used_omap":0,"kb_used_meta":8388608,"kb_avail":746584000,"statfs":{"total":773094113280,"available":764502016000,"internally_reserved":8589934592,"allocated":2097152,"data_stored":772252,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8589934592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001215"},"pg_stats":[{"pgid":"3.7","version":"0'0","reported_seq":"12","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T05:03:35.610591+0000","last_change":"2019-12-05T05:03:35.333820+0000","last_active":"2019-12-05T05:03:35.610591+0000","last_peered":"2019-12-05T05:03:35.610591+0000","last_clean":"2019-12-05T05:03:35.610591+0000","last_became_active":"2019-12-05T05:03:35.332717+0000","last_became_peered":"2019-12-05T05:03:35.332717+0000","last_unstale":"2019-12-05T05:03:35.610591+0000","last_undegraded":"2019-12-05T05:03:35.610591+0000","last_fullsized":"2019-12-05T05:03:35.610591+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:34.235090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:34.235090+0000","last_clean_scrub_stamp":"2019-12-05T05:03:34.235090+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7,0],"acting":[3,5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":"16","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T05:03:35.731480+0000","last_change":"2019-12-05T05:03:29.184885+0000","last_active":"2019-12-05T05:03:35.731480+0000","last_peered":"2019-12-05T05:03:35.731480+0000","last_clean":"2019-12-05T05:03:35.731480+0000","last_became_active":"2019-12-05T05:03:29.184431+0000","last_became_peered":"2019-12-05T05:03:29.184431+0000","last_unstale":"2019-12-05T05:03:35.731480+0000","last_undegraded":"2019-12-05T05:03:35.731480+0000","last_fullsized":"2019-12-05T05:03:35.731480+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:28.161818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:28.161818+0000","last_clean_scrub_stamp":"2019-12-05T05:03:28.161818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"24'4","reported_seq":"17","reported_epoch":"24","state":"active+clean","last_fresh":"2019-12-05T05:03:42.683004+0000","last_change":"2019-12-05T05:03:32.225175+0000","last_active":"2019-12-05T05:03:42.683004+0000","last_peered":"2019-12-05T05:03:42.683004+0000","last_clean":"2019-12-05T05:03:42.683004+0000","last_became_active":"2019-12-05T05:03:32.224827+0000","last_became_peered":"2019-12-05T05:03:32.224827+0000","last_unstale":"2019-12-05T05:03:42.683004+0000","last_undegraded":"2019-12-05T05:03:42.683004+0000","last_fullsized":"2019-12-05T05:03:42.683004+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:31.200243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:31.200243+0000","last_clean_scrub_stamp":"2019-12-05T05:03:31.200243+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":"12","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T05:03:36.529724+0000","last_change":"2019-12-05T05:03:35.333820+0000","last_active":"2019-12-05T05:03:36.529724+0000","last_peered":"2019-12-05T05:03:36.529724+0000","last_clean":"2019-12-05T05:03:36.529724+0000","last_became_active":"2019-12-05T05:03:35.332648+0000","last_became_peered":"2019-12-05T05:03:35.332648+0000","last_unstale":"2019-12-05T05:03:36.529724+0000","last_undegraded":"2019-12-05T05:03:36.529724+0000","last_fullsized":"2019-12-05T05:03:36.529724+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:34.235090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:34.235090+0000","last_clean_scrub_stamp":"2019-12-05T05:03:34.235090+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,7],"acting":[0,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":"16","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T05:03:35.610218+0000","last_change":"2019-12-05T05:03:29.182769+0000","last_active":"2019-12-05T05:03:35.610218+0000","last_peered":"2019-12-05T05:03:35.610218+0000","last_clean":"2019-12-05T05:03:35.610218+0000","last_became_active":"2019-12-05T05:03:29.182409+0000","last_became_peered":"2019-12-05T05:03:29.182409+0000","last_unstale":"2019-12-05T05:03:35.610218+0000","last_undegraded":"2019-12-05T05:03:35.610218+0000","last_fullsized":"2019-12-05T05:03:35.610218+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:28.161818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:28.161818+0000","last_clean_scrub_stamp":"2019-12-05T05:03:28.161818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"24'7","reported_seq":"19","reported_epoch":"24","state":"active+clean","last_fresh":"2019-12-05T05:03:42.685484+0000","last_change":"2019-12-05T05:03:32.228176+0000","last_active":"2019-12-05T05:03:42.685484+0000","last_peered":"2019-12-05T05:03:42.685484+0000","last_clean":"2019-12-05T05:03:42.685484+0000","last_became_active":"2019-12-05T05:03:32.227690+0000","last_became_peered":"2019-12-05T05:03:32.227690+0000","last_unstale":"2019-12-05T05:03:42.685484+0000","last_undegraded":"2019-12-05T05:03:42.685484+0000","last_fullsized":"2019-12-05T05:03:42.685484+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:31.200243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:31.200243+0000","last_clean_scrub_stamp":"2019-12-05T05:03:31.200243+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":566,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":"12","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T05:03:36.529831+0000","last_change":"2019-12-05T05:03:35.267212+0000","last_active":"2019-12-05T05:03:36.529831+0000","last_peered":"2019-12-05T05:03:36.529831+0000","last_clean":"2019-12-05T05:03:36.529831+0000","last_became_active":"2019-12-05T05:03:35.266235+0000","last_became_peered":"2019-12-05T05:03:35.266235+0000","last_unstale":"2019-12-05T05:03:36.529831+0000","last_undegraded":"2019-12-05T05:03:36.529831+0000","last_fullsized":"2019-12-05T05:03:36.529831+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:34.235090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:34.235090+0000","last_clean_scrub_stamp":"2019-12-05T05:03:34.235090+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3,5],"acting":[0,4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":"16","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T05:03:35.626677+0000","last_change":"2019-12-05T05:03:29.181115+0000","last_active":"2019-12-05T05:03:35.626677+0000","last_peered":"2019-12-05T05:03:35.626677+0000","last_clean":"2019-12-05T05:03:35.626677+0000","last_became_active":"2019-12-05T05:03:29.180832+0000","last_became_peered":"2019-12-05T05:03:29.180832+0000","last_unstale":"2019-12-05T05:03:35.626677+0000","last_undegraded":"2019-12-05T05:03:35.626677+0000","last_fullsized":"2019-12-05T05:03:35.626677+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:28.161818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:28.161818+0000","last_clean_scrub_stamp":"2019-12-05T05:03:28.161818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"24'8","reported_seq":"21","reported_epoch":"24","state":"active+clean","last_fresh":"2019-12-05T05:03:42.687799+0000","last_change":"2019-12-05T05:03:32.227259+0000","last_active":"2019-12-05T05:03:42.687799+0000","last_peered":"2019-12-05T05:03:42.687799+0000","last_clean":"2019-12-05T05:03:42.687799+0000","last_became_active":"2019-12-05T05:03:32.226895+0000","last_became_peered":"2019-12-05T05:03:32.226895+0000","last_unstale":"2019-12-05T05:03:42.687799+0000","last_undegraded":"2019-12-05T05:03:42.687799+0000","last_fullsized":"2019-12-05T05:03:42.687799+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:31.200243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:31.200243+0000","last_clean_scrub_stamp":"2019-12-05T05:03:31.200243+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":"12","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T05:03:35.628786+0000","last_change":"2019-12-05T05:03:35.628207+0000","last_active":"2019-12-05T05:03:35.628786+0000","last_peered":"2019-12-05T05:03:35.628786+0000","last_clean":"2019-12-05T05:03:35.628786+0000","last_became_active":"2019-12-05T05:03:35.627352+0000","last_became_peered":"2019-12-05T05:03:35.627352+0000","last_unstale":"2019-12-05T05:03:35.628786+0000","last_undegraded":"2019-12-05T05:03:35.628786+0000","last_fullsized":"2019-12-05T05:03:35.628786+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:34.235090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:34.235090+0000","last_clean_scrub_stamp":"2019-12-05T05:03:34.235090+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6,5],"acting":[1,2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":"16","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T05:03:36.847322+0000","last_change":"2019-12-05T05:03:29.408129+0000","last_active":"2019-12-05T05:03:36.847322+0000","last_peered":"2019-12-05T05:03:36.847322+0000","last_clean":"2019-12-05T05:03:36.847322+0000","last_became_active":"2019-12-05T05:03:29.407108+0000","last_became_peered":"2019-12-05T05:03:29.407108+0000","last_unstale":"2019-12-05T05:03:36.847322+0000","last_undegraded":"2019-12-05T05:03:36.847322+0000","last_fullsized":"2019-12-05T05:03:36.847322+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:28.161818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:28.161818+0000","last_clean_scrub_stamp":"2019-12-05T05:03:28.161818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"13","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T05:03:35.610338+0000","last_change":"2019-12-05T05:03:32.636592+0000","last_active":"2019-12-05T05:03:35.610338+0000","last_peered":"2019-12-05T05:03:35.610338+0000","last_clean":"2019-12-05T05:03:35.610338+0000","last_became_active":"2019-12-05T05:03:32.635937+0000","last_became_peered":"2019-12-05T05:03:32.635937+0000","last_unstale":"2019-12-05T05:03:35.610338+0000","last_undegraded":"2019-12-05T05:03:35.610338+0000","last_fullsized":"2019-12-05T05:03:35.610338+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:31.200243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:31.200243+0000","last_clean_scrub_stamp":"2019-12-05T05:03:31.200243+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":"12","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T05:03:36.315636+0000","last_change":"2019-12-05T05:03:36.315160+0000","last_active":"2019-12-05T05:03:36.315636+0000","last_peered":"2019-12-05T05:03:36.315636+0000","last_clean":"2019-12-05T05:03:36.315636+0000","last_became_active":"2019-12-05T05:03:36.314006+0000","last_became_peered":"2019-12-05T05:03:36.314006+0000","last_unstale":"2019-12-05T05:03:36.315636+0000","last_undegraded":"2019-12-05T05:03:36.315636+0000","last_fullsized":"2019-12-05T05:03:36.315636+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:34.235090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:34.235090+0000","last_clean_scrub_stamp":"2019-12-05T05:03:34.235090+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6,4],"acting":[3,5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"16","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T05:03:36.847644+0000","last_change":"2019-12-05T05:03:29.407882+0000","last_active":"2019-12-05T05:03:36.847644+0000","last_peered":"2019-12-05T05:03:36.847644+0000","last_clean":"2019-12-05T05:03:36.847644+0000","last_became_active":"2019-12-05T05:03:29.406770+0000","last_became_peered":"2019-12-05T05:03:29.406770+0000","last_unstale":"2019-12-05T05:03:36.847644+0000","last_undegraded":"2019-12-05T05:03:36.847644+0000","last_fullsized":"2019-12-05T05:03:36.847644+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:28.161818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:28.161818+0000","last_clean_scrub_stamp":"2019-12-05T05:03:28.161818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"24'5","reported_seq":"18","reported_epoch":"24","state":"active+clean","last_fresh":"2019-12-05T05:03:42.682907+0000","last_change":"2019-12-05T05:03:32.227654+0000","last_active":"2019-12-05T05:03:42.682907+0000","last_peered":"2019-12-05T05:03:42.682907+0000","last_clean":"2019-12-05T05:03:42.682907+0000","last_became_active":"2019-12-05T05:03:32.227052+0000","last_became_peered":"2019-12-05T05:03:32.227052+0000","last_unstale":"2019-12-05T05:03:42.682907+0000","last_undegraded":"2019-12-05T05:03:42.682907+0000","last_fullsized":"2019-12-05T05:03:42.682907+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:31.200243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:31.200243+0000","last_clean_scrub_stamp":"2019-12-05T05:03:31.200243+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":"12","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T05:03:35.734103+0000","last_change":"2019-12-05T05:03:35.733650+0000","last_active":"2019-12-05T05:03:35.734103+0000","last_peered":"2019-12-05T05:03:35.734103+0000","last_clean":"2019-12-05T05:03:35.734103+0000","last_became_active":"2019-12-05T05:03:35.732405+0000","last_became_peered":"2019-12-05T05:03:35.732405+0000","last_unstale":"2019-12-05T05:03:35.734103+0000","last_undegraded":"2019-12-05T05:03:35.734103+0000","last_fullsized":"2019-12-05T05:03:35.734103+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:34.235090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:34.235090+0000","last_clean_scrub_stamp":"2019-12-05T05:03:34.235090+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0,6],"acting":[4,5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":"16","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T05:03:35.629127+0000","last_change":"2019-12-05T05:03:29.407539+0000","last_active":"2019-12-05T05:03:35.629127+0000","last_peered":"2019-12-05T05:03:35.629127+0000","last_clean":"2019-12-05T05:03:35.629127+0000","last_became_active":"2019-12-05T05:03:29.406903+0000","last_became_peered":"2019-12-05T05:03:29.406903+0000","last_unstale":"2019-12-05T05:03:35.629127+0000","last_undegraded":"2019-12-05T05:03:35.629127+0000","last_fullsized":"2019-12-05T05:03:35.629127+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:28.161818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:28.161818+0000","last_clean_scrub_stamp":"2019-12-05T05:03:28.161818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"24'1","reported_seq":"14","reported_epoch":"24","state":"active+clean","last_fresh":"2019-12-05T05:03:37.365505+0000","last_change":"2019-12-05T05:03:32.227838+0000","last_active":"2019-12-05T05:03:37.365505+0000","last_peered":"2019-12-05T05:03:37.365505+0000","last_clean":"2019-12-05T05:03:37.365505+0000","last_became_active":"2019-12-05T05:03:32.227050+0000","last_became_peered":"2019-12-05T05:03:32.227050+0000","last_unstale":"2019-12-05T05:03:37.365505+0000","last_undegraded":"2019-12-05T05:03:37.365505+0000","last_fullsized":"2019-12-05T05:03:37.365505+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:31.200243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:31.200243+0000","last_clean_scrub_stamp":"2019-12-05T05:03:31.200243+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":"12","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T05:03:35.626774+0000","last_change":"2019-12-05T05:03:35.267419+0000","last_active":"2019-12-05T05:03:35.626774+0000","last_peered":"2019-12-05T05:03:35.626774+0000","last_clean":"2019-12-05T05:03:35.626774+0000","last_became_active":"2019-12-05T05:03:35.266868+0000","last_became_peered":"2019-12-05T05:03:35.266868+0000","last_unstale":"2019-12-05T05:03:35.626774+0000","last_undegraded":"2019-12-05T05:03:35.626774+0000","last_fullsized":"2019-12-05T05:03:35.626774+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:34.235090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:34.235090+0000","last_clean_scrub_stamp":"2019-12-05T05:03:34.235090+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0,5],"acting":[1,2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":"15","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T05:03:35.624631+0000","last_change":"2019-12-05T05:03:29.407541+0000","last_active":"2019-12-05T05:03:35.624631+0000","last_peered":"2019-12-05T05:03:35.624631+0000","last_clean":"2019-12-05T05:03:35.624631+0000","last_became_active":"2019-12-05T05:03:29.406857+0000","last_became_peered":"2019-12-05T05:03:29.406857+0000","last_unstale":"2019-12-05T05:03:35.624631+0000","last_undegraded":"2019-12-05T05:03:35.624631+0000","last_fullsized":"2019-12-05T05:03:35.624631+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:28.161818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:28.161818+0000","last_clean_scrub_stamp":"2019-12-05T05:03:28.161818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"24'6","reported_seq":"19","reported_epoch":"24","state":"active+clean","last_fresh":"2019-12-05T05:03:42.686730+0000","last_change":"2019-12-05T05:03:32.480464+0000","last_active":"2019-12-05T05:03:42.686730+0000","last_peered":"2019-12-05T05:03:42.686730+0000","last_clean":"2019-12-05T05:03:42.686730+0000","last_became_active":"2019-12-05T05:03:32.479565+0000","last_became_peered":"2019-12-05T05:03:32.479565+0000","last_unstale":"2019-12-05T05:03:42.686730+0000","last_undegraded":"2019-12-05T05:03:42.686730+0000","last_fullsized":"2019-12-05T05:03:42.686730+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:31.200243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:31.200243+0000","last_clean_scrub_stamp":"2019-12-05T05:03:31.200243+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":160,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":"12","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T05:03:35.630204+0000","last_change":"2019-12-05T05:03:35.629791+0000","last_active":"2019-12-05T05:03:35.630204+0000","last_peered":"2019-12-05T05:03:35.630204+0000","last_clean":"2019-12-05T05:03:35.630204+0000","last_became_active":"2019-12-05T05:03:35.629300+0000","last_became_peered":"2019-12-05T05:03:35.629300+0000","last_unstale":"2019-12-05T05:03:35.630204+0000","last_undegraded":"2019-12-05T05:03:35.630204+0000","last_fullsized":"2019-12-05T05:03:35.630204+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:34.235090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:34.235090+0000","last_clean_scrub_stamp":"2019-12-05T05:03:34.235090+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2,6],"acting":[5,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":"15","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T05:03:35.624610+0000","last_change":"2019-12-05T05:03:29.185245+0000","last_active":"2019-12-05T05:03:35.624610+0000","last_peered":"2019-12-05T05:03:35.624610+0000","last_clean":"2019-12-05T05:03:35.624610+0000","last_became_active":"2019-12-05T05:03:29.184716+0000","last_became_peered":"2019-12-05T05:03:29.184716+0000","last_unstale":"2019-12-05T05:03:35.624610+0000","last_undegraded":"2019-12-05T05:03:35.624610+0000","last_fullsized":"2019-12-05T05:03:35.624610+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:28.161818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:28.161818+0000","last_clean_scrub_stamp":"2019-12-05T05:03:28.161818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"24'8","reported_seq":"21","reported_epoch":"24","state":"active+clean","last_fresh":"2019-12-05T05:03:42.685962+0000","last_change":"2019-12-05T05:03:32.476846+0000","last_active":"2019-12-05T05:03:42.685962+0000","last_peered":"2019-12-05T05:03:42.685962+0000","last_clean":"2019-12-05T05:03:42.685962+0000","last_became_active":"2019-12-05T05:03:32.476140+0000","last_became_peered":"2019-12-05T05:03:32.476140+0000","last_unstale":"2019-12-05T05:03:42.685962+0000","last_undegraded":"2019-12-05T05:03:42.685962+0000","last_fullsized":"2019-12-05T05:03:42.685962+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:31.200243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:31.200243+0000","last_clean_scrub_stamp":"2019-12-05T05:03:31.200243+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1526,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":32,"acting":32,"num_store_stats":8},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2286,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574856,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048848,"kb_used_data":264,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322992,"statfs":{"total":96636764160,"available":95562743808,"internally_reserved":1073741824,"allocated":270336,"data_stored":96686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.801}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.102}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.038}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.982}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.732}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.176}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.348}]}]},{"osd":6,"up_from":13,"seq":55834574857,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048840,"kb_used_data":256,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323000,"statfs":{"total":96636764160,"available":95562752000,"internally_reserved":1073741824,"allocated":262144,"data_stored":96526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.790}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.931}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.325}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.134}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.086}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.210}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.383}]}]},{"osd":1,"up_from":13,"seq":55834574855,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048844,"kb_used_data":260,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322996,"statfs":{"total":96636764160,"available":95562747904,"internally_reserved":1073741824,"allocated":266240,"data_stored":97486,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.506}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.781}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.696}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.934}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.014}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.900}]}]},{"osd":0,"up_from":13,"seq":55834574857,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048852,"kb_used_data":268,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322988,"statfs":{"total":96636764160,"available":95562739712,"internally_reserved":1073741824,"allocated":274432,"data_stored":97646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.639}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.961}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.879}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.150}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.118}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.306}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.220}]}]},{"osd":2,"up_from":13,"seq":55834574855,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048836,"kb_used_data":252,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323004,"statfs":{"total":96636764160,"available":95562756096,"internally_reserved":1073741824,"allocated":258048,"data_stored":95994,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.925}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.738}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.648}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.430}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.231}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.200}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.159}]}]},{"osd":3,"up_from":13,"seq":55834574856,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048832,"kb_used_data":248,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323008,"statfs":{"total":96636764160,"available":95562760192,"internally_reserved":1073741824,"allocated":253952,"data_stored":95960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.858}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.668}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.712}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.812}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.026}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.092}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.733}]}]},{"osd":4,"up_from":13,"seq":55834574855,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048832,"kb_used_data":248,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323008,"statfs":{"total":96636764160,"available":95562760192,"internally_reserved":1073741824,"allocated":253952,"data_stored":95960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.828}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.744}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.916}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.036}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.356}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.295}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.398}]}]},{"osd":5,"up_from":13,"seq":55834574856,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048836,"kb_used_data":252,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323004,"statfs":{"total":96636764160,"available":95562756096,"internally_reserved":1073741824,"allocated":258048,"data_stored":95994,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.787}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.990}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.647}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.889}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.097}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.022}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.137}]}]}]}} 2019-12-05T05:03:50.373 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2019-12-05T05:03:50.486 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.476+0000 7efed66d4700 1 Processor -- start 2019-12-05T05:03:50.487 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.476+0000 7efed66d4700 1 -- start start 2019-12-05T05:03:50.487 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.476+0000 7efed66d4700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7efed008df70 0x7efed009f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:50.487 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.476+0000 7efed66d4700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efed009fc50 con 0x7efed009f820 2019-12-05T05:03:50.487 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.476+0000 7efed66d4700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efed00a0700 con 0x7efed00a7730 2019-12-05T05:03:50.487 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.476+0000 7efed66d4700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efed00a0af0 con 0x7efed008df70 2019-12-05T05:03:50.488 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.476+0000 7efed4c71700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7efed00a7730 0x7efed007e710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:53884/0 (socket says 172.21.15.42:53884) 2019-12-05T05:03:50.488 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.476+0000 7efed4c71700 1 -- 172.21.15.42:0/2854412428 learned_addr learned my addr 172.21.15.42:0/2854412428 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:03:50.488 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.476+0000 7efecffff700 1 --2- 172.21.15.42:0/2854412428 >> v2:172.21.15.40:3300/0 conn(0x7efed008df70 0x7efed009f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:50.488 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.476+0000 7efecf7fe700 1 -- 172.21.15.42:0/2854412428 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3189595127 0 0) 0x7efed00a0700 con 0x7efed00a7730 2019-12-05T05:03:50.488 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.476+0000 7efecf7fe700 1 -- 172.21.15.42:0/2854412428 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efec00030b0 con 0x7efed00a7730 2019-12-05T05:03:50.488 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.480+0000 7efecffff700 1 -- 172.21.15.42:0/2854412428 >> v1:172.21.15.42:6789/0 conn(0x7efed00a7730 legacy=0x7efed007e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:50.488 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.480+0000 7efecffff700 1 -- 172.21.15.42:0/2854412428 >> v1:172.21.15.40:6789/0 conn(0x7efed009f820 legacy=0x7efed014efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:50.489 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.480+0000 7efecffff700 1 -- 172.21.15.42:0/2854412428 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efed007ef10 con 0x7efed008df70 2019-12-05T05:03:50.489 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.480+0000 7efecffff700 1 --2- 172.21.15.42:0/2854412428 >> v2:172.21.15.40:3300/0 conn(0x7efed008df70 0x7efed009f2d0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7efebc008790 tx=0x7efebc00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:50.489 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.480+0000 7efecffff700 1 --2- 172.21.15.42:0/2854412428 >> v2:172.21.15.40:3300/0 conn(0x7efed008df70 0x7efed009f2d0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7efebc008790 tx=0x7efebc00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:50.489 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.480+0000 7efecffff700 1 --2- 172.21.15.42:0/2854412428 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efed008df70 0x7efed009f2d0 secure :-1 s=READY pgs=157 cs=0 l=1 rx=0x7efebc008790 tx=0x7efebc00fbf0).ready entity=mon.0 client_cookie=2c23e4cea0767fc4 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:50.489 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.480+0000 7efecffff700 1 --2- 172.21.15.42:0/2854412428 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efed008df70 0x7efed009f2d0 secure :-1 s=THROTTLE_DONE pgs=157 cs=0 l=1 rx=0x7efebc008790 tx=0x7efebc00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:50.489 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.480+0000 7efecffff700 1 --2- 172.21.15.42:0/2854412428 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efed008df70 0x7efed009f2d0 secure :-1 s=THROTTLE_DONE pgs=157 cs=0 l=1 rx=0x7efebc008790 tx=0x7efebc00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:50.490 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.480+0000 7efecf7fe700 1 -- 172.21.15.42:0/2854412428 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efebc011270 con 0x7efed008df70 2019-12-05T05:03:50.490 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.480+0000 7efecffff700 1 --2- 172.21.15.42:0/2854412428 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efed008df70 0x7efed009f2d0 secure :-1 s=THROTTLE_DONE pgs=157 cs=0 l=1 rx=0x7efebc008790 tx=0x7efebc00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:50.490 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.480+0000 7efecf7fe700 1 -- 172.21.15.42:0/2854412428 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efebc008bc0 con 0x7efed008df70 2019-12-05T05:03:50.490 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.480+0000 7efed66d4700 1 -- 172.21.15.42:0/2854412428 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efed008df70 msgr2=0x7efed009f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:50.490 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.480+0000 7efecf7fe700 1 -- 172.21.15.42:0/2854412428 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efebc011850 con 0x7efed008df70 2019-12-05T05:03:50.490 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.480+0000 7efed66d4700 1 --2- 172.21.15.42:0/2854412428 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efed008df70 0x7efed009f2d0 secure :-1 s=READY pgs=157 cs=0 l=1 rx=0x7efebc008790 tx=0x7efebc00fbf0).stop 2019-12-05T05:03:50.491 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.480+0000 7efed66d4700 1 -- 172.21.15.42:0/2854412428 shutdown_connections 2019-12-05T05:03:50.491 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.480+0000 7efed66d4700 1 --2- 172.21.15.42:0/2854412428 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efed008df70 0x7efed009f2d0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:50.491 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.480+0000 7efed66d4700 1 -- 172.21.15.42:0/2854412428 shutdown_connections 2019-12-05T05:03:50.491 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.480+0000 7efed66d4700 1 -- 172.21.15.42:0/2854412428 wait complete. 2019-12-05T05:03:50.491 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.480+0000 7efed66d4700 1 -- 172.21.15.42:0/2854412428 >> 172.21.15.42:0/2854412428 conn(0x7efed00aabd0 msgr2=0x7efed00a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:50.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.484+0000 7efed66d4700 1 Processor -- start 2019-12-05T05:03:50.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.484+0000 7efed66d4700 1 -- start start 2019-12-05T05:03:50.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.484+0000 7efed66d4700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efed0142a30 0x7efed0126720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:50.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.484+0000 7efed66d4700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efed01433e0 con 0x7efed008f770 2019-12-05T05:03:50.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.484+0000 7efed66d4700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efed007e110 con 0x7efed01142f0 2019-12-05T05:03:50.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.484+0000 7efed66d4700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efed01436f0 con 0x7efed0142a30 2019-12-05T05:03:50.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.484+0000 7efed5472700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7efed01142f0 0x7efed012db70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:53894/0 (socket says 172.21.15.42:53894) 2019-12-05T05:03:50.496 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.484+0000 7efed5472700 1 -- 172.21.15.42:0/3823366711 learned_addr learned my addr 172.21.15.42:0/3823366711 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:03:50.496 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.484+0000 7efecffff700 1 --2- 172.21.15.42:0/3823366711 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efed0142a30 0x7efed0126720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:50.496 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.484+0000 7efecd7fa700 1 -- 172.21.15.42:0/3823366711 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1804560111 0 0) 0x7efed007e110 con 0x7efed01142f0 2019-12-05T05:03:50.496 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.484+0000 7efecd7fa700 1 -- 172.21.15.42:0/3823366711 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efeb00030b0 con 0x7efed01142f0 2019-12-05T05:03:50.496 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.484+0000 7efecd7fa700 1 -- 172.21.15.42:0/3823366711 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1117545049 0 0) 0x7efed01433e0 con 0x7efed008f770 2019-12-05T05:03:50.496 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.484+0000 7efecd7fa700 1 -- 172.21.15.42:0/3823366711 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efed007e110 con 0x7efed008f770 2019-12-05T05:03:50.497 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.484+0000 7efecffff700 1 -- 172.21.15.42:0/3823366711 >> v1:172.21.15.42:6789/0 conn(0x7efed01142f0 legacy=0x7efed012db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:50.497 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.484+0000 7efecffff700 1 -- 172.21.15.42:0/3823366711 >> v1:172.21.15.40:6789/0 conn(0x7efed008f770 legacy=0x7efed007d910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:50.497 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.484+0000 7efecffff700 1 -- 172.21.15.42:0/3823366711 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efed0126c70 con 0x7efed0142a30 2019-12-05T05:03:50.497 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.484+0000 7efecffff700 1 --2- 172.21.15.42:0/3823366711 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efed0142a30 0x7efed0126720 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7efebc008790 tx=0x7efebc008df0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:50.497 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.484+0000 7efecffff700 1 --2- 172.21.15.42:0/3823366711 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efed0142a30 0x7efed0126720 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7efebc008790 tx=0x7efebc008df0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:50.497 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.484+0000 7efecffff700 1 --2- 172.21.15.42:0/3823366711 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efed0142a30 0x7efed0126720 secure :-1 s=READY pgs=46 cs=0 l=1 rx=0x7efebc008790 tx=0x7efebc008df0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:50.498 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.484+0000 7efecffff700 1 --2- 172.21.15.42:0/3823366711 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efed0142a30 0x7efed0126720 secure :-1 s=THROTTLE_DONE pgs=46 cs=0 l=1 rx=0x7efebc008790 tx=0x7efebc008df0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:50.498 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.484+0000 7efecffff700 1 --2- 172.21.15.42:0/3823366711 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efed0142a30 0x7efed0126720 secure :-1 s=THROTTLE_DONE pgs=46 cs=0 l=1 rx=0x7efebc008790 tx=0x7efebc008df0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:50.498 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.484+0000 7efecd7fa700 1 -- 172.21.15.42:0/3823366711 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efebc007150 con 0x7efed0142a30 2019-12-05T05:03:50.499 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.484+0000 7efecffff700 1 --2- 172.21.15.42:0/3823366711 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efed0142a30 0x7efed0126720 secure :-1 s=THROTTLE_DONE pgs=46 cs=0 l=1 rx=0x7efebc008790 tx=0x7efebc008df0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:50.499 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.484+0000 7efecd7fa700 1 -- 172.21.15.42:0/3823366711 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efebc011470 con 0x7efed0142a30 2019-12-05T05:03:50.499 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.484+0000 7efed66d4700 1 -- 172.21.15.42:0/3823366711 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efed0126f40 con 0x7efed0142a30 2019-12-05T05:03:50.499 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.484+0000 7efecd7fa700 1 -- 172.21.15.42:0/3823366711 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efebc00f550 con 0x7efed0142a30 2019-12-05T05:03:50.499 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.484+0000 7efed66d4700 1 -- 172.21.15.42:0/3823366711 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efed012f380 con 0x7efed0142a30 2019-12-05T05:03:50.499 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.488+0000 7efecffff700 1 --2- 172.21.15.42:0/3823366711 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efed0142a30 0x7efed0126720 secure :-1 s=THROTTLE_DONE pgs=46 cs=0 l=1 rx=0x7efebc008790 tx=0x7efebc008df0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:50.500 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.488+0000 7efecffff700 1 --2- 172.21.15.42:0/3823366711 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efed0142a30 0x7efed0126720 secure :-1 s=THROTTLE_DONE pgs=46 cs=0 l=1 rx=0x7efebc008790 tx=0x7efebc008df0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:50.500 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.488+0000 7efecd7fa700 1 -- 172.21.15.42:0/3823366711 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7efebc005540 con 0x7efed0142a30 2019-12-05T05:03:50.500 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.488+0000 7efecd7fa700 1 --2- 172.21.15.42:0/3823366711 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7efeb003b430 0x7efeb003d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:50.500 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.488+0000 7efecd7fa700 1 -- 172.21.15.42:0/3823366711 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6031+0+0 (secure 0 0 0) 0x7efebc04ba00 con 0x7efed0142a30 2019-12-05T05:03:50.500 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.488+0000 7efed4c71700 1 --2- 172.21.15.42:0/3823366711 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7efeb003b430 0x7efeb003d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:50.500 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.488+0000 7efed4c71700 1 --2- 172.21.15.42:0/3823366711 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7efeb003b430 0x7efeb003d940 crc :-1 s=READY pgs=101 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:50.501 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.492+0000 7efed66d4700 1 -- 172.21.15.42:0/3823366711 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efed005d100 con 0x7efed0142a30 2019-12-05T05:03:50.506 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.500+0000 7efecffff700 1 --2- 172.21.15.42:0/3823366711 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efed0142a30 0x7efed0126720 secure :-1 s=THROTTLE_DONE pgs=46 cs=0 l=1 rx=0x7efebc008790 tx=0x7efebc008df0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:50.506 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.500+0000 7efecd7fa700 1 -- 172.21.15.42:0/3823366711 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7efebc032a40 con 0x7efed0142a30 2019-12-05T05:03:50.677 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.672+0000 7efed66d4700 1 -- 172.21.15.42:0/3823366711 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efed005b8b0 con 0x7efed0142a30 2019-12-05T05:03:50.679 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.672+0000 7efecffff700 1 --2- 172.21.15.42:0/3823366711 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efed0142a30 0x7efed0126720 secure :-1 s=THROTTLE_DONE pgs=46 cs=0 l=1 rx=0x7efebc008790 tx=0x7efebc008df0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:50.679 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.672+0000 7efecd7fa700 1 -- 172.21.15.42:0/3823366711 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+14885 (secure 0 0 0) 0x7efebc00ec80 con 0x7efed0142a30 2019-12-05T05:03:50.680 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:03:50.680 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":24,"fsid":"5d836cfc-a83b-4433-9b3e-b9b2f31c9050","created":"2019-12-05T05:03:13.215602+0000","modified":"2019-12-05T05:03:36.607704+0000","last_up_change":"2019-12-05T05:03:22.664783+0000","last_in_change":"2019-12-05T05:03:22.664783+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2019-12-05T05:03:27.394605+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2019-12-05T05:03:30.419845+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2019-12-05T05:03:33.249075+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e3fc49bb-dcbc-43ff-ac7c-9eb0ea0bd257","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6810","nonce":12118},{"type":"v1","addr":"172.21.15.42:6811","nonce":12118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6812","nonce":12118},{"type":"v1","addr":"172.21.15.42:6813","nonce":12118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6816","nonce":12118},{"type":"v1","addr":"172.21.15.42:6817","nonce":12118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6814","nonce":12118},{"type":"v1","addr":"172.21.15.42:6815","nonce":12118}]},"public_addr":"172.21.15.42:6811/12118","cluster_addr":"172.21.15.42:6813/12118","heartbeat_back_addr":"172.21.15.42:6817/12118","heartbeat_front_addr":"172.21.15.42:6815/12118","state":["exists","up"]},{"osd":1,"uuid":"902543fb-f5a2-4321-9c9c-26f9b9ca8629","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6826","nonce":12119},{"type":"v1","addr":"172.21.15.42:6827","nonce":12119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6828","nonce":12119},{"type":"v1","addr":"172.21.15.42:6829","nonce":12119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6832","nonce":12119},{"type":"v1","addr":"172.21.15.42:6833","nonce":12119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6830","nonce":12119},{"type":"v1","addr":"172.21.15.42:6831","nonce":12119}]},"public_addr":"172.21.15.42:6827/12119","cluster_addr":"172.21.15.42:6829/12119","heartbeat_back_addr":"172.21.15.42:6833/12119","heartbeat_front_addr":"172.21.15.42:6831/12119","state":["exists","up"]},{"osd":2,"uuid":"28739ff7-bed7-4b75-8ea5-b57d088f1250","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6802","nonce":12117},{"type":"v1","addr":"172.21.15.42:6803","nonce":12117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6804","nonce":12117},{"type":"v1","addr":"172.21.15.42:6805","nonce":12117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6808","nonce":12117},{"type":"v1","addr":"172.21.15.42:6809","nonce":12117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6806","nonce":12117},{"type":"v1","addr":"172.21.15.42:6807","nonce":12117}]},"public_addr":"172.21.15.42:6803/12117","cluster_addr":"172.21.15.42:6805/12117","heartbeat_back_addr":"172.21.15.42:6809/12117","heartbeat_front_addr":"172.21.15.42:6807/12117","state":["exists","up"]},{"osd":3,"uuid":"1960f66b-2d40-4bb9-a289-ce8cd0c1c142","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6818","nonce":12116},{"type":"v1","addr":"172.21.15.42:6819","nonce":12116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6820","nonce":12116},{"type":"v1","addr":"172.21.15.42:6821","nonce":12116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6824","nonce":12116},{"type":"v1","addr":"172.21.15.42:6825","nonce":12116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6822","nonce":12116},{"type":"v1","addr":"172.21.15.42:6823","nonce":12116}]},"public_addr":"172.21.15.42:6819/12116","cluster_addr":"172.21.15.42:6821/12116","heartbeat_back_addr":"172.21.15.42:6825/12116","heartbeat_front_addr":"172.21.15.42:6823/12116","state":["exists","up"]},{"osd":4,"uuid":"45bbf1bd-7c8a-4a0d-b76b-afa6148a855c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":12488},{"type":"v1","addr":"172.21.15.40:6809","nonce":12488}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":12488},{"type":"v1","addr":"172.21.15.40:6812","nonce":12488}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":12488},{"type":"v1","addr":"172.21.15.40:6825","nonce":12488}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":12488},{"type":"v1","addr":"172.21.15.40:6819","nonce":12488}]},"public_addr":"172.21.15.40:6809/12488","cluster_addr":"172.21.15.40:6812/12488","heartbeat_back_addr":"172.21.15.40:6825/12488","heartbeat_front_addr":"172.21.15.40:6819/12488","state":["exists","up"]},{"osd":5,"uuid":"8a5511cd-925f-4869-a49e-924383e4ce53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":12489},{"type":"v1","addr":"172.21.15.40:6801","nonce":12489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":12489},{"type":"v1","addr":"172.21.15.40:6803","nonce":12489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":12489},{"type":"v1","addr":"172.21.15.40:6807","nonce":12489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":12489},{"type":"v1","addr":"172.21.15.40:6805","nonce":12489}]},"public_addr":"172.21.15.40:6801/12489","cluster_addr":"172.21.15.40:6803/12489","heartbeat_back_addr":"172.21.15.40:6807/12489","heartbeat_front_addr":"172.21.15.40:6805/12489","state":["exists","up"]},{"osd":6,"uuid":"d692c179-379e-446c-8484-a219a9241e04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6811","nonce":12491},{"type":"v1","addr":"172.21.15.40:6814","nonce":12491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6817","nonce":12491},{"type":"v1","addr":"172.21.15.40:6818","nonce":12491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":12491},{"type":"v1","addr":"172.21.15.40:6828","nonce":12491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":12491},{"type":"v1","addr":"172.21.15.40:6824","nonce":12491}]},"public_addr":"172.21.15.40:6814/12491","cluster_addr":"172.21.15.40:6818/12491","heartbeat_back_addr":"172.21.15.40:6828/12491","heartbeat_front_addr":"172.21.15.40:6824/12491","state":["exists","up"]},{"osd":7,"uuid":"79c3c243-190e-4e35-b769-319a2077e7cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6813","nonce":12490},{"type":"v1","addr":"172.21.15.40:6816","nonce":12490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":12490},{"type":"v1","addr":"172.21.15.40:6823","nonce":12490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":12490},{"type":"v1","addr":"172.21.15.40:6831","nonce":12490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6827","nonce":12490},{"type":"v1","addr":"172.21.15.40:6829","nonce":12490}]},"public_addr":"172.21.15.40:6816/12490","cluster_addr":"172.21.15.40:6823/12490","heartbeat_back_addr":"172.21.15.40:6831/12490","heartbeat_front_addr":"172.21.15.40:6829/12490","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T05:03:21.349150+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T05:03:21.644281+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T05:03:21.394040+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T05:03:21.587096+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T05:03:21.656497+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T05:03:21.510547+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T05:03:21.659358+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T05:03:21.628063+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2019-12-05T05:03:50.682 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.676+0000 7efed66d4700 1 -- 172.21.15.42:0/3823366711 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7efeb003b430 msgr2=0x7efeb003d940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:50.683 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.676+0000 7efed66d4700 1 --2- 172.21.15.42:0/3823366711 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7efeb003b430 0x7efeb003d940 crc :-1 s=READY pgs=101 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:50.683 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.676+0000 7efed66d4700 1 -- 172.21.15.42:0/3823366711 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efed0142a30 msgr2=0x7efed0126720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:50.683 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.676+0000 7efed66d4700 1 --2- 172.21.15.42:0/3823366711 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efed0142a30 0x7efed0126720 secure :-1 s=READY pgs=46 cs=0 l=1 rx=0x7efebc008790 tx=0x7efebc008df0).stop 2019-12-05T05:03:50.683 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.676+0000 7efed66d4700 1 -- 172.21.15.42:0/3823366711 shutdown_connections 2019-12-05T05:03:50.683 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.676+0000 7efed66d4700 1 --2- 172.21.15.42:0/3823366711 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7efeb003b430 0x7efeb003d940 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:50.683 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.676+0000 7efed66d4700 1 --2- 172.21.15.42:0/3823366711 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efed0142a30 0x7efed0126720 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:50.683 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.676+0000 7efed66d4700 1 -- 172.21.15.42:0/3823366711 shutdown_connections 2019-12-05T05:03:50.684 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.676+0000 7efed66d4700 1 -- 172.21.15.42:0/3823366711 wait complete. 2019-12-05T05:03:50.684 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.676+0000 7efed66d4700 1 -- 172.21.15.42:0/3823366711 >> 172.21.15.42:0/3823366711 conn(0x7efed00a8290 msgr2=0x7efed00a6ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:50.693 INFO:tasks.cephfs.filesystem.ceph_manager:[{u'cache_target_full_ratio_micro': 800000, u'fast_read': False, u'stripe_width': 0, u'flags_names': u'hashpspool', u'tier_of': -1, u'hit_set_grade_decay_rate': 0, u'pg_placement_num': 8, u'use_gmt_hitset': True, u'cache_target_dirty_ratio_micro': 400000, u'create_time': u'2019-12-05T05:03:27.394605+0000', u'snap_epoch': 0, u'erasure_code_profile': u'', u'pg_autoscale_mode': u'off', u'snap_seq': 0, u'expected_num_objects': 0, u'size': 2, u'pg_num_pending': 8, u'auid': 0, u'cache_min_flush_age': 0, u'hit_set_period': 0, u'min_read_recency_for_promote': 0, u'target_max_objects': 0, u'pg_placement_num_target': 8, u'pg_num': 8, u'type': 1, u'grade_table': [], u'pool_name': u'rbd', u'cache_min_evict_age': 0, u'snap_mode': u'selfmanaged', u'pg_num_target': 8, u'cache_mode': u'none', u'min_size': 1, u'cache_target_dirty_high_ratio_micro': 600000, u'object_hash': 2, u'last_pg_merge_meta': {u'ready_epoch': 0, u'source_version': u"0'0", u'source_pgid': u'0.0', u'last_epoch_clean': 0, u'target_version': u"0'0", u'last_epoch_started': 0}, u'write_tier': -1, u'last_force_op_resend_preluminous': u'0', u'pool': 1, u'removed_snaps': u'[]', u'crush_rule': 0, u'tiers': [], u'hit_set_params': {u'type': u'none'}, u'last_force_op_resend': u'0', u'pool_snaps': [], u'quota_max_objects': 0, u'last_force_op_resend_prenautilus': u'0', u'application_metadata': {u'rbd': {}}, u'options': {}, u'hit_set_count': 0, u'flags': 1, u'target_max_bytes': 0, u'quota_max_bytes': 0, u'hit_set_search_last_n': 0, u'last_change': u'17', u'min_write_recency_for_promote': 0, u'read_tier': -1}, {u'cache_target_full_ratio_micro': 800000, u'fast_read': False, u'stripe_width': 0, u'flags_names': u'hashpspool', u'tier_of': -1, u'hit_set_grade_decay_rate': 0, u'pg_placement_num': 8, u'use_gmt_hitset': True, u'cache_target_dirty_ratio_micro': 400000, u'create_time': u'2019-12-05T05:03:30.419845+0000', u'snap_epoch': 0, u'erasure_code_profile': u'', u'pg_autoscale_mode': u'off', u'snap_seq': 0, u'expected_num_objects': 0, u'size': 2, u'pg_num_pending': 8, u'auid': 0, u'cache_min_flush_age': 0, u'hit_set_period': 0, u'min_read_recency_for_promote': 0, u'target_max_objects': 0, u'pg_placement_num_target': 8, u'pg_num': 8, u'type': 1, u'grade_table': [], u'pool_name': u'cephfs_metadata', u'cache_min_evict_age': 0, u'snap_mode': u'selfmanaged', u'pg_num_target': 8, u'cache_mode': u'none', u'min_size': 1, u'cache_target_dirty_high_ratio_micro': 600000, u'object_hash': 2, u'last_pg_merge_meta': {u'ready_epoch': 0, u'source_version': u"0'0", u'source_pgid': u'0.0', u'last_epoch_clean': 0, u'target_version': u"0'0", u'last_epoch_started': 0}, u'write_tier': -1, u'last_force_op_resend_preluminous': u'0', u'pool': 2, u'removed_snaps': u'[]', u'crush_rule': 0, u'tiers': [], u'hit_set_params': {u'type': u'none'}, u'last_force_op_resend': u'0', u'pool_snaps': [], u'quota_max_objects': 0, u'last_force_op_resend_prenautilus': u'0', u'application_metadata': {u'cephfs': {u'metadata': u'cephfs'}}, u'options': {u'pg_autoscale_bias': 4, u'recovery_priority': 5, u'pg_num_min': 16}, u'hit_set_count': 0, u'flags': 1, u'target_max_bytes': 0, u'quota_max_bytes': 0, u'hit_set_search_last_n': 0, u'last_change': u'23', u'min_write_recency_for_promote': 0, u'read_tier': -1}, {u'cache_target_full_ratio_micro': 800000, u'fast_read': False, u'stripe_width': 8192, u'flags_names': u'hashpspool,ec_overwrites', u'tier_of': -1, u'hit_set_grade_decay_rate': 0, u'pg_placement_num': 8, u'use_gmt_hitset': True, u'cache_target_dirty_ratio_micro': 400000, u'create_time': u'2019-12-05T05:03:33.249075+0000', u'snap_epoch': 0, u'erasure_code_profile': u'cephfs_data', u'pg_autoscale_mode': u'off', u'snap_seq': 0, u'expected_num_objects': 0, u'size': 4, u'pg_num_pending': 8, u'auid': 0, u'cache_min_flush_age': 0, u'hit_set_period': 0, u'min_read_recency_for_promote': 0, u'target_max_objects': 0, u'pg_placement_num_target': 8, u'pg_num': 8, u'type': 3, u'grade_table': [], u'pool_name': u'cephfs_data', u'cache_min_evict_age': 0, u'snap_mode': u'selfmanaged', u'pg_num_target': 8, u'cache_mode': u'none', u'min_size': 3, u'cache_target_dirty_high_ratio_micro': 600000, u'object_hash': 2, u'last_pg_merge_meta': {u'ready_epoch': 0, u'source_version': u"0'0", u'source_pgid': u'0.0', u'last_epoch_clean': 0, u'target_version': u"0'0", u'last_epoch_started': 0}, u'write_tier': -1, u'last_force_op_resend_preluminous': u'0', u'pool': 3, u'removed_snaps': u'[]', u'crush_rule': 1, u'tiers': [], u'hit_set_params': {u'type': u'none'}, u'last_force_op_resend': u'0', u'pool_snaps': [], u'quota_max_objects': 0, u'last_force_op_resend_prenautilus': u'0', u'application_metadata': {u'cephfs': {u'data': u'cephfs'}}, u'options': {}, u'hit_set_count': 0, u'flags': 5, u'target_max_bytes': 0, u'quota_max_bytes': 0, u'hit_set_search_last_n': 0, u'last_change': u'24', u'min_write_recency_for_promote': 0, u'read_tier': -1}] 2019-12-05T05:03:50.694 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2019-12-05T05:03:50.795 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.784+0000 7fb106f45700 1 Processor -- start 2019-12-05T05:03:50.795 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.784+0000 7fb106f45700 1 -- start start 2019-12-05T05:03:50.795 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.784+0000 7fb106f45700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb10014df60 0x7fb1001503a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:50.796 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.784+0000 7fb106f45700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb10007f080 con 0x7fb1000a9a00 2019-12-05T05:03:50.796 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.784+0000 7fb106f45700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1000a1000 con 0x7fb10009f550 2019-12-05T05:03:50.796 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.784+0000 7fb106f45700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1000a8f80 con 0x7fb10014df60 2019-12-05T05:03:50.796 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.784+0000 7fb1054e2700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fb10009f550 0x7fb100144630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:53898/0 (socket says 172.21.15.42:53898) 2019-12-05T05:03:50.796 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.784+0000 7fb1054e2700 1 -- 172.21.15.42:0/3367907352 learned_addr learned my addr 172.21.15.42:0/3367907352 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:03:50.796 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.784+0000 7fb105ce3700 1 --2- 172.21.15.42:0/3367907352 >> v2:172.21.15.40:3300/0 conn(0x7fb10014df60 0x7fb1001503a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:50.797 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.784+0000 7fb0f7fff700 1 -- 172.21.15.42:0/3367907352 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2638725908 0 0) 0x7fb1000a1000 con 0x7fb10009f550 2019-12-05T05:03:50.797 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.784+0000 7fb0f7fff700 1 -- 172.21.15.42:0/3367907352 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0ec0030b0 con 0x7fb10009f550 2019-12-05T05:03:50.797 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.788+0000 7fb105ce3700 1 -- 172.21.15.42:0/3367907352 >> v1:172.21.15.42:6789/0 conn(0x7fb10009f550 legacy=0x7fb100144630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:50.797 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.788+0000 7fb0f7fff700 1 -- 172.21.15.42:0/3367907352 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1977319481 0 0) 0x7fb0ec0030b0 con 0x7fb10009f550 2019-12-05T05:03:50.797 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.788+0000 7fb105ce3700 1 -- 172.21.15.42:0/3367907352 >> v1:172.21.15.40:6789/0 conn(0x7fb1000a9a00 legacy=0x7fb10007e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:50.797 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.788+0000 7fb105ce3700 1 -- 172.21.15.42:0/3367907352 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb10009ee30 con 0x7fb10014df60 2019-12-05T05:03:50.797 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.788+0000 7fb105ce3700 1 --2- 172.21.15.42:0/3367907352 >> v2:172.21.15.40:3300/0 conn(0x7fb10014df60 0x7fb1001503a0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fb0f0001110 tx=0x7fb0f0001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:50.798 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.788+0000 7fb105ce3700 1 --2- 172.21.15.42:0/3367907352 >> v2:172.21.15.40:3300/0 conn(0x7fb10014df60 0x7fb1001503a0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fb0f0001110 tx=0x7fb0f0001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:50.798 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.788+0000 7fb105ce3700 1 --2- 172.21.15.42:0/3367907352 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb10014df60 0x7fb1001503a0 secure :-1 s=READY pgs=160 cs=0 l=1 rx=0x7fb0f0001110 tx=0x7fb0f0001150).ready entity=mon.0 client_cookie=82fa2967df82404f server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:50.798 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.788+0000 7fb105ce3700 1 --2- 172.21.15.42:0/3367907352 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb10014df60 0x7fb1001503a0 secure :-1 s=THROTTLE_DONE pgs=160 cs=0 l=1 rx=0x7fb0f0001110 tx=0x7fb0f0001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:50.798 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.788+0000 7fb105ce3700 1 --2- 172.21.15.42:0/3367907352 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb10014df60 0x7fb1001503a0 secure :-1 s=THROTTLE_DONE pgs=160 cs=0 l=1 rx=0x7fb0f0001110 tx=0x7fb0f0001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:50.798 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.788+0000 7fb0f7fff700 1 -- 172.21.15.42:0/3367907352 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb0fc001750 con 0x7fb10014df60 2019-12-05T05:03:50.798 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.788+0000 7fb105ce3700 1 --2- 172.21.15.42:0/3367907352 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb10014df60 0x7fb1001503a0 secure :-1 s=THROTTLE_DONE pgs=160 cs=0 l=1 rx=0x7fb0f0001110 tx=0x7fb0f0001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:50.799 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.788+0000 7fb0f7fff700 1 -- 172.21.15.42:0/3367907352 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0fc001560 con 0x7fb10014df60 2019-12-05T05:03:50.799 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.788+0000 7fb0f7fff700 1 -- 172.21.15.42:0/3367907352 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb0f8010be0 con 0x7fb10014df60 2019-12-05T05:03:50.799 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.788+0000 7fb106f45700 1 -- 172.21.15.42:0/3367907352 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb10014df60 msgr2=0x7fb1001503a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:50.799 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.788+0000 7fb106f45700 1 --2- 172.21.15.42:0/3367907352 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb10014df60 0x7fb1001503a0 secure :-1 s=READY pgs=160 cs=0 l=1 rx=0x7fb0f0001110 tx=0x7fb0f0001150).stop 2019-12-05T05:03:50.799 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.788+0000 7fb106f45700 1 -- 172.21.15.42:0/3367907352 shutdown_connections 2019-12-05T05:03:50.799 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.788+0000 7fb106f45700 1 --2- 172.21.15.42:0/3367907352 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb10014df60 0x7fb1001503a0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:50.799 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.788+0000 7fb106f45700 1 -- 172.21.15.42:0/3367907352 shutdown_connections 2019-12-05T05:03:50.800 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.788+0000 7fb106f45700 1 -- 172.21.15.42:0/3367907352 wait complete. 2019-12-05T05:03:50.800 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.788+0000 7fb106f45700 1 -- 172.21.15.42:0/3367907352 >> 172.21.15.42:0/3367907352 conn(0x7fb1000a9ef0 msgr2=0x7fb1000aa320 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:50.803 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.792+0000 7fb106f45700 1 Processor -- start 2019-12-05T05:03:50.803 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.792+0000 7fb106f45700 1 -- start start 2019-12-05T05:03:50.803 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.792+0000 7fb106f45700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb100121b10 0x7fb100147cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:50.803 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.792+0000 7fb106f45700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb1001468c0 0x7fb10007d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:50.803 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.792+0000 7fb106f45700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb10007e1e0 con 0x7fb10007ddb0 2019-12-05T05:03:50.803 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.792+0000 7fb106f45700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb10008fea0 con 0x7fb100121b10 2019-12-05T05:03:50.804 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.792+0000 7fb106f45700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb100090140 con 0x7fb1001468c0 2019-12-05T05:03:50.804 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.792+0000 7fb1054e2700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb100121b10 0x7fb100147cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:50.804 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.792+0000 7fb1054e2700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb100121b10 0x7fb100147cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:53866/0 (socket says 172.21.15.42:53866) 2019-12-05T05:03:50.804 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.792+0000 7fb1054e2700 1 -- 172.21.15.42:0/3262720526 learned_addr learned my addr 172.21.15.42:0/3262720526 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:03:50.804 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.792+0000 7fb104ce1700 1 --2- 172.21.15.42:0/3262720526 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb1001468c0 0x7fb10007d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:50.805 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.796+0000 7fb1054e2700 1 -- 172.21.15.42:0/3262720526 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb1001468c0 msgr2=0x7fb10007d860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:50.805 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.796+0000 7fb1054e2700 1 --2- 172.21.15.42:0/3262720526 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb1001468c0 0x7fb10007d860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:50.805 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.796+0000 7fb0f5ffb700 1 -- 172.21.15.42:0/3262720526 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3295731309 0 0) 0x7fb10007e1e0 con 0x7fb10007ddb0 2019-12-05T05:03:50.805 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.796+0000 7fb1054e2700 1 -- 172.21.15.42:0/3262720526 >> v1:172.21.15.40:6789/0 conn(0x7fb10007ddb0 legacy=0x7fb10008dd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:50.805 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.796+0000 7fb1054e2700 1 -- 172.21.15.42:0/3262720526 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb100145a80 con 0x7fb100121b10 2019-12-05T05:03:50.805 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.796+0000 7fb1054e2700 1 --2- 172.21.15.42:0/3262720526 >> v2:172.21.15.42:3300/0 conn(0x7fb100121b10 0x7fb100147cf0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fb1001476d0 tx=0x7fb0fc00c430).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:50.806 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.796+0000 7fb1054e2700 1 --2- 172.21.15.42:0/3262720526 >> v2:172.21.15.42:3300/0 conn(0x7fb100121b10 0x7fb100147cf0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fb1001476d0 tx=0x7fb0fc00c430).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:50.806 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.796+0000 7fb1054e2700 1 --2- 172.21.15.42:0/3262720526 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb100121b10 0x7fb100147cf0 secure :-1 s=READY pgs=191 cs=0 l=1 rx=0x7fb1001476d0 tx=0x7fb0fc00c430).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:50.806 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.796+0000 7fb1054e2700 1 --2- 172.21.15.42:0/3262720526 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb100121b10 0x7fb100147cf0 secure :-1 s=THROTTLE_DONE pgs=191 cs=0 l=1 rx=0x7fb1001476d0 tx=0x7fb0fc00c430).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:50.806 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.796+0000 7fb1054e2700 1 --2- 172.21.15.42:0/3262720526 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb100121b10 0x7fb100147cf0 secure :-1 s=THROTTLE_DONE pgs=191 cs=0 l=1 rx=0x7fb1001476d0 tx=0x7fb0fc00c430).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:50.806 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.796+0000 7fb0f5ffb700 1 -- 172.21.15.42:0/3262720526 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb0fc00c890 con 0x7fb100121b10 2019-12-05T05:03:50.807 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.796+0000 7fb1054e2700 1 --2- 172.21.15.42:0/3262720526 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb100121b10 0x7fb100147cf0 secure :-1 s=THROTTLE_DONE pgs=191 cs=0 l=1 rx=0x7fb1001476d0 tx=0x7fb0fc00c430).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:50.807 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.796+0000 7fb0f5ffb700 1 -- 172.21.15.42:0/3262720526 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0fc00c4b0 con 0x7fb100121b10 2019-12-05T05:03:50.807 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.796+0000 7fb106f45700 1 -- 172.21.15.42:0/3262720526 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb100146cf0 con 0x7fb100121b10 2019-12-05T05:03:50.807 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.796+0000 7fb0f5ffb700 1 -- 172.21.15.42:0/3262720526 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb0fc012440 con 0x7fb100121b10 2019-12-05T05:03:50.807 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.796+0000 7fb106f45700 1 -- 172.21.15.42:0/3262720526 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb100146190 con 0x7fb100121b10 2019-12-05T05:03:50.807 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.796+0000 7fb1054e2700 1 --2- 172.21.15.42:0/3262720526 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb100121b10 0x7fb100147cf0 secure :-1 s=THROTTLE_DONE pgs=191 cs=0 l=1 rx=0x7fb1001476d0 tx=0x7fb0fc00c430).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:50.807 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.796+0000 7fb0f5ffb700 1 -- 172.21.15.42:0/3262720526 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fb0fc012750 con 0x7fb100121b10 2019-12-05T05:03:50.808 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.796+0000 7fb1054e2700 1 --2- 172.21.15.42:0/3262720526 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb100121b10 0x7fb100147cf0 secure :-1 s=THROTTLE_DONE pgs=191 cs=0 l=1 rx=0x7fb1001476d0 tx=0x7fb0fc00c430).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:50.808 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.796+0000 7fb0f5ffb700 1 --2- 172.21.15.42:0/3262720526 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb0e003ace0 0x7fb0e003d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:50.808 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.796+0000 7fb0f5ffb700 1 -- 172.21.15.42:0/3262720526 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6031+0+0 (secure 0 0 0) 0x7fb0fc04ac50 con 0x7fb100121b10 2019-12-05T05:03:50.808 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.796+0000 7fb104ce1700 1 --2- 172.21.15.42:0/3262720526 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb0e003ace0 0x7fb0e003d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:50.809 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.796+0000 7fb104ce1700 1 --2- 172.21.15.42:0/3262720526 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb0e003ace0 0x7fb0e003d1f0 crc :-1 s=READY pgs=102 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:50.809 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.800+0000 7fb106f45700 1 -- 172.21.15.42:0/3262720526 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb10005d110 con 0x7fb100121b10 2019-12-05T05:03:50.811 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.804+0000 7fb1054e2700 1 --2- 172.21.15.42:0/3262720526 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb100121b10 0x7fb100147cf0 secure :-1 s=THROTTLE_DONE pgs=191 cs=0 l=1 rx=0x7fb1001476d0 tx=0x7fb0fc00c430).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:50.811 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.804+0000 7fb0f5ffb700 1 -- 172.21.15.42:0/3262720526 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fb0fc031cd0 con 0x7fb100121b10 2019-12-05T05:03:50.969 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.964+0000 7fb106f45700 1 -- 172.21.15.42:0/3262720526 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7fb1000a77a0 con 0x7fb100121b10 2019-12-05T05:03:50.969 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.964+0000 7fb1054e2700 1 --2- 172.21.15.42:0/3262720526 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb100121b10 0x7fb100147cf0 secure :-1 s=THROTTLE_DONE pgs=191 cs=0 l=1 rx=0x7fb1001476d0 tx=0x7fb0fc00c430).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:50.969 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.964+0000 7fb0f5ffb700 1 -- 172.21.15.42:0/3262720526 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v24) v1 ==== 92+0+10 (secure 0 0 0) 0x7fb0fc031f30 con 0x7fb100121b10 2019-12-05T05:03:50.970 INFO:teuthology.orchestra.run.smithi042.stdout:pg_num: 8 2019-12-05T05:03:50.972 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.968+0000 7fb106f45700 1 -- 172.21.15.42:0/3262720526 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb0e003ace0 msgr2=0x7fb0e003d1f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:50.973 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.968+0000 7fb106f45700 1 --2- 172.21.15.42:0/3262720526 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb0e003ace0 0x7fb0e003d1f0 crc :-1 s=READY pgs=102 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:50.973 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.968+0000 7fb106f45700 1 -- 172.21.15.42:0/3262720526 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb100121b10 msgr2=0x7fb100147cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:50.973 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.968+0000 7fb106f45700 1 --2- 172.21.15.42:0/3262720526 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb100121b10 0x7fb100147cf0 secure :-1 s=READY pgs=191 cs=0 l=1 rx=0x7fb1001476d0 tx=0x7fb0fc00c430).stop 2019-12-05T05:03:50.973 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.968+0000 7fb106f45700 1 -- 172.21.15.42:0/3262720526 shutdown_connections 2019-12-05T05:03:50.973 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.968+0000 7fb106f45700 1 --2- 172.21.15.42:0/3262720526 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb0e003ace0 0x7fb0e003d1f0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:50.973 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.968+0000 7fb106f45700 1 --2- 172.21.15.42:0/3262720526 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb100121b10 0x7fb100147cf0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:50.973 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.968+0000 7fb106f45700 1 --2- 172.21.15.42:0/3262720526 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb1001468c0 0x7fb10007d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:50.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.968+0000 7fb106f45700 1 -- 172.21.15.42:0/3262720526 shutdown_connections 2019-12-05T05:03:50.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.968+0000 7fb106f45700 1 -- 172.21.15.42:0/3262720526 wait complete. 2019-12-05T05:03:50.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:50.968+0000 7fb106f45700 1 -- 172.21.15.42:0/3262720526 >> 172.21.15.42:0/3262720526 conn(0x7fb1000a8b30 msgr2=0x7fb1000a4ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:50.982 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2019-12-05T05:03:51.081 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.072+0000 7fb30db03700 1 Processor -- start 2019-12-05T05:03:51.082 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.072+0000 7fb30db03700 1 -- start start 2019-12-05T05:03:51.082 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.072+0000 7fb30db03700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb30809f550 0x7fb308144630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:51.082 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.072+0000 7fb30db03700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb3080a9a00 0x7fb30807e880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:51.082 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.072+0000 7fb30db03700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb30807edd0 0x7fb308147b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:51.082 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.072+0000 7fb30db03700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3080a8f80 con 0x7fb30807edd0 2019-12-05T05:03:51.082 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.072+0000 7fb30db03700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3080a9220 con 0x7fb3080a9a00 2019-12-05T05:03:51.083 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.072+0000 7fb30db03700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3080a94c0 con 0x7fb30809f550 2019-12-05T05:03:51.083 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.072+0000 7fb3077fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb3080a9a00 0x7fb30807e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:51.083 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.072+0000 7fb3077fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb3080a9a00 0x7fb30807e880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:53874/0 (socket says 172.21.15.42:53874) 2019-12-05T05:03:51.083 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.072+0000 7fb3077fe700 1 -- 172.21.15.42:0/318643347 learned_addr learned my addr 172.21.15.42:0/318643347 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:03:51.083 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.072+0000 7fb307fff700 1 --2- 172.21.15.42:0/318643347 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb30809f550 0x7fb308144630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:51.083 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.072+0000 7fb30c8a1700 1 --2- 172.21.15.42:0/318643347 >> v2:172.21.15.40:3300/0 conn(0x7fb30807edd0 0x7fb308147b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:51.083 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.072+0000 7fb3077fe700 1 -- 172.21.15.42:0/318643347 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb30809f550 msgr2=0x7fb308144630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:03:51.084 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.072+0000 7fb3077fe700 1 --2- 172.21.15.42:0/318643347 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb30809f550 0x7fb308144630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:03:51.084 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.072+0000 7fb3077fe700 1 -- 172.21.15.42:0/318643347 >> v2:172.21.15.40:3300/0 conn(0x7fb30807edd0 msgr2=0x7fb308147b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:03:51.084 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.072+0000 7fb3077fe700 1 --2- 172.21.15.42:0/318643347 >> v2:172.21.15.40:3300/0 conn(0x7fb30807edd0 0x7fb308147b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:03:51.084 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.072+0000 7fb3077fe700 1 -- 172.21.15.42:0/318643347 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3080a1000 con 0x7fb3080a9a00 2019-12-05T05:03:51.084 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.072+0000 7fb30c8a1700 1 --2- 172.21.15.42:0/318643347 >> v2:172.21.15.40:3300/0 conn(0x7fb30807edd0 0x7fb308147b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T05:03:51.084 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.072+0000 7fb3077fe700 1 --2- 172.21.15.42:0/318643347 >> v2:172.21.15.42:3300/0 conn(0x7fb3080a9a00 0x7fb30807e880 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fb3080a93e0 tx=0x7fb2f0010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:51.084 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.072+0000 7fb3077fe700 1 --2- 172.21.15.42:0/318643347 >> v2:172.21.15.42:3300/0 conn(0x7fb3080a9a00 0x7fb30807e880 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fb3080a93e0 tx=0x7fb2f0010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:51.085 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.072+0000 7fb3077fe700 1 --2- 172.21.15.42:0/318643347 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb3080a9a00 0x7fb30807e880 secure :-1 s=READY pgs=192 cs=0 l=1 rx=0x7fb3080a93e0 tx=0x7fb2f0010130).ready entity=mon.1 client_cookie=c6caf25550aee7f8 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:51.085 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.072+0000 7fb3077fe700 1 --2- 172.21.15.42:0/318643347 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb3080a9a00 0x7fb30807e880 secure :-1 s=THROTTLE_DONE pgs=192 cs=0 l=1 rx=0x7fb3080a93e0 tx=0x7fb2f0010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:51.085 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.072+0000 7fb3077fe700 1 --2- 172.21.15.42:0/318643347 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb3080a9a00 0x7fb30807e880 secure :-1 s=THROTTLE_DONE pgs=192 cs=0 l=1 rx=0x7fb3080a93e0 tx=0x7fb2f0010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:51.085 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.072+0000 7fb306ffd700 1 -- 172.21.15.42:0/318643347 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb2f000a160 con 0x7fb3080a9a00 2019-12-05T05:03:51.085 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.072+0000 7fb3077fe700 1 --2- 172.21.15.42:0/318643347 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb3080a9a00 0x7fb30807e880 secure :-1 s=THROTTLE_DONE pgs=192 cs=0 l=1 rx=0x7fb3080a93e0 tx=0x7fb2f0010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:51.085 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.072+0000 7fb306ffd700 1 -- 172.21.15.42:0/318643347 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2f0011a70 con 0x7fb3080a9a00 2019-12-05T05:03:51.085 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.072+0000 7fb306ffd700 1 -- 172.21.15.42:0/318643347 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb2f000a740 con 0x7fb3080a9a00 2019-12-05T05:03:51.086 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.072+0000 7fb30db03700 1 -- 172.21.15.42:0/318643347 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb3080a9a00 msgr2=0x7fb30807e880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:51.086 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.072+0000 7fb30db03700 1 --2- 172.21.15.42:0/318643347 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb3080a9a00 0x7fb30807e880 secure :-1 s=READY pgs=192 cs=0 l=1 rx=0x7fb3080a93e0 tx=0x7fb2f0010130).stop 2019-12-05T05:03:51.086 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.076+0000 7fb30db03700 1 -- 172.21.15.42:0/318643347 shutdown_connections 2019-12-05T05:03:51.086 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.076+0000 7fb30db03700 1 --2- 172.21.15.42:0/318643347 >> v2:172.21.15.40:3300/0 conn(0x7fb30807edd0 0x7fb308147b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:03:51.086 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.076+0000 7fb30db03700 1 --2- 172.21.15.42:0/318643347 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb30809f550 0x7fb308144630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:03:51.086 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.076+0000 7fb30db03700 1 --2- 172.21.15.42:0/318643347 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb3080a9a00 0x7fb30807e880 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:51.086 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.076+0000 7fb30db03700 1 -- 172.21.15.42:0/318643347 shutdown_connections 2019-12-05T05:03:51.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.076+0000 7fb30db03700 1 -- 172.21.15.42:0/318643347 wait complete. 2019-12-05T05:03:51.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.076+0000 7fb30db03700 1 -- 172.21.15.42:0/318643347 >> 172.21.15.42:0/318643347 conn(0x7fb3080a9ef0 msgr2=0x7fb3080aa320 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:51.091 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.076+0000 7fb30db03700 1 Processor -- start 2019-12-05T05:03:51.093 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.076+0000 7fb30db03700 1 -- start start 2019-12-05T05:03:51.093 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.080+0000 7fb30db03700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb308120170 0x7fb3081466e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:51.093 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.080+0000 7fb30db03700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb308146c30 con 0x7fb30812bae0 2019-12-05T05:03:51.094 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.080+0000 7fb30db03700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb308144630 con 0x7fb30812e840 2019-12-05T05:03:51.094 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.080+0000 7fb30db03700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb308147050 con 0x7fb308120170 2019-12-05T05:03:51.094 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.080+0000 7fb3077fe700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fb30812e840 0x7fb30812b2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:53918/0 (socket says 172.21.15.42:53918) 2019-12-05T05:03:51.094 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.080+0000 7fb3077fe700 1 -- 172.21.15.42:0/685445755 learned_addr learned my addr 172.21.15.42:0/685445755 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:03:51.094 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.080+0000 7fb307fff700 1 --2- 172.21.15.42:0/685445755 >> v2:172.21.15.40:3300/0 conn(0x7fb308120170 0x7fb3081466e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:51.094 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.080+0000 7fb304ff9700 1 -- 172.21.15.42:0/685445755 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 315746288 0 0) 0x7fb308144630 con 0x7fb30812e840 2019-12-05T05:03:51.094 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.080+0000 7fb304ff9700 1 -- 172.21.15.42:0/685445755 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2ec0030b0 con 0x7fb30812e840 2019-12-05T05:03:51.095 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.080+0000 7fb304ff9700 1 -- 172.21.15.42:0/685445755 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 4088447313 0 0) 0x7fb2ec0030b0 con 0x7fb30812e840 2019-12-05T05:03:51.095 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.080+0000 7fb304ff9700 1 -- 172.21.15.42:0/685445755 --> v1:172.21.15.42:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb308144630 con 0x7fb30812e840 2019-12-05T05:03:51.095 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.080+0000 7fb304ff9700 1 -- 172.21.15.42:0/685445755 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7fb2f000a160 con 0x7fb30812e840 2019-12-05T05:03:51.095 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.080+0000 7fb304ff9700 1 -- 172.21.15.42:0/685445755 >> v2:172.21.15.40:3300/0 conn(0x7fb308120170 msgr2=0x7fb3081466e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:51.095 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.080+0000 7fb304ff9700 1 --2- 172.21.15.42:0/685445755 >> v2:172.21.15.40:3300/0 conn(0x7fb308120170 0x7fb3081466e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:51.095 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.080+0000 7fb304ff9700 1 -- 172.21.15.42:0/685445755 >> v1:172.21.15.42:6789/0 conn(0x7fb30812e840 legacy=0x7fb30812b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:51.095 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.080+0000 7fb304ff9700 1 -- 172.21.15.42:0/685445755 >> v1:172.21.15.40:6789/0 conn(0x7fb30812bae0 legacy=0x7fb3081357c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:51.096 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.080+0000 7fb304ff9700 1 --2- 172.21.15.42:0/685445755 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb2ec0038a0 0x7fb2ec003cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:51.096 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.080+0000 7fb304ff9700 1 --2- 172.21.15.42:0/685445755 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb2ec004420 0x7fb2ec004850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:51.096 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.080+0000 7fb304ff9700 1 --2- 172.21.15.42:0/685445755 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb2ec004e20 0x7fb2ec00b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:51.096 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.080+0000 7fb304ff9700 1 -- 172.21.15.42:0/685445755 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb308147050 con 0x7fb2ec0038a0 2019-12-05T05:03:51.096 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.080+0000 7fb304ff9700 1 -- 172.21.15.42:0/685445755 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2ec00bcb0 con 0x7fb2ec004e20 2019-12-05T05:03:51.096 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.080+0000 7fb304ff9700 1 -- 172.21.15.42:0/685445755 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2ec00bef0 con 0x7fb2ec004420 2019-12-05T05:03:51.096 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.080+0000 7fb307fff700 1 --2- 172.21.15.42:0/685445755 >> v2:172.21.15.40:3300/0 conn(0x7fb308120170 0x7fb3081466e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:03:51.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.080+0000 7fb30c8a1700 1 --2- 172.21.15.42:0/685445755 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb2ec004e20 0x7fb2ec00b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:51.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.080+0000 7fb3077fe700 1 --2- 172.21.15.42:0/685445755 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb2ec004420 0x7fb2ec004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:51.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.080+0000 7fb3077fe700 1 -- 172.21.15.42:0/685445755 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb2ec004420 msgr2=0x7fb2ec004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2019-12-05T05:03:51.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.080+0000 7fb3077fe700 1 -- 172.21.15.42:0/685445755 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb2ec004420 msgr2=0x7fb2ec004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2019-12-05T05:03:51.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.080+0000 7fb3077fe700 1 --2- 172.21.15.42:0/685445755 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb2ec004420 0x7fb2ec004850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-12-05T05:03:51.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.080+0000 7fb3077fe700 1 --2- 172.21.15.42:0/685445755 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb2ec004420 0x7fb2ec004850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rx=0 tx=0)._fault waiting 0.200000 2019-12-05T05:03:51.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.080+0000 7fb307fff700 1 --2- 172.21.15.42:0/685445755 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb2ec0038a0 0x7fb2ec003cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:51.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.080+0000 7fb30c8a1700 1 -- 172.21.15.42:0/685445755 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb2ec004420 msgr2=0x7fb2ec004850 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2019-12-05T05:03:51.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.080+0000 7fb30c8a1700 1 --2- 172.21.15.42:0/685445755 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb2ec004420 0x7fb2ec004850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:51.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.080+0000 7fb30c8a1700 1 -- 172.21.15.42:0/685445755 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb2ec0038a0 msgr2=0x7fb2ec003cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:51.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.080+0000 7fb30c8a1700 1 --2- 172.21.15.42:0/685445755 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb2ec0038a0 0x7fb2ec003cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:51.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.080+0000 7fb30c8a1700 1 -- 172.21.15.42:0/685445755 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb30812ec70 con 0x7fb2ec004e20 2019-12-05T05:03:51.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.080+0000 7fb30db03700 1 -- 172.21.15.42:0/685445755 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb30812ef40 con 0x7fb2ec004e20 2019-12-05T05:03:51.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.080+0000 7fb30c8a1700 1 --2- 172.21.15.42:0/685445755 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb2ec004e20 0x7fb2ec00b670 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fb3000078c0 tx=0x7fb300010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:51.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.080+0000 7fb30db03700 1 -- 172.21.15.42:0/685445755 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb308136fd0 con 0x7fb2ec004e20 2019-12-05T05:03:51.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.080+0000 7fb30c8a1700 1 -- 172.21.15.42:0/685445755 reap_dead start 2019-12-05T05:03:51.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.080+0000 7fb30c8a1700 1 --2- 172.21.15.42:0/685445755 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb2ec004e20 0x7fb2ec00b670 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fb3000078c0 tx=0x7fb300010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:51.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.080+0000 7fb30c8a1700 1 --2- 172.21.15.42:0/685445755 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb2ec004e20 0x7fb2ec00b670 secure :-1 s=READY pgs=194 cs=0 l=1 rx=0x7fb3000078c0 tx=0x7fb300010b50).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:51.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.084+0000 7fb30c8a1700 1 --2- 172.21.15.42:0/685445755 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb2ec004e20 0x7fb2ec00b670 secure :-1 s=THROTTLE_DONE pgs=194 cs=0 l=1 rx=0x7fb3000078c0 tx=0x7fb300010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:51.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.084+0000 7fb304ff9700 1 -- 172.21.15.42:0/685445755 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb300005540 con 0x7fb2ec004e20 2019-12-05T05:03:51.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.084+0000 7fb30c8a1700 1 --2- 172.21.15.42:0/685445755 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb2ec004e20 0x7fb2ec00b670 secure :-1 s=THROTTLE_DONE pgs=194 cs=0 l=1 rx=0x7fb3000078c0 tx=0x7fb300010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:51.100 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.084+0000 7fb304ff9700 1 -- 172.21.15.42:0/685445755 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb300007180 con 0x7fb2ec004e20 2019-12-05T05:03:51.100 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.084+0000 7fb30c8a1700 1 --2- 172.21.15.42:0/685445755 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb2ec004e20 0x7fb2ec00b670 secure :-1 s=THROTTLE_DONE pgs=194 cs=0 l=1 rx=0x7fb3000078c0 tx=0x7fb300010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:51.100 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.084+0000 7fb304ff9700 1 -- 172.21.15.42:0/685445755 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb300005b20 con 0x7fb2ec004e20 2019-12-05T05:03:51.100 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.084+0000 7fb30c8a1700 1 --2- 172.21.15.42:0/685445755 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb2ec004e20 0x7fb2ec00b670 secure :-1 s=THROTTLE_DONE pgs=194 cs=0 l=1 rx=0x7fb3000078c0 tx=0x7fb300010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:51.100 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.084+0000 7fb304ff9700 1 -- 172.21.15.42:0/685445755 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fb30000e4d0 con 0x7fb2ec004e20 2019-12-05T05:03:51.100 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.084+0000 7fb30c8a1700 1 --2- 172.21.15.42:0/685445755 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb2ec004e20 0x7fb2ec00b670 secure :-1 s=THROTTLE_DONE pgs=194 cs=0 l=1 rx=0x7fb3000078c0 tx=0x7fb300010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:51.100 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.084+0000 7fb304ff9700 1 --2- 172.21.15.42:0/685445755 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb2ec03e9b0 0x7fb2ec040ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:51.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.084+0000 7fb304ff9700 1 -- 172.21.15.42:0/685445755 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6031+0+0 (secure 0 0 0) 0x7fb30004ad50 con 0x7fb2ec004e20 2019-12-05T05:03:51.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.084+0000 7fb307fff700 1 --2- 172.21.15.42:0/685445755 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb2ec03e9b0 0x7fb2ec040ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:51.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.084+0000 7fb307fff700 1 --2- 172.21.15.42:0/685445755 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb2ec03e9b0 0x7fb2ec040ec0 crc :-1 s=READY pgs=103 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:51.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.088+0000 7fb30db03700 1 -- 172.21.15.42:0/685445755 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2e0003200 con 0x7fb2ec004e20 2019-12-05T05:03:51.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.096+0000 7fb30c8a1700 1 --2- 172.21.15.42:0/685445755 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb2ec004e20 0x7fb2ec00b670 secure :-1 s=THROTTLE_DONE pgs=194 cs=0 l=1 rx=0x7fb3000078c0 tx=0x7fb300010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:51.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.096+0000 7fb304ff9700 1 -- 172.21.15.42:0/685445755 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fb300031db0 con 0x7fb2ec004e20 2019-12-05T05:03:51.264 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.260+0000 7fb30db03700 1 -- 172.21.15.42:0/685445755 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7fb2e0003650 con 0x7fb2ec004e20 2019-12-05T05:03:51.264 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.260+0000 7fb30c8a1700 1 --2- 172.21.15.42:0/685445755 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb2ec004e20 0x7fb2ec00b670 secure :-1 s=THROTTLE_DONE pgs=194 cs=0 l=1 rx=0x7fb3000078c0 tx=0x7fb300010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:51.265 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.260+0000 7fb304ff9700 1 -- 172.21.15.42:0/685445755 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v24) v1 ==== 104+0+10 (secure 0 0 0) 0x7fb300032010 con 0x7fb2ec004e20 2019-12-05T05:03:51.267 INFO:teuthology.orchestra.run.smithi042.stdout:pg_num: 8 2019-12-05T05:03:51.270 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.264+0000 7fb30db03700 1 -- 172.21.15.42:0/685445755 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb2ec03e9b0 msgr2=0x7fb2ec040ec0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:51.270 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.264+0000 7fb30db03700 1 --2- 172.21.15.42:0/685445755 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb2ec03e9b0 0x7fb2ec040ec0 crc :-1 s=READY pgs=103 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:51.270 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.264+0000 7fb30db03700 1 -- 172.21.15.42:0/685445755 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb2ec004e20 msgr2=0x7fb2ec00b670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:51.270 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.264+0000 7fb30db03700 1 --2- 172.21.15.42:0/685445755 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb2ec004e20 0x7fb2ec00b670 secure :-1 s=READY pgs=194 cs=0 l=1 rx=0x7fb3000078c0 tx=0x7fb300010b50).stop 2019-12-05T05:03:51.270 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.264+0000 7fb30db03700 1 -- 172.21.15.42:0/685445755 shutdown_connections 2019-12-05T05:03:51.270 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.264+0000 7fb30db03700 1 --2- 172.21.15.42:0/685445755 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb2ec03e9b0 0x7fb2ec040ec0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:51.271 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.264+0000 7fb30db03700 1 --2- 172.21.15.42:0/685445755 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb2ec004e20 0x7fb2ec00b670 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:51.271 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.264+0000 7fb30db03700 1 -- 172.21.15.42:0/685445755 shutdown_connections 2019-12-05T05:03:51.271 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.264+0000 7fb30db03700 1 -- 172.21.15.42:0/685445755 wait complete. 2019-12-05T05:03:51.271 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.264+0000 7fb30db03700 1 -- 172.21.15.42:0/685445755 >> 172.21.15.42:0/685445755 conn(0x7fb308066fe0 msgr2=0x7fb30805a230 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:51.280 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2019-12-05T05:03:51.382 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.372+0000 7f668ddc0700 1 Processor -- start 2019-12-05T05:03:51.382 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.372+0000 7f668ddc0700 1 -- start start 2019-12-05T05:03:51.382 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.372+0000 7f668ddc0700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f668809f550 0x7f6688144630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:51.382 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.372+0000 7f668ddc0700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f668814cf50 0x7f668814f390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:51.382 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.372+0000 7f668ddc0700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f668807f080 con 0x7f66880a9a00 2019-12-05T05:03:51.383 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.372+0000 7f668ddc0700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66880a8f80 con 0x7f668814cf50 2019-12-05T05:03:51.383 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.372+0000 7f668ddc0700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66880a9220 con 0x7f668809f550 2019-12-05T05:03:51.383 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.372+0000 7f6687fff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f668809f550 0x7f6688144630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:51.383 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.372+0000 7f6687fff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f668809f550 0x7f6688144630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:53894/0 (socket says 172.21.15.42:53894) 2019-12-05T05:03:51.383 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.372+0000 7f6687fff700 1 -- 172.21.15.42:0/1623098730 learned_addr learned my addr 172.21.15.42:0/1623098730 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:03:51.383 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.372+0000 7f668cb5e700 1 --2- 172.21.15.42:0/1623098730 >> v2:172.21.15.40:3300/0 conn(0x7f668814cf50 0x7f668814f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:51.383 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.372+0000 7f6687fff700 1 -- 172.21.15.42:0/1623098730 >> v2:172.21.15.40:3300/0 conn(0x7f668814cf50 msgr2=0x7f668814f390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:03:51.384 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.372+0000 7f6687fff700 1 --2- 172.21.15.42:0/1623098730 >> v2:172.21.15.40:3300/0 conn(0x7f668814cf50 0x7f668814f390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:03:51.384 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.372+0000 7f6687fff700 1 -- 172.21.15.42:0/1623098730 >> v1:172.21.15.40:6789/0 conn(0x7f66880a9a00 legacy=0x7f668807e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:51.384 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.372+0000 7f6687fff700 1 -- 172.21.15.42:0/1623098730 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66880a1000 con 0x7f668809f550 2019-12-05T05:03:51.384 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.372+0000 7f6687fff700 1 --2- 172.21.15.42:0/1623098730 >> v2:172.21.15.42:3300/0 conn(0x7f668809f550 0x7f6688144630 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f6678002f30 tx=0x7f667800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:51.384 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.372+0000 7f6687fff700 1 --2- 172.21.15.42:0/1623098730 >> v2:172.21.15.42:3300/0 conn(0x7f668809f550 0x7f6688144630 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f6678002f30 tx=0x7f667800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:51.385 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.372+0000 7f6687fff700 1 --2- 172.21.15.42:0/1623098730 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f668809f550 0x7f6688144630 secure :-1 s=READY pgs=195 cs=0 l=1 rx=0x7f6678002f30 tx=0x7f667800fd30).ready entity=mon.1 client_cookie=c71f161f7751e058 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:51.385 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.372+0000 7f6687fff700 1 --2- 172.21.15.42:0/1623098730 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f668809f550 0x7f6688144630 secure :-1 s=THROTTLE_DONE pgs=195 cs=0 l=1 rx=0x7f6678002f30 tx=0x7f667800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:51.385 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.372+0000 7f66877fe700 1 -- 172.21.15.42:0/1623098730 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6678008c90 con 0x7f668809f550 2019-12-05T05:03:51.385 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.372+0000 7f6687fff700 1 --2- 172.21.15.42:0/1623098730 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f668809f550 0x7f6688144630 secure :-1 s=THROTTLE_DONE pgs=195 cs=0 l=1 rx=0x7f6678002f30 tx=0x7f667800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:51.385 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.372+0000 7f6687fff700 1 --2- 172.21.15.42:0/1623098730 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f668809f550 0x7f6688144630 secure :-1 s=THROTTLE_DONE pgs=195 cs=0 l=1 rx=0x7f6678002f30 tx=0x7f667800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:51.385 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.372+0000 7f66877fe700 1 -- 172.21.15.42:0/1623098730 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66780088b0 con 0x7f668809f550 2019-12-05T05:03:51.385 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.376+0000 7f66877fe700 1 -- 172.21.15.42:0/1623098730 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6678011ad0 con 0x7f668809f550 2019-12-05T05:03:51.386 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.376+0000 7f668ddc0700 1 -- 172.21.15.42:0/1623098730 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f668809f550 msgr2=0x7f6688144630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:51.386 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.376+0000 7f668ddc0700 1 --2- 172.21.15.42:0/1623098730 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f668809f550 0x7f6688144630 secure :-1 s=READY pgs=195 cs=0 l=1 rx=0x7f6678002f30 tx=0x7f667800fd30).stop 2019-12-05T05:03:51.386 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.376+0000 7f668ddc0700 1 -- 172.21.15.42:0/1623098730 shutdown_connections 2019-12-05T05:03:51.386 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.376+0000 7f668ddc0700 1 --2- 172.21.15.42:0/1623098730 >> v2:172.21.15.40:3300/0 conn(0x7f668814cf50 0x7f668814f390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:03:51.386 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.376+0000 7f668ddc0700 1 --2- 172.21.15.42:0/1623098730 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f668809f550 0x7f6688144630 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:51.386 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.376+0000 7f668ddc0700 1 -- 172.21.15.42:0/1623098730 shutdown_connections 2019-12-05T05:03:51.386 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.376+0000 7f668ddc0700 1 -- 172.21.15.42:0/1623098730 wait complete. 2019-12-05T05:03:51.387 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.376+0000 7f668ddc0700 1 -- 172.21.15.42:0/1623098730 >> 172.21.15.42:0/1623098730 conn(0x7f66880a9ef0 msgr2=0x7f66880aa320 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:51.389 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.376+0000 7f668ddc0700 1 Processor -- start 2019-12-05T05:03:51.390 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.376+0000 7f668ddc0700 1 -- start start 2019-12-05T05:03:51.390 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.380+0000 7f668ddc0700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6688143510 0x7f668808e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:51.390 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.380+0000 7f668ddc0700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f668807d940 0x7f668807dd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:51.390 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.380+0000 7f668ddc0700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f668807e2c0 con 0x7f66880a3f50 2019-12-05T05:03:51.390 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.380+0000 7f668ddc0700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f668811ab00 con 0x7f668807d940 2019-12-05T05:03:51.390 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.380+0000 7f668ddc0700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f668808ec20 con 0x7f6688143510 2019-12-05T05:03:51.391 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.380+0000 7f667f7fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6688143510 0x7f668808e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:51.391 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.380+0000 7f6687fff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f66880a3f50 0x7f668808fb30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:53940/0 (socket says 172.21.15.42:53940) 2019-12-05T05:03:51.391 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.380+0000 7f6687fff700 1 -- 172.21.15.42:0/3369102783 learned_addr learned my addr 172.21.15.42:0/3369102783 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:03:51.391 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.380+0000 7f668cb5e700 1 --2- 172.21.15.42:0/3369102783 >> v2:172.21.15.40:3300/0 conn(0x7f668807d940 0x7f668807dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:51.391 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.380+0000 7f667f7fe700 1 -- 172.21.15.42:0/3369102783 >> v2:172.21.15.40:3300/0 conn(0x7f668807d940 msgr2=0x7f668807dd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:51.391 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.380+0000 7f667f7fe700 1 --2- 172.21.15.42:0/3369102783 >> v2:172.21.15.40:3300/0 conn(0x7f668807d940 0x7f668807dd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:51.391 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.380+0000 7f667f7fe700 1 -- 172.21.15.42:0/3369102783 >> v1:172.21.15.42:6789/0 conn(0x7f66880a3f50 legacy=0x7f668808fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:51.392 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.380+0000 7f667f7fe700 1 -- 172.21.15.42:0/3369102783 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f668808de00 con 0x7f6688143510 2019-12-05T05:03:51.392 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.380+0000 7f66857fa700 1 -- 172.21.15.42:0/3369102783 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1429434674 0 0) 0x7f6678008470 con 0x7f66880a3f50 2019-12-05T05:03:51.392 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.380+0000 7f667f7fe700 1 --2- 172.21.15.42:0/3369102783 >> v2:172.21.15.42:3300/0 conn(0x7f6688143510 0x7f668808e6d0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f6670000e90 tx=0x7f66700108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:51.392 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.380+0000 7f667f7fe700 1 --2- 172.21.15.42:0/3369102783 >> v2:172.21.15.42:3300/0 conn(0x7f6688143510 0x7f668808e6d0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f6670000e90 tx=0x7f66700108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:51.392 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.380+0000 7f667f7fe700 1 --2- 172.21.15.42:0/3369102783 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6688143510 0x7f668808e6d0 secure :-1 s=READY pgs=197 cs=0 l=1 rx=0x7f6670000e90 tx=0x7f66700108f0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:51.392 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.380+0000 7f667f7fe700 1 --2- 172.21.15.42:0/3369102783 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6688143510 0x7f668808e6d0 secure :-1 s=THROTTLE_DONE pgs=197 cs=0 l=1 rx=0x7f6670000e90 tx=0x7f66700108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:51.392 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.380+0000 7f66857fa700 1 -- 172.21.15.42:0/3369102783 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f667000cb90 con 0x7f6688143510 2019-12-05T05:03:51.393 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.380+0000 7f667f7fe700 1 --2- 172.21.15.42:0/3369102783 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6688143510 0x7f668808e6d0 secure :-1 s=THROTTLE_DONE pgs=197 cs=0 l=1 rx=0x7f6670000e90 tx=0x7f66700108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:51.393 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.380+0000 7f667f7fe700 1 --2- 172.21.15.42:0/3369102783 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6688143510 0x7f668808e6d0 secure :-1 s=THROTTLE_DONE pgs=197 cs=0 l=1 rx=0x7f6670000e90 tx=0x7f66700108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:51.393 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.380+0000 7f66857fa700 1 -- 172.21.15.42:0/3369102783 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f667000c7b0 con 0x7f6688143510 2019-12-05T05:03:51.393 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.380+0000 7f668ddc0700 1 -- 172.21.15.42:0/3369102783 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6688130cb0 con 0x7f6688143510 2019-12-05T05:03:51.393 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.380+0000 7f66857fa700 1 -- 172.21.15.42:0/3369102783 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f66700110b0 con 0x7f6688143510 2019-12-05T05:03:51.393 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.380+0000 7f668ddc0700 1 -- 172.21.15.42:0/3369102783 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66881311e0 con 0x7f6688143510 2019-12-05T05:03:51.393 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.380+0000 7f667f7fe700 1 --2- 172.21.15.42:0/3369102783 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6688143510 0x7f668808e6d0 secure :-1 s=THROTTLE_DONE pgs=197 cs=0 l=1 rx=0x7f6670000e90 tx=0x7f66700108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:51.394 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.384+0000 7f66857fa700 1 -- 172.21.15.42:0/3369102783 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f6670011340 con 0x7f6688143510 2019-12-05T05:03:51.394 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.384+0000 7f667f7fe700 1 --2- 172.21.15.42:0/3369102783 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6688143510 0x7f668808e6d0 secure :-1 s=THROTTLE_DONE pgs=197 cs=0 l=1 rx=0x7f6670000e90 tx=0x7f66700108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:51.394 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.384+0000 7f66857fa700 1 --2- 172.21.15.42:0/3369102783 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f666c03ac50 0x7f666c03d160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:51.394 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.384+0000 7f66857fa700 1 -- 172.21.15.42:0/3369102783 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f667004b190 con 0x7f6688143510 2019-12-05T05:03:51.394 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.384+0000 7f6687fff700 1 --2- 172.21.15.42:0/3369102783 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f666c03ac50 0x7f666c03d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:51.394 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.384+0000 7f6687fff700 1 --2- 172.21.15.42:0/3369102783 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f666c03ac50 0x7f666c03d160 crc :-1 s=READY pgs=104 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:51.395 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.388+0000 7f668ddc0700 1 -- 172.21.15.42:0/3369102783 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f668805d110 con 0x7f6688143510 2019-12-05T05:03:51.398 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.392+0000 7f667f7fe700 1 --2- 172.21.15.42:0/3369102783 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6688143510 0x7f668808e6d0 secure :-1 s=THROTTLE_DONE pgs=197 cs=0 l=1 rx=0x7f6670000e90 tx=0x7f66700108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:51.398 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.392+0000 7f66857fa700 1 -- 172.21.15.42:0/3369102783 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f6670032210 con 0x7f6688143510 2019-12-05T05:03:51.554 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.548+0000 7f668ddc0700 1 -- 172.21.15.42:0/3369102783 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7f668805b8c0 con 0x7f6688143510 2019-12-05T05:03:51.555 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.548+0000 7f667f7fe700 1 --2- 172.21.15.42:0/3369102783 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6688143510 0x7f668808e6d0 secure :-1 s=THROTTLE_DONE pgs=197 cs=0 l=1 rx=0x7f6670000e90 tx=0x7f66700108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:51.555 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.548+0000 7f66857fa700 1 -- 172.21.15.42:0/3369102783 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v24) v1 ==== 100+0+10 (secure 0 0 0) 0x7f6670032470 con 0x7f6688143510 2019-12-05T05:03:51.555 INFO:teuthology.orchestra.run.smithi042.stdout:pg_num: 8 2019-12-05T05:03:51.558 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.552+0000 7f668ddc0700 1 -- 172.21.15.42:0/3369102783 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f666c03ac50 msgr2=0x7f666c03d160 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:51.558 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.552+0000 7f668ddc0700 1 --2- 172.21.15.42:0/3369102783 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f666c03ac50 0x7f666c03d160 crc :-1 s=READY pgs=104 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:51.558 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.552+0000 7f668ddc0700 1 -- 172.21.15.42:0/3369102783 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6688143510 msgr2=0x7f668808e6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:51.558 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.552+0000 7f668ddc0700 1 --2- 172.21.15.42:0/3369102783 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6688143510 0x7f668808e6d0 secure :-1 s=READY pgs=197 cs=0 l=1 rx=0x7f6670000e90 tx=0x7f66700108f0).stop 2019-12-05T05:03:51.558 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.552+0000 7f668ddc0700 1 -- 172.21.15.42:0/3369102783 shutdown_connections 2019-12-05T05:03:51.558 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.552+0000 7f668ddc0700 1 --2- 172.21.15.42:0/3369102783 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f666c03ac50 0x7f666c03d160 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:51.559 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.552+0000 7f668ddc0700 1 --2- 172.21.15.42:0/3369102783 >> v2:172.21.15.40:3300/0 conn(0x7f668807d940 0x7f668807dd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:51.559 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.552+0000 7f668ddc0700 1 --2- 172.21.15.42:0/3369102783 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6688143510 0x7f668808e6d0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:51.559 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.552+0000 7f668ddc0700 1 -- 172.21.15.42:0/3369102783 shutdown_connections 2019-12-05T05:03:51.559 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.552+0000 7f668ddc0700 1 -- 172.21.15.42:0/3369102783 wait complete. 2019-12-05T05:03:51.559 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.552+0000 7f668ddc0700 1 -- 172.21.15.42:0/3369102783 >> 172.21.15.42:0/3369102783 conn(0x7f668809f550 msgr2=0x7f66880a5180 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:51.568 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:03:51.664 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.652+0000 7fa161131700 1 Processor -- start 2019-12-05T05:03:51.665 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.652+0000 7fa161131700 1 -- start start 2019-12-05T05:03:51.665 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.656+0000 7fa161131700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa15c0a7730 0x7fa15c08df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:51.665 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.656+0000 7fa161131700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa15c0a0700 con 0x7fa15c07e710 2019-12-05T05:03:51.665 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.656+0000 7fa161131700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa15c0a9b90 con 0x7fa15c09fad0 2019-12-05T05:03:51.665 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.656+0000 7fa161131700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa15c0a0af0 con 0x7fa15c0a7730 2019-12-05T05:03:51.665 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.656+0000 7fa160930700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fa15c09fad0 0x7fa15c14efd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:53950/0 (socket says 172.21.15.42:53950) 2019-12-05T05:03:51.666 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.656+0000 7fa160930700 1 -- 172.21.15.42:0/3474703948 learned_addr learned my addr 172.21.15.42:0/3474703948 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:03:51.666 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.656+0000 7fa15b59e700 1 --2- 172.21.15.42:0/3474703948 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa15c0a7730 0x7fa15c08df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:51.666 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.656+0000 7fa15a59c700 1 -- 172.21.15.42:0/3474703948 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2097876376 0 0) 0x7fa15c0a9b90 con 0x7fa15c09fad0 2019-12-05T05:03:51.666 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.656+0000 7fa15b59e700 1 -- 172.21.15.42:0/3474703948 >> v1:172.21.15.42:6789/0 conn(0x7fa15c09fad0 legacy=0x7fa15c14efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:51.666 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.656+0000 7fa15b59e700 1 -- 172.21.15.42:0/3474703948 >> v1:172.21.15.40:6789/0 conn(0x7fa15c07e710 legacy=0x7fa15c09f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:51.666 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.656+0000 7fa15b59e700 1 -- 172.21.15.42:0/3474703948 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa15c07eb40 con 0x7fa15c0a7730 2019-12-05T05:03:51.666 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.656+0000 7fa15b59e700 1 --2- 172.21.15.42:0/3474703948 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa15c0a7730 0x7fa15c08df70 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fa144001110 tx=0x7fa144001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:51.667 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.656+0000 7fa15b59e700 1 --2- 172.21.15.42:0/3474703948 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa15c0a7730 0x7fa15c08df70 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fa144001110 tx=0x7fa144001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:51.667 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.656+0000 7fa15b59e700 1 --2- 172.21.15.42:0/3474703948 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa15c0a7730 0x7fa15c08df70 secure :-1 s=READY pgs=47 cs=0 l=1 rx=0x7fa144001110 tx=0x7fa144001150).ready entity=mon.2 client_cookie=89bb21659230f0a9 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:51.667 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.656+0000 7fa15b59e700 1 --2- 172.21.15.42:0/3474703948 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa15c0a7730 0x7fa15c08df70 secure :-1 s=THROTTLE_DONE pgs=47 cs=0 l=1 rx=0x7fa144001110 tx=0x7fa144001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:51.667 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.656+0000 7fa15a59c700 1 -- 172.21.15.42:0/3474703948 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa14c010870 con 0x7fa15c0a7730 2019-12-05T05:03:51.667 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.656+0000 7fa15b59e700 1 --2- 172.21.15.42:0/3474703948 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa15c0a7730 0x7fa15c08df70 secure :-1 s=THROTTLE_DONE pgs=47 cs=0 l=1 rx=0x7fa144001110 tx=0x7fa144001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:51.667 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.656+0000 7fa15b59e700 1 --2- 172.21.15.42:0/3474703948 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa15c0a7730 0x7fa15c08df70 secure :-1 s=THROTTLE_DONE pgs=47 cs=0 l=1 rx=0x7fa144001110 tx=0x7fa144001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:51.667 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.656+0000 7fa15a59c700 1 -- 172.21.15.42:0/3474703948 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa14c008b50 con 0x7fa15c0a7730 2019-12-05T05:03:51.668 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.656+0000 7fa15a59c700 1 -- 172.21.15.42:0/3474703948 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa14c0110c0 con 0x7fa15c0a7730 2019-12-05T05:03:51.668 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.656+0000 7fa161131700 1 -- 172.21.15.42:0/3474703948 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa15c0a7730 msgr2=0x7fa15c08df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:51.668 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.656+0000 7fa161131700 1 --2- 172.21.15.42:0/3474703948 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa15c0a7730 0x7fa15c08df70 secure :-1 s=READY pgs=47 cs=0 l=1 rx=0x7fa144001110 tx=0x7fa144001150).stop 2019-12-05T05:03:51.668 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.656+0000 7fa161131700 1 -- 172.21.15.42:0/3474703948 shutdown_connections 2019-12-05T05:03:51.668 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.656+0000 7fa161131700 1 --2- 172.21.15.42:0/3474703948 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa15c0a7730 0x7fa15c08df70 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:51.668 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.656+0000 7fa161131700 1 -- 172.21.15.42:0/3474703948 shutdown_connections 2019-12-05T05:03:51.668 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.656+0000 7fa161131700 1 -- 172.21.15.42:0/3474703948 wait complete. 2019-12-05T05:03:51.669 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.656+0000 7fa161131700 1 -- 172.21.15.42:0/3474703948 >> 172.21.15.42:0/3474703948 conn(0x7fa15c0aabd0 msgr2=0x7fa15c0a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:51.672 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.660+0000 7fa161131700 1 Processor -- start 2019-12-05T05:03:51.672 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.660+0000 7fa161131700 1 -- start start 2019-12-05T05:03:51.672 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.660+0000 7fa161131700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fa15c08f790 0x7fa15c0a3870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:51.672 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.660+0000 7fa161131700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa15c11f2f0 0x7fa15c11f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:51.672 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.660+0000 7fa161131700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa15c11a670 con 0x7fa15c124310 2019-12-05T05:03:51.673 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.660+0000 7fa161131700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa15c11b340 con 0x7fa15c11f2f0 2019-12-05T05:03:51.673 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.660+0000 7fa161131700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa15c11b5e0 con 0x7fa15c08f790 2019-12-05T05:03:51.673 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.660+0000 7fa15b59e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fa15c08f790 0x7fa15c0a3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:51.673 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.660+0000 7fa15b59e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fa15c08f790 0x7fa15c0a3870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:53914/0 (socket says 172.21.15.42:53914) 2019-12-05T05:03:51.673 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.660+0000 7fa15b59e700 1 -- 172.21.15.42:0/4067501569 learned_addr learned my addr 172.21.15.42:0/4067501569 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:03:51.673 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.660+0000 7fa15ad9d700 1 --2- 172.21.15.42:0/4067501569 >> v2:172.21.15.40:3300/0 conn(0x7fa15c11f2f0 0x7fa15c11f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:51.673 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.660+0000 7fa15b59e700 1 -- 172.21.15.42:0/4067501569 >> v2:172.21.15.40:3300/0 conn(0x7fa15c11f2f0 msgr2=0x7fa15c11f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:51.674 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.660+0000 7fa15b59e700 1 --2- 172.21.15.42:0/4067501569 >> v2:172.21.15.40:3300/0 conn(0x7fa15c11f2f0 0x7fa15c11f720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:51.674 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.660+0000 7fa15b59e700 1 -- 172.21.15.42:0/4067501569 >> v1:172.21.15.40:6789/0 conn(0x7fa15c124310 legacy=0x7fa15c124740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:51.674 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.660+0000 7fa15b59e700 1 -- 172.21.15.42:0/4067501569 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa15c1314f0 con 0x7fa15c08f790 2019-12-05T05:03:51.674 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.660+0000 7fa153fff700 1 -- 172.21.15.42:0/4067501569 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1111660652 0 0) 0x7fa15c11a670 con 0x7fa15c124310 2019-12-05T05:03:51.674 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.660+0000 7fa15b59e700 1 --2- 172.21.15.42:0/4067501569 >> v2:172.21.15.42:3300/0 conn(0x7fa15c08f790 0x7fa15c0a3870 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fa15c08eb90 tx=0x7fa14c010f10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:51.674 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.664+0000 7fa15b59e700 1 --2- 172.21.15.42:0/4067501569 >> v2:172.21.15.42:3300/0 conn(0x7fa15c08f790 0x7fa15c0a3870 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fa15c08eb90 tx=0x7fa14c010f10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:51.674 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.664+0000 7fa15b59e700 1 --2- 172.21.15.42:0/4067501569 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa15c08f790 0x7fa15c0a3870 secure :-1 s=READY pgs=199 cs=0 l=1 rx=0x7fa15c08eb90 tx=0x7fa14c010f10).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:51.675 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.664+0000 7fa15b59e700 1 --2- 172.21.15.42:0/4067501569 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa15c08f790 0x7fa15c0a3870 secure :-1 s=THROTTLE_DONE pgs=199 cs=0 l=1 rx=0x7fa15c08eb90 tx=0x7fa14c010f10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:51.675 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.664+0000 7fa153fff700 1 -- 172.21.15.42:0/4067501569 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa14c005ab0 con 0x7fa15c08f790 2019-12-05T05:03:51.675 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.664+0000 7fa15b59e700 1 --2- 172.21.15.42:0/4067501569 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa15c08f790 0x7fa15c0a3870 secure :-1 s=THROTTLE_DONE pgs=199 cs=0 l=1 rx=0x7fa15c08eb90 tx=0x7fa14c010f10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:51.675 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.664+0000 7fa15b59e700 1 --2- 172.21.15.42:0/4067501569 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa15c08f790 0x7fa15c0a3870 secure :-1 s=THROTTLE_DONE pgs=199 cs=0 l=1 rx=0x7fa15c08eb90 tx=0x7fa14c010f10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:51.675 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.664+0000 7fa153fff700 1 -- 172.21.15.42:0/4067501569 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa14c010a70 con 0x7fa15c08f790 2019-12-05T05:03:51.675 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.664+0000 7fa161131700 1 -- 172.21.15.42:0/4067501569 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa15c11aee0 con 0x7fa15c08f790 2019-12-05T05:03:51.675 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.664+0000 7fa153fff700 1 -- 172.21.15.42:0/4067501569 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa14c00f680 con 0x7fa15c08f790 2019-12-05T05:03:51.676 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.664+0000 7fa161131700 1 -- 172.21.15.42:0/4067501569 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa15c078530 con 0x7fa15c08f790 2019-12-05T05:03:51.676 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.664+0000 7fa15b59e700 1 --2- 172.21.15.42:0/4067501569 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa15c08f790 0x7fa15c0a3870 secure :-1 s=THROTTLE_DONE pgs=199 cs=0 l=1 rx=0x7fa15c08eb90 tx=0x7fa14c010f10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:51.676 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.664+0000 7fa15b59e700 1 --2- 172.21.15.42:0/4067501569 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa15c08f790 0x7fa15c0a3870 secure :-1 s=THROTTLE_DONE pgs=199 cs=0 l=1 rx=0x7fa15c08eb90 tx=0x7fa14c010f10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:51.676 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.664+0000 7fa153fff700 1 -- 172.21.15.42:0/4067501569 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fa14c011360 con 0x7fa15c08f790 2019-12-05T05:03:51.676 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.664+0000 7fa153fff700 1 --2- 172.21.15.42:0/4067501569 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa14003ace0 0x7fa14003d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:51.676 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.664+0000 7fa153fff700 1 -- 172.21.15.42:0/4067501569 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6031+0+0 (secure 0 0 0) 0x7fa14c04b1e0 con 0x7fa15c08f790 2019-12-05T05:03:51.676 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.664+0000 7fa15ad9d700 1 --2- 172.21.15.42:0/4067501569 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa14003ace0 0x7fa14003d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:51.677 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.664+0000 7fa15ad9d700 1 --2- 172.21.15.42:0/4067501569 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa14003ace0 0x7fa14003d1f0 crc :-1 s=READY pgs=105 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:51.677 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.668+0000 7fa161131700 1 -- 172.21.15.42:0/4067501569 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa15c05d100 con 0x7fa15c08f790 2019-12-05T05:03:51.681 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.676+0000 7fa15b59e700 1 --2- 172.21.15.42:0/4067501569 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa15c08f790 0x7fa15c0a3870 secure :-1 s=THROTTLE_DONE pgs=199 cs=0 l=1 rx=0x7fa15c08eb90 tx=0x7fa14c010f10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:51.682 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.676+0000 7fa153fff700 1 -- 172.21.15.42:0/4067501569 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fa14c011c60 con 0x7fa15c08f790 2019-12-05T05:03:51.843 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.836+0000 7fa161131700 1 -- 172.21.15.42:0/4067501569 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa15c05b8b0 con 0x7fa15c08f790 2019-12-05T05:03:51.844 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.840+0000 7fa15b59e700 1 --2- 172.21.15.42:0/4067501569 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa15c08f790 0x7fa15c0a3870 secure :-1 s=THROTTLE_DONE pgs=199 cs=0 l=1 rx=0x7fa15c08eb90 tx=0x7fa14c010f10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:51.844 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.840+0000 7fa153fff700 1 -- 172.21.15.42:0/4067501569 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+2102 (secure 0 0 0) 0x7fa14c00f990 con 0x7fa15c08f790 2019-12-05T05:03:51.845 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 6 2019-12-05T05:03:51.845 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:03:51.845 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4354,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/2019200199","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2019200199},{"type":"v1","addr":"172.21.15.42:6835","nonce":2019200199}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":4}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:03:43.669739+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4341},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4341":{"gid":4341,"name":"b","rank":0,"incarnation":5,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6833/3296198322","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3296198322},{"type":"v1","addr":"172.21.15.40:6833","nonce":3296198322}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:03:51.847 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.840+0000 7fa161131700 1 -- 172.21.15.42:0/4067501569 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa14003ace0 msgr2=0x7fa14003d1f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:51.848 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.840+0000 7fa161131700 1 --2- 172.21.15.42:0/4067501569 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa14003ace0 0x7fa14003d1f0 crc :-1 s=READY pgs=105 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:51.848 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.840+0000 7fa161131700 1 -- 172.21.15.42:0/4067501569 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa15c08f790 msgr2=0x7fa15c0a3870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:51.848 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.840+0000 7fa161131700 1 --2- 172.21.15.42:0/4067501569 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa15c08f790 0x7fa15c0a3870 secure :-1 s=READY pgs=199 cs=0 l=1 rx=0x7fa15c08eb90 tx=0x7fa14c010f10).stop 2019-12-05T05:03:51.848 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.840+0000 7fa161131700 1 -- 172.21.15.42:0/4067501569 shutdown_connections 2019-12-05T05:03:51.849 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.840+0000 7fa161131700 1 --2- 172.21.15.42:0/4067501569 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa14003ace0 0x7fa14003d1f0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:51.849 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.840+0000 7fa161131700 1 --2- 172.21.15.42:0/4067501569 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa15c08f790 0x7fa15c0a3870 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:51.849 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.840+0000 7fa161131700 1 --2- 172.21.15.42:0/4067501569 >> v2:172.21.15.40:3300/0 conn(0x7fa15c11f2f0 0x7fa15c11f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:51.849 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.840+0000 7fa161131700 1 -- 172.21.15.42:0/4067501569 shutdown_connections 2019-12-05T05:03:51.850 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.840+0000 7fa161131700 1 -- 172.21.15.42:0/4067501569 wait complete. 2019-12-05T05:03:51.850 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.840+0000 7fa161131700 1 -- 172.21.15.42:0/4067501569 >> 172.21.15.42:0/4067501569 conn(0x7fa15c0a7730 msgr2=0x7fa15c0a9780 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:51.857 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2019-12-05T05:03:51.964 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.944+0000 7f8a5c49b700 1 Processor -- start 2019-12-05T05:03:51.964 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.944+0000 7f8a5c49b700 1 -- start start 2019-12-05T05:03:51.964 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.948+0000 7f8a5c49b700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f8a5408df80 0x7f8a5409f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:51.965 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.948+0000 7f8a5c49b700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a5409fc60 con 0x7f8a540a7740 2019-12-05T05:03:51.965 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.948+0000 7f8a5c49b700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a540a0710 con 0x7f8a5409f830 2019-12-05T05:03:51.965 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.948+0000 7f8a5c49b700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a540a0b00 con 0x7f8a5408df80 2019-12-05T05:03:51.965 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.948+0000 7f8a5a237700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f8a5408df80 0x7f8a5409f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:51.966 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.948+0000 7f8a5b239700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f8a5409f830 0x7f8a5414efe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:53964/0 (socket says 172.21.15.42:53964) 2019-12-05T05:03:51.966 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.948+0000 7f8a5b239700 1 -- 172.21.15.42:0/2120529496 learned_addr learned my addr 172.21.15.42:0/2120529496 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:03:51.966 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.948+0000 7f8a5a237700 1 -- 172.21.15.42:0/2120529496 >> v1:172.21.15.42:6789/0 conn(0x7f8a5409f830 legacy=0x7f8a5414efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:51.966 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.948+0000 7f8a5a237700 1 -- 172.21.15.42:0/2120529496 >> v1:172.21.15.40:6789/0 conn(0x7f8a540a7740 legacy=0x7f8a5407e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:51.967 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.948+0000 7f8a59a36700 1 -- 172.21.15.42:0/2120529496 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3223553777 0 0) 0x7f8a540a0710 con 0x7f8a5409f830 2019-12-05T05:03:51.967 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.948+0000 7f8a5a237700 1 -- 172.21.15.42:0/2120529496 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a5407ef20 con 0x7f8a5408df80 2019-12-05T05:03:51.967 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.948+0000 7f8a5a237700 1 --2- 172.21.15.42:0/2120529496 >> v2:172.21.15.42:3300/0 conn(0x7f8a5408df80 0x7f8a5409f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f8a44008790 tx=0x7f8a4400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:51.967 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.948+0000 7f8a5a237700 1 --2- 172.21.15.42:0/2120529496 >> v2:172.21.15.42:3300/0 conn(0x7f8a5408df80 0x7f8a5409f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f8a44008790 tx=0x7f8a4400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:51.968 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.948+0000 7f8a5a237700 1 --2- 172.21.15.42:0/2120529496 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8a5408df80 0x7f8a5409f2e0 secure :-1 s=READY pgs=201 cs=0 l=1 rx=0x7f8a44008790 tx=0x7f8a4400fbf0).ready entity=mon.1 client_cookie=78aba746383931e7 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:51.968 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.948+0000 7f8a5a237700 1 --2- 172.21.15.42:0/2120529496 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8a5408df80 0x7f8a5409f2e0 secure :-1 s=THROTTLE_DONE pgs=201 cs=0 l=1 rx=0x7f8a44008790 tx=0x7f8a4400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:51.968 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.948+0000 7f8a59a36700 1 -- 172.21.15.42:0/2120529496 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8a44011270 con 0x7f8a5408df80 2019-12-05T05:03:51.968 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.948+0000 7f8a5a237700 1 --2- 172.21.15.42:0/2120529496 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8a5408df80 0x7f8a5409f2e0 secure :-1 s=THROTTLE_DONE pgs=201 cs=0 l=1 rx=0x7f8a44008790 tx=0x7f8a4400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:51.969 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.948+0000 7f8a59a36700 1 -- 172.21.15.42:0/2120529496 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a44008bc0 con 0x7f8a5408df80 2019-12-05T05:03:51.969 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.948+0000 7f8a5a237700 1 --2- 172.21.15.42:0/2120529496 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8a5408df80 0x7f8a5409f2e0 secure :-1 s=THROTTLE_DONE pgs=201 cs=0 l=1 rx=0x7f8a44008790 tx=0x7f8a4400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:51.969 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.948+0000 7f8a59a36700 1 -- 172.21.15.42:0/2120529496 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8a44011850 con 0x7f8a5408df80 2019-12-05T05:03:51.969 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.948+0000 7f8a5c49b700 1 -- 172.21.15.42:0/2120529496 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8a5408df80 msgr2=0x7f8a5409f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:51.969 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.948+0000 7f8a5c49b700 1 --2- 172.21.15.42:0/2120529496 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8a5408df80 0x7f8a5409f2e0 secure :-1 s=READY pgs=201 cs=0 l=1 rx=0x7f8a44008790 tx=0x7f8a4400fbf0).stop 2019-12-05T05:03:51.970 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.948+0000 7f8a5c49b700 1 -- 172.21.15.42:0/2120529496 shutdown_connections 2019-12-05T05:03:51.970 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.948+0000 7f8a5c49b700 1 --2- 172.21.15.42:0/2120529496 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8a5408df80 0x7f8a5409f2e0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:51.970 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.948+0000 7f8a5c49b700 1 -- 172.21.15.42:0/2120529496 shutdown_connections 2019-12-05T05:03:51.970 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.948+0000 7f8a5c49b700 1 -- 172.21.15.42:0/2120529496 wait complete. 2019-12-05T05:03:51.971 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.948+0000 7f8a5c49b700 1 -- 172.21.15.42:0/2120529496 >> 172.21.15.42:0/2120529496 conn(0x7f8a540aabe0 msgr2=0x7f8a540a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:51.971 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.952+0000 7f8a5c49b700 1 Processor -- start 2019-12-05T05:03:51.971 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.952+0000 7f8a5c49b700 1 -- start start 2019-12-05T05:03:51.971 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.952+0000 7f8a5c49b700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f8a54120db0 0x7f8a541387f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:51.972 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.952+0000 7f8a5c49b700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a5411c410 con 0x7f8a54116500 2019-12-05T05:03:51.972 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.952+0000 7f8a5c49b700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a541204f0 con 0x7f8a5411bfe0 2019-12-05T05:03:51.972 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.952+0000 7f8a5c49b700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a5411baa0 con 0x7f8a54120db0 2019-12-05T05:03:51.972 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.952+0000 7f8a5a237700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f8a5411bfe0 0x7f8a5407d920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:53968/0 (socket says 172.21.15.42:53968) 2019-12-05T05:03:51.973 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.952+0000 7f8a5a237700 1 -- 172.21.15.42:0/3662681280 learned_addr learned my addr 172.21.15.42:0/3662681280 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:03:51.973 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.952+0000 7f8a5b239700 1 --2- 172.21.15.42:0/3662681280 >> v2:172.21.15.42:3300/0 conn(0x7f8a54120db0 0x7f8a541387f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:51.973 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.952+0000 7f8a4b7fe700 1 -- 172.21.15.42:0/3662681280 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2520536476 0 0) 0x7f8a541204f0 con 0x7f8a5411bfe0 2019-12-05T05:03:51.973 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.952+0000 7f8a4b7fe700 1 -- 172.21.15.42:0/3662681280 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a3c0030b0 con 0x7f8a5411bfe0 2019-12-05T05:03:51.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.952+0000 7f8a4b7fe700 1 -- 172.21.15.42:0/3662681280 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3318657246 0 0) 0x7f8a5409fc60 con 0x7f8a54116500 2019-12-05T05:03:51.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.952+0000 7f8a4b7fe700 1 -- 172.21.15.42:0/3662681280 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a541204f0 con 0x7f8a54116500 2019-12-05T05:03:51.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.956+0000 7f8a5b239700 1 -- 172.21.15.42:0/3662681280 >> v1:172.21.15.42:6789/0 conn(0x7f8a5411bfe0 legacy=0x7f8a5407d920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:51.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.956+0000 7f8a4b7fe700 1 -- 172.21.15.42:0/3662681280 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 67623450 0 0) 0x7f8a3c0030b0 con 0x7f8a5411bfe0 2019-12-05T05:03:51.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.956+0000 7f8a5b239700 1 -- 172.21.15.42:0/3662681280 >> v1:172.21.15.40:6789/0 conn(0x7f8a54116500 legacy=0x7f8a5411ada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:51.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.956+0000 7f8a5b239700 1 -- 172.21.15.42:0/3662681280 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a5407e120 con 0x7f8a54120db0 2019-12-05T05:03:51.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.956+0000 7f8a5b239700 1 --2- 172.21.15.42:0/3662681280 >> v2:172.21.15.42:3300/0 conn(0x7f8a54120db0 0x7f8a541387f0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f8a44007700 tx=0x7f8a44010100).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:51.976 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.956+0000 7f8a5b239700 1 --2- 172.21.15.42:0/3662681280 >> v2:172.21.15.42:3300/0 conn(0x7f8a54120db0 0x7f8a541387f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f8a44007700 tx=0x7f8a44010100).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:51.976 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.956+0000 7f8a5b239700 1 --2- 172.21.15.42:0/3662681280 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8a54120db0 0x7f8a541387f0 secure :-1 s=READY pgs=203 cs=0 l=1 rx=0x7f8a44007700 tx=0x7f8a44010100).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:51.976 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.956+0000 7f8a5b239700 1 --2- 172.21.15.42:0/3662681280 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8a54120db0 0x7f8a541387f0 secure :-1 s=THROTTLE_DONE pgs=203 cs=0 l=1 rx=0x7f8a44007700 tx=0x7f8a44010100).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:51.976 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.956+0000 7f8a4b7fe700 1 -- 172.21.15.42:0/3662681280 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8a44011270 con 0x7f8a54120db0 2019-12-05T05:03:51.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.956+0000 7f8a5b239700 1 --2- 172.21.15.42:0/3662681280 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8a54120db0 0x7f8a541387f0 secure :-1 s=THROTTLE_DONE pgs=203 cs=0 l=1 rx=0x7f8a44007700 tx=0x7f8a44010100).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:51.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.956+0000 7f8a5b239700 1 --2- 172.21.15.42:0/3662681280 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8a54120db0 0x7f8a541387f0 secure :-1 s=THROTTLE_DONE pgs=203 cs=0 l=1 rx=0x7f8a44007700 tx=0x7f8a44010100).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:51.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.956+0000 7f8a4b7fe700 1 -- 172.21.15.42:0/3662681280 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a44011470 con 0x7f8a54120db0 2019-12-05T05:03:51.978 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.956+0000 7f8a5c49b700 1 -- 172.21.15.42:0/3662681280 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a541211e0 con 0x7f8a54120db0 2019-12-05T05:03:51.978 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.956+0000 7f8a4b7fe700 1 -- 172.21.15.42:0/3662681280 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8a4c011670 con 0x7f8a54120db0 2019-12-05T05:03:51.978 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.956+0000 7f8a5c49b700 1 -- 172.21.15.42:0/3662681280 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a54138d40 con 0x7f8a54120db0 2019-12-05T05:03:51.978 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.956+0000 7f8a5b239700 1 --2- 172.21.15.42:0/3662681280 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8a54120db0 0x7f8a541387f0 secure :-1 s=THROTTLE_DONE pgs=203 cs=0 l=1 rx=0x7f8a44007700 tx=0x7f8a44010100).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:51.979 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.956+0000 7f8a4b7fe700 1 -- 172.21.15.42:0/3662681280 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f8a4c011980 con 0x7f8a54120db0 2019-12-05T05:03:51.979 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.956+0000 7f8a5b239700 1 --2- 172.21.15.42:0/3662681280 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8a54120db0 0x7f8a541387f0 secure :-1 s=THROTTLE_DONE pgs=203 cs=0 l=1 rx=0x7f8a44007700 tx=0x7f8a44010100).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:51.979 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.956+0000 7f8a4b7fe700 1 --2- 172.21.15.42:0/3662681280 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8a3c03b420 0x7f8a3c03d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:51.979 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.956+0000 7f8a4b7fe700 1 -- 172.21.15.42:0/3662681280 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f8a4c04abf0 con 0x7f8a54120db0 2019-12-05T05:03:51.980 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.956+0000 7f8a5aa38700 1 --2- 172.21.15.42:0/3662681280 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8a3c03b420 0x7f8a3c03d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:51.980 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.960+0000 7f8a5aa38700 1 --2- 172.21.15.42:0/3662681280 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8a3c03b420 0x7f8a3c03d930 crc :-1 s=READY pgs=106 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:51.981 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.960+0000 7f8a5c49b700 1 -- 172.21.15.42:0/3662681280 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a5405d110 con 0x7f8a54120db0 2019-12-05T05:03:51.981 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.972+0000 7f8a5b239700 1 --2- 172.21.15.42:0/3662681280 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8a54120db0 0x7f8a541387f0 secure :-1 s=THROTTLE_DONE pgs=203 cs=0 l=1 rx=0x7f8a44007700 tx=0x7f8a44010100).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:51.981 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:51.972+0000 7f8a4b7fe700 1 -- 172.21.15.42:0/3662681280 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f8a4c031c70 con 0x7f8a54120db0 2019-12-05T05:03:52.146 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.140+0000 7f8a5c49b700 1 -- 172.21.15.42:0/3662681280 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8a5405b8c0 con 0x7f8a54120db0 2019-12-05T05:03:52.147 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.144+0000 7f8a5b239700 1 --2- 172.21.15.42:0/3662681280 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8a54120db0 0x7f8a541387f0 secure :-1 s=THROTTLE_DONE pgs=203 cs=0 l=1 rx=0x7f8a44007700 tx=0x7f8a44010100).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:52.148 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.144+0000 7f8a4b7fe700 1 -- 172.21.15.42:0/3662681280 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+14885 (secure 0 0 0) 0x7f8a4c031ed0 con 0x7f8a54120db0 2019-12-05T05:03:52.149 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:03:52.149 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":24,"fsid":"5d836cfc-a83b-4433-9b3e-b9b2f31c9050","created":"2019-12-05T05:03:13.215602+0000","modified":"2019-12-05T05:03:36.607704+0000","last_up_change":"2019-12-05T05:03:22.664783+0000","last_in_change":"2019-12-05T05:03:22.664783+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2019-12-05T05:03:27.394605+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2019-12-05T05:03:30.419845+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2019-12-05T05:03:33.249075+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e3fc49bb-dcbc-43ff-ac7c-9eb0ea0bd257","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6810","nonce":12118},{"type":"v1","addr":"172.21.15.42:6811","nonce":12118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6812","nonce":12118},{"type":"v1","addr":"172.21.15.42:6813","nonce":12118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6816","nonce":12118},{"type":"v1","addr":"172.21.15.42:6817","nonce":12118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6814","nonce":12118},{"type":"v1","addr":"172.21.15.42:6815","nonce":12118}]},"public_addr":"172.21.15.42:6811/12118","cluster_addr":"172.21.15.42:6813/12118","heartbeat_back_addr":"172.21.15.42:6817/12118","heartbeat_front_addr":"172.21.15.42:6815/12118","state":["exists","up"]},{"osd":1,"uuid":"902543fb-f5a2-4321-9c9c-26f9b9ca8629","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6826","nonce":12119},{"type":"v1","addr":"172.21.15.42:6827","nonce":12119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6828","nonce":12119},{"type":"v1","addr":"172.21.15.42:6829","nonce":12119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6832","nonce":12119},{"type":"v1","addr":"172.21.15.42:6833","nonce":12119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6830","nonce":12119},{"type":"v1","addr":"172.21.15.42:6831","nonce":12119}]},"public_addr":"172.21.15.42:6827/12119","cluster_addr":"172.21.15.42:6829/12119","heartbeat_back_addr":"172.21.15.42:6833/12119","heartbeat_front_addr":"172.21.15.42:6831/12119","state":["exists","up"]},{"osd":2,"uuid":"28739ff7-bed7-4b75-8ea5-b57d088f1250","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6802","nonce":12117},{"type":"v1","addr":"172.21.15.42:6803","nonce":12117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6804","nonce":12117},{"type":"v1","addr":"172.21.15.42:6805","nonce":12117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6808","nonce":12117},{"type":"v1","addr":"172.21.15.42:6809","nonce":12117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6806","nonce":12117},{"type":"v1","addr":"172.21.15.42:6807","nonce":12117}]},"public_addr":"172.21.15.42:6803/12117","cluster_addr":"172.21.15.42:6805/12117","heartbeat_back_addr":"172.21.15.42:6809/12117","heartbeat_front_addr":"172.21.15.42:6807/12117","state":["exists","up"]},{"osd":3,"uuid":"1960f66b-2d40-4bb9-a289-ce8cd0c1c142","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6818","nonce":12116},{"type":"v1","addr":"172.21.15.42:6819","nonce":12116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6820","nonce":12116},{"type":"v1","addr":"172.21.15.42:6821","nonce":12116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6824","nonce":12116},{"type":"v1","addr":"172.21.15.42:6825","nonce":12116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6822","nonce":12116},{"type":"v1","addr":"172.21.15.42:6823","nonce":12116}]},"public_addr":"172.21.15.42:6819/12116","cluster_addr":"172.21.15.42:6821/12116","heartbeat_back_addr":"172.21.15.42:6825/12116","heartbeat_front_addr":"172.21.15.42:6823/12116","state":["exists","up"]},{"osd":4,"uuid":"45bbf1bd-7c8a-4a0d-b76b-afa6148a855c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":12488},{"type":"v1","addr":"172.21.15.40:6809","nonce":12488}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":12488},{"type":"v1","addr":"172.21.15.40:6812","nonce":12488}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":12488},{"type":"v1","addr":"172.21.15.40:6825","nonce":12488}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":12488},{"type":"v1","addr":"172.21.15.40:6819","nonce":12488}]},"public_addr":"172.21.15.40:6809/12488","cluster_addr":"172.21.15.40:6812/12488","heartbeat_back_addr":"172.21.15.40:6825/12488","heartbeat_front_addr":"172.21.15.40:6819/12488","state":["exists","up"]},{"osd":5,"uuid":"8a5511cd-925f-4869-a49e-924383e4ce53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":12489},{"type":"v1","addr":"172.21.15.40:6801","nonce":12489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":12489},{"type":"v1","addr":"172.21.15.40:6803","nonce":12489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":12489},{"type":"v1","addr":"172.21.15.40:6807","nonce":12489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":12489},{"type":"v1","addr":"172.21.15.40:6805","nonce":12489}]},"public_addr":"172.21.15.40:6801/12489","cluster_addr":"172.21.15.40:6803/12489","heartbeat_back_addr":"172.21.15.40:6807/12489","heartbeat_front_addr":"172.21.15.40:6805/12489","state":["exists","up"]},{"osd":6,"uuid":"d692c179-379e-446c-8484-a219a9241e04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6811","nonce":12491},{"type":"v1","addr":"172.21.15.40:6814","nonce":12491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6817","nonce":12491},{"type":"v1","addr":"172.21.15.40:6818","nonce":12491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":12491},{"type":"v1","addr":"172.21.15.40:6828","nonce":12491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":12491},{"type":"v1","addr":"172.21.15.40:6824","nonce":12491}]},"public_addr":"172.21.15.40:6814/12491","cluster_addr":"172.21.15.40:6818/12491","heartbeat_back_addr":"172.21.15.40:6828/12491","heartbeat_front_addr":"172.21.15.40:6824/12491","state":["exists","up"]},{"osd":7,"uuid":"79c3c243-190e-4e35-b769-319a2077e7cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6813","nonce":12490},{"type":"v1","addr":"172.21.15.40:6816","nonce":12490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":12490},{"type":"v1","addr":"172.21.15.40:6823","nonce":12490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":12490},{"type":"v1","addr":"172.21.15.40:6831","nonce":12490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6827","nonce":12490},{"type":"v1","addr":"172.21.15.40:6829","nonce":12490}]},"public_addr":"172.21.15.40:6816/12490","cluster_addr":"172.21.15.40:6823/12490","heartbeat_back_addr":"172.21.15.40:6831/12490","heartbeat_front_addr":"172.21.15.40:6829/12490","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T05:03:21.349150+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T05:03:21.644281+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T05:03:21.394040+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T05:03:21.587096+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T05:03:21.656497+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T05:03:21.510547+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T05:03:21.659358+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T05:03:21.628063+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2019-12-05T05:03:52.151 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.144+0000 7f8a5c49b700 1 -- 172.21.15.42:0/3662681280 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8a3c03b420 msgr2=0x7f8a3c03d930 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:52.151 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.144+0000 7f8a5c49b700 1 --2- 172.21.15.42:0/3662681280 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8a3c03b420 0x7f8a3c03d930 crc :-1 s=READY pgs=106 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:52.152 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.144+0000 7f8a5c49b700 1 -- 172.21.15.42:0/3662681280 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8a54120db0 msgr2=0x7f8a541387f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:52.152 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.144+0000 7f8a5c49b700 1 --2- 172.21.15.42:0/3662681280 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8a54120db0 0x7f8a541387f0 secure :-1 s=READY pgs=203 cs=0 l=1 rx=0x7f8a44007700 tx=0x7f8a44010100).stop 2019-12-05T05:03:52.152 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.144+0000 7f8a5c49b700 1 -- 172.21.15.42:0/3662681280 shutdown_connections 2019-12-05T05:03:52.153 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.144+0000 7f8a5c49b700 1 --2- 172.21.15.42:0/3662681280 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8a3c03b420 0x7f8a3c03d930 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:52.153 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.144+0000 7f8a5c49b700 1 --2- 172.21.15.42:0/3662681280 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8a54120db0 0x7f8a541387f0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:52.153 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.144+0000 7f8a5c49b700 1 -- 172.21.15.42:0/3662681280 shutdown_connections 2019-12-05T05:03:52.153 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.144+0000 7f8a5c49b700 1 -- 172.21.15.42:0/3662681280 wait complete. 2019-12-05T05:03:52.153 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.144+0000 7f8a5c49b700 1 -- 172.21.15.42:0/3662681280 >> 172.21.15.42:0/3662681280 conn(0x7f8a540a7f90 msgr2=0x7f8a5407e670 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:52.161 DEBUG:tasks.mds_thrash:Yielding 2019-12-05T05:03:52.161 INFO:teuthology.run_tasks:Running task ceph-fuse... 2019-12-05T05:03:52.180 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2019-12-05T05:03:52.180 INFO:tasks.ceph_fuse:config is None 2019-12-05T05:03:52.180 INFO:tasks.ceph_fuse:new config is {'client.0': None} 2019-12-05T05:03:52.180 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2019-12-05T05:03:52.181 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2019-12-05T05:03:52.181 INFO:tasks.mds_thrash.fs.[cephfs]:starting mds_do_thrash for fs cephfs 2019-12-05T05:03:52.181 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 36 secs before thrashing 2019-12-05T05:03:52.279 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.268+0000 7ffb403af700 1 Processor -- start 2019-12-05T05:03:52.279 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.268+0000 7ffb403af700 1 -- start start 2019-12-05T05:03:52.279 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.268+0000 7ffb403af700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ffb380a7740 0x7ffb3808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:52.279 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.268+0000 7ffb403af700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ffb3809fae0 0x7ffb3807eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:52.279 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.268+0000 7ffb403af700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb380a0710 con 0x7ffb3807e720 2019-12-05T05:03:52.280 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.268+0000 7ffb403af700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb380a0b00 con 0x7ffb3809fae0 2019-12-05T05:03:52.280 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.268+0000 7ffb403af700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb3808e5b0 con 0x7ffb380a7740 2019-12-05T05:03:52.280 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.272+0000 7ffb3e94c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ffb380a7740 0x7ffb3808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:52.280 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.272+0000 7ffb3e94c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ffb380a7740 0x7ffb3808df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:53936/0 (socket says 172.21.15.42:53936) 2019-12-05T05:03:52.280 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.272+0000 7ffb3e94c700 1 -- 172.21.15.42:0/666601116 learned_addr learned my addr 172.21.15.42:0/666601116 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:03:52.280 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.272+0000 7ffb3f14d700 1 --2- 172.21.15.42:0/666601116 >> v2:172.21.15.40:3300/0 conn(0x7ffb3809fae0 0x7ffb3807eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:52.281 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.272+0000 7ffb3e94c700 1 -- 172.21.15.42:0/666601116 >> v2:172.21.15.40:3300/0 conn(0x7ffb3809fae0 msgr2=0x7ffb3807eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:03:52.281 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.272+0000 7ffb3e94c700 1 --2- 172.21.15.42:0/666601116 >> v2:172.21.15.40:3300/0 conn(0x7ffb3809fae0 0x7ffb3807eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:03:52.281 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.272+0000 7ffb3e94c700 1 -- 172.21.15.42:0/666601116 >> v1:172.21.15.40:6789/0 conn(0x7ffb3807e720 legacy=0x7ffb3809f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:52.281 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.272+0000 7ffb3e94c700 1 -- 172.21.15.42:0/666601116 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb3807f0a0 con 0x7ffb380a7740 2019-12-05T05:03:52.281 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.272+0000 7ffb3e94c700 1 --2- 172.21.15.42:0/666601116 >> v2:172.21.15.42:3300/0 conn(0x7ffb380a7740 0x7ffb3808df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7ffb34002f30 tx=0x7ffb3400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:52.281 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.272+0000 7ffb3e94c700 1 --2- 172.21.15.42:0/666601116 >> v2:172.21.15.42:3300/0 conn(0x7ffb380a7740 0x7ffb3808df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7ffb34002f30 tx=0x7ffb3400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:52.281 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.272+0000 7ffb3e94c700 1 --2- 172.21.15.42:0/666601116 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ffb380a7740 0x7ffb3808df80 secure :-1 s=READY pgs=204 cs=0 l=1 rx=0x7ffb34002f30 tx=0x7ffb3400fd30).ready entity=mon.1 client_cookie=360fca70b529126b server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:52.282 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.272+0000 7ffb3e94c700 1 --2- 172.21.15.42:0/666601116 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ffb380a7740 0x7ffb3808df80 secure :-1 s=THROTTLE_DONE pgs=204 cs=0 l=1 rx=0x7ffb34002f30 tx=0x7ffb3400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:52.282 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.272+0000 7ffb3e94c700 1 --2- 172.21.15.42:0/666601116 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ffb380a7740 0x7ffb3808df80 secure :-1 s=THROTTLE_DONE pgs=204 cs=0 l=1 rx=0x7ffb34002f30 tx=0x7ffb3400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:52.282 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.272+0000 7ffb3d94a700 1 -- 172.21.15.42:0/666601116 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ffb34008c90 con 0x7ffb380a7740 2019-12-05T05:03:52.282 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.272+0000 7ffb3e94c700 1 --2- 172.21.15.42:0/666601116 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ffb380a7740 0x7ffb3808df80 secure :-1 s=THROTTLE_DONE pgs=204 cs=0 l=1 rx=0x7ffb34002f30 tx=0x7ffb3400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:52.282 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.272+0000 7ffb3d94a700 1 -- 172.21.15.42:0/666601116 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb340088b0 con 0x7ffb380a7740 2019-12-05T05:03:52.282 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.272+0000 7ffb3d94a700 1 -- 172.21.15.42:0/666601116 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ffb34011ad0 con 0x7ffb380a7740 2019-12-05T05:03:52.282 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.272+0000 7ffb403af700 1 -- 172.21.15.42:0/666601116 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ffb380a7740 msgr2=0x7ffb3808df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:52.283 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.272+0000 7ffb403af700 1 --2- 172.21.15.42:0/666601116 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ffb380a7740 0x7ffb3808df80 secure :-1 s=READY pgs=204 cs=0 l=1 rx=0x7ffb34002f30 tx=0x7ffb3400fd30).stop 2019-12-05T05:03:52.283 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.272+0000 7ffb403af700 1 -- 172.21.15.42:0/666601116 shutdown_connections 2019-12-05T05:03:52.283 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.272+0000 7ffb403af700 1 --2- 172.21.15.42:0/666601116 >> v2:172.21.15.40:3300/0 conn(0x7ffb3809fae0 0x7ffb3807eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:03:52.283 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.272+0000 7ffb403af700 1 --2- 172.21.15.42:0/666601116 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ffb380a7740 0x7ffb3808df80 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:52.283 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.272+0000 7ffb403af700 1 -- 172.21.15.42:0/666601116 shutdown_connections 2019-12-05T05:03:52.283 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.272+0000 7ffb403af700 1 -- 172.21.15.42:0/666601116 wait complete. 2019-12-05T05:03:52.283 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.272+0000 7ffb403af700 1 -- 172.21.15.42:0/666601116 >> 172.21.15.42:0/666601116 conn(0x7ffb380aabe0 msgr2=0x7ffb380a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:52.286 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.276+0000 7ffb403af700 1 Processor -- start 2019-12-05T05:03:52.287 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.276+0000 7ffb403af700 1 -- start start 2019-12-05T05:03:52.287 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.276+0000 7ffb403af700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ffb3812ad90 0x7ffb38065730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:52.287 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.276+0000 7ffb403af700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ffb380a64c0 0x7ffb380a68f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:52.287 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.276+0000 7ffb403af700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb380a6e40 con 0x7ffb3808ee90 2019-12-05T05:03:52.287 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.276+0000 7ffb403af700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb38063530 con 0x7ffb3812ad90 2019-12-05T05:03:52.288 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.276+0000 7ffb403af700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb38063c30 con 0x7ffb380a64c0 2019-12-05T05:03:52.288 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.276+0000 7ffb3f14d700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ffb380a64c0 0x7ffb380a68f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:52.288 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.276+0000 7ffb3e14b700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ffb3812ad90 0x7ffb38065730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:52.288 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.276+0000 7ffb3e14b700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ffb3812ad90 0x7ffb38065730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.42:41758/0 (socket says 172.21.15.42:41758) 2019-12-05T05:03:52.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.276+0000 7ffb3e94c700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7ffb3808ee90 0x7ffb3812a590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:35190/0 (socket says 172.21.15.42:35190) 2019-12-05T05:03:52.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.276+0000 7ffb3f14d700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ffb380a64c0 0x7ffb380a68f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:54412/0 (socket says 172.21.15.42:54412) 2019-12-05T05:03:52.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.276+0000 7ffb3e14b700 1 -- 172.21.15.42:0/590865946 learned_addr learned my addr 172.21.15.42:0/590865946 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:03:52.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.276+0000 7ffb3e14b700 1 -- 172.21.15.42:0/590865946 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ffb380a64c0 msgr2=0x7ffb380a68f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:52.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.276+0000 7ffb3e14b700 1 --2- 172.21.15.42:0/590865946 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ffb380a64c0 0x7ffb380a68f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:52.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.276+0000 7ffb3e14b700 1 -- 172.21.15.42:0/590865946 >> v1:172.21.15.40:6789/0 conn(0x7ffb3808ee90 legacy=0x7ffb3812a590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:52.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.276+0000 7ffb3e14b700 1 -- 172.21.15.42:0/590865946 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb38063800 con 0x7ffb3812ad90 2019-12-05T05:03:52.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.276+0000 7ffb3e14b700 1 --2- 172.21.15.42:0/590865946 >> v2:172.21.15.40:3300/0 conn(0x7ffb3812ad90 0x7ffb38065730 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7ffb34007f10 tx=0x7ffb3400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:52.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.280+0000 7ffb3e14b700 1 --2- 172.21.15.42:0/590865946 >> v2:172.21.15.40:3300/0 conn(0x7ffb3812ad90 0x7ffb38065730 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7ffb34007f10 tx=0x7ffb3400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:52.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.280+0000 7ffb3e14b700 1 --2- 172.21.15.42:0/590865946 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffb3812ad90 0x7ffb38065730 secure :-1 s=READY pgs=173 cs=0 l=1 rx=0x7ffb34007f10 tx=0x7ffb3400fd30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:52.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.280+0000 7ffb3e14b700 1 --2- 172.21.15.42:0/590865946 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffb3812ad90 0x7ffb38065730 secure :-1 s=THROTTLE_DONE pgs=173 cs=0 l=1 rx=0x7ffb34007f10 tx=0x7ffb3400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:52.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.280+0000 7ffb2f7fe700 1 -- 172.21.15.42:0/590865946 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ffb2800c9b0 con 0x7ffb3812ad90 2019-12-05T05:03:52.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.280+0000 7ffb3e14b700 1 --2- 172.21.15.42:0/590865946 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffb3812ad90 0x7ffb38065730 secure :-1 s=THROTTLE_DONE pgs=173 cs=0 l=1 rx=0x7ffb34007f10 tx=0x7ffb3400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:52.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.280+0000 7ffb3e14b700 1 --2- 172.21.15.42:0/590865946 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffb3812ad90 0x7ffb38065730 secure :-1 s=THROTTLE_DONE pgs=173 cs=0 l=1 rx=0x7ffb34007f10 tx=0x7ffb3400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:52.291 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.280+0000 7ffb2f7fe700 1 -- 172.21.15.42:0/590865946 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb2800c5d0 con 0x7ffb3812ad90 2019-12-05T05:03:52.291 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.280+0000 7ffb403af700 1 -- 172.21.15.42:0/590865946 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb38131580 con 0x7ffb3812ad90 2019-12-05T05:03:52.291 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.280+0000 7ffb2f7fe700 1 -- 172.21.15.42:0/590865946 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ffb28010eb0 con 0x7ffb3812ad90 2019-12-05T05:03:52.291 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.280+0000 7ffb403af700 1 -- 172.21.15.42:0/590865946 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb38131b30 con 0x7ffb3812ad90 2019-12-05T05:03:52.291 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.280+0000 7ffb3e14b700 1 --2- 172.21.15.42:0/590865946 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffb3812ad90 0x7ffb38065730 secure :-1 s=THROTTLE_DONE pgs=173 cs=0 l=1 rx=0x7ffb34007f10 tx=0x7ffb3400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:52.291 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.280+0000 7ffb3e14b700 1 --2- 172.21.15.42:0/590865946 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffb3812ad90 0x7ffb38065730 secure :-1 s=THROTTLE_DONE pgs=173 cs=0 l=1 rx=0x7ffb34007f10 tx=0x7ffb3400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:52.291 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.280+0000 7ffb2f7fe700 1 -- 172.21.15.42:0/590865946 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7ffb28007430 con 0x7ffb3812ad90 2019-12-05T05:03:52.292 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.280+0000 7ffb2f7fe700 1 --2- 172.21.15.42:0/590865946 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ffb2003ade0 0x7ffb2003d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:52.292 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.280+0000 7ffb2f7fe700 1 -- 172.21.15.42:0/590865946 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6031+0+0 (secure 0 0 0) 0x7ffb2804b190 con 0x7ffb3812ad90 2019-12-05T05:03:52.292 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.280+0000 7ffb3e94c700 1 --2- 172.21.15.42:0/590865946 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ffb2003ade0 0x7ffb2003d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:52.293 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.280+0000 7ffb3e94c700 1 --2- 172.21.15.42:0/590865946 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ffb2003ade0 0x7ffb2003d2f0 crc :-1 s=READY pgs=107 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:52.293 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.284+0000 7ffb403af700 1 -- 172.21.15.42:0/590865946 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb3805d110 con 0x7ffb3812ad90 2019-12-05T05:03:52.297 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.292+0000 7ffb3e14b700 1 --2- 172.21.15.42:0/590865946 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffb3812ad90 0x7ffb38065730 secure :-1 s=THROTTLE_DONE pgs=173 cs=0 l=1 rx=0x7ffb34007f10 tx=0x7ffb3400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:52.298 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.292+0000 7ffb2f7fe700 1 -- 172.21.15.42:0/590865946 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7ffb280321d0 con 0x7ffb3812ad90 2019-12-05T05:03:52.473 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.468+0000 7ffb403af700 1 -- 172.21.15.42:0/590865946 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffb3805b8c0 con 0x7ffb3812ad90 2019-12-05T05:03:52.475 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.468+0000 7ffb3e14b700 1 --2- 172.21.15.42:0/590865946 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffb3812ad90 0x7ffb38065730 secure :-1 s=THROTTLE_DONE pgs=173 cs=0 l=1 rx=0x7ffb34007f10 tx=0x7ffb3400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:52.475 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.468+0000 7ffb2f7fe700 1 -- 172.21.15.42:0/590865946 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+14885 (secure 0 0 0) 0x7ffb28032430 con 0x7ffb3812ad90 2019-12-05T05:03:52.476 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:03:52.477 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":24,"fsid":"5d836cfc-a83b-4433-9b3e-b9b2f31c9050","created":"2019-12-05T05:03:13.215602+0000","modified":"2019-12-05T05:03:36.607704+0000","last_up_change":"2019-12-05T05:03:22.664783+0000","last_in_change":"2019-12-05T05:03:22.664783+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2019-12-05T05:03:27.394605+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2019-12-05T05:03:30.419845+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2019-12-05T05:03:33.249075+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e3fc49bb-dcbc-43ff-ac7c-9eb0ea0bd257","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6810","nonce":12118},{"type":"v1","addr":"172.21.15.42:6811","nonce":12118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6812","nonce":12118},{"type":"v1","addr":"172.21.15.42:6813","nonce":12118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6816","nonce":12118},{"type":"v1","addr":"172.21.15.42:6817","nonce":12118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6814","nonce":12118},{"type":"v1","addr":"172.21.15.42:6815","nonce":12118}]},"public_addr":"172.21.15.42:6811/12118","cluster_addr":"172.21.15.42:6813/12118","heartbeat_back_addr":"172.21.15.42:6817/12118","heartbeat_front_addr":"172.21.15.42:6815/12118","state":["exists","up"]},{"osd":1,"uuid":"902543fb-f5a2-4321-9c9c-26f9b9ca8629","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6826","nonce":12119},{"type":"v1","addr":"172.21.15.42:6827","nonce":12119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6828","nonce":12119},{"type":"v1","addr":"172.21.15.42:6829","nonce":12119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6832","nonce":12119},{"type":"v1","addr":"172.21.15.42:6833","nonce":12119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6830","nonce":12119},{"type":"v1","addr":"172.21.15.42:6831","nonce":12119}]},"public_addr":"172.21.15.42:6827/12119","cluster_addr":"172.21.15.42:6829/12119","heartbeat_back_addr":"172.21.15.42:6833/12119","heartbeat_front_addr":"172.21.15.42:6831/12119","state":["exists","up"]},{"osd":2,"uuid":"28739ff7-bed7-4b75-8ea5-b57d088f1250","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6802","nonce":12117},{"type":"v1","addr":"172.21.15.42:6803","nonce":12117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6804","nonce":12117},{"type":"v1","addr":"172.21.15.42:6805","nonce":12117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6808","nonce":12117},{"type":"v1","addr":"172.21.15.42:6809","nonce":12117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6806","nonce":12117},{"type":"v1","addr":"172.21.15.42:6807","nonce":12117}]},"public_addr":"172.21.15.42:6803/12117","cluster_addr":"172.21.15.42:6805/12117","heartbeat_back_addr":"172.21.15.42:6809/12117","heartbeat_front_addr":"172.21.15.42:6807/12117","state":["exists","up"]},{"osd":3,"uuid":"1960f66b-2d40-4bb9-a289-ce8cd0c1c142","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6818","nonce":12116},{"type":"v1","addr":"172.21.15.42:6819","nonce":12116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6820","nonce":12116},{"type":"v1","addr":"172.21.15.42:6821","nonce":12116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6824","nonce":12116},{"type":"v1","addr":"172.21.15.42:6825","nonce":12116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6822","nonce":12116},{"type":"v1","addr":"172.21.15.42:6823","nonce":12116}]},"public_addr":"172.21.15.42:6819/12116","cluster_addr":"172.21.15.42:6821/12116","heartbeat_back_addr":"172.21.15.42:6825/12116","heartbeat_front_addr":"172.21.15.42:6823/12116","state":["exists","up"]},{"osd":4,"uuid":"45bbf1bd-7c8a-4a0d-b76b-afa6148a855c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":12488},{"type":"v1","addr":"172.21.15.40:6809","nonce":12488}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":12488},{"type":"v1","addr":"172.21.15.40:6812","nonce":12488}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":12488},{"type":"v1","addr":"172.21.15.40:6825","nonce":12488}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":12488},{"type":"v1","addr":"172.21.15.40:6819","nonce":12488}]},"public_addr":"172.21.15.40:6809/12488","cluster_addr":"172.21.15.40:6812/12488","heartbeat_back_addr":"172.21.15.40:6825/12488","heartbeat_front_addr":"172.21.15.40:6819/12488","state":["exists","up"]},{"osd":5,"uuid":"8a5511cd-925f-4869-a49e-924383e4ce53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":12489},{"type":"v1","addr":"172.21.15.40:6801","nonce":12489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":12489},{"type":"v1","addr":"172.21.15.40:6803","nonce":12489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":12489},{"type":"v1","addr":"172.21.15.40:6807","nonce":12489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":12489},{"type":"v1","addr":"172.21.15.40:6805","nonce":12489}]},"public_addr":"172.21.15.40:6801/12489","cluster_addr":"172.21.15.40:6803/12489","heartbeat_back_addr":"172.21.15.40:6807/12489","heartbeat_front_addr":"172.21.15.40:6805/12489","state":["exists","up"]},{"osd":6,"uuid":"d692c179-379e-446c-8484-a219a9241e04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6811","nonce":12491},{"type":"v1","addr":"172.21.15.40:6814","nonce":12491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6817","nonce":12491},{"type":"v1","addr":"172.21.15.40:6818","nonce":12491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":12491},{"type":"v1","addr":"172.21.15.40:6828","nonce":12491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":12491},{"type":"v1","addr":"172.21.15.40:6824","nonce":12491}]},"public_addr":"172.21.15.40:6814/12491","cluster_addr":"172.21.15.40:6818/12491","heartbeat_back_addr":"172.21.15.40:6828/12491","heartbeat_front_addr":"172.21.15.40:6824/12491","state":["exists","up"]},{"osd":7,"uuid":"79c3c243-190e-4e35-b769-319a2077e7cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6813","nonce":12490},{"type":"v1","addr":"172.21.15.40:6816","nonce":12490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":12490},{"type":"v1","addr":"172.21.15.40:6823","nonce":12490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":12490},{"type":"v1","addr":"172.21.15.40:6831","nonce":12490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6827","nonce":12490},{"type":"v1","addr":"172.21.15.40:6829","nonce":12490}]},"public_addr":"172.21.15.40:6816/12490","cluster_addr":"172.21.15.40:6823/12490","heartbeat_back_addr":"172.21.15.40:6831/12490","heartbeat_front_addr":"172.21.15.40:6829/12490","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T05:03:21.349150+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T05:03:21.644281+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T05:03:21.394040+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T05:03:21.587096+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T05:03:21.656497+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T05:03:21.510547+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T05:03:21.659358+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T05:03:21.628063+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2019-12-05T05:03:52.479 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.472+0000 7ffb403af700 1 -- 172.21.15.42:0/590865946 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ffb2003ade0 msgr2=0x7ffb2003d2f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:52.479 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.472+0000 7ffb403af700 1 --2- 172.21.15.42:0/590865946 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ffb2003ade0 0x7ffb2003d2f0 crc :-1 s=READY pgs=107 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:52.479 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.472+0000 7ffb403af700 1 -- 172.21.15.42:0/590865946 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffb3812ad90 msgr2=0x7ffb38065730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:52.479 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.472+0000 7ffb403af700 1 --2- 172.21.15.42:0/590865946 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffb3812ad90 0x7ffb38065730 secure :-1 s=READY pgs=173 cs=0 l=1 rx=0x7ffb34007f10 tx=0x7ffb3400fd30).stop 2019-12-05T05:03:52.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.472+0000 7ffb403af700 1 -- 172.21.15.42:0/590865946 shutdown_connections 2019-12-05T05:03:52.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.472+0000 7ffb403af700 1 --2- 172.21.15.42:0/590865946 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ffb2003ade0 0x7ffb2003d2f0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:52.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.472+0000 7ffb403af700 1 --2- 172.21.15.42:0/590865946 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ffb380a64c0 0x7ffb380a68f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:52.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.472+0000 7ffb403af700 1 --2- 172.21.15.42:0/590865946 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffb3812ad90 0x7ffb38065730 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:52.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.472+0000 7ffb403af700 1 -- 172.21.15.42:0/590865946 shutdown_connections 2019-12-05T05:03:52.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.472+0000 7ffb403af700 1 -- 172.21.15.42:0/590865946 wait complete. 2019-12-05T05:03:52.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.472+0000 7ffb403af700 1 -- 172.21.15.42:0/590865946 >> 172.21.15.42:0/590865946 conn(0x7ffb380aabe0 msgr2=0x7ffb38059d50 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:52.490 INFO:tasks.cephfs.filesystem.ceph_manager:[{u'cache_target_full_ratio_micro': 800000, u'fast_read': False, u'stripe_width': 0, u'flags_names': u'hashpspool', u'tier_of': -1, u'hit_set_grade_decay_rate': 0, u'pg_placement_num': 8, u'use_gmt_hitset': True, u'cache_target_dirty_ratio_micro': 400000, u'create_time': u'2019-12-05T05:03:27.394605+0000', u'snap_epoch': 0, u'erasure_code_profile': u'', u'pg_autoscale_mode': u'off', u'snap_seq': 0, u'expected_num_objects': 0, u'size': 2, u'pg_num_pending': 8, u'auid': 0, u'cache_min_flush_age': 0, u'hit_set_period': 0, u'min_read_recency_for_promote': 0, u'target_max_objects': 0, u'pg_placement_num_target': 8, u'pg_num': 8, u'type': 1, u'grade_table': [], u'pool_name': u'rbd', u'cache_min_evict_age': 0, u'snap_mode': u'selfmanaged', u'pg_num_target': 8, u'cache_mode': u'none', u'min_size': 1, u'cache_target_dirty_high_ratio_micro': 600000, u'object_hash': 2, u'last_pg_merge_meta': {u'ready_epoch': 0, u'source_version': u"0'0", u'source_pgid': u'0.0', u'last_epoch_clean': 0, u'target_version': u"0'0", u'last_epoch_started': 0}, u'write_tier': -1, u'last_force_op_resend_preluminous': u'0', u'pool': 1, u'removed_snaps': u'[]', u'crush_rule': 0, u'tiers': [], u'hit_set_params': {u'type': u'none'}, u'last_force_op_resend': u'0', u'pool_snaps': [], u'quota_max_objects': 0, u'last_force_op_resend_prenautilus': u'0', u'application_metadata': {u'rbd': {}}, u'options': {}, u'hit_set_count': 0, u'flags': 1, u'target_max_bytes': 0, u'quota_max_bytes': 0, u'hit_set_search_last_n': 0, u'last_change': u'17', u'min_write_recency_for_promote': 0, u'read_tier': -1}, {u'cache_target_full_ratio_micro': 800000, u'fast_read': False, u'stripe_width': 0, u'flags_names': u'hashpspool', u'tier_of': -1, u'hit_set_grade_decay_rate': 0, u'pg_placement_num': 8, u'use_gmt_hitset': True, u'cache_target_dirty_ratio_micro': 400000, u'create_time': u'2019-12-05T05:03:30.419845+0000', u'snap_epoch': 0, u'erasure_code_profile': u'', u'pg_autoscale_mode': u'off', u'snap_seq': 0, u'expected_num_objects': 0, u'size': 2, u'pg_num_pending': 8, u'auid': 0, u'cache_min_flush_age': 0, u'hit_set_period': 0, u'min_read_recency_for_promote': 0, u'target_max_objects': 0, u'pg_placement_num_target': 8, u'pg_num': 8, u'type': 1, u'grade_table': [], u'pool_name': u'cephfs_metadata', u'cache_min_evict_age': 0, u'snap_mode': u'selfmanaged', u'pg_num_target': 8, u'cache_mode': u'none', u'min_size': 1, u'cache_target_dirty_high_ratio_micro': 600000, u'object_hash': 2, u'last_pg_merge_meta': {u'ready_epoch': 0, u'source_version': u"0'0", u'source_pgid': u'0.0', u'last_epoch_clean': 0, u'target_version': u"0'0", u'last_epoch_started': 0}, u'write_tier': -1, u'last_force_op_resend_preluminous': u'0', u'pool': 2, u'removed_snaps': u'[]', u'crush_rule': 0, u'tiers': [], u'hit_set_params': {u'type': u'none'}, u'last_force_op_resend': u'0', u'pool_snaps': [], u'quota_max_objects': 0, u'last_force_op_resend_prenautilus': u'0', u'application_metadata': {u'cephfs': {u'metadata': u'cephfs'}}, u'options': {u'pg_autoscale_bias': 4, u'recovery_priority': 5, u'pg_num_min': 16}, u'hit_set_count': 0, u'flags': 1, u'target_max_bytes': 0, u'quota_max_bytes': 0, u'hit_set_search_last_n': 0, u'last_change': u'23', u'min_write_recency_for_promote': 0, u'read_tier': -1}, {u'cache_target_full_ratio_micro': 800000, u'fast_read': False, u'stripe_width': 8192, u'flags_names': u'hashpspool,ec_overwrites', u'tier_of': -1, u'hit_set_grade_decay_rate': 0, u'pg_placement_num': 8, u'use_gmt_hitset': True, u'cache_target_dirty_ratio_micro': 400000, u'create_time': u'2019-12-05T05:03:33.249075+0000', u'snap_epoch': 0, u'erasure_code_profile': u'cephfs_data', u'pg_autoscale_mode': u'off', u'snap_seq': 0, u'expected_num_objects': 0, u'size': 4, u'pg_num_pending': 8, u'auid': 0, u'cache_min_flush_age': 0, u'hit_set_period': 0, u'min_read_recency_for_promote': 0, u'target_max_objects': 0, u'pg_placement_num_target': 8, u'pg_num': 8, u'type': 3, u'grade_table': [], u'pool_name': u'cephfs_data', u'cache_min_evict_age': 0, u'snap_mode': u'selfmanaged', u'pg_num_target': 8, u'cache_mode': u'none', u'min_size': 3, u'cache_target_dirty_high_ratio_micro': 600000, u'object_hash': 2, u'last_pg_merge_meta': {u'ready_epoch': 0, u'source_version': u"0'0", u'source_pgid': u'0.0', u'last_epoch_clean': 0, u'target_version': u"0'0", u'last_epoch_started': 0}, u'write_tier': -1, u'last_force_op_resend_preluminous': u'0', u'pool': 3, u'removed_snaps': u'[]', u'crush_rule': 1, u'tiers': [], u'hit_set_params': {u'type': u'none'}, u'last_force_op_resend': u'0', u'pool_snaps': [], u'quota_max_objects': 0, u'last_force_op_resend_prenautilus': u'0', u'application_metadata': {u'cephfs': {u'data': u'cephfs'}}, u'options': {}, u'hit_set_count': 0, u'flags': 5, u'target_max_bytes': 0, u'quota_max_bytes': 0, u'hit_set_search_last_n': 0, u'last_change': u'24', u'min_write_recency_for_promote': 0, u'read_tier': -1}] 2019-12-05T05:03:52.490 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2019-12-05T05:03:52.600 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.588+0000 7f2d99616700 1 Processor -- start 2019-12-05T05:03:52.600 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.588+0000 7f2d99616700 1 -- start start 2019-12-05T05:03:52.601 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.592+0000 7f2d99616700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2d940a9a00 0x7f2d9407e880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:52.601 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.592+0000 7f2d99616700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d940a1000 con 0x7f2d9409f550 2019-12-05T05:03:52.601 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.592+0000 7f2d99616700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d9409ee30 con 0x7f2d9407edd0 2019-12-05T05:03:52.602 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.592+0000 7f2d99616700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d940a8f80 con 0x7f2d940a9a00 2019-12-05T05:03:52.602 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.592+0000 7f2d93fff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f2d9407edd0 0x7f2d9414ef60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:53992/0 (socket says 172.21.15.42:53992) 2019-12-05T05:03:52.602 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.592+0000 7f2d93fff700 1 -- 172.21.15.42:0/1378088130 learned_addr learned my addr 172.21.15.42:0/1378088130 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:03:52.602 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.592+0000 7f2d92ffd700 1 --2- 172.21.15.42:0/1378088130 >> v2:172.21.15.40:3300/0 conn(0x7f2d940a9a00 0x7f2d9407e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:52.602 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.592+0000 7f2d927fc700 1 -- 172.21.15.42:0/1378088130 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3333077806 0 0) 0x7f2d9409ee30 con 0x7f2d9407edd0 2019-12-05T05:03:52.602 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.592+0000 7f2d927fc700 1 -- 172.21.15.42:0/1378088130 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d780030b0 con 0x7f2d9407edd0 2019-12-05T05:03:52.602 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.592+0000 7f2d927fc700 1 -- 172.21.15.42:0/1378088130 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1371099270 0 0) 0x7f2d940a1000 con 0x7f2d9409f550 2019-12-05T05:03:52.603 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.592+0000 7f2d927fc700 1 -- 172.21.15.42:0/1378088130 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d9409ee30 con 0x7f2d9409f550 2019-12-05T05:03:52.603 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.592+0000 7f2d92ffd700 1 -- 172.21.15.42:0/1378088130 >> v1:172.21.15.42:6789/0 conn(0x7f2d9407edd0 legacy=0x7f2d9414ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:52.603 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.592+0000 7f2d927fc700 1 -- 172.21.15.42:0/1378088130 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3659161633 0 0) 0x7f2d780030b0 con 0x7f2d9407edd0 2019-12-05T05:03:52.603 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.592+0000 7f2d92ffd700 1 -- 172.21.15.42:0/1378088130 >> v1:172.21.15.40:6789/0 conn(0x7f2d9409f550 legacy=0x7f2d94144630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:52.603 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.592+0000 7f2d92ffd700 1 -- 172.21.15.42:0/1378088130 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d9408dd20 con 0x7f2d940a9a00 2019-12-05T05:03:52.603 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.592+0000 7f2d92ffd700 1 --2- 172.21.15.42:0/1378088130 >> v2:172.21.15.40:3300/0 conn(0x7f2d940a9a00 0x7f2d9407e880 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f2d8c0018a0 tx=0x7f2d8c001cd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:52.603 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.592+0000 7f2d92ffd700 1 --2- 172.21.15.42:0/1378088130 >> v2:172.21.15.40:3300/0 conn(0x7f2d940a9a00 0x7f2d9407e880 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f2d8c0018a0 tx=0x7f2d8c001cd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:52.604 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.592+0000 7f2d92ffd700 1 --2- 172.21.15.42:0/1378088130 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2d940a9a00 0x7f2d9407e880 secure :-1 s=READY pgs=175 cs=0 l=1 rx=0x7f2d8c0018a0 tx=0x7f2d8c001cd0).ready entity=mon.0 client_cookie=76b810a781bc48f0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:52.604 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.592+0000 7f2d92ffd700 1 --2- 172.21.15.42:0/1378088130 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2d940a9a00 0x7f2d9407e880 secure :-1 s=THROTTLE_DONE pgs=175 cs=0 l=1 rx=0x7f2d8c0018a0 tx=0x7f2d8c001cd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:52.604 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.592+0000 7f2d927fc700 1 -- 172.21.15.42:0/1378088130 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2d8c001ad0 con 0x7f2d940a9a00 2019-12-05T05:03:52.604 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.592+0000 7f2d92ffd700 1 --2- 172.21.15.42:0/1378088130 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2d940a9a00 0x7f2d9407e880 secure :-1 s=THROTTLE_DONE pgs=175 cs=0 l=1 rx=0x7f2d8c0018a0 tx=0x7f2d8c001cd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:52.604 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.592+0000 7f2d92ffd700 1 --2- 172.21.15.42:0/1378088130 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2d940a9a00 0x7f2d9407e880 secure :-1 s=THROTTLE_DONE pgs=175 cs=0 l=1 rx=0x7f2d8c0018a0 tx=0x7f2d8c001cd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:52.604 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.592+0000 7f2d927fc700 1 -- 172.21.15.42:0/1378088130 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d8c0018e0 con 0x7f2d940a9a00 2019-12-05T05:03:52.604 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.592+0000 7f2d927fc700 1 -- 172.21.15.42:0/1378088130 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2d80011400 con 0x7f2d940a9a00 2019-12-05T05:03:52.605 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.592+0000 7f2d99616700 1 -- 172.21.15.42:0/1378088130 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2d940a9a00 msgr2=0x7f2d9407e880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:52.605 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.592+0000 7f2d99616700 1 --2- 172.21.15.42:0/1378088130 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2d940a9a00 0x7f2d9407e880 secure :-1 s=READY pgs=175 cs=0 l=1 rx=0x7f2d8c0018a0 tx=0x7f2d8c001cd0).stop 2019-12-05T05:03:52.605 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.592+0000 7f2d99616700 1 -- 172.21.15.42:0/1378088130 shutdown_connections 2019-12-05T05:03:52.605 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.592+0000 7f2d99616700 1 --2- 172.21.15.42:0/1378088130 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2d940a9a00 0x7f2d9407e880 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:52.605 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.592+0000 7f2d99616700 1 -- 172.21.15.42:0/1378088130 shutdown_connections 2019-12-05T05:03:52.605 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.592+0000 7f2d99616700 1 -- 172.21.15.42:0/1378088130 wait complete. 2019-12-05T05:03:52.605 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.592+0000 7f2d99616700 1 -- 172.21.15.42:0/1378088130 >> 172.21.15.42:0/1378088130 conn(0x7f2d940a9ef0 msgr2=0x7f2d940aa320 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:52.609 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.596+0000 7f2d99616700 1 Processor -- start 2019-12-05T05:03:52.609 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.596+0000 7f2d99616700 1 -- start start 2019-12-05T05:03:52.609 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.596+0000 7f2d99616700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2d9412d260 0x7f2d9412d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:52.609 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.596+0000 7f2d99616700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2d941326a0 0x7f2d94134ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:52.609 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.596+0000 7f2d99616700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d9411c270 con 0x7f2d94143a40 2019-12-05T05:03:52.609 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.596+0000 7f2d99616700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d9407df20 con 0x7f2d9412d260 2019-12-05T05:03:52.610 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.596+0000 7f2d99616700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d9411c540 con 0x7f2d941326a0 2019-12-05T05:03:52.610 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.596+0000 7f2d937fe700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f2d94143a40 0x7f2d9407d720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:53994/0 (socket says 172.21.15.42:53994) 2019-12-05T05:03:52.610 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.596+0000 7f2d937fe700 1 -- 172.21.15.42:0/1939878524 learned_addr learned my addr 172.21.15.42:0/1939878524 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:03:52.610 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.596+0000 7f2d92ffd700 1 --2- 172.21.15.42:0/1939878524 >> v2:172.21.15.40:3300/0 conn(0x7f2d9412d260 0x7f2d9412d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:52.610 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.596+0000 7f2d93fff700 1 --2- 172.21.15.42:0/1939878524 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2d941326a0 0x7f2d94134ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:52.610 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.596+0000 7f2d8bfff700 1 -- 172.21.15.42:0/1939878524 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3483354683 0 0) 0x7f2d9411c270 con 0x7f2d94143a40 2019-12-05T05:03:52.610 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.596+0000 7f2d8bfff700 1 -- 172.21.15.42:0/1939878524 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d700030b0 con 0x7f2d94143a40 2019-12-05T05:03:52.611 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.600+0000 7f2d92ffd700 1 -- 172.21.15.42:0/1939878524 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2d941326a0 msgr2=0x7f2d94134ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:52.611 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.600+0000 7f2d92ffd700 1 --2- 172.21.15.42:0/1939878524 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2d941326a0 0x7f2d94134ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:52.611 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.600+0000 7f2d92ffd700 1 -- 172.21.15.42:0/1939878524 >> v1:172.21.15.42:6789/0 conn(0x7f2d94143a40 legacy=0x7f2d9407d720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:52.611 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.600+0000 7f2d92ffd700 1 -- 172.21.15.42:0/1939878524 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d94147b60 con 0x7f2d9412d260 2019-12-05T05:03:52.612 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.600+0000 7f2d92ffd700 1 --2- 172.21.15.42:0/1939878524 >> v2:172.21.15.40:3300/0 conn(0x7f2d9412d260 0x7f2d9412d690 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f2d9411c700 tx=0x7f2d80007a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:52.612 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.600+0000 7f2d92ffd700 1 --2- 172.21.15.42:0/1939878524 >> v2:172.21.15.40:3300/0 conn(0x7f2d9412d260 0x7f2d9412d690 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f2d9411c700 tx=0x7f2d80007a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:52.612 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.600+0000 7f2d92ffd700 1 --2- 172.21.15.42:0/1939878524 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2d9412d260 0x7f2d9412d690 secure :-1 s=READY pgs=176 cs=0 l=1 rx=0x7f2d9411c700 tx=0x7f2d80007a20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:52.612 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.600+0000 7f2d92ffd700 1 --2- 172.21.15.42:0/1939878524 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2d9412d260 0x7f2d9412d690 secure :-1 s=THROTTLE_DONE pgs=176 cs=0 l=1 rx=0x7f2d9411c700 tx=0x7f2d80007a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:52.612 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.600+0000 7f2d8bfff700 1 -- 172.21.15.42:0/1939878524 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2d80016060 con 0x7f2d9412d260 2019-12-05T05:03:52.612 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.600+0000 7f2d92ffd700 1 --2- 172.21.15.42:0/1939878524 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2d9412d260 0x7f2d9412d690 secure :-1 s=THROTTLE_DONE pgs=176 cs=0 l=1 rx=0x7f2d9411c700 tx=0x7f2d80007a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:52.612 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.600+0000 7f2d99616700 1 -- 172.21.15.42:0/1939878524 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d9412dbe0 con 0x7f2d9412d260 2019-12-05T05:03:52.613 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.600+0000 7f2d8bfff700 1 -- 172.21.15.42:0/1939878524 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d80008b60 con 0x7f2d9412d260 2019-12-05T05:03:52.613 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.600+0000 7f2d92ffd700 1 --2- 172.21.15.42:0/1939878524 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2d9412d260 0x7f2d9412d690 secure :-1 s=THROTTLE_DONE pgs=176 cs=0 l=1 rx=0x7f2d9411c700 tx=0x7f2d80007a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:52.613 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.600+0000 7f2d8bfff700 1 -- 172.21.15.42:0/1939878524 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2d80011900 con 0x7f2d9412d260 2019-12-05T05:03:52.613 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.600+0000 7f2d99616700 1 -- 172.21.15.42:0/1939878524 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d941487f0 con 0x7f2d9412d260 2019-12-05T05:03:52.613 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.600+0000 7f2d92ffd700 1 --2- 172.21.15.42:0/1939878524 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2d9412d260 0x7f2d9412d690 secure :-1 s=THROTTLE_DONE pgs=176 cs=0 l=1 rx=0x7f2d9411c700 tx=0x7f2d80007a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:52.613 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.600+0000 7f2d8bfff700 1 -- 172.21.15.42:0/1939878524 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f2d80011c20 con 0x7f2d9412d260 2019-12-05T05:03:52.613 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.600+0000 7f2d92ffd700 1 --2- 172.21.15.42:0/1939878524 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2d9412d260 0x7f2d9412d690 secure :-1 s=THROTTLE_DONE pgs=176 cs=0 l=1 rx=0x7f2d9411c700 tx=0x7f2d80007a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:52.614 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.600+0000 7f2d8bfff700 1 --2- 172.21.15.42:0/1939878524 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2d7003b280 0x7f2d7003d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:52.614 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.600+0000 7f2d8bfff700 1 -- 172.21.15.42:0/1939878524 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f2d8004ae20 con 0x7f2d9412d260 2019-12-05T05:03:52.614 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.600+0000 7f2d937fe700 1 --2- 172.21.15.42:0/1939878524 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2d7003b280 0x7f2d7003d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:52.614 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.604+0000 7f2d937fe700 1 --2- 172.21.15.42:0/1939878524 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2d7003b280 0x7f2d7003d790 crc :-1 s=READY pgs=108 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:52.614 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.604+0000 7f2d99616700 1 -- 172.21.15.42:0/1939878524 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d9405d110 con 0x7f2d9412d260 2019-12-05T05:03:52.620 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.616+0000 7f2d92ffd700 1 --2- 172.21.15.42:0/1939878524 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2d9412d260 0x7f2d9412d690 secure :-1 s=THROTTLE_DONE pgs=176 cs=0 l=1 rx=0x7f2d9411c700 tx=0x7f2d80007a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:52.620 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.616+0000 7f2d8bfff700 1 -- 172.21.15.42:0/1939878524 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f2d80031e80 con 0x7f2d9412d260 2019-12-05T05:03:52.787 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.780+0000 7f2d99616700 1 -- 172.21.15.42:0/1939878524 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f2d940a7570 con 0x7f2d9412d260 2019-12-05T05:03:52.788 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.784+0000 7f2d92ffd700 1 --2- 172.21.15.42:0/1939878524 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2d9412d260 0x7f2d9412d690 secure :-1 s=THROTTLE_DONE pgs=176 cs=0 l=1 rx=0x7f2d9411c700 tx=0x7f2d80007a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:52.789 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.784+0000 7f2d8bfff700 1 -- 172.21.15.42:0/1939878524 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v24) v1 ==== 92+0+10 (secure 0 0 0) 0x7f2d800320e0 con 0x7f2d9412d260 2019-12-05T05:03:52.789 INFO:teuthology.orchestra.run.smithi042.stdout:pg_num: 8 2019-12-05T05:03:52.791 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.784+0000 7f2d99616700 1 -- 172.21.15.42:0/1939878524 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2d7003b280 msgr2=0x7f2d7003d790 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:52.791 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.784+0000 7f2d99616700 1 --2- 172.21.15.42:0/1939878524 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2d7003b280 0x7f2d7003d790 crc :-1 s=READY pgs=108 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:52.791 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.784+0000 7f2d99616700 1 -- 172.21.15.42:0/1939878524 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2d9412d260 msgr2=0x7f2d9412d690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:52.791 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.784+0000 7f2d99616700 1 --2- 172.21.15.42:0/1939878524 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2d9412d260 0x7f2d9412d690 secure :-1 s=READY pgs=176 cs=0 l=1 rx=0x7f2d9411c700 tx=0x7f2d80007a20).stop 2019-12-05T05:03:52.791 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.784+0000 7f2d99616700 1 -- 172.21.15.42:0/1939878524 shutdown_connections 2019-12-05T05:03:52.792 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.784+0000 7f2d99616700 1 --2- 172.21.15.42:0/1939878524 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2d7003b280 0x7f2d7003d790 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:52.792 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.784+0000 7f2d99616700 1 --2- 172.21.15.42:0/1939878524 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2d941326a0 0x7f2d94134ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:52.792 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.784+0000 7f2d99616700 1 --2- 172.21.15.42:0/1939878524 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2d9412d260 0x7f2d9412d690 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:52.792 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.784+0000 7f2d99616700 1 -- 172.21.15.42:0/1939878524 shutdown_connections 2019-12-05T05:03:52.792 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.784+0000 7f2d99616700 1 -- 172.21.15.42:0/1939878524 wait complete. 2019-12-05T05:03:52.792 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.784+0000 7f2d99616700 1 -- 172.21.15.42:0/1939878524 >> 172.21.15.42:0/1939878524 conn(0x7f2d940a8b30 msgr2=0x7f2d940a5850 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:52.801 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2019-12-05T05:03:52.901 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.892+0000 7fdf0acce700 1 Processor -- start 2019-12-05T05:03:52.902 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.892+0000 7fdf0acce700 1 -- start start 2019-12-05T05:03:52.902 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.892+0000 7fdf0acce700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fdf0409f550 0x7fdf04144630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:52.902 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.892+0000 7fdf0acce700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fdf0414cf50 0x7fdf0414f390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:52.902 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.892+0000 7fdf0acce700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf0407f080 con 0x7fdf040a9a00 2019-12-05T05:03:52.902 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.892+0000 7fdf0acce700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf040a8f80 con 0x7fdf0414cf50 2019-12-05T05:03:52.902 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.892+0000 7fdf0acce700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf040a9220 con 0x7fdf0409f550 2019-12-05T05:03:52.903 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.892+0000 7fdf0926b700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fdf0409f550 0x7fdf04144630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:52.903 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.892+0000 7fdf0926b700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fdf0409f550 0x7fdf04144630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:53964/0 (socket says 172.21.15.42:53964) 2019-12-05T05:03:52.903 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.892+0000 7fdf0926b700 1 -- 172.21.15.42:0/3126230209 learned_addr learned my addr 172.21.15.42:0/3126230209 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:03:52.904 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.892+0000 7fdf09a6c700 1 --2- 172.21.15.42:0/3126230209 >> v2:172.21.15.40:3300/0 conn(0x7fdf0414cf50 0x7fdf0414f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:52.904 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.892+0000 7fdf0926b700 1 -- 172.21.15.42:0/3126230209 >> v2:172.21.15.40:3300/0 conn(0x7fdf0414cf50 msgr2=0x7fdf0414f390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:03:52.904 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.892+0000 7fdf0926b700 1 --2- 172.21.15.42:0/3126230209 >> v2:172.21.15.40:3300/0 conn(0x7fdf0414cf50 0x7fdf0414f390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:03:52.904 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.896+0000 7fdf0926b700 1 -- 172.21.15.42:0/3126230209 >> v1:172.21.15.40:6789/0 conn(0x7fdf040a9a00 legacy=0x7fdf0407e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:52.904 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.896+0000 7fdf0926b700 1 -- 172.21.15.42:0/3126230209 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf040a1000 con 0x7fdf0409f550 2019-12-05T05:03:52.904 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.896+0000 7fdf0926b700 1 --2- 172.21.15.42:0/3126230209 >> v2:172.21.15.42:3300/0 conn(0x7fdf0409f550 0x7fdf04144630 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fdf00002f30 tx=0x7fdf0000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:52.904 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.896+0000 7fdf0926b700 1 --2- 172.21.15.42:0/3126230209 >> v2:172.21.15.42:3300/0 conn(0x7fdf0409f550 0x7fdf04144630 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fdf00002f30 tx=0x7fdf0000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:52.905 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.896+0000 7fdf0926b700 1 --2- 172.21.15.42:0/3126230209 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdf0409f550 0x7fdf04144630 secure :-1 s=READY pgs=207 cs=0 l=1 rx=0x7fdf00002f30 tx=0x7fdf0000fd30).ready entity=mon.1 client_cookie=6a060b2fbd7a3f3a server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:52.905 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.896+0000 7fdf0926b700 1 --2- 172.21.15.42:0/3126230209 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdf0409f550 0x7fdf04144630 secure :-1 s=THROTTLE_DONE pgs=207 cs=0 l=1 rx=0x7fdf00002f30 tx=0x7fdf0000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:52.905 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.896+0000 7fdf0926b700 1 --2- 172.21.15.42:0/3126230209 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdf0409f550 0x7fdf04144630 secure :-1 s=THROTTLE_DONE pgs=207 cs=0 l=1 rx=0x7fdf00002f30 tx=0x7fdf0000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:52.905 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.896+0000 7fdefbfff700 1 -- 172.21.15.42:0/3126230209 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdf00008c90 con 0x7fdf0409f550 2019-12-05T05:03:52.905 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.896+0000 7fdf0926b700 1 --2- 172.21.15.42:0/3126230209 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdf0409f550 0x7fdf04144630 secure :-1 s=THROTTLE_DONE pgs=207 cs=0 l=1 rx=0x7fdf00002f30 tx=0x7fdf0000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:52.905 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.896+0000 7fdefbfff700 1 -- 172.21.15.42:0/3126230209 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf000088b0 con 0x7fdf0409f550 2019-12-05T05:03:52.905 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.896+0000 7fdefbfff700 1 -- 172.21.15.42:0/3126230209 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdf00011ad0 con 0x7fdf0409f550 2019-12-05T05:03:52.906 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.896+0000 7fdf0acce700 1 -- 172.21.15.42:0/3126230209 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdf0409f550 msgr2=0x7fdf04144630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:52.906 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.896+0000 7fdf0acce700 1 --2- 172.21.15.42:0/3126230209 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdf0409f550 0x7fdf04144630 secure :-1 s=READY pgs=207 cs=0 l=1 rx=0x7fdf00002f30 tx=0x7fdf0000fd30).stop 2019-12-05T05:03:52.906 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.896+0000 7fdf0acce700 1 -- 172.21.15.42:0/3126230209 shutdown_connections 2019-12-05T05:03:52.906 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.896+0000 7fdf0acce700 1 --2- 172.21.15.42:0/3126230209 >> v2:172.21.15.40:3300/0 conn(0x7fdf0414cf50 0x7fdf0414f390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:03:52.906 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.896+0000 7fdf0acce700 1 --2- 172.21.15.42:0/3126230209 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdf0409f550 0x7fdf04144630 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:52.906 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.896+0000 7fdf0acce700 1 -- 172.21.15.42:0/3126230209 shutdown_connections 2019-12-05T05:03:52.906 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.896+0000 7fdf0acce700 1 -- 172.21.15.42:0/3126230209 wait complete. 2019-12-05T05:03:52.907 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.896+0000 7fdf0acce700 1 -- 172.21.15.42:0/3126230209 >> 172.21.15.42:0/3126230209 conn(0x7fdf040a9ef0 msgr2=0x7fdf040aa320 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:52.910 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.896+0000 7fdf0acce700 1 Processor -- start 2019-12-05T05:03:52.910 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.896+0000 7fdf0acce700 1 -- start start 2019-12-05T05:03:52.910 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.896+0000 7fdf0acce700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdf041442e0 0x7fdf04126010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:52.910 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.896+0000 7fdf0acce700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf0407e2c0 con 0x7fdf041434e0 2019-12-05T05:03:52.910 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.896+0000 7fdf0acce700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf041447d0 con 0x7fdf04143d60 2019-12-05T05:03:52.910 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.896+0000 7fdf0acce700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf0408e700 con 0x7fdf041442e0 2019-12-05T05:03:52.911 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.896+0000 7fdf08a6a700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fdf04143d60 0x7fdf0408f610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:54008/0 (socket says 172.21.15.42:54008) 2019-12-05T05:03:52.911 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.896+0000 7fdf08a6a700 1 -- 172.21.15.42:0/1752519367 learned_addr learned my addr 172.21.15.42:0/1752519367 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:03:52.911 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.896+0000 7fdf0926b700 1 --2- 172.21.15.42:0/1752519367 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdf041442e0 0x7fdf04126010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:52.911 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.896+0000 7fdef9ffb700 1 -- 172.21.15.42:0/1752519367 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 329193844 0 0) 0x7fdf041447d0 con 0x7fdf04143d60 2019-12-05T05:03:52.911 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.896+0000 7fdef9ffb700 1 -- 172.21.15.42:0/1752519367 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdee40030b0 con 0x7fdf04143d60 2019-12-05T05:03:52.911 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.900+0000 7fdef9ffb700 1 -- 172.21.15.42:0/1752519367 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2922219955 0 0) 0x7fdefc002240 con 0x7fdf041434e0 2019-12-05T05:03:52.911 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.900+0000 7fdef9ffb700 1 -- 172.21.15.42:0/1752519367 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf041447d0 con 0x7fdf041434e0 2019-12-05T05:03:52.912 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.900+0000 7fdef9ffb700 1 -- 172.21.15.42:0/1752519367 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 2872065881 0 0) 0x7fdee40030b0 con 0x7fdf04143d60 2019-12-05T05:03:52.912 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.900+0000 7fdf0926b700 1 -- 172.21.15.42:0/1752519367 >> v1:172.21.15.42:6789/0 conn(0x7fdf04143d60 legacy=0x7fdf0408f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:52.912 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.900+0000 7fdf0926b700 1 -- 172.21.15.42:0/1752519367 >> v1:172.21.15.40:6789/0 conn(0x7fdf041434e0 legacy=0x7fdf0407dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:52.912 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.900+0000 7fdf0926b700 1 -- 172.21.15.42:0/1752519367 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf04126560 con 0x7fdf041442e0 2019-12-05T05:03:52.912 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.900+0000 7fdf0926b700 1 --2- 172.21.15.42:0/1752519367 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdf041442e0 0x7fdf04126010 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fdef4001830 tx=0x7fdef4001c60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:52.912 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.900+0000 7fdf0926b700 1 --2- 172.21.15.42:0/1752519367 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdf041442e0 0x7fdf04126010 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fdef4001830 tx=0x7fdef4001c60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:52.912 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.900+0000 7fdf0926b700 1 --2- 172.21.15.42:0/1752519367 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdf041442e0 0x7fdf04126010 secure :-1 s=READY pgs=48 cs=0 l=1 rx=0x7fdef4001830 tx=0x7fdef4001c60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:52.913 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.900+0000 7fdf0926b700 1 --2- 172.21.15.42:0/1752519367 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdf041442e0 0x7fdf04126010 secure :-1 s=THROTTLE_DONE pgs=48 cs=0 l=1 rx=0x7fdef4001830 tx=0x7fdef4001c60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:52.913 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.900+0000 7fdef9ffb700 1 -- 172.21.15.42:0/1752519367 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdef4001a60 con 0x7fdf041442e0 2019-12-05T05:03:52.913 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.900+0000 7fdf0926b700 1 --2- 172.21.15.42:0/1752519367 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdf041442e0 0x7fdf04126010 secure :-1 s=THROTTLE_DONE pgs=48 cs=0 l=1 rx=0x7fdef4001830 tx=0x7fdef4001c60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:52.913 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.900+0000 7fdef9ffb700 1 -- 172.21.15.42:0/1752519367 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdef4001870 con 0x7fdf041442e0 2019-12-05T05:03:52.914 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.900+0000 7fdf0926b700 1 --2- 172.21.15.42:0/1752519367 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdf041442e0 0x7fdf04126010 secure :-1 s=THROTTLE_DONE pgs=48 cs=0 l=1 rx=0x7fdef4001830 tx=0x7fdef4001c60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:52.914 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.900+0000 7fdf0acce700 1 -- 172.21.15.42:0/1752519367 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf04131a30 con 0x7fdf041442e0 2019-12-05T05:03:52.914 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.900+0000 7fdef9ffb700 1 -- 172.21.15.42:0/1752519367 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdf0000f660 con 0x7fdf041442e0 2019-12-05T05:03:52.914 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.900+0000 7fdf0acce700 1 -- 172.21.15.42:0/1752519367 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf04131f80 con 0x7fdf041442e0 2019-12-05T05:03:52.914 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.900+0000 7fdf0926b700 1 --2- 172.21.15.42:0/1752519367 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdf041442e0 0x7fdf04126010 secure :-1 s=THROTTLE_DONE pgs=48 cs=0 l=1 rx=0x7fdef4001830 tx=0x7fdef4001c60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:52.914 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.900+0000 7fdef9ffb700 1 -- 172.21.15.42:0/1752519367 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fdf00005540 con 0x7fdf041442e0 2019-12-05T05:03:52.914 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.900+0000 7fdf0926b700 1 --2- 172.21.15.42:0/1752519367 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdf041442e0 0x7fdf04126010 secure :-1 s=THROTTLE_DONE pgs=48 cs=0 l=1 rx=0x7fdef4001830 tx=0x7fdef4001c60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:52.915 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.900+0000 7fdef9ffb700 1 --2- 172.21.15.42:0/1752519367 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fdee403b420 0x7fdee403d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:52.915 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.900+0000 7fdef9ffb700 1 -- 172.21.15.42:0/1752519367 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6031+0+0 (secure 0 0 0) 0x7fdf0004b4b0 con 0x7fdf041442e0 2019-12-05T05:03:52.915 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.900+0000 7fdf08a6a700 1 --2- 172.21.15.42:0/1752519367 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fdee403b420 0x7fdee403d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:52.915 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.904+0000 7fdf08a6a700 1 --2- 172.21.15.42:0/1752519367 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fdee403b420 0x7fdee403d930 crc :-1 s=READY pgs=109 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:52.915 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.908+0000 7fdf0acce700 1 -- 172.21.15.42:0/1752519367 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf0405d110 con 0x7fdf041442e0 2019-12-05T05:03:52.922 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.916+0000 7fdf0926b700 1 --2- 172.21.15.42:0/1752519367 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdf041442e0 0x7fdf04126010 secure :-1 s=THROTTLE_DONE pgs=48 cs=0 l=1 rx=0x7fdef4001830 tx=0x7fdef4001c60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:52.922 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:52.916+0000 7fdef9ffb700 1 -- 172.21.15.42:0/1752519367 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fdf00032510 con 0x7fdf041442e0 2019-12-05T05:03:53.080 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.076+0000 7fdf0acce700 1 -- 172.21.15.42:0/1752519367 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7fdf0405b8c0 con 0x7fdf041442e0 2019-12-05T05:03:53.081 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.076+0000 7fdf0926b700 1 --2- 172.21.15.42:0/1752519367 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdf041442e0 0x7fdf04126010 secure :-1 s=THROTTLE_DONE pgs=48 cs=0 l=1 rx=0x7fdef4001830 tx=0x7fdef4001c60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:53.081 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.076+0000 7fdef9ffb700 1 -- 172.21.15.42:0/1752519367 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v24) v1 ==== 104+0+10 (secure 0 0 0) 0x7fdf00032770 con 0x7fdf041442e0 2019-12-05T05:03:53.083 INFO:teuthology.orchestra.run.smithi042.stdout:pg_num: 8 2019-12-05T05:03:53.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.080+0000 7fdf0acce700 1 -- 172.21.15.42:0/1752519367 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fdee403b420 msgr2=0x7fdee403d930 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:53.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.080+0000 7fdf0acce700 1 --2- 172.21.15.42:0/1752519367 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fdee403b420 0x7fdee403d930 crc :-1 s=READY pgs=109 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:53.088 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.080+0000 7fdf0acce700 1 -- 172.21.15.42:0/1752519367 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdf041442e0 msgr2=0x7fdf04126010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:53.088 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.080+0000 7fdf0acce700 1 --2- 172.21.15.42:0/1752519367 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdf041442e0 0x7fdf04126010 secure :-1 s=READY pgs=48 cs=0 l=1 rx=0x7fdef4001830 tx=0x7fdef4001c60).stop 2019-12-05T05:03:53.088 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.080+0000 7fdf0acce700 1 -- 172.21.15.42:0/1752519367 shutdown_connections 2019-12-05T05:03:53.088 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.080+0000 7fdf0acce700 1 --2- 172.21.15.42:0/1752519367 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fdee403b420 0x7fdee403d930 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:53.088 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.080+0000 7fdf0acce700 1 --2- 172.21.15.42:0/1752519367 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdf041442e0 0x7fdf04126010 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:53.088 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.080+0000 7fdf0acce700 1 -- 172.21.15.42:0/1752519367 shutdown_connections 2019-12-05T05:03:53.088 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.080+0000 7fdf0acce700 1 -- 172.21.15.42:0/1752519367 wait complete. 2019-12-05T05:03:53.089 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.080+0000 7fdf0acce700 1 -- 172.21.15.42:0/1752519367 >> 172.21.15.42:0/1752519367 conn(0x7fdf0409f550 msgr2=0x7fdf0407e650 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:53.100 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2019-12-05T05:03:53.214 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.200+0000 7fce5435d700 1 Processor -- start 2019-12-05T05:03:53.214 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.200+0000 7fce5435d700 1 -- start start 2019-12-05T05:03:53.214 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.204+0000 7fce5435d700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fce4c0a9a00 0x7fce4c07e880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:53.214 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.204+0000 7fce5435d700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fce4c07edd0 0x7fce4c14ef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:53.215 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.204+0000 7fce5435d700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce4c0a1000 con 0x7fce4c09f550 2019-12-05T05:03:53.215 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.204+0000 7fce5435d700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce4c0a8f80 con 0x7fce4c0a9a00 2019-12-05T05:03:53.215 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.204+0000 7fce5435d700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce4c0a9220 con 0x7fce4c07edd0 2019-12-05T05:03:53.215 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.204+0000 7fce530fb700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fce4c07edd0 0x7fce4c14ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:53.215 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.204+0000 7fce520f9700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fce4c0a9a00 0x7fce4c07e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:53.215 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.204+0000 7fce530fb700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fce4c07edd0 0x7fce4c14ef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:54446/0 (socket says 172.21.15.42:54446) 2019-12-05T05:03:53.215 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.204+0000 7fce530fb700 1 -- 172.21.15.42:0/3533925888 learned_addr learned my addr 172.21.15.42:0/3533925888 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:03:53.216 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.204+0000 7fce530fb700 1 -- 172.21.15.42:0/3533925888 >> v2:172.21.15.40:3300/0 conn(0x7fce4c0a9a00 msgr2=0x7fce4c07e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:03:53.216 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.204+0000 7fce530fb700 1 --2- 172.21.15.42:0/3533925888 >> v2:172.21.15.40:3300/0 conn(0x7fce4c0a9a00 0x7fce4c07e880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:03:53.216 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.204+0000 7fce530fb700 1 -- 172.21.15.42:0/3533925888 >> v1:172.21.15.40:6789/0 conn(0x7fce4c09f550 legacy=0x7fce4c144630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:53.216 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.204+0000 7fce530fb700 1 -- 172.21.15.42:0/3533925888 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce4c09ee30 con 0x7fce4c07edd0 2019-12-05T05:03:53.216 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.204+0000 7fce530fb700 1 --2- 172.21.15.42:0/3533925888 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fce4c07edd0 0x7fce4c14ef60 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fce44008b10 tx=0x7fce440100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:53.216 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.204+0000 7fce530fb700 1 --2- 172.21.15.42:0/3533925888 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fce4c07edd0 0x7fce4c14ef60 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fce44008b10 tx=0x7fce440100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:53.216 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.204+0000 7fce530fb700 1 --2- 172.21.15.42:0/3533925888 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fce4c07edd0 0x7fce4c14ef60 secure :-1 s=READY pgs=49 cs=0 l=1 rx=0x7fce44008b10 tx=0x7fce440100b0).ready entity=mon.2 client_cookie=78cc270362fcdc5 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:53.217 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.204+0000 7fce530fb700 1 --2- 172.21.15.42:0/3533925888 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fce4c07edd0 0x7fce4c14ef60 secure :-1 s=THROTTLE_DONE pgs=49 cs=0 l=1 rx=0x7fce44008b10 tx=0x7fce440100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:53.217 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.204+0000 7fce518f8700 1 -- 172.21.15.42:0/3533925888 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fce4400a160 con 0x7fce4c07edd0 2019-12-05T05:03:53.217 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.204+0000 7fce530fb700 1 --2- 172.21.15.42:0/3533925888 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fce4c07edd0 0x7fce4c14ef60 secure :-1 s=THROTTLE_DONE pgs=49 cs=0 l=1 rx=0x7fce44008b10 tx=0x7fce440100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:53.217 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.204+0000 7fce530fb700 1 --2- 172.21.15.42:0/3533925888 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fce4c07edd0 0x7fce4c14ef60 secure :-1 s=THROTTLE_DONE pgs=49 cs=0 l=1 rx=0x7fce44008b10 tx=0x7fce440100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:53.217 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.204+0000 7fce518f8700 1 -- 172.21.15.42:0/3533925888 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce44011a30 con 0x7fce4c07edd0 2019-12-05T05:03:53.217 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.204+0000 7fce518f8700 1 -- 172.21.15.42:0/3533925888 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fce4400a550 con 0x7fce4c07edd0 2019-12-05T05:03:53.217 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.204+0000 7fce5435d700 1 -- 172.21.15.42:0/3533925888 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fce4c07edd0 msgr2=0x7fce4c14ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:53.218 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.204+0000 7fce5435d700 1 --2- 172.21.15.42:0/3533925888 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fce4c07edd0 0x7fce4c14ef60 secure :-1 s=READY pgs=49 cs=0 l=1 rx=0x7fce44008b10 tx=0x7fce440100b0).stop 2019-12-05T05:03:53.218 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.208+0000 7fce5435d700 1 -- 172.21.15.42:0/3533925888 shutdown_connections 2019-12-05T05:03:53.218 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.208+0000 7fce5435d700 1 --2- 172.21.15.42:0/3533925888 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fce4c07edd0 0x7fce4c14ef60 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:53.218 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.208+0000 7fce5435d700 1 --2- 172.21.15.42:0/3533925888 >> v2:172.21.15.40:3300/0 conn(0x7fce4c0a9a00 0x7fce4c07e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:03:53.218 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.208+0000 7fce5435d700 1 -- 172.21.15.42:0/3533925888 shutdown_connections 2019-12-05T05:03:53.218 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.208+0000 7fce5435d700 1 -- 172.21.15.42:0/3533925888 wait complete. 2019-12-05T05:03:53.218 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.208+0000 7fce5435d700 1 -- 172.21.15.42:0/3533925888 >> 172.21.15.42:0/3533925888 conn(0x7fce4c0a9ef0 msgr2=0x7fce4c0aa320 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:53.222 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.208+0000 7fce5435d700 1 Processor -- start 2019-12-05T05:03:53.222 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.212+0000 7fce5435d700 1 -- start start 2019-12-05T05:03:53.222 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.212+0000 7fce5435d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fce4c1232e0 0x7fce4c0a1230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:53.222 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.212+0000 7fce5435d700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce4c144280 con 0x7fce4c130840 2019-12-05T05:03:53.223 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.212+0000 7fce5435d700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce4c0a1780 con 0x7fce4c130410 2019-12-05T05:03:53.223 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.212+0000 7fce5435d700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce4c130c70 con 0x7fce4c1232e0 2019-12-05T05:03:53.223 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.212+0000 7fce528fa700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fce4c1232e0 0x7fce4c0a1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:53.223 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.212+0000 7fce520f9700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fce4c130410 0x7fce4c07da80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:54024/0 (socket says 172.21.15.42:54024) 2019-12-05T05:03:53.223 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.212+0000 7fce528fa700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fce4c1232e0 0x7fce4c0a1230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:53984/0 (socket says 172.21.15.42:53984) 2019-12-05T05:03:53.223 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.212+0000 7fce520f9700 1 -- 172.21.15.42:0/318568562 learned_addr learned my addr 172.21.15.42:0/318568562 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:03:53.223 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.212+0000 7fce528fa700 1 -- 172.21.15.42:0/318568562 >> v1:172.21.15.42:6789/0 conn(0x7fce4c130410 legacy=0x7fce4c07da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:53.223 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.212+0000 7fce528fa700 1 -- 172.21.15.42:0/318568562 >> v1:172.21.15.40:6789/0 conn(0x7fce4c130840 legacy=0x7fce4c137860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:53.224 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.212+0000 7fce437fe700 1 -- 172.21.15.42:0/318568562 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 191844177 0 0) 0x7fce4c0a1780 con 0x7fce4c130410 2019-12-05T05:03:53.224 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.212+0000 7fce528fa700 1 -- 172.21.15.42:0/318568562 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce4c090050 con 0x7fce4c1232e0 2019-12-05T05:03:53.224 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.212+0000 7fce528fa700 1 --2- 172.21.15.42:0/318568562 >> v2:172.21.15.42:3300/0 conn(0x7fce4c1232e0 0x7fce4c0a1230 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fce48001430 tx=0x7fce440100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:53.224 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.212+0000 7fce528fa700 1 --2- 172.21.15.42:0/318568562 >> v2:172.21.15.42:3300/0 conn(0x7fce4c1232e0 0x7fce4c0a1230 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fce48001430 tx=0x7fce440100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:53.224 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.212+0000 7fce528fa700 1 --2- 172.21.15.42:0/318568562 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fce4c1232e0 0x7fce4c0a1230 secure :-1 s=READY pgs=210 cs=0 l=1 rx=0x7fce48001430 tx=0x7fce440100b0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:53.224 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.212+0000 7fce528fa700 1 --2- 172.21.15.42:0/318568562 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fce4c1232e0 0x7fce4c0a1230 secure :-1 s=THROTTLE_DONE pgs=210 cs=0 l=1 rx=0x7fce48001430 tx=0x7fce440100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:53.224 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.212+0000 7fce437fe700 1 -- 172.21.15.42:0/318568562 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fce4800c9a0 con 0x7fce4c1232e0 2019-12-05T05:03:53.225 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.212+0000 7fce528fa700 1 --2- 172.21.15.42:0/318568562 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fce4c1232e0 0x7fce4c0a1230 secure :-1 s=THROTTLE_DONE pgs=210 cs=0 l=1 rx=0x7fce48001430 tx=0x7fce440100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:53.225 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.212+0000 7fce437fe700 1 -- 172.21.15.42:0/318568562 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce48011e10 con 0x7fce4c1232e0 2019-12-05T05:03:53.225 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.212+0000 7fce528fa700 1 --2- 172.21.15.42:0/318568562 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fce4c1232e0 0x7fce4c0a1230 secure :-1 s=THROTTLE_DONE pgs=210 cs=0 l=1 rx=0x7fce48001430 tx=0x7fce440100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:53.225 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.212+0000 7fce5435d700 1 -- 172.21.15.42:0/318568562 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce4c07e280 con 0x7fce4c1232e0 2019-12-05T05:03:53.225 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.212+0000 7fce437fe700 1 -- 172.21.15.42:0/318568562 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fce48010e00 con 0x7fce4c1232e0 2019-12-05T05:03:53.225 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.212+0000 7fce5435d700 1 -- 172.21.15.42:0/318568562 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce4c139070 con 0x7fce4c1232e0 2019-12-05T05:03:53.225 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.212+0000 7fce528fa700 1 --2- 172.21.15.42:0/318568562 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fce4c1232e0 0x7fce4c0a1230 secure :-1 s=THROTTLE_DONE pgs=210 cs=0 l=1 rx=0x7fce48001430 tx=0x7fce440100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:53.226 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.216+0000 7fce437fe700 1 -- 172.21.15.42:0/318568562 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fce48007430 con 0x7fce4c1232e0 2019-12-05T05:03:53.226 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.216+0000 7fce528fa700 1 --2- 172.21.15.42:0/318568562 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fce4c1232e0 0x7fce4c0a1230 secure :-1 s=THROTTLE_DONE pgs=210 cs=0 l=1 rx=0x7fce48001430 tx=0x7fce440100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:53.226 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.216+0000 7fce437fe700 1 --2- 172.21.15.42:0/318568562 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fce3403acb0 0x7fce3403d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:53.226 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.216+0000 7fce437fe700 1 -- 172.21.15.42:0/318568562 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6031+0+0 (secure 0 0 0) 0x7fce4804afe0 con 0x7fce4c1232e0 2019-12-05T05:03:53.226 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.216+0000 7fce520f9700 1 --2- 172.21.15.42:0/318568562 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fce3403acb0 0x7fce3403d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:53.226 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.216+0000 7fce520f9700 1 --2- 172.21.15.42:0/318568562 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fce3403acb0 0x7fce3403d1c0 crc :-1 s=READY pgs=110 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:53.227 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.216+0000 7fce5435d700 1 -- 172.21.15.42:0/318568562 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce4c05d110 con 0x7fce4c1232e0 2019-12-05T05:03:53.230 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.224+0000 7fce528fa700 1 --2- 172.21.15.42:0/318568562 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fce4c1232e0 0x7fce4c0a1230 secure :-1 s=THROTTLE_DONE pgs=210 cs=0 l=1 rx=0x7fce48001430 tx=0x7fce440100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:53.230 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.224+0000 7fce437fe700 1 -- 172.21.15.42:0/318568562 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fce48031fe0 con 0x7fce4c1232e0 2019-12-05T05:03:53.399 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.392+0000 7fce5435d700 1 -- 172.21.15.42:0/318568562 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7fce4c05b8c0 con 0x7fce4c1232e0 2019-12-05T05:03:53.399 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.392+0000 7fce528fa700 1 --2- 172.21.15.42:0/318568562 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fce4c1232e0 0x7fce4c0a1230 secure :-1 s=THROTTLE_DONE pgs=210 cs=0 l=1 rx=0x7fce48001430 tx=0x7fce440100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:53.399 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.392+0000 7fce437fe700 1 -- 172.21.15.42:0/318568562 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v24) v1 ==== 100+0+10 (secure 0 0 0) 0x7fce48032240 con 0x7fce4c1232e0 2019-12-05T05:03:53.400 INFO:teuthology.orchestra.run.smithi042.stdout:pg_num: 8 2019-12-05T05:03:53.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.396+0000 7fce5435d700 1 -- 172.21.15.42:0/318568562 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fce3403acb0 msgr2=0x7fce3403d1c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:53.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.396+0000 7fce5435d700 1 --2- 172.21.15.42:0/318568562 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fce3403acb0 0x7fce3403d1c0 crc :-1 s=READY pgs=110 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:53.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.396+0000 7fce5435d700 1 -- 172.21.15.42:0/318568562 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fce4c1232e0 msgr2=0x7fce4c0a1230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:53.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.396+0000 7fce5435d700 1 --2- 172.21.15.42:0/318568562 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fce4c1232e0 0x7fce4c0a1230 secure :-1 s=READY pgs=210 cs=0 l=1 rx=0x7fce48001430 tx=0x7fce440100b0).stop 2019-12-05T05:03:53.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.396+0000 7fce5435d700 1 -- 172.21.15.42:0/318568562 shutdown_connections 2019-12-05T05:03:53.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.396+0000 7fce5435d700 1 --2- 172.21.15.42:0/318568562 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fce3403acb0 0x7fce3403d1c0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:53.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.396+0000 7fce5435d700 1 --2- 172.21.15.42:0/318568562 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fce4c1232e0 0x7fce4c0a1230 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:53.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.396+0000 7fce5435d700 1 -- 172.21.15.42:0/318568562 shutdown_connections 2019-12-05T05:03:53.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.396+0000 7fce5435d700 1 -- 172.21.15.42:0/318568562 wait complete. 2019-12-05T05:03:53.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.396+0000 7fce5435d700 1 -- 172.21.15.42:0/318568562 >> 172.21.15.42:0/318568562 conn(0x7fce4c05aa70 msgr2=0x7fce4c0a5370 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:53.412 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2019-12-05T05:03:53.412 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:03:53.510 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.500+0000 7f9db7c5c700 1 Processor -- start 2019-12-05T05:03:53.511 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.500+0000 7f9db7c5c700 1 -- start start 2019-12-05T05:03:53.511 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.500+0000 7f9db7c5c700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9db00a7740 0x7f9db008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:03:53.511 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.500+0000 7f9db7c5c700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9db00a0710 con 0x7f9db007e720 2019-12-05T05:03:53.511 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.500+0000 7f9db7c5c700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9db00a9ba0 con 0x7f9db009fae0 2019-12-05T05:03:53.512 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.500+0000 7f9db7c5c700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9db00a0b00 con 0x7f9db00a7740 2019-12-05T05:03:53.512 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.500+0000 7f9db69fa700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f9db009fae0 0x7f9db014efe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:54034/0 (socket says 172.21.15.42:54034) 2019-12-05T05:03:53.512 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.500+0000 7f9db69fa700 1 -- 172.21.15.42:0/1550477079 learned_addr learned my addr 172.21.15.42:0/1550477079 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:03:53.512 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.500+0000 7f9db61f9700 1 --2- 172.21.15.42:0/1550477079 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9db00a7740 0x7f9db008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:53.512 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.500+0000 7f9db51f7700 1 -- 172.21.15.42:0/1550477079 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 927238916 0 0) 0x7f9db00a9ba0 con 0x7f9db009fae0 2019-12-05T05:03:53.512 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.500+0000 7f9db51f7700 1 -- 172.21.15.42:0/1550477079 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d9c0030b0 con 0x7f9db009fae0 2019-12-05T05:03:53.512 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.500+0000 7f9db61f9700 1 -- 172.21.15.42:0/1550477079 >> v1:172.21.15.42:6789/0 conn(0x7f9db009fae0 legacy=0x7f9db014efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:53.513 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.500+0000 7f9db61f9700 1 -- 172.21.15.42:0/1550477079 >> v1:172.21.15.40:6789/0 conn(0x7f9db007e720 legacy=0x7f9db009f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:53.513 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.500+0000 7f9db61f9700 1 -- 172.21.15.42:0/1550477079 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9db007eb50 con 0x7f9db00a7740 2019-12-05T05:03:53.513 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.504+0000 7f9db61f9700 1 --2- 172.21.15.42:0/1550477079 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9db00a7740 0x7f9db008df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f9dac002f30 tx=0x7f9dac00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:53.513 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.504+0000 7f9db61f9700 1 --2- 172.21.15.42:0/1550477079 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9db00a7740 0x7f9db008df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f9dac002f30 tx=0x7f9dac00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:53.513 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.504+0000 7f9db61f9700 1 --2- 172.21.15.42:0/1550477079 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9db00a7740 0x7f9db008df80 secure :-1 s=READY pgs=50 cs=0 l=1 rx=0x7f9dac002f30 tx=0x7f9dac00fbf0).ready entity=mon.2 client_cookie=951083908a1b5db7 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:53.513 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.504+0000 7f9db61f9700 1 --2- 172.21.15.42:0/1550477079 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9db00a7740 0x7f9db008df80 secure :-1 s=THROTTLE_DONE pgs=50 cs=0 l=1 rx=0x7f9dac002f30 tx=0x7f9dac00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:53.513 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.504+0000 7f9db51f7700 1 -- 172.21.15.42:0/1550477079 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9dac0112b0 con 0x7f9db00a7740 2019-12-05T05:03:53.514 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.504+0000 7f9db61f9700 1 --2- 172.21.15.42:0/1550477079 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9db00a7740 0x7f9db008df80 secure :-1 s=THROTTLE_DONE pgs=50 cs=0 l=1 rx=0x7f9dac002f30 tx=0x7f9dac00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:53.514 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.504+0000 7f9db61f9700 1 --2- 172.21.15.42:0/1550477079 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9db00a7740 0x7f9db008df80 secure :-1 s=THROTTLE_DONE pgs=50 cs=0 l=1 rx=0x7f9dac002f30 tx=0x7f9dac00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:53.514 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.504+0000 7f9db51f7700 1 -- 172.21.15.42:0/1550477079 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9dac008bc0 con 0x7f9db00a7740 2019-12-05T05:03:53.514 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.504+0000 7f9db51f7700 1 -- 172.21.15.42:0/1550477079 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9dac011890 con 0x7f9db00a7740 2019-12-05T05:03:53.514 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.504+0000 7f9db7c5c700 1 -- 172.21.15.42:0/1550477079 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9db00a7740 msgr2=0x7f9db008df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:53.514 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.504+0000 7f9db7c5c700 1 --2- 172.21.15.42:0/1550477079 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9db00a7740 0x7f9db008df80 secure :-1 s=READY pgs=50 cs=0 l=1 rx=0x7f9dac002f30 tx=0x7f9dac00fbf0).stop 2019-12-05T05:03:53.514 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.504+0000 7f9db7c5c700 1 -- 172.21.15.42:0/1550477079 shutdown_connections 2019-12-05T05:03:53.515 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.504+0000 7f9db7c5c700 1 --2- 172.21.15.42:0/1550477079 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9db00a7740 0x7f9db008df80 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:53.515 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.504+0000 7f9db7c5c700 1 -- 172.21.15.42:0/1550477079 shutdown_connections 2019-12-05T05:03:53.515 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.504+0000 7f9db7c5c700 1 -- 172.21.15.42:0/1550477079 wait complete. 2019-12-05T05:03:53.515 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.504+0000 7f9db7c5c700 1 -- 172.21.15.42:0/1550477079 >> 172.21.15.42:0/1550477079 conn(0x7f9db00aabe0 msgr2=0x7f9db00a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:53.518 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.508+0000 7f9db7c5c700 1 Processor -- start 2019-12-05T05:03:53.518 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.508+0000 7f9db7c5c700 1 -- start start 2019-12-05T05:03:53.518 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.508+0000 7f9db7c5c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f9db0145860 0x7f9db0145c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:53.518 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.508+0000 7f9db7c5c700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f9db012bd60 0x7f9db012e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:53.518 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.508+0000 7f9db7c5c700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9db00a5390 con 0x7f9db0118af0 2019-12-05T05:03:53.518 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.508+0000 7f9db7c5c700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9db01461e0 con 0x7f9db012bd60 2019-12-05T05:03:53.518 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.508+0000 7f9db7c5c700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9db0147ee0 con 0x7f9db0145860 2019-12-05T05:03:53.519 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.508+0000 7f9db59f8700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f9db0145860 0x7f9db0145c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:53.519 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.508+0000 7f9db59f8700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f9db0145860 0x7f9db0145c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:54000/0 (socket says 172.21.15.42:54000) 2019-12-05T05:03:53.519 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.508+0000 7f9db59f8700 1 -- 172.21.15.42:0/1073160195 learned_addr learned my addr 172.21.15.42:0/1073160195 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:03:53.519 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.508+0000 7f9db69fa700 1 --2- 172.21.15.42:0/1073160195 >> v2:172.21.15.40:3300/0 conn(0x7f9db012bd60 0x7f9db012e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:53.519 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.508+0000 7f9db59f8700 1 -- 172.21.15.42:0/1073160195 >> v2:172.21.15.40:3300/0 conn(0x7f9db012bd60 msgr2=0x7f9db012e1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:53.519 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.508+0000 7f9db59f8700 1 --2- 172.21.15.42:0/1073160195 >> v2:172.21.15.40:3300/0 conn(0x7f9db012bd60 0x7f9db012e1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:53.519 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.508+0000 7f9db59f8700 1 -- 172.21.15.42:0/1073160195 >> v1:172.21.15.40:6789/0 conn(0x7f9db0118af0 legacy=0x7f9db0127050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:53.520 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.508+0000 7f9db59f8700 1 -- 172.21.15.42:0/1073160195 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9db01448a0 con 0x7f9db0145860 2019-12-05T05:03:53.520 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.508+0000 7f9db59f8700 1 --2- 172.21.15.42:0/1073160195 >> v2:172.21.15.42:3300/0 conn(0x7f9db0145860 0x7f9db0145c90 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f9db005d370 tx=0x7f9da00109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:53.520 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.508+0000 7f9db59f8700 1 --2- 172.21.15.42:0/1073160195 >> v2:172.21.15.42:3300/0 conn(0x7f9db0145860 0x7f9db0145c90 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f9db005d370 tx=0x7f9da00109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:53.520 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.508+0000 7f9db59f8700 1 --2- 172.21.15.42:0/1073160195 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9db0145860 0x7f9db0145c90 secure :-1 s=READY pgs=212 cs=0 l=1 rx=0x7f9db005d370 tx=0x7f9da00109a0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:53.520 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.508+0000 7f9db59f8700 1 --2- 172.21.15.42:0/1073160195 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9db0145860 0x7f9db0145c90 secure :-1 s=THROTTLE_DONE pgs=212 cs=0 l=1 rx=0x7f9db005d370 tx=0x7f9da00109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:53.520 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.508+0000 7f9da6ffd700 1 -- 172.21.15.42:0/1073160195 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9da000c7a0 con 0x7f9db0145860 2019-12-05T05:03:53.520 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.508+0000 7f9db59f8700 1 --2- 172.21.15.42:0/1073160195 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9db0145860 0x7f9db0145c90 secure :-1 s=THROTTLE_DONE pgs=212 cs=0 l=1 rx=0x7f9db005d370 tx=0x7f9da00109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:53.521 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.508+0000 7f9db59f8700 1 --2- 172.21.15.42:0/1073160195 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9db0145860 0x7f9db0145c90 secure :-1 s=THROTTLE_DONE pgs=212 cs=0 l=1 rx=0x7f9db005d370 tx=0x7f9da00109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:53.521 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.508+0000 7f9da6ffd700 1 -- 172.21.15.42:0/1073160195 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9da000c3c0 con 0x7f9db0145860 2019-12-05T05:03:53.521 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.508+0000 7f9db7c5c700 1 -- 172.21.15.42:0/1073160195 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9db0145380 con 0x7f9db0145860 2019-12-05T05:03:53.521 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.508+0000 7f9da6ffd700 1 -- 172.21.15.42:0/1073160195 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9da000cd80 con 0x7f9db0145860 2019-12-05T05:03:53.522 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.508+0000 7f9db7c5c700 1 -- 172.21.15.42:0/1073160195 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9db00a5c30 con 0x7f9db0145860 2019-12-05T05:03:53.522 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.512+0000 7f9db59f8700 1 --2- 172.21.15.42:0/1073160195 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9db0145860 0x7f9db0145c90 secure :-1 s=THROTTLE_DONE pgs=212 cs=0 l=1 rx=0x7f9db005d370 tx=0x7f9da00109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:53.522 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.512+0000 7f9da6ffd700 1 -- 172.21.15.42:0/1073160195 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f9da0007440 con 0x7f9db0145860 2019-12-05T05:03:53.522 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.512+0000 7f9db59f8700 1 --2- 172.21.15.42:0/1073160195 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9db0145860 0x7f9db0145c90 secure :-1 s=THROTTLE_DONE pgs=212 cs=0 l=1 rx=0x7f9db005d370 tx=0x7f9da00109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:53.522 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.512+0000 7f9da6ffd700 1 --2- 172.21.15.42:0/1073160195 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9d9803adb0 0x7f9d9803d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:03:53.522 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.512+0000 7f9da6ffd700 1 -- 172.21.15.42:0/1073160195 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f9da004b2c0 con 0x7f9db0145860 2019-12-05T05:03:53.522 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.512+0000 7f9db61f9700 1 --2- 172.21.15.42:0/1073160195 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9d9803adb0 0x7f9d9803d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:03:53.523 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.512+0000 7f9db61f9700 1 --2- 172.21.15.42:0/1073160195 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9d9803adb0 0x7f9d9803d2c0 crc :-1 s=READY pgs=111 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:03:53.523 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.516+0000 7f9db7c5c700 1 -- 172.21.15.42:0/1073160195 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9db005d110 con 0x7f9db0145860 2019-12-05T05:03:53.526 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.520+0000 7f9db59f8700 1 --2- 172.21.15.42:0/1073160195 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9db0145860 0x7f9db0145c90 secure :-1 s=THROTTLE_DONE pgs=212 cs=0 l=1 rx=0x7f9db005d370 tx=0x7f9da00109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:53.526 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.520+0000 7f9da6ffd700 1 -- 172.21.15.42:0/1073160195 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f9da0032300 con 0x7f9db0145860 2019-12-05T05:03:53.690 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.684+0000 7f9db7c5c700 1 -- 172.21.15.42:0/1073160195 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9db005b8c0 con 0x7f9db0145860 2019-12-05T05:03:53.691 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.684+0000 7f9db59f8700 1 --2- 172.21.15.42:0/1073160195 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9db0145860 0x7f9db0145c90 secure :-1 s=THROTTLE_DONE pgs=212 cs=0 l=1 rx=0x7f9db005d370 tx=0x7f9da00109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:03:53.691 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.684+0000 7f9da6ffd700 1 -- 172.21.15.42:0/1073160195 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+2102 (secure 0 0 0) 0x7f9da0032560 con 0x7f9db0145860 2019-12-05T05:03:53.691 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 6 2019-12-05T05:03:53.691 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:03:53.691 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4354,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/2019200199","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2019200199},{"type":"v1","addr":"172.21.15.42:6835","nonce":2019200199}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":4}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:03:43.669739+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4341},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4341":{"gid":4341,"name":"b","rank":0,"incarnation":5,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6833/3296198322","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3296198322},{"type":"v1","addr":"172.21.15.40:6833","nonce":3296198322}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:03:53.694 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.688+0000 7f9db7c5c700 1 -- 172.21.15.42:0/1073160195 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9d9803adb0 msgr2=0x7f9d9803d2c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:53.694 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.688+0000 7f9db7c5c700 1 --2- 172.21.15.42:0/1073160195 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9d9803adb0 0x7f9d9803d2c0 crc :-1 s=READY pgs=111 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:53.694 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.688+0000 7f9db7c5c700 1 -- 172.21.15.42:0/1073160195 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9db0145860 msgr2=0x7f9db0145c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:03:53.694 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.688+0000 7f9db7c5c700 1 --2- 172.21.15.42:0/1073160195 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9db0145860 0x7f9db0145c90 secure :-1 s=READY pgs=212 cs=0 l=1 rx=0x7f9db005d370 tx=0x7f9da00109a0).stop 2019-12-05T05:03:53.695 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.688+0000 7f9db7c5c700 1 -- 172.21.15.42:0/1073160195 shutdown_connections 2019-12-05T05:03:53.695 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.688+0000 7f9db7c5c700 1 --2- 172.21.15.42:0/1073160195 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9d9803adb0 0x7f9d9803d2c0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:53.695 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.688+0000 7f9db7c5c700 1 --2- 172.21.15.42:0/1073160195 >> v2:172.21.15.40:3300/0 conn(0x7f9db012bd60 0x7f9db012e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:53.695 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.688+0000 7f9db7c5c700 1 --2- 172.21.15.42:0/1073160195 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9db0145860 0x7f9db0145c90 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:03:53.695 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.688+0000 7f9db7c5c700 1 -- 172.21.15.42:0/1073160195 shutdown_connections 2019-12-05T05:03:53.695 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.688+0000 7f9db7c5c700 1 -- 172.21.15.42:0/1073160195 wait complete. 2019-12-05T05:03:53.695 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:03:53.688+0000 7f9db7c5c700 1 -- 172.21.15.42:0/1073160195 >> 172.21.15.42:0/1073160195 conn(0x7f9db009ffb0 msgr2=0x7f9db0059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:03:53.905 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {u'session_autoclose': 300, u'balancer': u'', u'up': {u'mds_0': 4341}, u'last_failure_osd_epoch': 0, u'in': [0], u'last_failure': 0, u'max_file_size': 1099511627776, u'explicitly_allowed_features': 0, u'damaged': [], u'tableserver': 0, u'metadata_pool': 2, u'failed': [], u'epoch': 6, u'flags': 18, u'max_mds': 1, u'compat': {u'compat': {}, u'ro_compat': {}, u'incompat': {u'feature_10': u'snaprealm v2', u'feature_8': u'no anchor table', u'feature_9': u'file layout v2', u'feature_2': u'client writeable ranges', u'feature_3': u'default file layouts on dirs', u'feature_1': u'base v0.20', u'feature_6': u'dirfrag is stored in omap', u'feature_4': u'dir inode in separate object', u'feature_5': u'mds uses versioned encoding'}}, u'min_compat_client': u'0 (unknown)', u'data_pools': [3], u'info': {u'gid_4341': {u'export_targets': [], u'name': u'b', u'incarnation': 5, u'state_seq': 3, u'state': u'up:active', u'gid': 4341, u'features': 4540138292836696063, u'rank': 0, u'flags': 0, u'addrs': {u'addrvec': [{u'nonce': 3296198322, u'type': u'v2', u'addr': u'172.21.15.40:6832'}, {u'nonce': 3296198322, u'type': u'v1', u'addr': u'172.21.15.40:6833'}]}, u'addr': u'172.21.15.40:6833/3296198322'}}, u'fs_name': u'cephfs', u'created': u'2019-12-05T05:03:35.604738+0000', u'standby_count_wanted': 0, u'enabled': True, u'modified': u'2019-12-05T05:03:43.669739+0000', u'session_timeout': 60, u'stopped': [], u'ever_allowed_features': 0, u'root': 0} 2019-12-05T05:03:53.905 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2019-12-05T05:03:53.906 INFO:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2019-12-05T05:03:53.998 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2019-12-05T05:03:53.999 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_fsid": "5d836cfc-a83b-4433-9b3e-b9b2f31c9050", 2019-12-05T05:03:53.999 INFO:teuthology.orchestra.run.smithi040.stdout: "whoami": 0, 2019-12-05T05:03:53.999 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 4341, 2019-12-05T05:03:53.999 INFO:teuthology.orchestra.run.smithi040.stdout: "want_state": "up:active", 2019-12-05T05:03:54.000 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "up:active", 2019-12-05T05:03:54.000 INFO:teuthology.orchestra.run.smithi040.stdout: "rank_uptime": 11.329632397999999, 2019-12-05T05:03:54.000 INFO:teuthology.orchestra.run.smithi040.stdout: "mdsmap_epoch": 6, 2019-12-05T05:03:54.000 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch": 24, 2019-12-05T05:03:54.000 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch_barrier": 24, 2019-12-05T05:03:54.000 INFO:teuthology.orchestra.run.smithi040.stdout: "uptime": 15.980077476 2019-12-05T05:03:54.000 INFO:teuthology.orchestra.run.smithi040.stdout:} 2019-12-05T05:03:54.005 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "5d836cfc-a83b-4433-9b3e-b9b2f31c9050", "whoami": 0, "id": 4341, "want_state": "up:active", "state": "up:active", "rank_uptime": 11.329632397999999, "mdsmap_epoch": 6, "osdmap_epoch": 24, "osdmap_epoch_barrier": 24, "uptime": 15.980077476 } 2019-12-05T05:03:54.005 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2019-12-05T05:03:54.005 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2019-12-05T05:03:54.005 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@smithi042.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2019-12-05T05:03:54.006 INFO:teuthology.orchestra.run:Running command with timeout 900 2019-12-05T05:03:54.006 INFO:teuthology.orchestra.run.smithi042:> mkdir -- /home/ubuntu/cephtest/mnt.0 2019-12-05T05:03:54.019 INFO:teuthology.orchestra.run:Running command with timeout 900 2019-12-05T05:03:54.019 INFO:teuthology.orchestra.run.smithi042:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2019-12-05T05:03:54.098 DEBUG:teuthology.orchestra.run:got remote process result: 32 2019-12-05T05:03:54.101 INFO:teuthology.orchestra.run.smithi042.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2019-12-05T05:03:54.101 INFO:teuthology.orchestra.run:Running command with timeout 900 2019-12-05T05:03:54.101 INFO:teuthology.orchestra.run.smithi042:> ls /sys/fs/fuse/connections 2019-12-05T05:03:54.177 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2019-12-05T05:03:54.177 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0 2019-12-05T05:03:54.245 INFO:teuthology.orchestra.run:Running command with timeout 900 2019-12-05T05:03:54.245 INFO:teuthology.orchestra.run.smithi042:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2019-12-05T05:03:54.263 DEBUG:teuthology.orchestra.run:got remote process result: 32 2019-12-05T05:03:54.265 INFO:teuthology.orchestra.run.smithi042.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2019-12-05T05:03:54.266 INFO:teuthology.orchestra.run:Running command with timeout 900 2019-12-05T05:03:54.266 INFO:teuthology.orchestra.run.smithi042:> ls /sys/fs/fuse/connections 2019-12-05T05:03:54.308 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi042.stderr:2019-12-05T05:03:54.304+0000 7f6a6d52cf40 -1 init, newargv = 0x561d9a997030 newargc=7 2019-12-05T05:03:54.308 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi042.stderr:ceph-fuse[14738]: starting ceph client 2019-12-05T05:03:54.318 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi042.stderr:ceph-fuse[14738]: starting fuse 2019-12-05T05:03:54.337 INFO:teuthology.orchestra.run.smithi042.stdout:49 2019-12-05T05:03:54.337 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [49] 2019-12-05T05:03:54.338 INFO:teuthology.orchestra.run:Running command with timeout 900 2019-12-05T05:03:54.338 INFO:teuthology.orchestra.run.smithi042:> sudo python2 -c ' 2019-12-05T05:03:54.338 INFO:teuthology.orchestra.run.smithi042:> import glob 2019-12-05T05:03:54.338 INFO:teuthology.orchestra.run.smithi042:> import re 2019-12-05T05:03:54.338 INFO:teuthology.orchestra.run.smithi042:> import os 2019-12-05T05:03:54.338 INFO:teuthology.orchestra.run.smithi042:> import subprocess 2019-12-05T05:03:54.339 INFO:teuthology.orchestra.run.smithi042:> 2019-12-05T05:03:54.339 INFO:teuthology.orchestra.run.smithi042:> def find_socket(client_name): 2019-12-05T05:03:54.339 INFO:teuthology.orchestra.run.smithi042:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2019-12-05T05:03:54.339 INFO:teuthology.orchestra.run.smithi042:> files = glob.glob(asok_path) 2019-12-05T05:03:54.339 INFO:teuthology.orchestra.run.smithi042:> 2019-12-05T05:03:54.340 INFO:teuthology.orchestra.run.smithi042:> # Given a non-glob path, it better be there 2019-12-05T05:03:54.340 INFO:teuthology.orchestra.run.smithi042:> if "*" not in asok_path: 2019-12-05T05:03:54.340 INFO:teuthology.orchestra.run.smithi042:> assert(len(files) == 1) 2019-12-05T05:03:54.340 INFO:teuthology.orchestra.run.smithi042:> return files[0] 2019-12-05T05:03:54.340 INFO:teuthology.orchestra.run.smithi042:> 2019-12-05T05:03:54.340 INFO:teuthology.orchestra.run.smithi042:> for f in files: 2019-12-05T05:03:54.340 INFO:teuthology.orchestra.run.smithi042:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2019-12-05T05:03:54.340 INFO:teuthology.orchestra.run.smithi042:> if os.path.exists("/proc/{0}".format(pid)): 2019-12-05T05:03:54.341 INFO:teuthology.orchestra.run.smithi042:> return f 2019-12-05T05:03:54.341 INFO:teuthology.orchestra.run.smithi042:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2019-12-05T05:03:54.341 INFO:teuthology.orchestra.run.smithi042:> 2019-12-05T05:03:54.341 INFO:teuthology.orchestra.run.smithi042:> print(find_socket("client.0")) 2019-12-05T05:03:54.341 INFO:teuthology.orchestra.run.smithi042:> ' 2019-12-05T05:03:54.441 INFO:teuthology.orchestra.run.smithi042.stdout:/var/run/ceph/ceph-client.0.14738.asok 2019-12-05T05:03:54.474 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.14738.asok 2019-12-05T05:03:54.475 INFO:teuthology.orchestra.run:Running command with timeout 900 2019-12-05T05:03:54.475 INFO:teuthology.orchestra.run.smithi042:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.14738.asok status 2019-12-05T05:03:54.609 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2019-12-05T05:03:54.609 INFO:teuthology.orchestra.run.smithi042.stdout: "metadata": { 2019-12-05T05:03:54.609 INFO:teuthology.orchestra.run.smithi042.stdout: "ceph_sha1": "c5290805033483ba0cec165c06bde07dae5f19ec", 2019-12-05T05:03:54.610 INFO:teuthology.orchestra.run.smithi042.stdout: "ceph_version": "ceph version 15.0.0-8014-gc529080 (c5290805033483ba0cec165c06bde07dae5f19ec) octopus (dev)", 2019-12-05T05:03:54.610 INFO:teuthology.orchestra.run.smithi042.stdout: "entity_id": "0", 2019-12-05T05:03:54.610 INFO:teuthology.orchestra.run.smithi042.stdout: "hostname": "smithi042", 2019-12-05T05:03:54.610 INFO:teuthology.orchestra.run.smithi042.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2019-12-05T05:03:54.610 INFO:teuthology.orchestra.run.smithi042.stdout: "pid": "14738", 2019-12-05T05:03:54.610 INFO:teuthology.orchestra.run.smithi042.stdout: "root": "/" 2019-12-05T05:03:54.610 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2019-12-05T05:03:54.611 INFO:teuthology.orchestra.run.smithi042.stdout: "dentry_count": 0, 2019-12-05T05:03:54.611 INFO:teuthology.orchestra.run.smithi042.stdout: "dentry_pinned_count": 0, 2019-12-05T05:03:54.611 INFO:teuthology.orchestra.run.smithi042.stdout: "id": 4741, 2019-12-05T05:03:54.611 INFO:teuthology.orchestra.run.smithi042.stdout: "inst": { 2019-12-05T05:03:54.611 INFO:teuthology.orchestra.run.smithi042.stdout: "name": { 2019-12-05T05:03:54.611 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "client", 2019-12-05T05:03:54.611 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 4741 2019-12-05T05:03:54.611 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2019-12-05T05:03:54.612 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": { 2019-12-05T05:03:54.612 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2019-12-05T05:03:54.612 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:0", 2019-12-05T05:03:54.612 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 2187206384 2019-12-05T05:03:54.612 INFO:teuthology.orchestra.run.smithi042.stdout: } 2019-12-05T05:03:54.612 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2019-12-05T05:03:54.613 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": { 2019-12-05T05:03:54.613 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2019-12-05T05:03:54.613 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:0", 2019-12-05T05:03:54.613 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 2187206384 2019-12-05T05:03:54.613 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2019-12-05T05:03:54.613 INFO:teuthology.orchestra.run.smithi042.stdout: "inst_str": "client.4741 172.21.15.42:0/2187206384", 2019-12-05T05:03:54.613 INFO:teuthology.orchestra.run.smithi042.stdout: "addr_str": "172.21.15.42:0/2187206384", 2019-12-05T05:03:54.614 INFO:teuthology.orchestra.run.smithi042.stdout: "inode_count": 1, 2019-12-05T05:03:54.614 INFO:teuthology.orchestra.run.smithi042.stdout: "mds_epoch": 6, 2019-12-05T05:03:54.614 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_epoch": 24, 2019-12-05T05:03:54.614 INFO:teuthology.orchestra.run.smithi042.stdout: "osd_epoch_barrier": 0, 2019-12-05T05:03:54.614 INFO:teuthology.orchestra.run.smithi042.stdout: "blacklisted": false 2019-12-05T05:03:54.614 INFO:teuthology.orchestra.run.smithi042.stdout:} 2019-12-05T05:03:54.615 INFO:teuthology.orchestra.run:Running command with timeout 900 2019-12-05T05:03:54.615 INFO:teuthology.orchestra.run.smithi042:> stat --file-system '--printf=%T 2019-12-05T05:03:54.615 INFO:teuthology.orchestra.run.smithi042:> ' -- /home/ubuntu/cephtest/mnt.0 2019-12-05T05:03:54.627 INFO:teuthology.orchestra.run.smithi042.stdout:fuseblk 2019-12-05T05:03:54.627 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2019-12-05T05:03:54.627 INFO:teuthology.orchestra.run:Running command with timeout 900 2019-12-05T05:03:54.627 INFO:teuthology.orchestra.run.smithi042:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2019-12-05T05:03:54.711 INFO:teuthology.run_tasks:Running task workunit... 2019-12-05T05:03:54.725 INFO:tasks.workunit:Pulling workunits from ref c5290805033483ba0cec165c06bde07dae5f19ec 2019-12-05T05:03:54.725 INFO:tasks.workunit:Making a separate scratch dir for every client... 2019-12-05T05:03:54.725 INFO:tasks.workunit:timeout=3h 2019-12-05T05:03:54.725 INFO:tasks.workunit:cleanup=True 2019-12-05T05:03:54.726 INFO:teuthology.orchestra.run.smithi042:> stat -- /home/ubuntu/cephtest/mnt.0 2019-12-05T05:03:54.767 INFO:teuthology.orchestra.run.smithi042.stdout: File: /home/ubuntu/cephtest/mnt.0 2019-12-05T05:03:54.767 INFO:teuthology.orchestra.run.smithi042.stdout: Size: 0 Blocks: 1 IO Block: 4194304 directory 2019-12-05T05:03:54.767 INFO:teuthology.orchestra.run.smithi042.stdout:Device: 31h/49d Inode: 1 Links: 2 2019-12-05T05:03:54.767 INFO:teuthology.orchestra.run.smithi042.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2019-12-05T05:03:54.768 INFO:teuthology.orchestra.run.smithi042.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2019-12-05T05:03:54.768 INFO:teuthology.orchestra.run.smithi042.stdout:Modify: 2019-12-05 05:03:42.669424572 +0000 2019-12-05T05:03:54.768 INFO:teuthology.orchestra.run.smithi042.stdout:Change: 2019-12-05 05:03:54.706007379 +0000 2019-12-05T05:03:54.769 INFO:teuthology.orchestra.run.smithi042.stdout: Birth: - 2019-12-05T05:03:54.769 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2019-12-05T05:03:54.769 INFO:teuthology.orchestra.run.smithi042:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2019-12-05T05:03:54.865 INFO:teuthology.orchestra.run.smithi042:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone git://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout c5290805033483ba0cec165c06bde07dae5f19ec 2019-12-05T05:03:54.979 INFO:tasks.workunit.client.0.smithi042.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2019-12-05T05:04:03.536 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:04:03.540 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:04:28.183 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:04:28.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.276+0000 7f67ed854700 1 Processor -- start 2019-12-05T05:04:28.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.276+0000 7f67ed854700 1 -- start start 2019-12-05T05:04:28.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.280+0000 7f67ed854700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f67e809fae0 0x7f67e814fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:04:28.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.280+0000 7f67ed854700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67e80a0710 con 0x7f67e808df80 2019-12-05T05:04:28.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.280+0000 7f67ed854700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67e807ef20 con 0x7f67e80a7740 2019-12-05T05:04:28.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.280+0000 7f67ed854700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67e80a0b00 con 0x7f67e809fae0 2019-12-05T05:04:28.291 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.280+0000 7f67e77fe700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f67e80a7740 0x7f67e807e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:54084/0 (socket says 172.21.15.42:54084) 2019-12-05T05:04:28.291 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.280+0000 7f67e77fe700 1 -- 172.21.15.42:0/1826229737 learned_addr learned my addr 172.21.15.42:0/1826229737 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:04:28.291 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.280+0000 7f67e7fff700 1 --2- 172.21.15.42:0/1826229737 >> v2:172.21.15.40:3300/0 conn(0x7f67e809fae0 0x7f67e814fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:04:28.291 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.280+0000 7f67e67fc700 1 -- 172.21.15.42:0/1826229737 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 728612215 0 0) 0x7f67e807ef20 con 0x7f67e80a7740 2019-12-05T05:04:28.291 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.280+0000 7f67e67fc700 1 -- 172.21.15.42:0/1826229737 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67d00030b0 con 0x7f67e80a7740 2019-12-05T05:04:28.291 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.280+0000 7f67e7fff700 1 -- 172.21.15.42:0/1826229737 >> v1:172.21.15.42:6789/0 conn(0x7f67e80a7740 legacy=0x7f67e807e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:04:28.291 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.280+0000 7f67e67fc700 1 -- 172.21.15.42:0/1826229737 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3713008024 0 0) 0x7f67d00030b0 con 0x7f67e80a7740 2019-12-05T05:04:28.292 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.280+0000 7f67e7fff700 1 -- 172.21.15.42:0/1826229737 >> v1:172.21.15.40:6789/0 conn(0x7f67e808df80 legacy=0x7f67e809f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:04:28.292 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.280+0000 7f67e7fff700 1 -- 172.21.15.42:0/1826229737 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67e80a9ba0 con 0x7f67e809fae0 2019-12-05T05:04:28.292 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.280+0000 7f67e7fff700 1 --2- 172.21.15.42:0/1826229737 >> v2:172.21.15.40:3300/0 conn(0x7f67e809fae0 0x7f67e814fff0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f67dc001110 tx=0x7f67d8001820).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:28.292 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.280+0000 7f67e7fff700 1 --2- 172.21.15.42:0/1826229737 >> v2:172.21.15.40:3300/0 conn(0x7f67e809fae0 0x7f67e814fff0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f67dc001110 tx=0x7f67d8001820).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:28.292 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.280+0000 7f67e7fff700 1 --2- 172.21.15.42:0/1826229737 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f67e809fae0 0x7f67e814fff0 secure :-1 s=READY pgs=185 cs=0 l=1 rx=0x7f67dc001110 tx=0x7f67d8001820).ready entity=mon.0 client_cookie=67402e73b0543a0a server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:04:28.292 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.280+0000 7f67e7fff700 1 --2- 172.21.15.42:0/1826229737 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f67e809fae0 0x7f67e814fff0 secure :-1 s=THROTTLE_DONE pgs=185 cs=0 l=1 rx=0x7f67dc001110 tx=0x7f67d8001820).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:28.292 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.280+0000 7f67e67fc700 1 -- 172.21.15.42:0/1826229737 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f67d8001ad0 con 0x7f67e809fae0 2019-12-05T05:04:28.293 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.284+0000 7f67e7fff700 1 --2- 172.21.15.42:0/1826229737 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f67e809fae0 0x7f67e814fff0 secure :-1 s=THROTTLE_DONE pgs=185 cs=0 l=1 rx=0x7f67dc001110 tx=0x7f67d8001820).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:28.293 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.284+0000 7f67e67fc700 1 -- 172.21.15.42:0/1826229737 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67d80018e0 con 0x7f67e809fae0 2019-12-05T05:04:28.293 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.284+0000 7f67e7fff700 1 --2- 172.21.15.42:0/1826229737 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f67e809fae0 0x7f67e814fff0 secure :-1 s=THROTTLE_DONE pgs=185 cs=0 l=1 rx=0x7f67dc001110 tx=0x7f67d8001820).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:28.293 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.284+0000 7f67e67fc700 1 -- 172.21.15.42:0/1826229737 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f67e0008d60 con 0x7f67e809fae0 2019-12-05T05:04:28.293 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.284+0000 7f67ed854700 1 -- 172.21.15.42:0/1826229737 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f67e809fae0 msgr2=0x7f67e814fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:04:28.293 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.284+0000 7f67ed854700 1 --2- 172.21.15.42:0/1826229737 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f67e809fae0 0x7f67e814fff0 secure :-1 s=READY pgs=185 cs=0 l=1 rx=0x7f67dc001110 tx=0x7f67d8001820).stop 2019-12-05T05:04:28.293 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.284+0000 7f67ed854700 1 -- 172.21.15.42:0/1826229737 shutdown_connections 2019-12-05T05:04:28.294 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.284+0000 7f67ed854700 1 --2- 172.21.15.42:0/1826229737 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f67e809fae0 0x7f67e814fff0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:04:28.294 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.284+0000 7f67ed854700 1 -- 172.21.15.42:0/1826229737 shutdown_connections 2019-12-05T05:04:28.294 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.284+0000 7f67ed854700 1 -- 172.21.15.42:0/1826229737 wait complete. 2019-12-05T05:04:28.294 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.284+0000 7f67ed854700 1 -- 172.21.15.42:0/1826229737 >> 172.21.15.42:0/1826229737 conn(0x7f67e80aabe0 msgr2=0x7f67e80a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:04:28.299 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.288+0000 7f67ed854700 1 Processor -- start 2019-12-05T05:04:28.300 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.288+0000 7f67ed854700 1 -- start start 2019-12-05T05:04:28.300 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.288+0000 7f67ed854700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f67e812d1e0 0x7f67e807d110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:04:28.300 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.288+0000 7f67ed854700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67e807d660 con 0x7f67e811c280 2019-12-05T05:04:28.300 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.288+0000 7f67ed854700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67e807d890 con 0x7f67e8119f60 2019-12-05T05:04:28.300 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.288+0000 7f67ed854700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67e807bb50 con 0x7f67e812d1e0 2019-12-05T05:04:28.300 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.288+0000 7f67e77fe700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f67e811c280 0x7f67e807bdf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:35298/0 (socket says 172.21.15.42:35298) 2019-12-05T05:04:28.301 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.288+0000 7f67e6ffd700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f67e8119f60 0x7f67e812da20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:54092/0 (socket says 172.21.15.42:54092) 2019-12-05T05:04:28.301 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.288+0000 7f67e77fe700 1 -- 172.21.15.42:0/1868168610 learned_addr learned my addr 172.21.15.42:0/1868168610 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:04:28.301 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.288+0000 7f67e7fff700 1 --2- 172.21.15.42:0/1868168610 >> v2:172.21.15.40:3300/0 conn(0x7f67e812d1e0 0x7f67e807d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:04:28.301 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.288+0000 7f67cffff700 1 -- 172.21.15.42:0/1868168610 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 609753040 0 0) 0x7f67e807d890 con 0x7f67e8119f60 2019-12-05T05:04:28.301 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.288+0000 7f67cffff700 1 -- 172.21.15.42:0/1868168610 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67c80030b0 con 0x7f67e8119f60 2019-12-05T05:04:28.301 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.288+0000 7f67cffff700 1 -- 172.21.15.42:0/1868168610 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2605750729 0 0) 0x7f67e807d660 con 0x7f67e811c280 2019-12-05T05:04:28.301 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.288+0000 7f67cffff700 1 -- 172.21.15.42:0/1868168610 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67e807d890 con 0x7f67e811c280 2019-12-05T05:04:28.302 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.288+0000 7f67cffff700 1 -- 172.21.15.42:0/1868168610 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 1322214911 0 0) 0x7f67c80030b0 con 0x7f67e8119f60 2019-12-05T05:04:28.302 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.288+0000 7f67cffff700 1 -- 172.21.15.42:0/1868168610 --> v1:172.21.15.42:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f67e807d660 con 0x7f67e8119f60 2019-12-05T05:04:28.302 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.288+0000 7f67cffff700 1 -- 172.21.15.42:0/1868168610 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7f67dc0019d0 con 0x7f67e8119f60 2019-12-05T05:04:28.302 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.288+0000 7f67e7fff700 1 -- 172.21.15.42:0/1868168610 >> v1:172.21.15.42:6789/0 conn(0x7f67e8119f60 legacy=0x7f67e812da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:04:28.302 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.288+0000 7f67e7fff700 1 -- 172.21.15.42:0/1868168610 >> v1:172.21.15.40:6789/0 conn(0x7f67e811c280 legacy=0x7f67e807bdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:04:28.302 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.288+0000 7f67e7fff700 1 -- 172.21.15.42:0/1868168610 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67e807dac0 con 0x7f67e812d1e0 2019-12-05T05:04:28.302 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.288+0000 7f67e7fff700 1 --2- 172.21.15.42:0/1868168610 >> v2:172.21.15.40:3300/0 conn(0x7f67e812d1e0 0x7f67e807d110 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f67d80020a0 tx=0x7f67d8001350).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:28.303 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.288+0000 7f67e7fff700 1 --2- 172.21.15.42:0/1868168610 >> v2:172.21.15.40:3300/0 conn(0x7f67e812d1e0 0x7f67e807d110 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f67d80020a0 tx=0x7f67d8001350).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:28.303 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.288+0000 7f67e7fff700 1 --2- 172.21.15.42:0/1868168610 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f67e812d1e0 0x7f67e807d110 secure :-1 s=READY pgs=187 cs=0 l=1 rx=0x7f67d80020a0 tx=0x7f67d8001350).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:04:28.303 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.292+0000 7f67e7fff700 1 --2- 172.21.15.42:0/1868168610 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f67e812d1e0 0x7f67e807d110 secure :-1 s=THROTTLE_DONE pgs=187 cs=0 l=1 rx=0x7f67d80020a0 tx=0x7f67d8001350).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:28.303 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.292+0000 7f67cffff700 1 -- 172.21.15.42:0/1868168610 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f67e0016060 con 0x7f67e812d1e0 2019-12-05T05:04:28.303 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.292+0000 7f67e7fff700 1 --2- 172.21.15.42:0/1868168610 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f67e812d1e0 0x7f67e807d110 secure :-1 s=THROTTLE_DONE pgs=187 cs=0 l=1 rx=0x7f67d80020a0 tx=0x7f67d8001350).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:28.303 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.292+0000 7f67e7fff700 1 --2- 172.21.15.42:0/1868168610 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f67e812d1e0 0x7f67e807d110 secure :-1 s=THROTTLE_DONE pgs=187 cs=0 l=1 rx=0x7f67d80020a0 tx=0x7f67d8001350).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:28.303 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.292+0000 7f67cffff700 1 -- 172.21.15.42:0/1868168610 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67d8001cb0 con 0x7f67e812d1e0 2019-12-05T05:04:28.304 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.292+0000 7f67cffff700 1 -- 172.21.15.42:0/1868168610 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f67d8001ea0 con 0x7f67e812d1e0 2019-12-05T05:04:28.304 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.292+0000 7f67ed854700 1 -- 172.21.15.42:0/1868168610 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67e813bc30 con 0x7f67e812d1e0 2019-12-05T05:04:28.304 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.292+0000 7f67ed854700 1 -- 172.21.15.42:0/1868168610 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67e813c180 con 0x7f67e812d1e0 2019-12-05T05:04:28.304 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.292+0000 7f67e7fff700 1 --2- 172.21.15.42:0/1868168610 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f67e812d1e0 0x7f67e807d110 secure :-1 s=THROTTLE_DONE pgs=187 cs=0 l=1 rx=0x7f67d80020a0 tx=0x7f67d8001350).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:28.304 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.292+0000 7f67cffff700 1 -- 172.21.15.42:0/1868168610 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f67e00115e0 con 0x7f67e812d1e0 2019-12-05T05:04:28.304 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.292+0000 7f67e7fff700 1 --2- 172.21.15.42:0/1868168610 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f67e812d1e0 0x7f67e807d110 secure :-1 s=THROTTLE_DONE pgs=187 cs=0 l=1 rx=0x7f67d80020a0 tx=0x7f67d8001350).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:28.304 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.292+0000 7f67cffff700 1 --2- 172.21.15.42:0/1868168610 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f67c803b940 0x7f67c803de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:04:28.305 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.292+0000 7f67cffff700 1 -- 172.21.15.42:0/1868168610 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f67e004a6d0 con 0x7f67e812d1e0 2019-12-05T05:04:28.305 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.292+0000 7f67e77fe700 1 --2- 172.21.15.42:0/1868168610 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f67c803b940 0x7f67c803de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:04:28.305 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.292+0000 7f67ed854700 1 -- 172.21.15.42:0/1868168610 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67e805d110 con 0x7f67e812d1e0 2019-12-05T05:04:28.305 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.292+0000 7f67e77fe700 1 --2- 172.21.15.42:0/1868168610 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f67c803b940 0x7f67c803de50 crc :-1 s=READY pgs=112 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:04:28.309 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.304+0000 7f67e7fff700 1 --2- 172.21.15.42:0/1868168610 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f67e812d1e0 0x7f67e807d110 secure :-1 s=THROTTLE_DONE pgs=187 cs=0 l=1 rx=0x7f67d80020a0 tx=0x7f67d8001350).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:28.309 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.304+0000 7f67cffff700 1 -- 172.21.15.42:0/1868168610 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f67e0031750 con 0x7f67e812d1e0 2019-12-05T05:04:28.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.472+0000 7f67ed854700 1 -- 172.21.15.42:0/1868168610 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f67e805b8c0 con 0x7f67e812d1e0 2019-12-05T05:04:28.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.472+0000 7f67e7fff700 1 --2- 172.21.15.42:0/1868168610 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f67e812d1e0 0x7f67e807d110 secure :-1 s=THROTTLE_DONE pgs=187 cs=0 l=1 rx=0x7f67d80020a0 tx=0x7f67d8001350).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:28.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.472+0000 7f67cffff700 1 -- 172.21.15.42:0/1868168610 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+2102 (secure 0 0 0) 0x7f67e0007700 con 0x7f67e812d1e0 2019-12-05T05:04:28.485 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:04:28.485 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4354,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/2019200199","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2019200199},{"type":"v1","addr":"172.21.15.42:6835","nonce":2019200199}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":4}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:03:43.669739+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4341},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4341":{"gid":4341,"name":"b","rank":0,"incarnation":5,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6833/3296198322","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3296198322},{"type":"v1","addr":"172.21.15.40:6833","nonce":3296198322}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:04:28.485 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 6 2019-12-05T05:04:28.485 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.476+0000 7f67ed854700 1 -- 172.21.15.42:0/1868168610 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f67c803b940 msgr2=0x7f67c803de50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:04:28.486 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.476+0000 7f67ed854700 1 --2- 172.21.15.42:0/1868168610 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f67c803b940 0x7f67c803de50 crc :-1 s=READY pgs=112 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:04:28.486 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.476+0000 7f67ed854700 1 -- 172.21.15.42:0/1868168610 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f67e812d1e0 msgr2=0x7f67e807d110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:04:28.486 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.476+0000 7f67ed854700 1 --2- 172.21.15.42:0/1868168610 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f67e812d1e0 0x7f67e807d110 secure :-1 s=READY pgs=187 cs=0 l=1 rx=0x7f67d80020a0 tx=0x7f67d8001350).stop 2019-12-05T05:04:28.487 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.480+0000 7f67ed854700 1 -- 172.21.15.42:0/1868168610 shutdown_connections 2019-12-05T05:04:28.487 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.480+0000 7f67ed854700 1 --2- 172.21.15.42:0/1868168610 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f67c803b940 0x7f67c803de50 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:04:28.487 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.480+0000 7f67ed854700 1 --2- 172.21.15.42:0/1868168610 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f67e812d1e0 0x7f67e807d110 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:04:28.488 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.480+0000 7f67ed854700 1 -- 172.21.15.42:0/1868168610 shutdown_connections 2019-12-05T05:04:28.488 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.480+0000 7f67ed854700 1 -- 172.21.15.42:0/1868168610 wait complete. 2019-12-05T05:04:28.488 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.480+0000 7f67ed854700 1 -- 172.21.15.42:0/1868168610 >> 172.21.15.42:0/1868168610 conn(0x7f67e80a82a0 msgr2=0x7f67e8059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:04:28.495 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.b (rank=0) 2019-12-05T05:04:28.496 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2019-12-05T05:04:28.499 INFO:teuthology.orchestra.run:waiting for 300 2019-12-05T05:04:28.508 INFO:tasks.ceph.mds.b:Stopped 2019-12-05T05:04:28.509 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:04:28.607 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.596+0000 7f985c4d1700 1 Processor -- start 2019-12-05T05:04:28.607 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.596+0000 7f985c4d1700 1 -- start start 2019-12-05T05:04:28.607 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.596+0000 7f985c4d1700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f985408df80 0x7f985409f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:04:28.608 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.596+0000 7f985c4d1700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f985409f830 0x7f985414efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:04:28.608 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.596+0000 7f985c4d1700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f985409fc60 con 0x7f98540a7740 2019-12-05T05:04:28.608 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.596+0000 7f985c4d1700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98540a0b00 con 0x7f985408df80 2019-12-05T05:04:28.608 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.596+0000 7f985c4d1700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f985408e5b0 con 0x7f985409f830 2019-12-05T05:04:28.608 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.596+0000 7f985aa6e700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f98540a7740 0x7f985407e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:54100/0 (socket says 172.21.15.42:54100) 2019-12-05T05:04:28.608 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.596+0000 7f985aa6e700 1 -- 172.21.15.42:0/42012240 learned_addr learned my addr 172.21.15.42:0/42012240 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:04:28.609 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.596+0000 7f985b26f700 1 --2- 172.21.15.42:0/42012240 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f985409f830 0x7f985414efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:04:28.609 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.596+0000 7f985a26d700 1 --2- 172.21.15.42:0/42012240 >> v2:172.21.15.40:3300/0 conn(0x7f985408df80 0x7f985409f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:04:28.609 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.596+0000 7f9859a6c700 1 -- 172.21.15.42:0/42012240 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2314504860 0 0) 0x7f985409fc60 con 0x7f98540a7740 2019-12-05T05:04:28.609 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.596+0000 7f9859a6c700 1 -- 172.21.15.42:0/42012240 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f983c0030b0 con 0x7f98540a7740 2019-12-05T05:04:28.609 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.596+0000 7f9859a6c700 1 -- 172.21.15.42:0/42012240 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3614607853 0 0) 0x7f983c0030b0 con 0x7f98540a7740 2019-12-05T05:04:28.610 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.596+0000 7f9859a6c700 1 -- 172.21.15.42:0/42012240 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f985409f830 msgr2=0x7f985414efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:04:28.610 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.596+0000 7f9859a6c700 1 --2- 172.21.15.42:0/42012240 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f985409f830 0x7f985414efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:04:28.610 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.596+0000 7f9859a6c700 1 -- 172.21.15.42:0/42012240 >> v2:172.21.15.40:3300/0 conn(0x7f985408df80 msgr2=0x7f985409f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:04:28.610 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.596+0000 7f9859a6c700 1 --2- 172.21.15.42:0/42012240 >> v2:172.21.15.40:3300/0 conn(0x7f985408df80 0x7f985409f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:04:28.610 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.596+0000 7f9859a6c700 1 -- 172.21.15.42:0/42012240 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98540a0710 con 0x7f98540a7740 2019-12-05T05:04:28.610 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.596+0000 7f9859a6c700 1 -- 172.21.15.42:0/42012240 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7f9848001750 con 0x7f98540a7740 2019-12-05T05:04:28.611 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.596+0000 7f9859a6c700 1 -- 172.21.15.42:0/42012240 >> v1:172.21.15.42:6789/0 conn(0x7f98540a7740 legacy=0x7f985407e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:04:28.611 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.596+0000 7f9859a6c700 1 --2- 172.21.15.42:0/42012240 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f983c0036e0 0x7f983c003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:04:28.611 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.596+0000 7f9859a6c700 1 --2- 172.21.15.42:0/42012240 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f983c004420 0x7f983c004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:04:28.611 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.596+0000 7f9859a6c700 1 --2- 172.21.15.42:0/42012240 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f983c004da0 0x7f983c04f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:04:28.612 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.596+0000 7f9859a6c700 1 -- 172.21.15.42:0/42012240 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98540a0b00 con 0x7f983c0036e0 2019-12-05T05:04:28.612 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.596+0000 7f9859a6c700 1 -- 172.21.15.42:0/42012240 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f985408e5b0 con 0x7f983c004da0 2019-12-05T05:04:28.612 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.596+0000 7f9859a6c700 1 -- 172.21.15.42:0/42012240 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f983c005250 con 0x7f983c004420 2019-12-05T05:04:28.612 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.596+0000 7f985b26f700 1 --2- 172.21.15.42:0/42012240 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f983c004da0 0x7f983c04f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:04:28.624 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.596+0000 7f985a26d700 1 --2- 172.21.15.42:0/42012240 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f983c004420 0x7f983c004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:04:28.624 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.600+0000 7f985aa6e700 1 --2- 172.21.15.42:0/42012240 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f983c0036e0 0x7f983c003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:04:28.625 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.600+0000 7f985b26f700 1 -- 172.21.15.42:0/42012240 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f983c004420 msgr2=0x7f983c004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:04:28.625 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.600+0000 7f985b26f700 1 --2- 172.21.15.42:0/42012240 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f983c004420 0x7f983c004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:04:28.625 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.600+0000 7f985b26f700 1 -- 172.21.15.42:0/42012240 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f983c0036e0 msgr2=0x7f983c003b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:04:28.625 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.600+0000 7f985b26f700 1 --2- 172.21.15.42:0/42012240 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f983c0036e0 0x7f983c003b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:04:28.625 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.600+0000 7f985b26f700 1 -- 172.21.15.42:0/42012240 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f983c04fd10 con 0x7f983c004da0 2019-12-05T05:04:28.625 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.600+0000 7f985aa6e700 1 --2- 172.21.15.42:0/42012240 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f983c0036e0 0x7f983c003b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T05:04:28.626 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.600+0000 7f985b26f700 1 --2- 172.21.15.42:0/42012240 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f983c004da0 0x7f983c04f680 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f984c002f30 tx=0x7f984c00ffa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:28.626 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.600+0000 7f985b26f700 1 --2- 172.21.15.42:0/42012240 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f983c004da0 0x7f983c04f680 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f984c002f30 tx=0x7f984c00ffa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:28.626 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.600+0000 7f985b26f700 1 --2- 172.21.15.42:0/42012240 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f983c004da0 0x7f983c04f680 secure :-1 s=READY pgs=219 cs=0 l=1 rx=0x7f984c002f30 tx=0x7f984c00ffa0).ready entity=mon.1 client_cookie=6c6a989d62ea6d7b server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:04:28.626 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.600+0000 7f985b26f700 1 --2- 172.21.15.42:0/42012240 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f983c004da0 0x7f983c04f680 secure :-1 s=THROTTLE_DONE pgs=219 cs=0 l=1 rx=0x7f984c002f30 tx=0x7f984c00ffa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:28.626 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.600+0000 7f9859a6c700 1 -- 172.21.15.42:0/42012240 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f984c010cd0 con 0x7f983c004da0 2019-12-05T05:04:28.626 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.600+0000 7f985b26f700 1 --2- 172.21.15.42:0/42012240 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f983c004da0 0x7f983c04f680 secure :-1 s=THROTTLE_DONE pgs=219 cs=0 l=1 rx=0x7f984c002f30 tx=0x7f984c00ffa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:28.626 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.600+0000 7f9859a6c700 1 -- 172.21.15.42:0/42012240 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f984c0108f0 con 0x7f983c004da0 2019-12-05T05:04:28.627 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.600+0000 7f985b26f700 1 --2- 172.21.15.42:0/42012240 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f983c004da0 0x7f983c04f680 secure :-1 s=THROTTLE_DONE pgs=219 cs=0 l=1 rx=0x7f984c002f30 tx=0x7f984c00ffa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:28.627 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.600+0000 7f9859a6c700 1 -- 172.21.15.42:0/42012240 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f984c008bb0 con 0x7f983c004da0 2019-12-05T05:04:28.627 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.600+0000 7f985c4d1700 1 -- 172.21.15.42:0/42012240 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f983c004da0 msgr2=0x7f983c04f680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:04:28.636 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.600+0000 7f985c4d1700 1 --2- 172.21.15.42:0/42012240 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f983c004da0 0x7f983c04f680 secure :-1 s=READY pgs=219 cs=0 l=1 rx=0x7f984c002f30 tx=0x7f984c00ffa0).stop 2019-12-05T05:04:28.636 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.600+0000 7f985b26f700 1 -- 172.21.15.42:0/42012240 reap_dead start 2019-12-05T05:04:28.636 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.600+0000 7f985c4d1700 1 -- 172.21.15.42:0/42012240 shutdown_connections 2019-12-05T05:04:28.636 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.600+0000 7f985c4d1700 1 -- 172.21.15.42:0/42012240 shutdown_connections 2019-12-05T05:04:28.637 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.600+0000 7f985c4d1700 1 -- 172.21.15.42:0/42012240 wait complete. 2019-12-05T05:04:28.637 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.600+0000 7f985c4d1700 1 -- 172.21.15.42:0/42012240 >> 172.21.15.42:0/42012240 conn(0x7f98540aabe0 msgr2=0x7f98540a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:04:28.643 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.604+0000 7f985c4d1700 1 Processor -- start 2019-12-05T05:04:28.643 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.604+0000 7f985c4d1700 1 -- start start 2019-12-05T05:04:28.643 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.604+0000 7f985c4d1700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f98540a6ea0 0x7f985412d200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:04:28.644 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.604+0000 7f985c4d1700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f98541310b0 0x7f9854141480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:04:28.644 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.604+0000 7f985c4d1700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98541488c0 con 0x7f9854143ad0 2019-12-05T05:04:28.644 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.604+0000 7f985c4d1700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98541308d0 con 0x7f98541310b0 2019-12-05T05:04:28.644 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.604+0000 7f985c4d1700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9854130b70 con 0x7f98540a6ea0 2019-12-05T05:04:28.644 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.604+0000 7f985aa6e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f98540a6ea0 0x7f985412d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:04:28.644 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.604+0000 7f985a26d700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f9854143ad0 0x7f98540653f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:54116/0 (socket says 172.21.15.42:54116) 2019-12-05T05:04:28.644 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.604+0000 7f985a26d700 1 -- 172.21.15.42:0/780198966 learned_addr learned my addr 172.21.15.42:0/780198966 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:04:28.645 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.604+0000 7f985b26f700 1 --2- 172.21.15.42:0/780198966 >> v2:172.21.15.40:3300/0 conn(0x7f98541310b0 0x7f9854141480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:04:28.645 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.604+0000 7f985aa6e700 1 -- 172.21.15.42:0/780198966 >> v2:172.21.15.40:3300/0 conn(0x7f98541310b0 msgr2=0x7f9854141480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:04:28.645 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.604+0000 7f985aa6e700 1 --2- 172.21.15.42:0/780198966 >> v2:172.21.15.40:3300/0 conn(0x7f98541310b0 0x7f9854141480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:04:28.645 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.604+0000 7f985aa6e700 1 -- 172.21.15.42:0/780198966 >> v1:172.21.15.42:6789/0 conn(0x7f9854143ad0 legacy=0x7f98540653f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:04:28.645 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.604+0000 7f98537fe700 1 -- 172.21.15.42:0/780198966 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 962713453 0 0) 0x7f98541488c0 con 0x7f9854143ad0 2019-12-05T05:04:28.645 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.604+0000 7f985aa6e700 1 -- 172.21.15.42:0/780198966 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f985405ae30 con 0x7f98540a6ea0 2019-12-05T05:04:28.646 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.604+0000 7f985aa6e700 1 --2- 172.21.15.42:0/780198966 >> v2:172.21.15.42:3300/0 conn(0x7f98540a6ea0 0x7f985412d200 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f9854143070 tx=0x7f9848010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:28.646 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.604+0000 7f985aa6e700 1 --2- 172.21.15.42:0/780198966 >> v2:172.21.15.42:3300/0 conn(0x7f98540a6ea0 0x7f985412d200 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f9854143070 tx=0x7f9848010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:28.647 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.604+0000 7f985aa6e700 1 --2- 172.21.15.42:0/780198966 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f98540a6ea0 0x7f985412d200 secure :-1 s=READY pgs=221 cs=0 l=1 rx=0x7f9854143070 tx=0x7f9848010ab0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:04:28.647 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.604+0000 7f985aa6e700 1 --2- 172.21.15.42:0/780198966 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f98540a6ea0 0x7f985412d200 secure :-1 s=THROTTLE_DONE pgs=221 cs=0 l=1 rx=0x7f9854143070 tx=0x7f9848010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:28.647 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.604+0000 7f98537fe700 1 -- 172.21.15.42:0/780198966 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f984800c820 con 0x7f98540a6ea0 2019-12-05T05:04:28.647 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.604+0000 7f985aa6e700 1 --2- 172.21.15.42:0/780198966 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f98540a6ea0 0x7f985412d200 secure :-1 s=THROTTLE_DONE pgs=221 cs=0 l=1 rx=0x7f9854143070 tx=0x7f9848010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:28.647 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.604+0000 7f98537fe700 1 -- 172.21.15.42:0/780198966 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9848010170 con 0x7f98540a6ea0 2019-12-05T05:04:28.647 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.604+0000 7f985aa6e700 1 --2- 172.21.15.42:0/780198966 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f98540a6ea0 0x7f985412d200 secure :-1 s=THROTTLE_DONE pgs=221 cs=0 l=1 rx=0x7f9854143070 tx=0x7f9848010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:28.647 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.604+0000 7f98537fe700 1 -- 172.21.15.42:0/780198966 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f984800ce00 con 0x7f98540a6ea0 2019-12-05T05:04:28.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.604+0000 7f985c4d1700 1 -- 172.21.15.42:0/780198966 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9854143f00 con 0x7f98540a6ea0 2019-12-05T05:04:28.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.604+0000 7f985c4d1700 1 -- 172.21.15.42:0/780198966 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98541314e0 con 0x7f98540a6ea0 2019-12-05T05:04:28.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.604+0000 7f985aa6e700 1 --2- 172.21.15.42:0/780198966 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f98540a6ea0 0x7f985412d200 secure :-1 s=THROTTLE_DONE pgs=221 cs=0 l=1 rx=0x7f9854143070 tx=0x7f9848010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:28.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.604+0000 7f98537fe700 1 -- 172.21.15.42:0/780198966 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f9848012440 con 0x7f98540a6ea0 2019-12-05T05:04:28.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.608+0000 7f985aa6e700 1 --2- 172.21.15.42:0/780198966 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f98540a6ea0 0x7f985412d200 secure :-1 s=THROTTLE_DONE pgs=221 cs=0 l=1 rx=0x7f9854143070 tx=0x7f9848010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:28.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.608+0000 7f98537fe700 1 --2- 172.21.15.42:0/780198966 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f983403ac40 0x7f983403d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:04:28.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.608+0000 7f98537fe700 1 -- 172.21.15.42:0/780198966 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f984804ad00 con 0x7f98540a6ea0 2019-12-05T05:04:28.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.608+0000 7f985a26d700 1 --2- 172.21.15.42:0/780198966 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f983403ac40 0x7f983403d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:04:28.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.608+0000 7f985a26d700 1 --2- 172.21.15.42:0/780198966 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f983403ac40 0x7f983403d150 crc :-1 s=READY pgs=113 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:04:28.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.612+0000 7f985c4d1700 1 -- 172.21.15.42:0/780198966 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f985405d110 con 0x7f98540a6ea0 2019-12-05T05:04:28.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.616+0000 7f985aa6e700 1 --2- 172.21.15.42:0/780198966 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f98540a6ea0 0x7f985412d200 secure :-1 s=THROTTLE_DONE pgs=221 cs=0 l=1 rx=0x7f9854143070 tx=0x7f9848010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:28.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.616+0000 7f98537fe700 1 -- 172.21.15.42:0/780198966 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f9848031d60 con 0x7f98540a6ea0 2019-12-05T05:04:28.785 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.780+0000 7f985c4d1700 1 -- 172.21.15.42:0/780198966 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f985405b8c0 con 0x7f98540a6ea0 2019-12-05T05:04:28.787 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.780+0000 7f985aa6e700 1 --2- 172.21.15.42:0/780198966 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f98540a6ea0 0x7f985412d200 secure :-1 s=THROTTLE_DONE pgs=221 cs=0 l=1 rx=0x7f9854143070 tx=0x7f9848010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:28.787 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.780+0000 7f98537fe700 1 -- 172.21.15.42:0/780198966 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+2102 (secure 0 0 0) 0x7f9848031fc0 con 0x7f98540a6ea0 2019-12-05T05:04:28.787 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 6 2019-12-05T05:04:28.788 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:04:28.788 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4354,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/2019200199","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2019200199},{"type":"v1","addr":"172.21.15.42:6835","nonce":2019200199}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":4}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:03:43.669739+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4341},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4341":{"gid":4341,"name":"b","rank":0,"incarnation":5,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6833/3296198322","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3296198322},{"type":"v1","addr":"172.21.15.40:6833","nonce":3296198322}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:04:28.790 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.784+0000 7f985c4d1700 1 -- 172.21.15.42:0/780198966 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f983403ac40 msgr2=0x7f983403d150 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:04:28.790 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.784+0000 7f985c4d1700 1 --2- 172.21.15.42:0/780198966 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f983403ac40 0x7f983403d150 crc :-1 s=READY pgs=113 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:04:28.790 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.784+0000 7f985c4d1700 1 -- 172.21.15.42:0/780198966 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f98540a6ea0 msgr2=0x7f985412d200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:04:28.791 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.784+0000 7f985c4d1700 1 --2- 172.21.15.42:0/780198966 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f98540a6ea0 0x7f985412d200 secure :-1 s=READY pgs=221 cs=0 l=1 rx=0x7f9854143070 tx=0x7f9848010ab0).stop 2019-12-05T05:04:28.791 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.784+0000 7f985c4d1700 1 -- 172.21.15.42:0/780198966 shutdown_connections 2019-12-05T05:04:28.791 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.784+0000 7f985c4d1700 1 --2- 172.21.15.42:0/780198966 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f983403ac40 0x7f983403d150 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:04:28.791 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.784+0000 7f985c4d1700 1 --2- 172.21.15.42:0/780198966 >> v2:172.21.15.40:3300/0 conn(0x7f98541310b0 0x7f9854141480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:04:28.791 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.784+0000 7f985c4d1700 1 --2- 172.21.15.42:0/780198966 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f98540a6ea0 0x7f985412d200 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:04:28.792 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.784+0000 7f985c4d1700 1 -- 172.21.15.42:0/780198966 shutdown_connections 2019-12-05T05:04:28.793 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.784+0000 7f985c4d1700 1 -- 172.21.15.42:0/780198966 wait complete. 2019-12-05T05:04:28.793 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:28.784+0000 7f985c4d1700 1 -- 172.21.15.42:0/780198966 >> 172.21.15.42:0/780198966 conn(0x7f98540a98e0 msgr2=0x7f985407be90 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:04:28.800 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T05:04:30.802 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:04:30.903 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:30.892+0000 7f2e5d16a700 1 Processor -- start 2019-12-05T05:04:30.903 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:30.892+0000 7f2e5d16a700 1 -- start start 2019-12-05T05:04:30.903 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:30.896+0000 7f2e5d16a700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2e580a7740 0x7f2e5808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:04:30.903 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:30.896+0000 7f2e5d16a700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2e5807e720 0x7f2e5807eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:04:30.903 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:30.896+0000 7f2e5d16a700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2e5809f2e0 0x7f2e5809f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:04:30.903 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:30.896+0000 7f2e5d16a700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e580a0b00 con 0x7f2e580a7740 2019-12-05T05:04:30.904 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:30.896+0000 7f2e5d16a700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e5808e5b0 con 0x7f2e5807e720 2019-12-05T05:04:30.904 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:30.896+0000 7f2e5d16a700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e5808e850 con 0x7f2e5809f2e0 2019-12-05T05:04:30.904 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:30.896+0000 7f2e4ed9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2e5807e720 0x7f2e5807eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:04:30.904 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:30.896+0000 7f2e4ed9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2e5807e720 0x7f2e5807eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:54092/0 (socket says 172.21.15.42:54092) 2019-12-05T05:04:30.904 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:30.896+0000 7f2e4ed9d700 1 -- 172.21.15.42:0/406891755 learned_addr learned my addr 172.21.15.42:0/406891755 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:04:30.904 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:30.896+0000 7f2e5c969700 1 --2- 172.21.15.42:0/406891755 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2e5809f2e0 0x7f2e5809f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:04:30.904 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:30.896+0000 7f2e5759e700 1 --2- 172.21.15.42:0/406891755 >> v2:172.21.15.40:3300/0 conn(0x7f2e580a7740 0x7f2e5808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:04:30.905 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:30.896+0000 7f2e4ed9d700 1 -- 172.21.15.42:0/406891755 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2e5809f2e0 msgr2=0x7f2e5809f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:04:30.905 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:30.896+0000 7f2e4ed9d700 1 --2- 172.21.15.42:0/406891755 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2e5809f2e0 0x7f2e5809f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:04:30.905 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:30.896+0000 7f2e4ed9d700 1 -- 172.21.15.42:0/406891755 >> v2:172.21.15.40:3300/0 conn(0x7f2e580a7740 msgr2=0x7f2e5808df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:04:30.905 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:30.896+0000 7f2e4ed9d700 1 --2- 172.21.15.42:0/406891755 >> v2:172.21.15.40:3300/0 conn(0x7f2e580a7740 0x7f2e5808df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:04:30.905 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:30.896+0000 7f2e4ed9d700 1 -- 172.21.15.42:0/406891755 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e5807f0a0 con 0x7f2e5807e720 2019-12-05T05:04:30.905 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:30.896+0000 7f2e5759e700 1 --2- 172.21.15.42:0/406891755 >> v2:172.21.15.40:3300/0 conn(0x7f2e580a7740 0x7f2e5808df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T05:04:30.905 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:30.896+0000 7f2e4ed9d700 1 --2- 172.21.15.42:0/406891755 >> v2:172.21.15.42:3300/0 conn(0x7f2e5807e720 0x7f2e5807eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f2e580a0f50 tx=0x7f2e4000fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:30.906 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:30.896+0000 7f2e4ed9d700 1 --2- 172.21.15.42:0/406891755 >> v2:172.21.15.42:3300/0 conn(0x7f2e5807e720 0x7f2e5807eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f2e580a0f50 tx=0x7f2e4000fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:30.906 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:30.896+0000 7f2e4ed9d700 1 --2- 172.21.15.42:0/406891755 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2e5807e720 0x7f2e5807eb50 secure :-1 s=READY pgs=222 cs=0 l=1 rx=0x7f2e580a0f50 tx=0x7f2e4000fdb0).ready entity=mon.1 client_cookie=78c279a66c7cb8e server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:04:30.906 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:30.896+0000 7f2e4ed9d700 1 --2- 172.21.15.42:0/406891755 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2e5807e720 0x7f2e5807eb50 secure :-1 s=THROTTLE_DONE pgs=222 cs=0 l=1 rx=0x7f2e580a0f50 tx=0x7f2e4000fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:30.906 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:30.896+0000 7f2e56d9d700 1 -- 172.21.15.42:0/406891755 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2e40008c90 con 0x7f2e5807e720 2019-12-05T05:04:30.906 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:30.896+0000 7f2e4ed9d700 1 --2- 172.21.15.42:0/406891755 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2e5807e720 0x7f2e5807eb50 secure :-1 s=THROTTLE_DONE pgs=222 cs=0 l=1 rx=0x7f2e580a0f50 tx=0x7f2e4000fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:30.906 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:30.896+0000 7f2e4ed9d700 1 --2- 172.21.15.42:0/406891755 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2e5807e720 0x7f2e5807eb50 secure :-1 s=THROTTLE_DONE pgs=222 cs=0 l=1 rx=0x7f2e580a0f50 tx=0x7f2e4000fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:30.906 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:30.896+0000 7f2e56d9d700 1 -- 172.21.15.42:0/406891755 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e400088b0 con 0x7f2e5807e720 2019-12-05T05:04:30.907 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:30.896+0000 7f2e56d9d700 1 -- 172.21.15.42:0/406891755 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2e40011b50 con 0x7f2e5807e720 2019-12-05T05:04:30.907 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:30.896+0000 7f2e5d16a700 1 -- 172.21.15.42:0/406891755 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2e5807e720 msgr2=0x7f2e5807eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:04:30.907 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:30.896+0000 7f2e5d16a700 1 --2- 172.21.15.42:0/406891755 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2e5807e720 0x7f2e5807eb50 secure :-1 s=READY pgs=222 cs=0 l=1 rx=0x7f2e580a0f50 tx=0x7f2e4000fdb0).stop 2019-12-05T05:04:30.907 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:30.896+0000 7f2e5d16a700 1 -- 172.21.15.42:0/406891755 shutdown_connections 2019-12-05T05:04:30.907 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:30.896+0000 7f2e5d16a700 1 --2- 172.21.15.42:0/406891755 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2e5809f2e0 0x7f2e5809f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:04:30.907 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:30.896+0000 7f2e5d16a700 1 --2- 172.21.15.42:0/406891755 >> v2:172.21.15.40:3300/0 conn(0x7f2e580a7740 0x7f2e5808df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:04:30.907 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:30.896+0000 7f2e5d16a700 1 --2- 172.21.15.42:0/406891755 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2e5807e720 0x7f2e5807eb50 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:04:30.907 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:30.896+0000 7f2e5d16a700 1 -- 172.21.15.42:0/406891755 shutdown_connections 2019-12-05T05:04:30.908 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:30.896+0000 7f2e5d16a700 1 -- 172.21.15.42:0/406891755 wait complete. 2019-12-05T05:04:30.908 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:30.896+0000 7f2e5d16a700 1 -- 172.21.15.42:0/406891755 >> 172.21.15.42:0/406891755 conn(0x7f2e580aabe0 msgr2=0x7f2e580a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:04:30.908 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:30.896+0000 7f2e5d16a700 1 Processor -- start 2019-12-05T05:04:30.908 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:30.896+0000 7f2e5d16a700 1 -- start start 2019-12-05T05:04:30.915 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:30.900+0000 7f2e5d16a700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2e5811a830 0x7f2e58123490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:04:30.916 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:30.900+0000 7f2e5d16a700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2e58126f10 0x7f2e5812d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:04:30.916 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:30.900+0000 7f2e5d16a700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e5807cc60 con 0x7f2e581239e0 2019-12-05T05:04:30.916 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:30.900+0000 7f2e5d16a700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e5807bdf0 con 0x7f2e58126f10 2019-12-05T05:04:30.916 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:30.900+0000 7f2e5d16a700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e5807c9c0 con 0x7f2e5811a830 2019-12-05T05:04:30.916 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:30.900+0000 7f2e5c969700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2e58126f10 0x7f2e5812d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:04:30.916 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:30.900+0000 7f2e5c969700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2e58126f10 0x7f2e5812d980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:54102/0 (socket says 172.21.15.42:54102) 2019-12-05T05:04:30.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:30.900+0000 7f2e5c969700 1 -- 172.21.15.42:0/2365765893 learned_addr learned my addr 172.21.15.42:0/2365765893 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:04:30.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:30.900+0000 7f2e5759e700 1 --2- 172.21.15.42:0/2365765893 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2e5811a830 0x7f2e58123490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:04:30.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:30.900+0000 7f2e5c969700 1 -- 172.21.15.42:0/2365765893 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2e5811a830 msgr2=0x7f2e58123490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:04:30.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:30.900+0000 7f2e5c969700 1 --2- 172.21.15.42:0/2365765893 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2e5811a830 0x7f2e58123490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:04:30.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:30.900+0000 7f2e5c969700 1 -- 172.21.15.42:0/2365765893 >> v1:172.21.15.40:6789/0 conn(0x7f2e581239e0 legacy=0x7f2e58126710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:04:30.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:30.900+0000 7f2e5c969700 1 -- 172.21.15.42:0/2365765893 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e5807d8f0 con 0x7f2e58126f10 2019-12-05T05:04:30.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:30.900+0000 7f2e5c969700 1 --2- 172.21.15.42:0/2365765893 >> v2:172.21.15.42:3300/0 conn(0x7f2e58126f10 0x7f2e5812d980 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f2e5807bfb0 tx=0x7f2e50010c40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:30.918 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:30.900+0000 7f2e5c969700 1 --2- 172.21.15.42:0/2365765893 >> v2:172.21.15.42:3300/0 conn(0x7f2e58126f10 0x7f2e5812d980 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f2e5807bfb0 tx=0x7f2e50010c40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:30.918 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:30.900+0000 7f2e5c969700 1 --2- 172.21.15.42:0/2365765893 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2e58126f10 0x7f2e5812d980 secure :-1 s=READY pgs=223 cs=0 l=1 rx=0x7f2e5807bfb0 tx=0x7f2e50010c40).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:04:30.918 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:30.900+0000 7f2e5c969700 1 --2- 172.21.15.42:0/2365765893 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2e58126f10 0x7f2e5812d980 secure :-1 s=THROTTLE_DONE pgs=223 cs=0 l=1 rx=0x7f2e5807bfb0 tx=0x7f2e50010c40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:30.918 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:30.900+0000 7f2e5c969700 1 --2- 172.21.15.42:0/2365765893 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2e58126f10 0x7f2e5812d980 secure :-1 s=THROTTLE_DONE pgs=223 cs=0 l=1 rx=0x7f2e5807bfb0 tx=0x7f2e50010c40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:30.918 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:30.900+0000 7f2e54d99700 1 -- 172.21.15.42:0/2365765893 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2e50007350 con 0x7f2e58126f10 2019-12-05T05:04:30.918 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:30.900+0000 7f2e5c969700 1 --2- 172.21.15.42:0/2365765893 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2e58126f10 0x7f2e5812d980 secure :-1 s=THROTTLE_DONE pgs=223 cs=0 l=1 rx=0x7f2e5807bfb0 tx=0x7f2e50010c40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:30.918 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:30.900+0000 7f2e54d99700 1 -- 172.21.15.42:0/2365765893 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e50005cb0 con 0x7f2e58126f10 2019-12-05T05:04:30.918 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:30.900+0000 7f2e5d16a700 1 -- 172.21.15.42:0/2365765893 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e5811ac60 con 0x7f2e58126f10 2019-12-05T05:04:30.919 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:30.900+0000 7f2e54d99700 1 -- 172.21.15.42:0/2365765893 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2e500124b0 con 0x7f2e58126f10 2019-12-05T05:04:30.919 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:30.900+0000 7f2e5d16a700 1 -- 172.21.15.42:0/2365765893 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e58123fb0 con 0x7f2e58126f10 2019-12-05T05:04:30.919 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:30.900+0000 7f2e5c969700 1 --2- 172.21.15.42:0/2365765893 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2e58126f10 0x7f2e5812d980 secure :-1 s=THROTTLE_DONE pgs=223 cs=0 l=1 rx=0x7f2e5807bfb0 tx=0x7f2e50010c40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:30.919 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:30.900+0000 7f2e54d99700 1 -- 172.21.15.42:0/2365765893 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f2e50012780 con 0x7f2e58126f10 2019-12-05T05:04:30.919 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:30.900+0000 7f2e5c969700 1 --2- 172.21.15.42:0/2365765893 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2e58126f10 0x7f2e5812d980 secure :-1 s=THROTTLE_DONE pgs=223 cs=0 l=1 rx=0x7f2e5807bfb0 tx=0x7f2e50010c40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:30.919 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:30.900+0000 7f2e54d99700 1 --2- 172.21.15.42:0/2365765893 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2e3003adb0 0x7f2e3003d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:04:30.919 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:30.900+0000 7f2e54d99700 1 -- 172.21.15.42:0/2365765893 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f2e5004b2d0 con 0x7f2e58126f10 2019-12-05T05:04:30.920 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:30.900+0000 7f2e5759e700 1 --2- 172.21.15.42:0/2365765893 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2e3003adb0 0x7f2e3003d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:04:30.920 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:30.900+0000 7f2e5759e700 1 --2- 172.21.15.42:0/2365765893 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2e3003adb0 0x7f2e3003d2c0 crc :-1 s=READY pgs=114 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:04:30.920 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:30.908+0000 7f2e5d16a700 1 -- 172.21.15.42:0/2365765893 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e5811a2e0 con 0x7f2e58126f10 2019-12-05T05:04:30.926 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:30.916+0000 7f2e5c969700 1 --2- 172.21.15.42:0/2365765893 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2e58126f10 0x7f2e5812d980 secure :-1 s=THROTTLE_DONE pgs=223 cs=0 l=1 rx=0x7f2e5807bfb0 tx=0x7f2e50010c40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:30.926 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:30.916+0000 7f2e54d99700 1 -- 172.21.15.42:0/2365765893 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f2e50032350 con 0x7f2e58126f10 2019-12-05T05:04:31.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:31.096+0000 7f2e5d16a700 1 -- 172.21.15.42:0/2365765893 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2e5805d110 con 0x7f2e58126f10 2019-12-05T05:04:31.103 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:31.096+0000 7f2e5c969700 1 --2- 172.21.15.42:0/2365765893 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2e58126f10 0x7f2e5812d980 secure :-1 s=THROTTLE_DONE pgs=223 cs=0 l=1 rx=0x7f2e5807bfb0 tx=0x7f2e50010c40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:31.104 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:31.096+0000 7f2e54d99700 1 -- 172.21.15.42:0/2365765893 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+2102 (secure 0 0 0) 0x7f2e500325b0 con 0x7f2e58126f10 2019-12-05T05:04:31.104 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 6 2019-12-05T05:04:31.104 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:04:31.105 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4354,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/2019200199","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2019200199},{"type":"v1","addr":"172.21.15.42:6835","nonce":2019200199}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":4}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:03:43.669739+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4341},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4341":{"gid":4341,"name":"b","rank":0,"incarnation":5,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6833/3296198322","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3296198322},{"type":"v1","addr":"172.21.15.40:6833","nonce":3296198322}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:04:31.107 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:31.100+0000 7f2e5d16a700 1 -- 172.21.15.42:0/2365765893 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2e3003adb0 msgr2=0x7f2e3003d2c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:04:31.107 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:31.100+0000 7f2e5d16a700 1 --2- 172.21.15.42:0/2365765893 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2e3003adb0 0x7f2e3003d2c0 crc :-1 s=READY pgs=114 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:04:31.107 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:31.100+0000 7f2e5d16a700 1 -- 172.21.15.42:0/2365765893 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2e58126f10 msgr2=0x7f2e5812d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:04:31.107 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:31.100+0000 7f2e5d16a700 1 --2- 172.21.15.42:0/2365765893 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2e58126f10 0x7f2e5812d980 secure :-1 s=READY pgs=223 cs=0 l=1 rx=0x7f2e5807bfb0 tx=0x7f2e50010c40).stop 2019-12-05T05:04:31.107 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:31.100+0000 7f2e5d16a700 1 -- 172.21.15.42:0/2365765893 shutdown_connections 2019-12-05T05:04:31.108 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:31.100+0000 7f2e5d16a700 1 --2- 172.21.15.42:0/2365765893 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2e3003adb0 0x7f2e3003d2c0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:04:31.108 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:31.100+0000 7f2e5d16a700 1 --2- 172.21.15.42:0/2365765893 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2e58126f10 0x7f2e5812d980 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:04:31.108 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:31.100+0000 7f2e5d16a700 1 --2- 172.21.15.42:0/2365765893 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2e5811a830 0x7f2e58123490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:04:31.108 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:31.100+0000 7f2e5d16a700 1 -- 172.21.15.42:0/2365765893 shutdown_connections 2019-12-05T05:04:31.109 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:31.100+0000 7f2e5d16a700 1 -- 172.21.15.42:0/2365765893 wait complete. 2019-12-05T05:04:31.109 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:31.100+0000 7f2e5d16a700 1 -- 172.21.15.42:0/2365765893 >> 172.21.15.42:0/2365765893 conn(0x7f2e5808fda0 msgr2=0x7f2e5807e580 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:04:31.115 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T05:04:33.116 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:04:33.217 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:33.208+0000 7f563bca6700 1 Processor -- start 2019-12-05T05:04:33.217 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:33.208+0000 7f563bca6700 1 -- start start 2019-12-05T05:04:33.217 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:33.208+0000 7f563bca6700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f56340a7730 0x7f563408df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:04:33.217 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:33.208+0000 7f563bca6700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f563407e710 0x7f563407eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:04:33.218 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:33.208+0000 7f563bca6700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f563409f2d0 0x7f563409f700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:04:33.218 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:33.208+0000 7f563bca6700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56340a0af0 con 0x7f56340a7730 2019-12-05T05:04:33.218 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:33.208+0000 7f563bca6700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f563408e5a0 con 0x7f563409f2d0 2019-12-05T05:04:33.218 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:33.208+0000 7f563bca6700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f563408e840 con 0x7f563407e710 2019-12-05T05:04:33.231 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:33.208+0000 7f563aa44700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f563409f2d0 0x7f563409f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:04:33.231 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:33.208+0000 7f563aa44700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f563409f2d0 0x7f563409f700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:54112/0 (socket says 172.21.15.42:54112) 2019-12-05T05:04:33.232 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:33.208+0000 7f563aa44700 1 -- 172.21.15.42:0/4211909293 learned_addr learned my addr 172.21.15.42:0/4211909293 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:04:33.232 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:33.208+0000 7f5639a42700 1 --2- 172.21.15.42:0/4211909293 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f563407e710 0x7f563407eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:04:33.232 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:33.208+0000 7f563a243700 1 --2- 172.21.15.42:0/4211909293 >> v2:172.21.15.40:3300/0 conn(0x7f56340a7730 0x7f563408df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:04:33.232 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:33.208+0000 7f563aa44700 1 -- 172.21.15.42:0/4211909293 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f563407e710 msgr2=0x7f563407eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:04:33.233 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:33.208+0000 7f563aa44700 1 --2- 172.21.15.42:0/4211909293 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f563407e710 0x7f563407eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:04:33.233 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:33.208+0000 7f563aa44700 1 -- 172.21.15.42:0/4211909293 >> v2:172.21.15.40:3300/0 conn(0x7f56340a7730 msgr2=0x7f563408df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:04:33.233 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:33.208+0000 7f563aa44700 1 --2- 172.21.15.42:0/4211909293 >> v2:172.21.15.40:3300/0 conn(0x7f56340a7730 0x7f563408df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:04:33.233 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:33.208+0000 7f563aa44700 1 -- 172.21.15.42:0/4211909293 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f563407f090 con 0x7f563409f2d0 2019-12-05T05:04:33.234 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:33.208+0000 7f563aa44700 1 --2- 172.21.15.42:0/4211909293 >> v2:172.21.15.42:3300/0 conn(0x7f563409f2d0 0x7f563409f700 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f563408e760 tx=0x7f562c010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:33.234 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:33.208+0000 7f563aa44700 1 --2- 172.21.15.42:0/4211909293 >> v2:172.21.15.42:3300/0 conn(0x7f563409f2d0 0x7f563409f700 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f563408e760 tx=0x7f562c010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:33.234 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:33.208+0000 7f563aa44700 1 --2- 172.21.15.42:0/4211909293 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f563409f2d0 0x7f563409f700 secure :-1 s=READY pgs=224 cs=0 l=1 rx=0x7f563408e760 tx=0x7f562c010130).ready entity=mon.1 client_cookie=a2e05c00e3f18de7 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:04:33.234 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:33.208+0000 7f563aa44700 1 --2- 172.21.15.42:0/4211909293 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f563409f2d0 0x7f563409f700 secure :-1 s=THROTTLE_DONE pgs=224 cs=0 l=1 rx=0x7f563408e760 tx=0x7f562c010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:33.234 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:33.208+0000 7f5639241700 1 -- 172.21.15.42:0/4211909293 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f562c010de0 con 0x7f563409f2d0 2019-12-05T05:04:33.235 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:33.208+0000 7f563aa44700 1 --2- 172.21.15.42:0/4211909293 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f563409f2d0 0x7f563409f700 secure :-1 s=THROTTLE_DONE pgs=224 cs=0 l=1 rx=0x7f563408e760 tx=0x7f562c010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:33.235 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:33.208+0000 7f5639241700 1 -- 172.21.15.42:0/4211909293 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f562c010a00 con 0x7f563409f2d0 2019-12-05T05:04:33.235 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:33.208+0000 7f563aa44700 1 --2- 172.21.15.42:0/4211909293 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f563409f2d0 0x7f563409f700 secure :-1 s=THROTTLE_DONE pgs=224 cs=0 l=1 rx=0x7f563408e760 tx=0x7f562c010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:33.235 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:33.208+0000 7f5639241700 1 -- 172.21.15.42:0/4211909293 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f562c0114a0 con 0x7f563409f2d0 2019-12-05T05:04:33.235 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:33.208+0000 7f563bca6700 1 -- 172.21.15.42:0/4211909293 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f563409f2d0 msgr2=0x7f563409f700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:04:33.236 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:33.208+0000 7f563bca6700 1 --2- 172.21.15.42:0/4211909293 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f563409f2d0 0x7f563409f700 secure :-1 s=READY pgs=224 cs=0 l=1 rx=0x7f563408e760 tx=0x7f562c010130).stop 2019-12-05T05:04:33.236 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:33.212+0000 7f563bca6700 1 -- 172.21.15.42:0/4211909293 shutdown_connections 2019-12-05T05:04:33.241 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:33.212+0000 7f563bca6700 1 --2- 172.21.15.42:0/4211909293 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f563409f2d0 0x7f563409f700 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:04:33.241 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:33.212+0000 7f563bca6700 1 --2- 172.21.15.42:0/4211909293 >> v2:172.21.15.40:3300/0 conn(0x7f56340a7730 0x7f563408df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:04:33.241 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:33.212+0000 7f563bca6700 1 --2- 172.21.15.42:0/4211909293 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f563407e710 0x7f563407eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:04:33.242 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:33.212+0000 7f563bca6700 1 -- 172.21.15.42:0/4211909293 shutdown_connections 2019-12-05T05:04:33.242 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:33.212+0000 7f563bca6700 1 -- 172.21.15.42:0/4211909293 wait complete. 2019-12-05T05:04:33.242 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:33.212+0000 7f563bca6700 1 -- 172.21.15.42:0/4211909293 >> 172.21.15.42:0/4211909293 conn(0x7f56340aabd0 msgr2=0x7f56340a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:04:33.251 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:33.212+0000 7f563bca6700 1 Processor -- start 2019-12-05T05:04:33.252 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:33.212+0000 7f563bca6700 1 -- start start 2019-12-05T05:04:33.252 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:33.216+0000 7f563bca6700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5634076af0 0x7f5634078f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:04:33.252 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:33.216+0000 7f563bca6700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56341182a0 con 0x7f56340a97d0 2019-12-05T05:04:33.253 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:33.216+0000 7f563bca6700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5634123550 con 0x7f5634131d70 2019-12-05T05:04:33.253 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:33.216+0000 7f563bca6700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5634123780 con 0x7f5634076af0 2019-12-05T05:04:33.253 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:33.216+0000 7f5639a42700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f5634131d70 0x7f56340752e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:54154/0 (socket says 172.21.15.42:54154) 2019-12-05T05:04:33.254 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:33.216+0000 7f5639a42700 1 -- 172.21.15.42:0/2275129607 learned_addr learned my addr 172.21.15.42:0/2275129607 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:04:33.254 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:33.216+0000 7f563aa44700 1 --2- 172.21.15.42:0/2275129607 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5634076af0 0x7f5634078f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:04:33.255 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:33.216+0000 7f562affd700 1 -- 172.21.15.42:0/2275129607 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2331995187 0 0) 0x7f5634123550 con 0x7f5634131d70 2019-12-05T05:04:33.255 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:33.216+0000 7f562affd700 1 -- 172.21.15.42:0/2275129607 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f561c0030b0 con 0x7f5634131d70 2019-12-05T05:04:33.255 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:33.216+0000 7f562affd700 1 -- 172.21.15.42:0/2275129607 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3981697124 0 0) 0x7f56341182a0 con 0x7f56340a97d0 2019-12-05T05:04:33.255 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:33.216+0000 7f562affd700 1 -- 172.21.15.42:0/2275129607 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5634123550 con 0x7f56340a97d0 2019-12-05T05:04:33.256 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:33.216+0000 7f562affd700 1 -- 172.21.15.42:0/2275129607 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 1090109915 0 0) 0x7f561c0030b0 con 0x7f5634131d70 2019-12-05T05:04:33.256 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:33.216+0000 7f562affd700 1 -- 172.21.15.42:0/2275129607 --> v1:172.21.15.42:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f56341182a0 con 0x7f5634131d70 2019-12-05T05:04:33.256 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:33.216+0000 7f562affd700 1 -- 172.21.15.42:0/2275129607 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7f5624002640 con 0x7f5634131d70 2019-12-05T05:04:33.256 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:33.216+0000 7f562affd700 1 -- 172.21.15.42:0/2275129607 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5634076af0 msgr2=0x7f5634078f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:04:33.257 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:33.216+0000 7f562affd700 1 --2- 172.21.15.42:0/2275129607 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5634076af0 0x7f5634078f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:04:33.257 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:33.216+0000 7f562affd700 1 -- 172.21.15.42:0/2275129607 >> v1:172.21.15.42:6789/0 conn(0x7f5634131d70 legacy=0x7f56340752e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:04:33.258 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:33.216+0000 7f562affd700 1 -- 172.21.15.42:0/2275129607 >> v1:172.21.15.40:6789/0 conn(0x7f56340a97d0 legacy=0x7f5634131570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:04:33.258 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:33.216+0000 7f562affd700 1 --2- 172.21.15.42:0/2275129607 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f561c003ab0 0x7f561c004420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:04:33.258 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:33.216+0000 7f562affd700 1 --2- 172.21.15.42:0/2275129607 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f561c004970 0x7f561c004da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:04:33.258 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:33.216+0000 7f562affd700 1 --2- 172.21.15.42:0/2275129607 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f561c009660 0x7f561c00baa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:04:33.259 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:33.216+0000 7f562affd700 1 -- 172.21.15.42:0/2275129607 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5634123780 con 0x7f561c004970 2019-12-05T05:04:33.259 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:33.216+0000 7f562affd700 1 -- 172.21.15.42:0/2275129607 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f561c00bff0 con 0x7f561c003ab0 2019-12-05T05:04:33.259 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:33.216+0000 7f562affd700 1 -- 172.21.15.42:0/2275129607 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f561c00c1b0 con 0x7f561c009660 2019-12-05T05:04:33.259 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:33.216+0000 7f563a243700 1 --2- 172.21.15.42:0/2275129607 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f561c003ab0 0x7f561c004420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:04:33.260 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:33.216+0000 7f5639a42700 1 --2- 172.21.15.42:0/2275129607 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f561c004970 0x7f561c004da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:04:33.260 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:33.216+0000 7f563aa44700 1 --2- 172.21.15.42:0/2275129607 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f561c009660 0x7f561c00baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:04:33.260 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:33.216+0000 7f563a243700 1 -- 172.21.15.42:0/2275129607 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f561c009660 msgr2=0x7f561c00baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:04:33.260 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:33.216+0000 7f563a243700 1 --2- 172.21.15.42:0/2275129607 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f561c009660 0x7f561c00baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:04:33.261 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:33.216+0000 7f563a243700 1 -- 172.21.15.42:0/2275129607 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f561c004970 msgr2=0x7f561c004da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:04:33.261 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:33.216+0000 7f563a243700 1 --2- 172.21.15.42:0/2275129607 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f561c004970 0x7f561c004da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:04:33.261 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:33.216+0000 7f563a243700 1 -- 172.21.15.42:0/2275129607 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5634121ea0 con 0x7f561c003ab0 2019-12-05T05:04:33.261 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:33.216+0000 7f563bca6700 1 -- 172.21.15.42:0/2275129607 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5634076af0 con 0x7f561c003ab0 2019-12-05T05:04:33.261 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:33.216+0000 7f563bca6700 1 -- 172.21.15.42:0/2275129607 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5634077040 con 0x7f561c003ab0 2019-12-05T05:04:33.262 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:33.216+0000 7f563a243700 1 --2- 172.21.15.42:0/2275129607 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f561c003ab0 0x7f561c004420 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f56300076c0 tx=0x7f5630010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:33.262 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:33.216+0000 7f563a243700 1 --2- 172.21.15.42:0/2275129607 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f561c003ab0 0x7f561c004420 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f56300076c0 tx=0x7f5630010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:33.262 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:33.216+0000 7f563a243700 1 --2- 172.21.15.42:0/2275129607 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f561c003ab0 0x7f561c004420 secure :-1 s=READY pgs=226 cs=0 l=1 rx=0x7f56300076c0 tx=0x7f5630010b60).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:04:33.262 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:33.216+0000 7f563a243700 1 --2- 172.21.15.42:0/2275129607 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f561c003ab0 0x7f561c004420 secure :-1 s=THROTTLE_DONE pgs=226 cs=0 l=1 rx=0x7f56300076c0 tx=0x7f5630010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:33.263 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:33.216+0000 7f562affd700 1 -- 172.21.15.42:0/2275129607 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5630008bf0 con 0x7f561c003ab0 2019-12-05T05:04:33.263 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:33.216+0000 7f563a243700 1 --2- 172.21.15.42:0/2275129607 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f561c003ab0 0x7f561c004420 secure :-1 s=THROTTLE_DONE pgs=226 cs=0 l=1 rx=0x7f56300076c0 tx=0x7f5630010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:33.263 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:33.216+0000 7f562affd700 1 -- 172.21.15.42:0/2275129607 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5630013060 con 0x7f561c003ab0 2019-12-05T05:04:33.263 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:33.216+0000 7f563a243700 1 --2- 172.21.15.42:0/2275129607 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f561c003ab0 0x7f561c004420 secure :-1 s=THROTTLE_DONE pgs=226 cs=0 l=1 rx=0x7f56300076c0 tx=0x7f5630010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:33.264 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:33.216+0000 7f562affd700 1 -- 172.21.15.42:0/2275129607 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5630005730 con 0x7f561c003ab0 2019-12-05T05:04:33.264 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:33.216+0000 7f563a243700 1 --2- 172.21.15.42:0/2275129607 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f561c003ab0 0x7f561c004420 secure :-1 s=THROTTLE_DONE pgs=226 cs=0 l=1 rx=0x7f56300076c0 tx=0x7f5630010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:33.265 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:33.216+0000 7f562affd700 1 -- 172.21.15.42:0/2275129607 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f5630005930 con 0x7f561c003ab0 2019-12-05T05:04:33.265 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:33.216+0000 7f563a243700 1 --2- 172.21.15.42:0/2275129607 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f561c003ab0 0x7f561c004420 secure :-1 s=THROTTLE_DONE pgs=226 cs=0 l=1 rx=0x7f56300076c0 tx=0x7f5630010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:33.265 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:33.220+0000 7f562affd700 1 --2- 172.21.15.42:0/2275129607 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f561c043fd0 0x7f561c0464e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:04:33.265 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:33.220+0000 7f562affd700 1 -- 172.21.15.42:0/2275129607 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f563004abd0 con 0x7f561c003ab0 2019-12-05T05:04:33.266 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:33.220+0000 7f5639a42700 1 --2- 172.21.15.42:0/2275129607 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f561c043fd0 0x7f561c0464e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:04:33.266 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:33.220+0000 7f5639a42700 1 --2- 172.21.15.42:0/2275129607 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f561c043fd0 0x7f561c0464e0 crc :-1 s=READY pgs=115 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:04:33.266 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:33.220+0000 7f563bca6700 1 -- 172.21.15.42:0/2275129607 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5610003200 con 0x7f561c003ab0 2019-12-05T05:04:33.266 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:33.228+0000 7f563a243700 1 --2- 172.21.15.42:0/2275129607 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f561c003ab0 0x7f561c004420 secure :-1 s=THROTTLE_DONE pgs=226 cs=0 l=1 rx=0x7f56300076c0 tx=0x7f5630010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:33.267 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:33.228+0000 7f562affd700 1 -- 172.21.15.42:0/2275129607 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f5630031c30 con 0x7f561c003ab0 2019-12-05T05:04:33.400 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:33.392+0000 7f563bca6700 1 -- 172.21.15.42:0/2275129607 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5610003630 con 0x7f561c003ab0 2019-12-05T05:04:33.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:33.392+0000 7f563a243700 1 --2- 172.21.15.42:0/2275129607 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f561c003ab0 0x7f561c004420 secure :-1 s=THROTTLE_DONE pgs=226 cs=0 l=1 rx=0x7f56300076c0 tx=0x7f5630010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:33.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:33.392+0000 7f562affd700 1 -- 172.21.15.42:0/2275129607 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+2102 (secure 0 0 0) 0x7f5630033de0 con 0x7f561c003ab0 2019-12-05T05:04:33.409 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 6 2019-12-05T05:04:33.409 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:04:33.409 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4354,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/2019200199","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2019200199},{"type":"v1","addr":"172.21.15.42:6835","nonce":2019200199}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":4}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:03:43.669739+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4341},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4341":{"gid":4341,"name":"b","rank":0,"incarnation":5,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6833/3296198322","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3296198322},{"type":"v1","addr":"172.21.15.40:6833","nonce":3296198322}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:04:33.412 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:33.396+0000 7f563bca6700 1 -- 172.21.15.42:0/2275129607 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f561c043fd0 msgr2=0x7f561c0464e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:04:33.412 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:33.396+0000 7f563bca6700 1 --2- 172.21.15.42:0/2275129607 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f561c043fd0 0x7f561c0464e0 crc :-1 s=READY pgs=115 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:04:33.422 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:33.396+0000 7f563bca6700 1 -- 172.21.15.42:0/2275129607 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f561c003ab0 msgr2=0x7f561c004420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:04:33.423 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:33.396+0000 7f563aa44700 1 -- 172.21.15.42:0/2275129607 reap_dead start 2019-12-05T05:04:33.423 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:33.396+0000 7f563bca6700 1 --2- 172.21.15.42:0/2275129607 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f561c003ab0 0x7f561c004420 secure :-1 s=READY pgs=226 cs=0 l=1 rx=0x7f56300076c0 tx=0x7f5630010b60).stop 2019-12-05T05:04:33.423 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:33.396+0000 7f563bca6700 1 -- 172.21.15.42:0/2275129607 shutdown_connections 2019-12-05T05:04:33.423 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:33.396+0000 7f563bca6700 1 --2- 172.21.15.42:0/2275129607 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f561c003ab0 0x7f561c004420 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:04:33.424 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:33.396+0000 7f563bca6700 1 -- 172.21.15.42:0/2275129607 shutdown_connections 2019-12-05T05:04:33.424 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:33.396+0000 7f563bca6700 1 -- 172.21.15.42:0/2275129607 wait complete. 2019-12-05T05:04:33.430 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:33.396+0000 7f563bca6700 1 -- 172.21.15.42:0/2275129607 >> 172.21.15.42:0/2275129607 conn(0x7f563409fa00 msgr2=0x7f56340596f0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:04:33.431 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T05:04:33.578 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:04:33.582 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:04:35.432 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:04:35.531 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:35.520+0000 7ff5a06b8700 1 Processor -- start 2019-12-05T05:04:35.531 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:35.520+0000 7ff5a06b8700 1 -- start start 2019-12-05T05:04:35.531 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:35.520+0000 7ff5a06b8700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff5980a7740 0x7ff59808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:04:35.531 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:35.520+0000 7ff5a06b8700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff59809fae0 0x7ff59807eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:04:35.531 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:35.520+0000 7ff5a06b8700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5980a0710 con 0x7ff59807e720 2019-12-05T05:04:35.532 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:35.520+0000 7ff5a06b8700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5980a0b00 con 0x7ff5980a7740 2019-12-05T05:04:35.532 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:35.520+0000 7ff5a06b8700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff59808e5b0 con 0x7ff59809fae0 2019-12-05T05:04:35.532 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:35.520+0000 7ff59f456700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff59809fae0 0x7ff59807eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:04:35.532 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:35.520+0000 7ff59ec55700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff5980a7740 0x7ff59808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:04:35.532 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:35.520+0000 7ff59f456700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff59809fae0 0x7ff59807eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:54600/0 (socket says 172.21.15.42:54600) 2019-12-05T05:04:35.532 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:35.520+0000 7ff59f456700 1 -- 172.21.15.42:0/1781710341 learned_addr learned my addr 172.21.15.42:0/1781710341 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:04:35.532 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:35.520+0000 7ff59e454700 1 --1- 172.21.15.42:0/1781710341 >> v1:172.21.15.40:6789/0 conn(0x7ff59807e720 0x7ff59809f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:35378/0 (socket says 172.21.15.42:35378) 2019-12-05T05:04:35.533 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:35.524+0000 7ff59f456700 1 -- 172.21.15.42:0/1781710341 >> v2:172.21.15.40:3300/0 conn(0x7ff5980a7740 msgr2=0x7ff59808df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:04:35.533 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:35.524+0000 7ff59f456700 1 --2- 172.21.15.42:0/1781710341 >> v2:172.21.15.40:3300/0 conn(0x7ff5980a7740 0x7ff59808df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:04:35.533 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:35.524+0000 7ff59f456700 1 -- 172.21.15.42:0/1781710341 >> v1:172.21.15.40:6789/0 conn(0x7ff59807e720 legacy=0x7ff59809f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:04:35.533 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:35.524+0000 7ff59f456700 1 -- 172.21.15.42:0/1781710341 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff59807f0a0 con 0x7ff59809fae0 2019-12-05T05:04:35.533 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:35.524+0000 7ff59ec55700 1 --2- 172.21.15.42:0/1781710341 >> v2:172.21.15.40:3300/0 conn(0x7ff5980a7740 0x7ff59808df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:04:35.533 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:35.524+0000 7ff59f456700 1 --2- 172.21.15.42:0/1781710341 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff59809fae0 0x7ff59807eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7ff590008b10 tx=0x7ff5900100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:35.533 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:35.524+0000 7ff59f456700 1 --2- 172.21.15.42:0/1781710341 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff59809fae0 0x7ff59807eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7ff590008b10 tx=0x7ff5900100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:35.534 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:35.524+0000 7ff59f456700 1 --2- 172.21.15.42:0/1781710341 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff59809fae0 0x7ff59807eb50 secure :-1 s=READY pgs=51 cs=0 l=1 rx=0x7ff590008b10 tx=0x7ff5900100b0).ready entity=mon.2 client_cookie=13e342ab8acf5e20 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:04:35.534 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:35.524+0000 7ff59f456700 1 --2- 172.21.15.42:0/1781710341 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff59809fae0 0x7ff59807eb50 secure :-1 s=THROTTLE_DONE pgs=51 cs=0 l=1 rx=0x7ff590008b10 tx=0x7ff5900100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:35.534 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:35.524+0000 7ff59dc53700 1 -- 172.21.15.42:0/1781710341 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff590010da0 con 0x7ff59809fae0 2019-12-05T05:04:35.534 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:35.524+0000 7ff59f456700 1 --2- 172.21.15.42:0/1781710341 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff59809fae0 0x7ff59807eb50 secure :-1 s=THROTTLE_DONE pgs=51 cs=0 l=1 rx=0x7ff590008b10 tx=0x7ff5900100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:35.534 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:35.524+0000 7ff59dc53700 1 -- 172.21.15.42:0/1781710341 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5900109c0 con 0x7ff59809fae0 2019-12-05T05:04:35.534 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:35.524+0000 7ff59f456700 1 --2- 172.21.15.42:0/1781710341 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff59809fae0 0x7ff59807eb50 secure :-1 s=THROTTLE_DONE pgs=51 cs=0 l=1 rx=0x7ff590008b10 tx=0x7ff5900100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:35.534 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:35.524+0000 7ff5a06b8700 1 -- 172.21.15.42:0/1781710341 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff59809fae0 msgr2=0x7ff59807eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:04:35.535 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:35.524+0000 7ff59dc53700 1 -- 172.21.15.42:0/1781710341 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff5900114a0 con 0x7ff59809fae0 2019-12-05T05:04:35.535 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:35.524+0000 7ff5a06b8700 1 --2- 172.21.15.42:0/1781710341 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff59809fae0 0x7ff59807eb50 secure :-1 s=READY pgs=51 cs=0 l=1 rx=0x7ff590008b10 tx=0x7ff5900100b0).stop 2019-12-05T05:04:35.535 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:35.524+0000 7ff5a06b8700 1 -- 172.21.15.42:0/1781710341 shutdown_connections 2019-12-05T05:04:35.535 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:35.524+0000 7ff5a06b8700 1 --2- 172.21.15.42:0/1781710341 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff59809fae0 0x7ff59807eb50 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:04:35.535 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:35.524+0000 7ff5a06b8700 1 --2- 172.21.15.42:0/1781710341 >> v2:172.21.15.40:3300/0 conn(0x7ff5980a7740 0x7ff59808df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:04:35.535 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:35.524+0000 7ff5a06b8700 1 -- 172.21.15.42:0/1781710341 shutdown_connections 2019-12-05T05:04:35.535 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:35.524+0000 7ff5a06b8700 1 -- 172.21.15.42:0/1781710341 wait complete. 2019-12-05T05:04:35.536 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:35.524+0000 7ff5a06b8700 1 -- 172.21.15.42:0/1781710341 >> 172.21.15.42:0/1781710341 conn(0x7ff5980aabe0 msgr2=0x7ff5980a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:04:35.539 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:35.528+0000 7ff5a06b8700 1 Processor -- start 2019-12-05T05:04:35.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:35.528+0000 7ff5a06b8700 1 -- start start 2019-12-05T05:04:35.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:35.528+0000 7ff5a06b8700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff598117d60 0x7ff5981176b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:04:35.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:35.528+0000 7ff5a06b8700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff598116600 0x7ff59811aff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:04:35.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:35.528+0000 7ff5a06b8700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff59811ad20 con 0x7ff59811b540 2019-12-05T05:04:35.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:35.528+0000 7ff5a06b8700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff598118270 con 0x7ff598116600 2019-12-05T05:04:35.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:35.528+0000 7ff5a06b8700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff598118510 con 0x7ff598117d60 2019-12-05T05:04:35.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:35.528+0000 7ff59e454700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff598116600 0x7ff59811aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:04:35.541 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:35.528+0000 7ff59e454700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff598116600 0x7ff59811aff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:54136/0 (socket says 172.21.15.42:54136) 2019-12-05T05:04:35.541 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:35.528+0000 7ff59e454700 1 -- 172.21.15.42:0/381671218 learned_addr learned my addr 172.21.15.42:0/381671218 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:04:35.541 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:35.528+0000 7ff59ec55700 1 --2- 172.21.15.42:0/381671218 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff598117d60 0x7ff5981176b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:04:35.541 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:35.528+0000 7ff58f7fe700 1 -- 172.21.15.42:0/381671218 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1551281607 0 0) 0x7ff59811ad20 con 0x7ff59811b540 2019-12-05T05:04:35.541 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:35.528+0000 7ff58f7fe700 1 -- 172.21.15.42:0/381671218 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5800030b0 con 0x7ff59811b540 2019-12-05T05:04:35.541 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:35.528+0000 7ff59e454700 1 -- 172.21.15.42:0/381671218 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff598117d60 msgr2=0x7ff5981176b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:04:35.541 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:35.528+0000 7ff59e454700 1 --2- 172.21.15.42:0/381671218 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff598117d60 0x7ff5981176b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:04:35.542 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:35.528+0000 7ff59e454700 1 -- 172.21.15.42:0/381671218 >> v1:172.21.15.40:6789/0 conn(0x7ff59811b540 legacy=0x7ff59808eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:04:35.542 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:35.528+0000 7ff59e454700 1 -- 172.21.15.42:0/381671218 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff59811b970 con 0x7ff598116600 2019-12-05T05:04:35.542 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:35.528+0000 7ff59e454700 1 --2- 172.21.15.42:0/381671218 >> v2:172.21.15.42:3300/0 conn(0x7ff598116600 0x7ff59811aff0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7ff598118430 tx=0x7ff5880109f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:35.542 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:35.528+0000 7ff59e454700 1 --2- 172.21.15.42:0/381671218 >> v2:172.21.15.42:3300/0 conn(0x7ff598116600 0x7ff59811aff0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7ff598118430 tx=0x7ff5880109f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:35.542 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:35.528+0000 7ff59e454700 1 --2- 172.21.15.42:0/381671218 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff598116600 0x7ff59811aff0 secure :-1 s=READY pgs=227 cs=0 l=1 rx=0x7ff598118430 tx=0x7ff5880109f0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:04:35.542 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:35.528+0000 7ff59e454700 1 --2- 172.21.15.42:0/381671218 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff598116600 0x7ff59811aff0 secure :-1 s=THROTTLE_DONE pgs=227 cs=0 l=1 rx=0x7ff598118430 tx=0x7ff5880109f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:35.542 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:35.528+0000 7ff59e454700 1 --2- 172.21.15.42:0/381671218 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff598116600 0x7ff59811aff0 secure :-1 s=THROTTLE_DONE pgs=227 cs=0 l=1 rx=0x7ff598118430 tx=0x7ff5880109f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:35.543 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:35.528+0000 7ff58f7fe700 1 -- 172.21.15.42:0/381671218 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff588011660 con 0x7ff598116600 2019-12-05T05:04:35.543 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:35.528+0000 7ff59e454700 1 --2- 172.21.15.42:0/381671218 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff598116600 0x7ff59811aff0 secure :-1 s=THROTTLE_DONE pgs=227 cs=0 l=1 rx=0x7ff598118430 tx=0x7ff5880109f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:35.543 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:35.528+0000 7ff58f7fe700 1 -- 172.21.15.42:0/381671218 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff588011280 con 0x7ff598116600 2019-12-05T05:04:35.543 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:35.528+0000 7ff5a06b8700 1 -- 172.21.15.42:0/381671218 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff59808f370 con 0x7ff598116600 2019-12-05T05:04:35.543 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:35.532+0000 7ff58f7fe700 1 -- 172.21.15.42:0/381671218 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff588011c40 con 0x7ff598116600 2019-12-05T05:04:35.543 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:35.532+0000 7ff5a06b8700 1 -- 172.21.15.42:0/381671218 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff59808f8f0 con 0x7ff598116600 2019-12-05T05:04:35.543 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:35.532+0000 7ff59e454700 1 --2- 172.21.15.42:0/381671218 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff598116600 0x7ff59811aff0 secure :-1 s=THROTTLE_DONE pgs=227 cs=0 l=1 rx=0x7ff598118430 tx=0x7ff5880109f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:35.544 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:35.532+0000 7ff58f7fe700 1 -- 172.21.15.42:0/381671218 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7ff588006460 con 0x7ff598116600 2019-12-05T05:04:35.544 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:35.532+0000 7ff59e454700 1 --2- 172.21.15.42:0/381671218 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff598116600 0x7ff59811aff0 secure :-1 s=THROTTLE_DONE pgs=227 cs=0 l=1 rx=0x7ff598118430 tx=0x7ff5880109f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:35.544 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:35.532+0000 7ff58f7fe700 1 --2- 172.21.15.42:0/381671218 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff58003b2d0 0x7ff58003d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:04:35.544 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:35.532+0000 7ff58f7fe700 1 -- 172.21.15.42:0/381671218 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6031+0+0 (secure 0 0 0) 0x7ff58804ad60 con 0x7ff598116600 2019-12-05T05:04:35.544 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:35.532+0000 7ff59ec55700 1 --2- 172.21.15.42:0/381671218 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff58003b2d0 0x7ff58003d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:04:35.544 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:35.532+0000 7ff59ec55700 1 --2- 172.21.15.42:0/381671218 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff58003b2d0 0x7ff58003d7e0 crc :-1 s=READY pgs=116 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:04:35.544 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:35.536+0000 7ff5a06b8700 1 -- 172.21.15.42:0/381671218 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff59805d110 con 0x7ff598116600 2019-12-05T05:04:35.547 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:35.544+0000 7ff59e454700 1 --2- 172.21.15.42:0/381671218 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff598116600 0x7ff59811aff0 secure :-1 s=THROTTLE_DONE pgs=227 cs=0 l=1 rx=0x7ff598118430 tx=0x7ff5880109f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:35.547 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:35.544+0000 7ff58f7fe700 1 -- 172.21.15.42:0/381671218 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7ff588031de0 con 0x7ff598116600 2019-12-05T05:04:35.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:35.704+0000 7ff5a06b8700 1 -- 172.21.15.42:0/381671218 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff59805b8c0 con 0x7ff598116600 2019-12-05T05:04:35.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:35.704+0000 7ff59e454700 1 --2- 172.21.15.42:0/381671218 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff598116600 0x7ff59811aff0 secure :-1 s=THROTTLE_DONE pgs=227 cs=0 l=1 rx=0x7ff598118430 tx=0x7ff5880109f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:35.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:35.704+0000 7ff58f7fe700 1 -- 172.21.15.42:0/381671218 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+2102 (secure 0 0 0) 0x7ff588032040 con 0x7ff598116600 2019-12-05T05:04:35.714 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 6 2019-12-05T05:04:35.715 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:35.708+0000 7ff5a06b8700 1 -- 172.21.15.42:0/381671218 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff58003b2d0 msgr2=0x7ff58003d7e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:04:35.715 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:35.708+0000 7ff5a06b8700 1 --2- 172.21.15.42:0/381671218 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff58003b2d0 0x7ff58003d7e0 crc :-1 s=READY pgs=116 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:04:35.715 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:35.708+0000 7ff5a06b8700 1 -- 172.21.15.42:0/381671218 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff598116600 msgr2=0x7ff59811aff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:04:35.715 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:35.708+0000 7ff5a06b8700 1 --2- 172.21.15.42:0/381671218 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff598116600 0x7ff59811aff0 secure :-1 s=READY pgs=227 cs=0 l=1 rx=0x7ff598118430 tx=0x7ff5880109f0).stop 2019-12-05T05:04:35.715 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:35.708+0000 7ff5a06b8700 1 -- 172.21.15.42:0/381671218 shutdown_connections 2019-12-05T05:04:35.715 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:35.708+0000 7ff5a06b8700 1 --2- 172.21.15.42:0/381671218 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff58003b2d0 0x7ff58003d7e0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:04:35.715 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:35.708+0000 7ff5a06b8700 1 --2- 172.21.15.42:0/381671218 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff598117d60 0x7ff5981176b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:04:35.716 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:35.708+0000 7ff5a06b8700 1 --2- 172.21.15.42:0/381671218 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff598116600 0x7ff59811aff0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:04:35.716 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:35.708+0000 7ff5a06b8700 1 -- 172.21.15.42:0/381671218 shutdown_connections 2019-12-05T05:04:35.716 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:35.708+0000 7ff5a06b8700 1 -- 172.21.15.42:0/381671218 wait complete. 2019-12-05T05:04:35.716 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:35.708+0000 7ff5a06b8700 1 -- 172.21.15.42:0/381671218 >> 172.21.15.42:0/381671218 conn(0x7ff5980aabe0 msgr2=0x7ff598059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:04:35.716 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:04:35.717 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4354,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/2019200199","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2019200199},{"type":"v1","addr":"172.21.15.42:6835","nonce":2019200199}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":4}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:03:43.669739+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4341},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4341":{"gid":4341,"name":"b","rank":0,"incarnation":5,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6833/3296198322","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3296198322},{"type":"v1","addr":"172.21.15.40:6833","nonce":3296198322}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:04:35.724 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T05:04:37.729 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:04:37.831 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:37.820+0000 7f8546960700 1 Processor -- start 2019-12-05T05:04:37.831 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:37.820+0000 7f8546960700 1 -- start start 2019-12-05T05:04:37.831 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:37.820+0000 7f8546960700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f85400a7740 0x7f854008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:04:37.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:37.820+0000 7f8546960700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f854007e720 0x7f854007eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:04:37.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:37.820+0000 7f8546960700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85400a0710 con 0x7f854009f2e0 2019-12-05T05:04:37.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:37.820+0000 7f8546960700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85400a0b00 con 0x7f854007e720 2019-12-05T05:04:37.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:37.820+0000 7f8546960700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f854008e5b0 con 0x7f85400a7740 2019-12-05T05:04:37.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:37.820+0000 7f8544efd700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f85400a7740 0x7f854008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:04:37.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:37.820+0000 7f8537fff700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f854007e720 0x7f854007eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:04:37.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:37.820+0000 7f8544efd700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f85400a7740 0x7f854008df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:54610/0 (socket says 172.21.15.42:54610) 2019-12-05T05:04:37.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:37.820+0000 7f8544efd700 1 -- 172.21.15.42:0/1573719272 learned_addr learned my addr 172.21.15.42:0/1573719272 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:04:37.837 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:37.824+0000 7f8537fff700 1 -- 172.21.15.42:0/1573719272 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f85400a7740 msgr2=0x7f854008df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:04:37.837 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:37.824+0000 7f8537fff700 1 --2- 172.21.15.42:0/1573719272 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f85400a7740 0x7f854008df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:04:37.838 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:37.824+0000 7f8537fff700 1 -- 172.21.15.42:0/1573719272 >> v1:172.21.15.40:6789/0 conn(0x7f854009f2e0 legacy=0x7f854009f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:04:37.838 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:37.824+0000 7f8537fff700 1 -- 172.21.15.42:0/1573719272 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f854007f0a0 con 0x7f854007e720 2019-12-05T05:04:37.840 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:37.824+0000 7f8544efd700 1 --2- 172.21.15.42:0/1573719272 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f85400a7740 0x7f854008df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:04:37.840 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:37.824+0000 7f8537fff700 1 --2- 172.21.15.42:0/1573719272 >> v2:172.21.15.40:3300/0 conn(0x7f854007e720 0x7f854007eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f8530008790 tx=0x7f853000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:37.840 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:37.824+0000 7f8537fff700 1 --2- 172.21.15.42:0/1573719272 >> v2:172.21.15.40:3300/0 conn(0x7f854007e720 0x7f854007eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f8530008790 tx=0x7f853000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:37.840 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:37.824+0000 7f8537fff700 1 --2- 172.21.15.42:0/1573719272 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f854007e720 0x7f854007eb50 secure :-1 s=READY pgs=193 cs=0 l=1 rx=0x7f8530008790 tx=0x7f853000fc70).ready entity=mon.0 client_cookie=128dd8f673072f70 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:04:37.840 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:37.824+0000 7f8537fff700 1 --2- 172.21.15.42:0/1573719272 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f854007e720 0x7f854007eb50 secure :-1 s=THROTTLE_DONE pgs=193 cs=0 l=1 rx=0x7f8530008790 tx=0x7f853000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:37.841 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:37.824+0000 7f85377fe700 1 -- 172.21.15.42:0/1573719272 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f85300112f0 con 0x7f854007e720 2019-12-05T05:04:37.841 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:37.824+0000 7f8537fff700 1 --2- 172.21.15.42:0/1573719272 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f854007e720 0x7f854007eb50 secure :-1 s=THROTTLE_DONE pgs=193 cs=0 l=1 rx=0x7f8530008790 tx=0x7f853000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:37.841 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:37.824+0000 7f85377fe700 1 -- 172.21.15.42:0/1573719272 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8530008ad0 con 0x7f854007e720 2019-12-05T05:04:37.841 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:37.824+0000 7f8537fff700 1 --2- 172.21.15.42:0/1573719272 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f854007e720 0x7f854007eb50 secure :-1 s=THROTTLE_DONE pgs=193 cs=0 l=1 rx=0x7f8530008790 tx=0x7f853000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:37.841 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:37.824+0000 7f85377fe700 1 -- 172.21.15.42:0/1573719272 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f85300118d0 con 0x7f854007e720 2019-12-05T05:04:37.841 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:37.824+0000 7f8546960700 1 -- 172.21.15.42:0/1573719272 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f854007e720 msgr2=0x7f854007eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:04:37.842 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:37.824+0000 7f8546960700 1 --2- 172.21.15.42:0/1573719272 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f854007e720 0x7f854007eb50 secure :-1 s=READY pgs=193 cs=0 l=1 rx=0x7f8530008790 tx=0x7f853000fc70).stop 2019-12-05T05:04:37.842 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:37.824+0000 7f8546960700 1 -- 172.21.15.42:0/1573719272 shutdown_connections 2019-12-05T05:04:37.842 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:37.824+0000 7f8546960700 1 --2- 172.21.15.42:0/1573719272 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f85400a7740 0x7f854008df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:04:37.842 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:37.824+0000 7f8546960700 1 --2- 172.21.15.42:0/1573719272 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f854007e720 0x7f854007eb50 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:04:37.842 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:37.824+0000 7f8546960700 1 -- 172.21.15.42:0/1573719272 shutdown_connections 2019-12-05T05:04:37.842 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:37.824+0000 7f8546960700 1 -- 172.21.15.42:0/1573719272 wait complete. 2019-12-05T05:04:37.843 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:37.824+0000 7f8546960700 1 -- 172.21.15.42:0/1573719272 >> 172.21.15.42:0/1573719272 conn(0x7f85400aabe0 msgr2=0x7f85400a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:04:37.843 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:37.824+0000 7f8546960700 1 Processor -- start 2019-12-05T05:04:37.843 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:37.824+0000 7f8546960700 1 -- start start 2019-12-05T05:04:37.843 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:37.828+0000 7f8546960700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f854012a5e0 0x7f854012ca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:04:37.843 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:37.828+0000 7f8546960700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85400a5f40 con 0x7f85400a5560 2019-12-05T05:04:37.843 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:37.828+0000 7f8546960700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85400a5990 con 0x7f854011b100 2019-12-05T05:04:37.843 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:37.828+0000 7f8546960700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85400a5ca0 con 0x7f854012a5e0 2019-12-05T05:04:37.844 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:37.828+0000 7f8544efd700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f854011b100 0x7f8540123050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:54188/0 (socket says 172.21.15.42:54188) 2019-12-05T05:04:37.844 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:37.828+0000 7f8544efd700 1 -- 172.21.15.42:0/3835548651 learned_addr learned my addr 172.21.15.42:0/3835548651 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:04:37.854 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:37.828+0000 7f85456fe700 1 --2- 172.21.15.42:0/3835548651 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f854012a5e0 0x7f854012ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:04:37.854 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:37.828+0000 7f85357fa700 1 -- 172.21.15.42:0/3835548651 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1569717115 0 0) 0x7f85400a5990 con 0x7f854011b100 2019-12-05T05:04:37.854 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:37.828+0000 7f85357fa700 1 -- 172.21.15.42:0/3835548651 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85200030b0 con 0x7f854011b100 2019-12-05T05:04:37.854 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:37.828+0000 7f85357fa700 1 -- 172.21.15.42:0/3835548651 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 636997394 0 0) 0x7f85400a5f40 con 0x7f85400a5560 2019-12-05T05:04:37.854 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:37.828+0000 7f85357fa700 1 -- 172.21.15.42:0/3835548651 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85400a5990 con 0x7f85400a5560 2019-12-05T05:04:37.854 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:37.828+0000 7f85357fa700 1 -- 172.21.15.42:0/3835548651 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 1089743484 0 0) 0x7f85200030b0 con 0x7f854011b100 2019-12-05T05:04:37.855 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:37.828+0000 7f85357fa700 1 -- 172.21.15.42:0/3835548651 --> v1:172.21.15.42:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f85400a5f40 con 0x7f854011b100 2019-12-05T05:04:37.855 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:37.828+0000 7f85357fa700 1 -- 172.21.15.42:0/3835548651 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7f853c001ec0 con 0x7f854011b100 2019-12-05T05:04:37.855 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:37.828+0000 7f85357fa700 1 -- 172.21.15.42:0/3835548651 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f854012a5e0 msgr2=0x7f854012ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:04:37.855 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:37.828+0000 7f85357fa700 1 --2- 172.21.15.42:0/3835548651 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f854012a5e0 0x7f854012ca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:04:37.855 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:37.828+0000 7f85357fa700 1 -- 172.21.15.42:0/3835548651 >> v1:172.21.15.42:6789/0 conn(0x7f854011b100 legacy=0x7f8540123050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:04:37.856 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:37.828+0000 7f85357fa700 1 -- 172.21.15.42:0/3835548651 >> v1:172.21.15.40:6789/0 conn(0x7f85400a5560 legacy=0x7f8540119a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:04:37.856 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:37.828+0000 7f85357fa700 1 --2- 172.21.15.42:0/3835548651 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8520003ab0 0x7f8520004420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:04:37.856 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:37.828+0000 7f85357fa700 1 --2- 172.21.15.42:0/3835548651 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8520003f90 0x7f8520004a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:04:37.856 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:37.828+0000 7f85357fa700 1 --2- 172.21.15.42:0/3835548651 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8520009660 0x7f852000baa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:04:37.856 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:37.828+0000 7f85357fa700 1 -- 172.21.15.42:0/3835548651 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85400a5ca0 con 0x7f8520003ab0 2019-12-05T05:04:37.856 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:37.828+0000 7f85357fa700 1 -- 172.21.15.42:0/3835548651 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85200050f0 con 0x7f8520003f90 2019-12-05T05:04:37.856 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:37.828+0000 7f85357fa700 1 -- 172.21.15.42:0/3835548651 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f852000bff0 con 0x7f8520009660 2019-12-05T05:04:37.857 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:37.828+0000 7f8537fff700 1 --2- 172.21.15.42:0/3835548651 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8520003f90 0x7f8520004a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:04:37.857 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:37.828+0000 7f8544efd700 1 --2- 172.21.15.42:0/3835548651 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8520003ab0 0x7f8520004420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:04:37.857 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:37.828+0000 7f85456fe700 1 --2- 172.21.15.42:0/3835548651 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8520009660 0x7f852000baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:04:37.857 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:37.828+0000 7f8537fff700 1 -- 172.21.15.42:0/3835548651 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8520009660 msgr2=0x7f852000baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:04:37.858 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:37.828+0000 7f8537fff700 1 --2- 172.21.15.42:0/3835548651 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8520009660 0x7f852000baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:04:37.859 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:37.828+0000 7f8537fff700 1 -- 172.21.15.42:0/3835548651 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8520003ab0 msgr2=0x7f8520004420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:04:37.859 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:37.828+0000 7f8537fff700 1 --2- 172.21.15.42:0/3835548651 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8520003ab0 0x7f8520004420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:04:37.859 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:37.828+0000 7f8537fff700 1 -- 172.21.15.42:0/3835548651 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8540138a50 con 0x7f8520003f90 2019-12-05T05:04:37.859 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:37.828+0000 7f8546960700 1 -- 172.21.15.42:0/3835548651 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f854011a220 con 0x7f8520003f90 2019-12-05T05:04:37.859 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:37.828+0000 7f8537fff700 1 --2- 172.21.15.42:0/3835548651 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8520003f90 0x7f8520004a10 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f8530010080 tx=0x7f8530008e00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:37.860 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:37.828+0000 7f8537fff700 1 --2- 172.21.15.42:0/3835548651 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8520003f90 0x7f8520004a10 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f8530010080 tx=0x7f8530008e00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:37.860 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:37.828+0000 7f8546960700 1 -- 172.21.15.42:0/3835548651 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f854012a5e0 con 0x7f8520003f90 2019-12-05T05:04:37.860 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:37.828+0000 7f8537fff700 1 --2- 172.21.15.42:0/3835548651 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8520003f90 0x7f8520004a10 secure :-1 s=READY pgs=229 cs=0 l=1 rx=0x7f8530010080 tx=0x7f8530008e00).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:04:37.860 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:37.828+0000 7f85456fe700 1 --2- 172.21.15.42:0/3835548651 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8520009660 0x7f852000baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T05:04:37.860 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:37.828+0000 7f8537fff700 1 --2- 172.21.15.42:0/3835548651 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8520003f90 0x7f8520004a10 secure :-1 s=THROTTLE_DONE pgs=229 cs=0 l=1 rx=0x7f8530010080 tx=0x7f8530008e00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:37.860 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:37.828+0000 7f85357fa700 1 -- 172.21.15.42:0/3835548651 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8530005540 con 0x7f8520003f90 2019-12-05T05:04:37.860 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:37.828+0000 7f8537fff700 1 --2- 172.21.15.42:0/3835548651 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8520003f90 0x7f8520004a10 secure :-1 s=THROTTLE_DONE pgs=229 cs=0 l=1 rx=0x7f8530010080 tx=0x7f8530008e00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:37.861 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:37.828+0000 7f8537fff700 1 --2- 172.21.15.42:0/3835548651 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8520003f90 0x7f8520004a10 secure :-1 s=THROTTLE_DONE pgs=229 cs=0 l=1 rx=0x7f8530010080 tx=0x7f8530008e00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:37.861 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:37.828+0000 7f85357fa700 1 -- 172.21.15.42:0/3835548651 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85300114f0 con 0x7f8520003f90 2019-12-05T05:04:37.861 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:37.828+0000 7f85357fa700 1 -- 172.21.15.42:0/3835548651 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8530005b20 con 0x7f8520003f90 2019-12-05T05:04:37.861 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:37.828+0000 7f8537fff700 1 --2- 172.21.15.42:0/3835548651 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8520003f90 0x7f8520004a10 secure :-1 s=THROTTLE_DONE pgs=229 cs=0 l=1 rx=0x7f8530010080 tx=0x7f8530008e00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:37.861 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:37.828+0000 7f85357fa700 1 -- 172.21.15.42:0/3835548651 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f8530013460 con 0x7f8520003f90 2019-12-05T05:04:37.861 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:37.828+0000 7f8537fff700 1 --2- 172.21.15.42:0/3835548651 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8520003f90 0x7f8520004a10 secure :-1 s=THROTTLE_DONE pgs=229 cs=0 l=1 rx=0x7f8530010080 tx=0x7f8530008e00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:37.861 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:37.828+0000 7f85357fa700 1 --2- 172.21.15.42:0/3835548651 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8520043e00 0x7f8520046310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:04:37.862 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:37.828+0000 7f85357fa700 1 -- 172.21.15.42:0/3835548651 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f853004b4d0 con 0x7f8520003f90 2019-12-05T05:04:37.862 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:37.828+0000 7f8544efd700 1 --2- 172.21.15.42:0/3835548651 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8520043e00 0x7f8520046310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:04:37.862 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:37.828+0000 7f8544efd700 1 --2- 172.21.15.42:0/3835548651 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8520043e00 0x7f8520046310 crc :-1 s=READY pgs=117 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:04:37.862 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:37.836+0000 7f8546960700 1 -- 172.21.15.42:0/3835548651 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8518003200 con 0x7f8520003f90 2019-12-05T05:04:37.862 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:37.844+0000 7f8537fff700 1 --2- 172.21.15.42:0/3835548651 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8520003f90 0x7f8520004a10 secure :-1 s=THROTTLE_DONE pgs=229 cs=0 l=1 rx=0x7f8530010080 tx=0x7f8530008e00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:37.862 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:37.844+0000 7f85357fa700 1 -- 172.21.15.42:0/3835548651 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f8530032550 con 0x7f8520003f90 2019-12-05T05:04:38.007 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:37.996+0000 7f8546960700 1 -- 172.21.15.42:0/3835548651 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8518003630 con 0x7f8520003f90 2019-12-05T05:04:38.007 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:37.996+0000 7f8537fff700 1 --2- 172.21.15.42:0/3835548651 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8520003f90 0x7f8520004a10 secure :-1 s=THROTTLE_DONE pgs=229 cs=0 l=1 rx=0x7f8530010080 tx=0x7f8530008e00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:38.007 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:37.996+0000 7f85357fa700 1 -- 172.21.15.42:0/3835548651 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+2102 (secure 0 0 0) 0x7f85300327b0 con 0x7f8520003f90 2019-12-05T05:04:38.007 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 6 2019-12-05T05:04:38.008 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:38.000+0000 7f8546960700 1 -- 172.21.15.42:0/3835548651 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8520043e00 msgr2=0x7f8520046310 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:04:38.008 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:38.000+0000 7f8546960700 1 --2- 172.21.15.42:0/3835548651 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8520043e00 0x7f8520046310 crc :-1 s=READY pgs=117 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:04:38.008 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:38.000+0000 7f8546960700 1 -- 172.21.15.42:0/3835548651 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8520003f90 msgr2=0x7f8520004a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:04:38.008 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:38.000+0000 7f8546960700 1 --2- 172.21.15.42:0/3835548651 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8520003f90 0x7f8520004a10 secure :-1 s=READY pgs=229 cs=0 l=1 rx=0x7f8530010080 tx=0x7f8530008e00).stop 2019-12-05T05:04:38.009 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:38.000+0000 7f85456fe700 1 -- 172.21.15.42:0/3835548651 reap_dead start 2019-12-05T05:04:38.009 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:38.000+0000 7f8546960700 1 -- 172.21.15.42:0/3835548651 shutdown_connections 2019-12-05T05:04:38.009 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:38.000+0000 7f8546960700 1 --2- 172.21.15.42:0/3835548651 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8520003f90 0x7f8520004a10 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:04:38.009 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:38.000+0000 7f8546960700 1 -- 172.21.15.42:0/3835548651 shutdown_connections 2019-12-05T05:04:38.010 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:38.000+0000 7f8546960700 1 -- 172.21.15.42:0/3835548651 wait complete. 2019-12-05T05:04:38.010 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:38.000+0000 7f8546960700 1 -- 172.21.15.42:0/3835548651 >> 172.21.15.42:0/3835548651 conn(0x7f85400a9b10 msgr2=0x7f85400a7f90 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:04:38.010 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:04:38.010 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4354,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/2019200199","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2019200199},{"type":"v1","addr":"172.21.15.42:6835","nonce":2019200199}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":4}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:03:43.669739+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4341},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4341":{"gid":4341,"name":"b","rank":0,"incarnation":5,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6833/3296198322","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3296198322},{"type":"v1","addr":"172.21.15.40:6833","nonce":3296198322}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:04:38.015 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T05:04:40.016 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:04:40.100 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:40.092+0000 7f1f510b3700 1 Processor -- start 2019-12-05T05:04:40.100 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:40.092+0000 7f1f510b3700 1 -- start start 2019-12-05T05:04:40.100 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:40.092+0000 7f1f510b3700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f1f4c0a7740 0x7f1f4c08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:04:40.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:40.092+0000 7f1f510b3700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1f4c07e720 0x7f1f4c07eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:04:40.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:40.092+0000 7f1f510b3700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f1f4c09f2e0 0x7f1f4c09f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:04:40.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:40.092+0000 7f1f510b3700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f4c0a0b00 con 0x7f1f4c09f2e0 2019-12-05T05:04:40.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:40.092+0000 7f1f510b3700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f4c08e5b0 con 0x7f1f4c0a7740 2019-12-05T05:04:40.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:40.092+0000 7f1f510b3700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f4c08e850 con 0x7f1f4c07e720 2019-12-05T05:04:40.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:40.092+0000 7f1f4b59e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f1f4c0a7740 0x7f1f4c08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:04:40.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:40.092+0000 7f1f4b59e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f1f4c0a7740 0x7f1f4c08df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:54168/0 (socket says 172.21.15.42:54168) 2019-12-05T05:04:40.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:40.092+0000 7f1f4b59e700 1 -- 172.21.15.42:0/2880371833 learned_addr learned my addr 172.21.15.42:0/2880371833 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:04:40.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:40.092+0000 7f1f4ad9d700 1 --2- 172.21.15.42:0/2880371833 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1f4c07e720 0x7f1f4c07eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:04:40.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:40.092+0000 7f1f508b2700 1 --2- 172.21.15.42:0/2880371833 >> v2:172.21.15.40:3300/0 conn(0x7f1f4c09f2e0 0x7f1f4c09f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:04:40.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:40.092+0000 7f1f4b59e700 1 -- 172.21.15.42:0/2880371833 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1f4c07e720 msgr2=0x7f1f4c07eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:04:40.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:40.092+0000 7f1f4b59e700 1 --2- 172.21.15.42:0/2880371833 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1f4c07e720 0x7f1f4c07eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:04:40.103 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:40.092+0000 7f1f4b59e700 1 -- 172.21.15.42:0/2880371833 >> v2:172.21.15.40:3300/0 conn(0x7f1f4c09f2e0 msgr2=0x7f1f4c09f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:04:40.103 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:40.092+0000 7f1f4b59e700 1 --2- 172.21.15.42:0/2880371833 >> v2:172.21.15.40:3300/0 conn(0x7f1f4c09f2e0 0x7f1f4c09f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:04:40.103 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:40.092+0000 7f1f4b59e700 1 -- 172.21.15.42:0/2880371833 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f4c07f0a0 con 0x7f1f4c0a7740 2019-12-05T05:04:40.103 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:40.092+0000 7f1f4b59e700 1 --2- 172.21.15.42:0/2880371833 >> v2:172.21.15.42:3300/0 conn(0x7f1f4c0a7740 0x7f1f4c08df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f1f4c08e770 tx=0x7f1f3c00fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:40.103 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:40.092+0000 7f1f4b59e700 1 --2- 172.21.15.42:0/2880371833 >> v2:172.21.15.42:3300/0 conn(0x7f1f4c0a7740 0x7f1f4c08df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f1f4c08e770 tx=0x7f1f3c00fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:40.103 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:40.092+0000 7f1f4b59e700 1 --2- 172.21.15.42:0/2880371833 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1f4c0a7740 0x7f1f4c08df80 secure :-1 s=READY pgs=230 cs=0 l=1 rx=0x7f1f4c08e770 tx=0x7f1f3c00fdb0).ready entity=mon.1 client_cookie=bc425264db584e7e server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:04:40.103 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:40.092+0000 7f1f4b59e700 1 --2- 172.21.15.42:0/2880371833 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1f4c0a7740 0x7f1f4c08df80 secure :-1 s=THROTTLE_DONE pgs=230 cs=0 l=1 rx=0x7f1f4c08e770 tx=0x7f1f3c00fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:40.104 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:40.092+0000 7f1f4a59c700 1 -- 172.21.15.42:0/2880371833 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1f3c008c90 con 0x7f1f4c0a7740 2019-12-05T05:04:40.104 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:40.092+0000 7f1f4b59e700 1 --2- 172.21.15.42:0/2880371833 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1f4c0a7740 0x7f1f4c08df80 secure :-1 s=THROTTLE_DONE pgs=230 cs=0 l=1 rx=0x7f1f4c08e770 tx=0x7f1f3c00fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:40.104 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:40.092+0000 7f1f4a59c700 1 -- 172.21.15.42:0/2880371833 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f3c0088b0 con 0x7f1f4c0a7740 2019-12-05T05:04:40.104 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:40.092+0000 7f1f4b59e700 1 --2- 172.21.15.42:0/2880371833 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1f4c0a7740 0x7f1f4c08df80 secure :-1 s=THROTTLE_DONE pgs=230 cs=0 l=1 rx=0x7f1f4c08e770 tx=0x7f1f3c00fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:40.104 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:40.092+0000 7f1f4a59c700 1 -- 172.21.15.42:0/2880371833 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1f3c011b50 con 0x7f1f4c0a7740 2019-12-05T05:04:40.104 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:40.092+0000 7f1f510b3700 1 -- 172.21.15.42:0/2880371833 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1f4c0a7740 msgr2=0x7f1f4c08df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:04:40.104 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:40.092+0000 7f1f510b3700 1 --2- 172.21.15.42:0/2880371833 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1f4c0a7740 0x7f1f4c08df80 secure :-1 s=READY pgs=230 cs=0 l=1 rx=0x7f1f4c08e770 tx=0x7f1f3c00fdb0).stop 2019-12-05T05:04:40.105 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:40.092+0000 7f1f510b3700 1 -- 172.21.15.42:0/2880371833 shutdown_connections 2019-12-05T05:04:40.105 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:40.092+0000 7f1f510b3700 1 --2- 172.21.15.42:0/2880371833 >> v2:172.21.15.40:3300/0 conn(0x7f1f4c09f2e0 0x7f1f4c09f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:04:40.105 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:40.092+0000 7f1f510b3700 1 --2- 172.21.15.42:0/2880371833 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1f4c0a7740 0x7f1f4c08df80 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:04:40.105 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:40.092+0000 7f1f510b3700 1 --2- 172.21.15.42:0/2880371833 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1f4c07e720 0x7f1f4c07eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:04:40.105 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:40.092+0000 7f1f510b3700 1 -- 172.21.15.42:0/2880371833 shutdown_connections 2019-12-05T05:04:40.105 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:40.092+0000 7f1f510b3700 1 -- 172.21.15.42:0/2880371833 wait complete. 2019-12-05T05:04:40.106 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:40.092+0000 7f1f510b3700 1 -- 172.21.15.42:0/2880371833 >> 172.21.15.42:0/2880371833 conn(0x7f1f4c0aabe0 msgr2=0x7f1f4c0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:04:40.106 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:40.096+0000 7f1f510b3700 1 Processor -- start 2019-12-05T05:04:40.106 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:40.096+0000 7f1f510b3700 1 -- start start 2019-12-05T05:04:40.110 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:40.096+0000 7f1f510b3700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1f4c145080 0x7f1f4c137950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:04:40.111 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:40.096+0000 7f1f510b3700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f1f4c137ea0 0x7f1f4c1382d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:04:40.112 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:40.096+0000 7f1f510b3700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f4c1454b0 con 0x7f1f4c0a6c60 2019-12-05T05:04:40.112 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:40.096+0000 7f1f510b3700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f4c143f00 con 0x7f1f4c137ea0 2019-12-05T05:04:40.112 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:40.096+0000 7f1f510b3700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f4c138820 con 0x7f1f4c145080 2019-12-05T05:04:40.112 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:40.096+0000 7f1f4b59e700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f1f4c0a6c60 0x7f1f4c142950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:54212/0 (socket says 172.21.15.42:54212) 2019-12-05T05:04:40.112 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:40.096+0000 7f1f4b59e700 1 -- 172.21.15.42:0/234480762 learned_addr learned my addr 172.21.15.42:0/234480762 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:04:40.113 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:40.096+0000 7f1f508b2700 1 --2- 172.21.15.42:0/234480762 >> v2:172.21.15.42:3300/0 conn(0x7f1f4c137ea0 0x7f1f4c1382d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:04:40.113 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:40.096+0000 7f1f4ad9d700 1 --2- 172.21.15.42:0/234480762 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1f4c145080 0x7f1f4c137950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:04:40.113 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:40.096+0000 7f1f43fff700 1 -- 172.21.15.42:0/234480762 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4120464698 0 0) 0x7f1f3c008470 con 0x7f1f4c0a6c60 2019-12-05T05:04:40.113 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:40.096+0000 7f1f43fff700 1 -- 172.21.15.42:0/234480762 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f300030b0 con 0x7f1f4c0a6c60 2019-12-05T05:04:40.113 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:40.096+0000 7f1f508b2700 1 -- 172.21.15.42:0/234480762 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1f4c145080 msgr2=0x7f1f4c137950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:04:40.113 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:40.096+0000 7f1f508b2700 1 --2- 172.21.15.42:0/234480762 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1f4c145080 0x7f1f4c137950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:04:40.113 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:40.096+0000 7f1f508b2700 1 -- 172.21.15.42:0/234480762 >> v1:172.21.15.42:6789/0 conn(0x7f1f4c0a6c60 legacy=0x7f1f4c142950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:04:40.114 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:40.096+0000 7f1f508b2700 1 -- 172.21.15.42:0/234480762 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f4c143830 con 0x7f1f4c137ea0 2019-12-05T05:04:40.114 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:40.096+0000 7f1f508b2700 1 --2- 172.21.15.42:0/234480762 >> v2:172.21.15.42:3300/0 conn(0x7f1f4c137ea0 0x7f1f4c1382d0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f1f440032b0 tx=0x7f1f44010ca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:40.114 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:40.096+0000 7f1f508b2700 1 --2- 172.21.15.42:0/234480762 >> v2:172.21.15.42:3300/0 conn(0x7f1f4c137ea0 0x7f1f4c1382d0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f1f440032b0 tx=0x7f1f44010ca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:40.114 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:40.096+0000 7f1f508b2700 1 --2- 172.21.15.42:0/234480762 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1f4c137ea0 0x7f1f4c1382d0 secure :-1 s=READY pgs=232 cs=0 l=1 rx=0x7f1f440032b0 tx=0x7f1f44010ca0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:04:40.114 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:40.096+0000 7f1f508b2700 1 --2- 172.21.15.42:0/234480762 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1f4c137ea0 0x7f1f4c1382d0 secure :-1 s=THROTTLE_DONE pgs=232 cs=0 l=1 rx=0x7f1f440032b0 tx=0x7f1f44010ca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:40.114 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:40.096+0000 7f1f43fff700 1 -- 172.21.15.42:0/234480762 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1f44005ca0 con 0x7f1f4c137ea0 2019-12-05T05:04:40.114 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:40.096+0000 7f1f508b2700 1 --2- 172.21.15.42:0/234480762 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1f4c137ea0 0x7f1f4c1382d0 secure :-1 s=THROTTLE_DONE pgs=232 cs=0 l=1 rx=0x7f1f440032b0 tx=0x7f1f44010ca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:40.115 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:40.096+0000 7f1f508b2700 1 --2- 172.21.15.42:0/234480762 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1f4c137ea0 0x7f1f4c1382d0 secure :-1 s=THROTTLE_DONE pgs=232 cs=0 l=1 rx=0x7f1f440032b0 tx=0x7f1f44010ca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:40.115 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:40.096+0000 7f1f43fff700 1 -- 172.21.15.42:0/234480762 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f440058c0 con 0x7f1f4c137ea0 2019-12-05T05:04:40.115 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:40.096+0000 7f1f510b3700 1 -- 172.21.15.42:0/234480762 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f4c1267b0 con 0x7f1f4c137ea0 2019-12-05T05:04:40.115 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:40.096+0000 7f1f43fff700 1 -- 172.21.15.42:0/234480762 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1f44007530 con 0x7f1f4c137ea0 2019-12-05T05:04:40.115 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:40.096+0000 7f1f510b3700 1 -- 172.21.15.42:0/234480762 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f4c126b10 con 0x7f1f4c137ea0 2019-12-05T05:04:40.115 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:40.096+0000 7f1f508b2700 1 --2- 172.21.15.42:0/234480762 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1f4c137ea0 0x7f1f4c1382d0 secure :-1 s=THROTTLE_DONE pgs=232 cs=0 l=1 rx=0x7f1f440032b0 tx=0x7f1f44010ca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:40.115 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:40.100+0000 7f1f43fff700 1 -- 172.21.15.42:0/234480762 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f1f44012750 con 0x7f1f4c137ea0 2019-12-05T05:04:40.116 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:40.100+0000 7f1f508b2700 1 --2- 172.21.15.42:0/234480762 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1f4c137ea0 0x7f1f4c1382d0 secure :-1 s=THROTTLE_DONE pgs=232 cs=0 l=1 rx=0x7f1f440032b0 tx=0x7f1f44010ca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:40.116 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:40.100+0000 7f1f43fff700 1 --2- 172.21.15.42:0/234480762 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f1f3003b260 0x7f1f3003d770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:04:40.116 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:40.100+0000 7f1f43fff700 1 -- 172.21.15.42:0/234480762 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f1f4404b400 con 0x7f1f4c137ea0 2019-12-05T05:04:40.116 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:40.100+0000 7f1f4b59e700 1 --2- 172.21.15.42:0/234480762 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f1f3003b260 0x7f1f3003d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:04:40.116 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:40.100+0000 7f1f510b3700 1 -- 172.21.15.42:0/234480762 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f24003200 con 0x7f1f4c137ea0 2019-12-05T05:04:40.116 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:40.100+0000 7f1f4b59e700 1 --2- 172.21.15.42:0/234480762 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f1f3003b260 0x7f1f3003d770 crc :-1 s=READY pgs=118 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:04:40.116 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:40.104+0000 7f1f508b2700 1 --2- 172.21.15.42:0/234480762 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1f4c137ea0 0x7f1f4c1382d0 secure :-1 s=THROTTLE_DONE pgs=232 cs=0 l=1 rx=0x7f1f440032b0 tx=0x7f1f44010ca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:40.117 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:40.104+0000 7f1f43fff700 1 -- 172.21.15.42:0/234480762 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f1f44032480 con 0x7f1f4c137ea0 2019-12-05T05:04:40.264 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:40.260+0000 7f1f510b3700 1 -- 172.21.15.42:0/234480762 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1f24003630 con 0x7f1f4c137ea0 2019-12-05T05:04:40.264 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:40.260+0000 7f1f508b2700 1 --2- 172.21.15.42:0/234480762 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1f4c137ea0 0x7f1f4c1382d0 secure :-1 s=THROTTLE_DONE pgs=232 cs=0 l=1 rx=0x7f1f440032b0 tx=0x7f1f44010ca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:40.264 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:40.260+0000 7f1f43fff700 1 -- 172.21.15.42:0/234480762 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+2102 (secure 0 0 0) 0x7f1f440326e0 con 0x7f1f4c137ea0 2019-12-05T05:04:40.268 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 6 2019-12-05T05:04:40.269 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:40.260+0000 7f1f510b3700 1 -- 172.21.15.42:0/234480762 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f1f3003b260 msgr2=0x7f1f3003d770 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:04:40.269 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:40.260+0000 7f1f510b3700 1 --2- 172.21.15.42:0/234480762 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f1f3003b260 0x7f1f3003d770 crc :-1 s=READY pgs=118 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:04:40.269 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:40.260+0000 7f1f510b3700 1 -- 172.21.15.42:0/234480762 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1f4c137ea0 msgr2=0x7f1f4c1382d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:04:40.270 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:40.260+0000 7f1f510b3700 1 --2- 172.21.15.42:0/234480762 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1f4c137ea0 0x7f1f4c1382d0 secure :-1 s=READY pgs=232 cs=0 l=1 rx=0x7f1f440032b0 tx=0x7f1f44010ca0).stop 2019-12-05T05:04:40.270 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:40.260+0000 7f1f510b3700 1 -- 172.21.15.42:0/234480762 shutdown_connections 2019-12-05T05:04:40.271 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:40.260+0000 7f1f510b3700 1 --2- 172.21.15.42:0/234480762 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f1f3003b260 0x7f1f3003d770 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:04:40.271 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:40.260+0000 7f1f510b3700 1 --2- 172.21.15.42:0/234480762 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1f4c137ea0 0x7f1f4c1382d0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:04:40.271 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:40.260+0000 7f1f510b3700 1 --2- 172.21.15.42:0/234480762 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1f4c145080 0x7f1f4c137950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:04:40.271 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:40.260+0000 7f1f510b3700 1 -- 172.21.15.42:0/234480762 shutdown_connections 2019-12-05T05:04:40.271 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:40.260+0000 7f1f510b3700 1 -- 172.21.15.42:0/234480762 wait complete. 2019-12-05T05:04:40.272 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:40.260+0000 7f1f510b3700 1 -- 172.21.15.42:0/234480762 >> 172.21.15.42:0/234480762 conn(0x7f1f4c07e910 msgr2=0x7f1f4c08eb70 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:04:40.272 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:04:40.272 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4354,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/2019200199","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2019200199},{"type":"v1","addr":"172.21.15.42:6835","nonce":2019200199}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":4}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:03:43.669739+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4341},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4341":{"gid":4341,"name":"b","rank":0,"incarnation":5,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6833/3296198322","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3296198322},{"type":"v1","addr":"172.21.15.40:6833","nonce":3296198322}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:04:40.279 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T05:04:42.285 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:04:42.399 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:42.376+0000 7fbefdfa0700 1 Processor -- start 2019-12-05T05:04:42.399 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:42.376+0000 7fbefdfa0700 1 -- start start 2019-12-05T05:04:42.399 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:42.380+0000 7fbefdfa0700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fbef808df80 0x7fbef809f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:04:42.400 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:42.380+0000 7fbefdfa0700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fbef809f830 0x7fbef814efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:04:42.400 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:42.380+0000 7fbefdfa0700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbef809fc60 con 0x7fbef80a7740 2019-12-05T05:04:42.400 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:42.380+0000 7fbefdfa0700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbef80a0b00 con 0x7fbef809f830 2019-12-05T05:04:42.400 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:42.380+0000 7fbefdfa0700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbef808e5b0 con 0x7fbef808df80 2019-12-05T05:04:42.400 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:42.380+0000 7fbeeffff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fbef808df80 0x7fbef809f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:04:42.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:42.380+0000 7fbeeffff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fbef808df80 0x7fbef809f2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:54186/0 (socket says 172.21.15.42:54186) 2019-12-05T05:04:42.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:42.380+0000 7fbef7fff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fbef80a7740 0x7fbef807e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:54222/0 (socket says 172.21.15.42:54222) 2019-12-05T05:04:42.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:42.380+0000 7fbeeffff700 1 -- 172.21.15.42:0/717981658 learned_addr learned my addr 172.21.15.42:0/717981658 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:04:42.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:42.380+0000 7fbefcd3e700 1 --2- 172.21.15.42:0/717981658 >> v2:172.21.15.40:3300/0 conn(0x7fbef809f830 0x7fbef814efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:04:42.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:42.380+0000 7fbeeffff700 1 -- 172.21.15.42:0/717981658 >> v2:172.21.15.40:3300/0 conn(0x7fbef809f830 msgr2=0x7fbef814efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:04:42.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:42.380+0000 7fbeeffff700 1 --2- 172.21.15.42:0/717981658 >> v2:172.21.15.40:3300/0 conn(0x7fbef809f830 0x7fbef814efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:04:42.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:42.380+0000 7fbeeffff700 1 -- 172.21.15.42:0/717981658 >> v1:172.21.15.42:6789/0 conn(0x7fbef80a7740 legacy=0x7fbef807e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:04:42.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:42.380+0000 7fbeeffff700 1 -- 172.21.15.42:0/717981658 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbef80a0710 con 0x7fbef808df80 2019-12-05T05:04:42.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:42.380+0000 7fbeeffff700 1 --2- 172.21.15.42:0/717981658 >> v2:172.21.15.42:3300/0 conn(0x7fbef808df80 0x7fbef809f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fbee0002f30 tx=0x7fbee000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:42.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:42.380+0000 7fbeeffff700 1 --2- 172.21.15.42:0/717981658 >> v2:172.21.15.42:3300/0 conn(0x7fbef808df80 0x7fbef809f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fbee0002f30 tx=0x7fbee000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:42.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:42.380+0000 7fbeeffff700 1 --2- 172.21.15.42:0/717981658 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbef808df80 0x7fbef809f2e0 secure :-1 s=READY pgs=234 cs=0 l=1 rx=0x7fbee0002f30 tx=0x7fbee000fd30).ready entity=mon.1 client_cookie=d8f21c539a7e59a8 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:04:42.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:42.380+0000 7fbeeffff700 1 --2- 172.21.15.42:0/717981658 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbef808df80 0x7fbef809f2e0 secure :-1 s=THROTTLE_DONE pgs=234 cs=0 l=1 rx=0x7fbee0002f30 tx=0x7fbee000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:42.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:42.380+0000 7fbef77fe700 1 -- 172.21.15.42:0/717981658 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbee0008c90 con 0x7fbef808df80 2019-12-05T05:04:42.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:42.380+0000 7fbeeffff700 1 --2- 172.21.15.42:0/717981658 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbef808df80 0x7fbef809f2e0 secure :-1 s=THROTTLE_DONE pgs=234 cs=0 l=1 rx=0x7fbee0002f30 tx=0x7fbee000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:42.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:42.380+0000 7fbeeffff700 1 --2- 172.21.15.42:0/717981658 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbef808df80 0x7fbef809f2e0 secure :-1 s=THROTTLE_DONE pgs=234 cs=0 l=1 rx=0x7fbee0002f30 tx=0x7fbee000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:42.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:42.380+0000 7fbef77fe700 1 -- 172.21.15.42:0/717981658 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbee00088b0 con 0x7fbef808df80 2019-12-05T05:04:42.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:42.380+0000 7fbef77fe700 1 -- 172.21.15.42:0/717981658 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbee0011ad0 con 0x7fbef808df80 2019-12-05T05:04:42.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:42.380+0000 7fbefdfa0700 1 -- 172.21.15.42:0/717981658 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbef808df80 msgr2=0x7fbef809f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:04:42.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:42.380+0000 7fbefdfa0700 1 --2- 172.21.15.42:0/717981658 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbef808df80 0x7fbef809f2e0 secure :-1 s=READY pgs=234 cs=0 l=1 rx=0x7fbee0002f30 tx=0x7fbee000fd30).stop 2019-12-05T05:04:42.404 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:42.380+0000 7fbefdfa0700 1 -- 172.21.15.42:0/717981658 shutdown_connections 2019-12-05T05:04:42.404 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:42.380+0000 7fbefdfa0700 1 --2- 172.21.15.42:0/717981658 >> v2:172.21.15.40:3300/0 conn(0x7fbef809f830 0x7fbef814efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:04:42.404 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:42.380+0000 7fbefdfa0700 1 --2- 172.21.15.42:0/717981658 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbef808df80 0x7fbef809f2e0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:04:42.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:42.380+0000 7fbefdfa0700 1 -- 172.21.15.42:0/717981658 shutdown_connections 2019-12-05T05:04:42.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:42.380+0000 7fbefdfa0700 1 -- 172.21.15.42:0/717981658 wait complete. 2019-12-05T05:04:42.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:42.380+0000 7fbefdfa0700 1 -- 172.21.15.42:0/717981658 >> 172.21.15.42:0/717981658 conn(0x7fbef80aabe0 msgr2=0x7fbef80a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:04:42.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:42.380+0000 7fbefdfa0700 1 Processor -- start 2019-12-05T05:04:42.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:42.380+0000 7fbefdfa0700 1 -- start start 2019-12-05T05:04:42.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:42.384+0000 7fbefdfa0700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fbef81485d0 0x7fbef814dba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:04:42.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:42.384+0000 7fbefdfa0700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fbef812a0a0 0x7fbef812a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:04:42.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:42.384+0000 7fbefdfa0700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbef8116db0 con 0x7fbef808eeb0 2019-12-05T05:04:42.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:42.384+0000 7fbefdfa0700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbef814e2b0 con 0x7fbef812a0a0 2019-12-05T05:04:42.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:42.384+0000 7fbefdfa0700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbef814e550 con 0x7fbef81485d0 2019-12-05T05:04:42.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:42.384+0000 7fbef7fff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fbef808eeb0 0x7fbef81165b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:54228/0 (socket says 172.21.15.42:54228) 2019-12-05T05:04:42.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:42.384+0000 7fbef7fff700 1 -- 172.21.15.42:0/3247002818 learned_addr learned my addr 172.21.15.42:0/3247002818 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:04:42.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:42.384+0000 7fbeeffff700 1 --2- 172.21.15.42:0/3247002818 >> v2:172.21.15.42:3300/0 conn(0x7fbef81485d0 0x7fbef814dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:04:42.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:42.384+0000 7fbefcd3e700 1 --2- 172.21.15.42:0/3247002818 >> v2:172.21.15.40:3300/0 conn(0x7fbef812a0a0 0x7fbef812a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:04:42.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:42.384+0000 7fbef57fa700 1 -- 172.21.15.42:0/3247002818 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2228700490 0 0) 0x7fbef8116db0 con 0x7fbef808eeb0 2019-12-05T05:04:42.411 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:42.384+0000 7fbef57fa700 1 -- 172.21.15.42:0/3247002818 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbedc0030b0 con 0x7fbef808eeb0 2019-12-05T05:04:42.411 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:42.384+0000 7fbeeffff700 1 -- 172.21.15.42:0/3247002818 >> v2:172.21.15.40:3300/0 conn(0x7fbef812a0a0 msgr2=0x7fbef812a4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:04:42.411 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:42.384+0000 7fbeeffff700 1 --2- 172.21.15.42:0/3247002818 >> v2:172.21.15.40:3300/0 conn(0x7fbef812a0a0 0x7fbef812a4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:04:42.411 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:42.384+0000 7fbeeffff700 1 -- 172.21.15.42:0/3247002818 >> v1:172.21.15.42:6789/0 conn(0x7fbef808eeb0 legacy=0x7fbef81165b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:04:42.412 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:42.384+0000 7fbeeffff700 1 -- 172.21.15.42:0/3247002818 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbef814e7f0 con 0x7fbef81485d0 2019-12-05T05:04:42.412 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:42.384+0000 7fbeeffff700 1 --2- 172.21.15.42:0/3247002818 >> v2:172.21.15.42:3300/0 conn(0x7fbef81485d0 0x7fbef814dba0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fbef808f480 tx=0x7fbee0008e30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:42.412 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:42.384+0000 7fbeeffff700 1 --2- 172.21.15.42:0/3247002818 >> v2:172.21.15.42:3300/0 conn(0x7fbef81485d0 0x7fbef814dba0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fbef808f480 tx=0x7fbee0008e30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:42.417 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:42.384+0000 7fbeeffff700 1 --2- 172.21.15.42:0/3247002818 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbef81485d0 0x7fbef814dba0 secure :-1 s=READY pgs=236 cs=0 l=1 rx=0x7fbef808f480 tx=0x7fbee0008e30).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:04:42.417 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:42.384+0000 7fbeeffff700 1 --2- 172.21.15.42:0/3247002818 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbef81485d0 0x7fbef814dba0 secure :-1 s=THROTTLE_DONE pgs=236 cs=0 l=1 rx=0x7fbef808f480 tx=0x7fbee0008e30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:42.417 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:42.384+0000 7fbeeffff700 1 --2- 172.21.15.42:0/3247002818 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbef81485d0 0x7fbef814dba0 secure :-1 s=THROTTLE_DONE pgs=236 cs=0 l=1 rx=0x7fbef808f480 tx=0x7fbee0008e30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:42.417 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:42.384+0000 7fbef57fa700 1 -- 172.21.15.42:0/3247002818 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbee000f680 con 0x7fbef81485d0 2019-12-05T05:04:42.417 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:42.384+0000 7fbeeffff700 1 --2- 172.21.15.42:0/3247002818 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbef81485d0 0x7fbef814dba0 secure :-1 s=THROTTLE_DONE pgs=236 cs=0 l=1 rx=0x7fbef808f480 tx=0x7fbee0008e30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:42.419 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:42.384+0000 7fbef57fa700 1 -- 172.21.15.42:0/3247002818 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbee00116f0 con 0x7fbef81485d0 2019-12-05T05:04:42.419 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:42.384+0000 7fbef57fa700 1 -- 172.21.15.42:0/3247002818 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbee0005920 con 0x7fbef81485d0 2019-12-05T05:04:42.420 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:42.384+0000 7fbefdfa0700 1 -- 172.21.15.42:0/3247002818 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbef812aa20 con 0x7fbef81485d0 2019-12-05T05:04:42.420 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:42.384+0000 7fbefdfa0700 1 -- 172.21.15.42:0/3247002818 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbef8131510 con 0x7fbef81485d0 2019-12-05T05:04:42.428 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:42.384+0000 7fbeeffff700 1 --2- 172.21.15.42:0/3247002818 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbef81485d0 0x7fbef814dba0 secure :-1 s=THROTTLE_DONE pgs=236 cs=0 l=1 rx=0x7fbef808f480 tx=0x7fbee0008e30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:42.429 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:42.384+0000 7fbef57fa700 1 -- 172.21.15.42:0/3247002818 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fbee0005bf0 con 0x7fbef81485d0 2019-12-05T05:04:42.429 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:42.384+0000 7fbeeffff700 1 --2- 172.21.15.42:0/3247002818 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbef81485d0 0x7fbef814dba0 secure :-1 s=THROTTLE_DONE pgs=236 cs=0 l=1 rx=0x7fbef808f480 tx=0x7fbee0008e30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:42.429 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:42.384+0000 7fbef57fa700 1 --2- 172.21.15.42:0/3247002818 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbedc03b270 0x7fbedc03d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:04:42.429 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:42.384+0000 7fbef57fa700 1 -- 172.21.15.42:0/3247002818 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6031+0+0 (secure 0 0 0) 0x7fbee004bb50 con 0x7fbef81485d0 2019-12-05T05:04:42.430 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:42.384+0000 7fbef7fff700 1 --2- 172.21.15.42:0/3247002818 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbedc03b270 0x7fbedc03d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:04:42.430 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:42.388+0000 7fbef7fff700 1 --2- 172.21.15.42:0/3247002818 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbedc03b270 0x7fbedc03d780 crc :-1 s=READY pgs=119 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:04:42.430 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:42.392+0000 7fbefdfa0700 1 -- 172.21.15.42:0/3247002818 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbed0003200 con 0x7fbef81485d0 2019-12-05T05:04:42.431 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:42.400+0000 7fbeeffff700 1 --2- 172.21.15.42:0/3247002818 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbef81485d0 0x7fbef814dba0 secure :-1 s=THROTTLE_DONE pgs=236 cs=0 l=1 rx=0x7fbef808f480 tx=0x7fbee0008e30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:42.431 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:42.400+0000 7fbef57fa700 1 -- 172.21.15.42:0/3247002818 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fbee0032e30 con 0x7fbef81485d0 2019-12-05T05:04:42.563 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:42.556+0000 7fbefdfa0700 1 -- 172.21.15.42:0/3247002818 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbed0003630 con 0x7fbef81485d0 2019-12-05T05:04:42.564 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:42.560+0000 7fbeeffff700 1 --2- 172.21.15.42:0/3247002818 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbef81485d0 0x7fbef814dba0 secure :-1 s=THROTTLE_DONE pgs=236 cs=0 l=1 rx=0x7fbef808f480 tx=0x7fbee0008e30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:42.564 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:42.560+0000 7fbef57fa700 1 -- 172.21.15.42:0/3247002818 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+2102 (secure 0 0 0) 0x7fbee00307e0 con 0x7fbef81485d0 2019-12-05T05:04:42.568 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 6 2019-12-05T05:04:42.568 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:42.560+0000 7fbefdfa0700 1 -- 172.21.15.42:0/3247002818 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbedc03b270 msgr2=0x7fbedc03d780 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:04:42.568 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:42.560+0000 7fbefdfa0700 1 --2- 172.21.15.42:0/3247002818 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbedc03b270 0x7fbedc03d780 crc :-1 s=READY pgs=119 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:04:42.569 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:42.560+0000 7fbefdfa0700 1 -- 172.21.15.42:0/3247002818 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbef81485d0 msgr2=0x7fbef814dba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:04:42.569 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:42.560+0000 7fbefdfa0700 1 --2- 172.21.15.42:0/3247002818 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbef81485d0 0x7fbef814dba0 secure :-1 s=READY pgs=236 cs=0 l=1 rx=0x7fbef808f480 tx=0x7fbee0008e30).stop 2019-12-05T05:04:42.569 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:42.560+0000 7fbefdfa0700 1 -- 172.21.15.42:0/3247002818 shutdown_connections 2019-12-05T05:04:42.569 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:42.560+0000 7fbefdfa0700 1 --2- 172.21.15.42:0/3247002818 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbedc03b270 0x7fbedc03d780 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:04:42.569 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:42.560+0000 7fbefdfa0700 1 --2- 172.21.15.42:0/3247002818 >> v2:172.21.15.40:3300/0 conn(0x7fbef812a0a0 0x7fbef812a4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:04:42.569 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:42.560+0000 7fbefdfa0700 1 --2- 172.21.15.42:0/3247002818 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbef81485d0 0x7fbef814dba0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:04:42.569 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:42.560+0000 7fbefdfa0700 1 -- 172.21.15.42:0/3247002818 shutdown_connections 2019-12-05T05:04:42.570 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:42.560+0000 7fbefdfa0700 1 -- 172.21.15.42:0/3247002818 wait complete. 2019-12-05T05:04:42.570 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:42.560+0000 7fbefdfa0700 1 -- 172.21.15.42:0/3247002818 >> 172.21.15.42:0/3247002818 conn(0x7fbef80aabe0 msgr2=0x7fbef80a9770 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:04:42.570 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:04:42.570 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4354,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/2019200199","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2019200199},{"type":"v1","addr":"172.21.15.42:6835","nonce":2019200199}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":4}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:03:43.669739+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4341},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4341":{"gid":4341,"name":"b","rank":0,"incarnation":5,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6833/3296198322","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3296198322},{"type":"v1","addr":"172.21.15.40:6833","nonce":3296198322}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:04:42.576 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T05:04:44.577 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:04:44.676 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.660+0000 7fafbb49e700 1 Processor -- start 2019-12-05T05:04:44.676 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.660+0000 7fafbb49e700 1 -- start start 2019-12-05T05:04:44.676 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.660+0000 7fafbb49e700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fafb40a7740 0x7fafb408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:04:44.676 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.660+0000 7fafbb49e700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fafb407e720 0x7fafb407eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:04:44.677 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.660+0000 7fafbb49e700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafb40a0710 con 0x7fafb409f2e0 2019-12-05T05:04:44.677 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.660+0000 7fafbb49e700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafb40a0b00 con 0x7fafb407e720 2019-12-05T05:04:44.677 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.660+0000 7fafbb49e700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafb408e5b0 con 0x7fafb40a7740 2019-12-05T05:04:44.677 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.664+0000 7fafb923a700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fafb407e720 0x7fafb407eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:04:44.677 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.664+0000 7fafb9a3b700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fafb40a7740 0x7fafb408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:04:44.677 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.664+0000 7fafb923a700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fafb407e720 0x7fafb407eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.42:42028/0 (socket says 172.21.15.42:42028) 2019-12-05T05:04:44.678 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.664+0000 7fafba23c700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fafb409f2e0 0x7fafb409f710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:35460/0 (socket says 172.21.15.42:35460) 2019-12-05T05:04:44.678 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.664+0000 7fafb923a700 1 -- 172.21.15.42:0/2569236533 learned_addr learned my addr 172.21.15.42:0/2569236533 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:04:44.678 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.664+0000 7fafb923a700 1 -- 172.21.15.42:0/2569236533 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fafb40a7740 msgr2=0x7fafb408df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:04:44.678 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.664+0000 7fafb923a700 1 --2- 172.21.15.42:0/2569236533 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fafb40a7740 0x7fafb408df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:04:44.678 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.664+0000 7fafb923a700 1 -- 172.21.15.42:0/2569236533 >> v1:172.21.15.40:6789/0 conn(0x7fafb409f2e0 legacy=0x7fafb409f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:04:44.678 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.664+0000 7fafb923a700 1 -- 172.21.15.42:0/2569236533 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafb407f0a0 con 0x7fafb407e720 2019-12-05T05:04:44.678 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.664+0000 7fafb923a700 1 --2- 172.21.15.42:0/2569236533 >> v2:172.21.15.40:3300/0 conn(0x7fafb407e720 0x7fafb407eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fafa4008790 tx=0x7fafa400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:44.679 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.664+0000 7fafb923a700 1 --2- 172.21.15.42:0/2569236533 >> v2:172.21.15.40:3300/0 conn(0x7fafb407e720 0x7fafb407eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fafa4008790 tx=0x7fafa400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:44.679 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.664+0000 7fafb923a700 1 --2- 172.21.15.42:0/2569236533 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fafb407e720 0x7fafb407eb50 secure :-1 s=READY pgs=196 cs=0 l=1 rx=0x7fafa4008790 tx=0x7fafa400fd30).ready entity=mon.0 client_cookie=90b6009376ea051e server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:04:44.679 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.664+0000 7fafb923a700 1 --2- 172.21.15.42:0/2569236533 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fafb407e720 0x7fafb407eb50 secure :-1 s=THROTTLE_DONE pgs=196 cs=0 l=1 rx=0x7fafa4008790 tx=0x7fafa400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:44.679 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.664+0000 7fafb8a39700 1 -- 172.21.15.42:0/2569236533 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fafa40113b0 con 0x7fafb407e720 2019-12-05T05:04:44.679 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.664+0000 7fafb923a700 1 --2- 172.21.15.42:0/2569236533 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fafb407e720 0x7fafb407eb50 secure :-1 s=THROTTLE_DONE pgs=196 cs=0 l=1 rx=0x7fafa4008790 tx=0x7fafa400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:44.679 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.664+0000 7fafb8a39700 1 -- 172.21.15.42:0/2569236533 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafa4008bb0 con 0x7fafb407e720 2019-12-05T05:04:44.679 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.664+0000 7fafb923a700 1 --2- 172.21.15.42:0/2569236533 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fafb407e720 0x7fafb407eb50 secure :-1 s=THROTTLE_DONE pgs=196 cs=0 l=1 rx=0x7fafa4008790 tx=0x7fafa400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:44.680 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.664+0000 7fafb8a39700 1 -- 172.21.15.42:0/2569236533 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fafa4011990 con 0x7fafb407e720 2019-12-05T05:04:44.680 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.664+0000 7fafbb49e700 1 -- 172.21.15.42:0/2569236533 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fafb407e720 msgr2=0x7fafb407eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:04:44.680 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.664+0000 7fafbb49e700 1 --2- 172.21.15.42:0/2569236533 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fafb407e720 0x7fafb407eb50 secure :-1 s=READY pgs=196 cs=0 l=1 rx=0x7fafa4008790 tx=0x7fafa400fd30).stop 2019-12-05T05:04:44.680 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.664+0000 7fafbb49e700 1 -- 172.21.15.42:0/2569236533 shutdown_connections 2019-12-05T05:04:44.680 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.664+0000 7fafbb49e700 1 --2- 172.21.15.42:0/2569236533 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fafb40a7740 0x7fafb408df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:04:44.680 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.664+0000 7fafbb49e700 1 --2- 172.21.15.42:0/2569236533 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fafb407e720 0x7fafb407eb50 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:04:44.681 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.664+0000 7fafbb49e700 1 -- 172.21.15.42:0/2569236533 shutdown_connections 2019-12-05T05:04:44.681 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.664+0000 7fafbb49e700 1 -- 172.21.15.42:0/2569236533 wait complete. 2019-12-05T05:04:44.681 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.664+0000 7fafbb49e700 1 -- 172.21.15.42:0/2569236533 >> 172.21.15.42:0/2569236533 conn(0x7fafb40aabe0 msgr2=0x7fafb40a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:04:44.681 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.664+0000 7fafbb49e700 1 Processor -- start 2019-12-05T05:04:44.681 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.664+0000 7fafbb49e700 1 -- start start 2019-12-05T05:04:44.681 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.668+0000 7fafbb49e700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fafb411a8e0 0x7fafb40a5890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:04:44.681 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.668+0000 7fafbb49e700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fafb412b960 0x7fafb4131cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:04:44.682 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.668+0000 7fafbb49e700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafb4119510 con 0x7fafb412b530 2019-12-05T05:04:44.682 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.668+0000 7fafbb49e700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafb411ad10 con 0x7fafb412b960 2019-12-05T05:04:44.682 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.668+0000 7fafbb49e700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafb4119940 con 0x7fafb411a8e0 2019-12-05T05:04:44.682 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.668+0000 7fafb923a700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fafb412b530 0x7fafb41314d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:54256/0 (socket says 172.21.15.42:54256) 2019-12-05T05:04:44.682 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.668+0000 7fafb923a700 1 -- 172.21.15.42:0/1673068411 learned_addr learned my addr 172.21.15.42:0/1673068411 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:04:44.682 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.668+0000 7fafba23c700 1 --2- 172.21.15.42:0/1673068411 >> v2:172.21.15.40:3300/0 conn(0x7fafb412b960 0x7fafb4131cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:04:44.682 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.668+0000 7fafb9a3b700 1 --2- 172.21.15.42:0/1673068411 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fafb411a8e0 0x7fafb40a5890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:04:44.683 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.668+0000 7fafaa7fc700 1 -- 172.21.15.42:0/1673068411 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 403700351 0 0) 0x7fafb4119510 con 0x7fafb412b530 2019-12-05T05:04:44.683 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.668+0000 7fafaa7fc700 1 -- 172.21.15.42:0/1673068411 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf9c0030b0 con 0x7fafb412b530 2019-12-05T05:04:44.683 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.668+0000 7fafaa7fc700 1 -- 172.21.15.42:0/1673068411 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 1577161989 0 0) 0x7faf9c0030b0 con 0x7fafb412b530 2019-12-05T05:04:44.683 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.668+0000 7fafaa7fc700 1 -- 172.21.15.42:0/1673068411 --> v1:172.21.15.42:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fafb4119510 con 0x7fafb412b530 2019-12-05T05:04:44.683 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.668+0000 7fafaa7fc700 1 -- 172.21.15.42:0/1673068411 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7fafa40113b0 con 0x7fafb412b530 2019-12-05T05:04:44.684 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.668+0000 7fafaa7fc700 1 -- 172.21.15.42:0/1673068411 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fafb411a8e0 msgr2=0x7fafb40a5890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:04:44.684 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.668+0000 7fafaa7fc700 1 --2- 172.21.15.42:0/1673068411 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fafb411a8e0 0x7fafb40a5890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:04:44.684 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.668+0000 7fafaa7fc700 1 -- 172.21.15.42:0/1673068411 >> v2:172.21.15.40:3300/0 conn(0x7fafb412b960 msgr2=0x7fafb4131cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:04:44.684 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.668+0000 7fafaa7fc700 1 --2- 172.21.15.42:0/1673068411 >> v2:172.21.15.40:3300/0 conn(0x7fafb412b960 0x7fafb4131cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:04:44.684 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.668+0000 7fafaa7fc700 1 -- 172.21.15.42:0/1673068411 >> v1:172.21.15.42:6789/0 conn(0x7fafb412b530 legacy=0x7fafb41314d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:04:44.684 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.668+0000 7fafaa7fc700 1 --2- 172.21.15.42:0/1673068411 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7faf9c0038a0 0x7faf9c003cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:04:44.684 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.668+0000 7fafaa7fc700 1 --2- 172.21.15.42:0/1673068411 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faf9c004420 0x7faf9c004850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:04:44.685 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.668+0000 7fafaa7fc700 1 --2- 172.21.15.42:0/1673068411 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf9c004da0 0x7faf9c00b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:04:44.685 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.668+0000 7fafaa7fc700 1 -- 172.21.15.42:0/1673068411 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafb411ad10 con 0x7faf9c004420 2019-12-05T05:04:44.685 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.668+0000 7fafaa7fc700 1 -- 172.21.15.42:0/1673068411 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafb4119940 con 0x7faf9c004da0 2019-12-05T05:04:44.685 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.668+0000 7fafaa7fc700 1 -- 172.21.15.42:0/1673068411 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf9c00bbc0 con 0x7faf9c0038a0 2019-12-05T05:04:44.685 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.668+0000 7fafba23c700 1 --2- 172.21.15.42:0/1673068411 >> v2:172.21.15.40:3300/0 conn(0x7fafb412b960 0x7fafb4131cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:04:44.685 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.668+0000 7fafba23c700 1 --2- 172.21.15.42:0/1673068411 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf9c004da0 0x7faf9c00b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:04:44.685 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.668+0000 7fafb923a700 1 --2- 172.21.15.42:0/1673068411 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faf9c004420 0x7faf9c004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:04:44.686 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.668+0000 7fafb9a3b700 1 --2- 172.21.15.42:0/1673068411 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7faf9c0038a0 0x7faf9c003cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:04:44.686 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.668+0000 7fafba23c700 1 -- 172.21.15.42:0/1673068411 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7faf9c0038a0 msgr2=0x7faf9c003cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:04:44.686 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.668+0000 7fafba23c700 1 --2- 172.21.15.42:0/1673068411 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7faf9c0038a0 0x7faf9c003cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:04:44.686 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.668+0000 7fafba23c700 1 -- 172.21.15.42:0/1673068411 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faf9c004420 msgr2=0x7faf9c004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:04:44.686 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.668+0000 7fafba23c700 1 --2- 172.21.15.42:0/1673068411 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faf9c004420 0x7faf9c004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:04:44.686 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.668+0000 7fafba23c700 1 -- 172.21.15.42:0/1673068411 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafb413cef0 con 0x7faf9c004da0 2019-12-05T05:04:44.686 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.668+0000 7fafbb49e700 1 -- 172.21.15.42:0/1673068411 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafb411a8e0 con 0x7faf9c004da0 2019-12-05T05:04:44.687 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.668+0000 7fafba23c700 1 --2- 172.21.15.42:0/1673068411 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf9c004da0 0x7faf9c00b670 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fafac001210 tx=0x7fafac011250).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:44.687 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.668+0000 7fafbb49e700 1 -- 172.21.15.42:0/1673068411 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafb40a5890 con 0x7faf9c004da0 2019-12-05T05:04:44.687 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.668+0000 7fafba23c700 1 --2- 172.21.15.42:0/1673068411 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf9c004da0 0x7faf9c00b670 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fafac001210 tx=0x7fafac011250).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:44.687 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.668+0000 7fafba23c700 1 --2- 172.21.15.42:0/1673068411 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf9c004da0 0x7faf9c00b670 secure :-1 s=READY pgs=238 cs=0 l=1 rx=0x7fafac001210 tx=0x7fafac011250).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:04:44.687 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.668+0000 7fafb9a3b700 1 --2- 172.21.15.42:0/1673068411 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7faf9c0038a0 0x7faf9c003cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).send_auth_request state changed! 2019-12-05T05:04:44.687 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.668+0000 7fafba23c700 1 --2- 172.21.15.42:0/1673068411 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf9c004da0 0x7faf9c00b670 secure :-1 s=THROTTLE_DONE pgs=238 cs=0 l=1 rx=0x7fafac001210 tx=0x7fafac011250).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:44.687 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.668+0000 7fafba23c700 1 --2- 172.21.15.42:0/1673068411 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf9c004da0 0x7faf9c00b670 secure :-1 s=THROTTLE_DONE pgs=238 cs=0 l=1 rx=0x7fafac001210 tx=0x7fafac011250).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:44.688 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.668+0000 7fafba23c700 1 --2- 172.21.15.42:0/1673068411 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf9c004da0 0x7faf9c00b670 secure :-1 s=THROTTLE_DONE pgs=238 cs=0 l=1 rx=0x7fafac001210 tx=0x7fafac011250).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:44.688 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.668+0000 7fafaa7fc700 1 -- 172.21.15.42:0/1673068411 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fafac00eb10 con 0x7faf9c004da0 2019-12-05T05:04:44.688 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.668+0000 7fafaa7fc700 1 -- 172.21.15.42:0/1673068411 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafac011c90 con 0x7faf9c004da0 2019-12-05T05:04:44.688 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.668+0000 7fafaa7fc700 1 -- 172.21.15.42:0/1673068411 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fafac0048e0 con 0x7faf9c004da0 2019-12-05T05:04:44.688 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.668+0000 7fafba23c700 1 --2- 172.21.15.42:0/1673068411 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf9c004da0 0x7faf9c00b670 secure :-1 s=THROTTLE_DONE pgs=238 cs=0 l=1 rx=0x7fafac001210 tx=0x7fafac011250).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:44.688 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.668+0000 7fafaa7fc700 1 -- 172.21.15.42:0/1673068411 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fafac004ae0 con 0x7faf9c004da0 2019-12-05T05:04:44.688 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.668+0000 7fafba23c700 1 --2- 172.21.15.42:0/1673068411 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf9c004da0 0x7faf9c00b670 secure :-1 s=THROTTLE_DONE pgs=238 cs=0 l=1 rx=0x7fafac001210 tx=0x7fafac011250).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:44.689 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.668+0000 7fafaa7fc700 1 --2- 172.21.15.42:0/1673068411 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7faf9c043a40 0x7faf9c045f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:04:44.689 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.668+0000 7fafaa7fc700 1 -- 172.21.15.42:0/1673068411 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6117+0+0 (secure 0 0 0) 0x7fafac04b2b0 con 0x7faf9c004da0 2019-12-05T05:04:44.689 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.668+0000 7fafb9a3b700 1 --2- 172.21.15.42:0/1673068411 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7faf9c043a40 0x7faf9c045f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:04:44.689 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.672+0000 7fafb9a3b700 1 --2- 172.21.15.42:0/1673068411 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7faf9c043a40 0x7faf9c045f50 crc :-1 s=READY pgs=120 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:04:44.690 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.676+0000 7fafbb49e700 1 -- 172.21.15.42:0/1673068411 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf90003200 con 0x7faf9c004da0 2019-12-05T05:04:44.690 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.684+0000 7fafba23c700 1 --2- 172.21.15.42:0/1673068411 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf9c004da0 0x7faf9c00b670 secure :-1 s=THROTTLE_DONE pgs=238 cs=0 l=1 rx=0x7fafac001210 tx=0x7fafac011250).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:44.690 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.684+0000 7fafaa7fc700 1 -- 172.21.15.42:0/1673068411 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fafac033140 con 0x7faf9c004da0 2019-12-05T05:04:44.855 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.848+0000 7fafbb49e700 1 -- 172.21.15.42:0/1673068411 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faf90003630 con 0x7faf9c004da0 2019-12-05T05:04:44.855 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.852+0000 7fafba23c700 1 --2- 172.21.15.42:0/1673068411 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf9c004da0 0x7faf9c00b670 secure :-1 s=THROTTLE_DONE pgs=238 cs=0 l=1 rx=0x7fafac001210 tx=0x7fafac011250).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:44.856 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.852+0000 7fafaa7fc700 1 -- 172.21.15.42:0/1673068411 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+1772 (secure 0 0 0) 0x7fafac032850 con 0x7faf9c004da0 2019-12-05T05:04:44.856 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 8 2019-12-05T05:04:44.857 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:04:44.857 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:04:44.177728+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4354},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4354":{"gid":4354,"name":"a","rank":0,"incarnation":7,"state":"up:reconnect","state_seq":18,"addr":"172.21.15.42:6835/2019200199","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2019200199},{"type":"v1","addr":"172.21.15.42:6835","nonce":2019200199}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:04:44.859 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.852+0000 7fafbb49e700 1 -- 172.21.15.42:0/1673068411 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7faf9c043a40 msgr2=0x7faf9c045f50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:04:44.859 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.852+0000 7fafbb49e700 1 --2- 172.21.15.42:0/1673068411 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7faf9c043a40 0x7faf9c045f50 crc :-1 s=READY pgs=120 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:04:44.859 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.852+0000 7fafbb49e700 1 -- 172.21.15.42:0/1673068411 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf9c004da0 msgr2=0x7faf9c00b670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:04:44.859 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.852+0000 7fafbb49e700 1 --2- 172.21.15.42:0/1673068411 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf9c004da0 0x7faf9c00b670 secure :-1 s=READY pgs=238 cs=0 l=1 rx=0x7fafac001210 tx=0x7fafac011250).stop 2019-12-05T05:04:44.859 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.852+0000 7fafba23c700 1 -- 172.21.15.42:0/1673068411 reap_dead start 2019-12-05T05:04:44.859 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.852+0000 7fafba23c700 1 -- 172.21.15.42:0/1673068411 reap_dead start 2019-12-05T05:04:44.860 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.852+0000 7fafbb49e700 1 -- 172.21.15.42:0/1673068411 shutdown_connections 2019-12-05T05:04:44.860 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.852+0000 7fafbb49e700 1 -- 172.21.15.42:0/1673068411 shutdown_connections 2019-12-05T05:04:44.860 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.852+0000 7fafbb49e700 1 -- 172.21.15.42:0/1673068411 wait complete. 2019-12-05T05:04:44.860 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.852+0000 7fafbb49e700 1 -- 172.21.15.42:0/1673068411 >> 172.21.15.42:0/1673068411 conn(0x7fafb40a8120 msgr2=0x7fafb411b690 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:04:44.870 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b down, removed from mdsmap 2019-12-05T05:04:44.870 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2019-12-05T05:04:44.870 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:04:44.973 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.964+0000 7fd51fa93700 1 Processor -- start 2019-12-05T05:04:44.973 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.964+0000 7fd51fa93700 1 -- start start 2019-12-05T05:04:44.973 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.964+0000 7fd51fa93700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd51809fae0 0x7fd51814fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:04:44.973 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.964+0000 7fd51fa93700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5180a0710 con 0x7fd51808df80 2019-12-05T05:04:44.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.964+0000 7fd51fa93700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd51807ef20 con 0x7fd5180a7740 2019-12-05T05:04:44.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.964+0000 7fd51fa93700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5180a0b00 con 0x7fd51809fae0 2019-12-05T05:04:44.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.964+0000 7fd51e030700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fd5180a7740 0x7fd51807e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:54268/0 (socket says 172.21.15.42:54268) 2019-12-05T05:04:44.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.964+0000 7fd51e030700 1 -- 172.21.15.42:0/3909690618 learned_addr learned my addr 172.21.15.42:0/3909690618 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:04:44.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.964+0000 7fd51e831700 1 --2- 172.21.15.42:0/3909690618 >> v2:172.21.15.40:3300/0 conn(0x7fd51809fae0 0x7fd51814fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:04:44.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.964+0000 7fd51d02e700 1 -- 172.21.15.42:0/3909690618 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2629304260 0 0) 0x7fd51807ef20 con 0x7fd5180a7740 2019-12-05T05:04:44.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.964+0000 7fd51d02e700 1 -- 172.21.15.42:0/3909690618 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5040030b0 con 0x7fd5180a7740 2019-12-05T05:04:44.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.964+0000 7fd51d02e700 1 -- 172.21.15.42:0/3909690618 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3904227065 0 0) 0x7fd5040030b0 con 0x7fd5180a7740 2019-12-05T05:04:44.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.964+0000 7fd51d02e700 1 -- 172.21.15.42:0/3909690618 >> v2:172.21.15.40:3300/0 conn(0x7fd51809fae0 msgr2=0x7fd51814fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:04:44.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.964+0000 7fd51d02e700 1 --2- 172.21.15.42:0/3909690618 >> v2:172.21.15.40:3300/0 conn(0x7fd51809fae0 0x7fd51814fff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:04:44.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.964+0000 7fd51d02e700 1 -- 172.21.15.42:0/3909690618 >> v1:172.21.15.40:6789/0 conn(0x7fd51808df80 legacy=0x7fd51809f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:04:44.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.964+0000 7fd51d02e700 1 -- 172.21.15.42:0/3909690618 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5180a9ba0 con 0x7fd5180a7740 2019-12-05T05:04:44.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.964+0000 7fd51d02e700 1 -- 172.21.15.42:0/3909690618 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7fd514001750 con 0x7fd5180a7740 2019-12-05T05:04:44.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.964+0000 7fd51d02e700 1 -- 172.21.15.42:0/3909690618 >> v1:172.21.15.42:6789/0 conn(0x7fd5180a7740 legacy=0x7fd51807e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:04:44.976 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.964+0000 7fd51d02e700 1 --2- 172.21.15.42:0/3909690618 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd5040036e0 0x7fd504003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:04:44.976 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.964+0000 7fd51d02e700 1 --2- 172.21.15.42:0/3909690618 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd504004420 0x7fd504004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:04:44.976 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.964+0000 7fd51d02e700 1 --2- 172.21.15.42:0/3909690618 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd504004da0 0x7fd50400b670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:04:44.976 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.964+0000 7fd51d02e700 1 -- 172.21.15.42:0/3909690618 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5180a0b00 con 0x7fd504004420 2019-12-05T05:04:44.976 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.964+0000 7fd51d02e700 1 -- 172.21.15.42:0/3909690618 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd50404fc60 con 0x7fd5040036e0 2019-12-05T05:04:44.976 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.964+0000 7fd51d02e700 1 -- 172.21.15.42:0/3909690618 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd50404fea0 con 0x7fd504004da0 2019-12-05T05:04:44.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.964+0000 7fd51e030700 1 --2- 172.21.15.42:0/3909690618 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd5040036e0 0x7fd504003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:04:44.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.964+0000 7fd51d82f700 1 --2- 172.21.15.42:0/3909690618 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd504004420 0x7fd504004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:04:44.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.964+0000 7fd51e831700 1 --2- 172.21.15.42:0/3909690618 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd504004da0 0x7fd50400b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:04:44.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.964+0000 7fd51e030700 1 -- 172.21.15.42:0/3909690618 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd504004da0 msgr2=0x7fd50400b670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:04:44.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.964+0000 7fd51e030700 1 --2- 172.21.15.42:0/3909690618 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd504004da0 0x7fd50400b670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:04:44.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.964+0000 7fd51e030700 1 -- 172.21.15.42:0/3909690618 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd504004420 msgr2=0x7fd504004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:04:44.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.964+0000 7fd51e030700 1 --2- 172.21.15.42:0/3909690618 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd504004420 0x7fd504004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:04:44.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.964+0000 7fd51e030700 1 -- 172.21.15.42:0/3909690618 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5040501a0 con 0x7fd5040036e0 2019-12-05T05:04:44.978 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.964+0000 7fd51e831700 1 -- 172.21.15.42:0/3909690618 reap_dead start 2019-12-05T05:04:44.978 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.964+0000 7fd51e030700 1 --2- 172.21.15.42:0/3909690618 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd5040036e0 0x7fd504003b50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fd514001460 tx=0x7fd51400c1e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:44.978 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.964+0000 7fd51e030700 1 --2- 172.21.15.42:0/3909690618 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd5040036e0 0x7fd504003b50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fd514001460 tx=0x7fd51400c1e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:44.978 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.964+0000 7fd51e030700 1 --2- 172.21.15.42:0/3909690618 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd5040036e0 0x7fd504003b50 secure :-1 s=READY pgs=240 cs=0 l=1 rx=0x7fd514001460 tx=0x7fd51400c1e0).ready entity=mon.1 client_cookie=ca1176eb58ea4256 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:04:44.978 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.964+0000 7fd51e030700 1 --2- 172.21.15.42:0/3909690618 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd5040036e0 0x7fd504003b50 secure :-1 s=THROTTLE_DONE pgs=240 cs=0 l=1 rx=0x7fd514001460 tx=0x7fd51400c1e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:44.978 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.964+0000 7fd51d02e700 1 -- 172.21.15.42:0/3909690618 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd51400e940 con 0x7fd5040036e0 2019-12-05T05:04:44.979 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.964+0000 7fd51e030700 1 --2- 172.21.15.42:0/3909690618 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd5040036e0 0x7fd504003b50 secure :-1 s=THROTTLE_DONE pgs=240 cs=0 l=1 rx=0x7fd514001460 tx=0x7fd51400c1e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:44.979 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.964+0000 7fd51e030700 1 --2- 172.21.15.42:0/3909690618 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd5040036e0 0x7fd504003b50 secure :-1 s=THROTTLE_DONE pgs=240 cs=0 l=1 rx=0x7fd514001460 tx=0x7fd51400c1e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:44.979 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.964+0000 7fd51d02e700 1 -- 172.21.15.42:0/3909690618 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd514010b90 con 0x7fd5040036e0 2019-12-05T05:04:44.979 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.964+0000 7fd51d02e700 1 -- 172.21.15.42:0/3909690618 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd51400a4f0 con 0x7fd5040036e0 2019-12-05T05:04:44.979 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.964+0000 7fd51fa93700 1 -- 172.21.15.42:0/3909690618 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd5040036e0 msgr2=0x7fd504003b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:04:44.979 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.964+0000 7fd51fa93700 1 --2- 172.21.15.42:0/3909690618 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd5040036e0 0x7fd504003b50 secure :-1 s=READY pgs=240 cs=0 l=1 rx=0x7fd514001460 tx=0x7fd51400c1e0).stop 2019-12-05T05:04:44.979 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.968+0000 7fd51fa93700 1 -- 172.21.15.42:0/3909690618 shutdown_connections 2019-12-05T05:04:44.979 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.968+0000 7fd51fa93700 1 --2- 172.21.15.42:0/3909690618 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd5040036e0 0x7fd504003b50 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:04:44.980 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.968+0000 7fd51fa93700 1 -- 172.21.15.42:0/3909690618 shutdown_connections 2019-12-05T05:04:44.980 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.968+0000 7fd51fa93700 1 -- 172.21.15.42:0/3909690618 wait complete. 2019-12-05T05:04:44.980 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.968+0000 7fd51fa93700 1 -- 172.21.15.42:0/3909690618 >> 172.21.15.42:0/3909690618 conn(0x7fd5180aabe0 msgr2=0x7fd5180a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:04:44.980 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.968+0000 7fd51fa93700 1 Processor -- start 2019-12-05T05:04:44.980 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.968+0000 7fd51fa93700 1 -- start start 2019-12-05T05:04:44.983 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.968+0000 7fd51fa93700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd518119190 0x7fd518118470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:04:44.983 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.968+0000 7fd51fa93700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd5181189c0 0x7fd518071ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:04:44.984 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.968+0000 7fd51fa93700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd518072020 con 0x7fd5181261c0 2019-12-05T05:04:44.984 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.968+0000 7fd51fa93700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5181172a0 con 0x7fd5181189c0 2019-12-05T05:04:44.984 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.968+0000 7fd51fa93700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd518118180 con 0x7fd518119190 2019-12-05T05:04:44.984 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.968+0000 7fd51e831700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd5181189c0 0x7fd518071ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:04:44.984 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.968+0000 7fd51e831700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd5181189c0 0x7fd518071ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:54246/0 (socket says 172.21.15.42:54246) 2019-12-05T05:04:44.984 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.968+0000 7fd51e831700 1 -- 172.21.15.42:0/2307186180 learned_addr learned my addr 172.21.15.42:0/2307186180 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:04:44.985 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.968+0000 7fd51e030700 1 --2- 172.21.15.42:0/2307186180 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd518119190 0x7fd518118470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:04:44.985 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.968+0000 7fd51e831700 1 -- 172.21.15.42:0/2307186180 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd518119190 msgr2=0x7fd518118470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:04:44.985 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.968+0000 7fd51e831700 1 --2- 172.21.15.42:0/2307186180 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd518119190 0x7fd518118470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:04:44.985 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.968+0000 7fd51e831700 1 -- 172.21.15.42:0/2307186180 >> v1:172.21.15.40:6789/0 conn(0x7fd5181261c0 legacy=0x7fd5180712d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:04:44.985 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.968+0000 7fd51e831700 1 -- 172.21.15.42:0/2307186180 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd51805ae30 con 0x7fd5181189c0 2019-12-05T05:04:44.985 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.968+0000 7fd51e831700 1 --2- 172.21.15.42:0/2307186180 >> v2:172.21.15.42:3300/0 conn(0x7fd5181189c0 0x7fd518071ad0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fd508000f10 tx=0x7fd5100109f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:44.985 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.968+0000 7fd51e831700 1 --2- 172.21.15.42:0/2307186180 >> v2:172.21.15.42:3300/0 conn(0x7fd5181189c0 0x7fd518071ad0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fd508000f10 tx=0x7fd5100109f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:44.986 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.968+0000 7fd51e831700 1 --2- 172.21.15.42:0/2307186180 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd5181189c0 0x7fd518071ad0 secure :-1 s=READY pgs=241 cs=0 l=1 rx=0x7fd508000f10 tx=0x7fd5100109f0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:04:44.986 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.968+0000 7fd51e831700 1 --2- 172.21.15.42:0/2307186180 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd5181189c0 0x7fd518071ad0 secure :-1 s=THROTTLE_DONE pgs=241 cs=0 l=1 rx=0x7fd508000f10 tx=0x7fd5100109f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:44.986 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.968+0000 7fd51e831700 1 --2- 172.21.15.42:0/2307186180 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd5181189c0 0x7fd518071ad0 secure :-1 s=THROTTLE_DONE pgs=241 cs=0 l=1 rx=0x7fd508000f10 tx=0x7fd5100109f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:44.986 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.968+0000 7fd51e831700 1 --2- 172.21.15.42:0/2307186180 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd5181189c0 0x7fd518071ad0 secure :-1 s=THROTTLE_DONE pgs=241 cs=0 l=1 rx=0x7fd508000f10 tx=0x7fd5100109f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:44.986 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.972+0000 7fd50effd700 1 -- 172.21.15.42:0/2307186180 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd510007350 con 0x7fd5181189c0 2019-12-05T05:04:44.986 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.972+0000 7fd50effd700 1 -- 172.21.15.42:0/2307186180 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd510005cb0 con 0x7fd5181189c0 2019-12-05T05:04:44.986 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.972+0000 7fd51fa93700 1 -- 172.21.15.42:0/2307186180 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd51811b040 con 0x7fd5181189c0 2019-12-05T05:04:44.987 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.972+0000 7fd50effd700 1 -- 172.21.15.42:0/2307186180 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd5100084a0 con 0x7fd5181189c0 2019-12-05T05:04:44.987 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.972+0000 7fd51fa93700 1 -- 172.21.15.42:0/2307186180 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5181198b0 con 0x7fd5181189c0 2019-12-05T05:04:44.987 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.972+0000 7fd51e831700 1 --2- 172.21.15.42:0/2307186180 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd5181189c0 0x7fd518071ad0 secure :-1 s=THROTTLE_DONE pgs=241 cs=0 l=1 rx=0x7fd508000f10 tx=0x7fd5100109f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:44.987 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.972+0000 7fd50effd700 1 -- 172.21.15.42:0/2307186180 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fd5100086a0 con 0x7fd5181189c0 2019-12-05T05:04:44.987 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.972+0000 7fd51e831700 1 --2- 172.21.15.42:0/2307186180 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd5181189c0 0x7fd518071ad0 secure :-1 s=THROTTLE_DONE pgs=241 cs=0 l=1 rx=0x7fd508000f10 tx=0x7fd5100109f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:44.987 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.972+0000 7fd50effd700 1 --2- 172.21.15.42:0/2307186180 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd50003ad90 0x7fd50003d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:04:44.987 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.972+0000 7fd50effd700 1 -- 172.21.15.42:0/2307186180 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6117+0+0 (secure 0 0 0) 0x7fd51004b2f0 con 0x7fd5181189c0 2019-12-05T05:04:44.988 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.972+0000 7fd51e030700 1 --2- 172.21.15.42:0/2307186180 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd50003ad90 0x7fd50003d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:04:44.988 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.972+0000 7fd51e030700 1 --2- 172.21.15.42:0/2307186180 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd50003ad90 0x7fd50003d2a0 crc :-1 s=READY pgs=121 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:04:44.988 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.980+0000 7fd51fa93700 1 -- 172.21.15.42:0/2307186180 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd51805d110 con 0x7fd5181189c0 2019-12-05T05:04:44.990 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.984+0000 7fd51e831700 1 --2- 172.21.15.42:0/2307186180 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd5181189c0 0x7fd518071ad0 secure :-1 s=THROTTLE_DONE pgs=241 cs=0 l=1 rx=0x7fd508000f10 tx=0x7fd5100109f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:44.990 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:44.984+0000 7fd50effd700 1 -- 172.21.15.42:0/2307186180 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fd51811b040 con 0x7fd5181189c0 2019-12-05T05:04:45.145 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:45.140+0000 7fd51fa93700 1 -- 172.21.15.42:0/2307186180 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd51805b8c0 con 0x7fd5181189c0 2019-12-05T05:04:45.145 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:45.140+0000 7fd51e831700 1 --2- 172.21.15.42:0/2307186180 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd5181189c0 0x7fd518071ad0 secure :-1 s=THROTTLE_DONE pgs=241 cs=0 l=1 rx=0x7fd508000f10 tx=0x7fd5100109f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:45.146 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:45.140+0000 7fd50effd700 1 -- 172.21.15.42:0/2307186180 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+1772 (secure 0 0 0) 0x7fd510032370 con 0x7fd5181189c0 2019-12-05T05:04:45.146 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 8 2019-12-05T05:04:45.146 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:04:45.146 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:04:44.177728+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4354},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4354":{"gid":4354,"name":"a","rank":0,"incarnation":7,"state":"up:reconnect","state_seq":18,"addr":"172.21.15.42:6835/2019200199","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2019200199},{"type":"v1","addr":"172.21.15.42:6835","nonce":2019200199}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:04:45.148 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:45.144+0000 7fd51fa93700 1 -- 172.21.15.42:0/2307186180 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd50003ad90 msgr2=0x7fd50003d2a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:04:45.148 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:45.144+0000 7fd51fa93700 1 --2- 172.21.15.42:0/2307186180 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd50003ad90 0x7fd50003d2a0 crc :-1 s=READY pgs=121 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:04:45.148 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:45.144+0000 7fd51fa93700 1 -- 172.21.15.42:0/2307186180 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd5181189c0 msgr2=0x7fd518071ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:04:45.149 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:45.144+0000 7fd51fa93700 1 --2- 172.21.15.42:0/2307186180 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd5181189c0 0x7fd518071ad0 secure :-1 s=READY pgs=241 cs=0 l=1 rx=0x7fd508000f10 tx=0x7fd5100109f0).stop 2019-12-05T05:04:45.149 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:45.144+0000 7fd51fa93700 1 -- 172.21.15.42:0/2307186180 shutdown_connections 2019-12-05T05:04:45.149 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:45.144+0000 7fd51fa93700 1 --2- 172.21.15.42:0/2307186180 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd50003ad90 0x7fd50003d2a0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:04:45.149 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:45.144+0000 7fd51fa93700 1 --2- 172.21.15.42:0/2307186180 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd5181189c0 0x7fd518071ad0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:04:45.149 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:45.144+0000 7fd51fa93700 1 --2- 172.21.15.42:0/2307186180 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd518119190 0x7fd518118470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:04:45.149 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:45.144+0000 7fd51fa93700 1 -- 172.21.15.42:0/2307186180 shutdown_connections 2019-12-05T05:04:45.149 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:45.144+0000 7fd51fa93700 1 -- 172.21.15.42:0/2307186180 wait complete. 2019-12-05T05:04:45.150 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:45.144+0000 7fd51fa93700 1 -- 172.21.15.42:0/2307186180 >> 172.21.15.42:0/2307186180 conn(0x7fd5180aabe0 msgr2=0x7fd51808eb70 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:04:45.155 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 8, "feature_flags": { "enable_multiple": false, "ever_enabled_multiple": false }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2019-12-05T05:03:35.604738+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 8, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_4354": { "addr": "172.21.15.42:6835/2019200199", "addrs": { "addrvec": [ { "addr": "172.21.15.42:6834", "nonce": 2019200199, "type": "v2" }, { "addr": "172.21.15.42:6835", "nonce": 2019200199, "type": "v1" } ] }, "export_targets": [], "features": 4540138292836696063, "flags": 0, "gid": 4354, "incarnation": 7, "name": "a", "rank": 0, "state": "up:reconnect", "state_seq": 18 } }, "last_failure": 0, "last_failure_osd_epoch": 25, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 2, "min_compat_client": "0 (unknown)", "modified": "2019-12-05T05:04:44.177728+0000", "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4354 } } } ], "standbys": [] } 2019-12-05T05:04:47.156 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:04:47.252 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:47.236+0000 7f0443b7e700 1 Processor -- start 2019-12-05T05:04:47.253 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:47.236+0000 7f0443b7e700 1 -- start start 2019-12-05T05:04:47.253 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:47.236+0000 7f0443b7e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f043c0a7740 0x7f043c08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:04:47.253 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:47.236+0000 7f0443b7e700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f043c0a0710 con 0x7f043c07e720 2019-12-05T05:04:47.253 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:47.236+0000 7f0443b7e700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f043c0a9ba0 con 0x7f043c09fae0 2019-12-05T05:04:47.253 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:47.236+0000 7f0443b7e700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f043c0a0b00 con 0x7f043c0a7740 2019-12-05T05:04:47.254 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:47.236+0000 7f044211b700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f043c0a7740 0x7f043c08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:04:47.254 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:47.236+0000 7f044291c700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f043c09fae0 0x7f043c14efe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:54292/0 (socket says 172.21.15.42:54292) 2019-12-05T05:04:47.254 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:47.236+0000 7f044211b700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f043c0a7740 0x7f043c08df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:54250/0 (socket says 172.21.15.42:54250) 2019-12-05T05:04:47.254 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:47.236+0000 7f044291c700 1 -- 172.21.15.42:0/3979422449 learned_addr learned my addr 172.21.15.42:0/3979422449 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:04:47.254 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:47.236+0000 7f044211b700 1 -- 172.21.15.42:0/3979422449 >> v1:172.21.15.42:6789/0 conn(0x7f043c09fae0 legacy=0x7f043c14efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:04:47.255 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:47.236+0000 7f044211b700 1 -- 172.21.15.42:0/3979422449 >> v1:172.21.15.40:6789/0 conn(0x7f043c07e720 legacy=0x7f043c09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:04:47.255 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:47.236+0000 7f044211b700 1 -- 172.21.15.42:0/3979422449 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f043c07eb50 con 0x7f043c0a7740 2019-12-05T05:04:47.255 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:47.236+0000 7f044211b700 1 --2- 172.21.15.42:0/3979422449 >> v2:172.21.15.42:3300/0 conn(0x7f043c0a7740 0x7f043c08df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f042c008790 tx=0x7f042c00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:47.255 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:47.236+0000 7f044211b700 1 --2- 172.21.15.42:0/3979422449 >> v2:172.21.15.42:3300/0 conn(0x7f043c0a7740 0x7f043c08df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f042c008790 tx=0x7f042c00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:47.255 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:47.236+0000 7f044211b700 1 --2- 172.21.15.42:0/3979422449 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f043c0a7740 0x7f043c08df80 secure :-1 s=READY pgs=243 cs=0 l=1 rx=0x7f042c008790 tx=0x7f042c00fbf0).ready entity=mon.1 client_cookie=60012354556aa20 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:04:47.255 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:47.240+0000 7f044211b700 1 --2- 172.21.15.42:0/3979422449 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f043c0a7740 0x7f043c08df80 secure :-1 s=THROTTLE_DONE pgs=243 cs=0 l=1 rx=0x7f042c008790 tx=0x7f042c00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:47.256 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:47.240+0000 7f0441119700 1 -- 172.21.15.42:0/3979422449 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f042c011270 con 0x7f043c0a7740 2019-12-05T05:04:47.256 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:47.240+0000 7f044211b700 1 --2- 172.21.15.42:0/3979422449 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f043c0a7740 0x7f043c08df80 secure :-1 s=THROTTLE_DONE pgs=243 cs=0 l=1 rx=0x7f042c008790 tx=0x7f042c00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:47.256 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:47.240+0000 7f044211b700 1 --2- 172.21.15.42:0/3979422449 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f043c0a7740 0x7f043c08df80 secure :-1 s=THROTTLE_DONE pgs=243 cs=0 l=1 rx=0x7f042c008790 tx=0x7f042c00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:47.256 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:47.240+0000 7f0441119700 1 -- 172.21.15.42:0/3979422449 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f042c008bc0 con 0x7f043c0a7740 2019-12-05T05:04:47.256 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:47.240+0000 7f0441119700 1 -- 172.21.15.42:0/3979422449 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f042c011850 con 0x7f043c0a7740 2019-12-05T05:04:47.256 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:47.240+0000 7f0443b7e700 1 -- 172.21.15.42:0/3979422449 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f043c0a7740 msgr2=0x7f043c08df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:04:47.256 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:47.240+0000 7f0443b7e700 1 --2- 172.21.15.42:0/3979422449 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f043c0a7740 0x7f043c08df80 secure :-1 s=READY pgs=243 cs=0 l=1 rx=0x7f042c008790 tx=0x7f042c00fbf0).stop 2019-12-05T05:04:47.257 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:47.240+0000 7f0443b7e700 1 -- 172.21.15.42:0/3979422449 shutdown_connections 2019-12-05T05:04:47.257 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:47.240+0000 7f0443b7e700 1 --2- 172.21.15.42:0/3979422449 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f043c0a7740 0x7f043c08df80 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:04:47.257 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:47.240+0000 7f0443b7e700 1 -- 172.21.15.42:0/3979422449 shutdown_connections 2019-12-05T05:04:47.257 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:47.240+0000 7f0443b7e700 1 -- 172.21.15.42:0/3979422449 wait complete. 2019-12-05T05:04:47.257 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:47.240+0000 7f0443b7e700 1 -- 172.21.15.42:0/3979422449 >> 172.21.15.42:0/3979422449 conn(0x7f043c0aabe0 msgr2=0x7f043c0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:04:47.257 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:47.240+0000 7f0443b7e700 1 Processor -- start 2019-12-05T05:04:47.257 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:47.240+0000 7f0443b7e700 1 -- start start 2019-12-05T05:04:47.258 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:47.240+0000 7f0443b7e700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f043c0a1510 0x7f043c0a1940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:04:47.258 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:47.240+0000 7f0443b7e700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f043c0a4f80 con 0x7f043c118a50 2019-12-05T05:04:47.258 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:47.240+0000 7f0443b7e700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f043c065980 con 0x7f043c064010 2019-12-05T05:04:47.258 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:47.240+0000 7f0443b7e700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f043c063d70 con 0x7f043c0a1510 2019-12-05T05:04:47.258 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:47.240+0000 7f044211b700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f043c064010 0x7f043c065180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:54294/0 (socket says 172.21.15.42:54294) 2019-12-05T05:04:47.258 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:47.240+0000 7f044211b700 1 -- 172.21.15.42:0/1368179069 learned_addr learned my addr 172.21.15.42:0/1368179069 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:04:47.258 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:47.240+0000 7f044291c700 1 --2- 172.21.15.42:0/1368179069 >> v2:172.21.15.40:3300/0 conn(0x7f043c0a1510 0x7f043c0a1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:04:47.258 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:47.240+0000 7f043affd700 1 -- 172.21.15.42:0/1368179069 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1687162308 0 0) 0x7f043c065980 con 0x7f043c064010 2019-12-05T05:04:47.259 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:47.240+0000 7f043affd700 1 -- 172.21.15.42:0/1368179069 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f041c0030b0 con 0x7f043c064010 2019-12-05T05:04:47.259 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:47.240+0000 7f043affd700 1 -- 172.21.15.42:0/1368179069 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 3130400511 0 0) 0x7f041c0030b0 con 0x7f043c064010 2019-12-05T05:04:47.259 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:47.240+0000 7f043affd700 1 -- 172.21.15.42:0/1368179069 --> v1:172.21.15.42:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f043c065980 con 0x7f043c064010 2019-12-05T05:04:47.259 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:47.240+0000 7f043affd700 1 -- 172.21.15.42:0/1368179069 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7f042c011270 con 0x7f043c064010 2019-12-05T05:04:47.259 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:47.244+0000 7f043affd700 1 -- 172.21.15.42:0/1368179069 >> v2:172.21.15.40:3300/0 conn(0x7f043c0a1510 msgr2=0x7f043c0a1940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:04:47.259 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:47.244+0000 7f043affd700 1 --2- 172.21.15.42:0/1368179069 >> v2:172.21.15.40:3300/0 conn(0x7f043c0a1510 0x7f043c0a1940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:04:47.259 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:47.244+0000 7f043affd700 1 -- 172.21.15.42:0/1368179069 >> v1:172.21.15.42:6789/0 conn(0x7f043c064010 legacy=0x7f043c065180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:04:47.260 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:47.244+0000 7f043affd700 1 -- 172.21.15.42:0/1368179069 >> v1:172.21.15.40:6789/0 conn(0x7f043c118a50 legacy=0x7f043c0a3670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:04:47.260 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:47.244+0000 7f043affd700 1 --2- 172.21.15.42:0/1368179069 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f041c0038a0 0x7f041c003d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:04:47.260 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:47.244+0000 7f043affd700 1 --2- 172.21.15.42:0/1368179069 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f041c004420 0x7f041c004850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:04:47.260 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:47.244+0000 7f043affd700 1 --2- 172.21.15.42:0/1368179069 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f041c004ec0 0x7f041c00b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:04:47.260 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:47.244+0000 7f043affd700 1 -- 172.21.15.42:0/1368179069 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f043c063d70 con 0x7f041c0038a0 2019-12-05T05:04:47.260 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:47.244+0000 7f043affd700 1 -- 172.21.15.42:0/1368179069 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f041c00bcc0 con 0x7f041c004ec0 2019-12-05T05:04:47.260 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:47.244+0000 7f043affd700 1 -- 172.21.15.42:0/1368179069 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f041c00bf00 con 0x7f041c004420 2019-12-05T05:04:47.261 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:47.244+0000 7f044291c700 1 --2- 172.21.15.42:0/1368179069 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f041c004ec0 0x7f041c00b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:04:47.261 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:47.244+0000 7f044211b700 1 --2- 172.21.15.42:0/1368179069 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f041c0038a0 0x7f041c003d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:04:47.261 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:47.244+0000 7f044191a700 1 --2- 172.21.15.42:0/1368179069 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f041c004420 0x7f041c004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:04:47.261 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:47.244+0000 7f044291c700 1 -- 172.21.15.42:0/1368179069 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f041c004420 msgr2=0x7f041c004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:04:47.262 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:47.244+0000 7f044291c700 1 --2- 172.21.15.42:0/1368179069 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f041c004420 0x7f041c004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:04:47.262 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:47.244+0000 7f044291c700 1 -- 172.21.15.42:0/1368179069 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f041c0038a0 msgr2=0x7f041c003d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:04:47.262 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:47.244+0000 7f044291c700 1 --2- 172.21.15.42:0/1368179069 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f041c0038a0 0x7f041c003d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:04:47.262 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:47.244+0000 7f044291c700 1 -- 172.21.15.42:0/1368179069 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f043c0a3e70 con 0x7f041c004ec0 2019-12-05T05:04:47.262 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:47.244+0000 7f044291c700 1 --2- 172.21.15.42:0/1368179069 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f041c004ec0 0x7f041c00b670 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f0434004720 tx=0x7f0434010f90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:47.263 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:47.244+0000 7f0443b7e700 1 -- 172.21.15.42:0/1368179069 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f043c0a1e90 con 0x7f041c004ec0 2019-12-05T05:04:47.263 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:47.244+0000 7f044291c700 1 -- 172.21.15.42:0/1368179069 reap_dead start 2019-12-05T05:04:47.263 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:47.244+0000 7f044291c700 1 --2- 172.21.15.42:0/1368179069 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f041c004ec0 0x7f041c00b670 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f0434004720 tx=0x7f0434010f90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:47.263 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:47.244+0000 7f0443b7e700 1 -- 172.21.15.42:0/1368179069 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f043c064010 con 0x7f041c004ec0 2019-12-05T05:04:47.263 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:47.244+0000 7f044291c700 1 --2- 172.21.15.42:0/1368179069 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f041c004ec0 0x7f041c00b670 secure :-1 s=READY pgs=245 cs=0 l=1 rx=0x7f0434004720 tx=0x7f0434010f90).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:04:47.263 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:47.244+0000 7f044291c700 1 --2- 172.21.15.42:0/1368179069 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f041c004ec0 0x7f041c00b670 secure :-1 s=THROTTLE_DONE pgs=245 cs=0 l=1 rx=0x7f0434004720 tx=0x7f0434010f90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:47.263 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:47.244+0000 7f043affd700 1 -- 172.21.15.42:0/1368179069 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f04340047b0 con 0x7f041c004ec0 2019-12-05T05:04:47.264 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:47.244+0000 7f044291c700 1 --2- 172.21.15.42:0/1368179069 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f041c004ec0 0x7f041c00b670 secure :-1 s=THROTTLE_DONE pgs=245 cs=0 l=1 rx=0x7f0434004720 tx=0x7f0434010f90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:47.264 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:47.244+0000 7f044291c700 1 --2- 172.21.15.42:0/1368179069 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f041c004ec0 0x7f041c00b670 secure :-1 s=THROTTLE_DONE pgs=245 cs=0 l=1 rx=0x7f0434004720 tx=0x7f0434010f90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:47.264 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:47.244+0000 7f043affd700 1 -- 172.21.15.42:0/1368179069 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f043400cb30 con 0x7f041c004ec0 2019-12-05T05:04:47.264 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:47.244+0000 7f043affd700 1 -- 172.21.15.42:0/1368179069 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0434004db0 con 0x7f041c004ec0 2019-12-05T05:04:47.264 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:47.244+0000 7f044291c700 1 --2- 172.21.15.42:0/1368179069 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f041c004ec0 0x7f041c00b670 secure :-1 s=THROTTLE_DONE pgs=245 cs=0 l=1 rx=0x7f0434004720 tx=0x7f0434010f90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:47.264 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:47.244+0000 7f043affd700 1 -- 172.21.15.42:0/1368179069 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f0434006180 con 0x7f041c004ec0 2019-12-05T05:04:47.264 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:47.244+0000 7f044291c700 1 --2- 172.21.15.42:0/1368179069 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f041c004ec0 0x7f041c00b670 secure :-1 s=THROTTLE_DONE pgs=245 cs=0 l=1 rx=0x7f0434004720 tx=0x7f0434010f90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:47.265 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:47.244+0000 7f043affd700 1 --2- 172.21.15.42:0/1368179069 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f041c03e990 0x7f041c040ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:04:47.265 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:47.244+0000 7f043affd700 1 -- 172.21.15.42:0/1368179069 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6117+0+0 (secure 0 0 0) 0x7f043404ab50 con 0x7f041c004ec0 2019-12-05T05:04:47.265 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:47.244+0000 7f044211b700 1 --2- 172.21.15.42:0/1368179069 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f041c03e990 0x7f041c040ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:04:47.265 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:47.244+0000 7f044211b700 1 --2- 172.21.15.42:0/1368179069 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f041c03e990 0x7f041c040ea0 crc :-1 s=READY pgs=122 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:04:47.266 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:47.252+0000 7f0443b7e700 1 -- 172.21.15.42:0/1368179069 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0418003200 con 0x7f041c004ec0 2019-12-05T05:04:47.266 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:47.260+0000 7f044291c700 1 --2- 172.21.15.42:0/1368179069 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f041c004ec0 0x7f041c00b670 secure :-1 s=THROTTLE_DONE pgs=245 cs=0 l=1 rx=0x7f0434004720 tx=0x7f0434010f90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:47.266 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:47.260+0000 7f043affd700 1 -- 172.21.15.42:0/1368179069 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f0434031e30 con 0x7f041c004ec0 2019-12-05T05:04:47.417 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:47.412+0000 7f0443b7e700 1 -- 172.21.15.42:0/1368179069 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0418003630 con 0x7f041c004ec0 2019-12-05T05:04:47.418 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:47.412+0000 7f044291c700 1 --2- 172.21.15.42:0/1368179069 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f041c004ec0 0x7f041c00b670 secure :-1 s=THROTTLE_DONE pgs=245 cs=0 l=1 rx=0x7f0434004720 tx=0x7f0434010f90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:47.418 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:47.412+0000 7f043affd700 1 -- 172.21.15.42:0/1368179069 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+1771 (secure 0 0 0) 0x7f04340329e0 con 0x7f041c004ec0 2019-12-05T05:04:47.418 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 10 2019-12-05T05:04:47.418 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:04:47.419 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":10,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:04:46.187363+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4354},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4354":{"gid":4354,"name":"a","rank":0,"incarnation":7,"state":"up:active","state_seq":20,"addr":"172.21.15.42:6835/2019200199","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2019200199},{"type":"v1","addr":"172.21.15.42:6835","nonce":2019200199}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:04:47.422 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:47.416+0000 7f0443b7e700 1 -- 172.21.15.42:0/1368179069 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f041c03e990 msgr2=0x7f041c040ea0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:04:47.422 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:47.416+0000 7f0443b7e700 1 --2- 172.21.15.42:0/1368179069 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f041c03e990 0x7f041c040ea0 crc :-1 s=READY pgs=122 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:04:47.422 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:47.416+0000 7f0443b7e700 1 -- 172.21.15.42:0/1368179069 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f041c004ec0 msgr2=0x7f041c00b670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:04:47.422 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:47.416+0000 7f0443b7e700 1 --2- 172.21.15.42:0/1368179069 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f041c004ec0 0x7f041c00b670 secure :-1 s=READY pgs=245 cs=0 l=1 rx=0x7f0434004720 tx=0x7f0434010f90).stop 2019-12-05T05:04:47.422 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:47.416+0000 7f0443b7e700 1 -- 172.21.15.42:0/1368179069 shutdown_connections 2019-12-05T05:04:47.422 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:47.416+0000 7f0443b7e700 1 --2- 172.21.15.42:0/1368179069 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f041c03e990 0x7f041c040ea0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:04:47.423 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:47.416+0000 7f0443b7e700 1 --2- 172.21.15.42:0/1368179069 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f041c004ec0 0x7f041c00b670 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:04:47.423 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:47.416+0000 7f0443b7e700 1 -- 172.21.15.42:0/1368179069 shutdown_connections 2019-12-05T05:04:47.423 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:47.416+0000 7f0443b7e700 1 -- 172.21.15.42:0/1368179069 wait complete. 2019-12-05T05:04:47.423 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:47.416+0000 7f0443b7e700 1 -- 172.21.15.42:0/1368179069 >> 172.21.15.42:0/1368179069 conn(0x7f043c07ee20 msgr2=0x7f043c059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:04:47.432 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a has gained rank=0, replacing gid=4341 2019-12-05T05:04:47.433 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 5 secs before reviving mds.b 2019-12-05T05:04:52.433 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.b 2019-12-05T05:04:52.434 INFO:tasks.ceph.mds.b:Restarting daemon 2019-12-05T05:04:52.434 INFO:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2019-12-05T05:04:52.437 INFO:tasks.ceph.mds.b:Started 2019-12-05T05:04:52.437 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:04:52.497 INFO:tasks.ceph.mds.b.smithi040.stdout:starting mds.b at 2019-12-05T05:04:52.522 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:52.512+0000 7faa3ce8f700 1 Processor -- start 2019-12-05T05:04:52.522 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:52.512+0000 7faa3ce8f700 1 -- start start 2019-12-05T05:04:52.522 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:52.512+0000 7faa3ce8f700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7faa3808df70 0x7faa3809f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:04:52.522 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:52.512+0000 7faa3ce8f700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa3809fc50 con 0x7faa380a7730 2019-12-05T05:04:52.522 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:52.512+0000 7faa3ce8f700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa380a0700 con 0x7faa3809f820 2019-12-05T05:04:52.523 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:52.512+0000 7faa3ce8f700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa380a0af0 con 0x7faa3808df70 2019-12-05T05:04:52.523 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:52.512+0000 7faa3759e700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7faa3809f820 0x7faa3814efd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:54328/0 (socket says 172.21.15.42:54328) 2019-12-05T05:04:52.523 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:52.512+0000 7faa3759e700 1 -- 172.21.15.42:0/1624121783 learned_addr learned my addr 172.21.15.42:0/1624121783 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:04:52.523 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:52.512+0000 7faa2e59c700 1 --2- 172.21.15.42:0/1624121783 >> v2:172.21.15.40:3300/0 conn(0x7faa3808df70 0x7faa3809f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:04:52.523 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:52.512+0000 7faa3659c700 1 -- 172.21.15.42:0/1624121783 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1071669632 0 0) 0x7faa380a0700 con 0x7faa3809f820 2019-12-05T05:04:52.523 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:52.512+0000 7faa3659c700 1 -- 172.21.15.42:0/1624121783 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa240030b0 con 0x7faa3809f820 2019-12-05T05:04:52.523 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:52.512+0000 7faa3659c700 1 -- 172.21.15.42:0/1624121783 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1340937790 0 0) 0x7faa3809fc50 con 0x7faa380a7730 2019-12-05T05:04:52.524 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:52.512+0000 7faa3659c700 1 -- 172.21.15.42:0/1624121783 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa380a0700 con 0x7faa380a7730 2019-12-05T05:04:52.524 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:52.512+0000 7faa3659c700 1 -- 172.21.15.42:0/1624121783 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2542151400 0 0) 0x7faa240030b0 con 0x7faa3809f820 2019-12-05T05:04:52.524 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:52.512+0000 7faa3659c700 1 -- 172.21.15.42:0/1624121783 >> v2:172.21.15.40:3300/0 conn(0x7faa3808df70 msgr2=0x7faa3809f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:04:52.524 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:52.512+0000 7faa3659c700 1 --2- 172.21.15.42:0/1624121783 >> v2:172.21.15.40:3300/0 conn(0x7faa3808df70 0x7faa3809f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:04:52.524 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:52.512+0000 7faa3659c700 1 -- 172.21.15.42:0/1624121783 >> v1:172.21.15.40:6789/0 conn(0x7faa380a7730 legacy=0x7faa3807e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:04:52.524 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:52.512+0000 7faa3659c700 1 -- 172.21.15.42:0/1624121783 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa3807ef10 con 0x7faa3809f820 2019-12-05T05:04:52.524 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:52.512+0000 7faa3659c700 1 -- 172.21.15.42:0/1624121783 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7faa30001ad0 con 0x7faa3809f820 2019-12-05T05:04:52.525 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:52.512+0000 7faa3659c700 1 -- 172.21.15.42:0/1624121783 >> v1:172.21.15.42:6789/0 conn(0x7faa3809f820 legacy=0x7faa3814efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:04:52.525 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:52.512+0000 7faa3659c700 1 --2- 172.21.15.42:0/1624121783 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7faa24003910 0x7faa24003d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:04:52.525 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:52.512+0000 7faa3659c700 1 --2- 172.21.15.42:0/1624121783 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faa24004420 0x7faa240048f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:04:52.525 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:52.512+0000 7faa3659c700 1 --2- 172.21.15.42:0/1624121783 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faa2404d670 0x7faa2404fab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:04:52.525 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:52.512+0000 7faa3659c700 1 -- 172.21.15.42:0/1624121783 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa380a0af0 con 0x7faa24004420 2019-12-05T05:04:52.525 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:52.512+0000 7faa3659c700 1 -- 172.21.15.42:0/1624121783 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa24050000 con 0x7faa2404d670 2019-12-05T05:04:52.526 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:52.512+0000 7faa3659c700 1 -- 172.21.15.42:0/1624121783 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa240501c0 con 0x7faa24003910 2019-12-05T05:04:52.526 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:52.512+0000 7faa3759e700 1 --2- 172.21.15.42:0/1624121783 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faa2404d670 0x7faa2404fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:04:52.526 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:52.512+0000 7faa36d9d700 1 --2- 172.21.15.42:0/1624121783 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7faa24003910 0x7faa24003d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:04:52.526 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:52.512+0000 7faa2e59c700 1 --2- 172.21.15.42:0/1624121783 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faa24004420 0x7faa240048f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:04:52.528 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:52.512+0000 7faa3759e700 1 -- 172.21.15.42:0/1624121783 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7faa24003910 msgr2=0x7faa24003d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:04:52.528 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:52.512+0000 7faa3759e700 1 --2- 172.21.15.42:0/1624121783 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7faa24003910 0x7faa24003d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:04:52.528 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:52.512+0000 7faa3759e700 1 -- 172.21.15.42:0/1624121783 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faa24004420 msgr2=0x7faa240048f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:04:52.528 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:52.512+0000 7faa3759e700 1 --2- 172.21.15.42:0/1624121783 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faa24004420 0x7faa240048f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:04:52.528 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:52.512+0000 7faa3759e700 1 -- 172.21.15.42:0/1624121783 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa24050430 con 0x7faa2404d670 2019-12-05T05:04:52.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:52.512+0000 7faa3759e700 1 --2- 172.21.15.42:0/1624121783 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faa2404d670 0x7faa2404fab0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7faa240053f0 tx=0x7faa3000c4e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:52.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:52.512+0000 7faa3759e700 1 -- 172.21.15.42:0/1624121783 reap_dead start 2019-12-05T05:04:52.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:52.512+0000 7faa3759e700 1 --2- 172.21.15.42:0/1624121783 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faa2404d670 0x7faa2404fab0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7faa240053f0 tx=0x7faa3000c4e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:52.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:52.512+0000 7faa3759e700 1 --2- 172.21.15.42:0/1624121783 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faa2404d670 0x7faa2404fab0 secure :-1 s=READY pgs=255 cs=0 l=1 rx=0x7faa240053f0 tx=0x7faa3000c4e0).ready entity=mon.1 client_cookie=ef0880cdf2626cdf server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:04:52.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:52.512+0000 7faa3759e700 1 --2- 172.21.15.42:0/1624121783 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faa2404d670 0x7faa2404fab0 secure :-1 s=THROTTLE_DONE pgs=255 cs=0 l=1 rx=0x7faa240053f0 tx=0x7faa3000c4e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:52.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:52.512+0000 7faa3659c700 1 -- 172.21.15.42:0/1624121783 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faa30005bc0 con 0x7faa2404d670 2019-12-05T05:04:52.530 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:52.512+0000 7faa3759e700 1 --2- 172.21.15.42:0/1624121783 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faa2404d670 0x7faa2404fab0 secure :-1 s=THROTTLE_DONE pgs=255 cs=0 l=1 rx=0x7faa240053f0 tx=0x7faa3000c4e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:52.530 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:52.512+0000 7faa3759e700 1 --2- 172.21.15.42:0/1624121783 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faa2404d670 0x7faa2404fab0 secure :-1 s=THROTTLE_DONE pgs=255 cs=0 l=1 rx=0x7faa240053f0 tx=0x7faa3000c4e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:52.530 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:52.512+0000 7faa3659c700 1 -- 172.21.15.42:0/1624121783 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa300057e0 con 0x7faa2404d670 2019-12-05T05:04:52.530 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:52.512+0000 7faa3659c700 1 -- 172.21.15.42:0/1624121783 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faa30004fa0 con 0x7faa2404d670 2019-12-05T05:04:52.530 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:52.512+0000 7faa3ce8f700 1 -- 172.21.15.42:0/1624121783 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faa2404d670 msgr2=0x7faa2404fab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:04:52.530 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:52.512+0000 7faa3ce8f700 1 --2- 172.21.15.42:0/1624121783 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faa2404d670 0x7faa2404fab0 secure :-1 s=READY pgs=255 cs=0 l=1 rx=0x7faa240053f0 tx=0x7faa3000c4e0).stop 2019-12-05T05:04:52.530 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:52.516+0000 7faa3ce8f700 1 -- 172.21.15.42:0/1624121783 shutdown_connections 2019-12-05T05:04:52.531 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:52.516+0000 7faa3ce8f700 1 --2- 172.21.15.42:0/1624121783 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faa2404d670 0x7faa2404fab0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:04:52.531 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:52.516+0000 7faa3ce8f700 1 -- 172.21.15.42:0/1624121783 shutdown_connections 2019-12-05T05:04:52.531 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:52.516+0000 7faa3ce8f700 1 -- 172.21.15.42:0/1624121783 wait complete. 2019-12-05T05:04:52.531 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:52.516+0000 7faa3ce8f700 1 -- 172.21.15.42:0/1624121783 >> 172.21.15.42:0/1624121783 conn(0x7faa380aabd0 msgr2=0x7faa380a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:04:52.531 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:52.516+0000 7faa3ce8f700 1 Processor -- start 2019-12-05T05:04:52.531 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:52.516+0000 7faa3ce8f700 1 -- start start 2019-12-05T05:04:52.536 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:52.516+0000 7faa3ce8f700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7faa38141840 0x7faa38141c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:04:52.536 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:52.516+0000 7faa3ce8f700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7faa38076500 0x7faa38078940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:04:52.536 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:52.516+0000 7faa3ce8f700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa38126a50 con 0x7faa380a3a50 2019-12-05T05:04:52.536 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:52.516+0000 7faa3ce8f700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa380a4830 con 0x7faa38076500 2019-12-05T05:04:52.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:52.516+0000 7faa3ce8f700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa3811b3f0 con 0x7faa38141840 2019-12-05T05:04:52.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:52.516+0000 7faa3759e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7faa38076500 0x7faa38078940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:04:52.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:52.516+0000 7faa3759e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7faa38076500 0x7faa38078940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:54302/0 (socket says 172.21.15.42:54302) 2019-12-05T05:04:52.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:52.516+0000 7faa3759e700 1 -- 172.21.15.42:0/3899891077 learned_addr learned my addr 172.21.15.42:0/3899891077 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:04:52.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:52.516+0000 7faa2e59c700 1 --2- 172.21.15.42:0/3899891077 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7faa38141840 0x7faa38141c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:04:52.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:52.516+0000 7faa3759e700 1 -- 172.21.15.42:0/3899891077 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7faa38141840 msgr2=0x7faa38141c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:04:52.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:52.516+0000 7faa3759e700 1 --2- 172.21.15.42:0/3899891077 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7faa38141840 0x7faa38141c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:04:52.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:52.516+0000 7faa3759e700 1 -- 172.21.15.42:0/3899891077 >> v1:172.21.15.40:6789/0 conn(0x7faa380a3a50 legacy=0x7faa38134420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:04:52.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:52.516+0000 7faa3759e700 1 -- 172.21.15.42:0/3899891077 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa3811bec0 con 0x7faa38076500 2019-12-05T05:04:52.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:52.516+0000 7faa3759e700 1 --2- 172.21.15.42:0/3899891077 >> v2:172.21.15.42:3300/0 conn(0x7faa38076500 0x7faa38078940 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7faa3811b5b0 tx=0x7faa3000b3b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:52.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:52.520+0000 7faa3759e700 1 --2- 172.21.15.42:0/3899891077 >> v2:172.21.15.42:3300/0 conn(0x7faa38076500 0x7faa38078940 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7faa3811b5b0 tx=0x7faa3000b3b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:52.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:52.520+0000 7faa3759e700 1 --2- 172.21.15.42:0/3899891077 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faa38076500 0x7faa38078940 secure :-1 s=READY pgs=256 cs=0 l=1 rx=0x7faa3811b5b0 tx=0x7faa3000b3b0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:04:52.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:52.520+0000 7faa3759e700 1 --2- 172.21.15.42:0/3899891077 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faa38076500 0x7faa38078940 secure :-1 s=THROTTLE_DONE pgs=256 cs=0 l=1 rx=0x7faa3811b5b0 tx=0x7faa3000b3b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:52.541 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:52.520+0000 7faa3759e700 1 --2- 172.21.15.42:0/3899891077 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faa38076500 0x7faa38078940 secure :-1 s=THROTTLE_DONE pgs=256 cs=0 l=1 rx=0x7faa3811b5b0 tx=0x7faa3000b3b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:52.541 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:52.520+0000 7faa2ffff700 1 -- 172.21.15.42:0/3899891077 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faa30010350 con 0x7faa38076500 2019-12-05T05:04:52.541 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:52.520+0000 7faa3759e700 1 --2- 172.21.15.42:0/3899891077 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faa38076500 0x7faa38078940 secure :-1 s=THROTTLE_DONE pgs=256 cs=0 l=1 rx=0x7faa3811b5b0 tx=0x7faa3000b3b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:52.541 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:52.520+0000 7faa2ffff700 1 -- 172.21.15.42:0/3899891077 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa30004ba0 con 0x7faa38076500 2019-12-05T05:04:52.541 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:52.520+0000 7faa3ce8f700 1 -- 172.21.15.42:0/3899891077 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa3811b930 con 0x7faa38076500 2019-12-05T05:04:52.541 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:52.520+0000 7faa2ffff700 1 -- 172.21.15.42:0/3899891077 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faa30010930 con 0x7faa38076500 2019-12-05T05:04:52.541 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:52.520+0000 7faa3ce8f700 1 -- 172.21.15.42:0/3899891077 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa3811afd0 con 0x7faa38076500 2019-12-05T05:04:52.542 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:52.520+0000 7faa3759e700 1 --2- 172.21.15.42:0/3899891077 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faa38076500 0x7faa38078940 secure :-1 s=THROTTLE_DONE pgs=256 cs=0 l=1 rx=0x7faa3811b5b0 tx=0x7faa3000b3b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:52.542 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:52.520+0000 7faa2ffff700 1 -- 172.21.15.42:0/3899891077 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7faa30010b30 con 0x7faa38076500 2019-12-05T05:04:52.542 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:52.520+0000 7faa3759e700 1 --2- 172.21.15.42:0/3899891077 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faa38076500 0x7faa38078940 secure :-1 s=THROTTLE_DONE pgs=256 cs=0 l=1 rx=0x7faa3811b5b0 tx=0x7faa3000b3b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:52.542 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:52.520+0000 7faa2ffff700 1 --2- 172.21.15.42:0/3899891077 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7faa1c03f130 0x7faa1c041640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:04:52.542 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:52.520+0000 7faa2ffff700 1 -- 172.21.15.42:0/3899891077 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6117+0+0 (secure 0 0 0) 0x7faa3004b960 con 0x7faa38076500 2019-12-05T05:04:52.542 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:52.520+0000 7faa36d9d700 1 --2- 172.21.15.42:0/3899891077 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7faa1c03f130 0x7faa1c041640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:04:52.542 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:52.520+0000 7faa36d9d700 1 --2- 172.21.15.42:0/3899891077 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7faa1c03f130 0x7faa1c041640 crc :-1 s=READY pgs=123 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:04:52.542 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:52.528+0000 7faa3ce8f700 1 -- 172.21.15.42:0/3899891077 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa3805d100 con 0x7faa38076500 2019-12-05T05:04:52.543 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:52.532+0000 7faa3759e700 1 --2- 172.21.15.42:0/3899891077 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faa38076500 0x7faa38078940 secure :-1 s=THROTTLE_DONE pgs=256 cs=0 l=1 rx=0x7faa3811b5b0 tx=0x7faa3000b3b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:52.543 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:52.532+0000 7faa2ffff700 1 -- 172.21.15.42:0/3899891077 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7faa300329e0 con 0x7faa38076500 2019-12-05T05:04:52.743 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:52.736+0000 7faa3ce8f700 1 -- 172.21.15.42:0/3899891077 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faa3805ae20 con 0x7faa38076500 2019-12-05T05:04:52.743 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:52.740+0000 7faa3759e700 1 --2- 172.21.15.42:0/3899891077 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faa38076500 0x7faa38078940 secure :-1 s=THROTTLE_DONE pgs=256 cs=0 l=1 rx=0x7faa3811b5b0 tx=0x7faa3000b3b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:52.743 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:52.740+0000 7faa2ffff700 1 -- 172.21.15.42:0/3899891077 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+1771 (secure 0 0 0) 0x7faa30032c40 con 0x7faa38076500 2019-12-05T05:04:52.744 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 10 2019-12-05T05:04:52.744 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:04:52.744 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":10,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:04:46.187363+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4354},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4354":{"gid":4354,"name":"a","rank":0,"incarnation":7,"state":"up:active","state_seq":20,"addr":"172.21.15.42:6835/2019200199","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2019200199},{"type":"v1","addr":"172.21.15.42:6835","nonce":2019200199}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:04:52.746 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:52.740+0000 7faa3ce8f700 1 -- 172.21.15.42:0/3899891077 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7faa1c03f130 msgr2=0x7faa1c041640 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:04:52.747 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:52.740+0000 7faa3ce8f700 1 --2- 172.21.15.42:0/3899891077 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7faa1c03f130 0x7faa1c041640 crc :-1 s=READY pgs=123 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:04:52.747 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:52.740+0000 7faa3ce8f700 1 -- 172.21.15.42:0/3899891077 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faa38076500 msgr2=0x7faa38078940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:04:52.747 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:52.740+0000 7faa3ce8f700 1 --2- 172.21.15.42:0/3899891077 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faa38076500 0x7faa38078940 secure :-1 s=READY pgs=256 cs=0 l=1 rx=0x7faa3811b5b0 tx=0x7faa3000b3b0).stop 2019-12-05T05:04:52.747 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:52.740+0000 7faa3ce8f700 1 -- 172.21.15.42:0/3899891077 shutdown_connections 2019-12-05T05:04:52.748 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:52.740+0000 7faa3ce8f700 1 --2- 172.21.15.42:0/3899891077 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7faa1c03f130 0x7faa1c041640 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:04:52.748 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:52.740+0000 7faa3ce8f700 1 --2- 172.21.15.42:0/3899891077 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faa38076500 0x7faa38078940 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:04:52.748 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:52.740+0000 7faa3ce8f700 1 --2- 172.21.15.42:0/3899891077 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7faa38141840 0x7faa38141c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:04:52.748 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:52.740+0000 7faa3ce8f700 1 -- 172.21.15.42:0/3899891077 shutdown_connections 2019-12-05T05:04:52.748 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:52.740+0000 7faa3ce8f700 1 -- 172.21.15.42:0/3899891077 wait complete. 2019-12-05T05:04:52.748 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:52.740+0000 7faa3ce8f700 1 -- 172.21.15.42:0/3899891077 >> 172.21.15.42:0/3899891077 conn(0x7faa380a0f60 msgr2=0x7faa3811c270 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:04:52.755 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2019-12-05T05:04:54.756 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:04:54.852 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:54.840+0000 7f954809e700 1 Processor -- start 2019-12-05T05:04:54.852 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:54.840+0000 7f954809e700 1 -- start start 2019-12-05T05:04:54.852 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:54.840+0000 7f954809e700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f954009fae0 0x7f954014fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:04:54.852 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:54.840+0000 7f954809e700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95400a0710 con 0x7f95400a7740 2019-12-05T05:04:54.853 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:54.840+0000 7f954809e700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f954007ef20 con 0x7f954008df80 2019-12-05T05:04:54.853 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:54.840+0000 7f954809e700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95400a0b00 con 0x7f954009fae0 2019-12-05T05:04:54.853 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:54.840+0000 7f9545e3a700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f954008df80 0x7f954009f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:54346/0 (socket says 172.21.15.42:54346) 2019-12-05T05:04:54.853 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:54.840+0000 7f9545e3a700 1 -- 172.21.15.42:0/1535856511 learned_addr learned my addr 172.21.15.42:0/1535856511 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:04:54.853 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:54.840+0000 7f9546e3c700 1 --2- 172.21.15.42:0/1535856511 >> v2:172.21.15.40:3300/0 conn(0x7f954009fae0 0x7f954014fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:04:54.853 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:54.840+0000 7f9545639700 1 -- 172.21.15.42:0/1535856511 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3704036961 0 0) 0x7f954007ef20 con 0x7f954008df80 2019-12-05T05:04:54.854 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:54.840+0000 7f9545639700 1 -- 172.21.15.42:0/1535856511 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f952c0030b0 con 0x7f954008df80 2019-12-05T05:04:54.854 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:54.840+0000 7f9545639700 1 -- 172.21.15.42:0/1535856511 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 350261320 0 0) 0x7f952c0030b0 con 0x7f954008df80 2019-12-05T05:04:54.854 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:54.840+0000 7f9545639700 1 -- 172.21.15.42:0/1535856511 >> v2:172.21.15.40:3300/0 conn(0x7f954009fae0 msgr2=0x7f954014fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:04:54.854 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:54.840+0000 7f9545639700 1 --2- 172.21.15.42:0/1535856511 >> v2:172.21.15.40:3300/0 conn(0x7f954009fae0 0x7f954014fff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:04:54.854 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:54.840+0000 7f9545639700 1 -- 172.21.15.42:0/1535856511 >> v1:172.21.15.40:6789/0 conn(0x7f95400a7740 legacy=0x7f954007e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:04:54.854 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:54.840+0000 7f9545639700 1 -- 172.21.15.42:0/1535856511 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95400a9ba0 con 0x7f954008df80 2019-12-05T05:04:54.854 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:54.840+0000 7f9545639700 1 -- 172.21.15.42:0/1535856511 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7f9530001750 con 0x7f954008df80 2019-12-05T05:04:54.855 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:54.840+0000 7f9546e3c700 1 --2- 172.21.15.42:0/1535856511 >> v2:172.21.15.40:3300/0 conn(0x7f954009fae0 0x7f954014fff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:04:54.855 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:54.840+0000 7f9545639700 1 -- 172.21.15.42:0/1535856511 >> v1:172.21.15.42:6789/0 conn(0x7f954008df80 legacy=0x7f954009f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:04:54.855 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:54.840+0000 7f9545639700 1 --2- 172.21.15.42:0/1535856511 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f952c0036e0 0x7f952c003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:04:54.855 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:54.840+0000 7f9545639700 1 --2- 172.21.15.42:0/1535856511 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f952c004420 0x7f952c004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:04:54.855 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:54.840+0000 7f9545639700 1 --2- 172.21.15.42:0/1535856511 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f952c004da0 0x7f952c00b670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:04:54.855 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:54.840+0000 7f9545639700 1 -- 172.21.15.42:0/1535856511 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95400a0b00 con 0x7f952c004420 2019-12-05T05:04:54.855 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:54.840+0000 7f9545639700 1 -- 172.21.15.42:0/1535856511 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f952c04fc60 con 0x7f952c0036e0 2019-12-05T05:04:54.855 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:54.840+0000 7f954663b700 1 --2- 172.21.15.42:0/1535856511 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f952c0036e0 0x7f952c003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:04:54.856 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:54.840+0000 7f9545639700 1 -- 172.21.15.42:0/1535856511 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f952c04fea0 con 0x7f952c004da0 2019-12-05T05:04:54.856 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:54.840+0000 7f9545e3a700 1 --2- 172.21.15.42:0/1535856511 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f952c004420 0x7f952c004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:04:54.856 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:54.840+0000 7f9546e3c700 1 --2- 172.21.15.42:0/1535856511 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f952c004da0 0x7f952c00b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:04:54.856 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:54.840+0000 7f954663b700 1 -- 172.21.15.42:0/1535856511 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f952c004da0 msgr2=0x7f952c00b670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:04:54.858 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:54.840+0000 7f954663b700 1 --2- 172.21.15.42:0/1535856511 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f952c004da0 0x7f952c00b670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:04:54.858 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:54.840+0000 7f954663b700 1 -- 172.21.15.42:0/1535856511 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f952c004420 msgr2=0x7f952c004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:04:54.858 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:54.840+0000 7f954663b700 1 --2- 172.21.15.42:0/1535856511 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f952c004420 0x7f952c004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:04:54.858 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:54.840+0000 7f954663b700 1 -- 172.21.15.42:0/1535856511 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f952c0501a0 con 0x7f952c0036e0 2019-12-05T05:04:54.858 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:54.840+0000 7f9546e3c700 1 --2- 172.21.15.42:0/1535856511 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f952c004da0 0x7f952c00b670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T05:04:54.858 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:54.840+0000 7f9545e3a700 1 --2- 172.21.15.42:0/1535856511 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f952c004420 0x7f952c004850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T05:04:54.859 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:54.840+0000 7f9546e3c700 1 -- 172.21.15.42:0/1535856511 reap_dead start 2019-12-05T05:04:54.862 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:54.840+0000 7f954663b700 1 --2- 172.21.15.42:0/1535856511 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f952c0036e0 0x7f952c003b50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f953c001000 tx=0x7f953c00fcd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:54.862 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:54.840+0000 7f954663b700 1 --2- 172.21.15.42:0/1535856511 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f952c0036e0 0x7f952c003b50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f953c001000 tx=0x7f953c00fcd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:54.862 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:54.840+0000 7f954663b700 1 --2- 172.21.15.42:0/1535856511 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f952c0036e0 0x7f952c003b50 secure :-1 s=READY pgs=258 cs=0 l=1 rx=0x7f953c001000 tx=0x7f953c00fcd0).ready entity=mon.1 client_cookie=f94a5595194bcdea server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:04:54.862 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:54.840+0000 7f954663b700 1 --2- 172.21.15.42:0/1535856511 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f952c0036e0 0x7f952c003b50 secure :-1 s=THROTTLE_DONE pgs=258 cs=0 l=1 rx=0x7f953c001000 tx=0x7f953c00fcd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:54.863 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:54.840+0000 7f9545639700 1 -- 172.21.15.42:0/1535856511 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f953c003540 con 0x7f952c0036e0 2019-12-05T05:04:54.863 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:54.840+0000 7f954663b700 1 --2- 172.21.15.42:0/1535856511 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f952c0036e0 0x7f952c003b50 secure :-1 s=THROTTLE_DONE pgs=258 cs=0 l=1 rx=0x7f953c001000 tx=0x7f953c00fcd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:54.863 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:54.840+0000 7f954663b700 1 --2- 172.21.15.42:0/1535856511 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f952c0036e0 0x7f952c003b50 secure :-1 s=THROTTLE_DONE pgs=258 cs=0 l=1 rx=0x7f953c001000 tx=0x7f953c00fcd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:54.863 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:54.840+0000 7f9545639700 1 -- 172.21.15.42:0/1535856511 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f953c003160 con 0x7f952c0036e0 2019-12-05T05:04:54.863 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:54.840+0000 7f9545639700 1 -- 172.21.15.42:0/1535856511 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f953c003b20 con 0x7f952c0036e0 2019-12-05T05:04:54.863 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:54.840+0000 7f954809e700 1 -- 172.21.15.42:0/1535856511 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f952c0036e0 msgr2=0x7f952c003b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:04:54.863 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:54.840+0000 7f954809e700 1 --2- 172.21.15.42:0/1535856511 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f952c0036e0 0x7f952c003b50 secure :-1 s=READY pgs=258 cs=0 l=1 rx=0x7f953c001000 tx=0x7f953c00fcd0).stop 2019-12-05T05:04:54.864 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:54.840+0000 7f954809e700 1 -- 172.21.15.42:0/1535856511 shutdown_connections 2019-12-05T05:04:54.864 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:54.840+0000 7f954809e700 1 --2- 172.21.15.42:0/1535856511 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f952c0036e0 0x7f952c003b50 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:04:54.864 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:54.840+0000 7f954809e700 1 -- 172.21.15.42:0/1535856511 shutdown_connections 2019-12-05T05:04:54.864 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:54.840+0000 7f954809e700 1 -- 172.21.15.42:0/1535856511 wait complete. 2019-12-05T05:04:54.864 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:54.840+0000 7f954809e700 1 -- 172.21.15.42:0/1535856511 >> 172.21.15.42:0/1535856511 conn(0x7f95400aabe0 msgr2=0x7f95400a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:04:54.864 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:54.844+0000 7f954809e700 1 Processor -- start 2019-12-05T05:04:54.865 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:54.844+0000 7f954809e700 1 -- start start 2019-12-05T05:04:54.865 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:54.844+0000 7f954809e700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f95400a6ea0 0x7f9540077b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:04:54.865 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:54.844+0000 7f954809e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f95400a40c0 0x7f9540078060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:04:54.865 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:54.844+0000 7f954809e700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f9540143aa0 0x7f9540143ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:04:54.865 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:54.844+0000 7f954809e700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9540144eb0 con 0x7f9540143aa0 2019-12-05T05:04:54.865 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:54.844+0000 7f954809e700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9540145150 con 0x7f95400a40c0 2019-12-05T05:04:54.865 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:54.844+0000 7f954809e700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9540144420 con 0x7f95400a6ea0 2019-12-05T05:04:54.866 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:54.844+0000 7f9545e3a700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f95400a40c0 0x7f9540078060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:04:54.866 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:54.844+0000 7f9545e3a700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f95400a40c0 0x7f9540078060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:54320/0 (socket says 172.21.15.42:54320) 2019-12-05T05:04:54.866 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:54.844+0000 7f9545e3a700 1 -- 172.21.15.42:0/3364083127 learned_addr learned my addr 172.21.15.42:0/3364083127 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:04:54.866 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:54.844+0000 7f954663b700 1 --2- 172.21.15.42:0/3364083127 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f95400a6ea0 0x7f9540077b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:04:54.866 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:54.844+0000 7f9546e3c700 1 --2- 172.21.15.42:0/3364083127 >> v2:172.21.15.40:3300/0 conn(0x7f9540143aa0 0x7f9540143ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:04:54.866 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:54.844+0000 7f9545e3a700 1 -- 172.21.15.42:0/3364083127 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f95400a6ea0 msgr2=0x7f9540077b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:04:54.866 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:54.844+0000 7f9545e3a700 1 --2- 172.21.15.42:0/3364083127 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f95400a6ea0 0x7f9540077b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:04:54.867 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:54.844+0000 7f9545e3a700 1 -- 172.21.15.42:0/3364083127 >> v2:172.21.15.40:3300/0 conn(0x7f9540143aa0 msgr2=0x7f9540143ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:04:54.867 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:54.844+0000 7f9545e3a700 1 --2- 172.21.15.42:0/3364083127 >> v2:172.21.15.40:3300/0 conn(0x7f9540143aa0 0x7f9540143ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:04:54.867 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:54.844+0000 7f9545e3a700 1 -- 172.21.15.42:0/3364083127 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9540144b70 con 0x7f95400a40c0 2019-12-05T05:04:54.867 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:54.844+0000 7f9546e3c700 1 --2- 172.21.15.42:0/3364083127 >> v2:172.21.15.40:3300/0 conn(0x7f9540143aa0 0x7f9540143ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).send_auth_request state changed! 2019-12-05T05:04:54.867 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:54.844+0000 7f9545e3a700 1 --2- 172.21.15.42:0/3364083127 >> v2:172.21.15.42:3300/0 conn(0x7f95400a40c0 0x7f9540078060 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f9530001a10 tx=0x7f9530010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:54.867 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:54.844+0000 7f9545e3a700 1 --2- 172.21.15.42:0/3364083127 >> v2:172.21.15.42:3300/0 conn(0x7f95400a40c0 0x7f9540078060 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f9530001a10 tx=0x7f9530010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:54.867 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:54.844+0000 7f9545e3a700 1 --2- 172.21.15.42:0/3364083127 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f95400a40c0 0x7f9540078060 secure :-1 s=READY pgs=259 cs=0 l=1 rx=0x7f9530001a10 tx=0x7f9530010b50).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:04:54.868 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:54.844+0000 7f9545e3a700 1 --2- 172.21.15.42:0/3364083127 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f95400a40c0 0x7f9540078060 secure :-1 s=THROTTLE_DONE pgs=259 cs=0 l=1 rx=0x7f9530001a10 tx=0x7f9530010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:54.868 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:54.844+0000 7f9536ffd700 1 -- 172.21.15.42:0/3364083127 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f953000cd60 con 0x7f95400a40c0 2019-12-05T05:04:54.868 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:54.844+0000 7f9545e3a700 1 --2- 172.21.15.42:0/3364083127 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f95400a40c0 0x7f9540078060 secure :-1 s=THROTTLE_DONE pgs=259 cs=0 l=1 rx=0x7f9530001a10 tx=0x7f9530010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:54.868 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:54.844+0000 7f954809e700 1 -- 172.21.15.42:0/3364083127 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95401453f0 con 0x7f95400a40c0 2019-12-05T05:04:54.868 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:54.844+0000 7f9545e3a700 1 --2- 172.21.15.42:0/3364083127 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f95400a40c0 0x7f9540078060 secure :-1 s=THROTTLE_DONE pgs=259 cs=0 l=1 rx=0x7f9530001a10 tx=0x7f9530010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:54.868 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:54.844+0000 7f9536ffd700 1 -- 172.21.15.42:0/3364083127 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f953000c980 con 0x7f95400a40c0 2019-12-05T05:04:54.868 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:54.844+0000 7f9536ffd700 1 -- 172.21.15.42:0/3364083127 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9530008950 con 0x7f95400a40c0 2019-12-05T05:04:54.869 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:54.844+0000 7f954809e700 1 -- 172.21.15.42:0/3364083127 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f954012d4c0 con 0x7f95400a40c0 2019-12-05T05:04:54.869 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:54.848+0000 7f9545e3a700 1 --2- 172.21.15.42:0/3364083127 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f95400a40c0 0x7f9540078060 secure :-1 s=THROTTLE_DONE pgs=259 cs=0 l=1 rx=0x7f9530001a10 tx=0x7f9530010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:54.870 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:54.848+0000 7f9536ffd700 1 -- 172.21.15.42:0/3364083127 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f9530008c60 con 0x7f95400a40c0 2019-12-05T05:04:54.870 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:54.848+0000 7f9545e3a700 1 --2- 172.21.15.42:0/3364083127 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f95400a40c0 0x7f9540078060 secure :-1 s=THROTTLE_DONE pgs=259 cs=0 l=1 rx=0x7f9530001a10 tx=0x7f9530010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:54.870 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:54.848+0000 7f9536ffd700 1 --2- 172.21.15.42:0/3364083127 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f952803ceb0 0x7f952803f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:04:54.870 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:54.848+0000 7f9536ffd700 1 -- 172.21.15.42:0/3364083127 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6117+0+0 (secure 0 0 0) 0x7f953004acc0 con 0x7f95400a40c0 2019-12-05T05:04:54.870 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:54.848+0000 7f954663b700 1 --2- 172.21.15.42:0/3364083127 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f952803ceb0 0x7f952803f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:04:54.871 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:54.848+0000 7f954663b700 1 --2- 172.21.15.42:0/3364083127 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f952803ceb0 0x7f952803f3c0 crc :-1 s=READY pgs=124 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:04:54.872 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:54.856+0000 7f954809e700 1 -- 172.21.15.42:0/3364083127 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f954005d110 con 0x7f95400a40c0 2019-12-05T05:04:54.872 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:54.860+0000 7f9545e3a700 1 --2- 172.21.15.42:0/3364083127 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f95400a40c0 0x7f9540078060 secure :-1 s=THROTTLE_DONE pgs=259 cs=0 l=1 rx=0x7f9530001a10 tx=0x7f9530010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:54.872 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:54.860+0000 7f9536ffd700 1 -- 172.21.15.42:0/3364083127 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f9530031d40 con 0x7f95400a40c0 2019-12-05T05:04:55.020 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:55.016+0000 7f954809e700 1 -- 172.21.15.42:0/3364083127 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f95400a3bf0 con 0x7f95400a40c0 2019-12-05T05:04:55.021 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:55.016+0000 7f9545e3a700 1 --2- 172.21.15.42:0/3364083127 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f95400a40c0 0x7f9540078060 secure :-1 s=THROTTLE_DONE pgs=259 cs=0 l=1 rx=0x7f9530001a10 tx=0x7f9530010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:55.021 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:55.016+0000 7f9536ffd700 1 -- 172.21.15.42:0/3364083127 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+1771 (secure 0 0 0) 0x7f9530031fa0 con 0x7f95400a40c0 2019-12-05T05:04:55.021 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 10 2019-12-05T05:04:55.021 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:04:55.022 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":10,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:04:46.187363+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4354},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4354":{"gid":4354,"name":"a","rank":0,"incarnation":7,"state":"up:active","state_seq":20,"addr":"172.21.15.42:6835/2019200199","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2019200199},{"type":"v1","addr":"172.21.15.42:6835","nonce":2019200199}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:04:55.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:55.020+0000 7f954809e700 1 -- 172.21.15.42:0/3364083127 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f952803ceb0 msgr2=0x7f952803f3c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:04:55.024 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:55.020+0000 7f954809e700 1 --2- 172.21.15.42:0/3364083127 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f952803ceb0 0x7f952803f3c0 crc :-1 s=READY pgs=124 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:04:55.024 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:55.020+0000 7f954809e700 1 -- 172.21.15.42:0/3364083127 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f95400a40c0 msgr2=0x7f9540078060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:04:55.024 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:55.020+0000 7f954809e700 1 --2- 172.21.15.42:0/3364083127 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f95400a40c0 0x7f9540078060 secure :-1 s=READY pgs=259 cs=0 l=1 rx=0x7f9530001a10 tx=0x7f9530010b50).stop 2019-12-05T05:04:55.024 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:55.020+0000 7f954809e700 1 -- 172.21.15.42:0/3364083127 shutdown_connections 2019-12-05T05:04:55.024 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:55.020+0000 7f954809e700 1 --2- 172.21.15.42:0/3364083127 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f952803ceb0 0x7f952803f3c0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:04:55.024 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:55.020+0000 7f954809e700 1 --2- 172.21.15.42:0/3364083127 >> v2:172.21.15.40:3300/0 conn(0x7f9540143aa0 0x7f9540143ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:04:55.025 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:55.020+0000 7f954809e700 1 --2- 172.21.15.42:0/3364083127 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f95400a6ea0 0x7f9540077b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:04:55.025 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:55.020+0000 7f954809e700 1 --2- 172.21.15.42:0/3364083127 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f95400a40c0 0x7f9540078060 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:04:55.025 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:55.020+0000 7f954809e700 1 -- 172.21.15.42:0/3364083127 shutdown_connections 2019-12-05T05:04:55.025 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:55.020+0000 7f954809e700 1 -- 172.21.15.42:0/3364083127 wait complete. 2019-12-05T05:04:55.025 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:55.020+0000 7f954809e700 1 -- 172.21.15.42:0/3364083127 >> 172.21.15.42:0/3364083127 conn(0x7f95400a7740 msgr2=0x7f954007bce0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:04:55.033 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2019-12-05T05:04:57.034 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:04:57.136 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:57.124+0000 7fd955ea3700 1 Processor -- start 2019-12-05T05:04:57.137 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:57.124+0000 7fd955ea3700 1 -- start start 2019-12-05T05:04:57.137 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:57.128+0000 7fd955ea3700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd9500a7740 0x7fd95008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:04:57.137 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:57.128+0000 7fd955ea3700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9500a0710 con 0x7fd95007e720 2019-12-05T05:04:57.137 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:57.128+0000 7fd955ea3700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9500a9ba0 con 0x7fd95009fae0 2019-12-05T05:04:57.137 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:57.128+0000 7fd955ea3700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9500a0b00 con 0x7fd9500a7740 2019-12-05T05:04:57.137 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:57.128+0000 7fd954c41700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fd95009fae0 0x7fd95014efe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:54372/0 (socket says 172.21.15.42:54372) 2019-12-05T05:04:57.138 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:57.128+0000 7fd954c41700 1 -- 172.21.15.42:0/3085351350 learned_addr learned my addr 172.21.15.42:0/3085351350 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:04:57.138 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:57.128+0000 7fd94ffff700 1 --2- 172.21.15.42:0/3085351350 >> v2:172.21.15.42:3300/0 conn(0x7fd9500a7740 0x7fd95008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:04:57.138 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:57.128+0000 7fd94effd700 1 -- 172.21.15.42:0/3085351350 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3426211174 0 0) 0x7fd9500a9ba0 con 0x7fd95009fae0 2019-12-05T05:04:57.138 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:57.128+0000 7fd94effd700 1 -- 172.21.15.42:0/3085351350 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9380030b0 con 0x7fd95009fae0 2019-12-05T05:04:57.138 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:57.128+0000 7fd94ffff700 1 -- 172.21.15.42:0/3085351350 >> v1:172.21.15.42:6789/0 conn(0x7fd95009fae0 legacy=0x7fd95014efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:04:57.138 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:57.128+0000 7fd94ffff700 1 -- 172.21.15.42:0/3085351350 >> v1:172.21.15.40:6789/0 conn(0x7fd95007e720 legacy=0x7fd95009f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:04:57.138 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:57.128+0000 7fd94ffff700 1 -- 172.21.15.42:0/3085351350 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd95007eb50 con 0x7fd9500a7740 2019-12-05T05:04:57.139 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:57.128+0000 7fd94ffff700 1 --2- 172.21.15.42:0/3085351350 >> v2:172.21.15.42:3300/0 conn(0x7fd9500a7740 0x7fd95008df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fd940008790 tx=0x7fd94000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:57.139 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:57.128+0000 7fd94ffff700 1 --2- 172.21.15.42:0/3085351350 >> v2:172.21.15.42:3300/0 conn(0x7fd9500a7740 0x7fd95008df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fd940008790 tx=0x7fd94000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:57.139 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:57.128+0000 7fd94ffff700 1 --2- 172.21.15.42:0/3085351350 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd9500a7740 0x7fd95008df80 secure :-1 s=READY pgs=261 cs=0 l=1 rx=0x7fd940008790 tx=0x7fd94000fbf0).ready entity=mon.1 client_cookie=a7bfc7dfa0b9de5d server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:04:57.139 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:57.128+0000 7fd94ffff700 1 --2- 172.21.15.42:0/3085351350 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd9500a7740 0x7fd95008df80 secure :-1 s=THROTTLE_DONE pgs=261 cs=0 l=1 rx=0x7fd940008790 tx=0x7fd94000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:57.139 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:57.128+0000 7fd94ffff700 1 --2- 172.21.15.42:0/3085351350 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd9500a7740 0x7fd95008df80 secure :-1 s=THROTTLE_DONE pgs=261 cs=0 l=1 rx=0x7fd940008790 tx=0x7fd94000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:57.139 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:57.128+0000 7fd94effd700 1 -- 172.21.15.42:0/3085351350 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd940011270 con 0x7fd9500a7740 2019-12-05T05:04:57.139 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:57.128+0000 7fd94ffff700 1 --2- 172.21.15.42:0/3085351350 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd9500a7740 0x7fd95008df80 secure :-1 s=THROTTLE_DONE pgs=261 cs=0 l=1 rx=0x7fd940008790 tx=0x7fd94000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:57.140 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:57.128+0000 7fd94effd700 1 -- 172.21.15.42:0/3085351350 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd940008bc0 con 0x7fd9500a7740 2019-12-05T05:04:57.140 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:57.128+0000 7fd94effd700 1 -- 172.21.15.42:0/3085351350 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd940011850 con 0x7fd9500a7740 2019-12-05T05:04:57.140 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:57.128+0000 7fd955ea3700 1 -- 172.21.15.42:0/3085351350 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd9500a7740 msgr2=0x7fd95008df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:04:57.140 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:57.128+0000 7fd955ea3700 1 --2- 172.21.15.42:0/3085351350 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd9500a7740 0x7fd95008df80 secure :-1 s=READY pgs=261 cs=0 l=1 rx=0x7fd940008790 tx=0x7fd94000fbf0).stop 2019-12-05T05:04:57.140 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:57.132+0000 7fd955ea3700 1 -- 172.21.15.42:0/3085351350 shutdown_connections 2019-12-05T05:04:57.140 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:57.132+0000 7fd955ea3700 1 --2- 172.21.15.42:0/3085351350 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd9500a7740 0x7fd95008df80 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:04:57.141 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:57.132+0000 7fd955ea3700 1 -- 172.21.15.42:0/3085351350 shutdown_connections 2019-12-05T05:04:57.141 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:57.132+0000 7fd955ea3700 1 -- 172.21.15.42:0/3085351350 wait complete. 2019-12-05T05:04:57.142 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:57.132+0000 7fd955ea3700 1 -- 172.21.15.42:0/3085351350 >> 172.21.15.42:0/3085351350 conn(0x7fd9500aabe0 msgr2=0x7fd9500a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:04:57.147 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:57.132+0000 7fd955ea3700 1 Processor -- start 2019-12-05T05:04:57.147 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:57.132+0000 7fd955ea3700 1 -- start start 2019-12-05T05:04:57.147 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:57.136+0000 7fd955ea3700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd950113320 0x7fd950119640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:04:57.147 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:57.136+0000 7fd955ea3700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd95007df40 0x7fd950118670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:04:57.147 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:57.136+0000 7fd955ea3700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd950119b90 con 0x7fd950144fe0 2019-12-05T05:04:57.147 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:57.136+0000 7fd955ea3700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd950119ee0 con 0x7fd95007df40 2019-12-05T05:04:57.148 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:57.136+0000 7fd955ea3700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd95011a180 con 0x7fd950113320 2019-12-05T05:04:57.148 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:57.136+0000 7fd94ffff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd950113320 0x7fd950119640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:04:57.148 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:57.136+0000 7fd94ffff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd950113320 0x7fd950119640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:54336/0 (socket says 172.21.15.42:54336) 2019-12-05T05:04:57.148 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:57.136+0000 7fd94ffff700 1 -- 172.21.15.42:0/1888087902 learned_addr learned my addr 172.21.15.42:0/1888087902 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:04:57.148 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:57.136+0000 7fd954c41700 1 --2- 172.21.15.42:0/1888087902 >> v2:172.21.15.40:3300/0 conn(0x7fd95007df40 0x7fd950118670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:04:57.148 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:57.136+0000 7fd94ffff700 1 -- 172.21.15.42:0/1888087902 >> v2:172.21.15.40:3300/0 conn(0x7fd95007df40 msgr2=0x7fd950118670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:04:57.149 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:57.136+0000 7fd94ffff700 1 --2- 172.21.15.42:0/1888087902 >> v2:172.21.15.40:3300/0 conn(0x7fd95007df40 0x7fd950118670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:04:57.149 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:57.136+0000 7fd94ffff700 1 -- 172.21.15.42:0/1888087902 >> v1:172.21.15.40:6789/0 conn(0x7fd950144fe0 legacy=0x7fd95007d740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:04:57.149 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:57.136+0000 7fd94ffff700 1 -- 172.21.15.42:0/1888087902 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd950118bc0 con 0x7fd950113320 2019-12-05T05:04:57.149 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:57.136+0000 7fd94ffff700 1 --2- 172.21.15.42:0/1888087902 >> v2:172.21.15.42:3300/0 conn(0x7fd950113320 0x7fd950119640 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fd940008790 tx=0x7fd940008e70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:57.149 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:57.136+0000 7fd94ffff700 1 --2- 172.21.15.42:0/1888087902 >> v2:172.21.15.42:3300/0 conn(0x7fd950113320 0x7fd950119640 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fd940008790 tx=0x7fd940008e70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:57.149 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:57.136+0000 7fd94ffff700 1 --2- 172.21.15.42:0/1888087902 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd950113320 0x7fd950119640 secure :-1 s=READY pgs=262 cs=0 l=1 rx=0x7fd940008790 tx=0x7fd940008e70).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:04:57.149 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:57.136+0000 7fd94ffff700 1 --2- 172.21.15.42:0/1888087902 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd950113320 0x7fd950119640 secure :-1 s=THROTTLE_DONE pgs=262 cs=0 l=1 rx=0x7fd940008790 tx=0x7fd940008e70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:57.150 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:57.136+0000 7fd94ffff700 1 --2- 172.21.15.42:0/1888087902 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd950113320 0x7fd950119640 secure :-1 s=THROTTLE_DONE pgs=262 cs=0 l=1 rx=0x7fd940008790 tx=0x7fd940008e70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:57.150 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:57.136+0000 7fd94cff9700 1 -- 172.21.15.42:0/1888087902 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd940007340 con 0x7fd950113320 2019-12-05T05:04:57.150 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:57.136+0000 7fd94ffff700 1 --2- 172.21.15.42:0/1888087902 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd950113320 0x7fd950119640 secure :-1 s=THROTTLE_DONE pgs=262 cs=0 l=1 rx=0x7fd940008790 tx=0x7fd940008e70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:57.150 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:57.136+0000 7fd94cff9700 1 -- 172.21.15.42:0/1888087902 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd940011470 con 0x7fd950113320 2019-12-05T05:04:57.150 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:57.136+0000 7fd94cff9700 1 -- 172.21.15.42:0/1888087902 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd940005540 con 0x7fd950113320 2019-12-05T05:04:57.150 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:57.136+0000 7fd955ea3700 1 -- 172.21.15.42:0/1888087902 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd950118e90 con 0x7fd950113320 2019-12-05T05:04:57.150 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:57.136+0000 7fd955ea3700 1 -- 172.21.15.42:0/1888087902 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd95012db60 con 0x7fd950113320 2019-12-05T05:04:57.151 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:57.136+0000 7fd94ffff700 1 --2- 172.21.15.42:0/1888087902 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd950113320 0x7fd950119640 secure :-1 s=THROTTLE_DONE pgs=262 cs=0 l=1 rx=0x7fd940008790 tx=0x7fd940008e70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:57.151 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:57.140+0000 7fd94cff9700 1 -- 172.21.15.42:0/1888087902 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fd940005850 con 0x7fd950113320 2019-12-05T05:04:57.151 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:57.140+0000 7fd94ffff700 1 --2- 172.21.15.42:0/1888087902 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd950113320 0x7fd950119640 secure :-1 s=THROTTLE_DONE pgs=262 cs=0 l=1 rx=0x7fd940008790 tx=0x7fd940008e70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:57.151 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:57.140+0000 7fd94cff9700 1 --2- 172.21.15.42:0/1888087902 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd93003ada0 0x7fd93003d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:04:57.151 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:57.140+0000 7fd94cff9700 1 -- 172.21.15.42:0/1888087902 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6117+0+0 (secure 0 0 0) 0x7fd94004bb80 con 0x7fd950113320 2019-12-05T05:04:57.151 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:57.140+0000 7fd94f7fe700 1 --2- 172.21.15.42:0/1888087902 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd93003ada0 0x7fd93003d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:04:57.151 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:57.140+0000 7fd94f7fe700 1 --2- 172.21.15.42:0/1888087902 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd93003ada0 0x7fd93003d2b0 crc :-1 s=READY pgs=125 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:04:57.152 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:57.140+0000 7fd955ea3700 1 -- 172.21.15.42:0/1888087902 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd95005d110 con 0x7fd950113320 2019-12-05T05:04:57.153 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:57.148+0000 7fd94ffff700 1 --2- 172.21.15.42:0/1888087902 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd950113320 0x7fd950119640 secure :-1 s=THROTTLE_DONE pgs=262 cs=0 l=1 rx=0x7fd940008790 tx=0x7fd940008e70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:57.153 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:57.148+0000 7fd94cff9700 1 -- 172.21.15.42:0/1888087902 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fd940030840 con 0x7fd950113320 2019-12-05T05:04:57.323 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:57.320+0000 7fd955ea3700 1 -- 172.21.15.42:0/1888087902 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd95005b8c0 con 0x7fd950113320 2019-12-05T05:04:57.324 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:57.320+0000 7fd94ffff700 1 --2- 172.21.15.42:0/1888087902 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd950113320 0x7fd950119640 secure :-1 s=THROTTLE_DONE pgs=262 cs=0 l=1 rx=0x7fd940008790 tx=0x7fd940008e70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:04:57.324 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:57.320+0000 7fd94cff9700 1 -- 172.21.15.42:0/1888087902 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) v1 ==== 94+0+2101 (secure 0 0 0) 0x7fd940014d50 con 0x7fd950113320 2019-12-05T05:04:57.325 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 11 2019-12-05T05:04:57.325 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:04:57.325 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":11,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4831,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/90248709","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":90248709},{"type":"v1","addr":"172.21.15.40:6833","nonce":90248709}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":11}],"filesystems":[{"mdsmap":{"epoch":10,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:04:46.187363+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4354},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4354":{"gid":4354,"name":"a","rank":0,"incarnation":7,"state":"up:active","state_seq":20,"addr":"172.21.15.42:6835/2019200199","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2019200199},{"type":"v1","addr":"172.21.15.42:6835","nonce":2019200199}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:04:57.329 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:57.320+0000 7fd955ea3700 1 -- 172.21.15.42:0/1888087902 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd93003ada0 msgr2=0x7fd93003d2b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:04:57.329 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:57.320+0000 7fd955ea3700 1 --2- 172.21.15.42:0/1888087902 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd93003ada0 0x7fd93003d2b0 crc :-1 s=READY pgs=125 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:04:57.329 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:57.320+0000 7fd955ea3700 1 -- 172.21.15.42:0/1888087902 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd950113320 msgr2=0x7fd950119640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:04:57.329 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:57.320+0000 7fd955ea3700 1 --2- 172.21.15.42:0/1888087902 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd950113320 0x7fd950119640 secure :-1 s=READY pgs=262 cs=0 l=1 rx=0x7fd940008790 tx=0x7fd940008e70).stop 2019-12-05T05:04:57.330 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:57.320+0000 7fd955ea3700 1 -- 172.21.15.42:0/1888087902 shutdown_connections 2019-12-05T05:04:57.330 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:57.320+0000 7fd955ea3700 1 --2- 172.21.15.42:0/1888087902 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd93003ada0 0x7fd93003d2b0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:04:57.330 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:57.320+0000 7fd955ea3700 1 --2- 172.21.15.42:0/1888087902 >> v2:172.21.15.40:3300/0 conn(0x7fd95007df40 0x7fd950118670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:04:57.330 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:57.320+0000 7fd955ea3700 1 --2- 172.21.15.42:0/1888087902 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd950113320 0x7fd950119640 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:04:57.331 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:57.320+0000 7fd955ea3700 1 -- 172.21.15.42:0/1888087902 shutdown_connections 2019-12-05T05:04:57.331 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:57.320+0000 7fd955ea3700 1 -- 172.21.15.42:0/1888087902 wait complete. 2019-12-05T05:04:57.331 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:04:57.320+0000 7fd955ea3700 1 -- 172.21.15.42:0/1888087902 >> 172.21.15.42:0/1888087902 conn(0x7fd9500aabe0 msgr2=0x7fd9500a9640 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:04:57.341 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b reported in up:standby state 2019-12-05T05:04:57.341 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 117 secs before thrashing 2019-12-05T05:04:58.799 INFO:tasks.workunit.client.0.smithi042.stderr:Note: checking out 'c5290805033483ba0cec165c06bde07dae5f19ec'. 2019-12-05T05:04:58.799 INFO:tasks.workunit.client.0.smithi042.stderr: 2019-12-05T05:04:58.799 INFO:tasks.workunit.client.0.smithi042.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2019-12-05T05:04:58.800 INFO:tasks.workunit.client.0.smithi042.stderr:changes and commit them, and you can discard any commits you make in this 2019-12-05T05:04:58.800 INFO:tasks.workunit.client.0.smithi042.stderr:state without impacting any branches by performing another checkout. 2019-12-05T05:04:58.800 INFO:tasks.workunit.client.0.smithi042.stderr: 2019-12-05T05:04:58.800 INFO:tasks.workunit.client.0.smithi042.stderr:If you want to create a new branch to retain commits you create, you may 2019-12-05T05:04:58.800 INFO:tasks.workunit.client.0.smithi042.stderr:do so (now or later) by using -b with the checkout command again. Example: 2019-12-05T05:04:58.800 INFO:tasks.workunit.client.0.smithi042.stderr: 2019-12-05T05:04:58.801 INFO:tasks.workunit.client.0.smithi042.stderr: git checkout -b 2019-12-05T05:04:58.801 INFO:tasks.workunit.client.0.smithi042.stderr: 2019-12-05T05:04:58.801 INFO:tasks.workunit.client.0.smithi042.stderr:HEAD is now at c529080503 Merge PR #27866 into wip-pdonnell-testing-20191204.231200 2019-12-05T05:04:58.804 INFO:teuthology.orchestra.run.smithi042:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2019-12-05T05:04:58.813 INFO:tasks.workunit.client.0.smithi042.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2019-12-05T05:04:58.815 INFO:tasks.workunit.client.0.smithi042.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2019-12-05T05:04:58.816 INFO:tasks.workunit.client.0.smithi042.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2019-12-05T05:04:59.579 INFO:tasks.workunit.client.0.smithi042.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2019-12-05T05:04:59.722 INFO:tasks.workunit.client.0.smithi042.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2019-12-05T05:04:59.778 INFO:tasks.workunit.client.0.smithi042.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2019-12-05T05:04:59.780 INFO:tasks.workunit.client.0.smithi042.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2019-12-05T05:04:59.780 INFO:tasks.workunit.client.0.smithi042.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2019-12-05T05:04:59.830 INFO:tasks.workunit.client.0.smithi042.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2019-12-05T05:04:59.910 DEBUG:teuthology.orchestra.remote:smithi042:/home/ubuntu/cephtest/workunits.list.client.0 is 4KB 2019-12-05T05:04:59.961 INFO:tasks.workunit:Running workunits matching fs/snaps on client.0... 2019-12-05T05:04:59.962 INFO:tasks.workunit:Running workunit fs/snaps/snap-rm-diff.sh... 2019-12-05T05:04:59.962 INFO:teuthology.orchestra.run.smithi042:workunit test fs/snaps/snap-rm-diff.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c5290805033483ba0cec165c06bde07dae5f19ec TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snap-rm-diff.sh 2019-12-05T05:04:59.984 INFO:tasks.workunit.client.0.smithi042.stderr:+ ceph fs set cephfs allow_new_snaps true --yes-i-really-mean-it 2019-12-05T05:05:00.094 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:00.084+0000 7fe5130f9700 1 Processor -- start 2019-12-05T05:05:00.094 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:00.084+0000 7fe5130f9700 1 -- start start 2019-12-05T05:05:00.094 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:00.084+0000 7fe5130f9700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe50c0a69f0 0x7fe50c0a2840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:05:00.094 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:00.084+0000 7fe5130f9700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe50c0652f0 0x7fe50c14f200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:05:00.094 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:00.084+0000 7fe5130f9700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe50c0a7b60 con 0x7fe50c0a8030 2019-12-05T05:05:00.094 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:00.084+0000 7fe5130f9700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe50c065b20 con 0x7fe50c0a69f0 2019-12-05T05:05:00.095 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:00.084+0000 7fe5130f9700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe50c0a25a0 con 0x7fe50c0652f0 2019-12-05T05:05:00.095 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:00.084+0000 7fe511e97700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe50c0652f0 0x7fe50c14f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:05:00.095 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:00.084+0000 7fe511e97700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe50c0652f0 0x7fe50c14f200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:54348/0 (socket says 172.21.15.42:54348) 2019-12-05T05:05:00.095 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:00.084+0000 7fe511e97700 1 -- 172.21.15.42:0/3307201884 learned_addr learned my addr 172.21.15.42:0/3307201884 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:05:00.095 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:00.084+0000 7fe511696700 1 --2- 172.21.15.42:0/3307201884 >> v2:172.21.15.40:3300/0 conn(0x7fe50c0a69f0 0x7fe50c0a2840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:05:00.095 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:00.084+0000 7fe511e97700 1 -- 172.21.15.42:0/3307201884 >> v2:172.21.15.40:3300/0 conn(0x7fe50c0a69f0 msgr2=0x7fe50c0a2840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:05:00.096 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:00.084+0000 7fe511e97700 1 --2- 172.21.15.42:0/3307201884 >> v2:172.21.15.40:3300/0 conn(0x7fe50c0a69f0 0x7fe50c0a2840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:05:00.096 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:00.084+0000 7fe511e97700 1 -- 172.21.15.42:0/3307201884 >> v1:172.21.15.40:6789/0 conn(0x7fe50c0a8030 legacy=0x7fe50c144fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:05:00.096 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:00.084+0000 7fe511e97700 1 -- 172.21.15.42:0/3307201884 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe50c0a8460 con 0x7fe50c0652f0 2019-12-05T05:05:00.096 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:00.084+0000 7fe511e97700 1 --2- 172.21.15.42:0/3307201884 >> v2:172.21.15.42:3300/0 conn(0x7fe50c0652f0 0x7fe50c14f200 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fe50c0a2020 tx=0x7fe5040100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:05:00.096 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:00.084+0000 7fe511e97700 1 --2- 172.21.15.42:0/3307201884 >> v2:172.21.15.42:3300/0 conn(0x7fe50c0652f0 0x7fe50c14f200 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fe50c0a2020 tx=0x7fe5040100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:05:00.096 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:00.084+0000 7fe511e97700 1 --2- 172.21.15.42:0/3307201884 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe50c0652f0 0x7fe50c14f200 secure :-1 s=READY pgs=263 cs=0 l=1 rx=0x7fe50c0a2020 tx=0x7fe5040100b0).ready entity=mon.1 client_cookie=189bf4af6b4ae420 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:05:00.096 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:00.088+0000 7fe511e97700 1 --2- 172.21.15.42:0/3307201884 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe50c0652f0 0x7fe50c14f200 secure :-1 s=THROTTLE_DONE pgs=263 cs=0 l=1 rx=0x7fe50c0a2020 tx=0x7fe5040100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:05:00.097 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:00.088+0000 7fe503fff700 1 -- 172.21.15.42:0/3307201884 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe504011d90 con 0x7fe50c0652f0 2019-12-05T05:05:00.097 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:00.088+0000 7fe511e97700 1 --2- 172.21.15.42:0/3307201884 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe50c0652f0 0x7fe50c14f200 secure :-1 s=THROTTLE_DONE pgs=263 cs=0 l=1 rx=0x7fe50c0a2020 tx=0x7fe5040100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:05:00.097 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:00.088+0000 7fe511e97700 1 --2- 172.21.15.42:0/3307201884 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe50c0652f0 0x7fe50c14f200 secure :-1 s=THROTTLE_DONE pgs=263 cs=0 l=1 rx=0x7fe50c0a2020 tx=0x7fe5040100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:05:00.097 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:00.088+0000 7fe503fff700 1 -- 172.21.15.42:0/3307201884 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5040119b0 con 0x7fe50c0652f0 2019-12-05T05:05:00.097 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:00.088+0000 7fe503fff700 1 -- 172.21.15.42:0/3307201884 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe50400a540 con 0x7fe50c0652f0 2019-12-05T05:05:00.097 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:00.088+0000 7fe5130f9700 1 -- 172.21.15.42:0/3307201884 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe50c0652f0 msgr2=0x7fe50c14f200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:05:00.098 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:00.088+0000 7fe5130f9700 1 --2- 172.21.15.42:0/3307201884 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe50c0652f0 0x7fe50c14f200 secure :-1 s=READY pgs=263 cs=0 l=1 rx=0x7fe50c0a2020 tx=0x7fe5040100b0).stop 2019-12-05T05:05:00.098 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:00.088+0000 7fe5130f9700 1 -- 172.21.15.42:0/3307201884 shutdown_connections 2019-12-05T05:05:00.098 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:00.088+0000 7fe5130f9700 1 --2- 172.21.15.42:0/3307201884 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe50c0652f0 0x7fe50c14f200 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:05:00.098 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:00.088+0000 7fe5130f9700 1 --2- 172.21.15.42:0/3307201884 >> v2:172.21.15.40:3300/0 conn(0x7fe50c0a69f0 0x7fe50c0a2840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:05:00.098 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:00.088+0000 7fe5130f9700 1 -- 172.21.15.42:0/3307201884 shutdown_connections 2019-12-05T05:05:00.098 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:00.088+0000 7fe5130f9700 1 -- 172.21.15.42:0/3307201884 wait complete. 2019-12-05T05:05:00.098 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:00.088+0000 7fe5130f9700 1 -- 172.21.15.42:0/3307201884 >> 172.21.15.42:0/3307201884 conn(0x7fe50c059570 msgr2=0x7fe50c0599a0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:05:00.102 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:00.092+0000 7fe5130f9700 1 Processor -- start 2019-12-05T05:05:00.102 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:00.092+0000 7fe5130f9700 1 -- start start 2019-12-05T05:05:00.102 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:00.092+0000 7fe5130f9700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe50c11bda0 0x7fe50c11c660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:05:00.103 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:00.092+0000 7fe5130f9700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe50c12c9c0 0x7fe50c12cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:05:00.103 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:00.092+0000 7fe5130f9700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe50c0632f0 0x7fe50c063720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:05:00.103 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:00.092+0000 7fe5130f9700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe50c064d20 con 0x7fe50c12c9c0 2019-12-05T05:05:00.103 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:00.092+0000 7fe5130f9700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe50c12d340 con 0x7fe50c11bda0 2019-12-05T05:05:00.104 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:00.092+0000 7fe5130f9700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe50c07ed30 con 0x7fe50c0632f0 2019-12-05T05:05:00.104 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:00.092+0000 7fe511696700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe50c11bda0 0x7fe50c11c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:05:00.104 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:00.092+0000 7fe511696700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe50c11bda0 0x7fe50c11c660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:54350/0 (socket says 172.21.15.42:54350) 2019-12-05T05:05:00.104 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:00.092+0000 7fe511696700 1 -- 172.21.15.42:0/2220832571 learned_addr learned my addr 172.21.15.42:0/2220832571 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:05:00.105 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:00.092+0000 7fe511e97700 1 --2- 172.21.15.42:0/2220832571 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe50c0632f0 0x7fe50c063720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:05:00.105 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:00.092+0000 7fe510e95700 1 --2- 172.21.15.42:0/2220832571 >> v2:172.21.15.40:3300/0 conn(0x7fe50c12c9c0 0x7fe50c12cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:05:00.105 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:00.092+0000 7fe511696700 1 -- 172.21.15.42:0/2220832571 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe50c0632f0 msgr2=0x7fe50c063720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:05:00.105 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:00.092+0000 7fe511696700 1 --2- 172.21.15.42:0/2220832571 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe50c0632f0 0x7fe50c063720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:05:00.106 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:00.092+0000 7fe511696700 1 -- 172.21.15.42:0/2220832571 >> v2:172.21.15.40:3300/0 conn(0x7fe50c12c9c0 msgr2=0x7fe50c12cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:05:00.106 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:00.092+0000 7fe511696700 1 --2- 172.21.15.42:0/2220832571 >> v2:172.21.15.40:3300/0 conn(0x7fe50c12c9c0 0x7fe50c12cdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:05:00.106 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:00.092+0000 7fe511696700 1 -- 172.21.15.42:0/2220832571 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe50c063c70 con 0x7fe50c11bda0 2019-12-05T05:05:00.106 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:00.092+0000 7fe511696700 1 --2- 172.21.15.42:0/2220832571 >> v2:172.21.15.42:3300/0 conn(0x7fe50c11bda0 0x7fe50c11c660 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fe50c07eef0 tx=0x7fe508010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:05:00.107 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:00.092+0000 7fe511696700 1 --2- 172.21.15.42:0/2220832571 >> v2:172.21.15.42:3300/0 conn(0x7fe50c11bda0 0x7fe50c11c660 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fe50c07eef0 tx=0x7fe508010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:05:00.107 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:00.092+0000 7fe511696700 1 --2- 172.21.15.42:0/2220832571 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe50c11bda0 0x7fe50c11c660 secure :-1 s=READY pgs=264 cs=0 l=1 rx=0x7fe50c07eef0 tx=0x7fe508010b60).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:05:00.107 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:00.092+0000 7fe511696700 1 --2- 172.21.15.42:0/2220832571 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe50c11bda0 0x7fe50c11c660 secure :-1 s=THROTTLE_DONE pgs=264 cs=0 l=1 rx=0x7fe50c07eef0 tx=0x7fe508010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:05:00.107 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:00.092+0000 7fe511696700 1 --2- 172.21.15.42:0/2220832571 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe50c11bda0 0x7fe50c11c660 secure :-1 s=THROTTLE_DONE pgs=264 cs=0 l=1 rx=0x7fe50c07eef0 tx=0x7fe508010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:05:00.108 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:00.092+0000 7fe501ffb700 1 -- 172.21.15.42:0/2220832571 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe508005930 con 0x7fe50c11bda0 2019-12-05T05:05:00.108 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:00.092+0000 7fe511696700 1 --2- 172.21.15.42:0/2220832571 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe50c11bda0 0x7fe50c11c660 secure :-1 s=THROTTLE_DONE pgs=264 cs=0 l=1 rx=0x7fe50c07eef0 tx=0x7fe508010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:05:00.108 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:00.092+0000 7fe501ffb700 1 -- 172.21.15.42:0/2220832571 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe508007160 con 0x7fe50c11bda0 2019-12-05T05:05:00.108 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:00.092+0000 7fe5130f9700 1 -- 172.21.15.42:0/2220832571 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe50c11cdb0 con 0x7fe50c11bda0 2019-12-05T05:05:00.109 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:00.096+0000 7fe501ffb700 1 -- 172.21.15.42:0/2220832571 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe5080121a0 con 0x7fe50c11bda0 2019-12-05T05:05:00.109 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:00.096+0000 7fe5130f9700 1 -- 172.21.15.42:0/2220832571 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe50c07a520 con 0x7fe50c11bda0 2019-12-05T05:05:00.109 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:00.096+0000 7fe511696700 1 --2- 172.21.15.42:0/2220832571 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe50c11bda0 0x7fe50c11c660 secure :-1 s=THROTTLE_DONE pgs=264 cs=0 l=1 rx=0x7fe50c07eef0 tx=0x7fe508010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:05:00.109 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:00.096+0000 7fe501ffb700 1 -- 172.21.15.42:0/2220832571 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fe508012470 con 0x7fe50c11bda0 2019-12-05T05:05:00.109 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:00.096+0000 7fe511696700 1 --2- 172.21.15.42:0/2220832571 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe50c11bda0 0x7fe50c11c660 secure :-1 s=THROTTLE_DONE pgs=264 cs=0 l=1 rx=0x7fe50c07eef0 tx=0x7fe508010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:05:00.109 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:00.096+0000 7fe501ffb700 1 --2- 172.21.15.42:0/2220832571 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe4ec03aef0 0x7fe4ec03d400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:05:00.109 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:00.096+0000 7fe501ffb700 1 -- 172.21.15.42:0/2220832571 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6117+0+0 (secure 0 0 0) 0x7fe50804aea0 con 0x7fe50c11bda0 2019-12-05T05:05:00.110 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:00.096+0000 7fe510e95700 1 --2- 172.21.15.42:0/2220832571 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe4ec03aef0 0x7fe4ec03d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:05:00.110 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:00.096+0000 7fe510e95700 1 --2- 172.21.15.42:0/2220832571 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe4ec03aef0 0x7fe4ec03d400 crc :-1 s=READY pgs=127 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:05:00.110 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:00.100+0000 7fe5130f9700 1 -- 172.21.15.42:0/2220832571 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe50c11cbb0 con 0x7fe50c11bda0 2019-12-05T05:05:00.112 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:00.108+0000 7fe511696700 1 --2- 172.21.15.42:0/2220832571 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe50c11bda0 0x7fe50c11c660 secure :-1 s=THROTTLE_DONE pgs=264 cs=0 l=1 rx=0x7fe50c07eef0 tx=0x7fe508010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:05:00.112 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:00.108+0000 7fe501ffb700 1 -- 172.21.15.42:0/2220832571 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fe508031f20 con 0x7fe50c11bda0 2019-12-05T05:05:00.278 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:00.272+0000 7fe5130f9700 1 -- 172.21.15.42:0/2220832571 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"var": "allow_new_snaps", "prefix": "fs set", "fs_name": "cephfs", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fe50c05d2c0 con 0x7fe50c11bda0 2019-12-05T05:05:00.687 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:00.680+0000 7fe511696700 1 --2- 172.21.15.42:0/2220832571 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe50c11bda0 0x7fe50c11c660 secure :-1 s=THROTTLE_DONE pgs=264 cs=0 l=1 rx=0x7fe50c07eef0 tx=0x7fe508010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:05:00.687 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:00.680+0000 7fe501ffb700 1 -- 172.21.15.42:0/2220832571 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"var": "allow_new_snaps", "prefix": "fs set", "fs_name": "cephfs", "val": "true", "yes_i_really_mean_it": true}]=0 enabled new snapshots v12) v1 ==== 167+0+0 (secure 0 0 0) 0x7fe508032180 con 0x7fe50c11bda0 2019-12-05T05:05:00.812 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:00.808+0000 7fe5130f9700 1 -- 172.21.15.42:0/2220832571 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"var": "allow_new_snaps", "prefix": "fs set", "fs_name": "cephfs", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fe50c05ba70 con 0x7fe50c11bda0 2019-12-05T05:05:01.697 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:01.692+0000 7fe511696700 1 --2- 172.21.15.42:0/2220832571 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe50c11bda0 0x7fe50c11c660 secure :-1 s=THROTTLE_DONE pgs=264 cs=0 l=1 rx=0x7fe50c07eef0 tx=0x7fe508010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:05:01.697 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:01.692+0000 7fe501ffb700 1 -- 172.21.15.42:0/2220832571 <== mon.1 v2:172.21.15.42:3300/0 8 ==== mon_command_ack([{"var": "allow_new_snaps", "prefix": "fs set", "fs_name": "cephfs", "val": "true", "yes_i_really_mean_it": true}]=0 enabled new snapshots v13) v1 ==== 167+0+0 (secure 0 0 0) 0x7fe508032d30 con 0x7fe50c11bda0 2019-12-05T05:05:01.727 INFO:tasks.workunit.client.0.smithi042.stderr:enabled new snapshots 2019-12-05T05:05:01.732 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:01.728+0000 7fe5130f9700 1 -- 172.21.15.42:0/2220832571 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe4ec03aef0 msgr2=0x7fe4ec03d400 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:05:01.732 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:01.728+0000 7fe5130f9700 1 --2- 172.21.15.42:0/2220832571 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe4ec03aef0 0x7fe4ec03d400 crc :-1 s=READY pgs=127 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:05:01.732 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:01.728+0000 7fe5130f9700 1 -- 172.21.15.42:0/2220832571 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe50c11bda0 msgr2=0x7fe50c11c660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:05:01.733 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:01.728+0000 7fe5130f9700 1 --2- 172.21.15.42:0/2220832571 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe50c11bda0 0x7fe50c11c660 secure :-1 s=READY pgs=264 cs=0 l=1 rx=0x7fe50c07eef0 tx=0x7fe508010b60).stop 2019-12-05T05:05:01.733 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:01.728+0000 7fe5130f9700 1 -- 172.21.15.42:0/2220832571 shutdown_connections 2019-12-05T05:05:01.733 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:01.728+0000 7fe5130f9700 1 --2- 172.21.15.42:0/2220832571 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe4ec03aef0 0x7fe4ec03d400 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:05:01.733 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:01.728+0000 7fe5130f9700 1 --2- 172.21.15.42:0/2220832571 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe50c0632f0 0x7fe50c063720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:05:01.733 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:01.728+0000 7fe5130f9700 1 --2- 172.21.15.42:0/2220832571 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe50c11bda0 0x7fe50c11c660 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:05:01.733 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:01.728+0000 7fe5130f9700 1 --2- 172.21.15.42:0/2220832571 >> v2:172.21.15.40:3300/0 conn(0x7fe50c12c9c0 0x7fe50c12cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:05:01.733 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:01.728+0000 7fe5130f9700 1 -- 172.21.15.42:0/2220832571 shutdown_connections 2019-12-05T05:05:01.734 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:01.728+0000 7fe5130f9700 1 -- 172.21.15.42:0/2220832571 wait complete. 2019-12-05T05:05:01.734 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:01.728+0000 7fe5130f9700 1 -- 172.21.15.42:0/2220832571 >> 172.21.15.42:0/2220832571 conn(0x7fe50c0a7470 msgr2=0x7fe50c09c8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:05:01.745 INFO:tasks.workunit.client.0.smithi042.stderr:+ wget -q http://download.ceph.com/qa/linux-2.6.33.tar.bz2 2019-12-05T05:05:03.632 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:05:03.635 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:05:06.083 INFO:tasks.workunit.client.0.smithi042.stderr:+ mkdir foo 2019-12-05T05:05:06.096 INFO:tasks.workunit.client.0.smithi042.stderr:+ cp linux-2.6.33.tar.bz2 foo 2019-12-05T05:05:07.777 INFO:tasks.workunit.client.0.smithi042.stderr:+ mkdir foo/.snap/barsnap 2019-12-05T05:05:07.792 INFO:tasks.workunit.client.0.smithi042.stderr:+ rm foo/linux-2.6.33.tar.bz2 2019-12-05T05:05:07.805 INFO:tasks.workunit.client.0.smithi042.stderr:+ diff -q foo/.snap/barsnap/linux-2.6.33.tar.bz2 linux-2.6.33.tar.bz2 2019-12-05T05:05:08.016 INFO:tasks.workunit.client.0.smithi042.stderr:+ echo passed: files are identical 2019-12-05T05:05:08.016 INFO:tasks.workunit.client.0.smithi042.stderr:+ rmdir foo/.snap/barsnap 2019-12-05T05:05:08.016 INFO:tasks.workunit.client.0.smithi042.stdout:passed: files are identical 2019-12-05T05:05:08.034 INFO:tasks.workunit.client.0.smithi042.stderr:+ echo OK 2019-12-05T05:05:08.035 INFO:tasks.workunit.client.0.smithi042.stdout:OK 2019-12-05T05:05:08.035 INFO:teuthology.orchestra.run:Running command with timeout 3600 2019-12-05T05:05:08.035 INFO:teuthology.orchestra.run.smithi042:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2019-12-05T05:05:08.103 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-0.sh... 2019-12-05T05:05:08.103 INFO:teuthology.orchestra.run.smithi042:workunit test fs/snaps/snaptest-0.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c5290805033483ba0cec165c06bde07dae5f19ec TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-0.sh 2019-12-05T05:05:08.142 INFO:tasks.workunit.client.0.smithi042.stderr:+ set -e 2019-12-05T05:05:08.142 INFO:tasks.workunit.client.0.smithi042.stderr:+ ceph fs set cephfs allow_new_snaps false 2019-12-05T05:05:08.238 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:08.228+0000 7fdf9e932700 1 Processor -- start 2019-12-05T05:05:08.239 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:08.228+0000 7fdf9e932700 1 -- start start 2019-12-05T05:05:08.239 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:08.228+0000 7fdf9e932700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fdf9805a0b0 0x7fdf980a29b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:05:08.239 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:08.228+0000 7fdf9e932700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf980a2f00 con 0x7fdf981448a0 2019-12-05T05:05:08.239 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:08.228+0000 7fdf9e932700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf980a3620 con 0x7fdf980a6960 2019-12-05T05:05:08.239 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:08.228+0000 7fdf9e932700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf980a0d50 con 0x7fdf9805a0b0 2019-12-05T05:05:08.239 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:08.232+0000 7fdf97fff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fdf980a6960 0x7fdf980a6d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:54432/0 (socket says 172.21.15.42:54432) 2019-12-05T05:05:08.240 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:08.232+0000 7fdf97fff700 1 -- 172.21.15.42:0/1531816302 learned_addr learned my addr 172.21.15.42:0/1531816302 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:05:08.240 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:08.232+0000 7fdf9cecf700 1 --2- 172.21.15.42:0/1531816302 >> v2:172.21.15.40:3300/0 conn(0x7fdf9805a0b0 0x7fdf980a29b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:05:08.240 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:08.232+0000 7fdf977fe700 1 -- 172.21.15.42:0/1531816302 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 703836735 0 0) 0x7fdf980a3620 con 0x7fdf980a6960 2019-12-05T05:05:08.240 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:08.232+0000 7fdf977fe700 1 -- 172.21.15.42:0/1531816302 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf800030b0 con 0x7fdf980a6960 2019-12-05T05:05:08.240 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:08.232+0000 7fdf977fe700 1 -- 172.21.15.42:0/1531816302 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3405282117 0 0) 0x7fdf980a2f00 con 0x7fdf981448a0 2019-12-05T05:05:08.240 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:08.232+0000 7fdf9cecf700 1 -- 172.21.15.42:0/1531816302 >> v1:172.21.15.42:6789/0 conn(0x7fdf980a6960 legacy=0x7fdf980a6d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:05:08.240 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:08.232+0000 7fdf9cecf700 1 -- 172.21.15.42:0/1531816302 >> v1:172.21.15.40:6789/0 conn(0x7fdf981448a0 legacy=0x7fdf98144cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:05:08.241 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:08.232+0000 7fdf9cecf700 1 -- 172.21.15.42:0/1531816302 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf981454d0 con 0x7fdf9805a0b0 2019-12-05T05:05:08.241 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:08.232+0000 7fdf9cecf700 1 --2- 172.21.15.42:0/1531816302 >> v2:172.21.15.40:3300/0 conn(0x7fdf9805a0b0 0x7fdf980a29b0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fdf88008b10 tx=0x7fdf8800ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:05:08.241 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:08.232+0000 7fdf9cecf700 1 --2- 172.21.15.42:0/1531816302 >> v2:172.21.15.40:3300/0 conn(0x7fdf9805a0b0 0x7fdf980a29b0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fdf88008b10 tx=0x7fdf8800ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:05:08.241 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:08.232+0000 7fdf9cecf700 1 --2- 172.21.15.42:0/1531816302 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf9805a0b0 0x7fdf980a29b0 secure :-1 s=READY pgs=210 cs=0 l=1 rx=0x7fdf88008b10 tx=0x7fdf8800ff70).ready entity=mon.0 client_cookie=6c68a36961dff071 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:05:08.241 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:08.232+0000 7fdf9cecf700 1 --2- 172.21.15.42:0/1531816302 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf9805a0b0 0x7fdf980a29b0 secure :-1 s=THROTTLE_DONE pgs=210 cs=0 l=1 rx=0x7fdf88008b10 tx=0x7fdf8800ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:05:08.241 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:08.232+0000 7fdf977fe700 1 -- 172.21.15.42:0/1531816302 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdf880119d0 con 0x7fdf9805a0b0 2019-12-05T05:05:08.242 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:08.232+0000 7fdf9cecf700 1 --2- 172.21.15.42:0/1531816302 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf9805a0b0 0x7fdf980a29b0 secure :-1 s=THROTTLE_DONE pgs=210 cs=0 l=1 rx=0x7fdf88008b10 tx=0x7fdf8800ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:05:08.242 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:08.232+0000 7fdf977fe700 1 -- 172.21.15.42:0/1531816302 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf880115f0 con 0x7fdf9805a0b0 2019-12-05T05:05:08.242 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:08.232+0000 7fdf9cecf700 1 --2- 172.21.15.42:0/1531816302 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf9805a0b0 0x7fdf980a29b0 secure :-1 s=THROTTLE_DONE pgs=210 cs=0 l=1 rx=0x7fdf88008b10 tx=0x7fdf8800ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:05:08.242 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:08.232+0000 7fdf977fe700 1 -- 172.21.15.42:0/1531816302 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdf8800a160 con 0x7fdf9805a0b0 2019-12-05T05:05:08.242 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:08.232+0000 7fdf9e932700 1 -- 172.21.15.42:0/1531816302 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf9805a0b0 msgr2=0x7fdf980a29b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:05:08.242 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:08.232+0000 7fdf9e932700 1 --2- 172.21.15.42:0/1531816302 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf9805a0b0 0x7fdf980a29b0 secure :-1 s=READY pgs=210 cs=0 l=1 rx=0x7fdf88008b10 tx=0x7fdf8800ff70).stop 2019-12-05T05:05:08.242 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:08.232+0000 7fdf9e932700 1 -- 172.21.15.42:0/1531816302 shutdown_connections 2019-12-05T05:05:08.243 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:08.232+0000 7fdf9e932700 1 --2- 172.21.15.42:0/1531816302 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf9805a0b0 0x7fdf980a29b0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:05:08.243 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:08.232+0000 7fdf9e932700 1 -- 172.21.15.42:0/1531816302 shutdown_connections 2019-12-05T05:05:08.243 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:08.232+0000 7fdf9e932700 1 -- 172.21.15.42:0/1531816302 wait complete. 2019-12-05T05:05:08.243 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:08.232+0000 7fdf9e932700 1 -- 172.21.15.42:0/1531816302 >> 172.21.15.42:0/1531816302 conn(0x7fdf9805ee60 msgr2=0x7fdf98059570 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:05:08.247 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:08.236+0000 7fdf9e932700 1 Processor -- start 2019-12-05T05:05:08.247 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:08.236+0000 7fdf9e932700 1 -- start start 2019-12-05T05:05:08.247 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:08.236+0000 7fdf9e932700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fdf980a1450 0x7fdf98140830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:05:08.248 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:08.236+0000 7fdf9e932700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdf9811a290 0x7fdf9811b930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:05:08.248 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:08.236+0000 7fdf9e932700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf9807c240 con 0x7fdf9811be80 2019-12-05T05:05:08.248 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:08.236+0000 7fdf9e932700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf98140050 con 0x7fdf980a1450 2019-12-05T05:05:08.248 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:08.236+0000 7fdf9e932700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf981402f0 con 0x7fdf9811a290 2019-12-05T05:05:08.248 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:08.236+0000 7fdf97fff700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdf9811a290 0x7fdf9811b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:05:08.248 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:08.236+0000 7fdf9cecf700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fdf980a1450 0x7fdf98140830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:05:08.248 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:08.236+0000 7fdf97fff700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdf9811a290 0x7fdf9811b930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:54866/0 (socket says 172.21.15.42:54866) 2019-12-05T05:05:08.249 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:08.236+0000 7fdf9cecf700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fdf980a1450 0x7fdf98140830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.42:42214/0 (socket says 172.21.15.42:42214) 2019-12-05T05:05:08.249 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:08.236+0000 7fdf97fff700 1 -- 172.21.15.42:0/1847396649 learned_addr learned my addr 172.21.15.42:0/1847396649 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:05:08.249 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:08.240+0000 7fdf97fff700 1 -- 172.21.15.42:0/1847396649 >> v2:172.21.15.40:3300/0 conn(0x7fdf980a1450 msgr2=0x7fdf98140830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:05:08.249 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:08.240+0000 7fdf97fff700 1 --2- 172.21.15.42:0/1847396649 >> v2:172.21.15.40:3300/0 conn(0x7fdf980a1450 0x7fdf98140830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:05:08.249 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:08.240+0000 7fdf97fff700 1 -- 172.21.15.42:0/1847396649 >> v1:172.21.15.40:6789/0 conn(0x7fdf9811be80 legacy=0x7fdf9807ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:05:08.249 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:08.240+0000 7fdf97fff700 1 -- 172.21.15.42:0/1847396649 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf9811a6c0 con 0x7fdf9811a290 2019-12-05T05:05:08.250 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:08.240+0000 7fdf9cecf700 1 --2- 172.21.15.42:0/1847396649 >> v2:172.21.15.40:3300/0 conn(0x7fdf980a1450 0x7fdf98140830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:05:08.250 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:08.240+0000 7fdf97fff700 1 --2- 172.21.15.42:0/1847396649 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdf9811a290 0x7fdf9811b930 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fdf8c0043a0 tx=0x7fdf8c0108e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:05:08.250 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:08.240+0000 7fdf97fff700 1 --2- 172.21.15.42:0/1847396649 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdf9811a290 0x7fdf9811b930 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fdf8c0043a0 tx=0x7fdf8c0108e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:05:08.250 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:08.240+0000 7fdf97fff700 1 --2- 172.21.15.42:0/1847396649 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdf9811a290 0x7fdf9811b930 secure :-1 s=READY pgs=53 cs=0 l=1 rx=0x7fdf8c0043a0 tx=0x7fdf8c0108e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:05:08.250 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:08.240+0000 7fdf97fff700 1 --2- 172.21.15.42:0/1847396649 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdf9811a290 0x7fdf9811b930 secure :-1 s=THROTTLE_DONE pgs=53 cs=0 l=1 rx=0x7fdf8c0043a0 tx=0x7fdf8c0108e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:05:08.250 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:08.240+0000 7fdf97fff700 1 --2- 172.21.15.42:0/1847396649 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdf9811a290 0x7fdf9811b930 secure :-1 s=THROTTLE_DONE pgs=53 cs=0 l=1 rx=0x7fdf8c0043a0 tx=0x7fdf8c0108e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:05:08.250 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:08.240+0000 7fdf957fa700 1 -- 172.21.15.42:0/1847396649 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdf8c005b40 con 0x7fdf9811a290 2019-12-05T05:05:08.251 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:08.240+0000 7fdf97fff700 1 --2- 172.21.15.42:0/1847396649 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdf9811a290 0x7fdf9811b930 secure :-1 s=THROTTLE_DONE pgs=53 cs=0 l=1 rx=0x7fdf8c0043a0 tx=0x7fdf8c0108e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:05:08.251 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:08.240+0000 7fdf957fa700 1 -- 172.21.15.42:0/1847396649 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf8c005760 con 0x7fdf9811a290 2019-12-05T05:05:08.251 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:08.240+0000 7fdf957fa700 1 -- 172.21.15.42:0/1847396649 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdf8c0062b0 con 0x7fdf9811a290 2019-12-05T05:05:08.251 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:08.240+0000 7fdf9e932700 1 -- 172.21.15.42:0/1847396649 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf981320b0 con 0x7fdf9811a290 2019-12-05T05:05:08.251 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:08.240+0000 7fdf9e932700 1 -- 172.21.15.42:0/1847396649 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf981325e0 con 0x7fdf9811a290 2019-12-05T05:05:08.251 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:08.240+0000 7fdf97fff700 1 --2- 172.21.15.42:0/1847396649 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdf9811a290 0x7fdf9811b930 secure :-1 s=THROTTLE_DONE pgs=53 cs=0 l=1 rx=0x7fdf8c0043a0 tx=0x7fdf8c0108e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:05:08.251 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:08.240+0000 7fdf97fff700 1 -- 172.21.15.42:0/1847396649 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdf9811a290 msgr2=0x7fdf9811b930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2019-12-05T05:05:08.252 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:08.240+0000 7fdf957fa700 1 -- 172.21.15.42:0/1847396649 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fdf8c0065c0 con 0x7fdf9811a290 2019-12-05T05:05:08.252 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:08.240+0000 7fdf97fff700 1 -- 172.21.15.42:0/1847396649 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdf9811a290 msgr2=0x7fdf9811b930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2019-12-05T05:05:08.252 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:08.240+0000 7fdf97fff700 1 --2- 172.21.15.42:0/1847396649 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdf9811a290 0x7fdf9811b930 secure :-1 s=READY pgs=53 cs=0 l=1 rx=0x7fdf8c0043a0 tx=0x7fdf8c0108e0).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-12-05T05:05:08.252 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:08.240+0000 7fdf97fff700 1 --2- 172.21.15.42:0/1847396649 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdf9811a290 0x7fdf9811b930 secure :-1 s=READY pgs=53 cs=0 l=1 rx=0x7fdf8c0043a0 tx=0x7fdf8c0108e0).stop 2019-12-05T05:05:08.252 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:08.240+0000 7fdf957fa700 1 --2- 172.21.15.42:0/1847396649 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fdf7803ad80 0x7fdf7803d290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:05:08.252 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:08.240+0000 7fdf957fa700 1 -- 172.21.15.42:0/1847396649 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdf9811a290 msgr2=0x7fdf9811b930 unknown :-1 s=STATE_CLOSED l=1).mark_down 2019-12-05T05:05:08.252 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:08.240+0000 7fdf957fa700 1 --2- 172.21.15.42:0/1847396649 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdf9811a290 0x7fdf9811b930 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:05:08.253 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:08.240+0000 7fdf957fa700 1 --2- 172.21.15.42:0/1847396649 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf7803d9a0 0x7fdf7803fde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:05:08.253 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:08.240+0000 7fdf957fa700 1 --2- 172.21.15.42:0/1847396649 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdf78040330 0x7fdf78042810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:05:08.253 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:08.240+0000 7fdf957fa700 1 --2- 172.21.15.42:0/1847396649 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdf78042ef0 0x7fdf780453d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:05:08.253 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:08.240+0000 7fdf957fa700 1 -- 172.21.15.42:0/1847396649 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf78045ae0 con 0x7fdf7803d9a0 2019-12-05T05:05:08.253 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:08.240+0000 7fdf957fa700 1 -- 172.21.15.42:0/1847396649 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf78045cd0 con 0x7fdf78040330 2019-12-05T05:05:08.253 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:08.240+0000 7fdf957fa700 1 -- 172.21.15.42:0/1847396649 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf78045f10 con 0x7fdf78042ef0 2019-12-05T05:05:08.254 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:08.240+0000 7fdf9cecf700 1 --2- 172.21.15.42:0/1847396649 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fdf7803ad80 0x7fdf7803d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:05:08.254 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:08.240+0000 7fdf9d6d0700 1 --2- 172.21.15.42:0/1847396649 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdf78040330 0x7fdf78042810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:05:08.254 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:08.240+0000 7fdf97fff700 1 --2- 172.21.15.42:0/1847396649 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf7803d9a0 0x7fdf7803fde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:05:08.254 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:08.244+0000 7fdf9cecf700 1 --2- 172.21.15.42:0/1847396649 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdf78042ef0 0x7fdf780453d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:05:08.254 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:08.244+0000 7fdf9cecf700 1 --2- 172.21.15.42:0/1847396649 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fdf7803ad80 0x7fdf7803d290 crc :-1 s=READY pgs=130 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:05:08.256 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:08.244+0000 7fdf97fff700 1 -- 172.21.15.42:0/1847396649 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdf78042ef0 msgr2=0x7fdf780453d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:05:08.256 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:08.244+0000 7fdf97fff700 1 --2- 172.21.15.42:0/1847396649 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdf78042ef0 0x7fdf780453d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:05:08.258 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:08.244+0000 7fdf97fff700 1 -- 172.21.15.42:0/1847396649 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdf78040330 msgr2=0x7fdf78042810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:05:08.258 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:08.244+0000 7fdf97fff700 1 --2- 172.21.15.42:0/1847396649 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdf78040330 0x7fdf78042810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:05:08.258 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:08.244+0000 7fdf97fff700 1 -- 172.21.15.42:0/1847396649 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+,osdmap=0}) v3 -- 0x7fdf78046150 con 0x7fdf7803d9a0 2019-12-05T05:05:08.259 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:08.244+0000 7fdf97fff700 1 --2- 172.21.15.42:0/1847396649 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf7803d9a0 0x7fdf7803fde0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fdf8c0043a0 tx=0x7fdf8c02fa40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:05:08.259 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:08.244+0000 7fdf97fff700 1 --2- 172.21.15.42:0/1847396649 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf7803d9a0 0x7fdf7803fde0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fdf8c0043a0 tx=0x7fdf8c02fa40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:05:08.259 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:08.244+0000 7fdf97fff700 1 --2- 172.21.15.42:0/1847396649 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf7803d9a0 0x7fdf7803fde0 secure :-1 s=READY pgs=212 cs=0 l=1 rx=0x7fdf8c0043a0 tx=0x7fdf8c02fa40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:05:08.259 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:08.244+0000 7fdf97fff700 1 --2- 172.21.15.42:0/1847396649 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf7803d9a0 0x7fdf7803fde0 secure :-1 s=THROTTLE_DONE pgs=212 cs=0 l=1 rx=0x7fdf8c0043a0 tx=0x7fdf8c02fa40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:05:08.259 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:08.244+0000 7fdf957fa700 1 -- 172.21.15.42:0/1847396649 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdf8c032f30 con 0x7fdf7803d9a0 2019-12-05T05:05:08.260 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:08.244+0000 7fdf97fff700 1 --2- 172.21.15.42:0/1847396649 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf7803d9a0 0x7fdf7803fde0 secure :-1 s=THROTTLE_DONE pgs=212 cs=0 l=1 rx=0x7fdf8c0043a0 tx=0x7fdf8c02fa40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:05:08.260 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:08.244+0000 7fdf957fa700 1 -- 172.21.15.42:0/1847396649 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf8c005d40 con 0x7fdf7803d9a0 2019-12-05T05:05:08.260 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:08.244+0000 7fdf97fff700 1 --2- 172.21.15.42:0/1847396649 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf7803d9a0 0x7fdf7803fde0 secure :-1 s=THROTTLE_DONE pgs=212 cs=0 l=1 rx=0x7fdf8c0043a0 tx=0x7fdf8c02fa40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:05:08.260 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:08.244+0000 7fdf9e932700 1 -- 172.21.15.42:0/1847396649 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf9805d2c0 con 0x7fdf7803d9a0 2019-12-05T05:05:08.260 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:08.244+0000 7fdf957fa700 1 -- 172.21.15.42:0/1847396649 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fdf8c033a10 con 0x7fdf7803d9a0 2019-12-05T05:05:08.261 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:08.244+0000 7fdf97fff700 1 --2- 172.21.15.42:0/1847396649 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf7803d9a0 0x7fdf7803fde0 secure :-1 s=THROTTLE_DONE pgs=212 cs=0 l=1 rx=0x7fdf8c0043a0 tx=0x7fdf8c02fa40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:05:08.261 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:08.244+0000 7fdf957fa700 1 -- 172.21.15.42:0/1847396649 <== mon.0 v2:172.21.15.40:3300/0 4 ==== osd_map(25..25 src has 1..25) v4 ==== 6117+0+0 (secure 0 0 0) 0x7fdf8c04e100 con 0x7fdf7803d9a0 2019-12-05T05:05:08.262 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:08.256+0000 7fdf97fff700 1 --2- 172.21.15.42:0/1847396649 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf7803d9a0 0x7fdf7803fde0 secure :-1 s=THROTTLE_DONE pgs=212 cs=0 l=1 rx=0x7fdf8c0043a0 tx=0x7fdf8c02fa40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:05:08.262 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:08.256+0000 7fdf957fa700 1 -- 172.21.15.42:0/1847396649 <== mon.0 v2:172.21.15.40:3300/0 5 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fdf8c033d20 con 0x7fdf7803d9a0 2019-12-05T05:05:08.427 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:08.420+0000 7fdf9e932700 1 -- 172.21.15.42:0/1847396649 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"var": "allow_new_snaps", "prefix": "fs set", "fs_name": "cephfs", "val": "false"} v 0) v1 -- 0x7fdf9813fe30 con 0x7fdf7803d9a0 2019-12-05T05:05:09.198 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:09.192+0000 7fdf97fff700 1 --2- 172.21.15.42:0/1847396649 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf7803d9a0 0x7fdf7803fde0 secure :-1 s=THROTTLE_DONE pgs=212 cs=0 l=1 rx=0x7fdf8c0043a0 tx=0x7fdf8c02fa40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:05:09.198 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:09.192+0000 7fdf957fa700 1 -- 172.21.15.42:0/1847396649 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"var": "allow_new_snaps", "prefix": "fs set", "fs_name": "cephfs", "val": "false"}]=0 disabled new snapshots v14) v1 ==== 139+0+0 (secure 0 0 0) 0x7fdf8c033f40 con 0x7fdf7803d9a0 2019-12-05T05:05:09.359 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:09.352+0000 7fdf9e932700 1 -- 172.21.15.42:0/1847396649 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"var": "allow_new_snaps", "prefix": "fs set", "fs_name": "cephfs", "val": "false"} v 0) v1 -- 0x7fdf981404b0 con 0x7fdf7803d9a0 2019-12-05T05:05:10.204 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:10.200+0000 7fdf97fff700 1 --2- 172.21.15.42:0/1847396649 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf7803d9a0 0x7fdf7803fde0 secure :-1 s=THROTTLE_DONE pgs=212 cs=0 l=1 rx=0x7fdf8c0043a0 tx=0x7fdf8c02fa40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:05:10.204 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:10.200+0000 7fdf957fa700 1 -- 172.21.15.42:0/1847396649 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"var": "allow_new_snaps", "prefix": "fs set", "fs_name": "cephfs", "val": "false"}]=0 disabled new snapshots v15) v1 ==== 139+0+0 (secure 0 0 0) 0x7fdf8c049a10 con 0x7fdf7803d9a0 2019-12-05T05:05:10.225 INFO:tasks.workunit.client.0.smithi042.stderr:disabled new snapshots 2019-12-05T05:05:10.230 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:10.224+0000 7fdf9e932700 1 -- 172.21.15.42:0/1847396649 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fdf7803ad80 msgr2=0x7fdf7803d290 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:05:10.230 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:10.224+0000 7fdf9e932700 1 --2- 172.21.15.42:0/1847396649 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fdf7803ad80 0x7fdf7803d290 crc :-1 s=READY pgs=130 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:05:10.230 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:10.224+0000 7fdf9e932700 1 -- 172.21.15.42:0/1847396649 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf7803d9a0 msgr2=0x7fdf7803fde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:05:10.231 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:10.224+0000 7fdf9e932700 1 --2- 172.21.15.42:0/1847396649 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf7803d9a0 0x7fdf7803fde0 secure :-1 s=READY pgs=212 cs=0 l=1 rx=0x7fdf8c0043a0 tx=0x7fdf8c02fa40).stop 2019-12-05T05:05:10.231 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:10.224+0000 7fdf9d6d0700 1 -- 172.21.15.42:0/1847396649 reap_dead start 2019-12-05T05:05:10.231 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:10.224+0000 7fdf9e932700 1 -- 172.21.15.42:0/1847396649 shutdown_connections 2019-12-05T05:05:10.231 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:10.224+0000 7fdf9e932700 1 -- 172.21.15.42:0/1847396649 shutdown_connections 2019-12-05T05:05:10.231 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:10.224+0000 7fdf9e932700 1 -- 172.21.15.42:0/1847396649 wait complete. 2019-12-05T05:05:10.232 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:10.224+0000 7fdf9e932700 1 -- 172.21.15.42:0/1847396649 >> 172.21.15.42:0/1847396649 conn(0x7fdf9805edc0 msgr2=0x7fdf980a9b70 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:05:10.240 INFO:tasks.workunit.client.0.smithi042.stderr:+ expect_failure mkdir .snap/foo 2019-12-05T05:05:10.241 INFO:tasks.workunit.client.0.smithi042.stderr:+ mkdir .snap/foo 2019-12-05T05:05:10.247 INFO:tasks.workunit.client.0.smithi042.stderr:mkdir: cannot create directory '.snap/foo': Operation not permitted 2019-12-05T05:05:10.247 INFO:tasks.workunit.client.0.smithi042.stderr:+ return 0 2019-12-05T05:05:10.247 INFO:tasks.workunit.client.0.smithi042.stderr:+ ceph fs set cephfs allow_new_snaps true --yes-i-really-mean-it 2019-12-05T05:05:10.386 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:10.364+0000 7f55e26ec700 1 Processor -- start 2019-12-05T05:05:10.386 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:10.364+0000 7f55e26ec700 1 -- start start 2019-12-05T05:05:10.386 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:10.368+0000 7f55e26ec700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f55dc0a69f0 0x7f55dc0a2840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:05:10.386 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:10.368+0000 7f55e26ec700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f55dc0652f0 0x7f55dc14f200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:05:10.387 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:10.368+0000 7f55e26ec700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55dc0a7b60 con 0x7f55dc0a8030 2019-12-05T05:05:10.387 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:10.368+0000 7f55e26ec700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55dc065b20 con 0x7f55dc0a69f0 2019-12-05T05:05:10.387 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:10.368+0000 7f55e26ec700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55dc0a25a0 con 0x7f55dc0652f0 2019-12-05T05:05:10.388 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:10.368+0000 7f55d3fff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f55dc0a8030 0x7f55dc144fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:54464/0 (socket says 172.21.15.42:54464) 2019-12-05T05:05:10.389 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:10.368+0000 7f55d3fff700 1 -- 172.21.15.42:0/333541989 learned_addr learned my addr 172.21.15.42:0/333541989 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:05:10.389 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:10.368+0000 7f55e0c89700 1 --2- 172.21.15.42:0/333541989 >> v2:172.21.15.40:3300/0 conn(0x7f55dc0a69f0 0x7f55dc0a2840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:05:10.389 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:10.368+0000 7f55e148a700 1 --2- 172.21.15.42:0/333541989 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f55dc0652f0 0x7f55dc14f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:05:10.389 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:10.368+0000 7f55d37fe700 1 -- 172.21.15.42:0/333541989 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 71680485 0 0) 0x7f55dc0a7b60 con 0x7f55dc0a8030 2019-12-05T05:05:10.391 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:10.368+0000 7f55d37fe700 1 -- 172.21.15.42:0/333541989 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55c40030b0 con 0x7f55dc0a8030 2019-12-05T05:05:10.391 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:10.368+0000 7f55e0c89700 1 -- 172.21.15.42:0/333541989 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f55dc0652f0 msgr2=0x7f55dc14f200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:05:10.391 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:10.368+0000 7f55e0c89700 1 --2- 172.21.15.42:0/333541989 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f55dc0652f0 0x7f55dc14f200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:05:10.392 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:10.368+0000 7f55e0c89700 1 -- 172.21.15.42:0/333541989 >> v1:172.21.15.42:6789/0 conn(0x7f55dc0a8030 legacy=0x7f55dc144fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:05:10.392 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:10.368+0000 7f55e0c89700 1 -- 172.21.15.42:0/333541989 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55dc0a8460 con 0x7f55dc0a69f0 2019-12-05T05:05:10.392 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:10.368+0000 7f55e0c89700 1 --2- 172.21.15.42:0/333541989 >> v2:172.21.15.40:3300/0 conn(0x7f55dc0a69f0 0x7f55dc0a2840 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f55dc0a7630 tx=0x7f55d800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:05:10.396 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:10.368+0000 7f55e0c89700 1 --2- 172.21.15.42:0/333541989 >> v2:172.21.15.40:3300/0 conn(0x7f55dc0a69f0 0x7f55dc0a2840 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f55dc0a7630 tx=0x7f55d800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:05:10.396 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:10.368+0000 7f55e0c89700 1 --2- 172.21.15.42:0/333541989 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f55dc0a69f0 0x7f55dc0a2840 secure :-1 s=READY pgs=213 cs=0 l=1 rx=0x7f55dc0a7630 tx=0x7f55d800fc70).ready entity=mon.0 client_cookie=29b8a9138de6883b server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:05:10.396 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:10.368+0000 7f55e0c89700 1 --2- 172.21.15.42:0/333541989 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f55dc0a69f0 0x7f55dc0a2840 secure :-1 s=THROTTLE_DONE pgs=213 cs=0 l=1 rx=0x7f55dc0a7630 tx=0x7f55d800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:05:10.396 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:10.368+0000 7f55d37fe700 1 -- 172.21.15.42:0/333541989 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f55d80112b0 con 0x7f55dc0a69f0 2019-12-05T05:05:10.396 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:10.368+0000 7f55e0c89700 1 --2- 172.21.15.42:0/333541989 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f55dc0a69f0 0x7f55dc0a2840 secure :-1 s=THROTTLE_DONE pgs=213 cs=0 l=1 rx=0x7f55dc0a7630 tx=0x7f55d800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:05:10.397 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:10.368+0000 7f55d37fe700 1 -- 172.21.15.42:0/333541989 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55d8008ad0 con 0x7f55dc0a69f0 2019-12-05T05:05:10.397 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:10.368+0000 7f55e0c89700 1 --2- 172.21.15.42:0/333541989 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f55dc0a69f0 0x7f55dc0a2840 secure :-1 s=THROTTLE_DONE pgs=213 cs=0 l=1 rx=0x7f55dc0a7630 tx=0x7f55d800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:05:10.397 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:10.368+0000 7f55d37fe700 1 -- 172.21.15.42:0/333541989 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f55d8011890 con 0x7f55dc0a69f0 2019-12-05T05:05:10.397 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:10.368+0000 7f55e26ec700 1 -- 172.21.15.42:0/333541989 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f55dc0a69f0 msgr2=0x7f55dc0a2840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:05:10.397 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:10.368+0000 7f55e26ec700 1 --2- 172.21.15.42:0/333541989 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f55dc0a69f0 0x7f55dc0a2840 secure :-1 s=READY pgs=213 cs=0 l=1 rx=0x7f55dc0a7630 tx=0x7f55d800fc70).stop 2019-12-05T05:05:10.397 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:10.368+0000 7f55e26ec700 1 -- 172.21.15.42:0/333541989 shutdown_connections 2019-12-05T05:05:10.398 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:10.368+0000 7f55e26ec700 1 --2- 172.21.15.42:0/333541989 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f55dc0652f0 0x7f55dc14f200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:05:10.398 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:10.368+0000 7f55e26ec700 1 --2- 172.21.15.42:0/333541989 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f55dc0a69f0 0x7f55dc0a2840 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:05:10.398 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:10.368+0000 7f55e26ec700 1 -- 172.21.15.42:0/333541989 shutdown_connections 2019-12-05T05:05:10.399 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:10.368+0000 7f55e26ec700 1 -- 172.21.15.42:0/333541989 wait complete. 2019-12-05T05:05:10.399 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:10.368+0000 7f55e26ec700 1 -- 172.21.15.42:0/333541989 >> 172.21.15.42:0/333541989 conn(0x7f55dc059570 msgr2=0x7f55dc0599a0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:05:10.399 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:10.372+0000 7f55e26ec700 1 Processor -- start 2019-12-05T05:05:10.399 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:10.372+0000 7f55e26ec700 1 -- start start 2019-12-05T05:05:10.399 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:10.372+0000 7f55e26ec700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f55dc11f1a0 0x7f55dc1212d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:05:10.399 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:10.372+0000 7f55e26ec700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f55dc11f5d0 0x7f55dc121820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:05:10.399 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:10.372+0000 7f55e26ec700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55dc0a3620 con 0x7f55dc0a5070 2019-12-05T05:05:10.400 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:10.372+0000 7f55e26ec700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55dc12ff30 con 0x7f55dc11f5d0 2019-12-05T05:05:10.400 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:10.372+0000 7f55e26ec700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55dc1301d0 con 0x7f55dc11f1a0 2019-12-05T05:05:10.400 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:10.376+0000 7f55e0c89700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f55dc0a5070 0x7f55dc12f690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:54468/0 (socket says 172.21.15.42:54468) 2019-12-05T05:05:10.400 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:10.376+0000 7f55e0c89700 1 -- 172.21.15.42:0/2809578427 learned_addr learned my addr 172.21.15.42:0/2809578427 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:05:10.403 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:10.376+0000 7f55d3fff700 1 --2- 172.21.15.42:0/2809578427 >> v2:172.21.15.42:3300/0 conn(0x7f55dc11f1a0 0x7f55dc1212d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:05:10.403 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:10.376+0000 7f55e148a700 1 --2- 172.21.15.42:0/2809578427 >> v2:172.21.15.40:3300/0 conn(0x7f55dc11f5d0 0x7f55dc121820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:05:10.404 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:10.376+0000 7f55d17fa700 1 -- 172.21.15.42:0/2809578427 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1601776808 0 0) 0x7f55dc0a3620 con 0x7f55dc0a5070 2019-12-05T05:05:10.404 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:10.376+0000 7f55d17fa700 1 -- 172.21.15.42:0/2809578427 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55bc0030b0 con 0x7f55dc0a5070 2019-12-05T05:05:10.404 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:10.376+0000 7f55e148a700 1 -- 172.21.15.42:0/2809578427 >> v2:172.21.15.42:3300/0 conn(0x7f55dc11f1a0 msgr2=0x7f55dc1212d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:05:10.404 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:10.376+0000 7f55e148a700 1 --2- 172.21.15.42:0/2809578427 >> v2:172.21.15.42:3300/0 conn(0x7f55dc11f1a0 0x7f55dc1212d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:05:10.408 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:10.376+0000 7f55e148a700 1 -- 172.21.15.42:0/2809578427 >> v1:172.21.15.42:6789/0 conn(0x7f55dc0a5070 legacy=0x7f55dc12f690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:05:10.408 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:10.376+0000 7f55e148a700 1 -- 172.21.15.42:0/2809578427 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55dc0a95b0 con 0x7f55dc11f5d0 2019-12-05T05:05:10.409 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:10.376+0000 7f55d3fff700 1 --2- 172.21.15.42:0/2809578427 >> v2:172.21.15.42:3300/0 conn(0x7f55dc11f1a0 0x7f55dc1212d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:05:10.409 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:10.376+0000 7f55e148a700 1 --2- 172.21.15.42:0/2809578427 >> v2:172.21.15.40:3300/0 conn(0x7f55dc11f5d0 0x7f55dc121820 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f55dc068e20 tx=0x7f55d4010e10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:05:10.409 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:10.376+0000 7f55e148a700 1 --2- 172.21.15.42:0/2809578427 >> v2:172.21.15.40:3300/0 conn(0x7f55dc11f5d0 0x7f55dc121820 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f55dc068e20 tx=0x7f55d4010e10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:05:10.409 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:10.376+0000 7f55e148a700 1 --2- 172.21.15.42:0/2809578427 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f55dc11f5d0 0x7f55dc121820 secure :-1 s=READY pgs=214 cs=0 l=1 rx=0x7f55dc068e20 tx=0x7f55d4010e10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:05:10.410 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:10.376+0000 7f55e148a700 1 --2- 172.21.15.42:0/2809578427 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f55dc11f5d0 0x7f55dc121820 secure :-1 s=THROTTLE_DONE pgs=214 cs=0 l=1 rx=0x7f55dc068e20 tx=0x7f55d4010e10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:05:10.410 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:10.376+0000 7f55e148a700 1 --2- 172.21.15.42:0/2809578427 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f55dc11f5d0 0x7f55dc121820 secure :-1 s=THROTTLE_DONE pgs=214 cs=0 l=1 rx=0x7f55dc068e20 tx=0x7f55d4010e10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:05:10.410 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:10.376+0000 7f55d17fa700 1 -- 172.21.15.42:0/2809578427 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f55d4007350 con 0x7f55dc11f5d0 2019-12-05T05:05:10.410 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:10.376+0000 7f55e148a700 1 --2- 172.21.15.42:0/2809578427 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f55dc11f5d0 0x7f55dc121820 secure :-1 s=THROTTLE_DONE pgs=214 cs=0 l=1 rx=0x7f55dc068e20 tx=0x7f55d4010e10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:05:10.410 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:10.376+0000 7f55d17fa700 1 -- 172.21.15.42:0/2809578427 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55d4005d50 con 0x7f55dc11f5d0 2019-12-05T05:05:10.411 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:10.376+0000 7f55e26ec700 1 -- 172.21.15.42:0/2809578427 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55dc0a97c0 con 0x7f55dc11f5d0 2019-12-05T05:05:10.412 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:10.376+0000 7f55d17fa700 1 -- 172.21.15.42:0/2809578427 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f55d4012490 con 0x7f55dc11f5d0 2019-12-05T05:05:10.412 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:10.376+0000 7f55e26ec700 1 -- 172.21.15.42:0/2809578427 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55dc0a9b50 con 0x7f55dc11f5d0 2019-12-05T05:05:10.413 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:10.376+0000 7f55e148a700 1 --2- 172.21.15.42:0/2809578427 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f55dc11f5d0 0x7f55dc121820 secure :-1 s=THROTTLE_DONE pgs=214 cs=0 l=1 rx=0x7f55dc068e20 tx=0x7f55d4010e10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:05:10.413 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:10.380+0000 7f55e148a700 1 --2- 172.21.15.42:0/2809578427 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f55dc11f5d0 0x7f55dc121820 secure :-1 s=THROTTLE_DONE pgs=214 cs=0 l=1 rx=0x7f55dc068e20 tx=0x7f55d4010e10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:05:10.413 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:10.380+0000 7f55d17fa700 1 -- 172.21.15.42:0/2809578427 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f55d40127a0 con 0x7f55dc11f5d0 2019-12-05T05:05:10.413 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:10.380+0000 7f55d17fa700 1 --2- 172.21.15.42:0/2809578427 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f55bc03b2b0 0x7f55bc03d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:05:10.413 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:10.380+0000 7f55d17fa700 1 -- 172.21.15.42:0/2809578427 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6117+0+0 (secure 0 0 0) 0x7f55d404b560 con 0x7f55dc11f5d0 2019-12-05T05:05:10.413 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:10.380+0000 7f55e0c89700 1 --2- 172.21.15.42:0/2809578427 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f55bc03b2b0 0x7f55bc03d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:05:10.413 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:10.380+0000 7f55e0c89700 1 --2- 172.21.15.42:0/2809578427 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f55bc03b2b0 0x7f55bc03d7c0 crc :-1 s=READY pgs=131 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:05:10.414 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:10.380+0000 7f55e26ec700 1 -- 172.21.15.42:0/2809578427 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55dc05d2c0 con 0x7f55dc11f5d0 2019-12-05T05:05:10.414 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:10.392+0000 7f55e148a700 1 --2- 172.21.15.42:0/2809578427 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f55dc11f5d0 0x7f55dc121820 secure :-1 s=THROTTLE_DONE pgs=214 cs=0 l=1 rx=0x7f55dc068e20 tx=0x7f55d4010e10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:05:10.415 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:10.392+0000 7f55d17fa700 1 -- 172.21.15.42:0/2809578427 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f55d40325a0 con 0x7f55dc11f5d0 2019-12-05T05:05:10.578 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:10.560+0000 7f55e26ec700 1 -- 172.21.15.42:0/2809578427 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"var": "allow_new_snaps", "prefix": "fs set", "fs_name": "cephfs", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f55dc05ba70 con 0x7f55dc11f5d0 2019-12-05T05:05:11.207 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:11.204+0000 7f55e148a700 1 --2- 172.21.15.42:0/2809578427 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f55dc11f5d0 0x7f55dc121820 secure :-1 s=THROTTLE_DONE pgs=214 cs=0 l=1 rx=0x7f55dc068e20 tx=0x7f55d4010e10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:05:11.207 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:11.204+0000 7f55d17fa700 1 -- 172.21.15.42:0/2809578427 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"var": "allow_new_snaps", "prefix": "fs set", "fs_name": "cephfs", "val": "true", "yes_i_really_mean_it": true}]=0 enabled new snapshots v16) v1 ==== 167+0+0 (secure 0 0 0) 0x7f55d4032800 con 0x7f55dc11f5d0 2019-12-05T05:05:11.351 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:11.344+0000 7f55e26ec700 1 -- 172.21.15.42:0/2809578427 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"var": "allow_new_snaps", "prefix": "fs set", "fs_name": "cephfs", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f55dc0a0e20 con 0x7f55dc11f5d0 2019-12-05T05:05:12.224 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:12.220+0000 7f55e148a700 1 --2- 172.21.15.42:0/2809578427 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f55dc11f5d0 0x7f55dc121820 secure :-1 s=THROTTLE_DONE pgs=214 cs=0 l=1 rx=0x7f55dc068e20 tx=0x7f55d4010e10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:05:12.224 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:12.220+0000 7f55d17fa700 1 -- 172.21.15.42:0/2809578427 <== mon.0 v2:172.21.15.40:3300/0 8 ==== mon_command_ack([{"var": "allow_new_snaps", "prefix": "fs set", "fs_name": "cephfs", "val": "true", "yes_i_really_mean_it": true}]=0 enabled new snapshots v17) v1 ==== 167+0+0 (secure 0 0 0) 0x7f55d4032240 con 0x7f55dc11f5d0 2019-12-05T05:05:12.266 INFO:tasks.workunit.client.0.smithi042.stderr:enabled new snapshots 2019-12-05T05:05:12.272 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:12.268+0000 7f55e26ec700 1 -- 172.21.15.42:0/2809578427 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f55bc03b2b0 msgr2=0x7f55bc03d7c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:05:12.272 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:12.268+0000 7f55e26ec700 1 --2- 172.21.15.42:0/2809578427 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f55bc03b2b0 0x7f55bc03d7c0 crc :-1 s=READY pgs=131 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:05:12.272 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:12.268+0000 7f55e26ec700 1 -- 172.21.15.42:0/2809578427 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f55dc11f5d0 msgr2=0x7f55dc121820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:05:12.272 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:12.268+0000 7f55e26ec700 1 --2- 172.21.15.42:0/2809578427 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f55dc11f5d0 0x7f55dc121820 secure :-1 s=READY pgs=214 cs=0 l=1 rx=0x7f55dc068e20 tx=0x7f55d4010e10).stop 2019-12-05T05:05:12.272 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:12.268+0000 7f55e26ec700 1 -- 172.21.15.42:0/2809578427 shutdown_connections 2019-12-05T05:05:12.272 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:12.268+0000 7f55e26ec700 1 --2- 172.21.15.42:0/2809578427 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f55bc03b2b0 0x7f55bc03d7c0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:05:12.273 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:12.268+0000 7f55e26ec700 1 --2- 172.21.15.42:0/2809578427 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f55dc11f5d0 0x7f55dc121820 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:05:12.273 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:12.268+0000 7f55e26ec700 1 --2- 172.21.15.42:0/2809578427 >> v2:172.21.15.42:3300/0 conn(0x7f55dc11f1a0 0x7f55dc1212d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:05:12.273 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:12.268+0000 7f55e26ec700 1 -- 172.21.15.42:0/2809578427 shutdown_connections 2019-12-05T05:05:12.273 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:12.268+0000 7f55e26ec700 1 -- 172.21.15.42:0/2809578427 wait complete. 2019-12-05T05:05:12.273 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:12.268+0000 7f55e26ec700 1 -- 172.21.15.42:0/2809578427 >> 172.21.15.42:0/2809578427 conn(0x7f55dc0a11e0 msgr2=0x7f55dc123110 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:05:12.288 INFO:tasks.workunit.client.0.smithi042.stderr:+ echo asdf 2019-12-05T05:05:12.288 INFO:tasks.workunit.client.0.smithi042.stderr:+ mkdir .snap/foo 2019-12-05T05:05:12.303 INFO:tasks.workunit.client.0.smithi042.stderr:+ grep asdf .snap/foo/foo 2019-12-05T05:05:12.313 INFO:tasks.workunit.client.0.smithi042.stdout:asdf 2019-12-05T05:05:12.313 INFO:tasks.workunit.client.0.smithi042.stderr:+ rmdir .snap/foo 2019-12-05T05:05:13.119 INFO:tasks.workunit.client.0.smithi042.stderr:+ echo asdf 2019-12-05T05:05:13.120 INFO:tasks.workunit.client.0.smithi042.stderr:+ mkdir .snap/bar 2019-12-05T05:05:13.133 INFO:tasks.workunit.client.0.smithi042.stderr:+ rm bar 2019-12-05T05:05:13.136 INFO:tasks.workunit.client.0.smithi042.stderr:+ grep asdf .snap/bar/bar 2019-12-05T05:05:13.158 INFO:tasks.workunit.client.0.smithi042.stdout:asdf 2019-12-05T05:05:13.159 INFO:tasks.workunit.client.0.smithi042.stderr:+ rmdir .snap/bar 2019-12-05T05:05:13.159 INFO:tasks.workunit.client.0.smithi042.stderr:+ rm foo 2019-12-05T05:05:13.159 INFO:tasks.workunit.client.0.smithi042.stderr:+ ceph fs set cephfs allow_new_snaps false 2019-12-05T05:05:13.257 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:13.248+0000 7ff7c2f9a700 1 Processor -- start 2019-12-05T05:05:13.258 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:13.248+0000 7ff7c2f9a700 1 -- start start 2019-12-05T05:05:13.258 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:13.248+0000 7ff7c2f9a700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff7bc0a7160 0x7ff7bc0a29b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:05:13.258 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:13.248+0000 7ff7c2f9a700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7bc0a2f00 con 0x7ff7bc05a0b0 2019-12-05T05:05:13.258 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:13.248+0000 7ff7c2f9a700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7bc0a3620 con 0x7ff7bc1448a0 2019-12-05T05:05:13.258 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:13.248+0000 7ff7c2f9a700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7bc0a0d50 con 0x7ff7bc0a7160 2019-12-05T05:05:13.259 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:13.248+0000 7ff7c1d38700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7ff7bc1448a0 0x7ff7bc144cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:54486/0 (socket says 172.21.15.42:54486) 2019-12-05T05:05:13.259 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:13.248+0000 7ff7c1d38700 1 -- 172.21.15.42:0/1465734275 learned_addr learned my addr 172.21.15.42:0/1465734275 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:05:13.259 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:13.248+0000 7ff7c0d36700 1 --2- 172.21.15.42:0/1465734275 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff7bc0a7160 0x7ff7bc0a29b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:05:13.259 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:13.248+0000 7ff7b3fff700 1 -- 172.21.15.42:0/1465734275 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2228178280 0 0) 0x7ff7bc0a3620 con 0x7ff7bc1448a0 2019-12-05T05:05:13.259 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:13.248+0000 7ff7b3fff700 1 -- 172.21.15.42:0/1465734275 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7a80030b0 con 0x7ff7bc1448a0 2019-12-05T05:05:13.259 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:13.248+0000 7ff7b3fff700 1 -- 172.21.15.42:0/1465734275 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1594064460 0 0) 0x7ff7bc0a2f00 con 0x7ff7bc05a0b0 2019-12-05T05:05:13.260 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:13.248+0000 7ff7b3fff700 1 -- 172.21.15.42:0/1465734275 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7bc0a3620 con 0x7ff7bc05a0b0 2019-12-05T05:05:13.260 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:13.248+0000 7ff7c0d36700 1 -- 172.21.15.42:0/1465734275 >> v1:172.21.15.42:6789/0 conn(0x7ff7bc1448a0 legacy=0x7ff7bc144cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:05:13.260 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:13.248+0000 7ff7c0d36700 1 -- 172.21.15.42:0/1465734275 >> v1:172.21.15.40:6789/0 conn(0x7ff7bc05a0b0 legacy=0x7ff7bc0a6960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:05:13.260 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:13.248+0000 7ff7c0d36700 1 -- 172.21.15.42:0/1465734275 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7bc1454d0 con 0x7ff7bc0a7160 2019-12-05T05:05:13.260 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:13.248+0000 7ff7c0d36700 1 --2- 172.21.15.42:0/1465734275 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff7bc0a7160 0x7ff7bc0a29b0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7ff7ac008790 tx=0x7ff7ac00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:05:13.260 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:13.248+0000 7ff7c0d36700 1 --2- 172.21.15.42:0/1465734275 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff7bc0a7160 0x7ff7bc0a29b0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7ff7ac008790 tx=0x7ff7ac00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:05:13.261 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:13.248+0000 7ff7c0d36700 1 --2- 172.21.15.42:0/1465734275 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff7bc0a7160 0x7ff7bc0a29b0 secure :-1 s=READY pgs=54 cs=0 l=1 rx=0x7ff7ac008790 tx=0x7ff7ac00fbf0).ready entity=mon.2 client_cookie=3e51c14fd98e9459 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:05:13.261 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:13.248+0000 7ff7c0d36700 1 --2- 172.21.15.42:0/1465734275 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff7bc0a7160 0x7ff7bc0a29b0 secure :-1 s=THROTTLE_DONE pgs=54 cs=0 l=1 rx=0x7ff7ac008790 tx=0x7ff7ac00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:05:13.261 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:13.248+0000 7ff7b3fff700 1 -- 172.21.15.42:0/1465734275 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff7ac011270 con 0x7ff7bc0a7160 2019-12-05T05:05:13.261 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:13.248+0000 7ff7c0d36700 1 --2- 172.21.15.42:0/1465734275 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff7bc0a7160 0x7ff7bc0a29b0 secure :-1 s=THROTTLE_DONE pgs=54 cs=0 l=1 rx=0x7ff7ac008790 tx=0x7ff7ac00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:05:13.261 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:13.248+0000 7ff7c0d36700 1 --2- 172.21.15.42:0/1465734275 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff7bc0a7160 0x7ff7bc0a29b0 secure :-1 s=THROTTLE_DONE pgs=54 cs=0 l=1 rx=0x7ff7ac008790 tx=0x7ff7ac00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:05:13.261 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:13.248+0000 7ff7b3fff700 1 -- 172.21.15.42:0/1465734275 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7ac008bc0 con 0x7ff7bc0a7160 2019-12-05T05:05:13.262 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:13.248+0000 7ff7b3fff700 1 -- 172.21.15.42:0/1465734275 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff7ac011850 con 0x7ff7bc0a7160 2019-12-05T05:05:13.262 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:13.248+0000 7ff7c2f9a700 1 -- 172.21.15.42:0/1465734275 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff7bc0a7160 msgr2=0x7ff7bc0a29b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:05:13.263 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:13.248+0000 7ff7c2f9a700 1 --2- 172.21.15.42:0/1465734275 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff7bc0a7160 0x7ff7bc0a29b0 secure :-1 s=READY pgs=54 cs=0 l=1 rx=0x7ff7ac008790 tx=0x7ff7ac00fbf0).stop 2019-12-05T05:05:13.263 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:13.252+0000 7ff7c2f9a700 1 -- 172.21.15.42:0/1465734275 shutdown_connections 2019-12-05T05:05:13.263 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:13.252+0000 7ff7c2f9a700 1 --2- 172.21.15.42:0/1465734275 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff7bc0a7160 0x7ff7bc0a29b0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:05:13.263 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:13.252+0000 7ff7c2f9a700 1 -- 172.21.15.42:0/1465734275 shutdown_connections 2019-12-05T05:05:13.263 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:13.252+0000 7ff7c2f9a700 1 -- 172.21.15.42:0/1465734275 wait complete. 2019-12-05T05:05:13.263 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:13.252+0000 7ff7c2f9a700 1 -- 172.21.15.42:0/1465734275 >> 172.21.15.42:0/1465734275 conn(0x7ff7bc05ee60 msgr2=0x7ff7bc059570 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:05:13.264 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:13.252+0000 7ff7c2f9a700 1 Processor -- start 2019-12-05T05:05:13.264 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:13.252+0000 7ff7c2f9a700 1 -- start start 2019-12-05T05:05:13.270 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:13.252+0000 7ff7c2f9a700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff7bc0a45d0 0x7ff7bc12a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:05:13.270 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:13.252+0000 7ff7c2f9a700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7bc12a750 con 0x7ff7bc0a6150 2019-12-05T05:05:13.270 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:13.252+0000 7ff7c2f9a700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7bc14fd10 con 0x7ff7bc0a5710 2019-12-05T05:05:13.270 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:13.252+0000 7ff7c2f9a700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7bc0a51d0 con 0x7ff7bc0a45d0 2019-12-05T05:05:13.271 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:13.252+0000 7ff7c1d38700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7ff7bc0a5710 0x7ff7bc14f510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:54492/0 (socket says 172.21.15.42:54492) 2019-12-05T05:05:13.271 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:13.252+0000 7ff7c1d38700 1 -- 172.21.15.42:0/1149341690 learned_addr learned my addr 172.21.15.42:0/1149341690 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:05:13.272 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:13.252+0000 7ff7c0d36700 1 --2- 172.21.15.42:0/1149341690 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff7bc0a45d0 0x7ff7bc12a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:05:13.272 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:13.252+0000 7ff7b1ffb700 1 -- 172.21.15.42:0/1149341690 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2153594717 0 0) 0x7ff7bc14fd10 con 0x7ff7bc0a5710 2019-12-05T05:05:13.272 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:13.252+0000 7ff7b1ffb700 1 -- 172.21.15.42:0/1149341690 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff79c0030b0 con 0x7ff7bc0a5710 2019-12-05T05:05:13.272 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:13.252+0000 7ff7b1ffb700 1 -- 172.21.15.42:0/1149341690 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1565517454 0 0) 0x7ff7bc12a750 con 0x7ff7bc0a6150 2019-12-05T05:05:13.272 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:13.252+0000 7ff7b1ffb700 1 -- 172.21.15.42:0/1149341690 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7bc14fd10 con 0x7ff7bc0a6150 2019-12-05T05:05:13.273 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:13.256+0000 7ff7c0d36700 1 -- 172.21.15.42:0/1149341690 >> v1:172.21.15.42:6789/0 conn(0x7ff7bc0a5710 legacy=0x7ff7bc14f510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:05:13.273 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:13.256+0000 7ff7c0d36700 1 -- 172.21.15.42:0/1149341690 >> v1:172.21.15.40:6789/0 conn(0x7ff7bc0a6150 legacy=0x7ff7bc143c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:05:13.273 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:13.256+0000 7ff7c0d36700 1 -- 172.21.15.42:0/1149341690 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7bc134680 con 0x7ff7bc0a45d0 2019-12-05T05:05:13.274 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:13.256+0000 7ff7c0d36700 1 --2- 172.21.15.42:0/1149341690 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff7bc0a45d0 0x7ff7bc12a200 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7ff7ac008790 tx=0x7ff7ac008bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:05:13.274 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:13.256+0000 7ff7c0d36700 1 --2- 172.21.15.42:0/1149341690 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff7bc0a45d0 0x7ff7bc12a200 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7ff7ac008790 tx=0x7ff7ac008bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:05:13.274 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:13.256+0000 7ff7c0d36700 1 --2- 172.21.15.42:0/1149341690 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff7bc0a45d0 0x7ff7bc12a200 secure :-1 s=READY pgs=55 cs=0 l=1 rx=0x7ff7ac008790 tx=0x7ff7ac008bc0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:05:13.274 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:13.256+0000 7ff7c0d36700 1 --2- 172.21.15.42:0/1149341690 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff7bc0a45d0 0x7ff7bc12a200 secure :-1 s=THROTTLE_DONE pgs=55 cs=0 l=1 rx=0x7ff7ac008790 tx=0x7ff7ac008bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:05:13.275 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:13.256+0000 7ff7b1ffb700 1 -- 172.21.15.42:0/1149341690 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff7ac007150 con 0x7ff7bc0a45d0 2019-12-05T05:05:13.275 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:13.256+0000 7ff7c0d36700 1 --2- 172.21.15.42:0/1149341690 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff7bc0a45d0 0x7ff7bc12a200 secure :-1 s=THROTTLE_DONE pgs=55 cs=0 l=1 rx=0x7ff7ac008790 tx=0x7ff7ac008bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:05:13.275 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:13.256+0000 7ff7c0d36700 1 --2- 172.21.15.42:0/1149341690 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff7bc0a45d0 0x7ff7bc12a200 secure :-1 s=THROTTLE_DONE pgs=55 cs=0 l=1 rx=0x7ff7ac008790 tx=0x7ff7ac008bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:05:13.275 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:13.256+0000 7ff7b1ffb700 1 -- 172.21.15.42:0/1149341690 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7ac011470 con 0x7ff7bc0a45d0 2019-12-05T05:05:13.275 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:13.256+0000 7ff7c2f9a700 1 -- 172.21.15.42:0/1149341690 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7bc132660 con 0x7ff7bc0a45d0 2019-12-05T05:05:13.275 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:13.256+0000 7ff7b1ffb700 1 -- 172.21.15.42:0/1149341690 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff7ac00f550 con 0x7ff7bc0a45d0 2019-12-05T05:05:13.276 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:13.256+0000 7ff7c2f9a700 1 -- 172.21.15.42:0/1149341690 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7bc132bb0 con 0x7ff7bc0a45d0 2019-12-05T05:05:13.276 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:13.256+0000 7ff7c0d36700 1 --2- 172.21.15.42:0/1149341690 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff7bc0a45d0 0x7ff7bc12a200 secure :-1 s=THROTTLE_DONE pgs=55 cs=0 l=1 rx=0x7ff7ac008790 tx=0x7ff7ac008bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:05:13.276 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:13.256+0000 7ff7c0d36700 1 --2- 172.21.15.42:0/1149341690 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff7bc0a45d0 0x7ff7bc12a200 secure :-1 s=THROTTLE_DONE pgs=55 cs=0 l=1 rx=0x7ff7ac008790 tx=0x7ff7ac008bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:05:13.276 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:13.256+0000 7ff7b1ffb700 1 -- 172.21.15.42:0/1149341690 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7ff7ac005540 con 0x7ff7bc0a45d0 2019-12-05T05:05:13.276 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:13.256+0000 7ff7b1ffb700 1 --2- 172.21.15.42:0/1149341690 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff79c03b4b0 0x7ff79c03d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:05:13.276 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:13.256+0000 7ff7b1ffb700 1 -- 172.21.15.42:0/1149341690 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6173+0+0 (secure 0 0 0) 0x7ff7ac04b8f0 con 0x7ff7bc0a45d0 2019-12-05T05:05:13.276 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:13.256+0000 7ff7c1537700 1 --2- 172.21.15.42:0/1149341690 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff79c03b4b0 0x7ff79c03d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:05:13.277 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:13.256+0000 7ff7c1537700 1 --2- 172.21.15.42:0/1149341690 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff79c03b4b0 0x7ff79c03d9c0 crc :-1 s=READY pgs=132 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:05:13.277 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:13.264+0000 7ff7c2f9a700 1 -- 172.21.15.42:0/1149341690 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7bc05d2c0 con 0x7ff7bc0a45d0 2019-12-05T05:05:13.277 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:13.272+0000 7ff7c0d36700 1 --2- 172.21.15.42:0/1149341690 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff7bc0a45d0 0x7ff7bc12a200 secure :-1 s=THROTTLE_DONE pgs=55 cs=0 l=1 rx=0x7ff7ac008790 tx=0x7ff7ac008bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:05:13.278 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:13.272+0000 7ff7b1ffb700 1 -- 172.21.15.42:0/1149341690 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7ff7ac008dc0 con 0x7ff7bc0a45d0 2019-12-05T05:05:13.448 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:13.444+0000 7ff7c2f9a700 1 -- 172.21.15.42:0/1149341690 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"var": "allow_new_snaps", "prefix": "fs set", "fs_name": "cephfs", "val": "false"} v 0) v1 -- 0x7ff7bc1444c0 con 0x7ff7bc0a45d0 2019-12-05T05:05:14.246 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:14.240+0000 7ff7c0d36700 1 --2- 172.21.15.42:0/1149341690 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff7bc0a45d0 0x7ff7bc12a200 secure :-1 s=THROTTLE_DONE pgs=55 cs=0 l=1 rx=0x7ff7ac008790 tx=0x7ff7ac008bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:05:14.246 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:14.240+0000 7ff7b1ffb700 1 -- 172.21.15.42:0/1149341690 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"var": "allow_new_snaps", "prefix": "fs set", "fs_name": "cephfs", "val": "false"}]=0 disabled new snapshots v18) v1 ==== 139+0+0 (secure 0 0 0) 0x7ff7ac032b10 con 0x7ff7bc0a45d0 2019-12-05T05:05:14.377 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:14.372+0000 7ff7c2f9a700 1 -- 172.21.15.42:0/1149341690 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"var": "allow_new_snaps", "prefix": "fs set", "fs_name": "cephfs", "val": "false"} v 0) v1 -- 0x7ff7bc0a4f10 con 0x7ff7bc0a45d0 2019-12-05T05:05:15.248 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:15.244+0000 7ff7c0d36700 1 --2- 172.21.15.42:0/1149341690 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff7bc0a45d0 0x7ff7bc12a200 secure :-1 s=THROTTLE_DONE pgs=55 cs=0 l=1 rx=0x7ff7ac008790 tx=0x7ff7ac008bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:05:15.249 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:15.244+0000 7ff7b1ffb700 1 -- 172.21.15.42:0/1149341690 <== mon.2 v2:172.21.15.40:3301/0 8 ==== mon_command_ack([{"var": "allow_new_snaps", "prefix": "fs set", "fs_name": "cephfs", "val": "false"}]=0 disabled new snapshots v19) v1 ==== 139+0+0 (secure 0 0 0) 0x7ff7ac0336c0 con 0x7ff7bc0a45d0 2019-12-05T05:05:15.292 INFO:tasks.workunit.client.0.smithi042.stderr:disabled new snapshots 2019-12-05T05:05:15.297 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:15.292+0000 7ff7c2f9a700 1 -- 172.21.15.42:0/1149341690 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff79c03b4b0 msgr2=0x7ff79c03d9c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:05:15.298 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:15.292+0000 7ff7c2f9a700 1 --2- 172.21.15.42:0/1149341690 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff79c03b4b0 0x7ff79c03d9c0 crc :-1 s=READY pgs=132 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:05:15.298 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:15.292+0000 7ff7c2f9a700 1 -- 172.21.15.42:0/1149341690 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff7bc0a45d0 msgr2=0x7ff7bc12a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:05:15.298 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:15.292+0000 7ff7c2f9a700 1 --2- 172.21.15.42:0/1149341690 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff7bc0a45d0 0x7ff7bc12a200 secure :-1 s=READY pgs=55 cs=0 l=1 rx=0x7ff7ac008790 tx=0x7ff7ac008bc0).stop 2019-12-05T05:05:15.298 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:15.292+0000 7ff7c2f9a700 1 -- 172.21.15.42:0/1149341690 shutdown_connections 2019-12-05T05:05:15.299 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:15.292+0000 7ff7c2f9a700 1 --2- 172.21.15.42:0/1149341690 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff79c03b4b0 0x7ff79c03d9c0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:05:15.299 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:15.292+0000 7ff7c2f9a700 1 --2- 172.21.15.42:0/1149341690 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff7bc0a45d0 0x7ff7bc12a200 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:05:15.299 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:15.292+0000 7ff7c2f9a700 1 -- 172.21.15.42:0/1149341690 shutdown_connections 2019-12-05T05:05:15.299 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:15.292+0000 7ff7c2f9a700 1 -- 172.21.15.42:0/1149341690 wait complete. 2019-12-05T05:05:15.299 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:15.292+0000 7ff7c2f9a700 1 -- 172.21.15.42:0/1149341690 >> 172.21.15.42:0/1149341690 conn(0x7ff7bc0a2840 msgr2=0x7ff7bc0a6cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:05:15.325 INFO:tasks.workunit.client.0.smithi042.stderr:+ expect_failure mkdir .snap/baz 2019-12-05T05:05:15.326 INFO:tasks.workunit.client.0.smithi042.stderr:+ mkdir .snap/baz 2019-12-05T05:05:15.326 INFO:tasks.workunit.client.0.smithi042.stderr:mkdir: cannot create directory '.snap/baz': Operation not permitted 2019-12-05T05:05:15.326 INFO:tasks.workunit.client.0.smithi042.stderr:+ return 0 2019-12-05T05:05:15.326 INFO:tasks.workunit.client.0.smithi042.stderr:+ echo OK 2019-12-05T05:05:15.326 INFO:tasks.workunit.client.0.smithi042.stdout:OK 2019-12-05T05:05:15.327 INFO:teuthology.orchestra.run:Running command with timeout 3600 2019-12-05T05:05:15.327 INFO:teuthology.orchestra.run.smithi042:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2019-12-05T05:05:15.433 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-1.sh... 2019-12-05T05:05:15.433 INFO:teuthology.orchestra.run.smithi042:workunit test fs/snaps/snaptest-1.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c5290805033483ba0cec165c06bde07dae5f19ec TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-1.sh 2019-12-05T05:05:15.525 INFO:tasks.workunit.client.0.smithi042.stderr:+ ceph fs set cephfs allow_new_snaps true --yes-i-really-mean-it 2019-12-05T05:05:15.640 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:15.628+0000 7f076f02b700 1 Processor -- start 2019-12-05T05:05:15.640 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:15.632+0000 7f076f02b700 1 -- start start 2019-12-05T05:05:15.640 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:15.632+0000 7f076f02b700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f07680a2840 0x7f07680a8030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:05:15.640 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:15.632+0000 7f076f02b700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07680a7b60 con 0x7f07680652f0 2019-12-05T05:05:15.640 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:15.632+0000 7f076f02b700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07680a2c70 con 0x7f07680a69f0 2019-12-05T05:05:15.641 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:15.632+0000 7f076f02b700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0768065b20 con 0x7f07680a2840 2019-12-05T05:05:15.641 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:15.632+0000 7f076d5c8700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f07680a69f0 0x7f0768144fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:54498/0 (socket says 172.21.15.42:54498) 2019-12-05T05:05:15.641 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:15.632+0000 7f076d5c8700 1 -- 172.21.15.42:0/320755692 learned_addr learned my addr 172.21.15.42:0/320755692 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:05:15.641 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:15.632+0000 7f076cdc7700 1 --2- 172.21.15.42:0/320755692 >> v2:172.21.15.40:3300/0 conn(0x7f07680a2840 0x7f07680a8030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:05:15.641 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:15.632+0000 7f0767fff700 1 -- 172.21.15.42:0/320755692 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2290841450 0 0) 0x7f07680a2c70 con 0x7f07680a69f0 2019-12-05T05:05:15.641 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:15.632+0000 7f0767fff700 1 -- 172.21.15.42:0/320755692 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07500030b0 con 0x7f07680a69f0 2019-12-05T05:05:15.641 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:15.632+0000 7f076cdc7700 1 -- 172.21.15.42:0/320755692 >> v1:172.21.15.42:6789/0 conn(0x7f07680a69f0 legacy=0x7f0768144fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:05:15.642 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:15.632+0000 7f076cdc7700 1 -- 172.21.15.42:0/320755692 >> v1:172.21.15.40:6789/0 conn(0x7f07680652f0 legacy=0x7f076814f200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:05:15.642 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:15.632+0000 7f076cdc7700 1 -- 172.21.15.42:0/320755692 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07680a21e0 con 0x7f07680a2840 2019-12-05T05:05:15.642 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:15.632+0000 7f076cdc7700 1 --2- 172.21.15.42:0/320755692 >> v2:172.21.15.40:3300/0 conn(0x7f07680a2840 0x7f07680a8030 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f075c008790 tx=0x7f075c00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:05:15.642 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:15.632+0000 7f076cdc7700 1 --2- 172.21.15.42:0/320755692 >> v2:172.21.15.40:3300/0 conn(0x7f07680a2840 0x7f07680a8030 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f075c008790 tx=0x7f075c00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:05:15.642 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:15.632+0000 7f076cdc7700 1 --2- 172.21.15.42:0/320755692 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f07680a2840 0x7f07680a8030 secure :-1 s=READY pgs=218 cs=0 l=1 rx=0x7f075c008790 tx=0x7f075c00fbf0).ready entity=mon.0 client_cookie=f4b2d0fbbca529fb server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:05:15.642 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:15.632+0000 7f076cdc7700 1 --2- 172.21.15.42:0/320755692 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f07680a2840 0x7f07680a8030 secure :-1 s=THROTTLE_DONE pgs=218 cs=0 l=1 rx=0x7f075c008790 tx=0x7f075c00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:05:15.642 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:15.632+0000 7f0767fff700 1 -- 172.21.15.42:0/320755692 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f075c011230 con 0x7f07680a2840 2019-12-05T05:05:15.643 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:15.632+0000 7f076cdc7700 1 --2- 172.21.15.42:0/320755692 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f07680a2840 0x7f07680a8030 secure :-1 s=THROTTLE_DONE pgs=218 cs=0 l=1 rx=0x7f075c008790 tx=0x7f075c00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:05:15.643 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:15.632+0000 7f076cdc7700 1 --2- 172.21.15.42:0/320755692 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f07680a2840 0x7f07680a8030 secure :-1 s=THROTTLE_DONE pgs=218 cs=0 l=1 rx=0x7f075c008790 tx=0x7f075c00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:05:15.643 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:15.632+0000 7f0767fff700 1 -- 172.21.15.42:0/320755692 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f075c008bc0 con 0x7f07680a2840 2019-12-05T05:05:15.643 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:15.632+0000 7f0767fff700 1 -- 172.21.15.42:0/320755692 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f075c011810 con 0x7f07680a2840 2019-12-05T05:05:15.643 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:15.632+0000 7f076f02b700 1 -- 172.21.15.42:0/320755692 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f07680a2840 msgr2=0x7f07680a8030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:05:15.643 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:15.632+0000 7f076f02b700 1 --2- 172.21.15.42:0/320755692 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f07680a2840 0x7f07680a8030 secure :-1 s=READY pgs=218 cs=0 l=1 rx=0x7f075c008790 tx=0x7f075c00fbf0).stop 2019-12-05T05:05:15.643 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:15.636+0000 7f076f02b700 1 -- 172.21.15.42:0/320755692 shutdown_connections 2019-12-05T05:05:15.644 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:15.636+0000 7f076f02b700 1 --2- 172.21.15.42:0/320755692 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f07680a2840 0x7f07680a8030 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:05:15.644 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:15.636+0000 7f076f02b700 1 -- 172.21.15.42:0/320755692 shutdown_connections 2019-12-05T05:05:15.644 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:15.636+0000 7f076f02b700 1 -- 172.21.15.42:0/320755692 wait complete. 2019-12-05T05:05:15.644 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:15.636+0000 7f076f02b700 1 -- 172.21.15.42:0/320755692 >> 172.21.15.42:0/320755692 conn(0x7f0768059570 msgr2=0x7f07680599a0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:05:15.647 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:15.636+0000 7f076f02b700 1 Processor -- start 2019-12-05T05:05:15.647 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:15.636+0000 7f076f02b700 1 -- start start 2019-12-05T05:05:15.647 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:15.636+0000 7f076f02b700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f076807db80 0x7f0768063710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:05:15.647 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:15.636+0000 7f076f02b700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f07680a9db0 0x7f0768132ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:05:15.648 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:15.636+0000 7f076f02b700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f076811ca90 con 0x7f0768064120 2019-12-05T05:05:15.648 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:15.636+0000 7f076f02b700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f076811bf40 con 0x7f076807db80 2019-12-05T05:05:15.648 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:15.636+0000 7f076f02b700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f076811c1e0 con 0x7f07680a9db0 2019-12-05T05:05:15.648 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:15.636+0000 7f076ddc9700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f07680a9db0 0x7f0768132ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:05:15.648 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:15.636+0000 7f076ddc9700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f07680a9db0 0x7f0768132ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:54468/0 (socket says 172.21.15.42:54468) 2019-12-05T05:05:15.648 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:15.636+0000 7f076ddc9700 1 -- 172.21.15.42:0/1573536173 learned_addr learned my addr 172.21.15.42:0/1573536173 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:05:15.649 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:15.636+0000 7f076d5c8700 1 --2- 172.21.15.42:0/1573536173 >> v2:172.21.15.40:3300/0 conn(0x7f076807db80 0x7f0768063710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:05:15.649 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:15.640+0000 7f076ddc9700 1 -- 172.21.15.42:0/1573536173 >> v2:172.21.15.40:3300/0 conn(0x7f076807db80 msgr2=0x7f0768063710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:05:15.649 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:15.640+0000 7f076ddc9700 1 --2- 172.21.15.42:0/1573536173 >> v2:172.21.15.40:3300/0 conn(0x7f076807db80 0x7f0768063710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:05:15.649 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:15.640+0000 7f0765ffb700 1 -- 172.21.15.42:0/1573536173 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2793046712 0 0) 0x7f076811ca90 con 0x7f0768064120 2019-12-05T05:05:15.649 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:15.640+0000 7f076ddc9700 1 -- 172.21.15.42:0/1573536173 >> v1:172.21.15.40:6789/0 conn(0x7f0768064120 legacy=0x7f07680a95b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:05:15.649 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:15.640+0000 7f076ddc9700 1 -- 172.21.15.42:0/1573536173 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0768077680 con 0x7f07680a9db0 2019-12-05T05:05:15.649 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:15.640+0000 7f076ddc9700 1 --2- 172.21.15.42:0/1573536173 >> v2:172.21.15.42:3300/0 conn(0x7f07680a9db0 0x7f0768132ad0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f0758000e90 tx=0x7f07580107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:05:15.650 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:15.640+0000 7f076ddc9700 1 --2- 172.21.15.42:0/1573536173 >> v2:172.21.15.42:3300/0 conn(0x7f07680a9db0 0x7f0768132ad0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f0758000e90 tx=0x7f07580107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:05:15.650 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:15.640+0000 7f076ddc9700 1 --2- 172.21.15.42:0/1573536173 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f07680a9db0 0x7f0768132ad0 secure :-1 s=READY pgs=273 cs=0 l=1 rx=0x7f0758000e90 tx=0x7f07580107c0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:05:15.650 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:15.640+0000 7f076ddc9700 1 --2- 172.21.15.42:0/1573536173 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f07680a9db0 0x7f0768132ad0 secure :-1 s=THROTTLE_DONE pgs=273 cs=0 l=1 rx=0x7f0758000e90 tx=0x7f07580107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:05:15.650 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:15.640+0000 7f0765ffb700 1 -- 172.21.15.42:0/1573536173 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0758006a40 con 0x7f07680a9db0 2019-12-05T05:05:15.650 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:15.640+0000 7f076ddc9700 1 --2- 172.21.15.42:0/1573536173 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f07680a9db0 0x7f0768132ad0 secure :-1 s=THROTTLE_DONE pgs=273 cs=0 l=1 rx=0x7f0758000e90 tx=0x7f07580107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:05:15.650 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:15.640+0000 7f076ddc9700 1 --2- 172.21.15.42:0/1573536173 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f07680a9db0 0x7f0768132ad0 secure :-1 s=THROTTLE_DONE pgs=273 cs=0 l=1 rx=0x7f0758000e90 tx=0x7f07580107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:05:15.650 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:15.640+0000 7f0765ffb700 1 -- 172.21.15.42:0/1573536173 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0758006660 con 0x7f07680a9db0 2019-12-05T05:05:15.651 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:15.640+0000 7f076f02b700 1 -- 172.21.15.42:0/1573536173 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0768063c60 con 0x7f07680a9db0 2019-12-05T05:05:15.651 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:15.640+0000 7f0765ffb700 1 -- 172.21.15.42:0/1573536173 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0758011000 con 0x7f07680a9db0 2019-12-05T05:05:15.651 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:15.640+0000 7f076f02b700 1 -- 172.21.15.42:0/1573536173 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f076811c600 con 0x7f07680a9db0 2019-12-05T05:05:15.651 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:15.640+0000 7f076ddc9700 1 --2- 172.21.15.42:0/1573536173 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f07680a9db0 0x7f0768132ad0 secure :-1 s=THROTTLE_DONE pgs=273 cs=0 l=1 rx=0x7f0758000e90 tx=0x7f07580107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:05:15.651 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:15.640+0000 7f0765ffb700 1 -- 172.21.15.42:0/1573536173 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f07580112d0 con 0x7f07680a9db0 2019-12-05T05:05:15.651 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:15.640+0000 7f076ddc9700 1 --2- 172.21.15.42:0/1573536173 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f07680a9db0 0x7f0768132ad0 secure :-1 s=THROTTLE_DONE pgs=273 cs=0 l=1 rx=0x7f0758000e90 tx=0x7f07580107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:05:15.651 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:15.640+0000 7f0765ffb700 1 --2- 172.21.15.42:0/1573536173 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f074803acf0 0x7f074803d200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:05:15.652 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:15.640+0000 7f0765ffb700 1 -- 172.21.15.42:0/1573536173 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6173+0+0 (secure 0 0 0) 0x7f075804b060 con 0x7f07680a9db0 2019-12-05T05:05:15.652 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:15.640+0000 7f076d5c8700 1 --2- 172.21.15.42:0/1573536173 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f074803acf0 0x7f074803d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:05:15.652 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:15.644+0000 7f076d5c8700 1 --2- 172.21.15.42:0/1573536173 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f074803acf0 0x7f074803d200 crc :-1 s=READY pgs=133 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:05:15.652 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:15.648+0000 7f076f02b700 1 -- 172.21.15.42:0/1573536173 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f076811c3a0 con 0x7f07680a9db0 2019-12-05T05:05:15.659 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:15.652+0000 7f076ddc9700 1 --2- 172.21.15.42:0/1573536173 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f07680a9db0 0x7f0768132ad0 secure :-1 s=THROTTLE_DONE pgs=273 cs=0 l=1 rx=0x7f0758000e90 tx=0x7f07580107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:05:15.659 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:15.652+0000 7f0765ffb700 1 -- 172.21.15.42:0/1573536173 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f07580320e0 con 0x7f07680a9db0 2019-12-05T05:05:15.829 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:15.824+0000 7f076f02b700 1 -- 172.21.15.42:0/1573536173 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"var": "allow_new_snaps", "prefix": "fs set", "fs_name": "cephfs", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f076805d2c0 con 0x7f07680a9db0 2019-12-05T05:05:16.264 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:16.260+0000 7f076ddc9700 1 --2- 172.21.15.42:0/1573536173 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f07680a9db0 0x7f0768132ad0 secure :-1 s=THROTTLE_DONE pgs=273 cs=0 l=1 rx=0x7f0758000e90 tx=0x7f07580107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:05:16.265 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:16.260+0000 7f0765ffb700 1 -- 172.21.15.42:0/1573536173 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"var": "allow_new_snaps", "prefix": "fs set", "fs_name": "cephfs", "val": "true", "yes_i_really_mean_it": true}]=0 enabled new snapshots v20) v1 ==== 167+0+0 (secure 0 0 0) 0x7f0758032340 con 0x7f07680a9db0 2019-12-05T05:05:16.412 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:16.408+0000 7f076f02b700 1 -- 172.21.15.42:0/1573536173 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"var": "allow_new_snaps", "prefix": "fs set", "fs_name": "cephfs", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f076805ba70 con 0x7f07680a9db0 2019-12-05T05:05:17.269 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:17.264+0000 7f076ddc9700 1 --2- 172.21.15.42:0/1573536173 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f07680a9db0 0x7f0768132ad0 secure :-1 s=THROTTLE_DONE pgs=273 cs=0 l=1 rx=0x7f0758000e90 tx=0x7f07580107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:05:17.269 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:17.264+0000 7f0765ffb700 1 -- 172.21.15.42:0/1573536173 <== mon.1 v2:172.21.15.42:3300/0 8 ==== mon_command_ack([{"var": "allow_new_snaps", "prefix": "fs set", "fs_name": "cephfs", "val": "true", "yes_i_really_mean_it": true}]=0 enabled new snapshots v21) v1 ==== 167+0+0 (secure 0 0 0) 0x7f0758032ef0 con 0x7f07680a9db0 2019-12-05T05:05:17.277 INFO:tasks.workunit.client.0.smithi042.stderr:enabled new snapshots 2019-12-05T05:05:17.282 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:17.276+0000 7f076f02b700 1 -- 172.21.15.42:0/1573536173 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f074803acf0 msgr2=0x7f074803d200 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:05:17.282 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:17.276+0000 7f076f02b700 1 --2- 172.21.15.42:0/1573536173 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f074803acf0 0x7f074803d200 crc :-1 s=READY pgs=133 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:05:17.283 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:17.276+0000 7f076f02b700 1 -- 172.21.15.42:0/1573536173 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f07680a9db0 msgr2=0x7f0768132ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:05:17.283 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:17.276+0000 7f076f02b700 1 --2- 172.21.15.42:0/1573536173 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f07680a9db0 0x7f0768132ad0 secure :-1 s=READY pgs=273 cs=0 l=1 rx=0x7f0758000e90 tx=0x7f07580107c0).stop 2019-12-05T05:05:17.283 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:17.276+0000 7f076f02b700 1 -- 172.21.15.42:0/1573536173 shutdown_connections 2019-12-05T05:05:17.283 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:17.276+0000 7f076f02b700 1 --2- 172.21.15.42:0/1573536173 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f074803acf0 0x7f074803d200 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:05:17.284 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:17.276+0000 7f076f02b700 1 --2- 172.21.15.42:0/1573536173 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f07680a9db0 0x7f0768132ad0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:05:17.284 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:17.276+0000 7f076f02b700 1 --2- 172.21.15.42:0/1573536173 >> v2:172.21.15.40:3300/0 conn(0x7f076807db80 0x7f0768063710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:05:17.284 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:17.276+0000 7f076f02b700 1 -- 172.21.15.42:0/1573536173 shutdown_connections 2019-12-05T05:05:17.286 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:17.276+0000 7f076f02b700 1 -- 172.21.15.42:0/1573536173 wait complete. 2019-12-05T05:05:17.286 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:17.276+0000 7f076f02b700 1 -- 172.21.15.42:0/1573536173 >> 172.21.15.42:0/1573536173 conn(0x7f07680a2c60 msgr2=0x7f07680a36e0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:05:17.297 INFO:tasks.workunit.client.0.smithi042.stderr:+ echo 1 2019-12-05T05:05:17.301 INFO:tasks.workunit.client.0.smithi042.stderr:+ echo 2 2019-12-05T05:05:17.308 INFO:tasks.workunit.client.0.smithi042.stderr:+ echo 3 2019-12-05T05:05:17.308 INFO:tasks.workunit.client.0.smithi042.stderr:+ '[' -e file4 ']' 2019-12-05T05:05:17.308 INFO:tasks.workunit.client.0.smithi042.stderr:+ mkdir .snap/snap1 2019-12-05T05:05:17.321 INFO:tasks.workunit.client.0.smithi042.stderr:+ echo 4 2019-12-05T05:05:17.325 INFO:tasks.workunit.client.0.smithi042.stderr:++ ls 2019-12-05T05:05:17.331 INFO:tasks.workunit.client.0.smithi042.stderr:+ now='file1 2019-12-05T05:05:17.331 INFO:tasks.workunit.client.0.smithi042.stderr:file2 2019-12-05T05:05:17.331 INFO:tasks.workunit.client.0.smithi042.stderr:file3 2019-12-05T05:05:17.331 INFO:tasks.workunit.client.0.smithi042.stderr:file4' 2019-12-05T05:05:17.332 INFO:tasks.workunit.client.0.smithi042.stderr:++ ls .snap/snap1 2019-12-05T05:05:17.336 INFO:tasks.workunit.client.0.smithi042.stderr:+ then='file1 2019-12-05T05:05:17.337 INFO:tasks.workunit.client.0.smithi042.stderr:file2 2019-12-05T05:05:17.337 INFO:tasks.workunit.client.0.smithi042.stderr:file3' 2019-12-05T05:05:17.337 INFO:tasks.workunit.client.0.smithi042.stderr:+ rmdir .snap/snap1 2019-12-05T05:05:17.346 INFO:tasks.workunit.client.0.smithi042.stderr:+ '[' 'file1 2019-12-05T05:05:17.346 INFO:tasks.workunit.client.0.smithi042.stderr:file2 2019-12-05T05:05:17.347 INFO:tasks.workunit.client.0.smithi042.stderr:file3 2019-12-05T05:05:17.347 INFO:tasks.workunit.client.0.smithi042.stderr:file4' = 'file1 2019-12-05T05:05:17.347 INFO:tasks.workunit.client.0.smithi042.stderr:file2 2019-12-05T05:05:17.347 INFO:tasks.workunit.client.0.smithi042.stderr:file3' ']' 2019-12-05T05:05:17.348 INFO:tasks.workunit.client.0.smithi042.stderr:+ echo 1 2019-12-05T05:05:17.352 INFO:tasks.workunit.client.0.smithi042.stderr:+ echo 2 2019-12-05T05:05:17.359 INFO:tasks.workunit.client.0.smithi042.stderr:+ echo 3 2019-12-05T05:05:17.367 INFO:tasks.workunit.client.0.smithi042.stderr:+ mkdir .snap/snap1 2019-12-05T05:05:17.376 INFO:tasks.workunit.client.0.smithi042.stderr:+ echo 4 2019-12-05T05:05:17.383 INFO:tasks.workunit.client.0.smithi042.stderr:+ rmdir .snap/snap1 2019-12-05T05:05:17.393 INFO:tasks.workunit.client.0.smithi042.stderr:+ rm file1 file2 file3 file4 2019-12-05T05:05:17.404 INFO:tasks.workunit.client.0.smithi042.stderr:+ echo OK 2019-12-05T05:05:17.404 INFO:tasks.workunit.client.0.smithi042.stdout:OK 2019-12-05T05:05:17.404 INFO:teuthology.orchestra.run:Running command with timeout 3600 2019-12-05T05:05:17.405 INFO:teuthology.orchestra.run.smithi042:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2019-12-05T05:05:17.506 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-2.sh... 2019-12-05T05:05:17.506 INFO:teuthology.orchestra.run.smithi042:workunit test fs/snaps/snaptest-2.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c5290805033483ba0cec165c06bde07dae5f19ec TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-2.sh 2019-12-05T05:05:17.706 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:17.696+0000 7f2d284e0700 1 Processor -- start 2019-12-05T05:05:17.706 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:17.696+0000 7f2d284e0700 1 -- start start 2019-12-05T05:05:17.706 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:17.696+0000 7f2d284e0700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2d200a2840 0x7f2d200a8030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:05:17.706 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:17.696+0000 7f2d284e0700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2d200652f0 0x7f2d2014f200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:05:17.707 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:17.696+0000 7f2d284e0700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d200a7b60 con 0x7f2d200a69f0 2019-12-05T05:05:17.707 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:17.696+0000 7f2d284e0700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d20065b20 con 0x7f2d200a2840 2019-12-05T05:05:17.707 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:17.696+0000 7f2d284e0700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d200a25a0 con 0x7f2d200652f0 2019-12-05T05:05:17.707 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:17.696+0000 7f2d2627c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2d200a2840 0x7f2d200a8030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:05:17.707 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:17.700+0000 7f2d2627c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2d200a2840 0x7f2d200a8030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:54486/0 (socket says 172.21.15.42:54486) 2019-12-05T05:05:17.707 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:17.700+0000 7f2d2627c700 1 -- 172.21.15.42:0/74800990 learned_addr learned my addr 172.21.15.42:0/74800990 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:05:17.707 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:17.700+0000 7f2d2727e700 1 --2- 172.21.15.42:0/74800990 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2d200652f0 0x7f2d2014f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:05:17.708 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:17.700+0000 7f2d2627c700 1 -- 172.21.15.42:0/74800990 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2d200652f0 msgr2=0x7f2d2014f200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:05:17.708 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:17.700+0000 7f2d2627c700 1 --2- 172.21.15.42:0/74800990 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2d200652f0 0x7f2d2014f200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:05:17.708 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:17.700+0000 7f2d2627c700 1 -- 172.21.15.42:0/74800990 >> v1:172.21.15.40:6789/0 conn(0x7f2d200a69f0 legacy=0x7f2d20144fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:05:17.708 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:17.700+0000 7f2d2627c700 1 -- 172.21.15.42:0/74800990 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d200a2c70 con 0x7f2d200a2840 2019-12-05T05:05:17.708 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:17.700+0000 7f2d2627c700 1 --2- 172.21.15.42:0/74800990 >> v2:172.21.15.42:3300/0 conn(0x7f2d200a2840 0x7f2d200a8030 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f2d200a7630 tx=0x7f2d1000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:05:17.708 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:17.700+0000 7f2d2627c700 1 --2- 172.21.15.42:0/74800990 >> v2:172.21.15.42:3300/0 conn(0x7f2d200a2840 0x7f2d200a8030 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f2d200a7630 tx=0x7f2d1000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:05:17.709 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:17.700+0000 7f2d2627c700 1 --2- 172.21.15.42:0/74800990 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2d200a2840 0x7f2d200a8030 secure :-1 s=READY pgs=274 cs=0 l=1 rx=0x7f2d200a7630 tx=0x7f2d1000fd30).ready entity=mon.1 client_cookie=491040146f7e54f8 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:05:17.709 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:17.700+0000 7f2d2627c700 1 --2- 172.21.15.42:0/74800990 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2d200a2840 0x7f2d200a8030 secure :-1 s=THROTTLE_DONE pgs=274 cs=0 l=1 rx=0x7f2d200a7630 tx=0x7f2d1000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:05:17.709 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:17.700+0000 7f2d25a7b700 1 -- 172.21.15.42:0/74800990 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2d100113b0 con 0x7f2d200a2840 2019-12-05T05:05:17.709 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:17.700+0000 7f2d2627c700 1 --2- 172.21.15.42:0/74800990 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2d200a2840 0x7f2d200a8030 secure :-1 s=THROTTLE_DONE pgs=274 cs=0 l=1 rx=0x7f2d200a7630 tx=0x7f2d1000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:05:17.709 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:17.700+0000 7f2d2627c700 1 --2- 172.21.15.42:0/74800990 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2d200a2840 0x7f2d200a8030 secure :-1 s=THROTTLE_DONE pgs=274 cs=0 l=1 rx=0x7f2d200a7630 tx=0x7f2d1000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:05:17.709 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:17.700+0000 7f2d25a7b700 1 -- 172.21.15.42:0/74800990 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d10008bb0 con 0x7f2d200a2840 2019-12-05T05:05:17.710 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:17.700+0000 7f2d25a7b700 1 -- 172.21.15.42:0/74800990 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2d10011990 con 0x7f2d200a2840 2019-12-05T05:05:17.710 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:17.700+0000 7f2d284e0700 1 -- 172.21.15.42:0/74800990 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2d200a2840 msgr2=0x7f2d200a8030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:05:17.710 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:17.700+0000 7f2d284e0700 1 --2- 172.21.15.42:0/74800990 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2d200a2840 0x7f2d200a8030 secure :-1 s=READY pgs=274 cs=0 l=1 rx=0x7f2d200a7630 tx=0x7f2d1000fd30).stop 2019-12-05T05:05:17.710 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:17.700+0000 7f2d284e0700 1 -- 172.21.15.42:0/74800990 shutdown_connections 2019-12-05T05:05:17.710 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:17.700+0000 7f2d284e0700 1 --2- 172.21.15.42:0/74800990 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2d200652f0 0x7f2d2014f200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:05:17.710 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:17.700+0000 7f2d284e0700 1 --2- 172.21.15.42:0/74800990 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2d200a2840 0x7f2d200a8030 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:05:17.711 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:17.700+0000 7f2d284e0700 1 -- 172.21.15.42:0/74800990 shutdown_connections 2019-12-05T05:05:17.711 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:17.700+0000 7f2d284e0700 1 -- 172.21.15.42:0/74800990 wait complete. 2019-12-05T05:05:17.712 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:17.700+0000 7f2d284e0700 1 -- 172.21.15.42:0/74800990 >> 172.21.15.42:0/74800990 conn(0x7f2d20059570 msgr2=0x7f2d200599a0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:05:17.716 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:17.704+0000 7f2d284e0700 1 Processor -- start 2019-12-05T05:05:17.717 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:17.704+0000 7f2d284e0700 1 -- start start 2019-12-05T05:05:17.717 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:17.704+0000 7f2d284e0700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2d2011c3a0 0x7f2d20076d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:05:17.717 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:17.704+0000 7f2d284e0700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2d200a60d0 0x7f2d200772b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:05:17.717 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:17.704+0000 7f2d284e0700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d20078060 con 0x7f2d20078d80 2019-12-05T05:05:17.717 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:17.704+0000 7f2d284e0700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d20079290 con 0x7f2d200a60d0 2019-12-05T05:05:17.717 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:17.704+0000 7f2d284e0700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d200a5b90 con 0x7f2d2011c3a0 2019-12-05T05:05:17.718 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:17.704+0000 7f2d26a7d700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f2d20078d80 0x7f2d2007a190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:54530/0 (socket says 172.21.15.42:54530) 2019-12-05T05:05:17.718 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:17.704+0000 7f2d26a7d700 1 -- 172.21.15.42:0/2896975971 learned_addr learned my addr 172.21.15.42:0/2896975971 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:05:17.718 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:17.704+0000 7f2d2627c700 1 --2- 172.21.15.42:0/2896975971 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2d2011c3a0 0x7f2d20076d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:05:17.718 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:17.704+0000 7f2d2727e700 1 --2- 172.21.15.42:0/2896975971 >> v2:172.21.15.40:3300/0 conn(0x7f2d200a60d0 0x7f2d200772b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:05:17.718 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:17.704+0000 7f2d177fe700 1 -- 172.21.15.42:0/2896975971 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2860198622 0 0) 0x7f2d20078060 con 0x7f2d20078d80 2019-12-05T05:05:17.718 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:17.704+0000 7f2d177fe700 1 -- 172.21.15.42:0/2896975971 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d080030b0 con 0x7f2d20078d80 2019-12-05T05:05:17.718 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:17.708+0000 7f2d2627c700 1 -- 172.21.15.42:0/2896975971 >> v2:172.21.15.40:3300/0 conn(0x7f2d200a60d0 msgr2=0x7f2d200772b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:05:17.719 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:17.708+0000 7f2d2627c700 1 --2- 172.21.15.42:0/2896975971 >> v2:172.21.15.40:3300/0 conn(0x7f2d200a60d0 0x7f2d200772b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:05:17.719 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:17.708+0000 7f2d2627c700 1 -- 172.21.15.42:0/2896975971 >> v1:172.21.15.42:6789/0 conn(0x7f2d20078d80 legacy=0x7f2d2007a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:05:17.719 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:17.708+0000 7f2d2627c700 1 -- 172.21.15.42:0/2896975971 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d200784e0 con 0x7f2d2011c3a0 2019-12-05T05:05:17.719 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:17.708+0000 7f2d2627c700 1 --2- 172.21.15.42:0/2896975971 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2d2011c3a0 0x7f2d20076d60 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f2d200a7630 tx=0x7f2d10011f10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:05:17.719 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:17.708+0000 7f2d2627c700 1 --2- 172.21.15.42:0/2896975971 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2d2011c3a0 0x7f2d20076d60 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f2d200a7630 tx=0x7f2d10011f10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:05:17.719 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:17.708+0000 7f2d2627c700 1 --2- 172.21.15.42:0/2896975971 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2d2011c3a0 0x7f2d20076d60 secure :-1 s=READY pgs=56 cs=0 l=1 rx=0x7f2d200a7630 tx=0x7f2d10011f10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:05:17.719 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:17.708+0000 7f2d2627c700 1 --2- 172.21.15.42:0/2896975971 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2d2011c3a0 0x7f2d20076d60 secure :-1 s=THROTTLE_DONE pgs=56 cs=0 l=1 rx=0x7f2d200a7630 tx=0x7f2d10011f10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:05:17.720 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:17.708+0000 7f2d2627c700 1 --2- 172.21.15.42:0/2896975971 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2d2011c3a0 0x7f2d20076d60 secure :-1 s=THROTTLE_DONE pgs=56 cs=0 l=1 rx=0x7f2d200a7630 tx=0x7f2d10011f10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:05:17.720 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:17.708+0000 7f2d177fe700 1 -- 172.21.15.42:0/2896975971 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2d10007340 con 0x7f2d2011c3a0 2019-12-05T05:05:17.720 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:17.708+0000 7f2d2627c700 1 --2- 172.21.15.42:0/2896975971 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2d2011c3a0 0x7f2d20076d60 secure :-1 s=THROTTLE_DONE pgs=56 cs=0 l=1 rx=0x7f2d200a7630 tx=0x7f2d10011f10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:05:17.720 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:17.708+0000 7f2d177fe700 1 -- 172.21.15.42:0/2896975971 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d100115b0 con 0x7f2d2011c3a0 2019-12-05T05:05:17.722 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:17.708+0000 7f2d284e0700 1 -- 172.21.15.42:0/2896975971 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d20077800 con 0x7f2d2011c3a0 2019-12-05T05:05:17.722 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:17.708+0000 7f2d177fe700 1 -- 172.21.15.42:0/2896975971 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2d10005540 con 0x7f2d2011c3a0 2019-12-05T05:05:17.722 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:17.708+0000 7f2d284e0700 1 -- 172.21.15.42:0/2896975971 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d20130870 con 0x7f2d2011c3a0 2019-12-05T05:05:17.723 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:17.708+0000 7f2d2627c700 1 --2- 172.21.15.42:0/2896975971 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2d2011c3a0 0x7f2d20076d60 secure :-1 s=THROTTLE_DONE pgs=56 cs=0 l=1 rx=0x7f2d200a7630 tx=0x7f2d10011f10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:05:17.723 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:17.708+0000 7f2d177fe700 1 -- 172.21.15.42:0/2896975971 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f2d10005790 con 0x7f2d2011c3a0 2019-12-05T05:05:17.723 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:17.708+0000 7f2d2627c700 1 --2- 172.21.15.42:0/2896975971 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2d2011c3a0 0x7f2d20076d60 secure :-1 s=THROTTLE_DONE pgs=56 cs=0 l=1 rx=0x7f2d200a7630 tx=0x7f2d10011f10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:05:17.723 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:17.708+0000 7f2d177fe700 1 --2- 172.21.15.42:0/2896975971 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2d0803b2a0 0x7f2d0803d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:05:17.723 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:17.708+0000 7f2d177fe700 1 -- 172.21.15.42:0/2896975971 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6173+0+0 (secure 0 0 0) 0x7f2d1004b250 con 0x7f2d2011c3a0 2019-12-05T05:05:17.723 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:17.708+0000 7f2d26a7d700 1 --2- 172.21.15.42:0/2896975971 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2d0803b2a0 0x7f2d0803d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:05:17.724 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:17.712+0000 7f2d26a7d700 1 --2- 172.21.15.42:0/2896975971 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2d0803b2a0 0x7f2d0803d7b0 crc :-1 s=READY pgs=134 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:05:17.729 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:17.712+0000 7f2d284e0700 1 -- 172.21.15.42:0/2896975971 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d2005d2c0 con 0x7f2d2011c3a0 2019-12-05T05:05:17.730 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:17.716+0000 7f2d2627c700 1 --2- 172.21.15.42:0/2896975971 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2d2011c3a0 0x7f2d20076d60 secure :-1 s=THROTTLE_DONE pgs=56 cs=0 l=1 rx=0x7f2d200a7630 tx=0x7f2d10011f10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:05:17.730 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:17.716+0000 7f2d177fe700 1 -- 172.21.15.42:0/2896975971 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f2d100322d0 con 0x7f2d2011c3a0 2019-12-05T05:05:17.892 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:17.888+0000 7f2d284e0700 1 -- 172.21.15.42:0/2896975971 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"var": "allow_new_snaps", "prefix": "fs set", "fs_name": "cephfs", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f2d2005ba70 con 0x7f2d2011c3a0 2019-12-05T05:05:18.291 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:18.284+0000 7f2d2627c700 1 --2- 172.21.15.42:0/2896975971 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2d2011c3a0 0x7f2d20076d60 secure :-1 s=THROTTLE_DONE pgs=56 cs=0 l=1 rx=0x7f2d200a7630 tx=0x7f2d10011f10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:05:18.291 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:18.284+0000 7f2d177fe700 1 -- 172.21.15.42:0/2896975971 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"var": "allow_new_snaps", "prefix": "fs set", "fs_name": "cephfs", "val": "true", "yes_i_really_mean_it": true}]=0 enabled new snapshots v22) v1 ==== 167+0+0 (secure 0 0 0) 0x7f2d10032530 con 0x7f2d2011c3a0 2019-12-05T05:05:18.419 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:18.416+0000 7f2d284e0700 1 -- 172.21.15.42:0/2896975971 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"var": "allow_new_snaps", "prefix": "fs set", "fs_name": "cephfs", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f2d20069770 con 0x7f2d2011c3a0 2019-12-05T05:05:19.295 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:19.292+0000 7f2d2627c700 1 --2- 172.21.15.42:0/2896975971 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2d2011c3a0 0x7f2d20076d60 secure :-1 s=THROTTLE_DONE pgs=56 cs=0 l=1 rx=0x7f2d200a7630 tx=0x7f2d10011f10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:05:19.295 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:19.292+0000 7f2d177fe700 1 -- 172.21.15.42:0/2896975971 <== mon.2 v2:172.21.15.40:3301/0 8 ==== mon_command_ack([{"var": "allow_new_snaps", "prefix": "fs set", "fs_name": "cephfs", "val": "true", "yes_i_really_mean_it": true}]=0 enabled new snapshots v23) v1 ==== 167+0+0 (secure 0 0 0) 0x7f2d100330e0 con 0x7f2d2011c3a0 2019-12-05T05:05:19.335 INFO:tasks.workunit.client.0.smithi042.stderr:enabled new snapshots 2019-12-05T05:05:19.340 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:19.336+0000 7f2d284e0700 1 -- 172.21.15.42:0/2896975971 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2d0803b2a0 msgr2=0x7f2d0803d7b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:05:19.340 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:19.336+0000 7f2d284e0700 1 --2- 172.21.15.42:0/2896975971 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2d0803b2a0 0x7f2d0803d7b0 crc :-1 s=READY pgs=134 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:05:19.340 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:19.336+0000 7f2d284e0700 1 -- 172.21.15.42:0/2896975971 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2d2011c3a0 msgr2=0x7f2d20076d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:05:19.341 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:19.336+0000 7f2d284e0700 1 --2- 172.21.15.42:0/2896975971 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2d2011c3a0 0x7f2d20076d60 secure :-1 s=READY pgs=56 cs=0 l=1 rx=0x7f2d200a7630 tx=0x7f2d10011f10).stop 2019-12-05T05:05:19.341 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:19.336+0000 7f2d284e0700 1 -- 172.21.15.42:0/2896975971 shutdown_connections 2019-12-05T05:05:19.341 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:19.336+0000 7f2d284e0700 1 --2- 172.21.15.42:0/2896975971 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2d0803b2a0 0x7f2d0803d7b0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:05:19.341 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:19.336+0000 7f2d284e0700 1 --2- 172.21.15.42:0/2896975971 >> v2:172.21.15.40:3300/0 conn(0x7f2d200a60d0 0x7f2d200772b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:05:19.341 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:19.336+0000 7f2d284e0700 1 --2- 172.21.15.42:0/2896975971 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2d2011c3a0 0x7f2d20076d60 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:05:19.341 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:19.336+0000 7f2d284e0700 1 -- 172.21.15.42:0/2896975971 shutdown_connections 2019-12-05T05:05:19.341 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:19.336+0000 7f2d284e0700 1 -- 172.21.15.42:0/2896975971 wait complete. 2019-12-05T05:05:19.342 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:05:19.336+0000 7f2d284e0700 1 -- 172.21.15.42:0/2896975971 >> 172.21.15.42:0/2896975971 conn(0x7f2d200a2840 msgr2=0x7f2d2009c8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:05:19.351 INFO:tasks.workunit.client.0.smithi042.stdout:Create dir 100 to 199 ... 2019-12-05T05:05:19.362 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 100 2019-12-05T05:05:19.401 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 101 2019-12-05T05:05:19.438 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 102 2019-12-05T05:05:19.453 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 103 2019-12-05T05:05:19.480 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 104 2019-12-05T05:05:19.505 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 105 2019-12-05T05:05:19.530 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 106 2019-12-05T05:05:19.556 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 107 2019-12-05T05:05:19.583 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 108 2019-12-05T05:05:19.608 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 109 2019-12-05T05:05:19.634 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 110 2019-12-05T05:05:19.659 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 111 2019-12-05T05:05:19.685 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 112 2019-12-05T05:05:19.710 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 113 2019-12-05T05:05:19.736 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 114 2019-12-05T05:05:19.761 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 115 2019-12-05T05:05:19.786 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 116 2019-12-05T05:05:19.810 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 117 2019-12-05T05:05:19.834 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 118 2019-12-05T05:05:19.858 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 119 2019-12-05T05:05:20.133 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 120 2019-12-05T05:05:20.162 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 121 2019-12-05T05:05:20.189 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 122 2019-12-05T05:05:20.248 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 123 2019-12-05T05:05:20.281 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 124 2019-12-05T05:05:20.308 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 125 2019-12-05T05:05:20.332 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 126 2019-12-05T05:05:20.357 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 127 2019-12-05T05:05:20.384 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 128 2019-12-05T05:05:20.409 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 129 2019-12-05T05:05:20.434 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 130 2019-12-05T05:05:20.460 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 131 2019-12-05T05:05:20.486 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 132 2019-12-05T05:05:20.511 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 133 2019-12-05T05:05:20.536 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 134 2019-12-05T05:05:20.560 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 135 2019-12-05T05:05:20.585 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 136 2019-12-05T05:05:20.610 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 137 2019-12-05T05:05:20.636 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 138 2019-12-05T05:05:20.661 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 139 2019-12-05T05:05:20.685 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 140 2019-12-05T05:05:20.710 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 141 2019-12-05T05:05:20.734 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 142 2019-12-05T05:05:20.761 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 143 2019-12-05T05:05:20.786 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 144 2019-12-05T05:05:20.810 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 145 2019-12-05T05:05:20.834 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 146 2019-12-05T05:05:20.859 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 147 2019-12-05T05:05:20.884 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 148 2019-12-05T05:05:21.875 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 149 2019-12-05T05:05:21.905 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 150 2019-12-05T05:05:21.935 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 151 2019-12-05T05:05:21.960 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 152 2019-12-05T05:05:21.986 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 153 2019-12-05T05:05:22.012 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 154 2019-12-05T05:05:22.037 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 155 2019-12-05T05:05:22.065 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 156 2019-12-05T05:05:22.090 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 157 2019-12-05T05:05:22.117 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 158 2019-12-05T05:05:22.144 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 159 2019-12-05T05:05:22.169 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 160 2019-12-05T05:05:22.196 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 161 2019-12-05T05:05:22.222 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 162 2019-12-05T05:05:22.249 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 163 2019-12-05T05:05:22.275 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 164 2019-12-05T05:05:22.299 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 165 2019-12-05T05:05:22.326 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 166 2019-12-05T05:05:22.352 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 167 2019-12-05T05:05:22.377 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 168 2019-12-05T05:05:22.404 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 169 2019-12-05T05:05:22.428 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 170 2019-12-05T05:05:22.453 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 171 2019-12-05T05:05:22.481 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 172 2019-12-05T05:05:22.506 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 173 2019-12-05T05:05:22.530 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 174 2019-12-05T05:05:22.556 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 175 2019-12-05T05:05:22.581 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 176 2019-12-05T05:05:22.606 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 177 2019-12-05T05:05:22.633 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 178 2019-12-05T05:05:22.658 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 179 2019-12-05T05:05:22.685 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 180 2019-12-05T05:05:22.709 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 181 2019-12-05T05:05:23.064 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 182 2019-12-05T05:05:24.264 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 183 2019-12-05T05:05:24.293 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 184 2019-12-05T05:05:24.320 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 185 2019-12-05T05:05:24.348 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 186 2019-12-05T05:05:24.371 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 187 2019-12-05T05:05:24.397 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 188 2019-12-05T05:05:24.422 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 189 2019-12-05T05:05:24.450 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 190 2019-12-05T05:05:24.476 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 191 2019-12-05T05:05:24.499 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 192 2019-12-05T05:05:24.527 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 193 2019-12-05T05:05:24.550 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 194 2019-12-05T05:05:24.578 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 195 2019-12-05T05:05:24.603 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 196 2019-12-05T05:05:24.629 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 197 2019-12-05T05:05:24.656 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 198 2019-12-05T05:05:24.685 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 199 2019-12-05T05:05:24.704 INFO:tasks.workunit.client.0.smithi042.stdout:Take first snapshot .snap/test1 2019-12-05T05:05:25.468 INFO:tasks.workunit.client.0.smithi042.stdout:Create dir 200 to 299 ... 2019-12-05T05:05:25.468 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 200 2019-12-05T05:05:26.480 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 201 2019-12-05T05:05:26.508 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 202 2019-12-05T05:05:26.528 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 203 2019-12-05T05:05:26.628 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 204 2019-12-05T05:05:26.662 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 205 2019-12-05T05:05:26.683 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 206 2019-12-05T05:05:26.702 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 207 2019-12-05T05:05:26.727 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 208 2019-12-05T05:05:26.746 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 209 2019-12-05T05:05:26.768 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 210 2019-12-05T05:05:27.334 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 211 2019-12-05T05:05:27.447 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 212 2019-12-05T05:05:27.469 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 213 2019-12-05T05:05:27.495 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 214 2019-12-05T05:05:27.516 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 215 2019-12-05T05:05:27.538 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 216 2019-12-05T05:05:27.559 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 217 2019-12-05T05:05:27.582 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 218 2019-12-05T05:05:27.604 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 219 2019-12-05T05:05:27.624 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 220 2019-12-05T05:05:27.646 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 221 2019-12-05T05:05:27.668 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 222 2019-12-05T05:05:27.688 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 223 2019-12-05T05:05:27.710 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 224 2019-12-05T05:05:27.732 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 225 2019-12-05T05:05:27.754 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 226 2019-12-05T05:05:27.777 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 227 2019-12-05T05:05:27.800 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 228 2019-12-05T05:05:27.819 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 229 2019-12-05T05:05:27.850 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 230 2019-12-05T05:05:27.862 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 231 2019-12-05T05:05:27.884 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 232 2019-12-05T05:05:27.905 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 233 2019-12-05T05:05:27.926 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 234 2019-12-05T05:05:27.949 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 235 2019-12-05T05:05:28.103 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 236 2019-12-05T05:05:29.356 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 237 2019-12-05T05:05:29.403 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 238 2019-12-05T05:05:29.408 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 239 2019-12-05T05:05:29.433 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 240 2019-12-05T05:05:29.457 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 241 2019-12-05T05:05:29.480 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 242 2019-12-05T05:05:29.504 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 243 2019-12-05T05:05:29.527 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 244 2019-12-05T05:05:29.550 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 245 2019-12-05T05:05:30.441 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 246 2019-12-05T05:05:30.460 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 247 2019-12-05T05:05:30.485 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 248 2019-12-05T05:05:30.509 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 249 2019-12-05T05:05:30.538 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 250 2019-12-05T05:05:30.588 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 251 2019-12-05T05:05:33.090 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 252 2019-12-05T05:05:33.115 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 253 2019-12-05T05:05:33.140 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 254 2019-12-05T05:05:33.166 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 255 2019-12-05T05:05:33.191 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 256 2019-12-05T05:05:33.217 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 257 2019-12-05T05:05:33.238 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 258 2019-12-05T05:05:33.262 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 259 2019-12-05T05:05:33.286 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 260 2019-12-05T05:05:33.309 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 261 2019-12-05T05:05:33.451 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 262 2019-12-05T05:05:33.477 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 263 2019-12-05T05:05:33.502 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 264 2019-12-05T05:05:33.527 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 265 2019-12-05T05:05:33.550 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 266 2019-12-05T05:05:33.575 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 267 2019-12-05T05:05:33.599 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 268 2019-12-05T05:05:33.624 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 269 2019-12-05T05:05:33.648 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 270 2019-12-05T05:05:33.673 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 271 2019-12-05T05:05:33.675 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:05:33.678 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:05:33.699 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 272 2019-12-05T05:05:33.723 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 273 2019-12-05T05:05:33.747 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 274 2019-12-05T05:05:33.771 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 275 2019-12-05T05:05:33.795 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 276 2019-12-05T05:05:33.818 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 277 2019-12-05T05:05:33.841 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 278 2019-12-05T05:05:33.864 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 279 2019-12-05T05:05:33.887 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 280 2019-12-05T05:05:33.910 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 281 2019-12-05T05:05:33.933 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 282 2019-12-05T05:05:33.956 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 283 2019-12-05T05:05:33.980 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 284 2019-12-05T05:05:34.005 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 285 2019-12-05T05:05:34.028 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 286 2019-12-05T05:05:34.053 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 287 2019-12-05T05:05:34.078 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 288 2019-12-05T05:05:34.102 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 289 2019-12-05T05:05:34.126 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 290 2019-12-05T05:05:34.151 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 291 2019-12-05T05:05:34.174 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 292 2019-12-05T05:05:34.201 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 293 2019-12-05T05:05:34.224 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 294 2019-12-05T05:05:34.247 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 295 2019-12-05T05:05:34.271 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 296 2019-12-05T05:05:34.296 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 297 2019-12-05T05:05:34.323 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 298 2019-12-05T05:05:34.348 INFO:tasks.workunit.client.0.smithi042.stdout: create dir 299 2019-12-05T05:05:34.371 INFO:tasks.workunit.client.0.smithi042.stdout:Create a snapshot in every first level dir ... 2019-12-05T05:05:35.628 INFO:tasks.workunit.client.0.smithi042.stdout: create 100/.snap/snap-subdir-test 2019-12-05T05:05:35.682 INFO:tasks.workunit.client.0.smithi042.stdout: create 100/30 file after the snapshot 2019-12-05T05:05:35.686 INFO:tasks.workunit.client.0.smithi042.stdout: create 100/31 file after the snapshot 2019-12-05T05:05:35.689 INFO:tasks.workunit.client.0.smithi042.stdout: create 100/32 file after the snapshot 2019-12-05T05:05:35.692 INFO:tasks.workunit.client.0.smithi042.stdout: create 100/33 file after the snapshot 2019-12-05T05:05:35.695 INFO:tasks.workunit.client.0.smithi042.stdout: create 100/34 file after the snapshot 2019-12-05T05:05:35.697 INFO:tasks.workunit.client.0.smithi042.stdout: create 100/35 file after the snapshot 2019-12-05T05:05:35.700 INFO:tasks.workunit.client.0.smithi042.stdout: create 100/36 file after the snapshot 2019-12-05T05:05:35.702 INFO:tasks.workunit.client.0.smithi042.stdout: create 100/37 file after the snapshot 2019-12-05T05:05:35.704 INFO:tasks.workunit.client.0.smithi042.stdout: create 100/38 file after the snapshot 2019-12-05T05:05:35.706 INFO:tasks.workunit.client.0.smithi042.stdout: create 100/39 file after the snapshot 2019-12-05T05:05:35.708 INFO:tasks.workunit.client.0.smithi042.stdout: create 101/.snap/snap-subdir-test 2019-12-05T05:05:35.734 INFO:tasks.workunit.client.0.smithi042.stdout: create 101/30 file after the snapshot 2019-12-05T05:05:35.737 INFO:tasks.workunit.client.0.smithi042.stdout: create 101/31 file after the snapshot 2019-12-05T05:05:35.739 INFO:tasks.workunit.client.0.smithi042.stdout: create 101/32 file after the snapshot 2019-12-05T05:05:35.742 INFO:tasks.workunit.client.0.smithi042.stdout: create 101/33 file after the snapshot 2019-12-05T05:05:35.745 INFO:tasks.workunit.client.0.smithi042.stdout: create 101/34 file after the snapshot 2019-12-05T05:05:35.747 INFO:tasks.workunit.client.0.smithi042.stdout: create 101/35 file after the snapshot 2019-12-05T05:05:35.751 INFO:tasks.workunit.client.0.smithi042.stdout: create 101/36 file after the snapshot 2019-12-05T05:05:35.753 INFO:tasks.workunit.client.0.smithi042.stdout: create 101/37 file after the snapshot 2019-12-05T05:05:35.755 INFO:tasks.workunit.client.0.smithi042.stdout: create 101/38 file after the snapshot 2019-12-05T05:05:35.759 INFO:tasks.workunit.client.0.smithi042.stdout: create 101/39 file after the snapshot 2019-12-05T05:05:35.760 INFO:tasks.workunit.client.0.smithi042.stdout: create 102/.snap/snap-subdir-test 2019-12-05T05:05:35.786 INFO:tasks.workunit.client.0.smithi042.stdout: create 102/30 file after the snapshot 2019-12-05T05:05:35.788 INFO:tasks.workunit.client.0.smithi042.stdout: create 102/31 file after the snapshot 2019-12-05T05:05:35.790 INFO:tasks.workunit.client.0.smithi042.stdout: create 102/32 file after the snapshot 2019-12-05T05:05:35.794 INFO:tasks.workunit.client.0.smithi042.stdout: create 102/33 file after the snapshot 2019-12-05T05:05:35.797 INFO:tasks.workunit.client.0.smithi042.stdout: create 102/34 file after the snapshot 2019-12-05T05:05:35.799 INFO:tasks.workunit.client.0.smithi042.stdout: create 102/35 file after the snapshot 2019-12-05T05:05:35.801 INFO:tasks.workunit.client.0.smithi042.stdout: create 102/36 file after the snapshot 2019-12-05T05:05:35.804 INFO:tasks.workunit.client.0.smithi042.stdout: create 102/37 file after the snapshot 2019-12-05T05:05:35.806 INFO:tasks.workunit.client.0.smithi042.stdout: create 102/38 file after the snapshot 2019-12-05T05:05:35.808 INFO:tasks.workunit.client.0.smithi042.stdout: create 102/39 file after the snapshot 2019-12-05T05:05:35.810 INFO:tasks.workunit.client.0.smithi042.stdout: create 103/.snap/snap-subdir-test 2019-12-05T05:05:35.835 INFO:tasks.workunit.client.0.smithi042.stdout: create 103/30 file after the snapshot 2019-12-05T05:05:35.837 INFO:tasks.workunit.client.0.smithi042.stdout: create 103/31 file after the snapshot 2019-12-05T05:05:35.840 INFO:tasks.workunit.client.0.smithi042.stdout: create 103/32 file after the snapshot 2019-12-05T05:05:35.844 INFO:tasks.workunit.client.0.smithi042.stdout: create 103/33 file after the snapshot 2019-12-05T05:05:35.847 INFO:tasks.workunit.client.0.smithi042.stdout: create 103/34 file after the snapshot 2019-12-05T05:05:35.849 INFO:tasks.workunit.client.0.smithi042.stdout: create 103/35 file after the snapshot 2019-12-05T05:05:35.851 INFO:tasks.workunit.client.0.smithi042.stdout: create 103/36 file after the snapshot 2019-12-05T05:05:35.854 INFO:tasks.workunit.client.0.smithi042.stdout: create 103/37 file after the snapshot 2019-12-05T05:05:35.856 INFO:tasks.workunit.client.0.smithi042.stdout: create 103/38 file after the snapshot 2019-12-05T05:05:35.858 INFO:tasks.workunit.client.0.smithi042.stdout: create 103/39 file after the snapshot 2019-12-05T05:05:35.861 INFO:tasks.workunit.client.0.smithi042.stdout: create 104/.snap/snap-subdir-test 2019-12-05T05:05:36.445 INFO:tasks.workunit.client.0.smithi042.stdout: create 104/30 file after the snapshot 2019-12-05T05:05:36.449 INFO:tasks.workunit.client.0.smithi042.stdout: create 104/31 file after the snapshot 2019-12-05T05:05:36.453 INFO:tasks.workunit.client.0.smithi042.stdout: create 104/32 file after the snapshot 2019-12-05T05:05:36.456 INFO:tasks.workunit.client.0.smithi042.stdout: create 104/33 file after the snapshot 2019-12-05T05:05:36.460 INFO:tasks.workunit.client.0.smithi042.stdout: create 104/34 file after the snapshot 2019-12-05T05:05:36.463 INFO:tasks.workunit.client.0.smithi042.stdout: create 104/35 file after the snapshot 2019-12-05T05:05:36.466 INFO:tasks.workunit.client.0.smithi042.stdout: create 104/36 file after the snapshot 2019-12-05T05:05:36.468 INFO:tasks.workunit.client.0.smithi042.stdout: create 104/37 file after the snapshot 2019-12-05T05:05:36.471 INFO:tasks.workunit.client.0.smithi042.stdout: create 104/38 file after the snapshot 2019-12-05T05:05:36.473 INFO:tasks.workunit.client.0.smithi042.stdout: create 104/39 file after the snapshot 2019-12-05T05:05:36.475 INFO:tasks.workunit.client.0.smithi042.stdout: create 105/.snap/snap-subdir-test 2019-12-05T05:05:36.519 INFO:tasks.workunit.client.0.smithi042.stdout: create 105/30 file after the snapshot 2019-12-05T05:05:36.522 INFO:tasks.workunit.client.0.smithi042.stdout: create 105/31 file after the snapshot 2019-12-05T05:05:36.527 INFO:tasks.workunit.client.0.smithi042.stdout: create 105/32 file after the snapshot 2019-12-05T05:05:36.529 INFO:tasks.workunit.client.0.smithi042.stdout: create 105/33 file after the snapshot 2019-12-05T05:05:36.532 INFO:tasks.workunit.client.0.smithi042.stdout: create 105/34 file after the snapshot 2019-12-05T05:05:36.535 INFO:tasks.workunit.client.0.smithi042.stdout: create 105/35 file after the snapshot 2019-12-05T05:05:36.537 INFO:tasks.workunit.client.0.smithi042.stdout: create 105/36 file after the snapshot 2019-12-05T05:05:36.540 INFO:tasks.workunit.client.0.smithi042.stdout: create 105/37 file after the snapshot 2019-12-05T05:05:36.543 INFO:tasks.workunit.client.0.smithi042.stdout: create 105/38 file after the snapshot 2019-12-05T05:05:36.545 INFO:tasks.workunit.client.0.smithi042.stdout: create 105/39 file after the snapshot 2019-12-05T05:05:36.547 INFO:tasks.workunit.client.0.smithi042.stdout: create 106/.snap/snap-subdir-test 2019-12-05T05:05:37.414 INFO:tasks.workunit.client.0.smithi042.stdout: create 106/30 file after the snapshot 2019-12-05T05:05:37.419 INFO:tasks.workunit.client.0.smithi042.stdout: create 106/31 file after the snapshot 2019-12-05T05:05:37.424 INFO:tasks.workunit.client.0.smithi042.stdout: create 106/32 file after the snapshot 2019-12-05T05:05:37.428 INFO:tasks.workunit.client.0.smithi042.stdout: create 106/33 file after the snapshot 2019-12-05T05:05:37.431 INFO:tasks.workunit.client.0.smithi042.stdout: create 106/34 file after the snapshot 2019-12-05T05:05:37.433 INFO:tasks.workunit.client.0.smithi042.stdout: create 106/35 file after the snapshot 2019-12-05T05:05:37.436 INFO:tasks.workunit.client.0.smithi042.stdout: create 106/36 file after the snapshot 2019-12-05T05:05:37.438 INFO:tasks.workunit.client.0.smithi042.stdout: create 106/37 file after the snapshot 2019-12-05T05:05:37.441 INFO:tasks.workunit.client.0.smithi042.stdout: create 106/38 file after the snapshot 2019-12-05T05:05:37.443 INFO:tasks.workunit.client.0.smithi042.stdout: create 106/39 file after the snapshot 2019-12-05T05:05:37.446 INFO:tasks.workunit.client.0.smithi042.stdout: create 107/.snap/snap-subdir-test 2019-12-05T05:05:37.488 INFO:tasks.workunit.client.0.smithi042.stdout: create 107/30 file after the snapshot 2019-12-05T05:05:37.488 INFO:tasks.workunit.client.0.smithi042.stdout: create 107/31 file after the snapshot 2019-12-05T05:05:37.489 INFO:tasks.workunit.client.0.smithi042.stdout: create 107/32 file after the snapshot 2019-12-05T05:05:37.490 INFO:tasks.workunit.client.0.smithi042.stdout: create 107/33 file after the snapshot 2019-12-05T05:05:37.490 INFO:tasks.workunit.client.0.smithi042.stdout: create 107/34 file after the snapshot 2019-12-05T05:05:37.490 INFO:tasks.workunit.client.0.smithi042.stdout: create 107/35 file after the snapshot 2019-12-05T05:05:37.490 INFO:tasks.workunit.client.0.smithi042.stdout: create 107/36 file after the snapshot 2019-12-05T05:05:37.667 INFO:tasks.workunit.client.0.smithi042.stdout: create 107/37 file after the snapshot 2019-12-05T05:05:37.673 INFO:tasks.workunit.client.0.smithi042.stdout: create 107/38 file after the snapshot 2019-12-05T05:05:37.676 INFO:tasks.workunit.client.0.smithi042.stdout: create 107/39 file after the snapshot 2019-12-05T05:05:37.679 INFO:tasks.workunit.client.0.smithi042.stdout: create 108/.snap/snap-subdir-test 2019-12-05T05:05:37.719 INFO:tasks.workunit.client.0.smithi042.stdout: create 108/30 file after the snapshot 2019-12-05T05:05:37.722 INFO:tasks.workunit.client.0.smithi042.stdout: create 108/31 file after the snapshot 2019-12-05T05:05:37.726 INFO:tasks.workunit.client.0.smithi042.stdout: create 108/32 file after the snapshot 2019-12-05T05:05:37.729 INFO:tasks.workunit.client.0.smithi042.stdout: create 108/33 file after the snapshot 2019-12-05T05:05:37.731 INFO:tasks.workunit.client.0.smithi042.stdout: create 108/34 file after the snapshot 2019-12-05T05:05:37.735 INFO:tasks.workunit.client.0.smithi042.stdout: create 108/35 file after the snapshot 2019-12-05T05:05:37.738 INFO:tasks.workunit.client.0.smithi042.stdout: create 108/36 file after the snapshot 2019-12-05T05:05:37.740 INFO:tasks.workunit.client.0.smithi042.stdout: create 108/37 file after the snapshot 2019-12-05T05:05:37.742 INFO:tasks.workunit.client.0.smithi042.stdout: create 108/38 file after the snapshot 2019-12-05T05:05:37.760 INFO:tasks.workunit.client.0.smithi042.stdout: create 108/39 file after the snapshot 2019-12-05T05:05:37.760 INFO:tasks.workunit.client.0.smithi042.stdout: create 109/.snap/snap-subdir-test 2019-12-05T05:05:37.772 INFO:tasks.workunit.client.0.smithi042.stdout: create 109/30 file after the snapshot 2019-12-05T05:05:37.774 INFO:tasks.workunit.client.0.smithi042.stdout: create 109/31 file after the snapshot 2019-12-05T05:05:37.776 INFO:tasks.workunit.client.0.smithi042.stdout: create 109/32 file after the snapshot 2019-12-05T05:05:37.778 INFO:tasks.workunit.client.0.smithi042.stdout: create 109/33 file after the snapshot 2019-12-05T05:05:37.780 INFO:tasks.workunit.client.0.smithi042.stdout: create 109/34 file after the snapshot 2019-12-05T05:05:37.782 INFO:tasks.workunit.client.0.smithi042.stdout: create 109/35 file after the snapshot 2019-12-05T05:05:37.784 INFO:tasks.workunit.client.0.smithi042.stdout: create 109/36 file after the snapshot 2019-12-05T05:05:37.786 INFO:tasks.workunit.client.0.smithi042.stdout: create 109/37 file after the snapshot 2019-12-05T05:05:37.788 INFO:tasks.workunit.client.0.smithi042.stdout: create 109/38 file after the snapshot 2019-12-05T05:05:37.790 INFO:tasks.workunit.client.0.smithi042.stdout: create 109/39 file after the snapshot 2019-12-05T05:05:37.792 INFO:tasks.workunit.client.0.smithi042.stdout: create 110/.snap/snap-subdir-test 2019-12-05T05:05:37.829 INFO:tasks.workunit.client.0.smithi042.stdout: create 110/30 file after the snapshot 2019-12-05T05:05:37.832 INFO:tasks.workunit.client.0.smithi042.stdout: create 110/31 file after the snapshot 2019-12-05T05:05:37.835 INFO:tasks.workunit.client.0.smithi042.stdout: create 110/32 file after the snapshot 2019-12-05T05:05:37.838 INFO:tasks.workunit.client.0.smithi042.stdout: create 110/33 file after the snapshot 2019-12-05T05:05:37.841 INFO:tasks.workunit.client.0.smithi042.stdout: create 110/34 file after the snapshot 2019-12-05T05:05:37.843 INFO:tasks.workunit.client.0.smithi042.stdout: create 110/35 file after the snapshot 2019-12-05T05:05:37.845 INFO:tasks.workunit.client.0.smithi042.stdout: create 110/36 file after the snapshot 2019-12-05T05:05:37.847 INFO:tasks.workunit.client.0.smithi042.stdout: create 110/37 file after the snapshot 2019-12-05T05:05:37.849 INFO:tasks.workunit.client.0.smithi042.stdout: create 110/38 file after the snapshot 2019-12-05T05:05:37.851 INFO:tasks.workunit.client.0.smithi042.stdout: create 110/39 file after the snapshot 2019-12-05T05:05:37.853 INFO:tasks.workunit.client.0.smithi042.stdout: create 111/.snap/snap-subdir-test 2019-12-05T05:05:37.878 INFO:tasks.workunit.client.0.smithi042.stdout: create 111/30 file after the snapshot 2019-12-05T05:05:37.881 INFO:tasks.workunit.client.0.smithi042.stdout: create 111/31 file after the snapshot 2019-12-05T05:05:37.883 INFO:tasks.workunit.client.0.smithi042.stdout: create 111/32 file after the snapshot 2019-12-05T05:05:37.886 INFO:tasks.workunit.client.0.smithi042.stdout: create 111/33 file after the snapshot 2019-12-05T05:05:37.889 INFO:tasks.workunit.client.0.smithi042.stdout: create 111/34 file after the snapshot 2019-12-05T05:05:37.893 INFO:tasks.workunit.client.0.smithi042.stdout: create 111/35 file after the snapshot 2019-12-05T05:05:37.895 INFO:tasks.workunit.client.0.smithi042.stdout: create 111/36 file after the snapshot 2019-12-05T05:05:37.898 INFO:tasks.workunit.client.0.smithi042.stdout: create 111/37 file after the snapshot 2019-12-05T05:05:37.900 INFO:tasks.workunit.client.0.smithi042.stdout: create 111/38 file after the snapshot 2019-12-05T05:05:37.902 INFO:tasks.workunit.client.0.smithi042.stdout: create 111/39 file after the snapshot 2019-12-05T05:05:37.905 INFO:tasks.workunit.client.0.smithi042.stdout: create 112/.snap/snap-subdir-test 2019-12-05T05:05:37.929 INFO:tasks.workunit.client.0.smithi042.stdout: create 112/30 file after the snapshot 2019-12-05T05:05:37.931 INFO:tasks.workunit.client.0.smithi042.stdout: create 112/31 file after the snapshot 2019-12-05T05:05:37.934 INFO:tasks.workunit.client.0.smithi042.stdout: create 112/32 file after the snapshot 2019-12-05T05:05:37.937 INFO:tasks.workunit.client.0.smithi042.stdout: create 112/33 file after the snapshot 2019-12-05T05:05:37.939 INFO:tasks.workunit.client.0.smithi042.stdout: create 112/34 file after the snapshot 2019-12-05T05:05:37.941 INFO:tasks.workunit.client.0.smithi042.stdout: create 112/35 file after the snapshot 2019-12-05T05:05:37.943 INFO:tasks.workunit.client.0.smithi042.stdout: create 112/36 file after the snapshot 2019-12-05T05:05:37.945 INFO:tasks.workunit.client.0.smithi042.stdout: create 112/37 file after the snapshot 2019-12-05T05:05:37.947 INFO:tasks.workunit.client.0.smithi042.stdout: create 112/38 file after the snapshot 2019-12-05T05:05:37.949 INFO:tasks.workunit.client.0.smithi042.stdout: create 112/39 file after the snapshot 2019-12-05T05:05:37.951 INFO:tasks.workunit.client.0.smithi042.stdout: create 113/.snap/snap-subdir-test 2019-12-05T05:05:37.976 INFO:tasks.workunit.client.0.smithi042.stdout: create 113/30 file after the snapshot 2019-12-05T05:05:37.978 INFO:tasks.workunit.client.0.smithi042.stdout: create 113/31 file after the snapshot 2019-12-05T05:05:37.981 INFO:tasks.workunit.client.0.smithi042.stdout: create 113/32 file after the snapshot 2019-12-05T05:05:37.983 INFO:tasks.workunit.client.0.smithi042.stdout: create 113/33 file after the snapshot 2019-12-05T05:05:37.986 INFO:tasks.workunit.client.0.smithi042.stdout: create 113/34 file after the snapshot 2019-12-05T05:05:37.989 INFO:tasks.workunit.client.0.smithi042.stdout: create 113/35 file after the snapshot 2019-12-05T05:05:37.991 INFO:tasks.workunit.client.0.smithi042.stdout: create 113/36 file after the snapshot 2019-12-05T05:05:37.993 INFO:tasks.workunit.client.0.smithi042.stdout: create 113/37 file after the snapshot 2019-12-05T05:05:37.996 INFO:tasks.workunit.client.0.smithi042.stdout: create 113/38 file after the snapshot 2019-12-05T05:05:37.997 INFO:tasks.workunit.client.0.smithi042.stdout: create 113/39 file after the snapshot 2019-12-05T05:05:38.000 INFO:tasks.workunit.client.0.smithi042.stdout: create 114/.snap/snap-subdir-test 2019-12-05T05:05:38.040 INFO:tasks.workunit.client.0.smithi042.stdout: create 114/30 file after the snapshot 2019-12-05T05:05:38.041 INFO:tasks.workunit.client.0.smithi042.stdout: create 114/31 file after the snapshot 2019-12-05T05:05:38.041 INFO:tasks.workunit.client.0.smithi042.stdout: create 114/32 file after the snapshot 2019-12-05T05:05:38.046 INFO:tasks.workunit.client.0.smithi042.stdout: create 114/33 file after the snapshot 2019-12-05T05:05:38.047 INFO:tasks.workunit.client.0.smithi042.stdout: create 114/34 file after the snapshot 2019-12-05T05:05:38.050 INFO:tasks.workunit.client.0.smithi042.stdout: create 114/35 file after the snapshot 2019-12-05T05:05:38.053 INFO:tasks.workunit.client.0.smithi042.stdout: create 114/36 file after the snapshot 2019-12-05T05:05:38.054 INFO:tasks.workunit.client.0.smithi042.stdout: create 114/37 file after the snapshot 2019-12-05T05:05:38.055 INFO:tasks.workunit.client.0.smithi042.stdout: create 114/38 file after the snapshot 2019-12-05T05:05:38.058 INFO:tasks.workunit.client.0.smithi042.stdout: create 114/39 file after the snapshot 2019-12-05T05:05:38.060 INFO:tasks.workunit.client.0.smithi042.stdout: create 115/.snap/snap-subdir-test 2019-12-05T05:05:39.363 INFO:tasks.workunit.client.0.smithi042.stdout: create 115/30 file after the snapshot 2019-12-05T05:05:39.786 INFO:tasks.workunit.client.0.smithi042.stdout: create 115/31 file after the snapshot 2019-12-05T05:05:39.790 INFO:tasks.workunit.client.0.smithi042.stdout: create 115/32 file after the snapshot 2019-12-05T05:05:39.794 INFO:tasks.workunit.client.0.smithi042.stdout: create 115/33 file after the snapshot 2019-12-05T05:05:39.798 INFO:tasks.workunit.client.0.smithi042.stdout: create 115/34 file after the snapshot 2019-12-05T05:05:39.802 INFO:tasks.workunit.client.0.smithi042.stdout: create 115/35 file after the snapshot 2019-12-05T05:05:39.804 INFO:tasks.workunit.client.0.smithi042.stdout: create 115/36 file after the snapshot 2019-12-05T05:05:39.807 INFO:tasks.workunit.client.0.smithi042.stdout: create 115/37 file after the snapshot 2019-12-05T05:05:39.809 INFO:tasks.workunit.client.0.smithi042.stdout: create 115/38 file after the snapshot 2019-12-05T05:05:39.812 INFO:tasks.workunit.client.0.smithi042.stdout: create 115/39 file after the snapshot 2019-12-05T05:05:39.815 INFO:tasks.workunit.client.0.smithi042.stdout: create 116/.snap/snap-subdir-test 2019-12-05T05:05:39.918 INFO:tasks.workunit.client.0.smithi042.stdout: create 116/30 file after the snapshot 2019-12-05T05:05:39.921 INFO:tasks.workunit.client.0.smithi042.stdout: create 116/31 file after the snapshot 2019-12-05T05:05:39.925 INFO:tasks.workunit.client.0.smithi042.stdout: create 116/32 file after the snapshot 2019-12-05T05:05:39.928 INFO:tasks.workunit.client.0.smithi042.stdout: create 116/33 file after the snapshot 2019-12-05T05:05:39.930 INFO:tasks.workunit.client.0.smithi042.stdout: create 116/34 file after the snapshot 2019-12-05T05:05:39.933 INFO:tasks.workunit.client.0.smithi042.stdout: create 116/35 file after the snapshot 2019-12-05T05:05:39.936 INFO:tasks.workunit.client.0.smithi042.stdout: create 116/36 file after the snapshot 2019-12-05T05:05:39.938 INFO:tasks.workunit.client.0.smithi042.stdout: create 116/37 file after the snapshot 2019-12-05T05:05:39.940 INFO:tasks.workunit.client.0.smithi042.stdout: create 116/38 file after the snapshot 2019-12-05T05:05:39.942 INFO:tasks.workunit.client.0.smithi042.stdout: create 116/39 file after the snapshot 2019-12-05T05:05:39.944 INFO:tasks.workunit.client.0.smithi042.stdout: create 117/.snap/snap-subdir-test 2019-12-05T05:05:39.970 INFO:tasks.workunit.client.0.smithi042.stdout: create 117/30 file after the snapshot 2019-12-05T05:05:39.973 INFO:tasks.workunit.client.0.smithi042.stdout: create 117/31 file after the snapshot 2019-12-05T05:05:39.975 INFO:tasks.workunit.client.0.smithi042.stdout: create 117/32 file after the snapshot 2019-12-05T05:05:39.977 INFO:tasks.workunit.client.0.smithi042.stdout: create 117/33 file after the snapshot 2019-12-05T05:05:39.980 INFO:tasks.workunit.client.0.smithi042.stdout: create 117/34 file after the snapshot 2019-12-05T05:05:39.983 INFO:tasks.workunit.client.0.smithi042.stdout: create 117/35 file after the snapshot 2019-12-05T05:05:39.985 INFO:tasks.workunit.client.0.smithi042.stdout: create 117/36 file after the snapshot 2019-12-05T05:05:39.987 INFO:tasks.workunit.client.0.smithi042.stdout: create 117/37 file after the snapshot 2019-12-05T05:05:39.992 INFO:tasks.workunit.client.0.smithi042.stdout: create 117/38 file after the snapshot 2019-12-05T05:05:39.992 INFO:tasks.workunit.client.0.smithi042.stdout: create 117/39 file after the snapshot 2019-12-05T05:05:39.994 INFO:tasks.workunit.client.0.smithi042.stdout: create 118/.snap/snap-subdir-test 2019-12-05T05:05:40.019 INFO:tasks.workunit.client.0.smithi042.stdout: create 118/30 file after the snapshot 2019-12-05T05:05:40.021 INFO:tasks.workunit.client.0.smithi042.stdout: create 118/31 file after the snapshot 2019-12-05T05:05:40.024 INFO:tasks.workunit.client.0.smithi042.stdout: create 118/32 file after the snapshot 2019-12-05T05:05:40.026 INFO:tasks.workunit.client.0.smithi042.stdout: create 118/33 file after the snapshot 2019-12-05T05:05:40.029 INFO:tasks.workunit.client.0.smithi042.stdout: create 118/34 file after the snapshot 2019-12-05T05:05:40.032 INFO:tasks.workunit.client.0.smithi042.stdout: create 118/35 file after the snapshot 2019-12-05T05:05:40.034 INFO:tasks.workunit.client.0.smithi042.stdout: create 118/36 file after the snapshot 2019-12-05T05:05:40.036 INFO:tasks.workunit.client.0.smithi042.stdout: create 118/37 file after the snapshot 2019-12-05T05:05:40.038 INFO:tasks.workunit.client.0.smithi042.stdout: create 118/38 file after the snapshot 2019-12-05T05:05:40.040 INFO:tasks.workunit.client.0.smithi042.stdout: create 118/39 file after the snapshot 2019-12-05T05:05:40.042 INFO:tasks.workunit.client.0.smithi042.stdout: create 119/.snap/snap-subdir-test 2019-12-05T05:05:40.067 INFO:tasks.workunit.client.0.smithi042.stdout: create 119/30 file after the snapshot 2019-12-05T05:05:40.068 INFO:tasks.workunit.client.0.smithi042.stdout: create 119/31 file after the snapshot 2019-12-05T05:05:40.071 INFO:tasks.workunit.client.0.smithi042.stdout: create 119/32 file after the snapshot 2019-12-05T05:05:40.074 INFO:tasks.workunit.client.0.smithi042.stdout: create 119/33 file after the snapshot 2019-12-05T05:05:40.076 INFO:tasks.workunit.client.0.smithi042.stdout: create 119/34 file after the snapshot 2019-12-05T05:05:40.079 INFO:tasks.workunit.client.0.smithi042.stdout: create 119/35 file after the snapshot 2019-12-05T05:05:40.081 INFO:tasks.workunit.client.0.smithi042.stdout: create 119/36 file after the snapshot 2019-12-05T05:05:40.083 INFO:tasks.workunit.client.0.smithi042.stdout: create 119/37 file after the snapshot 2019-12-05T05:05:40.086 INFO:tasks.workunit.client.0.smithi042.stdout: create 119/38 file after the snapshot 2019-12-05T05:05:40.088 INFO:tasks.workunit.client.0.smithi042.stdout: create 119/39 file after the snapshot 2019-12-05T05:05:40.090 INFO:tasks.workunit.client.0.smithi042.stdout: create 120/.snap/snap-subdir-test 2019-12-05T05:05:40.114 INFO:tasks.workunit.client.0.smithi042.stdout: create 120/30 file after the snapshot 2019-12-05T05:05:40.117 INFO:tasks.workunit.client.0.smithi042.stdout: create 120/31 file after the snapshot 2019-12-05T05:05:40.119 INFO:tasks.workunit.client.0.smithi042.stdout: create 120/32 file after the snapshot 2019-12-05T05:05:40.121 INFO:tasks.workunit.client.0.smithi042.stdout: create 120/33 file after the snapshot 2019-12-05T05:05:40.124 INFO:tasks.workunit.client.0.smithi042.stdout: create 120/34 file after the snapshot 2019-12-05T05:05:40.126 INFO:tasks.workunit.client.0.smithi042.stdout: create 120/35 file after the snapshot 2019-12-05T05:05:40.128 INFO:tasks.workunit.client.0.smithi042.stdout: create 120/36 file after the snapshot 2019-12-05T05:05:40.130 INFO:tasks.workunit.client.0.smithi042.stdout: create 120/37 file after the snapshot 2019-12-05T05:05:40.132 INFO:tasks.workunit.client.0.smithi042.stdout: create 120/38 file after the snapshot 2019-12-05T05:05:40.135 INFO:tasks.workunit.client.0.smithi042.stdout: create 120/39 file after the snapshot 2019-12-05T05:05:40.137 INFO:tasks.workunit.client.0.smithi042.stdout: create 121/.snap/snap-subdir-test 2019-12-05T05:05:40.174 INFO:tasks.workunit.client.0.smithi042.stdout: create 121/30 file after the snapshot 2019-12-05T05:05:40.177 INFO:tasks.workunit.client.0.smithi042.stdout: create 121/31 file after the snapshot 2019-12-05T05:05:40.179 INFO:tasks.workunit.client.0.smithi042.stdout: create 121/32 file after the snapshot 2019-12-05T05:05:40.182 INFO:tasks.workunit.client.0.smithi042.stdout: create 121/33 file after the snapshot 2019-12-05T05:05:40.826 INFO:tasks.workunit.client.0.smithi042.stdout: create 121/34 file after the snapshot 2019-12-05T05:05:40.826 INFO:tasks.workunit.client.0.smithi042.stdout: create 121/35 file after the snapshot 2019-12-05T05:05:40.826 INFO:tasks.workunit.client.0.smithi042.stdout: create 121/36 file after the snapshot 2019-12-05T05:05:40.826 INFO:tasks.workunit.client.0.smithi042.stdout: create 121/37 file after the snapshot 2019-12-05T05:05:40.827 INFO:tasks.workunit.client.0.smithi042.stdout: create 121/38 file after the snapshot 2019-12-05T05:05:40.829 INFO:tasks.workunit.client.0.smithi042.stdout: create 121/39 file after the snapshot 2019-12-05T05:05:40.831 INFO:tasks.workunit.client.0.smithi042.stdout: create 122/.snap/snap-subdir-test 2019-12-05T05:05:40.865 INFO:tasks.workunit.client.0.smithi042.stdout: create 122/30 file after the snapshot 2019-12-05T05:05:40.867 INFO:tasks.workunit.client.0.smithi042.stdout: create 122/31 file after the snapshot 2019-12-05T05:05:40.870 INFO:tasks.workunit.client.0.smithi042.stdout: create 122/32 file after the snapshot 2019-12-05T05:05:40.873 INFO:tasks.workunit.client.0.smithi042.stdout: create 122/33 file after the snapshot 2019-12-05T05:05:40.875 INFO:tasks.workunit.client.0.smithi042.stdout: create 122/34 file after the snapshot 2019-12-05T05:05:40.878 INFO:tasks.workunit.client.0.smithi042.stdout: create 122/35 file after the snapshot 2019-12-05T05:05:40.880 INFO:tasks.workunit.client.0.smithi042.stdout: create 122/36 file after the snapshot 2019-12-05T05:05:40.883 INFO:tasks.workunit.client.0.smithi042.stdout: create 122/37 file after the snapshot 2019-12-05T05:05:40.886 INFO:tasks.workunit.client.0.smithi042.stdout: create 122/38 file after the snapshot 2019-12-05T05:05:40.888 INFO:tasks.workunit.client.0.smithi042.stdout: create 122/39 file after the snapshot 2019-12-05T05:05:40.890 INFO:tasks.workunit.client.0.smithi042.stdout: create 123/.snap/snap-subdir-test 2019-12-05T05:05:40.918 INFO:tasks.workunit.client.0.smithi042.stdout: create 123/30 file after the snapshot 2019-12-05T05:05:40.942 INFO:tasks.workunit.client.0.smithi042.stdout: create 123/31 file after the snapshot 2019-12-05T05:05:40.942 INFO:tasks.workunit.client.0.smithi042.stdout: create 123/32 file after the snapshot 2019-12-05T05:05:40.942 INFO:tasks.workunit.client.0.smithi042.stdout: create 123/33 file after the snapshot 2019-12-05T05:05:40.942 INFO:tasks.workunit.client.0.smithi042.stdout: create 123/34 file after the snapshot 2019-12-05T05:05:40.942 INFO:tasks.workunit.client.0.smithi042.stdout: create 123/35 file after the snapshot 2019-12-05T05:05:40.942 INFO:tasks.workunit.client.0.smithi042.stdout: create 123/36 file after the snapshot 2019-12-05T05:05:40.942 INFO:tasks.workunit.client.0.smithi042.stdout: create 123/37 file after the snapshot 2019-12-05T05:05:40.943 INFO:tasks.workunit.client.0.smithi042.stdout: create 123/38 file after the snapshot 2019-12-05T05:05:40.943 INFO:tasks.workunit.client.0.smithi042.stdout: create 123/39 file after the snapshot 2019-12-05T05:05:40.943 INFO:tasks.workunit.client.0.smithi042.stdout: create 124/.snap/snap-subdir-test 2019-12-05T05:05:40.969 INFO:tasks.workunit.client.0.smithi042.stdout: create 124/30 file after the snapshot 2019-12-05T05:05:40.971 INFO:tasks.workunit.client.0.smithi042.stdout: create 124/31 file after the snapshot 2019-12-05T05:05:40.974 INFO:tasks.workunit.client.0.smithi042.stdout: create 124/32 file after the snapshot 2019-12-05T05:05:40.976 INFO:tasks.workunit.client.0.smithi042.stdout: create 124/33 file after the snapshot 2019-12-05T05:05:40.980 INFO:tasks.workunit.client.0.smithi042.stdout: create 124/34 file after the snapshot 2019-12-05T05:05:40.984 INFO:tasks.workunit.client.0.smithi042.stdout: create 124/35 file after the snapshot 2019-12-05T05:05:40.986 INFO:tasks.workunit.client.0.smithi042.stdout: create 124/36 file after the snapshot 2019-12-05T05:05:40.988 INFO:tasks.workunit.client.0.smithi042.stdout: create 124/37 file after the snapshot 2019-12-05T05:05:40.990 INFO:tasks.workunit.client.0.smithi042.stdout: create 124/38 file after the snapshot 2019-12-05T05:05:40.992 INFO:tasks.workunit.client.0.smithi042.stdout: create 124/39 file after the snapshot 2019-12-05T05:05:40.994 INFO:tasks.workunit.client.0.smithi042.stdout: create 125/.snap/snap-subdir-test 2019-12-05T05:05:41.329 INFO:tasks.workunit.client.0.smithi042.stdout: create 125/30 file after the snapshot 2019-12-05T05:05:41.546 INFO:tasks.workunit.client.0.smithi042.stdout: create 125/31 file after the snapshot 2019-12-05T05:05:41.550 INFO:tasks.workunit.client.0.smithi042.stdout: create 125/32 file after the snapshot 2019-12-05T05:05:41.554 INFO:tasks.workunit.client.0.smithi042.stdout: create 125/33 file after the snapshot 2019-12-05T05:05:41.557 INFO:tasks.workunit.client.0.smithi042.stdout: create 125/34 file after the snapshot 2019-12-05T05:05:41.559 INFO:tasks.workunit.client.0.smithi042.stdout: create 125/35 file after the snapshot 2019-12-05T05:05:41.562 INFO:tasks.workunit.client.0.smithi042.stdout: create 125/36 file after the snapshot 2019-12-05T05:05:41.564 INFO:tasks.workunit.client.0.smithi042.stdout: create 125/37 file after the snapshot 2019-12-05T05:05:41.566 INFO:tasks.workunit.client.0.smithi042.stdout: create 125/38 file after the snapshot 2019-12-05T05:05:41.569 INFO:tasks.workunit.client.0.smithi042.stdout: create 125/39 file after the snapshot 2019-12-05T05:05:41.572 INFO:tasks.workunit.client.0.smithi042.stdout: create 126/.snap/snap-subdir-test 2019-12-05T05:05:41.608 INFO:tasks.workunit.client.0.smithi042.stdout: create 126/30 file after the snapshot 2019-12-05T05:05:41.610 INFO:tasks.workunit.client.0.smithi042.stdout: create 126/31 file after the snapshot 2019-12-05T05:05:41.612 INFO:tasks.workunit.client.0.smithi042.stdout: create 126/32 file after the snapshot 2019-12-05T05:05:41.615 INFO:tasks.workunit.client.0.smithi042.stdout: create 126/33 file after the snapshot 2019-12-05T05:05:41.617 INFO:tasks.workunit.client.0.smithi042.stdout: create 126/34 file after the snapshot 2019-12-05T05:05:41.620 INFO:tasks.workunit.client.0.smithi042.stdout: create 126/35 file after the snapshot 2019-12-05T05:05:41.623 INFO:tasks.workunit.client.0.smithi042.stdout: create 126/36 file after the snapshot 2019-12-05T05:05:41.625 INFO:tasks.workunit.client.0.smithi042.stdout: create 126/37 file after the snapshot 2019-12-05T05:05:41.628 INFO:tasks.workunit.client.0.smithi042.stdout: create 126/38 file after the snapshot 2019-12-05T05:05:41.631 INFO:tasks.workunit.client.0.smithi042.stdout: create 126/39 file after the snapshot 2019-12-05T05:05:41.634 INFO:tasks.workunit.client.0.smithi042.stdout: create 127/.snap/snap-subdir-test 2019-12-05T05:05:41.671 INFO:tasks.workunit.client.0.smithi042.stdout: create 127/30 file after the snapshot 2019-12-05T05:05:41.676 INFO:tasks.workunit.client.0.smithi042.stdout: create 127/31 file after the snapshot 2019-12-05T05:05:41.679 INFO:tasks.workunit.client.0.smithi042.stdout: create 127/32 file after the snapshot 2019-12-05T05:05:41.681 INFO:tasks.workunit.client.0.smithi042.stdout: create 127/33 file after the snapshot 2019-12-05T05:05:41.686 INFO:tasks.workunit.client.0.smithi042.stdout: create 127/34 file after the snapshot 2019-12-05T05:05:41.688 INFO:tasks.workunit.client.0.smithi042.stdout: create 127/35 file after the snapshot 2019-12-05T05:05:41.690 INFO:tasks.workunit.client.0.smithi042.stdout: create 127/36 file after the snapshot 2019-12-05T05:05:41.692 INFO:tasks.workunit.client.0.smithi042.stdout: create 127/37 file after the snapshot 2019-12-05T05:05:41.694 INFO:tasks.workunit.client.0.smithi042.stdout: create 127/38 file after the snapshot 2019-12-05T05:05:41.700 INFO:tasks.workunit.client.0.smithi042.stdout: create 127/39 file after the snapshot 2019-12-05T05:05:41.700 INFO:tasks.workunit.client.0.smithi042.stdout: create 128/.snap/snap-subdir-test 2019-12-05T05:05:41.725 INFO:tasks.workunit.client.0.smithi042.stdout: create 128/30 file after the snapshot 2019-12-05T05:05:41.728 INFO:tasks.workunit.client.0.smithi042.stdout: create 128/31 file after the snapshot 2019-12-05T05:05:41.731 INFO:tasks.workunit.client.0.smithi042.stdout: create 128/32 file after the snapshot 2019-12-05T05:05:41.733 INFO:tasks.workunit.client.0.smithi042.stdout: create 128/33 file after the snapshot 2019-12-05T05:05:41.735 INFO:tasks.workunit.client.0.smithi042.stdout: create 128/34 file after the snapshot 2019-12-05T05:05:41.738 INFO:tasks.workunit.client.0.smithi042.stdout: create 128/35 file after the snapshot 2019-12-05T05:05:41.740 INFO:tasks.workunit.client.0.smithi042.stdout: create 128/36 file after the snapshot 2019-12-05T05:05:41.742 INFO:tasks.workunit.client.0.smithi042.stdout: create 128/37 file after the snapshot 2019-12-05T05:05:41.745 INFO:tasks.workunit.client.0.smithi042.stdout: create 128/38 file after the snapshot 2019-12-05T05:05:41.747 INFO:tasks.workunit.client.0.smithi042.stdout: create 128/39 file after the snapshot 2019-12-05T05:05:41.749 INFO:tasks.workunit.client.0.smithi042.stdout: create 129/.snap/snap-subdir-test 2019-12-05T05:05:41.775 INFO:tasks.workunit.client.0.smithi042.stdout: create 129/30 file after the snapshot 2019-12-05T05:05:41.776 INFO:tasks.workunit.client.0.smithi042.stdout: create 129/31 file after the snapshot 2019-12-05T05:05:41.780 INFO:tasks.workunit.client.0.smithi042.stdout: create 129/32 file after the snapshot 2019-12-05T05:05:41.781 INFO:tasks.workunit.client.0.smithi042.stdout: create 129/33 file after the snapshot 2019-12-05T05:05:41.783 INFO:tasks.workunit.client.0.smithi042.stdout: create 129/34 file after the snapshot 2019-12-05T05:05:41.785 INFO:tasks.workunit.client.0.smithi042.stdout: create 129/35 file after the snapshot 2019-12-05T05:05:41.788 INFO:tasks.workunit.client.0.smithi042.stdout: create 129/36 file after the snapshot 2019-12-05T05:05:41.791 INFO:tasks.workunit.client.0.smithi042.stdout: create 129/37 file after the snapshot 2019-12-05T05:05:41.792 INFO:tasks.workunit.client.0.smithi042.stdout: create 129/38 file after the snapshot 2019-12-05T05:05:41.794 INFO:tasks.workunit.client.0.smithi042.stdout: create 129/39 file after the snapshot 2019-12-05T05:05:41.796 INFO:tasks.workunit.client.0.smithi042.stdout: create 130/.snap/snap-subdir-test 2019-12-05T05:05:41.821 INFO:tasks.workunit.client.0.smithi042.stdout: create 130/30 file after the snapshot 2019-12-05T05:05:41.824 INFO:tasks.workunit.client.0.smithi042.stdout: create 130/31 file after the snapshot 2019-12-05T05:05:41.827 INFO:tasks.workunit.client.0.smithi042.stdout: create 130/32 file after the snapshot 2019-12-05T05:05:41.828 INFO:tasks.workunit.client.0.smithi042.stdout: create 130/33 file after the snapshot 2019-12-05T05:05:41.832 INFO:tasks.workunit.client.0.smithi042.stdout: create 130/34 file after the snapshot 2019-12-05T05:05:41.835 INFO:tasks.workunit.client.0.smithi042.stdout: create 130/35 file after the snapshot 2019-12-05T05:05:41.836 INFO:tasks.workunit.client.0.smithi042.stdout: create 130/36 file after the snapshot 2019-12-05T05:05:41.839 INFO:tasks.workunit.client.0.smithi042.stdout: create 130/37 file after the snapshot 2019-12-05T05:05:41.841 INFO:tasks.workunit.client.0.smithi042.stdout: create 130/38 file after the snapshot 2019-12-05T05:05:41.842 INFO:tasks.workunit.client.0.smithi042.stdout: create 130/39 file after the snapshot 2019-12-05T05:05:41.844 INFO:tasks.workunit.client.0.smithi042.stdout: create 131/.snap/snap-subdir-test 2019-12-05T05:05:41.869 INFO:tasks.workunit.client.0.smithi042.stdout: create 131/30 file after the snapshot 2019-12-05T05:05:41.871 INFO:tasks.workunit.client.0.smithi042.stdout: create 131/31 file after the snapshot 2019-12-05T05:05:41.874 INFO:tasks.workunit.client.0.smithi042.stdout: create 131/32 file after the snapshot 2019-12-05T05:05:41.876 INFO:tasks.workunit.client.0.smithi042.stdout: create 131/33 file after the snapshot 2019-12-05T05:05:41.878 INFO:tasks.workunit.client.0.smithi042.stdout: create 131/34 file after the snapshot 2019-12-05T05:05:41.881 INFO:tasks.workunit.client.0.smithi042.stdout: create 131/35 file after the snapshot 2019-12-05T05:05:41.884 INFO:tasks.workunit.client.0.smithi042.stdout: create 131/36 file after the snapshot 2019-12-05T05:05:41.886 INFO:tasks.workunit.client.0.smithi042.stdout: create 131/37 file after the snapshot 2019-12-05T05:05:41.888 INFO:tasks.workunit.client.0.smithi042.stdout: create 131/38 file after the snapshot 2019-12-05T05:05:41.891 INFO:tasks.workunit.client.0.smithi042.stdout: create 131/39 file after the snapshot 2019-12-05T05:05:41.893 INFO:tasks.workunit.client.0.smithi042.stdout: create 132/.snap/snap-subdir-test 2019-12-05T05:05:41.929 INFO:tasks.workunit.client.0.smithi042.stdout: create 132/30 file after the snapshot 2019-12-05T05:05:41.932 INFO:tasks.workunit.client.0.smithi042.stdout: create 132/31 file after the snapshot 2019-12-05T05:05:41.934 INFO:tasks.workunit.client.0.smithi042.stdout: create 132/32 file after the snapshot 2019-12-05T05:05:41.938 INFO:tasks.workunit.client.0.smithi042.stdout: create 132/33 file after the snapshot 2019-12-05T05:05:41.943 INFO:tasks.workunit.client.0.smithi042.stdout: create 132/34 file after the snapshot 2019-12-05T05:05:41.945 INFO:tasks.workunit.client.0.smithi042.stdout: create 132/35 file after the snapshot 2019-12-05T05:05:41.947 INFO:tasks.workunit.client.0.smithi042.stdout: create 132/36 file after the snapshot 2019-12-05T05:05:41.950 INFO:tasks.workunit.client.0.smithi042.stdout: create 132/37 file after the snapshot 2019-12-05T05:05:41.952 INFO:tasks.workunit.client.0.smithi042.stdout: create 132/38 file after the snapshot 2019-12-05T05:05:41.954 INFO:tasks.workunit.client.0.smithi042.stdout: create 132/39 file after the snapshot 2019-12-05T05:05:41.956 INFO:tasks.workunit.client.0.smithi042.stdout: create 133/.snap/snap-subdir-test 2019-12-05T05:05:41.981 INFO:tasks.workunit.client.0.smithi042.stdout: create 133/30 file after the snapshot 2019-12-05T05:05:41.984 INFO:tasks.workunit.client.0.smithi042.stdout: create 133/31 file after the snapshot 2019-12-05T05:05:41.986 INFO:tasks.workunit.client.0.smithi042.stdout: create 133/32 file after the snapshot 2019-12-05T05:05:41.989 INFO:tasks.workunit.client.0.smithi042.stdout: create 133/33 file after the snapshot 2019-12-05T05:05:41.991 INFO:tasks.workunit.client.0.smithi042.stdout: create 133/34 file after the snapshot 2019-12-05T05:05:41.994 INFO:tasks.workunit.client.0.smithi042.stdout: create 133/35 file after the snapshot 2019-12-05T05:05:41.996 INFO:tasks.workunit.client.0.smithi042.stdout: create 133/36 file after the snapshot 2019-12-05T05:05:41.998 INFO:tasks.workunit.client.0.smithi042.stdout: create 133/37 file after the snapshot 2019-12-05T05:05:42.000 INFO:tasks.workunit.client.0.smithi042.stdout: create 133/38 file after the snapshot 2019-12-05T05:05:42.003 INFO:tasks.workunit.client.0.smithi042.stdout: create 133/39 file after the snapshot 2019-12-05T05:05:42.005 INFO:tasks.workunit.client.0.smithi042.stdout: create 134/.snap/snap-subdir-test 2019-12-05T05:05:42.029 INFO:tasks.workunit.client.0.smithi042.stdout: create 134/30 file after the snapshot 2019-12-05T05:05:42.031 INFO:tasks.workunit.client.0.smithi042.stdout: create 134/31 file after the snapshot 2019-12-05T05:05:42.034 INFO:tasks.workunit.client.0.smithi042.stdout: create 134/32 file after the snapshot 2019-12-05T05:05:42.036 INFO:tasks.workunit.client.0.smithi042.stdout: create 134/33 file after the snapshot 2019-12-05T05:05:42.039 INFO:tasks.workunit.client.0.smithi042.stdout: create 134/34 file after the snapshot 2019-12-05T05:05:42.041 INFO:tasks.workunit.client.0.smithi042.stdout: create 134/35 file after the snapshot 2019-12-05T05:05:42.043 INFO:tasks.workunit.client.0.smithi042.stdout: create 134/36 file after the snapshot 2019-12-05T05:05:42.045 INFO:tasks.workunit.client.0.smithi042.stdout: create 134/37 file after the snapshot 2019-12-05T05:05:42.048 INFO:tasks.workunit.client.0.smithi042.stdout: create 134/38 file after the snapshot 2019-12-05T05:05:42.050 INFO:tasks.workunit.client.0.smithi042.stdout: create 134/39 file after the snapshot 2019-12-05T05:05:42.052 INFO:tasks.workunit.client.0.smithi042.stdout: create 135/.snap/snap-subdir-test 2019-12-05T05:05:42.074 INFO:tasks.workunit.client.0.smithi042.stdout: create 135/30 file after the snapshot 2019-12-05T05:05:42.076 INFO:tasks.workunit.client.0.smithi042.stdout: create 135/31 file after the snapshot 2019-12-05T05:05:42.078 INFO:tasks.workunit.client.0.smithi042.stdout: create 135/32 file after the snapshot 2019-12-05T05:05:42.081 INFO:tasks.workunit.client.0.smithi042.stdout: create 135/33 file after the snapshot 2019-12-05T05:05:42.084 INFO:tasks.workunit.client.0.smithi042.stdout: create 135/34 file after the snapshot 2019-12-05T05:05:42.086 INFO:tasks.workunit.client.0.smithi042.stdout: create 135/35 file after the snapshot 2019-12-05T05:05:42.088 INFO:tasks.workunit.client.0.smithi042.stdout: create 135/36 file after the snapshot 2019-12-05T05:05:42.090 INFO:tasks.workunit.client.0.smithi042.stdout: create 135/37 file after the snapshot 2019-12-05T05:05:42.092 INFO:tasks.workunit.client.0.smithi042.stdout: create 135/38 file after the snapshot 2019-12-05T05:05:42.094 INFO:tasks.workunit.client.0.smithi042.stdout: create 135/39 file after the snapshot 2019-12-05T05:05:42.096 INFO:tasks.workunit.client.0.smithi042.stdout: create 136/.snap/snap-subdir-test 2019-12-05T05:05:42.934 INFO:tasks.workunit.client.0.smithi042.stdout: create 136/30 file after the snapshot 2019-12-05T05:05:42.938 INFO:tasks.workunit.client.0.smithi042.stdout: create 136/31 file after the snapshot 2019-12-05T05:05:42.942 INFO:tasks.workunit.client.0.smithi042.stdout: create 136/32 file after the snapshot 2019-12-05T05:05:42.948 INFO:tasks.workunit.client.0.smithi042.stdout: create 136/33 file after the snapshot 2019-12-05T05:05:42.950 INFO:tasks.workunit.client.0.smithi042.stdout: create 136/34 file after the snapshot 2019-12-05T05:05:42.953 INFO:tasks.workunit.client.0.smithi042.stdout: create 136/35 file after the snapshot 2019-12-05T05:05:42.956 INFO:tasks.workunit.client.0.smithi042.stdout: create 136/36 file after the snapshot 2019-12-05T05:05:42.958 INFO:tasks.workunit.client.0.smithi042.stdout: create 136/37 file after the snapshot 2019-12-05T05:05:42.961 INFO:tasks.workunit.client.0.smithi042.stdout: create 136/38 file after the snapshot 2019-12-05T05:05:42.963 INFO:tasks.workunit.client.0.smithi042.stdout: create 136/39 file after the snapshot 2019-12-05T05:05:42.966 INFO:tasks.workunit.client.0.smithi042.stdout: create 137/.snap/snap-subdir-test 2019-12-05T05:05:42.997 INFO:tasks.workunit.client.0.smithi042.stdout: create 137/30 file after the snapshot 2019-12-05T05:05:42.998 INFO:tasks.workunit.client.0.smithi042.stdout: create 137/31 file after the snapshot 2019-12-05T05:05:43.000 INFO:tasks.workunit.client.0.smithi042.stdout: create 137/32 file after the snapshot 2019-12-05T05:05:43.005 INFO:tasks.workunit.client.0.smithi042.stdout: create 137/33 file after the snapshot 2019-12-05T05:05:43.007 INFO:tasks.workunit.client.0.smithi042.stdout: create 137/34 file after the snapshot 2019-12-05T05:05:43.010 INFO:tasks.workunit.client.0.smithi042.stdout: create 137/35 file after the snapshot 2019-12-05T05:05:43.012 INFO:tasks.workunit.client.0.smithi042.stdout: create 137/36 file after the snapshot 2019-12-05T05:05:43.015 INFO:tasks.workunit.client.0.smithi042.stdout: create 137/37 file after the snapshot 2019-12-05T05:05:43.018 INFO:tasks.workunit.client.0.smithi042.stdout: create 137/38 file after the snapshot 2019-12-05T05:05:43.028 INFO:tasks.workunit.client.0.smithi042.stdout: create 137/39 file after the snapshot 2019-12-05T05:05:43.028 INFO:tasks.workunit.client.0.smithi042.stdout: create 138/.snap/snap-subdir-test 2019-12-05T05:05:43.054 INFO:tasks.workunit.client.0.smithi042.stdout: create 138/30 file after the snapshot 2019-12-05T05:05:43.057 INFO:tasks.workunit.client.0.smithi042.stdout: create 138/31 file after the snapshot 2019-12-05T05:05:43.061 INFO:tasks.workunit.client.0.smithi042.stdout: create 138/32 file after the snapshot 2019-12-05T05:05:43.064 INFO:tasks.workunit.client.0.smithi042.stdout: create 138/33 file after the snapshot 2019-12-05T05:05:43.067 INFO:tasks.workunit.client.0.smithi042.stdout: create 138/34 file after the snapshot 2019-12-05T05:05:43.069 INFO:tasks.workunit.client.0.smithi042.stdout: create 138/35 file after the snapshot 2019-12-05T05:05:43.072 INFO:tasks.workunit.client.0.smithi042.stdout: create 138/36 file after the snapshot 2019-12-05T05:05:43.074 INFO:tasks.workunit.client.0.smithi042.stdout: create 138/37 file after the snapshot 2019-12-05T05:05:43.076 INFO:tasks.workunit.client.0.smithi042.stdout: create 138/38 file after the snapshot 2019-12-05T05:05:43.078 INFO:tasks.workunit.client.0.smithi042.stdout: create 138/39 file after the snapshot 2019-12-05T05:05:43.081 INFO:tasks.workunit.client.0.smithi042.stdout: create 139/.snap/snap-subdir-test 2019-12-05T05:05:43.107 INFO:tasks.workunit.client.0.smithi042.stdout: create 139/30 file after the snapshot 2019-12-05T05:05:43.109 INFO:tasks.workunit.client.0.smithi042.stdout: create 139/31 file after the snapshot 2019-12-05T05:05:43.111 INFO:tasks.workunit.client.0.smithi042.stdout: create 139/32 file after the snapshot 2019-12-05T05:05:43.114 INFO:tasks.workunit.client.0.smithi042.stdout: create 139/33 file after the snapshot 2019-12-05T05:05:43.117 INFO:tasks.workunit.client.0.smithi042.stdout: create 139/34 file after the snapshot 2019-12-05T05:05:43.119 INFO:tasks.workunit.client.0.smithi042.stdout: create 139/35 file after the snapshot 2019-12-05T05:05:43.128 INFO:tasks.workunit.client.0.smithi042.stdout: create 139/36 file after the snapshot 2019-12-05T05:05:43.128 INFO:tasks.workunit.client.0.smithi042.stdout: create 139/37 file after the snapshot 2019-12-05T05:05:43.128 INFO:tasks.workunit.client.0.smithi042.stdout: create 139/38 file after the snapshot 2019-12-05T05:05:43.129 INFO:tasks.workunit.client.0.smithi042.stdout: create 139/39 file after the snapshot 2019-12-05T05:05:43.137 INFO:tasks.workunit.client.0.smithi042.stdout: create 140/.snap/snap-subdir-test 2019-12-05T05:05:43.678 INFO:tasks.workunit.client.0.smithi042.stdout: create 140/30 file after the snapshot 2019-12-05T05:05:43.683 INFO:tasks.workunit.client.0.smithi042.stdout: create 140/31 file after the snapshot 2019-12-05T05:05:43.688 INFO:tasks.workunit.client.0.smithi042.stdout: create 140/32 file after the snapshot 2019-12-05T05:05:43.691 INFO:tasks.workunit.client.0.smithi042.stdout: create 140/33 file after the snapshot 2019-12-05T05:05:43.694 INFO:tasks.workunit.client.0.smithi042.stdout: create 140/34 file after the snapshot 2019-12-05T05:05:43.696 INFO:tasks.workunit.client.0.smithi042.stdout: create 140/35 file after the snapshot 2019-12-05T05:05:44.105 INFO:tasks.workunit.client.0.smithi042.stdout: create 140/36 file after the snapshot 2019-12-05T05:05:44.111 INFO:tasks.workunit.client.0.smithi042.stdout: create 140/37 file after the snapshot 2019-12-05T05:05:44.113 INFO:tasks.workunit.client.0.smithi042.stdout: create 140/38 file after the snapshot 2019-12-05T05:05:44.116 INFO:tasks.workunit.client.0.smithi042.stdout: create 140/39 file after the snapshot 2019-12-05T05:05:44.119 INFO:tasks.workunit.client.0.smithi042.stdout: create 141/.snap/snap-subdir-test 2019-12-05T05:05:45.059 INFO:tasks.workunit.client.0.smithi042.stdout: create 141/30 file after the snapshot 2019-12-05T05:05:45.065 INFO:tasks.workunit.client.0.smithi042.stdout: create 141/31 file after the snapshot 2019-12-05T05:05:45.071 INFO:tasks.workunit.client.0.smithi042.stdout: create 141/32 file after the snapshot 2019-12-05T05:05:45.076 INFO:tasks.workunit.client.0.smithi042.stdout: create 141/33 file after the snapshot 2019-12-05T05:05:45.079 INFO:tasks.workunit.client.0.smithi042.stdout: create 141/34 file after the snapshot 2019-12-05T05:05:45.082 INFO:tasks.workunit.client.0.smithi042.stdout: create 141/35 file after the snapshot 2019-12-05T05:05:45.084 INFO:tasks.workunit.client.0.smithi042.stdout: create 141/36 file after the snapshot 2019-12-05T05:05:45.086 INFO:tasks.workunit.client.0.smithi042.stdout: create 141/37 file after the snapshot 2019-12-05T05:05:45.089 INFO:tasks.workunit.client.0.smithi042.stdout: create 141/38 file after the snapshot 2019-12-05T05:05:45.092 INFO:tasks.workunit.client.0.smithi042.stdout: create 141/39 file after the snapshot 2019-12-05T05:05:45.094 INFO:tasks.workunit.client.0.smithi042.stdout: create 142/.snap/snap-subdir-test 2019-12-05T05:05:45.121 INFO:tasks.workunit.client.0.smithi042.stdout: create 142/30 file after the snapshot 2019-12-05T05:05:45.123 INFO:tasks.workunit.client.0.smithi042.stdout: create 142/31 file after the snapshot 2019-12-05T05:05:45.126 INFO:tasks.workunit.client.0.smithi042.stdout: create 142/32 file after the snapshot 2019-12-05T05:05:45.129 INFO:tasks.workunit.client.0.smithi042.stdout: create 142/33 file after the snapshot 2019-12-05T05:05:45.132 INFO:tasks.workunit.client.0.smithi042.stdout: create 142/34 file after the snapshot 2019-12-05T05:05:45.135 INFO:tasks.workunit.client.0.smithi042.stdout: create 142/35 file after the snapshot 2019-12-05T05:05:45.137 INFO:tasks.workunit.client.0.smithi042.stdout: create 142/36 file after the snapshot 2019-12-05T05:05:45.140 INFO:tasks.workunit.client.0.smithi042.stdout: create 142/37 file after the snapshot 2019-12-05T05:05:45.142 INFO:tasks.workunit.client.0.smithi042.stdout: create 142/38 file after the snapshot 2019-12-05T05:05:45.144 INFO:tasks.workunit.client.0.smithi042.stdout: create 142/39 file after the snapshot 2019-12-05T05:05:45.146 INFO:tasks.workunit.client.0.smithi042.stdout: create 143/.snap/snap-subdir-test 2019-12-05T05:05:45.172 INFO:tasks.workunit.client.0.smithi042.stdout: create 143/30 file after the snapshot 2019-12-05T05:05:45.175 INFO:tasks.workunit.client.0.smithi042.stdout: create 143/31 file after the snapshot 2019-12-05T05:05:45.177 INFO:tasks.workunit.client.0.smithi042.stdout: create 143/32 file after the snapshot 2019-12-05T05:05:45.180 INFO:tasks.workunit.client.0.smithi042.stdout: create 143/33 file after the snapshot 2019-12-05T05:05:45.182 INFO:tasks.workunit.client.0.smithi042.stdout: create 143/34 file after the snapshot 2019-12-05T05:05:45.185 INFO:tasks.workunit.client.0.smithi042.stdout: create 143/35 file after the snapshot 2019-12-05T05:05:45.187 INFO:tasks.workunit.client.0.smithi042.stdout: create 143/36 file after the snapshot 2019-12-05T05:05:45.190 INFO:tasks.workunit.client.0.smithi042.stdout: create 143/37 file after the snapshot 2019-12-05T05:05:45.192 INFO:tasks.workunit.client.0.smithi042.stdout: create 143/38 file after the snapshot 2019-12-05T05:05:45.194 INFO:tasks.workunit.client.0.smithi042.stdout: create 143/39 file after the snapshot 2019-12-05T05:05:45.197 INFO:tasks.workunit.client.0.smithi042.stdout: create 144/.snap/snap-subdir-test 2019-12-05T05:05:45.224 INFO:tasks.workunit.client.0.smithi042.stdout: create 144/30 file after the snapshot 2019-12-05T05:05:45.227 INFO:tasks.workunit.client.0.smithi042.stdout: create 144/31 file after the snapshot 2019-12-05T05:05:45.230 INFO:tasks.workunit.client.0.smithi042.stdout: create 144/32 file after the snapshot 2019-12-05T05:05:45.233 INFO:tasks.workunit.client.0.smithi042.stdout: create 144/33 file after the snapshot 2019-12-05T05:05:45.235 INFO:tasks.workunit.client.0.smithi042.stdout: create 144/34 file after the snapshot 2019-12-05T05:05:45.237 INFO:tasks.workunit.client.0.smithi042.stdout: create 144/35 file after the snapshot 2019-12-05T05:05:45.240 INFO:tasks.workunit.client.0.smithi042.stdout: create 144/36 file after the snapshot 2019-12-05T05:05:45.242 INFO:tasks.workunit.client.0.smithi042.stdout: create 144/37 file after the snapshot 2019-12-05T05:05:45.245 INFO:tasks.workunit.client.0.smithi042.stdout: create 144/38 file after the snapshot 2019-12-05T05:05:45.247 INFO:tasks.workunit.client.0.smithi042.stdout: create 144/39 file after the snapshot 2019-12-05T05:05:45.249 INFO:tasks.workunit.client.0.smithi042.stdout: create 145/.snap/snap-subdir-test 2019-12-05T05:05:45.283 INFO:tasks.workunit.client.0.smithi042.stdout: create 145/30 file after the snapshot 2019-12-05T05:05:45.286 INFO:tasks.workunit.client.0.smithi042.stdout: create 145/31 file after the snapshot 2019-12-05T05:05:45.290 INFO:tasks.workunit.client.0.smithi042.stdout: create 145/32 file after the snapshot 2019-12-05T05:05:45.294 INFO:tasks.workunit.client.0.smithi042.stdout: create 145/33 file after the snapshot 2019-12-05T05:05:45.298 INFO:tasks.workunit.client.0.smithi042.stdout: create 145/34 file after the snapshot 2019-12-05T05:05:45.301 INFO:tasks.workunit.client.0.smithi042.stdout: create 145/35 file after the snapshot 2019-12-05T05:05:45.303 INFO:tasks.workunit.client.0.smithi042.stdout: create 145/36 file after the snapshot 2019-12-05T05:05:45.305 INFO:tasks.workunit.client.0.smithi042.stdout: create 145/37 file after the snapshot 2019-12-05T05:05:45.307 INFO:tasks.workunit.client.0.smithi042.stdout: create 145/38 file after the snapshot 2019-12-05T05:05:45.310 INFO:tasks.workunit.client.0.smithi042.stdout: create 145/39 file after the snapshot 2019-12-05T05:05:45.313 INFO:tasks.workunit.client.0.smithi042.stdout: create 146/.snap/snap-subdir-test 2019-12-05T05:05:45.338 INFO:tasks.workunit.client.0.smithi042.stdout: create 146/30 file after the snapshot 2019-12-05T05:05:45.370 INFO:tasks.workunit.client.0.smithi042.stdout: create 146/31 file after the snapshot 2019-12-05T05:05:45.381 INFO:tasks.workunit.client.0.smithi042.stdout: create 146/32 file after the snapshot 2019-12-05T05:05:45.381 INFO:tasks.workunit.client.0.smithi042.stdout: create 146/33 file after the snapshot 2019-12-05T05:05:45.381 INFO:tasks.workunit.client.0.smithi042.stdout: create 146/34 file after the snapshot 2019-12-05T05:05:45.381 INFO:tasks.workunit.client.0.smithi042.stdout: create 146/35 file after the snapshot 2019-12-05T05:05:45.382 INFO:tasks.workunit.client.0.smithi042.stdout: create 146/36 file after the snapshot 2019-12-05T05:05:45.384 INFO:tasks.workunit.client.0.smithi042.stdout: create 146/37 file after the snapshot 2019-12-05T05:05:45.386 INFO:tasks.workunit.client.0.smithi042.stdout: create 146/38 file after the snapshot 2019-12-05T05:05:45.388 INFO:tasks.workunit.client.0.smithi042.stdout: create 146/39 file after the snapshot 2019-12-05T05:05:45.390 INFO:tasks.workunit.client.0.smithi042.stdout: create 147/.snap/snap-subdir-test 2019-12-05T05:05:45.428 INFO:tasks.workunit.client.0.smithi042.stdout: create 147/30 file after the snapshot 2019-12-05T05:05:45.431 INFO:tasks.workunit.client.0.smithi042.stdout: create 147/31 file after the snapshot 2019-12-05T05:05:45.433 INFO:tasks.workunit.client.0.smithi042.stdout: create 147/32 file after the snapshot 2019-12-05T05:05:45.436 INFO:tasks.workunit.client.0.smithi042.stdout: create 147/33 file after the snapshot 2019-12-05T05:05:45.438 INFO:tasks.workunit.client.0.smithi042.stdout: create 147/34 file after the snapshot 2019-12-05T05:05:45.441 INFO:tasks.workunit.client.0.smithi042.stdout: create 147/35 file after the snapshot 2019-12-05T05:05:45.443 INFO:tasks.workunit.client.0.smithi042.stdout: create 147/36 file after the snapshot 2019-12-05T05:05:45.445 INFO:tasks.workunit.client.0.smithi042.stdout: create 147/37 file after the snapshot 2019-12-05T05:05:45.448 INFO:tasks.workunit.client.0.smithi042.stdout: create 147/38 file after the snapshot 2019-12-05T05:05:45.450 INFO:tasks.workunit.client.0.smithi042.stdout: create 147/39 file after the snapshot 2019-12-05T05:05:45.453 INFO:tasks.workunit.client.0.smithi042.stdout: create 148/.snap/snap-subdir-test 2019-12-05T05:05:45.480 INFO:tasks.workunit.client.0.smithi042.stdout: create 148/30 file after the snapshot 2019-12-05T05:05:45.482 INFO:tasks.workunit.client.0.smithi042.stdout: create 148/31 file after the snapshot 2019-12-05T05:05:45.485 INFO:tasks.workunit.client.0.smithi042.stdout: create 148/32 file after the snapshot 2019-12-05T05:05:45.488 INFO:tasks.workunit.client.0.smithi042.stdout: create 148/33 file after the snapshot 2019-12-05T05:05:45.490 INFO:tasks.workunit.client.0.smithi042.stdout: create 148/34 file after the snapshot 2019-12-05T05:05:45.493 INFO:tasks.workunit.client.0.smithi042.stdout: create 148/35 file after the snapshot 2019-12-05T05:05:45.495 INFO:tasks.workunit.client.0.smithi042.stdout: create 148/36 file after the snapshot 2019-12-05T05:05:45.498 INFO:tasks.workunit.client.0.smithi042.stdout: create 148/37 file after the snapshot 2019-12-05T05:05:45.500 INFO:tasks.workunit.client.0.smithi042.stdout: create 148/38 file after the snapshot 2019-12-05T05:05:45.502 INFO:tasks.workunit.client.0.smithi042.stdout: create 148/39 file after the snapshot 2019-12-05T05:05:45.504 INFO:tasks.workunit.client.0.smithi042.stdout: create 149/.snap/snap-subdir-test 2019-12-05T05:05:45.528 INFO:tasks.workunit.client.0.smithi042.stdout: create 149/30 file after the snapshot 2019-12-05T05:05:45.530 INFO:tasks.workunit.client.0.smithi042.stdout: create 149/31 file after the snapshot 2019-12-05T05:05:45.532 INFO:tasks.workunit.client.0.smithi042.stdout: create 149/32 file after the snapshot 2019-12-05T05:05:45.535 INFO:tasks.workunit.client.0.smithi042.stdout: create 149/33 file after the snapshot 2019-12-05T05:05:45.541 INFO:tasks.workunit.client.0.smithi042.stdout: create 149/34 file after the snapshot 2019-12-05T05:05:45.541 INFO:tasks.workunit.client.0.smithi042.stdout: create 149/35 file after the snapshot 2019-12-05T05:05:45.541 INFO:tasks.workunit.client.0.smithi042.stdout: create 149/36 file after the snapshot 2019-12-05T05:05:45.542 INFO:tasks.workunit.client.0.smithi042.stdout: create 149/37 file after the snapshot 2019-12-05T05:05:45.546 INFO:tasks.workunit.client.0.smithi042.stdout: create 149/38 file after the snapshot 2019-12-05T05:05:45.547 INFO:tasks.workunit.client.0.smithi042.stdout: create 149/39 file after the snapshot 2019-12-05T05:05:45.550 INFO:tasks.workunit.client.0.smithi042.stdout: create 150/.snap/snap-subdir-test 2019-12-05T05:05:45.571 INFO:tasks.workunit.client.0.smithi042.stdout: create 150/30 file after the snapshot 2019-12-05T05:05:45.574 INFO:tasks.workunit.client.0.smithi042.stdout: create 150/31 file after the snapshot 2019-12-05T05:05:45.577 INFO:tasks.workunit.client.0.smithi042.stdout: create 150/32 file after the snapshot 2019-12-05T05:05:45.580 INFO:tasks.workunit.client.0.smithi042.stdout: create 150/33 file after the snapshot 2019-12-05T05:05:45.583 INFO:tasks.workunit.client.0.smithi042.stdout: create 150/34 file after the snapshot 2019-12-05T05:05:45.584 INFO:tasks.workunit.client.0.smithi042.stdout: create 150/35 file after the snapshot 2019-12-05T05:05:45.587 INFO:tasks.workunit.client.0.smithi042.stdout: create 150/36 file after the snapshot 2019-12-05T05:05:45.976 INFO:tasks.workunit.client.0.smithi042.stdout: create 150/37 file after the snapshot 2019-12-05T05:05:45.981 INFO:tasks.workunit.client.0.smithi042.stdout: create 150/38 file after the snapshot 2019-12-05T05:05:45.983 INFO:tasks.workunit.client.0.smithi042.stdout: create 150/39 file after the snapshot 2019-12-05T05:05:45.987 INFO:tasks.workunit.client.0.smithi042.stdout: create 151/.snap/snap-subdir-test 2019-12-05T05:05:46.013 INFO:tasks.workunit.client.0.smithi042.stdout: create 151/30 file after the snapshot 2019-12-05T05:05:46.016 INFO:tasks.workunit.client.0.smithi042.stdout: create 151/31 file after the snapshot 2019-12-05T05:05:46.019 INFO:tasks.workunit.client.0.smithi042.stdout: create 151/32 file after the snapshot 2019-12-05T05:05:46.021 INFO:tasks.workunit.client.0.smithi042.stdout: create 151/33 file after the snapshot 2019-12-05T05:05:46.024 INFO:tasks.workunit.client.0.smithi042.stdout: create 151/34 file after the snapshot 2019-12-05T05:05:46.027 INFO:tasks.workunit.client.0.smithi042.stdout: create 151/35 file after the snapshot 2019-12-05T05:05:46.029 INFO:tasks.workunit.client.0.smithi042.stdout: create 151/36 file after the snapshot 2019-12-05T05:05:46.031 INFO:tasks.workunit.client.0.smithi042.stdout: create 151/37 file after the snapshot 2019-12-05T05:05:46.035 INFO:tasks.workunit.client.0.smithi042.stdout: create 151/38 file after the snapshot 2019-12-05T05:05:46.037 INFO:tasks.workunit.client.0.smithi042.stdout: create 151/39 file after the snapshot 2019-12-05T05:05:46.039 INFO:tasks.workunit.client.0.smithi042.stdout: create 152/.snap/snap-subdir-test 2019-12-05T05:05:46.063 INFO:tasks.workunit.client.0.smithi042.stdout: create 152/30 file after the snapshot 2019-12-05T05:05:46.066 INFO:tasks.workunit.client.0.smithi042.stdout: create 152/31 file after the snapshot 2019-12-05T05:05:46.069 INFO:tasks.workunit.client.0.smithi042.stdout: create 152/32 file after the snapshot 2019-12-05T05:05:46.072 INFO:tasks.workunit.client.0.smithi042.stdout: create 152/33 file after the snapshot 2019-12-05T05:05:46.073 INFO:tasks.workunit.client.0.smithi042.stdout: create 152/34 file after the snapshot 2019-12-05T05:05:46.076 INFO:tasks.workunit.client.0.smithi042.stdout: create 152/35 file after the snapshot 2019-12-05T05:05:46.077 INFO:tasks.workunit.client.0.smithi042.stdout: create 152/36 file after the snapshot 2019-12-05T05:05:46.080 INFO:tasks.workunit.client.0.smithi042.stdout: create 152/37 file after the snapshot 2019-12-05T05:05:46.083 INFO:tasks.workunit.client.0.smithi042.stdout: create 152/38 file after the snapshot 2019-12-05T05:05:46.085 INFO:tasks.workunit.client.0.smithi042.stdout: create 152/39 file after the snapshot 2019-12-05T05:05:46.087 INFO:tasks.workunit.client.0.smithi042.stdout: create 153/.snap/snap-subdir-test 2019-12-05T05:05:46.113 INFO:tasks.workunit.client.0.smithi042.stdout: create 153/30 file after the snapshot 2019-12-05T05:05:46.114 INFO:tasks.workunit.client.0.smithi042.stdout: create 153/31 file after the snapshot 2019-12-05T05:05:46.117 INFO:tasks.workunit.client.0.smithi042.stdout: create 153/32 file after the snapshot 2019-12-05T05:05:46.119 INFO:tasks.workunit.client.0.smithi042.stdout: create 153/33 file after the snapshot 2019-12-05T05:05:46.122 INFO:tasks.workunit.client.0.smithi042.stdout: create 153/34 file after the snapshot 2019-12-05T05:05:46.124 INFO:tasks.workunit.client.0.smithi042.stdout: create 153/35 file after the snapshot 2019-12-05T05:05:46.127 INFO:tasks.workunit.client.0.smithi042.stdout: create 153/36 file after the snapshot 2019-12-05T05:05:46.130 INFO:tasks.workunit.client.0.smithi042.stdout: create 153/37 file after the snapshot 2019-12-05T05:05:46.131 INFO:tasks.workunit.client.0.smithi042.stdout: create 153/38 file after the snapshot 2019-12-05T05:05:46.134 INFO:tasks.workunit.client.0.smithi042.stdout: create 153/39 file after the snapshot 2019-12-05T05:05:46.136 INFO:tasks.workunit.client.0.smithi042.stdout: create 154/.snap/snap-subdir-test 2019-12-05T05:05:46.172 INFO:tasks.workunit.client.0.smithi042.stdout: create 154/30 file after the snapshot 2019-12-05T05:05:46.175 INFO:tasks.workunit.client.0.smithi042.stdout: create 154/31 file after the snapshot 2019-12-05T05:05:46.179 INFO:tasks.workunit.client.0.smithi042.stdout: create 154/32 file after the snapshot 2019-12-05T05:05:46.181 INFO:tasks.workunit.client.0.smithi042.stdout: create 154/33 file after the snapshot 2019-12-05T05:05:46.183 INFO:tasks.workunit.client.0.smithi042.stdout: create 154/34 file after the snapshot 2019-12-05T05:05:46.185 INFO:tasks.workunit.client.0.smithi042.stdout: create 154/35 file after the snapshot 2019-12-05T05:05:46.187 INFO:tasks.workunit.client.0.smithi042.stdout: create 154/36 file after the snapshot 2019-12-05T05:05:46.189 INFO:tasks.workunit.client.0.smithi042.stdout: create 154/37 file after the snapshot 2019-12-05T05:05:46.192 INFO:tasks.workunit.client.0.smithi042.stdout: create 154/38 file after the snapshot 2019-12-05T05:05:46.194 INFO:tasks.workunit.client.0.smithi042.stdout: create 154/39 file after the snapshot 2019-12-05T05:05:46.196 INFO:tasks.workunit.client.0.smithi042.stdout: create 155/.snap/snap-subdir-test 2019-12-05T05:05:46.231 INFO:tasks.workunit.client.0.smithi042.stdout: create 155/30 file after the snapshot 2019-12-05T05:05:46.233 INFO:tasks.workunit.client.0.smithi042.stdout: create 155/31 file after the snapshot 2019-12-05T05:05:46.238 INFO:tasks.workunit.client.0.smithi042.stdout: create 155/32 file after the snapshot 2019-12-05T05:05:46.242 INFO:tasks.workunit.client.0.smithi042.stdout: create 155/33 file after the snapshot 2019-12-05T05:05:46.245 INFO:tasks.workunit.client.0.smithi042.stdout: create 155/34 file after the snapshot 2019-12-05T05:05:46.247 INFO:tasks.workunit.client.0.smithi042.stdout: create 155/35 file after the snapshot 2019-12-05T05:05:46.250 INFO:tasks.workunit.client.0.smithi042.stdout: create 155/36 file after the snapshot 2019-12-05T05:05:46.252 INFO:tasks.workunit.client.0.smithi042.stdout: create 155/37 file after the snapshot 2019-12-05T05:05:46.254 INFO:tasks.workunit.client.0.smithi042.stdout: create 155/38 file after the snapshot 2019-12-05T05:05:46.256 INFO:tasks.workunit.client.0.smithi042.stdout: create 155/39 file after the snapshot 2019-12-05T05:05:46.259 INFO:tasks.workunit.client.0.smithi042.stdout: create 156/.snap/snap-subdir-test 2019-12-05T05:05:46.282 INFO:tasks.workunit.client.0.smithi042.stdout: create 156/30 file after the snapshot 2019-12-05T05:05:46.284 INFO:tasks.workunit.client.0.smithi042.stdout: create 156/31 file after the snapshot 2019-12-05T05:05:46.287 INFO:tasks.workunit.client.0.smithi042.stdout: create 156/32 file after the snapshot 2019-12-05T05:05:46.289 INFO:tasks.workunit.client.0.smithi042.stdout: create 156/33 file after the snapshot 2019-12-05T05:05:46.291 INFO:tasks.workunit.client.0.smithi042.stdout: create 156/34 file after the snapshot 2019-12-05T05:05:46.293 INFO:tasks.workunit.client.0.smithi042.stdout: create 156/35 file after the snapshot 2019-12-05T05:05:46.296 INFO:tasks.workunit.client.0.smithi042.stdout: create 156/36 file after the snapshot 2019-12-05T05:05:46.298 INFO:tasks.workunit.client.0.smithi042.stdout: create 156/37 file after the snapshot 2019-12-05T05:05:46.300 INFO:tasks.workunit.client.0.smithi042.stdout: create 156/38 file after the snapshot 2019-12-05T05:05:46.302 INFO:tasks.workunit.client.0.smithi042.stdout: create 156/39 file after the snapshot 2019-12-05T05:05:46.304 INFO:tasks.workunit.client.0.smithi042.stdout: create 157/.snap/snap-subdir-test 2019-12-05T05:05:46.338 INFO:tasks.workunit.client.0.smithi042.stdout: create 157/30 file after the snapshot 2019-12-05T05:05:46.362 INFO:tasks.workunit.client.0.smithi042.stdout: create 157/31 file after the snapshot 2019-12-05T05:05:46.364 INFO:tasks.workunit.client.0.smithi042.stdout: create 157/32 file after the snapshot 2019-12-05T05:05:46.367 INFO:tasks.workunit.client.0.smithi042.stdout: create 157/33 file after the snapshot 2019-12-05T05:05:46.369 INFO:tasks.workunit.client.0.smithi042.stdout: create 157/34 file after the snapshot 2019-12-05T05:05:46.371 INFO:tasks.workunit.client.0.smithi042.stdout: create 157/35 file after the snapshot 2019-12-05T05:05:46.374 INFO:tasks.workunit.client.0.smithi042.stdout: create 157/36 file after the snapshot 2019-12-05T05:05:46.379 INFO:tasks.workunit.client.0.smithi042.stdout: create 157/37 file after the snapshot 2019-12-05T05:05:46.380 INFO:tasks.workunit.client.0.smithi042.stdout: create 157/38 file after the snapshot 2019-12-05T05:05:46.382 INFO:tasks.workunit.client.0.smithi042.stdout: create 157/39 file after the snapshot 2019-12-05T05:05:46.383 INFO:tasks.workunit.client.0.smithi042.stdout: create 158/.snap/snap-subdir-test 2019-12-05T05:05:47.134 INFO:tasks.workunit.client.0.smithi042.stdout: create 158/30 file after the snapshot 2019-12-05T05:05:47.139 INFO:tasks.workunit.client.0.smithi042.stdout: create 158/31 file after the snapshot 2019-12-05T05:05:47.142 INFO:tasks.workunit.client.0.smithi042.stdout: create 158/32 file after the snapshot 2019-12-05T05:05:47.145 INFO:tasks.workunit.client.0.smithi042.stdout: create 158/33 file after the snapshot 2019-12-05T05:05:47.148 INFO:tasks.workunit.client.0.smithi042.stdout: create 158/34 file after the snapshot 2019-12-05T05:05:47.150 INFO:tasks.workunit.client.0.smithi042.stdout: create 158/35 file after the snapshot 2019-12-05T05:05:47.152 INFO:tasks.workunit.client.0.smithi042.stdout: create 158/36 file after the snapshot 2019-12-05T05:05:47.155 INFO:tasks.workunit.client.0.smithi042.stdout: create 158/37 file after the snapshot 2019-12-05T05:05:47.156 INFO:tasks.workunit.client.0.smithi042.stdout: create 158/38 file after the snapshot 2019-12-05T05:05:47.159 INFO:tasks.workunit.client.0.smithi042.stdout: create 158/39 file after the snapshot 2019-12-05T05:05:47.161 INFO:tasks.workunit.client.0.smithi042.stdout: create 159/.snap/snap-subdir-test 2019-12-05T05:05:47.188 INFO:tasks.workunit.client.0.smithi042.stdout: create 159/30 file after the snapshot 2019-12-05T05:05:47.190 INFO:tasks.workunit.client.0.smithi042.stdout: create 159/31 file after the snapshot 2019-12-05T05:05:47.193 INFO:tasks.workunit.client.0.smithi042.stdout: create 159/32 file after the snapshot 2019-12-05T05:05:47.195 INFO:tasks.workunit.client.0.smithi042.stdout: create 159/33 file after the snapshot 2019-12-05T05:05:47.198 INFO:tasks.workunit.client.0.smithi042.stdout: create 159/34 file after the snapshot 2019-12-05T05:05:47.200 INFO:tasks.workunit.client.0.smithi042.stdout: create 159/35 file after the snapshot 2019-12-05T05:05:47.203 INFO:tasks.workunit.client.0.smithi042.stdout: create 159/36 file after the snapshot 2019-12-05T05:05:47.205 INFO:tasks.workunit.client.0.smithi042.stdout: create 159/37 file after the snapshot 2019-12-05T05:05:47.207 INFO:tasks.workunit.client.0.smithi042.stdout: create 159/38 file after the snapshot 2019-12-05T05:05:47.209 INFO:tasks.workunit.client.0.smithi042.stdout: create 159/39 file after the snapshot 2019-12-05T05:05:47.212 INFO:tasks.workunit.client.0.smithi042.stdout: create 160/.snap/snap-subdir-test 2019-12-05T05:05:47.249 INFO:tasks.workunit.client.0.smithi042.stdout: create 160/30 file after the snapshot 2019-12-05T05:05:47.250 INFO:tasks.workunit.client.0.smithi042.stdout: create 160/31 file after the snapshot 2019-12-05T05:05:47.262 INFO:tasks.workunit.client.0.smithi042.stdout: create 160/32 file after the snapshot 2019-12-05T05:05:47.263 INFO:tasks.workunit.client.0.smithi042.stdout: create 160/33 file after the snapshot 2019-12-05T05:05:47.263 INFO:tasks.workunit.client.0.smithi042.stdout: create 160/34 file after the snapshot 2019-12-05T05:05:47.263 INFO:tasks.workunit.client.0.smithi042.stdout: create 160/35 file after the snapshot 2019-12-05T05:05:47.264 INFO:tasks.workunit.client.0.smithi042.stdout: create 160/36 file after the snapshot 2019-12-05T05:05:47.267 INFO:tasks.workunit.client.0.smithi042.stdout: create 160/37 file after the snapshot 2019-12-05T05:05:47.269 INFO:tasks.workunit.client.0.smithi042.stdout: create 160/38 file after the snapshot 2019-12-05T05:05:47.271 INFO:tasks.workunit.client.0.smithi042.stdout: create 160/39 file after the snapshot 2019-12-05T05:05:47.274 INFO:tasks.workunit.client.0.smithi042.stdout: create 161/.snap/snap-subdir-test 2019-12-05T05:05:47.297 INFO:tasks.workunit.client.0.smithi042.stdout: create 161/30 file after the snapshot 2019-12-05T05:05:47.300 INFO:tasks.workunit.client.0.smithi042.stdout: create 161/31 file after the snapshot 2019-12-05T05:05:47.302 INFO:tasks.workunit.client.0.smithi042.stdout: create 161/32 file after the snapshot 2019-12-05T05:05:47.305 INFO:tasks.workunit.client.0.smithi042.stdout: create 161/33 file after the snapshot 2019-12-05T05:05:47.307 INFO:tasks.workunit.client.0.smithi042.stdout: create 161/34 file after the snapshot 2019-12-05T05:05:47.309 INFO:tasks.workunit.client.0.smithi042.stdout: create 161/35 file after the snapshot 2019-12-05T05:05:47.312 INFO:tasks.workunit.client.0.smithi042.stdout: create 161/36 file after the snapshot 2019-12-05T05:05:47.314 INFO:tasks.workunit.client.0.smithi042.stdout: create 161/37 file after the snapshot 2019-12-05T05:05:47.316 INFO:tasks.workunit.client.0.smithi042.stdout: create 161/38 file after the snapshot 2019-12-05T05:05:47.318 INFO:tasks.workunit.client.0.smithi042.stdout: create 161/39 file after the snapshot 2019-12-05T05:05:47.320 INFO:tasks.workunit.client.0.smithi042.stdout: create 162/.snap/snap-subdir-test 2019-12-05T05:05:47.421 INFO:tasks.workunit.client.0.smithi042.stdout: create 162/30 file after the snapshot 2019-12-05T05:05:47.423 INFO:tasks.workunit.client.0.smithi042.stdout: create 162/31 file after the snapshot 2019-12-05T05:05:47.427 INFO:tasks.workunit.client.0.smithi042.stdout: create 162/32 file after the snapshot 2019-12-05T05:05:47.431 INFO:tasks.workunit.client.0.smithi042.stdout: create 162/33 file after the snapshot 2019-12-05T05:05:47.434 INFO:tasks.workunit.client.0.smithi042.stdout: create 162/34 file after the snapshot 2019-12-05T05:05:47.437 INFO:tasks.workunit.client.0.smithi042.stdout: create 162/35 file after the snapshot 2019-12-05T05:05:47.440 INFO:tasks.workunit.client.0.smithi042.stdout: create 162/36 file after the snapshot 2019-12-05T05:05:47.442 INFO:tasks.workunit.client.0.smithi042.stdout: create 162/37 file after the snapshot 2019-12-05T05:05:47.444 INFO:tasks.workunit.client.0.smithi042.stdout: create 162/38 file after the snapshot 2019-12-05T05:05:47.446 INFO:tasks.workunit.client.0.smithi042.stdout: create 162/39 file after the snapshot 2019-12-05T05:05:47.449 INFO:tasks.workunit.client.0.smithi042.stdout: create 163/.snap/snap-subdir-test 2019-12-05T05:05:47.470 INFO:tasks.workunit.client.0.smithi042.stdout: create 163/30 file after the snapshot 2019-12-05T05:05:47.472 INFO:tasks.workunit.client.0.smithi042.stdout: create 163/31 file after the snapshot 2019-12-05T05:05:47.474 INFO:tasks.workunit.client.0.smithi042.stdout: create 163/32 file after the snapshot 2019-12-05T05:05:47.477 INFO:tasks.workunit.client.0.smithi042.stdout: create 163/33 file after the snapshot 2019-12-05T05:05:47.479 INFO:tasks.workunit.client.0.smithi042.stdout: create 163/34 file after the snapshot 2019-12-05T05:05:47.481 INFO:tasks.workunit.client.0.smithi042.stdout: create 163/35 file after the snapshot 2019-12-05T05:05:47.483 INFO:tasks.workunit.client.0.smithi042.stdout: create 163/36 file after the snapshot 2019-12-05T05:05:47.485 INFO:tasks.workunit.client.0.smithi042.stdout: create 163/37 file after the snapshot 2019-12-05T05:05:47.488 INFO:tasks.workunit.client.0.smithi042.stdout: create 163/38 file after the snapshot 2019-12-05T05:05:47.490 INFO:tasks.workunit.client.0.smithi042.stdout: create 163/39 file after the snapshot 2019-12-05T05:05:47.492 INFO:tasks.workunit.client.0.smithi042.stdout: create 164/.snap/snap-subdir-test 2019-12-05T05:05:47.514 INFO:tasks.workunit.client.0.smithi042.stdout: create 164/30 file after the snapshot 2019-12-05T05:05:47.516 INFO:tasks.workunit.client.0.smithi042.stdout: create 164/31 file after the snapshot 2019-12-05T05:05:47.518 INFO:tasks.workunit.client.0.smithi042.stdout: create 164/32 file after the snapshot 2019-12-05T05:05:47.520 INFO:tasks.workunit.client.0.smithi042.stdout: create 164/33 file after the snapshot 2019-12-05T05:05:47.523 INFO:tasks.workunit.client.0.smithi042.stdout: create 164/34 file after the snapshot 2019-12-05T05:05:47.525 INFO:tasks.workunit.client.0.smithi042.stdout: create 164/35 file after the snapshot 2019-12-05T05:05:47.527 INFO:tasks.workunit.client.0.smithi042.stdout: create 164/36 file after the snapshot 2019-12-05T05:05:47.529 INFO:tasks.workunit.client.0.smithi042.stdout: create 164/37 file after the snapshot 2019-12-05T05:05:47.532 INFO:tasks.workunit.client.0.smithi042.stdout: create 164/38 file after the snapshot 2019-12-05T05:05:47.534 INFO:tasks.workunit.client.0.smithi042.stdout: create 164/39 file after the snapshot 2019-12-05T05:05:47.536 INFO:tasks.workunit.client.0.smithi042.stdout: create 165/.snap/snap-subdir-test 2019-12-05T05:05:47.558 INFO:tasks.workunit.client.0.smithi042.stdout: create 165/30 file after the snapshot 2019-12-05T05:05:47.561 INFO:tasks.workunit.client.0.smithi042.stdout: create 165/31 file after the snapshot 2019-12-05T05:05:47.564 INFO:tasks.workunit.client.0.smithi042.stdout: create 165/32 file after the snapshot 2019-12-05T05:05:47.567 INFO:tasks.workunit.client.0.smithi042.stdout: create 165/33 file after the snapshot 2019-12-05T05:05:47.569 INFO:tasks.workunit.client.0.smithi042.stdout: create 165/34 file after the snapshot 2019-12-05T05:05:47.571 INFO:tasks.workunit.client.0.smithi042.stdout: create 165/35 file after the snapshot 2019-12-05T05:05:47.573 INFO:tasks.workunit.client.0.smithi042.stdout: create 165/36 file after the snapshot 2019-12-05T05:05:47.575 INFO:tasks.workunit.client.0.smithi042.stdout: create 165/37 file after the snapshot 2019-12-05T05:05:47.578 INFO:tasks.workunit.client.0.smithi042.stdout: create 165/38 file after the snapshot 2019-12-05T05:05:47.580 INFO:tasks.workunit.client.0.smithi042.stdout: create 165/39 file after the snapshot 2019-12-05T05:05:47.582 INFO:tasks.workunit.client.0.smithi042.stdout: create 166/.snap/snap-subdir-test 2019-12-05T05:05:48.518 INFO:tasks.workunit.client.0.smithi042.stdout: create 166/30 file after the snapshot 2019-12-05T05:05:48.523 INFO:tasks.workunit.client.0.smithi042.stdout: create 166/31 file after the snapshot 2019-12-05T05:05:48.526 INFO:tasks.workunit.client.0.smithi042.stdout: create 166/32 file after the snapshot 2019-12-05T05:05:48.529 INFO:tasks.workunit.client.0.smithi042.stdout: create 166/33 file after the snapshot 2019-12-05T05:05:48.532 INFO:tasks.workunit.client.0.smithi042.stdout: create 166/34 file after the snapshot 2019-12-05T05:05:48.535 INFO:tasks.workunit.client.0.smithi042.stdout: create 166/35 file after the snapshot 2019-12-05T05:05:48.537 INFO:tasks.workunit.client.0.smithi042.stdout: create 166/36 file after the snapshot 2019-12-05T05:05:48.540 INFO:tasks.workunit.client.0.smithi042.stdout: create 166/37 file after the snapshot 2019-12-05T05:05:48.544 INFO:tasks.workunit.client.0.smithi042.stdout: create 166/38 file after the snapshot 2019-12-05T05:05:48.546 INFO:tasks.workunit.client.0.smithi042.stdout: create 166/39 file after the snapshot 2019-12-05T05:05:48.549 INFO:tasks.workunit.client.0.smithi042.stdout: create 167/.snap/snap-subdir-test 2019-12-05T05:05:48.572 INFO:tasks.workunit.client.0.smithi042.stdout: create 167/30 file after the snapshot 2019-12-05T05:05:48.575 INFO:tasks.workunit.client.0.smithi042.stdout: create 167/31 file after the snapshot 2019-12-05T05:05:48.578 INFO:tasks.workunit.client.0.smithi042.stdout: create 167/32 file after the snapshot 2019-12-05T05:05:48.580 INFO:tasks.workunit.client.0.smithi042.stdout: create 167/33 file after the snapshot 2019-12-05T05:05:48.583 INFO:tasks.workunit.client.0.smithi042.stdout: create 167/34 file after the snapshot 2019-12-05T05:05:48.586 INFO:tasks.workunit.client.0.smithi042.stdout: create 167/35 file after the snapshot 2019-12-05T05:05:48.589 INFO:tasks.workunit.client.0.smithi042.stdout: create 167/36 file after the snapshot 2019-12-05T05:05:48.591 INFO:tasks.workunit.client.0.smithi042.stdout: create 167/37 file after the snapshot 2019-12-05T05:05:48.593 INFO:tasks.workunit.client.0.smithi042.stdout: create 167/38 file after the snapshot 2019-12-05T05:05:48.596 INFO:tasks.workunit.client.0.smithi042.stdout: create 167/39 file after the snapshot 2019-12-05T05:05:48.598 INFO:tasks.workunit.client.0.smithi042.stdout: create 168/.snap/snap-subdir-test 2019-12-05T05:05:48.619 INFO:tasks.workunit.client.0.smithi042.stdout: create 168/30 file after the snapshot 2019-12-05T05:05:48.622 INFO:tasks.workunit.client.0.smithi042.stdout: create 168/31 file after the snapshot 2019-12-05T05:05:48.624 INFO:tasks.workunit.client.0.smithi042.stdout: create 168/32 file after the snapshot 2019-12-05T05:05:48.627 INFO:tasks.workunit.client.0.smithi042.stdout: create 168/33 file after the snapshot 2019-12-05T05:05:48.630 INFO:tasks.workunit.client.0.smithi042.stdout: create 168/34 file after the snapshot 2019-12-05T05:05:48.633 INFO:tasks.workunit.client.0.smithi042.stdout: create 168/35 file after the snapshot 2019-12-05T05:05:48.634 INFO:tasks.workunit.client.0.smithi042.stdout: create 168/36 file after the snapshot 2019-12-05T05:05:48.637 INFO:tasks.workunit.client.0.smithi042.stdout: create 168/37 file after the snapshot 2019-12-05T05:05:48.640 INFO:tasks.workunit.client.0.smithi042.stdout: create 168/38 file after the snapshot 2019-12-05T05:05:48.642 INFO:tasks.workunit.client.0.smithi042.stdout: create 168/39 file after the snapshot 2019-12-05T05:05:48.645 INFO:tasks.workunit.client.0.smithi042.stdout: create 169/.snap/snap-subdir-test 2019-12-05T05:05:48.726 INFO:tasks.workunit.client.0.smithi042.stdout: create 169/30 file after the snapshot 2019-12-05T05:05:48.728 INFO:tasks.workunit.client.0.smithi042.stdout: create 169/31 file after the snapshot 2019-12-05T05:05:48.730 INFO:tasks.workunit.client.0.smithi042.stdout: create 169/32 file after the snapshot 2019-12-05T05:05:48.734 INFO:tasks.workunit.client.0.smithi042.stdout: create 169/33 file after the snapshot 2019-12-05T05:05:48.738 INFO:tasks.workunit.client.0.smithi042.stdout: create 169/34 file after the snapshot 2019-12-05T05:05:48.740 INFO:tasks.workunit.client.0.smithi042.stdout: create 169/35 file after the snapshot 2019-12-05T05:05:48.742 INFO:tasks.workunit.client.0.smithi042.stdout: create 169/36 file after the snapshot 2019-12-05T05:05:48.745 INFO:tasks.workunit.client.0.smithi042.stdout: create 169/37 file after the snapshot 2019-12-05T05:05:48.747 INFO:tasks.workunit.client.0.smithi042.stdout: create 169/38 file after the snapshot 2019-12-05T05:05:48.749 INFO:tasks.workunit.client.0.smithi042.stdout: create 169/39 file after the snapshot 2019-12-05T05:05:48.752 INFO:tasks.workunit.client.0.smithi042.stdout: create 170/.snap/snap-subdir-test 2019-12-05T05:05:48.775 INFO:tasks.workunit.client.0.smithi042.stdout: create 170/30 file after the snapshot 2019-12-05T05:05:48.777 INFO:tasks.workunit.client.0.smithi042.stdout: create 170/31 file after the snapshot 2019-12-05T05:05:48.779 INFO:tasks.workunit.client.0.smithi042.stdout: create 170/32 file after the snapshot 2019-12-05T05:05:48.783 INFO:tasks.workunit.client.0.smithi042.stdout: create 170/33 file after the snapshot 2019-12-05T05:05:48.785 INFO:tasks.workunit.client.0.smithi042.stdout: create 170/34 file after the snapshot 2019-12-05T05:05:48.788 INFO:tasks.workunit.client.0.smithi042.stdout: create 170/35 file after the snapshot 2019-12-05T05:05:48.790 INFO:tasks.workunit.client.0.smithi042.stdout: create 170/36 file after the snapshot 2019-12-05T05:05:48.793 INFO:tasks.workunit.client.0.smithi042.stdout: create 170/37 file after the snapshot 2019-12-05T05:05:48.795 INFO:tasks.workunit.client.0.smithi042.stdout: create 170/38 file after the snapshot 2019-12-05T05:05:48.797 INFO:tasks.workunit.client.0.smithi042.stdout: create 170/39 file after the snapshot 2019-12-05T05:05:48.800 INFO:tasks.workunit.client.0.smithi042.stdout: create 171/.snap/snap-subdir-test 2019-12-05T05:05:48.823 INFO:tasks.workunit.client.0.smithi042.stdout: create 171/30 file after the snapshot 2019-12-05T05:05:48.826 INFO:tasks.workunit.client.0.smithi042.stdout: create 171/31 file after the snapshot 2019-12-05T05:05:48.828 INFO:tasks.workunit.client.0.smithi042.stdout: create 171/32 file after the snapshot 2019-12-05T05:05:48.830 INFO:tasks.workunit.client.0.smithi042.stdout: create 171/33 file after the snapshot 2019-12-05T05:05:48.832 INFO:tasks.workunit.client.0.smithi042.stdout: create 171/34 file after the snapshot 2019-12-05T05:05:48.834 INFO:tasks.workunit.client.0.smithi042.stdout: create 171/35 file after the snapshot 2019-12-05T05:05:48.836 INFO:tasks.workunit.client.0.smithi042.stdout: create 171/36 file after the snapshot 2019-12-05T05:05:48.839 INFO:tasks.workunit.client.0.smithi042.stdout: create 171/37 file after the snapshot 2019-12-05T05:05:48.841 INFO:tasks.workunit.client.0.smithi042.stdout: create 171/38 file after the snapshot 2019-12-05T05:05:48.843 INFO:tasks.workunit.client.0.smithi042.stdout: create 171/39 file after the snapshot 2019-12-05T05:05:48.845 INFO:tasks.workunit.client.0.smithi042.stdout: create 172/.snap/snap-subdir-test 2019-12-05T05:05:48.866 INFO:tasks.workunit.client.0.smithi042.stdout: create 172/30 file after the snapshot 2019-12-05T05:05:48.868 INFO:tasks.workunit.client.0.smithi042.stdout: create 172/31 file after the snapshot 2019-12-05T05:05:48.870 INFO:tasks.workunit.client.0.smithi042.stdout: create 172/32 file after the snapshot 2019-12-05T05:05:48.873 INFO:tasks.workunit.client.0.smithi042.stdout: create 172/33 file after the snapshot 2019-12-05T05:05:48.875 INFO:tasks.workunit.client.0.smithi042.stdout: create 172/34 file after the snapshot 2019-12-05T05:05:48.878 INFO:tasks.workunit.client.0.smithi042.stdout: create 172/35 file after the snapshot 2019-12-05T05:05:48.879 INFO:tasks.workunit.client.0.smithi042.stdout: create 172/36 file after the snapshot 2019-12-05T05:05:48.882 INFO:tasks.workunit.client.0.smithi042.stdout: create 172/37 file after the snapshot 2019-12-05T05:05:48.884 INFO:tasks.workunit.client.0.smithi042.stdout: create 172/38 file after the snapshot 2019-12-05T05:05:48.886 INFO:tasks.workunit.client.0.smithi042.stdout: create 172/39 file after the snapshot 2019-12-05T05:05:48.888 INFO:tasks.workunit.client.0.smithi042.stdout: create 173/.snap/snap-subdir-test 2019-12-05T05:05:48.911 INFO:tasks.workunit.client.0.smithi042.stdout: create 173/30 file after the snapshot 2019-12-05T05:05:48.913 INFO:tasks.workunit.client.0.smithi042.stdout: create 173/31 file after the snapshot 2019-12-05T05:05:48.915 INFO:tasks.workunit.client.0.smithi042.stdout: create 173/32 file after the snapshot 2019-12-05T05:05:48.918 INFO:tasks.workunit.client.0.smithi042.stdout: create 173/33 file after the snapshot 2019-12-05T05:05:48.920 INFO:tasks.workunit.client.0.smithi042.stdout: create 173/34 file after the snapshot 2019-12-05T05:05:48.921 INFO:tasks.workunit.client.0.smithi042.stdout: create 173/35 file after the snapshot 2019-12-05T05:05:48.924 INFO:tasks.workunit.client.0.smithi042.stdout: create 173/36 file after the snapshot 2019-12-05T05:05:48.926 INFO:tasks.workunit.client.0.smithi042.stdout: create 173/37 file after the snapshot 2019-12-05T05:05:48.928 INFO:tasks.workunit.client.0.smithi042.stdout: create 173/38 file after the snapshot 2019-12-05T05:05:48.930 INFO:tasks.workunit.client.0.smithi042.stdout: create 173/39 file after the snapshot 2019-12-05T05:05:48.932 INFO:tasks.workunit.client.0.smithi042.stdout: create 174/.snap/snap-subdir-test 2019-12-05T05:05:48.956 INFO:tasks.workunit.client.0.smithi042.stdout: create 174/30 file after the snapshot 2019-12-05T05:05:48.958 INFO:tasks.workunit.client.0.smithi042.stdout: create 174/31 file after the snapshot 2019-12-05T05:05:48.960 INFO:tasks.workunit.client.0.smithi042.stdout: create 174/32 file after the snapshot 2019-12-05T05:05:48.962 INFO:tasks.workunit.client.0.smithi042.stdout: create 174/33 file after the snapshot 2019-12-05T05:05:48.964 INFO:tasks.workunit.client.0.smithi042.stdout: create 174/34 file after the snapshot 2019-12-05T05:05:48.967 INFO:tasks.workunit.client.0.smithi042.stdout: create 174/35 file after the snapshot 2019-12-05T05:05:48.969 INFO:tasks.workunit.client.0.smithi042.stdout: create 174/36 file after the snapshot 2019-12-05T05:05:48.971 INFO:tasks.workunit.client.0.smithi042.stdout: create 174/37 file after the snapshot 2019-12-05T05:05:48.973 INFO:tasks.workunit.client.0.smithi042.stdout: create 174/38 file after the snapshot 2019-12-05T05:05:48.976 INFO:tasks.workunit.client.0.smithi042.stdout: create 174/39 file after the snapshot 2019-12-05T05:05:48.978 INFO:tasks.workunit.client.0.smithi042.stdout: create 175/.snap/snap-subdir-test 2019-12-05T05:05:49.000 INFO:tasks.workunit.client.0.smithi042.stdout: create 175/30 file after the snapshot 2019-12-05T05:05:49.002 INFO:tasks.workunit.client.0.smithi042.stdout: create 175/31 file after the snapshot 2019-12-05T05:05:49.004 INFO:tasks.workunit.client.0.smithi042.stdout: create 175/32 file after the snapshot 2019-12-05T05:05:49.007 INFO:tasks.workunit.client.0.smithi042.stdout: create 175/33 file after the snapshot 2019-12-05T05:05:49.010 INFO:tasks.workunit.client.0.smithi042.stdout: create 175/34 file after the snapshot 2019-12-05T05:05:49.012 INFO:tasks.workunit.client.0.smithi042.stdout: create 175/35 file after the snapshot 2019-12-05T05:05:49.015 INFO:tasks.workunit.client.0.smithi042.stdout: create 175/36 file after the snapshot 2019-12-05T05:05:49.018 INFO:tasks.workunit.client.0.smithi042.stdout: create 175/37 file after the snapshot 2019-12-05T05:05:49.021 INFO:tasks.workunit.client.0.smithi042.stdout: create 175/38 file after the snapshot 2019-12-05T05:05:49.023 INFO:tasks.workunit.client.0.smithi042.stdout: create 175/39 file after the snapshot 2019-12-05T05:05:49.025 INFO:tasks.workunit.client.0.smithi042.stdout: create 176/.snap/snap-subdir-test 2019-12-05T05:05:49.046 INFO:tasks.workunit.client.0.smithi042.stdout: create 176/30 file after the snapshot 2019-12-05T05:05:49.048 INFO:tasks.workunit.client.0.smithi042.stdout: create 176/31 file after the snapshot 2019-12-05T05:05:49.050 INFO:tasks.workunit.client.0.smithi042.stdout: create 176/32 file after the snapshot 2019-12-05T05:05:49.053 INFO:tasks.workunit.client.0.smithi042.stdout: create 176/33 file after the snapshot 2019-12-05T05:05:49.055 INFO:tasks.workunit.client.0.smithi042.stdout: create 176/34 file after the snapshot 2019-12-05T05:05:49.058 INFO:tasks.workunit.client.0.smithi042.stdout: create 176/35 file after the snapshot 2019-12-05T05:05:49.060 INFO:tasks.workunit.client.0.smithi042.stdout: create 176/36 file after the snapshot 2019-12-05T05:05:49.063 INFO:tasks.workunit.client.0.smithi042.stdout: create 176/37 file after the snapshot 2019-12-05T05:05:49.065 INFO:tasks.workunit.client.0.smithi042.stdout: create 176/38 file after the snapshot 2019-12-05T05:05:49.067 INFO:tasks.workunit.client.0.smithi042.stdout: create 176/39 file after the snapshot 2019-12-05T05:05:49.069 INFO:tasks.workunit.client.0.smithi042.stdout: create 177/.snap/snap-subdir-test 2019-12-05T05:05:49.091 INFO:tasks.workunit.client.0.smithi042.stdout: create 177/30 file after the snapshot 2019-12-05T05:05:49.093 INFO:tasks.workunit.client.0.smithi042.stdout: create 177/31 file after the snapshot 2019-12-05T05:05:49.095 INFO:tasks.workunit.client.0.smithi042.stdout: create 177/32 file after the snapshot 2019-12-05T05:05:49.097 INFO:tasks.workunit.client.0.smithi042.stdout: create 177/33 file after the snapshot 2019-12-05T05:05:49.099 INFO:tasks.workunit.client.0.smithi042.stdout: create 177/34 file after the snapshot 2019-12-05T05:05:49.101 INFO:tasks.workunit.client.0.smithi042.stdout: create 177/35 file after the snapshot 2019-12-05T05:05:49.104 INFO:tasks.workunit.client.0.smithi042.stdout: create 177/36 file after the snapshot 2019-12-05T05:05:49.106 INFO:tasks.workunit.client.0.smithi042.stdout: create 177/37 file after the snapshot 2019-12-05T05:05:49.108 INFO:tasks.workunit.client.0.smithi042.stdout: create 177/38 file after the snapshot 2019-12-05T05:05:49.110 INFO:tasks.workunit.client.0.smithi042.stdout: create 177/39 file after the snapshot 2019-12-05T05:05:49.113 INFO:tasks.workunit.client.0.smithi042.stdout: create 178/.snap/snap-subdir-test 2019-12-05T05:05:49.135 INFO:tasks.workunit.client.0.smithi042.stdout: create 178/30 file after the snapshot 2019-12-05T05:05:49.137 INFO:tasks.workunit.client.0.smithi042.stdout: create 178/31 file after the snapshot 2019-12-05T05:05:49.139 INFO:tasks.workunit.client.0.smithi042.stdout: create 178/32 file after the snapshot 2019-12-05T05:05:49.141 INFO:tasks.workunit.client.0.smithi042.stdout: create 178/33 file after the snapshot 2019-12-05T05:05:49.144 INFO:tasks.workunit.client.0.smithi042.stdout: create 178/34 file after the snapshot 2019-12-05T05:05:49.147 INFO:tasks.workunit.client.0.smithi042.stdout: create 178/35 file after the snapshot 2019-12-05T05:05:49.149 INFO:tasks.workunit.client.0.smithi042.stdout: create 178/36 file after the snapshot 2019-12-05T05:05:49.151 INFO:tasks.workunit.client.0.smithi042.stdout: create 178/37 file after the snapshot 2019-12-05T05:05:49.154 INFO:tasks.workunit.client.0.smithi042.stdout: create 178/38 file after the snapshot 2019-12-05T05:05:49.156 INFO:tasks.workunit.client.0.smithi042.stdout: create 178/39 file after the snapshot 2019-12-05T05:05:49.157 INFO:tasks.workunit.client.0.smithi042.stdout: create 179/.snap/snap-subdir-test 2019-12-05T05:05:49.182 INFO:tasks.workunit.client.0.smithi042.stdout: create 179/30 file after the snapshot 2019-12-05T05:05:49.184 INFO:tasks.workunit.client.0.smithi042.stdout: create 179/31 file after the snapshot 2019-12-05T05:05:49.187 INFO:tasks.workunit.client.0.smithi042.stdout: create 179/32 file after the snapshot 2019-12-05T05:05:49.190 INFO:tasks.workunit.client.0.smithi042.stdout: create 179/33 file after the snapshot 2019-12-05T05:05:49.192 INFO:tasks.workunit.client.0.smithi042.stdout: create 179/34 file after the snapshot 2019-12-05T05:05:49.195 INFO:tasks.workunit.client.0.smithi042.stdout: create 179/35 file after the snapshot 2019-12-05T05:05:49.197 INFO:tasks.workunit.client.0.smithi042.stdout: create 179/36 file after the snapshot 2019-12-05T05:05:49.200 INFO:tasks.workunit.client.0.smithi042.stdout: create 179/37 file after the snapshot 2019-12-05T05:05:49.202 INFO:tasks.workunit.client.0.smithi042.stdout: create 179/38 file after the snapshot 2019-12-05T05:05:49.204 INFO:tasks.workunit.client.0.smithi042.stdout: create 179/39 file after the snapshot 2019-12-05T05:05:49.206 INFO:tasks.workunit.client.0.smithi042.stdout: create 180/.snap/snap-subdir-test 2019-12-05T05:05:49.237 INFO:tasks.workunit.client.0.smithi042.stdout: create 180/30 file after the snapshot 2019-12-05T05:05:49.239 INFO:tasks.workunit.client.0.smithi042.stdout: create 180/31 file after the snapshot 2019-12-05T05:05:49.241 INFO:tasks.workunit.client.0.smithi042.stdout: create 180/32 file after the snapshot 2019-12-05T05:05:49.245 INFO:tasks.workunit.client.0.smithi042.stdout: create 180/33 file after the snapshot 2019-12-05T05:05:49.249 INFO:tasks.workunit.client.0.smithi042.stdout: create 180/34 file after the snapshot 2019-12-05T05:05:49.252 INFO:tasks.workunit.client.0.smithi042.stdout: create 180/35 file after the snapshot 2019-12-05T05:05:49.255 INFO:tasks.workunit.client.0.smithi042.stdout: create 180/36 file after the snapshot 2019-12-05T05:05:49.257 INFO:tasks.workunit.client.0.smithi042.stdout: create 180/37 file after the snapshot 2019-12-05T05:05:49.259 INFO:tasks.workunit.client.0.smithi042.stdout: create 180/38 file after the snapshot 2019-12-05T05:05:49.261 INFO:tasks.workunit.client.0.smithi042.stdout: create 180/39 file after the snapshot 2019-12-05T05:05:49.263 INFO:tasks.workunit.client.0.smithi042.stdout: create 181/.snap/snap-subdir-test 2019-12-05T05:05:49.284 INFO:tasks.workunit.client.0.smithi042.stdout: create 181/30 file after the snapshot 2019-12-05T05:05:49.286 INFO:tasks.workunit.client.0.smithi042.stdout: create 181/31 file after the snapshot 2019-12-05T05:05:49.289 INFO:tasks.workunit.client.0.smithi042.stdout: create 181/32 file after the snapshot 2019-12-05T05:05:49.291 INFO:tasks.workunit.client.0.smithi042.stdout: create 181/33 file after the snapshot 2019-12-05T05:05:49.293 INFO:tasks.workunit.client.0.smithi042.stdout: create 181/34 file after the snapshot 2019-12-05T05:05:49.296 INFO:tasks.workunit.client.0.smithi042.stdout: create 181/35 file after the snapshot 2019-12-05T05:05:49.298 INFO:tasks.workunit.client.0.smithi042.stdout: create 181/36 file after the snapshot 2019-12-05T05:05:49.300 INFO:tasks.workunit.client.0.smithi042.stdout: create 181/37 file after the snapshot 2019-12-05T05:05:49.302 INFO:tasks.workunit.client.0.smithi042.stdout: create 181/38 file after the snapshot 2019-12-05T05:05:49.304 INFO:tasks.workunit.client.0.smithi042.stdout: create 181/39 file after the snapshot 2019-12-05T05:05:49.307 INFO:tasks.workunit.client.0.smithi042.stdout: create 182/.snap/snap-subdir-test 2019-12-05T05:05:49.327 INFO:tasks.workunit.client.0.smithi042.stdout: create 182/30 file after the snapshot 2019-12-05T05:05:49.332 INFO:tasks.workunit.client.0.smithi042.stdout: create 182/31 file after the snapshot 2019-12-05T05:05:49.334 INFO:tasks.workunit.client.0.smithi042.stdout: create 182/32 file after the snapshot 2019-12-05T05:05:49.336 INFO:tasks.workunit.client.0.smithi042.stdout: create 182/33 file after the snapshot 2019-12-05T05:05:49.338 INFO:tasks.workunit.client.0.smithi042.stdout: create 182/34 file after the snapshot 2019-12-05T05:05:49.340 INFO:tasks.workunit.client.0.smithi042.stdout: create 182/35 file after the snapshot 2019-12-05T05:05:49.342 INFO:tasks.workunit.client.0.smithi042.stdout: create 182/36 file after the snapshot 2019-12-05T05:05:49.345 INFO:tasks.workunit.client.0.smithi042.stdout: create 182/37 file after the snapshot 2019-12-05T05:05:49.347 INFO:tasks.workunit.client.0.smithi042.stdout: create 182/38 file after the snapshot 2019-12-05T05:05:49.349 INFO:tasks.workunit.client.0.smithi042.stdout: create 182/39 file after the snapshot 2019-12-05T05:05:49.351 INFO:tasks.workunit.client.0.smithi042.stdout: create 183/.snap/snap-subdir-test 2019-12-05T05:05:49.375 INFO:tasks.workunit.client.0.smithi042.stdout: create 183/30 file after the snapshot 2019-12-05T05:05:49.377 INFO:tasks.workunit.client.0.smithi042.stdout: create 183/31 file after the snapshot 2019-12-05T05:05:49.379 INFO:tasks.workunit.client.0.smithi042.stdout: create 183/32 file after the snapshot 2019-12-05T05:05:49.382 INFO:tasks.workunit.client.0.smithi042.stdout: create 183/33 file after the snapshot 2019-12-05T05:05:49.385 INFO:tasks.workunit.client.0.smithi042.stdout: create 183/34 file after the snapshot 2019-12-05T05:05:49.387 INFO:tasks.workunit.client.0.smithi042.stdout: create 183/35 file after the snapshot 2019-12-05T05:05:49.389 INFO:tasks.workunit.client.0.smithi042.stdout: create 183/36 file after the snapshot 2019-12-05T05:05:49.391 INFO:tasks.workunit.client.0.smithi042.stdout: create 183/37 file after the snapshot 2019-12-05T05:05:49.393 INFO:tasks.workunit.client.0.smithi042.stdout: create 183/38 file after the snapshot 2019-12-05T05:05:49.396 INFO:tasks.workunit.client.0.smithi042.stdout: create 183/39 file after the snapshot 2019-12-05T05:05:49.398 INFO:tasks.workunit.client.0.smithi042.stdout: create 184/.snap/snap-subdir-test 2019-12-05T05:05:49.421 INFO:tasks.workunit.client.0.smithi042.stdout: create 184/30 file after the snapshot 2019-12-05T05:05:49.423 INFO:tasks.workunit.client.0.smithi042.stdout: create 184/31 file after the snapshot 2019-12-05T05:05:49.425 INFO:tasks.workunit.client.0.smithi042.stdout: create 184/32 file after the snapshot 2019-12-05T05:05:49.433 INFO:tasks.workunit.client.0.smithi042.stdout: create 184/33 file after the snapshot 2019-12-05T05:05:49.433 INFO:tasks.workunit.client.0.smithi042.stdout: create 184/34 file after the snapshot 2019-12-05T05:05:49.434 INFO:tasks.workunit.client.0.smithi042.stdout: create 184/35 file after the snapshot 2019-12-05T05:05:49.436 INFO:tasks.workunit.client.0.smithi042.stdout: create 184/36 file after the snapshot 2019-12-05T05:05:49.438 INFO:tasks.workunit.client.0.smithi042.stdout: create 184/37 file after the snapshot 2019-12-05T05:05:49.440 INFO:tasks.workunit.client.0.smithi042.stdout: create 184/38 file after the snapshot 2019-12-05T05:05:49.442 INFO:tasks.workunit.client.0.smithi042.stdout: create 184/39 file after the snapshot 2019-12-05T05:05:49.444 INFO:tasks.workunit.client.0.smithi042.stdout: create 185/.snap/snap-subdir-test 2019-12-05T05:05:49.473 INFO:tasks.workunit.client.0.smithi042.stdout: create 185/30 file after the snapshot 2019-12-05T05:05:49.473 INFO:tasks.workunit.client.0.smithi042.stdout: create 185/31 file after the snapshot 2019-12-05T05:05:49.473 INFO:tasks.workunit.client.0.smithi042.stdout: create 185/32 file after the snapshot 2019-12-05T05:05:49.476 INFO:tasks.workunit.client.0.smithi042.stdout: create 185/33 file after the snapshot 2019-12-05T05:05:49.478 INFO:tasks.workunit.client.0.smithi042.stdout: create 185/34 file after the snapshot 2019-12-05T05:05:49.481 INFO:tasks.workunit.client.0.smithi042.stdout: create 185/35 file after the snapshot 2019-12-05T05:05:49.483 INFO:tasks.workunit.client.0.smithi042.stdout: create 185/36 file after the snapshot 2019-12-05T05:05:49.486 INFO:tasks.workunit.client.0.smithi042.stdout: create 185/37 file after the snapshot 2019-12-05T05:05:49.487 INFO:tasks.workunit.client.0.smithi042.stdout: create 185/38 file after the snapshot 2019-12-05T05:05:49.490 INFO:tasks.workunit.client.0.smithi042.stdout: create 185/39 file after the snapshot 2019-12-05T05:05:49.492 INFO:tasks.workunit.client.0.smithi042.stdout: create 186/.snap/snap-subdir-test 2019-12-05T05:05:49.726 INFO:tasks.workunit.client.0.smithi042.stdout: create 186/30 file after the snapshot 2019-12-05T05:05:49.729 INFO:tasks.workunit.client.0.smithi042.stdout: create 186/31 file after the snapshot 2019-12-05T05:05:49.732 INFO:tasks.workunit.client.0.smithi042.stdout: create 186/32 file after the snapshot 2019-12-05T05:05:49.736 INFO:tasks.workunit.client.0.smithi042.stdout: create 186/33 file after the snapshot 2019-12-05T05:05:49.739 INFO:tasks.workunit.client.0.smithi042.stdout: create 186/34 file after the snapshot 2019-12-05T05:05:49.742 INFO:tasks.workunit.client.0.smithi042.stdout: create 186/35 file after the snapshot 2019-12-05T05:05:49.745 INFO:tasks.workunit.client.0.smithi042.stdout: create 186/36 file after the snapshot 2019-12-05T05:05:49.747 INFO:tasks.workunit.client.0.smithi042.stdout: create 186/37 file after the snapshot 2019-12-05T05:05:49.750 INFO:tasks.workunit.client.0.smithi042.stdout: create 186/38 file after the snapshot 2019-12-05T05:05:49.752 INFO:tasks.workunit.client.0.smithi042.stdout: create 186/39 file after the snapshot 2019-12-05T05:05:49.770 INFO:tasks.workunit.client.0.smithi042.stdout: create 187/.snap/snap-subdir-test 2019-12-05T05:05:50.349 INFO:tasks.workunit.client.0.smithi042.stdout: create 187/30 file after the snapshot 2019-12-05T05:05:51.032 INFO:tasks.workunit.client.0.smithi042.stdout: create 187/31 file after the snapshot 2019-12-05T05:05:51.040 INFO:tasks.workunit.client.0.smithi042.stdout: create 187/32 file after the snapshot 2019-12-05T05:05:51.043 INFO:tasks.workunit.client.0.smithi042.stdout: create 187/33 file after the snapshot 2019-12-05T05:05:51.051 INFO:tasks.workunit.client.0.smithi042.stdout: create 187/34 file after the snapshot 2019-12-05T05:05:51.051 INFO:tasks.workunit.client.0.smithi042.stdout: create 187/35 file after the snapshot 2019-12-05T05:05:51.052 INFO:tasks.workunit.client.0.smithi042.stdout: create 187/36 file after the snapshot 2019-12-05T05:05:51.054 INFO:tasks.workunit.client.0.smithi042.stdout: create 187/37 file after the snapshot 2019-12-05T05:05:51.057 INFO:tasks.workunit.client.0.smithi042.stdout: create 187/38 file after the snapshot 2019-12-05T05:05:51.060 INFO:tasks.workunit.client.0.smithi042.stdout: create 187/39 file after the snapshot 2019-12-05T05:05:51.090 INFO:tasks.workunit.client.0.smithi042.stdout: create 188/.snap/snap-subdir-test 2019-12-05T05:05:51.117 INFO:tasks.workunit.client.0.smithi042.stdout: create 188/30 file after the snapshot 2019-12-05T05:05:51.121 INFO:tasks.workunit.client.0.smithi042.stdout: create 188/31 file after the snapshot 2019-12-05T05:05:51.125 INFO:tasks.workunit.client.0.smithi042.stdout: create 188/32 file after the snapshot 2019-12-05T05:05:51.128 INFO:tasks.workunit.client.0.smithi042.stdout: create 188/33 file after the snapshot 2019-12-05T05:05:51.131 INFO:tasks.workunit.client.0.smithi042.stdout: create 188/34 file after the snapshot 2019-12-05T05:05:51.133 INFO:tasks.workunit.client.0.smithi042.stdout: create 188/35 file after the snapshot 2019-12-05T05:05:51.136 INFO:tasks.workunit.client.0.smithi042.stdout: create 188/36 file after the snapshot 2019-12-05T05:05:51.138 INFO:tasks.workunit.client.0.smithi042.stdout: create 188/37 file after the snapshot 2019-12-05T05:05:51.140 INFO:tasks.workunit.client.0.smithi042.stdout: create 188/38 file after the snapshot 2019-12-05T05:05:51.142 INFO:tasks.workunit.client.0.smithi042.stdout: create 188/39 file after the snapshot 2019-12-05T05:05:51.145 INFO:tasks.workunit.client.0.smithi042.stdout: create 189/.snap/snap-subdir-test 2019-12-05T05:05:51.167 INFO:tasks.workunit.client.0.smithi042.stdout: create 189/30 file after the snapshot 2019-12-05T05:05:51.169 INFO:tasks.workunit.client.0.smithi042.stdout: create 189/31 file after the snapshot 2019-12-05T05:05:51.172 INFO:tasks.workunit.client.0.smithi042.stdout: create 189/32 file after the snapshot 2019-12-05T05:05:51.174 INFO:tasks.workunit.client.0.smithi042.stdout: create 189/33 file after the snapshot 2019-12-05T05:05:51.176 INFO:tasks.workunit.client.0.smithi042.stdout: create 189/34 file after the snapshot 2019-12-05T05:05:51.178 INFO:tasks.workunit.client.0.smithi042.stdout: create 189/35 file after the snapshot 2019-12-05T05:05:51.181 INFO:tasks.workunit.client.0.smithi042.stdout: create 189/36 file after the snapshot 2019-12-05T05:05:51.183 INFO:tasks.workunit.client.0.smithi042.stdout: create 189/37 file after the snapshot 2019-12-05T05:05:51.185 INFO:tasks.workunit.client.0.smithi042.stdout: create 189/38 file after the snapshot 2019-12-05T05:05:51.188 INFO:tasks.workunit.client.0.smithi042.stdout: create 189/39 file after the snapshot 2019-12-05T05:05:51.190 INFO:tasks.workunit.client.0.smithi042.stdout: create 190/.snap/snap-subdir-test 2019-12-05T05:05:51.224 INFO:tasks.workunit.client.0.smithi042.stdout: create 190/30 file after the snapshot 2019-12-05T05:05:51.226 INFO:tasks.workunit.client.0.smithi042.stdout: create 190/31 file after the snapshot 2019-12-05T05:05:51.230 INFO:tasks.workunit.client.0.smithi042.stdout: create 190/32 file after the snapshot 2019-12-05T05:05:51.232 INFO:tasks.workunit.client.0.smithi042.stdout: create 190/33 file after the snapshot 2019-12-05T05:05:51.235 INFO:tasks.workunit.client.0.smithi042.stdout: create 190/34 file after the snapshot 2019-12-05T05:05:51.238 INFO:tasks.workunit.client.0.smithi042.stdout: create 190/35 file after the snapshot 2019-12-05T05:05:51.240 INFO:tasks.workunit.client.0.smithi042.stdout: create 190/36 file after the snapshot 2019-12-05T05:05:51.243 INFO:tasks.workunit.client.0.smithi042.stdout: create 190/37 file after the snapshot 2019-12-05T05:05:51.245 INFO:tasks.workunit.client.0.smithi042.stdout: create 190/38 file after the snapshot 2019-12-05T05:05:51.247 INFO:tasks.workunit.client.0.smithi042.stdout: create 190/39 file after the snapshot 2019-12-05T05:05:51.251 INFO:tasks.workunit.client.0.smithi042.stdout: create 191/.snap/snap-subdir-test 2019-12-05T05:05:52.061 INFO:tasks.workunit.client.0.smithi042.stdout: create 191/30 file after the snapshot 2019-12-05T05:05:52.944 INFO:tasks.workunit.client.0.smithi042.stdout: create 191/31 file after the snapshot 2019-12-05T05:05:52.949 INFO:tasks.workunit.client.0.smithi042.stdout: create 191/32 file after the snapshot 2019-12-05T05:05:52.952 INFO:tasks.workunit.client.0.smithi042.stdout: create 191/33 file after the snapshot 2019-12-05T05:05:52.954 INFO:tasks.workunit.client.0.smithi042.stdout: create 191/34 file after the snapshot 2019-12-05T05:05:52.959 INFO:tasks.workunit.client.0.smithi042.stdout: create 191/35 file after the snapshot 2019-12-05T05:05:52.961 INFO:tasks.workunit.client.0.smithi042.stdout: create 191/36 file after the snapshot 2019-12-05T05:05:52.963 INFO:tasks.workunit.client.0.smithi042.stdout: create 191/37 file after the snapshot 2019-12-05T05:05:52.966 INFO:tasks.workunit.client.0.smithi042.stdout: create 191/38 file after the snapshot 2019-12-05T05:05:53.045 INFO:tasks.workunit.client.0.smithi042.stdout: create 191/39 file after the snapshot 2019-12-05T05:05:53.051 INFO:tasks.workunit.client.0.smithi042.stdout: create 192/.snap/snap-subdir-test 2019-12-05T05:05:53.143 INFO:tasks.workunit.client.0.smithi042.stdout: create 192/30 file after the snapshot 2019-12-05T05:05:53.147 INFO:tasks.workunit.client.0.smithi042.stdout: create 192/31 file after the snapshot 2019-12-05T05:05:53.151 INFO:tasks.workunit.client.0.smithi042.stdout: create 192/32 file after the snapshot 2019-12-05T05:05:53.155 INFO:tasks.workunit.client.0.smithi042.stdout: create 192/33 file after the snapshot 2019-12-05T05:05:53.158 INFO:tasks.workunit.client.0.smithi042.stdout: create 192/34 file after the snapshot 2019-12-05T05:05:53.161 INFO:tasks.workunit.client.0.smithi042.stdout: create 192/35 file after the snapshot 2019-12-05T05:05:53.163 INFO:tasks.workunit.client.0.smithi042.stdout: create 192/36 file after the snapshot 2019-12-05T05:05:53.165 INFO:tasks.workunit.client.0.smithi042.stdout: create 192/37 file after the snapshot 2019-12-05T05:05:53.167 INFO:tasks.workunit.client.0.smithi042.stdout: create 192/38 file after the snapshot 2019-12-05T05:05:53.169 INFO:tasks.workunit.client.0.smithi042.stdout: create 192/39 file after the snapshot 2019-12-05T05:05:53.172 INFO:tasks.workunit.client.0.smithi042.stdout: create 193/.snap/snap-subdir-test 2019-12-05T05:05:53.969 INFO:tasks.workunit.client.0.smithi042.stdout: create 193/30 file after the snapshot 2019-12-05T05:05:53.972 INFO:tasks.workunit.client.0.smithi042.stdout: create 193/31 file after the snapshot 2019-12-05T05:05:53.976 INFO:tasks.workunit.client.0.smithi042.stdout: create 193/32 file after the snapshot 2019-12-05T05:05:53.979 INFO:tasks.workunit.client.0.smithi042.stdout: create 193/33 file after the snapshot 2019-12-05T05:05:53.981 INFO:tasks.workunit.client.0.smithi042.stdout: create 193/34 file after the snapshot 2019-12-05T05:05:53.984 INFO:tasks.workunit.client.0.smithi042.stdout: create 193/35 file after the snapshot 2019-12-05T05:05:53.986 INFO:tasks.workunit.client.0.smithi042.stdout: create 193/36 file after the snapshot 2019-12-05T05:05:53.988 INFO:tasks.workunit.client.0.smithi042.stdout: create 193/37 file after the snapshot 2019-12-05T05:05:53.991 INFO:tasks.workunit.client.0.smithi042.stdout: create 193/38 file after the snapshot 2019-12-05T05:05:53.994 INFO:tasks.workunit.client.0.smithi042.stdout: create 193/39 file after the snapshot 2019-12-05T05:05:53.997 INFO:tasks.workunit.client.0.smithi042.stdout: create 194/.snap/snap-subdir-test 2019-12-05T05:05:54.035 INFO:tasks.workunit.client.0.smithi042.stdout: create 194/30 file after the snapshot 2019-12-05T05:05:54.035 INFO:tasks.workunit.client.0.smithi042.stdout: create 194/31 file after the snapshot 2019-12-05T05:05:54.040 INFO:tasks.workunit.client.0.smithi042.stdout: create 194/32 file after the snapshot 2019-12-05T05:05:54.040 INFO:tasks.workunit.client.0.smithi042.stdout: create 194/33 file after the snapshot 2019-12-05T05:05:54.048 INFO:tasks.workunit.client.0.smithi042.stdout: create 194/34 file after the snapshot 2019-12-05T05:05:54.049 INFO:tasks.workunit.client.0.smithi042.stdout: create 194/35 file after the snapshot 2019-12-05T05:05:54.050 INFO:tasks.workunit.client.0.smithi042.stdout: create 194/36 file after the snapshot 2019-12-05T05:05:54.061 INFO:tasks.workunit.client.0.smithi042.stdout: create 194/37 file after the snapshot 2019-12-05T05:05:54.061 INFO:tasks.workunit.client.0.smithi042.stdout: create 194/38 file after the snapshot 2019-12-05T05:05:54.061 INFO:tasks.workunit.client.0.smithi042.stdout: create 194/39 file after the snapshot 2019-12-05T05:05:54.061 INFO:tasks.workunit.client.0.smithi042.stdout: create 195/.snap/snap-subdir-test 2019-12-05T05:05:54.420 INFO:tasks.workunit.client.0.smithi042.stdout: create 195/30 file after the snapshot 2019-12-05T05:05:54.427 INFO:tasks.workunit.client.0.smithi042.stdout: create 195/31 file after the snapshot 2019-12-05T05:05:54.432 INFO:tasks.workunit.client.0.smithi042.stdout: create 195/32 file after the snapshot 2019-12-05T05:05:54.438 INFO:tasks.workunit.client.0.smithi042.stdout: create 195/33 file after the snapshot 2019-12-05T05:05:54.442 INFO:tasks.workunit.client.0.smithi042.stdout: create 195/34 file after the snapshot 2019-12-05T05:05:54.449 INFO:tasks.workunit.client.0.smithi042.stdout: create 195/35 file after the snapshot 2019-12-05T05:05:54.454 INFO:tasks.workunit.client.0.smithi042.stdout: create 195/36 file after the snapshot 2019-12-05T05:05:54.458 INFO:tasks.workunit.client.0.smithi042.stdout: create 195/37 file after the snapshot 2019-12-05T05:05:54.461 INFO:tasks.workunit.client.0.smithi042.stdout: create 195/38 file after the snapshot 2019-12-05T05:05:54.465 INFO:tasks.workunit.client.0.smithi042.stdout: create 195/39 file after the snapshot 2019-12-05T05:05:54.467 INFO:tasks.workunit.client.0.smithi042.stdout: create 196/.snap/snap-subdir-test 2019-12-05T05:05:54.493 INFO:tasks.workunit.client.0.smithi042.stdout: create 196/30 file after the snapshot 2019-12-05T05:05:54.495 INFO:tasks.workunit.client.0.smithi042.stdout: create 196/31 file after the snapshot 2019-12-05T05:05:54.498 INFO:tasks.workunit.client.0.smithi042.stdout: create 196/32 file after the snapshot 2019-12-05T05:05:54.501 INFO:tasks.workunit.client.0.smithi042.stdout: create 196/33 file after the snapshot 2019-12-05T05:05:54.503 INFO:tasks.workunit.client.0.smithi042.stdout: create 196/34 file after the snapshot 2019-12-05T05:05:54.506 INFO:tasks.workunit.client.0.smithi042.stdout: create 196/35 file after the snapshot 2019-12-05T05:05:54.507 INFO:tasks.workunit.client.0.smithi042.stdout: create 196/36 file after the snapshot 2019-12-05T05:05:54.510 INFO:tasks.workunit.client.0.smithi042.stdout: create 196/37 file after the snapshot 2019-12-05T05:05:54.512 INFO:tasks.workunit.client.0.smithi042.stdout: create 196/38 file after the snapshot 2019-12-05T05:05:54.514 INFO:tasks.workunit.client.0.smithi042.stdout: create 196/39 file after the snapshot 2019-12-05T05:05:54.516 INFO:tasks.workunit.client.0.smithi042.stdout: create 197/.snap/snap-subdir-test 2019-12-05T05:05:54.543 INFO:tasks.workunit.client.0.smithi042.stdout: create 197/30 file after the snapshot 2019-12-05T05:05:54.546 INFO:tasks.workunit.client.0.smithi042.stdout: create 197/31 file after the snapshot 2019-12-05T05:05:54.548 INFO:tasks.workunit.client.0.smithi042.stdout: create 197/32 file after the snapshot 2019-12-05T05:05:54.554 INFO:tasks.workunit.client.0.smithi042.stdout: create 197/33 file after the snapshot 2019-12-05T05:05:54.555 INFO:tasks.workunit.client.0.smithi042.stdout: create 197/34 file after the snapshot 2019-12-05T05:05:54.559 INFO:tasks.workunit.client.0.smithi042.stdout: create 197/35 file after the snapshot 2019-12-05T05:05:54.560 INFO:tasks.workunit.client.0.smithi042.stdout: create 197/36 file after the snapshot 2019-12-05T05:05:54.563 INFO:tasks.workunit.client.0.smithi042.stdout: create 197/37 file after the snapshot 2019-12-05T05:05:54.565 INFO:tasks.workunit.client.0.smithi042.stdout: create 197/38 file after the snapshot 2019-12-05T05:05:54.566 INFO:tasks.workunit.client.0.smithi042.stdout: create 197/39 file after the snapshot 2019-12-05T05:05:54.569 INFO:tasks.workunit.client.0.smithi042.stdout: create 198/.snap/snap-subdir-test 2019-12-05T05:05:54.589 INFO:tasks.workunit.client.0.smithi042.stdout: create 198/30 file after the snapshot 2019-12-05T05:05:54.619 INFO:tasks.workunit.client.0.smithi042.stdout: create 198/31 file after the snapshot 2019-12-05T05:05:54.619 INFO:tasks.workunit.client.0.smithi042.stdout: create 198/32 file after the snapshot 2019-12-05T05:05:54.620 INFO:tasks.workunit.client.0.smithi042.stdout: create 198/33 file after the snapshot 2019-12-05T05:05:54.620 INFO:tasks.workunit.client.0.smithi042.stdout: create 198/34 file after the snapshot 2019-12-05T05:05:54.620 INFO:tasks.workunit.client.0.smithi042.stdout: create 198/35 file after the snapshot 2019-12-05T05:05:54.620 INFO:tasks.workunit.client.0.smithi042.stdout: create 198/36 file after the snapshot 2019-12-05T05:05:54.621 INFO:tasks.workunit.client.0.smithi042.stdout: create 198/37 file after the snapshot 2019-12-05T05:05:54.621 INFO:tasks.workunit.client.0.smithi042.stdout: create 198/38 file after the snapshot 2019-12-05T05:05:54.621 INFO:tasks.workunit.client.0.smithi042.stdout: create 198/39 file after the snapshot 2019-12-05T05:05:54.621 INFO:tasks.workunit.client.0.smithi042.stdout: create 199/.snap/snap-subdir-test 2019-12-05T05:05:54.633 INFO:tasks.workunit.client.0.smithi042.stdout: create 199/30 file after the snapshot 2019-12-05T05:05:54.635 INFO:tasks.workunit.client.0.smithi042.stdout: create 199/31 file after the snapshot 2019-12-05T05:05:54.637 INFO:tasks.workunit.client.0.smithi042.stdout: create 199/32 file after the snapshot 2019-12-05T05:05:54.639 INFO:tasks.workunit.client.0.smithi042.stdout: create 199/33 file after the snapshot 2019-12-05T05:05:54.641 INFO:tasks.workunit.client.0.smithi042.stdout: create 199/34 file after the snapshot 2019-12-05T05:05:54.643 INFO:tasks.workunit.client.0.smithi042.stdout: create 199/35 file after the snapshot 2019-12-05T05:05:54.645 INFO:tasks.workunit.client.0.smithi042.stdout: create 199/36 file after the snapshot 2019-12-05T05:05:54.648 INFO:tasks.workunit.client.0.smithi042.stdout: create 199/37 file after the snapshot 2019-12-05T05:05:54.650 INFO:tasks.workunit.client.0.smithi042.stdout: create 199/38 file after the snapshot 2019-12-05T05:05:54.654 INFO:tasks.workunit.client.0.smithi042.stdout: create 199/39 file after the snapshot 2019-12-05T05:05:54.655 INFO:tasks.workunit.client.0.smithi042.stdout: create 200/.snap/snap-subdir-test 2019-12-05T05:05:54.677 INFO:tasks.workunit.client.0.smithi042.stdout: create 200/30 file after the snapshot 2019-12-05T05:05:54.679 INFO:tasks.workunit.client.0.smithi042.stdout: create 200/31 file after the snapshot 2019-12-05T05:05:54.682 INFO:tasks.workunit.client.0.smithi042.stdout: create 200/32 file after the snapshot 2019-12-05T05:05:54.684 INFO:tasks.workunit.client.0.smithi042.stdout: create 200/33 file after the snapshot 2019-12-05T05:05:54.686 INFO:tasks.workunit.client.0.smithi042.stdout: create 200/34 file after the snapshot 2019-12-05T05:05:54.689 INFO:tasks.workunit.client.0.smithi042.stdout: create 200/35 file after the snapshot 2019-12-05T05:05:54.692 INFO:tasks.workunit.client.0.smithi042.stdout: create 200/36 file after the snapshot 2019-12-05T05:05:54.694 INFO:tasks.workunit.client.0.smithi042.stdout: create 200/37 file after the snapshot 2019-12-05T05:05:54.697 INFO:tasks.workunit.client.0.smithi042.stdout: create 200/38 file after the snapshot 2019-12-05T05:05:54.698 INFO:tasks.workunit.client.0.smithi042.stdout: create 200/39 file after the snapshot 2019-12-05T05:05:54.700 INFO:tasks.workunit.client.0.smithi042.stdout: create 201/.snap/snap-subdir-test 2019-12-05T05:05:54.719 INFO:tasks.workunit.client.0.smithi042.stdout: create 201/30 file after the snapshot 2019-12-05T05:05:54.721 INFO:tasks.workunit.client.0.smithi042.stdout: create 201/31 file after the snapshot 2019-12-05T05:05:54.723 INFO:tasks.workunit.client.0.smithi042.stdout: create 201/32 file after the snapshot 2019-12-05T05:05:54.725 INFO:tasks.workunit.client.0.smithi042.stdout: create 201/33 file after the snapshot 2019-12-05T05:05:54.728 INFO:tasks.workunit.client.0.smithi042.stdout: create 201/34 file after the snapshot 2019-12-05T05:05:54.730 INFO:tasks.workunit.client.0.smithi042.stdout: create 201/35 file after the snapshot 2019-12-05T05:05:54.733 INFO:tasks.workunit.client.0.smithi042.stdout: create 201/36 file after the snapshot 2019-12-05T05:05:54.735 INFO:tasks.workunit.client.0.smithi042.stdout: create 201/37 file after the snapshot 2019-12-05T05:05:54.737 INFO:tasks.workunit.client.0.smithi042.stdout: create 201/38 file after the snapshot 2019-12-05T05:05:54.740 INFO:tasks.workunit.client.0.smithi042.stdout: create 201/39 file after the snapshot 2019-12-05T05:05:54.742 INFO:tasks.workunit.client.0.smithi042.stdout: create 202/.snap/snap-subdir-test 2019-12-05T05:05:54.764 INFO:tasks.workunit.client.0.smithi042.stdout: create 202/30 file after the snapshot 2019-12-05T05:05:54.766 INFO:tasks.workunit.client.0.smithi042.stdout: create 202/31 file after the snapshot 2019-12-05T05:05:54.768 INFO:tasks.workunit.client.0.smithi042.stdout: create 202/32 file after the snapshot 2019-12-05T05:05:54.771 INFO:tasks.workunit.client.0.smithi042.stdout: create 202/33 file after the snapshot 2019-12-05T05:05:54.773 INFO:tasks.workunit.client.0.smithi042.stdout: create 202/34 file after the snapshot 2019-12-05T05:05:54.776 INFO:tasks.workunit.client.0.smithi042.stdout: create 202/35 file after the snapshot 2019-12-05T05:05:54.778 INFO:tasks.workunit.client.0.smithi042.stdout: create 202/36 file after the snapshot 2019-12-05T05:05:54.782 INFO:tasks.workunit.client.0.smithi042.stdout: create 202/37 file after the snapshot 2019-12-05T05:05:54.784 INFO:tasks.workunit.client.0.smithi042.stdout: create 202/38 file after the snapshot 2019-12-05T05:05:54.786 INFO:tasks.workunit.client.0.smithi042.stdout: create 202/39 file after the snapshot 2019-12-05T05:05:54.788 INFO:tasks.workunit.client.0.smithi042.stdout: create 203/.snap/snap-subdir-test 2019-12-05T05:05:55.028 INFO:tasks.workunit.client.0.smithi042.stdout: create 203/30 file after the snapshot 2019-12-05T05:05:55.030 INFO:tasks.workunit.client.0.smithi042.stdout: create 203/31 file after the snapshot 2019-12-05T05:05:55.033 INFO:tasks.workunit.client.0.smithi042.stdout: create 203/32 file after the snapshot 2019-12-05T05:05:55.036 INFO:tasks.workunit.client.0.smithi042.stdout: create 203/33 file after the snapshot 2019-12-05T05:05:55.040 INFO:tasks.workunit.client.0.smithi042.stdout: create 203/34 file after the snapshot 2019-12-05T05:05:55.042 INFO:tasks.workunit.client.0.smithi042.stdout: create 203/35 file after the snapshot 2019-12-05T05:05:55.045 INFO:tasks.workunit.client.0.smithi042.stdout: create 203/36 file after the snapshot 2019-12-05T05:05:55.083 INFO:tasks.workunit.client.0.smithi042.stdout: create 203/37 file after the snapshot 2019-12-05T05:05:55.083 INFO:tasks.workunit.client.0.smithi042.stdout: create 203/38 file after the snapshot 2019-12-05T05:05:55.083 INFO:tasks.workunit.client.0.smithi042.stdout: create 203/39 file after the snapshot 2019-12-05T05:05:55.083 INFO:tasks.workunit.client.0.smithi042.stdout: create 204/.snap/snap-subdir-test 2019-12-05T05:05:55.084 INFO:tasks.workunit.client.0.smithi042.stdout: create 204/30 file after the snapshot 2019-12-05T05:05:55.084 INFO:tasks.workunit.client.0.smithi042.stdout: create 204/31 file after the snapshot 2019-12-05T05:05:55.084 INFO:tasks.workunit.client.0.smithi042.stdout: create 204/32 file after the snapshot 2019-12-05T05:05:55.087 INFO:tasks.workunit.client.0.smithi042.stdout: create 204/33 file after the snapshot 2019-12-05T05:05:55.090 INFO:tasks.workunit.client.0.smithi042.stdout: create 204/34 file after the snapshot 2019-12-05T05:05:55.092 INFO:tasks.workunit.client.0.smithi042.stdout: create 204/35 file after the snapshot 2019-12-05T05:05:55.095 INFO:tasks.workunit.client.0.smithi042.stdout: create 204/36 file after the snapshot 2019-12-05T05:05:55.097 INFO:tasks.workunit.client.0.smithi042.stdout: create 204/37 file after the snapshot 2019-12-05T05:05:55.101 INFO:tasks.workunit.client.0.smithi042.stdout: create 204/38 file after the snapshot 2019-12-05T05:05:55.103 INFO:tasks.workunit.client.0.smithi042.stdout: create 204/39 file after the snapshot 2019-12-05T05:05:55.104 INFO:tasks.workunit.client.0.smithi042.stdout: create 205/.snap/snap-subdir-test 2019-12-05T05:05:55.125 INFO:tasks.workunit.client.0.smithi042.stdout: create 205/30 file after the snapshot 2019-12-05T05:05:55.126 INFO:tasks.workunit.client.0.smithi042.stdout: create 205/31 file after the snapshot 2019-12-05T05:05:55.129 INFO:tasks.workunit.client.0.smithi042.stdout: create 205/32 file after the snapshot 2019-12-05T05:05:55.132 INFO:tasks.workunit.client.0.smithi042.stdout: create 205/33 file after the snapshot 2019-12-05T05:05:55.135 INFO:tasks.workunit.client.0.smithi042.stdout: create 205/34 file after the snapshot 2019-12-05T05:05:55.137 INFO:tasks.workunit.client.0.smithi042.stdout: create 205/35 file after the snapshot 2019-12-05T05:05:55.139 INFO:tasks.workunit.client.0.smithi042.stdout: create 205/36 file after the snapshot 2019-12-05T05:05:55.141 INFO:tasks.workunit.client.0.smithi042.stdout: create 205/37 file after the snapshot 2019-12-05T05:05:55.144 INFO:tasks.workunit.client.0.smithi042.stdout: create 205/38 file after the snapshot 2019-12-05T05:05:55.146 INFO:tasks.workunit.client.0.smithi042.stdout: create 205/39 file after the snapshot 2019-12-05T05:05:55.149 INFO:tasks.workunit.client.0.smithi042.stdout: create 206/.snap/snap-subdir-test 2019-12-05T05:05:55.170 INFO:tasks.workunit.client.0.smithi042.stdout: create 206/30 file after the snapshot 2019-12-05T05:05:55.172 INFO:tasks.workunit.client.0.smithi042.stdout: create 206/31 file after the snapshot 2019-12-05T05:05:55.174 INFO:tasks.workunit.client.0.smithi042.stdout: create 206/32 file after the snapshot 2019-12-05T05:05:55.176 INFO:tasks.workunit.client.0.smithi042.stdout: create 206/33 file after the snapshot 2019-12-05T05:05:55.180 INFO:tasks.workunit.client.0.smithi042.stdout: create 206/34 file after the snapshot 2019-12-05T05:05:55.182 INFO:tasks.workunit.client.0.smithi042.stdout: create 206/35 file after the snapshot 2019-12-05T05:05:55.184 INFO:tasks.workunit.client.0.smithi042.stdout: create 206/36 file after the snapshot 2019-12-05T05:05:55.186 INFO:tasks.workunit.client.0.smithi042.stdout: create 206/37 file after the snapshot 2019-12-05T05:05:55.188 INFO:tasks.workunit.client.0.smithi042.stdout: create 206/38 file after the snapshot 2019-12-05T05:05:55.206 INFO:tasks.workunit.client.0.smithi042.stdout: create 206/39 file after the snapshot 2019-12-05T05:05:55.209 INFO:tasks.workunit.client.0.smithi042.stdout: create 207/.snap/snap-subdir-test 2019-12-05T05:05:55.737 INFO:tasks.workunit.client.0.smithi042.stdout: create 207/30 file after the snapshot 2019-12-05T05:05:55.742 INFO:tasks.workunit.client.0.smithi042.stdout: create 207/31 file after the snapshot 2019-12-05T05:05:55.747 INFO:tasks.workunit.client.0.smithi042.stdout: create 207/32 file after the snapshot 2019-12-05T05:05:55.751 INFO:tasks.workunit.client.0.smithi042.stdout: create 207/33 file after the snapshot 2019-12-05T05:05:55.753 INFO:tasks.workunit.client.0.smithi042.stdout: create 207/34 file after the snapshot 2019-12-05T05:05:55.758 INFO:tasks.workunit.client.0.smithi042.stdout: create 207/35 file after the snapshot 2019-12-05T05:05:55.760 INFO:tasks.workunit.client.0.smithi042.stdout: create 207/36 file after the snapshot 2019-12-05T05:05:55.762 INFO:tasks.workunit.client.0.smithi042.stdout: create 207/37 file after the snapshot 2019-12-05T05:05:55.765 INFO:tasks.workunit.client.0.smithi042.stdout: create 207/38 file after the snapshot 2019-12-05T05:05:55.768 INFO:tasks.workunit.client.0.smithi042.stdout: create 207/39 file after the snapshot 2019-12-05T05:05:55.770 INFO:tasks.workunit.client.0.smithi042.stdout: create 208/.snap/snap-subdir-test 2019-12-05T05:05:56.309 INFO:tasks.workunit.client.0.smithi042.stdout: create 208/30 file after the snapshot 2019-12-05T05:05:56.312 INFO:tasks.workunit.client.0.smithi042.stdout: create 208/31 file after the snapshot 2019-12-05T05:05:56.316 INFO:tasks.workunit.client.0.smithi042.stdout: create 208/32 file after the snapshot 2019-12-05T05:05:56.320 INFO:tasks.workunit.client.0.smithi042.stdout: create 208/33 file after the snapshot 2019-12-05T05:05:56.324 INFO:tasks.workunit.client.0.smithi042.stdout: create 208/34 file after the snapshot 2019-12-05T05:05:56.347 INFO:tasks.workunit.client.0.smithi042.stdout: create 208/35 file after the snapshot 2019-12-05T05:05:56.349 INFO:tasks.workunit.client.0.smithi042.stdout: create 208/36 file after the snapshot 2019-12-05T05:05:56.353 INFO:tasks.workunit.client.0.smithi042.stdout: create 208/37 file after the snapshot 2019-12-05T05:05:56.355 INFO:tasks.workunit.client.0.smithi042.stdout: create 208/38 file after the snapshot 2019-12-05T05:05:56.358 INFO:tasks.workunit.client.0.smithi042.stdout: create 208/39 file after the snapshot 2019-12-05T05:05:56.359 INFO:tasks.workunit.client.0.smithi042.stdout: create 209/.snap/snap-subdir-test 2019-12-05T05:05:56.510 INFO:tasks.workunit.client.0.smithi042.stdout: create 209/30 file after the snapshot 2019-12-05T05:05:56.513 INFO:tasks.workunit.client.0.smithi042.stdout: create 209/31 file after the snapshot 2019-12-05T05:05:56.516 INFO:tasks.workunit.client.0.smithi042.stdout: create 209/32 file after the snapshot 2019-12-05T05:05:56.520 INFO:tasks.workunit.client.0.smithi042.stdout: create 209/33 file after the snapshot 2019-12-05T05:05:56.522 INFO:tasks.workunit.client.0.smithi042.stdout: create 209/34 file after the snapshot 2019-12-05T05:05:56.525 INFO:tasks.workunit.client.0.smithi042.stdout: create 209/35 file after the snapshot 2019-12-05T05:05:56.527 INFO:tasks.workunit.client.0.smithi042.stdout: create 209/36 file after the snapshot 2019-12-05T05:05:56.529 INFO:tasks.workunit.client.0.smithi042.stdout: create 209/37 file after the snapshot 2019-12-05T05:05:56.532 INFO:tasks.workunit.client.0.smithi042.stdout: create 209/38 file after the snapshot 2019-12-05T05:05:56.534 INFO:tasks.workunit.client.0.smithi042.stdout: create 209/39 file after the snapshot 2019-12-05T05:05:56.537 INFO:tasks.workunit.client.0.smithi042.stdout: create 210/.snap/snap-subdir-test 2019-12-05T05:05:56.926 INFO:tasks.workunit.client.0.smithi042.stdout: create 210/30 file after the snapshot 2019-12-05T05:05:56.930 INFO:tasks.workunit.client.0.smithi042.stdout: create 210/31 file after the snapshot 2019-12-05T05:05:56.935 INFO:tasks.workunit.client.0.smithi042.stdout: create 210/32 file after the snapshot 2019-12-05T05:05:57.424 INFO:tasks.workunit.client.0.smithi042.stdout: create 210/33 file after the snapshot 2019-12-05T05:05:57.430 INFO:tasks.workunit.client.0.smithi042.stdout: create 210/34 file after the snapshot 2019-12-05T05:05:57.439 INFO:tasks.workunit.client.0.smithi042.stdout: create 210/35 file after the snapshot 2019-12-05T05:05:57.439 INFO:tasks.workunit.client.0.smithi042.stdout: create 210/36 file after the snapshot 2019-12-05T05:05:57.441 INFO:tasks.workunit.client.0.smithi042.stdout: create 210/37 file after the snapshot 2019-12-05T05:05:58.041 INFO:tasks.workunit.client.0.smithi042.stdout: create 210/38 file after the snapshot 2019-12-05T05:05:58.055 INFO:tasks.workunit.client.0.smithi042.stdout: create 210/39 file after the snapshot 2019-12-05T05:05:58.066 INFO:tasks.workunit.client.0.smithi042.stdout: create 211/.snap/snap-subdir-test 2019-12-05T05:05:58.084 INFO:tasks.workunit.client.0.smithi042.stdout: create 211/30 file after the snapshot 2019-12-05T05:05:58.087 INFO:tasks.workunit.client.0.smithi042.stdout: create 211/31 file after the snapshot 2019-12-05T05:05:58.089 INFO:tasks.workunit.client.0.smithi042.stdout: create 211/32 file after the snapshot 2019-12-05T05:05:58.091 INFO:tasks.workunit.client.0.smithi042.stdout: create 211/33 file after the snapshot 2019-12-05T05:05:58.095 INFO:tasks.workunit.client.0.smithi042.stdout: create 211/34 file after the snapshot 2019-12-05T05:05:58.097 INFO:tasks.workunit.client.0.smithi042.stdout: create 211/35 file after the snapshot 2019-12-05T05:05:58.099 INFO:tasks.workunit.client.0.smithi042.stdout: create 211/36 file after the snapshot 2019-12-05T05:05:58.101 INFO:tasks.workunit.client.0.smithi042.stdout: create 211/37 file after the snapshot 2019-12-05T05:05:58.104 INFO:tasks.workunit.client.0.smithi042.stdout: create 211/38 file after the snapshot 2019-12-05T05:05:58.106 INFO:tasks.workunit.client.0.smithi042.stdout: create 211/39 file after the snapshot 2019-12-05T05:05:58.108 INFO:tasks.workunit.client.0.smithi042.stdout: create 212/.snap/snap-subdir-test 2019-12-05T05:05:58.130 INFO:tasks.workunit.client.0.smithi042.stdout: create 212/30 file after the snapshot 2019-12-05T05:05:58.137 INFO:tasks.workunit.client.0.smithi042.stdout: create 212/31 file after the snapshot 2019-12-05T05:05:58.137 INFO:tasks.workunit.client.0.smithi042.stdout: create 212/32 file after the snapshot 2019-12-05T05:05:58.138 INFO:tasks.workunit.client.0.smithi042.stdout: create 212/33 file after the snapshot 2019-12-05T05:05:58.139 INFO:tasks.workunit.client.0.smithi042.stdout: create 212/34 file after the snapshot 2019-12-05T05:05:58.142 INFO:tasks.workunit.client.0.smithi042.stdout: create 212/35 file after the snapshot 2019-12-05T05:05:58.144 INFO:tasks.workunit.client.0.smithi042.stdout: create 212/36 file after the snapshot 2019-12-05T05:05:58.146 INFO:tasks.workunit.client.0.smithi042.stdout: create 212/37 file after the snapshot 2019-12-05T05:05:58.149 INFO:tasks.workunit.client.0.smithi042.stdout: create 212/38 file after the snapshot 2019-12-05T05:05:58.151 INFO:tasks.workunit.client.0.smithi042.stdout: create 212/39 file after the snapshot 2019-12-05T05:05:58.153 INFO:tasks.workunit.client.0.smithi042.stdout: create 213/.snap/snap-subdir-test 2019-12-05T05:05:58.173 INFO:tasks.workunit.client.0.smithi042.stdout: create 213/30 file after the snapshot 2019-12-05T05:05:58.175 INFO:tasks.workunit.client.0.smithi042.stdout: create 213/31 file after the snapshot 2019-12-05T05:05:58.177 INFO:tasks.workunit.client.0.smithi042.stdout: create 213/32 file after the snapshot 2019-12-05T05:05:58.179 INFO:tasks.workunit.client.0.smithi042.stdout: create 213/33 file after the snapshot 2019-12-05T05:05:58.182 INFO:tasks.workunit.client.0.smithi042.stdout: create 213/34 file after the snapshot 2019-12-05T05:05:58.184 INFO:tasks.workunit.client.0.smithi042.stdout: create 213/35 file after the snapshot 2019-12-05T05:05:58.187 INFO:tasks.workunit.client.0.smithi042.stdout: create 213/36 file after the snapshot 2019-12-05T05:05:58.189 INFO:tasks.workunit.client.0.smithi042.stdout: create 213/37 file after the snapshot 2019-12-05T05:05:58.191 INFO:tasks.workunit.client.0.smithi042.stdout: create 213/38 file after the snapshot 2019-12-05T05:05:58.193 INFO:tasks.workunit.client.0.smithi042.stdout: create 213/39 file after the snapshot 2019-12-05T05:05:58.195 INFO:tasks.workunit.client.0.smithi042.stdout: create 214/.snap/snap-subdir-test 2019-12-05T05:05:59.182 INFO:tasks.workunit.client.0.smithi042.stdout: create 214/30 file after the snapshot 2019-12-05T05:05:59.188 INFO:tasks.workunit.client.0.smithi042.stdout: create 214/31 file after the snapshot 2019-12-05T05:05:59.193 INFO:tasks.workunit.client.0.smithi042.stdout: create 214/32 file after the snapshot 2019-12-05T05:05:59.196 INFO:tasks.workunit.client.0.smithi042.stdout: create 214/33 file after the snapshot 2019-12-05T05:05:59.198 INFO:tasks.workunit.client.0.smithi042.stdout: create 214/34 file after the snapshot 2019-12-05T05:05:59.203 INFO:tasks.workunit.client.0.smithi042.stdout: create 214/35 file after the snapshot 2019-12-05T05:05:59.205 INFO:tasks.workunit.client.0.smithi042.stdout: create 214/36 file after the snapshot 2019-12-05T05:05:59.208 INFO:tasks.workunit.client.0.smithi042.stdout: create 214/37 file after the snapshot 2019-12-05T05:05:59.210 INFO:tasks.workunit.client.0.smithi042.stdout: create 214/38 file after the snapshot 2019-12-05T05:05:59.212 INFO:tasks.workunit.client.0.smithi042.stdout: create 214/39 file after the snapshot 2019-12-05T05:05:59.215 INFO:tasks.workunit.client.0.smithi042.stdout: create 215/.snap/snap-subdir-test 2019-12-05T05:05:59.244 INFO:tasks.workunit.client.0.smithi042.stdout: create 215/30 file after the snapshot 2019-12-05T05:05:59.246 INFO:tasks.workunit.client.0.smithi042.stdout: create 215/31 file after the snapshot 2019-12-05T05:05:59.252 INFO:tasks.workunit.client.0.smithi042.stdout: create 215/32 file after the snapshot 2019-12-05T05:05:59.253 INFO:tasks.workunit.client.0.smithi042.stdout: create 215/33 file after the snapshot 2019-12-05T05:05:59.261 INFO:tasks.workunit.client.0.smithi042.stdout: create 215/34 file after the snapshot 2019-12-05T05:05:59.261 INFO:tasks.workunit.client.0.smithi042.stdout: create 215/35 file after the snapshot 2019-12-05T05:05:59.261 INFO:tasks.workunit.client.0.smithi042.stdout: create 215/36 file after the snapshot 2019-12-05T05:05:59.264 INFO:tasks.workunit.client.0.smithi042.stdout: create 215/37 file after the snapshot 2019-12-05T05:05:59.265 INFO:tasks.workunit.client.0.smithi042.stdout: create 215/38 file after the snapshot 2019-12-05T05:05:59.270 INFO:tasks.workunit.client.0.smithi042.stdout: create 215/39 file after the snapshot 2019-12-05T05:05:59.271 INFO:tasks.workunit.client.0.smithi042.stdout: create 216/.snap/snap-subdir-test 2019-12-05T05:05:59.292 INFO:tasks.workunit.client.0.smithi042.stdout: create 216/30 file after the snapshot 2019-12-05T05:05:59.295 INFO:tasks.workunit.client.0.smithi042.stdout: create 216/31 file after the snapshot 2019-12-05T05:05:59.297 INFO:tasks.workunit.client.0.smithi042.stdout: create 216/32 file after the snapshot 2019-12-05T05:05:59.299 INFO:tasks.workunit.client.0.smithi042.stdout: create 216/33 file after the snapshot 2019-12-05T05:05:59.302 INFO:tasks.workunit.client.0.smithi042.stdout: create 216/34 file after the snapshot 2019-12-05T05:05:59.304 INFO:tasks.workunit.client.0.smithi042.stdout: create 216/35 file after the snapshot 2019-12-05T05:05:59.307 INFO:tasks.workunit.client.0.smithi042.stdout: create 216/36 file after the snapshot 2019-12-05T05:05:59.309 INFO:tasks.workunit.client.0.smithi042.stdout: create 216/37 file after the snapshot 2019-12-05T05:05:59.311 INFO:tasks.workunit.client.0.smithi042.stdout: create 216/38 file after the snapshot 2019-12-05T05:05:59.313 INFO:tasks.workunit.client.0.smithi042.stdout: create 216/39 file after the snapshot 2019-12-05T05:05:59.315 INFO:tasks.workunit.client.0.smithi042.stdout: create 217/.snap/snap-subdir-test 2019-12-05T05:05:59.343 INFO:tasks.workunit.client.0.smithi042.stdout: create 217/30 file after the snapshot 2019-12-05T05:05:59.348 INFO:tasks.workunit.client.0.smithi042.stdout: create 217/31 file after the snapshot 2019-12-05T05:05:59.350 INFO:tasks.workunit.client.0.smithi042.stdout: create 217/32 file after the snapshot 2019-12-05T05:05:59.353 INFO:tasks.workunit.client.0.smithi042.stdout: create 217/33 file after the snapshot 2019-12-05T05:05:59.355 INFO:tasks.workunit.client.0.smithi042.stdout: create 217/34 file after the snapshot 2019-12-05T05:05:59.357 INFO:tasks.workunit.client.0.smithi042.stdout: create 217/35 file after the snapshot 2019-12-05T05:05:59.359 INFO:tasks.workunit.client.0.smithi042.stdout: create 217/36 file after the snapshot 2019-12-05T05:05:59.362 INFO:tasks.workunit.client.0.smithi042.stdout: create 217/37 file after the snapshot 2019-12-05T05:05:59.364 INFO:tasks.workunit.client.0.smithi042.stdout: create 217/38 file after the snapshot 2019-12-05T05:05:59.366 INFO:tasks.workunit.client.0.smithi042.stdout: create 217/39 file after the snapshot 2019-12-05T05:05:59.368 INFO:tasks.workunit.client.0.smithi042.stdout: create 218/.snap/snap-subdir-test 2019-12-05T05:05:59.401 INFO:tasks.workunit.client.0.smithi042.stdout: create 218/30 file after the snapshot 2019-12-05T05:05:59.404 INFO:tasks.workunit.client.0.smithi042.stdout: create 218/31 file after the snapshot 2019-12-05T05:05:59.408 INFO:tasks.workunit.client.0.smithi042.stdout: create 218/32 file after the snapshot 2019-12-05T05:05:59.411 INFO:tasks.workunit.client.0.smithi042.stdout: create 218/33 file after the snapshot 2019-12-05T05:05:59.413 INFO:tasks.workunit.client.0.smithi042.stdout: create 218/34 file after the snapshot 2019-12-05T05:05:59.416 INFO:tasks.workunit.client.0.smithi042.stdout: create 218/35 file after the snapshot 2019-12-05T05:05:59.418 INFO:tasks.workunit.client.0.smithi042.stdout: create 218/36 file after the snapshot 2019-12-05T05:06:00.274 INFO:tasks.workunit.client.0.smithi042.stdout: create 218/37 file after the snapshot 2019-12-05T05:06:00.279 INFO:tasks.workunit.client.0.smithi042.stdout: create 218/38 file after the snapshot 2019-12-05T05:06:00.282 INFO:tasks.workunit.client.0.smithi042.stdout: create 218/39 file after the snapshot 2019-12-05T05:06:00.284 INFO:tasks.workunit.client.0.smithi042.stdout: create 219/.snap/snap-subdir-test 2019-12-05T05:06:00.308 INFO:tasks.workunit.client.0.smithi042.stdout: create 219/30 file after the snapshot 2019-12-05T05:06:00.311 INFO:tasks.workunit.client.0.smithi042.stdout: create 219/31 file after the snapshot 2019-12-05T05:06:00.313 INFO:tasks.workunit.client.0.smithi042.stdout: create 219/32 file after the snapshot 2019-12-05T05:06:00.316 INFO:tasks.workunit.client.0.smithi042.stdout: create 219/33 file after the snapshot 2019-12-05T05:06:00.319 INFO:tasks.workunit.client.0.smithi042.stdout: create 219/34 file after the snapshot 2019-12-05T05:06:00.321 INFO:tasks.workunit.client.0.smithi042.stdout: create 219/35 file after the snapshot 2019-12-05T05:06:00.384 INFO:tasks.workunit.client.0.smithi042.stdout: create 219/36 file after the snapshot 2019-12-05T05:06:00.385 INFO:tasks.workunit.client.0.smithi042.stdout: create 219/37 file after the snapshot 2019-12-05T05:06:00.388 INFO:tasks.workunit.client.0.smithi042.stdout: create 219/38 file after the snapshot 2019-12-05T05:06:00.391 INFO:tasks.workunit.client.0.smithi042.stdout: create 219/39 file after the snapshot 2019-12-05T05:06:00.394 INFO:tasks.workunit.client.0.smithi042.stdout: create 220/.snap/snap-subdir-test 2019-12-05T05:06:01.142 INFO:tasks.workunit.client.0.smithi042.stdout: create 220/30 file after the snapshot 2019-12-05T05:06:01.146 INFO:tasks.workunit.client.0.smithi042.stdout: create 220/31 file after the snapshot 2019-12-05T05:06:01.152 INFO:tasks.workunit.client.0.smithi042.stdout: create 220/32 file after the snapshot 2019-12-05T05:06:01.155 INFO:tasks.workunit.client.0.smithi042.stdout: create 220/33 file after the snapshot 2019-12-05T05:06:01.158 INFO:tasks.workunit.client.0.smithi042.stdout: create 220/34 file after the snapshot 2019-12-05T05:06:01.160 INFO:tasks.workunit.client.0.smithi042.stdout: create 220/35 file after the snapshot 2019-12-05T05:06:01.163 INFO:tasks.workunit.client.0.smithi042.stdout: create 220/36 file after the snapshot 2019-12-05T05:06:01.165 INFO:tasks.workunit.client.0.smithi042.stdout: create 220/37 file after the snapshot 2019-12-05T05:06:01.167 INFO:tasks.workunit.client.0.smithi042.stdout: create 220/38 file after the snapshot 2019-12-05T05:06:01.170 INFO:tasks.workunit.client.0.smithi042.stdout: create 220/39 file after the snapshot 2019-12-05T05:06:01.173 INFO:tasks.workunit.client.0.smithi042.stdout: create 221/.snap/snap-subdir-test 2019-12-05T05:06:01.194 INFO:tasks.workunit.client.0.smithi042.stdout: create 221/30 file after the snapshot 2019-12-05T05:06:01.198 INFO:tasks.workunit.client.0.smithi042.stdout: create 221/31 file after the snapshot 2019-12-05T05:06:01.199 INFO:tasks.workunit.client.0.smithi042.stdout: create 221/32 file after the snapshot 2019-12-05T05:06:01.224 INFO:tasks.workunit.client.0.smithi042.stdout: create 221/33 file after the snapshot 2019-12-05T05:06:01.224 INFO:tasks.workunit.client.0.smithi042.stdout: create 221/34 file after the snapshot 2019-12-05T05:06:01.224 INFO:tasks.workunit.client.0.smithi042.stdout: create 221/35 file after the snapshot 2019-12-05T05:06:01.225 INFO:tasks.workunit.client.0.smithi042.stdout: create 221/36 file after the snapshot 2019-12-05T05:06:01.225 INFO:tasks.workunit.client.0.smithi042.stdout: create 221/37 file after the snapshot 2019-12-05T05:06:01.225 INFO:tasks.workunit.client.0.smithi042.stdout: create 221/38 file after the snapshot 2019-12-05T05:06:01.225 INFO:tasks.workunit.client.0.smithi042.stdout: create 221/39 file after the snapshot 2019-12-05T05:06:01.225 INFO:tasks.workunit.client.0.smithi042.stdout: create 222/.snap/snap-subdir-test 2019-12-05T05:06:01.551 INFO:tasks.workunit.client.0.smithi042.stdout: create 222/30 file after the snapshot 2019-12-05T05:06:01.554 INFO:tasks.workunit.client.0.smithi042.stdout: create 222/31 file after the snapshot 2019-12-05T05:06:01.557 INFO:tasks.workunit.client.0.smithi042.stdout: create 222/32 file after the snapshot 2019-12-05T05:06:01.560 INFO:tasks.workunit.client.0.smithi042.stdout: create 222/33 file after the snapshot 2019-12-05T05:06:01.563 INFO:tasks.workunit.client.0.smithi042.stdout: create 222/34 file after the snapshot 2019-12-05T05:06:01.565 INFO:tasks.workunit.client.0.smithi042.stdout: create 222/35 file after the snapshot 2019-12-05T05:06:01.567 INFO:tasks.workunit.client.0.smithi042.stdout: create 222/36 file after the snapshot 2019-12-05T05:06:01.569 INFO:tasks.workunit.client.0.smithi042.stdout: create 222/37 file after the snapshot 2019-12-05T05:06:01.572 INFO:tasks.workunit.client.0.smithi042.stdout: create 222/38 file after the snapshot 2019-12-05T05:06:01.575 INFO:tasks.workunit.client.0.smithi042.stdout: create 222/39 file after the snapshot 2019-12-05T05:06:01.577 INFO:tasks.workunit.client.0.smithi042.stdout: create 223/.snap/snap-subdir-test 2019-12-05T05:06:01.606 INFO:tasks.workunit.client.0.smithi042.stdout: create 223/30 file after the snapshot 2019-12-05T05:06:01.609 INFO:tasks.workunit.client.0.smithi042.stdout: create 223/31 file after the snapshot 2019-12-05T05:06:01.613 INFO:tasks.workunit.client.0.smithi042.stdout: create 223/32 file after the snapshot 2019-12-05T05:06:01.615 INFO:tasks.workunit.client.0.smithi042.stdout: create 223/33 file after the snapshot 2019-12-05T05:06:01.620 INFO:tasks.workunit.client.0.smithi042.stdout: create 223/34 file after the snapshot 2019-12-05T05:06:01.622 INFO:tasks.workunit.client.0.smithi042.stdout: create 223/35 file after the snapshot 2019-12-05T05:06:01.626 INFO:tasks.workunit.client.0.smithi042.stdout: create 223/36 file after the snapshot 2019-12-05T05:06:01.632 INFO:tasks.workunit.client.0.smithi042.stdout: create 223/37 file after the snapshot 2019-12-05T05:06:01.632 INFO:tasks.workunit.client.0.smithi042.stdout: create 223/38 file after the snapshot 2019-12-05T05:06:01.633 INFO:tasks.workunit.client.0.smithi042.stdout: create 223/39 file after the snapshot 2019-12-05T05:06:01.635 INFO:tasks.workunit.client.0.smithi042.stdout: create 224/.snap/snap-subdir-test 2019-12-05T05:06:01.657 INFO:tasks.workunit.client.0.smithi042.stdout: create 224/30 file after the snapshot 2019-12-05T05:06:01.660 INFO:tasks.workunit.client.0.smithi042.stdout: create 224/31 file after the snapshot 2019-12-05T05:06:01.662 INFO:tasks.workunit.client.0.smithi042.stdout: create 224/32 file after the snapshot 2019-12-05T05:06:01.665 INFO:tasks.workunit.client.0.smithi042.stdout: create 224/33 file after the snapshot 2019-12-05T05:06:01.667 INFO:tasks.workunit.client.0.smithi042.stdout: create 224/34 file after the snapshot 2019-12-05T05:06:01.684 INFO:tasks.workunit.client.0.smithi042.stdout: create 224/35 file after the snapshot 2019-12-05T05:06:01.688 INFO:tasks.workunit.client.0.smithi042.stdout: create 224/36 file after the snapshot 2019-12-05T05:06:01.689 INFO:tasks.workunit.client.0.smithi042.stdout: create 224/37 file after the snapshot 2019-12-05T05:06:01.689 INFO:tasks.workunit.client.0.smithi042.stdout: create 224/38 file after the snapshot 2019-12-05T05:06:01.689 INFO:tasks.workunit.client.0.smithi042.stdout: create 224/39 file after the snapshot 2019-12-05T05:06:01.689 INFO:tasks.workunit.client.0.smithi042.stdout: create 225/.snap/snap-subdir-test 2019-12-05T05:06:01.702 INFO:tasks.workunit.client.0.smithi042.stdout: create 225/30 file after the snapshot 2019-12-05T05:06:01.704 INFO:tasks.workunit.client.0.smithi042.stdout: create 225/31 file after the snapshot 2019-12-05T05:06:01.707 INFO:tasks.workunit.client.0.smithi042.stdout: create 225/32 file after the snapshot 2019-12-05T05:06:01.708 INFO:tasks.workunit.client.0.smithi042.stdout: create 225/33 file after the snapshot 2019-12-05T05:06:01.710 INFO:tasks.workunit.client.0.smithi042.stdout: create 225/34 file after the snapshot 2019-12-05T05:06:01.713 INFO:tasks.workunit.client.0.smithi042.stdout: create 225/35 file after the snapshot 2019-12-05T05:06:01.715 INFO:tasks.workunit.client.0.smithi042.stdout: create 225/36 file after the snapshot 2019-12-05T05:06:01.717 INFO:tasks.workunit.client.0.smithi042.stdout: create 225/37 file after the snapshot 2019-12-05T05:06:01.720 INFO:tasks.workunit.client.0.smithi042.stdout: create 225/38 file after the snapshot 2019-12-05T05:06:01.722 INFO:tasks.workunit.client.0.smithi042.stdout: create 225/39 file after the snapshot 2019-12-05T05:06:01.724 INFO:tasks.workunit.client.0.smithi042.stdout: create 226/.snap/snap-subdir-test 2019-12-05T05:06:01.750 INFO:tasks.workunit.client.0.smithi042.stdout: create 226/30 file after the snapshot 2019-12-05T05:06:01.753 INFO:tasks.workunit.client.0.smithi042.stdout: create 226/31 file after the snapshot 2019-12-05T05:06:01.755 INFO:tasks.workunit.client.0.smithi042.stdout: create 226/32 file after the snapshot 2019-12-05T05:06:01.758 INFO:tasks.workunit.client.0.smithi042.stdout: create 226/33 file after the snapshot 2019-12-05T05:06:01.761 INFO:tasks.workunit.client.0.smithi042.stdout: create 226/34 file after the snapshot 2019-12-05T05:06:01.765 INFO:tasks.workunit.client.0.smithi042.stdout: create 226/35 file after the snapshot 2019-12-05T05:06:01.765 INFO:tasks.workunit.client.0.smithi042.stdout: create 226/36 file after the snapshot 2019-12-05T05:06:01.768 INFO:tasks.workunit.client.0.smithi042.stdout: create 226/37 file after the snapshot 2019-12-05T05:06:01.772 INFO:tasks.workunit.client.0.smithi042.stdout: create 226/38 file after the snapshot 2019-12-05T05:06:01.772 INFO:tasks.workunit.client.0.smithi042.stdout: create 226/39 file after the snapshot 2019-12-05T05:06:01.774 INFO:tasks.workunit.client.0.smithi042.stdout: create 227/.snap/snap-subdir-test 2019-12-05T05:06:01.805 INFO:tasks.workunit.client.0.smithi042.stdout: create 227/30 file after the snapshot 2019-12-05T05:06:01.805 INFO:tasks.workunit.client.0.smithi042.stdout: create 227/31 file after the snapshot 2019-12-05T05:06:01.805 INFO:tasks.workunit.client.0.smithi042.stdout: create 227/32 file after the snapshot 2019-12-05T05:06:01.806 INFO:tasks.workunit.client.0.smithi042.stdout: create 227/33 file after the snapshot 2019-12-05T05:06:01.806 INFO:tasks.workunit.client.0.smithi042.stdout: create 227/34 file after the snapshot 2019-12-05T05:06:01.807 INFO:tasks.workunit.client.0.smithi042.stdout: create 227/35 file after the snapshot 2019-12-05T05:06:01.809 INFO:tasks.workunit.client.0.smithi042.stdout: create 227/36 file after the snapshot 2019-12-05T05:06:01.813 INFO:tasks.workunit.client.0.smithi042.stdout: create 227/37 file after the snapshot 2019-12-05T05:06:01.814 INFO:tasks.workunit.client.0.smithi042.stdout: create 227/38 file after the snapshot 2019-12-05T05:06:01.815 INFO:tasks.workunit.client.0.smithi042.stdout: create 227/39 file after the snapshot 2019-12-05T05:06:01.818 INFO:tasks.workunit.client.0.smithi042.stdout: create 228/.snap/snap-subdir-test 2019-12-05T05:06:01.839 INFO:tasks.workunit.client.0.smithi042.stdout: create 228/30 file after the snapshot 2019-12-05T05:06:01.841 INFO:tasks.workunit.client.0.smithi042.stdout: create 228/31 file after the snapshot 2019-12-05T05:06:01.843 INFO:tasks.workunit.client.0.smithi042.stdout: create 228/32 file after the snapshot 2019-12-05T05:06:01.847 INFO:tasks.workunit.client.0.smithi042.stdout: create 228/33 file after the snapshot 2019-12-05T05:06:01.849 INFO:tasks.workunit.client.0.smithi042.stdout: create 228/34 file after the snapshot 2019-12-05T05:06:01.851 INFO:tasks.workunit.client.0.smithi042.stdout: create 228/35 file after the snapshot 2019-12-05T05:06:01.854 INFO:tasks.workunit.client.0.smithi042.stdout: create 228/36 file after the snapshot 2019-12-05T05:06:01.856 INFO:tasks.workunit.client.0.smithi042.stdout: create 228/37 file after the snapshot 2019-12-05T05:06:01.859 INFO:tasks.workunit.client.0.smithi042.stdout: create 228/38 file after the snapshot 2019-12-05T05:06:01.861 INFO:tasks.workunit.client.0.smithi042.stdout: create 228/39 file after the snapshot 2019-12-05T05:06:01.863 INFO:tasks.workunit.client.0.smithi042.stdout: create 229/.snap/snap-subdir-test 2019-12-05T05:06:01.882 INFO:tasks.workunit.client.0.smithi042.stdout: create 229/30 file after the snapshot 2019-12-05T05:06:01.885 INFO:tasks.workunit.client.0.smithi042.stdout: create 229/31 file after the snapshot 2019-12-05T05:06:01.887 INFO:tasks.workunit.client.0.smithi042.stdout: create 229/32 file after the snapshot 2019-12-05T05:06:01.890 INFO:tasks.workunit.client.0.smithi042.stdout: create 229/33 file after the snapshot 2019-12-05T05:06:01.894 INFO:tasks.workunit.client.0.smithi042.stdout: create 229/34 file after the snapshot 2019-12-05T05:06:01.896 INFO:tasks.workunit.client.0.smithi042.stdout: create 229/35 file after the snapshot 2019-12-05T05:06:01.898 INFO:tasks.workunit.client.0.smithi042.stdout: create 229/36 file after the snapshot 2019-12-05T05:06:01.901 INFO:tasks.workunit.client.0.smithi042.stdout: create 229/37 file after the snapshot 2019-12-05T05:06:01.903 INFO:tasks.workunit.client.0.smithi042.stdout: create 229/38 file after the snapshot 2019-12-05T05:06:01.906 INFO:tasks.workunit.client.0.smithi042.stdout: create 229/39 file after the snapshot 2019-12-05T05:06:01.907 INFO:tasks.workunit.client.0.smithi042.stdout: create 230/.snap/snap-subdir-test 2019-12-05T05:06:01.927 INFO:tasks.workunit.client.0.smithi042.stdout: create 230/30 file after the snapshot 2019-12-05T05:06:01.928 INFO:tasks.workunit.client.0.smithi042.stdout: create 230/31 file after the snapshot 2019-12-05T05:06:01.931 INFO:tasks.workunit.client.0.smithi042.stdout: create 230/32 file after the snapshot 2019-12-05T05:06:01.933 INFO:tasks.workunit.client.0.smithi042.stdout: create 230/33 file after the snapshot 2019-12-05T05:06:01.937 INFO:tasks.workunit.client.0.smithi042.stdout: create 230/34 file after the snapshot 2019-12-05T05:06:01.938 INFO:tasks.workunit.client.0.smithi042.stdout: create 230/35 file after the snapshot 2019-12-05T05:06:01.940 INFO:tasks.workunit.client.0.smithi042.stdout: create 230/36 file after the snapshot 2019-12-05T05:06:01.942 INFO:tasks.workunit.client.0.smithi042.stdout: create 230/37 file after the snapshot 2019-12-05T05:06:01.945 INFO:tasks.workunit.client.0.smithi042.stdout: create 230/38 file after the snapshot 2019-12-05T05:06:01.946 INFO:tasks.workunit.client.0.smithi042.stdout: create 230/39 file after the snapshot 2019-12-05T05:06:01.948 INFO:tasks.workunit.client.0.smithi042.stdout: create 231/.snap/snap-subdir-test 2019-12-05T05:06:01.969 INFO:tasks.workunit.client.0.smithi042.stdout: create 231/30 file after the snapshot 2019-12-05T05:06:01.971 INFO:tasks.workunit.client.0.smithi042.stdout: create 231/31 file after the snapshot 2019-12-05T05:06:01.973 INFO:tasks.workunit.client.0.smithi042.stdout: create 231/32 file after the snapshot 2019-12-05T05:06:01.977 INFO:tasks.workunit.client.0.smithi042.stdout: create 231/33 file after the snapshot 2019-12-05T05:06:01.979 INFO:tasks.workunit.client.0.smithi042.stdout: create 231/34 file after the snapshot 2019-12-05T05:06:01.981 INFO:tasks.workunit.client.0.smithi042.stdout: create 231/35 file after the snapshot 2019-12-05T05:06:01.984 INFO:tasks.workunit.client.0.smithi042.stdout: create 231/36 file after the snapshot 2019-12-05T05:06:01.988 INFO:tasks.workunit.client.0.smithi042.stdout: create 231/37 file after the snapshot 2019-12-05T05:06:01.988 INFO:tasks.workunit.client.0.smithi042.stdout: create 231/38 file after the snapshot 2019-12-05T05:06:01.991 INFO:tasks.workunit.client.0.smithi042.stdout: create 231/39 file after the snapshot 2019-12-05T05:06:01.993 INFO:tasks.workunit.client.0.smithi042.stdout: create 232/.snap/snap-subdir-test 2019-12-05T05:06:02.049 INFO:tasks.workunit.client.0.smithi042.stdout: create 232/30 file after the snapshot 2019-12-05T05:06:02.052 INFO:tasks.workunit.client.0.smithi042.stdout: create 232/31 file after the snapshot 2019-12-05T05:06:02.055 INFO:tasks.workunit.client.0.smithi042.stdout: create 232/32 file after the snapshot 2019-12-05T05:06:02.058 INFO:tasks.workunit.client.0.smithi042.stdout: create 232/33 file after the snapshot 2019-12-05T05:06:02.061 INFO:tasks.workunit.client.0.smithi042.stdout: create 232/34 file after the snapshot 2019-12-05T05:06:02.062 INFO:tasks.workunit.client.0.smithi042.stdout: create 232/35 file after the snapshot 2019-12-05T05:06:02.064 INFO:tasks.workunit.client.0.smithi042.stdout: create 232/36 file after the snapshot 2019-12-05T05:06:02.067 INFO:tasks.workunit.client.0.smithi042.stdout: create 232/37 file after the snapshot 2019-12-05T05:06:02.069 INFO:tasks.workunit.client.0.smithi042.stdout: create 232/38 file after the snapshot 2019-12-05T05:06:02.072 INFO:tasks.workunit.client.0.smithi042.stdout: create 232/39 file after the snapshot 2019-12-05T05:06:02.073 INFO:tasks.workunit.client.0.smithi042.stdout: create 233/.snap/snap-subdir-test 2019-12-05T05:06:02.464 INFO:tasks.workunit.client.0.smithi042.stdout: create 233/30 file after the snapshot 2019-12-05T05:06:02.467 INFO:tasks.workunit.client.0.smithi042.stdout: create 233/31 file after the snapshot 2019-12-05T05:06:02.470 INFO:tasks.workunit.client.0.smithi042.stdout: create 233/32 file after the snapshot 2019-12-05T05:06:02.475 INFO:tasks.workunit.client.0.smithi042.stdout: create 233/33 file after the snapshot 2019-12-05T05:06:02.479 INFO:tasks.workunit.client.0.smithi042.stdout: create 233/34 file after the snapshot 2019-12-05T05:06:02.481 INFO:tasks.workunit.client.0.smithi042.stdout: create 233/35 file after the snapshot 2019-12-05T05:06:02.484 INFO:tasks.workunit.client.0.smithi042.stdout: create 233/36 file after the snapshot 2019-12-05T05:06:02.487 INFO:tasks.workunit.client.0.smithi042.stdout: create 233/37 file after the snapshot 2019-12-05T05:06:02.489 INFO:tasks.workunit.client.0.smithi042.stdout: create 233/38 file after the snapshot 2019-12-05T05:06:02.491 INFO:tasks.workunit.client.0.smithi042.stdout: create 233/39 file after the snapshot 2019-12-05T05:06:02.494 INFO:tasks.workunit.client.0.smithi042.stdout: create 234/.snap/snap-subdir-test 2019-12-05T05:06:02.520 INFO:tasks.workunit.client.0.smithi042.stdout: create 234/30 file after the snapshot 2019-12-05T05:06:02.523 INFO:tasks.workunit.client.0.smithi042.stdout: create 234/31 file after the snapshot 2019-12-05T05:06:02.526 INFO:tasks.workunit.client.0.smithi042.stdout: create 234/32 file after the snapshot 2019-12-05T05:06:02.529 INFO:tasks.workunit.client.0.smithi042.stdout: create 234/33 file after the snapshot 2019-12-05T05:06:02.531 INFO:tasks.workunit.client.0.smithi042.stdout: create 234/34 file after the snapshot 2019-12-05T05:06:02.533 INFO:tasks.workunit.client.0.smithi042.stdout: create 234/35 file after the snapshot 2019-12-05T05:06:02.536 INFO:tasks.workunit.client.0.smithi042.stdout: create 234/36 file after the snapshot 2019-12-05T05:06:02.538 INFO:tasks.workunit.client.0.smithi042.stdout: create 234/37 file after the snapshot 2019-12-05T05:06:02.552 INFO:tasks.workunit.client.0.smithi042.stdout: create 234/38 file after the snapshot 2019-12-05T05:06:02.552 INFO:tasks.workunit.client.0.smithi042.stdout: create 234/39 file after the snapshot 2019-12-05T05:06:02.552 INFO:tasks.workunit.client.0.smithi042.stdout: create 235/.snap/snap-subdir-test 2019-12-05T05:06:02.567 INFO:tasks.workunit.client.0.smithi042.stdout: create 235/30 file after the snapshot 2019-12-05T05:06:02.569 INFO:tasks.workunit.client.0.smithi042.stdout: create 235/31 file after the snapshot 2019-12-05T05:06:02.572 INFO:tasks.workunit.client.0.smithi042.stdout: create 235/32 file after the snapshot 2019-12-05T05:06:02.574 INFO:tasks.workunit.client.0.smithi042.stdout: create 235/33 file after the snapshot 2019-12-05T05:06:02.577 INFO:tasks.workunit.client.0.smithi042.stdout: create 235/34 file after the snapshot 2019-12-05T05:06:02.579 INFO:tasks.workunit.client.0.smithi042.stdout: create 235/35 file after the snapshot 2019-12-05T05:06:02.582 INFO:tasks.workunit.client.0.smithi042.stdout: create 235/36 file after the snapshot 2019-12-05T05:06:02.584 INFO:tasks.workunit.client.0.smithi042.stdout: create 235/37 file after the snapshot 2019-12-05T05:06:02.586 INFO:tasks.workunit.client.0.smithi042.stdout: create 235/38 file after the snapshot 2019-12-05T05:06:02.589 INFO:tasks.workunit.client.0.smithi042.stdout: create 235/39 file after the snapshot 2019-12-05T05:06:02.591 INFO:tasks.workunit.client.0.smithi042.stdout: create 236/.snap/snap-subdir-test 2019-12-05T05:06:02.611 INFO:tasks.workunit.client.0.smithi042.stdout: create 236/30 file after the snapshot 2019-12-05T05:06:02.613 INFO:tasks.workunit.client.0.smithi042.stdout: create 236/31 file after the snapshot 2019-12-05T05:06:02.616 INFO:tasks.workunit.client.0.smithi042.stdout: create 236/32 file after the snapshot 2019-12-05T05:06:02.620 INFO:tasks.workunit.client.0.smithi042.stdout: create 236/33 file after the snapshot 2019-12-05T05:06:02.622 INFO:tasks.workunit.client.0.smithi042.stdout: create 236/34 file after the snapshot 2019-12-05T05:06:02.624 INFO:tasks.workunit.client.0.smithi042.stdout: create 236/35 file after the snapshot 2019-12-05T05:06:02.627 INFO:tasks.workunit.client.0.smithi042.stdout: create 236/36 file after the snapshot 2019-12-05T05:06:02.629 INFO:tasks.workunit.client.0.smithi042.stdout: create 236/37 file after the snapshot 2019-12-05T05:06:02.632 INFO:tasks.workunit.client.0.smithi042.stdout: create 236/38 file after the snapshot 2019-12-05T05:06:02.633 INFO:tasks.workunit.client.0.smithi042.stdout: create 236/39 file after the snapshot 2019-12-05T05:06:02.635 INFO:tasks.workunit.client.0.smithi042.stdout: create 237/.snap/snap-subdir-test 2019-12-05T05:06:02.655 INFO:tasks.workunit.client.0.smithi042.stdout: create 237/30 file after the snapshot 2019-12-05T05:06:02.658 INFO:tasks.workunit.client.0.smithi042.stdout: create 237/31 file after the snapshot 2019-12-05T05:06:02.661 INFO:tasks.workunit.client.0.smithi042.stdout: create 237/32 file after the snapshot 2019-12-05T05:06:02.663 INFO:tasks.workunit.client.0.smithi042.stdout: create 237/33 file after the snapshot 2019-12-05T05:06:02.666 INFO:tasks.workunit.client.0.smithi042.stdout: create 237/34 file after the snapshot 2019-12-05T05:06:02.668 INFO:tasks.workunit.client.0.smithi042.stdout: create 237/35 file after the snapshot 2019-12-05T05:06:02.670 INFO:tasks.workunit.client.0.smithi042.stdout: create 237/36 file after the snapshot 2019-12-05T05:06:02.672 INFO:tasks.workunit.client.0.smithi042.stdout: create 237/37 file after the snapshot 2019-12-05T05:06:02.674 INFO:tasks.workunit.client.0.smithi042.stdout: create 237/38 file after the snapshot 2019-12-05T05:06:02.676 INFO:tasks.workunit.client.0.smithi042.stdout: create 237/39 file after the snapshot 2019-12-05T05:06:02.678 INFO:tasks.workunit.client.0.smithi042.stdout: create 238/.snap/snap-subdir-test 2019-12-05T05:06:02.699 INFO:tasks.workunit.client.0.smithi042.stdout: create 238/30 file after the snapshot 2019-12-05T05:06:02.702 INFO:tasks.workunit.client.0.smithi042.stdout: create 238/31 file after the snapshot 2019-12-05T05:06:02.704 INFO:tasks.workunit.client.0.smithi042.stdout: create 238/32 file after the snapshot 2019-12-05T05:06:02.706 INFO:tasks.workunit.client.0.smithi042.stdout: create 238/33 file after the snapshot 2019-12-05T05:06:02.708 INFO:tasks.workunit.client.0.smithi042.stdout: create 238/34 file after the snapshot 2019-12-05T05:06:02.711 INFO:tasks.workunit.client.0.smithi042.stdout: create 238/35 file after the snapshot 2019-12-05T05:06:02.713 INFO:tasks.workunit.client.0.smithi042.stdout: create 238/36 file after the snapshot 2019-12-05T05:06:02.715 INFO:tasks.workunit.client.0.smithi042.stdout: create 238/37 file after the snapshot 2019-12-05T05:06:02.718 INFO:tasks.workunit.client.0.smithi042.stdout: create 238/38 file after the snapshot 2019-12-05T05:06:02.720 INFO:tasks.workunit.client.0.smithi042.stdout: create 238/39 file after the snapshot 2019-12-05T05:06:02.722 INFO:tasks.workunit.client.0.smithi042.stdout: create 239/.snap/snap-subdir-test 2019-12-05T05:06:02.742 INFO:tasks.workunit.client.0.smithi042.stdout: create 239/30 file after the snapshot 2019-12-05T05:06:02.744 INFO:tasks.workunit.client.0.smithi042.stdout: create 239/31 file after the snapshot 2019-12-05T05:06:02.745 INFO:tasks.workunit.client.0.smithi042.stdout: create 239/32 file after the snapshot 2019-12-05T05:06:02.748 INFO:tasks.workunit.client.0.smithi042.stdout: create 239/33 file after the snapshot 2019-12-05T05:06:02.751 INFO:tasks.workunit.client.0.smithi042.stdout: create 239/34 file after the snapshot 2019-12-05T05:06:02.754 INFO:tasks.workunit.client.0.smithi042.stdout: create 239/35 file after the snapshot 2019-12-05T05:06:02.975 INFO:tasks.workunit.client.0.smithi042.stdout: create 239/36 file after the snapshot 2019-12-05T05:06:02.981 INFO:tasks.workunit.client.0.smithi042.stdout: create 239/37 file after the snapshot 2019-12-05T05:06:02.984 INFO:tasks.workunit.client.0.smithi042.stdout: create 239/38 file after the snapshot 2019-12-05T05:06:02.986 INFO:tasks.workunit.client.0.smithi042.stdout: create 239/39 file after the snapshot 2019-12-05T05:06:02.989 INFO:tasks.workunit.client.0.smithi042.stdout: create 240/.snap/snap-subdir-test 2019-12-05T05:06:03.012 INFO:tasks.workunit.client.0.smithi042.stdout: create 240/30 file after the snapshot 2019-12-05T05:06:03.014 INFO:tasks.workunit.client.0.smithi042.stdout: create 240/31 file after the snapshot 2019-12-05T05:06:03.016 INFO:tasks.workunit.client.0.smithi042.stdout: create 240/32 file after the snapshot 2019-12-05T05:06:03.019 INFO:tasks.workunit.client.0.smithi042.stdout: create 240/33 file after the snapshot 2019-12-05T05:06:03.021 INFO:tasks.workunit.client.0.smithi042.stdout: create 240/34 file after the snapshot 2019-12-05T05:06:03.024 INFO:tasks.workunit.client.0.smithi042.stdout: create 240/35 file after the snapshot 2019-12-05T05:06:03.027 INFO:tasks.workunit.client.0.smithi042.stdout: create 240/36 file after the snapshot 2019-12-05T05:06:03.030 INFO:tasks.workunit.client.0.smithi042.stdout: create 240/37 file after the snapshot 2019-12-05T05:06:03.032 INFO:tasks.workunit.client.0.smithi042.stdout: create 240/38 file after the snapshot 2019-12-05T05:06:03.035 INFO:tasks.workunit.client.0.smithi042.stdout: create 240/39 file after the snapshot 2019-12-05T05:06:03.037 INFO:tasks.workunit.client.0.smithi042.stdout: create 241/.snap/snap-subdir-test 2019-12-05T05:06:03.055 INFO:tasks.workunit.client.0.smithi042.stdout: create 241/30 file after the snapshot 2019-12-05T05:06:03.057 INFO:tasks.workunit.client.0.smithi042.stdout: create 241/31 file after the snapshot 2019-12-05T05:06:03.059 INFO:tasks.workunit.client.0.smithi042.stdout: create 241/32 file after the snapshot 2019-12-05T05:06:03.062 INFO:tasks.workunit.client.0.smithi042.stdout: create 241/33 file after the snapshot 2019-12-05T05:06:03.065 INFO:tasks.workunit.client.0.smithi042.stdout: create 241/34 file after the snapshot 2019-12-05T05:06:03.066 INFO:tasks.workunit.client.0.smithi042.stdout: create 241/35 file after the snapshot 2019-12-05T05:06:03.069 INFO:tasks.workunit.client.0.smithi042.stdout: create 241/36 file after the snapshot 2019-12-05T05:06:03.071 INFO:tasks.workunit.client.0.smithi042.stdout: create 241/37 file after the snapshot 2019-12-05T05:06:03.074 INFO:tasks.workunit.client.0.smithi042.stdout: create 241/38 file after the snapshot 2019-12-05T05:06:03.076 INFO:tasks.workunit.client.0.smithi042.stdout: create 241/39 file after the snapshot 2019-12-05T05:06:03.078 INFO:tasks.workunit.client.0.smithi042.stdout: create 242/.snap/snap-subdir-test 2019-12-05T05:06:03.099 INFO:tasks.workunit.client.0.smithi042.stdout: create 242/30 file after the snapshot 2019-12-05T05:06:03.101 INFO:tasks.workunit.client.0.smithi042.stdout: create 242/31 file after the snapshot 2019-12-05T05:06:03.103 INFO:tasks.workunit.client.0.smithi042.stdout: create 242/32 file after the snapshot 2019-12-05T05:06:03.106 INFO:tasks.workunit.client.0.smithi042.stdout: create 242/33 file after the snapshot 2019-12-05T05:06:03.110 INFO:tasks.workunit.client.0.smithi042.stdout: create 242/34 file after the snapshot 2019-12-05T05:06:03.112 INFO:tasks.workunit.client.0.smithi042.stdout: create 242/35 file after the snapshot 2019-12-05T05:06:03.114 INFO:tasks.workunit.client.0.smithi042.stdout: create 242/36 file after the snapshot 2019-12-05T05:06:03.117 INFO:tasks.workunit.client.0.smithi042.stdout: create 242/37 file after the snapshot 2019-12-05T05:06:03.119 INFO:tasks.workunit.client.0.smithi042.stdout: create 242/38 file after the snapshot 2019-12-05T05:06:03.121 INFO:tasks.workunit.client.0.smithi042.stdout: create 242/39 file after the snapshot 2019-12-05T05:06:03.123 INFO:tasks.workunit.client.0.smithi042.stdout: create 243/.snap/snap-subdir-test 2019-12-05T05:06:03.143 INFO:tasks.workunit.client.0.smithi042.stdout: create 243/30 file after the snapshot 2019-12-05T05:06:03.145 INFO:tasks.workunit.client.0.smithi042.stdout: create 243/31 file after the snapshot 2019-12-05T05:06:03.147 INFO:tasks.workunit.client.0.smithi042.stdout: create 243/32 file after the snapshot 2019-12-05T05:06:03.150 INFO:tasks.workunit.client.0.smithi042.stdout: create 243/33 file after the snapshot 2019-12-05T05:06:03.152 INFO:tasks.workunit.client.0.smithi042.stdout: create 243/34 file after the snapshot 2019-12-05T05:06:03.154 INFO:tasks.workunit.client.0.smithi042.stdout: create 243/35 file after the snapshot 2019-12-05T05:06:03.157 INFO:tasks.workunit.client.0.smithi042.stdout: create 243/36 file after the snapshot 2019-12-05T05:06:03.159 INFO:tasks.workunit.client.0.smithi042.stdout: create 243/37 file after the snapshot 2019-12-05T05:06:03.161 INFO:tasks.workunit.client.0.smithi042.stdout: create 243/38 file after the snapshot 2019-12-05T05:06:03.163 INFO:tasks.workunit.client.0.smithi042.stdout: create 243/39 file after the snapshot 2019-12-05T05:06:03.165 INFO:tasks.workunit.client.0.smithi042.stdout: create 244/.snap/snap-subdir-test 2019-12-05T05:06:03.183 INFO:tasks.workunit.client.0.smithi042.stdout: create 244/30 file after the snapshot 2019-12-05T05:06:03.185 INFO:tasks.workunit.client.0.smithi042.stdout: create 244/31 file after the snapshot 2019-12-05T05:06:03.188 INFO:tasks.workunit.client.0.smithi042.stdout: create 244/32 file after the snapshot 2019-12-05T05:06:03.190 INFO:tasks.workunit.client.0.smithi042.stdout: create 244/33 file after the snapshot 2019-12-05T05:06:03.192 INFO:tasks.workunit.client.0.smithi042.stdout: create 244/34 file after the snapshot 2019-12-05T05:06:03.195 INFO:tasks.workunit.client.0.smithi042.stdout: create 244/35 file after the snapshot 2019-12-05T05:06:03.197 INFO:tasks.workunit.client.0.smithi042.stdout: create 244/36 file after the snapshot 2019-12-05T05:06:03.199 INFO:tasks.workunit.client.0.smithi042.stdout: create 244/37 file after the snapshot 2019-12-05T05:06:03.201 INFO:tasks.workunit.client.0.smithi042.stdout: create 244/38 file after the snapshot 2019-12-05T05:06:03.204 INFO:tasks.workunit.client.0.smithi042.stdout: create 244/39 file after the snapshot 2019-12-05T05:06:03.206 INFO:tasks.workunit.client.0.smithi042.stdout: create 245/.snap/snap-subdir-test 2019-12-05T05:06:03.226 INFO:tasks.workunit.client.0.smithi042.stdout: create 245/30 file after the snapshot 2019-12-05T05:06:03.229 INFO:tasks.workunit.client.0.smithi042.stdout: create 245/31 file after the snapshot 2019-12-05T05:06:03.231 INFO:tasks.workunit.client.0.smithi042.stdout: create 245/32 file after the snapshot 2019-12-05T05:06:03.234 INFO:tasks.workunit.client.0.smithi042.stdout: create 245/33 file after the snapshot 2019-12-05T05:06:03.236 INFO:tasks.workunit.client.0.smithi042.stdout: create 245/34 file after the snapshot 2019-12-05T05:06:03.239 INFO:tasks.workunit.client.0.smithi042.stdout: create 245/35 file after the snapshot 2019-12-05T05:06:03.241 INFO:tasks.workunit.client.0.smithi042.stdout: create 245/36 file after the snapshot 2019-12-05T05:06:03.243 INFO:tasks.workunit.client.0.smithi042.stdout: create 245/37 file after the snapshot 2019-12-05T05:06:03.246 INFO:tasks.workunit.client.0.smithi042.stdout: create 245/38 file after the snapshot 2019-12-05T05:06:03.248 INFO:tasks.workunit.client.0.smithi042.stdout: create 245/39 file after the snapshot 2019-12-05T05:06:03.250 INFO:tasks.workunit.client.0.smithi042.stdout: create 246/.snap/snap-subdir-test 2019-12-05T05:06:03.718 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:06:03.722 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:06:04.146 INFO:tasks.workunit.client.0.smithi042.stdout: create 246/30 file after the snapshot 2019-12-05T05:06:04.426 INFO:tasks.workunit.client.0.smithi042.stdout: create 246/31 file after the snapshot 2019-12-05T05:06:04.460 INFO:tasks.workunit.client.0.smithi042.stdout: create 246/32 file after the snapshot 2019-12-05T05:06:04.464 INFO:tasks.workunit.client.0.smithi042.stdout: create 246/33 file after the snapshot 2019-12-05T05:06:04.467 INFO:tasks.workunit.client.0.smithi042.stdout: create 246/34 file after the snapshot 2019-12-05T05:06:04.470 INFO:tasks.workunit.client.0.smithi042.stdout: create 246/35 file after the snapshot 2019-12-05T05:06:04.473 INFO:tasks.workunit.client.0.smithi042.stdout: create 246/36 file after the snapshot 2019-12-05T05:06:04.475 INFO:tasks.workunit.client.0.smithi042.stdout: create 246/37 file after the snapshot 2019-12-05T05:06:04.477 INFO:tasks.workunit.client.0.smithi042.stdout: create 246/38 file after the snapshot 2019-12-05T05:06:04.480 INFO:tasks.workunit.client.0.smithi042.stdout: create 246/39 file after the snapshot 2019-12-05T05:06:04.482 INFO:tasks.workunit.client.0.smithi042.stdout: create 247/.snap/snap-subdir-test 2019-12-05T05:06:04.524 INFO:tasks.workunit.client.0.smithi042.stdout: create 247/30 file after the snapshot 2019-12-05T05:06:04.527 INFO:tasks.workunit.client.0.smithi042.stdout: create 247/31 file after the snapshot 2019-12-05T05:06:04.529 INFO:tasks.workunit.client.0.smithi042.stdout: create 247/32 file after the snapshot 2019-12-05T05:06:04.534 INFO:tasks.workunit.client.0.smithi042.stdout: create 247/33 file after the snapshot 2019-12-05T05:06:04.536 INFO:tasks.workunit.client.0.smithi042.stdout: create 247/34 file after the snapshot 2019-12-05T05:06:04.540 INFO:tasks.workunit.client.0.smithi042.stdout: create 247/35 file after the snapshot 2019-12-05T05:06:04.542 INFO:tasks.workunit.client.0.smithi042.stdout: create 247/36 file after the snapshot 2019-12-05T05:06:04.545 INFO:tasks.workunit.client.0.smithi042.stdout: create 247/37 file after the snapshot 2019-12-05T05:06:04.548 INFO:tasks.workunit.client.0.smithi042.stdout: create 247/38 file after the snapshot 2019-12-05T05:06:04.550 INFO:tasks.workunit.client.0.smithi042.stdout: create 247/39 file after the snapshot 2019-12-05T05:06:04.553 INFO:tasks.workunit.client.0.smithi042.stdout: create 248/.snap/snap-subdir-test 2019-12-05T05:06:05.241 INFO:tasks.workunit.client.0.smithi042.stdout: create 248/30 file after the snapshot 2019-12-05T05:06:05.246 INFO:tasks.workunit.client.0.smithi042.stdout: create 248/31 file after the snapshot 2019-12-05T05:06:05.250 INFO:tasks.workunit.client.0.smithi042.stdout: create 248/32 file after the snapshot 2019-12-05T05:06:05.257 INFO:tasks.workunit.client.0.smithi042.stdout: create 248/33 file after the snapshot 2019-12-05T05:06:05.259 INFO:tasks.workunit.client.0.smithi042.stdout: create 248/34 file after the snapshot 2019-12-05T05:06:05.261 INFO:tasks.workunit.client.0.smithi042.stdout: create 248/35 file after the snapshot 2019-12-05T05:06:05.264 INFO:tasks.workunit.client.0.smithi042.stdout: create 248/36 file after the snapshot 2019-12-05T05:06:05.267 INFO:tasks.workunit.client.0.smithi042.stdout: create 248/37 file after the snapshot 2019-12-05T05:06:05.270 INFO:tasks.workunit.client.0.smithi042.stdout: create 248/38 file after the snapshot 2019-12-05T05:06:05.272 INFO:tasks.workunit.client.0.smithi042.stdout: create 248/39 file after the snapshot 2019-12-05T05:06:05.274 INFO:tasks.workunit.client.0.smithi042.stdout: create 249/.snap/snap-subdir-test 2019-12-05T05:06:05.297 INFO:tasks.workunit.client.0.smithi042.stdout: create 249/30 file after the snapshot 2019-12-05T05:06:05.299 INFO:tasks.workunit.client.0.smithi042.stdout: create 249/31 file after the snapshot 2019-12-05T05:06:05.302 INFO:tasks.workunit.client.0.smithi042.stdout: create 249/32 file after the snapshot 2019-12-05T05:06:05.305 INFO:tasks.workunit.client.0.smithi042.stdout: create 249/33 file after the snapshot 2019-12-05T05:06:05.307 INFO:tasks.workunit.client.0.smithi042.stdout: create 249/34 file after the snapshot 2019-12-05T05:06:05.311 INFO:tasks.workunit.client.0.smithi042.stdout: create 249/35 file after the snapshot 2019-12-05T05:06:05.313 INFO:tasks.workunit.client.0.smithi042.stdout: create 249/36 file after the snapshot 2019-12-05T05:06:05.316 INFO:tasks.workunit.client.0.smithi042.stdout: create 249/37 file after the snapshot 2019-12-05T05:06:05.317 INFO:tasks.workunit.client.0.smithi042.stdout: create 249/38 file after the snapshot 2019-12-05T05:06:05.320 INFO:tasks.workunit.client.0.smithi042.stdout: create 249/39 file after the snapshot 2019-12-05T05:06:05.322 INFO:tasks.workunit.client.0.smithi042.stdout: create 250/.snap/snap-subdir-test 2019-12-05T05:06:06.045 INFO:tasks.workunit.client.0.smithi042.stdout: create 250/30 file after the snapshot 2019-12-05T05:06:06.053 INFO:tasks.workunit.client.0.smithi042.stdout: create 250/31 file after the snapshot 2019-12-05T05:06:06.053 INFO:tasks.workunit.client.0.smithi042.stdout: create 250/32 file after the snapshot 2019-12-05T05:06:06.055 INFO:tasks.workunit.client.0.smithi042.stdout: create 250/33 file after the snapshot 2019-12-05T05:06:06.058 INFO:tasks.workunit.client.0.smithi042.stdout: create 250/34 file after the snapshot 2019-12-05T05:06:06.061 INFO:tasks.workunit.client.0.smithi042.stdout: create 250/35 file after the snapshot 2019-12-05T05:06:06.063 INFO:tasks.workunit.client.0.smithi042.stdout: create 250/36 file after the snapshot 2019-12-05T05:06:06.065 INFO:tasks.workunit.client.0.smithi042.stdout: create 250/37 file after the snapshot 2019-12-05T05:06:06.069 INFO:tasks.workunit.client.0.smithi042.stdout: create 250/38 file after the snapshot 2019-12-05T05:06:06.071 INFO:tasks.workunit.client.0.smithi042.stdout: create 250/39 file after the snapshot 2019-12-05T05:06:06.074 INFO:tasks.workunit.client.0.smithi042.stdout: create 251/.snap/snap-subdir-test 2019-12-05T05:06:06.097 INFO:tasks.workunit.client.0.smithi042.stdout: create 251/30 file after the snapshot 2019-12-05T05:06:06.100 INFO:tasks.workunit.client.0.smithi042.stdout: create 251/31 file after the snapshot 2019-12-05T05:06:06.103 INFO:tasks.workunit.client.0.smithi042.stdout: create 251/32 file after the snapshot 2019-12-05T05:06:06.106 INFO:tasks.workunit.client.0.smithi042.stdout: create 251/33 file after the snapshot 2019-12-05T05:06:06.109 INFO:tasks.workunit.client.0.smithi042.stdout: create 251/34 file after the snapshot 2019-12-05T05:06:06.112 INFO:tasks.workunit.client.0.smithi042.stdout: create 251/35 file after the snapshot 2019-12-05T05:06:06.115 INFO:tasks.workunit.client.0.smithi042.stdout: create 251/36 file after the snapshot 2019-12-05T05:06:06.118 INFO:tasks.workunit.client.0.smithi042.stdout: create 251/37 file after the snapshot 2019-12-05T05:06:06.120 INFO:tasks.workunit.client.0.smithi042.stdout: create 251/38 file after the snapshot 2019-12-05T05:06:06.123 INFO:tasks.workunit.client.0.smithi042.stdout: create 251/39 file after the snapshot 2019-12-05T05:06:06.125 INFO:tasks.workunit.client.0.smithi042.stdout: create 252/.snap/snap-subdir-test 2019-12-05T05:06:06.146 INFO:tasks.workunit.client.0.smithi042.stdout: create 252/30 file after the snapshot 2019-12-05T05:06:06.148 INFO:tasks.workunit.client.0.smithi042.stdout: create 252/31 file after the snapshot 2019-12-05T05:06:06.150 INFO:tasks.workunit.client.0.smithi042.stdout: create 252/32 file after the snapshot 2019-12-05T05:06:06.153 INFO:tasks.workunit.client.0.smithi042.stdout: create 252/33 file after the snapshot 2019-12-05T05:06:06.155 INFO:tasks.workunit.client.0.smithi042.stdout: create 252/34 file after the snapshot 2019-12-05T05:06:06.157 INFO:tasks.workunit.client.0.smithi042.stdout: create 252/35 file after the snapshot 2019-12-05T05:06:06.159 INFO:tasks.workunit.client.0.smithi042.stdout: create 252/36 file after the snapshot 2019-12-05T05:06:06.162 INFO:tasks.workunit.client.0.smithi042.stdout: create 252/37 file after the snapshot 2019-12-05T05:06:06.164 INFO:tasks.workunit.client.0.smithi042.stdout: create 252/38 file after the snapshot 2019-12-05T05:06:06.166 INFO:tasks.workunit.client.0.smithi042.stdout: create 252/39 file after the snapshot 2019-12-05T05:06:06.169 INFO:tasks.workunit.client.0.smithi042.stdout: create 253/.snap/snap-subdir-test 2019-12-05T05:06:06.187 INFO:tasks.workunit.client.0.smithi042.stdout: create 253/30 file after the snapshot 2019-12-05T05:06:06.189 INFO:tasks.workunit.client.0.smithi042.stdout: create 253/31 file after the snapshot 2019-12-05T05:06:06.191 INFO:tasks.workunit.client.0.smithi042.stdout: create 253/32 file after the snapshot 2019-12-05T05:06:06.194 INFO:tasks.workunit.client.0.smithi042.stdout: create 253/33 file after the snapshot 2019-12-05T05:06:06.196 INFO:tasks.workunit.client.0.smithi042.stdout: create 253/34 file after the snapshot 2019-12-05T05:06:06.199 INFO:tasks.workunit.client.0.smithi042.stdout: create 253/35 file after the snapshot 2019-12-05T05:06:06.201 INFO:tasks.workunit.client.0.smithi042.stdout: create 253/36 file after the snapshot 2019-12-05T05:06:06.203 INFO:tasks.workunit.client.0.smithi042.stdout: create 253/37 file after the snapshot 2019-12-05T05:06:06.205 INFO:tasks.workunit.client.0.smithi042.stdout: create 253/38 file after the snapshot 2019-12-05T05:06:06.208 INFO:tasks.workunit.client.0.smithi042.stdout: create 253/39 file after the snapshot 2019-12-05T05:06:06.210 INFO:tasks.workunit.client.0.smithi042.stdout: create 254/.snap/snap-subdir-test 2019-12-05T05:06:06.229 INFO:tasks.workunit.client.0.smithi042.stdout: create 254/30 file after the snapshot 2019-12-05T05:06:06.230 INFO:tasks.workunit.client.0.smithi042.stdout: create 254/31 file after the snapshot 2019-12-05T05:06:06.233 INFO:tasks.workunit.client.0.smithi042.stdout: create 254/32 file after the snapshot 2019-12-05T05:06:06.235 INFO:tasks.workunit.client.0.smithi042.stdout: create 254/33 file after the snapshot 2019-12-05T05:06:06.239 INFO:tasks.workunit.client.0.smithi042.stdout: create 254/34 file after the snapshot 2019-12-05T05:06:06.241 INFO:tasks.workunit.client.0.smithi042.stdout: create 254/35 file after the snapshot 2019-12-05T05:06:06.243 INFO:tasks.workunit.client.0.smithi042.stdout: create 254/36 file after the snapshot 2019-12-05T05:06:06.245 INFO:tasks.workunit.client.0.smithi042.stdout: create 254/37 file after the snapshot 2019-12-05T05:06:06.247 INFO:tasks.workunit.client.0.smithi042.stdout: create 254/38 file after the snapshot 2019-12-05T05:06:06.249 INFO:tasks.workunit.client.0.smithi042.stdout: create 254/39 file after the snapshot 2019-12-05T05:06:06.252 INFO:tasks.workunit.client.0.smithi042.stdout: create 255/.snap/snap-subdir-test 2019-12-05T05:06:06.964 INFO:tasks.workunit.client.0.smithi042.stdout: create 255/30 file after the snapshot 2019-12-05T05:06:06.968 INFO:tasks.workunit.client.0.smithi042.stdout: create 255/31 file after the snapshot 2019-12-05T05:06:06.972 INFO:tasks.workunit.client.0.smithi042.stdout: create 255/32 file after the snapshot 2019-12-05T05:06:06.975 INFO:tasks.workunit.client.0.smithi042.stdout: create 255/33 file after the snapshot 2019-12-05T05:06:06.978 INFO:tasks.workunit.client.0.smithi042.stdout: create 255/34 file after the snapshot 2019-12-05T05:06:06.981 INFO:tasks.workunit.client.0.smithi042.stdout: create 255/35 file after the snapshot 2019-12-05T05:06:06.984 INFO:tasks.workunit.client.0.smithi042.stdout: create 255/36 file after the snapshot 2019-12-05T05:06:06.986 INFO:tasks.workunit.client.0.smithi042.stdout: create 255/37 file after the snapshot 2019-12-05T05:06:06.989 INFO:tasks.workunit.client.0.smithi042.stdout: create 255/38 file after the snapshot 2019-12-05T05:06:06.992 INFO:tasks.workunit.client.0.smithi042.stdout: create 255/39 file after the snapshot 2019-12-05T05:06:07.001 INFO:tasks.workunit.client.0.smithi042.stdout: create 256/.snap/snap-subdir-test 2019-12-05T05:06:07.020 INFO:tasks.workunit.client.0.smithi042.stdout: create 256/30 file after the snapshot 2019-12-05T05:06:07.022 INFO:tasks.workunit.client.0.smithi042.stdout: create 256/31 file after the snapshot 2019-12-05T05:06:07.025 INFO:tasks.workunit.client.0.smithi042.stdout: create 256/32 file after the snapshot 2019-12-05T05:06:07.026 INFO:tasks.workunit.client.0.smithi042.stdout: create 256/33 file after the snapshot 2019-12-05T05:06:07.029 INFO:tasks.workunit.client.0.smithi042.stdout: create 256/34 file after the snapshot 2019-12-05T05:06:07.032 INFO:tasks.workunit.client.0.smithi042.stdout: create 256/35 file after the snapshot 2019-12-05T05:06:07.034 INFO:tasks.workunit.client.0.smithi042.stdout: create 256/36 file after the snapshot 2019-12-05T05:06:07.037 INFO:tasks.workunit.client.0.smithi042.stdout: create 256/37 file after the snapshot 2019-12-05T05:06:07.039 INFO:tasks.workunit.client.0.smithi042.stdout: create 256/38 file after the snapshot 2019-12-05T05:06:07.041 INFO:tasks.workunit.client.0.smithi042.stdout: create 256/39 file after the snapshot 2019-12-05T05:06:07.053 INFO:tasks.workunit.client.0.smithi042.stdout: create 257/.snap/snap-subdir-test 2019-12-05T05:06:07.064 INFO:tasks.workunit.client.0.smithi042.stdout: create 257/30 file after the snapshot 2019-12-05T05:06:07.066 INFO:tasks.workunit.client.0.smithi042.stdout: create 257/31 file after the snapshot 2019-12-05T05:06:07.068 INFO:tasks.workunit.client.0.smithi042.stdout: create 257/32 file after the snapshot 2019-12-05T05:06:07.071 INFO:tasks.workunit.client.0.smithi042.stdout: create 257/33 file after the snapshot 2019-12-05T05:06:07.073 INFO:tasks.workunit.client.0.smithi042.stdout: create 257/34 file after the snapshot 2019-12-05T05:06:07.337 INFO:tasks.workunit.client.0.smithi042.stdout: create 257/35 file after the snapshot 2019-12-05T05:06:07.388 INFO:tasks.workunit.client.0.smithi042.stdout: create 257/36 file after the snapshot 2019-12-05T05:06:07.390 INFO:tasks.workunit.client.0.smithi042.stdout: create 257/37 file after the snapshot 2019-12-05T05:06:07.392 INFO:tasks.workunit.client.0.smithi042.stdout: create 257/38 file after the snapshot 2019-12-05T05:06:07.393 INFO:tasks.workunit.client.0.smithi042.stdout: create 257/39 file after the snapshot 2019-12-05T05:06:07.396 INFO:tasks.workunit.client.0.smithi042.stdout: create 258/.snap/snap-subdir-test 2019-12-05T05:06:08.125 INFO:tasks.workunit.client.0.smithi042.stdout: create 258/30 file after the snapshot 2019-12-05T05:06:08.127 INFO:tasks.workunit.client.0.smithi042.stdout: create 258/31 file after the snapshot 2019-12-05T05:06:08.129 INFO:tasks.workunit.client.0.smithi042.stdout: create 258/32 file after the snapshot 2019-12-05T05:06:08.336 INFO:tasks.workunit.client.0.smithi042.stdout: create 258/33 file after the snapshot 2019-12-05T05:06:08.413 INFO:tasks.workunit.client.0.smithi042.stdout: create 258/34 file after the snapshot 2019-12-05T05:06:08.414 INFO:tasks.workunit.client.0.smithi042.stdout: create 258/35 file after the snapshot 2019-12-05T05:06:08.417 INFO:tasks.workunit.client.0.smithi042.stdout: create 258/36 file after the snapshot 2019-12-05T05:06:08.419 INFO:tasks.workunit.client.0.smithi042.stdout: create 258/37 file after the snapshot 2019-12-05T05:06:08.421 INFO:tasks.workunit.client.0.smithi042.stdout: create 258/38 file after the snapshot 2019-12-05T05:06:08.423 INFO:tasks.workunit.client.0.smithi042.stdout: create 258/39 file after the snapshot 2019-12-05T05:06:08.425 INFO:tasks.workunit.client.0.smithi042.stdout: create 259/.snap/snap-subdir-test 2019-12-05T05:06:08.469 INFO:tasks.workunit.client.0.smithi042.stdout: create 259/30 file after the snapshot 2019-12-05T05:06:08.471 INFO:tasks.workunit.client.0.smithi042.stdout: create 259/31 file after the snapshot 2019-12-05T05:06:08.474 INFO:tasks.workunit.client.0.smithi042.stdout: create 259/32 file after the snapshot 2019-12-05T05:06:08.476 INFO:tasks.workunit.client.0.smithi042.stdout: create 259/33 file after the snapshot 2019-12-05T05:06:08.479 INFO:tasks.workunit.client.0.smithi042.stdout: create 259/34 file after the snapshot 2019-12-05T05:06:08.482 INFO:tasks.workunit.client.0.smithi042.stdout: create 259/35 file after the snapshot 2019-12-05T05:06:08.485 INFO:tasks.workunit.client.0.smithi042.stdout: create 259/36 file after the snapshot 2019-12-05T05:06:08.486 INFO:tasks.workunit.client.0.smithi042.stdout: create 259/37 file after the snapshot 2019-12-05T05:06:08.489 INFO:tasks.workunit.client.0.smithi042.stdout: create 259/38 file after the snapshot 2019-12-05T05:06:08.491 INFO:tasks.workunit.client.0.smithi042.stdout: create 259/39 file after the snapshot 2019-12-05T05:06:08.493 INFO:tasks.workunit.client.0.smithi042.stdout: create 260/.snap/snap-subdir-test 2019-12-05T05:06:09.188 INFO:tasks.workunit.client.0.smithi042.stdout: create 260/30 file after the snapshot 2019-12-05T05:06:09.192 INFO:tasks.workunit.client.0.smithi042.stdout: create 260/31 file after the snapshot 2019-12-05T05:06:09.196 INFO:tasks.workunit.client.0.smithi042.stdout: create 260/32 file after the snapshot 2019-12-05T05:06:09.199 INFO:tasks.workunit.client.0.smithi042.stdout: create 260/33 file after the snapshot 2019-12-05T05:06:09.204 INFO:tasks.workunit.client.0.smithi042.stdout: create 260/34 file after the snapshot 2019-12-05T05:06:09.207 INFO:tasks.workunit.client.0.smithi042.stdout: create 260/35 file after the snapshot 2019-12-05T05:06:09.219 INFO:tasks.workunit.client.0.smithi042.stdout: create 260/36 file after the snapshot 2019-12-05T05:06:09.219 INFO:tasks.workunit.client.0.smithi042.stdout: create 260/37 file after the snapshot 2019-12-05T05:06:09.219 INFO:tasks.workunit.client.0.smithi042.stdout: create 260/38 file after the snapshot 2019-12-05T05:06:09.220 INFO:tasks.workunit.client.0.smithi042.stdout: create 260/39 file after the snapshot 2019-12-05T05:06:09.220 INFO:tasks.workunit.client.0.smithi042.stdout: create 261/.snap/snap-subdir-test 2019-12-05T05:06:09.240 INFO:tasks.workunit.client.0.smithi042.stdout: create 261/30 file after the snapshot 2019-12-05T05:06:09.242 INFO:tasks.workunit.client.0.smithi042.stdout: create 261/31 file after the snapshot 2019-12-05T05:06:09.245 INFO:tasks.workunit.client.0.smithi042.stdout: create 261/32 file after the snapshot 2019-12-05T05:06:09.247 INFO:tasks.workunit.client.0.smithi042.stdout: create 261/33 file after the snapshot 2019-12-05T05:06:09.249 INFO:tasks.workunit.client.0.smithi042.stdout: create 261/34 file after the snapshot 2019-12-05T05:06:09.251 INFO:tasks.workunit.client.0.smithi042.stdout: create 261/35 file after the snapshot 2019-12-05T05:06:09.254 INFO:tasks.workunit.client.0.smithi042.stdout: create 261/36 file after the snapshot 2019-12-05T05:06:09.256 INFO:tasks.workunit.client.0.smithi042.stdout: create 261/37 file after the snapshot 2019-12-05T05:06:09.258 INFO:tasks.workunit.client.0.smithi042.stdout: create 261/38 file after the snapshot 2019-12-05T05:06:09.261 INFO:tasks.workunit.client.0.smithi042.stdout: create 261/39 file after the snapshot 2019-12-05T05:06:09.263 INFO:tasks.workunit.client.0.smithi042.stdout: create 262/.snap/snap-subdir-test 2019-12-05T05:06:09.282 INFO:tasks.workunit.client.0.smithi042.stdout: create 262/30 file after the snapshot 2019-12-05T05:06:09.283 INFO:tasks.workunit.client.0.smithi042.stdout: create 262/31 file after the snapshot 2019-12-05T05:06:09.961 INFO:tasks.workunit.client.0.smithi042.stdout: create 262/32 file after the snapshot 2019-12-05T05:06:09.966 INFO:tasks.workunit.client.0.smithi042.stdout: create 262/33 file after the snapshot 2019-12-05T05:06:09.970 INFO:tasks.workunit.client.0.smithi042.stdout: create 262/34 file after the snapshot 2019-12-05T05:06:09.974 INFO:tasks.workunit.client.0.smithi042.stdout: create 262/35 file after the snapshot 2019-12-05T05:06:09.977 INFO:tasks.workunit.client.0.smithi042.stdout: create 262/36 file after the snapshot 2019-12-05T05:06:09.979 INFO:tasks.workunit.client.0.smithi042.stdout: create 262/37 file after the snapshot 2019-12-05T05:06:09.982 INFO:tasks.workunit.client.0.smithi042.stdout: create 262/38 file after the snapshot 2019-12-05T05:06:09.984 INFO:tasks.workunit.client.0.smithi042.stdout: create 262/39 file after the snapshot 2019-12-05T05:06:09.987 INFO:tasks.workunit.client.0.smithi042.stdout: create 263/.snap/snap-subdir-test 2019-12-05T05:06:10.754 INFO:tasks.workunit.client.0.smithi042.stdout: create 263/30 file after the snapshot 2019-12-05T05:06:10.758 INFO:tasks.workunit.client.0.smithi042.stdout: create 263/31 file after the snapshot 2019-12-05T05:06:10.764 INFO:tasks.workunit.client.0.smithi042.stdout: create 263/32 file after the snapshot 2019-12-05T05:06:10.767 INFO:tasks.workunit.client.0.smithi042.stdout: create 263/33 file after the snapshot 2019-12-05T05:06:10.769 INFO:tasks.workunit.client.0.smithi042.stdout: create 263/34 file after the snapshot 2019-12-05T05:06:10.772 INFO:tasks.workunit.client.0.smithi042.stdout: create 263/35 file after the snapshot 2019-12-05T05:06:10.782 INFO:tasks.workunit.client.0.smithi042.stdout: create 263/36 file after the snapshot 2019-12-05T05:06:10.782 INFO:tasks.workunit.client.0.smithi042.stdout: create 263/37 file after the snapshot 2019-12-05T05:06:10.782 INFO:tasks.workunit.client.0.smithi042.stdout: create 263/38 file after the snapshot 2019-12-05T05:06:10.783 INFO:tasks.workunit.client.0.smithi042.stdout: create 263/39 file after the snapshot 2019-12-05T05:06:10.786 INFO:tasks.workunit.client.0.smithi042.stdout: create 264/.snap/snap-subdir-test 2019-12-05T05:06:10.817 INFO:tasks.workunit.client.0.smithi042.stdout: create 264/30 file after the snapshot 2019-12-05T05:06:10.820 INFO:tasks.workunit.client.0.smithi042.stdout: create 264/31 file after the snapshot 2019-12-05T05:06:10.823 INFO:tasks.workunit.client.0.smithi042.stdout: create 264/32 file after the snapshot 2019-12-05T05:06:10.826 INFO:tasks.workunit.client.0.smithi042.stdout: create 264/33 file after the snapshot 2019-12-05T05:06:10.829 INFO:tasks.workunit.client.0.smithi042.stdout: create 264/34 file after the snapshot 2019-12-05T05:06:10.831 INFO:tasks.workunit.client.0.smithi042.stdout: create 264/35 file after the snapshot 2019-12-05T05:06:10.834 INFO:tasks.workunit.client.0.smithi042.stdout: create 264/36 file after the snapshot 2019-12-05T05:06:10.836 INFO:tasks.workunit.client.0.smithi042.stdout: create 264/37 file after the snapshot 2019-12-05T05:06:10.839 INFO:tasks.workunit.client.0.smithi042.stdout: create 264/38 file after the snapshot 2019-12-05T05:06:10.841 INFO:tasks.workunit.client.0.smithi042.stdout: create 264/39 file after the snapshot 2019-12-05T05:06:10.843 INFO:tasks.workunit.client.0.smithi042.stdout: create 265/.snap/snap-subdir-test 2019-12-05T05:06:10.864 INFO:tasks.workunit.client.0.smithi042.stdout: create 265/30 file after the snapshot 2019-12-05T05:06:10.866 INFO:tasks.workunit.client.0.smithi042.stdout: create 265/31 file after the snapshot 2019-12-05T05:06:10.868 INFO:tasks.workunit.client.0.smithi042.stdout: create 265/32 file after the snapshot 2019-12-05T05:06:10.870 INFO:tasks.workunit.client.0.smithi042.stdout: create 265/33 file after the snapshot 2019-12-05T05:06:10.874 INFO:tasks.workunit.client.0.smithi042.stdout: create 265/34 file after the snapshot 2019-12-05T05:06:10.876 INFO:tasks.workunit.client.0.smithi042.stdout: create 265/35 file after the snapshot 2019-12-05T05:06:10.878 INFO:tasks.workunit.client.0.smithi042.stdout: create 265/36 file after the snapshot 2019-12-05T05:06:10.881 INFO:tasks.workunit.client.0.smithi042.stdout: create 265/37 file after the snapshot 2019-12-05T05:06:10.883 INFO:tasks.workunit.client.0.smithi042.stdout: create 265/38 file after the snapshot 2019-12-05T05:06:10.885 INFO:tasks.workunit.client.0.smithi042.stdout: create 265/39 file after the snapshot 2019-12-05T05:06:10.888 INFO:tasks.workunit.client.0.smithi042.stdout: create 266/.snap/snap-subdir-test 2019-12-05T05:06:10.914 INFO:tasks.workunit.client.0.smithi042.stdout: create 266/30 file after the snapshot 2019-12-05T05:06:10.916 INFO:tasks.workunit.client.0.smithi042.stdout: create 266/31 file after the snapshot 2019-12-05T05:06:10.919 INFO:tasks.workunit.client.0.smithi042.stdout: create 266/32 file after the snapshot 2019-12-05T05:06:10.922 INFO:tasks.workunit.client.0.smithi042.stdout: create 266/33 file after the snapshot 2019-12-05T05:06:10.924 INFO:tasks.workunit.client.0.smithi042.stdout: create 266/34 file after the snapshot 2019-12-05T05:06:10.927 INFO:tasks.workunit.client.0.smithi042.stdout: create 266/35 file after the snapshot 2019-12-05T05:06:10.929 INFO:tasks.workunit.client.0.smithi042.stdout: create 266/36 file after the snapshot 2019-12-05T05:06:10.931 INFO:tasks.workunit.client.0.smithi042.stdout: create 266/37 file after the snapshot 2019-12-05T05:06:10.934 INFO:tasks.workunit.client.0.smithi042.stdout: create 266/38 file after the snapshot 2019-12-05T05:06:10.936 INFO:tasks.workunit.client.0.smithi042.stdout: create 266/39 file after the snapshot 2019-12-05T05:06:10.938 INFO:tasks.workunit.client.0.smithi042.stdout: create 267/.snap/snap-subdir-test 2019-12-05T05:06:10.958 INFO:tasks.workunit.client.0.smithi042.stdout: create 267/30 file after the snapshot 2019-12-05T05:06:11.440 INFO:tasks.workunit.client.0.smithi042.stdout: create 267/31 file after the snapshot 2019-12-05T05:06:11.444 INFO:tasks.workunit.client.0.smithi042.stdout: create 267/32 file after the snapshot 2019-12-05T05:06:11.448 INFO:tasks.workunit.client.0.smithi042.stdout: create 267/33 file after the snapshot 2019-12-05T05:06:11.451 INFO:tasks.workunit.client.0.smithi042.stdout: create 267/34 file after the snapshot 2019-12-05T05:06:11.456 INFO:tasks.workunit.client.0.smithi042.stdout: create 267/35 file after the snapshot 2019-12-05T05:06:11.457 INFO:tasks.workunit.client.0.smithi042.stdout: create 267/36 file after the snapshot 2019-12-05T05:06:11.459 INFO:tasks.workunit.client.0.smithi042.stdout: create 267/37 file after the snapshot 2019-12-05T05:06:11.461 INFO:tasks.workunit.client.0.smithi042.stdout: create 267/38 file after the snapshot 2019-12-05T05:06:11.463 INFO:tasks.workunit.client.0.smithi042.stdout: create 267/39 file after the snapshot 2019-12-05T05:06:11.465 INFO:tasks.workunit.client.0.smithi042.stdout: create 268/.snap/snap-subdir-test 2019-12-05T05:06:11.499 INFO:tasks.workunit.client.0.smithi042.stdout: create 268/30 file after the snapshot 2019-12-05T05:06:11.501 INFO:tasks.workunit.client.0.smithi042.stdout: create 268/31 file after the snapshot 2019-12-05T05:06:11.503 INFO:tasks.workunit.client.0.smithi042.stdout: create 268/32 file after the snapshot 2019-12-05T05:06:11.506 INFO:tasks.workunit.client.0.smithi042.stdout: create 268/33 file after the snapshot 2019-12-05T05:06:11.510 INFO:tasks.workunit.client.0.smithi042.stdout: create 268/34 file after the snapshot 2019-12-05T05:06:11.513 INFO:tasks.workunit.client.0.smithi042.stdout: create 268/35 file after the snapshot 2019-12-05T05:06:11.515 INFO:tasks.workunit.client.0.smithi042.stdout: create 268/36 file after the snapshot 2019-12-05T05:06:11.518 INFO:tasks.workunit.client.0.smithi042.stdout: create 268/37 file after the snapshot 2019-12-05T05:06:11.520 INFO:tasks.workunit.client.0.smithi042.stdout: create 268/38 file after the snapshot 2019-12-05T05:06:11.522 INFO:tasks.workunit.client.0.smithi042.stdout: create 268/39 file after the snapshot 2019-12-05T05:06:11.524 INFO:tasks.workunit.client.0.smithi042.stdout: create 269/.snap/snap-subdir-test 2019-12-05T05:06:11.544 INFO:tasks.workunit.client.0.smithi042.stdout: create 269/30 file after the snapshot 2019-12-05T05:06:11.546 INFO:tasks.workunit.client.0.smithi042.stdout: create 269/31 file after the snapshot 2019-12-05T05:06:11.548 INFO:tasks.workunit.client.0.smithi042.stdout: create 269/32 file after the snapshot 2019-12-05T05:06:11.551 INFO:tasks.workunit.client.0.smithi042.stdout: create 269/33 file after the snapshot 2019-12-05T05:06:11.554 INFO:tasks.workunit.client.0.smithi042.stdout: create 269/34 file after the snapshot 2019-12-05T05:06:11.557 INFO:tasks.workunit.client.0.smithi042.stdout: create 269/35 file after the snapshot 2019-12-05T05:06:11.564 INFO:tasks.workunit.client.0.smithi042.stdout: create 269/36 file after the snapshot 2019-12-05T05:06:11.564 INFO:tasks.workunit.client.0.smithi042.stdout: create 269/37 file after the snapshot 2019-12-05T05:06:11.565 INFO:tasks.workunit.client.0.smithi042.stdout: create 269/38 file after the snapshot 2019-12-05T05:06:11.567 INFO:tasks.workunit.client.0.smithi042.stdout: create 269/39 file after the snapshot 2019-12-05T05:06:11.570 INFO:tasks.workunit.client.0.smithi042.stdout: create 270/.snap/snap-subdir-test 2019-12-05T05:06:11.589 INFO:tasks.workunit.client.0.smithi042.stdout: create 270/30 file after the snapshot 2019-12-05T05:06:11.590 INFO:tasks.workunit.client.0.smithi042.stdout: create 270/31 file after the snapshot 2019-12-05T05:06:11.593 INFO:tasks.workunit.client.0.smithi042.stdout: create 270/32 file after the snapshot 2019-12-05T05:06:11.595 INFO:tasks.workunit.client.0.smithi042.stdout: create 270/33 file after the snapshot 2019-12-05T05:06:11.597 INFO:tasks.workunit.client.0.smithi042.stdout: create 270/34 file after the snapshot 2019-12-05T05:06:11.600 INFO:tasks.workunit.client.0.smithi042.stdout: create 270/35 file after the snapshot 2019-12-05T05:06:11.602 INFO:tasks.workunit.client.0.smithi042.stdout: create 270/36 file after the snapshot 2019-12-05T05:06:11.607 INFO:tasks.workunit.client.0.smithi042.stdout: create 270/37 file after the snapshot 2019-12-05T05:06:11.607 INFO:tasks.workunit.client.0.smithi042.stdout: create 270/38 file after the snapshot 2019-12-05T05:06:11.609 INFO:tasks.workunit.client.0.smithi042.stdout: create 270/39 file after the snapshot 2019-12-05T05:06:11.611 INFO:tasks.workunit.client.0.smithi042.stdout: create 271/.snap/snap-subdir-test 2019-12-05T05:06:11.629 INFO:tasks.workunit.client.0.smithi042.stdout: create 271/30 file after the snapshot 2019-12-05T05:06:11.631 INFO:tasks.workunit.client.0.smithi042.stdout: create 271/31 file after the snapshot 2019-12-05T05:06:11.634 INFO:tasks.workunit.client.0.smithi042.stdout: create 271/32 file after the snapshot 2019-12-05T05:06:11.636 INFO:tasks.workunit.client.0.smithi042.stdout: create 271/33 file after the snapshot 2019-12-05T05:06:11.639 INFO:tasks.workunit.client.0.smithi042.stdout: create 271/34 file after the snapshot 2019-12-05T05:06:11.641 INFO:tasks.workunit.client.0.smithi042.stdout: create 271/35 file after the snapshot 2019-12-05T05:06:11.643 INFO:tasks.workunit.client.0.smithi042.stdout: create 271/36 file after the snapshot 2019-12-05T05:06:11.645 INFO:tasks.workunit.client.0.smithi042.stdout: create 271/37 file after the snapshot 2019-12-05T05:06:11.648 INFO:tasks.workunit.client.0.smithi042.stdout: create 271/38 file after the snapshot 2019-12-05T05:06:11.651 INFO:tasks.workunit.client.0.smithi042.stdout: create 271/39 file after the snapshot 2019-12-05T05:06:11.653 INFO:tasks.workunit.client.0.smithi042.stdout: create 272/.snap/snap-subdir-test 2019-12-05T05:06:11.673 INFO:tasks.workunit.client.0.smithi042.stdout: create 272/30 file after the snapshot 2019-12-05T05:06:11.676 INFO:tasks.workunit.client.0.smithi042.stdout: create 272/31 file after the snapshot 2019-12-05T05:06:11.678 INFO:tasks.workunit.client.0.smithi042.stdout: create 272/32 file after the snapshot 2019-12-05T05:06:11.681 INFO:tasks.workunit.client.0.smithi042.stdout: create 272/33 file after the snapshot 2019-12-05T05:06:11.684 INFO:tasks.workunit.client.0.smithi042.stdout: create 272/34 file after the snapshot 2019-12-05T05:06:11.686 INFO:tasks.workunit.client.0.smithi042.stdout: create 272/35 file after the snapshot 2019-12-05T05:06:11.690 INFO:tasks.workunit.client.0.smithi042.stdout: create 272/36 file after the snapshot 2019-12-05T05:06:11.692 INFO:tasks.workunit.client.0.smithi042.stdout: create 272/37 file after the snapshot 2019-12-05T05:06:11.694 INFO:tasks.workunit.client.0.smithi042.stdout: create 272/38 file after the snapshot 2019-12-05T05:06:11.697 INFO:tasks.workunit.client.0.smithi042.stdout: create 272/39 file after the snapshot 2019-12-05T05:06:11.699 INFO:tasks.workunit.client.0.smithi042.stdout: create 273/.snap/snap-subdir-test 2019-12-05T05:06:11.717 INFO:tasks.workunit.client.0.smithi042.stdout: create 273/30 file after the snapshot 2019-12-05T05:06:11.720 INFO:tasks.workunit.client.0.smithi042.stdout: create 273/31 file after the snapshot 2019-12-05T05:06:11.723 INFO:tasks.workunit.client.0.smithi042.stdout: create 273/32 file after the snapshot 2019-12-05T05:06:11.725 INFO:tasks.workunit.client.0.smithi042.stdout: create 273/33 file after the snapshot 2019-12-05T05:06:11.727 INFO:tasks.workunit.client.0.smithi042.stdout: create 273/34 file after the snapshot 2019-12-05T05:06:11.730 INFO:tasks.workunit.client.0.smithi042.stdout: create 273/35 file after the snapshot 2019-12-05T05:06:11.732 INFO:tasks.workunit.client.0.smithi042.stdout: create 273/36 file after the snapshot 2019-12-05T05:06:11.734 INFO:tasks.workunit.client.0.smithi042.stdout: create 273/37 file after the snapshot 2019-12-05T05:06:11.736 INFO:tasks.workunit.client.0.smithi042.stdout: create 273/38 file after the snapshot 2019-12-05T05:06:11.738 INFO:tasks.workunit.client.0.smithi042.stdout: create 273/39 file after the snapshot 2019-12-05T05:06:11.740 INFO:tasks.workunit.client.0.smithi042.stdout: create 274/.snap/snap-subdir-test 2019-12-05T05:06:11.759 INFO:tasks.workunit.client.0.smithi042.stdout: create 274/30 file after the snapshot 2019-12-05T05:06:11.762 INFO:tasks.workunit.client.0.smithi042.stdout: create 274/31 file after the snapshot 2019-12-05T05:06:11.765 INFO:tasks.workunit.client.0.smithi042.stdout: create 274/32 file after the snapshot 2019-12-05T05:06:11.767 INFO:tasks.workunit.client.0.smithi042.stdout: create 274/33 file after the snapshot 2019-12-05T05:06:11.771 INFO:tasks.workunit.client.0.smithi042.stdout: create 274/34 file after the snapshot 2019-12-05T05:06:11.772 INFO:tasks.workunit.client.0.smithi042.stdout: create 274/35 file after the snapshot 2019-12-05T05:06:11.774 INFO:tasks.workunit.client.0.smithi042.stdout: create 274/36 file after the snapshot 2019-12-05T05:06:11.776 INFO:tasks.workunit.client.0.smithi042.stdout: create 274/37 file after the snapshot 2019-12-05T05:06:11.778 INFO:tasks.workunit.client.0.smithi042.stdout: create 274/38 file after the snapshot 2019-12-05T05:06:11.780 INFO:tasks.workunit.client.0.smithi042.stdout: create 274/39 file after the snapshot 2019-12-05T05:06:11.784 INFO:tasks.workunit.client.0.smithi042.stdout: create 275/.snap/snap-subdir-test 2019-12-05T05:06:11.800 INFO:tasks.workunit.client.0.smithi042.stdout: create 275/30 file after the snapshot 2019-12-05T05:06:11.802 INFO:tasks.workunit.client.0.smithi042.stdout: create 275/31 file after the snapshot 2019-12-05T05:06:11.806 INFO:tasks.workunit.client.0.smithi042.stdout: create 275/32 file after the snapshot 2019-12-05T05:06:11.807 INFO:tasks.workunit.client.0.smithi042.stdout: create 275/33 file after the snapshot 2019-12-05T05:06:11.809 INFO:tasks.workunit.client.0.smithi042.stdout: create 275/34 file after the snapshot 2019-12-05T05:06:11.812 INFO:tasks.workunit.client.0.smithi042.stdout: create 275/35 file after the snapshot 2019-12-05T05:06:11.814 INFO:tasks.workunit.client.0.smithi042.stdout: create 275/36 file after the snapshot 2019-12-05T05:06:11.818 INFO:tasks.workunit.client.0.smithi042.stdout: create 275/37 file after the snapshot 2019-12-05T05:06:11.819 INFO:tasks.workunit.client.0.smithi042.stdout: create 275/38 file after the snapshot 2019-12-05T05:06:11.822 INFO:tasks.workunit.client.0.smithi042.stdout: create 275/39 file after the snapshot 2019-12-05T05:06:11.824 INFO:tasks.workunit.client.0.smithi042.stdout: create 276/.snap/snap-subdir-test 2019-12-05T05:06:11.842 INFO:tasks.workunit.client.0.smithi042.stdout: create 276/30 file after the snapshot 2019-12-05T05:06:11.847 INFO:tasks.workunit.client.0.smithi042.stdout: create 276/31 file after the snapshot 2019-12-05T05:06:11.848 INFO:tasks.workunit.client.0.smithi042.stdout: create 276/32 file after the snapshot 2019-12-05T05:06:11.853 INFO:tasks.workunit.client.0.smithi042.stdout: create 276/33 file after the snapshot 2019-12-05T05:06:11.854 INFO:tasks.workunit.client.0.smithi042.stdout: create 276/34 file after the snapshot 2019-12-05T05:06:11.857 INFO:tasks.workunit.client.0.smithi042.stdout: create 276/35 file after the snapshot 2019-12-05T05:06:11.859 INFO:tasks.workunit.client.0.smithi042.stdout: create 276/36 file after the snapshot 2019-12-05T05:06:11.861 INFO:tasks.workunit.client.0.smithi042.stdout: create 276/37 file after the snapshot 2019-12-05T05:06:11.865 INFO:tasks.workunit.client.0.smithi042.stdout: create 276/38 file after the snapshot 2019-12-05T05:06:11.866 INFO:tasks.workunit.client.0.smithi042.stdout: create 276/39 file after the snapshot 2019-12-05T05:06:11.868 INFO:tasks.workunit.client.0.smithi042.stdout: create 277/.snap/snap-subdir-test 2019-12-05T05:06:11.890 INFO:tasks.workunit.client.0.smithi042.stdout: create 277/30 file after the snapshot 2019-12-05T05:06:11.892 INFO:tasks.workunit.client.0.smithi042.stdout: create 277/31 file after the snapshot 2019-12-05T05:06:11.896 INFO:tasks.workunit.client.0.smithi042.stdout: create 277/32 file after the snapshot 2019-12-05T05:06:11.898 INFO:tasks.workunit.client.0.smithi042.stdout: create 277/33 file after the snapshot 2019-12-05T05:06:11.900 INFO:tasks.workunit.client.0.smithi042.stdout: create 277/34 file after the snapshot 2019-12-05T05:06:11.903 INFO:tasks.workunit.client.0.smithi042.stdout: create 277/35 file after the snapshot 2019-12-05T05:06:11.905 INFO:tasks.workunit.client.0.smithi042.stdout: create 277/36 file after the snapshot 2019-12-05T05:06:11.907 INFO:tasks.workunit.client.0.smithi042.stdout: create 277/37 file after the snapshot 2019-12-05T05:06:11.911 INFO:tasks.workunit.client.0.smithi042.stdout: create 277/38 file after the snapshot 2019-12-05T05:06:11.913 INFO:tasks.workunit.client.0.smithi042.stdout: create 277/39 file after the snapshot 2019-12-05T05:06:11.915 INFO:tasks.workunit.client.0.smithi042.stdout: create 278/.snap/snap-subdir-test 2019-12-05T05:06:11.940 INFO:tasks.workunit.client.0.smithi042.stdout: create 278/30 file after the snapshot 2019-12-05T05:06:11.941 INFO:tasks.workunit.client.0.smithi042.stdout: create 278/31 file after the snapshot 2019-12-05T05:06:11.943 INFO:tasks.workunit.client.0.smithi042.stdout: create 278/32 file after the snapshot 2019-12-05T05:06:11.946 INFO:tasks.workunit.client.0.smithi042.stdout: create 278/33 file after the snapshot 2019-12-05T05:06:11.949 INFO:tasks.workunit.client.0.smithi042.stdout: create 278/34 file after the snapshot 2019-12-05T05:06:11.952 INFO:tasks.workunit.client.0.smithi042.stdout: create 278/35 file after the snapshot 2019-12-05T05:06:11.954 INFO:tasks.workunit.client.0.smithi042.stdout: create 278/36 file after the snapshot 2019-12-05T05:06:11.956 INFO:tasks.workunit.client.0.smithi042.stdout: create 278/37 file after the snapshot 2019-12-05T05:06:11.958 INFO:tasks.workunit.client.0.smithi042.stdout: create 278/38 file after the snapshot 2019-12-05T05:06:11.960 INFO:tasks.workunit.client.0.smithi042.stdout: create 278/39 file after the snapshot 2019-12-05T05:06:11.963 INFO:tasks.workunit.client.0.smithi042.stdout: create 279/.snap/snap-subdir-test 2019-12-05T05:06:11.983 INFO:tasks.workunit.client.0.smithi042.stdout: create 279/30 file after the snapshot 2019-12-05T05:06:11.985 INFO:tasks.workunit.client.0.smithi042.stdout: create 279/31 file after the snapshot 2019-12-05T05:06:11.988 INFO:tasks.workunit.client.0.smithi042.stdout: create 279/32 file after the snapshot 2019-12-05T05:06:11.990 INFO:tasks.workunit.client.0.smithi042.stdout: create 279/33 file after the snapshot 2019-12-05T05:06:11.992 INFO:tasks.workunit.client.0.smithi042.stdout: create 279/34 file after the snapshot 2019-12-05T05:06:11.994 INFO:tasks.workunit.client.0.smithi042.stdout: create 279/35 file after the snapshot 2019-12-05T05:06:11.997 INFO:tasks.workunit.client.0.smithi042.stdout: create 279/36 file after the snapshot 2019-12-05T05:06:12.000 INFO:tasks.workunit.client.0.smithi042.stdout: create 279/37 file after the snapshot 2019-12-05T05:06:12.002 INFO:tasks.workunit.client.0.smithi042.stdout: create 279/38 file after the snapshot 2019-12-05T05:06:12.005 INFO:tasks.workunit.client.0.smithi042.stdout: create 279/39 file after the snapshot 2019-12-05T05:06:12.006 INFO:tasks.workunit.client.0.smithi042.stdout: create 280/.snap/snap-subdir-test 2019-12-05T05:06:12.026 INFO:tasks.workunit.client.0.smithi042.stdout: create 280/30 file after the snapshot 2019-12-05T05:06:12.028 INFO:tasks.workunit.client.0.smithi042.stdout: create 280/31 file after the snapshot 2019-12-05T05:06:12.031 INFO:tasks.workunit.client.0.smithi042.stdout: create 280/32 file after the snapshot 2019-12-05T05:06:12.035 INFO:tasks.workunit.client.0.smithi042.stdout: create 280/33 file after the snapshot 2019-12-05T05:06:12.037 INFO:tasks.workunit.client.0.smithi042.stdout: create 280/34 file after the snapshot 2019-12-05T05:06:12.039 INFO:tasks.workunit.client.0.smithi042.stdout: create 280/35 file after the snapshot 2019-12-05T05:06:12.041 INFO:tasks.workunit.client.0.smithi042.stdout: create 280/36 file after the snapshot 2019-12-05T05:06:12.043 INFO:tasks.workunit.client.0.smithi042.stdout: create 280/37 file after the snapshot 2019-12-05T05:06:12.045 INFO:tasks.workunit.client.0.smithi042.stdout: create 280/38 file after the snapshot 2019-12-05T05:06:12.048 INFO:tasks.workunit.client.0.smithi042.stdout: create 280/39 file after the snapshot 2019-12-05T05:06:12.050 INFO:tasks.workunit.client.0.smithi042.stdout: create 281/.snap/snap-subdir-test 2019-12-05T05:06:12.068 INFO:tasks.workunit.client.0.smithi042.stdout: create 281/30 file after the snapshot 2019-12-05T05:06:12.070 INFO:tasks.workunit.client.0.smithi042.stdout: create 281/31 file after the snapshot 2019-12-05T05:06:12.073 INFO:tasks.workunit.client.0.smithi042.stdout: create 281/32 file after the snapshot 2019-12-05T05:06:12.075 INFO:tasks.workunit.client.0.smithi042.stdout: create 281/33 file after the snapshot 2019-12-05T05:06:12.078 INFO:tasks.workunit.client.0.smithi042.stdout: create 281/34 file after the snapshot 2019-12-05T05:06:12.081 INFO:tasks.workunit.client.0.smithi042.stdout: create 281/35 file after the snapshot 2019-12-05T05:06:12.083 INFO:tasks.workunit.client.0.smithi042.stdout: create 281/36 file after the snapshot 2019-12-05T05:06:12.086 INFO:tasks.workunit.client.0.smithi042.stdout: create 281/37 file after the snapshot 2019-12-05T05:06:12.089 INFO:tasks.workunit.client.0.smithi042.stdout: create 281/38 file after the snapshot 2019-12-05T05:06:12.091 INFO:tasks.workunit.client.0.smithi042.stdout: create 281/39 file after the snapshot 2019-12-05T05:06:12.094 INFO:tasks.workunit.client.0.smithi042.stdout: create 282/.snap/snap-subdir-test 2019-12-05T05:06:12.112 INFO:tasks.workunit.client.0.smithi042.stdout: create 282/30 file after the snapshot 2019-12-05T05:06:12.113 INFO:tasks.workunit.client.0.smithi042.stdout: create 282/31 file after the snapshot 2019-12-05T05:06:12.116 INFO:tasks.workunit.client.0.smithi042.stdout: create 282/32 file after the snapshot 2019-12-05T05:06:12.118 INFO:tasks.workunit.client.0.smithi042.stdout: create 282/33 file after the snapshot 2019-12-05T05:06:12.121 INFO:tasks.workunit.client.0.smithi042.stdout: create 282/34 file after the snapshot 2019-12-05T05:06:12.135 INFO:tasks.workunit.client.0.smithi042.stdout: create 282/35 file after the snapshot 2019-12-05T05:06:12.135 INFO:tasks.workunit.client.0.smithi042.stdout: create 282/36 file after the snapshot 2019-12-05T05:06:12.135 INFO:tasks.workunit.client.0.smithi042.stdout: create 282/37 file after the snapshot 2019-12-05T05:06:12.136 INFO:tasks.workunit.client.0.smithi042.stdout: create 282/38 file after the snapshot 2019-12-05T05:06:12.136 INFO:tasks.workunit.client.0.smithi042.stdout: create 282/39 file after the snapshot 2019-12-05T05:06:12.136 INFO:tasks.workunit.client.0.smithi042.stdout: create 283/.snap/snap-subdir-test 2019-12-05T05:06:12.152 INFO:tasks.workunit.client.0.smithi042.stdout: create 283/30 file after the snapshot 2019-12-05T05:06:12.155 INFO:tasks.workunit.client.0.smithi042.stdout: create 283/31 file after the snapshot 2019-12-05T05:06:12.157 INFO:tasks.workunit.client.0.smithi042.stdout: create 283/32 file after the snapshot 2019-12-05T05:06:12.160 INFO:tasks.workunit.client.0.smithi042.stdout: create 283/33 file after the snapshot 2019-12-05T05:06:12.162 INFO:tasks.workunit.client.0.smithi042.stdout: create 283/34 file after the snapshot 2019-12-05T05:06:12.163 INFO:tasks.workunit.client.0.smithi042.stdout: create 283/35 file after the snapshot 2019-12-05T05:06:12.165 INFO:tasks.workunit.client.0.smithi042.stdout: create 283/36 file after the snapshot 2019-12-05T05:06:12.168 INFO:tasks.workunit.client.0.smithi042.stdout: create 283/37 file after the snapshot 2019-12-05T05:06:12.170 INFO:tasks.workunit.client.0.smithi042.stdout: create 283/38 file after the snapshot 2019-12-05T05:06:12.172 INFO:tasks.workunit.client.0.smithi042.stdout: create 283/39 file after the snapshot 2019-12-05T05:06:12.175 INFO:tasks.workunit.client.0.smithi042.stdout: create 284/.snap/snap-subdir-test 2019-12-05T05:06:13.033 INFO:tasks.workunit.client.0.smithi042.stdout: create 284/30 file after the snapshot 2019-12-05T05:06:13.040 INFO:tasks.workunit.client.0.smithi042.stdout: create 284/31 file after the snapshot 2019-12-05T05:06:13.047 INFO:tasks.workunit.client.0.smithi042.stdout: create 284/32 file after the snapshot 2019-12-05T05:06:13.187 INFO:tasks.workunit.client.0.smithi042.stdout: create 284/33 file after the snapshot 2019-12-05T05:06:13.197 INFO:tasks.workunit.client.0.smithi042.stdout: create 284/34 file after the snapshot 2019-12-05T05:06:13.200 INFO:tasks.workunit.client.0.smithi042.stdout: create 284/35 file after the snapshot 2019-12-05T05:06:13.203 INFO:tasks.workunit.client.0.smithi042.stdout: create 284/36 file after the snapshot 2019-12-05T05:06:13.205 INFO:tasks.workunit.client.0.smithi042.stdout: create 284/37 file after the snapshot 2019-12-05T05:06:13.208 INFO:tasks.workunit.client.0.smithi042.stdout: create 284/38 file after the snapshot 2019-12-05T05:06:13.211 INFO:tasks.workunit.client.0.smithi042.stdout: create 284/39 file after the snapshot 2019-12-05T05:06:13.214 INFO:tasks.workunit.client.0.smithi042.stdout: create 285/.snap/snap-subdir-test 2019-12-05T05:06:13.239 INFO:tasks.workunit.client.0.smithi042.stdout: create 285/30 file after the snapshot 2019-12-05T05:06:13.239 INFO:tasks.workunit.client.0.smithi042.stdout: create 285/31 file after the snapshot 2019-12-05T05:06:13.239 INFO:tasks.workunit.client.0.smithi042.stdout: create 285/32 file after the snapshot 2019-12-05T05:06:13.241 INFO:tasks.workunit.client.0.smithi042.stdout: create 285/33 file after the snapshot 2019-12-05T05:06:13.244 INFO:tasks.workunit.client.0.smithi042.stdout: create 285/34 file after the snapshot 2019-12-05T05:06:13.246 INFO:tasks.workunit.client.0.smithi042.stdout: create 285/35 file after the snapshot 2019-12-05T05:06:13.249 INFO:tasks.workunit.client.0.smithi042.stdout: create 285/36 file after the snapshot 2019-12-05T05:06:13.252 INFO:tasks.workunit.client.0.smithi042.stdout: create 285/37 file after the snapshot 2019-12-05T05:06:13.254 INFO:tasks.workunit.client.0.smithi042.stdout: create 285/38 file after the snapshot 2019-12-05T05:06:13.256 INFO:tasks.workunit.client.0.smithi042.stdout: create 285/39 file after the snapshot 2019-12-05T05:06:13.258 INFO:tasks.workunit.client.0.smithi042.stdout: create 286/.snap/snap-subdir-test 2019-12-05T05:06:13.803 INFO:tasks.workunit.client.0.smithi042.stdout: create 286/30 file after the snapshot 2019-12-05T05:06:13.804 INFO:tasks.workunit.client.0.smithi042.stdout: create 286/31 file after the snapshot 2019-12-05T05:06:13.807 INFO:tasks.workunit.client.0.smithi042.stdout: create 286/32 file after the snapshot 2019-12-05T05:06:13.810 INFO:tasks.workunit.client.0.smithi042.stdout: create 286/33 file after the snapshot 2019-12-05T05:06:13.812 INFO:tasks.workunit.client.0.smithi042.stdout: create 286/34 file after the snapshot 2019-12-05T05:06:13.814 INFO:tasks.workunit.client.0.smithi042.stdout: create 286/35 file after the snapshot 2019-12-05T05:06:13.817 INFO:tasks.workunit.client.0.smithi042.stdout: create 286/36 file after the snapshot 2019-12-05T05:06:13.820 INFO:tasks.workunit.client.0.smithi042.stdout: create 286/37 file after the snapshot 2019-12-05T05:06:13.822 INFO:tasks.workunit.client.0.smithi042.stdout: create 286/38 file after the snapshot 2019-12-05T05:06:13.825 INFO:tasks.workunit.client.0.smithi042.stdout: create 286/39 file after the snapshot 2019-12-05T05:06:13.827 INFO:tasks.workunit.client.0.smithi042.stdout: create 287/.snap/snap-subdir-test 2019-12-05T05:06:14.214 INFO:tasks.workunit.client.0.smithi042.stdout: create 287/30 file after the snapshot 2019-12-05T05:06:14.221 INFO:tasks.workunit.client.0.smithi042.stdout: create 287/31 file after the snapshot 2019-12-05T05:06:14.224 INFO:tasks.workunit.client.0.smithi042.stdout: create 287/32 file after the snapshot 2019-12-05T05:06:14.227 INFO:tasks.workunit.client.0.smithi042.stdout: create 287/33 file after the snapshot 2019-12-05T05:06:14.230 INFO:tasks.workunit.client.0.smithi042.stdout: create 287/34 file after the snapshot 2019-12-05T05:06:14.232 INFO:tasks.workunit.client.0.smithi042.stdout: create 287/35 file after the snapshot 2019-12-05T05:06:14.234 INFO:tasks.workunit.client.0.smithi042.stdout: create 287/36 file after the snapshot 2019-12-05T05:06:14.237 INFO:tasks.workunit.client.0.smithi042.stdout: create 287/37 file after the snapshot 2019-12-05T05:06:14.239 INFO:tasks.workunit.client.0.smithi042.stdout: create 287/38 file after the snapshot 2019-12-05T05:06:14.243 INFO:tasks.workunit.client.0.smithi042.stdout: create 287/39 file after the snapshot 2019-12-05T05:06:14.245 INFO:tasks.workunit.client.0.smithi042.stdout: create 288/.snap/snap-subdir-test 2019-12-05T05:06:14.266 INFO:tasks.workunit.client.0.smithi042.stdout: create 288/30 file after the snapshot 2019-12-05T05:06:14.268 INFO:tasks.workunit.client.0.smithi042.stdout: create 288/31 file after the snapshot 2019-12-05T05:06:14.270 INFO:tasks.workunit.client.0.smithi042.stdout: create 288/32 file after the snapshot 2019-12-05T05:06:14.278 INFO:tasks.workunit.client.0.smithi042.stdout: create 288/33 file after the snapshot 2019-12-05T05:06:14.279 INFO:tasks.workunit.client.0.smithi042.stdout: create 288/34 file after the snapshot 2019-12-05T05:06:14.279 INFO:tasks.workunit.client.0.smithi042.stdout: create 288/35 file after the snapshot 2019-12-05T05:06:14.285 INFO:tasks.workunit.client.0.smithi042.stdout: create 288/36 file after the snapshot 2019-12-05T05:06:14.285 INFO:tasks.workunit.client.0.smithi042.stdout: create 288/37 file after the snapshot 2019-12-05T05:06:14.289 INFO:tasks.workunit.client.0.smithi042.stdout: create 288/38 file after the snapshot 2019-12-05T05:06:14.289 INFO:tasks.workunit.client.0.smithi042.stdout: create 288/39 file after the snapshot 2019-12-05T05:06:14.290 INFO:tasks.workunit.client.0.smithi042.stdout: create 289/.snap/snap-subdir-test 2019-12-05T05:06:14.799 INFO:tasks.workunit.client.0.smithi042.stdout: create 289/30 file after the snapshot 2019-12-05T05:06:14.803 INFO:tasks.workunit.client.0.smithi042.stdout: create 289/31 file after the snapshot 2019-12-05T05:06:14.810 INFO:tasks.workunit.client.0.smithi042.stdout: create 289/32 file after the snapshot 2019-12-05T05:06:14.812 INFO:tasks.workunit.client.0.smithi042.stdout: create 289/33 file after the snapshot 2019-12-05T05:06:14.815 INFO:tasks.workunit.client.0.smithi042.stdout: create 289/34 file after the snapshot 2019-12-05T05:06:14.819 INFO:tasks.workunit.client.0.smithi042.stdout: create 289/35 file after the snapshot 2019-12-05T05:06:14.822 INFO:tasks.workunit.client.0.smithi042.stdout: create 289/36 file after the snapshot 2019-12-05T05:06:14.824 INFO:tasks.workunit.client.0.smithi042.stdout: create 289/37 file after the snapshot 2019-12-05T05:06:14.827 INFO:tasks.workunit.client.0.smithi042.stdout: create 289/38 file after the snapshot 2019-12-05T05:06:14.829 INFO:tasks.workunit.client.0.smithi042.stdout: create 289/39 file after the snapshot 2019-12-05T05:06:14.832 INFO:tasks.workunit.client.0.smithi042.stdout: create 290/.snap/snap-subdir-test 2019-12-05T05:06:14.867 INFO:tasks.workunit.client.0.smithi042.stdout: create 290/30 file after the snapshot 2019-12-05T05:06:14.869 INFO:tasks.workunit.client.0.smithi042.stdout: create 290/31 file after the snapshot 2019-12-05T05:06:14.872 INFO:tasks.workunit.client.0.smithi042.stdout: create 290/32 file after the snapshot 2019-12-05T05:06:14.875 INFO:tasks.workunit.client.0.smithi042.stdout: create 290/33 file after the snapshot 2019-12-05T05:06:14.878 INFO:tasks.workunit.client.0.smithi042.stdout: create 290/34 file after the snapshot 2019-12-05T05:06:14.880 INFO:tasks.workunit.client.0.smithi042.stdout: create 290/35 file after the snapshot 2019-12-05T05:06:14.883 INFO:tasks.workunit.client.0.smithi042.stdout: create 290/36 file after the snapshot 2019-12-05T05:06:14.886 INFO:tasks.workunit.client.0.smithi042.stdout: create 290/37 file after the snapshot 2019-12-05T05:06:14.888 INFO:tasks.workunit.client.0.smithi042.stdout: create 290/38 file after the snapshot 2019-12-05T05:06:14.890 INFO:tasks.workunit.client.0.smithi042.stdout: create 290/39 file after the snapshot 2019-12-05T05:06:14.892 INFO:tasks.workunit.client.0.smithi042.stdout: create 291/.snap/snap-subdir-test 2019-12-05T05:06:14.910 INFO:tasks.workunit.client.0.smithi042.stdout: create 291/30 file after the snapshot 2019-12-05T05:06:14.917 INFO:tasks.workunit.client.0.smithi042.stdout: create 291/31 file after the snapshot 2019-12-05T05:06:14.917 INFO:tasks.workunit.client.0.smithi042.stdout: create 291/32 file after the snapshot 2019-12-05T05:06:14.917 INFO:tasks.workunit.client.0.smithi042.stdout: create 291/33 file after the snapshot 2019-12-05T05:06:14.925 INFO:tasks.workunit.client.0.smithi042.stdout: create 291/34 file after the snapshot 2019-12-05T05:06:14.925 INFO:tasks.workunit.client.0.smithi042.stdout: create 291/35 file after the snapshot 2019-12-05T05:06:14.925 INFO:tasks.workunit.client.0.smithi042.stdout: create 291/36 file after the snapshot 2019-12-05T05:06:14.930 INFO:tasks.workunit.client.0.smithi042.stdout: create 291/37 file after the snapshot 2019-12-05T05:06:14.930 INFO:tasks.workunit.client.0.smithi042.stdout: create 291/38 file after the snapshot 2019-12-05T05:06:14.935 INFO:tasks.workunit.client.0.smithi042.stdout: create 291/39 file after the snapshot 2019-12-05T05:06:14.935 INFO:tasks.workunit.client.0.smithi042.stdout: create 292/.snap/snap-subdir-test 2019-12-05T05:06:14.952 INFO:tasks.workunit.client.0.smithi042.stdout: create 292/30 file after the snapshot 2019-12-05T05:06:14.954 INFO:tasks.workunit.client.0.smithi042.stdout: create 292/31 file after the snapshot 2019-12-05T05:06:14.956 INFO:tasks.workunit.client.0.smithi042.stdout: create 292/32 file after the snapshot 2019-12-05T05:06:14.959 INFO:tasks.workunit.client.0.smithi042.stdout: create 292/33 file after the snapshot 2019-12-05T05:06:14.961 INFO:tasks.workunit.client.0.smithi042.stdout: create 292/34 file after the snapshot 2019-12-05T05:06:14.963 INFO:tasks.workunit.client.0.smithi042.stdout: create 292/35 file after the snapshot 2019-12-05T05:06:14.966 INFO:tasks.workunit.client.0.smithi042.stdout: create 292/36 file after the snapshot 2019-12-05T05:06:14.969 INFO:tasks.workunit.client.0.smithi042.stdout: create 292/37 file after the snapshot 2019-12-05T05:06:14.971 INFO:tasks.workunit.client.0.smithi042.stdout: create 292/38 file after the snapshot 2019-12-05T05:06:14.974 INFO:tasks.workunit.client.0.smithi042.stdout: create 292/39 file after the snapshot 2019-12-05T05:06:14.976 INFO:tasks.workunit.client.0.smithi042.stdout: create 293/.snap/snap-subdir-test 2019-12-05T05:06:14.997 INFO:tasks.workunit.client.0.smithi042.stdout: create 293/30 file after the snapshot 2019-12-05T05:06:15.000 INFO:tasks.workunit.client.0.smithi042.stdout: create 293/31 file after the snapshot 2019-12-05T05:06:15.003 INFO:tasks.workunit.client.0.smithi042.stdout: create 293/32 file after the snapshot 2019-12-05T05:06:15.006 INFO:tasks.workunit.client.0.smithi042.stdout: create 293/33 file after the snapshot 2019-12-05T05:06:15.008 INFO:tasks.workunit.client.0.smithi042.stdout: create 293/34 file after the snapshot 2019-12-05T05:06:15.011 INFO:tasks.workunit.client.0.smithi042.stdout: create 293/35 file after the snapshot 2019-12-05T05:06:15.014 INFO:tasks.workunit.client.0.smithi042.stdout: create 293/36 file after the snapshot 2019-12-05T05:06:15.016 INFO:tasks.workunit.client.0.smithi042.stdout: create 293/37 file after the snapshot 2019-12-05T05:06:15.018 INFO:tasks.workunit.client.0.smithi042.stdout: create 293/38 file after the snapshot 2019-12-05T05:06:15.021 INFO:tasks.workunit.client.0.smithi042.stdout: create 293/39 file after the snapshot 2019-12-05T05:06:15.023 INFO:tasks.workunit.client.0.smithi042.stdout: create 294/.snap/snap-subdir-test 2019-12-05T05:06:15.040 INFO:tasks.workunit.client.0.smithi042.stdout: create 294/30 file after the snapshot 2019-12-05T05:06:15.042 INFO:tasks.workunit.client.0.smithi042.stdout: create 294/31 file after the snapshot 2019-12-05T05:06:15.044 INFO:tasks.workunit.client.0.smithi042.stdout: create 294/32 file after the snapshot 2019-12-05T05:06:15.047 INFO:tasks.workunit.client.0.smithi042.stdout: create 294/33 file after the snapshot 2019-12-05T05:06:15.050 INFO:tasks.workunit.client.0.smithi042.stdout: create 294/34 file after the snapshot 2019-12-05T05:06:15.052 INFO:tasks.workunit.client.0.smithi042.stdout: create 294/35 file after the snapshot 2019-12-05T05:06:15.055 INFO:tasks.workunit.client.0.smithi042.stdout: create 294/36 file after the snapshot 2019-12-05T05:06:15.057 INFO:tasks.workunit.client.0.smithi042.stdout: create 294/37 file after the snapshot 2019-12-05T05:06:15.060 INFO:tasks.workunit.client.0.smithi042.stdout: create 294/38 file after the snapshot 2019-12-05T05:06:15.062 INFO:tasks.workunit.client.0.smithi042.stdout: create 294/39 file after the snapshot 2019-12-05T05:06:15.064 INFO:tasks.workunit.client.0.smithi042.stdout: create 295/.snap/snap-subdir-test 2019-12-05T05:06:15.088 INFO:tasks.workunit.client.0.smithi042.stdout: create 295/30 file after the snapshot 2019-12-05T05:06:15.089 INFO:tasks.workunit.client.0.smithi042.stdout: create 295/31 file after the snapshot 2019-12-05T05:06:15.089 INFO:tasks.workunit.client.0.smithi042.stdout: create 295/32 file after the snapshot 2019-12-05T05:06:15.091 INFO:tasks.workunit.client.0.smithi042.stdout: create 295/33 file after the snapshot 2019-12-05T05:06:15.094 INFO:tasks.workunit.client.0.smithi042.stdout: create 295/34 file after the snapshot 2019-12-05T05:06:15.097 INFO:tasks.workunit.client.0.smithi042.stdout: create 295/35 file after the snapshot 2019-12-05T05:06:15.099 INFO:tasks.workunit.client.0.smithi042.stdout: create 295/36 file after the snapshot 2019-12-05T05:06:15.103 INFO:tasks.workunit.client.0.smithi042.stdout: create 295/37 file after the snapshot 2019-12-05T05:06:15.109 INFO:tasks.workunit.client.0.smithi042.stdout: create 295/38 file after the snapshot 2019-12-05T05:06:15.109 INFO:tasks.workunit.client.0.smithi042.stdout: create 295/39 file after the snapshot 2019-12-05T05:06:15.109 INFO:tasks.workunit.client.0.smithi042.stdout: create 296/.snap/snap-subdir-test 2019-12-05T05:06:15.440 INFO:tasks.workunit.client.0.smithi042.stdout: create 296/30 file after the snapshot 2019-12-05T05:06:15.444 INFO:tasks.workunit.client.0.smithi042.stdout: create 296/31 file after the snapshot 2019-12-05T05:06:15.449 INFO:tasks.workunit.client.0.smithi042.stdout: create 296/32 file after the snapshot 2019-12-05T05:06:15.452 INFO:tasks.workunit.client.0.smithi042.stdout: create 296/33 file after the snapshot 2019-12-05T05:06:15.455 INFO:tasks.workunit.client.0.smithi042.stdout: create 296/34 file after the snapshot 2019-12-05T05:06:15.458 INFO:tasks.workunit.client.0.smithi042.stdout: create 296/35 file after the snapshot 2019-12-05T05:06:15.461 INFO:tasks.workunit.client.0.smithi042.stdout: create 296/36 file after the snapshot 2019-12-05T05:06:15.464 INFO:tasks.workunit.client.0.smithi042.stdout: create 296/37 file after the snapshot 2019-12-05T05:06:15.467 INFO:tasks.workunit.client.0.smithi042.stdout: create 296/38 file after the snapshot 2019-12-05T05:06:15.469 INFO:tasks.workunit.client.0.smithi042.stdout: create 296/39 file after the snapshot 2019-12-05T05:06:15.471 INFO:tasks.workunit.client.0.smithi042.stdout: create 297/.snap/snap-subdir-test 2019-12-05T05:06:15.493 INFO:tasks.workunit.client.0.smithi042.stdout: create 297/30 file after the snapshot 2019-12-05T05:06:15.495 INFO:tasks.workunit.client.0.smithi042.stdout: create 297/31 file after the snapshot 2019-12-05T05:06:15.498 INFO:tasks.workunit.client.0.smithi042.stdout: create 297/32 file after the snapshot 2019-12-05T05:06:15.501 INFO:tasks.workunit.client.0.smithi042.stdout: create 297/33 file after the snapshot 2019-12-05T05:06:15.503 INFO:tasks.workunit.client.0.smithi042.stdout: create 297/34 file after the snapshot 2019-12-05T05:06:15.505 INFO:tasks.workunit.client.0.smithi042.stdout: create 297/35 file after the snapshot 2019-12-05T05:06:15.508 INFO:tasks.workunit.client.0.smithi042.stdout: create 297/36 file after the snapshot 2019-12-05T05:06:15.509 INFO:tasks.workunit.client.0.smithi042.stdout: create 297/37 file after the snapshot 2019-12-05T05:06:15.512 INFO:tasks.workunit.client.0.smithi042.stdout: create 297/38 file after the snapshot 2019-12-05T05:06:15.514 INFO:tasks.workunit.client.0.smithi042.stdout: create 297/39 file after the snapshot 2019-12-05T05:06:15.516 INFO:tasks.workunit.client.0.smithi042.stdout: create 298/.snap/snap-subdir-test 2019-12-05T05:06:15.535 INFO:tasks.workunit.client.0.smithi042.stdout: create 298/30 file after the snapshot 2019-12-05T05:06:15.537 INFO:tasks.workunit.client.0.smithi042.stdout: create 298/31 file after the snapshot 2019-12-05T05:06:15.539 INFO:tasks.workunit.client.0.smithi042.stdout: create 298/32 file after the snapshot 2019-12-05T05:06:15.543 INFO:tasks.workunit.client.0.smithi042.stdout: create 298/33 file after the snapshot 2019-12-05T05:06:15.544 INFO:tasks.workunit.client.0.smithi042.stdout: create 298/34 file after the snapshot 2019-12-05T05:06:15.547 INFO:tasks.workunit.client.0.smithi042.stdout: create 298/35 file after the snapshot 2019-12-05T05:06:15.550 INFO:tasks.workunit.client.0.smithi042.stdout: create 298/36 file after the snapshot 2019-12-05T05:06:15.552 INFO:tasks.workunit.client.0.smithi042.stdout: create 298/37 file after the snapshot 2019-12-05T05:06:15.557 INFO:tasks.workunit.client.0.smithi042.stdout: create 298/38 file after the snapshot 2019-12-05T05:06:15.558 INFO:tasks.workunit.client.0.smithi042.stdout: create 298/39 file after the snapshot 2019-12-05T05:06:15.559 INFO:tasks.workunit.client.0.smithi042.stdout: create 299/.snap/snap-subdir-test 2019-12-05T05:06:15.576 INFO:tasks.workunit.client.0.smithi042.stdout: create 299/30 file after the snapshot 2019-12-05T05:06:15.578 INFO:tasks.workunit.client.0.smithi042.stdout: create 299/31 file after the snapshot 2019-12-05T05:06:15.580 INFO:tasks.workunit.client.0.smithi042.stdout: create 299/32 file after the snapshot 2019-12-05T05:06:15.582 INFO:tasks.workunit.client.0.smithi042.stdout: create 299/33 file after the snapshot 2019-12-05T05:06:15.585 INFO:tasks.workunit.client.0.smithi042.stdout: create 299/34 file after the snapshot 2019-12-05T05:06:15.588 INFO:tasks.workunit.client.0.smithi042.stdout: create 299/35 file after the snapshot 2019-12-05T05:06:15.591 INFO:tasks.workunit.client.0.smithi042.stdout: create 299/36 file after the snapshot 2019-12-05T05:06:15.594 INFO:tasks.workunit.client.0.smithi042.stdout: create 299/37 file after the snapshot 2019-12-05T05:06:15.604 INFO:tasks.workunit.client.0.smithi042.stdout: create 299/38 file after the snapshot 2019-12-05T05:06:15.604 INFO:tasks.workunit.client.0.smithi042.stdout: create 299/39 file after the snapshot 2019-12-05T05:06:15.605 INFO:tasks.workunit.client.0.smithi042.stdout:Take second snapshot .snap/test2 2019-12-05T05:06:15.720 INFO:tasks.workunit.client.0.smithi042.stdout:Copy content of .snap/test1 to copyofsnap1 ... 2019-12-05T05:06:23.222 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1' -> 'copyofsnap1/test1' 2019-12-05T05:06:23.223 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/100' -> 'copyofsnap1/test1/100' 2019-12-05T05:06:23.223 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/100/10' -> 'copyofsnap1/test1/100/10' 2019-12-05T05:06:23.223 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/100/11' -> 'copyofsnap1/test1/100/11' 2019-12-05T05:06:23.223 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/100/12' -> 'copyofsnap1/test1/100/12' 2019-12-05T05:06:23.223 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/100/13' -> 'copyofsnap1/test1/100/13' 2019-12-05T05:06:23.223 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/100/14' -> 'copyofsnap1/test1/100/14' 2019-12-05T05:06:23.224 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/100/15' -> 'copyofsnap1/test1/100/15' 2019-12-05T05:06:23.224 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/100/16' -> 'copyofsnap1/test1/100/16' 2019-12-05T05:06:23.224 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/100/17' -> 'copyofsnap1/test1/100/17' 2019-12-05T05:06:23.224 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/100/18' -> 'copyofsnap1/test1/100/18' 2019-12-05T05:06:23.224 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/100/19' -> 'copyofsnap1/test1/100/19' 2019-12-05T05:06:23.224 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/100/20' -> 'copyofsnap1/test1/100/20' 2019-12-05T05:06:23.224 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/101' -> 'copyofsnap1/test1/101' 2019-12-05T05:06:23.225 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/101/10' -> 'copyofsnap1/test1/101/10' 2019-12-05T05:06:23.225 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/101/11' -> 'copyofsnap1/test1/101/11' 2019-12-05T05:06:23.225 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/101/12' -> 'copyofsnap1/test1/101/12' 2019-12-05T05:06:23.225 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/101/13' -> 'copyofsnap1/test1/101/13' 2019-12-05T05:06:23.225 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/101/14' -> 'copyofsnap1/test1/101/14' 2019-12-05T05:06:23.225 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/101/15' -> 'copyofsnap1/test1/101/15' 2019-12-05T05:06:23.225 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/101/16' -> 'copyofsnap1/test1/101/16' 2019-12-05T05:06:23.225 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/101/17' -> 'copyofsnap1/test1/101/17' 2019-12-05T05:06:23.226 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/101/18' -> 'copyofsnap1/test1/101/18' 2019-12-05T05:06:23.226 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/101/19' -> 'copyofsnap1/test1/101/19' 2019-12-05T05:06:23.226 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/101/20' -> 'copyofsnap1/test1/101/20' 2019-12-05T05:06:23.226 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/102' -> 'copyofsnap1/test1/102' 2019-12-05T05:06:23.226 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/102/10' -> 'copyofsnap1/test1/102/10' 2019-12-05T05:06:23.226 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/102/11' -> 'copyofsnap1/test1/102/11' 2019-12-05T05:06:23.226 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/102/12' -> 'copyofsnap1/test1/102/12' 2019-12-05T05:06:23.227 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/102/13' -> 'copyofsnap1/test1/102/13' 2019-12-05T05:06:23.227 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/102/14' -> 'copyofsnap1/test1/102/14' 2019-12-05T05:06:23.227 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/102/15' -> 'copyofsnap1/test1/102/15' 2019-12-05T05:06:23.227 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/102/16' -> 'copyofsnap1/test1/102/16' 2019-12-05T05:06:23.227 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/102/17' -> 'copyofsnap1/test1/102/17' 2019-12-05T05:06:23.227 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/102/18' -> 'copyofsnap1/test1/102/18' 2019-12-05T05:06:23.227 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/102/19' -> 'copyofsnap1/test1/102/19' 2019-12-05T05:06:23.228 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/102/20' -> 'copyofsnap1/test1/102/20' 2019-12-05T05:06:23.228 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/103' -> 'copyofsnap1/test1/103' 2019-12-05T05:06:23.228 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/103/10' -> 'copyofsnap1/test1/103/10' 2019-12-05T05:06:23.228 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/103/11' -> 'copyofsnap1/test1/103/11' 2019-12-05T05:06:23.228 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/103/12' -> 'copyofsnap1/test1/103/12' 2019-12-05T05:06:23.228 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/103/13' -> 'copyofsnap1/test1/103/13' 2019-12-05T05:06:23.229 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/103/14' -> 'copyofsnap1/test1/103/14' 2019-12-05T05:06:23.229 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/103/15' -> 'copyofsnap1/test1/103/15' 2019-12-05T05:06:23.229 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/103/16' -> 'copyofsnap1/test1/103/16' 2019-12-05T05:06:23.229 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/103/17' -> 'copyofsnap1/test1/103/17' 2019-12-05T05:06:23.230 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/103/18' -> 'copyofsnap1/test1/103/18' 2019-12-05T05:06:23.230 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/103/19' -> 'copyofsnap1/test1/103/19' 2019-12-05T05:06:23.230 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/103/20' -> 'copyofsnap1/test1/103/20' 2019-12-05T05:06:23.230 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/104' -> 'copyofsnap1/test1/104' 2019-12-05T05:06:23.230 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/104/10' -> 'copyofsnap1/test1/104/10' 2019-12-05T05:06:23.230 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/104/11' -> 'copyofsnap1/test1/104/11' 2019-12-05T05:06:23.230 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/104/12' -> 'copyofsnap1/test1/104/12' 2019-12-05T05:06:23.231 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/104/13' -> 'copyofsnap1/test1/104/13' 2019-12-05T05:06:23.231 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/104/14' -> 'copyofsnap1/test1/104/14' 2019-12-05T05:06:23.231 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/104/15' -> 'copyofsnap1/test1/104/15' 2019-12-05T05:06:23.231 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/104/16' -> 'copyofsnap1/test1/104/16' 2019-12-05T05:06:23.231 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/104/17' -> 'copyofsnap1/test1/104/17' 2019-12-05T05:06:23.232 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/104/18' -> 'copyofsnap1/test1/104/18' 2019-12-05T05:06:23.232 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/104/19' -> 'copyofsnap1/test1/104/19' 2019-12-05T05:06:23.232 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/104/20' -> 'copyofsnap1/test1/104/20' 2019-12-05T05:06:23.232 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/105' -> 'copyofsnap1/test1/105' 2019-12-05T05:06:23.232 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/105/10' -> 'copyofsnap1/test1/105/10' 2019-12-05T05:06:23.232 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/105/11' -> 'copyofsnap1/test1/105/11' 2019-12-05T05:06:23.232 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/105/12' -> 'copyofsnap1/test1/105/12' 2019-12-05T05:06:23.233 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/105/13' -> 'copyofsnap1/test1/105/13' 2019-12-05T05:06:23.233 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/105/14' -> 'copyofsnap1/test1/105/14' 2019-12-05T05:06:23.233 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/105/15' -> 'copyofsnap1/test1/105/15' 2019-12-05T05:06:23.233 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/105/16' -> 'copyofsnap1/test1/105/16' 2019-12-05T05:06:23.233 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/105/17' -> 'copyofsnap1/test1/105/17' 2019-12-05T05:06:23.233 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/105/18' -> 'copyofsnap1/test1/105/18' 2019-12-05T05:06:23.233 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/105/19' -> 'copyofsnap1/test1/105/19' 2019-12-05T05:06:23.234 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/105/20' -> 'copyofsnap1/test1/105/20' 2019-12-05T05:06:23.234 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/106' -> 'copyofsnap1/test1/106' 2019-12-05T05:06:23.234 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/106/10' -> 'copyofsnap1/test1/106/10' 2019-12-05T05:06:23.234 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/106/11' -> 'copyofsnap1/test1/106/11' 2019-12-05T05:06:23.234 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/106/12' -> 'copyofsnap1/test1/106/12' 2019-12-05T05:06:23.234 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/106/13' -> 'copyofsnap1/test1/106/13' 2019-12-05T05:06:23.234 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/106/14' -> 'copyofsnap1/test1/106/14' 2019-12-05T05:06:23.234 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/106/15' -> 'copyofsnap1/test1/106/15' 2019-12-05T05:06:23.235 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/106/16' -> 'copyofsnap1/test1/106/16' 2019-12-05T05:06:26.966 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/106/17' -> 'copyofsnap1/test1/106/17' 2019-12-05T05:06:26.967 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/106/18' -> 'copyofsnap1/test1/106/18' 2019-12-05T05:06:26.967 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/106/19' -> 'copyofsnap1/test1/106/19' 2019-12-05T05:06:26.967 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/106/20' -> 'copyofsnap1/test1/106/20' 2019-12-05T05:06:26.967 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/107' -> 'copyofsnap1/test1/107' 2019-12-05T05:06:26.967 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/107/10' -> 'copyofsnap1/test1/107/10' 2019-12-05T05:06:26.967 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/107/11' -> 'copyofsnap1/test1/107/11' 2019-12-05T05:06:26.968 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/107/12' -> 'copyofsnap1/test1/107/12' 2019-12-05T05:06:26.968 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/107/13' -> 'copyofsnap1/test1/107/13' 2019-12-05T05:06:26.968 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/107/14' -> 'copyofsnap1/test1/107/14' 2019-12-05T05:06:26.968 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/107/15' -> 'copyofsnap1/test1/107/15' 2019-12-05T05:06:26.968 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/107/16' -> 'copyofsnap1/test1/107/16' 2019-12-05T05:06:26.968 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/107/17' -> 'copyofsnap1/test1/107/17' 2019-12-05T05:06:26.968 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/107/18' -> 'copyofsnap1/test1/107/18' 2019-12-05T05:06:26.969 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/107/19' -> 'copyofsnap1/test1/107/19' 2019-12-05T05:06:26.969 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/107/20' -> 'copyofsnap1/test1/107/20' 2019-12-05T05:06:26.969 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/108' -> 'copyofsnap1/test1/108' 2019-12-05T05:06:26.969 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/108/10' -> 'copyofsnap1/test1/108/10' 2019-12-05T05:06:26.969 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/108/11' -> 'copyofsnap1/test1/108/11' 2019-12-05T05:06:26.969 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/108/12' -> 'copyofsnap1/test1/108/12' 2019-12-05T05:06:26.970 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/108/13' -> 'copyofsnap1/test1/108/13' 2019-12-05T05:06:26.970 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/108/14' -> 'copyofsnap1/test1/108/14' 2019-12-05T05:06:26.970 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/108/15' -> 'copyofsnap1/test1/108/15' 2019-12-05T05:06:26.970 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/108/16' -> 'copyofsnap1/test1/108/16' 2019-12-05T05:06:26.970 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/108/17' -> 'copyofsnap1/test1/108/17' 2019-12-05T05:06:26.970 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/108/18' -> 'copyofsnap1/test1/108/18' 2019-12-05T05:06:26.971 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/108/19' -> 'copyofsnap1/test1/108/19' 2019-12-05T05:06:26.971 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/108/20' -> 'copyofsnap1/test1/108/20' 2019-12-05T05:06:26.971 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/109' -> 'copyofsnap1/test1/109' 2019-12-05T05:06:26.971 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/109/10' -> 'copyofsnap1/test1/109/10' 2019-12-05T05:06:26.971 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/109/11' -> 'copyofsnap1/test1/109/11' 2019-12-05T05:06:26.971 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/109/12' -> 'copyofsnap1/test1/109/12' 2019-12-05T05:06:26.972 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/109/13' -> 'copyofsnap1/test1/109/13' 2019-12-05T05:06:26.972 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/109/14' -> 'copyofsnap1/test1/109/14' 2019-12-05T05:06:26.972 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/109/15' -> 'copyofsnap1/test1/109/15' 2019-12-05T05:06:26.972 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/109/16' -> 'copyofsnap1/test1/109/16' 2019-12-05T05:06:26.972 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/109/17' -> 'copyofsnap1/test1/109/17' 2019-12-05T05:06:26.972 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/109/18' -> 'copyofsnap1/test1/109/18' 2019-12-05T05:06:26.973 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/109/19' -> 'copyofsnap1/test1/109/19' 2019-12-05T05:06:26.973 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/109/20' -> 'copyofsnap1/test1/109/20' 2019-12-05T05:06:26.973 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/110' -> 'copyofsnap1/test1/110' 2019-12-05T05:06:26.973 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/110/10' -> 'copyofsnap1/test1/110/10' 2019-12-05T05:06:26.973 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/110/11' -> 'copyofsnap1/test1/110/11' 2019-12-05T05:06:26.973 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/110/12' -> 'copyofsnap1/test1/110/12' 2019-12-05T05:06:26.974 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/110/13' -> 'copyofsnap1/test1/110/13' 2019-12-05T05:06:26.974 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/110/14' -> 'copyofsnap1/test1/110/14' 2019-12-05T05:06:26.974 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/110/15' -> 'copyofsnap1/test1/110/15' 2019-12-05T05:06:26.974 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/110/16' -> 'copyofsnap1/test1/110/16' 2019-12-05T05:06:26.974 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/110/17' -> 'copyofsnap1/test1/110/17' 2019-12-05T05:06:26.974 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/110/18' -> 'copyofsnap1/test1/110/18' 2019-12-05T05:06:26.974 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/110/19' -> 'copyofsnap1/test1/110/19' 2019-12-05T05:06:26.975 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/110/20' -> 'copyofsnap1/test1/110/20' 2019-12-05T05:06:26.975 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/111' -> 'copyofsnap1/test1/111' 2019-12-05T05:06:26.975 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/111/10' -> 'copyofsnap1/test1/111/10' 2019-12-05T05:06:26.975 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/111/11' -> 'copyofsnap1/test1/111/11' 2019-12-05T05:06:26.975 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/111/12' -> 'copyofsnap1/test1/111/12' 2019-12-05T05:06:26.975 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/111/13' -> 'copyofsnap1/test1/111/13' 2019-12-05T05:06:26.976 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/111/14' -> 'copyofsnap1/test1/111/14' 2019-12-05T05:06:26.976 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/111/15' -> 'copyofsnap1/test1/111/15' 2019-12-05T05:06:26.976 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/111/16' -> 'copyofsnap1/test1/111/16' 2019-12-05T05:06:26.976 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/111/17' -> 'copyofsnap1/test1/111/17' 2019-12-05T05:06:26.976 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/111/18' -> 'copyofsnap1/test1/111/18' 2019-12-05T05:06:26.976 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/111/19' -> 'copyofsnap1/test1/111/19' 2019-12-05T05:06:26.977 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/111/20' -> 'copyofsnap1/test1/111/20' 2019-12-05T05:06:26.977 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/112' -> 'copyofsnap1/test1/112' 2019-12-05T05:06:26.977 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/112/10' -> 'copyofsnap1/test1/112/10' 2019-12-05T05:06:26.977 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/112/11' -> 'copyofsnap1/test1/112/11' 2019-12-05T05:06:26.977 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/112/12' -> 'copyofsnap1/test1/112/12' 2019-12-05T05:06:26.977 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/112/13' -> 'copyofsnap1/test1/112/13' 2019-12-05T05:06:26.977 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/112/14' -> 'copyofsnap1/test1/112/14' 2019-12-05T05:06:26.978 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/112/15' -> 'copyofsnap1/test1/112/15' 2019-12-05T05:06:26.978 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/112/16' -> 'copyofsnap1/test1/112/16' 2019-12-05T05:06:26.978 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/112/17' -> 'copyofsnap1/test1/112/17' 2019-12-05T05:06:26.978 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/112/18' -> 'copyofsnap1/test1/112/18' 2019-12-05T05:06:26.978 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/112/19' -> 'copyofsnap1/test1/112/19' 2019-12-05T05:06:26.978 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/112/20' -> 'copyofsnap1/test1/112/20' 2019-12-05T05:06:26.978 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/113' -> 'copyofsnap1/test1/113' 2019-12-05T05:06:26.979 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/113/10' -> 'copyofsnap1/test1/113/10' 2019-12-05T05:06:26.979 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/113/11' -> 'copyofsnap1/test1/113/11' 2019-12-05T05:06:26.979 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/113/12' -> 'copyofsnap1/test1/113/12' 2019-12-05T05:06:26.979 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/113/13' -> 'copyofsnap1/test1/113/13' 2019-12-05T05:06:31.844 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/113/14' -> 'copyofsnap1/test1/113/14' 2019-12-05T05:06:31.845 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/113/15' -> 'copyofsnap1/test1/113/15' 2019-12-05T05:06:31.845 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/113/16' -> 'copyofsnap1/test1/113/16' 2019-12-05T05:06:31.845 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/113/17' -> 'copyofsnap1/test1/113/17' 2019-12-05T05:06:31.845 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/113/18' -> 'copyofsnap1/test1/113/18' 2019-12-05T05:06:31.845 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/113/19' -> 'copyofsnap1/test1/113/19' 2019-12-05T05:06:31.845 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/113/20' -> 'copyofsnap1/test1/113/20' 2019-12-05T05:06:31.845 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/114' -> 'copyofsnap1/test1/114' 2019-12-05T05:06:31.846 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/114/10' -> 'copyofsnap1/test1/114/10' 2019-12-05T05:06:31.846 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/114/11' -> 'copyofsnap1/test1/114/11' 2019-12-05T05:06:31.846 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/114/12' -> 'copyofsnap1/test1/114/12' 2019-12-05T05:06:31.846 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/114/13' -> 'copyofsnap1/test1/114/13' 2019-12-05T05:06:31.846 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/114/14' -> 'copyofsnap1/test1/114/14' 2019-12-05T05:06:31.846 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/114/15' -> 'copyofsnap1/test1/114/15' 2019-12-05T05:06:31.846 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/114/16' -> 'copyofsnap1/test1/114/16' 2019-12-05T05:06:31.847 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/114/17' -> 'copyofsnap1/test1/114/17' 2019-12-05T05:06:31.847 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/114/18' -> 'copyofsnap1/test1/114/18' 2019-12-05T05:06:31.847 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/114/19' -> 'copyofsnap1/test1/114/19' 2019-12-05T05:06:31.847 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/114/20' -> 'copyofsnap1/test1/114/20' 2019-12-05T05:06:31.847 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/115' -> 'copyofsnap1/test1/115' 2019-12-05T05:06:31.847 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/115/10' -> 'copyofsnap1/test1/115/10' 2019-12-05T05:06:31.847 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/115/11' -> 'copyofsnap1/test1/115/11' 2019-12-05T05:06:31.848 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/115/12' -> 'copyofsnap1/test1/115/12' 2019-12-05T05:06:31.848 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/115/13' -> 'copyofsnap1/test1/115/13' 2019-12-05T05:06:31.848 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/115/14' -> 'copyofsnap1/test1/115/14' 2019-12-05T05:06:31.849 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/115/15' -> 'copyofsnap1/test1/115/15' 2019-12-05T05:06:31.849 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/115/16' -> 'copyofsnap1/test1/115/16' 2019-12-05T05:06:31.849 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/115/17' -> 'copyofsnap1/test1/115/17' 2019-12-05T05:06:31.849 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/115/18' -> 'copyofsnap1/test1/115/18' 2019-12-05T05:06:31.849 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/115/19' -> 'copyofsnap1/test1/115/19' 2019-12-05T05:06:31.849 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/115/20' -> 'copyofsnap1/test1/115/20' 2019-12-05T05:06:31.850 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/116' -> 'copyofsnap1/test1/116' 2019-12-05T05:06:31.850 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/116/10' -> 'copyofsnap1/test1/116/10' 2019-12-05T05:06:31.850 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/116/11' -> 'copyofsnap1/test1/116/11' 2019-12-05T05:06:31.850 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/116/12' -> 'copyofsnap1/test1/116/12' 2019-12-05T05:06:31.850 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/116/13' -> 'copyofsnap1/test1/116/13' 2019-12-05T05:06:31.850 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/116/14' -> 'copyofsnap1/test1/116/14' 2019-12-05T05:06:31.850 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/116/15' -> 'copyofsnap1/test1/116/15' 2019-12-05T05:06:31.851 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/116/16' -> 'copyofsnap1/test1/116/16' 2019-12-05T05:06:31.851 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/116/17' -> 'copyofsnap1/test1/116/17' 2019-12-05T05:06:31.851 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/116/18' -> 'copyofsnap1/test1/116/18' 2019-12-05T05:06:31.851 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/116/19' -> 'copyofsnap1/test1/116/19' 2019-12-05T05:06:31.851 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/116/20' -> 'copyofsnap1/test1/116/20' 2019-12-05T05:06:31.851 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/117' -> 'copyofsnap1/test1/117' 2019-12-05T05:06:31.851 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/117/10' -> 'copyofsnap1/test1/117/10' 2019-12-05T05:06:31.852 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/117/11' -> 'copyofsnap1/test1/117/11' 2019-12-05T05:06:31.852 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/117/12' -> 'copyofsnap1/test1/117/12' 2019-12-05T05:06:31.852 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/117/13' -> 'copyofsnap1/test1/117/13' 2019-12-05T05:06:31.852 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/117/14' -> 'copyofsnap1/test1/117/14' 2019-12-05T05:06:31.852 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/117/15' -> 'copyofsnap1/test1/117/15' 2019-12-05T05:06:31.852 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/117/16' -> 'copyofsnap1/test1/117/16' 2019-12-05T05:06:31.852 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/117/17' -> 'copyofsnap1/test1/117/17' 2019-12-05T05:06:31.853 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/117/18' -> 'copyofsnap1/test1/117/18' 2019-12-05T05:06:31.853 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/117/19' -> 'copyofsnap1/test1/117/19' 2019-12-05T05:06:31.853 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/117/20' -> 'copyofsnap1/test1/117/20' 2019-12-05T05:06:31.853 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/118' -> 'copyofsnap1/test1/118' 2019-12-05T05:06:31.853 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/118/10' -> 'copyofsnap1/test1/118/10' 2019-12-05T05:06:31.853 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/118/11' -> 'copyofsnap1/test1/118/11' 2019-12-05T05:06:31.853 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/118/12' -> 'copyofsnap1/test1/118/12' 2019-12-05T05:06:31.854 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/118/13' -> 'copyofsnap1/test1/118/13' 2019-12-05T05:06:31.854 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/118/14' -> 'copyofsnap1/test1/118/14' 2019-12-05T05:06:31.854 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/118/15' -> 'copyofsnap1/test1/118/15' 2019-12-05T05:06:31.854 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/118/16' -> 'copyofsnap1/test1/118/16' 2019-12-05T05:06:31.854 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/118/17' -> 'copyofsnap1/test1/118/17' 2019-12-05T05:06:31.854 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/118/18' -> 'copyofsnap1/test1/118/18' 2019-12-05T05:06:31.855 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/118/19' -> 'copyofsnap1/test1/118/19' 2019-12-05T05:06:31.855 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/118/20' -> 'copyofsnap1/test1/118/20' 2019-12-05T05:06:31.855 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/119' -> 'copyofsnap1/test1/119' 2019-12-05T05:06:31.855 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/119/10' -> 'copyofsnap1/test1/119/10' 2019-12-05T05:06:31.855 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/119/11' -> 'copyofsnap1/test1/119/11' 2019-12-05T05:06:31.855 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/119/12' -> 'copyofsnap1/test1/119/12' 2019-12-05T05:06:31.856 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/119/13' -> 'copyofsnap1/test1/119/13' 2019-12-05T05:06:31.856 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/119/14' -> 'copyofsnap1/test1/119/14' 2019-12-05T05:06:31.856 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/119/15' -> 'copyofsnap1/test1/119/15' 2019-12-05T05:06:31.856 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/119/16' -> 'copyofsnap1/test1/119/16' 2019-12-05T05:06:31.856 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/119/17' -> 'copyofsnap1/test1/119/17' 2019-12-05T05:06:31.856 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/119/18' -> 'copyofsnap1/test1/119/18' 2019-12-05T05:06:31.856 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/119/19' -> 'copyofsnap1/test1/119/19' 2019-12-05T05:06:31.857 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/119/20' -> 'copyofsnap1/test1/119/20' 2019-12-05T05:06:31.857 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/120' -> 'copyofsnap1/test1/120' 2019-12-05T05:06:31.857 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/120/10' -> 'copyofsnap1/test1/120/10' 2019-12-05T05:06:34.674 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:06:34.679 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:06:36.661 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/120/11' -> 'copyofsnap1/test1/120/11' 2019-12-05T05:06:36.662 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/120/12' -> 'copyofsnap1/test1/120/12' 2019-12-05T05:06:36.662 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/120/13' -> 'copyofsnap1/test1/120/13' 2019-12-05T05:06:36.662 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/120/14' -> 'copyofsnap1/test1/120/14' 2019-12-05T05:06:36.662 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/120/15' -> 'copyofsnap1/test1/120/15' 2019-12-05T05:06:36.663 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/120/16' -> 'copyofsnap1/test1/120/16' 2019-12-05T05:06:36.663 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/120/17' -> 'copyofsnap1/test1/120/17' 2019-12-05T05:06:36.663 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/120/18' -> 'copyofsnap1/test1/120/18' 2019-12-05T05:06:36.663 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/120/19' -> 'copyofsnap1/test1/120/19' 2019-12-05T05:06:36.664 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/120/20' -> 'copyofsnap1/test1/120/20' 2019-12-05T05:06:36.664 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/121' -> 'copyofsnap1/test1/121' 2019-12-05T05:06:36.664 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/121/10' -> 'copyofsnap1/test1/121/10' 2019-12-05T05:06:36.664 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/121/11' -> 'copyofsnap1/test1/121/11' 2019-12-05T05:06:36.665 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/121/12' -> 'copyofsnap1/test1/121/12' 2019-12-05T05:06:36.665 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/121/13' -> 'copyofsnap1/test1/121/13' 2019-12-05T05:06:36.665 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/121/14' -> 'copyofsnap1/test1/121/14' 2019-12-05T05:06:36.665 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/121/15' -> 'copyofsnap1/test1/121/15' 2019-12-05T05:06:36.666 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/121/16' -> 'copyofsnap1/test1/121/16' 2019-12-05T05:06:36.666 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/121/17' -> 'copyofsnap1/test1/121/17' 2019-12-05T05:06:36.666 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/121/18' -> 'copyofsnap1/test1/121/18' 2019-12-05T05:06:36.666 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/121/19' -> 'copyofsnap1/test1/121/19' 2019-12-05T05:06:36.666 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/121/20' -> 'copyofsnap1/test1/121/20' 2019-12-05T05:06:36.666 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/122' -> 'copyofsnap1/test1/122' 2019-12-05T05:06:36.666 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/122/10' -> 'copyofsnap1/test1/122/10' 2019-12-05T05:06:36.667 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/122/11' -> 'copyofsnap1/test1/122/11' 2019-12-05T05:06:36.667 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/122/12' -> 'copyofsnap1/test1/122/12' 2019-12-05T05:06:36.667 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/122/13' -> 'copyofsnap1/test1/122/13' 2019-12-05T05:06:36.667 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/122/14' -> 'copyofsnap1/test1/122/14' 2019-12-05T05:06:36.667 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/122/15' -> 'copyofsnap1/test1/122/15' 2019-12-05T05:06:36.668 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/122/16' -> 'copyofsnap1/test1/122/16' 2019-12-05T05:06:36.668 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/122/17' -> 'copyofsnap1/test1/122/17' 2019-12-05T05:06:36.669 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/122/18' -> 'copyofsnap1/test1/122/18' 2019-12-05T05:06:36.669 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/122/19' -> 'copyofsnap1/test1/122/19' 2019-12-05T05:06:36.669 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/122/20' -> 'copyofsnap1/test1/122/20' 2019-12-05T05:06:36.669 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/123' -> 'copyofsnap1/test1/123' 2019-12-05T05:06:36.669 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/123/10' -> 'copyofsnap1/test1/123/10' 2019-12-05T05:06:36.669 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/123/11' -> 'copyofsnap1/test1/123/11' 2019-12-05T05:06:36.669 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/123/12' -> 'copyofsnap1/test1/123/12' 2019-12-05T05:06:36.669 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/123/13' -> 'copyofsnap1/test1/123/13' 2019-12-05T05:06:36.670 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/123/14' -> 'copyofsnap1/test1/123/14' 2019-12-05T05:06:36.670 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/123/15' -> 'copyofsnap1/test1/123/15' 2019-12-05T05:06:36.670 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/123/16' -> 'copyofsnap1/test1/123/16' 2019-12-05T05:06:36.670 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/123/17' -> 'copyofsnap1/test1/123/17' 2019-12-05T05:06:36.670 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/123/18' -> 'copyofsnap1/test1/123/18' 2019-12-05T05:06:36.671 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/123/19' -> 'copyofsnap1/test1/123/19' 2019-12-05T05:06:36.671 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/123/20' -> 'copyofsnap1/test1/123/20' 2019-12-05T05:06:36.671 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/124' -> 'copyofsnap1/test1/124' 2019-12-05T05:06:36.671 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/124/10' -> 'copyofsnap1/test1/124/10' 2019-12-05T05:06:36.671 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/124/11' -> 'copyofsnap1/test1/124/11' 2019-12-05T05:06:36.671 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/124/12' -> 'copyofsnap1/test1/124/12' 2019-12-05T05:06:36.671 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/124/13' -> 'copyofsnap1/test1/124/13' 2019-12-05T05:06:36.672 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/124/14' -> 'copyofsnap1/test1/124/14' 2019-12-05T05:06:36.672 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/124/15' -> 'copyofsnap1/test1/124/15' 2019-12-05T05:06:36.672 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/124/16' -> 'copyofsnap1/test1/124/16' 2019-12-05T05:06:36.672 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/124/17' -> 'copyofsnap1/test1/124/17' 2019-12-05T05:06:36.672 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/124/18' -> 'copyofsnap1/test1/124/18' 2019-12-05T05:06:36.672 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/124/19' -> 'copyofsnap1/test1/124/19' 2019-12-05T05:06:36.672 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/124/20' -> 'copyofsnap1/test1/124/20' 2019-12-05T05:06:36.673 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/125' -> 'copyofsnap1/test1/125' 2019-12-05T05:06:36.673 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/125/10' -> 'copyofsnap1/test1/125/10' 2019-12-05T05:06:36.673 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/125/11' -> 'copyofsnap1/test1/125/11' 2019-12-05T05:06:36.673 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/125/12' -> 'copyofsnap1/test1/125/12' 2019-12-05T05:06:36.673 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/125/13' -> 'copyofsnap1/test1/125/13' 2019-12-05T05:06:36.673 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/125/14' -> 'copyofsnap1/test1/125/14' 2019-12-05T05:06:36.673 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/125/15' -> 'copyofsnap1/test1/125/15' 2019-12-05T05:06:36.673 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/125/16' -> 'copyofsnap1/test1/125/16' 2019-12-05T05:06:36.674 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/125/17' -> 'copyofsnap1/test1/125/17' 2019-12-05T05:06:36.674 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/125/18' -> 'copyofsnap1/test1/125/18' 2019-12-05T05:06:36.674 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/125/19' -> 'copyofsnap1/test1/125/19' 2019-12-05T05:06:36.674 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/125/20' -> 'copyofsnap1/test1/125/20' 2019-12-05T05:06:36.674 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/126' -> 'copyofsnap1/test1/126' 2019-12-05T05:06:36.674 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/126/10' -> 'copyofsnap1/test1/126/10' 2019-12-05T05:06:36.674 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/126/11' -> 'copyofsnap1/test1/126/11' 2019-12-05T05:06:36.675 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/126/12' -> 'copyofsnap1/test1/126/12' 2019-12-05T05:06:36.675 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/126/13' -> 'copyofsnap1/test1/126/13' 2019-12-05T05:06:36.675 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/126/14' -> 'copyofsnap1/test1/126/14' 2019-12-05T05:06:36.675 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/126/15' -> 'copyofsnap1/test1/126/15' 2019-12-05T05:06:36.675 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/126/16' -> 'copyofsnap1/test1/126/16' 2019-12-05T05:06:36.675 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/126/17' -> 'copyofsnap1/test1/126/17' 2019-12-05T05:06:36.675 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/126/18' -> 'copyofsnap1/test1/126/18' 2019-12-05T05:06:36.675 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/126/19' -> 'copyofsnap1/test1/126/19' 2019-12-05T05:06:39.493 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/126/20' -> 'copyofsnap1/test1/126/20' 2019-12-05T05:06:39.493 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/127' -> 'copyofsnap1/test1/127' 2019-12-05T05:06:39.493 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/127/10' -> 'copyofsnap1/test1/127/10' 2019-12-05T05:06:39.494 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/127/11' -> 'copyofsnap1/test1/127/11' 2019-12-05T05:06:39.494 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/127/12' -> 'copyofsnap1/test1/127/12' 2019-12-05T05:06:39.494 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/127/13' -> 'copyofsnap1/test1/127/13' 2019-12-05T05:06:39.494 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/127/14' -> 'copyofsnap1/test1/127/14' 2019-12-05T05:06:39.494 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/127/15' -> 'copyofsnap1/test1/127/15' 2019-12-05T05:06:39.494 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/127/16' -> 'copyofsnap1/test1/127/16' 2019-12-05T05:06:39.494 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/127/17' -> 'copyofsnap1/test1/127/17' 2019-12-05T05:06:39.495 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/127/18' -> 'copyofsnap1/test1/127/18' 2019-12-05T05:06:39.495 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/127/19' -> 'copyofsnap1/test1/127/19' 2019-12-05T05:06:39.495 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/127/20' -> 'copyofsnap1/test1/127/20' 2019-12-05T05:06:39.495 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/128' -> 'copyofsnap1/test1/128' 2019-12-05T05:06:39.495 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/128/10' -> 'copyofsnap1/test1/128/10' 2019-12-05T05:06:39.495 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/128/11' -> 'copyofsnap1/test1/128/11' 2019-12-05T05:06:39.495 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/128/12' -> 'copyofsnap1/test1/128/12' 2019-12-05T05:06:39.495 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/128/13' -> 'copyofsnap1/test1/128/13' 2019-12-05T05:06:39.496 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/128/14' -> 'copyofsnap1/test1/128/14' 2019-12-05T05:06:39.496 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/128/15' -> 'copyofsnap1/test1/128/15' 2019-12-05T05:06:39.496 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/128/16' -> 'copyofsnap1/test1/128/16' 2019-12-05T05:06:39.496 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/128/17' -> 'copyofsnap1/test1/128/17' 2019-12-05T05:06:39.496 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/128/18' -> 'copyofsnap1/test1/128/18' 2019-12-05T05:06:39.496 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/128/19' -> 'copyofsnap1/test1/128/19' 2019-12-05T05:06:39.496 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/128/20' -> 'copyofsnap1/test1/128/20' 2019-12-05T05:06:39.497 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/129' -> 'copyofsnap1/test1/129' 2019-12-05T05:06:39.497 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/129/10' -> 'copyofsnap1/test1/129/10' 2019-12-05T05:06:39.497 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/129/11' -> 'copyofsnap1/test1/129/11' 2019-12-05T05:06:39.497 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/129/12' -> 'copyofsnap1/test1/129/12' 2019-12-05T05:06:39.497 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/129/13' -> 'copyofsnap1/test1/129/13' 2019-12-05T05:06:39.497 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/129/14' -> 'copyofsnap1/test1/129/14' 2019-12-05T05:06:39.497 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/129/15' -> 'copyofsnap1/test1/129/15' 2019-12-05T05:06:39.498 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/129/16' -> 'copyofsnap1/test1/129/16' 2019-12-05T05:06:39.498 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/129/17' -> 'copyofsnap1/test1/129/17' 2019-12-05T05:06:39.498 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/129/18' -> 'copyofsnap1/test1/129/18' 2019-12-05T05:06:39.498 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/129/19' -> 'copyofsnap1/test1/129/19' 2019-12-05T05:06:39.498 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/129/20' -> 'copyofsnap1/test1/129/20' 2019-12-05T05:06:39.498 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/130' -> 'copyofsnap1/test1/130' 2019-12-05T05:06:39.498 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/130/10' -> 'copyofsnap1/test1/130/10' 2019-12-05T05:06:39.498 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/130/11' -> 'copyofsnap1/test1/130/11' 2019-12-05T05:06:39.499 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/130/12' -> 'copyofsnap1/test1/130/12' 2019-12-05T05:06:39.499 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/130/13' -> 'copyofsnap1/test1/130/13' 2019-12-05T05:06:39.499 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/130/14' -> 'copyofsnap1/test1/130/14' 2019-12-05T05:06:39.499 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/130/15' -> 'copyofsnap1/test1/130/15' 2019-12-05T05:06:39.499 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/130/16' -> 'copyofsnap1/test1/130/16' 2019-12-05T05:06:39.499 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/130/17' -> 'copyofsnap1/test1/130/17' 2019-12-05T05:06:39.499 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/130/18' -> 'copyofsnap1/test1/130/18' 2019-12-05T05:06:39.500 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/130/19' -> 'copyofsnap1/test1/130/19' 2019-12-05T05:06:39.500 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/130/20' -> 'copyofsnap1/test1/130/20' 2019-12-05T05:06:39.500 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/131' -> 'copyofsnap1/test1/131' 2019-12-05T05:06:39.500 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/131/10' -> 'copyofsnap1/test1/131/10' 2019-12-05T05:06:39.500 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/131/11' -> 'copyofsnap1/test1/131/11' 2019-12-05T05:06:39.500 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/131/12' -> 'copyofsnap1/test1/131/12' 2019-12-05T05:06:39.500 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/131/13' -> 'copyofsnap1/test1/131/13' 2019-12-05T05:06:39.501 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/131/14' -> 'copyofsnap1/test1/131/14' 2019-12-05T05:06:39.501 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/131/15' -> 'copyofsnap1/test1/131/15' 2019-12-05T05:06:39.501 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/131/16' -> 'copyofsnap1/test1/131/16' 2019-12-05T05:06:39.501 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/131/17' -> 'copyofsnap1/test1/131/17' 2019-12-05T05:06:39.501 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/131/18' -> 'copyofsnap1/test1/131/18' 2019-12-05T05:06:39.501 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/131/19' -> 'copyofsnap1/test1/131/19' 2019-12-05T05:06:39.501 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/131/20' -> 'copyofsnap1/test1/131/20' 2019-12-05T05:06:39.501 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/132' -> 'copyofsnap1/test1/132' 2019-12-05T05:06:39.502 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/132/10' -> 'copyofsnap1/test1/132/10' 2019-12-05T05:06:39.502 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/132/11' -> 'copyofsnap1/test1/132/11' 2019-12-05T05:06:39.502 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/132/12' -> 'copyofsnap1/test1/132/12' 2019-12-05T05:06:39.502 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/132/13' -> 'copyofsnap1/test1/132/13' 2019-12-05T05:06:39.502 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/132/14' -> 'copyofsnap1/test1/132/14' 2019-12-05T05:06:39.502 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/132/15' -> 'copyofsnap1/test1/132/15' 2019-12-05T05:06:39.502 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/132/16' -> 'copyofsnap1/test1/132/16' 2019-12-05T05:06:39.503 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/132/17' -> 'copyofsnap1/test1/132/17' 2019-12-05T05:06:39.503 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/132/18' -> 'copyofsnap1/test1/132/18' 2019-12-05T05:06:39.503 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/132/19' -> 'copyofsnap1/test1/132/19' 2019-12-05T05:06:39.503 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/132/20' -> 'copyofsnap1/test1/132/20' 2019-12-05T05:06:39.503 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/133' -> 'copyofsnap1/test1/133' 2019-12-05T05:06:39.503 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/133/10' -> 'copyofsnap1/test1/133/10' 2019-12-05T05:06:39.503 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/133/11' -> 'copyofsnap1/test1/133/11' 2019-12-05T05:06:39.504 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/133/12' -> 'copyofsnap1/test1/133/12' 2019-12-05T05:06:39.504 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/133/13' -> 'copyofsnap1/test1/133/13' 2019-12-05T05:06:39.504 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/133/14' -> 'copyofsnap1/test1/133/14' 2019-12-05T05:06:39.504 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/133/15' -> 'copyofsnap1/test1/133/15' 2019-12-05T05:06:39.504 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/133/16' -> 'copyofsnap1/test1/133/16' 2019-12-05T05:06:45.425 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/133/17' -> 'copyofsnap1/test1/133/17' 2019-12-05T05:06:45.425 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/133/18' -> 'copyofsnap1/test1/133/18' 2019-12-05T05:06:45.425 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/133/19' -> 'copyofsnap1/test1/133/19' 2019-12-05T05:06:45.425 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/133/20' -> 'copyofsnap1/test1/133/20' 2019-12-05T05:06:45.425 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/134' -> 'copyofsnap1/test1/134' 2019-12-05T05:06:45.426 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/134/10' -> 'copyofsnap1/test1/134/10' 2019-12-05T05:06:45.426 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/134/11' -> 'copyofsnap1/test1/134/11' 2019-12-05T05:06:45.426 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/134/12' -> 'copyofsnap1/test1/134/12' 2019-12-05T05:06:45.426 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/134/13' -> 'copyofsnap1/test1/134/13' 2019-12-05T05:06:45.426 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/134/14' -> 'copyofsnap1/test1/134/14' 2019-12-05T05:06:45.426 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/134/15' -> 'copyofsnap1/test1/134/15' 2019-12-05T05:06:45.426 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/134/16' -> 'copyofsnap1/test1/134/16' 2019-12-05T05:06:45.427 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/134/17' -> 'copyofsnap1/test1/134/17' 2019-12-05T05:06:45.427 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/134/18' -> 'copyofsnap1/test1/134/18' 2019-12-05T05:06:45.427 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/134/19' -> 'copyofsnap1/test1/134/19' 2019-12-05T05:06:45.427 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/134/20' -> 'copyofsnap1/test1/134/20' 2019-12-05T05:06:45.427 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/135' -> 'copyofsnap1/test1/135' 2019-12-05T05:06:45.427 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/135/10' -> 'copyofsnap1/test1/135/10' 2019-12-05T05:06:45.427 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/135/11' -> 'copyofsnap1/test1/135/11' 2019-12-05T05:06:45.428 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/135/12' -> 'copyofsnap1/test1/135/12' 2019-12-05T05:06:45.428 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/135/13' -> 'copyofsnap1/test1/135/13' 2019-12-05T05:06:45.428 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/135/14' -> 'copyofsnap1/test1/135/14' 2019-12-05T05:06:45.428 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/135/15' -> 'copyofsnap1/test1/135/15' 2019-12-05T05:06:45.428 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/135/16' -> 'copyofsnap1/test1/135/16' 2019-12-05T05:06:45.428 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/135/17' -> 'copyofsnap1/test1/135/17' 2019-12-05T05:06:45.429 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/135/18' -> 'copyofsnap1/test1/135/18' 2019-12-05T05:06:45.429 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/135/19' -> 'copyofsnap1/test1/135/19' 2019-12-05T05:06:45.429 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/135/20' -> 'copyofsnap1/test1/135/20' 2019-12-05T05:06:45.429 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/136' -> 'copyofsnap1/test1/136' 2019-12-05T05:06:45.429 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/136/10' -> 'copyofsnap1/test1/136/10' 2019-12-05T05:06:45.429 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/136/11' -> 'copyofsnap1/test1/136/11' 2019-12-05T05:06:45.429 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/136/12' -> 'copyofsnap1/test1/136/12' 2019-12-05T05:06:45.430 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/136/13' -> 'copyofsnap1/test1/136/13' 2019-12-05T05:06:45.430 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/136/14' -> 'copyofsnap1/test1/136/14' 2019-12-05T05:06:45.430 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/136/15' -> 'copyofsnap1/test1/136/15' 2019-12-05T05:06:45.430 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/136/16' -> 'copyofsnap1/test1/136/16' 2019-12-05T05:06:45.430 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/136/17' -> 'copyofsnap1/test1/136/17' 2019-12-05T05:06:45.430 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/136/18' -> 'copyofsnap1/test1/136/18' 2019-12-05T05:06:45.430 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/136/19' -> 'copyofsnap1/test1/136/19' 2019-12-05T05:06:45.430 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/136/20' -> 'copyofsnap1/test1/136/20' 2019-12-05T05:06:45.431 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/137' -> 'copyofsnap1/test1/137' 2019-12-05T05:06:45.431 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/137/10' -> 'copyofsnap1/test1/137/10' 2019-12-05T05:06:45.431 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/137/11' -> 'copyofsnap1/test1/137/11' 2019-12-05T05:06:45.431 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/137/12' -> 'copyofsnap1/test1/137/12' 2019-12-05T05:06:45.431 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/137/13' -> 'copyofsnap1/test1/137/13' 2019-12-05T05:06:45.431 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/137/14' -> 'copyofsnap1/test1/137/14' 2019-12-05T05:06:45.431 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/137/15' -> 'copyofsnap1/test1/137/15' 2019-12-05T05:06:45.432 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/137/16' -> 'copyofsnap1/test1/137/16' 2019-12-05T05:06:45.432 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/137/17' -> 'copyofsnap1/test1/137/17' 2019-12-05T05:06:45.432 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/137/18' -> 'copyofsnap1/test1/137/18' 2019-12-05T05:06:45.432 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/137/19' -> 'copyofsnap1/test1/137/19' 2019-12-05T05:06:45.432 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/137/20' -> 'copyofsnap1/test1/137/20' 2019-12-05T05:06:45.432 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/138' -> 'copyofsnap1/test1/138' 2019-12-05T05:06:45.433 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/138/10' -> 'copyofsnap1/test1/138/10' 2019-12-05T05:06:45.433 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/138/11' -> 'copyofsnap1/test1/138/11' 2019-12-05T05:06:45.433 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/138/12' -> 'copyofsnap1/test1/138/12' 2019-12-05T05:06:45.433 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/138/13' -> 'copyofsnap1/test1/138/13' 2019-12-05T05:06:45.433 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/138/14' -> 'copyofsnap1/test1/138/14' 2019-12-05T05:06:45.433 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/138/15' -> 'copyofsnap1/test1/138/15' 2019-12-05T05:06:45.434 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/138/16' -> 'copyofsnap1/test1/138/16' 2019-12-05T05:06:45.434 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/138/17' -> 'copyofsnap1/test1/138/17' 2019-12-05T05:06:45.434 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/138/18' -> 'copyofsnap1/test1/138/18' 2019-12-05T05:06:45.434 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/138/19' -> 'copyofsnap1/test1/138/19' 2019-12-05T05:06:45.435 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/138/20' -> 'copyofsnap1/test1/138/20' 2019-12-05T05:06:45.435 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/139' -> 'copyofsnap1/test1/139' 2019-12-05T05:06:45.435 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/139/10' -> 'copyofsnap1/test1/139/10' 2019-12-05T05:06:45.435 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/139/11' -> 'copyofsnap1/test1/139/11' 2019-12-05T05:06:45.435 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/139/12' -> 'copyofsnap1/test1/139/12' 2019-12-05T05:06:45.436 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/139/13' -> 'copyofsnap1/test1/139/13' 2019-12-05T05:06:45.436 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/139/14' -> 'copyofsnap1/test1/139/14' 2019-12-05T05:06:45.436 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/139/15' -> 'copyofsnap1/test1/139/15' 2019-12-05T05:06:45.436 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/139/16' -> 'copyofsnap1/test1/139/16' 2019-12-05T05:06:45.437 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/139/17' -> 'copyofsnap1/test1/139/17' 2019-12-05T05:06:45.438 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/139/18' -> 'copyofsnap1/test1/139/18' 2019-12-05T05:06:45.438 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/139/19' -> 'copyofsnap1/test1/139/19' 2019-12-05T05:06:45.438 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/139/20' -> 'copyofsnap1/test1/139/20' 2019-12-05T05:06:45.438 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/140' -> 'copyofsnap1/test1/140' 2019-12-05T05:06:45.438 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/140/10' -> 'copyofsnap1/test1/140/10' 2019-12-05T05:06:45.438 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/140/11' -> 'copyofsnap1/test1/140/11' 2019-12-05T05:06:45.440 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/140/12' -> 'copyofsnap1/test1/140/12' 2019-12-05T05:06:45.440 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/140/13' -> 'copyofsnap1/test1/140/13' 2019-12-05T05:06:51.692 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/140/14' -> 'copyofsnap1/test1/140/14' 2019-12-05T05:06:51.692 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/140/15' -> 'copyofsnap1/test1/140/15' 2019-12-05T05:06:51.693 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/140/16' -> 'copyofsnap1/test1/140/16' 2019-12-05T05:06:51.693 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/140/17' -> 'copyofsnap1/test1/140/17' 2019-12-05T05:06:51.693 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/140/18' -> 'copyofsnap1/test1/140/18' 2019-12-05T05:06:51.693 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/140/19' -> 'copyofsnap1/test1/140/19' 2019-12-05T05:06:51.693 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/140/20' -> 'copyofsnap1/test1/140/20' 2019-12-05T05:06:51.693 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/141' -> 'copyofsnap1/test1/141' 2019-12-05T05:06:51.693 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/141/10' -> 'copyofsnap1/test1/141/10' 2019-12-05T05:06:51.694 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/141/11' -> 'copyofsnap1/test1/141/11' 2019-12-05T05:06:51.694 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/141/12' -> 'copyofsnap1/test1/141/12' 2019-12-05T05:06:51.694 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/141/13' -> 'copyofsnap1/test1/141/13' 2019-12-05T05:06:51.694 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/141/14' -> 'copyofsnap1/test1/141/14' 2019-12-05T05:06:51.694 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/141/15' -> 'copyofsnap1/test1/141/15' 2019-12-05T05:06:51.694 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/141/16' -> 'copyofsnap1/test1/141/16' 2019-12-05T05:06:51.694 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/141/17' -> 'copyofsnap1/test1/141/17' 2019-12-05T05:06:51.695 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/141/18' -> 'copyofsnap1/test1/141/18' 2019-12-05T05:06:51.695 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/141/19' -> 'copyofsnap1/test1/141/19' 2019-12-05T05:06:51.695 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/141/20' -> 'copyofsnap1/test1/141/20' 2019-12-05T05:06:51.695 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/142' -> 'copyofsnap1/test1/142' 2019-12-05T05:06:51.695 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/142/10' -> 'copyofsnap1/test1/142/10' 2019-12-05T05:06:51.695 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/142/11' -> 'copyofsnap1/test1/142/11' 2019-12-05T05:06:51.695 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/142/12' -> 'copyofsnap1/test1/142/12' 2019-12-05T05:06:51.696 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/142/13' -> 'copyofsnap1/test1/142/13' 2019-12-05T05:06:51.696 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/142/14' -> 'copyofsnap1/test1/142/14' 2019-12-05T05:06:51.696 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/142/15' -> 'copyofsnap1/test1/142/15' 2019-12-05T05:06:51.696 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/142/16' -> 'copyofsnap1/test1/142/16' 2019-12-05T05:06:51.696 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/142/17' -> 'copyofsnap1/test1/142/17' 2019-12-05T05:06:51.696 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/142/18' -> 'copyofsnap1/test1/142/18' 2019-12-05T05:06:51.696 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/142/19' -> 'copyofsnap1/test1/142/19' 2019-12-05T05:06:51.697 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/142/20' -> 'copyofsnap1/test1/142/20' 2019-12-05T05:06:51.697 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/143' -> 'copyofsnap1/test1/143' 2019-12-05T05:06:51.697 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/143/10' -> 'copyofsnap1/test1/143/10' 2019-12-05T05:06:51.697 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/143/11' -> 'copyofsnap1/test1/143/11' 2019-12-05T05:06:51.697 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/143/12' -> 'copyofsnap1/test1/143/12' 2019-12-05T05:06:51.697 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/143/13' -> 'copyofsnap1/test1/143/13' 2019-12-05T05:06:51.698 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/143/14' -> 'copyofsnap1/test1/143/14' 2019-12-05T05:06:51.698 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/143/15' -> 'copyofsnap1/test1/143/15' 2019-12-05T05:06:51.698 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/143/16' -> 'copyofsnap1/test1/143/16' 2019-12-05T05:06:51.698 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/143/17' -> 'copyofsnap1/test1/143/17' 2019-12-05T05:06:51.698 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/143/18' -> 'copyofsnap1/test1/143/18' 2019-12-05T05:06:51.698 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/143/19' -> 'copyofsnap1/test1/143/19' 2019-12-05T05:06:51.698 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/143/20' -> 'copyofsnap1/test1/143/20' 2019-12-05T05:06:51.699 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/144' -> 'copyofsnap1/test1/144' 2019-12-05T05:06:51.699 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/144/10' -> 'copyofsnap1/test1/144/10' 2019-12-05T05:06:51.699 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/144/11' -> 'copyofsnap1/test1/144/11' 2019-12-05T05:06:51.699 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/144/12' -> 'copyofsnap1/test1/144/12' 2019-12-05T05:06:51.699 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/144/13' -> 'copyofsnap1/test1/144/13' 2019-12-05T05:06:51.699 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/144/14' -> 'copyofsnap1/test1/144/14' 2019-12-05T05:06:51.700 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/144/15' -> 'copyofsnap1/test1/144/15' 2019-12-05T05:06:51.701 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/144/16' -> 'copyofsnap1/test1/144/16' 2019-12-05T05:06:51.701 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/144/17' -> 'copyofsnap1/test1/144/17' 2019-12-05T05:06:51.701 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/144/18' -> 'copyofsnap1/test1/144/18' 2019-12-05T05:06:51.701 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/144/19' -> 'copyofsnap1/test1/144/19' 2019-12-05T05:06:51.701 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/144/20' -> 'copyofsnap1/test1/144/20' 2019-12-05T05:06:51.701 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/145' -> 'copyofsnap1/test1/145' 2019-12-05T05:06:51.701 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/145/10' -> 'copyofsnap1/test1/145/10' 2019-12-05T05:06:51.702 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/145/11' -> 'copyofsnap1/test1/145/11' 2019-12-05T05:06:51.702 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/145/12' -> 'copyofsnap1/test1/145/12' 2019-12-05T05:06:51.702 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/145/13' -> 'copyofsnap1/test1/145/13' 2019-12-05T05:06:51.702 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/145/14' -> 'copyofsnap1/test1/145/14' 2019-12-05T05:06:51.702 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/145/15' -> 'copyofsnap1/test1/145/15' 2019-12-05T05:06:51.702 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/145/16' -> 'copyofsnap1/test1/145/16' 2019-12-05T05:06:51.702 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/145/17' -> 'copyofsnap1/test1/145/17' 2019-12-05T05:06:51.702 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/145/18' -> 'copyofsnap1/test1/145/18' 2019-12-05T05:06:51.703 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/145/19' -> 'copyofsnap1/test1/145/19' 2019-12-05T05:06:51.703 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/145/20' -> 'copyofsnap1/test1/145/20' 2019-12-05T05:06:51.703 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/146' -> 'copyofsnap1/test1/146' 2019-12-05T05:06:51.703 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/146/10' -> 'copyofsnap1/test1/146/10' 2019-12-05T05:06:51.703 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/146/11' -> 'copyofsnap1/test1/146/11' 2019-12-05T05:06:51.703 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/146/12' -> 'copyofsnap1/test1/146/12' 2019-12-05T05:06:51.704 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/146/13' -> 'copyofsnap1/test1/146/13' 2019-12-05T05:06:51.704 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/146/14' -> 'copyofsnap1/test1/146/14' 2019-12-05T05:06:51.704 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/146/15' -> 'copyofsnap1/test1/146/15' 2019-12-05T05:06:51.704 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/146/16' -> 'copyofsnap1/test1/146/16' 2019-12-05T05:06:51.704 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/146/17' -> 'copyofsnap1/test1/146/17' 2019-12-05T05:06:51.704 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/146/18' -> 'copyofsnap1/test1/146/18' 2019-12-05T05:06:51.704 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/146/19' -> 'copyofsnap1/test1/146/19' 2019-12-05T05:06:51.705 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/146/20' -> 'copyofsnap1/test1/146/20' 2019-12-05T05:06:51.705 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/147' -> 'copyofsnap1/test1/147' 2019-12-05T05:06:51.705 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/147/10' -> 'copyofsnap1/test1/147/10' 2019-12-05T05:06:51.705 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/147/11' -> 'copyofsnap1/test1/147/11' 2019-12-05T05:06:54.342 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:06:54.432 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.424+0000 7fe8f692e700 1 Processor -- start 2019-12-05T05:06:54.432 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.424+0000 7fe8f692e700 1 -- start start 2019-12-05T05:06:54.433 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.424+0000 7fe8f692e700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe8f00a7740 0x7fe8f008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:06:54.433 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.424+0000 7fe8f692e700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe8f009fae0 0x7fe8f007eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:06:54.433 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.424+0000 7fe8f692e700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8f00a0710 con 0x7fe8f007e720 2019-12-05T05:06:54.433 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.424+0000 7fe8f692e700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8f00a0b00 con 0x7fe8f009fae0 2019-12-05T05:06:54.433 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.424+0000 7fe8f692e700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8f008e5b0 con 0x7fe8f00a7740 2019-12-05T05:06:54.433 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.424+0000 7fe8f4ecb700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe8f00a7740 0x7fe8f008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:06:54.434 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.424+0000 7fe8effff700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fe8f007e720 0x7fe8f009f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:36602/0 (socket says 172.21.15.42:36602) 2019-12-05T05:06:54.434 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.424+0000 7fe8effff700 1 -- 172.21.15.42:0/326073734 learned_addr learned my addr 172.21.15.42:0/326073734 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:06:54.434 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.424+0000 7fe8f56cc700 1 --2- 172.21.15.42:0/326073734 >> v2:172.21.15.40:3300/0 conn(0x7fe8f009fae0 0x7fe8f007eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:06:54.434 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.424+0000 7fe8f4ecb700 1 -- 172.21.15.42:0/326073734 >> v2:172.21.15.40:3300/0 conn(0x7fe8f009fae0 msgr2=0x7fe8f007eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:06:54.434 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.424+0000 7fe8f4ecb700 1 --2- 172.21.15.42:0/326073734 >> v2:172.21.15.40:3300/0 conn(0x7fe8f009fae0 0x7fe8f007eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:06:54.434 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.424+0000 7fe8f4ecb700 1 -- 172.21.15.42:0/326073734 >> v1:172.21.15.40:6789/0 conn(0x7fe8f007e720 legacy=0x7fe8f009f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:06:54.434 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.424+0000 7fe8f4ecb700 1 -- 172.21.15.42:0/326073734 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8f007f0a0 con 0x7fe8f00a7740 2019-12-05T05:06:54.434 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.424+0000 7fe8ef7fe700 1 -- 172.21.15.42:0/326073734 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2367662371 0 0) 0x7fe8f00a0710 con 0x7fe8f007e720 2019-12-05T05:06:54.435 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.424+0000 7fe8f56cc700 1 --2- 172.21.15.42:0/326073734 >> v2:172.21.15.40:3300/0 conn(0x7fe8f009fae0 0x7fe8f007eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:06:54.435 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.424+0000 7fe8f4ecb700 1 --2- 172.21.15.42:0/326073734 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe8f00a7740 0x7fe8f008df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fe8e4008790 tx=0x7fe8e400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:06:54.435 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.428+0000 7fe8f4ecb700 1 --2- 172.21.15.42:0/326073734 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe8f00a7740 0x7fe8f008df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fe8e4008790 tx=0x7fe8e400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:06:54.435 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.428+0000 7fe8f4ecb700 1 --2- 172.21.15.42:0/326073734 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe8f00a7740 0x7fe8f008df80 secure :-1 s=READY pgs=58 cs=0 l=1 rx=0x7fe8e4008790 tx=0x7fe8e400fc70).ready entity=mon.2 client_cookie=4f9f9e39321c3fbf server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:06:54.435 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.428+0000 7fe8f4ecb700 1 --2- 172.21.15.42:0/326073734 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe8f00a7740 0x7fe8f008df80 secure :-1 s=THROTTLE_DONE pgs=58 cs=0 l=1 rx=0x7fe8e4008790 tx=0x7fe8e400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:06:54.435 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.428+0000 7fe8ef7fe700 1 -- 172.21.15.42:0/326073734 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe8e4008bb0 con 0x7fe8f00a7740 2019-12-05T05:06:54.435 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.428+0000 7fe8f4ecb700 1 --2- 172.21.15.42:0/326073734 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe8f00a7740 0x7fe8f008df80 secure :-1 s=THROTTLE_DONE pgs=58 cs=0 l=1 rx=0x7fe8e4008790 tx=0x7fe8e400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:06:54.436 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.428+0000 7fe8ef7fe700 1 -- 172.21.15.42:0/326073734 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8e40087d0 con 0x7fe8f00a7740 2019-12-05T05:06:54.436 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.428+0000 7fe8f4ecb700 1 --2- 172.21.15.42:0/326073734 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe8f00a7740 0x7fe8f008df80 secure :-1 s=THROTTLE_DONE pgs=58 cs=0 l=1 rx=0x7fe8e4008790 tx=0x7fe8e400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:06:54.436 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.428+0000 7fe8ef7fe700 1 -- 172.21.15.42:0/326073734 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe8e4011950 con 0x7fe8f00a7740 2019-12-05T05:06:54.436 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.428+0000 7fe8f692e700 1 -- 172.21.15.42:0/326073734 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe8f00a7740 msgr2=0x7fe8f008df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:06:54.436 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.428+0000 7fe8f692e700 1 --2- 172.21.15.42:0/326073734 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe8f00a7740 0x7fe8f008df80 secure :-1 s=READY pgs=58 cs=0 l=1 rx=0x7fe8e4008790 tx=0x7fe8e400fc70).stop 2019-12-05T05:06:54.436 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.428+0000 7fe8f692e700 1 -- 172.21.15.42:0/326073734 shutdown_connections 2019-12-05T05:06:54.437 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.428+0000 7fe8f692e700 1 --2- 172.21.15.42:0/326073734 >> v2:172.21.15.40:3300/0 conn(0x7fe8f009fae0 0x7fe8f007eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:06:54.437 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.428+0000 7fe8f692e700 1 --2- 172.21.15.42:0/326073734 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe8f00a7740 0x7fe8f008df80 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:06:54.437 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.428+0000 7fe8f692e700 1 -- 172.21.15.42:0/326073734 shutdown_connections 2019-12-05T05:06:54.437 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.428+0000 7fe8f692e700 1 -- 172.21.15.42:0/326073734 wait complete. 2019-12-05T05:06:54.437 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.428+0000 7fe8f692e700 1 -- 172.21.15.42:0/326073734 >> 172.21.15.42:0/326073734 conn(0x7fe8f00aabe0 msgr2=0x7fe8f00a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:06:54.437 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.428+0000 7fe8f692e700 1 Processor -- start 2019-12-05T05:06:54.437 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.428+0000 7fe8f692e700 1 -- start start 2019-12-05T05:06:54.443 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.428+0000 7fe8f692e700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe8f0114bc0 0x7fe8f0144a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:06:54.443 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.428+0000 7fe8f692e700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe8f0125390 0x7fe8f0122240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:06:54.443 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.428+0000 7fe8f692e700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8f0144f80 con 0x7fe8f0116bc0 2019-12-05T05:06:54.443 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.428+0000 7fe8f692e700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8f01453d0 con 0x7fe8f0114bc0 2019-12-05T05:06:54.444 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.428+0000 7fe8f692e700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8f0125910 con 0x7fe8f0125390 2019-12-05T05:06:54.444 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.428+0000 7fe8f4ecb700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe8f0114bc0 0x7fe8f0144a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:06:54.444 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.428+0000 7fe8f4ecb700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe8f0114bc0 0x7fe8f0144a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.42:43176/0 (socket says 172.21.15.42:43176) 2019-12-05T05:06:54.444 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.428+0000 7fe8f56cc700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe8f0125390 0x7fe8f0122240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:06:54.444 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.428+0000 7fe8f4ecb700 1 -- 172.21.15.42:0/4129789894 learned_addr learned my addr 172.21.15.42:0/4129789894 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:06:54.444 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.428+0000 7fe8f4ecb700 1 -- 172.21.15.42:0/4129789894 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe8f0125390 msgr2=0x7fe8f0122240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:06:54.444 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.428+0000 7fe8f4ecb700 1 --2- 172.21.15.42:0/4129789894 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe8f0125390 0x7fe8f0122240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:06:54.445 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.428+0000 7fe8f4ecb700 1 -- 172.21.15.42:0/4129789894 >> v1:172.21.15.40:6789/0 conn(0x7fe8f0116bc0 legacy=0x7fe8f0121a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:06:54.445 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.428+0000 7fe8f4ecb700 1 -- 172.21.15.42:0/4129789894 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8f012d2e0 con 0x7fe8f0114bc0 2019-12-05T05:06:54.445 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.428+0000 7fe8f4ecb700 1 --2- 172.21.15.42:0/4129789894 >> v2:172.21.15.40:3300/0 conn(0x7fe8f0114bc0 0x7fe8f0144a30 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fe8f0145590 tx=0x7fe8e4011f60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:06:54.445 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.428+0000 7fe8f4ecb700 1 --2- 172.21.15.42:0/4129789894 >> v2:172.21.15.40:3300/0 conn(0x7fe8f0114bc0 0x7fe8f0144a30 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fe8f0145590 tx=0x7fe8e4011f60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:06:54.445 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.432+0000 7fe8f4ecb700 1 --2- 172.21.15.42:0/4129789894 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe8f0114bc0 0x7fe8f0144a30 secure :-1 s=READY pgs=231 cs=0 l=1 rx=0x7fe8f0145590 tx=0x7fe8e4011f60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:06:54.445 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.432+0000 7fe8f4ecb700 1 --2- 172.21.15.42:0/4129789894 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe8f0114bc0 0x7fe8f0144a30 secure :-1 s=THROTTLE_DONE pgs=231 cs=0 l=1 rx=0x7fe8f0145590 tx=0x7fe8e4011f60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:06:54.446 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.432+0000 7fe8f4ecb700 1 --2- 172.21.15.42:0/4129789894 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe8f0114bc0 0x7fe8f0144a30 secure :-1 s=THROTTLE_DONE pgs=231 cs=0 l=1 rx=0x7fe8f0145590 tx=0x7fe8e4011f60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:06:54.446 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.432+0000 7fe8ed7fa700 1 -- 172.21.15.42:0/4129789894 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe8e400f470 con 0x7fe8f0114bc0 2019-12-05T05:06:54.446 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.432+0000 7fe8f4ecb700 1 --2- 172.21.15.42:0/4129789894 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe8f0114bc0 0x7fe8f0144a30 secure :-1 s=THROTTLE_DONE pgs=231 cs=0 l=1 rx=0x7fe8f0145590 tx=0x7fe8e4011f60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:06:54.446 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.432+0000 7fe8ed7fa700 1 -- 172.21.15.42:0/4129789894 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8e4008db0 con 0x7fe8f0114bc0 2019-12-05T05:06:54.446 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.432+0000 7fe8f692e700 1 -- 172.21.15.42:0/4129789894 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8f012c2d0 con 0x7fe8f0114bc0 2019-12-05T05:06:54.446 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.432+0000 7fe8ed7fa700 1 -- 172.21.15.42:0/4129789894 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe8e4005730 con 0x7fe8f0114bc0 2019-12-05T05:06:54.446 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.432+0000 7fe8f692e700 1 -- 172.21.15.42:0/4129789894 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8f012c7d0 con 0x7fe8f0114bc0 2019-12-05T05:06:54.447 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.432+0000 7fe8f4ecb700 1 --2- 172.21.15.42:0/4129789894 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe8f0114bc0 0x7fe8f0144a30 secure :-1 s=THROTTLE_DONE pgs=231 cs=0 l=1 rx=0x7fe8f0145590 tx=0x7fe8e4011f60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:06:54.447 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.432+0000 7fe8ed7fa700 1 -- 172.21.15.42:0/4129789894 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fe8e4005a40 con 0x7fe8f0114bc0 2019-12-05T05:06:54.447 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.432+0000 7fe8f4ecb700 1 --2- 172.21.15.42:0/4129789894 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe8f0114bc0 0x7fe8f0144a30 secure :-1 s=THROTTLE_DONE pgs=231 cs=0 l=1 rx=0x7fe8f0145590 tx=0x7fe8e4011f60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:06:54.447 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.432+0000 7fe8f692e700 1 -- 172.21.15.42:0/4129789894 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8f005d110 con 0x7fe8f0114bc0 2019-12-05T05:06:54.447 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.432+0000 7fe8ed7fa700 1 --2- 172.21.15.42:0/4129789894 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe8d003ae30 0x7fe8d003d340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:06:54.448 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.432+0000 7fe8ed7fa700 1 -- 172.21.15.42:0/4129789894 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6145+0+0 (secure 0 0 0) 0x7fe8e404b570 con 0x7fe8f0114bc0 2019-12-05T05:06:54.448 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.432+0000 7fe8effff700 1 --2- 172.21.15.42:0/4129789894 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe8d003ae30 0x7fe8d003d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:06:54.448 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.432+0000 7fe8effff700 1 --2- 172.21.15.42:0/4129789894 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe8d003ae30 0x7fe8d003d340 crc :-1 s=READY pgs=135 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:06:54.449 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.436+0000 7fe8f4ecb700 1 --2- 172.21.15.42:0/4129789894 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe8f0114bc0 0x7fe8f0144a30 secure :-1 s=THROTTLE_DONE pgs=231 cs=0 l=1 rx=0x7fe8f0145590 tx=0x7fe8e4011f60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:06:54.449 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.436+0000 7fe8ed7fa700 1 -- 172.21.15.42:0/4129789894 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fe8e4005d50 con 0x7fe8f0114bc0 2019-12-05T05:06:54.634 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.628+0000 7fe8f692e700 1 -- 172.21.15.42:0/4129789894 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe8f005b8c0 con 0x7fe8f0114bc0 2019-12-05T05:06:54.634 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.628+0000 7fe8f4ecb700 1 --2- 172.21.15.42:0/4129789894 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe8f0114bc0 0x7fe8f0144a30 secure :-1 s=THROTTLE_DONE pgs=231 cs=0 l=1 rx=0x7fe8f0145590 tx=0x7fe8e4011f60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:06:54.634 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.628+0000 7fe8ed7fa700 1 -- 172.21.15.42:0/4129789894 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+2101 (secure 0 0 0) 0x7fe8e4033420 con 0x7fe8f0114bc0 2019-12-05T05:06:54.634 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 23 2019-12-05T05:06:54.635 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:06:54.635 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4831,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/90248709","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":90248709},{"type":"v1","addr":"172.21.15.40:6833","nonce":90248709}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":11}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:05:19.282511+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4354},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4354":{"gid":4354,"name":"a","rank":0,"incarnation":7,"state":"up:active","state_seq":20,"addr":"172.21.15.42:6835/2019200199","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2019200199},{"type":"v1","addr":"172.21.15.42:6835","nonce":2019200199}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:06:54.637 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.632+0000 7fe8f692e700 1 -- 172.21.15.42:0/4129789894 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe8d003ae30 msgr2=0x7fe8d003d340 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:06:54.637 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.632+0000 7fe8f692e700 1 --2- 172.21.15.42:0/4129789894 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe8d003ae30 0x7fe8d003d340 crc :-1 s=READY pgs=135 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:06:54.637 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.632+0000 7fe8f692e700 1 -- 172.21.15.42:0/4129789894 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe8f0114bc0 msgr2=0x7fe8f0144a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:06:54.637 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.632+0000 7fe8f692e700 1 --2- 172.21.15.42:0/4129789894 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe8f0114bc0 0x7fe8f0144a30 secure :-1 s=READY pgs=231 cs=0 l=1 rx=0x7fe8f0145590 tx=0x7fe8e4011f60).stop 2019-12-05T05:06:54.638 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.632+0000 7fe8f692e700 1 -- 172.21.15.42:0/4129789894 shutdown_connections 2019-12-05T05:06:54.638 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.632+0000 7fe8f692e700 1 --2- 172.21.15.42:0/4129789894 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe8d003ae30 0x7fe8d003d340 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:06:54.638 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.632+0000 7fe8f692e700 1 --2- 172.21.15.42:0/4129789894 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe8f0125390 0x7fe8f0122240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:06:54.638 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.632+0000 7fe8f692e700 1 --2- 172.21.15.42:0/4129789894 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe8f0114bc0 0x7fe8f0144a30 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:06:54.639 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.632+0000 7fe8f692e700 1 -- 172.21.15.42:0/4129789894 shutdown_connections 2019-12-05T05:06:54.639 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.632+0000 7fe8f692e700 1 -- 172.21.15.42:0/4129789894 wait complete. 2019-12-05T05:06:54.639 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.632+0000 7fe8f692e700 1 -- 172.21.15.42:0/4129789894 >> 172.21.15.42:0/4129789894 conn(0x7fe8f00aabe0 msgr2=0x7fe8f00a9790 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:06:54.646 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.a (rank=0) 2019-12-05T05:06:54.647 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2019-12-05T05:06:54.647 INFO:teuthology.orchestra.run:waiting for 300 2019-12-05T05:06:54.656 INFO:tasks.ceph.mds.a:Stopped 2019-12-05T05:06:54.657 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:06:54.823 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.812+0000 7f6b809d0700 1 Processor -- start 2019-12-05T05:06:54.823 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.812+0000 7f6b809d0700 1 -- start start 2019-12-05T05:06:54.823 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.816+0000 7f6b809d0700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f6b7c0a7740 0x7f6b7c08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:06:54.824 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.816+0000 7f6b809d0700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6b7c09fae0 0x7f6b7c07eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:06:54.824 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.816+0000 7f6b809d0700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b7c0a0710 con 0x7f6b7c07e720 2019-12-05T05:06:54.824 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.816+0000 7f6b809d0700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b7c0a0b00 con 0x7f6b7c0a7740 2019-12-05T05:06:54.824 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.816+0000 7f6b809d0700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b7c08e5b0 con 0x7f6b7c09fae0 2019-12-05T05:06:54.825 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.816+0000 7f6b7a59c700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f6b7c07e720 0x7f6b7c09f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:55410/0 (socket says 172.21.15.42:55410) 2019-12-05T05:06:54.825 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.816+0000 7f6b7a59c700 1 -- 172.21.15.42:0/2344066297 learned_addr learned my addr 172.21.15.42:0/2344066297 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:06:54.825 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.816+0000 7f6b7ad9d700 1 --2- 172.21.15.42:0/2344066297 >> v2:172.21.15.40:3300/0 conn(0x7f6b7c0a7740 0x7f6b7c08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:06:54.825 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.816+0000 7f6b7b59e700 1 --2- 172.21.15.42:0/2344066297 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6b7c09fae0 0x7f6b7c07eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:06:54.826 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.816+0000 7f6b79d9b700 1 -- 172.21.15.42:0/2344066297 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1918004580 0 0) 0x7f6b7c0a0710 con 0x7f6b7c07e720 2019-12-05T05:06:54.826 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.816+0000 7f6b79d9b700 1 -- 172.21.15.42:0/2344066297 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b640030b0 con 0x7f6b7c07e720 2019-12-05T05:06:54.826 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.816+0000 7f6b7ad9d700 1 -- 172.21.15.42:0/2344066297 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6b7c09fae0 msgr2=0x7f6b7c07eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:06:54.826 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.816+0000 7f6b7ad9d700 1 --2- 172.21.15.42:0/2344066297 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6b7c09fae0 0x7f6b7c07eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:06:54.826 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.816+0000 7f6b7ad9d700 1 -- 172.21.15.42:0/2344066297 >> v1:172.21.15.42:6789/0 conn(0x7f6b7c07e720 legacy=0x7f6b7c09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:06:54.827 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.816+0000 7f6b7ad9d700 1 -- 172.21.15.42:0/2344066297 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b7c07f0a0 con 0x7f6b7c0a7740 2019-12-05T05:06:54.827 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.816+0000 7f6b7b59e700 1 --2- 172.21.15.42:0/2344066297 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6b7c09fae0 0x7f6b7c07eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:06:54.827 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.816+0000 7f6b7ad9d700 1 --2- 172.21.15.42:0/2344066297 >> v2:172.21.15.40:3300/0 conn(0x7f6b7c0a7740 0x7f6b7c08df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f6b6c008790 tx=0x7f6b6c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:06:54.827 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.816+0000 7f6b7ad9d700 1 --2- 172.21.15.42:0/2344066297 >> v2:172.21.15.40:3300/0 conn(0x7f6b7c0a7740 0x7f6b7c08df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f6b6c008790 tx=0x7f6b6c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:06:54.828 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.816+0000 7f6b7ad9d700 1 --2- 172.21.15.42:0/2344066297 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6b7c0a7740 0x7f6b7c08df80 secure :-1 s=READY pgs=232 cs=0 l=1 rx=0x7f6b6c008790 tx=0x7f6b6c00fc70).ready entity=mon.0 client_cookie=15e02a0f08522d6d server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:06:54.828 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.816+0000 7f6b7ad9d700 1 --2- 172.21.15.42:0/2344066297 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6b7c0a7740 0x7f6b7c08df80 secure :-1 s=THROTTLE_DONE pgs=232 cs=0 l=1 rx=0x7f6b6c008790 tx=0x7f6b6c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:06:54.828 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.816+0000 7f6b79d9b700 1 -- 172.21.15.42:0/2344066297 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6b6c008bb0 con 0x7f6b7c0a7740 2019-12-05T05:06:54.829 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.816+0000 7f6b7ad9d700 1 --2- 172.21.15.42:0/2344066297 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6b7c0a7740 0x7f6b7c08df80 secure :-1 s=THROTTLE_DONE pgs=232 cs=0 l=1 rx=0x7f6b6c008790 tx=0x7f6b6c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:06:54.829 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.816+0000 7f6b79d9b700 1 -- 172.21.15.42:0/2344066297 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b6c0087d0 con 0x7f6b7c0a7740 2019-12-05T05:06:54.829 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.816+0000 7f6b7ad9d700 1 --2- 172.21.15.42:0/2344066297 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6b7c0a7740 0x7f6b7c08df80 secure :-1 s=THROTTLE_DONE pgs=232 cs=0 l=1 rx=0x7f6b6c008790 tx=0x7f6b6c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:06:54.829 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.816+0000 7f6b79d9b700 1 -- 172.21.15.42:0/2344066297 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6b6c011860 con 0x7f6b7c0a7740 2019-12-05T05:06:54.830 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.816+0000 7f6b809d0700 1 -- 172.21.15.42:0/2344066297 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6b7c0a7740 msgr2=0x7f6b7c08df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:06:54.830 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.816+0000 7f6b809d0700 1 --2- 172.21.15.42:0/2344066297 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6b7c0a7740 0x7f6b7c08df80 secure :-1 s=READY pgs=232 cs=0 l=1 rx=0x7f6b6c008790 tx=0x7f6b6c00fc70).stop 2019-12-05T05:06:54.830 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.816+0000 7f6b809d0700 1 -- 172.21.15.42:0/2344066297 shutdown_connections 2019-12-05T05:06:54.830 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.816+0000 7f6b809d0700 1 --2- 172.21.15.42:0/2344066297 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6b7c09fae0 0x7f6b7c07eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:06:54.831 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.816+0000 7f6b809d0700 1 --2- 172.21.15.42:0/2344066297 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6b7c0a7740 0x7f6b7c08df80 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:06:54.831 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.816+0000 7f6b809d0700 1 -- 172.21.15.42:0/2344066297 shutdown_connections 2019-12-05T05:06:54.831 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.816+0000 7f6b809d0700 1 -- 172.21.15.42:0/2344066297 wait complete. 2019-12-05T05:06:54.831 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.816+0000 7f6b809d0700 1 -- 172.21.15.42:0/2344066297 >> 172.21.15.42:0/2344066297 conn(0x7f6b7c0aabe0 msgr2=0x7f6b7c0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:06:54.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.820+0000 7f6b809d0700 1 Processor -- start 2019-12-05T05:06:54.837 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.820+0000 7f6b809d0700 1 -- start start 2019-12-05T05:06:54.837 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.820+0000 7f6b809d0700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f6b7c11c070 0x7f6b7c130430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:06:54.837 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.820+0000 7f6b809d0700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b7c1228f0 con 0x7f6b7c116500 2019-12-05T05:06:54.837 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.820+0000 7f6b809d0700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b7c121330 con 0x7f6b7c130000 2019-12-05T05:06:54.838 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.820+0000 7f6b809d0700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b7c122110 con 0x7f6b7c11c070 2019-12-05T05:06:54.838 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.820+0000 7f6b7a59c700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f6b7c130000 0x7f6b7c11b870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:55414/0 (socket says 172.21.15.42:55414) 2019-12-05T05:06:54.838 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.820+0000 7f6b7a59c700 1 -- 172.21.15.42:0/2271117537 learned_addr learned my addr 172.21.15.42:0/2271117537 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:06:54.838 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.820+0000 7f6b7b59e700 1 --2- 172.21.15.42:0/2271117537 >> v2:172.21.15.40:3300/0 conn(0x7f6b7c11c070 0x7f6b7c130430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:06:54.839 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.820+0000 7f6b6b7fe700 1 -- 172.21.15.42:0/2271117537 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2718154209 0 0) 0x7f6b7c121330 con 0x7f6b7c130000 2019-12-05T05:06:54.839 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.820+0000 7f6b6b7fe700 1 -- 172.21.15.42:0/2271117537 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b600030b0 con 0x7f6b7c130000 2019-12-05T05:06:54.839 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.820+0000 7f6b7b59e700 1 -- 172.21.15.42:0/2271117537 >> v1:172.21.15.42:6789/0 conn(0x7f6b7c130000 legacy=0x7f6b7c11b870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:06:54.839 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.820+0000 7f6b6b7fe700 1 -- 172.21.15.42:0/2271117537 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 276510006 0 0) 0x7f6b600030b0 con 0x7f6b7c130000 2019-12-05T05:06:54.840 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.824+0000 7f6b7b59e700 1 -- 172.21.15.42:0/2271117537 >> v1:172.21.15.40:6789/0 conn(0x7f6b7c116500 legacy=0x7f6b7c120b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:06:54.840 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.824+0000 7f6b7b59e700 1 -- 172.21.15.42:0/2271117537 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b7c130980 con 0x7f6b7c11c070 2019-12-05T05:06:54.840 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.824+0000 7f6b7b59e700 1 --2- 172.21.15.42:0/2271117537 >> v2:172.21.15.40:3300/0 conn(0x7f6b7c11c070 0x7f6b7c130430 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f6b70001b90 tx=0x7f6b70001bd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:06:54.840 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.824+0000 7f6b7b59e700 1 --2- 172.21.15.42:0/2271117537 >> v2:172.21.15.40:3300/0 conn(0x7f6b7c11c070 0x7f6b7c130430 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f6b70001b90 tx=0x7f6b70001bd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:06:54.841 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.824+0000 7f6b7b59e700 1 --2- 172.21.15.42:0/2271117537 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6b7c11c070 0x7f6b7c130430 secure :-1 s=READY pgs=234 cs=0 l=1 rx=0x7f6b70001b90 tx=0x7f6b70001bd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:06:54.841 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.824+0000 7f6b7b59e700 1 --2- 172.21.15.42:0/2271117537 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6b7c11c070 0x7f6b7c130430 secure :-1 s=THROTTLE_DONE pgs=234 cs=0 l=1 rx=0x7f6b70001b90 tx=0x7f6b70001bd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:06:54.841 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.824+0000 7f6b7b59e700 1 --2- 172.21.15.42:0/2271117537 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6b7c11c070 0x7f6b7c130430 secure :-1 s=THROTTLE_DONE pgs=234 cs=0 l=1 rx=0x7f6b70001b90 tx=0x7f6b70001bd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:06:54.841 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.824+0000 7f6b6b7fe700 1 -- 172.21.15.42:0/2271117537 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6b70001990 con 0x7f6b7c11c070 2019-12-05T05:06:54.842 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.824+0000 7f6b7b59e700 1 --2- 172.21.15.42:0/2271117537 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6b7c11c070 0x7f6b7c130430 secure :-1 s=THROTTLE_DONE pgs=234 cs=0 l=1 rx=0x7f6b70001b90 tx=0x7f6b70001bd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:06:54.842 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.824+0000 7f6b6b7fe700 1 -- 172.21.15.42:0/2271117537 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b700017a0 con 0x7f6b7c11c070 2019-12-05T05:06:54.842 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.824+0000 7f6b6b7fe700 1 -- 172.21.15.42:0/2271117537 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6b740127d0 con 0x7f6b7c11c070 2019-12-05T05:06:54.842 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.824+0000 7f6b809d0700 1 -- 172.21.15.42:0/2271117537 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b7c07b340 con 0x7f6b7c11c070 2019-12-05T05:06:54.843 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.824+0000 7f6b809d0700 1 -- 172.21.15.42:0/2271117537 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b7c07b890 con 0x7f6b7c11c070 2019-12-05T05:06:54.843 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.824+0000 7f6b7b59e700 1 --2- 172.21.15.42:0/2271117537 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6b7c11c070 0x7f6b7c130430 secure :-1 s=THROTTLE_DONE pgs=234 cs=0 l=1 rx=0x7f6b70001b90 tx=0x7f6b70001bd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:06:54.843 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.824+0000 7f6b6b7fe700 1 -- 172.21.15.42:0/2271117537 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f6b74012ae0 con 0x7f6b7c11c070 2019-12-05T05:06:54.843 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.824+0000 7f6b7b59e700 1 --2- 172.21.15.42:0/2271117537 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6b7c11c070 0x7f6b7c130430 secure :-1 s=THROTTLE_DONE pgs=234 cs=0 l=1 rx=0x7f6b70001b90 tx=0x7f6b70001bd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:06:54.843 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.824+0000 7f6b6b7fe700 1 --2- 172.21.15.42:0/2271117537 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6b6003b260 0x7f6b6003d770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:06:54.844 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.824+0000 7f6b6b7fe700 1 -- 172.21.15.42:0/2271117537 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6145+0+0 (secure 0 0 0) 0x7f6b7404afd0 con 0x7f6b7c11c070 2019-12-05T05:06:54.844 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.824+0000 7f6b7ad9d700 1 --2- 172.21.15.42:0/2271117537 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6b6003b260 0x7f6b6003d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:06:54.844 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.824+0000 7f6b7ad9d700 1 --2- 172.21.15.42:0/2271117537 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6b6003b260 0x7f6b6003d770 crc :-1 s=READY pgs=136 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:06:54.844 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.828+0000 7f6b809d0700 1 -- 172.21.15.42:0/2271117537 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b7c05d110 con 0x7f6b7c11c070 2019-12-05T05:06:54.845 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.836+0000 7f6b7b59e700 1 --2- 172.21.15.42:0/2271117537 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6b7c11c070 0x7f6b7c130430 secure :-1 s=THROTTLE_DONE pgs=234 cs=0 l=1 rx=0x7f6b70001b90 tx=0x7f6b70001bd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:06:54.846 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:54.836+0000 7f6b6b7fe700 1 -- 172.21.15.42:0/2271117537 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f6b740322b0 con 0x7f6b7c11c070 2019-12-05T05:06:55.009 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:55.004+0000 7f6b809d0700 1 -- 172.21.15.42:0/2271117537 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6b7c05b8c0 con 0x7f6b7c11c070 2019-12-05T05:06:55.010 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:55.008+0000 7f6b7b59e700 1 --2- 172.21.15.42:0/2271117537 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6b7c11c070 0x7f6b7c130430 secure :-1 s=THROTTLE_DONE pgs=234 cs=0 l=1 rx=0x7f6b70001b90 tx=0x7f6b70001bd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:06:55.010 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:55.008+0000 7f6b6b7fe700 1 -- 172.21.15.42:0/2271117537 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+2101 (secure 0 0 0) 0x7f6b74007140 con 0x7f6b7c11c070 2019-12-05T05:06:55.012 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 23 2019-12-05T05:06:55.013 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:06:55.013 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4831,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/90248709","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":90248709},{"type":"v1","addr":"172.21.15.40:6833","nonce":90248709}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":11}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:05:19.282511+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4354},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4354":{"gid":4354,"name":"a","rank":0,"incarnation":7,"state":"up:active","state_seq":20,"addr":"172.21.15.42:6835/2019200199","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2019200199},{"type":"v1","addr":"172.21.15.42:6835","nonce":2019200199}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:06:55.016 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:55.012+0000 7f6b809d0700 1 -- 172.21.15.42:0/2271117537 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6b6003b260 msgr2=0x7f6b6003d770 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:06:55.016 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:55.012+0000 7f6b809d0700 1 --2- 172.21.15.42:0/2271117537 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6b6003b260 0x7f6b6003d770 crc :-1 s=READY pgs=136 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:06:55.016 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:55.012+0000 7f6b809d0700 1 -- 172.21.15.42:0/2271117537 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6b7c11c070 msgr2=0x7f6b7c130430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:06:55.016 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:55.012+0000 7f6b809d0700 1 --2- 172.21.15.42:0/2271117537 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6b7c11c070 0x7f6b7c130430 secure :-1 s=READY pgs=234 cs=0 l=1 rx=0x7f6b70001b90 tx=0x7f6b70001bd0).stop 2019-12-05T05:06:55.017 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:55.012+0000 7f6b809d0700 1 -- 172.21.15.42:0/2271117537 shutdown_connections 2019-12-05T05:06:55.017 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:55.012+0000 7f6b809d0700 1 --2- 172.21.15.42:0/2271117537 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6b6003b260 0x7f6b6003d770 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:06:55.017 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:55.012+0000 7f6b809d0700 1 --2- 172.21.15.42:0/2271117537 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6b7c11c070 0x7f6b7c130430 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:06:55.017 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:55.012+0000 7f6b809d0700 1 -- 172.21.15.42:0/2271117537 shutdown_connections 2019-12-05T05:06:55.018 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:55.012+0000 7f6b809d0700 1 -- 172.21.15.42:0/2271117537 wait complete. 2019-12-05T05:06:55.018 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:55.012+0000 7f6b809d0700 1 -- 172.21.15.42:0/2271117537 >> 172.21.15.42:0/2271117537 conn(0x7f6b7c0a7740 msgr2=0x7f6b7c11da30 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:06:55.026 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T05:06:57.028 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:06:57.143 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:57.132+0000 7f79088c7700 1 Processor -- start 2019-12-05T05:06:57.145 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:57.132+0000 7f79088c7700 1 -- start start 2019-12-05T05:06:57.146 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:57.136+0000 7f79088c7700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f79040a7730 0x7f790408df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:06:57.146 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:57.136+0000 7f79088c7700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f790407e710 0x7f790407eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:06:57.146 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:57.136+0000 7f79088c7700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79040a0700 con 0x7f790409f2d0 2019-12-05T05:06:57.149 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:57.136+0000 7f79088c7700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79040a0af0 con 0x7f79040a7730 2019-12-05T05:06:57.149 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:57.136+0000 7f79088c7700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f790408e5a0 con 0x7f790407e710 2019-12-05T05:06:57.149 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:57.136+0000 7f7902d9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f79040a7730 0x7f790408df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:06:57.149 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:57.136+0000 7f7902d9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f79040a7730 0x7f790408df70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:55390/0 (socket says 172.21.15.42:55390) 2019-12-05T05:06:57.149 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:57.136+0000 7f7902d9d700 1 -- 172.21.15.42:0/3388756932 learned_addr learned my addr 172.21.15.42:0/3388756932 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:06:57.150 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:57.136+0000 7f790259c700 1 --2- 172.21.15.42:0/3388756932 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f790407e710 0x7f790407eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:06:57.150 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:57.136+0000 7f7902d9d700 1 -- 172.21.15.42:0/3388756932 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f790407e710 msgr2=0x7f790407eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:06:57.150 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:57.136+0000 7f7902d9d700 1 --2- 172.21.15.42:0/3388756932 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f790407e710 0x7f790407eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:06:57.150 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:57.136+0000 7f7902d9d700 1 -- 172.21.15.42:0/3388756932 >> v1:172.21.15.40:6789/0 conn(0x7f790409f2d0 legacy=0x7f790409f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:06:57.150 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:57.136+0000 7f7901d9b700 1 -- 172.21.15.42:0/3388756932 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3072463893 0 0) 0x7f79040a0700 con 0x7f790409f2d0 2019-12-05T05:06:57.151 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:57.136+0000 7f7902d9d700 1 -- 172.21.15.42:0/3388756932 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f790407f090 con 0x7f79040a7730 2019-12-05T05:06:57.151 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:57.136+0000 7f7902d9d700 1 --2- 172.21.15.42:0/3388756932 >> v2:172.21.15.42:3300/0 conn(0x7f79040a7730 0x7f790408df70 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f78fc008b10 tx=0x7f78fc0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:06:57.151 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:57.136+0000 7f7902d9d700 1 --2- 172.21.15.42:0/3388756932 >> v2:172.21.15.42:3300/0 conn(0x7f79040a7730 0x7f790408df70 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f78fc008b10 tx=0x7f78fc0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:06:57.151 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:57.136+0000 7f7902d9d700 1 --2- 172.21.15.42:0/3388756932 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f79040a7730 0x7f790408df70 secure :-1 s=READY pgs=285 cs=0 l=1 rx=0x7f78fc008b10 tx=0x7f78fc0100b0).ready entity=mon.1 client_cookie=e722614767bb7464 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:06:57.151 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:57.136+0000 7f7902d9d700 1 --2- 172.21.15.42:0/3388756932 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f79040a7730 0x7f790408df70 secure :-1 s=THROTTLE_DONE pgs=285 cs=0 l=1 rx=0x7f78fc008b10 tx=0x7f78fc0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:06:57.151 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:57.136+0000 7f7901d9b700 1 -- 172.21.15.42:0/3388756932 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f78fc00a160 con 0x7f79040a7730 2019-12-05T05:06:57.152 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:57.136+0000 7f7902d9d700 1 --2- 172.21.15.42:0/3388756932 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f79040a7730 0x7f790408df70 secure :-1 s=THROTTLE_DONE pgs=285 cs=0 l=1 rx=0x7f78fc008b10 tx=0x7f78fc0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:06:57.152 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:57.136+0000 7f7902d9d700 1 --2- 172.21.15.42:0/3388756932 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f79040a7730 0x7f790408df70 secure :-1 s=THROTTLE_DONE pgs=285 cs=0 l=1 rx=0x7f78fc008b10 tx=0x7f78fc0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:06:57.152 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:57.136+0000 7f7901d9b700 1 -- 172.21.15.42:0/3388756932 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78fc011a30 con 0x7f79040a7730 2019-12-05T05:06:57.152 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:57.136+0000 7f7901d9b700 1 -- 172.21.15.42:0/3388756932 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f78fc00a550 con 0x7f79040a7730 2019-12-05T05:06:57.152 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:57.136+0000 7f79088c7700 1 -- 172.21.15.42:0/3388756932 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f79040a7730 msgr2=0x7f790408df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:06:57.152 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:57.136+0000 7f79088c7700 1 --2- 172.21.15.42:0/3388756932 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f79040a7730 0x7f790408df70 secure :-1 s=READY pgs=285 cs=0 l=1 rx=0x7f78fc008b10 tx=0x7f78fc0100b0).stop 2019-12-05T05:06:57.152 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:57.136+0000 7f79088c7700 1 -- 172.21.15.42:0/3388756932 shutdown_connections 2019-12-05T05:06:57.153 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:57.136+0000 7f79088c7700 1 --2- 172.21.15.42:0/3388756932 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f79040a7730 0x7f790408df70 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:06:57.153 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:57.136+0000 7f79088c7700 1 --2- 172.21.15.42:0/3388756932 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f790407e710 0x7f790407eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:06:57.153 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:57.136+0000 7f79088c7700 1 -- 172.21.15.42:0/3388756932 shutdown_connections 2019-12-05T05:06:57.153 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:57.140+0000 7f79088c7700 1 -- 172.21.15.42:0/3388756932 wait complete. 2019-12-05T05:06:57.153 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:57.140+0000 7f79088c7700 1 -- 172.21.15.42:0/3388756932 >> 172.21.15.42:0/3388756932 conn(0x7f79040aabd0 msgr2=0x7f79040a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:06:57.157 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:57.140+0000 7f79088c7700 1 Processor -- start 2019-12-05T05:06:57.157 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:57.140+0000 7f79088c7700 1 -- start start 2019-12-05T05:06:57.157 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:57.140+0000 7f79088c7700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f790407ba50 0x7f7904143750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:06:57.157 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:57.140+0000 7f79088c7700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f790407be80 0x7f7904143ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:06:57.158 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:57.140+0000 7f79088c7700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7904142710 con 0x7f7904059f40 2019-12-05T05:06:57.158 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:57.140+0000 7f79088c7700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7904142b40 con 0x7f790407ba50 2019-12-05T05:06:57.158 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:57.140+0000 7f79088c7700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7904142de0 con 0x7f790407be80 2019-12-05T05:06:57.158 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:57.140+0000 7f7902d9d700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f7904059f40 0x7f7904118100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:36642/0 (socket says 172.21.15.42:36642) 2019-12-05T05:06:57.158 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:57.140+0000 7f7902d9d700 1 -- 172.21.15.42:0/1956100717 learned_addr learned my addr 172.21.15.42:0/1956100717 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:06:57.158 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:57.140+0000 7f790359e700 1 --2- 172.21.15.42:0/1956100717 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f790407be80 0x7f7904143ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:06:57.158 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:57.140+0000 7f790259c700 1 --2- 172.21.15.42:0/1956100717 >> v2:172.21.15.40:3300/0 conn(0x7f790407ba50 0x7f7904143750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:06:57.159 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:57.140+0000 7f78fb7fe700 1 -- 172.21.15.42:0/1956100717 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 101662655 0 0) 0x7f7904142710 con 0x7f7904059f40 2019-12-05T05:06:57.159 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:57.140+0000 7f78fb7fe700 1 -- 172.21.15.42:0/1956100717 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78e80030b0 con 0x7f7904059f40 2019-12-05T05:06:57.159 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:57.140+0000 7f790359e700 1 -- 172.21.15.42:0/1956100717 >> v2:172.21.15.40:3300/0 conn(0x7f790407ba50 msgr2=0x7f7904143750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:06:57.159 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:57.140+0000 7f790359e700 1 --2- 172.21.15.42:0/1956100717 >> v2:172.21.15.40:3300/0 conn(0x7f790407ba50 0x7f7904143750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:06:57.159 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:57.140+0000 7f790359e700 1 -- 172.21.15.42:0/1956100717 >> v1:172.21.15.40:6789/0 conn(0x7f7904059f40 legacy=0x7f7904118100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:06:57.159 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:57.140+0000 7f790359e700 1 -- 172.21.15.42:0/1956100717 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7904136710 con 0x7f790407be80 2019-12-05T05:06:57.160 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:57.140+0000 7f790259c700 1 --2- 172.21.15.42:0/1956100717 >> v2:172.21.15.40:3300/0 conn(0x7f790407ba50 0x7f7904143750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:06:57.160 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:57.144+0000 7f790359e700 1 --2- 172.21.15.42:0/1956100717 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f790407be80 0x7f7904143ca0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f78f4000b80 tx=0x7f78f4010780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:06:57.162 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:57.144+0000 7f790359e700 1 --2- 172.21.15.42:0/1956100717 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f790407be80 0x7f7904143ca0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f78f4000b80 tx=0x7f78f4010780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:06:57.162 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:57.144+0000 7f790359e700 1 --2- 172.21.15.42:0/1956100717 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f790407be80 0x7f7904143ca0 secure :-1 s=READY pgs=59 cs=0 l=1 rx=0x7f78f4000b80 tx=0x7f78f4010780).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:06:57.162 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:57.144+0000 7f790359e700 1 --2- 172.21.15.42:0/1956100717 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f790407be80 0x7f7904143ca0 secure :-1 s=THROTTLE_DONE pgs=59 cs=0 l=1 rx=0x7f78f4000b80 tx=0x7f78f4010780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:06:57.162 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:57.144+0000 7f78fb7fe700 1 -- 172.21.15.42:0/1956100717 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f78f4006a40 con 0x7f790407be80 2019-12-05T05:06:57.162 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:57.144+0000 7f790359e700 1 --2- 172.21.15.42:0/1956100717 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f790407be80 0x7f7904143ca0 secure :-1 s=THROTTLE_DONE pgs=59 cs=0 l=1 rx=0x7f78f4000b80 tx=0x7f78f4010780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:06:57.162 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:57.144+0000 7f790359e700 1 --2- 172.21.15.42:0/1956100717 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f790407be80 0x7f7904143ca0 secure :-1 s=THROTTLE_DONE pgs=59 cs=0 l=1 rx=0x7f78f4000b80 tx=0x7f78f4010780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:06:57.163 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:57.144+0000 7f78fb7fe700 1 -- 172.21.15.42:0/1956100717 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78f4006660 con 0x7f790407be80 2019-12-05T05:06:57.163 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:57.144+0000 7f79088c7700 1 -- 172.21.15.42:0/1956100717 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7904135700 con 0x7f790407be80 2019-12-05T05:06:57.163 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:57.144+0000 7f78fb7fe700 1 -- 172.21.15.42:0/1956100717 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f78f4010fc0 con 0x7f790407be80 2019-12-05T05:06:57.163 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:57.144+0000 7f79088c7700 1 -- 172.21.15.42:0/1956100717 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7904135c50 con 0x7f790407be80 2019-12-05T05:06:57.163 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:57.144+0000 7f790359e700 1 --2- 172.21.15.42:0/1956100717 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f790407be80 0x7f7904143ca0 secure :-1 s=THROTTLE_DONE pgs=59 cs=0 l=1 rx=0x7f78f4000b80 tx=0x7f78f4010780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:06:57.163 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:57.144+0000 7f78fb7fe700 1 -- 172.21.15.42:0/1956100717 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f78f40112d0 con 0x7f790407be80 2019-12-05T05:06:57.163 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:57.144+0000 7f790359e700 1 --2- 172.21.15.42:0/1956100717 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f790407be80 0x7f7904143ca0 secure :-1 s=THROTTLE_DONE pgs=59 cs=0 l=1 rx=0x7f78f4000b80 tx=0x7f78f4010780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:06:57.164 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:57.144+0000 7f78fb7fe700 1 --2- 172.21.15.42:0/1956100717 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f78e803b270 0x7f78e803d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:06:57.164 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:57.144+0000 7f78fb7fe700 1 -- 172.21.15.42:0/1956100717 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6145+0+0 (secure 0 0 0) 0x7f78f404afd0 con 0x7f790407be80 2019-12-05T05:06:57.164 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:57.144+0000 7f7902d9d700 1 --2- 172.21.15.42:0/1956100717 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f78e803b270 0x7f78e803d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:06:57.164 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:57.144+0000 7f7902d9d700 1 --2- 172.21.15.42:0/1956100717 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f78e803b270 0x7f78e803d780 crc :-1 s=READY pgs=137 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:06:57.164 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:57.152+0000 7f79088c7700 1 -- 172.21.15.42:0/1956100717 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7904142940 con 0x7f790407be80 2019-12-05T05:06:57.165 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:57.160+0000 7f790359e700 1 --2- 172.21.15.42:0/1956100717 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f790407be80 0x7f7904143ca0 secure :-1 s=THROTTLE_DONE pgs=59 cs=0 l=1 rx=0x7f78f4000b80 tx=0x7f78f4010780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:06:57.165 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:57.160+0000 7f78fb7fe700 1 -- 172.21.15.42:0/1956100717 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f78f4032030 con 0x7f790407be80 2019-12-05T05:06:57.335 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:57.332+0000 7f79088c7700 1 -- 172.21.15.42:0/1956100717 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f790405d100 con 0x7f790407be80 2019-12-05T05:06:57.335 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:57.332+0000 7f790359e700 1 --2- 172.21.15.42:0/1956100717 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f790407be80 0x7f7904143ca0 secure :-1 s=THROTTLE_DONE pgs=59 cs=0 l=1 rx=0x7f78f4000b80 tx=0x7f78f4010780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:06:57.336 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:57.332+0000 7f78fb7fe700 1 -- 172.21.15.42:0/1956100717 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+2101 (secure 0 0 0) 0x7f78f4032290 con 0x7f790407be80 2019-12-05T05:06:57.338 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 23 2019-12-05T05:06:57.338 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:06:57.338 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4831,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/90248709","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":90248709},{"type":"v1","addr":"172.21.15.40:6833","nonce":90248709}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":11}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:05:19.282511+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4354},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4354":{"gid":4354,"name":"a","rank":0,"incarnation":7,"state":"up:active","state_seq":20,"addr":"172.21.15.42:6835/2019200199","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2019200199},{"type":"v1","addr":"172.21.15.42:6835","nonce":2019200199}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:06:57.342 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:57.336+0000 7f79088c7700 1 -- 172.21.15.42:0/1956100717 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f78e803b270 msgr2=0x7f78e803d780 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:06:57.342 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:57.336+0000 7f79088c7700 1 --2- 172.21.15.42:0/1956100717 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f78e803b270 0x7f78e803d780 crc :-1 s=READY pgs=137 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:06:57.342 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:57.336+0000 7f79088c7700 1 -- 172.21.15.42:0/1956100717 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f790407be80 msgr2=0x7f7904143ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:06:57.342 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:57.336+0000 7f79088c7700 1 --2- 172.21.15.42:0/1956100717 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f790407be80 0x7f7904143ca0 secure :-1 s=READY pgs=59 cs=0 l=1 rx=0x7f78f4000b80 tx=0x7f78f4010780).stop 2019-12-05T05:06:57.342 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:57.336+0000 7f79088c7700 1 -- 172.21.15.42:0/1956100717 shutdown_connections 2019-12-05T05:06:57.343 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:57.336+0000 7f79088c7700 1 --2- 172.21.15.42:0/1956100717 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f78e803b270 0x7f78e803d780 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:06:57.343 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:57.336+0000 7f79088c7700 1 --2- 172.21.15.42:0/1956100717 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f790407be80 0x7f7904143ca0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:06:57.343 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:57.336+0000 7f79088c7700 1 --2- 172.21.15.42:0/1956100717 >> v2:172.21.15.40:3300/0 conn(0x7f790407ba50 0x7f7904143750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:06:57.343 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:57.336+0000 7f79088c7700 1 -- 172.21.15.42:0/1956100717 shutdown_connections 2019-12-05T05:06:57.343 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:57.336+0000 7f79088c7700 1 -- 172.21.15.42:0/1956100717 wait complete. 2019-12-05T05:06:57.343 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:57.340+0000 7f79088c7700 1 -- 172.21.15.42:0/1956100717 >> 172.21.15.42:0/1956100717 conn(0x7f790407de80 msgr2=0x7f790411d540 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:06:57.356 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T05:06:59.357 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:06:59.463 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:59.452+0000 7fafea28c700 1 Processor -- start 2019-12-05T05:06:59.463 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:59.452+0000 7fafea28c700 1 -- start start 2019-12-05T05:06:59.464 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:59.456+0000 7fafea28c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fafe40a7740 0x7fafe408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:06:59.464 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:59.456+0000 7fafea28c700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fafe409fae0 0x7fafe407eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:06:59.464 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:59.456+0000 7fafea28c700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafe40a0710 con 0x7fafe407e720 2019-12-05T05:06:59.464 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:59.456+0000 7fafea28c700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafe40a0b00 con 0x7fafe409fae0 2019-12-05T05:06:59.464 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:59.456+0000 7fafea28c700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafe408e5b0 con 0x7fafe40a7740 2019-12-05T05:06:59.464 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:59.456+0000 7fafe8829700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fafe40a7740 0x7fafe408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:06:59.465 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:59.456+0000 7fafe8829700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fafe40a7740 0x7fafe408df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:55406/0 (socket says 172.21.15.42:55406) 2019-12-05T05:06:59.465 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:59.456+0000 7fafe8829700 1 -- 172.21.15.42:0/2448229096 learned_addr learned my addr 172.21.15.42:0/2448229096 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:06:59.465 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:59.456+0000 7fafe3fff700 1 --1- 172.21.15.42:0/2448229096 >> v1:172.21.15.42:6789/0 conn(0x7fafe407e720 0x7fafe409f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:55446/0 (socket says 172.21.15.42:55446) 2019-12-05T05:06:59.465 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:59.456+0000 7fafe902a700 1 --2- 172.21.15.42:0/2448229096 >> v2:172.21.15.40:3300/0 conn(0x7fafe409fae0 0x7fafe407eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:06:59.465 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:59.456+0000 7fafe8829700 1 -- 172.21.15.42:0/2448229096 >> v2:172.21.15.40:3300/0 conn(0x7fafe409fae0 msgr2=0x7fafe407eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:06:59.466 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:59.456+0000 7fafe8829700 1 --2- 172.21.15.42:0/2448229096 >> v2:172.21.15.40:3300/0 conn(0x7fafe409fae0 0x7fafe407eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:06:59.466 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:59.456+0000 7fafe8829700 1 -- 172.21.15.42:0/2448229096 >> v1:172.21.15.42:6789/0 conn(0x7fafe407e720 legacy=0x7fafe409f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:06:59.466 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:59.456+0000 7fafe8829700 1 -- 172.21.15.42:0/2448229096 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafe407f0a0 con 0x7fafe40a7740 2019-12-05T05:06:59.466 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:59.456+0000 7fafe902a700 1 --2- 172.21.15.42:0/2448229096 >> v2:172.21.15.40:3300/0 conn(0x7fafe409fae0 0x7fafe407eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T05:06:59.466 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:59.456+0000 7fafe8829700 1 --2- 172.21.15.42:0/2448229096 >> v2:172.21.15.42:3300/0 conn(0x7fafe40a7740 0x7fafe408df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fafdc002f30 tx=0x7fafdc00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:06:59.467 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:59.456+0000 7fafe8829700 1 --2- 172.21.15.42:0/2448229096 >> v2:172.21.15.42:3300/0 conn(0x7fafe40a7740 0x7fafe408df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fafdc002f30 tx=0x7fafdc00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:06:59.467 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:59.456+0000 7fafe8829700 1 --2- 172.21.15.42:0/2448229096 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fafe40a7740 0x7fafe408df80 secure :-1 s=READY pgs=287 cs=0 l=1 rx=0x7fafdc002f30 tx=0x7fafdc00fd30).ready entity=mon.1 client_cookie=c14f38656ac552f8 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:06:59.467 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:59.456+0000 7fafe8829700 1 --2- 172.21.15.42:0/2448229096 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fafe40a7740 0x7fafe408df80 secure :-1 s=THROTTLE_DONE pgs=287 cs=0 l=1 rx=0x7fafdc002f30 tx=0x7fafdc00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:06:59.467 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:59.456+0000 7fafe8829700 1 --2- 172.21.15.42:0/2448229096 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fafe40a7740 0x7fafe408df80 secure :-1 s=THROTTLE_DONE pgs=287 cs=0 l=1 rx=0x7fafdc002f30 tx=0x7fafdc00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:06:59.467 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:59.456+0000 7fafe37fe700 1 -- 172.21.15.42:0/2448229096 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fafdc008c90 con 0x7fafe40a7740 2019-12-05T05:06:59.467 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:59.456+0000 7fafe8829700 1 --2- 172.21.15.42:0/2448229096 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fafe40a7740 0x7fafe408df80 secure :-1 s=THROTTLE_DONE pgs=287 cs=0 l=1 rx=0x7fafdc002f30 tx=0x7fafdc00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:06:59.468 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:59.456+0000 7fafe37fe700 1 -- 172.21.15.42:0/2448229096 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafdc0088b0 con 0x7fafe40a7740 2019-12-05T05:06:59.468 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:59.456+0000 7fafe37fe700 1 -- 172.21.15.42:0/2448229096 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fafdc011ad0 con 0x7fafe40a7740 2019-12-05T05:06:59.468 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:59.456+0000 7fafea28c700 1 -- 172.21.15.42:0/2448229096 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fafe40a7740 msgr2=0x7fafe408df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:06:59.468 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:59.456+0000 7fafea28c700 1 --2- 172.21.15.42:0/2448229096 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fafe40a7740 0x7fafe408df80 secure :-1 s=READY pgs=287 cs=0 l=1 rx=0x7fafdc002f30 tx=0x7fafdc00fd30).stop 2019-12-05T05:06:59.468 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:59.456+0000 7fafea28c700 1 -- 172.21.15.42:0/2448229096 shutdown_connections 2019-12-05T05:06:59.469 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:59.456+0000 7fafea28c700 1 --2- 172.21.15.42:0/2448229096 >> v2:172.21.15.40:3300/0 conn(0x7fafe409fae0 0x7fafe407eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:06:59.469 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:59.456+0000 7fafea28c700 1 --2- 172.21.15.42:0/2448229096 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fafe40a7740 0x7fafe408df80 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:06:59.469 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:59.456+0000 7fafea28c700 1 -- 172.21.15.42:0/2448229096 shutdown_connections 2019-12-05T05:06:59.469 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:59.456+0000 7fafea28c700 1 -- 172.21.15.42:0/2448229096 wait complete. 2019-12-05T05:06:59.469 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:59.456+0000 7fafea28c700 1 -- 172.21.15.42:0/2448229096 >> 172.21.15.42:0/2448229096 conn(0x7fafe40aabe0 msgr2=0x7fafe40a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:06:59.469 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:59.460+0000 7fafea28c700 1 Processor -- start 2019-12-05T05:06:59.470 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:59.460+0000 7fafea28c700 1 -- start start 2019-12-05T05:06:59.474 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:59.460+0000 7fafea28c700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fafe4122da0 0x7fafe4117780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:06:59.474 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:59.460+0000 7fafea28c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fafe4118690 0x7fafe411b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:06:59.474 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:59.460+0000 7fafea28c700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafe4118ac0 con 0x7fafe411bca0 2019-12-05T05:06:59.474 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:59.460+0000 7fafea28c700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafe411c0d0 con 0x7fafe4118690 2019-12-05T05:06:59.474 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:59.460+0000 7fafea28c700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafe4117240 con 0x7fafe4122da0 2019-12-05T05:06:59.475 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:59.460+0000 7fafe902a700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fafe411bca0 0x7fafe411a310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:55452/0 (socket says 172.21.15.42:55452) 2019-12-05T05:06:59.475 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:59.460+0000 7fafe902a700 1 -- 172.21.15.42:0/1242698086 learned_addr learned my addr 172.21.15.42:0/1242698086 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:06:59.475 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:59.460+0000 7fafe3fff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fafe4118690 0x7fafe411b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:06:59.475 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:59.460+0000 7fafe8829700 1 --2- 172.21.15.42:0/1242698086 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fafe4122da0 0x7fafe4117780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:06:59.475 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:59.460+0000 7fafe17fa700 1 -- 172.21.15.42:0/1242698086 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4236264886 0 0) 0x7fafe4118ac0 con 0x7fafe411bca0 2019-12-05T05:06:59.475 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:59.460+0000 7fafe17fa700 1 -- 172.21.15.42:0/1242698086 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fafc80030b0 con 0x7fafe411bca0 2019-12-05T05:06:59.476 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:59.460+0000 7fafe3fff700 1 -- 172.21.15.42:0/1242698086 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fafe4122da0 msgr2=0x7fafe4117780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:06:59.476 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:59.460+0000 7fafe3fff700 1 --2- 172.21.15.42:0/1242698086 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fafe4122da0 0x7fafe4117780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:06:59.476 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:59.460+0000 7fafe3fff700 1 -- 172.21.15.42:0/1242698086 >> v1:172.21.15.42:6789/0 conn(0x7fafe411bca0 legacy=0x7fafe411a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:06:59.476 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:59.460+0000 7fafe3fff700 1 -- 172.21.15.42:0/1242698086 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafe411ab10 con 0x7fafe4118690 2019-12-05T05:06:59.476 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:59.460+0000 7fafe8829700 1 --2- 172.21.15.42:0/1242698086 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fafe4122da0 0x7fafe4117780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:06:59.476 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:59.460+0000 7fafe3fff700 1 --2- 172.21.15.42:0/1242698086 >> v2:172.21.15.42:3300/0 conn(0x7fafe4118690 0x7fafe411b750 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fafe4117400 tx=0x7fafcc00c6f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:06:59.477 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:59.460+0000 7fafe3fff700 1 --2- 172.21.15.42:0/1242698086 >> v2:172.21.15.42:3300/0 conn(0x7fafe4118690 0x7fafe411b750 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fafe4117400 tx=0x7fafcc00c6f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:06:59.477 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:59.460+0000 7fafe3fff700 1 --2- 172.21.15.42:0/1242698086 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fafe4118690 0x7fafe411b750 secure :-1 s=READY pgs=289 cs=0 l=1 rx=0x7fafe4117400 tx=0x7fafcc00c6f0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:06:59.477 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:59.460+0000 7fafe3fff700 1 --2- 172.21.15.42:0/1242698086 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fafe4118690 0x7fafe411b750 secure :-1 s=THROTTLE_DONE pgs=289 cs=0 l=1 rx=0x7fafe4117400 tx=0x7fafcc00c6f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:06:59.477 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:59.460+0000 7fafe3fff700 1 --2- 172.21.15.42:0/1242698086 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fafe4118690 0x7fafe411b750 secure :-1 s=THROTTLE_DONE pgs=289 cs=0 l=1 rx=0x7fafe4117400 tx=0x7fafcc00c6f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:06:59.477 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:59.460+0000 7fafe17fa700 1 -- 172.21.15.42:0/1242698086 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fafcc00cb50 con 0x7fafe4118690 2019-12-05T05:06:59.477 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:59.460+0000 7fafe3fff700 1 --2- 172.21.15.42:0/1242698086 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fafe4118690 0x7fafe411b750 secure :-1 s=THROTTLE_DONE pgs=289 cs=0 l=1 rx=0x7fafe4117400 tx=0x7fafcc00c6f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:06:59.478 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:59.460+0000 7fafe17fa700 1 -- 172.21.15.42:0/1242698086 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafcc00c770 con 0x7fafe4118690 2019-12-05T05:06:59.478 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:59.460+0000 7fafea28c700 1 -- 172.21.15.42:0/1242698086 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafe411ade0 con 0x7fafe4118690 2019-12-05T05:06:59.478 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:59.460+0000 7fafe17fa700 1 -- 172.21.15.42:0/1242698086 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fafcc0048e0 con 0x7fafe4118690 2019-12-05T05:06:59.478 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:59.460+0000 7fafea28c700 1 -- 172.21.15.42:0/1242698086 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafe4077300 con 0x7fafe4118690 2019-12-05T05:06:59.478 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:59.464+0000 7fafe3fff700 1 --2- 172.21.15.42:0/1242698086 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fafe4118690 0x7fafe411b750 secure :-1 s=THROTTLE_DONE pgs=289 cs=0 l=1 rx=0x7fafe4117400 tx=0x7fafcc00c6f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:06:59.478 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:59.464+0000 7fafe17fa700 1 -- 172.21.15.42:0/1242698086 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fafcc004bf0 con 0x7fafe4118690 2019-12-05T05:06:59.479 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:59.464+0000 7fafe3fff700 1 --2- 172.21.15.42:0/1242698086 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fafe4118690 0x7fafe411b750 secure :-1 s=THROTTLE_DONE pgs=289 cs=0 l=1 rx=0x7fafe4117400 tx=0x7fafcc00c6f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:06:59.479 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:59.464+0000 7fafe17fa700 1 --2- 172.21.15.42:0/1242698086 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fafc803b2a0 0x7fafc803d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:06:59.479 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:59.464+0000 7fafe17fa700 1 -- 172.21.15.42:0/1242698086 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6145+0+0 (secure 0 0 0) 0x7fafcc04b110 con 0x7fafe4118690 2019-12-05T05:06:59.479 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:59.464+0000 7fafe8829700 1 --2- 172.21.15.42:0/1242698086 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fafc803b2a0 0x7fafc803d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:06:59.479 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:59.464+0000 7fafe8829700 1 --2- 172.21.15.42:0/1242698086 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fafc803b2a0 0x7fafc803d7b0 crc :-1 s=READY pgs=138 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:06:59.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:59.468+0000 7fafea28c700 1 -- 172.21.15.42:0/1242698086 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafe405d110 con 0x7fafe4118690 2019-12-05T05:06:59.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:59.476+0000 7fafe3fff700 1 --2- 172.21.15.42:0/1242698086 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fafe4118690 0x7fafe411b750 secure :-1 s=THROTTLE_DONE pgs=289 cs=0 l=1 rx=0x7fafe4117400 tx=0x7fafcc00c6f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:06:59.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:59.476+0000 7fafe17fa700 1 -- 172.21.15.42:0/1242698086 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fafcc032190 con 0x7fafe4118690 2019-12-05T05:06:59.642 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:59.636+0000 7fafea28c700 1 -- 172.21.15.42:0/1242698086 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fafe405b8c0 con 0x7fafe4118690 2019-12-05T05:06:59.643 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:59.640+0000 7fafe3fff700 1 --2- 172.21.15.42:0/1242698086 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fafe4118690 0x7fafe411b750 secure :-1 s=THROTTLE_DONE pgs=289 cs=0 l=1 rx=0x7fafe4117400 tx=0x7fafcc00c6f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:06:59.643 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:59.640+0000 7fafe17fa700 1 -- 172.21.15.42:0/1242698086 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+2101 (secure 0 0 0) 0x7fafcc0323f0 con 0x7fafe4118690 2019-12-05T05:06:59.643 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 23 2019-12-05T05:06:59.643 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:06:59.643 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4831,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/90248709","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":90248709},{"type":"v1","addr":"172.21.15.40:6833","nonce":90248709}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":11}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:05:19.282511+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4354},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4354":{"gid":4354,"name":"a","rank":0,"incarnation":7,"state":"up:active","state_seq":20,"addr":"172.21.15.42:6835/2019200199","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2019200199},{"type":"v1","addr":"172.21.15.42:6835","nonce":2019200199}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:06:59.645 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:59.640+0000 7fafea28c700 1 -- 172.21.15.42:0/1242698086 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fafc803b2a0 msgr2=0x7fafc803d7b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:06:59.646 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:59.640+0000 7fafea28c700 1 --2- 172.21.15.42:0/1242698086 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fafc803b2a0 0x7fafc803d7b0 crc :-1 s=READY pgs=138 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:06:59.646 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:59.640+0000 7fafea28c700 1 -- 172.21.15.42:0/1242698086 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fafe4118690 msgr2=0x7fafe411b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:06:59.646 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:59.640+0000 7fafea28c700 1 --2- 172.21.15.42:0/1242698086 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fafe4118690 0x7fafe411b750 secure :-1 s=READY pgs=289 cs=0 l=1 rx=0x7fafe4117400 tx=0x7fafcc00c6f0).stop 2019-12-05T05:06:59.646 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:59.640+0000 7fafea28c700 1 -- 172.21.15.42:0/1242698086 shutdown_connections 2019-12-05T05:06:59.646 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:59.640+0000 7fafea28c700 1 --2- 172.21.15.42:0/1242698086 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fafc803b2a0 0x7fafc803d7b0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:06:59.647 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:59.640+0000 7fafea28c700 1 --2- 172.21.15.42:0/1242698086 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fafe4122da0 0x7fafe4117780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:06:59.647 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:59.640+0000 7fafea28c700 1 --2- 172.21.15.42:0/1242698086 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fafe4118690 0x7fafe411b750 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:06:59.647 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:59.640+0000 7fafea28c700 1 -- 172.21.15.42:0/1242698086 shutdown_connections 2019-12-05T05:06:59.647 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:59.640+0000 7fafea28c700 1 -- 172.21.15.42:0/1242698086 wait complete. 2019-12-05T05:06:59.647 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:06:59.640+0000 7fafea28c700 1 -- 172.21.15.42:0/1242698086 >> 172.21.15.42:0/1242698086 conn(0x7fafe40aabe0 msgr2=0x7fafe411d8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:06:59.655 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T05:07:01.656 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:07:01.767 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:01.756+0000 7f0e5a1a2700 1 Processor -- start 2019-12-05T05:07:01.768 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:01.756+0000 7f0e5a1a2700 1 -- start start 2019-12-05T05:07:01.768 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:01.756+0000 7f0e5a1a2700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f0e540a7740 0x7f0e5408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:07:01.768 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:01.760+0000 7f0e5a1a2700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e540a0710 con 0x7f0e5407e720 2019-12-05T05:07:01.768 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:01.760+0000 7f0e5a1a2700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e540a9ba0 con 0x7f0e5409fae0 2019-12-05T05:07:01.768 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:01.760+0000 7f0e5a1a2700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e540a0b00 con 0x7f0e540a7740 2019-12-05T05:07:01.768 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:01.760+0000 7f0e58f40700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f0e5409fae0 0x7f0e5414efe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:55466/0 (socket says 172.21.15.42:55466) 2019-12-05T05:07:01.769 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:01.760+0000 7f0e58f40700 1 -- 172.21.15.42:0/995857044 learned_addr learned my addr 172.21.15.42:0/995857044 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:07:01.769 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:01.760+0000 7f0e53fff700 1 --2- 172.21.15.42:0/995857044 >> v2:172.21.15.40:3300/0 conn(0x7f0e540a7740 0x7f0e5408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:07:01.769 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:01.760+0000 7f0e52ffd700 1 -- 172.21.15.42:0/995857044 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 284905243 0 0) 0x7f0e540a9ba0 con 0x7f0e5409fae0 2019-12-05T05:07:01.769 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:01.760+0000 7f0e52ffd700 1 -- 172.21.15.42:0/995857044 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e380030b0 con 0x7f0e5409fae0 2019-12-05T05:07:01.769 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:01.760+0000 7f0e52ffd700 1 -- 172.21.15.42:0/995857044 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3516085455 0 0) 0x7f0e540a0710 con 0x7f0e5407e720 2019-12-05T05:07:01.769 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:01.760+0000 7f0e52ffd700 1 -- 172.21.15.42:0/995857044 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e540a9ba0 con 0x7f0e5407e720 2019-12-05T05:07:01.769 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:01.760+0000 7f0e52ffd700 1 -- 172.21.15.42:0/995857044 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2069011193 0 0) 0x7f0e380030b0 con 0x7f0e5409fae0 2019-12-05T05:07:01.770 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:01.760+0000 7f0e53fff700 1 -- 172.21.15.42:0/995857044 >> v1:172.21.15.42:6789/0 conn(0x7f0e5409fae0 legacy=0x7f0e5414efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:07:01.770 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:01.760+0000 7f0e53fff700 1 -- 172.21.15.42:0/995857044 >> v1:172.21.15.40:6789/0 conn(0x7f0e5407e720 legacy=0x7f0e5409f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:07:01.770 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:01.760+0000 7f0e53fff700 1 -- 172.21.15.42:0/995857044 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e5407eb50 con 0x7f0e540a7740 2019-12-05T05:07:01.770 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:01.760+0000 7f0e53fff700 1 --2- 172.21.15.42:0/995857044 >> v2:172.21.15.40:3300/0 conn(0x7f0e540a7740 0x7f0e5408df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f0e44008b10 tx=0x7f0e4c001950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:07:01.770 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:01.760+0000 7f0e53fff700 1 --2- 172.21.15.42:0/995857044 >> v2:172.21.15.40:3300/0 conn(0x7f0e540a7740 0x7f0e5408df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f0e44008b10 tx=0x7f0e4c001950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:07:01.770 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:01.760+0000 7f0e53fff700 1 --2- 172.21.15.42:0/995857044 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0e540a7740 0x7f0e5408df80 secure :-1 s=READY pgs=240 cs=0 l=1 rx=0x7f0e44008b10 tx=0x7f0e4c001950).ready entity=mon.0 client_cookie=c4f31cc95604879c server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:07:01.770 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:01.760+0000 7f0e53fff700 1 --2- 172.21.15.42:0/995857044 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0e540a7740 0x7f0e5408df80 secure :-1 s=THROTTLE_DONE pgs=240 cs=0 l=1 rx=0x7f0e44008b10 tx=0x7f0e4c001950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:07:01.771 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:01.760+0000 7f0e52ffd700 1 -- 172.21.15.42:0/995857044 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0e4c001750 con 0x7f0e540a7740 2019-12-05T05:07:01.771 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:01.760+0000 7f0e53fff700 1 --2- 172.21.15.42:0/995857044 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0e540a7740 0x7f0e5408df80 secure :-1 s=THROTTLE_DONE pgs=240 cs=0 l=1 rx=0x7f0e44008b10 tx=0x7f0e4c001950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:07:01.771 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:01.760+0000 7f0e52ffd700 1 -- 172.21.15.42:0/995857044 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e4c001560 con 0x7f0e540a7740 2019-12-05T05:07:01.771 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:01.760+0000 7f0e53fff700 1 --2- 172.21.15.42:0/995857044 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0e540a7740 0x7f0e5408df80 secure :-1 s=THROTTLE_DONE pgs=240 cs=0 l=1 rx=0x7f0e44008b10 tx=0x7f0e4c001950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:07:01.771 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:01.760+0000 7f0e52ffd700 1 -- 172.21.15.42:0/995857044 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0e44011ac0 con 0x7f0e540a7740 2019-12-05T05:07:01.771 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:01.760+0000 7f0e5a1a2700 1 -- 172.21.15.42:0/995857044 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0e540a7740 msgr2=0x7f0e5408df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:07:01.771 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:01.760+0000 7f0e5a1a2700 1 --2- 172.21.15.42:0/995857044 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0e540a7740 0x7f0e5408df80 secure :-1 s=READY pgs=240 cs=0 l=1 rx=0x7f0e44008b10 tx=0x7f0e4c001950).stop 2019-12-05T05:07:01.772 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:01.760+0000 7f0e5a1a2700 1 -- 172.21.15.42:0/995857044 shutdown_connections 2019-12-05T05:07:01.772 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:01.760+0000 7f0e5a1a2700 1 --2- 172.21.15.42:0/995857044 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0e540a7740 0x7f0e5408df80 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:07:01.772 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:01.760+0000 7f0e5a1a2700 1 -- 172.21.15.42:0/995857044 shutdown_connections 2019-12-05T05:07:01.772 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:01.760+0000 7f0e5a1a2700 1 -- 172.21.15.42:0/995857044 wait complete. 2019-12-05T05:07:01.772 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:01.760+0000 7f0e5a1a2700 1 -- 172.21.15.42:0/995857044 >> 172.21.15.42:0/995857044 conn(0x7f0e540aabe0 msgr2=0x7f0e540a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:07:01.772 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:01.764+0000 7f0e5a1a2700 1 Processor -- start 2019-12-05T05:07:01.773 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:01.764+0000 7f0e5a1a2700 1 -- start start 2019-12-05T05:07:01.776 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:01.764+0000 7f0e5a1a2700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0e540a6270 0x7f0e5411b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:07:01.776 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:01.764+0000 7f0e5a1a2700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0e54118360 0x7f0e5413a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:07:01.776 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:01.764+0000 7f0e5a1a2700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e5411bef0 con 0x7f0e54117f30 2019-12-05T05:07:01.776 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:01.764+0000 7f0e5a1a2700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e540a6820 con 0x7f0e54118360 2019-12-05T05:07:01.777 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:01.764+0000 7f0e5a1a2700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e540a6ac0 con 0x7f0e540a6270 2019-12-05T05:07:01.777 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:01.764+0000 7f0e58f40700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0e54118360 0x7f0e5413a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:07:01.777 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:01.764+0000 7f0e58f40700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0e54118360 0x7f0e5413a830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:55430/0 (socket says 172.21.15.42:55430) 2019-12-05T05:07:01.777 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:01.764+0000 7f0e58f40700 1 -- 172.21.15.42:0/2845410296 learned_addr learned my addr 172.21.15.42:0/2845410296 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:07:01.778 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:01.764+0000 7f0e537fe700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f0e54117f30 0x7f0e540a5190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:55470/0 (socket says 172.21.15.42:55470) 2019-12-05T05:07:01.778 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:01.764+0000 7f0e53fff700 1 --2- 172.21.15.42:0/2845410296 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0e540a6270 0x7f0e5411b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:07:01.778 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:01.764+0000 7f0e58f40700 1 -- 172.21.15.42:0/2845410296 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0e540a6270 msgr2=0x7f0e5411b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:07:01.778 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:01.764+0000 7f0e58f40700 1 --2- 172.21.15.42:0/2845410296 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0e540a6270 0x7f0e5411b9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:07:01.778 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:01.764+0000 7f0e58f40700 1 -- 172.21.15.42:0/2845410296 >> v1:172.21.15.42:6789/0 conn(0x7f0e54117f30 legacy=0x7f0e540a5190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:07:01.778 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:01.764+0000 7f0e58f40700 1 -- 172.21.15.42:0/2845410296 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e540a5990 con 0x7f0e54118360 2019-12-05T05:07:01.779 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:01.764+0000 7f0e50ff9700 1 -- 172.21.15.42:0/2845410296 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1668563051 0 0) 0x7f0e5411bef0 con 0x7f0e54117f30 2019-12-05T05:07:01.779 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:01.764+0000 7f0e58f40700 1 --2- 172.21.15.42:0/2845410296 >> v2:172.21.15.42:3300/0 conn(0x7f0e54118360 0x7f0e5413a830 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f0e540a69e0 tx=0x7f0e4c010910).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:07:01.779 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:01.764+0000 7f0e58f40700 1 --2- 172.21.15.42:0/2845410296 >> v2:172.21.15.42:3300/0 conn(0x7f0e54118360 0x7f0e5413a830 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f0e540a69e0 tx=0x7f0e4c010910).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:07:01.779 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:01.764+0000 7f0e58f40700 1 --2- 172.21.15.42:0/2845410296 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0e54118360 0x7f0e5413a830 secure :-1 s=READY pgs=292 cs=0 l=1 rx=0x7f0e540a69e0 tx=0x7f0e4c010910).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:07:01.779 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:01.764+0000 7f0e58f40700 1 --2- 172.21.15.42:0/2845410296 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0e54118360 0x7f0e5413a830 secure :-1 s=THROTTLE_DONE pgs=292 cs=0 l=1 rx=0x7f0e540a69e0 tx=0x7f0e4c010910).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:07:01.779 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:01.764+0000 7f0e50ff9700 1 -- 172.21.15.42:0/2845410296 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0e4c0116f0 con 0x7f0e54118360 2019-12-05T05:07:01.780 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:01.764+0000 7f0e5a1a2700 1 -- 172.21.15.42:0/2845410296 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e5413ad80 con 0x7f0e54118360 2019-12-05T05:07:01.780 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:01.764+0000 7f0e58f40700 1 --2- 172.21.15.42:0/2845410296 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0e54118360 0x7f0e5413a830 secure :-1 s=THROTTLE_DONE pgs=292 cs=0 l=1 rx=0x7f0e540a69e0 tx=0x7f0e4c010910).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:07:01.780 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:01.764+0000 7f0e58f40700 1 --2- 172.21.15.42:0/2845410296 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0e54118360 0x7f0e5413a830 secure :-1 s=THROTTLE_DONE pgs=292 cs=0 l=1 rx=0x7f0e540a69e0 tx=0x7f0e4c010910).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:07:01.780 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:01.764+0000 7f0e50ff9700 1 -- 172.21.15.42:0/2845410296 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e4c011310 con 0x7f0e54118360 2019-12-05T05:07:01.780 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:01.764+0000 7f0e5a1a2700 1 -- 172.21.15.42:0/2845410296 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e5413b2b0 con 0x7f0e54118360 2019-12-05T05:07:01.780 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:01.764+0000 7f0e50ff9700 1 -- 172.21.15.42:0/2845410296 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0e4c011cd0 con 0x7f0e54118360 2019-12-05T05:07:01.781 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:01.768+0000 7f0e58f40700 1 --2- 172.21.15.42:0/2845410296 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0e54118360 0x7f0e5413a830 secure :-1 s=THROTTLE_DONE pgs=292 cs=0 l=1 rx=0x7f0e540a69e0 tx=0x7f0e4c010910).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:07:01.781 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:01.768+0000 7f0e50ff9700 1 -- 172.21.15.42:0/2845410296 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f0e4c00b330 con 0x7f0e54118360 2019-12-05T05:07:01.782 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:01.768+0000 7f0e58f40700 1 --2- 172.21.15.42:0/2845410296 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0e54118360 0x7f0e5413a830 secure :-1 s=THROTTLE_DONE pgs=292 cs=0 l=1 rx=0x7f0e540a69e0 tx=0x7f0e4c010910).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:07:01.782 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:01.768+0000 7f0e50ff9700 1 --2- 172.21.15.42:0/2845410296 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0e3003ac90 0x7f0e3003d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:07:01.782 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:01.768+0000 7f0e50ff9700 1 -- 172.21.15.42:0/2845410296 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6145+0+0 (secure 0 0 0) 0x7f0e4c04b1c0 con 0x7f0e54118360 2019-12-05T05:07:01.782 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:01.768+0000 7f0e53fff700 1 --2- 172.21.15.42:0/2845410296 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0e3003ac90 0x7f0e3003d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:07:01.782 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:01.768+0000 7f0e53fff700 1 --2- 172.21.15.42:0/2845410296 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0e3003ac90 0x7f0e3003d1a0 crc :-1 s=READY pgs=139 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:07:01.784 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:01.772+0000 7f0e5a1a2700 1 -- 172.21.15.42:0/2845410296 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e5405d110 con 0x7f0e54118360 2019-12-05T05:07:01.784 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:01.780+0000 7f0e58f40700 1 --2- 172.21.15.42:0/2845410296 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0e54118360 0x7f0e5413a830 secure :-1 s=THROTTLE_DONE pgs=292 cs=0 l=1 rx=0x7f0e540a69e0 tx=0x7f0e4c010910).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:07:01.784 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:01.780+0000 7f0e50ff9700 1 -- 172.21.15.42:0/2845410296 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f0e4c032220 con 0x7f0e54118360 2019-12-05T05:07:01.946 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:01.944+0000 7f0e5a1a2700 1 -- 172.21.15.42:0/2845410296 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0e5408e4d0 con 0x7f0e54118360 2019-12-05T05:07:01.947 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:01.944+0000 7f0e58f40700 1 --2- 172.21.15.42:0/2845410296 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0e54118360 0x7f0e5413a830 secure :-1 s=THROTTLE_DONE pgs=292 cs=0 l=1 rx=0x7f0e540a69e0 tx=0x7f0e4c010910).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:07:01.948 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:01.944+0000 7f0e50ff9700 1 -- 172.21.15.42:0/2845410296 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+2101 (secure 0 0 0) 0x7f0e4c032480 con 0x7f0e54118360 2019-12-05T05:07:01.948 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 23 2019-12-05T05:07:01.948 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:07:01.948 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4831,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/90248709","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":90248709},{"type":"v1","addr":"172.21.15.40:6833","nonce":90248709}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":11}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:05:19.282511+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4354},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4354":{"gid":4354,"name":"a","rank":0,"incarnation":7,"state":"up:active","state_seq":20,"addr":"172.21.15.42:6835/2019200199","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2019200199},{"type":"v1","addr":"172.21.15.42:6835","nonce":2019200199}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:07:01.950 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:01.944+0000 7f0e5a1a2700 1 -- 172.21.15.42:0/2845410296 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0e3003ac90 msgr2=0x7f0e3003d1a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:07:01.950 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:01.944+0000 7f0e5a1a2700 1 --2- 172.21.15.42:0/2845410296 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0e3003ac90 0x7f0e3003d1a0 crc :-1 s=READY pgs=139 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:07:01.951 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:01.944+0000 7f0e5a1a2700 1 -- 172.21.15.42:0/2845410296 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0e54118360 msgr2=0x7f0e5413a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:07:01.951 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:01.944+0000 7f0e5a1a2700 1 --2- 172.21.15.42:0/2845410296 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0e54118360 0x7f0e5413a830 secure :-1 s=READY pgs=292 cs=0 l=1 rx=0x7f0e540a69e0 tx=0x7f0e4c010910).stop 2019-12-05T05:07:01.951 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:01.948+0000 7f0e5a1a2700 1 -- 172.21.15.42:0/2845410296 shutdown_connections 2019-12-05T05:07:01.951 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:01.948+0000 7f0e5a1a2700 1 --2- 172.21.15.42:0/2845410296 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0e3003ac90 0x7f0e3003d1a0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:07:01.951 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:01.948+0000 7f0e5a1a2700 1 --2- 172.21.15.42:0/2845410296 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0e54118360 0x7f0e5413a830 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:07:01.951 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:01.948+0000 7f0e5a1a2700 1 --2- 172.21.15.42:0/2845410296 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0e540a6270 0x7f0e5411b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:07:01.951 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:01.948+0000 7f0e5a1a2700 1 -- 172.21.15.42:0/2845410296 shutdown_connections 2019-12-05T05:07:01.952 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:01.948+0000 7f0e5a1a2700 1 -- 172.21.15.42:0/2845410296 wait complete. 2019-12-05T05:07:01.952 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:01.948+0000 7f0e5a1a2700 1 -- 172.21.15.42:0/2845410296 >> 172.21.15.42:0/2845410296 conn(0x7f0e5407ee20 msgr2=0x7f0e5409f3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:07:01.961 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T05:07:03.964 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:07:04.073 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:04.064+0000 7f4dd839f700 1 Processor -- start 2019-12-05T05:07:04.074 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:04.064+0000 7f4dd839f700 1 -- start start 2019-12-05T05:07:04.079 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:04.064+0000 7f4dd839f700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4dd00a7740 0x7f4dd008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:07:04.079 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:04.064+0000 7f4dd839f700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f4dd009fae0 0x7f4dd007eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:07:04.080 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:04.064+0000 7f4dd839f700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4dd00a0710 con 0x7f4dd007e720 2019-12-05T05:07:04.080 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:04.064+0000 7f4dd839f700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4dd00a0b00 con 0x7f4dd009fae0 2019-12-05T05:07:04.083 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:04.064+0000 7f4dd839f700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4dd008e5b0 con 0x7f4dd00a7740 2019-12-05T05:07:04.084 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:04.064+0000 7f4dd613b700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f4dd007e720 0x7f4dd009f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:55476/0 (socket says 172.21.15.42:55476) 2019-12-05T05:07:04.084 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:04.064+0000 7f4dd613b700 1 -- 172.21.15.42:0/1994241631 learned_addr learned my addr 172.21.15.42:0/1994241631 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:07:04.084 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:04.064+0000 7f4dd713d700 1 --2- 172.21.15.42:0/1994241631 >> v2:172.21.15.40:3300/0 conn(0x7f4dd009fae0 0x7f4dd007eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:07:04.084 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:04.064+0000 7f4dd693c700 1 --2- 172.21.15.42:0/1994241631 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4dd00a7740 0x7f4dd008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:07:04.085 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:04.064+0000 7f4dd593a700 1 -- 172.21.15.42:0/1994241631 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4033929686 0 0) 0x7f4dd00a0710 con 0x7f4dd007e720 2019-12-05T05:07:04.085 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:04.064+0000 7f4dd593a700 1 -- 172.21.15.42:0/1994241631 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4dbc0030b0 con 0x7f4dd007e720 2019-12-05T05:07:04.085 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:04.064+0000 7f4dd693c700 1 -- 172.21.15.42:0/1994241631 >> v2:172.21.15.40:3300/0 conn(0x7f4dd009fae0 msgr2=0x7f4dd007eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:07:04.085 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:04.064+0000 7f4dd693c700 1 --2- 172.21.15.42:0/1994241631 >> v2:172.21.15.40:3300/0 conn(0x7f4dd009fae0 0x7f4dd007eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:07:04.086 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:04.064+0000 7f4dd693c700 1 -- 172.21.15.42:0/1994241631 >> v1:172.21.15.42:6789/0 conn(0x7f4dd007e720 legacy=0x7f4dd009f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:07:04.086 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:04.064+0000 7f4dd693c700 1 -- 172.21.15.42:0/1994241631 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dd007f0a0 con 0x7f4dd00a7740 2019-12-05T05:07:04.086 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:04.064+0000 7f4dd713d700 1 --2- 172.21.15.42:0/1994241631 >> v2:172.21.15.40:3300/0 conn(0x7f4dd009fae0 0x7f4dd007eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:07:04.086 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:04.064+0000 7f4dd693c700 1 --2- 172.21.15.42:0/1994241631 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4dd00a7740 0x7f4dd008df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f4dcc008790 tx=0x7f4dcc00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:07:04.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:04.064+0000 7f4dd693c700 1 --2- 172.21.15.42:0/1994241631 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4dd00a7740 0x7f4dd008df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f4dcc008790 tx=0x7f4dcc00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:07:04.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:04.064+0000 7f4dd693c700 1 --2- 172.21.15.42:0/1994241631 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4dd00a7740 0x7f4dd008df80 secure :-1 s=READY pgs=60 cs=0 l=1 rx=0x7f4dcc008790 tx=0x7f4dcc00fc70).ready entity=mon.2 client_cookie=59c284517bf156ba server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:07:04.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:04.068+0000 7f4dd693c700 1 --2- 172.21.15.42:0/1994241631 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4dd00a7740 0x7f4dd008df80 secure :-1 s=THROTTLE_DONE pgs=60 cs=0 l=1 rx=0x7f4dcc008790 tx=0x7f4dcc00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:07:04.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:04.068+0000 7f4dd693c700 1 --2- 172.21.15.42:0/1994241631 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4dd00a7740 0x7f4dd008df80 secure :-1 s=THROTTLE_DONE pgs=60 cs=0 l=1 rx=0x7f4dcc008790 tx=0x7f4dcc00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:07:04.089 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:04.068+0000 7f4dd593a700 1 -- 172.21.15.42:0/1994241631 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4dcc008bb0 con 0x7f4dd00a7740 2019-12-05T05:07:04.090 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:04.068+0000 7f4dd693c700 1 --2- 172.21.15.42:0/1994241631 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4dd00a7740 0x7f4dd008df80 secure :-1 s=THROTTLE_DONE pgs=60 cs=0 l=1 rx=0x7f4dcc008790 tx=0x7f4dcc00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:07:04.090 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:04.068+0000 7f4dd593a700 1 -- 172.21.15.42:0/1994241631 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dcc0087d0 con 0x7f4dd00a7740 2019-12-05T05:07:04.090 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:04.068+0000 7f4dd593a700 1 -- 172.21.15.42:0/1994241631 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4dcc011860 con 0x7f4dd00a7740 2019-12-05T05:07:04.091 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:04.068+0000 7f4dd839f700 1 -- 172.21.15.42:0/1994241631 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4dd00a7740 msgr2=0x7f4dd008df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:07:04.091 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:04.068+0000 7f4dd839f700 1 --2- 172.21.15.42:0/1994241631 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4dd00a7740 0x7f4dd008df80 secure :-1 s=READY pgs=60 cs=0 l=1 rx=0x7f4dcc008790 tx=0x7f4dcc00fc70).stop 2019-12-05T05:07:04.091 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:04.068+0000 7f4dd839f700 1 -- 172.21.15.42:0/1994241631 shutdown_connections 2019-12-05T05:07:04.092 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:04.068+0000 7f4dd839f700 1 --2- 172.21.15.42:0/1994241631 >> v2:172.21.15.40:3300/0 conn(0x7f4dd009fae0 0x7f4dd007eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:07:04.092 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:04.068+0000 7f4dd839f700 1 --2- 172.21.15.42:0/1994241631 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4dd00a7740 0x7f4dd008df80 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:07:04.092 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:04.068+0000 7f4dd839f700 1 -- 172.21.15.42:0/1994241631 shutdown_connections 2019-12-05T05:07:04.093 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:04.068+0000 7f4dd839f700 1 -- 172.21.15.42:0/1994241631 wait complete. 2019-12-05T05:07:04.093 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:04.068+0000 7f4dd839f700 1 -- 172.21.15.42:0/1994241631 >> 172.21.15.42:0/1994241631 conn(0x7f4dd00aabe0 msgr2=0x7f4dd00a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:07:04.093 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:04.068+0000 7f4dd839f700 1 Processor -- start 2019-12-05T05:07:04.093 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:04.068+0000 7f4dd839f700 1 -- start start 2019-12-05T05:07:04.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:04.068+0000 7f4dd839f700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4dd01243f0 0x7f4dd011f910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:07:04.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:04.068+0000 7f4dd839f700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4dd01386e0 con 0x7f4dd0130400 2019-12-05T05:07:04.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:04.068+0000 7f4dd839f700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4dd0139920 con 0x7f4dd0119480 2019-12-05T05:07:04.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:04.068+0000 7f4dd839f700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4dd0119990 con 0x7f4dd01243f0 2019-12-05T05:07:04.100 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:04.072+0000 7f4dd693c700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f4dd0119480 0x7f4dd007d890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:55484/0 (socket says 172.21.15.42:55484) 2019-12-05T05:07:04.100 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:04.072+0000 7f4dd693c700 1 -- 172.21.15.42:0/1176883443 learned_addr learned my addr 172.21.15.42:0/1176883443 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:07:04.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:04.072+0000 7f4dd613b700 1 --2- 172.21.15.42:0/1176883443 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4dd01243f0 0x7f4dd011f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:07:04.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:04.072+0000 7f4dc37fe700 1 -- 172.21.15.42:0/1176883443 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2069345226 0 0) 0x7f4dd0139920 con 0x7f4dd0119480 2019-12-05T05:07:04.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:04.072+0000 7f4dc37fe700 1 -- 172.21.15.42:0/1176883443 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4db80030b0 con 0x7f4dd0119480 2019-12-05T05:07:04.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:04.072+0000 7f4dc37fe700 1 -- 172.21.15.42:0/1176883443 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3714952260 0 0) 0x7f4dd01386e0 con 0x7f4dd0130400 2019-12-05T05:07:04.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:04.072+0000 7f4dc37fe700 1 -- 172.21.15.42:0/1176883443 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4dd0139920 con 0x7f4dd0130400 2019-12-05T05:07:04.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:04.072+0000 7f4dd613b700 1 -- 172.21.15.42:0/1176883443 >> v1:172.21.15.42:6789/0 conn(0x7f4dd0119480 legacy=0x7f4dd007d890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:07:04.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:04.072+0000 7f4dd613b700 1 -- 172.21.15.42:0/1176883443 >> v1:172.21.15.40:6789/0 conn(0x7f4dd0130400 legacy=0x7f4dd0130830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:07:04.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:04.072+0000 7f4dd613b700 1 -- 172.21.15.42:0/1176883443 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dd008eb70 con 0x7f4dd01243f0 2019-12-05T05:07:04.109 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:04.072+0000 7f4dd613b700 1 --2- 172.21.15.42:0/1176883443 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4dd01243f0 0x7f4dd011f910 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f4dc4004720 tx=0x7f4dc4010c10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:07:04.110 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:04.072+0000 7f4dd613b700 1 --2- 172.21.15.42:0/1176883443 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4dd01243f0 0x7f4dd011f910 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f4dc4004720 tx=0x7f4dc4010c10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:07:04.110 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:04.072+0000 7f4dd613b700 1 --2- 172.21.15.42:0/1176883443 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4dd01243f0 0x7f4dd011f910 secure :-1 s=READY pgs=61 cs=0 l=1 rx=0x7f4dc4004720 tx=0x7f4dc4010c10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:07:04.110 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:04.072+0000 7f4dd613b700 1 --2- 172.21.15.42:0/1176883443 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4dd01243f0 0x7f4dd011f910 secure :-1 s=THROTTLE_DONE pgs=61 cs=0 l=1 rx=0x7f4dc4004720 tx=0x7f4dc4010c10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:07:04.111 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:04.072+0000 7f4dd613b700 1 --2- 172.21.15.42:0/1176883443 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4dd01243f0 0x7f4dd011f910 secure :-1 s=THROTTLE_DONE pgs=61 cs=0 l=1 rx=0x7f4dc4004720 tx=0x7f4dc4010c10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:07:04.111 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:04.072+0000 7f4dc37fe700 1 -- 172.21.15.42:0/1176883443 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4dc40122d0 con 0x7f4dd01243f0 2019-12-05T05:07:04.111 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:04.072+0000 7f4dd613b700 1 --2- 172.21.15.42:0/1176883443 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4dd01243f0 0x7f4dd011f910 secure :-1 s=THROTTLE_DONE pgs=61 cs=0 l=1 rx=0x7f4dc4004720 tx=0x7f4dc4010c10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:07:04.112 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:04.072+0000 7f4dc37fe700 1 -- 172.21.15.42:0/1176883443 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dc4005bd0 con 0x7f4dd01243f0 2019-12-05T05:07:04.112 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:04.072+0000 7f4dd839f700 1 -- 172.21.15.42:0/1176883443 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4dd008ee40 con 0x7f4dd01243f0 2019-12-05T05:07:04.112 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:04.072+0000 7f4dc37fe700 1 -- 172.21.15.42:0/1176883443 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4dc40128b0 con 0x7f4dd01243f0 2019-12-05T05:07:04.112 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:04.072+0000 7f4dd839f700 1 -- 172.21.15.42:0/1176883443 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4dd008f3c0 con 0x7f4dd01243f0 2019-12-05T05:07:04.113 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:04.072+0000 7f4dd613b700 1 --2- 172.21.15.42:0/1176883443 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4dd01243f0 0x7f4dd011f910 secure :-1 s=THROTTLE_DONE pgs=61 cs=0 l=1 rx=0x7f4dc4004720 tx=0x7f4dc4010c10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:07:04.113 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:04.072+0000 7f4dc37fe700 1 -- 172.21.15.42:0/1176883443 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f4dc4012ab0 con 0x7f4dd01243f0 2019-12-05T05:07:04.113 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:04.072+0000 7f4dd613b700 1 --2- 172.21.15.42:0/1176883443 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4dd01243f0 0x7f4dd011f910 secure :-1 s=THROTTLE_DONE pgs=61 cs=0 l=1 rx=0x7f4dc4004720 tx=0x7f4dc4010c10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:07:04.113 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:04.076+0000 7f4dc37fe700 1 --2- 172.21.15.42:0/1176883443 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4db803b450 0x7f4db803d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:07:04.114 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:04.076+0000 7f4dc37fe700 1 -- 172.21.15.42:0/1176883443 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6145+0+0 (secure 0 0 0) 0x7f4dc404b530 con 0x7f4dd01243f0 2019-12-05T05:07:04.114 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:04.076+0000 7f4dd693c700 1 --2- 172.21.15.42:0/1176883443 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4db803b450 0x7f4db803d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:07:04.115 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:04.076+0000 7f4dd693c700 1 --2- 172.21.15.42:0/1176883443 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4db803b450 0x7f4db803d960 crc :-1 s=READY pgs=140 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:07:04.115 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:04.080+0000 7f4dd839f700 1 -- 172.21.15.42:0/1176883443 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4dd005d110 con 0x7f4dd01243f0 2019-12-05T05:07:04.115 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:04.088+0000 7f4dd613b700 1 --2- 172.21.15.42:0/1176883443 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4dd01243f0 0x7f4dd011f910 secure :-1 s=THROTTLE_DONE pgs=61 cs=0 l=1 rx=0x7f4dc4004720 tx=0x7f4dc4010c10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:07:04.115 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:04.088+0000 7f4dc37fe700 1 -- 172.21.15.42:0/1176883443 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f4dc40325b0 con 0x7f4dd01243f0 2019-12-05T05:07:04.262 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:04.256+0000 7f4dd839f700 1 -- 172.21.15.42:0/1176883443 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4dd005b8c0 con 0x7f4dd01243f0 2019-12-05T05:07:04.263 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:04.260+0000 7f4dd613b700 1 --2- 172.21.15.42:0/1176883443 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4dd01243f0 0x7f4dd011f910 secure :-1 s=THROTTLE_DONE pgs=61 cs=0 l=1 rx=0x7f4dc4004720 tx=0x7f4dc4010c10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:07:04.263 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:04.260+0000 7f4dc37fe700 1 -- 172.21.15.42:0/1176883443 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+2101 (secure 0 0 0) 0x7f4dc4032810 con 0x7f4dd01243f0 2019-12-05T05:07:04.265 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:07:04.265 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4831,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/90248709","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":90248709},{"type":"v1","addr":"172.21.15.40:6833","nonce":90248709}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":11}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:05:19.282511+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4354},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4354":{"gid":4354,"name":"a","rank":0,"incarnation":7,"state":"up:active","state_seq":20,"addr":"172.21.15.42:6835/2019200199","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2019200199},{"type":"v1","addr":"172.21.15.42:6835","nonce":2019200199}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:07:04.265 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 23 2019-12-05T05:07:04.267 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:04.264+0000 7f4dd839f700 1 -- 172.21.15.42:0/1176883443 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4db803b450 msgr2=0x7f4db803d960 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:07:04.268 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:04.264+0000 7f4dd839f700 1 --2- 172.21.15.42:0/1176883443 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4db803b450 0x7f4db803d960 crc :-1 s=READY pgs=140 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:07:04.269 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:04.264+0000 7f4dd839f700 1 -- 172.21.15.42:0/1176883443 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4dd01243f0 msgr2=0x7f4dd011f910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:07:04.269 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:04.264+0000 7f4dd839f700 1 --2- 172.21.15.42:0/1176883443 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4dd01243f0 0x7f4dd011f910 secure :-1 s=READY pgs=61 cs=0 l=1 rx=0x7f4dc4004720 tx=0x7f4dc4010c10).stop 2019-12-05T05:07:04.269 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:04.264+0000 7f4dd839f700 1 -- 172.21.15.42:0/1176883443 shutdown_connections 2019-12-05T05:07:04.269 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:04.264+0000 7f4dd839f700 1 --2- 172.21.15.42:0/1176883443 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4db803b450 0x7f4db803d960 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:07:04.270 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:04.264+0000 7f4dd839f700 1 --2- 172.21.15.42:0/1176883443 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4dd01243f0 0x7f4dd011f910 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:07:04.270 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:04.264+0000 7f4dd839f700 1 -- 172.21.15.42:0/1176883443 shutdown_connections 2019-12-05T05:07:04.270 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:04.264+0000 7f4dd839f700 1 -- 172.21.15.42:0/1176883443 wait complete. 2019-12-05T05:07:04.270 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:04.264+0000 7f4dd839f700 1 -- 172.21.15.42:0/1176883443 >> 172.21.15.42:0/1176883443 conn(0x7f4dd00aabe0 msgr2=0x7f4dd011d890 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:07:04.278 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T05:07:04.730 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:07:04.734 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:07:06.279 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:07:06.390 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:06.380+0000 7f9f4855b700 1 Processor -- start 2019-12-05T05:07:06.390 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:06.380+0000 7f9f4855b700 1 -- start start 2019-12-05T05:07:06.390 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:06.380+0000 7f9f4855b700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f9f400a7740 0x7f9f4008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:07:06.391 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:06.380+0000 7f9f4855b700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9f4007e720 0x7f9f4007eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:07:06.391 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:06.380+0000 7f9f4855b700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f400a0710 con 0x7f9f4009f2e0 2019-12-05T05:07:06.391 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:06.380+0000 7f9f4855b700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f400a0b00 con 0x7f9f400a7740 2019-12-05T05:07:06.391 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:06.380+0000 7f9f4855b700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f4008e5b0 con 0x7f9f4007e720 2019-12-05T05:07:06.391 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:06.380+0000 7f9f46af8700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f9f400a7740 0x7f9f4008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:07:06.392 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:06.380+0000 7f9f472f9700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f9f4009f2e0 0x7f9f4009f710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:55492/0 (socket says 172.21.15.42:55492) 2019-12-05T05:07:06.392 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:06.380+0000 7f9f472f9700 1 -- 172.21.15.42:0/849335407 learned_addr learned my addr 172.21.15.42:0/849335407 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:07:06.392 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:06.380+0000 7f9f462f7700 1 --2- 172.21.15.42:0/849335407 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9f4007e720 0x7f9f4007eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:07:06.392 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:06.380+0000 7f9f46af8700 1 -- 172.21.15.42:0/849335407 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9f4007e720 msgr2=0x7f9f4007eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:07:06.392 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:06.380+0000 7f9f46af8700 1 --2- 172.21.15.42:0/849335407 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9f4007e720 0x7f9f4007eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:07:06.393 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:06.380+0000 7f9f46af8700 1 -- 172.21.15.42:0/849335407 >> v1:172.21.15.42:6789/0 conn(0x7f9f4009f2e0 legacy=0x7f9f4009f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:07:06.393 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:06.380+0000 7f9f46af8700 1 -- 172.21.15.42:0/849335407 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f4007f0a0 con 0x7f9f400a7740 2019-12-05T05:07:06.393 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:06.380+0000 7f9f45af6700 1 -- 172.21.15.42:0/849335407 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2094859001 0 0) 0x7f9f400a0710 con 0x7f9f4009f2e0 2019-12-05T05:07:06.393 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:06.380+0000 7f9f46af8700 1 --2- 172.21.15.42:0/849335407 >> v2:172.21.15.42:3300/0 conn(0x7f9f400a7740 0x7f9f4008df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f9f3c008790 tx=0x7f9f3c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:07:06.399 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:06.384+0000 7f9f46af8700 1 --2- 172.21.15.42:0/849335407 >> v2:172.21.15.42:3300/0 conn(0x7f9f400a7740 0x7f9f4008df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f9f3c008790 tx=0x7f9f3c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:07:06.400 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:06.384+0000 7f9f46af8700 1 --2- 172.21.15.42:0/849335407 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9f400a7740 0x7f9f4008df80 secure :-1 s=READY pgs=296 cs=0 l=1 rx=0x7f9f3c008790 tx=0x7f9f3c00fc70).ready entity=mon.1 client_cookie=12cedd86cb2872f8 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:07:06.400 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:06.384+0000 7f9f46af8700 1 --2- 172.21.15.42:0/849335407 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9f400a7740 0x7f9f4008df80 secure :-1 s=THROTTLE_DONE pgs=296 cs=0 l=1 rx=0x7f9f3c008790 tx=0x7f9f3c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:07:06.400 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:06.384+0000 7f9f46af8700 1 --2- 172.21.15.42:0/849335407 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9f400a7740 0x7f9f4008df80 secure :-1 s=THROTTLE_DONE pgs=296 cs=0 l=1 rx=0x7f9f3c008790 tx=0x7f9f3c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:07:06.400 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:06.384+0000 7f9f45af6700 1 -- 172.21.15.42:0/849335407 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9f3c0112f0 con 0x7f9f400a7740 2019-12-05T05:07:06.400 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:06.384+0000 7f9f46af8700 1 --2- 172.21.15.42:0/849335407 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9f400a7740 0x7f9f4008df80 secure :-1 s=THROTTLE_DONE pgs=296 cs=0 l=1 rx=0x7f9f3c008790 tx=0x7f9f3c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:07:06.400 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:06.384+0000 7f9f45af6700 1 -- 172.21.15.42:0/849335407 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f3c008ad0 con 0x7f9f400a7740 2019-12-05T05:07:06.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:06.384+0000 7f9f45af6700 1 -- 172.21.15.42:0/849335407 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9f3c0118d0 con 0x7f9f400a7740 2019-12-05T05:07:06.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:06.384+0000 7f9f4855b700 1 -- 172.21.15.42:0/849335407 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9f400a7740 msgr2=0x7f9f4008df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:07:06.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:06.384+0000 7f9f4855b700 1 --2- 172.21.15.42:0/849335407 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9f400a7740 0x7f9f4008df80 secure :-1 s=READY pgs=296 cs=0 l=1 rx=0x7f9f3c008790 tx=0x7f9f3c00fc70).stop 2019-12-05T05:07:06.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:06.384+0000 7f9f4855b700 1 -- 172.21.15.42:0/849335407 shutdown_connections 2019-12-05T05:07:06.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:06.384+0000 7f9f4855b700 1 --2- 172.21.15.42:0/849335407 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9f400a7740 0x7f9f4008df80 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:07:06.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:06.384+0000 7f9f4855b700 1 --2- 172.21.15.42:0/849335407 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9f4007e720 0x7f9f4007eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:07:06.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:06.384+0000 7f9f4855b700 1 -- 172.21.15.42:0/849335407 shutdown_connections 2019-12-05T05:07:06.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:06.384+0000 7f9f4855b700 1 -- 172.21.15.42:0/849335407 wait complete. 2019-12-05T05:07:06.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:06.384+0000 7f9f4855b700 1 -- 172.21.15.42:0/849335407 >> 172.21.15.42:0/849335407 conn(0x7f9f400aabe0 msgr2=0x7f9f400a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:07:06.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:06.384+0000 7f9f4855b700 1 Processor -- start 2019-12-05T05:07:06.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:06.384+0000 7f9f4855b700 1 -- start start 2019-12-05T05:07:06.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:06.388+0000 7f9f4855b700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f9f400a2ac0 0x7f9f400a5ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:07:06.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:06.388+0000 7f9f4855b700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9f400a1ac0 0x7f9f4008f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:07:06.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:06.388+0000 7f9f4855b700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f400a6420 con 0x7f9f400a1690 2019-12-05T05:07:06.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:06.388+0000 7f9f4855b700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f4011bf50 con 0x7f9f400a2ac0 2019-12-05T05:07:06.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:06.388+0000 7f9f4855b700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f400a30e0 con 0x7f9f400a1ac0 2019-12-05T05:07:06.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:06.388+0000 7f9f46af8700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f9f400a2ac0 0x7f9f400a5ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:07:06.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:06.388+0000 7f9f472f9700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9f400a1ac0 0x7f9f4008f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:07:06.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:06.388+0000 7f9f46af8700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f9f400a2ac0 0x7f9f400a5ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.42:43272/0 (socket says 172.21.15.42:43272) 2019-12-05T05:07:06.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:06.388+0000 7f9f46af8700 1 -- 172.21.15.42:0/3272345996 learned_addr learned my addr 172.21.15.42:0/3272345996 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:07:06.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:06.388+0000 7f9f46af8700 1 -- 172.21.15.42:0/3272345996 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9f400a1ac0 msgr2=0x7f9f4008f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:07:06.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:06.388+0000 7f9f46af8700 1 --2- 172.21.15.42:0/3272345996 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9f400a1ac0 0x7f9f4008f370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:07:06.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:06.388+0000 7f9f46af8700 1 -- 172.21.15.42:0/3272345996 >> v1:172.21.15.40:6789/0 conn(0x7f9f400a1690 legacy=0x7f9f4008eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:07:06.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:06.388+0000 7f9f46af8700 1 -- 172.21.15.42:0/3272345996 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f4008f8c0 con 0x7f9f400a2ac0 2019-12-05T05:07:06.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:06.388+0000 7f9f46af8700 1 --2- 172.21.15.42:0/3272345996 >> v2:172.21.15.40:3300/0 conn(0x7f9f400a2ac0 0x7f9f400a5ed0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f9f3c008790 tx=0x7f9f3c011a10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:07:06.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:06.388+0000 7f9f46af8700 1 --2- 172.21.15.42:0/3272345996 >> v2:172.21.15.40:3300/0 conn(0x7f9f400a2ac0 0x7f9f400a5ed0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f9f3c008790 tx=0x7f9f3c011a10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:07:06.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:06.388+0000 7f9f46af8700 1 --2- 172.21.15.42:0/3272345996 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9f400a2ac0 0x7f9f400a5ed0 secure :-1 s=READY pgs=243 cs=0 l=1 rx=0x7f9f3c008790 tx=0x7f9f3c011a10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:07:06.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:06.388+0000 7f9f46af8700 1 --2- 172.21.15.42:0/3272345996 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9f400a2ac0 0x7f9f400a5ed0 secure :-1 s=THROTTLE_DONE pgs=243 cs=0 l=1 rx=0x7f9f3c008790 tx=0x7f9f3c011a10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:07:06.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:06.388+0000 7f9f46af8700 1 --2- 172.21.15.42:0/3272345996 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9f400a2ac0 0x7f9f400a5ed0 secure :-1 s=THROTTLE_DONE pgs=243 cs=0 l=1 rx=0x7f9f3c008790 tx=0x7f9f3c011a10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:07:06.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:06.388+0000 7f9f377fe700 1 -- 172.21.15.42:0/3272345996 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9f3c007150 con 0x7f9f400a2ac0 2019-12-05T05:07:06.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:06.388+0000 7f9f46af8700 1 --2- 172.21.15.42:0/3272345996 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9f400a2ac0 0x7f9f400a5ed0 secure :-1 s=THROTTLE_DONE pgs=243 cs=0 l=1 rx=0x7f9f3c008790 tx=0x7f9f3c011a10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:07:06.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:06.388+0000 7f9f377fe700 1 -- 172.21.15.42:0/3272345996 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f3c0114f0 con 0x7f9f400a2ac0 2019-12-05T05:07:06.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:06.388+0000 7f9f377fe700 1 -- 172.21.15.42:0/3272345996 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9f3c00f550 con 0x7f9f400a2ac0 2019-12-05T05:07:06.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:06.388+0000 7f9f4855b700 1 -- 172.21.15.42:0/3272345996 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f400a1ef0 con 0x7f9f400a2ac0 2019-12-05T05:07:06.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:06.388+0000 7f9f4855b700 1 -- 172.21.15.42:0/3272345996 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f40119c70 con 0x7f9f400a2ac0 2019-12-05T05:07:06.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:06.392+0000 7f9f46af8700 1 --2- 172.21.15.42:0/3272345996 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9f400a2ac0 0x7f9f400a5ed0 secure :-1 s=THROTTLE_DONE pgs=243 cs=0 l=1 rx=0x7f9f3c008790 tx=0x7f9f3c011a10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:07:06.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:06.392+0000 7f9f377fe700 1 -- 172.21.15.42:0/3272345996 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f9f3c005540 con 0x7f9f400a2ac0 2019-12-05T05:07:06.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:06.392+0000 7f9f46af8700 1 --2- 172.21.15.42:0/3272345996 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9f400a2ac0 0x7f9f400a5ed0 secure :-1 s=THROTTLE_DONE pgs=243 cs=0 l=1 rx=0x7f9f3c008790 tx=0x7f9f3c011a10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:07:06.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:06.392+0000 7f9f377fe700 1 --2- 172.21.15.42:0/3272345996 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9f2003ada0 0x7f9f2003d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:07:06.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:06.392+0000 7f9f377fe700 1 -- 172.21.15.42:0/3272345996 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6145+0+0 (secure 0 0 0) 0x7f9f3c04b9b0 con 0x7f9f400a2ac0 2019-12-05T05:07:06.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:06.392+0000 7f9f462f7700 1 --2- 172.21.15.42:0/3272345996 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9f2003ada0 0x7f9f2003d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:07:06.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:06.392+0000 7f9f462f7700 1 --2- 172.21.15.42:0/3272345996 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9f2003ada0 0x7f9f2003d2b0 crc :-1 s=READY pgs=141 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:07:06.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:06.396+0000 7f9f4855b700 1 -- 172.21.15.42:0/3272345996 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f4005d110 con 0x7f9f400a2ac0 2019-12-05T05:07:06.411 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:06.404+0000 7f9f46af8700 1 --2- 172.21.15.42:0/3272345996 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9f400a2ac0 0x7f9f400a5ed0 secure :-1 s=THROTTLE_DONE pgs=243 cs=0 l=1 rx=0x7f9f3c008790 tx=0x7f9f3c011a10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:07:06.411 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:06.404+0000 7f9f377fe700 1 -- 172.21.15.42:0/3272345996 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f9f3c032a30 con 0x7f9f400a2ac0 2019-12-05T05:07:06.589 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:06.576+0000 7f9f4855b700 1 -- 172.21.15.42:0/3272345996 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9f4005b8c0 con 0x7f9f400a2ac0 2019-12-05T05:07:06.604 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:06.576+0000 7f9f46af8700 1 --2- 172.21.15.42:0/3272345996 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9f400a2ac0 0x7f9f400a5ed0 secure :-1 s=THROTTLE_DONE pgs=243 cs=0 l=1 rx=0x7f9f3c008790 tx=0x7f9f3c011a10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:07:06.604 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:06.576+0000 7f9f377fe700 1 -- 172.21.15.42:0/3272345996 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+2101 (secure 0 0 0) 0x7f9f3c032c90 con 0x7f9f400a2ac0 2019-12-05T05:07:06.605 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 23 2019-12-05T05:07:06.605 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:06.580+0000 7f9f4855b700 1 -- 172.21.15.42:0/3272345996 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9f2003ada0 msgr2=0x7f9f2003d2b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:07:06.605 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:06.580+0000 7f9f4855b700 1 --2- 172.21.15.42:0/3272345996 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9f2003ada0 0x7f9f2003d2b0 crc :-1 s=READY pgs=141 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:07:06.605 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:06.580+0000 7f9f4855b700 1 -- 172.21.15.42:0/3272345996 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9f400a2ac0 msgr2=0x7f9f400a5ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:07:06.605 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:06.580+0000 7f9f4855b700 1 --2- 172.21.15.42:0/3272345996 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9f400a2ac0 0x7f9f400a5ed0 secure :-1 s=READY pgs=243 cs=0 l=1 rx=0x7f9f3c008790 tx=0x7f9f3c011a10).stop 2019-12-05T05:07:06.605 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:06.580+0000 7f9f4855b700 1 -- 172.21.15.42:0/3272345996 shutdown_connections 2019-12-05T05:07:06.606 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:06.580+0000 7f9f4855b700 1 --2- 172.21.15.42:0/3272345996 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9f2003ada0 0x7f9f2003d2b0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:07:06.606 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:06.580+0000 7f9f4855b700 1 --2- 172.21.15.42:0/3272345996 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9f400a1ac0 0x7f9f4008f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:07:06.606 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:06.580+0000 7f9f4855b700 1 --2- 172.21.15.42:0/3272345996 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9f400a2ac0 0x7f9f400a5ed0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:07:06.606 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:06.580+0000 7f9f4855b700 1 -- 172.21.15.42:0/3272345996 shutdown_connections 2019-12-05T05:07:06.606 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:06.580+0000 7f9f4855b700 1 -- 172.21.15.42:0/3272345996 wait complete. 2019-12-05T05:07:06.606 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:06.580+0000 7f9f4855b700 1 -- 172.21.15.42:0/3272345996 >> 172.21.15.42:0/3272345996 conn(0x7f9f4011c610 msgr2=0x7f9f4009f380 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:07:06.606 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:07:06.607 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4831,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/90248709","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":90248709},{"type":"v1","addr":"172.21.15.40:6833","nonce":90248709}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":11}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:05:19.282511+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4354},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4354":{"gid":4354,"name":"a","rank":0,"incarnation":7,"state":"up:active","state_seq":20,"addr":"172.21.15.42:6835/2019200199","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2019200199},{"type":"v1","addr":"172.21.15.42:6835","nonce":2019200199}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:07:06.608 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T05:07:08.609 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:07:08.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:08.708+0000 7fcc53da7700 1 Processor -- start 2019-12-05T05:07:08.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:08.712+0000 7fcc53da7700 1 -- start start 2019-12-05T05:07:08.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:08.712+0000 7fcc53da7700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcc4c0a7740 0x7fcc4c08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:07:08.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:08.712+0000 7fcc53da7700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fcc4c07e720 0x7fcc4c07eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:07:08.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:08.712+0000 7fcc53da7700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc4c0a0710 con 0x7fcc4c09f2e0 2019-12-05T05:07:08.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:08.712+0000 7fcc53da7700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc4c0a0b00 con 0x7fcc4c07e720 2019-12-05T05:07:08.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:08.712+0000 7fcc53da7700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc4c08e5b0 con 0x7fcc4c0a7740 2019-12-05T05:07:08.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:08.712+0000 7fcc52344700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcc4c0a7740 0x7fcc4c08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:07:08.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:08.712+0000 7fcc52b45700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fcc4c09f2e0 0x7fcc4c09f710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:55512/0 (socket says 172.21.15.42:55512) 2019-12-05T05:07:08.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:08.712+0000 7fcc52b45700 1 -- 172.21.15.42:0/2785824326 learned_addr learned my addr 172.21.15.42:0/2785824326 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:07:08.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:08.712+0000 7fcc51b43700 1 --2- 172.21.15.42:0/2785824326 >> v2:172.21.15.40:3300/0 conn(0x7fcc4c07e720 0x7fcc4c07eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:07:08.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:08.712+0000 7fcc51342700 1 -- 172.21.15.42:0/2785824326 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 326594500 0 0) 0x7fcc4c0a0710 con 0x7fcc4c09f2e0 2019-12-05T05:07:08.724 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:08.712+0000 7fcc51342700 1 -- 172.21.15.42:0/2785824326 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc380030b0 con 0x7fcc4c09f2e0 2019-12-05T05:07:08.724 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:08.712+0000 7fcc52344700 1 -- 172.21.15.42:0/2785824326 >> v2:172.21.15.40:3300/0 conn(0x7fcc4c07e720 msgr2=0x7fcc4c07eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:07:08.725 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:08.712+0000 7fcc52344700 1 --2- 172.21.15.42:0/2785824326 >> v2:172.21.15.40:3300/0 conn(0x7fcc4c07e720 0x7fcc4c07eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:07:08.725 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:08.712+0000 7fcc52344700 1 -- 172.21.15.42:0/2785824326 >> v1:172.21.15.42:6789/0 conn(0x7fcc4c09f2e0 legacy=0x7fcc4c09f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:07:08.725 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:08.712+0000 7fcc52344700 1 -- 172.21.15.42:0/2785824326 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc4c07f0a0 con 0x7fcc4c0a7740 2019-12-05T05:07:08.725 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:08.712+0000 7fcc51b43700 1 --2- 172.21.15.42:0/2785824326 >> v2:172.21.15.40:3300/0 conn(0x7fcc4c07e720 0x7fcc4c07eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:07:08.726 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:08.712+0000 7fcc52344700 1 --2- 172.21.15.42:0/2785824326 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcc4c0a7740 0x7fcc4c08df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fcc48008790 tx=0x7fcc4800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:07:08.726 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:08.712+0000 7fcc52344700 1 --2- 172.21.15.42:0/2785824326 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcc4c0a7740 0x7fcc4c08df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fcc48008790 tx=0x7fcc4800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:07:08.727 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:08.712+0000 7fcc52344700 1 --2- 172.21.15.42:0/2785824326 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcc4c0a7740 0x7fcc4c08df80 secure :-1 s=READY pgs=62 cs=0 l=1 rx=0x7fcc48008790 tx=0x7fcc4800fc70).ready entity=mon.2 client_cookie=6f28698fcfb4ede4 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:07:08.727 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:08.712+0000 7fcc52344700 1 --2- 172.21.15.42:0/2785824326 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcc4c0a7740 0x7fcc4c08df80 secure :-1 s=THROTTLE_DONE pgs=62 cs=0 l=1 rx=0x7fcc48008790 tx=0x7fcc4800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:07:08.727 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:08.712+0000 7fcc52344700 1 --2- 172.21.15.42:0/2785824326 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcc4c0a7740 0x7fcc4c08df80 secure :-1 s=THROTTLE_DONE pgs=62 cs=0 l=1 rx=0x7fcc48008790 tx=0x7fcc4800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:07:08.728 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:08.712+0000 7fcc51342700 1 -- 172.21.15.42:0/2785824326 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcc480112f0 con 0x7fcc4c0a7740 2019-12-05T05:07:08.729 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:08.712+0000 7fcc52344700 1 --2- 172.21.15.42:0/2785824326 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcc4c0a7740 0x7fcc4c08df80 secure :-1 s=THROTTLE_DONE pgs=62 cs=0 l=1 rx=0x7fcc48008790 tx=0x7fcc4800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:07:08.729 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:08.712+0000 7fcc51342700 1 -- 172.21.15.42:0/2785824326 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc48008ad0 con 0x7fcc4c0a7740 2019-12-05T05:07:08.729 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:08.712+0000 7fcc51342700 1 -- 172.21.15.42:0/2785824326 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcc480118d0 con 0x7fcc4c0a7740 2019-12-05T05:07:08.729 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:08.712+0000 7fcc53da7700 1 -- 172.21.15.42:0/2785824326 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcc4c0a7740 msgr2=0x7fcc4c08df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:07:08.730 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:08.712+0000 7fcc53da7700 1 --2- 172.21.15.42:0/2785824326 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcc4c0a7740 0x7fcc4c08df80 secure :-1 s=READY pgs=62 cs=0 l=1 rx=0x7fcc48008790 tx=0x7fcc4800fc70).stop 2019-12-05T05:07:08.730 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:08.716+0000 7fcc53da7700 1 -- 172.21.15.42:0/2785824326 shutdown_connections 2019-12-05T05:07:08.730 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:08.716+0000 7fcc53da7700 1 --2- 172.21.15.42:0/2785824326 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcc4c0a7740 0x7fcc4c08df80 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:07:08.731 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:08.716+0000 7fcc53da7700 1 --2- 172.21.15.42:0/2785824326 >> v2:172.21.15.40:3300/0 conn(0x7fcc4c07e720 0x7fcc4c07eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:07:08.731 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:08.716+0000 7fcc53da7700 1 -- 172.21.15.42:0/2785824326 shutdown_connections 2019-12-05T05:07:08.731 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:08.716+0000 7fcc53da7700 1 -- 172.21.15.42:0/2785824326 wait complete. 2019-12-05T05:07:08.731 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:08.716+0000 7fcc53da7700 1 -- 172.21.15.42:0/2785824326 >> 172.21.15.42:0/2785824326 conn(0x7fcc4c0aabe0 msgr2=0x7fcc4c0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:07:08.735 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:08.716+0000 7fcc53da7700 1 Processor -- start 2019-12-05T05:07:08.735 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:08.716+0000 7fcc53da7700 1 -- start start 2019-12-05T05:07:08.735 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:08.720+0000 7fcc53da7700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fcc4c116500 0x7fcc4c117e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:07:08.735 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:08.720+0000 7fcc53da7700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fcc4c0702c0 0x7fcc4c0706f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:07:08.736 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:08.720+0000 7fcc53da7700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc4c117310 con 0x7fcc4c11b860 2019-12-05T05:07:08.736 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:08.720+0000 7fcc53da7700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc4c117540 con 0x7fcc4c0702c0 2019-12-05T05:07:08.736 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:08.720+0000 7fcc53da7700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc4c11a1a0 con 0x7fcc4c116500 2019-12-05T05:07:08.736 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:08.720+0000 7fcc52344700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fcc4c116500 0x7fcc4c117e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:07:08.736 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:08.720+0000 7fcc52b45700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fcc4c0702c0 0x7fcc4c0706f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:07:08.736 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:08.720+0000 7fcc52344700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fcc4c116500 0x7fcc4c117e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:55476/0 (socket says 172.21.15.42:55476) 2019-12-05T05:07:08.737 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:08.720+0000 7fcc51b43700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fcc4c11b860 0x7fcc4c11afb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:55516/0 (socket says 172.21.15.42:55516) 2019-12-05T05:07:08.737 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:08.720+0000 7fcc52344700 1 -- 172.21.15.42:0/3967512066 learned_addr learned my addr 172.21.15.42:0/3967512066 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:07:08.737 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:08.720+0000 7fcc52344700 1 -- 172.21.15.42:0/3967512066 >> v2:172.21.15.40:3300/0 conn(0x7fcc4c0702c0 msgr2=0x7fcc4c0706f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:07:08.737 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:08.720+0000 7fcc52344700 1 --2- 172.21.15.42:0/3967512066 >> v2:172.21.15.40:3300/0 conn(0x7fcc4c0702c0 0x7fcc4c0706f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:07:08.737 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:08.720+0000 7fcc52344700 1 -- 172.21.15.42:0/3967512066 >> v1:172.21.15.42:6789/0 conn(0x7fcc4c11b860 legacy=0x7fcc4c11afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:07:08.737 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:08.720+0000 7fcc52344700 1 -- 172.21.15.42:0/3967512066 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc4c070c40 con 0x7fcc4c116500 2019-12-05T05:07:08.737 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:08.720+0000 7fcc52344700 1 --2- 172.21.15.42:0/3967512066 >> v2:172.21.15.42:3300/0 conn(0x7fcc4c116500 0x7fcc4c117e60 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fcc48008790 tx=0x7fcc48011990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:07:08.738 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:08.720+0000 7fcc52344700 1 --2- 172.21.15.42:0/3967512066 >> v2:172.21.15.42:3300/0 conn(0x7fcc4c116500 0x7fcc4c117e60 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fcc48008790 tx=0x7fcc48011990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:07:08.738 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:08.720+0000 7fcc52344700 1 --2- 172.21.15.42:0/3967512066 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcc4c116500 0x7fcc4c117e60 secure :-1 s=READY pgs=299 cs=0 l=1 rx=0x7fcc48008790 tx=0x7fcc48011990).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:07:08.738 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:08.720+0000 7fcc52344700 1 --2- 172.21.15.42:0/3967512066 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcc4c116500 0x7fcc4c117e60 secure :-1 s=THROTTLE_DONE pgs=299 cs=0 l=1 rx=0x7fcc48008790 tx=0x7fcc48011990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:07:08.738 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:08.720+0000 7fcc42ffd700 1 -- 172.21.15.42:0/3967512066 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcc48007150 con 0x7fcc4c116500 2019-12-05T05:07:08.738 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:08.720+0000 7fcc52344700 1 --2- 172.21.15.42:0/3967512066 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcc4c116500 0x7fcc4c117e60 secure :-1 s=THROTTLE_DONE pgs=299 cs=0 l=1 rx=0x7fcc48008790 tx=0x7fcc48011990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:07:08.738 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:08.720+0000 7fcc52344700 1 --2- 172.21.15.42:0/3967512066 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcc4c116500 0x7fcc4c117e60 secure :-1 s=THROTTLE_DONE pgs=299 cs=0 l=1 rx=0x7fcc48008790 tx=0x7fcc48011990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:07:08.739 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:08.720+0000 7fcc42ffd700 1 -- 172.21.15.42:0/3967512066 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc480114f0 con 0x7fcc4c116500 2019-12-05T05:07:08.739 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:08.720+0000 7fcc53da7700 1 -- 172.21.15.42:0/3967512066 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc4c070f10 con 0x7fcc4c116500 2019-12-05T05:07:08.739 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:08.720+0000 7fcc42ffd700 1 -- 172.21.15.42:0/3967512066 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcc48008cc0 con 0x7fcc4c116500 2019-12-05T05:07:08.739 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:08.720+0000 7fcc53da7700 1 -- 172.21.15.42:0/3967512066 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc4c077530 con 0x7fcc4c116500 2019-12-05T05:07:08.739 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:08.720+0000 7fcc52344700 1 --2- 172.21.15.42:0/3967512066 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcc4c116500 0x7fcc4c117e60 secure :-1 s=THROTTLE_DONE pgs=299 cs=0 l=1 rx=0x7fcc48008790 tx=0x7fcc48011990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:07:08.740 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:08.724+0000 7fcc42ffd700 1 -- 172.21.15.42:0/3967512066 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fcc48005540 con 0x7fcc4c116500 2019-12-05T05:07:08.740 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:08.724+0000 7fcc52344700 1 --2- 172.21.15.42:0/3967512066 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcc4c116500 0x7fcc4c117e60 secure :-1 s=THROTTLE_DONE pgs=299 cs=0 l=1 rx=0x7fcc48008790 tx=0x7fcc48011990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:07:08.740 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:08.724+0000 7fcc42ffd700 1 --2- 172.21.15.42:0/3967512066 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fcc3403add0 0x7fcc3403d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:07:08.740 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:08.724+0000 7fcc42ffd700 1 -- 172.21.15.42:0/3967512066 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6203+0+0 (secure 0 0 0) 0x7fcc4808f630 con 0x7fcc4c116500 2019-12-05T05:07:08.740 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:08.724+0000 7fcc51b43700 1 --2- 172.21.15.42:0/3967512066 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fcc3403add0 0x7fcc3403d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:07:08.740 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:08.724+0000 7fcc51b43700 1 --2- 172.21.15.42:0/3967512066 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fcc3403add0 0x7fcc3403d2e0 crc :-1 s=READY pgs=142 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:07:08.740 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:08.724+0000 7fcc53da7700 1 -- 172.21.15.42:0/3967512066 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc4c05d110 con 0x7fcc4c116500 2019-12-05T05:07:08.741 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:08.736+0000 7fcc52344700 1 --2- 172.21.15.42:0/3967512066 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcc4c116500 0x7fcc4c117e60 secure :-1 s=THROTTLE_DONE pgs=299 cs=0 l=1 rx=0x7fcc48008790 tx=0x7fcc48011990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:07:08.741 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:08.736+0000 7fcc42ffd700 1 -- 172.21.15.42:0/3967512066 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fcc48007690 con 0x7fcc4c116500 2019-12-05T05:07:08.908 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:08.904+0000 7fcc53da7700 1 -- 172.21.15.42:0/3967512066 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcc4c05b8c0 con 0x7fcc4c116500 2019-12-05T05:07:08.911 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:08.904+0000 7fcc52344700 1 --2- 172.21.15.42:0/3967512066 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcc4c116500 0x7fcc4c117e60 secure :-1 s=THROTTLE_DONE pgs=299 cs=0 l=1 rx=0x7fcc48008790 tx=0x7fcc48011990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:07:08.911 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:08.904+0000 7fcc42ffd700 1 -- 172.21.15.42:0/3967512066 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+1765 (secure 0 0 0) 0x7fcc480766b0 con 0x7fcc4c116500 2019-12-05T05:07:08.911 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 24 2019-12-05T05:07:08.912 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:07:08.912 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:07:08.209582+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4831},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4831":{"gid":4831,"name":"b","rank":0,"incarnation":24,"state":"up:replay","state_seq":2,"addr":"172.21.15.40:6833/90248709","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":90248709},{"type":"v1","addr":"172.21.15.40:6833","nonce":90248709}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:07:08.914 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:08.908+0000 7fcc53da7700 1 -- 172.21.15.42:0/3967512066 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fcc3403add0 msgr2=0x7fcc3403d2e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:07:08.914 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:08.908+0000 7fcc53da7700 1 --2- 172.21.15.42:0/3967512066 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fcc3403add0 0x7fcc3403d2e0 crc :-1 s=READY pgs=142 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:07:08.915 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:08.908+0000 7fcc53da7700 1 -- 172.21.15.42:0/3967512066 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcc4c116500 msgr2=0x7fcc4c117e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:07:08.915 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:08.908+0000 7fcc53da7700 1 --2- 172.21.15.42:0/3967512066 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcc4c116500 0x7fcc4c117e60 secure :-1 s=READY pgs=299 cs=0 l=1 rx=0x7fcc48008790 tx=0x7fcc48011990).stop 2019-12-05T05:07:08.915 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:08.908+0000 7fcc53da7700 1 -- 172.21.15.42:0/3967512066 shutdown_connections 2019-12-05T05:07:08.915 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:08.908+0000 7fcc53da7700 1 --2- 172.21.15.42:0/3967512066 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fcc3403add0 0x7fcc3403d2e0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:07:08.916 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:08.908+0000 7fcc53da7700 1 --2- 172.21.15.42:0/3967512066 >> v2:172.21.15.40:3300/0 conn(0x7fcc4c0702c0 0x7fcc4c0706f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:07:08.916 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:08.908+0000 7fcc53da7700 1 --2- 172.21.15.42:0/3967512066 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcc4c116500 0x7fcc4c117e60 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:07:08.916 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:08.908+0000 7fcc53da7700 1 -- 172.21.15.42:0/3967512066 shutdown_connections 2019-12-05T05:07:08.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:08.908+0000 7fcc53da7700 1 -- 172.21.15.42:0/3967512066 wait complete. 2019-12-05T05:07:08.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:08.908+0000 7fcc53da7700 1 -- 172.21.15.42:0/3967512066 >> 172.21.15.42:0/3967512066 conn(0x7fcc4c0a82a0 msgr2=0x7fcc4c0a9790 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:07:08.923 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a down, removed from mdsmap 2019-12-05T05:07:08.924 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2019-12-05T05:07:08.924 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:07:09.032 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:09.016+0000 7fc125468700 1 Processor -- start 2019-12-05T05:07:09.032 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:09.016+0000 7fc125468700 1 -- start start 2019-12-05T05:07:09.032 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:09.020+0000 7fc125468700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc1200a7740 0x7fc12008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:07:09.033 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:09.020+0000 7fc125468700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc12007e720 0x7fc12007eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:07:09.033 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:09.020+0000 7fc125468700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1200a0710 con 0x7fc12009f2e0 2019-12-05T05:07:09.033 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:09.020+0000 7fc125468700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1200a0b00 con 0x7fc1200a7740 2019-12-05T05:07:09.033 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:09.020+0000 7fc125468700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc12008e5b0 con 0x7fc12007e720 2019-12-05T05:07:09.033 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:09.020+0000 7fc11effd700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc12007e720 0x7fc12007eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:07:09.033 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:09.020+0000 7fc11ffff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fc12009f2e0 0x7fc12009f710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:55526/0 (socket says 172.21.15.42:55526) 2019-12-05T05:07:09.033 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:09.020+0000 7fc11ffff700 1 -- 172.21.15.42:0/1806497565 learned_addr learned my addr 172.21.15.42:0/1806497565 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:07:09.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:09.020+0000 7fc11f7fe700 1 --2- 172.21.15.42:0/1806497565 >> v2:172.21.15.40:3300/0 conn(0x7fc1200a7740 0x7fc12008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:07:09.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:09.020+0000 7fc11effd700 1 -- 172.21.15.42:0/1806497565 >> v2:172.21.15.40:3300/0 conn(0x7fc1200a7740 msgr2=0x7fc12008df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:07:09.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:09.020+0000 7fc11effd700 1 --2- 172.21.15.42:0/1806497565 >> v2:172.21.15.40:3300/0 conn(0x7fc1200a7740 0x7fc12008df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:07:09.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:09.020+0000 7fc11effd700 1 -- 172.21.15.42:0/1806497565 >> v1:172.21.15.42:6789/0 conn(0x7fc12009f2e0 legacy=0x7fc12009f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:07:09.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:09.020+0000 7fc11effd700 1 -- 172.21.15.42:0/1806497565 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc12007f0a0 con 0x7fc12007e720 2019-12-05T05:07:09.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:09.020+0000 7fc11effd700 1 --2- 172.21.15.42:0/1806497565 >> v2:172.21.15.42:3300/0 conn(0x7fc12007e720 0x7fc12007eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fc108002f30 tx=0x7fc10800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:07:09.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:09.020+0000 7fc11effd700 1 --2- 172.21.15.42:0/1806497565 >> v2:172.21.15.42:3300/0 conn(0x7fc12007e720 0x7fc12007eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fc108002f30 tx=0x7fc10800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:07:09.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:09.020+0000 7fc11effd700 1 --2- 172.21.15.42:0/1806497565 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc12007e720 0x7fc12007eb50 secure :-1 s=READY pgs=301 cs=0 l=1 rx=0x7fc108002f30 tx=0x7fc10800fc70).ready entity=mon.1 client_cookie=cfb7a2f6037c62aa server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:07:09.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:09.020+0000 7fc11effd700 1 --2- 172.21.15.42:0/1806497565 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc12007e720 0x7fc12007eb50 secure :-1 s=THROTTLE_DONE pgs=301 cs=0 l=1 rx=0x7fc108002f30 tx=0x7fc10800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:07:09.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:09.020+0000 7fc11e7fc700 1 -- 172.21.15.42:0/1806497565 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc1080112b0 con 0x7fc12007e720 2019-12-05T05:07:09.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:09.020+0000 7fc11effd700 1 --2- 172.21.15.42:0/1806497565 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc12007e720 0x7fc12007eb50 secure :-1 s=THROTTLE_DONE pgs=301 cs=0 l=1 rx=0x7fc108002f30 tx=0x7fc10800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:07:09.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:09.020+0000 7fc11e7fc700 1 -- 172.21.15.42:0/1806497565 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc108008ad0 con 0x7fc12007e720 2019-12-05T05:07:09.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:09.020+0000 7fc11effd700 1 --2- 172.21.15.42:0/1806497565 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc12007e720 0x7fc12007eb50 secure :-1 s=THROTTLE_DONE pgs=301 cs=0 l=1 rx=0x7fc108002f30 tx=0x7fc10800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:07:09.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:09.020+0000 7fc125468700 1 -- 172.21.15.42:0/1806497565 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc12007e720 msgr2=0x7fc12007eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:07:09.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:09.020+0000 7fc11e7fc700 1 -- 172.21.15.42:0/1806497565 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc108011890 con 0x7fc12007e720 2019-12-05T05:07:09.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:09.020+0000 7fc125468700 1 --2- 172.21.15.42:0/1806497565 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc12007e720 0x7fc12007eb50 secure :-1 s=READY pgs=301 cs=0 l=1 rx=0x7fc108002f30 tx=0x7fc10800fc70).stop 2019-12-05T05:07:09.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:09.020+0000 7fc125468700 1 -- 172.21.15.42:0/1806497565 shutdown_connections 2019-12-05T05:07:09.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:09.020+0000 7fc125468700 1 --2- 172.21.15.42:0/1806497565 >> v2:172.21.15.40:3300/0 conn(0x7fc1200a7740 0x7fc12008df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:07:09.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:09.020+0000 7fc125468700 1 --2- 172.21.15.42:0/1806497565 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc12007e720 0x7fc12007eb50 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:07:09.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:09.020+0000 7fc125468700 1 -- 172.21.15.42:0/1806497565 shutdown_connections 2019-12-05T05:07:09.040 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:09.020+0000 7fc125468700 1 -- 172.21.15.42:0/1806497565 wait complete. 2019-12-05T05:07:09.040 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:09.020+0000 7fc125468700 1 -- 172.21.15.42:0/1806497565 >> 172.21.15.42:0/1806497565 conn(0x7fc1200aabe0 msgr2=0x7fc1200a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:07:09.040 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:09.024+0000 7fc125468700 1 Processor -- start 2019-12-05T05:07:09.040 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:09.024+0000 7fc125468700 1 -- start start 2019-12-05T05:07:09.040 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:09.024+0000 7fc125468700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc12011be50 0x7fc120117d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:07:09.041 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:09.024+0000 7fc125468700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc1200a5930 0x7fc120130840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:07:09.041 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:09.024+0000 7fc125468700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1200a6610 con 0x7fc1200a5d60 2019-12-05T05:07:09.041 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:09.024+0000 7fc125468700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1200a5160 con 0x7fc12011be50 2019-12-05T05:07:09.041 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:09.024+0000 7fc125468700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc120118610 con 0x7fc1200a5930 2019-12-05T05:07:09.041 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:09.024+0000 7fc11f7fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc12011be50 0x7fc120117d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:07:09.041 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:09.024+0000 7fc11f7fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc12011be50 0x7fc120117d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:55490/0 (socket says 172.21.15.42:55490) 2019-12-05T05:07:09.041 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:09.024+0000 7fc11f7fe700 1 -- 172.21.15.42:0/2341433115 learned_addr learned my addr 172.21.15.42:0/2341433115 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:07:09.042 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:09.024+0000 7fc11effd700 1 --2- 172.21.15.42:0/2341433115 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc1200a5930 0x7fc120130840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:07:09.042 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:09.024+0000 7fc11f7fe700 1 -- 172.21.15.42:0/2341433115 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc1200a5930 msgr2=0x7fc120130840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:07:09.042 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:09.024+0000 7fc11f7fe700 1 --2- 172.21.15.42:0/2341433115 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc1200a5930 0x7fc120130840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:07:09.042 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:09.024+0000 7fc11f7fe700 1 -- 172.21.15.42:0/2341433115 >> v1:172.21.15.42:6789/0 conn(0x7fc1200a5d60 legacy=0x7fc12007c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:07:09.042 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:09.024+0000 7fc11f7fe700 1 -- 172.21.15.42:0/2341433115 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc120118250 con 0x7fc12011be50 2019-12-05T05:07:09.042 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:09.024+0000 7fc11f7fe700 1 --2- 172.21.15.42:0/2341433115 >> v2:172.21.15.42:3300/0 conn(0x7fc12011be50 0x7fc120117d00 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fc1200a5320 tx=0x7fc1100130f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:07:09.043 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:09.024+0000 7fc11f7fe700 1 --2- 172.21.15.42:0/2341433115 >> v2:172.21.15.42:3300/0 conn(0x7fc12011be50 0x7fc120117d00 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fc1200a5320 tx=0x7fc1100130f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:07:09.043 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:09.024+0000 7fc11f7fe700 1 --2- 172.21.15.42:0/2341433115 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc12011be50 0x7fc120117d00 secure :-1 s=READY pgs=303 cs=0 l=1 rx=0x7fc1200a5320 tx=0x7fc1100130f0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:07:09.043 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:09.024+0000 7fc11f7fe700 1 --2- 172.21.15.42:0/2341433115 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc12011be50 0x7fc120117d00 secure :-1 s=THROTTLE_DONE pgs=303 cs=0 l=1 rx=0x7fc1200a5320 tx=0x7fc1100130f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:07:09.043 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:09.024+0000 7fc11f7fe700 1 --2- 172.21.15.42:0/2341433115 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc12011be50 0x7fc120117d00 secure :-1 s=THROTTLE_DONE pgs=303 cs=0 l=1 rx=0x7fc1200a5320 tx=0x7fc1100130f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:07:09.043 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:09.024+0000 7fc117fff700 1 -- 172.21.15.42:0/2341433115 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc110014b20 con 0x7fc12011be50 2019-12-05T05:07:09.043 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:09.024+0000 7fc11f7fe700 1 --2- 172.21.15.42:0/2341433115 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc12011be50 0x7fc120117d00 secure :-1 s=THROTTLE_DONE pgs=303 cs=0 l=1 rx=0x7fc1200a5320 tx=0x7fc1100130f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:07:09.043 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:09.024+0000 7fc117fff700 1 -- 172.21.15.42:0/2341433115 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc110010160 con 0x7fc12011be50 2019-12-05T05:07:09.044 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:09.024+0000 7fc117fff700 1 -- 172.21.15.42:0/2341433115 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc110005ab0 con 0x7fc12011be50 2019-12-05T05:07:09.044 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:09.024+0000 7fc125468700 1 -- 172.21.15.42:0/2341433115 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc120130d90 con 0x7fc12011be50 2019-12-05T05:07:09.044 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:09.024+0000 7fc125468700 1 -- 172.21.15.42:0/2341433115 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc12007d090 con 0x7fc12011be50 2019-12-05T05:07:09.044 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:09.028+0000 7fc11f7fe700 1 --2- 172.21.15.42:0/2341433115 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc12011be50 0x7fc120117d00 secure :-1 s=THROTTLE_DONE pgs=303 cs=0 l=1 rx=0x7fc1200a5320 tx=0x7fc1100130f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:07:09.044 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:09.028+0000 7fc117fff700 1 -- 172.21.15.42:0/2341433115 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fc110007180 con 0x7fc12011be50 2019-12-05T05:07:09.044 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:09.028+0000 7fc11f7fe700 1 --2- 172.21.15.42:0/2341433115 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc12011be50 0x7fc120117d00 secure :-1 s=THROTTLE_DONE pgs=303 cs=0 l=1 rx=0x7fc1200a5320 tx=0x7fc1100130f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:07:09.044 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:09.028+0000 7fc117fff700 1 --2- 172.21.15.42:0/2341433115 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc10403ad80 0x7fc10403d290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:07:09.045 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:09.028+0000 7fc117fff700 1 -- 172.21.15.42:0/2341433115 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6203+0+0 (secure 0 0 0) 0x7fc11004d1c0 con 0x7fc12011be50 2019-12-05T05:07:09.045 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:09.028+0000 7fc11effd700 1 --2- 172.21.15.42:0/2341433115 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc10403ad80 0x7fc10403d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:07:09.045 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:09.028+0000 7fc11effd700 1 --2- 172.21.15.42:0/2341433115 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc10403ad80 0x7fc10403d290 crc :-1 s=READY pgs=143 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:07:09.045 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:09.032+0000 7fc125468700 1 -- 172.21.15.42:0/2341433115 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc12005d110 con 0x7fc12011be50 2019-12-05T05:07:09.046 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:09.044+0000 7fc11f7fe700 1 --2- 172.21.15.42:0/2341433115 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc12011be50 0x7fc120117d00 secure :-1 s=THROTTLE_DONE pgs=303 cs=0 l=1 rx=0x7fc1200a5320 tx=0x7fc1100130f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:07:09.046 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:09.044+0000 7fc117fff700 1 -- 172.21.15.42:0/2341433115 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fc110034240 con 0x7fc12011be50 2019-12-05T05:07:09.218 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:09.212+0000 7fc125468700 1 -- 172.21.15.42:0/2341433115 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc12005b8c0 con 0x7fc12011be50 2019-12-05T05:07:09.219 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:09.216+0000 7fc11f7fe700 1 --2- 172.21.15.42:0/2341433115 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc12011be50 0x7fc120117d00 secure :-1 s=THROTTLE_DONE pgs=303 cs=0 l=1 rx=0x7fc1200a5320 tx=0x7fc1100130f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:07:09.219 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:09.216+0000 7fc117fff700 1 -- 172.21.15.42:0/2341433115 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+1765 (secure 0 0 0) 0x7fc110005dc0 con 0x7fc12011be50 2019-12-05T05:07:09.219 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 24 2019-12-05T05:07:09.219 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:07:09.219 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:07:08.209582+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4831},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4831":{"gid":4831,"name":"b","rank":0,"incarnation":24,"state":"up:replay","state_seq":2,"addr":"172.21.15.40:6833/90248709","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":90248709},{"type":"v1","addr":"172.21.15.40:6833","nonce":90248709}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:07:09.227 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:09.216+0000 7fc125468700 1 -- 172.21.15.42:0/2341433115 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc10403ad80 msgr2=0x7fc10403d290 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:07:09.228 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:09.216+0000 7fc125468700 1 --2- 172.21.15.42:0/2341433115 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc10403ad80 0x7fc10403d290 crc :-1 s=READY pgs=143 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:07:09.228 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:09.216+0000 7fc125468700 1 -- 172.21.15.42:0/2341433115 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc12011be50 msgr2=0x7fc120117d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:07:09.228 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:09.216+0000 7fc125468700 1 --2- 172.21.15.42:0/2341433115 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc12011be50 0x7fc120117d00 secure :-1 s=READY pgs=303 cs=0 l=1 rx=0x7fc1200a5320 tx=0x7fc1100130f0).stop 2019-12-05T05:07:09.228 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:09.216+0000 7fc125468700 1 -- 172.21.15.42:0/2341433115 shutdown_connections 2019-12-05T05:07:09.228 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:09.216+0000 7fc125468700 1 --2- 172.21.15.42:0/2341433115 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc10403ad80 0x7fc10403d290 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:07:09.229 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:09.216+0000 7fc125468700 1 --2- 172.21.15.42:0/2341433115 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc12011be50 0x7fc120117d00 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:07:09.229 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:09.216+0000 7fc125468700 1 --2- 172.21.15.42:0/2341433115 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc1200a5930 0x7fc120130840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:07:09.229 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:09.216+0000 7fc125468700 1 -- 172.21.15.42:0/2341433115 shutdown_connections 2019-12-05T05:07:09.229 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:09.216+0000 7fc125468700 1 -- 172.21.15.42:0/2341433115 wait complete. 2019-12-05T05:07:09.229 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:09.216+0000 7fc125468700 1 -- 172.21.15.42:0/2341433115 >> 172.21.15.42:0/2341433115 conn(0x7fc1200a7f90 msgr2=0x7fc120059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:07:09.248 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 24, "feature_flags": { "enable_multiple": false, "ever_enabled_multiple": false }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2019-12-05T05:03:35.604738+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 24, "ever_allowed_features": 2, "explicitly_allowed_features": 2, "failed": [], "flags": 18, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_4831": { "addr": "172.21.15.40:6833/90248709", "addrs": { "addrvec": [ { "addr": "172.21.15.40:6832", "nonce": 90248709, "type": "v2" }, { "addr": "172.21.15.40:6833", "nonce": 90248709, "type": "v1" } ] }, "export_targets": [], "features": 4540138292836696063, "flags": 0, "gid": 4831, "incarnation": 24, "name": "b", "rank": 0, "state": "up:replay", "state_seq": 2 } }, "last_failure": 0, "last_failure_osd_epoch": 31, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 2, "min_compat_client": "0 (unknown)", "modified": "2019-12-05T05:07:08.209582+0000", "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4831 } } } ], "standbys": [] } 2019-12-05T05:07:11.249 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:07:11.342 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:11.336+0000 7f81282e8700 1 Processor -- start 2019-12-05T05:07:11.342 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:11.336+0000 7f81282e8700 1 -- start start 2019-12-05T05:07:11.343 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:11.336+0000 7f81282e8700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f81200a7740 0x7f812008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:07:11.343 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:11.336+0000 7f81282e8700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f812007e720 0x7f812007eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:07:11.343 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:11.336+0000 7f81282e8700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f812009f2e0 0x7f812009f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:07:11.343 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:11.336+0000 7f81282e8700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81200a0b00 con 0x7f812009f2e0 2019-12-05T05:07:11.343 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:11.336+0000 7f81282e8700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f812008e5b0 con 0x7f81200a7740 2019-12-05T05:07:11.343 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:11.336+0000 7f81282e8700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f812008e850 con 0x7f812007e720 2019-12-05T05:07:11.343 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:11.336+0000 7f8126885700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f81200a7740 0x7f812008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:07:11.344 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:11.336+0000 7f8126084700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f812007e720 0x7f812007eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:07:11.344 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:11.336+0000 7f8127086700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f812009f2e0 0x7f812009f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:07:11.344 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:11.336+0000 7f8126885700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f81200a7740 0x7f812008df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:55506/0 (socket says 172.21.15.42:55506) 2019-12-05T05:07:11.344 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:11.336+0000 7f8126885700 1 -- 172.21.15.42:0/271288291 learned_addr learned my addr 172.21.15.42:0/271288291 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:07:11.344 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:11.336+0000 7f8126084700 1 -- 172.21.15.42:0/271288291 >> v2:172.21.15.42:3300/0 conn(0x7f81200a7740 msgr2=0x7f812008df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:07:11.344 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:11.336+0000 7f8126084700 1 --2- 172.21.15.42:0/271288291 >> v2:172.21.15.42:3300/0 conn(0x7f81200a7740 0x7f812008df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:07:11.344 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:11.336+0000 7f8126084700 1 -- 172.21.15.42:0/271288291 >> v2:172.21.15.40:3300/0 conn(0x7f812009f2e0 msgr2=0x7f812009f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:07:11.345 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:11.336+0000 7f8126084700 1 --2- 172.21.15.42:0/271288291 >> v2:172.21.15.40:3300/0 conn(0x7f812009f2e0 0x7f812009f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:07:11.345 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:11.336+0000 7f8126084700 1 -- 172.21.15.42:0/271288291 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f812007f0a0 con 0x7f812007e720 2019-12-05T05:07:11.345 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:11.336+0000 7f8126885700 1 --2- 172.21.15.42:0/271288291 >> v2:172.21.15.42:3300/0 conn(0x7f81200a7740 0x7f812008df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:07:11.345 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:11.336+0000 7f8126084700 1 --2- 172.21.15.42:0/271288291 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f812007e720 0x7f812007eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f8110002f30 tx=0x7f811000fcf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:07:11.345 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:11.336+0000 7f8126084700 1 --2- 172.21.15.42:0/271288291 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f812007e720 0x7f812007eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f8110002f30 tx=0x7f811000fcf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:07:11.345 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:11.336+0000 7f8126084700 1 --2- 172.21.15.42:0/271288291 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f812007e720 0x7f812007eb50 secure :-1 s=READY pgs=63 cs=0 l=1 rx=0x7f8110002f30 tx=0x7f811000fcf0).ready entity=mon.2 client_cookie=8101cdef36e923f5 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:07:11.345 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:11.336+0000 7f8126084700 1 --2- 172.21.15.42:0/271288291 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f812007e720 0x7f812007eb50 secure :-1 s=THROTTLE_DONE pgs=63 cs=0 l=1 rx=0x7f8110002f30 tx=0x7f811000fcf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:07:11.346 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:11.336+0000 7f8125883700 1 -- 172.21.15.42:0/271288291 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8110008bb0 con 0x7f812007e720 2019-12-05T05:07:11.346 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:11.336+0000 7f8126084700 1 --2- 172.21.15.42:0/271288291 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f812007e720 0x7f812007eb50 secure :-1 s=THROTTLE_DONE pgs=63 cs=0 l=1 rx=0x7f8110002f30 tx=0x7f811000fcf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:07:11.346 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:11.336+0000 7f8126084700 1 --2- 172.21.15.42:0/271288291 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f812007e720 0x7f812007eb50 secure :-1 s=THROTTLE_DONE pgs=63 cs=0 l=1 rx=0x7f8110002f30 tx=0x7f811000fcf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:07:11.346 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:11.336+0000 7f8125883700 1 -- 172.21.15.42:0/271288291 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81100087d0 con 0x7f812007e720 2019-12-05T05:07:11.346 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:11.336+0000 7f8125883700 1 -- 172.21.15.42:0/271288291 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f81100118a0 con 0x7f812007e720 2019-12-05T05:07:11.346 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:11.336+0000 7f81282e8700 1 -- 172.21.15.42:0/271288291 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f812007e720 msgr2=0x7f812007eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:07:11.346 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:11.336+0000 7f81282e8700 1 --2- 172.21.15.42:0/271288291 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f812007e720 0x7f812007eb50 secure :-1 s=READY pgs=63 cs=0 l=1 rx=0x7f8110002f30 tx=0x7f811000fcf0).stop 2019-12-05T05:07:11.347 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:11.336+0000 7f81282e8700 1 -- 172.21.15.42:0/271288291 shutdown_connections 2019-12-05T05:07:11.347 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:11.336+0000 7f81282e8700 1 --2- 172.21.15.42:0/271288291 >> v2:172.21.15.40:3300/0 conn(0x7f812009f2e0 0x7f812009f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:07:11.347 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:11.336+0000 7f81282e8700 1 --2- 172.21.15.42:0/271288291 >> v2:172.21.15.42:3300/0 conn(0x7f81200a7740 0x7f812008df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:07:11.347 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:11.336+0000 7f81282e8700 1 --2- 172.21.15.42:0/271288291 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f812007e720 0x7f812007eb50 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:07:11.347 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:11.336+0000 7f81282e8700 1 -- 172.21.15.42:0/271288291 shutdown_connections 2019-12-05T05:07:11.347 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:11.336+0000 7f81282e8700 1 -- 172.21.15.42:0/271288291 wait complete. 2019-12-05T05:07:11.347 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:11.336+0000 7f81282e8700 1 -- 172.21.15.42:0/271288291 >> 172.21.15.42:0/271288291 conn(0x7f81200aabe0 msgr2=0x7f81200a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:07:11.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:11.340+0000 7f81282e8700 1 Processor -- start 2019-12-05T05:07:11.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:11.340+0000 7f81282e8700 1 -- start start 2019-12-05T05:07:11.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:11.340+0000 7f81282e8700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f812011b5f0 0x7f812011ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:07:11.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:11.340+0000 7f81282e8700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f812011bf70 con 0x7f8120120460 2019-12-05T05:07:11.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:11.340+0000 7f81282e8700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8120139920 con 0x7f812011fa00 2019-12-05T05:07:11.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:11.340+0000 7f81282e8700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f812011f220 con 0x7f812011b5f0 2019-12-05T05:07:11.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:11.340+0000 7f8127086700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f812011b5f0 0x7f812011ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:07:11.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:11.340+0000 7f8126084700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f812011fa00 0x7f812007d8a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:55552/0 (socket says 172.21.15.42:55552) 2019-12-05T05:07:11.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:11.340+0000 7f8126885700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f8120120460 0x7f812012cef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:36758/0 (socket says 172.21.15.42:36758) 2019-12-05T05:07:11.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:11.340+0000 7f8126084700 1 -- 172.21.15.42:0/1515964001 learned_addr learned my addr 172.21.15.42:0/1515964001 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:07:11.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:11.340+0000 7f81177fe700 1 -- 172.21.15.42:0/1515964001 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2299434496 0 0) 0x7f812011bf70 con 0x7f8120120460 2019-12-05T05:07:11.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:11.340+0000 7f81177fe700 1 -- 172.21.15.42:0/1515964001 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81080030b0 con 0x7f8120120460 2019-12-05T05:07:11.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:11.340+0000 7f8127086700 1 -- 172.21.15.42:0/1515964001 >> v1:172.21.15.42:6789/0 conn(0x7f812011fa00 legacy=0x7f812007d8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:07:11.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:11.340+0000 7f81177fe700 1 -- 172.21.15.42:0/1515964001 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 232011388 0 0) 0x7f8120139920 con 0x7f812011fa00 2019-12-05T05:07:11.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:11.340+0000 7f8127086700 1 -- 172.21.15.42:0/1515964001 >> v1:172.21.15.40:6789/0 conn(0x7f8120120460 legacy=0x7f812012cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:07:11.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:11.340+0000 7f8127086700 1 -- 172.21.15.42:0/1515964001 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f812013ab60 con 0x7f812011b5f0 2019-12-05T05:07:11.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:11.340+0000 7f8127086700 1 --2- 172.21.15.42:0/1515964001 >> v2:172.21.15.42:3300/0 conn(0x7f812011b5f0 0x7f812011ba20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f81180032b0 tx=0x7f8118010c90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:07:11.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:11.340+0000 7f8127086700 1 --2- 172.21.15.42:0/1515964001 >> v2:172.21.15.42:3300/0 conn(0x7f812011b5f0 0x7f812011ba20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f81180032b0 tx=0x7f8118010c90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:07:11.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:11.340+0000 7f8127086700 1 --2- 172.21.15.42:0/1515964001 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f812011b5f0 0x7f812011ba20 secure :-1 s=READY pgs=305 cs=0 l=1 rx=0x7f81180032b0 tx=0x7f8118010c90).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:07:11.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:11.340+0000 7f8127086700 1 --2- 172.21.15.42:0/1515964001 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f812011b5f0 0x7f812011ba20 secure :-1 s=THROTTLE_DONE pgs=305 cs=0 l=1 rx=0x7f81180032b0 tx=0x7f8118010c90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:07:11.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:11.340+0000 7f81177fe700 1 -- 172.21.15.42:0/1515964001 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8118007350 con 0x7f812011b5f0 2019-12-05T05:07:11.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:11.340+0000 7f8127086700 1 --2- 172.21.15.42:0/1515964001 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f812011b5f0 0x7f812011ba20 secure :-1 s=THROTTLE_DONE pgs=305 cs=0 l=1 rx=0x7f81180032b0 tx=0x7f8118010c90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:07:11.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:11.340+0000 7f81177fe700 1 -- 172.21.15.42:0/1515964001 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8118005cb0 con 0x7f812011b5f0 2019-12-05T05:07:11.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:11.340+0000 7f8127086700 1 --2- 172.21.15.42:0/1515964001 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f812011b5f0 0x7f812011ba20 secure :-1 s=THROTTLE_DONE pgs=305 cs=0 l=1 rx=0x7f81180032b0 tx=0x7f8118010c90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:07:11.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:11.340+0000 7f81282e8700 1 -- 172.21.15.42:0/1515964001 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8120139b50 con 0x7f812011b5f0 2019-12-05T05:07:11.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:11.340+0000 7f81177fe700 1 -- 172.21.15.42:0/1515964001 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8118012350 con 0x7f812011b5f0 2019-12-05T05:07:11.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:11.340+0000 7f81282e8700 1 -- 172.21.15.42:0/1515964001 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8120139dc0 con 0x7f812011b5f0 2019-12-05T05:07:11.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:11.340+0000 7f8127086700 1 --2- 172.21.15.42:0/1515964001 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f812011b5f0 0x7f812011ba20 secure :-1 s=THROTTLE_DONE pgs=305 cs=0 l=1 rx=0x7f81180032b0 tx=0x7f8118010c90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:07:11.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:11.344+0000 7f81177fe700 1 -- 172.21.15.42:0/1515964001 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f81180125f0 con 0x7f812011b5f0 2019-12-05T05:07:11.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:11.344+0000 7f8127086700 1 --2- 172.21.15.42:0/1515964001 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f812011b5f0 0x7f812011ba20 secure :-1 s=THROTTLE_DONE pgs=305 cs=0 l=1 rx=0x7f81180032b0 tx=0x7f8118010c90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:07:11.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:11.344+0000 7f81177fe700 1 --2- 172.21.15.42:0/1515964001 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f810803b1e0 0x7f810803d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:07:11.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:11.344+0000 7f81177fe700 1 -- 172.21.15.42:0/1515964001 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6203+0+0 (secure 0 0 0) 0x7f811804b620 con 0x7f812011b5f0 2019-12-05T05:07:11.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:11.344+0000 7f8126885700 1 --2- 172.21.15.42:0/1515964001 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f810803b1e0 0x7f810803d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:07:11.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:11.344+0000 7f8126885700 1 --2- 172.21.15.42:0/1515964001 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f810803b1e0 0x7f810803d6f0 crc :-1 s=READY pgs=144 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:07:11.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:11.344+0000 7f81282e8700 1 -- 172.21.15.42:0/1515964001 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f812005d110 con 0x7f812011b5f0 2019-12-05T05:07:11.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:11.348+0000 7f8127086700 1 --2- 172.21.15.42:0/1515964001 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f812011b5f0 0x7f812011ba20 secure :-1 s=THROTTLE_DONE pgs=305 cs=0 l=1 rx=0x7f81180032b0 tx=0x7f8118010c90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:07:11.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:11.348+0000 7f81177fe700 1 -- 172.21.15.42:0/1515964001 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f8118032680 con 0x7f812011b5f0 2019-12-05T05:07:11.513 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:11.508+0000 7f81282e8700 1 -- 172.21.15.42:0/1515964001 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f812005b8c0 con 0x7f812011b5f0 2019-12-05T05:07:11.523 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:11.512+0000 7f8127086700 1 --2- 172.21.15.42:0/1515964001 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f812011b5f0 0x7f812011ba20 secure :-1 s=THROTTLE_DONE pgs=305 cs=0 l=1 rx=0x7f81180032b0 tx=0x7f8118010c90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:07:11.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:11.512+0000 7f81177fe700 1 -- 172.21.15.42:0/1515964001 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+1772 (secure 0 0 0) 0x7f81180328e0 con 0x7f812011b5f0 2019-12-05T05:07:11.529 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 27 2019-12-05T05:07:11.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:11.512+0000 7f81282e8700 1 -- 172.21.15.42:0/1515964001 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f810803b1e0 msgr2=0x7f810803d6f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:07:11.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:11.512+0000 7f81282e8700 1 --2- 172.21.15.42:0/1515964001 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f810803b1e0 0x7f810803d6f0 crc :-1 s=READY pgs=144 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:07:11.530 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:11.512+0000 7f81282e8700 1 -- 172.21.15.42:0/1515964001 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f812011b5f0 msgr2=0x7f812011ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:07:11.530 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:11.512+0000 7f81282e8700 1 --2- 172.21.15.42:0/1515964001 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f812011b5f0 0x7f812011ba20 secure :-1 s=READY pgs=305 cs=0 l=1 rx=0x7f81180032b0 tx=0x7f8118010c90).stop 2019-12-05T05:07:11.530 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:11.512+0000 7f81282e8700 1 -- 172.21.15.42:0/1515964001 shutdown_connections 2019-12-05T05:07:11.530 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:11.512+0000 7f81282e8700 1 --2- 172.21.15.42:0/1515964001 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f810803b1e0 0x7f810803d6f0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:07:11.530 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:11.512+0000 7f81282e8700 1 --2- 172.21.15.42:0/1515964001 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f812011b5f0 0x7f812011ba20 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:07:11.530 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:11.512+0000 7f81282e8700 1 -- 172.21.15.42:0/1515964001 shutdown_connections 2019-12-05T05:07:11.530 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:11.512+0000 7f81282e8700 1 -- 172.21.15.42:0/1515964001 wait complete. 2019-12-05T05:07:11.531 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:11.512+0000 7f81282e8700 1 -- 172.21.15.42:0/1515964001 >> 172.21.15.42:0/1515964001 conn(0x7f812007e4f0 msgr2=0x7f81200a9640 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:07:11.531 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:07:11.532 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:07:11.251519+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4831},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4831":{"gid":4831,"name":"b","rank":0,"incarnation":24,"state":"up:clientreplay","state_seq":38,"addr":"172.21.15.40:6833/90248709","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":90248709},{"type":"v1","addr":"172.21.15.40:6833","nonce":90248709}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:07:11.533 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2019-12-05T05:07:12.496 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/147/12' -> 'copyofsnap1/test1/147/12' 2019-12-05T05:07:12.496 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/147/13' -> 'copyofsnap1/test1/147/13' 2019-12-05T05:07:12.496 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/147/14' -> 'copyofsnap1/test1/147/14' 2019-12-05T05:07:12.497 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/147/15' -> 'copyofsnap1/test1/147/15' 2019-12-05T05:07:12.497 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/147/16' -> 'copyofsnap1/test1/147/16' 2019-12-05T05:07:12.497 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/147/17' -> 'copyofsnap1/test1/147/17' 2019-12-05T05:07:12.497 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/147/18' -> 'copyofsnap1/test1/147/18' 2019-12-05T05:07:12.498 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/147/19' -> 'copyofsnap1/test1/147/19' 2019-12-05T05:07:12.498 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/147/20' -> 'copyofsnap1/test1/147/20' 2019-12-05T05:07:12.498 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/148' -> 'copyofsnap1/test1/148' 2019-12-05T05:07:12.498 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/148/10' -> 'copyofsnap1/test1/148/10' 2019-12-05T05:07:12.498 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/148/11' -> 'copyofsnap1/test1/148/11' 2019-12-05T05:07:12.498 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/148/12' -> 'copyofsnap1/test1/148/12' 2019-12-05T05:07:12.498 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/148/13' -> 'copyofsnap1/test1/148/13' 2019-12-05T05:07:12.499 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/148/14' -> 'copyofsnap1/test1/148/14' 2019-12-05T05:07:12.499 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/148/15' -> 'copyofsnap1/test1/148/15' 2019-12-05T05:07:12.499 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/148/16' -> 'copyofsnap1/test1/148/16' 2019-12-05T05:07:12.499 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/148/17' -> 'copyofsnap1/test1/148/17' 2019-12-05T05:07:12.499 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/148/18' -> 'copyofsnap1/test1/148/18' 2019-12-05T05:07:12.499 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/148/19' -> 'copyofsnap1/test1/148/19' 2019-12-05T05:07:12.499 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/148/20' -> 'copyofsnap1/test1/148/20' 2019-12-05T05:07:12.501 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/149' -> 'copyofsnap1/test1/149' 2019-12-05T05:07:12.501 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/149/10' -> 'copyofsnap1/test1/149/10' 2019-12-05T05:07:12.501 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/149/11' -> 'copyofsnap1/test1/149/11' 2019-12-05T05:07:12.501 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/149/12' -> 'copyofsnap1/test1/149/12' 2019-12-05T05:07:12.501 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/149/13' -> 'copyofsnap1/test1/149/13' 2019-12-05T05:07:12.501 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/149/14' -> 'copyofsnap1/test1/149/14' 2019-12-05T05:07:12.502 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/149/15' -> 'copyofsnap1/test1/149/15' 2019-12-05T05:07:12.502 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/149/16' -> 'copyofsnap1/test1/149/16' 2019-12-05T05:07:12.502 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/149/17' -> 'copyofsnap1/test1/149/17' 2019-12-05T05:07:12.502 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/149/18' -> 'copyofsnap1/test1/149/18' 2019-12-05T05:07:12.502 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/149/19' -> 'copyofsnap1/test1/149/19' 2019-12-05T05:07:12.502 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/149/20' -> 'copyofsnap1/test1/149/20' 2019-12-05T05:07:12.502 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/150' -> 'copyofsnap1/test1/150' 2019-12-05T05:07:12.503 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/150/10' -> 'copyofsnap1/test1/150/10' 2019-12-05T05:07:12.503 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/150/11' -> 'copyofsnap1/test1/150/11' 2019-12-05T05:07:12.503 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/150/12' -> 'copyofsnap1/test1/150/12' 2019-12-05T05:07:12.503 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/150/13' -> 'copyofsnap1/test1/150/13' 2019-12-05T05:07:12.503 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/150/14' -> 'copyofsnap1/test1/150/14' 2019-12-05T05:07:12.503 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/150/15' -> 'copyofsnap1/test1/150/15' 2019-12-05T05:07:12.503 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/150/16' -> 'copyofsnap1/test1/150/16' 2019-12-05T05:07:12.503 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/150/17' -> 'copyofsnap1/test1/150/17' 2019-12-05T05:07:12.504 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/150/18' -> 'copyofsnap1/test1/150/18' 2019-12-05T05:07:12.504 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/150/19' -> 'copyofsnap1/test1/150/19' 2019-12-05T05:07:12.504 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/150/20' -> 'copyofsnap1/test1/150/20' 2019-12-05T05:07:12.504 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/151' -> 'copyofsnap1/test1/151' 2019-12-05T05:07:12.504 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/151/10' -> 'copyofsnap1/test1/151/10' 2019-12-05T05:07:12.504 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/151/11' -> 'copyofsnap1/test1/151/11' 2019-12-05T05:07:12.504 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/151/12' -> 'copyofsnap1/test1/151/12' 2019-12-05T05:07:12.505 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/151/13' -> 'copyofsnap1/test1/151/13' 2019-12-05T05:07:12.505 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/151/14' -> 'copyofsnap1/test1/151/14' 2019-12-05T05:07:12.505 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/151/15' -> 'copyofsnap1/test1/151/15' 2019-12-05T05:07:12.505 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/151/16' -> 'copyofsnap1/test1/151/16' 2019-12-05T05:07:12.505 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/151/17' -> 'copyofsnap1/test1/151/17' 2019-12-05T05:07:12.505 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/151/18' -> 'copyofsnap1/test1/151/18' 2019-12-05T05:07:12.505 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/151/19' -> 'copyofsnap1/test1/151/19' 2019-12-05T05:07:12.506 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/151/20' -> 'copyofsnap1/test1/151/20' 2019-12-05T05:07:12.506 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/152' -> 'copyofsnap1/test1/152' 2019-12-05T05:07:12.506 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/152/10' -> 'copyofsnap1/test1/152/10' 2019-12-05T05:07:12.506 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/152/11' -> 'copyofsnap1/test1/152/11' 2019-12-05T05:07:12.506 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/152/12' -> 'copyofsnap1/test1/152/12' 2019-12-05T05:07:12.506 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/152/13' -> 'copyofsnap1/test1/152/13' 2019-12-05T05:07:12.506 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/152/14' -> 'copyofsnap1/test1/152/14' 2019-12-05T05:07:12.506 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/152/15' -> 'copyofsnap1/test1/152/15' 2019-12-05T05:07:12.507 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/152/16' -> 'copyofsnap1/test1/152/16' 2019-12-05T05:07:12.507 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/152/17' -> 'copyofsnap1/test1/152/17' 2019-12-05T05:07:12.507 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/152/18' -> 'copyofsnap1/test1/152/18' 2019-12-05T05:07:12.507 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/152/19' -> 'copyofsnap1/test1/152/19' 2019-12-05T05:07:12.507 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/152/20' -> 'copyofsnap1/test1/152/20' 2019-12-05T05:07:12.507 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/153' -> 'copyofsnap1/test1/153' 2019-12-05T05:07:12.507 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/153/10' -> 'copyofsnap1/test1/153/10' 2019-12-05T05:07:12.508 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/153/11' -> 'copyofsnap1/test1/153/11' 2019-12-05T05:07:12.508 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/153/12' -> 'copyofsnap1/test1/153/12' 2019-12-05T05:07:12.508 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/153/13' -> 'copyofsnap1/test1/153/13' 2019-12-05T05:07:12.508 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/153/14' -> 'copyofsnap1/test1/153/14' 2019-12-05T05:07:12.508 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/153/15' -> 'copyofsnap1/test1/153/15' 2019-12-05T05:07:12.509 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/153/16' -> 'copyofsnap1/test1/153/16' 2019-12-05T05:07:12.509 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/153/17' -> 'copyofsnap1/test1/153/17' 2019-12-05T05:07:12.509 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/153/18' -> 'copyofsnap1/test1/153/18' 2019-12-05T05:07:12.509 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/153/19' -> 'copyofsnap1/test1/153/19' 2019-12-05T05:07:12.509 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/153/20' -> 'copyofsnap1/test1/153/20' 2019-12-05T05:07:13.534 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:07:13.631 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:13.624+0000 7f85acd83700 1 Processor -- start 2019-12-05T05:07:13.632 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:13.624+0000 7f85acd83700 1 -- start start 2019-12-05T05:07:13.632 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:13.624+0000 7f85acd83700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f85a808df80 0x7f85a809f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:07:13.632 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:13.624+0000 7f85acd83700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85a809fc60 con 0x7f85a809f830 2019-12-05T05:07:13.632 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:13.624+0000 7f85acd83700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85a80a0710 con 0x7f85a80a7740 2019-12-05T05:07:13.632 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:13.624+0000 7f85acd83700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85a80a0b00 con 0x7f85a808df80 2019-12-05T05:07:13.633 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:13.624+0000 7f85a6d9d700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f85a80a7740 0x7f85a807e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:55578/0 (socket says 172.21.15.42:55578) 2019-12-05T05:07:13.633 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:13.624+0000 7f85a6d9d700 1 -- 172.21.15.42:0/2717632662 learned_addr learned my addr 172.21.15.42:0/2717632662 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:07:13.633 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:13.624+0000 7f85a659c700 1 --2- 172.21.15.42:0/2717632662 >> v2:172.21.15.40:3300/0 conn(0x7f85a808df80 0x7f85a809f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:07:13.633 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:13.624+0000 7f85a5d9b700 1 -- 172.21.15.42:0/2717632662 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 926071237 0 0) 0x7f85a80a0710 con 0x7f85a80a7740 2019-12-05T05:07:13.633 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:13.624+0000 7f85a5d9b700 1 -- 172.21.15.42:0/2717632662 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85880030b0 con 0x7f85a80a7740 2019-12-05T05:07:13.633 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:13.624+0000 7f85a759e700 1 -- 172.21.15.42:0/2717632662 >> v1:172.21.15.40:6789/0 conn(0x7f85a809f830 legacy=0x7f85a814efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2019-12-05T05:07:13.633 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:13.624+0000 7f85a759e700 1 -- 172.21.15.42:0/2717632662 >> v1:172.21.15.40:6789/0 conn(0x7f85a809f830 legacy=0x7f85a814efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2019-12-05T05:07:13.634 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:13.624+0000 7f85a759e700 1 --1- 172.21.15.42:0/2717632662 >> v1:172.21.15.40:6789/0 conn(0x7f85a809f830 0x7f85a814efe0 :-1 s=OPENED pgs=245 cs=1 l=1).handle_message read tag failed 2019-12-05T05:07:13.634 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:13.624+0000 7f85a759e700 1 --1- 172.21.15.42:0/2717632662 >> v1:172.21.15.40:6789/0 conn(0x7f85a809f830 0x7f85a814efe0 :-1 s=OPENED pgs=245 cs=1 l=1).fault on lossy channel, failing 2019-12-05T05:07:13.634 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:13.624+0000 7f85a5d9b700 1 -- 172.21.15.42:0/2717632662 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 687756848 0 0) 0x7f85880030b0 con 0x7f85a80a7740 2019-12-05T05:07:13.634 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:13.624+0000 7f85a5d9b700 1 -- 172.21.15.42:0/2717632662 >> v2:172.21.15.40:3300/0 conn(0x7f85a808df80 msgr2=0x7f85a809f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:07:13.634 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:13.624+0000 7f85a5d9b700 1 --2- 172.21.15.42:0/2717632662 >> v2:172.21.15.40:3300/0 conn(0x7f85a808df80 0x7f85a809f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:07:13.634 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:13.624+0000 7f85a5d9b700 1 -- 172.21.15.42:0/2717632662 >> v1:172.21.15.40:6789/0 conn(0x7f85a809f830 legacy=0x7f85a814efe0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2019-12-05T05:07:13.634 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:13.624+0000 7f85a5d9b700 1 -- 172.21.15.42:0/2717632662 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85a807ef20 con 0x7f85a80a7740 2019-12-05T05:07:13.635 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:13.624+0000 7f85a5d9b700 1 -- 172.21.15.42:0/2717632662 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7f8590001750 con 0x7f85a80a7740 2019-12-05T05:07:13.635 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:13.624+0000 7f85a5d9b700 1 -- 172.21.15.42:0/2717632662 >> v1:172.21.15.42:6789/0 conn(0x7f85a80a7740 legacy=0x7f85a807e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:07:13.635 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:13.624+0000 7f85a5d9b700 1 --2- 172.21.15.42:0/2717632662 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f85880036e0 0x7f8588003b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:07:13.635 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:13.624+0000 7f85a5d9b700 1 --2- 172.21.15.42:0/2717632662 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8588004420 0x7f8588004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:07:13.635 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:13.624+0000 7f85a5d9b700 1 --2- 172.21.15.42:0/2717632662 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8588004da0 0x7f858804f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:07:13.635 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:13.624+0000 7f85a5d9b700 1 -- 172.21.15.42:0/2717632662 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85a80a0b00 con 0x7f8588004da0 2019-12-05T05:07:13.635 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:13.624+0000 7f85a5d9b700 1 -- 172.21.15.42:0/2717632662 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f858804fc30 con 0x7f8588004420 2019-12-05T05:07:13.635 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:13.624+0000 7f85a5d9b700 1 -- 172.21.15.42:0/2717632662 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f858804fe70 con 0x7f85880036e0 2019-12-05T05:07:13.636 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:13.624+0000 7f85a6d9d700 1 --2- 172.21.15.42:0/2717632662 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f85880036e0 0x7f8588003b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:07:13.636 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:13.624+0000 7f85a659c700 1 --2- 172.21.15.42:0/2717632662 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8588004420 0x7f8588004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:07:13.636 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:13.624+0000 7f85a759e700 1 --2- 172.21.15.42:0/2717632662 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8588004da0 0x7f858804f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:07:13.636 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:13.624+0000 7f85a659c700 1 -- 172.21.15.42:0/2717632662 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f85880036e0 msgr2=0x7f8588003b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:07:13.636 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:13.624+0000 7f85a659c700 1 --2- 172.21.15.42:0/2717632662 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f85880036e0 0x7f8588003b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:07:13.636 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:13.624+0000 7f85a659c700 1 -- 172.21.15.42:0/2717632662 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8588004da0 msgr2=0x7f858804f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:07:13.636 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:13.624+0000 7f85a659c700 1 --2- 172.21.15.42:0/2717632662 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8588004da0 0x7f858804f680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:07:13.637 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:13.624+0000 7f85a659c700 1 -- 172.21.15.42:0/2717632662 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8588050170 con 0x7f8588004420 2019-12-05T05:07:13.637 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:13.624+0000 7f85a759e700 1 --2- 172.21.15.42:0/2717632662 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8588004da0 0x7f858804f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T05:07:13.637 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:13.624+0000 7f85a759e700 1 -- 172.21.15.42:0/2717632662 reap_dead start 2019-12-05T05:07:13.637 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:13.628+0000 7f85a659c700 1 --2- 172.21.15.42:0/2717632662 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8588004420 0x7f8588004850 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f85980032b0 tx=0x7f8598010360).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:07:13.637 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:13.628+0000 7f85a659c700 1 --2- 172.21.15.42:0/2717632662 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8588004420 0x7f8588004850 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f85980032b0 tx=0x7f8598010360).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:07:13.637 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:13.628+0000 7f85a659c700 1 --2- 172.21.15.42:0/2717632662 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8588004420 0x7f8588004850 secure :-1 s=READY pgs=309 cs=0 l=1 rx=0x7f85980032b0 tx=0x7f8598010360).ready entity=mon.1 client_cookie=5b2900525505b374 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:07:13.637 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:13.628+0000 7f85a659c700 1 --2- 172.21.15.42:0/2717632662 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8588004420 0x7f8588004850 secure :-1 s=THROTTLE_DONE pgs=309 cs=0 l=1 rx=0x7f85980032b0 tx=0x7f8598010360).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:07:13.638 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:13.628+0000 7f85a5d9b700 1 -- 172.21.15.42:0/2717632662 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8598008d40 con 0x7f8588004420 2019-12-05T05:07:13.638 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:13.628+0000 7f85a659c700 1 --2- 172.21.15.42:0/2717632662 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8588004420 0x7f8588004850 secure :-1 s=THROTTLE_DONE pgs=309 cs=0 l=1 rx=0x7f85980032b0 tx=0x7f8598010360).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:07:13.638 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:13.628+0000 7f85a659c700 1 --2- 172.21.15.42:0/2717632662 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8588004420 0x7f8588004850 secure :-1 s=THROTTLE_DONE pgs=309 cs=0 l=1 rx=0x7f85980032b0 tx=0x7f8598010360).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:07:13.638 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:13.628+0000 7f85a5d9b700 1 -- 172.21.15.42:0/2717632662 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8598010cb0 con 0x7f8588004420 2019-12-05T05:07:13.638 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:13.628+0000 7f85a5d9b700 1 -- 172.21.15.42:0/2717632662 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8598005ca0 con 0x7f8588004420 2019-12-05T05:07:13.638 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:13.628+0000 7f85acd83700 1 -- 172.21.15.42:0/2717632662 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8588004420 msgr2=0x7f8588004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:07:13.638 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:13.628+0000 7f85acd83700 1 --2- 172.21.15.42:0/2717632662 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8588004420 0x7f8588004850 secure :-1 s=READY pgs=309 cs=0 l=1 rx=0x7f85980032b0 tx=0x7f8598010360).stop 2019-12-05T05:07:13.639 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:13.628+0000 7f85acd83700 1 -- 172.21.15.42:0/2717632662 shutdown_connections 2019-12-05T05:07:13.639 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:13.628+0000 7f85acd83700 1 --2- 172.21.15.42:0/2717632662 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8588004420 0x7f8588004850 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:07:13.639 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:13.628+0000 7f85acd83700 1 -- 172.21.15.42:0/2717632662 shutdown_connections 2019-12-05T05:07:13.639 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:13.628+0000 7f85acd83700 1 -- 172.21.15.42:0/2717632662 wait complete. 2019-12-05T05:07:13.639 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:13.628+0000 7f85acd83700 1 -- 172.21.15.42:0/2717632662 >> 172.21.15.42:0/2717632662 conn(0x7f85a80aabe0 msgr2=0x7f85a80a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:07:13.642 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:13.628+0000 7f85acd83700 1 Processor -- start 2019-12-05T05:07:13.643 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:13.628+0000 7f85acd83700 1 -- start start 2019-12-05T05:07:13.643 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:13.628+0000 7f85acd83700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f85a8124d10 0x7f85a8121d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:07:13.643 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:13.628+0000 7f85acd83700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f85a811cbf0 0x7f85a80752e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:07:13.643 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:13.628+0000 7f85acd83700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85a8125140 con 0x7f85a808edb0 2019-12-05T05:07:13.643 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:13.628+0000 7f85acd83700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85a8122980 con 0x7f85a8124d10 2019-12-05T05:07:13.643 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:13.628+0000 7f85acd83700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85a8122280 con 0x7f85a811cbf0 2019-12-05T05:07:13.644 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:13.628+0000 7f85a759e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f85a811cbf0 0x7f85a80752e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:07:13.644 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:13.628+0000 7f85a759e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f85a811cbf0 0x7f85a80752e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:55554/0 (socket says 172.21.15.42:55554) 2019-12-05T05:07:13.644 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:13.628+0000 7f85a759e700 1 -- 172.21.15.42:0/3531802943 learned_addr learned my addr 172.21.15.42:0/3531802943 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:07:13.644 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:13.628+0000 7f85a6d9d700 1 --2- 172.21.15.42:0/3531802943 >> v2:172.21.15.40:3300/0 conn(0x7f85a8124d10 0x7f85a8121d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:07:13.644 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:13.628+0000 7f85a759e700 1 -- 172.21.15.42:0/3531802943 >> v2:172.21.15.40:3300/0 conn(0x7f85a8124d10 msgr2=0x7f85a8121d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:07:13.644 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:13.628+0000 7f85a759e700 1 --2- 172.21.15.42:0/3531802943 >> v2:172.21.15.40:3300/0 conn(0x7f85a8124d10 0x7f85a8121d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:07:13.644 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:13.628+0000 7f85a759e700 1 -- 172.21.15.42:0/3531802943 >> v1:172.21.15.42:6789/0 conn(0x7f85a808edb0 legacy=0x7f85a811c3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:07:13.645 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:13.628+0000 7f85a759e700 1 -- 172.21.15.42:0/3531802943 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85a808f1e0 con 0x7f85a811cbf0 2019-12-05T05:07:13.645 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:13.632+0000 7f85a759e700 1 --2- 172.21.15.42:0/3531802943 >> v2:172.21.15.42:3300/0 conn(0x7f85a811cbf0 0x7f85a80752e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f85a808fbd0 tx=0x7f85a0010620).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:07:13.645 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:13.632+0000 7f85a759e700 1 --2- 172.21.15.42:0/3531802943 >> v2:172.21.15.42:3300/0 conn(0x7f85a811cbf0 0x7f85a80752e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f85a808fbd0 tx=0x7f85a0010620).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:07:13.645 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:13.632+0000 7f85a759e700 1 --2- 172.21.15.42:0/3531802943 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f85a811cbf0 0x7f85a80752e0 secure :-1 s=READY pgs=311 cs=0 l=1 rx=0x7f85a808fbd0 tx=0x7f85a0010620).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:07:13.645 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:13.632+0000 7f85a759e700 1 --2- 172.21.15.42:0/3531802943 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f85a811cbf0 0x7f85a80752e0 secure :-1 s=THROTTLE_DONE pgs=311 cs=0 l=1 rx=0x7f85a808fbd0 tx=0x7f85a0010620).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:07:13.645 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:13.632+0000 7f85a759e700 1 --2- 172.21.15.42:0/3531802943 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f85a811cbf0 0x7f85a80752e0 secure :-1 s=THROTTLE_DONE pgs=311 cs=0 l=1 rx=0x7f85a808fbd0 tx=0x7f85a0010620).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:07:13.646 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:13.632+0000 7f859f7fe700 1 -- 172.21.15.42:0/3531802943 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f85a0005730 con 0x7f85a811cbf0 2019-12-05T05:07:13.646 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:13.632+0000 7f85a759e700 1 --2- 172.21.15.42:0/3531802943 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f85a811cbf0 0x7f85a80752e0 secure :-1 s=THROTTLE_DONE pgs=311 cs=0 l=1 rx=0x7f85a808fbd0 tx=0x7f85a0010620).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:07:13.646 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:13.632+0000 7f859f7fe700 1 -- 172.21.15.42:0/3531802943 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85a0005350 con 0x7f85a811cbf0 2019-12-05T05:07:13.646 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:13.632+0000 7f85acd83700 1 -- 172.21.15.42:0/3531802943 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85a811d020 con 0x7f85a811cbf0 2019-12-05T05:07:13.646 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:13.632+0000 7f859f7fe700 1 -- 172.21.15.42:0/3531802943 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f85a0005d10 con 0x7f85a811cbf0 2019-12-05T05:07:13.646 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:13.632+0000 7f85acd83700 1 -- 172.21.15.42:0/3531802943 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85a8122b90 con 0x7f85a811cbf0 2019-12-05T05:07:13.646 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:13.632+0000 7f85a759e700 1 --2- 172.21.15.42:0/3531802943 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f85a811cbf0 0x7f85a80752e0 secure :-1 s=THROTTLE_DONE pgs=311 cs=0 l=1 rx=0x7f85a808fbd0 tx=0x7f85a0010620).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:07:13.646 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:13.632+0000 7f859f7fe700 1 -- 172.21.15.42:0/3531802943 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f85a000a180 con 0x7f85a811cbf0 2019-12-05T05:07:13.647 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:13.632+0000 7f85a759e700 1 --2- 172.21.15.42:0/3531802943 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f85a811cbf0 0x7f85a80752e0 secure :-1 s=THROTTLE_DONE pgs=311 cs=0 l=1 rx=0x7f85a808fbd0 tx=0x7f85a0010620).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:07:13.647 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:13.632+0000 7f859f7fe700 1 --2- 172.21.15.42:0/3531802943 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f858003ad50 0x7f858003d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:07:13.647 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:13.632+0000 7f859f7fe700 1 -- 172.21.15.42:0/3531802943 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6203+0+0 (secure 0 0 0) 0x7f85a004a970 con 0x7f85a811cbf0 2019-12-05T05:07:13.647 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:13.632+0000 7f85a6d9d700 1 --2- 172.21.15.42:0/3531802943 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f858003ad50 0x7f858003d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:07:13.647 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:13.632+0000 7f85a6d9d700 1 --2- 172.21.15.42:0/3531802943 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f858003ad50 0x7f858003d260 crc :-1 s=READY pgs=145 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:07:13.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:13.640+0000 7f85acd83700 1 -- 172.21.15.42:0/3531802943 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85a805d110 con 0x7f85a811cbf0 2019-12-05T05:07:13.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:13.648+0000 7f85a759e700 1 --2- 172.21.15.42:0/3531802943 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f85a811cbf0 0x7f85a80752e0 secure :-1 s=THROTTLE_DONE pgs=311 cs=0 l=1 rx=0x7f85a808fbd0 tx=0x7f85a0010620).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:07:13.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:13.648+0000 7f859f7fe700 1 -- 172.21.15.42:0/3531802943 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f85a00319f0 con 0x7f85a811cbf0 2019-12-05T05:07:13.821 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:13.816+0000 7f85acd83700 1 -- 172.21.15.42:0/3531802943 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f85a805b8c0 con 0x7f85a811cbf0 2019-12-05T05:07:13.822 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:13.820+0000 7f85a759e700 1 --2- 172.21.15.42:0/3531802943 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f85a811cbf0 0x7f85a80752e0 secure :-1 s=THROTTLE_DONE pgs=311 cs=0 l=1 rx=0x7f85a808fbd0 tx=0x7f85a0010620).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:07:13.822 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:13.820+0000 7f859f7fe700 1 -- 172.21.15.42:0/3531802943 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+1766 (secure 0 0 0) 0x7f85a0031c50 con 0x7f85a811cbf0 2019-12-05T05:07:13.822 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 28 2019-12-05T05:07:13.822 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:07:13.823 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:07:12.263209+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4831},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4831":{"gid":4831,"name":"b","rank":0,"incarnation":24,"state":"up:active","state_seq":39,"addr":"172.21.15.40:6833/90248709","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":90248709},{"type":"v1","addr":"172.21.15.40:6833","nonce":90248709}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:07:13.825 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:13.820+0000 7f85acd83700 1 -- 172.21.15.42:0/3531802943 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f858003ad50 msgr2=0x7f858003d260 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:07:13.825 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:13.820+0000 7f85acd83700 1 --2- 172.21.15.42:0/3531802943 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f858003ad50 0x7f858003d260 crc :-1 s=READY pgs=145 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:07:13.825 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:13.820+0000 7f85acd83700 1 -- 172.21.15.42:0/3531802943 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f85a811cbf0 msgr2=0x7f85a80752e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:07:13.825 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:13.820+0000 7f85acd83700 1 --2- 172.21.15.42:0/3531802943 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f85a811cbf0 0x7f85a80752e0 secure :-1 s=READY pgs=311 cs=0 l=1 rx=0x7f85a808fbd0 tx=0x7f85a0010620).stop 2019-12-05T05:07:13.825 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:13.820+0000 7f85acd83700 1 -- 172.21.15.42:0/3531802943 shutdown_connections 2019-12-05T05:07:13.825 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:13.820+0000 7f85acd83700 1 --2- 172.21.15.42:0/3531802943 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f858003ad50 0x7f858003d260 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:07:13.826 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:13.820+0000 7f85acd83700 1 --2- 172.21.15.42:0/3531802943 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f85a811cbf0 0x7f85a80752e0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:07:13.826 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:13.820+0000 7f85acd83700 1 --2- 172.21.15.42:0/3531802943 >> v2:172.21.15.40:3300/0 conn(0x7f85a8124d10 0x7f85a8121d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:07:13.826 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:13.820+0000 7f85acd83700 1 -- 172.21.15.42:0/3531802943 shutdown_connections 2019-12-05T05:07:13.826 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:13.820+0000 7f85acd83700 1 -- 172.21.15.42:0/3531802943 wait complete. 2019-12-05T05:07:13.826 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:13.820+0000 7f85acd83700 1 -- 172.21.15.42:0/3531802943 >> 172.21.15.42:0/3531802943 conn(0x7f85a808fde0 msgr2=0x7f85a80a9790 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:07:13.835 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b has gained rank=0, replacing gid=4354 2019-12-05T05:07:13.835 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 4 secs before reviving mds.a 2019-12-05T05:07:17.617 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/154' -> 'copyofsnap1/test1/154' 2019-12-05T05:07:17.617 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/154/10' -> 'copyofsnap1/test1/154/10' 2019-12-05T05:07:17.617 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/154/11' -> 'copyofsnap1/test1/154/11' 2019-12-05T05:07:17.617 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/154/12' -> 'copyofsnap1/test1/154/12' 2019-12-05T05:07:17.617 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/154/13' -> 'copyofsnap1/test1/154/13' 2019-12-05T05:07:17.618 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/154/14' -> 'copyofsnap1/test1/154/14' 2019-12-05T05:07:17.618 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/154/15' -> 'copyofsnap1/test1/154/15' 2019-12-05T05:07:17.618 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/154/16' -> 'copyofsnap1/test1/154/16' 2019-12-05T05:07:17.618 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/154/17' -> 'copyofsnap1/test1/154/17' 2019-12-05T05:07:17.618 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/154/18' -> 'copyofsnap1/test1/154/18' 2019-12-05T05:07:17.618 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/154/19' -> 'copyofsnap1/test1/154/19' 2019-12-05T05:07:17.618 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/154/20' -> 'copyofsnap1/test1/154/20' 2019-12-05T05:07:17.619 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/155' -> 'copyofsnap1/test1/155' 2019-12-05T05:07:17.619 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/155/10' -> 'copyofsnap1/test1/155/10' 2019-12-05T05:07:17.619 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/155/11' -> 'copyofsnap1/test1/155/11' 2019-12-05T05:07:17.619 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/155/12' -> 'copyofsnap1/test1/155/12' 2019-12-05T05:07:17.619 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/155/13' -> 'copyofsnap1/test1/155/13' 2019-12-05T05:07:17.619 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/155/14' -> 'copyofsnap1/test1/155/14' 2019-12-05T05:07:17.619 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/155/15' -> 'copyofsnap1/test1/155/15' 2019-12-05T05:07:17.619 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/155/16' -> 'copyofsnap1/test1/155/16' 2019-12-05T05:07:17.620 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/155/17' -> 'copyofsnap1/test1/155/17' 2019-12-05T05:07:17.620 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/155/18' -> 'copyofsnap1/test1/155/18' 2019-12-05T05:07:17.620 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/155/19' -> 'copyofsnap1/test1/155/19' 2019-12-05T05:07:17.620 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/155/20' -> 'copyofsnap1/test1/155/20' 2019-12-05T05:07:17.621 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/156' -> 'copyofsnap1/test1/156' 2019-12-05T05:07:17.621 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/156/10' -> 'copyofsnap1/test1/156/10' 2019-12-05T05:07:17.621 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/156/11' -> 'copyofsnap1/test1/156/11' 2019-12-05T05:07:17.621 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/156/12' -> 'copyofsnap1/test1/156/12' 2019-12-05T05:07:17.621 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/156/13' -> 'copyofsnap1/test1/156/13' 2019-12-05T05:07:17.622 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/156/14' -> 'copyofsnap1/test1/156/14' 2019-12-05T05:07:17.622 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/156/15' -> 'copyofsnap1/test1/156/15' 2019-12-05T05:07:17.622 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/156/16' -> 'copyofsnap1/test1/156/16' 2019-12-05T05:07:17.622 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/156/17' -> 'copyofsnap1/test1/156/17' 2019-12-05T05:07:17.622 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/156/18' -> 'copyofsnap1/test1/156/18' 2019-12-05T05:07:17.622 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/156/19' -> 'copyofsnap1/test1/156/19' 2019-12-05T05:07:17.622 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/156/20' -> 'copyofsnap1/test1/156/20' 2019-12-05T05:07:17.623 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/157' -> 'copyofsnap1/test1/157' 2019-12-05T05:07:17.623 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/157/10' -> 'copyofsnap1/test1/157/10' 2019-12-05T05:07:17.623 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/157/11' -> 'copyofsnap1/test1/157/11' 2019-12-05T05:07:17.623 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/157/12' -> 'copyofsnap1/test1/157/12' 2019-12-05T05:07:17.623 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/157/13' -> 'copyofsnap1/test1/157/13' 2019-12-05T05:07:17.623 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/157/14' -> 'copyofsnap1/test1/157/14' 2019-12-05T05:07:17.623 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/157/15' -> 'copyofsnap1/test1/157/15' 2019-12-05T05:07:17.624 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/157/16' -> 'copyofsnap1/test1/157/16' 2019-12-05T05:07:17.624 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/157/17' -> 'copyofsnap1/test1/157/17' 2019-12-05T05:07:17.624 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/157/18' -> 'copyofsnap1/test1/157/18' 2019-12-05T05:07:17.624 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/157/19' -> 'copyofsnap1/test1/157/19' 2019-12-05T05:07:17.624 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/157/20' -> 'copyofsnap1/test1/157/20' 2019-12-05T05:07:17.624 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/158' -> 'copyofsnap1/test1/158' 2019-12-05T05:07:17.624 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/158/10' -> 'copyofsnap1/test1/158/10' 2019-12-05T05:07:17.625 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/158/11' -> 'copyofsnap1/test1/158/11' 2019-12-05T05:07:17.625 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/158/12' -> 'copyofsnap1/test1/158/12' 2019-12-05T05:07:17.625 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/158/13' -> 'copyofsnap1/test1/158/13' 2019-12-05T05:07:17.625 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/158/14' -> 'copyofsnap1/test1/158/14' 2019-12-05T05:07:17.625 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/158/15' -> 'copyofsnap1/test1/158/15' 2019-12-05T05:07:17.625 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/158/16' -> 'copyofsnap1/test1/158/16' 2019-12-05T05:07:17.626 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/158/17' -> 'copyofsnap1/test1/158/17' 2019-12-05T05:07:17.626 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/158/18' -> 'copyofsnap1/test1/158/18' 2019-12-05T05:07:17.626 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/158/19' -> 'copyofsnap1/test1/158/19' 2019-12-05T05:07:17.626 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/158/20' -> 'copyofsnap1/test1/158/20' 2019-12-05T05:07:17.626 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/159' -> 'copyofsnap1/test1/159' 2019-12-05T05:07:17.626 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/159/10' -> 'copyofsnap1/test1/159/10' 2019-12-05T05:07:17.627 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/159/11' -> 'copyofsnap1/test1/159/11' 2019-12-05T05:07:17.627 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/159/12' -> 'copyofsnap1/test1/159/12' 2019-12-05T05:07:17.627 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/159/13' -> 'copyofsnap1/test1/159/13' 2019-12-05T05:07:17.627 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/159/14' -> 'copyofsnap1/test1/159/14' 2019-12-05T05:07:17.627 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/159/15' -> 'copyofsnap1/test1/159/15' 2019-12-05T05:07:17.627 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/159/16' -> 'copyofsnap1/test1/159/16' 2019-12-05T05:07:17.627 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/159/17' -> 'copyofsnap1/test1/159/17' 2019-12-05T05:07:17.628 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/159/18' -> 'copyofsnap1/test1/159/18' 2019-12-05T05:07:17.628 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/159/19' -> 'copyofsnap1/test1/159/19' 2019-12-05T05:07:17.628 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/159/20' -> 'copyofsnap1/test1/159/20' 2019-12-05T05:07:17.628 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/160' -> 'copyofsnap1/test1/160' 2019-12-05T05:07:17.628 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/160/10' -> 'copyofsnap1/test1/160/10' 2019-12-05T05:07:17.628 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/160/11' -> 'copyofsnap1/test1/160/11' 2019-12-05T05:07:17.628 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/160/12' -> 'copyofsnap1/test1/160/12' 2019-12-05T05:07:17.629 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/160/13' -> 'copyofsnap1/test1/160/13' 2019-12-05T05:07:17.629 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/160/14' -> 'copyofsnap1/test1/160/14' 2019-12-05T05:07:17.629 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/160/15' -> 'copyofsnap1/test1/160/15' 2019-12-05T05:07:17.629 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/160/16' -> 'copyofsnap1/test1/160/16' 2019-12-05T05:07:17.629 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/160/17' -> 'copyofsnap1/test1/160/17' 2019-12-05T05:07:17.836 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.a 2019-12-05T05:07:17.836 INFO:tasks.ceph.mds.a:Restarting daemon 2019-12-05T05:07:17.836 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2019-12-05T05:07:17.840 INFO:tasks.ceph.mds.a:Started 2019-12-05T05:07:17.841 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:07:17.898 INFO:tasks.ceph.mds.a.smithi042.stdout:starting mds.a at 2019-12-05T05:07:17.938 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:17.928+0000 7f4b6a313700 1 Processor -- start 2019-12-05T05:07:17.938 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:17.928+0000 7f4b6a313700 1 -- start start 2019-12-05T05:07:17.938 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:17.928+0000 7f4b6a313700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4b6408df80 0x7f4b6409f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:07:17.939 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:17.928+0000 7f4b6a313700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b6409fc60 con 0x7f4b640a7740 2019-12-05T05:07:17.939 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:17.928+0000 7f4b6a313700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b640a0710 con 0x7f4b6409f830 2019-12-05T05:07:17.939 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:17.928+0000 7f4b6a313700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b640a0b00 con 0x7f4b6408df80 2019-12-05T05:07:17.939 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:17.928+0000 7f4b690b1700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f4b6409f830 0x7f4b6414efe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:55638/0 (socket says 172.21.15.42:55638) 2019-12-05T05:07:17.939 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:17.928+0000 7f4b690b1700 1 -- 172.21.15.42:0/2776716009 learned_addr learned my addr 172.21.15.42:0/2776716009 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:07:17.939 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:17.928+0000 7f4b63fff700 1 --2- 172.21.15.42:0/2776716009 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4b6408df80 0x7f4b6409f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:07:17.940 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:17.928+0000 7f4b637fe700 1 -- 172.21.15.42:0/2776716009 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 202444861 0 0) 0x7f4b640a0710 con 0x7f4b6409f830 2019-12-05T05:07:17.940 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:17.928+0000 7f4b637fe700 1 -- 172.21.15.42:0/2776716009 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b500030b0 con 0x7f4b6409f830 2019-12-05T05:07:17.940 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:17.928+0000 7f4b637fe700 1 -- 172.21.15.42:0/2776716009 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2498595353 0 0) 0x7f4b6409fc60 con 0x7f4b640a7740 2019-12-05T05:07:17.940 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:17.928+0000 7f4b637fe700 1 -- 172.21.15.42:0/2776716009 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b640a0710 con 0x7f4b640a7740 2019-12-05T05:07:17.940 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:17.932+0000 7f4b637fe700 1 -- 172.21.15.42:0/2776716009 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1346674422 0 0) 0x7f4b500030b0 con 0x7f4b6409f830 2019-12-05T05:07:17.940 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:17.932+0000 7f4b637fe700 1 -- 172.21.15.42:0/2776716009 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4b6408df80 msgr2=0x7f4b6409f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:07:17.940 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:17.932+0000 7f4b637fe700 1 --2- 172.21.15.42:0/2776716009 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4b6408df80 0x7f4b6409f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:07:17.941 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:17.932+0000 7f4b637fe700 1 -- 172.21.15.42:0/2776716009 >> v1:172.21.15.40:6789/0 conn(0x7f4b640a7740 legacy=0x7f4b6407e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:07:17.941 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:17.932+0000 7f4b637fe700 1 -- 172.21.15.42:0/2776716009 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b6407ef20 con 0x7f4b6409f830 2019-12-05T05:07:17.941 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:17.932+0000 7f4b637fe700 1 -- 172.21.15.42:0/2776716009 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7f4b5c001ad0 con 0x7f4b6409f830 2019-12-05T05:07:17.941 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:17.932+0000 7f4b637fe700 1 -- 172.21.15.42:0/2776716009 >> v1:172.21.15.42:6789/0 conn(0x7f4b6409f830 legacy=0x7f4b6414efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:07:17.941 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:17.932+0000 7f4b637fe700 1 --2- 172.21.15.42:0/2776716009 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4b50003940 0x7f4b50003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:07:17.941 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:17.932+0000 7f4b637fe700 1 --2- 172.21.15.42:0/2776716009 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4b50004470 0x7f4b500048a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:07:17.941 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:17.932+0000 7f4b637fe700 1 --2- 172.21.15.42:0/2776716009 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4b50004e60 0x7f4b5004f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:07:17.942 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:17.932+0000 7f4b637fe700 1 -- 172.21.15.42:0/2776716009 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b640a0b00 con 0x7f4b50003940 2019-12-05T05:07:17.942 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:17.932+0000 7f4b637fe700 1 -- 172.21.15.42:0/2776716009 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b5004fc60 con 0x7f4b50004e60 2019-12-05T05:07:17.942 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:17.932+0000 7f4b637fe700 1 -- 172.21.15.42:0/2776716009 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b5004fea0 con 0x7f4b50004470 2019-12-05T05:07:17.942 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:17.932+0000 7f4b690b1700 1 --2- 172.21.15.42:0/2776716009 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4b50004e60 0x7f4b5004f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:07:17.942 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:17.932+0000 7f4b63fff700 1 --2- 172.21.15.42:0/2776716009 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4b50004470 0x7f4b500048a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:07:17.942 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:17.932+0000 7f4b688b0700 1 --2- 172.21.15.42:0/2776716009 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4b50003940 0x7f4b50003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:07:17.942 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:17.932+0000 7f4b690b1700 1 -- 172.21.15.42:0/2776716009 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4b50004470 msgr2=0x7f4b500048a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:07:17.943 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:17.932+0000 7f4b690b1700 1 --2- 172.21.15.42:0/2776716009 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4b50004470 0x7f4b500048a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:07:17.943 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:17.932+0000 7f4b690b1700 1 -- 172.21.15.42:0/2776716009 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4b50003940 msgr2=0x7f4b50003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:07:17.943 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:17.932+0000 7f4b690b1700 1 --2- 172.21.15.42:0/2776716009 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4b50003940 0x7f4b50003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:07:17.943 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:17.932+0000 7f4b690b1700 1 -- 172.21.15.42:0/2776716009 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b500501a0 con 0x7f4b50004e60 2019-12-05T05:07:17.943 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:17.932+0000 7f4b690b1700 1 --2- 172.21.15.42:0/2776716009 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4b50004e60 0x7f4b5004f680 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f4b500053f0 tx=0x7f4b5c00c440).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:07:17.943 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:17.932+0000 7f4b690b1700 1 --2- 172.21.15.42:0/2776716009 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4b50004e60 0x7f4b5004f680 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f4b500053f0 tx=0x7f4b5c00c440).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:07:17.944 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:17.932+0000 7f4b690b1700 1 --2- 172.21.15.42:0/2776716009 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4b50004e60 0x7f4b5004f680 secure :-1 s=READY pgs=316 cs=0 l=1 rx=0x7f4b500053f0 tx=0x7f4b5c00c440).ready entity=mon.1 client_cookie=6ce4c6598ba2da50 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:07:17.944 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:17.932+0000 7f4b690b1700 1 --2- 172.21.15.42:0/2776716009 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4b50004e60 0x7f4b5004f680 secure :-1 s=THROTTLE_DONE pgs=316 cs=0 l=1 rx=0x7f4b500053f0 tx=0x7f4b5c00c440).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:07:17.944 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:17.932+0000 7f4b690b1700 1 --2- 172.21.15.42:0/2776716009 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4b50004e60 0x7f4b5004f680 secure :-1 s=THROTTLE_DONE pgs=316 cs=0 l=1 rx=0x7f4b500053f0 tx=0x7f4b5c00c440).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:07:17.944 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:17.932+0000 7f4b637fe700 1 -- 172.21.15.42:0/2776716009 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4b5c005bc0 con 0x7f4b50004e60 2019-12-05T05:07:17.944 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:17.932+0000 7f4b690b1700 1 --2- 172.21.15.42:0/2776716009 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4b50004e60 0x7f4b5004f680 secure :-1 s=THROTTLE_DONE pgs=316 cs=0 l=1 rx=0x7f4b500053f0 tx=0x7f4b5c00c440).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:07:17.944 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:17.932+0000 7f4b637fe700 1 -- 172.21.15.42:0/2776716009 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b5c0057e0 con 0x7f4b50004e60 2019-12-05T05:07:17.944 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:17.932+0000 7f4b637fe700 1 -- 172.21.15.42:0/2776716009 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4b5c010350 con 0x7f4b50004e60 2019-12-05T05:07:17.945 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:17.932+0000 7f4b6a313700 1 -- 172.21.15.42:0/2776716009 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4b50004e60 msgr2=0x7f4b5004f680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:07:17.945 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:17.932+0000 7f4b6a313700 1 --2- 172.21.15.42:0/2776716009 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4b50004e60 0x7f4b5004f680 secure :-1 s=READY pgs=316 cs=0 l=1 rx=0x7f4b500053f0 tx=0x7f4b5c00c440).stop 2019-12-05T05:07:17.945 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:17.932+0000 7f4b690b1700 1 -- 172.21.15.42:0/2776716009 reap_dead start 2019-12-05T05:07:17.945 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:17.932+0000 7f4b6a313700 1 -- 172.21.15.42:0/2776716009 shutdown_connections 2019-12-05T05:07:17.945 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:17.932+0000 7f4b6a313700 1 -- 172.21.15.42:0/2776716009 shutdown_connections 2019-12-05T05:07:17.945 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:17.932+0000 7f4b6a313700 1 -- 172.21.15.42:0/2776716009 wait complete. 2019-12-05T05:07:17.945 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:17.932+0000 7f4b6a313700 1 -- 172.21.15.42:0/2776716009 >> 172.21.15.42:0/2776716009 conn(0x7f4b640aabe0 msgr2=0x7f4b640a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:07:17.950 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:17.936+0000 7f4b6a313700 1 Processor -- start 2019-12-05T05:07:17.950 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:17.936+0000 7f4b6a313700 1 -- start start 2019-12-05T05:07:17.950 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:17.936+0000 7f4b6a313700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f4b64073ad0 0x7f4b64075f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:07:17.950 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:17.936+0000 7f4b6a313700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b64142030 con 0x7f4b64121570 2019-12-05T05:07:17.951 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:17.936+0000 7f4b6a313700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b640a5ff0 con 0x7f4b64122640 2019-12-05T05:07:17.951 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:17.936+0000 7f4b6a313700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b641232e0 con 0x7f4b64073ad0 2019-12-05T05:07:17.951 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:17.936+0000 7f4b688b0700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f4b64122640 0x7f4b6411b700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:55646/0 (socket says 172.21.15.42:55646) 2019-12-05T05:07:17.951 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:17.936+0000 7f4b688b0700 1 -- 172.21.15.42:0/2771267420 learned_addr learned my addr 172.21.15.42:0/2771267420 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:07:17.951 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:17.936+0000 7f4b690b1700 1 --2- 172.21.15.42:0/2771267420 >> v2:172.21.15.40:3300/0 conn(0x7f4b64073ad0 0x7f4b64075f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:07:17.951 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:17.936+0000 7f4b617fa700 1 -- 172.21.15.42:0/2771267420 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 416462486 0 0) 0x7f4b640a5ff0 con 0x7f4b64122640 2019-12-05T05:07:17.951 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:17.936+0000 7f4b617fa700 1 -- 172.21.15.42:0/2771267420 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b480030b0 con 0x7f4b64122640 2019-12-05T05:07:17.952 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:17.936+0000 7f4b617fa700 1 -- 172.21.15.42:0/2771267420 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 1162050273 0 0) 0x7f4b480030b0 con 0x7f4b64122640 2019-12-05T05:07:17.952 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:17.936+0000 7f4b690b1700 1 -- 172.21.15.42:0/2771267420 >> v1:172.21.15.42:6789/0 conn(0x7f4b64122640 legacy=0x7f4b6411b700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:07:17.952 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:17.936+0000 7f4b690b1700 1 -- 172.21.15.42:0/2771267420 >> v1:172.21.15.40:6789/0 conn(0x7f4b64121570 legacy=0x7f4b640722c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:07:17.952 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:17.936+0000 7f4b690b1700 1 -- 172.21.15.42:0/2771267420 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b64123670 con 0x7f4b64073ad0 2019-12-05T05:07:17.952 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:17.936+0000 7f4b690b1700 1 --2- 172.21.15.42:0/2771267420 >> v2:172.21.15.40:3300/0 conn(0x7f4b64073ad0 0x7f4b64075f10 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f4b4c0024d0 tx=0x7f4b4c001210).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:07:17.953 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:17.936+0000 7f4b690b1700 1 --2- 172.21.15.42:0/2771267420 >> v2:172.21.15.40:3300/0 conn(0x7f4b64073ad0 0x7f4b64075f10 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f4b4c0024d0 tx=0x7f4b4c001210).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:07:17.953 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:17.936+0000 7f4b690b1700 1 --2- 172.21.15.42:0/2771267420 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4b64073ad0 0x7f4b64075f10 secure :-1 s=READY pgs=248 cs=0 l=1 rx=0x7f4b4c0024d0 tx=0x7f4b4c001210).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:07:17.953 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:17.940+0000 7f4b690b1700 1 --2- 172.21.15.42:0/2771267420 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4b64073ad0 0x7f4b64075f10 secure :-1 s=THROTTLE_DONE pgs=248 cs=0 l=1 rx=0x7f4b4c0024d0 tx=0x7f4b4c001210).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:07:17.954 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:17.940+0000 7f4b617fa700 1 -- 172.21.15.42:0/2771267420 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4b4c0022d0 con 0x7f4b64073ad0 2019-12-05T05:07:17.954 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:17.940+0000 7f4b690b1700 1 --2- 172.21.15.42:0/2771267420 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4b64073ad0 0x7f4b64075f10 secure :-1 s=THROTTLE_DONE pgs=248 cs=0 l=1 rx=0x7f4b4c0024d0 tx=0x7f4b4c001210).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:07:17.954 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:17.940+0000 7f4b6a313700 1 -- 172.21.15.42:0/2771267420 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b641219a0 con 0x7f4b64073ad0 2019-12-05T05:07:17.954 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:17.940+0000 7f4b690b1700 1 --2- 172.21.15.42:0/2771267420 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4b64073ad0 0x7f4b64075f10 secure :-1 s=THROTTLE_DONE pgs=248 cs=0 l=1 rx=0x7f4b4c0024d0 tx=0x7f4b4c001210).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:07:17.954 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:17.940+0000 7f4b617fa700 1 -- 172.21.15.42:0/2771267420 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b4c001c80 con 0x7f4b64073ad0 2019-12-05T05:07:17.954 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:17.940+0000 7f4b617fa700 1 -- 172.21.15.42:0/2771267420 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4b5c011200 con 0x7f4b64073ad0 2019-12-05T05:07:17.954 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:17.940+0000 7f4b6a313700 1 -- 172.21.15.42:0/2771267420 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b6411bf00 con 0x7f4b64073ad0 2019-12-05T05:07:17.955 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:17.940+0000 7f4b690b1700 1 --2- 172.21.15.42:0/2771267420 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4b64073ad0 0x7f4b64075f10 secure :-1 s=THROTTLE_DONE pgs=248 cs=0 l=1 rx=0x7f4b4c0024d0 tx=0x7f4b4c001210).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:07:17.955 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:17.940+0000 7f4b617fa700 1 -- 172.21.15.42:0/2771267420 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f4b5c011400 con 0x7f4b64073ad0 2019-12-05T05:07:17.955 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:17.940+0000 7f4b690b1700 1 --2- 172.21.15.42:0/2771267420 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4b64073ad0 0x7f4b64075f10 secure :-1 s=THROTTLE_DONE pgs=248 cs=0 l=1 rx=0x7f4b4c0024d0 tx=0x7f4b4c001210).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:07:17.955 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:17.940+0000 7f4b617fa700 1 --2- 172.21.15.42:0/2771267420 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4b4803b190 0x7f4b4803d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:07:17.955 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:17.940+0000 7f4b617fa700 1 -- 172.21.15.42:0/2771267420 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6203+0+0 (secure 0 0 0) 0x7f4b5c04aa30 con 0x7f4b64073ad0 2019-12-05T05:07:17.955 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:17.940+0000 7f4b688b0700 1 --2- 172.21.15.42:0/2771267420 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4b4803b190 0x7f4b4803d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:07:17.955 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:17.940+0000 7f4b688b0700 1 --2- 172.21.15.42:0/2771267420 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4b4803b190 0x7f4b4803d6a0 crc :-1 s=READY pgs=146 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:07:17.956 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:17.944+0000 7f4b6a313700 1 -- 172.21.15.42:0/2771267420 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b6405d110 con 0x7f4b64073ad0 2019-12-05T05:07:17.958 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:17.956+0000 7f4b690b1700 1 --2- 172.21.15.42:0/2771267420 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4b64073ad0 0x7f4b64075f10 secure :-1 s=THROTTLE_DONE pgs=248 cs=0 l=1 rx=0x7f4b4c0024d0 tx=0x7f4b4c001210).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:07:17.958 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:17.956+0000 7f4b617fa700 1 -- 172.21.15.42:0/2771267420 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f4b5c031a90 con 0x7f4b64073ad0 2019-12-05T05:07:18.127 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:18.124+0000 7f4b6a313700 1 -- 172.21.15.42:0/2771267420 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4b64122a70 con 0x7f4b64073ad0 2019-12-05T05:07:18.127 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:18.124+0000 7f4b690b1700 1 --2- 172.21.15.42:0/2771267420 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4b64073ad0 0x7f4b64075f10 secure :-1 s=THROTTLE_DONE pgs=248 cs=0 l=1 rx=0x7f4b4c0024d0 tx=0x7f4b4c001210).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:07:18.128 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:18.124+0000 7f4b617fa700 1 -- 172.21.15.42:0/2771267420 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+1766 (secure 0 0 0) 0x7f4b5c031cf0 con 0x7f4b64073ad0 2019-12-05T05:07:18.129 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 28 2019-12-05T05:07:18.129 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:07:18.129 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:07:12.263209+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4831},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4831":{"gid":4831,"name":"b","rank":0,"incarnation":24,"state":"up:active","state_seq":39,"addr":"172.21.15.40:6833/90248709","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":90248709},{"type":"v1","addr":"172.21.15.40:6833","nonce":90248709}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:07:18.132 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:18.128+0000 7f4b6a313700 1 -- 172.21.15.42:0/2771267420 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4b4803b190 msgr2=0x7f4b4803d6a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:07:18.132 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:18.128+0000 7f4b6a313700 1 --2- 172.21.15.42:0/2771267420 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4b4803b190 0x7f4b4803d6a0 crc :-1 s=READY pgs=146 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:07:18.132 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:18.128+0000 7f4b6a313700 1 -- 172.21.15.42:0/2771267420 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4b64073ad0 msgr2=0x7f4b64075f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:07:18.133 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:18.128+0000 7f4b6a313700 1 --2- 172.21.15.42:0/2771267420 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4b64073ad0 0x7f4b64075f10 secure :-1 s=READY pgs=248 cs=0 l=1 rx=0x7f4b4c0024d0 tx=0x7f4b4c001210).stop 2019-12-05T05:07:18.133 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:18.128+0000 7f4b6a313700 1 -- 172.21.15.42:0/2771267420 shutdown_connections 2019-12-05T05:07:18.133 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:18.128+0000 7f4b6a313700 1 --2- 172.21.15.42:0/2771267420 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4b4803b190 0x7f4b4803d6a0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:07:18.133 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:18.128+0000 7f4b6a313700 1 --2- 172.21.15.42:0/2771267420 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4b64073ad0 0x7f4b64075f10 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:07:18.133 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:18.128+0000 7f4b6a313700 1 -- 172.21.15.42:0/2771267420 shutdown_connections 2019-12-05T05:07:18.133 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:18.128+0000 7f4b6a313700 1 -- 172.21.15.42:0/2771267420 wait complete. 2019-12-05T05:07:18.133 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:18.128+0000 7f4b6a313700 1 -- 172.21.15.42:0/2771267420 >> 172.21.15.42:0/2771267420 conn(0x7f4b640a0f50 msgr2=0x7f4b6407e650 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:07:18.142 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2019-12-05T05:07:20.143 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:07:20.248 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:20.236+0000 7f7afc5fb700 1 Processor -- start 2019-12-05T05:07:20.249 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:20.236+0000 7f7afc5fb700 1 -- start start 2019-12-05T05:07:20.249 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:20.240+0000 7f7afc5fb700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7af409fae0 0x7f7af414fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:07:20.249 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:20.240+0000 7f7afc5fb700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7af40a0710 con 0x7f7af40a7740 2019-12-05T05:07:20.249 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:20.240+0000 7f7afc5fb700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7af407ef20 con 0x7f7af408df80 2019-12-05T05:07:20.249 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:20.240+0000 7f7afc5fb700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7af40a0b00 con 0x7f7af409fae0 2019-12-05T05:07:20.249 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:20.240+0000 7f7afa397700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f7af408df80 0x7f7af409f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:55660/0 (socket says 172.21.15.42:55660) 2019-12-05T05:07:20.249 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:20.240+0000 7f7afa397700 1 -- 172.21.15.42:0/3291550604 learned_addr learned my addr 172.21.15.42:0/3291550604 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:07:20.250 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:20.240+0000 7f7afb399700 1 --2- 172.21.15.42:0/3291550604 >> v2:172.21.15.40:3300/0 conn(0x7f7af409fae0 0x7f7af414fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:07:20.250 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:20.240+0000 7f7af9b96700 1 -- 172.21.15.42:0/3291550604 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 676400380 0 0) 0x7f7af407ef20 con 0x7f7af408df80 2019-12-05T05:07:20.250 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:20.240+0000 7f7af9b96700 1 -- 172.21.15.42:0/3291550604 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7adc0030b0 con 0x7f7af408df80 2019-12-05T05:07:20.250 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:20.240+0000 7f7afb399700 1 -- 172.21.15.42:0/3291550604 >> v1:172.21.15.42:6789/0 conn(0x7f7af408df80 legacy=0x7f7af409f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:07:20.250 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:20.240+0000 7f7afb399700 1 -- 172.21.15.42:0/3291550604 >> v1:172.21.15.40:6789/0 conn(0x7f7af40a7740 legacy=0x7f7af407e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:07:20.250 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:20.240+0000 7f7af9b96700 1 -- 172.21.15.42:0/3291550604 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 724215904 0 0) 0x7f7af40a0710 con 0x7f7af40a7740 2019-12-05T05:07:20.250 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:20.240+0000 7f7afb399700 1 -- 172.21.15.42:0/3291550604 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7af40a9ba0 con 0x7f7af409fae0 2019-12-05T05:07:20.251 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:20.240+0000 7f7afb399700 1 --2- 172.21.15.42:0/3291550604 >> v2:172.21.15.40:3300/0 conn(0x7f7af409fae0 0x7f7af414fff0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f7aec008790 tx=0x7f7aec00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:07:20.251 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:20.240+0000 7f7afb399700 1 --2- 172.21.15.42:0/3291550604 >> v2:172.21.15.40:3300/0 conn(0x7f7af409fae0 0x7f7af414fff0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f7aec008790 tx=0x7f7aec00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:07:20.251 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:20.240+0000 7f7afb399700 1 --2- 172.21.15.42:0/3291550604 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7af409fae0 0x7f7af414fff0 secure :-1 s=READY pgs=250 cs=0 l=1 rx=0x7f7aec008790 tx=0x7f7aec00fbf0).ready entity=mon.0 client_cookie=bc1a9342f90ca881 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:07:20.251 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:20.240+0000 7f7afb399700 1 --2- 172.21.15.42:0/3291550604 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7af409fae0 0x7f7af414fff0 secure :-1 s=THROTTLE_DONE pgs=250 cs=0 l=1 rx=0x7f7aec008790 tx=0x7f7aec00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:07:20.251 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:20.240+0000 7f7af9b96700 1 -- 172.21.15.42:0/3291550604 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7aec008bb0 con 0x7f7af409fae0 2019-12-05T05:07:20.251 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:20.240+0000 7f7afb399700 1 --2- 172.21.15.42:0/3291550604 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7af409fae0 0x7f7af414fff0 secure :-1 s=THROTTLE_DONE pgs=250 cs=0 l=1 rx=0x7f7aec008790 tx=0x7f7aec00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:07:20.252 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:20.244+0000 7f7afb399700 1 --2- 172.21.15.42:0/3291550604 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7af409fae0 0x7f7af414fff0 secure :-1 s=THROTTLE_DONE pgs=250 cs=0 l=1 rx=0x7f7aec008790 tx=0x7f7aec00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:07:20.252 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:20.244+0000 7f7af9b96700 1 -- 172.21.15.42:0/3291550604 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7aec0087d0 con 0x7f7af409fae0 2019-12-05T05:07:20.252 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:20.244+0000 7f7af9b96700 1 -- 172.21.15.42:0/3291550604 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7aec0118d0 con 0x7f7af409fae0 2019-12-05T05:07:20.252 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:20.244+0000 7f7afc5fb700 1 -- 172.21.15.42:0/3291550604 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7af409fae0 msgr2=0x7f7af414fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:07:20.252 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:20.244+0000 7f7afc5fb700 1 --2- 172.21.15.42:0/3291550604 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7af409fae0 0x7f7af414fff0 secure :-1 s=READY pgs=250 cs=0 l=1 rx=0x7f7aec008790 tx=0x7f7aec00fbf0).stop 2019-12-05T05:07:20.252 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:20.244+0000 7f7afc5fb700 1 -- 172.21.15.42:0/3291550604 shutdown_connections 2019-12-05T05:07:20.252 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:20.244+0000 7f7afc5fb700 1 --2- 172.21.15.42:0/3291550604 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7af409fae0 0x7f7af414fff0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:07:20.253 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:20.244+0000 7f7afc5fb700 1 -- 172.21.15.42:0/3291550604 shutdown_connections 2019-12-05T05:07:20.253 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:20.244+0000 7f7afc5fb700 1 -- 172.21.15.42:0/3291550604 wait complete. 2019-12-05T05:07:20.253 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:20.244+0000 7f7afc5fb700 1 -- 172.21.15.42:0/3291550604 >> 172.21.15.42:0/3291550604 conn(0x7f7af40aabe0 msgr2=0x7f7af40a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:07:20.282 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:20.248+0000 7f7afc5fb700 1 Processor -- start 2019-12-05T05:07:20.282 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:20.248+0000 7f7afc5fb700 1 -- start start 2019-12-05T05:07:20.282 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:20.248+0000 7f7afc5fb700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f7af407c830 0x7f7af4066fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:07:20.282 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:20.248+0000 7f7afc5fb700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7af407bde0 0x7f7af411b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:07:20.283 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:20.248+0000 7f7afc5fb700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7af411a8b0 0x7f7af411b600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:07:20.283 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:20.248+0000 7f7afc5fb700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7af407c2f0 con 0x7f7af407bde0 2019-12-05T05:07:20.283 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:20.248+0000 7f7afc5fb700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7af407c590 con 0x7f7af407c830 2019-12-05T05:07:20.283 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:20.248+0000 7f7afc5fb700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7af4067530 con 0x7f7af411a8b0 2019-12-05T05:07:20.283 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:20.248+0000 7f7afab98700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f7af407c830 0x7f7af4066fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:07:20.284 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:20.248+0000 7f7afab98700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f7af407c830 0x7f7af4066fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:55626/0 (socket says 172.21.15.42:55626) 2019-12-05T05:07:20.284 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:20.248+0000 7f7afab98700 1 -- 172.21.15.42:0/837218544 learned_addr learned my addr 172.21.15.42:0/837218544 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:07:20.284 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:20.248+0000 7f7afb399700 1 --2- 172.21.15.42:0/837218544 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7af411a8b0 0x7f7af411b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:07:20.284 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:20.248+0000 7f7afa397700 1 --2- 172.21.15.42:0/837218544 >> v2:172.21.15.40:3300/0 conn(0x7f7af407bde0 0x7f7af411b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:07:20.286 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:20.248+0000 7f7afab98700 1 -- 172.21.15.42:0/837218544 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7af411a8b0 msgr2=0x7f7af411b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:07:20.286 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:20.248+0000 7f7afab98700 1 --2- 172.21.15.42:0/837218544 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7af411a8b0 0x7f7af411b600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:07:20.286 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:20.248+0000 7f7afab98700 1 -- 172.21.15.42:0/837218544 >> v2:172.21.15.40:3300/0 conn(0x7f7af407bde0 msgr2=0x7f7af411b0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:07:20.287 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:20.248+0000 7f7afab98700 1 --2- 172.21.15.42:0/837218544 >> v2:172.21.15.40:3300/0 conn(0x7f7af407bde0 0x7f7af411b0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:07:20.288 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:20.248+0000 7f7afab98700 1 -- 172.21.15.42:0/837218544 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7af4118890 con 0x7f7af407c830 2019-12-05T05:07:20.288 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:20.248+0000 7f7afab98700 1 --2- 172.21.15.42:0/837218544 >> v2:172.21.15.42:3300/0 conn(0x7f7af407c830 0x7f7af4066fe0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f7af405d370 tx=0x7f7ae4010da0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:07:20.288 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:20.248+0000 7f7afab98700 1 --2- 172.21.15.42:0/837218544 >> v2:172.21.15.42:3300/0 conn(0x7f7af407c830 0x7f7af4066fe0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f7af405d370 tx=0x7f7ae4010da0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:07:20.288 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:20.248+0000 7f7afab98700 1 --2- 172.21.15.42:0/837218544 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7af407c830 0x7f7af4066fe0 secure :-1 s=READY pgs=319 cs=0 l=1 rx=0x7f7af405d370 tx=0x7f7ae4010da0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:07:20.288 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:20.248+0000 7f7afab98700 1 --2- 172.21.15.42:0/837218544 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7af407c830 0x7f7af4066fe0 secure :-1 s=THROTTLE_DONE pgs=319 cs=0 l=1 rx=0x7f7af405d370 tx=0x7f7ae4010da0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:07:20.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:20.248+0000 7f7afab98700 1 --2- 172.21.15.42:0/837218544 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7af407c830 0x7f7af4066fe0 secure :-1 s=THROTTLE_DONE pgs=319 cs=0 l=1 rx=0x7f7af405d370 tx=0x7f7ae4010da0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:07:20.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:20.248+0000 7f7afab98700 1 --2- 172.21.15.42:0/837218544 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7af407c830 0x7f7af4066fe0 secure :-1 s=THROTTLE_DONE pgs=319 cs=0 l=1 rx=0x7f7af405d370 tx=0x7f7ae4010da0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:07:20.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:20.248+0000 7f7af37fe700 1 -- 172.21.15.42:0/837218544 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7ae400ca40 con 0x7f7af407c830 2019-12-05T05:07:20.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:20.248+0000 7f7af37fe700 1 -- 172.21.15.42:0/837218544 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ae40101c0 con 0x7f7af407c830 2019-12-05T05:07:20.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:20.248+0000 7f7afc5fb700 1 -- 172.21.15.42:0/837218544 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7af407b140 con 0x7f7af407c830 2019-12-05T05:07:20.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:20.248+0000 7f7af37fe700 1 -- 172.21.15.42:0/837218544 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7ae40125d0 con 0x7f7af407c830 2019-12-05T05:07:20.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:20.248+0000 7f7afc5fb700 1 -- 172.21.15.42:0/837218544 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7af408ed10 con 0x7f7af407c830 2019-12-05T05:07:20.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:20.252+0000 7f7afab98700 1 --2- 172.21.15.42:0/837218544 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7af407c830 0x7f7af4066fe0 secure :-1 s=THROTTLE_DONE pgs=319 cs=0 l=1 rx=0x7f7af405d370 tx=0x7f7ae4010da0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:07:20.291 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:20.252+0000 7f7afab98700 1 --2- 172.21.15.42:0/837218544 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7af407c830 0x7f7af4066fe0 secure :-1 s=THROTTLE_DONE pgs=319 cs=0 l=1 rx=0x7f7af405d370 tx=0x7f7ae4010da0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:07:20.291 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:20.252+0000 7f7af37fe700 1 -- 172.21.15.42:0/837218544 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f7ae40128a0 con 0x7f7af407c830 2019-12-05T05:07:20.291 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:20.252+0000 7f7af37fe700 1 --2- 172.21.15.42:0/837218544 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f7ad403ae00 0x7f7ad403d310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:07:20.291 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:20.252+0000 7f7af37fe700 1 -- 172.21.15.42:0/837218544 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6203+0+0 (secure 0 0 0) 0x7f7ae404b500 con 0x7f7af407c830 2019-12-05T05:07:20.292 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:20.252+0000 7f7afa397700 1 --2- 172.21.15.42:0/837218544 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f7ad403ae00 0x7f7ad403d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:07:20.292 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:20.252+0000 7f7afa397700 1 --2- 172.21.15.42:0/837218544 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f7ad403ae00 0x7f7ad403d310 crc :-1 s=READY pgs=147 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:07:20.292 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:20.252+0000 7f7afc5fb700 1 -- 172.21.15.42:0/837218544 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7af405d110 con 0x7f7af407c830 2019-12-05T05:07:20.292 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:20.260+0000 7f7afab98700 1 --2- 172.21.15.42:0/837218544 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7af407c830 0x7f7af4066fe0 secure :-1 s=THROTTLE_DONE pgs=319 cs=0 l=1 rx=0x7f7af405d370 tx=0x7f7ae4010da0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:07:20.292 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:20.260+0000 7f7af37fe700 1 -- 172.21.15.42:0/837218544 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f7ae4032540 con 0x7f7af407c830 2019-12-05T05:07:20.442 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:20.440+0000 7f7afc5fb700 1 -- 172.21.15.42:0/837218544 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7af405b8c0 con 0x7f7af407c830 2019-12-05T05:07:20.443 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:20.440+0000 7f7afab98700 1 --2- 172.21.15.42:0/837218544 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7af407c830 0x7f7af4066fe0 secure :-1 s=THROTTLE_DONE pgs=319 cs=0 l=1 rx=0x7f7af405d370 tx=0x7f7ae4010da0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:07:20.444 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:20.440+0000 7f7af37fe700 1 -- 172.21.15.42:0/837218544 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+1766 (secure 0 0 0) 0x7f7ae40327a0 con 0x7f7af407c830 2019-12-05T05:07:20.444 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 28 2019-12-05T05:07:20.444 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:07:20.445 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:07:12.263209+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4831},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4831":{"gid":4831,"name":"b","rank":0,"incarnation":24,"state":"up:active","state_seq":39,"addr":"172.21.15.40:6833/90248709","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":90248709},{"type":"v1","addr":"172.21.15.40:6833","nonce":90248709}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:07:20.446 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:20.440+0000 7f7afc5fb700 1 -- 172.21.15.42:0/837218544 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f7ad403ae00 msgr2=0x7f7ad403d310 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:07:20.446 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:20.440+0000 7f7afc5fb700 1 --2- 172.21.15.42:0/837218544 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f7ad403ae00 0x7f7ad403d310 crc :-1 s=READY pgs=147 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:07:20.447 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:20.440+0000 7f7afc5fb700 1 -- 172.21.15.42:0/837218544 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7af407c830 msgr2=0x7f7af4066fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:07:20.447 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:20.440+0000 7f7afc5fb700 1 --2- 172.21.15.42:0/837218544 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7af407c830 0x7f7af4066fe0 secure :-1 s=READY pgs=319 cs=0 l=1 rx=0x7f7af405d370 tx=0x7f7ae4010da0).stop 2019-12-05T05:07:20.447 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:20.440+0000 7f7afc5fb700 1 -- 172.21.15.42:0/837218544 shutdown_connections 2019-12-05T05:07:20.447 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:20.440+0000 7f7afc5fb700 1 --2- 172.21.15.42:0/837218544 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f7ad403ae00 0x7f7ad403d310 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:07:20.447 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:20.440+0000 7f7afc5fb700 1 --2- 172.21.15.42:0/837218544 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7af411a8b0 0x7f7af411b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:07:20.448 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:20.440+0000 7f7afc5fb700 1 --2- 172.21.15.42:0/837218544 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7af407c830 0x7f7af4066fe0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:07:20.449 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:20.440+0000 7f7afc5fb700 1 --2- 172.21.15.42:0/837218544 >> v2:172.21.15.40:3300/0 conn(0x7f7af407bde0 0x7f7af411b0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:07:20.449 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:20.444+0000 7f7afc5fb700 1 -- 172.21.15.42:0/837218544 shutdown_connections 2019-12-05T05:07:20.449 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:20.444+0000 7f7afc5fb700 1 -- 172.21.15.42:0/837218544 wait complete. 2019-12-05T05:07:20.449 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:20.444+0000 7f7afc5fb700 1 -- 172.21.15.42:0/837218544 >> 172.21.15.42:0/837218544 conn(0x7f7af409f240 msgr2=0x7f7af4059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:07:20.455 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2019-12-05T05:07:22.457 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:07:22.559 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:22.552+0000 7f0e85367700 1 Processor -- start 2019-12-05T05:07:22.559 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:22.552+0000 7f0e85367700 1 -- start start 2019-12-05T05:07:22.559 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:22.552+0000 7f0e85367700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0e8009fad0 0x7f0e8014ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:07:22.559 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:22.552+0000 7f0e85367700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e800a0700 con 0x7f0e800a7730 2019-12-05T05:07:22.560 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:22.552+0000 7f0e85367700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e8007ef10 con 0x7f0e8008df70 2019-12-05T05:07:22.560 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:22.552+0000 7f0e85367700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e800a0af0 con 0x7f0e8009fad0 2019-12-05T05:07:22.560 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:22.552+0000 7f0e7ffff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0e8009fad0 0x7f0e8014ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:07:22.560 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:22.552+0000 7f0e7ffff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0e8009fad0 0x7f0e8014ffe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:55644/0 (socket says 172.21.15.42:55644) 2019-12-05T05:07:22.560 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:22.552+0000 7f0e7ffff700 1 -- 172.21.15.42:0/4225628436 learned_addr learned my addr 172.21.15.42:0/4225628436 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:07:22.560 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:22.552+0000 7f0e7f7fe700 1 --1- 172.21.15.42:0/4225628436 >> v1:172.21.15.40:6789/0 conn(0x7f0e800a7730 0x7f0e8007e710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:36886/0 (socket says 172.21.15.42:36886) 2019-12-05T05:07:22.560 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:22.552+0000 7f0e76ffd700 1 --1- 172.21.15.42:0/4225628436 >> v1:172.21.15.42:6789/0 conn(0x7f0e8008df70 0x7f0e8009f2d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:55678/0 (socket says 172.21.15.42:55678) 2019-12-05T05:07:22.561 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:22.552+0000 7f0e7effd700 1 -- 172.21.15.42:0/4225628436 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3355597339 0 0) 0x7f0e800a0700 con 0x7f0e800a7730 2019-12-05T05:07:22.561 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:22.552+0000 7f0e7ffff700 1 -- 172.21.15.42:0/4225628436 >> v1:172.21.15.42:6789/0 conn(0x7f0e8008df70 legacy=0x7f0e8009f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:07:22.561 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:22.552+0000 7f0e7ffff700 1 -- 172.21.15.42:0/4225628436 >> v1:172.21.15.40:6789/0 conn(0x7f0e800a7730 legacy=0x7f0e8007e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:07:22.561 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:22.552+0000 7f0e7ffff700 1 -- 172.21.15.42:0/4225628436 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e800a9b90 con 0x7f0e8009fad0 2019-12-05T05:07:22.561 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:22.552+0000 7f0e7ffff700 1 --2- 172.21.15.42:0/4225628436 >> v2:172.21.15.42:3300/0 conn(0x7f0e8009fad0 0x7f0e8014ffe0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f0e78008b10 tx=0x7f0e78010030).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:07:22.561 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:22.552+0000 7f0e7ffff700 1 --2- 172.21.15.42:0/4225628436 >> v2:172.21.15.42:3300/0 conn(0x7f0e8009fad0 0x7f0e8014ffe0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f0e78008b10 tx=0x7f0e78010030).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:07:22.562 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:22.552+0000 7f0e7ffff700 1 --2- 172.21.15.42:0/4225628436 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0e8009fad0 0x7f0e8014ffe0 secure :-1 s=READY pgs=321 cs=0 l=1 rx=0x7f0e78008b10 tx=0x7f0e78010030).ready entity=mon.1 client_cookie=4108d02e4f15009a server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:07:22.562 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:22.552+0000 7f0e7ffff700 1 --2- 172.21.15.42:0/4225628436 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0e8009fad0 0x7f0e8014ffe0 secure :-1 s=THROTTLE_DONE pgs=321 cs=0 l=1 rx=0x7f0e78008b10 tx=0x7f0e78010030).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:07:22.562 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:22.552+0000 7f0e7effd700 1 -- 172.21.15.42:0/4225628436 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0e780110c0 con 0x7f0e8009fad0 2019-12-05T05:07:22.562 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:22.552+0000 7f0e7ffff700 1 --2- 172.21.15.42:0/4225628436 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0e8009fad0 0x7f0e8014ffe0 secure :-1 s=THROTTLE_DONE pgs=321 cs=0 l=1 rx=0x7f0e78008b10 tx=0x7f0e78010030).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:07:22.562 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:22.552+0000 7f0e7effd700 1 -- 172.21.15.42:0/4225628436 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e78010a30 con 0x7f0e8009fad0 2019-12-05T05:07:22.562 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:22.552+0000 7f0e7ffff700 1 --2- 172.21.15.42:0/4225628436 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0e8009fad0 0x7f0e8014ffe0 secure :-1 s=THROTTLE_DONE pgs=321 cs=0 l=1 rx=0x7f0e78008b10 tx=0x7f0e78010030).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:07:22.562 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:22.552+0000 7f0e7effd700 1 -- 172.21.15.42:0/4225628436 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0e780114b0 con 0x7f0e8009fad0 2019-12-05T05:07:22.563 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:22.552+0000 7f0e85367700 1 -- 172.21.15.42:0/4225628436 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0e8009fad0 msgr2=0x7f0e8014ffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:07:22.563 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:22.552+0000 7f0e85367700 1 --2- 172.21.15.42:0/4225628436 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0e8009fad0 0x7f0e8014ffe0 secure :-1 s=READY pgs=321 cs=0 l=1 rx=0x7f0e78008b10 tx=0x7f0e78010030).stop 2019-12-05T05:07:22.563 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:22.556+0000 7f0e85367700 1 -- 172.21.15.42:0/4225628436 shutdown_connections 2019-12-05T05:07:22.563 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:22.556+0000 7f0e85367700 1 --2- 172.21.15.42:0/4225628436 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0e8009fad0 0x7f0e8014ffe0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:07:22.563 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:22.556+0000 7f0e85367700 1 -- 172.21.15.42:0/4225628436 shutdown_connections 2019-12-05T05:07:22.563 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:22.556+0000 7f0e85367700 1 -- 172.21.15.42:0/4225628436 wait complete. 2019-12-05T05:07:22.564 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:22.556+0000 7f0e85367700 1 -- 172.21.15.42:0/4225628436 >> 172.21.15.42:0/4225628436 conn(0x7f0e800aabd0 msgr2=0x7f0e800a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:07:22.567 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:22.556+0000 7f0e85367700 1 Processor -- start 2019-12-05T05:07:22.567 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:22.556+0000 7f0e85367700 1 -- start start 2019-12-05T05:07:22.567 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:22.560+0000 7f0e85367700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0e80147fb0 0x7f0e800a1a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:07:22.567 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:22.560+0000 7f0e85367700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0e80119dc0 0x7f0e8007bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:07:22.567 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:22.560+0000 7f0e85367700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e80145810 con 0x7f0e8007b790 2019-12-05T05:07:22.567 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:22.560+0000 7f0e85367700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e8011b960 con 0x7f0e80147fb0 2019-12-05T05:07:22.567 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:22.560+0000 7f0e85367700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e8011af20 con 0x7f0e80119dc0 2019-12-05T05:07:22.568 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:22.560+0000 7f0e7f7fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0e80147fb0 0x7f0e800a1a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:07:22.568 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:22.560+0000 7f0e7ffff700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0e80119dc0 0x7f0e8007bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:07:22.568 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:22.560+0000 7f0e76ffd700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f0e8007b790 0x7f0e801195c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:36896/0 (socket says 172.21.15.42:36896) 2019-12-05T05:07:22.568 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:22.560+0000 7f0e76ffd700 1 -- 172.21.15.42:0/3373776462 learned_addr learned my addr 172.21.15.42:0/3373776462 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:07:22.568 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:22.560+0000 7f0e7cff9700 1 -- 172.21.15.42:0/3373776462 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 813287524 0 0) 0x7f0e80145810 con 0x7f0e8007b790 2019-12-05T05:07:22.568 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:22.560+0000 7f0e7cff9700 1 -- 172.21.15.42:0/3373776462 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e640030b0 con 0x7f0e8007b790 2019-12-05T05:07:22.568 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:22.560+0000 7f0e7f7fe700 1 -- 172.21.15.42:0/3373776462 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0e80119dc0 msgr2=0x7f0e8007bbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:07:22.569 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:22.560+0000 7f0e7f7fe700 1 --2- 172.21.15.42:0/3373776462 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0e80119dc0 0x7f0e8007bbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:07:22.569 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:22.560+0000 7f0e7f7fe700 1 -- 172.21.15.42:0/3373776462 >> v1:172.21.15.40:6789/0 conn(0x7f0e8007b790 legacy=0x7f0e801195c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:07:22.569 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:22.560+0000 7f0e7f7fe700 1 -- 172.21.15.42:0/3373776462 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e8007c110 con 0x7f0e80147fb0 2019-12-05T05:07:22.569 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:22.560+0000 7f0e7ffff700 1 --2- 172.21.15.42:0/3373776462 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0e80119dc0 0x7f0e8007bbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:07:22.569 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:22.560+0000 7f0e7f7fe700 1 --2- 172.21.15.42:0/3373776462 >> v2:172.21.15.42:3300/0 conn(0x7f0e80147fb0 0x7f0e800a1a10 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f0e70000b80 tx=0x7f0e70010840).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:07:22.569 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:22.560+0000 7f0e7f7fe700 1 --2- 172.21.15.42:0/3373776462 >> v2:172.21.15.42:3300/0 conn(0x7f0e80147fb0 0x7f0e800a1a10 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f0e70000b80 tx=0x7f0e70010840).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:07:22.569 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:22.560+0000 7f0e7f7fe700 1 --2- 172.21.15.42:0/3373776462 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0e80147fb0 0x7f0e800a1a10 secure :-1 s=READY pgs=322 cs=0 l=1 rx=0x7f0e70000b80 tx=0x7f0e70010840).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:07:22.570 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:22.560+0000 7f0e7f7fe700 1 --2- 172.21.15.42:0/3373776462 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0e80147fb0 0x7f0e800a1a10 secure :-1 s=THROTTLE_DONE pgs=322 cs=0 l=1 rx=0x7f0e70000b80 tx=0x7f0e70010840).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:07:22.570 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:22.560+0000 7f0e7cff9700 1 -- 172.21.15.42:0/3373776462 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0e70006850 con 0x7f0e80147fb0 2019-12-05T05:07:22.570 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:22.560+0000 7f0e7f7fe700 1 --2- 172.21.15.42:0/3373776462 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0e80147fb0 0x7f0e800a1a10 secure :-1 s=THROTTLE_DONE pgs=322 cs=0 l=1 rx=0x7f0e70000b80 tx=0x7f0e70010840).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:07:22.570 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:22.560+0000 7f0e85367700 1 -- 172.21.15.42:0/3373776462 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e8011aa80 con 0x7f0e80147fb0 2019-12-05T05:07:22.570 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:22.560+0000 7f0e7cff9700 1 -- 172.21.15.42:0/3373776462 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e70006470 con 0x7f0e80147fb0 2019-12-05T05:07:22.570 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:22.560+0000 7f0e7f7fe700 1 --2- 172.21.15.42:0/3373776462 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0e80147fb0 0x7f0e800a1a10 secure :-1 s=THROTTLE_DONE pgs=322 cs=0 l=1 rx=0x7f0e70000b80 tx=0x7f0e70010840).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:07:22.570 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:22.560+0000 7f0e7cff9700 1 -- 172.21.15.42:0/3373776462 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0e70011190 con 0x7f0e80147fb0 2019-12-05T05:07:22.571 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:22.560+0000 7f0e85367700 1 -- 172.21.15.42:0/3373776462 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e80135920 con 0x7f0e80147fb0 2019-12-05T05:07:22.571 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:22.560+0000 7f0e7f7fe700 1 --2- 172.21.15.42:0/3373776462 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0e80147fb0 0x7f0e800a1a10 secure :-1 s=THROTTLE_DONE pgs=322 cs=0 l=1 rx=0x7f0e70000b80 tx=0x7f0e70010840).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:07:22.571 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:22.560+0000 7f0e7cff9700 1 -- 172.21.15.42:0/3373776462 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f0e700114b0 con 0x7f0e80147fb0 2019-12-05T05:07:22.571 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:22.560+0000 7f0e7f7fe700 1 --2- 172.21.15.42:0/3373776462 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0e80147fb0 0x7f0e800a1a10 secure :-1 s=THROTTLE_DONE pgs=322 cs=0 l=1 rx=0x7f0e70000b80 tx=0x7f0e70010840).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:07:22.571 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:22.564+0000 7f0e7cff9700 1 --2- 172.21.15.42:0/3373776462 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0e6403b2a0 0x7f0e6403d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:07:22.571 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:22.564+0000 7f0e7cff9700 1 -- 172.21.15.42:0/3373776462 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6203+0+0 (secure 0 0 0) 0x7f0e7004afe0 con 0x7f0e80147fb0 2019-12-05T05:07:22.571 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:22.564+0000 7f0e76ffd700 1 --2- 172.21.15.42:0/3373776462 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0e6403b2a0 0x7f0e6403d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:07:22.572 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:22.564+0000 7f0e76ffd700 1 --2- 172.21.15.42:0/3373776462 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0e6403b2a0 0x7f0e6403d7b0 crc :-1 s=READY pgs=148 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:07:22.572 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:22.568+0000 7f0e85367700 1 -- 172.21.15.42:0/3373776462 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e8005d100 con 0x7f0e80147fb0 2019-12-05T05:07:22.580 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:22.576+0000 7f0e7f7fe700 1 --2- 172.21.15.42:0/3373776462 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0e80147fb0 0x7f0e800a1a10 secure :-1 s=THROTTLE_DONE pgs=322 cs=0 l=1 rx=0x7f0e70000b80 tx=0x7f0e70010840).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:07:22.581 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:22.576+0000 7f0e7cff9700 1 -- 172.21.15.42:0/3373776462 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f0e70032040 con 0x7f0e80147fb0 2019-12-05T05:07:22.778 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:22.776+0000 7f0e85367700 1 -- 172.21.15.42:0/3373776462 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0e800a7140 con 0x7f0e80147fb0 2019-12-05T05:07:22.779 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:22.776+0000 7f0e7f7fe700 1 --2- 172.21.15.42:0/3373776462 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0e80147fb0 0x7f0e800a1a10 secure :-1 s=THROTTLE_DONE pgs=322 cs=0 l=1 rx=0x7f0e70000b80 tx=0x7f0e70010840).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:07:22.779 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:22.776+0000 7f0e7cff9700 1 -- 172.21.15.42:0/3373776462 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+2102 (secure 0 0 0) 0x7f0e700322a0 con 0x7f0e80147fb0 2019-12-05T05:07:22.780 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 29 2019-12-05T05:07:22.780 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:07:22.780 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4978,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/3650564018","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3650564018},{"type":"v1","addr":"172.21.15.42:6835","nonce":3650564018}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":29}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:07:12.263209+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4831},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4831":{"gid":4831,"name":"b","rank":0,"incarnation":24,"state":"up:active","state_seq":39,"addr":"172.21.15.40:6833/90248709","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":90248709},{"type":"v1","addr":"172.21.15.40:6833","nonce":90248709}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:07:22.782 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:22.780+0000 7f0e85367700 1 -- 172.21.15.42:0/3373776462 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0e6403b2a0 msgr2=0x7f0e6403d7b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:07:22.783 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:22.780+0000 7f0e85367700 1 --2- 172.21.15.42:0/3373776462 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0e6403b2a0 0x7f0e6403d7b0 crc :-1 s=READY pgs=148 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:07:22.783 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:22.780+0000 7f0e85367700 1 -- 172.21.15.42:0/3373776462 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0e80147fb0 msgr2=0x7f0e800a1a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:07:22.783 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:22.780+0000 7f0e85367700 1 --2- 172.21.15.42:0/3373776462 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0e80147fb0 0x7f0e800a1a10 secure :-1 s=READY pgs=322 cs=0 l=1 rx=0x7f0e70000b80 tx=0x7f0e70010840).stop 2019-12-05T05:07:22.783 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:22.780+0000 7f0e85367700 1 -- 172.21.15.42:0/3373776462 shutdown_connections 2019-12-05T05:07:22.783 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:22.780+0000 7f0e85367700 1 --2- 172.21.15.42:0/3373776462 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0e6403b2a0 0x7f0e6403d7b0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:07:22.784 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:22.780+0000 7f0e85367700 1 --2- 172.21.15.42:0/3373776462 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0e80119dc0 0x7f0e8007bbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:07:22.784 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:22.780+0000 7f0e85367700 1 --2- 172.21.15.42:0/3373776462 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0e80147fb0 0x7f0e800a1a10 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:07:22.784 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:22.780+0000 7f0e85367700 1 -- 172.21.15.42:0/3373776462 shutdown_connections 2019-12-05T05:07:22.784 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:22.780+0000 7f0e85367700 1 -- 172.21.15.42:0/3373776462 wait complete. 2019-12-05T05:07:22.784 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:07:22.780+0000 7f0e85367700 1 -- 172.21.15.42:0/3373776462 >> 172.21.15.42:0/3373776462 conn(0x7f0e800a7730 msgr2=0x7f0e8009f360 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:07:22.792 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a reported in up:standby state 2019-12-05T05:07:22.792 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 103 secs before thrashing 2019-12-05T05:07:22.850 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/160/18' -> 'copyofsnap1/test1/160/18' 2019-12-05T05:07:22.850 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/160/19' -> 'copyofsnap1/test1/160/19' 2019-12-05T05:07:22.850 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/160/20' -> 'copyofsnap1/test1/160/20' 2019-12-05T05:07:22.850 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/161' -> 'copyofsnap1/test1/161' 2019-12-05T05:07:22.850 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/161/10' -> 'copyofsnap1/test1/161/10' 2019-12-05T05:07:22.850 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/161/11' -> 'copyofsnap1/test1/161/11' 2019-12-05T05:07:22.851 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/161/12' -> 'copyofsnap1/test1/161/12' 2019-12-05T05:07:22.851 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/161/13' -> 'copyofsnap1/test1/161/13' 2019-12-05T05:07:22.851 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/161/14' -> 'copyofsnap1/test1/161/14' 2019-12-05T05:07:22.851 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/161/15' -> 'copyofsnap1/test1/161/15' 2019-12-05T05:07:22.851 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/161/16' -> 'copyofsnap1/test1/161/16' 2019-12-05T05:07:22.851 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/161/17' -> 'copyofsnap1/test1/161/17' 2019-12-05T05:07:22.851 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/161/18' -> 'copyofsnap1/test1/161/18' 2019-12-05T05:07:22.852 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/161/19' -> 'copyofsnap1/test1/161/19' 2019-12-05T05:07:22.852 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/161/20' -> 'copyofsnap1/test1/161/20' 2019-12-05T05:07:22.852 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/162' -> 'copyofsnap1/test1/162' 2019-12-05T05:07:22.852 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/162/10' -> 'copyofsnap1/test1/162/10' 2019-12-05T05:07:22.852 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/162/11' -> 'copyofsnap1/test1/162/11' 2019-12-05T05:07:22.852 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/162/12' -> 'copyofsnap1/test1/162/12' 2019-12-05T05:07:22.852 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/162/13' -> 'copyofsnap1/test1/162/13' 2019-12-05T05:07:22.852 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/162/14' -> 'copyofsnap1/test1/162/14' 2019-12-05T05:07:22.853 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/162/15' -> 'copyofsnap1/test1/162/15' 2019-12-05T05:07:22.853 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/162/16' -> 'copyofsnap1/test1/162/16' 2019-12-05T05:07:22.853 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/162/17' -> 'copyofsnap1/test1/162/17' 2019-12-05T05:07:22.853 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/162/18' -> 'copyofsnap1/test1/162/18' 2019-12-05T05:07:22.853 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/162/19' -> 'copyofsnap1/test1/162/19' 2019-12-05T05:07:22.853 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/162/20' -> 'copyofsnap1/test1/162/20' 2019-12-05T05:07:22.853 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/163' -> 'copyofsnap1/test1/163' 2019-12-05T05:07:22.854 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/163/10' -> 'copyofsnap1/test1/163/10' 2019-12-05T05:07:22.854 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/163/11' -> 'copyofsnap1/test1/163/11' 2019-12-05T05:07:22.854 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/163/12' -> 'copyofsnap1/test1/163/12' 2019-12-05T05:07:22.854 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/163/13' -> 'copyofsnap1/test1/163/13' 2019-12-05T05:07:22.854 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/163/14' -> 'copyofsnap1/test1/163/14' 2019-12-05T05:07:22.854 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/163/15' -> 'copyofsnap1/test1/163/15' 2019-12-05T05:07:22.854 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/163/16' -> 'copyofsnap1/test1/163/16' 2019-12-05T05:07:22.854 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/163/17' -> 'copyofsnap1/test1/163/17' 2019-12-05T05:07:22.855 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/163/18' -> 'copyofsnap1/test1/163/18' 2019-12-05T05:07:22.855 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/163/19' -> 'copyofsnap1/test1/163/19' 2019-12-05T05:07:22.855 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/163/20' -> 'copyofsnap1/test1/163/20' 2019-12-05T05:07:22.855 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/164' -> 'copyofsnap1/test1/164' 2019-12-05T05:07:22.855 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/164/10' -> 'copyofsnap1/test1/164/10' 2019-12-05T05:07:22.855 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/164/11' -> 'copyofsnap1/test1/164/11' 2019-12-05T05:07:22.855 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/164/12' -> 'copyofsnap1/test1/164/12' 2019-12-05T05:07:22.856 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/164/13' -> 'copyofsnap1/test1/164/13' 2019-12-05T05:07:22.856 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/164/14' -> 'copyofsnap1/test1/164/14' 2019-12-05T05:07:22.856 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/164/15' -> 'copyofsnap1/test1/164/15' 2019-12-05T05:07:22.856 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/164/16' -> 'copyofsnap1/test1/164/16' 2019-12-05T05:07:22.856 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/164/17' -> 'copyofsnap1/test1/164/17' 2019-12-05T05:07:22.856 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/164/18' -> 'copyofsnap1/test1/164/18' 2019-12-05T05:07:22.856 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/164/19' -> 'copyofsnap1/test1/164/19' 2019-12-05T05:07:22.856 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/164/20' -> 'copyofsnap1/test1/164/20' 2019-12-05T05:07:22.857 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/165' -> 'copyofsnap1/test1/165' 2019-12-05T05:07:22.857 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/165/10' -> 'copyofsnap1/test1/165/10' 2019-12-05T05:07:22.857 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/165/11' -> 'copyofsnap1/test1/165/11' 2019-12-05T05:07:22.857 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/165/12' -> 'copyofsnap1/test1/165/12' 2019-12-05T05:07:22.857 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/165/13' -> 'copyofsnap1/test1/165/13' 2019-12-05T05:07:22.857 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/165/14' -> 'copyofsnap1/test1/165/14' 2019-12-05T05:07:22.857 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/165/15' -> 'copyofsnap1/test1/165/15' 2019-12-05T05:07:22.858 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/165/16' -> 'copyofsnap1/test1/165/16' 2019-12-05T05:07:22.858 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/165/17' -> 'copyofsnap1/test1/165/17' 2019-12-05T05:07:22.858 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/165/18' -> 'copyofsnap1/test1/165/18' 2019-12-05T05:07:22.858 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/165/19' -> 'copyofsnap1/test1/165/19' 2019-12-05T05:07:22.858 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/165/20' -> 'copyofsnap1/test1/165/20' 2019-12-05T05:07:22.858 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/166' -> 'copyofsnap1/test1/166' 2019-12-05T05:07:22.858 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/166/10' -> 'copyofsnap1/test1/166/10' 2019-12-05T05:07:22.858 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/166/11' -> 'copyofsnap1/test1/166/11' 2019-12-05T05:07:22.859 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/166/12' -> 'copyofsnap1/test1/166/12' 2019-12-05T05:07:22.859 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/166/13' -> 'copyofsnap1/test1/166/13' 2019-12-05T05:07:22.859 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/166/14' -> 'copyofsnap1/test1/166/14' 2019-12-05T05:07:22.859 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/166/15' -> 'copyofsnap1/test1/166/15' 2019-12-05T05:07:22.859 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/166/16' -> 'copyofsnap1/test1/166/16' 2019-12-05T05:07:22.859 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/166/17' -> 'copyofsnap1/test1/166/17' 2019-12-05T05:07:22.859 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/166/18' -> 'copyofsnap1/test1/166/18' 2019-12-05T05:07:22.860 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/166/19' -> 'copyofsnap1/test1/166/19' 2019-12-05T05:07:22.860 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/166/20' -> 'copyofsnap1/test1/166/20' 2019-12-05T05:07:22.860 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/167' -> 'copyofsnap1/test1/167' 2019-12-05T05:07:22.860 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/167/10' -> 'copyofsnap1/test1/167/10' 2019-12-05T05:07:22.860 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/167/11' -> 'copyofsnap1/test1/167/11' 2019-12-05T05:07:22.860 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/167/12' -> 'copyofsnap1/test1/167/12' 2019-12-05T05:07:22.860 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/167/13' -> 'copyofsnap1/test1/167/13' 2019-12-05T05:07:22.860 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/167/14' -> 'copyofsnap1/test1/167/14' 2019-12-05T05:07:26.086 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/167/15' -> 'copyofsnap1/test1/167/15' 2019-12-05T05:07:26.086 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/167/16' -> 'copyofsnap1/test1/167/16' 2019-12-05T05:07:26.086 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/167/17' -> 'copyofsnap1/test1/167/17' 2019-12-05T05:07:26.086 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/167/18' -> 'copyofsnap1/test1/167/18' 2019-12-05T05:07:26.087 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/167/19' -> 'copyofsnap1/test1/167/19' 2019-12-05T05:07:26.087 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/167/20' -> 'copyofsnap1/test1/167/20' 2019-12-05T05:07:26.087 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/168' -> 'copyofsnap1/test1/168' 2019-12-05T05:07:26.087 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/168/10' -> 'copyofsnap1/test1/168/10' 2019-12-05T05:07:26.088 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/168/11' -> 'copyofsnap1/test1/168/11' 2019-12-05T05:07:26.088 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/168/12' -> 'copyofsnap1/test1/168/12' 2019-12-05T05:07:26.088 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/168/13' -> 'copyofsnap1/test1/168/13' 2019-12-05T05:07:26.088 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/168/14' -> 'copyofsnap1/test1/168/14' 2019-12-05T05:07:26.089 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/168/15' -> 'copyofsnap1/test1/168/15' 2019-12-05T05:07:26.089 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/168/16' -> 'copyofsnap1/test1/168/16' 2019-12-05T05:07:26.089 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/168/17' -> 'copyofsnap1/test1/168/17' 2019-12-05T05:07:26.089 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/168/18' -> 'copyofsnap1/test1/168/18' 2019-12-05T05:07:26.089 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/168/19' -> 'copyofsnap1/test1/168/19' 2019-12-05T05:07:26.090 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/168/20' -> 'copyofsnap1/test1/168/20' 2019-12-05T05:07:26.090 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/169' -> 'copyofsnap1/test1/169' 2019-12-05T05:07:26.090 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/169/10' -> 'copyofsnap1/test1/169/10' 2019-12-05T05:07:26.090 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/169/11' -> 'copyofsnap1/test1/169/11' 2019-12-05T05:07:26.091 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/169/12' -> 'copyofsnap1/test1/169/12' 2019-12-05T05:07:26.091 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/169/13' -> 'copyofsnap1/test1/169/13' 2019-12-05T05:07:26.091 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/169/14' -> 'copyofsnap1/test1/169/14' 2019-12-05T05:07:26.109 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/169/15' -> 'copyofsnap1/test1/169/15' 2019-12-05T05:07:26.109 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/169/16' -> 'copyofsnap1/test1/169/16' 2019-12-05T05:07:26.109 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/169/17' -> 'copyofsnap1/test1/169/17' 2019-12-05T05:07:26.110 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/169/18' -> 'copyofsnap1/test1/169/18' 2019-12-05T05:07:26.110 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/169/19' -> 'copyofsnap1/test1/169/19' 2019-12-05T05:07:26.110 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/169/20' -> 'copyofsnap1/test1/169/20' 2019-12-05T05:07:26.110 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/170' -> 'copyofsnap1/test1/170' 2019-12-05T05:07:26.110 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/170/10' -> 'copyofsnap1/test1/170/10' 2019-12-05T05:07:26.111 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/170/11' -> 'copyofsnap1/test1/170/11' 2019-12-05T05:07:26.111 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/170/12' -> 'copyofsnap1/test1/170/12' 2019-12-05T05:07:26.111 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/170/13' -> 'copyofsnap1/test1/170/13' 2019-12-05T05:07:26.111 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/170/14' -> 'copyofsnap1/test1/170/14' 2019-12-05T05:07:26.112 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/170/15' -> 'copyofsnap1/test1/170/15' 2019-12-05T05:07:26.112 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/170/16' -> 'copyofsnap1/test1/170/16' 2019-12-05T05:07:26.118 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/170/17' -> 'copyofsnap1/test1/170/17' 2019-12-05T05:07:26.118 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/170/18' -> 'copyofsnap1/test1/170/18' 2019-12-05T05:07:26.118 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/170/19' -> 'copyofsnap1/test1/170/19' 2019-12-05T05:07:26.118 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/170/20' -> 'copyofsnap1/test1/170/20' 2019-12-05T05:07:26.119 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/171' -> 'copyofsnap1/test1/171' 2019-12-05T05:07:26.119 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/171/10' -> 'copyofsnap1/test1/171/10' 2019-12-05T05:07:26.119 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/171/11' -> 'copyofsnap1/test1/171/11' 2019-12-05T05:07:26.119 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/171/12' -> 'copyofsnap1/test1/171/12' 2019-12-05T05:07:26.120 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/171/13' -> 'copyofsnap1/test1/171/13' 2019-12-05T05:07:26.120 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/171/14' -> 'copyofsnap1/test1/171/14' 2019-12-05T05:07:26.122 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/171/15' -> 'copyofsnap1/test1/171/15' 2019-12-05T05:07:26.122 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/171/16' -> 'copyofsnap1/test1/171/16' 2019-12-05T05:07:26.122 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/171/17' -> 'copyofsnap1/test1/171/17' 2019-12-05T05:07:26.123 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/171/18' -> 'copyofsnap1/test1/171/18' 2019-12-05T05:07:26.124 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/171/19' -> 'copyofsnap1/test1/171/19' 2019-12-05T05:07:26.124 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/171/20' -> 'copyofsnap1/test1/171/20' 2019-12-05T05:07:26.124 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/172' -> 'copyofsnap1/test1/172' 2019-12-05T05:07:26.124 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/172/10' -> 'copyofsnap1/test1/172/10' 2019-12-05T05:07:26.124 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/172/11' -> 'copyofsnap1/test1/172/11' 2019-12-05T05:07:26.124 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/172/12' -> 'copyofsnap1/test1/172/12' 2019-12-05T05:07:26.125 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/172/13' -> 'copyofsnap1/test1/172/13' 2019-12-05T05:07:26.125 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/172/14' -> 'copyofsnap1/test1/172/14' 2019-12-05T05:07:26.125 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/172/15' -> 'copyofsnap1/test1/172/15' 2019-12-05T05:07:26.125 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/172/16' -> 'copyofsnap1/test1/172/16' 2019-12-05T05:07:26.125 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/172/17' -> 'copyofsnap1/test1/172/17' 2019-12-05T05:07:26.125 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/172/18' -> 'copyofsnap1/test1/172/18' 2019-12-05T05:07:26.125 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/172/19' -> 'copyofsnap1/test1/172/19' 2019-12-05T05:07:26.125 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/172/20' -> 'copyofsnap1/test1/172/20' 2019-12-05T05:07:26.126 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/173' -> 'copyofsnap1/test1/173' 2019-12-05T05:07:26.126 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/173/10' -> 'copyofsnap1/test1/173/10' 2019-12-05T05:07:26.126 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/173/11' -> 'copyofsnap1/test1/173/11' 2019-12-05T05:07:26.126 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/173/12' -> 'copyofsnap1/test1/173/12' 2019-12-05T05:07:26.126 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/173/13' -> 'copyofsnap1/test1/173/13' 2019-12-05T05:07:26.126 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/173/14' -> 'copyofsnap1/test1/173/14' 2019-12-05T05:07:26.126 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/173/15' -> 'copyofsnap1/test1/173/15' 2019-12-05T05:07:26.127 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/173/16' -> 'copyofsnap1/test1/173/16' 2019-12-05T05:07:26.127 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/173/17' -> 'copyofsnap1/test1/173/17' 2019-12-05T05:07:26.127 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/173/18' -> 'copyofsnap1/test1/173/18' 2019-12-05T05:07:26.127 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/173/19' -> 'copyofsnap1/test1/173/19' 2019-12-05T05:07:26.127 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/173/20' -> 'copyofsnap1/test1/173/20' 2019-12-05T05:07:26.128 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/174' -> 'copyofsnap1/test1/174' 2019-12-05T05:07:26.128 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/174/10' -> 'copyofsnap1/test1/174/10' 2019-12-05T05:07:26.128 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/174/11' -> 'copyofsnap1/test1/174/11' 2019-12-05T05:07:27.313 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/174/12' -> 'copyofsnap1/test1/174/12' 2019-12-05T05:07:27.313 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/174/13' -> 'copyofsnap1/test1/174/13' 2019-12-05T05:07:27.314 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/174/14' -> 'copyofsnap1/test1/174/14' 2019-12-05T05:07:27.314 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/174/15' -> 'copyofsnap1/test1/174/15' 2019-12-05T05:07:27.314 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/174/16' -> 'copyofsnap1/test1/174/16' 2019-12-05T05:07:27.314 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/174/17' -> 'copyofsnap1/test1/174/17' 2019-12-05T05:07:27.315 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/174/18' -> 'copyofsnap1/test1/174/18' 2019-12-05T05:07:27.315 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/174/19' -> 'copyofsnap1/test1/174/19' 2019-12-05T05:07:27.315 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/174/20' -> 'copyofsnap1/test1/174/20' 2019-12-05T05:07:27.315 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/175' -> 'copyofsnap1/test1/175' 2019-12-05T05:07:27.315 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/175/10' -> 'copyofsnap1/test1/175/10' 2019-12-05T05:07:27.315 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/175/11' -> 'copyofsnap1/test1/175/11' 2019-12-05T05:07:27.316 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/175/12' -> 'copyofsnap1/test1/175/12' 2019-12-05T05:07:27.316 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/175/13' -> 'copyofsnap1/test1/175/13' 2019-12-05T05:07:27.316 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/175/14' -> 'copyofsnap1/test1/175/14' 2019-12-05T05:07:27.318 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/175/15' -> 'copyofsnap1/test1/175/15' 2019-12-05T05:07:27.319 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/175/16' -> 'copyofsnap1/test1/175/16' 2019-12-05T05:07:27.319 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/175/17' -> 'copyofsnap1/test1/175/17' 2019-12-05T05:07:27.319 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/175/18' -> 'copyofsnap1/test1/175/18' 2019-12-05T05:07:27.319 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/175/19' -> 'copyofsnap1/test1/175/19' 2019-12-05T05:07:27.319 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/175/20' -> 'copyofsnap1/test1/175/20' 2019-12-05T05:07:27.319 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/176' -> 'copyofsnap1/test1/176' 2019-12-05T05:07:27.320 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/176/10' -> 'copyofsnap1/test1/176/10' 2019-12-05T05:07:27.320 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/176/11' -> 'copyofsnap1/test1/176/11' 2019-12-05T05:07:27.321 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/176/12' -> 'copyofsnap1/test1/176/12' 2019-12-05T05:07:27.321 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/176/13' -> 'copyofsnap1/test1/176/13' 2019-12-05T05:07:27.321 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/176/14' -> 'copyofsnap1/test1/176/14' 2019-12-05T05:07:27.321 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/176/15' -> 'copyofsnap1/test1/176/15' 2019-12-05T05:07:27.321 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/176/16' -> 'copyofsnap1/test1/176/16' 2019-12-05T05:07:27.321 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/176/17' -> 'copyofsnap1/test1/176/17' 2019-12-05T05:07:27.322 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/176/18' -> 'copyofsnap1/test1/176/18' 2019-12-05T05:07:27.322 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/176/19' -> 'copyofsnap1/test1/176/19' 2019-12-05T05:07:27.322 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/176/20' -> 'copyofsnap1/test1/176/20' 2019-12-05T05:07:27.322 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/177' -> 'copyofsnap1/test1/177' 2019-12-05T05:07:27.322 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/177/10' -> 'copyofsnap1/test1/177/10' 2019-12-05T05:07:27.322 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/177/11' -> 'copyofsnap1/test1/177/11' 2019-12-05T05:07:27.322 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/177/12' -> 'copyofsnap1/test1/177/12' 2019-12-05T05:07:27.323 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/177/13' -> 'copyofsnap1/test1/177/13' 2019-12-05T05:07:27.323 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/177/14' -> 'copyofsnap1/test1/177/14' 2019-12-05T05:07:27.323 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/177/15' -> 'copyofsnap1/test1/177/15' 2019-12-05T05:07:27.323 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/177/16' -> 'copyofsnap1/test1/177/16' 2019-12-05T05:07:27.323 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/177/17' -> 'copyofsnap1/test1/177/17' 2019-12-05T05:07:27.323 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/177/18' -> 'copyofsnap1/test1/177/18' 2019-12-05T05:07:27.323 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/177/19' -> 'copyofsnap1/test1/177/19' 2019-12-05T05:07:27.324 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/177/20' -> 'copyofsnap1/test1/177/20' 2019-12-05T05:07:27.324 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/178' -> 'copyofsnap1/test1/178' 2019-12-05T05:07:27.324 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/178/10' -> 'copyofsnap1/test1/178/10' 2019-12-05T05:07:27.324 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/178/11' -> 'copyofsnap1/test1/178/11' 2019-12-05T05:07:27.324 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/178/12' -> 'copyofsnap1/test1/178/12' 2019-12-05T05:07:27.324 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/178/13' -> 'copyofsnap1/test1/178/13' 2019-12-05T05:07:27.324 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/178/14' -> 'copyofsnap1/test1/178/14' 2019-12-05T05:07:27.325 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/178/15' -> 'copyofsnap1/test1/178/15' 2019-12-05T05:07:27.325 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/178/16' -> 'copyofsnap1/test1/178/16' 2019-12-05T05:07:27.325 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/178/17' -> 'copyofsnap1/test1/178/17' 2019-12-05T05:07:27.325 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/178/18' -> 'copyofsnap1/test1/178/18' 2019-12-05T05:07:27.325 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/178/19' -> 'copyofsnap1/test1/178/19' 2019-12-05T05:07:27.325 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/178/20' -> 'copyofsnap1/test1/178/20' 2019-12-05T05:07:27.326 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/179' -> 'copyofsnap1/test1/179' 2019-12-05T05:07:27.326 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/179/10' -> 'copyofsnap1/test1/179/10' 2019-12-05T05:07:27.326 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/179/11' -> 'copyofsnap1/test1/179/11' 2019-12-05T05:07:27.326 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/179/12' -> 'copyofsnap1/test1/179/12' 2019-12-05T05:07:27.326 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/179/13' -> 'copyofsnap1/test1/179/13' 2019-12-05T05:07:27.327 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/179/14' -> 'copyofsnap1/test1/179/14' 2019-12-05T05:07:27.327 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/179/15' -> 'copyofsnap1/test1/179/15' 2019-12-05T05:07:27.327 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/179/16' -> 'copyofsnap1/test1/179/16' 2019-12-05T05:07:27.327 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/179/17' -> 'copyofsnap1/test1/179/17' 2019-12-05T05:07:27.327 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/179/18' -> 'copyofsnap1/test1/179/18' 2019-12-05T05:07:27.329 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/179/19' -> 'copyofsnap1/test1/179/19' 2019-12-05T05:07:27.329 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/179/20' -> 'copyofsnap1/test1/179/20' 2019-12-05T05:07:27.329 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/180' -> 'copyofsnap1/test1/180' 2019-12-05T05:07:27.329 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/180/10' -> 'copyofsnap1/test1/180/10' 2019-12-05T05:07:27.330 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/180/11' -> 'copyofsnap1/test1/180/11' 2019-12-05T05:07:27.330 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/180/12' -> 'copyofsnap1/test1/180/12' 2019-12-05T05:07:27.332 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/180/13' -> 'copyofsnap1/test1/180/13' 2019-12-05T05:07:27.333 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/180/14' -> 'copyofsnap1/test1/180/14' 2019-12-05T05:07:27.333 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/180/15' -> 'copyofsnap1/test1/180/15' 2019-12-05T05:07:27.333 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/180/16' -> 'copyofsnap1/test1/180/16' 2019-12-05T05:07:27.333 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/180/17' -> 'copyofsnap1/test1/180/17' 2019-12-05T05:07:27.333 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/180/18' -> 'copyofsnap1/test1/180/18' 2019-12-05T05:07:27.334 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/180/19' -> 'copyofsnap1/test1/180/19' 2019-12-05T05:07:27.334 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/180/20' -> 'copyofsnap1/test1/180/20' 2019-12-05T05:07:29.465 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/181' -> 'copyofsnap1/test1/181' 2019-12-05T05:07:29.465 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/181/10' -> 'copyofsnap1/test1/181/10' 2019-12-05T05:07:29.465 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/181/11' -> 'copyofsnap1/test1/181/11' 2019-12-05T05:07:29.465 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/181/12' -> 'copyofsnap1/test1/181/12' 2019-12-05T05:07:29.466 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/181/13' -> 'copyofsnap1/test1/181/13' 2019-12-05T05:07:29.466 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/181/14' -> 'copyofsnap1/test1/181/14' 2019-12-05T05:07:29.466 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/181/15' -> 'copyofsnap1/test1/181/15' 2019-12-05T05:07:29.466 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/181/16' -> 'copyofsnap1/test1/181/16' 2019-12-05T05:07:29.466 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/181/17' -> 'copyofsnap1/test1/181/17' 2019-12-05T05:07:29.466 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/181/18' -> 'copyofsnap1/test1/181/18' 2019-12-05T05:07:29.466 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/181/19' -> 'copyofsnap1/test1/181/19' 2019-12-05T05:07:29.467 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/181/20' -> 'copyofsnap1/test1/181/20' 2019-12-05T05:07:29.467 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/182' -> 'copyofsnap1/test1/182' 2019-12-05T05:07:29.467 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/182/10' -> 'copyofsnap1/test1/182/10' 2019-12-05T05:07:29.467 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/182/11' -> 'copyofsnap1/test1/182/11' 2019-12-05T05:07:29.467 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/182/13' -> 'copyofsnap1/test1/182/13' 2019-12-05T05:07:29.467 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/182/14' -> 'copyofsnap1/test1/182/14' 2019-12-05T05:07:29.467 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/182/15' -> 'copyofsnap1/test1/182/15' 2019-12-05T05:07:29.467 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/182/16' -> 'copyofsnap1/test1/182/16' 2019-12-05T05:07:29.468 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/182/17' -> 'copyofsnap1/test1/182/17' 2019-12-05T05:07:29.468 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/182/18' -> 'copyofsnap1/test1/182/18' 2019-12-05T05:07:29.468 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/182/19' -> 'copyofsnap1/test1/182/19' 2019-12-05T05:07:29.468 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/182/20' -> 'copyofsnap1/test1/182/20' 2019-12-05T05:07:29.468 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/182/12' -> 'copyofsnap1/test1/182/12' 2019-12-05T05:07:29.468 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/183' -> 'copyofsnap1/test1/183' 2019-12-05T05:07:29.468 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/183/10' -> 'copyofsnap1/test1/183/10' 2019-12-05T05:07:29.469 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/183/11' -> 'copyofsnap1/test1/183/11' 2019-12-05T05:07:29.469 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/183/12' -> 'copyofsnap1/test1/183/12' 2019-12-05T05:07:29.469 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/183/13' -> 'copyofsnap1/test1/183/13' 2019-12-05T05:07:29.469 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/183/14' -> 'copyofsnap1/test1/183/14' 2019-12-05T05:07:29.469 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/183/15' -> 'copyofsnap1/test1/183/15' 2019-12-05T05:07:29.469 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/183/16' -> 'copyofsnap1/test1/183/16' 2019-12-05T05:07:29.469 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/183/17' -> 'copyofsnap1/test1/183/17' 2019-12-05T05:07:29.470 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/183/18' -> 'copyofsnap1/test1/183/18' 2019-12-05T05:07:29.470 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/183/19' -> 'copyofsnap1/test1/183/19' 2019-12-05T05:07:29.470 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/183/20' -> 'copyofsnap1/test1/183/20' 2019-12-05T05:07:29.471 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/184' -> 'copyofsnap1/test1/184' 2019-12-05T05:07:29.471 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/184/10' -> 'copyofsnap1/test1/184/10' 2019-12-05T05:07:29.471 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/184/11' -> 'copyofsnap1/test1/184/11' 2019-12-05T05:07:29.471 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/184/12' -> 'copyofsnap1/test1/184/12' 2019-12-05T05:07:29.471 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/184/13' -> 'copyofsnap1/test1/184/13' 2019-12-05T05:07:29.471 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/184/14' -> 'copyofsnap1/test1/184/14' 2019-12-05T05:07:29.472 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/184/15' -> 'copyofsnap1/test1/184/15' 2019-12-05T05:07:29.472 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/184/16' -> 'copyofsnap1/test1/184/16' 2019-12-05T05:07:29.472 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/184/17' -> 'copyofsnap1/test1/184/17' 2019-12-05T05:07:29.472 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/184/18' -> 'copyofsnap1/test1/184/18' 2019-12-05T05:07:29.472 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/184/19' -> 'copyofsnap1/test1/184/19' 2019-12-05T05:07:29.472 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/184/20' -> 'copyofsnap1/test1/184/20' 2019-12-05T05:07:29.472 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/185' -> 'copyofsnap1/test1/185' 2019-12-05T05:07:29.472 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/185/10' -> 'copyofsnap1/test1/185/10' 2019-12-05T05:07:29.473 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/185/11' -> 'copyofsnap1/test1/185/11' 2019-12-05T05:07:29.473 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/185/12' -> 'copyofsnap1/test1/185/12' 2019-12-05T05:07:29.473 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/185/13' -> 'copyofsnap1/test1/185/13' 2019-12-05T05:07:29.473 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/185/14' -> 'copyofsnap1/test1/185/14' 2019-12-05T05:07:29.473 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/185/15' -> 'copyofsnap1/test1/185/15' 2019-12-05T05:07:29.473 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/185/16' -> 'copyofsnap1/test1/185/16' 2019-12-05T05:07:29.473 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/185/17' -> 'copyofsnap1/test1/185/17' 2019-12-05T05:07:29.474 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/185/18' -> 'copyofsnap1/test1/185/18' 2019-12-05T05:07:29.474 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/185/19' -> 'copyofsnap1/test1/185/19' 2019-12-05T05:07:29.474 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/185/20' -> 'copyofsnap1/test1/185/20' 2019-12-05T05:07:29.474 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/186' -> 'copyofsnap1/test1/186' 2019-12-05T05:07:29.474 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/186/10' -> 'copyofsnap1/test1/186/10' 2019-12-05T05:07:29.474 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/186/11' -> 'copyofsnap1/test1/186/11' 2019-12-05T05:07:29.474 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/186/12' -> 'copyofsnap1/test1/186/12' 2019-12-05T05:07:29.475 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/186/13' -> 'copyofsnap1/test1/186/13' 2019-12-05T05:07:29.475 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/186/14' -> 'copyofsnap1/test1/186/14' 2019-12-05T05:07:29.475 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/186/15' -> 'copyofsnap1/test1/186/15' 2019-12-05T05:07:29.475 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/186/16' -> 'copyofsnap1/test1/186/16' 2019-12-05T05:07:29.475 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/186/17' -> 'copyofsnap1/test1/186/17' 2019-12-05T05:07:29.475 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/186/18' -> 'copyofsnap1/test1/186/18' 2019-12-05T05:07:29.475 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/186/19' -> 'copyofsnap1/test1/186/19' 2019-12-05T05:07:29.475 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/186/20' -> 'copyofsnap1/test1/186/20' 2019-12-05T05:07:29.476 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/187' -> 'copyofsnap1/test1/187' 2019-12-05T05:07:29.476 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/187/10' -> 'copyofsnap1/test1/187/10' 2019-12-05T05:07:29.476 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/187/11' -> 'copyofsnap1/test1/187/11' 2019-12-05T05:07:29.476 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/187/12' -> 'copyofsnap1/test1/187/12' 2019-12-05T05:07:29.476 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/187/13' -> 'copyofsnap1/test1/187/13' 2019-12-05T05:07:29.477 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/187/14' -> 'copyofsnap1/test1/187/14' 2019-12-05T05:07:29.477 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/187/15' -> 'copyofsnap1/test1/187/15' 2019-12-05T05:07:29.477 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/187/16' -> 'copyofsnap1/test1/187/16' 2019-12-05T05:07:29.477 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/187/17' -> 'copyofsnap1/test1/187/17' 2019-12-05T05:07:32.982 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/187/18' -> 'copyofsnap1/test1/187/18' 2019-12-05T05:07:32.982 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/187/19' -> 'copyofsnap1/test1/187/19' 2019-12-05T05:07:32.982 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/187/20' -> 'copyofsnap1/test1/187/20' 2019-12-05T05:07:32.982 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/188' -> 'copyofsnap1/test1/188' 2019-12-05T05:07:32.982 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/188/10' -> 'copyofsnap1/test1/188/10' 2019-12-05T05:07:32.983 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/188/11' -> 'copyofsnap1/test1/188/11' 2019-12-05T05:07:32.983 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/188/12' -> 'copyofsnap1/test1/188/12' 2019-12-05T05:07:32.983 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/188/13' -> 'copyofsnap1/test1/188/13' 2019-12-05T05:07:32.983 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/188/14' -> 'copyofsnap1/test1/188/14' 2019-12-05T05:07:32.983 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/188/15' -> 'copyofsnap1/test1/188/15' 2019-12-05T05:07:32.983 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/188/16' -> 'copyofsnap1/test1/188/16' 2019-12-05T05:07:32.983 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/188/17' -> 'copyofsnap1/test1/188/17' 2019-12-05T05:07:32.984 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/188/18' -> 'copyofsnap1/test1/188/18' 2019-12-05T05:07:32.984 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/188/19' -> 'copyofsnap1/test1/188/19' 2019-12-05T05:07:32.984 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/188/20' -> 'copyofsnap1/test1/188/20' 2019-12-05T05:07:32.984 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/189' -> 'copyofsnap1/test1/189' 2019-12-05T05:07:32.984 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/189/10' -> 'copyofsnap1/test1/189/10' 2019-12-05T05:07:32.984 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/189/11' -> 'copyofsnap1/test1/189/11' 2019-12-05T05:07:32.985 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/189/12' -> 'copyofsnap1/test1/189/12' 2019-12-05T05:07:32.985 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/189/13' -> 'copyofsnap1/test1/189/13' 2019-12-05T05:07:32.985 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/189/14' -> 'copyofsnap1/test1/189/14' 2019-12-05T05:07:32.985 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/189/15' -> 'copyofsnap1/test1/189/15' 2019-12-05T05:07:32.985 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/189/16' -> 'copyofsnap1/test1/189/16' 2019-12-05T05:07:32.985 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/189/17' -> 'copyofsnap1/test1/189/17' 2019-12-05T05:07:32.986 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/189/18' -> 'copyofsnap1/test1/189/18' 2019-12-05T05:07:32.986 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/189/19' -> 'copyofsnap1/test1/189/19' 2019-12-05T05:07:32.986 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/189/20' -> 'copyofsnap1/test1/189/20' 2019-12-05T05:07:32.986 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/190' -> 'copyofsnap1/test1/190' 2019-12-05T05:07:32.989 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/190/10' -> 'copyofsnap1/test1/190/10' 2019-12-05T05:07:32.990 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/190/11' -> 'copyofsnap1/test1/190/11' 2019-12-05T05:07:32.990 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/190/12' -> 'copyofsnap1/test1/190/12' 2019-12-05T05:07:32.990 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/190/13' -> 'copyofsnap1/test1/190/13' 2019-12-05T05:07:32.990 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/190/14' -> 'copyofsnap1/test1/190/14' 2019-12-05T05:07:32.991 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/190/15' -> 'copyofsnap1/test1/190/15' 2019-12-05T05:07:32.991 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/190/16' -> 'copyofsnap1/test1/190/16' 2019-12-05T05:07:32.991 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/190/17' -> 'copyofsnap1/test1/190/17' 2019-12-05T05:07:32.991 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/190/18' -> 'copyofsnap1/test1/190/18' 2019-12-05T05:07:32.991 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/190/19' -> 'copyofsnap1/test1/190/19' 2019-12-05T05:07:32.992 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/190/20' -> 'copyofsnap1/test1/190/20' 2019-12-05T05:07:32.992 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/191' -> 'copyofsnap1/test1/191' 2019-12-05T05:07:32.992 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/191/10' -> 'copyofsnap1/test1/191/10' 2019-12-05T05:07:32.992 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/191/11' -> 'copyofsnap1/test1/191/11' 2019-12-05T05:07:32.993 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/191/12' -> 'copyofsnap1/test1/191/12' 2019-12-05T05:07:32.993 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/191/13' -> 'copyofsnap1/test1/191/13' 2019-12-05T05:07:32.993 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/191/14' -> 'copyofsnap1/test1/191/14' 2019-12-05T05:07:32.993 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/191/15' -> 'copyofsnap1/test1/191/15' 2019-12-05T05:07:32.993 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/191/16' -> 'copyofsnap1/test1/191/16' 2019-12-05T05:07:32.994 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/191/17' -> 'copyofsnap1/test1/191/17' 2019-12-05T05:07:32.996 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/191/18' -> 'copyofsnap1/test1/191/18' 2019-12-05T05:07:32.996 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/191/19' -> 'copyofsnap1/test1/191/19' 2019-12-05T05:07:33.013 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/191/20' -> 'copyofsnap1/test1/191/20' 2019-12-05T05:07:33.014 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/192' -> 'copyofsnap1/test1/192' 2019-12-05T05:07:33.014 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/192/10' -> 'copyofsnap1/test1/192/10' 2019-12-05T05:07:33.014 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/192/11' -> 'copyofsnap1/test1/192/11' 2019-12-05T05:07:33.015 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/192/12' -> 'copyofsnap1/test1/192/12' 2019-12-05T05:07:33.015 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/192/13' -> 'copyofsnap1/test1/192/13' 2019-12-05T05:07:33.015 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/192/14' -> 'copyofsnap1/test1/192/14' 2019-12-05T05:07:33.015 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/192/15' -> 'copyofsnap1/test1/192/15' 2019-12-05T05:07:33.015 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/192/16' -> 'copyofsnap1/test1/192/16' 2019-12-05T05:07:33.016 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/192/17' -> 'copyofsnap1/test1/192/17' 2019-12-05T05:07:33.016 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/192/18' -> 'copyofsnap1/test1/192/18' 2019-12-05T05:07:33.016 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/192/19' -> 'copyofsnap1/test1/192/19' 2019-12-05T05:07:33.016 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/192/20' -> 'copyofsnap1/test1/192/20' 2019-12-05T05:07:33.016 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/193' -> 'copyofsnap1/test1/193' 2019-12-05T05:07:33.016 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/193/10' -> 'copyofsnap1/test1/193/10' 2019-12-05T05:07:33.016 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/193/11' -> 'copyofsnap1/test1/193/11' 2019-12-05T05:07:33.017 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/193/12' -> 'copyofsnap1/test1/193/12' 2019-12-05T05:07:33.017 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/193/13' -> 'copyofsnap1/test1/193/13' 2019-12-05T05:07:33.017 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/193/14' -> 'copyofsnap1/test1/193/14' 2019-12-05T05:07:33.017 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/193/15' -> 'copyofsnap1/test1/193/15' 2019-12-05T05:07:33.017 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/193/16' -> 'copyofsnap1/test1/193/16' 2019-12-05T05:07:33.017 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/193/17' -> 'copyofsnap1/test1/193/17' 2019-12-05T05:07:33.017 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/193/18' -> 'copyofsnap1/test1/193/18' 2019-12-05T05:07:33.017 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/193/19' -> 'copyofsnap1/test1/193/19' 2019-12-05T05:07:33.018 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/193/20' -> 'copyofsnap1/test1/193/20' 2019-12-05T05:07:33.018 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/194' -> 'copyofsnap1/test1/194' 2019-12-05T05:07:33.018 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/194/10' -> 'copyofsnap1/test1/194/10' 2019-12-05T05:07:33.018 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/194/11' -> 'copyofsnap1/test1/194/11' 2019-12-05T05:07:33.018 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/194/12' -> 'copyofsnap1/test1/194/12' 2019-12-05T05:07:33.018 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/194/13' -> 'copyofsnap1/test1/194/13' 2019-12-05T05:07:33.018 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/194/14' -> 'copyofsnap1/test1/194/14' 2019-12-05T05:07:34.788 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:07:34.793 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:07:36.419 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/194/15' -> 'copyofsnap1/test1/194/15' 2019-12-05T05:07:36.419 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/194/16' -> 'copyofsnap1/test1/194/16' 2019-12-05T05:07:36.420 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/194/17' -> 'copyofsnap1/test1/194/17' 2019-12-05T05:07:36.420 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/194/18' -> 'copyofsnap1/test1/194/18' 2019-12-05T05:07:36.420 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/194/19' -> 'copyofsnap1/test1/194/19' 2019-12-05T05:07:36.420 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/194/20' -> 'copyofsnap1/test1/194/20' 2019-12-05T05:07:36.420 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/195' -> 'copyofsnap1/test1/195' 2019-12-05T05:07:36.421 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/195/10' -> 'copyofsnap1/test1/195/10' 2019-12-05T05:07:36.421 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/195/11' -> 'copyofsnap1/test1/195/11' 2019-12-05T05:07:36.421 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/195/12' -> 'copyofsnap1/test1/195/12' 2019-12-05T05:07:36.421 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/195/13' -> 'copyofsnap1/test1/195/13' 2019-12-05T05:07:36.422 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/195/14' -> 'copyofsnap1/test1/195/14' 2019-12-05T05:07:36.422 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/195/15' -> 'copyofsnap1/test1/195/15' 2019-12-05T05:07:36.422 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/195/16' -> 'copyofsnap1/test1/195/16' 2019-12-05T05:07:36.422 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/195/17' -> 'copyofsnap1/test1/195/17' 2019-12-05T05:07:36.422 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/195/18' -> 'copyofsnap1/test1/195/18' 2019-12-05T05:07:36.423 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/195/19' -> 'copyofsnap1/test1/195/19' 2019-12-05T05:07:36.423 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/195/20' -> 'copyofsnap1/test1/195/20' 2019-12-05T05:07:36.423 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/196' -> 'copyofsnap1/test1/196' 2019-12-05T05:07:36.423 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/196/10' -> 'copyofsnap1/test1/196/10' 2019-12-05T05:07:36.423 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/196/11' -> 'copyofsnap1/test1/196/11' 2019-12-05T05:07:36.424 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/196/12' -> 'copyofsnap1/test1/196/12' 2019-12-05T05:07:36.424 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/196/13' -> 'copyofsnap1/test1/196/13' 2019-12-05T05:07:36.424 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/196/14' -> 'copyofsnap1/test1/196/14' 2019-12-05T05:07:36.426 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/196/15' -> 'copyofsnap1/test1/196/15' 2019-12-05T05:07:36.426 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/196/16' -> 'copyofsnap1/test1/196/16' 2019-12-05T05:07:36.426 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/196/17' -> 'copyofsnap1/test1/196/17' 2019-12-05T05:07:36.427 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/196/18' -> 'copyofsnap1/test1/196/18' 2019-12-05T05:07:36.428 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/196/19' -> 'copyofsnap1/test1/196/19' 2019-12-05T05:07:36.428 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/196/20' -> 'copyofsnap1/test1/196/20' 2019-12-05T05:07:36.428 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/197' -> 'copyofsnap1/test1/197' 2019-12-05T05:07:36.429 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/197/10' -> 'copyofsnap1/test1/197/10' 2019-12-05T05:07:36.429 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/197/11' -> 'copyofsnap1/test1/197/11' 2019-12-05T05:07:36.429 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/197/12' -> 'copyofsnap1/test1/197/12' 2019-12-05T05:07:36.429 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/197/13' -> 'copyofsnap1/test1/197/13' 2019-12-05T05:07:36.429 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/197/14' -> 'copyofsnap1/test1/197/14' 2019-12-05T05:07:36.430 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/197/15' -> 'copyofsnap1/test1/197/15' 2019-12-05T05:07:36.430 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/197/16' -> 'copyofsnap1/test1/197/16' 2019-12-05T05:07:36.430 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/197/17' -> 'copyofsnap1/test1/197/17' 2019-12-05T05:07:36.430 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/197/18' -> 'copyofsnap1/test1/197/18' 2019-12-05T05:07:36.430 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/197/19' -> 'copyofsnap1/test1/197/19' 2019-12-05T05:07:36.430 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/197/20' -> 'copyofsnap1/test1/197/20' 2019-12-05T05:07:36.431 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/198' -> 'copyofsnap1/test1/198' 2019-12-05T05:07:36.431 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/198/10' -> 'copyofsnap1/test1/198/10' 2019-12-05T05:07:36.431 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/198/11' -> 'copyofsnap1/test1/198/11' 2019-12-05T05:07:36.432 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/198/12' -> 'copyofsnap1/test1/198/12' 2019-12-05T05:07:36.432 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/198/13' -> 'copyofsnap1/test1/198/13' 2019-12-05T05:07:36.432 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/198/14' -> 'copyofsnap1/test1/198/14' 2019-12-05T05:07:36.432 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/198/15' -> 'copyofsnap1/test1/198/15' 2019-12-05T05:07:36.433 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/198/16' -> 'copyofsnap1/test1/198/16' 2019-12-05T05:07:36.433 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/198/17' -> 'copyofsnap1/test1/198/17' 2019-12-05T05:07:36.433 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/198/18' -> 'copyofsnap1/test1/198/18' 2019-12-05T05:07:36.433 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/198/19' -> 'copyofsnap1/test1/198/19' 2019-12-05T05:07:36.433 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/198/20' -> 'copyofsnap1/test1/198/20' 2019-12-05T05:07:36.433 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/199' -> 'copyofsnap1/test1/199' 2019-12-05T05:07:36.434 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/199/10' -> 'copyofsnap1/test1/199/10' 2019-12-05T05:07:36.434 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/199/11' -> 'copyofsnap1/test1/199/11' 2019-12-05T05:07:36.434 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/199/12' -> 'copyofsnap1/test1/199/12' 2019-12-05T05:07:36.434 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/199/13' -> 'copyofsnap1/test1/199/13' 2019-12-05T05:07:36.434 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/199/14' -> 'copyofsnap1/test1/199/14' 2019-12-05T05:07:36.434 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/199/15' -> 'copyofsnap1/test1/199/15' 2019-12-05T05:07:36.434 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/199/16' -> 'copyofsnap1/test1/199/16' 2019-12-05T05:07:36.435 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/199/17' -> 'copyofsnap1/test1/199/17' 2019-12-05T05:07:36.435 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/199/18' -> 'copyofsnap1/test1/199/18' 2019-12-05T05:07:36.437 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/199/19' -> 'copyofsnap1/test1/199/19' 2019-12-05T05:07:36.437 INFO:tasks.workunit.client.0.smithi042.stdout:'.snap/test1/199/20' -> 'copyofsnap1/test1/199/20' 2019-12-05T05:07:36.437 INFO:tasks.workunit.client.0.smithi042.stdout:Take third snapshot .snap/test3 2019-12-05T05:07:36.458 INFO:tasks.workunit.client.0.smithi042.stdout:Delete the snapshots... 2019-12-05T05:07:40.258 INFO:tasks.workunit.client.0.smithi042.stdout:.//.snap/test1 2019-12-05T05:07:40.258 INFO:tasks.workunit.client.0.smithi042.stdout:.//.snap/test2 2019-12-05T05:07:40.258 INFO:tasks.workunit.client.0.smithi042.stdout:.//.snap/test3 2019-12-05T05:07:41.171 INFO:tasks.workunit.client.0.smithi042.stdout:./274/.snap/snap-subdir-test 2019-12-05T05:07:41.197 INFO:tasks.workunit.client.0.smithi042.stdout:./166/.snap/snap-subdir-test 2019-12-05T05:07:42.380 INFO:tasks.workunit.client.0.smithi042.stdout:./256/.snap/snap-subdir-test 2019-12-05T05:07:42.407 INFO:tasks.workunit.client.0.smithi042.stdout:./247/.snap/snap-subdir-test 2019-12-05T05:07:42.431 INFO:tasks.workunit.client.0.smithi042.stdout:./248/.snap/snap-subdir-test 2019-12-05T05:07:42.459 INFO:tasks.workunit.client.0.smithi042.stdout:./147/.snap/snap-subdir-test 2019-12-05T05:07:42.485 INFO:tasks.workunit.client.0.smithi042.stdout:./151/.snap/snap-subdir-test 2019-12-05T05:07:42.760 INFO:tasks.workunit.client.0.smithi042.stdout:./170/.snap/snap-subdir-test 2019-12-05T05:07:42.788 INFO:tasks.workunit.client.0.smithi042.stdout:./167/.snap/snap-subdir-test 2019-12-05T05:07:42.812 INFO:tasks.workunit.client.0.smithi042.stdout:./140/.snap/snap-subdir-test 2019-12-05T05:07:42.835 INFO:tasks.workunit.client.0.smithi042.stdout:./142/.snap/snap-subdir-test 2019-12-05T05:07:42.865 INFO:tasks.workunit.client.0.smithi042.stdout:./150/.snap/snap-subdir-test 2019-12-05T05:07:42.881 INFO:tasks.workunit.client.0.smithi042.stdout:./273/.snap/snap-subdir-test 2019-12-05T05:07:42.903 INFO:tasks.workunit.client.0.smithi042.stdout:./269/.snap/snap-subdir-test 2019-12-05T05:07:42.928 INFO:tasks.workunit.client.0.smithi042.stdout:./105/.snap/snap-subdir-test 2019-12-05T05:07:42.950 INFO:tasks.workunit.client.0.smithi042.stdout:./210/.snap/snap-subdir-test 2019-12-05T05:07:42.973 INFO:tasks.workunit.client.0.smithi042.stdout:./265/.snap/snap-subdir-test 2019-12-05T05:07:42.996 INFO:tasks.workunit.client.0.smithi042.stdout:./218/.snap/snap-subdir-test 2019-12-05T05:07:43.017 INFO:tasks.workunit.client.0.smithi042.stdout:./290/.snap/snap-subdir-test 2019-12-05T05:07:43.042 INFO:tasks.workunit.client.0.smithi042.stdout:./149/.snap/snap-subdir-test 2019-12-05T05:07:43.067 INFO:tasks.workunit.client.0.smithi042.stdout:./109/.snap/snap-subdir-test 2019-12-05T05:07:43.106 INFO:tasks.workunit.client.0.smithi042.stdout:./292/.snap/snap-subdir-test 2019-12-05T05:07:43.121 INFO:tasks.workunit.client.0.smithi042.stdout:./242/.snap/snap-subdir-test 2019-12-05T05:07:43.523 INFO:tasks.workunit.client.0.smithi042.stdout:./158/.snap/snap-subdir-test 2019-12-05T05:07:43.549 INFO:tasks.workunit.client.0.smithi042.stdout:./243/.snap/snap-subdir-test 2019-12-05T05:07:43.573 INFO:tasks.workunit.client.0.smithi042.stdout:./106/.snap/snap-subdir-test 2019-12-05T05:07:43.593 INFO:tasks.workunit.client.0.smithi042.stdout:./250/.snap/snap-subdir-test 2019-12-05T05:07:43.613 INFO:tasks.workunit.client.0.smithi042.stdout:./221/.snap/snap-subdir-test 2019-12-05T05:07:43.633 INFO:tasks.workunit.client.0.smithi042.stdout:./244/.snap/snap-subdir-test 2019-12-05T05:07:43.654 INFO:tasks.workunit.client.0.smithi042.stdout:./258/.snap/snap-subdir-test 2019-12-05T05:07:43.677 INFO:tasks.workunit.client.0.smithi042.stdout:./268/.snap/snap-subdir-test 2019-12-05T05:07:43.701 INFO:tasks.workunit.client.0.smithi042.stdout:./189/.snap/snap-subdir-test 2019-12-05T05:07:43.722 INFO:tasks.workunit.client.0.smithi042.stdout:./285/.snap/snap-subdir-test 2019-12-05T05:07:43.746 INFO:tasks.workunit.client.0.smithi042.stdout:./208/.snap/snap-subdir-test 2019-12-05T05:07:43.767 INFO:tasks.workunit.client.0.smithi042.stdout:./288/.snap/snap-subdir-test 2019-12-05T05:07:43.790 INFO:tasks.workunit.client.0.smithi042.stdout:./209/.snap/snap-subdir-test 2019-12-05T05:07:43.812 INFO:tasks.workunit.client.0.smithi042.stdout:./253/.snap/snap-subdir-test 2019-12-05T05:07:43.835 INFO:tasks.workunit.client.0.smithi042.stdout:./249/.snap/snap-subdir-test 2019-12-05T05:07:43.858 INFO:tasks.workunit.client.0.smithi042.stdout:./133/.snap/snap-subdir-test 2019-12-05T05:07:43.880 INFO:tasks.workunit.client.0.smithi042.stdout:./291/.snap/snap-subdir-test 2019-12-05T05:07:43.902 INFO:tasks.workunit.client.0.smithi042.stdout:./226/.snap/snap-subdir-test 2019-12-05T05:07:43.926 INFO:tasks.workunit.client.0.smithi042.stdout:./173/.snap/snap-subdir-test 2019-12-05T05:07:43.951 INFO:tasks.workunit.client.0.smithi042.stdout:./174/.snap/snap-subdir-test 2019-12-05T05:07:43.974 INFO:tasks.workunit.client.0.smithi042.stdout:./299/.snap/snap-subdir-test 2019-12-05T05:07:43.997 INFO:tasks.workunit.client.0.smithi042.stdout:./262/.snap/snap-subdir-test 2019-12-05T05:07:44.020 INFO:tasks.workunit.client.0.smithi042.stdout:./231/.snap/snap-subdir-test 2019-12-05T05:07:44.042 INFO:tasks.workunit.client.0.smithi042.stdout:./214/.snap/snap-subdir-test 2019-12-05T05:07:44.064 INFO:tasks.workunit.client.0.smithi042.stdout:./281/.snap/snap-subdir-test 2019-12-05T05:07:44.086 INFO:tasks.workunit.client.0.smithi042.stdout:./289/.snap/snap-subdir-test 2019-12-05T05:07:44.106 INFO:tasks.workunit.client.0.smithi042.stdout:./283/.snap/snap-subdir-test 2019-12-05T05:07:44.127 INFO:tasks.workunit.client.0.smithi042.stdout:./232/.snap/snap-subdir-test 2019-12-05T05:07:44.149 INFO:tasks.workunit.client.0.smithi042.stdout:./298/.snap/snap-subdir-test 2019-12-05T05:07:44.172 INFO:tasks.workunit.client.0.smithi042.stdout:./233/.snap/snap-subdir-test 2019-12-05T05:07:44.196 INFO:tasks.workunit.client.0.smithi042.stdout:./155/.snap/snap-subdir-test 2019-12-05T05:07:44.220 INFO:tasks.workunit.client.0.smithi042.stdout:./114/.snap/snap-subdir-test 2019-12-05T05:07:44.242 INFO:tasks.workunit.client.0.smithi042.stdout:./159/.snap/snap-subdir-test 2019-12-05T05:07:44.269 INFO:tasks.workunit.client.0.smithi042.stdout:./187/.snap/snap-subdir-test 2019-12-05T05:07:44.291 INFO:tasks.workunit.client.0.smithi042.stdout:./185/.snap/snap-subdir-test 2019-12-05T05:07:44.314 INFO:tasks.workunit.client.0.smithi042.stdout:./293/.snap/snap-subdir-test 2019-12-05T05:07:44.338 INFO:tasks.workunit.client.0.smithi042.stdout:./228/.snap/snap-subdir-test 2019-12-05T05:07:45.282 INFO:tasks.workunit.client.0.smithi042.stdout:./120/.snap/snap-subdir-test 2019-12-05T05:07:45.309 INFO:tasks.workunit.client.0.smithi042.stdout:./110/.snap/snap-subdir-test 2019-12-05T05:07:45.331 INFO:tasks.workunit.client.0.smithi042.stdout:./267/.snap/snap-subdir-test 2019-12-05T05:07:45.356 INFO:tasks.workunit.client.0.smithi042.stdout:./202/.snap/snap-subdir-test 2019-12-05T05:07:45.383 INFO:tasks.workunit.client.0.smithi042.stdout:./137/.snap/snap-subdir-test 2019-12-05T05:07:45.406 INFO:tasks.workunit.client.0.smithi042.stdout:./246/.snap/snap-subdir-test 2019-12-05T05:07:45.429 INFO:tasks.workunit.client.0.smithi042.stdout:./219/.snap/snap-subdir-test 2019-12-05T05:07:45.455 INFO:tasks.workunit.client.0.smithi042.stdout:./198/.snap/snap-subdir-test 2019-12-05T05:07:45.480 INFO:tasks.workunit.client.0.smithi042.stdout:./136/.snap/snap-subdir-test 2019-12-05T05:07:45.501 INFO:tasks.workunit.client.0.smithi042.stdout:./270/.snap/snap-subdir-test 2019-12-05T05:07:45.527 INFO:tasks.workunit.client.0.smithi042.stdout:./125/.snap/snap-subdir-test 2019-12-05T05:07:45.553 INFO:tasks.workunit.client.0.smithi042.stdout:./194/.snap/snap-subdir-test 2019-12-05T05:07:45.575 INFO:tasks.workunit.client.0.smithi042.stdout:./119/.snap/snap-subdir-test 2019-12-05T05:07:45.595 INFO:tasks.workunit.client.0.smithi042.stdout:./297/.snap/snap-subdir-test 2019-12-05T05:07:45.616 INFO:tasks.workunit.client.0.smithi042.stdout:./230/.snap/snap-subdir-test 2019-12-05T05:07:45.636 INFO:tasks.workunit.client.0.smithi042.stdout:./260/.snap/snap-subdir-test 2019-12-05T05:07:45.657 INFO:tasks.workunit.client.0.smithi042.stdout:./157/.snap/snap-subdir-test 2019-12-05T05:07:45.680 INFO:tasks.workunit.client.0.smithi042.stdout:./172/.snap/snap-subdir-test 2019-12-05T05:07:45.701 INFO:tasks.workunit.client.0.smithi042.stdout:./277/.snap/snap-subdir-test 2019-12-05T05:07:45.725 INFO:tasks.workunit.client.0.smithi042.stdout:./186/.snap/snap-subdir-test 2019-12-05T05:07:45.748 INFO:tasks.workunit.client.0.smithi042.stdout:./237/.snap/snap-subdir-test 2019-12-05T05:07:45.776 INFO:tasks.workunit.client.0.smithi042.stdout:./190/.snap/snap-subdir-test 2019-12-05T05:07:45.795 INFO:tasks.workunit.client.0.smithi042.stdout:./213/.snap/snap-subdir-test 2019-12-05T05:07:45.819 INFO:tasks.workunit.client.0.smithi042.stdout:./104/.snap/snap-subdir-test 2019-12-05T05:07:45.844 INFO:tasks.workunit.client.0.smithi042.stdout:./164/.snap/snap-subdir-test 2019-12-05T05:07:45.868 INFO:tasks.workunit.client.0.smithi042.stdout:./112/.snap/snap-subdir-test 2019-12-05T05:07:45.892 INFO:tasks.workunit.client.0.smithi042.stdout:./295/.snap/snap-subdir-test 2019-12-05T05:07:45.913 INFO:tasks.workunit.client.0.smithi042.stdout:./272/.snap/snap-subdir-test 2019-12-05T05:07:45.939 INFO:tasks.workunit.client.0.smithi042.stdout:./100/.snap/snap-subdir-test 2019-12-05T05:07:45.968 INFO:tasks.workunit.client.0.smithi042.stdout:./134/.snap/snap-subdir-test 2019-12-05T05:07:45.985 INFO:tasks.workunit.client.0.smithi042.stdout:./200/.snap/snap-subdir-test 2019-12-05T05:07:46.008 INFO:tasks.workunit.client.0.smithi042.stdout:./115/.snap/snap-subdir-test 2019-12-05T05:07:46.029 INFO:tasks.workunit.client.0.smithi042.stdout:./284/.snap/snap-subdir-test 2019-12-05T05:07:46.051 INFO:tasks.workunit.client.0.smithi042.stdout:./280/.snap/snap-subdir-test 2019-12-05T05:07:46.077 INFO:tasks.workunit.client.0.smithi042.stdout:./116/.snap/snap-subdir-test 2019-12-05T05:07:46.099 INFO:tasks.workunit.client.0.smithi042.stdout:./245/.snap/snap-subdir-test 2019-12-05T05:07:46.122 INFO:tasks.workunit.client.0.smithi042.stdout:./195/.snap/snap-subdir-test 2019-12-05T05:07:46.146 INFO:tasks.workunit.client.0.smithi042.stdout:./138/.snap/snap-subdir-test 2019-12-05T05:07:46.168 INFO:tasks.workunit.client.0.smithi042.stdout:./135/.snap/snap-subdir-test 2019-12-05T05:07:46.190 INFO:tasks.workunit.client.0.smithi042.stdout:./117/.snap/snap-subdir-test 2019-12-05T05:07:46.214 INFO:tasks.workunit.client.0.smithi042.stdout:./241/.snap/snap-subdir-test 2019-12-05T05:07:46.238 INFO:tasks.workunit.client.0.smithi042.stdout:./143/.snap/snap-subdir-test 2019-12-05T05:07:46.262 INFO:tasks.workunit.client.0.smithi042.stdout:./180/.snap/snap-subdir-test 2019-12-05T05:07:46.287 INFO:tasks.workunit.client.0.smithi042.stdout:./126/.snap/snap-subdir-test 2019-12-05T05:07:46.312 INFO:tasks.workunit.client.0.smithi042.stdout:./161/.snap/snap-subdir-test 2019-12-05T05:07:46.336 INFO:tasks.workunit.client.0.smithi042.stdout:./204/.snap/snap-subdir-test 2019-12-05T05:07:46.362 INFO:tasks.workunit.client.0.smithi042.stdout:./176/.snap/snap-subdir-test 2019-12-05T05:07:46.385 INFO:tasks.workunit.client.0.smithi042.stdout:./255/.snap/snap-subdir-test 2019-12-05T05:07:46.406 INFO:tasks.workunit.client.0.smithi042.stdout:./215/.snap/snap-subdir-test 2019-12-05T05:07:46.430 INFO:tasks.workunit.client.0.smithi042.stdout:./111/.snap/snap-subdir-test 2019-12-05T05:07:46.453 INFO:tasks.workunit.client.0.smithi042.stdout:./238/.snap/snap-subdir-test 2019-12-05T05:07:46.776 INFO:tasks.workunit.client.0.smithi042.stdout:./191/.snap/snap-subdir-test 2019-12-05T05:07:46.803 INFO:tasks.workunit.client.0.smithi042.stdout:./184/.snap/snap-subdir-test 2019-12-05T05:07:46.828 INFO:tasks.workunit.client.0.smithi042.stdout:./192/.snap/snap-subdir-test 2019-12-05T05:07:46.855 INFO:tasks.workunit.client.0.smithi042.stdout:./103/.snap/snap-subdir-test 2019-12-05T05:07:46.877 INFO:tasks.workunit.client.0.smithi042.stdout:./128/.snap/snap-subdir-test 2019-12-05T05:07:47.399 INFO:tasks.workunit.client.0.smithi042.stdout:./229/.snap/snap-subdir-test 2019-12-05T05:07:47.425 INFO:tasks.workunit.client.0.smithi042.stdout:./144/.snap/snap-subdir-test 2019-12-05T05:07:47.450 INFO:tasks.workunit.client.0.smithi042.stdout:./146/.snap/snap-subdir-test 2019-12-05T05:07:47.474 INFO:tasks.workunit.client.0.smithi042.stdout:./227/.snap/snap-subdir-test 2019-12-05T05:07:47.498 INFO:tasks.workunit.client.0.smithi042.stdout:./278/.snap/snap-subdir-test 2019-12-05T05:07:47.524 INFO:tasks.workunit.client.0.smithi042.stdout:./141/.snap/snap-subdir-test 2019-12-05T05:07:47.538 INFO:tasks.workunit.client.0.smithi042.stdout:./234/.snap/snap-subdir-test 2019-12-05T05:07:47.561 INFO:tasks.workunit.client.0.smithi042.stdout:./225/.snap/snap-subdir-test 2019-12-05T05:07:47.582 INFO:tasks.workunit.client.0.smithi042.stdout:./182/.snap/snap-subdir-test 2019-12-05T05:07:47.606 INFO:tasks.workunit.client.0.smithi042.stdout:./107/.snap/snap-subdir-test 2019-12-05T05:07:47.631 INFO:tasks.workunit.client.0.smithi042.stdout:./178/.snap/snap-subdir-test 2019-12-05T05:07:47.653 INFO:tasks.workunit.client.0.smithi042.stdout:./266/.snap/snap-subdir-test 2019-12-05T05:07:47.676 INFO:tasks.workunit.client.0.smithi042.stdout:./240/.snap/snap-subdir-test 2019-12-05T05:07:47.698 INFO:tasks.workunit.client.0.smithi042.stdout:./156/.snap/snap-subdir-test 2019-12-05T05:07:47.723 INFO:tasks.workunit.client.0.smithi042.stdout:./131/.snap/snap-subdir-test 2019-12-05T05:07:47.744 INFO:tasks.workunit.client.0.smithi042.stdout:./123/.snap/snap-subdir-test 2019-12-05T05:07:47.768 INFO:tasks.workunit.client.0.smithi042.stdout:./160/.snap/snap-subdir-test 2019-12-05T05:07:47.791 INFO:tasks.workunit.client.0.smithi042.stdout:./165/.snap/snap-subdir-test 2019-12-05T05:07:47.817 INFO:tasks.workunit.client.0.smithi042.stdout:./101/.snap/snap-subdir-test 2019-12-05T05:07:47.840 INFO:tasks.workunit.client.0.smithi042.stdout:./148/.snap/snap-subdir-test 2019-12-05T05:07:47.866 INFO:tasks.workunit.client.0.smithi042.stdout:./168/.snap/snap-subdir-test 2019-12-05T05:07:47.893 INFO:tasks.workunit.client.0.smithi042.stdout:./179/.snap/snap-subdir-test 2019-12-05T05:07:47.917 INFO:tasks.workunit.client.0.smithi042.stdout:./257/.snap/snap-subdir-test 2019-12-05T05:07:47.938 INFO:tasks.workunit.client.0.smithi042.stdout:./252/.snap/snap-subdir-test 2019-12-05T05:07:47.962 INFO:tasks.workunit.client.0.smithi042.stdout:./132/.snap/snap-subdir-test 2019-12-05T05:07:48.887 INFO:tasks.workunit.client.0.smithi042.stdout:./236/.snap/snap-subdir-test 2019-12-05T05:07:48.911 INFO:tasks.workunit.client.0.smithi042.stdout:./211/.snap/snap-subdir-test 2019-12-05T05:07:48.937 INFO:tasks.workunit.client.0.smithi042.stdout:./199/.snap/snap-subdir-test 2019-12-05T05:07:48.959 INFO:tasks.workunit.client.0.smithi042.stdout:./205/.snap/snap-subdir-test 2019-12-05T05:07:48.983 INFO:tasks.workunit.client.0.smithi042.stdout:./163/.snap/snap-subdir-test 2019-12-05T05:07:49.004 INFO:tasks.workunit.client.0.smithi042.stdout:./261/.snap/snap-subdir-test 2019-12-05T05:07:49.027 INFO:tasks.workunit.client.0.smithi042.stdout:./201/.snap/snap-subdir-test 2019-12-05T05:07:49.053 INFO:tasks.workunit.client.0.smithi042.stdout:./282/.snap/snap-subdir-test 2019-12-05T05:07:49.072 INFO:tasks.workunit.client.0.smithi042.stdout:./296/.snap/snap-subdir-test 2019-12-05T05:07:49.096 INFO:tasks.workunit.client.0.smithi042.stdout:./217/.snap/snap-subdir-test 2019-12-05T05:07:49.121 INFO:tasks.workunit.client.0.smithi042.stdout:./153/.snap/snap-subdir-test 2019-12-05T05:07:49.153 INFO:tasks.workunit.client.0.smithi042.stdout:./206/.snap/snap-subdir-test 2019-12-05T05:07:49.170 INFO:tasks.workunit.client.0.smithi042.stdout:./122/.snap/snap-subdir-test 2019-12-05T05:07:49.195 INFO:tasks.workunit.client.0.smithi042.stdout:./162/.snap/snap-subdir-test 2019-12-05T05:07:49.220 INFO:tasks.workunit.client.0.smithi042.stdout:./197/.snap/snap-subdir-test 2019-12-05T05:07:49.242 INFO:tasks.workunit.client.0.smithi042.stdout:./216/.snap/snap-subdir-test 2019-12-05T05:07:53.436 INFO:tasks.workunit.client.0.smithi042.stdout:./254/.snap/snap-subdir-test 2019-12-05T05:07:53.459 INFO:tasks.workunit.client.0.smithi042.stdout:./113/.snap/snap-subdir-test 2019-12-05T05:07:54.002 INFO:tasks.workunit.client.0.smithi042.stdout:./152/.snap/snap-subdir-test 2019-12-05T05:07:54.027 INFO:tasks.workunit.client.0.smithi042.stdout:./121/.snap/snap-subdir-test 2019-12-05T05:07:54.052 INFO:tasks.workunit.client.0.smithi042.stdout:./118/.snap/snap-subdir-test 2019-12-05T05:07:54.077 INFO:tasks.workunit.client.0.smithi042.stdout:./196/.snap/snap-subdir-test 2019-12-05T05:07:54.101 INFO:tasks.workunit.client.0.smithi042.stdout:./124/.snap/snap-subdir-test 2019-12-05T05:07:54.125 INFO:tasks.workunit.client.0.smithi042.stdout:./239/.snap/snap-subdir-test 2019-12-05T05:07:54.151 INFO:tasks.workunit.client.0.smithi042.stdout:./188/.snap/snap-subdir-test 2019-12-05T05:07:54.172 INFO:tasks.workunit.client.0.smithi042.stdout:./287/.snap/snap-subdir-test 2019-12-05T05:07:54.193 INFO:tasks.workunit.client.0.smithi042.stdout:./275/.snap/snap-subdir-test 2019-12-05T05:07:54.216 INFO:tasks.workunit.client.0.smithi042.stdout:./259/.snap/snap-subdir-test 2019-12-05T05:07:54.238 INFO:tasks.workunit.client.0.smithi042.stdout:./224/.snap/snap-subdir-test 2019-12-05T05:07:54.259 INFO:tasks.workunit.client.0.smithi042.stdout:./264/.snap/snap-subdir-test 2019-12-05T05:07:54.287 INFO:tasks.workunit.client.0.smithi042.stdout:./145/.snap/snap-subdir-test 2019-12-05T05:07:54.310 INFO:tasks.workunit.client.0.smithi042.stdout:./279/.snap/snap-subdir-test 2019-12-05T05:07:54.333 INFO:tasks.workunit.client.0.smithi042.stdout:./263/.snap/snap-subdir-test 2019-12-05T05:07:54.359 INFO:tasks.workunit.client.0.smithi042.stdout:./130/.snap/snap-subdir-test 2019-12-05T05:07:54.729 INFO:tasks.workunit.client.0.smithi042.stdout:./193/.snap/snap-subdir-test 2019-12-05T05:07:54.753 INFO:tasks.workunit.client.0.smithi042.stdout:./223/.snap/snap-subdir-test 2019-12-05T05:07:54.776 INFO:tasks.workunit.client.0.smithi042.stdout:./207/.snap/snap-subdir-test 2019-12-05T05:07:54.798 INFO:tasks.workunit.client.0.smithi042.stdout:./276/.snap/snap-subdir-test 2019-12-05T05:07:54.820 INFO:tasks.workunit.client.0.smithi042.stdout:./271/.snap/snap-subdir-test 2019-12-05T05:07:54.842 INFO:tasks.workunit.client.0.smithi042.stdout:./251/.snap/snap-subdir-test 2019-12-05T05:07:55.684 INFO:tasks.workunit.client.0.smithi042.stdout:./175/.snap/snap-subdir-test 2019-12-05T05:07:55.708 INFO:tasks.workunit.client.0.smithi042.stdout:./222/.snap/snap-subdir-test 2019-12-05T05:07:55.731 INFO:tasks.workunit.client.0.smithi042.stdout:./220/.snap/snap-subdir-test 2019-12-05T05:07:56.046 INFO:tasks.workunit.client.0.smithi042.stdout:./203/.snap/snap-subdir-test 2019-12-05T05:07:56.073 INFO:tasks.workunit.client.0.smithi042.stdout:./181/.snap/snap-subdir-test 2019-12-05T05:07:56.094 INFO:tasks.workunit.client.0.smithi042.stdout:./286/.snap/snap-subdir-test 2019-12-05T05:07:56.118 INFO:tasks.workunit.client.0.smithi042.stdout:./127/.snap/snap-subdir-test 2019-12-05T05:07:56.665 INFO:tasks.workunit.client.0.smithi042.stdout:./108/.snap/snap-subdir-test 2019-12-05T05:07:56.690 INFO:tasks.workunit.client.0.smithi042.stdout:./183/.snap/snap-subdir-test 2019-12-05T05:07:56.711 INFO:tasks.workunit.client.0.smithi042.stdout:./294/.snap/snap-subdir-test 2019-12-05T05:07:56.731 INFO:tasks.workunit.client.0.smithi042.stdout:./235/.snap/snap-subdir-test 2019-12-05T05:07:56.753 INFO:tasks.workunit.client.0.smithi042.stdout:./177/.snap/snap-subdir-test 2019-12-05T05:07:56.778 INFO:tasks.workunit.client.0.smithi042.stdout:./169/.snap/snap-subdir-test 2019-12-05T05:07:56.801 INFO:tasks.workunit.client.0.smithi042.stdout:./212/.snap/snap-subdir-test 2019-12-05T05:07:56.826 INFO:tasks.workunit.client.0.smithi042.stdout:./154/.snap/snap-subdir-test 2019-12-05T05:07:56.852 INFO:tasks.workunit.client.0.smithi042.stdout:./171/.snap/snap-subdir-test 2019-12-05T05:07:56.877 INFO:tasks.workunit.client.0.smithi042.stdout:./139/.snap/snap-subdir-test 2019-12-05T05:07:56.902 INFO:tasks.workunit.client.0.smithi042.stdout:./102/.snap/snap-subdir-test 2019-12-05T05:07:56.928 INFO:tasks.workunit.client.0.smithi042.stdout:./129/.snap/snap-subdir-test 2019-12-05T05:08:04.410 INFO:tasks.workunit.client.0.smithi042.stdout:Delete all the files and directories ... 2019-12-05T05:08:04.840 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:08:04.844 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:08:08.238 INFO:tasks.workunit.client.0.smithi042.stdout:removed './100/19' 2019-12-05T05:08:08.238 INFO:tasks.workunit.client.0.smithi042.stdout:removed './100/15' 2019-12-05T05:08:08.238 INFO:tasks.workunit.client.0.smithi042.stdout:removed './100/31' 2019-12-05T05:08:08.238 INFO:tasks.workunit.client.0.smithi042.stdout:removed './100/36' 2019-12-05T05:08:08.238 INFO:tasks.workunit.client.0.smithi042.stdout:removed './100/20' 2019-12-05T05:08:08.238 INFO:tasks.workunit.client.0.smithi042.stdout:removed './100/12' 2019-12-05T05:08:08.239 INFO:tasks.workunit.client.0.smithi042.stdout:removed './100/39' 2019-12-05T05:08:08.239 INFO:tasks.workunit.client.0.smithi042.stdout:removed './100/10' 2019-12-05T05:08:08.239 INFO:tasks.workunit.client.0.smithi042.stdout:removed './100/34' 2019-12-05T05:08:08.239 INFO:tasks.workunit.client.0.smithi042.stdout:removed './100/38' 2019-12-05T05:08:08.239 INFO:tasks.workunit.client.0.smithi042.stdout:removed './100/11' 2019-12-05T05:08:08.239 INFO:tasks.workunit.client.0.smithi042.stdout:removed './100/16' 2019-12-05T05:08:08.239 INFO:tasks.workunit.client.0.smithi042.stdout:removed './100/32' 2019-12-05T05:08:08.240 INFO:tasks.workunit.client.0.smithi042.stdout:removed './100/30' 2019-12-05T05:08:08.240 INFO:tasks.workunit.client.0.smithi042.stdout:removed './100/14' 2019-12-05T05:08:08.240 INFO:tasks.workunit.client.0.smithi042.stdout:removed './100/18' 2019-12-05T05:08:08.240 INFO:tasks.workunit.client.0.smithi042.stdout:removed './100/13' 2019-12-05T05:08:08.240 INFO:tasks.workunit.client.0.smithi042.stdout:removed './100/33' 2019-12-05T05:08:08.240 INFO:tasks.workunit.client.0.smithi042.stdout:removed './100/17' 2019-12-05T05:08:08.240 INFO:tasks.workunit.client.0.smithi042.stdout:removed './100/35' 2019-12-05T05:08:08.241 INFO:tasks.workunit.client.0.smithi042.stdout:removed './100/37' 2019-12-05T05:08:08.241 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './100' 2019-12-05T05:08:08.241 INFO:tasks.workunit.client.0.smithi042.stdout:removed './101/19' 2019-12-05T05:08:08.241 INFO:tasks.workunit.client.0.smithi042.stdout:removed './101/15' 2019-12-05T05:08:08.241 INFO:tasks.workunit.client.0.smithi042.stdout:removed './101/31' 2019-12-05T05:08:08.241 INFO:tasks.workunit.client.0.smithi042.stdout:removed './101/36' 2019-12-05T05:08:08.241 INFO:tasks.workunit.client.0.smithi042.stdout:removed './101/20' 2019-12-05T05:08:08.242 INFO:tasks.workunit.client.0.smithi042.stdout:removed './101/12' 2019-12-05T05:08:08.242 INFO:tasks.workunit.client.0.smithi042.stdout:removed './101/39' 2019-12-05T05:08:08.242 INFO:tasks.workunit.client.0.smithi042.stdout:removed './101/10' 2019-12-05T05:08:08.242 INFO:tasks.workunit.client.0.smithi042.stdout:removed './101/34' 2019-12-05T05:08:08.242 INFO:tasks.workunit.client.0.smithi042.stdout:removed './101/38' 2019-12-05T05:08:08.242 INFO:tasks.workunit.client.0.smithi042.stdout:removed './101/11' 2019-12-05T05:08:08.242 INFO:tasks.workunit.client.0.smithi042.stdout:removed './101/16' 2019-12-05T05:08:08.243 INFO:tasks.workunit.client.0.smithi042.stdout:removed './101/32' 2019-12-05T05:08:08.243 INFO:tasks.workunit.client.0.smithi042.stdout:removed './101/30' 2019-12-05T05:08:08.243 INFO:tasks.workunit.client.0.smithi042.stdout:removed './101/14' 2019-12-05T05:08:08.243 INFO:tasks.workunit.client.0.smithi042.stdout:removed './101/18' 2019-12-05T05:08:08.243 INFO:tasks.workunit.client.0.smithi042.stdout:removed './101/13' 2019-12-05T05:08:08.243 INFO:tasks.workunit.client.0.smithi042.stdout:removed './101/33' 2019-12-05T05:08:08.243 INFO:tasks.workunit.client.0.smithi042.stdout:removed './101/17' 2019-12-05T05:08:08.243 INFO:tasks.workunit.client.0.smithi042.stdout:removed './101/35' 2019-12-05T05:08:08.244 INFO:tasks.workunit.client.0.smithi042.stdout:removed './101/37' 2019-12-05T05:08:08.244 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './101' 2019-12-05T05:08:08.244 INFO:tasks.workunit.client.0.smithi042.stdout:removed './102/19' 2019-12-05T05:08:08.244 INFO:tasks.workunit.client.0.smithi042.stdout:removed './102/15' 2019-12-05T05:08:08.244 INFO:tasks.workunit.client.0.smithi042.stdout:removed './102/31' 2019-12-05T05:08:08.244 INFO:tasks.workunit.client.0.smithi042.stdout:removed './102/36' 2019-12-05T05:08:08.244 INFO:tasks.workunit.client.0.smithi042.stdout:removed './102/20' 2019-12-05T05:08:08.245 INFO:tasks.workunit.client.0.smithi042.stdout:removed './102/12' 2019-12-05T05:08:08.245 INFO:tasks.workunit.client.0.smithi042.stdout:removed './102/39' 2019-12-05T05:08:08.245 INFO:tasks.workunit.client.0.smithi042.stdout:removed './102/10' 2019-12-05T05:08:08.245 INFO:tasks.workunit.client.0.smithi042.stdout:removed './102/34' 2019-12-05T05:08:08.245 INFO:tasks.workunit.client.0.smithi042.stdout:removed './102/38' 2019-12-05T05:08:08.246 INFO:tasks.workunit.client.0.smithi042.stdout:removed './102/11' 2019-12-05T05:08:08.246 INFO:tasks.workunit.client.0.smithi042.stdout:removed './102/16' 2019-12-05T05:08:08.246 INFO:tasks.workunit.client.0.smithi042.stdout:removed './102/32' 2019-12-05T05:08:08.246 INFO:tasks.workunit.client.0.smithi042.stdout:removed './102/30' 2019-12-05T05:08:08.246 INFO:tasks.workunit.client.0.smithi042.stdout:removed './102/14' 2019-12-05T05:08:08.246 INFO:tasks.workunit.client.0.smithi042.stdout:removed './102/18' 2019-12-05T05:08:08.246 INFO:tasks.workunit.client.0.smithi042.stdout:removed './102/13' 2019-12-05T05:08:08.247 INFO:tasks.workunit.client.0.smithi042.stdout:removed './102/33' 2019-12-05T05:08:08.247 INFO:tasks.workunit.client.0.smithi042.stdout:removed './102/17' 2019-12-05T05:08:08.247 INFO:tasks.workunit.client.0.smithi042.stdout:removed './102/35' 2019-12-05T05:08:08.247 INFO:tasks.workunit.client.0.smithi042.stdout:removed './102/37' 2019-12-05T05:08:08.247 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './102' 2019-12-05T05:08:08.247 INFO:tasks.workunit.client.0.smithi042.stdout:removed './103/19' 2019-12-05T05:08:08.247 INFO:tasks.workunit.client.0.smithi042.stdout:removed './103/15' 2019-12-05T05:08:08.247 INFO:tasks.workunit.client.0.smithi042.stdout:removed './103/31' 2019-12-05T05:08:08.248 INFO:tasks.workunit.client.0.smithi042.stdout:removed './103/36' 2019-12-05T05:08:08.248 INFO:tasks.workunit.client.0.smithi042.stdout:removed './103/20' 2019-12-05T05:08:08.248 INFO:tasks.workunit.client.0.smithi042.stdout:removed './103/12' 2019-12-05T05:08:08.248 INFO:tasks.workunit.client.0.smithi042.stdout:removed './103/39' 2019-12-05T05:08:08.248 INFO:tasks.workunit.client.0.smithi042.stdout:removed './103/10' 2019-12-05T05:08:08.248 INFO:tasks.workunit.client.0.smithi042.stdout:removed './103/34' 2019-12-05T05:08:08.248 INFO:tasks.workunit.client.0.smithi042.stdout:removed './103/38' 2019-12-05T05:08:08.248 INFO:tasks.workunit.client.0.smithi042.stdout:removed './103/11' 2019-12-05T05:08:08.249 INFO:tasks.workunit.client.0.smithi042.stdout:removed './103/16' 2019-12-05T05:08:08.249 INFO:tasks.workunit.client.0.smithi042.stdout:removed './103/32' 2019-12-05T05:08:08.249 INFO:tasks.workunit.client.0.smithi042.stdout:removed './103/30' 2019-12-05T05:08:08.249 INFO:tasks.workunit.client.0.smithi042.stdout:removed './103/14' 2019-12-05T05:08:08.249 INFO:tasks.workunit.client.0.smithi042.stdout:removed './103/18' 2019-12-05T05:08:08.249 INFO:tasks.workunit.client.0.smithi042.stdout:removed './103/13' 2019-12-05T05:08:08.250 INFO:tasks.workunit.client.0.smithi042.stdout:removed './103/33' 2019-12-05T05:08:08.250 INFO:tasks.workunit.client.0.smithi042.stdout:removed './103/17' 2019-12-05T05:08:08.250 INFO:tasks.workunit.client.0.smithi042.stdout:removed './103/35' 2019-12-05T05:08:08.251 INFO:tasks.workunit.client.0.smithi042.stdout:removed './103/37' 2019-12-05T05:08:08.251 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './103' 2019-12-05T05:08:08.251 INFO:tasks.workunit.client.0.smithi042.stdout:removed './104/19' 2019-12-05T05:08:08.251 INFO:tasks.workunit.client.0.smithi042.stdout:removed './104/15' 2019-12-05T05:08:08.251 INFO:tasks.workunit.client.0.smithi042.stdout:removed './104/31' 2019-12-05T05:08:08.251 INFO:tasks.workunit.client.0.smithi042.stdout:removed './104/36' 2019-12-05T05:08:08.252 INFO:tasks.workunit.client.0.smithi042.stdout:removed './104/20' 2019-12-05T05:08:08.252 INFO:tasks.workunit.client.0.smithi042.stdout:removed './104/12' 2019-12-05T05:08:08.252 INFO:tasks.workunit.client.0.smithi042.stdout:removed './104/39' 2019-12-05T05:08:08.252 INFO:tasks.workunit.client.0.smithi042.stdout:removed './104/10' 2019-12-05T05:08:08.252 INFO:tasks.workunit.client.0.smithi042.stdout:removed './104/34' 2019-12-05T05:08:08.252 INFO:tasks.workunit.client.0.smithi042.stdout:removed './104/38' 2019-12-05T05:08:08.252 INFO:tasks.workunit.client.0.smithi042.stdout:removed './104/11' 2019-12-05T05:08:08.253 INFO:tasks.workunit.client.0.smithi042.stdout:removed './104/16' 2019-12-05T05:08:08.253 INFO:tasks.workunit.client.0.smithi042.stdout:removed './104/32' 2019-12-05T05:08:08.253 INFO:tasks.workunit.client.0.smithi042.stdout:removed './104/30' 2019-12-05T05:08:08.253 INFO:tasks.workunit.client.0.smithi042.stdout:removed './104/14' 2019-12-05T05:08:08.253 INFO:tasks.workunit.client.0.smithi042.stdout:removed './104/18' 2019-12-05T05:08:08.253 INFO:tasks.workunit.client.0.smithi042.stdout:removed './104/13' 2019-12-05T05:08:08.253 INFO:tasks.workunit.client.0.smithi042.stdout:removed './104/33' 2019-12-05T05:08:08.254 INFO:tasks.workunit.client.0.smithi042.stdout:removed './104/17' 2019-12-05T05:08:08.254 INFO:tasks.workunit.client.0.smithi042.stdout:removed './104/35' 2019-12-05T05:08:08.254 INFO:tasks.workunit.client.0.smithi042.stdout:removed './104/37' 2019-12-05T05:08:08.254 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './104' 2019-12-05T05:08:08.254 INFO:tasks.workunit.client.0.smithi042.stdout:removed './105/19' 2019-12-05T05:08:08.254 INFO:tasks.workunit.client.0.smithi042.stdout:removed './105/15' 2019-12-05T05:08:08.254 INFO:tasks.workunit.client.0.smithi042.stdout:removed './105/31' 2019-12-05T05:08:08.255 INFO:tasks.workunit.client.0.smithi042.stdout:removed './105/36' 2019-12-05T05:08:08.255 INFO:tasks.workunit.client.0.smithi042.stdout:removed './105/20' 2019-12-05T05:08:08.255 INFO:tasks.workunit.client.0.smithi042.stdout:removed './105/12' 2019-12-05T05:08:08.255 INFO:tasks.workunit.client.0.smithi042.stdout:removed './105/39' 2019-12-05T05:08:08.255 INFO:tasks.workunit.client.0.smithi042.stdout:removed './105/10' 2019-12-05T05:08:08.255 INFO:tasks.workunit.client.0.smithi042.stdout:removed './105/34' 2019-12-05T05:08:08.255 INFO:tasks.workunit.client.0.smithi042.stdout:removed './105/38' 2019-12-05T05:08:08.255 INFO:tasks.workunit.client.0.smithi042.stdout:removed './105/11' 2019-12-05T05:08:08.256 INFO:tasks.workunit.client.0.smithi042.stdout:removed './105/16' 2019-12-05T05:08:08.256 INFO:tasks.workunit.client.0.smithi042.stdout:removed './105/32' 2019-12-05T05:08:08.256 INFO:tasks.workunit.client.0.smithi042.stdout:removed './105/30' 2019-12-05T05:08:08.256 INFO:tasks.workunit.client.0.smithi042.stdout:removed './105/14' 2019-12-05T05:08:08.256 INFO:tasks.workunit.client.0.smithi042.stdout:removed './105/18' 2019-12-05T05:08:08.256 INFO:tasks.workunit.client.0.smithi042.stdout:removed './105/13' 2019-12-05T05:08:08.256 INFO:tasks.workunit.client.0.smithi042.stdout:removed './105/33' 2019-12-05T05:08:08.257 INFO:tasks.workunit.client.0.smithi042.stdout:removed './105/17' 2019-12-05T05:08:08.257 INFO:tasks.workunit.client.0.smithi042.stdout:removed './105/35' 2019-12-05T05:08:08.257 INFO:tasks.workunit.client.0.smithi042.stdout:removed './105/37' 2019-12-05T05:08:08.257 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './105' 2019-12-05T05:08:08.257 INFO:tasks.workunit.client.0.smithi042.stdout:removed './106/19' 2019-12-05T05:08:08.257 INFO:tasks.workunit.client.0.smithi042.stdout:removed './106/15' 2019-12-05T05:08:08.257 INFO:tasks.workunit.client.0.smithi042.stdout:removed './106/31' 2019-12-05T05:08:08.258 INFO:tasks.workunit.client.0.smithi042.stdout:removed './106/36' 2019-12-05T05:08:08.258 INFO:tasks.workunit.client.0.smithi042.stdout:removed './106/20' 2019-12-05T05:08:08.258 INFO:tasks.workunit.client.0.smithi042.stdout:removed './106/12' 2019-12-05T05:08:08.258 INFO:tasks.workunit.client.0.smithi042.stdout:removed './106/39' 2019-12-05T05:08:08.258 INFO:tasks.workunit.client.0.smithi042.stdout:removed './106/10' 2019-12-05T05:08:08.258 INFO:tasks.workunit.client.0.smithi042.stdout:removed './106/34' 2019-12-05T05:08:08.258 INFO:tasks.workunit.client.0.smithi042.stdout:removed './106/38' 2019-12-05T05:08:08.259 INFO:tasks.workunit.client.0.smithi042.stdout:removed './106/11' 2019-12-05T05:08:08.259 INFO:tasks.workunit.client.0.smithi042.stdout:removed './106/16' 2019-12-05T05:08:08.259 INFO:tasks.workunit.client.0.smithi042.stdout:removed './106/32' 2019-12-05T05:08:08.259 INFO:tasks.workunit.client.0.smithi042.stdout:removed './106/30' 2019-12-05T05:08:08.259 INFO:tasks.workunit.client.0.smithi042.stdout:removed './106/14' 2019-12-05T05:08:08.259 INFO:tasks.workunit.client.0.smithi042.stdout:removed './106/18' 2019-12-05T05:08:08.260 INFO:tasks.workunit.client.0.smithi042.stdout:removed './106/13' 2019-12-05T05:08:08.260 INFO:tasks.workunit.client.0.smithi042.stdout:removed './106/33' 2019-12-05T05:08:08.260 INFO:tasks.workunit.client.0.smithi042.stdout:removed './106/17' 2019-12-05T05:08:08.260 INFO:tasks.workunit.client.0.smithi042.stdout:removed './106/35' 2019-12-05T05:08:08.260 INFO:tasks.workunit.client.0.smithi042.stdout:removed './106/37' 2019-12-05T05:08:08.260 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './106' 2019-12-05T05:08:08.261 INFO:tasks.workunit.client.0.smithi042.stdout:removed './107/19' 2019-12-05T05:08:08.261 INFO:tasks.workunit.client.0.smithi042.stdout:removed './107/15' 2019-12-05T05:08:08.261 INFO:tasks.workunit.client.0.smithi042.stdout:removed './107/31' 2019-12-05T05:08:08.261 INFO:tasks.workunit.client.0.smithi042.stdout:removed './107/36' 2019-12-05T05:08:08.261 INFO:tasks.workunit.client.0.smithi042.stdout:removed './107/20' 2019-12-05T05:08:08.261 INFO:tasks.workunit.client.0.smithi042.stdout:removed './107/12' 2019-12-05T05:08:08.261 INFO:tasks.workunit.client.0.smithi042.stdout:removed './107/39' 2019-12-05T05:08:08.261 INFO:tasks.workunit.client.0.smithi042.stdout:removed './107/10' 2019-12-05T05:08:08.262 INFO:tasks.workunit.client.0.smithi042.stdout:removed './107/34' 2019-12-05T05:08:08.262 INFO:tasks.workunit.client.0.smithi042.stdout:removed './107/38' 2019-12-05T05:08:08.262 INFO:tasks.workunit.client.0.smithi042.stdout:removed './107/11' 2019-12-05T05:08:08.262 INFO:tasks.workunit.client.0.smithi042.stdout:removed './107/16' 2019-12-05T05:08:08.262 INFO:tasks.workunit.client.0.smithi042.stdout:removed './107/32' 2019-12-05T05:08:08.262 INFO:tasks.workunit.client.0.smithi042.stdout:removed './107/30' 2019-12-05T05:08:08.262 INFO:tasks.workunit.client.0.smithi042.stdout:removed './107/14' 2019-12-05T05:08:08.263 INFO:tasks.workunit.client.0.smithi042.stdout:removed './107/18' 2019-12-05T05:08:08.263 INFO:tasks.workunit.client.0.smithi042.stdout:removed './107/13' 2019-12-05T05:08:08.263 INFO:tasks.workunit.client.0.smithi042.stdout:removed './107/33' 2019-12-05T05:08:08.263 INFO:tasks.workunit.client.0.smithi042.stdout:removed './107/17' 2019-12-05T05:08:08.263 INFO:tasks.workunit.client.0.smithi042.stdout:removed './107/35' 2019-12-05T05:08:08.263 INFO:tasks.workunit.client.0.smithi042.stdout:removed './107/37' 2019-12-05T05:08:08.263 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './107' 2019-12-05T05:08:08.263 INFO:tasks.workunit.client.0.smithi042.stdout:removed './108/19' 2019-12-05T05:08:08.264 INFO:tasks.workunit.client.0.smithi042.stdout:removed './108/15' 2019-12-05T05:08:08.264 INFO:tasks.workunit.client.0.smithi042.stdout:removed './108/31' 2019-12-05T05:08:08.264 INFO:tasks.workunit.client.0.smithi042.stdout:removed './108/36' 2019-12-05T05:08:08.264 INFO:tasks.workunit.client.0.smithi042.stdout:removed './108/20' 2019-12-05T05:08:08.264 INFO:tasks.workunit.client.0.smithi042.stdout:removed './108/12' 2019-12-05T05:08:08.264 INFO:tasks.workunit.client.0.smithi042.stdout:removed './108/39' 2019-12-05T05:08:08.264 INFO:tasks.workunit.client.0.smithi042.stdout:removed './108/10' 2019-12-05T05:08:08.264 INFO:tasks.workunit.client.0.smithi042.stdout:removed './108/34' 2019-12-05T05:08:08.265 INFO:tasks.workunit.client.0.smithi042.stdout:removed './108/38' 2019-12-05T05:08:08.265 INFO:tasks.workunit.client.0.smithi042.stdout:removed './108/11' 2019-12-05T05:08:08.265 INFO:tasks.workunit.client.0.smithi042.stdout:removed './108/16' 2019-12-05T05:08:08.265 INFO:tasks.workunit.client.0.smithi042.stdout:removed './108/32' 2019-12-05T05:08:08.265 INFO:tasks.workunit.client.0.smithi042.stdout:removed './108/30' 2019-12-05T05:08:08.265 INFO:tasks.workunit.client.0.smithi042.stdout:removed './108/14' 2019-12-05T05:08:08.265 INFO:tasks.workunit.client.0.smithi042.stdout:removed './108/18' 2019-12-05T05:08:08.266 INFO:tasks.workunit.client.0.smithi042.stdout:removed './108/13' 2019-12-05T05:08:08.266 INFO:tasks.workunit.client.0.smithi042.stdout:removed './108/33' 2019-12-05T05:08:08.266 INFO:tasks.workunit.client.0.smithi042.stdout:removed './108/17' 2019-12-05T05:08:08.266 INFO:tasks.workunit.client.0.smithi042.stdout:removed './108/35' 2019-12-05T05:08:08.266 INFO:tasks.workunit.client.0.smithi042.stdout:removed './108/37' 2019-12-05T05:08:08.266 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './108' 2019-12-05T05:08:08.266 INFO:tasks.workunit.client.0.smithi042.stdout:removed './109/19' 2019-12-05T05:08:08.267 INFO:tasks.workunit.client.0.smithi042.stdout:removed './109/15' 2019-12-05T05:08:08.267 INFO:tasks.workunit.client.0.smithi042.stdout:removed './109/31' 2019-12-05T05:08:08.267 INFO:tasks.workunit.client.0.smithi042.stdout:removed './109/36' 2019-12-05T05:08:08.267 INFO:tasks.workunit.client.0.smithi042.stdout:removed './109/20' 2019-12-05T05:08:08.267 INFO:tasks.workunit.client.0.smithi042.stdout:removed './109/12' 2019-12-05T05:08:08.267 INFO:tasks.workunit.client.0.smithi042.stdout:removed './109/39' 2019-12-05T05:08:08.267 INFO:tasks.workunit.client.0.smithi042.stdout:removed './109/10' 2019-12-05T05:08:08.268 INFO:tasks.workunit.client.0.smithi042.stdout:removed './109/34' 2019-12-05T05:08:08.268 INFO:tasks.workunit.client.0.smithi042.stdout:removed './109/38' 2019-12-05T05:08:08.268 INFO:tasks.workunit.client.0.smithi042.stdout:removed './109/11' 2019-12-05T05:08:08.268 INFO:tasks.workunit.client.0.smithi042.stdout:removed './109/16' 2019-12-05T05:08:08.268 INFO:tasks.workunit.client.0.smithi042.stdout:removed './109/32' 2019-12-05T05:08:08.270 INFO:tasks.workunit.client.0.smithi042.stdout:removed './109/30' 2019-12-05T05:08:11.615 INFO:tasks.workunit.client.0.smithi042.stdout:removed './109/14' 2019-12-05T05:08:11.615 INFO:tasks.workunit.client.0.smithi042.stdout:removed './109/18' 2019-12-05T05:08:11.615 INFO:tasks.workunit.client.0.smithi042.stdout:removed './109/13' 2019-12-05T05:08:11.615 INFO:tasks.workunit.client.0.smithi042.stdout:removed './109/33' 2019-12-05T05:08:11.615 INFO:tasks.workunit.client.0.smithi042.stdout:removed './109/17' 2019-12-05T05:08:11.616 INFO:tasks.workunit.client.0.smithi042.stdout:removed './109/35' 2019-12-05T05:08:11.616 INFO:tasks.workunit.client.0.smithi042.stdout:removed './109/37' 2019-12-05T05:08:11.616 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './109' 2019-12-05T05:08:11.616 INFO:tasks.workunit.client.0.smithi042.stdout:removed './110/19' 2019-12-05T05:08:11.616 INFO:tasks.workunit.client.0.smithi042.stdout:removed './110/15' 2019-12-05T05:08:11.616 INFO:tasks.workunit.client.0.smithi042.stdout:removed './110/31' 2019-12-05T05:08:11.616 INFO:tasks.workunit.client.0.smithi042.stdout:removed './110/36' 2019-12-05T05:08:11.617 INFO:tasks.workunit.client.0.smithi042.stdout:removed './110/20' 2019-12-05T05:08:11.617 INFO:tasks.workunit.client.0.smithi042.stdout:removed './110/12' 2019-12-05T05:08:11.617 INFO:tasks.workunit.client.0.smithi042.stdout:removed './110/39' 2019-12-05T05:08:11.617 INFO:tasks.workunit.client.0.smithi042.stdout:removed './110/10' 2019-12-05T05:08:11.617 INFO:tasks.workunit.client.0.smithi042.stdout:removed './110/34' 2019-12-05T05:08:11.617 INFO:tasks.workunit.client.0.smithi042.stdout:removed './110/38' 2019-12-05T05:08:11.617 INFO:tasks.workunit.client.0.smithi042.stdout:removed './110/11' 2019-12-05T05:08:11.617 INFO:tasks.workunit.client.0.smithi042.stdout:removed './110/16' 2019-12-05T05:08:11.618 INFO:tasks.workunit.client.0.smithi042.stdout:removed './110/32' 2019-12-05T05:08:11.618 INFO:tasks.workunit.client.0.smithi042.stdout:removed './110/30' 2019-12-05T05:08:11.618 INFO:tasks.workunit.client.0.smithi042.stdout:removed './110/14' 2019-12-05T05:08:11.618 INFO:tasks.workunit.client.0.smithi042.stdout:removed './110/18' 2019-12-05T05:08:11.618 INFO:tasks.workunit.client.0.smithi042.stdout:removed './110/13' 2019-12-05T05:08:11.619 INFO:tasks.workunit.client.0.smithi042.stdout:removed './110/33' 2019-12-05T05:08:11.619 INFO:tasks.workunit.client.0.smithi042.stdout:removed './110/17' 2019-12-05T05:08:11.619 INFO:tasks.workunit.client.0.smithi042.stdout:removed './110/35' 2019-12-05T05:08:11.619 INFO:tasks.workunit.client.0.smithi042.stdout:removed './110/37' 2019-12-05T05:08:11.619 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './110' 2019-12-05T05:08:11.619 INFO:tasks.workunit.client.0.smithi042.stdout:removed './111/19' 2019-12-05T05:08:11.619 INFO:tasks.workunit.client.0.smithi042.stdout:removed './111/15' 2019-12-05T05:08:11.620 INFO:tasks.workunit.client.0.smithi042.stdout:removed './111/31' 2019-12-05T05:08:11.620 INFO:tasks.workunit.client.0.smithi042.stdout:removed './111/36' 2019-12-05T05:08:11.620 INFO:tasks.workunit.client.0.smithi042.stdout:removed './111/20' 2019-12-05T05:08:11.620 INFO:tasks.workunit.client.0.smithi042.stdout:removed './111/12' 2019-12-05T05:08:11.620 INFO:tasks.workunit.client.0.smithi042.stdout:removed './111/39' 2019-12-05T05:08:11.620 INFO:tasks.workunit.client.0.smithi042.stdout:removed './111/10' 2019-12-05T05:08:11.620 INFO:tasks.workunit.client.0.smithi042.stdout:removed './111/34' 2019-12-05T05:08:11.620 INFO:tasks.workunit.client.0.smithi042.stdout:removed './111/38' 2019-12-05T05:08:11.621 INFO:tasks.workunit.client.0.smithi042.stdout:removed './111/11' 2019-12-05T05:08:11.621 INFO:tasks.workunit.client.0.smithi042.stdout:removed './111/16' 2019-12-05T05:08:11.621 INFO:tasks.workunit.client.0.smithi042.stdout:removed './111/32' 2019-12-05T05:08:11.621 INFO:tasks.workunit.client.0.smithi042.stdout:removed './111/30' 2019-12-05T05:08:11.621 INFO:tasks.workunit.client.0.smithi042.stdout:removed './111/14' 2019-12-05T05:08:11.621 INFO:tasks.workunit.client.0.smithi042.stdout:removed './111/18' 2019-12-05T05:08:11.622 INFO:tasks.workunit.client.0.smithi042.stdout:removed './111/13' 2019-12-05T05:08:11.622 INFO:tasks.workunit.client.0.smithi042.stdout:removed './111/33' 2019-12-05T05:08:11.622 INFO:tasks.workunit.client.0.smithi042.stdout:removed './111/17' 2019-12-05T05:08:11.623 INFO:tasks.workunit.client.0.smithi042.stdout:removed './111/35' 2019-12-05T05:08:11.623 INFO:tasks.workunit.client.0.smithi042.stdout:removed './111/37' 2019-12-05T05:08:11.623 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './111' 2019-12-05T05:08:11.623 INFO:tasks.workunit.client.0.smithi042.stdout:removed './112/19' 2019-12-05T05:08:11.623 INFO:tasks.workunit.client.0.smithi042.stdout:removed './112/15' 2019-12-05T05:08:11.623 INFO:tasks.workunit.client.0.smithi042.stdout:removed './112/31' 2019-12-05T05:08:11.623 INFO:tasks.workunit.client.0.smithi042.stdout:removed './112/36' 2019-12-05T05:08:11.624 INFO:tasks.workunit.client.0.smithi042.stdout:removed './112/20' 2019-12-05T05:08:11.624 INFO:tasks.workunit.client.0.smithi042.stdout:removed './112/12' 2019-12-05T05:08:11.624 INFO:tasks.workunit.client.0.smithi042.stdout:removed './112/39' 2019-12-05T05:08:11.624 INFO:tasks.workunit.client.0.smithi042.stdout:removed './112/10' 2019-12-05T05:08:11.624 INFO:tasks.workunit.client.0.smithi042.stdout:removed './112/34' 2019-12-05T05:08:11.624 INFO:tasks.workunit.client.0.smithi042.stdout:removed './112/38' 2019-12-05T05:08:11.624 INFO:tasks.workunit.client.0.smithi042.stdout:removed './112/11' 2019-12-05T05:08:11.625 INFO:tasks.workunit.client.0.smithi042.stdout:removed './112/16' 2019-12-05T05:08:11.625 INFO:tasks.workunit.client.0.smithi042.stdout:removed './112/32' 2019-12-05T05:08:11.625 INFO:tasks.workunit.client.0.smithi042.stdout:removed './112/30' 2019-12-05T05:08:11.625 INFO:tasks.workunit.client.0.smithi042.stdout:removed './112/14' 2019-12-05T05:08:11.625 INFO:tasks.workunit.client.0.smithi042.stdout:removed './112/18' 2019-12-05T05:08:11.625 INFO:tasks.workunit.client.0.smithi042.stdout:removed './112/13' 2019-12-05T05:08:11.626 INFO:tasks.workunit.client.0.smithi042.stdout:removed './112/33' 2019-12-05T05:08:11.626 INFO:tasks.workunit.client.0.smithi042.stdout:removed './112/17' 2019-12-05T05:08:11.626 INFO:tasks.workunit.client.0.smithi042.stdout:removed './112/35' 2019-12-05T05:08:11.626 INFO:tasks.workunit.client.0.smithi042.stdout:removed './112/37' 2019-12-05T05:08:11.626 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './112' 2019-12-05T05:08:11.627 INFO:tasks.workunit.client.0.smithi042.stdout:removed './113/19' 2019-12-05T05:08:11.627 INFO:tasks.workunit.client.0.smithi042.stdout:removed './113/15' 2019-12-05T05:08:11.627 INFO:tasks.workunit.client.0.smithi042.stdout:removed './113/31' 2019-12-05T05:08:11.627 INFO:tasks.workunit.client.0.smithi042.stdout:removed './113/36' 2019-12-05T05:08:11.627 INFO:tasks.workunit.client.0.smithi042.stdout:removed './113/20' 2019-12-05T05:08:11.628 INFO:tasks.workunit.client.0.smithi042.stdout:removed './113/12' 2019-12-05T05:08:11.628 INFO:tasks.workunit.client.0.smithi042.stdout:removed './113/39' 2019-12-05T05:08:11.628 INFO:tasks.workunit.client.0.smithi042.stdout:removed './113/10' 2019-12-05T05:08:11.628 INFO:tasks.workunit.client.0.smithi042.stdout:removed './113/34' 2019-12-05T05:08:11.628 INFO:tasks.workunit.client.0.smithi042.stdout:removed './113/38' 2019-12-05T05:08:11.629 INFO:tasks.workunit.client.0.smithi042.stdout:removed './113/11' 2019-12-05T05:08:11.629 INFO:tasks.workunit.client.0.smithi042.stdout:removed './113/16' 2019-12-05T05:08:11.629 INFO:tasks.workunit.client.0.smithi042.stdout:removed './113/32' 2019-12-05T05:08:11.629 INFO:tasks.workunit.client.0.smithi042.stdout:removed './113/30' 2019-12-05T05:08:11.629 INFO:tasks.workunit.client.0.smithi042.stdout:removed './113/14' 2019-12-05T05:08:11.629 INFO:tasks.workunit.client.0.smithi042.stdout:removed './113/18' 2019-12-05T05:08:11.629 INFO:tasks.workunit.client.0.smithi042.stdout:removed './113/13' 2019-12-05T05:08:11.630 INFO:tasks.workunit.client.0.smithi042.stdout:removed './113/33' 2019-12-05T05:08:11.630 INFO:tasks.workunit.client.0.smithi042.stdout:removed './113/17' 2019-12-05T05:08:11.630 INFO:tasks.workunit.client.0.smithi042.stdout:removed './113/35' 2019-12-05T05:08:11.630 INFO:tasks.workunit.client.0.smithi042.stdout:removed './113/37' 2019-12-05T05:08:11.630 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './113' 2019-12-05T05:08:11.630 INFO:tasks.workunit.client.0.smithi042.stdout:removed './114/19' 2019-12-05T05:08:11.630 INFO:tasks.workunit.client.0.smithi042.stdout:removed './114/15' 2019-12-05T05:08:11.630 INFO:tasks.workunit.client.0.smithi042.stdout:removed './114/31' 2019-12-05T05:08:11.631 INFO:tasks.workunit.client.0.smithi042.stdout:removed './114/36' 2019-12-05T05:08:11.631 INFO:tasks.workunit.client.0.smithi042.stdout:removed './114/20' 2019-12-05T05:08:11.631 INFO:tasks.workunit.client.0.smithi042.stdout:removed './114/12' 2019-12-05T05:08:11.631 INFO:tasks.workunit.client.0.smithi042.stdout:removed './114/39' 2019-12-05T05:08:11.631 INFO:tasks.workunit.client.0.smithi042.stdout:removed './114/10' 2019-12-05T05:08:11.631 INFO:tasks.workunit.client.0.smithi042.stdout:removed './114/34' 2019-12-05T05:08:11.631 INFO:tasks.workunit.client.0.smithi042.stdout:removed './114/38' 2019-12-05T05:08:11.631 INFO:tasks.workunit.client.0.smithi042.stdout:removed './114/11' 2019-12-05T05:08:11.632 INFO:tasks.workunit.client.0.smithi042.stdout:removed './114/16' 2019-12-05T05:08:11.632 INFO:tasks.workunit.client.0.smithi042.stdout:removed './114/32' 2019-12-05T05:08:11.632 INFO:tasks.workunit.client.0.smithi042.stdout:removed './114/30' 2019-12-05T05:08:11.633 INFO:tasks.workunit.client.0.smithi042.stdout:removed './114/14' 2019-12-05T05:08:11.633 INFO:tasks.workunit.client.0.smithi042.stdout:removed './114/18' 2019-12-05T05:08:11.633 INFO:tasks.workunit.client.0.smithi042.stdout:removed './114/13' 2019-12-05T05:08:11.633 INFO:tasks.workunit.client.0.smithi042.stdout:removed './114/33' 2019-12-05T05:08:11.633 INFO:tasks.workunit.client.0.smithi042.stdout:removed './114/17' 2019-12-05T05:08:11.634 INFO:tasks.workunit.client.0.smithi042.stdout:removed './114/35' 2019-12-05T05:08:11.634 INFO:tasks.workunit.client.0.smithi042.stdout:removed './114/37' 2019-12-05T05:08:11.634 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './114' 2019-12-05T05:08:11.634 INFO:tasks.workunit.client.0.smithi042.stdout:removed './115/19' 2019-12-05T05:08:11.634 INFO:tasks.workunit.client.0.smithi042.stdout:removed './115/15' 2019-12-05T05:08:11.634 INFO:tasks.workunit.client.0.smithi042.stdout:removed './115/31' 2019-12-05T05:08:11.634 INFO:tasks.workunit.client.0.smithi042.stdout:removed './115/36' 2019-12-05T05:08:11.635 INFO:tasks.workunit.client.0.smithi042.stdout:removed './115/20' 2019-12-05T05:08:11.635 INFO:tasks.workunit.client.0.smithi042.stdout:removed './115/12' 2019-12-05T05:08:11.635 INFO:tasks.workunit.client.0.smithi042.stdout:removed './115/39' 2019-12-05T05:08:11.635 INFO:tasks.workunit.client.0.smithi042.stdout:removed './115/10' 2019-12-05T05:08:11.635 INFO:tasks.workunit.client.0.smithi042.stdout:removed './115/34' 2019-12-05T05:08:11.635 INFO:tasks.workunit.client.0.smithi042.stdout:removed './115/38' 2019-12-05T05:08:11.635 INFO:tasks.workunit.client.0.smithi042.stdout:removed './115/11' 2019-12-05T05:08:11.636 INFO:tasks.workunit.client.0.smithi042.stdout:removed './115/16' 2019-12-05T05:08:11.636 INFO:tasks.workunit.client.0.smithi042.stdout:removed './115/32' 2019-12-05T05:08:11.636 INFO:tasks.workunit.client.0.smithi042.stdout:removed './115/30' 2019-12-05T05:08:11.636 INFO:tasks.workunit.client.0.smithi042.stdout:removed './115/14' 2019-12-05T05:08:11.636 INFO:tasks.workunit.client.0.smithi042.stdout:removed './115/18' 2019-12-05T05:08:11.636 INFO:tasks.workunit.client.0.smithi042.stdout:removed './115/13' 2019-12-05T05:08:11.637 INFO:tasks.workunit.client.0.smithi042.stdout:removed './115/33' 2019-12-05T05:08:11.637 INFO:tasks.workunit.client.0.smithi042.stdout:removed './115/17' 2019-12-05T05:08:11.637 INFO:tasks.workunit.client.0.smithi042.stdout:removed './115/35' 2019-12-05T05:08:11.637 INFO:tasks.workunit.client.0.smithi042.stdout:removed './115/37' 2019-12-05T05:08:11.637 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './115' 2019-12-05T05:08:11.637 INFO:tasks.workunit.client.0.smithi042.stdout:removed './116/19' 2019-12-05T05:08:11.637 INFO:tasks.workunit.client.0.smithi042.stdout:removed './116/15' 2019-12-05T05:08:11.637 INFO:tasks.workunit.client.0.smithi042.stdout:removed './116/31' 2019-12-05T05:08:11.638 INFO:tasks.workunit.client.0.smithi042.stdout:removed './116/36' 2019-12-05T05:08:11.638 INFO:tasks.workunit.client.0.smithi042.stdout:removed './116/20' 2019-12-05T05:08:11.638 INFO:tasks.workunit.client.0.smithi042.stdout:removed './116/12' 2019-12-05T05:08:11.638 INFO:tasks.workunit.client.0.smithi042.stdout:removed './116/39' 2019-12-05T05:08:11.638 INFO:tasks.workunit.client.0.smithi042.stdout:removed './116/10' 2019-12-05T05:08:11.638 INFO:tasks.workunit.client.0.smithi042.stdout:removed './116/34' 2019-12-05T05:08:11.638 INFO:tasks.workunit.client.0.smithi042.stdout:removed './116/38' 2019-12-05T05:08:11.639 INFO:tasks.workunit.client.0.smithi042.stdout:removed './116/11' 2019-12-05T05:08:11.639 INFO:tasks.workunit.client.0.smithi042.stdout:removed './116/16' 2019-12-05T05:08:11.639 INFO:tasks.workunit.client.0.smithi042.stdout:removed './116/32' 2019-12-05T05:08:11.639 INFO:tasks.workunit.client.0.smithi042.stdout:removed './116/30' 2019-12-05T05:08:11.639 INFO:tasks.workunit.client.0.smithi042.stdout:removed './116/14' 2019-12-05T05:08:11.640 INFO:tasks.workunit.client.0.smithi042.stdout:removed './116/18' 2019-12-05T05:08:11.640 INFO:tasks.workunit.client.0.smithi042.stdout:removed './116/13' 2019-12-05T05:08:11.640 INFO:tasks.workunit.client.0.smithi042.stdout:removed './116/33' 2019-12-05T05:08:11.640 INFO:tasks.workunit.client.0.smithi042.stdout:removed './116/17' 2019-12-05T05:08:11.640 INFO:tasks.workunit.client.0.smithi042.stdout:removed './116/35' 2019-12-05T05:08:11.641 INFO:tasks.workunit.client.0.smithi042.stdout:removed './116/37' 2019-12-05T05:08:11.641 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './116' 2019-12-05T05:08:11.641 INFO:tasks.workunit.client.0.smithi042.stdout:removed './117/19' 2019-12-05T05:08:11.641 INFO:tasks.workunit.client.0.smithi042.stdout:removed './117/15' 2019-12-05T05:08:11.641 INFO:tasks.workunit.client.0.smithi042.stdout:removed './117/31' 2019-12-05T05:08:11.641 INFO:tasks.workunit.client.0.smithi042.stdout:removed './117/36' 2019-12-05T05:08:11.641 INFO:tasks.workunit.client.0.smithi042.stdout:removed './117/20' 2019-12-05T05:08:11.641 INFO:tasks.workunit.client.0.smithi042.stdout:removed './117/12' 2019-12-05T05:08:11.642 INFO:tasks.workunit.client.0.smithi042.stdout:removed './117/39' 2019-12-05T05:08:11.642 INFO:tasks.workunit.client.0.smithi042.stdout:removed './117/10' 2019-12-05T05:08:11.642 INFO:tasks.workunit.client.0.smithi042.stdout:removed './117/34' 2019-12-05T05:08:11.642 INFO:tasks.workunit.client.0.smithi042.stdout:removed './117/38' 2019-12-05T05:08:11.642 INFO:tasks.workunit.client.0.smithi042.stdout:removed './117/11' 2019-12-05T05:08:11.643 INFO:tasks.workunit.client.0.smithi042.stdout:removed './117/16' 2019-12-05T05:08:11.643 INFO:tasks.workunit.client.0.smithi042.stdout:removed './117/32' 2019-12-05T05:08:11.643 INFO:tasks.workunit.client.0.smithi042.stdout:removed './117/30' 2019-12-05T05:08:11.643 INFO:tasks.workunit.client.0.smithi042.stdout:removed './117/14' 2019-12-05T05:08:11.644 INFO:tasks.workunit.client.0.smithi042.stdout:removed './117/18' 2019-12-05T05:08:11.644 INFO:tasks.workunit.client.0.smithi042.stdout:removed './117/13' 2019-12-05T05:08:11.644 INFO:tasks.workunit.client.0.smithi042.stdout:removed './117/33' 2019-12-05T05:08:11.644 INFO:tasks.workunit.client.0.smithi042.stdout:removed './117/17' 2019-12-05T05:08:11.644 INFO:tasks.workunit.client.0.smithi042.stdout:removed './117/35' 2019-12-05T05:08:11.644 INFO:tasks.workunit.client.0.smithi042.stdout:removed './117/37' 2019-12-05T05:08:11.644 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './117' 2019-12-05T05:08:11.645 INFO:tasks.workunit.client.0.smithi042.stdout:removed './118/19' 2019-12-05T05:08:11.645 INFO:tasks.workunit.client.0.smithi042.stdout:removed './118/15' 2019-12-05T05:08:11.645 INFO:tasks.workunit.client.0.smithi042.stdout:removed './118/31' 2019-12-05T05:08:11.645 INFO:tasks.workunit.client.0.smithi042.stdout:removed './118/36' 2019-12-05T05:08:11.645 INFO:tasks.workunit.client.0.smithi042.stdout:removed './118/20' 2019-12-05T05:08:11.645 INFO:tasks.workunit.client.0.smithi042.stdout:removed './118/12' 2019-12-05T05:08:11.645 INFO:tasks.workunit.client.0.smithi042.stdout:removed './118/39' 2019-12-05T05:08:11.646 INFO:tasks.workunit.client.0.smithi042.stdout:removed './118/10' 2019-12-05T05:08:11.646 INFO:tasks.workunit.client.0.smithi042.stdout:removed './118/34' 2019-12-05T05:08:11.646 INFO:tasks.workunit.client.0.smithi042.stdout:removed './118/38' 2019-12-05T05:08:11.646 INFO:tasks.workunit.client.0.smithi042.stdout:removed './118/11' 2019-12-05T05:08:11.646 INFO:tasks.workunit.client.0.smithi042.stdout:removed './118/16' 2019-12-05T05:08:11.646 INFO:tasks.workunit.client.0.smithi042.stdout:removed './118/32' 2019-12-05T05:08:11.646 INFO:tasks.workunit.client.0.smithi042.stdout:removed './118/30' 2019-12-05T05:08:11.647 INFO:tasks.workunit.client.0.smithi042.stdout:removed './118/14' 2019-12-05T05:08:11.647 INFO:tasks.workunit.client.0.smithi042.stdout:removed './118/18' 2019-12-05T05:08:11.647 INFO:tasks.workunit.client.0.smithi042.stdout:removed './118/13' 2019-12-05T05:08:11.647 INFO:tasks.workunit.client.0.smithi042.stdout:removed './118/33' 2019-12-05T05:08:11.647 INFO:tasks.workunit.client.0.smithi042.stdout:removed './118/17' 2019-12-05T05:08:11.647 INFO:tasks.workunit.client.0.smithi042.stdout:removed './118/35' 2019-12-05T05:08:11.647 INFO:tasks.workunit.client.0.smithi042.stdout:removed './118/37' 2019-12-05T05:08:11.647 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './118' 2019-12-05T05:08:11.648 INFO:tasks.workunit.client.0.smithi042.stdout:removed './119/19' 2019-12-05T05:08:11.648 INFO:tasks.workunit.client.0.smithi042.stdout:removed './119/15' 2019-12-05T05:08:11.648 INFO:tasks.workunit.client.0.smithi042.stdout:removed './119/31' 2019-12-05T05:08:11.648 INFO:tasks.workunit.client.0.smithi042.stdout:removed './119/36' 2019-12-05T05:08:11.648 INFO:tasks.workunit.client.0.smithi042.stdout:removed './119/20' 2019-12-05T05:08:11.648 INFO:tasks.workunit.client.0.smithi042.stdout:removed './119/12' 2019-12-05T05:08:14.479 INFO:tasks.workunit.client.0.smithi042.stdout:removed './119/39' 2019-12-05T05:08:14.479 INFO:tasks.workunit.client.0.smithi042.stdout:removed './119/10' 2019-12-05T05:08:14.479 INFO:tasks.workunit.client.0.smithi042.stdout:removed './119/34' 2019-12-05T05:08:14.479 INFO:tasks.workunit.client.0.smithi042.stdout:removed './119/38' 2019-12-05T05:08:14.479 INFO:tasks.workunit.client.0.smithi042.stdout:removed './119/11' 2019-12-05T05:08:14.480 INFO:tasks.workunit.client.0.smithi042.stdout:removed './119/16' 2019-12-05T05:08:14.480 INFO:tasks.workunit.client.0.smithi042.stdout:removed './119/32' 2019-12-05T05:08:14.480 INFO:tasks.workunit.client.0.smithi042.stdout:removed './119/30' 2019-12-05T05:08:14.480 INFO:tasks.workunit.client.0.smithi042.stdout:removed './119/14' 2019-12-05T05:08:14.480 INFO:tasks.workunit.client.0.smithi042.stdout:removed './119/18' 2019-12-05T05:08:14.480 INFO:tasks.workunit.client.0.smithi042.stdout:removed './119/13' 2019-12-05T05:08:14.480 INFO:tasks.workunit.client.0.smithi042.stdout:removed './119/33' 2019-12-05T05:08:14.481 INFO:tasks.workunit.client.0.smithi042.stdout:removed './119/17' 2019-12-05T05:08:14.481 INFO:tasks.workunit.client.0.smithi042.stdout:removed './119/35' 2019-12-05T05:08:14.481 INFO:tasks.workunit.client.0.smithi042.stdout:removed './119/37' 2019-12-05T05:08:14.481 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './119' 2019-12-05T05:08:14.481 INFO:tasks.workunit.client.0.smithi042.stdout:removed './120/19' 2019-12-05T05:08:14.481 INFO:tasks.workunit.client.0.smithi042.stdout:removed './120/15' 2019-12-05T05:08:14.481 INFO:tasks.workunit.client.0.smithi042.stdout:removed './120/31' 2019-12-05T05:08:14.481 INFO:tasks.workunit.client.0.smithi042.stdout:removed './120/36' 2019-12-05T05:08:14.482 INFO:tasks.workunit.client.0.smithi042.stdout:removed './120/20' 2019-12-05T05:08:14.482 INFO:tasks.workunit.client.0.smithi042.stdout:removed './120/12' 2019-12-05T05:08:14.482 INFO:tasks.workunit.client.0.smithi042.stdout:removed './120/39' 2019-12-05T05:08:14.482 INFO:tasks.workunit.client.0.smithi042.stdout:removed './120/10' 2019-12-05T05:08:14.482 INFO:tasks.workunit.client.0.smithi042.stdout:removed './120/34' 2019-12-05T05:08:14.482 INFO:tasks.workunit.client.0.smithi042.stdout:removed './120/38' 2019-12-05T05:08:14.482 INFO:tasks.workunit.client.0.smithi042.stdout:removed './120/11' 2019-12-05T05:08:14.483 INFO:tasks.workunit.client.0.smithi042.stdout:removed './120/16' 2019-12-05T05:08:14.483 INFO:tasks.workunit.client.0.smithi042.stdout:removed './120/32' 2019-12-05T05:08:14.483 INFO:tasks.workunit.client.0.smithi042.stdout:removed './120/30' 2019-12-05T05:08:14.483 INFO:tasks.workunit.client.0.smithi042.stdout:removed './120/14' 2019-12-05T05:08:14.484 INFO:tasks.workunit.client.0.smithi042.stdout:removed './120/18' 2019-12-05T05:08:14.484 INFO:tasks.workunit.client.0.smithi042.stdout:removed './120/13' 2019-12-05T05:08:14.484 INFO:tasks.workunit.client.0.smithi042.stdout:removed './120/33' 2019-12-05T05:08:14.485 INFO:tasks.workunit.client.0.smithi042.stdout:removed './120/17' 2019-12-05T05:08:14.485 INFO:tasks.workunit.client.0.smithi042.stdout:removed './120/35' 2019-12-05T05:08:14.485 INFO:tasks.workunit.client.0.smithi042.stdout:removed './120/37' 2019-12-05T05:08:14.485 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './120' 2019-12-05T05:08:14.485 INFO:tasks.workunit.client.0.smithi042.stdout:removed './121/19' 2019-12-05T05:08:14.485 INFO:tasks.workunit.client.0.smithi042.stdout:removed './121/15' 2019-12-05T05:08:14.485 INFO:tasks.workunit.client.0.smithi042.stdout:removed './121/31' 2019-12-05T05:08:14.485 INFO:tasks.workunit.client.0.smithi042.stdout:removed './121/36' 2019-12-05T05:08:14.486 INFO:tasks.workunit.client.0.smithi042.stdout:removed './121/20' 2019-12-05T05:08:14.486 INFO:tasks.workunit.client.0.smithi042.stdout:removed './121/12' 2019-12-05T05:08:14.486 INFO:tasks.workunit.client.0.smithi042.stdout:removed './121/39' 2019-12-05T05:08:14.486 INFO:tasks.workunit.client.0.smithi042.stdout:removed './121/10' 2019-12-05T05:08:14.486 INFO:tasks.workunit.client.0.smithi042.stdout:removed './121/34' 2019-12-05T05:08:14.486 INFO:tasks.workunit.client.0.smithi042.stdout:removed './121/38' 2019-12-05T05:08:14.486 INFO:tasks.workunit.client.0.smithi042.stdout:removed './121/11' 2019-12-05T05:08:14.487 INFO:tasks.workunit.client.0.smithi042.stdout:removed './121/16' 2019-12-05T05:08:14.487 INFO:tasks.workunit.client.0.smithi042.stdout:removed './121/32' 2019-12-05T05:08:14.487 INFO:tasks.workunit.client.0.smithi042.stdout:removed './121/30' 2019-12-05T05:08:14.487 INFO:tasks.workunit.client.0.smithi042.stdout:removed './121/14' 2019-12-05T05:08:14.487 INFO:tasks.workunit.client.0.smithi042.stdout:removed './121/18' 2019-12-05T05:08:14.487 INFO:tasks.workunit.client.0.smithi042.stdout:removed './121/13' 2019-12-05T05:08:14.487 INFO:tasks.workunit.client.0.smithi042.stdout:removed './121/33' 2019-12-05T05:08:14.488 INFO:tasks.workunit.client.0.smithi042.stdout:removed './121/17' 2019-12-05T05:08:14.488 INFO:tasks.workunit.client.0.smithi042.stdout:removed './121/35' 2019-12-05T05:08:14.488 INFO:tasks.workunit.client.0.smithi042.stdout:removed './121/37' 2019-12-05T05:08:14.488 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './121' 2019-12-05T05:08:14.488 INFO:tasks.workunit.client.0.smithi042.stdout:removed './122/19' 2019-12-05T05:08:14.488 INFO:tasks.workunit.client.0.smithi042.stdout:removed './122/15' 2019-12-05T05:08:14.489 INFO:tasks.workunit.client.0.smithi042.stdout:removed './122/31' 2019-12-05T05:08:14.489 INFO:tasks.workunit.client.0.smithi042.stdout:removed './122/36' 2019-12-05T05:08:14.489 INFO:tasks.workunit.client.0.smithi042.stdout:removed './122/20' 2019-12-05T05:08:14.489 INFO:tasks.workunit.client.0.smithi042.stdout:removed './122/12' 2019-12-05T05:08:14.489 INFO:tasks.workunit.client.0.smithi042.stdout:removed './122/39' 2019-12-05T05:08:14.489 INFO:tasks.workunit.client.0.smithi042.stdout:removed './122/10' 2019-12-05T05:08:14.490 INFO:tasks.workunit.client.0.smithi042.stdout:removed './122/34' 2019-12-05T05:08:14.490 INFO:tasks.workunit.client.0.smithi042.stdout:removed './122/38' 2019-12-05T05:08:14.490 INFO:tasks.workunit.client.0.smithi042.stdout:removed './122/11' 2019-12-05T05:08:14.490 INFO:tasks.workunit.client.0.smithi042.stdout:removed './122/16' 2019-12-05T05:08:14.490 INFO:tasks.workunit.client.0.smithi042.stdout:removed './122/32' 2019-12-05T05:08:14.490 INFO:tasks.workunit.client.0.smithi042.stdout:removed './122/30' 2019-12-05T05:08:14.491 INFO:tasks.workunit.client.0.smithi042.stdout:removed './122/14' 2019-12-05T05:08:14.491 INFO:tasks.workunit.client.0.smithi042.stdout:removed './122/18' 2019-12-05T05:08:14.491 INFO:tasks.workunit.client.0.smithi042.stdout:removed './122/13' 2019-12-05T05:08:14.491 INFO:tasks.workunit.client.0.smithi042.stdout:removed './122/33' 2019-12-05T05:08:14.491 INFO:tasks.workunit.client.0.smithi042.stdout:removed './122/17' 2019-12-05T05:08:14.491 INFO:tasks.workunit.client.0.smithi042.stdout:removed './122/35' 2019-12-05T05:08:14.491 INFO:tasks.workunit.client.0.smithi042.stdout:removed './122/37' 2019-12-05T05:08:14.492 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './122' 2019-12-05T05:08:14.492 INFO:tasks.workunit.client.0.smithi042.stdout:removed './123/19' 2019-12-05T05:08:14.492 INFO:tasks.workunit.client.0.smithi042.stdout:removed './123/15' 2019-12-05T05:08:14.492 INFO:tasks.workunit.client.0.smithi042.stdout:removed './123/31' 2019-12-05T05:08:14.492 INFO:tasks.workunit.client.0.smithi042.stdout:removed './123/36' 2019-12-05T05:08:14.492 INFO:tasks.workunit.client.0.smithi042.stdout:removed './123/20' 2019-12-05T05:08:14.494 INFO:tasks.workunit.client.0.smithi042.stdout:removed './123/12' 2019-12-05T05:08:14.494 INFO:tasks.workunit.client.0.smithi042.stdout:removed './123/39' 2019-12-05T05:08:14.495 INFO:tasks.workunit.client.0.smithi042.stdout:removed './123/10' 2019-12-05T05:08:14.495 INFO:tasks.workunit.client.0.smithi042.stdout:removed './123/34' 2019-12-05T05:08:14.495 INFO:tasks.workunit.client.0.smithi042.stdout:removed './123/38' 2019-12-05T05:08:14.495 INFO:tasks.workunit.client.0.smithi042.stdout:removed './123/11' 2019-12-05T05:08:14.495 INFO:tasks.workunit.client.0.smithi042.stdout:removed './123/16' 2019-12-05T05:08:14.495 INFO:tasks.workunit.client.0.smithi042.stdout:removed './123/32' 2019-12-05T05:08:14.495 INFO:tasks.workunit.client.0.smithi042.stdout:removed './123/30' 2019-12-05T05:08:14.495 INFO:tasks.workunit.client.0.smithi042.stdout:removed './123/14' 2019-12-05T05:08:14.498 INFO:tasks.workunit.client.0.smithi042.stdout:removed './123/18' 2019-12-05T05:08:14.498 INFO:tasks.workunit.client.0.smithi042.stdout:removed './123/13' 2019-12-05T05:08:14.498 INFO:tasks.workunit.client.0.smithi042.stdout:removed './123/33' 2019-12-05T05:08:14.498 INFO:tasks.workunit.client.0.smithi042.stdout:removed './123/17' 2019-12-05T05:08:14.498 INFO:tasks.workunit.client.0.smithi042.stdout:removed './123/35' 2019-12-05T05:08:14.499 INFO:tasks.workunit.client.0.smithi042.stdout:removed './123/37' 2019-12-05T05:08:14.499 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './123' 2019-12-05T05:08:14.499 INFO:tasks.workunit.client.0.smithi042.stdout:removed './124/19' 2019-12-05T05:08:14.499 INFO:tasks.workunit.client.0.smithi042.stdout:removed './124/15' 2019-12-05T05:08:14.499 INFO:tasks.workunit.client.0.smithi042.stdout:removed './124/31' 2019-12-05T05:08:14.499 INFO:tasks.workunit.client.0.smithi042.stdout:removed './124/36' 2019-12-05T05:08:14.499 INFO:tasks.workunit.client.0.smithi042.stdout:removed './124/20' 2019-12-05T05:08:14.500 INFO:tasks.workunit.client.0.smithi042.stdout:removed './124/12' 2019-12-05T05:08:14.500 INFO:tasks.workunit.client.0.smithi042.stdout:removed './124/39' 2019-12-05T05:08:14.500 INFO:tasks.workunit.client.0.smithi042.stdout:removed './124/10' 2019-12-05T05:08:14.500 INFO:tasks.workunit.client.0.smithi042.stdout:removed './124/34' 2019-12-05T05:08:14.500 INFO:tasks.workunit.client.0.smithi042.stdout:removed './124/38' 2019-12-05T05:08:14.500 INFO:tasks.workunit.client.0.smithi042.stdout:removed './124/11' 2019-12-05T05:08:14.501 INFO:tasks.workunit.client.0.smithi042.stdout:removed './124/16' 2019-12-05T05:08:14.501 INFO:tasks.workunit.client.0.smithi042.stdout:removed './124/32' 2019-12-05T05:08:14.501 INFO:tasks.workunit.client.0.smithi042.stdout:removed './124/30' 2019-12-05T05:08:14.501 INFO:tasks.workunit.client.0.smithi042.stdout:removed './124/14' 2019-12-05T05:08:14.501 INFO:tasks.workunit.client.0.smithi042.stdout:removed './124/18' 2019-12-05T05:08:14.501 INFO:tasks.workunit.client.0.smithi042.stdout:removed './124/13' 2019-12-05T05:08:14.502 INFO:tasks.workunit.client.0.smithi042.stdout:removed './124/33' 2019-12-05T05:08:14.502 INFO:tasks.workunit.client.0.smithi042.stdout:removed './124/17' 2019-12-05T05:08:14.502 INFO:tasks.workunit.client.0.smithi042.stdout:removed './124/35' 2019-12-05T05:08:14.502 INFO:tasks.workunit.client.0.smithi042.stdout:removed './124/37' 2019-12-05T05:08:14.502 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './124' 2019-12-05T05:08:14.502 INFO:tasks.workunit.client.0.smithi042.stdout:removed './125/19' 2019-12-05T05:08:14.503 INFO:tasks.workunit.client.0.smithi042.stdout:removed './125/15' 2019-12-05T05:08:14.503 INFO:tasks.workunit.client.0.smithi042.stdout:removed './125/31' 2019-12-05T05:08:14.503 INFO:tasks.workunit.client.0.smithi042.stdout:removed './125/36' 2019-12-05T05:08:14.503 INFO:tasks.workunit.client.0.smithi042.stdout:removed './125/20' 2019-12-05T05:08:14.503 INFO:tasks.workunit.client.0.smithi042.stdout:removed './125/12' 2019-12-05T05:08:14.503 INFO:tasks.workunit.client.0.smithi042.stdout:removed './125/39' 2019-12-05T05:08:14.504 INFO:tasks.workunit.client.0.smithi042.stdout:removed './125/10' 2019-12-05T05:08:14.504 INFO:tasks.workunit.client.0.smithi042.stdout:removed './125/34' 2019-12-05T05:08:14.504 INFO:tasks.workunit.client.0.smithi042.stdout:removed './125/38' 2019-12-05T05:08:14.504 INFO:tasks.workunit.client.0.smithi042.stdout:removed './125/11' 2019-12-05T05:08:14.505 INFO:tasks.workunit.client.0.smithi042.stdout:removed './125/16' 2019-12-05T05:08:14.505 INFO:tasks.workunit.client.0.smithi042.stdout:removed './125/32' 2019-12-05T05:08:14.505 INFO:tasks.workunit.client.0.smithi042.stdout:removed './125/30' 2019-12-05T05:08:14.505 INFO:tasks.workunit.client.0.smithi042.stdout:removed './125/14' 2019-12-05T05:08:14.505 INFO:tasks.workunit.client.0.smithi042.stdout:removed './125/18' 2019-12-05T05:08:14.505 INFO:tasks.workunit.client.0.smithi042.stdout:removed './125/13' 2019-12-05T05:08:14.505 INFO:tasks.workunit.client.0.smithi042.stdout:removed './125/33' 2019-12-05T05:08:14.506 INFO:tasks.workunit.client.0.smithi042.stdout:removed './125/17' 2019-12-05T05:08:14.506 INFO:tasks.workunit.client.0.smithi042.stdout:removed './125/35' 2019-12-05T05:08:14.506 INFO:tasks.workunit.client.0.smithi042.stdout:removed './125/37' 2019-12-05T05:08:14.506 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './125' 2019-12-05T05:08:14.506 INFO:tasks.workunit.client.0.smithi042.stdout:removed './126/19' 2019-12-05T05:08:14.506 INFO:tasks.workunit.client.0.smithi042.stdout:removed './126/15' 2019-12-05T05:08:14.506 INFO:tasks.workunit.client.0.smithi042.stdout:removed './126/31' 2019-12-05T05:08:14.507 INFO:tasks.workunit.client.0.smithi042.stdout:removed './126/36' 2019-12-05T05:08:14.507 INFO:tasks.workunit.client.0.smithi042.stdout:removed './126/20' 2019-12-05T05:08:14.507 INFO:tasks.workunit.client.0.smithi042.stdout:removed './126/12' 2019-12-05T05:08:14.507 INFO:tasks.workunit.client.0.smithi042.stdout:removed './126/39' 2019-12-05T05:08:14.507 INFO:tasks.workunit.client.0.smithi042.stdout:removed './126/10' 2019-12-05T05:08:14.507 INFO:tasks.workunit.client.0.smithi042.stdout:removed './126/34' 2019-12-05T05:08:14.507 INFO:tasks.workunit.client.0.smithi042.stdout:removed './126/38' 2019-12-05T05:08:14.508 INFO:tasks.workunit.client.0.smithi042.stdout:removed './126/11' 2019-12-05T05:08:14.508 INFO:tasks.workunit.client.0.smithi042.stdout:removed './126/16' 2019-12-05T05:08:14.508 INFO:tasks.workunit.client.0.smithi042.stdout:removed './126/32' 2019-12-05T05:08:14.508 INFO:tasks.workunit.client.0.smithi042.stdout:removed './126/30' 2019-12-05T05:08:14.508 INFO:tasks.workunit.client.0.smithi042.stdout:removed './126/14' 2019-12-05T05:08:14.508 INFO:tasks.workunit.client.0.smithi042.stdout:removed './126/18' 2019-12-05T05:08:14.509 INFO:tasks.workunit.client.0.smithi042.stdout:removed './126/13' 2019-12-05T05:08:14.509 INFO:tasks.workunit.client.0.smithi042.stdout:removed './126/33' 2019-12-05T05:08:14.509 INFO:tasks.workunit.client.0.smithi042.stdout:removed './126/17' 2019-12-05T05:08:14.509 INFO:tasks.workunit.client.0.smithi042.stdout:removed './126/35' 2019-12-05T05:08:14.509 INFO:tasks.workunit.client.0.smithi042.stdout:removed './126/37' 2019-12-05T05:08:14.509 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './126' 2019-12-05T05:08:14.510 INFO:tasks.workunit.client.0.smithi042.stdout:removed './127/19' 2019-12-05T05:08:14.510 INFO:tasks.workunit.client.0.smithi042.stdout:removed './127/15' 2019-12-05T05:08:14.510 INFO:tasks.workunit.client.0.smithi042.stdout:removed './127/31' 2019-12-05T05:08:14.510 INFO:tasks.workunit.client.0.smithi042.stdout:removed './127/36' 2019-12-05T05:08:14.510 INFO:tasks.workunit.client.0.smithi042.stdout:removed './127/20' 2019-12-05T05:08:14.510 INFO:tasks.workunit.client.0.smithi042.stdout:removed './127/12' 2019-12-05T05:08:14.510 INFO:tasks.workunit.client.0.smithi042.stdout:removed './127/39' 2019-12-05T05:08:14.511 INFO:tasks.workunit.client.0.smithi042.stdout:removed './127/10' 2019-12-05T05:08:14.511 INFO:tasks.workunit.client.0.smithi042.stdout:removed './127/34' 2019-12-05T05:08:14.511 INFO:tasks.workunit.client.0.smithi042.stdout:removed './127/38' 2019-12-05T05:08:14.511 INFO:tasks.workunit.client.0.smithi042.stdout:removed './127/11' 2019-12-05T05:08:14.512 INFO:tasks.workunit.client.0.smithi042.stdout:removed './127/16' 2019-12-05T05:08:14.512 INFO:tasks.workunit.client.0.smithi042.stdout:removed './127/32' 2019-12-05T05:08:14.512 INFO:tasks.workunit.client.0.smithi042.stdout:removed './127/30' 2019-12-05T05:08:14.512 INFO:tasks.workunit.client.0.smithi042.stdout:removed './127/14' 2019-12-05T05:08:14.512 INFO:tasks.workunit.client.0.smithi042.stdout:removed './127/18' 2019-12-05T05:08:14.512 INFO:tasks.workunit.client.0.smithi042.stdout:removed './127/13' 2019-12-05T05:08:14.512 INFO:tasks.workunit.client.0.smithi042.stdout:removed './127/33' 2019-12-05T05:08:14.513 INFO:tasks.workunit.client.0.smithi042.stdout:removed './127/17' 2019-12-05T05:08:14.513 INFO:tasks.workunit.client.0.smithi042.stdout:removed './127/35' 2019-12-05T05:08:14.513 INFO:tasks.workunit.client.0.smithi042.stdout:removed './127/37' 2019-12-05T05:08:14.513 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './127' 2019-12-05T05:08:14.513 INFO:tasks.workunit.client.0.smithi042.stdout:removed './128/19' 2019-12-05T05:08:14.513 INFO:tasks.workunit.client.0.smithi042.stdout:removed './128/15' 2019-12-05T05:08:14.513 INFO:tasks.workunit.client.0.smithi042.stdout:removed './128/31' 2019-12-05T05:08:14.513 INFO:tasks.workunit.client.0.smithi042.stdout:removed './128/36' 2019-12-05T05:08:14.514 INFO:tasks.workunit.client.0.smithi042.stdout:removed './128/20' 2019-12-05T05:08:14.514 INFO:tasks.workunit.client.0.smithi042.stdout:removed './128/12' 2019-12-05T05:08:14.514 INFO:tasks.workunit.client.0.smithi042.stdout:removed './128/39' 2019-12-05T05:08:14.514 INFO:tasks.workunit.client.0.smithi042.stdout:removed './128/10' 2019-12-05T05:08:14.514 INFO:tasks.workunit.client.0.smithi042.stdout:removed './128/34' 2019-12-05T05:08:14.514 INFO:tasks.workunit.client.0.smithi042.stdout:removed './128/38' 2019-12-05T05:08:14.514 INFO:tasks.workunit.client.0.smithi042.stdout:removed './128/11' 2019-12-05T05:08:14.516 INFO:tasks.workunit.client.0.smithi042.stdout:removed './128/16' 2019-12-05T05:08:14.516 INFO:tasks.workunit.client.0.smithi042.stdout:removed './128/32' 2019-12-05T05:08:14.516 INFO:tasks.workunit.client.0.smithi042.stdout:removed './128/30' 2019-12-05T05:08:14.516 INFO:tasks.workunit.client.0.smithi042.stdout:removed './128/14' 2019-12-05T05:08:14.516 INFO:tasks.workunit.client.0.smithi042.stdout:removed './128/18' 2019-12-05T05:08:14.517 INFO:tasks.workunit.client.0.smithi042.stdout:removed './128/13' 2019-12-05T05:08:14.517 INFO:tasks.workunit.client.0.smithi042.stdout:removed './128/33' 2019-12-05T05:08:14.517 INFO:tasks.workunit.client.0.smithi042.stdout:removed './128/17' 2019-12-05T05:08:14.518 INFO:tasks.workunit.client.0.smithi042.stdout:removed './128/35' 2019-12-05T05:08:19.769 INFO:tasks.workunit.client.0.smithi042.stdout:removed './128/37' 2019-12-05T05:08:19.769 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './128' 2019-12-05T05:08:19.769 INFO:tasks.workunit.client.0.smithi042.stdout:removed './129/19' 2019-12-05T05:08:19.770 INFO:tasks.workunit.client.0.smithi042.stdout:removed './129/15' 2019-12-05T05:08:19.770 INFO:tasks.workunit.client.0.smithi042.stdout:removed './129/31' 2019-12-05T05:08:19.770 INFO:tasks.workunit.client.0.smithi042.stdout:removed './129/36' 2019-12-05T05:08:19.770 INFO:tasks.workunit.client.0.smithi042.stdout:removed './129/20' 2019-12-05T05:08:19.770 INFO:tasks.workunit.client.0.smithi042.stdout:removed './129/12' 2019-12-05T05:08:19.770 INFO:tasks.workunit.client.0.smithi042.stdout:removed './129/39' 2019-12-05T05:08:19.770 INFO:tasks.workunit.client.0.smithi042.stdout:removed './129/10' 2019-12-05T05:08:19.771 INFO:tasks.workunit.client.0.smithi042.stdout:removed './129/34' 2019-12-05T05:08:19.771 INFO:tasks.workunit.client.0.smithi042.stdout:removed './129/38' 2019-12-05T05:08:19.771 INFO:tasks.workunit.client.0.smithi042.stdout:removed './129/11' 2019-12-05T05:08:19.771 INFO:tasks.workunit.client.0.smithi042.stdout:removed './129/16' 2019-12-05T05:08:19.771 INFO:tasks.workunit.client.0.smithi042.stdout:removed './129/32' 2019-12-05T05:08:19.771 INFO:tasks.workunit.client.0.smithi042.stdout:removed './129/30' 2019-12-05T05:08:19.771 INFO:tasks.workunit.client.0.smithi042.stdout:removed './129/14' 2019-12-05T05:08:19.772 INFO:tasks.workunit.client.0.smithi042.stdout:removed './129/18' 2019-12-05T05:08:19.772 INFO:tasks.workunit.client.0.smithi042.stdout:removed './129/13' 2019-12-05T05:08:19.772 INFO:tasks.workunit.client.0.smithi042.stdout:removed './129/33' 2019-12-05T05:08:19.772 INFO:tasks.workunit.client.0.smithi042.stdout:removed './129/17' 2019-12-05T05:08:19.772 INFO:tasks.workunit.client.0.smithi042.stdout:removed './129/35' 2019-12-05T05:08:19.772 INFO:tasks.workunit.client.0.smithi042.stdout:removed './129/37' 2019-12-05T05:08:19.773 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './129' 2019-12-05T05:08:19.773 INFO:tasks.workunit.client.0.smithi042.stdout:removed './130/19' 2019-12-05T05:08:19.773 INFO:tasks.workunit.client.0.smithi042.stdout:removed './130/15' 2019-12-05T05:08:19.773 INFO:tasks.workunit.client.0.smithi042.stdout:removed './130/31' 2019-12-05T05:08:19.773 INFO:tasks.workunit.client.0.smithi042.stdout:removed './130/36' 2019-12-05T05:08:19.773 INFO:tasks.workunit.client.0.smithi042.stdout:removed './130/20' 2019-12-05T05:08:19.773 INFO:tasks.workunit.client.0.smithi042.stdout:removed './130/12' 2019-12-05T05:08:19.773 INFO:tasks.workunit.client.0.smithi042.stdout:removed './130/39' 2019-12-05T05:08:19.774 INFO:tasks.workunit.client.0.smithi042.stdout:removed './130/10' 2019-12-05T05:08:19.774 INFO:tasks.workunit.client.0.smithi042.stdout:removed './130/34' 2019-12-05T05:08:19.774 INFO:tasks.workunit.client.0.smithi042.stdout:removed './130/38' 2019-12-05T05:08:19.774 INFO:tasks.workunit.client.0.smithi042.stdout:removed './130/11' 2019-12-05T05:08:19.774 INFO:tasks.workunit.client.0.smithi042.stdout:removed './130/16' 2019-12-05T05:08:19.774 INFO:tasks.workunit.client.0.smithi042.stdout:removed './130/32' 2019-12-05T05:08:19.774 INFO:tasks.workunit.client.0.smithi042.stdout:removed './130/30' 2019-12-05T05:08:19.775 INFO:tasks.workunit.client.0.smithi042.stdout:removed './130/14' 2019-12-05T05:08:19.775 INFO:tasks.workunit.client.0.smithi042.stdout:removed './130/18' 2019-12-05T05:08:19.775 INFO:tasks.workunit.client.0.smithi042.stdout:removed './130/13' 2019-12-05T05:08:19.775 INFO:tasks.workunit.client.0.smithi042.stdout:removed './130/33' 2019-12-05T05:08:19.775 INFO:tasks.workunit.client.0.smithi042.stdout:removed './130/17' 2019-12-05T05:08:19.775 INFO:tasks.workunit.client.0.smithi042.stdout:removed './130/35' 2019-12-05T05:08:19.775 INFO:tasks.workunit.client.0.smithi042.stdout:removed './130/37' 2019-12-05T05:08:19.776 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './130' 2019-12-05T05:08:19.776 INFO:tasks.workunit.client.0.smithi042.stdout:removed './131/19' 2019-12-05T05:08:19.776 INFO:tasks.workunit.client.0.smithi042.stdout:removed './131/15' 2019-12-05T05:08:19.776 INFO:tasks.workunit.client.0.smithi042.stdout:removed './131/31' 2019-12-05T05:08:19.780 INFO:tasks.workunit.client.0.smithi042.stdout:removed './131/36' 2019-12-05T05:08:19.780 INFO:tasks.workunit.client.0.smithi042.stdout:removed './131/20' 2019-12-05T05:08:19.780 INFO:tasks.workunit.client.0.smithi042.stdout:removed './131/12' 2019-12-05T05:08:19.780 INFO:tasks.workunit.client.0.smithi042.stdout:removed './131/39' 2019-12-05T05:08:19.780 INFO:tasks.workunit.client.0.smithi042.stdout:removed './131/10' 2019-12-05T05:08:19.781 INFO:tasks.workunit.client.0.smithi042.stdout:removed './131/34' 2019-12-05T05:08:19.781 INFO:tasks.workunit.client.0.smithi042.stdout:removed './131/38' 2019-12-05T05:08:19.781 INFO:tasks.workunit.client.0.smithi042.stdout:removed './131/11' 2019-12-05T05:08:19.781 INFO:tasks.workunit.client.0.smithi042.stdout:removed './131/16' 2019-12-05T05:08:19.781 INFO:tasks.workunit.client.0.smithi042.stdout:removed './131/32' 2019-12-05T05:08:19.781 INFO:tasks.workunit.client.0.smithi042.stdout:removed './131/30' 2019-12-05T05:08:19.781 INFO:tasks.workunit.client.0.smithi042.stdout:removed './131/14' 2019-12-05T05:08:19.782 INFO:tasks.workunit.client.0.smithi042.stdout:removed './131/18' 2019-12-05T05:08:19.782 INFO:tasks.workunit.client.0.smithi042.stdout:removed './131/13' 2019-12-05T05:08:19.782 INFO:tasks.workunit.client.0.smithi042.stdout:removed './131/33' 2019-12-05T05:08:19.782 INFO:tasks.workunit.client.0.smithi042.stdout:removed './131/17' 2019-12-05T05:08:19.782 INFO:tasks.workunit.client.0.smithi042.stdout:removed './131/35' 2019-12-05T05:08:19.782 INFO:tasks.workunit.client.0.smithi042.stdout:removed './131/37' 2019-12-05T05:08:19.782 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './131' 2019-12-05T05:08:19.782 INFO:tasks.workunit.client.0.smithi042.stdout:removed './132/19' 2019-12-05T05:08:19.783 INFO:tasks.workunit.client.0.smithi042.stdout:removed './132/15' 2019-12-05T05:08:19.783 INFO:tasks.workunit.client.0.smithi042.stdout:removed './132/31' 2019-12-05T05:08:19.783 INFO:tasks.workunit.client.0.smithi042.stdout:removed './132/36' 2019-12-05T05:08:19.783 INFO:tasks.workunit.client.0.smithi042.stdout:removed './132/20' 2019-12-05T05:08:19.783 INFO:tasks.workunit.client.0.smithi042.stdout:removed './132/12' 2019-12-05T05:08:19.783 INFO:tasks.workunit.client.0.smithi042.stdout:removed './132/39' 2019-12-05T05:08:19.783 INFO:tasks.workunit.client.0.smithi042.stdout:removed './132/10' 2019-12-05T05:08:19.784 INFO:tasks.workunit.client.0.smithi042.stdout:removed './132/34' 2019-12-05T05:08:19.784 INFO:tasks.workunit.client.0.smithi042.stdout:removed './132/38' 2019-12-05T05:08:19.784 INFO:tasks.workunit.client.0.smithi042.stdout:removed './132/11' 2019-12-05T05:08:19.784 INFO:tasks.workunit.client.0.smithi042.stdout:removed './132/16' 2019-12-05T05:08:19.784 INFO:tasks.workunit.client.0.smithi042.stdout:removed './132/32' 2019-12-05T05:08:19.784 INFO:tasks.workunit.client.0.smithi042.stdout:removed './132/30' 2019-12-05T05:08:19.784 INFO:tasks.workunit.client.0.smithi042.stdout:removed './132/14' 2019-12-05T05:08:19.785 INFO:tasks.workunit.client.0.smithi042.stdout:removed './132/18' 2019-12-05T05:08:19.785 INFO:tasks.workunit.client.0.smithi042.stdout:removed './132/13' 2019-12-05T05:08:19.785 INFO:tasks.workunit.client.0.smithi042.stdout:removed './132/33' 2019-12-05T05:08:19.785 INFO:tasks.workunit.client.0.smithi042.stdout:removed './132/17' 2019-12-05T05:08:19.785 INFO:tasks.workunit.client.0.smithi042.stdout:removed './132/35' 2019-12-05T05:08:19.785 INFO:tasks.workunit.client.0.smithi042.stdout:removed './132/37' 2019-12-05T05:08:19.785 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './132' 2019-12-05T05:08:19.786 INFO:tasks.workunit.client.0.smithi042.stdout:removed './133/19' 2019-12-05T05:08:19.786 INFO:tasks.workunit.client.0.smithi042.stdout:removed './133/15' 2019-12-05T05:08:19.786 INFO:tasks.workunit.client.0.smithi042.stdout:removed './133/31' 2019-12-05T05:08:19.786 INFO:tasks.workunit.client.0.smithi042.stdout:removed './133/36' 2019-12-05T05:08:19.786 INFO:tasks.workunit.client.0.smithi042.stdout:removed './133/20' 2019-12-05T05:08:19.804 INFO:tasks.workunit.client.0.smithi042.stdout:removed './133/12' 2019-12-05T05:08:19.804 INFO:tasks.workunit.client.0.smithi042.stdout:removed './133/39' 2019-12-05T05:08:19.804 INFO:tasks.workunit.client.0.smithi042.stdout:removed './133/10' 2019-12-05T05:08:19.804 INFO:tasks.workunit.client.0.smithi042.stdout:removed './133/34' 2019-12-05T05:08:19.805 INFO:tasks.workunit.client.0.smithi042.stdout:removed './133/38' 2019-12-05T05:08:19.805 INFO:tasks.workunit.client.0.smithi042.stdout:removed './133/11' 2019-12-05T05:08:19.805 INFO:tasks.workunit.client.0.smithi042.stdout:removed './133/16' 2019-12-05T05:08:19.805 INFO:tasks.workunit.client.0.smithi042.stdout:removed './133/32' 2019-12-05T05:08:19.805 INFO:tasks.workunit.client.0.smithi042.stdout:removed './133/30' 2019-12-05T05:08:19.805 INFO:tasks.workunit.client.0.smithi042.stdout:removed './133/14' 2019-12-05T05:08:19.806 INFO:tasks.workunit.client.0.smithi042.stdout:removed './133/18' 2019-12-05T05:08:19.806 INFO:tasks.workunit.client.0.smithi042.stdout:removed './133/13' 2019-12-05T05:08:19.806 INFO:tasks.workunit.client.0.smithi042.stdout:removed './133/33' 2019-12-05T05:08:19.806 INFO:tasks.workunit.client.0.smithi042.stdout:removed './133/17' 2019-12-05T05:08:19.806 INFO:tasks.workunit.client.0.smithi042.stdout:removed './133/35' 2019-12-05T05:08:19.807 INFO:tasks.workunit.client.0.smithi042.stdout:removed './133/37' 2019-12-05T05:08:19.807 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './133' 2019-12-05T05:08:19.807 INFO:tasks.workunit.client.0.smithi042.stdout:removed './134/19' 2019-12-05T05:08:19.807 INFO:tasks.workunit.client.0.smithi042.stdout:removed './134/15' 2019-12-05T05:08:19.813 INFO:tasks.workunit.client.0.smithi042.stdout:removed './134/31' 2019-12-05T05:08:19.814 INFO:tasks.workunit.client.0.smithi042.stdout:removed './134/36' 2019-12-05T05:08:19.814 INFO:tasks.workunit.client.0.smithi042.stdout:removed './134/20' 2019-12-05T05:08:19.815 INFO:tasks.workunit.client.0.smithi042.stdout:removed './134/12' 2019-12-05T05:08:19.815 INFO:tasks.workunit.client.0.smithi042.stdout:removed './134/39' 2019-12-05T05:08:19.815 INFO:tasks.workunit.client.0.smithi042.stdout:removed './134/10' 2019-12-05T05:08:19.815 INFO:tasks.workunit.client.0.smithi042.stdout:removed './134/34' 2019-12-05T05:08:19.815 INFO:tasks.workunit.client.0.smithi042.stdout:removed './134/38' 2019-12-05T05:08:19.815 INFO:tasks.workunit.client.0.smithi042.stdout:removed './134/11' 2019-12-05T05:08:19.816 INFO:tasks.workunit.client.0.smithi042.stdout:removed './134/16' 2019-12-05T05:08:19.816 INFO:tasks.workunit.client.0.smithi042.stdout:removed './134/32' 2019-12-05T05:08:19.816 INFO:tasks.workunit.client.0.smithi042.stdout:removed './134/30' 2019-12-05T05:08:19.816 INFO:tasks.workunit.client.0.smithi042.stdout:removed './134/14' 2019-12-05T05:08:19.816 INFO:tasks.workunit.client.0.smithi042.stdout:removed './134/18' 2019-12-05T05:08:19.816 INFO:tasks.workunit.client.0.smithi042.stdout:removed './134/13' 2019-12-05T05:08:19.816 INFO:tasks.workunit.client.0.smithi042.stdout:removed './134/33' 2019-12-05T05:08:19.817 INFO:tasks.workunit.client.0.smithi042.stdout:removed './134/17' 2019-12-05T05:08:19.817 INFO:tasks.workunit.client.0.smithi042.stdout:removed './134/35' 2019-12-05T05:08:19.817 INFO:tasks.workunit.client.0.smithi042.stdout:removed './134/37' 2019-12-05T05:08:19.817 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './134' 2019-12-05T05:08:19.817 INFO:tasks.workunit.client.0.smithi042.stdout:removed './135/19' 2019-12-05T05:08:19.817 INFO:tasks.workunit.client.0.smithi042.stdout:removed './135/15' 2019-12-05T05:08:19.817 INFO:tasks.workunit.client.0.smithi042.stdout:removed './135/31' 2019-12-05T05:08:19.817 INFO:tasks.workunit.client.0.smithi042.stdout:removed './135/36' 2019-12-05T05:08:19.818 INFO:tasks.workunit.client.0.smithi042.stdout:removed './135/20' 2019-12-05T05:08:19.818 INFO:tasks.workunit.client.0.smithi042.stdout:removed './135/12' 2019-12-05T05:08:19.818 INFO:tasks.workunit.client.0.smithi042.stdout:removed './135/39' 2019-12-05T05:08:19.818 INFO:tasks.workunit.client.0.smithi042.stdout:removed './135/10' 2019-12-05T05:08:19.818 INFO:tasks.workunit.client.0.smithi042.stdout:removed './135/34' 2019-12-05T05:08:19.818 INFO:tasks.workunit.client.0.smithi042.stdout:removed './135/38' 2019-12-05T05:08:19.818 INFO:tasks.workunit.client.0.smithi042.stdout:removed './135/11' 2019-12-05T05:08:19.819 INFO:tasks.workunit.client.0.smithi042.stdout:removed './135/16' 2019-12-05T05:08:19.819 INFO:tasks.workunit.client.0.smithi042.stdout:removed './135/32' 2019-12-05T05:08:19.819 INFO:tasks.workunit.client.0.smithi042.stdout:removed './135/30' 2019-12-05T05:08:19.819 INFO:tasks.workunit.client.0.smithi042.stdout:removed './135/14' 2019-12-05T05:08:19.819 INFO:tasks.workunit.client.0.smithi042.stdout:removed './135/18' 2019-12-05T05:08:19.819 INFO:tasks.workunit.client.0.smithi042.stdout:removed './135/13' 2019-12-05T05:08:19.819 INFO:tasks.workunit.client.0.smithi042.stdout:removed './135/33' 2019-12-05T05:08:19.820 INFO:tasks.workunit.client.0.smithi042.stdout:removed './135/17' 2019-12-05T05:08:19.820 INFO:tasks.workunit.client.0.smithi042.stdout:removed './135/35' 2019-12-05T05:08:19.820 INFO:tasks.workunit.client.0.smithi042.stdout:removed './135/37' 2019-12-05T05:08:19.820 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './135' 2019-12-05T05:08:19.820 INFO:tasks.workunit.client.0.smithi042.stdout:removed './136/19' 2019-12-05T05:08:19.820 INFO:tasks.workunit.client.0.smithi042.stdout:removed './136/15' 2019-12-05T05:08:19.820 INFO:tasks.workunit.client.0.smithi042.stdout:removed './136/31' 2019-12-05T05:08:19.821 INFO:tasks.workunit.client.0.smithi042.stdout:removed './136/36' 2019-12-05T05:08:19.821 INFO:tasks.workunit.client.0.smithi042.stdout:removed './136/20' 2019-12-05T05:08:19.821 INFO:tasks.workunit.client.0.smithi042.stdout:removed './136/12' 2019-12-05T05:08:19.821 INFO:tasks.workunit.client.0.smithi042.stdout:removed './136/39' 2019-12-05T05:08:19.821 INFO:tasks.workunit.client.0.smithi042.stdout:removed './136/10' 2019-12-05T05:08:19.821 INFO:tasks.workunit.client.0.smithi042.stdout:removed './136/34' 2019-12-05T05:08:19.821 INFO:tasks.workunit.client.0.smithi042.stdout:removed './136/38' 2019-12-05T05:08:19.822 INFO:tasks.workunit.client.0.smithi042.stdout:removed './136/11' 2019-12-05T05:08:19.822 INFO:tasks.workunit.client.0.smithi042.stdout:removed './136/16' 2019-12-05T05:08:19.822 INFO:tasks.workunit.client.0.smithi042.stdout:removed './136/32' 2019-12-05T05:08:19.822 INFO:tasks.workunit.client.0.smithi042.stdout:removed './136/30' 2019-12-05T05:08:19.822 INFO:tasks.workunit.client.0.smithi042.stdout:removed './136/14' 2019-12-05T05:08:19.822 INFO:tasks.workunit.client.0.smithi042.stdout:removed './136/18' 2019-12-05T05:08:19.822 INFO:tasks.workunit.client.0.smithi042.stdout:removed './136/13' 2019-12-05T05:08:19.823 INFO:tasks.workunit.client.0.smithi042.stdout:removed './136/33' 2019-12-05T05:08:19.823 INFO:tasks.workunit.client.0.smithi042.stdout:removed './136/17' 2019-12-05T05:08:19.823 INFO:tasks.workunit.client.0.smithi042.stdout:removed './136/35' 2019-12-05T05:08:19.823 INFO:tasks.workunit.client.0.smithi042.stdout:removed './136/37' 2019-12-05T05:08:19.823 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './136' 2019-12-05T05:08:19.823 INFO:tasks.workunit.client.0.smithi042.stdout:removed './137/19' 2019-12-05T05:08:19.823 INFO:tasks.workunit.client.0.smithi042.stdout:removed './137/15' 2019-12-05T05:08:19.823 INFO:tasks.workunit.client.0.smithi042.stdout:removed './137/31' 2019-12-05T05:08:19.824 INFO:tasks.workunit.client.0.smithi042.stdout:removed './137/36' 2019-12-05T05:08:19.824 INFO:tasks.workunit.client.0.smithi042.stdout:removed './137/20' 2019-12-05T05:08:19.824 INFO:tasks.workunit.client.0.smithi042.stdout:removed './137/12' 2019-12-05T05:08:19.824 INFO:tasks.workunit.client.0.smithi042.stdout:removed './137/39' 2019-12-05T05:08:19.824 INFO:tasks.workunit.client.0.smithi042.stdout:removed './137/10' 2019-12-05T05:08:19.824 INFO:tasks.workunit.client.0.smithi042.stdout:removed './137/34' 2019-12-05T05:08:19.824 INFO:tasks.workunit.client.0.smithi042.stdout:removed './137/38' 2019-12-05T05:08:19.825 INFO:tasks.workunit.client.0.smithi042.stdout:removed './137/11' 2019-12-05T05:08:19.825 INFO:tasks.workunit.client.0.smithi042.stdout:removed './137/16' 2019-12-05T05:08:19.825 INFO:tasks.workunit.client.0.smithi042.stdout:removed './137/32' 2019-12-05T05:08:19.825 INFO:tasks.workunit.client.0.smithi042.stdout:removed './137/30' 2019-12-05T05:08:19.825 INFO:tasks.workunit.client.0.smithi042.stdout:removed './137/14' 2019-12-05T05:08:19.825 INFO:tasks.workunit.client.0.smithi042.stdout:removed './137/18' 2019-12-05T05:08:19.825 INFO:tasks.workunit.client.0.smithi042.stdout:removed './137/13' 2019-12-05T05:08:19.826 INFO:tasks.workunit.client.0.smithi042.stdout:removed './137/33' 2019-12-05T05:08:19.826 INFO:tasks.workunit.client.0.smithi042.stdout:removed './137/17' 2019-12-05T05:08:19.826 INFO:tasks.workunit.client.0.smithi042.stdout:removed './137/35' 2019-12-05T05:08:19.826 INFO:tasks.workunit.client.0.smithi042.stdout:removed './137/37' 2019-12-05T05:08:19.826 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './137' 2019-12-05T05:08:19.826 INFO:tasks.workunit.client.0.smithi042.stdout:removed './138/19' 2019-12-05T05:08:19.826 INFO:tasks.workunit.client.0.smithi042.stdout:removed './138/15' 2019-12-05T05:08:19.826 INFO:tasks.workunit.client.0.smithi042.stdout:removed './138/31' 2019-12-05T05:08:19.827 INFO:tasks.workunit.client.0.smithi042.stdout:removed './138/36' 2019-12-05T05:08:19.827 INFO:tasks.workunit.client.0.smithi042.stdout:removed './138/20' 2019-12-05T05:08:19.827 INFO:tasks.workunit.client.0.smithi042.stdout:removed './138/12' 2019-12-05T05:08:19.827 INFO:tasks.workunit.client.0.smithi042.stdout:removed './138/39' 2019-12-05T05:08:19.827 INFO:tasks.workunit.client.0.smithi042.stdout:removed './138/10' 2019-12-05T05:08:19.827 INFO:tasks.workunit.client.0.smithi042.stdout:removed './138/34' 2019-12-05T05:08:19.827 INFO:tasks.workunit.client.0.smithi042.stdout:removed './138/38' 2019-12-05T05:08:19.828 INFO:tasks.workunit.client.0.smithi042.stdout:removed './138/11' 2019-12-05T05:08:19.828 INFO:tasks.workunit.client.0.smithi042.stdout:removed './138/16' 2019-12-05T05:08:25.090 INFO:tasks.workunit.client.0.smithi042.stdout:removed './138/32' 2019-12-05T05:08:25.091 INFO:tasks.workunit.client.0.smithi042.stdout:removed './138/30' 2019-12-05T05:08:25.091 INFO:tasks.workunit.client.0.smithi042.stdout:removed './138/14' 2019-12-05T05:08:25.091 INFO:tasks.workunit.client.0.smithi042.stdout:removed './138/18' 2019-12-05T05:08:25.091 INFO:tasks.workunit.client.0.smithi042.stdout:removed './138/13' 2019-12-05T05:08:25.091 INFO:tasks.workunit.client.0.smithi042.stdout:removed './138/33' 2019-12-05T05:08:25.091 INFO:tasks.workunit.client.0.smithi042.stdout:removed './138/17' 2019-12-05T05:08:25.091 INFO:tasks.workunit.client.0.smithi042.stdout:removed './138/35' 2019-12-05T05:08:25.092 INFO:tasks.workunit.client.0.smithi042.stdout:removed './138/37' 2019-12-05T05:08:25.092 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './138' 2019-12-05T05:08:25.092 INFO:tasks.workunit.client.0.smithi042.stdout:removed './139/19' 2019-12-05T05:08:25.092 INFO:tasks.workunit.client.0.smithi042.stdout:removed './139/15' 2019-12-05T05:08:25.092 INFO:tasks.workunit.client.0.smithi042.stdout:removed './139/31' 2019-12-05T05:08:25.093 INFO:tasks.workunit.client.0.smithi042.stdout:removed './139/36' 2019-12-05T05:08:25.093 INFO:tasks.workunit.client.0.smithi042.stdout:removed './139/20' 2019-12-05T05:08:25.093 INFO:tasks.workunit.client.0.smithi042.stdout:removed './139/12' 2019-12-05T05:08:25.093 INFO:tasks.workunit.client.0.smithi042.stdout:removed './139/39' 2019-12-05T05:08:25.093 INFO:tasks.workunit.client.0.smithi042.stdout:removed './139/10' 2019-12-05T05:08:25.093 INFO:tasks.workunit.client.0.smithi042.stdout:removed './139/34' 2019-12-05T05:08:25.093 INFO:tasks.workunit.client.0.smithi042.stdout:removed './139/38' 2019-12-05T05:08:25.094 INFO:tasks.workunit.client.0.smithi042.stdout:removed './139/11' 2019-12-05T05:08:25.094 INFO:tasks.workunit.client.0.smithi042.stdout:removed './139/16' 2019-12-05T05:08:25.094 INFO:tasks.workunit.client.0.smithi042.stdout:removed './139/32' 2019-12-05T05:08:25.094 INFO:tasks.workunit.client.0.smithi042.stdout:removed './139/30' 2019-12-05T05:08:25.094 INFO:tasks.workunit.client.0.smithi042.stdout:removed './139/14' 2019-12-05T05:08:25.094 INFO:tasks.workunit.client.0.smithi042.stdout:removed './139/18' 2019-12-05T05:08:25.094 INFO:tasks.workunit.client.0.smithi042.stdout:removed './139/13' 2019-12-05T05:08:25.095 INFO:tasks.workunit.client.0.smithi042.stdout:removed './139/33' 2019-12-05T05:08:25.095 INFO:tasks.workunit.client.0.smithi042.stdout:removed './139/17' 2019-12-05T05:08:25.095 INFO:tasks.workunit.client.0.smithi042.stdout:removed './139/35' 2019-12-05T05:08:25.095 INFO:tasks.workunit.client.0.smithi042.stdout:removed './139/37' 2019-12-05T05:08:25.095 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './139' 2019-12-05T05:08:25.095 INFO:tasks.workunit.client.0.smithi042.stdout:removed './140/19' 2019-12-05T05:08:25.095 INFO:tasks.workunit.client.0.smithi042.stdout:removed './140/15' 2019-12-05T05:08:25.095 INFO:tasks.workunit.client.0.smithi042.stdout:removed './140/31' 2019-12-05T05:08:25.096 INFO:tasks.workunit.client.0.smithi042.stdout:removed './140/36' 2019-12-05T05:08:25.096 INFO:tasks.workunit.client.0.smithi042.stdout:removed './140/20' 2019-12-05T05:08:25.097 INFO:tasks.workunit.client.0.smithi042.stdout:removed './140/12' 2019-12-05T05:08:25.097 INFO:tasks.workunit.client.0.smithi042.stdout:removed './140/39' 2019-12-05T05:08:25.097 INFO:tasks.workunit.client.0.smithi042.stdout:removed './140/10' 2019-12-05T05:08:25.097 INFO:tasks.workunit.client.0.smithi042.stdout:removed './140/34' 2019-12-05T05:08:25.098 INFO:tasks.workunit.client.0.smithi042.stdout:removed './140/38' 2019-12-05T05:08:25.098 INFO:tasks.workunit.client.0.smithi042.stdout:removed './140/11' 2019-12-05T05:08:25.098 INFO:tasks.workunit.client.0.smithi042.stdout:removed './140/16' 2019-12-05T05:08:25.099 INFO:tasks.workunit.client.0.smithi042.stdout:removed './140/32' 2019-12-05T05:08:25.099 INFO:tasks.workunit.client.0.smithi042.stdout:removed './140/30' 2019-12-05T05:08:25.099 INFO:tasks.workunit.client.0.smithi042.stdout:removed './140/14' 2019-12-05T05:08:25.099 INFO:tasks.workunit.client.0.smithi042.stdout:removed './140/18' 2019-12-05T05:08:25.099 INFO:tasks.workunit.client.0.smithi042.stdout:removed './140/13' 2019-12-05T05:08:25.099 INFO:tasks.workunit.client.0.smithi042.stdout:removed './140/33' 2019-12-05T05:08:25.099 INFO:tasks.workunit.client.0.smithi042.stdout:removed './140/17' 2019-12-05T05:08:25.100 INFO:tasks.workunit.client.0.smithi042.stdout:removed './140/35' 2019-12-05T05:08:25.100 INFO:tasks.workunit.client.0.smithi042.stdout:removed './140/37' 2019-12-05T05:08:25.100 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './140' 2019-12-05T05:08:25.100 INFO:tasks.workunit.client.0.smithi042.stdout:removed './141/19' 2019-12-05T05:08:25.100 INFO:tasks.workunit.client.0.smithi042.stdout:removed './141/15' 2019-12-05T05:08:25.100 INFO:tasks.workunit.client.0.smithi042.stdout:removed './141/31' 2019-12-05T05:08:25.100 INFO:tasks.workunit.client.0.smithi042.stdout:removed './141/36' 2019-12-05T05:08:25.101 INFO:tasks.workunit.client.0.smithi042.stdout:removed './141/20' 2019-12-05T05:08:25.101 INFO:tasks.workunit.client.0.smithi042.stdout:removed './141/12' 2019-12-05T05:08:25.101 INFO:tasks.workunit.client.0.smithi042.stdout:removed './141/39' 2019-12-05T05:08:25.101 INFO:tasks.workunit.client.0.smithi042.stdout:removed './141/10' 2019-12-05T05:08:25.101 INFO:tasks.workunit.client.0.smithi042.stdout:removed './141/34' 2019-12-05T05:08:25.101 INFO:tasks.workunit.client.0.smithi042.stdout:removed './141/38' 2019-12-05T05:08:25.101 INFO:tasks.workunit.client.0.smithi042.stdout:removed './141/11' 2019-12-05T05:08:25.102 INFO:tasks.workunit.client.0.smithi042.stdout:removed './141/16' 2019-12-05T05:08:25.102 INFO:tasks.workunit.client.0.smithi042.stdout:removed './141/32' 2019-12-05T05:08:25.102 INFO:tasks.workunit.client.0.smithi042.stdout:removed './141/30' 2019-12-05T05:08:25.102 INFO:tasks.workunit.client.0.smithi042.stdout:removed './141/14' 2019-12-05T05:08:25.102 INFO:tasks.workunit.client.0.smithi042.stdout:removed './141/18' 2019-12-05T05:08:25.102 INFO:tasks.workunit.client.0.smithi042.stdout:removed './141/13' 2019-12-05T05:08:25.103 INFO:tasks.workunit.client.0.smithi042.stdout:removed './141/33' 2019-12-05T05:08:25.103 INFO:tasks.workunit.client.0.smithi042.stdout:removed './141/17' 2019-12-05T05:08:25.103 INFO:tasks.workunit.client.0.smithi042.stdout:removed './141/35' 2019-12-05T05:08:25.103 INFO:tasks.workunit.client.0.smithi042.stdout:removed './141/37' 2019-12-05T05:08:25.103 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './141' 2019-12-05T05:08:25.104 INFO:tasks.workunit.client.0.smithi042.stdout:removed './142/19' 2019-12-05T05:08:25.104 INFO:tasks.workunit.client.0.smithi042.stdout:removed './142/15' 2019-12-05T05:08:25.104 INFO:tasks.workunit.client.0.smithi042.stdout:removed './142/31' 2019-12-05T05:08:25.104 INFO:tasks.workunit.client.0.smithi042.stdout:removed './142/36' 2019-12-05T05:08:25.104 INFO:tasks.workunit.client.0.smithi042.stdout:removed './142/20' 2019-12-05T05:08:25.104 INFO:tasks.workunit.client.0.smithi042.stdout:removed './142/12' 2019-12-05T05:08:25.104 INFO:tasks.workunit.client.0.smithi042.stdout:removed './142/39' 2019-12-05T05:08:25.105 INFO:tasks.workunit.client.0.smithi042.stdout:removed './142/10' 2019-12-05T05:08:25.105 INFO:tasks.workunit.client.0.smithi042.stdout:removed './142/34' 2019-12-05T05:08:25.105 INFO:tasks.workunit.client.0.smithi042.stdout:removed './142/38' 2019-12-05T05:08:25.105 INFO:tasks.workunit.client.0.smithi042.stdout:removed './142/11' 2019-12-05T05:08:25.105 INFO:tasks.workunit.client.0.smithi042.stdout:removed './142/16' 2019-12-05T05:08:25.105 INFO:tasks.workunit.client.0.smithi042.stdout:removed './142/32' 2019-12-05T05:08:25.105 INFO:tasks.workunit.client.0.smithi042.stdout:removed './142/30' 2019-12-05T05:08:25.106 INFO:tasks.workunit.client.0.smithi042.stdout:removed './142/14' 2019-12-05T05:08:25.106 INFO:tasks.workunit.client.0.smithi042.stdout:removed './142/18' 2019-12-05T05:08:25.106 INFO:tasks.workunit.client.0.smithi042.stdout:removed './142/13' 2019-12-05T05:08:25.106 INFO:tasks.workunit.client.0.smithi042.stdout:removed './142/33' 2019-12-05T05:08:25.107 INFO:tasks.workunit.client.0.smithi042.stdout:removed './142/17' 2019-12-05T05:08:25.107 INFO:tasks.workunit.client.0.smithi042.stdout:removed './142/35' 2019-12-05T05:08:25.107 INFO:tasks.workunit.client.0.smithi042.stdout:removed './142/37' 2019-12-05T05:08:25.107 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './142' 2019-12-05T05:08:25.108 INFO:tasks.workunit.client.0.smithi042.stdout:removed './143/19' 2019-12-05T05:08:25.108 INFO:tasks.workunit.client.0.smithi042.stdout:removed './143/15' 2019-12-05T05:08:25.108 INFO:tasks.workunit.client.0.smithi042.stdout:removed './143/31' 2019-12-05T05:08:25.108 INFO:tasks.workunit.client.0.smithi042.stdout:removed './143/36' 2019-12-05T05:08:25.108 INFO:tasks.workunit.client.0.smithi042.stdout:removed './143/20' 2019-12-05T05:08:25.108 INFO:tasks.workunit.client.0.smithi042.stdout:removed './143/12' 2019-12-05T05:08:25.109 INFO:tasks.workunit.client.0.smithi042.stdout:removed './143/39' 2019-12-05T05:08:25.109 INFO:tasks.workunit.client.0.smithi042.stdout:removed './143/10' 2019-12-05T05:08:25.109 INFO:tasks.workunit.client.0.smithi042.stdout:removed './143/34' 2019-12-05T05:08:25.110 INFO:tasks.workunit.client.0.smithi042.stdout:removed './143/38' 2019-12-05T05:08:25.110 INFO:tasks.workunit.client.0.smithi042.stdout:removed './143/11' 2019-12-05T05:08:25.110 INFO:tasks.workunit.client.0.smithi042.stdout:removed './143/16' 2019-12-05T05:08:25.110 INFO:tasks.workunit.client.0.smithi042.stdout:removed './143/32' 2019-12-05T05:08:25.111 INFO:tasks.workunit.client.0.smithi042.stdout:removed './143/30' 2019-12-05T05:08:25.112 INFO:tasks.workunit.client.0.smithi042.stdout:removed './143/14' 2019-12-05T05:08:25.112 INFO:tasks.workunit.client.0.smithi042.stdout:removed './143/18' 2019-12-05T05:08:25.112 INFO:tasks.workunit.client.0.smithi042.stdout:removed './143/13' 2019-12-05T05:08:25.112 INFO:tasks.workunit.client.0.smithi042.stdout:removed './143/33' 2019-12-05T05:08:25.112 INFO:tasks.workunit.client.0.smithi042.stdout:removed './143/17' 2019-12-05T05:08:25.112 INFO:tasks.workunit.client.0.smithi042.stdout:removed './143/35' 2019-12-05T05:08:25.112 INFO:tasks.workunit.client.0.smithi042.stdout:removed './143/37' 2019-12-05T05:08:25.113 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './143' 2019-12-05T05:08:25.113 INFO:tasks.workunit.client.0.smithi042.stdout:removed './144/19' 2019-12-05T05:08:25.113 INFO:tasks.workunit.client.0.smithi042.stdout:removed './144/15' 2019-12-05T05:08:25.113 INFO:tasks.workunit.client.0.smithi042.stdout:removed './144/31' 2019-12-05T05:08:25.113 INFO:tasks.workunit.client.0.smithi042.stdout:removed './144/36' 2019-12-05T05:08:25.113 INFO:tasks.workunit.client.0.smithi042.stdout:removed './144/20' 2019-12-05T05:08:25.114 INFO:tasks.workunit.client.0.smithi042.stdout:removed './144/12' 2019-12-05T05:08:25.114 INFO:tasks.workunit.client.0.smithi042.stdout:removed './144/39' 2019-12-05T05:08:25.114 INFO:tasks.workunit.client.0.smithi042.stdout:removed './144/10' 2019-12-05T05:08:25.114 INFO:tasks.workunit.client.0.smithi042.stdout:removed './144/34' 2019-12-05T05:08:25.114 INFO:tasks.workunit.client.0.smithi042.stdout:removed './144/38' 2019-12-05T05:08:25.114 INFO:tasks.workunit.client.0.smithi042.stdout:removed './144/11' 2019-12-05T05:08:25.114 INFO:tasks.workunit.client.0.smithi042.stdout:removed './144/16' 2019-12-05T05:08:25.115 INFO:tasks.workunit.client.0.smithi042.stdout:removed './144/32' 2019-12-05T05:08:25.115 INFO:tasks.workunit.client.0.smithi042.stdout:removed './144/30' 2019-12-05T05:08:25.115 INFO:tasks.workunit.client.0.smithi042.stdout:removed './144/14' 2019-12-05T05:08:25.115 INFO:tasks.workunit.client.0.smithi042.stdout:removed './144/18' 2019-12-05T05:08:25.115 INFO:tasks.workunit.client.0.smithi042.stdout:removed './144/13' 2019-12-05T05:08:25.116 INFO:tasks.workunit.client.0.smithi042.stdout:removed './144/33' 2019-12-05T05:08:25.116 INFO:tasks.workunit.client.0.smithi042.stdout:removed './144/17' 2019-12-05T05:08:25.116 INFO:tasks.workunit.client.0.smithi042.stdout:removed './144/35' 2019-12-05T05:08:25.116 INFO:tasks.workunit.client.0.smithi042.stdout:removed './144/37' 2019-12-05T05:08:25.116 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './144' 2019-12-05T05:08:25.116 INFO:tasks.workunit.client.0.smithi042.stdout:removed './145/19' 2019-12-05T05:08:25.116 INFO:tasks.workunit.client.0.smithi042.stdout:removed './145/15' 2019-12-05T05:08:25.117 INFO:tasks.workunit.client.0.smithi042.stdout:removed './145/31' 2019-12-05T05:08:25.117 INFO:tasks.workunit.client.0.smithi042.stdout:removed './145/36' 2019-12-05T05:08:25.117 INFO:tasks.workunit.client.0.smithi042.stdout:removed './145/20' 2019-12-05T05:08:25.117 INFO:tasks.workunit.client.0.smithi042.stdout:removed './145/12' 2019-12-05T05:08:25.117 INFO:tasks.workunit.client.0.smithi042.stdout:removed './145/39' 2019-12-05T05:08:25.118 INFO:tasks.workunit.client.0.smithi042.stdout:removed './145/10' 2019-12-05T05:08:25.118 INFO:tasks.workunit.client.0.smithi042.stdout:removed './145/34' 2019-12-05T05:08:25.118 INFO:tasks.workunit.client.0.smithi042.stdout:removed './145/38' 2019-12-05T05:08:25.118 INFO:tasks.workunit.client.0.smithi042.stdout:removed './145/11' 2019-12-05T05:08:25.118 INFO:tasks.workunit.client.0.smithi042.stdout:removed './145/16' 2019-12-05T05:08:25.118 INFO:tasks.workunit.client.0.smithi042.stdout:removed './145/32' 2019-12-05T05:08:25.118 INFO:tasks.workunit.client.0.smithi042.stdout:removed './145/30' 2019-12-05T05:08:25.119 INFO:tasks.workunit.client.0.smithi042.stdout:removed './145/14' 2019-12-05T05:08:25.119 INFO:tasks.workunit.client.0.smithi042.stdout:removed './145/18' 2019-12-05T05:08:25.119 INFO:tasks.workunit.client.0.smithi042.stdout:removed './145/13' 2019-12-05T05:08:25.120 INFO:tasks.workunit.client.0.smithi042.stdout:removed './145/33' 2019-12-05T05:08:25.120 INFO:tasks.workunit.client.0.smithi042.stdout:removed './145/17' 2019-12-05T05:08:25.120 INFO:tasks.workunit.client.0.smithi042.stdout:removed './145/35' 2019-12-05T05:08:25.120 INFO:tasks.workunit.client.0.smithi042.stdout:removed './145/37' 2019-12-05T05:08:25.120 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './145' 2019-12-05T05:08:25.120 INFO:tasks.workunit.client.0.smithi042.stdout:removed './146/19' 2019-12-05T05:08:25.120 INFO:tasks.workunit.client.0.smithi042.stdout:removed './146/15' 2019-12-05T05:08:25.121 INFO:tasks.workunit.client.0.smithi042.stdout:removed './146/31' 2019-12-05T05:08:25.121 INFO:tasks.workunit.client.0.smithi042.stdout:removed './146/36' 2019-12-05T05:08:25.121 INFO:tasks.workunit.client.0.smithi042.stdout:removed './146/20' 2019-12-05T05:08:25.121 INFO:tasks.workunit.client.0.smithi042.stdout:removed './146/12' 2019-12-05T05:08:25.121 INFO:tasks.workunit.client.0.smithi042.stdout:removed './146/39' 2019-12-05T05:08:25.121 INFO:tasks.workunit.client.0.smithi042.stdout:removed './146/10' 2019-12-05T05:08:25.121 INFO:tasks.workunit.client.0.smithi042.stdout:removed './146/34' 2019-12-05T05:08:25.122 INFO:tasks.workunit.client.0.smithi042.stdout:removed './146/38' 2019-12-05T05:08:25.122 INFO:tasks.workunit.client.0.smithi042.stdout:removed './146/11' 2019-12-05T05:08:25.122 INFO:tasks.workunit.client.0.smithi042.stdout:removed './146/16' 2019-12-05T05:08:25.122 INFO:tasks.workunit.client.0.smithi042.stdout:removed './146/32' 2019-12-05T05:08:25.122 INFO:tasks.workunit.client.0.smithi042.stdout:removed './146/30' 2019-12-05T05:08:25.122 INFO:tasks.workunit.client.0.smithi042.stdout:removed './146/14' 2019-12-05T05:08:25.122 INFO:tasks.workunit.client.0.smithi042.stdout:removed './146/18' 2019-12-05T05:08:25.122 INFO:tasks.workunit.client.0.smithi042.stdout:removed './146/13' 2019-12-05T05:08:25.123 INFO:tasks.workunit.client.0.smithi042.stdout:removed './146/33' 2019-12-05T05:08:25.123 INFO:tasks.workunit.client.0.smithi042.stdout:removed './146/17' 2019-12-05T05:08:25.123 INFO:tasks.workunit.client.0.smithi042.stdout:removed './146/35' 2019-12-05T05:08:25.123 INFO:tasks.workunit.client.0.smithi042.stdout:removed './146/37' 2019-12-05T05:08:25.123 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './146' 2019-12-05T05:08:25.123 INFO:tasks.workunit.client.0.smithi042.stdout:removed './147/19' 2019-12-05T05:08:25.124 INFO:tasks.workunit.client.0.smithi042.stdout:removed './147/15' 2019-12-05T05:08:25.124 INFO:tasks.workunit.client.0.smithi042.stdout:removed './147/31' 2019-12-05T05:08:25.124 INFO:tasks.workunit.client.0.smithi042.stdout:removed './147/36' 2019-12-05T05:08:25.124 INFO:tasks.workunit.client.0.smithi042.stdout:removed './147/20' 2019-12-05T05:08:25.124 INFO:tasks.workunit.client.0.smithi042.stdout:removed './147/12' 2019-12-05T05:08:25.125 INFO:tasks.workunit.client.0.smithi042.stdout:removed './147/39' 2019-12-05T05:08:25.125 INFO:tasks.workunit.client.0.smithi042.stdout:removed './147/10' 2019-12-05T05:08:25.125 INFO:tasks.workunit.client.0.smithi042.stdout:removed './147/34' 2019-12-05T05:08:25.125 INFO:tasks.workunit.client.0.smithi042.stdout:removed './147/38' 2019-12-05T05:08:25.125 INFO:tasks.workunit.client.0.smithi042.stdout:removed './147/11' 2019-12-05T05:08:25.132 INFO:tasks.workunit.client.0.smithi042.stdout:removed './147/16' 2019-12-05T05:08:25.132 INFO:tasks.workunit.client.0.smithi042.stdout:removed './147/32' 2019-12-05T05:08:25.132 INFO:tasks.workunit.client.0.smithi042.stdout:removed './147/30' 2019-12-05T05:08:25.132 INFO:tasks.workunit.client.0.smithi042.stdout:removed './147/14' 2019-12-05T05:08:25.132 INFO:tasks.workunit.client.0.smithi042.stdout:removed './147/18' 2019-12-05T05:08:25.132 INFO:tasks.workunit.client.0.smithi042.stdout:removed './147/13' 2019-12-05T05:08:25.133 INFO:tasks.workunit.client.0.smithi042.stdout:removed './147/33' 2019-12-05T05:08:25.133 INFO:tasks.workunit.client.0.smithi042.stdout:removed './147/17' 2019-12-05T05:08:25.133 INFO:tasks.workunit.client.0.smithi042.stdout:removed './147/35' 2019-12-05T05:08:25.133 INFO:tasks.workunit.client.0.smithi042.stdout:removed './147/37' 2019-12-05T05:08:25.133 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './147' 2019-12-05T05:08:25.133 INFO:tasks.workunit.client.0.smithi042.stdout:removed './148/19' 2019-12-05T05:08:25.133 INFO:tasks.workunit.client.0.smithi042.stdout:removed './148/15' 2019-12-05T05:08:25.134 INFO:tasks.workunit.client.0.smithi042.stdout:removed './148/31' 2019-12-05T05:08:25.134 INFO:tasks.workunit.client.0.smithi042.stdout:removed './148/36' 2019-12-05T05:08:28.803 INFO:tasks.workunit.client.0.smithi042.stdout:removed './148/20' 2019-12-05T05:08:28.803 INFO:tasks.workunit.client.0.smithi042.stdout:removed './148/12' 2019-12-05T05:08:28.803 INFO:tasks.workunit.client.0.smithi042.stdout:removed './148/39' 2019-12-05T05:08:28.803 INFO:tasks.workunit.client.0.smithi042.stdout:removed './148/10' 2019-12-05T05:08:28.804 INFO:tasks.workunit.client.0.smithi042.stdout:removed './148/34' 2019-12-05T05:08:28.804 INFO:tasks.workunit.client.0.smithi042.stdout:removed './148/38' 2019-12-05T05:08:28.804 INFO:tasks.workunit.client.0.smithi042.stdout:removed './148/11' 2019-12-05T05:08:28.804 INFO:tasks.workunit.client.0.smithi042.stdout:removed './148/16' 2019-12-05T05:08:28.804 INFO:tasks.workunit.client.0.smithi042.stdout:removed './148/32' 2019-12-05T05:08:28.804 INFO:tasks.workunit.client.0.smithi042.stdout:removed './148/30' 2019-12-05T05:08:28.804 INFO:tasks.workunit.client.0.smithi042.stdout:removed './148/14' 2019-12-05T05:08:28.804 INFO:tasks.workunit.client.0.smithi042.stdout:removed './148/18' 2019-12-05T05:08:28.805 INFO:tasks.workunit.client.0.smithi042.stdout:removed './148/13' 2019-12-05T05:08:28.805 INFO:tasks.workunit.client.0.smithi042.stdout:removed './148/33' 2019-12-05T05:08:28.805 INFO:tasks.workunit.client.0.smithi042.stdout:removed './148/17' 2019-12-05T05:08:28.805 INFO:tasks.workunit.client.0.smithi042.stdout:removed './148/35' 2019-12-05T05:08:28.805 INFO:tasks.workunit.client.0.smithi042.stdout:removed './148/37' 2019-12-05T05:08:28.805 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './148' 2019-12-05T05:08:28.805 INFO:tasks.workunit.client.0.smithi042.stdout:removed './149/19' 2019-12-05T05:08:28.806 INFO:tasks.workunit.client.0.smithi042.stdout:removed './149/15' 2019-12-05T05:08:28.806 INFO:tasks.workunit.client.0.smithi042.stdout:removed './149/31' 2019-12-05T05:08:28.806 INFO:tasks.workunit.client.0.smithi042.stdout:removed './149/36' 2019-12-05T05:08:28.806 INFO:tasks.workunit.client.0.smithi042.stdout:removed './149/20' 2019-12-05T05:08:28.806 INFO:tasks.workunit.client.0.smithi042.stdout:removed './149/12' 2019-12-05T05:08:28.806 INFO:tasks.workunit.client.0.smithi042.stdout:removed './149/39' 2019-12-05T05:08:28.806 INFO:tasks.workunit.client.0.smithi042.stdout:removed './149/10' 2019-12-05T05:08:28.807 INFO:tasks.workunit.client.0.smithi042.stdout:removed './149/34' 2019-12-05T05:08:28.807 INFO:tasks.workunit.client.0.smithi042.stdout:removed './149/38' 2019-12-05T05:08:28.807 INFO:tasks.workunit.client.0.smithi042.stdout:removed './149/11' 2019-12-05T05:08:28.807 INFO:tasks.workunit.client.0.smithi042.stdout:removed './149/16' 2019-12-05T05:08:28.807 INFO:tasks.workunit.client.0.smithi042.stdout:removed './149/32' 2019-12-05T05:08:28.807 INFO:tasks.workunit.client.0.smithi042.stdout:removed './149/30' 2019-12-05T05:08:28.807 INFO:tasks.workunit.client.0.smithi042.stdout:removed './149/14' 2019-12-05T05:08:28.807 INFO:tasks.workunit.client.0.smithi042.stdout:removed './149/18' 2019-12-05T05:08:28.808 INFO:tasks.workunit.client.0.smithi042.stdout:removed './149/13' 2019-12-05T05:08:28.808 INFO:tasks.workunit.client.0.smithi042.stdout:removed './149/33' 2019-12-05T05:08:28.808 INFO:tasks.workunit.client.0.smithi042.stdout:removed './149/17' 2019-12-05T05:08:28.808 INFO:tasks.workunit.client.0.smithi042.stdout:removed './149/35' 2019-12-05T05:08:28.808 INFO:tasks.workunit.client.0.smithi042.stdout:removed './149/37' 2019-12-05T05:08:28.808 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './149' 2019-12-05T05:08:28.808 INFO:tasks.workunit.client.0.smithi042.stdout:removed './150/19' 2019-12-05T05:08:28.809 INFO:tasks.workunit.client.0.smithi042.stdout:removed './150/15' 2019-12-05T05:08:28.809 INFO:tasks.workunit.client.0.smithi042.stdout:removed './150/31' 2019-12-05T05:08:28.809 INFO:tasks.workunit.client.0.smithi042.stdout:removed './150/36' 2019-12-05T05:08:28.809 INFO:tasks.workunit.client.0.smithi042.stdout:removed './150/20' 2019-12-05T05:08:28.809 INFO:tasks.workunit.client.0.smithi042.stdout:removed './150/12' 2019-12-05T05:08:28.809 INFO:tasks.workunit.client.0.smithi042.stdout:removed './150/39' 2019-12-05T05:08:28.809 INFO:tasks.workunit.client.0.smithi042.stdout:removed './150/10' 2019-12-05T05:08:28.809 INFO:tasks.workunit.client.0.smithi042.stdout:removed './150/34' 2019-12-05T05:08:28.810 INFO:tasks.workunit.client.0.smithi042.stdout:removed './150/38' 2019-12-05T05:08:28.810 INFO:tasks.workunit.client.0.smithi042.stdout:removed './150/11' 2019-12-05T05:08:28.810 INFO:tasks.workunit.client.0.smithi042.stdout:removed './150/16' 2019-12-05T05:08:28.810 INFO:tasks.workunit.client.0.smithi042.stdout:removed './150/32' 2019-12-05T05:08:28.810 INFO:tasks.workunit.client.0.smithi042.stdout:removed './150/30' 2019-12-05T05:08:28.810 INFO:tasks.workunit.client.0.smithi042.stdout:removed './150/14' 2019-12-05T05:08:28.810 INFO:tasks.workunit.client.0.smithi042.stdout:removed './150/18' 2019-12-05T05:08:28.811 INFO:tasks.workunit.client.0.smithi042.stdout:removed './150/13' 2019-12-05T05:08:28.811 INFO:tasks.workunit.client.0.smithi042.stdout:removed './150/33' 2019-12-05T05:08:28.811 INFO:tasks.workunit.client.0.smithi042.stdout:removed './150/17' 2019-12-05T05:08:28.811 INFO:tasks.workunit.client.0.smithi042.stdout:removed './150/35' 2019-12-05T05:08:28.811 INFO:tasks.workunit.client.0.smithi042.stdout:removed './150/37' 2019-12-05T05:08:28.811 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './150' 2019-12-05T05:08:28.811 INFO:tasks.workunit.client.0.smithi042.stdout:removed './151/19' 2019-12-05T05:08:28.811 INFO:tasks.workunit.client.0.smithi042.stdout:removed './151/15' 2019-12-05T05:08:28.812 INFO:tasks.workunit.client.0.smithi042.stdout:removed './151/31' 2019-12-05T05:08:28.812 INFO:tasks.workunit.client.0.smithi042.stdout:removed './151/36' 2019-12-05T05:08:28.812 INFO:tasks.workunit.client.0.smithi042.stdout:removed './151/20' 2019-12-05T05:08:28.812 INFO:tasks.workunit.client.0.smithi042.stdout:removed './151/12' 2019-12-05T05:08:28.812 INFO:tasks.workunit.client.0.smithi042.stdout:removed './151/39' 2019-12-05T05:08:28.812 INFO:tasks.workunit.client.0.smithi042.stdout:removed './151/10' 2019-12-05T05:08:28.812 INFO:tasks.workunit.client.0.smithi042.stdout:removed './151/34' 2019-12-05T05:08:28.813 INFO:tasks.workunit.client.0.smithi042.stdout:removed './151/38' 2019-12-05T05:08:28.813 INFO:tasks.workunit.client.0.smithi042.stdout:removed './151/11' 2019-12-05T05:08:28.813 INFO:tasks.workunit.client.0.smithi042.stdout:removed './151/16' 2019-12-05T05:08:28.813 INFO:tasks.workunit.client.0.smithi042.stdout:removed './151/32' 2019-12-05T05:08:28.813 INFO:tasks.workunit.client.0.smithi042.stdout:removed './151/30' 2019-12-05T05:08:28.813 INFO:tasks.workunit.client.0.smithi042.stdout:removed './151/14' 2019-12-05T05:08:28.813 INFO:tasks.workunit.client.0.smithi042.stdout:removed './151/18' 2019-12-05T05:08:28.813 INFO:tasks.workunit.client.0.smithi042.stdout:removed './151/13' 2019-12-05T05:08:28.814 INFO:tasks.workunit.client.0.smithi042.stdout:removed './151/33' 2019-12-05T05:08:28.814 INFO:tasks.workunit.client.0.smithi042.stdout:removed './151/17' 2019-12-05T05:08:28.814 INFO:tasks.workunit.client.0.smithi042.stdout:removed './151/35' 2019-12-05T05:08:28.814 INFO:tasks.workunit.client.0.smithi042.stdout:removed './151/37' 2019-12-05T05:08:28.814 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './151' 2019-12-05T05:08:28.814 INFO:tasks.workunit.client.0.smithi042.stdout:removed './152/19' 2019-12-05T05:08:28.814 INFO:tasks.workunit.client.0.smithi042.stdout:removed './152/15' 2019-12-05T05:08:28.815 INFO:tasks.workunit.client.0.smithi042.stdout:removed './152/31' 2019-12-05T05:08:28.815 INFO:tasks.workunit.client.0.smithi042.stdout:removed './152/36' 2019-12-05T05:08:28.815 INFO:tasks.workunit.client.0.smithi042.stdout:removed './152/20' 2019-12-05T05:08:28.815 INFO:tasks.workunit.client.0.smithi042.stdout:removed './152/12' 2019-12-05T05:08:28.815 INFO:tasks.workunit.client.0.smithi042.stdout:removed './152/39' 2019-12-05T05:08:28.815 INFO:tasks.workunit.client.0.smithi042.stdout:removed './152/10' 2019-12-05T05:08:28.815 INFO:tasks.workunit.client.0.smithi042.stdout:removed './152/34' 2019-12-05T05:08:28.815 INFO:tasks.workunit.client.0.smithi042.stdout:removed './152/38' 2019-12-05T05:08:28.816 INFO:tasks.workunit.client.0.smithi042.stdout:removed './152/11' 2019-12-05T05:08:28.816 INFO:tasks.workunit.client.0.smithi042.stdout:removed './152/16' 2019-12-05T05:08:28.816 INFO:tasks.workunit.client.0.smithi042.stdout:removed './152/32' 2019-12-05T05:08:28.816 INFO:tasks.workunit.client.0.smithi042.stdout:removed './152/30' 2019-12-05T05:08:28.816 INFO:tasks.workunit.client.0.smithi042.stdout:removed './152/14' 2019-12-05T05:08:28.816 INFO:tasks.workunit.client.0.smithi042.stdout:removed './152/18' 2019-12-05T05:08:28.816 INFO:tasks.workunit.client.0.smithi042.stdout:removed './152/13' 2019-12-05T05:08:28.817 INFO:tasks.workunit.client.0.smithi042.stdout:removed './152/33' 2019-12-05T05:08:28.817 INFO:tasks.workunit.client.0.smithi042.stdout:removed './152/17' 2019-12-05T05:08:28.817 INFO:tasks.workunit.client.0.smithi042.stdout:removed './152/35' 2019-12-05T05:08:28.817 INFO:tasks.workunit.client.0.smithi042.stdout:removed './152/37' 2019-12-05T05:08:28.817 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './152' 2019-12-05T05:08:28.817 INFO:tasks.workunit.client.0.smithi042.stdout:removed './153/19' 2019-12-05T05:08:28.817 INFO:tasks.workunit.client.0.smithi042.stdout:removed './153/15' 2019-12-05T05:08:28.817 INFO:tasks.workunit.client.0.smithi042.stdout:removed './153/31' 2019-12-05T05:08:28.818 INFO:tasks.workunit.client.0.smithi042.stdout:removed './153/36' 2019-12-05T05:08:28.818 INFO:tasks.workunit.client.0.smithi042.stdout:removed './153/20' 2019-12-05T05:08:28.818 INFO:tasks.workunit.client.0.smithi042.stdout:removed './153/12' 2019-12-05T05:08:28.818 INFO:tasks.workunit.client.0.smithi042.stdout:removed './153/39' 2019-12-05T05:08:28.818 INFO:tasks.workunit.client.0.smithi042.stdout:removed './153/10' 2019-12-05T05:08:28.818 INFO:tasks.workunit.client.0.smithi042.stdout:removed './153/34' 2019-12-05T05:08:28.818 INFO:tasks.workunit.client.0.smithi042.stdout:removed './153/38' 2019-12-05T05:08:28.819 INFO:tasks.workunit.client.0.smithi042.stdout:removed './153/11' 2019-12-05T05:08:28.819 INFO:tasks.workunit.client.0.smithi042.stdout:removed './153/16' 2019-12-05T05:08:28.819 INFO:tasks.workunit.client.0.smithi042.stdout:removed './153/32' 2019-12-05T05:08:28.819 INFO:tasks.workunit.client.0.smithi042.stdout:removed './153/30' 2019-12-05T05:08:28.819 INFO:tasks.workunit.client.0.smithi042.stdout:removed './153/14' 2019-12-05T05:08:28.819 INFO:tasks.workunit.client.0.smithi042.stdout:removed './153/18' 2019-12-05T05:08:28.819 INFO:tasks.workunit.client.0.smithi042.stdout:removed './153/13' 2019-12-05T05:08:28.819 INFO:tasks.workunit.client.0.smithi042.stdout:removed './153/33' 2019-12-05T05:08:28.820 INFO:tasks.workunit.client.0.smithi042.stdout:removed './153/17' 2019-12-05T05:08:28.820 INFO:tasks.workunit.client.0.smithi042.stdout:removed './153/35' 2019-12-05T05:08:28.820 INFO:tasks.workunit.client.0.smithi042.stdout:removed './153/37' 2019-12-05T05:08:28.820 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './153' 2019-12-05T05:08:28.820 INFO:tasks.workunit.client.0.smithi042.stdout:removed './154/19' 2019-12-05T05:08:28.820 INFO:tasks.workunit.client.0.smithi042.stdout:removed './154/15' 2019-12-05T05:08:28.820 INFO:tasks.workunit.client.0.smithi042.stdout:removed './154/31' 2019-12-05T05:08:28.821 INFO:tasks.workunit.client.0.smithi042.stdout:removed './154/36' 2019-12-05T05:08:28.821 INFO:tasks.workunit.client.0.smithi042.stdout:removed './154/20' 2019-12-05T05:08:28.821 INFO:tasks.workunit.client.0.smithi042.stdout:removed './154/12' 2019-12-05T05:08:28.821 INFO:tasks.workunit.client.0.smithi042.stdout:removed './154/39' 2019-12-05T05:08:28.821 INFO:tasks.workunit.client.0.smithi042.stdout:removed './154/10' 2019-12-05T05:08:28.821 INFO:tasks.workunit.client.0.smithi042.stdout:removed './154/34' 2019-12-05T05:08:28.821 INFO:tasks.workunit.client.0.smithi042.stdout:removed './154/38' 2019-12-05T05:08:28.821 INFO:tasks.workunit.client.0.smithi042.stdout:removed './154/11' 2019-12-05T05:08:28.822 INFO:tasks.workunit.client.0.smithi042.stdout:removed './154/16' 2019-12-05T05:08:28.822 INFO:tasks.workunit.client.0.smithi042.stdout:removed './154/32' 2019-12-05T05:08:28.822 INFO:tasks.workunit.client.0.smithi042.stdout:removed './154/30' 2019-12-05T05:08:28.822 INFO:tasks.workunit.client.0.smithi042.stdout:removed './154/14' 2019-12-05T05:08:28.822 INFO:tasks.workunit.client.0.smithi042.stdout:removed './154/18' 2019-12-05T05:08:28.822 INFO:tasks.workunit.client.0.smithi042.stdout:removed './154/13' 2019-12-05T05:08:28.822 INFO:tasks.workunit.client.0.smithi042.stdout:removed './154/33' 2019-12-05T05:08:28.823 INFO:tasks.workunit.client.0.smithi042.stdout:removed './154/17' 2019-12-05T05:08:28.823 INFO:tasks.workunit.client.0.smithi042.stdout:removed './154/35' 2019-12-05T05:08:28.823 INFO:tasks.workunit.client.0.smithi042.stdout:removed './154/37' 2019-12-05T05:08:28.823 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './154' 2019-12-05T05:08:28.823 INFO:tasks.workunit.client.0.smithi042.stdout:removed './155/19' 2019-12-05T05:08:28.823 INFO:tasks.workunit.client.0.smithi042.stdout:removed './155/15' 2019-12-05T05:08:28.823 INFO:tasks.workunit.client.0.smithi042.stdout:removed './155/31' 2019-12-05T05:08:28.824 INFO:tasks.workunit.client.0.smithi042.stdout:removed './155/36' 2019-12-05T05:08:28.824 INFO:tasks.workunit.client.0.smithi042.stdout:removed './155/20' 2019-12-05T05:08:28.824 INFO:tasks.workunit.client.0.smithi042.stdout:removed './155/12' 2019-12-05T05:08:28.824 INFO:tasks.workunit.client.0.smithi042.stdout:removed './155/39' 2019-12-05T05:08:28.824 INFO:tasks.workunit.client.0.smithi042.stdout:removed './155/10' 2019-12-05T05:08:28.824 INFO:tasks.workunit.client.0.smithi042.stdout:removed './155/34' 2019-12-05T05:08:28.824 INFO:tasks.workunit.client.0.smithi042.stdout:removed './155/38' 2019-12-05T05:08:28.825 INFO:tasks.workunit.client.0.smithi042.stdout:removed './155/11' 2019-12-05T05:08:28.825 INFO:tasks.workunit.client.0.smithi042.stdout:removed './155/16' 2019-12-05T05:08:28.825 INFO:tasks.workunit.client.0.smithi042.stdout:removed './155/32' 2019-12-05T05:08:28.825 INFO:tasks.workunit.client.0.smithi042.stdout:removed './155/30' 2019-12-05T05:08:28.826 INFO:tasks.workunit.client.0.smithi042.stdout:removed './155/14' 2019-12-05T05:08:28.826 INFO:tasks.workunit.client.0.smithi042.stdout:removed './155/18' 2019-12-05T05:08:28.826 INFO:tasks.workunit.client.0.smithi042.stdout:removed './155/13' 2019-12-05T05:08:28.826 INFO:tasks.workunit.client.0.smithi042.stdout:removed './155/33' 2019-12-05T05:08:28.826 INFO:tasks.workunit.client.0.smithi042.stdout:removed './155/17' 2019-12-05T05:08:28.826 INFO:tasks.workunit.client.0.smithi042.stdout:removed './155/35' 2019-12-05T05:08:28.826 INFO:tasks.workunit.client.0.smithi042.stdout:removed './155/37' 2019-12-05T05:08:28.826 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './155' 2019-12-05T05:08:28.827 INFO:tasks.workunit.client.0.smithi042.stdout:removed './156/19' 2019-12-05T05:08:28.827 INFO:tasks.workunit.client.0.smithi042.stdout:removed './156/15' 2019-12-05T05:08:28.827 INFO:tasks.workunit.client.0.smithi042.stdout:removed './156/31' 2019-12-05T05:08:28.827 INFO:tasks.workunit.client.0.smithi042.stdout:removed './156/36' 2019-12-05T05:08:28.827 INFO:tasks.workunit.client.0.smithi042.stdout:removed './156/20' 2019-12-05T05:08:28.827 INFO:tasks.workunit.client.0.smithi042.stdout:removed './156/12' 2019-12-05T05:08:28.827 INFO:tasks.workunit.client.0.smithi042.stdout:removed './156/39' 2019-12-05T05:08:28.828 INFO:tasks.workunit.client.0.smithi042.stdout:removed './156/10' 2019-12-05T05:08:28.828 INFO:tasks.workunit.client.0.smithi042.stdout:removed './156/34' 2019-12-05T05:08:28.828 INFO:tasks.workunit.client.0.smithi042.stdout:removed './156/38' 2019-12-05T05:08:28.828 INFO:tasks.workunit.client.0.smithi042.stdout:removed './156/11' 2019-12-05T05:08:28.828 INFO:tasks.workunit.client.0.smithi042.stdout:removed './156/16' 2019-12-05T05:08:28.828 INFO:tasks.workunit.client.0.smithi042.stdout:removed './156/32' 2019-12-05T05:08:28.828 INFO:tasks.workunit.client.0.smithi042.stdout:removed './156/30' 2019-12-05T05:08:28.829 INFO:tasks.workunit.client.0.smithi042.stdout:removed './156/14' 2019-12-05T05:08:28.829 INFO:tasks.workunit.client.0.smithi042.stdout:removed './156/18' 2019-12-05T05:08:28.829 INFO:tasks.workunit.client.0.smithi042.stdout:removed './156/13' 2019-12-05T05:08:28.829 INFO:tasks.workunit.client.0.smithi042.stdout:removed './156/33' 2019-12-05T05:08:28.829 INFO:tasks.workunit.client.0.smithi042.stdout:removed './156/17' 2019-12-05T05:08:28.829 INFO:tasks.workunit.client.0.smithi042.stdout:removed './156/35' 2019-12-05T05:08:28.829 INFO:tasks.workunit.client.0.smithi042.stdout:removed './156/37' 2019-12-05T05:08:28.829 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './156' 2019-12-05T05:08:28.830 INFO:tasks.workunit.client.0.smithi042.stdout:removed './157/19' 2019-12-05T05:08:28.830 INFO:tasks.workunit.client.0.smithi042.stdout:removed './157/15' 2019-12-05T05:08:28.830 INFO:tasks.workunit.client.0.smithi042.stdout:removed './157/31' 2019-12-05T05:08:28.830 INFO:tasks.workunit.client.0.smithi042.stdout:removed './157/36' 2019-12-05T05:08:28.830 INFO:tasks.workunit.client.0.smithi042.stdout:removed './157/20' 2019-12-05T05:08:28.830 INFO:tasks.workunit.client.0.smithi042.stdout:removed './157/12' 2019-12-05T05:08:28.830 INFO:tasks.workunit.client.0.smithi042.stdout:removed './157/39' 2019-12-05T05:08:28.831 INFO:tasks.workunit.client.0.smithi042.stdout:removed './157/10' 2019-12-05T05:08:28.831 INFO:tasks.workunit.client.0.smithi042.stdout:removed './157/34' 2019-12-05T05:08:28.831 INFO:tasks.workunit.client.0.smithi042.stdout:removed './157/38' 2019-12-05T05:08:28.831 INFO:tasks.workunit.client.0.smithi042.stdout:removed './157/11' 2019-12-05T05:08:28.831 INFO:tasks.workunit.client.0.smithi042.stdout:removed './157/16' 2019-12-05T05:08:28.831 INFO:tasks.workunit.client.0.smithi042.stdout:removed './157/32' 2019-12-05T05:08:28.831 INFO:tasks.workunit.client.0.smithi042.stdout:removed './157/30' 2019-12-05T05:08:28.832 INFO:tasks.workunit.client.0.smithi042.stdout:removed './157/14' 2019-12-05T05:08:28.832 INFO:tasks.workunit.client.0.smithi042.stdout:removed './157/18' 2019-12-05T05:08:28.832 INFO:tasks.workunit.client.0.smithi042.stdout:removed './157/13' 2019-12-05T05:08:28.832 INFO:tasks.workunit.client.0.smithi042.stdout:removed './157/33' 2019-12-05T05:08:31.682 INFO:tasks.workunit.client.0.smithi042.stdout:removed './157/17' 2019-12-05T05:08:31.682 INFO:tasks.workunit.client.0.smithi042.stdout:removed './157/35' 2019-12-05T05:08:31.682 INFO:tasks.workunit.client.0.smithi042.stdout:removed './157/37' 2019-12-05T05:08:31.682 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './157' 2019-12-05T05:08:31.682 INFO:tasks.workunit.client.0.smithi042.stdout:removed './158/19' 2019-12-05T05:08:31.682 INFO:tasks.workunit.client.0.smithi042.stdout:removed './158/15' 2019-12-05T05:08:31.683 INFO:tasks.workunit.client.0.smithi042.stdout:removed './158/31' 2019-12-05T05:08:31.683 INFO:tasks.workunit.client.0.smithi042.stdout:removed './158/36' 2019-12-05T05:08:31.683 INFO:tasks.workunit.client.0.smithi042.stdout:removed './158/20' 2019-12-05T05:08:31.683 INFO:tasks.workunit.client.0.smithi042.stdout:removed './158/12' 2019-12-05T05:08:31.683 INFO:tasks.workunit.client.0.smithi042.stdout:removed './158/39' 2019-12-05T05:08:31.683 INFO:tasks.workunit.client.0.smithi042.stdout:removed './158/10' 2019-12-05T05:08:31.683 INFO:tasks.workunit.client.0.smithi042.stdout:removed './158/34' 2019-12-05T05:08:31.684 INFO:tasks.workunit.client.0.smithi042.stdout:removed './158/38' 2019-12-05T05:08:31.684 INFO:tasks.workunit.client.0.smithi042.stdout:removed './158/11' 2019-12-05T05:08:31.684 INFO:tasks.workunit.client.0.smithi042.stdout:removed './158/16' 2019-12-05T05:08:31.684 INFO:tasks.workunit.client.0.smithi042.stdout:removed './158/32' 2019-12-05T05:08:31.684 INFO:tasks.workunit.client.0.smithi042.stdout:removed './158/30' 2019-12-05T05:08:31.684 INFO:tasks.workunit.client.0.smithi042.stdout:removed './158/14' 2019-12-05T05:08:31.684 INFO:tasks.workunit.client.0.smithi042.stdout:removed './158/18' 2019-12-05T05:08:31.685 INFO:tasks.workunit.client.0.smithi042.stdout:removed './158/13' 2019-12-05T05:08:31.685 INFO:tasks.workunit.client.0.smithi042.stdout:removed './158/33' 2019-12-05T05:08:31.685 INFO:tasks.workunit.client.0.smithi042.stdout:removed './158/17' 2019-12-05T05:08:31.685 INFO:tasks.workunit.client.0.smithi042.stdout:removed './158/35' 2019-12-05T05:08:31.685 INFO:tasks.workunit.client.0.smithi042.stdout:removed './158/37' 2019-12-05T05:08:31.685 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './158' 2019-12-05T05:08:31.685 INFO:tasks.workunit.client.0.smithi042.stdout:removed './159/19' 2019-12-05T05:08:31.685 INFO:tasks.workunit.client.0.smithi042.stdout:removed './159/15' 2019-12-05T05:08:31.686 INFO:tasks.workunit.client.0.smithi042.stdout:removed './159/31' 2019-12-05T05:08:31.686 INFO:tasks.workunit.client.0.smithi042.stdout:removed './159/36' 2019-12-05T05:08:31.686 INFO:tasks.workunit.client.0.smithi042.stdout:removed './159/20' 2019-12-05T05:08:31.686 INFO:tasks.workunit.client.0.smithi042.stdout:removed './159/12' 2019-12-05T05:08:31.686 INFO:tasks.workunit.client.0.smithi042.stdout:removed './159/39' 2019-12-05T05:08:31.686 INFO:tasks.workunit.client.0.smithi042.stdout:removed './159/10' 2019-12-05T05:08:31.686 INFO:tasks.workunit.client.0.smithi042.stdout:removed './159/34' 2019-12-05T05:08:31.687 INFO:tasks.workunit.client.0.smithi042.stdout:removed './159/38' 2019-12-05T05:08:31.687 INFO:tasks.workunit.client.0.smithi042.stdout:removed './159/11' 2019-12-05T05:08:31.687 INFO:tasks.workunit.client.0.smithi042.stdout:removed './159/16' 2019-12-05T05:08:31.687 INFO:tasks.workunit.client.0.smithi042.stdout:removed './159/32' 2019-12-05T05:08:31.687 INFO:tasks.workunit.client.0.smithi042.stdout:removed './159/30' 2019-12-05T05:08:31.687 INFO:tasks.workunit.client.0.smithi042.stdout:removed './159/14' 2019-12-05T05:08:31.688 INFO:tasks.workunit.client.0.smithi042.stdout:removed './159/18' 2019-12-05T05:08:31.688 INFO:tasks.workunit.client.0.smithi042.stdout:removed './159/13' 2019-12-05T05:08:31.688 INFO:tasks.workunit.client.0.smithi042.stdout:removed './159/33' 2019-12-05T05:08:31.688 INFO:tasks.workunit.client.0.smithi042.stdout:removed './159/17' 2019-12-05T05:08:31.688 INFO:tasks.workunit.client.0.smithi042.stdout:removed './159/35' 2019-12-05T05:08:31.688 INFO:tasks.workunit.client.0.smithi042.stdout:removed './159/37' 2019-12-05T05:08:31.688 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './159' 2019-12-05T05:08:31.689 INFO:tasks.workunit.client.0.smithi042.stdout:removed './160/19' 2019-12-05T05:08:31.689 INFO:tasks.workunit.client.0.smithi042.stdout:removed './160/15' 2019-12-05T05:08:31.689 INFO:tasks.workunit.client.0.smithi042.stdout:removed './160/31' 2019-12-05T05:08:31.689 INFO:tasks.workunit.client.0.smithi042.stdout:removed './160/36' 2019-12-05T05:08:31.689 INFO:tasks.workunit.client.0.smithi042.stdout:removed './160/20' 2019-12-05T05:08:31.689 INFO:tasks.workunit.client.0.smithi042.stdout:removed './160/12' 2019-12-05T05:08:31.689 INFO:tasks.workunit.client.0.smithi042.stdout:removed './160/39' 2019-12-05T05:08:31.690 INFO:tasks.workunit.client.0.smithi042.stdout:removed './160/10' 2019-12-05T05:08:31.690 INFO:tasks.workunit.client.0.smithi042.stdout:removed './160/34' 2019-12-05T05:08:31.690 INFO:tasks.workunit.client.0.smithi042.stdout:removed './160/38' 2019-12-05T05:08:31.690 INFO:tasks.workunit.client.0.smithi042.stdout:removed './160/11' 2019-12-05T05:08:31.690 INFO:tasks.workunit.client.0.smithi042.stdout:removed './160/16' 2019-12-05T05:08:31.690 INFO:tasks.workunit.client.0.smithi042.stdout:removed './160/32' 2019-12-05T05:08:31.690 INFO:tasks.workunit.client.0.smithi042.stdout:removed './160/30' 2019-12-05T05:08:31.691 INFO:tasks.workunit.client.0.smithi042.stdout:removed './160/14' 2019-12-05T05:08:31.691 INFO:tasks.workunit.client.0.smithi042.stdout:removed './160/18' 2019-12-05T05:08:31.691 INFO:tasks.workunit.client.0.smithi042.stdout:removed './160/13' 2019-12-05T05:08:31.691 INFO:tasks.workunit.client.0.smithi042.stdout:removed './160/33' 2019-12-05T05:08:31.691 INFO:tasks.workunit.client.0.smithi042.stdout:removed './160/17' 2019-12-05T05:08:31.691 INFO:tasks.workunit.client.0.smithi042.stdout:removed './160/35' 2019-12-05T05:08:31.691 INFO:tasks.workunit.client.0.smithi042.stdout:removed './160/37' 2019-12-05T05:08:31.692 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './160' 2019-12-05T05:08:31.692 INFO:tasks.workunit.client.0.smithi042.stdout:removed './161/19' 2019-12-05T05:08:31.692 INFO:tasks.workunit.client.0.smithi042.stdout:removed './161/15' 2019-12-05T05:08:31.692 INFO:tasks.workunit.client.0.smithi042.stdout:removed './161/31' 2019-12-05T05:08:31.692 INFO:tasks.workunit.client.0.smithi042.stdout:removed './161/36' 2019-12-05T05:08:31.692 INFO:tasks.workunit.client.0.smithi042.stdout:removed './161/20' 2019-12-05T05:08:31.692 INFO:tasks.workunit.client.0.smithi042.stdout:removed './161/12' 2019-12-05T05:08:31.693 INFO:tasks.workunit.client.0.smithi042.stdout:removed './161/39' 2019-12-05T05:08:31.693 INFO:tasks.workunit.client.0.smithi042.stdout:removed './161/10' 2019-12-05T05:08:31.693 INFO:tasks.workunit.client.0.smithi042.stdout:removed './161/34' 2019-12-05T05:08:31.693 INFO:tasks.workunit.client.0.smithi042.stdout:removed './161/38' 2019-12-05T05:08:31.693 INFO:tasks.workunit.client.0.smithi042.stdout:removed './161/11' 2019-12-05T05:08:31.693 INFO:tasks.workunit.client.0.smithi042.stdout:removed './161/16' 2019-12-05T05:08:31.693 INFO:tasks.workunit.client.0.smithi042.stdout:removed './161/32' 2019-12-05T05:08:31.694 INFO:tasks.workunit.client.0.smithi042.stdout:removed './161/30' 2019-12-05T05:08:31.694 INFO:tasks.workunit.client.0.smithi042.stdout:removed './161/14' 2019-12-05T05:08:31.694 INFO:tasks.workunit.client.0.smithi042.stdout:removed './161/18' 2019-12-05T05:08:31.694 INFO:tasks.workunit.client.0.smithi042.stdout:removed './161/13' 2019-12-05T05:08:31.694 INFO:tasks.workunit.client.0.smithi042.stdout:removed './161/33' 2019-12-05T05:08:31.694 INFO:tasks.workunit.client.0.smithi042.stdout:removed './161/17' 2019-12-05T05:08:31.695 INFO:tasks.workunit.client.0.smithi042.stdout:removed './161/35' 2019-12-05T05:08:31.695 INFO:tasks.workunit.client.0.smithi042.stdout:removed './161/37' 2019-12-05T05:08:31.695 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './161' 2019-12-05T05:08:31.695 INFO:tasks.workunit.client.0.smithi042.stdout:removed './162/19' 2019-12-05T05:08:31.695 INFO:tasks.workunit.client.0.smithi042.stdout:removed './162/15' 2019-12-05T05:08:31.695 INFO:tasks.workunit.client.0.smithi042.stdout:removed './162/31' 2019-12-05T05:08:31.695 INFO:tasks.workunit.client.0.smithi042.stdout:removed './162/36' 2019-12-05T05:08:31.695 INFO:tasks.workunit.client.0.smithi042.stdout:removed './162/20' 2019-12-05T05:08:31.696 INFO:tasks.workunit.client.0.smithi042.stdout:removed './162/12' 2019-12-05T05:08:31.696 INFO:tasks.workunit.client.0.smithi042.stdout:removed './162/39' 2019-12-05T05:08:31.696 INFO:tasks.workunit.client.0.smithi042.stdout:removed './162/10' 2019-12-05T05:08:31.696 INFO:tasks.workunit.client.0.smithi042.stdout:removed './162/34' 2019-12-05T05:08:31.696 INFO:tasks.workunit.client.0.smithi042.stdout:removed './162/38' 2019-12-05T05:08:31.696 INFO:tasks.workunit.client.0.smithi042.stdout:removed './162/11' 2019-12-05T05:08:31.696 INFO:tasks.workunit.client.0.smithi042.stdout:removed './162/16' 2019-12-05T05:08:31.697 INFO:tasks.workunit.client.0.smithi042.stdout:removed './162/32' 2019-12-05T05:08:31.697 INFO:tasks.workunit.client.0.smithi042.stdout:removed './162/30' 2019-12-05T05:08:31.697 INFO:tasks.workunit.client.0.smithi042.stdout:removed './162/14' 2019-12-05T05:08:31.698 INFO:tasks.workunit.client.0.smithi042.stdout:removed './162/18' 2019-12-05T05:08:31.698 INFO:tasks.workunit.client.0.smithi042.stdout:removed './162/13' 2019-12-05T05:08:31.698 INFO:tasks.workunit.client.0.smithi042.stdout:removed './162/33' 2019-12-05T05:08:31.698 INFO:tasks.workunit.client.0.smithi042.stdout:removed './162/17' 2019-12-05T05:08:31.698 INFO:tasks.workunit.client.0.smithi042.stdout:removed './162/35' 2019-12-05T05:08:31.698 INFO:tasks.workunit.client.0.smithi042.stdout:removed './162/37' 2019-12-05T05:08:31.698 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './162' 2019-12-05T05:08:31.699 INFO:tasks.workunit.client.0.smithi042.stdout:removed './163/19' 2019-12-05T05:08:31.699 INFO:tasks.workunit.client.0.smithi042.stdout:removed './163/15' 2019-12-05T05:08:31.699 INFO:tasks.workunit.client.0.smithi042.stdout:removed './163/31' 2019-12-05T05:08:31.699 INFO:tasks.workunit.client.0.smithi042.stdout:removed './163/36' 2019-12-05T05:08:31.699 INFO:tasks.workunit.client.0.smithi042.stdout:removed './163/20' 2019-12-05T05:08:31.699 INFO:tasks.workunit.client.0.smithi042.stdout:removed './163/12' 2019-12-05T05:08:31.699 INFO:tasks.workunit.client.0.smithi042.stdout:removed './163/39' 2019-12-05T05:08:31.699 INFO:tasks.workunit.client.0.smithi042.stdout:removed './163/10' 2019-12-05T05:08:31.700 INFO:tasks.workunit.client.0.smithi042.stdout:removed './163/34' 2019-12-05T05:08:31.700 INFO:tasks.workunit.client.0.smithi042.stdout:removed './163/38' 2019-12-05T05:08:31.700 INFO:tasks.workunit.client.0.smithi042.stdout:removed './163/11' 2019-12-05T05:08:31.700 INFO:tasks.workunit.client.0.smithi042.stdout:removed './163/16' 2019-12-05T05:08:31.700 INFO:tasks.workunit.client.0.smithi042.stdout:removed './163/32' 2019-12-05T05:08:31.700 INFO:tasks.workunit.client.0.smithi042.stdout:removed './163/30' 2019-12-05T05:08:31.700 INFO:tasks.workunit.client.0.smithi042.stdout:removed './163/14' 2019-12-05T05:08:31.701 INFO:tasks.workunit.client.0.smithi042.stdout:removed './163/18' 2019-12-05T05:08:31.701 INFO:tasks.workunit.client.0.smithi042.stdout:removed './163/13' 2019-12-05T05:08:31.701 INFO:tasks.workunit.client.0.smithi042.stdout:removed './163/33' 2019-12-05T05:08:31.701 INFO:tasks.workunit.client.0.smithi042.stdout:removed './163/17' 2019-12-05T05:08:31.701 INFO:tasks.workunit.client.0.smithi042.stdout:removed './163/35' 2019-12-05T05:08:31.701 INFO:tasks.workunit.client.0.smithi042.stdout:removed './163/37' 2019-12-05T05:08:31.702 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './163' 2019-12-05T05:08:31.702 INFO:tasks.workunit.client.0.smithi042.stdout:removed './164/19' 2019-12-05T05:08:31.702 INFO:tasks.workunit.client.0.smithi042.stdout:removed './164/15' 2019-12-05T05:08:31.702 INFO:tasks.workunit.client.0.smithi042.stdout:removed './164/31' 2019-12-05T05:08:31.702 INFO:tasks.workunit.client.0.smithi042.stdout:removed './164/36' 2019-12-05T05:08:31.702 INFO:tasks.workunit.client.0.smithi042.stdout:removed './164/20' 2019-12-05T05:08:31.702 INFO:tasks.workunit.client.0.smithi042.stdout:removed './164/12' 2019-12-05T05:08:31.702 INFO:tasks.workunit.client.0.smithi042.stdout:removed './164/39' 2019-12-05T05:08:31.703 INFO:tasks.workunit.client.0.smithi042.stdout:removed './164/10' 2019-12-05T05:08:31.703 INFO:tasks.workunit.client.0.smithi042.stdout:removed './164/34' 2019-12-05T05:08:31.703 INFO:tasks.workunit.client.0.smithi042.stdout:removed './164/38' 2019-12-05T05:08:31.703 INFO:tasks.workunit.client.0.smithi042.stdout:removed './164/11' 2019-12-05T05:08:31.703 INFO:tasks.workunit.client.0.smithi042.stdout:removed './164/16' 2019-12-05T05:08:31.703 INFO:tasks.workunit.client.0.smithi042.stdout:removed './164/32' 2019-12-05T05:08:31.703 INFO:tasks.workunit.client.0.smithi042.stdout:removed './164/30' 2019-12-05T05:08:31.704 INFO:tasks.workunit.client.0.smithi042.stdout:removed './164/14' 2019-12-05T05:08:31.704 INFO:tasks.workunit.client.0.smithi042.stdout:removed './164/18' 2019-12-05T05:08:31.704 INFO:tasks.workunit.client.0.smithi042.stdout:removed './164/13' 2019-12-05T05:08:31.704 INFO:tasks.workunit.client.0.smithi042.stdout:removed './164/33' 2019-12-05T05:08:31.704 INFO:tasks.workunit.client.0.smithi042.stdout:removed './164/17' 2019-12-05T05:08:31.704 INFO:tasks.workunit.client.0.smithi042.stdout:removed './164/35' 2019-12-05T05:08:31.704 INFO:tasks.workunit.client.0.smithi042.stdout:removed './164/37' 2019-12-05T05:08:31.704 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './164' 2019-12-05T05:08:31.705 INFO:tasks.workunit.client.0.smithi042.stdout:removed './165/19' 2019-12-05T05:08:31.705 INFO:tasks.workunit.client.0.smithi042.stdout:removed './165/15' 2019-12-05T05:08:31.705 INFO:tasks.workunit.client.0.smithi042.stdout:removed './165/31' 2019-12-05T05:08:31.705 INFO:tasks.workunit.client.0.smithi042.stdout:removed './165/36' 2019-12-05T05:08:31.705 INFO:tasks.workunit.client.0.smithi042.stdout:removed './165/20' 2019-12-05T05:08:31.705 INFO:tasks.workunit.client.0.smithi042.stdout:removed './165/12' 2019-12-05T05:08:31.705 INFO:tasks.workunit.client.0.smithi042.stdout:removed './165/39' 2019-12-05T05:08:31.706 INFO:tasks.workunit.client.0.smithi042.stdout:removed './165/10' 2019-12-05T05:08:31.706 INFO:tasks.workunit.client.0.smithi042.stdout:removed './165/34' 2019-12-05T05:08:31.706 INFO:tasks.workunit.client.0.smithi042.stdout:removed './165/38' 2019-12-05T05:08:31.706 INFO:tasks.workunit.client.0.smithi042.stdout:removed './165/11' 2019-12-05T05:08:31.706 INFO:tasks.workunit.client.0.smithi042.stdout:removed './165/16' 2019-12-05T05:08:31.706 INFO:tasks.workunit.client.0.smithi042.stdout:removed './165/32' 2019-12-05T05:08:31.706 INFO:tasks.workunit.client.0.smithi042.stdout:removed './165/30' 2019-12-05T05:08:31.706 INFO:tasks.workunit.client.0.smithi042.stdout:removed './165/14' 2019-12-05T05:08:31.707 INFO:tasks.workunit.client.0.smithi042.stdout:removed './165/18' 2019-12-05T05:08:31.707 INFO:tasks.workunit.client.0.smithi042.stdout:removed './165/13' 2019-12-05T05:08:31.707 INFO:tasks.workunit.client.0.smithi042.stdout:removed './165/33' 2019-12-05T05:08:31.707 INFO:tasks.workunit.client.0.smithi042.stdout:removed './165/17' 2019-12-05T05:08:31.707 INFO:tasks.workunit.client.0.smithi042.stdout:removed './165/35' 2019-12-05T05:08:31.707 INFO:tasks.workunit.client.0.smithi042.stdout:removed './165/37' 2019-12-05T05:08:31.707 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './165' 2019-12-05T05:08:31.708 INFO:tasks.workunit.client.0.smithi042.stdout:removed './166/19' 2019-12-05T05:08:31.708 INFO:tasks.workunit.client.0.smithi042.stdout:removed './166/15' 2019-12-05T05:08:31.708 INFO:tasks.workunit.client.0.smithi042.stdout:removed './166/31' 2019-12-05T05:08:31.708 INFO:tasks.workunit.client.0.smithi042.stdout:removed './166/36' 2019-12-05T05:08:31.708 INFO:tasks.workunit.client.0.smithi042.stdout:removed './166/20' 2019-12-05T05:08:31.708 INFO:tasks.workunit.client.0.smithi042.stdout:removed './166/12' 2019-12-05T05:08:31.708 INFO:tasks.workunit.client.0.smithi042.stdout:removed './166/39' 2019-12-05T05:08:31.708 INFO:tasks.workunit.client.0.smithi042.stdout:removed './166/10' 2019-12-05T05:08:31.709 INFO:tasks.workunit.client.0.smithi042.stdout:removed './166/34' 2019-12-05T05:08:31.709 INFO:tasks.workunit.client.0.smithi042.stdout:removed './166/38' 2019-12-05T05:08:31.709 INFO:tasks.workunit.client.0.smithi042.stdout:removed './166/11' 2019-12-05T05:08:31.709 INFO:tasks.workunit.client.0.smithi042.stdout:removed './166/16' 2019-12-05T05:08:31.709 INFO:tasks.workunit.client.0.smithi042.stdout:removed './166/32' 2019-12-05T05:08:31.709 INFO:tasks.workunit.client.0.smithi042.stdout:removed './166/30' 2019-12-05T05:08:31.710 INFO:tasks.workunit.client.0.smithi042.stdout:removed './166/14' 2019-12-05T05:08:31.710 INFO:tasks.workunit.client.0.smithi042.stdout:removed './166/18' 2019-12-05T05:08:31.710 INFO:tasks.workunit.client.0.smithi042.stdout:removed './166/13' 2019-12-05T05:08:31.710 INFO:tasks.workunit.client.0.smithi042.stdout:removed './166/33' 2019-12-05T05:08:31.710 INFO:tasks.workunit.client.0.smithi042.stdout:removed './166/17' 2019-12-05T05:08:31.710 INFO:tasks.workunit.client.0.smithi042.stdout:removed './166/35' 2019-12-05T05:08:31.710 INFO:tasks.workunit.client.0.smithi042.stdout:removed './166/37' 2019-12-05T05:08:31.710 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './166' 2019-12-05T05:08:31.711 INFO:tasks.workunit.client.0.smithi042.stdout:removed './167/19' 2019-12-05T05:08:31.711 INFO:tasks.workunit.client.0.smithi042.stdout:removed './167/15' 2019-12-05T05:08:31.711 INFO:tasks.workunit.client.0.smithi042.stdout:removed './167/31' 2019-12-05T05:08:31.711 INFO:tasks.workunit.client.0.smithi042.stdout:removed './167/36' 2019-12-05T05:08:31.711 INFO:tasks.workunit.client.0.smithi042.stdout:removed './167/20' 2019-12-05T05:08:31.711 INFO:tasks.workunit.client.0.smithi042.stdout:removed './167/12' 2019-12-05T05:08:31.711 INFO:tasks.workunit.client.0.smithi042.stdout:removed './167/39' 2019-12-05T05:08:31.712 INFO:tasks.workunit.client.0.smithi042.stdout:removed './167/10' 2019-12-05T05:08:31.712 INFO:tasks.workunit.client.0.smithi042.stdout:removed './167/34' 2019-12-05T05:08:31.712 INFO:tasks.workunit.client.0.smithi042.stdout:removed './167/38' 2019-12-05T05:08:34.732 INFO:tasks.workunit.client.0.smithi042.stdout:removed './167/11' 2019-12-05T05:08:34.732 INFO:tasks.workunit.client.0.smithi042.stdout:removed './167/16' 2019-12-05T05:08:34.732 INFO:tasks.workunit.client.0.smithi042.stdout:removed './167/32' 2019-12-05T05:08:34.732 INFO:tasks.workunit.client.0.smithi042.stdout:removed './167/30' 2019-12-05T05:08:34.732 INFO:tasks.workunit.client.0.smithi042.stdout:removed './167/14' 2019-12-05T05:08:34.733 INFO:tasks.workunit.client.0.smithi042.stdout:removed './167/18' 2019-12-05T05:08:34.733 INFO:tasks.workunit.client.0.smithi042.stdout:removed './167/13' 2019-12-05T05:08:34.744 INFO:tasks.workunit.client.0.smithi042.stdout:removed './167/33' 2019-12-05T05:08:34.745 INFO:tasks.workunit.client.0.smithi042.stdout:removed './167/17' 2019-12-05T05:08:34.745 INFO:tasks.workunit.client.0.smithi042.stdout:removed './167/35' 2019-12-05T05:08:34.745 INFO:tasks.workunit.client.0.smithi042.stdout:removed './167/37' 2019-12-05T05:08:34.745 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './167' 2019-12-05T05:08:34.746 INFO:tasks.workunit.client.0.smithi042.stdout:removed './168/19' 2019-12-05T05:08:34.747 INFO:tasks.workunit.client.0.smithi042.stdout:removed './168/15' 2019-12-05T05:08:34.747 INFO:tasks.workunit.client.0.smithi042.stdout:removed './168/31' 2019-12-05T05:08:34.747 INFO:tasks.workunit.client.0.smithi042.stdout:removed './168/36' 2019-12-05T05:08:34.747 INFO:tasks.workunit.client.0.smithi042.stdout:removed './168/20' 2019-12-05T05:08:34.747 INFO:tasks.workunit.client.0.smithi042.stdout:removed './168/12' 2019-12-05T05:08:34.747 INFO:tasks.workunit.client.0.smithi042.stdout:removed './168/39' 2019-12-05T05:08:34.748 INFO:tasks.workunit.client.0.smithi042.stdout:removed './168/10' 2019-12-05T05:08:34.748 INFO:tasks.workunit.client.0.smithi042.stdout:removed './168/34' 2019-12-05T05:08:34.748 INFO:tasks.workunit.client.0.smithi042.stdout:removed './168/38' 2019-12-05T05:08:34.748 INFO:tasks.workunit.client.0.smithi042.stdout:removed './168/11' 2019-12-05T05:08:34.748 INFO:tasks.workunit.client.0.smithi042.stdout:removed './168/16' 2019-12-05T05:08:34.748 INFO:tasks.workunit.client.0.smithi042.stdout:removed './168/32' 2019-12-05T05:08:34.748 INFO:tasks.workunit.client.0.smithi042.stdout:removed './168/30' 2019-12-05T05:08:34.748 INFO:tasks.workunit.client.0.smithi042.stdout:removed './168/14' 2019-12-05T05:08:34.749 INFO:tasks.workunit.client.0.smithi042.stdout:removed './168/18' 2019-12-05T05:08:34.749 INFO:tasks.workunit.client.0.smithi042.stdout:removed './168/13' 2019-12-05T05:08:34.749 INFO:tasks.workunit.client.0.smithi042.stdout:removed './168/33' 2019-12-05T05:08:34.749 INFO:tasks.workunit.client.0.smithi042.stdout:removed './168/17' 2019-12-05T05:08:34.749 INFO:tasks.workunit.client.0.smithi042.stdout:removed './168/35' 2019-12-05T05:08:34.749 INFO:tasks.workunit.client.0.smithi042.stdout:removed './168/37' 2019-12-05T05:08:34.749 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './168' 2019-12-05T05:08:34.750 INFO:tasks.workunit.client.0.smithi042.stdout:removed './169/19' 2019-12-05T05:08:34.750 INFO:tasks.workunit.client.0.smithi042.stdout:removed './169/15' 2019-12-05T05:08:34.750 INFO:tasks.workunit.client.0.smithi042.stdout:removed './169/31' 2019-12-05T05:08:34.750 INFO:tasks.workunit.client.0.smithi042.stdout:removed './169/36' 2019-12-05T05:08:34.750 INFO:tasks.workunit.client.0.smithi042.stdout:removed './169/20' 2019-12-05T05:08:34.750 INFO:tasks.workunit.client.0.smithi042.stdout:removed './169/12' 2019-12-05T05:08:34.750 INFO:tasks.workunit.client.0.smithi042.stdout:removed './169/39' 2019-12-05T05:08:34.751 INFO:tasks.workunit.client.0.smithi042.stdout:removed './169/10' 2019-12-05T05:08:34.751 INFO:tasks.workunit.client.0.smithi042.stdout:removed './169/34' 2019-12-05T05:08:34.751 INFO:tasks.workunit.client.0.smithi042.stdout:removed './169/38' 2019-12-05T05:08:34.751 INFO:tasks.workunit.client.0.smithi042.stdout:removed './169/11' 2019-12-05T05:08:34.751 INFO:tasks.workunit.client.0.smithi042.stdout:removed './169/16' 2019-12-05T05:08:34.751 INFO:tasks.workunit.client.0.smithi042.stdout:removed './169/32' 2019-12-05T05:08:34.762 INFO:tasks.workunit.client.0.smithi042.stdout:removed './169/30' 2019-12-05T05:08:34.762 INFO:tasks.workunit.client.0.smithi042.stdout:removed './169/14' 2019-12-05T05:08:34.762 INFO:tasks.workunit.client.0.smithi042.stdout:removed './169/18' 2019-12-05T05:08:34.762 INFO:tasks.workunit.client.0.smithi042.stdout:removed './169/13' 2019-12-05T05:08:34.762 INFO:tasks.workunit.client.0.smithi042.stdout:removed './169/33' 2019-12-05T05:08:34.762 INFO:tasks.workunit.client.0.smithi042.stdout:removed './169/17' 2019-12-05T05:08:34.763 INFO:tasks.workunit.client.0.smithi042.stdout:removed './169/35' 2019-12-05T05:08:34.763 INFO:tasks.workunit.client.0.smithi042.stdout:removed './169/37' 2019-12-05T05:08:34.763 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './169' 2019-12-05T05:08:34.763 INFO:tasks.workunit.client.0.smithi042.stdout:removed './170/19' 2019-12-05T05:08:34.763 INFO:tasks.workunit.client.0.smithi042.stdout:removed './170/15' 2019-12-05T05:08:34.763 INFO:tasks.workunit.client.0.smithi042.stdout:removed './170/31' 2019-12-05T05:08:34.763 INFO:tasks.workunit.client.0.smithi042.stdout:removed './170/36' 2019-12-05T05:08:34.763 INFO:tasks.workunit.client.0.smithi042.stdout:removed './170/20' 2019-12-05T05:08:34.764 INFO:tasks.workunit.client.0.smithi042.stdout:removed './170/12' 2019-12-05T05:08:34.764 INFO:tasks.workunit.client.0.smithi042.stdout:removed './170/39' 2019-12-05T05:08:34.764 INFO:tasks.workunit.client.0.smithi042.stdout:removed './170/10' 2019-12-05T05:08:34.764 INFO:tasks.workunit.client.0.smithi042.stdout:removed './170/34' 2019-12-05T05:08:34.765 INFO:tasks.workunit.client.0.smithi042.stdout:removed './170/38' 2019-12-05T05:08:34.765 INFO:tasks.workunit.client.0.smithi042.stdout:removed './170/11' 2019-12-05T05:08:34.765 INFO:tasks.workunit.client.0.smithi042.stdout:removed './170/16' 2019-12-05T05:08:34.765 INFO:tasks.workunit.client.0.smithi042.stdout:removed './170/32' 2019-12-05T05:08:34.765 INFO:tasks.workunit.client.0.smithi042.stdout:removed './170/30' 2019-12-05T05:08:34.765 INFO:tasks.workunit.client.0.smithi042.stdout:removed './170/14' 2019-12-05T05:08:34.766 INFO:tasks.workunit.client.0.smithi042.stdout:removed './170/18' 2019-12-05T05:08:34.766 INFO:tasks.workunit.client.0.smithi042.stdout:removed './170/13' 2019-12-05T05:08:34.766 INFO:tasks.workunit.client.0.smithi042.stdout:removed './170/33' 2019-12-05T05:08:34.766 INFO:tasks.workunit.client.0.smithi042.stdout:removed './170/17' 2019-12-05T05:08:34.766 INFO:tasks.workunit.client.0.smithi042.stdout:removed './170/35' 2019-12-05T05:08:34.766 INFO:tasks.workunit.client.0.smithi042.stdout:removed './170/37' 2019-12-05T05:08:34.766 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './170' 2019-12-05T05:08:34.767 INFO:tasks.workunit.client.0.smithi042.stdout:removed './171/19' 2019-12-05T05:08:34.767 INFO:tasks.workunit.client.0.smithi042.stdout:removed './171/15' 2019-12-05T05:08:34.767 INFO:tasks.workunit.client.0.smithi042.stdout:removed './171/31' 2019-12-05T05:08:34.767 INFO:tasks.workunit.client.0.smithi042.stdout:removed './171/36' 2019-12-05T05:08:34.767 INFO:tasks.workunit.client.0.smithi042.stdout:removed './171/20' 2019-12-05T05:08:34.767 INFO:tasks.workunit.client.0.smithi042.stdout:removed './171/12' 2019-12-05T05:08:34.769 INFO:tasks.workunit.client.0.smithi042.stdout:removed './171/39' 2019-12-05T05:08:34.771 INFO:tasks.workunit.client.0.smithi042.stdout:removed './171/10' 2019-12-05T05:08:34.771 INFO:tasks.workunit.client.0.smithi042.stdout:removed './171/34' 2019-12-05T05:08:34.772 INFO:tasks.workunit.client.0.smithi042.stdout:removed './171/38' 2019-12-05T05:08:34.772 INFO:tasks.workunit.client.0.smithi042.stdout:removed './171/11' 2019-12-05T05:08:34.772 INFO:tasks.workunit.client.0.smithi042.stdout:removed './171/16' 2019-12-05T05:08:34.772 INFO:tasks.workunit.client.0.smithi042.stdout:removed './171/32' 2019-12-05T05:08:34.773 INFO:tasks.workunit.client.0.smithi042.stdout:removed './171/30' 2019-12-05T05:08:34.774 INFO:tasks.workunit.client.0.smithi042.stdout:removed './171/14' 2019-12-05T05:08:34.774 INFO:tasks.workunit.client.0.smithi042.stdout:removed './171/18' 2019-12-05T05:08:34.774 INFO:tasks.workunit.client.0.smithi042.stdout:removed './171/13' 2019-12-05T05:08:34.774 INFO:tasks.workunit.client.0.smithi042.stdout:removed './171/33' 2019-12-05T05:08:34.775 INFO:tasks.workunit.client.0.smithi042.stdout:removed './171/17' 2019-12-05T05:08:34.775 INFO:tasks.workunit.client.0.smithi042.stdout:removed './171/35' 2019-12-05T05:08:34.775 INFO:tasks.workunit.client.0.smithi042.stdout:removed './171/37' 2019-12-05T05:08:34.775 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './171' 2019-12-05T05:08:34.775 INFO:tasks.workunit.client.0.smithi042.stdout:removed './172/19' 2019-12-05T05:08:34.775 INFO:tasks.workunit.client.0.smithi042.stdout:removed './172/15' 2019-12-05T05:08:34.776 INFO:tasks.workunit.client.0.smithi042.stdout:removed './172/31' 2019-12-05T05:08:34.776 INFO:tasks.workunit.client.0.smithi042.stdout:removed './172/36' 2019-12-05T05:08:34.776 INFO:tasks.workunit.client.0.smithi042.stdout:removed './172/20' 2019-12-05T05:08:34.776 INFO:tasks.workunit.client.0.smithi042.stdout:removed './172/12' 2019-12-05T05:08:34.776 INFO:tasks.workunit.client.0.smithi042.stdout:removed './172/39' 2019-12-05T05:08:34.777 INFO:tasks.workunit.client.0.smithi042.stdout:removed './172/10' 2019-12-05T05:08:34.777 INFO:tasks.workunit.client.0.smithi042.stdout:removed './172/34' 2019-12-05T05:08:34.777 INFO:tasks.workunit.client.0.smithi042.stdout:removed './172/38' 2019-12-05T05:08:34.777 INFO:tasks.workunit.client.0.smithi042.stdout:removed './172/11' 2019-12-05T05:08:34.777 INFO:tasks.workunit.client.0.smithi042.stdout:removed './172/16' 2019-12-05T05:08:34.778 INFO:tasks.workunit.client.0.smithi042.stdout:removed './172/32' 2019-12-05T05:08:34.778 INFO:tasks.workunit.client.0.smithi042.stdout:removed './172/30' 2019-12-05T05:08:34.778 INFO:tasks.workunit.client.0.smithi042.stdout:removed './172/14' 2019-12-05T05:08:34.778 INFO:tasks.workunit.client.0.smithi042.stdout:removed './172/18' 2019-12-05T05:08:34.779 INFO:tasks.workunit.client.0.smithi042.stdout:removed './172/13' 2019-12-05T05:08:34.779 INFO:tasks.workunit.client.0.smithi042.stdout:removed './172/33' 2019-12-05T05:08:34.779 INFO:tasks.workunit.client.0.smithi042.stdout:removed './172/17' 2019-12-05T05:08:34.779 INFO:tasks.workunit.client.0.smithi042.stdout:removed './172/35' 2019-12-05T05:08:34.779 INFO:tasks.workunit.client.0.smithi042.stdout:removed './172/37' 2019-12-05T05:08:34.779 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './172' 2019-12-05T05:08:34.780 INFO:tasks.workunit.client.0.smithi042.stdout:removed './173/19' 2019-12-05T05:08:34.780 INFO:tasks.workunit.client.0.smithi042.stdout:removed './173/15' 2019-12-05T05:08:34.780 INFO:tasks.workunit.client.0.smithi042.stdout:removed './173/31' 2019-12-05T05:08:34.780 INFO:tasks.workunit.client.0.smithi042.stdout:removed './173/36' 2019-12-05T05:08:34.780 INFO:tasks.workunit.client.0.smithi042.stdout:removed './173/20' 2019-12-05T05:08:34.780 INFO:tasks.workunit.client.0.smithi042.stdout:removed './173/12' 2019-12-05T05:08:34.780 INFO:tasks.workunit.client.0.smithi042.stdout:removed './173/39' 2019-12-05T05:08:34.781 INFO:tasks.workunit.client.0.smithi042.stdout:removed './173/10' 2019-12-05T05:08:34.781 INFO:tasks.workunit.client.0.smithi042.stdout:removed './173/34' 2019-12-05T05:08:34.781 INFO:tasks.workunit.client.0.smithi042.stdout:removed './173/38' 2019-12-05T05:08:34.783 INFO:tasks.workunit.client.0.smithi042.stdout:removed './173/11' 2019-12-05T05:08:34.783 INFO:tasks.workunit.client.0.smithi042.stdout:removed './173/16' 2019-12-05T05:08:34.783 INFO:tasks.workunit.client.0.smithi042.stdout:removed './173/32' 2019-12-05T05:08:34.783 INFO:tasks.workunit.client.0.smithi042.stdout:removed './173/30' 2019-12-05T05:08:34.784 INFO:tasks.workunit.client.0.smithi042.stdout:removed './173/14' 2019-12-05T05:08:34.784 INFO:tasks.workunit.client.0.smithi042.stdout:removed './173/18' 2019-12-05T05:08:34.784 INFO:tasks.workunit.client.0.smithi042.stdout:removed './173/13' 2019-12-05T05:08:34.784 INFO:tasks.workunit.client.0.smithi042.stdout:removed './173/33' 2019-12-05T05:08:34.785 INFO:tasks.workunit.client.0.smithi042.stdout:removed './173/17' 2019-12-05T05:08:34.785 INFO:tasks.workunit.client.0.smithi042.stdout:removed './173/35' 2019-12-05T05:08:34.785 INFO:tasks.workunit.client.0.smithi042.stdout:removed './173/37' 2019-12-05T05:08:34.785 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './173' 2019-12-05T05:08:34.785 INFO:tasks.workunit.client.0.smithi042.stdout:removed './174/19' 2019-12-05T05:08:34.785 INFO:tasks.workunit.client.0.smithi042.stdout:removed './174/15' 2019-12-05T05:08:34.786 INFO:tasks.workunit.client.0.smithi042.stdout:removed './174/31' 2019-12-05T05:08:34.786 INFO:tasks.workunit.client.0.smithi042.stdout:removed './174/36' 2019-12-05T05:08:34.786 INFO:tasks.workunit.client.0.smithi042.stdout:removed './174/20' 2019-12-05T05:08:34.786 INFO:tasks.workunit.client.0.smithi042.stdout:removed './174/12' 2019-12-05T05:08:34.786 INFO:tasks.workunit.client.0.smithi042.stdout:removed './174/39' 2019-12-05T05:08:34.788 INFO:tasks.workunit.client.0.smithi042.stdout:removed './174/10' 2019-12-05T05:08:34.788 INFO:tasks.workunit.client.0.smithi042.stdout:removed './174/34' 2019-12-05T05:08:34.788 INFO:tasks.workunit.client.0.smithi042.stdout:removed './174/38' 2019-12-05T05:08:34.788 INFO:tasks.workunit.client.0.smithi042.stdout:removed './174/11' 2019-12-05T05:08:34.789 INFO:tasks.workunit.client.0.smithi042.stdout:removed './174/16' 2019-12-05T05:08:34.795 INFO:tasks.workunit.client.0.smithi042.stdout:removed './174/32' 2019-12-05T05:08:34.796 INFO:tasks.workunit.client.0.smithi042.stdout:removed './174/30' 2019-12-05T05:08:34.796 INFO:tasks.workunit.client.0.smithi042.stdout:removed './174/14' 2019-12-05T05:08:34.797 INFO:tasks.workunit.client.0.smithi042.stdout:removed './174/18' 2019-12-05T05:08:34.797 INFO:tasks.workunit.client.0.smithi042.stdout:removed './174/13' 2019-12-05T05:08:34.799 INFO:tasks.workunit.client.0.smithi042.stdout:removed './174/33' 2019-12-05T05:08:34.799 INFO:tasks.workunit.client.0.smithi042.stdout:removed './174/17' 2019-12-05T05:08:34.800 INFO:tasks.workunit.client.0.smithi042.stdout:removed './174/35' 2019-12-05T05:08:34.800 INFO:tasks.workunit.client.0.smithi042.stdout:removed './174/37' 2019-12-05T05:08:34.800 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './174' 2019-12-05T05:08:34.800 INFO:tasks.workunit.client.0.smithi042.stdout:removed './175/19' 2019-12-05T05:08:34.800 INFO:tasks.workunit.client.0.smithi042.stdout:removed './175/15' 2019-12-05T05:08:34.806 INFO:tasks.workunit.client.0.smithi042.stdout:removed './175/31' 2019-12-05T05:08:34.806 INFO:tasks.workunit.client.0.smithi042.stdout:removed './175/36' 2019-12-05T05:08:34.806 INFO:tasks.workunit.client.0.smithi042.stdout:removed './175/20' 2019-12-05T05:08:34.806 INFO:tasks.workunit.client.0.smithi042.stdout:removed './175/12' 2019-12-05T05:08:34.806 INFO:tasks.workunit.client.0.smithi042.stdout:removed './175/39' 2019-12-05T05:08:34.806 INFO:tasks.workunit.client.0.smithi042.stdout:removed './175/10' 2019-12-05T05:08:34.806 INFO:tasks.workunit.client.0.smithi042.stdout:removed './175/34' 2019-12-05T05:08:34.807 INFO:tasks.workunit.client.0.smithi042.stdout:removed './175/38' 2019-12-05T05:08:34.807 INFO:tasks.workunit.client.0.smithi042.stdout:removed './175/11' 2019-12-05T05:08:34.807 INFO:tasks.workunit.client.0.smithi042.stdout:removed './175/16' 2019-12-05T05:08:34.807 INFO:tasks.workunit.client.0.smithi042.stdout:removed './175/32' 2019-12-05T05:08:34.807 INFO:tasks.workunit.client.0.smithi042.stdout:removed './175/30' 2019-12-05T05:08:34.807 INFO:tasks.workunit.client.0.smithi042.stdout:removed './175/14' 2019-12-05T05:08:34.807 INFO:tasks.workunit.client.0.smithi042.stdout:removed './175/18' 2019-12-05T05:08:34.808 INFO:tasks.workunit.client.0.smithi042.stdout:removed './175/13' 2019-12-05T05:08:34.809 INFO:tasks.workunit.client.0.smithi042.stdout:removed './175/33' 2019-12-05T05:08:34.809 INFO:tasks.workunit.client.0.smithi042.stdout:removed './175/17' 2019-12-05T05:08:34.809 INFO:tasks.workunit.client.0.smithi042.stdout:removed './175/35' 2019-12-05T05:08:34.809 INFO:tasks.workunit.client.0.smithi042.stdout:removed './175/37' 2019-12-05T05:08:34.809 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './175' 2019-12-05T05:08:34.809 INFO:tasks.workunit.client.0.smithi042.stdout:removed './176/19' 2019-12-05T05:08:34.809 INFO:tasks.workunit.client.0.smithi042.stdout:removed './176/15' 2019-12-05T05:08:34.810 INFO:tasks.workunit.client.0.smithi042.stdout:removed './176/31' 2019-12-05T05:08:34.810 INFO:tasks.workunit.client.0.smithi042.stdout:removed './176/36' 2019-12-05T05:08:34.810 INFO:tasks.workunit.client.0.smithi042.stdout:removed './176/20' 2019-12-05T05:08:34.810 INFO:tasks.workunit.client.0.smithi042.stdout:removed './176/12' 2019-12-05T05:08:34.810 INFO:tasks.workunit.client.0.smithi042.stdout:removed './176/39' 2019-12-05T05:08:34.810 INFO:tasks.workunit.client.0.smithi042.stdout:removed './176/10' 2019-12-05T05:08:34.810 INFO:tasks.workunit.client.0.smithi042.stdout:removed './176/34' 2019-12-05T05:08:34.810 INFO:tasks.workunit.client.0.smithi042.stdout:removed './176/38' 2019-12-05T05:08:34.811 INFO:tasks.workunit.client.0.smithi042.stdout:removed './176/11' 2019-12-05T05:08:34.811 INFO:tasks.workunit.client.0.smithi042.stdout:removed './176/16' 2019-12-05T05:08:34.811 INFO:tasks.workunit.client.0.smithi042.stdout:removed './176/32' 2019-12-05T05:08:34.811 INFO:tasks.workunit.client.0.smithi042.stdout:removed './176/30' 2019-12-05T05:08:34.811 INFO:tasks.workunit.client.0.smithi042.stdout:removed './176/14' 2019-12-05T05:08:34.811 INFO:tasks.workunit.client.0.smithi042.stdout:removed './176/18' 2019-12-05T05:08:34.811 INFO:tasks.workunit.client.0.smithi042.stdout:removed './176/13' 2019-12-05T05:08:34.811 INFO:tasks.workunit.client.0.smithi042.stdout:removed './176/33' 2019-12-05T05:08:34.812 INFO:tasks.workunit.client.0.smithi042.stdout:removed './176/17' 2019-12-05T05:08:34.812 INFO:tasks.workunit.client.0.smithi042.stdout:removed './176/35' 2019-12-05T05:08:34.812 INFO:tasks.workunit.client.0.smithi042.stdout:removed './176/37' 2019-12-05T05:08:34.812 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './176' 2019-12-05T05:08:34.812 INFO:tasks.workunit.client.0.smithi042.stdout:removed './177/19' 2019-12-05T05:08:34.812 INFO:tasks.workunit.client.0.smithi042.stdout:removed './177/15' 2019-12-05T05:08:34.892 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:08:34.895 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:08:39.660 INFO:tasks.workunit.client.0.smithi042.stdout:removed './177/31' 2019-12-05T05:08:39.660 INFO:tasks.workunit.client.0.smithi042.stdout:removed './177/36' 2019-12-05T05:08:39.660 INFO:tasks.workunit.client.0.smithi042.stdout:removed './177/20' 2019-12-05T05:08:39.661 INFO:tasks.workunit.client.0.smithi042.stdout:removed './177/12' 2019-12-05T05:08:39.661 INFO:tasks.workunit.client.0.smithi042.stdout:removed './177/39' 2019-12-05T05:08:39.661 INFO:tasks.workunit.client.0.smithi042.stdout:removed './177/10' 2019-12-05T05:08:39.661 INFO:tasks.workunit.client.0.smithi042.stdout:removed './177/34' 2019-12-05T05:08:39.661 INFO:tasks.workunit.client.0.smithi042.stdout:removed './177/38' 2019-12-05T05:08:39.661 INFO:tasks.workunit.client.0.smithi042.stdout:removed './177/11' 2019-12-05T05:08:39.661 INFO:tasks.workunit.client.0.smithi042.stdout:removed './177/16' 2019-12-05T05:08:39.662 INFO:tasks.workunit.client.0.smithi042.stdout:removed './177/32' 2019-12-05T05:08:39.662 INFO:tasks.workunit.client.0.smithi042.stdout:removed './177/30' 2019-12-05T05:08:39.662 INFO:tasks.workunit.client.0.smithi042.stdout:removed './177/14' 2019-12-05T05:08:39.662 INFO:tasks.workunit.client.0.smithi042.stdout:removed './177/18' 2019-12-05T05:08:39.662 INFO:tasks.workunit.client.0.smithi042.stdout:removed './177/13' 2019-12-05T05:08:39.662 INFO:tasks.workunit.client.0.smithi042.stdout:removed './177/33' 2019-12-05T05:08:39.662 INFO:tasks.workunit.client.0.smithi042.stdout:removed './177/17' 2019-12-05T05:08:39.662 INFO:tasks.workunit.client.0.smithi042.stdout:removed './177/35' 2019-12-05T05:08:39.663 INFO:tasks.workunit.client.0.smithi042.stdout:removed './177/37' 2019-12-05T05:08:39.663 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './177' 2019-12-05T05:08:39.663 INFO:tasks.workunit.client.0.smithi042.stdout:removed './178/19' 2019-12-05T05:08:39.663 INFO:tasks.workunit.client.0.smithi042.stdout:removed './178/15' 2019-12-05T05:08:39.663 INFO:tasks.workunit.client.0.smithi042.stdout:removed './178/31' 2019-12-05T05:08:39.663 INFO:tasks.workunit.client.0.smithi042.stdout:removed './178/36' 2019-12-05T05:08:39.663 INFO:tasks.workunit.client.0.smithi042.stdout:removed './178/20' 2019-12-05T05:08:39.664 INFO:tasks.workunit.client.0.smithi042.stdout:removed './178/12' 2019-12-05T05:08:39.664 INFO:tasks.workunit.client.0.smithi042.stdout:removed './178/39' 2019-12-05T05:08:39.664 INFO:tasks.workunit.client.0.smithi042.stdout:removed './178/10' 2019-12-05T05:08:39.664 INFO:tasks.workunit.client.0.smithi042.stdout:removed './178/34' 2019-12-05T05:08:39.664 INFO:tasks.workunit.client.0.smithi042.stdout:removed './178/38' 2019-12-05T05:08:39.664 INFO:tasks.workunit.client.0.smithi042.stdout:removed './178/11' 2019-12-05T05:08:39.665 INFO:tasks.workunit.client.0.smithi042.stdout:removed './178/16' 2019-12-05T05:08:39.665 INFO:tasks.workunit.client.0.smithi042.stdout:removed './178/32' 2019-12-05T05:08:39.665 INFO:tasks.workunit.client.0.smithi042.stdout:removed './178/30' 2019-12-05T05:08:39.665 INFO:tasks.workunit.client.0.smithi042.stdout:removed './178/14' 2019-12-05T05:08:39.665 INFO:tasks.workunit.client.0.smithi042.stdout:removed './178/18' 2019-12-05T05:08:39.665 INFO:tasks.workunit.client.0.smithi042.stdout:removed './178/13' 2019-12-05T05:08:39.665 INFO:tasks.workunit.client.0.smithi042.stdout:removed './178/33' 2019-12-05T05:08:39.665 INFO:tasks.workunit.client.0.smithi042.stdout:removed './178/17' 2019-12-05T05:08:39.666 INFO:tasks.workunit.client.0.smithi042.stdout:removed './178/35' 2019-12-05T05:08:39.666 INFO:tasks.workunit.client.0.smithi042.stdout:removed './178/37' 2019-12-05T05:08:39.666 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './178' 2019-12-05T05:08:39.666 INFO:tasks.workunit.client.0.smithi042.stdout:removed './179/19' 2019-12-05T05:08:39.666 INFO:tasks.workunit.client.0.smithi042.stdout:removed './179/15' 2019-12-05T05:08:39.666 INFO:tasks.workunit.client.0.smithi042.stdout:removed './179/31' 2019-12-05T05:08:39.667 INFO:tasks.workunit.client.0.smithi042.stdout:removed './179/36' 2019-12-05T05:08:39.667 INFO:tasks.workunit.client.0.smithi042.stdout:removed './179/20' 2019-12-05T05:08:39.667 INFO:tasks.workunit.client.0.smithi042.stdout:removed './179/12' 2019-12-05T05:08:39.667 INFO:tasks.workunit.client.0.smithi042.stdout:removed './179/39' 2019-12-05T05:08:39.667 INFO:tasks.workunit.client.0.smithi042.stdout:removed './179/10' 2019-12-05T05:08:39.667 INFO:tasks.workunit.client.0.smithi042.stdout:removed './179/34' 2019-12-05T05:08:39.667 INFO:tasks.workunit.client.0.smithi042.stdout:removed './179/38' 2019-12-05T05:08:39.667 INFO:tasks.workunit.client.0.smithi042.stdout:removed './179/11' 2019-12-05T05:08:39.668 INFO:tasks.workunit.client.0.smithi042.stdout:removed './179/16' 2019-12-05T05:08:39.668 INFO:tasks.workunit.client.0.smithi042.stdout:removed './179/32' 2019-12-05T05:08:39.668 INFO:tasks.workunit.client.0.smithi042.stdout:removed './179/30' 2019-12-05T05:08:39.668 INFO:tasks.workunit.client.0.smithi042.stdout:removed './179/14' 2019-12-05T05:08:39.668 INFO:tasks.workunit.client.0.smithi042.stdout:removed './179/18' 2019-12-05T05:08:39.668 INFO:tasks.workunit.client.0.smithi042.stdout:removed './179/13' 2019-12-05T05:08:39.668 INFO:tasks.workunit.client.0.smithi042.stdout:removed './179/33' 2019-12-05T05:08:39.669 INFO:tasks.workunit.client.0.smithi042.stdout:removed './179/17' 2019-12-05T05:08:39.669 INFO:tasks.workunit.client.0.smithi042.stdout:removed './179/35' 2019-12-05T05:08:39.669 INFO:tasks.workunit.client.0.smithi042.stdout:removed './179/37' 2019-12-05T05:08:39.669 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './179' 2019-12-05T05:08:39.669 INFO:tasks.workunit.client.0.smithi042.stdout:removed './180/19' 2019-12-05T05:08:39.669 INFO:tasks.workunit.client.0.smithi042.stdout:removed './180/15' 2019-12-05T05:08:39.669 INFO:tasks.workunit.client.0.smithi042.stdout:removed './180/31' 2019-12-05T05:08:39.670 INFO:tasks.workunit.client.0.smithi042.stdout:removed './180/36' 2019-12-05T05:08:39.670 INFO:tasks.workunit.client.0.smithi042.stdout:removed './180/20' 2019-12-05T05:08:39.670 INFO:tasks.workunit.client.0.smithi042.stdout:removed './180/12' 2019-12-05T05:08:39.670 INFO:tasks.workunit.client.0.smithi042.stdout:removed './180/39' 2019-12-05T05:08:39.670 INFO:tasks.workunit.client.0.smithi042.stdout:removed './180/10' 2019-12-05T05:08:39.670 INFO:tasks.workunit.client.0.smithi042.stdout:removed './180/34' 2019-12-05T05:08:39.670 INFO:tasks.workunit.client.0.smithi042.stdout:removed './180/38' 2019-12-05T05:08:39.670 INFO:tasks.workunit.client.0.smithi042.stdout:removed './180/11' 2019-12-05T05:08:39.671 INFO:tasks.workunit.client.0.smithi042.stdout:removed './180/16' 2019-12-05T05:08:39.671 INFO:tasks.workunit.client.0.smithi042.stdout:removed './180/32' 2019-12-05T05:08:39.671 INFO:tasks.workunit.client.0.smithi042.stdout:removed './180/30' 2019-12-05T05:08:39.671 INFO:tasks.workunit.client.0.smithi042.stdout:removed './180/14' 2019-12-05T05:08:39.671 INFO:tasks.workunit.client.0.smithi042.stdout:removed './180/18' 2019-12-05T05:08:39.671 INFO:tasks.workunit.client.0.smithi042.stdout:removed './180/13' 2019-12-05T05:08:39.671 INFO:tasks.workunit.client.0.smithi042.stdout:removed './180/33' 2019-12-05T05:08:39.672 INFO:tasks.workunit.client.0.smithi042.stdout:removed './180/17' 2019-12-05T05:08:39.672 INFO:tasks.workunit.client.0.smithi042.stdout:removed './180/35' 2019-12-05T05:08:39.672 INFO:tasks.workunit.client.0.smithi042.stdout:removed './180/37' 2019-12-05T05:08:39.672 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './180' 2019-12-05T05:08:39.672 INFO:tasks.workunit.client.0.smithi042.stdout:removed './181/19' 2019-12-05T05:08:39.672 INFO:tasks.workunit.client.0.smithi042.stdout:removed './181/15' 2019-12-05T05:08:39.672 INFO:tasks.workunit.client.0.smithi042.stdout:removed './181/31' 2019-12-05T05:08:39.673 INFO:tasks.workunit.client.0.smithi042.stdout:removed './181/36' 2019-12-05T05:08:39.673 INFO:tasks.workunit.client.0.smithi042.stdout:removed './181/20' 2019-12-05T05:08:39.673 INFO:tasks.workunit.client.0.smithi042.stdout:removed './181/12' 2019-12-05T05:08:39.673 INFO:tasks.workunit.client.0.smithi042.stdout:removed './181/39' 2019-12-05T05:08:39.673 INFO:tasks.workunit.client.0.smithi042.stdout:removed './181/10' 2019-12-05T05:08:39.673 INFO:tasks.workunit.client.0.smithi042.stdout:removed './181/34' 2019-12-05T05:08:39.674 INFO:tasks.workunit.client.0.smithi042.stdout:removed './181/38' 2019-12-05T05:08:39.674 INFO:tasks.workunit.client.0.smithi042.stdout:removed './181/11' 2019-12-05T05:08:39.674 INFO:tasks.workunit.client.0.smithi042.stdout:removed './181/16' 2019-12-05T05:08:39.674 INFO:tasks.workunit.client.0.smithi042.stdout:removed './181/32' 2019-12-05T05:08:39.674 INFO:tasks.workunit.client.0.smithi042.stdout:removed './181/30' 2019-12-05T05:08:39.674 INFO:tasks.workunit.client.0.smithi042.stdout:removed './181/14' 2019-12-05T05:08:39.675 INFO:tasks.workunit.client.0.smithi042.stdout:removed './181/18' 2019-12-05T05:08:39.675 INFO:tasks.workunit.client.0.smithi042.stdout:removed './181/13' 2019-12-05T05:08:39.675 INFO:tasks.workunit.client.0.smithi042.stdout:removed './181/33' 2019-12-05T05:08:39.675 INFO:tasks.workunit.client.0.smithi042.stdout:removed './181/17' 2019-12-05T05:08:39.675 INFO:tasks.workunit.client.0.smithi042.stdout:removed './181/35' 2019-12-05T05:08:39.675 INFO:tasks.workunit.client.0.smithi042.stdout:removed './181/37' 2019-12-05T05:08:39.676 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './181' 2019-12-05T05:08:39.676 INFO:tasks.workunit.client.0.smithi042.stdout:removed './182/19' 2019-12-05T05:08:39.676 INFO:tasks.workunit.client.0.smithi042.stdout:removed './182/15' 2019-12-05T05:08:39.676 INFO:tasks.workunit.client.0.smithi042.stdout:removed './182/31' 2019-12-05T05:08:39.676 INFO:tasks.workunit.client.0.smithi042.stdout:removed './182/36' 2019-12-05T05:08:39.676 INFO:tasks.workunit.client.0.smithi042.stdout:removed './182/20' 2019-12-05T05:08:39.676 INFO:tasks.workunit.client.0.smithi042.stdout:removed './182/12' 2019-12-05T05:08:39.677 INFO:tasks.workunit.client.0.smithi042.stdout:removed './182/39' 2019-12-05T05:08:39.677 INFO:tasks.workunit.client.0.smithi042.stdout:removed './182/10' 2019-12-05T05:08:39.677 INFO:tasks.workunit.client.0.smithi042.stdout:removed './182/34' 2019-12-05T05:08:39.677 INFO:tasks.workunit.client.0.smithi042.stdout:removed './182/38' 2019-12-05T05:08:39.677 INFO:tasks.workunit.client.0.smithi042.stdout:removed './182/11' 2019-12-05T05:08:39.677 INFO:tasks.workunit.client.0.smithi042.stdout:removed './182/16' 2019-12-05T05:08:39.677 INFO:tasks.workunit.client.0.smithi042.stdout:removed './182/32' 2019-12-05T05:08:39.678 INFO:tasks.workunit.client.0.smithi042.stdout:removed './182/30' 2019-12-05T05:08:39.678 INFO:tasks.workunit.client.0.smithi042.stdout:removed './182/14' 2019-12-05T05:08:39.678 INFO:tasks.workunit.client.0.smithi042.stdout:removed './182/18' 2019-12-05T05:08:39.678 INFO:tasks.workunit.client.0.smithi042.stdout:removed './182/13' 2019-12-05T05:08:39.678 INFO:tasks.workunit.client.0.smithi042.stdout:removed './182/33' 2019-12-05T05:08:39.678 INFO:tasks.workunit.client.0.smithi042.stdout:removed './182/17' 2019-12-05T05:08:39.678 INFO:tasks.workunit.client.0.smithi042.stdout:removed './182/35' 2019-12-05T05:08:39.678 INFO:tasks.workunit.client.0.smithi042.stdout:removed './182/37' 2019-12-05T05:08:39.679 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './182' 2019-12-05T05:08:39.679 INFO:tasks.workunit.client.0.smithi042.stdout:removed './183/19' 2019-12-05T05:08:39.679 INFO:tasks.workunit.client.0.smithi042.stdout:removed './183/15' 2019-12-05T05:08:39.679 INFO:tasks.workunit.client.0.smithi042.stdout:removed './183/31' 2019-12-05T05:08:39.679 INFO:tasks.workunit.client.0.smithi042.stdout:removed './183/36' 2019-12-05T05:08:39.679 INFO:tasks.workunit.client.0.smithi042.stdout:removed './183/20' 2019-12-05T05:08:39.679 INFO:tasks.workunit.client.0.smithi042.stdout:removed './183/12' 2019-12-05T05:08:39.680 INFO:tasks.workunit.client.0.smithi042.stdout:removed './183/39' 2019-12-05T05:08:39.680 INFO:tasks.workunit.client.0.smithi042.stdout:removed './183/10' 2019-12-05T05:08:39.680 INFO:tasks.workunit.client.0.smithi042.stdout:removed './183/34' 2019-12-05T05:08:39.680 INFO:tasks.workunit.client.0.smithi042.stdout:removed './183/38' 2019-12-05T05:08:39.680 INFO:tasks.workunit.client.0.smithi042.stdout:removed './183/11' 2019-12-05T05:08:39.680 INFO:tasks.workunit.client.0.smithi042.stdout:removed './183/16' 2019-12-05T05:08:39.680 INFO:tasks.workunit.client.0.smithi042.stdout:removed './183/32' 2019-12-05T05:08:39.681 INFO:tasks.workunit.client.0.smithi042.stdout:removed './183/30' 2019-12-05T05:08:39.681 INFO:tasks.workunit.client.0.smithi042.stdout:removed './183/14' 2019-12-05T05:08:39.681 INFO:tasks.workunit.client.0.smithi042.stdout:removed './183/18' 2019-12-05T05:08:39.681 INFO:tasks.workunit.client.0.smithi042.stdout:removed './183/13' 2019-12-05T05:08:39.681 INFO:tasks.workunit.client.0.smithi042.stdout:removed './183/33' 2019-12-05T05:08:39.681 INFO:tasks.workunit.client.0.smithi042.stdout:removed './183/17' 2019-12-05T05:08:39.681 INFO:tasks.workunit.client.0.smithi042.stdout:removed './183/35' 2019-12-05T05:08:39.681 INFO:tasks.workunit.client.0.smithi042.stdout:removed './183/37' 2019-12-05T05:08:39.682 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './183' 2019-12-05T05:08:39.682 INFO:tasks.workunit.client.0.smithi042.stdout:removed './184/19' 2019-12-05T05:08:39.682 INFO:tasks.workunit.client.0.smithi042.stdout:removed './184/15' 2019-12-05T05:08:39.682 INFO:tasks.workunit.client.0.smithi042.stdout:removed './184/31' 2019-12-05T05:08:39.682 INFO:tasks.workunit.client.0.smithi042.stdout:removed './184/36' 2019-12-05T05:08:39.682 INFO:tasks.workunit.client.0.smithi042.stdout:removed './184/20' 2019-12-05T05:08:39.682 INFO:tasks.workunit.client.0.smithi042.stdout:removed './184/12' 2019-12-05T05:08:39.683 INFO:tasks.workunit.client.0.smithi042.stdout:removed './184/39' 2019-12-05T05:08:39.683 INFO:tasks.workunit.client.0.smithi042.stdout:removed './184/10' 2019-12-05T05:08:39.683 INFO:tasks.workunit.client.0.smithi042.stdout:removed './184/34' 2019-12-05T05:08:39.683 INFO:tasks.workunit.client.0.smithi042.stdout:removed './184/38' 2019-12-05T05:08:39.683 INFO:tasks.workunit.client.0.smithi042.stdout:removed './184/11' 2019-12-05T05:08:39.683 INFO:tasks.workunit.client.0.smithi042.stdout:removed './184/16' 2019-12-05T05:08:39.683 INFO:tasks.workunit.client.0.smithi042.stdout:removed './184/32' 2019-12-05T05:08:39.683 INFO:tasks.workunit.client.0.smithi042.stdout:removed './184/30' 2019-12-05T05:08:39.684 INFO:tasks.workunit.client.0.smithi042.stdout:removed './184/14' 2019-12-05T05:08:39.684 INFO:tasks.workunit.client.0.smithi042.stdout:removed './184/18' 2019-12-05T05:08:39.685 INFO:tasks.workunit.client.0.smithi042.stdout:removed './184/13' 2019-12-05T05:08:39.685 INFO:tasks.workunit.client.0.smithi042.stdout:removed './184/33' 2019-12-05T05:08:39.685 INFO:tasks.workunit.client.0.smithi042.stdout:removed './184/17' 2019-12-05T05:08:39.685 INFO:tasks.workunit.client.0.smithi042.stdout:removed './184/35' 2019-12-05T05:08:39.685 INFO:tasks.workunit.client.0.smithi042.stdout:removed './184/37' 2019-12-05T05:08:39.685 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './184' 2019-12-05T05:08:39.685 INFO:tasks.workunit.client.0.smithi042.stdout:removed './185/19' 2019-12-05T05:08:39.686 INFO:tasks.workunit.client.0.smithi042.stdout:removed './185/15' 2019-12-05T05:08:39.686 INFO:tasks.workunit.client.0.smithi042.stdout:removed './185/31' 2019-12-05T05:08:39.686 INFO:tasks.workunit.client.0.smithi042.stdout:removed './185/36' 2019-12-05T05:08:39.686 INFO:tasks.workunit.client.0.smithi042.stdout:removed './185/20' 2019-12-05T05:08:39.686 INFO:tasks.workunit.client.0.smithi042.stdout:removed './185/12' 2019-12-05T05:08:39.686 INFO:tasks.workunit.client.0.smithi042.stdout:removed './185/39' 2019-12-05T05:08:39.686 INFO:tasks.workunit.client.0.smithi042.stdout:removed './185/10' 2019-12-05T05:08:39.687 INFO:tasks.workunit.client.0.smithi042.stdout:removed './185/34' 2019-12-05T05:08:39.687 INFO:tasks.workunit.client.0.smithi042.stdout:removed './185/38' 2019-12-05T05:08:39.687 INFO:tasks.workunit.client.0.smithi042.stdout:removed './185/11' 2019-12-05T05:08:39.687 INFO:tasks.workunit.client.0.smithi042.stdout:removed './185/16' 2019-12-05T05:08:39.687 INFO:tasks.workunit.client.0.smithi042.stdout:removed './185/32' 2019-12-05T05:08:39.687 INFO:tasks.workunit.client.0.smithi042.stdout:removed './185/30' 2019-12-05T05:08:39.687 INFO:tasks.workunit.client.0.smithi042.stdout:removed './185/14' 2019-12-05T05:08:39.687 INFO:tasks.workunit.client.0.smithi042.stdout:removed './185/18' 2019-12-05T05:08:39.688 INFO:tasks.workunit.client.0.smithi042.stdout:removed './185/13' 2019-12-05T05:08:39.688 INFO:tasks.workunit.client.0.smithi042.stdout:removed './185/33' 2019-12-05T05:08:39.688 INFO:tasks.workunit.client.0.smithi042.stdout:removed './185/17' 2019-12-05T05:08:39.688 INFO:tasks.workunit.client.0.smithi042.stdout:removed './185/35' 2019-12-05T05:08:39.688 INFO:tasks.workunit.client.0.smithi042.stdout:removed './185/37' 2019-12-05T05:08:39.688 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './185' 2019-12-05T05:08:39.688 INFO:tasks.workunit.client.0.smithi042.stdout:removed './186/19' 2019-12-05T05:08:39.689 INFO:tasks.workunit.client.0.smithi042.stdout:removed './186/15' 2019-12-05T05:08:39.689 INFO:tasks.workunit.client.0.smithi042.stdout:removed './186/31' 2019-12-05T05:08:39.689 INFO:tasks.workunit.client.0.smithi042.stdout:removed './186/36' 2019-12-05T05:08:39.689 INFO:tasks.workunit.client.0.smithi042.stdout:removed './186/20' 2019-12-05T05:08:39.689 INFO:tasks.workunit.client.0.smithi042.stdout:removed './186/12' 2019-12-05T05:08:39.689 INFO:tasks.workunit.client.0.smithi042.stdout:removed './186/39' 2019-12-05T05:08:39.689 INFO:tasks.workunit.client.0.smithi042.stdout:removed './186/10' 2019-12-05T05:08:39.689 INFO:tasks.workunit.client.0.smithi042.stdout:removed './186/34' 2019-12-05T05:08:39.690 INFO:tasks.workunit.client.0.smithi042.stdout:removed './186/38' 2019-12-05T05:08:39.690 INFO:tasks.workunit.client.0.smithi042.stdout:removed './186/11' 2019-12-05T05:08:39.690 INFO:tasks.workunit.client.0.smithi042.stdout:removed './186/16' 2019-12-05T05:08:39.690 INFO:tasks.workunit.client.0.smithi042.stdout:removed './186/32' 2019-12-05T05:08:39.690 INFO:tasks.workunit.client.0.smithi042.stdout:removed './186/30' 2019-12-05T05:08:39.690 INFO:tasks.workunit.client.0.smithi042.stdout:removed './186/14' 2019-12-05T05:08:39.690 INFO:tasks.workunit.client.0.smithi042.stdout:removed './186/18' 2019-12-05T05:08:46.292 INFO:tasks.workunit.client.0.smithi042.stdout:removed './186/13' 2019-12-05T05:08:46.293 INFO:tasks.workunit.client.0.smithi042.stdout:removed './186/33' 2019-12-05T05:08:46.293 INFO:tasks.workunit.client.0.smithi042.stdout:removed './186/17' 2019-12-05T05:08:46.293 INFO:tasks.workunit.client.0.smithi042.stdout:removed './186/35' 2019-12-05T05:08:46.295 INFO:tasks.workunit.client.0.smithi042.stdout:removed './186/37' 2019-12-05T05:08:46.295 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './186' 2019-12-05T05:08:46.296 INFO:tasks.workunit.client.0.smithi042.stdout:removed './187/19' 2019-12-05T05:08:46.296 INFO:tasks.workunit.client.0.smithi042.stdout:removed './187/15' 2019-12-05T05:08:46.296 INFO:tasks.workunit.client.0.smithi042.stdout:removed './187/31' 2019-12-05T05:08:46.296 INFO:tasks.workunit.client.0.smithi042.stdout:removed './187/36' 2019-12-05T05:08:46.296 INFO:tasks.workunit.client.0.smithi042.stdout:removed './187/20' 2019-12-05T05:08:46.296 INFO:tasks.workunit.client.0.smithi042.stdout:removed './187/12' 2019-12-05T05:08:46.297 INFO:tasks.workunit.client.0.smithi042.stdout:removed './187/39' 2019-12-05T05:08:46.297 INFO:tasks.workunit.client.0.smithi042.stdout:removed './187/10' 2019-12-05T05:08:46.297 INFO:tasks.workunit.client.0.smithi042.stdout:removed './187/34' 2019-12-05T05:08:46.297 INFO:tasks.workunit.client.0.smithi042.stdout:removed './187/38' 2019-12-05T05:08:46.297 INFO:tasks.workunit.client.0.smithi042.stdout:removed './187/11' 2019-12-05T05:08:46.297 INFO:tasks.workunit.client.0.smithi042.stdout:removed './187/16' 2019-12-05T05:08:46.297 INFO:tasks.workunit.client.0.smithi042.stdout:removed './187/32' 2019-12-05T05:08:46.297 INFO:tasks.workunit.client.0.smithi042.stdout:removed './187/30' 2019-12-05T05:08:46.298 INFO:tasks.workunit.client.0.smithi042.stdout:removed './187/14' 2019-12-05T05:08:46.298 INFO:tasks.workunit.client.0.smithi042.stdout:removed './187/18' 2019-12-05T05:08:46.298 INFO:tasks.workunit.client.0.smithi042.stdout:removed './187/13' 2019-12-05T05:08:46.298 INFO:tasks.workunit.client.0.smithi042.stdout:removed './187/33' 2019-12-05T05:08:46.298 INFO:tasks.workunit.client.0.smithi042.stdout:removed './187/17' 2019-12-05T05:08:46.298 INFO:tasks.workunit.client.0.smithi042.stdout:removed './187/35' 2019-12-05T05:08:46.298 INFO:tasks.workunit.client.0.smithi042.stdout:removed './187/37' 2019-12-05T05:08:46.299 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './187' 2019-12-05T05:08:46.299 INFO:tasks.workunit.client.0.smithi042.stdout:removed './188/19' 2019-12-05T05:08:46.299 INFO:tasks.workunit.client.0.smithi042.stdout:removed './188/15' 2019-12-05T05:08:46.299 INFO:tasks.workunit.client.0.smithi042.stdout:removed './188/31' 2019-12-05T05:08:46.299 INFO:tasks.workunit.client.0.smithi042.stdout:removed './188/36' 2019-12-05T05:08:46.299 INFO:tasks.workunit.client.0.smithi042.stdout:removed './188/20' 2019-12-05T05:08:46.299 INFO:tasks.workunit.client.0.smithi042.stdout:removed './188/12' 2019-12-05T05:08:46.299 INFO:tasks.workunit.client.0.smithi042.stdout:removed './188/39' 2019-12-05T05:08:46.300 INFO:tasks.workunit.client.0.smithi042.stdout:removed './188/10' 2019-12-05T05:08:46.300 INFO:tasks.workunit.client.0.smithi042.stdout:removed './188/34' 2019-12-05T05:08:46.300 INFO:tasks.workunit.client.0.smithi042.stdout:removed './188/38' 2019-12-05T05:08:46.300 INFO:tasks.workunit.client.0.smithi042.stdout:removed './188/11' 2019-12-05T05:08:46.300 INFO:tasks.workunit.client.0.smithi042.stdout:removed './188/16' 2019-12-05T05:08:46.300 INFO:tasks.workunit.client.0.smithi042.stdout:removed './188/32' 2019-12-05T05:08:46.300 INFO:tasks.workunit.client.0.smithi042.stdout:removed './188/30' 2019-12-05T05:08:46.301 INFO:tasks.workunit.client.0.smithi042.stdout:removed './188/14' 2019-12-05T05:08:46.301 INFO:tasks.workunit.client.0.smithi042.stdout:removed './188/18' 2019-12-05T05:08:46.301 INFO:tasks.workunit.client.0.smithi042.stdout:removed './188/13' 2019-12-05T05:08:46.301 INFO:tasks.workunit.client.0.smithi042.stdout:removed './188/33' 2019-12-05T05:08:46.301 INFO:tasks.workunit.client.0.smithi042.stdout:removed './188/17' 2019-12-05T05:08:46.301 INFO:tasks.workunit.client.0.smithi042.stdout:removed './188/35' 2019-12-05T05:08:46.301 INFO:tasks.workunit.client.0.smithi042.stdout:removed './188/37' 2019-12-05T05:08:46.301 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './188' 2019-12-05T05:08:46.302 INFO:tasks.workunit.client.0.smithi042.stdout:removed './189/19' 2019-12-05T05:08:46.302 INFO:tasks.workunit.client.0.smithi042.stdout:removed './189/15' 2019-12-05T05:08:46.302 INFO:tasks.workunit.client.0.smithi042.stdout:removed './189/31' 2019-12-05T05:08:46.302 INFO:tasks.workunit.client.0.smithi042.stdout:removed './189/36' 2019-12-05T05:08:46.302 INFO:tasks.workunit.client.0.smithi042.stdout:removed './189/20' 2019-12-05T05:08:46.302 INFO:tasks.workunit.client.0.smithi042.stdout:removed './189/12' 2019-12-05T05:08:46.302 INFO:tasks.workunit.client.0.smithi042.stdout:removed './189/39' 2019-12-05T05:08:46.303 INFO:tasks.workunit.client.0.smithi042.stdout:removed './189/10' 2019-12-05T05:08:46.303 INFO:tasks.workunit.client.0.smithi042.stdout:removed './189/34' 2019-12-05T05:08:46.303 INFO:tasks.workunit.client.0.smithi042.stdout:removed './189/38' 2019-12-05T05:08:46.303 INFO:tasks.workunit.client.0.smithi042.stdout:removed './189/11' 2019-12-05T05:08:46.303 INFO:tasks.workunit.client.0.smithi042.stdout:removed './189/16' 2019-12-05T05:08:46.303 INFO:tasks.workunit.client.0.smithi042.stdout:removed './189/32' 2019-12-05T05:08:46.303 INFO:tasks.workunit.client.0.smithi042.stdout:removed './189/30' 2019-12-05T05:08:46.303 INFO:tasks.workunit.client.0.smithi042.stdout:removed './189/14' 2019-12-05T05:08:46.304 INFO:tasks.workunit.client.0.smithi042.stdout:removed './189/18' 2019-12-05T05:08:46.304 INFO:tasks.workunit.client.0.smithi042.stdout:removed './189/13' 2019-12-05T05:08:46.304 INFO:tasks.workunit.client.0.smithi042.stdout:removed './189/33' 2019-12-05T05:08:46.304 INFO:tasks.workunit.client.0.smithi042.stdout:removed './189/17' 2019-12-05T05:08:46.304 INFO:tasks.workunit.client.0.smithi042.stdout:removed './189/35' 2019-12-05T05:08:46.304 INFO:tasks.workunit.client.0.smithi042.stdout:removed './189/37' 2019-12-05T05:08:46.304 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './189' 2019-12-05T05:08:46.305 INFO:tasks.workunit.client.0.smithi042.stdout:removed './190/19' 2019-12-05T05:08:46.305 INFO:tasks.workunit.client.0.smithi042.stdout:removed './190/15' 2019-12-05T05:08:46.305 INFO:tasks.workunit.client.0.smithi042.stdout:removed './190/31' 2019-12-05T05:08:46.306 INFO:tasks.workunit.client.0.smithi042.stdout:removed './190/36' 2019-12-05T05:08:46.306 INFO:tasks.workunit.client.0.smithi042.stdout:removed './190/20' 2019-12-05T05:08:46.306 INFO:tasks.workunit.client.0.smithi042.stdout:removed './190/12' 2019-12-05T05:08:46.306 INFO:tasks.workunit.client.0.smithi042.stdout:removed './190/39' 2019-12-05T05:08:46.306 INFO:tasks.workunit.client.0.smithi042.stdout:removed './190/10' 2019-12-05T05:08:46.307 INFO:tasks.workunit.client.0.smithi042.stdout:removed './190/34' 2019-12-05T05:08:46.307 INFO:tasks.workunit.client.0.smithi042.stdout:removed './190/38' 2019-12-05T05:08:46.307 INFO:tasks.workunit.client.0.smithi042.stdout:removed './190/11' 2019-12-05T05:08:46.307 INFO:tasks.workunit.client.0.smithi042.stdout:removed './190/16' 2019-12-05T05:08:46.307 INFO:tasks.workunit.client.0.smithi042.stdout:removed './190/32' 2019-12-05T05:08:46.307 INFO:tasks.workunit.client.0.smithi042.stdout:removed './190/30' 2019-12-05T05:08:46.307 INFO:tasks.workunit.client.0.smithi042.stdout:removed './190/14' 2019-12-05T05:08:46.307 INFO:tasks.workunit.client.0.smithi042.stdout:removed './190/18' 2019-12-05T05:08:46.308 INFO:tasks.workunit.client.0.smithi042.stdout:removed './190/13' 2019-12-05T05:08:46.308 INFO:tasks.workunit.client.0.smithi042.stdout:removed './190/33' 2019-12-05T05:08:46.308 INFO:tasks.workunit.client.0.smithi042.stdout:removed './190/17' 2019-12-05T05:08:46.308 INFO:tasks.workunit.client.0.smithi042.stdout:removed './190/35' 2019-12-05T05:08:46.308 INFO:tasks.workunit.client.0.smithi042.stdout:removed './190/37' 2019-12-05T05:08:46.308 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './190' 2019-12-05T05:08:46.308 INFO:tasks.workunit.client.0.smithi042.stdout:removed './191/19' 2019-12-05T05:08:46.309 INFO:tasks.workunit.client.0.smithi042.stdout:removed './191/15' 2019-12-05T05:08:46.309 INFO:tasks.workunit.client.0.smithi042.stdout:removed './191/31' 2019-12-05T05:08:46.309 INFO:tasks.workunit.client.0.smithi042.stdout:removed './191/36' 2019-12-05T05:08:46.309 INFO:tasks.workunit.client.0.smithi042.stdout:removed './191/20' 2019-12-05T05:08:46.309 INFO:tasks.workunit.client.0.smithi042.stdout:removed './191/12' 2019-12-05T05:08:46.309 INFO:tasks.workunit.client.0.smithi042.stdout:removed './191/39' 2019-12-05T05:08:46.309 INFO:tasks.workunit.client.0.smithi042.stdout:removed './191/10' 2019-12-05T05:08:46.309 INFO:tasks.workunit.client.0.smithi042.stdout:removed './191/34' 2019-12-05T05:08:46.311 INFO:tasks.workunit.client.0.smithi042.stdout:removed './191/38' 2019-12-05T05:08:46.311 INFO:tasks.workunit.client.0.smithi042.stdout:removed './191/11' 2019-12-05T05:08:46.311 INFO:tasks.workunit.client.0.smithi042.stdout:removed './191/16' 2019-12-05T05:08:46.312 INFO:tasks.workunit.client.0.smithi042.stdout:removed './191/32' 2019-12-05T05:08:46.312 INFO:tasks.workunit.client.0.smithi042.stdout:removed './191/30' 2019-12-05T05:08:46.312 INFO:tasks.workunit.client.0.smithi042.stdout:removed './191/14' 2019-12-05T05:08:46.312 INFO:tasks.workunit.client.0.smithi042.stdout:removed './191/18' 2019-12-05T05:08:46.312 INFO:tasks.workunit.client.0.smithi042.stdout:removed './191/13' 2019-12-05T05:08:46.312 INFO:tasks.workunit.client.0.smithi042.stdout:removed './191/33' 2019-12-05T05:08:46.313 INFO:tasks.workunit.client.0.smithi042.stdout:removed './191/17' 2019-12-05T05:08:46.313 INFO:tasks.workunit.client.0.smithi042.stdout:removed './191/35' 2019-12-05T05:08:46.313 INFO:tasks.workunit.client.0.smithi042.stdout:removed './191/37' 2019-12-05T05:08:46.313 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './191' 2019-12-05T05:08:46.313 INFO:tasks.workunit.client.0.smithi042.stdout:removed './192/19' 2019-12-05T05:08:46.313 INFO:tasks.workunit.client.0.smithi042.stdout:removed './192/15' 2019-12-05T05:08:46.313 INFO:tasks.workunit.client.0.smithi042.stdout:removed './192/31' 2019-12-05T05:08:46.314 INFO:tasks.workunit.client.0.smithi042.stdout:removed './192/36' 2019-12-05T05:08:46.314 INFO:tasks.workunit.client.0.smithi042.stdout:removed './192/20' 2019-12-05T05:08:46.314 INFO:tasks.workunit.client.0.smithi042.stdout:removed './192/12' 2019-12-05T05:08:46.314 INFO:tasks.workunit.client.0.smithi042.stdout:removed './192/39' 2019-12-05T05:08:46.314 INFO:tasks.workunit.client.0.smithi042.stdout:removed './192/10' 2019-12-05T05:08:46.314 INFO:tasks.workunit.client.0.smithi042.stdout:removed './192/34' 2019-12-05T05:08:46.314 INFO:tasks.workunit.client.0.smithi042.stdout:removed './192/38' 2019-12-05T05:08:46.314 INFO:tasks.workunit.client.0.smithi042.stdout:removed './192/11' 2019-12-05T05:08:46.315 INFO:tasks.workunit.client.0.smithi042.stdout:removed './192/16' 2019-12-05T05:08:46.315 INFO:tasks.workunit.client.0.smithi042.stdout:removed './192/32' 2019-12-05T05:08:46.316 INFO:tasks.workunit.client.0.smithi042.stdout:removed './192/30' 2019-12-05T05:08:46.316 INFO:tasks.workunit.client.0.smithi042.stdout:removed './192/14' 2019-12-05T05:08:46.316 INFO:tasks.workunit.client.0.smithi042.stdout:removed './192/18' 2019-12-05T05:08:46.317 INFO:tasks.workunit.client.0.smithi042.stdout:removed './192/13' 2019-12-05T05:08:46.317 INFO:tasks.workunit.client.0.smithi042.stdout:removed './192/33' 2019-12-05T05:08:46.317 INFO:tasks.workunit.client.0.smithi042.stdout:removed './192/17' 2019-12-05T05:08:46.317 INFO:tasks.workunit.client.0.smithi042.stdout:removed './192/35' 2019-12-05T05:08:46.317 INFO:tasks.workunit.client.0.smithi042.stdout:removed './192/37' 2019-12-05T05:08:46.317 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './192' 2019-12-05T05:08:46.317 INFO:tasks.workunit.client.0.smithi042.stdout:removed './193/19' 2019-12-05T05:08:46.317 INFO:tasks.workunit.client.0.smithi042.stdout:removed './193/15' 2019-12-05T05:08:46.318 INFO:tasks.workunit.client.0.smithi042.stdout:removed './193/31' 2019-12-05T05:08:46.318 INFO:tasks.workunit.client.0.smithi042.stdout:removed './193/36' 2019-12-05T05:08:46.319 INFO:tasks.workunit.client.0.smithi042.stdout:removed './193/20' 2019-12-05T05:08:46.319 INFO:tasks.workunit.client.0.smithi042.stdout:removed './193/12' 2019-12-05T05:08:46.319 INFO:tasks.workunit.client.0.smithi042.stdout:removed './193/39' 2019-12-05T05:08:46.319 INFO:tasks.workunit.client.0.smithi042.stdout:removed './193/10' 2019-12-05T05:08:46.319 INFO:tasks.workunit.client.0.smithi042.stdout:removed './193/34' 2019-12-05T05:08:46.319 INFO:tasks.workunit.client.0.smithi042.stdout:removed './193/38' 2019-12-05T05:08:46.320 INFO:tasks.workunit.client.0.smithi042.stdout:removed './193/11' 2019-12-05T05:08:46.320 INFO:tasks.workunit.client.0.smithi042.stdout:removed './193/16' 2019-12-05T05:08:46.320 INFO:tasks.workunit.client.0.smithi042.stdout:removed './193/32' 2019-12-05T05:08:46.320 INFO:tasks.workunit.client.0.smithi042.stdout:removed './193/30' 2019-12-05T05:08:46.320 INFO:tasks.workunit.client.0.smithi042.stdout:removed './193/14' 2019-12-05T05:08:46.320 INFO:tasks.workunit.client.0.smithi042.stdout:removed './193/18' 2019-12-05T05:08:46.320 INFO:tasks.workunit.client.0.smithi042.stdout:removed './193/13' 2019-12-05T05:08:46.321 INFO:tasks.workunit.client.0.smithi042.stdout:removed './193/33' 2019-12-05T05:08:46.321 INFO:tasks.workunit.client.0.smithi042.stdout:removed './193/17' 2019-12-05T05:08:46.321 INFO:tasks.workunit.client.0.smithi042.stdout:removed './193/35' 2019-12-05T05:08:46.321 INFO:tasks.workunit.client.0.smithi042.stdout:removed './193/37' 2019-12-05T05:08:46.321 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './193' 2019-12-05T05:08:46.321 INFO:tasks.workunit.client.0.smithi042.stdout:removed './194/19' 2019-12-05T05:08:46.321 INFO:tasks.workunit.client.0.smithi042.stdout:removed './194/15' 2019-12-05T05:08:46.321 INFO:tasks.workunit.client.0.smithi042.stdout:removed './194/31' 2019-12-05T05:08:46.322 INFO:tasks.workunit.client.0.smithi042.stdout:removed './194/36' 2019-12-05T05:08:46.322 INFO:tasks.workunit.client.0.smithi042.stdout:removed './194/20' 2019-12-05T05:08:46.322 INFO:tasks.workunit.client.0.smithi042.stdout:removed './194/12' 2019-12-05T05:08:46.322 INFO:tasks.workunit.client.0.smithi042.stdout:removed './194/39' 2019-12-05T05:08:46.322 INFO:tasks.workunit.client.0.smithi042.stdout:removed './194/10' 2019-12-05T05:08:46.322 INFO:tasks.workunit.client.0.smithi042.stdout:removed './194/34' 2019-12-05T05:08:46.323 INFO:tasks.workunit.client.0.smithi042.stdout:removed './194/38' 2019-12-05T05:08:46.323 INFO:tasks.workunit.client.0.smithi042.stdout:removed './194/11' 2019-12-05T05:08:46.323 INFO:tasks.workunit.client.0.smithi042.stdout:removed './194/16' 2019-12-05T05:08:46.323 INFO:tasks.workunit.client.0.smithi042.stdout:removed './194/32' 2019-12-05T05:08:46.323 INFO:tasks.workunit.client.0.smithi042.stdout:removed './194/30' 2019-12-05T05:08:46.323 INFO:tasks.workunit.client.0.smithi042.stdout:removed './194/14' 2019-12-05T05:08:46.323 INFO:tasks.workunit.client.0.smithi042.stdout:removed './194/18' 2019-12-05T05:08:46.324 INFO:tasks.workunit.client.0.smithi042.stdout:removed './194/13' 2019-12-05T05:08:46.324 INFO:tasks.workunit.client.0.smithi042.stdout:removed './194/33' 2019-12-05T05:08:46.324 INFO:tasks.workunit.client.0.smithi042.stdout:removed './194/17' 2019-12-05T05:08:46.324 INFO:tasks.workunit.client.0.smithi042.stdout:removed './194/35' 2019-12-05T05:08:46.324 INFO:tasks.workunit.client.0.smithi042.stdout:removed './194/37' 2019-12-05T05:08:46.324 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './194' 2019-12-05T05:08:46.324 INFO:tasks.workunit.client.0.smithi042.stdout:removed './195/19' 2019-12-05T05:08:46.325 INFO:tasks.workunit.client.0.smithi042.stdout:removed './195/15' 2019-12-05T05:08:46.325 INFO:tasks.workunit.client.0.smithi042.stdout:removed './195/31' 2019-12-05T05:08:46.325 INFO:tasks.workunit.client.0.smithi042.stdout:removed './195/36' 2019-12-05T05:08:46.325 INFO:tasks.workunit.client.0.smithi042.stdout:removed './195/20' 2019-12-05T05:08:46.325 INFO:tasks.workunit.client.0.smithi042.stdout:removed './195/12' 2019-12-05T05:08:46.325 INFO:tasks.workunit.client.0.smithi042.stdout:removed './195/39' 2019-12-05T05:08:46.325 INFO:tasks.workunit.client.0.smithi042.stdout:removed './195/10' 2019-12-05T05:08:46.325 INFO:tasks.workunit.client.0.smithi042.stdout:removed './195/34' 2019-12-05T05:08:46.326 INFO:tasks.workunit.client.0.smithi042.stdout:removed './195/38' 2019-12-05T05:08:46.326 INFO:tasks.workunit.client.0.smithi042.stdout:removed './195/11' 2019-12-05T05:08:46.327 INFO:tasks.workunit.client.0.smithi042.stdout:removed './195/16' 2019-12-05T05:08:46.327 INFO:tasks.workunit.client.0.smithi042.stdout:removed './195/32' 2019-12-05T05:08:46.327 INFO:tasks.workunit.client.0.smithi042.stdout:removed './195/30' 2019-12-05T05:08:46.327 INFO:tasks.workunit.client.0.smithi042.stdout:removed './195/14' 2019-12-05T05:08:46.327 INFO:tasks.workunit.client.0.smithi042.stdout:removed './195/18' 2019-12-05T05:08:46.327 INFO:tasks.workunit.client.0.smithi042.stdout:removed './195/13' 2019-12-05T05:08:46.327 INFO:tasks.workunit.client.0.smithi042.stdout:removed './195/33' 2019-12-05T05:08:46.328 INFO:tasks.workunit.client.0.smithi042.stdout:removed './195/17' 2019-12-05T05:08:46.328 INFO:tasks.workunit.client.0.smithi042.stdout:removed './195/35' 2019-12-05T05:08:46.328 INFO:tasks.workunit.client.0.smithi042.stdout:removed './195/37' 2019-12-05T05:08:46.328 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './195' 2019-12-05T05:08:46.328 INFO:tasks.workunit.client.0.smithi042.stdout:removed './196/19' 2019-12-05T05:08:46.328 INFO:tasks.workunit.client.0.smithi042.stdout:removed './196/15' 2019-12-05T05:08:46.328 INFO:tasks.workunit.client.0.smithi042.stdout:removed './196/31' 2019-12-05T05:08:46.328 INFO:tasks.workunit.client.0.smithi042.stdout:removed './196/36' 2019-12-05T05:08:46.329 INFO:tasks.workunit.client.0.smithi042.stdout:removed './196/20' 2019-12-05T05:08:46.329 INFO:tasks.workunit.client.0.smithi042.stdout:removed './196/12' 2019-12-05T05:08:46.329 INFO:tasks.workunit.client.0.smithi042.stdout:removed './196/39' 2019-12-05T05:08:46.329 INFO:tasks.workunit.client.0.smithi042.stdout:removed './196/10' 2019-12-05T05:08:49.101 INFO:tasks.workunit.client.0.smithi042.stdout:removed './196/34' 2019-12-05T05:08:49.101 INFO:tasks.workunit.client.0.smithi042.stdout:removed './196/38' 2019-12-05T05:08:49.101 INFO:tasks.workunit.client.0.smithi042.stdout:removed './196/11' 2019-12-05T05:08:49.101 INFO:tasks.workunit.client.0.smithi042.stdout:removed './196/16' 2019-12-05T05:08:49.101 INFO:tasks.workunit.client.0.smithi042.stdout:removed './196/32' 2019-12-05T05:08:49.102 INFO:tasks.workunit.client.0.smithi042.stdout:removed './196/30' 2019-12-05T05:08:49.102 INFO:tasks.workunit.client.0.smithi042.stdout:removed './196/14' 2019-12-05T05:08:49.102 INFO:tasks.workunit.client.0.smithi042.stdout:removed './196/18' 2019-12-05T05:08:49.102 INFO:tasks.workunit.client.0.smithi042.stdout:removed './196/13' 2019-12-05T05:08:49.102 INFO:tasks.workunit.client.0.smithi042.stdout:removed './196/33' 2019-12-05T05:08:49.102 INFO:tasks.workunit.client.0.smithi042.stdout:removed './196/17' 2019-12-05T05:08:49.103 INFO:tasks.workunit.client.0.smithi042.stdout:removed './196/35' 2019-12-05T05:08:49.103 INFO:tasks.workunit.client.0.smithi042.stdout:removed './196/37' 2019-12-05T05:08:49.103 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './196' 2019-12-05T05:08:49.103 INFO:tasks.workunit.client.0.smithi042.stdout:removed './197/19' 2019-12-05T05:08:49.103 INFO:tasks.workunit.client.0.smithi042.stdout:removed './197/15' 2019-12-05T05:08:49.103 INFO:tasks.workunit.client.0.smithi042.stdout:removed './197/31' 2019-12-05T05:08:49.103 INFO:tasks.workunit.client.0.smithi042.stdout:removed './197/36' 2019-12-05T05:08:49.104 INFO:tasks.workunit.client.0.smithi042.stdout:removed './197/20' 2019-12-05T05:08:49.104 INFO:tasks.workunit.client.0.smithi042.stdout:removed './197/12' 2019-12-05T05:08:49.104 INFO:tasks.workunit.client.0.smithi042.stdout:removed './197/39' 2019-12-05T05:08:49.104 INFO:tasks.workunit.client.0.smithi042.stdout:removed './197/10' 2019-12-05T05:08:49.104 INFO:tasks.workunit.client.0.smithi042.stdout:removed './197/34' 2019-12-05T05:08:49.104 INFO:tasks.workunit.client.0.smithi042.stdout:removed './197/38' 2019-12-05T05:08:49.105 INFO:tasks.workunit.client.0.smithi042.stdout:removed './197/11' 2019-12-05T05:08:49.105 INFO:tasks.workunit.client.0.smithi042.stdout:removed './197/16' 2019-12-05T05:08:49.105 INFO:tasks.workunit.client.0.smithi042.stdout:removed './197/32' 2019-12-05T05:08:49.105 INFO:tasks.workunit.client.0.smithi042.stdout:removed './197/30' 2019-12-05T05:08:49.105 INFO:tasks.workunit.client.0.smithi042.stdout:removed './197/14' 2019-12-05T05:08:49.105 INFO:tasks.workunit.client.0.smithi042.stdout:removed './197/18' 2019-12-05T05:08:49.105 INFO:tasks.workunit.client.0.smithi042.stdout:removed './197/13' 2019-12-05T05:08:49.106 INFO:tasks.workunit.client.0.smithi042.stdout:removed './197/33' 2019-12-05T05:08:49.106 INFO:tasks.workunit.client.0.smithi042.stdout:removed './197/17' 2019-12-05T05:08:49.106 INFO:tasks.workunit.client.0.smithi042.stdout:removed './197/35' 2019-12-05T05:08:49.106 INFO:tasks.workunit.client.0.smithi042.stdout:removed './197/37' 2019-12-05T05:08:49.106 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './197' 2019-12-05T05:08:49.106 INFO:tasks.workunit.client.0.smithi042.stdout:removed './198/19' 2019-12-05T05:08:49.107 INFO:tasks.workunit.client.0.smithi042.stdout:removed './198/15' 2019-12-05T05:08:49.107 INFO:tasks.workunit.client.0.smithi042.stdout:removed './198/31' 2019-12-05T05:08:49.107 INFO:tasks.workunit.client.0.smithi042.stdout:removed './198/36' 2019-12-05T05:08:49.107 INFO:tasks.workunit.client.0.smithi042.stdout:removed './198/20' 2019-12-05T05:08:49.107 INFO:tasks.workunit.client.0.smithi042.stdout:removed './198/12' 2019-12-05T05:08:49.107 INFO:tasks.workunit.client.0.smithi042.stdout:removed './198/39' 2019-12-05T05:08:49.108 INFO:tasks.workunit.client.0.smithi042.stdout:removed './198/10' 2019-12-05T05:08:49.108 INFO:tasks.workunit.client.0.smithi042.stdout:removed './198/34' 2019-12-05T05:08:49.108 INFO:tasks.workunit.client.0.smithi042.stdout:removed './198/38' 2019-12-05T05:08:49.108 INFO:tasks.workunit.client.0.smithi042.stdout:removed './198/11' 2019-12-05T05:08:49.109 INFO:tasks.workunit.client.0.smithi042.stdout:removed './198/16' 2019-12-05T05:08:49.109 INFO:tasks.workunit.client.0.smithi042.stdout:removed './198/32' 2019-12-05T05:08:49.109 INFO:tasks.workunit.client.0.smithi042.stdout:removed './198/30' 2019-12-05T05:08:49.109 INFO:tasks.workunit.client.0.smithi042.stdout:removed './198/14' 2019-12-05T05:08:49.110 INFO:tasks.workunit.client.0.smithi042.stdout:removed './198/18' 2019-12-05T05:08:49.110 INFO:tasks.workunit.client.0.smithi042.stdout:removed './198/13' 2019-12-05T05:08:49.110 INFO:tasks.workunit.client.0.smithi042.stdout:removed './198/33' 2019-12-05T05:08:49.110 INFO:tasks.workunit.client.0.smithi042.stdout:removed './198/17' 2019-12-05T05:08:49.110 INFO:tasks.workunit.client.0.smithi042.stdout:removed './198/35' 2019-12-05T05:08:49.110 INFO:tasks.workunit.client.0.smithi042.stdout:removed './198/37' 2019-12-05T05:08:49.111 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './198' 2019-12-05T05:08:49.111 INFO:tasks.workunit.client.0.smithi042.stdout:removed './199/19' 2019-12-05T05:08:49.111 INFO:tasks.workunit.client.0.smithi042.stdout:removed './199/15' 2019-12-05T05:08:49.111 INFO:tasks.workunit.client.0.smithi042.stdout:removed './199/31' 2019-12-05T05:08:49.111 INFO:tasks.workunit.client.0.smithi042.stdout:removed './199/36' 2019-12-05T05:08:49.111 INFO:tasks.workunit.client.0.smithi042.stdout:removed './199/20' 2019-12-05T05:08:49.111 INFO:tasks.workunit.client.0.smithi042.stdout:removed './199/12' 2019-12-05T05:08:49.112 INFO:tasks.workunit.client.0.smithi042.stdout:removed './199/39' 2019-12-05T05:08:49.112 INFO:tasks.workunit.client.0.smithi042.stdout:removed './199/10' 2019-12-05T05:08:49.112 INFO:tasks.workunit.client.0.smithi042.stdout:removed './199/34' 2019-12-05T05:08:49.112 INFO:tasks.workunit.client.0.smithi042.stdout:removed './199/38' 2019-12-05T05:08:49.112 INFO:tasks.workunit.client.0.smithi042.stdout:removed './199/11' 2019-12-05T05:08:49.112 INFO:tasks.workunit.client.0.smithi042.stdout:removed './199/16' 2019-12-05T05:08:49.113 INFO:tasks.workunit.client.0.smithi042.stdout:removed './199/32' 2019-12-05T05:08:49.113 INFO:tasks.workunit.client.0.smithi042.stdout:removed './199/30' 2019-12-05T05:08:49.113 INFO:tasks.workunit.client.0.smithi042.stdout:removed './199/14' 2019-12-05T05:08:49.113 INFO:tasks.workunit.client.0.smithi042.stdout:removed './199/18' 2019-12-05T05:08:49.113 INFO:tasks.workunit.client.0.smithi042.stdout:removed './199/13' 2019-12-05T05:08:49.113 INFO:tasks.workunit.client.0.smithi042.stdout:removed './199/33' 2019-12-05T05:08:49.113 INFO:tasks.workunit.client.0.smithi042.stdout:removed './199/17' 2019-12-05T05:08:49.114 INFO:tasks.workunit.client.0.smithi042.stdout:removed './199/35' 2019-12-05T05:08:49.114 INFO:tasks.workunit.client.0.smithi042.stdout:removed './199/37' 2019-12-05T05:08:49.114 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './199' 2019-12-05T05:08:49.114 INFO:tasks.workunit.client.0.smithi042.stdout:removed './200/21' 2019-12-05T05:08:49.114 INFO:tasks.workunit.client.0.smithi042.stdout:removed './200/27' 2019-12-05T05:08:49.114 INFO:tasks.workunit.client.0.smithi042.stdout:removed './200/31' 2019-12-05T05:08:49.115 INFO:tasks.workunit.client.0.smithi042.stdout:removed './200/36' 2019-12-05T05:08:49.115 INFO:tasks.workunit.client.0.smithi042.stdout:removed './200/20' 2019-12-05T05:08:49.115 INFO:tasks.workunit.client.0.smithi042.stdout:removed './200/39' 2019-12-05T05:08:49.115 INFO:tasks.workunit.client.0.smithi042.stdout:removed './200/34' 2019-12-05T05:08:49.115 INFO:tasks.workunit.client.0.smithi042.stdout:removed './200/38' 2019-12-05T05:08:49.115 INFO:tasks.workunit.client.0.smithi042.stdout:removed './200/28' 2019-12-05T05:08:49.115 INFO:tasks.workunit.client.0.smithi042.stdout:removed './200/24' 2019-12-05T05:08:49.116 INFO:tasks.workunit.client.0.smithi042.stdout:removed './200/29' 2019-12-05T05:08:49.116 INFO:tasks.workunit.client.0.smithi042.stdout:removed './200/32' 2019-12-05T05:08:49.116 INFO:tasks.workunit.client.0.smithi042.stdout:removed './200/22' 2019-12-05T05:08:49.116 INFO:tasks.workunit.client.0.smithi042.stdout:removed './200/25' 2019-12-05T05:08:49.116 INFO:tasks.workunit.client.0.smithi042.stdout:removed './200/30' 2019-12-05T05:08:49.116 INFO:tasks.workunit.client.0.smithi042.stdout:removed './200/33' 2019-12-05T05:08:49.117 INFO:tasks.workunit.client.0.smithi042.stdout:removed './200/23' 2019-12-05T05:08:49.117 INFO:tasks.workunit.client.0.smithi042.stdout:removed './200/35' 2019-12-05T05:08:49.117 INFO:tasks.workunit.client.0.smithi042.stdout:removed './200/26' 2019-12-05T05:08:49.117 INFO:tasks.workunit.client.0.smithi042.stdout:removed './200/37' 2019-12-05T05:08:49.117 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './200' 2019-12-05T05:08:49.117 INFO:tasks.workunit.client.0.smithi042.stdout:removed './201/21' 2019-12-05T05:08:49.117 INFO:tasks.workunit.client.0.smithi042.stdout:removed './201/27' 2019-12-05T05:08:49.118 INFO:tasks.workunit.client.0.smithi042.stdout:removed './201/31' 2019-12-05T05:08:49.118 INFO:tasks.workunit.client.0.smithi042.stdout:removed './201/36' 2019-12-05T05:08:49.118 INFO:tasks.workunit.client.0.smithi042.stdout:removed './201/20' 2019-12-05T05:08:49.121 INFO:tasks.workunit.client.0.smithi042.stdout:removed './201/39' 2019-12-05T05:08:49.121 INFO:tasks.workunit.client.0.smithi042.stdout:removed './201/34' 2019-12-05T05:08:49.121 INFO:tasks.workunit.client.0.smithi042.stdout:removed './201/38' 2019-12-05T05:08:49.121 INFO:tasks.workunit.client.0.smithi042.stdout:removed './201/28' 2019-12-05T05:08:49.122 INFO:tasks.workunit.client.0.smithi042.stdout:removed './201/24' 2019-12-05T05:08:49.122 INFO:tasks.workunit.client.0.smithi042.stdout:removed './201/29' 2019-12-05T05:08:49.123 INFO:tasks.workunit.client.0.smithi042.stdout:removed './201/32' 2019-12-05T05:08:49.123 INFO:tasks.workunit.client.0.smithi042.stdout:removed './201/22' 2019-12-05T05:08:49.124 INFO:tasks.workunit.client.0.smithi042.stdout:removed './201/25' 2019-12-05T05:08:49.125 INFO:tasks.workunit.client.0.smithi042.stdout:removed './201/30' 2019-12-05T05:08:49.125 INFO:tasks.workunit.client.0.smithi042.stdout:removed './201/33' 2019-12-05T05:08:49.125 INFO:tasks.workunit.client.0.smithi042.stdout:removed './201/23' 2019-12-05T05:08:49.125 INFO:tasks.workunit.client.0.smithi042.stdout:removed './201/35' 2019-12-05T05:08:49.126 INFO:tasks.workunit.client.0.smithi042.stdout:removed './201/26' 2019-12-05T05:08:49.126 INFO:tasks.workunit.client.0.smithi042.stdout:removed './201/37' 2019-12-05T05:08:49.126 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './201' 2019-12-05T05:08:49.126 INFO:tasks.workunit.client.0.smithi042.stdout:removed './202/21' 2019-12-05T05:08:49.126 INFO:tasks.workunit.client.0.smithi042.stdout:removed './202/27' 2019-12-05T05:08:49.126 INFO:tasks.workunit.client.0.smithi042.stdout:removed './202/31' 2019-12-05T05:08:49.127 INFO:tasks.workunit.client.0.smithi042.stdout:removed './202/36' 2019-12-05T05:08:49.127 INFO:tasks.workunit.client.0.smithi042.stdout:removed './202/20' 2019-12-05T05:08:49.127 INFO:tasks.workunit.client.0.smithi042.stdout:removed './202/39' 2019-12-05T05:08:49.127 INFO:tasks.workunit.client.0.smithi042.stdout:removed './202/34' 2019-12-05T05:08:49.127 INFO:tasks.workunit.client.0.smithi042.stdout:removed './202/38' 2019-12-05T05:08:49.127 INFO:tasks.workunit.client.0.smithi042.stdout:removed './202/28' 2019-12-05T05:08:49.127 INFO:tasks.workunit.client.0.smithi042.stdout:removed './202/24' 2019-12-05T05:08:49.128 INFO:tasks.workunit.client.0.smithi042.stdout:removed './202/29' 2019-12-05T05:08:49.128 INFO:tasks.workunit.client.0.smithi042.stdout:removed './202/32' 2019-12-05T05:08:49.129 INFO:tasks.workunit.client.0.smithi042.stdout:removed './202/22' 2019-12-05T05:08:49.129 INFO:tasks.workunit.client.0.smithi042.stdout:removed './202/25' 2019-12-05T05:08:49.129 INFO:tasks.workunit.client.0.smithi042.stdout:removed './202/30' 2019-12-05T05:08:49.130 INFO:tasks.workunit.client.0.smithi042.stdout:removed './202/33' 2019-12-05T05:08:49.130 INFO:tasks.workunit.client.0.smithi042.stdout:removed './202/23' 2019-12-05T05:08:49.130 INFO:tasks.workunit.client.0.smithi042.stdout:removed './202/35' 2019-12-05T05:08:49.130 INFO:tasks.workunit.client.0.smithi042.stdout:removed './202/26' 2019-12-05T05:08:49.130 INFO:tasks.workunit.client.0.smithi042.stdout:removed './202/37' 2019-12-05T05:08:49.131 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './202' 2019-12-05T05:08:49.131 INFO:tasks.workunit.client.0.smithi042.stdout:removed './203/21' 2019-12-05T05:08:49.131 INFO:tasks.workunit.client.0.smithi042.stdout:removed './203/27' 2019-12-05T05:08:49.131 INFO:tasks.workunit.client.0.smithi042.stdout:removed './203/31' 2019-12-05T05:08:49.131 INFO:tasks.workunit.client.0.smithi042.stdout:removed './203/36' 2019-12-05T05:08:49.131 INFO:tasks.workunit.client.0.smithi042.stdout:removed './203/20' 2019-12-05T05:08:49.132 INFO:tasks.workunit.client.0.smithi042.stdout:removed './203/39' 2019-12-05T05:08:49.132 INFO:tasks.workunit.client.0.smithi042.stdout:removed './203/34' 2019-12-05T05:08:49.132 INFO:tasks.workunit.client.0.smithi042.stdout:removed './203/38' 2019-12-05T05:08:49.132 INFO:tasks.workunit.client.0.smithi042.stdout:removed './203/28' 2019-12-05T05:08:49.132 INFO:tasks.workunit.client.0.smithi042.stdout:removed './203/24' 2019-12-05T05:08:49.132 INFO:tasks.workunit.client.0.smithi042.stdout:removed './203/29' 2019-12-05T05:08:49.133 INFO:tasks.workunit.client.0.smithi042.stdout:removed './203/32' 2019-12-05T05:08:49.133 INFO:tasks.workunit.client.0.smithi042.stdout:removed './203/22' 2019-12-05T05:08:49.133 INFO:tasks.workunit.client.0.smithi042.stdout:removed './203/25' 2019-12-05T05:08:49.133 INFO:tasks.workunit.client.0.smithi042.stdout:removed './203/30' 2019-12-05T05:08:49.133 INFO:tasks.workunit.client.0.smithi042.stdout:removed './203/33' 2019-12-05T05:08:49.133 INFO:tasks.workunit.client.0.smithi042.stdout:removed './203/23' 2019-12-05T05:08:49.133 INFO:tasks.workunit.client.0.smithi042.stdout:removed './203/35' 2019-12-05T05:08:49.134 INFO:tasks.workunit.client.0.smithi042.stdout:removed './203/26' 2019-12-05T05:08:49.134 INFO:tasks.workunit.client.0.smithi042.stdout:removed './203/37' 2019-12-05T05:08:49.134 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './203' 2019-12-05T05:08:49.134 INFO:tasks.workunit.client.0.smithi042.stdout:removed './204/21' 2019-12-05T05:08:49.134 INFO:tasks.workunit.client.0.smithi042.stdout:removed './204/27' 2019-12-05T05:08:49.134 INFO:tasks.workunit.client.0.smithi042.stdout:removed './204/31' 2019-12-05T05:08:49.134 INFO:tasks.workunit.client.0.smithi042.stdout:removed './204/36' 2019-12-05T05:08:49.135 INFO:tasks.workunit.client.0.smithi042.stdout:removed './204/20' 2019-12-05T05:08:49.135 INFO:tasks.workunit.client.0.smithi042.stdout:removed './204/39' 2019-12-05T05:08:49.135 INFO:tasks.workunit.client.0.smithi042.stdout:removed './204/34' 2019-12-05T05:08:49.135 INFO:tasks.workunit.client.0.smithi042.stdout:removed './204/38' 2019-12-05T05:08:49.135 INFO:tasks.workunit.client.0.smithi042.stdout:removed './204/28' 2019-12-05T05:08:49.135 INFO:tasks.workunit.client.0.smithi042.stdout:removed './204/24' 2019-12-05T05:08:49.135 INFO:tasks.workunit.client.0.smithi042.stdout:removed './204/29' 2019-12-05T05:08:49.136 INFO:tasks.workunit.client.0.smithi042.stdout:removed './204/32' 2019-12-05T05:08:49.136 INFO:tasks.workunit.client.0.smithi042.stdout:removed './204/22' 2019-12-05T05:08:49.136 INFO:tasks.workunit.client.0.smithi042.stdout:removed './204/25' 2019-12-05T05:08:49.136 INFO:tasks.workunit.client.0.smithi042.stdout:removed './204/30' 2019-12-05T05:08:49.136 INFO:tasks.workunit.client.0.smithi042.stdout:removed './204/33' 2019-12-05T05:08:49.136 INFO:tasks.workunit.client.0.smithi042.stdout:removed './204/23' 2019-12-05T05:08:49.137 INFO:tasks.workunit.client.0.smithi042.stdout:removed './204/35' 2019-12-05T05:08:49.137 INFO:tasks.workunit.client.0.smithi042.stdout:removed './204/26' 2019-12-05T05:08:49.137 INFO:tasks.workunit.client.0.smithi042.stdout:removed './204/37' 2019-12-05T05:08:49.137 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './204' 2019-12-05T05:08:49.137 INFO:tasks.workunit.client.0.smithi042.stdout:removed './205/21' 2019-12-05T05:08:49.138 INFO:tasks.workunit.client.0.smithi042.stdout:removed './205/27' 2019-12-05T05:08:49.138 INFO:tasks.workunit.client.0.smithi042.stdout:removed './205/31' 2019-12-05T05:08:49.138 INFO:tasks.workunit.client.0.smithi042.stdout:removed './205/36' 2019-12-05T05:08:49.138 INFO:tasks.workunit.client.0.smithi042.stdout:removed './205/20' 2019-12-05T05:08:49.138 INFO:tasks.workunit.client.0.smithi042.stdout:removed './205/39' 2019-12-05T05:08:49.138 INFO:tasks.workunit.client.0.smithi042.stdout:removed './205/34' 2019-12-05T05:08:49.139 INFO:tasks.workunit.client.0.smithi042.stdout:removed './205/38' 2019-12-05T05:08:49.139 INFO:tasks.workunit.client.0.smithi042.stdout:removed './205/28' 2019-12-05T05:08:49.139 INFO:tasks.workunit.client.0.smithi042.stdout:removed './205/24' 2019-12-05T05:08:49.139 INFO:tasks.workunit.client.0.smithi042.stdout:removed './205/29' 2019-12-05T05:08:49.139 INFO:tasks.workunit.client.0.smithi042.stdout:removed './205/32' 2019-12-05T05:08:49.139 INFO:tasks.workunit.client.0.smithi042.stdout:removed './205/22' 2019-12-05T05:08:49.140 INFO:tasks.workunit.client.0.smithi042.stdout:removed './205/25' 2019-12-05T05:08:49.140 INFO:tasks.workunit.client.0.smithi042.stdout:removed './205/30' 2019-12-05T05:08:49.141 INFO:tasks.workunit.client.0.smithi042.stdout:removed './205/33' 2019-12-05T05:08:49.141 INFO:tasks.workunit.client.0.smithi042.stdout:removed './205/23' 2019-12-05T05:08:49.141 INFO:tasks.workunit.client.0.smithi042.stdout:removed './205/35' 2019-12-05T05:08:49.141 INFO:tasks.workunit.client.0.smithi042.stdout:removed './205/26' 2019-12-05T05:08:49.141 INFO:tasks.workunit.client.0.smithi042.stdout:removed './205/37' 2019-12-05T05:08:49.142 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './205' 2019-12-05T05:08:49.142 INFO:tasks.workunit.client.0.smithi042.stdout:removed './206/21' 2019-12-05T05:08:49.142 INFO:tasks.workunit.client.0.smithi042.stdout:removed './206/27' 2019-12-05T05:08:49.142 INFO:tasks.workunit.client.0.smithi042.stdout:removed './206/31' 2019-12-05T05:08:49.143 INFO:tasks.workunit.client.0.smithi042.stdout:removed './206/36' 2019-12-05T05:08:49.143 INFO:tasks.workunit.client.0.smithi042.stdout:removed './206/20' 2019-12-05T05:08:49.143 INFO:tasks.workunit.client.0.smithi042.stdout:removed './206/39' 2019-12-05T05:08:51.387 INFO:tasks.workunit.client.0.smithi042.stdout:removed './206/34' 2019-12-05T05:08:51.388 INFO:tasks.workunit.client.0.smithi042.stdout:removed './206/38' 2019-12-05T05:08:51.388 INFO:tasks.workunit.client.0.smithi042.stdout:removed './206/28' 2019-12-05T05:08:51.388 INFO:tasks.workunit.client.0.smithi042.stdout:removed './206/24' 2019-12-05T05:08:51.388 INFO:tasks.workunit.client.0.smithi042.stdout:removed './206/29' 2019-12-05T05:08:51.388 INFO:tasks.workunit.client.0.smithi042.stdout:removed './206/32' 2019-12-05T05:08:51.389 INFO:tasks.workunit.client.0.smithi042.stdout:removed './206/22' 2019-12-05T05:08:51.389 INFO:tasks.workunit.client.0.smithi042.stdout:removed './206/25' 2019-12-05T05:08:51.389 INFO:tasks.workunit.client.0.smithi042.stdout:removed './206/30' 2019-12-05T05:08:51.389 INFO:tasks.workunit.client.0.smithi042.stdout:removed './206/33' 2019-12-05T05:08:51.389 INFO:tasks.workunit.client.0.smithi042.stdout:removed './206/23' 2019-12-05T05:08:51.389 INFO:tasks.workunit.client.0.smithi042.stdout:removed './206/35' 2019-12-05T05:08:51.390 INFO:tasks.workunit.client.0.smithi042.stdout:removed './206/26' 2019-12-05T05:08:51.390 INFO:tasks.workunit.client.0.smithi042.stdout:removed './206/37' 2019-12-05T05:08:51.390 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './206' 2019-12-05T05:08:51.390 INFO:tasks.workunit.client.0.smithi042.stdout:removed './207/21' 2019-12-05T05:08:51.390 INFO:tasks.workunit.client.0.smithi042.stdout:removed './207/27' 2019-12-05T05:08:51.390 INFO:tasks.workunit.client.0.smithi042.stdout:removed './207/31' 2019-12-05T05:08:51.390 INFO:tasks.workunit.client.0.smithi042.stdout:removed './207/36' 2019-12-05T05:08:51.391 INFO:tasks.workunit.client.0.smithi042.stdout:removed './207/20' 2019-12-05T05:08:51.391 INFO:tasks.workunit.client.0.smithi042.stdout:removed './207/39' 2019-12-05T05:08:51.391 INFO:tasks.workunit.client.0.smithi042.stdout:removed './207/34' 2019-12-05T05:08:51.391 INFO:tasks.workunit.client.0.smithi042.stdout:removed './207/38' 2019-12-05T05:08:51.392 INFO:tasks.workunit.client.0.smithi042.stdout:removed './207/28' 2019-12-05T05:08:51.393 INFO:tasks.workunit.client.0.smithi042.stdout:removed './207/24' 2019-12-05T05:08:51.393 INFO:tasks.workunit.client.0.smithi042.stdout:removed './207/29' 2019-12-05T05:08:51.393 INFO:tasks.workunit.client.0.smithi042.stdout:removed './207/32' 2019-12-05T05:08:51.394 INFO:tasks.workunit.client.0.smithi042.stdout:removed './207/22' 2019-12-05T05:08:51.394 INFO:tasks.workunit.client.0.smithi042.stdout:removed './207/25' 2019-12-05T05:08:51.394 INFO:tasks.workunit.client.0.smithi042.stdout:removed './207/30' 2019-12-05T05:08:51.394 INFO:tasks.workunit.client.0.smithi042.stdout:removed './207/33' 2019-12-05T05:08:51.394 INFO:tasks.workunit.client.0.smithi042.stdout:removed './207/23' 2019-12-05T05:08:51.394 INFO:tasks.workunit.client.0.smithi042.stdout:removed './207/35' 2019-12-05T05:08:51.394 INFO:tasks.workunit.client.0.smithi042.stdout:removed './207/26' 2019-12-05T05:08:51.395 INFO:tasks.workunit.client.0.smithi042.stdout:removed './207/37' 2019-12-05T05:08:51.395 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './207' 2019-12-05T05:08:51.395 INFO:tasks.workunit.client.0.smithi042.stdout:removed './208/21' 2019-12-05T05:08:51.395 INFO:tasks.workunit.client.0.smithi042.stdout:removed './208/27' 2019-12-05T05:08:51.395 INFO:tasks.workunit.client.0.smithi042.stdout:removed './208/31' 2019-12-05T05:08:51.395 INFO:tasks.workunit.client.0.smithi042.stdout:removed './208/36' 2019-12-05T05:08:51.396 INFO:tasks.workunit.client.0.smithi042.stdout:removed './208/20' 2019-12-05T05:08:51.396 INFO:tasks.workunit.client.0.smithi042.stdout:removed './208/39' 2019-12-05T05:08:51.396 INFO:tasks.workunit.client.0.smithi042.stdout:removed './208/34' 2019-12-05T05:08:51.396 INFO:tasks.workunit.client.0.smithi042.stdout:removed './208/38' 2019-12-05T05:08:51.396 INFO:tasks.workunit.client.0.smithi042.stdout:removed './208/28' 2019-12-05T05:08:51.397 INFO:tasks.workunit.client.0.smithi042.stdout:removed './208/24' 2019-12-05T05:08:51.397 INFO:tasks.workunit.client.0.smithi042.stdout:removed './208/29' 2019-12-05T05:08:51.397 INFO:tasks.workunit.client.0.smithi042.stdout:removed './208/32' 2019-12-05T05:08:51.397 INFO:tasks.workunit.client.0.smithi042.stdout:removed './208/22' 2019-12-05T05:08:51.397 INFO:tasks.workunit.client.0.smithi042.stdout:removed './208/25' 2019-12-05T05:08:51.397 INFO:tasks.workunit.client.0.smithi042.stdout:removed './208/30' 2019-12-05T05:08:51.398 INFO:tasks.workunit.client.0.smithi042.stdout:removed './208/33' 2019-12-05T05:08:51.398 INFO:tasks.workunit.client.0.smithi042.stdout:removed './208/23' 2019-12-05T05:08:51.399 INFO:tasks.workunit.client.0.smithi042.stdout:removed './208/35' 2019-12-05T05:08:51.399 INFO:tasks.workunit.client.0.smithi042.stdout:removed './208/26' 2019-12-05T05:08:51.399 INFO:tasks.workunit.client.0.smithi042.stdout:removed './208/37' 2019-12-05T05:08:51.399 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './208' 2019-12-05T05:08:51.399 INFO:tasks.workunit.client.0.smithi042.stdout:removed './209/21' 2019-12-05T05:08:51.399 INFO:tasks.workunit.client.0.smithi042.stdout:removed './209/27' 2019-12-05T05:08:51.400 INFO:tasks.workunit.client.0.smithi042.stdout:removed './209/31' 2019-12-05T05:08:51.400 INFO:tasks.workunit.client.0.smithi042.stdout:removed './209/36' 2019-12-05T05:08:51.400 INFO:tasks.workunit.client.0.smithi042.stdout:removed './209/20' 2019-12-05T05:08:51.400 INFO:tasks.workunit.client.0.smithi042.stdout:removed './209/39' 2019-12-05T05:08:51.401 INFO:tasks.workunit.client.0.smithi042.stdout:removed './209/34' 2019-12-05T05:08:51.401 INFO:tasks.workunit.client.0.smithi042.stdout:removed './209/38' 2019-12-05T05:08:51.401 INFO:tasks.workunit.client.0.smithi042.stdout:removed './209/28' 2019-12-05T05:08:51.401 INFO:tasks.workunit.client.0.smithi042.stdout:removed './209/24' 2019-12-05T05:08:51.401 INFO:tasks.workunit.client.0.smithi042.stdout:removed './209/29' 2019-12-05T05:08:51.401 INFO:tasks.workunit.client.0.smithi042.stdout:removed './209/32' 2019-12-05T05:08:51.401 INFO:tasks.workunit.client.0.smithi042.stdout:removed './209/22' 2019-12-05T05:08:51.402 INFO:tasks.workunit.client.0.smithi042.stdout:removed './209/25' 2019-12-05T05:08:51.402 INFO:tasks.workunit.client.0.smithi042.stdout:removed './209/30' 2019-12-05T05:08:51.402 INFO:tasks.workunit.client.0.smithi042.stdout:removed './209/33' 2019-12-05T05:08:51.403 INFO:tasks.workunit.client.0.smithi042.stdout:removed './209/23' 2019-12-05T05:08:51.403 INFO:tasks.workunit.client.0.smithi042.stdout:removed './209/35' 2019-12-05T05:08:51.403 INFO:tasks.workunit.client.0.smithi042.stdout:removed './209/26' 2019-12-05T05:08:51.403 INFO:tasks.workunit.client.0.smithi042.stdout:removed './209/37' 2019-12-05T05:08:51.404 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './209' 2019-12-05T05:08:51.404 INFO:tasks.workunit.client.0.smithi042.stdout:removed './210/21' 2019-12-05T05:08:51.404 INFO:tasks.workunit.client.0.smithi042.stdout:removed './210/27' 2019-12-05T05:08:51.404 INFO:tasks.workunit.client.0.smithi042.stdout:removed './210/31' 2019-12-05T05:08:51.404 INFO:tasks.workunit.client.0.smithi042.stdout:removed './210/36' 2019-12-05T05:08:51.405 INFO:tasks.workunit.client.0.smithi042.stdout:removed './210/20' 2019-12-05T05:08:51.405 INFO:tasks.workunit.client.0.smithi042.stdout:removed './210/39' 2019-12-05T05:08:51.405 INFO:tasks.workunit.client.0.smithi042.stdout:removed './210/34' 2019-12-05T05:08:51.405 INFO:tasks.workunit.client.0.smithi042.stdout:removed './210/38' 2019-12-05T05:08:51.405 INFO:tasks.workunit.client.0.smithi042.stdout:removed './210/28' 2019-12-05T05:08:51.405 INFO:tasks.workunit.client.0.smithi042.stdout:removed './210/24' 2019-12-05T05:08:51.405 INFO:tasks.workunit.client.0.smithi042.stdout:removed './210/29' 2019-12-05T05:08:51.406 INFO:tasks.workunit.client.0.smithi042.stdout:removed './210/32' 2019-12-05T05:08:51.406 INFO:tasks.workunit.client.0.smithi042.stdout:removed './210/22' 2019-12-05T05:08:51.406 INFO:tasks.workunit.client.0.smithi042.stdout:removed './210/25' 2019-12-05T05:08:51.406 INFO:tasks.workunit.client.0.smithi042.stdout:removed './210/30' 2019-12-05T05:08:51.406 INFO:tasks.workunit.client.0.smithi042.stdout:removed './210/33' 2019-12-05T05:08:51.407 INFO:tasks.workunit.client.0.smithi042.stdout:removed './210/23' 2019-12-05T05:08:51.407 INFO:tasks.workunit.client.0.smithi042.stdout:removed './210/35' 2019-12-05T05:08:51.407 INFO:tasks.workunit.client.0.smithi042.stdout:removed './210/26' 2019-12-05T05:08:51.407 INFO:tasks.workunit.client.0.smithi042.stdout:removed './210/37' 2019-12-05T05:08:51.407 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './210' 2019-12-05T05:08:51.408 INFO:tasks.workunit.client.0.smithi042.stdout:removed './211/21' 2019-12-05T05:08:51.408 INFO:tasks.workunit.client.0.smithi042.stdout:removed './211/27' 2019-12-05T05:08:51.409 INFO:tasks.workunit.client.0.smithi042.stdout:removed './211/31' 2019-12-05T05:08:51.409 INFO:tasks.workunit.client.0.smithi042.stdout:removed './211/36' 2019-12-05T05:08:51.409 INFO:tasks.workunit.client.0.smithi042.stdout:removed './211/20' 2019-12-05T05:08:51.409 INFO:tasks.workunit.client.0.smithi042.stdout:removed './211/39' 2019-12-05T05:08:51.409 INFO:tasks.workunit.client.0.smithi042.stdout:removed './211/34' 2019-12-05T05:08:51.409 INFO:tasks.workunit.client.0.smithi042.stdout:removed './211/38' 2019-12-05T05:08:51.410 INFO:tasks.workunit.client.0.smithi042.stdout:removed './211/28' 2019-12-05T05:08:51.410 INFO:tasks.workunit.client.0.smithi042.stdout:removed './211/24' 2019-12-05T05:08:51.410 INFO:tasks.workunit.client.0.smithi042.stdout:removed './211/29' 2019-12-05T05:08:51.410 INFO:tasks.workunit.client.0.smithi042.stdout:removed './211/32' 2019-12-05T05:08:51.410 INFO:tasks.workunit.client.0.smithi042.stdout:removed './211/22' 2019-12-05T05:08:51.410 INFO:tasks.workunit.client.0.smithi042.stdout:removed './211/25' 2019-12-05T05:08:51.411 INFO:tasks.workunit.client.0.smithi042.stdout:removed './211/30' 2019-12-05T05:08:51.411 INFO:tasks.workunit.client.0.smithi042.stdout:removed './211/33' 2019-12-05T05:08:51.411 INFO:tasks.workunit.client.0.smithi042.stdout:removed './211/23' 2019-12-05T05:08:51.411 INFO:tasks.workunit.client.0.smithi042.stdout:removed './211/35' 2019-12-05T05:08:51.411 INFO:tasks.workunit.client.0.smithi042.stdout:removed './211/26' 2019-12-05T05:08:51.411 INFO:tasks.workunit.client.0.smithi042.stdout:removed './211/37' 2019-12-05T05:08:51.412 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './211' 2019-12-05T05:08:51.412 INFO:tasks.workunit.client.0.smithi042.stdout:removed './212/21' 2019-12-05T05:08:51.412 INFO:tasks.workunit.client.0.smithi042.stdout:removed './212/27' 2019-12-05T05:08:51.412 INFO:tasks.workunit.client.0.smithi042.stdout:removed './212/31' 2019-12-05T05:08:51.412 INFO:tasks.workunit.client.0.smithi042.stdout:removed './212/36' 2019-12-05T05:08:51.413 INFO:tasks.workunit.client.0.smithi042.stdout:removed './212/20' 2019-12-05T05:08:51.413 INFO:tasks.workunit.client.0.smithi042.stdout:removed './212/39' 2019-12-05T05:08:51.413 INFO:tasks.workunit.client.0.smithi042.stdout:removed './212/34' 2019-12-05T05:08:51.413 INFO:tasks.workunit.client.0.smithi042.stdout:removed './212/38' 2019-12-05T05:08:51.414 INFO:tasks.workunit.client.0.smithi042.stdout:removed './212/28' 2019-12-05T05:08:51.414 INFO:tasks.workunit.client.0.smithi042.stdout:removed './212/24' 2019-12-05T05:08:51.414 INFO:tasks.workunit.client.0.smithi042.stdout:removed './212/29' 2019-12-05T05:08:51.414 INFO:tasks.workunit.client.0.smithi042.stdout:removed './212/32' 2019-12-05T05:08:51.414 INFO:tasks.workunit.client.0.smithi042.stdout:removed './212/22' 2019-12-05T05:08:51.415 INFO:tasks.workunit.client.0.smithi042.stdout:removed './212/25' 2019-12-05T05:08:51.415 INFO:tasks.workunit.client.0.smithi042.stdout:removed './212/30' 2019-12-05T05:08:51.415 INFO:tasks.workunit.client.0.smithi042.stdout:removed './212/33' 2019-12-05T05:08:51.415 INFO:tasks.workunit.client.0.smithi042.stdout:removed './212/23' 2019-12-05T05:08:51.415 INFO:tasks.workunit.client.0.smithi042.stdout:removed './212/35' 2019-12-05T05:08:51.415 INFO:tasks.workunit.client.0.smithi042.stdout:removed './212/26' 2019-12-05T05:08:51.416 INFO:tasks.workunit.client.0.smithi042.stdout:removed './212/37' 2019-12-05T05:08:51.416 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './212' 2019-12-05T05:08:51.416 INFO:tasks.workunit.client.0.smithi042.stdout:removed './213/21' 2019-12-05T05:08:51.416 INFO:tasks.workunit.client.0.smithi042.stdout:removed './213/27' 2019-12-05T05:08:51.416 INFO:tasks.workunit.client.0.smithi042.stdout:removed './213/31' 2019-12-05T05:08:51.416 INFO:tasks.workunit.client.0.smithi042.stdout:removed './213/36' 2019-12-05T05:08:51.417 INFO:tasks.workunit.client.0.smithi042.stdout:removed './213/20' 2019-12-05T05:08:51.417 INFO:tasks.workunit.client.0.smithi042.stdout:removed './213/39' 2019-12-05T05:08:51.417 INFO:tasks.workunit.client.0.smithi042.stdout:removed './213/34' 2019-12-05T05:08:51.417 INFO:tasks.workunit.client.0.smithi042.stdout:removed './213/38' 2019-12-05T05:08:51.417 INFO:tasks.workunit.client.0.smithi042.stdout:removed './213/28' 2019-12-05T05:08:51.417 INFO:tasks.workunit.client.0.smithi042.stdout:removed './213/24' 2019-12-05T05:08:51.417 INFO:tasks.workunit.client.0.smithi042.stdout:removed './213/29' 2019-12-05T05:08:51.418 INFO:tasks.workunit.client.0.smithi042.stdout:removed './213/32' 2019-12-05T05:08:51.418 INFO:tasks.workunit.client.0.smithi042.stdout:removed './213/22' 2019-12-05T05:08:51.418 INFO:tasks.workunit.client.0.smithi042.stdout:removed './213/25' 2019-12-05T05:08:51.418 INFO:tasks.workunit.client.0.smithi042.stdout:removed './213/30' 2019-12-05T05:08:51.418 INFO:tasks.workunit.client.0.smithi042.stdout:removed './213/33' 2019-12-05T05:08:51.419 INFO:tasks.workunit.client.0.smithi042.stdout:removed './213/23' 2019-12-05T05:08:51.419 INFO:tasks.workunit.client.0.smithi042.stdout:removed './213/35' 2019-12-05T05:08:51.419 INFO:tasks.workunit.client.0.smithi042.stdout:removed './213/26' 2019-12-05T05:08:51.420 INFO:tasks.workunit.client.0.smithi042.stdout:removed './213/37' 2019-12-05T05:08:51.420 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './213' 2019-12-05T05:08:51.420 INFO:tasks.workunit.client.0.smithi042.stdout:removed './214/21' 2019-12-05T05:08:51.420 INFO:tasks.workunit.client.0.smithi042.stdout:removed './214/27' 2019-12-05T05:08:51.420 INFO:tasks.workunit.client.0.smithi042.stdout:removed './214/31' 2019-12-05T05:08:51.420 INFO:tasks.workunit.client.0.smithi042.stdout:removed './214/36' 2019-12-05T05:08:51.420 INFO:tasks.workunit.client.0.smithi042.stdout:removed './214/20' 2019-12-05T05:08:51.421 INFO:tasks.workunit.client.0.smithi042.stdout:removed './214/39' 2019-12-05T05:08:51.421 INFO:tasks.workunit.client.0.smithi042.stdout:removed './214/34' 2019-12-05T05:08:51.421 INFO:tasks.workunit.client.0.smithi042.stdout:removed './214/38' 2019-12-05T05:08:51.421 INFO:tasks.workunit.client.0.smithi042.stdout:removed './214/28' 2019-12-05T05:08:51.421 INFO:tasks.workunit.client.0.smithi042.stdout:removed './214/24' 2019-12-05T05:08:51.421 INFO:tasks.workunit.client.0.smithi042.stdout:removed './214/29' 2019-12-05T05:08:51.422 INFO:tasks.workunit.client.0.smithi042.stdout:removed './214/32' 2019-12-05T05:08:51.422 INFO:tasks.workunit.client.0.smithi042.stdout:removed './214/22' 2019-12-05T05:08:51.422 INFO:tasks.workunit.client.0.smithi042.stdout:removed './214/25' 2019-12-05T05:08:51.422 INFO:tasks.workunit.client.0.smithi042.stdout:removed './214/30' 2019-12-05T05:08:51.422 INFO:tasks.workunit.client.0.smithi042.stdout:removed './214/33' 2019-12-05T05:08:51.422 INFO:tasks.workunit.client.0.smithi042.stdout:removed './214/23' 2019-12-05T05:08:51.423 INFO:tasks.workunit.client.0.smithi042.stdout:removed './214/35' 2019-12-05T05:08:51.423 INFO:tasks.workunit.client.0.smithi042.stdout:removed './214/26' 2019-12-05T05:08:51.423 INFO:tasks.workunit.client.0.smithi042.stdout:removed './214/37' 2019-12-05T05:08:51.424 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './214' 2019-12-05T05:08:51.424 INFO:tasks.workunit.client.0.smithi042.stdout:removed './215/21' 2019-12-05T05:08:51.424 INFO:tasks.workunit.client.0.smithi042.stdout:removed './215/27' 2019-12-05T05:08:51.424 INFO:tasks.workunit.client.0.smithi042.stdout:removed './215/31' 2019-12-05T05:08:51.425 INFO:tasks.workunit.client.0.smithi042.stdout:removed './215/36' 2019-12-05T05:08:51.425 INFO:tasks.workunit.client.0.smithi042.stdout:removed './215/20' 2019-12-05T05:08:51.425 INFO:tasks.workunit.client.0.smithi042.stdout:removed './215/39' 2019-12-05T05:08:51.425 INFO:tasks.workunit.client.0.smithi042.stdout:removed './215/34' 2019-12-05T05:08:51.425 INFO:tasks.workunit.client.0.smithi042.stdout:removed './215/38' 2019-12-05T05:08:51.425 INFO:tasks.workunit.client.0.smithi042.stdout:removed './215/28' 2019-12-05T05:08:51.426 INFO:tasks.workunit.client.0.smithi042.stdout:removed './215/24' 2019-12-05T05:08:51.426 INFO:tasks.workunit.client.0.smithi042.stdout:removed './215/29' 2019-12-05T05:08:51.426 INFO:tasks.workunit.client.0.smithi042.stdout:removed './215/32' 2019-12-05T05:08:51.426 INFO:tasks.workunit.client.0.smithi042.stdout:removed './215/22' 2019-12-05T05:08:51.426 INFO:tasks.workunit.client.0.smithi042.stdout:removed './215/25' 2019-12-05T05:08:51.426 INFO:tasks.workunit.client.0.smithi042.stdout:removed './215/30' 2019-12-05T05:08:51.427 INFO:tasks.workunit.client.0.smithi042.stdout:removed './215/33' 2019-12-05T05:08:51.427 INFO:tasks.workunit.client.0.smithi042.stdout:removed './215/23' 2019-12-05T05:08:51.427 INFO:tasks.workunit.client.0.smithi042.stdout:removed './215/35' 2019-12-05T05:08:51.427 INFO:tasks.workunit.client.0.smithi042.stdout:removed './215/26' 2019-12-05T05:08:51.427 INFO:tasks.workunit.client.0.smithi042.stdout:removed './215/37' 2019-12-05T05:08:51.428 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './215' 2019-12-05T05:08:51.428 INFO:tasks.workunit.client.0.smithi042.stdout:removed './216/21' 2019-12-05T05:08:51.428 INFO:tasks.workunit.client.0.smithi042.stdout:removed './216/27' 2019-12-05T05:08:51.428 INFO:tasks.workunit.client.0.smithi042.stdout:removed './216/31' 2019-12-05T05:08:51.428 INFO:tasks.workunit.client.0.smithi042.stdout:removed './216/36' 2019-12-05T05:08:51.428 INFO:tasks.workunit.client.0.smithi042.stdout:removed './216/20' 2019-12-05T05:08:51.428 INFO:tasks.workunit.client.0.smithi042.stdout:removed './216/39' 2019-12-05T05:08:51.429 INFO:tasks.workunit.client.0.smithi042.stdout:removed './216/34' 2019-12-05T05:08:51.429 INFO:tasks.workunit.client.0.smithi042.stdout:removed './216/38' 2019-12-05T05:08:56.037 INFO:tasks.workunit.client.0.smithi042.stdout:removed './216/28' 2019-12-05T05:08:56.037 INFO:tasks.workunit.client.0.smithi042.stdout:removed './216/24' 2019-12-05T05:08:56.037 INFO:tasks.workunit.client.0.smithi042.stdout:removed './216/29' 2019-12-05T05:08:56.037 INFO:tasks.workunit.client.0.smithi042.stdout:removed './216/32' 2019-12-05T05:08:56.038 INFO:tasks.workunit.client.0.smithi042.stdout:removed './216/22' 2019-12-05T05:08:56.038 INFO:tasks.workunit.client.0.smithi042.stdout:removed './216/25' 2019-12-05T05:08:56.039 INFO:tasks.workunit.client.0.smithi042.stdout:removed './216/30' 2019-12-05T05:08:56.039 INFO:tasks.workunit.client.0.smithi042.stdout:removed './216/33' 2019-12-05T05:08:56.039 INFO:tasks.workunit.client.0.smithi042.stdout:removed './216/23' 2019-12-05T05:08:56.039 INFO:tasks.workunit.client.0.smithi042.stdout:removed './216/35' 2019-12-05T05:08:56.039 INFO:tasks.workunit.client.0.smithi042.stdout:removed './216/26' 2019-12-05T05:08:56.039 INFO:tasks.workunit.client.0.smithi042.stdout:removed './216/37' 2019-12-05T05:08:56.039 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './216' 2019-12-05T05:08:56.040 INFO:tasks.workunit.client.0.smithi042.stdout:removed './217/21' 2019-12-05T05:08:56.040 INFO:tasks.workunit.client.0.smithi042.stdout:removed './217/27' 2019-12-05T05:08:56.040 INFO:tasks.workunit.client.0.smithi042.stdout:removed './217/31' 2019-12-05T05:08:56.040 INFO:tasks.workunit.client.0.smithi042.stdout:removed './217/36' 2019-12-05T05:08:56.040 INFO:tasks.workunit.client.0.smithi042.stdout:removed './217/20' 2019-12-05T05:08:56.040 INFO:tasks.workunit.client.0.smithi042.stdout:removed './217/39' 2019-12-05T05:08:56.040 INFO:tasks.workunit.client.0.smithi042.stdout:removed './217/34' 2019-12-05T05:08:56.041 INFO:tasks.workunit.client.0.smithi042.stdout:removed './217/38' 2019-12-05T05:08:56.041 INFO:tasks.workunit.client.0.smithi042.stdout:removed './217/28' 2019-12-05T05:08:56.041 INFO:tasks.workunit.client.0.smithi042.stdout:removed './217/24' 2019-12-05T05:08:56.041 INFO:tasks.workunit.client.0.smithi042.stdout:removed './217/29' 2019-12-05T05:08:56.041 INFO:tasks.workunit.client.0.smithi042.stdout:removed './217/32' 2019-12-05T05:08:56.042 INFO:tasks.workunit.client.0.smithi042.stdout:removed './217/22' 2019-12-05T05:08:56.043 INFO:tasks.workunit.client.0.smithi042.stdout:removed './217/25' 2019-12-05T05:08:56.043 INFO:tasks.workunit.client.0.smithi042.stdout:removed './217/30' 2019-12-05T05:08:56.043 INFO:tasks.workunit.client.0.smithi042.stdout:removed './217/33' 2019-12-05T05:08:56.044 INFO:tasks.workunit.client.0.smithi042.stdout:removed './217/23' 2019-12-05T05:08:56.044 INFO:tasks.workunit.client.0.smithi042.stdout:removed './217/35' 2019-12-05T05:08:56.044 INFO:tasks.workunit.client.0.smithi042.stdout:removed './217/26' 2019-12-05T05:08:56.044 INFO:tasks.workunit.client.0.smithi042.stdout:removed './217/37' 2019-12-05T05:08:56.044 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './217' 2019-12-05T05:08:56.044 INFO:tasks.workunit.client.0.smithi042.stdout:removed './218/21' 2019-12-05T05:08:56.045 INFO:tasks.workunit.client.0.smithi042.stdout:removed './218/27' 2019-12-05T05:08:56.045 INFO:tasks.workunit.client.0.smithi042.stdout:removed './218/31' 2019-12-05T05:08:56.045 INFO:tasks.workunit.client.0.smithi042.stdout:removed './218/36' 2019-12-05T05:08:56.045 INFO:tasks.workunit.client.0.smithi042.stdout:removed './218/20' 2019-12-05T05:08:56.046 INFO:tasks.workunit.client.0.smithi042.stdout:removed './218/39' 2019-12-05T05:08:56.046 INFO:tasks.workunit.client.0.smithi042.stdout:removed './218/34' 2019-12-05T05:08:56.046 INFO:tasks.workunit.client.0.smithi042.stdout:removed './218/38' 2019-12-05T05:08:56.047 INFO:tasks.workunit.client.0.smithi042.stdout:removed './218/28' 2019-12-05T05:08:56.047 INFO:tasks.workunit.client.0.smithi042.stdout:removed './218/24' 2019-12-05T05:08:56.047 INFO:tasks.workunit.client.0.smithi042.stdout:removed './218/29' 2019-12-05T05:08:56.047 INFO:tasks.workunit.client.0.smithi042.stdout:removed './218/32' 2019-12-05T05:08:56.047 INFO:tasks.workunit.client.0.smithi042.stdout:removed './218/22' 2019-12-05T05:08:56.047 INFO:tasks.workunit.client.0.smithi042.stdout:removed './218/25' 2019-12-05T05:08:56.047 INFO:tasks.workunit.client.0.smithi042.stdout:removed './218/30' 2019-12-05T05:08:56.048 INFO:tasks.workunit.client.0.smithi042.stdout:removed './218/33' 2019-12-05T05:08:56.048 INFO:tasks.workunit.client.0.smithi042.stdout:removed './218/23' 2019-12-05T05:08:56.048 INFO:tasks.workunit.client.0.smithi042.stdout:removed './218/35' 2019-12-05T05:08:56.048 INFO:tasks.workunit.client.0.smithi042.stdout:removed './218/26' 2019-12-05T05:08:56.049 INFO:tasks.workunit.client.0.smithi042.stdout:removed './218/37' 2019-12-05T05:08:56.049 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './218' 2019-12-05T05:08:56.049 INFO:tasks.workunit.client.0.smithi042.stdout:removed './219/21' 2019-12-05T05:08:56.049 INFO:tasks.workunit.client.0.smithi042.stdout:removed './219/27' 2019-12-05T05:08:56.049 INFO:tasks.workunit.client.0.smithi042.stdout:removed './219/31' 2019-12-05T05:08:56.049 INFO:tasks.workunit.client.0.smithi042.stdout:removed './219/36' 2019-12-05T05:08:56.049 INFO:tasks.workunit.client.0.smithi042.stdout:removed './219/20' 2019-12-05T05:08:56.050 INFO:tasks.workunit.client.0.smithi042.stdout:removed './219/39' 2019-12-05T05:08:56.050 INFO:tasks.workunit.client.0.smithi042.stdout:removed './219/34' 2019-12-05T05:08:56.050 INFO:tasks.workunit.client.0.smithi042.stdout:removed './219/38' 2019-12-05T05:08:56.050 INFO:tasks.workunit.client.0.smithi042.stdout:removed './219/28' 2019-12-05T05:08:56.050 INFO:tasks.workunit.client.0.smithi042.stdout:removed './219/24' 2019-12-05T05:08:56.050 INFO:tasks.workunit.client.0.smithi042.stdout:removed './219/29' 2019-12-05T05:08:56.051 INFO:tasks.workunit.client.0.smithi042.stdout:removed './219/32' 2019-12-05T05:08:56.051 INFO:tasks.workunit.client.0.smithi042.stdout:removed './219/22' 2019-12-05T05:08:56.051 INFO:tasks.workunit.client.0.smithi042.stdout:removed './219/25' 2019-12-05T05:08:56.051 INFO:tasks.workunit.client.0.smithi042.stdout:removed './219/30' 2019-12-05T05:08:56.051 INFO:tasks.workunit.client.0.smithi042.stdout:removed './219/33' 2019-12-05T05:08:56.051 INFO:tasks.workunit.client.0.smithi042.stdout:removed './219/23' 2019-12-05T05:08:56.051 INFO:tasks.workunit.client.0.smithi042.stdout:removed './219/35' 2019-12-05T05:08:56.051 INFO:tasks.workunit.client.0.smithi042.stdout:removed './219/26' 2019-12-05T05:08:56.052 INFO:tasks.workunit.client.0.smithi042.stdout:removed './219/37' 2019-12-05T05:08:56.052 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './219' 2019-12-05T05:08:56.052 INFO:tasks.workunit.client.0.smithi042.stdout:removed './220/21' 2019-12-05T05:08:56.053 INFO:tasks.workunit.client.0.smithi042.stdout:removed './220/27' 2019-12-05T05:08:56.053 INFO:tasks.workunit.client.0.smithi042.stdout:removed './220/31' 2019-12-05T05:08:56.053 INFO:tasks.workunit.client.0.smithi042.stdout:removed './220/36' 2019-12-05T05:08:56.053 INFO:tasks.workunit.client.0.smithi042.stdout:removed './220/20' 2019-12-05T05:08:56.053 INFO:tasks.workunit.client.0.smithi042.stdout:removed './220/39' 2019-12-05T05:08:56.054 INFO:tasks.workunit.client.0.smithi042.stdout:removed './220/34' 2019-12-05T05:08:56.054 INFO:tasks.workunit.client.0.smithi042.stdout:removed './220/38' 2019-12-05T05:08:56.054 INFO:tasks.workunit.client.0.smithi042.stdout:removed './220/28' 2019-12-05T05:08:56.054 INFO:tasks.workunit.client.0.smithi042.stdout:removed './220/24' 2019-12-05T05:08:56.054 INFO:tasks.workunit.client.0.smithi042.stdout:removed './220/29' 2019-12-05T05:08:56.054 INFO:tasks.workunit.client.0.smithi042.stdout:removed './220/32' 2019-12-05T05:08:56.055 INFO:tasks.workunit.client.0.smithi042.stdout:removed './220/22' 2019-12-05T05:08:56.055 INFO:tasks.workunit.client.0.smithi042.stdout:removed './220/25' 2019-12-05T05:08:56.055 INFO:tasks.workunit.client.0.smithi042.stdout:removed './220/30' 2019-12-05T05:08:56.055 INFO:tasks.workunit.client.0.smithi042.stdout:removed './220/33' 2019-12-05T05:08:56.055 INFO:tasks.workunit.client.0.smithi042.stdout:removed './220/23' 2019-12-05T05:08:56.055 INFO:tasks.workunit.client.0.smithi042.stdout:removed './220/35' 2019-12-05T05:08:56.055 INFO:tasks.workunit.client.0.smithi042.stdout:removed './220/26' 2019-12-05T05:08:56.056 INFO:tasks.workunit.client.0.smithi042.stdout:removed './220/37' 2019-12-05T05:08:56.056 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './220' 2019-12-05T05:08:56.056 INFO:tasks.workunit.client.0.smithi042.stdout:removed './221/21' 2019-12-05T05:08:56.056 INFO:tasks.workunit.client.0.smithi042.stdout:removed './221/27' 2019-12-05T05:08:56.056 INFO:tasks.workunit.client.0.smithi042.stdout:removed './221/31' 2019-12-05T05:08:56.056 INFO:tasks.workunit.client.0.smithi042.stdout:removed './221/36' 2019-12-05T05:08:56.056 INFO:tasks.workunit.client.0.smithi042.stdout:removed './221/20' 2019-12-05T05:08:56.057 INFO:tasks.workunit.client.0.smithi042.stdout:removed './221/39' 2019-12-05T05:08:56.057 INFO:tasks.workunit.client.0.smithi042.stdout:removed './221/34' 2019-12-05T05:08:56.057 INFO:tasks.workunit.client.0.smithi042.stdout:removed './221/38' 2019-12-05T05:08:56.057 INFO:tasks.workunit.client.0.smithi042.stdout:removed './221/28' 2019-12-05T05:08:56.057 INFO:tasks.workunit.client.0.smithi042.stdout:removed './221/24' 2019-12-05T05:08:56.057 INFO:tasks.workunit.client.0.smithi042.stdout:removed './221/29' 2019-12-05T05:08:56.057 INFO:tasks.workunit.client.0.smithi042.stdout:removed './221/32' 2019-12-05T05:08:56.057 INFO:tasks.workunit.client.0.smithi042.stdout:removed './221/22' 2019-12-05T05:08:56.058 INFO:tasks.workunit.client.0.smithi042.stdout:removed './221/25' 2019-12-05T05:08:56.058 INFO:tasks.workunit.client.0.smithi042.stdout:removed './221/30' 2019-12-05T05:08:56.058 INFO:tasks.workunit.client.0.smithi042.stdout:removed './221/33' 2019-12-05T05:08:56.058 INFO:tasks.workunit.client.0.smithi042.stdout:removed './221/23' 2019-12-05T05:08:56.058 INFO:tasks.workunit.client.0.smithi042.stdout:removed './221/35' 2019-12-05T05:08:56.058 INFO:tasks.workunit.client.0.smithi042.stdout:removed './221/26' 2019-12-05T05:08:56.058 INFO:tasks.workunit.client.0.smithi042.stdout:removed './221/37' 2019-12-05T05:08:56.059 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './221' 2019-12-05T05:08:56.059 INFO:tasks.workunit.client.0.smithi042.stdout:removed './222/21' 2019-12-05T05:08:56.059 INFO:tasks.workunit.client.0.smithi042.stdout:removed './222/27' 2019-12-05T05:08:56.059 INFO:tasks.workunit.client.0.smithi042.stdout:removed './222/31' 2019-12-05T05:08:56.059 INFO:tasks.workunit.client.0.smithi042.stdout:removed './222/36' 2019-12-05T05:08:56.060 INFO:tasks.workunit.client.0.smithi042.stdout:removed './222/20' 2019-12-05T05:08:56.060 INFO:tasks.workunit.client.0.smithi042.stdout:removed './222/39' 2019-12-05T05:08:56.060 INFO:tasks.workunit.client.0.smithi042.stdout:removed './222/34' 2019-12-05T05:08:56.060 INFO:tasks.workunit.client.0.smithi042.stdout:removed './222/38' 2019-12-05T05:08:56.060 INFO:tasks.workunit.client.0.smithi042.stdout:removed './222/28' 2019-12-05T05:08:56.060 INFO:tasks.workunit.client.0.smithi042.stdout:removed './222/24' 2019-12-05T05:08:56.060 INFO:tasks.workunit.client.0.smithi042.stdout:removed './222/29' 2019-12-05T05:08:56.061 INFO:tasks.workunit.client.0.smithi042.stdout:removed './222/32' 2019-12-05T05:08:56.061 INFO:tasks.workunit.client.0.smithi042.stdout:removed './222/22' 2019-12-05T05:08:56.061 INFO:tasks.workunit.client.0.smithi042.stdout:removed './222/25' 2019-12-05T05:08:56.061 INFO:tasks.workunit.client.0.smithi042.stdout:removed './222/30' 2019-12-05T05:08:56.061 INFO:tasks.workunit.client.0.smithi042.stdout:removed './222/33' 2019-12-05T05:08:56.061 INFO:tasks.workunit.client.0.smithi042.stdout:removed './222/23' 2019-12-05T05:08:56.061 INFO:tasks.workunit.client.0.smithi042.stdout:removed './222/35' 2019-12-05T05:08:56.062 INFO:tasks.workunit.client.0.smithi042.stdout:removed './222/26' 2019-12-05T05:08:56.062 INFO:tasks.workunit.client.0.smithi042.stdout:removed './222/37' 2019-12-05T05:08:56.062 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './222' 2019-12-05T05:08:56.062 INFO:tasks.workunit.client.0.smithi042.stdout:removed './223/21' 2019-12-05T05:08:56.064 INFO:tasks.workunit.client.0.smithi042.stdout:removed './223/27' 2019-12-05T05:08:56.064 INFO:tasks.workunit.client.0.smithi042.stdout:removed './223/31' 2019-12-05T05:08:56.064 INFO:tasks.workunit.client.0.smithi042.stdout:removed './223/36' 2019-12-05T05:08:56.064 INFO:tasks.workunit.client.0.smithi042.stdout:removed './223/20' 2019-12-05T05:08:56.065 INFO:tasks.workunit.client.0.smithi042.stdout:removed './223/39' 2019-12-05T05:08:56.065 INFO:tasks.workunit.client.0.smithi042.stdout:removed './223/34' 2019-12-05T05:08:56.065 INFO:tasks.workunit.client.0.smithi042.stdout:removed './223/38' 2019-12-05T05:08:56.065 INFO:tasks.workunit.client.0.smithi042.stdout:removed './223/28' 2019-12-05T05:08:56.065 INFO:tasks.workunit.client.0.smithi042.stdout:removed './223/24' 2019-12-05T05:08:56.066 INFO:tasks.workunit.client.0.smithi042.stdout:removed './223/29' 2019-12-05T05:08:56.066 INFO:tasks.workunit.client.0.smithi042.stdout:removed './223/32' 2019-12-05T05:08:56.066 INFO:tasks.workunit.client.0.smithi042.stdout:removed './223/22' 2019-12-05T05:08:56.066 INFO:tasks.workunit.client.0.smithi042.stdout:removed './223/25' 2019-12-05T05:08:56.066 INFO:tasks.workunit.client.0.smithi042.stdout:removed './223/30' 2019-12-05T05:08:56.066 INFO:tasks.workunit.client.0.smithi042.stdout:removed './223/33' 2019-12-05T05:08:56.066 INFO:tasks.workunit.client.0.smithi042.stdout:removed './223/23' 2019-12-05T05:08:56.067 INFO:tasks.workunit.client.0.smithi042.stdout:removed './223/35' 2019-12-05T05:08:56.067 INFO:tasks.workunit.client.0.smithi042.stdout:removed './223/26' 2019-12-05T05:08:56.067 INFO:tasks.workunit.client.0.smithi042.stdout:removed './223/37' 2019-12-05T05:08:56.067 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './223' 2019-12-05T05:08:56.067 INFO:tasks.workunit.client.0.smithi042.stdout:removed './224/21' 2019-12-05T05:08:56.067 INFO:tasks.workunit.client.0.smithi042.stdout:removed './224/27' 2019-12-05T05:08:56.068 INFO:tasks.workunit.client.0.smithi042.stdout:removed './224/31' 2019-12-05T05:08:56.068 INFO:tasks.workunit.client.0.smithi042.stdout:removed './224/36' 2019-12-05T05:08:56.068 INFO:tasks.workunit.client.0.smithi042.stdout:removed './224/20' 2019-12-05T05:08:56.068 INFO:tasks.workunit.client.0.smithi042.stdout:removed './224/39' 2019-12-05T05:08:56.068 INFO:tasks.workunit.client.0.smithi042.stdout:removed './224/34' 2019-12-05T05:08:56.068 INFO:tasks.workunit.client.0.smithi042.stdout:removed './224/38' 2019-12-05T05:08:56.068 INFO:tasks.workunit.client.0.smithi042.stdout:removed './224/28' 2019-12-05T05:08:56.069 INFO:tasks.workunit.client.0.smithi042.stdout:removed './224/24' 2019-12-05T05:08:56.070 INFO:tasks.workunit.client.0.smithi042.stdout:removed './224/29' 2019-12-05T05:08:56.070 INFO:tasks.workunit.client.0.smithi042.stdout:removed './224/32' 2019-12-05T05:08:56.070 INFO:tasks.workunit.client.0.smithi042.stdout:removed './224/22' 2019-12-05T05:08:56.070 INFO:tasks.workunit.client.0.smithi042.stdout:removed './224/25' 2019-12-05T05:08:56.070 INFO:tasks.workunit.client.0.smithi042.stdout:removed './224/30' 2019-12-05T05:08:56.070 INFO:tasks.workunit.client.0.smithi042.stdout:removed './224/33' 2019-12-05T05:08:56.071 INFO:tasks.workunit.client.0.smithi042.stdout:removed './224/23' 2019-12-05T05:08:56.071 INFO:tasks.workunit.client.0.smithi042.stdout:removed './224/35' 2019-12-05T05:08:56.072 INFO:tasks.workunit.client.0.smithi042.stdout:removed './224/26' 2019-12-05T05:08:56.073 INFO:tasks.workunit.client.0.smithi042.stdout:removed './224/37' 2019-12-05T05:08:56.074 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './224' 2019-12-05T05:08:56.075 INFO:tasks.workunit.client.0.smithi042.stdout:removed './225/21' 2019-12-05T05:08:56.075 INFO:tasks.workunit.client.0.smithi042.stdout:removed './225/27' 2019-12-05T05:08:56.075 INFO:tasks.workunit.client.0.smithi042.stdout:removed './225/31' 2019-12-05T05:08:56.075 INFO:tasks.workunit.client.0.smithi042.stdout:removed './225/36' 2019-12-05T05:08:56.075 INFO:tasks.workunit.client.0.smithi042.stdout:removed './225/20' 2019-12-05T05:08:56.076 INFO:tasks.workunit.client.0.smithi042.stdout:removed './225/39' 2019-12-05T05:08:56.076 INFO:tasks.workunit.client.0.smithi042.stdout:removed './225/34' 2019-12-05T05:08:56.076 INFO:tasks.workunit.client.0.smithi042.stdout:removed './225/38' 2019-12-05T05:08:56.076 INFO:tasks.workunit.client.0.smithi042.stdout:removed './225/28' 2019-12-05T05:08:56.076 INFO:tasks.workunit.client.0.smithi042.stdout:removed './225/24' 2019-12-05T05:08:56.077 INFO:tasks.workunit.client.0.smithi042.stdout:removed './225/29' 2019-12-05T05:08:56.077 INFO:tasks.workunit.client.0.smithi042.stdout:removed './225/32' 2019-12-05T05:08:56.077 INFO:tasks.workunit.client.0.smithi042.stdout:removed './225/22' 2019-12-05T05:08:56.077 INFO:tasks.workunit.client.0.smithi042.stdout:removed './225/25' 2019-12-05T05:08:56.078 INFO:tasks.workunit.client.0.smithi042.stdout:removed './225/30' 2019-12-05T05:08:56.078 INFO:tasks.workunit.client.0.smithi042.stdout:removed './225/33' 2019-12-05T05:08:56.078 INFO:tasks.workunit.client.0.smithi042.stdout:removed './225/23' 2019-12-05T05:08:56.078 INFO:tasks.workunit.client.0.smithi042.stdout:removed './225/35' 2019-12-05T05:08:56.079 INFO:tasks.workunit.client.0.smithi042.stdout:removed './225/26' 2019-12-05T05:08:56.079 INFO:tasks.workunit.client.0.smithi042.stdout:removed './225/37' 2019-12-05T05:08:56.079 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './225' 2019-12-05T05:08:56.079 INFO:tasks.workunit.client.0.smithi042.stdout:removed './226/21' 2019-12-05T05:08:56.079 INFO:tasks.workunit.client.0.smithi042.stdout:removed './226/27' 2019-12-05T05:08:56.080 INFO:tasks.workunit.client.0.smithi042.stdout:removed './226/31' 2019-12-05T05:08:56.080 INFO:tasks.workunit.client.0.smithi042.stdout:removed './226/36' 2019-12-05T05:08:56.080 INFO:tasks.workunit.client.0.smithi042.stdout:removed './226/20' 2019-12-05T05:08:56.080 INFO:tasks.workunit.client.0.smithi042.stdout:removed './226/39' 2019-12-05T05:08:56.081 INFO:tasks.workunit.client.0.smithi042.stdout:removed './226/34' 2019-12-05T05:08:56.081 INFO:tasks.workunit.client.0.smithi042.stdout:removed './226/38' 2019-12-05T05:08:56.081 INFO:tasks.workunit.client.0.smithi042.stdout:removed './226/28' 2019-12-05T05:08:56.081 INFO:tasks.workunit.client.0.smithi042.stdout:removed './226/24' 2019-12-05T05:09:00.826 INFO:tasks.workunit.client.0.smithi042.stdout:removed './226/29' 2019-12-05T05:09:00.827 INFO:tasks.workunit.client.0.smithi042.stdout:removed './226/32' 2019-12-05T05:09:00.827 INFO:tasks.workunit.client.0.smithi042.stdout:removed './226/22' 2019-12-05T05:09:00.827 INFO:tasks.workunit.client.0.smithi042.stdout:removed './226/25' 2019-12-05T05:09:00.827 INFO:tasks.workunit.client.0.smithi042.stdout:removed './226/30' 2019-12-05T05:09:00.827 INFO:tasks.workunit.client.0.smithi042.stdout:removed './226/33' 2019-12-05T05:09:00.827 INFO:tasks.workunit.client.0.smithi042.stdout:removed './226/23' 2019-12-05T05:09:00.828 INFO:tasks.workunit.client.0.smithi042.stdout:removed './226/35' 2019-12-05T05:09:00.828 INFO:tasks.workunit.client.0.smithi042.stdout:removed './226/26' 2019-12-05T05:09:00.828 INFO:tasks.workunit.client.0.smithi042.stdout:removed './226/37' 2019-12-05T05:09:00.828 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './226' 2019-12-05T05:09:00.828 INFO:tasks.workunit.client.0.smithi042.stdout:removed './227/21' 2019-12-05T05:09:00.828 INFO:tasks.workunit.client.0.smithi042.stdout:removed './227/27' 2019-12-05T05:09:00.828 INFO:tasks.workunit.client.0.smithi042.stdout:removed './227/31' 2019-12-05T05:09:00.829 INFO:tasks.workunit.client.0.smithi042.stdout:removed './227/36' 2019-12-05T05:09:00.829 INFO:tasks.workunit.client.0.smithi042.stdout:removed './227/20' 2019-12-05T05:09:00.829 INFO:tasks.workunit.client.0.smithi042.stdout:removed './227/39' 2019-12-05T05:09:00.829 INFO:tasks.workunit.client.0.smithi042.stdout:removed './227/34' 2019-12-05T05:09:00.829 INFO:tasks.workunit.client.0.smithi042.stdout:removed './227/38' 2019-12-05T05:09:00.829 INFO:tasks.workunit.client.0.smithi042.stdout:removed './227/28' 2019-12-05T05:09:00.829 INFO:tasks.workunit.client.0.smithi042.stdout:removed './227/24' 2019-12-05T05:09:00.830 INFO:tasks.workunit.client.0.smithi042.stdout:removed './227/29' 2019-12-05T05:09:00.830 INFO:tasks.workunit.client.0.smithi042.stdout:removed './227/32' 2019-12-05T05:09:00.830 INFO:tasks.workunit.client.0.smithi042.stdout:removed './227/22' 2019-12-05T05:09:00.830 INFO:tasks.workunit.client.0.smithi042.stdout:removed './227/25' 2019-12-05T05:09:00.830 INFO:tasks.workunit.client.0.smithi042.stdout:removed './227/30' 2019-12-05T05:09:00.831 INFO:tasks.workunit.client.0.smithi042.stdout:removed './227/33' 2019-12-05T05:09:00.831 INFO:tasks.workunit.client.0.smithi042.stdout:removed './227/23' 2019-12-05T05:09:00.831 INFO:tasks.workunit.client.0.smithi042.stdout:removed './227/35' 2019-12-05T05:09:00.831 INFO:tasks.workunit.client.0.smithi042.stdout:removed './227/26' 2019-12-05T05:09:00.831 INFO:tasks.workunit.client.0.smithi042.stdout:removed './227/37' 2019-12-05T05:09:00.832 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './227' 2019-12-05T05:09:00.832 INFO:tasks.workunit.client.0.smithi042.stdout:removed './228/21' 2019-12-05T05:09:00.832 INFO:tasks.workunit.client.0.smithi042.stdout:removed './228/27' 2019-12-05T05:09:00.832 INFO:tasks.workunit.client.0.smithi042.stdout:removed './228/31' 2019-12-05T05:09:00.832 INFO:tasks.workunit.client.0.smithi042.stdout:removed './228/36' 2019-12-05T05:09:00.832 INFO:tasks.workunit.client.0.smithi042.stdout:removed './228/20' 2019-12-05T05:09:00.833 INFO:tasks.workunit.client.0.smithi042.stdout:removed './228/39' 2019-12-05T05:09:00.833 INFO:tasks.workunit.client.0.smithi042.stdout:removed './228/34' 2019-12-05T05:09:00.833 INFO:tasks.workunit.client.0.smithi042.stdout:removed './228/38' 2019-12-05T05:09:00.833 INFO:tasks.workunit.client.0.smithi042.stdout:removed './228/28' 2019-12-05T05:09:00.833 INFO:tasks.workunit.client.0.smithi042.stdout:removed './228/24' 2019-12-05T05:09:00.833 INFO:tasks.workunit.client.0.smithi042.stdout:removed './228/29' 2019-12-05T05:09:00.834 INFO:tasks.workunit.client.0.smithi042.stdout:removed './228/32' 2019-12-05T05:09:00.834 INFO:tasks.workunit.client.0.smithi042.stdout:removed './228/22' 2019-12-05T05:09:00.834 INFO:tasks.workunit.client.0.smithi042.stdout:removed './228/25' 2019-12-05T05:09:00.834 INFO:tasks.workunit.client.0.smithi042.stdout:removed './228/30' 2019-12-05T05:09:00.834 INFO:tasks.workunit.client.0.smithi042.stdout:removed './228/33' 2019-12-05T05:09:00.834 INFO:tasks.workunit.client.0.smithi042.stdout:removed './228/23' 2019-12-05T05:09:00.834 INFO:tasks.workunit.client.0.smithi042.stdout:removed './228/35' 2019-12-05T05:09:00.835 INFO:tasks.workunit.client.0.smithi042.stdout:removed './228/26' 2019-12-05T05:09:00.835 INFO:tasks.workunit.client.0.smithi042.stdout:removed './228/37' 2019-12-05T05:09:00.836 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './228' 2019-12-05T05:09:00.836 INFO:tasks.workunit.client.0.smithi042.stdout:removed './229/21' 2019-12-05T05:09:00.836 INFO:tasks.workunit.client.0.smithi042.stdout:removed './229/27' 2019-12-05T05:09:00.836 INFO:tasks.workunit.client.0.smithi042.stdout:removed './229/31' 2019-12-05T05:09:00.837 INFO:tasks.workunit.client.0.smithi042.stdout:removed './229/36' 2019-12-05T05:09:00.837 INFO:tasks.workunit.client.0.smithi042.stdout:removed './229/20' 2019-12-05T05:09:00.837 INFO:tasks.workunit.client.0.smithi042.stdout:removed './229/39' 2019-12-05T05:09:00.837 INFO:tasks.workunit.client.0.smithi042.stdout:removed './229/34' 2019-12-05T05:09:00.837 INFO:tasks.workunit.client.0.smithi042.stdout:removed './229/38' 2019-12-05T05:09:00.837 INFO:tasks.workunit.client.0.smithi042.stdout:removed './229/28' 2019-12-05T05:09:00.838 INFO:tasks.workunit.client.0.smithi042.stdout:removed './229/24' 2019-12-05T05:09:00.838 INFO:tasks.workunit.client.0.smithi042.stdout:removed './229/29' 2019-12-05T05:09:00.838 INFO:tasks.workunit.client.0.smithi042.stdout:removed './229/32' 2019-12-05T05:09:00.838 INFO:tasks.workunit.client.0.smithi042.stdout:removed './229/22' 2019-12-05T05:09:00.838 INFO:tasks.workunit.client.0.smithi042.stdout:removed './229/25' 2019-12-05T05:09:00.838 INFO:tasks.workunit.client.0.smithi042.stdout:removed './229/30' 2019-12-05T05:09:00.838 INFO:tasks.workunit.client.0.smithi042.stdout:removed './229/33' 2019-12-05T05:09:00.839 INFO:tasks.workunit.client.0.smithi042.stdout:removed './229/23' 2019-12-05T05:09:00.839 INFO:tasks.workunit.client.0.smithi042.stdout:removed './229/35' 2019-12-05T05:09:00.839 INFO:tasks.workunit.client.0.smithi042.stdout:removed './229/26' 2019-12-05T05:09:00.839 INFO:tasks.workunit.client.0.smithi042.stdout:removed './229/37' 2019-12-05T05:09:00.839 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './229' 2019-12-05T05:09:00.839 INFO:tasks.workunit.client.0.smithi042.stdout:removed './230/21' 2019-12-05T05:09:00.839 INFO:tasks.workunit.client.0.smithi042.stdout:removed './230/27' 2019-12-05T05:09:00.840 INFO:tasks.workunit.client.0.smithi042.stdout:removed './230/31' 2019-12-05T05:09:00.840 INFO:tasks.workunit.client.0.smithi042.stdout:removed './230/36' 2019-12-05T05:09:00.841 INFO:tasks.workunit.client.0.smithi042.stdout:removed './230/20' 2019-12-05T05:09:00.842 INFO:tasks.workunit.client.0.smithi042.stdout:removed './230/39' 2019-12-05T05:09:00.842 INFO:tasks.workunit.client.0.smithi042.stdout:removed './230/34' 2019-12-05T05:09:00.842 INFO:tasks.workunit.client.0.smithi042.stdout:removed './230/38' 2019-12-05T05:09:00.842 INFO:tasks.workunit.client.0.smithi042.stdout:removed './230/28' 2019-12-05T05:09:00.844 INFO:tasks.workunit.client.0.smithi042.stdout:removed './230/24' 2019-12-05T05:09:00.844 INFO:tasks.workunit.client.0.smithi042.stdout:removed './230/29' 2019-12-05T05:09:00.845 INFO:tasks.workunit.client.0.smithi042.stdout:removed './230/32' 2019-12-05T05:09:00.845 INFO:tasks.workunit.client.0.smithi042.stdout:removed './230/22' 2019-12-05T05:09:00.845 INFO:tasks.workunit.client.0.smithi042.stdout:removed './230/25' 2019-12-05T05:09:00.845 INFO:tasks.workunit.client.0.smithi042.stdout:removed './230/30' 2019-12-05T05:09:00.846 INFO:tasks.workunit.client.0.smithi042.stdout:removed './230/33' 2019-12-05T05:09:00.846 INFO:tasks.workunit.client.0.smithi042.stdout:removed './230/23' 2019-12-05T05:09:00.846 INFO:tasks.workunit.client.0.smithi042.stdout:removed './230/35' 2019-12-05T05:09:00.846 INFO:tasks.workunit.client.0.smithi042.stdout:removed './230/26' 2019-12-05T05:09:00.846 INFO:tasks.workunit.client.0.smithi042.stdout:removed './230/37' 2019-12-05T05:09:00.846 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './230' 2019-12-05T05:09:00.846 INFO:tasks.workunit.client.0.smithi042.stdout:removed './231/21' 2019-12-05T05:09:00.847 INFO:tasks.workunit.client.0.smithi042.stdout:removed './231/27' 2019-12-05T05:09:00.847 INFO:tasks.workunit.client.0.smithi042.stdout:removed './231/31' 2019-12-05T05:09:00.847 INFO:tasks.workunit.client.0.smithi042.stdout:removed './231/36' 2019-12-05T05:09:00.847 INFO:tasks.workunit.client.0.smithi042.stdout:removed './231/20' 2019-12-05T05:09:00.847 INFO:tasks.workunit.client.0.smithi042.stdout:removed './231/39' 2019-12-05T05:09:00.847 INFO:tasks.workunit.client.0.smithi042.stdout:removed './231/34' 2019-12-05T05:09:00.847 INFO:tasks.workunit.client.0.smithi042.stdout:removed './231/38' 2019-12-05T05:09:00.848 INFO:tasks.workunit.client.0.smithi042.stdout:removed './231/28' 2019-12-05T05:09:00.848 INFO:tasks.workunit.client.0.smithi042.stdout:removed './231/24' 2019-12-05T05:09:00.848 INFO:tasks.workunit.client.0.smithi042.stdout:removed './231/29' 2019-12-05T05:09:00.848 INFO:tasks.workunit.client.0.smithi042.stdout:removed './231/32' 2019-12-05T05:09:00.848 INFO:tasks.workunit.client.0.smithi042.stdout:removed './231/22' 2019-12-05T05:09:00.848 INFO:tasks.workunit.client.0.smithi042.stdout:removed './231/25' 2019-12-05T05:09:00.848 INFO:tasks.workunit.client.0.smithi042.stdout:removed './231/30' 2019-12-05T05:09:00.848 INFO:tasks.workunit.client.0.smithi042.stdout:removed './231/33' 2019-12-05T05:09:00.849 INFO:tasks.workunit.client.0.smithi042.stdout:removed './231/23' 2019-12-05T05:09:00.849 INFO:tasks.workunit.client.0.smithi042.stdout:removed './231/35' 2019-12-05T05:09:00.849 INFO:tasks.workunit.client.0.smithi042.stdout:removed './231/26' 2019-12-05T05:09:00.849 INFO:tasks.workunit.client.0.smithi042.stdout:removed './231/37' 2019-12-05T05:09:00.849 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './231' 2019-12-05T05:09:00.849 INFO:tasks.workunit.client.0.smithi042.stdout:removed './232/21' 2019-12-05T05:09:00.849 INFO:tasks.workunit.client.0.smithi042.stdout:removed './232/27' 2019-12-05T05:09:00.850 INFO:tasks.workunit.client.0.smithi042.stdout:removed './232/31' 2019-12-05T05:09:00.850 INFO:tasks.workunit.client.0.smithi042.stdout:removed './232/36' 2019-12-05T05:09:00.850 INFO:tasks.workunit.client.0.smithi042.stdout:removed './232/20' 2019-12-05T05:09:00.850 INFO:tasks.workunit.client.0.smithi042.stdout:removed './232/39' 2019-12-05T05:09:00.850 INFO:tasks.workunit.client.0.smithi042.stdout:removed './232/34' 2019-12-05T05:09:00.851 INFO:tasks.workunit.client.0.smithi042.stdout:removed './232/38' 2019-12-05T05:09:00.851 INFO:tasks.workunit.client.0.smithi042.stdout:removed './232/28' 2019-12-05T05:09:00.852 INFO:tasks.workunit.client.0.smithi042.stdout:removed './232/24' 2019-12-05T05:09:00.852 INFO:tasks.workunit.client.0.smithi042.stdout:removed './232/29' 2019-12-05T05:09:00.852 INFO:tasks.workunit.client.0.smithi042.stdout:removed './232/32' 2019-12-05T05:09:00.852 INFO:tasks.workunit.client.0.smithi042.stdout:removed './232/22' 2019-12-05T05:09:00.852 INFO:tasks.workunit.client.0.smithi042.stdout:removed './232/25' 2019-12-05T05:09:00.852 INFO:tasks.workunit.client.0.smithi042.stdout:removed './232/30' 2019-12-05T05:09:00.852 INFO:tasks.workunit.client.0.smithi042.stdout:removed './232/33' 2019-12-05T05:09:00.853 INFO:tasks.workunit.client.0.smithi042.stdout:removed './232/23' 2019-12-05T05:09:00.853 INFO:tasks.workunit.client.0.smithi042.stdout:removed './232/35' 2019-12-05T05:09:00.853 INFO:tasks.workunit.client.0.smithi042.stdout:removed './232/26' 2019-12-05T05:09:00.853 INFO:tasks.workunit.client.0.smithi042.stdout:removed './232/37' 2019-12-05T05:09:00.854 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './232' 2019-12-05T05:09:00.854 INFO:tasks.workunit.client.0.smithi042.stdout:removed './233/21' 2019-12-05T05:09:00.854 INFO:tasks.workunit.client.0.smithi042.stdout:removed './233/27' 2019-12-05T05:09:00.854 INFO:tasks.workunit.client.0.smithi042.stdout:removed './233/31' 2019-12-05T05:09:00.854 INFO:tasks.workunit.client.0.smithi042.stdout:removed './233/36' 2019-12-05T05:09:00.854 INFO:tasks.workunit.client.0.smithi042.stdout:removed './233/20' 2019-12-05T05:09:00.855 INFO:tasks.workunit.client.0.smithi042.stdout:removed './233/39' 2019-12-05T05:09:00.855 INFO:tasks.workunit.client.0.smithi042.stdout:removed './233/34' 2019-12-05T05:09:00.855 INFO:tasks.workunit.client.0.smithi042.stdout:removed './233/38' 2019-12-05T05:09:00.856 INFO:tasks.workunit.client.0.smithi042.stdout:removed './233/28' 2019-12-05T05:09:00.856 INFO:tasks.workunit.client.0.smithi042.stdout:removed './233/24' 2019-12-05T05:09:00.856 INFO:tasks.workunit.client.0.smithi042.stdout:removed './233/29' 2019-12-05T05:09:00.856 INFO:tasks.workunit.client.0.smithi042.stdout:removed './233/32' 2019-12-05T05:09:00.857 INFO:tasks.workunit.client.0.smithi042.stdout:removed './233/22' 2019-12-05T05:09:00.857 INFO:tasks.workunit.client.0.smithi042.stdout:removed './233/25' 2019-12-05T05:09:00.857 INFO:tasks.workunit.client.0.smithi042.stdout:removed './233/30' 2019-12-05T05:09:00.857 INFO:tasks.workunit.client.0.smithi042.stdout:removed './233/33' 2019-12-05T05:09:00.857 INFO:tasks.workunit.client.0.smithi042.stdout:removed './233/23' 2019-12-05T05:09:00.857 INFO:tasks.workunit.client.0.smithi042.stdout:removed './233/35' 2019-12-05T05:09:00.858 INFO:tasks.workunit.client.0.smithi042.stdout:removed './233/26' 2019-12-05T05:09:00.858 INFO:tasks.workunit.client.0.smithi042.stdout:removed './233/37' 2019-12-05T05:09:00.858 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './233' 2019-12-05T05:09:00.858 INFO:tasks.workunit.client.0.smithi042.stdout:removed './234/21' 2019-12-05T05:09:00.858 INFO:tasks.workunit.client.0.smithi042.stdout:removed './234/27' 2019-12-05T05:09:00.858 INFO:tasks.workunit.client.0.smithi042.stdout:removed './234/31' 2019-12-05T05:09:00.858 INFO:tasks.workunit.client.0.smithi042.stdout:removed './234/36' 2019-12-05T05:09:00.859 INFO:tasks.workunit.client.0.smithi042.stdout:removed './234/20' 2019-12-05T05:09:00.859 INFO:tasks.workunit.client.0.smithi042.stdout:removed './234/39' 2019-12-05T05:09:00.859 INFO:tasks.workunit.client.0.smithi042.stdout:removed './234/34' 2019-12-05T05:09:00.859 INFO:tasks.workunit.client.0.smithi042.stdout:removed './234/38' 2019-12-05T05:09:00.859 INFO:tasks.workunit.client.0.smithi042.stdout:removed './234/28' 2019-12-05T05:09:00.859 INFO:tasks.workunit.client.0.smithi042.stdout:removed './234/24' 2019-12-05T05:09:00.859 INFO:tasks.workunit.client.0.smithi042.stdout:removed './234/29' 2019-12-05T05:09:00.859 INFO:tasks.workunit.client.0.smithi042.stdout:removed './234/32' 2019-12-05T05:09:00.860 INFO:tasks.workunit.client.0.smithi042.stdout:removed './234/22' 2019-12-05T05:09:00.860 INFO:tasks.workunit.client.0.smithi042.stdout:removed './234/25' 2019-12-05T05:09:00.860 INFO:tasks.workunit.client.0.smithi042.stdout:removed './234/30' 2019-12-05T05:09:00.860 INFO:tasks.workunit.client.0.smithi042.stdout:removed './234/33' 2019-12-05T05:09:00.860 INFO:tasks.workunit.client.0.smithi042.stdout:removed './234/23' 2019-12-05T05:09:00.860 INFO:tasks.workunit.client.0.smithi042.stdout:removed './234/35' 2019-12-05T05:09:00.860 INFO:tasks.workunit.client.0.smithi042.stdout:removed './234/26' 2019-12-05T05:09:00.861 INFO:tasks.workunit.client.0.smithi042.stdout:removed './234/37' 2019-12-05T05:09:00.861 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './234' 2019-12-05T05:09:00.861 INFO:tasks.workunit.client.0.smithi042.stdout:removed './235/21' 2019-12-05T05:09:00.861 INFO:tasks.workunit.client.0.smithi042.stdout:removed './235/27' 2019-12-05T05:09:00.861 INFO:tasks.workunit.client.0.smithi042.stdout:removed './235/31' 2019-12-05T05:09:00.862 INFO:tasks.workunit.client.0.smithi042.stdout:removed './235/36' 2019-12-05T05:09:00.862 INFO:tasks.workunit.client.0.smithi042.stdout:removed './235/20' 2019-12-05T05:09:00.862 INFO:tasks.workunit.client.0.smithi042.stdout:removed './235/39' 2019-12-05T05:09:00.862 INFO:tasks.workunit.client.0.smithi042.stdout:removed './235/34' 2019-12-05T05:09:00.863 INFO:tasks.workunit.client.0.smithi042.stdout:removed './235/38' 2019-12-05T05:09:00.863 INFO:tasks.workunit.client.0.smithi042.stdout:removed './235/28' 2019-12-05T05:09:00.863 INFO:tasks.workunit.client.0.smithi042.stdout:removed './235/24' 2019-12-05T05:09:00.863 INFO:tasks.workunit.client.0.smithi042.stdout:removed './235/29' 2019-12-05T05:09:00.864 INFO:tasks.workunit.client.0.smithi042.stdout:removed './235/32' 2019-12-05T05:09:00.864 INFO:tasks.workunit.client.0.smithi042.stdout:removed './235/22' 2019-12-05T05:09:00.864 INFO:tasks.workunit.client.0.smithi042.stdout:removed './235/25' 2019-12-05T05:09:00.864 INFO:tasks.workunit.client.0.smithi042.stdout:removed './235/30' 2019-12-05T05:09:00.865 INFO:tasks.workunit.client.0.smithi042.stdout:removed './235/33' 2019-12-05T05:09:00.865 INFO:tasks.workunit.client.0.smithi042.stdout:removed './235/23' 2019-12-05T05:09:00.865 INFO:tasks.workunit.client.0.smithi042.stdout:removed './235/35' 2019-12-05T05:09:00.865 INFO:tasks.workunit.client.0.smithi042.stdout:removed './235/26' 2019-12-05T05:09:00.866 INFO:tasks.workunit.client.0.smithi042.stdout:removed './235/37' 2019-12-05T05:09:00.866 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './235' 2019-12-05T05:09:00.866 INFO:tasks.workunit.client.0.smithi042.stdout:removed './236/21' 2019-12-05T05:09:00.866 INFO:tasks.workunit.client.0.smithi042.stdout:removed './236/27' 2019-12-05T05:09:00.867 INFO:tasks.workunit.client.0.smithi042.stdout:removed './236/31' 2019-12-05T05:09:00.867 INFO:tasks.workunit.client.0.smithi042.stdout:removed './236/36' 2019-12-05T05:09:00.867 INFO:tasks.workunit.client.0.smithi042.stdout:removed './236/20' 2019-12-05T05:09:00.867 INFO:tasks.workunit.client.0.smithi042.stdout:removed './236/39' 2019-12-05T05:09:00.867 INFO:tasks.workunit.client.0.smithi042.stdout:removed './236/34' 2019-12-05T05:09:00.867 INFO:tasks.workunit.client.0.smithi042.stdout:removed './236/38' 2019-12-05T05:09:00.867 INFO:tasks.workunit.client.0.smithi042.stdout:removed './236/28' 2019-12-05T05:09:00.868 INFO:tasks.workunit.client.0.smithi042.stdout:removed './236/24' 2019-12-05T05:09:00.868 INFO:tasks.workunit.client.0.smithi042.stdout:removed './236/29' 2019-12-05T05:09:00.868 INFO:tasks.workunit.client.0.smithi042.stdout:removed './236/32' 2019-12-05T05:09:04.935 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:09:04.941 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:09:05.794 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:09:05.880 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:05.871+0000 7f5859c98700 1 Processor -- start 2019-12-05T05:09:05.880 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:05.871+0000 7f5859c98700 1 -- start start 2019-12-05T05:09:05.880 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:05.875+0000 7f5859c98700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f585408df70 0x7f585409f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:09:05.881 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:05.875+0000 7f5859c98700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f585409f820 0x7f585414efd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:09:05.881 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:05.875+0000 7f5859c98700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f585409fc50 con 0x7f58540a7730 2019-12-05T05:09:05.881 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:05.875+0000 7f5859c98700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58540a0af0 con 0x7f585409f820 2019-12-05T05:09:05.881 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:05.875+0000 7f5859c98700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f585408e5a0 con 0x7f585408df70 2019-12-05T05:09:05.881 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:05.875+0000 7f58537fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f585408df70 0x7f585409f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:09:05.881 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:05.875+0000 7f58537fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f585408df70 0x7f585409f2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:56468/0 (socket says 172.21.15.42:56468) 2019-12-05T05:09:05.882 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:05.875+0000 7f58537fe700 1 -- 172.21.15.42:0/1668567766 learned_addr learned my addr 172.21.15.42:0/1668567766 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:09:05.882 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:05.875+0000 7f5858a36700 1 --2- 172.21.15.42:0/1668567766 >> v2:172.21.15.40:3300/0 conn(0x7f585409f820 0x7f585414efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:09:05.882 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:05.875+0000 7f58537fe700 1 -- 172.21.15.42:0/1668567766 >> v2:172.21.15.40:3300/0 conn(0x7f585409f820 msgr2=0x7f585414efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:09:05.882 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:05.875+0000 7f58537fe700 1 --2- 172.21.15.42:0/1668567766 >> v2:172.21.15.40:3300/0 conn(0x7f585409f820 0x7f585414efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:09:05.882 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:05.875+0000 7f58537fe700 1 -- 172.21.15.42:0/1668567766 >> v1:172.21.15.40:6789/0 conn(0x7f58540a7730 legacy=0x7f585407e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:09:05.882 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:05.875+0000 7f58537fe700 1 -- 172.21.15.42:0/1668567766 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58540a0700 con 0x7f585408df70 2019-12-05T05:09:05.882 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:05.875+0000 7f58537fe700 1 --2- 172.21.15.42:0/1668567766 >> v2:172.21.15.42:3300/0 conn(0x7f585408df70 0x7f585409f2d0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f5848002f30 tx=0x7f584800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:05.883 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:05.875+0000 7f58537fe700 1 --2- 172.21.15.42:0/1668567766 >> v2:172.21.15.42:3300/0 conn(0x7f585408df70 0x7f585409f2d0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f5848002f30 tx=0x7f584800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:05.883 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:05.875+0000 7f58537fe700 1 --2- 172.21.15.42:0/1668567766 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f585408df70 0x7f585409f2d0 secure :-1 s=READY pgs=325 cs=0 l=1 rx=0x7f5848002f30 tx=0x7f584800fd30).ready entity=mon.1 client_cookie=2bb2a8520dc74dd3 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:09:05.883 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:05.875+0000 7f58537fe700 1 --2- 172.21.15.42:0/1668567766 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f585408df70 0x7f585409f2d0 secure :-1 s=THROTTLE_DONE pgs=325 cs=0 l=1 rx=0x7f5848002f30 tx=0x7f584800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:05.883 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:05.875+0000 7f5852ffd700 1 -- 172.21.15.42:0/1668567766 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5848008c90 con 0x7f585408df70 2019-12-05T05:09:05.883 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:05.875+0000 7f58537fe700 1 --2- 172.21.15.42:0/1668567766 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f585408df70 0x7f585409f2d0 secure :-1 s=THROTTLE_DONE pgs=325 cs=0 l=1 rx=0x7f5848002f30 tx=0x7f584800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:05.883 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:05.875+0000 7f58537fe700 1 --2- 172.21.15.42:0/1668567766 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f585408df70 0x7f585409f2d0 secure :-1 s=THROTTLE_DONE pgs=325 cs=0 l=1 rx=0x7f5848002f30 tx=0x7f584800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:05.883 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:05.875+0000 7f5852ffd700 1 -- 172.21.15.42:0/1668567766 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58480088b0 con 0x7f585408df70 2019-12-05T05:09:05.884 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:05.875+0000 7f5852ffd700 1 -- 172.21.15.42:0/1668567766 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5848011ad0 con 0x7f585408df70 2019-12-05T05:09:05.884 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:05.875+0000 7f5859c98700 1 -- 172.21.15.42:0/1668567766 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f585408df70 msgr2=0x7f585409f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:09:05.884 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:05.875+0000 7f5859c98700 1 --2- 172.21.15.42:0/1668567766 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f585408df70 0x7f585409f2d0 secure :-1 s=READY pgs=325 cs=0 l=1 rx=0x7f5848002f30 tx=0x7f584800fd30).stop 2019-12-05T05:09:05.884 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:05.875+0000 7f5859c98700 1 -- 172.21.15.42:0/1668567766 shutdown_connections 2019-12-05T05:09:05.884 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:05.875+0000 7f5859c98700 1 --2- 172.21.15.42:0/1668567766 >> v2:172.21.15.40:3300/0 conn(0x7f585409f820 0x7f585414efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:09:05.884 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:05.875+0000 7f5859c98700 1 --2- 172.21.15.42:0/1668567766 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f585408df70 0x7f585409f2d0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:09:05.885 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:05.875+0000 7f5859c98700 1 -- 172.21.15.42:0/1668567766 shutdown_connections 2019-12-05T05:09:05.885 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:05.875+0000 7f5859c98700 1 -- 172.21.15.42:0/1668567766 wait complete. 2019-12-05T05:09:05.885 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:05.875+0000 7f5859c98700 1 -- 172.21.15.42:0/1668567766 >> 172.21.15.42:0/1668567766 conn(0x7f58540aabd0 msgr2=0x7f58540a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:09:05.889 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:05.875+0000 7f5859c98700 1 Processor -- start 2019-12-05T05:09:05.889 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:05.875+0000 7f5859c98700 1 -- start start 2019-12-05T05:09:05.889 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:05.875+0000 7f5859c98700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f585412be70 0x7f585412e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:09:05.889 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:05.875+0000 7f5859c98700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f585407c230 con 0x7f5854116350 2019-12-05T05:09:05.889 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:05.875+0000 7f5859c98700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5854118d10 con 0x7f585407ca70 2019-12-05T05:09:05.889 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:05.875+0000 7f5859c98700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f585407c570 con 0x7f585412be70 2019-12-05T05:09:05.891 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:05.875+0000 7f5858a36700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f585412be70 0x7f585412e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:09:05.891 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:05.875+0000 7f58537fe700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f585407ca70 0x7f585412a660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:56510/0 (socket says 172.21.15.42:56510) 2019-12-05T05:09:05.891 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:05.875+0000 7f58537fe700 1 -- 172.21.15.42:0/1834385468 learned_addr learned my addr 172.21.15.42:0/1834385468 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:09:05.892 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:05.875+0000 7f5858a36700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f585412be70 0x7f585412e2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:56476/0 (socket says 172.21.15.42:56476) 2019-12-05T05:09:05.892 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:05.875+0000 7f5858a36700 1 -- 172.21.15.42:0/1834385468 >> v1:172.21.15.42:6789/0 conn(0x7f585407ca70 legacy=0x7f585412a660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:09:05.892 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:05.875+0000 7f5850ff9700 1 -- 172.21.15.42:0/1834385468 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3490280730 0 0) 0x7f585407c230 con 0x7f5854116350 2019-12-05T05:09:05.892 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:05.875+0000 7f5858a36700 1 -- 172.21.15.42:0/1834385468 >> v1:172.21.15.40:6789/0 conn(0x7f5854116350 legacy=0x7f585411b670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:09:05.892 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:05.875+0000 7f5858a36700 1 -- 172.21.15.42:0/1834385468 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f585411ac40 con 0x7f585412be70 2019-12-05T05:09:05.892 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:05.875+0000 7f5858a36700 1 --2- 172.21.15.42:0/1834385468 >> v2:172.21.15.42:3300/0 conn(0x7f585412be70 0x7f585412e2b0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f5848007f10 tx=0x7f584800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:05.892 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:05.875+0000 7f5858a36700 1 --2- 172.21.15.42:0/1834385468 >> v2:172.21.15.42:3300/0 conn(0x7f585412be70 0x7f585412e2b0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f5848007f10 tx=0x7f584800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:05.893 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:05.875+0000 7f5858a36700 1 --2- 172.21.15.42:0/1834385468 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f585412be70 0x7f585412e2b0 secure :-1 s=READY pgs=327 cs=0 l=1 rx=0x7f5848007f10 tx=0x7f584800fd30).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:09:05.893 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:05.879+0000 7f5858a36700 1 --2- 172.21.15.42:0/1834385468 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f585412be70 0x7f585412e2b0 secure :-1 s=THROTTLE_DONE pgs=327 cs=0 l=1 rx=0x7f5848007f10 tx=0x7f584800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:05.893 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:05.879+0000 7f5850ff9700 1 -- 172.21.15.42:0/1834385468 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f584c005ca0 con 0x7f585412be70 2019-12-05T05:09:05.893 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:05.879+0000 7f5858a36700 1 --2- 172.21.15.42:0/1834385468 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f585412be70 0x7f585412e2b0 secure :-1 s=THROTTLE_DONE pgs=327 cs=0 l=1 rx=0x7f5848007f10 tx=0x7f584800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:05.893 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:05.879+0000 7f5858a36700 1 --2- 172.21.15.42:0/1834385468 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f585412be70 0x7f585412e2b0 secure :-1 s=THROTTLE_DONE pgs=327 cs=0 l=1 rx=0x7f5848007f10 tx=0x7f584800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:05.895 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:05.879+0000 7f5850ff9700 1 -- 172.21.15.42:0/1834385468 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f584c0058c0 con 0x7f585412be70 2019-12-05T05:09:05.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:05.879+0000 7f5850ff9700 1 -- 172.21.15.42:0/1834385468 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f584c012a60 con 0x7f585412be70 2019-12-05T05:09:05.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:05.879+0000 7f5859c98700 1 -- 172.21.15.42:0/1834385468 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f585407cea0 con 0x7f585412be70 2019-12-05T05:09:05.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:05.879+0000 7f5859c98700 1 -- 172.21.15.42:0/1834385468 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f585412e800 con 0x7f585412be70 2019-12-05T05:09:05.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:05.879+0000 7f5858a36700 1 --2- 172.21.15.42:0/1834385468 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f585412be70 0x7f585412e2b0 secure :-1 s=THROTTLE_DONE pgs=327 cs=0 l=1 rx=0x7f5848007f10 tx=0x7f584800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:05.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:05.879+0000 7f5850ff9700 1 -- 172.21.15.42:0/1834385468 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f584c00c3f0 con 0x7f585412be70 2019-12-05T05:09:05.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:05.879+0000 7f5858a36700 1 --2- 172.21.15.42:0/1834385468 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f585412be70 0x7f585412e2b0 secure :-1 s=THROTTLE_DONE pgs=327 cs=0 l=1 rx=0x7f5848007f10 tx=0x7f584800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:05.897 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:05.879+0000 7f5850ff9700 1 --2- 172.21.15.42:0/1834385468 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f583403ac00 0x7f583403d110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:09:05.897 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:05.879+0000 7f5850ff9700 1 -- 172.21.15.42:0/1834385468 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f584c04b2f0 con 0x7f585412be70 2019-12-05T05:09:05.900 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:05.879+0000 7f5853fff700 1 --2- 172.21.15.42:0/1834385468 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f583403ac00 0x7f583403d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:09:05.901 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:05.879+0000 7f5853fff700 1 --2- 172.21.15.42:0/1834385468 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f583403ac00 0x7f583403d110 crc :-1 s=READY pgs=150 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:09:05.901 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:05.879+0000 7f5859c98700 1 -- 172.21.15.42:0/1834385468 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f585411b3a0 con 0x7f585412be70 2019-12-05T05:09:05.902 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:05.883+0000 7f5858a36700 1 --2- 172.21.15.42:0/1834385468 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f585412be70 0x7f585412e2b0 secure :-1 s=THROTTLE_DONE pgs=327 cs=0 l=1 rx=0x7f5848007f10 tx=0x7f584800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:05.902 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:05.883+0000 7f5850ff9700 1 -- 172.21.15.42:0/1834385468 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f584c032330 con 0x7f585412be70 2019-12-05T05:09:06.053 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:06.047+0000 7f5859c98700 1 -- 172.21.15.42:0/1834385468 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f585411af90 con 0x7f585412be70 2019-12-05T05:09:06.054 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:06.051+0000 7f5858a36700 1 --2- 172.21.15.42:0/1834385468 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f585412be70 0x7f585412e2b0 secure :-1 s=THROTTLE_DONE pgs=327 cs=0 l=1 rx=0x7f5848007f10 tx=0x7f584800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:06.055 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:06.051+0000 7f5850ff9700 1 -- 172.21.15.42:0/1834385468 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+2102 (secure 0 0 0) 0x7f584c032590 con 0x7f585412be70 2019-12-05T05:09:06.056 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 29 2019-12-05T05:09:06.057 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:09:06.057 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4978,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/3650564018","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3650564018},{"type":"v1","addr":"172.21.15.42:6835","nonce":3650564018}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":29}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:07:12.263209+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4831},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4831":{"gid":4831,"name":"b","rank":0,"incarnation":24,"state":"up:active","state_seq":39,"addr":"172.21.15.40:6833/90248709","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":90248709},{"type":"v1","addr":"172.21.15.40:6833","nonce":90248709}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:09:06.059 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:06.055+0000 7f5859c98700 1 -- 172.21.15.42:0/1834385468 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f583403ac00 msgr2=0x7f583403d110 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:09:06.059 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:06.055+0000 7f5859c98700 1 --2- 172.21.15.42:0/1834385468 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f583403ac00 0x7f583403d110 crc :-1 s=READY pgs=150 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:09:06.059 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:06.055+0000 7f5859c98700 1 -- 172.21.15.42:0/1834385468 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f585412be70 msgr2=0x7f585412e2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:09:06.060 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:06.055+0000 7f5859c98700 1 --2- 172.21.15.42:0/1834385468 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f585412be70 0x7f585412e2b0 secure :-1 s=READY pgs=327 cs=0 l=1 rx=0x7f5848007f10 tx=0x7f584800fd30).stop 2019-12-05T05:09:06.060 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:06.055+0000 7f5859c98700 1 -- 172.21.15.42:0/1834385468 shutdown_connections 2019-12-05T05:09:06.060 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:06.055+0000 7f5859c98700 1 --2- 172.21.15.42:0/1834385468 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f583403ac00 0x7f583403d110 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:09:06.060 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:06.055+0000 7f5859c98700 1 --2- 172.21.15.42:0/1834385468 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f585412be70 0x7f585412e2b0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:09:06.060 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:06.055+0000 7f5859c98700 1 -- 172.21.15.42:0/1834385468 shutdown_connections 2019-12-05T05:09:06.060 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:06.055+0000 7f5859c98700 1 -- 172.21.15.42:0/1834385468 wait complete. 2019-12-05T05:09:06.060 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:06.055+0000 7f5859c98700 1 -- 172.21.15.42:0/1834385468 >> 172.21.15.42:0/1834385468 conn(0x7f58540a97d0 msgr2=0x7f58540596f0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:09:06.071 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.b (rank=0) 2019-12-05T05:09:06.071 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2019-12-05T05:09:06.071 INFO:teuthology.orchestra.run:waiting for 300 2019-12-05T05:09:06.085 INFO:tasks.ceph.mds.b:Stopped 2019-12-05T05:09:06.086 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:09:06.180 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:06.167+0000 7fd8fe36e700 1 Processor -- start 2019-12-05T05:09:06.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:06.167+0000 7fd8fe36e700 1 -- start start 2019-12-05T05:09:06.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:06.167+0000 7fd8fe36e700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd8f80a7740 0x7fd8f808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:09:06.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:06.167+0000 7fd8fe36e700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8f80a0710 con 0x7fd8f809fae0 2019-12-05T05:09:06.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:06.167+0000 7fd8fe36e700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8f80a9ba0 con 0x7fd8f807e720 2019-12-05T05:09:06.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:06.167+0000 7fd8fe36e700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8f80a0b00 con 0x7fd8f80a7740 2019-12-05T05:09:06.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:06.167+0000 7fd8f7fff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fd8f807e720 0x7fd8f809f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:56522/0 (socket says 172.21.15.42:56522) 2019-12-05T05:09:06.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:06.167+0000 7fd8f7fff700 1 -- 172.21.15.42:0/147066147 learned_addr learned my addr 172.21.15.42:0/147066147 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:09:06.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:06.167+0000 7fd8fc90b700 1 --2- 172.21.15.42:0/147066147 >> v2:172.21.15.40:3300/0 conn(0x7fd8f80a7740 0x7fd8f808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:09:06.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:06.171+0000 7fd8f77fe700 1 -- 172.21.15.42:0/147066147 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1778591905 0 0) 0x7fd8f80a9ba0 con 0x7fd8f807e720 2019-12-05T05:09:06.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:06.171+0000 7fd8f77fe700 1 -- 172.21.15.42:0/147066147 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8e40030b0 con 0x7fd8f807e720 2019-12-05T05:09:06.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:06.171+0000 7fd8fc90b700 1 -- 172.21.15.42:0/147066147 >> v1:172.21.15.42:6789/0 conn(0x7fd8f807e720 legacy=0x7fd8f809f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:09:06.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:06.171+0000 7fd8fc90b700 1 -- 172.21.15.42:0/147066147 >> v1:172.21.15.40:6789/0 conn(0x7fd8f809fae0 legacy=0x7fd8f814efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:09:06.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:06.171+0000 7fd8fc90b700 1 -- 172.21.15.42:0/147066147 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8f807eb50 con 0x7fd8f80a7740 2019-12-05T05:09:06.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:06.171+0000 7fd8fc90b700 1 --2- 172.21.15.42:0/147066147 >> v2:172.21.15.40:3300/0 conn(0x7fd8f80a7740 0x7fd8f808df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fd8e8008790 tx=0x7fd8e800fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:06.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:06.171+0000 7fd8fc90b700 1 --2- 172.21.15.42:0/147066147 >> v2:172.21.15.40:3300/0 conn(0x7fd8f80a7740 0x7fd8f808df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fd8e8008790 tx=0x7fd8e800fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:06.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:06.171+0000 7fd8fc90b700 1 --2- 172.21.15.42:0/147066147 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd8f80a7740 0x7fd8f808df80 secure :-1 s=READY pgs=257 cs=0 l=1 rx=0x7fd8e8008790 tx=0x7fd8e800fbf0).ready entity=mon.0 client_cookie=ac8e39dda8bbd0b7 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:09:06.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:06.171+0000 7fd8fc90b700 1 --2- 172.21.15.42:0/147066147 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd8f80a7740 0x7fd8f808df80 secure :-1 s=THROTTLE_DONE pgs=257 cs=0 l=1 rx=0x7fd8e8008790 tx=0x7fd8e800fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:06.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:06.171+0000 7fd8f77fe700 1 -- 172.21.15.42:0/147066147 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd8e8011270 con 0x7fd8f80a7740 2019-12-05T05:09:06.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:06.171+0000 7fd8fc90b700 1 --2- 172.21.15.42:0/147066147 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd8f80a7740 0x7fd8f808df80 secure :-1 s=THROTTLE_DONE pgs=257 cs=0 l=1 rx=0x7fd8e8008790 tx=0x7fd8e800fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:06.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:06.171+0000 7fd8fc90b700 1 --2- 172.21.15.42:0/147066147 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd8f80a7740 0x7fd8f808df80 secure :-1 s=THROTTLE_DONE pgs=257 cs=0 l=1 rx=0x7fd8e8008790 tx=0x7fd8e800fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:06.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:06.171+0000 7fd8f77fe700 1 -- 172.21.15.42:0/147066147 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8e8008bc0 con 0x7fd8f80a7740 2019-12-05T05:09:06.184 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:06.171+0000 7fd8f77fe700 1 -- 172.21.15.42:0/147066147 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd8e8011850 con 0x7fd8f80a7740 2019-12-05T05:09:06.184 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:06.171+0000 7fd8fe36e700 1 -- 172.21.15.42:0/147066147 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd8f80a7740 msgr2=0x7fd8f808df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:09:06.184 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:06.171+0000 7fd8fe36e700 1 --2- 172.21.15.42:0/147066147 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd8f80a7740 0x7fd8f808df80 secure :-1 s=READY pgs=257 cs=0 l=1 rx=0x7fd8e8008790 tx=0x7fd8e800fbf0).stop 2019-12-05T05:09:06.184 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:06.171+0000 7fd8fe36e700 1 -- 172.21.15.42:0/147066147 shutdown_connections 2019-12-05T05:09:06.185 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:06.171+0000 7fd8fe36e700 1 --2- 172.21.15.42:0/147066147 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd8f80a7740 0x7fd8f808df80 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:09:06.185 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:06.171+0000 7fd8fe36e700 1 -- 172.21.15.42:0/147066147 shutdown_connections 2019-12-05T05:09:06.185 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:06.171+0000 7fd8fe36e700 1 -- 172.21.15.42:0/147066147 wait complete. 2019-12-05T05:09:06.185 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:06.171+0000 7fd8fe36e700 1 -- 172.21.15.42:0/147066147 >> 172.21.15.42:0/147066147 conn(0x7fd8f80aabe0 msgr2=0x7fd8f80a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:09:06.186 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:06.175+0000 7fd8fe36e700 1 Processor -- start 2019-12-05T05:09:06.186 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:06.175+0000 7fd8fe36e700 1 -- start start 2019-12-05T05:09:06.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:06.175+0000 7fd8fe36e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd8f807ddc0 0x7fd8f8124240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:09:06.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:06.175+0000 7fd8fe36e700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd8f80a1b10 0x7fd8f8137650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:09:06.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:06.175+0000 7fd8fe36e700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8f811afb0 con 0x7fd8f80a16e0 2019-12-05T05:09:06.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:06.175+0000 7fd8fe36e700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8f80a2050 con 0x7fd8f807ddc0 2019-12-05T05:09:06.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:06.175+0000 7fd8fe36e700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8f811b280 con 0x7fd8f80a1b10 2019-12-05T05:09:06.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:06.175+0000 7fd8fc90b700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd8f807ddc0 0x7fd8f8124240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:09:06.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:06.175+0000 7fd8fc90b700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd8f807ddc0 0x7fd8f8124240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:56488/0 (socket says 172.21.15.42:56488) 2019-12-05T05:09:06.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:06.175+0000 7fd8fc90b700 1 -- 172.21.15.42:0/211913396 learned_addr learned my addr 172.21.15.42:0/211913396 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:09:06.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:06.175+0000 7fd8fd10c700 1 --2- 172.21.15.42:0/211913396 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd8f80a1b10 0x7fd8f8137650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:09:06.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:06.175+0000 7fd8fc90b700 1 -- 172.21.15.42:0/211913396 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd8f80a1b10 msgr2=0x7fd8f8137650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:09:06.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:06.175+0000 7fd8fc90b700 1 --2- 172.21.15.42:0/211913396 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd8f80a1b10 0x7fd8f8137650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:09:06.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:06.175+0000 7fd8fc90b700 1 -- 172.21.15.42:0/211913396 >> v1:172.21.15.42:6789/0 conn(0x7fd8f80a16e0 legacy=0x7fd8f811b520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:09:06.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:06.175+0000 7fd8fc90b700 1 -- 172.21.15.42:0/211913396 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8f8124790 con 0x7fd8f807ddc0 2019-12-05T05:09:06.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:06.175+0000 7fd8fc90b700 1 --2- 172.21.15.42:0/211913396 >> v2:172.21.15.42:3300/0 conn(0x7fd8f807ddc0 0x7fd8f8124240 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fd8f80a2210 tx=0x7fd8e8008ea0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:06.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:06.179+0000 7fd8fc90b700 1 --2- 172.21.15.42:0/211913396 >> v2:172.21.15.42:3300/0 conn(0x7fd8f807ddc0 0x7fd8f8124240 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fd8f80a2210 tx=0x7fd8e8008ea0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:06.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:06.179+0000 7fd8fc90b700 1 --2- 172.21.15.42:0/211913396 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd8f807ddc0 0x7fd8f8124240 secure :-1 s=READY pgs=330 cs=0 l=1 rx=0x7fd8f80a2210 tx=0x7fd8e8008ea0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:09:06.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:06.179+0000 7fd8fc90b700 1 --2- 172.21.15.42:0/211913396 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd8f807ddc0 0x7fd8f8124240 secure :-1 s=THROTTLE_DONE pgs=330 cs=0 l=1 rx=0x7fd8f80a2210 tx=0x7fd8e8008ea0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:06.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:06.179+0000 7fd8f57fa700 1 -- 172.21.15.42:0/211913396 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd8e8007340 con 0x7fd8f807ddc0 2019-12-05T05:09:06.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:06.179+0000 7fd8fc90b700 1 --2- 172.21.15.42:0/211913396 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd8f807ddc0 0x7fd8f8124240 secure :-1 s=THROTTLE_DONE pgs=330 cs=0 l=1 rx=0x7fd8f80a2210 tx=0x7fd8e8008ea0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:06.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:06.179+0000 7fd8f57fa700 1 -- 172.21.15.42:0/211913396 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8e8011470 con 0x7fd8f807ddc0 2019-12-05T05:09:06.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:06.179+0000 7fd8fc90b700 1 --2- 172.21.15.42:0/211913396 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd8f807ddc0 0x7fd8f8124240 secure :-1 s=THROTTLE_DONE pgs=330 cs=0 l=1 rx=0x7fd8f80a2210 tx=0x7fd8e8008ea0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:06.193 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:06.179+0000 7fd8fe36e700 1 -- 172.21.15.42:0/211913396 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8f811bd20 con 0x7fd8f807ddc0 2019-12-05T05:09:06.193 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:06.179+0000 7fd8f57fa700 1 -- 172.21.15.42:0/211913396 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd8e8005540 con 0x7fd8f807ddc0 2019-12-05T05:09:06.193 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:06.179+0000 7fd8fe36e700 1 -- 172.21.15.42:0/211913396 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8f8118dd0 con 0x7fd8f807ddc0 2019-12-05T05:09:06.193 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:06.179+0000 7fd8fc90b700 1 --2- 172.21.15.42:0/211913396 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd8f807ddc0 0x7fd8f8124240 secure :-1 s=THROTTLE_DONE pgs=330 cs=0 l=1 rx=0x7fd8f80a2210 tx=0x7fd8e8008ea0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:06.193 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:06.179+0000 7fd8f57fa700 1 -- 172.21.15.42:0/211913396 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fd8e8005820 con 0x7fd8f807ddc0 2019-12-05T05:09:06.193 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:06.179+0000 7fd8fc90b700 1 --2- 172.21.15.42:0/211913396 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd8f807ddc0 0x7fd8f8124240 secure :-1 s=THROTTLE_DONE pgs=330 cs=0 l=1 rx=0x7fd8f80a2210 tx=0x7fd8e8008ea0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:06.193 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:06.179+0000 7fd8f57fa700 1 --2- 172.21.15.42:0/211913396 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd8dc03adb0 0x7fd8dc03d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:09:06.194 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:06.179+0000 7fd8f57fa700 1 -- 172.21.15.42:0/211913396 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fd8e804bc00 con 0x7fd8f807ddc0 2019-12-05T05:09:06.194 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:06.179+0000 7fd8f7fff700 1 --2- 172.21.15.42:0/211913396 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd8dc03adb0 0x7fd8dc03d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:09:06.194 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:06.179+0000 7fd8f7fff700 1 --2- 172.21.15.42:0/211913396 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd8dc03adb0 0x7fd8dc03d2c0 crc :-1 s=READY pgs=151 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:09:06.194 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:06.183+0000 7fd8fe36e700 1 -- 172.21.15.42:0/211913396 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8f805d110 con 0x7fd8f807ddc0 2019-12-05T05:09:06.196 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:06.191+0000 7fd8fc90b700 1 --2- 172.21.15.42:0/211913396 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd8f807ddc0 0x7fd8f8124240 secure :-1 s=THROTTLE_DONE pgs=330 cs=0 l=1 rx=0x7fd8f80a2210 tx=0x7fd8e8008ea0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:06.197 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:06.191+0000 7fd8f57fa700 1 -- 172.21.15.42:0/211913396 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fd8e8032c60 con 0x7fd8f807ddc0 2019-12-05T05:09:06.365 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:06.359+0000 7fd8fe36e700 1 -- 172.21.15.42:0/211913396 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd8f805b8c0 con 0x7fd8f807ddc0 2019-12-05T05:09:06.366 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:06.359+0000 7fd8fc90b700 1 --2- 172.21.15.42:0/211913396 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd8f807ddc0 0x7fd8f8124240 secure :-1 s=THROTTLE_DONE pgs=330 cs=0 l=1 rx=0x7fd8f80a2210 tx=0x7fd8e8008ea0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:06.366 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:06.359+0000 7fd8f57fa700 1 -- 172.21.15.42:0/211913396 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+2102 (secure 0 0 0) 0x7fd8e8032ec0 con 0x7fd8f807ddc0 2019-12-05T05:09:06.368 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 29 2019-12-05T05:09:06.368 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:09:06.368 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4978,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/3650564018","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3650564018},{"type":"v1","addr":"172.21.15.42:6835","nonce":3650564018}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":29}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:07:12.263209+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4831},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4831":{"gid":4831,"name":"b","rank":0,"incarnation":24,"state":"up:active","state_seq":39,"addr":"172.21.15.40:6833/90248709","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":90248709},{"type":"v1","addr":"172.21.15.40:6833","nonce":90248709}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:09:06.373 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:06.367+0000 7fd8fe36e700 1 -- 172.21.15.42:0/211913396 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd8dc03adb0 msgr2=0x7fd8dc03d2c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:09:06.373 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:06.367+0000 7fd8fe36e700 1 --2- 172.21.15.42:0/211913396 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd8dc03adb0 0x7fd8dc03d2c0 crc :-1 s=READY pgs=151 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:09:06.373 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:06.367+0000 7fd8fe36e700 1 -- 172.21.15.42:0/211913396 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd8f807ddc0 msgr2=0x7fd8f8124240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:09:06.373 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:06.367+0000 7fd8fe36e700 1 --2- 172.21.15.42:0/211913396 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd8f807ddc0 0x7fd8f8124240 secure :-1 s=READY pgs=330 cs=0 l=1 rx=0x7fd8f80a2210 tx=0x7fd8e8008ea0).stop 2019-12-05T05:09:06.374 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:06.367+0000 7fd8fe36e700 1 -- 172.21.15.42:0/211913396 shutdown_connections 2019-12-05T05:09:06.374 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:06.367+0000 7fd8fe36e700 1 --2- 172.21.15.42:0/211913396 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd8dc03adb0 0x7fd8dc03d2c0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:09:06.374 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:06.367+0000 7fd8fe36e700 1 --2- 172.21.15.42:0/211913396 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd8f80a1b10 0x7fd8f8137650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:09:06.374 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:06.367+0000 7fd8fe36e700 1 --2- 172.21.15.42:0/211913396 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd8f807ddc0 0x7fd8f8124240 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:09:06.375 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:06.367+0000 7fd8fe36e700 1 -- 172.21.15.42:0/211913396 shutdown_connections 2019-12-05T05:09:06.375 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:06.367+0000 7fd8fe36e700 1 -- 172.21.15.42:0/211913396 wait complete. 2019-12-05T05:09:06.375 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:06.367+0000 7fd8fe36e700 1 -- 172.21.15.42:0/211913396 >> 172.21.15.42:0/211913396 conn(0x7fd8f80a8410 msgr2=0x7fd8f80a9640 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:09:06.386 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T05:09:08.387 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:09:08.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:08.483+0000 7fbcdc654700 1 Processor -- start 2019-12-05T05:09:08.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:08.483+0000 7fbcdc654700 1 -- start start 2019-12-05T05:09:08.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:08.487+0000 7fbcdc654700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbcd408df80 0x7fbcd409f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:09:08.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:08.487+0000 7fbcdc654700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fbcd409f830 0x7fbcd414efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:09:08.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:08.487+0000 7fbcdc654700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcd409fc60 con 0x7fbcd40a7740 2019-12-05T05:09:08.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:08.487+0000 7fbcdc654700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbcd40a0b00 con 0x7fbcd409f830 2019-12-05T05:09:08.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:08.487+0000 7fbcdc654700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcd408e5b0 con 0x7fbcd408df80 2019-12-05T05:09:08.496 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:08.487+0000 7fbcdb3f2700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fbcd409f830 0x7fbcd414efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:09:08.496 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:08.487+0000 7fbcdb3f2700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fbcd409f830 0x7fbcd414efe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:56510/0 (socket says 172.21.15.42:56510) 2019-12-05T05:09:08.496 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:08.487+0000 7fbcdb3f2700 1 -- 172.21.15.42:0/1131998351 learned_addr learned my addr 172.21.15.42:0/1131998351 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:09:08.496 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:08.487+0000 7fbcda3f0700 1 --2- 172.21.15.42:0/1131998351 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbcd408df80 0x7fbcd409f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:09:08.497 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:08.487+0000 7fbcdb3f2700 1 -- 172.21.15.42:0/1131998351 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbcd408df80 msgr2=0x7fbcd409f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:09:08.497 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:08.487+0000 7fbcdb3f2700 1 --2- 172.21.15.42:0/1131998351 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbcd408df80 0x7fbcd409f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:09:08.497 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:08.487+0000 7fbcdb3f2700 1 -- 172.21.15.42:0/1131998351 >> v1:172.21.15.40:6789/0 conn(0x7fbcd40a7740 legacy=0x7fbcd407e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:09:08.497 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:08.487+0000 7fbcdb3f2700 1 -- 172.21.15.42:0/1131998351 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcd40a0710 con 0x7fbcd409f830 2019-12-05T05:09:08.497 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:08.487+0000 7fbcdb3f2700 1 --2- 172.21.15.42:0/1131998351 >> v2:172.21.15.42:3300/0 conn(0x7fbcd409f830 0x7fbcd414efe0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fbccc008b10 tx=0x7fbccc0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:08.497 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:08.487+0000 7fbcdb3f2700 1 --2- 172.21.15.42:0/1131998351 >> v2:172.21.15.42:3300/0 conn(0x7fbcd409f830 0x7fbcd414efe0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fbccc008b10 tx=0x7fbccc0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:08.497 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:08.487+0000 7fbcdb3f2700 1 --2- 172.21.15.42:0/1131998351 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbcd409f830 0x7fbcd414efe0 secure :-1 s=READY pgs=331 cs=0 l=1 rx=0x7fbccc008b10 tx=0x7fbccc0100b0).ready entity=mon.1 client_cookie=35120a3f92762e97 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:09:08.498 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:08.487+0000 7fbcdb3f2700 1 --2- 172.21.15.42:0/1131998351 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbcd409f830 0x7fbcd414efe0 secure :-1 s=THROTTLE_DONE pgs=331 cs=0 l=1 rx=0x7fbccc008b10 tx=0x7fbccc0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:08.498 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:08.487+0000 7fbcdb3f2700 1 --2- 172.21.15.42:0/1131998351 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbcd409f830 0x7fbcd414efe0 secure :-1 s=THROTTLE_DONE pgs=331 cs=0 l=1 rx=0x7fbccc008b10 tx=0x7fbccc0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:08.498 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:08.487+0000 7fbcd9bef700 1 -- 172.21.15.42:0/1131998351 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbccc010da0 con 0x7fbcd409f830 2019-12-05T05:09:08.498 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:08.487+0000 7fbcdb3f2700 1 --2- 172.21.15.42:0/1131998351 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbcd409f830 0x7fbcd414efe0 secure :-1 s=THROTTLE_DONE pgs=331 cs=0 l=1 rx=0x7fbccc008b10 tx=0x7fbccc0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:08.498 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:08.487+0000 7fbcd9bef700 1 -- 172.21.15.42:0/1131998351 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbccc0109c0 con 0x7fbcd409f830 2019-12-05T05:09:08.498 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:08.487+0000 7fbcd9bef700 1 -- 172.21.15.42:0/1131998351 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbccc0114a0 con 0x7fbcd409f830 2019-12-05T05:09:08.498 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:08.487+0000 7fbcdc654700 1 -- 172.21.15.42:0/1131998351 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbcd409f830 msgr2=0x7fbcd414efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:09:08.499 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:08.487+0000 7fbcdc654700 1 --2- 172.21.15.42:0/1131998351 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbcd409f830 0x7fbcd414efe0 secure :-1 s=READY pgs=331 cs=0 l=1 rx=0x7fbccc008b10 tx=0x7fbccc0100b0).stop 2019-12-05T05:09:08.499 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:08.487+0000 7fbcdc654700 1 -- 172.21.15.42:0/1131998351 shutdown_connections 2019-12-05T05:09:08.499 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:08.487+0000 7fbcdc654700 1 --2- 172.21.15.42:0/1131998351 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbcd409f830 0x7fbcd414efe0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:09:08.499 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:08.487+0000 7fbcdc654700 1 --2- 172.21.15.42:0/1131998351 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbcd408df80 0x7fbcd409f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:09:08.499 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:08.487+0000 7fbcdc654700 1 -- 172.21.15.42:0/1131998351 shutdown_connections 2019-12-05T05:09:08.499 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:08.487+0000 7fbcdc654700 1 -- 172.21.15.42:0/1131998351 wait complete. 2019-12-05T05:09:08.499 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:08.487+0000 7fbcdc654700 1 -- 172.21.15.42:0/1131998351 >> 172.21.15.42:0/1131998351 conn(0x7fbcd40aabe0 msgr2=0x7fbcd40a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:09:08.503 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:08.491+0000 7fbcdc654700 1 Processor -- start 2019-12-05T05:09:08.503 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:08.491+0000 7fbcdc654700 1 -- start start 2019-12-05T05:09:08.503 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:08.495+0000 7fbcdc654700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fbcd4116500 0x7fbcd4117c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:09:08.504 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:08.495+0000 7fbcdc654700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbcd4118150 0x7fbcd4133600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:09:08.504 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:08.495+0000 7fbcdc654700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fbcd4133b50 0x7fbcd408eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:09:08.504 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:08.495+0000 7fbcdc654700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbcd4118580 con 0x7fbcd4133b50 2019-12-05T05:09:08.504 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:08.495+0000 7fbcdc654700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbcd4133f80 con 0x7fbcd4116500 2019-12-05T05:09:08.504 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:08.495+0000 7fbcdc654700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcd4134170 con 0x7fbcd4118150 2019-12-05T05:09:08.504 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:08.495+0000 7fbcdabf1700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fbcd4116500 0x7fbcd4117c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:09:08.504 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:08.495+0000 7fbcdabf1700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fbcd4116500 0x7fbcd4117c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:56512/0 (socket says 172.21.15.42:56512) 2019-12-05T05:09:08.505 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:08.495+0000 7fbcdabf1700 1 -- 172.21.15.42:0/4243419094 learned_addr learned my addr 172.21.15.42:0/4243419094 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:09:08.505 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:08.495+0000 7fbcda3f0700 1 --2- 172.21.15.42:0/4243419094 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbcd4118150 0x7fbcd4133600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:09:08.505 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:08.495+0000 7fbcdb3f2700 1 --2- 172.21.15.42:0/4243419094 >> v2:172.21.15.40:3300/0 conn(0x7fbcd4133b50 0x7fbcd408eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:09:08.505 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:08.495+0000 7fbcdabf1700 1 -- 172.21.15.42:0/4243419094 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbcd4118150 msgr2=0x7fbcd4133600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:09:08.505 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:08.495+0000 7fbcdabf1700 1 --2- 172.21.15.42:0/4243419094 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbcd4118150 0x7fbcd4133600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:09:08.505 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:08.495+0000 7fbcdabf1700 1 -- 172.21.15.42:0/4243419094 >> v2:172.21.15.40:3300/0 conn(0x7fbcd4133b50 msgr2=0x7fbcd408eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:09:08.506 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:08.495+0000 7fbcdabf1700 1 --2- 172.21.15.42:0/4243419094 >> v2:172.21.15.40:3300/0 conn(0x7fbcd4133b50 0x7fbcd408eb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:09:08.506 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:08.495+0000 7fbcdabf1700 1 -- 172.21.15.42:0/4243419094 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcd41175f0 con 0x7fbcd4116500 2019-12-05T05:09:08.506 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:08.495+0000 7fbcdabf1700 1 --2- 172.21.15.42:0/4243419094 >> v2:172.21.15.42:3300/0 conn(0x7fbcd4116500 0x7fbcd4117c00 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fbcd0000e90 tx=0x7fbcd0010a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:08.506 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:08.495+0000 7fbcdabf1700 1 --2- 172.21.15.42:0/4243419094 >> v2:172.21.15.42:3300/0 conn(0x7fbcd4116500 0x7fbcd4117c00 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fbcd0000e90 tx=0x7fbcd0010a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:08.507 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:08.495+0000 7fbcdabf1700 1 --2- 172.21.15.42:0/4243419094 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbcd4116500 0x7fbcd4117c00 secure :-1 s=READY pgs=332 cs=0 l=1 rx=0x7fbcd0000e90 tx=0x7fbcd0010a20).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:09:08.508 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:08.495+0000 7fbcdabf1700 1 --2- 172.21.15.42:0/4243419094 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbcd4116500 0x7fbcd4117c00 secure :-1 s=THROTTLE_DONE pgs=332 cs=0 l=1 rx=0x7fbcd0000e90 tx=0x7fbcd0010a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:08.508 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:08.495+0000 7fbcdabf1700 1 --2- 172.21.15.42:0/4243419094 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbcd4116500 0x7fbcd4117c00 secure :-1 s=THROTTLE_DONE pgs=332 cs=0 l=1 rx=0x7fbcd0000e90 tx=0x7fbcd0010a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:08.508 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:08.495+0000 7fbccb7fe700 1 -- 172.21.15.42:0/4243419094 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbcd000caa0 con 0x7fbcd4116500 2019-12-05T05:09:08.508 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:08.495+0000 7fbcdabf1700 1 --2- 172.21.15.42:0/4243419094 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbcd4116500 0x7fbcd4117c00 secure :-1 s=THROTTLE_DONE pgs=332 cs=0 l=1 rx=0x7fbcd0000e90 tx=0x7fbcd0010a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:08.508 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:08.495+0000 7fbccb7fe700 1 -- 172.21.15.42:0/4243419094 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcd000c6c0 con 0x7fbcd4116500 2019-12-05T05:09:08.508 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:08.495+0000 7fbcdc654700 1 -- 172.21.15.42:0/4243419094 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcd408f0c0 con 0x7fbcd4116500 2019-12-05T05:09:08.508 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:08.495+0000 7fbccb7fe700 1 -- 172.21.15.42:0/4243419094 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbcd0012290 con 0x7fbcd4116500 2019-12-05T05:09:08.509 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:08.495+0000 7fbcdc654700 1 -- 172.21.15.42:0/4243419094 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcd408f5c0 con 0x7fbcd4116500 2019-12-05T05:09:08.509 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:08.495+0000 7fbcdabf1700 1 --2- 172.21.15.42:0/4243419094 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbcd4116500 0x7fbcd4117c00 secure :-1 s=THROTTLE_DONE pgs=332 cs=0 l=1 rx=0x7fbcd0000e90 tx=0x7fbcd0010a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:08.509 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:08.495+0000 7fbccb7fe700 1 -- 172.21.15.42:0/4243419094 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fbcd00125a0 con 0x7fbcd4116500 2019-12-05T05:09:08.509 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:08.499+0000 7fbcdabf1700 1 --2- 172.21.15.42:0/4243419094 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbcd4116500 0x7fbcd4117c00 secure :-1 s=THROTTLE_DONE pgs=332 cs=0 l=1 rx=0x7fbcd0000e90 tx=0x7fbcd0010a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:08.509 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:08.499+0000 7fbccb7fe700 1 --2- 172.21.15.42:0/4243419094 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbcbc03ae90 0x7fbcbc03d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:09:08.509 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:08.499+0000 7fbccb7fe700 1 -- 172.21.15.42:0/4243419094 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fbcd004b370 con 0x7fbcd4116500 2019-12-05T05:09:08.509 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:08.499+0000 7fbcda3f0700 1 --2- 172.21.15.42:0/4243419094 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbcbc03ae90 0x7fbcbc03d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:09:08.510 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:08.499+0000 7fbcda3f0700 1 --2- 172.21.15.42:0/4243419094 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbcbc03ae90 0x7fbcbc03d3a0 crc :-1 s=READY pgs=152 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:09:08.510 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:08.499+0000 7fbcdc654700 1 -- 172.21.15.42:0/4243419094 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbcd405d110 con 0x7fbcd4116500 2019-12-05T05:09:08.510 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:08.503+0000 7fbcdabf1700 1 --2- 172.21.15.42:0/4243419094 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbcd4116500 0x7fbcd4117c00 secure :-1 s=THROTTLE_DONE pgs=332 cs=0 l=1 rx=0x7fbcd0000e90 tx=0x7fbcd0010a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:08.511 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:08.503+0000 7fbccb7fe700 1 -- 172.21.15.42:0/4243419094 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fbcd00323f0 con 0x7fbcd4116500 2019-12-05T05:09:08.675 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:08.671+0000 7fbcdc654700 1 -- 172.21.15.42:0/4243419094 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbcd405b8c0 con 0x7fbcd4116500 2019-12-05T05:09:08.676 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:08.671+0000 7fbcdabf1700 1 --2- 172.21.15.42:0/4243419094 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbcd4116500 0x7fbcd4117c00 secure :-1 s=THROTTLE_DONE pgs=332 cs=0 l=1 rx=0x7fbcd0000e90 tx=0x7fbcd0010a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:08.677 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:08.671+0000 7fbccb7fe700 1 -- 172.21.15.42:0/4243419094 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+2102 (secure 0 0 0) 0x7fbcd0032650 con 0x7fbcd4116500 2019-12-05T05:09:08.677 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 29 2019-12-05T05:09:08.677 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:09:08.677 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4978,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/3650564018","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3650564018},{"type":"v1","addr":"172.21.15.42:6835","nonce":3650564018}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":29}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:07:12.263209+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4831},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4831":{"gid":4831,"name":"b","rank":0,"incarnation":24,"state":"up:active","state_seq":39,"addr":"172.21.15.40:6833/90248709","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":90248709},{"type":"v1","addr":"172.21.15.40:6833","nonce":90248709}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:09:08.679 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:08.675+0000 7fbcdc654700 1 -- 172.21.15.42:0/4243419094 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbcbc03ae90 msgr2=0x7fbcbc03d3a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:09:08.679 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:08.675+0000 7fbcdc654700 1 --2- 172.21.15.42:0/4243419094 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbcbc03ae90 0x7fbcbc03d3a0 crc :-1 s=READY pgs=152 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:09:08.680 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:08.675+0000 7fbcdc654700 1 -- 172.21.15.42:0/4243419094 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbcd4116500 msgr2=0x7fbcd4117c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:09:08.680 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:08.675+0000 7fbcdc654700 1 --2- 172.21.15.42:0/4243419094 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbcd4116500 0x7fbcd4117c00 secure :-1 s=READY pgs=332 cs=0 l=1 rx=0x7fbcd0000e90 tx=0x7fbcd0010a20).stop 2019-12-05T05:09:08.680 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:08.675+0000 7fbcdc654700 1 -- 172.21.15.42:0/4243419094 shutdown_connections 2019-12-05T05:09:08.680 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:08.675+0000 7fbcdc654700 1 --2- 172.21.15.42:0/4243419094 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbcbc03ae90 0x7fbcbc03d3a0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:09:08.680 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:08.675+0000 7fbcdc654700 1 --2- 172.21.15.42:0/4243419094 >> v2:172.21.15.40:3300/0 conn(0x7fbcd4133b50 0x7fbcd408eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:09:08.681 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:08.675+0000 7fbcdc654700 1 --2- 172.21.15.42:0/4243419094 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbcd4116500 0x7fbcd4117c00 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:09:08.681 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:08.675+0000 7fbcdc654700 1 --2- 172.21.15.42:0/4243419094 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbcd4118150 0x7fbcd4133600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:09:08.681 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:08.675+0000 7fbcdc654700 1 -- 172.21.15.42:0/4243419094 shutdown_connections 2019-12-05T05:09:08.681 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:08.675+0000 7fbcdc654700 1 -- 172.21.15.42:0/4243419094 wait complete. 2019-12-05T05:09:08.681 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:08.675+0000 7fbcdc654700 1 -- 172.21.15.42:0/4243419094 >> 172.21.15.42:0/4243419094 conn(0x7fbcd40a9a90 msgr2=0x7fbcd409f220 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:09:08.689 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T05:09:10.691 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:09:10.794 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:10.783+0000 7fe445e55700 1 Processor -- start 2019-12-05T05:09:10.794 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:10.783+0000 7fe445e55700 1 -- start start 2019-12-05T05:09:10.794 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:10.783+0000 7fe445e55700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe44008df80 0x7fe44009f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:09:10.795 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:10.783+0000 7fe445e55700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe44009fc60 con 0x7fe4400a7740 2019-12-05T05:09:10.795 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:10.783+0000 7fe445e55700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4400a0710 con 0x7fe44009f830 2019-12-05T05:09:10.795 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:10.783+0000 7fe445e55700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4400a0b00 con 0x7fe44008df80 2019-12-05T05:09:10.795 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:10.783+0000 7fe444bf3700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fe44009f830 0x7fe44014efe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:56570/0 (socket says 172.21.15.42:56570) 2019-12-05T05:09:10.795 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:10.783+0000 7fe444bf3700 1 -- 172.21.15.42:0/3271302717 learned_addr learned my addr 172.21.15.42:0/3271302717 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:09:10.795 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:10.783+0000 7fe43f7fe700 1 --2- 172.21.15.42:0/3271302717 >> v2:172.21.15.40:3300/0 conn(0x7fe44008df80 0x7fe44009f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:09:10.795 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:10.783+0000 7fe43effd700 1 -- 172.21.15.42:0/3271302717 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2737962758 0 0) 0x7fe4400a0710 con 0x7fe44009f830 2019-12-05T05:09:10.796 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:10.787+0000 7fe43effd700 1 -- 172.21.15.42:0/3271302717 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4280030b0 con 0x7fe44009f830 2019-12-05T05:09:10.796 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:10.787+0000 7fe43effd700 1 -- 172.21.15.42:0/3271302717 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2995567759 0 0) 0x7fe44009fc60 con 0x7fe4400a7740 2019-12-05T05:09:10.796 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:10.787+0000 7fe43effd700 1 -- 172.21.15.42:0/3271302717 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4400a0710 con 0x7fe4400a7740 2019-12-05T05:09:10.796 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:10.787+0000 7fe43f7fe700 1 -- 172.21.15.42:0/3271302717 >> v1:172.21.15.42:6789/0 conn(0x7fe44009f830 legacy=0x7fe44014efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:09:10.811 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:10.787+0000 7fe43f7fe700 1 -- 172.21.15.42:0/3271302717 >> v1:172.21.15.40:6789/0 conn(0x7fe4400a7740 legacy=0x7fe44007e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:09:10.811 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:10.787+0000 7fe43f7fe700 1 -- 172.21.15.42:0/3271302717 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe44007ef20 con 0x7fe44008df80 2019-12-05T05:09:10.811 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:10.787+0000 7fe43f7fe700 1 --2- 172.21.15.42:0/3271302717 >> v2:172.21.15.40:3300/0 conn(0x7fe44008df80 0x7fe44009f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fe434008790 tx=0x7fe43400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:10.811 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:10.787+0000 7fe43f7fe700 1 --2- 172.21.15.42:0/3271302717 >> v2:172.21.15.40:3300/0 conn(0x7fe44008df80 0x7fe44009f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fe434008790 tx=0x7fe43400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:10.812 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:10.787+0000 7fe43f7fe700 1 --2- 172.21.15.42:0/3271302717 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe44008df80 0x7fe44009f2e0 secure :-1 s=READY pgs=261 cs=0 l=1 rx=0x7fe434008790 tx=0x7fe43400fbf0).ready entity=mon.0 client_cookie=d0d9b088444647ad server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:09:10.812 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:10.787+0000 7fe43f7fe700 1 --2- 172.21.15.42:0/3271302717 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe44008df80 0x7fe44009f2e0 secure :-1 s=THROTTLE_DONE pgs=261 cs=0 l=1 rx=0x7fe434008790 tx=0x7fe43400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:10.812 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:10.787+0000 7fe43f7fe700 1 --2- 172.21.15.42:0/3271302717 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe44008df80 0x7fe44009f2e0 secure :-1 s=THROTTLE_DONE pgs=261 cs=0 l=1 rx=0x7fe434008790 tx=0x7fe43400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:10.812 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:10.787+0000 7fe43effd700 1 -- 172.21.15.42:0/3271302717 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe434011270 con 0x7fe44008df80 2019-12-05T05:09:10.812 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:10.787+0000 7fe43f7fe700 1 --2- 172.21.15.42:0/3271302717 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe44008df80 0x7fe44009f2e0 secure :-1 s=THROTTLE_DONE pgs=261 cs=0 l=1 rx=0x7fe434008790 tx=0x7fe43400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:10.812 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:10.787+0000 7fe43effd700 1 -- 172.21.15.42:0/3271302717 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe434008bc0 con 0x7fe44008df80 2019-12-05T05:09:10.813 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:10.787+0000 7fe43effd700 1 -- 172.21.15.42:0/3271302717 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe434011850 con 0x7fe44008df80 2019-12-05T05:09:10.813 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:10.787+0000 7fe445e55700 1 -- 172.21.15.42:0/3271302717 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe44008df80 msgr2=0x7fe44009f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:09:10.813 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:10.787+0000 7fe445e55700 1 --2- 172.21.15.42:0/3271302717 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe44008df80 0x7fe44009f2e0 secure :-1 s=READY pgs=261 cs=0 l=1 rx=0x7fe434008790 tx=0x7fe43400fbf0).stop 2019-12-05T05:09:10.813 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:10.787+0000 7fe445e55700 1 -- 172.21.15.42:0/3271302717 shutdown_connections 2019-12-05T05:09:10.813 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:10.787+0000 7fe445e55700 1 --2- 172.21.15.42:0/3271302717 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe44008df80 0x7fe44009f2e0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:09:10.813 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:10.787+0000 7fe445e55700 1 -- 172.21.15.42:0/3271302717 shutdown_connections 2019-12-05T05:09:10.813 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:10.787+0000 7fe445e55700 1 -- 172.21.15.42:0/3271302717 wait complete. 2019-12-05T05:09:10.814 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:10.787+0000 7fe445e55700 1 -- 172.21.15.42:0/3271302717 >> 172.21.15.42:0/3271302717 conn(0x7fe4400aabe0 msgr2=0x7fe4400a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:09:10.818 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:10.791+0000 7fe445e55700 1 Processor -- start 2019-12-05T05:09:10.818 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:10.791+0000 7fe445e55700 1 -- start start 2019-12-05T05:09:10.818 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:10.791+0000 7fe445e55700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe4400a62a0 0x7fe44011ad60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:09:10.818 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:10.791+0000 7fe445e55700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe440139680 0x7fe44013bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:09:10.818 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:10.791+0000 7fe445e55700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4400a66d0 con 0x7fe440118300 2019-12-05T05:09:10.818 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:10.791+0000 7fe445e55700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4400a5da0 con 0x7fe4400a62a0 2019-12-05T05:09:10.819 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:10.791+0000 7fe445e55700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4400a5b20 con 0x7fe440139680 2019-12-05T05:09:10.819 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:10.791+0000 7fe43ffff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe4400a62a0 0x7fe44011ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:09:10.819 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:10.791+0000 7fe43ffff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe4400a62a0 0x7fe44011ad60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:56534/0 (socket says 172.21.15.42:56534) 2019-12-05T05:09:10.819 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:10.791+0000 7fe43ffff700 1 -- 172.21.15.42:0/401655355 learned_addr learned my addr 172.21.15.42:0/401655355 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:09:10.819 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:10.791+0000 7fe444bf3700 1 --2- 172.21.15.42:0/401655355 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe440139680 0x7fe44013bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:09:10.819 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:10.795+0000 7fe43ffff700 1 -- 172.21.15.42:0/401655355 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe440139680 msgr2=0x7fe44013bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:09:10.820 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:10.795+0000 7fe43ffff700 1 --2- 172.21.15.42:0/401655355 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe440139680 0x7fe44013bac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:09:10.820 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:10.795+0000 7fe43ffff700 1 -- 172.21.15.42:0/401655355 >> v1:172.21.15.40:6789/0 conn(0x7fe440118300 legacy=0x7fe4400a6b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:09:10.820 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:10.795+0000 7fe43ffff700 1 -- 172.21.15.42:0/401655355 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe44007b540 con 0x7fe4400a62a0 2019-12-05T05:09:10.820 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:10.795+0000 7fe43ffff700 1 --2- 172.21.15.42:0/401655355 >> v2:172.21.15.42:3300/0 conn(0x7fe4400a62a0 0x7fe44011ad60 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fe430000e90 tx=0x7fe4300109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:10.820 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:10.795+0000 7fe43ffff700 1 --2- 172.21.15.42:0/401655355 >> v2:172.21.15.42:3300/0 conn(0x7fe4400a62a0 0x7fe44011ad60 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fe430000e90 tx=0x7fe4300109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:10.820 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:10.795+0000 7fe43ffff700 1 --2- 172.21.15.42:0/401655355 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe4400a62a0 0x7fe44011ad60 secure :-1 s=READY pgs=334 cs=0 l=1 rx=0x7fe430000e90 tx=0x7fe4300109a0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:09:10.821 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:10.795+0000 7fe43ffff700 1 --2- 172.21.15.42:0/401655355 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe4400a62a0 0x7fe44011ad60 secure :-1 s=THROTTLE_DONE pgs=334 cs=0 l=1 rx=0x7fe430000e90 tx=0x7fe4300109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:10.821 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:10.795+0000 7fe43cff9700 1 -- 172.21.15.42:0/401655355 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe43000caa0 con 0x7fe4400a62a0 2019-12-05T05:09:10.821 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:10.795+0000 7fe43ffff700 1 --2- 172.21.15.42:0/401655355 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe4400a62a0 0x7fe44011ad60 secure :-1 s=THROTTLE_DONE pgs=334 cs=0 l=1 rx=0x7fe430000e90 tx=0x7fe4300109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:10.822 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:10.795+0000 7fe43ffff700 1 --2- 172.21.15.42:0/401655355 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe4400a62a0 0x7fe44011ad60 secure :-1 s=THROTTLE_DONE pgs=334 cs=0 l=1 rx=0x7fe430000e90 tx=0x7fe4300109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:10.822 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:10.795+0000 7fe43cff9700 1 -- 172.21.15.42:0/401655355 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe43000c6c0 con 0x7fe4400a62a0 2019-12-05T05:09:10.822 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:10.795+0000 7fe445e55700 1 -- 172.21.15.42:0/401655355 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe440118730 con 0x7fe4400a62a0 2019-12-05T05:09:10.822 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:10.795+0000 7fe43cff9700 1 -- 172.21.15.42:0/401655355 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe43000d6a0 con 0x7fe4400a62a0 2019-12-05T05:09:10.822 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:10.795+0000 7fe445e55700 1 -- 172.21.15.42:0/401655355 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe44011c350 con 0x7fe4400a62a0 2019-12-05T05:09:10.822 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:10.795+0000 7fe43ffff700 1 --2- 172.21.15.42:0/401655355 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe4400a62a0 0x7fe44011ad60 secure :-1 s=THROTTLE_DONE pgs=334 cs=0 l=1 rx=0x7fe430000e90 tx=0x7fe4300109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:10.822 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:10.795+0000 7fe43cff9700 1 -- 172.21.15.42:0/401655355 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fe43000d8a0 con 0x7fe4400a62a0 2019-12-05T05:09:10.823 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:10.795+0000 7fe43ffff700 1 --2- 172.21.15.42:0/401655355 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe4400a62a0 0x7fe44011ad60 secure :-1 s=THROTTLE_DONE pgs=334 cs=0 l=1 rx=0x7fe430000e90 tx=0x7fe4300109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:10.823 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:10.795+0000 7fe43cff9700 1 --2- 172.21.15.42:0/401655355 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe42003ad50 0x7fe42003d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:09:10.823 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:10.795+0000 7fe43cff9700 1 -- 172.21.15.42:0/401655355 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fe43004b2e0 con 0x7fe4400a62a0 2019-12-05T05:09:10.823 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:10.795+0000 7fe43f7fe700 1 --2- 172.21.15.42:0/401655355 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe42003ad50 0x7fe42003d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:09:10.823 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:10.795+0000 7fe43f7fe700 1 --2- 172.21.15.42:0/401655355 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe42003ad50 0x7fe42003d260 crc :-1 s=READY pgs=153 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:09:10.823 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:10.799+0000 7fe445e55700 1 -- 172.21.15.42:0/401655355 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe44005d110 con 0x7fe4400a62a0 2019-12-05T05:09:10.824 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:10.807+0000 7fe43ffff700 1 --2- 172.21.15.42:0/401655355 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe4400a62a0 0x7fe44011ad60 secure :-1 s=THROTTLE_DONE pgs=334 cs=0 l=1 rx=0x7fe430000e90 tx=0x7fe4300109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:10.824 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:10.807+0000 7fe43cff9700 1 -- 172.21.15.42:0/401655355 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fe430032360 con 0x7fe4400a62a0 2019-12-05T05:09:10.981 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:10.975+0000 7fe445e55700 1 -- 172.21.15.42:0/401655355 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe44005b8c0 con 0x7fe4400a62a0 2019-12-05T05:09:10.982 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:10.975+0000 7fe43ffff700 1 --2- 172.21.15.42:0/401655355 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe4400a62a0 0x7fe44011ad60 secure :-1 s=THROTTLE_DONE pgs=334 cs=0 l=1 rx=0x7fe430000e90 tx=0x7fe4300109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:10.983 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:10.975+0000 7fe43cff9700 1 -- 172.21.15.42:0/401655355 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+2102 (secure 0 0 0) 0x7fe4300325c0 con 0x7fe4400a62a0 2019-12-05T05:09:10.983 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 29 2019-12-05T05:09:10.983 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:09:10.984 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4978,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/3650564018","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3650564018},{"type":"v1","addr":"172.21.15.42:6835","nonce":3650564018}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":29}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:07:12.263209+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4831},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4831":{"gid":4831,"name":"b","rank":0,"incarnation":24,"state":"up:active","state_seq":39,"addr":"172.21.15.40:6833/90248709","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":90248709},{"type":"v1","addr":"172.21.15.40:6833","nonce":90248709}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:09:10.986 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:10.979+0000 7fe445e55700 1 -- 172.21.15.42:0/401655355 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe42003ad50 msgr2=0x7fe42003d260 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:09:10.986 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:10.979+0000 7fe445e55700 1 --2- 172.21.15.42:0/401655355 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe42003ad50 0x7fe42003d260 crc :-1 s=READY pgs=153 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:09:10.986 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:10.979+0000 7fe445e55700 1 -- 172.21.15.42:0/401655355 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe4400a62a0 msgr2=0x7fe44011ad60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:09:10.986 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:10.979+0000 7fe445e55700 1 --2- 172.21.15.42:0/401655355 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe4400a62a0 0x7fe44011ad60 secure :-1 s=READY pgs=334 cs=0 l=1 rx=0x7fe430000e90 tx=0x7fe4300109a0).stop 2019-12-05T05:09:10.987 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:10.979+0000 7fe445e55700 1 -- 172.21.15.42:0/401655355 shutdown_connections 2019-12-05T05:09:10.987 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:10.979+0000 7fe445e55700 1 --2- 172.21.15.42:0/401655355 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe42003ad50 0x7fe42003d260 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:09:10.987 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:10.979+0000 7fe445e55700 1 --2- 172.21.15.42:0/401655355 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe440139680 0x7fe44013bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:09:10.987 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:10.979+0000 7fe445e55700 1 --2- 172.21.15.42:0/401655355 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe4400a62a0 0x7fe44011ad60 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:09:10.987 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:10.979+0000 7fe445e55700 1 -- 172.21.15.42:0/401655355 shutdown_connections 2019-12-05T05:09:10.988 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:10.979+0000 7fe445e55700 1 -- 172.21.15.42:0/401655355 wait complete. 2019-12-05T05:09:10.988 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:10.979+0000 7fe445e55700 1 -- 172.21.15.42:0/401655355 >> 172.21.15.42:0/401655355 conn(0x7fe4400a7740 msgr2=0x7fe440059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:09:10.995 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T05:09:12.995 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:09:13.104 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:13.091+0000 7ff49b07b700 1 Processor -- start 2019-12-05T05:09:13.104 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:13.091+0000 7ff49b07b700 1 -- start start 2019-12-05T05:09:13.104 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:13.095+0000 7ff49b07b700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff49408df70 0x7ff49409f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:09:13.105 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:13.095+0000 7ff49b07b700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff49409f820 0x7ff49414efd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:09:13.105 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:13.095+0000 7ff49b07b700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff49409fc50 con 0x7ff4940a7730 2019-12-05T05:09:13.105 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:13.095+0000 7ff49b07b700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4940a0af0 con 0x7ff49408df70 2019-12-05T05:09:13.105 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:13.095+0000 7ff49b07b700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff49408e5a0 con 0x7ff49409f820 2019-12-05T05:09:13.105 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:13.095+0000 7ff498e17700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff49408df70 0x7ff49409f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:09:13.105 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:13.095+0000 7ff498e17700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff49408df70 0x7ff49409f2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:56548/0 (socket says 172.21.15.42:56548) 2019-12-05T05:09:13.106 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:13.095+0000 7ff498e17700 1 -- 172.21.15.42:0/3411861021 learned_addr learned my addr 172.21.15.42:0/3411861021 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:09:13.106 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:13.095+0000 7ff499618700 1 --1- 172.21.15.42:0/3411861021 >> v1:172.21.15.42:6789/0 conn(0x7ff4940a7730 0x7ff49407e710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:56586/0 (socket says 172.21.15.42:56586) 2019-12-05T05:09:13.106 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:13.095+0000 7ff499e19700 1 --2- 172.21.15.42:0/3411861021 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff49409f820 0x7ff49414efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:09:13.106 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:13.095+0000 7ff498e17700 1 -- 172.21.15.42:0/3411861021 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff49409f820 msgr2=0x7ff49414efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:09:13.106 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:13.095+0000 7ff498e17700 1 --2- 172.21.15.42:0/3411861021 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff49409f820 0x7ff49414efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:09:13.107 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:13.095+0000 7ff498e17700 1 -- 172.21.15.42:0/3411861021 >> v1:172.21.15.42:6789/0 conn(0x7ff4940a7730 legacy=0x7ff49407e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:09:13.107 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:13.095+0000 7ff498e17700 1 -- 172.21.15.42:0/3411861021 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4940a0700 con 0x7ff49408df70 2019-12-05T05:09:13.108 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:13.095+0000 7ff487fff700 1 -- 172.21.15.42:0/3411861021 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2393850912 0 0) 0x7ff49409fc50 con 0x7ff4940a7730 2019-12-05T05:09:13.108 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:13.095+0000 7ff498e17700 1 --2- 172.21.15.42:0/3411861021 >> v2:172.21.15.42:3300/0 conn(0x7ff49408df70 0x7ff49409f2d0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7ff488008b10 tx=0x7ff4880100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:13.108 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:13.095+0000 7ff498e17700 1 --2- 172.21.15.42:0/3411861021 >> v2:172.21.15.42:3300/0 conn(0x7ff49408df70 0x7ff49409f2d0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7ff488008b10 tx=0x7ff4880100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:13.109 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:13.095+0000 7ff498e17700 1 --2- 172.21.15.42:0/3411861021 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff49408df70 0x7ff49409f2d0 secure :-1 s=READY pgs=336 cs=0 l=1 rx=0x7ff488008b10 tx=0x7ff4880100b0).ready entity=mon.1 client_cookie=4911d93941524706 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:09:13.109 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:13.095+0000 7ff498e17700 1 --2- 172.21.15.42:0/3411861021 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff49408df70 0x7ff49409f2d0 secure :-1 s=THROTTLE_DONE pgs=336 cs=0 l=1 rx=0x7ff488008b10 tx=0x7ff4880100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:13.109 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:13.095+0000 7ff498e17700 1 --2- 172.21.15.42:0/3411861021 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff49408df70 0x7ff49409f2d0 secure :-1 s=THROTTLE_DONE pgs=336 cs=0 l=1 rx=0x7ff488008b10 tx=0x7ff4880100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:13.109 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:13.095+0000 7ff487fff700 1 -- 172.21.15.42:0/3411861021 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff488010da0 con 0x7ff49408df70 2019-12-05T05:09:13.109 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:13.095+0000 7ff498e17700 1 --2- 172.21.15.42:0/3411861021 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff49408df70 0x7ff49409f2d0 secure :-1 s=THROTTLE_DONE pgs=336 cs=0 l=1 rx=0x7ff488008b10 tx=0x7ff4880100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:13.109 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:13.095+0000 7ff487fff700 1 -- 172.21.15.42:0/3411861021 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4880109c0 con 0x7ff49408df70 2019-12-05T05:09:13.110 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:13.095+0000 7ff487fff700 1 -- 172.21.15.42:0/3411861021 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff4880114a0 con 0x7ff49408df70 2019-12-05T05:09:13.110 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:13.095+0000 7ff49b07b700 1 -- 172.21.15.42:0/3411861021 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff49408df70 msgr2=0x7ff49409f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:09:13.110 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:13.095+0000 7ff49b07b700 1 --2- 172.21.15.42:0/3411861021 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff49408df70 0x7ff49409f2d0 secure :-1 s=READY pgs=336 cs=0 l=1 rx=0x7ff488008b10 tx=0x7ff4880100b0).stop 2019-12-05T05:09:13.110 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:13.095+0000 7ff49b07b700 1 -- 172.21.15.42:0/3411861021 shutdown_connections 2019-12-05T05:09:13.111 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:13.095+0000 7ff49b07b700 1 --2- 172.21.15.42:0/3411861021 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff49409f820 0x7ff49414efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:09:13.111 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:13.095+0000 7ff49b07b700 1 --2- 172.21.15.42:0/3411861021 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff49408df70 0x7ff49409f2d0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:09:13.111 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:13.095+0000 7ff49b07b700 1 -- 172.21.15.42:0/3411861021 shutdown_connections 2019-12-05T05:09:13.112 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:13.095+0000 7ff49b07b700 1 -- 172.21.15.42:0/3411861021 wait complete. 2019-12-05T05:09:13.112 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:13.095+0000 7ff49b07b700 1 -- 172.21.15.42:0/3411861021 >> 172.21.15.42:0/3411861021 conn(0x7ff4940aabd0 msgr2=0x7ff4940a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:09:13.116 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:13.099+0000 7ff49b07b700 1 Processor -- start 2019-12-05T05:09:13.116 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:13.099+0000 7ff49b07b700 1 -- start start 2019-12-05T05:09:13.117 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:13.099+0000 7ff49b07b700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff4941314f0 0x7ff4940a4da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:09:13.117 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:13.099+0000 7ff49b07b700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff494131c20 0x7ff494132050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:09:13.117 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:13.099+0000 7ff49b07b700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff49411aa70 con 0x7ff494123580 2019-12-05T05:09:13.118 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:13.099+0000 7ff49b07b700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4940a5720 con 0x7ff4941314f0 2019-12-05T05:09:13.118 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:13.099+0000 7ff49b07b700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff49411b200 con 0x7ff494131c20 2019-12-05T05:09:13.118 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:13.099+0000 7ff499e19700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff494131c20 0x7ff494132050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:09:13.119 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:13.103+0000 7ff499e19700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff494131c20 0x7ff494132050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:56558/0 (socket says 172.21.15.42:56558) 2019-12-05T05:09:13.119 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:13.103+0000 7ff499e19700 1 -- 172.21.15.42:0/1571944007 learned_addr learned my addr 172.21.15.42:0/1571944007 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:09:13.119 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:13.103+0000 7ff499618700 1 --2- 172.21.15.42:0/1571944007 >> v2:172.21.15.40:3300/0 conn(0x7ff4941314f0 0x7ff4940a4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:09:13.119 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:13.103+0000 7ff499e19700 1 -- 172.21.15.42:0/1571944007 >> v2:172.21.15.40:3300/0 conn(0x7ff4941314f0 msgr2=0x7ff4940a4da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:09:13.119 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:13.103+0000 7ff499e19700 1 --2- 172.21.15.42:0/1571944007 >> v2:172.21.15.40:3300/0 conn(0x7ff4941314f0 0x7ff4940a4da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:09:13.119 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:13.103+0000 7ff499e19700 1 -- 172.21.15.42:0/1571944007 >> v1:172.21.15.40:6789/0 conn(0x7ff494123580 legacy=0x7ff49411b4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:09:13.119 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:13.103+0000 7ff499e19700 1 -- 172.21.15.42:0/1571944007 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4940a52f0 con 0x7ff494131c20 2019-12-05T05:09:13.120 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:13.103+0000 7ff499e19700 1 --2- 172.21.15.42:0/1571944007 >> v2:172.21.15.42:3300/0 conn(0x7ff494131c20 0x7ff494132050 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7ff48c002f30 tx=0x7ff48c010990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:13.120 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:13.103+0000 7ff499e19700 1 --2- 172.21.15.42:0/1571944007 >> v2:172.21.15.42:3300/0 conn(0x7ff494131c20 0x7ff494132050 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7ff48c002f30 tx=0x7ff48c010990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:13.120 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:13.103+0000 7ff499e19700 1 --2- 172.21.15.42:0/1571944007 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff494131c20 0x7ff494132050 secure :-1 s=READY pgs=337 cs=0 l=1 rx=0x7ff48c002f30 tx=0x7ff48c010990).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:09:13.120 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:13.103+0000 7ff499e19700 1 --2- 172.21.15.42:0/1571944007 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff494131c20 0x7ff494132050 secure :-1 s=THROTTLE_DONE pgs=337 cs=0 l=1 rx=0x7ff48c002f30 tx=0x7ff48c010990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:13.120 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:13.103+0000 7ff499e19700 1 --2- 172.21.15.42:0/1571944007 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff494131c20 0x7ff494132050 secure :-1 s=THROTTLE_DONE pgs=337 cs=0 l=1 rx=0x7ff48c002f30 tx=0x7ff48c010990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:13.120 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:13.103+0000 7ff499e19700 1 --2- 172.21.15.42:0/1571944007 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff494131c20 0x7ff494132050 secure :-1 s=THROTTLE_DONE pgs=337 cs=0 l=1 rx=0x7ff48c002f30 tx=0x7ff48c010990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:13.121 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:13.103+0000 7ff485ffb700 1 -- 172.21.15.42:0/1571944007 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff48c005930 con 0x7ff494131c20 2019-12-05T05:09:13.124 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:13.103+0000 7ff485ffb700 1 -- 172.21.15.42:0/1571944007 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff48c007160 con 0x7ff494131c20 2019-12-05T05:09:13.124 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:13.103+0000 7ff49b07b700 1 -- 172.21.15.42:0/1571944007 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff494138940 con 0x7ff494131c20 2019-12-05T05:09:13.124 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:13.103+0000 7ff485ffb700 1 -- 172.21.15.42:0/1571944007 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff48c0060c0 con 0x7ff494131c20 2019-12-05T05:09:13.125 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:13.103+0000 7ff49b07b700 1 -- 172.21.15.42:0/1571944007 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff494138ef0 con 0x7ff494131c20 2019-12-05T05:09:13.125 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:13.103+0000 7ff499e19700 1 --2- 172.21.15.42:0/1571944007 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff494131c20 0x7ff494132050 secure :-1 s=THROTTLE_DONE pgs=337 cs=0 l=1 rx=0x7ff48c002f30 tx=0x7ff48c010990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:13.125 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:13.103+0000 7ff485ffb700 1 -- 172.21.15.42:0/1571944007 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7ff48c0063d0 con 0x7ff494131c20 2019-12-05T05:09:13.125 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:13.103+0000 7ff499e19700 1 --2- 172.21.15.42:0/1571944007 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff494131c20 0x7ff494132050 secure :-1 s=THROTTLE_DONE pgs=337 cs=0 l=1 rx=0x7ff48c002f30 tx=0x7ff48c010990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:13.125 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:13.103+0000 7ff485ffb700 1 --2- 172.21.15.42:0/1571944007 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff47403ae20 0x7ff47403d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:09:13.126 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:13.103+0000 7ff485ffb700 1 -- 172.21.15.42:0/1571944007 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6663+0+0 (secure 0 0 0) 0x7ff48c04ad10 con 0x7ff494131c20 2019-12-05T05:09:13.126 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:13.103+0000 7ff499618700 1 --2- 172.21.15.42:0/1571944007 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff47403ae20 0x7ff47403d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:09:13.126 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:13.103+0000 7ff499618700 1 --2- 172.21.15.42:0/1571944007 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff47403ae20 0x7ff47403d330 crc :-1 s=READY pgs=154 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:09:13.126 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:13.107+0000 7ff49b07b700 1 -- 172.21.15.42:0/1571944007 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff49405d100 con 0x7ff494131c20 2019-12-05T05:09:13.127 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:13.115+0000 7ff499e19700 1 --2- 172.21.15.42:0/1571944007 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff494131c20 0x7ff494132050 secure :-1 s=THROTTLE_DONE pgs=337 cs=0 l=1 rx=0x7ff48c002f30 tx=0x7ff48c010990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:13.127 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:13.115+0000 7ff485ffb700 1 -- 172.21.15.42:0/1571944007 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7ff48c031d90 con 0x7ff494131c20 2019-12-05T05:09:13.293 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:13.287+0000 7ff49b07b700 1 -- 172.21.15.42:0/1571944007 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff49405b8b0 con 0x7ff494131c20 2019-12-05T05:09:13.294 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:13.287+0000 7ff499e19700 1 --2- 172.21.15.42:0/1571944007 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff494131c20 0x7ff494132050 secure :-1 s=THROTTLE_DONE pgs=337 cs=0 l=1 rx=0x7ff48c002f30 tx=0x7ff48c010990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:13.294 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:13.287+0000 7ff485ffb700 1 -- 172.21.15.42:0/1571944007 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+2102 (secure 0 0 0) 0x7ff48c031ff0 con 0x7ff494131c20 2019-12-05T05:09:13.294 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 29 2019-12-05T05:09:13.295 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:09:13.296 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4978,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/3650564018","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3650564018},{"type":"v1","addr":"172.21.15.42:6835","nonce":3650564018}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":29}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:07:12.263209+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4831},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4831":{"gid":4831,"name":"b","rank":0,"incarnation":24,"state":"up:active","state_seq":39,"addr":"172.21.15.40:6833/90248709","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":90248709},{"type":"v1","addr":"172.21.15.40:6833","nonce":90248709}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:09:13.298 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:13.291+0000 7ff49b07b700 1 -- 172.21.15.42:0/1571944007 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff47403ae20 msgr2=0x7ff47403d330 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:09:13.298 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:13.291+0000 7ff49b07b700 1 --2- 172.21.15.42:0/1571944007 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff47403ae20 0x7ff47403d330 crc :-1 s=READY pgs=154 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:09:13.298 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:13.291+0000 7ff49b07b700 1 -- 172.21.15.42:0/1571944007 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff494131c20 msgr2=0x7ff494132050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:09:13.298 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:13.291+0000 7ff49b07b700 1 --2- 172.21.15.42:0/1571944007 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff494131c20 0x7ff494132050 secure :-1 s=READY pgs=337 cs=0 l=1 rx=0x7ff48c002f30 tx=0x7ff48c010990).stop 2019-12-05T05:09:13.298 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:13.291+0000 7ff49b07b700 1 -- 172.21.15.42:0/1571944007 shutdown_connections 2019-12-05T05:09:13.298 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:13.291+0000 7ff49b07b700 1 --2- 172.21.15.42:0/1571944007 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff47403ae20 0x7ff47403d330 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:09:13.299 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:13.291+0000 7ff49b07b700 1 --2- 172.21.15.42:0/1571944007 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff494131c20 0x7ff494132050 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:09:13.299 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:13.291+0000 7ff49b07b700 1 --2- 172.21.15.42:0/1571944007 >> v2:172.21.15.40:3300/0 conn(0x7ff4941314f0 0x7ff4940a4da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:09:13.299 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:13.291+0000 7ff49b07b700 1 -- 172.21.15.42:0/1571944007 shutdown_connections 2019-12-05T05:09:13.299 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:13.291+0000 7ff49b07b700 1 -- 172.21.15.42:0/1571944007 wait complete. 2019-12-05T05:09:13.299 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:13.291+0000 7ff49b07b700 1 -- 172.21.15.42:0/1571944007 >> 172.21.15.42:0/1571944007 conn(0x7ff49408fdd0 msgr2=0x7ff4940596f0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:09:13.307 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T05:09:15.308 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:09:15.416 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:15.406+0000 7f5572f8b700 1 Processor -- start 2019-12-05T05:09:15.416 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:15.406+0000 7f5572f8b700 1 -- start start 2019-12-05T05:09:15.417 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:15.406+0000 7f5572f8b700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f556c0a7740 0x7f556c08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:09:15.417 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:15.406+0000 7f5572f8b700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f556c07e720 0x7f556c07eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:09:15.417 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:15.406+0000 7f5572f8b700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f556c0a0710 con 0x7f556c09f2e0 2019-12-05T05:09:15.417 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:15.406+0000 7f5572f8b700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f556c0a0b00 con 0x7f556c0a7740 2019-12-05T05:09:15.417 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:15.406+0000 7f5572f8b700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f556c08e5b0 con 0x7f556c07e720 2019-12-05T05:09:15.417 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:15.406+0000 7f5571d29700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f556c09f2e0 0x7f556c09f710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:56608/0 (socket says 172.21.15.42:56608) 2019-12-05T05:09:15.418 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:15.406+0000 7f5571d29700 1 -- 172.21.15.42:0/1576470080 learned_addr learned my addr 172.21.15.42:0/1576470080 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:09:15.418 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:15.406+0000 7f5570d27700 1 --2- 172.21.15.42:0/1576470080 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f556c07e720 0x7f556c07eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:09:15.418 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:15.406+0000 7f5571528700 1 --2- 172.21.15.42:0/1576470080 >> v2:172.21.15.40:3300/0 conn(0x7f556c0a7740 0x7f556c08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:09:15.418 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:15.406+0000 7f5563fff700 1 -- 172.21.15.42:0/1576470080 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 895809194 0 0) 0x7f556c0a0710 con 0x7f556c09f2e0 2019-12-05T05:09:15.418 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:15.406+0000 7f5563fff700 1 -- 172.21.15.42:0/1576470080 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55580030b0 con 0x7f556c09f2e0 2019-12-05T05:09:15.418 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:15.406+0000 7f5570d27700 1 -- 172.21.15.42:0/1576470080 >> v2:172.21.15.40:3300/0 conn(0x7f556c0a7740 msgr2=0x7f556c08df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:09:15.418 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:15.406+0000 7f5570d27700 1 --2- 172.21.15.42:0/1576470080 >> v2:172.21.15.40:3300/0 conn(0x7f556c0a7740 0x7f556c08df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:09:15.419 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:15.406+0000 7f5570d27700 1 -- 172.21.15.42:0/1576470080 >> v1:172.21.15.42:6789/0 conn(0x7f556c09f2e0 legacy=0x7f556c09f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:09:15.419 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:15.406+0000 7f5570d27700 1 -- 172.21.15.42:0/1576470080 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f556c07f0a0 con 0x7f556c07e720 2019-12-05T05:09:15.419 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:15.406+0000 7f5571528700 1 --2- 172.21.15.42:0/1576470080 >> v2:172.21.15.40:3300/0 conn(0x7f556c0a7740 0x7f556c08df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:09:15.419 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:15.406+0000 7f5570d27700 1 --2- 172.21.15.42:0/1576470080 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f556c07e720 0x7f556c07eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f555c008790 tx=0x7f555c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:15.419 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:15.406+0000 7f5570d27700 1 --2- 172.21.15.42:0/1576470080 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f556c07e720 0x7f556c07eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f555c008790 tx=0x7f555c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:15.419 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:15.406+0000 7f5570d27700 1 --2- 172.21.15.42:0/1576470080 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f556c07e720 0x7f556c07eb50 secure :-1 s=READY pgs=65 cs=0 l=1 rx=0x7f555c008790 tx=0x7f555c00fc70).ready entity=mon.2 client_cookie=731993b95bfe5e6e server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:09:15.420 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:15.406+0000 7f5570d27700 1 --2- 172.21.15.42:0/1576470080 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f556c07e720 0x7f556c07eb50 secure :-1 s=THROTTLE_DONE pgs=65 cs=0 l=1 rx=0x7f555c008790 tx=0x7f555c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:15.420 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:15.406+0000 7f5563fff700 1 -- 172.21.15.42:0/1576470080 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f555c0112f0 con 0x7f556c07e720 2019-12-05T05:09:15.420 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:15.406+0000 7f5570d27700 1 --2- 172.21.15.42:0/1576470080 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f556c07e720 0x7f556c07eb50 secure :-1 s=THROTTLE_DONE pgs=65 cs=0 l=1 rx=0x7f555c008790 tx=0x7f555c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:15.420 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:15.406+0000 7f5570d27700 1 --2- 172.21.15.42:0/1576470080 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f556c07e720 0x7f556c07eb50 secure :-1 s=THROTTLE_DONE pgs=65 cs=0 l=1 rx=0x7f555c008790 tx=0x7f555c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:15.420 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:15.406+0000 7f5563fff700 1 -- 172.21.15.42:0/1576470080 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f555c008ad0 con 0x7f556c07e720 2019-12-05T05:09:15.420 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:15.406+0000 7f5563fff700 1 -- 172.21.15.42:0/1576470080 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f555c0118d0 con 0x7f556c07e720 2019-12-05T05:09:15.420 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:15.406+0000 7f5572f8b700 1 -- 172.21.15.42:0/1576470080 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f556c07e720 msgr2=0x7f556c07eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:09:15.421 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:15.406+0000 7f5572f8b700 1 --2- 172.21.15.42:0/1576470080 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f556c07e720 0x7f556c07eb50 secure :-1 s=READY pgs=65 cs=0 l=1 rx=0x7f555c008790 tx=0x7f555c00fc70).stop 2019-12-05T05:09:15.421 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:15.410+0000 7f5572f8b700 1 -- 172.21.15.42:0/1576470080 shutdown_connections 2019-12-05T05:09:15.421 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:15.410+0000 7f5572f8b700 1 --2- 172.21.15.42:0/1576470080 >> v2:172.21.15.40:3300/0 conn(0x7f556c0a7740 0x7f556c08df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:09:15.421 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:15.410+0000 7f5572f8b700 1 --2- 172.21.15.42:0/1576470080 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f556c07e720 0x7f556c07eb50 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:09:15.421 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:15.410+0000 7f5572f8b700 1 -- 172.21.15.42:0/1576470080 shutdown_connections 2019-12-05T05:09:15.421 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:15.410+0000 7f5572f8b700 1 -- 172.21.15.42:0/1576470080 wait complete. 2019-12-05T05:09:15.421 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:15.410+0000 7f5572f8b700 1 -- 172.21.15.42:0/1576470080 >> 172.21.15.42:0/1576470080 conn(0x7f556c0aabe0 msgr2=0x7f556c0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:09:15.425 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:15.410+0000 7f5572f8b700 1 Processor -- start 2019-12-05T05:09:15.425 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:15.410+0000 7f5572f8b700 1 -- start start 2019-12-05T05:09:15.425 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:15.414+0000 7f5572f8b700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f556c122a40 0x7f556c119120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:09:15.425 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:15.414+0000 7f5572f8b700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f556c12d470 con 0x7f556c12d040 2019-12-05T05:09:15.425 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:15.414+0000 7f5572f8b700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f556c139a70 con 0x7f556c122190 2019-12-05T05:09:15.426 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:15.414+0000 7f5572f8b700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f556c1176d0 con 0x7f556c122a40 2019-12-05T05:09:15.426 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:15.414+0000 7f5571528700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f556c122190 0x7f556c07b370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:56610/0 (socket says 172.21.15.42:56610) 2019-12-05T05:09:15.426 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:15.414+0000 7f5571528700 1 -- 172.21.15.42:0/1994543972 learned_addr learned my addr 172.21.15.42:0/1994543972 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:09:15.426 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:15.414+0000 7f5570d27700 1 --2- 172.21.15.42:0/1994543972 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f556c122a40 0x7f556c119120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:09:15.426 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:15.414+0000 7f5561ffb700 1 -- 172.21.15.42:0/1994543972 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 577855390 0 0) 0x7f556c139a70 con 0x7f556c122190 2019-12-05T05:09:15.426 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:15.414+0000 7f5561ffb700 1 -- 172.21.15.42:0/1994543972 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f554c0030b0 con 0x7f556c122190 2019-12-05T05:09:15.427 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:15.414+0000 7f5561ffb700 1 -- 172.21.15.42:0/1994543972 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2098569354 0 0) 0x7f556c12d470 con 0x7f556c12d040 2019-12-05T05:09:15.427 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:15.414+0000 7f5561ffb700 1 -- 172.21.15.42:0/1994543972 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f556c139a70 con 0x7f556c12d040 2019-12-05T05:09:15.427 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:15.414+0000 7f5570d27700 1 -- 172.21.15.42:0/1994543972 >> v1:172.21.15.42:6789/0 conn(0x7f556c122190 legacy=0x7f556c07b370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:09:15.427 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:15.414+0000 7f5570d27700 1 -- 172.21.15.42:0/1994543972 >> v1:172.21.15.40:6789/0 conn(0x7f556c12d040 legacy=0x7f556c116ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:09:15.427 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:15.414+0000 7f5570d27700 1 -- 172.21.15.42:0/1994543972 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f556c13acb0 con 0x7f556c122a40 2019-12-05T05:09:15.427 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:15.414+0000 7f5570d27700 1 --2- 172.21.15.42:0/1994543972 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f556c122a40 0x7f556c119120 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f555c008790 tx=0x7f555c0116e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:15.428 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:15.414+0000 7f5570d27700 1 --2- 172.21.15.42:0/1994543972 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f556c122a40 0x7f556c119120 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f555c008790 tx=0x7f555c0116e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:15.428 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:15.414+0000 7f5570d27700 1 --2- 172.21.15.42:0/1994543972 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f556c122a40 0x7f556c119120 secure :-1 s=READY pgs=66 cs=0 l=1 rx=0x7f555c008790 tx=0x7f555c0116e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:09:15.428 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:15.414+0000 7f5570d27700 1 --2- 172.21.15.42:0/1994543972 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f556c122a40 0x7f556c119120 secure :-1 s=THROTTLE_DONE pgs=66 cs=0 l=1 rx=0x7f555c008790 tx=0x7f555c0116e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:15.428 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:15.414+0000 7f5561ffb700 1 -- 172.21.15.42:0/1994543972 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f555c007150 con 0x7f556c122a40 2019-12-05T05:09:15.428 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:15.414+0000 7f5570d27700 1 --2- 172.21.15.42:0/1994543972 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f556c122a40 0x7f556c119120 secure :-1 s=THROTTLE_DONE pgs=66 cs=0 l=1 rx=0x7f555c008790 tx=0x7f555c0116e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:15.428 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:15.414+0000 7f5570d27700 1 --2- 172.21.15.42:0/1994543972 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f556c122a40 0x7f556c119120 secure :-1 s=THROTTLE_DONE pgs=66 cs=0 l=1 rx=0x7f555c008790 tx=0x7f555c0116e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:15.428 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:15.414+0000 7f5561ffb700 1 -- 172.21.15.42:0/1994543972 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f555c0114f0 con 0x7f556c122a40 2019-12-05T05:09:15.429 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:15.414+0000 7f5572f8b700 1 -- 172.21.15.42:0/1994543972 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f556c138a60 con 0x7f556c122a40 2019-12-05T05:09:15.429 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:15.414+0000 7f5561ffb700 1 -- 172.21.15.42:0/1994543972 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f555c008cc0 con 0x7f556c122a40 2019-12-05T05:09:15.429 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:15.414+0000 7f5572f8b700 1 -- 172.21.15.42:0/1994543972 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f556c138fb0 con 0x7f556c122a40 2019-12-05T05:09:15.429 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:15.414+0000 7f5570d27700 1 --2- 172.21.15.42:0/1994543972 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f556c122a40 0x7f556c119120 secure :-1 s=THROTTLE_DONE pgs=66 cs=0 l=1 rx=0x7f555c008790 tx=0x7f555c0116e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:15.429 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:15.418+0000 7f5561ffb700 1 -- 172.21.15.42:0/1994543972 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f555c005540 con 0x7f556c122a40 2019-12-05T05:09:15.429 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:15.418+0000 7f5570d27700 1 --2- 172.21.15.42:0/1994543972 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f556c122a40 0x7f556c119120 secure :-1 s=THROTTLE_DONE pgs=66 cs=0 l=1 rx=0x7f555c008790 tx=0x7f555c0116e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:15.429 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:15.418+0000 7f5561ffb700 1 --2- 172.21.15.42:0/1994543972 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f554c03b410 0x7f554c03d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:09:15.430 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:15.418+0000 7f5561ffb700 1 -- 172.21.15.42:0/1994543972 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f555c04b3b0 con 0x7f556c122a40 2019-12-05T05:09:15.430 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:15.418+0000 7f5571528700 1 --2- 172.21.15.42:0/1994543972 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f554c03b410 0x7f554c03d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:09:15.430 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:15.418+0000 7f5571528700 1 --2- 172.21.15.42:0/1994543972 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f554c03b410 0x7f554c03d920 crc :-1 s=READY pgs=155 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:09:15.431 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:15.418+0000 7f5572f8b700 1 -- 172.21.15.42:0/1994543972 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f556c05d110 con 0x7f556c122a40 2019-12-05T05:09:15.435 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:15.430+0000 7f5570d27700 1 --2- 172.21.15.42:0/1994543972 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f556c122a40 0x7f556c119120 secure :-1 s=THROTTLE_DONE pgs=66 cs=0 l=1 rx=0x7f555c008790 tx=0x7f555c0116e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:15.436 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:15.430+0000 7f5561ffb700 1 -- 172.21.15.42:0/1994543972 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f555c011920 con 0x7f556c122a40 2019-12-05T05:09:15.606 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:15.598+0000 7f5572f8b700 1 -- 172.21.15.42:0/1994543972 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f556c05b8c0 con 0x7f556c122a40 2019-12-05T05:09:15.607 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:15.602+0000 7f5570d27700 1 --2- 172.21.15.42:0/1994543972 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f556c122a40 0x7f556c119120 secure :-1 s=THROTTLE_DONE pgs=66 cs=0 l=1 rx=0x7f555c008790 tx=0x7f555c0116e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:15.607 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:15.602+0000 7f5561ffb700 1 -- 172.21.15.42:0/1994543972 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+2102 (secure 0 0 0) 0x7f555c032410 con 0x7f556c122a40 2019-12-05T05:09:15.609 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 29 2019-12-05T05:09:15.609 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:09:15.610 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4978,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/3650564018","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3650564018},{"type":"v1","addr":"172.21.15.42:6835","nonce":3650564018}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":29}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:07:12.263209+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4831},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4831":{"gid":4831,"name":"b","rank":0,"incarnation":24,"state":"up:active","state_seq":39,"addr":"172.21.15.40:6833/90248709","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":90248709},{"type":"v1","addr":"172.21.15.40:6833","nonce":90248709}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:09:15.612 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:15.606+0000 7f5572f8b700 1 -- 172.21.15.42:0/1994543972 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f554c03b410 msgr2=0x7f554c03d920 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:09:15.612 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:15.606+0000 7f5572f8b700 1 --2- 172.21.15.42:0/1994543972 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f554c03b410 0x7f554c03d920 crc :-1 s=READY pgs=155 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:09:15.612 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:15.606+0000 7f5572f8b700 1 -- 172.21.15.42:0/1994543972 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f556c122a40 msgr2=0x7f556c119120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:09:15.613 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:15.606+0000 7f5572f8b700 1 --2- 172.21.15.42:0/1994543972 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f556c122a40 0x7f556c119120 secure :-1 s=READY pgs=66 cs=0 l=1 rx=0x7f555c008790 tx=0x7f555c0116e0).stop 2019-12-05T05:09:15.613 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:15.606+0000 7f5572f8b700 1 -- 172.21.15.42:0/1994543972 shutdown_connections 2019-12-05T05:09:15.613 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:15.606+0000 7f5572f8b700 1 --2- 172.21.15.42:0/1994543972 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f554c03b410 0x7f554c03d920 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:09:15.613 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:15.606+0000 7f5572f8b700 1 --2- 172.21.15.42:0/1994543972 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f556c122a40 0x7f556c119120 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:09:15.613 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:15.606+0000 7f5572f8b700 1 -- 172.21.15.42:0/1994543972 shutdown_connections 2019-12-05T05:09:15.613 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:15.606+0000 7f5572f8b700 1 -- 172.21.15.42:0/1994543972 wait complete. 2019-12-05T05:09:15.614 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:15.606+0000 7f5572f8b700 1 -- 172.21.15.42:0/1994543972 >> 172.21.15.42:0/1994543972 conn(0x7f556c0aabe0 msgr2=0x7f556c059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:09:15.622 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T05:09:17.623 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:09:17.727 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:17.714+0000 7fb00110a700 1 Processor -- start 2019-12-05T05:09:17.728 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:17.714+0000 7fb00110a700 1 -- start start 2019-12-05T05:09:17.728 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:17.718+0000 7fb00110a700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7faffc0a7740 0x7faffc08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:09:17.728 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:17.718+0000 7fb00110a700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7faffc07e720 0x7faffc07eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:09:17.728 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:17.718+0000 7fb00110a700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7faffc09f2e0 0x7faffc09f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:09:17.728 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:17.718+0000 7fb00110a700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faffc0a0b00 con 0x7faffc0a7740 2019-12-05T05:09:17.728 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:17.718+0000 7fb00110a700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faffc08e5b0 con 0x7faffc09f2e0 2019-12-05T05:09:17.728 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:17.718+0000 7fb00110a700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faffc08e850 con 0x7faffc07e720 2019-12-05T05:09:17.729 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:17.718+0000 7fb000909700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7faffc09f2e0 0x7faffc09f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:09:17.729 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:17.718+0000 7faffad9d700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7faffc07e720 0x7faffc07eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:09:17.729 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:17.718+0000 7faffb59e700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7faffc0a7740 0x7faffc08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:09:17.729 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:17.718+0000 7fb000909700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7faffc09f2e0 0x7faffc09f710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:56584/0 (socket says 172.21.15.42:56584) 2019-12-05T05:09:17.730 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:17.718+0000 7fb000909700 1 -- 172.21.15.42:0/3286668709 learned_addr learned my addr 172.21.15.42:0/3286668709 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:09:17.730 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:17.718+0000 7fb000909700 1 -- 172.21.15.42:0/3286668709 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7faffc07e720 msgr2=0x7faffc07eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:09:17.730 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:17.718+0000 7fb000909700 1 --2- 172.21.15.42:0/3286668709 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7faffc07e720 0x7faffc07eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:09:17.730 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:17.718+0000 7fb000909700 1 -- 172.21.15.42:0/3286668709 >> v2:172.21.15.40:3300/0 conn(0x7faffc0a7740 msgr2=0x7faffc08df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:09:17.730 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:17.718+0000 7fb000909700 1 --2- 172.21.15.42:0/3286668709 >> v2:172.21.15.40:3300/0 conn(0x7faffc0a7740 0x7faffc08df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:09:17.730 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:17.718+0000 7fb000909700 1 -- 172.21.15.42:0/3286668709 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faffc07f0a0 con 0x7faffc09f2e0 2019-12-05T05:09:17.730 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:17.718+0000 7fb000909700 1 --2- 172.21.15.42:0/3286668709 >> v2:172.21.15.42:3300/0 conn(0x7faffc09f2e0 0x7faffc09f710 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7faffc08e770 tx=0x7fafec00fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:17.731 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:17.718+0000 7fb000909700 1 --2- 172.21.15.42:0/3286668709 >> v2:172.21.15.42:3300/0 conn(0x7faffc09f2e0 0x7faffc09f710 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7faffc08e770 tx=0x7fafec00fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:17.731 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:17.718+0000 7fb000909700 1 --2- 172.21.15.42:0/3286668709 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faffc09f2e0 0x7faffc09f710 secure :-1 s=READY pgs=340 cs=0 l=1 rx=0x7faffc08e770 tx=0x7fafec00fdb0).ready entity=mon.1 client_cookie=e4a8941b5c885d7c server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:09:17.731 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:17.718+0000 7fb000909700 1 --2- 172.21.15.42:0/3286668709 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faffc09f2e0 0x7faffc09f710 secure :-1 s=THROTTLE_DONE pgs=340 cs=0 l=1 rx=0x7faffc08e770 tx=0x7fafec00fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:17.731 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:17.718+0000 7fb000909700 1 --2- 172.21.15.42:0/3286668709 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faffc09f2e0 0x7faffc09f710 secure :-1 s=THROTTLE_DONE pgs=340 cs=0 l=1 rx=0x7faffc08e770 tx=0x7fafec00fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:17.731 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:17.718+0000 7faffa59c700 1 -- 172.21.15.42:0/3286668709 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fafec008c90 con 0x7faffc09f2e0 2019-12-05T05:09:17.731 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:17.718+0000 7fb000909700 1 --2- 172.21.15.42:0/3286668709 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faffc09f2e0 0x7faffc09f710 secure :-1 s=THROTTLE_DONE pgs=340 cs=0 l=1 rx=0x7faffc08e770 tx=0x7fafec00fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:17.731 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:17.718+0000 7faffa59c700 1 -- 172.21.15.42:0/3286668709 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafec0088b0 con 0x7faffc09f2e0 2019-12-05T05:09:17.732 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:17.718+0000 7faffa59c700 1 -- 172.21.15.42:0/3286668709 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fafec011b50 con 0x7faffc09f2e0 2019-12-05T05:09:17.732 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:17.718+0000 7fb00110a700 1 -- 172.21.15.42:0/3286668709 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faffc09f2e0 msgr2=0x7faffc09f710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:09:17.732 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:17.718+0000 7fb00110a700 1 --2- 172.21.15.42:0/3286668709 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faffc09f2e0 0x7faffc09f710 secure :-1 s=READY pgs=340 cs=0 l=1 rx=0x7faffc08e770 tx=0x7fafec00fdb0).stop 2019-12-05T05:09:17.732 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:17.718+0000 7fb00110a700 1 -- 172.21.15.42:0/3286668709 shutdown_connections 2019-12-05T05:09:17.732 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:17.718+0000 7fb00110a700 1 --2- 172.21.15.42:0/3286668709 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faffc09f2e0 0x7faffc09f710 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:09:17.732 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:17.718+0000 7fb00110a700 1 --2- 172.21.15.42:0/3286668709 >> v2:172.21.15.40:3300/0 conn(0x7faffc0a7740 0x7faffc08df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:09:17.732 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:17.718+0000 7fb00110a700 1 --2- 172.21.15.42:0/3286668709 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7faffc07e720 0x7faffc07eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:09:17.733 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:17.722+0000 7fb00110a700 1 -- 172.21.15.42:0/3286668709 shutdown_connections 2019-12-05T05:09:17.733 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:17.722+0000 7fb00110a700 1 -- 172.21.15.42:0/3286668709 wait complete. 2019-12-05T05:09:17.733 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:17.722+0000 7fb00110a700 1 -- 172.21.15.42:0/3286668709 >> 172.21.15.42:0/3286668709 conn(0x7faffc0aabe0 msgr2=0x7faffc0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:09:17.736 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:17.722+0000 7fb00110a700 1 Processor -- start 2019-12-05T05:09:17.736 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:17.722+0000 7fb00110a700 1 -- start start 2019-12-05T05:09:17.737 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:17.722+0000 7fb00110a700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7faffc130cd0 0x7faffc11b290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:09:17.737 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:17.722+0000 7fb00110a700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faffc07d760 con 0x7faffc0a6b00 2019-12-05T05:09:17.737 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:17.722+0000 7fb00110a700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faffc11a060 con 0x7faffc07b4b0 2019-12-05T05:09:17.737 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:17.722+0000 7fb00110a700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faffc11ba00 con 0x7faffc130cd0 2019-12-05T05:09:17.737 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:17.722+0000 7faffb59e700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7faffc07b4b0 0x7faffc1304d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:56624/0 (socket says 172.21.15.42:56624) 2019-12-05T05:09:17.737 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:17.722+0000 7faffb59e700 1 -- 172.21.15.42:0/1026924685 learned_addr learned my addr 172.21.15.42:0/1026924685 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:09:17.737 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:17.722+0000 7faffad9d700 1 --2- 172.21.15.42:0/1026924685 >> v2:172.21.15.42:3300/0 conn(0x7faffc130cd0 0x7faffc11b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:09:17.738 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:17.726+0000 7faff3fff700 1 -- 172.21.15.42:0/1026924685 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3744461729 0 0) 0x7faffc11a060 con 0x7faffc07b4b0 2019-12-05T05:09:17.738 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:17.726+0000 7faff3fff700 1 -- 172.21.15.42:0/1026924685 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fafe00030b0 con 0x7faffc07b4b0 2019-12-05T05:09:17.738 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:17.726+0000 7faff3fff700 1 -- 172.21.15.42:0/1026924685 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 834569397 0 0) 0x7faffc07d760 con 0x7faffc0a6b00 2019-12-05T05:09:17.738 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:17.726+0000 7faff3fff700 1 -- 172.21.15.42:0/1026924685 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faffc11a060 con 0x7faffc0a6b00 2019-12-05T05:09:17.738 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:17.726+0000 7faffad9d700 1 -- 172.21.15.42:0/1026924685 >> v1:172.21.15.42:6789/0 conn(0x7faffc07b4b0 legacy=0x7faffc1304d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:09:17.738 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:17.726+0000 7faffad9d700 1 -- 172.21.15.42:0/1026924685 >> v1:172.21.15.40:6789/0 conn(0x7faffc0a6b00 legacy=0x7faffc137680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:09:17.738 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:17.726+0000 7faffad9d700 1 -- 172.21.15.42:0/1026924685 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faffc0a6f30 con 0x7faffc130cd0 2019-12-05T05:09:17.739 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:17.726+0000 7faffad9d700 1 --2- 172.21.15.42:0/1026924685 >> v2:172.21.15.42:3300/0 conn(0x7faffc130cd0 0x7faffc11b290 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fafe4002f30 tx=0x7fafe4010a70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:17.739 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:17.726+0000 7faffad9d700 1 --2- 172.21.15.42:0/1026924685 >> v2:172.21.15.42:3300/0 conn(0x7faffc130cd0 0x7faffc11b290 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fafe4002f30 tx=0x7fafe4010a70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:17.739 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:17.726+0000 7faffad9d700 1 --2- 172.21.15.42:0/1026924685 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faffc130cd0 0x7faffc11b290 secure :-1 s=READY pgs=342 cs=0 l=1 rx=0x7fafe4002f30 tx=0x7fafe4010a70).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:09:17.739 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:17.726+0000 7faffad9d700 1 --2- 172.21.15.42:0/1026924685 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faffc130cd0 0x7faffc11b290 secure :-1 s=THROTTLE_DONE pgs=342 cs=0 l=1 rx=0x7fafe4002f30 tx=0x7fafe4010a70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:17.739 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:17.726+0000 7faffad9d700 1 --2- 172.21.15.42:0/1026924685 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faffc130cd0 0x7faffc11b290 secure :-1 s=THROTTLE_DONE pgs=342 cs=0 l=1 rx=0x7fafe4002f30 tx=0x7fafe4010a70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:17.740 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:17.726+0000 7faff3fff700 1 -- 172.21.15.42:0/1026924685 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fafe4005920 con 0x7faffc130cd0 2019-12-05T05:09:17.740 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:17.726+0000 7faffad9d700 1 --2- 172.21.15.42:0/1026924685 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faffc130cd0 0x7faffc11b290 secure :-1 s=THROTTLE_DONE pgs=342 cs=0 l=1 rx=0x7fafe4002f30 tx=0x7fafe4010a70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:17.740 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:17.726+0000 7faff3fff700 1 -- 172.21.15.42:0/1026924685 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafe4005540 con 0x7faffc130cd0 2019-12-05T05:09:17.740 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:17.726+0000 7fb00110a700 1 -- 172.21.15.42:0/1026924685 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faffc138e90 con 0x7faffc130cd0 2019-12-05T05:09:17.740 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:17.726+0000 7faff3fff700 1 -- 172.21.15.42:0/1026924685 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fafe4012520 con 0x7faffc130cd0 2019-12-05T05:09:17.740 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:17.726+0000 7fb00110a700 1 -- 172.21.15.42:0/1026924685 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faffc139410 con 0x7faffc130cd0 2019-12-05T05:09:17.740 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:17.726+0000 7faffad9d700 1 --2- 172.21.15.42:0/1026924685 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faffc130cd0 0x7faffc11b290 secure :-1 s=THROTTLE_DONE pgs=342 cs=0 l=1 rx=0x7fafe4002f30 tx=0x7fafe4010a70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:17.741 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:17.726+0000 7faff3fff700 1 -- 172.21.15.42:0/1026924685 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fafe4007180 con 0x7faffc130cd0 2019-12-05T05:09:17.741 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:17.726+0000 7faffad9d700 1 --2- 172.21.15.42:0/1026924685 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faffc130cd0 0x7faffc11b290 secure :-1 s=THROTTLE_DONE pgs=342 cs=0 l=1 rx=0x7fafe4002f30 tx=0x7fafe4010a70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:17.741 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:17.726+0000 7faff3fff700 1 --2- 172.21.15.42:0/1026924685 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fafe003b4a0 0x7fafe003d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:09:17.741 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:17.726+0000 7faff3fff700 1 -- 172.21.15.42:0/1026924685 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fafe404ae60 con 0x7faffc130cd0 2019-12-05T05:09:17.741 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:17.726+0000 7faffb59e700 1 --2- 172.21.15.42:0/1026924685 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fafe003b4a0 0x7fafe003d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:09:17.741 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:17.730+0000 7faffb59e700 1 --2- 172.21.15.42:0/1026924685 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fafe003b4a0 0x7fafe003d9b0 crc :-1 s=READY pgs=156 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:09:17.742 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:17.730+0000 7fb00110a700 1 -- 172.21.15.42:0/1026924685 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faffc05d110 con 0x7faffc130cd0 2019-12-05T05:09:17.744 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:17.738+0000 7faffad9d700 1 --2- 172.21.15.42:0/1026924685 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faffc130cd0 0x7faffc11b290 secure :-1 s=THROTTLE_DONE pgs=342 cs=0 l=1 rx=0x7fafe4002f30 tx=0x7fafe4010a70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:17.745 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:17.738+0000 7faff3fff700 1 -- 172.21.15.42:0/1026924685 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fafe4031ee0 con 0x7faffc130cd0 2019-12-05T05:09:17.906 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:17.898+0000 7fb00110a700 1 -- 172.21.15.42:0/1026924685 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faffc05b8c0 con 0x7faffc130cd0 2019-12-05T05:09:17.907 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:17.902+0000 7faffad9d700 1 --2- 172.21.15.42:0/1026924685 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faffc130cd0 0x7faffc11b290 secure :-1 s=THROTTLE_DONE pgs=342 cs=0 l=1 rx=0x7fafe4002f30 tx=0x7fafe4010a70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:17.907 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:17.902+0000 7faff3fff700 1 -- 172.21.15.42:0/1026924685 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+2102 (secure 0 0 0) 0x7fafe4032140 con 0x7faffc130cd0 2019-12-05T05:09:17.907 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 29 2019-12-05T05:09:17.907 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:09:17.908 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4978,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/3650564018","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3650564018},{"type":"v1","addr":"172.21.15.42:6835","nonce":3650564018}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":29}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:07:12.263209+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4831},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4831":{"gid":4831,"name":"b","rank":0,"incarnation":24,"state":"up:active","state_seq":39,"addr":"172.21.15.40:6833/90248709","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":90248709},{"type":"v1","addr":"172.21.15.40:6833","nonce":90248709}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:09:17.909 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:17.902+0000 7fb00110a700 1 -- 172.21.15.42:0/1026924685 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fafe003b4a0 msgr2=0x7fafe003d9b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:09:17.909 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:17.902+0000 7fb00110a700 1 --2- 172.21.15.42:0/1026924685 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fafe003b4a0 0x7fafe003d9b0 crc :-1 s=READY pgs=156 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:09:17.909 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:17.902+0000 7fb00110a700 1 -- 172.21.15.42:0/1026924685 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faffc130cd0 msgr2=0x7faffc11b290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:09:17.909 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:17.902+0000 7fb00110a700 1 --2- 172.21.15.42:0/1026924685 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faffc130cd0 0x7faffc11b290 secure :-1 s=READY pgs=342 cs=0 l=1 rx=0x7fafe4002f30 tx=0x7fafe4010a70).stop 2019-12-05T05:09:17.910 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:17.902+0000 7fb00110a700 1 -- 172.21.15.42:0/1026924685 shutdown_connections 2019-12-05T05:09:17.910 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:17.902+0000 7fb00110a700 1 --2- 172.21.15.42:0/1026924685 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fafe003b4a0 0x7fafe003d9b0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:09:17.910 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:17.902+0000 7fb00110a700 1 --2- 172.21.15.42:0/1026924685 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faffc130cd0 0x7faffc11b290 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:09:17.911 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:17.902+0000 7fb00110a700 1 -- 172.21.15.42:0/1026924685 shutdown_connections 2019-12-05T05:09:17.911 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:17.902+0000 7fb00110a700 1 -- 172.21.15.42:0/1026924685 wait complete. 2019-12-05T05:09:17.911 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:17.902+0000 7fb00110a700 1 -- 172.21.15.42:0/1026924685 >> 172.21.15.42:0/1026924685 conn(0x7faffc09fad0 msgr2=0x7faffc059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:09:17.921 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T05:09:19.922 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:09:20.033 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:20.010+0000 7f82e8dc4700 1 Processor -- start 2019-12-05T05:09:20.033 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:20.010+0000 7f82e8dc4700 1 -- start start 2019-12-05T05:09:20.033 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:20.010+0000 7f82e8dc4700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f82e408df70 0x7f82e409f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:09:20.033 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:20.010+0000 7f82e8dc4700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82e409fc50 con 0x7f82e40a7730 2019-12-05T05:09:20.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:20.010+0000 7f82e8dc4700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82e40a0700 con 0x7f82e409f820 2019-12-05T05:09:20.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:20.010+0000 7f82e8dc4700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82e40a0af0 con 0x7f82e408df70 2019-12-05T05:09:20.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:20.014+0000 7f82e259c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f82e408df70 0x7f82e409f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:09:20.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:20.014+0000 7f82e259c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f82e408df70 0x7f82e409f2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:56596/0 (socket says 172.21.15.42:56596) 2019-12-05T05:09:20.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:20.014+0000 7f82e259c700 1 -- 172.21.15.42:0/1671269639 learned_addr learned my addr 172.21.15.42:0/1671269639 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:09:20.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:20.014+0000 7f82e259c700 1 -- 172.21.15.42:0/1671269639 >> v1:172.21.15.42:6789/0 conn(0x7f82e409f820 legacy=0x7f82e414efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:09:20.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:20.014+0000 7f82e259c700 1 -- 172.21.15.42:0/1671269639 >> v1:172.21.15.40:6789/0 conn(0x7f82e40a7730 legacy=0x7f82e407e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:09:20.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:20.014+0000 7f82e259c700 1 -- 172.21.15.42:0/1671269639 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82e407ef10 con 0x7f82e408df70 2019-12-05T05:09:20.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:20.014+0000 7f82e259c700 1 --2- 172.21.15.42:0/1671269639 >> v2:172.21.15.42:3300/0 conn(0x7f82e408df70 0x7f82e409f2d0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f82cc008790 tx=0x7f82cc00fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:20.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:20.014+0000 7f82e259c700 1 --2- 172.21.15.42:0/1671269639 >> v2:172.21.15.42:3300/0 conn(0x7f82e408df70 0x7f82e409f2d0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f82cc008790 tx=0x7f82cc00fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:20.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:20.014+0000 7f82e259c700 1 --2- 172.21.15.42:0/1671269639 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f82e408df70 0x7f82e409f2d0 secure :-1 s=READY pgs=344 cs=0 l=1 rx=0x7f82cc008790 tx=0x7f82cc00fcb0).ready entity=mon.1 client_cookie=8f1cacf00acb2bb7 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:09:20.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:20.014+0000 7f82e259c700 1 --2- 172.21.15.42:0/1671269639 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f82e408df70 0x7f82e409f2d0 secure :-1 s=THROTTLE_DONE pgs=344 cs=0 l=1 rx=0x7f82cc008790 tx=0x7f82cc00fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:20.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:20.014+0000 7f82e1d9b700 1 -- 172.21.15.42:0/1671269639 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f82cc011520 con 0x7f82e408df70 2019-12-05T05:09:20.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:20.014+0000 7f82e259c700 1 --2- 172.21.15.42:0/1671269639 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f82e408df70 0x7f82e409f2d0 secure :-1 s=THROTTLE_DONE pgs=344 cs=0 l=1 rx=0x7f82cc008790 tx=0x7f82cc00fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:20.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:20.014+0000 7f82e259c700 1 --2- 172.21.15.42:0/1671269639 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f82e408df70 0x7f82e409f2d0 secure :-1 s=THROTTLE_DONE pgs=344 cs=0 l=1 rx=0x7f82cc008790 tx=0x7f82cc00fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:20.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:20.014+0000 7f82e1d9b700 1 -- 172.21.15.42:0/1671269639 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82cc008ca0 con 0x7f82e408df70 2019-12-05T05:09:20.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:20.014+0000 7f82e1d9b700 1 -- 172.21.15.42:0/1671269639 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f82cc011b00 con 0x7f82e408df70 2019-12-05T05:09:20.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:20.014+0000 7f82e8dc4700 1 -- 172.21.15.42:0/1671269639 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f82e408df70 msgr2=0x7f82e409f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:09:20.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:20.014+0000 7f82e8dc4700 1 --2- 172.21.15.42:0/1671269639 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f82e408df70 0x7f82e409f2d0 secure :-1 s=READY pgs=344 cs=0 l=1 rx=0x7f82cc008790 tx=0x7f82cc00fcb0).stop 2019-12-05T05:09:20.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:20.014+0000 7f82e8dc4700 1 -- 172.21.15.42:0/1671269639 shutdown_connections 2019-12-05T05:09:20.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:20.014+0000 7f82e8dc4700 1 --2- 172.21.15.42:0/1671269639 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f82e408df70 0x7f82e409f2d0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:09:20.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:20.014+0000 7f82e8dc4700 1 -- 172.21.15.42:0/1671269639 shutdown_connections 2019-12-05T05:09:20.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:20.014+0000 7f82e8dc4700 1 -- 172.21.15.42:0/1671269639 wait complete. 2019-12-05T05:09:20.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:20.014+0000 7f82e8dc4700 1 -- 172.21.15.42:0/1671269639 >> 172.21.15.42:0/1671269639 conn(0x7f82e40aabd0 msgr2=0x7f82e40a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:09:20.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:20.014+0000 7f82e8dc4700 1 Processor -- start 2019-12-05T05:09:20.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:20.014+0000 7f82e8dc4700 1 -- start start 2019-12-05T05:09:20.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:20.018+0000 7f82e8dc4700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f82e4064600 0x7f82e4124e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:09:20.039 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:20.018+0000 7f82e8dc4700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f82e40a5b10 0x7f82e40a5f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:09:20.039 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:20.018+0000 7f82e8dc4700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82e40a6ae0 con 0x7f82e407d7b0 2019-12-05T05:09:20.039 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:20.018+0000 7f82e8dc4700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82e4124690 con 0x7f82e4064600 2019-12-05T05:09:20.039 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:20.018+0000 7f82e8dc4700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82e4124930 con 0x7f82e40a5b10 2019-12-05T05:09:20.039 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:20.018+0000 7f82e359e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f82e40a5b10 0x7f82e40a5f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:09:20.040 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:20.018+0000 7f82e359e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f82e40a5b10 0x7f82e40a5f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:56606/0 (socket says 172.21.15.42:56606) 2019-12-05T05:09:20.040 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:20.018+0000 7f82e359e700 1 -- 172.21.15.42:0/3890150922 learned_addr learned my addr 172.21.15.42:0/3890150922 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:09:20.040 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:20.018+0000 7f82e2d9d700 1 --2- 172.21.15.42:0/3890150922 >> v2:172.21.15.40:3300/0 conn(0x7f82e4064600 0x7f82e4124e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:09:20.040 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:20.018+0000 7f82e359e700 1 -- 172.21.15.42:0/3890150922 >> v2:172.21.15.40:3300/0 conn(0x7f82e4064600 msgr2=0x7f82e4124e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:09:20.040 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:20.018+0000 7f82e359e700 1 --2- 172.21.15.42:0/3890150922 >> v2:172.21.15.40:3300/0 conn(0x7f82e4064600 0x7f82e4124e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:09:20.040 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:20.018+0000 7f82e359e700 1 -- 172.21.15.42:0/3890150922 >> v1:172.21.15.40:6789/0 conn(0x7f82e407d7b0 legacy=0x7f82e407dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:09:20.040 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:20.018+0000 7f82e359e700 1 -- 172.21.15.42:0/3890150922 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82e41253c0 con 0x7f82e40a5b10 2019-12-05T05:09:20.041 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:20.018+0000 7f82e359e700 1 --2- 172.21.15.42:0/3890150922 >> v2:172.21.15.42:3300/0 conn(0x7f82e40a5b10 0x7f82e40a5f40 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f82dc002ef0 tx=0x7f82dc0107d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:20.041 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:20.018+0000 7f82e359e700 1 --2- 172.21.15.42:0/3890150922 >> v2:172.21.15.42:3300/0 conn(0x7f82e40a5b10 0x7f82e40a5f40 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f82dc002ef0 tx=0x7f82dc0107d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:20.041 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:20.018+0000 7f82e359e700 1 --2- 172.21.15.42:0/3890150922 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f82e40a5b10 0x7f82e40a5f40 secure :-1 s=READY pgs=345 cs=0 l=1 rx=0x7f82dc002ef0 tx=0x7f82dc0107d0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:09:20.041 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:20.018+0000 7f82e359e700 1 --2- 172.21.15.42:0/3890150922 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f82e40a5b10 0x7f82e40a5f40 secure :-1 s=THROTTLE_DONE pgs=345 cs=0 l=1 rx=0x7f82dc002ef0 tx=0x7f82dc0107d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:20.041 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:20.018+0000 7f82e359e700 1 --2- 172.21.15.42:0/3890150922 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f82e40a5b10 0x7f82e40a5f40 secure :-1 s=THROTTLE_DONE pgs=345 cs=0 l=1 rx=0x7f82dc002ef0 tx=0x7f82dc0107d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:20.041 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:20.018+0000 7f82e359e700 1 --2- 172.21.15.42:0/3890150922 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f82e40a5b10 0x7f82e40a5f40 secure :-1 s=THROTTLE_DONE pgs=345 cs=0 l=1 rx=0x7f82dc002ef0 tx=0x7f82dc0107d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:20.041 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:20.018+0000 7f82db7fe700 1 -- 172.21.15.42:0/3890150922 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f82dc0057c0 con 0x7f82e40a5b10 2019-12-05T05:09:20.042 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:20.018+0000 7f82db7fe700 1 -- 172.21.15.42:0/3890150922 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82dc007160 con 0x7f82e40a5b10 2019-12-05T05:09:20.042 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:20.018+0000 7f82e8dc4700 1 -- 172.21.15.42:0/3890150922 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82e40a6490 con 0x7f82e40a5b10 2019-12-05T05:09:20.042 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:20.018+0000 7f82db7fe700 1 -- 172.21.15.42:0/3890150922 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f82dc005de0 con 0x7f82e40a5b10 2019-12-05T05:09:20.042 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:20.018+0000 7f82e8dc4700 1 -- 172.21.15.42:0/3890150922 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82e412f000 con 0x7f82e40a5b10 2019-12-05T05:09:20.042 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:20.018+0000 7f82e359e700 1 --2- 172.21.15.42:0/3890150922 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f82e40a5b10 0x7f82e40a5f40 secure :-1 s=THROTTLE_DONE pgs=345 cs=0 l=1 rx=0x7f82dc002ef0 tx=0x7f82dc0107d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:20.042 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:20.018+0000 7f82db7fe700 1 -- 172.21.15.42:0/3890150922 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f82dc0087b0 con 0x7f82e40a5b10 2019-12-05T05:09:20.042 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:20.018+0000 7f82e359e700 1 --2- 172.21.15.42:0/3890150922 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f82e40a5b10 0x7f82e40a5f40 secure :-1 s=THROTTLE_DONE pgs=345 cs=0 l=1 rx=0x7f82dc002ef0 tx=0x7f82dc0107d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:20.043 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:20.018+0000 7f82db7fe700 1 --2- 172.21.15.42:0/3890150922 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f82c803ae40 0x7f82c803d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:09:20.043 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:20.018+0000 7f82db7fe700 1 -- 172.21.15.42:0/3890150922 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f82dc04acb0 con 0x7f82e40a5b10 2019-12-05T05:09:20.043 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:20.018+0000 7f82e2d9d700 1 --2- 172.21.15.42:0/3890150922 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f82c803ae40 0x7f82c803d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:09:20.043 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:20.018+0000 7f82e2d9d700 1 --2- 172.21.15.42:0/3890150922 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f82c803ae40 0x7f82c803d350 crc :-1 s=READY pgs=157 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:09:20.043 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:20.026+0000 7f82e8dc4700 1 -- 172.21.15.42:0/3890150922 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82e405d100 con 0x7f82e40a5b10 2019-12-05T05:09:20.044 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:20.034+0000 7f82e359e700 1 --2- 172.21.15.42:0/3890150922 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f82e40a5b10 0x7f82e40a5f40 secure :-1 s=THROTTLE_DONE pgs=345 cs=0 l=1 rx=0x7f82dc002ef0 tx=0x7f82dc0107d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:20.044 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:20.034+0000 7f82db7fe700 1 -- 172.21.15.42:0/3890150922 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f82dc031d30 con 0x7f82e40a5b10 2019-12-05T05:09:20.212 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:20.210+0000 7f82e8dc4700 1 -- 172.21.15.42:0/3890150922 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f82e405b8b0 con 0x7f82e40a5b10 2019-12-05T05:09:20.213 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:20.210+0000 7f82e359e700 1 --2- 172.21.15.42:0/3890150922 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f82e40a5b10 0x7f82e40a5f40 secure :-1 s=THROTTLE_DONE pgs=345 cs=0 l=1 rx=0x7f82dc002ef0 tx=0x7f82dc0107d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:20.213 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:20.210+0000 7f82db7fe700 1 -- 172.21.15.42:0/3890150922 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+2102 (secure 0 0 0) 0x7f82dc031f90 con 0x7f82e40a5b10 2019-12-05T05:09:20.213 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 29 2019-12-05T05:09:20.213 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:09:20.214 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4978,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/3650564018","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3650564018},{"type":"v1","addr":"172.21.15.42:6835","nonce":3650564018}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":29}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:07:12.263209+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4831},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4831":{"gid":4831,"name":"b","rank":0,"incarnation":24,"state":"up:active","state_seq":39,"addr":"172.21.15.40:6833/90248709","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":90248709},{"type":"v1","addr":"172.21.15.40:6833","nonce":90248709}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:09:20.217 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:20.214+0000 7f82e8dc4700 1 -- 172.21.15.42:0/3890150922 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f82c803ae40 msgr2=0x7f82c803d350 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:09:20.217 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:20.214+0000 7f82e8dc4700 1 --2- 172.21.15.42:0/3890150922 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f82c803ae40 0x7f82c803d350 crc :-1 s=READY pgs=157 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:09:20.217 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:20.214+0000 7f82e8dc4700 1 -- 172.21.15.42:0/3890150922 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f82e40a5b10 msgr2=0x7f82e40a5f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:09:20.217 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:20.214+0000 7f82e8dc4700 1 --2- 172.21.15.42:0/3890150922 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f82e40a5b10 0x7f82e40a5f40 secure :-1 s=READY pgs=345 cs=0 l=1 rx=0x7f82dc002ef0 tx=0x7f82dc0107d0).stop 2019-12-05T05:09:20.217 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:20.214+0000 7f82e8dc4700 1 -- 172.21.15.42:0/3890150922 shutdown_connections 2019-12-05T05:09:20.217 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:20.214+0000 7f82e8dc4700 1 --2- 172.21.15.42:0/3890150922 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f82c803ae40 0x7f82c803d350 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:09:20.218 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:20.214+0000 7f82e8dc4700 1 --2- 172.21.15.42:0/3890150922 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f82e40a5b10 0x7f82e40a5f40 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:09:20.218 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:20.214+0000 7f82e8dc4700 1 --2- 172.21.15.42:0/3890150922 >> v2:172.21.15.40:3300/0 conn(0x7f82e4064600 0x7f82e4124e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:09:20.218 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:20.214+0000 7f82e8dc4700 1 -- 172.21.15.42:0/3890150922 shutdown_connections 2019-12-05T05:09:20.218 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:20.214+0000 7f82e8dc4700 1 -- 172.21.15.42:0/3890150922 wait complete. 2019-12-05T05:09:20.218 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:20.214+0000 7f82e8dc4700 1 -- 172.21.15.42:0/3890150922 >> 172.21.15.42:0/3890150922 conn(0x7f82e40a7f80 msgr2=0x7f82e407e4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:09:20.225 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T05:09:22.227 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:09:22.336 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:22.326+0000 7f1a13fbd700 1 Processor -- start 2019-12-05T05:09:22.337 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:22.326+0000 7f1a13fbd700 1 -- start start 2019-12-05T05:09:22.337 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:22.330+0000 7f1a13fbd700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f1a0c09fae0 0x7f1a0c14fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:09:22.337 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:22.330+0000 7f1a13fbd700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a0c0a0710 con 0x7f1a0c0a7740 2019-12-05T05:09:22.337 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:22.330+0000 7f1a13fbd700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a0c07ef20 con 0x7f1a0c08df80 2019-12-05T05:09:22.337 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:22.330+0000 7f1a13fbd700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a0c0a0b00 con 0x7f1a0c09fae0 2019-12-05T05:09:22.337 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:22.330+0000 7f1a11d59700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f1a0c08df80 0x7f1a0c09f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:56650/0 (socket says 172.21.15.42:56650) 2019-12-05T05:09:22.337 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:22.330+0000 7f1a11d59700 1 -- 172.21.15.42:0/1635659525 learned_addr learned my addr 172.21.15.42:0/1635659525 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:09:22.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:22.330+0000 7f1a12d5b700 1 --2- 172.21.15.42:0/1635659525 >> v2:172.21.15.42:3300/0 conn(0x7f1a0c09fae0 0x7f1a0c14fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:09:22.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:22.330+0000 7f1a11558700 1 -- 172.21.15.42:0/1635659525 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2594986954 0 0) 0x7f1a0c07ef20 con 0x7f1a0c08df80 2019-12-05T05:09:22.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:22.330+0000 7f1a11558700 1 -- 172.21.15.42:0/1635659525 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19f80030b0 con 0x7f1a0c08df80 2019-12-05T05:09:22.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:22.330+0000 7f1a11558700 1 -- 172.21.15.42:0/1635659525 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3269207119 0 0) 0x7f1a0c0a0710 con 0x7f1a0c0a7740 2019-12-05T05:09:22.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:22.330+0000 7f1a12d5b700 1 -- 172.21.15.42:0/1635659525 >> v1:172.21.15.42:6789/0 conn(0x7f1a0c08df80 legacy=0x7f1a0c09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:09:22.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:22.330+0000 7f1a12d5b700 1 -- 172.21.15.42:0/1635659525 >> v1:172.21.15.40:6789/0 conn(0x7f1a0c0a7740 legacy=0x7f1a0c07e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:09:22.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:22.330+0000 7f1a12d5b700 1 -- 172.21.15.42:0/1635659525 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a0c0a9ba0 con 0x7f1a0c09fae0 2019-12-05T05:09:22.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:22.330+0000 7f1a12d5b700 1 --2- 172.21.15.42:0/1635659525 >> v2:172.21.15.42:3300/0 conn(0x7f1a0c09fae0 0x7f1a0c14fff0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f1a04008b10 tx=0x7f1a0400ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:22.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:22.330+0000 7f1a12d5b700 1 --2- 172.21.15.42:0/1635659525 >> v2:172.21.15.42:3300/0 conn(0x7f1a0c09fae0 0x7f1a0c14fff0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f1a04008b10 tx=0x7f1a0400ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:22.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:22.330+0000 7f1a12d5b700 1 --2- 172.21.15.42:0/1635659525 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1a0c09fae0 0x7f1a0c14fff0 secure :-1 s=READY pgs=347 cs=0 l=1 rx=0x7f1a04008b10 tx=0x7f1a0400ff70).ready entity=mon.1 client_cookie=55217f3b3fcb979f server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:09:22.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:22.330+0000 7f1a12d5b700 1 --2- 172.21.15.42:0/1635659525 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1a0c09fae0 0x7f1a0c14fff0 secure :-1 s=THROTTLE_DONE pgs=347 cs=0 l=1 rx=0x7f1a04008b10 tx=0x7f1a0400ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:22.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:22.330+0000 7f1a11558700 1 -- 172.21.15.42:0/1635659525 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1a04010970 con 0x7f1a0c09fae0 2019-12-05T05:09:22.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:22.330+0000 7f1a12d5b700 1 --2- 172.21.15.42:0/1635659525 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1a0c09fae0 0x7f1a0c14fff0 secure :-1 s=THROTTLE_DONE pgs=347 cs=0 l=1 rx=0x7f1a04008b10 tx=0x7f1a0400ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:22.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:22.330+0000 7f1a12d5b700 1 --2- 172.21.15.42:0/1635659525 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1a0c09fae0 0x7f1a0c14fff0 secure :-1 s=THROTTLE_DONE pgs=347 cs=0 l=1 rx=0x7f1a04008b10 tx=0x7f1a0400ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:22.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:22.330+0000 7f1a11558700 1 -- 172.21.15.42:0/1635659525 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a04008b50 con 0x7f1a0c09fae0 2019-12-05T05:09:22.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:22.330+0000 7f1a11558700 1 -- 172.21.15.42:0/1635659525 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1a040110c0 con 0x7f1a0c09fae0 2019-12-05T05:09:22.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:22.330+0000 7f1a13fbd700 1 -- 172.21.15.42:0/1635659525 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1a0c09fae0 msgr2=0x7f1a0c14fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:09:22.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:22.330+0000 7f1a13fbd700 1 --2- 172.21.15.42:0/1635659525 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1a0c09fae0 0x7f1a0c14fff0 secure :-1 s=READY pgs=347 cs=0 l=1 rx=0x7f1a04008b10 tx=0x7f1a0400ff70).stop 2019-12-05T05:09:22.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:22.334+0000 7f1a13fbd700 1 -- 172.21.15.42:0/1635659525 shutdown_connections 2019-12-05T05:09:22.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:22.334+0000 7f1a13fbd700 1 --2- 172.21.15.42:0/1635659525 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1a0c09fae0 0x7f1a0c14fff0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:09:22.341 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:22.334+0000 7f1a13fbd700 1 -- 172.21.15.42:0/1635659525 shutdown_connections 2019-12-05T05:09:22.341 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:22.334+0000 7f1a13fbd700 1 -- 172.21.15.42:0/1635659525 wait complete. 2019-12-05T05:09:22.342 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:22.334+0000 7f1a13fbd700 1 -- 172.21.15.42:0/1635659525 >> 172.21.15.42:0/1635659525 conn(0x7f1a0c0aabe0 msgr2=0x7f1a0c0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:09:22.345 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:22.334+0000 7f1a13fbd700 1 Processor -- start 2019-12-05T05:09:22.345 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:22.334+0000 7f1a13fbd700 1 -- start start 2019-12-05T05:09:22.345 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:22.338+0000 7f1a13fbd700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f1a0c08ef40 0x7f1a0c063840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:09:22.346 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:22.338+0000 7f1a13fbd700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1a0c064ec0 0x7f1a0c0652f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:09:22.346 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:22.338+0000 7f1a13fbd700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a0c148860 con 0x7f1a0c1475a0 2019-12-05T05:09:22.346 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:22.338+0000 7f1a13fbd700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a0c064180 con 0x7f1a0c08ef40 2019-12-05T05:09:22.346 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:22.338+0000 7f1a13fbd700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a0c064c20 con 0x7f1a0c064ec0 2019-12-05T05:09:22.346 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:22.338+0000 7f1a1255a700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f1a0c08ef40 0x7f1a0c063840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:09:22.346 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:22.338+0000 7f1a12d5b700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f1a0c1475a0 0x7f1a0c0a5d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:56658/0 (socket says 172.21.15.42:56658) 2019-12-05T05:09:22.346 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:22.338+0000 7f1a1255a700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f1a0c08ef40 0x7f1a0c063840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:56616/0 (socket says 172.21.15.42:56616) 2019-12-05T05:09:22.347 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:22.338+0000 7f1a12d5b700 1 -- 172.21.15.42:0/2361722532 learned_addr learned my addr 172.21.15.42:0/2361722532 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:09:22.347 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:22.338+0000 7f1a11d59700 1 --2- 172.21.15.42:0/2361722532 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1a0c064ec0 0x7f1a0c0652f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:09:22.347 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:22.338+0000 7f1a11d59700 1 -- 172.21.15.42:0/2361722532 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1a0c064ec0 msgr2=0x7f1a0c0652f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2019-12-05T05:09:22.347 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:22.338+0000 7f1a11d59700 1 -- 172.21.15.42:0/2361722532 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1a0c064ec0 msgr2=0x7f1a0c0652f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2019-12-05T05:09:22.347 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:22.338+0000 7f1a11d59700 1 --2- 172.21.15.42:0/2361722532 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1a0c064ec0 0x7f1a0c0652f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-12-05T05:09:22.347 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:22.338+0000 7f1a1255a700 1 -- 172.21.15.42:0/2361722532 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1a0c064ec0 msgr2=0x7f1a0c0652f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:09:22.347 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:22.338+0000 7f1a11d59700 1 --2- 172.21.15.42:0/2361722532 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1a0c064ec0 0x7f1a0c0652f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rx=0 tx=0)._fault waiting 0.200000 2019-12-05T05:09:22.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:22.338+0000 7f1a1255a700 1 --2- 172.21.15.42:0/2361722532 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1a0c064ec0 0x7f1a0c0652f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:09:22.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:22.338+0000 7f1a1255a700 1 -- 172.21.15.42:0/2361722532 >> v1:172.21.15.42:6789/0 conn(0x7f1a0c1475a0 legacy=0x7f1a0c0a5d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:09:22.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:22.338+0000 7f1a02ffd700 1 -- 172.21.15.42:0/2361722532 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1310145153 0 0) 0x7f1a0c148860 con 0x7f1a0c1475a0 2019-12-05T05:09:22.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:22.338+0000 7f1a1255a700 1 -- 172.21.15.42:0/2361722532 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a0c063d90 con 0x7f1a0c08ef40 2019-12-05T05:09:22.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:22.338+0000 7f1a1255a700 1 --2- 172.21.15.42:0/2361722532 >> v2:172.21.15.42:3300/0 conn(0x7f1a0c08ef40 0x7f1a0c063840 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f1a0c064340 tx=0x7f1a08010840).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:22.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:22.338+0000 7f1a1255a700 1 --2- 172.21.15.42:0/2361722532 >> v2:172.21.15.42:3300/0 conn(0x7f1a0c08ef40 0x7f1a0c063840 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f1a0c064340 tx=0x7f1a08010840).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:22.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:22.338+0000 7f1a1255a700 1 --2- 172.21.15.42:0/2361722532 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1a0c08ef40 0x7f1a0c063840 secure :-1 s=READY pgs=349 cs=0 l=1 rx=0x7f1a0c064340 tx=0x7f1a08010840).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:09:22.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:22.338+0000 7f1a1255a700 1 --2- 172.21.15.42:0/2361722532 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1a0c08ef40 0x7f1a0c063840 secure :-1 s=THROTTLE_DONE pgs=349 cs=0 l=1 rx=0x7f1a0c064340 tx=0x7f1a08010840).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:22.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:22.338+0000 7f1a02ffd700 1 -- 172.21.15.42:0/2361722532 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1a08006a40 con 0x7f1a0c08ef40 2019-12-05T05:09:22.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:22.338+0000 7f1a1255a700 1 --2- 172.21.15.42:0/2361722532 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1a0c08ef40 0x7f1a0c063840 secure :-1 s=THROTTLE_DONE pgs=349 cs=0 l=1 rx=0x7f1a0c064340 tx=0x7f1a08010840).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:22.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:22.338+0000 7f1a02ffd700 1 -- 172.21.15.42:0/2361722532 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a08006660 con 0x7f1a0c08ef40 2019-12-05T05:09:22.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:22.338+0000 7f1a1255a700 1 --2- 172.21.15.42:0/2361722532 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1a0c08ef40 0x7f1a0c063840 secure :-1 s=THROTTLE_DONE pgs=349 cs=0 l=1 rx=0x7f1a0c064340 tx=0x7f1a08010840).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:22.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:22.338+0000 7f1a13fbd700 1 -- 172.21.15.42:0/2361722532 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a0c130630 con 0x7f1a0c08ef40 2019-12-05T05:09:22.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:22.338+0000 7f1a02ffd700 1 -- 172.21.15.42:0/2361722532 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1a08011040 con 0x7f1a0c08ef40 2019-12-05T05:09:22.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:22.338+0000 7f1a13fbd700 1 -- 172.21.15.42:0/2361722532 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a0c147bf0 con 0x7f1a0c08ef40 2019-12-05T05:09:22.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:22.338+0000 7f1a1255a700 1 --2- 172.21.15.42:0/2361722532 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1a0c08ef40 0x7f1a0c063840 secure :-1 s=THROTTLE_DONE pgs=349 cs=0 l=1 rx=0x7f1a0c064340 tx=0x7f1a08010840).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:22.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:22.342+0000 7f1a02ffd700 1 -- 172.21.15.42:0/2361722532 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f1a08011360 con 0x7f1a0c08ef40 2019-12-05T05:09:22.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:22.342+0000 7f1a1255a700 1 --2- 172.21.15.42:0/2361722532 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1a0c08ef40 0x7f1a0c063840 secure :-1 s=THROTTLE_DONE pgs=349 cs=0 l=1 rx=0x7f1a0c064340 tx=0x7f1a08010840).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:22.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:22.342+0000 7f1a02ffd700 1 --2- 172.21.15.42:0/2361722532 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f19f403acc0 0x7f19f403d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:09:22.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:22.342+0000 7f1a02ffd700 1 -- 172.21.15.42:0/2361722532 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f1a0804b0b0 con 0x7f1a0c08ef40 2019-12-05T05:09:22.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:22.342+0000 7f1a11d59700 1 --2- 172.21.15.42:0/2361722532 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f19f403acc0 0x7f19f403d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:09:22.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:22.342+0000 7f1a11d59700 1 --2- 172.21.15.42:0/2361722532 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f19f403acc0 0x7f19f403d1d0 crc :-1 s=READY pgs=158 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:09:22.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:22.346+0000 7f1a13fbd700 1 -- 172.21.15.42:0/2361722532 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a0c05d110 con 0x7f1a0c08ef40 2019-12-05T05:09:22.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:22.350+0000 7f1a1255a700 1 --2- 172.21.15.42:0/2361722532 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1a0c08ef40 0x7f1a0c063840 secure :-1 s=THROTTLE_DONE pgs=349 cs=0 l=1 rx=0x7f1a0c064340 tx=0x7f1a08010840).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:22.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:22.350+0000 7f1a02ffd700 1 -- 172.21.15.42:0/2361722532 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f1a08032110 con 0x7f1a0c08ef40 2019-12-05T05:09:22.515 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:22.514+0000 7f1a13fbd700 1 -- 172.21.15.42:0/2361722532 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1a0c0aadf0 con 0x7f1a0c08ef40 2019-12-05T05:09:22.516 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:22.514+0000 7f1a1255a700 1 --2- 172.21.15.42:0/2361722532 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1a0c08ef40 0x7f1a0c063840 secure :-1 s=THROTTLE_DONE pgs=349 cs=0 l=1 rx=0x7f1a0c064340 tx=0x7f1a08010840).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:22.517 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:22.514+0000 7f1a02ffd700 1 -- 172.21.15.42:0/2361722532 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+2102 (secure 0 0 0) 0x7f1a08032370 con 0x7f1a0c08ef40 2019-12-05T05:09:22.517 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 29 2019-12-05T05:09:22.517 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:09:22.518 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4978,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/3650564018","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3650564018},{"type":"v1","addr":"172.21.15.42:6835","nonce":3650564018}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":29}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:07:12.263209+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4831},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4831":{"gid":4831,"name":"b","rank":0,"incarnation":24,"state":"up:active","state_seq":39,"addr":"172.21.15.40:6833/90248709","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":90248709},{"type":"v1","addr":"172.21.15.40:6833","nonce":90248709}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:09:22.520 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:22.514+0000 7f1a13fbd700 1 -- 172.21.15.42:0/2361722532 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f19f403acc0 msgr2=0x7f19f403d1d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:09:22.520 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:22.514+0000 7f1a13fbd700 1 --2- 172.21.15.42:0/2361722532 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f19f403acc0 0x7f19f403d1d0 crc :-1 s=READY pgs=158 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:09:22.520 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:22.514+0000 7f1a13fbd700 1 -- 172.21.15.42:0/2361722532 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1a0c08ef40 msgr2=0x7f1a0c063840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:09:22.520 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:22.514+0000 7f1a13fbd700 1 --2- 172.21.15.42:0/2361722532 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1a0c08ef40 0x7f1a0c063840 secure :-1 s=READY pgs=349 cs=0 l=1 rx=0x7f1a0c064340 tx=0x7f1a08010840).stop 2019-12-05T05:09:22.521 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:22.514+0000 7f1a13fbd700 1 -- 172.21.15.42:0/2361722532 shutdown_connections 2019-12-05T05:09:22.521 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:22.514+0000 7f1a13fbd700 1 --2- 172.21.15.42:0/2361722532 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f19f403acc0 0x7f19f403d1d0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:09:22.521 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:22.514+0000 7f1a13fbd700 1 --2- 172.21.15.42:0/2361722532 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1a0c08ef40 0x7f1a0c063840 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:09:22.521 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:22.514+0000 7f1a13fbd700 1 --2- 172.21.15.42:0/2361722532 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1a0c064ec0 0x7f1a0c0652f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:09:22.521 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:22.518+0000 7f1a13fbd700 1 -- 172.21.15.42:0/2361722532 shutdown_connections 2019-12-05T05:09:22.521 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:22.518+0000 7f1a13fbd700 1 -- 172.21.15.42:0/2361722532 wait complete. 2019-12-05T05:09:22.522 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:22.518+0000 7f1a13fbd700 1 -- 172.21.15.42:0/2361722532 >> 172.21.15.42:0/2361722532 conn(0x7f1a0c0a6fc0 msgr2=0x7f1a0c07e4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:09:22.535 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T05:09:24.536 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:09:24.657 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.618+0000 7fcceec20700 1 Processor -- start 2019-12-05T05:09:24.658 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.618+0000 7fcceec20700 1 -- start start 2019-12-05T05:09:24.658 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.618+0000 7fcceec20700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcce80a7740 0x7fcce808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:09:24.658 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.618+0000 7fcceec20700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fcce807e720 0x7fcce807eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:09:24.658 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.618+0000 7fcceec20700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fcce809f2e0 0x7fcce809f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:09:24.658 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.618+0000 7fcceec20700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcce80a0b00 con 0x7fcce807e720 2019-12-05T05:09:24.658 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.618+0000 7fcceec20700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcce808e5b0 con 0x7fcce809f2e0 2019-12-05T05:09:24.659 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.618+0000 7fcceec20700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcce808e850 con 0x7fcce80a7740 2019-12-05T05:09:24.659 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.618+0000 7fcced9be700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fcce809f2e0 0x7fcce809f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:09:24.659 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.618+0000 7fcced9be700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fcce809f2e0 0x7fcce809f710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:56650/0 (socket says 172.21.15.42:56650) 2019-12-05T05:09:24.659 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.618+0000 7fcced9be700 1 -- 172.21.15.42:0/4273856060 learned_addr learned my addr 172.21.15.42:0/4273856060 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:09:24.659 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.618+0000 7fccec9bc700 1 --2- 172.21.15.42:0/4273856060 >> v2:172.21.15.40:3300/0 conn(0x7fcce807e720 0x7fcce807eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:09:24.659 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.618+0000 7fcced1bd700 1 --2- 172.21.15.42:0/4273856060 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcce80a7740 0x7fcce808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:09:24.659 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.618+0000 7fcced9be700 1 -- 172.21.15.42:0/4273856060 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcce80a7740 msgr2=0x7fcce808df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:09:24.659 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.618+0000 7fcced9be700 1 --2- 172.21.15.42:0/4273856060 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcce80a7740 0x7fcce808df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:09:24.660 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.618+0000 7fcced9be700 1 -- 172.21.15.42:0/4273856060 >> v2:172.21.15.40:3300/0 conn(0x7fcce807e720 msgr2=0x7fcce807eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:09:24.660 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.618+0000 7fcced9be700 1 --2- 172.21.15.42:0/4273856060 >> v2:172.21.15.40:3300/0 conn(0x7fcce807e720 0x7fcce807eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:09:24.660 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.618+0000 7fcced9be700 1 -- 172.21.15.42:0/4273856060 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcce807f0a0 con 0x7fcce809f2e0 2019-12-05T05:09:24.660 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.618+0000 7fcced1bd700 1 --2- 172.21.15.42:0/4273856060 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcce80a7740 0x7fcce808df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T05:09:24.664 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.618+0000 7fcced9be700 1 --2- 172.21.15.42:0/4273856060 >> v2:172.21.15.42:3300/0 conn(0x7fcce809f2e0 0x7fcce809f710 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fcce808e770 tx=0x7fcce400fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:24.665 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.618+0000 7fcced9be700 1 --2- 172.21.15.42:0/4273856060 >> v2:172.21.15.42:3300/0 conn(0x7fcce809f2e0 0x7fcce809f710 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fcce808e770 tx=0x7fcce400fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:24.665 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.618+0000 7fcced9be700 1 --2- 172.21.15.42:0/4273856060 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcce809f2e0 0x7fcce809f710 secure :-1 s=READY pgs=350 cs=0 l=1 rx=0x7fcce808e770 tx=0x7fcce400fdb0).ready entity=mon.1 client_cookie=1b28ef2dcc3acfd server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:09:24.665 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.618+0000 7fcced9be700 1 --2- 172.21.15.42:0/4273856060 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcce809f2e0 0x7fcce809f710 secure :-1 s=THROTTLE_DONE pgs=350 cs=0 l=1 rx=0x7fcce808e770 tx=0x7fcce400fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:24.665 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.618+0000 7fcced9be700 1 --2- 172.21.15.42:0/4273856060 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcce809f2e0 0x7fcce809f710 secure :-1 s=THROTTLE_DONE pgs=350 cs=0 l=1 rx=0x7fcce808e770 tx=0x7fcce400fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:24.667 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.618+0000 7fccdbfff700 1 -- 172.21.15.42:0/4273856060 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcce4008c90 con 0x7fcce809f2e0 2019-12-05T05:09:24.667 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.618+0000 7fcced9be700 1 --2- 172.21.15.42:0/4273856060 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcce809f2e0 0x7fcce809f710 secure :-1 s=THROTTLE_DONE pgs=350 cs=0 l=1 rx=0x7fcce808e770 tx=0x7fcce400fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:24.668 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.618+0000 7fccdbfff700 1 -- 172.21.15.42:0/4273856060 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcce40088b0 con 0x7fcce809f2e0 2019-12-05T05:09:24.668 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.618+0000 7fccdbfff700 1 -- 172.21.15.42:0/4273856060 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcce4011b50 con 0x7fcce809f2e0 2019-12-05T05:09:24.668 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.618+0000 7fcceec20700 1 -- 172.21.15.42:0/4273856060 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcce809f2e0 msgr2=0x7fcce809f710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:09:24.668 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.618+0000 7fcceec20700 1 --2- 172.21.15.42:0/4273856060 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcce809f2e0 0x7fcce809f710 secure :-1 s=READY pgs=350 cs=0 l=1 rx=0x7fcce808e770 tx=0x7fcce400fdb0).stop 2019-12-05T05:09:24.670 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.618+0000 7fcceec20700 1 -- 172.21.15.42:0/4273856060 shutdown_connections 2019-12-05T05:09:24.670 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.618+0000 7fcceec20700 1 --2- 172.21.15.42:0/4273856060 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcce809f2e0 0x7fcce809f710 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:09:24.670 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.618+0000 7fcceec20700 1 --2- 172.21.15.42:0/4273856060 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcce80a7740 0x7fcce808df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:09:24.671 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.618+0000 7fcceec20700 1 --2- 172.21.15.42:0/4273856060 >> v2:172.21.15.40:3300/0 conn(0x7fcce807e720 0x7fcce807eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:09:24.671 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.618+0000 7fcceec20700 1 -- 172.21.15.42:0/4273856060 shutdown_connections 2019-12-05T05:09:24.671 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.618+0000 7fcceec20700 1 -- 172.21.15.42:0/4273856060 wait complete. 2019-12-05T05:09:24.671 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.618+0000 7fcceec20700 1 -- 172.21.15.42:0/4273856060 >> 172.21.15.42:0/4273856060 conn(0x7fcce80aabe0 msgr2=0x7fcce80a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:09:24.671 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.622+0000 7fcceec20700 1 Processor -- start 2019-12-05T05:09:24.671 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.622+0000 7fcceec20700 1 -- start start 2019-12-05T05:09:24.671 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.622+0000 7fcceec20700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fcce81245a0 0x7fcce8063190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:09:24.672 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.622+0000 7fcceec20700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcce8064740 con 0x7fcce80636e0 2019-12-05T05:09:24.672 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.622+0000 7fcceec20700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcce8065ca0 con 0x7fcce8064f50 2019-12-05T05:09:24.673 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.622+0000 7fcceec20700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcce8064a10 con 0x7fcce81245a0 2019-12-05T05:09:24.673 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.622+0000 7fcced1bd700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fcce8064f50 0x7fcce8133690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:56690/0 (socket says 172.21.15.42:56690) 2019-12-05T05:09:24.673 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.622+0000 7fcced1bd700 1 -- 172.21.15.42:0/1927184956 learned_addr learned my addr 172.21.15.42:0/1927184956 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:09:24.673 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.622+0000 7fccec9bc700 1 --2- 172.21.15.42:0/1927184956 >> v2:172.21.15.40:3300/0 conn(0x7fcce81245a0 0x7fcce8063190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:09:24.673 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.622+0000 7fccd9ffb700 1 -- 172.21.15.42:0/1927184956 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3354637647 0 0) 0x7fcce8065ca0 con 0x7fcce8064f50 2019-12-05T05:09:24.673 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.622+0000 7fccd9ffb700 1 -- 172.21.15.42:0/1927184956 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fccc80030b0 con 0x7fcce8064f50 2019-12-05T05:09:24.673 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.622+0000 7fccd9ffb700 1 -- 172.21.15.42:0/1927184956 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 2378635371 0 0) 0x7fccc80030b0 con 0x7fcce8064f50 2019-12-05T05:09:24.674 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.622+0000 7fccd9ffb700 1 -- 172.21.15.42:0/1927184956 --> v1:172.21.15.42:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcce8065ca0 con 0x7fcce8064f50 2019-12-05T05:09:24.674 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.622+0000 7fccd9ffb700 1 -- 172.21.15.42:0/1927184956 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7fcce0002940 con 0x7fcce8064f50 2019-12-05T05:09:24.674 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.622+0000 7fccd9ffb700 1 -- 172.21.15.42:0/1927184956 >> v2:172.21.15.40:3300/0 conn(0x7fcce81245a0 msgr2=0x7fcce8063190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:09:24.674 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.622+0000 7fccd9ffb700 1 --2- 172.21.15.42:0/1927184956 >> v2:172.21.15.40:3300/0 conn(0x7fcce81245a0 0x7fcce8063190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:09:24.674 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.622+0000 7fccd9ffb700 1 -- 172.21.15.42:0/1927184956 >> v1:172.21.15.42:6789/0 conn(0x7fcce8064f50 legacy=0x7fcce8133690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:09:24.674 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.622+0000 7fccd9ffb700 1 -- 172.21.15.42:0/1927184956 >> v1:172.21.15.40:6789/0 conn(0x7fcce80636e0 legacy=0x7fcce80702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:09:24.674 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.622+0000 7fccd9ffb700 1 --2- 172.21.15.42:0/1927184956 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fccc80038a0 0x7fccc8003d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:09:24.675 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.622+0000 7fccd9ffb700 1 --2- 172.21.15.42:0/1927184956 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fccc8004420 0x7fccc8004870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:09:24.675 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.622+0000 7fccd9ffb700 1 --2- 172.21.15.42:0/1927184956 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fccc8004ed0 0x7fccc800b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:09:24.675 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.622+0000 7fccd9ffb700 1 -- 172.21.15.42:0/1927184956 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcce8064a10 con 0x7fccc8004420 2019-12-05T05:09:24.675 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.622+0000 7fccd9ffb700 1 -- 172.21.15.42:0/1927184956 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccc800bda0 con 0x7fccc8004ed0 2019-12-05T05:09:24.675 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.622+0000 7fccd9ffb700 1 -- 172.21.15.42:0/1927184956 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccc800bfe0 con 0x7fccc80038a0 2019-12-05T05:09:24.675 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.622+0000 7fcced9be700 1 --2- 172.21.15.42:0/1927184956 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fccc8004ed0 0x7fccc800b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:09:24.675 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.622+0000 7fccec9bc700 1 --2- 172.21.15.42:0/1927184956 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fccc8004420 0x7fccc8004870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:09:24.676 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.622+0000 7fcced1bd700 1 --2- 172.21.15.42:0/1927184956 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fccc80038a0 0x7fccc8003d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:09:24.676 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.622+0000 7fcced1bd700 1 -- 172.21.15.42:0/1927184956 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fccc80038a0 msgr2=0x7fccc8003d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2019-12-05T05:09:24.676 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.622+0000 7fcced1bd700 1 -- 172.21.15.42:0/1927184956 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fccc80038a0 msgr2=0x7fccc8003d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2019-12-05T05:09:24.676 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.622+0000 7fcced1bd700 1 --2- 172.21.15.42:0/1927184956 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fccc80038a0 0x7fccc8003d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-12-05T05:09:24.676 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.622+0000 7fcced1bd700 1 --2- 172.21.15.42:0/1927184956 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fccc80038a0 0x7fccc8003d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rx=0 tx=0)._fault waiting 0.200000 2019-12-05T05:09:24.676 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.622+0000 7fcced9be700 1 -- 172.21.15.42:0/1927184956 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fccc80038a0 msgr2=0x7fccc8003d10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2019-12-05T05:09:24.677 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.622+0000 7fcced9be700 1 --2- 172.21.15.42:0/1927184956 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fccc80038a0 0x7fccc8003d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:09:24.677 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.622+0000 7fcced9be700 1 -- 172.21.15.42:0/1927184956 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fccc8004420 msgr2=0x7fccc8004870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:09:24.677 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.622+0000 7fcced9be700 1 --2- 172.21.15.42:0/1927184956 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fccc8004420 0x7fccc8004870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:09:24.677 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.622+0000 7fcced9be700 1 -- 172.21.15.42:0/1927184956 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcce8070ac0 con 0x7fccc8004ed0 2019-12-05T05:09:24.677 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.622+0000 7fcceec20700 1 -- 172.21.15.42:0/1927184956 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcce8070d90 con 0x7fccc8004ed0 2019-12-05T05:09:24.677 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.622+0000 7fcceec20700 1 -- 172.21.15.42:0/1927184956 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcce8078540 con 0x7fccc8004ed0 2019-12-05T05:09:24.677 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.622+0000 7fcced9be700 1 --2- 172.21.15.42:0/1927184956 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fccc8004ed0 0x7fccc800b670 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fcce40101c0 tx=0x7fcce4008be0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:24.678 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.622+0000 7fcced9be700 1 -- 172.21.15.42:0/1927184956 reap_dead start 2019-12-05T05:09:24.678 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.626+0000 7fcced9be700 1 --2- 172.21.15.42:0/1927184956 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fccc8004ed0 0x7fccc800b670 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fcce40101c0 tx=0x7fcce4008be0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:24.678 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.626+0000 7fcced9be700 1 --2- 172.21.15.42:0/1927184956 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fccc8004ed0 0x7fccc800b670 secure :-1 s=READY pgs=352 cs=0 l=1 rx=0x7fcce40101c0 tx=0x7fcce4008be0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:09:24.678 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.626+0000 7fcced9be700 1 --2- 172.21.15.42:0/1927184956 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fccc8004ed0 0x7fccc800b670 secure :-1 s=THROTTLE_DONE pgs=352 cs=0 l=1 rx=0x7fcce40101c0 tx=0x7fcce4008be0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:24.678 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.626+0000 7fccd9ffb700 1 -- 172.21.15.42:0/1927184956 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcce400f470 con 0x7fccc8004ed0 2019-12-05T05:09:24.678 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.626+0000 7fcced9be700 1 --2- 172.21.15.42:0/1927184956 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fccc8004ed0 0x7fccc800b670 secure :-1 s=THROTTLE_DONE pgs=352 cs=0 l=1 rx=0x7fcce40101c0 tx=0x7fcce4008be0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:24.678 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.626+0000 7fcced9be700 1 --2- 172.21.15.42:0/1927184956 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fccc8004ed0 0x7fccc800b670 secure :-1 s=THROTTLE_DONE pgs=352 cs=0 l=1 rx=0x7fcce40101c0 tx=0x7fcce4008be0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:24.679 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.626+0000 7fccd9ffb700 1 -- 172.21.15.42:0/1927184956 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcce4011770 con 0x7fccc8004ed0 2019-12-05T05:09:24.679 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.626+0000 7fccd9ffb700 1 -- 172.21.15.42:0/1927184956 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcce4005730 con 0x7fccc8004ed0 2019-12-05T05:09:24.679 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.626+0000 7fcced9be700 1 --2- 172.21.15.42:0/1927184956 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fccc8004ed0 0x7fccc800b670 secure :-1 s=THROTTLE_DONE pgs=352 cs=0 l=1 rx=0x7fcce40101c0 tx=0x7fcce4008be0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:24.679 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.626+0000 7fccd9ffb700 1 -- 172.21.15.42:0/1927184956 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fcce4005930 con 0x7fccc8004ed0 2019-12-05T05:09:24.679 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.626+0000 7fcced9be700 1 --2- 172.21.15.42:0/1927184956 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fccc8004ed0 0x7fccc800b670 secure :-1 s=THROTTLE_DONE pgs=352 cs=0 l=1 rx=0x7fcce40101c0 tx=0x7fcce4008be0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:24.679 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.626+0000 7fccd9ffb700 1 --2- 172.21.15.42:0/1927184956 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fccc803eaa0 0x7fccc8040fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:09:24.679 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.626+0000 7fccd9ffb700 1 -- 172.21.15.42:0/1927184956 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6289+0+0 (secure 0 0 0) 0x7fcce408f1a0 con 0x7fccc8004ed0 2019-12-05T05:09:24.680 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.626+0000 7fcced1bd700 1 --2- 172.21.15.42:0/1927184956 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fccc803eaa0 0x7fccc8040fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:09:24.680 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.626+0000 7fcced1bd700 1 --2- 172.21.15.42:0/1927184956 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fccc803eaa0 0x7fccc8040fb0 crc :-1 s=READY pgs=159 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:09:24.680 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.634+0000 7fcceec20700 1 -- 172.21.15.42:0/1927184956 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccc0003200 con 0x7fccc8004ed0 2019-12-05T05:09:24.680 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.642+0000 7fcced9be700 1 --2- 172.21.15.42:0/1927184956 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fccc8004ed0 0x7fccc800b670 secure :-1 s=THROTTLE_DONE pgs=352 cs=0 l=1 rx=0x7fcce40101c0 tx=0x7fcce4008be0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:24.680 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.642+0000 7fccd9ffb700 1 -- 172.21.15.42:0/1927184956 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fcce4008de0 con 0x7fccc8004ed0 2019-12-05T05:09:24.805 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.802+0000 7fcceec20700 1 -- 172.21.15.42:0/1927184956 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fccc0003630 con 0x7fccc8004ed0 2019-12-05T05:09:24.807 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.802+0000 7fcced9be700 1 --2- 172.21.15.42:0/1927184956 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fccc8004ed0 0x7fccc800b670 secure :-1 s=THROTTLE_DONE pgs=352 cs=0 l=1 rx=0x7fcce40101c0 tx=0x7fcce4008be0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:24.807 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.802+0000 7fccd9ffb700 1 -- 172.21.15.42:0/1927184956 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+1771 (secure 0 0 0) 0x7fcce4076200 con 0x7fccc8004ed0 2019-12-05T05:09:24.807 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 30 2019-12-05T05:09:24.807 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:09:24.808 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:09:23.258857+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4978},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4978":{"gid":4978,"name":"a","rank":0,"incarnation":30,"state":"up:replay","state_seq":2,"addr":"172.21.15.42:6835/3650564018","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3650564018},{"type":"v1","addr":"172.21.15.42:6835","nonce":3650564018}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:09:24.809 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.806+0000 7fcceec20700 1 -- 172.21.15.42:0/1927184956 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fccc803eaa0 msgr2=0x7fccc8040fb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:09:24.809 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.806+0000 7fcceec20700 1 --2- 172.21.15.42:0/1927184956 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fccc803eaa0 0x7fccc8040fb0 crc :-1 s=READY pgs=159 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:09:24.810 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.806+0000 7fcceec20700 1 -- 172.21.15.42:0/1927184956 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fccc8004ed0 msgr2=0x7fccc800b670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:09:24.810 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.806+0000 7fcceec20700 1 --2- 172.21.15.42:0/1927184956 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fccc8004ed0 0x7fccc800b670 secure :-1 s=READY pgs=352 cs=0 l=1 rx=0x7fcce40101c0 tx=0x7fcce4008be0).stop 2019-12-05T05:09:24.810 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.806+0000 7fcceec20700 1 -- 172.21.15.42:0/1927184956 shutdown_connections 2019-12-05T05:09:24.810 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.806+0000 7fcceec20700 1 --2- 172.21.15.42:0/1927184956 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fccc803eaa0 0x7fccc8040fb0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:09:24.824 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.806+0000 7fcceec20700 1 --2- 172.21.15.42:0/1927184956 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fccc8004ed0 0x7fccc800b670 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:09:24.825 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.806+0000 7fcceec20700 1 -- 172.21.15.42:0/1927184956 shutdown_connections 2019-12-05T05:09:24.828 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.806+0000 7fcceec20700 1 -- 172.21.15.42:0/1927184956 wait complete. 2019-12-05T05:09:24.828 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.806+0000 7fcceec20700 1 -- 172.21.15.42:0/1927184956 >> 172.21.15.42:0/1927184956 conn(0x7fcce807e4f0 msgr2=0x7fcce809f460 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:09:24.829 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b down, removed from mdsmap 2019-12-05T05:09:24.829 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2019-12-05T05:09:24.829 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:09:24.924 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.914+0000 7f6302f88700 1 Processor -- start 2019-12-05T05:09:24.924 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.914+0000 7f6302f88700 1 -- start start 2019-12-05T05:09:24.924 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.918+0000 7f6302f88700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f62fc0a7740 0x7f62fc08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:09:24.924 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.918+0000 7f6302f88700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f62fc07e720 0x7f62fc07eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:09:24.925 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.918+0000 7f6302f88700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62fc0a0710 con 0x7f62fc09f2e0 2019-12-05T05:09:24.925 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.918+0000 7f6302f88700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62fc0a0b00 con 0x7f62fc07e720 2019-12-05T05:09:24.925 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.918+0000 7f6302f88700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62fc08e5b0 con 0x7f62fc0a7740 2019-12-05T05:09:24.925 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.918+0000 7f6301d26700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f62fc09f2e0 0x7f62fc09f710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:56708/0 (socket says 172.21.15.42:56708) 2019-12-05T05:09:24.925 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.918+0000 7f6301d26700 1 -- 172.21.15.42:0/1487288596 learned_addr learned my addr 172.21.15.42:0/1487288596 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:09:24.925 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.918+0000 7f6301525700 1 --2- 172.21.15.42:0/1487288596 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f62fc0a7740 0x7f62fc08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:09:24.926 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.918+0000 7f6300d24700 1 --2- 172.21.15.42:0/1487288596 >> v2:172.21.15.40:3300/0 conn(0x7f62fc07e720 0x7f62fc07eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:09:24.926 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.918+0000 7f62fbfff700 1 -- 172.21.15.42:0/1487288596 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4224324611 0 0) 0x7f62fc0a0710 con 0x7f62fc09f2e0 2019-12-05T05:09:24.926 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.918+0000 7f62fbfff700 1 -- 172.21.15.42:0/1487288596 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62e40030b0 con 0x7f62fc09f2e0 2019-12-05T05:09:24.926 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.918+0000 7f62fbfff700 1 -- 172.21.15.42:0/1487288596 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 570320858 0 0) 0x7f62e40030b0 con 0x7f62fc09f2e0 2019-12-05T05:09:24.926 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.918+0000 7f62fbfff700 1 -- 172.21.15.42:0/1487288596 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f62fc0a7740 msgr2=0x7f62fc08df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:09:24.926 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.918+0000 7f62fbfff700 1 --2- 172.21.15.42:0/1487288596 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f62fc0a7740 0x7f62fc08df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:09:24.926 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.918+0000 7f62fbfff700 1 -- 172.21.15.42:0/1487288596 >> v2:172.21.15.40:3300/0 conn(0x7f62fc07e720 msgr2=0x7f62fc07eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:09:24.927 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.918+0000 7f62fbfff700 1 --2- 172.21.15.42:0/1487288596 >> v2:172.21.15.40:3300/0 conn(0x7f62fc07e720 0x7f62fc07eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:09:24.927 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.918+0000 7f62fbfff700 1 -- 172.21.15.42:0/1487288596 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62fc07f0a0 con 0x7f62fc09f2e0 2019-12-05T05:09:24.927 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.918+0000 7f62fbfff700 1 -- 172.21.15.42:0/1487288596 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7f62ec001750 con 0x7f62fc09f2e0 2019-12-05T05:09:24.927 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.918+0000 7f62fbfff700 1 -- 172.21.15.42:0/1487288596 >> v1:172.21.15.42:6789/0 conn(0x7f62fc09f2e0 legacy=0x7f62fc09f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:09:24.927 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.918+0000 7f62fbfff700 1 --2- 172.21.15.42:0/1487288596 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f62e40036e0 0x7f62e4003b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:09:24.927 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.918+0000 7f62fbfff700 1 --2- 172.21.15.42:0/1487288596 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f62e4004420 0x7f62e4004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:09:24.927 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.918+0000 7f62fbfff700 1 --2- 172.21.15.42:0/1487288596 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f62e4004da0 0x7f62e404f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:09:24.928 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.918+0000 7f62fbfff700 1 -- 172.21.15.42:0/1487288596 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62fc0a0b00 con 0x7f62e4004da0 2019-12-05T05:09:24.928 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.918+0000 7f62fbfff700 1 -- 172.21.15.42:0/1487288596 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62fc08e5b0 con 0x7f62e4004420 2019-12-05T05:09:24.928 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.918+0000 7f62fbfff700 1 -- 172.21.15.42:0/1487288596 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62e404fbd0 con 0x7f62e40036e0 2019-12-05T05:09:24.928 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.918+0000 7f6300d24700 1 --2- 172.21.15.42:0/1487288596 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f62e4004420 0x7f62e4004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:09:24.928 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.918+0000 7f6301525700 1 --2- 172.21.15.42:0/1487288596 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f62e40036e0 0x7f62e4003b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:09:24.928 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.918+0000 7f6301d26700 1 --2- 172.21.15.42:0/1487288596 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f62e4004da0 0x7f62e404f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:09:24.928 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.918+0000 7f6300d24700 1 -- 172.21.15.42:0/1487288596 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f62e40036e0 msgr2=0x7f62e4003b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:09:24.929 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.918+0000 7f6300d24700 1 --2- 172.21.15.42:0/1487288596 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f62e40036e0 0x7f62e4003b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:09:24.929 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.918+0000 7f6300d24700 1 -- 172.21.15.42:0/1487288596 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f62e4004da0 msgr2=0x7f62e404f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:09:24.929 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.918+0000 7f6300d24700 1 --2- 172.21.15.42:0/1487288596 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f62e4004da0 0x7f62e404f680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:09:24.929 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.918+0000 7f6300d24700 1 -- 172.21.15.42:0/1487288596 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62e404fd90 con 0x7f62e4004420 2019-12-05T05:09:24.929 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.918+0000 7f6300d24700 1 --2- 172.21.15.42:0/1487288596 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f62e4004420 0x7f62e4004850 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f62f00032b0 tx=0x7f62f0010360).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:24.929 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.918+0000 7f6300d24700 1 --2- 172.21.15.42:0/1487288596 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f62e4004420 0x7f62e4004850 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f62f00032b0 tx=0x7f62f0010360).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:24.929 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.918+0000 7f6300d24700 1 --2- 172.21.15.42:0/1487288596 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f62e4004420 0x7f62e4004850 secure :-1 s=READY pgs=354 cs=0 l=1 rx=0x7f62f00032b0 tx=0x7f62f0010360).ready entity=mon.1 client_cookie=c7e4b935d2e7477e server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:09:24.930 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.918+0000 7f6300d24700 1 --2- 172.21.15.42:0/1487288596 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f62e4004420 0x7f62e4004850 secure :-1 s=THROTTLE_DONE pgs=354 cs=0 l=1 rx=0x7f62f00032b0 tx=0x7f62f0010360).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:24.930 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.918+0000 7f62fbfff700 1 -- 172.21.15.42:0/1487288596 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f62f0008d60 con 0x7f62e4004420 2019-12-05T05:09:24.930 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.918+0000 7f6300d24700 1 --2- 172.21.15.42:0/1487288596 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f62e4004420 0x7f62e4004850 secure :-1 s=THROTTLE_DONE pgs=354 cs=0 l=1 rx=0x7f62f00032b0 tx=0x7f62f0010360).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:24.930 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.918+0000 7f6300d24700 1 --2- 172.21.15.42:0/1487288596 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f62e4004420 0x7f62e4004850 secure :-1 s=THROTTLE_DONE pgs=354 cs=0 l=1 rx=0x7f62f00032b0 tx=0x7f62f0010360).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:24.930 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.918+0000 7f62fbfff700 1 -- 172.21.15.42:0/1487288596 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62f0010cb0 con 0x7f62e4004420 2019-12-05T05:09:24.930 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.918+0000 7f62fbfff700 1 -- 172.21.15.42:0/1487288596 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f62f00114a0 con 0x7f62e4004420 2019-12-05T05:09:24.931 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.918+0000 7f6302f88700 1 -- 172.21.15.42:0/1487288596 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f62e4004420 msgr2=0x7f62e4004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:09:24.931 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.918+0000 7f6302f88700 1 --2- 172.21.15.42:0/1487288596 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f62e4004420 0x7f62e4004850 secure :-1 s=READY pgs=354 cs=0 l=1 rx=0x7f62f00032b0 tx=0x7f62f0010360).stop 2019-12-05T05:09:24.932 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.918+0000 7f6301d26700 1 -- 172.21.15.42:0/1487288596 reap_dead start 2019-12-05T05:09:24.932 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.918+0000 7f6302f88700 1 -- 172.21.15.42:0/1487288596 shutdown_connections 2019-12-05T05:09:24.932 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.918+0000 7f6302f88700 1 -- 172.21.15.42:0/1487288596 shutdown_connections 2019-12-05T05:09:24.932 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.918+0000 7f6302f88700 1 -- 172.21.15.42:0/1487288596 wait complete. 2019-12-05T05:09:24.932 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.918+0000 7f6302f88700 1 -- 172.21.15.42:0/1487288596 >> 172.21.15.42:0/1487288596 conn(0x7f62fc0aabe0 msgr2=0x7f62fc0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:09:24.933 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.922+0000 7f6302f88700 1 Processor -- start 2019-12-05T05:09:24.933 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.922+0000 7f6302f88700 1 -- start start 2019-12-05T05:09:24.936 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.922+0000 7f6302f88700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f62fc0a1eb0 0x7f62fc0a22e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:09:24.936 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.922+0000 7f6302f88700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62fc11b7e0 con 0x7f62fc0a5bf0 2019-12-05T05:09:24.936 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.922+0000 7f6302f88700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62fc0a2830 con 0x7f62fc121630 2019-12-05T05:09:24.936 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.922+0000 7f6302f88700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62fc0a6130 con 0x7f62fc0a1eb0 2019-12-05T05:09:24.937 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.922+0000 7f6301525700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f62fc121630 0x7f62fc11afe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:56716/0 (socket says 172.21.15.42:56716) 2019-12-05T05:09:24.937 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.922+0000 7f6301525700 1 -- 172.21.15.42:0/424925729 learned_addr learned my addr 172.21.15.42:0/424925729 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:09:24.937 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.922+0000 7f6301d26700 1 --2- 172.21.15.42:0/424925729 >> v2:172.21.15.42:3300/0 conn(0x7f62fc0a1eb0 0x7f62fc0a22e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:09:24.937 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.922+0000 7f62f9ffb700 1 -- 172.21.15.42:0/424925729 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2799764698 0 0) 0x7f62fc0a2830 con 0x7f62fc121630 2019-12-05T05:09:24.937 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.922+0000 7f62f9ffb700 1 -- 172.21.15.42:0/424925729 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62dc0030b0 con 0x7f62fc121630 2019-12-05T05:09:24.937 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.922+0000 7f62f9ffb700 1 -- 172.21.15.42:0/424925729 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 597545357 0 0) 0x7f62dc0030b0 con 0x7f62fc121630 2019-12-05T05:09:24.938 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.922+0000 7f6301d26700 1 -- 172.21.15.42:0/424925729 >> v1:172.21.15.42:6789/0 conn(0x7f62fc121630 legacy=0x7f62fc11afe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:09:24.938 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.922+0000 7f6301d26700 1 -- 172.21.15.42:0/424925729 >> v1:172.21.15.40:6789/0 conn(0x7f62fc0a5bf0 legacy=0x7f62fc07d890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:09:24.938 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.922+0000 7f6301d26700 1 -- 172.21.15.42:0/424925729 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62fc0a6dc0 con 0x7f62fc0a1eb0 2019-12-05T05:09:24.938 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.922+0000 7f6301d26700 1 --2- 172.21.15.42:0/424925729 >> v2:172.21.15.42:3300/0 conn(0x7f62fc0a1eb0 0x7f62fc0a22e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f62f4001690 tx=0x7f62ec001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:24.938 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.922+0000 7f6301d26700 1 --2- 172.21.15.42:0/424925729 >> v2:172.21.15.42:3300/0 conn(0x7f62fc0a1eb0 0x7f62fc0a22e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f62f4001690 tx=0x7f62ec001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:24.939 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.922+0000 7f6301d26700 1 --2- 172.21.15.42:0/424925729 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f62fc0a1eb0 0x7f62fc0a22e0 secure :-1 s=READY pgs=356 cs=0 l=1 rx=0x7f62f4001690 tx=0x7f62ec001150).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:09:24.939 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.922+0000 7f6301d26700 1 --2- 172.21.15.42:0/424925729 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f62fc0a1eb0 0x7f62fc0a22e0 secure :-1 s=THROTTLE_DONE pgs=356 cs=0 l=1 rx=0x7f62f4001690 tx=0x7f62ec001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:24.939 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.922+0000 7f62f9ffb700 1 -- 172.21.15.42:0/424925729 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f62f4001490 con 0x7f62fc0a1eb0 2019-12-05T05:09:24.940 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.922+0000 7f6301d26700 1 --2- 172.21.15.42:0/424925729 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f62fc0a1eb0 0x7f62fc0a22e0 secure :-1 s=THROTTLE_DONE pgs=356 cs=0 l=1 rx=0x7f62f4001690 tx=0x7f62ec001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:24.940 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.922+0000 7f6301d26700 1 --2- 172.21.15.42:0/424925729 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f62fc0a1eb0 0x7f62fc0a22e0 secure :-1 s=THROTTLE_DONE pgs=356 cs=0 l=1 rx=0x7f62f4001690 tx=0x7f62ec001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:24.940 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.922+0000 7f62f9ffb700 1 -- 172.21.15.42:0/424925729 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62f40012a0 con 0x7f62fc0a1eb0 2019-12-05T05:09:24.940 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.922+0000 7f6302f88700 1 -- 172.21.15.42:0/424925729 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62fc07e090 con 0x7f62fc0a1eb0 2019-12-05T05:09:24.940 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.922+0000 7f62f9ffb700 1 -- 172.21.15.42:0/424925729 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f62ec00cc00 con 0x7f62fc0a1eb0 2019-12-05T05:09:24.940 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.922+0000 7f6302f88700 1 -- 172.21.15.42:0/424925729 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62fc138900 con 0x7f62fc0a1eb0 2019-12-05T05:09:24.940 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.926+0000 7f6301d26700 1 --2- 172.21.15.42:0/424925729 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f62fc0a1eb0 0x7f62fc0a22e0 secure :-1 s=THROTTLE_DONE pgs=356 cs=0 l=1 rx=0x7f62f4001690 tx=0x7f62ec001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:24.941 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.926+0000 7f62f9ffb700 1 -- 172.21.15.42:0/424925729 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f62ec004400 con 0x7f62fc0a1eb0 2019-12-05T05:09:24.941 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.926+0000 7f6301d26700 1 --2- 172.21.15.42:0/424925729 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f62fc0a1eb0 0x7f62fc0a22e0 secure :-1 s=THROTTLE_DONE pgs=356 cs=0 l=1 rx=0x7f62f4001690 tx=0x7f62ec001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:24.941 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.926+0000 7f62f9ffb700 1 --2- 172.21.15.42:0/424925729 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f62dc03b230 0x7f62dc03d740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:09:24.941 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.926+0000 7f62f9ffb700 1 -- 172.21.15.42:0/424925729 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6289+0+0 (secure 0 0 0) 0x7f62ec04a220 con 0x7f62fc0a1eb0 2019-12-05T05:09:24.941 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.926+0000 7f6301525700 1 --2- 172.21.15.42:0/424925729 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f62dc03b230 0x7f62dc03d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:09:24.941 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.926+0000 7f6301525700 1 --2- 172.21.15.42:0/424925729 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f62dc03b230 0x7f62dc03d740 crc :-1 s=READY pgs=160 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:09:24.941 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.930+0000 7f6302f88700 1 -- 172.21.15.42:0/424925729 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62fc05d110 con 0x7f62fc0a1eb0 2019-12-05T05:09:24.942 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.934+0000 7f6301d26700 1 --2- 172.21.15.42:0/424925729 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f62fc0a1eb0 0x7f62fc0a22e0 secure :-1 s=THROTTLE_DONE pgs=356 cs=0 l=1 rx=0x7f62f4001690 tx=0x7f62ec001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:24.942 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:24.934+0000 7f62f9ffb700 1 -- 172.21.15.42:0/424925729 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f62ec0312a0 con 0x7f62fc0a1eb0 2019-12-05T05:09:25.092 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:25.090+0000 7f6302f88700 1 -- 172.21.15.42:0/424925729 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f62fc05b8c0 con 0x7f62fc0a1eb0 2019-12-05T05:09:25.093 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:25.090+0000 7f6301d26700 1 --2- 172.21.15.42:0/424925729 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f62fc0a1eb0 0x7f62fc0a22e0 secure :-1 s=THROTTLE_DONE pgs=356 cs=0 l=1 rx=0x7f62f4001690 tx=0x7f62ec001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:25.093 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:25.090+0000 7f62f9ffb700 1 -- 172.21.15.42:0/424925729 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+1771 (secure 0 0 0) 0x7f62ec001230 con 0x7f62fc0a1eb0 2019-12-05T05:09:25.093 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 30 2019-12-05T05:09:25.094 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:09:25.094 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:09:23.258857+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4978},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4978":{"gid":4978,"name":"a","rank":0,"incarnation":30,"state":"up:replay","state_seq":2,"addr":"172.21.15.42:6835/3650564018","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3650564018},{"type":"v1","addr":"172.21.15.42:6835","nonce":3650564018}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:09:25.095 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:25.090+0000 7f6302f88700 1 -- 172.21.15.42:0/424925729 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f62dc03b230 msgr2=0x7f62dc03d740 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:09:25.095 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:25.090+0000 7f6302f88700 1 --2- 172.21.15.42:0/424925729 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f62dc03b230 0x7f62dc03d740 crc :-1 s=READY pgs=160 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:09:25.095 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:25.090+0000 7f6302f88700 1 -- 172.21.15.42:0/424925729 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f62fc0a1eb0 msgr2=0x7f62fc0a22e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:09:25.096 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:25.090+0000 7f6302f88700 1 --2- 172.21.15.42:0/424925729 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f62fc0a1eb0 0x7f62fc0a22e0 secure :-1 s=READY pgs=356 cs=0 l=1 rx=0x7f62f4001690 tx=0x7f62ec001150).stop 2019-12-05T05:09:25.096 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:25.090+0000 7f6302f88700 1 -- 172.21.15.42:0/424925729 shutdown_connections 2019-12-05T05:09:25.096 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:25.090+0000 7f6302f88700 1 --2- 172.21.15.42:0/424925729 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f62dc03b230 0x7f62dc03d740 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:09:25.096 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:25.090+0000 7f6302f88700 1 --2- 172.21.15.42:0/424925729 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f62fc0a1eb0 0x7f62fc0a22e0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:09:25.096 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:25.090+0000 7f6302f88700 1 -- 172.21.15.42:0/424925729 shutdown_connections 2019-12-05T05:09:25.096 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:25.090+0000 7f6302f88700 1 -- 172.21.15.42:0/424925729 wait complete. 2019-12-05T05:09:25.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:25.090+0000 7f6302f88700 1 -- 172.21.15.42:0/424925729 >> 172.21.15.42:0/424925729 conn(0x7f62fc11bac0 msgr2=0x7f62fc059bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:09:25.108 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 30, "feature_flags": { "enable_multiple": false, "ever_enabled_multiple": false }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2019-12-05T05:03:35.604738+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 30, "ever_allowed_features": 2, "explicitly_allowed_features": 2, "failed": [], "flags": 18, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_4978": { "addr": "172.21.15.42:6835/3650564018", "addrs": { "addrvec": [ { "addr": "172.21.15.42:6834", "nonce": 3650564018, "type": "v2" }, { "addr": "172.21.15.42:6835", "nonce": 3650564018, "type": "v1" } ] }, "export_targets": [], "features": 4540138292836696063, "flags": 0, "gid": 4978, "incarnation": 30, "name": "a", "rank": 0, "state": "up:replay", "state_seq": 2 } }, "last_failure": 0, "last_failure_osd_epoch": 38, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 2, "min_compat_client": "0 (unknown)", "modified": "2019-12-05T05:09:23.258857+0000", "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4978 } } } ], "standbys": [] } 2019-12-05T05:09:27.117 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:09:27.225 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:27.218+0000 7f4d4c67c700 1 Processor -- start 2019-12-05T05:09:27.225 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:27.218+0000 7f4d4c67c700 1 -- start start 2019-12-05T05:09:27.226 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:27.218+0000 7f4d4c67c700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f4d440a7740 0x7f4d4408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:09:27.226 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:27.218+0000 7f4d4c67c700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4d4407e720 0x7f4d4407eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:09:27.226 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:27.218+0000 7f4d4c67c700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d440a0710 con 0x7f4d4409f2e0 2019-12-05T05:09:27.226 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:27.218+0000 7f4d4c67c700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d440a0b00 con 0x7f4d440a7740 2019-12-05T05:09:27.226 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:27.218+0000 7f4d4c67c700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d4408e5b0 con 0x7f4d4407e720 2019-12-05T05:09:27.226 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:27.218+0000 7f4d4a418700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4d4407e720 0x7f4d4407eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:09:27.227 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:27.218+0000 7f4d4a418700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4d4407e720 0x7f4d4407eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:57156/0 (socket says 172.21.15.42:57156) 2019-12-05T05:09:27.227 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:27.218+0000 7f4d4ac19700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f4d440a7740 0x7f4d4408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:09:27.227 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:27.218+0000 7f4d4a418700 1 -- 172.21.15.42:0/675130948 learned_addr learned my addr 172.21.15.42:0/675130948 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:09:27.227 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:27.218+0000 7f4d4a418700 1 -- 172.21.15.42:0/675130948 >> v2:172.21.15.40:3300/0 conn(0x7f4d440a7740 msgr2=0x7f4d4408df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:09:27.227 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:27.218+0000 7f4d4a418700 1 --2- 172.21.15.42:0/675130948 >> v2:172.21.15.40:3300/0 conn(0x7f4d440a7740 0x7f4d4408df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:09:27.228 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:27.218+0000 7f4d4a418700 1 -- 172.21.15.42:0/675130948 >> v1:172.21.15.40:6789/0 conn(0x7f4d4409f2e0 legacy=0x7f4d4409f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:09:27.228 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:27.218+0000 7f4d4a418700 1 -- 172.21.15.42:0/675130948 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d4407f0a0 con 0x7f4d4407e720 2019-12-05T05:09:27.228 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:27.218+0000 7f4d4ac19700 1 --2- 172.21.15.42:0/675130948 >> v2:172.21.15.40:3300/0 conn(0x7f4d440a7740 0x7f4d4408df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:09:27.229 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:27.218+0000 7f4d4a418700 1 --2- 172.21.15.42:0/675130948 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4d4407e720 0x7f4d4407eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f4d34008790 tx=0x7f4d3400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:27.229 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:27.218+0000 7f4d4a418700 1 --2- 172.21.15.42:0/675130948 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4d4407e720 0x7f4d4407eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f4d34008790 tx=0x7f4d3400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:27.230 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:27.222+0000 7f4d4a418700 1 --2- 172.21.15.42:0/675130948 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4d4407e720 0x7f4d4407eb50 secure :-1 s=READY pgs=69 cs=0 l=1 rx=0x7f4d34008790 tx=0x7f4d3400fd30).ready entity=mon.2 client_cookie=168fa88a88c5f054 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:09:27.230 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:27.222+0000 7f4d4a418700 1 --2- 172.21.15.42:0/675130948 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4d4407e720 0x7f4d4407eb50 secure :-1 s=THROTTLE_DONE pgs=69 cs=0 l=1 rx=0x7f4d34008790 tx=0x7f4d3400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:27.230 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:27.222+0000 7f4d4a418700 1 --2- 172.21.15.42:0/675130948 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4d4407e720 0x7f4d4407eb50 secure :-1 s=THROTTLE_DONE pgs=69 cs=0 l=1 rx=0x7f4d34008790 tx=0x7f4d3400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:27.230 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:27.222+0000 7f4d4a418700 1 --2- 172.21.15.42:0/675130948 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4d4407e720 0x7f4d4407eb50 secure :-1 s=THROTTLE_DONE pgs=69 cs=0 l=1 rx=0x7f4d34008790 tx=0x7f4d3400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:27.231 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:27.222+0000 7f4d49c17700 1 -- 172.21.15.42:0/675130948 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4d340113b0 con 0x7f4d4407e720 2019-12-05T05:09:27.232 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:27.222+0000 7f4d49c17700 1 -- 172.21.15.42:0/675130948 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d34008bb0 con 0x7f4d4407e720 2019-12-05T05:09:27.232 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:27.222+0000 7f4d49c17700 1 -- 172.21.15.42:0/675130948 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4d34011990 con 0x7f4d4407e720 2019-12-05T05:09:27.232 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:27.222+0000 7f4d4c67c700 1 -- 172.21.15.42:0/675130948 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4d4407e720 msgr2=0x7f4d4407eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:09:27.232 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:27.222+0000 7f4d4c67c700 1 --2- 172.21.15.42:0/675130948 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4d4407e720 0x7f4d4407eb50 secure :-1 s=READY pgs=69 cs=0 l=1 rx=0x7f4d34008790 tx=0x7f4d3400fd30).stop 2019-12-05T05:09:27.233 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:27.222+0000 7f4d4c67c700 1 -- 172.21.15.42:0/675130948 shutdown_connections 2019-12-05T05:09:27.233 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:27.222+0000 7f4d4c67c700 1 --2- 172.21.15.42:0/675130948 >> v2:172.21.15.40:3300/0 conn(0x7f4d440a7740 0x7f4d4408df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:09:27.233 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:27.222+0000 7f4d4c67c700 1 --2- 172.21.15.42:0/675130948 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4d4407e720 0x7f4d4407eb50 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:09:27.233 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:27.222+0000 7f4d4c67c700 1 -- 172.21.15.42:0/675130948 shutdown_connections 2019-12-05T05:09:27.234 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:27.222+0000 7f4d4c67c700 1 -- 172.21.15.42:0/675130948 wait complete. 2019-12-05T05:09:27.234 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:27.222+0000 7f4d4c67c700 1 -- 172.21.15.42:0/675130948 >> 172.21.15.42:0/675130948 conn(0x7f4d440aabe0 msgr2=0x7f4d440a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:09:27.241 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:27.226+0000 7f4d4c67c700 1 Processor -- start 2019-12-05T05:09:27.242 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:27.226+0000 7f4d4c67c700 1 -- start start 2019-12-05T05:09:27.242 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:27.226+0000 7f4d4c67c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f4d4411bf30 0x7f4d4407afe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:09:27.242 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:27.226+0000 7f4d4c67c700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d440a68d0 con 0x7f4d440a2080 2019-12-05T05:09:27.243 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:27.226+0000 7f4d4c67c700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d4407b530 con 0x7f4d440a40f0 2019-12-05T05:09:27.243 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:27.226+0000 7f4d4c67c700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d440a25f0 con 0x7f4d4411bf30 2019-12-05T05:09:27.243 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:27.226+0000 7f4d4ac19700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f4d4411bf30 0x7f4d4407afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:09:27.243 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:27.226+0000 7f4d4ac19700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f4d4411bf30 0x7f4d4407afe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:56694/0 (socket says 172.21.15.42:56694) 2019-12-05T05:09:27.244 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:27.226+0000 7f4d4ac19700 1 -- 172.21.15.42:0/4147488866 learned_addr learned my addr 172.21.15.42:0/4147488866 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:09:27.244 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:27.226+0000 7f4d4ac19700 1 -- 172.21.15.42:0/4147488866 >> v1:172.21.15.42:6789/0 conn(0x7f4d440a40f0 legacy=0x7f4d440a4c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:09:27.244 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:27.226+0000 7f4d4ac19700 1 -- 172.21.15.42:0/4147488866 >> v1:172.21.15.40:6789/0 conn(0x7f4d440a2080 legacy=0x7f4d44138a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:09:27.246 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:27.226+0000 7f4d4ac19700 1 -- 172.21.15.42:0/4147488866 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d440a2b30 con 0x7f4d4411bf30 2019-12-05T05:09:27.246 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:27.226+0000 7f4d3b7fe700 1 -- 172.21.15.42:0/4147488866 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3630641838 0 0) 0x7f4d440a68d0 con 0x7f4d440a2080 2019-12-05T05:09:27.246 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:27.226+0000 7f4d4ac19700 1 --2- 172.21.15.42:0/4147488866 >> v2:172.21.15.42:3300/0 conn(0x7f4d4411bf30 0x7f4d4407afe0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f4d40002ef0 tx=0x7f4d40010cb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:27.247 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:27.226+0000 7f4d4ac19700 1 --2- 172.21.15.42:0/4147488866 >> v2:172.21.15.42:3300/0 conn(0x7f4d4411bf30 0x7f4d4407afe0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f4d40002ef0 tx=0x7f4d40010cb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:27.247 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:27.226+0000 7f4d4ac19700 1 --2- 172.21.15.42:0/4147488866 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4d4411bf30 0x7f4d4407afe0 secure :-1 s=READY pgs=358 cs=0 l=1 rx=0x7f4d40002ef0 tx=0x7f4d40010cb0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:09:27.247 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:27.226+0000 7f4d4ac19700 1 --2- 172.21.15.42:0/4147488866 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4d4411bf30 0x7f4d4407afe0 secure :-1 s=THROTTLE_DONE pgs=358 cs=0 l=1 rx=0x7f4d40002ef0 tx=0x7f4d40010cb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:27.252 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:27.226+0000 7f4d3b7fe700 1 -- 172.21.15.42:0/4147488866 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4d40012680 con 0x7f4d4411bf30 2019-12-05T05:09:27.253 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:27.226+0000 7f4d4c67c700 1 -- 172.21.15.42:0/4147488866 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d4407b760 con 0x7f4d4411bf30 2019-12-05T05:09:27.253 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:27.226+0000 7f4d4ac19700 1 --2- 172.21.15.42:0/4147488866 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4d4411bf30 0x7f4d4407afe0 secure :-1 s=THROTTLE_DONE pgs=358 cs=0 l=1 rx=0x7f4d40002ef0 tx=0x7f4d40010cb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:27.253 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:27.226+0000 7f4d4c67c700 1 -- 172.21.15.42:0/4147488866 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d440a5480 con 0x7f4d4411bf30 2019-12-05T05:09:27.253 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:27.226+0000 7f4d3b7fe700 1 -- 172.21.15.42:0/4147488866 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d40007150 con 0x7f4d4411bf30 2019-12-05T05:09:27.254 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:27.226+0000 7f4d4ac19700 1 --2- 172.21.15.42:0/4147488866 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4d4411bf30 0x7f4d4407afe0 secure :-1 s=THROTTLE_DONE pgs=358 cs=0 l=1 rx=0x7f4d40002ef0 tx=0x7f4d40010cb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:27.254 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:27.226+0000 7f4d3b7fe700 1 -- 172.21.15.42:0/4147488866 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4d40012ca0 con 0x7f4d4411bf30 2019-12-05T05:09:27.254 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:27.230+0000 7f4d4ac19700 1 --2- 172.21.15.42:0/4147488866 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4d4411bf30 0x7f4d4407afe0 secure :-1 s=THROTTLE_DONE pgs=358 cs=0 l=1 rx=0x7f4d40002ef0 tx=0x7f4d40010cb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:27.254 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:27.230+0000 7f4d3b7fe700 1 -- 172.21.15.42:0/4147488866 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f4d40005540 con 0x7f4d4411bf30 2019-12-05T05:09:27.255 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:27.230+0000 7f4d4ac19700 1 --2- 172.21.15.42:0/4147488866 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4d4411bf30 0x7f4d4407afe0 secure :-1 s=THROTTLE_DONE pgs=358 cs=0 l=1 rx=0x7f4d40002ef0 tx=0x7f4d40010cb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:27.255 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:27.230+0000 7f4d3b7fe700 1 --2- 172.21.15.42:0/4147488866 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4d2c03ac80 0x7f4d2c03d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:09:27.255 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:27.230+0000 7f4d3b7fe700 1 -- 172.21.15.42:0/4147488866 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6289+0+0 (secure 0 0 0) 0x7f4d4004b840 con 0x7f4d4411bf30 2019-12-05T05:09:27.256 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:27.230+0000 7f4d4a418700 1 --2- 172.21.15.42:0/4147488866 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4d2c03ac80 0x7f4d2c03d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:09:27.256 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:27.230+0000 7f4d4a418700 1 --2- 172.21.15.42:0/4147488866 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4d2c03ac80 0x7f4d2c03d190 crc :-1 s=READY pgs=161 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:09:27.256 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:27.234+0000 7f4d4c67c700 1 -- 172.21.15.42:0/4147488866 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d4405d110 con 0x7f4d4411bf30 2019-12-05T05:09:27.257 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:27.242+0000 7f4d4ac19700 1 --2- 172.21.15.42:0/4147488866 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4d4411bf30 0x7f4d4407afe0 secure :-1 s=THROTTLE_DONE pgs=358 cs=0 l=1 rx=0x7f4d40002ef0 tx=0x7f4d40010cb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:27.257 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:27.242+0000 7f4d3b7fe700 1 -- 172.21.15.42:0/4147488866 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f4d400328a0 con 0x7f4d4411bf30 2019-12-05T05:09:27.451 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:27.450+0000 7f4d4c67c700 1 -- 172.21.15.42:0/4147488866 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4d4405b8c0 con 0x7f4d4411bf30 2019-12-05T05:09:27.452 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:27.450+0000 7f4d4ac19700 1 --2- 172.21.15.42:0/4147488866 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4d4411bf30 0x7f4d4407afe0 secure :-1 s=THROTTLE_DONE pgs=358 cs=0 l=1 rx=0x7f4d40002ef0 tx=0x7f4d40010cb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:27.452 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:27.450+0000 7f4d3b7fe700 1 -- 172.21.15.42:0/4147488866 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+1778 (secure 0 0 0) 0x7f4d40032b00 con 0x7f4d4411bf30 2019-12-05T05:09:27.452 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 33 2019-12-05T05:09:27.452 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:09:27.453 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:09:27.287703+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4978},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4978":{"gid":4978,"name":"a","rank":0,"incarnation":30,"state":"up:clientreplay","state_seq":35,"addr":"172.21.15.42:6835/3650564018","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3650564018},{"type":"v1","addr":"172.21.15.42:6835","nonce":3650564018}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:09:27.455 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:27.450+0000 7f4d4c67c700 1 -- 172.21.15.42:0/4147488866 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4d2c03ac80 msgr2=0x7f4d2c03d190 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:09:27.455 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:27.450+0000 7f4d4c67c700 1 --2- 172.21.15.42:0/4147488866 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4d2c03ac80 0x7f4d2c03d190 crc :-1 s=READY pgs=161 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:09:27.455 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:27.450+0000 7f4d4c67c700 1 -- 172.21.15.42:0/4147488866 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4d4411bf30 msgr2=0x7f4d4407afe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:09:27.455 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:27.450+0000 7f4d4c67c700 1 --2- 172.21.15.42:0/4147488866 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4d4411bf30 0x7f4d4407afe0 secure :-1 s=READY pgs=358 cs=0 l=1 rx=0x7f4d40002ef0 tx=0x7f4d40010cb0).stop 2019-12-05T05:09:27.455 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:27.450+0000 7f4d4c67c700 1 -- 172.21.15.42:0/4147488866 shutdown_connections 2019-12-05T05:09:27.456 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:27.450+0000 7f4d4c67c700 1 --2- 172.21.15.42:0/4147488866 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4d2c03ac80 0x7f4d2c03d190 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:09:27.456 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:27.450+0000 7f4d4c67c700 1 --2- 172.21.15.42:0/4147488866 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4d4411bf30 0x7f4d4407afe0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:09:27.456 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:27.450+0000 7f4d4c67c700 1 -- 172.21.15.42:0/4147488866 shutdown_connections 2019-12-05T05:09:27.456 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:27.450+0000 7f4d4c67c700 1 -- 172.21.15.42:0/4147488866 wait complete. 2019-12-05T05:09:27.456 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:27.450+0000 7f4d4c67c700 1 -- 172.21.15.42:0/4147488866 >> 172.21.15.42:0/4147488866 conn(0x7f4d440a6d70 msgr2=0x7f4d44059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:09:27.464 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2019-12-05T05:09:28.451 INFO:tasks.workunit.client.0.smithi042.stdout:removed './236/22' 2019-12-05T05:09:28.451 INFO:tasks.workunit.client.0.smithi042.stdout:removed './236/25' 2019-12-05T05:09:28.451 INFO:tasks.workunit.client.0.smithi042.stdout:removed './236/30' 2019-12-05T05:09:28.452 INFO:tasks.workunit.client.0.smithi042.stdout:removed './236/33' 2019-12-05T05:09:28.452 INFO:tasks.workunit.client.0.smithi042.stdout:removed './236/23' 2019-12-05T05:09:28.452 INFO:tasks.workunit.client.0.smithi042.stdout:removed './236/35' 2019-12-05T05:09:28.452 INFO:tasks.workunit.client.0.smithi042.stdout:removed './236/26' 2019-12-05T05:09:28.452 INFO:tasks.workunit.client.0.smithi042.stdout:removed './236/37' 2019-12-05T05:09:28.452 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './236' 2019-12-05T05:09:28.452 INFO:tasks.workunit.client.0.smithi042.stdout:removed './237/21' 2019-12-05T05:09:28.452 INFO:tasks.workunit.client.0.smithi042.stdout:removed './237/27' 2019-12-05T05:09:28.453 INFO:tasks.workunit.client.0.smithi042.stdout:removed './237/31' 2019-12-05T05:09:28.453 INFO:tasks.workunit.client.0.smithi042.stdout:removed './237/36' 2019-12-05T05:09:28.453 INFO:tasks.workunit.client.0.smithi042.stdout:removed './237/20' 2019-12-05T05:09:28.453 INFO:tasks.workunit.client.0.smithi042.stdout:removed './237/39' 2019-12-05T05:09:28.453 INFO:tasks.workunit.client.0.smithi042.stdout:removed './237/34' 2019-12-05T05:09:28.453 INFO:tasks.workunit.client.0.smithi042.stdout:removed './237/38' 2019-12-05T05:09:28.453 INFO:tasks.workunit.client.0.smithi042.stdout:removed './237/28' 2019-12-05T05:09:28.454 INFO:tasks.workunit.client.0.smithi042.stdout:removed './237/24' 2019-12-05T05:09:28.454 INFO:tasks.workunit.client.0.smithi042.stdout:removed './237/29' 2019-12-05T05:09:28.454 INFO:tasks.workunit.client.0.smithi042.stdout:removed './237/32' 2019-12-05T05:09:28.454 INFO:tasks.workunit.client.0.smithi042.stdout:removed './237/22' 2019-12-05T05:09:28.454 INFO:tasks.workunit.client.0.smithi042.stdout:removed './237/25' 2019-12-05T05:09:28.454 INFO:tasks.workunit.client.0.smithi042.stdout:removed './237/30' 2019-12-05T05:09:28.454 INFO:tasks.workunit.client.0.smithi042.stdout:removed './237/33' 2019-12-05T05:09:28.454 INFO:tasks.workunit.client.0.smithi042.stdout:removed './237/23' 2019-12-05T05:09:28.455 INFO:tasks.workunit.client.0.smithi042.stdout:removed './237/35' 2019-12-05T05:09:28.455 INFO:tasks.workunit.client.0.smithi042.stdout:removed './237/26' 2019-12-05T05:09:28.455 INFO:tasks.workunit.client.0.smithi042.stdout:removed './237/37' 2019-12-05T05:09:28.455 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './237' 2019-12-05T05:09:28.455 INFO:tasks.workunit.client.0.smithi042.stdout:removed './238/21' 2019-12-05T05:09:28.456 INFO:tasks.workunit.client.0.smithi042.stdout:removed './238/27' 2019-12-05T05:09:28.456 INFO:tasks.workunit.client.0.smithi042.stdout:removed './238/31' 2019-12-05T05:09:28.456 INFO:tasks.workunit.client.0.smithi042.stdout:removed './238/36' 2019-12-05T05:09:28.456 INFO:tasks.workunit.client.0.smithi042.stdout:removed './238/20' 2019-12-05T05:09:28.456 INFO:tasks.workunit.client.0.smithi042.stdout:removed './238/39' 2019-12-05T05:09:28.456 INFO:tasks.workunit.client.0.smithi042.stdout:removed './238/34' 2019-12-05T05:09:28.456 INFO:tasks.workunit.client.0.smithi042.stdout:removed './238/38' 2019-12-05T05:09:28.457 INFO:tasks.workunit.client.0.smithi042.stdout:removed './238/28' 2019-12-05T05:09:28.457 INFO:tasks.workunit.client.0.smithi042.stdout:removed './238/24' 2019-12-05T05:09:28.457 INFO:tasks.workunit.client.0.smithi042.stdout:removed './238/29' 2019-12-05T05:09:28.457 INFO:tasks.workunit.client.0.smithi042.stdout:removed './238/32' 2019-12-05T05:09:28.457 INFO:tasks.workunit.client.0.smithi042.stdout:removed './238/22' 2019-12-05T05:09:28.457 INFO:tasks.workunit.client.0.smithi042.stdout:removed './238/25' 2019-12-05T05:09:28.457 INFO:tasks.workunit.client.0.smithi042.stdout:removed './238/30' 2019-12-05T05:09:28.458 INFO:tasks.workunit.client.0.smithi042.stdout:removed './238/33' 2019-12-05T05:09:28.458 INFO:tasks.workunit.client.0.smithi042.stdout:removed './238/23' 2019-12-05T05:09:28.458 INFO:tasks.workunit.client.0.smithi042.stdout:removed './238/35' 2019-12-05T05:09:28.458 INFO:tasks.workunit.client.0.smithi042.stdout:removed './238/26' 2019-12-05T05:09:28.458 INFO:tasks.workunit.client.0.smithi042.stdout:removed './238/37' 2019-12-05T05:09:28.458 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './238' 2019-12-05T05:09:28.458 INFO:tasks.workunit.client.0.smithi042.stdout:removed './239/21' 2019-12-05T05:09:28.459 INFO:tasks.workunit.client.0.smithi042.stdout:removed './239/27' 2019-12-05T05:09:28.459 INFO:tasks.workunit.client.0.smithi042.stdout:removed './239/31' 2019-12-05T05:09:28.459 INFO:tasks.workunit.client.0.smithi042.stdout:removed './239/36' 2019-12-05T05:09:28.459 INFO:tasks.workunit.client.0.smithi042.stdout:removed './239/20' 2019-12-05T05:09:28.460 INFO:tasks.workunit.client.0.smithi042.stdout:removed './239/39' 2019-12-05T05:09:28.461 INFO:tasks.workunit.client.0.smithi042.stdout:removed './239/34' 2019-12-05T05:09:28.461 INFO:tasks.workunit.client.0.smithi042.stdout:removed './239/38' 2019-12-05T05:09:28.461 INFO:tasks.workunit.client.0.smithi042.stdout:removed './239/28' 2019-12-05T05:09:28.461 INFO:tasks.workunit.client.0.smithi042.stdout:removed './239/24' 2019-12-05T05:09:28.461 INFO:tasks.workunit.client.0.smithi042.stdout:removed './239/29' 2019-12-05T05:09:28.461 INFO:tasks.workunit.client.0.smithi042.stdout:removed './239/32' 2019-12-05T05:09:28.461 INFO:tasks.workunit.client.0.smithi042.stdout:removed './239/22' 2019-12-05T05:09:28.462 INFO:tasks.workunit.client.0.smithi042.stdout:removed './239/25' 2019-12-05T05:09:28.462 INFO:tasks.workunit.client.0.smithi042.stdout:removed './239/30' 2019-12-05T05:09:28.462 INFO:tasks.workunit.client.0.smithi042.stdout:removed './239/33' 2019-12-05T05:09:28.462 INFO:tasks.workunit.client.0.smithi042.stdout:removed './239/23' 2019-12-05T05:09:28.462 INFO:tasks.workunit.client.0.smithi042.stdout:removed './239/35' 2019-12-05T05:09:28.462 INFO:tasks.workunit.client.0.smithi042.stdout:removed './239/26' 2019-12-05T05:09:28.462 INFO:tasks.workunit.client.0.smithi042.stdout:removed './239/37' 2019-12-05T05:09:28.463 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './239' 2019-12-05T05:09:28.463 INFO:tasks.workunit.client.0.smithi042.stdout:removed './240/21' 2019-12-05T05:09:28.463 INFO:tasks.workunit.client.0.smithi042.stdout:removed './240/27' 2019-12-05T05:09:28.463 INFO:tasks.workunit.client.0.smithi042.stdout:removed './240/31' 2019-12-05T05:09:28.463 INFO:tasks.workunit.client.0.smithi042.stdout:removed './240/36' 2019-12-05T05:09:28.463 INFO:tasks.workunit.client.0.smithi042.stdout:removed './240/20' 2019-12-05T05:09:28.463 INFO:tasks.workunit.client.0.smithi042.stdout:removed './240/39' 2019-12-05T05:09:28.463 INFO:tasks.workunit.client.0.smithi042.stdout:removed './240/34' 2019-12-05T05:09:28.464 INFO:tasks.workunit.client.0.smithi042.stdout:removed './240/38' 2019-12-05T05:09:28.464 INFO:tasks.workunit.client.0.smithi042.stdout:removed './240/28' 2019-12-05T05:09:28.464 INFO:tasks.workunit.client.0.smithi042.stdout:removed './240/24' 2019-12-05T05:09:28.464 INFO:tasks.workunit.client.0.smithi042.stdout:removed './240/29' 2019-12-05T05:09:28.464 INFO:tasks.workunit.client.0.smithi042.stdout:removed './240/32' 2019-12-05T05:09:28.464 INFO:tasks.workunit.client.0.smithi042.stdout:removed './240/22' 2019-12-05T05:09:28.464 INFO:tasks.workunit.client.0.smithi042.stdout:removed './240/25' 2019-12-05T05:09:28.465 INFO:tasks.workunit.client.0.smithi042.stdout:removed './240/30' 2019-12-05T05:09:28.465 INFO:tasks.workunit.client.0.smithi042.stdout:removed './240/33' 2019-12-05T05:09:28.465 INFO:tasks.workunit.client.0.smithi042.stdout:removed './240/23' 2019-12-05T05:09:28.465 INFO:tasks.workunit.client.0.smithi042.stdout:removed './240/35' 2019-12-05T05:09:28.465 INFO:tasks.workunit.client.0.smithi042.stdout:removed './240/26' 2019-12-05T05:09:28.465 INFO:tasks.workunit.client.0.smithi042.stdout:removed './240/37' 2019-12-05T05:09:28.465 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './240' 2019-12-05T05:09:28.465 INFO:tasks.workunit.client.0.smithi042.stdout:removed './241/21' 2019-12-05T05:09:28.466 INFO:tasks.workunit.client.0.smithi042.stdout:removed './241/27' 2019-12-05T05:09:28.466 INFO:tasks.workunit.client.0.smithi042.stdout:removed './241/31' 2019-12-05T05:09:28.466 INFO:tasks.workunit.client.0.smithi042.stdout:removed './241/36' 2019-12-05T05:09:28.466 INFO:tasks.workunit.client.0.smithi042.stdout:removed './241/20' 2019-12-05T05:09:28.466 INFO:tasks.workunit.client.0.smithi042.stdout:removed './241/39' 2019-12-05T05:09:28.466 INFO:tasks.workunit.client.0.smithi042.stdout:removed './241/34' 2019-12-05T05:09:28.466 INFO:tasks.workunit.client.0.smithi042.stdout:removed './241/38' 2019-12-05T05:09:28.467 INFO:tasks.workunit.client.0.smithi042.stdout:removed './241/28' 2019-12-05T05:09:28.467 INFO:tasks.workunit.client.0.smithi042.stdout:removed './241/24' 2019-12-05T05:09:28.467 INFO:tasks.workunit.client.0.smithi042.stdout:removed './241/29' 2019-12-05T05:09:28.467 INFO:tasks.workunit.client.0.smithi042.stdout:removed './241/32' 2019-12-05T05:09:28.467 INFO:tasks.workunit.client.0.smithi042.stdout:removed './241/22' 2019-12-05T05:09:28.467 INFO:tasks.workunit.client.0.smithi042.stdout:removed './241/25' 2019-12-05T05:09:28.467 INFO:tasks.workunit.client.0.smithi042.stdout:removed './241/30' 2019-12-05T05:09:28.468 INFO:tasks.workunit.client.0.smithi042.stdout:removed './241/33' 2019-12-05T05:09:28.468 INFO:tasks.workunit.client.0.smithi042.stdout:removed './241/23' 2019-12-05T05:09:28.468 INFO:tasks.workunit.client.0.smithi042.stdout:removed './241/35' 2019-12-05T05:09:28.468 INFO:tasks.workunit.client.0.smithi042.stdout:removed './241/26' 2019-12-05T05:09:28.468 INFO:tasks.workunit.client.0.smithi042.stdout:removed './241/37' 2019-12-05T05:09:28.468 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './241' 2019-12-05T05:09:28.468 INFO:tasks.workunit.client.0.smithi042.stdout:removed './242/21' 2019-12-05T05:09:28.469 INFO:tasks.workunit.client.0.smithi042.stdout:removed './242/27' 2019-12-05T05:09:28.469 INFO:tasks.workunit.client.0.smithi042.stdout:removed './242/31' 2019-12-05T05:09:28.469 INFO:tasks.workunit.client.0.smithi042.stdout:removed './242/36' 2019-12-05T05:09:28.469 INFO:tasks.workunit.client.0.smithi042.stdout:removed './242/20' 2019-12-05T05:09:28.469 INFO:tasks.workunit.client.0.smithi042.stdout:removed './242/39' 2019-12-05T05:09:28.470 INFO:tasks.workunit.client.0.smithi042.stdout:removed './242/34' 2019-12-05T05:09:28.470 INFO:tasks.workunit.client.0.smithi042.stdout:removed './242/38' 2019-12-05T05:09:28.470 INFO:tasks.workunit.client.0.smithi042.stdout:removed './242/28' 2019-12-05T05:09:28.470 INFO:tasks.workunit.client.0.smithi042.stdout:removed './242/24' 2019-12-05T05:09:28.471 INFO:tasks.workunit.client.0.smithi042.stdout:removed './242/29' 2019-12-05T05:09:28.471 INFO:tasks.workunit.client.0.smithi042.stdout:removed './242/32' 2019-12-05T05:09:28.471 INFO:tasks.workunit.client.0.smithi042.stdout:removed './242/22' 2019-12-05T05:09:28.471 INFO:tasks.workunit.client.0.smithi042.stdout:removed './242/25' 2019-12-05T05:09:28.472 INFO:tasks.workunit.client.0.smithi042.stdout:removed './242/30' 2019-12-05T05:09:28.472 INFO:tasks.workunit.client.0.smithi042.stdout:removed './242/33' 2019-12-05T05:09:28.472 INFO:tasks.workunit.client.0.smithi042.stdout:removed './242/23' 2019-12-05T05:09:28.472 INFO:tasks.workunit.client.0.smithi042.stdout:removed './242/35' 2019-12-05T05:09:28.472 INFO:tasks.workunit.client.0.smithi042.stdout:removed './242/26' 2019-12-05T05:09:28.472 INFO:tasks.workunit.client.0.smithi042.stdout:removed './242/37' 2019-12-05T05:09:28.472 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './242' 2019-12-05T05:09:28.473 INFO:tasks.workunit.client.0.smithi042.stdout:removed './243/21' 2019-12-05T05:09:28.473 INFO:tasks.workunit.client.0.smithi042.stdout:removed './243/27' 2019-12-05T05:09:28.473 INFO:tasks.workunit.client.0.smithi042.stdout:removed './243/31' 2019-12-05T05:09:28.473 INFO:tasks.workunit.client.0.smithi042.stdout:removed './243/36' 2019-12-05T05:09:28.473 INFO:tasks.workunit.client.0.smithi042.stdout:removed './243/20' 2019-12-05T05:09:28.473 INFO:tasks.workunit.client.0.smithi042.stdout:removed './243/39' 2019-12-05T05:09:28.473 INFO:tasks.workunit.client.0.smithi042.stdout:removed './243/34' 2019-12-05T05:09:28.474 INFO:tasks.workunit.client.0.smithi042.stdout:removed './243/38' 2019-12-05T05:09:28.474 INFO:tasks.workunit.client.0.smithi042.stdout:removed './243/28' 2019-12-05T05:09:28.474 INFO:tasks.workunit.client.0.smithi042.stdout:removed './243/24' 2019-12-05T05:09:28.474 INFO:tasks.workunit.client.0.smithi042.stdout:removed './243/29' 2019-12-05T05:09:28.474 INFO:tasks.workunit.client.0.smithi042.stdout:removed './243/32' 2019-12-05T05:09:28.475 INFO:tasks.workunit.client.0.smithi042.stdout:removed './243/22' 2019-12-05T05:09:28.475 INFO:tasks.workunit.client.0.smithi042.stdout:removed './243/25' 2019-12-05T05:09:28.475 INFO:tasks.workunit.client.0.smithi042.stdout:removed './243/30' 2019-12-05T05:09:28.475 INFO:tasks.workunit.client.0.smithi042.stdout:removed './243/33' 2019-12-05T05:09:28.475 INFO:tasks.workunit.client.0.smithi042.stdout:removed './243/23' 2019-12-05T05:09:28.475 INFO:tasks.workunit.client.0.smithi042.stdout:removed './243/35' 2019-12-05T05:09:28.475 INFO:tasks.workunit.client.0.smithi042.stdout:removed './243/26' 2019-12-05T05:09:28.476 INFO:tasks.workunit.client.0.smithi042.stdout:removed './243/37' 2019-12-05T05:09:28.476 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './243' 2019-12-05T05:09:28.476 INFO:tasks.workunit.client.0.smithi042.stdout:removed './244/21' 2019-12-05T05:09:28.476 INFO:tasks.workunit.client.0.smithi042.stdout:removed './244/27' 2019-12-05T05:09:28.476 INFO:tasks.workunit.client.0.smithi042.stdout:removed './244/31' 2019-12-05T05:09:28.477 INFO:tasks.workunit.client.0.smithi042.stdout:removed './244/36' 2019-12-05T05:09:28.477 INFO:tasks.workunit.client.0.smithi042.stdout:removed './244/20' 2019-12-05T05:09:28.477 INFO:tasks.workunit.client.0.smithi042.stdout:removed './244/39' 2019-12-05T05:09:28.477 INFO:tasks.workunit.client.0.smithi042.stdout:removed './244/34' 2019-12-05T05:09:28.477 INFO:tasks.workunit.client.0.smithi042.stdout:removed './244/38' 2019-12-05T05:09:28.477 INFO:tasks.workunit.client.0.smithi042.stdout:removed './244/28' 2019-12-05T05:09:28.478 INFO:tasks.workunit.client.0.smithi042.stdout:removed './244/24' 2019-12-05T05:09:28.478 INFO:tasks.workunit.client.0.smithi042.stdout:removed './244/29' 2019-12-05T05:09:28.478 INFO:tasks.workunit.client.0.smithi042.stdout:removed './244/32' 2019-12-05T05:09:28.478 INFO:tasks.workunit.client.0.smithi042.stdout:removed './244/22' 2019-12-05T05:09:28.478 INFO:tasks.workunit.client.0.smithi042.stdout:removed './244/25' 2019-12-05T05:09:28.478 INFO:tasks.workunit.client.0.smithi042.stdout:removed './244/30' 2019-12-05T05:09:28.479 INFO:tasks.workunit.client.0.smithi042.stdout:removed './244/33' 2019-12-05T05:09:28.479 INFO:tasks.workunit.client.0.smithi042.stdout:removed './244/23' 2019-12-05T05:09:28.479 INFO:tasks.workunit.client.0.smithi042.stdout:removed './244/35' 2019-12-05T05:09:28.479 INFO:tasks.workunit.client.0.smithi042.stdout:removed './244/26' 2019-12-05T05:09:28.479 INFO:tasks.workunit.client.0.smithi042.stdout:removed './244/37' 2019-12-05T05:09:28.479 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './244' 2019-12-05T05:09:28.480 INFO:tasks.workunit.client.0.smithi042.stdout:removed './245/21' 2019-12-05T05:09:28.480 INFO:tasks.workunit.client.0.smithi042.stdout:removed './245/27' 2019-12-05T05:09:28.480 INFO:tasks.workunit.client.0.smithi042.stdout:removed './245/31' 2019-12-05T05:09:28.480 INFO:tasks.workunit.client.0.smithi042.stdout:removed './245/36' 2019-12-05T05:09:28.480 INFO:tasks.workunit.client.0.smithi042.stdout:removed './245/20' 2019-12-05T05:09:28.481 INFO:tasks.workunit.client.0.smithi042.stdout:removed './245/39' 2019-12-05T05:09:28.481 INFO:tasks.workunit.client.0.smithi042.stdout:removed './245/34' 2019-12-05T05:09:28.482 INFO:tasks.workunit.client.0.smithi042.stdout:removed './245/38' 2019-12-05T05:09:28.482 INFO:tasks.workunit.client.0.smithi042.stdout:removed './245/28' 2019-12-05T05:09:28.482 INFO:tasks.workunit.client.0.smithi042.stdout:removed './245/24' 2019-12-05T05:09:28.482 INFO:tasks.workunit.client.0.smithi042.stdout:removed './245/29' 2019-12-05T05:09:28.482 INFO:tasks.workunit.client.0.smithi042.stdout:removed './245/32' 2019-12-05T05:09:28.482 INFO:tasks.workunit.client.0.smithi042.stdout:removed './245/22' 2019-12-05T05:09:28.483 INFO:tasks.workunit.client.0.smithi042.stdout:removed './245/25' 2019-12-05T05:09:28.483 INFO:tasks.workunit.client.0.smithi042.stdout:removed './245/30' 2019-12-05T05:09:28.483 INFO:tasks.workunit.client.0.smithi042.stdout:removed './245/33' 2019-12-05T05:09:28.483 INFO:tasks.workunit.client.0.smithi042.stdout:removed './245/23' 2019-12-05T05:09:28.483 INFO:tasks.workunit.client.0.smithi042.stdout:removed './245/35' 2019-12-05T05:09:28.484 INFO:tasks.workunit.client.0.smithi042.stdout:removed './245/26' 2019-12-05T05:09:28.484 INFO:tasks.workunit.client.0.smithi042.stdout:removed './245/37' 2019-12-05T05:09:28.484 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './245' 2019-12-05T05:09:28.484 INFO:tasks.workunit.client.0.smithi042.stdout:removed './246/21' 2019-12-05T05:09:28.484 INFO:tasks.workunit.client.0.smithi042.stdout:removed './246/27' 2019-12-05T05:09:28.484 INFO:tasks.workunit.client.0.smithi042.stdout:removed './246/31' 2019-12-05T05:09:28.485 INFO:tasks.workunit.client.0.smithi042.stdout:removed './246/36' 2019-12-05T05:09:28.485 INFO:tasks.workunit.client.0.smithi042.stdout:removed './246/20' 2019-12-05T05:09:28.485 INFO:tasks.workunit.client.0.smithi042.stdout:removed './246/39' 2019-12-05T05:09:28.485 INFO:tasks.workunit.client.0.smithi042.stdout:removed './246/34' 2019-12-05T05:09:28.485 INFO:tasks.workunit.client.0.smithi042.stdout:removed './246/38' 2019-12-05T05:09:28.485 INFO:tasks.workunit.client.0.smithi042.stdout:removed './246/28' 2019-12-05T05:09:28.485 INFO:tasks.workunit.client.0.smithi042.stdout:removed './246/24' 2019-12-05T05:09:28.486 INFO:tasks.workunit.client.0.smithi042.stdout:removed './246/29' 2019-12-05T05:09:28.486 INFO:tasks.workunit.client.0.smithi042.stdout:removed './246/32' 2019-12-05T05:09:28.486 INFO:tasks.workunit.client.0.smithi042.stdout:removed './246/22' 2019-12-05T05:09:29.465 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:09:29.564 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:29.554+0000 7fe8df74e700 1 Processor -- start 2019-12-05T05:09:29.564 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:29.554+0000 7fe8df74e700 1 -- start start 2019-12-05T05:09:29.565 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:29.554+0000 7fe8df74e700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe8d80a7740 0x7fe8d808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:09:29.565 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:29.554+0000 7fe8df74e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe8d807e720 0x7fe8d807eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:09:29.565 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:29.554+0000 7fe8df74e700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe8d809f2e0 0x7fe8d809f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:09:29.565 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:29.554+0000 7fe8df74e700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8d80a0b00 con 0x7fe8d80a7740 2019-12-05T05:09:29.566 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:29.554+0000 7fe8df74e700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8d808e5b0 con 0x7fe8d807e720 2019-12-05T05:09:29.566 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:29.554+0000 7fe8df74e700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8d808e850 con 0x7fe8d809f2e0 2019-12-05T05:09:29.566 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:29.558+0000 7fe8dd4ea700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe8d807e720 0x7fe8d807eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:09:29.566 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:29.558+0000 7fe8dd4ea700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe8d807e720 0x7fe8d807eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:56740/0 (socket says 172.21.15.42:56740) 2019-12-05T05:09:29.567 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:29.558+0000 7fe8dd4ea700 1 -- 172.21.15.42:0/1206579790 learned_addr learned my addr 172.21.15.42:0/1206579790 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:09:29.567 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:29.558+0000 7fe8ddceb700 1 --2- 172.21.15.42:0/1206579790 >> v2:172.21.15.40:3300/0 conn(0x7fe8d80a7740 0x7fe8d808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:09:29.567 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:29.558+0000 7fe8de4ec700 1 --2- 172.21.15.42:0/1206579790 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe8d809f2e0 0x7fe8d809f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:09:29.567 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:29.558+0000 7fe8dd4ea700 1 -- 172.21.15.42:0/1206579790 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe8d809f2e0 msgr2=0x7fe8d809f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:09:29.567 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:29.558+0000 7fe8dd4ea700 1 --2- 172.21.15.42:0/1206579790 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe8d809f2e0 0x7fe8d809f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:09:29.568 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:29.558+0000 7fe8dd4ea700 1 -- 172.21.15.42:0/1206579790 >> v2:172.21.15.40:3300/0 conn(0x7fe8d80a7740 msgr2=0x7fe8d808df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:09:29.568 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:29.558+0000 7fe8dd4ea700 1 --2- 172.21.15.42:0/1206579790 >> v2:172.21.15.40:3300/0 conn(0x7fe8d80a7740 0x7fe8d808df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:09:29.568 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:29.558+0000 7fe8dd4ea700 1 -- 172.21.15.42:0/1206579790 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8d807f0a0 con 0x7fe8d807e720 2019-12-05T05:09:29.568 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:29.558+0000 7fe8dd4ea700 1 --2- 172.21.15.42:0/1206579790 >> v2:172.21.15.42:3300/0 conn(0x7fe8d807e720 0x7fe8d807eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fe8d80a0f50 tx=0x7fe8c800fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:29.569 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:29.558+0000 7fe8dd4ea700 1 --2- 172.21.15.42:0/1206579790 >> v2:172.21.15.42:3300/0 conn(0x7fe8d807e720 0x7fe8d807eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fe8d80a0f50 tx=0x7fe8c800fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:29.569 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:29.558+0000 7fe8dd4ea700 1 --2- 172.21.15.42:0/1206579790 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe8d807e720 0x7fe8d807eb50 secure :-1 s=READY pgs=359 cs=0 l=1 rx=0x7fe8d80a0f50 tx=0x7fe8c800fdb0).ready entity=mon.1 client_cookie=d3490edff5be55f7 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:09:29.569 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:29.558+0000 7fe8dd4ea700 1 --2- 172.21.15.42:0/1206579790 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe8d807e720 0x7fe8d807eb50 secure :-1 s=THROTTLE_DONE pgs=359 cs=0 l=1 rx=0x7fe8d80a0f50 tx=0x7fe8c800fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:29.569 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:29.558+0000 7fe8dd4ea700 1 --2- 172.21.15.42:0/1206579790 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe8d807e720 0x7fe8d807eb50 secure :-1 s=THROTTLE_DONE pgs=359 cs=0 l=1 rx=0x7fe8d80a0f50 tx=0x7fe8c800fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:29.570 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:29.558+0000 7fe8dcce9700 1 -- 172.21.15.42:0/1206579790 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe8c8008c90 con 0x7fe8d807e720 2019-12-05T05:09:29.584 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:29.558+0000 7fe8dd4ea700 1 --2- 172.21.15.42:0/1206579790 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe8d807e720 0x7fe8d807eb50 secure :-1 s=THROTTLE_DONE pgs=359 cs=0 l=1 rx=0x7fe8d80a0f50 tx=0x7fe8c800fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:29.584 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:29.558+0000 7fe8dcce9700 1 -- 172.21.15.42:0/1206579790 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8c80088b0 con 0x7fe8d807e720 2019-12-05T05:09:29.585 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:29.558+0000 7fe8dcce9700 1 -- 172.21.15.42:0/1206579790 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe8c8011b50 con 0x7fe8d807e720 2019-12-05T05:09:29.585 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:29.558+0000 7fe8df74e700 1 -- 172.21.15.42:0/1206579790 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe8d807e720 msgr2=0x7fe8d807eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:09:29.585 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:29.558+0000 7fe8df74e700 1 --2- 172.21.15.42:0/1206579790 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe8d807e720 0x7fe8d807eb50 secure :-1 s=READY pgs=359 cs=0 l=1 rx=0x7fe8d80a0f50 tx=0x7fe8c800fdb0).stop 2019-12-05T05:09:29.585 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:29.558+0000 7fe8df74e700 1 -- 172.21.15.42:0/1206579790 shutdown_connections 2019-12-05T05:09:29.586 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:29.558+0000 7fe8df74e700 1 --2- 172.21.15.42:0/1206579790 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe8d809f2e0 0x7fe8d809f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:09:29.586 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:29.558+0000 7fe8df74e700 1 --2- 172.21.15.42:0/1206579790 >> v2:172.21.15.40:3300/0 conn(0x7fe8d80a7740 0x7fe8d808df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:09:29.586 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:29.558+0000 7fe8df74e700 1 --2- 172.21.15.42:0/1206579790 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe8d807e720 0x7fe8d807eb50 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:09:29.586 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:29.558+0000 7fe8df74e700 1 -- 172.21.15.42:0/1206579790 shutdown_connections 2019-12-05T05:09:29.587 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:29.558+0000 7fe8df74e700 1 -- 172.21.15.42:0/1206579790 wait complete. 2019-12-05T05:09:29.587 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:29.558+0000 7fe8df74e700 1 -- 172.21.15.42:0/1206579790 >> 172.21.15.42:0/1206579790 conn(0x7fe8d80aabe0 msgr2=0x7fe8d80a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:09:29.595 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:29.562+0000 7fe8df74e700 1 Processor -- start 2019-12-05T05:09:29.596 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:29.562+0000 7fe8df74e700 1 -- start start 2019-12-05T05:09:29.596 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:29.562+0000 7fe8df74e700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe8d807c580 0x7fe8d812ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:09:29.596 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:29.562+0000 7fe8df74e700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8d808f080 con 0x7fe8d811abd0 2019-12-05T05:09:29.596 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:29.562+0000 7fe8df74e700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8d811a500 con 0x7fe8d811b000 2019-12-05T05:09:29.597 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:29.562+0000 7fe8df74e700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8d808f400 con 0x7fe8d807c580 2019-12-05T05:09:29.597 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:29.562+0000 7fe8dd4ea700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fe8d811b000 0x7fe8d812a5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:56786/0 (socket says 172.21.15.42:56786) 2019-12-05T05:09:29.597 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:29.562+0000 7fe8dd4ea700 1 -- 172.21.15.42:0/2574701733 learned_addr learned my addr 172.21.15.42:0/2574701733 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:09:29.597 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:29.562+0000 7fe8de4ec700 1 --2- 172.21.15.42:0/2574701733 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe8d807c580 0x7fe8d812ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:09:29.597 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:29.562+0000 7fe8ce7fc700 1 -- 172.21.15.42:0/2574701733 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4188272663 0 0) 0x7fe8d811a500 con 0x7fe8d811b000 2019-12-05T05:09:29.597 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:29.562+0000 7fe8ce7fc700 1 -- 172.21.15.42:0/2574701733 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8c00030b0 con 0x7fe8d811b000 2019-12-05T05:09:29.598 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:29.562+0000 7fe8ce7fc700 1 -- 172.21.15.42:0/2574701733 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4161555258 0 0) 0x7fe8d808f080 con 0x7fe8d811abd0 2019-12-05T05:09:29.598 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:29.562+0000 7fe8ce7fc700 1 -- 172.21.15.42:0/2574701733 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8d811a500 con 0x7fe8d811abd0 2019-12-05T05:09:29.598 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:29.562+0000 7fe8de4ec700 1 -- 172.21.15.42:0/2574701733 >> v1:172.21.15.42:6789/0 conn(0x7fe8d811b000 legacy=0x7fe8d812a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:09:29.598 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:29.562+0000 7fe8de4ec700 1 -- 172.21.15.42:0/2574701733 >> v1:172.21.15.40:6789/0 conn(0x7fe8d811abd0 legacy=0x7fe8d807bd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:09:29.598 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:29.562+0000 7fe8de4ec700 1 -- 172.21.15.42:0/2574701733 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8d811a0e0 con 0x7fe8d807c580 2019-12-05T05:09:29.598 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:29.566+0000 7fe8de4ec700 1 --2- 172.21.15.42:0/2574701733 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe8d807c580 0x7fe8d812ddc0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fe8d00032b0 tx=0x7fe8d0010c80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:29.598 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:29.566+0000 7fe8de4ec700 1 --2- 172.21.15.42:0/2574701733 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe8d807c580 0x7fe8d812ddc0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fe8d00032b0 tx=0x7fe8d0010c80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:29.599 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:29.566+0000 7fe8de4ec700 1 --2- 172.21.15.42:0/2574701733 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe8d807c580 0x7fe8d812ddc0 secure :-1 s=READY pgs=70 cs=0 l=1 rx=0x7fe8d00032b0 tx=0x7fe8d0010c80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:09:29.599 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:29.566+0000 7fe8de4ec700 1 --2- 172.21.15.42:0/2574701733 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe8d807c580 0x7fe8d812ddc0 secure :-1 s=THROTTLE_DONE pgs=70 cs=0 l=1 rx=0x7fe8d00032b0 tx=0x7fe8d0010c80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:29.599 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:29.566+0000 7fe8ce7fc700 1 -- 172.21.15.42:0/2574701733 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe8d0012a00 con 0x7fe8d807c580 2019-12-05T05:09:29.599 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:29.566+0000 7fe8de4ec700 1 --2- 172.21.15.42:0/2574701733 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe8d807c580 0x7fe8d812ddc0 secure :-1 s=THROTTLE_DONE pgs=70 cs=0 l=1 rx=0x7fe8d00032b0 tx=0x7fe8d0010c80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:29.599 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:29.566+0000 7fe8ce7fc700 1 -- 172.21.15.42:0/2574701733 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8d0012620 con 0x7fe8d807c580 2019-12-05T05:09:29.600 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:29.566+0000 7fe8df74e700 1 -- 172.21.15.42:0/2574701733 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8d812e310 con 0x7fe8d807c580 2019-12-05T05:09:29.600 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:29.566+0000 7fe8de4ec700 1 --2- 172.21.15.42:0/2574701733 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe8d807c580 0x7fe8d812ddc0 secure :-1 s=THROTTLE_DONE pgs=70 cs=0 l=1 rx=0x7fe8d00032b0 tx=0x7fe8d0010c80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:29.600 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:29.566+0000 7fe8ce7fc700 1 -- 172.21.15.42:0/2574701733 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe8d000e570 con 0x7fe8d807c580 2019-12-05T05:09:29.600 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:29.566+0000 7fe8df74e700 1 -- 172.21.15.42:0/2574701733 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8d812e890 con 0x7fe8d807c580 2019-12-05T05:09:29.601 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:29.566+0000 7fe8de4ec700 1 --2- 172.21.15.42:0/2574701733 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe8d807c580 0x7fe8d812ddc0 secure :-1 s=THROTTLE_DONE pgs=70 cs=0 l=1 rx=0x7fe8d00032b0 tx=0x7fe8d0010c80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:29.601 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:29.566+0000 7fe8de4ec700 1 --2- 172.21.15.42:0/2574701733 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe8d807c580 0x7fe8d812ddc0 secure :-1 s=THROTTLE_DONE pgs=70 cs=0 l=1 rx=0x7fe8d00032b0 tx=0x7fe8d0010c80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:29.601 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:29.566+0000 7fe8ce7fc700 1 -- 172.21.15.42:0/2574701733 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fe8d0013060 con 0x7fe8d807c580 2019-12-05T05:09:29.601 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:29.566+0000 7fe8ce7fc700 1 --2- 172.21.15.42:0/2574701733 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe8c003b450 0x7fe8c003d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:09:29.601 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:29.566+0000 7fe8ce7fc700 1 -- 172.21.15.42:0/2574701733 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6289+0+0 (secure 0 0 0) 0x7fe8d004b1d0 con 0x7fe8d807c580 2019-12-05T05:09:29.601 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:29.566+0000 7fe8ddceb700 1 --2- 172.21.15.42:0/2574701733 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe8c003b450 0x7fe8c003d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:09:29.602 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:29.566+0000 7fe8ddceb700 1 --2- 172.21.15.42:0/2574701733 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe8c003b450 0x7fe8c003d960 crc :-1 s=READY pgs=162 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:09:29.602 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:29.570+0000 7fe8df74e700 1 -- 172.21.15.42:0/2574701733 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8d805d110 con 0x7fe8d807c580 2019-12-05T05:09:29.602 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:29.578+0000 7fe8de4ec700 1 --2- 172.21.15.42:0/2574701733 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe8d807c580 0x7fe8d812ddc0 secure :-1 s=THROTTLE_DONE pgs=70 cs=0 l=1 rx=0x7fe8d00032b0 tx=0x7fe8d0010c80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:29.602 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:29.578+0000 7fe8ce7fc700 1 -- 172.21.15.42:0/2574701733 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fe8d0032210 con 0x7fe8d807c580 2019-12-05T05:09:29.753 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:29.750+0000 7fe8df74e700 1 -- 172.21.15.42:0/2574701733 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe8d805b8c0 con 0x7fe8d807c580 2019-12-05T05:09:29.754 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:29.750+0000 7fe8de4ec700 1 --2- 172.21.15.42:0/2574701733 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe8d807c580 0x7fe8d812ddc0 secure :-1 s=THROTTLE_DONE pgs=70 cs=0 l=1 rx=0x7fe8d00032b0 tx=0x7fe8d0010c80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:29.754 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:29.750+0000 7fe8ce7fc700 1 -- 172.21.15.42:0/2574701733 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) v1 ==== 94+0+1772 (secure 0 0 0) 0x7fe8d0032470 con 0x7fe8d807c580 2019-12-05T05:09:29.757 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 34 2019-12-05T05:09:29.757 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:09:29.757 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":34,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:09:28.296949+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4978},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4978":{"gid":4978,"name":"a","rank":0,"incarnation":30,"state":"up:active","state_seq":36,"addr":"172.21.15.42:6835/3650564018","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3650564018},{"type":"v1","addr":"172.21.15.42:6835","nonce":3650564018}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:09:29.759 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:29.754+0000 7fe8df74e700 1 -- 172.21.15.42:0/2574701733 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe8c003b450 msgr2=0x7fe8c003d960 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:09:29.760 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:29.754+0000 7fe8df74e700 1 --2- 172.21.15.42:0/2574701733 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe8c003b450 0x7fe8c003d960 crc :-1 s=READY pgs=162 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:09:29.760 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:29.754+0000 7fe8df74e700 1 -- 172.21.15.42:0/2574701733 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe8d807c580 msgr2=0x7fe8d812ddc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:09:29.760 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:29.754+0000 7fe8df74e700 1 --2- 172.21.15.42:0/2574701733 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe8d807c580 0x7fe8d812ddc0 secure :-1 s=READY pgs=70 cs=0 l=1 rx=0x7fe8d00032b0 tx=0x7fe8d0010c80).stop 2019-12-05T05:09:29.760 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:29.754+0000 7fe8df74e700 1 -- 172.21.15.42:0/2574701733 shutdown_connections 2019-12-05T05:09:29.761 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:29.754+0000 7fe8df74e700 1 --2- 172.21.15.42:0/2574701733 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe8c003b450 0x7fe8c003d960 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:09:29.761 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:29.754+0000 7fe8df74e700 1 --2- 172.21.15.42:0/2574701733 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe8d807c580 0x7fe8d812ddc0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:09:29.761 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:29.754+0000 7fe8df74e700 1 -- 172.21.15.42:0/2574701733 shutdown_connections 2019-12-05T05:09:29.761 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:29.754+0000 7fe8df74e700 1 -- 172.21.15.42:0/2574701733 wait complete. 2019-12-05T05:09:29.761 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:29.754+0000 7fe8df74e700 1 -- 172.21.15.42:0/2574701733 >> 172.21.15.42:0/2574701733 conn(0x7fe8d808fda0 msgr2=0x7fe8d8059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:09:29.768 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a has gained rank=0, replacing gid=4831 2019-12-05T05:09:29.768 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 9 secs before reviving mds.b 2019-12-05T05:09:31.506 INFO:tasks.workunit.client.0.smithi042.stdout:removed './246/25' 2019-12-05T05:09:31.506 INFO:tasks.workunit.client.0.smithi042.stdout:removed './246/30' 2019-12-05T05:09:31.506 INFO:tasks.workunit.client.0.smithi042.stdout:removed './246/33' 2019-12-05T05:09:31.506 INFO:tasks.workunit.client.0.smithi042.stdout:removed './246/23' 2019-12-05T05:09:31.506 INFO:tasks.workunit.client.0.smithi042.stdout:removed './246/35' 2019-12-05T05:09:31.507 INFO:tasks.workunit.client.0.smithi042.stdout:removed './246/26' 2019-12-05T05:09:31.507 INFO:tasks.workunit.client.0.smithi042.stdout:removed './246/37' 2019-12-05T05:09:31.507 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './246' 2019-12-05T05:09:31.508 INFO:tasks.workunit.client.0.smithi042.stdout:removed './247/21' 2019-12-05T05:09:31.508 INFO:tasks.workunit.client.0.smithi042.stdout:removed './247/27' 2019-12-05T05:09:31.508 INFO:tasks.workunit.client.0.smithi042.stdout:removed './247/31' 2019-12-05T05:09:31.508 INFO:tasks.workunit.client.0.smithi042.stdout:removed './247/36' 2019-12-05T05:09:31.508 INFO:tasks.workunit.client.0.smithi042.stdout:removed './247/20' 2019-12-05T05:09:31.509 INFO:tasks.workunit.client.0.smithi042.stdout:removed './247/39' 2019-12-05T05:09:31.509 INFO:tasks.workunit.client.0.smithi042.stdout:removed './247/34' 2019-12-05T05:09:31.509 INFO:tasks.workunit.client.0.smithi042.stdout:removed './247/38' 2019-12-05T05:09:31.509 INFO:tasks.workunit.client.0.smithi042.stdout:removed './247/28' 2019-12-05T05:09:31.509 INFO:tasks.workunit.client.0.smithi042.stdout:removed './247/24' 2019-12-05T05:09:31.510 INFO:tasks.workunit.client.0.smithi042.stdout:removed './247/29' 2019-12-05T05:09:31.510 INFO:tasks.workunit.client.0.smithi042.stdout:removed './247/32' 2019-12-05T05:09:31.510 INFO:tasks.workunit.client.0.smithi042.stdout:removed './247/22' 2019-12-05T05:09:31.510 INFO:tasks.workunit.client.0.smithi042.stdout:removed './247/25' 2019-12-05T05:09:31.510 INFO:tasks.workunit.client.0.smithi042.stdout:removed './247/30' 2019-12-05T05:09:31.510 INFO:tasks.workunit.client.0.smithi042.stdout:removed './247/33' 2019-12-05T05:09:31.510 INFO:tasks.workunit.client.0.smithi042.stdout:removed './247/23' 2019-12-05T05:09:31.511 INFO:tasks.workunit.client.0.smithi042.stdout:removed './247/35' 2019-12-05T05:09:31.511 INFO:tasks.workunit.client.0.smithi042.stdout:removed './247/26' 2019-12-05T05:09:31.511 INFO:tasks.workunit.client.0.smithi042.stdout:removed './247/37' 2019-12-05T05:09:31.511 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './247' 2019-12-05T05:09:31.511 INFO:tasks.workunit.client.0.smithi042.stdout:removed './248/21' 2019-12-05T05:09:31.511 INFO:tasks.workunit.client.0.smithi042.stdout:removed './248/27' 2019-12-05T05:09:31.511 INFO:tasks.workunit.client.0.smithi042.stdout:removed './248/31' 2019-12-05T05:09:31.512 INFO:tasks.workunit.client.0.smithi042.stdout:removed './248/36' 2019-12-05T05:09:31.512 INFO:tasks.workunit.client.0.smithi042.stdout:removed './248/20' 2019-12-05T05:09:31.512 INFO:tasks.workunit.client.0.smithi042.stdout:removed './248/39' 2019-12-05T05:09:31.512 INFO:tasks.workunit.client.0.smithi042.stdout:removed './248/34' 2019-12-05T05:09:31.512 INFO:tasks.workunit.client.0.smithi042.stdout:removed './248/38' 2019-12-05T05:09:31.512 INFO:tasks.workunit.client.0.smithi042.stdout:removed './248/28' 2019-12-05T05:09:31.512 INFO:tasks.workunit.client.0.smithi042.stdout:removed './248/24' 2019-12-05T05:09:31.513 INFO:tasks.workunit.client.0.smithi042.stdout:removed './248/29' 2019-12-05T05:09:31.513 INFO:tasks.workunit.client.0.smithi042.stdout:removed './248/32' 2019-12-05T05:09:31.513 INFO:tasks.workunit.client.0.smithi042.stdout:removed './248/22' 2019-12-05T05:09:31.514 INFO:tasks.workunit.client.0.smithi042.stdout:removed './248/25' 2019-12-05T05:09:31.514 INFO:tasks.workunit.client.0.smithi042.stdout:removed './248/30' 2019-12-05T05:09:31.514 INFO:tasks.workunit.client.0.smithi042.stdout:removed './248/33' 2019-12-05T05:09:31.514 INFO:tasks.workunit.client.0.smithi042.stdout:removed './248/23' 2019-12-05T05:09:31.514 INFO:tasks.workunit.client.0.smithi042.stdout:removed './248/35' 2019-12-05T05:09:31.514 INFO:tasks.workunit.client.0.smithi042.stdout:removed './248/26' 2019-12-05T05:09:31.514 INFO:tasks.workunit.client.0.smithi042.stdout:removed './248/37' 2019-12-05T05:09:31.514 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './248' 2019-12-05T05:09:31.515 INFO:tasks.workunit.client.0.smithi042.stdout:removed './249/21' 2019-12-05T05:09:31.515 INFO:tasks.workunit.client.0.smithi042.stdout:removed './249/27' 2019-12-05T05:09:31.515 INFO:tasks.workunit.client.0.smithi042.stdout:removed './249/31' 2019-12-05T05:09:31.515 INFO:tasks.workunit.client.0.smithi042.stdout:removed './249/36' 2019-12-05T05:09:31.515 INFO:tasks.workunit.client.0.smithi042.stdout:removed './249/20' 2019-12-05T05:09:31.515 INFO:tasks.workunit.client.0.smithi042.stdout:removed './249/39' 2019-12-05T05:09:31.515 INFO:tasks.workunit.client.0.smithi042.stdout:removed './249/34' 2019-12-05T05:09:31.516 INFO:tasks.workunit.client.0.smithi042.stdout:removed './249/38' 2019-12-05T05:09:31.516 INFO:tasks.workunit.client.0.smithi042.stdout:removed './249/28' 2019-12-05T05:09:31.516 INFO:tasks.workunit.client.0.smithi042.stdout:removed './249/24' 2019-12-05T05:09:31.516 INFO:tasks.workunit.client.0.smithi042.stdout:removed './249/29' 2019-12-05T05:09:31.516 INFO:tasks.workunit.client.0.smithi042.stdout:removed './249/32' 2019-12-05T05:09:31.516 INFO:tasks.workunit.client.0.smithi042.stdout:removed './249/22' 2019-12-05T05:09:31.516 INFO:tasks.workunit.client.0.smithi042.stdout:removed './249/25' 2019-12-05T05:09:31.517 INFO:tasks.workunit.client.0.smithi042.stdout:removed './249/30' 2019-12-05T05:09:31.517 INFO:tasks.workunit.client.0.smithi042.stdout:removed './249/33' 2019-12-05T05:09:31.517 INFO:tasks.workunit.client.0.smithi042.stdout:removed './249/23' 2019-12-05T05:09:31.517 INFO:tasks.workunit.client.0.smithi042.stdout:removed './249/35' 2019-12-05T05:09:31.517 INFO:tasks.workunit.client.0.smithi042.stdout:removed './249/26' 2019-12-05T05:09:31.517 INFO:tasks.workunit.client.0.smithi042.stdout:removed './249/37' 2019-12-05T05:09:31.517 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './249' 2019-12-05T05:09:31.518 INFO:tasks.workunit.client.0.smithi042.stdout:removed './250/21' 2019-12-05T05:09:31.518 INFO:tasks.workunit.client.0.smithi042.stdout:removed './250/27' 2019-12-05T05:09:31.518 INFO:tasks.workunit.client.0.smithi042.stdout:removed './250/31' 2019-12-05T05:09:31.518 INFO:tasks.workunit.client.0.smithi042.stdout:removed './250/36' 2019-12-05T05:09:31.518 INFO:tasks.workunit.client.0.smithi042.stdout:removed './250/20' 2019-12-05T05:09:31.518 INFO:tasks.workunit.client.0.smithi042.stdout:removed './250/39' 2019-12-05T05:09:31.518 INFO:tasks.workunit.client.0.smithi042.stdout:removed './250/34' 2019-12-05T05:09:31.519 INFO:tasks.workunit.client.0.smithi042.stdout:removed './250/38' 2019-12-05T05:09:31.519 INFO:tasks.workunit.client.0.smithi042.stdout:removed './250/28' 2019-12-05T05:09:31.519 INFO:tasks.workunit.client.0.smithi042.stdout:removed './250/24' 2019-12-05T05:09:31.519 INFO:tasks.workunit.client.0.smithi042.stdout:removed './250/29' 2019-12-05T05:09:31.520 INFO:tasks.workunit.client.0.smithi042.stdout:removed './250/32' 2019-12-05T05:09:31.520 INFO:tasks.workunit.client.0.smithi042.stdout:removed './250/22' 2019-12-05T05:09:31.520 INFO:tasks.workunit.client.0.smithi042.stdout:removed './250/25' 2019-12-05T05:09:31.520 INFO:tasks.workunit.client.0.smithi042.stdout:removed './250/30' 2019-12-05T05:09:31.520 INFO:tasks.workunit.client.0.smithi042.stdout:removed './250/33' 2019-12-05T05:09:31.520 INFO:tasks.workunit.client.0.smithi042.stdout:removed './250/23' 2019-12-05T05:09:31.520 INFO:tasks.workunit.client.0.smithi042.stdout:removed './250/35' 2019-12-05T05:09:31.521 INFO:tasks.workunit.client.0.smithi042.stdout:removed './250/26' 2019-12-05T05:09:31.521 INFO:tasks.workunit.client.0.smithi042.stdout:removed './250/37' 2019-12-05T05:09:31.521 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './250' 2019-12-05T05:09:31.521 INFO:tasks.workunit.client.0.smithi042.stdout:removed './251/21' 2019-12-05T05:09:31.521 INFO:tasks.workunit.client.0.smithi042.stdout:removed './251/27' 2019-12-05T05:09:31.521 INFO:tasks.workunit.client.0.smithi042.stdout:removed './251/31' 2019-12-05T05:09:31.521 INFO:tasks.workunit.client.0.smithi042.stdout:removed './251/36' 2019-12-05T05:09:31.522 INFO:tasks.workunit.client.0.smithi042.stdout:removed './251/20' 2019-12-05T05:09:31.522 INFO:tasks.workunit.client.0.smithi042.stdout:removed './251/39' 2019-12-05T05:09:31.522 INFO:tasks.workunit.client.0.smithi042.stdout:removed './251/34' 2019-12-05T05:09:31.522 INFO:tasks.workunit.client.0.smithi042.stdout:removed './251/38' 2019-12-05T05:09:31.522 INFO:tasks.workunit.client.0.smithi042.stdout:removed './251/28' 2019-12-05T05:09:31.522 INFO:tasks.workunit.client.0.smithi042.stdout:removed './251/24' 2019-12-05T05:09:31.522 INFO:tasks.workunit.client.0.smithi042.stdout:removed './251/29' 2019-12-05T05:09:31.522 INFO:tasks.workunit.client.0.smithi042.stdout:removed './251/32' 2019-12-05T05:09:31.523 INFO:tasks.workunit.client.0.smithi042.stdout:removed './251/22' 2019-12-05T05:09:31.523 INFO:tasks.workunit.client.0.smithi042.stdout:removed './251/25' 2019-12-05T05:09:31.523 INFO:tasks.workunit.client.0.smithi042.stdout:removed './251/30' 2019-12-05T05:09:31.523 INFO:tasks.workunit.client.0.smithi042.stdout:removed './251/33' 2019-12-05T05:09:31.524 INFO:tasks.workunit.client.0.smithi042.stdout:removed './251/23' 2019-12-05T05:09:31.524 INFO:tasks.workunit.client.0.smithi042.stdout:removed './251/35' 2019-12-05T05:09:31.524 INFO:tasks.workunit.client.0.smithi042.stdout:removed './251/26' 2019-12-05T05:09:31.524 INFO:tasks.workunit.client.0.smithi042.stdout:removed './251/37' 2019-12-05T05:09:31.524 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './251' 2019-12-05T05:09:31.524 INFO:tasks.workunit.client.0.smithi042.stdout:removed './252/21' 2019-12-05T05:09:31.524 INFO:tasks.workunit.client.0.smithi042.stdout:removed './252/27' 2019-12-05T05:09:31.525 INFO:tasks.workunit.client.0.smithi042.stdout:removed './252/31' 2019-12-05T05:09:31.525 INFO:tasks.workunit.client.0.smithi042.stdout:removed './252/36' 2019-12-05T05:09:31.525 INFO:tasks.workunit.client.0.smithi042.stdout:removed './252/20' 2019-12-05T05:09:31.525 INFO:tasks.workunit.client.0.smithi042.stdout:removed './252/39' 2019-12-05T05:09:31.525 INFO:tasks.workunit.client.0.smithi042.stdout:removed './252/34' 2019-12-05T05:09:31.525 INFO:tasks.workunit.client.0.smithi042.stdout:removed './252/38' 2019-12-05T05:09:31.525 INFO:tasks.workunit.client.0.smithi042.stdout:removed './252/28' 2019-12-05T05:09:31.525 INFO:tasks.workunit.client.0.smithi042.stdout:removed './252/24' 2019-12-05T05:09:31.526 INFO:tasks.workunit.client.0.smithi042.stdout:removed './252/29' 2019-12-05T05:09:31.526 INFO:tasks.workunit.client.0.smithi042.stdout:removed './252/32' 2019-12-05T05:09:31.526 INFO:tasks.workunit.client.0.smithi042.stdout:removed './252/22' 2019-12-05T05:09:31.526 INFO:tasks.workunit.client.0.smithi042.stdout:removed './252/25' 2019-12-05T05:09:31.527 INFO:tasks.workunit.client.0.smithi042.stdout:removed './252/30' 2019-12-05T05:09:31.527 INFO:tasks.workunit.client.0.smithi042.stdout:removed './252/33' 2019-12-05T05:09:31.528 INFO:tasks.workunit.client.0.smithi042.stdout:removed './252/23' 2019-12-05T05:09:31.528 INFO:tasks.workunit.client.0.smithi042.stdout:removed './252/35' 2019-12-05T05:09:31.528 INFO:tasks.workunit.client.0.smithi042.stdout:removed './252/26' 2019-12-05T05:09:31.528 INFO:tasks.workunit.client.0.smithi042.stdout:removed './252/37' 2019-12-05T05:09:31.528 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './252' 2019-12-05T05:09:31.529 INFO:tasks.workunit.client.0.smithi042.stdout:removed './253/21' 2019-12-05T05:09:31.529 INFO:tasks.workunit.client.0.smithi042.stdout:removed './253/27' 2019-12-05T05:09:31.530 INFO:tasks.workunit.client.0.smithi042.stdout:removed './253/31' 2019-12-05T05:09:31.530 INFO:tasks.workunit.client.0.smithi042.stdout:removed './253/36' 2019-12-05T05:09:31.530 INFO:tasks.workunit.client.0.smithi042.stdout:removed './253/20' 2019-12-05T05:09:31.530 INFO:tasks.workunit.client.0.smithi042.stdout:removed './253/39' 2019-12-05T05:09:31.531 INFO:tasks.workunit.client.0.smithi042.stdout:removed './253/34' 2019-12-05T05:09:31.531 INFO:tasks.workunit.client.0.smithi042.stdout:removed './253/38' 2019-12-05T05:09:31.531 INFO:tasks.workunit.client.0.smithi042.stdout:removed './253/28' 2019-12-05T05:09:31.531 INFO:tasks.workunit.client.0.smithi042.stdout:removed './253/24' 2019-12-05T05:09:31.532 INFO:tasks.workunit.client.0.smithi042.stdout:removed './253/29' 2019-12-05T05:09:31.532 INFO:tasks.workunit.client.0.smithi042.stdout:removed './253/32' 2019-12-05T05:09:31.532 INFO:tasks.workunit.client.0.smithi042.stdout:removed './253/22' 2019-12-05T05:09:31.532 INFO:tasks.workunit.client.0.smithi042.stdout:removed './253/25' 2019-12-05T05:09:31.533 INFO:tasks.workunit.client.0.smithi042.stdout:removed './253/30' 2019-12-05T05:09:31.533 INFO:tasks.workunit.client.0.smithi042.stdout:removed './253/33' 2019-12-05T05:09:31.533 INFO:tasks.workunit.client.0.smithi042.stdout:removed './253/23' 2019-12-05T05:09:31.534 INFO:tasks.workunit.client.0.smithi042.stdout:removed './253/35' 2019-12-05T05:09:31.534 INFO:tasks.workunit.client.0.smithi042.stdout:removed './253/26' 2019-12-05T05:09:31.534 INFO:tasks.workunit.client.0.smithi042.stdout:removed './253/37' 2019-12-05T05:09:31.534 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './253' 2019-12-05T05:09:31.535 INFO:tasks.workunit.client.0.smithi042.stdout:removed './254/21' 2019-12-05T05:09:31.535 INFO:tasks.workunit.client.0.smithi042.stdout:removed './254/27' 2019-12-05T05:09:31.535 INFO:tasks.workunit.client.0.smithi042.stdout:removed './254/31' 2019-12-05T05:09:31.535 INFO:tasks.workunit.client.0.smithi042.stdout:removed './254/36' 2019-12-05T05:09:31.536 INFO:tasks.workunit.client.0.smithi042.stdout:removed './254/20' 2019-12-05T05:09:31.536 INFO:tasks.workunit.client.0.smithi042.stdout:removed './254/39' 2019-12-05T05:09:31.536 INFO:tasks.workunit.client.0.smithi042.stdout:removed './254/34' 2019-12-05T05:09:31.536 INFO:tasks.workunit.client.0.smithi042.stdout:removed './254/38' 2019-12-05T05:09:31.537 INFO:tasks.workunit.client.0.smithi042.stdout:removed './254/28' 2019-12-05T05:09:31.537 INFO:tasks.workunit.client.0.smithi042.stdout:removed './254/24' 2019-12-05T05:09:31.537 INFO:tasks.workunit.client.0.smithi042.stdout:removed './254/29' 2019-12-05T05:09:31.537 INFO:tasks.workunit.client.0.smithi042.stdout:removed './254/32' 2019-12-05T05:09:31.537 INFO:tasks.workunit.client.0.smithi042.stdout:removed './254/22' 2019-12-05T05:09:31.538 INFO:tasks.workunit.client.0.smithi042.stdout:removed './254/25' 2019-12-05T05:09:31.538 INFO:tasks.workunit.client.0.smithi042.stdout:removed './254/30' 2019-12-05T05:09:31.538 INFO:tasks.workunit.client.0.smithi042.stdout:removed './254/33' 2019-12-05T05:09:31.538 INFO:tasks.workunit.client.0.smithi042.stdout:removed './254/23' 2019-12-05T05:09:31.539 INFO:tasks.workunit.client.0.smithi042.stdout:removed './254/35' 2019-12-05T05:09:31.539 INFO:tasks.workunit.client.0.smithi042.stdout:removed './254/26' 2019-12-05T05:09:31.539 INFO:tasks.workunit.client.0.smithi042.stdout:removed './254/37' 2019-12-05T05:09:31.539 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './254' 2019-12-05T05:09:31.540 INFO:tasks.workunit.client.0.smithi042.stdout:removed './255/21' 2019-12-05T05:09:31.541 INFO:tasks.workunit.client.0.smithi042.stdout:removed './255/27' 2019-12-05T05:09:31.541 INFO:tasks.workunit.client.0.smithi042.stdout:removed './255/31' 2019-12-05T05:09:31.541 INFO:tasks.workunit.client.0.smithi042.stdout:removed './255/36' 2019-12-05T05:09:31.541 INFO:tasks.workunit.client.0.smithi042.stdout:removed './255/20' 2019-12-05T05:09:31.542 INFO:tasks.workunit.client.0.smithi042.stdout:removed './255/39' 2019-12-05T05:09:31.542 INFO:tasks.workunit.client.0.smithi042.stdout:removed './255/34' 2019-12-05T05:09:31.542 INFO:tasks.workunit.client.0.smithi042.stdout:removed './255/38' 2019-12-05T05:09:31.542 INFO:tasks.workunit.client.0.smithi042.stdout:removed './255/28' 2019-12-05T05:09:31.543 INFO:tasks.workunit.client.0.smithi042.stdout:removed './255/24' 2019-12-05T05:09:31.543 INFO:tasks.workunit.client.0.smithi042.stdout:removed './255/29' 2019-12-05T05:09:31.543 INFO:tasks.workunit.client.0.smithi042.stdout:removed './255/32' 2019-12-05T05:09:31.543 INFO:tasks.workunit.client.0.smithi042.stdout:removed './255/22' 2019-12-05T05:09:31.544 INFO:tasks.workunit.client.0.smithi042.stdout:removed './255/25' 2019-12-05T05:09:31.544 INFO:tasks.workunit.client.0.smithi042.stdout:removed './255/30' 2019-12-05T05:09:31.545 INFO:tasks.workunit.client.0.smithi042.stdout:removed './255/33' 2019-12-05T05:09:31.545 INFO:tasks.workunit.client.0.smithi042.stdout:removed './255/23' 2019-12-05T05:09:31.545 INFO:tasks.workunit.client.0.smithi042.stdout:removed './255/35' 2019-12-05T05:09:31.545 INFO:tasks.workunit.client.0.smithi042.stdout:removed './255/26' 2019-12-05T05:09:31.546 INFO:tasks.workunit.client.0.smithi042.stdout:removed './255/37' 2019-12-05T05:09:31.546 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './255' 2019-12-05T05:09:31.546 INFO:tasks.workunit.client.0.smithi042.stdout:removed './256/21' 2019-12-05T05:09:31.546 INFO:tasks.workunit.client.0.smithi042.stdout:removed './256/27' 2019-12-05T05:09:31.546 INFO:tasks.workunit.client.0.smithi042.stdout:removed './256/31' 2019-12-05T05:09:31.547 INFO:tasks.workunit.client.0.smithi042.stdout:removed './256/36' 2019-12-05T05:09:31.547 INFO:tasks.workunit.client.0.smithi042.stdout:removed './256/20' 2019-12-05T05:09:31.547 INFO:tasks.workunit.client.0.smithi042.stdout:removed './256/39' 2019-12-05T05:09:31.547 INFO:tasks.workunit.client.0.smithi042.stdout:removed './256/34' 2019-12-05T05:09:31.548 INFO:tasks.workunit.client.0.smithi042.stdout:removed './256/38' 2019-12-05T05:09:31.548 INFO:tasks.workunit.client.0.smithi042.stdout:removed './256/28' 2019-12-05T05:09:31.548 INFO:tasks.workunit.client.0.smithi042.stdout:removed './256/24' 2019-12-05T05:09:31.548 INFO:tasks.workunit.client.0.smithi042.stdout:removed './256/29' 2019-12-05T05:09:31.548 INFO:tasks.workunit.client.0.smithi042.stdout:removed './256/32' 2019-12-05T05:09:31.549 INFO:tasks.workunit.client.0.smithi042.stdout:removed './256/22' 2019-12-05T05:09:31.549 INFO:tasks.workunit.client.0.smithi042.stdout:removed './256/25' 2019-12-05T05:09:31.549 INFO:tasks.workunit.client.0.smithi042.stdout:removed './256/30' 2019-12-05T05:09:34.986 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:09:35.007 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:09:37.151 INFO:tasks.workunit.client.0.smithi042.stdout:removed './256/33' 2019-12-05T05:09:37.152 INFO:tasks.workunit.client.0.smithi042.stdout:removed './256/23' 2019-12-05T05:09:37.152 INFO:tasks.workunit.client.0.smithi042.stdout:removed './256/35' 2019-12-05T05:09:37.152 INFO:tasks.workunit.client.0.smithi042.stdout:removed './256/26' 2019-12-05T05:09:37.152 INFO:tasks.workunit.client.0.smithi042.stdout:removed './256/37' 2019-12-05T05:09:37.152 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './256' 2019-12-05T05:09:37.152 INFO:tasks.workunit.client.0.smithi042.stdout:removed './257/21' 2019-12-05T05:09:37.152 INFO:tasks.workunit.client.0.smithi042.stdout:removed './257/27' 2019-12-05T05:09:37.153 INFO:tasks.workunit.client.0.smithi042.stdout:removed './257/31' 2019-12-05T05:09:37.153 INFO:tasks.workunit.client.0.smithi042.stdout:removed './257/36' 2019-12-05T05:09:37.153 INFO:tasks.workunit.client.0.smithi042.stdout:removed './257/20' 2019-12-05T05:09:37.153 INFO:tasks.workunit.client.0.smithi042.stdout:removed './257/39' 2019-12-05T05:09:37.153 INFO:tasks.workunit.client.0.smithi042.stdout:removed './257/34' 2019-12-05T05:09:37.153 INFO:tasks.workunit.client.0.smithi042.stdout:removed './257/38' 2019-12-05T05:09:37.154 INFO:tasks.workunit.client.0.smithi042.stdout:removed './257/28' 2019-12-05T05:09:37.154 INFO:tasks.workunit.client.0.smithi042.stdout:removed './257/24' 2019-12-05T05:09:37.154 INFO:tasks.workunit.client.0.smithi042.stdout:removed './257/29' 2019-12-05T05:09:37.154 INFO:tasks.workunit.client.0.smithi042.stdout:removed './257/32' 2019-12-05T05:09:37.154 INFO:tasks.workunit.client.0.smithi042.stdout:removed './257/22' 2019-12-05T05:09:37.155 INFO:tasks.workunit.client.0.smithi042.stdout:removed './257/25' 2019-12-05T05:09:37.155 INFO:tasks.workunit.client.0.smithi042.stdout:removed './257/30' 2019-12-05T05:09:37.155 INFO:tasks.workunit.client.0.smithi042.stdout:removed './257/33' 2019-12-05T05:09:37.155 INFO:tasks.workunit.client.0.smithi042.stdout:removed './257/23' 2019-12-05T05:09:37.156 INFO:tasks.workunit.client.0.smithi042.stdout:removed './257/35' 2019-12-05T05:09:37.156 INFO:tasks.workunit.client.0.smithi042.stdout:removed './257/26' 2019-12-05T05:09:37.156 INFO:tasks.workunit.client.0.smithi042.stdout:removed './257/37' 2019-12-05T05:09:37.157 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './257' 2019-12-05T05:09:37.158 INFO:tasks.workunit.client.0.smithi042.stdout:removed './258/21' 2019-12-05T05:09:37.158 INFO:tasks.workunit.client.0.smithi042.stdout:removed './258/27' 2019-12-05T05:09:37.158 INFO:tasks.workunit.client.0.smithi042.stdout:removed './258/31' 2019-12-05T05:09:37.158 INFO:tasks.workunit.client.0.smithi042.stdout:removed './258/36' 2019-12-05T05:09:37.159 INFO:tasks.workunit.client.0.smithi042.stdout:removed './258/20' 2019-12-05T05:09:37.159 INFO:tasks.workunit.client.0.smithi042.stdout:removed './258/39' 2019-12-05T05:09:37.159 INFO:tasks.workunit.client.0.smithi042.stdout:removed './258/34' 2019-12-05T05:09:37.159 INFO:tasks.workunit.client.0.smithi042.stdout:removed './258/38' 2019-12-05T05:09:37.159 INFO:tasks.workunit.client.0.smithi042.stdout:removed './258/28' 2019-12-05T05:09:37.159 INFO:tasks.workunit.client.0.smithi042.stdout:removed './258/24' 2019-12-05T05:09:37.160 INFO:tasks.workunit.client.0.smithi042.stdout:removed './258/29' 2019-12-05T05:09:37.160 INFO:tasks.workunit.client.0.smithi042.stdout:removed './258/32' 2019-12-05T05:09:37.161 INFO:tasks.workunit.client.0.smithi042.stdout:removed './258/22' 2019-12-05T05:09:37.161 INFO:tasks.workunit.client.0.smithi042.stdout:removed './258/25' 2019-12-05T05:09:37.161 INFO:tasks.workunit.client.0.smithi042.stdout:removed './258/30' 2019-12-05T05:09:37.161 INFO:tasks.workunit.client.0.smithi042.stdout:removed './258/33' 2019-12-05T05:09:37.161 INFO:tasks.workunit.client.0.smithi042.stdout:removed './258/23' 2019-12-05T05:09:37.162 INFO:tasks.workunit.client.0.smithi042.stdout:removed './258/35' 2019-12-05T05:09:37.162 INFO:tasks.workunit.client.0.smithi042.stdout:removed './258/26' 2019-12-05T05:09:37.162 INFO:tasks.workunit.client.0.smithi042.stdout:removed './258/37' 2019-12-05T05:09:37.162 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './258' 2019-12-05T05:09:37.163 INFO:tasks.workunit.client.0.smithi042.stdout:removed './259/21' 2019-12-05T05:09:37.163 INFO:tasks.workunit.client.0.smithi042.stdout:removed './259/27' 2019-12-05T05:09:37.164 INFO:tasks.workunit.client.0.smithi042.stdout:removed './259/31' 2019-12-05T05:09:37.164 INFO:tasks.workunit.client.0.smithi042.stdout:removed './259/36' 2019-12-05T05:09:37.164 INFO:tasks.workunit.client.0.smithi042.stdout:removed './259/20' 2019-12-05T05:09:37.164 INFO:tasks.workunit.client.0.smithi042.stdout:removed './259/39' 2019-12-05T05:09:37.165 INFO:tasks.workunit.client.0.smithi042.stdout:removed './259/34' 2019-12-05T05:09:37.165 INFO:tasks.workunit.client.0.smithi042.stdout:removed './259/38' 2019-12-05T05:09:37.165 INFO:tasks.workunit.client.0.smithi042.stdout:removed './259/28' 2019-12-05T05:09:37.165 INFO:tasks.workunit.client.0.smithi042.stdout:removed './259/24' 2019-12-05T05:09:37.166 INFO:tasks.workunit.client.0.smithi042.stdout:removed './259/29' 2019-12-05T05:09:37.166 INFO:tasks.workunit.client.0.smithi042.stdout:removed './259/32' 2019-12-05T05:09:37.166 INFO:tasks.workunit.client.0.smithi042.stdout:removed './259/22' 2019-12-05T05:09:37.166 INFO:tasks.workunit.client.0.smithi042.stdout:removed './259/25' 2019-12-05T05:09:37.166 INFO:tasks.workunit.client.0.smithi042.stdout:removed './259/30' 2019-12-05T05:09:37.167 INFO:tasks.workunit.client.0.smithi042.stdout:removed './259/33' 2019-12-05T05:09:37.167 INFO:tasks.workunit.client.0.smithi042.stdout:removed './259/23' 2019-12-05T05:09:37.167 INFO:tasks.workunit.client.0.smithi042.stdout:removed './259/35' 2019-12-05T05:09:37.167 INFO:tasks.workunit.client.0.smithi042.stdout:removed './259/26' 2019-12-05T05:09:37.174 INFO:tasks.workunit.client.0.smithi042.stdout:removed './259/37' 2019-12-05T05:09:37.175 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './259' 2019-12-05T05:09:37.175 INFO:tasks.workunit.client.0.smithi042.stdout:removed './260/21' 2019-12-05T05:09:37.175 INFO:tasks.workunit.client.0.smithi042.stdout:removed './260/27' 2019-12-05T05:09:37.185 INFO:tasks.workunit.client.0.smithi042.stdout:removed './260/31' 2019-12-05T05:09:37.185 INFO:tasks.workunit.client.0.smithi042.stdout:removed './260/36' 2019-12-05T05:09:37.186 INFO:tasks.workunit.client.0.smithi042.stdout:removed './260/20' 2019-12-05T05:09:37.186 INFO:tasks.workunit.client.0.smithi042.stdout:removed './260/39' 2019-12-05T05:09:37.186 INFO:tasks.workunit.client.0.smithi042.stdout:removed './260/34' 2019-12-05T05:09:37.186 INFO:tasks.workunit.client.0.smithi042.stdout:removed './260/38' 2019-12-05T05:09:37.187 INFO:tasks.workunit.client.0.smithi042.stdout:removed './260/28' 2019-12-05T05:09:37.187 INFO:tasks.workunit.client.0.smithi042.stdout:removed './260/24' 2019-12-05T05:09:37.187 INFO:tasks.workunit.client.0.smithi042.stdout:removed './260/29' 2019-12-05T05:09:37.187 INFO:tasks.workunit.client.0.smithi042.stdout:removed './260/32' 2019-12-05T05:09:37.187 INFO:tasks.workunit.client.0.smithi042.stdout:removed './260/22' 2019-12-05T05:09:37.188 INFO:tasks.workunit.client.0.smithi042.stdout:removed './260/25' 2019-12-05T05:09:37.188 INFO:tasks.workunit.client.0.smithi042.stdout:removed './260/30' 2019-12-05T05:09:37.188 INFO:tasks.workunit.client.0.smithi042.stdout:removed './260/33' 2019-12-05T05:09:37.188 INFO:tasks.workunit.client.0.smithi042.stdout:removed './260/23' 2019-12-05T05:09:37.189 INFO:tasks.workunit.client.0.smithi042.stdout:removed './260/35' 2019-12-05T05:09:37.189 INFO:tasks.workunit.client.0.smithi042.stdout:removed './260/26' 2019-12-05T05:09:37.189 INFO:tasks.workunit.client.0.smithi042.stdout:removed './260/37' 2019-12-05T05:09:37.189 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './260' 2019-12-05T05:09:37.190 INFO:tasks.workunit.client.0.smithi042.stdout:removed './261/21' 2019-12-05T05:09:37.190 INFO:tasks.workunit.client.0.smithi042.stdout:removed './261/27' 2019-12-05T05:09:37.190 INFO:tasks.workunit.client.0.smithi042.stdout:removed './261/31' 2019-12-05T05:09:37.190 INFO:tasks.workunit.client.0.smithi042.stdout:removed './261/36' 2019-12-05T05:09:37.190 INFO:tasks.workunit.client.0.smithi042.stdout:removed './261/20' 2019-12-05T05:09:37.191 INFO:tasks.workunit.client.0.smithi042.stdout:removed './261/39' 2019-12-05T05:09:37.191 INFO:tasks.workunit.client.0.smithi042.stdout:removed './261/34' 2019-12-05T05:09:37.191 INFO:tasks.workunit.client.0.smithi042.stdout:removed './261/38' 2019-12-05T05:09:37.191 INFO:tasks.workunit.client.0.smithi042.stdout:removed './261/28' 2019-12-05T05:09:37.192 INFO:tasks.workunit.client.0.smithi042.stdout:removed './261/24' 2019-12-05T05:09:37.192 INFO:tasks.workunit.client.0.smithi042.stdout:removed './261/29' 2019-12-05T05:09:37.192 INFO:tasks.workunit.client.0.smithi042.stdout:removed './261/32' 2019-12-05T05:09:37.192 INFO:tasks.workunit.client.0.smithi042.stdout:removed './261/22' 2019-12-05T05:09:37.193 INFO:tasks.workunit.client.0.smithi042.stdout:removed './261/25' 2019-12-05T05:09:37.193 INFO:tasks.workunit.client.0.smithi042.stdout:removed './261/30' 2019-12-05T05:09:37.193 INFO:tasks.workunit.client.0.smithi042.stdout:removed './261/33' 2019-12-05T05:09:37.193 INFO:tasks.workunit.client.0.smithi042.stdout:removed './261/23' 2019-12-05T05:09:37.193 INFO:tasks.workunit.client.0.smithi042.stdout:removed './261/35' 2019-12-05T05:09:37.194 INFO:tasks.workunit.client.0.smithi042.stdout:removed './261/26' 2019-12-05T05:09:37.194 INFO:tasks.workunit.client.0.smithi042.stdout:removed './261/37' 2019-12-05T05:09:37.194 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './261' 2019-12-05T05:09:37.194 INFO:tasks.workunit.client.0.smithi042.stdout:removed './262/21' 2019-12-05T05:09:37.195 INFO:tasks.workunit.client.0.smithi042.stdout:removed './262/27' 2019-12-05T05:09:37.195 INFO:tasks.workunit.client.0.smithi042.stdout:removed './262/31' 2019-12-05T05:09:37.196 INFO:tasks.workunit.client.0.smithi042.stdout:removed './262/36' 2019-12-05T05:09:37.196 INFO:tasks.workunit.client.0.smithi042.stdout:removed './262/20' 2019-12-05T05:09:37.196 INFO:tasks.workunit.client.0.smithi042.stdout:removed './262/39' 2019-12-05T05:09:37.196 INFO:tasks.workunit.client.0.smithi042.stdout:removed './262/34' 2019-12-05T05:09:37.196 INFO:tasks.workunit.client.0.smithi042.stdout:removed './262/38' 2019-12-05T05:09:37.197 INFO:tasks.workunit.client.0.smithi042.stdout:removed './262/28' 2019-12-05T05:09:37.197 INFO:tasks.workunit.client.0.smithi042.stdout:removed './262/24' 2019-12-05T05:09:37.197 INFO:tasks.workunit.client.0.smithi042.stdout:removed './262/29' 2019-12-05T05:09:37.197 INFO:tasks.workunit.client.0.smithi042.stdout:removed './262/32' 2019-12-05T05:09:37.197 INFO:tasks.workunit.client.0.smithi042.stdout:removed './262/22' 2019-12-05T05:09:37.198 INFO:tasks.workunit.client.0.smithi042.stdout:removed './262/25' 2019-12-05T05:09:37.198 INFO:tasks.workunit.client.0.smithi042.stdout:removed './262/30' 2019-12-05T05:09:37.198 INFO:tasks.workunit.client.0.smithi042.stdout:removed './262/33' 2019-12-05T05:09:37.198 INFO:tasks.workunit.client.0.smithi042.stdout:removed './262/23' 2019-12-05T05:09:37.199 INFO:tasks.workunit.client.0.smithi042.stdout:removed './262/35' 2019-12-05T05:09:37.199 INFO:tasks.workunit.client.0.smithi042.stdout:removed './262/26' 2019-12-05T05:09:37.199 INFO:tasks.workunit.client.0.smithi042.stdout:removed './262/37' 2019-12-05T05:09:37.199 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './262' 2019-12-05T05:09:37.199 INFO:tasks.workunit.client.0.smithi042.stdout:removed './263/21' 2019-12-05T05:09:37.200 INFO:tasks.workunit.client.0.smithi042.stdout:removed './263/27' 2019-12-05T05:09:37.200 INFO:tasks.workunit.client.0.smithi042.stdout:removed './263/31' 2019-12-05T05:09:37.200 INFO:tasks.workunit.client.0.smithi042.stdout:removed './263/36' 2019-12-05T05:09:37.200 INFO:tasks.workunit.client.0.smithi042.stdout:removed './263/20' 2019-12-05T05:09:37.200 INFO:tasks.workunit.client.0.smithi042.stdout:removed './263/39' 2019-12-05T05:09:37.201 INFO:tasks.workunit.client.0.smithi042.stdout:removed './263/34' 2019-12-05T05:09:37.201 INFO:tasks.workunit.client.0.smithi042.stdout:removed './263/38' 2019-12-05T05:09:37.201 INFO:tasks.workunit.client.0.smithi042.stdout:removed './263/28' 2019-12-05T05:09:37.201 INFO:tasks.workunit.client.0.smithi042.stdout:removed './263/24' 2019-12-05T05:09:37.201 INFO:tasks.workunit.client.0.smithi042.stdout:removed './263/29' 2019-12-05T05:09:37.202 INFO:tasks.workunit.client.0.smithi042.stdout:removed './263/32' 2019-12-05T05:09:37.202 INFO:tasks.workunit.client.0.smithi042.stdout:removed './263/22' 2019-12-05T05:09:37.202 INFO:tasks.workunit.client.0.smithi042.stdout:removed './263/25' 2019-12-05T05:09:37.202 INFO:tasks.workunit.client.0.smithi042.stdout:removed './263/30' 2019-12-05T05:09:37.202 INFO:tasks.workunit.client.0.smithi042.stdout:removed './263/33' 2019-12-05T05:09:37.202 INFO:tasks.workunit.client.0.smithi042.stdout:removed './263/23' 2019-12-05T05:09:37.203 INFO:tasks.workunit.client.0.smithi042.stdout:removed './263/35' 2019-12-05T05:09:37.203 INFO:tasks.workunit.client.0.smithi042.stdout:removed './263/26' 2019-12-05T05:09:37.203 INFO:tasks.workunit.client.0.smithi042.stdout:removed './263/37' 2019-12-05T05:09:37.203 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './263' 2019-12-05T05:09:37.203 INFO:tasks.workunit.client.0.smithi042.stdout:removed './264/21' 2019-12-05T05:09:37.204 INFO:tasks.workunit.client.0.smithi042.stdout:removed './264/27' 2019-12-05T05:09:37.204 INFO:tasks.workunit.client.0.smithi042.stdout:removed './264/31' 2019-12-05T05:09:37.204 INFO:tasks.workunit.client.0.smithi042.stdout:removed './264/36' 2019-12-05T05:09:37.204 INFO:tasks.workunit.client.0.smithi042.stdout:removed './264/20' 2019-12-05T05:09:37.204 INFO:tasks.workunit.client.0.smithi042.stdout:removed './264/39' 2019-12-05T05:09:37.205 INFO:tasks.workunit.client.0.smithi042.stdout:removed './264/34' 2019-12-05T05:09:37.205 INFO:tasks.workunit.client.0.smithi042.stdout:removed './264/38' 2019-12-05T05:09:37.205 INFO:tasks.workunit.client.0.smithi042.stdout:removed './264/28' 2019-12-05T05:09:37.205 INFO:tasks.workunit.client.0.smithi042.stdout:removed './264/24' 2019-12-05T05:09:37.205 INFO:tasks.workunit.client.0.smithi042.stdout:removed './264/29' 2019-12-05T05:09:37.206 INFO:tasks.workunit.client.0.smithi042.stdout:removed './264/32' 2019-12-05T05:09:37.206 INFO:tasks.workunit.client.0.smithi042.stdout:removed './264/22' 2019-12-05T05:09:37.206 INFO:tasks.workunit.client.0.smithi042.stdout:removed './264/25' 2019-12-05T05:09:37.206 INFO:tasks.workunit.client.0.smithi042.stdout:removed './264/30' 2019-12-05T05:09:37.206 INFO:tasks.workunit.client.0.smithi042.stdout:removed './264/33' 2019-12-05T05:09:37.207 INFO:tasks.workunit.client.0.smithi042.stdout:removed './264/23' 2019-12-05T05:09:37.207 INFO:tasks.workunit.client.0.smithi042.stdout:removed './264/35' 2019-12-05T05:09:37.207 INFO:tasks.workunit.client.0.smithi042.stdout:removed './264/26' 2019-12-05T05:09:37.207 INFO:tasks.workunit.client.0.smithi042.stdout:removed './264/37' 2019-12-05T05:09:37.207 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './264' 2019-12-05T05:09:37.207 INFO:tasks.workunit.client.0.smithi042.stdout:removed './265/21' 2019-12-05T05:09:37.207 INFO:tasks.workunit.client.0.smithi042.stdout:removed './265/27' 2019-12-05T05:09:37.208 INFO:tasks.workunit.client.0.smithi042.stdout:removed './265/31' 2019-12-05T05:09:37.208 INFO:tasks.workunit.client.0.smithi042.stdout:removed './265/36' 2019-12-05T05:09:37.208 INFO:tasks.workunit.client.0.smithi042.stdout:removed './265/20' 2019-12-05T05:09:37.208 INFO:tasks.workunit.client.0.smithi042.stdout:removed './265/39' 2019-12-05T05:09:37.208 INFO:tasks.workunit.client.0.smithi042.stdout:removed './265/34' 2019-12-05T05:09:37.208 INFO:tasks.workunit.client.0.smithi042.stdout:removed './265/38' 2019-12-05T05:09:37.209 INFO:tasks.workunit.client.0.smithi042.stdout:removed './265/28' 2019-12-05T05:09:37.209 INFO:tasks.workunit.client.0.smithi042.stdout:removed './265/24' 2019-12-05T05:09:37.209 INFO:tasks.workunit.client.0.smithi042.stdout:removed './265/29' 2019-12-05T05:09:37.209 INFO:tasks.workunit.client.0.smithi042.stdout:removed './265/32' 2019-12-05T05:09:37.210 INFO:tasks.workunit.client.0.smithi042.stdout:removed './265/22' 2019-12-05T05:09:37.210 INFO:tasks.workunit.client.0.smithi042.stdout:removed './265/25' 2019-12-05T05:09:37.210 INFO:tasks.workunit.client.0.smithi042.stdout:removed './265/30' 2019-12-05T05:09:37.210 INFO:tasks.workunit.client.0.smithi042.stdout:removed './265/33' 2019-12-05T05:09:37.210 INFO:tasks.workunit.client.0.smithi042.stdout:removed './265/23' 2019-12-05T05:09:37.211 INFO:tasks.workunit.client.0.smithi042.stdout:removed './265/35' 2019-12-05T05:09:37.211 INFO:tasks.workunit.client.0.smithi042.stdout:removed './265/26' 2019-12-05T05:09:37.211 INFO:tasks.workunit.client.0.smithi042.stdout:removed './265/37' 2019-12-05T05:09:37.211 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './265' 2019-12-05T05:09:37.212 INFO:tasks.workunit.client.0.smithi042.stdout:removed './266/21' 2019-12-05T05:09:37.212 INFO:tasks.workunit.client.0.smithi042.stdout:removed './266/27' 2019-12-05T05:09:37.212 INFO:tasks.workunit.client.0.smithi042.stdout:removed './266/31' 2019-12-05T05:09:37.212 INFO:tasks.workunit.client.0.smithi042.stdout:removed './266/36' 2019-12-05T05:09:37.213 INFO:tasks.workunit.client.0.smithi042.stdout:removed './266/20' 2019-12-05T05:09:37.213 INFO:tasks.workunit.client.0.smithi042.stdout:removed './266/39' 2019-12-05T05:09:37.213 INFO:tasks.workunit.client.0.smithi042.stdout:removed './266/34' 2019-12-05T05:09:37.213 INFO:tasks.workunit.client.0.smithi042.stdout:removed './266/38' 2019-12-05T05:09:37.213 INFO:tasks.workunit.client.0.smithi042.stdout:removed './266/28' 2019-12-05T05:09:37.215 INFO:tasks.workunit.client.0.smithi042.stdout:removed './266/24' 2019-12-05T05:09:37.215 INFO:tasks.workunit.client.0.smithi042.stdout:removed './266/29' 2019-12-05T05:09:37.216 INFO:tasks.workunit.client.0.smithi042.stdout:removed './266/32' 2019-12-05T05:09:37.216 INFO:tasks.workunit.client.0.smithi042.stdout:removed './266/22' 2019-12-05T05:09:37.217 INFO:tasks.workunit.client.0.smithi042.stdout:removed './266/25' 2019-12-05T05:09:37.217 INFO:tasks.workunit.client.0.smithi042.stdout:removed './266/30' 2019-12-05T05:09:37.218 INFO:tasks.workunit.client.0.smithi042.stdout:removed './266/33' 2019-12-05T05:09:37.218 INFO:tasks.workunit.client.0.smithi042.stdout:removed './266/23' 2019-12-05T05:09:38.769 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.b 2019-12-05T05:09:38.769 INFO:tasks.ceph.mds.b:Restarting daemon 2019-12-05T05:09:38.769 INFO:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2019-12-05T05:09:38.773 INFO:tasks.ceph.mds.b:Started 2019-12-05T05:09:38.773 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:09:38.834 INFO:tasks.ceph.mds.b.smithi040.stdout:starting mds.b at 2019-12-05T05:09:38.884 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:38.874+0000 7f2cfc556700 1 Processor -- start 2019-12-05T05:09:38.884 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:38.874+0000 7f2cfc556700 1 -- start start 2019-12-05T05:09:38.884 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:38.874+0000 7f2cfc556700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2cf408df80 0x7f2cf409f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:09:38.884 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:38.874+0000 7f2cfc556700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2cf409f830 0x7f2cf414efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:09:38.884 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:38.874+0000 7f2cfc556700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cf409fc60 con 0x7f2cf40a7740 2019-12-05T05:09:38.884 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:38.874+0000 7f2cfc556700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2cf40a0b00 con 0x7f2cf408df80 2019-12-05T05:09:38.885 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:38.874+0000 7f2cfc556700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cf408e5b0 con 0x7f2cf409f830 2019-12-05T05:09:38.885 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:38.878+0000 7f2cfa2f2700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2cf408df80 0x7f2cf409f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:09:38.885 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:38.878+0000 7f2cfb2f4700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2cf409f830 0x7f2cf414efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:09:38.885 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:38.878+0000 7f2cfa2f2700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2cf408df80 0x7f2cf409f2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.42:44638/0 (socket says 172.21.15.42:44638) 2019-12-05T05:09:38.885 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:38.878+0000 7f2cfa2f2700 1 -- 172.21.15.42:0/1415699282 learned_addr learned my addr 172.21.15.42:0/1415699282 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:09:38.885 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:38.878+0000 7f2cfa2f2700 1 -- 172.21.15.42:0/1415699282 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2cf409f830 msgr2=0x7f2cf414efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:09:38.886 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:38.878+0000 7f2cfa2f2700 1 --2- 172.21.15.42:0/1415699282 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2cf409f830 0x7f2cf414efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:09:38.886 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:38.878+0000 7f2cfa2f2700 1 -- 172.21.15.42:0/1415699282 >> v1:172.21.15.40:6789/0 conn(0x7f2cf40a7740 legacy=0x7f2cf407e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:09:38.886 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:38.878+0000 7f2cfa2f2700 1 -- 172.21.15.42:0/1415699282 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cf40a0710 con 0x7f2cf408df80 2019-12-05T05:09:38.886 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:38.878+0000 7f2cfa2f2700 1 --2- 172.21.15.42:0/1415699282 >> v2:172.21.15.40:3300/0 conn(0x7f2cf408df80 0x7f2cf409f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f2ce8008790 tx=0x7f2ce800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:38.886 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:38.878+0000 7f2cfa2f2700 1 --2- 172.21.15.42:0/1415699282 >> v2:172.21.15.40:3300/0 conn(0x7f2cf408df80 0x7f2cf409f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f2ce8008790 tx=0x7f2ce800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:38.886 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:38.878+0000 7f2cfa2f2700 1 --2- 172.21.15.42:0/1415699282 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2cf408df80 0x7f2cf409f2e0 secure :-1 s=READY pgs=283 cs=0 l=1 rx=0x7f2ce8008790 tx=0x7f2ce800fd30).ready entity=mon.0 client_cookie=309282668314da40 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:09:38.886 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:38.878+0000 7f2cfa2f2700 1 --2- 172.21.15.42:0/1415699282 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2cf408df80 0x7f2cf409f2e0 secure :-1 s=THROTTLE_DONE pgs=283 cs=0 l=1 rx=0x7f2ce8008790 tx=0x7f2ce800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:38.887 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:38.878+0000 7f2cfa2f2700 1 --2- 172.21.15.42:0/1415699282 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2cf408df80 0x7f2cf409f2e0 secure :-1 s=THROTTLE_DONE pgs=283 cs=0 l=1 rx=0x7f2ce8008790 tx=0x7f2ce800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:38.887 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:38.878+0000 7f2cf9af1700 1 -- 172.21.15.42:0/1415699282 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2ce8008c90 con 0x7f2cf408df80 2019-12-05T05:09:38.887 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:38.878+0000 7f2cfa2f2700 1 --2- 172.21.15.42:0/1415699282 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2cf408df80 0x7f2cf409f2e0 secure :-1 s=THROTTLE_DONE pgs=283 cs=0 l=1 rx=0x7f2ce8008790 tx=0x7f2ce800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:38.887 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:38.878+0000 7f2cf9af1700 1 -- 172.21.15.42:0/1415699282 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ce80088b0 con 0x7f2cf408df80 2019-12-05T05:09:38.888 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:38.878+0000 7f2cf9af1700 1 -- 172.21.15.42:0/1415699282 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2ce8011b10 con 0x7f2cf408df80 2019-12-05T05:09:38.888 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:38.878+0000 7f2cfc556700 1 -- 172.21.15.42:0/1415699282 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2cf408df80 msgr2=0x7f2cf409f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:09:38.888 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:38.878+0000 7f2cfc556700 1 --2- 172.21.15.42:0/1415699282 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2cf408df80 0x7f2cf409f2e0 secure :-1 s=READY pgs=283 cs=0 l=1 rx=0x7f2ce8008790 tx=0x7f2ce800fd30).stop 2019-12-05T05:09:38.888 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:38.878+0000 7f2cfc556700 1 -- 172.21.15.42:0/1415699282 shutdown_connections 2019-12-05T05:09:38.888 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:38.878+0000 7f2cfc556700 1 --2- 172.21.15.42:0/1415699282 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2cf409f830 0x7f2cf414efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:09:38.888 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:38.878+0000 7f2cfc556700 1 --2- 172.21.15.42:0/1415699282 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2cf408df80 0x7f2cf409f2e0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:09:38.888 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:38.878+0000 7f2cfc556700 1 -- 172.21.15.42:0/1415699282 shutdown_connections 2019-12-05T05:09:38.889 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:38.878+0000 7f2cfc556700 1 -- 172.21.15.42:0/1415699282 wait complete. 2019-12-05T05:09:38.889 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:38.878+0000 7f2cfc556700 1 -- 172.21.15.42:0/1415699282 >> 172.21.15.42:0/1415699282 conn(0x7f2cf40aabe0 msgr2=0x7f2cf40a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:09:38.890 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:38.882+0000 7f2cfc556700 1 Processor -- start 2019-12-05T05:09:38.891 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:38.882+0000 7f2cfc556700 1 -- start start 2019-12-05T05:09:38.891 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:38.882+0000 7f2cfc556700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2cf41236f0 0x7f2cf4130b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:09:38.891 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:38.882+0000 7f2cfc556700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2cf412cd00 0x7f2cf4117610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:09:38.891 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:38.882+0000 7f2cfc556700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2cf412d5c0 0x7f2cf41191b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:09:38.891 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:38.882+0000 7f2cfc556700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2cf41187d0 con 0x7f2cf412d5c0 2019-12-05T05:09:38.891 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:38.882+0000 7f2cfc556700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2cf4130360 con 0x7f2cf41236f0 2019-12-05T05:09:38.892 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:38.882+0000 7f2cfc556700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cf4130600 con 0x7f2cf412cd00 2019-12-05T05:09:38.892 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:38.882+0000 7f2cfaaf3700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2cf41236f0 0x7f2cf4130b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:09:38.892 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:38.882+0000 7f2cfb2f4700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2cf412d5c0 0x7f2cf41191b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:09:38.892 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:38.882+0000 7f2cfaaf3700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2cf41236f0 0x7f2cf4130b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:56826/0 (socket says 172.21.15.42:56826) 2019-12-05T05:09:38.892 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:38.882+0000 7f2cfa2f2700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2cf412cd00 0x7f2cf4117610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:09:38.892 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:38.882+0000 7f2cfaaf3700 1 -- 172.21.15.42:0/2757407696 learned_addr learned my addr 172.21.15.42:0/2757407696 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:09:38.892 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:38.882+0000 7f2cfaaf3700 1 -- 172.21.15.42:0/2757407696 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2cf412cd00 msgr2=0x7f2cf4117610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:09:38.893 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:38.882+0000 7f2cfaaf3700 1 --2- 172.21.15.42:0/2757407696 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2cf412cd00 0x7f2cf4117610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:09:38.893 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:38.882+0000 7f2cfaaf3700 1 -- 172.21.15.42:0/2757407696 >> v2:172.21.15.40:3300/0 conn(0x7f2cf412d5c0 msgr2=0x7f2cf41191b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:09:38.893 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:38.882+0000 7f2cfaaf3700 1 --2- 172.21.15.42:0/2757407696 >> v2:172.21.15.40:3300/0 conn(0x7f2cf412d5c0 0x7f2cf41191b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:09:38.893 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:38.886+0000 7f2cfaaf3700 1 -- 172.21.15.42:0/2757407696 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cf4131090 con 0x7f2cf41236f0 2019-12-05T05:09:38.893 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:38.886+0000 7f2cfb2f4700 1 --2- 172.21.15.42:0/2757407696 >> v2:172.21.15.40:3300/0 conn(0x7f2cf412d5c0 0x7f2cf41191b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:09:38.893 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:38.886+0000 7f2cfa2f2700 1 --2- 172.21.15.42:0/2757407696 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2cf412cd00 0x7f2cf4117610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:09:38.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:38.886+0000 7f2cfaaf3700 1 --2- 172.21.15.42:0/2757407696 >> v2:172.21.15.42:3300/0 conn(0x7f2cf41236f0 0x7f2cf4130b40 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f2cf4118440 tx=0x7f2cf0010a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:38.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:38.886+0000 7f2cfaaf3700 1 --2- 172.21.15.42:0/2757407696 >> v2:172.21.15.42:3300/0 conn(0x7f2cf41236f0 0x7f2cf4130b40 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f2cf4118440 tx=0x7f2cf0010a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:38.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:38.886+0000 7f2cfaaf3700 1 --2- 172.21.15.42:0/2757407696 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2cf41236f0 0x7f2cf4130b40 secure :-1 s=READY pgs=361 cs=0 l=1 rx=0x7f2cf4118440 tx=0x7f2cf0010a20).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:09:38.898 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:38.886+0000 7f2cfaaf3700 1 --2- 172.21.15.42:0/2757407696 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2cf41236f0 0x7f2cf4130b40 secure :-1 s=THROTTLE_DONE pgs=361 cs=0 l=1 rx=0x7f2cf4118440 tx=0x7f2cf0010a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:38.898 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:38.886+0000 7f2cfaaf3700 1 --2- 172.21.15.42:0/2757407696 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2cf41236f0 0x7f2cf4130b40 secure :-1 s=THROTTLE_DONE pgs=361 cs=0 l=1 rx=0x7f2cf4118440 tx=0x7f2cf0010a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:38.898 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:38.886+0000 7f2ce77fe700 1 -- 172.21.15.42:0/2757407696 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2cf000caa0 con 0x7f2cf41236f0 2019-12-05T05:09:38.898 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:38.886+0000 7f2cfaaf3700 1 --2- 172.21.15.42:0/2757407696 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2cf41236f0 0x7f2cf4130b40 secure :-1 s=THROTTLE_DONE pgs=361 cs=0 l=1 rx=0x7f2cf4118440 tx=0x7f2cf0010a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:38.898 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:38.886+0000 7f2ce77fe700 1 -- 172.21.15.42:0/2757407696 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cf000c6c0 con 0x7f2cf41236f0 2019-12-05T05:09:38.899 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:38.886+0000 7f2cfc556700 1 -- 172.21.15.42:0/2757407696 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cf4117b60 con 0x7f2cf41236f0 2019-12-05T05:09:38.899 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:38.886+0000 7f2ce77fe700 1 -- 172.21.15.42:0/2757407696 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2cf0012250 con 0x7f2cf41236f0 2019-12-05T05:09:38.899 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:38.886+0000 7f2cfc556700 1 -- 172.21.15.42:0/2757407696 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cf411c370 con 0x7f2cf41236f0 2019-12-05T05:09:38.899 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:38.886+0000 7f2cfaaf3700 1 --2- 172.21.15.42:0/2757407696 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2cf41236f0 0x7f2cf4130b40 secure :-1 s=THROTTLE_DONE pgs=361 cs=0 l=1 rx=0x7f2cf4118440 tx=0x7f2cf0010a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:38.899 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:38.886+0000 7f2ce77fe700 1 -- 172.21.15.42:0/2757407696 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f2cf0012560 con 0x7f2cf41236f0 2019-12-05T05:09:38.899 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:38.886+0000 7f2cfaaf3700 1 --2- 172.21.15.42:0/2757407696 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2cf41236f0 0x7f2cf4130b40 secure :-1 s=THROTTLE_DONE pgs=361 cs=0 l=1 rx=0x7f2cf4118440 tx=0x7f2cf0010a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:38.899 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:38.886+0000 7f2ce77fe700 1 --2- 172.21.15.42:0/2757407696 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2cdc03ae60 0x7f2cdc03d370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:09:38.900 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:38.886+0000 7f2ce77fe700 1 -- 172.21.15.42:0/2757407696 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6289+0+0 (secure 0 0 0) 0x7f2cf004b2c0 con 0x7f2cf41236f0 2019-12-05T05:09:38.900 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:38.886+0000 7f2cfa2f2700 1 --2- 172.21.15.42:0/2757407696 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2cdc03ae60 0x7f2cdc03d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:09:38.900 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:38.890+0000 7f2cfa2f2700 1 --2- 172.21.15.42:0/2757407696 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2cdc03ae60 0x7f2cdc03d370 crc :-1 s=READY pgs=163 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:09:38.900 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:38.890+0000 7f2cfc556700 1 -- 172.21.15.42:0/2757407696 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cf405d110 con 0x7f2cf41236f0 2019-12-05T05:09:38.901 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:38.894+0000 7f2cfaaf3700 1 --2- 172.21.15.42:0/2757407696 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2cf41236f0 0x7f2cf4130b40 secure :-1 s=THROTTLE_DONE pgs=361 cs=0 l=1 rx=0x7f2cf4118440 tx=0x7f2cf0010a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:38.901 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:38.894+0000 7f2ce77fe700 1 -- 172.21.15.42:0/2757407696 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f2cf0032340 con 0x7f2cf41236f0 2019-12-05T05:09:39.071 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:39.066+0000 7f2cfc556700 1 -- 172.21.15.42:0/2757407696 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2cf405b8c0 con 0x7f2cf41236f0 2019-12-05T05:09:39.072 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:39.070+0000 7f2cfaaf3700 1 --2- 172.21.15.42:0/2757407696 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2cf41236f0 0x7f2cf4130b40 secure :-1 s=THROTTLE_DONE pgs=361 cs=0 l=1 rx=0x7f2cf4118440 tx=0x7f2cf0010a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:39.072 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:39.070+0000 7f2ce77fe700 1 -- 172.21.15.42:0/2757407696 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) v1 ==== 94+0+1772 (secure 0 0 0) 0x7f2cf00325a0 con 0x7f2cf41236f0 2019-12-05T05:09:39.072 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 34 2019-12-05T05:09:39.072 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:09:39.072 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":34,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:09:28.296949+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4978},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4978":{"gid":4978,"name":"a","rank":0,"incarnation":30,"state":"up:active","state_seq":36,"addr":"172.21.15.42:6835/3650564018","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3650564018},{"type":"v1","addr":"172.21.15.42:6835","nonce":3650564018}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:09:39.074 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:39.070+0000 7f2cfc556700 1 -- 172.21.15.42:0/2757407696 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2cdc03ae60 msgr2=0x7f2cdc03d370 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:09:39.074 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:39.070+0000 7f2cfc556700 1 --2- 172.21.15.42:0/2757407696 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2cdc03ae60 0x7f2cdc03d370 crc :-1 s=READY pgs=163 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:09:39.075 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:39.070+0000 7f2cfc556700 1 -- 172.21.15.42:0/2757407696 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2cf41236f0 msgr2=0x7f2cf4130b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:09:39.075 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:39.070+0000 7f2cfc556700 1 --2- 172.21.15.42:0/2757407696 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2cf41236f0 0x7f2cf4130b40 secure :-1 s=READY pgs=361 cs=0 l=1 rx=0x7f2cf4118440 tx=0x7f2cf0010a20).stop 2019-12-05T05:09:39.075 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:39.070+0000 7f2cfc556700 1 -- 172.21.15.42:0/2757407696 shutdown_connections 2019-12-05T05:09:39.075 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:39.070+0000 7f2cfc556700 1 --2- 172.21.15.42:0/2757407696 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2cdc03ae60 0x7f2cdc03d370 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:09:39.075 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:39.070+0000 7f2cfc556700 1 --2- 172.21.15.42:0/2757407696 >> v2:172.21.15.40:3300/0 conn(0x7f2cf412d5c0 0x7f2cf41191b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:09:39.075 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:39.070+0000 7f2cfc556700 1 --2- 172.21.15.42:0/2757407696 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2cf41236f0 0x7f2cf4130b40 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:09:39.075 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:39.070+0000 7f2cfc556700 1 --2- 172.21.15.42:0/2757407696 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2cf412cd00 0x7f2cf4117610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:09:39.076 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:39.070+0000 7f2cfc556700 1 -- 172.21.15.42:0/2757407696 shutdown_connections 2019-12-05T05:09:39.076 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:39.070+0000 7f2cfc556700 1 -- 172.21.15.42:0/2757407696 wait complete. 2019-12-05T05:09:39.076 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:39.070+0000 7f2cfc556700 1 -- 172.21.15.42:0/2757407696 >> 172.21.15.42:0/2757407696 conn(0x7f2cf40aabe0 msgr2=0x7f2cf4059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:09:39.084 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2019-12-05T05:09:41.085 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:09:41.195 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:41.186+0000 7fca7e8d0700 1 Processor -- start 2019-12-05T05:09:41.195 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:41.186+0000 7fca7e8d0700 1 -- start start 2019-12-05T05:09:41.195 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:41.186+0000 7fca7e8d0700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fca7808df80 0x7fca7809f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:09:41.195 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:41.186+0000 7fca7e8d0700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca7809fc60 con 0x7fca780a7740 2019-12-05T05:09:41.195 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:41.186+0000 7fca7e8d0700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca780a0710 con 0x7fca7809f830 2019-12-05T05:09:41.196 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:41.186+0000 7fca7e8d0700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca780a0b00 con 0x7fca7808df80 2019-12-05T05:09:41.196 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:41.186+0000 7fca77fff700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fca7808df80 0x7fca7809f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:09:41.196 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:41.186+0000 7fca7d66e700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fca7809f830 0x7fca7814efe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:56886/0 (socket says 172.21.15.42:56886) 2019-12-05T05:09:41.196 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:41.186+0000 7fca7d66e700 1 -- 172.21.15.42:0/1451328926 learned_addr learned my addr 172.21.15.42:0/1451328926 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:09:41.196 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:41.186+0000 7fca777fe700 1 -- 172.21.15.42:0/1451328926 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3010748529 0 0) 0x7fca780a0710 con 0x7fca7809f830 2019-12-05T05:09:41.196 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:41.186+0000 7fca777fe700 1 -- 172.21.15.42:0/1451328926 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca640030b0 con 0x7fca7809f830 2019-12-05T05:09:41.196 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:41.190+0000 7fca77fff700 1 -- 172.21.15.42:0/1451328926 >> v1:172.21.15.42:6789/0 conn(0x7fca7809f830 legacy=0x7fca7814efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:09:41.197 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:41.190+0000 7fca77fff700 1 -- 172.21.15.42:0/1451328926 >> v1:172.21.15.40:6789/0 conn(0x7fca780a7740 legacy=0x7fca7807e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:09:41.197 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:41.190+0000 7fca77fff700 1 -- 172.21.15.42:0/1451328926 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca7807ef20 con 0x7fca7808df80 2019-12-05T05:09:41.197 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:41.190+0000 7fca77fff700 1 --2- 172.21.15.42:0/1451328926 >> v2:172.21.15.40:3300/0 conn(0x7fca7808df80 0x7fca7809f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fca60008790 tx=0x7fca6000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:41.197 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:41.190+0000 7fca77fff700 1 --2- 172.21.15.42:0/1451328926 >> v2:172.21.15.40:3300/0 conn(0x7fca7808df80 0x7fca7809f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fca60008790 tx=0x7fca6000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:41.197 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:41.190+0000 7fca77fff700 1 --2- 172.21.15.42:0/1451328926 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fca7808df80 0x7fca7809f2e0 secure :-1 s=READY pgs=285 cs=0 l=1 rx=0x7fca60008790 tx=0x7fca6000fbf0).ready entity=mon.0 client_cookie=cd0eeae1e884ee28 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:09:41.197 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:41.190+0000 7fca77fff700 1 --2- 172.21.15.42:0/1451328926 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fca7808df80 0x7fca7809f2e0 secure :-1 s=THROTTLE_DONE pgs=285 cs=0 l=1 rx=0x7fca60008790 tx=0x7fca6000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:41.197 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:41.190+0000 7fca777fe700 1 -- 172.21.15.42:0/1451328926 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fca60011270 con 0x7fca7808df80 2019-12-05T05:09:41.198 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:41.190+0000 7fca77fff700 1 --2- 172.21.15.42:0/1451328926 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fca7808df80 0x7fca7809f2e0 secure :-1 s=THROTTLE_DONE pgs=285 cs=0 l=1 rx=0x7fca60008790 tx=0x7fca6000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:41.198 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:41.190+0000 7fca77fff700 1 --2- 172.21.15.42:0/1451328926 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fca7808df80 0x7fca7809f2e0 secure :-1 s=THROTTLE_DONE pgs=285 cs=0 l=1 rx=0x7fca60008790 tx=0x7fca6000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:41.198 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:41.190+0000 7fca777fe700 1 -- 172.21.15.42:0/1451328926 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca60008bc0 con 0x7fca7808df80 2019-12-05T05:09:41.198 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:41.190+0000 7fca777fe700 1 -- 172.21.15.42:0/1451328926 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fca60011850 con 0x7fca7808df80 2019-12-05T05:09:41.198 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:41.190+0000 7fca7e8d0700 1 -- 172.21.15.42:0/1451328926 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fca7808df80 msgr2=0x7fca7809f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:09:41.198 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:41.190+0000 7fca7e8d0700 1 --2- 172.21.15.42:0/1451328926 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fca7808df80 0x7fca7809f2e0 secure :-1 s=READY pgs=285 cs=0 l=1 rx=0x7fca60008790 tx=0x7fca6000fbf0).stop 2019-12-05T05:09:41.198 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:41.190+0000 7fca7e8d0700 1 -- 172.21.15.42:0/1451328926 shutdown_connections 2019-12-05T05:09:41.199 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:41.190+0000 7fca7e8d0700 1 --2- 172.21.15.42:0/1451328926 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fca7808df80 0x7fca7809f2e0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:09:41.199 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:41.190+0000 7fca7e8d0700 1 -- 172.21.15.42:0/1451328926 shutdown_connections 2019-12-05T05:09:41.199 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:41.190+0000 7fca7e8d0700 1 -- 172.21.15.42:0/1451328926 wait complete. 2019-12-05T05:09:41.199 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:41.190+0000 7fca7e8d0700 1 -- 172.21.15.42:0/1451328926 >> 172.21.15.42:0/1451328926 conn(0x7fca780aabe0 msgr2=0x7fca780a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:09:41.199 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:41.190+0000 7fca7e8d0700 1 Processor -- start 2019-12-05T05:09:41.199 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:41.190+0000 7fca7e8d0700 1 -- start start 2019-12-05T05:09:41.202 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:41.194+0000 7fca7e8d0700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fca780a1510 0x7fca78120a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:09:41.202 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:41.194+0000 7fca7e8d0700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fca780a3130 0x7fca781426d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:09:41.202 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:41.194+0000 7fca7e8d0700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca78120fe0 con 0x7fca780a3560 2019-12-05T05:09:41.203 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:41.194+0000 7fca7e8d0700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca780a24f0 con 0x7fca780a3130 2019-12-05T05:09:41.203 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:41.194+0000 7fca7e8d0700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca781202b0 con 0x7fca780a1510 2019-12-05T05:09:41.203 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:41.194+0000 7fca7ce6d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fca780a1510 0x7fca78120a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:09:41.203 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:41.194+0000 7fca7ce6d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fca780a1510 0x7fca78120a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:56850/0 (socket says 172.21.15.42:56850) 2019-12-05T05:09:41.203 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:41.194+0000 7fca7ce6d700 1 -- 172.21.15.42:0/1614950466 learned_addr learned my addr 172.21.15.42:0/1614950466 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:09:41.203 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:41.194+0000 7fca77fff700 1 --2- 172.21.15.42:0/1614950466 >> v2:172.21.15.40:3300/0 conn(0x7fca780a3130 0x7fca781426d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:09:41.203 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:41.194+0000 7fca7ce6d700 1 -- 172.21.15.42:0/1614950466 >> v2:172.21.15.40:3300/0 conn(0x7fca780a3130 msgr2=0x7fca781426d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:09:41.204 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:41.194+0000 7fca7ce6d700 1 --2- 172.21.15.42:0/1614950466 >> v2:172.21.15.40:3300/0 conn(0x7fca780a3130 0x7fca781426d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:09:41.204 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:41.194+0000 7fca7ce6d700 1 -- 172.21.15.42:0/1614950466 >> v1:172.21.15.42:6789/0 conn(0x7fca780a3560 legacy=0x7fca780a1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:09:41.204 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:41.194+0000 7fca7ce6d700 1 -- 172.21.15.42:0/1614950466 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca78142c20 con 0x7fca780a1510 2019-12-05T05:09:41.204 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:41.194+0000 7fca7ce6d700 1 --2- 172.21.15.42:0/1614950466 >> v2:172.21.15.42:3300/0 conn(0x7fca780a1510 0x7fca78120a90 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fca68001210 tx=0x7fca68010d20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:41.204 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:41.194+0000 7fca7ce6d700 1 --2- 172.21.15.42:0/1614950466 >> v2:172.21.15.42:3300/0 conn(0x7fca780a1510 0x7fca78120a90 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fca68001210 tx=0x7fca68010d20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:41.204 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:41.194+0000 7fca7ce6d700 1 --2- 172.21.15.42:0/1614950466 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fca780a1510 0x7fca78120a90 secure :-1 s=READY pgs=364 cs=0 l=1 rx=0x7fca68001210 tx=0x7fca68010d20).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:09:41.204 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:41.194+0000 7fca7ce6d700 1 --2- 172.21.15.42:0/1614950466 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fca780a1510 0x7fca78120a90 secure :-1 s=THROTTLE_DONE pgs=364 cs=0 l=1 rx=0x7fca68001210 tx=0x7fca68010d20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:41.205 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:41.194+0000 7fca757fa700 1 -- 172.21.15.42:0/1614950466 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fca6800ca40 con 0x7fca780a1510 2019-12-05T05:09:41.205 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:41.194+0000 7fca7ce6d700 1 --2- 172.21.15.42:0/1614950466 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fca780a1510 0x7fca78120a90 secure :-1 s=THROTTLE_DONE pgs=364 cs=0 l=1 rx=0x7fca68001210 tx=0x7fca68010d20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:41.205 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:41.194+0000 7fca7ce6d700 1 --2- 172.21.15.42:0/1614950466 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fca780a1510 0x7fca78120a90 secure :-1 s=THROTTLE_DONE pgs=364 cs=0 l=1 rx=0x7fca68001210 tx=0x7fca68010d20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:41.205 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:41.194+0000 7fca757fa700 1 -- 172.21.15.42:0/1614950466 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca680101c0 con 0x7fca780a1510 2019-12-05T05:09:41.205 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:41.194+0000 7fca7e8d0700 1 -- 172.21.15.42:0/1614950466 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca7812ab40 con 0x7fca780a1510 2019-12-05T05:09:41.205 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:41.194+0000 7fca757fa700 1 -- 172.21.15.42:0/1614950466 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fca68012590 con 0x7fca780a1510 2019-12-05T05:09:41.205 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:41.194+0000 7fca7e8d0700 1 -- 172.21.15.42:0/1614950466 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca7812b0f0 con 0x7fca780a1510 2019-12-05T05:09:41.206 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:41.198+0000 7fca7ce6d700 1 --2- 172.21.15.42:0/1614950466 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fca780a1510 0x7fca78120a90 secure :-1 s=THROTTLE_DONE pgs=364 cs=0 l=1 rx=0x7fca68001210 tx=0x7fca68010d20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:41.206 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:41.198+0000 7fca7ce6d700 1 --2- 172.21.15.42:0/1614950466 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fca780a1510 0x7fca78120a90 secure :-1 s=THROTTLE_DONE pgs=364 cs=0 l=1 rx=0x7fca68001210 tx=0x7fca68010d20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:41.206 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:41.198+0000 7fca757fa700 1 -- 172.21.15.42:0/1614950466 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fca68012870 con 0x7fca780a1510 2019-12-05T05:09:41.206 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:41.198+0000 7fca757fa700 1 --2- 172.21.15.42:0/1614950466 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fca5c03ada0 0x7fca5c03d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:09:41.206 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:41.198+0000 7fca757fa700 1 -- 172.21.15.42:0/1614950466 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6289+0+0 (secure 0 0 0) 0x7fca6804b5d0 con 0x7fca780a1510 2019-12-05T05:09:41.206 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:41.198+0000 7fca77fff700 1 --2- 172.21.15.42:0/1614950466 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fca5c03ada0 0x7fca5c03d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:09:41.206 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:41.198+0000 7fca77fff700 1 --2- 172.21.15.42:0/1614950466 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fca5c03ada0 0x7fca5c03d2b0 crc :-1 s=READY pgs=164 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:09:41.207 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:41.202+0000 7fca7e8d0700 1 -- 172.21.15.42:0/1614950466 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca7805d110 con 0x7fca780a1510 2019-12-05T05:09:41.213 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:41.210+0000 7fca7ce6d700 1 --2- 172.21.15.42:0/1614950466 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fca780a1510 0x7fca78120a90 secure :-1 s=THROTTLE_DONE pgs=364 cs=0 l=1 rx=0x7fca68001210 tx=0x7fca68010d20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:41.213 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:41.210+0000 7fca757fa700 1 -- 172.21.15.42:0/1614950466 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fca68032590 con 0x7fca780a1510 2019-12-05T05:09:41.393 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:41.390+0000 7fca7e8d0700 1 -- 172.21.15.42:0/1614950466 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fca7805b8c0 con 0x7fca780a1510 2019-12-05T05:09:41.393 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:41.390+0000 7fca7ce6d700 1 --2- 172.21.15.42:0/1614950466 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fca780a1510 0x7fca78120a90 secure :-1 s=THROTTLE_DONE pgs=364 cs=0 l=1 rx=0x7fca68001210 tx=0x7fca68010d20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:41.393 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:41.390+0000 7fca757fa700 1 -- 172.21.15.42:0/1614950466 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) v1 ==== 94+0+1772 (secure 0 0 0) 0x7fca680327f0 con 0x7fca780a1510 2019-12-05T05:09:41.394 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 34 2019-12-05T05:09:41.394 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:09:41.394 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":34,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:09:28.296949+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4978},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4978":{"gid":4978,"name":"a","rank":0,"incarnation":30,"state":"up:active","state_seq":36,"addr":"172.21.15.42:6835/3650564018","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3650564018},{"type":"v1","addr":"172.21.15.42:6835","nonce":3650564018}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:09:41.394 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:41.390+0000 7fca7e8d0700 1 -- 172.21.15.42:0/1614950466 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fca5c03ada0 msgr2=0x7fca5c03d2b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:09:41.395 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:41.390+0000 7fca7e8d0700 1 --2- 172.21.15.42:0/1614950466 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fca5c03ada0 0x7fca5c03d2b0 crc :-1 s=READY pgs=164 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:09:41.395 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:41.390+0000 7fca7e8d0700 1 -- 172.21.15.42:0/1614950466 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fca780a1510 msgr2=0x7fca78120a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:09:41.395 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:41.390+0000 7fca7e8d0700 1 --2- 172.21.15.42:0/1614950466 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fca780a1510 0x7fca78120a90 secure :-1 s=READY pgs=364 cs=0 l=1 rx=0x7fca68001210 tx=0x7fca68010d20).stop 2019-12-05T05:09:41.396 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:41.390+0000 7fca7e8d0700 1 -- 172.21.15.42:0/1614950466 shutdown_connections 2019-12-05T05:09:41.396 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:41.390+0000 7fca7e8d0700 1 --2- 172.21.15.42:0/1614950466 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fca5c03ada0 0x7fca5c03d2b0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:09:41.396 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:41.390+0000 7fca7e8d0700 1 --2- 172.21.15.42:0/1614950466 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fca780a1510 0x7fca78120a90 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:09:41.396 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:41.390+0000 7fca7e8d0700 1 --2- 172.21.15.42:0/1614950466 >> v2:172.21.15.40:3300/0 conn(0x7fca780a3130 0x7fca781426d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:09:41.396 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:41.390+0000 7fca7e8d0700 1 -- 172.21.15.42:0/1614950466 shutdown_connections 2019-12-05T05:09:41.396 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:41.394+0000 7fca7e8d0700 1 -- 172.21.15.42:0/1614950466 wait complete. 2019-12-05T05:09:41.397 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:41.394+0000 7fca7e8d0700 1 -- 172.21.15.42:0/1614950466 >> 172.21.15.42:0/1614950466 conn(0x7fca780a7f90 msgr2=0x7fca780a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:09:41.405 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2019-12-05T05:09:42.178 INFO:tasks.workunit.client.0.smithi042.stdout:removed './266/35' 2019-12-05T05:09:42.178 INFO:tasks.workunit.client.0.smithi042.stdout:removed './266/26' 2019-12-05T05:09:42.178 INFO:tasks.workunit.client.0.smithi042.stdout:removed './266/37' 2019-12-05T05:09:42.178 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './266' 2019-12-05T05:09:42.178 INFO:tasks.workunit.client.0.smithi042.stdout:removed './267/21' 2019-12-05T05:09:42.179 INFO:tasks.workunit.client.0.smithi042.stdout:removed './267/27' 2019-12-05T05:09:42.179 INFO:tasks.workunit.client.0.smithi042.stdout:removed './267/31' 2019-12-05T05:09:42.179 INFO:tasks.workunit.client.0.smithi042.stdout:removed './267/36' 2019-12-05T05:09:42.179 INFO:tasks.workunit.client.0.smithi042.stdout:removed './267/20' 2019-12-05T05:09:42.179 INFO:tasks.workunit.client.0.smithi042.stdout:removed './267/39' 2019-12-05T05:09:42.179 INFO:tasks.workunit.client.0.smithi042.stdout:removed './267/34' 2019-12-05T05:09:42.179 INFO:tasks.workunit.client.0.smithi042.stdout:removed './267/38' 2019-12-05T05:09:42.180 INFO:tasks.workunit.client.0.smithi042.stdout:removed './267/28' 2019-12-05T05:09:42.180 INFO:tasks.workunit.client.0.smithi042.stdout:removed './267/24' 2019-12-05T05:09:42.180 INFO:tasks.workunit.client.0.smithi042.stdout:removed './267/29' 2019-12-05T05:09:42.180 INFO:tasks.workunit.client.0.smithi042.stdout:removed './267/32' 2019-12-05T05:09:42.180 INFO:tasks.workunit.client.0.smithi042.stdout:removed './267/22' 2019-12-05T05:09:42.180 INFO:tasks.workunit.client.0.smithi042.stdout:removed './267/25' 2019-12-05T05:09:42.180 INFO:tasks.workunit.client.0.smithi042.stdout:removed './267/30' 2019-12-05T05:09:42.181 INFO:tasks.workunit.client.0.smithi042.stdout:removed './267/33' 2019-12-05T05:09:42.181 INFO:tasks.workunit.client.0.smithi042.stdout:removed './267/23' 2019-12-05T05:09:42.181 INFO:tasks.workunit.client.0.smithi042.stdout:removed './267/35' 2019-12-05T05:09:42.181 INFO:tasks.workunit.client.0.smithi042.stdout:removed './267/26' 2019-12-05T05:09:42.181 INFO:tasks.workunit.client.0.smithi042.stdout:removed './267/37' 2019-12-05T05:09:42.181 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './267' 2019-12-05T05:09:42.181 INFO:tasks.workunit.client.0.smithi042.stdout:removed './268/21' 2019-12-05T05:09:42.182 INFO:tasks.workunit.client.0.smithi042.stdout:removed './268/27' 2019-12-05T05:09:42.182 INFO:tasks.workunit.client.0.smithi042.stdout:removed './268/31' 2019-12-05T05:09:42.183 INFO:tasks.workunit.client.0.smithi042.stdout:removed './268/36' 2019-12-05T05:09:42.183 INFO:tasks.workunit.client.0.smithi042.stdout:removed './268/20' 2019-12-05T05:09:42.183 INFO:tasks.workunit.client.0.smithi042.stdout:removed './268/39' 2019-12-05T05:09:42.183 INFO:tasks.workunit.client.0.smithi042.stdout:removed './268/34' 2019-12-05T05:09:42.183 INFO:tasks.workunit.client.0.smithi042.stdout:removed './268/38' 2019-12-05T05:09:42.183 INFO:tasks.workunit.client.0.smithi042.stdout:removed './268/28' 2019-12-05T05:09:42.184 INFO:tasks.workunit.client.0.smithi042.stdout:removed './268/24' 2019-12-05T05:09:42.184 INFO:tasks.workunit.client.0.smithi042.stdout:removed './268/29' 2019-12-05T05:09:42.184 INFO:tasks.workunit.client.0.smithi042.stdout:removed './268/32' 2019-12-05T05:09:42.184 INFO:tasks.workunit.client.0.smithi042.stdout:removed './268/22' 2019-12-05T05:09:42.184 INFO:tasks.workunit.client.0.smithi042.stdout:removed './268/25' 2019-12-05T05:09:42.184 INFO:tasks.workunit.client.0.smithi042.stdout:removed './268/30' 2019-12-05T05:09:42.184 INFO:tasks.workunit.client.0.smithi042.stdout:removed './268/33' 2019-12-05T05:09:42.185 INFO:tasks.workunit.client.0.smithi042.stdout:removed './268/23' 2019-12-05T05:09:42.185 INFO:tasks.workunit.client.0.smithi042.stdout:removed './268/35' 2019-12-05T05:09:42.185 INFO:tasks.workunit.client.0.smithi042.stdout:removed './268/26' 2019-12-05T05:09:42.185 INFO:tasks.workunit.client.0.smithi042.stdout:removed './268/37' 2019-12-05T05:09:42.185 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './268' 2019-12-05T05:09:42.185 INFO:tasks.workunit.client.0.smithi042.stdout:removed './269/21' 2019-12-05T05:09:42.185 INFO:tasks.workunit.client.0.smithi042.stdout:removed './269/27' 2019-12-05T05:09:42.186 INFO:tasks.workunit.client.0.smithi042.stdout:removed './269/31' 2019-12-05T05:09:42.186 INFO:tasks.workunit.client.0.smithi042.stdout:removed './269/36' 2019-12-05T05:09:42.186 INFO:tasks.workunit.client.0.smithi042.stdout:removed './269/20' 2019-12-05T05:09:42.186 INFO:tasks.workunit.client.0.smithi042.stdout:removed './269/39' 2019-12-05T05:09:42.186 INFO:tasks.workunit.client.0.smithi042.stdout:removed './269/34' 2019-12-05T05:09:42.187 INFO:tasks.workunit.client.0.smithi042.stdout:removed './269/38' 2019-12-05T05:09:42.187 INFO:tasks.workunit.client.0.smithi042.stdout:removed './269/28' 2019-12-05T05:09:42.187 INFO:tasks.workunit.client.0.smithi042.stdout:removed './269/24' 2019-12-05T05:09:42.187 INFO:tasks.workunit.client.0.smithi042.stdout:removed './269/29' 2019-12-05T05:09:42.187 INFO:tasks.workunit.client.0.smithi042.stdout:removed './269/32' 2019-12-05T05:09:42.187 INFO:tasks.workunit.client.0.smithi042.stdout:removed './269/22' 2019-12-05T05:09:42.187 INFO:tasks.workunit.client.0.smithi042.stdout:removed './269/25' 2019-12-05T05:09:42.188 INFO:tasks.workunit.client.0.smithi042.stdout:removed './269/30' 2019-12-05T05:09:42.188 INFO:tasks.workunit.client.0.smithi042.stdout:removed './269/33' 2019-12-05T05:09:42.188 INFO:tasks.workunit.client.0.smithi042.stdout:removed './269/23' 2019-12-05T05:09:42.188 INFO:tasks.workunit.client.0.smithi042.stdout:removed './269/35' 2019-12-05T05:09:42.188 INFO:tasks.workunit.client.0.smithi042.stdout:removed './269/26' 2019-12-05T05:09:42.188 INFO:tasks.workunit.client.0.smithi042.stdout:removed './269/37' 2019-12-05T05:09:42.188 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './269' 2019-12-05T05:09:42.189 INFO:tasks.workunit.client.0.smithi042.stdout:removed './270/21' 2019-12-05T05:09:42.189 INFO:tasks.workunit.client.0.smithi042.stdout:removed './270/27' 2019-12-05T05:09:42.189 INFO:tasks.workunit.client.0.smithi042.stdout:removed './270/31' 2019-12-05T05:09:42.189 INFO:tasks.workunit.client.0.smithi042.stdout:removed './270/36' 2019-12-05T05:09:42.189 INFO:tasks.workunit.client.0.smithi042.stdout:removed './270/20' 2019-12-05T05:09:42.189 INFO:tasks.workunit.client.0.smithi042.stdout:removed './270/39' 2019-12-05T05:09:42.189 INFO:tasks.workunit.client.0.smithi042.stdout:removed './270/34' 2019-12-05T05:09:42.189 INFO:tasks.workunit.client.0.smithi042.stdout:removed './270/38' 2019-12-05T05:09:42.190 INFO:tasks.workunit.client.0.smithi042.stdout:removed './270/28' 2019-12-05T05:09:42.190 INFO:tasks.workunit.client.0.smithi042.stdout:removed './270/24' 2019-12-05T05:09:42.190 INFO:tasks.workunit.client.0.smithi042.stdout:removed './270/29' 2019-12-05T05:09:42.190 INFO:tasks.workunit.client.0.smithi042.stdout:removed './270/32' 2019-12-05T05:09:42.190 INFO:tasks.workunit.client.0.smithi042.stdout:removed './270/22' 2019-12-05T05:09:42.190 INFO:tasks.workunit.client.0.smithi042.stdout:removed './270/25' 2019-12-05T05:09:42.190 INFO:tasks.workunit.client.0.smithi042.stdout:removed './270/30' 2019-12-05T05:09:42.191 INFO:tasks.workunit.client.0.smithi042.stdout:removed './270/33' 2019-12-05T05:09:42.191 INFO:tasks.workunit.client.0.smithi042.stdout:removed './270/23' 2019-12-05T05:09:42.191 INFO:tasks.workunit.client.0.smithi042.stdout:removed './270/35' 2019-12-05T05:09:42.191 INFO:tasks.workunit.client.0.smithi042.stdout:removed './270/26' 2019-12-05T05:09:42.191 INFO:tasks.workunit.client.0.smithi042.stdout:removed './270/37' 2019-12-05T05:09:42.191 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './270' 2019-12-05T05:09:42.191 INFO:tasks.workunit.client.0.smithi042.stdout:removed './271/21' 2019-12-05T05:09:42.192 INFO:tasks.workunit.client.0.smithi042.stdout:removed './271/27' 2019-12-05T05:09:42.193 INFO:tasks.workunit.client.0.smithi042.stdout:removed './271/31' 2019-12-05T05:09:42.193 INFO:tasks.workunit.client.0.smithi042.stdout:removed './271/36' 2019-12-05T05:09:42.205 INFO:tasks.workunit.client.0.smithi042.stdout:removed './271/20' 2019-12-05T05:09:42.206 INFO:tasks.workunit.client.0.smithi042.stdout:removed './271/39' 2019-12-05T05:09:42.206 INFO:tasks.workunit.client.0.smithi042.stdout:removed './271/34' 2019-12-05T05:09:42.206 INFO:tasks.workunit.client.0.smithi042.stdout:removed './271/38' 2019-12-05T05:09:42.206 INFO:tasks.workunit.client.0.smithi042.stdout:removed './271/28' 2019-12-05T05:09:42.206 INFO:tasks.workunit.client.0.smithi042.stdout:removed './271/24' 2019-12-05T05:09:42.206 INFO:tasks.workunit.client.0.smithi042.stdout:removed './271/29' 2019-12-05T05:09:42.206 INFO:tasks.workunit.client.0.smithi042.stdout:removed './271/32' 2019-12-05T05:09:42.207 INFO:tasks.workunit.client.0.smithi042.stdout:removed './271/22' 2019-12-05T05:09:42.207 INFO:tasks.workunit.client.0.smithi042.stdout:removed './271/25' 2019-12-05T05:09:42.207 INFO:tasks.workunit.client.0.smithi042.stdout:removed './271/30' 2019-12-05T05:09:42.207 INFO:tasks.workunit.client.0.smithi042.stdout:removed './271/33' 2019-12-05T05:09:42.207 INFO:tasks.workunit.client.0.smithi042.stdout:removed './271/23' 2019-12-05T05:09:42.207 INFO:tasks.workunit.client.0.smithi042.stdout:removed './271/35' 2019-12-05T05:09:42.207 INFO:tasks.workunit.client.0.smithi042.stdout:removed './271/26' 2019-12-05T05:09:42.207 INFO:tasks.workunit.client.0.smithi042.stdout:removed './271/37' 2019-12-05T05:09:42.208 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './271' 2019-12-05T05:09:42.208 INFO:tasks.workunit.client.0.smithi042.stdout:removed './272/21' 2019-12-05T05:09:42.208 INFO:tasks.workunit.client.0.smithi042.stdout:removed './272/27' 2019-12-05T05:09:42.208 INFO:tasks.workunit.client.0.smithi042.stdout:removed './272/31' 2019-12-05T05:09:42.208 INFO:tasks.workunit.client.0.smithi042.stdout:removed './272/36' 2019-12-05T05:09:42.208 INFO:tasks.workunit.client.0.smithi042.stdout:removed './272/20' 2019-12-05T05:09:42.208 INFO:tasks.workunit.client.0.smithi042.stdout:removed './272/39' 2019-12-05T05:09:42.209 INFO:tasks.workunit.client.0.smithi042.stdout:removed './272/34' 2019-12-05T05:09:42.209 INFO:tasks.workunit.client.0.smithi042.stdout:removed './272/38' 2019-12-05T05:09:42.209 INFO:tasks.workunit.client.0.smithi042.stdout:removed './272/28' 2019-12-05T05:09:42.209 INFO:tasks.workunit.client.0.smithi042.stdout:removed './272/24' 2019-12-05T05:09:42.209 INFO:tasks.workunit.client.0.smithi042.stdout:removed './272/29' 2019-12-05T05:09:42.209 INFO:tasks.workunit.client.0.smithi042.stdout:removed './272/32' 2019-12-05T05:09:42.209 INFO:tasks.workunit.client.0.smithi042.stdout:removed './272/22' 2019-12-05T05:09:42.210 INFO:tasks.workunit.client.0.smithi042.stdout:removed './272/25' 2019-12-05T05:09:42.210 INFO:tasks.workunit.client.0.smithi042.stdout:removed './272/30' 2019-12-05T05:09:42.210 INFO:tasks.workunit.client.0.smithi042.stdout:removed './272/33' 2019-12-05T05:09:42.210 INFO:tasks.workunit.client.0.smithi042.stdout:removed './272/23' 2019-12-05T05:09:42.210 INFO:tasks.workunit.client.0.smithi042.stdout:removed './272/35' 2019-12-05T05:09:42.210 INFO:tasks.workunit.client.0.smithi042.stdout:removed './272/26' 2019-12-05T05:09:42.210 INFO:tasks.workunit.client.0.smithi042.stdout:removed './272/37' 2019-12-05T05:09:42.211 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './272' 2019-12-05T05:09:42.211 INFO:tasks.workunit.client.0.smithi042.stdout:removed './273/21' 2019-12-05T05:09:42.211 INFO:tasks.workunit.client.0.smithi042.stdout:removed './273/27' 2019-12-05T05:09:42.211 INFO:tasks.workunit.client.0.smithi042.stdout:removed './273/31' 2019-12-05T05:09:42.211 INFO:tasks.workunit.client.0.smithi042.stdout:removed './273/36' 2019-12-05T05:09:42.211 INFO:tasks.workunit.client.0.smithi042.stdout:removed './273/20' 2019-12-05T05:09:42.211 INFO:tasks.workunit.client.0.smithi042.stdout:removed './273/39' 2019-12-05T05:09:42.212 INFO:tasks.workunit.client.0.smithi042.stdout:removed './273/34' 2019-12-05T05:09:42.212 INFO:tasks.workunit.client.0.smithi042.stdout:removed './273/38' 2019-12-05T05:09:42.212 INFO:tasks.workunit.client.0.smithi042.stdout:removed './273/28' 2019-12-05T05:09:42.221 INFO:tasks.workunit.client.0.smithi042.stdout:removed './273/24' 2019-12-05T05:09:42.221 INFO:tasks.workunit.client.0.smithi042.stdout:removed './273/29' 2019-12-05T05:09:42.221 INFO:tasks.workunit.client.0.smithi042.stdout:removed './273/32' 2019-12-05T05:09:42.221 INFO:tasks.workunit.client.0.smithi042.stdout:removed './273/22' 2019-12-05T05:09:42.221 INFO:tasks.workunit.client.0.smithi042.stdout:removed './273/25' 2019-12-05T05:09:42.221 INFO:tasks.workunit.client.0.smithi042.stdout:removed './273/30' 2019-12-05T05:09:42.222 INFO:tasks.workunit.client.0.smithi042.stdout:removed './273/33' 2019-12-05T05:09:42.222 INFO:tasks.workunit.client.0.smithi042.stdout:removed './273/23' 2019-12-05T05:09:42.222 INFO:tasks.workunit.client.0.smithi042.stdout:removed './273/35' 2019-12-05T05:09:42.222 INFO:tasks.workunit.client.0.smithi042.stdout:removed './273/26' 2019-12-05T05:09:42.222 INFO:tasks.workunit.client.0.smithi042.stdout:removed './273/37' 2019-12-05T05:09:42.222 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './273' 2019-12-05T05:09:42.222 INFO:tasks.workunit.client.0.smithi042.stdout:removed './274/21' 2019-12-05T05:09:42.223 INFO:tasks.workunit.client.0.smithi042.stdout:removed './274/27' 2019-12-05T05:09:42.223 INFO:tasks.workunit.client.0.smithi042.stdout:removed './274/31' 2019-12-05T05:09:42.223 INFO:tasks.workunit.client.0.smithi042.stdout:removed './274/36' 2019-12-05T05:09:42.223 INFO:tasks.workunit.client.0.smithi042.stdout:removed './274/20' 2019-12-05T05:09:42.223 INFO:tasks.workunit.client.0.smithi042.stdout:removed './274/39' 2019-12-05T05:09:42.223 INFO:tasks.workunit.client.0.smithi042.stdout:removed './274/34' 2019-12-05T05:09:42.223 INFO:tasks.workunit.client.0.smithi042.stdout:removed './274/38' 2019-12-05T05:09:42.224 INFO:tasks.workunit.client.0.smithi042.stdout:removed './274/28' 2019-12-05T05:09:42.224 INFO:tasks.workunit.client.0.smithi042.stdout:removed './274/24' 2019-12-05T05:09:42.224 INFO:tasks.workunit.client.0.smithi042.stdout:removed './274/29' 2019-12-05T05:09:42.224 INFO:tasks.workunit.client.0.smithi042.stdout:removed './274/32' 2019-12-05T05:09:42.224 INFO:tasks.workunit.client.0.smithi042.stdout:removed './274/22' 2019-12-05T05:09:42.239 INFO:tasks.workunit.client.0.smithi042.stdout:removed './274/25' 2019-12-05T05:09:42.240 INFO:tasks.workunit.client.0.smithi042.stdout:removed './274/30' 2019-12-05T05:09:42.240 INFO:tasks.workunit.client.0.smithi042.stdout:removed './274/33' 2019-12-05T05:09:42.240 INFO:tasks.workunit.client.0.smithi042.stdout:removed './274/23' 2019-12-05T05:09:42.240 INFO:tasks.workunit.client.0.smithi042.stdout:removed './274/35' 2019-12-05T05:09:42.241 INFO:tasks.workunit.client.0.smithi042.stdout:removed './274/26' 2019-12-05T05:09:42.241 INFO:tasks.workunit.client.0.smithi042.stdout:removed './274/37' 2019-12-05T05:09:42.242 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './274' 2019-12-05T05:09:42.242 INFO:tasks.workunit.client.0.smithi042.stdout:removed './275/21' 2019-12-05T05:09:42.242 INFO:tasks.workunit.client.0.smithi042.stdout:removed './275/27' 2019-12-05T05:09:42.243 INFO:tasks.workunit.client.0.smithi042.stdout:removed './275/31' 2019-12-05T05:09:42.243 INFO:tasks.workunit.client.0.smithi042.stdout:removed './275/36' 2019-12-05T05:09:42.243 INFO:tasks.workunit.client.0.smithi042.stdout:removed './275/20' 2019-12-05T05:09:42.243 INFO:tasks.workunit.client.0.smithi042.stdout:removed './275/39' 2019-12-05T05:09:42.243 INFO:tasks.workunit.client.0.smithi042.stdout:removed './275/34' 2019-12-05T05:09:42.243 INFO:tasks.workunit.client.0.smithi042.stdout:removed './275/38' 2019-12-05T05:09:42.243 INFO:tasks.workunit.client.0.smithi042.stdout:removed './275/28' 2019-12-05T05:09:42.244 INFO:tasks.workunit.client.0.smithi042.stdout:removed './275/24' 2019-12-05T05:09:42.244 INFO:tasks.workunit.client.0.smithi042.stdout:removed './275/29' 2019-12-05T05:09:42.244 INFO:tasks.workunit.client.0.smithi042.stdout:removed './275/32' 2019-12-05T05:09:42.244 INFO:tasks.workunit.client.0.smithi042.stdout:removed './275/22' 2019-12-05T05:09:42.244 INFO:tasks.workunit.client.0.smithi042.stdout:removed './275/25' 2019-12-05T05:09:42.244 INFO:tasks.workunit.client.0.smithi042.stdout:removed './275/30' 2019-12-05T05:09:42.244 INFO:tasks.workunit.client.0.smithi042.stdout:removed './275/33' 2019-12-05T05:09:42.245 INFO:tasks.workunit.client.0.smithi042.stdout:removed './275/23' 2019-12-05T05:09:42.245 INFO:tasks.workunit.client.0.smithi042.stdout:removed './275/35' 2019-12-05T05:09:42.245 INFO:tasks.workunit.client.0.smithi042.stdout:removed './275/26' 2019-12-05T05:09:42.245 INFO:tasks.workunit.client.0.smithi042.stdout:removed './275/37' 2019-12-05T05:09:42.245 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './275' 2019-12-05T05:09:42.245 INFO:tasks.workunit.client.0.smithi042.stdout:removed './276/21' 2019-12-05T05:09:42.246 INFO:tasks.workunit.client.0.smithi042.stdout:removed './276/27' 2019-12-05T05:09:42.246 INFO:tasks.workunit.client.0.smithi042.stdout:removed './276/31' 2019-12-05T05:09:42.246 INFO:tasks.workunit.client.0.smithi042.stdout:removed './276/36' 2019-12-05T05:09:42.246 INFO:tasks.workunit.client.0.smithi042.stdout:removed './276/20' 2019-12-05T05:09:42.246 INFO:tasks.workunit.client.0.smithi042.stdout:removed './276/39' 2019-12-05T05:09:42.246 INFO:tasks.workunit.client.0.smithi042.stdout:removed './276/34' 2019-12-05T05:09:42.247 INFO:tasks.workunit.client.0.smithi042.stdout:removed './276/38' 2019-12-05T05:09:42.247 INFO:tasks.workunit.client.0.smithi042.stdout:removed './276/28' 2019-12-05T05:09:42.247 INFO:tasks.workunit.client.0.smithi042.stdout:removed './276/24' 2019-12-05T05:09:42.247 INFO:tasks.workunit.client.0.smithi042.stdout:removed './276/29' 2019-12-05T05:09:42.247 INFO:tasks.workunit.client.0.smithi042.stdout:removed './276/32' 2019-12-05T05:09:42.247 INFO:tasks.workunit.client.0.smithi042.stdout:removed './276/22' 2019-12-05T05:09:42.247 INFO:tasks.workunit.client.0.smithi042.stdout:removed './276/25' 2019-12-05T05:09:42.248 INFO:tasks.workunit.client.0.smithi042.stdout:removed './276/30' 2019-12-05T05:09:42.248 INFO:tasks.workunit.client.0.smithi042.stdout:removed './276/33' 2019-12-05T05:09:42.248 INFO:tasks.workunit.client.0.smithi042.stdout:removed './276/23' 2019-12-05T05:09:42.248 INFO:tasks.workunit.client.0.smithi042.stdout:removed './276/35' 2019-12-05T05:09:42.248 INFO:tasks.workunit.client.0.smithi042.stdout:removed './276/26' 2019-12-05T05:09:43.405 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:09:43.526 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:43.518+0000 7f887de11700 1 Processor -- start 2019-12-05T05:09:43.526 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:43.518+0000 7f887de11700 1 -- start start 2019-12-05T05:09:43.526 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:43.518+0000 7f887de11700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f88780a7730 0x7f887808df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:09:43.527 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:43.518+0000 7f887de11700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f887809fad0 0x7f887807eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:09:43.527 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:43.518+0000 7f887de11700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88780a0700 con 0x7f887807e710 2019-12-05T05:09:43.527 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:43.518+0000 7f887de11700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88780a0af0 con 0x7f887809fad0 2019-12-05T05:09:43.527 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:43.518+0000 7f887de11700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f887808e5a0 con 0x7f88780a7730 2019-12-05T05:09:43.527 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:43.518+0000 7f887cbaf700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f887809fad0 0x7f887807eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:09:43.527 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:43.518+0000 7f887cbaf700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f887809fad0 0x7f887807eb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:56866/0 (socket says 172.21.15.42:56866) 2019-12-05T05:09:43.527 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:43.518+0000 7f887cbaf700 1 -- 172.21.15.42:0/1461709891 learned_addr learned my addr 172.21.15.42:0/1461709891 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:09:43.528 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:43.518+0000 7f8877fff700 1 --2- 172.21.15.42:0/1461709891 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f88780a7730 0x7f887808df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:09:43.528 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:43.522+0000 7f88777fe700 1 -- 172.21.15.42:0/1461709891 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1040422873 0 0) 0x7f88780a0700 con 0x7f887807e710 2019-12-05T05:09:43.528 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:43.522+0000 7f8877fff700 1 -- 172.21.15.42:0/1461709891 >> v2:172.21.15.42:3300/0 conn(0x7f887809fad0 msgr2=0x7f887807eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:09:43.528 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:43.522+0000 7f8877fff700 1 --2- 172.21.15.42:0/1461709891 >> v2:172.21.15.42:3300/0 conn(0x7f887809fad0 0x7f887807eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:09:43.528 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:43.522+0000 7f8877fff700 1 -- 172.21.15.42:0/1461709891 >> v1:172.21.15.40:6789/0 conn(0x7f887807e710 legacy=0x7f887809f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:09:43.528 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:43.522+0000 7f8877fff700 1 -- 172.21.15.42:0/1461709891 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f887807f090 con 0x7f88780a7730 2019-12-05T05:09:43.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:43.522+0000 7f8877fff700 1 --2- 172.21.15.42:0/1461709891 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f88780a7730 0x7f887808df70 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f8868008790 tx=0x7f886800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:43.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:43.522+0000 7f887cbaf700 1 --2- 172.21.15.42:0/1461709891 >> v2:172.21.15.42:3300/0 conn(0x7f887809fad0 0x7f887807eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:09:43.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:43.522+0000 7f8877fff700 1 --2- 172.21.15.42:0/1461709891 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f88780a7730 0x7f887808df70 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f8868008790 tx=0x7f886800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:43.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:43.522+0000 7f8877fff700 1 --2- 172.21.15.42:0/1461709891 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f88780a7730 0x7f887808df70 secure :-1 s=READY pgs=74 cs=0 l=1 rx=0x7f8868008790 tx=0x7f886800fc70).ready entity=mon.2 client_cookie=72ebb9a6d825dd66 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:09:43.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:43.522+0000 7f8877fff700 1 --2- 172.21.15.42:0/1461709891 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f88780a7730 0x7f887808df70 secure :-1 s=THROTTLE_DONE pgs=74 cs=0 l=1 rx=0x7f8868008790 tx=0x7f886800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:43.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:43.522+0000 7f88777fe700 1 -- 172.21.15.42:0/1461709891 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8868008bb0 con 0x7f88780a7730 2019-12-05T05:09:43.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:43.522+0000 7f8877fff700 1 --2- 172.21.15.42:0/1461709891 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f88780a7730 0x7f887808df70 secure :-1 s=THROTTLE_DONE pgs=74 cs=0 l=1 rx=0x7f8868008790 tx=0x7f886800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:43.530 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:43.522+0000 7f8877fff700 1 --2- 172.21.15.42:0/1461709891 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f88780a7730 0x7f887808df70 secure :-1 s=THROTTLE_DONE pgs=74 cs=0 l=1 rx=0x7f8868008790 tx=0x7f886800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:43.530 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:43.522+0000 7f88777fe700 1 -- 172.21.15.42:0/1461709891 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88680087d0 con 0x7f88780a7730 2019-12-05T05:09:43.530 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:43.522+0000 7f88777fe700 1 -- 172.21.15.42:0/1461709891 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8868011950 con 0x7f88780a7730 2019-12-05T05:09:43.530 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:43.522+0000 7f887de11700 1 -- 172.21.15.42:0/1461709891 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f88780a7730 msgr2=0x7f887808df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:09:43.530 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:43.522+0000 7f887de11700 1 --2- 172.21.15.42:0/1461709891 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f88780a7730 0x7f887808df70 secure :-1 s=READY pgs=74 cs=0 l=1 rx=0x7f8868008790 tx=0x7f886800fc70).stop 2019-12-05T05:09:43.530 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:43.522+0000 7f887de11700 1 -- 172.21.15.42:0/1461709891 shutdown_connections 2019-12-05T05:09:43.530 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:43.522+0000 7f887de11700 1 --2- 172.21.15.42:0/1461709891 >> v2:172.21.15.42:3300/0 conn(0x7f887809fad0 0x7f887807eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:09:43.531 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:43.522+0000 7f887de11700 1 --2- 172.21.15.42:0/1461709891 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f88780a7730 0x7f887808df70 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:09:43.531 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:43.522+0000 7f887de11700 1 -- 172.21.15.42:0/1461709891 shutdown_connections 2019-12-05T05:09:43.531 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:43.522+0000 7f887de11700 1 -- 172.21.15.42:0/1461709891 wait complete. 2019-12-05T05:09:43.531 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:43.522+0000 7f887de11700 1 -- 172.21.15.42:0/1461709891 >> 172.21.15.42:0/1461709891 conn(0x7f88780aabd0 msgr2=0x7f88780a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:09:43.535 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:43.522+0000 7f887de11700 1 Processor -- start 2019-12-05T05:09:43.535 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:43.522+0000 7f887de11700 1 -- start start 2019-12-05T05:09:43.536 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:43.526+0000 7f887de11700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f887811ad80 0x7f8878133d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:09:43.536 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:43.526+0000 7f887de11700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f887811bb10 con 0x7f887811a950 2019-12-05T05:09:43.536 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:43.526+0000 7f887de11700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88781342a0 con 0x7f8878116350 2019-12-05T05:09:43.536 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:43.526+0000 7f887de11700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8878116780 con 0x7f887811ad80 2019-12-05T05:09:43.536 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:43.526+0000 7f8877fff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f8878116350 0x7f88780a1b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:56906/0 (socket says 172.21.15.42:56906) 2019-12-05T05:09:43.536 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:43.526+0000 7f8877fff700 1 -- 172.21.15.42:0/1599712358 learned_addr learned my addr 172.21.15.42:0/1599712358 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:09:43.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:43.526+0000 7f887cbaf700 1 --2- 172.21.15.42:0/1599712358 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f887811ad80 0x7f8878133d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:09:43.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:43.526+0000 7f88757fa700 1 -- 172.21.15.42:0/1599712358 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2249716818 0 0) 0x7f88781342a0 con 0x7f8878116350 2019-12-05T05:09:43.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:43.526+0000 7f88757fa700 1 -- 172.21.15.42:0/1599712358 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f885c0030b0 con 0x7f8878116350 2019-12-05T05:09:43.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:43.526+0000 7f88757fa700 1 -- 172.21.15.42:0/1599712358 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2199046107 0 0) 0x7f887811bb10 con 0x7f887811a950 2019-12-05T05:09:43.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:43.526+0000 7f88757fa700 1 -- 172.21.15.42:0/1599712358 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88781342a0 con 0x7f887811a950 2019-12-05T05:09:43.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:43.526+0000 7f887cbaf700 1 -- 172.21.15.42:0/1599712358 >> v1:172.21.15.42:6789/0 conn(0x7f8878116350 legacy=0x7f88780a1b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:09:43.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:43.526+0000 7f887cbaf700 1 -- 172.21.15.42:0/1599712358 >> v1:172.21.15.40:6789/0 conn(0x7f887811a950 legacy=0x7f8878133550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:09:43.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:43.526+0000 7f887cbaf700 1 -- 172.21.15.42:0/1599712358 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f887807a320 con 0x7f887811ad80 2019-12-05T05:09:43.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:43.526+0000 7f887cbaf700 1 --2- 172.21.15.42:0/1599712358 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f887811ad80 0x7f8878133d50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f88700032b0 tx=0x7f8870010ea0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:43.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:43.526+0000 7f887cbaf700 1 --2- 172.21.15.42:0/1599712358 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f887811ad80 0x7f8878133d50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f88700032b0 tx=0x7f8870010ea0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:43.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:43.526+0000 7f887cbaf700 1 --2- 172.21.15.42:0/1599712358 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f887811ad80 0x7f8878133d50 secure :-1 s=READY pgs=75 cs=0 l=1 rx=0x7f88700032b0 tx=0x7f8870010ea0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:09:43.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:43.526+0000 7f887cbaf700 1 --2- 172.21.15.42:0/1599712358 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f887811ad80 0x7f8878133d50 secure :-1 s=THROTTLE_DONE pgs=75 cs=0 l=1 rx=0x7f88700032b0 tx=0x7f8870010ea0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:43.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:43.526+0000 7f887cbaf700 1 --2- 172.21.15.42:0/1599712358 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f887811ad80 0x7f8878133d50 secure :-1 s=THROTTLE_DONE pgs=75 cs=0 l=1 rx=0x7f88700032b0 tx=0x7f8870010ea0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:43.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:43.526+0000 7f887cbaf700 1 --2- 172.21.15.42:0/1599712358 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f887811ad80 0x7f8878133d50 secure :-1 s=THROTTLE_DONE pgs=75 cs=0 l=1 rx=0x7f88700032b0 tx=0x7f8870010ea0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:43.539 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:43.526+0000 7f88757fa700 1 -- 172.21.15.42:0/1599712358 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8870007160 con 0x7f887811ad80 2019-12-05T05:09:43.539 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:43.526+0000 7f88757fa700 1 -- 172.21.15.42:0/1599712358 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8870005bc0 con 0x7f887811ad80 2019-12-05T05:09:43.539 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:43.526+0000 7f887de11700 1 -- 172.21.15.42:0/1599712358 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8878078300 con 0x7f887811ad80 2019-12-05T05:09:43.539 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:43.526+0000 7f88757fa700 1 -- 172.21.15.42:0/1599712358 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8870012560 con 0x7f887811ad80 2019-12-05T05:09:43.539 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:43.526+0000 7f887de11700 1 -- 172.21.15.42:0/1599712358 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8878078800 con 0x7f887811ad80 2019-12-05T05:09:43.539 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:43.530+0000 7f887cbaf700 1 --2- 172.21.15.42:0/1599712358 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f887811ad80 0x7f8878133d50 secure :-1 s=THROTTLE_DONE pgs=75 cs=0 l=1 rx=0x7f88700032b0 tx=0x7f8870010ea0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:43.539 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:43.530+0000 7f88757fa700 1 -- 172.21.15.42:0/1599712358 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f8870012760 con 0x7f887811ad80 2019-12-05T05:09:43.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:43.530+0000 7f887cbaf700 1 --2- 172.21.15.42:0/1599712358 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f887811ad80 0x7f8878133d50 secure :-1 s=THROTTLE_DONE pgs=75 cs=0 l=1 rx=0x7f88700032b0 tx=0x7f8870010ea0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:43.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:43.530+0000 7f88757fa700 1 --2- 172.21.15.42:0/1599712358 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f885c03b4c0 0x7f885c03d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:09:43.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:43.530+0000 7f88757fa700 1 -- 172.21.15.42:0/1599712358 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6289+0+0 (secure 0 0 0) 0x7f887004b3b0 con 0x7f887811ad80 2019-12-05T05:09:43.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:43.530+0000 7f8877fff700 1 --2- 172.21.15.42:0/1599712358 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f885c03b4c0 0x7f885c03d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:09:43.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:43.530+0000 7f8877fff700 1 --2- 172.21.15.42:0/1599712358 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f885c03b4c0 0x7f885c03d9d0 crc :-1 s=READY pgs=166 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:09:43.541 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:43.534+0000 7f887de11700 1 -- 172.21.15.42:0/1599712358 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f887805d100 con 0x7f887811ad80 2019-12-05T05:09:43.541 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:43.538+0000 7f887cbaf700 1 --2- 172.21.15.42:0/1599712358 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f887811ad80 0x7f8878133d50 secure :-1 s=THROTTLE_DONE pgs=75 cs=0 l=1 rx=0x7f88700032b0 tx=0x7f8870010ea0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:43.542 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:43.538+0000 7f88757fa700 1 -- 172.21.15.42:0/1599712358 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f8870032430 con 0x7f887811ad80 2019-12-05T05:09:43.712 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:43.706+0000 7f887de11700 1 -- 172.21.15.42:0/1599712358 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f887805b8b0 con 0x7f887811ad80 2019-12-05T05:09:43.712 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:43.706+0000 7f887cbaf700 1 --2- 172.21.15.42:0/1599712358 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f887811ad80 0x7f8878133d50 secure :-1 s=THROTTLE_DONE pgs=75 cs=0 l=1 rx=0x7f88700032b0 tx=0x7f8870010ea0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:43.712 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:43.706+0000 7f88757fa700 1 -- 172.21.15.42:0/1599712358 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+2105 (secure 0 0 0) 0x7f8870032690 con 0x7f887811ad80 2019-12-05T05:09:43.713 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 35 2019-12-05T05:09:43.713 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:09:43.713 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4595,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/211584380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":211584380},{"type":"v1","addr":"172.21.15.40:6833","nonce":211584380}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":35}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:09:28.296949+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4978},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4978":{"gid":4978,"name":"a","rank":0,"incarnation":30,"state":"up:active","state_seq":36,"addr":"172.21.15.42:6835/3650564018","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3650564018},{"type":"v1","addr":"172.21.15.42:6835","nonce":3650564018}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:09:43.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:43.710+0000 7f887de11700 1 -- 172.21.15.42:0/1599712358 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f885c03b4c0 msgr2=0x7f885c03d9d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:09:43.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:43.710+0000 7f887de11700 1 --2- 172.21.15.42:0/1599712358 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f885c03b4c0 0x7f885c03d9d0 crc :-1 s=READY pgs=166 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:09:43.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:43.710+0000 7f887de11700 1 -- 172.21.15.42:0/1599712358 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f887811ad80 msgr2=0x7f8878133d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:09:43.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:43.710+0000 7f887de11700 1 --2- 172.21.15.42:0/1599712358 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f887811ad80 0x7f8878133d50 secure :-1 s=READY pgs=75 cs=0 l=1 rx=0x7f88700032b0 tx=0x7f8870010ea0).stop 2019-12-05T05:09:43.715 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:43.710+0000 7f887de11700 1 -- 172.21.15.42:0/1599712358 shutdown_connections 2019-12-05T05:09:43.715 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:43.710+0000 7f887de11700 1 --2- 172.21.15.42:0/1599712358 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f885c03b4c0 0x7f885c03d9d0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:09:43.715 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:43.710+0000 7f887de11700 1 --2- 172.21.15.42:0/1599712358 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f887811ad80 0x7f8878133d50 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:09:43.715 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:43.710+0000 7f887de11700 1 -- 172.21.15.42:0/1599712358 shutdown_connections 2019-12-05T05:09:43.715 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:43.710+0000 7f887de11700 1 -- 172.21.15.42:0/1599712358 wait complete. 2019-12-05T05:09:43.715 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:43.710+0000 7f887de11700 1 -- 172.21.15.42:0/1599712358 >> 172.21.15.42:0/1599712358 conn(0x7f88780aabd0 msgr2=0x7f88780a7f80 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:09:43.724 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b reported in up:standby state 2019-12-05T05:09:43.724 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 5 secs before thrashing 2019-12-05T05:09:46.824 INFO:tasks.workunit.client.0.smithi042.stdout:removed './276/37' 2019-12-05T05:09:46.824 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './276' 2019-12-05T05:09:46.825 INFO:tasks.workunit.client.0.smithi042.stdout:removed './277/21' 2019-12-05T05:09:46.825 INFO:tasks.workunit.client.0.smithi042.stdout:removed './277/27' 2019-12-05T05:09:46.825 INFO:tasks.workunit.client.0.smithi042.stdout:removed './277/31' 2019-12-05T05:09:46.825 INFO:tasks.workunit.client.0.smithi042.stdout:removed './277/36' 2019-12-05T05:09:46.825 INFO:tasks.workunit.client.0.smithi042.stdout:removed './277/20' 2019-12-05T05:09:46.825 INFO:tasks.workunit.client.0.smithi042.stdout:removed './277/39' 2019-12-05T05:09:46.825 INFO:tasks.workunit.client.0.smithi042.stdout:removed './277/34' 2019-12-05T05:09:46.826 INFO:tasks.workunit.client.0.smithi042.stdout:removed './277/38' 2019-12-05T05:09:46.826 INFO:tasks.workunit.client.0.smithi042.stdout:removed './277/28' 2019-12-05T05:09:46.826 INFO:tasks.workunit.client.0.smithi042.stdout:removed './277/24' 2019-12-05T05:09:46.826 INFO:tasks.workunit.client.0.smithi042.stdout:removed './277/29' 2019-12-05T05:09:46.826 INFO:tasks.workunit.client.0.smithi042.stdout:removed './277/32' 2019-12-05T05:09:46.826 INFO:tasks.workunit.client.0.smithi042.stdout:removed './277/22' 2019-12-05T05:09:46.826 INFO:tasks.workunit.client.0.smithi042.stdout:removed './277/25' 2019-12-05T05:09:46.826 INFO:tasks.workunit.client.0.smithi042.stdout:removed './277/30' 2019-12-05T05:09:46.827 INFO:tasks.workunit.client.0.smithi042.stdout:removed './277/33' 2019-12-05T05:09:46.827 INFO:tasks.workunit.client.0.smithi042.stdout:removed './277/23' 2019-12-05T05:09:46.827 INFO:tasks.workunit.client.0.smithi042.stdout:removed './277/35' 2019-12-05T05:09:46.827 INFO:tasks.workunit.client.0.smithi042.stdout:removed './277/26' 2019-12-05T05:09:46.827 INFO:tasks.workunit.client.0.smithi042.stdout:removed './277/37' 2019-12-05T05:09:46.827 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './277' 2019-12-05T05:09:46.827 INFO:tasks.workunit.client.0.smithi042.stdout:removed './278/21' 2019-12-05T05:09:46.828 INFO:tasks.workunit.client.0.smithi042.stdout:removed './278/27' 2019-12-05T05:09:46.828 INFO:tasks.workunit.client.0.smithi042.stdout:removed './278/31' 2019-12-05T05:09:46.828 INFO:tasks.workunit.client.0.smithi042.stdout:removed './278/36' 2019-12-05T05:09:46.828 INFO:tasks.workunit.client.0.smithi042.stdout:removed './278/20' 2019-12-05T05:09:46.828 INFO:tasks.workunit.client.0.smithi042.stdout:removed './278/39' 2019-12-05T05:09:46.828 INFO:tasks.workunit.client.0.smithi042.stdout:removed './278/34' 2019-12-05T05:09:46.828 INFO:tasks.workunit.client.0.smithi042.stdout:removed './278/38' 2019-12-05T05:09:46.829 INFO:tasks.workunit.client.0.smithi042.stdout:removed './278/28' 2019-12-05T05:09:46.829 INFO:tasks.workunit.client.0.smithi042.stdout:removed './278/24' 2019-12-05T05:09:46.829 INFO:tasks.workunit.client.0.smithi042.stdout:removed './278/29' 2019-12-05T05:09:46.829 INFO:tasks.workunit.client.0.smithi042.stdout:removed './278/32' 2019-12-05T05:09:46.829 INFO:tasks.workunit.client.0.smithi042.stdout:removed './278/22' 2019-12-05T05:09:46.829 INFO:tasks.workunit.client.0.smithi042.stdout:removed './278/25' 2019-12-05T05:09:46.829 INFO:tasks.workunit.client.0.smithi042.stdout:removed './278/30' 2019-12-05T05:09:46.830 INFO:tasks.workunit.client.0.smithi042.stdout:removed './278/33' 2019-12-05T05:09:46.830 INFO:tasks.workunit.client.0.smithi042.stdout:removed './278/23' 2019-12-05T05:09:46.830 INFO:tasks.workunit.client.0.smithi042.stdout:removed './278/35' 2019-12-05T05:09:46.830 INFO:tasks.workunit.client.0.smithi042.stdout:removed './278/26' 2019-12-05T05:09:46.830 INFO:tasks.workunit.client.0.smithi042.stdout:removed './278/37' 2019-12-05T05:09:46.830 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './278' 2019-12-05T05:09:46.830 INFO:tasks.workunit.client.0.smithi042.stdout:removed './279/21' 2019-12-05T05:09:46.831 INFO:tasks.workunit.client.0.smithi042.stdout:removed './279/27' 2019-12-05T05:09:46.831 INFO:tasks.workunit.client.0.smithi042.stdout:removed './279/31' 2019-12-05T05:09:46.831 INFO:tasks.workunit.client.0.smithi042.stdout:removed './279/36' 2019-12-05T05:09:46.831 INFO:tasks.workunit.client.0.smithi042.stdout:removed './279/20' 2019-12-05T05:09:46.831 INFO:tasks.workunit.client.0.smithi042.stdout:removed './279/39' 2019-12-05T05:09:46.831 INFO:tasks.workunit.client.0.smithi042.stdout:removed './279/34' 2019-12-05T05:09:46.831 INFO:tasks.workunit.client.0.smithi042.stdout:removed './279/38' 2019-12-05T05:09:46.832 INFO:tasks.workunit.client.0.smithi042.stdout:removed './279/28' 2019-12-05T05:09:46.832 INFO:tasks.workunit.client.0.smithi042.stdout:removed './279/24' 2019-12-05T05:09:46.832 INFO:tasks.workunit.client.0.smithi042.stdout:removed './279/29' 2019-12-05T05:09:46.832 INFO:tasks.workunit.client.0.smithi042.stdout:removed './279/32' 2019-12-05T05:09:46.832 INFO:tasks.workunit.client.0.smithi042.stdout:removed './279/22' 2019-12-05T05:09:46.833 INFO:tasks.workunit.client.0.smithi042.stdout:removed './279/25' 2019-12-05T05:09:46.833 INFO:tasks.workunit.client.0.smithi042.stdout:removed './279/30' 2019-12-05T05:09:46.833 INFO:tasks.workunit.client.0.smithi042.stdout:removed './279/33' 2019-12-05T05:09:46.833 INFO:tasks.workunit.client.0.smithi042.stdout:removed './279/23' 2019-12-05T05:09:46.833 INFO:tasks.workunit.client.0.smithi042.stdout:removed './279/35' 2019-12-05T05:09:46.833 INFO:tasks.workunit.client.0.smithi042.stdout:removed './279/26' 2019-12-05T05:09:46.833 INFO:tasks.workunit.client.0.smithi042.stdout:removed './279/37' 2019-12-05T05:09:46.834 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './279' 2019-12-05T05:09:46.834 INFO:tasks.workunit.client.0.smithi042.stdout:removed './280/21' 2019-12-05T05:09:46.834 INFO:tasks.workunit.client.0.smithi042.stdout:removed './280/27' 2019-12-05T05:09:46.834 INFO:tasks.workunit.client.0.smithi042.stdout:removed './280/31' 2019-12-05T05:09:46.834 INFO:tasks.workunit.client.0.smithi042.stdout:removed './280/36' 2019-12-05T05:09:46.834 INFO:tasks.workunit.client.0.smithi042.stdout:removed './280/20' 2019-12-05T05:09:46.835 INFO:tasks.workunit.client.0.smithi042.stdout:removed './280/39' 2019-12-05T05:09:46.835 INFO:tasks.workunit.client.0.smithi042.stdout:removed './280/34' 2019-12-05T05:09:46.835 INFO:tasks.workunit.client.0.smithi042.stdout:removed './280/38' 2019-12-05T05:09:46.835 INFO:tasks.workunit.client.0.smithi042.stdout:removed './280/28' 2019-12-05T05:09:46.835 INFO:tasks.workunit.client.0.smithi042.stdout:removed './280/24' 2019-12-05T05:09:46.835 INFO:tasks.workunit.client.0.smithi042.stdout:removed './280/29' 2019-12-05T05:09:46.836 INFO:tasks.workunit.client.0.smithi042.stdout:removed './280/32' 2019-12-05T05:09:46.836 INFO:tasks.workunit.client.0.smithi042.stdout:removed './280/22' 2019-12-05T05:09:46.836 INFO:tasks.workunit.client.0.smithi042.stdout:removed './280/25' 2019-12-05T05:09:46.836 INFO:tasks.workunit.client.0.smithi042.stdout:removed './280/30' 2019-12-05T05:09:46.836 INFO:tasks.workunit.client.0.smithi042.stdout:removed './280/33' 2019-12-05T05:09:46.836 INFO:tasks.workunit.client.0.smithi042.stdout:removed './280/23' 2019-12-05T05:09:46.837 INFO:tasks.workunit.client.0.smithi042.stdout:removed './280/35' 2019-12-05T05:09:46.837 INFO:tasks.workunit.client.0.smithi042.stdout:removed './280/26' 2019-12-05T05:09:46.837 INFO:tasks.workunit.client.0.smithi042.stdout:removed './280/37' 2019-12-05T05:09:46.837 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './280' 2019-12-05T05:09:46.837 INFO:tasks.workunit.client.0.smithi042.stdout:removed './281/21' 2019-12-05T05:09:46.837 INFO:tasks.workunit.client.0.smithi042.stdout:removed './281/27' 2019-12-05T05:09:46.838 INFO:tasks.workunit.client.0.smithi042.stdout:removed './281/31' 2019-12-05T05:09:46.838 INFO:tasks.workunit.client.0.smithi042.stdout:removed './281/36' 2019-12-05T05:09:46.838 INFO:tasks.workunit.client.0.smithi042.stdout:removed './281/20' 2019-12-05T05:09:46.838 INFO:tasks.workunit.client.0.smithi042.stdout:removed './281/39' 2019-12-05T05:09:46.838 INFO:tasks.workunit.client.0.smithi042.stdout:removed './281/34' 2019-12-05T05:09:46.838 INFO:tasks.workunit.client.0.smithi042.stdout:removed './281/38' 2019-12-05T05:09:46.838 INFO:tasks.workunit.client.0.smithi042.stdout:removed './281/28' 2019-12-05T05:09:46.838 INFO:tasks.workunit.client.0.smithi042.stdout:removed './281/24' 2019-12-05T05:09:46.839 INFO:tasks.workunit.client.0.smithi042.stdout:removed './281/29' 2019-12-05T05:09:46.839 INFO:tasks.workunit.client.0.smithi042.stdout:removed './281/32' 2019-12-05T05:09:46.839 INFO:tasks.workunit.client.0.smithi042.stdout:removed './281/22' 2019-12-05T05:09:46.839 INFO:tasks.workunit.client.0.smithi042.stdout:removed './281/25' 2019-12-05T05:09:46.839 INFO:tasks.workunit.client.0.smithi042.stdout:removed './281/30' 2019-12-05T05:09:46.839 INFO:tasks.workunit.client.0.smithi042.stdout:removed './281/33' 2019-12-05T05:09:46.839 INFO:tasks.workunit.client.0.smithi042.stdout:removed './281/23' 2019-12-05T05:09:46.840 INFO:tasks.workunit.client.0.smithi042.stdout:removed './281/35' 2019-12-05T05:09:46.840 INFO:tasks.workunit.client.0.smithi042.stdout:removed './281/26' 2019-12-05T05:09:46.840 INFO:tasks.workunit.client.0.smithi042.stdout:removed './281/37' 2019-12-05T05:09:46.840 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './281' 2019-12-05T05:09:46.840 INFO:tasks.workunit.client.0.smithi042.stdout:removed './282/21' 2019-12-05T05:09:46.840 INFO:tasks.workunit.client.0.smithi042.stdout:removed './282/27' 2019-12-05T05:09:46.840 INFO:tasks.workunit.client.0.smithi042.stdout:removed './282/31' 2019-12-05T05:09:46.841 INFO:tasks.workunit.client.0.smithi042.stdout:removed './282/36' 2019-12-05T05:09:46.841 INFO:tasks.workunit.client.0.smithi042.stdout:removed './282/20' 2019-12-05T05:09:46.841 INFO:tasks.workunit.client.0.smithi042.stdout:removed './282/39' 2019-12-05T05:09:46.841 INFO:tasks.workunit.client.0.smithi042.stdout:removed './282/34' 2019-12-05T05:09:46.841 INFO:tasks.workunit.client.0.smithi042.stdout:removed './282/38' 2019-12-05T05:09:46.841 INFO:tasks.workunit.client.0.smithi042.stdout:removed './282/28' 2019-12-05T05:09:46.841 INFO:tasks.workunit.client.0.smithi042.stdout:removed './282/24' 2019-12-05T05:09:46.842 INFO:tasks.workunit.client.0.smithi042.stdout:removed './282/29' 2019-12-05T05:09:46.842 INFO:tasks.workunit.client.0.smithi042.stdout:removed './282/32' 2019-12-05T05:09:46.842 INFO:tasks.workunit.client.0.smithi042.stdout:removed './282/22' 2019-12-05T05:09:46.842 INFO:tasks.workunit.client.0.smithi042.stdout:removed './282/25' 2019-12-05T05:09:46.842 INFO:tasks.workunit.client.0.smithi042.stdout:removed './282/30' 2019-12-05T05:09:46.842 INFO:tasks.workunit.client.0.smithi042.stdout:removed './282/33' 2019-12-05T05:09:46.842 INFO:tasks.workunit.client.0.smithi042.stdout:removed './282/23' 2019-12-05T05:09:46.843 INFO:tasks.workunit.client.0.smithi042.stdout:removed './282/35' 2019-12-05T05:09:46.843 INFO:tasks.workunit.client.0.smithi042.stdout:removed './282/26' 2019-12-05T05:09:46.843 INFO:tasks.workunit.client.0.smithi042.stdout:removed './282/37' 2019-12-05T05:09:46.843 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './282' 2019-12-05T05:09:46.843 INFO:tasks.workunit.client.0.smithi042.stdout:removed './283/21' 2019-12-05T05:09:46.843 INFO:tasks.workunit.client.0.smithi042.stdout:removed './283/27' 2019-12-05T05:09:46.843 INFO:tasks.workunit.client.0.smithi042.stdout:removed './283/31' 2019-12-05T05:09:46.844 INFO:tasks.workunit.client.0.smithi042.stdout:removed './283/36' 2019-12-05T05:09:46.844 INFO:tasks.workunit.client.0.smithi042.stdout:removed './283/20' 2019-12-05T05:09:46.844 INFO:tasks.workunit.client.0.smithi042.stdout:removed './283/39' 2019-12-05T05:09:46.844 INFO:tasks.workunit.client.0.smithi042.stdout:removed './283/34' 2019-12-05T05:09:46.844 INFO:tasks.workunit.client.0.smithi042.stdout:removed './283/38' 2019-12-05T05:09:46.844 INFO:tasks.workunit.client.0.smithi042.stdout:removed './283/28' 2019-12-05T05:09:46.844 INFO:tasks.workunit.client.0.smithi042.stdout:removed './283/24' 2019-12-05T05:09:46.845 INFO:tasks.workunit.client.0.smithi042.stdout:removed './283/29' 2019-12-05T05:09:46.845 INFO:tasks.workunit.client.0.smithi042.stdout:removed './283/32' 2019-12-05T05:09:46.845 INFO:tasks.workunit.client.0.smithi042.stdout:removed './283/22' 2019-12-05T05:09:46.845 INFO:tasks.workunit.client.0.smithi042.stdout:removed './283/25' 2019-12-05T05:09:46.845 INFO:tasks.workunit.client.0.smithi042.stdout:removed './283/30' 2019-12-05T05:09:46.845 INFO:tasks.workunit.client.0.smithi042.stdout:removed './283/33' 2019-12-05T05:09:46.846 INFO:tasks.workunit.client.0.smithi042.stdout:removed './283/23' 2019-12-05T05:09:46.846 INFO:tasks.workunit.client.0.smithi042.stdout:removed './283/35' 2019-12-05T05:09:46.846 INFO:tasks.workunit.client.0.smithi042.stdout:removed './283/26' 2019-12-05T05:09:46.846 INFO:tasks.workunit.client.0.smithi042.stdout:removed './283/37' 2019-12-05T05:09:46.846 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './283' 2019-12-05T05:09:46.846 INFO:tasks.workunit.client.0.smithi042.stdout:removed './284/21' 2019-12-05T05:09:46.846 INFO:tasks.workunit.client.0.smithi042.stdout:removed './284/27' 2019-12-05T05:09:46.847 INFO:tasks.workunit.client.0.smithi042.stdout:removed './284/31' 2019-12-05T05:09:46.847 INFO:tasks.workunit.client.0.smithi042.stdout:removed './284/36' 2019-12-05T05:09:46.847 INFO:tasks.workunit.client.0.smithi042.stdout:removed './284/20' 2019-12-05T05:09:46.847 INFO:tasks.workunit.client.0.smithi042.stdout:removed './284/39' 2019-12-05T05:09:46.847 INFO:tasks.workunit.client.0.smithi042.stdout:removed './284/34' 2019-12-05T05:09:46.847 INFO:tasks.workunit.client.0.smithi042.stdout:removed './284/38' 2019-12-05T05:09:46.847 INFO:tasks.workunit.client.0.smithi042.stdout:removed './284/28' 2019-12-05T05:09:46.848 INFO:tasks.workunit.client.0.smithi042.stdout:removed './284/24' 2019-12-05T05:09:46.848 INFO:tasks.workunit.client.0.smithi042.stdout:removed './284/29' 2019-12-05T05:09:46.848 INFO:tasks.workunit.client.0.smithi042.stdout:removed './284/32' 2019-12-05T05:09:46.848 INFO:tasks.workunit.client.0.smithi042.stdout:removed './284/22' 2019-12-05T05:09:46.848 INFO:tasks.workunit.client.0.smithi042.stdout:removed './284/25' 2019-12-05T05:09:46.848 INFO:tasks.workunit.client.0.smithi042.stdout:removed './284/30' 2019-12-05T05:09:46.848 INFO:tasks.workunit.client.0.smithi042.stdout:removed './284/33' 2019-12-05T05:09:46.849 INFO:tasks.workunit.client.0.smithi042.stdout:removed './284/23' 2019-12-05T05:09:46.849 INFO:tasks.workunit.client.0.smithi042.stdout:removed './284/35' 2019-12-05T05:09:46.849 INFO:tasks.workunit.client.0.smithi042.stdout:removed './284/26' 2019-12-05T05:09:46.849 INFO:tasks.workunit.client.0.smithi042.stdout:removed './284/37' 2019-12-05T05:09:46.849 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './284' 2019-12-05T05:09:46.850 INFO:tasks.workunit.client.0.smithi042.stdout:removed './285/21' 2019-12-05T05:09:46.850 INFO:tasks.workunit.client.0.smithi042.stdout:removed './285/27' 2019-12-05T05:09:46.850 INFO:tasks.workunit.client.0.smithi042.stdout:removed './285/31' 2019-12-05T05:09:46.850 INFO:tasks.workunit.client.0.smithi042.stdout:removed './285/36' 2019-12-05T05:09:46.850 INFO:tasks.workunit.client.0.smithi042.stdout:removed './285/20' 2019-12-05T05:09:46.850 INFO:tasks.workunit.client.0.smithi042.stdout:removed './285/39' 2019-12-05T05:09:46.851 INFO:tasks.workunit.client.0.smithi042.stdout:removed './285/34' 2019-12-05T05:09:46.851 INFO:tasks.workunit.client.0.smithi042.stdout:removed './285/38' 2019-12-05T05:09:46.851 INFO:tasks.workunit.client.0.smithi042.stdout:removed './285/28' 2019-12-05T05:09:46.851 INFO:tasks.workunit.client.0.smithi042.stdout:removed './285/24' 2019-12-05T05:09:46.851 INFO:tasks.workunit.client.0.smithi042.stdout:removed './285/29' 2019-12-05T05:09:46.852 INFO:tasks.workunit.client.0.smithi042.stdout:removed './285/32' 2019-12-05T05:09:46.852 INFO:tasks.workunit.client.0.smithi042.stdout:removed './285/22' 2019-12-05T05:09:46.852 INFO:tasks.workunit.client.0.smithi042.stdout:removed './285/25' 2019-12-05T05:09:46.852 INFO:tasks.workunit.client.0.smithi042.stdout:removed './285/30' 2019-12-05T05:09:46.852 INFO:tasks.workunit.client.0.smithi042.stdout:removed './285/33' 2019-12-05T05:09:46.853 INFO:tasks.workunit.client.0.smithi042.stdout:removed './285/23' 2019-12-05T05:09:46.853 INFO:tasks.workunit.client.0.smithi042.stdout:removed './285/35' 2019-12-05T05:09:46.853 INFO:tasks.workunit.client.0.smithi042.stdout:removed './285/26' 2019-12-05T05:09:46.853 INFO:tasks.workunit.client.0.smithi042.stdout:removed './285/37' 2019-12-05T05:09:46.853 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './285' 2019-12-05T05:09:46.854 INFO:tasks.workunit.client.0.smithi042.stdout:removed './286/21' 2019-12-05T05:09:46.854 INFO:tasks.workunit.client.0.smithi042.stdout:removed './286/27' 2019-12-05T05:09:46.854 INFO:tasks.workunit.client.0.smithi042.stdout:removed './286/31' 2019-12-05T05:09:46.854 INFO:tasks.workunit.client.0.smithi042.stdout:removed './286/36' 2019-12-05T05:09:46.854 INFO:tasks.workunit.client.0.smithi042.stdout:removed './286/20' 2019-12-05T05:09:46.855 INFO:tasks.workunit.client.0.smithi042.stdout:removed './286/39' 2019-12-05T05:09:46.855 INFO:tasks.workunit.client.0.smithi042.stdout:removed './286/34' 2019-12-05T05:09:46.855 INFO:tasks.workunit.client.0.smithi042.stdout:removed './286/38' 2019-12-05T05:09:46.855 INFO:tasks.workunit.client.0.smithi042.stdout:removed './286/28' 2019-12-05T05:09:46.855 INFO:tasks.workunit.client.0.smithi042.stdout:removed './286/24' 2019-12-05T05:09:46.856 INFO:tasks.workunit.client.0.smithi042.stdout:removed './286/29' 2019-12-05T05:09:46.856 INFO:tasks.workunit.client.0.smithi042.stdout:removed './286/32' 2019-12-05T05:09:46.856 INFO:tasks.workunit.client.0.smithi042.stdout:removed './286/22' 2019-12-05T05:09:46.856 INFO:tasks.workunit.client.0.smithi042.stdout:removed './286/25' 2019-12-05T05:09:46.857 INFO:tasks.workunit.client.0.smithi042.stdout:removed './286/30' 2019-12-05T05:09:46.857 INFO:tasks.workunit.client.0.smithi042.stdout:removed './286/33' 2019-12-05T05:09:46.857 INFO:tasks.workunit.client.0.smithi042.stdout:removed './286/23' 2019-12-05T05:09:46.857 INFO:tasks.workunit.client.0.smithi042.stdout:removed './286/35' 2019-12-05T05:09:46.857 INFO:tasks.workunit.client.0.smithi042.stdout:removed './286/26' 2019-12-05T05:09:46.858 INFO:tasks.workunit.client.0.smithi042.stdout:removed './286/37' 2019-12-05T05:09:46.859 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './286' 2019-12-05T05:09:48.402 INFO:tasks.workunit.client.0.smithi042.stdout:removed './287/21' 2019-12-05T05:09:48.403 INFO:tasks.workunit.client.0.smithi042.stdout:removed './287/27' 2019-12-05T05:09:48.403 INFO:tasks.workunit.client.0.smithi042.stdout:removed './287/31' 2019-12-05T05:09:48.403 INFO:tasks.workunit.client.0.smithi042.stdout:removed './287/36' 2019-12-05T05:09:48.403 INFO:tasks.workunit.client.0.smithi042.stdout:removed './287/20' 2019-12-05T05:09:48.403 INFO:tasks.workunit.client.0.smithi042.stdout:removed './287/39' 2019-12-05T05:09:48.403 INFO:tasks.workunit.client.0.smithi042.stdout:removed './287/34' 2019-12-05T05:09:48.403 INFO:tasks.workunit.client.0.smithi042.stdout:removed './287/38' 2019-12-05T05:09:48.404 INFO:tasks.workunit.client.0.smithi042.stdout:removed './287/28' 2019-12-05T05:09:48.404 INFO:tasks.workunit.client.0.smithi042.stdout:removed './287/24' 2019-12-05T05:09:48.404 INFO:tasks.workunit.client.0.smithi042.stdout:removed './287/29' 2019-12-05T05:09:48.404 INFO:tasks.workunit.client.0.smithi042.stdout:removed './287/32' 2019-12-05T05:09:48.404 INFO:tasks.workunit.client.0.smithi042.stdout:removed './287/22' 2019-12-05T05:09:48.404 INFO:tasks.workunit.client.0.smithi042.stdout:removed './287/25' 2019-12-05T05:09:48.404 INFO:tasks.workunit.client.0.smithi042.stdout:removed './287/30' 2019-12-05T05:09:48.405 INFO:tasks.workunit.client.0.smithi042.stdout:removed './287/33' 2019-12-05T05:09:48.405 INFO:tasks.workunit.client.0.smithi042.stdout:removed './287/23' 2019-12-05T05:09:48.405 INFO:tasks.workunit.client.0.smithi042.stdout:removed './287/35' 2019-12-05T05:09:48.405 INFO:tasks.workunit.client.0.smithi042.stdout:removed './287/26' 2019-12-05T05:09:48.405 INFO:tasks.workunit.client.0.smithi042.stdout:removed './287/37' 2019-12-05T05:09:48.405 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './287' 2019-12-05T05:09:48.405 INFO:tasks.workunit.client.0.smithi042.stdout:removed './288/21' 2019-12-05T05:09:48.413 INFO:tasks.workunit.client.0.smithi042.stdout:removed './288/27' 2019-12-05T05:09:48.413 INFO:tasks.workunit.client.0.smithi042.stdout:removed './288/31' 2019-12-05T05:09:48.413 INFO:tasks.workunit.client.0.smithi042.stdout:removed './288/36' 2019-12-05T05:09:48.413 INFO:tasks.workunit.client.0.smithi042.stdout:removed './288/20' 2019-12-05T05:09:48.413 INFO:tasks.workunit.client.0.smithi042.stdout:removed './288/39' 2019-12-05T05:09:48.413 INFO:tasks.workunit.client.0.smithi042.stdout:removed './288/34' 2019-12-05T05:09:48.413 INFO:tasks.workunit.client.0.smithi042.stdout:removed './288/38' 2019-12-05T05:09:48.414 INFO:tasks.workunit.client.0.smithi042.stdout:removed './288/28' 2019-12-05T05:09:48.414 INFO:tasks.workunit.client.0.smithi042.stdout:removed './288/24' 2019-12-05T05:09:48.414 INFO:tasks.workunit.client.0.smithi042.stdout:removed './288/29' 2019-12-05T05:09:48.414 INFO:tasks.workunit.client.0.smithi042.stdout:removed './288/32' 2019-12-05T05:09:48.414 INFO:tasks.workunit.client.0.smithi042.stdout:removed './288/22' 2019-12-05T05:09:48.414 INFO:tasks.workunit.client.0.smithi042.stdout:removed './288/25' 2019-12-05T05:09:48.414 INFO:tasks.workunit.client.0.smithi042.stdout:removed './288/30' 2019-12-05T05:09:48.415 INFO:tasks.workunit.client.0.smithi042.stdout:removed './288/33' 2019-12-05T05:09:48.415 INFO:tasks.workunit.client.0.smithi042.stdout:removed './288/23' 2019-12-05T05:09:48.415 INFO:tasks.workunit.client.0.smithi042.stdout:removed './288/35' 2019-12-05T05:09:48.415 INFO:tasks.workunit.client.0.smithi042.stdout:removed './288/26' 2019-12-05T05:09:48.415 INFO:tasks.workunit.client.0.smithi042.stdout:removed './288/37' 2019-12-05T05:09:48.415 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './288' 2019-12-05T05:09:48.415 INFO:tasks.workunit.client.0.smithi042.stdout:removed './289/21' 2019-12-05T05:09:48.416 INFO:tasks.workunit.client.0.smithi042.stdout:removed './289/27' 2019-12-05T05:09:48.416 INFO:tasks.workunit.client.0.smithi042.stdout:removed './289/31' 2019-12-05T05:09:48.416 INFO:tasks.workunit.client.0.smithi042.stdout:removed './289/36' 2019-12-05T05:09:48.416 INFO:tasks.workunit.client.0.smithi042.stdout:removed './289/20' 2019-12-05T05:09:48.416 INFO:tasks.workunit.client.0.smithi042.stdout:removed './289/39' 2019-12-05T05:09:48.417 INFO:tasks.workunit.client.0.smithi042.stdout:removed './289/34' 2019-12-05T05:09:48.417 INFO:tasks.workunit.client.0.smithi042.stdout:removed './289/38' 2019-12-05T05:09:48.417 INFO:tasks.workunit.client.0.smithi042.stdout:removed './289/28' 2019-12-05T05:09:48.417 INFO:tasks.workunit.client.0.smithi042.stdout:removed './289/24' 2019-12-05T05:09:48.417 INFO:tasks.workunit.client.0.smithi042.stdout:removed './289/29' 2019-12-05T05:09:48.417 INFO:tasks.workunit.client.0.smithi042.stdout:removed './289/32' 2019-12-05T05:09:48.417 INFO:tasks.workunit.client.0.smithi042.stdout:removed './289/22' 2019-12-05T05:09:48.418 INFO:tasks.workunit.client.0.smithi042.stdout:removed './289/25' 2019-12-05T05:09:48.418 INFO:tasks.workunit.client.0.smithi042.stdout:removed './289/30' 2019-12-05T05:09:48.418 INFO:tasks.workunit.client.0.smithi042.stdout:removed './289/33' 2019-12-05T05:09:48.418 INFO:tasks.workunit.client.0.smithi042.stdout:removed './289/23' 2019-12-05T05:09:48.418 INFO:tasks.workunit.client.0.smithi042.stdout:removed './289/35' 2019-12-05T05:09:48.418 INFO:tasks.workunit.client.0.smithi042.stdout:removed './289/26' 2019-12-05T05:09:48.418 INFO:tasks.workunit.client.0.smithi042.stdout:removed './289/37' 2019-12-05T05:09:48.419 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './289' 2019-12-05T05:09:48.419 INFO:tasks.workunit.client.0.smithi042.stdout:removed './290/21' 2019-12-05T05:09:48.419 INFO:tasks.workunit.client.0.smithi042.stdout:removed './290/27' 2019-12-05T05:09:48.419 INFO:tasks.workunit.client.0.smithi042.stdout:removed './290/31' 2019-12-05T05:09:48.419 INFO:tasks.workunit.client.0.smithi042.stdout:removed './290/36' 2019-12-05T05:09:48.419 INFO:tasks.workunit.client.0.smithi042.stdout:removed './290/20' 2019-12-05T05:09:48.419 INFO:tasks.workunit.client.0.smithi042.stdout:removed './290/39' 2019-12-05T05:09:48.419 INFO:tasks.workunit.client.0.smithi042.stdout:removed './290/34' 2019-12-05T05:09:48.420 INFO:tasks.workunit.client.0.smithi042.stdout:removed './290/38' 2019-12-05T05:09:48.420 INFO:tasks.workunit.client.0.smithi042.stdout:removed './290/28' 2019-12-05T05:09:48.420 INFO:tasks.workunit.client.0.smithi042.stdout:removed './290/24' 2019-12-05T05:09:48.420 INFO:tasks.workunit.client.0.smithi042.stdout:removed './290/29' 2019-12-05T05:09:48.420 INFO:tasks.workunit.client.0.smithi042.stdout:removed './290/32' 2019-12-05T05:09:48.420 INFO:tasks.workunit.client.0.smithi042.stdout:removed './290/22' 2019-12-05T05:09:48.420 INFO:tasks.workunit.client.0.smithi042.stdout:removed './290/25' 2019-12-05T05:09:48.421 INFO:tasks.workunit.client.0.smithi042.stdout:removed './290/30' 2019-12-05T05:09:48.421 INFO:tasks.workunit.client.0.smithi042.stdout:removed './290/33' 2019-12-05T05:09:48.421 INFO:tasks.workunit.client.0.smithi042.stdout:removed './290/23' 2019-12-05T05:09:48.421 INFO:tasks.workunit.client.0.smithi042.stdout:removed './290/35' 2019-12-05T05:09:48.421 INFO:tasks.workunit.client.0.smithi042.stdout:removed './290/26' 2019-12-05T05:09:48.421 INFO:tasks.workunit.client.0.smithi042.stdout:removed './290/37' 2019-12-05T05:09:48.421 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './290' 2019-12-05T05:09:48.422 INFO:tasks.workunit.client.0.smithi042.stdout:removed './291/21' 2019-12-05T05:09:48.422 INFO:tasks.workunit.client.0.smithi042.stdout:removed './291/27' 2019-12-05T05:09:48.422 INFO:tasks.workunit.client.0.smithi042.stdout:removed './291/31' 2019-12-05T05:09:48.422 INFO:tasks.workunit.client.0.smithi042.stdout:removed './291/36' 2019-12-05T05:09:48.422 INFO:tasks.workunit.client.0.smithi042.stdout:removed './291/20' 2019-12-05T05:09:48.422 INFO:tasks.workunit.client.0.smithi042.stdout:removed './291/39' 2019-12-05T05:09:48.422 INFO:tasks.workunit.client.0.smithi042.stdout:removed './291/34' 2019-12-05T05:09:48.422 INFO:tasks.workunit.client.0.smithi042.stdout:removed './291/38' 2019-12-05T05:09:48.423 INFO:tasks.workunit.client.0.smithi042.stdout:removed './291/28' 2019-12-05T05:09:48.423 INFO:tasks.workunit.client.0.smithi042.stdout:removed './291/24' 2019-12-05T05:09:48.423 INFO:tasks.workunit.client.0.smithi042.stdout:removed './291/29' 2019-12-05T05:09:48.423 INFO:tasks.workunit.client.0.smithi042.stdout:removed './291/32' 2019-12-05T05:09:48.424 INFO:tasks.workunit.client.0.smithi042.stdout:removed './291/22' 2019-12-05T05:09:48.424 INFO:tasks.workunit.client.0.smithi042.stdout:removed './291/25' 2019-12-05T05:09:48.424 INFO:tasks.workunit.client.0.smithi042.stdout:removed './291/30' 2019-12-05T05:09:48.424 INFO:tasks.workunit.client.0.smithi042.stdout:removed './291/33' 2019-12-05T05:09:48.424 INFO:tasks.workunit.client.0.smithi042.stdout:removed './291/23' 2019-12-05T05:09:48.424 INFO:tasks.workunit.client.0.smithi042.stdout:removed './291/35' 2019-12-05T05:09:48.424 INFO:tasks.workunit.client.0.smithi042.stdout:removed './291/26' 2019-12-05T05:09:48.425 INFO:tasks.workunit.client.0.smithi042.stdout:removed './291/37' 2019-12-05T05:09:48.425 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './291' 2019-12-05T05:09:48.425 INFO:tasks.workunit.client.0.smithi042.stdout:removed './292/21' 2019-12-05T05:09:48.425 INFO:tasks.workunit.client.0.smithi042.stdout:removed './292/27' 2019-12-05T05:09:48.425 INFO:tasks.workunit.client.0.smithi042.stdout:removed './292/31' 2019-12-05T05:09:48.425 INFO:tasks.workunit.client.0.smithi042.stdout:removed './292/36' 2019-12-05T05:09:48.425 INFO:tasks.workunit.client.0.smithi042.stdout:removed './292/20' 2019-12-05T05:09:48.426 INFO:tasks.workunit.client.0.smithi042.stdout:removed './292/39' 2019-12-05T05:09:48.426 INFO:tasks.workunit.client.0.smithi042.stdout:removed './292/34' 2019-12-05T05:09:48.426 INFO:tasks.workunit.client.0.smithi042.stdout:removed './292/38' 2019-12-05T05:09:48.426 INFO:tasks.workunit.client.0.smithi042.stdout:removed './292/28' 2019-12-05T05:09:48.426 INFO:tasks.workunit.client.0.smithi042.stdout:removed './292/24' 2019-12-05T05:09:48.426 INFO:tasks.workunit.client.0.smithi042.stdout:removed './292/29' 2019-12-05T05:09:48.426 INFO:tasks.workunit.client.0.smithi042.stdout:removed './292/32' 2019-12-05T05:09:48.427 INFO:tasks.workunit.client.0.smithi042.stdout:removed './292/22' 2019-12-05T05:09:48.427 INFO:tasks.workunit.client.0.smithi042.stdout:removed './292/25' 2019-12-05T05:09:48.427 INFO:tasks.workunit.client.0.smithi042.stdout:removed './292/30' 2019-12-05T05:09:48.427 INFO:tasks.workunit.client.0.smithi042.stdout:removed './292/33' 2019-12-05T05:09:48.428 INFO:tasks.workunit.client.0.smithi042.stdout:removed './292/23' 2019-12-05T05:09:48.428 INFO:tasks.workunit.client.0.smithi042.stdout:removed './292/35' 2019-12-05T05:09:48.428 INFO:tasks.workunit.client.0.smithi042.stdout:removed './292/26' 2019-12-05T05:09:48.428 INFO:tasks.workunit.client.0.smithi042.stdout:removed './292/37' 2019-12-05T05:09:48.428 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './292' 2019-12-05T05:09:48.428 INFO:tasks.workunit.client.0.smithi042.stdout:removed './293/21' 2019-12-05T05:09:48.428 INFO:tasks.workunit.client.0.smithi042.stdout:removed './293/27' 2019-12-05T05:09:48.429 INFO:tasks.workunit.client.0.smithi042.stdout:removed './293/31' 2019-12-05T05:09:48.429 INFO:tasks.workunit.client.0.smithi042.stdout:removed './293/36' 2019-12-05T05:09:48.429 INFO:tasks.workunit.client.0.smithi042.stdout:removed './293/20' 2019-12-05T05:09:48.429 INFO:tasks.workunit.client.0.smithi042.stdout:removed './293/39' 2019-12-05T05:09:48.429 INFO:tasks.workunit.client.0.smithi042.stdout:removed './293/34' 2019-12-05T05:09:48.429 INFO:tasks.workunit.client.0.smithi042.stdout:removed './293/38' 2019-12-05T05:09:48.430 INFO:tasks.workunit.client.0.smithi042.stdout:removed './293/28' 2019-12-05T05:09:48.430 INFO:tasks.workunit.client.0.smithi042.stdout:removed './293/24' 2019-12-05T05:09:48.430 INFO:tasks.workunit.client.0.smithi042.stdout:removed './293/29' 2019-12-05T05:09:48.430 INFO:tasks.workunit.client.0.smithi042.stdout:removed './293/32' 2019-12-05T05:09:48.430 INFO:tasks.workunit.client.0.smithi042.stdout:removed './293/22' 2019-12-05T05:09:48.430 INFO:tasks.workunit.client.0.smithi042.stdout:removed './293/25' 2019-12-05T05:09:48.430 INFO:tasks.workunit.client.0.smithi042.stdout:removed './293/30' 2019-12-05T05:09:48.431 INFO:tasks.workunit.client.0.smithi042.stdout:removed './293/33' 2019-12-05T05:09:48.431 INFO:tasks.workunit.client.0.smithi042.stdout:removed './293/23' 2019-12-05T05:09:48.431 INFO:tasks.workunit.client.0.smithi042.stdout:removed './293/35' 2019-12-05T05:09:48.431 INFO:tasks.workunit.client.0.smithi042.stdout:removed './293/26' 2019-12-05T05:09:48.431 INFO:tasks.workunit.client.0.smithi042.stdout:removed './293/37' 2019-12-05T05:09:48.431 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './293' 2019-12-05T05:09:48.431 INFO:tasks.workunit.client.0.smithi042.stdout:removed './294/21' 2019-12-05T05:09:48.432 INFO:tasks.workunit.client.0.smithi042.stdout:removed './294/27' 2019-12-05T05:09:48.432 INFO:tasks.workunit.client.0.smithi042.stdout:removed './294/31' 2019-12-05T05:09:48.432 INFO:tasks.workunit.client.0.smithi042.stdout:removed './294/36' 2019-12-05T05:09:48.432 INFO:tasks.workunit.client.0.smithi042.stdout:removed './294/20' 2019-12-05T05:09:48.432 INFO:tasks.workunit.client.0.smithi042.stdout:removed './294/39' 2019-12-05T05:09:48.432 INFO:tasks.workunit.client.0.smithi042.stdout:removed './294/34' 2019-12-05T05:09:48.432 INFO:tasks.workunit.client.0.smithi042.stdout:removed './294/38' 2019-12-05T05:09:48.433 INFO:tasks.workunit.client.0.smithi042.stdout:removed './294/28' 2019-12-05T05:09:48.433 INFO:tasks.workunit.client.0.smithi042.stdout:removed './294/24' 2019-12-05T05:09:48.433 INFO:tasks.workunit.client.0.smithi042.stdout:removed './294/29' 2019-12-05T05:09:48.433 INFO:tasks.workunit.client.0.smithi042.stdout:removed './294/32' 2019-12-05T05:09:48.433 INFO:tasks.workunit.client.0.smithi042.stdout:removed './294/22' 2019-12-05T05:09:48.434 INFO:tasks.workunit.client.0.smithi042.stdout:removed './294/25' 2019-12-05T05:09:48.434 INFO:tasks.workunit.client.0.smithi042.stdout:removed './294/30' 2019-12-05T05:09:48.434 INFO:tasks.workunit.client.0.smithi042.stdout:removed './294/33' 2019-12-05T05:09:48.434 INFO:tasks.workunit.client.0.smithi042.stdout:removed './294/23' 2019-12-05T05:09:48.434 INFO:tasks.workunit.client.0.smithi042.stdout:removed './294/35' 2019-12-05T05:09:48.434 INFO:tasks.workunit.client.0.smithi042.stdout:removed './294/26' 2019-12-05T05:09:48.434 INFO:tasks.workunit.client.0.smithi042.stdout:removed './294/37' 2019-12-05T05:09:48.435 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './294' 2019-12-05T05:09:48.435 INFO:tasks.workunit.client.0.smithi042.stdout:removed './295/21' 2019-12-05T05:09:48.435 INFO:tasks.workunit.client.0.smithi042.stdout:removed './295/27' 2019-12-05T05:09:48.435 INFO:tasks.workunit.client.0.smithi042.stdout:removed './295/31' 2019-12-05T05:09:48.435 INFO:tasks.workunit.client.0.smithi042.stdout:removed './295/36' 2019-12-05T05:09:48.435 INFO:tasks.workunit.client.0.smithi042.stdout:removed './295/20' 2019-12-05T05:09:48.436 INFO:tasks.workunit.client.0.smithi042.stdout:removed './295/39' 2019-12-05T05:09:48.436 INFO:tasks.workunit.client.0.smithi042.stdout:removed './295/34' 2019-12-05T05:09:48.436 INFO:tasks.workunit.client.0.smithi042.stdout:removed './295/38' 2019-12-05T05:09:48.436 INFO:tasks.workunit.client.0.smithi042.stdout:removed './295/28' 2019-12-05T05:09:48.436 INFO:tasks.workunit.client.0.smithi042.stdout:removed './295/24' 2019-12-05T05:09:48.436 INFO:tasks.workunit.client.0.smithi042.stdout:removed './295/29' 2019-12-05T05:09:48.436 INFO:tasks.workunit.client.0.smithi042.stdout:removed './295/32' 2019-12-05T05:09:48.437 INFO:tasks.workunit.client.0.smithi042.stdout:removed './295/22' 2019-12-05T05:09:48.437 INFO:tasks.workunit.client.0.smithi042.stdout:removed './295/25' 2019-12-05T05:09:48.437 INFO:tasks.workunit.client.0.smithi042.stdout:removed './295/30' 2019-12-05T05:09:48.437 INFO:tasks.workunit.client.0.smithi042.stdout:removed './295/33' 2019-12-05T05:09:48.437 INFO:tasks.workunit.client.0.smithi042.stdout:removed './295/23' 2019-12-05T05:09:48.438 INFO:tasks.workunit.client.0.smithi042.stdout:removed './295/35' 2019-12-05T05:09:48.438 INFO:tasks.workunit.client.0.smithi042.stdout:removed './295/26' 2019-12-05T05:09:48.438 INFO:tasks.workunit.client.0.smithi042.stdout:removed './295/37' 2019-12-05T05:09:48.438 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './295' 2019-12-05T05:09:48.438 INFO:tasks.workunit.client.0.smithi042.stdout:removed './296/21' 2019-12-05T05:09:48.439 INFO:tasks.workunit.client.0.smithi042.stdout:removed './296/27' 2019-12-05T05:09:48.439 INFO:tasks.workunit.client.0.smithi042.stdout:removed './296/31' 2019-12-05T05:09:48.439 INFO:tasks.workunit.client.0.smithi042.stdout:removed './296/36' 2019-12-05T05:09:48.439 INFO:tasks.workunit.client.0.smithi042.stdout:removed './296/20' 2019-12-05T05:09:48.439 INFO:tasks.workunit.client.0.smithi042.stdout:removed './296/39' 2019-12-05T05:09:48.439 INFO:tasks.workunit.client.0.smithi042.stdout:removed './296/34' 2019-12-05T05:09:48.439 INFO:tasks.workunit.client.0.smithi042.stdout:removed './296/38' 2019-12-05T05:09:48.440 INFO:tasks.workunit.client.0.smithi042.stdout:removed './296/28' 2019-12-05T05:09:48.440 INFO:tasks.workunit.client.0.smithi042.stdout:removed './296/24' 2019-12-05T05:09:48.440 INFO:tasks.workunit.client.0.smithi042.stdout:removed './296/29' 2019-12-05T05:09:48.440 INFO:tasks.workunit.client.0.smithi042.stdout:removed './296/32' 2019-12-05T05:09:48.440 INFO:tasks.workunit.client.0.smithi042.stdout:removed './296/22' 2019-12-05T05:09:48.440 INFO:tasks.workunit.client.0.smithi042.stdout:removed './296/25' 2019-12-05T05:09:48.440 INFO:tasks.workunit.client.0.smithi042.stdout:removed './296/30' 2019-12-05T05:09:48.441 INFO:tasks.workunit.client.0.smithi042.stdout:removed './296/33' 2019-12-05T05:09:48.441 INFO:tasks.workunit.client.0.smithi042.stdout:removed './296/23' 2019-12-05T05:09:48.441 INFO:tasks.workunit.client.0.smithi042.stdout:removed './296/35' 2019-12-05T05:09:48.441 INFO:tasks.workunit.client.0.smithi042.stdout:removed './296/26' 2019-12-05T05:09:48.441 INFO:tasks.workunit.client.0.smithi042.stdout:removed './296/37' 2019-12-05T05:09:48.441 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './296' 2019-12-05T05:09:48.443 INFO:tasks.workunit.client.0.smithi042.stdout:removed './297/21' 2019-12-05T05:09:48.443 INFO:tasks.workunit.client.0.smithi042.stdout:removed './297/27' 2019-12-05T05:09:48.726 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:09:48.814 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:48.805+0000 7f603f15e700 1 Processor -- start 2019-12-05T05:09:48.814 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:48.805+0000 7f603f15e700 1 -- start start 2019-12-05T05:09:48.815 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:48.805+0000 7f603f15e700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f60380a7740 0x7f603808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:09:48.815 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:48.805+0000 7f603f15e700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f603809fae0 0x7f603807eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:09:48.815 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:48.805+0000 7f603f15e700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60380a0710 con 0x7f603807e720 2019-12-05T05:09:48.815 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:48.805+0000 7f603f15e700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60380a0b00 con 0x7f60380a7740 2019-12-05T05:09:48.815 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:48.805+0000 7f603f15e700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f603808e5b0 con 0x7f603809fae0 2019-12-05T05:09:48.815 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:48.805+0000 7f603d6fb700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f60380a7740 0x7f603808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:09:48.815 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:48.805+0000 7f603cefa700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f603807e720 0x7f603809f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:38164/0 (socket says 172.21.15.42:38164) 2019-12-05T05:09:48.816 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:48.805+0000 7f603cefa700 1 -- 172.21.15.42:0/3233676138 learned_addr learned my addr 172.21.15.42:0/3233676138 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:09:48.816 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:48.805+0000 7f603defc700 1 --2- 172.21.15.42:0/3233676138 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f603809fae0 0x7f603807eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:09:48.816 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:48.805+0000 7f602ffff700 1 -- 172.21.15.42:0/3233676138 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 473780042 0 0) 0x7f60380a0710 con 0x7f603807e720 2019-12-05T05:09:48.816 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:48.805+0000 7f602ffff700 1 -- 172.21.15.42:0/3233676138 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60240030b0 con 0x7f603807e720 2019-12-05T05:09:48.816 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:48.805+0000 7f603d6fb700 1 -- 172.21.15.42:0/3233676138 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f603809fae0 msgr2=0x7f603807eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:09:48.816 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:48.805+0000 7f603d6fb700 1 --2- 172.21.15.42:0/3233676138 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f603809fae0 0x7f603807eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:09:48.817 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:48.805+0000 7f603d6fb700 1 -- 172.21.15.42:0/3233676138 >> v1:172.21.15.40:6789/0 conn(0x7f603807e720 legacy=0x7f603809f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:09:48.817 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:48.805+0000 7f603d6fb700 1 -- 172.21.15.42:0/3233676138 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f603807f0a0 con 0x7f60380a7740 2019-12-05T05:09:48.817 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:48.809+0000 7f603d6fb700 1 --2- 172.21.15.42:0/3233676138 >> v2:172.21.15.40:3300/0 conn(0x7f60380a7740 0x7f603808df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f6034008790 tx=0x7f603400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:48.817 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:48.809+0000 7f603d6fb700 1 --2- 172.21.15.42:0/3233676138 >> v2:172.21.15.40:3300/0 conn(0x7f60380a7740 0x7f603808df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f6034008790 tx=0x7f603400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:48.817 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:48.809+0000 7f603d6fb700 1 --2- 172.21.15.42:0/3233676138 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f60380a7740 0x7f603808df80 secure :-1 s=READY pgs=291 cs=0 l=1 rx=0x7f6034008790 tx=0x7f603400fc70).ready entity=mon.0 client_cookie=524d3dda50fc0bae server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:09:48.817 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:48.809+0000 7f603d6fb700 1 --2- 172.21.15.42:0/3233676138 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f60380a7740 0x7f603808df80 secure :-1 s=THROTTLE_DONE pgs=291 cs=0 l=1 rx=0x7f6034008790 tx=0x7f603400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:48.817 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:48.809+0000 7f602ffff700 1 -- 172.21.15.42:0/3233676138 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6034008bb0 con 0x7f60380a7740 2019-12-05T05:09:48.818 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:48.809+0000 7f603d6fb700 1 --2- 172.21.15.42:0/3233676138 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f60380a7740 0x7f603808df80 secure :-1 s=THROTTLE_DONE pgs=291 cs=0 l=1 rx=0x7f6034008790 tx=0x7f603400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:48.818 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:48.809+0000 7f603d6fb700 1 --2- 172.21.15.42:0/3233676138 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f60380a7740 0x7f603808df80 secure :-1 s=THROTTLE_DONE pgs=291 cs=0 l=1 rx=0x7f6034008790 tx=0x7f603400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:48.818 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:48.809+0000 7f602ffff700 1 -- 172.21.15.42:0/3233676138 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60340087d0 con 0x7f60380a7740 2019-12-05T05:09:48.818 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:48.809+0000 7f602ffff700 1 -- 172.21.15.42:0/3233676138 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6034011860 con 0x7f60380a7740 2019-12-05T05:09:48.818 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:48.809+0000 7f603f15e700 1 -- 172.21.15.42:0/3233676138 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f60380a7740 msgr2=0x7f603808df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:09:48.818 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:48.809+0000 7f603f15e700 1 --2- 172.21.15.42:0/3233676138 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f60380a7740 0x7f603808df80 secure :-1 s=READY pgs=291 cs=0 l=1 rx=0x7f6034008790 tx=0x7f603400fc70).stop 2019-12-05T05:09:48.818 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:48.809+0000 7f603f15e700 1 -- 172.21.15.42:0/3233676138 shutdown_connections 2019-12-05T05:09:48.819 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:48.809+0000 7f603f15e700 1 --2- 172.21.15.42:0/3233676138 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f603809fae0 0x7f603807eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:09:48.819 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:48.809+0000 7f603f15e700 1 --2- 172.21.15.42:0/3233676138 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f60380a7740 0x7f603808df80 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:09:48.819 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:48.809+0000 7f603f15e700 1 -- 172.21.15.42:0/3233676138 shutdown_connections 2019-12-05T05:09:48.819 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:48.809+0000 7f603f15e700 1 -- 172.21.15.42:0/3233676138 wait complete. 2019-12-05T05:09:48.819 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:48.809+0000 7f603f15e700 1 -- 172.21.15.42:0/3233676138 >> 172.21.15.42:0/3233676138 conn(0x7f60380aabe0 msgr2=0x7f60380a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:09:48.819 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:48.809+0000 7f603f15e700 1 Processor -- start 2019-12-05T05:09:48.819 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:48.809+0000 7f603f15e700 1 -- start start 2019-12-05T05:09:48.823 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:48.809+0000 7f603f15e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6038124370 0x7f6038121570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:09:48.823 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:48.809+0000 7f603f15e700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6038122af0 con 0x7f603811a770 2019-12-05T05:09:48.823 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:48.809+0000 7f603f15e700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60381393b0 con 0x7f60381226c0 2019-12-05T05:09:48.824 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:48.809+0000 7f603f15e700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6038122e00 con 0x7f6038124370 2019-12-05T05:09:48.824 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:48.809+0000 7f603d6fb700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6038124370 0x7f6038121570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:09:48.824 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:48.809+0000 7f603d6fb700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6038124370 0x7f6038121570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:56920/0 (socket says 172.21.15.42:56920) 2019-12-05T05:09:48.824 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:48.809+0000 7f603d6fb700 1 -- 172.21.15.42:0/3471772576 learned_addr learned my addr 172.21.15.42:0/3471772576 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:09:48.825 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:48.813+0000 7f603d6fb700 1 -- 172.21.15.42:0/3471772576 >> v1:172.21.15.42:6789/0 conn(0x7f60381226c0 legacy=0x7f603807daf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:09:48.825 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:48.813+0000 7f603d6fb700 1 -- 172.21.15.42:0/3471772576 >> v1:172.21.15.40:6789/0 conn(0x7f603811a770 legacy=0x7f603811aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:09:48.825 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:48.813+0000 7f603d6fb700 1 -- 172.21.15.42:0/3471772576 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f603813a5f0 con 0x7f6038124370 2019-12-05T05:09:48.825 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:48.813+0000 7f603d6fb700 1 --2- 172.21.15.42:0/3471772576 >> v2:172.21.15.42:3300/0 conn(0x7f6038124370 0x7f6038121570 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f6030008190 tx=0x7f6030007aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:48.825 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:48.813+0000 7f603d6fb700 1 --2- 172.21.15.42:0/3471772576 >> v2:172.21.15.42:3300/0 conn(0x7f6038124370 0x7f6038121570 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f6030008190 tx=0x7f6030007aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:48.825 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:48.813+0000 7f603d6fb700 1 --2- 172.21.15.42:0/3471772576 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6038124370 0x7f6038121570 secure :-1 s=READY pgs=367 cs=0 l=1 rx=0x7f6030008190 tx=0x7f6030007aa0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:09:48.826 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:48.813+0000 7f603d6fb700 1 --2- 172.21.15.42:0/3471772576 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6038124370 0x7f6038121570 secure :-1 s=THROTTLE_DONE pgs=367 cs=0 l=1 rx=0x7f6030008190 tx=0x7f6030007aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:48.826 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:48.813+0000 7f603d6fb700 1 --2- 172.21.15.42:0/3471772576 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6038124370 0x7f6038121570 secure :-1 s=THROTTLE_DONE pgs=367 cs=0 l=1 rx=0x7f6030008190 tx=0x7f6030007aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:48.826 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:48.813+0000 7f603d6fb700 1 --2- 172.21.15.42:0/3471772576 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6038124370 0x7f6038121570 secure :-1 s=THROTTLE_DONE pgs=367 cs=0 l=1 rx=0x7f6030008190 tx=0x7f6030007aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:48.826 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:48.813+0000 7f602dffb700 1 -- 172.21.15.42:0/3471772576 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f60340087d0 con 0x7f6038124370 2019-12-05T05:09:48.827 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:48.813+0000 7f602dffb700 1 -- 172.21.15.42:0/3471772576 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6034008db0 con 0x7f6038124370 2019-12-05T05:09:48.827 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:48.813+0000 7f603f15e700 1 -- 172.21.15.42:0/3471772576 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60381395e0 con 0x7f6038124370 2019-12-05T05:09:48.827 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:48.813+0000 7f602dffb700 1 -- 172.21.15.42:0/3471772576 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6034005540 con 0x7f6038124370 2019-12-05T05:09:48.827 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:48.813+0000 7f603f15e700 1 -- 172.21.15.42:0/3471772576 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60381399f0 con 0x7f6038124370 2019-12-05T05:09:48.828 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:48.813+0000 7f603d6fb700 1 --2- 172.21.15.42:0/3471772576 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6038124370 0x7f6038121570 secure :-1 s=THROTTLE_DONE pgs=367 cs=0 l=1 rx=0x7f6030008190 tx=0x7f6030007aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:48.828 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:48.813+0000 7f602dffb700 1 -- 172.21.15.42:0/3471772576 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f6034005810 con 0x7f6038124370 2019-12-05T05:09:48.828 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:48.813+0000 7f603d6fb700 1 --2- 172.21.15.42:0/3471772576 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6038124370 0x7f6038121570 secure :-1 s=THROTTLE_DONE pgs=367 cs=0 l=1 rx=0x7f6030008190 tx=0x7f6030007aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:48.828 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:48.813+0000 7f602dffb700 1 --2- 172.21.15.42:0/3471772576 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f601803ae30 0x7f601803d340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:09:48.828 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:48.813+0000 7f602dffb700 1 -- 172.21.15.42:0/3471772576 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6289+0+0 (secure 0 0 0) 0x7f603404b3e0 con 0x7f6038124370 2019-12-05T05:09:48.828 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:48.813+0000 7f603cefa700 1 --2- 172.21.15.42:0/3471772576 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f601803ae30 0x7f601803d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:09:48.828 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:48.813+0000 7f603cefa700 1 --2- 172.21.15.42:0/3471772576 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f601803ae30 0x7f601803d340 crc :-1 s=READY pgs=167 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:09:48.829 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:48.821+0000 7f603f15e700 1 -- 172.21.15.42:0/3471772576 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f603805d110 con 0x7f6038124370 2019-12-05T05:09:48.829 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:48.825+0000 7f603d6fb700 1 --2- 172.21.15.42:0/3471772576 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6038124370 0x7f6038121570 secure :-1 s=THROTTLE_DONE pgs=367 cs=0 l=1 rx=0x7f6030008190 tx=0x7f6030007aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:48.830 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:48.825+0000 7f602dffb700 1 -- 172.21.15.42:0/3471772576 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f6034032460 con 0x7f6038124370 2019-12-05T05:09:48.994 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:48.989+0000 7f603f15e700 1 -- 172.21.15.42:0/3471772576 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f603805b8c0 con 0x7f6038124370 2019-12-05T05:09:48.994 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:48.989+0000 7f603d6fb700 1 --2- 172.21.15.42:0/3471772576 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6038124370 0x7f6038121570 secure :-1 s=THROTTLE_DONE pgs=367 cs=0 l=1 rx=0x7f6030008190 tx=0x7f6030007aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:48.995 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:48.989+0000 7f602dffb700 1 -- 172.21.15.42:0/3471772576 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+2105 (secure 0 0 0) 0x7f60340326c0 con 0x7f6038124370 2019-12-05T05:09:48.995 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 35 2019-12-05T05:09:48.995 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:09:48.996 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4595,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/211584380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":211584380},{"type":"v1","addr":"172.21.15.40:6833","nonce":211584380}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":35}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:09:28.296949+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4978},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4978":{"gid":4978,"name":"a","rank":0,"incarnation":30,"state":"up:active","state_seq":36,"addr":"172.21.15.42:6835/3650564018","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3650564018},{"type":"v1","addr":"172.21.15.42:6835","nonce":3650564018}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:09:48.998 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:48.993+0000 7f603f15e700 1 -- 172.21.15.42:0/3471772576 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f601803ae30 msgr2=0x7f601803d340 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:09:48.998 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:48.993+0000 7f603f15e700 1 --2- 172.21.15.42:0/3471772576 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f601803ae30 0x7f601803d340 crc :-1 s=READY pgs=167 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:09:48.998 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:48.993+0000 7f603f15e700 1 -- 172.21.15.42:0/3471772576 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6038124370 msgr2=0x7f6038121570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:09:48.998 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:48.993+0000 7f603f15e700 1 --2- 172.21.15.42:0/3471772576 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6038124370 0x7f6038121570 secure :-1 s=READY pgs=367 cs=0 l=1 rx=0x7f6030008190 tx=0x7f6030007aa0).stop 2019-12-05T05:09:48.999 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:48.993+0000 7f603f15e700 1 -- 172.21.15.42:0/3471772576 shutdown_connections 2019-12-05T05:09:48.999 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:48.993+0000 7f603f15e700 1 --2- 172.21.15.42:0/3471772576 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f601803ae30 0x7f601803d340 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:09:48.999 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:48.993+0000 7f603f15e700 1 --2- 172.21.15.42:0/3471772576 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6038124370 0x7f6038121570 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:09:48.999 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:48.993+0000 7f603f15e700 1 -- 172.21.15.42:0/3471772576 shutdown_connections 2019-12-05T05:09:48.999 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:48.993+0000 7f603f15e700 1 -- 172.21.15.42:0/3471772576 wait complete. 2019-12-05T05:09:49.000 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:48.993+0000 7f603f15e700 1 -- 172.21.15.42:0/3471772576 >> 172.21.15.42:0/3471772576 conn(0x7f60380aabe0 msgr2=0x7f60380a9640 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:09:49.009 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.a (rank=0) 2019-12-05T05:09:49.009 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2019-12-05T05:09:49.010 INFO:teuthology.orchestra.run:waiting for 300 2019-12-05T05:09:49.025 INFO:tasks.ceph.mds.a:Stopped 2019-12-05T05:09:49.025 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:09:49.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:49.181+0000 7fbfafe6c700 1 Processor -- start 2019-12-05T05:09:49.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:49.181+0000 7fbfafe6c700 1 -- start start 2019-12-05T05:09:49.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:49.181+0000 7fbfafe6c700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbfa80a7740 0x7fbfa808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:09:49.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:49.181+0000 7fbfafe6c700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfa80a0710 con 0x7fbfa809fae0 2019-12-05T05:09:49.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:49.181+0000 7fbfafe6c700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfa80a9ba0 con 0x7fbfa807e720 2019-12-05T05:09:49.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:49.181+0000 7fbfafe6c700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfa80a0b00 con 0x7fbfa80a7740 2019-12-05T05:09:49.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:49.185+0000 7fbfadc08700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fbfa807e720 0x7fbfa809f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:56970/0 (socket says 172.21.15.42:56970) 2019-12-05T05:09:49.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:49.185+0000 7fbfadc08700 1 -- 172.21.15.42:0/3591887133 learned_addr learned my addr 172.21.15.42:0/3591887133 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:09:49.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:49.185+0000 7fbfae409700 1 --2- 172.21.15.42:0/3591887133 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbfa80a7740 0x7fbfa808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:09:49.193 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:49.185+0000 7fbfad407700 1 -- 172.21.15.42:0/3591887133 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2040404983 0 0) 0x7fbfa80a9ba0 con 0x7fbfa807e720 2019-12-05T05:09:49.193 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:49.185+0000 7fbfad407700 1 -- 172.21.15.42:0/3591887133 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf940030b0 con 0x7fbfa807e720 2019-12-05T05:09:49.193 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:49.185+0000 7fbfad407700 1 -- 172.21.15.42:0/3591887133 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 278922325 0 0) 0x7fbfa80a0710 con 0x7fbfa809fae0 2019-12-05T05:09:49.193 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:49.185+0000 7fbfad407700 1 -- 172.21.15.42:0/3591887133 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbfa80a9ba0 con 0x7fbfa809fae0 2019-12-05T05:09:49.193 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:49.185+0000 7fbfae409700 1 -- 172.21.15.42:0/3591887133 >> v1:172.21.15.42:6789/0 conn(0x7fbfa807e720 legacy=0x7fbfa809f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:09:49.193 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:49.185+0000 7fbfae409700 1 -- 172.21.15.42:0/3591887133 >> v1:172.21.15.40:6789/0 conn(0x7fbfa809fae0 legacy=0x7fbfa814efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:09:49.193 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:49.185+0000 7fbfae409700 1 -- 172.21.15.42:0/3591887133 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfa807eb50 con 0x7fbfa80a7740 2019-12-05T05:09:49.194 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:49.185+0000 7fbfae409700 1 --2- 172.21.15.42:0/3591887133 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbfa80a7740 0x7fbfa808df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fbfa40032b0 tx=0x7fbfa400ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:49.194 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:49.185+0000 7fbfae409700 1 --2- 172.21.15.42:0/3591887133 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbfa80a7740 0x7fbfa808df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fbfa40032b0 tx=0x7fbfa400ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:49.194 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:49.185+0000 7fbfae409700 1 --2- 172.21.15.42:0/3591887133 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbfa80a7740 0x7fbfa808df80 secure :-1 s=READY pgs=77 cs=0 l=1 rx=0x7fbfa40032b0 tx=0x7fbfa400ff70).ready entity=mon.2 client_cookie=f4a93225cd95be15 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:09:49.194 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:49.185+0000 7fbfae409700 1 --2- 172.21.15.42:0/3591887133 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbfa80a7740 0x7fbfa808df80 secure :-1 s=THROTTLE_DONE pgs=77 cs=0 l=1 rx=0x7fbfa40032b0 tx=0x7fbfa400ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:49.194 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:49.185+0000 7fbfad407700 1 -- 172.21.15.42:0/3591887133 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbfa4011a10 con 0x7fbfa80a7740 2019-12-05T05:09:49.194 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:49.185+0000 7fbfae409700 1 --2- 172.21.15.42:0/3591887133 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbfa80a7740 0x7fbfa808df80 secure :-1 s=THROTTLE_DONE pgs=77 cs=0 l=1 rx=0x7fbfa40032b0 tx=0x7fbfa400ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:49.194 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:49.185+0000 7fbfae409700 1 --2- 172.21.15.42:0/3591887133 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbfa80a7740 0x7fbfa808df80 secure :-1 s=THROTTLE_DONE pgs=77 cs=0 l=1 rx=0x7fbfa40032b0 tx=0x7fbfa400ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:49.195 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:49.185+0000 7fbfad407700 1 -- 172.21.15.42:0/3591887133 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfa4011630 con 0x7fbfa80a7740 2019-12-05T05:09:49.195 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:49.185+0000 7fbfad407700 1 -- 172.21.15.42:0/3591887133 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbfa400a160 con 0x7fbfa80a7740 2019-12-05T05:09:49.195 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:49.185+0000 7fbfafe6c700 1 -- 172.21.15.42:0/3591887133 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbfa80a7740 msgr2=0x7fbfa808df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:09:49.195 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:49.185+0000 7fbfafe6c700 1 --2- 172.21.15.42:0/3591887133 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbfa80a7740 0x7fbfa808df80 secure :-1 s=READY pgs=77 cs=0 l=1 rx=0x7fbfa40032b0 tx=0x7fbfa400ff70).stop 2019-12-05T05:09:49.195 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:49.185+0000 7fbfafe6c700 1 -- 172.21.15.42:0/3591887133 shutdown_connections 2019-12-05T05:09:49.195 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:49.185+0000 7fbfafe6c700 1 --2- 172.21.15.42:0/3591887133 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbfa80a7740 0x7fbfa808df80 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:09:49.195 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:49.185+0000 7fbfafe6c700 1 -- 172.21.15.42:0/3591887133 shutdown_connections 2019-12-05T05:09:49.196 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:49.185+0000 7fbfafe6c700 1 -- 172.21.15.42:0/3591887133 wait complete. 2019-12-05T05:09:49.196 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:49.185+0000 7fbfafe6c700 1 -- 172.21.15.42:0/3591887133 >> 172.21.15.42:0/3591887133 conn(0x7fbfa80aabe0 msgr2=0x7fbfa80a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:09:49.199 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:49.189+0000 7fbfafe6c700 1 Processor -- start 2019-12-05T05:09:49.199 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:49.189+0000 7fbfafe6c700 1 -- start start 2019-12-05T05:09:49.199 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:49.189+0000 7fbfafe6c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fbfa8142ff0 0x7fbfa8125860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:09:49.200 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:49.189+0000 7fbfafe6c700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfa80a58b0 con 0x7fbfa80a5480 2019-12-05T05:09:49.200 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:49.189+0000 7fbfafe6c700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfa80a6600 con 0x7fbfa807cfb0 2019-12-05T05:09:49.200 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:49.189+0000 7fbfafe6c700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfa8145280 con 0x7fbfa8142ff0 2019-12-05T05:09:49.200 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:49.189+0000 7fbfae409700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fbfa807cfb0 0x7fbfa8144a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:56974/0 (socket says 172.21.15.42:56974) 2019-12-05T05:09:49.200 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:49.189+0000 7fbfae409700 1 -- 172.21.15.42:0/2950248667 learned_addr learned my addr 172.21.15.42:0/2950248667 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:09:49.200 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:49.189+0000 7fbfadc08700 1 --2- 172.21.15.42:0/2950248667 >> v2:172.21.15.42:3300/0 conn(0x7fbfa8142ff0 0x7fbfa8125860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:09:49.201 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:49.189+0000 7fbf9effd700 1 -- 172.21.15.42:0/2950248667 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3380168124 0 0) 0x7fbfa80a6600 con 0x7fbfa807cfb0 2019-12-05T05:09:49.201 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:49.189+0000 7fbf9effd700 1 -- 172.21.15.42:0/2950248667 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf900030b0 con 0x7fbfa807cfb0 2019-12-05T05:09:49.201 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:49.189+0000 7fbf9effd700 1 -- 172.21.15.42:0/2950248667 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3962954985 0 0) 0x7fbfa80a58b0 con 0x7fbfa80a5480 2019-12-05T05:09:49.201 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:49.189+0000 7fbf9effd700 1 -- 172.21.15.42:0/2950248667 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbfa80a6600 con 0x7fbfa80a5480 2019-12-05T05:09:49.201 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:49.189+0000 7fbfadc08700 1 -- 172.21.15.42:0/2950248667 >> v1:172.21.15.42:6789/0 conn(0x7fbfa807cfb0 legacy=0x7fbfa8144a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:09:49.201 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:49.189+0000 7fbfadc08700 1 -- 172.21.15.42:0/2950248667 >> v1:172.21.15.40:6789/0 conn(0x7fbfa80a5480 legacy=0x7fbfa812ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:09:49.201 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:49.189+0000 7fbfadc08700 1 -- 172.21.15.42:0/2950248667 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfa80a6830 con 0x7fbfa8142ff0 2019-12-05T05:09:49.202 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:49.189+0000 7fbfadc08700 1 --2- 172.21.15.42:0/2950248667 >> v2:172.21.15.42:3300/0 conn(0x7fbfa8142ff0 0x7fbfa8125860 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fbfa807f1c0 tx=0x7fbf98010870).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:49.202 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:49.189+0000 7fbfadc08700 1 --2- 172.21.15.42:0/2950248667 >> v2:172.21.15.42:3300/0 conn(0x7fbfa8142ff0 0x7fbfa8125860 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fbfa807f1c0 tx=0x7fbf98010870).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:49.202 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:49.189+0000 7fbfadc08700 1 --2- 172.21.15.42:0/2950248667 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbfa8142ff0 0x7fbfa8125860 secure :-1 s=READY pgs=370 cs=0 l=1 rx=0x7fbfa807f1c0 tx=0x7fbf98010870).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:09:49.202 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:49.189+0000 7fbfadc08700 1 --2- 172.21.15.42:0/2950248667 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbfa8142ff0 0x7fbfa8125860 secure :-1 s=THROTTLE_DONE pgs=370 cs=0 l=1 rx=0x7fbfa807f1c0 tx=0x7fbf98010870).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:49.202 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:49.189+0000 7fbf9effd700 1 -- 172.21.15.42:0/2950248667 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbf98005c30 con 0x7fbfa8142ff0 2019-12-05T05:09:49.202 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:49.189+0000 7fbfafe6c700 1 -- 172.21.15.42:0/2950248667 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfa80a6b00 con 0x7fbfa8142ff0 2019-12-05T05:09:49.203 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:49.189+0000 7fbfadc08700 1 --2- 172.21.15.42:0/2950248667 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbfa8142ff0 0x7fbfa8125860 secure :-1 s=THROTTLE_DONE pgs=370 cs=0 l=1 rx=0x7fbfa807f1c0 tx=0x7fbf98010870).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:49.203 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:49.189+0000 7fbf9effd700 1 -- 172.21.15.42:0/2950248667 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf98005850 con 0x7fbfa8142ff0 2019-12-05T05:09:49.203 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:49.189+0000 7fbfadc08700 1 --2- 172.21.15.42:0/2950248667 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbfa8142ff0 0x7fbfa8125860 secure :-1 s=THROTTLE_DONE pgs=370 cs=0 l=1 rx=0x7fbfa807f1c0 tx=0x7fbf98010870).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:49.203 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:49.193+0000 7fbfafe6c700 1 -- 172.21.15.42:0/2950248667 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfa812e4c0 con 0x7fbfa8142ff0 2019-12-05T05:09:49.203 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:49.193+0000 7fbf9effd700 1 -- 172.21.15.42:0/2950248667 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbf980064a0 con 0x7fbfa8142ff0 2019-12-05T05:09:49.203 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:49.193+0000 7fbfadc08700 1 --2- 172.21.15.42:0/2950248667 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbfa8142ff0 0x7fbfa8125860 secure :-1 s=THROTTLE_DONE pgs=370 cs=0 l=1 rx=0x7fbfa807f1c0 tx=0x7fbf98010870).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:49.203 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:49.193+0000 7fbf9effd700 1 -- 172.21.15.42:0/2950248667 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fbf980066f0 con 0x7fbfa8142ff0 2019-12-05T05:09:49.204 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:49.193+0000 7fbfadc08700 1 --2- 172.21.15.42:0/2950248667 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbfa8142ff0 0x7fbfa8125860 secure :-1 s=THROTTLE_DONE pgs=370 cs=0 l=1 rx=0x7fbfa807f1c0 tx=0x7fbf98010870).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:49.204 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:49.193+0000 7fbf9effd700 1 --2- 172.21.15.42:0/2950248667 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbf9003d630 0x7fbf9003fb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:09:49.204 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:49.193+0000 7fbf9effd700 1 -- 172.21.15.42:0/2950248667 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6289+0+0 (secure 0 0 0) 0x7fbf9804af50 con 0x7fbfa8142ff0 2019-12-05T05:09:49.204 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:49.193+0000 7fbfae409700 1 --2- 172.21.15.42:0/2950248667 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbf9003d630 0x7fbf9003fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:09:49.204 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:49.193+0000 7fbfae409700 1 --2- 172.21.15.42:0/2950248667 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbf9003d630 0x7fbf9003fb40 crc :-1 s=READY pgs=168 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:09:49.205 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:49.197+0000 7fbfafe6c700 1 -- 172.21.15.42:0/2950248667 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfa805d110 con 0x7fbfa8142ff0 2019-12-05T05:09:49.207 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:49.205+0000 7fbfadc08700 1 --2- 172.21.15.42:0/2950248667 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbfa8142ff0 0x7fbfa8125860 secure :-1 s=THROTTLE_DONE pgs=370 cs=0 l=1 rx=0x7fbfa807f1c0 tx=0x7fbf98010870).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:49.208 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:49.205+0000 7fbf9effd700 1 -- 172.21.15.42:0/2950248667 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fbf98031fb0 con 0x7fbfa8142ff0 2019-12-05T05:09:49.375 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:49.369+0000 7fbfafe6c700 1 -- 172.21.15.42:0/2950248667 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbfa805b8c0 con 0x7fbfa8142ff0 2019-12-05T05:09:49.379 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:49.373+0000 7fbfadc08700 1 --2- 172.21.15.42:0/2950248667 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbfa8142ff0 0x7fbfa8125860 secure :-1 s=THROTTLE_DONE pgs=370 cs=0 l=1 rx=0x7fbfa807f1c0 tx=0x7fbf98010870).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:49.379 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:49.373+0000 7fbf9effd700 1 -- 172.21.15.42:0/2950248667 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+2105 (secure 0 0 0) 0x7fbf98032210 con 0x7fbfa8142ff0 2019-12-05T05:09:49.379 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 35 2019-12-05T05:09:49.379 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:49.373+0000 7fbfafe6c700 1 -- 172.21.15.42:0/2950248667 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbf9003d630 msgr2=0x7fbf9003fb40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:09:49.380 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:49.373+0000 7fbfafe6c700 1 --2- 172.21.15.42:0/2950248667 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbf9003d630 0x7fbf9003fb40 crc :-1 s=READY pgs=168 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:09:49.380 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:49.373+0000 7fbfafe6c700 1 -- 172.21.15.42:0/2950248667 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbfa8142ff0 msgr2=0x7fbfa8125860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:09:49.380 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:49.373+0000 7fbfafe6c700 1 --2- 172.21.15.42:0/2950248667 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbfa8142ff0 0x7fbfa8125860 secure :-1 s=READY pgs=370 cs=0 l=1 rx=0x7fbfa807f1c0 tx=0x7fbf98010870).stop 2019-12-05T05:09:49.380 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:49.373+0000 7fbfafe6c700 1 -- 172.21.15.42:0/2950248667 shutdown_connections 2019-12-05T05:09:49.381 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:49.373+0000 7fbfafe6c700 1 --2- 172.21.15.42:0/2950248667 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbf9003d630 0x7fbf9003fb40 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:09:49.381 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:49.373+0000 7fbfafe6c700 1 --2- 172.21.15.42:0/2950248667 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbfa8142ff0 0x7fbfa8125860 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:09:49.381 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:49.373+0000 7fbfafe6c700 1 -- 172.21.15.42:0/2950248667 shutdown_connections 2019-12-05T05:09:49.381 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:49.373+0000 7fbfafe6c700 1 -- 172.21.15.42:0/2950248667 wait complete. 2019-12-05T05:09:49.381 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:49.373+0000 7fbfafe6c700 1 -- 172.21.15.42:0/2950248667 >> 172.21.15.42:0/2950248667 conn(0x7fbfa80a7740 msgr2=0x7fbfa8059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:09:49.381 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:09:49.382 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4595,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/211584380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":211584380},{"type":"v1","addr":"172.21.15.40:6833","nonce":211584380}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":35}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:09:28.296949+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4978},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4978":{"gid":4978,"name":"a","rank":0,"incarnation":30,"state":"up:active","state_seq":36,"addr":"172.21.15.42:6835/3650564018","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3650564018},{"type":"v1","addr":"172.21.15.42:6835","nonce":3650564018}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:09:49.388 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T05:09:51.389 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:09:51.497 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:51.489+0000 7f53f5c07700 1 Processor -- start 2019-12-05T05:09:51.498 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:51.489+0000 7f53f5c07700 1 -- start start 2019-12-05T05:09:51.498 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:51.489+0000 7f53f5c07700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f53f009fae0 0x7f53f014fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:09:51.498 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:51.489+0000 7f53f5c07700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53f00a0710 con 0x7f53f008df80 2019-12-05T05:09:51.498 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:51.489+0000 7f53f5c07700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53f007ef20 con 0x7f53f00a7740 2019-12-05T05:09:51.498 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:51.489+0000 7f53f5c07700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53f00a0b00 con 0x7f53f009fae0 2019-12-05T05:09:51.499 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:51.489+0000 7f53effff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f53f00a7740 0x7f53f007e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:56988/0 (socket says 172.21.15.42:56988) 2019-12-05T05:09:51.499 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:51.489+0000 7f53effff700 1 -- 172.21.15.42:0/84350134 learned_addr learned my addr 172.21.15.42:0/84350134 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:09:51.499 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:51.489+0000 7f53f49a5700 1 --2- 172.21.15.42:0/84350134 >> v2:172.21.15.40:3300/0 conn(0x7f53f009fae0 0x7f53f014fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:09:51.499 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:51.489+0000 7f53eeffd700 1 -- 172.21.15.42:0/84350134 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 205353304 0 0) 0x7f53f007ef20 con 0x7f53f00a7740 2019-12-05T05:09:51.499 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:51.489+0000 7f53eeffd700 1 -- 172.21.15.42:0/84350134 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53dc0030b0 con 0x7f53f00a7740 2019-12-05T05:09:51.499 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:51.489+0000 7f53f49a5700 1 -- 172.21.15.42:0/84350134 >> v1:172.21.15.42:6789/0 conn(0x7f53f00a7740 legacy=0x7f53f007e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:09:51.499 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:51.489+0000 7f53f49a5700 1 -- 172.21.15.42:0/84350134 >> v1:172.21.15.40:6789/0 conn(0x7f53f008df80 legacy=0x7f53f009f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:09:51.500 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:51.489+0000 7f53f49a5700 1 -- 172.21.15.42:0/84350134 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53f00a9ba0 con 0x7f53f009fae0 2019-12-05T05:09:51.500 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:51.489+0000 7f53f49a5700 1 --2- 172.21.15.42:0/84350134 >> v2:172.21.15.40:3300/0 conn(0x7f53f009fae0 0x7f53f014fff0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f53e8008b10 tx=0x7f53e0001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:51.500 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:51.489+0000 7f53f49a5700 1 --2- 172.21.15.42:0/84350134 >> v2:172.21.15.40:3300/0 conn(0x7f53f009fae0 0x7f53f014fff0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f53e8008b10 tx=0x7f53e0001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:51.500 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:51.489+0000 7f53f49a5700 1 --2- 172.21.15.42:0/84350134 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f53f009fae0 0x7f53f014fff0 secure :-1 s=READY pgs=296 cs=0 l=1 rx=0x7f53e8008b10 tx=0x7f53e0001150).ready entity=mon.0 client_cookie=dc17acb63c3cc281 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:09:51.500 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:51.489+0000 7f53f49a5700 1 --2- 172.21.15.42:0/84350134 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f53f009fae0 0x7f53f014fff0 secure :-1 s=THROTTLE_DONE pgs=296 cs=0 l=1 rx=0x7f53e8008b10 tx=0x7f53e0001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:51.500 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:51.489+0000 7f53eeffd700 1 -- 172.21.15.42:0/84350134 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f53e8010870 con 0x7f53f009fae0 2019-12-05T05:09:51.500 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:51.489+0000 7f53f49a5700 1 --2- 172.21.15.42:0/84350134 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f53f009fae0 0x7f53f014fff0 secure :-1 s=THROTTLE_DONE pgs=296 cs=0 l=1 rx=0x7f53e8008b10 tx=0x7f53e0001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:51.501 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:51.489+0000 7f53f49a5700 1 --2- 172.21.15.42:0/84350134 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f53f009fae0 0x7f53f014fff0 secure :-1 s=THROTTLE_DONE pgs=296 cs=0 l=1 rx=0x7f53e8008b10 tx=0x7f53e0001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:51.501 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:51.489+0000 7f53eeffd700 1 -- 172.21.15.42:0/84350134 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53e8008b50 con 0x7f53f009fae0 2019-12-05T05:09:51.501 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:51.489+0000 7f53eeffd700 1 -- 172.21.15.42:0/84350134 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f53e80110c0 con 0x7f53f009fae0 2019-12-05T05:09:51.501 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:51.489+0000 7f53f5c07700 1 -- 172.21.15.42:0/84350134 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f53f009fae0 msgr2=0x7f53f014fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:09:51.501 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:51.489+0000 7f53f5c07700 1 --2- 172.21.15.42:0/84350134 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f53f009fae0 0x7f53f014fff0 secure :-1 s=READY pgs=296 cs=0 l=1 rx=0x7f53e8008b10 tx=0x7f53e0001150).stop 2019-12-05T05:09:51.501 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:51.493+0000 7f53f5c07700 1 -- 172.21.15.42:0/84350134 shutdown_connections 2019-12-05T05:09:51.501 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:51.493+0000 7f53f5c07700 1 --2- 172.21.15.42:0/84350134 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f53f009fae0 0x7f53f014fff0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:09:51.502 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:51.493+0000 7f53f5c07700 1 -- 172.21.15.42:0/84350134 shutdown_connections 2019-12-05T05:09:51.502 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:51.493+0000 7f53f5c07700 1 -- 172.21.15.42:0/84350134 wait complete. 2019-12-05T05:09:51.502 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:51.493+0000 7f53f5c07700 1 -- 172.21.15.42:0/84350134 >> 172.21.15.42:0/84350134 conn(0x7f53f00aabe0 msgr2=0x7f53f00a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:09:51.505 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:51.493+0000 7f53f5c07700 1 Processor -- start 2019-12-05T05:09:51.505 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:51.493+0000 7f53f5c07700 1 -- start start 2019-12-05T05:09:51.505 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:51.497+0000 7f53f5c07700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f53f007b2a0 0x7f53f0112de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:09:51.506 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:51.497+0000 7f53f5c07700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53f01137c0 con 0x7f53f007a0b0 2019-12-05T05:09:51.506 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:51.497+0000 7f53f5c07700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53f01269e0 con 0x7f53f00a3a60 2019-12-05T05:09:51.506 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:51.497+0000 7f53f5c07700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53f0113520 con 0x7f53f007b2a0 2019-12-05T05:09:51.506 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:51.497+0000 7f53effff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f53f00a3a60 0x7f53f0127140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:56996/0 (socket says 172.21.15.42:56996) 2019-12-05T05:09:51.506 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:51.497+0000 7f53effff700 1 -- 172.21.15.42:0/3435522269 learned_addr learned my addr 172.21.15.42:0/3435522269 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:09:51.507 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:51.497+0000 7f53ef7fe700 1 --2- 172.21.15.42:0/3435522269 >> v2:172.21.15.40:3300/0 conn(0x7f53f007b2a0 0x7f53f0112de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:09:51.507 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:51.497+0000 7f53ecff9700 1 -- 172.21.15.42:0/3435522269 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 704671709 0 0) 0x7f53f01269e0 con 0x7f53f00a3a60 2019-12-05T05:09:51.507 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:51.497+0000 7f53ecff9700 1 -- 172.21.15.42:0/3435522269 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53d40030b0 con 0x7f53f00a3a60 2019-12-05T05:09:51.507 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:51.497+0000 7f53ef7fe700 1 -- 172.21.15.42:0/3435522269 >> v1:172.21.15.42:6789/0 conn(0x7f53f00a3a60 legacy=0x7f53f0127140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:09:51.507 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:51.497+0000 7f53ef7fe700 1 -- 172.21.15.42:0/3435522269 >> v1:172.21.15.40:6789/0 conn(0x7f53f007a0b0 legacy=0x7f53f012de30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:09:51.508 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:51.497+0000 7f53ecff9700 1 -- 172.21.15.42:0/3435522269 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1683854120 0 0) 0x7f53f01137c0 con 0x7f53f007a0b0 2019-12-05T05:09:51.508 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:51.497+0000 7f53ef7fe700 1 -- 172.21.15.42:0/3435522269 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53f007a4e0 con 0x7f53f007b2a0 2019-12-05T05:09:51.509 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:51.497+0000 7f53ef7fe700 1 --2- 172.21.15.42:0/3435522269 >> v2:172.21.15.40:3300/0 conn(0x7f53f007b2a0 0x7f53f0112de0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f53e0001510 tx=0x7f53e00109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:51.509 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:51.497+0000 7f53ef7fe700 1 --2- 172.21.15.42:0/3435522269 >> v2:172.21.15.40:3300/0 conn(0x7f53f007b2a0 0x7f53f0112de0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f53e0001510 tx=0x7f53e00109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:51.509 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:51.497+0000 7f53ef7fe700 1 --2- 172.21.15.42:0/3435522269 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f53f007b2a0 0x7f53f0112de0 secure :-1 s=READY pgs=298 cs=0 l=1 rx=0x7f53e0001510 tx=0x7f53e00109a0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:09:51.509 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:51.497+0000 7f53ef7fe700 1 --2- 172.21.15.42:0/3435522269 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f53f007b2a0 0x7f53f0112de0 secure :-1 s=THROTTLE_DONE pgs=298 cs=0 l=1 rx=0x7f53e0001510 tx=0x7f53e00109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:51.509 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:51.497+0000 7f53ecff9700 1 -- 172.21.15.42:0/3435522269 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f53e000ca80 con 0x7f53f007b2a0 2019-12-05T05:09:51.509 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:51.497+0000 7f53ef7fe700 1 --2- 172.21.15.42:0/3435522269 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f53f007b2a0 0x7f53f0112de0 secure :-1 s=THROTTLE_DONE pgs=298 cs=0 l=1 rx=0x7f53e0001510 tx=0x7f53e00109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:51.509 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:51.497+0000 7f53ef7fe700 1 --2- 172.21.15.42:0/3435522269 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f53f007b2a0 0x7f53f0112de0 secure :-1 s=THROTTLE_DONE pgs=298 cs=0 l=1 rx=0x7f53e0001510 tx=0x7f53e00109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:51.510 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:51.497+0000 7f53ecff9700 1 -- 172.21.15.42:0/3435522269 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53e000c6a0 con 0x7f53f007b2a0 2019-12-05T05:09:51.510 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:51.497+0000 7f53f5c07700 1 -- 172.21.15.42:0/3435522269 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53f007a7b0 con 0x7f53f007b2a0 2019-12-05T05:09:51.510 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:51.497+0000 7f53ecff9700 1 -- 172.21.15.42:0/3435522269 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f53e0012210 con 0x7f53f007b2a0 2019-12-05T05:09:51.510 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:51.497+0000 7f53f5c07700 1 -- 172.21.15.42:0/3435522269 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53f012f640 con 0x7f53f007b2a0 2019-12-05T05:09:51.510 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:51.497+0000 7f53ef7fe700 1 --2- 172.21.15.42:0/3435522269 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f53f007b2a0 0x7f53f0112de0 secure :-1 s=THROTTLE_DONE pgs=298 cs=0 l=1 rx=0x7f53e0001510 tx=0x7f53e00109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:51.510 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:51.501+0000 7f53ecff9700 1 -- 172.21.15.42:0/3435522269 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f53e00124e0 con 0x7f53f007b2a0 2019-12-05T05:09:51.510 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:51.501+0000 7f53ef7fe700 1 --2- 172.21.15.42:0/3435522269 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f53f007b2a0 0x7f53f0112de0 secure :-1 s=THROTTLE_DONE pgs=298 cs=0 l=1 rx=0x7f53e0001510 tx=0x7f53e00109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:51.511 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:51.501+0000 7f53ecff9700 1 --2- 172.21.15.42:0/3435522269 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f53d403b260 0x7f53d403d770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:09:51.511 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:51.501+0000 7f53ecff9700 1 -- 172.21.15.42:0/3435522269 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6289+0+0 (secure 0 0 0) 0x7f53e004b1a0 con 0x7f53f007b2a0 2019-12-05T05:09:51.511 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:51.501+0000 7f53effff700 1 --2- 172.21.15.42:0/3435522269 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f53d403b260 0x7f53d403d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:09:51.516 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:51.501+0000 7f53effff700 1 --2- 172.21.15.42:0/3435522269 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f53d403b260 0x7f53d403d770 crc :-1 s=READY pgs=169 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:09:51.517 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:51.501+0000 7f53f5c07700 1 -- 172.21.15.42:0/3435522269 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53f005d110 con 0x7f53f007b2a0 2019-12-05T05:09:51.517 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:51.513+0000 7f53ef7fe700 1 --2- 172.21.15.42:0/3435522269 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f53f007b2a0 0x7f53f0112de0 secure :-1 s=THROTTLE_DONE pgs=298 cs=0 l=1 rx=0x7f53e0001510 tx=0x7f53e00109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:51.517 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:51.513+0000 7f53ecff9700 1 -- 172.21.15.42:0/3435522269 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f53e0032220 con 0x7f53f007b2a0 2019-12-05T05:09:51.686 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:51.681+0000 7f53f5c07700 1 -- 172.21.15.42:0/3435522269 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f53f005b8c0 con 0x7f53f007b2a0 2019-12-05T05:09:51.687 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:51.681+0000 7f53ef7fe700 1 --2- 172.21.15.42:0/3435522269 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f53f007b2a0 0x7f53f0112de0 secure :-1 s=THROTTLE_DONE pgs=298 cs=0 l=1 rx=0x7f53e0001510 tx=0x7f53e00109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:51.687 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:51.681+0000 7f53ecff9700 1 -- 172.21.15.42:0/3435522269 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+2105 (secure 0 0 0) 0x7f53e0032480 con 0x7f53f007b2a0 2019-12-05T05:09:51.689 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 35 2019-12-05T05:09:51.689 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:09:51.689 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4595,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/211584380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":211584380},{"type":"v1","addr":"172.21.15.40:6833","nonce":211584380}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":35}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:09:28.296949+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4978},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4978":{"gid":4978,"name":"a","rank":0,"incarnation":30,"state":"up:active","state_seq":36,"addr":"172.21.15.42:6835/3650564018","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3650564018},{"type":"v1","addr":"172.21.15.42:6835","nonce":3650564018}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:09:51.692 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:51.685+0000 7f53f5c07700 1 -- 172.21.15.42:0/3435522269 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f53d403b260 msgr2=0x7f53d403d770 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:09:51.692 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:51.685+0000 7f53f5c07700 1 --2- 172.21.15.42:0/3435522269 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f53d403b260 0x7f53d403d770 crc :-1 s=READY pgs=169 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:09:51.692 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:51.685+0000 7f53f5c07700 1 -- 172.21.15.42:0/3435522269 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f53f007b2a0 msgr2=0x7f53f0112de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:09:51.692 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:51.685+0000 7f53f5c07700 1 --2- 172.21.15.42:0/3435522269 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f53f007b2a0 0x7f53f0112de0 secure :-1 s=READY pgs=298 cs=0 l=1 rx=0x7f53e0001510 tx=0x7f53e00109a0).stop 2019-12-05T05:09:51.692 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:51.689+0000 7f53f5c07700 1 -- 172.21.15.42:0/3435522269 shutdown_connections 2019-12-05T05:09:51.692 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:51.689+0000 7f53f5c07700 1 --2- 172.21.15.42:0/3435522269 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f53d403b260 0x7f53d403d770 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:09:51.693 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:51.689+0000 7f53f5c07700 1 --2- 172.21.15.42:0/3435522269 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f53f007b2a0 0x7f53f0112de0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:09:51.693 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:51.689+0000 7f53f5c07700 1 -- 172.21.15.42:0/3435522269 shutdown_connections 2019-12-05T05:09:51.693 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:51.689+0000 7f53f5c07700 1 -- 172.21.15.42:0/3435522269 wait complete. 2019-12-05T05:09:51.693 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:51.689+0000 7f53f5c07700 1 -- 172.21.15.42:0/3435522269 >> 172.21.15.42:0/3435522269 conn(0x7f53f0059700 msgr2=0x7f53f007e870 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:09:51.707 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T05:09:53.708 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:09:53.837 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:53.801+0000 7f675bca7700 1 Processor -- start 2019-12-05T05:09:53.837 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:53.801+0000 7f675bca7700 1 -- start start 2019-12-05T05:09:53.838 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:53.801+0000 7f675bca7700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f67540a7730 0x7f675408df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:09:53.838 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:53.801+0000 7f675bca7700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f675407e710 0x7f675407eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:09:53.838 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:53.801+0000 7f675bca7700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67540a0700 con 0x7f675409f2d0 2019-12-05T05:09:53.838 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:53.801+0000 7f675bca7700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67540a0af0 con 0x7f67540a7730 2019-12-05T05:09:53.839 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:53.801+0000 7f675bca7700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f675408e5a0 con 0x7f675407e710 2019-12-05T05:09:53.839 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:53.805+0000 7f675a244700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f67540a7730 0x7f675408df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:09:53.856 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:53.805+0000 7f675a244700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f67540a7730 0x7f675408df70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:56966/0 (socket says 172.21.15.42:56966) 2019-12-05T05:09:53.856 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:53.805+0000 7f675a244700 1 -- 172.21.15.42:0/2428360812 learned_addr learned my addr 172.21.15.42:0/2428360812 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:09:53.857 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:53.805+0000 7f6759a43700 1 --2- 172.21.15.42:0/2428360812 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f675407e710 0x7f675407eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:09:53.857 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:53.805+0000 7f675a244700 1 -- 172.21.15.42:0/2428360812 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f675407e710 msgr2=0x7f675407eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:09:53.857 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:53.805+0000 7f675a244700 1 --2- 172.21.15.42:0/2428360812 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f675407e710 0x7f675407eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:09:53.858 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:53.805+0000 7f675a244700 1 -- 172.21.15.42:0/2428360812 >> v1:172.21.15.40:6789/0 conn(0x7f675409f2d0 legacy=0x7f675409f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:09:53.858 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:53.805+0000 7f675a244700 1 -- 172.21.15.42:0/2428360812 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f675407f090 con 0x7f67540a7730 2019-12-05T05:09:53.858 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:53.805+0000 7f6759a43700 1 --2- 172.21.15.42:0/2428360812 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f675407e710 0x7f675407eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T05:09:53.858 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:53.805+0000 7f675a244700 1 --2- 172.21.15.42:0/2428360812 >> v2:172.21.15.42:3300/0 conn(0x7f67540a7730 0x7f675408df70 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f6750008b10 tx=0x7f67500100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:53.859 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:53.805+0000 7f675a244700 1 --2- 172.21.15.42:0/2428360812 >> v2:172.21.15.42:3300/0 conn(0x7f67540a7730 0x7f675408df70 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f6750008b10 tx=0x7f67500100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:53.859 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:53.805+0000 7f675a244700 1 --2- 172.21.15.42:0/2428360812 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f67540a7730 0x7f675408df70 secure :-1 s=READY pgs=373 cs=0 l=1 rx=0x7f6750008b10 tx=0x7f67500100b0).ready entity=mon.1 client_cookie=286d910140e01a0e server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:09:53.859 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:53.805+0000 7f675a244700 1 --2- 172.21.15.42:0/2428360812 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f67540a7730 0x7f675408df70 secure :-1 s=THROTTLE_DONE pgs=373 cs=0 l=1 rx=0x7f6750008b10 tx=0x7f67500100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:53.859 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:53.805+0000 7f675a244700 1 --2- 172.21.15.42:0/2428360812 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f67540a7730 0x7f675408df70 secure :-1 s=THROTTLE_DONE pgs=373 cs=0 l=1 rx=0x7f6750008b10 tx=0x7f67500100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:53.859 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:53.805+0000 7f675a244700 1 --2- 172.21.15.42:0/2428360812 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f67540a7730 0x7f675408df70 secure :-1 s=THROTTLE_DONE pgs=373 cs=0 l=1 rx=0x7f6750008b10 tx=0x7f67500100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:53.860 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:53.805+0000 7f6759242700 1 -- 172.21.15.42:0/2428360812 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f675000a160 con 0x7f67540a7730 2019-12-05T05:09:53.860 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:53.805+0000 7f6759242700 1 -- 172.21.15.42:0/2428360812 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6750011a30 con 0x7f67540a7730 2019-12-05T05:09:53.869 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:53.805+0000 7f6759242700 1 -- 172.21.15.42:0/2428360812 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f675000a550 con 0x7f67540a7730 2019-12-05T05:09:53.869 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:53.805+0000 7f675bca7700 1 -- 172.21.15.42:0/2428360812 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f67540a7730 msgr2=0x7f675408df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:09:53.870 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:53.805+0000 7f675bca7700 1 --2- 172.21.15.42:0/2428360812 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f67540a7730 0x7f675408df70 secure :-1 s=READY pgs=373 cs=0 l=1 rx=0x7f6750008b10 tx=0x7f67500100b0).stop 2019-12-05T05:09:53.870 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:53.805+0000 7f675bca7700 1 -- 172.21.15.42:0/2428360812 shutdown_connections 2019-12-05T05:09:53.870 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:53.805+0000 7f675bca7700 1 --2- 172.21.15.42:0/2428360812 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f67540a7730 0x7f675408df70 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:09:53.870 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:53.805+0000 7f675bca7700 1 --2- 172.21.15.42:0/2428360812 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f675407e710 0x7f675407eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:09:53.871 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:53.805+0000 7f675bca7700 1 -- 172.21.15.42:0/2428360812 shutdown_connections 2019-12-05T05:09:53.871 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:53.805+0000 7f675bca7700 1 -- 172.21.15.42:0/2428360812 wait complete. 2019-12-05T05:09:53.871 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:53.805+0000 7f675bca7700 1 -- 172.21.15.42:0/2428360812 >> 172.21.15.42:0/2428360812 conn(0x7f67540aabd0 msgr2=0x7f67540a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:09:53.871 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:53.809+0000 7f675bca7700 1 Processor -- start 2019-12-05T05:09:53.871 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:53.809+0000 7f675bca7700 1 -- start start 2019-12-05T05:09:53.872 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:53.809+0000 7f675bca7700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f675408eea0 0x7f67540a27f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:09:53.872 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:53.809+0000 7f675bca7700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67541259f0 con 0x7f675407c910 2019-12-05T05:09:53.872 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:53.809+0000 7f675bca7700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6754125c20 con 0x7f67540a2d40 2019-12-05T05:09:53.872 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:53.809+0000 7f675bca7700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6754124f50 con 0x7f675408eea0 2019-12-05T05:09:53.873 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:53.809+0000 7f6759a43700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f675408eea0 0x7f67540a27f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:09:53.873 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:53.809+0000 7f675aa45700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f67540a2d40 0x7f67541251f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:57012/0 (socket says 172.21.15.42:57012) 2019-12-05T05:09:53.873 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:53.809+0000 7f675aa45700 1 -- 172.21.15.42:0/3152109268 learned_addr learned my addr 172.21.15.42:0/3152109268 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:09:53.873 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:53.809+0000 7f674affd700 1 -- 172.21.15.42:0/3152109268 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3288240357 0 0) 0x7f6754125c20 con 0x7f67540a2d40 2019-12-05T05:09:53.873 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:53.809+0000 7f674affd700 1 -- 172.21.15.42:0/3152109268 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f673c0030b0 con 0x7f67540a2d40 2019-12-05T05:09:53.874 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:53.813+0000 7f674affd700 1 -- 172.21.15.42:0/3152109268 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1108307784 0 0) 0x7f67541259f0 con 0x7f675407c910 2019-12-05T05:09:53.874 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:53.813+0000 7f674affd700 1 -- 172.21.15.42:0/3152109268 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6754125c20 con 0x7f675407c910 2019-12-05T05:09:53.874 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:53.813+0000 7f6759a43700 1 -- 172.21.15.42:0/3152109268 >> v1:172.21.15.42:6789/0 conn(0x7f67540a2d40 legacy=0x7f67541251f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:09:53.874 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:53.813+0000 7f6759a43700 1 -- 172.21.15.42:0/3152109268 >> v1:172.21.15.40:6789/0 conn(0x7f675407c910 legacy=0x7f6754078a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:09:53.874 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:53.813+0000 7f6759a43700 1 -- 172.21.15.42:0/3152109268 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67540a3170 con 0x7f675408eea0 2019-12-05T05:09:53.874 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:53.813+0000 7f6759a43700 1 --2- 172.21.15.42:0/3152109268 >> v2:172.21.15.42:3300/0 conn(0x7f675408eea0 0x7f67540a27f0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f6744002f30 tx=0x7f67440108d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:53.875 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:53.813+0000 7f6759a43700 1 --2- 172.21.15.42:0/3152109268 >> v2:172.21.15.42:3300/0 conn(0x7f675408eea0 0x7f67540a27f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f6744002f30 tx=0x7f67440108d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:53.875 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:53.813+0000 7f6759a43700 1 --2- 172.21.15.42:0/3152109268 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f675408eea0 0x7f67540a27f0 secure :-1 s=READY pgs=375 cs=0 l=1 rx=0x7f6744002f30 tx=0x7f67440108d0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:09:53.875 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:53.813+0000 7f6759a43700 1 --2- 172.21.15.42:0/3152109268 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f675408eea0 0x7f67540a27f0 secure :-1 s=THROTTLE_DONE pgs=375 cs=0 l=1 rx=0x7f6744002f30 tx=0x7f67440108d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:53.875 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:53.813+0000 7f674affd700 1 -- 172.21.15.42:0/3152109268 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6744005930 con 0x7f675408eea0 2019-12-05T05:09:53.875 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:53.813+0000 7f6759a43700 1 --2- 172.21.15.42:0/3152109268 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f675408eea0 0x7f67540a27f0 secure :-1 s=THROTTLE_DONE pgs=375 cs=0 l=1 rx=0x7f6744002f30 tx=0x7f67440108d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:53.876 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:53.813+0000 7f674affd700 1 -- 172.21.15.42:0/3152109268 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6744007160 con 0x7f675408eea0 2019-12-05T05:09:53.876 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:53.813+0000 7f6759a43700 1 --2- 172.21.15.42:0/3152109268 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f675408eea0 0x7f67540a27f0 secure :-1 s=THROTTLE_DONE pgs=375 cs=0 l=1 rx=0x7f6744002f30 tx=0x7f67440108d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:53.876 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:53.813+0000 7f674affd700 1 -- 172.21.15.42:0/3152109268 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f67440060c0 con 0x7f675408eea0 2019-12-05T05:09:53.876 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:53.813+0000 7f675bca7700 1 -- 172.21.15.42:0/3152109268 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6754130740 con 0x7f675408eea0 2019-12-05T05:09:53.876 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:53.813+0000 7f675bca7700 1 -- 172.21.15.42:0/3152109268 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6754130c90 con 0x7f675408eea0 2019-12-05T05:09:53.877 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:53.813+0000 7f6759a43700 1 --2- 172.21.15.42:0/3152109268 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f675408eea0 0x7f67540a27f0 secure :-1 s=THROTTLE_DONE pgs=375 cs=0 l=1 rx=0x7f6744002f30 tx=0x7f67440108d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:53.877 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:53.813+0000 7f6759a43700 1 --2- 172.21.15.42:0/3152109268 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f675408eea0 0x7f67540a27f0 secure :-1 s=THROTTLE_DONE pgs=375 cs=0 l=1 rx=0x7f6744002f30 tx=0x7f67440108d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:53.877 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:53.813+0000 7f674affd700 1 -- 172.21.15.42:0/3152109268 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f6744007be0 con 0x7f675408eea0 2019-12-05T05:09:53.877 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:53.813+0000 7f674affd700 1 --2- 172.21.15.42:0/3152109268 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f673c03b390 0x7f673c03d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:09:53.877 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:53.813+0000 7f674affd700 1 -- 172.21.15.42:0/3152109268 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6289+0+0 (secure 0 0 0) 0x7f674404ab20 con 0x7f675408eea0 2019-12-05T05:09:53.877 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:53.813+0000 7f675a244700 1 --2- 172.21.15.42:0/3152109268 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f673c03b390 0x7f673c03d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:09:53.878 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:53.813+0000 7f675a244700 1 --2- 172.21.15.42:0/3152109268 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f673c03b390 0x7f673c03d8a0 crc :-1 s=READY pgs=171 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:09:53.878 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:53.817+0000 7f675bca7700 1 -- 172.21.15.42:0/3152109268 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f675405d100 con 0x7f675408eea0 2019-12-05T05:09:53.878 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:53.825+0000 7f6759a43700 1 --2- 172.21.15.42:0/3152109268 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f675408eea0 0x7f67540a27f0 secure :-1 s=THROTTLE_DONE pgs=375 cs=0 l=1 rx=0x7f6744002f30 tx=0x7f67440108d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:53.878 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:53.825+0000 7f674affd700 1 -- 172.21.15.42:0/3152109268 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f6744031b80 con 0x7f675408eea0 2019-12-05T05:09:54.002 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:53.997+0000 7f675bca7700 1 -- 172.21.15.42:0/3152109268 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f675405b8b0 con 0x7f675408eea0 2019-12-05T05:09:54.003 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:53.997+0000 7f6759a43700 1 --2- 172.21.15.42:0/3152109268 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f675408eea0 0x7f67540a27f0 secure :-1 s=THROTTLE_DONE pgs=375 cs=0 l=1 rx=0x7f6744002f30 tx=0x7f67440108d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:54.003 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:53.997+0000 7f674affd700 1 -- 172.21.15.42:0/3152109268 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+2105 (secure 0 0 0) 0x7f6744031de0 con 0x7f675408eea0 2019-12-05T05:09:54.004 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 35 2019-12-05T05:09:54.004 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:09:54.004 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4595,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/211584380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":211584380},{"type":"v1","addr":"172.21.15.40:6833","nonce":211584380}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":35}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:09:28.296949+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4978},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4978":{"gid":4978,"name":"a","rank":0,"incarnation":30,"state":"up:active","state_seq":36,"addr":"172.21.15.42:6835/3650564018","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3650564018},{"type":"v1","addr":"172.21.15.42:6835","nonce":3650564018}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:09:54.006 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:54.001+0000 7f675bca7700 1 -- 172.21.15.42:0/3152109268 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f673c03b390 msgr2=0x7f673c03d8a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:09:54.007 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:54.001+0000 7f675bca7700 1 --2- 172.21.15.42:0/3152109268 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f673c03b390 0x7f673c03d8a0 crc :-1 s=READY pgs=171 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:09:54.007 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:54.001+0000 7f675bca7700 1 -- 172.21.15.42:0/3152109268 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f675408eea0 msgr2=0x7f67540a27f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:09:54.007 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:54.001+0000 7f675bca7700 1 --2- 172.21.15.42:0/3152109268 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f675408eea0 0x7f67540a27f0 secure :-1 s=READY pgs=375 cs=0 l=1 rx=0x7f6744002f30 tx=0x7f67440108d0).stop 2019-12-05T05:09:54.007 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:54.001+0000 7f675bca7700 1 -- 172.21.15.42:0/3152109268 shutdown_connections 2019-12-05T05:09:54.007 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:54.001+0000 7f675bca7700 1 --2- 172.21.15.42:0/3152109268 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f673c03b390 0x7f673c03d8a0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:09:54.007 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:54.001+0000 7f675bca7700 1 --2- 172.21.15.42:0/3152109268 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f675408eea0 0x7f67540a27f0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:09:54.007 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:54.001+0000 7f675bca7700 1 -- 172.21.15.42:0/3152109268 shutdown_connections 2019-12-05T05:09:54.008 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:54.001+0000 7f675bca7700 1 -- 172.21.15.42:0/3152109268 wait complete. 2019-12-05T05:09:54.008 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:54.001+0000 7f675bca7700 1 -- 172.21.15.42:0/3152109268 >> 172.21.15.42:0/3152109268 conn(0x7f67540a7730 msgr2=0x7f67540a9630 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:09:54.016 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T05:09:56.017 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:09:56.124 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:56.113+0000 7f3f832e0700 1 Processor -- start 2019-12-05T05:09:56.124 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:56.113+0000 7f3f832e0700 1 -- start start 2019-12-05T05:09:56.124 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:56.113+0000 7f3f832e0700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f3f7c0a7740 0x7f3f7c08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:09:56.125 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:56.113+0000 7f3f832e0700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3f7c07e720 0x7f3f7c07eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:09:56.125 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:56.117+0000 7f3f832e0700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f7c0a0710 con 0x7f3f7c09f2e0 2019-12-05T05:09:56.125 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:56.117+0000 7f3f832e0700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f7c0a0b00 con 0x7f3f7c0a7740 2019-12-05T05:09:56.125 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:56.117+0000 7f3f832e0700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f7c08e5b0 con 0x7f3f7c07e720 2019-12-05T05:09:56.125 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:56.117+0000 7f3f8107c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3f7c07e720 0x7f3f7c07eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:09:56.125 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:56.117+0000 7f3f8107c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3f7c07e720 0x7f3f7c07eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:56984/0 (socket says 172.21.15.42:56984) 2019-12-05T05:09:56.125 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:56.117+0000 7f3f8107c700 1 -- 172.21.15.42:0/160325868 learned_addr learned my addr 172.21.15.42:0/160325868 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:09:56.126 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:56.117+0000 7f3f8187d700 1 --2- 172.21.15.42:0/160325868 >> v2:172.21.15.40:3300/0 conn(0x7f3f7c0a7740 0x7f3f7c08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:09:56.126 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:56.117+0000 7f3f8107c700 1 -- 172.21.15.42:0/160325868 >> v2:172.21.15.40:3300/0 conn(0x7f3f7c0a7740 msgr2=0x7f3f7c08df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:09:56.126 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:56.117+0000 7f3f8107c700 1 --2- 172.21.15.42:0/160325868 >> v2:172.21.15.40:3300/0 conn(0x7f3f7c0a7740 0x7f3f7c08df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:09:56.126 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:56.117+0000 7f3f8107c700 1 -- 172.21.15.42:0/160325868 >> v1:172.21.15.40:6789/0 conn(0x7f3f7c09f2e0 legacy=0x7f3f7c09f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:09:56.126 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:56.117+0000 7f3f8107c700 1 -- 172.21.15.42:0/160325868 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f7c07f0a0 con 0x7f3f7c07e720 2019-12-05T05:09:56.126 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:56.117+0000 7f3f8107c700 1 --2- 172.21.15.42:0/160325868 >> v2:172.21.15.42:3300/0 conn(0x7f3f7c07e720 0x7f3f7c07eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f3f6c002f30 tx=0x7f3f6c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:56.126 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:56.117+0000 7f3f8107c700 1 --2- 172.21.15.42:0/160325868 >> v2:172.21.15.42:3300/0 conn(0x7f3f7c07e720 0x7f3f7c07eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f3f6c002f30 tx=0x7f3f6c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:56.127 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:56.117+0000 7f3f8107c700 1 --2- 172.21.15.42:0/160325868 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3f7c07e720 0x7f3f7c07eb50 secure :-1 s=READY pgs=376 cs=0 l=1 rx=0x7f3f6c002f30 tx=0x7f3f6c00fd30).ready entity=mon.1 client_cookie=f0c9f2b57196d217 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:09:56.127 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:56.117+0000 7f3f8107c700 1 --2- 172.21.15.42:0/160325868 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3f7c07e720 0x7f3f7c07eb50 secure :-1 s=THROTTLE_DONE pgs=376 cs=0 l=1 rx=0x7f3f6c002f30 tx=0x7f3f6c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:56.127 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:56.117+0000 7f3f8087b700 1 -- 172.21.15.42:0/160325868 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3f6c011370 con 0x7f3f7c07e720 2019-12-05T05:09:56.127 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:56.117+0000 7f3f8107c700 1 --2- 172.21.15.42:0/160325868 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3f7c07e720 0x7f3f7c07eb50 secure :-1 s=THROTTLE_DONE pgs=376 cs=0 l=1 rx=0x7f3f6c002f30 tx=0x7f3f6c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:56.127 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:56.117+0000 7f3f8107c700 1 --2- 172.21.15.42:0/160325868 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3f7c07e720 0x7f3f7c07eb50 secure :-1 s=THROTTLE_DONE pgs=376 cs=0 l=1 rx=0x7f3f6c002f30 tx=0x7f3f6c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:56.127 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:56.117+0000 7f3f8087b700 1 -- 172.21.15.42:0/160325868 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f6c008bb0 con 0x7f3f7c07e720 2019-12-05T05:09:56.127 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:56.117+0000 7f3f8087b700 1 -- 172.21.15.42:0/160325868 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3f6c011950 con 0x7f3f7c07e720 2019-12-05T05:09:56.128 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:56.117+0000 7f3f832e0700 1 -- 172.21.15.42:0/160325868 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3f7c07e720 msgr2=0x7f3f7c07eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:09:56.128 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:56.117+0000 7f3f832e0700 1 --2- 172.21.15.42:0/160325868 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3f7c07e720 0x7f3f7c07eb50 secure :-1 s=READY pgs=376 cs=0 l=1 rx=0x7f3f6c002f30 tx=0x7f3f6c00fd30).stop 2019-12-05T05:09:56.130 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:56.117+0000 7f3f832e0700 1 -- 172.21.15.42:0/160325868 shutdown_connections 2019-12-05T05:09:56.131 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:56.117+0000 7f3f832e0700 1 --2- 172.21.15.42:0/160325868 >> v2:172.21.15.40:3300/0 conn(0x7f3f7c0a7740 0x7f3f7c08df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:09:56.132 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:56.117+0000 7f3f832e0700 1 --2- 172.21.15.42:0/160325868 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3f7c07e720 0x7f3f7c07eb50 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:09:56.132 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:56.117+0000 7f3f832e0700 1 -- 172.21.15.42:0/160325868 shutdown_connections 2019-12-05T05:09:56.132 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:56.117+0000 7f3f832e0700 1 -- 172.21.15.42:0/160325868 wait complete. 2019-12-05T05:09:56.132 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:56.117+0000 7f3f832e0700 1 -- 172.21.15.42:0/160325868 >> 172.21.15.42:0/160325868 conn(0x7f3f7c0aabe0 msgr2=0x7f3f7c0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:09:56.153 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:56.121+0000 7f3f832e0700 1 Processor -- start 2019-12-05T05:09:56.154 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:56.121+0000 7f3f832e0700 1 -- start start 2019-12-05T05:09:56.154 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:56.121+0000 7f3f832e0700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f3f7c131940 0x7f3f7c13c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:09:56.154 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:56.121+0000 7f3f832e0700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f7c07b520 con 0x7f3f7c131510 2019-12-05T05:09:56.154 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:56.121+0000 7f3f832e0700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f7c13df00 con 0x7f3f7c0a5250 2019-12-05T05:09:56.155 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:56.121+0000 7f3f832e0700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f7c0a4a70 con 0x7f3f7c131940 2019-12-05T05:09:56.155 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:56.121+0000 7f3f8187d700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f3f7c0a5250 0x7f3f7c118450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:57026/0 (socket says 172.21.15.42:57026) 2019-12-05T05:09:56.155 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:56.121+0000 7f3f8187d700 1 -- 172.21.15.42:0/3511410058 learned_addr learned my addr 172.21.15.42:0/3511410058 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:09:56.155 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:56.121+0000 7f3f8207e700 1 --2- 172.21.15.42:0/3511410058 >> v2:172.21.15.40:3300/0 conn(0x7f3f7c131940 0x7f3f7c13c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:09:56.155 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:56.125+0000 7f3f727fc700 1 -- 172.21.15.42:0/3511410058 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1981203693 0 0) 0x7f3f7c13df00 con 0x7f3f7c0a5250 2019-12-05T05:09:56.155 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:56.125+0000 7f3f727fc700 1 -- 172.21.15.42:0/3511410058 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f640030b0 con 0x7f3f7c0a5250 2019-12-05T05:09:56.155 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:56.125+0000 7f3f727fc700 1 -- 172.21.15.42:0/3511410058 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1585520176 0 0) 0x7f3f7c07b520 con 0x7f3f7c131510 2019-12-05T05:09:56.156 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:56.125+0000 7f3f727fc700 1 -- 172.21.15.42:0/3511410058 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f7c13df00 con 0x7f3f7c131510 2019-12-05T05:09:56.156 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:56.125+0000 7f3f727fc700 1 -- 172.21.15.42:0/3511410058 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 4047647978 0 0) 0x7f3f640030b0 con 0x7f3f7c0a5250 2019-12-05T05:09:56.156 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:56.125+0000 7f3f727fc700 1 -- 172.21.15.42:0/3511410058 --> v1:172.21.15.42:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3f7c07b520 con 0x7f3f7c0a5250 2019-12-05T05:09:56.156 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:56.125+0000 7f3f727fc700 1 -- 172.21.15.42:0/3511410058 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7f3f78001490 con 0x7f3f7c0a5250 2019-12-05T05:09:56.156 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:56.125+0000 7f3f8207e700 1 -- 172.21.15.42:0/3511410058 >> v1:172.21.15.42:6789/0 conn(0x7f3f7c0a5250 legacy=0x7f3f7c118450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:09:56.156 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:56.125+0000 7f3f8207e700 1 -- 172.21.15.42:0/3511410058 >> v1:172.21.15.40:6789/0 conn(0x7f3f7c131510 legacy=0x7f3f7c0a6100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:09:56.156 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:56.125+0000 7f3f8207e700 1 -- 172.21.15.42:0/3511410058 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f7c13f140 con 0x7f3f7c131940 2019-12-05T05:09:56.157 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:56.125+0000 7f3f8207e700 1 --2- 172.21.15.42:0/3511410058 >> v2:172.21.15.40:3300/0 conn(0x7f3f7c131940 0x7f3f7c13c9a0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f3f74001210 tx=0x7f3f78002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:56.157 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:56.125+0000 7f3f8207e700 1 --2- 172.21.15.42:0/3511410058 >> v2:172.21.15.40:3300/0 conn(0x7f3f7c131940 0x7f3f7c13c9a0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f3f74001210 tx=0x7f3f78002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:56.157 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:56.125+0000 7f3f8207e700 1 --2- 172.21.15.42:0/3511410058 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f7c131940 0x7f3f7c13c9a0 secure :-1 s=READY pgs=303 cs=0 l=1 rx=0x7f3f74001210 tx=0x7f3f78002ff0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:09:56.157 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:56.125+0000 7f3f8207e700 1 --2- 172.21.15.42:0/3511410058 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f7c131940 0x7f3f7c13c9a0 secure :-1 s=THROTTLE_DONE pgs=303 cs=0 l=1 rx=0x7f3f74001210 tx=0x7f3f78002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:56.157 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:56.125+0000 7f3f727fc700 1 -- 172.21.15.42:0/3511410058 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3f74006ca0 con 0x7f3f7c131940 2019-12-05T05:09:56.157 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:56.125+0000 7f3f8207e700 1 --2- 172.21.15.42:0/3511410058 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f7c131940 0x7f3f7c13c9a0 secure :-1 s=THROTTLE_DONE pgs=303 cs=0 l=1 rx=0x7f3f74001210 tx=0x7f3f78002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:56.157 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:56.125+0000 7f3f832e0700 1 -- 172.21.15.42:0/3511410058 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f7c13cef0 con 0x7f3f7c131940 2019-12-05T05:09:56.158 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:56.125+0000 7f3f727fc700 1 -- 172.21.15.42:0/3511410058 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f740068c0 con 0x7f3f7c131940 2019-12-05T05:09:56.158 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:56.125+0000 7f3f8207e700 1 --2- 172.21.15.42:0/3511410058 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f7c131940 0x7f3f7c13c9a0 secure :-1 s=THROTTLE_DONE pgs=303 cs=0 l=1 rx=0x7f3f74001210 tx=0x7f3f78002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:56.158 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:56.125+0000 7f3f832e0700 1 -- 172.21.15.42:0/3511410058 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f7c13d420 con 0x7f3f7c131940 2019-12-05T05:09:56.158 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:56.125+0000 7f3f727fc700 1 -- 172.21.15.42:0/3511410058 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3f74002600 con 0x7f3f7c131940 2019-12-05T05:09:56.158 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:56.125+0000 7f3f8207e700 1 --2- 172.21.15.42:0/3511410058 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f7c131940 0x7f3f7c13c9a0 secure :-1 s=THROTTLE_DONE pgs=303 cs=0 l=1 rx=0x7f3f74001210 tx=0x7f3f78002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:56.159 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:56.129+0000 7f3f727fc700 1 -- 172.21.15.42:0/3511410058 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f3f74002920 con 0x7f3f7c131940 2019-12-05T05:09:56.159 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:56.129+0000 7f3f8207e700 1 --2- 172.21.15.42:0/3511410058 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f7c131940 0x7f3f7c13c9a0 secure :-1 s=THROTTLE_DONE pgs=303 cs=0 l=1 rx=0x7f3f74001210 tx=0x7f3f78002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:56.160 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:56.129+0000 7f3f727fc700 1 --2- 172.21.15.42:0/3511410058 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3f6403b850 0x7f3f6403dd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:09:56.160 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:56.129+0000 7f3f727fc700 1 -- 172.21.15.42:0/3511410058 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6289+0+0 (secure 0 0 0) 0x7f3f7404b000 con 0x7f3f7c131940 2019-12-05T05:09:56.160 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:56.129+0000 7f3f8187d700 1 --2- 172.21.15.42:0/3511410058 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3f6403b850 0x7f3f6403dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:09:56.164 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:56.129+0000 7f3f8187d700 1 --2- 172.21.15.42:0/3511410058 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3f6403b850 0x7f3f6403dd60 crc :-1 s=READY pgs=172 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:09:56.164 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:56.129+0000 7f3f832e0700 1 -- 172.21.15.42:0/3511410058 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f7c05d110 con 0x7f3f7c131940 2019-12-05T05:09:56.164 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:56.137+0000 7f3f8207e700 1 --2- 172.21.15.42:0/3511410058 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f7c131940 0x7f3f7c13c9a0 secure :-1 s=THROTTLE_DONE pgs=303 cs=0 l=1 rx=0x7f3f74001210 tx=0x7f3f78002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:56.165 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:56.137+0000 7f3f727fc700 1 -- 172.21.15.42:0/3511410058 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f3f740322c0 con 0x7f3f7c131940 2019-12-05T05:09:56.314 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:56.309+0000 7f3f832e0700 1 -- 172.21.15.42:0/3511410058 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3f7c05b8c0 con 0x7f3f7c131940 2019-12-05T05:09:56.315 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:56.313+0000 7f3f8207e700 1 --2- 172.21.15.42:0/3511410058 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f7c131940 0x7f3f7c13c9a0 secure :-1 s=THROTTLE_DONE pgs=303 cs=0 l=1 rx=0x7f3f74001210 tx=0x7f3f78002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:56.315 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:56.313+0000 7f3f727fc700 1 -- 172.21.15.42:0/3511410058 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+2105 (secure 0 0 0) 0x7f3f74002c30 con 0x7f3f7c131940 2019-12-05T05:09:56.318 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:09:56.318 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4595,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/211584380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":211584380},{"type":"v1","addr":"172.21.15.40:6833","nonce":211584380}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":35}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:09:28.296949+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4978},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4978":{"gid":4978,"name":"a","rank":0,"incarnation":30,"state":"up:active","state_seq":36,"addr":"172.21.15.42:6835/3650564018","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3650564018},{"type":"v1","addr":"172.21.15.42:6835","nonce":3650564018}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:09:56.318 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 35 2019-12-05T05:09:56.320 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:56.317+0000 7f3f832e0700 1 -- 172.21.15.42:0/3511410058 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3f6403b850 msgr2=0x7f3f6403dd60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:09:56.321 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:56.317+0000 7f3f832e0700 1 --2- 172.21.15.42:0/3511410058 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3f6403b850 0x7f3f6403dd60 crc :-1 s=READY pgs=172 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:09:56.321 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:56.317+0000 7f3f832e0700 1 -- 172.21.15.42:0/3511410058 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f7c131940 msgr2=0x7f3f7c13c9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:09:56.321 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:56.317+0000 7f3f832e0700 1 --2- 172.21.15.42:0/3511410058 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f7c131940 0x7f3f7c13c9a0 secure :-1 s=READY pgs=303 cs=0 l=1 rx=0x7f3f74001210 tx=0x7f3f78002ff0).stop 2019-12-05T05:09:56.321 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:56.317+0000 7f3f832e0700 1 -- 172.21.15.42:0/3511410058 shutdown_connections 2019-12-05T05:09:56.321 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:56.317+0000 7f3f832e0700 1 --2- 172.21.15.42:0/3511410058 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3f6403b850 0x7f3f6403dd60 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:09:56.321 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:56.317+0000 7f3f832e0700 1 --2- 172.21.15.42:0/3511410058 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f7c131940 0x7f3f7c13c9a0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:09:56.322 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:56.317+0000 7f3f832e0700 1 -- 172.21.15.42:0/3511410058 shutdown_connections 2019-12-05T05:09:56.322 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:56.317+0000 7f3f832e0700 1 -- 172.21.15.42:0/3511410058 wait complete. 2019-12-05T05:09:56.322 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:56.317+0000 7f3f832e0700 1 -- 172.21.15.42:0/3511410058 >> 172.21.15.42:0/3511410058 conn(0x7f3f7c0a7f90 msgr2=0x7f3f7c059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:09:56.331 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T05:09:58.332 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:09:58.447 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:58.429+0000 7f032eafb700 1 Processor -- start 2019-12-05T05:09:58.448 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:58.429+0000 7f032eafb700 1 -- start start 2019-12-05T05:09:58.448 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:58.433+0000 7f032eafb700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f032808df70 0x7f032809f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:09:58.448 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:58.433+0000 7f032eafb700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f032809f820 0x7f032814efd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:09:58.448 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:58.433+0000 7f032eafb700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f032809fc50 con 0x7f03280a7730 2019-12-05T05:09:58.448 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:58.433+0000 7f032eafb700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03280a0af0 con 0x7f032809f820 2019-12-05T05:09:58.448 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:58.433+0000 7f032eafb700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f032808e5a0 con 0x7f032808df70 2019-12-05T05:09:58.449 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:58.433+0000 7f032c897700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f032808df70 0x7f032809f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:09:58.449 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:58.433+0000 7f032c897700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f032808df70 0x7f032809f2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:57002/0 (socket says 172.21.15.42:57002) 2019-12-05T05:09:58.449 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:58.433+0000 7f032c897700 1 -- 172.21.15.42:0/3595808553 learned_addr learned my addr 172.21.15.42:0/3595808553 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:09:58.449 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:58.433+0000 7f032d899700 1 --2- 172.21.15.42:0/3595808553 >> v2:172.21.15.40:3300/0 conn(0x7f032809f820 0x7f032814efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:09:58.449 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:58.433+0000 7f032c897700 1 -- 172.21.15.42:0/3595808553 >> v2:172.21.15.40:3300/0 conn(0x7f032809f820 msgr2=0x7f032814efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:09:58.449 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:58.433+0000 7f032c897700 1 --2- 172.21.15.42:0/3595808553 >> v2:172.21.15.40:3300/0 conn(0x7f032809f820 0x7f032814efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:09:58.449 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:58.433+0000 7f032c897700 1 -- 172.21.15.42:0/3595808553 >> v1:172.21.15.40:6789/0 conn(0x7f03280a7730 legacy=0x7f032807e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:09:58.450 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:58.433+0000 7f032c897700 1 -- 172.21.15.42:0/3595808553 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03280a0700 con 0x7f032808df70 2019-12-05T05:09:58.450 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:58.433+0000 7f032c897700 1 --2- 172.21.15.42:0/3595808553 >> v2:172.21.15.42:3300/0 conn(0x7f032808df70 0x7f032809f2d0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f031c0032b0 tx=0x7f031c0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:58.450 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:58.433+0000 7f032c897700 1 --2- 172.21.15.42:0/3595808553 >> v2:172.21.15.42:3300/0 conn(0x7f032808df70 0x7f032809f2d0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f031c0032b0 tx=0x7f031c0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:58.450 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:58.433+0000 7f032c897700 1 --2- 172.21.15.42:0/3595808553 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f032808df70 0x7f032809f2d0 secure :-1 s=READY pgs=378 cs=0 l=1 rx=0x7f031c0032b0 tx=0x7f031c0100b0).ready entity=mon.1 client_cookie=e12bee8911c6c7b5 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:09:58.450 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:58.433+0000 7f032c897700 1 --2- 172.21.15.42:0/3595808553 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f032808df70 0x7f032809f2d0 secure :-1 s=THROTTLE_DONE pgs=378 cs=0 l=1 rx=0x7f031c0032b0 tx=0x7f031c0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:58.450 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:58.433+0000 7f031bfff700 1 -- 172.21.15.42:0/3595808553 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f031c010d60 con 0x7f032808df70 2019-12-05T05:09:58.450 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:58.433+0000 7f032c897700 1 --2- 172.21.15.42:0/3595808553 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f032808df70 0x7f032809f2d0 secure :-1 s=THROTTLE_DONE pgs=378 cs=0 l=1 rx=0x7f031c0032b0 tx=0x7f031c0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:58.451 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:58.433+0000 7f031bfff700 1 -- 172.21.15.42:0/3595808553 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f031c010980 con 0x7f032808df70 2019-12-05T05:09:58.451 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:58.433+0000 7f032c897700 1 --2- 172.21.15.42:0/3595808553 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f032808df70 0x7f032809f2d0 secure :-1 s=THROTTLE_DONE pgs=378 cs=0 l=1 rx=0x7f031c0032b0 tx=0x7f031c0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:58.451 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:58.433+0000 7f031bfff700 1 -- 172.21.15.42:0/3595808553 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f031c0114a0 con 0x7f032808df70 2019-12-05T05:09:58.451 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:58.433+0000 7f032eafb700 1 -- 172.21.15.42:0/3595808553 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f032808df70 msgr2=0x7f032809f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:09:58.451 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:58.433+0000 7f032eafb700 1 --2- 172.21.15.42:0/3595808553 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f032808df70 0x7f032809f2d0 secure :-1 s=READY pgs=378 cs=0 l=1 rx=0x7f031c0032b0 tx=0x7f031c0100b0).stop 2019-12-05T05:09:58.451 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:58.433+0000 7f032eafb700 1 -- 172.21.15.42:0/3595808553 shutdown_connections 2019-12-05T05:09:58.451 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:58.433+0000 7f032eafb700 1 --2- 172.21.15.42:0/3595808553 >> v2:172.21.15.40:3300/0 conn(0x7f032809f820 0x7f032814efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:09:58.452 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:58.433+0000 7f032eafb700 1 --2- 172.21.15.42:0/3595808553 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f032808df70 0x7f032809f2d0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:09:58.452 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:58.433+0000 7f032eafb700 1 -- 172.21.15.42:0/3595808553 shutdown_connections 2019-12-05T05:09:58.452 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:58.433+0000 7f032eafb700 1 -- 172.21.15.42:0/3595808553 wait complete. 2019-12-05T05:09:58.452 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:58.433+0000 7f032eafb700 1 -- 172.21.15.42:0/3595808553 >> 172.21.15.42:0/3595808553 conn(0x7f03280aabd0 msgr2=0x7f03280a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:09:58.452 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:58.437+0000 7f032eafb700 1 Processor -- start 2019-12-05T05:09:58.452 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:58.437+0000 7f032eafb700 1 -- start start 2019-12-05T05:09:58.452 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:58.437+0000 7f032eafb700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f032807c070 0x7f0328138a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:09:58.453 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:58.437+0000 7f032eafb700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f032807d0a0 con 0x7f0328118c10 2019-12-05T05:09:58.453 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:58.437+0000 7f032eafb700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0328130dc0 con 0x7f032807bc40 2019-12-05T05:09:58.453 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:58.437+0000 7f032eafb700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f032807d370 con 0x7f032807c070 2019-12-05T05:09:58.453 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:58.437+0000 7f032c897700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f032807bc40 0x7f03281305c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:57044/0 (socket says 172.21.15.42:57044) 2019-12-05T05:09:58.453 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:58.437+0000 7f032c897700 1 -- 172.21.15.42:0/4240002981 learned_addr learned my addr 172.21.15.42:0/4240002981 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:09:58.454 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:58.437+0000 7f032d899700 1 --2- 172.21.15.42:0/4240002981 >> v2:172.21.15.42:3300/0 conn(0x7f032807c070 0x7f0328138a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:09:58.454 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:58.437+0000 7f0319ffb700 1 -- 172.21.15.42:0/4240002981 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2568671799 0 0) 0x7f032807d0a0 con 0x7f0328118c10 2019-12-05T05:09:58.454 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:58.437+0000 7f032d899700 1 -- 172.21.15.42:0/4240002981 >> v1:172.21.15.42:6789/0 conn(0x7f032807bc40 legacy=0x7f03281305c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:09:58.454 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:58.437+0000 7f032d899700 1 -- 172.21.15.42:0/4240002981 >> v1:172.21.15.40:6789/0 conn(0x7f0328118c10 legacy=0x7f032807b140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:09:58.455 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:58.441+0000 7f032d899700 1 -- 172.21.15.42:0/4240002981 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f032813af90 con 0x7f032807c070 2019-12-05T05:09:58.455 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:58.441+0000 7f032d899700 1 --2- 172.21.15.42:0/4240002981 >> v2:172.21.15.42:3300/0 conn(0x7f032807c070 0x7f0328138a20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f031c008290 tx=0x7f031c0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:58.455 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:58.441+0000 7f032d899700 1 --2- 172.21.15.42:0/4240002981 >> v2:172.21.15.42:3300/0 conn(0x7f032807c070 0x7f0328138a20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f031c008290 tx=0x7f031c0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:58.455 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:58.441+0000 7f032d899700 1 --2- 172.21.15.42:0/4240002981 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f032807c070 0x7f0328138a20 secure :-1 s=READY pgs=380 cs=0 l=1 rx=0x7f031c008290 tx=0x7f031c0100b0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:09:58.456 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:58.441+0000 7f032d899700 1 --2- 172.21.15.42:0/4240002981 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f032807c070 0x7f0328138a20 secure :-1 s=THROTTLE_DONE pgs=380 cs=0 l=1 rx=0x7f031c008290 tx=0x7f031c0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:58.456 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:58.441+0000 7f0319ffb700 1 -- 172.21.15.42:0/4240002981 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0320005930 con 0x7f032807c070 2019-12-05T05:09:58.456 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:58.441+0000 7f032d899700 1 --2- 172.21.15.42:0/4240002981 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f032807c070 0x7f0328138a20 secure :-1 s=THROTTLE_DONE pgs=380 cs=0 l=1 rx=0x7f031c008290 tx=0x7f031c0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:58.456 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:58.441+0000 7f032d899700 1 --2- 172.21.15.42:0/4240002981 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f032807c070 0x7f0328138a20 secure :-1 s=THROTTLE_DONE pgs=380 cs=0 l=1 rx=0x7f031c008290 tx=0x7f031c0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:58.456 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:58.441+0000 7f0319ffb700 1 -- 172.21.15.42:0/4240002981 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0320007160 con 0x7f032807c070 2019-12-05T05:09:58.457 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:58.441+0000 7f032eafb700 1 -- 172.21.15.42:0/4240002981 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0328138f70 con 0x7f032807c070 2019-12-05T05:09:58.457 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:58.441+0000 7f0319ffb700 1 -- 172.21.15.42:0/4240002981 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f03200060c0 con 0x7f032807c070 2019-12-05T05:09:58.457 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:58.441+0000 7f032eafb700 1 -- 172.21.15.42:0/4240002981 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0328139320 con 0x7f032807c070 2019-12-05T05:09:58.457 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:58.441+0000 7f032d899700 1 --2- 172.21.15.42:0/4240002981 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f032807c070 0x7f0328138a20 secure :-1 s=THROTTLE_DONE pgs=380 cs=0 l=1 rx=0x7f031c008290 tx=0x7f031c0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:58.457 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:58.441+0000 7f0319ffb700 1 -- 172.21.15.42:0/4240002981 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f03200063d0 con 0x7f032807c070 2019-12-05T05:09:58.457 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:58.441+0000 7f032d899700 1 --2- 172.21.15.42:0/4240002981 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f032807c070 0x7f0328138a20 secure :-1 s=THROTTLE_DONE pgs=380 cs=0 l=1 rx=0x7f031c008290 tx=0x7f031c0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:58.457 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:58.441+0000 7f0319ffb700 1 --2- 172.21.15.42:0/4240002981 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f030803acd0 0x7f030803d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:09:58.458 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:58.441+0000 7f0319ffb700 1 -- 172.21.15.42:0/4240002981 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6289+0+0 (secure 0 0 0) 0x7f032004ab00 con 0x7f032807c070 2019-12-05T05:09:58.458 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:58.441+0000 7f032d098700 1 --2- 172.21.15.42:0/4240002981 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f030803acd0 0x7f030803d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:09:58.458 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:58.441+0000 7f032d098700 1 --2- 172.21.15.42:0/4240002981 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f030803acd0 0x7f030803d1e0 crc :-1 s=READY pgs=173 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:09:58.458 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:58.449+0000 7f032eafb700 1 -- 172.21.15.42:0/4240002981 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f032805d100 con 0x7f032807c070 2019-12-05T05:09:58.461 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:58.457+0000 7f032d899700 1 --2- 172.21.15.42:0/4240002981 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f032807c070 0x7f0328138a20 secure :-1 s=THROTTLE_DONE pgs=380 cs=0 l=1 rx=0x7f031c008290 tx=0x7f031c0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:58.461 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:58.457+0000 7f0319ffb700 1 -- 172.21.15.42:0/4240002981 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f0320031b80 con 0x7f032807c070 2019-12-05T05:09:58.631 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:58.629+0000 7f032eafb700 1 -- 172.21.15.42:0/4240002981 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f032805b8b0 con 0x7f032807c070 2019-12-05T05:09:58.633 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:58.629+0000 7f032d899700 1 --2- 172.21.15.42:0/4240002981 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f032807c070 0x7f0328138a20 secure :-1 s=THROTTLE_DONE pgs=380 cs=0 l=1 rx=0x7f031c008290 tx=0x7f031c0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:09:58.633 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:58.629+0000 7f0319ffb700 1 -- 172.21.15.42:0/4240002981 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+2105 (secure 0 0 0) 0x7f0320031de0 con 0x7f032807c070 2019-12-05T05:09:58.634 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 35 2019-12-05T05:09:58.634 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:09:58.634 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4595,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/211584380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":211584380},{"type":"v1","addr":"172.21.15.40:6833","nonce":211584380}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":35}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:09:28.296949+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4978},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4978":{"gid":4978,"name":"a","rank":0,"incarnation":30,"state":"up:active","state_seq":36,"addr":"172.21.15.42:6835/3650564018","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3650564018},{"type":"v1","addr":"172.21.15.42:6835","nonce":3650564018}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:09:58.635 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:58.629+0000 7f032eafb700 1 -- 172.21.15.42:0/4240002981 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f030803acd0 msgr2=0x7f030803d1e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:09:58.635 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:58.629+0000 7f032eafb700 1 --2- 172.21.15.42:0/4240002981 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f030803acd0 0x7f030803d1e0 crc :-1 s=READY pgs=173 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:09:58.635 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:58.629+0000 7f032eafb700 1 -- 172.21.15.42:0/4240002981 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f032807c070 msgr2=0x7f0328138a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:09:58.635 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:58.629+0000 7f032eafb700 1 --2- 172.21.15.42:0/4240002981 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f032807c070 0x7f0328138a20 secure :-1 s=READY pgs=380 cs=0 l=1 rx=0x7f031c008290 tx=0x7f031c0100b0).stop 2019-12-05T05:09:58.636 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:58.629+0000 7f032eafb700 1 -- 172.21.15.42:0/4240002981 shutdown_connections 2019-12-05T05:09:58.636 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:58.629+0000 7f032eafb700 1 --2- 172.21.15.42:0/4240002981 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f030803acd0 0x7f030803d1e0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:09:58.636 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:58.629+0000 7f032eafb700 1 --2- 172.21.15.42:0/4240002981 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f032807c070 0x7f0328138a20 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:09:58.636 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:58.629+0000 7f032eafb700 1 -- 172.21.15.42:0/4240002981 shutdown_connections 2019-12-05T05:09:58.636 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:58.629+0000 7f032eafb700 1 -- 172.21.15.42:0/4240002981 wait complete. 2019-12-05T05:09:58.636 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:09:58.629+0000 7f032eafb700 1 -- 172.21.15.42:0/4240002981 >> 172.21.15.42:0/4240002981 conn(0x7f032808fdd0 msgr2=0x7f03280596f0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:09:58.645 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T05:10:00.646 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:10:00.762 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:00.745+0000 7ff81ca0e700 1 Processor -- start 2019-12-05T05:10:00.762 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:00.745+0000 7ff81ca0e700 1 -- start start 2019-12-05T05:10:00.763 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:00.745+0000 7ff81ca0e700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff81808df80 0x7ff81809f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:10:00.763 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:00.745+0000 7ff81ca0e700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff81809fc60 con 0x7ff8180a7740 2019-12-05T05:10:00.763 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:00.745+0000 7ff81ca0e700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8180a0710 con 0x7ff81809f830 2019-12-05T05:10:00.763 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:00.745+0000 7ff81ca0e700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8180a0b00 con 0x7ff81808df80 2019-12-05T05:10:00.763 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:00.745+0000 7ff81759e700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7ff81809f830 0x7ff81814efe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:57056/0 (socket says 172.21.15.42:57056) 2019-12-05T05:10:00.763 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:00.745+0000 7ff81759e700 1 -- 172.21.15.42:0/1804091896 learned_addr learned my addr 172.21.15.42:0/1804091896 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:10:00.763 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:00.745+0000 7ff80e59c700 1 --2- 172.21.15.42:0/1804091896 >> v2:172.21.15.40:3300/0 conn(0x7ff81808df80 0x7ff81809f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:10:00.764 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:00.745+0000 7ff81659c700 1 -- 172.21.15.42:0/1804091896 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2543275694 0 0) 0x7ff8180a0710 con 0x7ff81809f830 2019-12-05T05:10:00.764 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:00.745+0000 7ff81659c700 1 -- 172.21.15.42:0/1804091896 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8040030b0 con 0x7ff81809f830 2019-12-05T05:10:00.764 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:00.749+0000 7ff81659c700 1 -- 172.21.15.42:0/1804091896 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2466776048 0 0) 0x7ff8040030b0 con 0x7ff81809f830 2019-12-05T05:10:00.764 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:00.749+0000 7ff80e59c700 1 -- 172.21.15.42:0/1804091896 >> v1:172.21.15.42:6789/0 conn(0x7ff81809f830 legacy=0x7ff81814efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:10:00.765 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:00.749+0000 7ff80e59c700 1 -- 172.21.15.42:0/1804091896 >> v1:172.21.15.40:6789/0 conn(0x7ff8180a7740 legacy=0x7ff81807e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:10:00.765 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:00.749+0000 7ff80e59c700 1 -- 172.21.15.42:0/1804091896 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff81807ef20 con 0x7ff81808df80 2019-12-05T05:10:00.765 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:00.749+0000 7ff80e59c700 1 --2- 172.21.15.42:0/1804091896 >> v2:172.21.15.40:3300/0 conn(0x7ff81808df80 0x7ff81809f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7ff800008790 tx=0x7ff80000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:00.765 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:00.749+0000 7ff80e59c700 1 --2- 172.21.15.42:0/1804091896 >> v2:172.21.15.40:3300/0 conn(0x7ff81808df80 0x7ff81809f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7ff800008790 tx=0x7ff80000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:00.765 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:00.749+0000 7ff80e59c700 1 --2- 172.21.15.42:0/1804091896 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff81808df80 0x7ff81809f2e0 secure :-1 s=READY pgs=307 cs=0 l=1 rx=0x7ff800008790 tx=0x7ff80000fbf0).ready entity=mon.0 client_cookie=1fd794a48563115 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:10:00.765 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:00.749+0000 7ff80e59c700 1 --2- 172.21.15.42:0/1804091896 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff81808df80 0x7ff81809f2e0 secure :-1 s=THROTTLE_DONE pgs=307 cs=0 l=1 rx=0x7ff800008790 tx=0x7ff80000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:00.766 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:00.749+0000 7ff81659c700 1 -- 172.21.15.42:0/1804091896 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff800011270 con 0x7ff81808df80 2019-12-05T05:10:00.766 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:00.749+0000 7ff80e59c700 1 --2- 172.21.15.42:0/1804091896 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff81808df80 0x7ff81809f2e0 secure :-1 s=THROTTLE_DONE pgs=307 cs=0 l=1 rx=0x7ff800008790 tx=0x7ff80000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:00.766 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:00.749+0000 7ff80e59c700 1 --2- 172.21.15.42:0/1804091896 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff81808df80 0x7ff81809f2e0 secure :-1 s=THROTTLE_DONE pgs=307 cs=0 l=1 rx=0x7ff800008790 tx=0x7ff80000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:00.766 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:00.749+0000 7ff81659c700 1 -- 172.21.15.42:0/1804091896 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff800008bc0 con 0x7ff81808df80 2019-12-05T05:10:00.766 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:00.749+0000 7ff81659c700 1 -- 172.21.15.42:0/1804091896 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff800011850 con 0x7ff81808df80 2019-12-05T05:10:00.766 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:00.749+0000 7ff81ca0e700 1 -- 172.21.15.42:0/1804091896 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff81808df80 msgr2=0x7ff81809f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:10:00.766 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:00.749+0000 7ff81ca0e700 1 --2- 172.21.15.42:0/1804091896 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff81808df80 0x7ff81809f2e0 secure :-1 s=READY pgs=307 cs=0 l=1 rx=0x7ff800008790 tx=0x7ff80000fbf0).stop 2019-12-05T05:10:00.767 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:00.749+0000 7ff81ca0e700 1 -- 172.21.15.42:0/1804091896 shutdown_connections 2019-12-05T05:10:00.767 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:00.749+0000 7ff81ca0e700 1 --2- 172.21.15.42:0/1804091896 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff81808df80 0x7ff81809f2e0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:10:00.768 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:00.749+0000 7ff81ca0e700 1 -- 172.21.15.42:0/1804091896 shutdown_connections 2019-12-05T05:10:00.768 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:00.749+0000 7ff81ca0e700 1 -- 172.21.15.42:0/1804091896 wait complete. 2019-12-05T05:10:00.768 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:00.749+0000 7ff81ca0e700 1 -- 172.21.15.42:0/1804091896 >> 172.21.15.42:0/1804091896 conn(0x7ff8180aabe0 msgr2=0x7ff8180a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:10:00.768 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:00.753+0000 7ff81ca0e700 1 Processor -- start 2019-12-05T05:10:00.769 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:00.753+0000 7ff81ca0e700 1 -- start start 2019-12-05T05:10:00.785 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:00.753+0000 7ff81ca0e700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff81811d110 0x7ff81812cd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:10:00.786 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:00.753+0000 7ff81ca0e700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8180a2130 con 0x7ff81807d390 2019-12-05T05:10:00.786 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:00.753+0000 7ff81ca0e700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff818131050 con 0x7ff818130c20 2019-12-05T05:10:00.786 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:00.753+0000 7ff81ca0e700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff818130440 con 0x7ff81811d110 2019-12-05T05:10:00.786 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:00.753+0000 7ff80e59c700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7ff818130c20 0x7ff8180a1930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:57060/0 (socket says 172.21.15.42:57060) 2019-12-05T05:10:00.786 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:00.753+0000 7ff80e59c700 1 -- 172.21.15.42:0/3796866776 learned_addr learned my addr 172.21.15.42:0/3796866776 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:10:00.786 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:00.753+0000 7ff816d9d700 1 --2- 172.21.15.42:0/3796866776 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff81811d110 0x7ff81812cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:10:00.787 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:00.753+0000 7ff80ffff700 1 -- 172.21.15.42:0/3796866776 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2345326141 0 0) 0x7ff818131050 con 0x7ff818130c20 2019-12-05T05:10:00.787 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:00.753+0000 7ff80ffff700 1 -- 172.21.15.42:0/3796866776 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7fc0030b0 con 0x7ff818130c20 2019-12-05T05:10:00.787 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:00.753+0000 7ff80ffff700 1 -- 172.21.15.42:0/3796866776 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2860187943 0 0) 0x7ff8180a2130 con 0x7ff81807d390 2019-12-05T05:10:00.787 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:00.753+0000 7ff80ffff700 1 -- 172.21.15.42:0/3796866776 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff818131050 con 0x7ff81807d390 2019-12-05T05:10:00.787 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:00.757+0000 7ff816d9d700 1 -- 172.21.15.42:0/3796866776 >> v1:172.21.15.42:6789/0 conn(0x7ff818130c20 legacy=0x7ff8180a1930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:10:00.804 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:00.757+0000 7ff816d9d700 1 -- 172.21.15.42:0/3796866776 >> v1:172.21.15.40:6789/0 conn(0x7ff81807d390 legacy=0x7ff81807d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:10:00.804 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:00.757+0000 7ff816d9d700 1 -- 172.21.15.42:0/3796866776 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff81807dfc0 con 0x7ff81811d110 2019-12-05T05:10:00.804 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:00.757+0000 7ff816d9d700 1 --2- 172.21.15.42:0/3796866776 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff81811d110 0x7ff81812cd70 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7ff808000b80 tx=0x7ff80800c2f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:00.805 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:00.757+0000 7ff816d9d700 1 --2- 172.21.15.42:0/3796866776 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff81811d110 0x7ff81812cd70 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7ff808000b80 tx=0x7ff80800c2f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:00.805 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:00.757+0000 7ff816d9d700 1 --2- 172.21.15.42:0/3796866776 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff81811d110 0x7ff81812cd70 secure :-1 s=READY pgs=78 cs=0 l=1 rx=0x7ff808000b80 tx=0x7ff80800c2f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:10:00.805 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:00.757+0000 7ff816d9d700 1 --2- 172.21.15.42:0/3796866776 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff81811d110 0x7ff81812cd70 secure :-1 s=THROTTLE_DONE pgs=78 cs=0 l=1 rx=0x7ff808000b80 tx=0x7ff80800c2f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:00.805 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:00.757+0000 7ff80ffff700 1 -- 172.21.15.42:0/3796866776 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff80800cb90 con 0x7ff81811d110 2019-12-05T05:10:00.805 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:00.757+0000 7ff816d9d700 1 --2- 172.21.15.42:0/3796866776 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff81811d110 0x7ff81812cd70 secure :-1 s=THROTTLE_DONE pgs=78 cs=0 l=1 rx=0x7ff808000b80 tx=0x7ff80800c2f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:00.805 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:00.757+0000 7ff816d9d700 1 --2- 172.21.15.42:0/3796866776 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff81811d110 0x7ff81812cd70 secure :-1 s=THROTTLE_DONE pgs=78 cs=0 l=1 rx=0x7ff808000b80 tx=0x7ff80800c2f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:00.806 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:00.757+0000 7ff80ffff700 1 -- 172.21.15.42:0/3796866776 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff80800c7b0 con 0x7ff81811d110 2019-12-05T05:10:00.806 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:00.757+0000 7ff81ca0e700 1 -- 172.21.15.42:0/3796866776 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff818138a60 con 0x7ff81811d110 2019-12-05T05:10:00.806 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:00.757+0000 7ff80ffff700 1 -- 172.21.15.42:0/3796866776 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff808011070 con 0x7ff81811d110 2019-12-05T05:10:00.806 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:00.757+0000 7ff81ca0e700 1 -- 172.21.15.42:0/3796866776 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff818138f90 con 0x7ff81811d110 2019-12-05T05:10:00.806 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:00.757+0000 7ff816d9d700 1 --2- 172.21.15.42:0/3796866776 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff81811d110 0x7ff81812cd70 secure :-1 s=THROTTLE_DONE pgs=78 cs=0 l=1 rx=0x7ff808000b80 tx=0x7ff80800c2f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:00.806 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:00.757+0000 7ff80ffff700 1 -- 172.21.15.42:0/3796866776 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7ff808011340 con 0x7ff81811d110 2019-12-05T05:10:00.806 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:00.757+0000 7ff816d9d700 1 --2- 172.21.15.42:0/3796866776 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff81811d110 0x7ff81812cd70 secure :-1 s=THROTTLE_DONE pgs=78 cs=0 l=1 rx=0x7ff808000b80 tx=0x7ff80800c2f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:00.808 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:00.757+0000 7ff80ffff700 1 --2- 172.21.15.42:0/3796866776 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff7fc03b3f0 0x7ff7fc03d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:10:00.809 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:00.757+0000 7ff80ffff700 1 -- 172.21.15.42:0/3796866776 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6289+0+0 (secure 0 0 0) 0x7ff80804b110 con 0x7ff81811d110 2019-12-05T05:10:00.809 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:00.761+0000 7ff80e59c700 1 --2- 172.21.15.42:0/3796866776 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff7fc03b3f0 0x7ff7fc03d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:10:00.809 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:00.761+0000 7ff81ca0e700 1 -- 172.21.15.42:0/3796866776 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff81805d110 con 0x7ff81811d110 2019-12-05T05:10:00.810 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:00.761+0000 7ff80e59c700 1 --2- 172.21.15.42:0/3796866776 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff7fc03b3f0 0x7ff7fc03d900 crc :-1 s=READY pgs=174 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:10:00.810 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:00.769+0000 7ff816d9d700 1 --2- 172.21.15.42:0/3796866776 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff81811d110 0x7ff81812cd70 secure :-1 s=THROTTLE_DONE pgs=78 cs=0 l=1 rx=0x7ff808000b80 tx=0x7ff80800c2f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:00.811 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:00.769+0000 7ff80ffff700 1 -- 172.21.15.42:0/3796866776 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7ff808032190 con 0x7ff81811d110 2019-12-05T05:10:00.941 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:00.937+0000 7ff81ca0e700 1 -- 172.21.15.42:0/3796866776 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff81805b8c0 con 0x7ff81811d110 2019-12-05T05:10:00.942 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:00.937+0000 7ff816d9d700 1 --2- 172.21.15.42:0/3796866776 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff81811d110 0x7ff81812cd70 secure :-1 s=THROTTLE_DONE pgs=78 cs=0 l=1 rx=0x7ff808000b80 tx=0x7ff80800c2f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:00.942 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:00.937+0000 7ff80ffff700 1 -- 172.21.15.42:0/3796866776 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+2105 (secure 0 0 0) 0x7ff8080323f0 con 0x7ff81811d110 2019-12-05T05:10:00.944 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:10:00.944 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4595,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/211584380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":211584380},{"type":"v1","addr":"172.21.15.40:6833","nonce":211584380}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":35}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:09:28.296949+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4978},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4978":{"gid":4978,"name":"a","rank":0,"incarnation":30,"state":"up:active","state_seq":36,"addr":"172.21.15.42:6835/3650564018","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3650564018},{"type":"v1","addr":"172.21.15.42:6835","nonce":3650564018}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:10:00.945 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 35 2019-12-05T05:10:00.947 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:00.941+0000 7ff81ca0e700 1 -- 172.21.15.42:0/3796866776 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff7fc03b3f0 msgr2=0x7ff7fc03d900 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:10:00.948 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:00.941+0000 7ff81ca0e700 1 --2- 172.21.15.42:0/3796866776 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff7fc03b3f0 0x7ff7fc03d900 crc :-1 s=READY pgs=174 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:10:00.948 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:00.941+0000 7ff81ca0e700 1 -- 172.21.15.42:0/3796866776 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff81811d110 msgr2=0x7ff81812cd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:10:00.948 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:00.941+0000 7ff81ca0e700 1 --2- 172.21.15.42:0/3796866776 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff81811d110 0x7ff81812cd70 secure :-1 s=READY pgs=78 cs=0 l=1 rx=0x7ff808000b80 tx=0x7ff80800c2f0).stop 2019-12-05T05:10:00.948 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:00.941+0000 7ff81ca0e700 1 -- 172.21.15.42:0/3796866776 shutdown_connections 2019-12-05T05:10:00.948 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:00.941+0000 7ff81ca0e700 1 --2- 172.21.15.42:0/3796866776 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff7fc03b3f0 0x7ff7fc03d900 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:10:00.948 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:00.941+0000 7ff81ca0e700 1 --2- 172.21.15.42:0/3796866776 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff81811d110 0x7ff81812cd70 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:10:00.949 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:00.941+0000 7ff81ca0e700 1 -- 172.21.15.42:0/3796866776 shutdown_connections 2019-12-05T05:10:00.949 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:00.941+0000 7ff81ca0e700 1 -- 172.21.15.42:0/3796866776 wait complete. 2019-12-05T05:10:00.949 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:00.941+0000 7ff81ca0e700 1 -- 172.21.15.42:0/3796866776 >> 172.21.15.42:0/3796866776 conn(0x7ff81808fda0 msgr2=0x7ff8180a6a10 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:10:01.006 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T05:10:03.006 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:10:03.114 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.105+0000 7f5b9a98e700 1 Processor -- start 2019-12-05T05:10:03.114 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.105+0000 7f5b9a98e700 1 -- start start 2019-12-05T05:10:03.114 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.105+0000 7f5b9a98e700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5b9408df80 0x7f5b9409f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:10:03.114 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.105+0000 7f5b9a98e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5b9409f830 0x7f5b9414efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:10:03.121 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.105+0000 7f5b9a98e700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b9409fc60 con 0x7f5b940a7740 2019-12-05T05:10:03.121 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.105+0000 7f5b9a98e700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b940a0b00 con 0x7f5b9409f830 2019-12-05T05:10:03.121 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.105+0000 7f5b9a98e700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b9408e5b0 con 0x7f5b9408df80 2019-12-05T05:10:03.121 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.105+0000 7f5b9972c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5b9409f830 0x7f5b9414efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:10:03.121 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.105+0000 7f5b98f2b700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f5b940a7740 0x7f5b9407e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:57068/0 (socket says 172.21.15.42:57068) 2019-12-05T05:10:03.121 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.105+0000 7f5b98f2b700 1 -- 172.21.15.42:0/884703513 learned_addr learned my addr 172.21.15.42:0/884703513 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:10:03.122 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.105+0000 7f5b9972c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5b9409f830 0x7f5b9414efe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:57034/0 (socket says 172.21.15.42:57034) 2019-12-05T05:10:03.125 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.105+0000 7f5b93fff700 1 --2- 172.21.15.42:0/884703513 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5b9408df80 0x7f5b9409f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:10:03.126 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.105+0000 7f5b9972c700 1 -- 172.21.15.42:0/884703513 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5b9408df80 msgr2=0x7f5b9409f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:10:03.126 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.105+0000 7f5b9972c700 1 --2- 172.21.15.42:0/884703513 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5b9408df80 0x7f5b9409f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:10:03.126 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.105+0000 7f5b9972c700 1 -- 172.21.15.42:0/884703513 >> v1:172.21.15.42:6789/0 conn(0x7f5b940a7740 legacy=0x7f5b9407e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:10:03.126 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.105+0000 7f5b9972c700 1 -- 172.21.15.42:0/884703513 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b940a0710 con 0x7f5b9409f830 2019-12-05T05:10:03.127 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.105+0000 7f5b93fff700 1 --2- 172.21.15.42:0/884703513 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5b9408df80 0x7f5b9409f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T05:10:03.127 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.105+0000 7f5b9972c700 1 --2- 172.21.15.42:0/884703513 >> v2:172.21.15.42:3300/0 conn(0x7f5b9409f830 0x7f5b9414efe0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f5b8c008790 tx=0x7f5b8c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:03.127 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.105+0000 7f5b9972c700 1 --2- 172.21.15.42:0/884703513 >> v2:172.21.15.42:3300/0 conn(0x7f5b9409f830 0x7f5b9414efe0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f5b8c008790 tx=0x7f5b8c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:03.127 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.105+0000 7f5b9972c700 1 --2- 172.21.15.42:0/884703513 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5b9409f830 0x7f5b9414efe0 secure :-1 s=READY pgs=384 cs=0 l=1 rx=0x7f5b8c008790 tx=0x7f5b8c00fc70).ready entity=mon.1 client_cookie=1c29e3201355a770 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:10:03.128 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.105+0000 7f5b9972c700 1 --2- 172.21.15.42:0/884703513 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5b9409f830 0x7f5b9414efe0 secure :-1 s=THROTTLE_DONE pgs=384 cs=0 l=1 rx=0x7f5b8c008790 tx=0x7f5b8c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:03.128 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.105+0000 7f5b9972c700 1 --2- 172.21.15.42:0/884703513 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5b9409f830 0x7f5b9414efe0 secure :-1 s=THROTTLE_DONE pgs=384 cs=0 l=1 rx=0x7f5b8c008790 tx=0x7f5b8c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:03.128 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.105+0000 7f5b937fe700 1 -- 172.21.15.42:0/884703513 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5b8c008bb0 con 0x7f5b9409f830 2019-12-05T05:10:03.128 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.105+0000 7f5b9972c700 1 --2- 172.21.15.42:0/884703513 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5b9409f830 0x7f5b9414efe0 secure :-1 s=THROTTLE_DONE pgs=384 cs=0 l=1 rx=0x7f5b8c008790 tx=0x7f5b8c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:03.129 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.105+0000 7f5b937fe700 1 -- 172.21.15.42:0/884703513 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b8c0087d0 con 0x7f5b9409f830 2019-12-05T05:10:03.129 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.105+0000 7f5b937fe700 1 -- 172.21.15.42:0/884703513 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5b8c011860 con 0x7f5b9409f830 2019-12-05T05:10:03.129 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.105+0000 7f5b9a98e700 1 -- 172.21.15.42:0/884703513 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5b9409f830 msgr2=0x7f5b9414efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:10:03.129 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.105+0000 7f5b9a98e700 1 --2- 172.21.15.42:0/884703513 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5b9409f830 0x7f5b9414efe0 secure :-1 s=READY pgs=384 cs=0 l=1 rx=0x7f5b8c008790 tx=0x7f5b8c00fc70).stop 2019-12-05T05:10:03.129 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.109+0000 7f5b9a98e700 1 -- 172.21.15.42:0/884703513 shutdown_connections 2019-12-05T05:10:03.130 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.109+0000 7f5b9a98e700 1 --2- 172.21.15.42:0/884703513 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5b9409f830 0x7f5b9414efe0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:10:03.130 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.109+0000 7f5b9a98e700 1 --2- 172.21.15.42:0/884703513 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5b9408df80 0x7f5b9409f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:10:03.130 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.109+0000 7f5b9a98e700 1 -- 172.21.15.42:0/884703513 shutdown_connections 2019-12-05T05:10:03.130 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.109+0000 7f5b9a98e700 1 -- 172.21.15.42:0/884703513 wait complete. 2019-12-05T05:10:03.131 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.109+0000 7f5b9a98e700 1 -- 172.21.15.42:0/884703513 >> 172.21.15.42:0/884703513 conn(0x7f5b940aabe0 msgr2=0x7f5b940a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:10:03.131 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.109+0000 7f5b9a98e700 1 Processor -- start 2019-12-05T05:10:03.131 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.109+0000 7f5b9a98e700 1 -- start start 2019-12-05T05:10:03.139 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.109+0000 7f5b9a98e700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5b9412b320 0x7f5b940a6ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:10:03.140 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.109+0000 7f5b9a98e700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5b9411cd50 0x7f5b9411acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:10:03.140 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.109+0000 7f5b9a98e700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b9412c180 con 0x7f5b9411b200 2019-12-05T05:10:03.140 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.109+0000 7f5b9a98e700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b940a57c0 con 0x7f5b9411cd50 2019-12-05T05:10:03.140 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.109+0000 7f5b9a98e700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b940a5a60 con 0x7f5b9412b320 2019-12-05T05:10:03.141 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.109+0000 7f5b9972c700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f5b9411b200 0x7f5b9412b980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:57078/0 (socket says 172.21.15.42:57078) 2019-12-05T05:10:03.141 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.109+0000 7f5b9972c700 1 -- 172.21.15.42:0/2500902722 learned_addr learned my addr 172.21.15.42:0/2500902722 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:10:03.141 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.109+0000 7f5b93fff700 1 --2- 172.21.15.42:0/2500902722 >> v2:172.21.15.40:3300/0 conn(0x7f5b9411cd50 0x7f5b9411acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:10:03.141 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.109+0000 7f5b98f2b700 1 --2- 172.21.15.42:0/2500902722 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5b9412b320 0x7f5b940a6ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:10:03.142 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.113+0000 7f5b917fa700 1 -- 172.21.15.42:0/2500902722 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1034356469 0 0) 0x7f5b9412c180 con 0x7f5b9411b200 2019-12-05T05:10:03.142 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.113+0000 7f5b917fa700 1 -- 172.21.15.42:0/2500902722 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b780030b0 con 0x7f5b9411b200 2019-12-05T05:10:03.142 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.113+0000 7f5b917fa700 1 -- 172.21.15.42:0/2500902722 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 2083333088 0 0) 0x7f5b780030b0 con 0x7f5b9411b200 2019-12-05T05:10:03.142 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.113+0000 7f5b93fff700 1 -- 172.21.15.42:0/2500902722 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5b9412b320 msgr2=0x7f5b940a6ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:10:03.142 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.113+0000 7f5b93fff700 1 --2- 172.21.15.42:0/2500902722 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5b9412b320 0x7f5b940a6ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:10:03.142 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.113+0000 7f5b93fff700 1 -- 172.21.15.42:0/2500902722 >> v1:172.21.15.42:6789/0 conn(0x7f5b9411b200 legacy=0x7f5b9412b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:10:03.143 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.113+0000 7f5b93fff700 1 -- 172.21.15.42:0/2500902722 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b940a5240 con 0x7f5b9411cd50 2019-12-05T05:10:03.143 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.113+0000 7f5b98f2b700 1 --2- 172.21.15.42:0/2500902722 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5b9412b320 0x7f5b940a6ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:10:03.143 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.113+0000 7f5b93fff700 1 --2- 172.21.15.42:0/2500902722 >> v2:172.21.15.40:3300/0 conn(0x7f5b9411cd50 0x7f5b9411acb0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f5b8c008670 tx=0x7f5b8c010200).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:03.143 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.113+0000 7f5b93fff700 1 --2- 172.21.15.42:0/2500902722 >> v2:172.21.15.40:3300/0 conn(0x7f5b9411cd50 0x7f5b9411acb0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f5b8c008670 tx=0x7f5b8c010200).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:03.143 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.113+0000 7f5b93fff700 1 --2- 172.21.15.42:0/2500902722 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b9411cd50 0x7f5b9411acb0 secure :-1 s=READY pgs=309 cs=0 l=1 rx=0x7f5b8c008670 tx=0x7f5b8c010200).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:10:03.143 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.113+0000 7f5b93fff700 1 --2- 172.21.15.42:0/2500902722 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b9411cd50 0x7f5b9411acb0 secure :-1 s=THROTTLE_DONE pgs=309 cs=0 l=1 rx=0x7f5b8c008670 tx=0x7f5b8c010200).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:03.143 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.113+0000 7f5b917fa700 1 -- 172.21.15.42:0/2500902722 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5b8c008470 con 0x7f5b9411cd50 2019-12-05T05:10:03.144 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.113+0000 7f5b93fff700 1 --2- 172.21.15.42:0/2500902722 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b9411cd50 0x7f5b9411acb0 secure :-1 s=THROTTLE_DONE pgs=309 cs=0 l=1 rx=0x7f5b8c008670 tx=0x7f5b8c010200).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:03.144 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.113+0000 7f5b93fff700 1 --2- 172.21.15.42:0/2500902722 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b9411cd50 0x7f5b9411acb0 secure :-1 s=THROTTLE_DONE pgs=309 cs=0 l=1 rx=0x7f5b8c008670 tx=0x7f5b8c010200).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:03.144 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.113+0000 7f5b917fa700 1 -- 172.21.15.42:0/2500902722 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b8c008db0 con 0x7f5b9411cd50 2019-12-05T05:10:03.144 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.113+0000 7f5b9a98e700 1 -- 172.21.15.42:0/2500902722 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b94132770 con 0x7f5b9411cd50 2019-12-05T05:10:03.144 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.113+0000 7f5b917fa700 1 -- 172.21.15.42:0/2500902722 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5b7c005b20 con 0x7f5b9411cd50 2019-12-05T05:10:03.144 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.113+0000 7f5b9a98e700 1 -- 172.21.15.42:0/2500902722 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b94132ca0 con 0x7f5b9411cd50 2019-12-05T05:10:03.144 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.113+0000 7f5b93fff700 1 --2- 172.21.15.42:0/2500902722 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b9411cd50 0x7f5b9411acb0 secure :-1 s=THROTTLE_DONE pgs=309 cs=0 l=1 rx=0x7f5b8c008670 tx=0x7f5b8c010200).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:03.145 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.113+0000 7f5b917fa700 1 -- 172.21.15.42:0/2500902722 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f5b7c00f180 con 0x7f5b9411cd50 2019-12-05T05:10:03.145 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.113+0000 7f5b93fff700 1 --2- 172.21.15.42:0/2500902722 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b9411cd50 0x7f5b9411acb0 secure :-1 s=THROTTLE_DONE pgs=309 cs=0 l=1 rx=0x7f5b8c008670 tx=0x7f5b8c010200).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:03.145 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.117+0000 7f5b917fa700 1 --2- 172.21.15.42:0/2500902722 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5b7803b230 0x7f5b7803d740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:10:03.145 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.117+0000 7f5b917fa700 1 -- 172.21.15.42:0/2500902722 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6289+0+0 (secure 0 0 0) 0x7f5b7c04a8f0 con 0x7f5b9411cd50 2019-12-05T05:10:03.145 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.117+0000 7f5b98f2b700 1 --2- 172.21.15.42:0/2500902722 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5b7803b230 0x7f5b7803d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:10:03.145 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.117+0000 7f5b98f2b700 1 --2- 172.21.15.42:0/2500902722 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5b7803b230 0x7f5b7803d740 crc :-1 s=READY pgs=175 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:10:03.145 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.121+0000 7f5b9a98e700 1 -- 172.21.15.42:0/2500902722 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b9405d110 con 0x7f5b9411cd50 2019-12-05T05:10:03.146 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.129+0000 7f5b93fff700 1 --2- 172.21.15.42:0/2500902722 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b9411cd50 0x7f5b9411acb0 secure :-1 s=THROTTLE_DONE pgs=309 cs=0 l=1 rx=0x7f5b8c008670 tx=0x7f5b8c010200).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:03.146 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.129+0000 7f5b917fa700 1 -- 172.21.15.42:0/2500902722 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f5b7c031950 con 0x7f5b9411cd50 2019-12-05T05:10:03.311 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.305+0000 7f5b9a98e700 1 -- 172.21.15.42:0/2500902722 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5b9405b8c0 con 0x7f5b9411cd50 2019-12-05T05:10:03.312 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.309+0000 7f5b93fff700 1 --2- 172.21.15.42:0/2500902722 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b9411cd50 0x7f5b9411acb0 secure :-1 s=THROTTLE_DONE pgs=309 cs=0 l=1 rx=0x7f5b8c008670 tx=0x7f5b8c010200).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:03.312 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.309+0000 7f5b917fa700 1 -- 172.21.15.42:0/2500902722 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) v1 ==== 94+0+1768 (secure 0 0 0) 0x7f5b7c005dc0 con 0x7f5b9411cd50 2019-12-05T05:10:03.314 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:10:03.314 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":36,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:10:03.270458+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4595},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4595":{"gid":4595,"name":"b","rank":0,"incarnation":36,"state":"up:replay","state_seq":2,"addr":"172.21.15.40:6833/211584380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":211584380},{"type":"v1","addr":"172.21.15.40:6833","nonce":211584380}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:10:03.314 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 36 2019-12-05T05:10:03.317 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.313+0000 7f5b9a98e700 1 -- 172.21.15.42:0/2500902722 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5b7803b230 msgr2=0x7f5b7803d740 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:10:03.317 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.313+0000 7f5b9a98e700 1 --2- 172.21.15.42:0/2500902722 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5b7803b230 0x7f5b7803d740 crc :-1 s=READY pgs=175 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:10:03.317 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.313+0000 7f5b9a98e700 1 -- 172.21.15.42:0/2500902722 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b9411cd50 msgr2=0x7f5b9411acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:10:03.317 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.313+0000 7f5b9a98e700 1 --2- 172.21.15.42:0/2500902722 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b9411cd50 0x7f5b9411acb0 secure :-1 s=READY pgs=309 cs=0 l=1 rx=0x7f5b8c008670 tx=0x7f5b8c010200).stop 2019-12-05T05:10:03.317 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.313+0000 7f5b9a98e700 1 -- 172.21.15.42:0/2500902722 shutdown_connections 2019-12-05T05:10:03.318 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.313+0000 7f5b9a98e700 1 --2- 172.21.15.42:0/2500902722 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5b7803b230 0x7f5b7803d740 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:10:03.318 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.313+0000 7f5b9a98e700 1 --2- 172.21.15.42:0/2500902722 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5b9412b320 0x7f5b940a6ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:10:03.318 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.313+0000 7f5b9a98e700 1 --2- 172.21.15.42:0/2500902722 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b9411cd50 0x7f5b9411acb0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:10:03.318 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.313+0000 7f5b9a98e700 1 -- 172.21.15.42:0/2500902722 shutdown_connections 2019-12-05T05:10:03.318 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.313+0000 7f5b9a98e700 1 -- 172.21.15.42:0/2500902722 wait complete. 2019-12-05T05:10:03.318 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.313+0000 7f5b9a98e700 1 -- 172.21.15.42:0/2500902722 >> 172.21.15.42:0/2500902722 conn(0x7f5b9409f4e0 msgr2=0x7f5b9411da10 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:10:03.327 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a down, removed from mdsmap 2019-12-05T05:10:03.327 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2019-12-05T05:10:03.328 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:10:03.423 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.413+0000 7f2a2ad8f700 1 Processor -- start 2019-12-05T05:10:03.423 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.413+0000 7f2a2ad8f700 1 -- start start 2019-12-05T05:10:03.423 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.413+0000 7f2a2ad8f700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2a240a7740 0x7f2a2408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:10:03.423 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.413+0000 7f2a2ad8f700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2a2407e720 0x7f2a2407eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:10:03.423 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.413+0000 7f2a2ad8f700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a240a0710 con 0x7f2a2409f2e0 2019-12-05T05:10:03.424 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.413+0000 7f2a2ad8f700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a240a0b00 con 0x7f2a2407e720 2019-12-05T05:10:03.424 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.413+0000 7f2a2ad8f700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a2408e5b0 con 0x7f2a240a7740 2019-12-05T05:10:03.424 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.413+0000 7f2a2932c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2a240a7740 0x7f2a2408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:10:03.424 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.413+0000 7f2a29b2d700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f2a2409f2e0 0x7f2a2409f710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:57086/0 (socket says 172.21.15.42:57086) 2019-12-05T05:10:03.424 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.413+0000 7f2a29b2d700 1 -- 172.21.15.42:0/4135409931 learned_addr learned my addr 172.21.15.42:0/4135409931 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:10:03.425 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.417+0000 7f2a28b2b700 1 --2- 172.21.15.42:0/4135409931 >> v2:172.21.15.40:3300/0 conn(0x7f2a2407e720 0x7f2a2407eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:10:03.425 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.417+0000 7f2a1bfff700 1 -- 172.21.15.42:0/4135409931 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 122001302 0 0) 0x7f2a240a0710 con 0x7f2a2409f2e0 2019-12-05T05:10:03.425 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.417+0000 7f2a1bfff700 1 -- 172.21.15.42:0/4135409931 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a100030b0 con 0x7f2a2409f2e0 2019-12-05T05:10:03.425 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.417+0000 7f2a2932c700 1 -- 172.21.15.42:0/4135409931 >> v2:172.21.15.40:3300/0 conn(0x7f2a2407e720 msgr2=0x7f2a2407eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:10:03.425 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.417+0000 7f2a2932c700 1 --2- 172.21.15.42:0/4135409931 >> v2:172.21.15.40:3300/0 conn(0x7f2a2407e720 0x7f2a2407eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:10:03.425 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.417+0000 7f2a2932c700 1 -- 172.21.15.42:0/4135409931 >> v1:172.21.15.42:6789/0 conn(0x7f2a2409f2e0 legacy=0x7f2a2409f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:10:03.425 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.417+0000 7f2a2932c700 1 -- 172.21.15.42:0/4135409931 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a2407f0a0 con 0x7f2a240a7740 2019-12-05T05:10:03.426 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.417+0000 7f2a28b2b700 1 --2- 172.21.15.42:0/4135409931 >> v2:172.21.15.40:3300/0 conn(0x7f2a2407e720 0x7f2a2407eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T05:10:03.426 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.417+0000 7f2a2932c700 1 --2- 172.21.15.42:0/4135409931 >> v2:172.21.15.42:3300/0 conn(0x7f2a240a7740 0x7f2a2408df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f2a20002f30 tx=0x7f2a2000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:03.426 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.417+0000 7f2a2932c700 1 --2- 172.21.15.42:0/4135409931 >> v2:172.21.15.42:3300/0 conn(0x7f2a240a7740 0x7f2a2408df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f2a20002f30 tx=0x7f2a2000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:03.426 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.417+0000 7f2a2932c700 1 --2- 172.21.15.42:0/4135409931 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2a240a7740 0x7f2a2408df80 secure :-1 s=READY pgs=387 cs=0 l=1 rx=0x7f2a20002f30 tx=0x7f2a2000fc70).ready entity=mon.1 client_cookie=b804bbec9d8b21cf server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:10:03.426 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.417+0000 7f2a2932c700 1 --2- 172.21.15.42:0/4135409931 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2a240a7740 0x7f2a2408df80 secure :-1 s=THROTTLE_DONE pgs=387 cs=0 l=1 rx=0x7f2a20002f30 tx=0x7f2a2000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:03.426 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.417+0000 7f2a1bfff700 1 -- 172.21.15.42:0/4135409931 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2a200112b0 con 0x7f2a240a7740 2019-12-05T05:10:03.426 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.417+0000 7f2a2932c700 1 --2- 172.21.15.42:0/4135409931 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2a240a7740 0x7f2a2408df80 secure :-1 s=THROTTLE_DONE pgs=387 cs=0 l=1 rx=0x7f2a20002f30 tx=0x7f2a2000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:03.427 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.417+0000 7f2a2932c700 1 --2- 172.21.15.42:0/4135409931 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2a240a7740 0x7f2a2408df80 secure :-1 s=THROTTLE_DONE pgs=387 cs=0 l=1 rx=0x7f2a20002f30 tx=0x7f2a2000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:03.427 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.417+0000 7f2a1bfff700 1 -- 172.21.15.42:0/4135409931 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a20008ad0 con 0x7f2a240a7740 2019-12-05T05:10:03.427 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.417+0000 7f2a1bfff700 1 -- 172.21.15.42:0/4135409931 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2a20011890 con 0x7f2a240a7740 2019-12-05T05:10:03.427 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.417+0000 7f2a2ad8f700 1 -- 172.21.15.42:0/4135409931 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2a240a7740 msgr2=0x7f2a2408df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:10:03.427 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.417+0000 7f2a2ad8f700 1 --2- 172.21.15.42:0/4135409931 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2a240a7740 0x7f2a2408df80 secure :-1 s=READY pgs=387 cs=0 l=1 rx=0x7f2a20002f30 tx=0x7f2a2000fc70).stop 2019-12-05T05:10:03.427 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.417+0000 7f2a2ad8f700 1 -- 172.21.15.42:0/4135409931 shutdown_connections 2019-12-05T05:10:03.427 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.417+0000 7f2a2ad8f700 1 --2- 172.21.15.42:0/4135409931 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2a240a7740 0x7f2a2408df80 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:10:03.428 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.417+0000 7f2a2ad8f700 1 --2- 172.21.15.42:0/4135409931 >> v2:172.21.15.40:3300/0 conn(0x7f2a2407e720 0x7f2a2407eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:10:03.428 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.417+0000 7f2a2ad8f700 1 -- 172.21.15.42:0/4135409931 shutdown_connections 2019-12-05T05:10:03.428 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.417+0000 7f2a2ad8f700 1 -- 172.21.15.42:0/4135409931 wait complete. 2019-12-05T05:10:03.428 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.417+0000 7f2a2ad8f700 1 -- 172.21.15.42:0/4135409931 >> 172.21.15.42:0/4135409931 conn(0x7f2a240aabe0 msgr2=0x7f2a240a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:10:03.433 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.421+0000 7f2a2ad8f700 1 Processor -- start 2019-12-05T05:10:03.433 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.421+0000 7f2a2ad8f700 1 -- start start 2019-12-05T05:10:03.433 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.421+0000 7f2a2ad8f700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2a24124150 0x7f2a240a61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:10:03.433 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.421+0000 7f2a2ad8f700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2a240a5540 0x7f2a2407db50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:10:03.433 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.421+0000 7f2a2ad8f700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a2407e0a0 con 0x7f2a2411a1d0 2019-12-05T05:10:03.433 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.421+0000 7f2a2ad8f700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a240a5e10 con 0x7f2a24124150 2019-12-05T05:10:03.434 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.421+0000 7f2a2ad8f700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a2411af50 con 0x7f2a240a5540 2019-12-05T05:10:03.434 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.421+0000 7f2a28b2b700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f2a2411a1d0 0x7f2a2411b1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:57090/0 (socket says 172.21.15.42:57090) 2019-12-05T05:10:03.434 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.421+0000 7f2a28b2b700 1 -- 172.21.15.42:0/3791901868 learned_addr learned my addr 172.21.15.42:0/3791901868 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:10:03.434 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.421+0000 7f2a2932c700 1 --2- 172.21.15.42:0/3791901868 >> v2:172.21.15.40:3300/0 conn(0x7f2a24124150 0x7f2a240a61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:10:03.435 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.421+0000 7f2a29b2d700 1 --2- 172.21.15.42:0/3791901868 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2a240a5540 0x7f2a2407db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:10:03.435 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.421+0000 7f2a19ffb700 1 -- 172.21.15.42:0/3791901868 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 841125646 0 0) 0x7f2a2407e0a0 con 0x7f2a2411a1d0 2019-12-05T05:10:03.435 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.421+0000 7f2a19ffb700 1 -- 172.21.15.42:0/3791901868 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a040030b0 con 0x7f2a2411a1d0 2019-12-05T05:10:03.435 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.421+0000 7f2a2932c700 1 -- 172.21.15.42:0/3791901868 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2a240a5540 msgr2=0x7f2a2407db50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:10:03.436 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.421+0000 7f2a2932c700 1 --2- 172.21.15.42:0/3791901868 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2a240a5540 0x7f2a2407db50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:10:03.436 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.421+0000 7f2a2932c700 1 -- 172.21.15.42:0/3791901868 >> v1:172.21.15.42:6789/0 conn(0x7f2a2411a1d0 legacy=0x7f2a2411b1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:10:03.436 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.421+0000 7f2a2932c700 1 -- 172.21.15.42:0/3791901868 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a240a5970 con 0x7f2a24124150 2019-12-05T05:10:03.436 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.421+0000 7f2a29b2d700 1 --2- 172.21.15.42:0/3791901868 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2a240a5540 0x7f2a2407db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:10:03.436 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.421+0000 7f2a2932c700 1 --2- 172.21.15.42:0/3791901868 >> v2:172.21.15.40:3300/0 conn(0x7f2a24124150 0x7f2a240a61b0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f2a240a5fd0 tx=0x7f2a20011950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:03.436 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.421+0000 7f2a2932c700 1 --2- 172.21.15.42:0/3791901868 >> v2:172.21.15.40:3300/0 conn(0x7f2a24124150 0x7f2a240a61b0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f2a240a5fd0 tx=0x7f2a20011950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:03.436 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.421+0000 7f2a2932c700 1 --2- 172.21.15.42:0/3791901868 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a24124150 0x7f2a240a61b0 secure :-1 s=READY pgs=310 cs=0 l=1 rx=0x7f2a240a5fd0 tx=0x7f2a20011950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:10:03.437 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.421+0000 7f2a2932c700 1 --2- 172.21.15.42:0/3791901868 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a24124150 0x7f2a240a61b0 secure :-1 s=THROTTLE_DONE pgs=310 cs=0 l=1 rx=0x7f2a240a5fd0 tx=0x7f2a20011950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:03.437 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.421+0000 7f2a19ffb700 1 -- 172.21.15.42:0/3791901868 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2a20007150 con 0x7f2a24124150 2019-12-05T05:10:03.437 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.421+0000 7f2a2932c700 1 --2- 172.21.15.42:0/3791901868 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a24124150 0x7f2a240a61b0 secure :-1 s=THROTTLE_DONE pgs=310 cs=0 l=1 rx=0x7f2a240a5fd0 tx=0x7f2a20011950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:03.437 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.421+0000 7f2a2932c700 1 --2- 172.21.15.42:0/3791901868 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a24124150 0x7f2a240a61b0 secure :-1 s=THROTTLE_DONE pgs=310 cs=0 l=1 rx=0x7f2a240a5fd0 tx=0x7f2a20011950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:03.437 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.421+0000 7f2a19ffb700 1 -- 172.21.15.42:0/3791901868 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a200114b0 con 0x7f2a24124150 2019-12-05T05:10:03.437 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.425+0000 7f2a2ad8f700 1 -- 172.21.15.42:0/3791901868 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a240a9770 con 0x7f2a24124150 2019-12-05T05:10:03.438 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.425+0000 7f2a19ffb700 1 -- 172.21.15.42:0/3791901868 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2a20005730 con 0x7f2a24124150 2019-12-05T05:10:03.438 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.425+0000 7f2a2ad8f700 1 -- 172.21.15.42:0/3791901868 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a240a9cf0 con 0x7f2a24124150 2019-12-05T05:10:03.438 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.425+0000 7f2a2932c700 1 --2- 172.21.15.42:0/3791901868 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a24124150 0x7f2a240a61b0 secure :-1 s=THROTTLE_DONE pgs=310 cs=0 l=1 rx=0x7f2a240a5fd0 tx=0x7f2a20011950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:03.438 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.425+0000 7f2a19ffb700 1 -- 172.21.15.42:0/3791901868 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f2a20005a40 con 0x7f2a24124150 2019-12-05T05:10:03.438 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.425+0000 7f2a2932c700 1 --2- 172.21.15.42:0/3791901868 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a24124150 0x7f2a240a61b0 secure :-1 s=THROTTLE_DONE pgs=310 cs=0 l=1 rx=0x7f2a240a5fd0 tx=0x7f2a20011950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:03.438 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.425+0000 7f2a19ffb700 1 --2- 172.21.15.42:0/3791901868 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2a0403b2a0 0x7f2a0403d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:10:03.438 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.425+0000 7f2a19ffb700 1 -- 172.21.15.42:0/3791901868 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6375+0+0 (secure 0 0 0) 0x7f2a2004c5f0 con 0x7f2a24124150 2019-12-05T05:10:03.439 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.425+0000 7f2a28b2b700 1 --2- 172.21.15.42:0/3791901868 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2a0403b2a0 0x7f2a0403d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:10:03.439 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.425+0000 7f2a28b2b700 1 --2- 172.21.15.42:0/3791901868 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2a0403b2a0 0x7f2a0403d7b0 crc :-1 s=READY pgs=176 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:10:03.439 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.429+0000 7f2a2ad8f700 1 -- 172.21.15.42:0/3791901868 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a2405d110 con 0x7f2a24124150 2019-12-05T05:10:03.439 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.437+0000 7f2a2932c700 1 --2- 172.21.15.42:0/3791901868 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a24124150 0x7f2a240a61b0 secure :-1 s=THROTTLE_DONE pgs=310 cs=0 l=1 rx=0x7f2a240a5fd0 tx=0x7f2a20011950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:03.440 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.437+0000 7f2a19ffb700 1 -- 172.21.15.42:0/3791901868 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f2a20005d50 con 0x7f2a24124150 2019-12-05T05:10:03.601 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.597+0000 7f2a2ad8f700 1 -- 172.21.15.42:0/3791901868 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2a2411ad30 con 0x7f2a24124150 2019-12-05T05:10:03.601 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.597+0000 7f2a2932c700 1 --2- 172.21.15.42:0/3791901868 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a24124150 0x7f2a240a61b0 secure :-1 s=THROTTLE_DONE pgs=310 cs=0 l=1 rx=0x7f2a240a5fd0 tx=0x7f2a20011950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:03.601 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.597+0000 7f2a19ffb700 1 -- 172.21.15.42:0/3791901868 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) v1 ==== 94+0+1768 (secure 0 0 0) 0x7f2a200338d0 con 0x7f2a24124150 2019-12-05T05:10:03.602 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:10:03.602 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":36,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:10:03.270458+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4595},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4595":{"gid":4595,"name":"b","rank":0,"incarnation":36,"state":"up:replay","state_seq":2,"addr":"172.21.15.40:6833/211584380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":211584380},{"type":"v1","addr":"172.21.15.40:6833","nonce":211584380}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:10:03.602 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 36 2019-12-05T05:10:03.604 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.601+0000 7f2a2ad8f700 1 -- 172.21.15.42:0/3791901868 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2a0403b2a0 msgr2=0x7f2a0403d7b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:10:03.604 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.601+0000 7f2a2ad8f700 1 --2- 172.21.15.42:0/3791901868 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2a0403b2a0 0x7f2a0403d7b0 crc :-1 s=READY pgs=176 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:10:03.604 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.601+0000 7f2a2ad8f700 1 -- 172.21.15.42:0/3791901868 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a24124150 msgr2=0x7f2a240a61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:10:03.604 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.601+0000 7f2a2ad8f700 1 --2- 172.21.15.42:0/3791901868 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a24124150 0x7f2a240a61b0 secure :-1 s=READY pgs=310 cs=0 l=1 rx=0x7f2a240a5fd0 tx=0x7f2a20011950).stop 2019-12-05T05:10:03.604 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.601+0000 7f2a2ad8f700 1 -- 172.21.15.42:0/3791901868 shutdown_connections 2019-12-05T05:10:03.604 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.601+0000 7f2a2ad8f700 1 --2- 172.21.15.42:0/3791901868 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2a0403b2a0 0x7f2a0403d7b0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:10:03.605 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.601+0000 7f2a2ad8f700 1 --2- 172.21.15.42:0/3791901868 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2a240a5540 0x7f2a2407db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:10:03.605 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.601+0000 7f2a2ad8f700 1 --2- 172.21.15.42:0/3791901868 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a24124150 0x7f2a240a61b0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:10:03.605 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.601+0000 7f2a2ad8f700 1 -- 172.21.15.42:0/3791901868 shutdown_connections 2019-12-05T05:10:03.605 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.601+0000 7f2a2ad8f700 1 -- 172.21.15.42:0/3791901868 wait complete. 2019-12-05T05:10:03.605 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:03.601+0000 7f2a2ad8f700 1 -- 172.21.15.42:0/3791901868 >> 172.21.15.42:0/3791901868 conn(0x7f2a240aabe0 msgr2=0x7f2a24059ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:10:03.618 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 36, "feature_flags": { "enable_multiple": false, "ever_enabled_multiple": false }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2019-12-05T05:03:35.604738+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 36, "ever_allowed_features": 2, "explicitly_allowed_features": 2, "failed": [], "flags": 18, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_4595": { "addr": "172.21.15.40:6833/211584380", "addrs": { "addrvec": [ { "addr": "172.21.15.40:6832", "nonce": 211584380, "type": "v2" }, { "addr": "172.21.15.40:6833", "nonce": 211584380, "type": "v1" } ] }, "export_targets": [], "features": 4540138292836696063, "flags": 0, "gid": 4595, "incarnation": 36, "name": "b", "rank": 0, "state": "up:replay", "state_seq": 2 } }, "last_failure": 0, "last_failure_osd_epoch": 39, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 2, "min_compat_client": "0 (unknown)", "modified": "2019-12-05T05:10:03.270458+0000", "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4595 } } } ], "standbys": [] } 2019-12-05T05:10:05.098 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:10:05.102 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:10:05.621 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:10:05.736 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:05.717+0000 7f16a2b4f700 1 Processor -- start 2019-12-05T05:10:05.737 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:05.717+0000 7f16a2b4f700 1 -- start start 2019-12-05T05:10:05.737 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:05.717+0000 7f16a2b4f700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f169c08df80 0x7f169c09f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:10:05.737 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:05.717+0000 7f16a2b4f700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f169c09fc60 con 0x7f169c09f830 2019-12-05T05:10:05.737 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:05.717+0000 7f16a2b4f700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f169c0a0710 con 0x7f169c0a7740 2019-12-05T05:10:05.737 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:05.717+0000 7f16a2b4f700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f169c0a0b00 con 0x7f169c08df80 2019-12-05T05:10:05.737 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:05.717+0000 7f16a10ec700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f169c0a7740 0x7f169c07e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:57102/0 (socket says 172.21.15.42:57102) 2019-12-05T05:10:05.738 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:05.717+0000 7f16a10ec700 1 -- 172.21.15.42:0/1979705268 learned_addr learned my addr 172.21.15.42:0/1979705268 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:10:05.738 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:05.717+0000 7f16a08eb700 1 --2- 172.21.15.42:0/1979705268 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f169c08df80 0x7f169c09f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:10:05.738 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:05.717+0000 7f1693fff700 1 -- 172.21.15.42:0/1979705268 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3794617181 0 0) 0x7f169c0a0710 con 0x7f169c0a7740 2019-12-05T05:10:05.738 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:05.717+0000 7f1693fff700 1 -- 172.21.15.42:0/1979705268 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16880030b0 con 0x7f169c0a7740 2019-12-05T05:10:05.738 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:05.717+0000 7f1693fff700 1 -- 172.21.15.42:0/1979705268 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3156307196 0 0) 0x7f169c09fc60 con 0x7f169c09f830 2019-12-05T05:10:05.738 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:05.717+0000 7f1693fff700 1 -- 172.21.15.42:0/1979705268 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f169c0a0710 con 0x7f169c09f830 2019-12-05T05:10:05.738 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:05.717+0000 7f1693fff700 1 -- 172.21.15.42:0/1979705268 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3153361467 0 0) 0x7f16880030b0 con 0x7f169c0a7740 2019-12-05T05:10:05.738 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:05.721+0000 7f1693fff700 1 -- 172.21.15.42:0/1979705268 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f169c08df80 msgr2=0x7f169c09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:10:05.739 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:05.721+0000 7f1693fff700 1 --2- 172.21.15.42:0/1979705268 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f169c08df80 0x7f169c09f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:10:05.739 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:05.721+0000 7f1693fff700 1 -- 172.21.15.42:0/1979705268 >> v1:172.21.15.40:6789/0 conn(0x7f169c09f830 legacy=0x7f169c14efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:10:05.739 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:05.721+0000 7f1693fff700 1 -- 172.21.15.42:0/1979705268 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f169c07ef20 con 0x7f169c0a7740 2019-12-05T05:10:05.739 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:05.721+0000 7f1693fff700 1 -- 172.21.15.42:0/1979705268 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7f1698001750 con 0x7f169c0a7740 2019-12-05T05:10:05.739 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:05.721+0000 7f16a08eb700 1 --2- 172.21.15.42:0/1979705268 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f169c08df80 0x7f169c09f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:10:05.739 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:05.721+0000 7f1693fff700 1 -- 172.21.15.42:0/1979705268 >> v1:172.21.15.42:6789/0 conn(0x7f169c0a7740 legacy=0x7f169c07e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:10:05.739 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:05.721+0000 7f1693fff700 1 --2- 172.21.15.42:0/1979705268 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1688003940 0x7f1688003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:10:05.740 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:05.721+0000 7f1693fff700 1 --2- 172.21.15.42:0/1979705268 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1688004420 0x7f1688004870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:10:05.740 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:05.721+0000 7f1693fff700 1 --2- 172.21.15.42:0/1979705268 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1688004e30 0x7f168804f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:10:05.740 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:05.721+0000 7f16a10ec700 1 --2- 172.21.15.42:0/1979705268 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1688003940 0x7f1688003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:10:05.740 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:05.721+0000 7f1693fff700 1 -- 172.21.15.42:0/1979705268 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f169c0a0b00 con 0x7f1688003940 2019-12-05T05:10:05.740 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:05.721+0000 7f1693fff700 1 -- 172.21.15.42:0/1979705268 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f168804fc60 con 0x7f1688004e30 2019-12-05T05:10:05.740 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:05.721+0000 7f1693fff700 1 -- 172.21.15.42:0/1979705268 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f168804fe70 con 0x7f1688004420 2019-12-05T05:10:05.740 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:05.721+0000 7f16a18ed700 1 --2- 172.21.15.42:0/1979705268 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1688004e30 0x7f168804f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:10:05.741 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:05.721+0000 7f16a08eb700 1 --2- 172.21.15.42:0/1979705268 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1688004420 0x7f1688004870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:10:05.741 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:05.721+0000 7f16a18ed700 1 -- 172.21.15.42:0/1979705268 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1688004420 msgr2=0x7f1688004870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:10:05.741 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:05.721+0000 7f16a18ed700 1 --2- 172.21.15.42:0/1979705268 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1688004420 0x7f1688004870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:10:05.741 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:05.721+0000 7f16a18ed700 1 -- 172.21.15.42:0/1979705268 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1688003940 msgr2=0x7f1688003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:10:05.741 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:05.721+0000 7f16a18ed700 1 --2- 172.21.15.42:0/1979705268 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1688003940 0x7f1688003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:10:05.741 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:05.721+0000 7f16a18ed700 1 -- 172.21.15.42:0/1979705268 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1688050170 con 0x7f1688004e30 2019-12-05T05:10:05.741 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:05.721+0000 7f16a18ed700 1 --2- 172.21.15.42:0/1979705268 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1688004e30 0x7f168804f680 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f1694001360 tx=0x7f169400c1f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:05.742 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:05.721+0000 7f16a18ed700 1 --2- 172.21.15.42:0/1979705268 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1688004e30 0x7f168804f680 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f1694001360 tx=0x7f169400c1f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:05.742 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:05.721+0000 7f16a18ed700 1 --2- 172.21.15.42:0/1979705268 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1688004e30 0x7f168804f680 secure :-1 s=READY pgs=390 cs=0 l=1 rx=0x7f1694001360 tx=0x7f169400c1f0).ready entity=mon.1 client_cookie=25d01b8fec4351b6 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:10:05.742 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:05.721+0000 7f16a18ed700 1 --2- 172.21.15.42:0/1979705268 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1688004e30 0x7f168804f680 secure :-1 s=THROTTLE_DONE pgs=390 cs=0 l=1 rx=0x7f1694001360 tx=0x7f169400c1f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:05.742 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:05.721+0000 7f16a18ed700 1 --2- 172.21.15.42:0/1979705268 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1688004e30 0x7f168804f680 secure :-1 s=THROTTLE_DONE pgs=390 cs=0 l=1 rx=0x7f1694001360 tx=0x7f169400c1f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:05.742 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:05.721+0000 7f1693fff700 1 -- 172.21.15.42:0/1979705268 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f169400c740 con 0x7f1688004e30 2019-12-05T05:10:05.742 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:05.721+0000 7f16a18ed700 1 --2- 172.21.15.42:0/1979705268 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1688004e30 0x7f168804f680 secure :-1 s=THROTTLE_DONE pgs=390 cs=0 l=1 rx=0x7f1694001360 tx=0x7f169400c1f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:05.742 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:05.721+0000 7f1693fff700 1 -- 172.21.15.42:0/1979705268 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1694010b80 con 0x7f1688004e30 2019-12-05T05:10:05.743 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:05.721+0000 7f1693fff700 1 -- 172.21.15.42:0/1979705268 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f169400cd20 con 0x7f1688004e30 2019-12-05T05:10:05.743 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:05.721+0000 7f16a2b4f700 1 -- 172.21.15.42:0/1979705268 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1688004e30 msgr2=0x7f168804f680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:10:05.743 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:05.721+0000 7f16a2b4f700 1 --2- 172.21.15.42:0/1979705268 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1688004e30 0x7f168804f680 secure :-1 s=READY pgs=390 cs=0 l=1 rx=0x7f1694001360 tx=0x7f169400c1f0).stop 2019-12-05T05:10:05.743 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:05.721+0000 7f16a18ed700 1 -- 172.21.15.42:0/1979705268 reap_dead start 2019-12-05T05:10:05.744 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:05.721+0000 7f16a2b4f700 1 -- 172.21.15.42:0/1979705268 shutdown_connections 2019-12-05T05:10:05.744 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:05.721+0000 7f16a2b4f700 1 -- 172.21.15.42:0/1979705268 shutdown_connections 2019-12-05T05:10:05.744 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:05.721+0000 7f16a2b4f700 1 -- 172.21.15.42:0/1979705268 wait complete. 2019-12-05T05:10:05.744 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:05.721+0000 7f16a2b4f700 1 -- 172.21.15.42:0/1979705268 >> 172.21.15.42:0/1979705268 conn(0x7f169c0aabe0 msgr2=0x7f169c0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:10:05.744 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:05.725+0000 7f16a2b4f700 1 Processor -- start 2019-12-05T05:10:05.744 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:05.725+0000 7f16a2b4f700 1 -- start start 2019-12-05T05:10:05.744 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:05.725+0000 7f16a2b4f700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f169c12d3f0 0x7f169c12d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:10:05.745 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:05.725+0000 7f16a2b4f700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f169c12dd70 0x7f169c134b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:10:05.745 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:05.725+0000 7f16a2b4f700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f169c148330 con 0x7f169c146620 2019-12-05T05:10:05.745 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:05.725+0000 7f16a2b4f700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f169c1457e0 con 0x7f169c12dd70 2019-12-05T05:10:05.745 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:05.725+0000 7f16a2b4f700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f169c07c010 con 0x7f169c12d3f0 2019-12-05T05:10:05.745 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:05.729+0000 7f16a10ec700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f169c146620 0x7f169c147b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:57114/0 (socket says 172.21.15.42:57114) 2019-12-05T05:10:05.745 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:05.729+0000 7f16a10ec700 1 -- 172.21.15.42:0/740264836 learned_addr learned my addr 172.21.15.42:0/740264836 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:10:05.745 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:05.729+0000 7f16a18ed700 1 --2- 172.21.15.42:0/740264836 >> v2:172.21.15.42:3300/0 conn(0x7f169c12dd70 0x7f169c134b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:10:05.746 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:05.729+0000 7f16a08eb700 1 --2- 172.21.15.42:0/740264836 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f169c12d3f0 0x7f169c12d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:10:05.746 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:05.729+0000 7f1691ffb700 1 -- 172.21.15.42:0/740264836 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2615300021 0 0) 0x7f169c148330 con 0x7f169c146620 2019-12-05T05:10:05.746 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:05.729+0000 7f1691ffb700 1 -- 172.21.15.42:0/740264836 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f167c0030b0 con 0x7f169c146620 2019-12-05T05:10:05.746 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:05.729+0000 7f16a18ed700 1 -- 172.21.15.42:0/740264836 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f169c12d3f0 msgr2=0x7f169c12d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:10:05.746 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:05.729+0000 7f16a18ed700 1 --2- 172.21.15.42:0/740264836 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f169c12d3f0 0x7f169c12d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:10:05.746 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:05.729+0000 7f16a18ed700 1 -- 172.21.15.42:0/740264836 >> v1:172.21.15.42:6789/0 conn(0x7f169c146620 legacy=0x7f169c147b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:10:05.746 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:05.729+0000 7f16a18ed700 1 -- 172.21.15.42:0/740264836 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f169c1486c0 con 0x7f169c12dd70 2019-12-05T05:10:05.747 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:05.729+0000 7f16a18ed700 1 --2- 172.21.15.42:0/740264836 >> v2:172.21.15.42:3300/0 conn(0x7f169c12dd70 0x7f169c134b40 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f169c1459a0 tx=0x7f169400cd40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:05.747 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:05.729+0000 7f16a18ed700 1 --2- 172.21.15.42:0/740264836 >> v2:172.21.15.42:3300/0 conn(0x7f169c12dd70 0x7f169c134b40 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f169c1459a0 tx=0x7f169400cd40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:05.747 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:05.729+0000 7f16a18ed700 1 --2- 172.21.15.42:0/740264836 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f169c12dd70 0x7f169c134b40 secure :-1 s=READY pgs=392 cs=0 l=1 rx=0x7f169c1459a0 tx=0x7f169400cd40).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:10:05.747 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:05.729+0000 7f16a18ed700 1 --2- 172.21.15.42:0/740264836 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f169c12dd70 0x7f169c134b40 secure :-1 s=THROTTLE_DONE pgs=392 cs=0 l=1 rx=0x7f169c1459a0 tx=0x7f169400cd40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:05.747 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:05.729+0000 7f1691ffb700 1 -- 172.21.15.42:0/740264836 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1694010460 con 0x7f169c12dd70 2019-12-05T05:10:05.747 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:05.729+0000 7f16a18ed700 1 --2- 172.21.15.42:0/740264836 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f169c12dd70 0x7f169c134b40 secure :-1 s=THROTTLE_DONE pgs=392 cs=0 l=1 rx=0x7f169c1459a0 tx=0x7f169400cd40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:05.747 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:05.729+0000 7f16a18ed700 1 --2- 172.21.15.42:0/740264836 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f169c12dd70 0x7f169c134b40 secure :-1 s=THROTTLE_DONE pgs=392 cs=0 l=1 rx=0x7f169c1459a0 tx=0x7f169400cd40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:05.748 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:05.729+0000 7f1691ffb700 1 -- 172.21.15.42:0/740264836 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f169400c940 con 0x7f169c12dd70 2019-12-05T05:10:05.748 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:05.729+0000 7f16a2b4f700 1 -- 172.21.15.42:0/740264836 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f169c12e1a0 con 0x7f169c12dd70 2019-12-05T05:10:05.748 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:05.729+0000 7f1691ffb700 1 -- 172.21.15.42:0/740264836 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1694002430 con 0x7f169c12dd70 2019-12-05T05:10:05.748 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:05.729+0000 7f16a2b4f700 1 -- 172.21.15.42:0/740264836 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f169c146a50 con 0x7f169c12dd70 2019-12-05T05:10:05.748 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:05.729+0000 7f16a18ed700 1 --2- 172.21.15.42:0/740264836 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f169c12dd70 0x7f169c134b40 secure :-1 s=THROTTLE_DONE pgs=392 cs=0 l=1 rx=0x7f169c1459a0 tx=0x7f169400cd40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:05.748 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:05.729+0000 7f1691ffb700 1 -- 172.21.15.42:0/740264836 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f16940066e0 con 0x7f169c12dd70 2019-12-05T05:10:05.748 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:05.729+0000 7f16a18ed700 1 --2- 172.21.15.42:0/740264836 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f169c12dd70 0x7f169c134b40 secure :-1 s=THROTTLE_DONE pgs=392 cs=0 l=1 rx=0x7f169c1459a0 tx=0x7f169400cd40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:05.749 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:05.729+0000 7f1691ffb700 1 --2- 172.21.15.42:0/740264836 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f167c03b2b0 0x7f167c03d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:10:05.749 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:05.733+0000 7f1691ffb700 1 -- 172.21.15.42:0/740264836 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6375+0+0 (secure 0 0 0) 0x7f169404ae40 con 0x7f169c12dd70 2019-12-05T05:10:05.749 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:05.733+0000 7f16a10ec700 1 --2- 172.21.15.42:0/740264836 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f167c03b2b0 0x7f167c03d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:10:05.749 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:05.733+0000 7f16a10ec700 1 --2- 172.21.15.42:0/740264836 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f167c03b2b0 0x7f167c03d7c0 crc :-1 s=READY pgs=177 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:10:05.749 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:05.733+0000 7f16a2b4f700 1 -- 172.21.15.42:0/740264836 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f169c05d110 con 0x7f169c12dd70 2019-12-05T05:10:05.750 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:05.741+0000 7f16a18ed700 1 --2- 172.21.15.42:0/740264836 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f169c12dd70 0x7f169c134b40 secure :-1 s=THROTTLE_DONE pgs=392 cs=0 l=1 rx=0x7f169c1459a0 tx=0x7f169400cd40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:05.750 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:05.741+0000 7f1691ffb700 1 -- 172.21.15.42:0/740264836 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f1694032120 con 0x7f169c12dd70 2019-12-05T05:10:05.903 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:05.897+0000 7f16a2b4f700 1 -- 172.21.15.42:0/740264836 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f169c05b8c0 con 0x7f169c12dd70 2019-12-05T05:10:05.952 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:05.901+0000 7f16a18ed700 1 --2- 172.21.15.42:0/740264836 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f169c12dd70 0x7f169c134b40 secure :-1 s=THROTTLE_DONE pgs=392 cs=0 l=1 rx=0x7f169c1459a0 tx=0x7f169400cd40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:05.953 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:05.901+0000 7f1691ffb700 1 -- 172.21.15.42:0/740264836 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) v1 ==== 94+0+1771 (secure 0 0 0) 0x7f1694032cd0 con 0x7f169c12dd70 2019-12-05T05:10:05.953 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 37 2019-12-05T05:10:05.953 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:05.905+0000 7f16a2b4f700 1 -- 172.21.15.42:0/740264836 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f167c03b2b0 msgr2=0x7f167c03d7c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:10:05.953 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:05.905+0000 7f16a2b4f700 1 --2- 172.21.15.42:0/740264836 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f167c03b2b0 0x7f167c03d7c0 crc :-1 s=READY pgs=177 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:10:05.954 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:05.905+0000 7f16a2b4f700 1 -- 172.21.15.42:0/740264836 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f169c12dd70 msgr2=0x7f169c134b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:10:05.954 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:05.905+0000 7f16a2b4f700 1 --2- 172.21.15.42:0/740264836 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f169c12dd70 0x7f169c134b40 secure :-1 s=READY pgs=392 cs=0 l=1 rx=0x7f169c1459a0 tx=0x7f169400cd40).stop 2019-12-05T05:10:05.954 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:05.905+0000 7f16a2b4f700 1 -- 172.21.15.42:0/740264836 shutdown_connections 2019-12-05T05:10:05.955 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:05.905+0000 7f16a2b4f700 1 --2- 172.21.15.42:0/740264836 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f167c03b2b0 0x7f167c03d7c0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:10:05.955 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:05.905+0000 7f16a2b4f700 1 --2- 172.21.15.42:0/740264836 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f169c12dd70 0x7f169c134b40 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:10:05.955 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:05.905+0000 7f16a2b4f700 1 --2- 172.21.15.42:0/740264836 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f169c12d3f0 0x7f169c12d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:10:05.955 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:05.905+0000 7f16a2b4f700 1 -- 172.21.15.42:0/740264836 shutdown_connections 2019-12-05T05:10:05.955 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:05.905+0000 7f16a2b4f700 1 -- 172.21.15.42:0/740264836 wait complete. 2019-12-05T05:10:05.956 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:05.905+0000 7f16a2b4f700 1 -- 172.21.15.42:0/740264836 >> 172.21.15.42:0/740264836 conn(0x7f169c0a1000 msgr2=0x7f169c07c7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:10:05.956 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:10:05.956 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":37,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:10:05.372500+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4595},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4595":{"gid":4595,"name":"b","rank":0,"incarnation":36,"state":"up:reconnect","state_seq":8,"addr":"172.21.15.40:6833/211584380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":211584380},{"type":"v1","addr":"172.21.15.40:6833","nonce":211584380}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:10:05.957 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2019-12-05T05:10:07.958 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:10:08.065 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:08.053+0000 7f3f6b657700 1 Processor -- start 2019-12-05T05:10:08.065 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:08.057+0000 7f3f6b657700 1 -- start start 2019-12-05T05:10:08.065 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:08.057+0000 7f3f6b657700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3f640a7740 0x7f3f6408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:10:08.066 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:08.057+0000 7f3f6b657700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3f6407e720 0x7f3f6407eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:10:08.066 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:08.057+0000 7f3f6b657700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f640a0710 con 0x7f3f6409f2e0 2019-12-05T05:10:08.066 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:08.057+0000 7f3f6b657700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f640a0b00 con 0x7f3f6407e720 2019-12-05T05:10:08.066 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:08.057+0000 7f3f6b657700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f6408e5b0 con 0x7f3f640a7740 2019-12-05T05:10:08.067 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:08.057+0000 7f3f693f3700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3f6407e720 0x7f3f6407eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:10:08.067 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:08.057+0000 7f3f693f3700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3f6407e720 0x7f3f6407eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:57086/0 (socket says 172.21.15.42:57086) 2019-12-05T05:10:08.067 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:08.057+0000 7f3f693f3700 1 -- 172.21.15.42:0/2577091940 learned_addr learned my addr 172.21.15.42:0/2577091940 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:10:08.067 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:08.057+0000 7f3f69bf4700 1 --2- 172.21.15.42:0/2577091940 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3f640a7740 0x7f3f6408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:10:08.067 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:08.057+0000 7f3f68bf2700 1 -- 172.21.15.42:0/2577091940 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2349711193 0 0) 0x7f3f640a0710 con 0x7f3f6409f2e0 2019-12-05T05:10:08.067 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:08.057+0000 7f3f693f3700 1 -- 172.21.15.42:0/2577091940 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3f640a7740 msgr2=0x7f3f6408df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:10:08.068 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:08.057+0000 7f3f693f3700 1 --2- 172.21.15.42:0/2577091940 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3f640a7740 0x7f3f6408df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:10:08.068 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:08.057+0000 7f3f693f3700 1 -- 172.21.15.42:0/2577091940 >> v1:172.21.15.40:6789/0 conn(0x7f3f6409f2e0 legacy=0x7f3f6409f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:10:08.068 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:08.057+0000 7f3f693f3700 1 -- 172.21.15.42:0/2577091940 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f6407f0a0 con 0x7f3f6407e720 2019-12-05T05:10:08.068 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:08.057+0000 7f3f693f3700 1 --2- 172.21.15.42:0/2577091940 >> v2:172.21.15.42:3300/0 conn(0x7f3f6407e720 0x7f3f6407eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f3f58008790 tx=0x7f3f5800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:08.068 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:08.057+0000 7f3f693f3700 1 --2- 172.21.15.42:0/2577091940 >> v2:172.21.15.42:3300/0 conn(0x7f3f6407e720 0x7f3f6407eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f3f58008790 tx=0x7f3f5800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:08.068 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:08.057+0000 7f3f693f3700 1 --2- 172.21.15.42:0/2577091940 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3f6407e720 0x7f3f6407eb50 secure :-1 s=READY pgs=393 cs=0 l=1 rx=0x7f3f58008790 tx=0x7f3f5800fd30).ready entity=mon.1 client_cookie=dcc9c705932f0cb3 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:10:08.068 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:08.057+0000 7f3f693f3700 1 --2- 172.21.15.42:0/2577091940 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3f6407e720 0x7f3f6407eb50 secure :-1 s=THROTTLE_DONE pgs=393 cs=0 l=1 rx=0x7f3f58008790 tx=0x7f3f5800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:08.069 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:08.057+0000 7f3f68bf2700 1 -- 172.21.15.42:0/2577091940 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3f580113b0 con 0x7f3f6407e720 2019-12-05T05:10:08.069 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:08.057+0000 7f3f693f3700 1 --2- 172.21.15.42:0/2577091940 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3f6407e720 0x7f3f6407eb50 secure :-1 s=THROTTLE_DONE pgs=393 cs=0 l=1 rx=0x7f3f58008790 tx=0x7f3f5800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:08.069 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:08.057+0000 7f3f693f3700 1 --2- 172.21.15.42:0/2577091940 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3f6407e720 0x7f3f6407eb50 secure :-1 s=THROTTLE_DONE pgs=393 cs=0 l=1 rx=0x7f3f58008790 tx=0x7f3f5800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:08.069 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:08.057+0000 7f3f68bf2700 1 -- 172.21.15.42:0/2577091940 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f58008bb0 con 0x7f3f6407e720 2019-12-05T05:10:08.069 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:08.057+0000 7f3f68bf2700 1 -- 172.21.15.42:0/2577091940 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3f58011990 con 0x7f3f6407e720 2019-12-05T05:10:08.070 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:08.057+0000 7f3f6b657700 1 -- 172.21.15.42:0/2577091940 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3f6407e720 msgr2=0x7f3f6407eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:10:08.070 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:08.057+0000 7f3f6b657700 1 --2- 172.21.15.42:0/2577091940 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3f6407e720 0x7f3f6407eb50 secure :-1 s=READY pgs=393 cs=0 l=1 rx=0x7f3f58008790 tx=0x7f3f5800fd30).stop 2019-12-05T05:10:08.070 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:08.061+0000 7f3f6b657700 1 -- 172.21.15.42:0/2577091940 shutdown_connections 2019-12-05T05:10:08.070 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:08.061+0000 7f3f6b657700 1 --2- 172.21.15.42:0/2577091940 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3f640a7740 0x7f3f6408df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:10:08.070 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:08.061+0000 7f3f6b657700 1 --2- 172.21.15.42:0/2577091940 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3f6407e720 0x7f3f6407eb50 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:10:08.070 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:08.061+0000 7f3f6b657700 1 -- 172.21.15.42:0/2577091940 shutdown_connections 2019-12-05T05:10:08.071 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:08.061+0000 7f3f6b657700 1 -- 172.21.15.42:0/2577091940 wait complete. 2019-12-05T05:10:08.071 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:08.061+0000 7f3f6b657700 1 -- 172.21.15.42:0/2577091940 >> 172.21.15.42:0/2577091940 conn(0x7f3f640aabe0 msgr2=0x7f3f640a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:10:08.075 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:08.061+0000 7f3f6b657700 1 Processor -- start 2019-12-05T05:10:08.075 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:08.061+0000 7f3f6b657700 1 -- start start 2019-12-05T05:10:08.075 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:08.065+0000 7f3f6b657700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f3f6412efb0 0x7f3f64123380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:10:08.076 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:08.065+0000 7f3f6b657700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3f6411bed0 0x7f3f640a3c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:10:08.076 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:08.065+0000 7f3f6b657700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3f640a20d0 0x7f3f640a2500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:10:08.076 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:08.065+0000 7f3f6b657700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f640a2a50 con 0x7f3f6412efb0 2019-12-05T05:10:08.077 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:08.065+0000 7f3f6b657700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f640a16b0 con 0x7f3f640a20d0 2019-12-05T05:10:08.077 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:08.065+0000 7f3f6b657700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f641238d0 con 0x7f3f6411bed0 2019-12-05T05:10:08.078 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:08.065+0000 7f3f6a3f5700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3f640a20d0 0x7f3f640a2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:10:08.078 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:08.065+0000 7f3f6a3f5700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3f640a20d0 0x7f3f640a2500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:57092/0 (socket says 172.21.15.42:57092) 2019-12-05T05:10:08.078 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:08.065+0000 7f3f6a3f5700 1 -- 172.21.15.42:0/3091215085 learned_addr learned my addr 172.21.15.42:0/3091215085 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:10:08.078 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:08.065+0000 7f3f69bf4700 1 --2- 172.21.15.42:0/3091215085 >> v2:172.21.15.40:3300/0 conn(0x7f3f6412efb0 0x7f3f64123380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:10:08.079 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:08.065+0000 7f3f693f3700 1 --2- 172.21.15.42:0/3091215085 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3f6411bed0 0x7f3f640a3c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:10:08.079 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:08.065+0000 7f3f6a3f5700 1 -- 172.21.15.42:0/3091215085 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3f6411bed0 msgr2=0x7f3f640a3c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:10:08.079 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:08.065+0000 7f3f6a3f5700 1 --2- 172.21.15.42:0/3091215085 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3f6411bed0 0x7f3f640a3c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:10:08.079 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:08.065+0000 7f3f6a3f5700 1 -- 172.21.15.42:0/3091215085 >> v2:172.21.15.40:3300/0 conn(0x7f3f6412efb0 msgr2=0x7f3f64123380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:10:08.080 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:08.065+0000 7f3f6a3f5700 1 --2- 172.21.15.42:0/3091215085 >> v2:172.21.15.40:3300/0 conn(0x7f3f6412efb0 0x7f3f64123380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:10:08.080 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:08.065+0000 7f3f6a3f5700 1 -- 172.21.15.42:0/3091215085 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f640a4160 con 0x7f3f640a20d0 2019-12-05T05:10:08.080 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:08.065+0000 7f3f6a3f5700 1 --2- 172.21.15.42:0/3091215085 >> v2:172.21.15.42:3300/0 conn(0x7f3f640a20d0 0x7f3f640a2500 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f3f5c000f00 tx=0x7f3f5c010b80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:08.080 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:08.065+0000 7f3f6a3f5700 1 --2- 172.21.15.42:0/3091215085 >> v2:172.21.15.42:3300/0 conn(0x7f3f640a20d0 0x7f3f640a2500 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f3f5c000f00 tx=0x7f3f5c010b80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:08.081 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:08.065+0000 7f3f6a3f5700 1 --2- 172.21.15.42:0/3091215085 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3f640a20d0 0x7f3f640a2500 secure :-1 s=READY pgs=394 cs=0 l=1 rx=0x7f3f5c000f00 tx=0x7f3f5c010b80).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:10:08.081 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:08.065+0000 7f3f6a3f5700 1 --2- 172.21.15.42:0/3091215085 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3f640a20d0 0x7f3f640a2500 secure :-1 s=THROTTLE_DONE pgs=394 cs=0 l=1 rx=0x7f3f5c000f00 tx=0x7f3f5c010b80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:08.081 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:08.065+0000 7f3f567fc700 1 -- 172.21.15.42:0/3091215085 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3f5c006dc0 con 0x7f3f640a20d0 2019-12-05T05:10:08.081 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:08.065+0000 7f3f6a3f5700 1 --2- 172.21.15.42:0/3091215085 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3f640a20d0 0x7f3f640a2500 secure :-1 s=THROTTLE_DONE pgs=394 cs=0 l=1 rx=0x7f3f5c000f00 tx=0x7f3f5c010b80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:08.082 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:08.065+0000 7f3f6a3f5700 1 --2- 172.21.15.42:0/3091215085 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3f640a20d0 0x7f3f640a2500 secure :-1 s=THROTTLE_DONE pgs=394 cs=0 l=1 rx=0x7f3f5c000f00 tx=0x7f3f5c010b80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:08.082 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:08.065+0000 7f3f567fc700 1 -- 172.21.15.42:0/3091215085 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f5c0069e0 con 0x7f3f640a20d0 2019-12-05T05:10:08.082 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:08.065+0000 7f3f6b657700 1 -- 172.21.15.42:0/3091215085 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f6411a260 con 0x7f3f640a20d0 2019-12-05T05:10:08.082 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:08.065+0000 7f3f567fc700 1 -- 172.21.15.42:0/3091215085 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3f5c002620 con 0x7f3f640a20d0 2019-12-05T05:10:08.083 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:08.065+0000 7f3f6b657700 1 -- 172.21.15.42:0/3091215085 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f6411a810 con 0x7f3f640a20d0 2019-12-05T05:10:08.083 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:08.065+0000 7f3f6a3f5700 1 --2- 172.21.15.42:0/3091215085 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3f640a20d0 0x7f3f640a2500 secure :-1 s=THROTTLE_DONE pgs=394 cs=0 l=1 rx=0x7f3f5c000f00 tx=0x7f3f5c010b80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:08.083 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:08.069+0000 7f3f567fc700 1 -- 172.21.15.42:0/3091215085 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f3f5c0028f0 con 0x7f3f640a20d0 2019-12-05T05:10:08.083 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:08.069+0000 7f3f6a3f5700 1 --2- 172.21.15.42:0/3091215085 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3f640a20d0 0x7f3f640a2500 secure :-1 s=THROTTLE_DONE pgs=394 cs=0 l=1 rx=0x7f3f5c000f00 tx=0x7f3f5c010b80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:08.084 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:08.069+0000 7f3f567fc700 1 --2- 172.21.15.42:0/3091215085 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3f4c03ad40 0x7f3f4c03d250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:10:08.084 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:08.069+0000 7f3f567fc700 1 -- 172.21.15.42:0/3091215085 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6375+0+0 (secure 0 0 0) 0x7f3f5c04b560 con 0x7f3f640a20d0 2019-12-05T05:10:08.084 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:08.069+0000 7f3f69bf4700 1 --2- 172.21.15.42:0/3091215085 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3f4c03ad40 0x7f3f4c03d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:10:08.084 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:08.069+0000 7f3f69bf4700 1 --2- 172.21.15.42:0/3091215085 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3f4c03ad40 0x7f3f4c03d250 crc :-1 s=READY pgs=178 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:10:08.086 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:08.073+0000 7f3f6b657700 1 -- 172.21.15.42:0/3091215085 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f6405d110 con 0x7f3f640a20d0 2019-12-05T05:10:08.086 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:08.081+0000 7f3f6a3f5700 1 --2- 172.21.15.42:0/3091215085 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3f640a20d0 0x7f3f640a2500 secure :-1 s=THROTTLE_DONE pgs=394 cs=0 l=1 rx=0x7f3f5c000f00 tx=0x7f3f5c010b80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:08.086 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:08.081+0000 7f3f567fc700 1 -- 172.21.15.42:0/3091215085 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f3f5c0325e0 con 0x7f3f640a20d0 2019-12-05T05:10:08.258 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:08.253+0000 7f3f6b657700 1 -- 172.21.15.42:0/3091215085 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3f6405b8c0 con 0x7f3f640a20d0 2019-12-05T05:10:08.259 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:08.253+0000 7f3f6a3f5700 1 --2- 172.21.15.42:0/3091215085 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3f640a20d0 0x7f3f640a2500 secure :-1 s=THROTTLE_DONE pgs=394 cs=0 l=1 rx=0x7f3f5c000f00 tx=0x7f3f5c010b80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:08.259 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:08.253+0000 7f3f567fc700 1 -- 172.21.15.42:0/3091215085 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) v1 ==== 94+0+1775 (secure 0 0 0) 0x7f3f5c032840 con 0x7f3f640a20d0 2019-12-05T05:10:08.260 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 39 2019-12-05T05:10:08.260 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:10:08.260 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":39,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":39,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:10:07.385453+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4595},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4595":{"gid":4595,"name":"b","rank":0,"incarnation":36,"state":"up:clientreplay","state_seq":10,"addr":"172.21.15.40:6833/211584380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":211584380},{"type":"v1","addr":"172.21.15.40:6833","nonce":211584380}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:10:08.262 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:08.257+0000 7f3f6b657700 1 -- 172.21.15.42:0/3091215085 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3f4c03ad40 msgr2=0x7f3f4c03d250 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:10:08.262 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:08.257+0000 7f3f6b657700 1 --2- 172.21.15.42:0/3091215085 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3f4c03ad40 0x7f3f4c03d250 crc :-1 s=READY pgs=178 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:10:08.263 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:08.257+0000 7f3f6b657700 1 -- 172.21.15.42:0/3091215085 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3f640a20d0 msgr2=0x7f3f640a2500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:10:08.263 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:08.257+0000 7f3f6b657700 1 --2- 172.21.15.42:0/3091215085 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3f640a20d0 0x7f3f640a2500 secure :-1 s=READY pgs=394 cs=0 l=1 rx=0x7f3f5c000f00 tx=0x7f3f5c010b80).stop 2019-12-05T05:10:08.263 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:08.257+0000 7f3f6b657700 1 -- 172.21.15.42:0/3091215085 shutdown_connections 2019-12-05T05:10:08.263 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:08.257+0000 7f3f6b657700 1 --2- 172.21.15.42:0/3091215085 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3f4c03ad40 0x7f3f4c03d250 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:10:08.263 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:08.257+0000 7f3f6b657700 1 --2- 172.21.15.42:0/3091215085 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3f640a20d0 0x7f3f640a2500 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:10:08.264 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:08.257+0000 7f3f6b657700 1 --2- 172.21.15.42:0/3091215085 >> v2:172.21.15.40:3300/0 conn(0x7f3f6412efb0 0x7f3f64123380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:10:08.264 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:08.257+0000 7f3f6b657700 1 --2- 172.21.15.42:0/3091215085 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3f6411bed0 0x7f3f640a3c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:10:08.264 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:08.257+0000 7f3f6b657700 1 -- 172.21.15.42:0/3091215085 shutdown_connections 2019-12-05T05:10:08.264 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:08.257+0000 7f3f6b657700 1 -- 172.21.15.42:0/3091215085 wait complete. 2019-12-05T05:10:08.265 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:08.257+0000 7f3f6b657700 1 -- 172.21.15.42:0/3091215085 >> 172.21.15.42:0/3091215085 conn(0x7f3f6407e870 msgr2=0x7f3f6411c470 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:10:08.272 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2019-12-05T05:10:10.273 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:10:10.383 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:10.373+0000 7f33de6c7700 1 Processor -- start 2019-12-05T05:10:10.384 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:10.373+0000 7f33de6c7700 1 -- start start 2019-12-05T05:10:10.384 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:10.377+0000 7f33de6c7700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f33d80a7740 0x7f33d808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:10:10.384 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:10.377+0000 7f33de6c7700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f33d807e720 0x7f33d807eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:10:10.384 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:10.377+0000 7f33de6c7700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f33d809f2e0 0x7f33d809f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:10:10.384 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:10.377+0000 7f33de6c7700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33d80a0b00 con 0x7f33d807e720 2019-12-05T05:10:10.385 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:10.377+0000 7f33de6c7700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33d808e5b0 con 0x7f33d80a7740 2019-12-05T05:10:10.385 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:10.377+0000 7f33de6c7700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33d808e850 con 0x7f33d809f2e0 2019-12-05T05:10:10.385 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:10.377+0000 7f33dcc64700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f33d80a7740 0x7f33d808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:10:10.389 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:10.377+0000 7f33dcc64700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f33d80a7740 0x7f33d808df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:57128/0 (socket says 172.21.15.42:57128) 2019-12-05T05:10:10.390 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:10.377+0000 7f33dcc64700 1 -- 172.21.15.42:0/3653253132 learned_addr learned my addr 172.21.15.42:0/3653253132 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:10:10.391 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:10.377+0000 7f33dd465700 1 --2- 172.21.15.42:0/3653253132 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f33d809f2e0 0x7f33d809f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:10:10.391 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:10.377+0000 7f33dcc64700 1 -- 172.21.15.42:0/3653253132 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f33d809f2e0 msgr2=0x7f33d809f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:10:10.392 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:10.377+0000 7f33dcc64700 1 --2- 172.21.15.42:0/3653253132 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f33d809f2e0 0x7f33d809f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:10:10.395 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:10.377+0000 7f33dcc64700 1 -- 172.21.15.42:0/3653253132 >> v2:172.21.15.40:3300/0 conn(0x7f33d807e720 msgr2=0x7f33d807eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:10:10.395 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:10.377+0000 7f33dcc64700 1 --2- 172.21.15.42:0/3653253132 >> v2:172.21.15.40:3300/0 conn(0x7f33d807e720 0x7f33d807eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:10:10.396 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:10.377+0000 7f33dcc64700 1 -- 172.21.15.42:0/3653253132 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33d807f0a0 con 0x7f33d80a7740 2019-12-05T05:10:10.396 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:10.377+0000 7f33dcc64700 1 --2- 172.21.15.42:0/3653253132 >> v2:172.21.15.42:3300/0 conn(0x7f33d80a7740 0x7f33d808df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f33d80a0f50 tx=0x7f33d0010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:10.396 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:10.377+0000 7f33dcc64700 1 --2- 172.21.15.42:0/3653253132 >> v2:172.21.15.42:3300/0 conn(0x7f33d80a7740 0x7f33d808df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f33d80a0f50 tx=0x7f33d0010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:10.399 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:10.377+0000 7f33dcc64700 1 --2- 172.21.15.42:0/3653253132 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f33d80a7740 0x7f33d808df80 secure :-1 s=READY pgs=395 cs=0 l=1 rx=0x7f33d80a0f50 tx=0x7f33d0010130).ready entity=mon.1 client_cookie=287c90aaf209ff42 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:10:10.400 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:10.377+0000 7f33dcc64700 1 --2- 172.21.15.42:0/3653253132 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f33d80a7740 0x7f33d808df80 secure :-1 s=THROTTLE_DONE pgs=395 cs=0 l=1 rx=0x7f33d80a0f50 tx=0x7f33d0010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:10.400 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:10.377+0000 7f33d77fe700 1 -- 172.21.15.42:0/3653253132 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f33d0010de0 con 0x7f33d80a7740 2019-12-05T05:10:10.400 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:10.377+0000 7f33dcc64700 1 --2- 172.21.15.42:0/3653253132 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f33d80a7740 0x7f33d808df80 secure :-1 s=THROTTLE_DONE pgs=395 cs=0 l=1 rx=0x7f33d80a0f50 tx=0x7f33d0010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:10.400 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:10.377+0000 7f33d77fe700 1 -- 172.21.15.42:0/3653253132 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33d0010a00 con 0x7f33d80a7740 2019-12-05T05:10:10.416 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:10.377+0000 7f33dcc64700 1 --2- 172.21.15.42:0/3653253132 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f33d80a7740 0x7f33d808df80 secure :-1 s=THROTTLE_DONE pgs=395 cs=0 l=1 rx=0x7f33d80a0f50 tx=0x7f33d0010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:10.417 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:10.377+0000 7f33de6c7700 1 -- 172.21.15.42:0/3653253132 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f33d80a7740 msgr2=0x7f33d808df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:10:10.417 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:10.377+0000 7f33d77fe700 1 -- 172.21.15.42:0/3653253132 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f33d00114a0 con 0x7f33d80a7740 2019-12-05T05:10:10.417 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:10.377+0000 7f33de6c7700 1 --2- 172.21.15.42:0/3653253132 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f33d80a7740 0x7f33d808df80 secure :-1 s=READY pgs=395 cs=0 l=1 rx=0x7f33d80a0f50 tx=0x7f33d0010130).stop 2019-12-05T05:10:10.417 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:10.377+0000 7f33de6c7700 1 -- 172.21.15.42:0/3653253132 shutdown_connections 2019-12-05T05:10:10.417 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:10.377+0000 7f33de6c7700 1 --2- 172.21.15.42:0/3653253132 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f33d809f2e0 0x7f33d809f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:10:10.418 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:10.377+0000 7f33de6c7700 1 --2- 172.21.15.42:0/3653253132 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f33d80a7740 0x7f33d808df80 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:10:10.418 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:10.377+0000 7f33de6c7700 1 --2- 172.21.15.42:0/3653253132 >> v2:172.21.15.40:3300/0 conn(0x7f33d807e720 0x7f33d807eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:10:10.418 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:10.377+0000 7f33de6c7700 1 -- 172.21.15.42:0/3653253132 shutdown_connections 2019-12-05T05:10:10.418 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:10.377+0000 7f33de6c7700 1 -- 172.21.15.42:0/3653253132 wait complete. 2019-12-05T05:10:10.418 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:10.377+0000 7f33de6c7700 1 -- 172.21.15.42:0/3653253132 >> 172.21.15.42:0/3653253132 conn(0x7f33d80aabe0 msgr2=0x7f33d80a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:10:10.427 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:10.381+0000 7f33de6c7700 1 Processor -- start 2019-12-05T05:10:10.428 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:10.381+0000 7f33de6c7700 1 -- start start 2019-12-05T05:10:10.428 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:10.381+0000 7f33de6c7700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f33d8142c90 0x7f33d8124a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:10:10.428 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:10.381+0000 7f33de6c7700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f33d8124fa0 0x7f33d81357f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:10:10.428 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:10.381+0000 7f33de6c7700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33d81430c0 con 0x7f33d8114790 2019-12-05T05:10:10.428 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:10.381+0000 7f33de6c7700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33d8113700 con 0x7f33d8142c90 2019-12-05T05:10:10.429 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:10.381+0000 7f33de6c7700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33d81139a0 con 0x7f33d8124fa0 2019-12-05T05:10:10.429 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:10.381+0000 7f33dd465700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f33d8124fa0 0x7f33d81357f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:10:10.429 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:10.381+0000 7f33d7fff700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f33d8142c90 0x7f33d8124a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:10:10.429 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:10.381+0000 7f33dd465700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f33d8124fa0 0x7f33d81357f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:57598/0 (socket says 172.21.15.42:57598) 2019-12-05T05:10:10.429 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:10.381+0000 7f33d7fff700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f33d8142c90 0x7f33d8124a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.42:44950/0 (socket says 172.21.15.42:44950) 2019-12-05T05:10:10.430 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:10.381+0000 7f33dd465700 1 -- 172.21.15.42:0/3335505348 learned_addr learned my addr 172.21.15.42:0/3335505348 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:10:10.430 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:10.381+0000 7f33d7fff700 1 -- 172.21.15.42:0/3335505348 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f33d8124fa0 msgr2=0x7f33d81357f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:10:10.430 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:10.381+0000 7f33d7fff700 1 --2- 172.21.15.42:0/3335505348 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f33d8124fa0 0x7f33d81357f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:10:10.430 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:10.381+0000 7f33d7fff700 1 -- 172.21.15.42:0/3335505348 >> v1:172.21.15.40:6789/0 conn(0x7f33d8114790 legacy=0x7f33d80a2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:10:10.430 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:10.381+0000 7f33d7fff700 1 -- 172.21.15.42:0/3335505348 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33d8113ee0 con 0x7f33d8142c90 2019-12-05T05:10:10.430 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:10.381+0000 7f33dd465700 1 --2- 172.21.15.42:0/3335505348 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f33d8124fa0 0x7f33d81357f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:10:10.436 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:10.381+0000 7f33d7fff700 1 --2- 172.21.15.42:0/3335505348 >> v2:172.21.15.40:3300/0 conn(0x7f33d8142c90 0x7f33d8124a50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f33d0008290 tx=0x7f33d0010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:10.436 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:10.381+0000 7f33d7fff700 1 --2- 172.21.15.42:0/3335505348 >> v2:172.21.15.40:3300/0 conn(0x7f33d8142c90 0x7f33d8124a50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f33d0008290 tx=0x7f33d0010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:10.437 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:10.381+0000 7f33d7fff700 1 --2- 172.21.15.42:0/3335505348 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f33d8142c90 0x7f33d8124a50 secure :-1 s=READY pgs=314 cs=0 l=1 rx=0x7f33d0008290 tx=0x7f33d0010130).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:10:10.437 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:10.381+0000 7f33d7fff700 1 --2- 172.21.15.42:0/3335505348 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f33d8142c90 0x7f33d8124a50 secure :-1 s=THROTTLE_DONE pgs=314 cs=0 l=1 rx=0x7f33d0008290 tx=0x7f33d0010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:10.437 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:10.381+0000 7f33d7fff700 1 --2- 172.21.15.42:0/3335505348 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f33d8142c90 0x7f33d8124a50 secure :-1 s=THROTTLE_DONE pgs=314 cs=0 l=1 rx=0x7f33d0008290 tx=0x7f33d0010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:10.437 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:10.381+0000 7f33d57fa700 1 -- 172.21.15.42:0/3335505348 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f33cc0057e0 con 0x7f33d8142c90 2019-12-05T05:10:10.437 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:10.381+0000 7f33d7fff700 1 --2- 172.21.15.42:0/3335505348 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f33d8142c90 0x7f33d8124a50 secure :-1 s=THROTTLE_DONE pgs=314 cs=0 l=1 rx=0x7f33d0008290 tx=0x7f33d0010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:10.438 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:10.381+0000 7f33d57fa700 1 -- 172.21.15.42:0/3335505348 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33cc007160 con 0x7f33d8142c90 2019-12-05T05:10:10.438 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:10.381+0000 7f33d57fa700 1 -- 172.21.15.42:0/3335505348 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f33cc005dc0 con 0x7f33d8142c90 2019-12-05T05:10:10.438 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:10.381+0000 7f33de6c7700 1 -- 172.21.15.42:0/3335505348 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33d81253d0 con 0x7f33d8142c90 2019-12-05T05:10:10.438 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:10.381+0000 7f33de6c7700 1 -- 172.21.15.42:0/3335505348 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33d80a34a0 con 0x7f33d8142c90 2019-12-05T05:10:10.438 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:10.381+0000 7f33d7fff700 1 --2- 172.21.15.42:0/3335505348 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f33d8142c90 0x7f33d8124a50 secure :-1 s=THROTTLE_DONE pgs=314 cs=0 l=1 rx=0x7f33d0008290 tx=0x7f33d0010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:10.438 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:10.381+0000 7f33d57fa700 1 -- 172.21.15.42:0/3335505348 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f33cc0086c0 con 0x7f33d8142c90 2019-12-05T05:10:10.439 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:10.381+0000 7f33d7fff700 1 --2- 172.21.15.42:0/3335505348 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f33d8142c90 0x7f33d8124a50 secure :-1 s=THROTTLE_DONE pgs=314 cs=0 l=1 rx=0x7f33d0008290 tx=0x7f33d0010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:10.439 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:10.385+0000 7f33d57fa700 1 --2- 172.21.15.42:0/3335505348 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f33b803ae10 0x7f33b803d320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:10:10.439 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:10.385+0000 7f33d57fa700 1 -- 172.21.15.42:0/3335505348 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6375+0+0 (secure 0 0 0) 0x7f33cc04aba0 con 0x7f33d8142c90 2019-12-05T05:10:10.439 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:10.385+0000 7f33dcc64700 1 --2- 172.21.15.42:0/3335505348 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f33b803ae10 0x7f33b803d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:10:10.439 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:10.385+0000 7f33dcc64700 1 --2- 172.21.15.42:0/3335505348 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f33b803ae10 0x7f33b803d320 crc :-1 s=READY pgs=179 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:10:10.439 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:10.389+0000 7f33de6c7700 1 -- 172.21.15.42:0/3335505348 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33d805d110 con 0x7f33d8142c90 2019-12-05T05:10:10.440 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:10.397+0000 7f33d7fff700 1 --2- 172.21.15.42:0/3335505348 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f33d8142c90 0x7f33d8124a50 secure :-1 s=THROTTLE_DONE pgs=314 cs=0 l=1 rx=0x7f33d0008290 tx=0x7f33d0010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:10.440 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:10.397+0000 7f33d57fa700 1 -- 172.21.15.42:0/3335505348 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f33cc031c20 con 0x7f33d8142c90 2019-12-05T05:10:10.562 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:10.557+0000 7f33de6c7700 1 -- 172.21.15.42:0/3335505348 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f33d805b8c0 con 0x7f33d8142c90 2019-12-05T05:10:10.563 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:10.557+0000 7f33d7fff700 1 --2- 172.21.15.42:0/3335505348 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f33d8142c90 0x7f33d8124a50 secure :-1 s=THROTTLE_DONE pgs=314 cs=0 l=1 rx=0x7f33d0008290 tx=0x7f33d0010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:10.564 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:10.557+0000 7f33d57fa700 1 -- 172.21.15.42:0/3335505348 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) v1 ==== 94+0+1769 (secure 0 0 0) 0x7f33cc031e80 con 0x7f33d8142c90 2019-12-05T05:10:10.564 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 40 2019-12-05T05:10:10.564 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:10:10.564 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":40,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:10:08.394068+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4595},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4595":{"gid":4595,"name":"b","rank":0,"incarnation":36,"state":"up:active","state_seq":11,"addr":"172.21.15.40:6833/211584380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":211584380},{"type":"v1","addr":"172.21.15.40:6833","nonce":211584380}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:10:10.566 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:10.561+0000 7f33de6c7700 1 -- 172.21.15.42:0/3335505348 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f33b803ae10 msgr2=0x7f33b803d320 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:10:10.566 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:10.561+0000 7f33de6c7700 1 --2- 172.21.15.42:0/3335505348 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f33b803ae10 0x7f33b803d320 crc :-1 s=READY pgs=179 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:10:10.566 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:10.561+0000 7f33de6c7700 1 -- 172.21.15.42:0/3335505348 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f33d8142c90 msgr2=0x7f33d8124a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:10:10.566 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:10.561+0000 7f33de6c7700 1 --2- 172.21.15.42:0/3335505348 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f33d8142c90 0x7f33d8124a50 secure :-1 s=READY pgs=314 cs=0 l=1 rx=0x7f33d0008290 tx=0x7f33d0010130).stop 2019-12-05T05:10:10.566 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:10.561+0000 7f33de6c7700 1 -- 172.21.15.42:0/3335505348 shutdown_connections 2019-12-05T05:10:10.567 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:10.561+0000 7f33de6c7700 1 --2- 172.21.15.42:0/3335505348 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f33b803ae10 0x7f33b803d320 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:10:10.567 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:10.561+0000 7f33de6c7700 1 --2- 172.21.15.42:0/3335505348 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f33d8124fa0 0x7f33d81357f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:10:10.567 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:10.561+0000 7f33de6c7700 1 --2- 172.21.15.42:0/3335505348 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f33d8142c90 0x7f33d8124a50 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:10:10.567 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:10.561+0000 7f33de6c7700 1 -- 172.21.15.42:0/3335505348 shutdown_connections 2019-12-05T05:10:10.567 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:10.561+0000 7f33de6c7700 1 -- 172.21.15.42:0/3335505348 wait complete. 2019-12-05T05:10:10.567 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:10.561+0000 7f33de6c7700 1 -- 172.21.15.42:0/3335505348 >> 172.21.15.42:0/3335505348 conn(0x7f33d80a82a0 msgr2=0x7f33d809f620 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:10:10.575 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b has gained rank=0, replacing gid=4978 2019-12-05T05:10:10.576 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 5 secs before reviving mds.a 2019-12-05T05:10:11.352 INFO:tasks.workunit.client.0.smithi042.stdout:removed './297/31' 2019-12-05T05:10:11.352 INFO:tasks.workunit.client.0.smithi042.stdout:removed './297/36' 2019-12-05T05:10:11.352 INFO:tasks.workunit.client.0.smithi042.stdout:removed './297/20' 2019-12-05T05:10:11.353 INFO:tasks.workunit.client.0.smithi042.stdout:removed './297/39' 2019-12-05T05:10:11.353 INFO:tasks.workunit.client.0.smithi042.stdout:removed './297/34' 2019-12-05T05:10:11.353 INFO:tasks.workunit.client.0.smithi042.stdout:removed './297/38' 2019-12-05T05:10:11.353 INFO:tasks.workunit.client.0.smithi042.stdout:removed './297/28' 2019-12-05T05:10:11.353 INFO:tasks.workunit.client.0.smithi042.stdout:removed './297/24' 2019-12-05T05:10:11.353 INFO:tasks.workunit.client.0.smithi042.stdout:removed './297/29' 2019-12-05T05:10:11.353 INFO:tasks.workunit.client.0.smithi042.stdout:removed './297/32' 2019-12-05T05:10:11.354 INFO:tasks.workunit.client.0.smithi042.stdout:removed './297/22' 2019-12-05T05:10:11.354 INFO:tasks.workunit.client.0.smithi042.stdout:removed './297/25' 2019-12-05T05:10:11.354 INFO:tasks.workunit.client.0.smithi042.stdout:removed './297/30' 2019-12-05T05:10:11.354 INFO:tasks.workunit.client.0.smithi042.stdout:removed './297/33' 2019-12-05T05:10:11.354 INFO:tasks.workunit.client.0.smithi042.stdout:removed './297/23' 2019-12-05T05:10:11.355 INFO:tasks.workunit.client.0.smithi042.stdout:removed './297/35' 2019-12-05T05:10:11.355 INFO:tasks.workunit.client.0.smithi042.stdout:removed './297/26' 2019-12-05T05:10:11.355 INFO:tasks.workunit.client.0.smithi042.stdout:removed './297/37' 2019-12-05T05:10:11.355 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './297' 2019-12-05T05:10:11.355 INFO:tasks.workunit.client.0.smithi042.stdout:removed './298/21' 2019-12-05T05:10:11.355 INFO:tasks.workunit.client.0.smithi042.stdout:removed './298/27' 2019-12-05T05:10:11.356 INFO:tasks.workunit.client.0.smithi042.stdout:removed './298/31' 2019-12-05T05:10:11.356 INFO:tasks.workunit.client.0.smithi042.stdout:removed './298/36' 2019-12-05T05:10:11.358 INFO:tasks.workunit.client.0.smithi042.stdout:removed './298/20' 2019-12-05T05:10:11.358 INFO:tasks.workunit.client.0.smithi042.stdout:removed './298/39' 2019-12-05T05:10:11.358 INFO:tasks.workunit.client.0.smithi042.stdout:removed './298/34' 2019-12-05T05:10:11.359 INFO:tasks.workunit.client.0.smithi042.stdout:removed './298/38' 2019-12-05T05:10:11.359 INFO:tasks.workunit.client.0.smithi042.stdout:removed './298/28' 2019-12-05T05:10:11.359 INFO:tasks.workunit.client.0.smithi042.stdout:removed './298/24' 2019-12-05T05:10:11.359 INFO:tasks.workunit.client.0.smithi042.stdout:removed './298/29' 2019-12-05T05:10:11.360 INFO:tasks.workunit.client.0.smithi042.stdout:removed './298/32' 2019-12-05T05:10:11.360 INFO:tasks.workunit.client.0.smithi042.stdout:removed './298/22' 2019-12-05T05:10:11.360 INFO:tasks.workunit.client.0.smithi042.stdout:removed './298/25' 2019-12-05T05:10:11.360 INFO:tasks.workunit.client.0.smithi042.stdout:removed './298/30' 2019-12-05T05:10:11.360 INFO:tasks.workunit.client.0.smithi042.stdout:removed './298/33' 2019-12-05T05:10:11.360 INFO:tasks.workunit.client.0.smithi042.stdout:removed './298/23' 2019-12-05T05:10:11.361 INFO:tasks.workunit.client.0.smithi042.stdout:removed './298/35' 2019-12-05T05:10:11.361 INFO:tasks.workunit.client.0.smithi042.stdout:removed './298/26' 2019-12-05T05:10:11.361 INFO:tasks.workunit.client.0.smithi042.stdout:removed './298/37' 2019-12-05T05:10:11.361 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './298' 2019-12-05T05:10:11.362 INFO:tasks.workunit.client.0.smithi042.stdout:removed './299/21' 2019-12-05T05:10:11.362 INFO:tasks.workunit.client.0.smithi042.stdout:removed './299/27' 2019-12-05T05:10:11.362 INFO:tasks.workunit.client.0.smithi042.stdout:removed './299/31' 2019-12-05T05:10:11.362 INFO:tasks.workunit.client.0.smithi042.stdout:removed './299/36' 2019-12-05T05:10:11.362 INFO:tasks.workunit.client.0.smithi042.stdout:removed './299/20' 2019-12-05T05:10:11.362 INFO:tasks.workunit.client.0.smithi042.stdout:removed './299/39' 2019-12-05T05:10:11.362 INFO:tasks.workunit.client.0.smithi042.stdout:removed './299/34' 2019-12-05T05:10:11.363 INFO:tasks.workunit.client.0.smithi042.stdout:removed './299/38' 2019-12-05T05:10:11.363 INFO:tasks.workunit.client.0.smithi042.stdout:removed './299/28' 2019-12-05T05:10:11.363 INFO:tasks.workunit.client.0.smithi042.stdout:removed './299/24' 2019-12-05T05:10:11.363 INFO:tasks.workunit.client.0.smithi042.stdout:removed './299/29' 2019-12-05T05:10:11.363 INFO:tasks.workunit.client.0.smithi042.stdout:removed './299/32' 2019-12-05T05:10:11.363 INFO:tasks.workunit.client.0.smithi042.stdout:removed './299/22' 2019-12-05T05:10:11.363 INFO:tasks.workunit.client.0.smithi042.stdout:removed './299/25' 2019-12-05T05:10:11.364 INFO:tasks.workunit.client.0.smithi042.stdout:removed './299/30' 2019-12-05T05:10:11.364 INFO:tasks.workunit.client.0.smithi042.stdout:removed './299/33' 2019-12-05T05:10:11.364 INFO:tasks.workunit.client.0.smithi042.stdout:removed './299/23' 2019-12-05T05:10:11.364 INFO:tasks.workunit.client.0.smithi042.stdout:removed './299/35' 2019-12-05T05:10:11.364 INFO:tasks.workunit.client.0.smithi042.stdout:removed './299/26' 2019-12-05T05:10:11.364 INFO:tasks.workunit.client.0.smithi042.stdout:removed './299/37' 2019-12-05T05:10:11.364 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './299' 2019-12-05T05:10:11.364 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/166/19' 2019-12-05T05:10:11.365 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/166/15' 2019-12-05T05:10:11.365 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/166/20' 2019-12-05T05:10:11.365 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/166/12' 2019-12-05T05:10:11.365 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/166/10' 2019-12-05T05:10:11.365 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/166/11' 2019-12-05T05:10:11.365 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/166/16' 2019-12-05T05:10:11.365 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/166/14' 2019-12-05T05:10:11.366 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/166/18' 2019-12-05T05:10:11.366 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/166/13' 2019-12-05T05:10:11.366 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/166/17' 2019-12-05T05:10:11.366 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './copyofsnap1/test1/166' 2019-12-05T05:10:11.366 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/147/19' 2019-12-05T05:10:11.366 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/147/15' 2019-12-05T05:10:11.366 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/147/20' 2019-12-05T05:10:11.367 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/147/12' 2019-12-05T05:10:11.367 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/147/10' 2019-12-05T05:10:11.367 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/147/11' 2019-12-05T05:10:11.367 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/147/16' 2019-12-05T05:10:11.367 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/147/14' 2019-12-05T05:10:11.367 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/147/18' 2019-12-05T05:10:11.367 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/147/13' 2019-12-05T05:10:11.367 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/147/17' 2019-12-05T05:10:11.368 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './copyofsnap1/test1/147' 2019-12-05T05:10:11.368 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/151/19' 2019-12-05T05:10:11.368 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/151/15' 2019-12-05T05:10:11.368 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/151/20' 2019-12-05T05:10:11.369 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/151/12' 2019-12-05T05:10:11.369 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/151/10' 2019-12-05T05:10:11.369 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/151/11' 2019-12-05T05:10:11.369 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/151/16' 2019-12-05T05:10:11.369 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/151/14' 2019-12-05T05:10:11.369 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/151/18' 2019-12-05T05:10:11.369 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/151/13' 2019-12-05T05:10:11.370 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/151/17' 2019-12-05T05:10:11.370 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './copyofsnap1/test1/151' 2019-12-05T05:10:11.370 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/170/19' 2019-12-05T05:10:11.370 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/170/15' 2019-12-05T05:10:11.370 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/170/20' 2019-12-05T05:10:11.370 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/170/12' 2019-12-05T05:10:11.370 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/170/10' 2019-12-05T05:10:11.371 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/170/11' 2019-12-05T05:10:11.371 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/170/16' 2019-12-05T05:10:11.371 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/170/14' 2019-12-05T05:10:11.371 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/170/18' 2019-12-05T05:10:11.371 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/170/13' 2019-12-05T05:10:11.371 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/170/17' 2019-12-05T05:10:11.371 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './copyofsnap1/test1/170' 2019-12-05T05:10:11.371 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/167/19' 2019-12-05T05:10:11.372 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/167/15' 2019-12-05T05:10:11.372 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/167/20' 2019-12-05T05:10:11.372 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/167/12' 2019-12-05T05:10:11.372 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/167/10' 2019-12-05T05:10:11.372 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/167/11' 2019-12-05T05:10:11.373 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/167/16' 2019-12-05T05:10:11.373 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/167/14' 2019-12-05T05:10:11.373 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/167/18' 2019-12-05T05:10:11.373 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/167/13' 2019-12-05T05:10:11.373 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/167/17' 2019-12-05T05:10:11.373 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './copyofsnap1/test1/167' 2019-12-05T05:10:11.374 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/140/19' 2019-12-05T05:10:11.374 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/140/15' 2019-12-05T05:10:11.374 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/140/20' 2019-12-05T05:10:11.374 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/140/12' 2019-12-05T05:10:11.374 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/140/10' 2019-12-05T05:10:11.374 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/140/11' 2019-12-05T05:10:11.375 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/140/16' 2019-12-05T05:10:11.375 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/140/14' 2019-12-05T05:10:11.375 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/140/18' 2019-12-05T05:10:11.375 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/140/13' 2019-12-05T05:10:11.375 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/140/17' 2019-12-05T05:10:11.375 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './copyofsnap1/test1/140' 2019-12-05T05:10:11.375 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/142/19' 2019-12-05T05:10:11.376 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/142/15' 2019-12-05T05:10:11.376 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/142/20' 2019-12-05T05:10:11.376 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/142/12' 2019-12-05T05:10:11.376 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/142/10' 2019-12-05T05:10:15.577 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.a 2019-12-05T05:10:15.577 INFO:tasks.ceph.mds.a:Restarting daemon 2019-12-05T05:10:15.578 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2019-12-05T05:10:15.582 INFO:tasks.ceph.mds.a:Started 2019-12-05T05:10:15.583 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:10:15.637 INFO:tasks.ceph.mds.a.smithi042.stdout:starting mds.a at 2019-12-05T05:10:15.685 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:15.673+0000 7fc038d78700 1 Processor -- start 2019-12-05T05:10:15.686 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:15.673+0000 7fc038d78700 1 -- start start 2019-12-05T05:10:15.686 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:15.677+0000 7fc038d78700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc0340a7740 0x7fc03408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:10:15.686 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:15.677+0000 7fc038d78700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0340a0710 con 0x7fc03407e720 2019-12-05T05:10:15.686 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:15.677+0000 7fc038d78700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0340a9ba0 con 0x7fc03409fae0 2019-12-05T05:10:15.686 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:15.677+0000 7fc038d78700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0340a0b00 con 0x7fc0340a7740 2019-12-05T05:10:15.687 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:15.677+0000 7fc03359e700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fc03409fae0 0x7fc03414efe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:57210/0 (socket says 172.21.15.42:57210) 2019-12-05T05:10:15.687 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:15.677+0000 7fc03359e700 1 -- 172.21.15.42:0/417949324 learned_addr learned my addr 172.21.15.42:0/417949324 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:10:15.687 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:15.677+0000 7fc032d9d700 1 --2- 172.21.15.42:0/417949324 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc0340a7740 0x7fc03408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:10:15.687 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:15.677+0000 7fc031d9b700 1 -- 172.21.15.42:0/417949324 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3432447853 0 0) 0x7fc0340a0710 con 0x7fc03407e720 2019-12-05T05:10:15.689 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:15.677+0000 7fc031d9b700 1 -- 172.21.15.42:0/417949324 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0200030b0 con 0x7fc03407e720 2019-12-05T05:10:15.689 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:15.677+0000 7fc031d9b700 1 -- 172.21.15.42:0/417949324 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2366292031 0 0) 0x7fc0340a9ba0 con 0x7fc03409fae0 2019-12-05T05:10:15.689 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:15.677+0000 7fc032d9d700 1 -- 172.21.15.42:0/417949324 >> v1:172.21.15.42:6789/0 conn(0x7fc03409fae0 legacy=0x7fc03414efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:10:15.689 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:15.677+0000 7fc032d9d700 1 -- 172.21.15.42:0/417949324 >> v1:172.21.15.40:6789/0 conn(0x7fc03407e720 legacy=0x7fc03409f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:10:15.690 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:15.677+0000 7fc032d9d700 1 -- 172.21.15.42:0/417949324 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc03407eb50 con 0x7fc0340a7740 2019-12-05T05:10:15.690 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:15.677+0000 7fc032d9d700 1 --2- 172.21.15.42:0/417949324 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc0340a7740 0x7fc03408df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fc01c002f30 tx=0x7fc01c00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:15.690 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:15.677+0000 7fc032d9d700 1 --2- 172.21.15.42:0/417949324 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc0340a7740 0x7fc03408df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fc01c002f30 tx=0x7fc01c00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:15.690 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:15.677+0000 7fc032d9d700 1 --2- 172.21.15.42:0/417949324 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc0340a7740 0x7fc03408df80 secure :-1 s=READY pgs=79 cs=0 l=1 rx=0x7fc01c002f30 tx=0x7fc01c00fbf0).ready entity=mon.2 client_cookie=4566590def2935d5 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:10:15.691 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:15.677+0000 7fc032d9d700 1 --2- 172.21.15.42:0/417949324 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc0340a7740 0x7fc03408df80 secure :-1 s=THROTTLE_DONE pgs=79 cs=0 l=1 rx=0x7fc01c002f30 tx=0x7fc01c00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:15.691 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:15.677+0000 7fc031d9b700 1 -- 172.21.15.42:0/417949324 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc01c0112b0 con 0x7fc0340a7740 2019-12-05T05:10:15.691 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:15.677+0000 7fc032d9d700 1 --2- 172.21.15.42:0/417949324 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc0340a7740 0x7fc03408df80 secure :-1 s=THROTTLE_DONE pgs=79 cs=0 l=1 rx=0x7fc01c002f30 tx=0x7fc01c00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:15.691 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:15.677+0000 7fc032d9d700 1 --2- 172.21.15.42:0/417949324 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc0340a7740 0x7fc03408df80 secure :-1 s=THROTTLE_DONE pgs=79 cs=0 l=1 rx=0x7fc01c002f30 tx=0x7fc01c00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:15.691 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:15.677+0000 7fc031d9b700 1 -- 172.21.15.42:0/417949324 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc01c008bc0 con 0x7fc0340a7740 2019-12-05T05:10:15.691 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:15.677+0000 7fc031d9b700 1 -- 172.21.15.42:0/417949324 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc01c011890 con 0x7fc0340a7740 2019-12-05T05:10:15.691 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:15.677+0000 7fc038d78700 1 -- 172.21.15.42:0/417949324 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc0340a7740 msgr2=0x7fc03408df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:10:15.692 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:15.677+0000 7fc038d78700 1 --2- 172.21.15.42:0/417949324 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc0340a7740 0x7fc03408df80 secure :-1 s=READY pgs=79 cs=0 l=1 rx=0x7fc01c002f30 tx=0x7fc01c00fbf0).stop 2019-12-05T05:10:15.692 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:15.677+0000 7fc038d78700 1 -- 172.21.15.42:0/417949324 shutdown_connections 2019-12-05T05:10:15.692 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:15.677+0000 7fc038d78700 1 --2- 172.21.15.42:0/417949324 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc0340a7740 0x7fc03408df80 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:10:15.692 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:15.677+0000 7fc038d78700 1 -- 172.21.15.42:0/417949324 shutdown_connections 2019-12-05T05:10:15.692 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:15.677+0000 7fc038d78700 1 -- 172.21.15.42:0/417949324 wait complete. 2019-12-05T05:10:15.692 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:15.677+0000 7fc038d78700 1 -- 172.21.15.42:0/417949324 >> 172.21.15.42:0/417949324 conn(0x7fc0340aabe0 msgr2=0x7fc0340a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:10:15.697 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:15.681+0000 7fc038d78700 1 Processor -- start 2019-12-05T05:10:15.697 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:15.681+0000 7fc038d78700 1 -- start start 2019-12-05T05:10:15.697 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:15.681+0000 7fc038d78700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc03408ef00 0x7fc034116940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:10:15.698 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:15.681+0000 7fc038d78700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc0340a6210 0x7fc034130eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:10:15.698 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:15.681+0000 7fc038d78700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc034116300 con 0x7fc0341188d0 2019-12-05T05:10:15.698 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:15.681+0000 7fc038d78700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0340a6790 con 0x7fc0340a6210 2019-12-05T05:10:15.698 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:15.681+0000 7fc038d78700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0340a6a30 con 0x7fc03408ef00 2019-12-05T05:10:15.698 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:15.681+0000 7fc03259c700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fc0341188d0 0x7fc0341306b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:57212/0 (socket says 172.21.15.42:57212) 2019-12-05T05:10:15.699 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:15.681+0000 7fc032d9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc03408ef00 0x7fc034116940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:10:15.699 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:15.681+0000 7fc03259c700 1 -- 172.21.15.42:0/3960367282 learned_addr learned my addr 172.21.15.42:0/3960367282 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:10:15.699 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:15.681+0000 7fc03359e700 1 --2- 172.21.15.42:0/3960367282 >> v2:172.21.15.40:3300/0 conn(0x7fc0340a6210 0x7fc034130eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:10:15.699 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:15.681+0000 7fc02b7fe700 1 -- 172.21.15.42:0/3960367282 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4117848753 0 0) 0x7fc034116300 con 0x7fc0341188d0 2019-12-05T05:10:15.700 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:15.681+0000 7fc02b7fe700 1 -- 172.21.15.42:0/3960367282 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0180030b0 con 0x7fc0341188d0 2019-12-05T05:10:15.700 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:15.681+0000 7fc032d9d700 1 -- 172.21.15.42:0/3960367282 >> v2:172.21.15.40:3300/0 conn(0x7fc0340a6210 msgr2=0x7fc034130eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:10:15.701 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:15.681+0000 7fc032d9d700 1 --2- 172.21.15.42:0/3960367282 >> v2:172.21.15.40:3300/0 conn(0x7fc0340a6210 0x7fc034130eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:10:15.702 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:15.681+0000 7fc032d9d700 1 -- 172.21.15.42:0/3960367282 >> v1:172.21.15.42:6789/0 conn(0x7fc0341188d0 legacy=0x7fc0341306b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:10:15.702 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:15.681+0000 7fc032d9d700 1 -- 172.21.15.42:0/3960367282 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc034116e90 con 0x7fc03408ef00 2019-12-05T05:10:15.702 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:15.681+0000 7fc03359e700 1 --2- 172.21.15.42:0/3960367282 >> v2:172.21.15.40:3300/0 conn(0x7fc0340a6210 0x7fc034130eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:10:15.702 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:15.681+0000 7fc032d9d700 1 --2- 172.21.15.42:0/3960367282 >> v2:172.21.15.42:3300/0 conn(0x7fc03408ef00 0x7fc034116940 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fc01c002f30 tx=0x7fc01c008e70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:15.703 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:15.681+0000 7fc032d9d700 1 --2- 172.21.15.42:0/3960367282 >> v2:172.21.15.42:3300/0 conn(0x7fc03408ef00 0x7fc034116940 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fc01c002f30 tx=0x7fc01c008e70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:15.703 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:15.681+0000 7fc032d9d700 1 --2- 172.21.15.42:0/3960367282 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc03408ef00 0x7fc034116940 secure :-1 s=READY pgs=401 cs=0 l=1 rx=0x7fc01c002f30 tx=0x7fc01c008e70).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:10:15.703 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:15.685+0000 7fc032d9d700 1 --2- 172.21.15.42:0/3960367282 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc03408ef00 0x7fc034116940 secure :-1 s=THROTTLE_DONE pgs=401 cs=0 l=1 rx=0x7fc01c002f30 tx=0x7fc01c008e70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:15.704 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:15.685+0000 7fc02b7fe700 1 -- 172.21.15.42:0/3960367282 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc01c007150 con 0x7fc03408ef00 2019-12-05T05:10:15.704 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:15.685+0000 7fc032d9d700 1 --2- 172.21.15.42:0/3960367282 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc03408ef00 0x7fc034116940 secure :-1 s=THROTTLE_DONE pgs=401 cs=0 l=1 rx=0x7fc01c002f30 tx=0x7fc01c008e70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:15.704 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:15.685+0000 7fc032d9d700 1 --2- 172.21.15.42:0/3960367282 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc03408ef00 0x7fc034116940 secure :-1 s=THROTTLE_DONE pgs=401 cs=0 l=1 rx=0x7fc01c002f30 tx=0x7fc01c008e70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:15.704 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:15.685+0000 7fc02b7fe700 1 -- 172.21.15.42:0/3960367282 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc01c0114b0 con 0x7fc03408ef00 2019-12-05T05:10:15.704 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:15.685+0000 7fc038d78700 1 -- 172.21.15.42:0/3960367282 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc034117160 con 0x7fc03408ef00 2019-12-05T05:10:15.705 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:15.685+0000 7fc02b7fe700 1 -- 172.21.15.42:0/3960367282 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc01c00f550 con 0x7fc03408ef00 2019-12-05T05:10:15.705 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:15.685+0000 7fc038d78700 1 -- 172.21.15.42:0/3960367282 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc034137a10 con 0x7fc03408ef00 2019-12-05T05:10:15.705 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:15.685+0000 7fc032d9d700 1 --2- 172.21.15.42:0/3960367282 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc03408ef00 0x7fc034116940 secure :-1 s=THROTTLE_DONE pgs=401 cs=0 l=1 rx=0x7fc01c002f30 tx=0x7fc01c008e70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:15.705 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:15.685+0000 7fc02b7fe700 1 -- 172.21.15.42:0/3960367282 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fc01c005540 con 0x7fc03408ef00 2019-12-05T05:10:15.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:15.685+0000 7fc032d9d700 1 --2- 172.21.15.42:0/3960367282 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc03408ef00 0x7fc034116940 secure :-1 s=THROTTLE_DONE pgs=401 cs=0 l=1 rx=0x7fc01c002f30 tx=0x7fc01c008e70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:15.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:15.685+0000 7fc02b7fe700 1 --2- 172.21.15.42:0/3960367282 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc01803b2f0 0x7fc01803d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:10:15.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:15.685+0000 7fc02b7fe700 1 -- 172.21.15.42:0/3960367282 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6375+0+0 (secure 0 0 0) 0x7fc01c04b770 con 0x7fc03408ef00 2019-12-05T05:10:15.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:15.685+0000 7fc03259c700 1 --2- 172.21.15.42:0/3960367282 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc01803b2f0 0x7fc01803d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:10:15.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:15.685+0000 7fc03259c700 1 --2- 172.21.15.42:0/3960367282 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc01803b2f0 0x7fc01803d800 crc :-1 s=READY pgs=180 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:10:15.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:15.689+0000 7fc038d78700 1 -- 172.21.15.42:0/3960367282 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0340a6bf0 con 0x7fc03408ef00 2019-12-05T05:10:15.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:15.697+0000 7fc032d9d700 1 --2- 172.21.15.42:0/3960367282 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc03408ef00 0x7fc034116940 secure :-1 s=THROTTLE_DONE pgs=401 cs=0 l=1 rx=0x7fc01c002f30 tx=0x7fc01c008e70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:15.708 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:15.697+0000 7fc02b7fe700 1 -- 172.21.15.42:0/3960367282 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fc01c032a50 con 0x7fc03408ef00 2019-12-05T05:10:15.868 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:15.865+0000 7fc038d78700 1 -- 172.21.15.42:0/3960367282 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc03405d110 con 0x7fc03408ef00 2019-12-05T05:10:15.870 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:15.865+0000 7fc032d9d700 1 --2- 172.21.15.42:0/3960367282 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc03408ef00 0x7fc034116940 secure :-1 s=THROTTLE_DONE pgs=401 cs=0 l=1 rx=0x7fc01c002f30 tx=0x7fc01c008e70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:15.870 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:15.865+0000 7fc02b7fe700 1 -- 172.21.15.42:0/3960367282 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) v1 ==== 94+0+2105 (secure 0 0 0) 0x7fc01c0304a0 con 0x7fc03408ef00 2019-12-05T05:10:15.870 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 41 2019-12-05T05:10:15.870 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:10:15.871 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":41,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5215,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6835/3176146895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3176146895},{"type":"v1","addr":"172.21.15.42:6835","nonce":3176146895}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":41}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:10:08.394068+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4595},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4595":{"gid":4595,"name":"b","rank":0,"incarnation":36,"state":"up:active","state_seq":11,"addr":"172.21.15.40:6833/211584380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":211584380},{"type":"v1","addr":"172.21.15.40:6833","nonce":211584380}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:10:15.872 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:15.869+0000 7fc038d78700 1 -- 172.21.15.42:0/3960367282 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc01803b2f0 msgr2=0x7fc01803d800 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:10:15.872 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:15.869+0000 7fc038d78700 1 --2- 172.21.15.42:0/3960367282 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc01803b2f0 0x7fc01803d800 crc :-1 s=READY pgs=180 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:10:15.872 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:15.869+0000 7fc038d78700 1 -- 172.21.15.42:0/3960367282 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc03408ef00 msgr2=0x7fc034116940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:10:15.872 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:15.869+0000 7fc038d78700 1 --2- 172.21.15.42:0/3960367282 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc03408ef00 0x7fc034116940 secure :-1 s=READY pgs=401 cs=0 l=1 rx=0x7fc01c002f30 tx=0x7fc01c008e70).stop 2019-12-05T05:10:15.873 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:15.869+0000 7fc038d78700 1 -- 172.21.15.42:0/3960367282 shutdown_connections 2019-12-05T05:10:15.873 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:15.869+0000 7fc038d78700 1 --2- 172.21.15.42:0/3960367282 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc01803b2f0 0x7fc01803d800 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:10:15.874 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:15.869+0000 7fc038d78700 1 --2- 172.21.15.42:0/3960367282 >> v2:172.21.15.40:3300/0 conn(0x7fc0340a6210 0x7fc034130eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:10:15.874 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:15.869+0000 7fc038d78700 1 --2- 172.21.15.42:0/3960367282 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc03408ef00 0x7fc034116940 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:10:15.874 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:15.869+0000 7fc038d78700 1 -- 172.21.15.42:0/3960367282 shutdown_connections 2019-12-05T05:10:15.874 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:15.869+0000 7fc038d78700 1 -- 172.21.15.42:0/3960367282 wait complete. 2019-12-05T05:10:15.874 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:10:15.869+0000 7fc038d78700 1 -- 172.21.15.42:0/3960367282 >> 172.21.15.42:0/3960367282 conn(0x7fc03407d840 msgr2=0x7fc03407be00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:10:15.882 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a reported in up:standby state 2019-12-05T05:10:15.882 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 46 secs before thrashing 2019-12-05T05:10:16.819 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/142/11' 2019-12-05T05:10:16.820 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/142/16' 2019-12-05T05:10:16.820 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/142/14' 2019-12-05T05:10:16.820 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/142/18' 2019-12-05T05:10:16.820 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/142/13' 2019-12-05T05:10:16.820 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/142/17' 2019-12-05T05:10:16.820 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './copyofsnap1/test1/142' 2019-12-05T05:10:16.821 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/150/19' 2019-12-05T05:10:16.821 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/150/15' 2019-12-05T05:10:16.821 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/150/20' 2019-12-05T05:10:16.821 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/150/12' 2019-12-05T05:10:16.821 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/150/10' 2019-12-05T05:10:16.821 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/150/11' 2019-12-05T05:10:16.821 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/150/16' 2019-12-05T05:10:16.822 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/150/14' 2019-12-05T05:10:16.822 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/150/18' 2019-12-05T05:10:16.822 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/150/13' 2019-12-05T05:10:16.822 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/150/17' 2019-12-05T05:10:16.822 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './copyofsnap1/test1/150' 2019-12-05T05:10:16.822 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/105/19' 2019-12-05T05:10:16.822 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/105/15' 2019-12-05T05:10:16.823 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/105/20' 2019-12-05T05:10:16.823 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/105/12' 2019-12-05T05:10:16.823 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/105/10' 2019-12-05T05:10:16.823 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/105/11' 2019-12-05T05:10:16.823 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/105/16' 2019-12-05T05:10:16.823 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/105/14' 2019-12-05T05:10:16.823 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/105/18' 2019-12-05T05:10:16.823 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/105/13' 2019-12-05T05:10:16.824 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/105/17' 2019-12-05T05:10:16.824 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './copyofsnap1/test1/105' 2019-12-05T05:10:16.824 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/149/19' 2019-12-05T05:10:16.824 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/149/15' 2019-12-05T05:10:16.824 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/149/20' 2019-12-05T05:10:16.824 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/149/12' 2019-12-05T05:10:16.824 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/149/10' 2019-12-05T05:10:16.825 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/149/11' 2019-12-05T05:10:16.825 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/149/16' 2019-12-05T05:10:16.825 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/149/14' 2019-12-05T05:10:16.825 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/149/18' 2019-12-05T05:10:16.825 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/149/13' 2019-12-05T05:10:16.825 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/149/17' 2019-12-05T05:10:16.825 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './copyofsnap1/test1/149' 2019-12-05T05:10:16.826 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/109/19' 2019-12-05T05:10:16.826 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/109/15' 2019-12-05T05:10:16.826 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/109/20' 2019-12-05T05:10:16.826 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/109/12' 2019-12-05T05:10:16.826 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/109/10' 2019-12-05T05:10:16.826 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/109/11' 2019-12-05T05:10:16.827 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/109/16' 2019-12-05T05:10:16.827 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/109/14' 2019-12-05T05:10:16.827 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/109/18' 2019-12-05T05:10:16.828 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/109/13' 2019-12-05T05:10:16.828 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/109/17' 2019-12-05T05:10:16.828 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './copyofsnap1/test1/109' 2019-12-05T05:10:16.828 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/158/19' 2019-12-05T05:10:16.828 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/158/15' 2019-12-05T05:10:16.828 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/158/20' 2019-12-05T05:10:16.828 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/158/12' 2019-12-05T05:10:16.829 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/158/10' 2019-12-05T05:10:16.829 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/158/11' 2019-12-05T05:10:16.829 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/158/16' 2019-12-05T05:10:16.829 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/158/14' 2019-12-05T05:10:16.829 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/158/18' 2019-12-05T05:10:16.829 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/158/13' 2019-12-05T05:10:16.829 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/158/17' 2019-12-05T05:10:16.830 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './copyofsnap1/test1/158' 2019-12-05T05:10:16.830 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/106/19' 2019-12-05T05:10:16.830 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/106/15' 2019-12-05T05:10:16.830 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/106/20' 2019-12-05T05:10:16.830 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/106/12' 2019-12-05T05:10:16.830 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/106/10' 2019-12-05T05:10:16.830 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/106/11' 2019-12-05T05:10:16.830 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/106/16' 2019-12-05T05:10:16.831 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/106/14' 2019-12-05T05:10:16.831 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/106/18' 2019-12-05T05:10:16.831 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/106/13' 2019-12-05T05:10:16.831 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/106/17' 2019-12-05T05:10:16.831 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './copyofsnap1/test1/106' 2019-12-05T05:10:16.831 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/189/19' 2019-12-05T05:10:16.831 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/189/15' 2019-12-05T05:10:16.832 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/189/20' 2019-12-05T05:10:16.832 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/189/12' 2019-12-05T05:10:16.832 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/189/10' 2019-12-05T05:10:16.832 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/189/11' 2019-12-05T05:10:16.832 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/189/16' 2019-12-05T05:10:16.832 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/189/14' 2019-12-05T05:10:16.832 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/189/18' 2019-12-05T05:10:16.833 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/189/13' 2019-12-05T05:10:16.833 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/189/17' 2019-12-05T05:10:16.833 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './copyofsnap1/test1/189' 2019-12-05T05:10:16.833 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/133/19' 2019-12-05T05:10:16.833 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/133/15' 2019-12-05T05:10:16.833 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/133/20' 2019-12-05T05:10:16.833 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/133/12' 2019-12-05T05:10:16.834 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/133/10' 2019-12-05T05:10:16.834 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/133/11' 2019-12-05T05:10:16.834 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/133/16' 2019-12-05T05:10:16.834 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/133/14' 2019-12-05T05:10:16.834 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/133/18' 2019-12-05T05:10:16.834 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/133/13' 2019-12-05T05:10:16.834 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/133/17' 2019-12-05T05:10:16.835 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './copyofsnap1/test1/133' 2019-12-05T05:10:16.835 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/173/19' 2019-12-05T05:10:16.835 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/173/15' 2019-12-05T05:10:16.835 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/173/20' 2019-12-05T05:10:16.835 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/173/12' 2019-12-05T05:10:16.835 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/173/10' 2019-12-05T05:10:16.835 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/173/11' 2019-12-05T05:10:18.753 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/173/16' 2019-12-05T05:10:18.753 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/173/14' 2019-12-05T05:10:18.753 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/173/18' 2019-12-05T05:10:18.754 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/173/13' 2019-12-05T05:10:18.754 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/173/17' 2019-12-05T05:10:18.754 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './copyofsnap1/test1/173' 2019-12-05T05:10:18.754 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/174/19' 2019-12-05T05:10:18.755 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/174/15' 2019-12-05T05:10:18.755 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/174/20' 2019-12-05T05:10:18.755 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/174/12' 2019-12-05T05:10:18.755 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/174/10' 2019-12-05T05:10:18.755 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/174/11' 2019-12-05T05:10:18.756 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/174/16' 2019-12-05T05:10:18.756 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/174/14' 2019-12-05T05:10:18.756 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/174/18' 2019-12-05T05:10:18.756 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/174/13' 2019-12-05T05:10:18.756 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/174/17' 2019-12-05T05:10:18.756 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './copyofsnap1/test1/174' 2019-12-05T05:10:18.757 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/155/19' 2019-12-05T05:10:18.757 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/155/15' 2019-12-05T05:10:18.757 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/155/20' 2019-12-05T05:10:18.757 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/155/12' 2019-12-05T05:10:18.757 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/155/10' 2019-12-05T05:10:18.757 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/155/11' 2019-12-05T05:10:18.758 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/155/16' 2019-12-05T05:10:18.758 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/155/14' 2019-12-05T05:10:18.758 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/155/18' 2019-12-05T05:10:18.758 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/155/13' 2019-12-05T05:10:18.758 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/155/17' 2019-12-05T05:10:18.758 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './copyofsnap1/test1/155' 2019-12-05T05:10:18.758 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/114/19' 2019-12-05T05:10:18.759 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/114/15' 2019-12-05T05:10:18.759 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/114/20' 2019-12-05T05:10:18.759 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/114/12' 2019-12-05T05:10:18.759 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/114/10' 2019-12-05T05:10:18.759 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/114/11' 2019-12-05T05:10:18.759 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/114/16' 2019-12-05T05:10:18.759 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/114/14' 2019-12-05T05:10:18.760 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/114/18' 2019-12-05T05:10:18.760 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/114/13' 2019-12-05T05:10:18.760 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/114/17' 2019-12-05T05:10:18.760 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './copyofsnap1/test1/114' 2019-12-05T05:10:18.760 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/159/19' 2019-12-05T05:10:18.760 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/159/15' 2019-12-05T05:10:18.760 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/159/20' 2019-12-05T05:10:18.761 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/159/12' 2019-12-05T05:10:18.761 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/159/10' 2019-12-05T05:10:18.761 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/159/11' 2019-12-05T05:10:18.761 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/159/16' 2019-12-05T05:10:18.761 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/159/14' 2019-12-05T05:10:18.761 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/159/18' 2019-12-05T05:10:18.762 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/159/13' 2019-12-05T05:10:18.762 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/159/17' 2019-12-05T05:10:18.762 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './copyofsnap1/test1/159' 2019-12-05T05:10:18.762 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/187/19' 2019-12-05T05:10:18.762 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/187/15' 2019-12-05T05:10:18.763 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/187/20' 2019-12-05T05:10:18.763 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/187/12' 2019-12-05T05:10:18.763 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/187/10' 2019-12-05T05:10:18.763 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/187/11' 2019-12-05T05:10:18.763 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/187/16' 2019-12-05T05:10:18.764 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/187/14' 2019-12-05T05:10:18.764 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/187/18' 2019-12-05T05:10:18.764 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/187/13' 2019-12-05T05:10:18.764 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/187/17' 2019-12-05T05:10:18.764 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './copyofsnap1/test1/187' 2019-12-05T05:10:18.764 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/185/19' 2019-12-05T05:10:18.765 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/185/15' 2019-12-05T05:10:18.765 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/185/20' 2019-12-05T05:10:18.765 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/185/12' 2019-12-05T05:10:18.765 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/185/10' 2019-12-05T05:10:18.766 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/185/11' 2019-12-05T05:10:18.766 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/185/16' 2019-12-05T05:10:18.766 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/185/14' 2019-12-05T05:10:18.766 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/185/18' 2019-12-05T05:10:18.766 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/185/13' 2019-12-05T05:10:18.766 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/185/17' 2019-12-05T05:10:18.766 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './copyofsnap1/test1/185' 2019-12-05T05:10:18.767 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/120/19' 2019-12-05T05:10:18.767 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/120/15' 2019-12-05T05:10:18.767 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/120/20' 2019-12-05T05:10:18.767 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/120/12' 2019-12-05T05:10:18.767 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/120/10' 2019-12-05T05:10:18.767 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/120/11' 2019-12-05T05:10:18.768 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/120/16' 2019-12-05T05:10:18.768 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/120/14' 2019-12-05T05:10:18.768 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/120/18' 2019-12-05T05:10:18.768 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/120/13' 2019-12-05T05:10:18.768 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/120/17' 2019-12-05T05:10:18.768 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './copyofsnap1/test1/120' 2019-12-05T05:10:18.768 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/110/19' 2019-12-05T05:10:18.769 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/110/15' 2019-12-05T05:10:18.769 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/110/20' 2019-12-05T05:10:18.769 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/110/12' 2019-12-05T05:10:18.769 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/110/10' 2019-12-05T05:10:18.769 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/110/11' 2019-12-05T05:10:18.769 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/110/16' 2019-12-05T05:10:18.769 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/110/14' 2019-12-05T05:10:18.770 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/110/18' 2019-12-05T05:10:18.770 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/110/13' 2019-12-05T05:10:18.770 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/110/17' 2019-12-05T05:10:18.770 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './copyofsnap1/test1/110' 2019-12-05T05:10:18.770 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/137/19' 2019-12-05T05:10:18.770 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/137/15' 2019-12-05T05:10:18.770 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/137/20' 2019-12-05T05:10:18.771 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/137/12' 2019-12-05T05:10:18.771 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/137/10' 2019-12-05T05:10:18.771 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/137/11' 2019-12-05T05:10:18.771 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/137/16' 2019-12-05T05:10:20.327 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/137/14' 2019-12-05T05:10:20.327 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/137/18' 2019-12-05T05:10:20.327 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/137/13' 2019-12-05T05:10:20.327 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/137/17' 2019-12-05T05:10:20.327 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './copyofsnap1/test1/137' 2019-12-05T05:10:20.327 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/198/19' 2019-12-05T05:10:20.328 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/198/15' 2019-12-05T05:10:20.328 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/198/20' 2019-12-05T05:10:20.328 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/198/12' 2019-12-05T05:10:20.329 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/198/10' 2019-12-05T05:10:20.329 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/198/11' 2019-12-05T05:10:20.329 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/198/16' 2019-12-05T05:10:20.329 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/198/14' 2019-12-05T05:10:20.329 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/198/18' 2019-12-05T05:10:20.329 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/198/13' 2019-12-05T05:10:20.329 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/198/17' 2019-12-05T05:10:20.330 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './copyofsnap1/test1/198' 2019-12-05T05:10:20.330 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/136/19' 2019-12-05T05:10:20.330 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/136/15' 2019-12-05T05:10:20.330 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/136/20' 2019-12-05T05:10:20.330 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/136/12' 2019-12-05T05:10:20.330 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/136/10' 2019-12-05T05:10:20.330 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/136/11' 2019-12-05T05:10:20.331 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/136/16' 2019-12-05T05:10:20.331 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/136/14' 2019-12-05T05:10:20.331 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/136/18' 2019-12-05T05:10:20.331 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/136/13' 2019-12-05T05:10:20.332 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/136/17' 2019-12-05T05:10:20.332 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './copyofsnap1/test1/136' 2019-12-05T05:10:20.332 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/125/19' 2019-12-05T05:10:20.333 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/125/15' 2019-12-05T05:10:20.333 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/125/20' 2019-12-05T05:10:20.333 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/125/12' 2019-12-05T05:10:20.333 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/125/10' 2019-12-05T05:10:20.333 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/125/11' 2019-12-05T05:10:20.333 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/125/16' 2019-12-05T05:10:20.333 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/125/14' 2019-12-05T05:10:20.333 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/125/18' 2019-12-05T05:10:20.334 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/125/13' 2019-12-05T05:10:20.334 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/125/17' 2019-12-05T05:10:20.334 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './copyofsnap1/test1/125' 2019-12-05T05:10:20.334 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/194/19' 2019-12-05T05:10:20.334 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/194/15' 2019-12-05T05:10:20.334 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/194/20' 2019-12-05T05:10:20.334 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/194/12' 2019-12-05T05:10:20.335 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/194/10' 2019-12-05T05:10:20.335 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/194/11' 2019-12-05T05:10:20.335 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/194/16' 2019-12-05T05:10:20.335 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/194/14' 2019-12-05T05:10:20.335 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/194/18' 2019-12-05T05:10:20.335 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/194/13' 2019-12-05T05:10:20.335 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/194/17' 2019-12-05T05:10:20.336 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './copyofsnap1/test1/194' 2019-12-05T05:10:20.336 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/119/19' 2019-12-05T05:10:20.336 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/119/15' 2019-12-05T05:10:20.336 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/119/20' 2019-12-05T05:10:20.336 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/119/12' 2019-12-05T05:10:20.336 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/119/10' 2019-12-05T05:10:20.336 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/119/11' 2019-12-05T05:10:20.337 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/119/16' 2019-12-05T05:10:20.337 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/119/14' 2019-12-05T05:10:20.337 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/119/18' 2019-12-05T05:10:20.337 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/119/13' 2019-12-05T05:10:20.337 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/119/17' 2019-12-05T05:10:20.337 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './copyofsnap1/test1/119' 2019-12-05T05:10:20.337 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/157/19' 2019-12-05T05:10:20.337 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/157/15' 2019-12-05T05:10:20.338 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/157/20' 2019-12-05T05:10:20.338 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/157/12' 2019-12-05T05:10:20.338 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/157/10' 2019-12-05T05:10:20.338 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/157/11' 2019-12-05T05:10:20.339 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/157/16' 2019-12-05T05:10:20.339 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/157/14' 2019-12-05T05:10:20.339 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/157/18' 2019-12-05T05:10:20.339 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/157/13' 2019-12-05T05:10:20.339 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/157/17' 2019-12-05T05:10:20.339 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './copyofsnap1/test1/157' 2019-12-05T05:10:20.339 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/172/19' 2019-12-05T05:10:20.340 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/172/15' 2019-12-05T05:10:20.340 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/172/20' 2019-12-05T05:10:20.340 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/172/12' 2019-12-05T05:10:20.340 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/172/10' 2019-12-05T05:10:20.340 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/172/11' 2019-12-05T05:10:20.340 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/172/16' 2019-12-05T05:10:20.340 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/172/14' 2019-12-05T05:10:20.340 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/172/18' 2019-12-05T05:10:20.341 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/172/13' 2019-12-05T05:10:20.341 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/172/17' 2019-12-05T05:10:20.341 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './copyofsnap1/test1/172' 2019-12-05T05:10:20.341 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/186/19' 2019-12-05T05:10:20.341 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/186/15' 2019-12-05T05:10:20.341 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/186/20' 2019-12-05T05:10:20.341 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/186/12' 2019-12-05T05:10:20.342 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/186/10' 2019-12-05T05:10:20.342 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/186/11' 2019-12-05T05:10:20.342 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/186/16' 2019-12-05T05:10:20.342 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/186/14' 2019-12-05T05:10:20.342 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/186/18' 2019-12-05T05:10:20.342 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/186/13' 2019-12-05T05:10:20.342 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/186/17' 2019-12-05T05:10:20.343 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './copyofsnap1/test1/186' 2019-12-05T05:10:20.343 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/190/19' 2019-12-05T05:10:20.343 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/190/15' 2019-12-05T05:10:20.343 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/190/20' 2019-12-05T05:10:20.343 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/190/12' 2019-12-05T05:10:20.343 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/190/10' 2019-12-05T05:10:20.343 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/190/11' 2019-12-05T05:10:20.343 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/190/16' 2019-12-05T05:10:20.344 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/190/14' 2019-12-05T05:10:23.078 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/190/18' 2019-12-05T05:10:23.078 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/190/13' 2019-12-05T05:10:23.078 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/190/17' 2019-12-05T05:10:23.078 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './copyofsnap1/test1/190' 2019-12-05T05:10:23.079 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/104/19' 2019-12-05T05:10:23.079 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/104/15' 2019-12-05T05:10:23.079 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/104/20' 2019-12-05T05:10:23.079 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/104/12' 2019-12-05T05:10:23.080 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/104/10' 2019-12-05T05:10:23.080 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/104/11' 2019-12-05T05:10:23.080 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/104/16' 2019-12-05T05:10:23.080 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/104/14' 2019-12-05T05:10:23.080 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/104/18' 2019-12-05T05:10:23.080 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/104/13' 2019-12-05T05:10:23.080 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/104/17' 2019-12-05T05:10:23.080 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './copyofsnap1/test1/104' 2019-12-05T05:10:23.081 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/164/19' 2019-12-05T05:10:23.081 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/164/15' 2019-12-05T05:10:23.081 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/164/20' 2019-12-05T05:10:23.081 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/164/12' 2019-12-05T05:10:23.081 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/164/10' 2019-12-05T05:10:23.081 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/164/11' 2019-12-05T05:10:23.081 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/164/16' 2019-12-05T05:10:23.082 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/164/14' 2019-12-05T05:10:23.082 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/164/18' 2019-12-05T05:10:23.082 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/164/13' 2019-12-05T05:10:23.082 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/164/17' 2019-12-05T05:10:23.082 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './copyofsnap1/test1/164' 2019-12-05T05:10:23.082 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/112/19' 2019-12-05T05:10:23.082 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/112/15' 2019-12-05T05:10:23.082 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/112/20' 2019-12-05T05:10:23.083 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/112/12' 2019-12-05T05:10:23.083 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/112/10' 2019-12-05T05:10:23.083 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/112/11' 2019-12-05T05:10:23.083 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/112/16' 2019-12-05T05:10:23.084 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/112/14' 2019-12-05T05:10:23.084 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/112/18' 2019-12-05T05:10:23.084 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/112/13' 2019-12-05T05:10:23.084 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/112/17' 2019-12-05T05:10:23.084 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './copyofsnap1/test1/112' 2019-12-05T05:10:23.084 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/100/19' 2019-12-05T05:10:23.084 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/100/15' 2019-12-05T05:10:23.085 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/100/20' 2019-12-05T05:10:23.085 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/100/12' 2019-12-05T05:10:23.085 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/100/10' 2019-12-05T05:10:23.085 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/100/11' 2019-12-05T05:10:23.085 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/100/16' 2019-12-05T05:10:23.085 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/100/14' 2019-12-05T05:10:23.085 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/100/18' 2019-12-05T05:10:23.085 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/100/13' 2019-12-05T05:10:23.086 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/100/17' 2019-12-05T05:10:23.086 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './copyofsnap1/test1/100' 2019-12-05T05:10:23.086 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/134/19' 2019-12-05T05:10:23.086 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/134/15' 2019-12-05T05:10:23.086 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/134/20' 2019-12-05T05:10:23.086 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/134/12' 2019-12-05T05:10:23.086 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/134/10' 2019-12-05T05:10:23.087 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/134/11' 2019-12-05T05:10:23.087 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/134/16' 2019-12-05T05:10:23.087 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/134/14' 2019-12-05T05:10:23.087 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/134/18' 2019-12-05T05:10:23.087 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/134/13' 2019-12-05T05:10:23.087 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/134/17' 2019-12-05T05:10:23.087 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './copyofsnap1/test1/134' 2019-12-05T05:10:23.087 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/115/19' 2019-12-05T05:10:23.088 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/115/15' 2019-12-05T05:10:23.088 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/115/20' 2019-12-05T05:10:23.088 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/115/12' 2019-12-05T05:10:23.088 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/115/10' 2019-12-05T05:10:23.088 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/115/11' 2019-12-05T05:10:23.088 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/115/16' 2019-12-05T05:10:23.088 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/115/14' 2019-12-05T05:10:23.089 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/115/18' 2019-12-05T05:10:23.089 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/115/13' 2019-12-05T05:10:23.089 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/115/17' 2019-12-05T05:10:23.089 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './copyofsnap1/test1/115' 2019-12-05T05:10:23.089 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/116/19' 2019-12-05T05:10:23.089 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/116/15' 2019-12-05T05:10:23.089 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/116/20' 2019-12-05T05:10:23.089 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/116/12' 2019-12-05T05:10:23.090 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/116/10' 2019-12-05T05:10:23.090 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/116/11' 2019-12-05T05:10:23.090 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/116/16' 2019-12-05T05:10:23.090 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/116/14' 2019-12-05T05:10:23.090 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/116/18' 2019-12-05T05:10:23.090 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/116/13' 2019-12-05T05:10:23.090 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/116/17' 2019-12-05T05:10:23.090 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './copyofsnap1/test1/116' 2019-12-05T05:10:23.091 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/195/19' 2019-12-05T05:10:23.091 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/195/15' 2019-12-05T05:10:23.091 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/195/20' 2019-12-05T05:10:23.091 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/195/12' 2019-12-05T05:10:23.091 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/195/10' 2019-12-05T05:10:23.091 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/195/11' 2019-12-05T05:10:23.091 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/195/16' 2019-12-05T05:10:23.092 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/195/14' 2019-12-05T05:10:23.092 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/195/18' 2019-12-05T05:10:23.092 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/195/13' 2019-12-05T05:10:23.092 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/195/17' 2019-12-05T05:10:23.092 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './copyofsnap1/test1/195' 2019-12-05T05:10:23.092 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/138/19' 2019-12-05T05:10:23.092 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/138/15' 2019-12-05T05:10:23.093 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/138/20' 2019-12-05T05:10:23.093 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/138/12' 2019-12-05T05:10:23.093 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/138/10' 2019-12-05T05:10:23.093 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/138/11' 2019-12-05T05:10:23.094 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/138/16' 2019-12-05T05:10:23.094 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/138/14' 2019-12-05T05:10:23.094 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/138/18' 2019-12-05T05:10:24.359 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/138/13' 2019-12-05T05:10:24.359 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/138/17' 2019-12-05T05:10:24.359 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './copyofsnap1/test1/138' 2019-12-05T05:10:24.359 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/135/19' 2019-12-05T05:10:24.359 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/135/15' 2019-12-05T05:10:24.359 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/135/20' 2019-12-05T05:10:24.360 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/135/12' 2019-12-05T05:10:24.360 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/135/10' 2019-12-05T05:10:24.360 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/135/11' 2019-12-05T05:10:24.360 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/135/16' 2019-12-05T05:10:24.361 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/135/14' 2019-12-05T05:10:24.361 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/135/18' 2019-12-05T05:10:24.361 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/135/13' 2019-12-05T05:10:24.361 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/135/17' 2019-12-05T05:10:24.361 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './copyofsnap1/test1/135' 2019-12-05T05:10:24.361 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/117/19' 2019-12-05T05:10:24.361 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/117/15' 2019-12-05T05:10:24.362 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/117/20' 2019-12-05T05:10:24.362 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/117/12' 2019-12-05T05:10:24.362 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/117/10' 2019-12-05T05:10:24.362 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/117/11' 2019-12-05T05:10:24.362 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/117/16' 2019-12-05T05:10:24.362 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/117/14' 2019-12-05T05:10:24.362 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/117/18' 2019-12-05T05:10:24.363 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/117/13' 2019-12-05T05:10:24.363 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/117/17' 2019-12-05T05:10:24.363 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './copyofsnap1/test1/117' 2019-12-05T05:10:24.363 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/143/19' 2019-12-05T05:10:24.363 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/143/15' 2019-12-05T05:10:24.363 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/143/20' 2019-12-05T05:10:24.363 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/143/12' 2019-12-05T05:10:24.364 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/143/10' 2019-12-05T05:10:24.364 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/143/11' 2019-12-05T05:10:24.364 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/143/16' 2019-12-05T05:10:24.364 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/143/14' 2019-12-05T05:10:24.364 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/143/18' 2019-12-05T05:10:24.364 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/143/13' 2019-12-05T05:10:24.364 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/143/17' 2019-12-05T05:10:24.364 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './copyofsnap1/test1/143' 2019-12-05T05:10:24.365 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/180/19' 2019-12-05T05:10:24.365 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/180/15' 2019-12-05T05:10:24.365 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/180/20' 2019-12-05T05:10:24.365 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/180/12' 2019-12-05T05:10:24.365 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/180/10' 2019-12-05T05:10:24.365 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/180/11' 2019-12-05T05:10:24.365 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/180/16' 2019-12-05T05:10:24.366 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/180/14' 2019-12-05T05:10:24.366 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/180/18' 2019-12-05T05:10:24.366 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/180/13' 2019-12-05T05:10:24.366 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/180/17' 2019-12-05T05:10:24.366 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './copyofsnap1/test1/180' 2019-12-05T05:10:24.366 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/126/19' 2019-12-05T05:10:24.366 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/126/15' 2019-12-05T05:10:24.367 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/126/20' 2019-12-05T05:10:24.367 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/126/12' 2019-12-05T05:10:24.367 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/126/10' 2019-12-05T05:10:24.367 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/126/11' 2019-12-05T05:10:24.367 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/126/16' 2019-12-05T05:10:24.367 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/126/14' 2019-12-05T05:10:24.367 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/126/18' 2019-12-05T05:10:24.367 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/126/13' 2019-12-05T05:10:24.368 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/126/17' 2019-12-05T05:10:24.368 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './copyofsnap1/test1/126' 2019-12-05T05:10:24.368 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/161/19' 2019-12-05T05:10:24.368 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/161/15' 2019-12-05T05:10:24.368 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/161/20' 2019-12-05T05:10:24.369 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/161/12' 2019-12-05T05:10:24.369 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/161/10' 2019-12-05T05:10:24.369 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/161/11' 2019-12-05T05:10:24.369 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/161/16' 2019-12-05T05:10:24.369 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/161/14' 2019-12-05T05:10:24.369 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/161/18' 2019-12-05T05:10:24.369 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/161/13' 2019-12-05T05:10:24.370 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/161/17' 2019-12-05T05:10:24.371 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './copyofsnap1/test1/161' 2019-12-05T05:10:24.371 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/176/19' 2019-12-05T05:10:24.371 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/176/15' 2019-12-05T05:10:24.371 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/176/20' 2019-12-05T05:10:24.372 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/176/12' 2019-12-05T05:10:24.372 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/176/10' 2019-12-05T05:10:24.372 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/176/11' 2019-12-05T05:10:24.372 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/176/16' 2019-12-05T05:10:24.372 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/176/14' 2019-12-05T05:10:24.372 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/176/18' 2019-12-05T05:10:24.372 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/176/13' 2019-12-05T05:10:24.373 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/176/17' 2019-12-05T05:10:24.373 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './copyofsnap1/test1/176' 2019-12-05T05:10:24.373 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/111/19' 2019-12-05T05:10:24.373 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/111/15' 2019-12-05T05:10:24.373 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/111/20' 2019-12-05T05:10:24.373 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/111/12' 2019-12-05T05:10:24.373 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/111/10' 2019-12-05T05:10:24.374 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/111/11' 2019-12-05T05:10:24.374 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/111/16' 2019-12-05T05:10:24.374 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/111/14' 2019-12-05T05:10:24.374 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/111/18' 2019-12-05T05:10:24.374 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/111/13' 2019-12-05T05:10:24.374 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/111/17' 2019-12-05T05:10:24.374 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './copyofsnap1/test1/111' 2019-12-05T05:10:24.374 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/191/19' 2019-12-05T05:10:24.375 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/191/15' 2019-12-05T05:10:24.375 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/191/20' 2019-12-05T05:10:24.375 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/191/12' 2019-12-05T05:10:24.375 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/191/10' 2019-12-05T05:10:24.375 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/191/11' 2019-12-05T05:10:24.375 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/191/16' 2019-12-05T05:10:24.375 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/191/14' 2019-12-05T05:10:24.376 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/191/18' 2019-12-05T05:10:24.376 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/191/13' 2019-12-05T05:10:27.882 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/191/17' 2019-12-05T05:10:27.882 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './copyofsnap1/test1/191' 2019-12-05T05:10:27.883 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/184/19' 2019-12-05T05:10:27.885 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/184/15' 2019-12-05T05:10:27.885 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/184/20' 2019-12-05T05:10:27.885 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/184/12' 2019-12-05T05:10:27.886 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/184/10' 2019-12-05T05:10:27.886 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/184/11' 2019-12-05T05:10:27.886 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/184/16' 2019-12-05T05:10:27.886 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/184/14' 2019-12-05T05:10:27.886 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/184/18' 2019-12-05T05:10:27.887 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/184/13' 2019-12-05T05:10:27.887 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/184/17' 2019-12-05T05:10:27.888 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './copyofsnap1/test1/184' 2019-12-05T05:10:27.889 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/192/19' 2019-12-05T05:10:27.889 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/192/15' 2019-12-05T05:10:27.889 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/192/20' 2019-12-05T05:10:27.889 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/192/12' 2019-12-05T05:10:27.889 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/192/10' 2019-12-05T05:10:27.889 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/192/11' 2019-12-05T05:10:27.890 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/192/16' 2019-12-05T05:10:27.890 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/192/14' 2019-12-05T05:10:27.890 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/192/18' 2019-12-05T05:10:27.890 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/192/13' 2019-12-05T05:10:27.890 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/192/17' 2019-12-05T05:10:27.890 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './copyofsnap1/test1/192' 2019-12-05T05:10:27.890 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/103/19' 2019-12-05T05:10:27.891 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/103/15' 2019-12-05T05:10:27.891 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/103/20' 2019-12-05T05:10:27.891 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/103/12' 2019-12-05T05:10:27.891 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/103/10' 2019-12-05T05:10:27.891 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/103/11' 2019-12-05T05:10:27.891 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/103/16' 2019-12-05T05:10:27.891 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/103/14' 2019-12-05T05:10:27.892 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/103/18' 2019-12-05T05:10:27.892 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/103/13' 2019-12-05T05:10:27.892 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/103/17' 2019-12-05T05:10:27.892 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './copyofsnap1/test1/103' 2019-12-05T05:10:27.892 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/128/19' 2019-12-05T05:10:27.892 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/128/15' 2019-12-05T05:10:27.892 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/128/20' 2019-12-05T05:10:27.893 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/128/12' 2019-12-05T05:10:27.893 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/128/10' 2019-12-05T05:10:27.893 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/128/11' 2019-12-05T05:10:27.893 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/128/16' 2019-12-05T05:10:27.900 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/128/14' 2019-12-05T05:10:27.900 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/128/18' 2019-12-05T05:10:27.900 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/128/13' 2019-12-05T05:10:27.900 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/128/17' 2019-12-05T05:10:27.901 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './copyofsnap1/test1/128' 2019-12-05T05:10:27.901 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/144/19' 2019-12-05T05:10:27.901 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/144/15' 2019-12-05T05:10:27.901 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/144/20' 2019-12-05T05:10:27.901 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/144/12' 2019-12-05T05:10:27.901 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/144/10' 2019-12-05T05:10:27.901 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/144/11' 2019-12-05T05:10:27.902 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/144/16' 2019-12-05T05:10:27.902 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/144/14' 2019-12-05T05:10:27.902 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/144/18' 2019-12-05T05:10:27.902 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/144/13' 2019-12-05T05:10:27.902 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/144/17' 2019-12-05T05:10:27.902 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './copyofsnap1/test1/144' 2019-12-05T05:10:27.902 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/146/19' 2019-12-05T05:10:27.903 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/146/15' 2019-12-05T05:10:27.903 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/146/20' 2019-12-05T05:10:27.903 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/146/12' 2019-12-05T05:10:27.903 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/146/10' 2019-12-05T05:10:27.903 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/146/11' 2019-12-05T05:10:27.903 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/146/16' 2019-12-05T05:10:27.903 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/146/14' 2019-12-05T05:10:27.904 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/146/18' 2019-12-05T05:10:27.904 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/146/13' 2019-12-05T05:10:27.904 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/146/17' 2019-12-05T05:10:27.904 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './copyofsnap1/test1/146' 2019-12-05T05:10:27.904 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/141/19' 2019-12-05T05:10:27.904 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/141/15' 2019-12-05T05:10:27.905 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/141/20' 2019-12-05T05:10:27.905 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/141/12' 2019-12-05T05:10:27.907 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/141/10' 2019-12-05T05:10:27.907 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/141/11' 2019-12-05T05:10:27.907 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/141/16' 2019-12-05T05:10:27.907 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/141/14' 2019-12-05T05:10:27.908 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/141/18' 2019-12-05T05:10:27.908 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/141/13' 2019-12-05T05:10:27.908 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/141/17' 2019-12-05T05:10:27.908 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './copyofsnap1/test1/141' 2019-12-05T05:10:27.908 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/182/19' 2019-12-05T05:10:27.908 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/182/15' 2019-12-05T05:10:27.908 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/182/20' 2019-12-05T05:10:27.909 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/182/12' 2019-12-05T05:10:27.909 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/182/10' 2019-12-05T05:10:27.909 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/182/11' 2019-12-05T05:10:27.909 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/182/16' 2019-12-05T05:10:27.909 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/182/14' 2019-12-05T05:10:27.909 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/182/18' 2019-12-05T05:10:27.909 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/182/13' 2019-12-05T05:10:27.910 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/182/17' 2019-12-05T05:10:27.910 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './copyofsnap1/test1/182' 2019-12-05T05:10:27.910 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/107/19' 2019-12-05T05:10:27.910 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/107/15' 2019-12-05T05:10:27.910 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/107/20' 2019-12-05T05:10:27.910 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/107/12' 2019-12-05T05:10:27.911 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/107/10' 2019-12-05T05:10:27.911 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/107/11' 2019-12-05T05:10:27.911 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/107/16' 2019-12-05T05:10:27.911 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/107/14' 2019-12-05T05:10:27.911 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/107/18' 2019-12-05T05:10:27.911 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/107/13' 2019-12-05T05:10:27.911 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/107/17' 2019-12-05T05:10:30.254 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './copyofsnap1/test1/107' 2019-12-05T05:10:30.254 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/178/19' 2019-12-05T05:10:30.254 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/178/15' 2019-12-05T05:10:30.254 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/178/20' 2019-12-05T05:10:30.254 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/178/12' 2019-12-05T05:10:30.255 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/178/10' 2019-12-05T05:10:30.255 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/178/11' 2019-12-05T05:10:30.255 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/178/16' 2019-12-05T05:10:30.255 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/178/14' 2019-12-05T05:10:30.255 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/178/18' 2019-12-05T05:10:30.255 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/178/13' 2019-12-05T05:10:30.255 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/178/17' 2019-12-05T05:10:30.256 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './copyofsnap1/test1/178' 2019-12-05T05:10:30.256 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/156/19' 2019-12-05T05:10:30.256 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/156/15' 2019-12-05T05:10:30.256 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/156/20' 2019-12-05T05:10:30.257 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/156/12' 2019-12-05T05:10:30.257 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/156/10' 2019-12-05T05:10:30.257 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/156/11' 2019-12-05T05:10:30.257 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/156/16' 2019-12-05T05:10:30.257 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/156/14' 2019-12-05T05:10:30.257 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/156/18' 2019-12-05T05:10:30.258 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/156/13' 2019-12-05T05:10:30.258 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/156/17' 2019-12-05T05:10:30.258 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './copyofsnap1/test1/156' 2019-12-05T05:10:30.258 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/131/19' 2019-12-05T05:10:30.258 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/131/15' 2019-12-05T05:10:30.258 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/131/20' 2019-12-05T05:10:30.259 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/131/12' 2019-12-05T05:10:30.259 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/131/10' 2019-12-05T05:10:30.259 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/131/11' 2019-12-05T05:10:30.265 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/131/16' 2019-12-05T05:10:30.265 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/131/14' 2019-12-05T05:10:30.265 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/131/18' 2019-12-05T05:10:30.265 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/131/13' 2019-12-05T05:10:30.265 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/131/17' 2019-12-05T05:10:30.271 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './copyofsnap1/test1/131' 2019-12-05T05:10:30.271 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/123/19' 2019-12-05T05:10:30.272 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/123/15' 2019-12-05T05:10:30.272 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/123/20' 2019-12-05T05:10:30.272 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/123/12' 2019-12-05T05:10:30.272 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/123/10' 2019-12-05T05:10:30.272 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/123/11' 2019-12-05T05:10:30.273 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/123/16' 2019-12-05T05:10:30.273 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/123/14' 2019-12-05T05:10:30.273 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/123/18' 2019-12-05T05:10:30.273 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/123/13' 2019-12-05T05:10:30.273 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/123/17' 2019-12-05T05:10:30.273 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './copyofsnap1/test1/123' 2019-12-05T05:10:30.274 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/160/19' 2019-12-05T05:10:30.277 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/160/15' 2019-12-05T05:10:30.277 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/160/20' 2019-12-05T05:10:30.277 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/160/12' 2019-12-05T05:10:30.277 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/160/10' 2019-12-05T05:10:30.277 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/160/11' 2019-12-05T05:10:30.278 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/160/16' 2019-12-05T05:10:30.278 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/160/14' 2019-12-05T05:10:30.278 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/160/18' 2019-12-05T05:10:30.278 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/160/13' 2019-12-05T05:10:30.278 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/160/17' 2019-12-05T05:10:30.278 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './copyofsnap1/test1/160' 2019-12-05T05:10:30.278 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/165/19' 2019-12-05T05:10:30.279 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/165/15' 2019-12-05T05:10:30.279 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/165/20' 2019-12-05T05:10:30.279 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/165/12' 2019-12-05T05:10:30.279 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/165/10' 2019-12-05T05:10:30.279 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/165/11' 2019-12-05T05:10:30.279 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/165/16' 2019-12-05T05:10:30.279 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/165/14' 2019-12-05T05:10:30.279 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/165/18' 2019-12-05T05:10:30.280 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/165/13' 2019-12-05T05:10:30.280 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/165/17' 2019-12-05T05:10:30.280 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './copyofsnap1/test1/165' 2019-12-05T05:10:30.280 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/101/19' 2019-12-05T05:10:30.280 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/101/15' 2019-12-05T05:10:30.282 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/101/20' 2019-12-05T05:10:30.282 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/101/12' 2019-12-05T05:10:30.283 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/101/10' 2019-12-05T05:10:30.283 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/101/11' 2019-12-05T05:10:30.283 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/101/16' 2019-12-05T05:10:30.283 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/101/14' 2019-12-05T05:10:30.293 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/101/18' 2019-12-05T05:10:30.293 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/101/13' 2019-12-05T05:10:30.294 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/101/17' 2019-12-05T05:10:30.294 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './copyofsnap1/test1/101' 2019-12-05T05:10:30.294 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/148/19' 2019-12-05T05:10:30.294 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/148/15' 2019-12-05T05:10:30.294 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/148/20' 2019-12-05T05:10:30.295 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/148/12' 2019-12-05T05:10:30.295 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/148/10' 2019-12-05T05:10:30.296 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/148/11' 2019-12-05T05:10:30.296 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/148/16' 2019-12-05T05:10:30.296 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/148/14' 2019-12-05T05:10:30.296 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/148/18' 2019-12-05T05:10:30.296 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/148/13' 2019-12-05T05:10:30.296 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/148/17' 2019-12-05T05:10:30.297 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './copyofsnap1/test1/148' 2019-12-05T05:10:30.297 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/168/19' 2019-12-05T05:10:30.297 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/168/15' 2019-12-05T05:10:30.297 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/168/20' 2019-12-05T05:10:30.297 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/168/12' 2019-12-05T05:10:30.297 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/168/10' 2019-12-05T05:10:30.297 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/168/11' 2019-12-05T05:10:30.298 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/168/16' 2019-12-05T05:10:30.298 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/168/14' 2019-12-05T05:10:30.298 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/168/18' 2019-12-05T05:10:30.298 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/168/13' 2019-12-05T05:10:30.298 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/168/17' 2019-12-05T05:10:30.298 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './copyofsnap1/test1/168' 2019-12-05T05:10:34.692 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/179/19' 2019-12-05T05:10:34.693 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/179/15' 2019-12-05T05:10:34.693 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/179/20' 2019-12-05T05:10:34.693 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/179/12' 2019-12-05T05:10:34.693 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/179/10' 2019-12-05T05:10:34.693 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/179/11' 2019-12-05T05:10:34.693 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/179/16' 2019-12-05T05:10:34.693 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/179/14' 2019-12-05T05:10:34.694 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/179/18' 2019-12-05T05:10:34.694 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/179/13' 2019-12-05T05:10:34.694 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/179/17' 2019-12-05T05:10:34.694 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './copyofsnap1/test1/179' 2019-12-05T05:10:34.694 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/132/19' 2019-12-05T05:10:34.694 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/132/15' 2019-12-05T05:10:34.694 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/132/20' 2019-12-05T05:10:34.695 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/132/12' 2019-12-05T05:10:34.695 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/132/10' 2019-12-05T05:10:34.695 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/132/11' 2019-12-05T05:10:34.695 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/132/16' 2019-12-05T05:10:34.696 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/132/14' 2019-12-05T05:10:34.696 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/132/18' 2019-12-05T05:10:34.696 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/132/13' 2019-12-05T05:10:34.696 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/132/17' 2019-12-05T05:10:34.696 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './copyofsnap1/test1/132' 2019-12-05T05:10:34.697 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/199/19' 2019-12-05T05:10:34.697 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/199/15' 2019-12-05T05:10:34.697 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/199/20' 2019-12-05T05:10:34.697 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/199/12' 2019-12-05T05:10:34.697 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/199/10' 2019-12-05T05:10:34.697 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/199/11' 2019-12-05T05:10:34.697 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/199/16' 2019-12-05T05:10:34.698 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/199/14' 2019-12-05T05:10:34.698 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/199/18' 2019-12-05T05:10:34.698 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/199/13' 2019-12-05T05:10:34.698 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/199/17' 2019-12-05T05:10:34.698 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './copyofsnap1/test1/199' 2019-12-05T05:10:34.698 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/163/19' 2019-12-05T05:10:34.698 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/163/15' 2019-12-05T05:10:34.699 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/163/20' 2019-12-05T05:10:34.699 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/163/12' 2019-12-05T05:10:34.699 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/163/10' 2019-12-05T05:10:34.699 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/163/11' 2019-12-05T05:10:34.699 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/163/16' 2019-12-05T05:10:34.699 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/163/14' 2019-12-05T05:10:34.700 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/163/18' 2019-12-05T05:10:34.700 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/163/13' 2019-12-05T05:10:34.700 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/163/17' 2019-12-05T05:10:34.700 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './copyofsnap1/test1/163' 2019-12-05T05:10:34.700 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/153/19' 2019-12-05T05:10:34.701 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/153/15' 2019-12-05T05:10:34.701 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/153/20' 2019-12-05T05:10:34.701 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/153/12' 2019-12-05T05:10:34.701 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/153/10' 2019-12-05T05:10:34.701 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/153/11' 2019-12-05T05:10:34.701 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/153/16' 2019-12-05T05:10:34.701 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/153/14' 2019-12-05T05:10:34.701 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/153/18' 2019-12-05T05:10:34.702 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/153/13' 2019-12-05T05:10:34.702 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/153/17' 2019-12-05T05:10:34.702 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './copyofsnap1/test1/153' 2019-12-05T05:10:34.702 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/122/19' 2019-12-05T05:10:34.702 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/122/15' 2019-12-05T05:10:34.702 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/122/20' 2019-12-05T05:10:34.702 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/122/12' 2019-12-05T05:10:34.703 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/122/10' 2019-12-05T05:10:34.703 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/122/11' 2019-12-05T05:10:34.703 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/122/16' 2019-12-05T05:10:34.704 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/122/14' 2019-12-05T05:10:34.704 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/122/18' 2019-12-05T05:10:34.704 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/122/13' 2019-12-05T05:10:34.704 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/122/17' 2019-12-05T05:10:34.705 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './copyofsnap1/test1/122' 2019-12-05T05:10:34.705 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/162/19' 2019-12-05T05:10:34.705 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/162/15' 2019-12-05T05:10:34.705 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/162/20' 2019-12-05T05:10:34.705 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/162/12' 2019-12-05T05:10:34.706 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/162/10' 2019-12-05T05:10:34.706 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/162/11' 2019-12-05T05:10:34.706 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/162/16' 2019-12-05T05:10:34.706 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/162/14' 2019-12-05T05:10:34.707 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/162/18' 2019-12-05T05:10:34.707 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/162/13' 2019-12-05T05:10:34.707 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/162/17' 2019-12-05T05:10:34.707 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './copyofsnap1/test1/162' 2019-12-05T05:10:34.707 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/197/19' 2019-12-05T05:10:34.708 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/197/15' 2019-12-05T05:10:34.708 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/197/20' 2019-12-05T05:10:34.708 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/197/12' 2019-12-05T05:10:34.708 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/197/10' 2019-12-05T05:10:34.708 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/197/11' 2019-12-05T05:10:34.708 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/197/16' 2019-12-05T05:10:34.708 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/197/14' 2019-12-05T05:10:34.709 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/197/18' 2019-12-05T05:10:34.709 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/197/13' 2019-12-05T05:10:34.709 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/197/17' 2019-12-05T05:10:34.709 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './copyofsnap1/test1/197' 2019-12-05T05:10:34.709 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/113/19' 2019-12-05T05:10:34.709 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/113/15' 2019-12-05T05:10:34.710 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/113/20' 2019-12-05T05:10:34.710 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/113/12' 2019-12-05T05:10:34.710 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/113/10' 2019-12-05T05:10:34.710 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/113/11' 2019-12-05T05:10:34.710 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/113/16' 2019-12-05T05:10:34.710 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/113/14' 2019-12-05T05:10:34.712 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/113/18' 2019-12-05T05:10:34.712 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/113/13' 2019-12-05T05:10:34.712 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/113/17' 2019-12-05T05:10:34.713 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './copyofsnap1/test1/113' 2019-12-05T05:10:34.726 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/152/19' 2019-12-05T05:10:35.138 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:10:35.142 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:10:37.613 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/152/15' 2019-12-05T05:10:37.613 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/152/20' 2019-12-05T05:10:37.613 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/152/12' 2019-12-05T05:10:37.614 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/152/10' 2019-12-05T05:10:37.614 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/152/11' 2019-12-05T05:10:37.614 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/152/16' 2019-12-05T05:10:37.614 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/152/14' 2019-12-05T05:10:37.614 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/152/18' 2019-12-05T05:10:37.614 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/152/13' 2019-12-05T05:10:37.614 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/152/17' 2019-12-05T05:10:37.614 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './copyofsnap1/test1/152' 2019-12-05T05:10:37.619 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/121/19' 2019-12-05T05:10:37.620 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/121/15' 2019-12-05T05:10:37.620 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/121/20' 2019-12-05T05:10:37.620 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/121/12' 2019-12-05T05:10:37.620 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/121/10' 2019-12-05T05:10:37.620 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/121/11' 2019-12-05T05:10:37.620 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/121/16' 2019-12-05T05:10:37.620 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/121/14' 2019-12-05T05:10:37.621 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/121/18' 2019-12-05T05:10:37.621 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/121/13' 2019-12-05T05:10:37.621 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/121/17' 2019-12-05T05:10:37.621 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './copyofsnap1/test1/121' 2019-12-05T05:10:37.621 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/118/19' 2019-12-05T05:10:37.621 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/118/15' 2019-12-05T05:10:37.621 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/118/20' 2019-12-05T05:10:37.621 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/118/12' 2019-12-05T05:10:37.622 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/118/10' 2019-12-05T05:10:37.622 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/118/11' 2019-12-05T05:10:37.631 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/118/16' 2019-12-05T05:10:37.631 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/118/14' 2019-12-05T05:10:37.631 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/118/18' 2019-12-05T05:10:37.631 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/118/13' 2019-12-05T05:10:37.631 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/118/17' 2019-12-05T05:10:37.631 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './copyofsnap1/test1/118' 2019-12-05T05:10:37.632 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/196/19' 2019-12-05T05:10:37.632 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/196/15' 2019-12-05T05:10:37.632 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/196/20' 2019-12-05T05:10:37.632 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/196/12' 2019-12-05T05:10:37.632 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/196/10' 2019-12-05T05:10:37.632 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/196/11' 2019-12-05T05:10:37.632 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/196/16' 2019-12-05T05:10:37.633 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/196/14' 2019-12-05T05:10:37.633 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/196/18' 2019-12-05T05:10:37.633 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/196/13' 2019-12-05T05:10:37.633 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/196/17' 2019-12-05T05:10:37.633 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './copyofsnap1/test1/196' 2019-12-05T05:10:37.636 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/124/19' 2019-12-05T05:10:37.636 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/124/15' 2019-12-05T05:10:37.636 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/124/20' 2019-12-05T05:10:37.637 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/124/12' 2019-12-05T05:10:37.637 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/124/10' 2019-12-05T05:10:37.637 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/124/11' 2019-12-05T05:10:37.637 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/124/16' 2019-12-05T05:10:37.637 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/124/14' 2019-12-05T05:10:37.637 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/124/18' 2019-12-05T05:10:37.637 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/124/13' 2019-12-05T05:10:37.638 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/124/17' 2019-12-05T05:10:37.638 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './copyofsnap1/test1/124' 2019-12-05T05:10:37.638 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/188/19' 2019-12-05T05:10:37.638 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/188/15' 2019-12-05T05:10:37.638 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/188/20' 2019-12-05T05:10:37.638 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/188/12' 2019-12-05T05:10:37.638 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/188/10' 2019-12-05T05:10:37.639 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/188/11' 2019-12-05T05:10:37.639 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/188/16' 2019-12-05T05:10:37.639 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/188/14' 2019-12-05T05:10:37.639 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/188/18' 2019-12-05T05:10:37.639 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/188/13' 2019-12-05T05:10:37.639 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/188/17' 2019-12-05T05:10:37.639 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './copyofsnap1/test1/188' 2019-12-05T05:10:37.639 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/145/19' 2019-12-05T05:10:37.640 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/145/15' 2019-12-05T05:10:37.640 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/145/20' 2019-12-05T05:10:37.640 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/145/12' 2019-12-05T05:10:37.640 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/145/10' 2019-12-05T05:10:37.640 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/145/11' 2019-12-05T05:10:37.640 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/145/16' 2019-12-05T05:10:37.640 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/145/14' 2019-12-05T05:10:37.641 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/145/18' 2019-12-05T05:10:37.641 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/145/13' 2019-12-05T05:10:37.641 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/145/17' 2019-12-05T05:10:37.641 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './copyofsnap1/test1/145' 2019-12-05T05:10:37.641 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/130/19' 2019-12-05T05:10:37.641 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/130/15' 2019-12-05T05:10:37.641 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/130/20' 2019-12-05T05:10:37.642 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/130/12' 2019-12-05T05:10:37.642 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/130/10' 2019-12-05T05:10:37.642 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/130/11' 2019-12-05T05:10:37.652 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/130/16' 2019-12-05T05:10:37.652 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/130/14' 2019-12-05T05:10:37.652 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/130/18' 2019-12-05T05:10:37.652 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/130/13' 2019-12-05T05:10:37.652 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/130/17' 2019-12-05T05:10:37.652 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './copyofsnap1/test1/130' 2019-12-05T05:10:37.653 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/193/19' 2019-12-05T05:10:37.653 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/193/15' 2019-12-05T05:10:37.653 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/193/20' 2019-12-05T05:10:37.653 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/193/12' 2019-12-05T05:10:37.653 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/193/10' 2019-12-05T05:10:37.654 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/193/11' 2019-12-05T05:10:37.654 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/193/16' 2019-12-05T05:10:37.654 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/193/14' 2019-12-05T05:10:37.654 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/193/18' 2019-12-05T05:10:37.654 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/193/13' 2019-12-05T05:10:37.654 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/193/17' 2019-12-05T05:10:37.655 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './copyofsnap1/test1/193' 2019-12-05T05:10:37.655 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/175/19' 2019-12-05T05:10:37.655 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/175/15' 2019-12-05T05:10:40.140 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/175/20' 2019-12-05T05:10:40.141 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/175/12' 2019-12-05T05:10:40.141 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/175/10' 2019-12-05T05:10:40.141 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/175/11' 2019-12-05T05:10:40.141 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/175/16' 2019-12-05T05:10:40.141 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/175/14' 2019-12-05T05:10:40.141 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/175/18' 2019-12-05T05:10:40.142 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/175/13' 2019-12-05T05:10:40.142 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/175/17' 2019-12-05T05:10:40.142 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './copyofsnap1/test1/175' 2019-12-05T05:10:40.142 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/181/19' 2019-12-05T05:10:40.142 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/181/15' 2019-12-05T05:10:40.142 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/181/20' 2019-12-05T05:10:40.143 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/181/12' 2019-12-05T05:10:40.143 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/181/10' 2019-12-05T05:10:40.143 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/181/11' 2019-12-05T05:10:40.143 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/181/16' 2019-12-05T05:10:40.143 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/181/14' 2019-12-05T05:10:40.143 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/181/18' 2019-12-05T05:10:40.143 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/181/13' 2019-12-05T05:10:40.144 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/181/17' 2019-12-05T05:10:40.144 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './copyofsnap1/test1/181' 2019-12-05T05:10:40.144 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/127/19' 2019-12-05T05:10:40.144 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/127/15' 2019-12-05T05:10:40.144 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/127/20' 2019-12-05T05:10:40.144 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/127/12' 2019-12-05T05:10:40.145 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/127/10' 2019-12-05T05:10:40.145 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/127/11' 2019-12-05T05:10:40.145 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/127/16' 2019-12-05T05:10:40.145 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/127/14' 2019-12-05T05:10:40.145 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/127/18' 2019-12-05T05:10:40.145 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/127/13' 2019-12-05T05:10:40.145 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/127/17' 2019-12-05T05:10:40.146 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './copyofsnap1/test1/127' 2019-12-05T05:10:40.146 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/108/19' 2019-12-05T05:10:40.146 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/108/15' 2019-12-05T05:10:40.146 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/108/20' 2019-12-05T05:10:40.146 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/108/12' 2019-12-05T05:10:40.146 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/108/10' 2019-12-05T05:10:40.147 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/108/11' 2019-12-05T05:10:40.147 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/108/16' 2019-12-05T05:10:40.147 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/108/14' 2019-12-05T05:10:40.147 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/108/18' 2019-12-05T05:10:40.147 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/108/13' 2019-12-05T05:10:40.147 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/108/17' 2019-12-05T05:10:40.147 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './copyofsnap1/test1/108' 2019-12-05T05:10:40.148 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/183/19' 2019-12-05T05:10:40.148 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/183/15' 2019-12-05T05:10:40.148 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/183/20' 2019-12-05T05:10:40.148 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/183/12' 2019-12-05T05:10:40.148 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/183/10' 2019-12-05T05:10:40.148 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/183/11' 2019-12-05T05:10:40.149 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/183/16' 2019-12-05T05:10:40.149 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/183/14' 2019-12-05T05:10:40.149 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/183/18' 2019-12-05T05:10:40.149 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/183/13' 2019-12-05T05:10:40.149 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/183/17' 2019-12-05T05:10:40.149 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './copyofsnap1/test1/183' 2019-12-05T05:10:40.149 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/177/19' 2019-12-05T05:10:40.150 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/177/15' 2019-12-05T05:10:40.150 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/177/20' 2019-12-05T05:10:40.150 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/177/12' 2019-12-05T05:10:40.150 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/177/10' 2019-12-05T05:10:40.150 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/177/11' 2019-12-05T05:10:40.150 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/177/16' 2019-12-05T05:10:40.150 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/177/14' 2019-12-05T05:10:40.151 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/177/18' 2019-12-05T05:10:40.151 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/177/13' 2019-12-05T05:10:40.151 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/177/17' 2019-12-05T05:10:40.151 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './copyofsnap1/test1/177' 2019-12-05T05:10:40.151 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/169/19' 2019-12-05T05:10:40.151 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/169/15' 2019-12-05T05:10:40.151 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/169/20' 2019-12-05T05:10:40.152 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/169/12' 2019-12-05T05:10:40.152 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/169/10' 2019-12-05T05:10:40.152 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/169/11' 2019-12-05T05:10:40.152 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/169/16' 2019-12-05T05:10:40.152 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/169/14' 2019-12-05T05:10:40.152 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/169/18' 2019-12-05T05:10:40.153 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/169/13' 2019-12-05T05:10:40.153 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/169/17' 2019-12-05T05:10:40.153 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './copyofsnap1/test1/169' 2019-12-05T05:10:40.153 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/154/19' 2019-12-05T05:10:40.153 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/154/15' 2019-12-05T05:10:40.153 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/154/20' 2019-12-05T05:10:40.153 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/154/12' 2019-12-05T05:10:40.154 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/154/10' 2019-12-05T05:10:40.154 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/154/11' 2019-12-05T05:10:40.154 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/154/16' 2019-12-05T05:10:40.154 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/154/14' 2019-12-05T05:10:40.154 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/154/18' 2019-12-05T05:10:40.154 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/154/13' 2019-12-05T05:10:40.154 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/154/17' 2019-12-05T05:10:40.155 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './copyofsnap1/test1/154' 2019-12-05T05:10:40.155 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/171/19' 2019-12-05T05:10:40.155 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/171/15' 2019-12-05T05:10:40.155 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/171/20' 2019-12-05T05:10:40.155 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/171/12' 2019-12-05T05:10:40.155 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/171/10' 2019-12-05T05:10:40.156 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/171/11' 2019-12-05T05:10:40.156 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/171/16' 2019-12-05T05:10:40.156 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/171/14' 2019-12-05T05:10:40.156 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/171/18' 2019-12-05T05:10:40.156 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/171/13' 2019-12-05T05:10:40.156 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/171/17' 2019-12-05T05:10:40.156 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './copyofsnap1/test1/171' 2019-12-05T05:10:40.157 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/139/19' 2019-12-05T05:10:40.157 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/139/15' 2019-12-05T05:10:40.157 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/139/20' 2019-12-05T05:10:40.986 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/139/12' 2019-12-05T05:10:40.986 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/139/10' 2019-12-05T05:10:40.987 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/139/11' 2019-12-05T05:10:40.987 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/139/16' 2019-12-05T05:10:40.987 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/139/14' 2019-12-05T05:10:40.987 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/139/18' 2019-12-05T05:10:40.987 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/139/13' 2019-12-05T05:10:40.987 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/139/17' 2019-12-05T05:10:40.987 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './copyofsnap1/test1/139' 2019-12-05T05:10:40.988 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/102/19' 2019-12-05T05:10:40.988 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/102/15' 2019-12-05T05:10:40.988 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/102/20' 2019-12-05T05:10:40.988 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/102/12' 2019-12-05T05:10:40.988 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/102/10' 2019-12-05T05:10:40.988 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/102/11' 2019-12-05T05:10:40.988 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/102/16' 2019-12-05T05:10:40.989 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/102/14' 2019-12-05T05:10:40.989 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/102/18' 2019-12-05T05:10:40.989 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/102/13' 2019-12-05T05:10:40.989 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/102/17' 2019-12-05T05:10:40.989 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './copyofsnap1/test1/102' 2019-12-05T05:10:40.989 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/129/19' 2019-12-05T05:10:40.990 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/129/15' 2019-12-05T05:10:40.990 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/129/20' 2019-12-05T05:10:40.990 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/129/12' 2019-12-05T05:10:40.990 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/129/10' 2019-12-05T05:10:40.990 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/129/11' 2019-12-05T05:10:40.990 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/129/16' 2019-12-05T05:10:40.990 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/129/14' 2019-12-05T05:10:40.990 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/129/18' 2019-12-05T05:10:40.991 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/129/13' 2019-12-05T05:10:40.991 INFO:tasks.workunit.client.0.smithi042.stdout:removed './copyofsnap1/test1/129/17' 2019-12-05T05:10:40.991 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './copyofsnap1/test1/129' 2019-12-05T05:10:40.991 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './copyofsnap1/test1' 2019-12-05T05:10:40.991 INFO:tasks.workunit.client.0.smithi042.stdout:removed directory './copyofsnap1' 2019-12-05T05:10:40.991 INFO:tasks.workunit.client.0.smithi042.stdout:OK 2019-12-05T05:10:40.992 INFO:teuthology.orchestra.run:Running command with timeout 3600 2019-12-05T05:10:40.992 INFO:teuthology.orchestra.run.smithi042:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2019-12-05T05:10:41.213 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-authwb.sh... 2019-12-05T05:10:41.213 INFO:teuthology.orchestra.run.smithi042:workunit test fs/snaps/snaptest-authwb.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c5290805033483ba0cec165c06bde07dae5f19ec TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-authwb.sh 2019-12-05T05:10:41.246 INFO:tasks.workunit.client.0.smithi042.stderr:+ set -e 2019-12-05T05:10:41.246 INFO:tasks.workunit.client.0.smithi042.stderr:+ ceph fs set cephfs allow_new_snaps true --yes-i-really-mean-it 2019-12-05T05:10:41.367 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:41.341+0000 7fbc65b47700 1 Processor -- start 2019-12-05T05:10:41.367 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:41.341+0000 7fbc65b47700 1 -- start start 2019-12-05T05:10:41.368 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:41.345+0000 7fbc65b47700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbc600a69f0 0x7fbc600a2840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:10:41.368 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:41.345+0000 7fbc65b47700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fbc600a8030 0x7fbc600652f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:10:41.368 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:41.345+0000 7fbc65b47700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc60065840 con 0x7fbc60144fe0 2019-12-05T05:10:41.368 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:41.345+0000 7fbc65b47700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc60065b20 con 0x7fbc600a8030 2019-12-05T05:10:41.370 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:41.345+0000 7fbc65b47700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc600a25a0 con 0x7fbc600a69f0 2019-12-05T05:10:41.370 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:41.345+0000 7fbc5f7fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fbc600a8030 0x7fbc600652f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:10:41.370 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:41.345+0000 7fbc5f7fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fbc600a8030 0x7fbc600652f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:57334/0 (socket says 172.21.15.42:57334) 2019-12-05T05:10:41.370 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:41.345+0000 7fbc5f7fe700 1 -- 172.21.15.42:0/1740788005 learned_addr learned my addr 172.21.15.42:0/1740788005 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:10:41.371 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:41.345+0000 7fbc5ffff700 1 --2- 172.21.15.42:0/1740788005 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbc600a69f0 0x7fbc600a2840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:10:41.371 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:41.345+0000 7fbc5f7fe700 1 -- 172.21.15.42:0/1740788005 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbc600a69f0 msgr2=0x7fbc600a2840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:10:41.372 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:41.345+0000 7fbc5f7fe700 1 --2- 172.21.15.42:0/1740788005 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbc600a69f0 0x7fbc600a2840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:10:41.373 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:41.345+0000 7fbc5effd700 1 -- 172.21.15.42:0/1740788005 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2638779146 0 0) 0x7fbc58001450 con 0x7fbc60144fe0 2019-12-05T05:10:41.373 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:41.345+0000 7fbc5f7fe700 1 -- 172.21.15.42:0/1740788005 >> v1:172.21.15.40:6789/0 conn(0x7fbc60144fe0 legacy=0x7fbc6009c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:10:41.373 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:41.345+0000 7fbc5f7fe700 1 -- 172.21.15.42:0/1740788005 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc600a7b60 con 0x7fbc600a8030 2019-12-05T05:10:41.373 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:41.345+0000 7fbc5f7fe700 1 --2- 172.21.15.42:0/1740788005 >> v2:172.21.15.42:3300/0 conn(0x7fbc600a8030 0x7fbc600652f0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fbc600a7630 tx=0x7fbc5400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:41.373 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:41.345+0000 7fbc5f7fe700 1 --2- 172.21.15.42:0/1740788005 >> v2:172.21.15.42:3300/0 conn(0x7fbc600a8030 0x7fbc600652f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fbc600a7630 tx=0x7fbc5400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:41.373 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:41.345+0000 7fbc5f7fe700 1 --2- 172.21.15.42:0/1740788005 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbc600a8030 0x7fbc600652f0 secure :-1 s=READY pgs=403 cs=0 l=1 rx=0x7fbc600a7630 tx=0x7fbc5400fd30).ready entity=mon.1 client_cookie=eb02447b17b67fc0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:10:41.374 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:41.345+0000 7fbc5f7fe700 1 --2- 172.21.15.42:0/1740788005 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbc600a8030 0x7fbc600652f0 secure :-1 s=THROTTLE_DONE pgs=403 cs=0 l=1 rx=0x7fbc600a7630 tx=0x7fbc5400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:41.374 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:41.345+0000 7fbc5f7fe700 1 --2- 172.21.15.42:0/1740788005 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbc600a8030 0x7fbc600652f0 secure :-1 s=THROTTLE_DONE pgs=403 cs=0 l=1 rx=0x7fbc600a7630 tx=0x7fbc5400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:41.374 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:41.345+0000 7fbc5effd700 1 -- 172.21.15.42:0/1740788005 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbc540113b0 con 0x7fbc600a8030 2019-12-05T05:10:41.374 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:41.345+0000 7fbc5f7fe700 1 --2- 172.21.15.42:0/1740788005 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbc600a8030 0x7fbc600652f0 secure :-1 s=THROTTLE_DONE pgs=403 cs=0 l=1 rx=0x7fbc600a7630 tx=0x7fbc5400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:41.374 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:41.345+0000 7fbc5effd700 1 -- 172.21.15.42:0/1740788005 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc54008bb0 con 0x7fbc600a8030 2019-12-05T05:10:41.374 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:41.345+0000 7fbc5effd700 1 -- 172.21.15.42:0/1740788005 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbc54011990 con 0x7fbc600a8030 2019-12-05T05:10:41.375 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:41.345+0000 7fbc65b47700 1 -- 172.21.15.42:0/1740788005 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbc600a8030 msgr2=0x7fbc600652f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:10:41.375 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:41.345+0000 7fbc65b47700 1 --2- 172.21.15.42:0/1740788005 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbc600a8030 0x7fbc600652f0 secure :-1 s=READY pgs=403 cs=0 l=1 rx=0x7fbc600a7630 tx=0x7fbc5400fd30).stop 2019-12-05T05:10:41.375 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:41.349+0000 7fbc65b47700 1 -- 172.21.15.42:0/1740788005 shutdown_connections 2019-12-05T05:10:41.375 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:41.349+0000 7fbc65b47700 1 --2- 172.21.15.42:0/1740788005 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbc600a69f0 0x7fbc600a2840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:10:41.375 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:41.349+0000 7fbc65b47700 1 --2- 172.21.15.42:0/1740788005 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbc600a8030 0x7fbc600652f0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:10:41.375 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:41.349+0000 7fbc65b47700 1 -- 172.21.15.42:0/1740788005 shutdown_connections 2019-12-05T05:10:41.375 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:41.349+0000 7fbc65b47700 1 -- 172.21.15.42:0/1740788005 wait complete. 2019-12-05T05:10:41.376 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:41.349+0000 7fbc65b47700 1 -- 172.21.15.42:0/1740788005 >> 172.21.15.42:0/1740788005 conn(0x7fbc60059570 msgr2=0x7fbc600599a0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:10:41.376 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:41.349+0000 7fbc65b47700 1 Processor -- start 2019-12-05T05:10:41.376 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:41.349+0000 7fbc65b47700 1 -- start start 2019-12-05T05:10:41.376 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:41.353+0000 7fbc65b47700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fbc600a5490 0x7fbc6007a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:10:41.377 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:41.353+0000 7fbc65b47700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fbc60139120 0x7fbc6013b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:10:41.377 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:41.353+0000 7fbc65b47700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc600a0e70 con 0x7fbc6007bbe0 2019-12-05T05:10:41.377 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:41.353+0000 7fbc65b47700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc6007ddb0 con 0x7fbc60139120 2019-12-05T05:10:41.377 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:41.353+0000 7fbc65b47700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc6007e050 con 0x7fbc600a5490 2019-12-05T05:10:41.377 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:41.353+0000 7fbc5ffff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fbc6007bbe0 0x7fbc600a4c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:57378/0 (socket says 172.21.15.42:57378) 2019-12-05T05:10:41.377 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:41.353+0000 7fbc5f7fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fbc600a5490 0x7fbc6007a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:10:41.378 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:41.353+0000 7fbc5ffff700 1 -- 172.21.15.42:0/1745473867 learned_addr learned my addr 172.21.15.42:0/1745473867 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:10:41.378 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:41.353+0000 7fbc648e5700 1 --2- 172.21.15.42:0/1745473867 >> v2:172.21.15.40:3300/0 conn(0x7fbc60139120 0x7fbc6013b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:10:41.378 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:41.353+0000 7fbc5f7fe700 1 -- 172.21.15.42:0/1745473867 >> v2:172.21.15.40:3300/0 conn(0x7fbc60139120 msgr2=0x7fbc6013b560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:10:41.378 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:41.353+0000 7fbc5f7fe700 1 --2- 172.21.15.42:0/1745473867 >> v2:172.21.15.40:3300/0 conn(0x7fbc60139120 0x7fbc6013b560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:10:41.378 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:41.353+0000 7fbc5f7fe700 1 -- 172.21.15.42:0/1745473867 >> v1:172.21.15.42:6789/0 conn(0x7fbc6007bbe0 legacy=0x7fbc600a4c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:10:41.378 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:41.353+0000 7fbc5f7fe700 1 -- 172.21.15.42:0/1745473867 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc6007daa0 con 0x7fbc600a5490 2019-12-05T05:10:41.379 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:41.353+0000 7fbc5f7fe700 1 --2- 172.21.15.42:0/1745473867 >> v2:172.21.15.42:3300/0 conn(0x7fbc600a5490 0x7fbc6007a120 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fbc600a7630 tx=0x7fbc54011f50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:41.380 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:41.353+0000 7fbc5f7fe700 1 --2- 172.21.15.42:0/1745473867 >> v2:172.21.15.42:3300/0 conn(0x7fbc600a5490 0x7fbc6007a120 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fbc600a7630 tx=0x7fbc54011f50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:41.380 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:41.353+0000 7fbc5f7fe700 1 --2- 172.21.15.42:0/1745473867 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbc600a5490 0x7fbc6007a120 secure :-1 s=READY pgs=405 cs=0 l=1 rx=0x7fbc600a7630 tx=0x7fbc54011f50).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:10:41.380 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:41.353+0000 7fbc5f7fe700 1 --2- 172.21.15.42:0/1745473867 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbc600a5490 0x7fbc6007a120 secure :-1 s=THROTTLE_DONE pgs=405 cs=0 l=1 rx=0x7fbc600a7630 tx=0x7fbc54011f50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:41.380 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:41.353+0000 7fbc5cff9700 1 -- 172.21.15.42:0/1745473867 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbc54007340 con 0x7fbc600a5490 2019-12-05T05:10:41.381 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:41.353+0000 7fbc5f7fe700 1 --2- 172.21.15.42:0/1745473867 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbc600a5490 0x7fbc6007a120 secure :-1 s=THROTTLE_DONE pgs=405 cs=0 l=1 rx=0x7fbc600a7630 tx=0x7fbc54011f50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:41.381 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:41.353+0000 7fbc5f7fe700 1 --2- 172.21.15.42:0/1745473867 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbc600a5490 0x7fbc6007a120 secure :-1 s=THROTTLE_DONE pgs=405 cs=0 l=1 rx=0x7fbc600a7630 tx=0x7fbc54011f50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:41.381 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:41.353+0000 7fbc5cff9700 1 -- 172.21.15.42:0/1745473867 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc540115b0 con 0x7fbc600a5490 2019-12-05T05:10:41.381 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:41.353+0000 7fbc65b47700 1 -- 172.21.15.42:0/1745473867 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc6007c010 con 0x7fbc600a5490 2019-12-05T05:10:41.382 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:41.353+0000 7fbc5cff9700 1 -- 172.21.15.42:0/1745473867 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbc54005540 con 0x7fbc600a5490 2019-12-05T05:10:41.382 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:41.353+0000 7fbc65b47700 1 -- 172.21.15.42:0/1745473867 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc6007a670 con 0x7fbc600a5490 2019-12-05T05:10:41.383 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:41.357+0000 7fbc5f7fe700 1 --2- 172.21.15.42:0/1745473867 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbc600a5490 0x7fbc6007a120 secure :-1 s=THROTTLE_DONE pgs=405 cs=0 l=1 rx=0x7fbc600a7630 tx=0x7fbc54011f50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:41.383 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:41.357+0000 7fbc5cff9700 1 -- 172.21.15.42:0/1745473867 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fbc54005850 con 0x7fbc600a5490 2019-12-05T05:10:41.383 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:41.357+0000 7fbc5f7fe700 1 --2- 172.21.15.42:0/1745473867 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbc600a5490 0x7fbc6007a120 secure :-1 s=THROTTLE_DONE pgs=405 cs=0 l=1 rx=0x7fbc600a7630 tx=0x7fbc54011f50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:41.383 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:41.357+0000 7fbc5cff9700 1 --2- 172.21.15.42:0/1745473867 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbc4003ae00 0x7fbc4003d310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:10:41.383 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:41.357+0000 7fbc5cff9700 1 -- 172.21.15.42:0/1745473867 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6375+0+0 (secure 0 0 0) 0x7fbc5404b280 con 0x7fbc600a5490 2019-12-05T05:10:41.384 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:41.357+0000 7fbc5ffff700 1 --2- 172.21.15.42:0/1745473867 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbc4003ae00 0x7fbc4003d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:10:41.384 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:41.357+0000 7fbc5ffff700 1 --2- 172.21.15.42:0/1745473867 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbc4003ae00 0x7fbc4003d310 crc :-1 s=READY pgs=182 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:10:41.384 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:41.357+0000 7fbc65b47700 1 -- 172.21.15.42:0/1745473867 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc6007e4f0 con 0x7fbc600a5490 2019-12-05T05:10:41.385 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:41.365+0000 7fbc5f7fe700 1 --2- 172.21.15.42:0/1745473867 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbc600a5490 0x7fbc6007a120 secure :-1 s=THROTTLE_DONE pgs=405 cs=0 l=1 rx=0x7fbc600a7630 tx=0x7fbc54011f50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:41.385 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:41.365+0000 7fbc5cff9700 1 -- 172.21.15.42:0/1745473867 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fbc54032300 con 0x7fbc600a5490 2019-12-05T05:10:41.536 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:41.533+0000 7fbc65b47700 1 -- 172.21.15.42:0/1745473867 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"var": "allow_new_snaps", "prefix": "fs set", "fs_name": "cephfs", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fbc6005d2c0 con 0x7fbc600a5490 2019-12-05T05:10:41.805 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:41.793+0000 7fbc5f7fe700 1 --2- 172.21.15.42:0/1745473867 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbc600a5490 0x7fbc6007a120 secure :-1 s=THROTTLE_DONE pgs=405 cs=0 l=1 rx=0x7fbc600a7630 tx=0x7fbc54011f50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:41.805 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:41.793+0000 7fbc5cff9700 1 -- 172.21.15.42:0/1745473867 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"var": "allow_new_snaps", "prefix": "fs set", "fs_name": "cephfs", "val": "true", "yes_i_really_mean_it": true}]=0 enabled new snapshots v42) v1 ==== 167+0+0 (secure 0 0 0) 0x7fbc54032560 con 0x7fbc600a5490 2019-12-05T05:10:41.925 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:41.921+0000 7fbc65b47700 1 -- 172.21.15.42:0/1745473867 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"var": "allow_new_snaps", "prefix": "fs set", "fs_name": "cephfs", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fbc6005ba70 con 0x7fbc600a5490 2019-12-05T05:10:42.802 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:42.797+0000 7fbc5f7fe700 1 --2- 172.21.15.42:0/1745473867 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbc600a5490 0x7fbc6007a120 secure :-1 s=THROTTLE_DONE pgs=405 cs=0 l=1 rx=0x7fbc600a7630 tx=0x7fbc54011f50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:42.802 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:42.797+0000 7fbc5cff9700 1 -- 172.21.15.42:0/1745473867 <== mon.1 v2:172.21.15.42:3300/0 8 ==== mon_command_ack([{"var": "allow_new_snaps", "prefix": "fs set", "fs_name": "cephfs", "val": "true", "yes_i_really_mean_it": true}]=0 enabled new snapshots v43) v1 ==== 167+0+0 (secure 0 0 0) 0x7fbc54033110 con 0x7fbc600a5490 2019-12-05T05:10:42.841 INFO:tasks.workunit.client.0.smithi042.stderr:enabled new snapshots 2019-12-05T05:10:42.845 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:42.841+0000 7fbc65b47700 1 -- 172.21.15.42:0/1745473867 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbc4003ae00 msgr2=0x7fbc4003d310 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:10:42.846 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:42.841+0000 7fbc65b47700 1 --2- 172.21.15.42:0/1745473867 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbc4003ae00 0x7fbc4003d310 crc :-1 s=READY pgs=182 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:10:42.846 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:42.841+0000 7fbc65b47700 1 -- 172.21.15.42:0/1745473867 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbc600a5490 msgr2=0x7fbc6007a120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:10:42.846 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:42.841+0000 7fbc65b47700 1 --2- 172.21.15.42:0/1745473867 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbc600a5490 0x7fbc6007a120 secure :-1 s=READY pgs=405 cs=0 l=1 rx=0x7fbc600a7630 tx=0x7fbc54011f50).stop 2019-12-05T05:10:42.846 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:42.841+0000 7fbc65b47700 1 -- 172.21.15.42:0/1745473867 shutdown_connections 2019-12-05T05:10:42.846 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:42.841+0000 7fbc65b47700 1 --2- 172.21.15.42:0/1745473867 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbc4003ae00 0x7fbc4003d310 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:10:42.846 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:42.841+0000 7fbc65b47700 1 --2- 172.21.15.42:0/1745473867 >> v2:172.21.15.40:3300/0 conn(0x7fbc60139120 0x7fbc6013b560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:10:42.847 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:42.841+0000 7fbc65b47700 1 --2- 172.21.15.42:0/1745473867 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbc600a5490 0x7fbc6007a120 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:10:42.847 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:42.841+0000 7fbc65b47700 1 -- 172.21.15.42:0/1745473867 shutdown_connections 2019-12-05T05:10:42.847 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:42.841+0000 7fbc65b47700 1 -- 172.21.15.42:0/1745473867 wait complete. 2019-12-05T05:10:42.847 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:42.841+0000 7fbc65b47700 1 -- 172.21.15.42:0/1745473867 >> 172.21.15.42:0/1745473867 conn(0x7fbc60144db0 msgr2=0x7fbc6009c8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:10:42.857 INFO:tasks.workunit.client.0.smithi042.stderr:+ touch foo 2019-12-05T05:10:42.864 INFO:tasks.workunit.client.0.smithi042.stderr:+ chmod +x foo 2019-12-05T05:10:42.865 INFO:tasks.workunit.client.0.smithi042.stderr:+ mkdir .snap/s 2019-12-05T05:10:42.898 INFO:tasks.workunit.client.0.smithi042.stderr:+ find .snap/s/foo -executable 2019-12-05T05:10:42.898 INFO:tasks.workunit.client.0.smithi042.stderr:+ grep foo 2019-12-05T05:10:42.909 INFO:tasks.workunit.client.0.smithi042.stdout:.snap/s/foo 2019-12-05T05:10:42.909 INFO:tasks.workunit.client.0.smithi042.stderr:+ rmdir .snap/s 2019-12-05T05:10:42.922 INFO:tasks.workunit.client.0.smithi042.stderr:+ rm foo 2019-12-05T05:10:42.926 INFO:tasks.workunit.client.0.smithi042.stderr:+ echo OK 2019-12-05T05:10:42.927 INFO:tasks.workunit.client.0.smithi042.stdout:OK 2019-12-05T05:10:42.927 INFO:teuthology.orchestra.run:Running command with timeout 3600 2019-12-05T05:10:42.927 INFO:teuthology.orchestra.run.smithi042:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2019-12-05T05:10:43.033 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-capwb.sh... 2019-12-05T05:10:43.033 INFO:teuthology.orchestra.run.smithi042:workunit test fs/snaps/snaptest-capwb.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c5290805033483ba0cec165c06bde07dae5f19ec TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-capwb.sh 2019-12-05T05:10:43.088 INFO:tasks.workunit.client.0.smithi042.stderr:+ set -e 2019-12-05T05:10:43.088 INFO:tasks.workunit.client.0.smithi042.stderr:+ mkdir foo 2019-12-05T05:10:43.093 INFO:tasks.workunit.client.0.smithi042.stderr:+ ceph fs set cephfs allow_new_snaps true --yes-i-really-mean-it 2019-12-05T05:10:43.206 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:43.197+0000 7fbe86293700 1 Processor -- start 2019-12-05T05:10:43.207 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:43.197+0000 7fbe86293700 1 -- start start 2019-12-05T05:10:43.207 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:43.197+0000 7fbe86293700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fbe800a8030 0x7fbe800652f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:10:43.207 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:43.197+0000 7fbe86293700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe80065840 con 0x7fbe800a2840 2019-12-05T05:10:43.207 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:43.197+0000 7fbe86293700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe800a7b60 con 0x7fbe800a69f0 2019-12-05T05:10:43.207 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:43.197+0000 7fbe86293700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe80065b20 con 0x7fbe800a8030 2019-12-05T05:10:43.207 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:43.197+0000 7fbe84830700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fbe800a69f0 0x7fbe80144fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:57396/0 (socket says 172.21.15.42:57396) 2019-12-05T05:10:43.207 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:43.197+0000 7fbe84830700 1 -- 172.21.15.42:0/2400091361 learned_addr learned my addr 172.21.15.42:0/2400091361 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:10:43.208 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:43.197+0000 7fbe85031700 1 --2- 172.21.15.42:0/2400091361 >> v2:172.21.15.40:3300/0 conn(0x7fbe800a8030 0x7fbe800652f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:10:43.208 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:43.197+0000 7fbe777fe700 1 -- 172.21.15.42:0/2400091361 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2568786036 0 0) 0x7fbe800a7b60 con 0x7fbe800a69f0 2019-12-05T05:10:43.208 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:43.197+0000 7fbe777fe700 1 -- 172.21.15.42:0/2400091361 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe680030b0 con 0x7fbe800a69f0 2019-12-05T05:10:43.208 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:43.201+0000 7fbe85031700 1 -- 172.21.15.42:0/2400091361 >> v1:172.21.15.42:6789/0 conn(0x7fbe800a69f0 legacy=0x7fbe80144fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:10:43.208 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:43.201+0000 7fbe85031700 1 -- 172.21.15.42:0/2400091361 >> v1:172.21.15.40:6789/0 conn(0x7fbe800a2840 legacy=0x7fbe8014d1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:10:43.208 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:43.201+0000 7fbe85031700 1 -- 172.21.15.42:0/2400091361 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe800a2c70 con 0x7fbe800a8030 2019-12-05T05:10:43.208 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:43.201+0000 7fbe85031700 1 --2- 172.21.15.42:0/2400091361 >> v2:172.21.15.40:3300/0 conn(0x7fbe800a8030 0x7fbe800652f0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fbe800a2020 tx=0x7fbe7800ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:43.209 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:43.201+0000 7fbe85031700 1 --2- 172.21.15.42:0/2400091361 >> v2:172.21.15.40:3300/0 conn(0x7fbe800a8030 0x7fbe800652f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fbe800a2020 tx=0x7fbe7800ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:43.209 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:43.201+0000 7fbe85031700 1 --2- 172.21.15.42:0/2400091361 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbe800a8030 0x7fbe800652f0 secure :-1 s=READY pgs=320 cs=0 l=1 rx=0x7fbe800a2020 tx=0x7fbe7800ff70).ready entity=mon.0 client_cookie=ef26119e9c08fac2 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:10:43.209 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:43.201+0000 7fbe85031700 1 --2- 172.21.15.42:0/2400091361 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbe800a8030 0x7fbe800652f0 secure :-1 s=THROTTLE_DONE pgs=320 cs=0 l=1 rx=0x7fbe800a2020 tx=0x7fbe7800ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:43.209 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:43.201+0000 7fbe777fe700 1 -- 172.21.15.42:0/2400091361 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbe78011990 con 0x7fbe800a8030 2019-12-05T05:10:43.209 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:43.201+0000 7fbe85031700 1 --2- 172.21.15.42:0/2400091361 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbe800a8030 0x7fbe800652f0 secure :-1 s=THROTTLE_DONE pgs=320 cs=0 l=1 rx=0x7fbe800a2020 tx=0x7fbe7800ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:43.209 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:43.201+0000 7fbe85031700 1 --2- 172.21.15.42:0/2400091361 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbe800a8030 0x7fbe800652f0 secure :-1 s=THROTTLE_DONE pgs=320 cs=0 l=1 rx=0x7fbe800a2020 tx=0x7fbe7800ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:43.209 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:43.201+0000 7fbe777fe700 1 -- 172.21.15.42:0/2400091361 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe780115b0 con 0x7fbe800a8030 2019-12-05T05:10:43.210 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:43.201+0000 7fbe777fe700 1 -- 172.21.15.42:0/2400091361 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbe7800a160 con 0x7fbe800a8030 2019-12-05T05:10:43.210 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:43.201+0000 7fbe86293700 1 -- 172.21.15.42:0/2400091361 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbe800a8030 msgr2=0x7fbe800652f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:10:43.210 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:43.201+0000 7fbe86293700 1 --2- 172.21.15.42:0/2400091361 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbe800a8030 0x7fbe800652f0 secure :-1 s=READY pgs=320 cs=0 l=1 rx=0x7fbe800a2020 tx=0x7fbe7800ff70).stop 2019-12-05T05:10:43.210 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:43.201+0000 7fbe86293700 1 -- 172.21.15.42:0/2400091361 shutdown_connections 2019-12-05T05:10:43.210 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:43.201+0000 7fbe86293700 1 --2- 172.21.15.42:0/2400091361 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbe800a8030 0x7fbe800652f0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:10:43.210 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:43.201+0000 7fbe86293700 1 -- 172.21.15.42:0/2400091361 shutdown_connections 2019-12-05T05:10:43.211 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:43.201+0000 7fbe86293700 1 -- 172.21.15.42:0/2400091361 wait complete. 2019-12-05T05:10:43.211 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:43.201+0000 7fbe86293700 1 -- 172.21.15.42:0/2400091361 >> 172.21.15.42:0/2400091361 conn(0x7fbe80059570 msgr2=0x7fbe800599a0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:10:43.215 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:43.201+0000 7fbe86293700 1 Processor -- start 2019-12-05T05:10:43.215 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:43.201+0000 7fbe86293700 1 -- start start 2019-12-05T05:10:43.216 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:43.205+0000 7fbe86293700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fbe8012bc40 0x7fbe8014edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:10:43.216 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:43.205+0000 7fbe86293700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fbe8014f300 0x7fbe8014f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:10:43.216 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:43.205+0000 7fbe86293700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe8009f700 con 0x7fbe800a0860 2019-12-05T05:10:43.216 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:43.205+0000 7fbe86293700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe8009ef20 con 0x7fbe8014f300 2019-12-05T05:10:43.216 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:43.205+0000 7fbe86293700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe8009f1c0 con 0x7fbe8012bc40 2019-12-05T05:10:43.216 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:43.205+0000 7fbe77fff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fbe8012bc40 0x7fbe8014edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:10:43.217 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:43.205+0000 7fbe85031700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fbe8014f300 0x7fbe8014f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:10:43.217 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:43.205+0000 7fbe77fff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fbe8012bc40 0x7fbe8014edb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:57366/0 (socket says 172.21.15.42:57366) 2019-12-05T05:10:43.217 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:43.205+0000 7fbe77fff700 1 -- 172.21.15.42:0/3992397244 learned_addr learned my addr 172.21.15.42:0/3992397244 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:10:43.217 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:43.205+0000 7fbe85031700 1 -- 172.21.15.42:0/3992397244 >> v2:172.21.15.42:3300/0 conn(0x7fbe8012bc40 msgr2=0x7fbe8014edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:10:43.217 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:43.205+0000 7fbe85031700 1 --2- 172.21.15.42:0/3992397244 >> v2:172.21.15.42:3300/0 conn(0x7fbe8012bc40 0x7fbe8014edb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:10:43.217 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:43.205+0000 7fbe85031700 1 -- 172.21.15.42:0/3992397244 >> v1:172.21.15.42:6789/0 conn(0x7fbe800a0860 legacy=0x7fbe8012b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:10:43.217 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:43.205+0000 7fbe85031700 1 -- 172.21.15.42:0/3992397244 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe800a6390 con 0x7fbe8014f300 2019-12-05T05:10:43.218 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:43.205+0000 7fbe77fff700 1 --2- 172.21.15.42:0/3992397244 >> v2:172.21.15.42:3300/0 conn(0x7fbe8012bc40 0x7fbe8014edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:10:43.218 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:43.205+0000 7fbe85031700 1 --2- 172.21.15.42:0/3992397244 >> v2:172.21.15.40:3300/0 conn(0x7fbe8014f300 0x7fbe8014f730 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fbe8009f0e0 tx=0x7fbe78007dc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:43.218 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:43.205+0000 7fbe85031700 1 --2- 172.21.15.42:0/3992397244 >> v2:172.21.15.40:3300/0 conn(0x7fbe8014f300 0x7fbe8014f730 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fbe8009f0e0 tx=0x7fbe78007dc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:43.218 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:43.205+0000 7fbe85031700 1 --2- 172.21.15.42:0/3992397244 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbe8014f300 0x7fbe8014f730 secure :-1 s=READY pgs=321 cs=0 l=1 rx=0x7fbe8009f0e0 tx=0x7fbe78007dc0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:10:43.218 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:43.205+0000 7fbe85031700 1 --2- 172.21.15.42:0/3992397244 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbe8014f300 0x7fbe8014f730 secure :-1 s=THROTTLE_DONE pgs=321 cs=0 l=1 rx=0x7fbe8009f0e0 tx=0x7fbe78007dc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:43.218 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:43.205+0000 7fbe85031700 1 --2- 172.21.15.42:0/3992397244 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbe8014f300 0x7fbe8014f730 secure :-1 s=THROTTLE_DONE pgs=321 cs=0 l=1 rx=0x7fbe8009f0e0 tx=0x7fbe78007dc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:43.218 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:43.205+0000 7fbe757fa700 1 -- 172.21.15.42:0/3992397244 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbe7800f760 con 0x7fbe8014f300 2019-12-05T05:10:43.219 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:43.205+0000 7fbe85031700 1 --2- 172.21.15.42:0/3992397244 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbe8014f300 0x7fbe8014f730 secure :-1 s=THROTTLE_DONE pgs=321 cs=0 l=1 rx=0x7fbe8009f0e0 tx=0x7fbe78007dc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:43.219 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:43.205+0000 7fbe757fa700 1 -- 172.21.15.42:0/3992397244 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe78011b90 con 0x7fbe8014f300 2019-12-05T05:10:43.219 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:43.205+0000 7fbe757fa700 1 -- 172.21.15.42:0/3992397244 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbe78005ab0 con 0x7fbe8014f300 2019-12-05T05:10:43.219 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:43.205+0000 7fbe86293700 1 -- 172.21.15.42:0/3992397244 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe8009db40 con 0x7fbe8014f300 2019-12-05T05:10:43.219 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:43.205+0000 7fbe86293700 1 -- 172.21.15.42:0/3992397244 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe8009eb00 con 0x7fbe8014f300 2019-12-05T05:10:43.219 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:43.205+0000 7fbe85031700 1 --2- 172.21.15.42:0/3992397244 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbe8014f300 0x7fbe8014f730 secure :-1 s=THROTTLE_DONE pgs=321 cs=0 l=1 rx=0x7fbe8009f0e0 tx=0x7fbe78007dc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:43.219 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:43.209+0000 7fbe757fa700 1 -- 172.21.15.42:0/3992397244 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fbe78007150 con 0x7fbe8014f300 2019-12-05T05:10:43.220 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:43.209+0000 7fbe85031700 1 --2- 172.21.15.42:0/3992397244 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbe8014f300 0x7fbe8014f730 secure :-1 s=THROTTLE_DONE pgs=321 cs=0 l=1 rx=0x7fbe8009f0e0 tx=0x7fbe78007dc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:43.220 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:43.209+0000 7fbe757fa700 1 --2- 172.21.15.42:0/3992397244 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbe6003acf0 0x7fbe6003d200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:10:43.220 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:43.209+0000 7fbe757fa700 1 -- 172.21.15.42:0/3992397244 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6375+0+0 (secure 0 0 0) 0x7fbe7804b6b0 con 0x7fbe8014f300 2019-12-05T05:10:43.220 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:43.209+0000 7fbe84830700 1 --2- 172.21.15.42:0/3992397244 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbe6003acf0 0x7fbe6003d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:10:43.220 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:43.209+0000 7fbe84830700 1 --2- 172.21.15.42:0/3992397244 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbe6003acf0 0x7fbe6003d200 crc :-1 s=READY pgs=183 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:10:43.221 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:43.213+0000 7fbe86293700 1 -- 172.21.15.42:0/3992397244 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe8005d2c0 con 0x7fbe8014f300 2019-12-05T05:10:43.226 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:43.221+0000 7fbe85031700 1 --2- 172.21.15.42:0/3992397244 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbe8014f300 0x7fbe8014f730 secure :-1 s=THROTTLE_DONE pgs=321 cs=0 l=1 rx=0x7fbe8009f0e0 tx=0x7fbe78007dc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:43.226 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:43.221+0000 7fbe757fa700 1 -- 172.21.15.42:0/3992397244 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fbe78032730 con 0x7fbe8014f300 2019-12-05T05:10:43.395 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:43.393+0000 7fbe86293700 1 -- 172.21.15.42:0/3992397244 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"var": "allow_new_snaps", "prefix": "fs set", "fs_name": "cephfs", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fbe8007ea70 con 0x7fbe8014f300 2019-12-05T05:10:44.298 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:44.293+0000 7fbe85031700 1 --2- 172.21.15.42:0/3992397244 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbe8014f300 0x7fbe8014f730 secure :-1 s=THROTTLE_DONE pgs=321 cs=0 l=1 rx=0x7fbe8009f0e0 tx=0x7fbe78007dc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:44.298 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:44.293+0000 7fbe757fa700 1 -- 172.21.15.42:0/3992397244 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"var": "allow_new_snaps", "prefix": "fs set", "fs_name": "cephfs", "val": "true", "yes_i_really_mean_it": true}]=0 enabled new snapshots v44) v1 ==== 167+0+0 (secure 0 0 0) 0x7fbe78005dc0 con 0x7fbe8014f300 2019-12-05T05:10:44.428 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:44.425+0000 7fbe86293700 1 -- 172.21.15.42:0/3992397244 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"var": "allow_new_snaps", "prefix": "fs set", "fs_name": "cephfs", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fbe8005ba70 con 0x7fbe8014f300 2019-12-05T05:10:45.302 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:45.297+0000 7fbe85031700 1 --2- 172.21.15.42:0/3992397244 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbe8014f300 0x7fbe8014f730 secure :-1 s=THROTTLE_DONE pgs=321 cs=0 l=1 rx=0x7fbe8009f0e0 tx=0x7fbe78007dc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:45.302 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:45.297+0000 7fbe757fa700 1 -- 172.21.15.42:0/3992397244 <== mon.0 v2:172.21.15.40:3300/0 8 ==== mon_command_ack([{"var": "allow_new_snaps", "prefix": "fs set", "fs_name": "cephfs", "val": "true", "yes_i_really_mean_it": true}]=0 enabled new snapshots v45) v1 ==== 167+0+0 (secure 0 0 0) 0x7fbe78032990 con 0x7fbe8014f300 2019-12-05T05:10:45.344 INFO:tasks.workunit.client.0.smithi042.stderr:enabled new snapshots 2019-12-05T05:10:45.350 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:45.345+0000 7fbe86293700 1 -- 172.21.15.42:0/3992397244 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbe6003acf0 msgr2=0x7fbe6003d200 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:10:45.350 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:45.345+0000 7fbe86293700 1 --2- 172.21.15.42:0/3992397244 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbe6003acf0 0x7fbe6003d200 crc :-1 s=READY pgs=183 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:10:45.351 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:45.345+0000 7fbe86293700 1 -- 172.21.15.42:0/3992397244 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbe8014f300 msgr2=0x7fbe8014f730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:10:45.351 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:45.345+0000 7fbe86293700 1 --2- 172.21.15.42:0/3992397244 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbe8014f300 0x7fbe8014f730 secure :-1 s=READY pgs=321 cs=0 l=1 rx=0x7fbe8009f0e0 tx=0x7fbe78007dc0).stop 2019-12-05T05:10:45.351 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:45.345+0000 7fbe86293700 1 -- 172.21.15.42:0/3992397244 shutdown_connections 2019-12-05T05:10:45.352 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:45.345+0000 7fbe86293700 1 --2- 172.21.15.42:0/3992397244 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbe6003acf0 0x7fbe6003d200 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:10:45.352 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:45.345+0000 7fbe86293700 1 --2- 172.21.15.42:0/3992397244 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbe8014f300 0x7fbe8014f730 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:10:45.352 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:45.345+0000 7fbe86293700 1 --2- 172.21.15.42:0/3992397244 >> v2:172.21.15.42:3300/0 conn(0x7fbe8012bc40 0x7fbe8014edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:10:45.352 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:45.345+0000 7fbe86293700 1 -- 172.21.15.42:0/3992397244 shutdown_connections 2019-12-05T05:10:45.352 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:45.345+0000 7fbe86293700 1 -- 172.21.15.42:0/3992397244 wait complete. 2019-12-05T05:10:45.353 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:45.345+0000 7fbe86293700 1 -- 172.21.15.42:0/3992397244 >> 172.21.15.42:0/3992397244 conn(0x7fbe800a3d90 msgr2=0x7fbe800a1300 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:10:45.367 INFO:tasks.workunit.client.0.smithi042.stderr:+ echo x 2019-12-05T05:10:45.367 INFO:tasks.workunit.client.0.smithi042.stderr:+ sync 2019-12-05T05:10:45.668 INFO:tasks.workunit.client.0.smithi042.stderr:+ mkdir foo/.snap/ss 2019-12-05T05:10:45.690 INFO:tasks.workunit.client.0.smithi042.stderr:+ ln foo/x foo/xx 2019-12-05T05:10:45.702 INFO:tasks.workunit.client.0.smithi042.stderr:+ cat foo/.snap/ss/x 2019-12-05T05:10:45.714 INFO:tasks.workunit.client.0.smithi042.stdout:x 2019-12-05T05:10:45.714 INFO:tasks.workunit.client.0.smithi042.stderr:+ rmdir foo/.snap/ss 2019-12-05T05:10:45.734 INFO:tasks.workunit.client.0.smithi042.stderr:+ echo a 2019-12-05T05:10:45.738 INFO:tasks.workunit.client.0.smithi042.stderr:+ echo b 2019-12-05T05:10:45.739 INFO:tasks.workunit.client.0.smithi042.stderr:+ mkdir foo/.snap/s 2019-12-05T05:10:45.755 INFO:tasks.workunit.client.0.smithi042.stderr:+ cat foo/.snap/s/a 2019-12-05T05:10:45.767 INFO:tasks.workunit.client.0.smithi042.stderr:+ r=a 2019-12-05T05:10:45.767 INFO:tasks.workunit.client.0.smithi042.stderr:+ [ -z a ] 2019-12-05T05:10:45.767 INFO:tasks.workunit.client.0.smithi042.stderr:+ ln foo/b foo/b2 2019-12-05T05:10:45.773 INFO:tasks.workunit.client.0.smithi042.stderr:+ cat foo/.snap/s/b 2019-12-05T05:10:45.796 INFO:tasks.workunit.client.0.smithi042.stdout:b 2019-12-05T05:10:45.796 INFO:tasks.workunit.client.0.smithi042.stdout:this used to hang: 2019-12-05T05:10:45.796 INFO:tasks.workunit.client.0.smithi042.stdout:oh, it didn't hang! good job. 2019-12-05T05:10:45.796 INFO:tasks.workunit.client.0.smithi042.stdout:b 2019-12-05T05:10:45.797 INFO:tasks.workunit.client.0.smithi042.stdout:more 2019-12-05T05:10:45.797 INFO:tasks.workunit.client.0.smithi042.stderr:+ echo this used to hang: 2019-12-05T05:10:45.797 INFO:tasks.workunit.client.0.smithi042.stderr:+ echo more 2019-12-05T05:10:45.797 INFO:tasks.workunit.client.0.smithi042.stderr:+ echo oh, it didn't hang! good job. 2019-12-05T05:10:45.797 INFO:tasks.workunit.client.0.smithi042.stderr:+ cat foo/b 2019-12-05T05:10:45.797 INFO:tasks.workunit.client.0.smithi042.stderr:+ rmdir foo/.snap/s 2019-12-05T05:10:45.806 INFO:tasks.workunit.client.0.smithi042.stderr:+ rm -r foo 2019-12-05T05:10:45.922 INFO:tasks.workunit.client.0.smithi042.stderr:+ echo OK 2019-12-05T05:10:45.923 INFO:tasks.workunit.client.0.smithi042.stdout:OK 2019-12-05T05:10:45.923 INFO:teuthology.orchestra.run:Running command with timeout 3600 2019-12-05T05:10:45.923 INFO:teuthology.orchestra.run.smithi042:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2019-12-05T05:10:46.035 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-dir-rename.sh... 2019-12-05T05:10:46.035 INFO:teuthology.orchestra.run.smithi042:workunit test fs/snaps/snaptest-dir-rename.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c5290805033483ba0cec165c06bde07dae5f19ec TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-dir-rename.sh 2019-12-05T05:10:46.133 INFO:tasks.workunit.client.0.smithi042.stderr:+ set -e 2019-12-05T05:10:46.134 INFO:tasks.workunit.client.0.smithi042.stderr:+ ceph fs set cephfs allow_new_snaps true --yes-i-really-mean-it 2019-12-05T05:10:46.237 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:46.229+0000 7efc54dd4700 1 Processor -- start 2019-12-05T05:10:46.238 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:46.229+0000 7efc54dd4700 1 -- start start 2019-12-05T05:10:46.238 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:46.229+0000 7efc54dd4700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7efc500a2840 0x7efc500a8030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:10:46.238 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:46.229+0000 7efc54dd4700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efc500652f0 0x7efc5014f200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:10:46.238 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:46.229+0000 7efc54dd4700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc500a7b60 con 0x7efc500a69f0 2019-12-05T05:10:46.238 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:46.229+0000 7efc54dd4700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc50065b20 con 0x7efc500a2840 2019-12-05T05:10:46.238 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:46.229+0000 7efc54dd4700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc500a25a0 con 0x7efc500652f0 2019-12-05T05:10:46.239 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:46.229+0000 7efc4ed9d700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7efc500a69f0 0x7efc50144fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:38624/0 (socket says 172.21.15.42:38624) 2019-12-05T05:10:46.239 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:46.229+0000 7efc4e59c700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7efc500a2840 0x7efc500a8030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:10:46.239 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:46.229+0000 7efc4ed9d700 1 -- 172.21.15.42:0/3119388423 learned_addr learned my addr 172.21.15.42:0/3119388423 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:10:46.239 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:46.229+0000 7efc4f59e700 1 --2- 172.21.15.42:0/3119388423 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efc500652f0 0x7efc5014f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:10:46.239 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:46.229+0000 7efc4e59c700 1 -- 172.21.15.42:0/3119388423 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efc500652f0 msgr2=0x7efc5014f200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:10:46.239 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:46.229+0000 7efc4e59c700 1 --2- 172.21.15.42:0/3119388423 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efc500652f0 0x7efc5014f200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:10:46.239 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:46.229+0000 7efc4e59c700 1 -- 172.21.15.42:0/3119388423 >> v1:172.21.15.40:6789/0 conn(0x7efc500a69f0 legacy=0x7efc50144fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:10:46.240 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:46.229+0000 7efc4dd9b700 1 -- 172.21.15.42:0/3119388423 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 344241811 0 0) 0x7efc500a7b60 con 0x7efc500a69f0 2019-12-05T05:10:46.240 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:46.229+0000 7efc4e59c700 1 -- 172.21.15.42:0/3119388423 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc500a2c70 con 0x7efc500a2840 2019-12-05T05:10:46.241 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:46.229+0000 7efc4e59c700 1 --2- 172.21.15.42:0/3119388423 >> v2:172.21.15.40:3300/0 conn(0x7efc500a2840 0x7efc500a8030 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7efc500a7630 tx=0x7efc3800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:46.241 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:46.229+0000 7efc4e59c700 1 --2- 172.21.15.42:0/3119388423 >> v2:172.21.15.40:3300/0 conn(0x7efc500a2840 0x7efc500a8030 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7efc500a7630 tx=0x7efc3800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:46.241 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:46.229+0000 7efc4e59c700 1 --2- 172.21.15.42:0/3119388423 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efc500a2840 0x7efc500a8030 secure :-1 s=READY pgs=323 cs=0 l=1 rx=0x7efc500a7630 tx=0x7efc3800fc70).ready entity=mon.0 client_cookie=814cef653ec0325e server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:10:46.241 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:46.229+0000 7efc4e59c700 1 --2- 172.21.15.42:0/3119388423 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efc500a2840 0x7efc500a8030 secure :-1 s=THROTTLE_DONE pgs=323 cs=0 l=1 rx=0x7efc500a7630 tx=0x7efc3800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:46.242 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:46.229+0000 7efc4dd9b700 1 -- 172.21.15.42:0/3119388423 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efc380112f0 con 0x7efc500a2840 2019-12-05T05:10:46.242 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:46.229+0000 7efc4e59c700 1 --2- 172.21.15.42:0/3119388423 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efc500a2840 0x7efc500a8030 secure :-1 s=THROTTLE_DONE pgs=323 cs=0 l=1 rx=0x7efc500a7630 tx=0x7efc3800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:46.242 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:46.229+0000 7efc4dd9b700 1 -- 172.21.15.42:0/3119388423 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc38008ad0 con 0x7efc500a2840 2019-12-05T05:10:46.243 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:46.229+0000 7efc4e59c700 1 --2- 172.21.15.42:0/3119388423 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efc500a2840 0x7efc500a8030 secure :-1 s=THROTTLE_DONE pgs=323 cs=0 l=1 rx=0x7efc500a7630 tx=0x7efc3800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:46.243 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:46.229+0000 7efc4dd9b700 1 -- 172.21.15.42:0/3119388423 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efc380118d0 con 0x7efc500a2840 2019-12-05T05:10:46.243 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:46.229+0000 7efc54dd4700 1 -- 172.21.15.42:0/3119388423 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efc500a2840 msgr2=0x7efc500a8030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:10:46.243 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:46.229+0000 7efc54dd4700 1 --2- 172.21.15.42:0/3119388423 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efc500a2840 0x7efc500a8030 secure :-1 s=READY pgs=323 cs=0 l=1 rx=0x7efc500a7630 tx=0x7efc3800fc70).stop 2019-12-05T05:10:46.243 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:46.233+0000 7efc54dd4700 1 -- 172.21.15.42:0/3119388423 shutdown_connections 2019-12-05T05:10:46.243 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:46.233+0000 7efc54dd4700 1 --2- 172.21.15.42:0/3119388423 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efc500652f0 0x7efc5014f200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:10:46.244 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:46.233+0000 7efc54dd4700 1 --2- 172.21.15.42:0/3119388423 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efc500a2840 0x7efc500a8030 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:10:46.244 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:46.233+0000 7efc54dd4700 1 -- 172.21.15.42:0/3119388423 shutdown_connections 2019-12-05T05:10:46.244 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:46.233+0000 7efc54dd4700 1 -- 172.21.15.42:0/3119388423 wait complete. 2019-12-05T05:10:46.244 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:46.233+0000 7efc54dd4700 1 -- 172.21.15.42:0/3119388423 >> 172.21.15.42:0/3119388423 conn(0x7efc50059570 msgr2=0x7efc500599a0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:10:46.248 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:46.237+0000 7efc54dd4700 1 Processor -- start 2019-12-05T05:10:46.248 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:46.237+0000 7efc54dd4700 1 -- start start 2019-12-05T05:10:46.248 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:46.237+0000 7efc54dd4700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efc50063880 0x7efc50064300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:10:46.249 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:46.237+0000 7efc54dd4700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc500632f0 con 0x7efc5007e440 2019-12-05T05:10:46.249 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:46.237+0000 7efc54dd4700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc500a4df0 con 0x7efc5007d0c0 2019-12-05T05:10:46.249 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:46.237+0000 7efc54dd4700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc500a0f30 con 0x7efc50063880 2019-12-05T05:10:46.249 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:46.237+0000 7efc4ed9d700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efc50063880 0x7efc50064300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:10:46.249 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:46.237+0000 7efc4e59c700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7efc5007e440 0x7efc500a2ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:38632/0 (socket says 172.21.15.42:38632) 2019-12-05T05:10:46.249 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:46.237+0000 7efc4e59c700 1 -- 172.21.15.42:0/2363763283 learned_addr learned my addr 172.21.15.42:0/2363763283 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:10:46.249 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:46.237+0000 7efc477fe700 1 -- 172.21.15.42:0/2363763283 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3403107088 0 0) 0x7efc500632f0 con 0x7efc5007e440 2019-12-05T05:10:46.250 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:46.237+0000 7efc477fe700 1 -- 172.21.15.42:0/2363763283 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc340030b0 con 0x7efc5007e440 2019-12-05T05:10:46.250 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:46.237+0000 7efc477fe700 1 -- 172.21.15.42:0/2363763283 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1189883554 0 0) 0x7efc500a4df0 con 0x7efc5007d0c0 2019-12-05T05:10:46.250 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:46.237+0000 7efc477fe700 1 -- 172.21.15.42:0/2363763283 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc500632f0 con 0x7efc5007d0c0 2019-12-05T05:10:46.250 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:46.237+0000 7efc4ed9d700 1 -- 172.21.15.42:0/2363763283 >> v1:172.21.15.42:6789/0 conn(0x7efc5007d0c0 legacy=0x7efc500a36f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:10:46.250 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:46.237+0000 7efc4ed9d700 1 -- 172.21.15.42:0/2363763283 >> v1:172.21.15.40:6789/0 conn(0x7efc5007e440 legacy=0x7efc500a2ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:10:46.250 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:46.237+0000 7efc4ed9d700 1 -- 172.21.15.42:0/2363763283 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc500a5020 con 0x7efc50063880 2019-12-05T05:10:46.250 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:46.237+0000 7efc4ed9d700 1 --2- 172.21.15.42:0/2363763283 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efc50063880 0x7efc50064300 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7efc400043a0 tx=0x7efc40010850).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:46.251 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:46.237+0000 7efc4ed9d700 1 --2- 172.21.15.42:0/2363763283 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efc50063880 0x7efc50064300 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7efc400043a0 tx=0x7efc40010850).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:46.251 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:46.237+0000 7efc4ed9d700 1 --2- 172.21.15.42:0/2363763283 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efc50063880 0x7efc50064300 secure :-1 s=READY pgs=81 cs=0 l=1 rx=0x7efc400043a0 tx=0x7efc40010850).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:10:46.251 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:46.241+0000 7efc4ed9d700 1 --2- 172.21.15.42:0/2363763283 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efc50063880 0x7efc50064300 secure :-1 s=THROTTLE_DONE pgs=81 cs=0 l=1 rx=0x7efc400043a0 tx=0x7efc40010850).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:46.251 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:46.241+0000 7efc477fe700 1 -- 172.21.15.42:0/2363763283 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efc40011580 con 0x7efc50063880 2019-12-05T05:10:46.251 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:46.241+0000 7efc4ed9d700 1 --2- 172.21.15.42:0/2363763283 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efc50063880 0x7efc50064300 secure :-1 s=THROTTLE_DONE pgs=81 cs=0 l=1 rx=0x7efc400043a0 tx=0x7efc40010850).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:46.251 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:46.241+0000 7efc4ed9d700 1 --2- 172.21.15.42:0/2363763283 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efc50063880 0x7efc50064300 secure :-1 s=THROTTLE_DONE pgs=81 cs=0 l=1 rx=0x7efc400043a0 tx=0x7efc40010850).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:46.251 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:46.241+0000 7efc477fe700 1 -- 172.21.15.42:0/2363763283 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc400111a0 con 0x7efc50063880 2019-12-05T05:10:46.252 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:46.241+0000 7efc477fe700 1 -- 172.21.15.42:0/2363763283 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efc40011b60 con 0x7efc50063880 2019-12-05T05:10:46.252 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:46.241+0000 7efc54dd4700 1 -- 172.21.15.42:0/2363763283 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc500a52f0 con 0x7efc50063880 2019-12-05T05:10:46.252 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:46.241+0000 7efc54dd4700 1 -- 172.21.15.42:0/2363763283 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc5011aa40 con 0x7efc50063880 2019-12-05T05:10:46.252 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:46.241+0000 7efc4ed9d700 1 --2- 172.21.15.42:0/2363763283 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efc50063880 0x7efc50064300 secure :-1 s=THROTTLE_DONE pgs=81 cs=0 l=1 rx=0x7efc400043a0 tx=0x7efc40010850).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:46.253 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:46.241+0000 7efc477fe700 1 -- 172.21.15.42:0/2363763283 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7efc4000b330 con 0x7efc50063880 2019-12-05T05:10:46.253 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:46.241+0000 7efc4ed9d700 1 --2- 172.21.15.42:0/2363763283 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efc50063880 0x7efc50064300 secure :-1 s=THROTTLE_DONE pgs=81 cs=0 l=1 rx=0x7efc400043a0 tx=0x7efc40010850).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:46.253 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:46.241+0000 7efc477fe700 1 --2- 172.21.15.42:0/2363763283 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7efc3403b4a0 0x7efc3403d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:10:46.253 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:46.241+0000 7efc477fe700 1 -- 172.21.15.42:0/2363763283 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6431+0+0 (secure 0 0 0) 0x7efc4004b030 con 0x7efc50063880 2019-12-05T05:10:46.253 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:46.241+0000 7efc4e59c700 1 --2- 172.21.15.42:0/2363763283 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7efc3403b4a0 0x7efc3403d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:10:46.253 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:46.241+0000 7efc4e59c700 1 --2- 172.21.15.42:0/2363763283 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7efc3403b4a0 0x7efc3403d9b0 crc :-1 s=READY pgs=184 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:10:46.254 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:46.245+0000 7efc54dd4700 1 -- 172.21.15.42:0/2363763283 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc5005d2c0 con 0x7efc50063880 2019-12-05T05:10:46.255 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:46.249+0000 7efc4ed9d700 1 --2- 172.21.15.42:0/2363763283 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efc50063880 0x7efc50064300 secure :-1 s=THROTTLE_DONE pgs=81 cs=0 l=1 rx=0x7efc400043a0 tx=0x7efc40010850).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:46.255 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:46.249+0000 7efc477fe700 1 -- 172.21.15.42:0/2363763283 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7efc500a52f0 con 0x7efc50063880 2019-12-05T05:10:46.429 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:46.417+0000 7efc54dd4700 1 -- 172.21.15.42:0/2363763283 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"var": "allow_new_snaps", "prefix": "fs set", "fs_name": "cephfs", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7efc5005ba70 con 0x7efc50063880 2019-12-05T05:10:47.328 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:47.321+0000 7efc4ed9d700 1 --2- 172.21.15.42:0/2363763283 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efc50063880 0x7efc50064300 secure :-1 s=THROTTLE_DONE pgs=81 cs=0 l=1 rx=0x7efc400043a0 tx=0x7efc40010850).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:47.328 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:47.321+0000 7efc477fe700 1 -- 172.21.15.42:0/2363763283 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"var": "allow_new_snaps", "prefix": "fs set", "fs_name": "cephfs", "val": "true", "yes_i_really_mean_it": true}]=0 enabled new snapshots v46) v1 ==== 167+0+0 (secure 0 0 0) 0x7efc40032090 con 0x7efc50063880 2019-12-05T05:10:47.455 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:47.449+0000 7efc54dd4700 1 -- 172.21.15.42:0/2363763283 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"var": "allow_new_snaps", "prefix": "fs set", "fs_name": "cephfs", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7efc50069770 con 0x7efc50063880 2019-12-05T05:10:48.342 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:48.337+0000 7efc4ed9d700 1 --2- 172.21.15.42:0/2363763283 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efc50063880 0x7efc50064300 secure :-1 s=THROTTLE_DONE pgs=81 cs=0 l=1 rx=0x7efc400043a0 tx=0x7efc40010850).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:48.343 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:48.337+0000 7efc477fe700 1 -- 172.21.15.42:0/2363763283 <== mon.2 v2:172.21.15.40:3301/0 8 ==== mon_command_ack([{"var": "allow_new_snaps", "prefix": "fs set", "fs_name": "cephfs", "val": "true", "yes_i_really_mean_it": true}]=0 enabled new snapshots v47) v1 ==== 167+0+0 (secure 0 0 0) 0x7efc400322f0 con 0x7efc50063880 2019-12-05T05:10:48.371 INFO:tasks.workunit.client.0.smithi042.stderr:enabled new snapshots 2019-12-05T05:10:48.375 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:48.369+0000 7efc54dd4700 1 -- 172.21.15.42:0/2363763283 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7efc3403b4a0 msgr2=0x7efc3403d9b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:10:48.376 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:48.369+0000 7efc54dd4700 1 --2- 172.21.15.42:0/2363763283 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7efc3403b4a0 0x7efc3403d9b0 crc :-1 s=READY pgs=184 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:10:48.376 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:48.369+0000 7efc54dd4700 1 -- 172.21.15.42:0/2363763283 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efc50063880 msgr2=0x7efc50064300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:10:48.376 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:48.369+0000 7efc54dd4700 1 --2- 172.21.15.42:0/2363763283 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efc50063880 0x7efc50064300 secure :-1 s=READY pgs=81 cs=0 l=1 rx=0x7efc400043a0 tx=0x7efc40010850).stop 2019-12-05T05:10:48.376 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:48.369+0000 7efc54dd4700 1 -- 172.21.15.42:0/2363763283 shutdown_connections 2019-12-05T05:10:48.376 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:48.373+0000 7efc54dd4700 1 --2- 172.21.15.42:0/2363763283 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7efc3403b4a0 0x7efc3403d9b0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:10:48.376 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:48.373+0000 7efc54dd4700 1 --2- 172.21.15.42:0/2363763283 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efc50063880 0x7efc50064300 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:10:48.377 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:48.373+0000 7efc54dd4700 1 -- 172.21.15.42:0/2363763283 shutdown_connections 2019-12-05T05:10:48.377 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:48.373+0000 7efc54dd4700 1 -- 172.21.15.42:0/2363763283 wait complete. 2019-12-05T05:10:48.377 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:48.373+0000 7efc54dd4700 1 -- 172.21.15.42:0/2363763283 >> 172.21.15.42:0/2363763283 conn(0x7efc50144db0 msgr2=0x7efc500a9650 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:10:48.393 INFO:tasks.workunit.client.0.smithi042.stderr:+ mkdir a 2019-12-05T05:10:48.394 INFO:tasks.workunit.client.0.smithi042.stderr:+ mkdir .snap/bar 2019-12-05T05:10:48.425 INFO:tasks.workunit.client.0.smithi042.stderr:+ mkdir a/.snap/foo 2019-12-05T05:10:48.429 INFO:tasks.workunit.client.0.smithi042.stderr:+ rmdir a/.snap/foo 2019-12-05T05:10:48.440 INFO:tasks.workunit.client.0.smithi042.stderr:+ rmdir a 2019-12-05T05:10:48.446 INFO:tasks.workunit.client.0.smithi042.stderr:+ stat .snap/bar/a 2019-12-05T05:10:48.458 INFO:tasks.workunit.client.0.smithi042.stdout: File: .snap/bar/a 2019-12-05T05:10:48.459 INFO:tasks.workunit.client.0.smithi042.stdout: Size: 0 Blocks: 1 IO Block: 4096 directory 2019-12-05T05:10:48.459 INFO:tasks.workunit.client.0.smithi042.stdout:Device: 31h/49d Inode: 59955269551003028 Links: 2 2019-12-05T05:10:48.459 INFO:tasks.workunit.client.0.smithi042.stdout:Access: (0775/drwxrwxr-x) Uid: ( 1000/ ubuntu) Gid: ( 1216/ ubuntu) 2019-12-05T05:10:48.459 INFO:tasks.workunit.client.0.smithi042.stdout:Access: 2019-12-05 05:10:48.388846688 +0000 2019-12-05T05:10:48.460 INFO:tasks.workunit.client.0.smithi042.stdout:Modify: 2019-12-05 05:10:48.388846688 +0000 2019-12-05T05:10:48.460 INFO:tasks.workunit.client.0.smithi042.stdout:Change: 2019-12-05 05:10:48.388846688 +0000 2019-12-05T05:10:48.460 INFO:tasks.workunit.client.0.smithi042.stdout: Birth: - 2019-12-05T05:10:48.460 INFO:tasks.workunit.client.0.smithi042.stderr:+ rmdir .snap/bar 2019-12-05T05:10:48.679 INFO:tasks.workunit.client.0.smithi042.stderr:+ echo OK 2019-12-05T05:10:48.680 INFO:tasks.workunit.client.0.smithi042.stdout:OK 2019-12-05T05:10:48.680 INFO:teuthology.orchestra.run:Running command with timeout 3600 2019-12-05T05:10:48.680 INFO:teuthology.orchestra.run.smithi042:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2019-12-05T05:10:48.793 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-double-null.sh... 2019-12-05T05:10:48.793 INFO:teuthology.orchestra.run.smithi042:workunit test fs/snaps/snaptest-double-null.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c5290805033483ba0cec165c06bde07dae5f19ec TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-double-null.sh 2019-12-05T05:10:48.842 INFO:tasks.workunit.client.0.smithi042.stderr:+ set -e 2019-12-05T05:10:48.843 INFO:tasks.workunit.client.0.smithi042.stderr:+ ceph fs set cephfs allow_new_snaps true --yes-i-really-mean-it 2019-12-05T05:10:48.936 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:48.925+0000 7fcad3554700 1 Processor -- start 2019-12-05T05:10:48.936 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:48.925+0000 7fcad3554700 1 -- start start 2019-12-05T05:10:48.936 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:48.929+0000 7fcad3554700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fcacc0a69f0 0x7fcacc0a2840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:10:48.936 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:48.929+0000 7fcad3554700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcacc0652f0 0x7fcacc14f200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:10:48.937 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:48.929+0000 7fcad3554700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcacc0a7b60 con 0x7fcacc0a8030 2019-12-05T05:10:48.937 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:48.929+0000 7fcad3554700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcacc065b20 con 0x7fcacc0a69f0 2019-12-05T05:10:48.937 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:48.929+0000 7fcad3554700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcacc0a25a0 con 0x7fcacc0652f0 2019-12-05T05:10:48.937 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:48.929+0000 7fcad1af1700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fcacc0a69f0 0x7fcacc0a2840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:10:48.937 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:48.929+0000 7fcad1af1700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fcacc0a69f0 0x7fcacc0a2840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:57396/0 (socket says 172.21.15.42:57396) 2019-12-05T05:10:48.937 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:48.929+0000 7fcad1af1700 1 -- 172.21.15.42:0/2130294383 learned_addr learned my addr 172.21.15.42:0/2130294383 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:10:48.937 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:48.929+0000 7fcad22f2700 1 --2- 172.21.15.42:0/2130294383 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcacc0652f0 0x7fcacc14f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:10:48.938 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:48.929+0000 7fcad1af1700 1 -- 172.21.15.42:0/2130294383 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcacc0652f0 msgr2=0x7fcacc14f200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:10:48.938 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:48.929+0000 7fcad1af1700 1 --2- 172.21.15.42:0/2130294383 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcacc0652f0 0x7fcacc14f200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:10:48.938 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:48.929+0000 7fcad1af1700 1 -- 172.21.15.42:0/2130294383 >> v1:172.21.15.40:6789/0 conn(0x7fcacc0a8030 legacy=0x7fcacc144fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:10:48.938 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:48.929+0000 7fcad1af1700 1 -- 172.21.15.42:0/2130294383 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcacc0a8460 con 0x7fcacc0a69f0 2019-12-05T05:10:48.938 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:48.929+0000 7fcad0aef700 1 -- 172.21.15.42:0/2130294383 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2760166266 0 0) 0x7fcacc0a7b60 con 0x7fcacc0a8030 2019-12-05T05:10:48.938 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:48.929+0000 7fcad1af1700 1 --2- 172.21.15.42:0/2130294383 >> v2:172.21.15.42:3300/0 conn(0x7fcacc0a69f0 0x7fcacc0a2840 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fcacc0a7630 tx=0x7fcabc00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:48.938 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:48.929+0000 7fcad1af1700 1 --2- 172.21.15.42:0/2130294383 >> v2:172.21.15.42:3300/0 conn(0x7fcacc0a69f0 0x7fcacc0a2840 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fcacc0a7630 tx=0x7fcabc00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:48.939 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:48.929+0000 7fcad1af1700 1 --2- 172.21.15.42:0/2130294383 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcacc0a69f0 0x7fcacc0a2840 secure :-1 s=READY pgs=410 cs=0 l=1 rx=0x7fcacc0a7630 tx=0x7fcabc00fd30).ready entity=mon.1 client_cookie=6ade21970f67b083 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:10:48.939 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:48.929+0000 7fcad1af1700 1 --2- 172.21.15.42:0/2130294383 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcacc0a69f0 0x7fcacc0a2840 secure :-1 s=THROTTLE_DONE pgs=410 cs=0 l=1 rx=0x7fcacc0a7630 tx=0x7fcabc00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:48.939 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:48.929+0000 7fcad0aef700 1 -- 172.21.15.42:0/2130294383 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcabc011370 con 0x7fcacc0a69f0 2019-12-05T05:10:48.939 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:48.929+0000 7fcad1af1700 1 --2- 172.21.15.42:0/2130294383 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcacc0a69f0 0x7fcacc0a2840 secure :-1 s=THROTTLE_DONE pgs=410 cs=0 l=1 rx=0x7fcacc0a7630 tx=0x7fcabc00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:48.939 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:48.929+0000 7fcad1af1700 1 --2- 172.21.15.42:0/2130294383 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcacc0a69f0 0x7fcacc0a2840 secure :-1 s=THROTTLE_DONE pgs=410 cs=0 l=1 rx=0x7fcacc0a7630 tx=0x7fcabc00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:48.939 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:48.929+0000 7fcad0aef700 1 -- 172.21.15.42:0/2130294383 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcabc008bb0 con 0x7fcacc0a69f0 2019-12-05T05:10:48.939 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:48.929+0000 7fcad0aef700 1 -- 172.21.15.42:0/2130294383 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcabc011950 con 0x7fcacc0a69f0 2019-12-05T05:10:48.941 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:48.929+0000 7fcad3554700 1 -- 172.21.15.42:0/2130294383 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcacc0a69f0 msgr2=0x7fcacc0a2840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:10:48.941 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:48.929+0000 7fcad3554700 1 --2- 172.21.15.42:0/2130294383 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcacc0a69f0 0x7fcacc0a2840 secure :-1 s=READY pgs=410 cs=0 l=1 rx=0x7fcacc0a7630 tx=0x7fcabc00fd30).stop 2019-12-05T05:10:48.941 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:48.929+0000 7fcad3554700 1 -- 172.21.15.42:0/2130294383 shutdown_connections 2019-12-05T05:10:48.941 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:48.929+0000 7fcad3554700 1 --2- 172.21.15.42:0/2130294383 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcacc0652f0 0x7fcacc14f200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:10:48.942 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:48.929+0000 7fcad3554700 1 --2- 172.21.15.42:0/2130294383 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcacc0a69f0 0x7fcacc0a2840 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:10:48.946 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:48.929+0000 7fcad3554700 1 -- 172.21.15.42:0/2130294383 shutdown_connections 2019-12-05T05:10:48.946 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:48.929+0000 7fcad3554700 1 -- 172.21.15.42:0/2130294383 wait complete. 2019-12-05T05:10:48.946 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:48.929+0000 7fcad3554700 1 -- 172.21.15.42:0/2130294383 >> 172.21.15.42:0/2130294383 conn(0x7fcacc059570 msgr2=0x7fcacc0599a0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:10:48.950 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:48.933+0000 7fcad3554700 1 Processor -- start 2019-12-05T05:10:48.950 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:48.933+0000 7fcad3554700 1 -- start start 2019-12-05T05:10:48.950 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:48.933+0000 7fcad3554700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcacc120e50 0x7fcacc07ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:10:48.950 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:48.933+0000 7fcad3554700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fcacc1205e0 0x7fcacc139f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:10:48.950 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:48.933+0000 7fcad3554700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcacc07cfa0 con 0x7fcacc1201b0 2019-12-05T05:10:48.950 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:48.933+0000 7fcad3554700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcacc0a5ee0 con 0x7fcacc1205e0 2019-12-05T05:10:48.951 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:48.933+0000 7fcad3554700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcacc07d260 con 0x7fcacc120e50 2019-12-05T05:10:48.951 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:48.933+0000 7fcad22f2700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fcacc1205e0 0x7fcacc139f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:10:48.951 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:48.933+0000 7fcad22f2700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fcacc1205e0 0x7fcacc139f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:57406/0 (socket says 172.21.15.42:57406) 2019-12-05T05:10:48.952 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:48.933+0000 7fcad22f2700 1 -- 172.21.15.42:0/2381490921 learned_addr learned my addr 172.21.15.42:0/2381490921 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:10:48.952 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:48.933+0000 7fcad12f0700 1 --1- 172.21.15.42:0/2381490921 >> v1:172.21.15.42:6789/0 conn(0x7fcacc1201b0 0x7fcacc1221c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:57440/0 (socket says 172.21.15.42:57440) 2019-12-05T05:10:48.952 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:48.933+0000 7fcad1af1700 1 --2- 172.21.15.42:0/2381490921 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcacc120e50 0x7fcacc07ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:10:48.952 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:48.933+0000 7fcad22f2700 1 -- 172.21.15.42:0/2381490921 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcacc120e50 msgr2=0x7fcacc07ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:10:48.953 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:48.933+0000 7fcad22f2700 1 --2- 172.21.15.42:0/2381490921 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcacc120e50 0x7fcacc07ca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:10:48.953 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:48.933+0000 7fcad22f2700 1 -- 172.21.15.42:0/2381490921 >> v1:172.21.15.42:6789/0 conn(0x7fcacc1201b0 legacy=0x7fcacc1221c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:10:48.953 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:48.933+0000 7fcad22f2700 1 -- 172.21.15.42:0/2381490921 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcacc1229c0 con 0x7fcacc1205e0 2019-12-05T05:10:48.953 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:48.933+0000 7fcad22f2700 1 --2- 172.21.15.42:0/2381490921 >> v2:172.21.15.42:3300/0 conn(0x7fcacc1205e0 0x7fcacc139f70 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fcac00010e0 tx=0x7fcac0001120).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:48.953 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:48.933+0000 7fcad22f2700 1 --2- 172.21.15.42:0/2381490921 >> v2:172.21.15.42:3300/0 conn(0x7fcacc1205e0 0x7fcacc139f70 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fcac00010e0 tx=0x7fcac0001120).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:48.953 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:48.933+0000 7fcad22f2700 1 --2- 172.21.15.42:0/2381490921 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcacc1205e0 0x7fcacc139f70 secure :-1 s=READY pgs=412 cs=0 l=1 rx=0x7fcac00010e0 tx=0x7fcac0001120).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:10:48.953 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:48.933+0000 7fcad22f2700 1 --2- 172.21.15.42:0/2381490921 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcacc1205e0 0x7fcacc139f70 secure :-1 s=THROTTLE_DONE pgs=412 cs=0 l=1 rx=0x7fcac00010e0 tx=0x7fcac0001120).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:48.954 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:48.933+0000 7fcad22f2700 1 --2- 172.21.15.42:0/2381490921 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcacc1205e0 0x7fcacc139f70 secure :-1 s=THROTTLE_DONE pgs=412 cs=0 l=1 rx=0x7fcac00010e0 tx=0x7fcac0001120).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:48.954 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:48.933+0000 7fcaca7fc700 1 -- 172.21.15.42:0/2381490921 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcac4005ca0 con 0x7fcacc1205e0 2019-12-05T05:10:48.954 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:48.933+0000 7fcad22f2700 1 --2- 172.21.15.42:0/2381490921 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcacc1205e0 0x7fcacc139f70 secure :-1 s=THROTTLE_DONE pgs=412 cs=0 l=1 rx=0x7fcac00010e0 tx=0x7fcac0001120).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:48.954 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:48.933+0000 7fcaca7fc700 1 -- 172.21.15.42:0/2381490921 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcac40058c0 con 0x7fcacc1205e0 2019-12-05T05:10:48.954 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:48.933+0000 7fcaca7fc700 1 -- 172.21.15.42:0/2381490921 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcac4007530 con 0x7fcacc1205e0 2019-12-05T05:10:48.954 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:48.933+0000 7fcad3554700 1 -- 172.21.15.42:0/2381490921 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcacc13a4c0 con 0x7fcacc1205e0 2019-12-05T05:10:48.954 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:48.933+0000 7fcad3554700 1 -- 172.21.15.42:0/2381490921 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcacc13a9f0 con 0x7fcacc1205e0 2019-12-05T05:10:48.955 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:48.937+0000 7fcad22f2700 1 --2- 172.21.15.42:0/2381490921 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcacc1205e0 0x7fcacc139f70 secure :-1 s=THROTTLE_DONE pgs=412 cs=0 l=1 rx=0x7fcac00010e0 tx=0x7fcac0001120).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:48.955 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:48.937+0000 7fcaca7fc700 1 -- 172.21.15.42:0/2381490921 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fcac400c700 con 0x7fcacc1205e0 2019-12-05T05:10:48.955 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:48.937+0000 7fcad22f2700 1 --2- 172.21.15.42:0/2381490921 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcacc1205e0 0x7fcacc139f70 secure :-1 s=THROTTLE_DONE pgs=412 cs=0 l=1 rx=0x7fcac00010e0 tx=0x7fcac0001120).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:48.955 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:48.937+0000 7fcaca7fc700 1 --2- 172.21.15.42:0/2381490921 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fcaac03adb0 0x7fcaac03d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:10:48.955 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:48.937+0000 7fcaca7fc700 1 -- 172.21.15.42:0/2381490921 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fcac40090a0 con 0x7fcacc1205e0 2019-12-05T05:10:48.955 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:48.937+0000 7fcad1af1700 1 --2- 172.21.15.42:0/2381490921 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fcaac03adb0 0x7fcaac03d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:10:48.955 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:48.937+0000 7fcad1af1700 1 --2- 172.21.15.42:0/2381490921 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fcaac03adb0 0x7fcaac03d2c0 crc :-1 s=READY pgs=185 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:10:48.956 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:48.941+0000 7fcad3554700 1 -- 172.21.15.42:0/2381490921 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcacc05d2c0 con 0x7fcacc1205e0 2019-12-05T05:10:48.956 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:48.949+0000 7fcad22f2700 1 --2- 172.21.15.42:0/2381490921 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcacc1205e0 0x7fcacc139f70 secure :-1 s=THROTTLE_DONE pgs=412 cs=0 l=1 rx=0x7fcac00010e0 tx=0x7fcac0001120).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:48.956 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:48.949+0000 7fcaca7fc700 1 -- 172.21.15.42:0/2381490921 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fcac4032880 con 0x7fcacc1205e0 2019-12-05T05:10:49.123 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:49.117+0000 7fcad3554700 1 -- 172.21.15.42:0/2381490921 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"var": "allow_new_snaps", "prefix": "fs set", "fs_name": "cephfs", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fcacc05ba70 con 0x7fcacc1205e0 2019-12-05T05:10:49.349 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:49.341+0000 7fcad22f2700 1 --2- 172.21.15.42:0/2381490921 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcacc1205e0 0x7fcacc139f70 secure :-1 s=THROTTLE_DONE pgs=412 cs=0 l=1 rx=0x7fcac00010e0 tx=0x7fcac0001120).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:49.349 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:49.341+0000 7fcaca7fc700 1 -- 172.21.15.42:0/2381490921 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"var": "allow_new_snaps", "prefix": "fs set", "fs_name": "cephfs", "val": "true", "yes_i_really_mean_it": true}]=0 enabled new snapshots v48) v1 ==== 167+0+0 (secure 0 0 0) 0x7fcac4032ae0 con 0x7fcacc1205e0 2019-12-05T05:10:49.502 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:49.497+0000 7fcad3554700 1 -- 172.21.15.42:0/2381490921 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"var": "allow_new_snaps", "prefix": "fs set", "fs_name": "cephfs", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fcacc069770 con 0x7fcacc1205e0 2019-12-05T05:10:49.515 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T05:10:49.509+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 41 pg[3.1s0( v 40'1400 (0'0,40'1400] local-lis/les=21/22 n=2 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=40'1400 lcod 40'1399 mlcod 40'1399 active+clean+snaptrim trimq=[1a6~2,1a9~2] ps=[1a2~4]] removing snap head 2019-12-05T05:10:50.363 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:50.357+0000 7fcad22f2700 1 --2- 172.21.15.42:0/2381490921 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcacc1205e0 0x7fcacc139f70 secure :-1 s=THROTTLE_DONE pgs=412 cs=0 l=1 rx=0x7fcac00010e0 tx=0x7fcac0001120).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:50.363 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:50.357+0000 7fcaca7fc700 1 -- 172.21.15.42:0/2381490921 <== mon.1 v2:172.21.15.42:3300/0 8 ==== mon_command_ack([{"var": "allow_new_snaps", "prefix": "fs set", "fs_name": "cephfs", "val": "true", "yes_i_really_mean_it": true}]=0 enabled new snapshots v49) v1 ==== 167+0+0 (secure 0 0 0) 0x7fcac4033690 con 0x7fcacc1205e0 2019-12-05T05:10:50.367 INFO:tasks.workunit.client.0.smithi042.stderr:enabled new snapshots 2019-12-05T05:10:50.371 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:50.365+0000 7fcad3554700 1 -- 172.21.15.42:0/2381490921 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fcaac03adb0 msgr2=0x7fcaac03d2c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:10:50.371 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:50.365+0000 7fcad3554700 1 --2- 172.21.15.42:0/2381490921 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fcaac03adb0 0x7fcaac03d2c0 crc :-1 s=READY pgs=185 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:10:50.372 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:50.365+0000 7fcad3554700 1 -- 172.21.15.42:0/2381490921 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcacc1205e0 msgr2=0x7fcacc139f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:10:50.372 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:50.365+0000 7fcad3554700 1 --2- 172.21.15.42:0/2381490921 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcacc1205e0 0x7fcacc139f70 secure :-1 s=READY pgs=412 cs=0 l=1 rx=0x7fcac00010e0 tx=0x7fcac0001120).stop 2019-12-05T05:10:50.372 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:50.365+0000 7fcad3554700 1 -- 172.21.15.42:0/2381490921 shutdown_connections 2019-12-05T05:10:50.373 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:50.365+0000 7fcad3554700 1 --2- 172.21.15.42:0/2381490921 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fcaac03adb0 0x7fcaac03d2c0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:10:50.373 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:50.365+0000 7fcad3554700 1 --2- 172.21.15.42:0/2381490921 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcacc1205e0 0x7fcacc139f70 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:10:50.373 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:50.365+0000 7fcad3554700 1 --2- 172.21.15.42:0/2381490921 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcacc120e50 0x7fcacc07ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:10:50.373 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:50.365+0000 7fcad3554700 1 -- 172.21.15.42:0/2381490921 shutdown_connections 2019-12-05T05:10:50.373 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:50.365+0000 7fcad3554700 1 -- 172.21.15.42:0/2381490921 wait complete. 2019-12-05T05:10:50.373 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:50.365+0000 7fcad3554700 1 -- 172.21.15.42:0/2381490921 >> 172.21.15.42:0/2381490921 conn(0x7fcacc0a8700 msgr2=0x7fcacc0aa6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:10:50.382 INFO:tasks.workunit.client.0.smithi042.stderr:+ seq 1 20 2019-12-05T05:10:50.383 INFO:tasks.workunit.client.0.smithi042.stderr:+ mkdir a 2019-12-05T05:10:50.388 INFO:tasks.workunit.client.0.smithi042.stderr:+ mkdir a/.snap/one 2019-12-05T05:10:50.390 INFO:tasks.workunit.client.0.smithi042.stderr:+ cat 2019-12-05T05:10:50.409 INFO:tasks.workunit.client.0.smithi042.stderr:+ mkdir a/.snap/two 2019-12-05T05:10:50.421 INFO:tasks.workunit.client.0.smithi042.stderr:+ chmod 777 a/foo 2019-12-05T05:10:50.423 INFO:tasks.workunit.client.0.smithi042.stderr:+ sync 2019-12-05T05:10:50.605 INFO:tasks.workunit.client.0.smithi042.stderr:+ ps 2019-12-05T05:10:50.617 INFO:tasks.workunit.client.0.smithi042.stdout: PID TTY TIME CMD 2019-12-05T05:10:50.618 INFO:tasks.workunit.client.0.smithi042.stdout: 1714 ? 00:00:00 systemd 2019-12-05T05:10:50.618 INFO:tasks.workunit.client.0.smithi042.stdout: 1715 ? 00:00:00 (sd-pam) 2019-12-05T05:10:50.618 INFO:tasks.workunit.client.0.smithi042.stdout: 7572 ? 00:00:01 sshd 2019-12-05T05:10:50.618 INFO:tasks.workunit.client.0.smithi042.stdout: 18572 ? 00:00:00 timeout 2019-12-05T05:10:50.618 INFO:tasks.workunit.client.0.smithi042.stdout: 18575 ? 00:00:00 snaptest-double 2019-12-05T05:10:50.619 INFO:tasks.workunit.client.0.smithi042.stdout: 18612 ? 00:00:00 ps 2019-12-05T05:10:50.622 INFO:tasks.workunit.client.0.smithi042.stderr:+ rmdir a/.snap/one a/.snap/two 2019-12-05T05:10:50.655 INFO:tasks.workunit.client.0.smithi042.stderr:+ rm a/foo 2019-12-05T05:10:50.660 INFO:tasks.workunit.client.0.smithi042.stderr:+ rmdir a 2019-12-05T05:10:50.673 INFO:tasks.workunit.client.0.smithi042.stderr:+ mkdir a 2019-12-05T05:10:50.680 INFO:tasks.workunit.client.0.smithi042.stderr:+ mkdir a/.snap/one 2019-12-05T05:10:50.680 INFO:tasks.workunit.client.0.smithi042.stderr:+ cat 2019-12-05T05:10:50.697 INFO:tasks.workunit.client.0.smithi042.stderr:+ mkdir a/.snap/two 2019-12-05T05:10:50.708 INFO:tasks.workunit.client.0.smithi042.stderr:+ chmod 777 a/foo 2019-12-05T05:10:50.715 INFO:tasks.workunit.client.0.smithi042.stderr:+ sync 2019-12-05T05:10:50.767 INFO:tasks.workunit.client.0.smithi042.stderr:+ ps 2019-12-05T05:10:50.779 INFO:tasks.workunit.client.0.smithi042.stdout: PID TTY TIME CMD 2019-12-05T05:10:50.779 INFO:tasks.workunit.client.0.smithi042.stdout: 1714 ? 00:00:00 systemd 2019-12-05T05:10:50.779 INFO:tasks.workunit.client.0.smithi042.stdout: 1715 ? 00:00:00 (sd-pam) 2019-12-05T05:10:50.780 INFO:tasks.workunit.client.0.smithi042.stdout: 7572 ? 00:00:01 sshd 2019-12-05T05:10:50.780 INFO:tasks.workunit.client.0.smithi042.stdout: 18572 ? 00:00:00 timeout 2019-12-05T05:10:50.780 INFO:tasks.workunit.client.0.smithi042.stdout: 18575 ? 00:00:00 snaptest-double 2019-12-05T05:10:50.780 INFO:tasks.workunit.client.0.smithi042.stdout: 18622 ? 00:00:00 ps 2019-12-05T05:10:50.785 INFO:tasks.workunit.client.0.smithi042.stderr:+ rmdir a/.snap/one a/.snap/two 2019-12-05T05:10:50.812 INFO:tasks.workunit.client.0.smithi042.stderr:+ rm a/foo 2019-12-05T05:10:50.816 INFO:tasks.workunit.client.0.smithi042.stderr:+ rmdir a 2019-12-05T05:10:50.827 INFO:tasks.workunit.client.0.smithi042.stderr:+ mkdir a 2019-12-05T05:10:50.830 INFO:tasks.workunit.client.0.smithi042.stderr:+ mkdir a/.snap/one 2019-12-05T05:10:50.834 INFO:tasks.workunit.client.0.smithi042.stderr:+ cat 2019-12-05T05:10:50.854 INFO:tasks.workunit.client.0.smithi042.stderr:+ mkdir a/.snap/two 2019-12-05T05:10:50.868 INFO:tasks.workunit.client.0.smithi042.stderr:+ chmod 777 a/foo 2019-12-05T05:10:50.869 INFO:tasks.workunit.client.0.smithi042.stderr:+ sync 2019-12-05T05:10:50.917 INFO:tasks.workunit.client.0.smithi042.stderr:+ ps 2019-12-05T05:10:50.930 INFO:tasks.workunit.client.0.smithi042.stdout: PID TTY TIME CMD 2019-12-05T05:10:50.930 INFO:tasks.workunit.client.0.smithi042.stdout: 1714 ? 00:00:00 systemd 2019-12-05T05:10:50.930 INFO:tasks.workunit.client.0.smithi042.stdout: 1715 ? 00:00:00 (sd-pam) 2019-12-05T05:10:50.930 INFO:tasks.workunit.client.0.smithi042.stdout: 7572 ? 00:00:01 sshd 2019-12-05T05:10:50.930 INFO:tasks.workunit.client.0.smithi042.stdout: 18572 ? 00:00:00 timeout 2019-12-05T05:10:50.931 INFO:tasks.workunit.client.0.smithi042.stdout: 18575 ? 00:00:00 snaptest-double 2019-12-05T05:10:50.931 INFO:tasks.workunit.client.0.smithi042.stdout: 18632 ? 00:00:00 ps 2019-12-05T05:10:50.935 INFO:tasks.workunit.client.0.smithi042.stderr:+ rmdir a/.snap/one a/.snap/two 2019-12-05T05:10:51.052 INFO:tasks.workunit.client.0.smithi042.stderr:+ rm a/foo 2019-12-05T05:10:51.058 INFO:tasks.workunit.client.0.smithi042.stderr:+ rmdir a 2019-12-05T05:10:51.073 INFO:tasks.workunit.client.0.smithi042.stderr:+ mkdir a 2019-12-05T05:10:51.077 INFO:tasks.workunit.client.0.smithi042.stderr:+ mkdir a/.snap/one 2019-12-05T05:10:51.081 INFO:tasks.workunit.client.0.smithi042.stderr:+ cat 2019-12-05T05:10:51.096 INFO:tasks.workunit.client.0.smithi042.stderr:+ mkdir a/.snap/two 2019-12-05T05:10:51.107 INFO:tasks.workunit.client.0.smithi042.stderr:+ chmod 777 a/foo 2019-12-05T05:10:51.109 INFO:tasks.workunit.client.0.smithi042.stderr:+ sync 2019-12-05T05:10:51.174 INFO:tasks.workunit.client.0.smithi042.stderr:+ ps 2019-12-05T05:10:51.183 INFO:tasks.workunit.client.0.smithi042.stdout: PID TTY TIME CMD 2019-12-05T05:10:51.183 INFO:tasks.workunit.client.0.smithi042.stdout: 1714 ? 00:00:00 systemd 2019-12-05T05:10:51.184 INFO:tasks.workunit.client.0.smithi042.stdout: 1715 ? 00:00:00 (sd-pam) 2019-12-05T05:10:51.184 INFO:tasks.workunit.client.0.smithi042.stdout: 7572 ? 00:00:01 sshd 2019-12-05T05:10:51.184 INFO:tasks.workunit.client.0.smithi042.stdout: 18572 ? 00:00:00 timeout 2019-12-05T05:10:51.184 INFO:tasks.workunit.client.0.smithi042.stdout: 18575 ? 00:00:00 snaptest-double 2019-12-05T05:10:51.185 INFO:tasks.workunit.client.0.smithi042.stdout: 18642 ? 00:00:00 ps 2019-12-05T05:10:51.189 INFO:tasks.workunit.client.0.smithi042.stderr:+ rmdir a/.snap/one a/.snap/two 2019-12-05T05:10:51.217 INFO:tasks.workunit.client.0.smithi042.stderr:+ rm a/foo 2019-12-05T05:10:51.222 INFO:tasks.workunit.client.0.smithi042.stderr:+ rmdir a 2019-12-05T05:10:51.250 INFO:tasks.workunit.client.0.smithi042.stderr:+ mkdir a 2019-12-05T05:10:51.251 INFO:tasks.workunit.client.0.smithi042.stderr:+ mkdir a/.snap/one 2019-12-05T05:10:51.251 INFO:tasks.workunit.client.0.smithi042.stderr:+ cat 2019-12-05T05:10:51.263 INFO:tasks.workunit.client.0.smithi042.stderr:+ mkdir a/.snap/two 2019-12-05T05:10:51.280 INFO:tasks.workunit.client.0.smithi042.stderr:+ chmod 777 a/foo 2019-12-05T05:10:51.281 INFO:tasks.workunit.client.0.smithi042.stderr:+ sync 2019-12-05T05:10:51.327 INFO:tasks.workunit.client.0.smithi042.stderr:+ ps 2019-12-05T05:10:51.338 INFO:tasks.workunit.client.0.smithi042.stdout: PID TTY TIME CMD 2019-12-05T05:10:51.339 INFO:tasks.workunit.client.0.smithi042.stdout: 1714 ? 00:00:00 systemd 2019-12-05T05:10:51.339 INFO:tasks.workunit.client.0.smithi042.stdout: 1715 ? 00:00:00 (sd-pam) 2019-12-05T05:10:51.339 INFO:tasks.workunit.client.0.smithi042.stdout: 7572 ? 00:00:01 sshd 2019-12-05T05:10:51.339 INFO:tasks.workunit.client.0.smithi042.stdout: 18572 ? 00:00:00 timeout 2019-12-05T05:10:51.340 INFO:tasks.workunit.client.0.smithi042.stdout: 18575 ? 00:00:00 snaptest-double 2019-12-05T05:10:51.340 INFO:tasks.workunit.client.0.smithi042.stdout: 18652 ? 00:00:00 ps 2019-12-05T05:10:51.344 INFO:tasks.workunit.client.0.smithi042.stderr:+ rmdir a/.snap/one a/.snap/two 2019-12-05T05:10:51.370 INFO:tasks.workunit.client.0.smithi042.stderr:+ rm a/foo 2019-12-05T05:10:51.374 INFO:tasks.workunit.client.0.smithi042.stderr:+ rmdir a 2019-12-05T05:10:51.443 INFO:tasks.workunit.client.0.smithi042.stderr:+ mkdir a 2019-12-05T05:10:51.443 INFO:tasks.workunit.client.0.smithi042.stderr:+ mkdir a/.snap/one 2019-12-05T05:10:51.444 INFO:tasks.workunit.client.0.smithi042.stderr:+ cat 2019-12-05T05:10:52.140 INFO:tasks.workunit.client.0.smithi042.stderr:+ mkdir a/.snap/two 2019-12-05T05:10:52.158 INFO:tasks.workunit.client.0.smithi042.stderr:+ chmod 777 a/foo 2019-12-05T05:10:52.164 INFO:tasks.workunit.client.0.smithi042.stderr:+ sync 2019-12-05T05:10:52.221 INFO:tasks.workunit.client.0.smithi042.stderr:+ ps 2019-12-05T05:10:52.234 INFO:tasks.workunit.client.0.smithi042.stdout: PID TTY TIME CMD 2019-12-05T05:10:52.241 INFO:tasks.workunit.client.0.smithi042.stdout: 1714 ? 00:00:00 systemd 2019-12-05T05:10:52.241 INFO:tasks.workunit.client.0.smithi042.stdout: 1715 ? 00:00:00 (sd-pam) 2019-12-05T05:10:52.241 INFO:tasks.workunit.client.0.smithi042.stdout: 7572 ? 00:00:01 sshd 2019-12-05T05:10:52.241 INFO:tasks.workunit.client.0.smithi042.stdout: 18572 ? 00:00:00 timeout 2019-12-05T05:10:52.241 INFO:tasks.workunit.client.0.smithi042.stdout: 18575 ? 00:00:00 snaptest-double 2019-12-05T05:10:52.242 INFO:tasks.workunit.client.0.smithi042.stdout: 18662 ? 00:00:00 ps 2019-12-05T05:10:52.242 INFO:tasks.workunit.client.0.smithi042.stderr:+ rmdir a/.snap/one a/.snap/two 2019-12-05T05:10:52.267 INFO:tasks.workunit.client.0.smithi042.stderr:+ rm a/foo 2019-12-05T05:10:52.274 INFO:tasks.workunit.client.0.smithi042.stderr:+ rmdir a 2019-12-05T05:10:52.285 INFO:tasks.workunit.client.0.smithi042.stderr:+ mkdir a 2019-12-05T05:10:52.288 INFO:tasks.workunit.client.0.smithi042.stderr:+ mkdir a/.snap/one 2019-12-05T05:10:52.290 INFO:tasks.workunit.client.0.smithi042.stderr:+ cat 2019-12-05T05:10:52.306 INFO:tasks.workunit.client.0.smithi042.stderr:+ mkdir a/.snap/two 2019-12-05T05:10:52.316 INFO:tasks.workunit.client.0.smithi042.stderr:+ chmod 777 a/foo 2019-12-05T05:10:52.317 INFO:tasks.workunit.client.0.smithi042.stderr:+ sync 2019-12-05T05:10:52.368 INFO:tasks.workunit.client.0.smithi042.stderr:+ ps 2019-12-05T05:10:52.384 INFO:tasks.workunit.client.0.smithi042.stdout: PID TTY TIME CMD 2019-12-05T05:10:52.384 INFO:tasks.workunit.client.0.smithi042.stdout: 1714 ? 00:00:00 systemd 2019-12-05T05:10:52.385 INFO:tasks.workunit.client.0.smithi042.stdout: 1715 ? 00:00:00 (sd-pam) 2019-12-05T05:10:52.385 INFO:tasks.workunit.client.0.smithi042.stdout: 7572 ? 00:00:01 sshd 2019-12-05T05:10:52.385 INFO:tasks.workunit.client.0.smithi042.stdout: 18572 ? 00:00:00 timeout 2019-12-05T05:10:52.385 INFO:tasks.workunit.client.0.smithi042.stdout: 18575 ? 00:00:00 snaptest-double 2019-12-05T05:10:52.385 INFO:tasks.workunit.client.0.smithi042.stdout: 18672 ? 00:00:00 ps 2019-12-05T05:10:52.395 INFO:tasks.workunit.client.0.smithi042.stderr:+ rmdir a/.snap/one a/.snap/two 2019-12-05T05:10:52.420 INFO:tasks.workunit.client.0.smithi042.stderr:+ rm a/foo 2019-12-05T05:10:52.425 INFO:tasks.workunit.client.0.smithi042.stderr:+ rmdir a 2019-12-05T05:10:52.439 INFO:tasks.workunit.client.0.smithi042.stderr:+ mkdir a 2019-12-05T05:10:52.442 INFO:tasks.workunit.client.0.smithi042.stderr:+ mkdir a/.snap/one 2019-12-05T05:10:52.446 INFO:tasks.workunit.client.0.smithi042.stderr:+ cat 2019-12-05T05:10:52.465 INFO:tasks.workunit.client.0.smithi042.stderr:+ mkdir a/.snap/two 2019-12-05T05:10:52.480 INFO:tasks.workunit.client.0.smithi042.stderr:+ chmod 777 a/foo 2019-12-05T05:10:52.482 INFO:tasks.workunit.client.0.smithi042.stderr:+ sync 2019-12-05T05:10:52.527 INFO:tasks.workunit.client.0.smithi042.stderr:+ ps 2019-12-05T05:10:52.542 INFO:tasks.workunit.client.0.smithi042.stdout: PID TTY TIME CMD 2019-12-05T05:10:52.543 INFO:tasks.workunit.client.0.smithi042.stdout: 1714 ? 00:00:00 systemd 2019-12-05T05:10:52.543 INFO:tasks.workunit.client.0.smithi042.stdout: 1715 ? 00:00:00 (sd-pam) 2019-12-05T05:10:52.543 INFO:tasks.workunit.client.0.smithi042.stdout: 7572 ? 00:00:01 sshd 2019-12-05T05:10:52.543 INFO:tasks.workunit.client.0.smithi042.stdout: 18572 ? 00:00:00 timeout 2019-12-05T05:10:52.543 INFO:tasks.workunit.client.0.smithi042.stdout: 18575 ? 00:00:00 snaptest-double 2019-12-05T05:10:52.544 INFO:tasks.workunit.client.0.smithi042.stdout: 18682 ? 00:00:00 ps 2019-12-05T05:10:52.547 INFO:tasks.workunit.client.0.smithi042.stderr:+ rmdir a/.snap/one a/.snap/two 2019-12-05T05:10:52.572 INFO:tasks.workunit.client.0.smithi042.stderr:+ rm a/foo 2019-12-05T05:10:52.591 INFO:tasks.workunit.client.0.smithi042.stderr:+ rmdir a 2019-12-05T05:10:52.591 INFO:tasks.workunit.client.0.smithi042.stderr:+ mkdir a 2019-12-05T05:10:52.594 INFO:tasks.workunit.client.0.smithi042.stderr:+ mkdir a/.snap/one 2019-12-05T05:10:52.604 INFO:tasks.workunit.client.0.smithi042.stderr:+ cat 2019-12-05T05:10:52.615 INFO:tasks.workunit.client.0.smithi042.stderr:+ mkdir a/.snap/two 2019-12-05T05:10:52.628 INFO:tasks.workunit.client.0.smithi042.stderr:+ chmod 777 a/foo 2019-12-05T05:10:52.629 INFO:tasks.workunit.client.0.smithi042.stderr:+ sync 2019-12-05T05:10:52.679 INFO:tasks.workunit.client.0.smithi042.stderr:+ ps 2019-12-05T05:10:52.689 INFO:tasks.workunit.client.0.smithi042.stdout: PID TTY TIME CMD 2019-12-05T05:10:52.690 INFO:tasks.workunit.client.0.smithi042.stdout: 1714 ? 00:00:00 systemd 2019-12-05T05:10:52.690 INFO:tasks.workunit.client.0.smithi042.stdout: 1715 ? 00:00:00 (sd-pam) 2019-12-05T05:10:52.690 INFO:tasks.workunit.client.0.smithi042.stdout: 7572 ? 00:00:01 sshd 2019-12-05T05:10:52.690 INFO:tasks.workunit.client.0.smithi042.stdout: 18572 ? 00:00:00 timeout 2019-12-05T05:10:52.691 INFO:tasks.workunit.client.0.smithi042.stdout: 18575 ? 00:00:00 snaptest-double 2019-12-05T05:10:52.691 INFO:tasks.workunit.client.0.smithi042.stdout: 18692 ? 00:00:00 ps 2019-12-05T05:10:52.695 INFO:tasks.workunit.client.0.smithi042.stderr:+ rmdir a/.snap/one a/.snap/two 2019-12-05T05:10:52.733 INFO:tasks.workunit.client.0.smithi042.stderr:+ rm a/foo 2019-12-05T05:10:52.733 INFO:tasks.workunit.client.0.smithi042.stderr:+ rmdir a 2019-12-05T05:10:52.740 INFO:tasks.workunit.client.0.smithi042.stderr:+ mkdir a 2019-12-05T05:10:52.744 INFO:tasks.workunit.client.0.smithi042.stderr:+ mkdir a/.snap/one 2019-12-05T05:10:52.747 INFO:tasks.workunit.client.0.smithi042.stderr:+ cat 2019-12-05T05:10:52.762 INFO:tasks.workunit.client.0.smithi042.stderr:+ mkdir a/.snap/two 2019-12-05T05:10:52.772 INFO:tasks.workunit.client.0.smithi042.stderr:+ chmod 777 a/foo 2019-12-05T05:10:52.775 INFO:tasks.workunit.client.0.smithi042.stderr:+ sync 2019-12-05T05:10:52.839 INFO:tasks.workunit.client.0.smithi042.stderr:+ ps 2019-12-05T05:10:52.855 INFO:tasks.workunit.client.0.smithi042.stdout: PID TTY TIME CMD 2019-12-05T05:10:52.855 INFO:tasks.workunit.client.0.smithi042.stdout: 1714 ? 00:00:00 systemd 2019-12-05T05:10:52.855 INFO:tasks.workunit.client.0.smithi042.stdout: 1715 ? 00:00:00 (sd-pam) 2019-12-05T05:10:52.856 INFO:tasks.workunit.client.0.smithi042.stdout: 7572 ? 00:00:01 sshd 2019-12-05T05:10:52.856 INFO:tasks.workunit.client.0.smithi042.stdout: 18572 ? 00:00:00 timeout 2019-12-05T05:10:52.856 INFO:tasks.workunit.client.0.smithi042.stdout: 18575 ? 00:00:00 snaptest-double 2019-12-05T05:10:52.856 INFO:tasks.workunit.client.0.smithi042.stdout: 18702 ? 00:00:00 ps 2019-12-05T05:10:52.857 INFO:tasks.workunit.client.0.smithi042.stderr:+ rmdir a/.snap/one a/.snap/two 2019-12-05T05:10:52.887 INFO:tasks.workunit.client.0.smithi042.stderr:+ rm a/foo 2019-12-05T05:10:52.891 INFO:tasks.workunit.client.0.smithi042.stderr:+ rmdir a 2019-12-05T05:10:52.904 INFO:tasks.workunit.client.0.smithi042.stderr:+ mkdir a 2019-12-05T05:10:52.908 INFO:tasks.workunit.client.0.smithi042.stderr:+ mkdir a/.snap/one 2019-12-05T05:10:52.911 INFO:tasks.workunit.client.0.smithi042.stderr:+ cat 2019-12-05T05:10:53.090 INFO:tasks.workunit.client.0.smithi042.stderr:+ mkdir a/.snap/two 2019-12-05T05:10:53.105 INFO:tasks.workunit.client.0.smithi042.stderr:+ chmod 777 a/foo 2019-12-05T05:10:53.105 INFO:tasks.workunit.client.0.smithi042.stderr:+ sync 2019-12-05T05:10:53.170 INFO:tasks.workunit.client.0.smithi042.stderr:+ ps 2019-12-05T05:10:53.184 INFO:tasks.workunit.client.0.smithi042.stdout: PID TTY TIME CMD 2019-12-05T05:10:53.184 INFO:tasks.workunit.client.0.smithi042.stdout: 1714 ? 00:00:00 systemd 2019-12-05T05:10:53.185 INFO:tasks.workunit.client.0.smithi042.stdout: 1715 ? 00:00:00 (sd-pam) 2019-12-05T05:10:53.185 INFO:tasks.workunit.client.0.smithi042.stdout: 7572 ? 00:00:01 sshd 2019-12-05T05:10:53.185 INFO:tasks.workunit.client.0.smithi042.stdout: 18572 ? 00:00:00 timeout 2019-12-05T05:10:53.185 INFO:tasks.workunit.client.0.smithi042.stdout: 18575 ? 00:00:00 snaptest-double 2019-12-05T05:10:53.185 INFO:tasks.workunit.client.0.smithi042.stdout: 18712 ? 00:00:00 ps 2019-12-05T05:10:53.189 INFO:tasks.workunit.client.0.smithi042.stderr:+ rmdir a/.snap/one a/.snap/two 2019-12-05T05:10:53.221 INFO:tasks.workunit.client.0.smithi042.stderr:+ rm a/foo 2019-12-05T05:10:53.228 INFO:tasks.workunit.client.0.smithi042.stderr:+ rmdir a 2019-12-05T05:10:53.241 INFO:tasks.workunit.client.0.smithi042.stderr:+ mkdir a 2019-12-05T05:10:53.246 INFO:tasks.workunit.client.0.smithi042.stderr:+ mkdir a/.snap/one 2019-12-05T05:10:53.248 INFO:tasks.workunit.client.0.smithi042.stderr:+ cat 2019-12-05T05:10:53.269 INFO:tasks.workunit.client.0.smithi042.stderr:+ mkdir a/.snap/two 2019-12-05T05:10:53.284 INFO:tasks.workunit.client.0.smithi042.stderr:+ chmod 777 a/foo 2019-12-05T05:10:53.285 INFO:tasks.workunit.client.0.smithi042.stderr:+ sync 2019-12-05T05:10:53.336 INFO:tasks.workunit.client.0.smithi042.stderr:+ ps 2019-12-05T05:10:53.354 INFO:tasks.workunit.client.0.smithi042.stdout: PID TTY TIME CMD 2019-12-05T05:10:53.354 INFO:tasks.workunit.client.0.smithi042.stdout: 1714 ? 00:00:00 systemd 2019-12-05T05:10:53.354 INFO:tasks.workunit.client.0.smithi042.stdout: 1715 ? 00:00:00 (sd-pam) 2019-12-05T05:10:53.354 INFO:tasks.workunit.client.0.smithi042.stdout: 7572 ? 00:00:01 sshd 2019-12-05T05:10:53.354 INFO:tasks.workunit.client.0.smithi042.stdout: 18572 ? 00:00:00 timeout 2019-12-05T05:10:53.354 INFO:tasks.workunit.client.0.smithi042.stdout: 18575 ? 00:00:00 snaptest-double 2019-12-05T05:10:53.355 INFO:tasks.workunit.client.0.smithi042.stdout: 18722 ? 00:00:00 ps 2019-12-05T05:10:53.362 INFO:tasks.workunit.client.0.smithi042.stderr:+ rmdir a/.snap/one a/.snap/two 2019-12-05T05:10:54.242 INFO:tasks.workunit.client.0.smithi042.stderr:+ rm a/foo 2019-12-05T05:10:54.248 INFO:tasks.workunit.client.0.smithi042.stderr:+ rmdir a 2019-12-05T05:10:54.261 INFO:tasks.workunit.client.0.smithi042.stderr:+ mkdir a 2019-12-05T05:10:54.267 INFO:tasks.workunit.client.0.smithi042.stderr:+ mkdir a/.snap/one 2019-12-05T05:10:54.270 INFO:tasks.workunit.client.0.smithi042.stderr:+ cat 2019-12-05T05:10:54.289 INFO:tasks.workunit.client.0.smithi042.stderr:+ mkdir a/.snap/two 2019-12-05T05:10:54.336 INFO:tasks.workunit.client.0.smithi042.stderr:+ chmod 777 a/foo 2019-12-05T05:10:54.339 INFO:tasks.workunit.client.0.smithi042.stderr:+ sync 2019-12-05T05:10:54.472 INFO:tasks.workunit.client.0.smithi042.stderr:+ ps 2019-12-05T05:10:54.478 INFO:tasks.workunit.client.0.smithi042.stdout: PID TTY TIME CMD 2019-12-05T05:10:54.478 INFO:tasks.workunit.client.0.smithi042.stdout: 1714 ? 00:00:00 systemd 2019-12-05T05:10:54.479 INFO:tasks.workunit.client.0.smithi042.stdout: 1715 ? 00:00:00 (sd-pam) 2019-12-05T05:10:54.479 INFO:tasks.workunit.client.0.smithi042.stdout: 7572 ? 00:00:01 sshd 2019-12-05T05:10:54.479 INFO:tasks.workunit.client.0.smithi042.stdout: 18572 ? 00:00:00 timeout 2019-12-05T05:10:54.479 INFO:tasks.workunit.client.0.smithi042.stdout: 18575 ? 00:00:00 snaptest-double 2019-12-05T05:10:54.480 INFO:tasks.workunit.client.0.smithi042.stdout: 18732 ? 00:00:00 ps 2019-12-05T05:10:54.480 INFO:tasks.workunit.client.0.smithi042.stderr:+ rmdir a/.snap/one a/.snap/two 2019-12-05T05:10:54.497 INFO:tasks.workunit.client.0.smithi042.stderr:+ rm a/foo 2019-12-05T05:10:54.501 INFO:tasks.workunit.client.0.smithi042.stderr:+ rmdir a 2019-12-05T05:10:54.510 INFO:tasks.workunit.client.0.smithi042.stderr:+ mkdir a 2019-12-05T05:10:54.514 INFO:tasks.workunit.client.0.smithi042.stderr:+ mkdir a/.snap/one 2019-12-05T05:10:54.518 INFO:tasks.workunit.client.0.smithi042.stderr:+ cat 2019-12-05T05:10:54.633 INFO:tasks.workunit.client.0.smithi042.stderr:+ mkdir a/.snap/two 2019-12-05T05:10:54.656 INFO:tasks.workunit.client.0.smithi042.stderr:+ chmod 777 a/foo 2019-12-05T05:10:54.657 INFO:tasks.workunit.client.0.smithi042.stderr:+ sync 2019-12-05T05:10:54.721 INFO:tasks.workunit.client.0.smithi042.stderr:+ ps 2019-12-05T05:10:54.736 INFO:tasks.workunit.client.0.smithi042.stdout: PID TTY TIME CMD 2019-12-05T05:10:54.736 INFO:tasks.workunit.client.0.smithi042.stdout: 1714 ? 00:00:00 systemd 2019-12-05T05:10:54.736 INFO:tasks.workunit.client.0.smithi042.stdout: 1715 ? 00:00:00 (sd-pam) 2019-12-05T05:10:54.737 INFO:tasks.workunit.client.0.smithi042.stdout: 7572 ? 00:00:01 sshd 2019-12-05T05:10:54.737 INFO:tasks.workunit.client.0.smithi042.stdout: 18572 ? 00:00:00 timeout 2019-12-05T05:10:54.737 INFO:tasks.workunit.client.0.smithi042.stdout: 18575 ? 00:00:00 snaptest-double 2019-12-05T05:10:54.737 INFO:tasks.workunit.client.0.smithi042.stdout: 18742 ? 00:00:00 ps 2019-12-05T05:10:54.743 INFO:tasks.workunit.client.0.smithi042.stderr:+ rmdir a/.snap/one a/.snap/two 2019-12-05T05:10:54.771 INFO:tasks.workunit.client.0.smithi042.stderr:+ rm a/foo 2019-12-05T05:10:54.775 INFO:tasks.workunit.client.0.smithi042.stderr:+ rmdir a 2019-12-05T05:10:54.785 INFO:tasks.workunit.client.0.smithi042.stderr:+ mkdir a 2019-12-05T05:10:54.788 INFO:tasks.workunit.client.0.smithi042.stderr:+ mkdir a/.snap/one 2019-12-05T05:10:54.791 INFO:tasks.workunit.client.0.smithi042.stderr:+ cat 2019-12-05T05:10:55.359 INFO:tasks.workunit.client.0.smithi042.stderr:+ mkdir a/.snap/two 2019-12-05T05:10:56.159 INFO:tasks.workunit.client.0.smithi042.stderr:+ chmod 777 a/foo 2019-12-05T05:10:56.160 INFO:tasks.workunit.client.0.smithi042.stderr:+ sync 2019-12-05T05:10:56.245 INFO:tasks.workunit.client.0.smithi042.stderr:+ ps 2019-12-05T05:10:56.260 INFO:tasks.workunit.client.0.smithi042.stdout: PID TTY TIME CMD 2019-12-05T05:10:56.260 INFO:tasks.workunit.client.0.smithi042.stdout: 1714 ? 00:00:00 systemd 2019-12-05T05:10:56.260 INFO:tasks.workunit.client.0.smithi042.stdout: 1715 ? 00:00:00 (sd-pam) 2019-12-05T05:10:56.260 INFO:tasks.workunit.client.0.smithi042.stdout: 7572 ? 00:00:01 sshd 2019-12-05T05:10:56.261 INFO:tasks.workunit.client.0.smithi042.stdout: 18572 ? 00:00:00 timeout 2019-12-05T05:10:56.261 INFO:tasks.workunit.client.0.smithi042.stdout: 18575 ? 00:00:00 snaptest-double 2019-12-05T05:10:56.261 INFO:tasks.workunit.client.0.smithi042.stdout: 18752 ? 00:00:00 ps 2019-12-05T05:10:56.266 INFO:tasks.workunit.client.0.smithi042.stderr:+ rmdir a/.snap/one a/.snap/two 2019-12-05T05:10:56.379 INFO:tasks.workunit.client.0.smithi042.stderr:+ rm a/foo 2019-12-05T05:10:56.382 INFO:tasks.workunit.client.0.smithi042.stderr:+ rmdir a 2019-12-05T05:10:56.390 INFO:tasks.workunit.client.0.smithi042.stderr:+ mkdir a 2019-12-05T05:10:56.394 INFO:tasks.workunit.client.0.smithi042.stderr:+ mkdir a/.snap/one 2019-12-05T05:10:56.396 INFO:tasks.workunit.client.0.smithi042.stderr:+ cat 2019-12-05T05:10:57.009 INFO:tasks.workunit.client.0.smithi042.stderr:+ mkdir a/.snap/two 2019-12-05T05:10:57.029 INFO:tasks.workunit.client.0.smithi042.stderr:+ chmod 777 a/foo 2019-12-05T05:10:57.029 INFO:tasks.workunit.client.0.smithi042.stderr:+ sync 2019-12-05T05:10:57.112 INFO:tasks.workunit.client.0.smithi042.stderr:+ ps 2019-12-05T05:10:57.125 INFO:tasks.workunit.client.0.smithi042.stdout: PID TTY TIME CMD 2019-12-05T05:10:57.125 INFO:tasks.workunit.client.0.smithi042.stdout: 1714 ? 00:00:00 systemd 2019-12-05T05:10:57.126 INFO:tasks.workunit.client.0.smithi042.stdout: 1715 ? 00:00:00 (sd-pam) 2019-12-05T05:10:57.126 INFO:tasks.workunit.client.0.smithi042.stdout: 7572 ? 00:00:01 sshd 2019-12-05T05:10:57.126 INFO:tasks.workunit.client.0.smithi042.stdout: 18572 ? 00:00:00 timeout 2019-12-05T05:10:57.126 INFO:tasks.workunit.client.0.smithi042.stdout: 18575 ? 00:00:00 snaptest-double 2019-12-05T05:10:57.126 INFO:tasks.workunit.client.0.smithi042.stdout: 18762 ? 00:00:00 ps 2019-12-05T05:10:57.131 INFO:tasks.workunit.client.0.smithi042.stderr:+ rmdir a/.snap/one a/.snap/two 2019-12-05T05:10:57.161 INFO:tasks.workunit.client.0.smithi042.stderr:+ rm a/foo 2019-12-05T05:10:57.165 INFO:tasks.workunit.client.0.smithi042.stderr:+ rmdir a 2019-12-05T05:10:57.264 INFO:tasks.workunit.client.0.smithi042.stderr:+ mkdir a 2019-12-05T05:10:57.270 INFO:tasks.workunit.client.0.smithi042.stderr:+ mkdir a/.snap/one 2019-12-05T05:10:57.274 INFO:tasks.workunit.client.0.smithi042.stderr:+ cat 2019-12-05T05:10:57.295 INFO:tasks.workunit.client.0.smithi042.stderr:+ mkdir a/.snap/two 2019-12-05T05:10:57.308 INFO:tasks.workunit.client.0.smithi042.stderr:+ chmod 777 a/foo 2019-12-05T05:10:57.309 INFO:tasks.workunit.client.0.smithi042.stderr:+ sync 2019-12-05T05:10:57.354 INFO:tasks.workunit.client.0.smithi042.stderr:+ ps 2019-12-05T05:10:57.369 INFO:tasks.workunit.client.0.smithi042.stdout: PID TTY TIME CMD 2019-12-05T05:10:57.369 INFO:tasks.workunit.client.0.smithi042.stdout: 1714 ? 00:00:00 systemd 2019-12-05T05:10:57.369 INFO:tasks.workunit.client.0.smithi042.stdout: 1715 ? 00:00:00 (sd-pam) 2019-12-05T05:10:57.370 INFO:tasks.workunit.client.0.smithi042.stdout: 7572 ? 00:00:01 sshd 2019-12-05T05:10:57.370 INFO:tasks.workunit.client.0.smithi042.stdout: 18572 ? 00:00:00 timeout 2019-12-05T05:10:57.370 INFO:tasks.workunit.client.0.smithi042.stdout: 18575 ? 00:00:00 snaptest-double 2019-12-05T05:10:57.370 INFO:tasks.workunit.client.0.smithi042.stdout: 18772 ? 00:00:00 ps 2019-12-05T05:10:57.375 INFO:tasks.workunit.client.0.smithi042.stderr:+ rmdir a/.snap/one a/.snap/two 2019-12-05T05:10:57.405 INFO:tasks.workunit.client.0.smithi042.stderr:+ rm a/foo 2019-12-05T05:10:57.415 INFO:tasks.workunit.client.0.smithi042.stderr:+ rmdir a 2019-12-05T05:10:57.422 INFO:tasks.workunit.client.0.smithi042.stderr:+ mkdir a 2019-12-05T05:10:57.426 INFO:tasks.workunit.client.0.smithi042.stderr:+ mkdir a/.snap/one 2019-12-05T05:10:57.429 INFO:tasks.workunit.client.0.smithi042.stderr:+ cat 2019-12-05T05:10:57.449 INFO:tasks.workunit.client.0.smithi042.stderr:+ mkdir a/.snap/two 2019-12-05T05:10:57.463 INFO:tasks.workunit.client.0.smithi042.stderr:+ chmod 777 a/foo 2019-12-05T05:10:57.467 INFO:tasks.workunit.client.0.smithi042.stderr:+ sync 2019-12-05T05:10:57.504 INFO:tasks.workunit.client.0.smithi042.stderr:+ ps 2019-12-05T05:10:57.519 INFO:tasks.workunit.client.0.smithi042.stdout: PID TTY TIME CMD 2019-12-05T05:10:57.519 INFO:tasks.workunit.client.0.smithi042.stdout: 1714 ? 00:00:00 systemd 2019-12-05T05:10:57.519 INFO:tasks.workunit.client.0.smithi042.stdout: 1715 ? 00:00:00 (sd-pam) 2019-12-05T05:10:57.519 INFO:tasks.workunit.client.0.smithi042.stdout: 7572 ? 00:00:01 sshd 2019-12-05T05:10:57.519 INFO:tasks.workunit.client.0.smithi042.stdout: 18572 ? 00:00:00 timeout 2019-12-05T05:10:57.519 INFO:tasks.workunit.client.0.smithi042.stdout: 18575 ? 00:00:00 snaptest-double 2019-12-05T05:10:57.520 INFO:tasks.workunit.client.0.smithi042.stdout: 18782 ? 00:00:00 ps 2019-12-05T05:10:57.531 INFO:tasks.workunit.client.0.smithi042.stderr:+ rmdir a/.snap/one a/.snap/two 2019-12-05T05:10:57.555 INFO:tasks.workunit.client.0.smithi042.stderr:+ rm a/foo 2019-12-05T05:10:57.560 INFO:tasks.workunit.client.0.smithi042.stderr:+ rmdir a 2019-12-05T05:10:57.575 INFO:tasks.workunit.client.0.smithi042.stderr:+ mkdir a 2019-12-05T05:10:57.581 INFO:tasks.workunit.client.0.smithi042.stderr:+ mkdir a/.snap/one 2019-12-05T05:10:57.592 INFO:tasks.workunit.client.0.smithi042.stderr:+ cat 2019-12-05T05:10:58.035 INFO:tasks.workunit.client.0.smithi042.stderr:+ mkdir a/.snap/two 2019-12-05T05:10:58.059 INFO:tasks.workunit.client.0.smithi042.stderr:+ chmod 777 a/foo 2019-12-05T05:10:58.059 INFO:tasks.workunit.client.0.smithi042.stderr:+ sync 2019-12-05T05:10:58.125 INFO:tasks.workunit.client.0.smithi042.stderr:+ ps 2019-12-05T05:10:58.136 INFO:tasks.workunit.client.0.smithi042.stdout: PID TTY TIME CMD 2019-12-05T05:10:58.136 INFO:tasks.workunit.client.0.smithi042.stdout: 1714 ? 00:00:00 systemd 2019-12-05T05:10:58.136 INFO:tasks.workunit.client.0.smithi042.stdout: 1715 ? 00:00:00 (sd-pam) 2019-12-05T05:10:58.136 INFO:tasks.workunit.client.0.smithi042.stdout: 7572 ? 00:00:01 sshd 2019-12-05T05:10:58.137 INFO:tasks.workunit.client.0.smithi042.stdout: 18572 ? 00:00:00 timeout 2019-12-05T05:10:58.137 INFO:tasks.workunit.client.0.smithi042.stdout: 18575 ? 00:00:00 snaptest-double 2019-12-05T05:10:58.137 INFO:tasks.workunit.client.0.smithi042.stdout: 18792 ? 00:00:00 ps 2019-12-05T05:10:58.145 INFO:tasks.workunit.client.0.smithi042.stderr:+ rmdir a/.snap/one a/.snap/two 2019-12-05T05:10:58.174 INFO:tasks.workunit.client.0.smithi042.stderr:+ rm a/foo 2019-12-05T05:10:58.181 INFO:tasks.workunit.client.0.smithi042.stderr:+ rmdir a 2019-12-05T05:10:58.196 INFO:tasks.workunit.client.0.smithi042.stderr:+ mkdir a 2019-12-05T05:10:58.205 INFO:tasks.workunit.client.0.smithi042.stderr:+ mkdir a/.snap/one 2019-12-05T05:10:58.206 INFO:tasks.workunit.client.0.smithi042.stderr:+ cat 2019-12-05T05:10:59.022 INFO:tasks.workunit.client.0.smithi042.stderr:+ mkdir a/.snap/two 2019-12-05T05:10:59.036 INFO:tasks.workunit.client.0.smithi042.stderr:+ chmod 777 a/foo 2019-12-05T05:10:59.037 INFO:tasks.workunit.client.0.smithi042.stderr:+ sync 2019-12-05T05:10:59.105 INFO:tasks.workunit.client.0.smithi042.stderr:+ ps 2019-12-05T05:10:59.120 INFO:tasks.workunit.client.0.smithi042.stdout: PID TTY TIME CMD 2019-12-05T05:10:59.120 INFO:tasks.workunit.client.0.smithi042.stdout: 1714 ? 00:00:00 systemd 2019-12-05T05:10:59.120 INFO:tasks.workunit.client.0.smithi042.stdout: 1715 ? 00:00:00 (sd-pam) 2019-12-05T05:10:59.120 INFO:tasks.workunit.client.0.smithi042.stdout: 7572 ? 00:00:01 sshd 2019-12-05T05:10:59.120 INFO:tasks.workunit.client.0.smithi042.stdout: 18572 ? 00:00:00 timeout 2019-12-05T05:10:59.121 INFO:tasks.workunit.client.0.smithi042.stdout: 18575 ? 00:00:00 snaptest-double 2019-12-05T05:10:59.121 INFO:tasks.workunit.client.0.smithi042.stdout: 18802 ? 00:00:00 ps 2019-12-05T05:10:59.125 INFO:tasks.workunit.client.0.smithi042.stderr:+ rmdir a/.snap/one a/.snap/two 2019-12-05T05:10:59.154 INFO:tasks.workunit.client.0.smithi042.stderr:+ rm a/foo 2019-12-05T05:10:59.158 INFO:tasks.workunit.client.0.smithi042.stderr:+ rmdir a 2019-12-05T05:10:59.170 INFO:tasks.workunit.client.0.smithi042.stderr:+ echo OK 2019-12-05T05:10:59.171 INFO:tasks.workunit.client.0.smithi042.stdout:OK 2019-12-05T05:10:59.171 INFO:teuthology.orchestra.run:Running command with timeout 3600 2019-12-05T05:10:59.171 INFO:teuthology.orchestra.run.smithi042:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2019-12-05T05:10:59.295 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-estale.sh... 2019-12-05T05:10:59.296 INFO:teuthology.orchestra.run.smithi042:workunit test fs/snaps/snaptest-estale.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c5290805033483ba0cec165c06bde07dae5f19ec TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-estale.sh 2019-12-05T05:10:59.343 INFO:tasks.workunit.client.0.smithi042.stderr:+ ceph fs set cephfs allow_new_snaps true --yes-i-really-mean-it 2019-12-05T05:10:59.444 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:59.433+0000 7f08db970700 1 Processor -- start 2019-12-05T05:10:59.444 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:59.433+0000 7f08db970700 1 -- start start 2019-12-05T05:10:59.445 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:59.433+0000 7f08db970700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f08d40a69f0 0x7f08d40a2840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:10:59.445 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:59.433+0000 7f08db970700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f08d40a8030 0x7f08d40652f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:10:59.445 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:59.433+0000 7f08db970700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08d4065840 con 0x7f08d4144fe0 2019-12-05T05:10:59.445 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:59.433+0000 7f08db970700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08d4065b20 con 0x7f08d40a69f0 2019-12-05T05:10:59.445 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:59.433+0000 7f08db970700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08d40a25a0 con 0x7f08d40a8030 2019-12-05T05:10:59.446 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:59.433+0000 7f08d970c700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f08d40a8030 0x7f08d40652f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:10:59.446 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:59.433+0000 7f08d970c700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f08d40a8030 0x7f08d40652f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:57930/0 (socket says 172.21.15.42:57930) 2019-12-05T05:10:59.446 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:59.433+0000 7f08d970c700 1 -- 172.21.15.42:0/2217230094 learned_addr learned my addr 172.21.15.42:0/2217230094 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:10:59.446 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:59.433+0000 7f08d9f0d700 1 --2- 172.21.15.42:0/2217230094 >> v2:172.21.15.40:3300/0 conn(0x7f08d40a69f0 0x7f08d40a2840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:10:59.446 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:59.433+0000 7f08d970c700 1 -- 172.21.15.42:0/2217230094 >> v2:172.21.15.40:3300/0 conn(0x7f08d40a69f0 msgr2=0x7f08d40a2840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:10:59.447 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:59.433+0000 7f08d970c700 1 --2- 172.21.15.42:0/2217230094 >> v2:172.21.15.40:3300/0 conn(0x7f08d40a69f0 0x7f08d40a2840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:10:59.447 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:59.433+0000 7f08d970c700 1 -- 172.21.15.42:0/2217230094 >> v1:172.21.15.40:6789/0 conn(0x7f08d4144fe0 legacy=0x7f08d409c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:10:59.447 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:59.433+0000 7f08d970c700 1 -- 172.21.15.42:0/2217230094 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08d40a7b60 con 0x7f08d40a8030 2019-12-05T05:10:59.447 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:59.433+0000 7f08d970c700 1 --2- 172.21.15.42:0/2217230094 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f08d40a8030 0x7f08d40652f0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f08d0008b10 tx=0x7f08d00100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:59.447 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:59.433+0000 7f08d970c700 1 --2- 172.21.15.42:0/2217230094 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f08d40a8030 0x7f08d40652f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f08d0008b10 tx=0x7f08d00100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:59.447 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:59.433+0000 7f08d970c700 1 --2- 172.21.15.42:0/2217230094 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f08d40a8030 0x7f08d40652f0 secure :-1 s=READY pgs=83 cs=0 l=1 rx=0x7f08d0008b10 tx=0x7f08d00100b0).ready entity=mon.2 client_cookie=97571ded538112b server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:10:59.447 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:59.433+0000 7f08d970c700 1 --2- 172.21.15.42:0/2217230094 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f08d40a8030 0x7f08d40652f0 secure :-1 s=THROTTLE_DONE pgs=83 cs=0 l=1 rx=0x7f08d0008b10 tx=0x7f08d00100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:59.448 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:59.433+0000 7f08d8f0b700 1 -- 172.21.15.42:0/2217230094 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f08d000a160 con 0x7f08d40a8030 2019-12-05T05:10:59.448 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:59.433+0000 7f08d970c700 1 --2- 172.21.15.42:0/2217230094 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f08d40a8030 0x7f08d40652f0 secure :-1 s=THROTTLE_DONE pgs=83 cs=0 l=1 rx=0x7f08d0008b10 tx=0x7f08d00100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:59.448 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:59.433+0000 7f08d970c700 1 --2- 172.21.15.42:0/2217230094 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f08d40a8030 0x7f08d40652f0 secure :-1 s=THROTTLE_DONE pgs=83 cs=0 l=1 rx=0x7f08d0008b10 tx=0x7f08d00100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:59.448 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:59.433+0000 7f08d8f0b700 1 -- 172.21.15.42:0/2217230094 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08d0011a30 con 0x7f08d40a8030 2019-12-05T05:10:59.448 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:59.433+0000 7f08d8f0b700 1 -- 172.21.15.42:0/2217230094 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f08d000a550 con 0x7f08d40a8030 2019-12-05T05:10:59.448 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:59.433+0000 7f08db970700 1 -- 172.21.15.42:0/2217230094 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f08d40a8030 msgr2=0x7f08d40652f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:10:59.448 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:59.433+0000 7f08db970700 1 --2- 172.21.15.42:0/2217230094 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f08d40a8030 0x7f08d40652f0 secure :-1 s=READY pgs=83 cs=0 l=1 rx=0x7f08d0008b10 tx=0x7f08d00100b0).stop 2019-12-05T05:10:59.449 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:59.433+0000 7f08db970700 1 -- 172.21.15.42:0/2217230094 shutdown_connections 2019-12-05T05:10:59.451 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:59.433+0000 7f08db970700 1 --2- 172.21.15.42:0/2217230094 >> v2:172.21.15.40:3300/0 conn(0x7f08d40a69f0 0x7f08d40a2840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:10:59.451 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:59.433+0000 7f08db970700 1 --2- 172.21.15.42:0/2217230094 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f08d40a8030 0x7f08d40652f0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:10:59.452 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:59.433+0000 7f08db970700 1 -- 172.21.15.42:0/2217230094 shutdown_connections 2019-12-05T05:10:59.452 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:59.433+0000 7f08db970700 1 -- 172.21.15.42:0/2217230094 wait complete. 2019-12-05T05:10:59.452 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:59.433+0000 7f08db970700 1 -- 172.21.15.42:0/2217230094 >> 172.21.15.42:0/2217230094 conn(0x7f08d4059570 msgr2=0x7f08d40599a0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:10:59.452 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:59.437+0000 7f08db970700 1 Processor -- start 2019-12-05T05:10:59.452 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:59.437+0000 7f08db970700 1 -- start start 2019-12-05T05:10:59.453 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:59.437+0000 7f08db970700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f08d414f880 0x7f08d4124fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:10:59.453 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:59.437+0000 7f08db970700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08d4125520 con 0x7f08d4116c60 2019-12-05T05:10:59.453 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:59.437+0000 7f08db970700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08d4131a00 con 0x7f08d4118cd0 2019-12-05T05:10:59.453 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:59.437+0000 7f08db970700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08d4125870 con 0x7f08d414f880 2019-12-05T05:10:59.453 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:59.437+0000 7f08d9f0d700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f08d414f880 0x7f08d4124fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:10:59.454 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:59.437+0000 7f08da70e700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f08d4118cd0 0x7f08d4119100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:57512/0 (socket says 172.21.15.42:57512) 2019-12-05T05:10:59.454 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:59.437+0000 7f08da70e700 1 -- 172.21.15.42:0/2675196121 learned_addr learned my addr 172.21.15.42:0/2675196121 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:10:59.454 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:59.437+0000 7f08c67fc700 1 -- 172.21.15.42:0/2675196121 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1406326117 0 0) 0x7f08d4131a00 con 0x7f08d4118cd0 2019-12-05T05:10:59.454 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:59.437+0000 7f08c67fc700 1 -- 172.21.15.42:0/2675196121 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08bc0030b0 con 0x7f08d4118cd0 2019-12-05T05:10:59.454 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:59.437+0000 7f08d9f0d700 1 -- 172.21.15.42:0/2675196121 >> v1:172.21.15.42:6789/0 conn(0x7f08d4118cd0 legacy=0x7f08d4119100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:10:59.454 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:59.437+0000 7f08c67fc700 1 -- 172.21.15.42:0/2675196121 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1714606473 0 0) 0x7f08d4125520 con 0x7f08d4116c60 2019-12-05T05:10:59.454 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:59.437+0000 7f08d9f0d700 1 -- 172.21.15.42:0/2675196121 >> v1:172.21.15.40:6789/0 conn(0x7f08d4116c60 legacy=0x7f08d4117090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:10:59.455 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:59.437+0000 7f08d9f0d700 1 -- 172.21.15.42:0/2675196121 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08d4132c40 con 0x7f08d414f880 2019-12-05T05:10:59.455 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:59.437+0000 7f08d9f0d700 1 --2- 172.21.15.42:0/2675196121 >> v2:172.21.15.40:3300/0 conn(0x7f08d414f880 0x7f08d4124fd0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f08c8002f30 tx=0x7f08c8010a70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:59.455 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:59.437+0000 7f08d9f0d700 1 --2- 172.21.15.42:0/2675196121 >> v2:172.21.15.40:3300/0 conn(0x7f08d414f880 0x7f08d4124fd0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f08c8002f30 tx=0x7f08c8010a70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:59.455 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:59.437+0000 7f08d9f0d700 1 --2- 172.21.15.42:0/2675196121 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f08d414f880 0x7f08d4124fd0 secure :-1 s=READY pgs=329 cs=0 l=1 rx=0x7f08c8002f30 tx=0x7f08c8010a70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:10:59.455 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:59.437+0000 7f08d9f0d700 1 --2- 172.21.15.42:0/2675196121 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f08d414f880 0x7f08d4124fd0 secure :-1 s=THROTTLE_DONE pgs=329 cs=0 l=1 rx=0x7f08c8002f30 tx=0x7f08c8010a70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:59.455 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:59.437+0000 7f08c67fc700 1 -- 172.21.15.42:0/2675196121 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f08c8005920 con 0x7f08d414f880 2019-12-05T05:10:59.456 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:59.437+0000 7f08d9f0d700 1 --2- 172.21.15.42:0/2675196121 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f08d414f880 0x7f08d4124fd0 secure :-1 s=THROTTLE_DONE pgs=329 cs=0 l=1 rx=0x7f08c8002f30 tx=0x7f08c8010a70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:59.456 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:59.437+0000 7f08c67fc700 1 -- 172.21.15.42:0/2675196121 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08c8005540 con 0x7f08d414f880 2019-12-05T05:10:59.457 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:59.437+0000 7f08db970700 1 -- 172.21.15.42:0/2675196121 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08d41309f0 con 0x7f08d414f880 2019-12-05T05:10:59.457 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:59.437+0000 7f08d9f0d700 1 --2- 172.21.15.42:0/2675196121 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f08d414f880 0x7f08d4124fd0 secure :-1 s=THROTTLE_DONE pgs=329 cs=0 l=1 rx=0x7f08c8002f30 tx=0x7f08c8010a70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:59.457 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:59.437+0000 7f08c67fc700 1 -- 172.21.15.42:0/2675196121 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f08c8012520 con 0x7f08d414f880 2019-12-05T05:10:59.457 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:59.437+0000 7f08db970700 1 -- 172.21.15.42:0/2675196121 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08d4130d80 con 0x7f08d414f880 2019-12-05T05:10:59.457 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:59.437+0000 7f08d9f0d700 1 --2- 172.21.15.42:0/2675196121 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f08d414f880 0x7f08d4124fd0 secure :-1 s=THROTTLE_DONE pgs=329 cs=0 l=1 rx=0x7f08c8002f30 tx=0x7f08c8010a70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:59.457 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:59.437+0000 7f08c67fc700 1 -- 172.21.15.42:0/2675196121 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f08c8007180 con 0x7f08d414f880 2019-12-05T05:10:59.457 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:59.437+0000 7f08d9f0d700 1 --2- 172.21.15.42:0/2675196121 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f08d414f880 0x7f08d4124fd0 secure :-1 s=THROTTLE_DONE pgs=329 cs=0 l=1 rx=0x7f08c8002f30 tx=0x7f08c8010a70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:59.458 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:59.437+0000 7f08c67fc700 1 --2- 172.21.15.42:0/2675196121 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f08bc03b150 0x7f08bc03d660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:10:59.458 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:59.437+0000 7f08c67fc700 1 -- 172.21.15.42:0/2675196121 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f08c804aee0 con 0x7f08d414f880 2019-12-05T05:10:59.458 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:59.437+0000 7f08d970c700 1 --2- 172.21.15.42:0/2675196121 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f08bc03b150 0x7f08bc03d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:10:59.458 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:59.441+0000 7f08d970c700 1 --2- 172.21.15.42:0/2675196121 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f08bc03b150 0x7f08bc03d660 crc :-1 s=READY pgs=189 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:10:59.458 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:59.441+0000 7f08db970700 1 -- 172.21.15.42:0/2675196121 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08d405d2c0 con 0x7f08d414f880 2019-12-05T05:10:59.477 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:59.445+0000 7f08d9f0d700 1 --2- 172.21.15.42:0/2675196121 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f08d414f880 0x7f08d4124fd0 secure :-1 s=THROTTLE_DONE pgs=329 cs=0 l=1 rx=0x7f08c8002f30 tx=0x7f08c8010a70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:10:59.477 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:59.445+0000 7f08c67fc700 1 -- 172.21.15.42:0/2675196121 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f08c8031f20 con 0x7f08d414f880 2019-12-05T05:10:59.609 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:10:59.605+0000 7f08db970700 1 -- 172.21.15.42:0/2675196121 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"var": "allow_new_snaps", "prefix": "fs set", "fs_name": "cephfs", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f08d405ba70 con 0x7f08d414f880 2019-12-05T05:11:00.319 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:11:00.313+0000 7f08d9f0d700 1 --2- 172.21.15.42:0/2675196121 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f08d414f880 0x7f08d4124fd0 secure :-1 s=THROTTLE_DONE pgs=329 cs=0 l=1 rx=0x7f08c8002f30 tx=0x7f08c8010a70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:00.319 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:11:00.313+0000 7f08c67fc700 1 -- 172.21.15.42:0/2675196121 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"var": "allow_new_snaps", "prefix": "fs set", "fs_name": "cephfs", "val": "true", "yes_i_really_mean_it": true}]=0 enabled new snapshots v50) v1 ==== 167+0+0 (secure 0 0 0) 0x7f08c8032180 con 0x7f08d414f880 2019-12-05T05:11:00.437 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:11:00.433+0000 7f08db970700 1 -- 172.21.15.42:0/2675196121 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"var": "allow_new_snaps", "prefix": "fs set", "fs_name": "cephfs", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f08d4131450 con 0x7f08d414f880 2019-12-05T05:11:01.328 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:11:01.321+0000 7f08d9f0d700 1 --2- 172.21.15.42:0/2675196121 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f08d414f880 0x7f08d4124fd0 secure :-1 s=THROTTLE_DONE pgs=329 cs=0 l=1 rx=0x7f08c8002f30 tx=0x7f08c8010a70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:01.329 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:11:01.325+0000 7f08c67fc700 1 -- 172.21.15.42:0/2675196121 <== mon.0 v2:172.21.15.40:3300/0 8 ==== mon_command_ack([{"var": "allow_new_snaps", "prefix": "fs set", "fs_name": "cephfs", "val": "true", "yes_i_really_mean_it": true}]=0 enabled new snapshots v51) v1 ==== 167+0+0 (secure 0 0 0) 0x7f08c8032d30 con 0x7f08d414f880 2019-12-05T05:11:01.353 INFO:tasks.workunit.client.0.smithi042.stderr:enabled new snapshots 2019-12-05T05:11:01.359 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:11:01.353+0000 7f08db970700 1 -- 172.21.15.42:0/2675196121 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f08bc03b150 msgr2=0x7f08bc03d660 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:11:01.359 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:11:01.353+0000 7f08db970700 1 --2- 172.21.15.42:0/2675196121 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f08bc03b150 0x7f08bc03d660 crc :-1 s=READY pgs=189 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:11:01.360 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:11:01.353+0000 7f08db970700 1 -- 172.21.15.42:0/2675196121 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f08d414f880 msgr2=0x7f08d4124fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:11:01.360 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:11:01.353+0000 7f08db970700 1 --2- 172.21.15.42:0/2675196121 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f08d414f880 0x7f08d4124fd0 secure :-1 s=READY pgs=329 cs=0 l=1 rx=0x7f08c8002f30 tx=0x7f08c8010a70).stop 2019-12-05T05:11:01.360 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:11:01.353+0000 7f08db970700 1 -- 172.21.15.42:0/2675196121 shutdown_connections 2019-12-05T05:11:01.360 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:11:01.353+0000 7f08db970700 1 --2- 172.21.15.42:0/2675196121 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f08bc03b150 0x7f08bc03d660 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:11:01.360 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:11:01.353+0000 7f08db970700 1 --2- 172.21.15.42:0/2675196121 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f08d414f880 0x7f08d4124fd0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:11:01.361 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:11:01.353+0000 7f08db970700 1 -- 172.21.15.42:0/2675196121 shutdown_connections 2019-12-05T05:11:01.362 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:11:01.353+0000 7f08db970700 1 -- 172.21.15.42:0/2675196121 wait complete. 2019-12-05T05:11:01.362 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:11:01.353+0000 7f08db970700 1 -- 172.21.15.42:0/2675196121 >> 172.21.15.42:0/2675196121 conn(0x7f08d4144db0 msgr2=0x7f08d409c8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:11:01.368 INFO:tasks.workunit.client.0.smithi042.stderr:+ mkdir .snap/foo 2019-12-05T05:11:01.390 INFO:tasks.workunit.client.0.smithi042.stderr:+ echo We want ENOENT, not ESTALE, here. 2019-12-05T05:11:01.391 INFO:tasks.workunit.client.0.smithi042.stderr:+ seq 1 100 2019-12-05T05:11:01.391 INFO:tasks.workunit.client.0.smithi042.stdout:We want ENOENT, not ESTALE, here. 2019-12-05T05:11:01.391 INFO:tasks.workunit.client.0.smithi042.stderr:+ stat .snap/foo/1 2019-12-05T05:11:01.392 INFO:tasks.workunit.client.0.smithi042.stderr:+ grep No such file 2019-12-05T05:11:01.396 INFO:tasks.workunit.client.0.smithi042.stdout:stat: cannot stat '.snap/foo/1': No such file or directory 2019-12-05T05:11:01.397 INFO:tasks.workunit.client.0.smithi042.stderr:+ grep No such file 2019-12-05T05:11:01.397 INFO:tasks.workunit.client.0.smithi042.stderr:+ stat .snap/foo/2 2019-12-05T05:11:01.402 INFO:tasks.workunit.client.0.smithi042.stdout:stat: cannot stat '.snap/foo/2': No such file or directory 2019-12-05T05:11:01.402 INFO:tasks.workunit.client.0.smithi042.stderr:+ stat .snap/foo/3 2019-12-05T05:11:01.403 INFO:tasks.workunit.client.0.smithi042.stderr:+ grep No such file 2019-12-05T05:11:01.406 INFO:tasks.workunit.client.0.smithi042.stdout:stat: cannot stat '.snap/foo/3': No such file or directory 2019-12-05T05:11:01.407 INFO:tasks.workunit.client.0.smithi042.stderr:+ stat .snap/foo/4 2019-12-05T05:11:01.408 INFO:tasks.workunit.client.0.smithi042.stderr:+ grep No such file 2019-12-05T05:11:01.412 INFO:tasks.workunit.client.0.smithi042.stdout:stat: cannot stat '.snap/foo/4': No such file or directory 2019-12-05T05:11:01.413 INFO:tasks.workunit.client.0.smithi042.stderr:+ stat .snap/foo/5 2019-12-05T05:11:01.413 INFO:tasks.workunit.client.0.smithi042.stderr:+ grep No such file 2019-12-05T05:11:01.418 INFO:tasks.workunit.client.0.smithi042.stdout:stat: cannot stat '.snap/foo/5': No such file or directory 2019-12-05T05:11:01.419 INFO:tasks.workunit.client.0.smithi042.stderr:+ stat .snap/foo/6 2019-12-05T05:11:01.419 INFO:tasks.workunit.client.0.smithi042.stderr:+ grep No such file 2019-12-05T05:11:01.427 INFO:tasks.workunit.client.0.smithi042.stdout:stat: cannot stat '.snap/foo/6': No such file or directory 2019-12-05T05:11:01.427 INFO:tasks.workunit.client.0.smithi042.stderr:+ stat .snap/foo/7 2019-12-05T05:11:01.428 INFO:tasks.workunit.client.0.smithi042.stderr:+ grep No such file 2019-12-05T05:11:01.435 INFO:tasks.workunit.client.0.smithi042.stdout:stat: cannot stat '.snap/foo/7': No such file or directory 2019-12-05T05:11:01.436 INFO:tasks.workunit.client.0.smithi042.stderr:+ stat .snap/foo/8 2019-12-05T05:11:01.436 INFO:tasks.workunit.client.0.smithi042.stderr:+ grep No such file 2019-12-05T05:11:01.441 INFO:tasks.workunit.client.0.smithi042.stdout:stat: cannot stat '.snap/foo/8': No such file or directory 2019-12-05T05:11:01.441 INFO:tasks.workunit.client.0.smithi042.stderr:+ stat .snap/foo/9+ 2019-12-05T05:11:01.441 INFO:tasks.workunit.client.0.smithi042.stderr:grep No such file 2019-12-05T05:11:01.445 INFO:tasks.workunit.client.0.smithi042.stdout:stat: cannot stat '.snap/foo/9': No such file or directory 2019-12-05T05:11:01.446 INFO:tasks.workunit.client.0.smithi042.stderr:+ stat .snap/foo/10 2019-12-05T05:11:01.446 INFO:tasks.workunit.client.0.smithi042.stderr:+ grep No such file 2019-12-05T05:11:01.452 INFO:tasks.workunit.client.0.smithi042.stdout:stat: cannot stat '.snap/foo/10': No such file or directory 2019-12-05T05:11:01.453 INFO:tasks.workunit.client.0.smithi042.stderr:+ stat+ .snap/foo/11 2019-12-05T05:11:01.453 INFO:tasks.workunit.client.0.smithi042.stderr:grep No such file 2019-12-05T05:11:01.460 INFO:tasks.workunit.client.0.smithi042.stdout:stat: cannot stat '.snap/foo/11': No such file or directory 2019-12-05T05:11:01.461 INFO:tasks.workunit.client.0.smithi042.stderr:+ stat .snap/foo/12 2019-12-05T05:11:01.461 INFO:tasks.workunit.client.0.smithi042.stderr:+ grep No such file 2019-12-05T05:11:01.467 INFO:tasks.workunit.client.0.smithi042.stdout:stat: cannot stat '.snap/foo/12': No such file or directory 2019-12-05T05:11:01.467 INFO:tasks.workunit.client.0.smithi042.stderr:+ stat .snap/foo/13 2019-12-05T05:11:01.467 INFO:tasks.workunit.client.0.smithi042.stderr:+ grep No such file 2019-12-05T05:11:01.473 INFO:tasks.workunit.client.0.smithi042.stdout:stat: cannot stat '.snap/foo/13': No such file or directory 2019-12-05T05:11:01.474 INFO:tasks.workunit.client.0.smithi042.stderr:+ stat .snap/foo/14 2019-12-05T05:11:01.474 INFO:tasks.workunit.client.0.smithi042.stderr:+ grep No such file 2019-12-05T05:11:01.480 INFO:tasks.workunit.client.0.smithi042.stdout:stat: cannot stat '.snap/foo/14': No such file or directory 2019-12-05T05:11:01.481 INFO:tasks.workunit.client.0.smithi042.stderr:+ stat .snap/foo/15 2019-12-05T05:11:01.481 INFO:tasks.workunit.client.0.smithi042.stderr:+ grep No such file 2019-12-05T05:11:01.487 INFO:tasks.workunit.client.0.smithi042.stdout:stat: cannot stat '.snap/foo/15': No such file or directory 2019-12-05T05:11:01.488 INFO:tasks.workunit.client.0.smithi042.stderr:+ stat .snap/foo/16 2019-12-05T05:11:01.488 INFO:tasks.workunit.client.0.smithi042.stderr:+ grep No such file 2019-12-05T05:11:01.494 INFO:tasks.workunit.client.0.smithi042.stdout:stat: cannot stat '.snap/foo/16': No such file or directory 2019-12-05T05:11:01.495 INFO:tasks.workunit.client.0.smithi042.stderr:+ stat .snap/foo/17 2019-12-05T05:11:01.495 INFO:tasks.workunit.client.0.smithi042.stderr:+ grep No such file 2019-12-05T05:11:01.506 INFO:tasks.workunit.client.0.smithi042.stdout:stat: cannot stat '.snap/foo/17': No such file or directory 2019-12-05T05:11:01.513 INFO:tasks.workunit.client.0.smithi042.stderr:+ + stat .snap/foo/18 2019-12-05T05:11:01.514 INFO:tasks.workunit.client.0.smithi042.stderr:grep No such file 2019-12-05T05:11:01.514 INFO:tasks.workunit.client.0.smithi042.stdout:stat: cannot stat '.snap/foo/18': No such file or directory 2019-12-05T05:11:01.515 INFO:tasks.workunit.client.0.smithi042.stderr:+ stat .snap/foo/19 2019-12-05T05:11:01.515 INFO:tasks.workunit.client.0.smithi042.stderr:+ grep No such file 2019-12-05T05:11:01.516 INFO:tasks.workunit.client.0.smithi042.stdout:stat: cannot stat '.snap/foo/19': No such file or directory 2019-12-05T05:11:01.516 INFO:tasks.workunit.client.0.smithi042.stderr:+ stat .snap/foo/20 2019-12-05T05:11:01.516 INFO:tasks.workunit.client.0.smithi042.stderr:+ grep No such file 2019-12-05T05:11:01.523 INFO:tasks.workunit.client.0.smithi042.stdout:stat: cannot stat '.snap/foo/20': No such file or directory 2019-12-05T05:11:01.524 INFO:tasks.workunit.client.0.smithi042.stderr:+ stat .snap/foo/21 2019-12-05T05:11:01.524 INFO:tasks.workunit.client.0.smithi042.stderr:+ grep No such file 2019-12-05T05:11:01.531 INFO:tasks.workunit.client.0.smithi042.stdout:stat: cannot stat '.snap/foo/21': No such file or directory 2019-12-05T05:11:01.531 INFO:tasks.workunit.client.0.smithi042.stderr:+ stat .snap/foo/22 2019-12-05T05:11:01.532 INFO:tasks.workunit.client.0.smithi042.stderr:+ grep No such file 2019-12-05T05:11:01.538 INFO:tasks.workunit.client.0.smithi042.stdout:stat: cannot stat '.snap/foo/22': No such file or directory 2019-12-05T05:11:01.538 INFO:tasks.workunit.client.0.smithi042.stderr:+ stat .snap/foo/23 2019-12-05T05:11:01.539 INFO:tasks.workunit.client.0.smithi042.stderr:+ grep No such file 2019-12-05T05:11:01.544 INFO:tasks.workunit.client.0.smithi042.stdout:stat: cannot stat '.snap/foo/23': No such file or directory 2019-12-05T05:11:01.545 INFO:tasks.workunit.client.0.smithi042.stderr:+ stat .snap/foo/24 2019-12-05T05:11:01.545 INFO:tasks.workunit.client.0.smithi042.stderr:+ grep No such file 2019-12-05T05:11:01.551 INFO:tasks.workunit.client.0.smithi042.stdout:stat: cannot stat '.snap/foo/24': No such file or directory 2019-12-05T05:11:01.552 INFO:tasks.workunit.client.0.smithi042.stderr:+ stat .snap/foo/25 2019-12-05T05:11:01.552 INFO:tasks.workunit.client.0.smithi042.stderr:+ grep No such file 2019-12-05T05:11:01.559 INFO:tasks.workunit.client.0.smithi042.stdout:stat: cannot stat '.snap/foo/25': No such file or directory 2019-12-05T05:11:01.559 INFO:tasks.workunit.client.0.smithi042.stderr:+ stat .snap/foo/26 2019-12-05T05:11:01.559 INFO:tasks.workunit.client.0.smithi042.stderr:+ grep No such file 2019-12-05T05:11:01.564 INFO:tasks.workunit.client.0.smithi042.stdout:stat: cannot stat '.snap/foo/26': No such file or directory 2019-12-05T05:11:01.565 INFO:tasks.workunit.client.0.smithi042.stderr:+ stat .snap/foo/27 2019-12-05T05:11:01.565 INFO:tasks.workunit.client.0.smithi042.stderr:+ grep No such file 2019-12-05T05:11:01.571 INFO:tasks.workunit.client.0.smithi042.stdout:stat: cannot stat '.snap/foo/27': No such file or directory 2019-12-05T05:11:01.572 INFO:tasks.workunit.client.0.smithi042.stderr:+ stat .snap/foo/28 2019-12-05T05:11:01.572 INFO:tasks.workunit.client.0.smithi042.stderr:+ grep No such file 2019-12-05T05:11:01.578 INFO:tasks.workunit.client.0.smithi042.stdout:stat: cannot stat '.snap/foo/28': No such file or directory 2019-12-05T05:11:01.578 INFO:tasks.workunit.client.0.smithi042.stderr:+ stat .snap/foo/29 2019-12-05T05:11:01.579 INFO:tasks.workunit.client.0.smithi042.stderr:+ grep No such file 2019-12-05T05:11:01.585 INFO:tasks.workunit.client.0.smithi042.stdout:stat: cannot stat '.snap/foo/29': No such file or directory 2019-12-05T05:11:01.585 INFO:tasks.workunit.client.0.smithi042.stderr:+ stat .snap/foo/30 2019-12-05T05:11:01.586 INFO:tasks.workunit.client.0.smithi042.stderr:+ grep No such file 2019-12-05T05:11:01.593 INFO:tasks.workunit.client.0.smithi042.stdout:stat: cannot stat '.snap/foo/30': No such file or directory 2019-12-05T05:11:01.594 INFO:tasks.workunit.client.0.smithi042.stderr:+ stat .snap/foo/31 2019-12-05T05:11:01.594 INFO:tasks.workunit.client.0.smithi042.stderr:+ grep No such file 2019-12-05T05:11:01.600 INFO:tasks.workunit.client.0.smithi042.stdout:stat: cannot stat '.snap/foo/31': No such file or directory 2019-12-05T05:11:01.601 INFO:tasks.workunit.client.0.smithi042.stderr:+ stat .snap/foo/32 2019-12-05T05:11:01.601 INFO:tasks.workunit.client.0.smithi042.stderr:+ grep No such file 2019-12-05T05:11:01.607 INFO:tasks.workunit.client.0.smithi042.stdout:stat: cannot stat '.snap/foo/32': No such file or directory 2019-12-05T05:11:01.608 INFO:tasks.workunit.client.0.smithi042.stderr:+ stat .snap/foo/33 2019-12-05T05:11:01.608 INFO:tasks.workunit.client.0.smithi042.stderr:+ grep No such file 2019-12-05T05:11:01.616 INFO:tasks.workunit.client.0.smithi042.stdout:stat: cannot stat '.snap/foo/33': No such file or directory 2019-12-05T05:11:01.616 INFO:tasks.workunit.client.0.smithi042.stderr:+ stat .snap/foo/34 2019-12-05T05:11:01.616 INFO:tasks.workunit.client.0.smithi042.stderr:+ grep No such file 2019-12-05T05:11:01.624 INFO:tasks.workunit.client.0.smithi042.stdout:stat: cannot stat '.snap/foo/34': No such file or directory 2019-12-05T05:11:01.624 INFO:tasks.workunit.client.0.smithi042.stderr:+ stat .snap/foo/35 2019-12-05T05:11:01.624 INFO:tasks.workunit.client.0.smithi042.stderr:+ grep No such file 2019-12-05T05:11:01.630 INFO:tasks.workunit.client.0.smithi042.stdout:stat: cannot stat '.snap/foo/35': No such file or directory 2019-12-05T05:11:01.631 INFO:tasks.workunit.client.0.smithi042.stderr:+ stat .snap/foo/36 2019-12-05T05:11:01.631 INFO:tasks.workunit.client.0.smithi042.stderr:+ grep No such file 2019-12-05T05:11:01.637 INFO:tasks.workunit.client.0.smithi042.stdout:stat: cannot stat '.snap/foo/36': No such file or directory 2019-12-05T05:11:01.638 INFO:tasks.workunit.client.0.smithi042.stderr:+ stat .snap/foo/37 2019-12-05T05:11:01.638 INFO:tasks.workunit.client.0.smithi042.stderr:+ grep No such file 2019-12-05T05:11:01.645 INFO:tasks.workunit.client.0.smithi042.stdout:stat: cannot stat '.snap/foo/37': No such file or directory 2019-12-05T05:11:01.646 INFO:tasks.workunit.client.0.smithi042.stderr:+ stat .snap/foo/38 2019-12-05T05:11:01.647 INFO:tasks.workunit.client.0.smithi042.stderr:+ grep No such file 2019-12-05T05:11:01.658 INFO:tasks.workunit.client.0.smithi042.stdout:stat: cannot stat '.snap/foo/38': No such file or directory 2019-12-05T05:11:01.658 INFO:tasks.workunit.client.0.smithi042.stderr:+ stat .snap/foo/39 2019-12-05T05:11:01.659 INFO:tasks.workunit.client.0.smithi042.stderr:+ grep No such file 2019-12-05T05:11:01.668 INFO:tasks.workunit.client.0.smithi042.stdout:stat: cannot stat '.snap/foo/39': No such file or directory 2019-12-05T05:11:01.668 INFO:tasks.workunit.client.0.smithi042.stdout:stat: cannot stat '.snap/foo/40': No such file or directory 2019-12-05T05:11:01.668 INFO:tasks.workunit.client.0.smithi042.stderr:+ stat .snap/foo/40 2019-12-05T05:11:01.668 INFO:tasks.workunit.client.0.smithi042.stderr:+ grep No such file 2019-12-05T05:11:01.669 INFO:tasks.workunit.client.0.smithi042.stderr:+ stat .snap/foo/41 2019-12-05T05:11:01.669 INFO:tasks.workunit.client.0.smithi042.stderr:+ grep No such file 2019-12-05T05:11:01.679 INFO:tasks.workunit.client.0.smithi042.stdout:stat: cannot stat '.snap/foo/41': No such file or directory 2019-12-05T05:11:01.679 INFO:tasks.workunit.client.0.smithi042.stderr:+ stat .snap/foo/42 2019-12-05T05:11:01.679 INFO:tasks.workunit.client.0.smithi042.stderr:+ grep No such file 2019-12-05T05:11:01.682 INFO:tasks.workunit.client.0.smithi042.stdout:stat: cannot stat '.snap/foo/42': No such file or directory 2019-12-05T05:11:01.683 INFO:tasks.workunit.client.0.smithi042.stderr:+ stat .snap/foo/43 2019-12-05T05:11:01.683 INFO:tasks.workunit.client.0.smithi042.stderr:+ grep No such file 2019-12-05T05:11:01.691 INFO:tasks.workunit.client.0.smithi042.stdout:stat: cannot stat '.snap/foo/43': No such file or directory 2019-12-05T05:11:01.691 INFO:tasks.workunit.client.0.smithi042.stderr:+ stat .snap/foo/44 2019-12-05T05:11:01.691 INFO:tasks.workunit.client.0.smithi042.stderr:+ grep No such file 2019-12-05T05:11:01.697 INFO:tasks.workunit.client.0.smithi042.stdout:stat: cannot stat '.snap/foo/44': No such file or directory 2019-12-05T05:11:01.699 INFO:tasks.workunit.client.0.smithi042.stderr:+ stat .snap/foo/45 2019-12-05T05:11:01.699 INFO:tasks.workunit.client.0.smithi042.stderr:+ grep No such file 2019-12-05T05:11:01.705 INFO:tasks.workunit.client.0.smithi042.stdout:stat: cannot stat '.snap/foo/45': No such file or directory 2019-12-05T05:11:01.706 INFO:tasks.workunit.client.0.smithi042.stderr:+ stat .snap/foo/46 2019-12-05T05:11:01.706 INFO:tasks.workunit.client.0.smithi042.stderr:+ grep No such file 2019-12-05T05:11:01.713 INFO:tasks.workunit.client.0.smithi042.stdout:stat: cannot stat '.snap/foo/46': No such file or directory 2019-12-05T05:11:01.714 INFO:tasks.workunit.client.0.smithi042.stderr:+ stat+ .snap/foo/47 2019-12-05T05:11:01.714 INFO:tasks.workunit.client.0.smithi042.stderr:grep No such file 2019-12-05T05:11:01.720 INFO:tasks.workunit.client.0.smithi042.stdout:stat: cannot stat '.snap/foo/47': No such file or directory 2019-12-05T05:11:01.721 INFO:tasks.workunit.client.0.smithi042.stderr:+ stat .snap/foo/48 2019-12-05T05:11:01.721 INFO:tasks.workunit.client.0.smithi042.stderr:+ grep No such file 2019-12-05T05:11:01.741 INFO:tasks.workunit.client.0.smithi042.stdout:stat: cannot stat '.snap/foo/48': No such file or directory 2019-12-05T05:11:01.742 INFO:tasks.workunit.client.0.smithi042.stdout:stat: cannot stat '.snap/foo/49': No such file or directory 2019-12-05T05:11:01.742 INFO:tasks.workunit.client.0.smithi042.stdout:stat: cannot stat '.snap/foo/50': No such file or directory 2019-12-05T05:11:01.742 INFO:tasks.workunit.client.0.smithi042.stderr:+ stat .snap/foo/49 2019-12-05T05:11:01.742 INFO:tasks.workunit.client.0.smithi042.stderr:+ grep No such file 2019-12-05T05:11:01.742 INFO:tasks.workunit.client.0.smithi042.stderr:+ stat .snap/foo/50 2019-12-05T05:11:01.743 INFO:tasks.workunit.client.0.smithi042.stderr:+ grep No such file 2019-12-05T05:11:01.743 INFO:tasks.workunit.client.0.smithi042.stderr:+ stat .snap/foo/51 2019-12-05T05:11:01.743 INFO:tasks.workunit.client.0.smithi042.stderr:+ grep No such file 2019-12-05T05:11:01.748 INFO:tasks.workunit.client.0.smithi042.stdout:stat: cannot stat '.snap/foo/51': No such file or directory 2019-12-05T05:11:01.748 INFO:tasks.workunit.client.0.smithi042.stderr:+ stat .snap/foo/52 2019-12-05T05:11:01.749 INFO:tasks.workunit.client.0.smithi042.stderr:+ grep No such file 2019-12-05T05:11:01.753 INFO:tasks.workunit.client.0.smithi042.stdout:stat: cannot stat '.snap/foo/52': No such file or directory 2019-12-05T05:11:01.754 INFO:tasks.workunit.client.0.smithi042.stderr:+ stat .snap/foo/53 2019-12-05T05:11:01.754 INFO:tasks.workunit.client.0.smithi042.stderr:+ grep No such file 2019-12-05T05:11:01.760 INFO:tasks.workunit.client.0.smithi042.stdout:stat: cannot stat '.snap/foo/53': No such file or directory 2019-12-05T05:11:01.761 INFO:tasks.workunit.client.0.smithi042.stderr:+ stat .snap/foo/54 2019-12-05T05:11:01.761 INFO:tasks.workunit.client.0.smithi042.stderr:+ grep No such file 2019-12-05T05:11:01.766 INFO:tasks.workunit.client.0.smithi042.stdout:stat: cannot stat '.snap/foo/54': No such file or directory 2019-12-05T05:11:01.767 INFO:tasks.workunit.client.0.smithi042.stderr:+ stat .snap/foo/55 2019-12-05T05:11:01.767 INFO:tasks.workunit.client.0.smithi042.stderr:+ grep No such file 2019-12-05T05:11:01.773 INFO:tasks.workunit.client.0.smithi042.stdout:stat: cannot stat '.snap/foo/55': No such file or directory 2019-12-05T05:11:01.774 INFO:tasks.workunit.client.0.smithi042.stderr:+ stat .snap/foo/56 2019-12-05T05:11:01.774 INFO:tasks.workunit.client.0.smithi042.stderr:+ grep No such file 2019-12-05T05:11:01.782 INFO:tasks.workunit.client.0.smithi042.stdout:stat: cannot stat '.snap/foo/56': No such file or directory 2019-12-05T05:11:01.782 INFO:tasks.workunit.client.0.smithi042.stderr:+ stat .snap/foo/57 2019-12-05T05:11:01.782 INFO:tasks.workunit.client.0.smithi042.stderr:+ grep No such file 2019-12-05T05:11:01.786 INFO:tasks.workunit.client.0.smithi042.stdout:stat: cannot stat '.snap/foo/57': No such file or directory 2019-12-05T05:11:01.787 INFO:tasks.workunit.client.0.smithi042.stderr:+ stat .snap/foo/58 2019-12-05T05:11:01.787 INFO:tasks.workunit.client.0.smithi042.stderr:+ grep No such file 2019-12-05T05:11:01.794 INFO:tasks.workunit.client.0.smithi042.stdout:stat: cannot stat '.snap/foo/58': No such file or directory 2019-12-05T05:11:01.794 INFO:tasks.workunit.client.0.smithi042.stderr:+ stat .snap/foo/59 2019-12-05T05:11:01.795 INFO:tasks.workunit.client.0.smithi042.stderr:+ grep No such file 2019-12-05T05:11:01.805 INFO:tasks.workunit.client.0.smithi042.stdout:stat: cannot stat '.snap/foo/59': No such file or directory 2019-12-05T05:11:01.805 INFO:tasks.workunit.client.0.smithi042.stderr:+ stat .snap/foo/60 2019-12-05T05:11:01.806 INFO:tasks.workunit.client.0.smithi042.stderr:+ grep No such file 2019-12-05T05:11:01.808 INFO:tasks.workunit.client.0.smithi042.stdout:stat: cannot stat '.snap/foo/60': No such file or directory 2019-12-05T05:11:01.809 INFO:tasks.workunit.client.0.smithi042.stderr:+ stat .snap/foo/61 2019-12-05T05:11:01.810 INFO:tasks.workunit.client.0.smithi042.stderr:+ grep No such file 2019-12-05T05:11:01.815 INFO:tasks.workunit.client.0.smithi042.stdout:stat: cannot stat '.snap/foo/61': No such file or directory 2019-12-05T05:11:01.816 INFO:tasks.workunit.client.0.smithi042.stderr:+ stat .snap/foo/62 2019-12-05T05:11:01.816 INFO:tasks.workunit.client.0.smithi042.stderr:+ grep No such file 2019-12-05T05:11:01.822 INFO:tasks.workunit.client.0.smithi042.stdout:stat: cannot stat '.snap/foo/62': No such file or directory 2019-12-05T05:11:01.824 INFO:tasks.workunit.client.0.smithi042.stderr:+ stat .snap/foo/63 2019-12-05T05:11:01.825 INFO:tasks.workunit.client.0.smithi042.stderr:+ grep No such file 2019-12-05T05:11:01.829 INFO:tasks.workunit.client.0.smithi042.stdout:stat: cannot stat '.snap/foo/63': No such file or directory 2019-12-05T05:11:01.830 INFO:tasks.workunit.client.0.smithi042.stderr:+ stat .snap/foo/64 2019-12-05T05:11:01.830 INFO:tasks.workunit.client.0.smithi042.stderr:+ grep No such file 2019-12-05T05:11:01.836 INFO:tasks.workunit.client.0.smithi042.stdout:stat: cannot stat '.snap/foo/64': No such file or directory 2019-12-05T05:11:01.837 INFO:tasks.workunit.client.0.smithi042.stderr:+ stat .snap/foo/65 2019-12-05T05:11:01.837 INFO:tasks.workunit.client.0.smithi042.stderr:+ grep No such file 2019-12-05T05:11:01.843 INFO:tasks.workunit.client.0.smithi042.stdout:stat: cannot stat '.snap/foo/65': No such file or directory 2019-12-05T05:11:01.844 INFO:tasks.workunit.client.0.smithi042.stderr:+ stat .snap/foo/66 2019-12-05T05:11:01.844 INFO:tasks.workunit.client.0.smithi042.stderr:+ grep No such file 2019-12-05T05:11:01.850 INFO:tasks.workunit.client.0.smithi042.stdout:stat: cannot stat '.snap/foo/66': No such file or directory 2019-12-05T05:11:01.851 INFO:tasks.workunit.client.0.smithi042.stderr:+ stat .snap/foo/67 2019-12-05T05:11:01.851 INFO:tasks.workunit.client.0.smithi042.stderr:+ grep No such file 2019-12-05T05:11:01.858 INFO:tasks.workunit.client.0.smithi042.stdout:stat: cannot stat '.snap/foo/67': No such file or directory 2019-12-05T05:11:01.858 INFO:tasks.workunit.client.0.smithi042.stderr:+ + stat .snap/foo/68 2019-12-05T05:11:01.859 INFO:tasks.workunit.client.0.smithi042.stderr:grep No such file 2019-12-05T05:11:01.864 INFO:tasks.workunit.client.0.smithi042.stdout:stat: cannot stat '.snap/foo/68': No such file or directory 2019-12-05T05:11:01.866 INFO:tasks.workunit.client.0.smithi042.stderr:+ stat .snap/foo/69 2019-12-05T05:11:01.866 INFO:tasks.workunit.client.0.smithi042.stderr:+ grep No such file 2019-12-05T05:11:01.871 INFO:tasks.workunit.client.0.smithi042.stdout:stat: cannot stat '.snap/foo/69': No such file or directory 2019-12-05T05:11:01.872 INFO:tasks.workunit.client.0.smithi042.stderr:+ stat .snap/foo/70 2019-12-05T05:11:01.872 INFO:tasks.workunit.client.0.smithi042.stderr:+ grep No such file 2019-12-05T05:11:01.878 INFO:tasks.workunit.client.0.smithi042.stdout:stat: cannot stat '.snap/foo/70': No such file or directory 2019-12-05T05:11:01.879 INFO:tasks.workunit.client.0.smithi042.stderr:+ stat+ .snap/foo/71 2019-12-05T05:11:01.879 INFO:tasks.workunit.client.0.smithi042.stderr:grep No such file 2019-12-05T05:11:01.883 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:11:01.886 INFO:tasks.workunit.client.0.smithi042.stdout:stat: cannot stat '.snap/foo/71': No such file or directory 2019-12-05T05:11:01.887 INFO:tasks.workunit.client.0.smithi042.stderr:+ stat .snap/foo/72 2019-12-05T05:11:01.887 INFO:tasks.workunit.client.0.smithi042.stderr:+ grep No such file 2019-12-05T05:11:01.904 INFO:tasks.workunit.client.0.smithi042.stdout:stat: cannot stat '.snap/foo/72': No such file or directory 2019-12-05T05:11:01.905 INFO:tasks.workunit.client.0.smithi042.stdout:stat: cannot stat '.snap/foo/73': No such file or directory 2019-12-05T05:11:01.905 INFO:tasks.workunit.client.0.smithi042.stderr:+ stat .snap/foo/73 2019-12-05T05:11:01.905 INFO:tasks.workunit.client.0.smithi042.stderr:+ grep No such file 2019-12-05T05:11:01.905 INFO:tasks.workunit.client.0.smithi042.stderr:+ stat .snap/foo/74 2019-12-05T05:11:01.906 INFO:tasks.workunit.client.0.smithi042.stderr:+ grep No such file 2019-12-05T05:11:01.915 INFO:tasks.workunit.client.0.smithi042.stdout:stat: cannot stat '.snap/foo/74': No such file or directory 2019-12-05T05:11:01.915 INFO:tasks.workunit.client.0.smithi042.stderr:+ stat .snap/foo/75 2019-12-05T05:11:01.915 INFO:tasks.workunit.client.0.smithi042.stderr:+ grep No such file 2019-12-05T05:11:01.916 INFO:tasks.workunit.client.0.smithi042.stdout:stat: cannot stat '.snap/foo/75': No such file or directory 2019-12-05T05:11:01.917 INFO:tasks.workunit.client.0.smithi042.stderr:+ stat .snap/foo/76 2019-12-05T05:11:01.917 INFO:tasks.workunit.client.0.smithi042.stderr:+ grep No such file 2019-12-05T05:11:01.924 INFO:tasks.workunit.client.0.smithi042.stdout:stat: cannot stat '.snap/foo/76': No such file or directory 2019-12-05T05:11:01.924 INFO:tasks.workunit.client.0.smithi042.stderr:+ stat .snap/foo/77 2019-12-05T05:11:01.924 INFO:tasks.workunit.client.0.smithi042.stderr:+ grep No such file 2019-12-05T05:11:01.930 INFO:tasks.workunit.client.0.smithi042.stdout:stat: cannot stat '.snap/foo/77': No such file or directory 2019-12-05T05:11:01.931 INFO:tasks.workunit.client.0.smithi042.stderr:+ stat .snap/foo/78 2019-12-05T05:11:01.931 INFO:tasks.workunit.client.0.smithi042.stderr:+ grep No such file 2019-12-05T05:11:01.937 INFO:tasks.workunit.client.0.smithi042.stdout:stat: cannot stat '.snap/foo/78': No such file or directory 2019-12-05T05:11:01.937 INFO:tasks.workunit.client.0.smithi042.stderr:+ stat .snap/foo/79 2019-12-05T05:11:01.938 INFO:tasks.workunit.client.0.smithi042.stderr:+ grep No such file 2019-12-05T05:11:01.945 INFO:tasks.workunit.client.0.smithi042.stdout:stat: cannot stat '.snap/foo/79': No such file or directory 2019-12-05T05:11:01.945 INFO:tasks.workunit.client.0.smithi042.stderr:+ stat .snap/foo/80 2019-12-05T05:11:01.946 INFO:tasks.workunit.client.0.smithi042.stderr:+ grep No such file 2019-12-05T05:11:01.952 INFO:tasks.workunit.client.0.smithi042.stdout:stat: cannot stat '.snap/foo/80': No such file or directory 2019-12-05T05:11:01.952 INFO:tasks.workunit.client.0.smithi042.stderr:+ stat .snap/foo/81 2019-12-05T05:11:01.953 INFO:tasks.workunit.client.0.smithi042.stderr:+ grep No such file 2019-12-05T05:11:01.987 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:01.977+0000 7ff9b6433700 1 Processor -- start 2019-12-05T05:11:01.987 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:01.977+0000 7ff9b6433700 1 -- start start 2019-12-05T05:11:01.987 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:01.977+0000 7ff9b6433700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff9b008df80 0x7ff9b009f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:11:01.987 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:01.977+0000 7ff9b6433700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff9b009f830 0x7ff9b014efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:11:01.988 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:01.977+0000 7ff9b6433700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9b009fc60 con 0x7ff9b00a7740 2019-12-05T05:11:01.988 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:01.977+0000 7ff9b6433700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9b00a0b00 con 0x7ff9b009f830 2019-12-05T05:11:01.988 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:01.977+0000 7ff9b6433700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9b008e5b0 con 0x7ff9b008df80 2019-12-05T05:11:01.988 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:01.977+0000 7ff9b49d0700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7ff9b00a7740 0x7ff9b007e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:57528/0 (socket says 172.21.15.42:57528) 2019-12-05T05:11:01.988 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:01.977+0000 7ff9b49d0700 1 -- 172.21.15.42:0/1778497258 learned_addr learned my addr 172.21.15.42:0/1778497258 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:11:01.988 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:01.977+0000 7ff9b51d1700 1 --2- 172.21.15.42:0/1778497258 >> v2:172.21.15.42:3300/0 conn(0x7ff9b009f830 0x7ff9b014efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:11:01.989 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:01.981+0000 7ff9affff700 1 --2- 172.21.15.42:0/1778497258 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff9b008df80 0x7ff9b009f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:11:01.989 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:01.981+0000 7ff9af7fe700 1 -- 172.21.15.42:0/1778497258 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 720936844 0 0) 0x7ff9b009fc60 con 0x7ff9b00a7740 2019-12-05T05:11:01.989 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:01.981+0000 7ff9af7fe700 1 -- 172.21.15.42:0/1778497258 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9a00030b0 con 0x7ff9b00a7740 2019-12-05T05:11:01.989 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:01.981+0000 7ff9affff700 1 -- 172.21.15.42:0/1778497258 >> v2:172.21.15.42:3300/0 conn(0x7ff9b009f830 msgr2=0x7ff9b014efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:11:01.989 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:01.981+0000 7ff9affff700 1 --2- 172.21.15.42:0/1778497258 >> v2:172.21.15.42:3300/0 conn(0x7ff9b009f830 0x7ff9b014efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:11:01.989 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:01.981+0000 7ff9affff700 1 -- 172.21.15.42:0/1778497258 >> v1:172.21.15.42:6789/0 conn(0x7ff9b00a7740 legacy=0x7ff9b007e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:11:01.989 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:01.981+0000 7ff9affff700 1 -- 172.21.15.42:0/1778497258 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9b00a0710 con 0x7ff9b008df80 2019-12-05T05:11:01.990 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:01.981+0000 7ff9affff700 1 --2- 172.21.15.42:0/1778497258 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff9b008df80 0x7ff9b009f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7ff99c008790 tx=0x7ff99c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:01.990 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:01.981+0000 7ff9affff700 1 --2- 172.21.15.42:0/1778497258 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff9b008df80 0x7ff9b009f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7ff99c008790 tx=0x7ff99c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:01.990 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:01.981+0000 7ff9affff700 1 --2- 172.21.15.42:0/1778497258 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff9b008df80 0x7ff9b009f2e0 secure :-1 s=READY pgs=84 cs=0 l=1 rx=0x7ff99c008790 tx=0x7ff99c00fc70).ready entity=mon.2 client_cookie=576bdc67f6f84e55 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:11:01.990 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:01.981+0000 7ff9affff700 1 --2- 172.21.15.42:0/1778497258 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff9b008df80 0x7ff9b009f2e0 secure :-1 s=THROTTLE_DONE pgs=84 cs=0 l=1 rx=0x7ff99c008790 tx=0x7ff99c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:01.990 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:01.981+0000 7ff9affff700 1 --2- 172.21.15.42:0/1778497258 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff9b008df80 0x7ff9b009f2e0 secure :-1 s=THROTTLE_DONE pgs=84 cs=0 l=1 rx=0x7ff99c008790 tx=0x7ff99c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:01.990 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:01.981+0000 7ff9af7fe700 1 -- 172.21.15.42:0/1778497258 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff99c008bb0 con 0x7ff9b008df80 2019-12-05T05:11:01.990 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:01.981+0000 7ff9affff700 1 --2- 172.21.15.42:0/1778497258 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff9b008df80 0x7ff9b009f2e0 secure :-1 s=THROTTLE_DONE pgs=84 cs=0 l=1 rx=0x7ff99c008790 tx=0x7ff99c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:01.991 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:01.981+0000 7ff9af7fe700 1 -- 172.21.15.42:0/1778497258 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99c0087d0 con 0x7ff9b008df80 2019-12-05T05:11:01.991 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:01.981+0000 7ff9af7fe700 1 -- 172.21.15.42:0/1778497258 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff99c011860 con 0x7ff9b008df80 2019-12-05T05:11:01.991 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:01.981+0000 7ff9b6433700 1 -- 172.21.15.42:0/1778497258 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff9b008df80 msgr2=0x7ff9b009f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:11:01.991 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:01.981+0000 7ff9b6433700 1 --2- 172.21.15.42:0/1778497258 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff9b008df80 0x7ff9b009f2e0 secure :-1 s=READY pgs=84 cs=0 l=1 rx=0x7ff99c008790 tx=0x7ff99c00fc70).stop 2019-12-05T05:11:01.991 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:01.981+0000 7ff9b6433700 1 -- 172.21.15.42:0/1778497258 shutdown_connections 2019-12-05T05:11:01.991 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:01.981+0000 7ff9b6433700 1 --2- 172.21.15.42:0/1778497258 >> v2:172.21.15.42:3300/0 conn(0x7ff9b009f830 0x7ff9b014efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:11:01.991 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:01.981+0000 7ff9b6433700 1 --2- 172.21.15.42:0/1778497258 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff9b008df80 0x7ff9b009f2e0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:11:01.992 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:01.981+0000 7ff9b6433700 1 -- 172.21.15.42:0/1778497258 shutdown_connections 2019-12-05T05:11:01.992 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:01.981+0000 7ff9b6433700 1 -- 172.21.15.42:0/1778497258 wait complete. 2019-12-05T05:11:01.992 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:01.981+0000 7ff9b6433700 1 -- 172.21.15.42:0/1778497258 >> 172.21.15.42:0/1778497258 conn(0x7ff9b00aabe0 msgr2=0x7ff9b00a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:11:01.995 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:01.985+0000 7ff9b6433700 1 Processor -- start 2019-12-05T05:11:01.995 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:01.985+0000 7ff9b6433700 1 -- start start 2019-12-05T05:11:01.995 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:01.985+0000 7ff9b6433700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff9b00a9640 0x7ff9b0131500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:11:01.995 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:01.985+0000 7ff9b6433700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff9b0131a50 0x7ff9b01176c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:11:01.996 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:01.985+0000 7ff9b6433700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9b0142060 con 0x7ff9b0117c10 2019-12-05T05:11:01.996 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:01.985+0000 7ff9b6433700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9b0131260 con 0x7ff9b00a9640 2019-12-05T05:11:01.996 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:01.985+0000 7ff9b6433700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9b0142eb0 con 0x7ff9b0131a50 2019-12-05T05:11:01.996 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:01.985+0000 7ff9affff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff9b0131a50 0x7ff9b01176c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:11:01.996 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:01.985+0000 7ff9affff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff9b0131a50 0x7ff9b01176c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:57496/0 (socket says 172.21.15.42:57496) 2019-12-05T05:11:01.996 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:01.985+0000 7ff9affff700 1 -- 172.21.15.42:0/210496168 learned_addr learned my addr 172.21.15.42:0/210496168 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:11:01.997 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:01.985+0000 7ff9b49d0700 1 --2- 172.21.15.42:0/210496168 >> v2:172.21.15.40:3300/0 conn(0x7ff9b00a9640 0x7ff9b0131500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:11:01.997 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:01.985+0000 7ff9affff700 1 -- 172.21.15.42:0/210496168 >> v2:172.21.15.40:3300/0 conn(0x7ff9b00a9640 msgr2=0x7ff9b0131500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:11:01.997 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:01.985+0000 7ff9affff700 1 --2- 172.21.15.42:0/210496168 >> v2:172.21.15.40:3300/0 conn(0x7ff9b00a9640 0x7ff9b0131500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:11:01.997 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:01.985+0000 7ff9affff700 1 -- 172.21.15.42:0/210496168 >> v1:172.21.15.40:6789/0 conn(0x7ff9b0117c10 legacy=0x7ff9b0116520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:11:01.997 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:01.985+0000 7ff9affff700 1 -- 172.21.15.42:0/210496168 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9b0131e80 con 0x7ff9b0131a50 2019-12-05T05:11:01.997 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:01.985+0000 7ff9affff700 1 --2- 172.21.15.42:0/210496168 >> v2:172.21.15.42:3300/0 conn(0x7ff9b0131a50 0x7ff9b01176c0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7ff99c008790 tx=0x7ff99c011980).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:01.997 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:01.985+0000 7ff9affff700 1 --2- 172.21.15.42:0/210496168 >> v2:172.21.15.42:3300/0 conn(0x7ff9b0131a50 0x7ff9b01176c0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7ff99c008790 tx=0x7ff99c011980).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:01.998 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:01.985+0000 7ff9affff700 1 --2- 172.21.15.42:0/210496168 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff9b0131a50 0x7ff9b01176c0 secure :-1 s=READY pgs=415 cs=0 l=1 rx=0x7ff99c008790 tx=0x7ff99c011980).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:11:01.998 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:01.985+0000 7ff9affff700 1 --2- 172.21.15.42:0/210496168 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff9b0131a50 0x7ff9b01176c0 secure :-1 s=THROTTLE_DONE pgs=415 cs=0 l=1 rx=0x7ff99c008790 tx=0x7ff99c011980).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:01.998 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:01.985+0000 7ff9ad7fa700 1 -- 172.21.15.42:0/210496168 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff99c00f280 con 0x7ff9b0131a50 2019-12-05T05:11:01.998 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:01.985+0000 7ff9affff700 1 --2- 172.21.15.42:0/210496168 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff9b0131a50 0x7ff9b01176c0 secure :-1 s=THROTTLE_DONE pgs=415 cs=0 l=1 rx=0x7ff99c008790 tx=0x7ff99c011980).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:01.998 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:01.985+0000 7ff9affff700 1 --2- 172.21.15.42:0/210496168 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff9b0131a50 0x7ff9b01176c0 secure :-1 s=THROTTLE_DONE pgs=415 cs=0 l=1 rx=0x7ff99c008790 tx=0x7ff99c011980).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:01.998 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:01.989+0000 7ff9ad7fa700 1 -- 172.21.15.42:0/210496168 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99c008db0 con 0x7ff9b0131a50 2019-12-05T05:11:01.999 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:01.989+0000 7ff9b6433700 1 -- 172.21.15.42:0/210496168 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9b0118040 con 0x7ff9b0131a50 2019-12-05T05:11:01.999 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:01.989+0000 7ff9ad7fa700 1 -- 172.21.15.42:0/210496168 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff99c005540 con 0x7ff9b0131a50 2019-12-05T05:11:01.999 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:01.989+0000 7ff9b6433700 1 -- 172.21.15.42:0/210496168 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9b0116d20 con 0x7ff9b0131a50 2019-12-05T05:11:01.999 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:01.989+0000 7ff9affff700 1 --2- 172.21.15.42:0/210496168 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff9b0131a50 0x7ff9b01176c0 secure :-1 s=THROTTLE_DONE pgs=415 cs=0 l=1 rx=0x7ff99c008790 tx=0x7ff99c011980).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:01.999 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:01.989+0000 7ff9ad7fa700 1 -- 172.21.15.42:0/210496168 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7ff99c005850 con 0x7ff9b0131a50 2019-12-05T05:11:01.999 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:01.989+0000 7ff9affff700 1 --2- 172.21.15.42:0/210496168 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff9b0131a50 0x7ff9b01176c0 secure :-1 s=THROTTLE_DONE pgs=415 cs=0 l=1 rx=0x7ff99c008790 tx=0x7ff99c011980).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:02.000 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:01.989+0000 7ff9ad7fa700 1 --2- 172.21.15.42:0/210496168 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff99003adc0 0x7ff99003d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:11:02.000 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:01.989+0000 7ff9ad7fa700 1 -- 172.21.15.42:0/210496168 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6431+0+0 (secure 0 0 0) 0x7ff99c08f800 con 0x7ff9b0131a50 2019-12-05T05:11:02.000 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:01.989+0000 7ff9b49d0700 1 --2- 172.21.15.42:0/210496168 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff99003adc0 0x7ff99003d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:11:02.000 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:01.989+0000 7ff9b49d0700 1 --2- 172.21.15.42:0/210496168 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff99003adc0 0x7ff99003d2d0 crc :-1 s=READY pgs=190 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:11:02.000 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:01.993+0000 7ff9b6433700 1 -- 172.21.15.42:0/210496168 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9b0117390 con 0x7ff9b0131a50 2019-12-05T05:11:02.001 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:01.997+0000 7ff9affff700 1 --2- 172.21.15.42:0/210496168 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff9b0131a50 0x7ff9b01176c0 secure :-1 s=THROTTLE_DONE pgs=415 cs=0 l=1 rx=0x7ff99c008790 tx=0x7ff99c011980).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:02.001 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:01.997+0000 7ff9ad7fa700 1 -- 172.21.15.42:0/210496168 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7ff99c076880 con 0x7ff9b0131a50 2019-12-05T05:11:02.066 INFO:tasks.workunit.client.0.smithi042.stdout:stat: cannot stat '.snap/foo/81': No such file or directory 2019-12-05T05:11:02.067 INFO:tasks.workunit.client.0.smithi042.stdout:stat: cannot stat '.snap/foo/82': No such file or directory 2019-12-05T05:11:02.067 INFO:tasks.workunit.client.0.smithi042.stdout:stat: cannot stat '.snap/foo/83': No such file or directory 2019-12-05T05:11:02.067 INFO:tasks.workunit.client.0.smithi042.stderr:+ stat .snap/foo/82 2019-12-05T05:11:02.067 INFO:tasks.workunit.client.0.smithi042.stderr:+ grep No such file 2019-12-05T05:11:02.067 INFO:tasks.workunit.client.0.smithi042.stderr:+ stat .snap/foo/83 2019-12-05T05:11:02.067 INFO:tasks.workunit.client.0.smithi042.stderr:+ grep No such file 2019-12-05T05:11:02.068 INFO:tasks.workunit.client.0.smithi042.stderr:+ stat .snap/foo/84 2019-12-05T05:11:02.068 INFO:tasks.workunit.client.0.smithi042.stderr:+ grep No such file 2019-12-05T05:11:02.072 INFO:tasks.workunit.client.0.smithi042.stdout:stat: cannot stat '.snap/foo/84': No such file or directory 2019-12-05T05:11:02.073 INFO:tasks.workunit.client.0.smithi042.stderr:+ stat .snap/foo/85 2019-12-05T05:11:02.073 INFO:tasks.workunit.client.0.smithi042.stderr:+ grep No such file 2019-12-05T05:11:02.080 INFO:tasks.workunit.client.0.smithi042.stdout:stat: cannot stat '.snap/foo/85': No such file or directory 2019-12-05T05:11:02.080 INFO:tasks.workunit.client.0.smithi042.stderr:+ stat .snap/foo/86 2019-12-05T05:11:02.081 INFO:tasks.workunit.client.0.smithi042.stderr:+ grep No such file 2019-12-05T05:11:02.087 INFO:tasks.workunit.client.0.smithi042.stdout:stat: cannot stat '.snap/foo/86': No such file or directory 2019-12-05T05:11:02.087 INFO:tasks.workunit.client.0.smithi042.stderr:+ stat .snap/foo/87 2019-12-05T05:11:02.088 INFO:tasks.workunit.client.0.smithi042.stderr:+ grep No such file 2019-12-05T05:11:02.094 INFO:tasks.workunit.client.0.smithi042.stdout:stat: cannot stat '.snap/foo/87': No such file or directory 2019-12-05T05:11:02.095 INFO:tasks.workunit.client.0.smithi042.stderr:+ + grep No such file 2019-12-05T05:11:02.095 INFO:tasks.workunit.client.0.smithi042.stderr:stat .snap/foo/88 2019-12-05T05:11:02.101 INFO:tasks.workunit.client.0.smithi042.stdout:stat: cannot stat '.snap/foo/88': No such file or directory 2019-12-05T05:11:02.101 INFO:tasks.workunit.client.0.smithi042.stderr:+ stat .snap/foo/89 2019-12-05T05:11:02.102 INFO:tasks.workunit.client.0.smithi042.stderr:+ grep No such file 2019-12-05T05:11:02.105 INFO:tasks.workunit.client.0.smithi042.stdout:stat: cannot stat '.snap/foo/89': No such file or directory 2019-12-05T05:11:02.105 INFO:tasks.workunit.client.0.smithi042.stderr:+ + stat .snap/foo/90 2019-12-05T05:11:02.106 INFO:tasks.workunit.client.0.smithi042.stderr:grep No such file 2019-12-05T05:11:02.111 INFO:tasks.workunit.client.0.smithi042.stdout:stat: cannot stat '.snap/foo/90': No such file or directory 2019-12-05T05:11:02.111 INFO:tasks.workunit.client.0.smithi042.stderr:+ + statgrep .snap/foo/91 No such file 2019-12-05T05:11:02.112 INFO:tasks.workunit.client.0.smithi042.stderr: 2019-12-05T05:11:02.117 INFO:tasks.workunit.client.0.smithi042.stdout:stat: cannot stat '.snap/foo/91': No such file or directory 2019-12-05T05:11:02.117 INFO:tasks.workunit.client.0.smithi042.stderr:+ stat .snap/foo/92 2019-12-05T05:11:02.118 INFO:tasks.workunit.client.0.smithi042.stderr:+ grep No such file 2019-12-05T05:11:02.122 INFO:tasks.workunit.client.0.smithi042.stdout:stat: cannot stat '.snap/foo/92': No such file or directory 2019-12-05T05:11:02.123 INFO:tasks.workunit.client.0.smithi042.stderr:+ + stat .snap/foo/93 2019-12-05T05:11:02.123 INFO:tasks.workunit.client.0.smithi042.stderr:grep No such file 2019-12-05T05:11:02.128 INFO:tasks.workunit.client.0.smithi042.stdout:stat: cannot stat '.snap/foo/93': No such file or directory 2019-12-05T05:11:02.128 INFO:tasks.workunit.client.0.smithi042.stderr:+ + grep No such file 2019-12-05T05:11:02.128 INFO:tasks.workunit.client.0.smithi042.stderr:stat .snap/foo/94 2019-12-05T05:11:02.134 INFO:tasks.workunit.client.0.smithi042.stdout:stat: cannot stat '.snap/foo/94': No such file or directory 2019-12-05T05:11:02.134 INFO:tasks.workunit.client.0.smithi042.stderr:+ + stat .snap/foo/95 2019-12-05T05:11:02.134 INFO:tasks.workunit.client.0.smithi042.stderr:grep No such file 2019-12-05T05:11:02.138 INFO:tasks.workunit.client.0.smithi042.stdout:stat: cannot stat '.snap/foo/95': No such file or directory 2019-12-05T05:11:02.139 INFO:tasks.workunit.client.0.smithi042.stderr:+ + stat .snap/foo/96 2019-12-05T05:11:02.139 INFO:tasks.workunit.client.0.smithi042.stderr:grep No such file 2019-12-05T05:11:02.144 INFO:tasks.workunit.client.0.smithi042.stdout:stat: cannot stat '.snap/foo/96': No such file or directory 2019-12-05T05:11:02.145 INFO:tasks.workunit.client.0.smithi042.stderr:+ stat .snap/foo/97 2019-12-05T05:11:02.145 INFO:tasks.workunit.client.0.smithi042.stderr:+ grep No such file 2019-12-05T05:11:02.151 INFO:tasks.workunit.client.0.smithi042.stdout:stat: cannot stat '.snap/foo/97': No such file or directory 2019-12-05T05:11:02.152 INFO:tasks.workunit.client.0.smithi042.stderr:+ stat .snap/foo/98 2019-12-05T05:11:02.152 INFO:tasks.workunit.client.0.smithi042.stderr:+ grep No such file 2019-12-05T05:11:02.158 INFO:tasks.workunit.client.0.smithi042.stdout:stat: cannot stat '.snap/foo/98': No such file or directory 2019-12-05T05:11:02.159 INFO:tasks.workunit.client.0.smithi042.stderr:+ stat .snap/foo/99 2019-12-05T05:11:02.159 INFO:tasks.workunit.client.0.smithi042.stderr:+ grep No such file 2019-12-05T05:11:02.164 INFO:tasks.workunit.client.0.smithi042.stdout:stat: cannot stat '.snap/foo/99': No such file or directory 2019-12-05T05:11:02.165 INFO:tasks.workunit.client.0.smithi042.stderr:+ stat .snap/foo/100 2019-12-05T05:11:02.165 INFO:tasks.workunit.client.0.smithi042.stderr:+ grep No such file 2019-12-05T05:11:02.170 INFO:tasks.workunit.client.0.smithi042.stdout:stat: cannot stat '.snap/foo/100': No such file or directory 2019-12-05T05:11:02.171 INFO:tasks.workunit.client.0.smithi042.stderr:+ rmdir .snap/foo 2019-12-05T05:11:02.174 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:02.169+0000 7ff9b6433700 1 -- 172.21.15.42:0/210496168 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff9b005d110 con 0x7ff9b0131a50 2019-12-05T05:11:02.175 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:02.169+0000 7ff9affff700 1 --2- 172.21.15.42:0/210496168 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff9b0131a50 0x7ff9b01176c0 secure :-1 s=THROTTLE_DONE pgs=415 cs=0 l=1 rx=0x7ff99c008790 tx=0x7ff99c011980).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:02.175 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:02.169+0000 7ff9ad7fa700 1 -- 172.21.15.42:0/210496168 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) v1 ==== 94+0+2105 (secure 0 0 0) 0x7ff99c076ae0 con 0x7ff9b0131a50 2019-12-05T05:11:02.176 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 51 2019-12-05T05:11:02.176 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:11:02.176 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":51,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5215,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6835/3176146895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3176146895},{"type":"v1","addr":"172.21.15.42:6835","nonce":3176146895}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":41}],"filesystems":[{"mdsmap":{"epoch":51,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:11:01.313715+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4595},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4595":{"gid":4595,"name":"b","rank":0,"incarnation":36,"state":"up:active","state_seq":11,"addr":"172.21.15.40:6833/211584380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":211584380},{"type":"v1","addr":"172.21.15.40:6833","nonce":211584380}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:11:02.178 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:02.173+0000 7ff9b6433700 1 -- 172.21.15.42:0/210496168 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff99003adc0 msgr2=0x7ff99003d2d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:11:02.178 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:02.173+0000 7ff9b6433700 1 --2- 172.21.15.42:0/210496168 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff99003adc0 0x7ff99003d2d0 crc :-1 s=READY pgs=190 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:11:02.179 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:02.173+0000 7ff9b6433700 1 -- 172.21.15.42:0/210496168 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff9b0131a50 msgr2=0x7ff9b01176c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:11:02.179 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:02.173+0000 7ff9b6433700 1 --2- 172.21.15.42:0/210496168 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff9b0131a50 0x7ff9b01176c0 secure :-1 s=READY pgs=415 cs=0 l=1 rx=0x7ff99c008790 tx=0x7ff99c011980).stop 2019-12-05T05:11:02.180 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:02.173+0000 7ff9b6433700 1 -- 172.21.15.42:0/210496168 shutdown_connections 2019-12-05T05:11:02.180 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:02.173+0000 7ff9b6433700 1 --2- 172.21.15.42:0/210496168 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff99003adc0 0x7ff99003d2d0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:11:02.180 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:02.173+0000 7ff9b6433700 1 --2- 172.21.15.42:0/210496168 >> v2:172.21.15.40:3300/0 conn(0x7ff9b00a9640 0x7ff9b0131500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:11:02.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:02.173+0000 7ff9b6433700 1 --2- 172.21.15.42:0/210496168 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff9b0131a50 0x7ff9b01176c0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:11:02.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:02.173+0000 7ff9b6433700 1 -- 172.21.15.42:0/210496168 shutdown_connections 2019-12-05T05:11:02.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:02.173+0000 7ff9b6433700 1 -- 172.21.15.42:0/210496168 wait complete. 2019-12-05T05:11:02.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:02.173+0000 7ff9b6433700 1 -- 172.21.15.42:0/210496168 >> 172.21.15.42:0/210496168 conn(0x7ff9b009ffb0 msgr2=0x7ff9b01185d0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:11:02.191 INFO:tasks.workunit.client.0.smithi042.stderr:+ echo OK 2019-12-05T05:11:02.192 INFO:tasks.workunit.client.0.smithi042.stdout:OK 2019-12-05T05:11:02.192 INFO:teuthology.orchestra.run:Running command with timeout 3600 2019-12-05T05:11:02.193 INFO:teuthology.orchestra.run.smithi042:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2019-12-05T05:11:02.194 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.b (rank=0) 2019-12-05T05:11:02.194 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2019-12-05T05:11:02.195 INFO:teuthology.orchestra.run:waiting for 300 2019-12-05T05:11:02.209 INFO:tasks.ceph.mds.b:Stopped 2019-12-05T05:11:02.209 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:11:02.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:02.337+0000 7f757e286700 1 Processor -- start 2019-12-05T05:11:02.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:02.337+0000 7f757e286700 1 -- start start 2019-12-05T05:11:02.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:02.337+0000 7f757e286700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f75780a7740 0x7f757808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:11:02.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:02.337+0000 7f757e286700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f757807e720 0x7f757807eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:11:02.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:02.337+0000 7f757e286700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f757809f2e0 0x7f757809f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:11:02.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:02.337+0000 7f757e286700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75780a0b00 con 0x7f75780a7740 2019-12-05T05:11:02.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:02.337+0000 7f757e286700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f757808e5b0 con 0x7f757807e720 2019-12-05T05:11:02.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:02.337+0000 7f757e286700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f757808e850 con 0x7f757809f2e0 2019-12-05T05:11:02.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:02.337+0000 7f7577fff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f757807e720 0x7f757807eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:11:02.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:02.337+0000 7f7577fff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f757807e720 0x7f757807eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:57508/0 (socket says 172.21.15.42:57508) 2019-12-05T05:11:02.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:02.337+0000 7f7577fff700 1 -- 172.21.15.42:0/678768583 learned_addr learned my addr 172.21.15.42:0/678768583 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:11:02.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:02.337+0000 7f757c823700 1 --2- 172.21.15.42:0/678768583 >> v2:172.21.15.40:3300/0 conn(0x7f75780a7740 0x7f757808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:11:02.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:02.337+0000 7f757d024700 1 --2- 172.21.15.42:0/678768583 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f757809f2e0 0x7f757809f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:11:02.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:02.337+0000 7f7577fff700 1 -- 172.21.15.42:0/678768583 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f757809f2e0 msgr2=0x7f757809f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:11:02.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:02.337+0000 7f7577fff700 1 --2- 172.21.15.42:0/678768583 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f757809f2e0 0x7f757809f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:11:02.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:02.337+0000 7f7577fff700 1 -- 172.21.15.42:0/678768583 >> v2:172.21.15.40:3300/0 conn(0x7f75780a7740 msgr2=0x7f757808df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:11:02.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:02.337+0000 7f7577fff700 1 --2- 172.21.15.42:0/678768583 >> v2:172.21.15.40:3300/0 conn(0x7f75780a7740 0x7f757808df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:11:02.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:02.337+0000 7f7577fff700 1 -- 172.21.15.42:0/678768583 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f757807f0a0 con 0x7f757807e720 2019-12-05T05:11:02.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:02.337+0000 7f7577fff700 1 --2- 172.21.15.42:0/678768583 >> v2:172.21.15.42:3300/0 conn(0x7f757807e720 0x7f757807eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f75780a0f50 tx=0x7f7568010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:02.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:02.337+0000 7f7577fff700 1 --2- 172.21.15.42:0/678768583 >> v2:172.21.15.42:3300/0 conn(0x7f757807e720 0x7f757807eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f75780a0f50 tx=0x7f7568010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:02.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:02.337+0000 7f7577fff700 1 --2- 172.21.15.42:0/678768583 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f757807e720 0x7f757807eb50 secure :-1 s=READY pgs=416 cs=0 l=1 rx=0x7f75780a0f50 tx=0x7f7568010130).ready entity=mon.1 client_cookie=5775b98585f85166 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:11:02.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:02.337+0000 7f7577fff700 1 --2- 172.21.15.42:0/678768583 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f757807e720 0x7f757807eb50 secure :-1 s=THROTTLE_DONE pgs=416 cs=0 l=1 rx=0x7f75780a0f50 tx=0x7f7568010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:02.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:02.337+0000 7f75777fe700 1 -- 172.21.15.42:0/678768583 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7568010de0 con 0x7f757807e720 2019-12-05T05:11:02.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:02.337+0000 7f7577fff700 1 --2- 172.21.15.42:0/678768583 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f757807e720 0x7f757807eb50 secure :-1 s=THROTTLE_DONE pgs=416 cs=0 l=1 rx=0x7f75780a0f50 tx=0x7f7568010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:02.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:02.337+0000 7f75777fe700 1 -- 172.21.15.42:0/678768583 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7568010a00 con 0x7f757807e720 2019-12-05T05:11:02.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:02.337+0000 7f7577fff700 1 --2- 172.21.15.42:0/678768583 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f757807e720 0x7f757807eb50 secure :-1 s=THROTTLE_DONE pgs=416 cs=0 l=1 rx=0x7f75780a0f50 tx=0x7f7568010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:02.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:02.337+0000 7f757e286700 1 -- 172.21.15.42:0/678768583 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f757807e720 msgr2=0x7f757807eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:11:02.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:02.337+0000 7f75777fe700 1 -- 172.21.15.42:0/678768583 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f75680114a0 con 0x7f757807e720 2019-12-05T05:11:02.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:02.337+0000 7f757e286700 1 --2- 172.21.15.42:0/678768583 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f757807e720 0x7f757807eb50 secure :-1 s=READY pgs=416 cs=0 l=1 rx=0x7f75780a0f50 tx=0x7f7568010130).stop 2019-12-05T05:11:02.362 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:02.341+0000 7f757e286700 1 -- 172.21.15.42:0/678768583 shutdown_connections 2019-12-05T05:11:02.362 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:02.341+0000 7f757e286700 1 --2- 172.21.15.42:0/678768583 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f757809f2e0 0x7f757809f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:11:02.362 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:02.341+0000 7f757e286700 1 --2- 172.21.15.42:0/678768583 >> v2:172.21.15.40:3300/0 conn(0x7f75780a7740 0x7f757808df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:11:02.362 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:02.341+0000 7f757e286700 1 --2- 172.21.15.42:0/678768583 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f757807e720 0x7f757807eb50 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:11:02.362 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:02.341+0000 7f757e286700 1 -- 172.21.15.42:0/678768583 shutdown_connections 2019-12-05T05:11:02.362 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:02.341+0000 7f757e286700 1 -- 172.21.15.42:0/678768583 wait complete. 2019-12-05T05:11:02.363 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:02.341+0000 7f757e286700 1 -- 172.21.15.42:0/678768583 >> 172.21.15.42:0/678768583 conn(0x7f75780aabe0 msgr2=0x7f75780a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:11:02.363 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:02.341+0000 7f757e286700 1 Processor -- start 2019-12-05T05:11:02.363 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:02.341+0000 7f757e286700 1 -- start start 2019-12-05T05:11:02.363 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:02.341+0000 7f757e286700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f7578119920 0x7f757811e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:11:02.363 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:02.341+0000 7f757e286700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f757811eec0 0x7f7578122730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:11:02.363 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:02.341+0000 7f757e286700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7578116cb0 con 0x7f757811b8b0 2019-12-05T05:11:02.363 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:02.341+0000 7f757e286700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75781164d0 con 0x7f7578119920 2019-12-05T05:11:02.363 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:02.341+0000 7f757e286700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7578116770 con 0x7f757811eec0 2019-12-05T05:11:02.364 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:02.341+0000 7f757c823700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f7578119920 0x7f757811e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:11:02.364 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:02.341+0000 7f7577fff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f757811b8b0 0x7f7578121f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:57552/0 (socket says 172.21.15.42:57552) 2019-12-05T05:11:02.364 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:02.341+0000 7f7577fff700 1 -- 172.21.15.42:0/65953444 learned_addr learned my addr 172.21.15.42:0/65953444 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:11:02.364 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:02.341+0000 7f757d024700 1 --2- 172.21.15.42:0/65953444 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f757811eec0 0x7f7578122730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:11:02.364 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:02.341+0000 7f75757fa700 1 -- 172.21.15.42:0/65953444 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3198221612 0 0) 0x7f7578116cb0 con 0x7f757811b8b0 2019-12-05T05:11:02.364 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:02.341+0000 7f75757fa700 1 -- 172.21.15.42:0/65953444 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f755c0030b0 con 0x7f757811b8b0 2019-12-05T05:11:02.365 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:02.345+0000 7f757c823700 1 -- 172.21.15.42:0/65953444 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f757811eec0 msgr2=0x7f7578122730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:11:02.365 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:02.345+0000 7f757c823700 1 --2- 172.21.15.42:0/65953444 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f757811eec0 0x7f7578122730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:11:02.365 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:02.345+0000 7f757c823700 1 -- 172.21.15.42:0/65953444 >> v1:172.21.15.42:6789/0 conn(0x7f757811b8b0 legacy=0x7f7578121f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:11:02.365 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:02.345+0000 7f757c823700 1 -- 172.21.15.42:0/65953444 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f757811f2f0 con 0x7f7578119920 2019-12-05T05:11:02.365 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:02.345+0000 7f757c823700 1 --2- 172.21.15.42:0/65953444 >> v2:172.21.15.42:3300/0 conn(0x7f7578119920 0x7f757811e970 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f7578116690 tx=0x7f7560010a60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:02.365 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:02.345+0000 7f757c823700 1 --2- 172.21.15.42:0/65953444 >> v2:172.21.15.42:3300/0 conn(0x7f7578119920 0x7f757811e970 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f7578116690 tx=0x7f7560010a60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:02.365 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:02.345+0000 7f757c823700 1 --2- 172.21.15.42:0/65953444 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7578119920 0x7f757811e970 secure :-1 s=READY pgs=418 cs=0 l=1 rx=0x7f7578116690 tx=0x7f7560010a60).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:11:02.366 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:02.345+0000 7f757c823700 1 --2- 172.21.15.42:0/65953444 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7578119920 0x7f757811e970 secure :-1 s=THROTTLE_DONE pgs=418 cs=0 l=1 rx=0x7f7578116690 tx=0x7f7560010a60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:02.366 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:02.345+0000 7f75757fa700 1 -- 172.21.15.42:0/65953444 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7560012930 con 0x7f7578119920 2019-12-05T05:11:02.366 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:02.345+0000 7f757c823700 1 --2- 172.21.15.42:0/65953444 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7578119920 0x7f757811e970 secure :-1 s=THROTTLE_DONE pgs=418 cs=0 l=1 rx=0x7f7578116690 tx=0x7f7560010a60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:02.366 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:02.345+0000 7f757c823700 1 --2- 172.21.15.42:0/65953444 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7578119920 0x7f757811e970 secure :-1 s=THROTTLE_DONE pgs=418 cs=0 l=1 rx=0x7f7578116690 tx=0x7f7560010a60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:02.366 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:02.345+0000 7f75757fa700 1 -- 172.21.15.42:0/65953444 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7560012550 con 0x7f7578119920 2019-12-05T05:11:02.366 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:02.345+0000 7f757e286700 1 -- 172.21.15.42:0/65953444 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75781161c0 con 0x7f7578119920 2019-12-05T05:11:02.366 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:02.345+0000 7f75757fa700 1 -- 172.21.15.42:0/65953444 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7560005540 con 0x7f7578119920 2019-12-05T05:11:02.367 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:02.345+0000 7f757e286700 1 -- 172.21.15.42:0/65953444 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7578122c80 con 0x7f7578119920 2019-12-05T05:11:02.367 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:02.345+0000 7f757c823700 1 --2- 172.21.15.42:0/65953444 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7578119920 0x7f757811e970 secure :-1 s=THROTTLE_DONE pgs=418 cs=0 l=1 rx=0x7f7578116690 tx=0x7f7560010a60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:02.367 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:02.345+0000 7f75757fa700 1 -- 172.21.15.42:0/65953444 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f7560005810 con 0x7f7578119920 2019-12-05T05:11:02.367 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:02.345+0000 7f757c823700 1 --2- 172.21.15.42:0/65953444 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7578119920 0x7f757811e970 secure :-1 s=THROTTLE_DONE pgs=418 cs=0 l=1 rx=0x7f7578116690 tx=0x7f7560010a60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:02.369 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:02.345+0000 7f75757fa700 1 --2- 172.21.15.42:0/65953444 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f755c03b2b0 0x7f755c03d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:11:02.369 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:02.345+0000 7f75757fa700 1 -- 172.21.15.42:0/65953444 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f756004ae20 con 0x7f7578119920 2019-12-05T05:11:02.369 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:02.345+0000 7f7577fff700 1 --2- 172.21.15.42:0/65953444 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f755c03b2b0 0x7f755c03d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:11:02.369 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:02.349+0000 7f7577fff700 1 --2- 172.21.15.42:0/65953444 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f755c03b2b0 0x7f755c03d7c0 crc :-1 s=READY pgs=191 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:11:02.369 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:02.353+0000 7f757e286700 1 -- 172.21.15.42:0/65953444 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f757805d110 con 0x7f7578119920 2019-12-05T05:11:02.369 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:02.361+0000 7f757c823700 1 --2- 172.21.15.42:0/65953444 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7578119920 0x7f757811e970 secure :-1 s=THROTTLE_DONE pgs=418 cs=0 l=1 rx=0x7f7578116690 tx=0x7f7560010a60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:02.370 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:02.361+0000 7f75757fa700 1 -- 172.21.15.42:0/65953444 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f7560031e80 con 0x7f7578119920 2019-12-05T05:11:02.536 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:02.533+0000 7f757e286700 1 -- 172.21.15.42:0/65953444 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f757805b8c0 con 0x7f7578119920 2019-12-05T05:11:02.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:02.533+0000 7f757c823700 1 --2- 172.21.15.42:0/65953444 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7578119920 0x7f757811e970 secure :-1 s=THROTTLE_DONE pgs=418 cs=0 l=1 rx=0x7f7578116690 tx=0x7f7560010a60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:02.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:02.533+0000 7f75757fa700 1 -- 172.21.15.42:0/65953444 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) v1 ==== 94+0+2105 (secure 0 0 0) 0x7f75600320e0 con 0x7f7578119920 2019-12-05T05:11:02.537 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 51 2019-12-05T05:11:02.538 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:11:02.538 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":51,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5215,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6835/3176146895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3176146895},{"type":"v1","addr":"172.21.15.42:6835","nonce":3176146895}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":41}],"filesystems":[{"mdsmap":{"epoch":51,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:11:01.313715+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4595},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4595":{"gid":4595,"name":"b","rank":0,"incarnation":36,"state":"up:active","state_seq":11,"addr":"172.21.15.40:6833/211584380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":211584380},{"type":"v1","addr":"172.21.15.40:6833","nonce":211584380}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:11:02.541 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:02.533+0000 7f757e286700 1 -- 172.21.15.42:0/65953444 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f755c03b2b0 msgr2=0x7f755c03d7c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:11:02.541 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:02.533+0000 7f757e286700 1 --2- 172.21.15.42:0/65953444 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f755c03b2b0 0x7f755c03d7c0 crc :-1 s=READY pgs=191 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:11:02.542 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:02.533+0000 7f757e286700 1 -- 172.21.15.42:0/65953444 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7578119920 msgr2=0x7f757811e970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:11:02.542 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:02.533+0000 7f757e286700 1 --2- 172.21.15.42:0/65953444 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7578119920 0x7f757811e970 secure :-1 s=READY pgs=418 cs=0 l=1 rx=0x7f7578116690 tx=0x7f7560010a60).stop 2019-12-05T05:11:02.542 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:02.533+0000 7f757e286700 1 -- 172.21.15.42:0/65953444 shutdown_connections 2019-12-05T05:11:02.542 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:02.533+0000 7f757e286700 1 --2- 172.21.15.42:0/65953444 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f755c03b2b0 0x7f755c03d7c0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:11:02.542 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:02.533+0000 7f757e286700 1 --2- 172.21.15.42:0/65953444 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f757811eec0 0x7f7578122730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:11:02.543 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:02.533+0000 7f757e286700 1 --2- 172.21.15.42:0/65953444 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7578119920 0x7f757811e970 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:11:02.543 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:02.533+0000 7f757e286700 1 -- 172.21.15.42:0/65953444 shutdown_connections 2019-12-05T05:11:02.543 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:02.533+0000 7f757e286700 1 -- 172.21.15.42:0/65953444 wait complete. 2019-12-05T05:11:02.543 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:02.533+0000 7f757e286700 1 -- 172.21.15.42:0/65953444 >> 172.21.15.42:0/65953444 conn(0x7f757808fda0 msgr2=0x7f757807e740 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:11:02.551 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T05:11:04.552 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:11:04.665 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:04.657+0000 7f0a980a8700 1 Processor -- start 2019-12-05T05:11:04.666 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:04.657+0000 7f0a980a8700 1 -- start start 2019-12-05T05:11:04.666 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:04.657+0000 7f0a980a8700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f0a900a7730 0x7f0a9008df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:11:04.666 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:04.657+0000 7f0a980a8700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0a9007e710 0x7f0a9007eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:11:04.666 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:04.657+0000 7f0a980a8700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0a9009f2d0 0x7f0a9009f700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:11:04.667 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:04.657+0000 7f0a980a8700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a900a0af0 con 0x7f0a900a7730 2019-12-05T05:11:04.667 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:04.657+0000 7f0a980a8700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a9008e5a0 con 0x7f0a9009f2d0 2019-12-05T05:11:04.667 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:04.657+0000 7f0a980a8700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a9008e840 con 0x7f0a9007e710 2019-12-05T05:11:04.667 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:04.657+0000 7f0a96e46700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0a9009f2d0 0x7f0a9009f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:11:04.668 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:04.657+0000 7f0a96e46700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0a9009f2d0 0x7f0a9009f700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:57528/0 (socket says 172.21.15.42:57528) 2019-12-05T05:11:04.668 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:04.657+0000 7f0a96e46700 1 -- 172.21.15.42:0/2947519079 learned_addr learned my addr 172.21.15.42:0/2947519079 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:11:04.668 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:04.657+0000 7f0a95e44700 1 --2- 172.21.15.42:0/2947519079 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0a9007e710 0x7f0a9007eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:11:04.668 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:04.657+0000 7f0a96645700 1 --2- 172.21.15.42:0/2947519079 >> v2:172.21.15.40:3300/0 conn(0x7f0a900a7730 0x7f0a9008df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:11:04.669 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:04.657+0000 7f0a96e46700 1 -- 172.21.15.42:0/2947519079 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0a9007e710 msgr2=0x7f0a9007eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:11:04.669 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:04.657+0000 7f0a96e46700 1 --2- 172.21.15.42:0/2947519079 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0a9007e710 0x7f0a9007eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:11:04.669 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:04.657+0000 7f0a96e46700 1 -- 172.21.15.42:0/2947519079 >> v2:172.21.15.40:3300/0 conn(0x7f0a900a7730 msgr2=0x7f0a9008df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:11:04.669 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:04.657+0000 7f0a96e46700 1 --2- 172.21.15.42:0/2947519079 >> v2:172.21.15.40:3300/0 conn(0x7f0a900a7730 0x7f0a9008df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:11:04.670 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:04.657+0000 7f0a96e46700 1 -- 172.21.15.42:0/2947519079 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a9007f090 con 0x7f0a9009f2d0 2019-12-05T05:11:04.670 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:04.657+0000 7f0a96e46700 1 --2- 172.21.15.42:0/2947519079 >> v2:172.21.15.42:3300/0 conn(0x7f0a9009f2d0 0x7f0a9009f700 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f0a9008e760 tx=0x7f0a88010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:04.670 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:04.657+0000 7f0a96e46700 1 --2- 172.21.15.42:0/2947519079 >> v2:172.21.15.42:3300/0 conn(0x7f0a9009f2d0 0x7f0a9009f700 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f0a9008e760 tx=0x7f0a88010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:04.674 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:04.657+0000 7f0a96e46700 1 --2- 172.21.15.42:0/2947519079 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0a9009f2d0 0x7f0a9009f700 secure :-1 s=READY pgs=419 cs=0 l=1 rx=0x7f0a9008e760 tx=0x7f0a88010130).ready entity=mon.1 client_cookie=ce255c0834e81641 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:11:04.675 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:04.657+0000 7f0a96e46700 1 --2- 172.21.15.42:0/2947519079 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0a9009f2d0 0x7f0a9009f700 secure :-1 s=THROTTLE_DONE pgs=419 cs=0 l=1 rx=0x7f0a9008e760 tx=0x7f0a88010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:04.675 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:04.657+0000 7f0a96e46700 1 --2- 172.21.15.42:0/2947519079 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0a9009f2d0 0x7f0a9009f700 secure :-1 s=THROTTLE_DONE pgs=419 cs=0 l=1 rx=0x7f0a9008e760 tx=0x7f0a88010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:04.675 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:04.657+0000 7f0a96e46700 1 --2- 172.21.15.42:0/2947519079 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0a9009f2d0 0x7f0a9009f700 secure :-1 s=THROTTLE_DONE pgs=419 cs=0 l=1 rx=0x7f0a9008e760 tx=0x7f0a88010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:04.675 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:04.657+0000 7f0a95643700 1 -- 172.21.15.42:0/2947519079 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0a88010de0 con 0x7f0a9009f2d0 2019-12-05T05:11:04.675 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:04.657+0000 7f0a95643700 1 -- 172.21.15.42:0/2947519079 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a88010a00 con 0x7f0a9009f2d0 2019-12-05T05:11:04.675 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:04.657+0000 7f0a95643700 1 -- 172.21.15.42:0/2947519079 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0a880114a0 con 0x7f0a9009f2d0 2019-12-05T05:11:04.676 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:04.657+0000 7f0a980a8700 1 -- 172.21.15.42:0/2947519079 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0a9009f2d0 msgr2=0x7f0a9009f700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:11:04.676 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:04.657+0000 7f0a980a8700 1 --2- 172.21.15.42:0/2947519079 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0a9009f2d0 0x7f0a9009f700 secure :-1 s=READY pgs=419 cs=0 l=1 rx=0x7f0a9008e760 tx=0x7f0a88010130).stop 2019-12-05T05:11:04.676 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:04.661+0000 7f0a980a8700 1 -- 172.21.15.42:0/2947519079 shutdown_connections 2019-12-05T05:11:04.676 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:04.661+0000 7f0a980a8700 1 --2- 172.21.15.42:0/2947519079 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0a9009f2d0 0x7f0a9009f700 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:11:04.676 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:04.661+0000 7f0a980a8700 1 --2- 172.21.15.42:0/2947519079 >> v2:172.21.15.40:3300/0 conn(0x7f0a900a7730 0x7f0a9008df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:11:04.676 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:04.661+0000 7f0a980a8700 1 --2- 172.21.15.42:0/2947519079 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0a9007e710 0x7f0a9007eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:11:04.677 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:04.661+0000 7f0a980a8700 1 -- 172.21.15.42:0/2947519079 shutdown_connections 2019-12-05T05:11:04.677 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:04.661+0000 7f0a980a8700 1 -- 172.21.15.42:0/2947519079 wait complete. 2019-12-05T05:11:04.677 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:04.661+0000 7f0a980a8700 1 -- 172.21.15.42:0/2947519079 >> 172.21.15.42:0/2947519079 conn(0x7f0a900aabd0 msgr2=0x7f0a900a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:11:04.681 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:04.661+0000 7f0a980a8700 1 Processor -- start 2019-12-05T05:11:04.681 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:04.661+0000 7f0a980a8700 1 -- start start 2019-12-05T05:11:04.681 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:04.665+0000 7f0a980a8700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0a900a1500 0x7f0a901309c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:11:04.681 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:04.665+0000 7f0a980a8700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0a901236d0 0x7f0a9007cf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:11:04.682 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:04.665+0000 7f0a980a8700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a900a5d60 con 0x7f0a900a5260 2019-12-05T05:11:04.682 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:04.665+0000 7f0a980a8700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a90130190 con 0x7f0a900a1500 2019-12-05T05:11:04.682 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:04.665+0000 7f0a980a8700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a900a4d00 con 0x7f0a901236d0 2019-12-05T05:11:04.682 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:04.665+0000 7f0a96645700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0a900a1500 0x7f0a901309c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:11:04.682 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:04.665+0000 7f0a96e46700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f0a900a5260 0x7f0a9007b7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:57574/0 (socket says 172.21.15.42:57574) 2019-12-05T05:11:04.682 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:04.665+0000 7f0a96e46700 1 -- 172.21.15.42:0/2878035365 learned_addr learned my addr 172.21.15.42:0/2878035365 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:11:04.683 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:04.665+0000 7f0a95e44700 1 --2- 172.21.15.42:0/2878035365 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0a901236d0 0x7f0a9007cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:11:04.683 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:04.665+0000 7f0a96645700 1 -- 172.21.15.42:0/2878035365 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0a901236d0 msgr2=0x7f0a9007cf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:11:04.683 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:04.665+0000 7f0a96645700 1 --2- 172.21.15.42:0/2878035365 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0a901236d0 0x7f0a9007cf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:11:04.683 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:04.665+0000 7f0a82ffd700 1 -- 172.21.15.42:0/2878035365 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4093818398 0 0) 0x7f0a900a5d60 con 0x7f0a900a5260 2019-12-05T05:11:04.683 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:04.665+0000 7f0a96645700 1 -- 172.21.15.42:0/2878035365 >> v1:172.21.15.42:6789/0 conn(0x7f0a900a5260 legacy=0x7f0a9007b7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:11:04.683 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:04.665+0000 7f0a96645700 1 -- 172.21.15.42:0/2878035365 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a90130f10 con 0x7f0a900a1500 2019-12-05T05:11:04.683 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:04.665+0000 7f0a96645700 1 --2- 172.21.15.42:0/2878035365 >> v2:172.21.15.42:3300/0 conn(0x7f0a900a1500 0x7f0a901309c0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f0a90124fb0 tx=0x7f0a8c010b10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:04.684 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:04.665+0000 7f0a96645700 1 --2- 172.21.15.42:0/2878035365 >> v2:172.21.15.42:3300/0 conn(0x7f0a900a1500 0x7f0a901309c0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f0a90124fb0 tx=0x7f0a8c010b10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:04.684 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:04.665+0000 7f0a96645700 1 --2- 172.21.15.42:0/2878035365 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0a900a1500 0x7f0a901309c0 secure :-1 s=READY pgs=421 cs=0 l=1 rx=0x7f0a90124fb0 tx=0x7f0a8c010b10).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:11:04.684 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:04.665+0000 7f0a96645700 1 --2- 172.21.15.42:0/2878035365 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0a900a1500 0x7f0a901309c0 secure :-1 s=THROTTLE_DONE pgs=421 cs=0 l=1 rx=0x7f0a90124fb0 tx=0x7f0a8c010b10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:04.684 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:04.665+0000 7f0a96645700 1 --2- 172.21.15.42:0/2878035365 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0a900a1500 0x7f0a901309c0 secure :-1 s=THROTTLE_DONE pgs=421 cs=0 l=1 rx=0x7f0a90124fb0 tx=0x7f0a8c010b10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:04.684 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:04.665+0000 7f0a82ffd700 1 -- 172.21.15.42:0/2878035365 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0a8c0129e0 con 0x7f0a900a1500 2019-12-05T05:11:04.685 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:04.665+0000 7f0a96645700 1 --2- 172.21.15.42:0/2878035365 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0a900a1500 0x7f0a901309c0 secure :-1 s=THROTTLE_DONE pgs=421 cs=0 l=1 rx=0x7f0a90124fb0 tx=0x7f0a8c010b10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:04.685 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:04.665+0000 7f0a82ffd700 1 -- 172.21.15.42:0/2878035365 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a8c012600 con 0x7f0a900a1500 2019-12-05T05:11:04.685 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:04.665+0000 7f0a980a8700 1 -- 172.21.15.42:0/2878035365 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a9007d460 con 0x7f0a900a1500 2019-12-05T05:11:04.685 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:04.665+0000 7f0a82ffd700 1 -- 172.21.15.42:0/2878035365 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0a8c00e160 con 0x7f0a900a1500 2019-12-05T05:11:04.685 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:04.665+0000 7f0a980a8700 1 -- 172.21.15.42:0/2878035365 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a9007bfb0 con 0x7f0a900a1500 2019-12-05T05:11:04.685 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:04.665+0000 7f0a96645700 1 --2- 172.21.15.42:0/2878035365 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0a900a1500 0x7f0a901309c0 secure :-1 s=THROTTLE_DONE pgs=421 cs=0 l=1 rx=0x7f0a90124fb0 tx=0x7f0a8c010b10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:04.685 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:04.665+0000 7f0a82ffd700 1 -- 172.21.15.42:0/2878035365 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f0a8c007170 con 0x7f0a900a1500 2019-12-05T05:11:04.686 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:04.665+0000 7f0a96645700 1 --2- 172.21.15.42:0/2878035365 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0a900a1500 0x7f0a901309c0 secure :-1 s=THROTTLE_DONE pgs=421 cs=0 l=1 rx=0x7f0a90124fb0 tx=0x7f0a8c010b10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:04.686 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:04.669+0000 7f0a82ffd700 1 --2- 172.21.15.42:0/2878035365 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0a7803ac60 0x7f0a7803d170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:11:04.686 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:04.669+0000 7f0a82ffd700 1 -- 172.21.15.42:0/2878035365 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f0a8c04af70 con 0x7f0a900a1500 2019-12-05T05:11:04.686 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:04.669+0000 7f0a95e44700 1 --2- 172.21.15.42:0/2878035365 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0a7803ac60 0x7f0a7803d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:11:04.686 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:04.669+0000 7f0a95e44700 1 --2- 172.21.15.42:0/2878035365 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0a7803ac60 0x7f0a7803d170 crc :-1 s=READY pgs=192 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:11:04.686 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:04.673+0000 7f0a980a8700 1 -- 172.21.15.42:0/2878035365 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a9005d100 con 0x7f0a900a1500 2019-12-05T05:11:04.687 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:04.681+0000 7f0a96645700 1 --2- 172.21.15.42:0/2878035365 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0a900a1500 0x7f0a901309c0 secure :-1 s=THROTTLE_DONE pgs=421 cs=0 l=1 rx=0x7f0a90124fb0 tx=0x7f0a8c010b10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:04.687 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:04.681+0000 7f0a82ffd700 1 -- 172.21.15.42:0/2878035365 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f0a8c031ff0 con 0x7f0a900a1500 2019-12-05T05:11:04.859 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:04.853+0000 7f0a980a8700 1 -- 172.21.15.42:0/2878035365 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0a9005b8b0 con 0x7f0a900a1500 2019-12-05T05:11:04.860 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:04.857+0000 7f0a96645700 1 --2- 172.21.15.42:0/2878035365 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0a900a1500 0x7f0a901309c0 secure :-1 s=THROTTLE_DONE pgs=421 cs=0 l=1 rx=0x7f0a90124fb0 tx=0x7f0a8c010b10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:04.860 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:04.857+0000 7f0a82ffd700 1 -- 172.21.15.42:0/2878035365 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) v1 ==== 94+0+2105 (secure 0 0 0) 0x7f0a8c02f0d0 con 0x7f0a900a1500 2019-12-05T05:11:04.860 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 51 2019-12-05T05:11:04.860 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:11:04.861 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":51,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5215,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6835/3176146895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3176146895},{"type":"v1","addr":"172.21.15.42:6835","nonce":3176146895}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":41}],"filesystems":[{"mdsmap":{"epoch":51,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:11:01.313715+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4595},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4595":{"gid":4595,"name":"b","rank":0,"incarnation":36,"state":"up:active","state_seq":11,"addr":"172.21.15.40:6833/211584380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":211584380},{"type":"v1","addr":"172.21.15.40:6833","nonce":211584380}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:11:04.862 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:04.857+0000 7f0a980a8700 1 -- 172.21.15.42:0/2878035365 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0a7803ac60 msgr2=0x7f0a7803d170 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:11:04.862 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:04.857+0000 7f0a980a8700 1 --2- 172.21.15.42:0/2878035365 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0a7803ac60 0x7f0a7803d170 crc :-1 s=READY pgs=192 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:11:04.863 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:04.857+0000 7f0a980a8700 1 -- 172.21.15.42:0/2878035365 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0a900a1500 msgr2=0x7f0a901309c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:11:04.863 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:04.857+0000 7f0a980a8700 1 --2- 172.21.15.42:0/2878035365 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0a900a1500 0x7f0a901309c0 secure :-1 s=READY pgs=421 cs=0 l=1 rx=0x7f0a90124fb0 tx=0x7f0a8c010b10).stop 2019-12-05T05:11:04.863 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:04.857+0000 7f0a980a8700 1 -- 172.21.15.42:0/2878035365 shutdown_connections 2019-12-05T05:11:04.863 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:04.857+0000 7f0a980a8700 1 --2- 172.21.15.42:0/2878035365 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0a7803ac60 0x7f0a7803d170 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:11:04.863 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:04.857+0000 7f0a980a8700 1 --2- 172.21.15.42:0/2878035365 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0a900a1500 0x7f0a901309c0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:11:04.863 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:04.857+0000 7f0a980a8700 1 --2- 172.21.15.42:0/2878035365 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0a901236d0 0x7f0a9007cf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:11:04.864 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:04.857+0000 7f0a980a8700 1 -- 172.21.15.42:0/2878035365 shutdown_connections 2019-12-05T05:11:04.864 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:04.857+0000 7f0a980a8700 1 -- 172.21.15.42:0/2878035365 wait complete. 2019-12-05T05:11:04.864 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:04.857+0000 7f0a980a8700 1 -- 172.21.15.42:0/2878035365 >> 172.21.15.42:0/2878035365 conn(0x7f0a9009fa00 msgr2=0x7f0a900596f0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:11:04.871 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T05:11:05.196 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:11:05.200 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:11:06.872 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:11:06.986 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:06.973+0000 7f59d7951700 1 Processor -- start 2019-12-05T05:11:06.987 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:06.973+0000 7f59d7951700 1 -- start start 2019-12-05T05:11:06.987 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:06.977+0000 7f59d7951700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f59d00a7740 0x7f59d008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:11:06.987 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:06.977+0000 7f59d7951700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f59d007e720 0x7f59d007eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:11:06.987 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:06.977+0000 7f59d7951700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59d00a0710 con 0x7f59d009f2e0 2019-12-05T05:11:06.987 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:06.977+0000 7f59d7951700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59d00a0b00 con 0x7f59d007e720 2019-12-05T05:11:06.987 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:06.977+0000 7f59d7951700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59d008e5b0 con 0x7f59d00a7740 2019-12-05T05:11:06.988 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:06.977+0000 7f59d5eee700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f59d00a7740 0x7f59d008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:11:06.988 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:06.977+0000 7f59d5eee700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f59d00a7740 0x7f59d008df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:57542/0 (socket says 172.21.15.42:57542) 2019-12-05T05:11:06.988 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:06.977+0000 7f59d5eee700 1 -- 172.21.15.42:0/4142863125 learned_addr learned my addr 172.21.15.42:0/4142863125 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:11:06.988 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:06.977+0000 7f59d56ed700 1 --2- 172.21.15.42:0/4142863125 >> v2:172.21.15.40:3300/0 conn(0x7f59d007e720 0x7f59d007eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:11:06.988 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:06.977+0000 7f59d5eee700 1 -- 172.21.15.42:0/4142863125 >> v2:172.21.15.40:3300/0 conn(0x7f59d007e720 msgr2=0x7f59d007eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:11:06.988 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:06.977+0000 7f59d5eee700 1 --2- 172.21.15.42:0/4142863125 >> v2:172.21.15.40:3300/0 conn(0x7f59d007e720 0x7f59d007eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:11:06.988 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:06.977+0000 7f59d5eee700 1 -- 172.21.15.42:0/4142863125 >> v1:172.21.15.40:6789/0 conn(0x7f59d009f2e0 legacy=0x7f59d009f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:11:06.989 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:06.977+0000 7f59d5eee700 1 -- 172.21.15.42:0/4142863125 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59d007f0a0 con 0x7f59d00a7740 2019-12-05T05:11:06.989 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:06.977+0000 7f59d5eee700 1 --2- 172.21.15.42:0/4142863125 >> v2:172.21.15.42:3300/0 conn(0x7f59d00a7740 0x7f59d008df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f59cc002f30 tx=0x7f59cc00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:06.989 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:06.977+0000 7f59d5eee700 1 --2- 172.21.15.42:0/4142863125 >> v2:172.21.15.42:3300/0 conn(0x7f59d00a7740 0x7f59d008df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f59cc002f30 tx=0x7f59cc00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:06.989 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:06.977+0000 7f59d5eee700 1 --2- 172.21.15.42:0/4142863125 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f59d00a7740 0x7f59d008df80 secure :-1 s=READY pgs=422 cs=0 l=1 rx=0x7f59cc002f30 tx=0x7f59cc00fd30).ready entity=mon.1 client_cookie=ed83bed912c1fb7e server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:11:06.989 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:06.977+0000 7f59d5eee700 1 --2- 172.21.15.42:0/4142863125 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f59d00a7740 0x7f59d008df80 secure :-1 s=THROTTLE_DONE pgs=422 cs=0 l=1 rx=0x7f59cc002f30 tx=0x7f59cc00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:06.989 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:06.977+0000 7f59d5eee700 1 --2- 172.21.15.42:0/4142863125 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f59d00a7740 0x7f59d008df80 secure :-1 s=THROTTLE_DONE pgs=422 cs=0 l=1 rx=0x7f59cc002f30 tx=0x7f59cc00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:06.989 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:06.977+0000 7f59d5eee700 1 --2- 172.21.15.42:0/4142863125 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f59d00a7740 0x7f59d008df80 secure :-1 s=THROTTLE_DONE pgs=422 cs=0 l=1 rx=0x7f59cc002f30 tx=0x7f59cc00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:06.990 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:06.977+0000 7f59d4eec700 1 -- 172.21.15.42:0/4142863125 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f59cc011370 con 0x7f59d00a7740 2019-12-05T05:11:06.990 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:06.977+0000 7f59d4eec700 1 -- 172.21.15.42:0/4142863125 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59cc008bb0 con 0x7f59d00a7740 2019-12-05T05:11:06.990 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:06.977+0000 7f59d4eec700 1 -- 172.21.15.42:0/4142863125 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f59cc011950 con 0x7f59d00a7740 2019-12-05T05:11:06.990 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:06.977+0000 7f59d7951700 1 -- 172.21.15.42:0/4142863125 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f59d00a7740 msgr2=0x7f59d008df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:11:06.990 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:06.977+0000 7f59d7951700 1 --2- 172.21.15.42:0/4142863125 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f59d00a7740 0x7f59d008df80 secure :-1 s=READY pgs=422 cs=0 l=1 rx=0x7f59cc002f30 tx=0x7f59cc00fd30).stop 2019-12-05T05:11:06.990 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:06.981+0000 7f59d7951700 1 -- 172.21.15.42:0/4142863125 shutdown_connections 2019-12-05T05:11:06.990 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:06.981+0000 7f59d7951700 1 --2- 172.21.15.42:0/4142863125 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f59d00a7740 0x7f59d008df80 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:11:06.991 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:06.981+0000 7f59d7951700 1 --2- 172.21.15.42:0/4142863125 >> v2:172.21.15.40:3300/0 conn(0x7f59d007e720 0x7f59d007eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:11:06.991 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:06.981+0000 7f59d7951700 1 -- 172.21.15.42:0/4142863125 shutdown_connections 2019-12-05T05:11:06.991 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:06.981+0000 7f59d7951700 1 -- 172.21.15.42:0/4142863125 wait complete. 2019-12-05T05:11:06.991 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:06.981+0000 7f59d7951700 1 -- 172.21.15.42:0/4142863125 >> 172.21.15.42:0/4142863125 conn(0x7f59d00aabe0 msgr2=0x7f59d00a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:11:06.995 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:06.985+0000 7f59d7951700 1 Processor -- start 2019-12-05T05:11:06.995 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:06.985+0000 7f59d7951700 1 -- start start 2019-12-05T05:11:06.995 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:06.985+0000 7f59d7951700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f59d011d600 0x7f59d011da30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:11:06.996 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:06.985+0000 7f59d7951700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59d0126d50 con 0x7f59d01278b0 2019-12-05T05:11:06.996 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:06.985+0000 7f59d7951700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59d01301c0 con 0x7f59d0142cb0 2019-12-05T05:11:06.996 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:06.985+0000 7f59d7951700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59d01270d0 con 0x7f59d011d600 2019-12-05T05:11:06.996 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:06.985+0000 7f59d5eee700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f59d0142cb0 0x7f59d0123790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:57584/0 (socket says 172.21.15.42:57584) 2019-12-05T05:11:06.996 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:06.985+0000 7f59d5eee700 1 -- 172.21.15.42:0/3910263625 learned_addr learned my addr 172.21.15.42:0/3910263625 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:11:06.996 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:06.985+0000 7f59d66ef700 1 --2- 172.21.15.42:0/3910263625 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f59d011d600 0x7f59d011da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:11:06.997 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:06.985+0000 7f59c67fc700 1 -- 172.21.15.42:0/3910263625 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3977032672 0 0) 0x7f59d01301c0 con 0x7f59d0142cb0 2019-12-05T05:11:06.997 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:06.985+0000 7f59c67fc700 1 -- 172.21.15.42:0/3910263625 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59b80030b0 con 0x7f59d0142cb0 2019-12-05T05:11:06.997 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:06.985+0000 7f59c67fc700 1 -- 172.21.15.42:0/3910263625 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3092216332 0 0) 0x7f59d0126d50 con 0x7f59d01278b0 2019-12-05T05:11:06.997 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:06.985+0000 7f59c67fc700 1 -- 172.21.15.42:0/3910263625 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59d01301c0 con 0x7f59d01278b0 2019-12-05T05:11:06.997 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:06.985+0000 7f59c67fc700 1 -- 172.21.15.42:0/3910263625 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 3615241209 0 0) 0x7f59b80030b0 con 0x7f59d0142cb0 2019-12-05T05:11:06.998 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:06.985+0000 7f59c67fc700 1 -- 172.21.15.42:0/3910263625 --> v1:172.21.15.42:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f59d0126d50 con 0x7f59d0142cb0 2019-12-05T05:11:06.998 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:06.985+0000 7f59c67fc700 1 -- 172.21.15.42:0/3910263625 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7f59cc011370 con 0x7f59d0142cb0 2019-12-05T05:11:06.998 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:06.985+0000 7f59c67fc700 1 -- 172.21.15.42:0/3910263625 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f59d011d600 msgr2=0x7f59d011da30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:11:06.998 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:06.985+0000 7f59c67fc700 1 --2- 172.21.15.42:0/3910263625 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f59d011d600 0x7f59d011da30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:11:06.998 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:06.985+0000 7f59c67fc700 1 -- 172.21.15.42:0/3910263625 >> v1:172.21.15.42:6789/0 conn(0x7f59d0142cb0 legacy=0x7f59d0123790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:11:06.998 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:06.985+0000 7f59c67fc700 1 -- 172.21.15.42:0/3910263625 >> v1:172.21.15.40:6789/0 conn(0x7f59d01278b0 legacy=0x7f59d00a32a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:11:06.998 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:06.985+0000 7f59c67fc700 1 --2- 172.21.15.42:0/3910263625 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f59b8003ab0 0x7f59b8004420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:11:06.999 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:06.985+0000 7f59c67fc700 1 --2- 172.21.15.42:0/3910263625 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f59b8003ee0 0x7f59b8004970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:11:06.999 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:06.985+0000 7f59c67fc700 1 --2- 172.21.15.42:0/3910263625 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f59b8004f60 0x7f59b800b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:11:06.999 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:06.985+0000 7f59c67fc700 1 -- 172.21.15.42:0/3910263625 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59d01270d0 con 0x7f59b8004f60 2019-12-05T05:11:06.999 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:06.985+0000 7f59c67fc700 1 -- 172.21.15.42:0/3910263625 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59b800bcf0 con 0x7f59b8003ee0 2019-12-05T05:11:06.999 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:06.985+0000 7f59c67fc700 1 -- 172.21.15.42:0/3910263625 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59b800bf30 con 0x7f59b8003ab0 2019-12-05T05:11:06.999 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:06.985+0000 7f59d56ed700 1 --2- 172.21.15.42:0/3910263625 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f59b8003ee0 0x7f59b8004970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:11:06.999 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:06.985+0000 7f59d66ef700 1 --2- 172.21.15.42:0/3910263625 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f59b8004f60 0x7f59b800b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:11:07.000 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:06.985+0000 7f59d5eee700 1 --2- 172.21.15.42:0/3910263625 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f59b8003ab0 0x7f59b8004420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:11:07.000 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:06.989+0000 7f59d56ed700 1 -- 172.21.15.42:0/3910263625 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f59b8003ab0 msgr2=0x7f59b8004420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:11:07.000 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:06.989+0000 7f59d56ed700 1 --2- 172.21.15.42:0/3910263625 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f59b8003ab0 0x7f59b8004420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:11:07.000 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:06.989+0000 7f59d56ed700 1 -- 172.21.15.42:0/3910263625 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f59b8004f60 msgr2=0x7f59b800b670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:11:07.000 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:06.989+0000 7f59d56ed700 1 --2- 172.21.15.42:0/3910263625 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f59b8004f60 0x7f59b800b670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:11:07.000 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:06.989+0000 7f59d56ed700 1 -- 172.21.15.42:0/3910263625 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59d0131400 con 0x7f59b8003ee0 2019-12-05T05:11:07.000 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:06.989+0000 7f59d7951700 1 -- 172.21.15.42:0/3910263625 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59d011d600 con 0x7f59b8003ee0 2019-12-05T05:11:07.001 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:06.989+0000 7f59d56ed700 1 --2- 172.21.15.42:0/3910263625 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f59b8003ee0 0x7f59b8004970 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f59c0007e90 tx=0x7f59c0010e40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:07.001 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:06.989+0000 7f59d7951700 1 -- 172.21.15.42:0/3910263625 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59d011d990 con 0x7f59b8003ee0 2019-12-05T05:11:07.001 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:06.989+0000 7f59d56ed700 1 --2- 172.21.15.42:0/3910263625 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f59b8003ee0 0x7f59b8004970 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f59c0007e90 tx=0x7f59c0010e40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:07.001 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:06.989+0000 7f59d56ed700 1 --2- 172.21.15.42:0/3910263625 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f59b8003ee0 0x7f59b8004970 secure :-1 s=READY pgs=424 cs=0 l=1 rx=0x7f59c0007e90 tx=0x7f59c0010e40).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:11:07.001 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:06.989+0000 7f59d56ed700 1 --2- 172.21.15.42:0/3910263625 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f59b8003ee0 0x7f59b8004970 secure :-1 s=THROTTLE_DONE pgs=424 cs=0 l=1 rx=0x7f59c0007e90 tx=0x7f59c0010e40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:07.001 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:06.989+0000 7f59d56ed700 1 --2- 172.21.15.42:0/3910263625 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f59b8003ee0 0x7f59b8004970 secure :-1 s=THROTTLE_DONE pgs=424 cs=0 l=1 rx=0x7f59c0007e90 tx=0x7f59c0010e40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:07.001 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:06.989+0000 7f59c67fc700 1 -- 172.21.15.42:0/3910263625 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f59c000eb70 con 0x7f59b8003ee0 2019-12-05T05:11:07.002 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:06.989+0000 7f59d56ed700 1 --2- 172.21.15.42:0/3910263625 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f59b8003ee0 0x7f59b8004970 secure :-1 s=THROTTLE_DONE pgs=424 cs=0 l=1 rx=0x7f59c0007e90 tx=0x7f59c0010e40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:07.002 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:06.989+0000 7f59c67fc700 1 -- 172.21.15.42:0/3910263625 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59c000e790 con 0x7f59b8003ee0 2019-12-05T05:11:07.002 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:06.989+0000 7f59c67fc700 1 -- 172.21.15.42:0/3910263625 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f59c0011680 con 0x7f59b8003ee0 2019-12-05T05:11:07.002 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:06.989+0000 7f59d56ed700 1 --2- 172.21.15.42:0/3910263625 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f59b8003ee0 0x7f59b8004970 secure :-1 s=THROTTLE_DONE pgs=424 cs=0 l=1 rx=0x7f59c0007e90 tx=0x7f59c0010e40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:07.002 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:06.989+0000 7f59c67fc700 1 -- 172.21.15.42:0/3910263625 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f59c0011880 con 0x7f59b8003ee0 2019-12-05T05:11:07.002 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:06.989+0000 7f59d56ed700 1 --2- 172.21.15.42:0/3910263625 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f59b8003ee0 0x7f59b8004970 secure :-1 s=THROTTLE_DONE pgs=424 cs=0 l=1 rx=0x7f59c0007e90 tx=0x7f59c0010e40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:07.002 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:06.989+0000 7f59c67fc700 1 --2- 172.21.15.42:0/3910263625 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f59b8046030 0x7f59b8048540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:11:07.003 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:06.989+0000 7f59c67fc700 1 -- 172.21.15.42:0/3910263625 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f59c004b0f0 con 0x7f59b8003ee0 2019-12-05T05:11:07.003 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:06.989+0000 7f59d5eee700 1 --2- 172.21.15.42:0/3910263625 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f59b8046030 0x7f59b8048540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:11:07.003 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:06.989+0000 7f59d5eee700 1 --2- 172.21.15.42:0/3910263625 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f59b8046030 0x7f59b8048540 crc :-1 s=READY pgs=193 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:11:07.003 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:06.989+0000 7f59d7951700 1 -- 172.21.15.42:0/3910263625 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59ac003200 con 0x7f59b8003ee0 2019-12-05T05:11:07.004 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:06.993+0000 7f59d56ed700 1 --2- 172.21.15.42:0/3910263625 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f59b8003ee0 0x7f59b8004970 secure :-1 s=THROTTLE_DONE pgs=424 cs=0 l=1 rx=0x7f59c0007e90 tx=0x7f59c0010e40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:07.004 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:06.993+0000 7f59c67fc700 1 -- 172.21.15.42:0/3910263625 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f59c0032170 con 0x7f59b8003ee0 2019-12-05T05:11:07.166 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:07.161+0000 7f59d7951700 1 -- 172.21.15.42:0/3910263625 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f59ac003630 con 0x7f59b8003ee0 2019-12-05T05:11:07.202 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:07.161+0000 7f59d56ed700 1 --2- 172.21.15.42:0/3910263625 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f59b8003ee0 0x7f59b8004970 secure :-1 s=THROTTLE_DONE pgs=424 cs=0 l=1 rx=0x7f59c0007e90 tx=0x7f59c0010e40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:07.202 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:07.161+0000 7f59c67fc700 1 -- 172.21.15.42:0/3910263625 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) v1 ==== 94+0+2105 (secure 0 0 0) 0x7f59c00323d0 con 0x7f59b8003ee0 2019-12-05T05:11:07.203 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 51 2019-12-05T05:11:07.203 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:07.165+0000 7f59d7951700 1 -- 172.21.15.42:0/3910263625 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f59b8046030 msgr2=0x7f59b8048540 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:11:07.204 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:07.165+0000 7f59d7951700 1 --2- 172.21.15.42:0/3910263625 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f59b8046030 0x7f59b8048540 crc :-1 s=READY pgs=193 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:11:07.204 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:07.165+0000 7f59d7951700 1 -- 172.21.15.42:0/3910263625 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f59b8003ee0 msgr2=0x7f59b8004970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:11:07.204 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:07.165+0000 7f59d7951700 1 --2- 172.21.15.42:0/3910263625 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f59b8003ee0 0x7f59b8004970 secure :-1 s=READY pgs=424 cs=0 l=1 rx=0x7f59c0007e90 tx=0x7f59c0010e40).stop 2019-12-05T05:11:07.204 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:07.165+0000 7f59d66ef700 1 -- 172.21.15.42:0/3910263625 reap_dead start 2019-12-05T05:11:07.204 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:07.165+0000 7f59d66ef700 1 -- 172.21.15.42:0/3910263625 reap_dead start 2019-12-05T05:11:07.205 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:07.165+0000 7f59d7951700 1 -- 172.21.15.42:0/3910263625 shutdown_connections 2019-12-05T05:11:07.205 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:07.165+0000 7f59d7951700 1 -- 172.21.15.42:0/3910263625 shutdown_connections 2019-12-05T05:11:07.205 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:07.165+0000 7f59d7951700 1 -- 172.21.15.42:0/3910263625 wait complete. 2019-12-05T05:11:07.205 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:07.165+0000 7f59d7951700 1 -- 172.21.15.42:0/3910263625 >> 172.21.15.42:0/3910263625 conn(0x7f59d0116dc0 msgr2=0x7f59d00a97e0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:11:07.206 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:11:07.206 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":51,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5215,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6835/3176146895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3176146895},{"type":"v1","addr":"172.21.15.42:6835","nonce":3176146895}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":41}],"filesystems":[{"mdsmap":{"epoch":51,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:11:01.313715+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4595},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4595":{"gid":4595,"name":"b","rank":0,"incarnation":36,"state":"up:active","state_seq":11,"addr":"172.21.15.40:6833/211584380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":211584380},{"type":"v1","addr":"172.21.15.40:6833","nonce":211584380}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:11:07.206 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T05:11:09.208 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:11:09.324 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:09.305+0000 7fdf5529a700 1 Processor -- start 2019-12-05T05:11:09.324 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:09.305+0000 7fdf5529a700 1 -- start start 2019-12-05T05:11:09.324 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:09.309+0000 7fdf5529a700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdf500a7730 0x7fdf5008df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:11:09.325 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:09.309+0000 7fdf5529a700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fdf5009fad0 0x7fdf5007eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:11:09.325 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:09.309+0000 7fdf5529a700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf500a0700 con 0x7fdf5007e710 2019-12-05T05:11:09.325 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:09.309+0000 7fdf5529a700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf500a0af0 con 0x7fdf5009fad0 2019-12-05T05:11:09.325 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:09.309+0000 7fdf5529a700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf5008e5a0 con 0x7fdf500a7730 2019-12-05T05:11:09.325 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:09.309+0000 7fdf4ffff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fdf5009fad0 0x7fdf5007eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:11:09.326 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:09.309+0000 7fdf4ffff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fdf5009fad0 0x7fdf5007eb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:57568/0 (socket says 172.21.15.42:57568) 2019-12-05T05:11:09.326 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:09.309+0000 7fdf4effd700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fdf5007e710 0x7fdf5009f2d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:57602/0 (socket says 172.21.15.42:57602) 2019-12-05T05:11:09.326 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:09.309+0000 7fdf4ffff700 1 -- 172.21.15.42:0/589152322 learned_addr learned my addr 172.21.15.42:0/589152322 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:11:09.327 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:09.309+0000 7fdf4f7fe700 1 --2- 172.21.15.42:0/589152322 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdf500a7730 0x7fdf5008df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:11:09.327 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:09.309+0000 7fdf4ffff700 1 -- 172.21.15.42:0/589152322 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdf500a7730 msgr2=0x7fdf5008df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:11:09.327 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:09.309+0000 7fdf4ffff700 1 --2- 172.21.15.42:0/589152322 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdf500a7730 0x7fdf5008df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:11:09.327 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:09.309+0000 7fdf4e7fc700 1 -- 172.21.15.42:0/589152322 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 457584616 0 0) 0x7fdf500a0700 con 0x7fdf5007e710 2019-12-05T05:11:09.327 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:09.309+0000 7fdf4ffff700 1 -- 172.21.15.42:0/589152322 >> v1:172.21.15.42:6789/0 conn(0x7fdf5007e710 legacy=0x7fdf5009f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:11:09.327 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:09.309+0000 7fdf4ffff700 1 -- 172.21.15.42:0/589152322 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf5007f090 con 0x7fdf5009fad0 2019-12-05T05:11:09.327 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:09.309+0000 7fdf4ffff700 1 --2- 172.21.15.42:0/589152322 >> v2:172.21.15.42:3300/0 conn(0x7fdf5009fad0 0x7fdf5007eb40 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fdf48008b10 tx=0x7fdf480100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:09.328 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:09.309+0000 7fdf4ffff700 1 --2- 172.21.15.42:0/589152322 >> v2:172.21.15.42:3300/0 conn(0x7fdf5009fad0 0x7fdf5007eb40 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fdf48008b10 tx=0x7fdf480100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:09.328 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:09.309+0000 7fdf4ffff700 1 --2- 172.21.15.42:0/589152322 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdf5009fad0 0x7fdf5007eb40 secure :-1 s=READY pgs=426 cs=0 l=1 rx=0x7fdf48008b10 tx=0x7fdf480100b0).ready entity=mon.1 client_cookie=dff0ff068e2dc339 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:11:09.328 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:09.309+0000 7fdf4ffff700 1 --2- 172.21.15.42:0/589152322 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdf5009fad0 0x7fdf5007eb40 secure :-1 s=THROTTLE_DONE pgs=426 cs=0 l=1 rx=0x7fdf48008b10 tx=0x7fdf480100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:09.328 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:09.309+0000 7fdf4e7fc700 1 -- 172.21.15.42:0/589152322 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdf48010da0 con 0x7fdf5009fad0 2019-12-05T05:11:09.328 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:09.309+0000 7fdf4ffff700 1 --2- 172.21.15.42:0/589152322 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdf5009fad0 0x7fdf5007eb40 secure :-1 s=THROTTLE_DONE pgs=426 cs=0 l=1 rx=0x7fdf48008b10 tx=0x7fdf480100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:09.328 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:09.309+0000 7fdf4e7fc700 1 -- 172.21.15.42:0/589152322 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf480109c0 con 0x7fdf5009fad0 2019-12-05T05:11:09.328 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:09.309+0000 7fdf4ffff700 1 --2- 172.21.15.42:0/589152322 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdf5009fad0 0x7fdf5007eb40 secure :-1 s=THROTTLE_DONE pgs=426 cs=0 l=1 rx=0x7fdf48008b10 tx=0x7fdf480100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:09.329 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:09.309+0000 7fdf5529a700 1 -- 172.21.15.42:0/589152322 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdf5009fad0 msgr2=0x7fdf5007eb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:11:09.329 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:09.309+0000 7fdf5529a700 1 --2- 172.21.15.42:0/589152322 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdf5009fad0 0x7fdf5007eb40 secure :-1 s=READY pgs=426 cs=0 l=1 rx=0x7fdf48008b10 tx=0x7fdf480100b0).stop 2019-12-05T05:11:09.329 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:09.309+0000 7fdf4e7fc700 1 -- 172.21.15.42:0/589152322 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdf480114a0 con 0x7fdf5009fad0 2019-12-05T05:11:09.329 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:09.309+0000 7fdf5529a700 1 -- 172.21.15.42:0/589152322 shutdown_connections 2019-12-05T05:11:09.329 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:09.309+0000 7fdf5529a700 1 --2- 172.21.15.42:0/589152322 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdf5009fad0 0x7fdf5007eb40 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:11:09.329 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:09.309+0000 7fdf5529a700 1 --2- 172.21.15.42:0/589152322 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdf500a7730 0x7fdf5008df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:11:09.329 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:09.309+0000 7fdf5529a700 1 -- 172.21.15.42:0/589152322 shutdown_connections 2019-12-05T05:11:09.330 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:09.309+0000 7fdf5529a700 1 -- 172.21.15.42:0/589152322 wait complete. 2019-12-05T05:11:09.330 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:09.309+0000 7fdf5529a700 1 -- 172.21.15.42:0/589152322 >> 172.21.15.42:0/589152322 conn(0x7fdf500aabd0 msgr2=0x7fdf500a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:11:09.330 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:09.313+0000 7fdf5529a700 1 Processor -- start 2019-12-05T05:11:09.330 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:09.313+0000 7fdf5529a700 1 -- start start 2019-12-05T05:11:09.330 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:09.313+0000 7fdf5529a700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fdf500a2a40 0x7fdf50142860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:11:09.330 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:09.313+0000 7fdf5529a700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdf50121d20 0x7fdf50122150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:11:09.330 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:09.313+0000 7fdf5529a700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf50142db0 con 0x7fdf500a1920 2019-12-05T05:11:09.331 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:09.313+0000 7fdf5529a700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf50121590 con 0x7fdf500a2a40 2019-12-05T05:11:09.331 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:09.313+0000 7fdf5529a700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf500a2f50 con 0x7fdf50121d20 2019-12-05T05:11:09.331 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:09.313+0000 7fdf4ffff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fdf500a1920 0x7fdf50124b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:57612/0 (socket says 172.21.15.42:57612) 2019-12-05T05:11:09.331 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:09.313+0000 7fdf4ffff700 1 -- 172.21.15.42:0/566705419 learned_addr learned my addr 172.21.15.42:0/566705419 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:11:09.332 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:09.313+0000 7fdf4effd700 1 --2- 172.21.15.42:0/566705419 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdf50121d20 0x7fdf50122150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:11:09.332 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:09.313+0000 7fdf4f7fe700 1 --2- 172.21.15.42:0/566705419 >> v2:172.21.15.40:3300/0 conn(0x7fdf500a2a40 0x7fdf50142860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:11:09.332 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:09.313+0000 7fdf37fff700 1 -- 172.21.15.42:0/566705419 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2173634701 0 0) 0x7fdf50142db0 con 0x7fdf500a1920 2019-12-05T05:11:09.332 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:09.313+0000 7fdf37fff700 1 -- 172.21.15.42:0/566705419 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf300030b0 con 0x7fdf500a1920 2019-12-05T05:11:09.333 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:09.313+0000 7fdf37fff700 1 -- 172.21.15.42:0/566705419 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 1381815956 0 0) 0x7fdf300030b0 con 0x7fdf500a1920 2019-12-05T05:11:09.333 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:09.313+0000 7fdf37fff700 1 -- 172.21.15.42:0/566705419 --> v1:172.21.15.42:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdf50142db0 con 0x7fdf500a1920 2019-12-05T05:11:09.333 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:09.313+0000 7fdf37fff700 1 -- 172.21.15.42:0/566705419 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7fdf480087f0 con 0x7fdf500a1920 2019-12-05T05:11:09.333 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:09.313+0000 7fdf4f7fe700 1 -- 172.21.15.42:0/566705419 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdf50121d20 msgr2=0x7fdf50122150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:11:09.333 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:09.313+0000 7fdf4f7fe700 1 --2- 172.21.15.42:0/566705419 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdf50121d20 0x7fdf50122150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:11:09.333 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:09.313+0000 7fdf4f7fe700 1 -- 172.21.15.42:0/566705419 >> v1:172.21.15.42:6789/0 conn(0x7fdf500a1920 legacy=0x7fdf50124b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:11:09.333 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:09.313+0000 7fdf4f7fe700 1 -- 172.21.15.42:0/566705419 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf501452b0 con 0x7fdf500a2a40 2019-12-05T05:11:09.334 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:09.313+0000 7fdf4effd700 1 --2- 172.21.15.42:0/566705419 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdf50121d20 0x7fdf50122150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:11:09.334 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:09.313+0000 7fdf4f7fe700 1 --2- 172.21.15.42:0/566705419 >> v2:172.21.15.40:3300/0 conn(0x7fdf500a2a40 0x7fdf50142860 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fdf50121750 tx=0x7fdf48008650).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:09.334 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:09.313+0000 7fdf4f7fe700 1 --2- 172.21.15.42:0/566705419 >> v2:172.21.15.40:3300/0 conn(0x7fdf500a2a40 0x7fdf50142860 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fdf50121750 tx=0x7fdf48008650).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:09.334 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:09.313+0000 7fdf4f7fe700 1 --2- 172.21.15.42:0/566705419 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf500a2a40 0x7fdf50142860 secure :-1 s=READY pgs=334 cs=0 l=1 rx=0x7fdf50121750 tx=0x7fdf48008650).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:11:09.334 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:09.313+0000 7fdf4f7fe700 1 --2- 172.21.15.42:0/566705419 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf500a2a40 0x7fdf50142860 secure :-1 s=THROTTLE_DONE pgs=334 cs=0 l=1 rx=0x7fdf50121750 tx=0x7fdf48008650).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:09.335 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:09.313+0000 7fdf37fff700 1 -- 172.21.15.42:0/566705419 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdf40005720 con 0x7fdf500a2a40 2019-12-05T05:11:09.335 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:09.317+0000 7fdf4f7fe700 1 --2- 172.21.15.42:0/566705419 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf500a2a40 0x7fdf50142860 secure :-1 s=THROTTLE_DONE pgs=334 cs=0 l=1 rx=0x7fdf50121750 tx=0x7fdf48008650).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:09.335 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:09.317+0000 7fdf37fff700 1 -- 172.21.15.42:0/566705419 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf40007160 con 0x7fdf500a2a40 2019-12-05T05:11:09.335 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:09.317+0000 7fdf5529a700 1 -- 172.21.15.42:0/566705419 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf500a1d50 con 0x7fdf500a2a40 2019-12-05T05:11:09.335 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:09.317+0000 7fdf4f7fe700 1 --2- 172.21.15.42:0/566705419 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf500a2a40 0x7fdf50142860 secure :-1 s=THROTTLE_DONE pgs=334 cs=0 l=1 rx=0x7fdf50121750 tx=0x7fdf48008650).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:09.335 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:09.317+0000 7fdf37fff700 1 -- 172.21.15.42:0/566705419 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdf40005d00 con 0x7fdf500a2a40 2019-12-05T05:11:09.335 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:09.317+0000 7fdf5529a700 1 -- 172.21.15.42:0/566705419 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf50125350 con 0x7fdf500a2a40 2019-12-05T05:11:09.336 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:09.317+0000 7fdf4f7fe700 1 --2- 172.21.15.42:0/566705419 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf500a2a40 0x7fdf50142860 secure :-1 s=THROTTLE_DONE pgs=334 cs=0 l=1 rx=0x7fdf50121750 tx=0x7fdf48008650).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:09.336 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:09.317+0000 7fdf37fff700 1 -- 172.21.15.42:0/566705419 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fdf4000a250 con 0x7fdf500a2a40 2019-12-05T05:11:09.336 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:09.317+0000 7fdf4f7fe700 1 --2- 172.21.15.42:0/566705419 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf500a2a40 0x7fdf50142860 secure :-1 s=THROTTLE_DONE pgs=334 cs=0 l=1 rx=0x7fdf50121750 tx=0x7fdf48008650).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:09.336 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:09.317+0000 7fdf37fff700 1 --2- 172.21.15.42:0/566705419 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fdf3003b650 0x7fdf3003db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:11:09.336 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:09.317+0000 7fdf37fff700 1 -- 172.21.15.42:0/566705419 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6403+0+0 (secure 0 0 0) 0x7fdf4004ac50 con 0x7fdf500a2a40 2019-12-05T05:11:09.336 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:09.317+0000 7fdf4effd700 1 --2- 172.21.15.42:0/566705419 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fdf3003b650 0x7fdf3003db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:11:09.337 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:09.317+0000 7fdf4effd700 1 --2- 172.21.15.42:0/566705419 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fdf3003b650 0x7fdf3003db60 crc :-1 s=READY pgs=194 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:11:09.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:09.321+0000 7fdf5529a700 1 -- 172.21.15.42:0/566705419 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf500a2280 con 0x7fdf500a2a40 2019-12-05T05:11:09.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:09.329+0000 7fdf4f7fe700 1 --2- 172.21.15.42:0/566705419 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf500a2a40 0x7fdf50142860 secure :-1 s=THROTTLE_DONE pgs=334 cs=0 l=1 rx=0x7fdf50121750 tx=0x7fdf48008650).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:09.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:09.329+0000 7fdf37fff700 1 -- 172.21.15.42:0/566705419 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fdf40031cb0 con 0x7fdf500a2a40 2019-12-05T05:11:09.505 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:09.501+0000 7fdf5529a700 1 -- 172.21.15.42:0/566705419 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdf5005d100 con 0x7fdf500a2a40 2019-12-05T05:11:09.506 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:09.501+0000 7fdf4f7fe700 1 --2- 172.21.15.42:0/566705419 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf500a2a40 0x7fdf50142860 secure :-1 s=THROTTLE_DONE pgs=334 cs=0 l=1 rx=0x7fdf50121750 tx=0x7fdf48008650).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:09.506 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:09.501+0000 7fdf37fff700 1 -- 172.21.15.42:0/566705419 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) v1 ==== 94+0+2105 (secure 0 0 0) 0x7fdf40031f10 con 0x7fdf500a2a40 2019-12-05T05:11:09.507 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:11:09.508 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":51,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5215,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6835/3176146895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3176146895},{"type":"v1","addr":"172.21.15.42:6835","nonce":3176146895}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":41}],"filesystems":[{"mdsmap":{"epoch":51,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:11:01.313715+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4595},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4595":{"gid":4595,"name":"b","rank":0,"incarnation":36,"state":"up:active","state_seq":11,"addr":"172.21.15.40:6833/211584380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":211584380},{"type":"v1","addr":"172.21.15.40:6833","nonce":211584380}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:11:09.508 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 51 2019-12-05T05:11:09.510 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:09.505+0000 7fdf5529a700 1 -- 172.21.15.42:0/566705419 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fdf3003b650 msgr2=0x7fdf3003db60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:11:09.510 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:09.505+0000 7fdf5529a700 1 --2- 172.21.15.42:0/566705419 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fdf3003b650 0x7fdf3003db60 crc :-1 s=READY pgs=194 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:11:09.510 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:09.505+0000 7fdf5529a700 1 -- 172.21.15.42:0/566705419 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf500a2a40 msgr2=0x7fdf50142860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:11:09.510 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:09.505+0000 7fdf5529a700 1 --2- 172.21.15.42:0/566705419 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf500a2a40 0x7fdf50142860 secure :-1 s=READY pgs=334 cs=0 l=1 rx=0x7fdf50121750 tx=0x7fdf48008650).stop 2019-12-05T05:11:09.511 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:09.505+0000 7fdf5529a700 1 -- 172.21.15.42:0/566705419 shutdown_connections 2019-12-05T05:11:09.511 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:09.505+0000 7fdf5529a700 1 --2- 172.21.15.42:0/566705419 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fdf3003b650 0x7fdf3003db60 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:11:09.511 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:09.505+0000 7fdf5529a700 1 --2- 172.21.15.42:0/566705419 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf500a2a40 0x7fdf50142860 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:11:09.511 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:09.505+0000 7fdf5529a700 1 --2- 172.21.15.42:0/566705419 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdf50121d20 0x7fdf50122150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:11:09.511 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:09.505+0000 7fdf5529a700 1 -- 172.21.15.42:0/566705419 shutdown_connections 2019-12-05T05:11:09.511 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:09.505+0000 7fdf5529a700 1 -- 172.21.15.42:0/566705419 wait complete. 2019-12-05T05:11:09.511 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:09.505+0000 7fdf5529a700 1 -- 172.21.15.42:0/566705419 >> 172.21.15.42:0/566705419 conn(0x7fdf500a8290 msgr2=0x7fdf5007da50 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:11:09.521 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T05:11:11.522 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:11:11.629 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:11.613+0000 7f0f8fed8700 1 Processor -- start 2019-12-05T05:11:11.630 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:11.613+0000 7f0f8fed8700 1 -- start start 2019-12-05T05:11:11.630 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:11.617+0000 7f0f8fed8700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f0f880a7740 0x7f0f8808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:11:11.631 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:11.617+0000 7f0f8fed8700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0f8809fae0 0x7f0f8807eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:11:11.631 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:11.617+0000 7f0f8fed8700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f880a0710 con 0x7f0f8807e720 2019-12-05T05:11:11.631 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:11.617+0000 7f0f8fed8700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f880a0b00 con 0x7f0f880a7740 2019-12-05T05:11:11.631 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:11.617+0000 7f0f8fed8700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f8808e5b0 con 0x7f0f8809fae0 2019-12-05T05:11:11.631 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:11.617+0000 7f0f8dc74700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f0f8807e720 0x7f0f8809f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:57616/0 (socket says 172.21.15.42:57616) 2019-12-05T05:11:11.631 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:11.617+0000 7f0f8dc74700 1 -- 172.21.15.42:0/2628475273 learned_addr learned my addr 172.21.15.42:0/2628475273 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:11:11.631 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:11.617+0000 7f0f8e475700 1 --2- 172.21.15.42:0/2628475273 >> v2:172.21.15.40:3300/0 conn(0x7f0f880a7740 0x7f0f8808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:11:11.632 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:11.617+0000 7f0f8ec76700 1 --2- 172.21.15.42:0/2628475273 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0f8809fae0 0x7f0f8807eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:11:11.632 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:11.617+0000 7f0f8d473700 1 -- 172.21.15.42:0/2628475273 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1638534770 0 0) 0x7f0f880a0710 con 0x7f0f8807e720 2019-12-05T05:11:11.632 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:11.617+0000 7f0f8d473700 1 -- 172.21.15.42:0/2628475273 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f740030b0 con 0x7f0f8807e720 2019-12-05T05:11:11.632 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:11.617+0000 7f0f8e475700 1 -- 172.21.15.42:0/2628475273 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0f8809fae0 msgr2=0x7f0f8807eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:11:11.632 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:11.617+0000 7f0f8e475700 1 --2- 172.21.15.42:0/2628475273 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0f8809fae0 0x7f0f8807eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:11:11.632 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:11.617+0000 7f0f8e475700 1 -- 172.21.15.42:0/2628475273 >> v1:172.21.15.42:6789/0 conn(0x7f0f8807e720 legacy=0x7f0f8809f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:11:11.632 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:11.617+0000 7f0f8e475700 1 -- 172.21.15.42:0/2628475273 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f8807f0a0 con 0x7f0f880a7740 2019-12-05T05:11:11.633 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:11.617+0000 7f0f8e475700 1 --2- 172.21.15.42:0/2628475273 >> v2:172.21.15.40:3300/0 conn(0x7f0f880a7740 0x7f0f8808df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f0f84008b10 tx=0x7f0f8400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:11.633 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:11.617+0000 7f0f8e475700 1 --2- 172.21.15.42:0/2628475273 >> v2:172.21.15.40:3300/0 conn(0x7f0f880a7740 0x7f0f8808df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f0f84008b10 tx=0x7f0f8400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:11.633 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:11.617+0000 7f0f8e475700 1 --2- 172.21.15.42:0/2628475273 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0f880a7740 0x7f0f8808df80 secure :-1 s=READY pgs=335 cs=0 l=1 rx=0x7f0f84008b10 tx=0x7f0f8400fff0).ready entity=mon.0 client_cookie=97af48f00b004c82 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:11:11.633 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:11.617+0000 7f0f8e475700 1 --2- 172.21.15.42:0/2628475273 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0f880a7740 0x7f0f8808df80 secure :-1 s=THROTTLE_DONE pgs=335 cs=0 l=1 rx=0x7f0f84008b10 tx=0x7f0f8400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:11.634 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:11.617+0000 7f0f8d473700 1 -- 172.21.15.42:0/2628475273 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0f84010900 con 0x7f0f880a7740 2019-12-05T05:11:11.634 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:11.617+0000 7f0f8e475700 1 --2- 172.21.15.42:0/2628475273 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0f880a7740 0x7f0f8808df80 secure :-1 s=THROTTLE_DONE pgs=335 cs=0 l=1 rx=0x7f0f84008b10 tx=0x7f0f8400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:11.634 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:11.617+0000 7f0f8d473700 1 -- 172.21.15.42:0/2628475273 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f84008b50 con 0x7f0f880a7740 2019-12-05T05:11:11.634 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:11.617+0000 7f0f8e475700 1 --2- 172.21.15.42:0/2628475273 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0f880a7740 0x7f0f8808df80 secure :-1 s=THROTTLE_DONE pgs=335 cs=0 l=1 rx=0x7f0f84008b10 tx=0x7f0f8400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:11.634 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:11.617+0000 7f0f8d473700 1 -- 172.21.15.42:0/2628475273 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0f840110c0 con 0x7f0f880a7740 2019-12-05T05:11:11.635 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:11.617+0000 7f0f8fed8700 1 -- 172.21.15.42:0/2628475273 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0f880a7740 msgr2=0x7f0f8808df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:11:11.635 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:11.617+0000 7f0f8fed8700 1 --2- 172.21.15.42:0/2628475273 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0f880a7740 0x7f0f8808df80 secure :-1 s=READY pgs=335 cs=0 l=1 rx=0x7f0f84008b10 tx=0x7f0f8400fff0).stop 2019-12-05T05:11:11.635 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:11.621+0000 7f0f8fed8700 1 -- 172.21.15.42:0/2628475273 shutdown_connections 2019-12-05T05:11:11.635 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:11.621+0000 7f0f8fed8700 1 --2- 172.21.15.42:0/2628475273 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0f8809fae0 0x7f0f8807eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:11:11.635 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:11.621+0000 7f0f8fed8700 1 --2- 172.21.15.42:0/2628475273 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0f880a7740 0x7f0f8808df80 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:11:11.636 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:11.621+0000 7f0f8fed8700 1 -- 172.21.15.42:0/2628475273 shutdown_connections 2019-12-05T05:11:11.637 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:11.621+0000 7f0f8fed8700 1 -- 172.21.15.42:0/2628475273 wait complete. 2019-12-05T05:11:11.638 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:11.621+0000 7f0f8fed8700 1 -- 172.21.15.42:0/2628475273 >> 172.21.15.42:0/2628475273 conn(0x7f0f880aabe0 msgr2=0x7f0f880a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:11:11.638 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:11.621+0000 7f0f8fed8700 1 Processor -- start 2019-12-05T05:11:11.638 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:11.621+0000 7f0f8fed8700 1 -- start start 2019-12-05T05:11:11.638 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:11.625+0000 7f0f8fed8700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f0f880a5f80 0x7f0f880a63b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:11:11.639 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:11.625+0000 7f0f8fed8700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0f8812c410 0x7f0f8812c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:11:11.639 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:11.625+0000 7f0f8fed8700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f8807e120 con 0x7f0f8811b770 2019-12-05T05:11:11.639 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:11.625+0000 7f0f8fed8700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f880a6900 con 0x7f0f880a5f80 2019-12-05T05:11:11.639 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:11.625+0000 7f0f8fed8700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f880a6b70 con 0x7f0f8812c410 2019-12-05T05:11:11.640 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:11.625+0000 7f0f8ec76700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0f8812c410 0x7f0f8812c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:11:11.640 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:11.625+0000 7f0f8dc74700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f0f880a5f80 0x7f0f880a63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:11:11.640 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:11.625+0000 7f0f8ec76700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0f8812c410 0x7f0f8812c840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:57588/0 (socket says 172.21.15.42:57588) 2019-12-05T05:11:11.640 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:11.625+0000 7f0f8ec76700 1 -- 172.21.15.42:0/3107747652 learned_addr learned my addr 172.21.15.42:0/3107747652 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:11:11.644 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:11.625+0000 7f0f8ec76700 1 -- 172.21.15.42:0/3107747652 >> v2:172.21.15.40:3300/0 conn(0x7f0f880a5f80 msgr2=0x7f0f880a63b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:11:11.644 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:11.625+0000 7f0f8ec76700 1 --2- 172.21.15.42:0/3107747652 >> v2:172.21.15.40:3300/0 conn(0x7f0f880a5f80 0x7f0f880a63b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:11:11.644 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:11.625+0000 7f0f8ec76700 1 -- 172.21.15.42:0/3107747652 >> v1:172.21.15.40:6789/0 conn(0x7f0f8811b770 legacy=0x7f0f8807d920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:11:11.644 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:11.625+0000 7f0f8ec76700 1 -- 172.21.15.42:0/3107747652 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f8812cd90 con 0x7f0f8812c410 2019-12-05T05:11:11.645 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:11.625+0000 7f0f8ec76700 1 --2- 172.21.15.42:0/3107747652 >> v2:172.21.15.42:3300/0 conn(0x7f0f8812c410 0x7f0f8812c840 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f0f880a5790 tx=0x7f0f80012cd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:11.645 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:11.625+0000 7f0f8ec76700 1 --2- 172.21.15.42:0/3107747652 >> v2:172.21.15.42:3300/0 conn(0x7f0f8812c410 0x7f0f8812c840 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f0f880a5790 tx=0x7f0f80012cd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:11.645 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:11.625+0000 7f0f8ec76700 1 --2- 172.21.15.42:0/3107747652 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0f8812c410 0x7f0f8812c840 secure :-1 s=READY pgs=429 cs=0 l=1 rx=0x7f0f880a5790 tx=0x7f0f80012cd0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:11:11.645 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:11.625+0000 7f0f8ec76700 1 --2- 172.21.15.42:0/3107747652 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0f8812c410 0x7f0f8812c840 secure :-1 s=THROTTLE_DONE pgs=429 cs=0 l=1 rx=0x7f0f880a5790 tx=0x7f0f80012cd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:11.645 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:11.625+0000 7f0f7effd700 1 -- 172.21.15.42:0/3107747652 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0f80014740 con 0x7f0f8812c410 2019-12-05T05:11:11.645 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:11.625+0000 7f0f8fed8700 1 -- 172.21.15.42:0/3107747652 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f880a9640 con 0x7f0f8812c410 2019-12-05T05:11:11.645 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:11.625+0000 7f0f8ec76700 1 --2- 172.21.15.42:0/3107747652 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0f8812c410 0x7f0f8812c840 secure :-1 s=THROTTLE_DONE pgs=429 cs=0 l=1 rx=0x7f0f880a5790 tx=0x7f0f80012cd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:11.646 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:11.625+0000 7f0f7effd700 1 -- 172.21.15.42:0/3107747652 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f80007150 con 0x7f0f8812c410 2019-12-05T05:11:11.646 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:11.625+0000 7f0f8ec76700 1 --2- 172.21.15.42:0/3107747652 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0f8812c410 0x7f0f8812c840 secure :-1 s=THROTTLE_DONE pgs=429 cs=0 l=1 rx=0x7f0f880a5790 tx=0x7f0f80012cd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:11.646 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:11.625+0000 7f0f7effd700 1 -- 172.21.15.42:0/3107747652 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0f80014d20 con 0x7f0f8812c410 2019-12-05T05:11:11.646 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:11.625+0000 7f0f8fed8700 1 -- 172.21.15.42:0/3107747652 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f880a9bf0 con 0x7f0f8812c410 2019-12-05T05:11:11.646 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:11.629+0000 7f0f8ec76700 1 --2- 172.21.15.42:0/3107747652 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0f8812c410 0x7f0f8812c840 secure :-1 s=THROTTLE_DONE pgs=429 cs=0 l=1 rx=0x7f0f880a5790 tx=0x7f0f80012cd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:11.646 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:11.629+0000 7f0f7effd700 1 -- 172.21.15.42:0/3107747652 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f0f80005540 con 0x7f0f8812c410 2019-12-05T05:11:11.646 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:11.629+0000 7f0f8ec76700 1 --2- 172.21.15.42:0/3107747652 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0f8812c410 0x7f0f8812c840 secure :-1 s=THROTTLE_DONE pgs=429 cs=0 l=1 rx=0x7f0f880a5790 tx=0x7f0f80012cd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:11.647 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:11.629+0000 7f0f7effd700 1 --2- 172.21.15.42:0/3107747652 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0f7003aca0 0x7f0f7003d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:11:11.647 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:11.629+0000 7f0f7effd700 1 -- 172.21.15.42:0/3107747652 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6403+0+0 (secure 0 0 0) 0x7f0f8004d400 con 0x7f0f8812c410 2019-12-05T05:11:11.647 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:11.629+0000 7f0f8e475700 1 --2- 172.21.15.42:0/3107747652 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0f7003aca0 0x7f0f7003d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:11:11.647 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:11.629+0000 7f0f8e475700 1 --2- 172.21.15.42:0/3107747652 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0f7003aca0 0x7f0f7003d1b0 crc :-1 s=READY pgs=195 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:11:11.647 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:11.629+0000 7f0f8fed8700 1 -- 172.21.15.42:0/3107747652 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f8805d110 con 0x7f0f8812c410 2019-12-05T05:11:11.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:11.637+0000 7f0f8ec76700 1 --2- 172.21.15.42:0/3107747652 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0f8812c410 0x7f0f8812c840 secure :-1 s=THROTTLE_DONE pgs=429 cs=0 l=1 rx=0x7f0f880a5790 tx=0x7f0f80012cd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:11.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:11.637+0000 7f0f7effd700 1 -- 172.21.15.42:0/3107747652 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f0f80034460 con 0x7f0f8812c410 2019-12-05T05:11:11.816 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:11.809+0000 7f0f8fed8700 1 -- 172.21.15.42:0/3107747652 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0f8805b8c0 con 0x7f0f8812c410 2019-12-05T05:11:11.816 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:11.813+0000 7f0f8ec76700 1 --2- 172.21.15.42:0/3107747652 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0f8812c410 0x7f0f8812c840 secure :-1 s=THROTTLE_DONE pgs=429 cs=0 l=1 rx=0x7f0f880a5790 tx=0x7f0f80012cd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:11.817 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:11.813+0000 7f0f7effd700 1 -- 172.21.15.42:0/3107747652 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) v1 ==== 94+0+2105 (secure 0 0 0) 0x7f0f800346c0 con 0x7f0f8812c410 2019-12-05T05:11:11.817 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 51 2019-12-05T05:11:11.817 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:11:11.817 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":51,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5215,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6835/3176146895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3176146895},{"type":"v1","addr":"172.21.15.42:6835","nonce":3176146895}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":41}],"filesystems":[{"mdsmap":{"epoch":51,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:11:01.313715+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4595},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4595":{"gid":4595,"name":"b","rank":0,"incarnation":36,"state":"up:active","state_seq":11,"addr":"172.21.15.40:6833/211584380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":211584380},{"type":"v1","addr":"172.21.15.40:6833","nonce":211584380}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:11:11.819 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:11.813+0000 7f0f8fed8700 1 -- 172.21.15.42:0/3107747652 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0f7003aca0 msgr2=0x7f0f7003d1b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:11:11.819 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:11.813+0000 7f0f8fed8700 1 --2- 172.21.15.42:0/3107747652 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0f7003aca0 0x7f0f7003d1b0 crc :-1 s=READY pgs=195 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:11:11.819 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:11.813+0000 7f0f8fed8700 1 -- 172.21.15.42:0/3107747652 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0f8812c410 msgr2=0x7f0f8812c840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:11:11.820 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:11.813+0000 7f0f8fed8700 1 --2- 172.21.15.42:0/3107747652 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0f8812c410 0x7f0f8812c840 secure :-1 s=READY pgs=429 cs=0 l=1 rx=0x7f0f880a5790 tx=0x7f0f80012cd0).stop 2019-12-05T05:11:11.820 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:11.813+0000 7f0f8fed8700 1 -- 172.21.15.42:0/3107747652 shutdown_connections 2019-12-05T05:11:11.820 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:11.813+0000 7f0f8fed8700 1 --2- 172.21.15.42:0/3107747652 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0f7003aca0 0x7f0f7003d1b0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:11:11.820 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:11.813+0000 7f0f8fed8700 1 --2- 172.21.15.42:0/3107747652 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0f8812c410 0x7f0f8812c840 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:11:11.820 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:11.813+0000 7f0f8fed8700 1 --2- 172.21.15.42:0/3107747652 >> v2:172.21.15.40:3300/0 conn(0x7f0f880a5f80 0x7f0f880a63b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:11:11.820 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:11.813+0000 7f0f8fed8700 1 -- 172.21.15.42:0/3107747652 shutdown_connections 2019-12-05T05:11:11.821 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:11.813+0000 7f0f8fed8700 1 -- 172.21.15.42:0/3107747652 wait complete. 2019-12-05T05:11:11.821 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:11.813+0000 7f0f8fed8700 1 -- 172.21.15.42:0/3107747652 >> 172.21.15.42:0/3107747652 conn(0x7f0f8809f580 msgr2=0x7f0f88059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:11:11.829 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T05:11:13.832 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:11:13.937 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:13.929+0000 7f9503548700 1 Processor -- start 2019-12-05T05:11:13.937 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:13.929+0000 7f9503548700 1 -- start start 2019-12-05T05:11:13.937 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:13.929+0000 7f9503548700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f94fc0a7740 0x7f94fc08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:11:13.938 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:13.929+0000 7f9503548700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f94fc07e720 0x7f94fc07eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:11:13.938 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:13.929+0000 7f9503548700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f94fc09f2e0 0x7f94fc09f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:11:13.938 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:13.929+0000 7f9503548700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94fc0a0b00 con 0x7f94fc0a7740 2019-12-05T05:11:13.938 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:13.929+0000 7f9503548700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94fc08e5b0 con 0x7f94fc07e720 2019-12-05T05:11:13.938 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:13.929+0000 7f9503548700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94fc08e850 con 0x7f94fc09f2e0 2019-12-05T05:11:13.938 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:13.929+0000 7f95012e4700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f94fc07e720 0x7f94fc07eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:11:13.938 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:13.929+0000 7f95012e4700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f94fc07e720 0x7f94fc07eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:57594/0 (socket says 172.21.15.42:57594) 2019-12-05T05:11:13.939 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:13.929+0000 7f95012e4700 1 -- 172.21.15.42:0/3724574418 learned_addr learned my addr 172.21.15.42:0/3724574418 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:11:13.939 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:13.929+0000 7f95022e6700 1 --2- 172.21.15.42:0/3724574418 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f94fc09f2e0 0x7f94fc09f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:11:13.939 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:13.929+0000 7f9501ae5700 1 --2- 172.21.15.42:0/3724574418 >> v2:172.21.15.40:3300/0 conn(0x7f94fc0a7740 0x7f94fc08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:11:13.939 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:13.929+0000 7f95012e4700 1 -- 172.21.15.42:0/3724574418 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f94fc09f2e0 msgr2=0x7f94fc09f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:11:13.939 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:13.929+0000 7f95012e4700 1 --2- 172.21.15.42:0/3724574418 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f94fc09f2e0 0x7f94fc09f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:11:13.939 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:13.929+0000 7f95012e4700 1 -- 172.21.15.42:0/3724574418 >> v2:172.21.15.40:3300/0 conn(0x7f94fc0a7740 msgr2=0x7f94fc08df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:11:13.939 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:13.929+0000 7f95012e4700 1 --2- 172.21.15.42:0/3724574418 >> v2:172.21.15.40:3300/0 conn(0x7f94fc0a7740 0x7f94fc08df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:11:13.939 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:13.929+0000 7f95012e4700 1 -- 172.21.15.42:0/3724574418 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94fc07f0a0 con 0x7f94fc07e720 2019-12-05T05:11:13.940 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:13.929+0000 7f95012e4700 1 --2- 172.21.15.42:0/3724574418 >> v2:172.21.15.42:3300/0 conn(0x7f94fc07e720 0x7f94fc07eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f94fc0a0f50 tx=0x7f94ec00fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:13.940 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:13.929+0000 7f95012e4700 1 --2- 172.21.15.42:0/3724574418 >> v2:172.21.15.42:3300/0 conn(0x7f94fc07e720 0x7f94fc07eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f94fc0a0f50 tx=0x7f94ec00fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:13.940 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:13.929+0000 7f95012e4700 1 --2- 172.21.15.42:0/3724574418 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f94fc07e720 0x7f94fc07eb50 secure :-1 s=READY pgs=430 cs=0 l=1 rx=0x7f94fc0a0f50 tx=0x7f94ec00fdb0).ready entity=mon.1 client_cookie=94a29db9adba29c6 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:11:13.940 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:13.929+0000 7f95012e4700 1 --2- 172.21.15.42:0/3724574418 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f94fc07e720 0x7f94fc07eb50 secure :-1 s=THROTTLE_DONE pgs=430 cs=0 l=1 rx=0x7f94fc0a0f50 tx=0x7f94ec00fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:13.941 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:13.929+0000 7f95012e4700 1 --2- 172.21.15.42:0/3724574418 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f94fc07e720 0x7f94fc07eb50 secure :-1 s=THROTTLE_DONE pgs=430 cs=0 l=1 rx=0x7f94fc0a0f50 tx=0x7f94ec00fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:13.942 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:13.929+0000 7f95012e4700 1 --2- 172.21.15.42:0/3724574418 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f94fc07e720 0x7f94fc07eb50 secure :-1 s=THROTTLE_DONE pgs=430 cs=0 l=1 rx=0x7f94fc0a0f50 tx=0x7f94ec00fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:13.942 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:13.929+0000 7f9500ae3700 1 -- 172.21.15.42:0/3724574418 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f94ec008c90 con 0x7f94fc07e720 2019-12-05T05:11:13.942 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:13.929+0000 7f9500ae3700 1 -- 172.21.15.42:0/3724574418 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94ec0088b0 con 0x7f94fc07e720 2019-12-05T05:11:13.942 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:13.929+0000 7f9500ae3700 1 -- 172.21.15.42:0/3724574418 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f94ec011b50 con 0x7f94fc07e720 2019-12-05T05:11:13.943 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:13.929+0000 7f9503548700 1 -- 172.21.15.42:0/3724574418 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f94fc07e720 msgr2=0x7f94fc07eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:11:13.943 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:13.929+0000 7f9503548700 1 --2- 172.21.15.42:0/3724574418 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f94fc07e720 0x7f94fc07eb50 secure :-1 s=READY pgs=430 cs=0 l=1 rx=0x7f94fc0a0f50 tx=0x7f94ec00fdb0).stop 2019-12-05T05:11:13.943 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:13.933+0000 7f9503548700 1 -- 172.21.15.42:0/3724574418 shutdown_connections 2019-12-05T05:11:13.943 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:13.933+0000 7f9503548700 1 --2- 172.21.15.42:0/3724574418 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f94fc09f2e0 0x7f94fc09f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:11:13.944 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:13.933+0000 7f9503548700 1 --2- 172.21.15.42:0/3724574418 >> v2:172.21.15.40:3300/0 conn(0x7f94fc0a7740 0x7f94fc08df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:11:13.944 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:13.933+0000 7f9503548700 1 --2- 172.21.15.42:0/3724574418 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f94fc07e720 0x7f94fc07eb50 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:11:13.945 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:13.933+0000 7f9503548700 1 -- 172.21.15.42:0/3724574418 shutdown_connections 2019-12-05T05:11:13.945 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:13.933+0000 7f9503548700 1 -- 172.21.15.42:0/3724574418 wait complete. 2019-12-05T05:11:13.945 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:13.933+0000 7f9503548700 1 -- 172.21.15.42:0/3724574418 >> 172.21.15.42:0/3724574418 conn(0x7f94fc0aabe0 msgr2=0x7f94fc0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:11:13.952 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:13.933+0000 7f9503548700 1 Processor -- start 2019-12-05T05:11:13.952 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:13.933+0000 7f9503548700 1 -- start start 2019-12-05T05:11:13.952 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:13.937+0000 7f9503548700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f94fc11ae40 0x7f94fc11b270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:11:13.953 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:13.937+0000 7f9503548700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f94fc138530 0x7f94fc13a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:11:13.953 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:13.937+0000 7f9503548700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94fc11baa0 con 0x7f94fc122f70 2019-12-05T05:11:13.953 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:13.937+0000 7f9503548700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94fc07d600 con 0x7f94fc138530 2019-12-05T05:11:13.953 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:13.937+0000 7f9503548700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94fc11aba0 con 0x7f94fc11ae40 2019-12-05T05:11:13.954 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:13.937+0000 7f95012e4700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f94fc11ae40 0x7f94fc11b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:11:13.954 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:13.937+0000 7f9501ae5700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f94fc122f70 0x7f94fc07cc80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:57636/0 (socket says 172.21.15.42:57636) 2019-12-05T05:11:13.954 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:13.937+0000 7f95012e4700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f94fc11ae40 0x7f94fc11b270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:57600/0 (socket says 172.21.15.42:57600) 2019-12-05T05:11:13.954 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:13.937+0000 7f9501ae5700 1 -- 172.21.15.42:0/3475995279 learned_addr learned my addr 172.21.15.42:0/3475995279 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:11:13.955 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:13.937+0000 7f95022e6700 1 --2- 172.21.15.42:0/3475995279 >> v2:172.21.15.40:3300/0 conn(0x7f94fc138530 0x7f94fc13a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:11:13.955 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:13.937+0000 7f95012e4700 1 -- 172.21.15.42:0/3475995279 >> v2:172.21.15.40:3300/0 conn(0x7f94fc138530 msgr2=0x7f94fc13a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:11:13.956 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:13.937+0000 7f95012e4700 1 --2- 172.21.15.42:0/3475995279 >> v2:172.21.15.40:3300/0 conn(0x7f94fc138530 0x7f94fc13a970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:11:13.956 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:13.937+0000 7f95012e4700 1 -- 172.21.15.42:0/3475995279 >> v1:172.21.15.42:6789/0 conn(0x7f94fc122f70 legacy=0x7f94fc07cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:11:13.956 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:13.937+0000 7f95012e4700 1 -- 172.21.15.42:0/3475995279 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94fc1300c0 con 0x7f94fc11ae40 2019-12-05T05:11:13.956 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:13.937+0000 7f94f27fc700 1 -- 172.21.15.42:0/3475995279 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 706767859 0 0) 0x7f94fc11baa0 con 0x7f94fc122f70 2019-12-05T05:11:13.957 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:13.937+0000 7f95012e4700 1 --2- 172.21.15.42:0/3475995279 >> v2:172.21.15.42:3300/0 conn(0x7f94fc11ae40 0x7f94fc11b270 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f94fc11c260 tx=0x7f94ec008930).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:13.957 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:13.937+0000 7f95012e4700 1 --2- 172.21.15.42:0/3475995279 >> v2:172.21.15.42:3300/0 conn(0x7f94fc11ae40 0x7f94fc11b270 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f94fc11c260 tx=0x7f94ec008930).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:13.957 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:13.937+0000 7f95012e4700 1 --2- 172.21.15.42:0/3475995279 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f94fc11ae40 0x7f94fc11b270 secure :-1 s=READY pgs=432 cs=0 l=1 rx=0x7f94fc11c260 tx=0x7f94ec008930).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:11:13.957 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:13.937+0000 7f95012e4700 1 --2- 172.21.15.42:0/3475995279 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f94fc11ae40 0x7f94fc11b270 secure :-1 s=THROTTLE_DONE pgs=432 cs=0 l=1 rx=0x7f94fc11c260 tx=0x7f94ec008930).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:13.958 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:13.937+0000 7f94f27fc700 1 -- 172.21.15.42:0/3475995279 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f94ec00f280 con 0x7f94fc11ae40 2019-12-05T05:11:13.958 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:13.937+0000 7f95012e4700 1 --2- 172.21.15.42:0/3475995279 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f94fc11ae40 0x7f94fc11b270 secure :-1 s=THROTTLE_DONE pgs=432 cs=0 l=1 rx=0x7f94fc11c260 tx=0x7f94ec008930).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:13.958 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:13.937+0000 7f95012e4700 1 --2- 172.21.15.42:0/3475995279 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f94fc11ae40 0x7f94fc11b270 secure :-1 s=THROTTLE_DONE pgs=432 cs=0 l=1 rx=0x7f94fc11c260 tx=0x7f94ec008930).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:13.959 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:13.937+0000 7f94f27fc700 1 -- 172.21.15.42:0/3475995279 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94ec011770 con 0x7f94fc11ae40 2019-12-05T05:11:13.959 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:13.937+0000 7f9503548700 1 -- 172.21.15.42:0/3475995279 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94fc1233a0 con 0x7f94fc11ae40 2019-12-05T05:11:13.959 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:13.937+0000 7f94f27fc700 1 -- 172.21.15.42:0/3475995279 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f94ec005540 con 0x7f94fc11ae40 2019-12-05T05:11:13.959 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:13.937+0000 7f9503548700 1 -- 172.21.15.42:0/3475995279 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94fc08ed90 con 0x7f94fc11ae40 2019-12-05T05:11:13.959 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:13.937+0000 7f95012e4700 1 --2- 172.21.15.42:0/3475995279 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f94fc11ae40 0x7f94fc11b270 secure :-1 s=THROTTLE_DONE pgs=432 cs=0 l=1 rx=0x7f94fc11c260 tx=0x7f94ec008930).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:13.960 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:13.937+0000 7f94f27fc700 1 -- 172.21.15.42:0/3475995279 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f94ec005810 con 0x7f94fc11ae40 2019-12-05T05:11:13.960 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:13.937+0000 7f95012e4700 1 --2- 172.21.15.42:0/3475995279 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f94fc11ae40 0x7f94fc11b270 secure :-1 s=THROTTLE_DONE pgs=432 cs=0 l=1 rx=0x7f94fc11c260 tx=0x7f94ec008930).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:13.960 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:13.937+0000 7f94f27fc700 1 --2- 172.21.15.42:0/3475995279 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f94e403cef0 0x7f94e403f400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:11:13.960 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:13.937+0000 7f94f27fc700 1 -- 172.21.15.42:0/3475995279 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6403+0+0 (secure 0 0 0) 0x7f94ec08f390 con 0x7f94fc11ae40 2019-12-05T05:11:13.960 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:13.937+0000 7f9501ae5700 1 --2- 172.21.15.42:0/3475995279 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f94e403cef0 0x7f94e403f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:11:13.961 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:13.941+0000 7f9501ae5700 1 --2- 172.21.15.42:0/3475995279 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f94e403cef0 0x7f94e403f400 crc :-1 s=READY pgs=196 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:11:13.961 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:13.945+0000 7f9503548700 1 -- 172.21.15.42:0/3475995279 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94fc05d110 con 0x7f94fc11ae40 2019-12-05T05:11:13.963 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:13.949+0000 7f95012e4700 1 --2- 172.21.15.42:0/3475995279 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f94fc11ae40 0x7f94fc11b270 secure :-1 s=THROTTLE_DONE pgs=432 cs=0 l=1 rx=0x7f94fc11c260 tx=0x7f94ec008930).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:13.963 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:13.949+0000 7f94f27fc700 1 -- 172.21.15.42:0/3475995279 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f94ec076410 con 0x7f94fc11ae40 2019-12-05T05:11:14.126 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:14.117+0000 7f9503548700 1 -- 172.21.15.42:0/3475995279 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f94fc05b8c0 con 0x7f94fc11ae40 2019-12-05T05:11:14.126 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:14.121+0000 7f95012e4700 1 --2- 172.21.15.42:0/3475995279 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f94fc11ae40 0x7f94fc11b270 secure :-1 s=THROTTLE_DONE pgs=432 cs=0 l=1 rx=0x7f94fc11c260 tx=0x7f94ec008930).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:14.126 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:14.121+0000 7f94f27fc700 1 -- 172.21.15.42:0/3475995279 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) v1 ==== 94+0+2105 (secure 0 0 0) 0x7f94ec076670 con 0x7f94fc11ae40 2019-12-05T05:11:14.126 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 51 2019-12-05T05:11:14.127 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:11:14.127 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":51,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5215,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6835/3176146895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3176146895},{"type":"v1","addr":"172.21.15.42:6835","nonce":3176146895}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":41}],"filesystems":[{"mdsmap":{"epoch":51,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:11:01.313715+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4595},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4595":{"gid":4595,"name":"b","rank":0,"incarnation":36,"state":"up:active","state_seq":11,"addr":"172.21.15.40:6833/211584380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":211584380},{"type":"v1","addr":"172.21.15.40:6833","nonce":211584380}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:11:14.128 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:14.121+0000 7f9503548700 1 -- 172.21.15.42:0/3475995279 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f94e403cef0 msgr2=0x7f94e403f400 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:11:14.128 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:14.121+0000 7f9503548700 1 --2- 172.21.15.42:0/3475995279 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f94e403cef0 0x7f94e403f400 crc :-1 s=READY pgs=196 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:11:14.128 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:14.121+0000 7f9503548700 1 -- 172.21.15.42:0/3475995279 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f94fc11ae40 msgr2=0x7f94fc11b270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:11:14.128 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:14.121+0000 7f9503548700 1 --2- 172.21.15.42:0/3475995279 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f94fc11ae40 0x7f94fc11b270 secure :-1 s=READY pgs=432 cs=0 l=1 rx=0x7f94fc11c260 tx=0x7f94ec008930).stop 2019-12-05T05:11:14.129 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:14.121+0000 7f9503548700 1 -- 172.21.15.42:0/3475995279 shutdown_connections 2019-12-05T05:11:14.129 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:14.121+0000 7f9503548700 1 --2- 172.21.15.42:0/3475995279 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f94e403cef0 0x7f94e403f400 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:11:14.129 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:14.121+0000 7f9503548700 1 --2- 172.21.15.42:0/3475995279 >> v2:172.21.15.40:3300/0 conn(0x7f94fc138530 0x7f94fc13a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:11:14.129 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:14.121+0000 7f9503548700 1 --2- 172.21.15.42:0/3475995279 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f94fc11ae40 0x7f94fc11b270 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:11:14.129 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:14.121+0000 7f9503548700 1 -- 172.21.15.42:0/3475995279 shutdown_connections 2019-12-05T05:11:14.129 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:14.121+0000 7f9503548700 1 -- 172.21.15.42:0/3475995279 wait complete. 2019-12-05T05:11:14.129 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:14.121+0000 7f9503548700 1 -- 172.21.15.42:0/3475995279 >> 172.21.15.42:0/3475995279 conn(0x7f94fc07e560 msgr2=0x7f94fc059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:11:14.138 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T05:11:16.139 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:11:16.247 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:16.233+0000 7f3e5fb1c700 1 Processor -- start 2019-12-05T05:11:16.247 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:16.233+0000 7f3e5fb1c700 1 -- start start 2019-12-05T05:11:16.248 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:16.237+0000 7f3e5fb1c700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f3e5808df80 0x7f3e5809f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:11:16.248 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:16.237+0000 7f3e5fb1c700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3e5809f830 0x7f3e5814efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:11:16.248 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:16.237+0000 7f3e5fb1c700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e5809fc60 con 0x7f3e580a7740 2019-12-05T05:11:16.248 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:16.237+0000 7f3e5fb1c700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e580a0b00 con 0x7f3e5808df80 2019-12-05T05:11:16.248 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:16.237+0000 7f3e5fb1c700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e5808e5b0 con 0x7f3e5809f830 2019-12-05T05:11:16.248 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:16.237+0000 7f3e5e0b9700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f3e580a7740 0x7f3e5807e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:57646/0 (socket says 172.21.15.42:57646) 2019-12-05T05:11:16.248 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:16.237+0000 7f3e5e0b9700 1 -- 172.21.15.42:0/2188017920 learned_addr learned my addr 172.21.15.42:0/2188017920 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:11:16.249 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:16.237+0000 7f3e5e8ba700 1 --2- 172.21.15.42:0/2188017920 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3e5809f830 0x7f3e5814efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:11:16.249 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:16.237+0000 7f3e5d8b8700 1 --2- 172.21.15.42:0/2188017920 >> v2:172.21.15.40:3300/0 conn(0x7f3e5808df80 0x7f3e5809f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:11:16.249 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:16.237+0000 7f3e5d0b7700 1 -- 172.21.15.42:0/2188017920 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 137456796 0 0) 0x7f3e5809fc60 con 0x7f3e580a7740 2019-12-05T05:11:16.249 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:16.237+0000 7f3e5d0b7700 1 -- 172.21.15.42:0/2188017920 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e440030b0 con 0x7f3e580a7740 2019-12-05T05:11:16.249 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:16.237+0000 7f3e5d0b7700 1 -- 172.21.15.42:0/2188017920 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1439159790 0 0) 0x7f3e440030b0 con 0x7f3e580a7740 2019-12-05T05:11:16.249 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:16.237+0000 7f3e5d0b7700 1 -- 172.21.15.42:0/2188017920 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3e5809f830 msgr2=0x7f3e5814efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:11:16.249 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:16.237+0000 7f3e5d0b7700 1 --2- 172.21.15.42:0/2188017920 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3e5809f830 0x7f3e5814efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:11:16.250 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:16.237+0000 7f3e5d0b7700 1 -- 172.21.15.42:0/2188017920 >> v2:172.21.15.40:3300/0 conn(0x7f3e5808df80 msgr2=0x7f3e5809f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:11:16.250 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:16.237+0000 7f3e5d0b7700 1 --2- 172.21.15.42:0/2188017920 >> v2:172.21.15.40:3300/0 conn(0x7f3e5808df80 0x7f3e5809f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:11:16.250 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:16.237+0000 7f3e5d0b7700 1 -- 172.21.15.42:0/2188017920 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e580a0710 con 0x7f3e580a7740 2019-12-05T05:11:16.250 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:16.237+0000 7f3e5d0b7700 1 -- 172.21.15.42:0/2188017920 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7f3e54001750 con 0x7f3e580a7740 2019-12-05T05:11:16.250 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:16.237+0000 7f3e5d0b7700 1 -- 172.21.15.42:0/2188017920 >> v1:172.21.15.42:6789/0 conn(0x7f3e580a7740 legacy=0x7f3e5807e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:11:16.250 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:16.237+0000 7f3e5d0b7700 1 --2- 172.21.15.42:0/2188017920 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e440036e0 0x7f3e44003b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:11:16.250 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:16.237+0000 7f3e5d0b7700 1 --2- 172.21.15.42:0/2188017920 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3e44004420 0x7f3e44004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:11:16.251 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:16.237+0000 7f3e5d0b7700 1 --2- 172.21.15.42:0/2188017920 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3e44004da0 0x7f3e4404f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:11:16.251 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:16.237+0000 7f3e5d0b7700 1 -- 172.21.15.42:0/2188017920 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e580a0b00 con 0x7f3e440036e0 2019-12-05T05:11:16.251 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:16.237+0000 7f3e5d0b7700 1 -- 172.21.15.42:0/2188017920 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e5808e5b0 con 0x7f3e44004420 2019-12-05T05:11:16.251 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:16.237+0000 7f3e5d0b7700 1 -- 172.21.15.42:0/2188017920 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e440051d0 con 0x7f3e44004da0 2019-12-05T05:11:16.251 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:16.237+0000 7f3e5e0b9700 1 --2- 172.21.15.42:0/2188017920 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e440036e0 0x7f3e44003b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:11:16.251 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:16.237+0000 7f3e5d8b8700 1 --2- 172.21.15.42:0/2188017920 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3e44004420 0x7f3e44004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:11:16.251 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:16.237+0000 7f3e5e8ba700 1 --2- 172.21.15.42:0/2188017920 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3e44004da0 0x7f3e4404f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:11:16.252 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:16.237+0000 7f3e5d8b8700 1 -- 172.21.15.42:0/2188017920 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3e44004da0 msgr2=0x7f3e4404f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:11:16.252 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:16.237+0000 7f3e5d8b8700 1 --2- 172.21.15.42:0/2188017920 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3e44004da0 0x7f3e4404f680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:11:16.252 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:16.237+0000 7f3e5d8b8700 1 -- 172.21.15.42:0/2188017920 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e440036e0 msgr2=0x7f3e44003b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:11:16.252 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:16.237+0000 7f3e5d8b8700 1 --2- 172.21.15.42:0/2188017920 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e440036e0 0x7f3e44003b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:11:16.252 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:16.237+0000 7f3e5d8b8700 1 -- 172.21.15.42:0/2188017920 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e4404fc90 con 0x7f3e44004420 2019-12-05T05:11:16.252 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:16.237+0000 7f3e5d8b8700 1 --2- 172.21.15.42:0/2188017920 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3e44004420 0x7f3e44004850 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f3e48002f30 tx=0x7f3e4800ffe0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:16.252 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:16.241+0000 7f3e5d8b8700 1 --2- 172.21.15.42:0/2188017920 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3e44004420 0x7f3e44004850 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f3e48002f30 tx=0x7f3e4800ffe0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:16.253 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:16.241+0000 7f3e5d8b8700 1 --2- 172.21.15.42:0/2188017920 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3e44004420 0x7f3e44004850 secure :-1 s=READY pgs=434 cs=0 l=1 rx=0x7f3e48002f30 tx=0x7f3e4800ffe0).ready entity=mon.1 client_cookie=ea7c8cf98428b2a server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:11:16.253 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:16.241+0000 7f3e5d8b8700 1 --2- 172.21.15.42:0/2188017920 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3e44004420 0x7f3e44004850 secure :-1 s=THROTTLE_DONE pgs=434 cs=0 l=1 rx=0x7f3e48002f30 tx=0x7f3e4800ffe0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:16.253 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:16.241+0000 7f3e5d0b7700 1 -- 172.21.15.42:0/2188017920 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3e48010d10 con 0x7f3e44004420 2019-12-05T05:11:16.253 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:16.241+0000 7f3e5d8b8700 1 --2- 172.21.15.42:0/2188017920 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3e44004420 0x7f3e44004850 secure :-1 s=THROTTLE_DONE pgs=434 cs=0 l=1 rx=0x7f3e48002f30 tx=0x7f3e4800ffe0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:16.253 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:16.241+0000 7f3e5d8b8700 1 --2- 172.21.15.42:0/2188017920 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3e44004420 0x7f3e44004850 secure :-1 s=THROTTLE_DONE pgs=434 cs=0 l=1 rx=0x7f3e48002f30 tx=0x7f3e4800ffe0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:16.253 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:16.241+0000 7f3e5d0b7700 1 -- 172.21.15.42:0/2188017920 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e48010930 con 0x7f3e44004420 2019-12-05T05:11:16.253 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:16.241+0000 7f3e5d0b7700 1 -- 172.21.15.42:0/2188017920 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3e48008bd0 con 0x7f3e44004420 2019-12-05T05:11:16.254 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:16.241+0000 7f3e5fb1c700 1 -- 172.21.15.42:0/2188017920 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3e44004420 msgr2=0x7f3e44004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:11:16.254 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:16.241+0000 7f3e5fb1c700 1 --2- 172.21.15.42:0/2188017920 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3e44004420 0x7f3e44004850 secure :-1 s=READY pgs=434 cs=0 l=1 rx=0x7f3e48002f30 tx=0x7f3e4800ffe0).stop 2019-12-05T05:11:16.264 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:16.241+0000 7f3e5e8ba700 1 -- 172.21.15.42:0/2188017920 reap_dead start 2019-12-05T05:11:16.264 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:16.241+0000 7f3e5fb1c700 1 -- 172.21.15.42:0/2188017920 shutdown_connections 2019-12-05T05:11:16.265 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:16.241+0000 7f3e5fb1c700 1 -- 172.21.15.42:0/2188017920 shutdown_connections 2019-12-05T05:11:16.265 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:16.241+0000 7f3e5fb1c700 1 -- 172.21.15.42:0/2188017920 wait complete. 2019-12-05T05:11:16.265 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:16.241+0000 7f3e5fb1c700 1 -- 172.21.15.42:0/2188017920 >> 172.21.15.42:0/2188017920 conn(0x7f3e580aabe0 msgr2=0x7f3e580a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:11:16.265 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:16.241+0000 7f3e5fb1c700 1 Processor -- start 2019-12-05T05:11:16.265 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:16.241+0000 7f3e5fb1c700 1 -- start start 2019-12-05T05:11:16.268 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:16.245+0000 7f3e5fb1c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3e5811a410 0x7f3e58123be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:11:16.288 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:16.245+0000 7f3e5fb1c700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f3e5808eeb0 0x7f3e5807c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:11:16.288 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:16.245+0000 7f3e5fb1c700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e58124130 con 0x7f3e5807ced0 2019-12-05T05:11:16.288 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:16.245+0000 7f3e5fb1c700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e5811a8e0 con 0x7f3e5808eeb0 2019-12-05T05:11:16.288 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:16.245+0000 7f3e5fb1c700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e5811ab80 con 0x7f3e5811a410 2019-12-05T05:11:16.288 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:16.245+0000 7f3e5e0b9700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3e5811a410 0x7f3e58123be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:11:16.288 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:16.245+0000 7f3e5e0b9700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3e5811a410 0x7f3e58123be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:57620/0 (socket says 172.21.15.42:57620) 2019-12-05T05:11:16.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:16.245+0000 7f3e5e0b9700 1 -- 172.21.15.42:0/3981548966 learned_addr learned my addr 172.21.15.42:0/3981548966 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:11:16.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:16.245+0000 7f3e5d8b8700 1 --2- 172.21.15.42:0/3981548966 >> v2:172.21.15.40:3300/0 conn(0x7f3e5808eeb0 0x7f3e5807c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:11:16.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:16.245+0000 7f3e5e0b9700 1 -- 172.21.15.42:0/3981548966 >> v2:172.21.15.40:3300/0 conn(0x7f3e5808eeb0 msgr2=0x7f3e5807c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:11:16.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:16.245+0000 7f3e5e0b9700 1 --2- 172.21.15.42:0/3981548966 >> v2:172.21.15.40:3300/0 conn(0x7f3e5808eeb0 0x7f3e5807c980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:11:16.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:16.245+0000 7f3e5e0b9700 1 -- 172.21.15.42:0/3981548966 >> v1:172.21.15.40:6789/0 conn(0x7f3e5807ced0 legacy=0x7f3e5807a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:11:16.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:16.245+0000 7f3e5e0b9700 1 -- 172.21.15.42:0/3981548966 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e5807d300 con 0x7f3e5811a410 2019-12-05T05:11:16.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:16.245+0000 7f3e5e0b9700 1 --2- 172.21.15.42:0/3981548966 >> v2:172.21.15.42:3300/0 conn(0x7f3e5811a410 0x7f3e58123be0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f3e54001000 tx=0x7f3e54010c50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:16.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:16.245+0000 7f3e5e0b9700 1 --2- 172.21.15.42:0/3981548966 >> v2:172.21.15.42:3300/0 conn(0x7f3e5811a410 0x7f3e58123be0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f3e54001000 tx=0x7f3e54010c50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:16.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:16.245+0000 7f3e5e0b9700 1 --2- 172.21.15.42:0/3981548966 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3e5811a410 0x7f3e58123be0 secure :-1 s=READY pgs=435 cs=0 l=1 rx=0x7f3e54001000 tx=0x7f3e54010c50).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:11:16.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:16.245+0000 7f3e5e0b9700 1 --2- 172.21.15.42:0/3981548966 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3e5811a410 0x7f3e58123be0 secure :-1 s=THROTTLE_DONE pgs=435 cs=0 l=1 rx=0x7f3e54001000 tx=0x7f3e54010c50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:16.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:16.245+0000 7f3e5e0b9700 1 --2- 172.21.15.42:0/3981548966 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3e5811a410 0x7f3e58123be0 secure :-1 s=THROTTLE_DONE pgs=435 cs=0 l=1 rx=0x7f3e54001000 tx=0x7f3e54010c50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:16.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:16.245+0000 7f3e4effd700 1 -- 172.21.15.42:0/3981548966 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3e5400e5e0 con 0x7f3e5811a410 2019-12-05T05:11:16.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:16.245+0000 7f3e5e0b9700 1 --2- 172.21.15.42:0/3981548966 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3e5811a410 0x7f3e58123be0 secure :-1 s=THROTTLE_DONE pgs=435 cs=0 l=1 rx=0x7f3e54001000 tx=0x7f3e54010c50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:16.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:16.245+0000 7f3e4effd700 1 -- 172.21.15.42:0/3981548966 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e5400e200 con 0x7f3e5811a410 2019-12-05T05:11:16.291 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:16.245+0000 7f3e5fb1c700 1 -- 172.21.15.42:0/3981548966 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e5807b0f0 con 0x7f3e5811a410 2019-12-05T05:11:16.291 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:16.245+0000 7f3e4effd700 1 -- 172.21.15.42:0/3981548966 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3e5400ebc0 con 0x7f3e5811a410 2019-12-05T05:11:16.291 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:16.245+0000 7f3e5fb1c700 1 -- 172.21.15.42:0/3981548966 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e5807b620 con 0x7f3e5811a410 2019-12-05T05:11:16.291 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:16.249+0000 7f3e5e0b9700 1 --2- 172.21.15.42:0/3981548966 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3e5811a410 0x7f3e58123be0 secure :-1 s=THROTTLE_DONE pgs=435 cs=0 l=1 rx=0x7f3e54001000 tx=0x7f3e54010c50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:16.291 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:16.249+0000 7f3e4effd700 1 -- 172.21.15.42:0/3981548966 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f3e54009260 con 0x7f3e5811a410 2019-12-05T05:11:16.291 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:16.249+0000 7f3e5e0b9700 1 --2- 172.21.15.42:0/3981548966 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3e5811a410 0x7f3e58123be0 secure :-1 s=THROTTLE_DONE pgs=435 cs=0 l=1 rx=0x7f3e54001000 tx=0x7f3e54010c50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:16.291 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:16.249+0000 7f3e4effd700 1 --2- 172.21.15.42:0/3981548966 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3e4003ad20 0x7f3e4003d230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:11:16.291 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:16.249+0000 7f3e4effd700 1 -- 172.21.15.42:0/3981548966 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6403+0+0 (secure 0 0 0) 0x7f3e5404b430 con 0x7f3e5811a410 2019-12-05T05:11:16.292 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:16.249+0000 7f3e5d8b8700 1 --2- 172.21.15.42:0/3981548966 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3e4003ad20 0x7f3e4003d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:11:16.292 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:16.249+0000 7f3e5d8b8700 1 --2- 172.21.15.42:0/3981548966 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3e4003ad20 0x7f3e4003d230 crc :-1 s=READY pgs=197 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:11:16.292 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:16.249+0000 7f3e5fb1c700 1 -- 172.21.15.42:0/3981548966 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e5805d110 con 0x7f3e5811a410 2019-12-05T05:11:16.292 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:16.257+0000 7f3e5e0b9700 1 --2- 172.21.15.42:0/3981548966 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3e5811a410 0x7f3e58123be0 secure :-1 s=THROTTLE_DONE pgs=435 cs=0 l=1 rx=0x7f3e54001000 tx=0x7f3e54010c50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:16.292 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:16.257+0000 7f3e4effd700 1 -- 172.21.15.42:0/3981548966 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f3e540324b0 con 0x7f3e5811a410 2019-12-05T05:11:16.430 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:16.425+0000 7f3e5fb1c700 1 -- 172.21.15.42:0/3981548966 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3e5807f090 con 0x7f3e5811a410 2019-12-05T05:11:16.431 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:16.425+0000 7f3e5e0b9700 1 --2- 172.21.15.42:0/3981548966 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3e5811a410 0x7f3e58123be0 secure :-1 s=THROTTLE_DONE pgs=435 cs=0 l=1 rx=0x7f3e54001000 tx=0x7f3e54010c50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:16.432 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:16.425+0000 7f3e4effd700 1 -- 172.21.15.42:0/3981548966 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) v1 ==== 94+0+2105 (secure 0 0 0) 0x7f3e54032710 con 0x7f3e5811a410 2019-12-05T05:11:16.433 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 51 2019-12-05T05:11:16.433 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:11:16.433 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":51,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5215,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6835/3176146895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3176146895},{"type":"v1","addr":"172.21.15.42:6835","nonce":3176146895}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":41}],"filesystems":[{"mdsmap":{"epoch":51,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:11:01.313715+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4595},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4595":{"gid":4595,"name":"b","rank":0,"incarnation":36,"state":"up:active","state_seq":11,"addr":"172.21.15.40:6833/211584380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":211584380},{"type":"v1","addr":"172.21.15.40:6833","nonce":211584380}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:11:16.434 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:16.429+0000 7f3e5fb1c700 1 -- 172.21.15.42:0/3981548966 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3e4003ad20 msgr2=0x7f3e4003d230 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:11:16.435 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:16.429+0000 7f3e5fb1c700 1 --2- 172.21.15.42:0/3981548966 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3e4003ad20 0x7f3e4003d230 crc :-1 s=READY pgs=197 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:11:16.435 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:16.429+0000 7f3e5fb1c700 1 -- 172.21.15.42:0/3981548966 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3e5811a410 msgr2=0x7f3e58123be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:11:16.435 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:16.429+0000 7f3e5fb1c700 1 --2- 172.21.15.42:0/3981548966 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3e5811a410 0x7f3e58123be0 secure :-1 s=READY pgs=435 cs=0 l=1 rx=0x7f3e54001000 tx=0x7f3e54010c50).stop 2019-12-05T05:11:16.435 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:16.429+0000 7f3e5fb1c700 1 -- 172.21.15.42:0/3981548966 shutdown_connections 2019-12-05T05:11:16.436 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:16.429+0000 7f3e5fb1c700 1 --2- 172.21.15.42:0/3981548966 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3e4003ad20 0x7f3e4003d230 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:11:16.436 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:16.429+0000 7f3e5fb1c700 1 --2- 172.21.15.42:0/3981548966 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3e5811a410 0x7f3e58123be0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:11:16.436 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:16.429+0000 7f3e5fb1c700 1 --2- 172.21.15.42:0/3981548966 >> v2:172.21.15.40:3300/0 conn(0x7f3e5808eeb0 0x7f3e5807c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:11:16.436 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:16.429+0000 7f3e5fb1c700 1 -- 172.21.15.42:0/3981548966 shutdown_connections 2019-12-05T05:11:16.436 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:16.429+0000 7f3e5fb1c700 1 -- 172.21.15.42:0/3981548966 wait complete. 2019-12-05T05:11:16.437 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:16.429+0000 7f3e5fb1c700 1 -- 172.21.15.42:0/3981548966 >> 172.21.15.42:0/3981548966 conn(0x7f3e5808fde0 msgr2=0x7f3e58059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:11:16.452 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T05:11:18.453 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:11:18.553 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.544+0000 7fdadd53a700 1 Processor -- start 2019-12-05T05:11:18.553 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.544+0000 7fdadd53a700 1 -- start start 2019-12-05T05:11:18.553 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.544+0000 7fdadd53a700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fdad808df80 0x7fdad809f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:11:18.553 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.544+0000 7fdadd53a700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fdad809f830 0x7fdad814efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:11:18.553 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.544+0000 7fdadd53a700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdad809fc60 con 0x7fdad80a7740 2019-12-05T05:11:18.553 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.544+0000 7fdadd53a700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdad80a0b00 con 0x7fdad808df80 2019-12-05T05:11:18.554 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.544+0000 7fdadd53a700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdad808e5b0 con 0x7fdad809f830 2019-12-05T05:11:18.554 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.544+0000 7fdad7fff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fdad809f830 0x7fdad814efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:11:18.554 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.544+0000 7fdad7fff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fdad809f830 0x7fdad814efe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:57646/0 (socket says 172.21.15.42:57646) 2019-12-05T05:11:18.554 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.544+0000 7fdad7fff700 1 -- 172.21.15.42:0/135276627 learned_addr learned my addr 172.21.15.42:0/135276627 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:11:18.554 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.544+0000 7fdad6ffd700 1 --2- 172.21.15.42:0/135276627 >> v2:172.21.15.40:3300/0 conn(0x7fdad808df80 0x7fdad809f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:11:18.554 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.544+0000 7fdad7fff700 1 -- 172.21.15.42:0/135276627 >> v2:172.21.15.40:3300/0 conn(0x7fdad808df80 msgr2=0x7fdad809f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:11:18.554 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.544+0000 7fdad7fff700 1 --2- 172.21.15.42:0/135276627 >> v2:172.21.15.40:3300/0 conn(0x7fdad808df80 0x7fdad809f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:11:18.555 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.544+0000 7fdad7fff700 1 -- 172.21.15.42:0/135276627 >> v1:172.21.15.40:6789/0 conn(0x7fdad80a7740 legacy=0x7fdad807e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:11:18.555 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.544+0000 7fdad7fff700 1 -- 172.21.15.42:0/135276627 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdad80a0710 con 0x7fdad809f830 2019-12-05T05:11:18.555 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.544+0000 7fdad7fff700 1 --2- 172.21.15.42:0/135276627 >> v2:172.21.15.42:3300/0 conn(0x7fdad809f830 0x7fdad814efe0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fdac80032b0 tx=0x7fdac80100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:18.555 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.544+0000 7fdad7fff700 1 --2- 172.21.15.42:0/135276627 >> v2:172.21.15.42:3300/0 conn(0x7fdad809f830 0x7fdad814efe0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fdac80032b0 tx=0x7fdac80100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:18.555 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.544+0000 7fdad7fff700 1 --2- 172.21.15.42:0/135276627 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdad809f830 0x7fdad814efe0 secure :-1 s=READY pgs=436 cs=0 l=1 rx=0x7fdac80032b0 tx=0x7fdac80100b0).ready entity=mon.1 client_cookie=417d0d0debd4f82c server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:11:18.555 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.544+0000 7fdad7fff700 1 --2- 172.21.15.42:0/135276627 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdad809f830 0x7fdad814efe0 secure :-1 s=THROTTLE_DONE pgs=436 cs=0 l=1 rx=0x7fdac80032b0 tx=0x7fdac80100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:18.555 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.544+0000 7fdad7fff700 1 --2- 172.21.15.42:0/135276627 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdad809f830 0x7fdad814efe0 secure :-1 s=THROTTLE_DONE pgs=436 cs=0 l=1 rx=0x7fdac80032b0 tx=0x7fdac80100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:18.556 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.544+0000 7fdad67fc700 1 -- 172.21.15.42:0/135276627 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdac8010d60 con 0x7fdad809f830 2019-12-05T05:11:18.556 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.544+0000 7fdad7fff700 1 --2- 172.21.15.42:0/135276627 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdad809f830 0x7fdad814efe0 secure :-1 s=THROTTLE_DONE pgs=436 cs=0 l=1 rx=0x7fdac80032b0 tx=0x7fdac80100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:18.556 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.544+0000 7fdad67fc700 1 -- 172.21.15.42:0/135276627 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdac8010980 con 0x7fdad809f830 2019-12-05T05:11:18.556 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.544+0000 7fdad67fc700 1 -- 172.21.15.42:0/135276627 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdac80114a0 con 0x7fdad809f830 2019-12-05T05:11:18.556 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.544+0000 7fdadd53a700 1 -- 172.21.15.42:0/135276627 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdad809f830 msgr2=0x7fdad814efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:11:18.556 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.544+0000 7fdadd53a700 1 --2- 172.21.15.42:0/135276627 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdad809f830 0x7fdad814efe0 secure :-1 s=READY pgs=436 cs=0 l=1 rx=0x7fdac80032b0 tx=0x7fdac80100b0).stop 2019-12-05T05:11:18.556 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.548+0000 7fdadd53a700 1 -- 172.21.15.42:0/135276627 shutdown_connections 2019-12-05T05:11:18.557 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.548+0000 7fdadd53a700 1 --2- 172.21.15.42:0/135276627 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdad809f830 0x7fdad814efe0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:11:18.557 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.548+0000 7fdadd53a700 1 --2- 172.21.15.42:0/135276627 >> v2:172.21.15.40:3300/0 conn(0x7fdad808df80 0x7fdad809f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:11:18.557 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.548+0000 7fdadd53a700 1 -- 172.21.15.42:0/135276627 shutdown_connections 2019-12-05T05:11:18.557 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.548+0000 7fdadd53a700 1 -- 172.21.15.42:0/135276627 wait complete. 2019-12-05T05:11:18.557 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.548+0000 7fdadd53a700 1 -- 172.21.15.42:0/135276627 >> 172.21.15.42:0/135276627 conn(0x7fdad80aabe0 msgr2=0x7fdad80a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:11:18.561 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.548+0000 7fdadd53a700 1 Processor -- start 2019-12-05T05:11:18.561 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.548+0000 7fdadd53a700 1 -- start start 2019-12-05T05:11:18.561 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.548+0000 7fdadd53a700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdad811b050 0x7fdad80a22a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:11:18.561 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.548+0000 7fdadd53a700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fdad80a27f0 0x7fdad8126a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:11:18.562 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.548+0000 7fdadd53a700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdad8126fe0 con 0x7fdad80a2e50 2019-12-05T05:11:18.563 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.548+0000 7fdadd53a700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdad8123700 con 0x7fdad80a27f0 2019-12-05T05:11:18.563 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.548+0000 7fdadd53a700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdad81239a0 con 0x7fdad811b050 2019-12-05T05:11:18.563 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.548+0000 7fdad6ffd700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdad811b050 0x7fdad80a22a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:11:18.563 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.548+0000 7fdad7fff700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fdad80a27f0 0x7fdad8126a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:11:18.563 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.548+0000 7fdad6ffd700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdad811b050 0x7fdad80a22a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:58116/0 (socket says 172.21.15.42:58116) 2019-12-05T05:11:18.563 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.548+0000 7fdad7fff700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fdad80a27f0 0x7fdad8126a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.42:45468/0 (socket says 172.21.15.42:45468) 2019-12-05T05:11:18.563 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.548+0000 7fdad6ffd700 1 -- 172.21.15.42:0/1233057214 learned_addr learned my addr 172.21.15.42:0/1233057214 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:11:18.564 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.552+0000 7fdad6ffd700 1 -- 172.21.15.42:0/1233057214 >> v2:172.21.15.40:3300/0 conn(0x7fdad80a27f0 msgr2=0x7fdad8126a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:11:18.564 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.552+0000 7fdad6ffd700 1 --2- 172.21.15.42:0/1233057214 >> v2:172.21.15.40:3300/0 conn(0x7fdad80a27f0 0x7fdad8126a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:11:18.564 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.552+0000 7fdad6ffd700 1 -- 172.21.15.42:0/1233057214 >> v1:172.21.15.40:6789/0 conn(0x7fdad80a2e50 legacy=0x7fdad8123c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:11:18.564 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.552+0000 7fdad6ffd700 1 -- 172.21.15.42:0/1233057214 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdad8127210 con 0x7fdad811b050 2019-12-05T05:11:18.564 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.552+0000 7fdad6ffd700 1 --2- 172.21.15.42:0/1233057214 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdad811b050 0x7fdad80a22a0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fdac0007980 tx=0x7fdac0010bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:18.568 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.552+0000 7fdad6ffd700 1 --2- 172.21.15.42:0/1233057214 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdad811b050 0x7fdad80a22a0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fdac0007980 tx=0x7fdac0010bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:18.568 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.552+0000 7fdad6ffd700 1 --2- 172.21.15.42:0/1233057214 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdad811b050 0x7fdad80a22a0 secure :-1 s=READY pgs=85 cs=0 l=1 rx=0x7fdac0007980 tx=0x7fdac0010bb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:11:18.569 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.552+0000 7fdad6ffd700 1 --2- 172.21.15.42:0/1233057214 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdad811b050 0x7fdad80a22a0 secure :-1 s=THROTTLE_DONE pgs=85 cs=0 l=1 rx=0x7fdac0007980 tx=0x7fdac0010bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:18.569 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.552+0000 7fdacffff700 1 -- 172.21.15.42:0/1233057214 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdac0005840 con 0x7fdad811b050 2019-12-05T05:11:18.569 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.552+0000 7fdad6ffd700 1 --2- 172.21.15.42:0/1233057214 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdad811b050 0x7fdad80a22a0 secure :-1 s=THROTTLE_DONE pgs=85 cs=0 l=1 rx=0x7fdac0007980 tx=0x7fdac0010bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:18.570 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.552+0000 7fdad6ffd700 1 --2- 172.21.15.42:0/1233057214 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdad811b050 0x7fdad80a22a0 secure :-1 s=THROTTLE_DONE pgs=85 cs=0 l=1 rx=0x7fdac0007980 tx=0x7fdac0010bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:18.570 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.552+0000 7fdacffff700 1 -- 172.21.15.42:0/1233057214 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdac0007160 con 0x7fdad811b050 2019-12-05T05:11:18.570 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.552+0000 7fdadd53a700 1 -- 172.21.15.42:0/1233057214 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdad81274e0 con 0x7fdad811b050 2019-12-05T05:11:18.570 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.552+0000 7fdacffff700 1 -- 172.21.15.42:0/1233057214 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdac0012230 con 0x7fdad811b050 2019-12-05T05:11:18.570 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.552+0000 7fdadd53a700 1 -- 172.21.15.42:0/1233057214 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdad812e260 con 0x7fdad811b050 2019-12-05T05:11:18.570 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.552+0000 7fdad6ffd700 1 --2- 172.21.15.42:0/1233057214 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdad811b050 0x7fdad80a22a0 secure :-1 s=THROTTLE_DONE pgs=85 cs=0 l=1 rx=0x7fdac0007980 tx=0x7fdac0010bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:18.570 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.552+0000 7fdacffff700 1 -- 172.21.15.42:0/1233057214 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fdac0007bc0 con 0x7fdad811b050 2019-12-05T05:11:18.571 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.552+0000 7fdad6ffd700 1 --2- 172.21.15.42:0/1233057214 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdad811b050 0x7fdad80a22a0 secure :-1 s=THROTTLE_DONE pgs=85 cs=0 l=1 rx=0x7fdac0007980 tx=0x7fdac0010bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:18.571 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.552+0000 7fdacffff700 1 --2- 172.21.15.42:0/1233057214 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fdab003ad20 0x7fdab003d230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:11:18.571 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.552+0000 7fdacffff700 1 -- 172.21.15.42:0/1233057214 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6461+0+0 (secure 0 0 0) 0x7fdac004af80 con 0x7fdad811b050 2019-12-05T05:11:18.571 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.552+0000 7fdad77fe700 1 --2- 172.21.15.42:0/1233057214 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fdab003ad20 0x7fdab003d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:11:18.571 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.552+0000 7fdad77fe700 1 --2- 172.21.15.42:0/1233057214 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fdab003ad20 0x7fdab003d230 crc :-1 s=READY pgs=198 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:11:18.574 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.560+0000 7fdadd53a700 1 -- 172.21.15.42:0/1233057214 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdad8124440 con 0x7fdad811b050 2019-12-05T05:11:18.574 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.568+0000 7fdad6ffd700 1 --2- 172.21.15.42:0/1233057214 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdad811b050 0x7fdad80a22a0 secure :-1 s=THROTTLE_DONE pgs=85 cs=0 l=1 rx=0x7fdac0007980 tx=0x7fdac0010bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:18.574 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.568+0000 7fdacffff700 1 -- 172.21.15.42:0/1233057214 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fdac0032000 con 0x7fdad811b050 2019-12-05T05:11:18.746 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.736+0000 7fdadd53a700 1 -- 172.21.15.42:0/1233057214 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdad805d110 con 0x7fdad811b050 2019-12-05T05:11:18.747 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.736+0000 7fdad6ffd700 1 --2- 172.21.15.42:0/1233057214 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdad811b050 0x7fdad80a22a0 secure :-1 s=THROTTLE_DONE pgs=85 cs=0 l=1 rx=0x7fdac0007980 tx=0x7fdac0010bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:18.747 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.736+0000 7fdacffff700 1 -- 172.21.15.42:0/1233057214 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 52 v52) v1 ==== 94+0+1771 (secure 0 0 0) 0x7fdac0032260 con 0x7fdad811b050 2019-12-05T05:11:18.747 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 52 2019-12-05T05:11:18.747 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:11:18.747 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":52,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":52,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:11:18.296472+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5215},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5215":{"gid":5215,"name":"a","rank":0,"incarnation":52,"state":"up:replay","state_seq":1,"addr":"172.21.15.42:6835/3176146895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3176146895},{"type":"v1","addr":"172.21.15.42:6835","nonce":3176146895}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:11:18.748 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.744+0000 7fdadd53a700 1 -- 172.21.15.42:0/1233057214 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fdab003ad20 msgr2=0x7fdab003d230 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:11:18.749 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.744+0000 7fdadd53a700 1 --2- 172.21.15.42:0/1233057214 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fdab003ad20 0x7fdab003d230 crc :-1 s=READY pgs=198 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:11:18.749 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.744+0000 7fdadd53a700 1 -- 172.21.15.42:0/1233057214 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdad811b050 msgr2=0x7fdad80a22a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:11:18.749 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.744+0000 7fdadd53a700 1 --2- 172.21.15.42:0/1233057214 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdad811b050 0x7fdad80a22a0 secure :-1 s=READY pgs=85 cs=0 l=1 rx=0x7fdac0007980 tx=0x7fdac0010bb0).stop 2019-12-05T05:11:18.749 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.744+0000 7fdadd53a700 1 -- 172.21.15.42:0/1233057214 shutdown_connections 2019-12-05T05:11:18.749 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.744+0000 7fdadd53a700 1 --2- 172.21.15.42:0/1233057214 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fdab003ad20 0x7fdab003d230 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:11:18.749 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.744+0000 7fdadd53a700 1 --2- 172.21.15.42:0/1233057214 >> v2:172.21.15.40:3300/0 conn(0x7fdad80a27f0 0x7fdad8126a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:11:18.750 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.744+0000 7fdadd53a700 1 --2- 172.21.15.42:0/1233057214 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdad811b050 0x7fdad80a22a0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:11:18.750 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.744+0000 7fdadd53a700 1 -- 172.21.15.42:0/1233057214 shutdown_connections 2019-12-05T05:11:18.750 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.744+0000 7fdadd53a700 1 -- 172.21.15.42:0/1233057214 wait complete. 2019-12-05T05:11:18.751 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.744+0000 7fdadd53a700 1 -- 172.21.15.42:0/1233057214 >> 172.21.15.42:0/1233057214 conn(0x7fdad80a82a0 msgr2=0x7fdad807cea0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:11:18.767 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b down, removed from mdsmap 2019-12-05T05:11:18.767 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2019-12-05T05:11:18.768 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:11:18.878 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.856+0000 7ff38ee29700 1 Processor -- start 2019-12-05T05:11:18.878 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.856+0000 7ff38ee29700 1 -- start start 2019-12-05T05:11:18.878 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.856+0000 7ff38ee29700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff38808df80 0x7ff38809f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:11:18.879 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.856+0000 7ff38ee29700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff38809f830 0x7ff38814efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:11:18.879 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.856+0000 7ff38ee29700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff38809fc60 con 0x7ff3880a7740 2019-12-05T05:11:18.879 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.856+0000 7ff38ee29700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3880a0b00 con 0x7ff38809f830 2019-12-05T05:11:18.879 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.856+0000 7ff38ee29700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff38808e5b0 con 0x7ff38808df80 2019-12-05T05:11:18.879 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.860+0000 7ff38cbc5700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff38808df80 0x7ff38809f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:11:18.879 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.860+0000 7ff38dbc7700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff38809f830 0x7ff38814efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:11:18.879 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.860+0000 7ff38cbc5700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff38808df80 0x7ff38809f2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:58126/0 (socket says 172.21.15.42:58126) 2019-12-05T05:11:18.880 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.860+0000 7ff38cbc5700 1 -- 172.21.15.42:0/549535493 learned_addr learned my addr 172.21.15.42:0/549535493 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:11:18.880 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.860+0000 7ff38dbc7700 1 -- 172.21.15.42:0/549535493 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff38808df80 msgr2=0x7ff38809f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:11:18.880 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.860+0000 7ff38dbc7700 1 --2- 172.21.15.42:0/549535493 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff38808df80 0x7ff38809f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:11:18.880 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.860+0000 7ff38dbc7700 1 -- 172.21.15.42:0/549535493 >> v1:172.21.15.40:6789/0 conn(0x7ff3880a7740 legacy=0x7ff38807e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:11:18.880 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.860+0000 7ff38dbc7700 1 -- 172.21.15.42:0/549535493 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3880a0710 con 0x7ff38809f830 2019-12-05T05:11:18.880 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.860+0000 7ff38dbc7700 1 --2- 172.21.15.42:0/549535493 >> v2:172.21.15.40:3300/0 conn(0x7ff38809f830 0x7ff38814efe0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7ff380008b10 tx=0x7ff38000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:18.880 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.860+0000 7ff38dbc7700 1 --2- 172.21.15.42:0/549535493 >> v2:172.21.15.40:3300/0 conn(0x7ff38809f830 0x7ff38814efe0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7ff380008b10 tx=0x7ff38000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:18.881 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.860+0000 7ff38dbc7700 1 --2- 172.21.15.42:0/549535493 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff38809f830 0x7ff38814efe0 secure :-1 s=READY pgs=343 cs=0 l=1 rx=0x7ff380008b10 tx=0x7ff38000fff0).ready entity=mon.0 client_cookie=a3db4fbc1734f59c server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:11:18.881 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.860+0000 7ff38dbc7700 1 --2- 172.21.15.42:0/549535493 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff38809f830 0x7ff38814efe0 secure :-1 s=THROTTLE_DONE pgs=343 cs=0 l=1 rx=0x7ff380008b10 tx=0x7ff38000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:18.881 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.860+0000 7ff37ffff700 1 -- 172.21.15.42:0/549535493 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff380010900 con 0x7ff38809f830 2019-12-05T05:11:18.881 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.860+0000 7ff38dbc7700 1 --2- 172.21.15.42:0/549535493 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff38809f830 0x7ff38814efe0 secure :-1 s=THROTTLE_DONE pgs=343 cs=0 l=1 rx=0x7ff380008b10 tx=0x7ff38000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:18.881 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.860+0000 7ff37ffff700 1 -- 172.21.15.42:0/549535493 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff380008b50 con 0x7ff38809f830 2019-12-05T05:11:18.881 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.860+0000 7ff38dbc7700 1 --2- 172.21.15.42:0/549535493 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff38809f830 0x7ff38814efe0 secure :-1 s=THROTTLE_DONE pgs=343 cs=0 l=1 rx=0x7ff380008b10 tx=0x7ff38000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:18.882 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.860+0000 7ff38ee29700 1 -- 172.21.15.42:0/549535493 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff38809f830 msgr2=0x7ff38814efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:11:18.882 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.860+0000 7ff37ffff700 1 -- 172.21.15.42:0/549535493 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff3800110c0 con 0x7ff38809f830 2019-12-05T05:11:18.882 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.860+0000 7ff38ee29700 1 --2- 172.21.15.42:0/549535493 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff38809f830 0x7ff38814efe0 secure :-1 s=READY pgs=343 cs=0 l=1 rx=0x7ff380008b10 tx=0x7ff38000fff0).stop 2019-12-05T05:11:18.882 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.860+0000 7ff38ee29700 1 -- 172.21.15.42:0/549535493 shutdown_connections 2019-12-05T05:11:18.882 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.860+0000 7ff38ee29700 1 --2- 172.21.15.42:0/549535493 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff38809f830 0x7ff38814efe0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:11:18.882 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.860+0000 7ff38ee29700 1 --2- 172.21.15.42:0/549535493 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff38808df80 0x7ff38809f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:11:18.882 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.860+0000 7ff38ee29700 1 -- 172.21.15.42:0/549535493 shutdown_connections 2019-12-05T05:11:18.883 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.860+0000 7ff38ee29700 1 -- 172.21.15.42:0/549535493 wait complete. 2019-12-05T05:11:18.883 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.860+0000 7ff38ee29700 1 -- 172.21.15.42:0/549535493 >> 172.21.15.42:0/549535493 conn(0x7ff3880aabe0 msgr2=0x7ff3880a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:11:18.883 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.860+0000 7ff38ee29700 1 Processor -- start 2019-12-05T05:11:18.883 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.860+0000 7ff38ee29700 1 -- start start 2019-12-05T05:11:18.883 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.864+0000 7ff38ee29700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff3881385a0 0x7ff38811e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:11:18.884 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.864+0000 7ff38ee29700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff38812b800 0x7ff38812bc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:11:18.884 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.864+0000 7ff38ee29700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff388120460 con 0x7ff388117590 2019-12-05T05:11:18.884 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.864+0000 7ff38ee29700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3881207e0 con 0x7ff3881385a0 2019-12-05T05:11:18.885 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.864+0000 7ff38ee29700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff388120a80 con 0x7ff38812b800 2019-12-05T05:11:18.885 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.864+0000 7ff38dbc7700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff38812b800 0x7ff38812bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:11:18.885 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.864+0000 7ff38dbc7700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff38812b800 0x7ff38812bc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:58134/0 (socket says 172.21.15.42:58134) 2019-12-05T05:11:18.885 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.864+0000 7ff38dbc7700 1 -- 172.21.15.42:0/4249989244 learned_addr learned my addr 172.21.15.42:0/4249989244 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:11:18.885 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.864+0000 7ff38cbc5700 1 --2- 172.21.15.42:0/4249989244 >> v2:172.21.15.40:3300/0 conn(0x7ff3881385a0 0x7ff38811e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:11:18.885 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.864+0000 7ff38dbc7700 1 -- 172.21.15.42:0/4249989244 >> v2:172.21.15.40:3300/0 conn(0x7ff3881385a0 msgr2=0x7ff38811e9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:11:18.885 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.864+0000 7ff38dbc7700 1 --2- 172.21.15.42:0/4249989244 >> v2:172.21.15.40:3300/0 conn(0x7ff3881385a0 0x7ff38811e9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:11:18.886 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.864+0000 7ff38dbc7700 1 -- 172.21.15.42:0/4249989244 >> v1:172.21.15.40:6789/0 conn(0x7ff388117590 legacy=0x7ff388137da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:11:18.886 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.864+0000 7ff38dbc7700 1 -- 172.21.15.42:0/4249989244 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff388118a50 con 0x7ff38812b800 2019-12-05T05:11:18.886 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.864+0000 7ff38cbc5700 1 --2- 172.21.15.42:0/4249989244 >> v2:172.21.15.40:3300/0 conn(0x7ff3881385a0 0x7ff38811e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:11:18.886 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.864+0000 7ff38dbc7700 1 --2- 172.21.15.42:0/4249989244 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff38812b800 0x7ff38812bc30 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7ff380008b10 tx=0x7ff380010cf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:18.887 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.864+0000 7ff38dbc7700 1 --2- 172.21.15.42:0/4249989244 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff38812b800 0x7ff38812bc30 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7ff380008b10 tx=0x7ff380010cf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:18.887 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.864+0000 7ff38dbc7700 1 --2- 172.21.15.42:0/4249989244 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff38812b800 0x7ff38812bc30 secure :-1 s=READY pgs=86 cs=0 l=1 rx=0x7ff380008b10 tx=0x7ff380010cf0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:11:18.887 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.864+0000 7ff38dbc7700 1 --2- 172.21.15.42:0/4249989244 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff38812b800 0x7ff38812bc30 secure :-1 s=THROTTLE_DONE pgs=86 cs=0 l=1 rx=0x7ff380008b10 tx=0x7ff380010cf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:18.887 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.864+0000 7ff37dffb700 1 -- 172.21.15.42:0/4249989244 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff380005ab0 con 0x7ff38812b800 2019-12-05T05:11:18.887 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.864+0000 7ff38dbc7700 1 --2- 172.21.15.42:0/4249989244 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff38812b800 0x7ff38812bc30 secure :-1 s=THROTTLE_DONE pgs=86 cs=0 l=1 rx=0x7ff380008b10 tx=0x7ff380010cf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:18.888 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.864+0000 7ff37dffb700 1 -- 172.21.15.42:0/4249989244 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff380010b00 con 0x7ff38812b800 2019-12-05T05:11:18.888 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.864+0000 7ff38dbc7700 1 --2- 172.21.15.42:0/4249989244 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff38812b800 0x7ff38812bc30 secure :-1 s=THROTTLE_DONE pgs=86 cs=0 l=1 rx=0x7ff380008b10 tx=0x7ff380010cf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:18.888 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.864+0000 7ff38ee29700 1 -- 172.21.15.42:0/4249989244 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff38812c180 con 0x7ff38812b800 2019-12-05T05:11:18.895 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.864+0000 7ff37dffb700 1 -- 172.21.15.42:0/4249989244 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff38000f680 con 0x7ff38812b800 2019-12-05T05:11:18.895 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.864+0000 7ff38ee29700 1 -- 172.21.15.42:0/4249989244 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff388132bc0 con 0x7ff38812b800 2019-12-05T05:11:18.895 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.864+0000 7ff38dbc7700 1 --2- 172.21.15.42:0/4249989244 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff38812b800 0x7ff38812bc30 secure :-1 s=THROTTLE_DONE pgs=86 cs=0 l=1 rx=0x7ff380008b10 tx=0x7ff380010cf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:18.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.864+0000 7ff37dffb700 1 -- 172.21.15.42:0/4249989244 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7ff380011360 con 0x7ff38812b800 2019-12-05T05:11:18.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.864+0000 7ff38dbc7700 1 --2- 172.21.15.42:0/4249989244 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff38812b800 0x7ff38812bc30 secure :-1 s=THROTTLE_DONE pgs=86 cs=0 l=1 rx=0x7ff380008b10 tx=0x7ff380010cf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:18.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.864+0000 7ff37dffb700 1 --2- 172.21.15.42:0/4249989244 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff36803ad70 0x7ff36803d280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:11:18.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.864+0000 7ff37dffb700 1 -- 172.21.15.42:0/4249989244 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6461+0+0 (secure 0 0 0) 0x7ff38004b140 con 0x7ff38812b800 2019-12-05T05:11:18.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.864+0000 7ff38d3c6700 1 --2- 172.21.15.42:0/4249989244 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff36803ad70 0x7ff36803d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:11:18.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.864+0000 7ff38d3c6700 1 --2- 172.21.15.42:0/4249989244 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff36803ad70 0x7ff36803d280 crc :-1 s=READY pgs=199 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:11:18.897 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.872+0000 7ff38ee29700 1 -- 172.21.15.42:0/4249989244 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff38811ef40 con 0x7ff38812b800 2019-12-05T05:11:18.897 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.880+0000 7ff38dbc7700 1 --2- 172.21.15.42:0/4249989244 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff38812b800 0x7ff38812bc30 secure :-1 s=THROTTLE_DONE pgs=86 cs=0 l=1 rx=0x7ff380008b10 tx=0x7ff380010cf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:18.897 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:18.880+0000 7ff37dffb700 1 -- 172.21.15.42:0/4249989244 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7ff38000f950 con 0x7ff38812b800 2019-12-05T05:11:19.048 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:19.044+0000 7ff38ee29700 1 -- 172.21.15.42:0/4249989244 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff38805d110 con 0x7ff38812b800 2019-12-05T05:11:19.049 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:19.044+0000 7ff38dbc7700 1 --2- 172.21.15.42:0/4249989244 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff38812b800 0x7ff38812bc30 secure :-1 s=THROTTLE_DONE pgs=86 cs=0 l=1 rx=0x7ff380008b10 tx=0x7ff380010cf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:19.049 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:19.044+0000 7ff37dffb700 1 -- 172.21.15.42:0/4249989244 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 52 v52) v1 ==== 94+0+1771 (secure 0 0 0) 0x7ff380011c60 con 0x7ff38812b800 2019-12-05T05:11:19.050 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 52 2019-12-05T05:11:19.050 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:11:19.050 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":52,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":52,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:11:18.296472+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5215},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5215":{"gid":5215,"name":"a","rank":0,"incarnation":52,"state":"up:replay","state_seq":1,"addr":"172.21.15.42:6835/3176146895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3176146895},{"type":"v1","addr":"172.21.15.42:6835","nonce":3176146895}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:11:19.053 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:19.048+0000 7ff38ee29700 1 -- 172.21.15.42:0/4249989244 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff36803ad70 msgr2=0x7ff36803d280 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:11:19.053 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:19.048+0000 7ff38ee29700 1 --2- 172.21.15.42:0/4249989244 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff36803ad70 0x7ff36803d280 crc :-1 s=READY pgs=199 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:11:19.053 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:19.048+0000 7ff38ee29700 1 -- 172.21.15.42:0/4249989244 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff38812b800 msgr2=0x7ff38812bc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:11:19.054 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:19.048+0000 7ff38ee29700 1 --2- 172.21.15.42:0/4249989244 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff38812b800 0x7ff38812bc30 secure :-1 s=READY pgs=86 cs=0 l=1 rx=0x7ff380008b10 tx=0x7ff380010cf0).stop 2019-12-05T05:11:19.054 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:19.048+0000 7ff38ee29700 1 -- 172.21.15.42:0/4249989244 shutdown_connections 2019-12-05T05:11:19.054 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:19.048+0000 7ff38ee29700 1 --2- 172.21.15.42:0/4249989244 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff36803ad70 0x7ff36803d280 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:11:19.054 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:19.048+0000 7ff38ee29700 1 --2- 172.21.15.42:0/4249989244 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff38812b800 0x7ff38812bc30 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:11:19.054 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:19.048+0000 7ff38ee29700 1 --2- 172.21.15.42:0/4249989244 >> v2:172.21.15.40:3300/0 conn(0x7ff3881385a0 0x7ff38811e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:11:19.054 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:19.048+0000 7ff38ee29700 1 -- 172.21.15.42:0/4249989244 shutdown_connections 2019-12-05T05:11:19.054 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:19.048+0000 7ff38ee29700 1 -- 172.21.15.42:0/4249989244 wait complete. 2019-12-05T05:11:19.055 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:19.048+0000 7ff38ee29700 1 -- 172.21.15.42:0/4249989244 >> 172.21.15.42:0/4249989244 conn(0x7ff3880a6ed0 msgr2=0x7ff3880a97c0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:11:19.062 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 52, "feature_flags": { "enable_multiple": false, "ever_enabled_multiple": false }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2019-12-05T05:03:35.604738+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 52, "ever_allowed_features": 2, "explicitly_allowed_features": 2, "failed": [], "flags": 18, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5215": { "addr": "172.21.15.42:6835/3176146895", "addrs": { "addrvec": [ { "addr": "172.21.15.42:6834", "nonce": 3176146895, "type": "v2" }, { "addr": "172.21.15.42:6835", "nonce": 3176146895, "type": "v1" } ] }, "export_targets": [], "features": 4540138292836696063, "flags": 0, "gid": 5215, "incarnation": 52, "name": "a", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 52, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 2, "min_compat_client": "0 (unknown)", "modified": "2019-12-05T05:11:18.296472+0000", "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5215 } } } ], "standbys": [] } 2019-12-05T05:11:21.064 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:11:21.171 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:21.160+0000 7fbdd4bd7700 1 Processor -- start 2019-12-05T05:11:21.172 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:21.160+0000 7fbdd4bd7700 1 -- start start 2019-12-05T05:11:21.172 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:21.160+0000 7fbdd4bd7700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fbdd00a7730 0x7fbdd008df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:11:21.173 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:21.160+0000 7fbdd4bd7700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fbdd009fad0 0x7fbdd007eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:11:21.173 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:21.160+0000 7fbdd4bd7700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdd00a0700 con 0x7fbdd007e710 2019-12-05T05:11:21.173 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:21.160+0000 7fbdd4bd7700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbdd00a0af0 con 0x7fbdd009fad0 2019-12-05T05:11:21.174 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:21.160+0000 7fbdd4bd7700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbdd008e5a0 con 0x7fbdd00a7730 2019-12-05T05:11:21.175 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:21.160+0000 7fbdced9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fbdd00a7730 0x7fbdd008df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:11:21.175 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:21.160+0000 7fbdced9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fbdd00a7730 0x7fbdd008df70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:57682/0 (socket says 172.21.15.42:57682) 2019-12-05T05:11:21.175 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:21.160+0000 7fbdced9d700 1 -- 172.21.15.42:0/3153740168 learned_addr learned my addr 172.21.15.42:0/3153740168 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:11:21.175 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:21.160+0000 7fbdcf59e700 1 --2- 172.21.15.42:0/3153740168 >> v2:172.21.15.40:3300/0 conn(0x7fbdd009fad0 0x7fbdd007eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:11:21.175 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:21.164+0000 7fbdced9d700 1 -- 172.21.15.42:0/3153740168 >> v2:172.21.15.40:3300/0 conn(0x7fbdd009fad0 msgr2=0x7fbdd007eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:11:21.175 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:21.164+0000 7fbdced9d700 1 --2- 172.21.15.42:0/3153740168 >> v2:172.21.15.40:3300/0 conn(0x7fbdd009fad0 0x7fbdd007eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:11:21.175 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:21.164+0000 7fbdced9d700 1 -- 172.21.15.42:0/3153740168 >> v1:172.21.15.42:6789/0 conn(0x7fbdd007e710 legacy=0x7fbdd009f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:11:21.176 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:21.164+0000 7fbdced9d700 1 -- 172.21.15.42:0/3153740168 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdd007f090 con 0x7fbdd00a7730 2019-12-05T05:11:21.176 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:21.164+0000 7fbdced9d700 1 --2- 172.21.15.42:0/3153740168 >> v2:172.21.15.42:3300/0 conn(0x7fbdd00a7730 0x7fbdd008df70 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fbdc0002f30 tx=0x7fbdc000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:21.176 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:21.164+0000 7fbdced9d700 1 --2- 172.21.15.42:0/3153740168 >> v2:172.21.15.42:3300/0 conn(0x7fbdd00a7730 0x7fbdd008df70 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fbdc0002f30 tx=0x7fbdc000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:21.176 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:21.164+0000 7fbdced9d700 1 --2- 172.21.15.42:0/3153740168 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbdd00a7730 0x7fbdd008df70 secure :-1 s=READY pgs=438 cs=0 l=1 rx=0x7fbdc0002f30 tx=0x7fbdc000fd30).ready entity=mon.1 client_cookie=10fe281b77fdfd88 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:11:21.176 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:21.164+0000 7fbdced9d700 1 --2- 172.21.15.42:0/3153740168 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbdd00a7730 0x7fbdd008df70 secure :-1 s=THROTTLE_DONE pgs=438 cs=0 l=1 rx=0x7fbdc0002f30 tx=0x7fbdc000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:21.176 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:21.164+0000 7fbdced9d700 1 --2- 172.21.15.42:0/3153740168 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbdd00a7730 0x7fbdd008df70 secure :-1 s=THROTTLE_DONE pgs=438 cs=0 l=1 rx=0x7fbdc0002f30 tx=0x7fbdc000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:21.177 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:21.164+0000 7fbdcdd9b700 1 -- 172.21.15.42:0/3153740168 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbdc0008c90 con 0x7fbdd00a7730 2019-12-05T05:11:21.177 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:21.164+0000 7fbdced9d700 1 --2- 172.21.15.42:0/3153740168 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbdd00a7730 0x7fbdd008df70 secure :-1 s=THROTTLE_DONE pgs=438 cs=0 l=1 rx=0x7fbdc0002f30 tx=0x7fbdc000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:21.177 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:21.164+0000 7fbdcdd9b700 1 -- 172.21.15.42:0/3153740168 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdc00088b0 con 0x7fbdd00a7730 2019-12-05T05:11:21.177 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:21.164+0000 7fbdcdd9b700 1 -- 172.21.15.42:0/3153740168 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbdc0011ad0 con 0x7fbdd00a7730 2019-12-05T05:11:21.177 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:21.164+0000 7fbdd4bd7700 1 -- 172.21.15.42:0/3153740168 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbdd00a7730 msgr2=0x7fbdd008df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:11:21.177 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:21.164+0000 7fbdd4bd7700 1 --2- 172.21.15.42:0/3153740168 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbdd00a7730 0x7fbdd008df70 secure :-1 s=READY pgs=438 cs=0 l=1 rx=0x7fbdc0002f30 tx=0x7fbdc000fd30).stop 2019-12-05T05:11:21.177 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:21.164+0000 7fbdd4bd7700 1 -- 172.21.15.42:0/3153740168 shutdown_connections 2019-12-05T05:11:21.178 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:21.164+0000 7fbdd4bd7700 1 --2- 172.21.15.42:0/3153740168 >> v2:172.21.15.40:3300/0 conn(0x7fbdd009fad0 0x7fbdd007eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:11:21.178 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:21.164+0000 7fbdd4bd7700 1 --2- 172.21.15.42:0/3153740168 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbdd00a7730 0x7fbdd008df70 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:11:21.178 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:21.164+0000 7fbdd4bd7700 1 -- 172.21.15.42:0/3153740168 shutdown_connections 2019-12-05T05:11:21.178 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:21.164+0000 7fbdd4bd7700 1 -- 172.21.15.42:0/3153740168 wait complete. 2019-12-05T05:11:21.178 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:21.164+0000 7fbdd4bd7700 1 -- 172.21.15.42:0/3153740168 >> 172.21.15.42:0/3153740168 conn(0x7fbdd00aabd0 msgr2=0x7fbdd00a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:11:21.178 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:21.164+0000 7fbdd4bd7700 1 Processor -- start 2019-12-05T05:11:21.178 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:21.168+0000 7fbdd4bd7700 1 -- start start 2019-12-05T05:11:21.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:21.168+0000 7fbdd4bd7700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fbdd011c140 0x7fbdd01161b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:11:21.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:21.168+0000 7fbdd4bd7700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fbdd011a7f0 0x7fbdd007d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:11:21.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:21.168+0000 7fbdd4bd7700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbdd011ac20 0x7fbdd007d740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:11:21.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:21.168+0000 7fbdd4bd7700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbdd0133c70 con 0x7fbdd011c140 2019-12-05T05:11:21.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:21.168+0000 7fbdd4bd7700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbdd0133f10 con 0x7fbdd011a7f0 2019-12-05T05:11:21.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:21.168+0000 7fbdd4bd7700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdd01341b0 con 0x7fbdd011ac20 2019-12-05T05:11:21.184 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:21.168+0000 7fbdce59c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fbdd011a7f0 0x7fbdd007d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:11:21.184 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:21.168+0000 7fbdce59c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fbdd011a7f0 0x7fbdd007d1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:57688/0 (socket says 172.21.15.42:57688) 2019-12-05T05:11:21.184 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:21.168+0000 7fbdce59c700 1 -- 172.21.15.42:0/916929102 learned_addr learned my addr 172.21.15.42:0/916929102 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:11:21.184 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:21.168+0000 7fbdced9d700 1 --2- 172.21.15.42:0/916929102 >> v2:172.21.15.40:3300/0 conn(0x7fbdd011c140 0x7fbdd01161b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:11:21.185 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:21.168+0000 7fbdcf59e700 1 --2- 172.21.15.42:0/916929102 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbdd011ac20 0x7fbdd007d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:11:21.185 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:21.168+0000 7fbdce59c700 1 -- 172.21.15.42:0/916929102 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbdd011ac20 msgr2=0x7fbdd007d740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:11:21.185 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:21.168+0000 7fbdce59c700 1 --2- 172.21.15.42:0/916929102 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbdd011ac20 0x7fbdd007d740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:11:21.185 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:21.168+0000 7fbdce59c700 1 -- 172.21.15.42:0/916929102 >> v2:172.21.15.40:3300/0 conn(0x7fbdd011c140 msgr2=0x7fbdd01161b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:11:21.185 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:21.168+0000 7fbdce59c700 1 --2- 172.21.15.42:0/916929102 >> v2:172.21.15.40:3300/0 conn(0x7fbdd011c140 0x7fbdd01161b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:11:21.186 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:21.168+0000 7fbdce59c700 1 -- 172.21.15.42:0/916929102 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdd0116700 con 0x7fbdd011a7f0 2019-12-05T05:11:21.186 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:21.168+0000 7fbdce59c700 1 --2- 172.21.15.42:0/916929102 >> v2:172.21.15.42:3300/0 conn(0x7fbdd011a7f0 0x7fbdd007d1f0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fbdd01340d0 tx=0x7fbdc4010da0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:21.186 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:21.168+0000 7fbdce59c700 1 --2- 172.21.15.42:0/916929102 >> v2:172.21.15.42:3300/0 conn(0x7fbdd011a7f0 0x7fbdd007d1f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fbdd01340d0 tx=0x7fbdc4010da0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:21.186 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:21.168+0000 7fbdce59c700 1 --2- 172.21.15.42:0/916929102 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbdd011a7f0 0x7fbdd007d1f0 secure :-1 s=READY pgs=439 cs=0 l=1 rx=0x7fbdd01340d0 tx=0x7fbdc4010da0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:11:21.186 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:21.168+0000 7fbdce59c700 1 --2- 172.21.15.42:0/916929102 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbdd011a7f0 0x7fbdd007d1f0 secure :-1 s=THROTTLE_DONE pgs=439 cs=0 l=1 rx=0x7fbdd01340d0 tx=0x7fbdc4010da0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:21.186 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:21.168+0000 7fbdbf7fe700 1 -- 172.21.15.42:0/916929102 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbdc400ca40 con 0x7fbdd011a7f0 2019-12-05T05:11:21.186 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:21.168+0000 7fbdce59c700 1 --2- 172.21.15.42:0/916929102 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbdd011a7f0 0x7fbdd007d1f0 secure :-1 s=THROTTLE_DONE pgs=439 cs=0 l=1 rx=0x7fbdd01340d0 tx=0x7fbdc4010da0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:21.187 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:21.168+0000 7fbdce59c700 1 --2- 172.21.15.42:0/916929102 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbdd011a7f0 0x7fbdd007d1f0 secure :-1 s=THROTTLE_DONE pgs=439 cs=0 l=1 rx=0x7fbdd01340d0 tx=0x7fbdc4010da0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:21.187 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:21.168+0000 7fbdbf7fe700 1 -- 172.21.15.42:0/916929102 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdc40101c0 con 0x7fbdd011a7f0 2019-12-05T05:11:21.187 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:21.168+0000 7fbdd4bd7700 1 -- 172.21.15.42:0/916929102 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdd007b730 con 0x7fbdd011a7f0 2019-12-05T05:11:21.187 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:21.168+0000 7fbdbf7fe700 1 -- 172.21.15.42:0/916929102 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbdc40125d0 con 0x7fbdd011a7f0 2019-12-05T05:11:21.187 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:21.168+0000 7fbdd4bd7700 1 -- 172.21.15.42:0/916929102 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdd0134450 con 0x7fbdd011a7f0 2019-12-05T05:11:21.187 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:21.168+0000 7fbdce59c700 1 --2- 172.21.15.42:0/916929102 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbdd011a7f0 0x7fbdd007d1f0 secure :-1 s=THROTTLE_DONE pgs=439 cs=0 l=1 rx=0x7fbdd01340d0 tx=0x7fbdc4010da0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:21.188 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:21.172+0000 7fbdbf7fe700 1 -- 172.21.15.42:0/916929102 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fbdc40128b0 con 0x7fbdd011a7f0 2019-12-05T05:11:21.188 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:21.172+0000 7fbdce59c700 1 --2- 172.21.15.42:0/916929102 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbdd011a7f0 0x7fbdd007d1f0 secure :-1 s=THROTTLE_DONE pgs=439 cs=0 l=1 rx=0x7fbdd01340d0 tx=0x7fbdc4010da0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:21.188 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:21.172+0000 7fbdbf7fe700 1 --2- 172.21.15.42:0/916929102 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbdb403ae10 0x7fbdb403d320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:11:21.188 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:21.172+0000 7fbdbf7fe700 1 -- 172.21.15.42:0/916929102 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6461+0+0 (secure 0 0 0) 0x7fbdc404b550 con 0x7fbdd011a7f0 2019-12-05T05:11:21.188 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:21.172+0000 7fbdced9d700 1 --2- 172.21.15.42:0/916929102 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbdb403ae10 0x7fbdb403d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:11:21.188 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:21.172+0000 7fbdced9d700 1 --2- 172.21.15.42:0/916929102 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbdb403ae10 0x7fbdb403d320 crc :-1 s=READY pgs=200 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:11:21.188 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:21.176+0000 7fbdd4bd7700 1 -- 172.21.15.42:0/916929102 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbdd005d100 con 0x7fbdd011a7f0 2019-12-05T05:11:21.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:21.184+0000 7fbdce59c700 1 --2- 172.21.15.42:0/916929102 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbdd011a7f0 0x7fbdd007d1f0 secure :-1 s=THROTTLE_DONE pgs=439 cs=0 l=1 rx=0x7fbdd01340d0 tx=0x7fbdc4010da0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:21.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:21.184+0000 7fbdbf7fe700 1 -- 172.21.15.42:0/916929102 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fbdc40325b0 con 0x7fbdd011a7f0 2019-12-05T05:11:21.380 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:21.356+0000 7fbdd4bd7700 1 -- 172.21.15.42:0/916929102 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbdd005b8b0 con 0x7fbdd011a7f0 2019-12-05T05:11:21.380 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:21.356+0000 7fbdce59c700 1 --2- 172.21.15.42:0/916929102 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbdd011a7f0 0x7fbdd007d1f0 secure :-1 s=THROTTLE_DONE pgs=439 cs=0 l=1 rx=0x7fbdd01340d0 tx=0x7fbdc4010da0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:21.380 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:21.356+0000 7fbdbf7fe700 1 -- 172.21.15.42:0/916929102 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) v1 ==== 94+0+1775 (secure 0 0 0) 0x7fbdc4032810 con 0x7fbdd011a7f0 2019-12-05T05:11:21.381 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 53 2019-12-05T05:11:21.381 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:21.360+0000 7fbdd4bd7700 1 -- 172.21.15.42:0/916929102 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbdb403ae10 msgr2=0x7fbdb403d320 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:11:21.381 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:21.360+0000 7fbdd4bd7700 1 --2- 172.21.15.42:0/916929102 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbdb403ae10 0x7fbdb403d320 crc :-1 s=READY pgs=200 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:11:21.381 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:21.360+0000 7fbdd4bd7700 1 -- 172.21.15.42:0/916929102 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbdd011a7f0 msgr2=0x7fbdd007d1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:11:21.381 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:21.360+0000 7fbdd4bd7700 1 --2- 172.21.15.42:0/916929102 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbdd011a7f0 0x7fbdd007d1f0 secure :-1 s=READY pgs=439 cs=0 l=1 rx=0x7fbdd01340d0 tx=0x7fbdc4010da0).stop 2019-12-05T05:11:21.382 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:21.360+0000 7fbdd4bd7700 1 -- 172.21.15.42:0/916929102 shutdown_connections 2019-12-05T05:11:21.382 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:21.360+0000 7fbdd4bd7700 1 --2- 172.21.15.42:0/916929102 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbdb403ae10 0x7fbdb403d320 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:11:21.382 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:21.360+0000 7fbdd4bd7700 1 --2- 172.21.15.42:0/916929102 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbdd011ac20 0x7fbdd007d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:11:21.382 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:21.360+0000 7fbdd4bd7700 1 --2- 172.21.15.42:0/916929102 >> v2:172.21.15.40:3300/0 conn(0x7fbdd011c140 0x7fbdd01161b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:11:21.382 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:21.360+0000 7fbdd4bd7700 1 --2- 172.21.15.42:0/916929102 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbdd011a7f0 0x7fbdd007d1f0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:11:21.383 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:21.360+0000 7fbdd4bd7700 1 -- 172.21.15.42:0/916929102 shutdown_connections 2019-12-05T05:11:21.383 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:21.360+0000 7fbdd4bd7700 1 -- 172.21.15.42:0/916929102 wait complete. 2019-12-05T05:11:21.383 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:21.360+0000 7fbdd4bd7700 1 -- 172.21.15.42:0/916929102 >> 172.21.15.42:0/916929102 conn(0x7fbdd008fd90 msgr2=0x7fbdd009f610 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:11:21.383 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:11:21.384 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":53,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":53,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:11:20.818745+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5215},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5215":{"gid":5215,"name":"a","rank":0,"incarnation":52,"state":"up:reconnect","state_seq":18,"addr":"172.21.15.42:6835/3176146895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3176146895},{"type":"v1","addr":"172.21.15.42:6835","nonce":3176146895}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:11:21.384 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2019-12-05T05:11:22.881 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-git-ceph.sh... 2019-12-05T05:11:22.882 INFO:teuthology.orchestra.run.smithi042:workunit test fs/snaps/snaptest-git-ceph.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c5290805033483ba0cec165c06bde07dae5f19ec TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-ceph.sh 2019-12-05T05:11:22.897 INFO:tasks.workunit.client.0.smithi042.stderr:+ set -e 2019-12-05T05:11:22.897 INFO:tasks.workunit.client.0.smithi042.stderr:+ ceph fs set cephfs allow_new_snaps true --yes-i-really-mean-it 2019-12-05T05:11:23.006 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:11:22.996+0000 7fe15e746700 1 Processor -- start 2019-12-05T05:11:23.006 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:11:22.996+0000 7fe15e746700 1 -- start start 2019-12-05T05:11:23.007 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:11:22.996+0000 7fe15e746700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe1580a69e0 0x7fe1580a2830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:11:23.007 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:11:22.996+0000 7fe15e746700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe1580a8020 0x7fe1580652e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:11:23.007 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:11:22.996+0000 7fe15e746700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe158065830 con 0x7fe158144fd0 2019-12-05T05:11:23.007 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:11:22.996+0000 7fe15e746700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe158065b10 con 0x7fe1580a69e0 2019-12-05T05:11:23.007 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:11:22.996+0000 7fe15e746700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1580a2590 con 0x7fe1580a8020 2019-12-05T05:11:23.007 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:11:22.996+0000 7fe157fff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe1580a8020 0x7fe1580652e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:11:23.007 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:11:22.996+0000 7fe157fff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe1580a8020 0x7fe1580652e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:57700/0 (socket says 172.21.15.42:57700) 2019-12-05T05:11:23.008 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:11:22.996+0000 7fe157fff700 1 -- 172.21.15.42:0/863882519 learned_addr learned my addr 172.21.15.42:0/863882519 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:11:23.008 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:11:22.996+0000 7fe15cce3700 1 --2- 172.21.15.42:0/863882519 >> v2:172.21.15.40:3300/0 conn(0x7fe1580a69e0 0x7fe1580a2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:11:23.008 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:11:22.996+0000 7fe157fff700 1 -- 172.21.15.42:0/863882519 >> v2:172.21.15.40:3300/0 conn(0x7fe1580a69e0 msgr2=0x7fe1580a2830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:11:23.008 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:11:22.996+0000 7fe157fff700 1 --2- 172.21.15.42:0/863882519 >> v2:172.21.15.40:3300/0 conn(0x7fe1580a69e0 0x7fe1580a2830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:11:23.008 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:11:22.996+0000 7fe157fff700 1 -- 172.21.15.42:0/863882519 >> v1:172.21.15.40:6789/0 conn(0x7fe158144fd0 legacy=0x7fe15809c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:11:23.009 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:11:22.996+0000 7fe157fff700 1 -- 172.21.15.42:0/863882519 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1580a7b50 con 0x7fe1580a8020 2019-12-05T05:11:23.009 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:11:22.996+0000 7fe157fff700 1 --2- 172.21.15.42:0/863882519 >> v2:172.21.15.42:3300/0 conn(0x7fe1580a8020 0x7fe1580652e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fe140008790 tx=0x7fe14000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:23.009 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:11:22.996+0000 7fe157fff700 1 --2- 172.21.15.42:0/863882519 >> v2:172.21.15.42:3300/0 conn(0x7fe1580a8020 0x7fe1580652e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fe140008790 tx=0x7fe14000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:23.009 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:11:22.996+0000 7fe157fff700 1 --2- 172.21.15.42:0/863882519 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe1580a8020 0x7fe1580652e0 secure :-1 s=READY pgs=440 cs=0 l=1 rx=0x7fe140008790 tx=0x7fe14000fd30).ready entity=mon.1 client_cookie=302b4a7c13a33a8 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:11:23.009 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:11:22.996+0000 7fe157fff700 1 --2- 172.21.15.42:0/863882519 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe1580a8020 0x7fe1580652e0 secure :-1 s=THROTTLE_DONE pgs=440 cs=0 l=1 rx=0x7fe140008790 tx=0x7fe14000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:23.009 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:11:22.996+0000 7fe1577fe700 1 -- 172.21.15.42:0/863882519 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe140011370 con 0x7fe1580a8020 2019-12-05T05:11:23.010 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:11:22.996+0000 7fe157fff700 1 --2- 172.21.15.42:0/863882519 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe1580a8020 0x7fe1580652e0 secure :-1 s=THROTTLE_DONE pgs=440 cs=0 l=1 rx=0x7fe140008790 tx=0x7fe14000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:23.010 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:11:22.996+0000 7fe1577fe700 1 -- 172.21.15.42:0/863882519 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe140008bb0 con 0x7fe1580a8020 2019-12-05T05:11:23.010 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:11:22.996+0000 7fe157fff700 1 --2- 172.21.15.42:0/863882519 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe1580a8020 0x7fe1580652e0 secure :-1 s=THROTTLE_DONE pgs=440 cs=0 l=1 rx=0x7fe140008790 tx=0x7fe14000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:23.010 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:11:22.996+0000 7fe1577fe700 1 -- 172.21.15.42:0/863882519 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe140011950 con 0x7fe1580a8020 2019-12-05T05:11:23.010 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:11:22.996+0000 7fe15e746700 1 -- 172.21.15.42:0/863882519 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe1580a8020 msgr2=0x7fe1580652e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:11:23.011 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:11:22.996+0000 7fe15e746700 1 --2- 172.21.15.42:0/863882519 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe1580a8020 0x7fe1580652e0 secure :-1 s=READY pgs=440 cs=0 l=1 rx=0x7fe140008790 tx=0x7fe14000fd30).stop 2019-12-05T05:11:23.011 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:11:23.000+0000 7fe15e746700 1 -- 172.21.15.42:0/863882519 shutdown_connections 2019-12-05T05:11:23.011 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:11:23.000+0000 7fe15e746700 1 --2- 172.21.15.42:0/863882519 >> v2:172.21.15.40:3300/0 conn(0x7fe1580a69e0 0x7fe1580a2830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:11:23.011 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:11:23.000+0000 7fe15e746700 1 --2- 172.21.15.42:0/863882519 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe1580a8020 0x7fe1580652e0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:11:23.011 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:11:23.000+0000 7fe15e746700 1 -- 172.21.15.42:0/863882519 shutdown_connections 2019-12-05T05:11:23.011 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:11:23.000+0000 7fe15e746700 1 -- 172.21.15.42:0/863882519 wait complete. 2019-12-05T05:11:23.011 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:11:23.000+0000 7fe15e746700 1 -- 172.21.15.42:0/863882519 >> 172.21.15.42:0/863882519 conn(0x7fe158059560 msgr2=0x7fe158059990 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:11:23.015 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:11:23.000+0000 7fe15e746700 1 Processor -- start 2019-12-05T05:11:23.015 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:11:23.000+0000 7fe15e746700 1 -- start start 2019-12-05T05:11:23.016 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:11:23.004+0000 7fe15e746700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe15807da80 0x7fe15813b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:11:23.016 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:11:23.004+0000 7fe15e746700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe15813b910 0x7fe15813dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:11:23.016 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:11:23.004+0000 7fe15e746700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe158133b80 con 0x7fe1580aa270 2019-12-05T05:11:23.016 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:11:23.004+0000 7fe15e746700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1580a5b60 con 0x7fe15813b910 2019-12-05T05:11:23.016 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:11:23.004+0000 7fe15e746700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1580a50d0 con 0x7fe15807da80 2019-12-05T05:11:23.016 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:11:23.004+0000 7fe15cce3700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fe1580aa270 0x7fe158135f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:57744/0 (socket says 172.21.15.42:57744) 2019-12-05T05:11:23.017 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:11:23.004+0000 7fe15cce3700 1 -- 172.21.15.42:0/4084801435 learned_addr learned my addr 172.21.15.42:0/4084801435 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:11:23.017 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:11:23.004+0000 7fe15d4e4700 1 --2- 172.21.15.42:0/4084801435 >> v2:172.21.15.42:3300/0 conn(0x7fe15813b910 0x7fe15813dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:11:23.017 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:11:23.004+0000 7fe157fff700 1 --2- 172.21.15.42:0/4084801435 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe15807da80 0x7fe15813b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:11:23.017 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:11:23.004+0000 7fe1557fa700 1 -- 172.21.15.42:0/4084801435 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2673604469 0 0) 0x7fe158133b80 con 0x7fe1580aa270 2019-12-05T05:11:23.017 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:11:23.004+0000 7fe1557fa700 1 -- 172.21.15.42:0/4084801435 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe13c0030b0 con 0x7fe1580aa270 2019-12-05T05:11:23.018 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:11:23.004+0000 7fe15d4e4700 1 -- 172.21.15.42:0/4084801435 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe15807da80 msgr2=0x7fe15813b3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:11:23.018 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:11:23.004+0000 7fe15d4e4700 1 --2- 172.21.15.42:0/4084801435 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe15807da80 0x7fe15813b3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:11:23.018 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:11:23.004+0000 7fe15d4e4700 1 -- 172.21.15.42:0/4084801435 >> v1:172.21.15.42:6789/0 conn(0x7fe1580aa270 legacy=0x7fe158135f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:11:23.018 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:11:23.004+0000 7fe15d4e4700 1 -- 172.21.15.42:0/4084801435 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1580a58b0 con 0x7fe15813b910 2019-12-05T05:11:23.018 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:11:23.004+0000 7fe157fff700 1 --2- 172.21.15.42:0/4084801435 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe15807da80 0x7fe15813b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:11:23.019 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:11:23.004+0000 7fe15d4e4700 1 --2- 172.21.15.42:0/4084801435 >> v2:172.21.15.42:3300/0 conn(0x7fe15813b910 0x7fe15813dd50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fe1580a5290 tx=0x7fe1500107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:23.019 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:11:23.004+0000 7fe15d4e4700 1 --2- 172.21.15.42:0/4084801435 >> v2:172.21.15.42:3300/0 conn(0x7fe15813b910 0x7fe15813dd50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fe1580a5290 tx=0x7fe1500107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:23.019 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:11:23.004+0000 7fe15d4e4700 1 --2- 172.21.15.42:0/4084801435 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe15813b910 0x7fe15813dd50 secure :-1 s=READY pgs=442 cs=0 l=1 rx=0x7fe1580a5290 tx=0x7fe1500107c0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:11:23.019 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:11:23.004+0000 7fe15d4e4700 1 --2- 172.21.15.42:0/4084801435 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe15813b910 0x7fe15813dd50 secure :-1 s=THROTTLE_DONE pgs=442 cs=0 l=1 rx=0x7fe1580a5290 tx=0x7fe1500107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:23.020 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:11:23.004+0000 7fe1557fa700 1 -- 172.21.15.42:0/4084801435 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe150006a40 con 0x7fe15813b910 2019-12-05T05:11:23.020 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:11:23.004+0000 7fe15d4e4700 1 --2- 172.21.15.42:0/4084801435 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe15813b910 0x7fe15813dd50 secure :-1 s=THROTTLE_DONE pgs=442 cs=0 l=1 rx=0x7fe1580a5290 tx=0x7fe1500107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:23.020 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:11:23.004+0000 7fe15d4e4700 1 --2- 172.21.15.42:0/4084801435 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe15813b910 0x7fe15813dd50 secure :-1 s=THROTTLE_DONE pgs=442 cs=0 l=1 rx=0x7fe1580a5290 tx=0x7fe1500107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:23.020 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:11:23.004+0000 7fe1557fa700 1 -- 172.21.15.42:0/4084801435 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe150006660 con 0x7fe15813b910 2019-12-05T05:11:23.020 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:11:23.004+0000 7fe15e746700 1 -- 172.21.15.42:0/4084801435 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1580a4d30 con 0x7fe15813b910 2019-12-05T05:11:23.021 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:11:23.004+0000 7fe1557fa700 1 -- 172.21.15.42:0/4084801435 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe150010fc0 con 0x7fe15813b910 2019-12-05T05:11:23.021 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:11:23.004+0000 7fe15e746700 1 -- 172.21.15.42:0/4084801435 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe158136780 con 0x7fe15813b910 2019-12-05T05:11:23.021 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:11:23.008+0000 7fe15d4e4700 1 --2- 172.21.15.42:0/4084801435 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe15813b910 0x7fe15813dd50 secure :-1 s=THROTTLE_DONE pgs=442 cs=0 l=1 rx=0x7fe1580a5290 tx=0x7fe1500107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:23.021 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:11:23.008+0000 7fe1557fa700 1 -- 172.21.15.42:0/4084801435 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fe1500112d0 con 0x7fe15813b910 2019-12-05T05:11:23.021 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:11:23.008+0000 7fe15d4e4700 1 --2- 172.21.15.42:0/4084801435 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe15813b910 0x7fe15813dd50 secure :-1 s=THROTTLE_DONE pgs=442 cs=0 l=1 rx=0x7fe1580a5290 tx=0x7fe1500107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:23.021 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:11:23.008+0000 7fe1557fa700 1 --2- 172.21.15.42:0/4084801435 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe13c03b220 0x7fe13c03d730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:11:23.022 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:11:23.008+0000 7fe1557fa700 1 -- 172.21.15.42:0/4084801435 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6461+0+0 (secure 0 0 0) 0x7fe158065830 con 0x7fe15813b910 2019-12-05T05:11:23.022 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:11:23.008+0000 7fe15cce3700 1 --2- 172.21.15.42:0/4084801435 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe13c03b220 0x7fe13c03d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:11:23.022 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:11:23.008+0000 7fe15cce3700 1 --2- 172.21.15.42:0/4084801435 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe13c03b220 0x7fe13c03d730 crc :-1 s=READY pgs=201 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:11:23.023 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:11:23.012+0000 7fe15e746700 1 -- 172.21.15.42:0/4084801435 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe15805d2b0 con 0x7fe15813b910 2019-12-05T05:11:23.025 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:11:23.020+0000 7fe15d4e4700 1 --2- 172.21.15.42:0/4084801435 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe15813b910 0x7fe15813dd50 secure :-1 s=THROTTLE_DONE pgs=442 cs=0 l=1 rx=0x7fe1580a5290 tx=0x7fe1500107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:23.025 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:11:23.020+0000 7fe1557fa700 1 -- 172.21.15.42:0/4084801435 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fe1580a58b0 con 0x7fe15813b910 2019-12-05T05:11:23.196 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:11:23.188+0000 7fe15e746700 1 -- 172.21.15.42:0/4084801435 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"var": "allow_new_snaps", "prefix": "fs set", "fs_name": "cephfs", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fe1580a6480 con 0x7fe15813b910 2019-12-05T05:11:23.399 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:11:23.509 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:23.496+0000 7f6af3add700 1 Processor -- start 2019-12-05T05:11:23.509 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:23.496+0000 7f6af3add700 1 -- start start 2019-12-05T05:11:23.509 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:23.500+0000 7f6af3add700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6aec0a7740 0x7f6aec08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:11:23.509 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:23.500+0000 7f6af3add700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6aec09fae0 0x7f6aec07eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:11:23.510 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:23.500+0000 7f6af3add700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6aec0a0710 con 0x7f6aec07e720 2019-12-05T05:11:23.510 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:23.500+0000 7f6af3add700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6aec0a0b00 con 0x7f6aec09fae0 2019-12-05T05:11:23.510 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:23.500+0000 7f6af3add700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6aec08e5b0 con 0x7f6aec0a7740 2019-12-05T05:11:23.510 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:23.500+0000 7f6af287b700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6aec09fae0 0x7f6aec07eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:11:23.511 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:23.500+0000 7f6af287b700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6aec09fae0 0x7f6aec07eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:57720/0 (socket says 172.21.15.42:57720) 2019-12-05T05:11:23.511 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:23.500+0000 7f6af207a700 1 --2- 172.21.15.42:0/2919935481 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6aec0a7740 0x7f6aec08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:11:23.511 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:23.500+0000 7f6af287b700 1 -- 172.21.15.42:0/2919935481 learned_addr learned my addr 172.21.15.42:0/2919935481 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:11:23.511 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:23.500+0000 7f6af287b700 1 -- 172.21.15.42:0/2919935481 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6aec0a7740 msgr2=0x7f6aec08df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:11:23.512 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:23.500+0000 7f6af1078700 1 -- 172.21.15.42:0/2919935481 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1978105846 0 0) 0x7f6aec0a0710 con 0x7f6aec07e720 2019-12-05T05:11:23.512 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:23.500+0000 7f6af287b700 1 --2- 172.21.15.42:0/2919935481 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6aec0a7740 0x7f6aec08df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:11:23.512 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:23.500+0000 7f6af287b700 1 -- 172.21.15.42:0/2919935481 >> v1:172.21.15.40:6789/0 conn(0x7f6aec07e720 legacy=0x7f6aec09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:11:23.512 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:23.500+0000 7f6af287b700 1 -- 172.21.15.42:0/2919935481 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6aec07f0a0 con 0x7f6aec09fae0 2019-12-05T05:11:23.512 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:23.500+0000 7f6af287b700 1 --2- 172.21.15.42:0/2919935481 >> v2:172.21.15.42:3300/0 conn(0x7f6aec09fae0 0x7f6aec07eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f6ae4008b10 tx=0x7f6ae40100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:23.513 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:23.500+0000 7f6af287b700 1 --2- 172.21.15.42:0/2919935481 >> v2:172.21.15.42:3300/0 conn(0x7f6aec09fae0 0x7f6aec07eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f6ae4008b10 tx=0x7f6ae40100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:23.513 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:23.500+0000 7f6af287b700 1 --2- 172.21.15.42:0/2919935481 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6aec09fae0 0x7f6aec07eb50 secure :-1 s=READY pgs=445 cs=0 l=1 rx=0x7f6ae4008b10 tx=0x7f6ae40100b0).ready entity=mon.1 client_cookie=47cd4188c9df1b34 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:11:23.513 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:23.500+0000 7f6af287b700 1 --2- 172.21.15.42:0/2919935481 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6aec09fae0 0x7f6aec07eb50 secure :-1 s=THROTTLE_DONE pgs=445 cs=0 l=1 rx=0x7f6ae4008b10 tx=0x7f6ae40100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:23.513 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:23.500+0000 7f6af1078700 1 -- 172.21.15.42:0/2919935481 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6ae4010da0 con 0x7f6aec09fae0 2019-12-05T05:11:23.514 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:23.500+0000 7f6af287b700 1 --2- 172.21.15.42:0/2919935481 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6aec09fae0 0x7f6aec07eb50 secure :-1 s=THROTTLE_DONE pgs=445 cs=0 l=1 rx=0x7f6ae4008b10 tx=0x7f6ae40100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:23.514 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:23.500+0000 7f6af1078700 1 -- 172.21.15.42:0/2919935481 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ae40109c0 con 0x7f6aec09fae0 2019-12-05T05:11:23.514 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:23.500+0000 7f6af287b700 1 --2- 172.21.15.42:0/2919935481 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6aec09fae0 0x7f6aec07eb50 secure :-1 s=THROTTLE_DONE pgs=445 cs=0 l=1 rx=0x7f6ae4008b10 tx=0x7f6ae40100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:23.514 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:23.500+0000 7f6af1078700 1 -- 172.21.15.42:0/2919935481 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6ae40114a0 con 0x7f6aec09fae0 2019-12-05T05:11:23.515 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:23.500+0000 7f6af3add700 1 -- 172.21.15.42:0/2919935481 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6aec09fae0 msgr2=0x7f6aec07eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:11:23.515 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:23.500+0000 7f6af3add700 1 --2- 172.21.15.42:0/2919935481 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6aec09fae0 0x7f6aec07eb50 secure :-1 s=READY pgs=445 cs=0 l=1 rx=0x7f6ae4008b10 tx=0x7f6ae40100b0).stop 2019-12-05T05:11:23.515 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:23.500+0000 7f6af3add700 1 -- 172.21.15.42:0/2919935481 shutdown_connections 2019-12-05T05:11:23.515 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:23.500+0000 7f6af3add700 1 --2- 172.21.15.42:0/2919935481 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6aec09fae0 0x7f6aec07eb50 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:11:23.515 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:23.500+0000 7f6af3add700 1 --2- 172.21.15.42:0/2919935481 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6aec0a7740 0x7f6aec08df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:11:23.516 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:23.500+0000 7f6af3add700 1 -- 172.21.15.42:0/2919935481 shutdown_connections 2019-12-05T05:11:23.516 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:23.500+0000 7f6af3add700 1 -- 172.21.15.42:0/2919935481 wait complete. 2019-12-05T05:11:23.516 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:23.500+0000 7f6af3add700 1 -- 172.21.15.42:0/2919935481 >> 172.21.15.42:0/2919935481 conn(0x7f6aec0aabe0 msgr2=0x7f6aec0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:11:23.522 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:23.504+0000 7f6af3add700 1 Processor -- start 2019-12-05T05:11:23.522 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:23.504+0000 7f6af3add700 1 -- start start 2019-12-05T05:11:23.522 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:23.504+0000 7f6af3add700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6aec0a6dc0 0x7f6aec0a5d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:11:23.523 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:23.504+0000 7f6af3add700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6aec07ba60 0x7f6aec07be90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:11:23.523 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:23.504+0000 7f6af3add700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6aec0a6290 con 0x7f6aec119530 2019-12-05T05:11:23.523 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:23.504+0000 7f6af3add700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6aec0a66c0 con 0x7f6aec0a6dc0 2019-12-05T05:11:23.523 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:23.504+0000 7f6af3add700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6aec119290 con 0x7f6aec07ba60 2019-12-05T05:11:23.523 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:23.508+0000 7f6af207a700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6aec0a6dc0 0x7f6aec0a5d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:11:23.524 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:23.508+0000 7f6af287b700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f6aec119530 0x7f6aec119960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:57762/0 (socket says 172.21.15.42:57762) 2019-12-05T05:11:23.524 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:23.508+0000 7f6af287b700 1 -- 172.21.15.42:0/1332464360 learned_addr learned my addr 172.21.15.42:0/1332464360 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:11:23.524 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:23.508+0000 7f6af1879700 1 --2- 172.21.15.42:0/1332464360 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6aec07ba60 0x7f6aec07be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:11:23.524 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:23.508+0000 7f6ae2ffd700 1 -- 172.21.15.42:0/1332464360 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2818713116 0 0) 0x7f6aec0a6290 con 0x7f6aec119530 2019-12-05T05:11:23.524 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:23.508+0000 7f6af207a700 1 -- 172.21.15.42:0/1332464360 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6aec07ba60 msgr2=0x7f6aec07be90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:11:23.525 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:23.508+0000 7f6af207a700 1 --2- 172.21.15.42:0/1332464360 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6aec07ba60 0x7f6aec07be90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:11:23.525 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:23.508+0000 7f6af207a700 1 -- 172.21.15.42:0/1332464360 >> v1:172.21.15.42:6789/0 conn(0x7f6aec119530 legacy=0x7f6aec119960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:11:23.525 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:23.508+0000 7f6af207a700 1 -- 172.21.15.42:0/1332464360 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6aec0a4b20 con 0x7f6aec0a6dc0 2019-12-05T05:11:23.525 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:23.508+0000 7f6af207a700 1 --2- 172.21.15.42:0/1332464360 >> v2:172.21.15.42:3300/0 conn(0x7f6aec0a6dc0 0x7f6aec0a5d40 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f6ae8002f30 tx=0x7f6ae80109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:23.525 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:23.508+0000 7f6af207a700 1 --2- 172.21.15.42:0/1332464360 >> v2:172.21.15.42:3300/0 conn(0x7f6aec0a6dc0 0x7f6aec0a5d40 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f6ae8002f30 tx=0x7f6ae80109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:23.526 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:23.508+0000 7f6af207a700 1 --2- 172.21.15.42:0/1332464360 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6aec0a6dc0 0x7f6aec0a5d40 secure :-1 s=READY pgs=447 cs=0 l=1 rx=0x7f6ae8002f30 tx=0x7f6ae80109e0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:11:23.526 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:23.508+0000 7f6af207a700 1 --2- 172.21.15.42:0/1332464360 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6aec0a6dc0 0x7f6aec0a5d40 secure :-1 s=THROTTLE_DONE pgs=447 cs=0 l=1 rx=0x7f6ae8002f30 tx=0x7f6ae80109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:23.526 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:23.508+0000 7f6ae2ffd700 1 -- 172.21.15.42:0/1332464360 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6ae8005920 con 0x7f6aec0a6dc0 2019-12-05T05:11:23.526 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:23.508+0000 7f6af207a700 1 --2- 172.21.15.42:0/1332464360 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6aec0a6dc0 0x7f6aec0a5d40 secure :-1 s=THROTTLE_DONE pgs=447 cs=0 l=1 rx=0x7f6ae8002f30 tx=0x7f6ae80109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:23.526 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:23.508+0000 7f6ae2ffd700 1 -- 172.21.15.42:0/1332464360 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ae8005540 con 0x7f6aec0a6dc0 2019-12-05T05:11:23.526 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:23.508+0000 7f6af207a700 1 --2- 172.21.15.42:0/1332464360 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6aec0a6dc0 0x7f6aec0a5d40 secure :-1 s=THROTTLE_DONE pgs=447 cs=0 l=1 rx=0x7f6ae8002f30 tx=0x7f6ae80109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:23.527 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:23.508+0000 7f6ae2ffd700 1 -- 172.21.15.42:0/1332464360 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6ae8012490 con 0x7f6aec0a6dc0 2019-12-05T05:11:23.527 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:23.508+0000 7f6af3add700 1 -- 172.21.15.42:0/1332464360 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6aec131690 con 0x7f6aec0a6dc0 2019-12-05T05:11:23.527 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:23.508+0000 7f6af3add700 1 -- 172.21.15.42:0/1332464360 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6aec131aa0 con 0x7f6aec0a6dc0 2019-12-05T05:11:23.527 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:23.508+0000 7f6af207a700 1 --2- 172.21.15.42:0/1332464360 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6aec0a6dc0 0x7f6aec0a5d40 secure :-1 s=THROTTLE_DONE pgs=447 cs=0 l=1 rx=0x7f6ae8002f30 tx=0x7f6ae80109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:23.527 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:23.508+0000 7f6ae2ffd700 1 -- 172.21.15.42:0/1332464360 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f6ae8007180 con 0x7f6aec0a6dc0 2019-12-05T05:11:23.528 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:23.508+0000 7f6af207a700 1 --2- 172.21.15.42:0/1332464360 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6aec0a6dc0 0x7f6aec0a5d40 secure :-1 s=THROTTLE_DONE pgs=447 cs=0 l=1 rx=0x7f6ae8002f30 tx=0x7f6ae80109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:23.528 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:23.512+0000 7f6ae2ffd700 1 --2- 172.21.15.42:0/1332464360 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6ad403ac00 0x7f6ad403d110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:11:23.528 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:23.512+0000 7f6ae2ffd700 1 -- 172.21.15.42:0/1332464360 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6461+0+0 (secure 0 0 0) 0x7f6ae804adb0 con 0x7f6aec0a6dc0 2019-12-05T05:11:23.528 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:23.512+0000 7f6af1879700 1 --2- 172.21.15.42:0/1332464360 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6ad403ac00 0x7f6ad403d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:11:23.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:23.512+0000 7f6af1879700 1 --2- 172.21.15.42:0/1332464360 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6ad403ac00 0x7f6ad403d110 crc :-1 s=READY pgs=202 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:11:23.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:23.512+0000 7f6af3add700 1 -- 172.21.15.42:0/1332464360 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6aec0a64c0 con 0x7f6aec0a6dc0 2019-12-05T05:11:23.530 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:23.520+0000 7f6af207a700 1 --2- 172.21.15.42:0/1332464360 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6aec0a6dc0 0x7f6aec0a5d40 secure :-1 s=THROTTLE_DONE pgs=447 cs=0 l=1 rx=0x7f6ae8002f30 tx=0x7f6ae80109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:23.531 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:23.520+0000 7f6ae2ffd700 1 -- 172.21.15.42:0/1332464360 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f6ae8031e10 con 0x7f6aec0a6dc0 2019-12-05T05:11:23.708 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:23.692+0000 7f6af3add700 1 -- 172.21.15.42:0/1332464360 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6aec05d110 con 0x7f6aec0a6dc0 2019-12-05T05:11:23.709 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:23.692+0000 7f6af207a700 1 --2- 172.21.15.42:0/1332464360 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6aec0a6dc0 0x7f6aec0a5d40 secure :-1 s=THROTTLE_DONE pgs=447 cs=0 l=1 rx=0x7f6ae8002f30 tx=0x7f6ae80109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:23.709 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:23.692+0000 7f6ae2ffd700 1 -- 172.21.15.42:0/1332464360 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 55 v55) v1 ==== 94+0+1772 (secure 0 0 0) 0x7f6ae8032070 con 0x7f6aec0a6dc0 2019-12-05T05:11:23.709 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 55 2019-12-05T05:11:23.709 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:23.696+0000 7f6af3add700 1 -- 172.21.15.42:0/1332464360 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6ad403ac00 msgr2=0x7f6ad403d110 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:11:23.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:23.696+0000 7f6af3add700 1 --2- 172.21.15.42:0/1332464360 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6ad403ac00 0x7f6ad403d110 crc :-1 s=READY pgs=202 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:11:23.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:23.696+0000 7f6af3add700 1 -- 172.21.15.42:0/1332464360 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6aec0a6dc0 msgr2=0x7f6aec0a5d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:11:23.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:23.696+0000 7f6af3add700 1 --2- 172.21.15.42:0/1332464360 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6aec0a6dc0 0x7f6aec0a5d40 secure :-1 s=READY pgs=447 cs=0 l=1 rx=0x7f6ae8002f30 tx=0x7f6ae80109e0).stop 2019-12-05T05:11:23.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:23.696+0000 7f6af3add700 1 -- 172.21.15.42:0/1332464360 shutdown_connections 2019-12-05T05:11:23.711 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:23.696+0000 7f6af3add700 1 --2- 172.21.15.42:0/1332464360 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6ad403ac00 0x7f6ad403d110 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:11:23.711 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:23.696+0000 7f6af3add700 1 --2- 172.21.15.42:0/1332464360 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6aec0a6dc0 0x7f6aec0a5d40 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:11:23.711 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:23.696+0000 7f6af3add700 1 --2- 172.21.15.42:0/1332464360 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6aec07ba60 0x7f6aec07be90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:11:23.711 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:23.696+0000 7f6af3add700 1 -- 172.21.15.42:0/1332464360 shutdown_connections 2019-12-05T05:11:23.712 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:23.696+0000 7f6af3add700 1 -- 172.21.15.42:0/1332464360 wait complete. 2019-12-05T05:11:23.712 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:23.696+0000 7f6af3add700 1 -- 172.21.15.42:0/1332464360 >> 172.21.15.42:0/1332464360 conn(0x7f6aec0a82a0 msgr2=0x7f6aec07e4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:11:23.712 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:11:23.713 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":55,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:11:22.830707+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5215},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5215":{"gid":5215,"name":"a","rank":0,"incarnation":52,"state":"up:active","state_seq":20,"addr":"172.21.15.42:6835/3176146895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3176146895},{"type":"v1","addr":"172.21.15.42:6835","nonce":3176146895}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:11:23.714 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a has gained rank=0, replacing gid=4595 2019-12-05T05:11:23.715 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 4 secs before reviving mds.b 2019-12-05T05:11:23.859 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:11:23.852+0000 7fe15d4e4700 1 --2- 172.21.15.42:0/4084801435 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe15813b910 0x7fe15813dd50 secure :-1 s=THROTTLE_DONE pgs=442 cs=0 l=1 rx=0x7fe1580a5290 tx=0x7fe1500107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:23.860 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:11:23.852+0000 7fe1557fa700 1 -- 172.21.15.42:0/4084801435 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"var": "allow_new_snaps", "prefix": "fs set", "fs_name": "cephfs", "val": "true", "yes_i_really_mean_it": true}]=0 enabled new snapshots v56) v1 ==== 167+0+0 (secure 0 0 0) 0x7fe150032090 con 0x7fe15813b910 2019-12-05T05:11:23.984 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:11:23.972+0000 7fe15e746700 1 -- 172.21.15.42:0/4084801435 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"var": "allow_new_snaps", "prefix": "fs set", "fs_name": "cephfs", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fe15805ba60 con 0x7fe15813b910 2019-12-05T05:11:24.862 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:11:24.856+0000 7fe15d4e4700 1 --2- 172.21.15.42:0/4084801435 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe15813b910 0x7fe15813dd50 secure :-1 s=THROTTLE_DONE pgs=442 cs=0 l=1 rx=0x7fe1580a5290 tx=0x7fe1500107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:24.863 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:11:24.856+0000 7fe1557fa700 1 -- 172.21.15.42:0/4084801435 <== mon.1 v2:172.21.15.42:3300/0 8 ==== mon_command_ack([{"var": "allow_new_snaps", "prefix": "fs set", "fs_name": "cephfs", "val": "true", "yes_i_really_mean_it": true}]=0 enabled new snapshots v57) v1 ==== 167+0+0 (secure 0 0 0) 0x7fe1500322f0 con 0x7fe15813b910 2019-12-05T05:11:24.894 INFO:tasks.workunit.client.0.smithi042.stderr:enabled new snapshots 2019-12-05T05:11:24.898 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:11:24.892+0000 7fe15e746700 1 -- 172.21.15.42:0/4084801435 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe13c03b220 msgr2=0x7fe13c03d730 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:11:24.899 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:11:24.892+0000 7fe15e746700 1 --2- 172.21.15.42:0/4084801435 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe13c03b220 0x7fe13c03d730 crc :-1 s=READY pgs=201 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:11:24.899 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:11:24.892+0000 7fe15e746700 1 -- 172.21.15.42:0/4084801435 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe15813b910 msgr2=0x7fe15813dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:11:24.899 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:11:24.892+0000 7fe15e746700 1 --2- 172.21.15.42:0/4084801435 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe15813b910 0x7fe15813dd50 secure :-1 s=READY pgs=442 cs=0 l=1 rx=0x7fe1580a5290 tx=0x7fe1500107c0).stop 2019-12-05T05:11:24.899 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:11:24.892+0000 7fe15e746700 1 -- 172.21.15.42:0/4084801435 shutdown_connections 2019-12-05T05:11:24.899 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:11:24.892+0000 7fe15e746700 1 --2- 172.21.15.42:0/4084801435 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe13c03b220 0x7fe13c03d730 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:11:24.899 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:11:24.892+0000 7fe15e746700 1 --2- 172.21.15.42:0/4084801435 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe15813b910 0x7fe15813dd50 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:11:24.900 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:11:24.892+0000 7fe15e746700 1 --2- 172.21.15.42:0/4084801435 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe15807da80 0x7fe15813b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:11:24.900 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:11:24.892+0000 7fe15e746700 1 -- 172.21.15.42:0/4084801435 shutdown_connections 2019-12-05T05:11:24.900 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:11:24.892+0000 7fe15e746700 1 -- 172.21.15.42:0/4084801435 wait complete. 2019-12-05T05:11:24.900 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T05:11:24.892+0000 7fe15e746700 1 -- 172.21.15.42:0/4084801435 >> 172.21.15.42:0/4084801435 conn(0x7fe158145280 msgr2=0x7fe158123400 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:11:24.908 INFO:tasks.workunit.client.0.smithi042.stderr:+ git clone git://git.ceph.com/ceph.git 2019-12-05T05:11:24.917 INFO:tasks.workunit.client.0.smithi042.stderr:Cloning into 'ceph'... 2019-12-05T05:11:27.716 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.b 2019-12-05T05:11:27.716 INFO:tasks.ceph.mds.b:Restarting daemon 2019-12-05T05:11:27.717 INFO:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2019-12-05T05:11:27.721 INFO:tasks.ceph.mds.b:Started 2019-12-05T05:11:27.724 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:11:27.781 INFO:tasks.ceph.mds.b.smithi040.stdout:starting mds.b at 2019-12-05T05:11:27.833 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:27.816+0000 7ffbaf64e700 1 Processor -- start 2019-12-05T05:11:27.833 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:27.820+0000 7ffbaf64e700 1 -- start start 2019-12-05T05:11:27.833 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:27.820+0000 7ffbaf64e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ffba808df80 0x7ffba809f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:11:27.833 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:27.820+0000 7ffbaf64e700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffba809fc60 con 0x7ffba809f830 2019-12-05T05:11:27.834 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:27.820+0000 7ffbaf64e700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffba80a0710 con 0x7ffba80a7740 2019-12-05T05:11:27.834 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:27.820+0000 7ffbaf64e700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffba80a0b00 con 0x7ffba808df80 2019-12-05T05:11:27.834 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:27.820+0000 7ffbad3ea700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ffba808df80 0x7ffba809f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:11:27.834 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:27.820+0000 7ffbadbeb700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7ffba80a7740 0x7ffba807e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:57778/0 (socket says 172.21.15.42:57778) 2019-12-05T05:11:27.834 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:27.820+0000 7ffbadbeb700 1 -- 172.21.15.42:0/612230497 learned_addr learned my addr 172.21.15.42:0/612230497 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:11:27.834 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:27.820+0000 7ffbacbe9700 1 -- 172.21.15.42:0/612230497 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 234538941 0 0) 0x7ffba80a0710 con 0x7ffba80a7740 2019-12-05T05:11:27.834 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:27.820+0000 7ffbacbe9700 1 -- 172.21.15.42:0/612230497 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb940030b0 con 0x7ffba80a7740 2019-12-05T05:11:27.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:27.820+0000 7ffbad3ea700 1 -- 172.21.15.42:0/612230497 >> v1:172.21.15.42:6789/0 conn(0x7ffba80a7740 legacy=0x7ffba807e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:11:27.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:27.820+0000 7ffbacbe9700 1 -- 172.21.15.42:0/612230497 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3042735411 0 0) 0x7ffba809fc60 con 0x7ffba809f830 2019-12-05T05:11:27.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:27.820+0000 7ffbad3ea700 1 -- 172.21.15.42:0/612230497 >> v1:172.21.15.40:6789/0 conn(0x7ffba809f830 legacy=0x7ffba814efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:11:27.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:27.820+0000 7ffbad3ea700 1 -- 172.21.15.42:0/612230497 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffba807ef20 con 0x7ffba808df80 2019-12-05T05:11:27.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:27.820+0000 7ffbad3ea700 1 --2- 172.21.15.42:0/612230497 >> v2:172.21.15.42:3300/0 conn(0x7ffba808df80 0x7ffba809f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7ffb98008790 tx=0x7ffb9800fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:27.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:27.820+0000 7ffbad3ea700 1 --2- 172.21.15.42:0/612230497 >> v2:172.21.15.42:3300/0 conn(0x7ffba808df80 0x7ffba809f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7ffb98008790 tx=0x7ffb9800fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:27.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:27.820+0000 7ffbad3ea700 1 --2- 172.21.15.42:0/612230497 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ffba808df80 0x7ffba809f2e0 secure :-1 s=READY pgs=450 cs=0 l=1 rx=0x7ffb98008790 tx=0x7ffb9800fbf0).ready entity=mon.1 client_cookie=dd299306344446ba server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:11:27.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:27.820+0000 7ffbad3ea700 1 --2- 172.21.15.42:0/612230497 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ffba808df80 0x7ffba809f2e0 secure :-1 s=THROTTLE_DONE pgs=450 cs=0 l=1 rx=0x7ffb98008790 tx=0x7ffb9800fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:27.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:27.820+0000 7ffbacbe9700 1 -- 172.21.15.42:0/612230497 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ffb98011270 con 0x7ffba808df80 2019-12-05T05:11:27.837 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:27.820+0000 7ffbad3ea700 1 --2- 172.21.15.42:0/612230497 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ffba808df80 0x7ffba809f2e0 secure :-1 s=THROTTLE_DONE pgs=450 cs=0 l=1 rx=0x7ffb98008790 tx=0x7ffb9800fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:27.837 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:27.820+0000 7ffbacbe9700 1 -- 172.21.15.42:0/612230497 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb98008bc0 con 0x7ffba808df80 2019-12-05T05:11:27.837 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:27.820+0000 7ffbad3ea700 1 --2- 172.21.15.42:0/612230497 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ffba808df80 0x7ffba809f2e0 secure :-1 s=THROTTLE_DONE pgs=450 cs=0 l=1 rx=0x7ffb98008790 tx=0x7ffb9800fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:27.837 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:27.820+0000 7ffbacbe9700 1 -- 172.21.15.42:0/612230497 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ffb98011850 con 0x7ffba808df80 2019-12-05T05:11:27.837 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:27.820+0000 7ffbaf64e700 1 -- 172.21.15.42:0/612230497 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ffba808df80 msgr2=0x7ffba809f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:11:27.838 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:27.820+0000 7ffbaf64e700 1 --2- 172.21.15.42:0/612230497 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ffba808df80 0x7ffba809f2e0 secure :-1 s=READY pgs=450 cs=0 l=1 rx=0x7ffb98008790 tx=0x7ffb9800fbf0).stop 2019-12-05T05:11:27.838 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:27.824+0000 7ffbaf64e700 1 -- 172.21.15.42:0/612230497 shutdown_connections 2019-12-05T05:11:27.838 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:27.824+0000 7ffbaf64e700 1 --2- 172.21.15.42:0/612230497 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ffba808df80 0x7ffba809f2e0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:11:27.838 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:27.824+0000 7ffbaf64e700 1 -- 172.21.15.42:0/612230497 shutdown_connections 2019-12-05T05:11:27.838 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:27.824+0000 7ffbaf64e700 1 -- 172.21.15.42:0/612230497 wait complete. 2019-12-05T05:11:27.838 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:27.824+0000 7ffbaf64e700 1 -- 172.21.15.42:0/612230497 >> 172.21.15.42:0/612230497 conn(0x7ffba80aabe0 msgr2=0x7ffba80a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:11:27.838 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:27.824+0000 7ffbaf64e700 1 Processor -- start 2019-12-05T05:11:27.839 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:27.824+0000 7ffbaf64e700 1 -- start start 2019-12-05T05:11:27.839 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:27.824+0000 7ffbaf64e700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ffba80a1bc0 0x7ffba8123fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:11:27.839 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:27.824+0000 7ffbaf64e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ffba807d900 0x7ffba807dd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:11:27.839 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:27.824+0000 7ffbaf64e700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffba81227d0 con 0x7ffba812cb40 2019-12-05T05:11:27.840 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:27.824+0000 7ffbaf64e700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffba8122b50 con 0x7ffba80a1bc0 2019-12-05T05:11:27.840 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:27.824+0000 7ffbaf64e700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffba8122df0 con 0x7ffba807d900 2019-12-05T05:11:27.840 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:27.824+0000 7ffbad3ea700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ffba807d900 0x7ffba807dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:11:27.840 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:27.824+0000 7ffbad3ea700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ffba807d900 0x7ffba807dd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:57748/0 (socket says 172.21.15.42:57748) 2019-12-05T05:11:27.840 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:27.824+0000 7ffbad3ea700 1 -- 172.21.15.42:0/171550314 learned_addr learned my addr 172.21.15.42:0/171550314 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:11:27.840 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:27.824+0000 7ffbadbeb700 1 --2- 172.21.15.42:0/171550314 >> v2:172.21.15.40:3300/0 conn(0x7ffba80a1bc0 0x7ffba8123fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:11:27.841 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:27.828+0000 7ffbad3ea700 1 -- 172.21.15.42:0/171550314 >> v2:172.21.15.40:3300/0 conn(0x7ffba80a1bc0 msgr2=0x7ffba8123fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:11:27.841 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:27.828+0000 7ffbad3ea700 1 --2- 172.21.15.42:0/171550314 >> v2:172.21.15.40:3300/0 conn(0x7ffba80a1bc0 0x7ffba8123fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:11:27.841 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:27.828+0000 7ffbad3ea700 1 -- 172.21.15.42:0/171550314 >> v1:172.21.15.42:6789/0 conn(0x7ffba812cb40 legacy=0x7ffba8130100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:11:27.841 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:27.828+0000 7ffbad3ea700 1 -- 172.21.15.42:0/171550314 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffba8124520 con 0x7ffba807d900 2019-12-05T05:11:27.841 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:27.828+0000 7ffbad3ea700 1 --2- 172.21.15.42:0/171550314 >> v2:172.21.15.42:3300/0 conn(0x7ffba807d900 0x7ffba807dd30 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7ffb98008790 tx=0x7ffb98011990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:27.841 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:27.828+0000 7ffbad3ea700 1 --2- 172.21.15.42:0/171550314 >> v2:172.21.15.42:3300/0 conn(0x7ffba807d900 0x7ffba807dd30 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7ffb98008790 tx=0x7ffb98011990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:27.842 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:27.828+0000 7ffbad3ea700 1 --2- 172.21.15.42:0/171550314 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ffba807d900 0x7ffba807dd30 secure :-1 s=READY pgs=452 cs=0 l=1 rx=0x7ffb98008790 tx=0x7ffb98011990).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:11:27.842 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:27.828+0000 7ffbad3ea700 1 --2- 172.21.15.42:0/171550314 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ffba807d900 0x7ffba807dd30 secure :-1 s=THROTTLE_DONE pgs=452 cs=0 l=1 rx=0x7ffb98008790 tx=0x7ffb98011990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:27.842 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:27.828+0000 7ffbad3ea700 1 --2- 172.21.15.42:0/171550314 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ffba807d900 0x7ffba807dd30 secure :-1 s=THROTTLE_DONE pgs=452 cs=0 l=1 rx=0x7ffb98008790 tx=0x7ffb98011990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:27.842 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:27.828+0000 7ffbad3ea700 1 --2- 172.21.15.42:0/171550314 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ffba807d900 0x7ffba807dd30 secure :-1 s=THROTTLE_DONE pgs=452 cs=0 l=1 rx=0x7ffb98008790 tx=0x7ffb98011990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:27.842 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:27.828+0000 7ffb9e7fc700 1 -- 172.21.15.42:0/171550314 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ffb98007340 con 0x7ffba807d900 2019-12-05T05:11:27.842 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:27.828+0000 7ffb9e7fc700 1 -- 172.21.15.42:0/171550314 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb98011470 con 0x7ffba807d900 2019-12-05T05:11:27.842 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:27.828+0000 7ffb9e7fc700 1 -- 172.21.15.42:0/171550314 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ffb98005540 con 0x7ffba807d900 2019-12-05T05:11:27.843 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:27.828+0000 7ffbaf64e700 1 -- 172.21.15.42:0/171550314 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffba812cf70 con 0x7ffba807d900 2019-12-05T05:11:27.843 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:27.828+0000 7ffbaf64e700 1 -- 172.21.15.42:0/171550314 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffba812d470 con 0x7ffba807d900 2019-12-05T05:11:27.843 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:27.828+0000 7ffbad3ea700 1 --2- 172.21.15.42:0/171550314 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ffba807d900 0x7ffba807dd30 secure :-1 s=THROTTLE_DONE pgs=452 cs=0 l=1 rx=0x7ffb98008790 tx=0x7ffb98011990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:27.844 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:27.828+0000 7ffb9e7fc700 1 -- 172.21.15.42:0/171550314 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7ffb98005850 con 0x7ffba807d900 2019-12-05T05:11:27.844 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:27.828+0000 7ffbad3ea700 1 --2- 172.21.15.42:0/171550314 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ffba807d900 0x7ffba807dd30 secure :-1 s=THROTTLE_DONE pgs=452 cs=0 l=1 rx=0x7ffb98008790 tx=0x7ffb98011990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:27.844 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:27.828+0000 7ffb9e7fc700 1 --2- 172.21.15.42:0/171550314 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ffb9003ad40 0x7ffb9003d250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:11:27.845 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:27.828+0000 7ffb9e7fc700 1 -- 172.21.15.42:0/171550314 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6517+0+0 (secure 0 0 0) 0x7ffb9804bbd0 con 0x7ffba807d900 2019-12-05T05:11:27.845 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:27.828+0000 7ffbadbeb700 1 --2- 172.21.15.42:0/171550314 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ffb9003ad40 0x7ffb9003d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:11:27.845 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:27.828+0000 7ffbadbeb700 1 --2- 172.21.15.42:0/171550314 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ffb9003ad40 0x7ffb9003d250 crc :-1 s=READY pgs=203 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:11:27.845 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:27.836+0000 7ffbaf64e700 1 -- 172.21.15.42:0/171550314 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffba805d110 con 0x7ffba807d900 2019-12-05T05:11:27.846 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:27.840+0000 7ffbad3ea700 1 --2- 172.21.15.42:0/171550314 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ffba807d900 0x7ffba807dd30 secure :-1 s=THROTTLE_DONE pgs=452 cs=0 l=1 rx=0x7ffb98008790 tx=0x7ffb98011990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:27.846 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:27.840+0000 7ffb9e7fc700 1 -- 172.21.15.42:0/171550314 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7ffb98032c50 con 0x7ffba807d900 2019-12-05T05:11:28.016 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:28.008+0000 7ffbaf64e700 1 -- 172.21.15.42:0/171550314 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffba805b8c0 con 0x7ffba807d900 2019-12-05T05:11:28.016 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:28.008+0000 7ffbad3ea700 1 --2- 172.21.15.42:0/171550314 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ffba807d900 0x7ffba807dd30 secure :-1 s=THROTTLE_DONE pgs=452 cs=0 l=1 rx=0x7ffb98008790 tx=0x7ffb98011990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:11:28.016 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:28.008+0000 7ffb9e7fc700 1 -- 172.21.15.42:0/171550314 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) v1 ==== 94+0+2105 (secure 0 0 0) 0x7ffb98032eb0 con 0x7ffba807d900 2019-12-05T05:11:28.016 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 58 2019-12-05T05:11:28.017 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:11:28.017 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":58,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5391,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/814397242","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":814397242},{"type":"v1","addr":"172.21.15.40:6833","nonce":814397242}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":58}],"filesystems":[{"mdsmap":{"epoch":57,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:11:24.854205+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5215},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5215":{"gid":5215,"name":"a","rank":0,"incarnation":52,"state":"up:active","state_seq":20,"addr":"172.21.15.42:6835/3176146895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3176146895},{"type":"v1","addr":"172.21.15.42:6835","nonce":3176146895}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:11:28.018 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:28.012+0000 7ffbaf64e700 1 -- 172.21.15.42:0/171550314 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ffb9003ad40 msgr2=0x7ffb9003d250 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:11:28.018 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:28.012+0000 7ffbaf64e700 1 --2- 172.21.15.42:0/171550314 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ffb9003ad40 0x7ffb9003d250 crc :-1 s=READY pgs=203 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:11:28.018 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:28.012+0000 7ffbaf64e700 1 -- 172.21.15.42:0/171550314 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ffba807d900 msgr2=0x7ffba807dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:11:28.019 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:28.012+0000 7ffbaf64e700 1 --2- 172.21.15.42:0/171550314 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ffba807d900 0x7ffba807dd30 secure :-1 s=READY pgs=452 cs=0 l=1 rx=0x7ffb98008790 tx=0x7ffb98011990).stop 2019-12-05T05:11:28.019 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:28.012+0000 7ffbaf64e700 1 -- 172.21.15.42:0/171550314 shutdown_connections 2019-12-05T05:11:28.020 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:28.012+0000 7ffbaf64e700 1 --2- 172.21.15.42:0/171550314 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ffb9003ad40 0x7ffb9003d250 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:11:28.020 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:28.012+0000 7ffbaf64e700 1 --2- 172.21.15.42:0/171550314 >> v2:172.21.15.40:3300/0 conn(0x7ffba80a1bc0 0x7ffba8123fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:11:28.020 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:28.012+0000 7ffbaf64e700 1 --2- 172.21.15.42:0/171550314 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ffba807d900 0x7ffba807dd30 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:11:28.020 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:28.012+0000 7ffbaf64e700 1 -- 172.21.15.42:0/171550314 shutdown_connections 2019-12-05T05:11:28.020 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:28.012+0000 7ffbaf64e700 1 -- 172.21.15.42:0/171550314 wait complete. 2019-12-05T05:11:28.020 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:11:28.012+0000 7ffbaf64e700 1 -- 172.21.15.42:0/171550314 >> 172.21.15.42:0/171550314 conn(0x7ffba807e990 msgr2=0x7ffba809f3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:11:28.029 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b reported in up:standby state 2019-12-05T05:11:28.029 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 60 secs before thrashing 2019-12-05T05:11:35.243 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:11:35.247 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:12:05.289 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:12:05.292 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:12:28.030 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:12:28.134 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.124+0000 7fd09ce9e700 1 Processor -- start 2019-12-05T05:12:28.134 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.124+0000 7fd09ce9e700 1 -- start start 2019-12-05T05:12:28.135 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.124+0000 7fd09ce9e700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd0980a7740 0x7fd09808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:12:28.135 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.124+0000 7fd09ce9e700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd09807e720 0x7fd09807eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:12:28.135 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.124+0000 7fd09ce9e700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0980a0710 con 0x7fd09809f2e0 2019-12-05T05:12:28.135 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.124+0000 7fd09ce9e700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0980a0b00 con 0x7fd09807e720 2019-12-05T05:12:28.135 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.124+0000 7fd09ce9e700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd09808e5b0 con 0x7fd0980a7740 2019-12-05T05:12:28.135 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.124+0000 7fd096d9d700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd0980a7740 0x7fd09808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:12:28.136 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.124+0000 7fd09659c700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd09807e720 0x7fd09807eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:12:28.136 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.124+0000 7fd09659c700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd09807e720 0x7fd09807eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.42:45608/0 (socket says 172.21.15.42:45608) 2019-12-05T05:12:28.136 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.124+0000 7fd096d9d700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd0980a7740 0x7fd09808df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:58256/0 (socket says 172.21.15.42:58256) 2019-12-05T05:12:28.136 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.124+0000 7fd09659c700 1 -- 172.21.15.42:0/1921719946 learned_addr learned my addr 172.21.15.42:0/1921719946 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:12:28.136 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.124+0000 7fd09759e700 1 --1- 172.21.15.42:0/1921719946 >> v1:172.21.15.40:6789/0 conn(0x7fd09809f2e0 0x7fd09809f710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:39040/0 (socket says 172.21.15.42:39040) 2019-12-05T05:12:28.136 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.124+0000 7fd096d9d700 1 -- 172.21.15.42:0/1921719946 >> v2:172.21.15.40:3300/0 conn(0x7fd09807e720 msgr2=0x7fd09807eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:12:28.136 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.124+0000 7fd096d9d700 1 --2- 172.21.15.42:0/1921719946 >> v2:172.21.15.40:3300/0 conn(0x7fd09807e720 0x7fd09807eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:12:28.137 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.124+0000 7fd096d9d700 1 -- 172.21.15.42:0/1921719946 >> v1:172.21.15.40:6789/0 conn(0x7fd09809f2e0 legacy=0x7fd09809f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:12:28.137 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.124+0000 7fd096d9d700 1 -- 172.21.15.42:0/1921719946 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd09807f0a0 con 0x7fd0980a7740 2019-12-05T05:12:28.137 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.124+0000 7fd096d9d700 1 --2- 172.21.15.42:0/1921719946 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd0980a7740 0x7fd09808df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fd080008790 tx=0x7fd08000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:28.137 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.124+0000 7fd096d9d700 1 --2- 172.21.15.42:0/1921719946 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd0980a7740 0x7fd09808df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fd080008790 tx=0x7fd08000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:28.137 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.124+0000 7fd096d9d700 1 --2- 172.21.15.42:0/1921719946 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd0980a7740 0x7fd09808df80 secure :-1 s=READY pgs=87 cs=0 l=1 rx=0x7fd080008790 tx=0x7fd08000fc70).ready entity=mon.2 client_cookie=400860b4cbe999f4 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:12:28.137 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.128+0000 7fd096d9d700 1 --2- 172.21.15.42:0/1921719946 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd0980a7740 0x7fd09808df80 secure :-1 s=THROTTLE_DONE pgs=87 cs=0 l=1 rx=0x7fd080008790 tx=0x7fd08000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:28.137 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.128+0000 7fd095d9b700 1 -- 172.21.15.42:0/1921719946 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd0800112f0 con 0x7fd0980a7740 2019-12-05T05:12:28.138 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.128+0000 7fd096d9d700 1 --2- 172.21.15.42:0/1921719946 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd0980a7740 0x7fd09808df80 secure :-1 s=THROTTLE_DONE pgs=87 cs=0 l=1 rx=0x7fd080008790 tx=0x7fd08000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:28.138 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.128+0000 7fd096d9d700 1 --2- 172.21.15.42:0/1921719946 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd0980a7740 0x7fd09808df80 secure :-1 s=THROTTLE_DONE pgs=87 cs=0 l=1 rx=0x7fd080008790 tx=0x7fd08000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:28.138 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.128+0000 7fd095d9b700 1 -- 172.21.15.42:0/1921719946 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd080008ad0 con 0x7fd0980a7740 2019-12-05T05:12:28.138 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.128+0000 7fd095d9b700 1 -- 172.21.15.42:0/1921719946 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd0800118d0 con 0x7fd0980a7740 2019-12-05T05:12:28.138 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.128+0000 7fd09ce9e700 1 -- 172.21.15.42:0/1921719946 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd0980a7740 msgr2=0x7fd09808df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:12:28.138 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.128+0000 7fd09ce9e700 1 --2- 172.21.15.42:0/1921719946 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd0980a7740 0x7fd09808df80 secure :-1 s=READY pgs=87 cs=0 l=1 rx=0x7fd080008790 tx=0x7fd08000fc70).stop 2019-12-05T05:12:28.138 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.128+0000 7fd09ce9e700 1 -- 172.21.15.42:0/1921719946 shutdown_connections 2019-12-05T05:12:28.138 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.128+0000 7fd09ce9e700 1 --2- 172.21.15.42:0/1921719946 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd0980a7740 0x7fd09808df80 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:12:28.139 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.128+0000 7fd09ce9e700 1 --2- 172.21.15.42:0/1921719946 >> v2:172.21.15.40:3300/0 conn(0x7fd09807e720 0x7fd09807eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:12:28.139 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.128+0000 7fd09ce9e700 1 -- 172.21.15.42:0/1921719946 shutdown_connections 2019-12-05T05:12:28.139 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.128+0000 7fd09ce9e700 1 -- 172.21.15.42:0/1921719946 wait complete. 2019-12-05T05:12:28.139 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.128+0000 7fd09ce9e700 1 -- 172.21.15.42:0/1921719946 >> 172.21.15.42:0/1921719946 conn(0x7fd0980aabe0 msgr2=0x7fd0980a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:12:28.142 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.132+0000 7fd09ce9e700 1 Processor -- start 2019-12-05T05:12:28.142 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.132+0000 7fd09ce9e700 1 -- start start 2019-12-05T05:12:28.142 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.132+0000 7fd09ce9e700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd09811c230 0x7fd098117a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:12:28.142 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.132+0000 7fd09ce9e700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd098118410 con 0x7fd0981209c0 2019-12-05T05:12:28.143 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.132+0000 7fd09ce9e700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0980a6e70 con 0x7fd098117fe0 2019-12-05T05:12:28.143 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.132+0000 7fd09ce9e700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd09811ab70 con 0x7fd09811c230 2019-12-05T05:12:28.143 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.132+0000 7fd09759e700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fd098117fe0 0x7fd098134860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:57838/0 (socket says 172.21.15.42:57838) 2019-12-05T05:12:28.143 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.132+0000 7fd09759e700 1 -- 172.21.15.42:0/3600717186 learned_addr learned my addr 172.21.15.42:0/3600717186 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:12:28.143 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.132+0000 7fd09659c700 1 --2- 172.21.15.42:0/3600717186 >> v2:172.21.15.40:3300/0 conn(0x7fd09811c230 0x7fd098117a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:12:28.143 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.132+0000 7fd08f7fe700 1 -- 172.21.15.42:0/3600717186 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2928043602 0 0) 0x7fd0980a6e70 con 0x7fd098117fe0 2019-12-05T05:12:28.143 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.132+0000 7fd08f7fe700 1 -- 172.21.15.42:0/3600717186 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd07c0030b0 con 0x7fd098117fe0 2019-12-05T05:12:28.144 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.132+0000 7fd09659c700 1 -- 172.21.15.42:0/3600717186 >> v1:172.21.15.42:6789/0 conn(0x7fd098117fe0 legacy=0x7fd098134860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:12:28.144 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.132+0000 7fd09659c700 1 -- 172.21.15.42:0/3600717186 >> v1:172.21.15.40:6789/0 conn(0x7fd0981209c0 legacy=0x7fd098077c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:12:28.144 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.132+0000 7fd09659c700 1 -- 172.21.15.42:0/3600717186 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd09811a740 con 0x7fd09811c230 2019-12-05T05:12:28.144 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.132+0000 7fd09659c700 1 --2- 172.21.15.42:0/3600717186 >> v2:172.21.15.40:3300/0 conn(0x7fd09811c230 0x7fd098117a90 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fd088002f30 tx=0x7fd088010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:28.144 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.132+0000 7fd09659c700 1 --2- 172.21.15.42:0/3600717186 >> v2:172.21.15.40:3300/0 conn(0x7fd09811c230 0x7fd098117a90 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fd088002f30 tx=0x7fd088010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:28.144 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.132+0000 7fd09659c700 1 --2- 172.21.15.42:0/3600717186 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd09811c230 0x7fd098117a90 secure :-1 s=READY pgs=358 cs=0 l=1 rx=0x7fd088002f30 tx=0x7fd088010b50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:12:28.144 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.132+0000 7fd09659c700 1 --2- 172.21.15.42:0/3600717186 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd09811c230 0x7fd098117a90 secure :-1 s=THROTTLE_DONE pgs=358 cs=0 l=1 rx=0x7fd088002f30 tx=0x7fd088010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:28.145 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.132+0000 7fd08f7fe700 1 -- 172.21.15.42:0/3600717186 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd088005930 con 0x7fd09811c230 2019-12-05T05:12:28.145 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.132+0000 7fd09659c700 1 --2- 172.21.15.42:0/3600717186 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd09811c230 0x7fd098117a90 secure :-1 s=THROTTLE_DONE pgs=358 cs=0 l=1 rx=0x7fd088002f30 tx=0x7fd088010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:28.145 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.132+0000 7fd09659c700 1 --2- 172.21.15.42:0/3600717186 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd09811c230 0x7fd098117a90 secure :-1 s=THROTTLE_DONE pgs=358 cs=0 l=1 rx=0x7fd088002f30 tx=0x7fd088010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:28.145 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.132+0000 7fd08f7fe700 1 -- 172.21.15.42:0/3600717186 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd088007160 con 0x7fd09811c230 2019-12-05T05:12:28.145 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.132+0000 7fd09ce9e700 1 -- 172.21.15.42:0/3600717186 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd098078440 con 0x7fd09811c230 2019-12-05T05:12:28.145 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.132+0000 7fd08f7fe700 1 -- 172.21.15.42:0/3600717186 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd0880088b0 con 0x7fd09811c230 2019-12-05T05:12:28.145 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.132+0000 7fd09ce9e700 1 -- 172.21.15.42:0/3600717186 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd098136070 con 0x7fd09811c230 2019-12-05T05:12:28.146 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.136+0000 7fd09659c700 1 --2- 172.21.15.42:0/3600717186 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd09811c230 0x7fd098117a90 secure :-1 s=THROTTLE_DONE pgs=358 cs=0 l=1 rx=0x7fd088002f30 tx=0x7fd088010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:28.146 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.136+0000 7fd08f7fe700 1 -- 172.21.15.42:0/3600717186 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fd088008b50 con 0x7fd09811c230 2019-12-05T05:12:28.146 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.136+0000 7fd09659c700 1 --2- 172.21.15.42:0/3600717186 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd09811c230 0x7fd098117a90 secure :-1 s=THROTTLE_DONE pgs=358 cs=0 l=1 rx=0x7fd088002f30 tx=0x7fd088010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:28.146 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.136+0000 7fd08f7fe700 1 --2- 172.21.15.42:0/3600717186 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd07c03b1d0 0x7fd07c03d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:12:28.146 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.136+0000 7fd08f7fe700 1 -- 172.21.15.42:0/3600717186 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6489+0+0 (secure 0 0 0) 0x7fd088013060 con 0x7fd09811c230 2019-12-05T05:12:28.146 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.136+0000 7fd096d9d700 1 --2- 172.21.15.42:0/3600717186 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd07c03b1d0 0x7fd07c03d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:12:28.146 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.136+0000 7fd096d9d700 1 --2- 172.21.15.42:0/3600717186 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd07c03b1d0 0x7fd07c03d6e0 crc :-1 s=READY pgs=206 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:12:28.147 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.140+0000 7fd09ce9e700 1 -- 172.21.15.42:0/3600717186 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd09805d110 con 0x7fd09811c230 2019-12-05T05:12:28.154 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.148+0000 7fd09659c700 1 --2- 172.21.15.42:0/3600717186 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd09811c230 0x7fd098117a90 secure :-1 s=THROTTLE_DONE pgs=358 cs=0 l=1 rx=0x7fd088002f30 tx=0x7fd088010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:28.154 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.148+0000 7fd08f7fe700 1 -- 172.21.15.42:0/3600717186 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fd0880321f0 con 0x7fd09811c230 2019-12-05T05:12:28.320 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.316+0000 7fd09ce9e700 1 -- 172.21.15.42:0/3600717186 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd09805b8c0 con 0x7fd09811c230 2019-12-05T05:12:28.322 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.316+0000 7fd09659c700 1 --2- 172.21.15.42:0/3600717186 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd09811c230 0x7fd098117a90 secure :-1 s=THROTTLE_DONE pgs=358 cs=0 l=1 rx=0x7fd088002f30 tx=0x7fd088010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:28.322 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.316+0000 7fd08f7fe700 1 -- 172.21.15.42:0/3600717186 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) v1 ==== 94+0+2105 (secure 0 0 0) 0x7fd088032450 con 0x7fd09811c230 2019-12-05T05:12:28.324 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 58 2019-12-05T05:12:28.325 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:12:28.325 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":58,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5391,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/814397242","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":814397242},{"type":"v1","addr":"172.21.15.40:6833","nonce":814397242}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":58}],"filesystems":[{"mdsmap":{"epoch":57,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:11:24.854205+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5215},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5215":{"gid":5215,"name":"a","rank":0,"incarnation":52,"state":"up:active","state_seq":20,"addr":"172.21.15.42:6835/3176146895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3176146895},{"type":"v1","addr":"172.21.15.42:6835","nonce":3176146895}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:12:28.326 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.320+0000 7fd09ce9e700 1 -- 172.21.15.42:0/3600717186 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd07c03b1d0 msgr2=0x7fd07c03d6e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:12:28.327 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.320+0000 7fd09ce9e700 1 --2- 172.21.15.42:0/3600717186 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd07c03b1d0 0x7fd07c03d6e0 crc :-1 s=READY pgs=206 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:12:28.327 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.320+0000 7fd09ce9e700 1 -- 172.21.15.42:0/3600717186 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd09811c230 msgr2=0x7fd098117a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:12:28.327 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.320+0000 7fd09ce9e700 1 --2- 172.21.15.42:0/3600717186 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd09811c230 0x7fd098117a90 secure :-1 s=READY pgs=358 cs=0 l=1 rx=0x7fd088002f30 tx=0x7fd088010b50).stop 2019-12-05T05:12:28.327 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.320+0000 7fd09ce9e700 1 -- 172.21.15.42:0/3600717186 shutdown_connections 2019-12-05T05:12:28.328 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.320+0000 7fd09ce9e700 1 --2- 172.21.15.42:0/3600717186 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd07c03b1d0 0x7fd07c03d6e0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:12:28.328 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.320+0000 7fd09ce9e700 1 --2- 172.21.15.42:0/3600717186 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd09811c230 0x7fd098117a90 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:12:28.328 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.320+0000 7fd09ce9e700 1 -- 172.21.15.42:0/3600717186 shutdown_connections 2019-12-05T05:12:28.328 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.320+0000 7fd09ce9e700 1 -- 172.21.15.42:0/3600717186 wait complete. 2019-12-05T05:12:28.328 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.320+0000 7fd09ce9e700 1 -- 172.21.15.42:0/3600717186 >> 172.21.15.42:0/3600717186 conn(0x7fd09809f620 msgr2=0x7fd0980a9790 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:12:28.336 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.a (rank=0) 2019-12-05T05:12:28.337 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2019-12-05T05:12:28.337 INFO:teuthology.orchestra.run:waiting for 300 2019-12-05T05:12:28.350 INFO:tasks.ceph.mds.a:Stopped 2019-12-05T05:12:28.351 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:12:28.519 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.508+0000 7f6aa030a700 1 Processor -- start 2019-12-05T05:12:28.519 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.508+0000 7f6aa030a700 1 -- start start 2019-12-05T05:12:28.519 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.508+0000 7f6aa030a700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6a9808df70 0x7f6a9809f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:12:28.519 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.508+0000 7f6aa030a700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f6a9809f820 0x7f6a9814efd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:12:28.519 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.508+0000 7f6aa030a700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a9809fc50 con 0x7f6a980a7730 2019-12-05T05:12:28.520 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.508+0000 7f6aa030a700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a980a0af0 con 0x7f6a9809f820 2019-12-05T05:12:28.520 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.508+0000 7f6aa030a700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a9808e5a0 con 0x7f6a9808df70 2019-12-05T05:12:28.520 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.508+0000 7f6a9e0a6700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6a9808df70 0x7f6a9809f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:12:28.520 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.508+0000 7f6a9f0a8700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f6a9809f820 0x7f6a9814efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:12:28.520 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.508+0000 7f6a9f0a8700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f6a9809f820 0x7f6a9814efd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.42:45626/0 (socket says 172.21.15.42:45626) 2019-12-05T05:12:28.520 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.508+0000 7f6a9e0a6700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6a9808df70 0x7f6a9809f2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:58274/0 (socket says 172.21.15.42:58274) 2019-12-05T05:12:28.521 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.508+0000 7f6a9f0a8700 1 -- 172.21.15.42:0/2052639639 learned_addr learned my addr 172.21.15.42:0/2052639639 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:12:28.521 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.512+0000 7f6a9e0a6700 1 -- 172.21.15.42:0/2052639639 >> v2:172.21.15.40:3300/0 conn(0x7f6a9809f820 msgr2=0x7f6a9814efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:12:28.521 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.512+0000 7f6a9e0a6700 1 --2- 172.21.15.42:0/2052639639 >> v2:172.21.15.40:3300/0 conn(0x7f6a9809f820 0x7f6a9814efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:12:28.521 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.512+0000 7f6a9e0a6700 1 -- 172.21.15.42:0/2052639639 >> v1:172.21.15.40:6789/0 conn(0x7f6a980a7730 legacy=0x7f6a9807e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:12:28.521 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.512+0000 7f6a9e0a6700 1 -- 172.21.15.42:0/2052639639 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a980a0700 con 0x7f6a9808df70 2019-12-05T05:12:28.521 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.512+0000 7f6a9e0a6700 1 --2- 172.21.15.42:0/2052639639 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6a9808df70 0x7f6a9809f2d0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f6a88008790 tx=0x7f6a8800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:28.522 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.512+0000 7f6a9e0a6700 1 --2- 172.21.15.42:0/2052639639 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6a9808df70 0x7f6a9809f2d0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f6a88008790 tx=0x7f6a8800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:28.522 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.512+0000 7f6a9e0a6700 1 --2- 172.21.15.42:0/2052639639 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6a9808df70 0x7f6a9809f2d0 secure :-1 s=READY pgs=88 cs=0 l=1 rx=0x7f6a88008790 tx=0x7f6a8800fc70).ready entity=mon.2 client_cookie=17e28a17d42a65e server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:12:28.522 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.512+0000 7f6a9e0a6700 1 --2- 172.21.15.42:0/2052639639 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6a9808df70 0x7f6a9809f2d0 secure :-1 s=THROTTLE_DONE pgs=88 cs=0 l=1 rx=0x7f6a88008790 tx=0x7f6a8800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:28.522 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.512+0000 7f6a9d8a5700 1 -- 172.21.15.42:0/2052639639 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6a88008bb0 con 0x7f6a9808df70 2019-12-05T05:12:28.522 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.512+0000 7f6a9e0a6700 1 --2- 172.21.15.42:0/2052639639 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6a9808df70 0x7f6a9809f2d0 secure :-1 s=THROTTLE_DONE pgs=88 cs=0 l=1 rx=0x7f6a88008790 tx=0x7f6a8800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:28.522 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.512+0000 7f6a9e0a6700 1 --2- 172.21.15.42:0/2052639639 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6a9808df70 0x7f6a9809f2d0 secure :-1 s=THROTTLE_DONE pgs=88 cs=0 l=1 rx=0x7f6a88008790 tx=0x7f6a8800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:28.523 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.512+0000 7f6a9d8a5700 1 -- 172.21.15.42:0/2052639639 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a880087d0 con 0x7f6a9808df70 2019-12-05T05:12:28.523 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.512+0000 7f6a9d8a5700 1 -- 172.21.15.42:0/2052639639 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6a88011860 con 0x7f6a9808df70 2019-12-05T05:12:28.523 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.512+0000 7f6aa030a700 1 -- 172.21.15.42:0/2052639639 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6a9808df70 msgr2=0x7f6a9809f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:12:28.523 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.512+0000 7f6aa030a700 1 --2- 172.21.15.42:0/2052639639 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6a9808df70 0x7f6a9809f2d0 secure :-1 s=READY pgs=88 cs=0 l=1 rx=0x7f6a88008790 tx=0x7f6a8800fc70).stop 2019-12-05T05:12:28.523 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.512+0000 7f6aa030a700 1 -- 172.21.15.42:0/2052639639 shutdown_connections 2019-12-05T05:12:28.523 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.512+0000 7f6aa030a700 1 --2- 172.21.15.42:0/2052639639 >> v2:172.21.15.40:3300/0 conn(0x7f6a9809f820 0x7f6a9814efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:12:28.523 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.512+0000 7f6aa030a700 1 --2- 172.21.15.42:0/2052639639 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6a9808df70 0x7f6a9809f2d0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:12:28.524 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.512+0000 7f6aa030a700 1 -- 172.21.15.42:0/2052639639 shutdown_connections 2019-12-05T05:12:28.524 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.512+0000 7f6aa030a700 1 -- 172.21.15.42:0/2052639639 wait complete. 2019-12-05T05:12:28.524 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.512+0000 7f6aa030a700 1 -- 172.21.15.42:0/2052639639 >> 172.21.15.42:0/2052639639 conn(0x7f6a980aabd0 msgr2=0x7f6a980a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:12:28.535 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.516+0000 7f6aa030a700 1 Processor -- start 2019-12-05T05:12:28.535 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.516+0000 7f6aa030a700 1 -- start start 2019-12-05T05:12:28.535 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.516+0000 7f6aa030a700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6a98123460 0x7f6a98123b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:12:28.535 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.516+0000 7f6aa030a700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6a98124080 0x7f6a98079c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:12:28.535 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.516+0000 7f6aa030a700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a9807a1c0 con 0x7f6a98144330 2019-12-05T05:12:28.535 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.516+0000 7f6aa030a700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a98126a20 con 0x7f6a98123460 2019-12-05T05:12:28.536 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.516+0000 7f6aa030a700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a98126cc0 con 0x7f6a98124080 2019-12-05T05:12:28.536 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.516+0000 7f6a9e0a6700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6a98123460 0x7f6a98123b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:12:28.536 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.516+0000 7f6a9e0a6700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6a98123460 0x7f6a98123b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:57814/0 (socket says 172.21.15.42:57814) 2019-12-05T05:12:28.536 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.516+0000 7f6a9e0a6700 1 -- 172.21.15.42:0/2621730654 learned_addr learned my addr 172.21.15.42:0/2621730654 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:12:28.536 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.516+0000 7f6a9f0a8700 1 --2- 172.21.15.42:0/2621730654 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6a98124080 0x7f6a98079c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:12:28.536 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.516+0000 7f6a9e0a6700 1 -- 172.21.15.42:0/2621730654 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6a98124080 msgr2=0x7f6a98079c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:12:28.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.516+0000 7f6a9e0a6700 1 --2- 172.21.15.42:0/2621730654 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6a98124080 0x7f6a98079c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:12:28.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.516+0000 7f6a9e0a6700 1 -- 172.21.15.42:0/2621730654 >> v1:172.21.15.40:6789/0 conn(0x7f6a98144330 legacy=0x7f6a98127200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:12:28.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.516+0000 7f6a9e0a6700 1 -- 172.21.15.42:0/2621730654 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a9807a3f0 con 0x7f6a98123460 2019-12-05T05:12:28.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.516+0000 7f6a9e0a6700 1 --2- 172.21.15.42:0/2621730654 >> v2:172.21.15.42:3300/0 conn(0x7f6a98123460 0x7f6a98123b30 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f6a98126be0 tx=0x7f6a88008690).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:28.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.516+0000 7f6a9e0a6700 1 --2- 172.21.15.42:0/2621730654 >> v2:172.21.15.42:3300/0 conn(0x7f6a98123460 0x7f6a98123b30 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f6a98126be0 tx=0x7f6a88008690).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:28.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.516+0000 7f6a9e0a6700 1 --2- 172.21.15.42:0/2621730654 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6a98123460 0x7f6a98123b30 secure :-1 s=READY pgs=457 cs=0 l=1 rx=0x7f6a98126be0 tx=0x7f6a88008690).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:12:28.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.516+0000 7f6a9e0a6700 1 --2- 172.21.15.42:0/2621730654 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6a98123460 0x7f6a98123b30 secure :-1 s=THROTTLE_DONE pgs=457 cs=0 l=1 rx=0x7f6a98126be0 tx=0x7f6a88008690).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:28.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.516+0000 7f6a9e0a6700 1 --2- 172.21.15.42:0/2621730654 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6a98123460 0x7f6a98123b30 secure :-1 s=THROTTLE_DONE pgs=457 cs=0 l=1 rx=0x7f6a98126be0 tx=0x7f6a88008690).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:28.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.516+0000 7f6a9e0a6700 1 --2- 172.21.15.42:0/2621730654 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6a98123460 0x7f6a98123b30 secure :-1 s=THROTTLE_DONE pgs=457 cs=0 l=1 rx=0x7f6a98126be0 tx=0x7f6a88008690).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:28.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.516+0000 7f6a8f7fe700 1 -- 172.21.15.42:0/2621730654 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6a8800f470 con 0x7f6a98123460 2019-12-05T05:12:28.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.520+0000 7f6a8f7fe700 1 -- 172.21.15.42:0/2621730654 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a88008db0 con 0x7f6a98123460 2019-12-05T05:12:28.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.520+0000 7f6a8f7fe700 1 -- 172.21.15.42:0/2621730654 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6a88016740 con 0x7f6a98123460 2019-12-05T05:12:28.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.520+0000 7f6aa030a700 1 -- 172.21.15.42:0/2621730654 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a9808eb60 con 0x7f6a98123460 2019-12-05T05:12:28.539 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.520+0000 7f6aa030a700 1 -- 172.21.15.42:0/2621730654 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a9808f110 con 0x7f6a98123460 2019-12-05T05:12:28.539 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.520+0000 7f6a9e0a6700 1 --2- 172.21.15.42:0/2621730654 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6a98123460 0x7f6a98123b30 secure :-1 s=THROTTLE_DONE pgs=457 cs=0 l=1 rx=0x7f6a98126be0 tx=0x7f6a88008690).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:28.539 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.520+0000 7f6a8f7fe700 1 -- 172.21.15.42:0/2621730654 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f6a88016a50 con 0x7f6a98123460 2019-12-05T05:12:28.539 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.520+0000 7f6a9e0a6700 1 --2- 172.21.15.42:0/2621730654 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6a98123460 0x7f6a98123b30 secure :-1 s=THROTTLE_DONE pgs=457 cs=0 l=1 rx=0x7f6a98126be0 tx=0x7f6a88008690).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:28.539 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.520+0000 7f6a8f7fe700 1 --2- 172.21.15.42:0/2621730654 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6a8003adb0 0x7f6a8003d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:12:28.539 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.520+0000 7f6a8f7fe700 1 -- 172.21.15.42:0/2621730654 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6489+0+0 (secure 0 0 0) 0x7f6a8804d7a0 con 0x7f6a98123460 2019-12-05T05:12:28.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.520+0000 7f6a9e8a7700 1 --2- 172.21.15.42:0/2621730654 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6a8003adb0 0x7f6a8003d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:12:28.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.520+0000 7f6a9e8a7700 1 --2- 172.21.15.42:0/2621730654 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6a8003adb0 0x7f6a8003d2c0 crc :-1 s=READY pgs=207 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:12:28.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.524+0000 7f6aa030a700 1 -- 172.21.15.42:0/2621730654 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a9805d100 con 0x7f6a98123460 2019-12-05T05:12:28.541 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.528+0000 7f6a9e0a6700 1 --2- 172.21.15.42:0/2621730654 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6a98123460 0x7f6a98123b30 secure :-1 s=THROTTLE_DONE pgs=457 cs=0 l=1 rx=0x7f6a98126be0 tx=0x7f6a88008690).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:28.541 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.532+0000 7f6a8f7fe700 1 -- 172.21.15.42:0/2621730654 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f6a88034a80 con 0x7f6a98123460 2019-12-05T05:12:28.701 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.696+0000 7f6aa030a700 1 -- 172.21.15.42:0/2621730654 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6a9805b8b0 con 0x7f6a98123460 2019-12-05T05:12:28.702 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.696+0000 7f6a9e0a6700 1 --2- 172.21.15.42:0/2621730654 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6a98123460 0x7f6a98123b30 secure :-1 s=THROTTLE_DONE pgs=457 cs=0 l=1 rx=0x7f6a98126be0 tx=0x7f6a88008690).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:28.702 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.696+0000 7f6a8f7fe700 1 -- 172.21.15.42:0/2621730654 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) v1 ==== 94+0+2105 (secure 0 0 0) 0x7f6a8800ebb0 con 0x7f6a98123460 2019-12-05T05:12:28.702 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 58 2019-12-05T05:12:28.703 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:12:28.703 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":58,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5391,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/814397242","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":814397242},{"type":"v1","addr":"172.21.15.40:6833","nonce":814397242}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":58}],"filesystems":[{"mdsmap":{"epoch":57,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:11:24.854205+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5215},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5215":{"gid":5215,"name":"a","rank":0,"incarnation":52,"state":"up:active","state_seq":20,"addr":"172.21.15.42:6835/3176146895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3176146895},{"type":"v1","addr":"172.21.15.42:6835","nonce":3176146895}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:12:28.705 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.700+0000 7f6aa030a700 1 -- 172.21.15.42:0/2621730654 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6a8003adb0 msgr2=0x7f6a8003d2c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:12:28.705 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.700+0000 7f6aa030a700 1 --2- 172.21.15.42:0/2621730654 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6a8003adb0 0x7f6a8003d2c0 crc :-1 s=READY pgs=207 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:12:28.705 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.700+0000 7f6aa030a700 1 -- 172.21.15.42:0/2621730654 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6a98123460 msgr2=0x7f6a98123b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:12:28.705 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.700+0000 7f6aa030a700 1 --2- 172.21.15.42:0/2621730654 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6a98123460 0x7f6a98123b30 secure :-1 s=READY pgs=457 cs=0 l=1 rx=0x7f6a98126be0 tx=0x7f6a88008690).stop 2019-12-05T05:12:28.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.700+0000 7f6aa030a700 1 -- 172.21.15.42:0/2621730654 shutdown_connections 2019-12-05T05:12:28.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.700+0000 7f6aa030a700 1 --2- 172.21.15.42:0/2621730654 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6a8003adb0 0x7f6a8003d2c0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:12:28.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.700+0000 7f6aa030a700 1 --2- 172.21.15.42:0/2621730654 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6a98124080 0x7f6a98079c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:12:28.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.700+0000 7f6aa030a700 1 --2- 172.21.15.42:0/2621730654 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6a98123460 0x7f6a98123b30 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:12:28.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.700+0000 7f6aa030a700 1 -- 172.21.15.42:0/2621730654 shutdown_connections 2019-12-05T05:12:28.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.700+0000 7f6aa030a700 1 -- 172.21.15.42:0/2621730654 wait complete. 2019-12-05T05:12:28.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:28.700+0000 7f6aa030a700 1 -- 172.21.15.42:0/2621730654 >> 172.21.15.42:0/2621730654 conn(0x7f6a980a7730 msgr2=0x7f6a980a6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:12:28.715 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T05:12:30.715 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:12:30.832 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:30.820+0000 7fd4b9399700 1 Processor -- start 2019-12-05T05:12:30.832 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:30.820+0000 7fd4b9399700 1 -- start start 2019-12-05T05:12:30.833 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:30.824+0000 7fd4b9399700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd4b40a7740 0x7fd4b408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:12:30.833 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:30.824+0000 7fd4b9399700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd4b409fae0 0x7fd4b407eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:12:30.833 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:30.824+0000 7fd4b9399700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4b40a0710 con 0x7fd4b407e720 2019-12-05T05:12:30.833 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:30.824+0000 7fd4b9399700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4b40a0b00 con 0x7fd4b40a7740 2019-12-05T05:12:30.833 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:30.824+0000 7fd4b9399700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4b408e5b0 con 0x7fd4b409fae0 2019-12-05T05:12:30.833 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:30.824+0000 7fd4b3fff700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd4b409fae0 0x7fd4b407eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:12:30.834 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:30.824+0000 7fd4b3fff700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd4b409fae0 0x7fd4b407eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:58296/0 (socket says 172.21.15.42:58296) 2019-12-05T05:12:30.834 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:30.824+0000 7fd4b3fff700 1 -- 172.21.15.42:0/4045606171 learned_addr learned my addr 172.21.15.42:0/4045606171 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:12:30.834 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:30.824+0000 7fd4b37fe700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd4b40a7740 0x7fd4b408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:12:30.834 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:30.824+0000 7fd4b3fff700 1 -- 172.21.15.42:0/4045606171 >> v2:172.21.15.40:3300/0 conn(0x7fd4b40a7740 msgr2=0x7fd4b408df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:12:30.834 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:30.824+0000 7fd4b3fff700 1 --2- 172.21.15.42:0/4045606171 >> v2:172.21.15.40:3300/0 conn(0x7fd4b40a7740 0x7fd4b408df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:12:30.834 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:30.824+0000 7fd4b3fff700 1 -- 172.21.15.42:0/4045606171 >> v1:172.21.15.40:6789/0 conn(0x7fd4b407e720 legacy=0x7fd4b409f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:12:30.834 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:30.824+0000 7fd4b3fff700 1 -- 172.21.15.42:0/4045606171 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4b407f0a0 con 0x7fd4b409fae0 2019-12-05T05:12:30.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:30.824+0000 7fd4b3fff700 1 --2- 172.21.15.42:0/4045606171 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd4b409fae0 0x7fd4b407eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fd4ac008b10 tx=0x7fd4ac0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:30.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:30.824+0000 7fd4b3fff700 1 --2- 172.21.15.42:0/4045606171 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd4b409fae0 0x7fd4b407eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fd4ac008b10 tx=0x7fd4ac0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:30.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:30.824+0000 7fd4b3fff700 1 --2- 172.21.15.42:0/4045606171 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd4b409fae0 0x7fd4b407eb50 secure :-1 s=READY pgs=89 cs=0 l=1 rx=0x7fd4ac008b10 tx=0x7fd4ac0100b0).ready entity=mon.2 client_cookie=b014d7f3908ac8b1 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:12:30.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:30.824+0000 7fd4b3fff700 1 --2- 172.21.15.42:0/4045606171 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd4b409fae0 0x7fd4b407eb50 secure :-1 s=THROTTLE_DONE pgs=89 cs=0 l=1 rx=0x7fd4ac008b10 tx=0x7fd4ac0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:30.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:30.824+0000 7fd4b3fff700 1 --2- 172.21.15.42:0/4045606171 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd4b409fae0 0x7fd4b407eb50 secure :-1 s=THROTTLE_DONE pgs=89 cs=0 l=1 rx=0x7fd4ac008b10 tx=0x7fd4ac0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:30.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:30.824+0000 7fd4b27fc700 1 -- 172.21.15.42:0/4045606171 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd4ac010da0 con 0x7fd4b409fae0 2019-12-05T05:12:30.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:30.824+0000 7fd4b3fff700 1 --2- 172.21.15.42:0/4045606171 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd4b409fae0 0x7fd4b407eb50 secure :-1 s=THROTTLE_DONE pgs=89 cs=0 l=1 rx=0x7fd4ac008b10 tx=0x7fd4ac0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:30.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:30.824+0000 7fd4b27fc700 1 -- 172.21.15.42:0/4045606171 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4ac0109c0 con 0x7fd4b409fae0 2019-12-05T05:12:30.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:30.824+0000 7fd4b27fc700 1 -- 172.21.15.42:0/4045606171 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd4ac0114a0 con 0x7fd4b409fae0 2019-12-05T05:12:30.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:30.824+0000 7fd4b9399700 1 -- 172.21.15.42:0/4045606171 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd4b409fae0 msgr2=0x7fd4b407eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:12:30.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:30.824+0000 7fd4b9399700 1 --2- 172.21.15.42:0/4045606171 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd4b409fae0 0x7fd4b407eb50 secure :-1 s=READY pgs=89 cs=0 l=1 rx=0x7fd4ac008b10 tx=0x7fd4ac0100b0).stop 2019-12-05T05:12:30.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:30.824+0000 7fd4b9399700 1 -- 172.21.15.42:0/4045606171 shutdown_connections 2019-12-05T05:12:30.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:30.824+0000 7fd4b9399700 1 --2- 172.21.15.42:0/4045606171 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd4b409fae0 0x7fd4b407eb50 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:12:30.837 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:30.824+0000 7fd4b9399700 1 --2- 172.21.15.42:0/4045606171 >> v2:172.21.15.40:3300/0 conn(0x7fd4b40a7740 0x7fd4b408df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:12:30.837 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:30.824+0000 7fd4b9399700 1 -- 172.21.15.42:0/4045606171 shutdown_connections 2019-12-05T05:12:30.837 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:30.824+0000 7fd4b9399700 1 -- 172.21.15.42:0/4045606171 wait complete. 2019-12-05T05:12:30.837 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:30.824+0000 7fd4b9399700 1 -- 172.21.15.42:0/4045606171 >> 172.21.15.42:0/4045606171 conn(0x7fd4b40aabe0 msgr2=0x7fd4b40a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:12:30.840 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:30.828+0000 7fd4b9399700 1 Processor -- start 2019-12-05T05:12:30.840 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:30.828+0000 7fd4b9399700 1 -- start start 2019-12-05T05:12:30.841 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:30.828+0000 7fd4b9399700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd4b4144b50 0x7fd4b4122430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:12:30.841 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:30.828+0000 7fd4b9399700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd4b4119df0 0x7fd4b4122980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:12:30.841 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:30.828+0000 7fd4b9399700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4b41241b0 con 0x7fd4b4119220 2019-12-05T05:12:30.841 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:30.828+0000 7fd4b9399700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4b4121340 con 0x7fd4b4144b50 2019-12-05T05:12:30.841 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:30.828+0000 7fd4b9399700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4b408f530 con 0x7fd4b4119df0 2019-12-05T05:12:30.841 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:30.828+0000 7fd4b3fff700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd4b4119df0 0x7fd4b4122980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:12:30.842 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:30.828+0000 7fd4b37fe700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd4b4144b50 0x7fd4b4122430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:12:30.842 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:30.828+0000 7fd4b3fff700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd4b4119df0 0x7fd4b4122980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:58302/0 (socket says 172.21.15.42:58302) 2019-12-05T05:12:30.842 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:30.828+0000 7fd4b3fff700 1 -- 172.21.15.42:0/2171431 learned_addr learned my addr 172.21.15.42:0/2171431 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:12:30.842 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:30.828+0000 7fd4b37fe700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd4b4144b50 0x7fd4b4122430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.42:45648/0 (socket says 172.21.15.42:45648) 2019-12-05T05:12:30.842 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:30.828+0000 7fd4b3fff700 1 -- 172.21.15.42:0/2171431 >> v2:172.21.15.40:3300/0 conn(0x7fd4b4144b50 msgr2=0x7fd4b4122430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:12:30.842 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:30.828+0000 7fd4b3fff700 1 --2- 172.21.15.42:0/2171431 >> v2:172.21.15.40:3300/0 conn(0x7fd4b4144b50 0x7fd4b4122430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:12:30.842 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:30.828+0000 7fd4b3fff700 1 -- 172.21.15.42:0/2171431 >> v1:172.21.15.40:6789/0 conn(0x7fd4b4119220 legacy=0x7fd4b4131640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:12:30.843 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:30.828+0000 7fd49bfff700 1 -- 172.21.15.42:0/2171431 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3110509013 0 0) 0x7fd4b41241b0 con 0x7fd4b4119220 2019-12-05T05:12:30.843 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:30.828+0000 7fd4b3fff700 1 -- 172.21.15.42:0/2171431 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4b408f220 con 0x7fd4b4119df0 2019-12-05T05:12:30.843 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:30.828+0000 7fd4b37fe700 1 --2- 172.21.15.42:0/2171431 >> v2:172.21.15.40:3300/0 conn(0x7fd4b4144b50 0x7fd4b4122430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:12:30.843 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:30.832+0000 7fd4b3fff700 1 --2- 172.21.15.42:0/2171431 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd4b4119df0 0x7fd4b4122980 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fd4ac008b10 tx=0x7fd4ac010cf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:30.843 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:30.832+0000 7fd4b3fff700 1 --2- 172.21.15.42:0/2171431 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd4b4119df0 0x7fd4b4122980 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fd4ac008b10 tx=0x7fd4ac010cf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:30.843 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:30.832+0000 7fd4b3fff700 1 --2- 172.21.15.42:0/2171431 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd4b4119df0 0x7fd4b4122980 secure :-1 s=READY pgs=90 cs=0 l=1 rx=0x7fd4ac008b10 tx=0x7fd4ac010cf0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:12:30.843 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:30.832+0000 7fd4b3fff700 1 --2- 172.21.15.42:0/2171431 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd4b4119df0 0x7fd4b4122980 secure :-1 s=THROTTLE_DONE pgs=90 cs=0 l=1 rx=0x7fd4ac008b10 tx=0x7fd4ac010cf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:30.844 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:30.832+0000 7fd49bfff700 1 -- 172.21.15.42:0/2171431 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd4ac011bb0 con 0x7fd4b4119df0 2019-12-05T05:12:30.844 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:30.832+0000 7fd4b3fff700 1 --2- 172.21.15.42:0/2171431 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd4b4119df0 0x7fd4b4122980 secure :-1 s=THROTTLE_DONE pgs=90 cs=0 l=1 rx=0x7fd4ac008b10 tx=0x7fd4ac010cf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:30.844 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:30.832+0000 7fd4b3fff700 1 --2- 172.21.15.42:0/2171431 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd4b4119df0 0x7fd4b4122980 secure :-1 s=THROTTLE_DONE pgs=90 cs=0 l=1 rx=0x7fd4ac008b10 tx=0x7fd4ac010cf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:30.844 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:30.832+0000 7fd49bfff700 1 -- 172.21.15.42:0/2171431 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4ac0110c0 con 0x7fd4b4119df0 2019-12-05T05:12:30.844 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:30.832+0000 7fd4b9399700 1 -- 172.21.15.42:0/2171431 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4b4131e40 con 0x7fd4b4119df0 2019-12-05T05:12:30.845 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:30.832+0000 7fd49bfff700 1 -- 172.21.15.42:0/2171431 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd4ac005ab0 con 0x7fd4b4119df0 2019-12-05T05:12:30.845 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:30.832+0000 7fd4b9399700 1 -- 172.21.15.42:0/2171431 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4b4132370 con 0x7fd4b4119df0 2019-12-05T05:12:30.845 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:30.832+0000 7fd4b3fff700 1 --2- 172.21.15.42:0/2171431 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd4b4119df0 0x7fd4b4122980 secure :-1 s=THROTTLE_DONE pgs=90 cs=0 l=1 rx=0x7fd4ac008b10 tx=0x7fd4ac010cf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:30.845 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:30.832+0000 7fd49bfff700 1 -- 172.21.15.42:0/2171431 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fd4ac00f280 con 0x7fd4b4119df0 2019-12-05T05:12:30.845 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:30.832+0000 7fd4b3fff700 1 --2- 172.21.15.42:0/2171431 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd4b4119df0 0x7fd4b4122980 secure :-1 s=THROTTLE_DONE pgs=90 cs=0 l=1 rx=0x7fd4ac008b10 tx=0x7fd4ac010cf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:30.845 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:30.832+0000 7fd49bfff700 1 --2- 172.21.15.42:0/2171431 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd49403ac90 0x7fd49403d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:12:30.845 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:30.832+0000 7fd49bfff700 1 -- 172.21.15.42:0/2171431 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6489+0+0 (secure 0 0 0) 0x7fd4ac04b510 con 0x7fd4b4119df0 2019-12-05T05:12:30.846 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:30.832+0000 7fd4b37fe700 1 --2- 172.21.15.42:0/2171431 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd49403ac90 0x7fd49403d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:12:30.846 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:30.836+0000 7fd4b37fe700 1 --2- 172.21.15.42:0/2171431 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd49403ac90 0x7fd49403d1a0 crc :-1 s=READY pgs=208 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:12:30.846 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:30.836+0000 7fd4b9399700 1 -- 172.21.15.42:0/2171431 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4b405d110 con 0x7fd4b4119df0 2019-12-05T05:12:30.851 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:30.844+0000 7fd4b3fff700 1 --2- 172.21.15.42:0/2171431 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd4b4119df0 0x7fd4b4122980 secure :-1 s=THROTTLE_DONE pgs=90 cs=0 l=1 rx=0x7fd4ac008b10 tx=0x7fd4ac010cf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:30.851 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:30.844+0000 7fd49bfff700 1 -- 172.21.15.42:0/2171431 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fd4ac032590 con 0x7fd4b4119df0 2019-12-05T05:12:31.024 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:31.020+0000 7fd4b9399700 1 -- 172.21.15.42:0/2171431 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd4b405b8c0 con 0x7fd4b4119df0 2019-12-05T05:12:31.025 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:31.020+0000 7fd4b3fff700 1 --2- 172.21.15.42:0/2171431 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd4b4119df0 0x7fd4b4122980 secure :-1 s=THROTTLE_DONE pgs=90 cs=0 l=1 rx=0x7fd4ac008b10 tx=0x7fd4ac010cf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:31.025 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:31.020+0000 7fd49bfff700 1 -- 172.21.15.42:0/2171431 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) v1 ==== 94+0+2105 (secure 0 0 0) 0x7fd4ac005dc0 con 0x7fd4b4119df0 2019-12-05T05:12:31.027 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 58 2019-12-05T05:12:31.027 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:12:31.027 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":58,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5391,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/814397242","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":814397242},{"type":"v1","addr":"172.21.15.40:6833","nonce":814397242}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":58}],"filesystems":[{"mdsmap":{"epoch":57,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:11:24.854205+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5215},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5215":{"gid":5215,"name":"a","rank":0,"incarnation":52,"state":"up:active","state_seq":20,"addr":"172.21.15.42:6835/3176146895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3176146895},{"type":"v1","addr":"172.21.15.42:6835","nonce":3176146895}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:12:31.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:31.024+0000 7fd4b9399700 1 -- 172.21.15.42:0/2171431 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd49403ac90 msgr2=0x7fd49403d1a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:12:31.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:31.024+0000 7fd4b9399700 1 --2- 172.21.15.42:0/2171431 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd49403ac90 0x7fd49403d1a0 crc :-1 s=READY pgs=208 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:12:31.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:31.024+0000 7fd4b9399700 1 -- 172.21.15.42:0/2171431 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd4b4119df0 msgr2=0x7fd4b4122980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:12:31.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:31.024+0000 7fd4b9399700 1 --2- 172.21.15.42:0/2171431 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd4b4119df0 0x7fd4b4122980 secure :-1 s=READY pgs=90 cs=0 l=1 rx=0x7fd4ac008b10 tx=0x7fd4ac010cf0).stop 2019-12-05T05:12:31.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:31.024+0000 7fd4b9399700 1 -- 172.21.15.42:0/2171431 shutdown_connections 2019-12-05T05:12:31.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:31.024+0000 7fd4b9399700 1 --2- 172.21.15.42:0/2171431 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd49403ac90 0x7fd49403d1a0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:12:31.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:31.024+0000 7fd4b9399700 1 --2- 172.21.15.42:0/2171431 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd4b4119df0 0x7fd4b4122980 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:12:31.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:31.024+0000 7fd4b9399700 1 --2- 172.21.15.42:0/2171431 >> v2:172.21.15.40:3300/0 conn(0x7fd4b4144b50 0x7fd4b4122430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:12:31.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:31.024+0000 7fd4b9399700 1 -- 172.21.15.42:0/2171431 shutdown_connections 2019-12-05T05:12:31.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:31.024+0000 7fd4b9399700 1 -- 172.21.15.42:0/2171431 wait complete. 2019-12-05T05:12:31.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:31.024+0000 7fd4b9399700 1 -- 172.21.15.42:0/2171431 >> 172.21.15.42:0/2171431 conn(0x7fd4b40a8080 msgr2=0x7fd4b40a9770 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:12:31.047 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T05:12:33.048 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:12:33.158 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:33.148+0000 7f34eedf1700 1 Processor -- start 2019-12-05T05:12:33.158 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:33.148+0000 7f34eedf1700 1 -- start start 2019-12-05T05:12:33.159 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:33.148+0000 7f34eedf1700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f34e808df80 0x7f34e809f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:12:33.160 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:33.148+0000 7f34eedf1700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34e809fc60 con 0x7f34e80a7740 2019-12-05T05:12:33.160 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:33.148+0000 7f34eedf1700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34e80a0710 con 0x7f34e809f830 2019-12-05T05:12:33.160 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:33.148+0000 7f34eedf1700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34e80a0b00 con 0x7f34e808df80 2019-12-05T05:12:33.160 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:33.148+0000 7f34edb8f700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f34e809f830 0x7f34e814efe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:57886/0 (socket says 172.21.15.42:57886) 2019-12-05T05:12:33.160 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:33.148+0000 7f34edb8f700 1 -- 172.21.15.42:0/2030478021 learned_addr learned my addr 172.21.15.42:0/2030478021 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:12:33.161 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:33.148+0000 7f34ecb8d700 1 --2- 172.21.15.42:0/2030478021 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f34e808df80 0x7f34e809f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:12:33.161 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:33.148+0000 7f34dffff700 1 -- 172.21.15.42:0/2030478021 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1595365464 0 0) 0x7f34e80a0710 con 0x7f34e809f830 2019-12-05T05:12:33.161 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:33.148+0000 7f34dffff700 1 -- 172.21.15.42:0/2030478021 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34d40030b0 con 0x7f34e809f830 2019-12-05T05:12:33.161 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:33.148+0000 7f34dffff700 1 -- 172.21.15.42:0/2030478021 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 969622129 0 0) 0x7f34e809fc60 con 0x7f34e80a7740 2019-12-05T05:12:33.161 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:33.148+0000 7f34dffff700 1 -- 172.21.15.42:0/2030478021 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34e80a0710 con 0x7f34e80a7740 2019-12-05T05:12:33.161 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:33.148+0000 7f34ecb8d700 1 -- 172.21.15.42:0/2030478021 >> v1:172.21.15.42:6789/0 conn(0x7f34e809f830 legacy=0x7f34e814efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:12:33.162 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:33.148+0000 7f34ecb8d700 1 -- 172.21.15.42:0/2030478021 >> v1:172.21.15.40:6789/0 conn(0x7f34e80a7740 legacy=0x7f34e807e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:12:33.162 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:33.148+0000 7f34ecb8d700 1 -- 172.21.15.42:0/2030478021 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34e807ef20 con 0x7f34e808df80 2019-12-05T05:12:33.162 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:33.148+0000 7f34ecb8d700 1 --2- 172.21.15.42:0/2030478021 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f34e808df80 0x7f34e809f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f34d8002f30 tx=0x7f34d800fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:33.162 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:33.148+0000 7f34ecb8d700 1 --2- 172.21.15.42:0/2030478021 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f34e808df80 0x7f34e809f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f34d8002f30 tx=0x7f34d800fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:33.162 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:33.148+0000 7f34ecb8d700 1 --2- 172.21.15.42:0/2030478021 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f34e808df80 0x7f34e809f2e0 secure :-1 s=READY pgs=91 cs=0 l=1 rx=0x7f34d8002f30 tx=0x7f34d800fbf0).ready entity=mon.2 client_cookie=77953ca6ef2ec02e server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:12:33.163 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:33.152+0000 7f34ecb8d700 1 --2- 172.21.15.42:0/2030478021 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f34e808df80 0x7f34e809f2e0 secure :-1 s=THROTTLE_DONE pgs=91 cs=0 l=1 rx=0x7f34d8002f30 tx=0x7f34d800fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:33.163 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:33.152+0000 7f34dffff700 1 -- 172.21.15.42:0/2030478021 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f34d80112b0 con 0x7f34e808df80 2019-12-05T05:12:33.163 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:33.152+0000 7f34ecb8d700 1 --2- 172.21.15.42:0/2030478021 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f34e808df80 0x7f34e809f2e0 secure :-1 s=THROTTLE_DONE pgs=91 cs=0 l=1 rx=0x7f34d8002f30 tx=0x7f34d800fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:33.163 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:33.152+0000 7f34ecb8d700 1 --2- 172.21.15.42:0/2030478021 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f34e808df80 0x7f34e809f2e0 secure :-1 s=THROTTLE_DONE pgs=91 cs=0 l=1 rx=0x7f34d8002f30 tx=0x7f34d800fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:33.163 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:33.152+0000 7f34dffff700 1 -- 172.21.15.42:0/2030478021 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34d8008bc0 con 0x7f34e808df80 2019-12-05T05:12:33.163 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:33.152+0000 7f34dffff700 1 -- 172.21.15.42:0/2030478021 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f34d8011890 con 0x7f34e808df80 2019-12-05T05:12:33.163 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:33.152+0000 7f34eedf1700 1 -- 172.21.15.42:0/2030478021 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f34e808df80 msgr2=0x7f34e809f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:12:33.164 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:33.152+0000 7f34eedf1700 1 --2- 172.21.15.42:0/2030478021 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f34e808df80 0x7f34e809f2e0 secure :-1 s=READY pgs=91 cs=0 l=1 rx=0x7f34d8002f30 tx=0x7f34d800fbf0).stop 2019-12-05T05:12:33.164 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:33.152+0000 7f34eedf1700 1 -- 172.21.15.42:0/2030478021 shutdown_connections 2019-12-05T05:12:33.164 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:33.152+0000 7f34eedf1700 1 --2- 172.21.15.42:0/2030478021 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f34e808df80 0x7f34e809f2e0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:12:33.164 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:33.152+0000 7f34eedf1700 1 -- 172.21.15.42:0/2030478021 shutdown_connections 2019-12-05T05:12:33.164 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:33.152+0000 7f34eedf1700 1 -- 172.21.15.42:0/2030478021 wait complete. 2019-12-05T05:12:33.164 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:33.152+0000 7f34eedf1700 1 -- 172.21.15.42:0/2030478021 >> 172.21.15.42:0/2030478021 conn(0x7f34e80aabe0 msgr2=0x7f34e80a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:12:33.168 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:33.156+0000 7f34eedf1700 1 Processor -- start 2019-12-05T05:12:33.168 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:33.156+0000 7f34eedf1700 1 -- start start 2019-12-05T05:12:33.168 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:33.156+0000 7f34eedf1700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f34e811bbe0 0x7f34e807c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:12:33.168 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:33.156+0000 7f34eedf1700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f34e8143ba0 0x7f34e807cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:12:33.169 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:33.156+0000 7f34eedf1700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34e80a44e0 con 0x7f34e81306e0 2019-12-05T05:12:33.169 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:33.156+0000 7f34eedf1700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34e8144180 con 0x7f34e8143ba0 2019-12-05T05:12:33.169 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:33.156+0000 7f34eedf1700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34e8144420 con 0x7f34e811bbe0 2019-12-05T05:12:33.169 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:33.156+0000 7f34edb8f700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f34e81306e0 0x7f34e8130b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:57892/0 (socket says 172.21.15.42:57892) 2019-12-05T05:12:33.171 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:33.156+0000 7f34edb8f700 1 -- 172.21.15.42:0/2052951202 learned_addr learned my addr 172.21.15.42:0/2052951202 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:12:33.172 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:33.156+0000 7f34ed38e700 1 --2- 172.21.15.42:0/2052951202 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f34e811bbe0 0x7f34e807c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:12:33.172 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:33.156+0000 7f34ecb8d700 1 --2- 172.21.15.42:0/2052951202 >> v2:172.21.15.40:3300/0 conn(0x7f34e8143ba0 0x7f34e807cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:12:33.172 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:33.156+0000 7f34ddffb700 1 -- 172.21.15.42:0/2052951202 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 915704921 0 0) 0x7f34e80a44e0 con 0x7f34e81306e0 2019-12-05T05:12:33.172 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:33.156+0000 7f34ddffb700 1 -- 172.21.15.42:0/2052951202 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34c80030b0 con 0x7f34e81306e0 2019-12-05T05:12:33.175 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:33.156+0000 7f34ed38e700 1 -- 172.21.15.42:0/2052951202 >> v2:172.21.15.40:3300/0 conn(0x7f34e8143ba0 msgr2=0x7f34e807cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:12:33.175 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:33.156+0000 7f34ed38e700 1 --2- 172.21.15.42:0/2052951202 >> v2:172.21.15.40:3300/0 conn(0x7f34e8143ba0 0x7f34e807cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:12:33.175 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:33.156+0000 7f34ed38e700 1 -- 172.21.15.42:0/2052951202 >> v1:172.21.15.42:6789/0 conn(0x7f34e81306e0 legacy=0x7f34e8130b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:12:33.175 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:33.156+0000 7f34ddffb700 1 -- 172.21.15.42:0/2052951202 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 4000890656 0 0) 0x7f34c80030b0 con 0x7f34e81306e0 2019-12-05T05:12:33.176 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:33.156+0000 7f34ed38e700 1 -- 172.21.15.42:0/2052951202 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34e80a1a20 con 0x7f34e811bbe0 2019-12-05T05:12:33.176 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:33.156+0000 7f34ed38e700 1 --2- 172.21.15.42:0/2052951202 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f34e811bbe0 0x7f34e807c7b0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f34e0001e20 tx=0x7f34e0001930).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:33.176 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:33.156+0000 7f34ed38e700 1 --2- 172.21.15.42:0/2052951202 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f34e811bbe0 0x7f34e807c7b0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f34e0001e20 tx=0x7f34e0001930).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:33.176 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:33.156+0000 7f34ed38e700 1 --2- 172.21.15.42:0/2052951202 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f34e811bbe0 0x7f34e807c7b0 secure :-1 s=READY pgs=92 cs=0 l=1 rx=0x7f34e0001e20 tx=0x7f34e0001930).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:12:33.176 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:33.160+0000 7f34ed38e700 1 --2- 172.21.15.42:0/2052951202 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f34e811bbe0 0x7f34e807c7b0 secure :-1 s=THROTTLE_DONE pgs=92 cs=0 l=1 rx=0x7f34e0001e20 tx=0x7f34e0001930).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:33.177 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:33.160+0000 7f34ddffb700 1 -- 172.21.15.42:0/2052951202 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f34e0001be0 con 0x7f34e811bbe0 2019-12-05T05:12:33.177 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:33.160+0000 7f34ed38e700 1 --2- 172.21.15.42:0/2052951202 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f34e811bbe0 0x7f34e807c7b0 secure :-1 s=THROTTLE_DONE pgs=92 cs=0 l=1 rx=0x7f34e0001e20 tx=0x7f34e0001930).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:33.177 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:33.160+0000 7f34ed38e700 1 --2- 172.21.15.42:0/2052951202 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f34e811bbe0 0x7f34e807c7b0 secure :-1 s=THROTTLE_DONE pgs=92 cs=0 l=1 rx=0x7f34e0001e20 tx=0x7f34e0001930).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:33.177 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:33.160+0000 7f34ddffb700 1 -- 172.21.15.42:0/2052951202 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34e4011e10 con 0x7f34e811bbe0 2019-12-05T05:12:33.177 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:33.160+0000 7f34eedf1700 1 -- 172.21.15.42:0/2052951202 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34e8137a90 con 0x7f34e811bbe0 2019-12-05T05:12:33.177 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:33.160+0000 7f34ddffb700 1 -- 172.21.15.42:0/2052951202 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f34e400ca80 con 0x7f34e811bbe0 2019-12-05T05:12:33.177 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:33.160+0000 7f34eedf1700 1 -- 172.21.15.42:0/2052951202 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34e8137fc0 con 0x7f34e811bbe0 2019-12-05T05:12:33.178 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:33.160+0000 7f34ed38e700 1 --2- 172.21.15.42:0/2052951202 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f34e811bbe0 0x7f34e807c7b0 secure :-1 s=THROTTLE_DONE pgs=92 cs=0 l=1 rx=0x7f34e0001e20 tx=0x7f34e0001930).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:33.178 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:33.160+0000 7f34ed38e700 1 --2- 172.21.15.42:0/2052951202 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f34e811bbe0 0x7f34e807c7b0 secure :-1 s=THROTTLE_DONE pgs=92 cs=0 l=1 rx=0x7f34e0001e20 tx=0x7f34e0001930).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:33.178 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:33.160+0000 7f34ddffb700 1 -- 172.21.15.42:0/2052951202 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f34e400d490 con 0x7f34e811bbe0 2019-12-05T05:12:33.178 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:33.160+0000 7f34ddffb700 1 --2- 172.21.15.42:0/2052951202 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f34c803b2d0 0x7f34c803d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:12:33.178 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:33.160+0000 7f34ddffb700 1 -- 172.21.15.42:0/2052951202 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6489+0+0 (secure 0 0 0) 0x7f34e404aa90 con 0x7f34e811bbe0 2019-12-05T05:12:33.178 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:33.160+0000 7f34ecb8d700 1 --2- 172.21.15.42:0/2052951202 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f34c803b2d0 0x7f34c803d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:12:33.178 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:33.160+0000 7f34ecb8d700 1 --2- 172.21.15.42:0/2052951202 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f34c803b2d0 0x7f34c803d7e0 crc :-1 s=READY pgs=209 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:12:33.179 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:33.164+0000 7f34eedf1700 1 -- 172.21.15.42:0/2052951202 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34e805d110 con 0x7f34e811bbe0 2019-12-05T05:12:33.179 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:33.172+0000 7f34ed38e700 1 --2- 172.21.15.42:0/2052951202 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f34e811bbe0 0x7f34e807c7b0 secure :-1 s=THROTTLE_DONE pgs=92 cs=0 l=1 rx=0x7f34e0001e20 tx=0x7f34e0001930).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:33.180 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:33.172+0000 7f34ddffb700 1 -- 172.21.15.42:0/2052951202 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f34e4031ad0 con 0x7f34e811bbe0 2019-12-05T05:12:33.345 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:33.340+0000 7f34eedf1700 1 -- 172.21.15.42:0/2052951202 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f34e805b8c0 con 0x7f34e811bbe0 2019-12-05T05:12:33.346 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:33.340+0000 7f34ed38e700 1 --2- 172.21.15.42:0/2052951202 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f34e811bbe0 0x7f34e807c7b0 secure :-1 s=THROTTLE_DONE pgs=92 cs=0 l=1 rx=0x7f34e0001e20 tx=0x7f34e0001930).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:33.346 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:33.340+0000 7f34ddffb700 1 -- 172.21.15.42:0/2052951202 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) v1 ==== 94+0+2105 (secure 0 0 0) 0x7f34e400cd90 con 0x7f34e811bbe0 2019-12-05T05:12:33.348 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:12:33.349 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":58,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5391,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/814397242","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":814397242},{"type":"v1","addr":"172.21.15.40:6833","nonce":814397242}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":58}],"filesystems":[{"mdsmap":{"epoch":57,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:11:24.854205+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5215},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5215":{"gid":5215,"name":"a","rank":0,"incarnation":52,"state":"up:active","state_seq":20,"addr":"172.21.15.42:6835/3176146895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3176146895},{"type":"v1","addr":"172.21.15.42:6835","nonce":3176146895}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:12:33.349 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 58 2019-12-05T05:12:33.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:33.344+0000 7f34eedf1700 1 -- 172.21.15.42:0/2052951202 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f34c803b2d0 msgr2=0x7f34c803d7e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:12:33.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:33.344+0000 7f34eedf1700 1 --2- 172.21.15.42:0/2052951202 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f34c803b2d0 0x7f34c803d7e0 crc :-1 s=READY pgs=209 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:12:33.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:33.344+0000 7f34eedf1700 1 -- 172.21.15.42:0/2052951202 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f34e811bbe0 msgr2=0x7f34e807c7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:12:33.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:33.344+0000 7f34eedf1700 1 --2- 172.21.15.42:0/2052951202 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f34e811bbe0 0x7f34e807c7b0 secure :-1 s=READY pgs=92 cs=0 l=1 rx=0x7f34e0001e20 tx=0x7f34e0001930).stop 2019-12-05T05:12:33.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:33.344+0000 7f34eedf1700 1 -- 172.21.15.42:0/2052951202 shutdown_connections 2019-12-05T05:12:33.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:33.344+0000 7f34eedf1700 1 --2- 172.21.15.42:0/2052951202 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f34c803b2d0 0x7f34c803d7e0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:12:33.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:33.344+0000 7f34eedf1700 1 --2- 172.21.15.42:0/2052951202 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f34e811bbe0 0x7f34e807c7b0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:12:33.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:33.344+0000 7f34eedf1700 1 --2- 172.21.15.42:0/2052951202 >> v2:172.21.15.40:3300/0 conn(0x7f34e8143ba0 0x7f34e807cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:12:33.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:33.344+0000 7f34eedf1700 1 -- 172.21.15.42:0/2052951202 shutdown_connections 2019-12-05T05:12:33.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:33.344+0000 7f34eedf1700 1 -- 172.21.15.42:0/2052951202 wait complete. 2019-12-05T05:12:33.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:33.344+0000 7f34eedf1700 1 -- 172.21.15.42:0/2052951202 >> 172.21.15.42:0/2052951202 conn(0x7f34e807e9a0 msgr2=0x7f34e8059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:12:33.362 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T05:12:35.353 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:12:35.357 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:12:35.368 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:12:35.541 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:35.528+0000 7f2258b47700 1 Processor -- start 2019-12-05T05:12:35.542 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:35.528+0000 7f2258b47700 1 -- start start 2019-12-05T05:12:35.542 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:35.528+0000 7f2258b47700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f22540a7740 0x7f225408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:12:35.542 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:35.528+0000 7f2258b47700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22540a0710 con 0x7f225407e720 2019-12-05T05:12:35.542 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:35.528+0000 7f2258b47700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22540a9ba0 con 0x7f225409fae0 2019-12-05T05:12:35.542 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:35.528+0000 7f2258b47700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22540a0b00 con 0x7f22540a7740 2019-12-05T05:12:35.542 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:35.528+0000 7f225359e700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f225409fae0 0x7f225414efe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:57902/0 (socket says 172.21.15.42:57902) 2019-12-05T05:12:35.543 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:35.528+0000 7f225359e700 1 -- 172.21.15.42:0/1063515710 learned_addr learned my addr 172.21.15.42:0/1063515710 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:12:35.543 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:35.532+0000 7f2252d9d700 1 --2- 172.21.15.42:0/1063515710 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f22540a7740 0x7f225408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:12:35.543 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:35.532+0000 7f2251d9b700 1 -- 172.21.15.42:0/1063515710 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 506781081 0 0) 0x7f22540a9ba0 con 0x7f225409fae0 2019-12-05T05:12:35.543 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:35.532+0000 7f2251d9b700 1 -- 172.21.15.42:0/1063515710 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f223c0030b0 con 0x7f225409fae0 2019-12-05T05:12:35.543 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:35.532+0000 7f2251d9b700 1 -- 172.21.15.42:0/1063515710 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 648119957 0 0) 0x7f22540a0710 con 0x7f225407e720 2019-12-05T05:12:35.543 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:35.532+0000 7f2251d9b700 1 -- 172.21.15.42:0/1063515710 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22540a9ba0 con 0x7f225407e720 2019-12-05T05:12:35.543 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:35.532+0000 7f2251d9b700 1 -- 172.21.15.42:0/1063515710 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1499221053 0 0) 0x7f223c0030b0 con 0x7f225409fae0 2019-12-05T05:12:35.543 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:35.532+0000 7f2251d9b700 1 -- 172.21.15.42:0/1063515710 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f22540a7740 msgr2=0x7f225408df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:12:35.544 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:35.532+0000 7f2251d9b700 1 --2- 172.21.15.42:0/1063515710 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f22540a7740 0x7f225408df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:12:35.544 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:35.532+0000 7f2251d9b700 1 -- 172.21.15.42:0/1063515710 >> v1:172.21.15.40:6789/0 conn(0x7f225407e720 legacy=0x7f225409f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:12:35.544 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:35.532+0000 7f2251d9b700 1 -- 172.21.15.42:0/1063515710 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f225407eb50 con 0x7f225409fae0 2019-12-05T05:12:35.544 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:35.532+0000 7f2251d9b700 1 -- 172.21.15.42:0/1063515710 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7f224c001ad0 con 0x7f225409fae0 2019-12-05T05:12:35.544 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:35.532+0000 7f2252d9d700 1 --2- 172.21.15.42:0/1063515710 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f22540a7740 0x7f225408df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:12:35.544 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:35.532+0000 7f2251d9b700 1 -- 172.21.15.42:0/1063515710 >> v1:172.21.15.42:6789/0 conn(0x7f225409fae0 legacy=0x7f225414efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:12:35.544 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:35.532+0000 7f2251d9b700 1 --2- 172.21.15.42:0/1063515710 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f223c003940 0x7f223c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:12:35.545 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:35.532+0000 7f2251d9b700 1 --2- 172.21.15.42:0/1063515710 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f223c004470 0x7f223c004920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:12:35.545 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:35.532+0000 7f2251d9b700 1 --2- 172.21.15.42:0/1063515710 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f223c04d670 0x7f223c04fab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:12:35.545 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:35.532+0000 7f2251d9b700 1 -- 172.21.15.42:0/1063515710 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22540a0b00 con 0x7f223c003940 2019-12-05T05:12:35.545 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:35.532+0000 7f2251d9b700 1 -- 172.21.15.42:0/1063515710 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f223c005130 con 0x7f223c004470 2019-12-05T05:12:35.545 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:35.532+0000 7f2251d9b700 1 -- 172.21.15.42:0/1063515710 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f223c050000 con 0x7f223c04d670 2019-12-05T05:12:35.545 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:35.532+0000 7f2252d9d700 1 --2- 172.21.15.42:0/1063515710 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f223c003940 0x7f223c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:12:35.545 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:35.532+0000 7f225259c700 1 --2- 172.21.15.42:0/1063515710 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f223c004470 0x7f223c004920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:12:35.546 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:35.532+0000 7f225359e700 1 --2- 172.21.15.42:0/1063515710 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f223c04d670 0x7f223c04fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:12:35.546 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:35.532+0000 7f225259c700 1 -- 172.21.15.42:0/1063515710 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f223c04d670 msgr2=0x7f223c04fab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:12:35.546 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:35.532+0000 7f225259c700 1 --2- 172.21.15.42:0/1063515710 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f223c04d670 0x7f223c04fab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:12:35.546 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:35.532+0000 7f225259c700 1 -- 172.21.15.42:0/1063515710 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f223c003940 msgr2=0x7f223c003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:12:35.546 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:35.532+0000 7f225259c700 1 --2- 172.21.15.42:0/1063515710 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f223c003940 0x7f223c003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:12:35.546 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:35.532+0000 7f225259c700 1 -- 172.21.15.42:0/1063515710 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f223c050280 con 0x7f223c004470 2019-12-05T05:12:35.546 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:35.532+0000 7f2252d9d700 1 --2- 172.21.15.42:0/1063515710 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f223c003940 0x7f223c003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:12:35.547 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:35.532+0000 7f225259c700 1 --2- 172.21.15.42:0/1063515710 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f223c004470 0x7f223c004920 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f2248000fe0 tx=0x7f224800be70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:35.547 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:35.532+0000 7f225259c700 1 --2- 172.21.15.42:0/1063515710 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f223c004470 0x7f223c004920 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f2248000fe0 tx=0x7f224800be70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:35.547 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:35.532+0000 7f225259c700 1 --2- 172.21.15.42:0/1063515710 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f223c004470 0x7f223c004920 secure :-1 s=READY pgs=461 cs=0 l=1 rx=0x7f2248000fe0 tx=0x7f224800be70).ready entity=mon.1 client_cookie=a7f9c1afdc5184b8 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:12:35.547 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:35.532+0000 7f225259c700 1 --2- 172.21.15.42:0/1063515710 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f223c004470 0x7f223c004920 secure :-1 s=THROTTLE_DONE pgs=461 cs=0 l=1 rx=0x7f2248000fe0 tx=0x7f224800be70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:35.547 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:35.532+0000 7f225259c700 1 --2- 172.21.15.42:0/1063515710 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f223c004470 0x7f223c004920 secure :-1 s=THROTTLE_DONE pgs=461 cs=0 l=1 rx=0x7f2248000fe0 tx=0x7f224800be70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:35.547 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:35.532+0000 7f2251d9b700 1 -- 172.21.15.42:0/1063515710 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2248010be0 con 0x7f223c004470 2019-12-05T05:12:35.547 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:35.532+0000 7f225259c700 1 --2- 172.21.15.42:0/1063515710 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f223c004470 0x7f223c004920 secure :-1 s=THROTTLE_DONE pgs=461 cs=0 l=1 rx=0x7f2248000fe0 tx=0x7f224800be70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:35.548 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:35.532+0000 7f2251d9b700 1 -- 172.21.15.42:0/1063515710 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2248010800 con 0x7f223c004470 2019-12-05T05:12:35.548 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:35.532+0000 7f2251d9b700 1 -- 172.21.15.42:0/1063515710 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f224800c5b0 con 0x7f223c004470 2019-12-05T05:12:35.548 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:35.532+0000 7f2258b47700 1 -- 172.21.15.42:0/1063515710 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f223c004470 msgr2=0x7f223c004920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:12:35.548 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:35.532+0000 7f2258b47700 1 --2- 172.21.15.42:0/1063515710 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f223c004470 0x7f223c004920 secure :-1 s=READY pgs=461 cs=0 l=1 rx=0x7f2248000fe0 tx=0x7f224800be70).stop 2019-12-05T05:12:35.548 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:35.532+0000 7f225359e700 1 -- 172.21.15.42:0/1063515710 reap_dead start 2019-12-05T05:12:35.548 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:35.532+0000 7f2258b47700 1 -- 172.21.15.42:0/1063515710 shutdown_connections 2019-12-05T05:12:35.548 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:35.536+0000 7f2258b47700 1 -- 172.21.15.42:0/1063515710 shutdown_connections 2019-12-05T05:12:35.549 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:35.536+0000 7f2258b47700 1 -- 172.21.15.42:0/1063515710 wait complete. 2019-12-05T05:12:35.549 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:35.536+0000 7f2258b47700 1 -- 172.21.15.42:0/1063515710 >> 172.21.15.42:0/1063515710 conn(0x7f22540aabe0 msgr2=0x7f22540a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:12:35.552 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:35.536+0000 7f2258b47700 1 Processor -- start 2019-12-05T05:12:35.552 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:35.536+0000 7f2258b47700 1 -- start start 2019-12-05T05:12:35.553 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:35.540+0000 7f2258b47700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2254116500 0x7f22540a3200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:12:35.553 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:35.540+0000 7f2258b47700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f22540a3910 0x7f22540a3d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:12:35.553 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:35.540+0000 7f2258b47700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f225408eb70 0x7f225408efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:12:35.553 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:35.540+0000 7f2258b47700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22540a1970 con 0x7f225408eb70 2019-12-05T05:12:35.553 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:35.540+0000 7f2258b47700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22540a1c10 con 0x7f22540a3910 2019-12-05T05:12:35.553 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:35.540+0000 7f2258b47700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22540a1eb0 con 0x7f2254116500 2019-12-05T05:12:35.554 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:35.540+0000 7f225259c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f22540a3910 0x7f22540a3d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:12:35.554 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:35.540+0000 7f2252d9d700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2254116500 0x7f22540a3200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:12:35.554 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:35.540+0000 7f225259c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f22540a3910 0x7f22540a3d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:57874/0 (socket says 172.21.15.42:57874) 2019-12-05T05:12:35.554 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:35.540+0000 7f2252d9d700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2254116500 0x7f22540a3200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:58338/0 (socket says 172.21.15.42:58338) 2019-12-05T05:12:35.554 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:35.540+0000 7f225259c700 1 -- 172.21.15.42:0/3641288824 learned_addr learned my addr 172.21.15.42:0/3641288824 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:12:35.554 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:35.540+0000 7f225359e700 1 --2- 172.21.15.42:0/3641288824 >> v2:172.21.15.40:3300/0 conn(0x7f225408eb70 0x7f225408efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:12:35.554 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:35.540+0000 7f2252d9d700 1 -- 172.21.15.42:0/3641288824 >> v2:172.21.15.42:3300/0 conn(0x7f22540a3910 msgr2=0x7f22540a3d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:12:35.554 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:35.540+0000 7f2252d9d700 1 --2- 172.21.15.42:0/3641288824 >> v2:172.21.15.42:3300/0 conn(0x7f22540a3910 0x7f22540a3d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:12:35.555 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:35.540+0000 7f2252d9d700 1 -- 172.21.15.42:0/3641288824 >> v2:172.21.15.40:3300/0 conn(0x7f225408eb70 msgr2=0x7f225408efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:12:35.555 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:35.540+0000 7f2252d9d700 1 --2- 172.21.15.42:0/3641288824 >> v2:172.21.15.40:3300/0 conn(0x7f225408eb70 0x7f225408efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:12:35.555 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:35.540+0000 7f2252d9d700 1 -- 172.21.15.42:0/3641288824 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22540a4290 con 0x7f2254116500 2019-12-05T05:12:35.555 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:35.540+0000 7f2252d9d700 1 --2- 172.21.15.42:0/3641288824 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2254116500 0x7f22540a3200 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f224400b200 tx=0x7f2244010d00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:35.555 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:35.540+0000 7f2252d9d700 1 --2- 172.21.15.42:0/3641288824 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2254116500 0x7f22540a3200 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f224400b200 tx=0x7f2244010d00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:35.555 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:35.540+0000 7f2252d9d700 1 --2- 172.21.15.42:0/3641288824 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2254116500 0x7f22540a3200 secure :-1 s=READY pgs=93 cs=0 l=1 rx=0x7f224400b200 tx=0x7f2244010d00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:12:35.555 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:35.540+0000 7f2252d9d700 1 --2- 172.21.15.42:0/3641288824 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2254116500 0x7f22540a3200 secure :-1 s=THROTTLE_DONE pgs=93 cs=0 l=1 rx=0x7f224400b200 tx=0x7f2244010d00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:35.556 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:35.540+0000 7f22437fe700 1 -- 172.21.15.42:0/3641288824 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2244005930 con 0x7f2254116500 2019-12-05T05:12:35.556 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:35.540+0000 7f2252d9d700 1 --2- 172.21.15.42:0/3641288824 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2254116500 0x7f22540a3200 secure :-1 s=THROTTLE_DONE pgs=93 cs=0 l=1 rx=0x7f224400b200 tx=0x7f2244010d00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:35.556 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:35.540+0000 7f22437fe700 1 -- 172.21.15.42:0/3641288824 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2244007160 con 0x7f2254116500 2019-12-05T05:12:35.556 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:35.540+0000 7f2252d9d700 1 --2- 172.21.15.42:0/3641288824 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2254116500 0x7f22540a3200 secure :-1 s=THROTTLE_DONE pgs=93 cs=0 l=1 rx=0x7f224400b200 tx=0x7f2244010d00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:35.556 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:35.540+0000 7f2258b47700 1 -- 172.21.15.42:0/3641288824 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22540a14d0 con 0x7f2254116500 2019-12-05T05:12:35.556 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:35.540+0000 7f22437fe700 1 -- 172.21.15.42:0/3641288824 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f224400e390 con 0x7f2254116500 2019-12-05T05:12:35.557 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:35.540+0000 7f2258b47700 1 -- 172.21.15.42:0/3641288824 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f225408f710 con 0x7f2254116500 2019-12-05T05:12:35.557 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:35.544+0000 7f2252d9d700 1 --2- 172.21.15.42:0/3641288824 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2254116500 0x7f22540a3200 secure :-1 s=THROTTLE_DONE pgs=93 cs=0 l=1 rx=0x7f224400b200 tx=0x7f2244010d00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:35.557 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:35.544+0000 7f22437fe700 1 -- 172.21.15.42:0/3641288824 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f224400e670 con 0x7f2254116500 2019-12-05T05:12:35.557 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:35.544+0000 7f2252d9d700 1 --2- 172.21.15.42:0/3641288824 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2254116500 0x7f22540a3200 secure :-1 s=THROTTLE_DONE pgs=93 cs=0 l=1 rx=0x7f224400b200 tx=0x7f2244010d00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:35.557 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:35.544+0000 7f2258b47700 1 -- 172.21.15.42:0/3641288824 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f225405d110 con 0x7f2254116500 2019-12-05T05:12:35.557 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:35.544+0000 7f22437fe700 1 --2- 172.21.15.42:0/3641288824 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f223803acf0 0x7f223803d200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:12:35.557 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:35.544+0000 7f22437fe700 1 -- 172.21.15.42:0/3641288824 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6489+0+0 (secure 0 0 0) 0x7f224404b900 con 0x7f2254116500 2019-12-05T05:12:35.557 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:35.544+0000 7f225259c700 1 --2- 172.21.15.42:0/3641288824 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f223803acf0 0x7f223803d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:12:35.558 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:35.544+0000 7f225259c700 1 --2- 172.21.15.42:0/3641288824 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f223803acf0 0x7f223803d200 crc :-1 s=READY pgs=210 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:12:35.558 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:35.552+0000 7f2252d9d700 1 --2- 172.21.15.42:0/3641288824 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2254116500 0x7f22540a3200 secure :-1 s=THROTTLE_DONE pgs=93 cs=0 l=1 rx=0x7f224400b200 tx=0x7f2244010d00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:35.559 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:35.552+0000 7f22437fe700 1 -- 172.21.15.42:0/3641288824 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f2244032960 con 0x7f2254116500 2019-12-05T05:12:35.728 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:35.724+0000 7f2258b47700 1 -- 172.21.15.42:0/3641288824 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f225405b8c0 con 0x7f2254116500 2019-12-05T05:12:35.729 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:35.724+0000 7f2252d9d700 1 --2- 172.21.15.42:0/3641288824 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2254116500 0x7f22540a3200 secure :-1 s=THROTTLE_DONE pgs=93 cs=0 l=1 rx=0x7f224400b200 tx=0x7f2244010d00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:35.729 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:35.724+0000 7f22437fe700 1 -- 172.21.15.42:0/3641288824 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) v1 ==== 94+0+2105 (secure 0 0 0) 0x7f2244032bc0 con 0x7f2254116500 2019-12-05T05:12:35.731 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:12:35.732 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":58,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5391,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/814397242","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":814397242},{"type":"v1","addr":"172.21.15.40:6833","nonce":814397242}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":58}],"filesystems":[{"mdsmap":{"epoch":57,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:11:24.854205+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5215},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5215":{"gid":5215,"name":"a","rank":0,"incarnation":52,"state":"up:active","state_seq":20,"addr":"172.21.15.42:6835/3176146895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3176146895},{"type":"v1","addr":"172.21.15.42:6835","nonce":3176146895}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:12:35.732 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 58 2019-12-05T05:12:35.734 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:35.728+0000 7f2258b47700 1 -- 172.21.15.42:0/3641288824 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f223803acf0 msgr2=0x7f223803d200 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:12:35.734 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:35.728+0000 7f2258b47700 1 --2- 172.21.15.42:0/3641288824 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f223803acf0 0x7f223803d200 crc :-1 s=READY pgs=210 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:12:35.735 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:35.728+0000 7f2258b47700 1 -- 172.21.15.42:0/3641288824 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2254116500 msgr2=0x7f22540a3200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:12:35.735 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:35.728+0000 7f2258b47700 1 --2- 172.21.15.42:0/3641288824 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2254116500 0x7f22540a3200 secure :-1 s=READY pgs=93 cs=0 l=1 rx=0x7f224400b200 tx=0x7f2244010d00).stop 2019-12-05T05:12:35.735 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:35.728+0000 7f2258b47700 1 -- 172.21.15.42:0/3641288824 shutdown_connections 2019-12-05T05:12:35.735 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:35.728+0000 7f2258b47700 1 --2- 172.21.15.42:0/3641288824 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f223803acf0 0x7f223803d200 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:12:35.735 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:35.728+0000 7f2258b47700 1 --2- 172.21.15.42:0/3641288824 >> v2:172.21.15.40:3300/0 conn(0x7f225408eb70 0x7f225408efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:12:35.735 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:35.728+0000 7f2258b47700 1 --2- 172.21.15.42:0/3641288824 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2254116500 0x7f22540a3200 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:12:35.735 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:35.728+0000 7f2258b47700 1 --2- 172.21.15.42:0/3641288824 >> v2:172.21.15.42:3300/0 conn(0x7f22540a3910 0x7f22540a3d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:12:35.736 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:35.728+0000 7f2258b47700 1 -- 172.21.15.42:0/3641288824 shutdown_connections 2019-12-05T05:12:35.736 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:35.728+0000 7f2258b47700 1 -- 172.21.15.42:0/3641288824 wait complete. 2019-12-05T05:12:35.736 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:35.728+0000 7f2258b47700 1 -- 172.21.15.42:0/3641288824 >> 172.21.15.42:0/3641288824 conn(0x7f22540aabe0 msgr2=0x7f22540a6a60 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:12:35.744 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T05:12:37.745 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:12:37.857 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:37.840+0000 7f6e4d3a6700 1 Processor -- start 2019-12-05T05:12:37.857 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:37.840+0000 7f6e4d3a6700 1 -- start start 2019-12-05T05:12:37.858 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:37.840+0000 7f6e4d3a6700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6e480a7740 0x7f6e4808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:12:37.858 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:37.840+0000 7f6e4d3a6700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f6e4807e720 0x7f6e4807eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:12:37.858 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:37.840+0000 7f6e4d3a6700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e480a0710 con 0x7f6e4809f2e0 2019-12-05T05:12:37.858 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:37.840+0000 7f6e4d3a6700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e480a0b00 con 0x7f6e4807e720 2019-12-05T05:12:37.858 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:37.840+0000 7f6e4d3a6700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e4808e5b0 con 0x7f6e480a7740 2019-12-05T05:12:37.858 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:37.840+0000 7f6e47fff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f6e4809f2e0 0x7f6e4809f710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:57920/0 (socket says 172.21.15.42:57920) 2019-12-05T05:12:37.859 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:37.840+0000 7f6e47fff700 1 -- 172.21.15.42:0/565431797 learned_addr learned my addr 172.21.15.42:0/565431797 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:12:37.859 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:37.840+0000 7f6e477fe700 1 --2- 172.21.15.42:0/565431797 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6e480a7740 0x7f6e4808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:12:37.869 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:37.840+0000 7f6e46ffd700 1 --2- 172.21.15.42:0/565431797 >> v2:172.21.15.40:3300/0 conn(0x7f6e4807e720 0x7f6e4807eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:12:37.869 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:37.840+0000 7f6e467fc700 1 -- 172.21.15.42:0/565431797 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 336969678 0 0) 0x7f6e480a0710 con 0x7f6e4809f2e0 2019-12-05T05:12:37.870 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:37.840+0000 7f6e467fc700 1 -- 172.21.15.42:0/565431797 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e340030b0 con 0x7f6e4809f2e0 2019-12-05T05:12:37.870 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:37.840+0000 7f6e477fe700 1 -- 172.21.15.42:0/565431797 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6e480a7740 msgr2=0x7f6e4808df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2019-12-05T05:12:37.870 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:37.840+0000 7f6e477fe700 1 -- 172.21.15.42:0/565431797 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6e480a7740 msgr2=0x7f6e4808df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2019-12-05T05:12:37.870 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:37.840+0000 7f6e477fe700 1 --2- 172.21.15.42:0/565431797 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6e480a7740 0x7f6e4808df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-12-05T05:12:37.871 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:37.840+0000 7f6e477fe700 1 --2- 172.21.15.42:0/565431797 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6e480a7740 0x7f6e4808df80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rx=0 tx=0)._fault waiting 0.200000 2019-12-05T05:12:37.871 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:37.840+0000 7f6e467fc700 1 -- 172.21.15.42:0/565431797 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2697893255 0 0) 0x7f6e340030b0 con 0x7f6e4809f2e0 2019-12-05T05:12:37.871 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:37.840+0000 7f6e467fc700 1 -- 172.21.15.42:0/565431797 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6e480a7740 msgr2=0x7f6e4808df80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2019-12-05T05:12:37.871 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:37.840+0000 7f6e467fc700 1 --2- 172.21.15.42:0/565431797 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6e480a7740 0x7f6e4808df80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:12:37.871 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:37.840+0000 7f6e467fc700 1 -- 172.21.15.42:0/565431797 >> v2:172.21.15.40:3300/0 conn(0x7f6e4807e720 msgr2=0x7f6e4807eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:12:37.872 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:37.840+0000 7f6e467fc700 1 --2- 172.21.15.42:0/565431797 >> v2:172.21.15.40:3300/0 conn(0x7f6e4807e720 0x7f6e4807eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:12:37.872 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:37.840+0000 7f6e467fc700 1 -- 172.21.15.42:0/565431797 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e4807f0a0 con 0x7f6e4809f2e0 2019-12-05T05:12:37.872 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:37.840+0000 7f6e467fc700 1 -- 172.21.15.42:0/565431797 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7f6e40001750 con 0x7f6e4809f2e0 2019-12-05T05:12:37.872 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:37.840+0000 7f6e467fc700 1 -- 172.21.15.42:0/565431797 >> v1:172.21.15.42:6789/0 conn(0x7f6e4809f2e0 legacy=0x7f6e4809f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:12:37.872 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:37.840+0000 7f6e467fc700 1 --2- 172.21.15.42:0/565431797 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6e34003760 0x7f6e34003b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:12:37.872 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:37.840+0000 7f6e467fc700 1 --2- 172.21.15.42:0/565431797 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6e34004420 0x7f6e34004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:12:37.872 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:37.840+0000 7f6e467fc700 1 --2- 172.21.15.42:0/565431797 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6e34004da0 0x7f6e3404f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:12:37.873 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:37.840+0000 7f6e467fc700 1 -- 172.21.15.42:0/565431797 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e480a0b00 con 0x7f6e34004420 2019-12-05T05:12:37.873 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:37.840+0000 7f6e467fc700 1 -- 172.21.15.42:0/565431797 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e4808e5b0 con 0x7f6e34004da0 2019-12-05T05:12:37.873 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:37.840+0000 7f6e467fc700 1 -- 172.21.15.42:0/565431797 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e3404fbd0 con 0x7f6e34003760 2019-12-05T05:12:37.873 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:37.844+0000 7f6e47fff700 1 --2- 172.21.15.42:0/565431797 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6e34004da0 0x7f6e3404f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:12:37.873 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:37.844+0000 7f6e46ffd700 1 --2- 172.21.15.42:0/565431797 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6e34004420 0x7f6e34004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:12:37.873 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:37.844+0000 7f6e477fe700 1 --2- 172.21.15.42:0/565431797 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6e34003760 0x7f6e34003b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:12:37.874 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:37.844+0000 7f6e47fff700 1 -- 172.21.15.42:0/565431797 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6e34003760 msgr2=0x7f6e34003b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:12:37.874 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:37.844+0000 7f6e47fff700 1 --2- 172.21.15.42:0/565431797 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6e34003760 0x7f6e34003b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:12:37.874 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:37.844+0000 7f6e47fff700 1 -- 172.21.15.42:0/565431797 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6e34004420 msgr2=0x7f6e34004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:12:37.874 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:37.844+0000 7f6e47fff700 1 --2- 172.21.15.42:0/565431797 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6e34004420 0x7f6e34004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:12:37.874 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:37.844+0000 7f6e47fff700 1 -- 172.21.15.42:0/565431797 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e3404fe50 con 0x7f6e34004da0 2019-12-05T05:12:37.874 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:37.844+0000 7f6e46ffd700 1 --2- 172.21.15.42:0/565431797 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6e34004420 0x7f6e34004850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:12:37.874 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:37.844+0000 7f6e47fff700 1 --2- 172.21.15.42:0/565431797 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6e34004da0 0x7f6e3404f680 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f6e40001000 tx=0x7f6e40010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:37.875 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:37.844+0000 7f6e47fff700 1 --2- 172.21.15.42:0/565431797 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6e34004da0 0x7f6e3404f680 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f6e40001000 tx=0x7f6e40010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:37.875 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:37.844+0000 7f6e47fff700 1 --2- 172.21.15.42:0/565431797 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6e34004da0 0x7f6e3404f680 secure :-1 s=READY pgs=463 cs=0 l=1 rx=0x7f6e40001000 tx=0x7f6e40010b50).ready entity=mon.1 client_cookie=2b0d920be3a90aee server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:12:37.875 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:37.844+0000 7f6e47fff700 1 --2- 172.21.15.42:0/565431797 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6e34004da0 0x7f6e3404f680 secure :-1 s=THROTTLE_DONE pgs=463 cs=0 l=1 rx=0x7f6e40001000 tx=0x7f6e40010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:37.875 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:37.844+0000 7f6e467fc700 1 -- 172.21.15.42:0/565431797 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6e4000e0c0 con 0x7f6e34004da0 2019-12-05T05:12:37.875 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:37.844+0000 7f6e47fff700 1 --2- 172.21.15.42:0/565431797 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6e34004da0 0x7f6e3404f680 secure :-1 s=THROTTLE_DONE pgs=463 cs=0 l=1 rx=0x7f6e40001000 tx=0x7f6e40010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:37.875 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:37.844+0000 7f6e47fff700 1 --2- 172.21.15.42:0/565431797 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6e34004da0 0x7f6e3404f680 secure :-1 s=THROTTLE_DONE pgs=463 cs=0 l=1 rx=0x7f6e40001000 tx=0x7f6e40010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:37.875 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:37.844+0000 7f6e467fc700 1 -- 172.21.15.42:0/565431797 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e40011460 con 0x7f6e34004da0 2019-12-05T05:12:37.876 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:37.844+0000 7f6e467fc700 1 -- 172.21.15.42:0/565431797 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6e4000e6a0 con 0x7f6e34004da0 2019-12-05T05:12:37.876 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:37.844+0000 7f6e4d3a6700 1 -- 172.21.15.42:0/565431797 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6e34004da0 msgr2=0x7f6e3404f680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:12:37.876 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:37.844+0000 7f6e4d3a6700 1 --2- 172.21.15.42:0/565431797 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6e34004da0 0x7f6e3404f680 secure :-1 s=READY pgs=463 cs=0 l=1 rx=0x7f6e40001000 tx=0x7f6e40010b50).stop 2019-12-05T05:12:37.876 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:37.844+0000 7f6e47fff700 1 -- 172.21.15.42:0/565431797 reap_dead start 2019-12-05T05:12:37.876 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:37.844+0000 7f6e4d3a6700 1 -- 172.21.15.42:0/565431797 shutdown_connections 2019-12-05T05:12:37.876 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:37.844+0000 7f6e4d3a6700 1 -- 172.21.15.42:0/565431797 shutdown_connections 2019-12-05T05:12:37.876 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:37.844+0000 7f6e4d3a6700 1 -- 172.21.15.42:0/565431797 wait complete. 2019-12-05T05:12:37.877 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:37.844+0000 7f6e4d3a6700 1 -- 172.21.15.42:0/565431797 >> 172.21.15.42:0/565431797 conn(0x7f6e480aabe0 msgr2=0x7f6e480a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:12:37.877 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:37.848+0000 7f6e4d3a6700 1 Processor -- start 2019-12-05T05:12:37.877 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:37.848+0000 7f6e4d3a6700 1 -- start start 2019-12-05T05:12:37.877 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:37.848+0000 7f6e4d3a6700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6e48122950 0x7f6e480a6290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:12:37.877 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:37.848+0000 7f6e4d3a6700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6e4811b6c0 0x7f6e480a3720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:12:37.877 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:37.848+0000 7f6e4d3a6700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e4807d460 con 0x7f6e4808eb70 2019-12-05T05:12:37.877 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:37.848+0000 7f6e4d3a6700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e480a6b30 con 0x7f6e4811b6c0 2019-12-05T05:12:37.878 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:37.848+0000 7f6e4d3a6700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e48122410 con 0x7f6e48122950 2019-12-05T05:12:37.878 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:37.848+0000 7f6e46ffd700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6e4811b6c0 0x7f6e480a3720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:12:37.878 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:37.848+0000 7f6e46ffd700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6e4811b6c0 0x7f6e480a3720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:57894/0 (socket says 172.21.15.42:57894) 2019-12-05T05:12:37.878 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:37.848+0000 7f6e46ffd700 1 -- 172.21.15.42:0/2997664479 learned_addr learned my addr 172.21.15.42:0/2997664479 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:12:37.878 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:37.848+0000 7f6e477fe700 1 --2- 172.21.15.42:0/2997664479 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6e48122950 0x7f6e480a6290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:12:37.878 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:37.848+0000 7f6e3ffff700 1 -- 172.21.15.42:0/2997664479 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3435717120 0 0) 0x7f6e4807d460 con 0x7f6e4808eb70 2019-12-05T05:12:37.878 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:37.852+0000 7f6e3ffff700 1 -- 172.21.15.42:0/2997664479 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e2c0030b0 con 0x7f6e4808eb70 2019-12-05T05:12:37.879 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:37.852+0000 7f6e46ffd700 1 -- 172.21.15.42:0/2997664479 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6e48122950 msgr2=0x7f6e480a6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:12:37.879 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:37.852+0000 7f6e46ffd700 1 --2- 172.21.15.42:0/2997664479 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6e48122950 0x7f6e480a6290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:12:37.879 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:37.852+0000 7f6e46ffd700 1 -- 172.21.15.42:0/2997664479 >> v1:172.21.15.40:6789/0 conn(0x7f6e4808eb70 legacy=0x7f6e4808efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:12:37.879 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:37.852+0000 7f6e46ffd700 1 -- 172.21.15.42:0/2997664479 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e480a67e0 con 0x7f6e4811b6c0 2019-12-05T05:12:37.879 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:37.852+0000 7f6e477fe700 1 --2- 172.21.15.42:0/2997664479 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6e48122950 0x7f6e480a6290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:12:37.879 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:37.852+0000 7f6e46ffd700 1 --2- 172.21.15.42:0/2997664479 >> v2:172.21.15.42:3300/0 conn(0x7f6e4811b6c0 0x7f6e480a3720 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f6e4807d690 tx=0x7f6e3000cb30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:37.879 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:37.852+0000 7f6e46ffd700 1 --2- 172.21.15.42:0/2997664479 >> v2:172.21.15.42:3300/0 conn(0x7f6e4811b6c0 0x7f6e480a3720 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f6e4807d690 tx=0x7f6e3000cb30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:37.880 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:37.852+0000 7f6e46ffd700 1 --2- 172.21.15.42:0/2997664479 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6e4811b6c0 0x7f6e480a3720 secure :-1 s=READY pgs=464 cs=0 l=1 rx=0x7f6e4807d690 tx=0x7f6e3000cb30).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:12:37.880 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:37.852+0000 7f6e46ffd700 1 --2- 172.21.15.42:0/2997664479 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6e4811b6c0 0x7f6e480a3720 secure :-1 s=THROTTLE_DONE pgs=464 cs=0 l=1 rx=0x7f6e4807d690 tx=0x7f6e3000cb30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:37.885 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:37.852+0000 7f6e3ffff700 1 -- 172.21.15.42:0/2997664479 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6e30008d60 con 0x7f6e4811b6c0 2019-12-05T05:12:37.885 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:37.852+0000 7f6e46ffd700 1 --2- 172.21.15.42:0/2997664479 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6e4811b6c0 0x7f6e480a3720 secure :-1 s=THROTTLE_DONE pgs=464 cs=0 l=1 rx=0x7f6e4807d690 tx=0x7f6e3000cb30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:37.885 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:37.852+0000 7f6e3ffff700 1 -- 172.21.15.42:0/2997664479 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e30005cb0 con 0x7f6e4811b6c0 2019-12-05T05:12:37.885 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:37.852+0000 7f6e46ffd700 1 --2- 172.21.15.42:0/2997664479 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6e4811b6c0 0x7f6e480a3720 secure :-1 s=THROTTLE_DONE pgs=464 cs=0 l=1 rx=0x7f6e4807d690 tx=0x7f6e3000cb30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:37.885 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:37.852+0000 7f6e3ffff700 1 -- 172.21.15.42:0/2997664479 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6e30007540 con 0x7f6e4811b6c0 2019-12-05T05:12:37.885 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:37.852+0000 7f6e4d3a6700 1 -- 172.21.15.42:0/2997664479 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e480a3c70 con 0x7f6e4811b6c0 2019-12-05T05:12:37.886 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:37.852+0000 7f6e4d3a6700 1 -- 172.21.15.42:0/2997664479 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e4808f7a0 con 0x7f6e4811b6c0 2019-12-05T05:12:37.887 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:37.852+0000 7f6e46ffd700 1 --2- 172.21.15.42:0/2997664479 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6e4811b6c0 0x7f6e480a3720 secure :-1 s=THROTTLE_DONE pgs=464 cs=0 l=1 rx=0x7f6e4807d690 tx=0x7f6e3000cb30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:37.887 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:37.852+0000 7f6e3ffff700 1 -- 172.21.15.42:0/2997664479 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f6e30012750 con 0x7f6e4811b6c0 2019-12-05T05:12:37.887 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:37.852+0000 7f6e46ffd700 1 --2- 172.21.15.42:0/2997664479 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6e4811b6c0 0x7f6e480a3720 secure :-1 s=THROTTLE_DONE pgs=464 cs=0 l=1 rx=0x7f6e4807d690 tx=0x7f6e3000cb30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:37.888 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:37.852+0000 7f6e3ffff700 1 --2- 172.21.15.42:0/2997664479 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6e2c03b280 0x7f6e2c03d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:12:37.888 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:37.852+0000 7f6e3ffff700 1 -- 172.21.15.42:0/2997664479 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6489+0+0 (secure 0 0 0) 0x7f6e3004b420 con 0x7f6e4811b6c0 2019-12-05T05:12:37.888 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:37.852+0000 7f6e477fe700 1 --2- 172.21.15.42:0/2997664479 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6e2c03b280 0x7f6e2c03d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:12:37.888 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:37.856+0000 7f6e477fe700 1 --2- 172.21.15.42:0/2997664479 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6e2c03b280 0x7f6e2c03d790 crc :-1 s=READY pgs=211 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:12:37.888 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:37.856+0000 7f6e4d3a6700 1 -- 172.21.15.42:0/2997664479 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e4805d110 con 0x7f6e4811b6c0 2019-12-05T05:12:37.889 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:37.864+0000 7f6e46ffd700 1 --2- 172.21.15.42:0/2997664479 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6e4811b6c0 0x7f6e480a3720 secure :-1 s=THROTTLE_DONE pgs=464 cs=0 l=1 rx=0x7f6e4807d690 tx=0x7f6e3000cb30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:37.889 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:37.864+0000 7f6e3ffff700 1 -- 172.21.15.42:0/2997664479 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f6e30032480 con 0x7f6e4811b6c0 2019-12-05T05:12:38.042 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:38.036+0000 7f6e4d3a6700 1 -- 172.21.15.42:0/2997664479 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6e4805b8c0 con 0x7f6e4811b6c0 2019-12-05T05:12:38.043 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:38.036+0000 7f6e46ffd700 1 --2- 172.21.15.42:0/2997664479 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6e4811b6c0 0x7f6e480a3720 secure :-1 s=THROTTLE_DONE pgs=464 cs=0 l=1 rx=0x7f6e4807d690 tx=0x7f6e3000cb30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:38.043 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:38.036+0000 7f6e3ffff700 1 -- 172.21.15.42:0/2997664479 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) v1 ==== 94+0+2105 (secure 0 0 0) 0x7f6e300326e0 con 0x7f6e4811b6c0 2019-12-05T05:12:38.043 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 58 2019-12-05T05:12:38.043 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:12:38.043 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":58,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5391,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/814397242","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":814397242},{"type":"v1","addr":"172.21.15.40:6833","nonce":814397242}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":58}],"filesystems":[{"mdsmap":{"epoch":57,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:11:24.854205+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5215},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5215":{"gid":5215,"name":"a","rank":0,"incarnation":52,"state":"up:active","state_seq":20,"addr":"172.21.15.42:6835/3176146895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3176146895},{"type":"v1","addr":"172.21.15.42:6835","nonce":3176146895}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:12:38.045 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:38.040+0000 7f6e4d3a6700 1 -- 172.21.15.42:0/2997664479 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6e2c03b280 msgr2=0x7f6e2c03d790 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:12:38.045 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:38.040+0000 7f6e4d3a6700 1 --2- 172.21.15.42:0/2997664479 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6e2c03b280 0x7f6e2c03d790 crc :-1 s=READY pgs=211 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:12:38.045 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:38.040+0000 7f6e4d3a6700 1 -- 172.21.15.42:0/2997664479 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6e4811b6c0 msgr2=0x7f6e480a3720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:12:38.046 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:38.040+0000 7f6e4d3a6700 1 --2- 172.21.15.42:0/2997664479 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6e4811b6c0 0x7f6e480a3720 secure :-1 s=READY pgs=464 cs=0 l=1 rx=0x7f6e4807d690 tx=0x7f6e3000cb30).stop 2019-12-05T05:12:38.046 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:38.040+0000 7f6e4d3a6700 1 -- 172.21.15.42:0/2997664479 shutdown_connections 2019-12-05T05:12:38.046 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:38.040+0000 7f6e4d3a6700 1 --2- 172.21.15.42:0/2997664479 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6e2c03b280 0x7f6e2c03d790 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:12:38.046 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:38.040+0000 7f6e4d3a6700 1 --2- 172.21.15.42:0/2997664479 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6e48122950 0x7f6e480a6290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:12:38.046 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:38.040+0000 7f6e4d3a6700 1 --2- 172.21.15.42:0/2997664479 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6e4811b6c0 0x7f6e480a3720 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:12:38.046 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:38.040+0000 7f6e4d3a6700 1 -- 172.21.15.42:0/2997664479 shutdown_connections 2019-12-05T05:12:38.047 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:38.040+0000 7f6e4d3a6700 1 -- 172.21.15.42:0/2997664479 wait complete. 2019-12-05T05:12:38.047 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:38.040+0000 7f6e4d3a6700 1 -- 172.21.15.42:0/2997664479 >> 172.21.15.42:0/2997664479 conn(0x7f6e480a9ae0 msgr2=0x7f6e4809f350 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:12:38.063 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T05:12:40.066 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:12:40.169 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:40.160+0000 7fdba923e700 1 Processor -- start 2019-12-05T05:12:40.169 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:40.160+0000 7fdba923e700 1 -- start start 2019-12-05T05:12:40.170 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:40.160+0000 7fdba923e700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdba40a7740 0x7fdba408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:12:40.170 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:40.160+0000 7fdba923e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fdba407e720 0x7fdba407eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:12:40.170 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:40.160+0000 7fdba923e700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fdba409f2e0 0x7fdba409f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:12:40.170 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:40.160+0000 7fdba923e700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdba40a0b00 con 0x7fdba409f2e0 2019-12-05T05:12:40.170 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:40.160+0000 7fdba923e700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdba408e5b0 con 0x7fdba407e720 2019-12-05T05:12:40.170 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:40.160+0000 7fdba923e700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdba408e850 con 0x7fdba40a7740 2019-12-05T05:12:40.171 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:40.160+0000 7fdba2d9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fdba407e720 0x7fdba407eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:12:40.171 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:40.160+0000 7fdba2d9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fdba407e720 0x7fdba407eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:57900/0 (socket says 172.21.15.42:57900) 2019-12-05T05:12:40.171 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:40.160+0000 7fdba2d9d700 1 -- 172.21.15.42:0/1891080815 learned_addr learned my addr 172.21.15.42:0/1891080815 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:12:40.171 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:40.160+0000 7fdba359e700 1 --2- 172.21.15.42:0/1891080815 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdba40a7740 0x7fdba408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:12:40.171 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:40.160+0000 7fdba8a3d700 1 --2- 172.21.15.42:0/1891080815 >> v2:172.21.15.40:3300/0 conn(0x7fdba409f2e0 0x7fdba409f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:12:40.171 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:40.160+0000 7fdba2d9d700 1 -- 172.21.15.42:0/1891080815 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdba40a7740 msgr2=0x7fdba408df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:12:40.171 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:40.160+0000 7fdba2d9d700 1 --2- 172.21.15.42:0/1891080815 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdba40a7740 0x7fdba408df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:12:40.172 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:40.160+0000 7fdba2d9d700 1 -- 172.21.15.42:0/1891080815 >> v2:172.21.15.40:3300/0 conn(0x7fdba409f2e0 msgr2=0x7fdba409f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:12:40.172 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:40.160+0000 7fdba2d9d700 1 --2- 172.21.15.42:0/1891080815 >> v2:172.21.15.40:3300/0 conn(0x7fdba409f2e0 0x7fdba409f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:12:40.172 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:40.160+0000 7fdba2d9d700 1 -- 172.21.15.42:0/1891080815 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdba407f0a0 con 0x7fdba407e720 2019-12-05T05:12:40.172 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:40.160+0000 7fdba2d9d700 1 --2- 172.21.15.42:0/1891080815 >> v2:172.21.15.42:3300/0 conn(0x7fdba407e720 0x7fdba407eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fdba408e770 tx=0x7fdb8c00fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:40.172 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:40.160+0000 7fdba2d9d700 1 --2- 172.21.15.42:0/1891080815 >> v2:172.21.15.42:3300/0 conn(0x7fdba407e720 0x7fdba407eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fdba408e770 tx=0x7fdb8c00fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:40.172 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:40.160+0000 7fdba2d9d700 1 --2- 172.21.15.42:0/1891080815 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdba407e720 0x7fdba407eb50 secure :-1 s=READY pgs=465 cs=0 l=1 rx=0x7fdba408e770 tx=0x7fdb8c00fdb0).ready entity=mon.1 client_cookie=5168fb8ac1123149 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:12:40.172 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:40.160+0000 7fdba2d9d700 1 --2- 172.21.15.42:0/1891080815 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdba407e720 0x7fdba407eb50 secure :-1 s=THROTTLE_DONE pgs=465 cs=0 l=1 rx=0x7fdba408e770 tx=0x7fdb8c00fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:40.173 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:40.160+0000 7fdba259c700 1 -- 172.21.15.42:0/1891080815 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdb8c008c90 con 0x7fdba407e720 2019-12-05T05:12:40.173 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:40.160+0000 7fdba2d9d700 1 --2- 172.21.15.42:0/1891080815 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdba407e720 0x7fdba407eb50 secure :-1 s=THROTTLE_DONE pgs=465 cs=0 l=1 rx=0x7fdba408e770 tx=0x7fdb8c00fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:40.173 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:40.164+0000 7fdba259c700 1 -- 172.21.15.42:0/1891080815 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb8c0088b0 con 0x7fdba407e720 2019-12-05T05:12:40.173 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:40.164+0000 7fdba2d9d700 1 --2- 172.21.15.42:0/1891080815 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdba407e720 0x7fdba407eb50 secure :-1 s=THROTTLE_DONE pgs=465 cs=0 l=1 rx=0x7fdba408e770 tx=0x7fdb8c00fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:40.173 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:40.164+0000 7fdba923e700 1 -- 172.21.15.42:0/1891080815 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdba407e720 msgr2=0x7fdba407eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:12:40.173 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:40.164+0000 7fdba259c700 1 -- 172.21.15.42:0/1891080815 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdb8c011b50 con 0x7fdba407e720 2019-12-05T05:12:40.173 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:40.164+0000 7fdba923e700 1 --2- 172.21.15.42:0/1891080815 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdba407e720 0x7fdba407eb50 secure :-1 s=READY pgs=465 cs=0 l=1 rx=0x7fdba408e770 tx=0x7fdb8c00fdb0).stop 2019-12-05T05:12:40.174 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:40.164+0000 7fdba923e700 1 -- 172.21.15.42:0/1891080815 shutdown_connections 2019-12-05T05:12:40.174 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:40.164+0000 7fdba923e700 1 --2- 172.21.15.42:0/1891080815 >> v2:172.21.15.40:3300/0 conn(0x7fdba409f2e0 0x7fdba409f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:12:40.174 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:40.164+0000 7fdba923e700 1 --2- 172.21.15.42:0/1891080815 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdba40a7740 0x7fdba408df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:12:40.174 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:40.164+0000 7fdba923e700 1 --2- 172.21.15.42:0/1891080815 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdba407e720 0x7fdba407eb50 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:12:40.174 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:40.164+0000 7fdba923e700 1 -- 172.21.15.42:0/1891080815 shutdown_connections 2019-12-05T05:12:40.174 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:40.164+0000 7fdba923e700 1 -- 172.21.15.42:0/1891080815 wait complete. 2019-12-05T05:12:40.174 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:40.164+0000 7fdba923e700 1 -- 172.21.15.42:0/1891080815 >> 172.21.15.42:0/1891080815 conn(0x7fdba40aabe0 msgr2=0x7fdba40a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:12:40.178 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:40.164+0000 7fdba923e700 1 Processor -- start 2019-12-05T05:12:40.178 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:40.164+0000 7fdba923e700 1 -- start start 2019-12-05T05:12:40.178 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:40.164+0000 7fdba923e700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fdba40a2120 0x7fdba4142d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:12:40.178 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:40.164+0000 7fdba923e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fdba40a3d50 0x7fdba407da80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:12:40.178 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:40.164+0000 7fdba923e700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdba41432e0 con 0x7fdba4144460 2019-12-05T05:12:40.178 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:40.164+0000 7fdba923e700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdba41435c0 con 0x7fdba40a2120 2019-12-05T05:12:40.179 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:40.164+0000 7fdba923e700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdba4143860 con 0x7fdba40a3d50 2019-12-05T05:12:40.179 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:40.164+0000 7fdba2d9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fdba40a3d50 0x7fdba407da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:12:40.179 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:40.164+0000 7fdba2d9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fdba40a3d50 0x7fdba407da80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:57906/0 (socket says 172.21.15.42:57906) 2019-12-05T05:12:40.179 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:40.164+0000 7fdba2d9d700 1 -- 172.21.15.42:0/388353515 learned_addr learned my addr 172.21.15.42:0/388353515 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:12:40.179 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:40.168+0000 7fdba359e700 1 --2- 172.21.15.42:0/388353515 >> v2:172.21.15.40:3300/0 conn(0x7fdba40a2120 0x7fdba4142d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:12:40.179 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:40.168+0000 7fdba2d9d700 1 -- 172.21.15.42:0/388353515 >> v2:172.21.15.40:3300/0 conn(0x7fdba40a2120 msgr2=0x7fdba4142d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:12:40.179 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:40.168+0000 7fdba2d9d700 1 --2- 172.21.15.42:0/388353515 >> v2:172.21.15.40:3300/0 conn(0x7fdba40a2120 0x7fdba4142d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:12:40.180 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:40.168+0000 7fdba2d9d700 1 -- 172.21.15.42:0/388353515 >> v1:172.21.15.40:6789/0 conn(0x7fdba4144460 legacy=0x7fdba4144890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:12:40.180 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:40.168+0000 7fdba2d9d700 1 -- 172.21.15.42:0/388353515 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdba407dfd0 con 0x7fdba40a3d50 2019-12-05T05:12:40.180 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:40.168+0000 7fdba359e700 1 --2- 172.21.15.42:0/388353515 >> v2:172.21.15.40:3300/0 conn(0x7fdba40a2120 0x7fdba4142d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T05:12:40.180 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:40.168+0000 7fdba2d9d700 1 --2- 172.21.15.42:0/388353515 >> v2:172.21.15.42:3300/0 conn(0x7fdba40a3d50 0x7fdba407da80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fdba408e770 tx=0x7fdb8c008bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:40.180 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:40.168+0000 7fdba2d9d700 1 --2- 172.21.15.42:0/388353515 >> v2:172.21.15.42:3300/0 conn(0x7fdba40a3d50 0x7fdba407da80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fdba408e770 tx=0x7fdb8c008bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:40.180 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:40.168+0000 7fdba2d9d700 1 --2- 172.21.15.42:0/388353515 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdba40a3d50 0x7fdba407da80 secure :-1 s=READY pgs=466 cs=0 l=1 rx=0x7fdba408e770 tx=0x7fdb8c008bc0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:12:40.180 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:40.168+0000 7fdba2d9d700 1 --2- 172.21.15.42:0/388353515 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdba40a3d50 0x7fdba407da80 secure :-1 s=THROTTLE_DONE pgs=466 cs=0 l=1 rx=0x7fdba408e770 tx=0x7fdb8c008bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:40.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:40.168+0000 7fdb9bfff700 1 -- 172.21.15.42:0/388353515 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdb8c00f680 con 0x7fdba40a3d50 2019-12-05T05:12:40.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:40.168+0000 7fdba2d9d700 1 --2- 172.21.15.42:0/388353515 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdba40a3d50 0x7fdba407da80 secure :-1 s=THROTTLE_DONE pgs=466 cs=0 l=1 rx=0x7fdba408e770 tx=0x7fdb8c008bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:40.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:40.168+0000 7fdba2d9d700 1 --2- 172.21.15.42:0/388353515 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdba40a3d50 0x7fdba407da80 secure :-1 s=THROTTLE_DONE pgs=466 cs=0 l=1 rx=0x7fdba408e770 tx=0x7fdb8c008bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:40.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:40.168+0000 7fdb9bfff700 1 -- 172.21.15.42:0/388353515 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb8c011770 con 0x7fdba40a3d50 2019-12-05T05:12:40.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:40.168+0000 7fdba923e700 1 -- 172.21.15.42:0/388353515 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdba4137740 con 0x7fdba40a3d50 2019-12-05T05:12:40.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:40.168+0000 7fdb9bfff700 1 -- 172.21.15.42:0/388353515 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdb8c005920 con 0x7fdba40a3d50 2019-12-05T05:12:40.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:40.168+0000 7fdba923e700 1 -- 172.21.15.42:0/388353515 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdba4137c70 con 0x7fdba40a3d50 2019-12-05T05:12:40.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:40.168+0000 7fdba2d9d700 1 --2- 172.21.15.42:0/388353515 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdba40a3d50 0x7fdba407da80 secure :-1 s=THROTTLE_DONE pgs=466 cs=0 l=1 rx=0x7fdba408e770 tx=0x7fdb8c008bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:40.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:40.168+0000 7fdb9bfff700 1 -- 172.21.15.42:0/388353515 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fdb8c005c30 con 0x7fdba40a3d50 2019-12-05T05:12:40.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:40.168+0000 7fdba2d9d700 1 --2- 172.21.15.42:0/388353515 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdba40a3d50 0x7fdba407da80 secure :-1 s=THROTTLE_DONE pgs=466 cs=0 l=1 rx=0x7fdba408e770 tx=0x7fdb8c008bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:40.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:40.168+0000 7fdb9bfff700 1 --2- 172.21.15.42:0/388353515 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fdb7c03ad10 0x7fdb7c03d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:12:40.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:40.168+0000 7fdb9bfff700 1 -- 172.21.15.42:0/388353515 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6489+0+0 (secure 0 0 0) 0x7fdb8c04b5a0 con 0x7fdba40a3d50 2019-12-05T05:12:40.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:40.168+0000 7fdba359e700 1 --2- 172.21.15.42:0/388353515 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fdb7c03ad10 0x7fdb7c03d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:12:40.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:40.172+0000 7fdba359e700 1 --2- 172.21.15.42:0/388353515 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fdb7c03ad10 0x7fdb7c03d220 crc :-1 s=READY pgs=212 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:12:40.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:40.176+0000 7fdba923e700 1 -- 172.21.15.42:0/388353515 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdba4143a20 con 0x7fdba40a3d50 2019-12-05T05:12:40.196 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:40.184+0000 7fdba2d9d700 1 --2- 172.21.15.42:0/388353515 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdba40a3d50 0x7fdba407da80 secure :-1 s=THROTTLE_DONE pgs=466 cs=0 l=1 rx=0x7fdba408e770 tx=0x7fdb8c008bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:40.196 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:40.184+0000 7fdb9bfff700 1 -- 172.21.15.42:0/388353515 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fdb8c030290 con 0x7fdba40a3d50 2019-12-05T05:12:40.359 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:40.344+0000 7fdba923e700 1 -- 172.21.15.42:0/388353515 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdba405d110 con 0x7fdba40a3d50 2019-12-05T05:12:40.359 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:40.344+0000 7fdba2d9d700 1 --2- 172.21.15.42:0/388353515 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdba40a3d50 0x7fdba407da80 secure :-1 s=THROTTLE_DONE pgs=466 cs=0 l=1 rx=0x7fdba408e770 tx=0x7fdb8c008bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:40.359 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:40.344+0000 7fdb9bfff700 1 -- 172.21.15.42:0/388353515 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) v1 ==== 94+0+2105 (secure 0 0 0) 0x7fdb8c010bb0 con 0x7fdba40a3d50 2019-12-05T05:12:40.360 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 58 2019-12-05T05:12:40.360 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:40.348+0000 7fdba923e700 1 -- 172.21.15.42:0/388353515 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fdb7c03ad10 msgr2=0x7fdb7c03d220 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:12:40.360 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:40.348+0000 7fdba923e700 1 --2- 172.21.15.42:0/388353515 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fdb7c03ad10 0x7fdb7c03d220 crc :-1 s=READY pgs=212 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:12:40.360 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:40.348+0000 7fdba923e700 1 -- 172.21.15.42:0/388353515 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdba40a3d50 msgr2=0x7fdba407da80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:12:40.361 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:40.348+0000 7fdba923e700 1 --2- 172.21.15.42:0/388353515 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdba40a3d50 0x7fdba407da80 secure :-1 s=READY pgs=466 cs=0 l=1 rx=0x7fdba408e770 tx=0x7fdb8c008bc0).stop 2019-12-05T05:12:40.361 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:40.348+0000 7fdba923e700 1 -- 172.21.15.42:0/388353515 shutdown_connections 2019-12-05T05:12:40.361 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:40.348+0000 7fdba923e700 1 --2- 172.21.15.42:0/388353515 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fdb7c03ad10 0x7fdb7c03d220 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:12:40.361 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:40.348+0000 7fdba923e700 1 --2- 172.21.15.42:0/388353515 >> v2:172.21.15.40:3300/0 conn(0x7fdba40a2120 0x7fdba4142d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:12:40.361 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:40.348+0000 7fdba923e700 1 --2- 172.21.15.42:0/388353515 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdba40a3d50 0x7fdba407da80 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:12:40.361 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:40.352+0000 7fdba923e700 1 -- 172.21.15.42:0/388353515 shutdown_connections 2019-12-05T05:12:40.362 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:40.352+0000 7fdba923e700 1 -- 172.21.15.42:0/388353515 wait complete. 2019-12-05T05:12:40.362 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:40.352+0000 7fdba923e700 1 -- 172.21.15.42:0/388353515 >> 172.21.15.42:0/388353515 conn(0x7fdba407e560 msgr2=0x7fdba411c560 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:12:40.362 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:12:40.362 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":58,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5391,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/814397242","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":814397242},{"type":"v1","addr":"172.21.15.40:6833","nonce":814397242}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":58}],"filesystems":[{"mdsmap":{"epoch":57,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:11:24.854205+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5215},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5215":{"gid":5215,"name":"a","rank":0,"incarnation":52,"state":"up:active","state_seq":20,"addr":"172.21.15.42:6835/3176146895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3176146895},{"type":"v1","addr":"172.21.15.42:6835","nonce":3176146895}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:12:40.367 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T05:12:42.368 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:12:42.476 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:42.464+0000 7ff2e1cbc700 1 Processor -- start 2019-12-05T05:12:42.477 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:42.464+0000 7ff2e1cbc700 1 -- start start 2019-12-05T05:12:42.477 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:42.468+0000 7ff2e1cbc700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff2dc0a7740 0x7ff2dc08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:12:42.477 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:42.468+0000 7ff2e1cbc700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff2dc07e720 0x7ff2dc07eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:12:42.477 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:42.468+0000 7ff2e1cbc700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2dc0a0710 con 0x7ff2dc09f2e0 2019-12-05T05:12:42.477 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:42.468+0000 7ff2e1cbc700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2dc0a0b00 con 0x7ff2dc07e720 2019-12-05T05:12:42.477 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:42.468+0000 7ff2e1cbc700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2dc08e5b0 con 0x7ff2dc0a7740 2019-12-05T05:12:42.478 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:42.468+0000 7ff2db7fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff2dc07e720 0x7ff2dc07eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:12:42.478 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:42.468+0000 7ff2db7fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff2dc07e720 0x7ff2dc07eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:57916/0 (socket says 172.21.15.42:57916) 2019-12-05T05:12:42.478 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:42.468+0000 7ff2db7fe700 1 -- 172.21.15.42:0/1281269353 learned_addr learned my addr 172.21.15.42:0/1281269353 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:12:42.478 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:42.468+0000 7ff2dbfff700 1 --2- 172.21.15.42:0/1281269353 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff2dc0a7740 0x7ff2dc08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:12:42.478 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:42.468+0000 7ff2db7fe700 1 -- 172.21.15.42:0/1281269353 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff2dc0a7740 msgr2=0x7ff2dc08df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:12:42.478 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:42.468+0000 7ff2db7fe700 1 --2- 172.21.15.42:0/1281269353 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff2dc0a7740 0x7ff2dc08df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:12:42.478 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:42.468+0000 7ff2db7fe700 1 -- 172.21.15.42:0/1281269353 >> v1:172.21.15.40:6789/0 conn(0x7ff2dc09f2e0 legacy=0x7ff2dc09f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:12:42.479 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:42.468+0000 7ff2db7fe700 1 -- 172.21.15.42:0/1281269353 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2dc07f0a0 con 0x7ff2dc07e720 2019-12-05T05:12:42.479 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:42.468+0000 7ff2db7fe700 1 --2- 172.21.15.42:0/1281269353 >> v2:172.21.15.42:3300/0 conn(0x7ff2dc07e720 0x7ff2dc07eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7ff2c4008790 tx=0x7ff2c400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:42.479 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:42.468+0000 7ff2db7fe700 1 --2- 172.21.15.42:0/1281269353 >> v2:172.21.15.42:3300/0 conn(0x7ff2dc07e720 0x7ff2dc07eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7ff2c4008790 tx=0x7ff2c400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:42.479 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:42.468+0000 7ff2db7fe700 1 --2- 172.21.15.42:0/1281269353 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff2dc07e720 0x7ff2dc07eb50 secure :-1 s=READY pgs=467 cs=0 l=1 rx=0x7ff2c4008790 tx=0x7ff2c400fd30).ready entity=mon.1 client_cookie=cfa0089068a7d0bb server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:12:42.479 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:42.468+0000 7ff2db7fe700 1 --2- 172.21.15.42:0/1281269353 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff2dc07e720 0x7ff2dc07eb50 secure :-1 s=THROTTLE_DONE pgs=467 cs=0 l=1 rx=0x7ff2c4008790 tx=0x7ff2c400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:42.479 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:42.468+0000 7ff2db7fe700 1 --2- 172.21.15.42:0/1281269353 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff2dc07e720 0x7ff2dc07eb50 secure :-1 s=THROTTLE_DONE pgs=467 cs=0 l=1 rx=0x7ff2c4008790 tx=0x7ff2c400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:42.479 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:42.468+0000 7ff2db7fe700 1 --2- 172.21.15.42:0/1281269353 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff2dc07e720 0x7ff2dc07eb50 secure :-1 s=THROTTLE_DONE pgs=467 cs=0 l=1 rx=0x7ff2c4008790 tx=0x7ff2c400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:42.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:42.468+0000 7ff2daffd700 1 -- 172.21.15.42:0/1281269353 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff2c40113b0 con 0x7ff2dc07e720 2019-12-05T05:12:42.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:42.468+0000 7ff2daffd700 1 -- 172.21.15.42:0/1281269353 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2c4008bb0 con 0x7ff2dc07e720 2019-12-05T05:12:42.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:42.468+0000 7ff2daffd700 1 -- 172.21.15.42:0/1281269353 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff2c4011990 con 0x7ff2dc07e720 2019-12-05T05:12:42.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:42.468+0000 7ff2e1cbc700 1 -- 172.21.15.42:0/1281269353 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff2dc07e720 msgr2=0x7ff2dc07eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:12:42.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:42.468+0000 7ff2e1cbc700 1 --2- 172.21.15.42:0/1281269353 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff2dc07e720 0x7ff2dc07eb50 secure :-1 s=READY pgs=467 cs=0 l=1 rx=0x7ff2c4008790 tx=0x7ff2c400fd30).stop 2019-12-05T05:12:42.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:42.468+0000 7ff2e1cbc700 1 -- 172.21.15.42:0/1281269353 shutdown_connections 2019-12-05T05:12:42.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:42.468+0000 7ff2e1cbc700 1 --2- 172.21.15.42:0/1281269353 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff2dc0a7740 0x7ff2dc08df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:12:42.482 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:42.468+0000 7ff2e1cbc700 1 --2- 172.21.15.42:0/1281269353 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff2dc07e720 0x7ff2dc07eb50 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:12:42.482 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:42.468+0000 7ff2e1cbc700 1 -- 172.21.15.42:0/1281269353 shutdown_connections 2019-12-05T05:12:42.482 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:42.468+0000 7ff2e1cbc700 1 -- 172.21.15.42:0/1281269353 wait complete. 2019-12-05T05:12:42.482 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:42.468+0000 7ff2e1cbc700 1 -- 172.21.15.42:0/1281269353 >> 172.21.15.42:0/1281269353 conn(0x7ff2dc0aabe0 msgr2=0x7ff2dc0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:12:42.485 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:42.472+0000 7ff2e1cbc700 1 Processor -- start 2019-12-05T05:12:42.486 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:42.472+0000 7ff2e1cbc700 1 -- start start 2019-12-05T05:12:42.486 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:42.472+0000 7ff2e1cbc700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff2dc07c870 0x7ff2dc0a56a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:12:42.486 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:42.472+0000 7ff2e1cbc700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff2dc07bab0 0x7ff2dc07bee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:12:42.486 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:42.472+0000 7ff2e1cbc700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2dc07c430 con 0x7ff2dc116500 2019-12-05T05:12:42.487 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:42.472+0000 7ff2e1cbc700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2dc0a4c80 con 0x7ff2dc07bab0 2019-12-05T05:12:42.487 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:42.472+0000 7ff2e1cbc700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2dc07cea0 con 0x7ff2dc07c870 2019-12-05T05:12:42.487 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:42.476+0000 7ff2db7fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff2dc07c870 0x7ff2dc0a56a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:12:42.487 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:42.476+0000 7ff2dbfff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7ff2dc116500 0x7ff2dc141950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:57960/0 (socket says 172.21.15.42:57960) 2019-12-05T05:12:42.487 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:42.476+0000 7ff2dbfff700 1 -- 172.21.15.42:0/1628984106 learned_addr learned my addr 172.21.15.42:0/1628984106 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:12:42.488 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:42.476+0000 7ff2e0a5a700 1 --2- 172.21.15.42:0/1628984106 >> v2:172.21.15.40:3300/0 conn(0x7ff2dc07bab0 0x7ff2dc07bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:12:42.492 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:42.476+0000 7ff2db7fe700 1 -- 172.21.15.42:0/1628984106 >> v2:172.21.15.40:3300/0 conn(0x7ff2dc07bab0 msgr2=0x7ff2dc07bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:12:42.492 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:42.476+0000 7ff2db7fe700 1 --2- 172.21.15.42:0/1628984106 >> v2:172.21.15.40:3300/0 conn(0x7ff2dc07bab0 0x7ff2dc07bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:12:42.492 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:42.476+0000 7ff2db7fe700 1 -- 172.21.15.42:0/1628984106 >> v1:172.21.15.42:6789/0 conn(0x7ff2dc116500 legacy=0x7ff2dc141950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:12:42.492 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:42.476+0000 7ff2d8ff9700 1 -- 172.21.15.42:0/1628984106 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3578374865 0 0) 0x7ff2dc07c430 con 0x7ff2dc116500 2019-12-05T05:12:42.492 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:42.476+0000 7ff2db7fe700 1 -- 172.21.15.42:0/1628984106 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2dc0752e0 con 0x7ff2dc07c870 2019-12-05T05:12:42.493 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:42.476+0000 7ff2db7fe700 1 --2- 172.21.15.42:0/1628984106 >> v2:172.21.15.42:3300/0 conn(0x7ff2dc07c870 0x7ff2dc0a56a0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7ff2c4008790 tx=0x7ff2c4008bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:42.493 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:42.476+0000 7ff2db7fe700 1 --2- 172.21.15.42:0/1628984106 >> v2:172.21.15.42:3300/0 conn(0x7ff2dc07c870 0x7ff2dc0a56a0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7ff2c4008790 tx=0x7ff2c4008bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:42.493 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:42.476+0000 7ff2db7fe700 1 --2- 172.21.15.42:0/1628984106 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff2dc07c870 0x7ff2dc0a56a0 secure :-1 s=READY pgs=469 cs=0 l=1 rx=0x7ff2c4008790 tx=0x7ff2c4008bb0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:12:42.493 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:42.476+0000 7ff2db7fe700 1 --2- 172.21.15.42:0/1628984106 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff2dc07c870 0x7ff2dc0a56a0 secure :-1 s=THROTTLE_DONE pgs=469 cs=0 l=1 rx=0x7ff2c4008790 tx=0x7ff2c4008bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:42.493 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:42.476+0000 7ff2d8ff9700 1 -- 172.21.15.42:0/1628984106 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff2c4007340 con 0x7ff2dc07c870 2019-12-05T05:12:42.493 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:42.476+0000 7ff2db7fe700 1 --2- 172.21.15.42:0/1628984106 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff2dc07c870 0x7ff2dc0a56a0 secure :-1 s=THROTTLE_DONE pgs=469 cs=0 l=1 rx=0x7ff2c4008790 tx=0x7ff2c4008bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:42.493 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:42.476+0000 7ff2db7fe700 1 --2- 172.21.15.42:0/1628984106 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff2dc07c870 0x7ff2dc0a56a0 secure :-1 s=THROTTLE_DONE pgs=469 cs=0 l=1 rx=0x7ff2c4008790 tx=0x7ff2c4008bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:42.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:42.476+0000 7ff2d8ff9700 1 -- 172.21.15.42:0/1628984106 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2c40115b0 con 0x7ff2dc07c870 2019-12-05T05:12:42.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:42.476+0000 7ff2e1cbc700 1 -- 172.21.15.42:0/1628984106 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2dc0742d0 con 0x7ff2dc07c870 2019-12-05T05:12:42.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:42.476+0000 7ff2d8ff9700 1 -- 172.21.15.42:0/1628984106 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff2c4005540 con 0x7ff2dc07c870 2019-12-05T05:12:42.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:42.476+0000 7ff2e1cbc700 1 -- 172.21.15.42:0/1628984106 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2dc0747d0 con 0x7ff2dc07c870 2019-12-05T05:12:42.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:42.476+0000 7ff2db7fe700 1 --2- 172.21.15.42:0/1628984106 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff2dc07c870 0x7ff2dc0a56a0 secure :-1 s=THROTTLE_DONE pgs=469 cs=0 l=1 rx=0x7ff2c4008790 tx=0x7ff2c4008bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:42.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:42.476+0000 7ff2d8ff9700 1 -- 172.21.15.42:0/1628984106 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7ff2c4005850 con 0x7ff2dc07c870 2019-12-05T05:12:42.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:42.476+0000 7ff2db7fe700 1 --2- 172.21.15.42:0/1628984106 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff2dc07c870 0x7ff2dc0a56a0 secure :-1 s=THROTTLE_DONE pgs=469 cs=0 l=1 rx=0x7ff2c4008790 tx=0x7ff2c4008bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:42.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:42.480+0000 7ff2d8ff9700 1 --2- 172.21.15.42:0/1628984106 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff2c003ad20 0x7ff2c003d230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:12:42.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:42.480+0000 7ff2d8ff9700 1 -- 172.21.15.42:0/1628984106 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6489+0+0 (secure 0 0 0) 0x7ff2c404b390 con 0x7ff2dc07c870 2019-12-05T05:12:42.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:42.480+0000 7ff2dbfff700 1 --2- 172.21.15.42:0/1628984106 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff2c003ad20 0x7ff2c003d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:12:42.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:42.480+0000 7ff2dbfff700 1 --2- 172.21.15.42:0/1628984106 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff2c003ad20 0x7ff2c003d230 crc :-1 s=READY pgs=213 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:12:42.496 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:42.480+0000 7ff2e1cbc700 1 -- 172.21.15.42:0/1628984106 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2dc07cca0 con 0x7ff2dc07c870 2019-12-05T05:12:42.496 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:42.488+0000 7ff2db7fe700 1 --2- 172.21.15.42:0/1628984106 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff2dc07c870 0x7ff2dc0a56a0 secure :-1 s=THROTTLE_DONE pgs=469 cs=0 l=1 rx=0x7ff2c4008790 tx=0x7ff2c4008bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:42.496 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:42.488+0000 7ff2d8ff9700 1 -- 172.21.15.42:0/1628984106 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7ff2c4032410 con 0x7ff2dc07c870 2019-12-05T05:12:42.663 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:42.656+0000 7ff2e1cbc700 1 -- 172.21.15.42:0/1628984106 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff2dc05d110 con 0x7ff2dc07c870 2019-12-05T05:12:42.664 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:42.656+0000 7ff2db7fe700 1 --2- 172.21.15.42:0/1628984106 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff2dc07c870 0x7ff2dc0a56a0 secure :-1 s=THROTTLE_DONE pgs=469 cs=0 l=1 rx=0x7ff2c4008790 tx=0x7ff2c4008bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:42.664 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:42.656+0000 7ff2d8ff9700 1 -- 172.21.15.42:0/1628984106 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) v1 ==== 94+0+2105 (secure 0 0 0) 0x7ff2c40321c0 con 0x7ff2dc07c870 2019-12-05T05:12:42.664 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 58 2019-12-05T05:12:42.664 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:12:42.664 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":58,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5391,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/814397242","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":814397242},{"type":"v1","addr":"172.21.15.40:6833","nonce":814397242}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":58}],"filesystems":[{"mdsmap":{"epoch":57,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:11:24.854205+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5215},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5215":{"gid":5215,"name":"a","rank":0,"incarnation":52,"state":"up:active","state_seq":20,"addr":"172.21.15.42:6835/3176146895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3176146895},{"type":"v1","addr":"172.21.15.42:6835","nonce":3176146895}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:12:42.666 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:42.660+0000 7ff2e1cbc700 1 -- 172.21.15.42:0/1628984106 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff2c003ad20 msgr2=0x7ff2c003d230 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:12:42.666 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:42.660+0000 7ff2e1cbc700 1 --2- 172.21.15.42:0/1628984106 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff2c003ad20 0x7ff2c003d230 crc :-1 s=READY pgs=213 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:12:42.667 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:42.660+0000 7ff2e1cbc700 1 -- 172.21.15.42:0/1628984106 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff2dc07c870 msgr2=0x7ff2dc0a56a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:12:42.667 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:42.660+0000 7ff2e1cbc700 1 --2- 172.21.15.42:0/1628984106 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff2dc07c870 0x7ff2dc0a56a0 secure :-1 s=READY pgs=469 cs=0 l=1 rx=0x7ff2c4008790 tx=0x7ff2c4008bb0).stop 2019-12-05T05:12:42.667 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:42.660+0000 7ff2e1cbc700 1 -- 172.21.15.42:0/1628984106 shutdown_connections 2019-12-05T05:12:42.667 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:42.660+0000 7ff2e1cbc700 1 --2- 172.21.15.42:0/1628984106 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff2c003ad20 0x7ff2c003d230 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:12:42.668 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:42.660+0000 7ff2e1cbc700 1 --2- 172.21.15.42:0/1628984106 >> v2:172.21.15.40:3300/0 conn(0x7ff2dc07bab0 0x7ff2dc07bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:12:42.668 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:42.660+0000 7ff2e1cbc700 1 --2- 172.21.15.42:0/1628984106 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff2dc07c870 0x7ff2dc0a56a0 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:12:42.668 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:42.660+0000 7ff2e1cbc700 1 -- 172.21.15.42:0/1628984106 shutdown_connections 2019-12-05T05:12:42.669 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:42.660+0000 7ff2e1cbc700 1 -- 172.21.15.42:0/1628984106 wait complete. 2019-12-05T05:12:42.669 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:42.660+0000 7ff2e1cbc700 1 -- 172.21.15.42:0/1628984106 >> 172.21.15.42:0/1628984106 conn(0x7ff2dc0a9870 msgr2=0x7ff2dc11c610 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:12:42.676 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T05:12:44.676 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:12:44.778 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:44.768+0000 7f1f8288a700 1 Processor -- start 2019-12-05T05:12:44.778 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:44.768+0000 7f1f8288a700 1 -- start start 2019-12-05T05:12:44.779 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:44.768+0000 7f1f8288a700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f1f7c0a7740 0x7f1f7c08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:12:44.779 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:44.768+0000 7f1f8288a700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1f7c09fae0 0x7f1f7c07eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:12:44.779 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:44.768+0000 7f1f8288a700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f7c0a0710 con 0x7f1f7c07e720 2019-12-05T05:12:44.779 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:44.768+0000 7f1f8288a700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f7c0a0b00 con 0x7f1f7c0a7740 2019-12-05T05:12:44.779 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:44.768+0000 7f1f8288a700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f7c08e5b0 con 0x7f1f7c09fae0 2019-12-05T05:12:44.779 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:44.768+0000 7f1f80e27700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f1f7c0a7740 0x7f1f7c08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:12:44.780 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:44.768+0000 7f1f7bfff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f1f7c07e720 0x7f1f7c09f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:57972/0 (socket says 172.21.15.42:57972) 2019-12-05T05:12:44.780 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:44.768+0000 7f1f7bfff700 1 -- 172.21.15.42:0/2309501448 learned_addr learned my addr 172.21.15.42:0/2309501448 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:12:44.780 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:44.768+0000 7f1f81628700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1f7c09fae0 0x7f1f7c07eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:12:44.780 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:44.768+0000 7f1f7b7fe700 1 -- 172.21.15.42:0/2309501448 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1338929777 0 0) 0x7f1f7c0a0710 con 0x7f1f7c07e720 2019-12-05T05:12:44.780 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:44.768+0000 7f1f7b7fe700 1 -- 172.21.15.42:0/2309501448 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f640030b0 con 0x7f1f7c07e720 2019-12-05T05:12:44.780 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:44.768+0000 7f1f80e27700 1 -- 172.21.15.42:0/2309501448 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1f7c09fae0 msgr2=0x7f1f7c07eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:12:44.781 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:44.768+0000 7f1f80e27700 1 --2- 172.21.15.42:0/2309501448 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1f7c09fae0 0x7f1f7c07eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:12:44.781 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:44.768+0000 7f1f80e27700 1 -- 172.21.15.42:0/2309501448 >> v1:172.21.15.42:6789/0 conn(0x7f1f7c07e720 legacy=0x7f1f7c09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:12:44.781 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:44.768+0000 7f1f80e27700 1 -- 172.21.15.42:0/2309501448 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f7c07f0a0 con 0x7f1f7c0a7740 2019-12-05T05:12:44.781 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:44.768+0000 7f1f80e27700 1 --2- 172.21.15.42:0/2309501448 >> v2:172.21.15.42:3300/0 conn(0x7f1f7c0a7740 0x7f1f7c08df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f1f70008790 tx=0x7f1f7000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:44.781 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:44.768+0000 7f1f80e27700 1 --2- 172.21.15.42:0/2309501448 >> v2:172.21.15.42:3300/0 conn(0x7f1f7c0a7740 0x7f1f7c08df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f1f70008790 tx=0x7f1f7000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:44.781 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:44.768+0000 7f1f80e27700 1 --2- 172.21.15.42:0/2309501448 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1f7c0a7740 0x7f1f7c08df80 secure :-1 s=READY pgs=471 cs=0 l=1 rx=0x7f1f70008790 tx=0x7f1f7000fc70).ready entity=mon.1 client_cookie=52a26afc7fb37dd9 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:12:44.781 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:44.768+0000 7f1f80e27700 1 --2- 172.21.15.42:0/2309501448 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1f7c0a7740 0x7f1f7c08df80 secure :-1 s=THROTTLE_DONE pgs=471 cs=0 l=1 rx=0x7f1f70008790 tx=0x7f1f7000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:44.782 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:44.772+0000 7f1f7b7fe700 1 -- 172.21.15.42:0/2309501448 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1f70008bb0 con 0x7f1f7c0a7740 2019-12-05T05:12:44.783 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:44.772+0000 7f1f80e27700 1 --2- 172.21.15.42:0/2309501448 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1f7c0a7740 0x7f1f7c08df80 secure :-1 s=THROTTLE_DONE pgs=471 cs=0 l=1 rx=0x7f1f70008790 tx=0x7f1f7000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:44.783 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:44.772+0000 7f1f80e27700 1 --2- 172.21.15.42:0/2309501448 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1f7c0a7740 0x7f1f7c08df80 secure :-1 s=THROTTLE_DONE pgs=471 cs=0 l=1 rx=0x7f1f70008790 tx=0x7f1f7000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:44.783 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:44.772+0000 7f1f7b7fe700 1 -- 172.21.15.42:0/2309501448 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f700087d0 con 0x7f1f7c0a7740 2019-12-05T05:12:44.783 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:44.772+0000 7f1f7b7fe700 1 -- 172.21.15.42:0/2309501448 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1f70011860 con 0x7f1f7c0a7740 2019-12-05T05:12:44.783 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:44.772+0000 7f1f8288a700 1 -- 172.21.15.42:0/2309501448 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1f7c0a7740 msgr2=0x7f1f7c08df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:12:44.784 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:44.772+0000 7f1f8288a700 1 --2- 172.21.15.42:0/2309501448 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1f7c0a7740 0x7f1f7c08df80 secure :-1 s=READY pgs=471 cs=0 l=1 rx=0x7f1f70008790 tx=0x7f1f7000fc70).stop 2019-12-05T05:12:44.784 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:44.772+0000 7f1f8288a700 1 -- 172.21.15.42:0/2309501448 shutdown_connections 2019-12-05T05:12:44.784 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:44.772+0000 7f1f8288a700 1 --2- 172.21.15.42:0/2309501448 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1f7c09fae0 0x7f1f7c07eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:12:44.784 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:44.772+0000 7f1f8288a700 1 --2- 172.21.15.42:0/2309501448 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1f7c0a7740 0x7f1f7c08df80 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:12:44.784 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:44.772+0000 7f1f8288a700 1 -- 172.21.15.42:0/2309501448 shutdown_connections 2019-12-05T05:12:44.784 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:44.772+0000 7f1f8288a700 1 -- 172.21.15.42:0/2309501448 wait complete. 2019-12-05T05:12:44.785 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:44.772+0000 7f1f8288a700 1 -- 172.21.15.42:0/2309501448 >> 172.21.15.42:0/2309501448 conn(0x7f1f7c0aabe0 msgr2=0x7f1f7c0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:12:44.788 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:44.776+0000 7f1f8288a700 1 Processor -- start 2019-12-05T05:12:44.788 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:44.776+0000 7f1f8288a700 1 -- start start 2019-12-05T05:12:44.788 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:44.776+0000 7f1f8288a700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f1f7c1233b0 0x7f1f7c1237e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:12:44.788 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:44.776+0000 7f1f8288a700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1f7c133610 0x7f1f7c133a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:12:44.788 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:44.776+0000 7f1f8288a700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f7c122290 con 0x7f1f7c121e60 2019-12-05T05:12:44.789 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:44.776+0000 7f1f8288a700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f7c1219b0 con 0x7f1f7c1233b0 2019-12-05T05:12:44.789 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:44.776+0000 7f1f8288a700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f7c123130 con 0x7f1f7c133610 2019-12-05T05:12:44.789 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:44.776+0000 7f1f81628700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1f7c133610 0x7f1f7c133a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:12:44.789 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:44.776+0000 7f1f7bfff700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f1f7c1233b0 0x7f1f7c1237e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:12:44.789 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:44.776+0000 7f1f81628700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1f7c133610 0x7f1f7c133a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:58408/0 (socket says 172.21.15.42:58408) 2019-12-05T05:12:44.790 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:44.776+0000 7f1f7bfff700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f1f7c1233b0 0x7f1f7c1237e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.42:45756/0 (socket says 172.21.15.42:45756) 2019-12-05T05:12:44.790 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:44.776+0000 7f1f81628700 1 -- 172.21.15.42:0/3085286591 learned_addr learned my addr 172.21.15.42:0/3085286591 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:12:44.790 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:44.776+0000 7f1f81628700 1 -- 172.21.15.42:0/3085286591 >> v2:172.21.15.40:3300/0 conn(0x7f1f7c1233b0 msgr2=0x7f1f7c1237e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:12:44.790 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:44.776+0000 7f1f81628700 1 --2- 172.21.15.42:0/3085286591 >> v2:172.21.15.40:3300/0 conn(0x7f1f7c1233b0 0x7f1f7c1237e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:12:44.790 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:44.776+0000 7f1f81628700 1 -- 172.21.15.42:0/3085286591 >> v1:172.21.15.40:6789/0 conn(0x7f1f7c121e60 legacy=0x7f1f7c123f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:12:44.790 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:44.776+0000 7f1f81628700 1 -- 172.21.15.42:0/3085286591 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f7c1196a0 con 0x7f1f7c133610 2019-12-05T05:12:44.791 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:44.776+0000 7f1f81628700 1 --2- 172.21.15.42:0/3085286591 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1f7c133610 0x7f1f7c133a40 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f1f7c118ee0 tx=0x7f1f74010d30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:44.791 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:44.776+0000 7f1f81628700 1 --2- 172.21.15.42:0/3085286591 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1f7c133610 0x7f1f7c133a40 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f1f7c118ee0 tx=0x7f1f74010d30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:44.791 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:44.776+0000 7f1f81628700 1 --2- 172.21.15.42:0/3085286591 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1f7c133610 0x7f1f7c133a40 secure :-1 s=READY pgs=94 cs=0 l=1 rx=0x7f1f7c118ee0 tx=0x7f1f74010d30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:12:44.791 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:44.776+0000 7f1f81628700 1 --2- 172.21.15.42:0/3085286591 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1f7c133610 0x7f1f7c133a40 secure :-1 s=THROTTLE_DONE pgs=94 cs=0 l=1 rx=0x7f1f7c118ee0 tx=0x7f1f74010d30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:44.791 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:44.776+0000 7f1f797fa700 1 -- 172.21.15.42:0/3085286591 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1f74012a20 con 0x7f1f7c133610 2019-12-05T05:12:44.791 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:44.776+0000 7f1f81628700 1 --2- 172.21.15.42:0/3085286591 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1f7c133610 0x7f1f7c133a40 secure :-1 s=THROTTLE_DONE pgs=94 cs=0 l=1 rx=0x7f1f7c118ee0 tx=0x7f1f74010d30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:44.792 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:44.776+0000 7f1f81628700 1 --2- 172.21.15.42:0/3085286591 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1f7c133610 0x7f1f7c133a40 secure :-1 s=THROTTLE_DONE pgs=94 cs=0 l=1 rx=0x7f1f7c118ee0 tx=0x7f1f74010d30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:44.793 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:44.776+0000 7f1f797fa700 1 -- 172.21.15.42:0/3085286591 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f74012640 con 0x7f1f7c133610 2019-12-05T05:12:44.793 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:44.776+0000 7f1f8288a700 1 -- 172.21.15.42:0/3085286591 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f7c133f90 con 0x7f1f7c133610 2019-12-05T05:12:44.793 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:44.776+0000 7f1f797fa700 1 -- 172.21.15.42:0/3085286591 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1f7400e570 con 0x7f1f7c133610 2019-12-05T05:12:44.793 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:44.776+0000 7f1f8288a700 1 -- 172.21.15.42:0/3085286591 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f7c077300 con 0x7f1f7c133610 2019-12-05T05:12:44.793 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:44.776+0000 7f1f81628700 1 --2- 172.21.15.42:0/3085286591 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1f7c133610 0x7f1f7c133a40 secure :-1 s=THROTTLE_DONE pgs=94 cs=0 l=1 rx=0x7f1f7c118ee0 tx=0x7f1f74010d30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:44.794 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:44.780+0000 7f1f797fa700 1 -- 172.21.15.42:0/3085286591 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f1f740100e0 con 0x7f1f7c133610 2019-12-05T05:12:44.794 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:44.780+0000 7f1f81628700 1 --2- 172.21.15.42:0/3085286591 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1f7c133610 0x7f1f7c133a40 secure :-1 s=THROTTLE_DONE pgs=94 cs=0 l=1 rx=0x7f1f7c118ee0 tx=0x7f1f74010d30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:44.794 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:44.780+0000 7f1f797fa700 1 --2- 172.21.15.42:0/3085286591 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f1f5c03ad80 0x7f1f5c03d290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:12:44.794 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:44.780+0000 7f1f797fa700 1 -- 172.21.15.42:0/3085286591 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6547+0+0 (secure 0 0 0) 0x7f1f7404b890 con 0x7f1f7c133610 2019-12-05T05:12:44.794 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:44.780+0000 7f1f80e27700 1 --2- 172.21.15.42:0/3085286591 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f1f5c03ad80 0x7f1f5c03d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:12:44.794 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:44.780+0000 7f1f80e27700 1 --2- 172.21.15.42:0/3085286591 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f1f5c03ad80 0x7f1f5c03d290 crc :-1 s=READY pgs=214 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:12:44.795 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:44.784+0000 7f1f8288a700 1 -- 172.21.15.42:0/3085286591 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f7c05d110 con 0x7f1f7c133610 2019-12-05T05:12:44.798 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:44.792+0000 7f1f81628700 1 --2- 172.21.15.42:0/3085286591 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1f7c133610 0x7f1f7c133a40 secure :-1 s=THROTTLE_DONE pgs=94 cs=0 l=1 rx=0x7f1f7c118ee0 tx=0x7f1f74010d30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:44.798 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:44.792+0000 7f1f797fa700 1 -- 172.21.15.42:0/3085286591 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f1f7c1196a0 con 0x7f1f7c133610 2019-12-05T05:12:44.966 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:44.960+0000 7f1f8288a700 1 -- 172.21.15.42:0/3085286591 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1f7c05b8c0 con 0x7f1f7c133610 2019-12-05T05:12:44.967 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:44.960+0000 7f1f81628700 1 --2- 172.21.15.42:0/3085286591 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1f7c133610 0x7f1f7c133a40 secure :-1 s=THROTTLE_DONE pgs=94 cs=0 l=1 rx=0x7f1f7c118ee0 tx=0x7f1f74010d30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:44.967 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:44.960+0000 7f1f797fa700 1 -- 172.21.15.42:0/3085286591 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) v1 ==== 94+0+1768 (secure 0 0 0) 0x7f1f740328f0 con 0x7f1f7c133610 2019-12-05T05:12:44.969 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:12:44.970 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":59,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":59,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:12:43.340063+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5391},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5391":{"gid":5391,"name":"b","rank":0,"incarnation":59,"state":"up:replay","state_seq":1,"addr":"172.21.15.40:6833/814397242","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":814397242},{"type":"v1","addr":"172.21.15.40:6833","nonce":814397242}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:12:44.970 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 59 2019-12-05T05:12:44.972 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:44.968+0000 7f1f8288a700 1 -- 172.21.15.42:0/3085286591 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f1f5c03ad80 msgr2=0x7f1f5c03d290 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:12:44.973 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:44.968+0000 7f1f8288a700 1 --2- 172.21.15.42:0/3085286591 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f1f5c03ad80 0x7f1f5c03d290 crc :-1 s=READY pgs=214 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:12:44.973 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:44.968+0000 7f1f8288a700 1 -- 172.21.15.42:0/3085286591 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1f7c133610 msgr2=0x7f1f7c133a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:12:44.973 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:44.968+0000 7f1f8288a700 1 --2- 172.21.15.42:0/3085286591 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1f7c133610 0x7f1f7c133a40 secure :-1 s=READY pgs=94 cs=0 l=1 rx=0x7f1f7c118ee0 tx=0x7f1f74010d30).stop 2019-12-05T05:12:44.973 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:44.968+0000 7f1f8288a700 1 -- 172.21.15.42:0/3085286591 shutdown_connections 2019-12-05T05:12:44.973 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:44.968+0000 7f1f8288a700 1 --2- 172.21.15.42:0/3085286591 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f1f5c03ad80 0x7f1f5c03d290 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:12:44.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:44.968+0000 7f1f8288a700 1 --2- 172.21.15.42:0/3085286591 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1f7c133610 0x7f1f7c133a40 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:12:44.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:44.968+0000 7f1f8288a700 1 --2- 172.21.15.42:0/3085286591 >> v2:172.21.15.40:3300/0 conn(0x7f1f7c1233b0 0x7f1f7c1237e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:12:44.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:44.968+0000 7f1f8288a700 1 -- 172.21.15.42:0/3085286591 shutdown_connections 2019-12-05T05:12:44.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:44.968+0000 7f1f8288a700 1 -- 172.21.15.42:0/3085286591 wait complete. 2019-12-05T05:12:44.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:44.968+0000 7f1f8288a700 1 -- 172.21.15.42:0/3085286591 >> 172.21.15.42:0/3085286591 conn(0x7f1f7c0aabe0 msgr2=0x7f1f7c0a9640 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:12:44.984 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a down, removed from mdsmap 2019-12-05T05:12:44.985 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2019-12-05T05:12:44.985 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:12:45.091 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:45.080+0000 7f36ad8cb700 1 Processor -- start 2019-12-05T05:12:45.091 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:45.080+0000 7f36ad8cb700 1 -- start start 2019-12-05T05:12:45.091 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:45.084+0000 7f36ad8cb700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f36a80a7730 0x7f36a808df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:12:45.092 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:45.084+0000 7f36ad8cb700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f36a807e710 0x7f36a807eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:12:45.092 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:45.084+0000 7f36ad8cb700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36a80a0700 con 0x7f36a809f2d0 2019-12-05T05:12:45.092 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:45.084+0000 7f36ad8cb700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36a80a0af0 con 0x7f36a80a7730 2019-12-05T05:12:45.092 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:45.084+0000 7f36ad8cb700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36a808e5a0 con 0x7f36a807e710 2019-12-05T05:12:45.092 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:45.084+0000 7f36a6ffd700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f36a807e710 0x7f36a807eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:12:45.093 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:45.084+0000 7f36a6ffd700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f36a807e710 0x7f36a807eb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:57946/0 (socket says 172.21.15.42:57946) 2019-12-05T05:12:45.093 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:45.084+0000 7f36a6ffd700 1 -- 172.21.15.42:0/2713913661 learned_addr learned my addr 172.21.15.42:0/2713913661 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:12:45.093 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:45.084+0000 7f36a77fe700 1 --2- 172.21.15.42:0/2713913661 >> v2:172.21.15.40:3300/0 conn(0x7f36a80a7730 0x7f36a808df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:12:45.093 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:45.084+0000 7f36a6ffd700 1 -- 172.21.15.42:0/2713913661 >> v2:172.21.15.40:3300/0 conn(0x7f36a80a7730 msgr2=0x7f36a808df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:12:45.093 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:45.084+0000 7f36a6ffd700 1 --2- 172.21.15.42:0/2713913661 >> v2:172.21.15.40:3300/0 conn(0x7f36a80a7730 0x7f36a808df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:12:45.093 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:45.084+0000 7f36a6ffd700 1 -- 172.21.15.42:0/2713913661 >> v1:172.21.15.40:6789/0 conn(0x7f36a809f2d0 legacy=0x7f36a809f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:12:45.093 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:45.084+0000 7f36a6ffd700 1 -- 172.21.15.42:0/2713913661 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36a807f090 con 0x7f36a807e710 2019-12-05T05:12:45.094 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:45.084+0000 7f36a77fe700 1 --2- 172.21.15.42:0/2713913661 >> v2:172.21.15.40:3300/0 conn(0x7f36a80a7730 0x7f36a808df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T05:12:45.094 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:45.084+0000 7f36a6ffd700 1 --2- 172.21.15.42:0/2713913661 >> v2:172.21.15.42:3300/0 conn(0x7f36a807e710 0x7f36a807eb40 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f3690002f30 tx=0x7f369000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:45.094 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:45.084+0000 7f36a6ffd700 1 --2- 172.21.15.42:0/2713913661 >> v2:172.21.15.42:3300/0 conn(0x7f36a807e710 0x7f36a807eb40 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f3690002f30 tx=0x7f369000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:45.094 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:45.084+0000 7f36a6ffd700 1 --2- 172.21.15.42:0/2713913661 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f36a807e710 0x7f36a807eb40 secure :-1 s=READY pgs=472 cs=0 l=1 rx=0x7f3690002f30 tx=0x7f369000fd30).ready entity=mon.1 client_cookie=eefe50eab8ef2a91 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:12:45.094 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:45.084+0000 7f36a6ffd700 1 --2- 172.21.15.42:0/2713913661 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f36a807e710 0x7f36a807eb40 secure :-1 s=THROTTLE_DONE pgs=472 cs=0 l=1 rx=0x7f3690002f30 tx=0x7f369000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:45.094 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:45.084+0000 7f36a67fc700 1 -- 172.21.15.42:0/2713913661 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3690011370 con 0x7f36a807e710 2019-12-05T05:12:45.094 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:45.084+0000 7f36a6ffd700 1 --2- 172.21.15.42:0/2713913661 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f36a807e710 0x7f36a807eb40 secure :-1 s=THROTTLE_DONE pgs=472 cs=0 l=1 rx=0x7f3690002f30 tx=0x7f369000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:45.095 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:45.084+0000 7f36a6ffd700 1 --2- 172.21.15.42:0/2713913661 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f36a807e710 0x7f36a807eb40 secure :-1 s=THROTTLE_DONE pgs=472 cs=0 l=1 rx=0x7f3690002f30 tx=0x7f369000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:45.095 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:45.084+0000 7f36a67fc700 1 -- 172.21.15.42:0/2713913661 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3690008bb0 con 0x7f36a807e710 2019-12-05T05:12:45.095 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:45.084+0000 7f36a67fc700 1 -- 172.21.15.42:0/2713913661 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3690011950 con 0x7f36a807e710 2019-12-05T05:12:45.095 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:45.084+0000 7f36ad8cb700 1 -- 172.21.15.42:0/2713913661 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f36a807e710 msgr2=0x7f36a807eb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:12:45.095 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:45.084+0000 7f36ad8cb700 1 --2- 172.21.15.42:0/2713913661 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f36a807e710 0x7f36a807eb40 secure :-1 s=READY pgs=472 cs=0 l=1 rx=0x7f3690002f30 tx=0x7f369000fd30).stop 2019-12-05T05:12:45.095 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:45.084+0000 7f36ad8cb700 1 -- 172.21.15.42:0/2713913661 shutdown_connections 2019-12-05T05:12:45.095 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:45.084+0000 7f36ad8cb700 1 --2- 172.21.15.42:0/2713913661 >> v2:172.21.15.40:3300/0 conn(0x7f36a80a7730 0x7f36a808df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:12:45.096 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:45.084+0000 7f36ad8cb700 1 --2- 172.21.15.42:0/2713913661 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f36a807e710 0x7f36a807eb40 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:12:45.096 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:45.084+0000 7f36ad8cb700 1 -- 172.21.15.42:0/2713913661 shutdown_connections 2019-12-05T05:12:45.096 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:45.084+0000 7f36ad8cb700 1 -- 172.21.15.42:0/2713913661 wait complete. 2019-12-05T05:12:45.096 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:45.084+0000 7f36ad8cb700 1 -- 172.21.15.42:0/2713913661 >> 172.21.15.42:0/2713913661 conn(0x7f36a80aabd0 msgr2=0x7f36a80a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:12:45.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:45.088+0000 7f36ad8cb700 1 Processor -- start 2019-12-05T05:12:45.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:45.088+0000 7f36ad8cb700 1 -- start start 2019-12-05T05:12:45.100 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:45.088+0000 7f36ad8cb700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f36a80a9aa0 0x7f36a8079340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:12:45.100 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:45.088+0000 7f36ad8cb700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36a81124d0 con 0x7f36a80a3d30 2019-12-05T05:12:45.100 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:45.088+0000 7f36ad8cb700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36a80a4330 con 0x7f36a8079890 2019-12-05T05:12:45.100 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:45.088+0000 7f36ad8cb700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36a8111f90 con 0x7f36a80a9aa0 2019-12-05T05:12:45.100 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:45.088+0000 7f36a7fff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f36a8079890 0x7f36a812f820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:57994/0 (socket says 172.21.15.42:57994) 2019-12-05T05:12:45.100 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:45.088+0000 7f36a7fff700 1 -- 172.21.15.42:0/898601570 learned_addr learned my addr 172.21.15.42:0/898601570 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:12:45.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:45.088+0000 7f36a6ffd700 1 --2- 172.21.15.42:0/898601570 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f36a80a9aa0 0x7f36a8079340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:12:45.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:45.092+0000 7f369ffff700 1 -- 172.21.15.42:0/898601570 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1281524367 0 0) 0x7f36a80a4330 con 0x7f36a8079890 2019-12-05T05:12:45.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:45.092+0000 7f369ffff700 1 -- 172.21.15.42:0/898601570 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f368c0030b0 con 0x7f36a8079890 2019-12-05T05:12:45.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:45.092+0000 7f369ffff700 1 -- 172.21.15.42:0/898601570 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 958056423 0 0) 0x7f36a81124d0 con 0x7f36a80a3d30 2019-12-05T05:12:45.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:45.092+0000 7f369ffff700 1 -- 172.21.15.42:0/898601570 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36a80a4330 con 0x7f36a80a3d30 2019-12-05T05:12:45.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:45.092+0000 7f36a6ffd700 1 -- 172.21.15.42:0/898601570 >> v1:172.21.15.42:6789/0 conn(0x7f36a8079890 legacy=0x7f36a812f820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:12:45.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:45.092+0000 7f36a6ffd700 1 -- 172.21.15.42:0/898601570 >> v1:172.21.15.40:6789/0 conn(0x7f36a80a3d30 legacy=0x7f36a8113270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:12:45.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:45.092+0000 7f36a6ffd700 1 -- 172.21.15.42:0/898601570 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36a8113a70 con 0x7f36a80a9aa0 2019-12-05T05:12:45.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:45.092+0000 7f36a6ffd700 1 --2- 172.21.15.42:0/898601570 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f36a80a9aa0 0x7f36a8079340 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f3690002f30 tx=0x7f3690011f10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:45.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:45.092+0000 7f36a6ffd700 1 --2- 172.21.15.42:0/898601570 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f36a80a9aa0 0x7f36a8079340 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f3690002f30 tx=0x7f3690011f10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:45.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:45.092+0000 7f36a6ffd700 1 --2- 172.21.15.42:0/898601570 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f36a80a9aa0 0x7f36a8079340 secure :-1 s=READY pgs=95 cs=0 l=1 rx=0x7f3690002f30 tx=0x7f3690011f10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:12:45.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:45.092+0000 7f36a6ffd700 1 --2- 172.21.15.42:0/898601570 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f36a80a9aa0 0x7f36a8079340 secure :-1 s=THROTTLE_DONE pgs=95 cs=0 l=1 rx=0x7f3690002f30 tx=0x7f3690011f10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:45.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:45.092+0000 7f369ffff700 1 -- 172.21.15.42:0/898601570 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3690007340 con 0x7f36a80a9aa0 2019-12-05T05:12:45.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:45.092+0000 7f36a6ffd700 1 --2- 172.21.15.42:0/898601570 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f36a80a9aa0 0x7f36a8079340 secure :-1 s=THROTTLE_DONE pgs=95 cs=0 l=1 rx=0x7f3690002f30 tx=0x7f3690011f10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:45.103 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:45.092+0000 7f369ffff700 1 -- 172.21.15.42:0/898601570 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3690011570 con 0x7f36a80a9aa0 2019-12-05T05:12:45.103 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:45.092+0000 7f36a6ffd700 1 --2- 172.21.15.42:0/898601570 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f36a80a9aa0 0x7f36a8079340 secure :-1 s=THROTTLE_DONE pgs=95 cs=0 l=1 rx=0x7f3690002f30 tx=0x7f3690011f10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:45.103 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:45.092+0000 7f36ad8cb700 1 -- 172.21.15.42:0/898601570 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36a8145140 con 0x7f36a80a9aa0 2019-12-05T05:12:45.103 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:45.092+0000 7f369ffff700 1 -- 172.21.15.42:0/898601570 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3690005540 con 0x7f36a80a9aa0 2019-12-05T05:12:45.103 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:45.092+0000 7f36ad8cb700 1 -- 172.21.15.42:0/898601570 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36a8079cc0 con 0x7f36a80a9aa0 2019-12-05T05:12:45.103 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:45.092+0000 7f36a6ffd700 1 --2- 172.21.15.42:0/898601570 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f36a80a9aa0 0x7f36a8079340 secure :-1 s=THROTTLE_DONE pgs=95 cs=0 l=1 rx=0x7f3690002f30 tx=0x7f3690011f10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:45.103 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:45.092+0000 7f369ffff700 1 -- 172.21.15.42:0/898601570 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f36900057e0 con 0x7f36a80a9aa0 2019-12-05T05:12:45.104 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:45.092+0000 7f36a6ffd700 1 --2- 172.21.15.42:0/898601570 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f36a80a9aa0 0x7f36a8079340 secure :-1 s=THROTTLE_DONE pgs=95 cs=0 l=1 rx=0x7f3690002f30 tx=0x7f3690011f10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:45.104 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:45.092+0000 7f369ffff700 1 --2- 172.21.15.42:0/898601570 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f368c03b380 0x7f368c03d890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:12:45.104 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:45.092+0000 7f369ffff700 1 -- 172.21.15.42:0/898601570 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6547+0+0 (secure 0 0 0) 0x7f369004b340 con 0x7f36a80a9aa0 2019-12-05T05:12:45.104 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:45.092+0000 7f36a77fe700 1 --2- 172.21.15.42:0/898601570 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f368c03b380 0x7f368c03d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:12:45.104 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:45.092+0000 7f36a77fe700 1 --2- 172.21.15.42:0/898601570 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f368c03b380 0x7f368c03d890 crc :-1 s=READY pgs=215 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:12:45.105 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:45.096+0000 7f36ad8cb700 1 -- 172.21.15.42:0/898601570 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36a805d100 con 0x7f36a80a9aa0 2019-12-05T05:12:45.112 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:45.108+0000 7f36a6ffd700 1 --2- 172.21.15.42:0/898601570 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f36a80a9aa0 0x7f36a8079340 secure :-1 s=THROTTLE_DONE pgs=95 cs=0 l=1 rx=0x7f3690002f30 tx=0x7f3690011f10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:45.112 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:45.108+0000 7f369ffff700 1 -- 172.21.15.42:0/898601570 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f36900323a0 con 0x7f36a80a9aa0 2019-12-05T05:12:45.282 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:45.276+0000 7f36ad8cb700 1 -- 172.21.15.42:0/898601570 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f36a805b8b0 con 0x7f36a80a9aa0 2019-12-05T05:12:45.283 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:45.276+0000 7f36a6ffd700 1 --2- 172.21.15.42:0/898601570 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f36a80a9aa0 0x7f36a8079340 secure :-1 s=THROTTLE_DONE pgs=95 cs=0 l=1 rx=0x7f3690002f30 tx=0x7f3690011f10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:45.283 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:45.280+0000 7f369ffff700 1 -- 172.21.15.42:0/898601570 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) v1 ==== 94+0+1768 (secure 0 0 0) 0x7f3690032600 con 0x7f36a80a9aa0 2019-12-05T05:12:45.286 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 59 2019-12-05T05:12:45.286 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:12:45.286 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":59,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":59,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:12:43.340063+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5391},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5391":{"gid":5391,"name":"b","rank":0,"incarnation":59,"state":"up:replay","state_seq":1,"addr":"172.21.15.40:6833/814397242","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":814397242},{"type":"v1","addr":"172.21.15.40:6833","nonce":814397242}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:12:45.288 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:45.284+0000 7f36ad8cb700 1 -- 172.21.15.42:0/898601570 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f368c03b380 msgr2=0x7f368c03d890 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:12:45.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:45.284+0000 7f36ad8cb700 1 --2- 172.21.15.42:0/898601570 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f368c03b380 0x7f368c03d890 crc :-1 s=READY pgs=215 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:12:45.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:45.284+0000 7f36ad8cb700 1 -- 172.21.15.42:0/898601570 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f36a80a9aa0 msgr2=0x7f36a8079340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:12:45.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:45.284+0000 7f36ad8cb700 1 --2- 172.21.15.42:0/898601570 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f36a80a9aa0 0x7f36a8079340 secure :-1 s=READY pgs=95 cs=0 l=1 rx=0x7f3690002f30 tx=0x7f3690011f10).stop 2019-12-05T05:12:45.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:45.284+0000 7f36ad8cb700 1 -- 172.21.15.42:0/898601570 shutdown_connections 2019-12-05T05:12:45.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:45.284+0000 7f36ad8cb700 1 --2- 172.21.15.42:0/898601570 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f368c03b380 0x7f368c03d890 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:12:45.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:45.284+0000 7f36ad8cb700 1 --2- 172.21.15.42:0/898601570 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f36a80a9aa0 0x7f36a8079340 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:12:45.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:45.284+0000 7f36ad8cb700 1 -- 172.21.15.42:0/898601570 shutdown_connections 2019-12-05T05:12:45.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:45.284+0000 7f36ad8cb700 1 -- 172.21.15.42:0/898601570 wait complete. 2019-12-05T05:12:45.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:45.284+0000 7f36ad8cb700 1 -- 172.21.15.42:0/898601570 >> 172.21.15.42:0/898601570 conn(0x7f36a80a7f80 msgr2=0x7f36a807cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:12:45.300 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 59, "feature_flags": { "enable_multiple": false, "ever_enabled_multiple": false }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2019-12-05T05:03:35.604738+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 59, "ever_allowed_features": 2, "explicitly_allowed_features": 2, "failed": [], "flags": 18, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5391": { "addr": "172.21.15.40:6833/814397242", "addrs": { "addrvec": [ { "addr": "172.21.15.40:6832", "nonce": 814397242, "type": "v2" }, { "addr": "172.21.15.40:6833", "nonce": 814397242, "type": "v1" } ] }, "export_targets": [], "features": 4540138292836696063, "flags": 0, "gid": 5391, "incarnation": 59, "name": "b", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 55, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 2, "min_compat_client": "0 (unknown)", "modified": "2019-12-05T05:12:43.340063+0000", "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5391 } } } ], "standbys": [] } 2019-12-05T05:12:47.301 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:12:47.411 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:47.400+0000 7f8aeaf7a700 1 Processor -- start 2019-12-05T05:12:47.412 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:47.400+0000 7f8aeaf7a700 1 -- start start 2019-12-05T05:12:47.412 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:47.404+0000 7f8aeaf7a700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f8ae40a7740 0x7f8ae408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:12:47.412 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:47.404+0000 7f8aeaf7a700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8ae407e720 0x7f8ae407eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:12:47.412 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:47.404+0000 7f8aeaf7a700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ae40a0710 con 0x7f8ae409f2e0 2019-12-05T05:12:47.412 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:47.404+0000 7f8aeaf7a700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ae40a0b00 con 0x7f8ae40a7740 2019-12-05T05:12:47.412 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:47.404+0000 7f8aeaf7a700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ae408e5b0 con 0x7f8ae407e720 2019-12-05T05:12:47.413 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:47.404+0000 7f8ae9517700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f8ae40a7740 0x7f8ae408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:12:47.413 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:47.404+0000 7f8ae9517700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f8ae40a7740 0x7f8ae408df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:57964/0 (socket says 172.21.15.42:57964) 2019-12-05T05:12:47.413 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:47.404+0000 7f8ae9517700 1 -- 172.21.15.42:0/4122390268 learned_addr learned my addr 172.21.15.42:0/4122390268 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:12:47.413 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:47.404+0000 7f8ae8d16700 1 --2- 172.21.15.42:0/4122390268 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8ae407e720 0x7f8ae407eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:12:47.413 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:47.404+0000 7f8ae9517700 1 -- 172.21.15.42:0/4122390268 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8ae407e720 msgr2=0x7f8ae407eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:12:47.413 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:47.404+0000 7f8ae9517700 1 --2- 172.21.15.42:0/4122390268 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8ae407e720 0x7f8ae407eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:12:47.414 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:47.404+0000 7f8ae9517700 1 -- 172.21.15.42:0/4122390268 >> v1:172.21.15.40:6789/0 conn(0x7f8ae409f2e0 legacy=0x7f8ae409f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:12:47.414 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:47.404+0000 7f8ae9517700 1 -- 172.21.15.42:0/4122390268 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ae407f0a0 con 0x7f8ae40a7740 2019-12-05T05:12:47.414 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:47.404+0000 7f8ae8d16700 1 --2- 172.21.15.42:0/4122390268 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8ae407e720 0x7f8ae407eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T05:12:47.414 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:47.404+0000 7f8ae9517700 1 --2- 172.21.15.42:0/4122390268 >> v2:172.21.15.42:3300/0 conn(0x7f8ae40a7740 0x7f8ae408df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f8ae0008790 tx=0x7f8ae000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:47.414 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:47.404+0000 7f8ae9517700 1 --2- 172.21.15.42:0/4122390268 >> v2:172.21.15.42:3300/0 conn(0x7f8ae40a7740 0x7f8ae408df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f8ae0008790 tx=0x7f8ae000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:47.414 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:47.404+0000 7f8ae9517700 1 --2- 172.21.15.42:0/4122390268 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8ae40a7740 0x7f8ae408df80 secure :-1 s=READY pgs=474 cs=0 l=1 rx=0x7f8ae0008790 tx=0x7f8ae000fd30).ready entity=mon.1 client_cookie=e2567da9af51c9cc server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:12:47.414 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:47.404+0000 7f8ae9517700 1 --2- 172.21.15.42:0/4122390268 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8ae40a7740 0x7f8ae408df80 secure :-1 s=THROTTLE_DONE pgs=474 cs=0 l=1 rx=0x7f8ae0008790 tx=0x7f8ae000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:47.415 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:47.404+0000 7f8adbfff700 1 -- 172.21.15.42:0/4122390268 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8ae00113b0 con 0x7f8ae40a7740 2019-12-05T05:12:47.415 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:47.404+0000 7f8ae9517700 1 --2- 172.21.15.42:0/4122390268 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8ae40a7740 0x7f8ae408df80 secure :-1 s=THROTTLE_DONE pgs=474 cs=0 l=1 rx=0x7f8ae0008790 tx=0x7f8ae000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:47.415 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:47.404+0000 7f8adbfff700 1 -- 172.21.15.42:0/4122390268 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ae0008bb0 con 0x7f8ae40a7740 2019-12-05T05:12:47.415 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:47.404+0000 7f8ae9517700 1 --2- 172.21.15.42:0/4122390268 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8ae40a7740 0x7f8ae408df80 secure :-1 s=THROTTLE_DONE pgs=474 cs=0 l=1 rx=0x7f8ae0008790 tx=0x7f8ae000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:47.415 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:47.404+0000 7f8adbfff700 1 -- 172.21.15.42:0/4122390268 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8ae0011990 con 0x7f8ae40a7740 2019-12-05T05:12:47.415 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:47.404+0000 7f8aeaf7a700 1 -- 172.21.15.42:0/4122390268 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8ae40a7740 msgr2=0x7f8ae408df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:12:47.415 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:47.404+0000 7f8aeaf7a700 1 --2- 172.21.15.42:0/4122390268 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8ae40a7740 0x7f8ae408df80 secure :-1 s=READY pgs=474 cs=0 l=1 rx=0x7f8ae0008790 tx=0x7f8ae000fd30).stop 2019-12-05T05:12:47.416 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:47.404+0000 7f8aeaf7a700 1 -- 172.21.15.42:0/4122390268 shutdown_connections 2019-12-05T05:12:47.416 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:47.404+0000 7f8aeaf7a700 1 --2- 172.21.15.42:0/4122390268 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8ae40a7740 0x7f8ae408df80 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:12:47.416 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:47.404+0000 7f8aeaf7a700 1 --2- 172.21.15.42:0/4122390268 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8ae407e720 0x7f8ae407eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:12:47.416 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:47.404+0000 7f8aeaf7a700 1 -- 172.21.15.42:0/4122390268 shutdown_connections 2019-12-05T05:12:47.416 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:47.404+0000 7f8aeaf7a700 1 -- 172.21.15.42:0/4122390268 wait complete. 2019-12-05T05:12:47.416 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:47.404+0000 7f8aeaf7a700 1 -- 172.21.15.42:0/4122390268 >> 172.21.15.42:0/4122390268 conn(0x7f8ae40aabe0 msgr2=0x7f8ae40a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:12:47.419 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:47.408+0000 7f8aeaf7a700 1 Processor -- start 2019-12-05T05:12:47.420 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:47.408+0000 7f8aeaf7a700 1 -- start start 2019-12-05T05:12:47.420 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:47.408+0000 7f8aeaf7a700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8ae408eeb0 0x7f8ae40658e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:12:47.420 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:47.408+0000 7f8aeaf7a700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8ae40a6590 0x7f8ae4063960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:12:47.420 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:47.408+0000 7f8aeaf7a700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ae40a69c0 con 0x7f8ae4063eb0 2019-12-05T05:12:47.420 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:47.408+0000 7f8aeaf7a700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ae4063700 con 0x7f8ae408eeb0 2019-12-05T05:12:47.420 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:47.408+0000 7f8aeaf7a700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ae40a6ca0 con 0x7f8ae40a6590 2019-12-05T05:12:47.421 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:47.408+0000 7f8ae9517700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8ae408eeb0 0x7f8ae40658e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:12:47.421 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:47.408+0000 7f8ae8d16700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8ae40a6590 0x7f8ae4063960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:12:47.421 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:47.408+0000 7f8ae9517700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8ae408eeb0 0x7f8ae40658e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.42:45786/0 (socket says 172.21.15.42:45786) 2019-12-05T05:12:47.421 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:47.408+0000 7f8ae9517700 1 -- 172.21.15.42:0/2909913361 learned_addr learned my addr 172.21.15.42:0/2909913361 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:12:47.421 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:47.408+0000 7f8ae9d18700 1 --1- 172.21.15.42:0/2909913361 >> v1:172.21.15.40:6789/0 conn(0x7f8ae4063eb0 0x7f8ae412a540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:39220/0 (socket says 172.21.15.42:39220) 2019-12-05T05:12:47.421 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:47.408+0000 7f8ae8d16700 1 -- 172.21.15.42:0/2909913361 >> v2:172.21.15.40:3300/0 conn(0x7f8ae408eeb0 msgr2=0x7f8ae40658e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:12:47.421 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:47.408+0000 7f8ae8d16700 1 --2- 172.21.15.42:0/2909913361 >> v2:172.21.15.40:3300/0 conn(0x7f8ae408eeb0 0x7f8ae40658e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:12:47.422 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:47.408+0000 7f8ae8d16700 1 -- 172.21.15.42:0/2909913361 >> v1:172.21.15.40:6789/0 conn(0x7f8ae4063eb0 legacy=0x7f8ae412a540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:12:47.422 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:47.408+0000 7f8ad9ffb700 1 -- 172.21.15.42:0/2909913361 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 583356645 0 0) 0x7f8ae40a0710 con 0x7f8ae4063eb0 2019-12-05T05:12:47.422 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:47.408+0000 7f8ae8d16700 1 -- 172.21.15.42:0/2909913361 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ae40642e0 con 0x7f8ae40a6590 2019-12-05T05:12:47.422 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:47.408+0000 7f8ae9517700 1 --2- 172.21.15.42:0/2909913361 >> v2:172.21.15.40:3300/0 conn(0x7f8ae408eeb0 0x7f8ae40658e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:12:47.422 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:47.408+0000 7f8ae8d16700 1 --2- 172.21.15.42:0/2909913361 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8ae40a6590 0x7f8ae4063960 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f8ad4002f30 tx=0x7f8ad4010980).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:47.422 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:47.408+0000 7f8ae8d16700 1 --2- 172.21.15.42:0/2909913361 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8ae40a6590 0x7f8ae4063960 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f8ad4002f30 tx=0x7f8ad4010980).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:47.422 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:47.408+0000 7f8ae8d16700 1 --2- 172.21.15.42:0/2909913361 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8ae40a6590 0x7f8ae4063960 secure :-1 s=READY pgs=96 cs=0 l=1 rx=0x7f8ad4002f30 tx=0x7f8ad4010980).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:12:47.423 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:47.408+0000 7f8ae8d16700 1 --2- 172.21.15.42:0/2909913361 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8ae40a6590 0x7f8ae4063960 secure :-1 s=THROTTLE_DONE pgs=96 cs=0 l=1 rx=0x7f8ad4002f30 tx=0x7f8ad4010980).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:47.423 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:47.408+0000 7f8ad9ffb700 1 -- 172.21.15.42:0/2909913361 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8ad4005930 con 0x7f8ae40a6590 2019-12-05T05:12:47.423 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:47.408+0000 7f8ae8d16700 1 --2- 172.21.15.42:0/2909913361 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8ae40a6590 0x7f8ae4063960 secure :-1 s=THROTTLE_DONE pgs=96 cs=0 l=1 rx=0x7f8ad4002f30 tx=0x7f8ad4010980).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:47.423 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:47.408+0000 7f8ae8d16700 1 --2- 172.21.15.42:0/2909913361 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8ae40a6590 0x7f8ae4063960 secure :-1 s=THROTTLE_DONE pgs=96 cs=0 l=1 rx=0x7f8ad4002f30 tx=0x7f8ad4010980).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:47.423 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:47.408+0000 7f8ad9ffb700 1 -- 172.21.15.42:0/2909913361 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ad4007160 con 0x7f8ae40a6590 2019-12-05T05:12:47.423 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:47.408+0000 7f8aeaf7a700 1 -- 172.21.15.42:0/2909913361 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ae412ad40 con 0x7f8ae40a6590 2019-12-05T05:12:47.423 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:47.408+0000 7f8ad9ffb700 1 -- 172.21.15.42:0/2909913361 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8ad40060c0 con 0x7f8ae40a6590 2019-12-05T05:12:47.424 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:47.408+0000 7f8aeaf7a700 1 -- 172.21.15.42:0/2909913361 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ae412b2f0 con 0x7f8ae40a6590 2019-12-05T05:12:47.424 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:47.412+0000 7f8ae8d16700 1 --2- 172.21.15.42:0/2909913361 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8ae40a6590 0x7f8ae4063960 secure :-1 s=THROTTLE_DONE pgs=96 cs=0 l=1 rx=0x7f8ad4002f30 tx=0x7f8ad4010980).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:47.424 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:47.412+0000 7f8ae8d16700 1 --2- 172.21.15.42:0/2909913361 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8ae40a6590 0x7f8ae4063960 secure :-1 s=THROTTLE_DONE pgs=96 cs=0 l=1 rx=0x7f8ad4002f30 tx=0x7f8ad4010980).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:47.424 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:47.412+0000 7f8ad9ffb700 1 -- 172.21.15.42:0/2909913361 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f8ad4007be0 con 0x7f8ae40a6590 2019-12-05T05:12:47.424 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:47.412+0000 7f8ad9ffb700 1 --2- 172.21.15.42:0/2909913361 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8ac403ad20 0x7f8ac403d230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:12:47.424 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:47.412+0000 7f8ad9ffb700 1 -- 172.21.15.42:0/2909913361 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6547+0+0 (secure 0 0 0) 0x7f8ad404abe0 con 0x7f8ae40a6590 2019-12-05T05:12:47.424 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:47.412+0000 7f8ae9517700 1 --2- 172.21.15.42:0/2909913361 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8ac403ad20 0x7f8ac403d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:12:47.425 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:47.412+0000 7f8ae9517700 1 --2- 172.21.15.42:0/2909913361 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8ac403ad20 0x7f8ac403d230 crc :-1 s=READY pgs=216 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:12:47.425 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:47.420+0000 7f8aeaf7a700 1 -- 172.21.15.42:0/2909913361 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ae405d110 con 0x7f8ae40a6590 2019-12-05T05:12:47.434 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:47.424+0000 7f8ae8d16700 1 --2- 172.21.15.42:0/2909913361 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8ae40a6590 0x7f8ae4063960 secure :-1 s=THROTTLE_DONE pgs=96 cs=0 l=1 rx=0x7f8ad4002f30 tx=0x7f8ad4010980).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:47.434 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:47.424+0000 7f8ad9ffb700 1 -- 172.21.15.42:0/2909913361 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f8ad4031c20 con 0x7f8ae40a6590 2019-12-05T05:12:47.595 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:47.588+0000 7f8aeaf7a700 1 -- 172.21.15.42:0/2909913361 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8ae405b8c0 con 0x7f8ae40a6590 2019-12-05T05:12:47.596 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:47.592+0000 7f8ae8d16700 1 --2- 172.21.15.42:0/2909913361 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8ae40a6590 0x7f8ae4063960 secure :-1 s=THROTTLE_DONE pgs=96 cs=0 l=1 rx=0x7f8ad4002f30 tx=0x7f8ad4010980).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:47.596 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:47.592+0000 7f8ad9ffb700 1 -- 172.21.15.42:0/2909913361 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) v1 ==== 94+0+1769 (secure 0 0 0) 0x7f8ad4031e80 con 0x7f8ae40a6590 2019-12-05T05:12:47.598 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:12:47.598 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":61,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:12:47.362915+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5391},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5391":{"gid":5391,"name":"b","rank":0,"incarnation":59,"state":"up:rejoin","state_seq":22,"addr":"172.21.15.40:6833/814397242","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":814397242},{"type":"v1","addr":"172.21.15.40:6833","nonce":814397242}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:12:47.599 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 61 2019-12-05T05:12:47.601 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:47.596+0000 7f8aeaf7a700 1 -- 172.21.15.42:0/2909913361 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8ac403ad20 msgr2=0x7f8ac403d230 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:12:47.601 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:47.596+0000 7f8aeaf7a700 1 --2- 172.21.15.42:0/2909913361 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8ac403ad20 0x7f8ac403d230 crc :-1 s=READY pgs=216 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:12:47.602 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:47.596+0000 7f8aeaf7a700 1 -- 172.21.15.42:0/2909913361 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8ae40a6590 msgr2=0x7f8ae4063960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:12:47.602 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:47.596+0000 7f8aeaf7a700 1 --2- 172.21.15.42:0/2909913361 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8ae40a6590 0x7f8ae4063960 secure :-1 s=READY pgs=96 cs=0 l=1 rx=0x7f8ad4002f30 tx=0x7f8ad4010980).stop 2019-12-05T05:12:47.602 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:47.596+0000 7f8aeaf7a700 1 -- 172.21.15.42:0/2909913361 shutdown_connections 2019-12-05T05:12:47.602 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:47.596+0000 7f8aeaf7a700 1 --2- 172.21.15.42:0/2909913361 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8ac403ad20 0x7f8ac403d230 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:12:47.602 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:47.596+0000 7f8aeaf7a700 1 --2- 172.21.15.42:0/2909913361 >> v2:172.21.15.40:3300/0 conn(0x7f8ae408eeb0 0x7f8ae40658e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:12:47.602 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:47.596+0000 7f8aeaf7a700 1 --2- 172.21.15.42:0/2909913361 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8ae40a6590 0x7f8ae4063960 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:12:47.603 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:47.596+0000 7f8aeaf7a700 1 -- 172.21.15.42:0/2909913361 shutdown_connections 2019-12-05T05:12:47.603 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:47.596+0000 7f8aeaf7a700 1 -- 172.21.15.42:0/2909913361 wait complete. 2019-12-05T05:12:47.603 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:47.596+0000 7f8aeaf7a700 1 -- 172.21.15.42:0/2909913361 >> 172.21.15.42:0/2909913361 conn(0x7f8ae40a7740 msgr2=0x7f8ae40a9640 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:12:47.612 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2019-12-05T05:12:49.612 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:12:49.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:49.704+0000 7f59d57a7700 1 Processor -- start 2019-12-05T05:12:49.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:49.704+0000 7f59d57a7700 1 -- start start 2019-12-05T05:12:49.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:49.708+0000 7f59d57a7700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f59d008df80 0x7f59d009f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:12:49.718 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:49.708+0000 7f59d57a7700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59d009fc60 con 0x7f59d00a7740 2019-12-05T05:12:49.718 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:49.708+0000 7f59d57a7700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59d00a0710 con 0x7f59d009f830 2019-12-05T05:12:49.718 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:49.708+0000 7f59d57a7700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59d00a0b00 con 0x7f59d008df80 2019-12-05T05:12:49.718 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:49.708+0000 7f59ceffd700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f59d008df80 0x7f59d009f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:12:49.718 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:49.708+0000 7f59cffff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f59d009f830 0x7f59d014efe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:58020/0 (socket says 172.21.15.42:58020) 2019-12-05T05:12:49.718 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:49.708+0000 7f59cffff700 1 -- 172.21.15.42:0/279973536 learned_addr learned my addr 172.21.15.42:0/279973536 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:12:49.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:49.708+0000 7f59ce7fc700 1 -- 172.21.15.42:0/279973536 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 132946568 0 0) 0x7f59d00a0710 con 0x7f59d009f830 2019-12-05T05:12:49.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:49.708+0000 7f59ce7fc700 1 -- 172.21.15.42:0/279973536 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59b80030b0 con 0x7f59d009f830 2019-12-05T05:12:49.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:49.708+0000 7f59ceffd700 1 -- 172.21.15.42:0/279973536 >> v1:172.21.15.42:6789/0 conn(0x7f59d009f830 legacy=0x7f59d014efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:12:49.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:49.708+0000 7f59ceffd700 1 -- 172.21.15.42:0/279973536 >> v1:172.21.15.40:6789/0 conn(0x7f59d00a7740 legacy=0x7f59d007e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:12:49.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:49.708+0000 7f59ce7fc700 1 -- 172.21.15.42:0/279973536 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 212596372 0 0) 0x7f59d009fc60 con 0x7f59d00a7740 2019-12-05T05:12:49.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:49.708+0000 7f59ceffd700 1 -- 172.21.15.42:0/279973536 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59d007ef20 con 0x7f59d008df80 2019-12-05T05:12:49.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:49.708+0000 7f59ceffd700 1 --2- 172.21.15.42:0/279973536 >> v2:172.21.15.40:3300/0 conn(0x7f59d008df80 0x7f59d009f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f59c4008790 tx=0x7f59c400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:49.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:49.708+0000 7f59ceffd700 1 --2- 172.21.15.42:0/279973536 >> v2:172.21.15.40:3300/0 conn(0x7f59d008df80 0x7f59d009f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f59c4008790 tx=0x7f59c400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:49.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:49.708+0000 7f59ceffd700 1 --2- 172.21.15.42:0/279973536 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f59d008df80 0x7f59d009f2e0 secure :-1 s=READY pgs=375 cs=0 l=1 rx=0x7f59c4008790 tx=0x7f59c400fbf0).ready entity=mon.0 client_cookie=73719d7e827100ba server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:12:49.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:49.708+0000 7f59ceffd700 1 --2- 172.21.15.42:0/279973536 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f59d008df80 0x7f59d009f2e0 secure :-1 s=THROTTLE_DONE pgs=375 cs=0 l=1 rx=0x7f59c4008790 tx=0x7f59c400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:49.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:49.708+0000 7f59ce7fc700 1 -- 172.21.15.42:0/279973536 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f59c4011270 con 0x7f59d008df80 2019-12-05T05:12:49.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:49.708+0000 7f59ceffd700 1 --2- 172.21.15.42:0/279973536 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f59d008df80 0x7f59d009f2e0 secure :-1 s=THROTTLE_DONE pgs=375 cs=0 l=1 rx=0x7f59c4008790 tx=0x7f59c400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:49.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:49.708+0000 7f59ceffd700 1 --2- 172.21.15.42:0/279973536 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f59d008df80 0x7f59d009f2e0 secure :-1 s=THROTTLE_DONE pgs=375 cs=0 l=1 rx=0x7f59c4008790 tx=0x7f59c400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:49.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:49.708+0000 7f59ce7fc700 1 -- 172.21.15.42:0/279973536 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59c4008bc0 con 0x7f59d008df80 2019-12-05T05:12:49.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:49.708+0000 7f59ce7fc700 1 -- 172.21.15.42:0/279973536 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f59c4011850 con 0x7f59d008df80 2019-12-05T05:12:49.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:49.708+0000 7f59d57a7700 1 -- 172.21.15.42:0/279973536 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f59d008df80 msgr2=0x7f59d009f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:12:49.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:49.708+0000 7f59d57a7700 1 --2- 172.21.15.42:0/279973536 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f59d008df80 0x7f59d009f2e0 secure :-1 s=READY pgs=375 cs=0 l=1 rx=0x7f59c4008790 tx=0x7f59c400fbf0).stop 2019-12-05T05:12:49.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:49.712+0000 7f59d57a7700 1 -- 172.21.15.42:0/279973536 shutdown_connections 2019-12-05T05:12:49.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:49.712+0000 7f59d57a7700 1 --2- 172.21.15.42:0/279973536 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f59d008df80 0x7f59d009f2e0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:12:49.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:49.712+0000 7f59d57a7700 1 -- 172.21.15.42:0/279973536 shutdown_connections 2019-12-05T05:12:49.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:49.712+0000 7f59d57a7700 1 -- 172.21.15.42:0/279973536 wait complete. 2019-12-05T05:12:49.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:49.712+0000 7f59d57a7700 1 -- 172.21.15.42:0/279973536 >> 172.21.15.42:0/279973536 conn(0x7f59d00aabe0 msgr2=0x7f59d00a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:12:49.725 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:49.712+0000 7f59d57a7700 1 Processor -- start 2019-12-05T05:12:49.725 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:49.712+0000 7f59d57a7700 1 -- start start 2019-12-05T05:12:49.725 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:49.716+0000 7f59d57a7700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f59d011a050 0x7f59d012dd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:12:49.726 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:49.716+0000 7f59d57a7700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59d011bac0 con 0x7f59d011a480 2019-12-05T05:12:49.726 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:49.716+0000 7f59d57a7700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59d01178b0 con 0x7f59d00a9ab0 2019-12-05T05:12:49.726 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:49.716+0000 7f59d57a7700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59d011a990 con 0x7f59d011a050 2019-12-05T05:12:49.726 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:49.716+0000 7f59cf7fe700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f59d00a9ab0 0x7f59d01419d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:58022/0 (socket says 172.21.15.42:58022) 2019-12-05T05:12:49.726 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:49.716+0000 7f59cf7fe700 1 -- 172.21.15.42:0/637562611 learned_addr learned my addr 172.21.15.42:0/637562611 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:12:49.726 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:49.716+0000 7f59ceffd700 1 --2- 172.21.15.42:0/637562611 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f59d011a050 0x7f59d012dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:12:49.726 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:49.716+0000 7f59cffff700 1 -- 172.21.15.42:0/637562611 >> v1:172.21.15.40:6789/0 conn(0x7f59d011a480 legacy=0x7f59d012e2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2019-12-05T05:12:49.727 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:49.716+0000 7f59cffff700 1 -- 172.21.15.42:0/637562611 >> v1:172.21.15.40:6789/0 conn(0x7f59d011a480 legacy=0x7f59d012e2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2019-12-05T05:12:49.727 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:49.716+0000 7f59cffff700 1 --1- 172.21.15.42:0/637562611 >> v1:172.21.15.40:6789/0 conn(0x7f59d011a480 0x7f59d012e2d0 :-1 s=OPENED pgs=376 cs=1 l=1).handle_message read tag failed 2019-12-05T05:12:49.727 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:49.716+0000 7f59cffff700 1 --1- 172.21.15.42:0/637562611 >> v1:172.21.15.40:6789/0 conn(0x7f59d011a480 0x7f59d012e2d0 :-1 s=OPENED pgs=376 cs=1 l=1).fault on lossy channel, failing 2019-12-05T05:12:49.727 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:49.716+0000 7f59b7fff700 1 -- 172.21.15.42:0/637562611 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 212384222 0 0) 0x7f59d01178b0 con 0x7f59d00a9ab0 2019-12-05T05:12:49.727 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:49.716+0000 7f59b7fff700 1 -- 172.21.15.42:0/637562611 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59b00030b0 con 0x7f59d00a9ab0 2019-12-05T05:12:49.727 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:49.716+0000 7f59ceffd700 1 -- 172.21.15.42:0/637562611 >> v1:172.21.15.42:6789/0 conn(0x7f59d00a9ab0 legacy=0x7f59d01419d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:12:49.727 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:49.716+0000 7f59ceffd700 1 -- 172.21.15.42:0/637562611 >> v1:172.21.15.40:6789/0 conn(0x7f59d011a480 legacy=0x7f59d012e2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2019-12-05T05:12:49.728 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:49.716+0000 7f59ceffd700 1 -- 172.21.15.42:0/637562611 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59d0117ae0 con 0x7f59d011a050 2019-12-05T05:12:49.728 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:49.716+0000 7f59ceffd700 1 --2- 172.21.15.42:0/637562611 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f59d011a050 0x7f59d012dd80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f59c4008790 tx=0x7f59c4008e70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:49.728 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:49.716+0000 7f59ceffd700 1 --2- 172.21.15.42:0/637562611 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f59d011a050 0x7f59d012dd80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f59c4008790 tx=0x7f59c4008e70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:49.728 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:49.716+0000 7f59ceffd700 1 --2- 172.21.15.42:0/637562611 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f59d011a050 0x7f59d012dd80 secure :-1 s=READY pgs=97 cs=0 l=1 rx=0x7f59c4008790 tx=0x7f59c4008e70).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:12:49.728 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:49.716+0000 7f59ceffd700 1 --2- 172.21.15.42:0/637562611 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f59d011a050 0x7f59d012dd80 secure :-1 s=THROTTLE_DONE pgs=97 cs=0 l=1 rx=0x7f59c4008790 tx=0x7f59c4008e70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:49.728 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:49.716+0000 7f59b7fff700 1 -- 172.21.15.42:0/637562611 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f59c4007340 con 0x7f59d011a050 2019-12-05T05:12:49.728 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:49.716+0000 7f59ceffd700 1 --2- 172.21.15.42:0/637562611 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f59d011a050 0x7f59d012dd80 secure :-1 s=THROTTLE_DONE pgs=97 cs=0 l=1 rx=0x7f59c4008790 tx=0x7f59c4008e70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:49.729 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:49.716+0000 7f59b7fff700 1 -- 172.21.15.42:0/637562611 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59c4011470 con 0x7f59d011a050 2019-12-05T05:12:49.729 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:49.716+0000 7f59ceffd700 1 --2- 172.21.15.42:0/637562611 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f59d011a050 0x7f59d012dd80 secure :-1 s=THROTTLE_DONE pgs=97 cs=0 l=1 rx=0x7f59c4008790 tx=0x7f59c4008e70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:49.729 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:49.716+0000 7f59d57a7700 1 -- 172.21.15.42:0/637562611 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59d0117db0 con 0x7f59d011a050 2019-12-05T05:12:49.729 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:49.716+0000 7f59b7fff700 1 -- 172.21.15.42:0/637562611 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f59c4005540 con 0x7f59d011a050 2019-12-05T05:12:49.729 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:49.716+0000 7f59d57a7700 1 -- 172.21.15.42:0/637562611 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59d01182b0 con 0x7f59d011a050 2019-12-05T05:12:49.729 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:49.716+0000 7f59ceffd700 1 --2- 172.21.15.42:0/637562611 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f59d011a050 0x7f59d012dd80 secure :-1 s=THROTTLE_DONE pgs=97 cs=0 l=1 rx=0x7f59c4008790 tx=0x7f59c4008e70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:49.729 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:49.716+0000 7f59b7fff700 1 -- 172.21.15.42:0/637562611 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f59c40057e0 con 0x7f59d011a050 2019-12-05T05:12:49.730 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:49.716+0000 7f59ceffd700 1 --2- 172.21.15.42:0/637562611 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f59d011a050 0x7f59d012dd80 secure :-1 s=THROTTLE_DONE pgs=97 cs=0 l=1 rx=0x7f59c4008790 tx=0x7f59c4008e70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:49.730 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:49.720+0000 7f59b7fff700 1 --2- 172.21.15.42:0/637562611 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f59b003b180 0x7f59b003d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:12:49.730 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:49.720+0000 7f59b7fff700 1 -- 172.21.15.42:0/637562611 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6547+0+0 (secure 0 0 0) 0x7f59c404ba80 con 0x7f59d011a050 2019-12-05T05:12:49.730 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:49.720+0000 7f59cf7fe700 1 --2- 172.21.15.42:0/637562611 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f59b003b180 0x7f59b003d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:12:49.730 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:49.720+0000 7f59cf7fe700 1 --2- 172.21.15.42:0/637562611 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f59b003b180 0x7f59b003d690 crc :-1 s=READY pgs=217 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:12:49.731 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:49.720+0000 7f59d57a7700 1 -- 172.21.15.42:0/637562611 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59d005d110 con 0x7f59d011a050 2019-12-05T05:12:49.735 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:49.732+0000 7f59ceffd700 1 --2- 172.21.15.42:0/637562611 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f59d011a050 0x7f59d012dd80 secure :-1 s=THROTTLE_DONE pgs=97 cs=0 l=1 rx=0x7f59c4008790 tx=0x7f59c4008e70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:49.736 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:49.732+0000 7f59b7fff700 1 -- 172.21.15.42:0/637562611 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f59c4032ae0 con 0x7f59d011a050 2019-12-05T05:12:49.905 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:49.900+0000 7f59d57a7700 1 -- 172.21.15.42:0/637562611 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f59d005b8c0 con 0x7f59d011a050 2019-12-05T05:12:49.906 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:49.900+0000 7f59ceffd700 1 --2- 172.21.15.42:0/637562611 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f59d011a050 0x7f59d012dd80 secure :-1 s=THROTTLE_DONE pgs=97 cs=0 l=1 rx=0x7f59c4008790 tx=0x7f59c4008e70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:49.906 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:49.900+0000 7f59b7fff700 1 -- 172.21.15.42:0/637562611 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) v1 ==== 94+0+1769 (secure 0 0 0) 0x7f59c4032d40 con 0x7f59d011a050 2019-12-05T05:12:49.906 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 62 2019-12-05T05:12:49.912 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:12:49.912 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":62,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":62,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:12:48.374585+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5391},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5391":{"gid":5391,"name":"b","rank":0,"incarnation":59,"state":"up:active","state_seq":23,"addr":"172.21.15.40:6833/814397242","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":814397242},{"type":"v1","addr":"172.21.15.40:6833","nonce":814397242}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:12:49.913 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:49.904+0000 7f59d57a7700 1 -- 172.21.15.42:0/637562611 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f59b003b180 msgr2=0x7f59b003d690 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:12:49.914 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:49.904+0000 7f59d57a7700 1 --2- 172.21.15.42:0/637562611 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f59b003b180 0x7f59b003d690 crc :-1 s=READY pgs=217 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:12:49.914 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:49.904+0000 7f59d57a7700 1 -- 172.21.15.42:0/637562611 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f59d011a050 msgr2=0x7f59d012dd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:12:49.914 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:49.904+0000 7f59d57a7700 1 --2- 172.21.15.42:0/637562611 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f59d011a050 0x7f59d012dd80 secure :-1 s=READY pgs=97 cs=0 l=1 rx=0x7f59c4008790 tx=0x7f59c4008e70).stop 2019-12-05T05:12:49.914 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:49.904+0000 7f59d57a7700 1 -- 172.21.15.42:0/637562611 shutdown_connections 2019-12-05T05:12:49.914 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:49.904+0000 7f59d57a7700 1 --2- 172.21.15.42:0/637562611 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f59b003b180 0x7f59b003d690 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:12:49.914 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:49.904+0000 7f59d57a7700 1 --2- 172.21.15.42:0/637562611 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f59d011a050 0x7f59d012dd80 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:12:49.915 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:49.904+0000 7f59d57a7700 1 -- 172.21.15.42:0/637562611 shutdown_connections 2019-12-05T05:12:49.915 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:49.904+0000 7f59d57a7700 1 -- 172.21.15.42:0/637562611 wait complete. 2019-12-05T05:12:49.915 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:49.904+0000 7f59d57a7700 1 -- 172.21.15.42:0/637562611 >> 172.21.15.42:0/637562611 conn(0x7f59d009f2c0 msgr2=0x7f59d0059ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:12:49.919 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b has gained rank=0, replacing gid=5215 2019-12-05T05:12:49.919 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 0 secs before reviving mds.a 2019-12-05T05:12:49.920 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.a 2019-12-05T05:12:49.920 INFO:tasks.ceph.mds.a:Restarting daemon 2019-12-05T05:12:49.920 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2019-12-05T05:12:49.923 INFO:tasks.ceph.mds.a:Started 2019-12-05T05:12:49.923 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:12:49.979 INFO:tasks.ceph.mds.a.smithi042.stdout:starting mds.a at 2019-12-05T05:12:50.027 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:50.012+0000 7f945573e700 1 Processor -- start 2019-12-05T05:12:50.027 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:50.012+0000 7f945573e700 1 -- start start 2019-12-05T05:12:50.028 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:50.012+0000 7f945573e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f94500a7740 0x7f945008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:12:50.028 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:50.012+0000 7f945573e700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f945007e720 0x7f945007eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:12:50.028 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:50.016+0000 7f945573e700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94500a0710 con 0x7f945009f2e0 2019-12-05T05:12:50.028 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:50.016+0000 7f945573e700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94500a0b00 con 0x7f945007e720 2019-12-05T05:12:50.028 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:50.016+0000 7f945573e700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f945008e5b0 con 0x7f94500a7740 2019-12-05T05:12:50.028 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:50.016+0000 7f944f7fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f94500a7740 0x7f945008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:12:50.028 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:50.016+0000 7f944f7fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f94500a7740 0x7f945008df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:58002/0 (socket says 172.21.15.42:58002) 2019-12-05T05:12:50.029 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:50.016+0000 7f944f7fe700 1 -- 172.21.15.42:0/3299520292 learned_addr learned my addr 172.21.15.42:0/3299520292 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:12:50.029 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:50.016+0000 7f944effd700 1 --2- 172.21.15.42:0/3299520292 >> v2:172.21.15.40:3300/0 conn(0x7f945007e720 0x7f945007eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:12:50.029 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:50.016+0000 7f944f7fe700 1 -- 172.21.15.42:0/3299520292 >> v2:172.21.15.40:3300/0 conn(0x7f945007e720 msgr2=0x7f945007eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:12:50.029 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:50.016+0000 7f944f7fe700 1 --2- 172.21.15.42:0/3299520292 >> v2:172.21.15.40:3300/0 conn(0x7f945007e720 0x7f945007eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:12:50.029 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:50.016+0000 7f944f7fe700 1 -- 172.21.15.42:0/3299520292 >> v1:172.21.15.40:6789/0 conn(0x7f945009f2e0 legacy=0x7f945009f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:12:50.029 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:50.016+0000 7f944f7fe700 1 -- 172.21.15.42:0/3299520292 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f945007f0a0 con 0x7f94500a7740 2019-12-05T05:12:50.029 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:50.016+0000 7f944f7fe700 1 --2- 172.21.15.42:0/3299520292 >> v2:172.21.15.42:3300/0 conn(0x7f94500a7740 0x7f945008df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f9440002f30 tx=0x7f944000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:50.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:50.016+0000 7f944f7fe700 1 --2- 172.21.15.42:0/3299520292 >> v2:172.21.15.42:3300/0 conn(0x7f94500a7740 0x7f945008df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f9440002f30 tx=0x7f944000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:50.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:50.016+0000 7f944f7fe700 1 --2- 172.21.15.42:0/3299520292 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f94500a7740 0x7f945008df80 secure :-1 s=READY pgs=477 cs=0 l=1 rx=0x7f9440002f30 tx=0x7f944000fd30).ready entity=mon.1 client_cookie=f183591b86949e51 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:12:50.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:50.016+0000 7f944f7fe700 1 --2- 172.21.15.42:0/3299520292 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f94500a7740 0x7f945008df80 secure :-1 s=THROTTLE_DONE pgs=477 cs=0 l=1 rx=0x7f9440002f30 tx=0x7f944000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:50.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:50.016+0000 7f944f7fe700 1 --2- 172.21.15.42:0/3299520292 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f94500a7740 0x7f945008df80 secure :-1 s=THROTTLE_DONE pgs=477 cs=0 l=1 rx=0x7f9440002f30 tx=0x7f944000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:50.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:50.016+0000 7f944e7fc700 1 -- 172.21.15.42:0/3299520292 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9440011370 con 0x7f94500a7740 2019-12-05T05:12:50.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:50.016+0000 7f944f7fe700 1 --2- 172.21.15.42:0/3299520292 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f94500a7740 0x7f945008df80 secure :-1 s=THROTTLE_DONE pgs=477 cs=0 l=1 rx=0x7f9440002f30 tx=0x7f944000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:50.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:50.016+0000 7f944e7fc700 1 -- 172.21.15.42:0/3299520292 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9440008bb0 con 0x7f94500a7740 2019-12-05T05:12:50.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:50.016+0000 7f944e7fc700 1 -- 172.21.15.42:0/3299520292 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9440011950 con 0x7f94500a7740 2019-12-05T05:12:50.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:50.016+0000 7f945573e700 1 -- 172.21.15.42:0/3299520292 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f94500a7740 msgr2=0x7f945008df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:12:50.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:50.016+0000 7f945573e700 1 --2- 172.21.15.42:0/3299520292 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f94500a7740 0x7f945008df80 secure :-1 s=READY pgs=477 cs=0 l=1 rx=0x7f9440002f30 tx=0x7f944000fd30).stop 2019-12-05T05:12:50.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:50.016+0000 7f945573e700 1 -- 172.21.15.42:0/3299520292 shutdown_connections 2019-12-05T05:12:50.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:50.016+0000 7f945573e700 1 --2- 172.21.15.42:0/3299520292 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f94500a7740 0x7f945008df80 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:12:50.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:50.016+0000 7f945573e700 1 --2- 172.21.15.42:0/3299520292 >> v2:172.21.15.40:3300/0 conn(0x7f945007e720 0x7f945007eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:12:50.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:50.016+0000 7f945573e700 1 -- 172.21.15.42:0/3299520292 shutdown_connections 2019-12-05T05:12:50.032 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:50.016+0000 7f945573e700 1 -- 172.21.15.42:0/3299520292 wait complete. 2019-12-05T05:12:50.032 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:50.016+0000 7f945573e700 1 -- 172.21.15.42:0/3299520292 >> 172.21.15.42:0/3299520292 conn(0x7f94500aabe0 msgr2=0x7f94500a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:12:50.032 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:50.020+0000 7f945573e700 1 Processor -- start 2019-12-05T05:12:50.032 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:50.020+0000 7f945573e700 1 -- start start 2019-12-05T05:12:50.033 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:50.020+0000 7f945573e700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f94500a6ed0 0x7f9450137ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:12:50.033 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:50.020+0000 7f945573e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f9450117930 0x7f945012ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:12:50.033 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:50.020+0000 7f945573e700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f945011aeb0 con 0x7f945005a210 2019-12-05T05:12:50.033 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:50.020+0000 7f945573e700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9450119450 con 0x7f94500a6ed0 2019-12-05T05:12:50.033 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:50.020+0000 7f945573e700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94501196f0 con 0x7f9450117930 2019-12-05T05:12:50.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:50.020+0000 7f944ffff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f9450117930 0x7f945012ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:12:50.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:50.020+0000 7f944ffff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f9450117930 0x7f945012ba40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:58008/0 (socket says 172.21.15.42:58008) 2019-12-05T05:12:50.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:50.020+0000 7f944ffff700 1 -- 172.21.15.42:0/2273957873 learned_addr learned my addr 172.21.15.42:0/2273957873 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:12:50.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:50.020+0000 7f944f7fe700 1 --2- 172.21.15.42:0/2273957873 >> v2:172.21.15.40:3300/0 conn(0x7f94500a6ed0 0x7f9450137ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:12:50.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:50.020+0000 7f9437fff700 1 -- 172.21.15.42:0/2273957873 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1729640114 0 0) 0x7f945011aeb0 con 0x7f945005a210 2019-12-05T05:12:50.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:50.020+0000 7f9437fff700 1 -- 172.21.15.42:0/2273957873 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94300030b0 con 0x7f945005a210 2019-12-05T05:12:50.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:50.020+0000 7f944ffff700 1 -- 172.21.15.42:0/2273957873 >> v2:172.21.15.40:3300/0 conn(0x7f94500a6ed0 msgr2=0x7f9450137ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:12:50.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:50.020+0000 7f944ffff700 1 --2- 172.21.15.42:0/2273957873 >> v2:172.21.15.40:3300/0 conn(0x7f94500a6ed0 0x7f9450137ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:12:50.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:50.020+0000 7f944ffff700 1 -- 172.21.15.42:0/2273957873 >> v1:172.21.15.40:6789/0 conn(0x7f945005a210 legacy=0x7f94501383f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:12:50.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:50.020+0000 7f944ffff700 1 -- 172.21.15.42:0/2273957873 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f945012bf90 con 0x7f9450117930 2019-12-05T05:12:50.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:50.020+0000 7f944ffff700 1 --2- 172.21.15.42:0/2273957873 >> v2:172.21.15.42:3300/0 conn(0x7f9450117930 0x7f945012ba40 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f9448001210 tx=0x7f9448010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:50.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:50.020+0000 7f944ffff700 1 --2- 172.21.15.42:0/2273957873 >> v2:172.21.15.42:3300/0 conn(0x7f9450117930 0x7f945012ba40 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f9448001210 tx=0x7f9448010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:50.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:50.020+0000 7f944ffff700 1 --2- 172.21.15.42:0/2273957873 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9450117930 0x7f945012ba40 secure :-1 s=READY pgs=478 cs=0 l=1 rx=0x7f9448001210 tx=0x7f9448010b40).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:12:50.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:50.020+0000 7f944ffff700 1 --2- 172.21.15.42:0/2273957873 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9450117930 0x7f945012ba40 secure :-1 s=THROTTLE_DONE pgs=478 cs=0 l=1 rx=0x7f9448001210 tx=0x7f9448010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:50.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:50.020+0000 7f9437fff700 1 -- 172.21.15.42:0/2273957873 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9448006dc0 con 0x7f9450117930 2019-12-05T05:12:50.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:50.020+0000 7f944ffff700 1 --2- 172.21.15.42:0/2273957873 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9450117930 0x7f945012ba40 secure :-1 s=THROTTLE_DONE pgs=478 cs=0 l=1 rx=0x7f9448001210 tx=0x7f9448010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:50.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:50.020+0000 7f944ffff700 1 --2- 172.21.15.42:0/2273957873 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9450117930 0x7f945012ba40 secure :-1 s=THROTTLE_DONE pgs=478 cs=0 l=1 rx=0x7f9448001210 tx=0x7f9448010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:50.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:50.020+0000 7f9437fff700 1 -- 172.21.15.42:0/2273957873 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94480069e0 con 0x7f9450117930 2019-12-05T05:12:50.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:50.020+0000 7f945573e700 1 -- 172.21.15.42:0/2273957873 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f945012c260 con 0x7f9450117930 2019-12-05T05:12:50.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:50.020+0000 7f9437fff700 1 -- 172.21.15.42:0/2273957873 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9448002620 con 0x7f9450117930 2019-12-05T05:12:50.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:50.020+0000 7f945573e700 1 -- 172.21.15.42:0/2273957873 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f945012c810 con 0x7f9450117930 2019-12-05T05:12:50.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:50.020+0000 7f944ffff700 1 --2- 172.21.15.42:0/2273957873 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9450117930 0x7f945012ba40 secure :-1 s=THROTTLE_DONE pgs=478 cs=0 l=1 rx=0x7f9448001210 tx=0x7f9448010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:50.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:50.024+0000 7f9437fff700 1 -- 172.21.15.42:0/2273957873 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f94480028c0 con 0x7f9450117930 2019-12-05T05:12:50.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:50.024+0000 7f944ffff700 1 --2- 172.21.15.42:0/2273957873 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9450117930 0x7f945012ba40 secure :-1 s=THROTTLE_DONE pgs=478 cs=0 l=1 rx=0x7f9448001210 tx=0x7f9448010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:50.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:50.024+0000 7f9437fff700 1 --2- 172.21.15.42:0/2273957873 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f943003b250 0x7f943003d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:12:50.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:50.024+0000 7f9437fff700 1 -- 172.21.15.42:0/2273957873 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6547+0+0 (secure 0 0 0) 0x7f944804b570 con 0x7f9450117930 2019-12-05T05:12:50.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:50.024+0000 7f944f7fe700 1 --2- 172.21.15.42:0/2273957873 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f943003b250 0x7f943003d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:12:50.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:50.024+0000 7f944f7fe700 1 --2- 172.21.15.42:0/2273957873 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f943003b250 0x7f943003d760 crc :-1 s=READY pgs=218 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:12:50.039 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:50.028+0000 7f945573e700 1 -- 172.21.15.42:0/2273957873 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f945005d110 con 0x7f9450117930 2019-12-05T05:12:50.041 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:50.036+0000 7f944ffff700 1 --2- 172.21.15.42:0/2273957873 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9450117930 0x7f945012ba40 secure :-1 s=THROTTLE_DONE pgs=478 cs=0 l=1 rx=0x7f9448001210 tx=0x7f9448010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:50.041 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:50.036+0000 7f9437fff700 1 -- 172.21.15.42:0/2273957873 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f94480325d0 con 0x7f9450117930 2019-12-05T05:12:50.205 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:50.200+0000 7f945573e700 1 -- 172.21.15.42:0/2273957873 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f945008fad0 con 0x7f9450117930 2019-12-05T05:12:50.206 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:50.200+0000 7f944ffff700 1 --2- 172.21.15.42:0/2273957873 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9450117930 0x7f945012ba40 secure :-1 s=THROTTLE_DONE pgs=478 cs=0 l=1 rx=0x7f9448001210 tx=0x7f9448010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:50.206 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:50.200+0000 7f9437fff700 1 -- 172.21.15.42:0/2273957873 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) v1 ==== 94+0+1769 (secure 0 0 0) 0x7f9448032830 con 0x7f9450117930 2019-12-05T05:12:50.206 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 62 2019-12-05T05:12:50.206 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:12:50.206 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":62,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":62,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:12:48.374585+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5391},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5391":{"gid":5391,"name":"b","rank":0,"incarnation":59,"state":"up:active","state_seq":23,"addr":"172.21.15.40:6833/814397242","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":814397242},{"type":"v1","addr":"172.21.15.40:6833","nonce":814397242}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:12:50.209 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:50.204+0000 7f945573e700 1 -- 172.21.15.42:0/2273957873 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f943003b250 msgr2=0x7f943003d760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:12:50.209 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:50.204+0000 7f945573e700 1 --2- 172.21.15.42:0/2273957873 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f943003b250 0x7f943003d760 crc :-1 s=READY pgs=218 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:12:50.209 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:50.204+0000 7f945573e700 1 -- 172.21.15.42:0/2273957873 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9450117930 msgr2=0x7f945012ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:12:50.209 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:50.204+0000 7f945573e700 1 --2- 172.21.15.42:0/2273957873 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9450117930 0x7f945012ba40 secure :-1 s=READY pgs=478 cs=0 l=1 rx=0x7f9448001210 tx=0x7f9448010b40).stop 2019-12-05T05:12:50.209 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:50.204+0000 7f945573e700 1 -- 172.21.15.42:0/2273957873 shutdown_connections 2019-12-05T05:12:50.210 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:50.204+0000 7f945573e700 1 --2- 172.21.15.42:0/2273957873 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f943003b250 0x7f943003d760 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:12:50.210 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:50.204+0000 7f945573e700 1 --2- 172.21.15.42:0/2273957873 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9450117930 0x7f945012ba40 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:12:50.210 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:50.204+0000 7f945573e700 1 --2- 172.21.15.42:0/2273957873 >> v2:172.21.15.40:3300/0 conn(0x7f94500a6ed0 0x7f9450137ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:12:50.210 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:50.204+0000 7f945573e700 1 -- 172.21.15.42:0/2273957873 shutdown_connections 2019-12-05T05:12:50.210 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:50.204+0000 7f945573e700 1 -- 172.21.15.42:0/2273957873 wait complete. 2019-12-05T05:12:50.211 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:50.204+0000 7f945573e700 1 -- 172.21.15.42:0/2273957873 >> 172.21.15.42:0/2273957873 conn(0x7f94500a9cb0 msgr2=0x7f945011b530 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:12:50.219 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2019-12-05T05:12:52.220 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:12:52.533 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:52.316+0000 7f3511fbc700 1 Processor -- start 2019-12-05T05:12:52.534 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:52.316+0000 7f3511fbc700 1 -- start start 2019-12-05T05:12:52.534 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:52.316+0000 7f3511fbc700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f350c08df80 0x7f350c09f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:12:52.534 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:52.316+0000 7f3511fbc700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f350c09f830 0x7f350c14efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:12:52.534 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:52.316+0000 7f3511fbc700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f350c09fc60 con 0x7f350c0a7740 2019-12-05T05:12:52.534 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:52.316+0000 7f3511fbc700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f350c0a0b00 con 0x7f350c08df80 2019-12-05T05:12:52.534 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:52.316+0000 7f3511fbc700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f350c08e5b0 con 0x7f350c09f830 2019-12-05T05:12:52.535 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:52.320+0000 7f3510d5a700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f350c09f830 0x7f350c14efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:12:52.535 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:52.320+0000 7f350b7fe700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f350c08df80 0x7f350c09f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:12:52.535 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:52.320+0000 7f3510d5a700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f350c09f830 0x7f350c14efe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:58020/0 (socket says 172.21.15.42:58020) 2019-12-05T05:12:52.535 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:52.320+0000 7f3510d5a700 1 -- 172.21.15.42:0/529066230 learned_addr learned my addr 172.21.15.42:0/529066230 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:12:52.535 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:52.320+0000 7f3510d5a700 1 -- 172.21.15.42:0/529066230 >> v2:172.21.15.40:3300/0 conn(0x7f350c08df80 msgr2=0x7f350c09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:12:52.536 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:52.320+0000 7f3510d5a700 1 --2- 172.21.15.42:0/529066230 >> v2:172.21.15.40:3300/0 conn(0x7f350c08df80 0x7f350c09f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:12:52.536 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:52.320+0000 7f3510d5a700 1 -- 172.21.15.42:0/529066230 >> v1:172.21.15.40:6789/0 conn(0x7f350c0a7740 legacy=0x7f350c07e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:12:52.536 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:52.320+0000 7f3510d5a700 1 -- 172.21.15.42:0/529066230 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f350c0a0710 con 0x7f350c09f830 2019-12-05T05:12:52.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:52.320+0000 7f3510d5a700 1 -- 172.21.15.42:0/529066230 >> v2:172.21.15.42:3300/0 conn(0x7f350c09f830 msgr2=0x7f350c14efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2019-12-05T05:12:52.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:52.320+0000 7f3510d5a700 1 -- 172.21.15.42:0/529066230 >> v2:172.21.15.42:3300/0 conn(0x7f350c09f830 msgr2=0x7f350c14efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2019-12-05T05:12:52.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:52.320+0000 7f3510d5a700 1 --2- 172.21.15.42:0/529066230 >> v2:172.21.15.42:3300/0 conn(0x7f350c09f830 0x7f350c14efe0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f35040032b0 tx=0x7f35040100b0).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-12-05T05:12:52.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:52.320+0000 7f3510d5a700 1 --2- 172.21.15.42:0/529066230 >> v2:172.21.15.42:3300/0 conn(0x7f350c09f830 0x7f350c14efe0 secure :-1 s=START_CONNECT pgs=0 cs=0 l=0 rx=0x7f35040032b0 tx=0x7f35040100b0)._fault waiting 0.200000 2019-12-05T05:12:52.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:52.524+0000 7f3510d5a700 1 --2- 172.21.15.42:0/529066230 >> v2:172.21.15.42:3300/0 conn(0x7f350c09f830 0x7f350c14efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:12:52.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:52.524+0000 7f3510d5a700 0 --2- 172.21.15.42:0/529066230 >> v2:172.21.15.42:3300/0 conn(0x7f350c09f830 0x7f350c14efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2019-12-05T05:12:52.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:52.524+0000 7f3510d5a700 1 --2- 172.21.15.42:0/529066230 >> v2:172.21.15.42:3300/0 conn(0x7f350c09f830 0x7f350c14efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:12:52.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:52.524+0000 7f350affd700 1 -- 172.21.15.42:0/529066230 >> v2:172.21.15.42:3300/0 conn(0x7f350c09f830 msgr2=0x7f350c14efe0 unknown :-1 s=STATE_CLOSED l=0).mark_down 2019-12-05T05:12:52.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:52.524+0000 7f350affd700 1 --2- 172.21.15.42:0/529066230 >> v2:172.21.15.42:3300/0 conn(0x7f350c09f830 0x7f350c14efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:12:52.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:52.524+0000 7f350affd700 1 --2- 172.21.15.42:0/529066230 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f34f804af50 0x7f34f804d460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:12:52.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:52.524+0000 7f350affd700 1 -- 172.21.15.42:0/529066230 --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34f804dba0 con 0x7f34f80470a0 2019-12-05T05:12:52.539 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:52.524+0000 7f350affd700 1 -- 172.21.15.42:0/529066230 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34f804ee70 con 0x7f34f804af50 2019-12-05T05:12:52.539 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:52.524+0000 7f350b7fe700 1 --2- 172.21.15.42:0/529066230 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f34f804af50 0x7f34f804d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:12:52.539 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:52.524+0000 7f350affd700 1 -- 172.21.15.42:0/529066230 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 923790725 0 0) 0x7f34f804dba0 con 0x7f34f80470a0 2019-12-05T05:12:52.539 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:52.524+0000 7f350affd700 1 -- 172.21.15.42:0/529066230 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34f804deb0 con 0x7f34f80470a0 2019-12-05T05:12:52.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:52.524+0000 7f350b7fe700 1 -- 172.21.15.42:0/529066230 >> v1:172.21.15.42:6789/0 conn(0x7f34f80470a0 legacy=0x7f34f8049570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:12:52.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:52.524+0000 7f350affd700 1 -- 172.21.15.42:0/529066230 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1031681293 0 0) 0x7f34f804deb0 con 0x7f34f80470a0 2019-12-05T05:12:52.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:52.524+0000 7f350b7fe700 1 -- 172.21.15.42:0/529066230 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34f804f170 con 0x7f34f804af50 2019-12-05T05:12:52.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:52.524+0000 7f350b7fe700 1 --2- 172.21.15.42:0/529066230 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f34f804af50 0x7f34f804d460 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f34fc001a30 tx=0x7f34fc001540).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:52.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:52.524+0000 7f350b7fe700 1 --2- 172.21.15.42:0/529066230 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f34f804af50 0x7f34f804d460 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f34fc001a30 tx=0x7f34fc001540).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:52.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:52.524+0000 7f350b7fe700 1 --2- 172.21.15.42:0/529066230 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f34f804af50 0x7f34f804d460 secure :-1 s=READY pgs=99 cs=0 l=1 rx=0x7f34fc001a30 tx=0x7f34fc001540).ready entity=mon.2 client_cookie=ebb25a64e5d39419 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:12:52.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:52.524+0000 7f350b7fe700 1 --2- 172.21.15.42:0/529066230 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f34f804af50 0x7f34f804d460 secure :-1 s=THROTTLE_DONE pgs=99 cs=0 l=1 rx=0x7f34fc001a30 tx=0x7f34fc001540).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:52.541 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:52.524+0000 7f350b7fe700 1 --2- 172.21.15.42:0/529066230 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f34f804af50 0x7f34f804d460 secure :-1 s=THROTTLE_DONE pgs=99 cs=0 l=1 rx=0x7f34fc001a30 tx=0x7f34fc001540).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:52.541 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:52.524+0000 7f350b7fe700 1 --2- 172.21.15.42:0/529066230 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f34f804af50 0x7f34f804d460 secure :-1 s=THROTTLE_DONE pgs=99 cs=0 l=1 rx=0x7f34fc001a30 tx=0x7f34fc001540).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:52.541 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:52.524+0000 7f350affd700 1 -- 172.21.15.42:0/529066230 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f34fc0017f0 con 0x7f34f804af50 2019-12-05T05:12:52.541 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:52.524+0000 7f350affd700 1 -- 172.21.15.42:0/529066230 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34fc001600 con 0x7f34f804af50 2019-12-05T05:12:52.541 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:52.524+0000 7f350affd700 1 -- 172.21.15.42:0/529066230 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f34f4011650 con 0x7f34f804af50 2019-12-05T05:12:52.541 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:52.524+0000 7f3511fbc700 1 -- 172.21.15.42:0/529066230 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f34f804af50 msgr2=0x7f34f804d460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:12:52.542 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:52.524+0000 7f3511fbc700 1 --2- 172.21.15.42:0/529066230 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f34f804af50 0x7f34f804d460 secure :-1 s=READY pgs=99 cs=0 l=1 rx=0x7f34fc001a30 tx=0x7f34fc001540).stop 2019-12-05T05:12:52.542 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:52.524+0000 7f3510d5a700 1 -- 172.21.15.42:0/529066230 reap_dead start 2019-12-05T05:12:52.542 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:52.528+0000 7f3511fbc700 1 -- 172.21.15.42:0/529066230 shutdown_connections 2019-12-05T05:12:52.542 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:52.528+0000 7f3511fbc700 1 -- 172.21.15.42:0/529066230 shutdown_connections 2019-12-05T05:12:52.543 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:52.528+0000 7f3511fbc700 1 -- 172.21.15.42:0/529066230 wait complete. 2019-12-05T05:12:52.543 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:52.528+0000 7f3511fbc700 1 -- 172.21.15.42:0/529066230 >> 172.21.15.42:0/529066230 conn(0x7f350c0aabe0 msgr2=0x7f350c0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:12:52.546 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:52.528+0000 7f3511fbc700 1 Processor -- start 2019-12-05T05:12:52.547 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:52.528+0000 7f3511fbc700 1 -- start start 2019-12-05T05:12:52.547 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:52.532+0000 7f3511fbc700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f350c12d1e0 0x7f350c0a1b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:12:52.547 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:52.532+0000 7f3511fbc700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f350c126d80 0x7f350c1271b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:12:52.547 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:52.532+0000 7f3511fbc700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f350c119150 con 0x7f350c118d20 2019-12-05T05:12:52.547 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:52.532+0000 7f3511fbc700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f350c117620 con 0x7f350c12d1e0 2019-12-05T05:12:52.548 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:52.532+0000 7f3511fbc700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f350c1178c0 con 0x7f350c126d80 2019-12-05T05:12:52.548 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:52.532+0000 7f350bfff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f350c12d1e0 0x7f350c0a1b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:12:52.548 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:52.532+0000 7f350bfff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f350c12d1e0 0x7f350c0a1b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:58030/0 (socket says 172.21.15.42:58030) 2019-12-05T05:12:52.548 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:52.532+0000 7f350bfff700 1 -- 172.21.15.42:0/1343483678 learned_addr learned my addr 172.21.15.42:0/1343483678 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:12:52.548 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:52.532+0000 7f350b7fe700 1 --1- 172.21.15.42:0/1343483678 >> v1:172.21.15.42:6789/0 conn(0x7f350c118d20 0x7f350c117e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:58066/0 (socket says 172.21.15.42:58066) 2019-12-05T05:12:52.548 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:52.532+0000 7f3510d5a700 1 --2- 172.21.15.42:0/1343483678 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f350c126d80 0x7f350c1271b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:12:52.548 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:52.532+0000 7f350bfff700 1 -- 172.21.15.42:0/1343483678 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f350c126d80 msgr2=0x7f350c1271b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:12:52.549 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:52.532+0000 7f350bfff700 1 --2- 172.21.15.42:0/1343483678 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f350c126d80 0x7f350c1271b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:12:52.549 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:52.532+0000 7f350bfff700 1 -- 172.21.15.42:0/1343483678 >> v1:172.21.15.42:6789/0 conn(0x7f350c118d20 legacy=0x7f350c117e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:12:52.549 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:52.532+0000 7f350bfff700 1 -- 172.21.15.42:0/1343483678 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f350c127700 con 0x7f350c12d1e0 2019-12-05T05:12:52.549 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:52.532+0000 7f350bfff700 1 --2- 172.21.15.42:0/1343483678 >> v2:172.21.15.42:3300/0 conn(0x7f350c12d1e0 0x7f350c0a1b60 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f350c0a21c0 tx=0x7f34fc0109f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:52.549 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:52.532+0000 7f350bfff700 1 --2- 172.21.15.42:0/1343483678 >> v2:172.21.15.42:3300/0 conn(0x7f350c12d1e0 0x7f350c0a1b60 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f350c0a21c0 tx=0x7f34fc0109f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:52.550 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:52.532+0000 7f350bfff700 1 --2- 172.21.15.42:0/1343483678 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f350c12d1e0 0x7f350c0a1b60 secure :-1 s=READY pgs=481 cs=0 l=1 rx=0x7f350c0a21c0 tx=0x7f34fc0109f0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:12:52.550 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:52.532+0000 7f350bfff700 1 --2- 172.21.15.42:0/1343483678 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f350c12d1e0 0x7f350c0a1b60 secure :-1 s=THROTTLE_DONE pgs=481 cs=0 l=1 rx=0x7f350c0a21c0 tx=0x7f34fc0109f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:52.550 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:52.532+0000 7f3508ff9700 1 -- 172.21.15.42:0/1343483678 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f34fc012800 con 0x7f350c12d1e0 2019-12-05T05:12:52.550 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:52.532+0000 7f350bfff700 1 --2- 172.21.15.42:0/1343483678 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f350c12d1e0 0x7f350c0a1b60 secure :-1 s=THROTTLE_DONE pgs=481 cs=0 l=1 rx=0x7f350c0a21c0 tx=0x7f34fc0109f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:52.550 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:52.532+0000 7f350bfff700 1 --2- 172.21.15.42:0/1343483678 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f350c12d1e0 0x7f350c0a1b60 secure :-1 s=THROTTLE_DONE pgs=481 cs=0 l=1 rx=0x7f350c0a21c0 tx=0x7f34fc0109f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:52.550 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:52.532+0000 7f3508ff9700 1 -- 172.21.15.42:0/1343483678 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34fc012420 con 0x7f350c12d1e0 2019-12-05T05:12:52.550 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:52.532+0000 7f3511fbc700 1 -- 172.21.15.42:0/1343483678 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f350c0a4b20 con 0x7f350c12d1e0 2019-12-05T05:12:52.551 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:52.532+0000 7f3508ff9700 1 -- 172.21.15.42:0/1343483678 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f34fc012de0 con 0x7f350c12d1e0 2019-12-05T05:12:52.551 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:52.532+0000 7f3511fbc700 1 -- 172.21.15.42:0/1343483678 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f350c0a5020 con 0x7f350c12d1e0 2019-12-05T05:12:52.551 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:52.536+0000 7f350bfff700 1 --2- 172.21.15.42:0/1343483678 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f350c12d1e0 0x7f350c0a1b60 secure :-1 s=THROTTLE_DONE pgs=481 cs=0 l=1 rx=0x7f350c0a21c0 tx=0x7f34fc0109f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:52.551 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:52.536+0000 7f3508ff9700 1 -- 172.21.15.42:0/1343483678 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f34fc00d1b0 con 0x7f350c12d1e0 2019-12-05T05:12:52.551 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:52.536+0000 7f350bfff700 1 --2- 172.21.15.42:0/1343483678 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f350c12d1e0 0x7f350c0a1b60 secure :-1 s=THROTTLE_DONE pgs=481 cs=0 l=1 rx=0x7f350c0a21c0 tx=0x7f34fc0109f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:52.551 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:52.536+0000 7f3508ff9700 1 --2- 172.21.15.42:0/1343483678 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f34f003ace0 0x7f34f003d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:12:52.552 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:52.536+0000 7f3508ff9700 1 -- 172.21.15.42:0/1343483678 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6547+0+0 (secure 0 0 0) 0x7f34fc04adf0 con 0x7f350c12d1e0 2019-12-05T05:12:52.552 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:52.536+0000 7f350b7fe700 1 --2- 172.21.15.42:0/1343483678 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f34f003ace0 0x7f34f003d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:12:52.552 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:52.536+0000 7f350b7fe700 1 --2- 172.21.15.42:0/1343483678 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f34f003ace0 0x7f34f003d1f0 crc :-1 s=READY pgs=219 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:12:52.588 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:52.584+0000 7f3511fbc700 1 -- 172.21.15.42:0/1343483678 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f350c05d110 con 0x7f350c12d1e0 2019-12-05T05:12:52.597 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:52.592+0000 7f350bfff700 1 --2- 172.21.15.42:0/1343483678 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f350c12d1e0 0x7f350c0a1b60 secure :-1 s=THROTTLE_DONE pgs=481 cs=0 l=1 rx=0x7f350c0a21c0 tx=0x7f34fc0109f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:52.597 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:52.592+0000 7f3508ff9700 1 -- 172.21.15.42:0/1343483678 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f34fc031e70 con 0x7f350c12d1e0 2019-12-05T05:12:52.775 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:52.760+0000 7f3511fbc700 1 -- 172.21.15.42:0/1343483678 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f350c1187b0 con 0x7f350c12d1e0 2019-12-05T05:12:52.776 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:52.764+0000 7f350bfff700 1 --2- 172.21.15.42:0/1343483678 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f350c12d1e0 0x7f350c0a1b60 secure :-1 s=THROTTLE_DONE pgs=481 cs=0 l=1 rx=0x7f350c0a21c0 tx=0x7f34fc0109f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:52.776 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:52.764+0000 7f3508ff9700 1 -- 172.21.15.42:0/1343483678 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) v1 ==== 94+0+1769 (secure 0 0 0) 0x7f34fc0320d0 con 0x7f350c12d1e0 2019-12-05T05:12:52.776 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 62 2019-12-05T05:12:52.777 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:52.764+0000 7f3511fbc700 1 -- 172.21.15.42:0/1343483678 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f34f003ace0 msgr2=0x7f34f003d1f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:12:52.777 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:52.764+0000 7f3511fbc700 1 --2- 172.21.15.42:0/1343483678 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f34f003ace0 0x7f34f003d1f0 crc :-1 s=READY pgs=219 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:12:52.777 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:52.764+0000 7f3511fbc700 1 -- 172.21.15.42:0/1343483678 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f350c12d1e0 msgr2=0x7f350c0a1b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:12:52.777 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:52.764+0000 7f3511fbc700 1 --2- 172.21.15.42:0/1343483678 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f350c12d1e0 0x7f350c0a1b60 secure :-1 s=READY pgs=481 cs=0 l=1 rx=0x7f350c0a21c0 tx=0x7f34fc0109f0).stop 2019-12-05T05:12:52.777 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:52.764+0000 7f3511fbc700 1 -- 172.21.15.42:0/1343483678 shutdown_connections 2019-12-05T05:12:52.777 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:52.764+0000 7f3511fbc700 1 --2- 172.21.15.42:0/1343483678 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f34f003ace0 0x7f34f003d1f0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:12:52.777 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:52.764+0000 7f3511fbc700 1 --2- 172.21.15.42:0/1343483678 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f350c126d80 0x7f350c1271b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:12:52.778 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:52.764+0000 7f3511fbc700 1 --2- 172.21.15.42:0/1343483678 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f350c12d1e0 0x7f350c0a1b60 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:12:52.778 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:52.764+0000 7f3511fbc700 1 -- 172.21.15.42:0/1343483678 shutdown_connections 2019-12-05T05:12:52.778 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:52.764+0000 7f3511fbc700 1 -- 172.21.15.42:0/1343483678 wait complete. 2019-12-05T05:12:52.778 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:52.764+0000 7f3511fbc700 1 -- 172.21.15.42:0/1343483678 >> 172.21.15.42:0/1343483678 conn(0x7f350c0a8940 msgr2=0x7f350c11c6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:12:52.778 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:12:52.778 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":62,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":62,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:12:48.374585+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5391},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5391":{"gid":5391,"name":"b","rank":0,"incarnation":59,"state":"up:active","state_seq":23,"addr":"172.21.15.40:6833/814397242","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":814397242},{"type":"v1","addr":"172.21.15.40:6833","nonce":814397242}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:12:52.781 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2019-12-05T05:12:54.781 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:12:54.890 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:54.876+0000 7ff2c3eae700 1 Processor -- start 2019-12-05T05:12:54.890 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:54.876+0000 7ff2c3eae700 1 -- start start 2019-12-05T05:12:54.891 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:54.880+0000 7ff2c3eae700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff2bc0a7740 0x7ff2bc08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:12:54.891 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:54.880+0000 7ff2c3eae700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff2bc09fae0 0x7ff2bc07eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:12:54.891 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:54.880+0000 7ff2c3eae700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2bc0a0710 con 0x7ff2bc07e720 2019-12-05T05:12:54.891 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:54.880+0000 7ff2c3eae700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2bc0a0b00 con 0x7ff2bc0a7740 2019-12-05T05:12:54.891 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:54.880+0000 7ff2c3eae700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2bc08e5b0 con 0x7ff2bc09fae0 2019-12-05T05:12:54.891 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:54.880+0000 7ff2c244b700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff2bc0a7740 0x7ff2bc08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:12:54.891 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:54.880+0000 7ff2c1c4a700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7ff2bc07e720 0x7ff2bc09f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:58080/0 (socket says 172.21.15.42:58080) 2019-12-05T05:12:54.892 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:54.880+0000 7ff2c1c4a700 1 -- 172.21.15.42:0/2162353389 learned_addr learned my addr 172.21.15.42:0/2162353389 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:12:54.892 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:54.880+0000 7ff2c244b700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff2bc0a7740 0x7ff2bc08df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:58040/0 (socket says 172.21.15.42:58040) 2019-12-05T05:12:54.892 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:54.880+0000 7ff2c2c4c700 1 --2- 172.21.15.42:0/2162353389 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff2bc09fae0 0x7ff2bc07eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:12:54.892 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:54.880+0000 7ff2c244b700 1 -- 172.21.15.42:0/2162353389 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff2bc09fae0 msgr2=0x7ff2bc07eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:12:54.892 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:54.880+0000 7ff2c244b700 1 --2- 172.21.15.42:0/2162353389 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff2bc09fae0 0x7ff2bc07eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:12:54.892 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:54.880+0000 7ff2c244b700 1 -- 172.21.15.42:0/2162353389 >> v1:172.21.15.42:6789/0 conn(0x7ff2bc07e720 legacy=0x7ff2bc09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:12:54.892 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:54.880+0000 7ff2c1449700 1 -- 172.21.15.42:0/2162353389 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1723752016 0 0) 0x7ff2bc0a0710 con 0x7ff2bc07e720 2019-12-05T05:12:54.893 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:54.880+0000 7ff2c244b700 1 -- 172.21.15.42:0/2162353389 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2bc07f0a0 con 0x7ff2bc0a7740 2019-12-05T05:12:54.893 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:54.880+0000 7ff2c244b700 1 --2- 172.21.15.42:0/2162353389 >> v2:172.21.15.42:3300/0 conn(0x7ff2bc0a7740 0x7ff2bc08df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7ff2b8008790 tx=0x7ff2b800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:54.893 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:54.880+0000 7ff2c244b700 1 --2- 172.21.15.42:0/2162353389 >> v2:172.21.15.42:3300/0 conn(0x7ff2bc0a7740 0x7ff2bc08df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7ff2b8008790 tx=0x7ff2b800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:54.893 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:54.880+0000 7ff2c244b700 1 --2- 172.21.15.42:0/2162353389 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff2bc0a7740 0x7ff2bc08df80 secure :-1 s=READY pgs=483 cs=0 l=1 rx=0x7ff2b8008790 tx=0x7ff2b800fc70).ready entity=mon.1 client_cookie=591619c40f9420a2 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:12:54.893 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:54.880+0000 7ff2c244b700 1 --2- 172.21.15.42:0/2162353389 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff2bc0a7740 0x7ff2bc08df80 secure :-1 s=THROTTLE_DONE pgs=483 cs=0 l=1 rx=0x7ff2b8008790 tx=0x7ff2b800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:54.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:54.880+0000 7ff2c1449700 1 -- 172.21.15.42:0/2162353389 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff2b8008bb0 con 0x7ff2bc0a7740 2019-12-05T05:12:54.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:54.880+0000 7ff2c244b700 1 --2- 172.21.15.42:0/2162353389 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff2bc0a7740 0x7ff2bc08df80 secure :-1 s=THROTTLE_DONE pgs=483 cs=0 l=1 rx=0x7ff2b8008790 tx=0x7ff2b800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:54.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:54.880+0000 7ff2c1449700 1 -- 172.21.15.42:0/2162353389 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2b80087d0 con 0x7ff2bc0a7740 2019-12-05T05:12:54.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:54.880+0000 7ff2c244b700 1 --2- 172.21.15.42:0/2162353389 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff2bc0a7740 0x7ff2bc08df80 secure :-1 s=THROTTLE_DONE pgs=483 cs=0 l=1 rx=0x7ff2b8008790 tx=0x7ff2b800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:54.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:54.880+0000 7ff2c3eae700 1 -- 172.21.15.42:0/2162353389 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff2bc0a7740 msgr2=0x7ff2bc08df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:12:54.895 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:54.880+0000 7ff2c1449700 1 -- 172.21.15.42:0/2162353389 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff2b8011860 con 0x7ff2bc0a7740 2019-12-05T05:12:54.895 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:54.880+0000 7ff2c3eae700 1 --2- 172.21.15.42:0/2162353389 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff2bc0a7740 0x7ff2bc08df80 secure :-1 s=READY pgs=483 cs=0 l=1 rx=0x7ff2b8008790 tx=0x7ff2b800fc70).stop 2019-12-05T05:12:54.895 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:54.884+0000 7ff2c3eae700 1 -- 172.21.15.42:0/2162353389 shutdown_connections 2019-12-05T05:12:54.895 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:54.884+0000 7ff2c3eae700 1 --2- 172.21.15.42:0/2162353389 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff2bc09fae0 0x7ff2bc07eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:12:54.895 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:54.884+0000 7ff2c3eae700 1 --2- 172.21.15.42:0/2162353389 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff2bc0a7740 0x7ff2bc08df80 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:12:54.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:54.884+0000 7ff2c3eae700 1 -- 172.21.15.42:0/2162353389 shutdown_connections 2019-12-05T05:12:54.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:54.884+0000 7ff2c3eae700 1 -- 172.21.15.42:0/2162353389 wait complete. 2019-12-05T05:12:54.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:54.884+0000 7ff2c3eae700 1 -- 172.21.15.42:0/2162353389 >> 172.21.15.42:0/2162353389 conn(0x7ff2bc0aabe0 msgr2=0x7ff2bc0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:12:54.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:54.884+0000 7ff2c3eae700 1 Processor -- start 2019-12-05T05:12:54.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:54.884+0000 7ff2c3eae700 1 -- start start 2019-12-05T05:12:54.900 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:54.884+0000 7ff2c3eae700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff2bc129ad0 0x7ff2bc129f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:12:54.900 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:54.884+0000 7ff2c3eae700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff2bc0a54c0 0x7ff2bc0a58f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:12:54.900 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:54.884+0000 7ff2c3eae700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2bc0a5e40 con 0x7ff2bc116990 2019-12-05T05:12:54.900 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:54.884+0000 7ff2c3eae700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2bc0a66a0 con 0x7ff2bc129ad0 2019-12-05T05:12:54.901 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:54.884+0000 7ff2c3eae700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2bc0633c0 con 0x7ff2bc0a54c0 2019-12-05T05:12:54.901 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:54.884+0000 7ff2c2c4c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff2bc0a54c0 0x7ff2bc0a58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:12:54.901 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:54.884+0000 7ff2c244b700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7ff2bc116990 0x7ff2bc08f370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:58084/0 (socket says 172.21.15.42:58084) 2019-12-05T05:12:54.901 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:54.884+0000 7ff2c244b700 1 -- 172.21.15.42:0/3996010062 learned_addr learned my addr 172.21.15.42:0/3996010062 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:12:54.901 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:54.884+0000 7ff2c1c4a700 1 --2- 172.21.15.42:0/3996010062 >> v2:172.21.15.40:3300/0 conn(0x7ff2bc129ad0 0x7ff2bc129f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:12:54.901 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:54.884+0000 7ff2b2ffd700 1 -- 172.21.15.42:0/3996010062 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 458117199 0 0) 0x7ff2bc0a5e40 con 0x7ff2bc116990 2019-12-05T05:12:54.902 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:54.884+0000 7ff2b2ffd700 1 -- 172.21.15.42:0/3996010062 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2a40030b0 con 0x7ff2bc116990 2019-12-05T05:12:54.902 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:54.888+0000 7ff2b2ffd700 1 -- 172.21.15.42:0/3996010062 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 3326748474 0 0) 0x7ff2a40030b0 con 0x7ff2bc116990 2019-12-05T05:12:54.902 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:54.888+0000 7ff2c2c4c700 1 -- 172.21.15.42:0/3996010062 >> v2:172.21.15.40:3300/0 conn(0x7ff2bc129ad0 msgr2=0x7ff2bc129f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:12:54.902 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:54.888+0000 7ff2c2c4c700 1 --2- 172.21.15.42:0/3996010062 >> v2:172.21.15.40:3300/0 conn(0x7ff2bc129ad0 0x7ff2bc129f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:12:54.902 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:54.888+0000 7ff2c2c4c700 1 -- 172.21.15.42:0/3996010062 >> v1:172.21.15.42:6789/0 conn(0x7ff2bc116990 legacy=0x7ff2bc08f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:12:54.903 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:54.888+0000 7ff2c2c4c700 1 -- 172.21.15.42:0/3996010062 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2bc1318c0 con 0x7ff2bc0a54c0 2019-12-05T05:12:54.903 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:54.888+0000 7ff2c2c4c700 1 --2- 172.21.15.42:0/3996010062 >> v2:172.21.15.42:3300/0 conn(0x7ff2bc0a54c0 0x7ff2bc0a58f0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7ff2b8008670 tx=0x7ff2b8010180).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:54.903 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:54.888+0000 7ff2c2c4c700 1 --2- 172.21.15.42:0/3996010062 >> v2:172.21.15.42:3300/0 conn(0x7ff2bc0a54c0 0x7ff2bc0a58f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7ff2b8008670 tx=0x7ff2b8010180).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:54.903 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:54.888+0000 7ff2c2c4c700 1 --2- 172.21.15.42:0/3996010062 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff2bc0a54c0 0x7ff2bc0a58f0 secure :-1 s=READY pgs=485 cs=0 l=1 rx=0x7ff2b8008670 tx=0x7ff2b8010180).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:12:54.903 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:54.888+0000 7ff2c2c4c700 1 --2- 172.21.15.42:0/3996010062 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff2bc0a54c0 0x7ff2bc0a58f0 secure :-1 s=THROTTLE_DONE pgs=485 cs=0 l=1 rx=0x7ff2b8008670 tx=0x7ff2b8010180).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:54.903 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:54.888+0000 7ff2b2ffd700 1 -- 172.21.15.42:0/3996010062 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff2b8008470 con 0x7ff2bc0a54c0 2019-12-05T05:12:54.904 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:54.888+0000 7ff2c2c4c700 1 --2- 172.21.15.42:0/3996010062 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff2bc0a54c0 0x7ff2bc0a58f0 secure :-1 s=THROTTLE_DONE pgs=485 cs=0 l=1 rx=0x7ff2b8008670 tx=0x7ff2b8010180).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:54.904 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:54.888+0000 7ff2c2c4c700 1 --2- 172.21.15.42:0/3996010062 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff2bc0a54c0 0x7ff2bc0a58f0 secure :-1 s=THROTTLE_DONE pgs=485 cs=0 l=1 rx=0x7ff2b8008670 tx=0x7ff2b8010180).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:54.904 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:54.888+0000 7ff2b2ffd700 1 -- 172.21.15.42:0/3996010062 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2b8008db0 con 0x7ff2bc0a54c0 2019-12-05T05:12:54.904 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:54.888+0000 7ff2b2ffd700 1 -- 172.21.15.42:0/3996010062 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff2b400eb40 con 0x7ff2bc0a54c0 2019-12-05T05:12:54.905 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:54.888+0000 7ff2c3eae700 1 -- 172.21.15.42:0/3996010062 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2bc1308b0 con 0x7ff2bc0a54c0 2019-12-05T05:12:54.905 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:54.888+0000 7ff2c3eae700 1 -- 172.21.15.42:0/3996010062 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2bc130e30 con 0x7ff2bc0a54c0 2019-12-05T05:12:54.905 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:54.888+0000 7ff2c2c4c700 1 --2- 172.21.15.42:0/3996010062 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff2bc0a54c0 0x7ff2bc0a58f0 secure :-1 s=THROTTLE_DONE pgs=485 cs=0 l=1 rx=0x7ff2b8008670 tx=0x7ff2b8010180).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:54.905 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:54.888+0000 7ff2b2ffd700 1 -- 172.21.15.42:0/3996010062 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7ff2b40058c0 con 0x7ff2bc0a54c0 2019-12-05T05:12:54.905 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:54.888+0000 7ff2c2c4c700 1 --2- 172.21.15.42:0/3996010062 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff2bc0a54c0 0x7ff2bc0a58f0 secure :-1 s=THROTTLE_DONE pgs=485 cs=0 l=1 rx=0x7ff2b8008670 tx=0x7ff2b8010180).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:54.905 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:54.888+0000 7ff2b2ffd700 1 --2- 172.21.15.42:0/3996010062 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff2a403b210 0x7ff2a403d720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:12:54.906 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:54.888+0000 7ff2b2ffd700 1 -- 172.21.15.42:0/3996010062 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6547+0+0 (secure 0 0 0) 0x7ff2b40120c0 con 0x7ff2bc0a54c0 2019-12-05T05:12:54.906 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:54.888+0000 7ff2c244b700 1 --2- 172.21.15.42:0/3996010062 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff2a403b210 0x7ff2a403d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:12:54.906 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:54.892+0000 7ff2c244b700 1 --2- 172.21.15.42:0/3996010062 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff2a403b210 0x7ff2a403d720 crc :-1 s=READY pgs=221 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:12:54.906 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:54.892+0000 7ff2c3eae700 1 -- 172.21.15.42:0/3996010062 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2bc05d110 con 0x7ff2bc0a54c0 2019-12-05T05:12:54.907 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:54.900+0000 7ff2c2c4c700 1 --2- 172.21.15.42:0/3996010062 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff2bc0a54c0 0x7ff2bc0a58f0 secure :-1 s=THROTTLE_DONE pgs=485 cs=0 l=1 rx=0x7ff2b8008670 tx=0x7ff2b8010180).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:54.907 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:54.900+0000 7ff2b2ffd700 1 -- 172.21.15.42:0/3996010062 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7ff2b4031f40 con 0x7ff2bc0a54c0 2019-12-05T05:12:55.076 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.072+0000 7ff2c3eae700 1 -- 172.21.15.42:0/3996010062 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff2bc05b8c0 con 0x7ff2bc0a54c0 2019-12-05T05:12:55.077 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.072+0000 7ff2c2c4c700 1 --2- 172.21.15.42:0/3996010062 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff2bc0a54c0 0x7ff2bc0a58f0 secure :-1 s=THROTTLE_DONE pgs=485 cs=0 l=1 rx=0x7ff2b8008670 tx=0x7ff2b8010180).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:55.077 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.072+0000 7ff2b2ffd700 1 -- 172.21.15.42:0/3996010062 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 63 v63) v1 ==== 94+0+2102 (secure 0 0 0) 0x7ff2b40321a0 con 0x7ff2bc0a54c0 2019-12-05T05:12:55.077 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 63 2019-12-05T05:12:55.078 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:12:55.078 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":63,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5505,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/211479572","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":211479572},{"type":"v1","addr":"172.21.15.42:6835","nonce":211479572}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":63}],"filesystems":[{"mdsmap":{"epoch":62,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:12:48.374585+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5391},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5391":{"gid":5391,"name":"b","rank":0,"incarnation":59,"state":"up:active","state_seq":23,"addr":"172.21.15.40:6833/814397242","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":814397242},{"type":"v1","addr":"172.21.15.40:6833","nonce":814397242}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:12:55.079 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.072+0000 7ff2c3eae700 1 -- 172.21.15.42:0/3996010062 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff2a403b210 msgr2=0x7ff2a403d720 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:12:55.080 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.072+0000 7ff2c3eae700 1 --2- 172.21.15.42:0/3996010062 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff2a403b210 0x7ff2a403d720 crc :-1 s=READY pgs=221 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:12:55.080 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.072+0000 7ff2c3eae700 1 -- 172.21.15.42:0/3996010062 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff2bc0a54c0 msgr2=0x7ff2bc0a58f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:12:55.080 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.072+0000 7ff2c3eae700 1 --2- 172.21.15.42:0/3996010062 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff2bc0a54c0 0x7ff2bc0a58f0 secure :-1 s=READY pgs=485 cs=0 l=1 rx=0x7ff2b8008670 tx=0x7ff2b8010180).stop 2019-12-05T05:12:55.080 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.072+0000 7ff2c3eae700 1 -- 172.21.15.42:0/3996010062 shutdown_connections 2019-12-05T05:12:55.080 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.072+0000 7ff2c3eae700 1 --2- 172.21.15.42:0/3996010062 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff2a403b210 0x7ff2a403d720 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:12:55.080 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.072+0000 7ff2c3eae700 1 --2- 172.21.15.42:0/3996010062 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff2bc0a54c0 0x7ff2bc0a58f0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:12:55.081 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.072+0000 7ff2c3eae700 1 --2- 172.21.15.42:0/3996010062 >> v2:172.21.15.40:3300/0 conn(0x7ff2bc129ad0 0x7ff2bc129f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:12:55.081 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.076+0000 7ff2c3eae700 1 -- 172.21.15.42:0/3996010062 shutdown_connections 2019-12-05T05:12:55.081 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.076+0000 7ff2c3eae700 1 -- 172.21.15.42:0/3996010062 wait complete. 2019-12-05T05:12:55.081 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.076+0000 7ff2c3eae700 1 -- 172.21.15.42:0/3996010062 >> 172.21.15.42:0/3996010062 conn(0x7ff2bc0a9b50 msgr2=0x7ff2bc141360 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:12:55.089 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a reported in up:standby state 2019-12-05T05:12:55.089 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:12:55.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.180+0000 7f817c2b4700 1 Processor -- start 2019-12-05T05:12:55.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.180+0000 7f817c2b4700 1 -- start start 2019-12-05T05:12:55.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.180+0000 7f817c2b4700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f81740a7740 0x7f817408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:12:55.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.180+0000 7f817c2b4700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f817409fae0 0x7f817407eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:12:55.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.180+0000 7f817c2b4700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81740a0710 con 0x7f817407e720 2019-12-05T05:12:55.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.180+0000 7f817c2b4700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81740a0b00 con 0x7f817409fae0 2019-12-05T05:12:55.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.180+0000 7f817c2b4700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f817408e5b0 con 0x7f81740a7740 2019-12-05T05:12:55.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.180+0000 7f817a851700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f81740a7740 0x7f817408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:12:55.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.180+0000 7f817a851700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f81740a7740 0x7f817408df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:58054/0 (socket says 172.21.15.42:58054) 2019-12-05T05:12:55.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.180+0000 7f817a050700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f817407e720 0x7f817409f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:58092/0 (socket says 172.21.15.42:58092) 2019-12-05T05:12:55.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.180+0000 7f817a851700 1 -- 172.21.15.42:0/1396920080 learned_addr learned my addr 172.21.15.42:0/1396920080 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:12:55.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.180+0000 7f817b052700 1 --2- 172.21.15.42:0/1396920080 >> v2:172.21.15.40:3300/0 conn(0x7f817409fae0 0x7f817407eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:12:55.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.180+0000 7f817a851700 1 -- 172.21.15.42:0/1396920080 >> v2:172.21.15.40:3300/0 conn(0x7f817409fae0 msgr2=0x7f817407eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:12:55.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.180+0000 7f817a851700 1 --2- 172.21.15.42:0/1396920080 >> v2:172.21.15.40:3300/0 conn(0x7f817409fae0 0x7f817407eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:12:55.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.180+0000 7f817a851700 1 -- 172.21.15.42:0/1396920080 >> v1:172.21.15.42:6789/0 conn(0x7f817407e720 legacy=0x7f817409f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:12:55.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.180+0000 7f817a851700 1 -- 172.21.15.42:0/1396920080 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f817407f0a0 con 0x7f81740a7740 2019-12-05T05:12:55.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.180+0000 7f817a851700 1 --2- 172.21.15.42:0/1396920080 >> v2:172.21.15.42:3300/0 conn(0x7f81740a7740 0x7f817408df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f8170002f30 tx=0x7f817000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:55.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.180+0000 7f817a851700 1 --2- 172.21.15.42:0/1396920080 >> v2:172.21.15.42:3300/0 conn(0x7f81740a7740 0x7f817408df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f8170002f30 tx=0x7f817000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:55.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.180+0000 7f817a851700 1 --2- 172.21.15.42:0/1396920080 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f81740a7740 0x7f817408df80 secure :-1 s=READY pgs=487 cs=0 l=1 rx=0x7f8170002f30 tx=0x7f817000fd30).ready entity=mon.1 client_cookie=d03803dc5a2e594f server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:12:55.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.180+0000 7f817a851700 1 --2- 172.21.15.42:0/1396920080 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f81740a7740 0x7f817408df80 secure :-1 s=THROTTLE_DONE pgs=487 cs=0 l=1 rx=0x7f8170002f30 tx=0x7f817000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:55.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.180+0000 7f817984f700 1 -- 172.21.15.42:0/1396920080 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8170008c90 con 0x7f81740a7740 2019-12-05T05:12:55.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.180+0000 7f817a851700 1 --2- 172.21.15.42:0/1396920080 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f81740a7740 0x7f817408df80 secure :-1 s=THROTTLE_DONE pgs=487 cs=0 l=1 rx=0x7f8170002f30 tx=0x7f817000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:55.193 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.180+0000 7f817a851700 1 --2- 172.21.15.42:0/1396920080 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f81740a7740 0x7f817408df80 secure :-1 s=THROTTLE_DONE pgs=487 cs=0 l=1 rx=0x7f8170002f30 tx=0x7f817000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:55.193 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.180+0000 7f817984f700 1 -- 172.21.15.42:0/1396920080 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81700088b0 con 0x7f81740a7740 2019-12-05T05:12:55.193 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.180+0000 7f817984f700 1 -- 172.21.15.42:0/1396920080 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8170011ad0 con 0x7f81740a7740 2019-12-05T05:12:55.193 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.180+0000 7f817c2b4700 1 -- 172.21.15.42:0/1396920080 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f81740a7740 msgr2=0x7f817408df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:12:55.193 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.180+0000 7f817c2b4700 1 --2- 172.21.15.42:0/1396920080 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f81740a7740 0x7f817408df80 secure :-1 s=READY pgs=487 cs=0 l=1 rx=0x7f8170002f30 tx=0x7f817000fd30).stop 2019-12-05T05:12:55.193 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.184+0000 7f817c2b4700 1 -- 172.21.15.42:0/1396920080 shutdown_connections 2019-12-05T05:12:55.193 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.184+0000 7f817c2b4700 1 --2- 172.21.15.42:0/1396920080 >> v2:172.21.15.40:3300/0 conn(0x7f817409fae0 0x7f817407eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:12:55.194 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.184+0000 7f817c2b4700 1 --2- 172.21.15.42:0/1396920080 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f81740a7740 0x7f817408df80 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:12:55.194 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.184+0000 7f817c2b4700 1 -- 172.21.15.42:0/1396920080 shutdown_connections 2019-12-05T05:12:55.194 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.184+0000 7f817c2b4700 1 -- 172.21.15.42:0/1396920080 wait complete. 2019-12-05T05:12:55.194 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.184+0000 7f817c2b4700 1 -- 172.21.15.42:0/1396920080 >> 172.21.15.42:0/1396920080 conn(0x7f81740aabe0 msgr2=0x7f81740a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:12:55.197 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.184+0000 7f817c2b4700 1 Processor -- start 2019-12-05T05:12:55.197 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.184+0000 7f817c2b4700 1 -- start start 2019-12-05T05:12:55.197 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.188+0000 7f817c2b4700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f81740a1850 0x7f817412ce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:12:55.197 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.188+0000 7f817c2b4700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f81741179f0 0x7f817412d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:12:55.198 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.188+0000 7f817c2b4700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f817412db00 0x7f8174118940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:12:55.198 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.188+0000 7f817c2b4700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8174117210 con 0x7f81740a1850 2019-12-05T05:12:55.198 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.188+0000 7f817c2b4700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81741174b0 con 0x7f817412db00 2019-12-05T05:12:55.198 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.188+0000 7f817c2b4700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8174117750 con 0x7f81741179f0 2019-12-05T05:12:55.198 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.188+0000 7f817b052700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f817412db00 0x7f8174118940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:12:55.198 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.188+0000 7f817a851700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f81740a1850 0x7f817412ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:12:55.198 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.188+0000 7f817a050700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f81741179f0 0x7f817412d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:12:55.199 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.188+0000 7f817b052700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f817412db00 0x7f8174118940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:58064/0 (socket says 172.21.15.42:58064) 2019-12-05T05:12:55.199 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.188+0000 7f817a851700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f81740a1850 0x7f817412ce10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.42:45876/0 (socket says 172.21.15.42:45876) 2019-12-05T05:12:55.199 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.188+0000 7f817b052700 1 -- 172.21.15.42:0/4113886884 learned_addr learned my addr 172.21.15.42:0/4113886884 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:12:55.199 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.188+0000 7f817a050700 1 -- 172.21.15.42:0/4113886884 >> v2:172.21.15.42:3300/0 conn(0x7f817412db00 msgr2=0x7f8174118940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:12:55.199 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.188+0000 7f817a050700 1 --2- 172.21.15.42:0/4113886884 >> v2:172.21.15.42:3300/0 conn(0x7f817412db00 0x7f8174118940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:12:55.199 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.188+0000 7f817a050700 1 -- 172.21.15.42:0/4113886884 >> v2:172.21.15.40:3300/0 conn(0x7f81740a1850 msgr2=0x7f817412ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:12:55.199 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.188+0000 7f817a050700 1 --2- 172.21.15.42:0/4113886884 >> v2:172.21.15.40:3300/0 conn(0x7f81740a1850 0x7f817412ce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:12:55.200 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.188+0000 7f817a050700 1 -- 172.21.15.42:0/4113886884 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8174117e20 con 0x7f81741179f0 2019-12-05T05:12:55.200 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.188+0000 7f817a050700 1 --2- 172.21.15.42:0/4113886884 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f81741179f0 0x7f817412d5b0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f8174117130 tx=0x7f81680109f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:55.200 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.188+0000 7f817a050700 1 --2- 172.21.15.42:0/4113886884 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f81741179f0 0x7f817412d5b0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f8174117130 tx=0x7f81680109f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:55.200 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.188+0000 7f817a050700 1 --2- 172.21.15.42:0/4113886884 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f81741179f0 0x7f817412d5b0 secure :-1 s=READY pgs=100 cs=0 l=1 rx=0x7f8174117130 tx=0x7f81680109f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:12:55.200 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.188+0000 7f817a050700 1 --2- 172.21.15.42:0/4113886884 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f81741179f0 0x7f817412d5b0 secure :-1 s=THROTTLE_DONE pgs=100 cs=0 l=1 rx=0x7f8174117130 tx=0x7f81680109f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:55.200 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.188+0000 7f817a050700 1 --2- 172.21.15.42:0/4113886884 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f81741179f0 0x7f817412d5b0 secure :-1 s=THROTTLE_DONE pgs=100 cs=0 l=1 rx=0x7f8174117130 tx=0x7f81680109f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:55.201 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.188+0000 7f81677fe700 1 -- 172.21.15.42:0/4113886884 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8168011750 con 0x7f81741179f0 2019-12-05T05:12:55.201 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.188+0000 7f817a050700 1 --2- 172.21.15.42:0/4113886884 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f81741179f0 0x7f817412d5b0 secure :-1 s=THROTTLE_DONE pgs=100 cs=0 l=1 rx=0x7f8174117130 tx=0x7f81680109f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:55.201 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.188+0000 7f81677fe700 1 -- 172.21.15.42:0/4113886884 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8168011370 con 0x7f81741179f0 2019-12-05T05:12:55.201 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.188+0000 7f817c2b4700 1 -- 172.21.15.42:0/4113886884 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8174118e90 con 0x7f81741179f0 2019-12-05T05:12:55.201 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.188+0000 7f81677fe700 1 -- 172.21.15.42:0/4113886884 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8168011d30 con 0x7f81741179f0 2019-12-05T05:12:55.201 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.188+0000 7f817c2b4700 1 -- 172.21.15.42:0/4113886884 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81741303d0 con 0x7f81741179f0 2019-12-05T05:12:55.201 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.188+0000 7f817a050700 1 --2- 172.21.15.42:0/4113886884 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f81741179f0 0x7f817412d5b0 secure :-1 s=THROTTLE_DONE pgs=100 cs=0 l=1 rx=0x7f8174117130 tx=0x7f81680109f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:55.202 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.188+0000 7f817a050700 1 --2- 172.21.15.42:0/4113886884 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f81741179f0 0x7f817412d5b0 secure :-1 s=THROTTLE_DONE pgs=100 cs=0 l=1 rx=0x7f8174117130 tx=0x7f81680109f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:55.202 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.188+0000 7f81677fe700 1 -- 172.21.15.42:0/4113886884 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f8168006460 con 0x7f81741179f0 2019-12-05T05:12:55.202 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.192+0000 7f81677fe700 1 --2- 172.21.15.42:0/4113886884 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f815c03ae60 0x7f815c03d370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:12:55.202 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.192+0000 7f81677fe700 1 -- 172.21.15.42:0/4113886884 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6547+0+0 (secure 0 0 0) 0x7f816804ae60 con 0x7f81741179f0 2019-12-05T05:12:55.202 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.192+0000 7f817a851700 1 --2- 172.21.15.42:0/4113886884 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f815c03ae60 0x7f815c03d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:12:55.202 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.192+0000 7f817a851700 1 --2- 172.21.15.42:0/4113886884 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f815c03ae60 0x7f815c03d370 crc :-1 s=READY pgs=222 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:12:55.203 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.196+0000 7f817c2b4700 1 -- 172.21.15.42:0/4113886884 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f817405d110 con 0x7f81741179f0 2019-12-05T05:12:55.205 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.200+0000 7f817a050700 1 --2- 172.21.15.42:0/4113886884 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f81741179f0 0x7f817412d5b0 secure :-1 s=THROTTLE_DONE pgs=100 cs=0 l=1 rx=0x7f8174117130 tx=0x7f81680109f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:55.205 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.200+0000 7f81677fe700 1 -- 172.21.15.42:0/4113886884 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f8168031ea0 con 0x7f81741179f0 2019-12-05T05:12:55.369 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.364+0000 7f817c2b4700 1 -- 172.21.15.42:0/4113886884 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f817405b8c0 con 0x7f81741179f0 2019-12-05T05:12:55.370 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.364+0000 7f817a050700 1 --2- 172.21.15.42:0/4113886884 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f81741179f0 0x7f817412d5b0 secure :-1 s=THROTTLE_DONE pgs=100 cs=0 l=1 rx=0x7f8174117130 tx=0x7f81680109f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:55.370 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.364+0000 7f81677fe700 1 -- 172.21.15.42:0/4113886884 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 63 v63) v1 ==== 94+0+2102 (secure 0 0 0) 0x7f8168032100 con 0x7f81741179f0 2019-12-05T05:12:55.372 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 63 2019-12-05T05:12:55.373 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:12:55.373 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":63,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5505,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/211479572","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":211479572},{"type":"v1","addr":"172.21.15.42:6835","nonce":211479572}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":63}],"filesystems":[{"mdsmap":{"epoch":62,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:12:48.374585+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5391},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5391":{"gid":5391,"name":"b","rank":0,"incarnation":59,"state":"up:active","state_seq":23,"addr":"172.21.15.40:6833/814397242","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":814397242},{"type":"v1","addr":"172.21.15.40:6833","nonce":814397242}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:12:55.376 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.368+0000 7f817c2b4700 1 -- 172.21.15.42:0/4113886884 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f815c03ae60 msgr2=0x7f815c03d370 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:12:55.376 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.368+0000 7f817c2b4700 1 --2- 172.21.15.42:0/4113886884 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f815c03ae60 0x7f815c03d370 crc :-1 s=READY pgs=222 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:12:55.377 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.368+0000 7f817c2b4700 1 -- 172.21.15.42:0/4113886884 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f81741179f0 msgr2=0x7f817412d5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:12:55.377 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.368+0000 7f817c2b4700 1 --2- 172.21.15.42:0/4113886884 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f81741179f0 0x7f817412d5b0 secure :-1 s=READY pgs=100 cs=0 l=1 rx=0x7f8174117130 tx=0x7f81680109f0).stop 2019-12-05T05:12:55.377 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.368+0000 7f817c2b4700 1 -- 172.21.15.42:0/4113886884 shutdown_connections 2019-12-05T05:12:55.377 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.368+0000 7f817c2b4700 1 --2- 172.21.15.42:0/4113886884 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f815c03ae60 0x7f815c03d370 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:12:55.378 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.368+0000 7f817c2b4700 1 --2- 172.21.15.42:0/4113886884 >> v2:172.21.15.42:3300/0 conn(0x7f817412db00 0x7f8174118940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:12:55.378 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.368+0000 7f817c2b4700 1 --2- 172.21.15.42:0/4113886884 >> v2:172.21.15.40:3300/0 conn(0x7f81740a1850 0x7f817412ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:12:55.378 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.368+0000 7f817c2b4700 1 --2- 172.21.15.42:0/4113886884 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f81741179f0 0x7f817412d5b0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:12:55.378 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.368+0000 7f817c2b4700 1 -- 172.21.15.42:0/4113886884 shutdown_connections 2019-12-05T05:12:55.379 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.368+0000 7f817c2b4700 1 -- 172.21.15.42:0/4113886884 wait complete. 2019-12-05T05:12:55.379 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.368+0000 7f817c2b4700 1 -- 172.21.15.42:0/4113886884 >> 172.21.15.42:0/4113886884 conn(0x7f81740a7740 msgr2=0x7f817409f2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:12:55.385 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.b (rank=0) 2019-12-05T05:12:55.386 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2019-12-05T05:12:55.386 INFO:teuthology.orchestra.run:waiting for 300 2019-12-05T05:12:55.401 INFO:tasks.ceph.mds.b:Stopped 2019-12-05T05:12:55.402 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:12:55.505 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.492+0000 7f08478cb700 1 Processor -- start 2019-12-05T05:12:55.505 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.492+0000 7f08478cb700 1 -- start start 2019-12-05T05:12:55.505 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.496+0000 7f08478cb700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f08400a7740 0x7f084008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:12:55.505 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.496+0000 7f08478cb700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f084009fae0 0x7f084007eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:12:55.505 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.496+0000 7f08478cb700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08400a0710 con 0x7f084007e720 2019-12-05T05:12:55.506 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.496+0000 7f08478cb700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08400a0b00 con 0x7f08400a7740 2019-12-05T05:12:55.506 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.496+0000 7f08478cb700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f084008e5b0 con 0x7f084009fae0 2019-12-05T05:12:55.506 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.496+0000 7f0846669700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f084009fae0 0x7f084007eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:12:55.506 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.496+0000 7f0846669700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f084009fae0 0x7f084007eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:58074/0 (socket says 172.21.15.42:58074) 2019-12-05T05:12:55.506 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.496+0000 7f0846669700 1 -- 172.21.15.42:0/102290193 learned_addr learned my addr 172.21.15.42:0/102290193 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:12:55.506 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.496+0000 7f0845e68700 1 --2- 172.21.15.42:0/102290193 >> v2:172.21.15.40:3300/0 conn(0x7f08400a7740 0x7f084008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:12:55.506 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.496+0000 7f0846669700 1 -- 172.21.15.42:0/102290193 >> v2:172.21.15.40:3300/0 conn(0x7f08400a7740 msgr2=0x7f084008df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:12:55.507 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.496+0000 7f0846669700 1 --2- 172.21.15.42:0/102290193 >> v2:172.21.15.40:3300/0 conn(0x7f08400a7740 0x7f084008df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:12:55.507 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.496+0000 7f0846669700 1 -- 172.21.15.42:0/102290193 >> v1:172.21.15.40:6789/0 conn(0x7f084007e720 legacy=0x7f084009f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:12:55.507 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.496+0000 7f0846669700 1 -- 172.21.15.42:0/102290193 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f084007f0a0 con 0x7f084009fae0 2019-12-05T05:12:55.507 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.496+0000 7f0846669700 1 --2- 172.21.15.42:0/102290193 >> v2:172.21.15.42:3300/0 conn(0x7f084009fae0 0x7f084007eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f0830002f30 tx=0x7f083000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:55.507 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.496+0000 7f0846669700 1 --2- 172.21.15.42:0/102290193 >> v2:172.21.15.42:3300/0 conn(0x7f084009fae0 0x7f084007eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f0830002f30 tx=0x7f083000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:55.507 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.496+0000 7f0846669700 1 --2- 172.21.15.42:0/102290193 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f084009fae0 0x7f084007eb50 secure :-1 s=READY pgs=488 cs=0 l=1 rx=0x7f0830002f30 tx=0x7f083000fd30).ready entity=mon.1 client_cookie=5dea33277de166d9 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:12:55.507 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.496+0000 7f0846669700 1 --2- 172.21.15.42:0/102290193 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f084009fae0 0x7f084007eb50 secure :-1 s=THROTTLE_DONE pgs=488 cs=0 l=1 rx=0x7f0830002f30 tx=0x7f083000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:55.508 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.496+0000 7f0846669700 1 --2- 172.21.15.42:0/102290193 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f084009fae0 0x7f084007eb50 secure :-1 s=THROTTLE_DONE pgs=488 cs=0 l=1 rx=0x7f0830002f30 tx=0x7f083000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:55.508 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.496+0000 7f0844e66700 1 -- 172.21.15.42:0/102290193 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0830008c90 con 0x7f084009fae0 2019-12-05T05:12:55.508 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.496+0000 7f0846669700 1 --2- 172.21.15.42:0/102290193 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f084009fae0 0x7f084007eb50 secure :-1 s=THROTTLE_DONE pgs=488 cs=0 l=1 rx=0x7f0830002f30 tx=0x7f083000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:55.508 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.496+0000 7f0844e66700 1 -- 172.21.15.42:0/102290193 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08300088b0 con 0x7f084009fae0 2019-12-05T05:12:55.509 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.496+0000 7f0844e66700 1 -- 172.21.15.42:0/102290193 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0830011ad0 con 0x7f084009fae0 2019-12-05T05:12:55.509 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.496+0000 7f08478cb700 1 -- 172.21.15.42:0/102290193 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f084009fae0 msgr2=0x7f084007eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:12:55.509 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.496+0000 7f08478cb700 1 --2- 172.21.15.42:0/102290193 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f084009fae0 0x7f084007eb50 secure :-1 s=READY pgs=488 cs=0 l=1 rx=0x7f0830002f30 tx=0x7f083000fd30).stop 2019-12-05T05:12:55.509 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.496+0000 7f08478cb700 1 -- 172.21.15.42:0/102290193 shutdown_connections 2019-12-05T05:12:55.509 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.496+0000 7f08478cb700 1 --2- 172.21.15.42:0/102290193 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f084009fae0 0x7f084007eb50 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:12:55.509 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.496+0000 7f08478cb700 1 --2- 172.21.15.42:0/102290193 >> v2:172.21.15.40:3300/0 conn(0x7f08400a7740 0x7f084008df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:12:55.510 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.496+0000 7f08478cb700 1 -- 172.21.15.42:0/102290193 shutdown_connections 2019-12-05T05:12:55.510 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.496+0000 7f08478cb700 1 -- 172.21.15.42:0/102290193 wait complete. 2019-12-05T05:12:55.510 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.496+0000 7f08478cb700 1 -- 172.21.15.42:0/102290193 >> 172.21.15.42:0/102290193 conn(0x7f08400aabe0 msgr2=0x7f08400a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:12:55.514 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.500+0000 7f08478cb700 1 Processor -- start 2019-12-05T05:12:55.514 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.500+0000 7f08478cb700 1 -- start start 2019-12-05T05:12:55.514 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.500+0000 7f08478cb700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0840123630 0x7f084007a310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:12:55.515 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.500+0000 7f08478cb700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0840124440 con 0x7f08400a1df0 2019-12-05T05:12:55.515 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.500+0000 7f08478cb700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08400a3180 con 0x7f08400a19c0 2019-12-05T05:12:55.515 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.500+0000 7f08478cb700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08400a2220 con 0x7f0840123630 2019-12-05T05:12:55.515 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.500+0000 7f0845667700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f08400a19c0 0x7f08400a2980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:58116/0 (socket says 172.21.15.42:58116) 2019-12-05T05:12:55.515 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.500+0000 7f0845667700 1 -- 172.21.15.42:0/3185903258 learned_addr learned my addr 172.21.15.42:0/3185903258 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:12:55.515 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.500+0000 7f0845e68700 1 --2- 172.21.15.42:0/3185903258 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0840123630 0x7f084007a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:12:55.516 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.500+0000 7f083e7fc700 1 -- 172.21.15.42:0/3185903258 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 37485312 0 0) 0x7f08400a3180 con 0x7f08400a19c0 2019-12-05T05:12:55.516 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.500+0000 7f083e7fc700 1 -- 172.21.15.42:0/3185903258 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08200030b0 con 0x7f08400a19c0 2019-12-05T05:12:55.516 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.500+0000 7f083e7fc700 1 -- 172.21.15.42:0/3185903258 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2492689362 0 0) 0x7f0840124440 con 0x7f08400a1df0 2019-12-05T05:12:55.516 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.500+0000 7f083e7fc700 1 -- 172.21.15.42:0/3185903258 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08400a3180 con 0x7f08400a1df0 2019-12-05T05:12:55.516 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.500+0000 7f083e7fc700 1 -- 172.21.15.42:0/3185903258 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 1702125508 0 0) 0x7f08200030b0 con 0x7f08400a19c0 2019-12-05T05:12:55.516 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.500+0000 7f083e7fc700 1 -- 172.21.15.42:0/3185903258 --> v1:172.21.15.42:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0840124440 con 0x7f08400a19c0 2019-12-05T05:12:55.516 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.500+0000 7f083e7fc700 1 -- 172.21.15.42:0/3185903258 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7f0834001a00 con 0x7f08400a19c0 2019-12-05T05:12:55.517 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.500+0000 7f083e7fc700 1 -- 172.21.15.42:0/3185903258 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0840123630 msgr2=0x7f084007a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:12:55.517 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.500+0000 7f083e7fc700 1 --2- 172.21.15.42:0/3185903258 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0840123630 0x7f084007a310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:12:55.517 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.500+0000 7f083e7fc700 1 -- 172.21.15.42:0/3185903258 >> v1:172.21.15.42:6789/0 conn(0x7f08400a19c0 legacy=0x7f08400a2980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:12:55.517 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.504+0000 7f083e7fc700 1 -- 172.21.15.42:0/3185903258 >> v1:172.21.15.40:6789/0 conn(0x7f08400a1df0 legacy=0x7f084012e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:12:55.517 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.504+0000 7f083e7fc700 1 --2- 172.21.15.42:0/3185903258 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0820003ab0 0x7f0820004420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:12:55.517 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.504+0000 7f083e7fc700 1 --2- 172.21.15.42:0/3185903258 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0820004970 0x7f0820004da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:12:55.517 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.504+0000 7f083e7fc700 1 --2- 172.21.15.42:0/3185903258 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0820009660 0x7f082000baa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:12:55.518 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.504+0000 7f083e7fc700 1 -- 172.21.15.42:0/3185903258 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08400a2220 con 0x7f0820004970 2019-12-05T05:12:55.518 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.504+0000 7f083e7fc700 1 -- 172.21.15.42:0/3185903258 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0820004120 con 0x7f0820003ab0 2019-12-05T05:12:55.518 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.504+0000 7f083e7fc700 1 -- 172.21.15.42:0/3185903258 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f082000bff0 con 0x7f0820009660 2019-12-05T05:12:55.518 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.504+0000 7f0845e68700 1 --2- 172.21.15.42:0/3185903258 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0820003ab0 0x7f0820004420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:12:55.518 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.504+0000 7f0845667700 1 --2- 172.21.15.42:0/3185903258 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0820004970 0x7f0820004da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:12:55.518 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.504+0000 7f0846669700 1 --2- 172.21.15.42:0/3185903258 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0820009660 0x7f082000baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:12:55.518 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.504+0000 7f0845e68700 1 -- 172.21.15.42:0/3185903258 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0820009660 msgr2=0x7f082000baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:12:55.519 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.504+0000 7f0845e68700 1 --2- 172.21.15.42:0/3185903258 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0820009660 0x7f082000baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:12:55.519 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.504+0000 7f0845e68700 1 -- 172.21.15.42:0/3185903258 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0820004970 msgr2=0x7f0820004da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:12:55.519 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.504+0000 7f0845e68700 1 --2- 172.21.15.42:0/3185903258 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0820004970 0x7f0820004da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:12:55.519 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.504+0000 7f0845e68700 1 -- 172.21.15.42:0/3185903258 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0840131870 con 0x7f0820003ab0 2019-12-05T05:12:55.519 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.504+0000 7f08478cb700 1 -- 172.21.15.42:0/3185903258 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0840123630 con 0x7f0820003ab0 2019-12-05T05:12:55.519 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.504+0000 7f0845e68700 1 --2- 172.21.15.42:0/3185903258 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0820003ab0 0x7f0820004420 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f08200043b0 tx=0x7f0838011070).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:55.519 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.504+0000 7f08478cb700 1 -- 172.21.15.42:0/3185903258 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f084007a310 con 0x7f0820003ab0 2019-12-05T05:12:55.520 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.504+0000 7f0845e68700 1 --2- 172.21.15.42:0/3185903258 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0820003ab0 0x7f0820004420 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f08200043b0 tx=0x7f0838011070).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:55.520 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.504+0000 7f0845e68700 1 --2- 172.21.15.42:0/3185903258 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0820003ab0 0x7f0820004420 secure :-1 s=READY pgs=490 cs=0 l=1 rx=0x7f08200043b0 tx=0x7f0838011070).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:12:55.520 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.504+0000 7f0845e68700 1 --2- 172.21.15.42:0/3185903258 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0820003ab0 0x7f0820004420 secure :-1 s=THROTTLE_DONE pgs=490 cs=0 l=1 rx=0x7f08200043b0 tx=0x7f0838011070).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:55.520 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.504+0000 7f083e7fc700 1 -- 172.21.15.42:0/3185903258 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f08380058c0 con 0x7f0820003ab0 2019-12-05T05:12:55.520 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.504+0000 7f0845e68700 1 --2- 172.21.15.42:0/3185903258 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0820003ab0 0x7f0820004420 secure :-1 s=THROTTLE_DONE pgs=490 cs=0 l=1 rx=0x7f08200043b0 tx=0x7f0838011070).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:55.520 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.504+0000 7f083e7fc700 1 -- 172.21.15.42:0/3185903258 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0838011a70 con 0x7f0820003ab0 2019-12-05T05:12:55.521 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.504+0000 7f0845e68700 1 --2- 172.21.15.42:0/3185903258 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0820003ab0 0x7f0820004420 secure :-1 s=THROTTLE_DONE pgs=490 cs=0 l=1 rx=0x7f08200043b0 tx=0x7f0838011070).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:55.521 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.504+0000 7f083e7fc700 1 -- 172.21.15.42:0/3185903258 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0838007150 con 0x7f0820003ab0 2019-12-05T05:12:55.521 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.504+0000 7f0845e68700 1 --2- 172.21.15.42:0/3185903258 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0820003ab0 0x7f0820004420 secure :-1 s=THROTTLE_DONE pgs=490 cs=0 l=1 rx=0x7f08200043b0 tx=0x7f0838011070).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:55.521 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.504+0000 7f083e7fc700 1 -- 172.21.15.42:0/3185903258 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f0838007350 con 0x7f0820003ab0 2019-12-05T05:12:55.521 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.504+0000 7f0845e68700 1 --2- 172.21.15.42:0/3185903258 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0820003ab0 0x7f0820004420 secure :-1 s=THROTTLE_DONE pgs=490 cs=0 l=1 rx=0x7f08200043b0 tx=0x7f0838011070).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:55.521 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.504+0000 7f083e7fc700 1 --2- 172.21.15.42:0/3185903258 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0820043ea0 0x7f08200463b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:12:55.521 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.504+0000 7f083e7fc700 1 -- 172.21.15.42:0/3185903258 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6547+0+0 (secure 0 0 0) 0x7f083804b400 con 0x7f0820003ab0 2019-12-05T05:12:55.522 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.504+0000 7f0845667700 1 --2- 172.21.15.42:0/3185903258 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0820043ea0 0x7f08200463b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:12:55.522 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.504+0000 7f0845667700 1 --2- 172.21.15.42:0/3185903258 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0820043ea0 0x7f08200463b0 crc :-1 s=READY pgs=223 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:12:55.522 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.508+0000 7f08478cb700 1 -- 172.21.15.42:0/3185903258 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f081c003200 con 0x7f0820003ab0 2019-12-05T05:12:55.523 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.516+0000 7f0845e68700 1 --2- 172.21.15.42:0/3185903258 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0820003ab0 0x7f0820004420 secure :-1 s=THROTTLE_DONE pgs=490 cs=0 l=1 rx=0x7f08200043b0 tx=0x7f0838011070).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:55.523 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.516+0000 7f083e7fc700 1 -- 172.21.15.42:0/3185903258 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f0838032480 con 0x7f0820003ab0 2019-12-05T05:12:55.690 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.684+0000 7f08478cb700 1 -- 172.21.15.42:0/3185903258 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f081c003630 con 0x7f0820003ab0 2019-12-05T05:12:55.691 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.684+0000 7f0845e68700 1 --2- 172.21.15.42:0/3185903258 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0820003ab0 0x7f0820004420 secure :-1 s=THROTTLE_DONE pgs=490 cs=0 l=1 rx=0x7f08200043b0 tx=0x7f0838011070).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:55.691 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.684+0000 7f083e7fc700 1 -- 172.21.15.42:0/3185903258 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 63 v63) v1 ==== 94+0+2102 (secure 0 0 0) 0x7f08380326e0 con 0x7f0820003ab0 2019-12-05T05:12:55.691 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 63 2019-12-05T05:12:55.691 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:12:55.691 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":63,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5505,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/211479572","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":211479572},{"type":"v1","addr":"172.21.15.42:6835","nonce":211479572}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":63}],"filesystems":[{"mdsmap":{"epoch":62,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:12:48.374585+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5391},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5391":{"gid":5391,"name":"b","rank":0,"incarnation":59,"state":"up:active","state_seq":23,"addr":"172.21.15.40:6833/814397242","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":814397242},{"type":"v1","addr":"172.21.15.40:6833","nonce":814397242}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:12:55.693 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.688+0000 7f08478cb700 1 -- 172.21.15.42:0/3185903258 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0820043ea0 msgr2=0x7f08200463b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:12:55.693 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.688+0000 7f08478cb700 1 --2- 172.21.15.42:0/3185903258 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0820043ea0 0x7f08200463b0 crc :-1 s=READY pgs=223 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:12:55.693 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.688+0000 7f08478cb700 1 -- 172.21.15.42:0/3185903258 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0820003ab0 msgr2=0x7f0820004420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:12:55.694 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.688+0000 7f0846669700 1 -- 172.21.15.42:0/3185903258 reap_dead start 2019-12-05T05:12:55.694 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.688+0000 7f08478cb700 1 --2- 172.21.15.42:0/3185903258 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0820003ab0 0x7f0820004420 secure :-1 s=READY pgs=490 cs=0 l=1 rx=0x7f08200043b0 tx=0x7f0838011070).stop 2019-12-05T05:12:55.694 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.688+0000 7f08478cb700 1 -- 172.21.15.42:0/3185903258 shutdown_connections 2019-12-05T05:12:55.694 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.688+0000 7f08478cb700 1 --2- 172.21.15.42:0/3185903258 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0820003ab0 0x7f0820004420 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:12:55.694 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.688+0000 7f08478cb700 1 -- 172.21.15.42:0/3185903258 shutdown_connections 2019-12-05T05:12:55.694 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.688+0000 7f08478cb700 1 -- 172.21.15.42:0/3185903258 wait complete. 2019-12-05T05:12:55.694 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:55.688+0000 7f08478cb700 1 -- 172.21.15.42:0/3185903258 >> 172.21.15.42:0/3185903258 conn(0x7f0840116160 msgr2=0x7f0840059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:12:55.703 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T05:12:57.705 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:12:57.809 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:57.796+0000 7f8c6b630700 1 Processor -- start 2019-12-05T05:12:57.810 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:57.796+0000 7f8c6b630700 1 -- start start 2019-12-05T05:12:57.810 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:57.800+0000 7f8c6b630700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8c640a7740 0x7f8c6408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:12:57.810 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:57.800+0000 7f8c6b630700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c640a0710 con 0x7f8c6409fae0 2019-12-05T05:12:57.810 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:57.800+0000 7f8c6b630700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c640a9ba0 con 0x7f8c6407e720 2019-12-05T05:12:57.810 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:57.800+0000 7f8c6b630700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c640a0b00 con 0x7f8c640a7740 2019-12-05T05:12:57.810 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:57.800+0000 7f8c693cc700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f8c6407e720 0x7f8c6409f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:58136/0 (socket says 172.21.15.42:58136) 2019-12-05T05:12:57.810 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:57.800+0000 7f8c693cc700 1 -- 172.21.15.42:0/3302257367 learned_addr learned my addr 172.21.15.42:0/3302257367 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:12:57.811 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:57.800+0000 7f8c69bcd700 1 --2- 172.21.15.42:0/3302257367 >> v2:172.21.15.40:3300/0 conn(0x7f8c640a7740 0x7f8c6408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:12:57.811 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:57.800+0000 7f8c68bcb700 1 -- 172.21.15.42:0/3302257367 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2963471889 0 0) 0x7f8c640a9ba0 con 0x7f8c6407e720 2019-12-05T05:12:57.811 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:57.800+0000 7f8c68bcb700 1 -- 172.21.15.42:0/3302257367 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c500030b0 con 0x7f8c6407e720 2019-12-05T05:12:57.811 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:57.800+0000 7f8c68bcb700 1 -- 172.21.15.42:0/3302257367 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 856601521 0 0) 0x7f8c500030b0 con 0x7f8c6407e720 2019-12-05T05:12:57.812 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:57.800+0000 7f8c69bcd700 1 -- 172.21.15.42:0/3302257367 >> v1:172.21.15.42:6789/0 conn(0x7f8c6407e720 legacy=0x7f8c6409f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:12:57.812 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:57.800+0000 7f8c69bcd700 1 -- 172.21.15.42:0/3302257367 >> v1:172.21.15.40:6789/0 conn(0x7f8c6409fae0 legacy=0x7f8c6414efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:12:57.812 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:57.800+0000 7f8c69bcd700 1 -- 172.21.15.42:0/3302257367 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c6407eb50 con 0x7f8c640a7740 2019-12-05T05:12:57.812 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:57.800+0000 7f8c69bcd700 1 --2- 172.21.15.42:0/3302257367 >> v2:172.21.15.40:3300/0 conn(0x7f8c640a7740 0x7f8c6408df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f8c60008b10 tx=0x7f8c54001990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:57.812 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:57.800+0000 7f8c69bcd700 1 --2- 172.21.15.42:0/3302257367 >> v2:172.21.15.40:3300/0 conn(0x7f8c640a7740 0x7f8c6408df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f8c60008b10 tx=0x7f8c54001990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:57.813 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:57.800+0000 7f8c69bcd700 1 --2- 172.21.15.42:0/3302257367 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8c640a7740 0x7f8c6408df80 secure :-1 s=READY pgs=386 cs=0 l=1 rx=0x7f8c60008b10 tx=0x7f8c54001990).ready entity=mon.0 client_cookie=f64ec238002a9af9 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:12:57.813 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:57.800+0000 7f8c69bcd700 1 --2- 172.21.15.42:0/3302257367 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8c640a7740 0x7f8c6408df80 secure :-1 s=THROTTLE_DONE pgs=386 cs=0 l=1 rx=0x7f8c60008b10 tx=0x7f8c54001990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:57.813 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:57.800+0000 7f8c68bcb700 1 -- 172.21.15.42:0/3302257367 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8c54001750 con 0x7f8c640a7740 2019-12-05T05:12:57.813 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:57.800+0000 7f8c69bcd700 1 --2- 172.21.15.42:0/3302257367 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8c640a7740 0x7f8c6408df80 secure :-1 s=THROTTLE_DONE pgs=386 cs=0 l=1 rx=0x7f8c60008b10 tx=0x7f8c54001990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:57.814 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:57.800+0000 7f8c69bcd700 1 --2- 172.21.15.42:0/3302257367 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8c640a7740 0x7f8c6408df80 secure :-1 s=THROTTLE_DONE pgs=386 cs=0 l=1 rx=0x7f8c60008b10 tx=0x7f8c54001990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:57.814 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:57.800+0000 7f8c68bcb700 1 -- 172.21.15.42:0/3302257367 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c54001560 con 0x7f8c640a7740 2019-12-05T05:12:57.814 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:57.800+0000 7f8c68bcb700 1 -- 172.21.15.42:0/3302257367 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8c60011ac0 con 0x7f8c640a7740 2019-12-05T05:12:57.814 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:57.800+0000 7f8c6b630700 1 -- 172.21.15.42:0/3302257367 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8c640a7740 msgr2=0x7f8c6408df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:12:57.814 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:57.800+0000 7f8c6b630700 1 --2- 172.21.15.42:0/3302257367 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8c640a7740 0x7f8c6408df80 secure :-1 s=READY pgs=386 cs=0 l=1 rx=0x7f8c60008b10 tx=0x7f8c54001990).stop 2019-12-05T05:12:57.814 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:57.804+0000 7f8c6b630700 1 -- 172.21.15.42:0/3302257367 shutdown_connections 2019-12-05T05:12:57.815 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:57.804+0000 7f8c6b630700 1 --2- 172.21.15.42:0/3302257367 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8c640a7740 0x7f8c6408df80 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:12:57.815 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:57.804+0000 7f8c6b630700 1 -- 172.21.15.42:0/3302257367 shutdown_connections 2019-12-05T05:12:57.815 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:57.804+0000 7f8c6b630700 1 -- 172.21.15.42:0/3302257367 wait complete. 2019-12-05T05:12:57.815 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:57.804+0000 7f8c6b630700 1 -- 172.21.15.42:0/3302257367 >> 172.21.15.42:0/3302257367 conn(0x7f8c640aabe0 msgr2=0x7f8c640a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:12:57.821 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:57.804+0000 7f8c6b630700 1 Processor -- start 2019-12-05T05:12:57.821 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:57.804+0000 7f8c6b630700 1 -- start start 2019-12-05T05:12:57.822 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:57.808+0000 7f8c6b630700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f8c641170a0 0x7f8c640a9770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:12:57.822 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:57.808+0000 7f8c6b630700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8c64138b20 0x7f8c6413af60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:12:57.822 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:57.808+0000 7f8c6b630700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c6411edc0 con 0x7f8c6411bab0 2019-12-05T05:12:57.822 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:57.808+0000 7f8c6b630700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c6411fcb0 con 0x7f8c641170a0 2019-12-05T05:12:57.823 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:57.808+0000 7f8c6b630700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c6411ff50 con 0x7f8c64138b20 2019-12-05T05:12:57.823 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:57.808+0000 7f8c693cc700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f8c641170a0 0x7f8c640a9770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:12:57.823 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:57.808+0000 7f8c693cc700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f8c641170a0 0x7f8c640a9770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:58104/0 (socket says 172.21.15.42:58104) 2019-12-05T05:12:57.823 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:57.808+0000 7f8c693cc700 1 -- 172.21.15.42:0/2591806313 learned_addr learned my addr 172.21.15.42:0/2591806313 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:12:57.824 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:57.808+0000 7f8c6a3ce700 1 --2- 172.21.15.42:0/2591806313 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8c64138b20 0x7f8c6413af60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:12:57.824 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:57.808+0000 7f8c693cc700 1 -- 172.21.15.42:0/2591806313 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8c64138b20 msgr2=0x7f8c6413af60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:12:57.837 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:57.808+0000 7f8c693cc700 1 --2- 172.21.15.42:0/2591806313 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8c64138b20 0x7f8c6413af60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:12:57.837 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:57.808+0000 7f8c693cc700 1 -- 172.21.15.42:0/2591806313 >> v1:172.21.15.40:6789/0 conn(0x7f8c6411bab0 legacy=0x7f8c6411e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:12:57.837 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:57.808+0000 7f8c693cc700 1 -- 172.21.15.42:0/2591806313 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c640a9cc0 con 0x7f8c641170a0 2019-12-05T05:12:57.837 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:57.808+0000 7f8c6a3ce700 1 --2- 172.21.15.42:0/2591806313 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8c64138b20 0x7f8c6413af60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T05:12:57.837 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:57.808+0000 7f8c693cc700 1 --2- 172.21.15.42:0/2591806313 >> v2:172.21.15.42:3300/0 conn(0x7f8c641170a0 0x7f8c640a9770 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f8c641204c0 tx=0x7f8c54012cd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:57.837 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:57.808+0000 7f8c693cc700 1 --2- 172.21.15.42:0/2591806313 >> v2:172.21.15.42:3300/0 conn(0x7f8c641170a0 0x7f8c640a9770 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f8c641204c0 tx=0x7f8c54012cd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:57.838 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:57.808+0000 7f8c693cc700 1 --2- 172.21.15.42:0/2591806313 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8c641170a0 0x7f8c640a9770 secure :-1 s=READY pgs=492 cs=0 l=1 rx=0x7f8c641204c0 tx=0x7f8c54012cd0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:12:57.838 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:57.808+0000 7f8c693cc700 1 --2- 172.21.15.42:0/2591806313 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8c641170a0 0x7f8c640a9770 secure :-1 s=THROTTLE_DONE pgs=492 cs=0 l=1 rx=0x7f8c641204c0 tx=0x7f8c54012cd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:57.838 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:57.808+0000 7f8c5a7fc700 1 -- 172.21.15.42:0/2591806313 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8c54014610 con 0x7f8c641170a0 2019-12-05T05:12:57.838 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:57.808+0000 7f8c693cc700 1 --2- 172.21.15.42:0/2591806313 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8c641170a0 0x7f8c640a9770 secure :-1 s=THROTTLE_DONE pgs=492 cs=0 l=1 rx=0x7f8c641204c0 tx=0x7f8c54012cd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:57.838 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:57.808+0000 7f8c693cc700 1 --2- 172.21.15.42:0/2591806313 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8c641170a0 0x7f8c640a9770 secure :-1 s=THROTTLE_DONE pgs=492 cs=0 l=1 rx=0x7f8c641204c0 tx=0x7f8c54012cd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:57.838 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:57.808+0000 7f8c5a7fc700 1 -- 172.21.15.42:0/2591806313 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c5400d160 con 0x7f8c641170a0 2019-12-05T05:12:57.838 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:57.808+0000 7f8c6b630700 1 -- 172.21.15.42:0/2591806313 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c641174d0 con 0x7f8c641170a0 2019-12-05T05:12:57.839 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:57.808+0000 7f8c5a7fc700 1 -- 172.21.15.42:0/2591806313 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8c54014bf0 con 0x7f8c641170a0 2019-12-05T05:12:57.839 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:57.808+0000 7f8c6b630700 1 -- 172.21.15.42:0/2591806313 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c64118250 con 0x7f8c641170a0 2019-12-05T05:12:57.839 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:57.808+0000 7f8c693cc700 1 --2- 172.21.15.42:0/2591806313 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8c641170a0 0x7f8c640a9770 secure :-1 s=THROTTLE_DONE pgs=492 cs=0 l=1 rx=0x7f8c641204c0 tx=0x7f8c54012cd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:57.839 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:57.812+0000 7f8c693cc700 1 --2- 172.21.15.42:0/2591806313 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8c641170a0 0x7f8c640a9770 secure :-1 s=THROTTLE_DONE pgs=492 cs=0 l=1 rx=0x7f8c641204c0 tx=0x7f8c54012cd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:57.839 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:57.812+0000 7f8c5a7fc700 1 -- 172.21.15.42:0/2591806313 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f8c54004430 con 0x7f8c641170a0 2019-12-05T05:12:57.839 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:57.812+0000 7f8c5a7fc700 1 --2- 172.21.15.42:0/2591806313 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8c4c03ad10 0x7f8c4c03d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:12:57.839 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:57.812+0000 7f8c5a7fc700 1 -- 172.21.15.42:0/2591806313 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6547+0+0 (secure 0 0 0) 0x7f8c5404d140 con 0x7f8c641170a0 2019-12-05T05:12:57.840 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:57.812+0000 7f8c69bcd700 1 --2- 172.21.15.42:0/2591806313 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8c4c03ad10 0x7f8c4c03d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:12:57.840 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:57.812+0000 7f8c69bcd700 1 --2- 172.21.15.42:0/2591806313 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8c4c03ad10 0x7f8c4c03d220 crc :-1 s=READY pgs=224 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:12:57.840 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:57.812+0000 7f8c6b630700 1 -- 172.21.15.42:0/2591806313 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c6405d110 con 0x7f8c641170a0 2019-12-05T05:12:57.853 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:57.820+0000 7f8c693cc700 1 --2- 172.21.15.42:0/2591806313 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8c641170a0 0x7f8c640a9770 secure :-1 s=THROTTLE_DONE pgs=492 cs=0 l=1 rx=0x7f8c641204c0 tx=0x7f8c54012cd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:57.853 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:57.820+0000 7f8c5a7fc700 1 -- 172.21.15.42:0/2591806313 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f8c54034180 con 0x7f8c641170a0 2019-12-05T05:12:57.999 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:57.992+0000 7f8c6b630700 1 -- 172.21.15.42:0/2591806313 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8c640aadf0 con 0x7f8c641170a0 2019-12-05T05:12:58.001 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:57.992+0000 7f8c693cc700 1 --2- 172.21.15.42:0/2591806313 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8c641170a0 0x7f8c640a9770 secure :-1 s=THROTTLE_DONE pgs=492 cs=0 l=1 rx=0x7f8c641204c0 tx=0x7f8c54012cd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:12:58.002 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:57.992+0000 7f8c5a7fc700 1 -- 172.21.15.42:0/2591806313 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 63 v63) v1 ==== 94+0+2102 (secure 0 0 0) 0x7f8c540343e0 con 0x7f8c641170a0 2019-12-05T05:12:58.002 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 63 2019-12-05T05:12:58.002 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:12:58.003 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":63,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5505,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/211479572","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":211479572},{"type":"v1","addr":"172.21.15.42:6835","nonce":211479572}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":63}],"filesystems":[{"mdsmap":{"epoch":62,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:12:48.374585+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5391},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5391":{"gid":5391,"name":"b","rank":0,"incarnation":59,"state":"up:active","state_seq":23,"addr":"172.21.15.40:6833/814397242","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":814397242},{"type":"v1","addr":"172.21.15.40:6833","nonce":814397242}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:12:58.005 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:57.996+0000 7f8c6b630700 1 -- 172.21.15.42:0/2591806313 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8c4c03ad10 msgr2=0x7f8c4c03d220 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:12:58.006 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:57.996+0000 7f8c6b630700 1 --2- 172.21.15.42:0/2591806313 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8c4c03ad10 0x7f8c4c03d220 crc :-1 s=READY pgs=224 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:12:58.006 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:57.996+0000 7f8c6b630700 1 -- 172.21.15.42:0/2591806313 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8c641170a0 msgr2=0x7f8c640a9770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:12:58.006 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:57.996+0000 7f8c6b630700 1 --2- 172.21.15.42:0/2591806313 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8c641170a0 0x7f8c640a9770 secure :-1 s=READY pgs=492 cs=0 l=1 rx=0x7f8c641204c0 tx=0x7f8c54012cd0).stop 2019-12-05T05:12:58.006 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:57.996+0000 7f8c6b630700 1 -- 172.21.15.42:0/2591806313 shutdown_connections 2019-12-05T05:12:58.007 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:57.996+0000 7f8c6b630700 1 --2- 172.21.15.42:0/2591806313 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8c4c03ad10 0x7f8c4c03d220 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:12:58.007 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:57.996+0000 7f8c6b630700 1 --2- 172.21.15.42:0/2591806313 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8c64138b20 0x7f8c6413af60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:12:58.007 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:57.996+0000 7f8c6b630700 1 --2- 172.21.15.42:0/2591806313 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8c641170a0 0x7f8c640a9770 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:12:58.007 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:57.996+0000 7f8c6b630700 1 -- 172.21.15.42:0/2591806313 shutdown_connections 2019-12-05T05:12:58.007 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:57.996+0000 7f8c6b630700 1 -- 172.21.15.42:0/2591806313 wait complete. 2019-12-05T05:12:58.008 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:12:57.996+0000 7f8c6b630700 1 -- 172.21.15.42:0/2591806313 >> 172.21.15.42:0/2591806313 conn(0x7f8c640a7f90 msgr2=0x7f8c6408eb70 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:12:58.012 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T05:13:00.014 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:13:00.120 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:00.108+0000 7fe9e5d19700 1 Processor -- start 2019-12-05T05:13:00.120 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:00.108+0000 7fe9e5d19700 1 -- start start 2019-12-05T05:13:00.120 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:00.108+0000 7fe9e5d19700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe9e00a7740 0x7fe9e008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:13:00.120 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:00.108+0000 7fe9e5d19700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe9e007e720 0x7fe9e007eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:13:00.120 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:00.108+0000 7fe9e5d19700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe9e009f2e0 0x7fe9e009f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:13:00.121 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:00.108+0000 7fe9e5d19700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9e00a0b00 con 0x7fe9e00a7740 2019-12-05T05:13:00.121 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:00.108+0000 7fe9e5d19700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9e008e5b0 con 0x7fe9e009f2e0 2019-12-05T05:13:00.121 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:00.108+0000 7fe9e5d19700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9e008e850 con 0x7fe9e007e720 2019-12-05T05:13:00.121 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:00.108+0000 7fe9e4ab7700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe9e009f2e0 0x7fe9e009f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:13:00.121 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:00.108+0000 7fe9e4ab7700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe9e009f2e0 0x7fe9e009f710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:58116/0 (socket says 172.21.15.42:58116) 2019-12-05T05:13:00.121 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:00.108+0000 7fe9e4ab7700 1 -- 172.21.15.42:0/4094167595 learned_addr learned my addr 172.21.15.42:0/4094167595 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:13:00.121 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:00.108+0000 7fe9d77fe700 1 --2- 172.21.15.42:0/4094167595 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe9e007e720 0x7fe9e007eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:13:00.122 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:00.108+0000 7fe9dffff700 1 --2- 172.21.15.42:0/4094167595 >> v2:172.21.15.40:3300/0 conn(0x7fe9e00a7740 0x7fe9e008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:13:00.122 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:00.108+0000 7fe9e4ab7700 1 -- 172.21.15.42:0/4094167595 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe9e007e720 msgr2=0x7fe9e007eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:13:00.122 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:00.108+0000 7fe9e4ab7700 1 --2- 172.21.15.42:0/4094167595 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe9e007e720 0x7fe9e007eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:13:00.122 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:00.108+0000 7fe9e4ab7700 1 -- 172.21.15.42:0/4094167595 >> v2:172.21.15.40:3300/0 conn(0x7fe9e00a7740 msgr2=0x7fe9e008df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:13:00.122 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:00.108+0000 7fe9e4ab7700 1 --2- 172.21.15.42:0/4094167595 >> v2:172.21.15.40:3300/0 conn(0x7fe9e00a7740 0x7fe9e008df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:13:00.122 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:00.108+0000 7fe9e4ab7700 1 -- 172.21.15.42:0/4094167595 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9e007f0a0 con 0x7fe9e009f2e0 2019-12-05T05:13:00.122 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:00.112+0000 7fe9e4ab7700 1 --2- 172.21.15.42:0/4094167595 >> v2:172.21.15.42:3300/0 conn(0x7fe9e009f2e0 0x7fe9e009f710 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fe9e008e770 tx=0x7fe9d8010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:00.123 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:00.112+0000 7fe9e4ab7700 1 --2- 172.21.15.42:0/4094167595 >> v2:172.21.15.42:3300/0 conn(0x7fe9e009f2e0 0x7fe9e009f710 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fe9e008e770 tx=0x7fe9d8010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:00.123 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:00.112+0000 7fe9e4ab7700 1 --2- 172.21.15.42:0/4094167595 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe9e009f2e0 0x7fe9e009f710 secure :-1 s=READY pgs=493 cs=0 l=1 rx=0x7fe9e008e770 tx=0x7fe9d8010130).ready entity=mon.1 client_cookie=1921f07da1e059db server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:13:00.124 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:00.112+0000 7fe9e4ab7700 1 --2- 172.21.15.42:0/4094167595 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe9e009f2e0 0x7fe9e009f710 secure :-1 s=THROTTLE_DONE pgs=493 cs=0 l=1 rx=0x7fe9e008e770 tx=0x7fe9d8010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:00.124 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:00.112+0000 7fe9df7fe700 1 -- 172.21.15.42:0/4094167595 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe9d8010de0 con 0x7fe9e009f2e0 2019-12-05T05:13:00.124 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:00.112+0000 7fe9e4ab7700 1 --2- 172.21.15.42:0/4094167595 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe9e009f2e0 0x7fe9e009f710 secure :-1 s=THROTTLE_DONE pgs=493 cs=0 l=1 rx=0x7fe9e008e770 tx=0x7fe9d8010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:00.124 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:00.112+0000 7fe9df7fe700 1 -- 172.21.15.42:0/4094167595 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9d8010a00 con 0x7fe9e009f2e0 2019-12-05T05:13:00.125 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:00.112+0000 7fe9e4ab7700 1 --2- 172.21.15.42:0/4094167595 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe9e009f2e0 0x7fe9e009f710 secure :-1 s=THROTTLE_DONE pgs=493 cs=0 l=1 rx=0x7fe9e008e770 tx=0x7fe9d8010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:00.125 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:00.112+0000 7fe9df7fe700 1 -- 172.21.15.42:0/4094167595 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe9d80114a0 con 0x7fe9e009f2e0 2019-12-05T05:13:00.125 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:00.112+0000 7fe9e5d19700 1 -- 172.21.15.42:0/4094167595 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe9e009f2e0 msgr2=0x7fe9e009f710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:13:00.125 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:00.112+0000 7fe9e5d19700 1 --2- 172.21.15.42:0/4094167595 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe9e009f2e0 0x7fe9e009f710 secure :-1 s=READY pgs=493 cs=0 l=1 rx=0x7fe9e008e770 tx=0x7fe9d8010130).stop 2019-12-05T05:13:00.125 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:00.112+0000 7fe9e5d19700 1 -- 172.21.15.42:0/4094167595 shutdown_connections 2019-12-05T05:13:00.125 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:00.112+0000 7fe9e5d19700 1 --2- 172.21.15.42:0/4094167595 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe9e009f2e0 0x7fe9e009f710 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:13:00.125 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:00.112+0000 7fe9e5d19700 1 --2- 172.21.15.42:0/4094167595 >> v2:172.21.15.40:3300/0 conn(0x7fe9e00a7740 0x7fe9e008df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:13:00.126 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:00.112+0000 7fe9e5d19700 1 --2- 172.21.15.42:0/4094167595 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe9e007e720 0x7fe9e007eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:13:00.126 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:00.112+0000 7fe9e5d19700 1 -- 172.21.15.42:0/4094167595 shutdown_connections 2019-12-05T05:13:00.126 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:00.112+0000 7fe9e5d19700 1 -- 172.21.15.42:0/4094167595 wait complete. 2019-12-05T05:13:00.126 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:00.112+0000 7fe9e5d19700 1 -- 172.21.15.42:0/4094167595 >> 172.21.15.42:0/4094167595 conn(0x7fe9e00aabe0 msgr2=0x7fe9e00a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:13:00.126 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:00.116+0000 7fe9e5d19700 1 Processor -- start 2019-12-05T05:13:00.126 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:00.116+0000 7fe9e5d19700 1 -- start start 2019-12-05T05:13:00.129 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:00.116+0000 7fe9e5d19700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe9e00a5190 0x7fe9e011b620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:13:00.129 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:00.116+0000 7fe9e5d19700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe9e012e990 0x7fe9e01365f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:13:00.130 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:00.116+0000 7fe9e5d19700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9e00a4b70 con 0x7fe9e011bb70 2019-12-05T05:13:00.130 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:00.116+0000 7fe9e5d19700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9e00a6b80 con 0x7fe9e012e990 2019-12-05T05:13:00.130 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:00.116+0000 7fe9e5d19700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9e00a6e20 con 0x7fe9e00a5190 2019-12-05T05:13:00.130 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:00.116+0000 7fe9dffff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe9e00a5190 0x7fe9e011b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:13:00.130 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:00.116+0000 7fe9dffff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe9e00a5190 0x7fe9e011b620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:58118/0 (socket says 172.21.15.42:58118) 2019-12-05T05:13:00.131 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:00.116+0000 7fe9d77fe700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fe9e011bb70 0x7fe9e012e190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:58158/0 (socket says 172.21.15.42:58158) 2019-12-05T05:13:00.131 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:00.116+0000 7fe9dffff700 1 -- 172.21.15.42:0/3363601682 learned_addr learned my addr 172.21.15.42:0/3363601682 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:13:00.131 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:00.116+0000 7fe9e4ab7700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe9e012e990 0x7fe9e01365f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:13:00.131 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:00.116+0000 7fe9dd7fa700 1 -- 172.21.15.42:0/3363601682 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3331482729 0 0) 0x7fe9e00a4b70 con 0x7fe9e011bb70 2019-12-05T05:13:00.131 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:00.116+0000 7fe9dffff700 1 -- 172.21.15.42:0/3363601682 >> v2:172.21.15.40:3300/0 conn(0x7fe9e012e990 msgr2=0x7fe9e01365f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:13:00.131 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:00.116+0000 7fe9dffff700 1 --2- 172.21.15.42:0/3363601682 >> v2:172.21.15.40:3300/0 conn(0x7fe9e012e990 0x7fe9e01365f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:13:00.131 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:00.116+0000 7fe9dffff700 1 -- 172.21.15.42:0/3363601682 >> v1:172.21.15.42:6789/0 conn(0x7fe9e011bb70 legacy=0x7fe9e012e190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:13:00.132 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:00.116+0000 7fe9dffff700 1 -- 172.21.15.42:0/3363601682 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9e0142530 con 0x7fe9e00a5190 2019-12-05T05:13:00.132 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:00.116+0000 7fe9dffff700 1 --2- 172.21.15.42:0/3363601682 >> v2:172.21.15.42:3300/0 conn(0x7fe9e00a5190 0x7fe9e011b620 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fe9d0003070 tx=0x7fe9d0010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:00.132 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:00.116+0000 7fe9dffff700 1 --2- 172.21.15.42:0/3363601682 >> v2:172.21.15.42:3300/0 conn(0x7fe9e00a5190 0x7fe9e011b620 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fe9d0003070 tx=0x7fe9d0010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:00.132 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:00.116+0000 7fe9dffff700 1 --2- 172.21.15.42:0/3363601682 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe9e00a5190 0x7fe9e011b620 secure :-1 s=READY pgs=495 cs=0 l=1 rx=0x7fe9d0003070 tx=0x7fe9d0010b60).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:13:00.132 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:00.116+0000 7fe9dffff700 1 --2- 172.21.15.42:0/3363601682 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe9e00a5190 0x7fe9e011b620 secure :-1 s=THROTTLE_DONE pgs=495 cs=0 l=1 rx=0x7fe9d0003070 tx=0x7fe9d0010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:00.132 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:00.116+0000 7fe9dd7fa700 1 -- 172.21.15.42:0/3363601682 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe9d0005540 con 0x7fe9e00a5190 2019-12-05T05:13:00.132 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:00.116+0000 7fe9dffff700 1 --2- 172.21.15.42:0/3363601682 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe9e00a5190 0x7fe9e011b620 secure :-1 s=THROTTLE_DONE pgs=495 cs=0 l=1 rx=0x7fe9d0003070 tx=0x7fe9d0010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:00.133 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:00.116+0000 7fe9dffff700 1 --2- 172.21.15.42:0/3363601682 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe9e00a5190 0x7fe9e011b620 secure :-1 s=THROTTLE_DONE pgs=495 cs=0 l=1 rx=0x7fe9d0003070 tx=0x7fe9d0010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:00.133 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:00.120+0000 7fe9dd7fa700 1 -- 172.21.15.42:0/3363601682 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9d0007150 con 0x7fe9e00a5190 2019-12-05T05:13:00.133 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:00.120+0000 7fe9e5d19700 1 -- 172.21.15.42:0/3363601682 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9e011c830 con 0x7fe9e00a5190 2019-12-05T05:13:00.133 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:00.120+0000 7fe9dd7fa700 1 -- 172.21.15.42:0/3363601682 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe9d0005b20 con 0x7fe9e00a5190 2019-12-05T05:13:00.133 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:00.120+0000 7fe9e5d19700 1 -- 172.21.15.42:0/3363601682 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9e00a70c0 con 0x7fe9e00a5190 2019-12-05T05:13:00.133 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:00.120+0000 7fe9dffff700 1 --2- 172.21.15.42:0/3363601682 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe9e00a5190 0x7fe9e011b620 secure :-1 s=THROTTLE_DONE pgs=495 cs=0 l=1 rx=0x7fe9d0003070 tx=0x7fe9d0010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:00.133 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:00.120+0000 7fe9dd7fa700 1 -- 172.21.15.42:0/3363601682 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fe9d00123c0 con 0x7fe9e00a5190 2019-12-05T05:13:00.134 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:00.120+0000 7fe9dffff700 1 --2- 172.21.15.42:0/3363601682 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe9e00a5190 0x7fe9e011b620 secure :-1 s=THROTTLE_DONE pgs=495 cs=0 l=1 rx=0x7fe9d0003070 tx=0x7fe9d0010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:00.134 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:00.120+0000 7fe9dd7fa700 1 --2- 172.21.15.42:0/3363601682 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe9c403acc0 0x7fe9c403d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:13:00.134 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:00.120+0000 7fe9dd7fa700 1 -- 172.21.15.42:0/3363601682 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6547+0+0 (secure 0 0 0) 0x7fe9d004ae20 con 0x7fe9e00a5190 2019-12-05T05:13:00.134 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:00.120+0000 7fe9d77fe700 1 --2- 172.21.15.42:0/3363601682 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe9c403acc0 0x7fe9c403d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:13:00.134 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:00.120+0000 7fe9d77fe700 1 --2- 172.21.15.42:0/3363601682 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe9c403acc0 0x7fe9c403d1d0 crc :-1 s=READY pgs=225 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:13:00.134 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:00.124+0000 7fe9e5d19700 1 -- 172.21.15.42:0/3363601682 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9e005d110 con 0x7fe9e00a5190 2019-12-05T05:13:00.135 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:00.128+0000 7fe9dffff700 1 --2- 172.21.15.42:0/3363601682 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe9e00a5190 0x7fe9e011b620 secure :-1 s=THROTTLE_DONE pgs=495 cs=0 l=1 rx=0x7fe9d0003070 tx=0x7fe9d0010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:00.136 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:00.128+0000 7fe9dd7fa700 1 -- 172.21.15.42:0/3363601682 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fe9d0031ea0 con 0x7fe9e00a5190 2019-12-05T05:13:00.301 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:00.296+0000 7fe9e5d19700 1 -- 172.21.15.42:0/3363601682 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe9e005b8c0 con 0x7fe9e00a5190 2019-12-05T05:13:00.301 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:00.296+0000 7fe9dffff700 1 --2- 172.21.15.42:0/3363601682 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe9e00a5190 0x7fe9e011b620 secure :-1 s=THROTTLE_DONE pgs=495 cs=0 l=1 rx=0x7fe9d0003070 tx=0x7fe9d0010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:00.302 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:00.296+0000 7fe9dd7fa700 1 -- 172.21.15.42:0/3363601682 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 63 v63) v1 ==== 94+0+2102 (secure 0 0 0) 0x7fe9d0032100 con 0x7fe9e00a5190 2019-12-05T05:13:00.302 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 63 2019-12-05T05:13:00.302 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:13:00.302 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":63,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5505,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/211479572","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":211479572},{"type":"v1","addr":"172.21.15.42:6835","nonce":211479572}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":63}],"filesystems":[{"mdsmap":{"epoch":62,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:12:48.374585+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5391},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5391":{"gid":5391,"name":"b","rank":0,"incarnation":59,"state":"up:active","state_seq":23,"addr":"172.21.15.40:6833/814397242","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":814397242},{"type":"v1","addr":"172.21.15.40:6833","nonce":814397242}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:13:00.304 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:00.300+0000 7fe9e5d19700 1 -- 172.21.15.42:0/3363601682 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe9c403acc0 msgr2=0x7fe9c403d1d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:13:00.304 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:00.300+0000 7fe9e5d19700 1 --2- 172.21.15.42:0/3363601682 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe9c403acc0 0x7fe9c403d1d0 crc :-1 s=READY pgs=225 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:13:00.305 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:00.300+0000 7fe9e5d19700 1 -- 172.21.15.42:0/3363601682 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe9e00a5190 msgr2=0x7fe9e011b620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:13:00.305 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:00.300+0000 7fe9e5d19700 1 --2- 172.21.15.42:0/3363601682 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe9e00a5190 0x7fe9e011b620 secure :-1 s=READY pgs=495 cs=0 l=1 rx=0x7fe9d0003070 tx=0x7fe9d0010b60).stop 2019-12-05T05:13:00.305 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:00.300+0000 7fe9e5d19700 1 -- 172.21.15.42:0/3363601682 shutdown_connections 2019-12-05T05:13:00.305 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:00.300+0000 7fe9e5d19700 1 --2- 172.21.15.42:0/3363601682 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe9c403acc0 0x7fe9c403d1d0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:13:00.306 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:00.300+0000 7fe9e5d19700 1 --2- 172.21.15.42:0/3363601682 >> v2:172.21.15.40:3300/0 conn(0x7fe9e012e990 0x7fe9e01365f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:13:00.306 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:00.300+0000 7fe9e5d19700 1 --2- 172.21.15.42:0/3363601682 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe9e00a5190 0x7fe9e011b620 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:13:00.306 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:00.300+0000 7fe9e5d19700 1 -- 172.21.15.42:0/3363601682 shutdown_connections 2019-12-05T05:13:00.306 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:00.300+0000 7fe9e5d19700 1 -- 172.21.15.42:0/3363601682 wait complete. 2019-12-05T05:13:00.306 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:00.300+0000 7fe9e5d19700 1 -- 172.21.15.42:0/3363601682 >> 172.21.15.42:0/3363601682 conn(0x7fe9e00a7740 msgr2=0x7fe9e009f380 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:13:00.319 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T05:13:02.320 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:13:02.429 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:02.412+0000 7f820bca7700 1 Processor -- start 2019-12-05T05:13:02.429 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:02.412+0000 7f820bca7700 1 -- start start 2019-12-05T05:13:02.429 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:02.416+0000 7f820bca7700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f82040a7740 0x7f820408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:13:02.429 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:02.416+0000 7f820bca7700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f820407e720 0x7f820407eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:13:02.430 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:02.416+0000 7f820bca7700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82040a0710 con 0x7f820409f2e0 2019-12-05T05:13:02.430 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:02.416+0000 7f820bca7700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82040a0b00 con 0x7f820407e720 2019-12-05T05:13:02.430 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:02.416+0000 7f820bca7700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f820408e5b0 con 0x7f82040a7740 2019-12-05T05:13:02.430 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:02.416+0000 7f820a244700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f82040a7740 0x7f820408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:13:02.431 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:02.416+0000 7f820a244700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f82040a7740 0x7f820408df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:58134/0 (socket says 172.21.15.42:58134) 2019-12-05T05:13:02.431 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:02.416+0000 7f820a244700 1 -- 172.21.15.42:0/1531919670 learned_addr learned my addr 172.21.15.42:0/1531919670 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:13:02.431 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:02.416+0000 7f8209a43700 1 --2- 172.21.15.42:0/1531919670 >> v2:172.21.15.40:3300/0 conn(0x7f820407e720 0x7f820407eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:13:02.431 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:02.416+0000 7f820a244700 1 -- 172.21.15.42:0/1531919670 >> v2:172.21.15.40:3300/0 conn(0x7f820407e720 msgr2=0x7f820407eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:13:02.431 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:02.416+0000 7f820a244700 1 --2- 172.21.15.42:0/1531919670 >> v2:172.21.15.40:3300/0 conn(0x7f820407e720 0x7f820407eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:13:02.432 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:02.416+0000 7f820a244700 1 -- 172.21.15.42:0/1531919670 >> v1:172.21.15.40:6789/0 conn(0x7f820409f2e0 legacy=0x7f820409f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:13:02.432 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:02.416+0000 7f820a244700 1 -- 172.21.15.42:0/1531919670 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f820407f0a0 con 0x7f82040a7740 2019-12-05T05:13:02.432 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:02.416+0000 7f820a244700 1 --2- 172.21.15.42:0/1531919670 >> v2:172.21.15.42:3300/0 conn(0x7f82040a7740 0x7f820408df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f82000032b0 tx=0x7f82000100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:02.432 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:02.416+0000 7f820a244700 1 --2- 172.21.15.42:0/1531919670 >> v2:172.21.15.42:3300/0 conn(0x7f82040a7740 0x7f820408df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f82000032b0 tx=0x7f82000100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:02.432 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:02.416+0000 7f820a244700 1 --2- 172.21.15.42:0/1531919670 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f82040a7740 0x7f820408df80 secure :-1 s=READY pgs=496 cs=0 l=1 rx=0x7f82000032b0 tx=0x7f82000100b0).ready entity=mon.1 client_cookie=9c758a45dfbe3644 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:13:02.432 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:02.416+0000 7f820a244700 1 --2- 172.21.15.42:0/1531919670 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f82040a7740 0x7f820408df80 secure :-1 s=THROTTLE_DONE pgs=496 cs=0 l=1 rx=0x7f82000032b0 tx=0x7f82000100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:02.432 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:02.416+0000 7f820a244700 1 --2- 172.21.15.42:0/1531919670 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f82040a7740 0x7f820408df80 secure :-1 s=THROTTLE_DONE pgs=496 cs=0 l=1 rx=0x7f82000032b0 tx=0x7f82000100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:02.433 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:02.416+0000 7f8209242700 1 -- 172.21.15.42:0/1531919670 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8200011dd0 con 0x7f82040a7740 2019-12-05T05:13:02.433 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:02.416+0000 7f820a244700 1 --2- 172.21.15.42:0/1531919670 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f82040a7740 0x7f820408df80 secure :-1 s=THROTTLE_DONE pgs=496 cs=0 l=1 rx=0x7f82000032b0 tx=0x7f82000100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:02.433 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:02.416+0000 7f8209242700 1 -- 172.21.15.42:0/1531919670 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82000119f0 con 0x7f82040a7740 2019-12-05T05:13:02.433 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:02.416+0000 7f8209242700 1 -- 172.21.15.42:0/1531919670 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f820000a540 con 0x7f82040a7740 2019-12-05T05:13:02.433 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:02.416+0000 7f820bca7700 1 -- 172.21.15.42:0/1531919670 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f82040a7740 msgr2=0x7f820408df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:13:02.434 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:02.416+0000 7f820bca7700 1 --2- 172.21.15.42:0/1531919670 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f82040a7740 0x7f820408df80 secure :-1 s=READY pgs=496 cs=0 l=1 rx=0x7f82000032b0 tx=0x7f82000100b0).stop 2019-12-05T05:13:02.434 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:02.420+0000 7f820bca7700 1 -- 172.21.15.42:0/1531919670 shutdown_connections 2019-12-05T05:13:02.435 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:02.420+0000 7f820bca7700 1 --2- 172.21.15.42:0/1531919670 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f82040a7740 0x7f820408df80 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:13:02.435 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:02.420+0000 7f820bca7700 1 --2- 172.21.15.42:0/1531919670 >> v2:172.21.15.40:3300/0 conn(0x7f820407e720 0x7f820407eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:13:02.435 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:02.420+0000 7f820bca7700 1 -- 172.21.15.42:0/1531919670 shutdown_connections 2019-12-05T05:13:02.435 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:02.420+0000 7f820bca7700 1 -- 172.21.15.42:0/1531919670 wait complete. 2019-12-05T05:13:02.435 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:02.420+0000 7f820bca7700 1 -- 172.21.15.42:0/1531919670 >> 172.21.15.42:0/1531919670 conn(0x7f82040aabe0 msgr2=0x7f82040a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:13:02.435 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:02.420+0000 7f820bca7700 1 Processor -- start 2019-12-05T05:13:02.450 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:02.420+0000 7f820bca7700 1 -- start start 2019-12-05T05:13:02.451 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:02.424+0000 7f820bca7700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8204123690 0x7f8204116ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:13:02.451 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:02.424+0000 7f820bca7700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8204121960 con 0x7f820407af50 2019-12-05T05:13:02.451 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:02.424+0000 7f820bca7700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8204118bb0 con 0x7f820407b380 2019-12-05T05:13:02.451 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:02.424+0000 7f820bca7700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8204118670 con 0x7f8204123690 2019-12-05T05:13:02.451 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:02.424+0000 7f820aa45700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f820407b380 0x7f8204138930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:58180/0 (socket says 172.21.15.42:58180) 2019-12-05T05:13:02.451 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:02.424+0000 7f820aa45700 1 -- 172.21.15.42:0/2253209354 learned_addr learned my addr 172.21.15.42:0/2253209354 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:13:02.458 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:02.424+0000 7f820a244700 1 --2- 172.21.15.42:0/2253209354 >> v2:172.21.15.40:3300/0 conn(0x7f8204123690 0x7f8204116ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:13:02.461 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:02.424+0000 7f81faffd700 1 -- 172.21.15.42:0/2253209354 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2115012988 0 0) 0x7f8204118bb0 con 0x7f820407b380 2019-12-05T05:13:02.461 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:02.424+0000 7f81faffd700 1 -- 172.21.15.42:0/2253209354 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81ec0030b0 con 0x7f820407b380 2019-12-05T05:13:02.461 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:02.424+0000 7f81faffd700 1 -- 172.21.15.42:0/2253209354 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 379600102 0 0) 0x7f8204121960 con 0x7f820407af50 2019-12-05T05:13:02.462 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:02.424+0000 7f820a244700 1 -- 172.21.15.42:0/2253209354 >> v1:172.21.15.42:6789/0 conn(0x7f820407b380 legacy=0x7f8204138930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:13:02.462 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:02.424+0000 7f820a244700 1 -- 172.21.15.42:0/2253209354 >> v1:172.21.15.40:6789/0 conn(0x7f820407af50 legacy=0x7f8204121160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:13:02.462 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:02.424+0000 7f820a244700 1 -- 172.21.15.42:0/2253209354 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8204120ce0 con 0x7f8204123690 2019-12-05T05:13:02.462 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:02.424+0000 7f820a244700 1 --2- 172.21.15.42:0/2253209354 >> v2:172.21.15.40:3300/0 conn(0x7f8204123690 0x7f8204116ed0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f82000032b0 tx=0x7f8200011ce0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:02.462 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:02.424+0000 7f820a244700 1 --2- 172.21.15.42:0/2253209354 >> v2:172.21.15.40:3300/0 conn(0x7f8204123690 0x7f8204116ed0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f82000032b0 tx=0x7f8200011ce0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:02.462 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:02.424+0000 7f820a244700 1 --2- 172.21.15.42:0/2253209354 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8204123690 0x7f8204116ed0 secure :-1 s=READY pgs=390 cs=0 l=1 rx=0x7f82000032b0 tx=0x7f8200011ce0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:13:02.463 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:02.424+0000 7f820a244700 1 --2- 172.21.15.42:0/2253209354 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8204123690 0x7f8204116ed0 secure :-1 s=THROTTLE_DONE pgs=390 cs=0 l=1 rx=0x7f82000032b0 tx=0x7f8200011ce0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:02.463 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:02.424+0000 7f81faffd700 1 -- 172.21.15.42:0/2253209354 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f820000ac10 con 0x7f8204123690 2019-12-05T05:13:02.463 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:02.424+0000 7f820a244700 1 --2- 172.21.15.42:0/2253209354 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8204123690 0x7f8204116ed0 secure :-1 s=THROTTLE_DONE pgs=390 cs=0 l=1 rx=0x7f82000032b0 tx=0x7f8200011ce0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:02.463 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:02.424+0000 7f81faffd700 1 -- 172.21.15.42:0/2253209354 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f820000a160 con 0x7f8204123690 2019-12-05T05:13:02.463 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:02.424+0000 7f820a244700 1 --2- 172.21.15.42:0/2253209354 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8204123690 0x7f8204116ed0 secure :-1 s=THROTTLE_DONE pgs=390 cs=0 l=1 rx=0x7f82000032b0 tx=0x7f8200011ce0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:02.463 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:02.424+0000 7f820bca7700 1 -- 172.21.15.42:0/2253209354 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f820413a140 con 0x7f8204123690 2019-12-05T05:13:02.463 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:02.424+0000 7f81faffd700 1 -- 172.21.15.42:0/2253209354 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8200005ab0 con 0x7f8204123690 2019-12-05T05:13:02.464 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:02.424+0000 7f820bca7700 1 -- 172.21.15.42:0/2253209354 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f820413a6f0 con 0x7f8204123690 2019-12-05T05:13:02.464 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:02.428+0000 7f820a244700 1 --2- 172.21.15.42:0/2253209354 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8204123690 0x7f8204116ed0 secure :-1 s=THROTTLE_DONE pgs=390 cs=0 l=1 rx=0x7f82000032b0 tx=0x7f8200011ce0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:02.464 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:02.428+0000 7f81faffd700 1 -- 172.21.15.42:0/2253209354 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f8200007150 con 0x7f8204123690 2019-12-05T05:13:02.464 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:02.428+0000 7f820a244700 1 --2- 172.21.15.42:0/2253209354 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8204123690 0x7f8204116ed0 secure :-1 s=THROTTLE_DONE pgs=390 cs=0 l=1 rx=0x7f82000032b0 tx=0x7f8200011ce0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:02.464 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:02.428+0000 7f81faffd700 1 --2- 172.21.15.42:0/2253209354 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f81ec03b150 0x7f81ec03d660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:13:02.464 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:02.428+0000 7f81faffd700 1 -- 172.21.15.42:0/2253209354 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6547+0+0 (secure 0 0 0) 0x7f820004bb90 con 0x7f8204123690 2019-12-05T05:13:02.464 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:02.428+0000 7f8209a43700 1 --2- 172.21.15.42:0/2253209354 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f81ec03b150 0x7f81ec03d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:13:02.465 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:02.428+0000 7f8209a43700 1 --2- 172.21.15.42:0/2253209354 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f81ec03b150 0x7f81ec03d660 crc :-1 s=READY pgs=226 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:13:02.465 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:02.428+0000 7f820bca7700 1 -- 172.21.15.42:0/2253209354 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f820405d110 con 0x7f8204123690 2019-12-05T05:13:02.466 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:02.440+0000 7f820a244700 1 --2- 172.21.15.42:0/2253209354 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8204123690 0x7f8204116ed0 secure :-1 s=THROTTLE_DONE pgs=390 cs=0 l=1 rx=0x7f82000032b0 tx=0x7f8200011ce0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:02.466 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:02.440+0000 7f81faffd700 1 -- 172.21.15.42:0/2253209354 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f8200032e50 con 0x7f8204123690 2019-12-05T05:13:02.613 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:02.608+0000 7f820bca7700 1 -- 172.21.15.42:0/2253209354 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f820405b8c0 con 0x7f8204123690 2019-12-05T05:13:02.614 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:02.608+0000 7f820a244700 1 --2- 172.21.15.42:0/2253209354 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8204123690 0x7f8204116ed0 secure :-1 s=THROTTLE_DONE pgs=390 cs=0 l=1 rx=0x7f82000032b0 tx=0x7f8200011ce0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:02.614 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:02.608+0000 7f81faffd700 1 -- 172.21.15.42:0/2253209354 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 63 v63) v1 ==== 94+0+2102 (secure 0 0 0) 0x7f8200005d50 con 0x7f8204123690 2019-12-05T05:13:02.617 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:13:02.617 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":63,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5505,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/211479572","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":211479572},{"type":"v1","addr":"172.21.15.42:6835","nonce":211479572}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":63}],"filesystems":[{"mdsmap":{"epoch":62,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:12:48.374585+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5391},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5391":{"gid":5391,"name":"b","rank":0,"incarnation":59,"state":"up:active","state_seq":23,"addr":"172.21.15.40:6833/814397242","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":814397242},{"type":"v1","addr":"172.21.15.40:6833","nonce":814397242}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:13:02.617 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 63 2019-12-05T05:13:02.619 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:02.612+0000 7f820bca7700 1 -- 172.21.15.42:0/2253209354 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f81ec03b150 msgr2=0x7f81ec03d660 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:13:02.619 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:02.612+0000 7f820bca7700 1 --2- 172.21.15.42:0/2253209354 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f81ec03b150 0x7f81ec03d660 crc :-1 s=READY pgs=226 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:13:02.620 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:02.612+0000 7f820bca7700 1 -- 172.21.15.42:0/2253209354 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8204123690 msgr2=0x7f8204116ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:13:02.620 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:02.612+0000 7f820bca7700 1 --2- 172.21.15.42:0/2253209354 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8204123690 0x7f8204116ed0 secure :-1 s=READY pgs=390 cs=0 l=1 rx=0x7f82000032b0 tx=0x7f8200011ce0).stop 2019-12-05T05:13:02.620 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:02.612+0000 7f820bca7700 1 -- 172.21.15.42:0/2253209354 shutdown_connections 2019-12-05T05:13:02.620 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:02.612+0000 7f820bca7700 1 --2- 172.21.15.42:0/2253209354 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f81ec03b150 0x7f81ec03d660 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:13:02.620 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:02.612+0000 7f820bca7700 1 --2- 172.21.15.42:0/2253209354 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8204123690 0x7f8204116ed0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:13:02.620 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:02.612+0000 7f820bca7700 1 -- 172.21.15.42:0/2253209354 shutdown_connections 2019-12-05T05:13:02.621 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:02.612+0000 7f820bca7700 1 -- 172.21.15.42:0/2253209354 wait complete. 2019-12-05T05:13:02.621 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:02.612+0000 7f820bca7700 1 -- 172.21.15.42:0/2253209354 >> 172.21.15.42:0/2253209354 conn(0x7f82040a97e0 msgr2=0x7f820409f580 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:13:02.629 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T05:13:04.632 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:13:04.738 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:04.724+0000 7f30cf43b700 1 Processor -- start 2019-12-05T05:13:04.738 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:04.724+0000 7f30cf43b700 1 -- start start 2019-12-05T05:13:04.738 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:04.728+0000 7f30cf43b700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f30c80a7740 0x7f30c808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:13:04.739 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:04.728+0000 7f30cf43b700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f30c807e720 0x7f30c807eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:13:04.739 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:04.728+0000 7f30cf43b700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f30c809f2e0 0x7f30c809f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:13:04.739 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:04.728+0000 7f30cf43b700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30c80a0b00 con 0x7f30c809f2e0 2019-12-05T05:13:04.739 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:04.728+0000 7f30cf43b700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30c808e5b0 con 0x7f30c807e720 2019-12-05T05:13:04.739 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:04.728+0000 7f30cf43b700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30c808e850 con 0x7f30c80a7740 2019-12-05T05:13:04.739 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:04.728+0000 7f30cd1d7700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f30c807e720 0x7f30c807eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:13:04.739 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:04.728+0000 7f30ce1d9700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f30c809f2e0 0x7f30c809f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:13:04.740 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:04.728+0000 7f30cd1d7700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f30c807e720 0x7f30c807eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:58150/0 (socket says 172.21.15.42:58150) 2019-12-05T05:13:04.740 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:04.728+0000 7f30cd1d7700 1 -- 172.21.15.42:0/3378012064 learned_addr learned my addr 172.21.15.42:0/3378012064 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:13:04.740 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:04.728+0000 7f30cd9d8700 1 --2- 172.21.15.42:0/3378012064 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f30c80a7740 0x7f30c808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:13:04.740 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:04.728+0000 7f30cd1d7700 1 -- 172.21.15.42:0/3378012064 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f30c80a7740 msgr2=0x7f30c808df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:13:04.740 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:04.728+0000 7f30cd1d7700 1 --2- 172.21.15.42:0/3378012064 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f30c80a7740 0x7f30c808df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:13:04.740 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:04.728+0000 7f30cd1d7700 1 -- 172.21.15.42:0/3378012064 >> v2:172.21.15.40:3300/0 conn(0x7f30c809f2e0 msgr2=0x7f30c809f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:13:04.741 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:04.728+0000 7f30cd1d7700 1 --2- 172.21.15.42:0/3378012064 >> v2:172.21.15.40:3300/0 conn(0x7f30c809f2e0 0x7f30c809f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:13:04.741 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:04.728+0000 7f30cd1d7700 1 -- 172.21.15.42:0/3378012064 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30c807f0a0 con 0x7f30c807e720 2019-12-05T05:13:04.741 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:04.728+0000 7f30cd1d7700 1 --2- 172.21.15.42:0/3378012064 >> v2:172.21.15.42:3300/0 conn(0x7f30c807e720 0x7f30c807eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f30c808e770 tx=0x7f30b800fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:04.741 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:04.728+0000 7f30cd1d7700 1 --2- 172.21.15.42:0/3378012064 >> v2:172.21.15.42:3300/0 conn(0x7f30c807e720 0x7f30c807eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f30c808e770 tx=0x7f30b800fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:04.741 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:04.728+0000 7f30cd1d7700 1 --2- 172.21.15.42:0/3378012064 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f30c807e720 0x7f30c807eb50 secure :-1 s=READY pgs=498 cs=0 l=1 rx=0x7f30c808e770 tx=0x7f30b800fdb0).ready entity=mon.1 client_cookie=3f724562508a3a5a server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:13:04.741 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:04.728+0000 7f30cd1d7700 1 --2- 172.21.15.42:0/3378012064 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f30c807e720 0x7f30c807eb50 secure :-1 s=THROTTLE_DONE pgs=498 cs=0 l=1 rx=0x7f30c808e770 tx=0x7f30b800fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:04.741 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:04.728+0000 7f30cd1d7700 1 --2- 172.21.15.42:0/3378012064 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f30c807e720 0x7f30c807eb50 secure :-1 s=THROTTLE_DONE pgs=498 cs=0 l=1 rx=0x7f30c808e770 tx=0x7f30b800fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:04.742 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:04.728+0000 7f30cc9d6700 1 -- 172.21.15.42:0/3378012064 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f30b8008c90 con 0x7f30c807e720 2019-12-05T05:13:04.742 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:04.728+0000 7f30cd1d7700 1 --2- 172.21.15.42:0/3378012064 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f30c807e720 0x7f30c807eb50 secure :-1 s=THROTTLE_DONE pgs=498 cs=0 l=1 rx=0x7f30c808e770 tx=0x7f30b800fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:04.742 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:04.728+0000 7f30cc9d6700 1 -- 172.21.15.42:0/3378012064 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30b80088b0 con 0x7f30c807e720 2019-12-05T05:13:04.742 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:04.728+0000 7f30cc9d6700 1 -- 172.21.15.42:0/3378012064 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f30b8011b50 con 0x7f30c807e720 2019-12-05T05:13:04.742 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:04.728+0000 7f30cf43b700 1 -- 172.21.15.42:0/3378012064 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f30c807e720 msgr2=0x7f30c807eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:13:04.742 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:04.728+0000 7f30cf43b700 1 --2- 172.21.15.42:0/3378012064 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f30c807e720 0x7f30c807eb50 secure :-1 s=READY pgs=498 cs=0 l=1 rx=0x7f30c808e770 tx=0x7f30b800fdb0).stop 2019-12-05T05:13:04.742 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:04.732+0000 7f30cf43b700 1 -- 172.21.15.42:0/3378012064 shutdown_connections 2019-12-05T05:13:04.743 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:04.732+0000 7f30cf43b700 1 --2- 172.21.15.42:0/3378012064 >> v2:172.21.15.40:3300/0 conn(0x7f30c809f2e0 0x7f30c809f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:13:04.743 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:04.732+0000 7f30cf43b700 1 --2- 172.21.15.42:0/3378012064 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f30c80a7740 0x7f30c808df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:13:04.743 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:04.732+0000 7f30cf43b700 1 --2- 172.21.15.42:0/3378012064 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f30c807e720 0x7f30c807eb50 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:13:04.743 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:04.732+0000 7f30cf43b700 1 -- 172.21.15.42:0/3378012064 shutdown_connections 2019-12-05T05:13:04.743 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:04.732+0000 7f30cf43b700 1 -- 172.21.15.42:0/3378012064 wait complete. 2019-12-05T05:13:04.743 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:04.732+0000 7f30cf43b700 1 -- 172.21.15.42:0/3378012064 >> 172.21.15.42:0/3378012064 conn(0x7f30c80aabe0 msgr2=0x7f30c80a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:13:04.748 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:04.736+0000 7f30cf43b700 1 Processor -- start 2019-12-05T05:13:04.749 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:04.736+0000 7f30cf43b700 1 -- start start 2019-12-05T05:13:04.749 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:04.736+0000 7f30cf43b700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f30c8123db0 0x7f30c81318e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:13:04.749 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:04.736+0000 7f30cf43b700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f30c8121050 0x7f30c811db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:13:04.749 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:04.736+0000 7f30cf43b700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30c811e4d0 con 0x7f30c8122620 2019-12-05T05:13:04.749 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:04.736+0000 7f30cf43b700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30c81215d0 con 0x7f30c8123db0 2019-12-05T05:13:04.749 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:04.736+0000 7f30cf43b700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30c811d8d0 con 0x7f30c8121050 2019-12-05T05:13:04.750 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:04.736+0000 7f30cd1d7700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f30c8122620 0x7f30c808eb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:58192/0 (socket says 172.21.15.42:58192) 2019-12-05T05:13:04.750 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:04.736+0000 7f30cd1d7700 1 -- 172.21.15.42:0/995997053 learned_addr learned my addr 172.21.15.42:0/995997053 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:13:04.750 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:04.736+0000 7f30ce1d9700 1 --2- 172.21.15.42:0/995997053 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f30c8121050 0x7f30c811db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:13:04.750 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:04.736+0000 7f30cd9d8700 1 --2- 172.21.15.42:0/995997053 >> v2:172.21.15.40:3300/0 conn(0x7f30c8123db0 0x7f30c81318e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:13:04.750 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:04.736+0000 7f30be7fc700 1 -- 172.21.15.42:0/995997053 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2121638944 0 0) 0x7f30c811e4d0 con 0x7f30c8122620 2019-12-05T05:13:04.750 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:04.736+0000 7f30be7fc700 1 -- 172.21.15.42:0/995997053 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30b00030b0 con 0x7f30c8122620 2019-12-05T05:13:04.751 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:04.736+0000 7f30cd9d8700 1 -- 172.21.15.42:0/995997053 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f30c8121050 msgr2=0x7f30c811db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:13:04.751 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:04.736+0000 7f30cd9d8700 1 --2- 172.21.15.42:0/995997053 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f30c8121050 0x7f30c811db70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:13:04.751 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:04.736+0000 7f30cd9d8700 1 -- 172.21.15.42:0/995997053 >> v1:172.21.15.42:6789/0 conn(0x7f30c8122620 legacy=0x7f30c808eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:13:04.751 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:04.736+0000 7f30cd9d8700 1 -- 172.21.15.42:0/995997053 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30c808f370 con 0x7f30c8123db0 2019-12-05T05:13:04.751 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:04.736+0000 7f30ce1d9700 1 --2- 172.21.15.42:0/995997053 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f30c8121050 0x7f30c811db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:13:04.751 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:04.736+0000 7f30cd9d8700 1 --2- 172.21.15.42:0/995997053 >> v2:172.21.15.40:3300/0 conn(0x7f30c8123db0 0x7f30c81318e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f30c8065ef0 tx=0x7f30c4010a10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:04.751 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:04.736+0000 7f30cd9d8700 1 --2- 172.21.15.42:0/995997053 >> v2:172.21.15.40:3300/0 conn(0x7f30c8123db0 0x7f30c81318e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f30c8065ef0 tx=0x7f30c4010a10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:04.752 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:04.736+0000 7f30cd9d8700 1 --2- 172.21.15.42:0/995997053 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f30c8123db0 0x7f30c81318e0 secure :-1 s=READY pgs=391 cs=0 l=1 rx=0x7f30c8065ef0 tx=0x7f30c4010a10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:13:04.752 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:04.736+0000 7f30cd9d8700 1 --2- 172.21.15.42:0/995997053 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f30c8123db0 0x7f30c81318e0 secure :-1 s=THROTTLE_DONE pgs=391 cs=0 l=1 rx=0x7f30c8065ef0 tx=0x7f30c4010a10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:04.752 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:04.736+0000 7f30be7fc700 1 -- 172.21.15.42:0/995997053 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f30c4005920 con 0x7f30c8123db0 2019-12-05T05:13:04.752 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:04.736+0000 7f30cd9d8700 1 --2- 172.21.15.42:0/995997053 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f30c8123db0 0x7f30c81318e0 secure :-1 s=THROTTLE_DONE pgs=391 cs=0 l=1 rx=0x7f30c8065ef0 tx=0x7f30c4010a10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:04.752 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:04.736+0000 7f30cd9d8700 1 --2- 172.21.15.42:0/995997053 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f30c8123db0 0x7f30c81318e0 secure :-1 s=THROTTLE_DONE pgs=391 cs=0 l=1 rx=0x7f30c8065ef0 tx=0x7f30c4010a10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:04.752 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:04.736+0000 7f30be7fc700 1 -- 172.21.15.42:0/995997053 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30c4005540 con 0x7f30c8123db0 2019-12-05T05:13:04.753 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:04.736+0000 7f30be7fc700 1 -- 172.21.15.42:0/995997053 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f30c4012500 con 0x7f30c8123db0 2019-12-05T05:13:04.753 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:04.736+0000 7f30cf43b700 1 -- 172.21.15.42:0/995997053 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30c808f640 con 0x7f30c8123db0 2019-12-05T05:13:04.753 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:04.736+0000 7f30cf43b700 1 -- 172.21.15.42:0/995997053 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30c8075710 con 0x7f30c8123db0 2019-12-05T05:13:04.753 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:04.740+0000 7f30cd9d8700 1 --2- 172.21.15.42:0/995997053 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f30c8123db0 0x7f30c81318e0 secure :-1 s=THROTTLE_DONE pgs=391 cs=0 l=1 rx=0x7f30c8065ef0 tx=0x7f30c4010a10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:04.753 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:04.740+0000 7f30be7fc700 1 -- 172.21.15.42:0/995997053 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f30c4012810 con 0x7f30c8123db0 2019-12-05T05:13:04.753 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:04.740+0000 7f30cd9d8700 1 --2- 172.21.15.42:0/995997053 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f30c8123db0 0x7f30c81318e0 secure :-1 s=THROTTLE_DONE pgs=391 cs=0 l=1 rx=0x7f30c8065ef0 tx=0x7f30c4010a10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:04.754 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:04.740+0000 7f30be7fc700 1 --2- 172.21.15.42:0/995997053 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f30b003b260 0x7f30b003d770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:13:04.754 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:04.740+0000 7f30be7fc700 1 -- 172.21.15.42:0/995997053 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6547+0+0 (secure 0 0 0) 0x7f30c404aad0 con 0x7f30c8123db0 2019-12-05T05:13:04.754 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:04.740+0000 7f30cd1d7700 1 --2- 172.21.15.42:0/995997053 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f30b003b260 0x7f30b003d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:13:04.754 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:04.740+0000 7f30cd1d7700 1 --2- 172.21.15.42:0/995997053 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f30b003b260 0x7f30b003d770 crc :-1 s=READY pgs=227 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:13:04.754 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:04.740+0000 7f30cf43b700 1 -- 172.21.15.42:0/995997053 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30c811e0c0 con 0x7f30c8123db0 2019-12-05T05:13:04.756 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:04.752+0000 7f30cd9d8700 1 --2- 172.21.15.42:0/995997053 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f30c8123db0 0x7f30c81318e0 secure :-1 s=THROTTLE_DONE pgs=391 cs=0 l=1 rx=0x7f30c8065ef0 tx=0x7f30c4010a10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:04.756 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:04.752+0000 7f30be7fc700 1 -- 172.21.15.42:0/995997053 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f30c4031b50 con 0x7f30c8123db0 2019-12-05T05:13:04.925 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:04.920+0000 7f30cf43b700 1 -- 172.21.15.42:0/995997053 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f30c805d110 con 0x7f30c8123db0 2019-12-05T05:13:04.926 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:04.920+0000 7f30cd9d8700 1 --2- 172.21.15.42:0/995997053 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f30c8123db0 0x7f30c81318e0 secure :-1 s=THROTTLE_DONE pgs=391 cs=0 l=1 rx=0x7f30c8065ef0 tx=0x7f30c4010a10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:04.926 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:04.920+0000 7f30be7fc700 1 -- 172.21.15.42:0/995997053 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 63 v63) v1 ==== 94+0+2102 (secure 0 0 0) 0x7f30c4031db0 con 0x7f30c8123db0 2019-12-05T05:13:04.928 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:13:04.929 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":63,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5505,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/211479572","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":211479572},{"type":"v1","addr":"172.21.15.42:6835","nonce":211479572}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":63}],"filesystems":[{"mdsmap":{"epoch":62,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:12:48.374585+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5391},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5391":{"gid":5391,"name":"b","rank":0,"incarnation":59,"state":"up:active","state_seq":23,"addr":"172.21.15.40:6833/814397242","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":814397242},{"type":"v1","addr":"172.21.15.40:6833","nonce":814397242}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:13:04.929 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 63 2019-12-05T05:13:04.931 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:04.924+0000 7f30cf43b700 1 -- 172.21.15.42:0/995997053 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f30b003b260 msgr2=0x7f30b003d770 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:13:04.931 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:04.924+0000 7f30cf43b700 1 --2- 172.21.15.42:0/995997053 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f30b003b260 0x7f30b003d770 crc :-1 s=READY pgs=227 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:13:04.931 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:04.924+0000 7f30cf43b700 1 -- 172.21.15.42:0/995997053 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f30c8123db0 msgr2=0x7f30c81318e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:13:04.932 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:04.924+0000 7f30cf43b700 1 --2- 172.21.15.42:0/995997053 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f30c8123db0 0x7f30c81318e0 secure :-1 s=READY pgs=391 cs=0 l=1 rx=0x7f30c8065ef0 tx=0x7f30c4010a10).stop 2019-12-05T05:13:04.932 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:04.924+0000 7f30cf43b700 1 -- 172.21.15.42:0/995997053 shutdown_connections 2019-12-05T05:13:04.932 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:04.924+0000 7f30cf43b700 1 --2- 172.21.15.42:0/995997053 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f30b003b260 0x7f30b003d770 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:13:04.932 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:04.924+0000 7f30cf43b700 1 --2- 172.21.15.42:0/995997053 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f30c8121050 0x7f30c811db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:13:04.932 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:04.924+0000 7f30cf43b700 1 --2- 172.21.15.42:0/995997053 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f30c8123db0 0x7f30c81318e0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:13:04.932 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:04.924+0000 7f30cf43b700 1 -- 172.21.15.42:0/995997053 shutdown_connections 2019-12-05T05:13:04.933 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:04.924+0000 7f30cf43b700 1 -- 172.21.15.42:0/995997053 wait complete. 2019-12-05T05:13:04.933 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:04.924+0000 7f30cf43b700 1 -- 172.21.15.42:0/995997053 >> 172.21.15.42:0/995997053 conn(0x7f30c80a82a0 msgr2=0x7f30c8059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:13:04.948 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T05:13:05.393 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:13:05.397 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:13:06.949 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:13:07.064 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:07.047+0000 7f8ec9c1b700 1 Processor -- start 2019-12-05T05:13:07.065 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:07.047+0000 7f8ec9c1b700 1 -- start start 2019-12-05T05:13:07.065 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:07.047+0000 7f8ec9c1b700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8ec408df80 0x7f8ec409f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:13:07.065 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:07.047+0000 7f8ec9c1b700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8ec409f830 0x7f8ec414efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:13:07.065 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:07.047+0000 7f8ec9c1b700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ec409fc60 con 0x7f8ec40a7740 2019-12-05T05:13:07.065 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:07.047+0000 7f8ec9c1b700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ec40a0b00 con 0x7f8ec409f830 2019-12-05T05:13:07.065 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:07.047+0000 7f8ec9c1b700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ec408e5b0 con 0x7f8ec408df80 2019-12-05T05:13:07.066 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:07.047+0000 7f8ec37fe700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8ec408df80 0x7f8ec409f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:13:07.066 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:07.047+0000 7f8ec89b9700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8ec409f830 0x7f8ec414efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:13:07.066 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:07.047+0000 7f8ec37fe700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8ec408df80 0x7f8ec409f2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:58630/0 (socket says 172.21.15.42:58630) 2019-12-05T05:13:07.066 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:07.047+0000 7f8ec37fe700 1 -- 172.21.15.42:0/1771723697 learned_addr learned my addr 172.21.15.42:0/1771723697 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:13:07.066 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:07.047+0000 7f8ec37fe700 1 -- 172.21.15.42:0/1771723697 >> v2:172.21.15.40:3300/0 conn(0x7f8ec409f830 msgr2=0x7f8ec414efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:13:07.066 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:07.047+0000 7f8ec37fe700 1 --2- 172.21.15.42:0/1771723697 >> v2:172.21.15.40:3300/0 conn(0x7f8ec409f830 0x7f8ec414efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:13:07.066 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:07.047+0000 7f8ec37fe700 1 -- 172.21.15.42:0/1771723697 >> v1:172.21.15.40:6789/0 conn(0x7f8ec40a7740 legacy=0x7f8ec407e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:13:07.067 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:07.047+0000 7f8ec37fe700 1 -- 172.21.15.42:0/1771723697 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ec40a0710 con 0x7f8ec408df80 2019-12-05T05:13:07.067 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:07.047+0000 7f8ec37fe700 1 --2- 172.21.15.42:0/1771723697 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8ec408df80 0x7f8ec409f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f8eb4008790 tx=0x7f8eb400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:07.067 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:07.051+0000 7f8ec37fe700 1 --2- 172.21.15.42:0/1771723697 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8ec408df80 0x7f8ec409f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f8eb4008790 tx=0x7f8eb400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:07.067 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:07.051+0000 7f8ec37fe700 1 --2- 172.21.15.42:0/1771723697 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8ec408df80 0x7f8ec409f2e0 secure :-1 s=READY pgs=101 cs=0 l=1 rx=0x7f8eb4008790 tx=0x7f8eb400fd30).ready entity=mon.2 client_cookie=cd138adf47b9a44e server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:13:07.067 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:07.051+0000 7f8ec37fe700 1 --2- 172.21.15.42:0/1771723697 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8ec408df80 0x7f8ec409f2e0 secure :-1 s=THROTTLE_DONE pgs=101 cs=0 l=1 rx=0x7f8eb4008790 tx=0x7f8eb400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:07.067 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:07.051+0000 7f8ec2ffd700 1 -- 172.21.15.42:0/1771723697 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8eb4008c90 con 0x7f8ec408df80 2019-12-05T05:13:07.067 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:07.051+0000 7f8ec37fe700 1 --2- 172.21.15.42:0/1771723697 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8ec408df80 0x7f8ec409f2e0 secure :-1 s=THROTTLE_DONE pgs=101 cs=0 l=1 rx=0x7f8eb4008790 tx=0x7f8eb400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:07.068 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:07.051+0000 7f8ec2ffd700 1 -- 172.21.15.42:0/1771723697 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8eb40088b0 con 0x7f8ec408df80 2019-12-05T05:13:07.068 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:07.051+0000 7f8ec37fe700 1 --2- 172.21.15.42:0/1771723697 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8ec408df80 0x7f8ec409f2e0 secure :-1 s=THROTTLE_DONE pgs=101 cs=0 l=1 rx=0x7f8eb4008790 tx=0x7f8eb400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:07.068 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:07.051+0000 7f8ec2ffd700 1 -- 172.21.15.42:0/1771723697 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8eb4011b10 con 0x7f8ec408df80 2019-12-05T05:13:07.068 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:07.051+0000 7f8ec9c1b700 1 -- 172.21.15.42:0/1771723697 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8ec408df80 msgr2=0x7f8ec409f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:13:07.073 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:07.051+0000 7f8ec9c1b700 1 --2- 172.21.15.42:0/1771723697 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8ec408df80 0x7f8ec409f2e0 secure :-1 s=READY pgs=101 cs=0 l=1 rx=0x7f8eb4008790 tx=0x7f8eb400fd30).stop 2019-12-05T05:13:07.073 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:07.051+0000 7f8ec9c1b700 1 -- 172.21.15.42:0/1771723697 shutdown_connections 2019-12-05T05:13:07.073 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:07.051+0000 7f8ec9c1b700 1 --2- 172.21.15.42:0/1771723697 >> v2:172.21.15.40:3300/0 conn(0x7f8ec409f830 0x7f8ec414efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:13:07.074 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:07.051+0000 7f8ec9c1b700 1 --2- 172.21.15.42:0/1771723697 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8ec408df80 0x7f8ec409f2e0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:13:07.074 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:07.051+0000 7f8ec9c1b700 1 -- 172.21.15.42:0/1771723697 shutdown_connections 2019-12-05T05:13:07.077 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:07.051+0000 7f8ec9c1b700 1 -- 172.21.15.42:0/1771723697 wait complete. 2019-12-05T05:13:07.078 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:07.051+0000 7f8ec9c1b700 1 -- 172.21.15.42:0/1771723697 >> 172.21.15.42:0/1771723697 conn(0x7f8ec40aabe0 msgr2=0x7f8ec40a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:13:07.078 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:07.051+0000 7f8ec9c1b700 1 Processor -- start 2019-12-05T05:13:07.078 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:07.051+0000 7f8ec9c1b700 1 -- start start 2019-12-05T05:13:07.079 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:07.055+0000 7f8ec9c1b700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8ec41223d0 0x7f8ec411afd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:13:07.079 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:07.055+0000 7f8ec9c1b700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f8ec4132f90 0x7f8ec40762f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:13:07.079 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:07.055+0000 7f8ec9c1b700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ec4076840 con 0x7f8ec4116790 2019-12-05T05:13:07.079 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:07.055+0000 7f8ec9c1b700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ec4117610 con 0x7f8ec41223d0 2019-12-05T05:13:07.080 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:07.055+0000 7f8ec9c1b700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ec4116bc0 con 0x7f8ec4132f90 2019-12-05T05:13:07.080 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:07.055+0000 7f8ec3fff700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8ec41223d0 0x7f8ec411afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:13:07.080 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:07.055+0000 7f8ec89b9700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f8ec4132f90 0x7f8ec40762f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:13:07.080 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:07.055+0000 7f8ec3fff700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8ec41223d0 0x7f8ec411afd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.42:45984/0 (socket says 172.21.15.42:45984) 2019-12-05T05:13:07.082 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:07.055+0000 7f8ec3fff700 1 -- 172.21.15.42:0/2912432804 learned_addr learned my addr 172.21.15.42:0/2912432804 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:13:07.082 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:07.055+0000 7f8ec89b9700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f8ec4132f90 0x7f8ec40762f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:58172/0 (socket says 172.21.15.42:58172) 2019-12-05T05:13:07.082 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:07.055+0000 7f8ec3fff700 1 -- 172.21.15.42:0/2912432804 >> v2:172.21.15.42:3300/0 conn(0x7f8ec4132f90 msgr2=0x7f8ec40762f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:13:07.082 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:07.055+0000 7f8ec3fff700 1 --2- 172.21.15.42:0/2912432804 >> v2:172.21.15.42:3300/0 conn(0x7f8ec4132f90 0x7f8ec40762f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:13:07.083 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:07.055+0000 7f8ec3fff700 1 -- 172.21.15.42:0/2912432804 >> v1:172.21.15.42:6789/0 conn(0x7f8ec4116790 legacy=0x7f8ec4132790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:13:07.084 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:07.055+0000 7f8ec3fff700 1 -- 172.21.15.42:0/2912432804 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ec4077a80 con 0x7f8ec41223d0 2019-12-05T05:13:07.084 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:07.055+0000 7f8ec89b9700 1 --2- 172.21.15.42:0/2912432804 >> v2:172.21.15.42:3300/0 conn(0x7f8ec4132f90 0x7f8ec40762f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:13:07.084 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:07.055+0000 7f8ec3fff700 1 --2- 172.21.15.42:0/2912432804 >> v2:172.21.15.40:3300/0 conn(0x7f8ec41223d0 0x7f8ec411afd0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f8ec411d0c0 tx=0x7f8eac0109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:07.084 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:07.055+0000 7f8ec3fff700 1 --2- 172.21.15.42:0/2912432804 >> v2:172.21.15.40:3300/0 conn(0x7f8ec41223d0 0x7f8ec411afd0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f8ec411d0c0 tx=0x7f8eac0109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:07.085 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:07.055+0000 7f8ec3fff700 1 --2- 172.21.15.42:0/2912432804 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8ec41223d0 0x7f8ec411afd0 secure :-1 s=READY pgs=393 cs=0 l=1 rx=0x7f8ec411d0c0 tx=0x7f8eac0109a0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:13:07.085 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:07.055+0000 7f8ec3fff700 1 --2- 172.21.15.42:0/2912432804 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8ec41223d0 0x7f8ec411afd0 secure :-1 s=THROTTLE_DONE pgs=393 cs=0 l=1 rx=0x7f8ec411d0c0 tx=0x7f8eac0109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:07.085 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:07.055+0000 7f8ec0ff9700 1 -- 172.21.15.42:0/2912432804 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8eac00c7a0 con 0x7f8ec41223d0 2019-12-05T05:13:07.085 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:07.055+0000 7f8ec3fff700 1 --2- 172.21.15.42:0/2912432804 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8ec41223d0 0x7f8ec411afd0 secure :-1 s=THROTTLE_DONE pgs=393 cs=0 l=1 rx=0x7f8ec411d0c0 tx=0x7f8eac0109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:07.086 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:07.055+0000 7f8ec3fff700 1 --2- 172.21.15.42:0/2912432804 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8ec41223d0 0x7f8ec411afd0 secure :-1 s=THROTTLE_DONE pgs=393 cs=0 l=1 rx=0x7f8ec411d0c0 tx=0x7f8eac0109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:07.086 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:07.055+0000 7f8ec0ff9700 1 -- 172.21.15.42:0/2912432804 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8eac00c3c0 con 0x7f8ec41223d0 2019-12-05T05:13:07.086 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:07.055+0000 7f8ec9c1b700 1 -- 172.21.15.42:0/2912432804 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ec4076a70 con 0x7f8ec41223d0 2019-12-05T05:13:07.086 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:07.055+0000 7f8ec0ff9700 1 -- 172.21.15.42:0/2912432804 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8eac00cd80 con 0x7f8ec41223d0 2019-12-05T05:13:07.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:07.055+0000 7f8ec9c1b700 1 -- 172.21.15.42:0/2912432804 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ec4077020 con 0x7f8ec41223d0 2019-12-05T05:13:07.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:07.055+0000 7f8ec3fff700 1 --2- 172.21.15.42:0/2912432804 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8ec41223d0 0x7f8ec411afd0 secure :-1 s=THROTTLE_DONE pgs=393 cs=0 l=1 rx=0x7f8ec411d0c0 tx=0x7f8eac0109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:07.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:07.059+0000 7f8ec0ff9700 1 -- 172.21.15.42:0/2912432804 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f8eac007440 con 0x7f8ec41223d0 2019-12-05T05:13:07.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:07.059+0000 7f8ec3fff700 1 --2- 172.21.15.42:0/2912432804 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8ec41223d0 0x7f8ec411afd0 secure :-1 s=THROTTLE_DONE pgs=393 cs=0 l=1 rx=0x7f8ec411d0c0 tx=0x7f8eac0109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:07.088 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:07.059+0000 7f8ec0ff9700 1 --2- 172.21.15.42:0/2912432804 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8ea803ae00 0x7f8ea803d310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:13:07.088 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:07.059+0000 7f8ec0ff9700 1 -- 172.21.15.42:0/2912432804 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6547+0+0 (secure 0 0 0) 0x7f8eac04b290 con 0x7f8ec41223d0 2019-12-05T05:13:07.088 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:07.059+0000 7f8ec37fe700 1 --2- 172.21.15.42:0/2912432804 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8ea803ae00 0x7f8ea803d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:13:07.089 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:07.059+0000 7f8ec37fe700 1 --2- 172.21.15.42:0/2912432804 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8ea803ae00 0x7f8ea803d310 crc :-1 s=READY pgs=228 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:13:07.090 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:07.063+0000 7f8ec9c1b700 1 -- 172.21.15.42:0/2912432804 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ec405d110 con 0x7f8ec41223d0 2019-12-05T05:13:07.090 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:07.071+0000 7f8ec3fff700 1 --2- 172.21.15.42:0/2912432804 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8ec41223d0 0x7f8ec411afd0 secure :-1 s=THROTTLE_DONE pgs=393 cs=0 l=1 rx=0x7f8ec411d0c0 tx=0x7f8eac0109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:07.090 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:07.071+0000 7f8ec0ff9700 1 -- 172.21.15.42:0/2912432804 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f8eac032310 con 0x7f8ec41223d0 2019-12-05T05:13:07.251 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:07.243+0000 7f8ec9c1b700 1 -- 172.21.15.42:0/2912432804 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8ec405b8c0 con 0x7f8ec41223d0 2019-12-05T05:13:07.252 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:07.247+0000 7f8ec3fff700 1 --2- 172.21.15.42:0/2912432804 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8ec41223d0 0x7f8ec411afd0 secure :-1 s=THROTTLE_DONE pgs=393 cs=0 l=1 rx=0x7f8ec411d0c0 tx=0x7f8eac0109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:07.252 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:07.247+0000 7f8ec0ff9700 1 -- 172.21.15.42:0/2912432804 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 63 v63) v1 ==== 94+0+2102 (secure 0 0 0) 0x7f8eac032570 con 0x7f8ec41223d0 2019-12-05T05:13:07.254 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 63 2019-12-05T05:13:07.254 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:13:07.254 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":63,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5505,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/211479572","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":211479572},{"type":"v1","addr":"172.21.15.42:6835","nonce":211479572}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":63}],"filesystems":[{"mdsmap":{"epoch":62,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:12:48.374585+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5391},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5391":{"gid":5391,"name":"b","rank":0,"incarnation":59,"state":"up:active","state_seq":23,"addr":"172.21.15.40:6833/814397242","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":814397242},{"type":"v1","addr":"172.21.15.40:6833","nonce":814397242}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:13:07.256 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:07.251+0000 7f8ec9c1b700 1 -- 172.21.15.42:0/2912432804 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8ea803ae00 msgr2=0x7f8ea803d310 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:13:07.257 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:07.251+0000 7f8ec9c1b700 1 --2- 172.21.15.42:0/2912432804 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8ea803ae00 0x7f8ea803d310 crc :-1 s=READY pgs=228 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:13:07.257 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:07.251+0000 7f8ec9c1b700 1 -- 172.21.15.42:0/2912432804 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8ec41223d0 msgr2=0x7f8ec411afd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:13:07.257 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:07.251+0000 7f8ec9c1b700 1 --2- 172.21.15.42:0/2912432804 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8ec41223d0 0x7f8ec411afd0 secure :-1 s=READY pgs=393 cs=0 l=1 rx=0x7f8ec411d0c0 tx=0x7f8eac0109a0).stop 2019-12-05T05:13:07.257 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:07.251+0000 7f8ec9c1b700 1 -- 172.21.15.42:0/2912432804 shutdown_connections 2019-12-05T05:13:07.257 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:07.251+0000 7f8ec9c1b700 1 --2- 172.21.15.42:0/2912432804 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8ea803ae00 0x7f8ea803d310 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:13:07.257 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:07.251+0000 7f8ec9c1b700 1 --2- 172.21.15.42:0/2912432804 >> v2:172.21.15.42:3300/0 conn(0x7f8ec4132f90 0x7f8ec40762f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:13:07.258 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:07.251+0000 7f8ec9c1b700 1 --2- 172.21.15.42:0/2912432804 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8ec41223d0 0x7f8ec411afd0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:13:07.258 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:07.251+0000 7f8ec9c1b700 1 -- 172.21.15.42:0/2912432804 shutdown_connections 2019-12-05T05:13:07.258 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:07.251+0000 7f8ec9c1b700 1 -- 172.21.15.42:0/2912432804 wait complete. 2019-12-05T05:13:07.258 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:07.251+0000 7f8ec9c1b700 1 -- 172.21.15.42:0/2912432804 >> 172.21.15.42:0/2912432804 conn(0x7f8ec40a7740 msgr2=0x7f8ec40a9840 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:13:07.267 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T05:13:09.271 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:13:09.366 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.351+0000 7efeeed37700 1 Processor -- start 2019-12-05T05:13:09.366 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.351+0000 7efeeed37700 1 -- start start 2019-12-05T05:13:09.366 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.351+0000 7efeeed37700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7efee80a7730 0x7efee808df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:13:09.366 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.351+0000 7efeeed37700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efee809fad0 0x7efee807eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:13:09.366 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.351+0000 7efeeed37700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efee80a0700 con 0x7efee807e710 2019-12-05T05:13:09.367 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.351+0000 7efeeed37700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efee80a0af0 con 0x7efee80a7730 2019-12-05T05:13:09.367 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.351+0000 7efeeed37700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efee808e5a0 con 0x7efee809fad0 2019-12-05T05:13:09.367 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.351+0000 7efeecad3700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7efee807e710 0x7efee809f2d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:58242/0 (socket says 172.21.15.42:58242) 2019-12-05T05:13:09.367 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.351+0000 7efeecad3700 1 -- 172.21.15.42:0/2819087433 learned_addr learned my addr 172.21.15.42:0/2819087433 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:13:09.367 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.351+0000 7efeed2d4700 1 --2- 172.21.15.42:0/2819087433 >> v2:172.21.15.40:3300/0 conn(0x7efee80a7730 0x7efee808df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:13:09.367 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.351+0000 7efeedad5700 1 --2- 172.21.15.42:0/2819087433 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efee809fad0 0x7efee807eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:13:09.368 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.351+0000 7efee7fff700 1 -- 172.21.15.42:0/2819087433 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 393826117 0 0) 0x7efee80a0700 con 0x7efee807e710 2019-12-05T05:13:09.368 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.351+0000 7efee7fff700 1 -- 172.21.15.42:0/2819087433 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efed00030b0 con 0x7efee807e710 2019-12-05T05:13:09.368 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.351+0000 7efee7fff700 1 -- 172.21.15.42:0/2819087433 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 368202488 0 0) 0x7efed00030b0 con 0x7efee807e710 2019-12-05T05:13:09.368 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.351+0000 7efee7fff700 1 -- 172.21.15.42:0/2819087433 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efee809fad0 msgr2=0x7efee807eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:13:09.371 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.351+0000 7efee7fff700 1 --2- 172.21.15.42:0/2819087433 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efee809fad0 0x7efee807eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:13:09.371 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.351+0000 7efee7fff700 1 -- 172.21.15.42:0/2819087433 >> v2:172.21.15.40:3300/0 conn(0x7efee80a7730 msgr2=0x7efee808df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:13:09.372 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.351+0000 7efee7fff700 1 --2- 172.21.15.42:0/2819087433 >> v2:172.21.15.40:3300/0 conn(0x7efee80a7730 0x7efee808df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:13:09.372 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.351+0000 7efee7fff700 1 -- 172.21.15.42:0/2819087433 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efee807f090 con 0x7efee807e710 2019-12-05T05:13:09.372 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.351+0000 7efee7fff700 1 -- 172.21.15.42:0/2819087433 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7efedc001750 con 0x7efee807e710 2019-12-05T05:13:09.372 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.351+0000 7efee7fff700 1 -- 172.21.15.42:0/2819087433 >> v1:172.21.15.42:6789/0 conn(0x7efee807e710 legacy=0x7efee809f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:13:09.372 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.351+0000 7efee7fff700 1 --2- 172.21.15.42:0/2819087433 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efed00036e0 0x7efed0003b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:13:09.372 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.351+0000 7efee7fff700 1 --2- 172.21.15.42:0/2819087433 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efed0004420 0x7efed0004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:13:09.373 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.351+0000 7efee7fff700 1 --2- 172.21.15.42:0/2819087433 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efed0004da0 0x7efed004f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:13:09.373 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.351+0000 7efee7fff700 1 -- 172.21.15.42:0/2819087433 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efee80a0af0 con 0x7efed0004420 2019-12-05T05:13:09.373 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.351+0000 7efee7fff700 1 -- 172.21.15.42:0/2819087433 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efee808e5a0 con 0x7efed0004da0 2019-12-05T05:13:09.373 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.351+0000 7efee7fff700 1 -- 172.21.15.42:0/2819087433 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efed0005250 con 0x7efed00036e0 2019-12-05T05:13:09.373 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.351+0000 7efeedad5700 1 --2- 172.21.15.42:0/2819087433 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efed0004da0 0x7efed004f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:13:09.373 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.351+0000 7efeed2d4700 1 --2- 172.21.15.42:0/2819087433 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efed00036e0 0x7efed0003b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:13:09.374 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.351+0000 7efeedad5700 1 -- 172.21.15.42:0/2819087433 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efed00036e0 msgr2=0x7efed0003b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:13:09.374 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.351+0000 7efeedad5700 1 --2- 172.21.15.42:0/2819087433 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efed00036e0 0x7efed0003b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:13:09.374 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.351+0000 7efeedad5700 1 -- 172.21.15.42:0/2819087433 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efed0004420 msgr2=0x7efed0004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:13:09.374 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.351+0000 7efeedad5700 1 --2- 172.21.15.42:0/2819087433 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efed0004420 0x7efed0004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:13:09.374 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.351+0000 7efeedad5700 1 -- 172.21.15.42:0/2819087433 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efed004fce0 con 0x7efed0004da0 2019-12-05T05:13:09.374 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.351+0000 7efeed2d4700 1 --2- 172.21.15.42:0/2819087433 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efed00036e0 0x7efed0003b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).send_auth_request state changed! 2019-12-05T05:13:09.375 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.351+0000 7efeedad5700 1 --2- 172.21.15.42:0/2819087433 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efed0004da0 0x7efed004f680 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7efed0004390 tx=0x7efee000ffe0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:09.375 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.351+0000 7efeedad5700 1 --2- 172.21.15.42:0/2819087433 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efed0004da0 0x7efed004f680 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7efed0004390 tx=0x7efee000ffe0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:09.375 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.351+0000 7efeedad5700 1 --2- 172.21.15.42:0/2819087433 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efed0004da0 0x7efed004f680 secure :-1 s=READY pgs=505 cs=0 l=1 rx=0x7efed0004390 tx=0x7efee000ffe0).ready entity=mon.1 client_cookie=c41c4ccbab7daef9 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:13:09.375 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.351+0000 7efeedad5700 1 --2- 172.21.15.42:0/2819087433 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efed0004da0 0x7efed004f680 secure :-1 s=THROTTLE_DONE pgs=505 cs=0 l=1 rx=0x7efed0004390 tx=0x7efee000ffe0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:09.375 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.351+0000 7efee7fff700 1 -- 172.21.15.42:0/2819087433 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efee0010cd0 con 0x7efed0004da0 2019-12-05T05:13:09.375 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.351+0000 7efeedad5700 1 --2- 172.21.15.42:0/2819087433 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efed0004da0 0x7efed004f680 secure :-1 s=THROTTLE_DONE pgs=505 cs=0 l=1 rx=0x7efed0004390 tx=0x7efee000ffe0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:09.375 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.351+0000 7efeedad5700 1 --2- 172.21.15.42:0/2819087433 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efed0004da0 0x7efed004f680 secure :-1 s=THROTTLE_DONE pgs=505 cs=0 l=1 rx=0x7efed0004390 tx=0x7efee000ffe0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:09.376 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.351+0000 7efee7fff700 1 -- 172.21.15.42:0/2819087433 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efee00108f0 con 0x7efed0004da0 2019-12-05T05:13:09.376 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.351+0000 7efee7fff700 1 -- 172.21.15.42:0/2819087433 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efee0008b90 con 0x7efed0004da0 2019-12-05T05:13:09.376 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.351+0000 7efeeed37700 1 -- 172.21.15.42:0/2819087433 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efed0004da0 msgr2=0x7efed004f680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:13:09.376 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.351+0000 7efeeed37700 1 --2- 172.21.15.42:0/2819087433 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efed0004da0 0x7efed004f680 secure :-1 s=READY pgs=505 cs=0 l=1 rx=0x7efed0004390 tx=0x7efee000ffe0).stop 2019-12-05T05:13:09.376 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.351+0000 7efeedad5700 1 -- 172.21.15.42:0/2819087433 reap_dead start 2019-12-05T05:13:09.377 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.355+0000 7efeeed37700 1 -- 172.21.15.42:0/2819087433 shutdown_connections 2019-12-05T05:13:09.377 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.355+0000 7efeeed37700 1 -- 172.21.15.42:0/2819087433 shutdown_connections 2019-12-05T05:13:09.377 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.355+0000 7efeeed37700 1 -- 172.21.15.42:0/2819087433 wait complete. 2019-12-05T05:13:09.377 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.355+0000 7efeeed37700 1 -- 172.21.15.42:0/2819087433 >> 172.21.15.42:0/2819087433 conn(0x7efee80aabd0 msgr2=0x7efee80a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:13:09.377 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.355+0000 7efeeed37700 1 Processor -- start 2019-12-05T05:13:09.377 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.355+0000 7efeeed37700 1 -- start start 2019-12-05T05:13:09.377 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.355+0000 7efeeed37700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7efee8064070 0x7efee80a3c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:13:09.378 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.355+0000 7efeeed37700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efee80742d0 0x7efee8076710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:13:09.378 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.355+0000 7efeeed37700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efee80a41a0 con 0x7efee80a5ae0 2019-12-05T05:13:09.378 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.355+0000 7efeeed37700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efee805ae20 con 0x7efee8064070 2019-12-05T05:13:09.378 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.355+0000 7efeeed37700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efee8065a60 con 0x7efee80742d0 2019-12-05T05:13:09.378 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.355+0000 7efeecad3700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7efee8064070 0x7efee80a3c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:13:09.378 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.355+0000 7efeecad3700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7efee8064070 0x7efee80a3c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:58216/0 (socket says 172.21.15.42:58216) 2019-12-05T05:13:09.379 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.355+0000 7efeecad3700 1 -- 172.21.15.42:0/1233304613 learned_addr learned my addr 172.21.15.42:0/1233304613 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:13:09.379 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.355+0000 7efeedad5700 1 --2- 172.21.15.42:0/1233304613 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efee80742d0 0x7efee8076710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:13:09.379 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.355+0000 7efeecad3700 1 -- 172.21.15.42:0/1233304613 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efee80742d0 msgr2=0x7efee8076710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:13:09.379 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.355+0000 7efeecad3700 1 --2- 172.21.15.42:0/1233304613 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efee80742d0 0x7efee8076710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:13:09.379 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.355+0000 7efeecad3700 1 -- 172.21.15.42:0/1233304613 >> v1:172.21.15.40:6789/0 conn(0x7efee80a5ae0 legacy=0x7efee8141920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:13:09.379 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.355+0000 7efeecad3700 1 -- 172.21.15.42:0/1233304613 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efee8063180 con 0x7efee8064070 2019-12-05T05:13:09.380 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.355+0000 7efeedad5700 1 --2- 172.21.15.42:0/1233304613 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efee80742d0 0x7efee8076710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T05:13:09.380 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.355+0000 7efeecad3700 1 --2- 172.21.15.42:0/1233304613 >> v2:172.21.15.42:3300/0 conn(0x7efee8064070 0x7efee80a3c50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7efedc001000 tx=0x7efedc010a60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:09.380 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.355+0000 7efeecad3700 1 --2- 172.21.15.42:0/1233304613 >> v2:172.21.15.42:3300/0 conn(0x7efee8064070 0x7efee80a3c50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7efedc001000 tx=0x7efedc010a60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:09.380 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.355+0000 7efeecad3700 1 --2- 172.21.15.42:0/1233304613 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efee8064070 0x7efee80a3c50 secure :-1 s=READY pgs=506 cs=0 l=1 rx=0x7efedc001000 tx=0x7efedc010a60).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:13:09.380 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.355+0000 7efeecad3700 1 --2- 172.21.15.42:0/1233304613 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efee8064070 0x7efee80a3c50 secure :-1 s=THROTTLE_DONE pgs=506 cs=0 l=1 rx=0x7efedc001000 tx=0x7efedc010a60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:09.380 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.355+0000 7efeecad3700 1 --2- 172.21.15.42:0/1233304613 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efee8064070 0x7efee80a3c50 secure :-1 s=THROTTLE_DONE pgs=506 cs=0 l=1 rx=0x7efedc001000 tx=0x7efedc010a60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:09.381 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.355+0000 7efeecad3700 1 --2- 172.21.15.42:0/1233304613 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efee8064070 0x7efee80a3c50 secure :-1 s=THROTTLE_DONE pgs=506 cs=0 l=1 rx=0x7efedc001000 tx=0x7efedc010a60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:09.381 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.355+0000 7efee5ffb700 1 -- 172.21.15.42:0/1233304613 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efedc00c830 con 0x7efee8064070 2019-12-05T05:13:09.381 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.355+0000 7efee5ffb700 1 -- 172.21.15.42:0/1233304613 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efedc00c450 con 0x7efee8064070 2019-12-05T05:13:09.381 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.355+0000 7efee5ffb700 1 -- 172.21.15.42:0/1233304613 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efedc004400 con 0x7efee8064070 2019-12-05T05:13:09.381 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.355+0000 7efeeed37700 1 -- 172.21.15.42:0/1233304613 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efee80634f0 con 0x7efee8064070 2019-12-05T05:13:09.382 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.355+0000 7efeeed37700 1 -- 172.21.15.42:0/1233304613 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efee8065750 con 0x7efee8064070 2019-12-05T05:13:09.382 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.359+0000 7efeecad3700 1 --2- 172.21.15.42:0/1233304613 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efee8064070 0x7efee80a3c50 secure :-1 s=THROTTLE_DONE pgs=506 cs=0 l=1 rx=0x7efedc001000 tx=0x7efedc010a60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:09.382 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.359+0000 7efee5ffb700 1 -- 172.21.15.42:0/1233304613 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7efedc004600 con 0x7efee8064070 2019-12-05T05:13:09.382 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.359+0000 7efeecad3700 1 --2- 172.21.15.42:0/1233304613 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efee8064070 0x7efee80a3c50 secure :-1 s=THROTTLE_DONE pgs=506 cs=0 l=1 rx=0x7efedc001000 tx=0x7efedc010a60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:09.382 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.359+0000 7efee5ffb700 1 --2- 172.21.15.42:0/1233304613 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7efec803ad60 0x7efec803d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:13:09.382 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.359+0000 7efee5ffb700 1 -- 172.21.15.42:0/1233304613 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6633+0+0 (secure 0 0 0) 0x7efedc04ab80 con 0x7efee8064070 2019-12-05T05:13:09.383 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.359+0000 7efeed2d4700 1 --2- 172.21.15.42:0/1233304613 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7efec803ad60 0x7efec803d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:13:09.383 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.359+0000 7efeed2d4700 1 --2- 172.21.15.42:0/1233304613 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7efec803ad60 0x7efec803d270 crc :-1 s=READY pgs=229 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:13:09.384 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.367+0000 7efeeed37700 1 -- 172.21.15.42:0/1233304613 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efee805d100 con 0x7efee8064070 2019-12-05T05:13:09.384 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.371+0000 7efeecad3700 1 --2- 172.21.15.42:0/1233304613 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efee8064070 0x7efee80a3c50 secure :-1 s=THROTTLE_DONE pgs=506 cs=0 l=1 rx=0x7efedc001000 tx=0x7efedc010a60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:09.384 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.371+0000 7efee5ffb700 1 -- 172.21.15.42:0/1233304613 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7efedc031c00 con 0x7efee8064070 2019-12-05T05:13:09.570 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.563+0000 7efeeed37700 1 -- 172.21.15.42:0/1233304613 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efee805b8b0 con 0x7efee8064070 2019-12-05T05:13:09.571 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.563+0000 7efeecad3700 1 --2- 172.21.15.42:0/1233304613 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efee8064070 0x7efee80a3c50 secure :-1 s=THROTTLE_DONE pgs=506 cs=0 l=1 rx=0x7efedc001000 tx=0x7efedc010a60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:09.571 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.563+0000 7efee5ffb700 1 -- 172.21.15.42:0/1233304613 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) v1 ==== 94+0+1768 (secure 0 0 0) 0x7efedc031e60 con 0x7efee8064070 2019-12-05T05:13:09.571 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 64 2019-12-05T05:13:09.571 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:13:09.571 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":64,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:13:08.347598+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5505":{"gid":5505,"name":"a","rank":0,"incarnation":64,"state":"up:replay","state_seq":2,"addr":"172.21.15.42:6835/211479572","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":211479572},{"type":"v1","addr":"172.21.15.42:6835","nonce":211479572}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:13:09.574 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.567+0000 7efeeed37700 1 -- 172.21.15.42:0/1233304613 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7efec803ad60 msgr2=0x7efec803d270 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:13:09.574 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.567+0000 7efeeed37700 1 --2- 172.21.15.42:0/1233304613 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7efec803ad60 0x7efec803d270 crc :-1 s=READY pgs=229 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:13:09.575 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.567+0000 7efeeed37700 1 -- 172.21.15.42:0/1233304613 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efee8064070 msgr2=0x7efee80a3c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:13:09.575 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.567+0000 7efeeed37700 1 --2- 172.21.15.42:0/1233304613 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efee8064070 0x7efee80a3c50 secure :-1 s=READY pgs=506 cs=0 l=1 rx=0x7efedc001000 tx=0x7efedc010a60).stop 2019-12-05T05:13:09.575 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.567+0000 7efeeed37700 1 -- 172.21.15.42:0/1233304613 shutdown_connections 2019-12-05T05:13:09.575 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.567+0000 7efeeed37700 1 --2- 172.21.15.42:0/1233304613 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7efec803ad60 0x7efec803d270 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:13:09.575 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.567+0000 7efeeed37700 1 --2- 172.21.15.42:0/1233304613 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efee80742d0 0x7efee8076710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:13:09.576 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.567+0000 7efeeed37700 1 --2- 172.21.15.42:0/1233304613 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efee8064070 0x7efee80a3c50 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:13:09.576 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.567+0000 7efeeed37700 1 -- 172.21.15.42:0/1233304613 shutdown_connections 2019-12-05T05:13:09.576 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.567+0000 7efeeed37700 1 -- 172.21.15.42:0/1233304613 wait complete. 2019-12-05T05:13:09.576 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.567+0000 7efeeed37700 1 -- 172.21.15.42:0/1233304613 >> 172.21.15.42:0/1233304613 conn(0x7efee80aabd0 msgr2=0x7efee80a9760 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:13:09.585 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b down, removed from mdsmap 2019-12-05T05:13:09.585 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2019-12-05T05:13:09.586 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:13:09.712 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.663+0000 7f865a5b3700 1 Processor -- start 2019-12-05T05:13:09.713 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.663+0000 7f865a5b3700 1 -- start start 2019-12-05T05:13:09.713 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.663+0000 7f865a5b3700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f86540a7740 0x7f865408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:13:09.713 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.663+0000 7f865a5b3700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86540a0710 con 0x7f865409fae0 2019-12-05T05:13:09.713 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.663+0000 7f865a5b3700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86540a9ba0 con 0x7f865407e720 2019-12-05T05:13:09.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.663+0000 7f865a5b3700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86540a0b00 con 0x7f86540a7740 2019-12-05T05:13:09.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.663+0000 7f8653fff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f865407e720 0x7f865409f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:58262/0 (socket says 172.21.15.42:58262) 2019-12-05T05:13:09.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.663+0000 7f8653fff700 1 -- 172.21.15.42:0/3768605051 learned_addr learned my addr 172.21.15.42:0/3768605051 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:13:09.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.663+0000 7f8658b50700 1 --2- 172.21.15.42:0/3768605051 >> v2:172.21.15.40:3300/0 conn(0x7f86540a7740 0x7f865408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:13:09.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.663+0000 7f86537fe700 1 -- 172.21.15.42:0/3768605051 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3281345661 0 0) 0x7f86540a9ba0 con 0x7f865407e720 2019-12-05T05:13:09.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.663+0000 7f86537fe700 1 -- 172.21.15.42:0/3768605051 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86400030b0 con 0x7f865407e720 2019-12-05T05:13:09.716 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.663+0000 7f86537fe700 1 -- 172.21.15.42:0/3768605051 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2542245157 0 0) 0x7f86400030b0 con 0x7f865407e720 2019-12-05T05:13:09.716 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.663+0000 7f86537fe700 1 -- 172.21.15.42:0/3768605051 >> v2:172.21.15.40:3300/0 conn(0x7f86540a7740 msgr2=0x7f865408df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:13:09.716 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.663+0000 7f86537fe700 1 --2- 172.21.15.42:0/3768605051 >> v2:172.21.15.40:3300/0 conn(0x7f86540a7740 0x7f865408df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:13:09.716 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.663+0000 7f86537fe700 1 -- 172.21.15.42:0/3768605051 >> v1:172.21.15.40:6789/0 conn(0x7f865409fae0 legacy=0x7f865414efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:13:09.716 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.663+0000 7f86537fe700 1 -- 172.21.15.42:0/3768605051 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f865407eb50 con 0x7f865407e720 2019-12-05T05:13:09.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.663+0000 7f86537fe700 1 -- 172.21.15.42:0/3768605051 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7f863c001750 con 0x7f865407e720 2019-12-05T05:13:09.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.663+0000 7f86537fe700 1 -- 172.21.15.42:0/3768605051 >> v1:172.21.15.42:6789/0 conn(0x7f865407e720 legacy=0x7f865409f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:13:09.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.663+0000 7f86537fe700 1 --2- 172.21.15.42:0/3768605051 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f86400036e0 0x7f8640003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:13:09.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.663+0000 7f86537fe700 1 --2- 172.21.15.42:0/3768605051 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8640004420 0x7f8640004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:13:09.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.663+0000 7f86537fe700 1 --2- 172.21.15.42:0/3768605051 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8640004da0 0x7f864000b670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:13:09.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.663+0000 7f86537fe700 1 -- 172.21.15.42:0/3768605051 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86540a0b00 con 0x7f86400036e0 2019-12-05T05:13:09.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.663+0000 7f86537fe700 1 -- 172.21.15.42:0/3768605051 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f864004fc60 con 0x7f8640004420 2019-12-05T05:13:09.718 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.663+0000 7f86537fe700 1 -- 172.21.15.42:0/3768605051 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f864004fea0 con 0x7f8640004da0 2019-12-05T05:13:09.718 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.663+0000 7f8653fff700 1 --2- 172.21.15.42:0/3768605051 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8640004420 0x7f8640004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:13:09.718 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.663+0000 7f8658b50700 1 --2- 172.21.15.42:0/3768605051 >> v2:172.21.15.40:3300/0 conn(0x7f86540a7740 0x7f865408df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:13:09.718 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.663+0000 7f8659351700 1 --2- 172.21.15.42:0/3768605051 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8640004da0 0x7f864000b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:13:09.718 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.663+0000 7f8653fff700 1 -- 172.21.15.42:0/3768605051 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8640004da0 msgr2=0x7f864000b670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:13:09.718 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.663+0000 7f8653fff700 1 --2- 172.21.15.42:0/3768605051 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8640004da0 0x7f864000b670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:13:09.718 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.663+0000 7f8653fff700 1 -- 172.21.15.42:0/3768605051 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f86400036e0 msgr2=0x7f8640003b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:13:09.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.663+0000 7f8653fff700 1 --2- 172.21.15.42:0/3768605051 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f86400036e0 0x7f8640003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:13:09.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.663+0000 7f8653fff700 1 -- 172.21.15.42:0/3768605051 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86400501a0 con 0x7f8640004420 2019-12-05T05:13:09.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.663+0000 7f8659351700 1 -- 172.21.15.42:0/3768605051 reap_dead start 2019-12-05T05:13:09.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.663+0000 7f8653fff700 1 --2- 172.21.15.42:0/3768605051 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8640004420 0x7f8640004850 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f863c001950 tx=0x7f863c00c260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:09.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.663+0000 7f8653fff700 1 --2- 172.21.15.42:0/3768605051 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8640004420 0x7f8640004850 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f863c001950 tx=0x7f863c00c260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:09.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.663+0000 7f8653fff700 1 --2- 172.21.15.42:0/3768605051 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8640004420 0x7f8640004850 secure :-1 s=READY pgs=508 cs=0 l=1 rx=0x7f863c001950 tx=0x7f863c00c260).ready entity=mon.1 client_cookie=89d4e77249fc95e0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:13:09.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.663+0000 7f8653fff700 1 --2- 172.21.15.42:0/3768605051 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8640004420 0x7f8640004850 secure :-1 s=THROTTLE_DONE pgs=508 cs=0 l=1 rx=0x7f863c001950 tx=0x7f863c00c260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:09.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.663+0000 7f86537fe700 1 -- 172.21.15.42:0/3768605051 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f863c010350 con 0x7f8640004420 2019-12-05T05:13:09.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.663+0000 7f8653fff700 1 --2- 172.21.15.42:0/3768605051 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8640004420 0x7f8640004850 secure :-1 s=THROTTLE_DONE pgs=508 cs=0 l=1 rx=0x7f863c001950 tx=0x7f863c00c260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:09.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.663+0000 7f86537fe700 1 -- 172.21.15.42:0/3768605051 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f863c00ed20 con 0x7f8640004420 2019-12-05T05:13:09.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.663+0000 7f8653fff700 1 --2- 172.21.15.42:0/3768605051 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8640004420 0x7f8640004850 secure :-1 s=THROTTLE_DONE pgs=508 cs=0 l=1 rx=0x7f863c001950 tx=0x7f863c00c260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:09.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.663+0000 7f86537fe700 1 -- 172.21.15.42:0/3768605051 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f863c010930 con 0x7f8640004420 2019-12-05T05:13:09.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.663+0000 7f865a5b3700 1 -- 172.21.15.42:0/3768605051 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8640004420 msgr2=0x7f8640004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:13:09.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.663+0000 7f865a5b3700 1 --2- 172.21.15.42:0/3768605051 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8640004420 0x7f8640004850 secure :-1 s=READY pgs=508 cs=0 l=1 rx=0x7f863c001950 tx=0x7f863c00c260).stop 2019-12-05T05:13:09.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.667+0000 7f865a5b3700 1 -- 172.21.15.42:0/3768605051 shutdown_connections 2019-12-05T05:13:09.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.667+0000 7f865a5b3700 1 --2- 172.21.15.42:0/3768605051 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8640004420 0x7f8640004850 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:13:09.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.667+0000 7f865a5b3700 1 -- 172.21.15.42:0/3768605051 shutdown_connections 2019-12-05T05:13:09.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.667+0000 7f865a5b3700 1 -- 172.21.15.42:0/3768605051 wait complete. 2019-12-05T05:13:09.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.667+0000 7f865a5b3700 1 -- 172.21.15.42:0/3768605051 >> 172.21.15.42:0/3768605051 conn(0x7f86540aabe0 msgr2=0x7f86540a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:13:09.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.667+0000 7f865a5b3700 1 Processor -- start 2019-12-05T05:13:09.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.667+0000 7f865a5b3700 1 -- start start 2019-12-05T05:13:09.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.667+0000 7f865a5b3700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f865407dcf0 0x7f865411b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:13:09.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.667+0000 7f865a5b3700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f86540a54f0 0x7f865411e120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:13:09.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.667+0000 7f865a5b3700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f865411a100 0x7f865411c280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:13:09.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.667+0000 7f865a5b3700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86540a4d50 con 0x7f865407dcf0 2019-12-05T05:13:09.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.667+0000 7f865a5b3700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86540a4ff0 con 0x7f865411a100 2019-12-05T05:13:09.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.667+0000 7f865a5b3700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f865411a530 con 0x7f86540a54f0 2019-12-05T05:13:09.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.667+0000 7f8659351700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f865411a100 0x7f865411c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:13:09.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.667+0000 7f8659351700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f865411a100 0x7f865411c280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:58238/0 (socket says 172.21.15.42:58238) 2019-12-05T05:13:09.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.667+0000 7f8659351700 1 -- 172.21.15.42:0/1020110759 learned_addr learned my addr 172.21.15.42:0/1020110759 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:13:09.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.667+0000 7f8653fff700 1 --2- 172.21.15.42:0/1020110759 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f86540a54f0 0x7f865411e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:13:09.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.667+0000 7f8658b50700 1 --2- 172.21.15.42:0/1020110759 >> v2:172.21.15.40:3300/0 conn(0x7f865407dcf0 0x7f865411b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:13:09.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.667+0000 7f8659351700 1 -- 172.21.15.42:0/1020110759 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f86540a54f0 msgr2=0x7f865411e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:13:09.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.667+0000 7f8659351700 1 --2- 172.21.15.42:0/1020110759 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f86540a54f0 0x7f865411e120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:13:09.724 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.667+0000 7f8659351700 1 -- 172.21.15.42:0/1020110759 >> v2:172.21.15.40:3300/0 conn(0x7f865407dcf0 msgr2=0x7f865411b170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:13:09.724 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.667+0000 7f8659351700 1 --2- 172.21.15.42:0/1020110759 >> v2:172.21.15.40:3300/0 conn(0x7f865407dcf0 0x7f865411b170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:13:09.724 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.667+0000 7f8659351700 1 -- 172.21.15.42:0/1020110759 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f865411a720 con 0x7f865411a100 2019-12-05T05:13:09.724 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.667+0000 7f8659351700 1 --2- 172.21.15.42:0/1020110759 >> v2:172.21.15.42:3300/0 conn(0x7f865411a100 0x7f865411c280 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f8644000f10 tx=0x7f8644010660).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:09.724 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.667+0000 7f8659351700 1 --2- 172.21.15.42:0/1020110759 >> v2:172.21.15.42:3300/0 conn(0x7f865411a100 0x7f865411c280 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f8644000f10 tx=0x7f8644010660).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:09.724 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.667+0000 7f8659351700 1 --2- 172.21.15.42:0/1020110759 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f865411a100 0x7f865411c280 secure :-1 s=READY pgs=509 cs=0 l=1 rx=0x7f8644000f10 tx=0x7f8644010660).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:13:09.724 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.671+0000 7f8659351700 1 --2- 172.21.15.42:0/1020110759 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f865411a100 0x7f865411c280 secure :-1 s=THROTTLE_DONE pgs=509 cs=0 l=1 rx=0x7f8644000f10 tx=0x7f8644010660).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:09.725 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.671+0000 7f8659351700 1 --2- 172.21.15.42:0/1020110759 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f865411a100 0x7f865411c280 secure :-1 s=THROTTLE_DONE pgs=509 cs=0 l=1 rx=0x7f8644000f10 tx=0x7f8644010660).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:09.725 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.671+0000 7f86517fa700 1 -- 172.21.15.42:0/1020110759 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8644006a40 con 0x7f865411a100 2019-12-05T05:13:09.725 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.671+0000 7f8659351700 1 --2- 172.21.15.42:0/1020110759 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f865411a100 0x7f865411c280 secure :-1 s=THROTTLE_DONE pgs=509 cs=0 l=1 rx=0x7f8644000f10 tx=0x7f8644010660).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:09.726 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.671+0000 7f86517fa700 1 -- 172.21.15.42:0/1020110759 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8644006660 con 0x7f865411a100 2019-12-05T05:13:09.726 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.671+0000 7f865a5b3700 1 -- 172.21.15.42:0/1020110759 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f865411c7d0 con 0x7f865411a100 2019-12-05T05:13:09.726 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.671+0000 7f86517fa700 1 -- 172.21.15.42:0/1020110759 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8644007640 con 0x7f865411a100 2019-12-05T05:13:09.727 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.671+0000 7f865a5b3700 1 -- 172.21.15.42:0/1020110759 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f865411cd00 con 0x7f865411a100 2019-12-05T05:13:09.727 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.671+0000 7f8659351700 1 --2- 172.21.15.42:0/1020110759 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f865411a100 0x7f865411c280 secure :-1 s=THROTTLE_DONE pgs=509 cs=0 l=1 rx=0x7f8644000f10 tx=0x7f8644010660).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:09.727 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.671+0000 7f86517fa700 1 -- 172.21.15.42:0/1020110759 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f8644007840 con 0x7f865411a100 2019-12-05T05:13:09.727 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.671+0000 7f8659351700 1 --2- 172.21.15.42:0/1020110759 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f865411a100 0x7f865411c280 secure :-1 s=THROTTLE_DONE pgs=509 cs=0 l=1 rx=0x7f8644000f10 tx=0x7f8644010660).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:09.727 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.671+0000 7f86517fa700 1 --2- 172.21.15.42:0/1020110759 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f863803ae60 0x7f863803d370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:13:09.727 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.671+0000 7f86517fa700 1 -- 172.21.15.42:0/1020110759 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6633+0+0 (secure 0 0 0) 0x7f864c0082d0 con 0x7f865411a100 2019-12-05T05:13:09.727 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.671+0000 7f8658b50700 1 --2- 172.21.15.42:0/1020110759 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f863803ae60 0x7f863803d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:13:09.728 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.671+0000 7f8658b50700 1 --2- 172.21.15.42:0/1020110759 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f863803ae60 0x7f863803d370 crc :-1 s=READY pgs=230 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:13:09.728 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.675+0000 7f865a5b3700 1 -- 172.21.15.42:0/1020110759 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f865405d110 con 0x7f865411a100 2019-12-05T05:13:09.728 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.683+0000 7f8659351700 1 --2- 172.21.15.42:0/1020110759 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f865411a100 0x7f865411c280 secure :-1 s=THROTTLE_DONE pgs=509 cs=0 l=1 rx=0x7f8644000f10 tx=0x7f8644010660).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:09.728 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.683+0000 7f86517fa700 1 -- 172.21.15.42:0/1020110759 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f8644031f40 con 0x7f865411a100 2019-12-05T05:13:09.849 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.843+0000 7f865a5b3700 1 -- 172.21.15.42:0/1020110759 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f865405b8c0 con 0x7f865411a100 2019-12-05T05:13:09.850 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.843+0000 7f8659351700 1 --2- 172.21.15.42:0/1020110759 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f865411a100 0x7f865411c280 secure :-1 s=THROTTLE_DONE pgs=509 cs=0 l=1 rx=0x7f8644000f10 tx=0x7f8644010660).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:09.850 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.843+0000 7f86517fa700 1 -- 172.21.15.42:0/1020110759 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) v1 ==== 94+0+1768 (secure 0 0 0) 0x7f86440321a0 con 0x7f865411a100 2019-12-05T05:13:09.851 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 64 2019-12-05T05:13:09.851 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:13:09.851 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":64,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:13:08.347598+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5505":{"gid":5505,"name":"a","rank":0,"incarnation":64,"state":"up:replay","state_seq":2,"addr":"172.21.15.42:6835/211479572","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":211479572},{"type":"v1","addr":"172.21.15.42:6835","nonce":211479572}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:13:09.854 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.847+0000 7f865a5b3700 1 -- 172.21.15.42:0/1020110759 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f863803ae60 msgr2=0x7f863803d370 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:13:09.854 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.847+0000 7f865a5b3700 1 --2- 172.21.15.42:0/1020110759 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f863803ae60 0x7f863803d370 crc :-1 s=READY pgs=230 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:13:09.854 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.847+0000 7f865a5b3700 1 -- 172.21.15.42:0/1020110759 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f865411a100 msgr2=0x7f865411c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:13:09.855 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.847+0000 7f865a5b3700 1 --2- 172.21.15.42:0/1020110759 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f865411a100 0x7f865411c280 secure :-1 s=READY pgs=509 cs=0 l=1 rx=0x7f8644000f10 tx=0x7f8644010660).stop 2019-12-05T05:13:09.855 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.847+0000 7f865a5b3700 1 -- 172.21.15.42:0/1020110759 shutdown_connections 2019-12-05T05:13:09.855 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.847+0000 7f865a5b3700 1 --2- 172.21.15.42:0/1020110759 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f863803ae60 0x7f863803d370 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:13:09.855 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.847+0000 7f865a5b3700 1 --2- 172.21.15.42:0/1020110759 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f865411a100 0x7f865411c280 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:13:09.855 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.847+0000 7f865a5b3700 1 --2- 172.21.15.42:0/1020110759 >> v2:172.21.15.40:3300/0 conn(0x7f865407dcf0 0x7f865411b170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:13:09.855 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.847+0000 7f865a5b3700 1 --2- 172.21.15.42:0/1020110759 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f86540a54f0 0x7f865411e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:13:09.855 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.847+0000 7f865a5b3700 1 -- 172.21.15.42:0/1020110759 shutdown_connections 2019-12-05T05:13:09.856 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.847+0000 7f865a5b3700 1 -- 172.21.15.42:0/1020110759 wait complete. 2019-12-05T05:13:09.856 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:09.847+0000 7f865a5b3700 1 -- 172.21.15.42:0/1020110759 >> 172.21.15.42:0/1020110759 conn(0x7f86540a02c0 msgr2=0x7f865407e4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:13:09.862 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 64, "feature_flags": { "enable_multiple": false, "ever_enabled_multiple": false }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2019-12-05T05:03:35.604738+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 64, "ever_allowed_features": 2, "explicitly_allowed_features": 2, "failed": [], "flags": 18, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5505": { "addr": "172.21.15.42:6835/211479572", "addrs": { "addrvec": [ { "addr": "172.21.15.42:6834", "nonce": 211479572, "type": "v2" }, { "addr": "172.21.15.42:6835", "nonce": 211479572, "type": "v1" } ] }, "export_targets": [], "features": 4540138292836696063, "flags": 0, "gid": 5505, "incarnation": 64, "name": "a", "rank": 0, "state": "up:replay", "state_seq": 2 } }, "last_failure": 0, "last_failure_osd_epoch": 56, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 2, "min_compat_client": "0 (unknown)", "modified": "2019-12-05T05:13:08.347598+0000", "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5505 } } } ], "standbys": [] } 2019-12-05T05:13:11.863 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:13:11.966 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:11.955+0000 7f8fd5cbc700 1 Processor -- start 2019-12-05T05:13:11.966 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:11.955+0000 7f8fd5cbc700 1 -- start start 2019-12-05T05:13:11.966 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:11.955+0000 7f8fd5cbc700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f8fd00a7740 0x7f8fd008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:13:11.967 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:11.955+0000 7f8fd5cbc700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8fd009fae0 0x7f8fd007eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:13:11.967 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:11.955+0000 7f8fd5cbc700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fd00a0710 con 0x7f8fd007e720 2019-12-05T05:13:11.967 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:11.955+0000 7f8fd5cbc700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fd00a0b00 con 0x7f8fd009fae0 2019-12-05T05:13:11.967 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:11.955+0000 7f8fd5cbc700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fd008e5b0 con 0x7f8fd00a7740 2019-12-05T05:13:11.967 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:11.955+0000 7f8fcffff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f8fd00a7740 0x7f8fd008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:13:11.968 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:11.955+0000 7f8fcffff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f8fd00a7740 0x7f8fd008df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:58244/0 (socket says 172.21.15.42:58244) 2019-12-05T05:13:11.968 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:11.955+0000 7f8fcffff700 1 -- 172.21.15.42:0/198551933 learned_addr learned my addr 172.21.15.42:0/198551933 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:13:11.968 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:11.955+0000 7f8fd4a5a700 1 --2- 172.21.15.42:0/198551933 >> v2:172.21.15.40:3300/0 conn(0x7f8fd009fae0 0x7f8fd007eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:13:11.968 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:11.955+0000 7f8fcffff700 1 -- 172.21.15.42:0/198551933 >> v2:172.21.15.40:3300/0 conn(0x7f8fd009fae0 msgr2=0x7f8fd007eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:13:11.968 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:11.955+0000 7f8fcffff700 1 --2- 172.21.15.42:0/198551933 >> v2:172.21.15.40:3300/0 conn(0x7f8fd009fae0 0x7f8fd007eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:13:11.968 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:11.955+0000 7f8fcffff700 1 -- 172.21.15.42:0/198551933 >> v1:172.21.15.40:6789/0 conn(0x7f8fd007e720 legacy=0x7f8fd009f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:13:11.969 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:11.955+0000 7f8fcffff700 1 -- 172.21.15.42:0/198551933 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fd007f0a0 con 0x7f8fd00a7740 2019-12-05T05:13:11.969 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:11.955+0000 7f8fcffff700 1 --2- 172.21.15.42:0/198551933 >> v2:172.21.15.42:3300/0 conn(0x7f8fd00a7740 0x7f8fd008df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f8fc0002f30 tx=0x7f8fc000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:11.969 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:11.955+0000 7f8fcffff700 1 --2- 172.21.15.42:0/198551933 >> v2:172.21.15.42:3300/0 conn(0x7f8fd00a7740 0x7f8fd008df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f8fc0002f30 tx=0x7f8fc000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:11.969 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:11.955+0000 7f8fcffff700 1 --2- 172.21.15.42:0/198551933 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8fd00a7740 0x7f8fd008df80 secure :-1 s=READY pgs=510 cs=0 l=1 rx=0x7f8fc0002f30 tx=0x7f8fc000fd30).ready entity=mon.1 client_cookie=cfe68df7934e15c9 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:13:11.970 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:11.955+0000 7f8fcffff700 1 --2- 172.21.15.42:0/198551933 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8fd00a7740 0x7f8fd008df80 secure :-1 s=THROTTLE_DONE pgs=510 cs=0 l=1 rx=0x7f8fc0002f30 tx=0x7f8fc000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:11.970 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:11.955+0000 7f8fcffff700 1 --2- 172.21.15.42:0/198551933 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8fd00a7740 0x7f8fd008df80 secure :-1 s=THROTTLE_DONE pgs=510 cs=0 l=1 rx=0x7f8fc0002f30 tx=0x7f8fc000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:11.970 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:11.955+0000 7f8fceffd700 1 -- 172.21.15.42:0/198551933 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8fc0008c90 con 0x7f8fd00a7740 2019-12-05T05:13:11.970 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:11.955+0000 7f8fcffff700 1 --2- 172.21.15.42:0/198551933 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8fd00a7740 0x7f8fd008df80 secure :-1 s=THROTTLE_DONE pgs=510 cs=0 l=1 rx=0x7f8fc0002f30 tx=0x7f8fc000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:11.970 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:11.959+0000 7f8fceffd700 1 -- 172.21.15.42:0/198551933 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fc00088b0 con 0x7f8fd00a7740 2019-12-05T05:13:11.970 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:11.959+0000 7f8fceffd700 1 -- 172.21.15.42:0/198551933 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8fc0011ad0 con 0x7f8fd00a7740 2019-12-05T05:13:11.970 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:11.959+0000 7f8fd5cbc700 1 -- 172.21.15.42:0/198551933 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8fd00a7740 msgr2=0x7f8fd008df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:13:11.971 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:11.959+0000 7f8fd5cbc700 1 --2- 172.21.15.42:0/198551933 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8fd00a7740 0x7f8fd008df80 secure :-1 s=READY pgs=510 cs=0 l=1 rx=0x7f8fc0002f30 tx=0x7f8fc000fd30).stop 2019-12-05T05:13:11.971 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:11.959+0000 7f8fd5cbc700 1 -- 172.21.15.42:0/198551933 shutdown_connections 2019-12-05T05:13:11.971 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:11.959+0000 7f8fd5cbc700 1 --2- 172.21.15.42:0/198551933 >> v2:172.21.15.40:3300/0 conn(0x7f8fd009fae0 0x7f8fd007eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:13:11.971 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:11.959+0000 7f8fd5cbc700 1 --2- 172.21.15.42:0/198551933 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8fd00a7740 0x7f8fd008df80 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:13:11.971 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:11.959+0000 7f8fd5cbc700 1 -- 172.21.15.42:0/198551933 shutdown_connections 2019-12-05T05:13:11.971 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:11.959+0000 7f8fd5cbc700 1 -- 172.21.15.42:0/198551933 wait complete. 2019-12-05T05:13:11.971 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:11.959+0000 7f8fd5cbc700 1 -- 172.21.15.42:0/198551933 >> 172.21.15.42:0/198551933 conn(0x7f8fd00aabe0 msgr2=0x7f8fd00a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:13:11.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:11.963+0000 7f8fd5cbc700 1 Processor -- start 2019-12-05T05:13:11.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:11.963+0000 7f8fd5cbc700 1 -- start start 2019-12-05T05:13:11.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:11.963+0000 7f8fd5cbc700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8fd0143ee0 0x7f8fd0142530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:13:11.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:11.963+0000 7f8fd5cbc700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8fd0123ab0 0x7f8fd0123ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:13:11.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:11.963+0000 7f8fd5cbc700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fd0143770 con 0x7f8fd014e140 2019-12-05T05:13:11.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:11.963+0000 7f8fd5cbc700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fd0142cb0 con 0x7f8fd0123ab0 2019-12-05T05:13:11.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:11.963+0000 7f8fd5cbc700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fd0142f50 con 0x7f8fd0143ee0 2019-12-05T05:13:11.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:11.963+0000 7f8fcffff700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8fd0143ee0 0x7f8fd0142530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:13:11.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:11.963+0000 7f8fd4a5a700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8fd0123ab0 0x7f8fd0123ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:13:11.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:11.963+0000 7f8fcffff700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8fd0143ee0 0x7f8fd0142530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:58716/0 (socket says 172.21.15.42:58716) 2019-12-05T05:13:11.976 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:11.963+0000 7f8fd4a5a700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8fd0123ab0 0x7f8fd0123ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.42:46070/0 (socket says 172.21.15.42:46070) 2019-12-05T05:13:11.976 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:11.963+0000 7f8fd4a5a700 1 -- 172.21.15.42:0/1039473273 learned_addr learned my addr 172.21.15.42:0/1039473273 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:13:11.976 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:11.963+0000 7f8fcffff700 1 -- 172.21.15.42:0/1039473273 >> v2:172.21.15.40:3300/0 conn(0x7f8fd0123ab0 msgr2=0x7f8fd0123ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:13:11.976 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:11.963+0000 7f8fcffff700 1 --2- 172.21.15.42:0/1039473273 >> v2:172.21.15.40:3300/0 conn(0x7f8fd0123ab0 0x7f8fd0123ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:13:11.976 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:11.963+0000 7f8fcffff700 1 -- 172.21.15.42:0/1039473273 >> v1:172.21.15.40:6789/0 conn(0x7f8fd014e140 legacy=0x7f8fd014e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:13:11.976 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:11.963+0000 7f8fcffff700 1 -- 172.21.15.42:0/1039473273 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fd0124430 con 0x7f8fd0143ee0 2019-12-05T05:13:11.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:11.963+0000 7f8fd4a5a700 1 --2- 172.21.15.42:0/1039473273 >> v2:172.21.15.40:3300/0 conn(0x7f8fd0123ab0 0x7f8fd0123ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:13:11.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:11.963+0000 7f8fcffff700 1 --2- 172.21.15.42:0/1039473273 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8fd0143ee0 0x7f8fd0142530 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f8fc0002f30 tx=0x7f8fc0008b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:11.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:11.963+0000 7f8fcffff700 1 --2- 172.21.15.42:0/1039473273 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8fd0143ee0 0x7f8fd0142530 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f8fc0002f30 tx=0x7f8fc0008b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:11.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:11.963+0000 7f8fcffff700 1 --2- 172.21.15.42:0/1039473273 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8fd0143ee0 0x7f8fd0142530 secure :-1 s=READY pgs=102 cs=0 l=1 rx=0x7f8fc0002f30 tx=0x7f8fc0008b60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:13:11.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:11.963+0000 7f8fcffff700 1 --2- 172.21.15.42:0/1039473273 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8fd0143ee0 0x7f8fd0142530 secure :-1 s=THROTTLE_DONE pgs=102 cs=0 l=1 rx=0x7f8fc0002f30 tx=0x7f8fc0008b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:11.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:11.963+0000 7f8fccff9700 1 -- 172.21.15.42:0/1039473273 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8fc000f490 con 0x7f8fd0143ee0 2019-12-05T05:13:11.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:11.963+0000 7f8fcffff700 1 --2- 172.21.15.42:0/1039473273 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8fd0143ee0 0x7f8fd0142530 secure :-1 s=THROTTLE_DONE pgs=102 cs=0 l=1 rx=0x7f8fc0002f30 tx=0x7f8fc0008b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:11.978 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:11.963+0000 7f8fcffff700 1 --2- 172.21.15.42:0/1039473273 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8fd0143ee0 0x7f8fd0142530 secure :-1 s=THROTTLE_DONE pgs=102 cs=0 l=1 rx=0x7f8fc0002f30 tx=0x7f8fc0008b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:11.978 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:11.967+0000 7f8fccff9700 1 -- 172.21.15.42:0/1039473273 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fc00116f0 con 0x7f8fd0143ee0 2019-12-05T05:13:11.978 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:11.967+0000 7f8fd5cbc700 1 -- 172.21.15.42:0/1039473273 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fd012e420 con 0x7f8fd0143ee0 2019-12-05T05:13:11.978 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:11.967+0000 7f8fccff9700 1 -- 172.21.15.42:0/1039473273 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8fc0005730 con 0x7f8fd0143ee0 2019-12-05T05:13:11.978 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:11.967+0000 7f8fd5cbc700 1 -- 172.21.15.42:0/1039473273 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fd012e9a0 con 0x7f8fd0143ee0 2019-12-05T05:13:11.978 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:11.967+0000 7f8fcffff700 1 --2- 172.21.15.42:0/1039473273 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8fd0143ee0 0x7f8fd0142530 secure :-1 s=THROTTLE_DONE pgs=102 cs=0 l=1 rx=0x7f8fc0002f30 tx=0x7f8fc0008b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:11.978 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:11.967+0000 7f8fccff9700 1 -- 172.21.15.42:0/1039473273 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f8fc0005a40 con 0x7f8fd0143ee0 2019-12-05T05:13:11.979 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:11.967+0000 7f8fcffff700 1 --2- 172.21.15.42:0/1039473273 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8fd0143ee0 0x7f8fd0142530 secure :-1 s=THROTTLE_DONE pgs=102 cs=0 l=1 rx=0x7f8fc0002f30 tx=0x7f8fc0008b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:11.979 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:11.967+0000 7f8fccff9700 1 --2- 172.21.15.42:0/1039473273 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8fb003ae00 0x7f8fb003d310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:13:11.979 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:11.967+0000 7f8fccff9700 1 -- 172.21.15.42:0/1039473273 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6633+0+0 (secure 0 0 0) 0x7f8fc004b940 con 0x7f8fd0143ee0 2019-12-05T05:13:11.979 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:11.967+0000 7f8fcf7fe700 1 --2- 172.21.15.42:0/1039473273 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8fb003ae00 0x7f8fb003d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:13:11.979 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:11.967+0000 7f8fcf7fe700 1 --2- 172.21.15.42:0/1039473273 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8fb003ae00 0x7f8fb003d310 crc :-1 s=READY pgs=231 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:13:11.980 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:11.971+0000 7f8fd5cbc700 1 -- 172.21.15.42:0/1039473273 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fd005d110 con 0x7f8fd0143ee0 2019-12-05T05:13:11.985 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:11.979+0000 7f8fcffff700 1 --2- 172.21.15.42:0/1039473273 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8fd0143ee0 0x7f8fd0142530 secure :-1 s=THROTTLE_DONE pgs=102 cs=0 l=1 rx=0x7f8fc0002f30 tx=0x7f8fc0008b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:11.985 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:11.979+0000 7f8fccff9700 1 -- 172.21.15.42:0/1039473273 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f8fc00329c0 con 0x7f8fd0143ee0 2019-12-05T05:13:12.161 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:12.147+0000 7f8fd5cbc700 1 -- 172.21.15.42:0/1039473273 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8fd005b8c0 con 0x7f8fd0143ee0 2019-12-05T05:13:12.161 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:12.147+0000 7f8fcffff700 1 --2- 172.21.15.42:0/1039473273 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8fd0143ee0 0x7f8fd0142530 secure :-1 s=THROTTLE_DONE pgs=102 cs=0 l=1 rx=0x7f8fc0002f30 tx=0x7f8fc0008b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:12.161 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:12.147+0000 7f8fccff9700 1 -- 172.21.15.42:0/1039473273 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 65 v65) v1 ==== 94+0+1771 (secure 0 0 0) 0x7f8fc0032c20 con 0x7f8fd0143ee0 2019-12-05T05:13:12.161 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 65 2019-12-05T05:13:12.162 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:12.147+0000 7f8fd5cbc700 1 -- 172.21.15.42:0/1039473273 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8fb003ae00 msgr2=0x7f8fb003d310 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:13:12.162 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:12.147+0000 7f8fd5cbc700 1 --2- 172.21.15.42:0/1039473273 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8fb003ae00 0x7f8fb003d310 crc :-1 s=READY pgs=231 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:13:12.162 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:12.147+0000 7f8fd5cbc700 1 -- 172.21.15.42:0/1039473273 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8fd0143ee0 msgr2=0x7f8fd0142530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:13:12.162 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:12.147+0000 7f8fd5cbc700 1 --2- 172.21.15.42:0/1039473273 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8fd0143ee0 0x7f8fd0142530 secure :-1 s=READY pgs=102 cs=0 l=1 rx=0x7f8fc0002f30 tx=0x7f8fc0008b60).stop 2019-12-05T05:13:12.162 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:12.147+0000 7f8fd5cbc700 1 -- 172.21.15.42:0/1039473273 shutdown_connections 2019-12-05T05:13:12.164 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:12.147+0000 7f8fd5cbc700 1 --2- 172.21.15.42:0/1039473273 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8fb003ae00 0x7f8fb003d310 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:13:12.164 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:12.147+0000 7f8fd5cbc700 1 --2- 172.21.15.42:0/1039473273 >> v2:172.21.15.40:3300/0 conn(0x7f8fd0123ab0 0x7f8fd0123ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:13:12.164 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:12.147+0000 7f8fd5cbc700 1 --2- 172.21.15.42:0/1039473273 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8fd0143ee0 0x7f8fd0142530 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:13:12.165 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:12.151+0000 7f8fd5cbc700 1 -- 172.21.15.42:0/1039473273 shutdown_connections 2019-12-05T05:13:12.165 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:12.151+0000 7f8fd5cbc700 1 -- 172.21.15.42:0/1039473273 wait complete. 2019-12-05T05:13:12.165 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:12.151+0000 7f8fd5cbc700 1 -- 172.21.15.42:0/1039473273 >> 172.21.15.42:0/1039473273 conn(0x7f8fd00a89d0 msgr2=0x7f8fd009f620 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:13:12.165 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:13:12.165 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":65,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:13:11.380521+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5505":{"gid":5505,"name":"a","rank":0,"incarnation":64,"state":"up:reconnect","state_seq":7,"addr":"172.21.15.42:6835/211479572","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":211479572},{"type":"v1","addr":"172.21.15.42:6835","nonce":211479572}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:13:12.166 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2019-12-05T05:13:14.167 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:13:14.279 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:14.267+0000 7f6bebe76700 1 Processor -- start 2019-12-05T05:13:14.279 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:14.267+0000 7f6bebe76700 1 -- start start 2019-12-05T05:13:14.280 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:14.267+0000 7f6bebe76700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6be40a7740 0x7f6be408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:13:14.280 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:14.267+0000 7f6bebe76700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f6be409fae0 0x7f6be407eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:13:14.280 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:14.267+0000 7f6bebe76700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6be40a0710 con 0x7f6be407e720 2019-12-05T05:13:14.280 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:14.267+0000 7f6bebe76700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6be40a0b00 con 0x7f6be409fae0 2019-12-05T05:13:14.280 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:14.267+0000 7f6bebe76700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6be408e5b0 con 0x7f6be40a7740 2019-12-05T05:13:14.281 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:14.267+0000 7f6bea413700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6be40a7740 0x7f6be408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:13:14.281 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:14.267+0000 7f6be9c12700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f6be407e720 0x7f6be409f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:58300/0 (socket says 172.21.15.42:58300) 2019-12-05T05:13:14.281 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:14.267+0000 7f6be9c12700 1 -- 172.21.15.42:0/275829830 learned_addr learned my addr 172.21.15.42:0/275829830 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:13:14.281 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:14.267+0000 7f6beac14700 1 --2- 172.21.15.42:0/275829830 >> v2:172.21.15.40:3300/0 conn(0x7f6be409fae0 0x7f6be407eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:13:14.281 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:14.267+0000 7f6bea413700 1 -- 172.21.15.42:0/275829830 >> v2:172.21.15.40:3300/0 conn(0x7f6be409fae0 msgr2=0x7f6be407eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:13:14.281 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:14.267+0000 7f6bea413700 1 --2- 172.21.15.42:0/275829830 >> v2:172.21.15.40:3300/0 conn(0x7f6be409fae0 0x7f6be407eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:13:14.281 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:14.267+0000 7f6be9411700 1 -- 172.21.15.42:0/275829830 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 73376372 0 0) 0x7f6be40a0710 con 0x7f6be407e720 2019-12-05T05:13:14.282 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:14.267+0000 7f6bea413700 1 -- 172.21.15.42:0/275829830 >> v1:172.21.15.42:6789/0 conn(0x7f6be407e720 legacy=0x7f6be409f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:13:14.282 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:14.267+0000 7f6bea413700 1 -- 172.21.15.42:0/275829830 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6be407f0a0 con 0x7f6be40a7740 2019-12-05T05:13:14.282 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:14.271+0000 7f6beac14700 1 --2- 172.21.15.42:0/275829830 >> v2:172.21.15.40:3300/0 conn(0x7f6be409fae0 0x7f6be407eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T05:13:14.282 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:14.271+0000 7f6bea413700 1 --2- 172.21.15.42:0/275829830 >> v2:172.21.15.42:3300/0 conn(0x7f6be40a7740 0x7f6be408df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f6be0002f30 tx=0x7f6be000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:14.282 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:14.271+0000 7f6bea413700 1 --2- 172.21.15.42:0/275829830 >> v2:172.21.15.42:3300/0 conn(0x7f6be40a7740 0x7f6be408df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f6be0002f30 tx=0x7f6be000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:14.282 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:14.271+0000 7f6bea413700 1 --2- 172.21.15.42:0/275829830 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6be40a7740 0x7f6be408df80 secure :-1 s=READY pgs=512 cs=0 l=1 rx=0x7f6be0002f30 tx=0x7f6be000fc70).ready entity=mon.1 client_cookie=407d69ae5155df0b server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:13:14.283 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:14.271+0000 7f6bea413700 1 --2- 172.21.15.42:0/275829830 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6be40a7740 0x7f6be408df80 secure :-1 s=THROTTLE_DONE pgs=512 cs=0 l=1 rx=0x7f6be0002f30 tx=0x7f6be000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:14.283 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:14.271+0000 7f6bea413700 1 --2- 172.21.15.42:0/275829830 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6be40a7740 0x7f6be408df80 secure :-1 s=THROTTLE_DONE pgs=512 cs=0 l=1 rx=0x7f6be0002f30 tx=0x7f6be000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:14.283 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:14.271+0000 7f6be9411700 1 -- 172.21.15.42:0/275829830 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6be0008bb0 con 0x7f6be40a7740 2019-12-05T05:13:14.283 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:14.271+0000 7f6be9411700 1 -- 172.21.15.42:0/275829830 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6be00087d0 con 0x7f6be40a7740 2019-12-05T05:13:14.283 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:14.271+0000 7f6bea413700 1 --2- 172.21.15.42:0/275829830 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6be40a7740 0x7f6be408df80 secure :-1 s=THROTTLE_DONE pgs=512 cs=0 l=1 rx=0x7f6be0002f30 tx=0x7f6be000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:14.284 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:14.271+0000 7f6be9411700 1 -- 172.21.15.42:0/275829830 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6be0011820 con 0x7f6be40a7740 2019-12-05T05:13:14.284 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:14.271+0000 7f6bebe76700 1 -- 172.21.15.42:0/275829830 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6be40a7740 msgr2=0x7f6be408df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:13:14.284 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:14.271+0000 7f6bebe76700 1 --2- 172.21.15.42:0/275829830 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6be40a7740 0x7f6be408df80 secure :-1 s=READY pgs=512 cs=0 l=1 rx=0x7f6be0002f30 tx=0x7f6be000fc70).stop 2019-12-05T05:13:14.284 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:14.271+0000 7f6bebe76700 1 -- 172.21.15.42:0/275829830 shutdown_connections 2019-12-05T05:13:14.284 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:14.271+0000 7f6bebe76700 1 --2- 172.21.15.42:0/275829830 >> v2:172.21.15.40:3300/0 conn(0x7f6be409fae0 0x7f6be407eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:13:14.284 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:14.271+0000 7f6bebe76700 1 --2- 172.21.15.42:0/275829830 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6be40a7740 0x7f6be408df80 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:13:14.285 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:14.271+0000 7f6bebe76700 1 -- 172.21.15.42:0/275829830 shutdown_connections 2019-12-05T05:13:14.285 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:14.271+0000 7f6bebe76700 1 -- 172.21.15.42:0/275829830 wait complete. 2019-12-05T05:13:14.285 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:14.271+0000 7f6bebe76700 1 -- 172.21.15.42:0/275829830 >> 172.21.15.42:0/275829830 conn(0x7f6be40aabe0 msgr2=0x7f6be40a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:13:14.288 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:14.275+0000 7f6bebe76700 1 Processor -- start 2019-12-05T05:13:14.288 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:14.275+0000 7f6bebe76700 1 -- start start 2019-12-05T05:13:14.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:14.275+0000 7f6bebe76700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f6be408ee90 0x7f6be4065050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:13:14.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:14.275+0000 7f6bebe76700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6be4115550 0x7f6be4115980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:13:14.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:14.275+0000 7f6bebe76700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6be407c100 con 0x7f6be4063be0 2019-12-05T05:13:14.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:14.275+0000 7f6bebe76700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6be40632e0 con 0x7f6be408ee90 2019-12-05T05:13:14.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:14.275+0000 7f6bebe76700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6be4065710 con 0x7f6be4115550 2019-12-05T05:13:14.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:14.275+0000 7f6bea413700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f6be408ee90 0x7f6be4065050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:13:14.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:14.275+0000 7f6beac14700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6be4115550 0x7f6be4115980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:13:14.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:14.275+0000 7f6bea413700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f6be408ee90 0x7f6be4065050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.42:46084/0 (socket says 172.21.15.42:46084) 2019-12-05T05:13:14.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:14.275+0000 7f6bea413700 1 -- 172.21.15.42:0/25980306 learned_addr learned my addr 172.21.15.42:0/25980306 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:13:14.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:14.275+0000 7f6bea413700 1 -- 172.21.15.42:0/25980306 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6be4115550 msgr2=0x7f6be4115980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:13:14.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:14.275+0000 7f6bea413700 1 --2- 172.21.15.42:0/25980306 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6be4115550 0x7f6be4115980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:13:14.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:14.275+0000 7f6bea413700 1 -- 172.21.15.42:0/25980306 >> v1:172.21.15.40:6789/0 conn(0x7f6be4063be0 legacy=0x7f6be407b900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:13:14.291 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:14.275+0000 7f6bea413700 1 -- 172.21.15.42:0/25980306 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6be4115ed0 con 0x7f6be408ee90 2019-12-05T05:13:14.291 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:14.275+0000 7f6beac14700 1 --2- 172.21.15.42:0/25980306 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6be4115550 0x7f6be4115980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:13:14.291 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:14.275+0000 7f6bea413700 1 --2- 172.21.15.42:0/25980306 >> v2:172.21.15.40:3300/0 conn(0x7f6be408ee90 0x7f6be4065050 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f6be40658d0 tx=0x7f6be0008a70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:14.291 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:14.275+0000 7f6bea413700 1 --2- 172.21.15.42:0/25980306 >> v2:172.21.15.40:3300/0 conn(0x7f6be408ee90 0x7f6be4065050 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f6be40658d0 tx=0x7f6be0008a70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:14.291 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:14.275+0000 7f6bea413700 1 --2- 172.21.15.42:0/25980306 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6be408ee90 0x7f6be4065050 secure :-1 s=READY pgs=400 cs=0 l=1 rx=0x7f6be40658d0 tx=0x7f6be0008a70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:13:14.291 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:14.275+0000 7f6bea413700 1 --2- 172.21.15.42:0/25980306 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6be408ee90 0x7f6be4065050 secure :-1 s=THROTTLE_DONE pgs=400 cs=0 l=1 rx=0x7f6be40658d0 tx=0x7f6be0008a70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:14.291 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:14.275+0000 7f6bea413700 1 --2- 172.21.15.42:0/25980306 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6be408ee90 0x7f6be4065050 secure :-1 s=THROTTLE_DONE pgs=400 cs=0 l=1 rx=0x7f6be40658d0 tx=0x7f6be0008a70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:14.292 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:14.275+0000 7f6bea413700 1 --2- 172.21.15.42:0/25980306 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6be408ee90 0x7f6be4065050 secure :-1 s=THROTTLE_DONE pgs=400 cs=0 l=1 rx=0x7f6be40658d0 tx=0x7f6be0008a70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:14.292 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:14.275+0000 7f6bdaffd700 1 -- 172.21.15.42:0/25980306 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6be0005540 con 0x7f6be408ee90 2019-12-05T05:13:14.292 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:14.275+0000 7f6bdaffd700 1 -- 172.21.15.42:0/25980306 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6be0008db0 con 0x7f6be408ee90 2019-12-05T05:13:14.292 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:14.275+0000 7f6bebe76700 1 -- 172.21.15.42:0/25980306 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6be4131640 con 0x7f6be408ee90 2019-12-05T05:13:14.292 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:14.275+0000 7f6bdaffd700 1 -- 172.21.15.42:0/25980306 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6be0005b20 con 0x7f6be408ee90 2019-12-05T05:13:14.292 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:14.275+0000 7f6bebe76700 1 -- 172.21.15.42:0/25980306 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6be4131bc0 con 0x7f6be408ee90 2019-12-05T05:13:14.293 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:14.279+0000 7f6bea413700 1 --2- 172.21.15.42:0/25980306 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6be408ee90 0x7f6be4065050 secure :-1 s=THROTTLE_DONE pgs=400 cs=0 l=1 rx=0x7f6be40658d0 tx=0x7f6be0008a70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:14.293 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:14.279+0000 7f6bea413700 1 --2- 172.21.15.42:0/25980306 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6be408ee90 0x7f6be4065050 secure :-1 s=THROTTLE_DONE pgs=400 cs=0 l=1 rx=0x7f6be40658d0 tx=0x7f6be0008a70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:14.293 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:14.279+0000 7f6bdaffd700 1 -- 172.21.15.42:0/25980306 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f6be00130b0 con 0x7f6be408ee90 2019-12-05T05:13:14.293 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:14.279+0000 7f6bdaffd700 1 --2- 172.21.15.42:0/25980306 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6bcc03ad70 0x7f6bcc03d280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:13:14.293 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:14.279+0000 7f6bdaffd700 1 -- 172.21.15.42:0/25980306 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6633+0+0 (secure 0 0 0) 0x7f6be004b850 con 0x7f6be408ee90 2019-12-05T05:13:14.294 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:14.279+0000 7f6be9c12700 1 --2- 172.21.15.42:0/25980306 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6bcc03ad70 0x7f6bcc03d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:13:14.294 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:14.279+0000 7f6be9c12700 1 --2- 172.21.15.42:0/25980306 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6bcc03ad70 0x7f6bcc03d280 crc :-1 s=READY pgs=232 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:13:14.294 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:14.283+0000 7f6bebe76700 1 -- 172.21.15.42:0/25980306 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6be405d110 con 0x7f6be408ee90 2019-12-05T05:13:14.297 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:14.291+0000 7f6bea413700 1 --2- 172.21.15.42:0/25980306 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6be408ee90 0x7f6be4065050 secure :-1 s=THROTTLE_DONE pgs=400 cs=0 l=1 rx=0x7f6be40658d0 tx=0x7f6be0008a70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:14.298 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:14.291+0000 7f6bdaffd700 1 -- 172.21.15.42:0/25980306 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f6be0030530 con 0x7f6be408ee90 2019-12-05T05:13:14.467 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:14.459+0000 7f6bebe76700 1 -- 172.21.15.42:0/25980306 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6be405b8c0 con 0x7f6be408ee90 2019-12-05T05:13:14.468 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:14.463+0000 7f6bea413700 1 --2- 172.21.15.42:0/25980306 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6be408ee90 0x7f6be4065050 secure :-1 s=THROTTLE_DONE pgs=400 cs=0 l=1 rx=0x7f6be40658d0 tx=0x7f6be0008a70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:14.468 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:14.463+0000 7f6bdaffd700 1 -- 172.21.15.42:0/25980306 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+1768 (secure 0 0 0) 0x7f6be00336e0 con 0x7f6be408ee90 2019-12-05T05:13:14.469 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 67 2019-12-05T05:13:14.469 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:13:14.469 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":67,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:13:13.391909+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5505":{"gid":5505,"name":"a","rank":0,"incarnation":64,"state":"up:active","state_seq":9,"addr":"172.21.15.42:6835/211479572","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":211479572},{"type":"v1","addr":"172.21.15.42:6835","nonce":211479572}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:13:14.471 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:14.463+0000 7f6bebe76700 1 -- 172.21.15.42:0/25980306 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6bcc03ad70 msgr2=0x7f6bcc03d280 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:13:14.471 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:14.463+0000 7f6bebe76700 1 --2- 172.21.15.42:0/25980306 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6bcc03ad70 0x7f6bcc03d280 crc :-1 s=READY pgs=232 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:13:14.471 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:14.463+0000 7f6bebe76700 1 -- 172.21.15.42:0/25980306 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6be408ee90 msgr2=0x7f6be4065050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:13:14.472 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:14.463+0000 7f6bebe76700 1 --2- 172.21.15.42:0/25980306 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6be408ee90 0x7f6be4065050 secure :-1 s=READY pgs=400 cs=0 l=1 rx=0x7f6be40658d0 tx=0x7f6be0008a70).stop 2019-12-05T05:13:14.472 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:14.463+0000 7f6bebe76700 1 -- 172.21.15.42:0/25980306 shutdown_connections 2019-12-05T05:13:14.472 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:14.463+0000 7f6bebe76700 1 --2- 172.21.15.42:0/25980306 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6bcc03ad70 0x7f6bcc03d280 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:13:14.472 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:14.463+0000 7f6bebe76700 1 --2- 172.21.15.42:0/25980306 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6be4115550 0x7f6be4115980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:13:14.472 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:14.463+0000 7f6bebe76700 1 --2- 172.21.15.42:0/25980306 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6be408ee90 0x7f6be4065050 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:13:14.472 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:14.463+0000 7f6bebe76700 1 -- 172.21.15.42:0/25980306 shutdown_connections 2019-12-05T05:13:14.473 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:14.463+0000 7f6bebe76700 1 -- 172.21.15.42:0/25980306 wait complete. 2019-12-05T05:13:14.473 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:14.463+0000 7f6bebe76700 1 -- 172.21.15.42:0/25980306 >> 172.21.15.42:0/25980306 conn(0x7f6be40aabe0 msgr2=0x7f6be40a9750 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:13:14.481 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a has gained rank=0, replacing gid=5391 2019-12-05T05:13:14.481 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 1 secs before reviving mds.b 2019-12-05T05:13:15.483 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.b 2019-12-05T05:13:15.483 INFO:tasks.ceph.mds.b:Restarting daemon 2019-12-05T05:13:15.483 INFO:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2019-12-05T05:13:15.487 INFO:tasks.ceph.mds.b:Started 2019-12-05T05:13:15.487 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:13:15.551 INFO:tasks.ceph.mds.b.smithi040.stdout:starting mds.b at 2019-12-05T05:13:15.593 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:15.579+0000 7f2acc3d3700 1 Processor -- start 2019-12-05T05:13:15.593 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:15.579+0000 7f2acc3d3700 1 -- start start 2019-12-05T05:13:15.593 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:15.579+0000 7f2acc3d3700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2ac409fae0 0x7f2ac414fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:13:15.593 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:15.579+0000 7f2acc3d3700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ac40a0710 con 0x7f2ac40a7740 2019-12-05T05:13:15.593 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:15.579+0000 7f2acc3d3700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ac407ef20 con 0x7f2ac408df80 2019-12-05T05:13:15.593 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:15.579+0000 7f2acc3d3700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ac40a0b00 con 0x7f2ac409fae0 2019-12-05T05:13:15.594 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:15.579+0000 7f2aca16f700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f2ac408df80 0x7f2ac409f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:58318/0 (socket says 172.21.15.42:58318) 2019-12-05T05:13:15.594 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:15.579+0000 7f2aca16f700 1 -- 172.21.15.42:0/2792513643 learned_addr learned my addr 172.21.15.42:0/2792513643 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:13:15.594 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:15.579+0000 7f2acb171700 1 --2- 172.21.15.42:0/2792513643 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2ac409fae0 0x7f2ac414fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:13:15.594 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:15.579+0000 7f2ac996e700 1 -- 172.21.15.42:0/2792513643 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2556374119 0 0) 0x7f2ac407ef20 con 0x7f2ac408df80 2019-12-05T05:13:15.594 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:15.579+0000 7f2ac996e700 1 -- 172.21.15.42:0/2792513643 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ab00030b0 con 0x7f2ac408df80 2019-12-05T05:13:15.594 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:15.579+0000 7f2ac996e700 1 -- 172.21.15.42:0/2792513643 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 389108085 0 0) 0x7f2ac40a0710 con 0x7f2ac40a7740 2019-12-05T05:13:15.594 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:15.579+0000 7f2acb171700 1 -- 172.21.15.42:0/2792513643 >> v1:172.21.15.42:6789/0 conn(0x7f2ac408df80 legacy=0x7f2ac409f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:13:15.595 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:15.579+0000 7f2acb171700 1 -- 172.21.15.42:0/2792513643 >> v1:172.21.15.40:6789/0 conn(0x7f2ac40a7740 legacy=0x7f2ac407e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:13:15.595 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:15.579+0000 7f2acb171700 1 -- 172.21.15.42:0/2792513643 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ac40a9ba0 con 0x7f2ac409fae0 2019-12-05T05:13:15.595 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:15.579+0000 7f2acb171700 1 --2- 172.21.15.42:0/2792513643 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2ac409fae0 0x7f2ac414fff0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f2abc0032b0 tx=0x7f2abc00ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:15.597 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:15.579+0000 7f2acb171700 1 --2- 172.21.15.42:0/2792513643 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2ac409fae0 0x7f2ac414fff0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f2abc0032b0 tx=0x7f2abc00ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:15.597 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:15.583+0000 7f2acb171700 1 --2- 172.21.15.42:0/2792513643 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2ac409fae0 0x7f2ac414fff0 secure :-1 s=READY pgs=105 cs=0 l=1 rx=0x7f2abc0032b0 tx=0x7f2abc00ff70).ready entity=mon.2 client_cookie=f56c3dd0d4ef9f4b server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:13:15.597 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:15.583+0000 7f2acb171700 1 --2- 172.21.15.42:0/2792513643 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2ac409fae0 0x7f2ac414fff0 secure :-1 s=THROTTLE_DONE pgs=105 cs=0 l=1 rx=0x7f2abc0032b0 tx=0x7f2abc00ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:15.597 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:15.583+0000 7f2ac996e700 1 -- 172.21.15.42:0/2792513643 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2abc0109b0 con 0x7f2ac409fae0 2019-12-05T05:13:15.597 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:15.583+0000 7f2acb171700 1 --2- 172.21.15.42:0/2792513643 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2ac409fae0 0x7f2ac414fff0 secure :-1 s=THROTTLE_DONE pgs=105 cs=0 l=1 rx=0x7f2abc0032b0 tx=0x7f2abc00ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:15.597 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:15.583+0000 7f2acb171700 1 --2- 172.21.15.42:0/2792513643 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2ac409fae0 0x7f2ac414fff0 secure :-1 s=THROTTLE_DONE pgs=105 cs=0 l=1 rx=0x7f2abc0032b0 tx=0x7f2abc00ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:15.597 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:15.583+0000 7f2ac996e700 1 -- 172.21.15.42:0/2792513643 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2abc008b50 con 0x7f2ac409fae0 2019-12-05T05:13:15.598 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:15.583+0000 7f2ac996e700 1 -- 172.21.15.42:0/2792513643 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2abc0110c0 con 0x7f2ac409fae0 2019-12-05T05:13:15.598 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:15.583+0000 7f2acc3d3700 1 -- 172.21.15.42:0/2792513643 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2ac409fae0 msgr2=0x7f2ac414fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:13:15.598 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:15.583+0000 7f2acc3d3700 1 --2- 172.21.15.42:0/2792513643 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2ac409fae0 0x7f2ac414fff0 secure :-1 s=READY pgs=105 cs=0 l=1 rx=0x7f2abc0032b0 tx=0x7f2abc00ff70).stop 2019-12-05T05:13:15.598 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:15.583+0000 7f2acc3d3700 1 -- 172.21.15.42:0/2792513643 shutdown_connections 2019-12-05T05:13:15.598 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:15.583+0000 7f2acc3d3700 1 --2- 172.21.15.42:0/2792513643 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2ac409fae0 0x7f2ac414fff0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:13:15.598 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:15.583+0000 7f2acc3d3700 1 -- 172.21.15.42:0/2792513643 shutdown_connections 2019-12-05T05:13:15.598 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:15.583+0000 7f2acc3d3700 1 -- 172.21.15.42:0/2792513643 wait complete. 2019-12-05T05:13:15.599 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:15.583+0000 7f2acc3d3700 1 -- 172.21.15.42:0/2792513643 >> 172.21.15.42:0/2792513643 conn(0x7f2ac40aabe0 msgr2=0x7f2ac40a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:13:15.599 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:15.587+0000 7f2acc3d3700 1 Processor -- start 2019-12-05T05:13:15.599 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:15.587+0000 7f2acc3d3700 1 -- start start 2019-12-05T05:13:15.603 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:15.587+0000 7f2acc3d3700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2ac41165d0 0x7f2ac4116f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:13:15.603 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:15.587+0000 7f2acc3d3700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2ac4130af0 0x7f2ac4130f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:13:15.604 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:15.587+0000 7f2acc3d3700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ac4130580 con 0x7f2ac4118790 2019-12-05T05:13:15.604 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:15.587+0000 7f2acc3d3700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ac4131470 con 0x7f2ac41165d0 2019-12-05T05:13:15.604 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:15.587+0000 7f2acc3d3700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ac4117620 con 0x7f2ac4130af0 2019-12-05T05:13:15.604 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:15.587+0000 7f2acb171700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2ac4130af0 0x7f2ac4130f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:13:15.604 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:15.587+0000 7f2acb171700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2ac4130af0 0x7f2ac4130f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:58288/0 (socket says 172.21.15.42:58288) 2019-12-05T05:13:15.604 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:15.587+0000 7f2acb171700 1 -- 172.21.15.42:0/3000008398 learned_addr learned my addr 172.21.15.42:0/3000008398 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:13:15.604 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:15.587+0000 7f2aca970700 1 --2- 172.21.15.42:0/3000008398 >> v2:172.21.15.40:3300/0 conn(0x7f2ac41165d0 0x7f2ac4116f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:13:15.605 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:15.587+0000 7f2acb171700 1 -- 172.21.15.42:0/3000008398 >> v2:172.21.15.40:3300/0 conn(0x7f2ac41165d0 msgr2=0x7f2ac4116f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:13:15.605 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:15.587+0000 7f2acb171700 1 --2- 172.21.15.42:0/3000008398 >> v2:172.21.15.40:3300/0 conn(0x7f2ac41165d0 0x7f2ac4116f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:13:15.605 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:15.587+0000 7f2acb171700 1 -- 172.21.15.42:0/3000008398 >> v1:172.21.15.40:6789/0 conn(0x7f2ac4118790 legacy=0x7f2ac41192c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:13:15.605 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:15.587+0000 7f2acb171700 1 -- 172.21.15.42:0/3000008398 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ac4119ac0 con 0x7f2ac4130af0 2019-12-05T05:13:15.605 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:15.587+0000 7f2acb171700 1 --2- 172.21.15.42:0/3000008398 >> v2:172.21.15.42:3300/0 conn(0x7f2ac4130af0 0x7f2ac4130f20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f2abc0032b0 tx=0x7f2abc010ea0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:15.607 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:15.591+0000 7f2acb171700 1 --2- 172.21.15.42:0/3000008398 >> v2:172.21.15.42:3300/0 conn(0x7f2ac4130af0 0x7f2ac4130f20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f2abc0032b0 tx=0x7f2abc010ea0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:15.607 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:15.591+0000 7f2acb171700 1 --2- 172.21.15.42:0/3000008398 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2ac4130af0 0x7f2ac4130f20 secure :-1 s=READY pgs=514 cs=0 l=1 rx=0x7f2abc0032b0 tx=0x7f2abc010ea0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:13:15.607 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:15.591+0000 7f2acb171700 1 --2- 172.21.15.42:0/3000008398 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2ac4130af0 0x7f2ac4130f20 secure :-1 s=THROTTLE_DONE pgs=514 cs=0 l=1 rx=0x7f2abc0032b0 tx=0x7f2abc010ea0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:15.607 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:15.591+0000 7f2acb171700 1 --2- 172.21.15.42:0/3000008398 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2ac4130af0 0x7f2ac4130f20 secure :-1 s=THROTTLE_DONE pgs=514 cs=0 l=1 rx=0x7f2abc0032b0 tx=0x7f2abc010ea0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:15.607 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:15.591+0000 7f2abb7fe700 1 -- 172.21.15.42:0/3000008398 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2abc005ab0 con 0x7f2ac4130af0 2019-12-05T05:13:15.607 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:15.591+0000 7f2acb171700 1 --2- 172.21.15.42:0/3000008398 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2ac4130af0 0x7f2ac4130f20 secure :-1 s=THROTTLE_DONE pgs=514 cs=0 l=1 rx=0x7f2abc0032b0 tx=0x7f2abc010ea0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:15.607 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:15.591+0000 7f2abb7fe700 1 -- 172.21.15.42:0/3000008398 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2abc010bb0 con 0x7f2ac4130af0 2019-12-05T05:13:15.608 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:15.591+0000 7f2acc3d3700 1 -- 172.21.15.42:0/3000008398 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ac4118bc0 con 0x7f2ac4130af0 2019-12-05T05:13:15.608 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:15.591+0000 7f2abb7fe700 1 -- 172.21.15.42:0/3000008398 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2abc00f680 con 0x7f2ac4130af0 2019-12-05T05:13:15.608 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:15.591+0000 7f2acc3d3700 1 -- 172.21.15.42:0/3000008398 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ac41379d0 con 0x7f2ac4130af0 2019-12-05T05:13:15.608 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:15.591+0000 7f2acb171700 1 --2- 172.21.15.42:0/3000008398 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2ac4130af0 0x7f2ac4130f20 secure :-1 s=THROTTLE_DONE pgs=514 cs=0 l=1 rx=0x7f2abc0032b0 tx=0x7f2abc010ea0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:15.608 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:15.591+0000 7f2abb7fe700 1 -- 172.21.15.42:0/3000008398 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f2abc0113d0 con 0x7f2ac4130af0 2019-12-05T05:13:15.609 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:15.591+0000 7f2acb171700 1 --2- 172.21.15.42:0/3000008398 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2ac4130af0 0x7f2ac4130f20 secure :-1 s=THROTTLE_DONE pgs=514 cs=0 l=1 rx=0x7f2abc0032b0 tx=0x7f2abc010ea0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:15.609 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:15.591+0000 7f2abb7fe700 1 --2- 172.21.15.42:0/3000008398 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2aac03ace0 0x7f2aac03d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:13:15.609 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:15.591+0000 7f2abb7fe700 1 -- 172.21.15.42:0/3000008398 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6633+0+0 (secure 0 0 0) 0x7f2abc04b080 con 0x7f2ac4130af0 2019-12-05T05:13:15.609 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:15.591+0000 7f2aca970700 1 --2- 172.21.15.42:0/3000008398 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2aac03ace0 0x7f2aac03d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:13:15.609 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:15.591+0000 7f2aca970700 1 --2- 172.21.15.42:0/3000008398 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2aac03ace0 0x7f2aac03d1f0 crc :-1 s=READY pgs=233 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:13:15.610 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:15.595+0000 7f2acc3d3700 1 -- 172.21.15.42:0/3000008398 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ac405d110 con 0x7f2ac4130af0 2019-12-05T05:13:15.610 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:15.599+0000 7f2acb171700 1 --2- 172.21.15.42:0/3000008398 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2ac4130af0 0x7f2ac4130f20 secure :-1 s=THROTTLE_DONE pgs=514 cs=0 l=1 rx=0x7f2abc0032b0 tx=0x7f2abc010ea0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:15.610 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:15.599+0000 7f2abb7fe700 1 -- 172.21.15.42:0/3000008398 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f2abc011cd0 con 0x7f2ac4130af0 2019-12-05T05:13:15.769 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:15.763+0000 7f2acc3d3700 1 -- 172.21.15.42:0/3000008398 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2ac405b8c0 con 0x7f2ac4130af0 2019-12-05T05:13:15.770 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:15.763+0000 7f2acb171700 1 --2- 172.21.15.42:0/3000008398 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2ac4130af0 0x7f2ac4130f20 secure :-1 s=THROTTLE_DONE pgs=514 cs=0 l=1 rx=0x7f2abc0032b0 tx=0x7f2abc010ea0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:15.771 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:15.763+0000 7f2abb7fe700 1 -- 172.21.15.42:0/3000008398 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+1768 (secure 0 0 0) 0x7f2abc032f10 con 0x7f2ac4130af0 2019-12-05T05:13:15.771 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 67 2019-12-05T05:13:15.771 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:13:15.771 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":67,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:13:13.391909+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5505":{"gid":5505,"name":"a","rank":0,"incarnation":64,"state":"up:active","state_seq":9,"addr":"172.21.15.42:6835/211479572","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":211479572},{"type":"v1","addr":"172.21.15.42:6835","nonce":211479572}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:13:15.772 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:15.767+0000 7f2acc3d3700 1 -- 172.21.15.42:0/3000008398 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2aac03ace0 msgr2=0x7f2aac03d1f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:13:15.773 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:15.767+0000 7f2acc3d3700 1 --2- 172.21.15.42:0/3000008398 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2aac03ace0 0x7f2aac03d1f0 crc :-1 s=READY pgs=233 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:13:15.773 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:15.767+0000 7f2acc3d3700 1 -- 172.21.15.42:0/3000008398 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2ac4130af0 msgr2=0x7f2ac4130f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:13:15.773 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:15.767+0000 7f2acc3d3700 1 --2- 172.21.15.42:0/3000008398 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2ac4130af0 0x7f2ac4130f20 secure :-1 s=READY pgs=514 cs=0 l=1 rx=0x7f2abc0032b0 tx=0x7f2abc010ea0).stop 2019-12-05T05:13:15.774 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:15.767+0000 7f2acc3d3700 1 -- 172.21.15.42:0/3000008398 shutdown_connections 2019-12-05T05:13:15.774 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:15.767+0000 7f2acc3d3700 1 --2- 172.21.15.42:0/3000008398 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2aac03ace0 0x7f2aac03d1f0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:13:15.774 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:15.767+0000 7f2acc3d3700 1 --2- 172.21.15.42:0/3000008398 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2ac4130af0 0x7f2ac4130f20 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:13:15.774 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:15.767+0000 7f2acc3d3700 1 --2- 172.21.15.42:0/3000008398 >> v2:172.21.15.40:3300/0 conn(0x7f2ac41165d0 0x7f2ac4116f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:13:15.774 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:15.767+0000 7f2acc3d3700 1 -- 172.21.15.42:0/3000008398 shutdown_connections 2019-12-05T05:13:15.775 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:15.767+0000 7f2acc3d3700 1 -- 172.21.15.42:0/3000008398 wait complete. 2019-12-05T05:13:15.775 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:15.767+0000 7f2acc3d3700 1 -- 172.21.15.42:0/3000008398 >> 172.21.15.42:0/3000008398 conn(0x7f2ac409fa20 msgr2=0x7f2ac411d960 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:13:15.783 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2019-12-05T05:13:17.784 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:13:17.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:17.879+0000 7f5fdef82700 1 Processor -- start 2019-12-05T05:13:17.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:17.879+0000 7f5fdef82700 1 -- start start 2019-12-05T05:13:17.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:17.883+0000 7f5fdef82700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5fd80a7730 0x7f5fd808df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:13:17.897 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:17.883+0000 7f5fdef82700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fd80a0700 con 0x7f5fd809fad0 2019-12-05T05:13:17.897 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:17.883+0000 7f5fdef82700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fd80a9b90 con 0x7f5fd807e710 2019-12-05T05:13:17.897 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:17.883+0000 7f5fdef82700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fd80a0af0 con 0x7f5fd80a7730 2019-12-05T05:13:17.897 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:17.883+0000 7f5fdcd1e700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f5fd807e710 0x7f5fd809f2d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:58332/0 (socket says 172.21.15.42:58332) 2019-12-05T05:13:17.897 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:17.883+0000 7f5fdcd1e700 1 -- 172.21.15.42:0/3800153606 learned_addr learned my addr 172.21.15.42:0/3800153606 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:13:17.897 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:17.883+0000 7f5fdd51f700 1 --2- 172.21.15.42:0/3800153606 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5fd80a7730 0x7f5fd808df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:13:17.898 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:17.883+0000 7f5fcbfff700 1 -- 172.21.15.42:0/3800153606 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3929581971 0 0) 0x7f5fd80a9b90 con 0x7f5fd807e710 2019-12-05T05:13:17.898 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:17.883+0000 7f5fcbfff700 1 -- 172.21.15.42:0/3800153606 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5fc40030b0 con 0x7f5fd807e710 2019-12-05T05:13:17.898 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:17.883+0000 7f5fcbfff700 1 -- 172.21.15.42:0/3800153606 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 818454733 0 0) 0x7f5fd80a0700 con 0x7f5fd809fad0 2019-12-05T05:13:17.898 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:17.883+0000 7f5fcbfff700 1 -- 172.21.15.42:0/3800153606 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5fd80a9b90 con 0x7f5fd809fad0 2019-12-05T05:13:17.898 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:17.883+0000 7f5fdd51f700 1 -- 172.21.15.42:0/3800153606 >> v1:172.21.15.42:6789/0 conn(0x7f5fd807e710 legacy=0x7f5fd809f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:13:17.898 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:17.883+0000 7f5fdd51f700 1 -- 172.21.15.42:0/3800153606 >> v1:172.21.15.40:6789/0 conn(0x7f5fd809fad0 legacy=0x7f5fd814efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:13:17.898 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:17.883+0000 7f5fddd20700 1 -- 172.21.15.42:0/3800153606 >> v1:172.21.15.40:6789/0 conn(0x7f5fd809fad0 legacy=0x7f5fd814efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2019-12-05T05:13:17.899 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:17.883+0000 7f5fddd20700 1 -- 172.21.15.42:0/3800153606 >> v1:172.21.15.40:6789/0 conn(0x7f5fd809fad0 legacy=0x7f5fd814efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2019-12-05T05:13:17.899 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:17.883+0000 7f5fddd20700 1 --1- 172.21.15.42:0/3800153606 >> v1:172.21.15.40:6789/0 conn(0x7f5fd809fad0 0x7f5fd814efd0 :-1 s=OPENED pgs=405 cs=1 l=1).handle_message read tag failed 2019-12-05T05:13:17.899 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:17.883+0000 7f5fddd20700 1 --1- 172.21.15.42:0/3800153606 >> v1:172.21.15.40:6789/0 conn(0x7f5fd809fad0 0x7f5fd814efd0 :-1 s=OPENED pgs=405 cs=1 l=1).fault on lossy channel, failing 2019-12-05T05:13:17.899 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:17.883+0000 7f5fdd51f700 1 -- 172.21.15.42:0/3800153606 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fd807eb40 con 0x7f5fd80a7730 2019-12-05T05:13:17.899 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:17.883+0000 7f5fdd51f700 1 --2- 172.21.15.42:0/3800153606 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5fd80a7730 0x7f5fd808df70 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f5fd4002f30 tx=0x7f5fd400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:17.899 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:17.883+0000 7f5fdd51f700 1 --2- 172.21.15.42:0/3800153606 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5fd80a7730 0x7f5fd808df70 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f5fd4002f30 tx=0x7f5fd400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:17.900 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:17.883+0000 7f5fdd51f700 1 --2- 172.21.15.42:0/3800153606 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5fd80a7730 0x7f5fd808df70 secure :-1 s=READY pgs=106 cs=0 l=1 rx=0x7f5fd4002f30 tx=0x7f5fd400fbf0).ready entity=mon.2 client_cookie=4832a160c738580c server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:13:17.900 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:17.883+0000 7f5fdd51f700 1 --2- 172.21.15.42:0/3800153606 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5fd80a7730 0x7f5fd808df70 secure :-1 s=THROTTLE_DONE pgs=106 cs=0 l=1 rx=0x7f5fd4002f30 tx=0x7f5fd400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:17.901 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:17.883+0000 7f5fcbfff700 1 -- 172.21.15.42:0/3800153606 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5fd4008bb0 con 0x7f5fd80a7730 2019-12-05T05:13:17.901 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:17.883+0000 7f5fdd51f700 1 --2- 172.21.15.42:0/3800153606 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5fd80a7730 0x7f5fd808df70 secure :-1 s=THROTTLE_DONE pgs=106 cs=0 l=1 rx=0x7f5fd4002f30 tx=0x7f5fd400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:17.901 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:17.883+0000 7f5fcbfff700 1 -- 172.21.15.42:0/3800153606 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fd40087d0 con 0x7f5fd80a7730 2019-12-05T05:13:17.901 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:17.883+0000 7f5fdd51f700 1 --2- 172.21.15.42:0/3800153606 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5fd80a7730 0x7f5fd808df70 secure :-1 s=THROTTLE_DONE pgs=106 cs=0 l=1 rx=0x7f5fd4002f30 tx=0x7f5fd400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:17.901 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:17.883+0000 7f5fcbfff700 1 -- 172.21.15.42:0/3800153606 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5fd40118d0 con 0x7f5fd80a7730 2019-12-05T05:13:17.901 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:17.883+0000 7f5fdef82700 1 -- 172.21.15.42:0/3800153606 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5fd80a7730 msgr2=0x7f5fd808df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:13:17.901 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:17.883+0000 7f5fdef82700 1 --2- 172.21.15.42:0/3800153606 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5fd80a7730 0x7f5fd808df70 secure :-1 s=READY pgs=106 cs=0 l=1 rx=0x7f5fd4002f30 tx=0x7f5fd400fbf0).stop 2019-12-05T05:13:17.902 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:17.883+0000 7f5fdef82700 1 -- 172.21.15.42:0/3800153606 shutdown_connections 2019-12-05T05:13:17.902 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:17.883+0000 7f5fdef82700 1 --2- 172.21.15.42:0/3800153606 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5fd80a7730 0x7f5fd808df70 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:13:17.902 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:17.883+0000 7f5fdef82700 1 -- 172.21.15.42:0/3800153606 shutdown_connections 2019-12-05T05:13:17.902 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:17.883+0000 7f5fdef82700 1 -- 172.21.15.42:0/3800153606 wait complete. 2019-12-05T05:13:17.902 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:17.883+0000 7f5fdef82700 1 -- 172.21.15.42:0/3800153606 >> 172.21.15.42:0/3800153606 conn(0x7f5fd80aabd0 msgr2=0x7f5fd80a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:13:17.902 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:17.887+0000 7f5fdef82700 1 Processor -- start 2019-12-05T05:13:17.903 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:17.887+0000 7f5fdef82700 1 -- start start 2019-12-05T05:13:17.903 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:17.887+0000 7f5fdef82700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5fd8122130 0x7f5fd8122a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:13:17.903 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:17.887+0000 7f5fdef82700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5fd8119e00 0x7f5fd8125e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:13:17.903 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:17.887+0000 7f5fdef82700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5fd80a20f0 0x7f5fd80a2520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:13:17.903 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:17.887+0000 7f5fdef82700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5fd80a2a70 con 0x7f5fd80a20f0 2019-12-05T05:13:17.903 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:17.887+0000 7f5fdef82700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5fd8125920 con 0x7f5fd8122130 2019-12-05T05:13:17.904 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:17.887+0000 7f5fdef82700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fd8125bc0 con 0x7f5fd8119e00 2019-12-05T05:13:17.904 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:17.887+0000 7f5fdd51f700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5fd8122130 0x7f5fd8122a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:13:17.905 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:17.887+0000 7f5fdd51f700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5fd8122130 0x7f5fd8122a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:58298/0 (socket says 172.21.15.42:58298) 2019-12-05T05:13:17.905 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:17.887+0000 7f5fdd51f700 1 -- 172.21.15.42:0/1744210401 learned_addr learned my addr 172.21.15.42:0/1744210401 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:13:17.905 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:17.887+0000 7f5fddd20700 1 --2- 172.21.15.42:0/1744210401 >> v2:172.21.15.40:3300/0 conn(0x7f5fd80a20f0 0x7f5fd80a2520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:13:17.905 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:17.887+0000 7f5fdcd1e700 1 --2- 172.21.15.42:0/1744210401 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5fd8119e00 0x7f5fd8125e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:13:17.905 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:17.891+0000 7f5fdd51f700 1 -- 172.21.15.42:0/1744210401 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5fd8119e00 msgr2=0x7f5fd8125e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:13:17.905 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:17.891+0000 7f5fdd51f700 1 --2- 172.21.15.42:0/1744210401 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5fd8119e00 0x7f5fd8125e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:13:17.905 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:17.891+0000 7f5fdd51f700 1 -- 172.21.15.42:0/1744210401 >> v2:172.21.15.40:3300/0 conn(0x7f5fd80a20f0 msgr2=0x7f5fd80a2520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:13:17.906 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:17.891+0000 7f5fdd51f700 1 --2- 172.21.15.42:0/1744210401 >> v2:172.21.15.40:3300/0 conn(0x7f5fd80a20f0 0x7f5fd80a2520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:13:17.906 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:17.891+0000 7f5fdd51f700 1 -- 172.21.15.42:0/1744210401 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fd8122f70 con 0x7f5fd8122130 2019-12-05T05:13:17.906 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:17.891+0000 7f5fdd51f700 1 --2- 172.21.15.42:0/1744210401 >> v2:172.21.15.42:3300/0 conn(0x7f5fd8122130 0x7f5fd8122a20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f5fd8125ae0 tx=0x7f5fd4011990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:17.906 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:17.891+0000 7f5fdd51f700 1 --2- 172.21.15.42:0/1744210401 >> v2:172.21.15.42:3300/0 conn(0x7f5fd8122130 0x7f5fd8122a20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f5fd8125ae0 tx=0x7f5fd4011990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:17.906 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:17.891+0000 7f5fdd51f700 1 --2- 172.21.15.42:0/1744210401 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5fd8122130 0x7f5fd8122a20 secure :-1 s=READY pgs=516 cs=0 l=1 rx=0x7f5fd8125ae0 tx=0x7f5fd4011990).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:13:17.908 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:17.891+0000 7f5fdd51f700 1 --2- 172.21.15.42:0/1744210401 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5fd8122130 0x7f5fd8122a20 secure :-1 s=THROTTLE_DONE pgs=516 cs=0 l=1 rx=0x7f5fd8125ae0 tx=0x7f5fd4011990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:17.908 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:17.891+0000 7f5fdd51f700 1 --2- 172.21.15.42:0/1744210401 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5fd8122130 0x7f5fd8122a20 secure :-1 s=THROTTLE_DONE pgs=516 cs=0 l=1 rx=0x7f5fd8125ae0 tx=0x7f5fd4011990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:17.908 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:17.891+0000 7f5fc9ffb700 1 -- 172.21.15.42:0/1744210401 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5fd400f280 con 0x7f5fd8122130 2019-12-05T05:13:17.909 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:17.891+0000 7f5fdd51f700 1 --2- 172.21.15.42:0/1744210401 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5fd8122130 0x7f5fd8122a20 secure :-1 s=THROTTLE_DONE pgs=516 cs=0 l=1 rx=0x7f5fd8125ae0 tx=0x7f5fd4011990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:17.909 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:17.891+0000 7f5fc9ffb700 1 -- 172.21.15.42:0/1744210401 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fd4008db0 con 0x7f5fd8122130 2019-12-05T05:13:17.909 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:17.891+0000 7f5fdef82700 1 -- 172.21.15.42:0/1744210401 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fd81263b0 con 0x7f5fd8122130 2019-12-05T05:13:17.909 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:17.891+0000 7f5fc9ffb700 1 -- 172.21.15.42:0/1744210401 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5fd4005540 con 0x7f5fd8122130 2019-12-05T05:13:17.909 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:17.891+0000 7f5fdef82700 1 -- 172.21.15.42:0/1744210401 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fd8118630 con 0x7f5fd8122130 2019-12-05T05:13:17.909 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:17.891+0000 7f5fdd51f700 1 --2- 172.21.15.42:0/1744210401 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5fd8122130 0x7f5fd8122a20 secure :-1 s=THROTTLE_DONE pgs=516 cs=0 l=1 rx=0x7f5fd8125ae0 tx=0x7f5fd4011990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:17.910 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:17.891+0000 7f5fc9ffb700 1 -- 172.21.15.42:0/1744210401 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f5fd4005850 con 0x7f5fd8122130 2019-12-05T05:13:17.910 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:17.891+0000 7f5fdd51f700 1 --2- 172.21.15.42:0/1744210401 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5fd8122130 0x7f5fd8122a20 secure :-1 s=THROTTLE_DONE pgs=516 cs=0 l=1 rx=0x7f5fd8125ae0 tx=0x7f5fd4011990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:17.910 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:17.891+0000 7f5fc9ffb700 1 --2- 172.21.15.42:0/1744210401 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5fb803d010 0x7f5fb803f520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:13:17.910 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:17.891+0000 7f5fc9ffb700 1 -- 172.21.15.42:0/1744210401 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6633+0+0 (secure 0 0 0) 0x7f5fd4012060 con 0x7f5fd8122130 2019-12-05T05:13:17.910 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:17.891+0000 7f5fdcd1e700 1 --2- 172.21.15.42:0/1744210401 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5fb803d010 0x7f5fb803f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:13:17.910 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:17.895+0000 7f5fdcd1e700 1 --2- 172.21.15.42:0/1744210401 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5fb803d010 0x7f5fb803f520 crc :-1 s=READY pgs=234 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:13:17.910 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:17.895+0000 7f5fdef82700 1 -- 172.21.15.42:0/1744210401 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fd805d100 con 0x7f5fd8122130 2019-12-05T05:13:17.911 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:17.903+0000 7f5fdd51f700 1 --2- 172.21.15.42:0/1744210401 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5fd8122130 0x7f5fd8122a20 secure :-1 s=THROTTLE_DONE pgs=516 cs=0 l=1 rx=0x7f5fd8125ae0 tx=0x7f5fd4011990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:17.911 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:17.903+0000 7f5fc9ffb700 1 -- 172.21.15.42:0/1744210401 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f5fd40327a0 con 0x7f5fd8122130 2019-12-05T05:13:18.081 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:18.067+0000 7f5fdef82700 1 -- 172.21.15.42:0/1744210401 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5fd814ab60 con 0x7f5fd8122130 2019-12-05T05:13:18.081 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:18.071+0000 7f5fdd51f700 1 --2- 172.21.15.42:0/1744210401 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5fd8122130 0x7f5fd8122a20 secure :-1 s=THROTTLE_DONE pgs=516 cs=0 l=1 rx=0x7f5fd8125ae0 tx=0x7f5fd4011990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:18.081 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:18.071+0000 7f5fc9ffb700 1 -- 172.21.15.42:0/1744210401 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+1768 (secure 0 0 0) 0x7f5fd4032a00 con 0x7f5fd8122130 2019-12-05T05:13:18.081 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 67 2019-12-05T05:13:18.082 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:18.075+0000 7f5fdef82700 1 -- 172.21.15.42:0/1744210401 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5fb803d010 msgr2=0x7f5fb803f520 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:13:18.082 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:18.075+0000 7f5fdef82700 1 --2- 172.21.15.42:0/1744210401 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5fb803d010 0x7f5fb803f520 crc :-1 s=READY pgs=234 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:13:18.082 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:18.075+0000 7f5fdef82700 1 -- 172.21.15.42:0/1744210401 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5fd8122130 msgr2=0x7f5fd8122a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:13:18.082 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:18.075+0000 7f5fdef82700 1 --2- 172.21.15.42:0/1744210401 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5fd8122130 0x7f5fd8122a20 secure :-1 s=READY pgs=516 cs=0 l=1 rx=0x7f5fd8125ae0 tx=0x7f5fd4011990).stop 2019-12-05T05:13:18.082 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:18.075+0000 7f5fdef82700 1 -- 172.21.15.42:0/1744210401 shutdown_connections 2019-12-05T05:13:18.082 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:18.075+0000 7f5fdef82700 1 --2- 172.21.15.42:0/1744210401 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5fb803d010 0x7f5fb803f520 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:13:18.082 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:18.075+0000 7f5fdef82700 1 --2- 172.21.15.42:0/1744210401 >> v2:172.21.15.40:3300/0 conn(0x7f5fd80a20f0 0x7f5fd80a2520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:13:18.083 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:18.075+0000 7f5fdef82700 1 --2- 172.21.15.42:0/1744210401 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5fd8122130 0x7f5fd8122a20 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:13:18.083 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:18.075+0000 7f5fdef82700 1 --2- 172.21.15.42:0/1744210401 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5fd8119e00 0x7f5fd8125e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:13:18.083 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:18.075+0000 7f5fdef82700 1 -- 172.21.15.42:0/1744210401 shutdown_connections 2019-12-05T05:13:18.083 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:18.075+0000 7f5fdef82700 1 -- 172.21.15.42:0/1744210401 wait complete. 2019-12-05T05:13:18.083 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:18.075+0000 7f5fdef82700 1 -- 172.21.15.42:0/1744210401 >> 172.21.15.42:0/1744210401 conn(0x7f5fd807bd40 msgr2=0x7f5fd81161f0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:13:18.083 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:13:18.084 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":67,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:13:13.391909+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5505":{"gid":5505,"name":"a","rank":0,"incarnation":64,"state":"up:active","state_seq":9,"addr":"172.21.15.42:6835/211479572","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":211479572},{"type":"v1","addr":"172.21.15.42:6835","nonce":211479572}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:13:18.091 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2019-12-05T05:13:20.092 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:13:20.202 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:20.191+0000 7fd079f24700 1 Processor -- start 2019-12-05T05:13:20.202 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:20.191+0000 7fd079f24700 1 -- start start 2019-12-05T05:13:20.203 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:20.191+0000 7fd079f24700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd07408df80 0x7fd07409f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:13:20.203 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:20.191+0000 7fd079f24700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd07409fc60 con 0x7fd0740a7740 2019-12-05T05:13:20.203 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:20.191+0000 7fd079f24700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0740a0710 con 0x7fd07409f830 2019-12-05T05:13:20.203 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:20.191+0000 7fd079f24700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0740a0b00 con 0x7fd07408df80 2019-12-05T05:13:20.204 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:20.191+0000 7fd078cc2700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fd07409f830 0x7fd07414efe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:58348/0 (socket says 172.21.15.42:58348) 2019-12-05T05:13:20.204 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:20.191+0000 7fd078cc2700 1 -- 172.21.15.42:0/1308537987 learned_addr learned my addr 172.21.15.42:0/1308537987 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:13:20.204 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:20.191+0000 7fd0737fe700 1 --2- 172.21.15.42:0/1308537987 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd07408df80 0x7fd07409f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:13:20.204 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:20.191+0000 7fd072ffd700 1 -- 172.21.15.42:0/1308537987 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2993009111 0 0) 0x7fd0740a0710 con 0x7fd07409f830 2019-12-05T05:13:20.204 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:20.191+0000 7fd072ffd700 1 -- 172.21.15.42:0/1308537987 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd0540030b0 con 0x7fd07409f830 2019-12-05T05:13:20.205 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:20.191+0000 7fd072ffd700 1 -- 172.21.15.42:0/1308537987 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2282570703 0 0) 0x7fd07409fc60 con 0x7fd0740a7740 2019-12-05T05:13:20.205 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:20.191+0000 7fd072ffd700 1 -- 172.21.15.42:0/1308537987 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd0740a0710 con 0x7fd0740a7740 2019-12-05T05:13:20.205 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:20.191+0000 7fd0737fe700 1 -- 172.21.15.42:0/1308537987 >> v1:172.21.15.42:6789/0 conn(0x7fd07409f830 legacy=0x7fd07414efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:13:20.205 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:20.191+0000 7fd0737fe700 1 -- 172.21.15.42:0/1308537987 >> v1:172.21.15.40:6789/0 conn(0x7fd0740a7740 legacy=0x7fd07407e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:13:20.205 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:20.191+0000 7fd0737fe700 1 -- 172.21.15.42:0/1308537987 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd07407ef20 con 0x7fd07408df80 2019-12-05T05:13:20.206 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:20.191+0000 7fd0737fe700 1 --2- 172.21.15.42:0/1308537987 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd07408df80 0x7fd07409f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fd05c002f30 tx=0x7fd05c00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:20.206 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:20.191+0000 7fd0737fe700 1 --2- 172.21.15.42:0/1308537987 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd07408df80 0x7fd07409f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fd05c002f30 tx=0x7fd05c00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:20.206 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:20.191+0000 7fd0737fe700 1 --2- 172.21.15.42:0/1308537987 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd07408df80 0x7fd07409f2e0 secure :-1 s=READY pgs=107 cs=0 l=1 rx=0x7fd05c002f30 tx=0x7fd05c00fbf0).ready entity=mon.2 client_cookie=5ed508946cf9e04c server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:13:20.206 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:20.195+0000 7fd0737fe700 1 --2- 172.21.15.42:0/1308537987 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd07408df80 0x7fd07409f2e0 secure :-1 s=THROTTLE_DONE pgs=107 cs=0 l=1 rx=0x7fd05c002f30 tx=0x7fd05c00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:20.206 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:20.195+0000 7fd072ffd700 1 -- 172.21.15.42:0/1308537987 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd05c0112b0 con 0x7fd07408df80 2019-12-05T05:13:20.206 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:20.195+0000 7fd0737fe700 1 --2- 172.21.15.42:0/1308537987 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd07408df80 0x7fd07409f2e0 secure :-1 s=THROTTLE_DONE pgs=107 cs=0 l=1 rx=0x7fd05c002f30 tx=0x7fd05c00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:20.206 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:20.195+0000 7fd072ffd700 1 -- 172.21.15.42:0/1308537987 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd05c008bc0 con 0x7fd07408df80 2019-12-05T05:13:20.207 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:20.195+0000 7fd0737fe700 1 --2- 172.21.15.42:0/1308537987 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd07408df80 0x7fd07409f2e0 secure :-1 s=THROTTLE_DONE pgs=107 cs=0 l=1 rx=0x7fd05c002f30 tx=0x7fd05c00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:20.207 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:20.195+0000 7fd079f24700 1 -- 172.21.15.42:0/1308537987 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd07408df80 msgr2=0x7fd07409f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:13:20.207 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:20.195+0000 7fd072ffd700 1 -- 172.21.15.42:0/1308537987 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd05c011890 con 0x7fd07408df80 2019-12-05T05:13:20.207 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:20.195+0000 7fd079f24700 1 --2- 172.21.15.42:0/1308537987 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd07408df80 0x7fd07409f2e0 secure :-1 s=READY pgs=107 cs=0 l=1 rx=0x7fd05c002f30 tx=0x7fd05c00fbf0).stop 2019-12-05T05:13:20.207 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:20.195+0000 7fd079f24700 1 -- 172.21.15.42:0/1308537987 shutdown_connections 2019-12-05T05:13:20.207 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:20.195+0000 7fd079f24700 1 --2- 172.21.15.42:0/1308537987 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd07408df80 0x7fd07409f2e0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:13:20.208 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:20.195+0000 7fd079f24700 1 -- 172.21.15.42:0/1308537987 shutdown_connections 2019-12-05T05:13:20.208 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:20.195+0000 7fd079f24700 1 -- 172.21.15.42:0/1308537987 wait complete. 2019-12-05T05:13:20.208 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:20.195+0000 7fd079f24700 1 -- 172.21.15.42:0/1308537987 >> 172.21.15.42:0/1308537987 conn(0x7fd0740aabe0 msgr2=0x7fd0740a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:13:20.211 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:20.195+0000 7fd079f24700 1 Processor -- start 2019-12-05T05:13:20.211 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:20.199+0000 7fd079f24700 1 -- start start 2019-12-05T05:13:20.211 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:20.199+0000 7fd079f24700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd0741501c0 0x7fd0740a2930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:13:20.212 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:20.199+0000 7fd079f24700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd0740a2e80 0x7fd07407d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:13:20.212 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:20.199+0000 7fd079f24700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd07407e1d0 con 0x7fd07407dda0 2019-12-05T05:13:20.212 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:20.199+0000 7fd079f24700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0741225e0 con 0x7fd0740a2e80 2019-12-05T05:13:20.212 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:20.199+0000 7fd079f24700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0740a3440 con 0x7fd0741501c0 2019-12-05T05:13:20.212 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:20.199+0000 7fd078cc2700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fd07407dda0 0x7fd074122840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:58354/0 (socket says 172.21.15.42:58354) 2019-12-05T05:13:20.212 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:20.199+0000 7fd078cc2700 1 -- 172.21.15.42:0/4257038335 learned_addr learned my addr 172.21.15.42:0/4257038335 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:13:20.212 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:20.199+0000 7fd073fff700 1 --2- 172.21.15.42:0/4257038335 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd0741501c0 0x7fd0740a2930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:13:20.213 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:20.199+0000 7fd0737fe700 1 --2- 172.21.15.42:0/4257038335 >> v2:172.21.15.40:3300/0 conn(0x7fd0740a2e80 0x7fd07407d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:13:20.213 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:20.199+0000 7fd070ff9700 1 -- 172.21.15.42:0/4257038335 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 639432170 0 0) 0x7fd07407e1d0 con 0x7fd07407dda0 2019-12-05T05:13:20.213 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:20.199+0000 7fd070ff9700 1 -- 172.21.15.42:0/4257038335 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd04c0030b0 con 0x7fd07407dda0 2019-12-05T05:13:20.213 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:20.199+0000 7fd0737fe700 1 -- 172.21.15.42:0/4257038335 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd0741501c0 msgr2=0x7fd0740a2930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:13:20.213 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:20.199+0000 7fd0737fe700 1 --2- 172.21.15.42:0/4257038335 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd0741501c0 0x7fd0740a2930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:13:20.215 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:20.199+0000 7fd0737fe700 1 -- 172.21.15.42:0/4257038335 >> v1:172.21.15.42:6789/0 conn(0x7fd07407dda0 legacy=0x7fd074122840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:13:20.215 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:20.199+0000 7fd0737fe700 1 -- 172.21.15.42:0/4257038335 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0740a36e0 con 0x7fd0740a2e80 2019-12-05T05:13:20.216 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:20.199+0000 7fd0737fe700 1 --2- 172.21.15.42:0/4257038335 >> v2:172.21.15.40:3300/0 conn(0x7fd0740a2e80 0x7fd07407d850 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fd05c002f30 tx=0x7fd05c008bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:20.216 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:20.199+0000 7fd0737fe700 1 --2- 172.21.15.42:0/4257038335 >> v2:172.21.15.40:3300/0 conn(0x7fd0740a2e80 0x7fd07407d850 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fd05c002f30 tx=0x7fd05c008bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:20.216 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:20.199+0000 7fd0737fe700 1 --2- 172.21.15.42:0/4257038335 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd0740a2e80 0x7fd07407d850 secure :-1 s=READY pgs=407 cs=0 l=1 rx=0x7fd05c002f30 tx=0x7fd05c008bc0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:13:20.216 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:20.199+0000 7fd0737fe700 1 --2- 172.21.15.42:0/4257038335 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd0740a2e80 0x7fd07407d850 secure :-1 s=THROTTLE_DONE pgs=407 cs=0 l=1 rx=0x7fd05c002f30 tx=0x7fd05c008bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:20.216 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:20.199+0000 7fd070ff9700 1 -- 172.21.15.42:0/4257038335 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd05c011d80 con 0x7fd0740a2e80 2019-12-05T05:13:20.216 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:20.199+0000 7fd0737fe700 1 --2- 172.21.15.42:0/4257038335 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd0740a2e80 0x7fd07407d850 secure :-1 s=THROTTLE_DONE pgs=407 cs=0 l=1 rx=0x7fd05c002f30 tx=0x7fd05c008bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:20.217 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:20.199+0000 7fd0737fe700 1 --2- 172.21.15.42:0/4257038335 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd0740a2e80 0x7fd07407d850 secure :-1 s=THROTTLE_DONE pgs=407 cs=0 l=1 rx=0x7fd05c002f30 tx=0x7fd05c008bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:20.217 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:20.199+0000 7fd070ff9700 1 -- 172.21.15.42:0/4257038335 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd05c0114b0 con 0x7fd0740a2e80 2019-12-05T05:13:20.217 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:20.199+0000 7fd079f24700 1 -- 172.21.15.42:0/4257038335 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd074123040 con 0x7fd0740a2e80 2019-12-05T05:13:20.217 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:20.199+0000 7fd070ff9700 1 -- 172.21.15.42:0/4257038335 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd05c00f360 con 0x7fd0740a2e80 2019-12-05T05:13:20.217 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:20.199+0000 7fd079f24700 1 -- 172.21.15.42:0/4257038335 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd07412cee0 con 0x7fd0740a2e80 2019-12-05T05:13:20.217 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:20.199+0000 7fd0737fe700 1 --2- 172.21.15.42:0/4257038335 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd0740a2e80 0x7fd07407d850 secure :-1 s=THROTTLE_DONE pgs=407 cs=0 l=1 rx=0x7fd05c002f30 tx=0x7fd05c008bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:20.217 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:20.203+0000 7fd070ff9700 1 -- 172.21.15.42:0/4257038335 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fd05c005540 con 0x7fd0740a2e80 2019-12-05T05:13:20.218 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:20.203+0000 7fd0737fe700 1 --2- 172.21.15.42:0/4257038335 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd0740a2e80 0x7fd07407d850 secure :-1 s=THROTTLE_DONE pgs=407 cs=0 l=1 rx=0x7fd05c002f30 tx=0x7fd05c008bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:20.218 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:20.203+0000 7fd070ff9700 1 --2- 172.21.15.42:0/4257038335 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd04c03b270 0x7fd04c03d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:13:20.218 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:20.203+0000 7fd070ff9700 1 -- 172.21.15.42:0/4257038335 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6633+0+0 (secure 0 0 0) 0x7fd05c04b5e0 con 0x7fd0740a2e80 2019-12-05T05:13:20.218 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:20.203+0000 7fd073fff700 1 --2- 172.21.15.42:0/4257038335 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd04c03b270 0x7fd04c03d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:13:20.218 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:20.203+0000 7fd073fff700 1 --2- 172.21.15.42:0/4257038335 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd04c03b270 0x7fd04c03d780 crc :-1 s=READY pgs=236 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:13:20.219 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:20.207+0000 7fd079f24700 1 -- 172.21.15.42:0/4257038335 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd07405d110 con 0x7fd0740a2e80 2019-12-05T05:13:20.223 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:20.215+0000 7fd0737fe700 1 --2- 172.21.15.42:0/4257038335 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd0740a2e80 0x7fd07407d850 secure :-1 s=THROTTLE_DONE pgs=407 cs=0 l=1 rx=0x7fd05c002f30 tx=0x7fd05c008bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:20.223 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:20.215+0000 7fd070ff9700 1 -- 172.21.15.42:0/4257038335 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fd05c00f600 con 0x7fd0740a2e80 2019-12-05T05:13:20.397 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:20.387+0000 7fd079f24700 1 -- 172.21.15.42:0/4257038335 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd074123420 con 0x7fd0740a2e80 2019-12-05T05:13:20.397 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:20.391+0000 7fd0737fe700 1 --2- 172.21.15.42:0/4257038335 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd0740a2e80 0x7fd07407d850 secure :-1 s=THROTTLE_DONE pgs=407 cs=0 l=1 rx=0x7fd05c002f30 tx=0x7fd05c008bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:20.397 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:20.391+0000 7fd070ff9700 1 -- 172.21.15.42:0/4257038335 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) v1 ==== 94+0+2104 (secure 0 0 0) 0x7fd05c033450 con 0x7fd0740a2e80 2019-12-05T05:13:20.400 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 68 2019-12-05T05:13:20.401 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:13:20.401 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":68,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4841,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/3150831963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3150831963},{"type":"v1","addr":"172.21.15.40:6833","nonce":3150831963}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":68}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:13:13.391909+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5505":{"gid":5505,"name":"a","rank":0,"incarnation":64,"state":"up:active","state_seq":9,"addr":"172.21.15.42:6835/211479572","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":211479572},{"type":"v1","addr":"172.21.15.42:6835","nonce":211479572}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:13:20.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:20.395+0000 7fd079f24700 1 -- 172.21.15.42:0/4257038335 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd04c03b270 msgr2=0x7fd04c03d780 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:13:20.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:20.395+0000 7fd079f24700 1 --2- 172.21.15.42:0/4257038335 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd04c03b270 0x7fd04c03d780 crc :-1 s=READY pgs=236 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:13:20.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:20.395+0000 7fd079f24700 1 -- 172.21.15.42:0/4257038335 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd0740a2e80 msgr2=0x7fd07407d850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:13:20.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:20.395+0000 7fd079f24700 1 --2- 172.21.15.42:0/4257038335 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd0740a2e80 0x7fd07407d850 secure :-1 s=READY pgs=407 cs=0 l=1 rx=0x7fd05c002f30 tx=0x7fd05c008bc0).stop 2019-12-05T05:13:20.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:20.395+0000 7fd079f24700 1 -- 172.21.15.42:0/4257038335 shutdown_connections 2019-12-05T05:13:20.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:20.395+0000 7fd079f24700 1 --2- 172.21.15.42:0/4257038335 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd04c03b270 0x7fd04c03d780 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:13:20.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:20.395+0000 7fd079f24700 1 --2- 172.21.15.42:0/4257038335 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd0741501c0 0x7fd0740a2930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:13:20.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:20.395+0000 7fd079f24700 1 --2- 172.21.15.42:0/4257038335 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd0740a2e80 0x7fd07407d850 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:13:20.404 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:20.395+0000 7fd079f24700 1 -- 172.21.15.42:0/4257038335 shutdown_connections 2019-12-05T05:13:20.404 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:20.395+0000 7fd079f24700 1 -- 172.21.15.42:0/4257038335 wait complete. 2019-12-05T05:13:20.404 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:20.395+0000 7fd079f24700 1 -- 172.21.15.42:0/4257038335 >> 172.21.15.42:0/4257038335 conn(0x7fd0740a0050 msgr2=0x7fd0740a4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:13:20.415 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b reported in up:standby state 2019-12-05T05:13:20.415 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 39 secs before thrashing 2019-12-05T05:13:35.440 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:13:35.444 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:13:59.416 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:13:59.521 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.507+0000 7f5d1ecc2700 1 Processor -- start 2019-12-05T05:13:59.521 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.507+0000 7f5d1ecc2700 1 -- start start 2019-12-05T05:13:59.521 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.511+0000 7f5d1ecc2700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5d180a7740 0x7f5d1808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:13:59.521 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.511+0000 7f5d1ecc2700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5d1807e720 0x7f5d1807eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:13:59.522 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.511+0000 7f5d1ecc2700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d180a0710 con 0x7f5d1809f2e0 2019-12-05T05:13:59.522 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.511+0000 7f5d1ecc2700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d180a0b00 con 0x7f5d1807e720 2019-12-05T05:13:59.522 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.511+0000 7f5d1ecc2700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d1808e5b0 con 0x7f5d180a7740 2019-12-05T05:13:59.522 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.511+0000 7f5d1ca5e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5d1807e720 0x7f5d1807eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:13:59.522 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.511+0000 7f5d1ca5e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5d1807e720 0x7f5d1807eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:58332/0 (socket says 172.21.15.42:58332) 2019-12-05T05:13:59.522 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.511+0000 7f5d1ca5e700 1 -- 172.21.15.42:0/1528223308 learned_addr learned my addr 172.21.15.42:0/1528223308 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:13:59.522 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.511+0000 7f5d1d25f700 1 --2- 172.21.15.42:0/1528223308 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5d180a7740 0x7f5d1808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:13:59.523 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.511+0000 7f5d1ca5e700 1 -- 172.21.15.42:0/1528223308 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5d180a7740 msgr2=0x7f5d1808df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:13:59.523 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.511+0000 7f5d1ca5e700 1 --2- 172.21.15.42:0/1528223308 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5d180a7740 0x7f5d1808df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:13:59.523 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.511+0000 7f5d1ca5e700 1 -- 172.21.15.42:0/1528223308 >> v1:172.21.15.40:6789/0 conn(0x7f5d1809f2e0 legacy=0x7f5d1809f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:13:59.523 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.511+0000 7f5d0ffff700 1 -- 172.21.15.42:0/1528223308 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1678290091 0 0) 0x7f5d180a0710 con 0x7f5d1809f2e0 2019-12-05T05:13:59.523 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.511+0000 7f5d1ca5e700 1 -- 172.21.15.42:0/1528223308 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d1807f0a0 con 0x7f5d1807e720 2019-12-05T05:13:59.523 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.511+0000 7f5d1ca5e700 1 --2- 172.21.15.42:0/1528223308 >> v2:172.21.15.42:3300/0 conn(0x7f5d1807e720 0x7f5d1807eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f5d08008790 tx=0x7f5d0800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:59.534 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.511+0000 7f5d1ca5e700 1 --2- 172.21.15.42:0/1528223308 >> v2:172.21.15.42:3300/0 conn(0x7f5d1807e720 0x7f5d1807eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f5d08008790 tx=0x7f5d0800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:59.534 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.511+0000 7f5d1ca5e700 1 --2- 172.21.15.42:0/1528223308 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5d1807e720 0x7f5d1807eb50 secure :-1 s=READY pgs=519 cs=0 l=1 rx=0x7f5d08008790 tx=0x7f5d0800fd30).ready entity=mon.1 client_cookie=1608b544d348558 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:13:59.535 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.511+0000 7f5d1ca5e700 1 --2- 172.21.15.42:0/1528223308 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5d1807e720 0x7f5d1807eb50 secure :-1 s=THROTTLE_DONE pgs=519 cs=0 l=1 rx=0x7f5d08008790 tx=0x7f5d0800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:59.535 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.511+0000 7f5d0ffff700 1 -- 172.21.15.42:0/1528223308 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5d080113b0 con 0x7f5d1807e720 2019-12-05T05:13:59.535 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.511+0000 7f5d1ca5e700 1 --2- 172.21.15.42:0/1528223308 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5d1807e720 0x7f5d1807eb50 secure :-1 s=THROTTLE_DONE pgs=519 cs=0 l=1 rx=0x7f5d08008790 tx=0x7f5d0800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:59.535 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.511+0000 7f5d0ffff700 1 -- 172.21.15.42:0/1528223308 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d08008bb0 con 0x7f5d1807e720 2019-12-05T05:13:59.535 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.511+0000 7f5d1ca5e700 1 --2- 172.21.15.42:0/1528223308 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5d1807e720 0x7f5d1807eb50 secure :-1 s=THROTTLE_DONE pgs=519 cs=0 l=1 rx=0x7f5d08008790 tx=0x7f5d0800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:59.535 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.511+0000 7f5d0ffff700 1 -- 172.21.15.42:0/1528223308 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5d08011990 con 0x7f5d1807e720 2019-12-05T05:13:59.536 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.511+0000 7f5d1ecc2700 1 -- 172.21.15.42:0/1528223308 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5d1807e720 msgr2=0x7f5d1807eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:13:59.536 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.511+0000 7f5d1ecc2700 1 --2- 172.21.15.42:0/1528223308 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5d1807e720 0x7f5d1807eb50 secure :-1 s=READY pgs=519 cs=0 l=1 rx=0x7f5d08008790 tx=0x7f5d0800fd30).stop 2019-12-05T05:13:59.536 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.515+0000 7f5d1ecc2700 1 -- 172.21.15.42:0/1528223308 shutdown_connections 2019-12-05T05:13:59.536 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.515+0000 7f5d1ecc2700 1 --2- 172.21.15.42:0/1528223308 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5d180a7740 0x7f5d1808df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:13:59.536 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.515+0000 7f5d1ecc2700 1 --2- 172.21.15.42:0/1528223308 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5d1807e720 0x7f5d1807eb50 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:13:59.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.515+0000 7f5d1ecc2700 1 -- 172.21.15.42:0/1528223308 shutdown_connections 2019-12-05T05:13:59.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.515+0000 7f5d1ecc2700 1 -- 172.21.15.42:0/1528223308 wait complete. 2019-12-05T05:13:59.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.515+0000 7f5d1ecc2700 1 -- 172.21.15.42:0/1528223308 >> 172.21.15.42:0/1528223308 conn(0x7f5d180aabe0 msgr2=0x7f5d180a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:13:59.544 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.515+0000 7f5d1ecc2700 1 Processor -- start 2019-12-05T05:13:59.545 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.515+0000 7f5d1ecc2700 1 -- start start 2019-12-05T05:13:59.545 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.519+0000 7f5d1ecc2700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5d18114b80 0x7f5d181140a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:13:59.545 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.519+0000 7f5d1ecc2700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d1807e050 con 0x7f5d181145f0 2019-12-05T05:13:59.546 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.519+0000 7f5d1ecc2700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d18115710 con 0x7f5d18117cf0 2019-12-05T05:13:59.546 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.519+0000 7f5d1ecc2700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d18115940 con 0x7f5d18114b80 2019-12-05T05:13:59.546 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.519+0000 7f5d1d25f700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f5d18117cf0 0x7f5d18126d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:58374/0 (socket says 172.21.15.42:58374) 2019-12-05T05:13:59.546 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.519+0000 7f5d1d25f700 1 -- 172.21.15.42:0/1068161905 learned_addr learned my addr 172.21.15.42:0/1068161905 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:13:59.547 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.519+0000 7f5d1ca5e700 1 --2- 172.21.15.42:0/1068161905 >> v2:172.21.15.40:3300/0 conn(0x7f5d18114b80 0x7f5d181140a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:13:59.547 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.519+0000 7f5d0dffb700 1 -- 172.21.15.42:0/1068161905 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3788014533 0 0) 0x7f5d18115710 con 0x7f5d18117cf0 2019-12-05T05:13:59.547 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.519+0000 7f5d0dffb700 1 -- 172.21.15.42:0/1068161905 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5cf80030b0 con 0x7f5d18117cf0 2019-12-05T05:13:59.547 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.519+0000 7f5d1ca5e700 1 -- 172.21.15.42:0/1068161905 >> v1:172.21.15.42:6789/0 conn(0x7f5d18117cf0 legacy=0x7f5d18126d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:13:59.547 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.519+0000 7f5d1ca5e700 1 -- 172.21.15.42:0/1068161905 >> v1:172.21.15.40:6789/0 conn(0x7f5d181145f0 legacy=0x7f5d1807d850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:13:59.548 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.519+0000 7f5d1ca5e700 1 -- 172.21.15.42:0/1068161905 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d181311e0 con 0x7f5d18114b80 2019-12-05T05:13:59.548 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.519+0000 7f5d1ca5e700 1 --2- 172.21.15.42:0/1068161905 >> v2:172.21.15.40:3300/0 conn(0x7f5d18114b80 0x7f5d181140a0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f5d18115b80 tx=0x7f5d08011f20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:59.548 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.519+0000 7f5d1ca5e700 1 --2- 172.21.15.42:0/1068161905 >> v2:172.21.15.40:3300/0 conn(0x7f5d18114b80 0x7f5d181140a0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f5d18115b80 tx=0x7f5d08011f20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:59.549 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.519+0000 7f5d1ca5e700 1 --2- 172.21.15.42:0/1068161905 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5d18114b80 0x7f5d181140a0 secure :-1 s=READY pgs=411 cs=0 l=1 rx=0x7f5d18115b80 tx=0x7f5d08011f20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:13:59.549 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.519+0000 7f5d1ca5e700 1 --2- 172.21.15.42:0/1068161905 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5d18114b80 0x7f5d181140a0 secure :-1 s=THROTTLE_DONE pgs=411 cs=0 l=1 rx=0x7f5d18115b80 tx=0x7f5d08011f20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:59.549 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.519+0000 7f5d0dffb700 1 -- 172.21.15.42:0/1068161905 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5d08007340 con 0x7f5d18114b80 2019-12-05T05:13:59.549 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.519+0000 7f5d1ca5e700 1 --2- 172.21.15.42:0/1068161905 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5d18114b80 0x7f5d181140a0 secure :-1 s=THROTTLE_DONE pgs=411 cs=0 l=1 rx=0x7f5d18115b80 tx=0x7f5d08011f20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:59.549 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.519+0000 7f5d0dffb700 1 -- 172.21.15.42:0/1068161905 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d080115b0 con 0x7f5d18114b80 2019-12-05T05:13:59.550 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.519+0000 7f5d1ca5e700 1 --2- 172.21.15.42:0/1068161905 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5d18114b80 0x7f5d181140a0 secure :-1 s=THROTTLE_DONE pgs=411 cs=0 l=1 rx=0x7f5d18115b80 tx=0x7f5d08011f20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:59.550 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.519+0000 7f5d1ecc2700 1 -- 172.21.15.42:0/1068161905 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d1812f1c0 con 0x7f5d18114b80 2019-12-05T05:13:59.550 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.519+0000 7f5d0dffb700 1 -- 172.21.15.42:0/1068161905 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5d08005540 con 0x7f5d18114b80 2019-12-05T05:13:59.550 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.519+0000 7f5d1ecc2700 1 -- 172.21.15.42:0/1068161905 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d1812f770 con 0x7f5d18114b80 2019-12-05T05:13:59.550 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.519+0000 7f5d1ca5e700 1 --2- 172.21.15.42:0/1068161905 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5d18114b80 0x7f5d181140a0 secure :-1 s=THROTTLE_DONE pgs=411 cs=0 l=1 rx=0x7f5d18115b80 tx=0x7f5d08011f20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:59.550 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.523+0000 7f5d0dffb700 1 -- 172.21.15.42:0/1068161905 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f5d080057e0 con 0x7f5d18114b80 2019-12-05T05:13:59.551 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.523+0000 7f5d1ca5e700 1 --2- 172.21.15.42:0/1068161905 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5d18114b80 0x7f5d181140a0 secure :-1 s=THROTTLE_DONE pgs=411 cs=0 l=1 rx=0x7f5d18115b80 tx=0x7f5d08011f20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:59.551 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.523+0000 7f5d0dffb700 1 --2- 172.21.15.42:0/1068161905 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5cf803b1d0 0x7f5cf803d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:13:59.551 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.523+0000 7f5d0dffb700 1 -- 172.21.15.42:0/1068161905 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6633+0+0 (secure 0 0 0) 0x7f5d0804b430 con 0x7f5d18114b80 2019-12-05T05:13:59.551 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.523+0000 7f5d1d25f700 1 --2- 172.21.15.42:0/1068161905 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5cf803b1d0 0x7f5cf803d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:13:59.551 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.523+0000 7f5d1d25f700 1 --2- 172.21.15.42:0/1068161905 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5cf803b1d0 0x7f5cf803d6e0 crc :-1 s=READY pgs=238 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:13:59.552 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.523+0000 7f5d1ecc2700 1 -- 172.21.15.42:0/1068161905 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d1805d110 con 0x7f5d18114b80 2019-12-05T05:13:59.552 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.535+0000 7f5d1ca5e700 1 --2- 172.21.15.42:0/1068161905 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5d18114b80 0x7f5d181140a0 secure :-1 s=THROTTLE_DONE pgs=411 cs=0 l=1 rx=0x7f5d18115b80 tx=0x7f5d08011f20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:59.553 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.535+0000 7f5d0dffb700 1 -- 172.21.15.42:0/1068161905 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f5d08032490 con 0x7f5d18114b80 2019-12-05T05:13:59.709 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.703+0000 7f5d1ecc2700 1 -- 172.21.15.42:0/1068161905 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5d1805b8c0 con 0x7f5d18114b80 2019-12-05T05:13:59.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.703+0000 7f5d1ca5e700 1 --2- 172.21.15.42:0/1068161905 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5d18114b80 0x7f5d181140a0 secure :-1 s=THROTTLE_DONE pgs=411 cs=0 l=1 rx=0x7f5d18115b80 tx=0x7f5d08011f20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:59.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.703+0000 7f5d0dffb700 1 -- 172.21.15.42:0/1068161905 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) v1 ==== 94+0+2104 (secure 0 0 0) 0x7f5d080326f0 con 0x7f5d18114b80 2019-12-05T05:13:59.712 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:13:59.712 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":68,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4841,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/3150831963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3150831963},{"type":"v1","addr":"172.21.15.40:6833","nonce":3150831963}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":68}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:13:13.391909+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5505":{"gid":5505,"name":"a","rank":0,"incarnation":64,"state":"up:active","state_seq":9,"addr":"172.21.15.42:6835/211479572","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":211479572},{"type":"v1","addr":"172.21.15.42:6835","nonce":211479572}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:13:59.713 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 68 2019-12-05T05:13:59.715 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.707+0000 7f5d1ecc2700 1 -- 172.21.15.42:0/1068161905 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5cf803b1d0 msgr2=0x7f5cf803d6e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:13:59.715 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.707+0000 7f5d1ecc2700 1 --2- 172.21.15.42:0/1068161905 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5cf803b1d0 0x7f5cf803d6e0 crc :-1 s=READY pgs=238 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:13:59.715 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.707+0000 7f5d1ecc2700 1 -- 172.21.15.42:0/1068161905 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5d18114b80 msgr2=0x7f5d181140a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:13:59.715 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.707+0000 7f5d1ecc2700 1 --2- 172.21.15.42:0/1068161905 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5d18114b80 0x7f5d181140a0 secure :-1 s=READY pgs=411 cs=0 l=1 rx=0x7f5d18115b80 tx=0x7f5d08011f20).stop 2019-12-05T05:13:59.716 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.707+0000 7f5d1ecc2700 1 -- 172.21.15.42:0/1068161905 shutdown_connections 2019-12-05T05:13:59.716 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.707+0000 7f5d1ecc2700 1 --2- 172.21.15.42:0/1068161905 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5cf803b1d0 0x7f5cf803d6e0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:13:59.716 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.707+0000 7f5d1ecc2700 1 --2- 172.21.15.42:0/1068161905 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5d18114b80 0x7f5d181140a0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:13:59.716 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.707+0000 7f5d1ecc2700 1 -- 172.21.15.42:0/1068161905 shutdown_connections 2019-12-05T05:13:59.716 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.707+0000 7f5d1ecc2700 1 -- 172.21.15.42:0/1068161905 wait complete. 2019-12-05T05:13:59.716 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.707+0000 7f5d1ecc2700 1 -- 172.21.15.42:0/1068161905 >> 172.21.15.42:0/1068161905 conn(0x7f5d180a8240 msgr2=0x7f5d180a9790 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:13:59.725 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.a (rank=0) 2019-12-05T05:13:59.725 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2019-12-05T05:13:59.726 INFO:teuthology.orchestra.run:waiting for 300 2019-12-05T05:13:59.745 INFO:tasks.ceph.mds.a:Stopped 2019-12-05T05:13:59.746 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:13:59.916 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.903+0000 7f3075108700 1 Processor -- start 2019-12-05T05:13:59.916 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.903+0000 7f3075108700 1 -- start start 2019-12-05T05:13:59.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.907+0000 7f3075108700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f307008df80 0x7f307009f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:13:59.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.907+0000 7f3075108700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f307009f830 0x7f307014efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:13:59.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.907+0000 7f3075108700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f307009fc60 con 0x7f30700a7740 2019-12-05T05:13:59.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.907+0000 7f3075108700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30700a0b00 con 0x7f307008df80 2019-12-05T05:13:59.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.907+0000 7f3075108700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f307008e5b0 con 0x7f307009f830 2019-12-05T05:13:59.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.907+0000 7f306ed9d700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f307008df80 0x7f307009f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:13:59.918 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.907+0000 7f3074907700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f307009f830 0x7f307014efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:13:59.918 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.907+0000 7f3074907700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f307009f830 0x7f307014efe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:58814/0 (socket says 172.21.15.42:58814) 2019-12-05T05:13:59.918 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.907+0000 7f306f59e700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f30700a7740 0x7f307007e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:39590/0 (socket says 172.21.15.42:39590) 2019-12-05T05:13:59.918 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.907+0000 7f3074907700 1 -- 172.21.15.42:0/1553641039 learned_addr learned my addr 172.21.15.42:0/1553641039 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:13:59.918 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.907+0000 7f3074907700 1 -- 172.21.15.42:0/1553641039 >> v2:172.21.15.40:3300/0 conn(0x7f307008df80 msgr2=0x7f307009f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:13:59.918 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.907+0000 7f3074907700 1 --2- 172.21.15.42:0/1553641039 >> v2:172.21.15.40:3300/0 conn(0x7f307008df80 0x7f307009f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:13:59.918 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.907+0000 7f3074907700 1 -- 172.21.15.42:0/1553641039 >> v1:172.21.15.40:6789/0 conn(0x7f30700a7740 legacy=0x7f307007e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:13:59.919 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.907+0000 7f3074907700 1 -- 172.21.15.42:0/1553641039 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30700a0710 con 0x7f307009f830 2019-12-05T05:13:59.919 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.907+0000 7f3074907700 1 --2- 172.21.15.42:0/1553641039 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f307009f830 0x7f307014efe0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f3068008790 tx=0x7f306800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:59.919 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.907+0000 7f3074907700 1 --2- 172.21.15.42:0/1553641039 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f307009f830 0x7f307014efe0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f3068008790 tx=0x7f306800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:59.919 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.907+0000 7f3074907700 1 --2- 172.21.15.42:0/1553641039 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f307009f830 0x7f307014efe0 secure :-1 s=READY pgs=108 cs=0 l=1 rx=0x7f3068008790 tx=0x7f306800fd30).ready entity=mon.2 client_cookie=5c2c9fa5fa997d21 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:13:59.919 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.907+0000 7f3074907700 1 --2- 172.21.15.42:0/1553641039 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f307009f830 0x7f307014efe0 secure :-1 s=THROTTLE_DONE pgs=108 cs=0 l=1 rx=0x7f3068008790 tx=0x7f306800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:59.919 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.907+0000 7f3074907700 1 --2- 172.21.15.42:0/1553641039 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f307009f830 0x7f307014efe0 secure :-1 s=THROTTLE_DONE pgs=108 cs=0 l=1 rx=0x7f3068008790 tx=0x7f306800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:59.919 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.907+0000 7f306e59c700 1 -- 172.21.15.42:0/1553641039 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3068008c90 con 0x7f307009f830 2019-12-05T05:13:59.920 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.907+0000 7f3074907700 1 --2- 172.21.15.42:0/1553641039 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f307009f830 0x7f307014efe0 secure :-1 s=THROTTLE_DONE pgs=108 cs=0 l=1 rx=0x7f3068008790 tx=0x7f306800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:59.920 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.907+0000 7f306e59c700 1 -- 172.21.15.42:0/1553641039 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30680088b0 con 0x7f307009f830 2019-12-05T05:13:59.920 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.907+0000 7f306e59c700 1 -- 172.21.15.42:0/1553641039 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3068011b10 con 0x7f307009f830 2019-12-05T05:13:59.920 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.907+0000 7f3075108700 1 -- 172.21.15.42:0/1553641039 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f307009f830 msgr2=0x7f307014efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:13:59.920 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.907+0000 7f3075108700 1 --2- 172.21.15.42:0/1553641039 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f307009f830 0x7f307014efe0 secure :-1 s=READY pgs=108 cs=0 l=1 rx=0x7f3068008790 tx=0x7f306800fd30).stop 2019-12-05T05:13:59.920 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.911+0000 7f3075108700 1 -- 172.21.15.42:0/1553641039 shutdown_connections 2019-12-05T05:13:59.920 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.911+0000 7f3075108700 1 --2- 172.21.15.42:0/1553641039 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f307009f830 0x7f307014efe0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:13:59.921 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.911+0000 7f3075108700 1 --2- 172.21.15.42:0/1553641039 >> v2:172.21.15.40:3300/0 conn(0x7f307008df80 0x7f307009f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:13:59.921 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.911+0000 7f3075108700 1 -- 172.21.15.42:0/1553641039 shutdown_connections 2019-12-05T05:13:59.921 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.911+0000 7f3075108700 1 -- 172.21.15.42:0/1553641039 wait complete. 2019-12-05T05:13:59.921 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.911+0000 7f3075108700 1 -- 172.21.15.42:0/1553641039 >> 172.21.15.42:0/1553641039 conn(0x7f30700aabe0 msgr2=0x7f30700a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:13:59.924 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.911+0000 7f3075108700 1 Processor -- start 2019-12-05T05:13:59.924 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.911+0000 7f3075108700 1 -- start start 2019-12-05T05:13:59.925 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.915+0000 7f3075108700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f30700a6d90 0x7f30700a2780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:13:59.925 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.915+0000 7f3075108700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f307014e420 0x7f30700a1930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:13:59.925 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.915+0000 7f3075108700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3070142b60 con 0x7f307014e850 2019-12-05T05:13:59.925 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.915+0000 7f3075108700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30700a3100 con 0x7f30700a6d90 2019-12-05T05:13:59.925 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.915+0000 7f3075108700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30700a1690 con 0x7f307014e420 2019-12-05T05:13:59.925 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.915+0000 7f306ed9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f307014e420 0x7f30700a1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:13:59.926 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.915+0000 7f306ed9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f307014e420 0x7f30700a1930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:58352/0 (socket says 172.21.15.42:58352) 2019-12-05T05:13:59.926 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.915+0000 7f306ed9d700 1 -- 172.21.15.42:0/897138456 learned_addr learned my addr 172.21.15.42:0/897138456 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:13:59.926 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.915+0000 7f306f59e700 1 --2- 172.21.15.42:0/897138456 >> v2:172.21.15.40:3300/0 conn(0x7f30700a6d90 0x7f30700a2780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:13:59.926 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.915+0000 7f306ed9d700 1 -- 172.21.15.42:0/897138456 >> v2:172.21.15.40:3300/0 conn(0x7f30700a6d90 msgr2=0x7f30700a2780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:13:59.926 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.915+0000 7f306ed9d700 1 --2- 172.21.15.42:0/897138456 >> v2:172.21.15.40:3300/0 conn(0x7f30700a6d90 0x7f30700a2780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:13:59.926 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.915+0000 7f306ed9d700 1 -- 172.21.15.42:0/897138456 >> v1:172.21.15.40:6789/0 conn(0x7f307014e850 legacy=0x7f307008eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:13:59.926 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.915+0000 7f306ed9d700 1 -- 172.21.15.42:0/897138456 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30700a2cd0 con 0x7f307014e420 2019-12-05T05:13:59.927 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.915+0000 7f306ed9d700 1 --2- 172.21.15.42:0/897138456 >> v2:172.21.15.42:3300/0 conn(0x7f307014e420 0x7f30700a1930 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f30640033f0 tx=0x7f3064010fc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:59.927 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.915+0000 7f306ed9d700 1 --2- 172.21.15.42:0/897138456 >> v2:172.21.15.42:3300/0 conn(0x7f307014e420 0x7f30700a1930 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f30640033f0 tx=0x7f3064010fc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:59.927 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.915+0000 7f306ed9d700 1 --2- 172.21.15.42:0/897138456 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f307014e420 0x7f30700a1930 secure :-1 s=READY pgs=521 cs=0 l=1 rx=0x7f30640033f0 tx=0x7f3064010fc0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:13:59.927 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.915+0000 7f306ed9d700 1 --2- 172.21.15.42:0/897138456 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f307014e420 0x7f30700a1930 secure :-1 s=THROTTLE_DONE pgs=521 cs=0 l=1 rx=0x7f30640033f0 tx=0x7f3064010fc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:59.927 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.915+0000 7f306ed9d700 1 --2- 172.21.15.42:0/897138456 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f307014e420 0x7f30700a1930 secure :-1 s=THROTTLE_DONE pgs=521 cs=0 l=1 rx=0x7f30640033f0 tx=0x7f3064010fc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:59.927 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.915+0000 7f306ed9d700 1 --2- 172.21.15.42:0/897138456 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f307014e420 0x7f30700a1930 secure :-1 s=THROTTLE_DONE pgs=521 cs=0 l=1 rx=0x7f30640033f0 tx=0x7f3064010fc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:59.927 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.915+0000 7f3057fff700 1 -- 172.21.15.42:0/897138456 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3064007350 con 0x7f307014e420 2019-12-05T05:13:59.928 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.915+0000 7f3057fff700 1 -- 172.21.15.42:0/897138456 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3064005cb0 con 0x7f307014e420 2019-12-05T05:13:59.928 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.915+0000 7f3075108700 1 -- 172.21.15.42:0/897138456 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30700a1e80 con 0x7f307014e420 2019-12-05T05:13:59.928 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.915+0000 7f3057fff700 1 -- 172.21.15.42:0/897138456 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3064012830 con 0x7f307014e420 2019-12-05T05:13:59.928 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.915+0000 7f3075108700 1 -- 172.21.15.42:0/897138456 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f307008f370 con 0x7f307014e420 2019-12-05T05:13:59.928 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.915+0000 7f306ed9d700 1 --2- 172.21.15.42:0/897138456 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f307014e420 0x7f30700a1930 secure :-1 s=THROTTLE_DONE pgs=521 cs=0 l=1 rx=0x7f30640033f0 tx=0x7f3064010fc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:59.928 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.915+0000 7f3057fff700 1 -- 172.21.15.42:0/897138456 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f306400e180 con 0x7f307014e420 2019-12-05T05:13:59.928 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.915+0000 7f306ed9d700 1 --2- 172.21.15.42:0/897138456 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f307014e420 0x7f30700a1930 secure :-1 s=THROTTLE_DONE pgs=521 cs=0 l=1 rx=0x7f30640033f0 tx=0x7f3064010fc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:59.929 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.915+0000 7f3057fff700 1 --2- 172.21.15.42:0/897138456 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f305003ad20 0x7f305003d230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:13:59.929 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.919+0000 7f3057fff700 1 -- 172.21.15.42:0/897138456 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6633+0+0 (secure 0 0 0) 0x7f306404b170 con 0x7f307014e420 2019-12-05T05:13:59.929 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.919+0000 7f306f59e700 1 --2- 172.21.15.42:0/897138456 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f305003ad20 0x7f305003d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:13:59.929 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.919+0000 7f306f59e700 1 --2- 172.21.15.42:0/897138456 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f305003ad20 0x7f305003d230 crc :-1 s=READY pgs=239 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:13:59.930 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.919+0000 7f3075108700 1 -- 172.21.15.42:0/897138456 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f307005d110 con 0x7f307014e420 2019-12-05T05:13:59.931 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.923+0000 7f306ed9d700 1 --2- 172.21.15.42:0/897138456 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f307014e420 0x7f30700a1930 secure :-1 s=THROTTLE_DONE pgs=521 cs=0 l=1 rx=0x7f30640033f0 tx=0x7f3064010fc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:13:59.931 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:13:59.923+0000 7f3057fff700 1 -- 172.21.15.42:0/897138456 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f3064032450 con 0x7f307014e420 2019-12-05T05:14:00.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:00.091+0000 7f3075108700 1 -- 172.21.15.42:0/897138456 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f307005b8c0 con 0x7f307014e420 2019-12-05T05:14:00.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:00.091+0000 7f306ed9d700 1 --2- 172.21.15.42:0/897138456 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f307014e420 0x7f30700a1930 secure :-1 s=THROTTLE_DONE pgs=521 cs=0 l=1 rx=0x7f30640033f0 tx=0x7f3064010fc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:00.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:00.091+0000 7f3057fff700 1 -- 172.21.15.42:0/897138456 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) v1 ==== 94+0+2104 (secure 0 0 0) 0x7f3064033000 con 0x7f307014e420 2019-12-05T05:14:00.100 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 68 2019-12-05T05:14:00.100 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:14:00.100 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":68,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4841,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/3150831963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3150831963},{"type":"v1","addr":"172.21.15.40:6833","nonce":3150831963}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":68}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:13:13.391909+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5505":{"gid":5505,"name":"a","rank":0,"incarnation":64,"state":"up:active","state_seq":9,"addr":"172.21.15.42:6835/211479572","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":211479572},{"type":"v1","addr":"172.21.15.42:6835","nonce":211479572}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:14:00.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:00.095+0000 7f3075108700 1 -- 172.21.15.42:0/897138456 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f305003ad20 msgr2=0x7f305003d230 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:14:00.103 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:00.095+0000 7f3075108700 1 --2- 172.21.15.42:0/897138456 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f305003ad20 0x7f305003d230 crc :-1 s=READY pgs=239 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:14:00.103 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:00.095+0000 7f3075108700 1 -- 172.21.15.42:0/897138456 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f307014e420 msgr2=0x7f30700a1930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:14:00.103 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:00.095+0000 7f3075108700 1 --2- 172.21.15.42:0/897138456 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f307014e420 0x7f30700a1930 secure :-1 s=READY pgs=521 cs=0 l=1 rx=0x7f30640033f0 tx=0x7f3064010fc0).stop 2019-12-05T05:14:00.103 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:00.095+0000 7f3075108700 1 -- 172.21.15.42:0/897138456 shutdown_connections 2019-12-05T05:14:00.103 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:00.095+0000 7f3075108700 1 --2- 172.21.15.42:0/897138456 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f305003ad20 0x7f305003d230 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:14:00.104 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:00.095+0000 7f3075108700 1 --2- 172.21.15.42:0/897138456 >> v2:172.21.15.40:3300/0 conn(0x7f30700a6d90 0x7f30700a2780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:14:00.104 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:00.095+0000 7f3075108700 1 --2- 172.21.15.42:0/897138456 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f307014e420 0x7f30700a1930 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:14:00.104 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:00.095+0000 7f3075108700 1 -- 172.21.15.42:0/897138456 shutdown_connections 2019-12-05T05:14:00.104 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:00.095+0000 7f3075108700 1 -- 172.21.15.42:0/897138456 wait complete. 2019-12-05T05:14:00.104 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:00.095+0000 7f3075108700 1 -- 172.21.15.42:0/897138456 >> 172.21.15.42:0/897138456 conn(0x7f307008fde0 msgr2=0x7f307007cd60 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:14:00.113 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T05:14:02.114 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:14:02.224 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:02.211+0000 7fd6e8938700 1 Processor -- start 2019-12-05T05:14:02.224 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:02.211+0000 7fd6e8938700 1 -- start start 2019-12-05T05:14:02.224 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:02.211+0000 7fd6e8938700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd6e40a7740 0x7fd6e408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:14:02.224 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:02.211+0000 7fd6e8938700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd6e407e720 0x7fd6e407eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:14:02.224 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:02.211+0000 7fd6e8938700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6e40a0710 con 0x7fd6e409f2e0 2019-12-05T05:14:02.225 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:02.211+0000 7fd6e8938700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6e40a0b00 con 0x7fd6e40a7740 2019-12-05T05:14:02.225 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:02.211+0000 7fd6e8938700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6e408e5b0 con 0x7fd6e407e720 2019-12-05T05:14:02.225 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:02.215+0000 7fd6e259c700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd6e407e720 0x7fd6e407eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:14:02.225 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:02.215+0000 7fd6e359e700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fd6e409f2e0 0x7fd6e409f710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:39612/0 (socket says 172.21.15.42:39612) 2019-12-05T05:14:02.225 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:02.215+0000 7fd6e2d9d700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd6e40a7740 0x7fd6e408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:14:02.225 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:02.215+0000 7fd6e359e700 1 -- 172.21.15.42:0/947097727 learned_addr learned my addr 172.21.15.42:0/947097727 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:14:02.225 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:02.215+0000 7fd6e1d9b700 1 -- 172.21.15.42:0/947097727 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 971880624 0 0) 0x7fd6e40a0710 con 0x7fd6e409f2e0 2019-12-05T05:14:02.226 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:02.215+0000 7fd6e1d9b700 1 -- 172.21.15.42:0/947097727 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6d00030b0 con 0x7fd6e409f2e0 2019-12-05T05:14:02.226 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:02.215+0000 7fd6e2d9d700 1 -- 172.21.15.42:0/947097727 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd6e407e720 msgr2=0x7fd6e407eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:14:02.226 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:02.215+0000 7fd6e2d9d700 1 --2- 172.21.15.42:0/947097727 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd6e407e720 0x7fd6e407eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:14:02.226 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:02.215+0000 7fd6e2d9d700 1 -- 172.21.15.42:0/947097727 >> v1:172.21.15.40:6789/0 conn(0x7fd6e409f2e0 legacy=0x7fd6e409f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:14:02.226 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:02.215+0000 7fd6e2d9d700 1 -- 172.21.15.42:0/947097727 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6e407f0a0 con 0x7fd6e40a7740 2019-12-05T05:14:02.226 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:02.215+0000 7fd6e259c700 1 --2- 172.21.15.42:0/947097727 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd6e407e720 0x7fd6e407eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:14:02.227 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:02.215+0000 7fd6e2d9d700 1 --2- 172.21.15.42:0/947097727 >> v2:172.21.15.40:3300/0 conn(0x7fd6e40a7740 0x7fd6e408df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fd6cc008790 tx=0x7fd6cc00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:02.227 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:02.215+0000 7fd6e2d9d700 1 --2- 172.21.15.42:0/947097727 >> v2:172.21.15.40:3300/0 conn(0x7fd6e40a7740 0x7fd6e408df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fd6cc008790 tx=0x7fd6cc00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:02.227 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:02.215+0000 7fd6e2d9d700 1 --2- 172.21.15.42:0/947097727 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd6e40a7740 0x7fd6e408df80 secure :-1 s=READY pgs=415 cs=0 l=1 rx=0x7fd6cc008790 tx=0x7fd6cc00fc70).ready entity=mon.0 client_cookie=524287b8d2b04174 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:14:02.227 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:02.215+0000 7fd6e2d9d700 1 --2- 172.21.15.42:0/947097727 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd6e40a7740 0x7fd6e408df80 secure :-1 s=THROTTLE_DONE pgs=415 cs=0 l=1 rx=0x7fd6cc008790 tx=0x7fd6cc00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:02.227 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:02.215+0000 7fd6e1d9b700 1 -- 172.21.15.42:0/947097727 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd6cc0112f0 con 0x7fd6e40a7740 2019-12-05T05:14:02.227 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:02.215+0000 7fd6e2d9d700 1 --2- 172.21.15.42:0/947097727 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd6e40a7740 0x7fd6e408df80 secure :-1 s=THROTTLE_DONE pgs=415 cs=0 l=1 rx=0x7fd6cc008790 tx=0x7fd6cc00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:02.228 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:02.215+0000 7fd6e1d9b700 1 -- 172.21.15.42:0/947097727 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6cc008ad0 con 0x7fd6e40a7740 2019-12-05T05:14:02.228 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:02.215+0000 7fd6e2d9d700 1 --2- 172.21.15.42:0/947097727 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd6e40a7740 0x7fd6e408df80 secure :-1 s=THROTTLE_DONE pgs=415 cs=0 l=1 rx=0x7fd6cc008790 tx=0x7fd6cc00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:02.228 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:02.215+0000 7fd6e1d9b700 1 -- 172.21.15.42:0/947097727 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd6cc0118d0 con 0x7fd6e40a7740 2019-12-05T05:14:02.228 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:02.215+0000 7fd6e8938700 1 -- 172.21.15.42:0/947097727 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd6e40a7740 msgr2=0x7fd6e408df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:14:02.228 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:02.215+0000 7fd6e8938700 1 --2- 172.21.15.42:0/947097727 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd6e40a7740 0x7fd6e408df80 secure :-1 s=READY pgs=415 cs=0 l=1 rx=0x7fd6cc008790 tx=0x7fd6cc00fc70).stop 2019-12-05T05:14:02.228 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:02.215+0000 7fd6e8938700 1 -- 172.21.15.42:0/947097727 shutdown_connections 2019-12-05T05:14:02.229 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:02.215+0000 7fd6e8938700 1 --2- 172.21.15.42:0/947097727 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd6e40a7740 0x7fd6e408df80 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:14:02.229 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:02.215+0000 7fd6e8938700 1 --2- 172.21.15.42:0/947097727 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd6e407e720 0x7fd6e407eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:14:02.229 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:02.215+0000 7fd6e8938700 1 -- 172.21.15.42:0/947097727 shutdown_connections 2019-12-05T05:14:02.229 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:02.215+0000 7fd6e8938700 1 -- 172.21.15.42:0/947097727 wait complete. 2019-12-05T05:14:02.229 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:02.215+0000 7fd6e8938700 1 -- 172.21.15.42:0/947097727 >> 172.21.15.42:0/947097727 conn(0x7fd6e40aabe0 msgr2=0x7fd6e40a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:14:02.235 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:02.219+0000 7fd6e8938700 1 Processor -- start 2019-12-05T05:14:02.235 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:02.219+0000 7fd6e8938700 1 -- start start 2019-12-05T05:14:02.235 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:02.219+0000 7fd6e8938700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd6e407d710 0x7fd6e407db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:14:02.236 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:02.219+0000 7fd6e8938700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd6e411afc0 0x7fd6e411b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:14:02.236 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:02.219+0000 7fd6e8938700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6e411b940 con 0x7fd6e4119420 2019-12-05T05:14:02.236 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:02.219+0000 7fd6e8938700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6e4123cf0 con 0x7fd6e407d710 2019-12-05T05:14:02.236 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:02.219+0000 7fd6e8938700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6e411ad20 con 0x7fd6e411afc0 2019-12-05T05:14:02.236 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:02.219+0000 7fd6e359e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd6e411afc0 0x7fd6e411b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:14:02.236 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:02.219+0000 7fd6e359e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd6e411afc0 0x7fd6e411b3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:58374/0 (socket says 172.21.15.42:58374) 2019-12-05T05:14:02.236 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:02.219+0000 7fd6e359e700 1 -- 172.21.15.42:0/1297005825 learned_addr learned my addr 172.21.15.42:0/1297005825 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:14:02.237 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:02.219+0000 7fd6e259c700 1 --2- 172.21.15.42:0/1297005825 >> v2:172.21.15.40:3300/0 conn(0x7fd6e407d710 0x7fd6e407db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:14:02.237 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:02.219+0000 7fd6e359e700 1 -- 172.21.15.42:0/1297005825 >> v2:172.21.15.40:3300/0 conn(0x7fd6e407d710 msgr2=0x7fd6e407db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:14:02.237 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:02.219+0000 7fd6e359e700 1 --2- 172.21.15.42:0/1297005825 >> v2:172.21.15.40:3300/0 conn(0x7fd6e407d710 0x7fd6e407db40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:14:02.237 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:02.219+0000 7fd6e359e700 1 -- 172.21.15.42:0/1297005825 >> v1:172.21.15.40:6789/0 conn(0x7fd6e4119420 legacy=0x7fd6e4123450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:14:02.237 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:02.219+0000 7fd6e359e700 1 -- 172.21.15.42:0/1297005825 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6e4142e70 con 0x7fd6e411afc0 2019-12-05T05:14:02.237 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:02.219+0000 7fd6e259c700 1 --2- 172.21.15.42:0/1297005825 >> v2:172.21.15.40:3300/0 conn(0x7fd6e407d710 0x7fd6e407db40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:14:02.237 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:02.219+0000 7fd6e359e700 1 --2- 172.21.15.42:0/1297005825 >> v2:172.21.15.42:3300/0 conn(0x7fd6e411afc0 0x7fd6e411b3f0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fd6dc004720 tx=0x7fd6dc010b90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:02.238 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:02.219+0000 7fd6e359e700 1 --2- 172.21.15.42:0/1297005825 >> v2:172.21.15.42:3300/0 conn(0x7fd6e411afc0 0x7fd6e411b3f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fd6dc004720 tx=0x7fd6dc010b90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:02.238 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:02.219+0000 7fd6e359e700 1 --2- 172.21.15.42:0/1297005825 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd6e411afc0 0x7fd6e411b3f0 secure :-1 s=READY pgs=522 cs=0 l=1 rx=0x7fd6dc004720 tx=0x7fd6dc010b90).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:14:02.238 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:02.219+0000 7fd6e359e700 1 --2- 172.21.15.42:0/1297005825 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd6e411afc0 0x7fd6e411b3f0 secure :-1 s=THROTTLE_DONE pgs=522 cs=0 l=1 rx=0x7fd6dc004720 tx=0x7fd6dc010b90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:02.238 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:02.219+0000 7fd6db7fe700 1 -- 172.21.15.42:0/1297005825 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd6dc005bc0 con 0x7fd6e411afc0 2019-12-05T05:14:02.238 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:02.223+0000 7fd6e359e700 1 --2- 172.21.15.42:0/1297005825 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd6e411afc0 0x7fd6e411b3f0 secure :-1 s=THROTTLE_DONE pgs=522 cs=0 l=1 rx=0x7fd6dc004720 tx=0x7fd6dc010b90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:02.238 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:02.223+0000 7fd6db7fe700 1 -- 172.21.15.42:0/1297005825 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6dc0057e0 con 0x7fd6e411afc0 2019-12-05T05:14:02.238 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:02.223+0000 7fd6e359e700 1 --2- 172.21.15.42:0/1297005825 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd6e411afc0 0x7fd6e411b3f0 secure :-1 s=THROTTLE_DONE pgs=522 cs=0 l=1 rx=0x7fd6dc004720 tx=0x7fd6dc010b90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:02.239 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:02.223+0000 7fd6e8938700 1 -- 172.21.15.42:0/1297005825 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6e4137600 con 0x7fd6e411afc0 2019-12-05T05:14:02.239 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:02.223+0000 7fd6db7fe700 1 -- 172.21.15.42:0/1297005825 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd6dc0125b0 con 0x7fd6e411afc0 2019-12-05T05:14:02.239 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:02.223+0000 7fd6e8938700 1 -- 172.21.15.42:0/1297005825 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6e4137a10 con 0x7fd6e411afc0 2019-12-05T05:14:02.239 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:02.223+0000 7fd6e359e700 1 --2- 172.21.15.42:0/1297005825 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd6e411afc0 0x7fd6e411b3f0 secure :-1 s=THROTTLE_DONE pgs=522 cs=0 l=1 rx=0x7fd6dc004720 tx=0x7fd6dc010b90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:02.239 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:02.223+0000 7fd6db7fe700 1 -- 172.21.15.42:0/1297005825 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fd6dc012840 con 0x7fd6e411afc0 2019-12-05T05:14:02.240 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:02.223+0000 7fd6e359e700 1 --2- 172.21.15.42:0/1297005825 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd6e411afc0 0x7fd6e411b3f0 secure :-1 s=THROTTLE_DONE pgs=522 cs=0 l=1 rx=0x7fd6dc004720 tx=0x7fd6dc010b90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:02.240 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:02.223+0000 7fd6db7fe700 1 --2- 172.21.15.42:0/1297005825 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd6c803ae20 0x7fd6c803d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:14:02.240 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:02.223+0000 7fd6db7fe700 1 -- 172.21.15.42:0/1297005825 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6633+0+0 (secure 0 0 0) 0x7fd6dc04b0d0 con 0x7fd6e411afc0 2019-12-05T05:14:02.240 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:02.223+0000 7fd6e2d9d700 1 --2- 172.21.15.42:0/1297005825 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd6c803ae20 0x7fd6c803d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:14:02.240 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:02.223+0000 7fd6e2d9d700 1 --2- 172.21.15.42:0/1297005825 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd6c803ae20 0x7fd6c803d330 crc :-1 s=READY pgs=240 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:14:02.240 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:02.227+0000 7fd6e8938700 1 -- 172.21.15.42:0/1297005825 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6e405d110 con 0x7fd6e411afc0 2019-12-05T05:14:02.243 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:02.235+0000 7fd6e359e700 1 --2- 172.21.15.42:0/1297005825 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd6e411afc0 0x7fd6e411b3f0 secure :-1 s=THROTTLE_DONE pgs=522 cs=0 l=1 rx=0x7fd6dc004720 tx=0x7fd6dc010b90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:02.243 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:02.235+0000 7fd6db7fe700 1 -- 172.21.15.42:0/1297005825 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fd6dc013060 con 0x7fd6e411afc0 2019-12-05T05:14:02.413 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:02.407+0000 7fd6e8938700 1 -- 172.21.15.42:0/1297005825 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd6e405b8c0 con 0x7fd6e411afc0 2019-12-05T05:14:02.414 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:02.407+0000 7fd6e359e700 1 --2- 172.21.15.42:0/1297005825 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd6e411afc0 0x7fd6e411b3f0 secure :-1 s=THROTTLE_DONE pgs=522 cs=0 l=1 rx=0x7fd6dc004720 tx=0x7fd6dc010b90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:02.414 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:02.407+0000 7fd6db7fe700 1 -- 172.21.15.42:0/1297005825 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) v1 ==== 94+0+2104 (secure 0 0 0) 0x7fd6dc032130 con 0x7fd6e411afc0 2019-12-05T05:14:02.414 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 68 2019-12-05T05:14:02.415 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:14:02.415 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":68,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4841,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/3150831963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3150831963},{"type":"v1","addr":"172.21.15.40:6833","nonce":3150831963}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":68}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:13:13.391909+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5505":{"gid":5505,"name":"a","rank":0,"incarnation":64,"state":"up:active","state_seq":9,"addr":"172.21.15.42:6835/211479572","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":211479572},{"type":"v1","addr":"172.21.15.42:6835","nonce":211479572}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:14:02.416 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:02.411+0000 7fd6e8938700 1 -- 172.21.15.42:0/1297005825 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd6c803ae20 msgr2=0x7fd6c803d330 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:14:02.417 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:02.411+0000 7fd6e8938700 1 --2- 172.21.15.42:0/1297005825 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd6c803ae20 0x7fd6c803d330 crc :-1 s=READY pgs=240 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:14:02.417 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:02.411+0000 7fd6e8938700 1 -- 172.21.15.42:0/1297005825 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd6e411afc0 msgr2=0x7fd6e411b3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:14:02.417 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:02.411+0000 7fd6e8938700 1 --2- 172.21.15.42:0/1297005825 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd6e411afc0 0x7fd6e411b3f0 secure :-1 s=READY pgs=522 cs=0 l=1 rx=0x7fd6dc004720 tx=0x7fd6dc010b90).stop 2019-12-05T05:14:02.417 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:02.411+0000 7fd6e8938700 1 -- 172.21.15.42:0/1297005825 shutdown_connections 2019-12-05T05:14:02.417 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:02.411+0000 7fd6e8938700 1 --2- 172.21.15.42:0/1297005825 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd6c803ae20 0x7fd6c803d330 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:14:02.417 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:02.411+0000 7fd6e8938700 1 --2- 172.21.15.42:0/1297005825 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd6e411afc0 0x7fd6e411b3f0 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:14:02.418 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:02.411+0000 7fd6e8938700 1 --2- 172.21.15.42:0/1297005825 >> v2:172.21.15.40:3300/0 conn(0x7fd6e407d710 0x7fd6e407db40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:14:02.418 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:02.411+0000 7fd6e8938700 1 -- 172.21.15.42:0/1297005825 shutdown_connections 2019-12-05T05:14:02.418 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:02.411+0000 7fd6e8938700 1 -- 172.21.15.42:0/1297005825 wait complete. 2019-12-05T05:14:02.418 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:02.411+0000 7fd6e8938700 1 -- 172.21.15.42:0/1297005825 >> 172.21.15.42:0/1297005825 conn(0x7fd6e407ea00 msgr2=0x7fd6e411c6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:14:02.426 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T05:14:04.427 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:14:04.544 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:04.531+0000 7fd50491c700 1 Processor -- start 2019-12-05T05:14:04.544 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:04.531+0000 7fd50491c700 1 -- start start 2019-12-05T05:14:04.545 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:04.535+0000 7fd50491c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd50008df80 0x7fd50009f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:14:04.545 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:04.535+0000 7fd50491c700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd50009f830 0x7fd50014efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:14:04.545 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:04.535+0000 7fd50491c700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd50009fc60 con 0x7fd5000a7740 2019-12-05T05:14:04.545 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:04.535+0000 7fd50491c700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5000a0b00 con 0x7fd50008df80 2019-12-05T05:14:04.545 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:04.535+0000 7fd50491c700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd50008e5b0 con 0x7fd50009f830 2019-12-05T05:14:04.545 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:04.535+0000 7fd4fe59c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd50008df80 0x7fd50009f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:14:04.545 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:04.535+0000 7fd4fed9d700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fd5000a7740 0x7fd50007e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:58422/0 (socket says 172.21.15.42:58422) 2019-12-05T05:14:04.546 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:04.535+0000 7fd4fed9d700 1 -- 172.21.15.42:0/1599816465 learned_addr learned my addr 172.21.15.42:0/1599816465 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:14:04.546 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:04.535+0000 7fd4ff59e700 1 --2- 172.21.15.42:0/1599816465 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd50009f830 0x7fd50014efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:14:04.546 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:04.535+0000 7fd4fe59c700 1 -- 172.21.15.42:0/1599816465 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd50009f830 msgr2=0x7fd50014efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:14:04.546 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:04.535+0000 7fd4fe59c700 1 --2- 172.21.15.42:0/1599816465 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd50009f830 0x7fd50014efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:14:04.546 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:04.535+0000 7fd4fe59c700 1 -- 172.21.15.42:0/1599816465 >> v1:172.21.15.42:6789/0 conn(0x7fd5000a7740 legacy=0x7fd50007e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:14:04.546 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:04.535+0000 7fd4fdd9b700 1 -- 172.21.15.42:0/1599816465 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2887892460 0 0) 0x7fd50009fc60 con 0x7fd5000a7740 2019-12-05T05:14:04.546 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:04.535+0000 7fd4fe59c700 1 -- 172.21.15.42:0/1599816465 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5000a0710 con 0x7fd50008df80 2019-12-05T05:14:04.547 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:04.535+0000 7fd4fe59c700 1 --2- 172.21.15.42:0/1599816465 >> v2:172.21.15.42:3300/0 conn(0x7fd50008df80 0x7fd50009f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fd4f4008790 tx=0x7fd4f400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:04.547 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:04.535+0000 7fd4fe59c700 1 --2- 172.21.15.42:0/1599816465 >> v2:172.21.15.42:3300/0 conn(0x7fd50008df80 0x7fd50009f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fd4f4008790 tx=0x7fd4f400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:04.547 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:04.535+0000 7fd4fe59c700 1 --2- 172.21.15.42:0/1599816465 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd50008df80 0x7fd50009f2e0 secure :-1 s=READY pgs=524 cs=0 l=1 rx=0x7fd4f4008790 tx=0x7fd4f400fc70).ready entity=mon.1 client_cookie=ce14ad3ea6a47371 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:14:04.547 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:04.535+0000 7fd4fe59c700 1 --2- 172.21.15.42:0/1599816465 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd50008df80 0x7fd50009f2e0 secure :-1 s=THROTTLE_DONE pgs=524 cs=0 l=1 rx=0x7fd4f4008790 tx=0x7fd4f400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:04.547 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:04.535+0000 7fd4fdd9b700 1 -- 172.21.15.42:0/1599816465 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd4f4008bb0 con 0x7fd50008df80 2019-12-05T05:14:04.547 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:04.535+0000 7fd4fe59c700 1 --2- 172.21.15.42:0/1599816465 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd50008df80 0x7fd50009f2e0 secure :-1 s=THROTTLE_DONE pgs=524 cs=0 l=1 rx=0x7fd4f4008790 tx=0x7fd4f400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:04.547 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:04.535+0000 7fd4fe59c700 1 --2- 172.21.15.42:0/1599816465 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd50008df80 0x7fd50009f2e0 secure :-1 s=THROTTLE_DONE pgs=524 cs=0 l=1 rx=0x7fd4f4008790 tx=0x7fd4f400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:04.548 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:04.535+0000 7fd4fdd9b700 1 -- 172.21.15.42:0/1599816465 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4f40087d0 con 0x7fd50008df80 2019-12-05T05:14:04.548 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:04.535+0000 7fd4fdd9b700 1 -- 172.21.15.42:0/1599816465 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd4f4011860 con 0x7fd50008df80 2019-12-05T05:14:04.548 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:04.535+0000 7fd50491c700 1 -- 172.21.15.42:0/1599816465 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd50008df80 msgr2=0x7fd50009f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:14:04.548 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:04.535+0000 7fd50491c700 1 --2- 172.21.15.42:0/1599816465 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd50008df80 0x7fd50009f2e0 secure :-1 s=READY pgs=524 cs=0 l=1 rx=0x7fd4f4008790 tx=0x7fd4f400fc70).stop 2019-12-05T05:14:04.548 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:04.535+0000 7fd50491c700 1 -- 172.21.15.42:0/1599816465 shutdown_connections 2019-12-05T05:14:04.558 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:04.535+0000 7fd50491c700 1 --2- 172.21.15.42:0/1599816465 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd50009f830 0x7fd50014efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:14:04.558 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:04.535+0000 7fd50491c700 1 --2- 172.21.15.42:0/1599816465 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd50008df80 0x7fd50009f2e0 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:14:04.558 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:04.535+0000 7fd50491c700 1 -- 172.21.15.42:0/1599816465 shutdown_connections 2019-12-05T05:14:04.558 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:04.535+0000 7fd50491c700 1 -- 172.21.15.42:0/1599816465 wait complete. 2019-12-05T05:14:04.558 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:04.535+0000 7fd50491c700 1 -- 172.21.15.42:0/1599816465 >> 172.21.15.42:0/1599816465 conn(0x7fd5000aabe0 msgr2=0x7fd5000a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:14:04.562 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:04.539+0000 7fd50491c700 1 Processor -- start 2019-12-05T05:14:04.562 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:04.539+0000 7fd50491c700 1 -- start start 2019-12-05T05:14:04.576 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:04.539+0000 7fd50491c700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd500137030 0x7fd500139470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:14:04.576 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:04.539+0000 7fd50491c700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5000a7130 con 0x7fd50007d430 2019-12-05T05:14:04.576 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:04.539+0000 7fd50491c700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5000a4260 con 0x7fd5000a6d00 2019-12-05T05:14:04.576 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:04.539+0000 7fd50491c700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5000a4a30 con 0x7fd500137030 2019-12-05T05:14:04.577 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:04.539+0000 7fd4fed9d700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fd5000a6d00 0x7fd50012ce20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:58428/0 (socket says 172.21.15.42:58428) 2019-12-05T05:14:04.577 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:04.539+0000 7fd4fed9d700 1 -- 172.21.15.42:0/1492891225 learned_addr learned my addr 172.21.15.42:0/1492891225 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:14:04.577 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:04.539+0000 7fd4ff59e700 1 --2- 172.21.15.42:0/1492891225 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd500137030 0x7fd500139470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:14:04.577 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:04.539+0000 7fd4ef7fe700 1 -- 172.21.15.42:0/1492891225 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1336792737 0 0) 0x7fd5000a7130 con 0x7fd50007d430 2019-12-05T05:14:04.577 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:04.539+0000 7fd4ef7fe700 1 -- 172.21.15.42:0/1492891225 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4e40030b0 con 0x7fd50007d430 2019-12-05T05:14:04.578 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:04.539+0000 7fd4ef7fe700 1 -- 172.21.15.42:0/1492891225 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1537898425 0 0) 0x7fd5000a4260 con 0x7fd5000a6d00 2019-12-05T05:14:04.578 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:04.539+0000 7fd4ef7fe700 1 -- 172.21.15.42:0/1492891225 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5000a7130 con 0x7fd5000a6d00 2019-12-05T05:14:04.578 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:04.539+0000 7fd4ff59e700 1 -- 172.21.15.42:0/1492891225 >> v1:172.21.15.42:6789/0 conn(0x7fd5000a6d00 legacy=0x7fd50012ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:14:04.579 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:04.539+0000 7fd4ff59e700 1 -- 172.21.15.42:0/1492891225 >> v1:172.21.15.40:6789/0 conn(0x7fd50007d430 legacy=0x7fd500135820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:14:04.580 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:04.539+0000 7fd4ff59e700 1 -- 172.21.15.42:0/1492891225 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd50007d860 con 0x7fd500137030 2019-12-05T05:14:04.580 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:04.543+0000 7fd4ff59e700 1 --2- 172.21.15.42:0/1492891225 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd500137030 0x7fd500139470 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fd4f80032b0 tx=0x7fd4f8010c90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:04.580 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:04.543+0000 7fd4ff59e700 1 --2- 172.21.15.42:0/1492891225 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd500137030 0x7fd500139470 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fd4f80032b0 tx=0x7fd4f8010c90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:04.580 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:04.543+0000 7fd4ff59e700 1 --2- 172.21.15.42:0/1492891225 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd500137030 0x7fd500139470 secure :-1 s=READY pgs=109 cs=0 l=1 rx=0x7fd4f80032b0 tx=0x7fd4f8010c90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:14:04.580 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:04.543+0000 7fd4ff59e700 1 --2- 172.21.15.42:0/1492891225 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd500137030 0x7fd500139470 secure :-1 s=THROTTLE_DONE pgs=109 cs=0 l=1 rx=0x7fd4f80032b0 tx=0x7fd4f8010c90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:04.580 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:04.543+0000 7fd4ef7fe700 1 -- 172.21.15.42:0/1492891225 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd4f8007350 con 0x7fd500137030 2019-12-05T05:14:04.581 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:04.543+0000 7fd4ff59e700 1 --2- 172.21.15.42:0/1492891225 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd500137030 0x7fd500139470 secure :-1 s=THROTTLE_DONE pgs=109 cs=0 l=1 rx=0x7fd4f80032b0 tx=0x7fd4f8010c90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:04.581 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:04.543+0000 7fd4ff59e700 1 --2- 172.21.15.42:0/1492891225 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd500137030 0x7fd500139470 secure :-1 s=THROTTLE_DONE pgs=109 cs=0 l=1 rx=0x7fd4f80032b0 tx=0x7fd4f8010c90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:04.581 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:04.543+0000 7fd4ef7fe700 1 -- 172.21.15.42:0/1492891225 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4f8005cb0 con 0x7fd500137030 2019-12-05T05:14:04.581 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:04.543+0000 7fd50491c700 1 -- 172.21.15.42:0/1492891225 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd50012d620 con 0x7fd500137030 2019-12-05T05:14:04.581 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:04.543+0000 7fd4ef7fe700 1 -- 172.21.15.42:0/1492891225 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd4f8012350 con 0x7fd500137030 2019-12-05T05:14:04.581 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:04.543+0000 7fd50491c700 1 -- 172.21.15.42:0/1492891225 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5000a51a0 con 0x7fd500137030 2019-12-05T05:14:04.581 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:04.543+0000 7fd4ff59e700 1 --2- 172.21.15.42:0/1492891225 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd500137030 0x7fd500139470 secure :-1 s=THROTTLE_DONE pgs=109 cs=0 l=1 rx=0x7fd4f80032b0 tx=0x7fd4f8010c90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:04.582 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:04.543+0000 7fd4ef7fe700 1 -- 172.21.15.42:0/1492891225 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fd4f8012620 con 0x7fd500137030 2019-12-05T05:14:04.582 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:04.543+0000 7fd4ff59e700 1 --2- 172.21.15.42:0/1492891225 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd500137030 0x7fd500139470 secure :-1 s=THROTTLE_DONE pgs=109 cs=0 l=1 rx=0x7fd4f80032b0 tx=0x7fd4f8010c90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:04.582 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:04.543+0000 7fd4ef7fe700 1 --2- 172.21.15.42:0/1492891225 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd4e403b520 0x7fd4e403da30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:14:04.582 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:04.543+0000 7fd4ef7fe700 1 -- 172.21.15.42:0/1492891225 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6633+0+0 (secure 0 0 0) 0x7fd4f804b640 con 0x7fd500137030 2019-12-05T05:14:04.582 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:04.543+0000 7fd4fed9d700 1 --2- 172.21.15.42:0/1492891225 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd4e403b520 0x7fd4e403da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:14:04.582 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:04.543+0000 7fd4fed9d700 1 -- 172.21.15.42:0/1492891225 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd4e403b520 msgr2=0x7fd4e403da30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2019-12-05T05:14:04.582 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:04.543+0000 7fd4fed9d700 1 -- 172.21.15.42:0/1492891225 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd4e403b520 msgr2=0x7fd4e403da30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2019-12-05T05:14:04.583 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:04.543+0000 7fd4fed9d700 1 --2- 172.21.15.42:0/1492891225 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd4e403b520 0x7fd4e403da30 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-12-05T05:14:04.583 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:04.543+0000 7fd4fed9d700 1 --2- 172.21.15.42:0/1492891225 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd4e403b520 0x7fd4e403da30 crc :-1 s=START_CONNECT pgs=0 cs=0 l=1 rx=0 tx=0)._fault waiting 0.200000 2019-12-05T05:14:04.583 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:04.547+0000 7fd50491c700 1 -- 172.21.15.42:0/1492891225 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd50007dad0 con 0x7fd500137030 2019-12-05T05:14:04.589 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:04.559+0000 7fd4ff59e700 1 --2- 172.21.15.42:0/1492891225 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd500137030 0x7fd500139470 secure :-1 s=THROTTLE_DONE pgs=109 cs=0 l=1 rx=0x7fd4f80032b0 tx=0x7fd4f8010c90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:04.589 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:04.559+0000 7fd4ef7fe700 1 -- 172.21.15.42:0/1492891225 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fd4f80326c0 con 0x7fd500137030 2019-12-05T05:14:04.731 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:04.723+0000 7fd50491c700 1 -- 172.21.15.42:0/1492891225 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd50005d110 con 0x7fd500137030 2019-12-05T05:14:04.732 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:04.727+0000 7fd4ff59e700 1 --2- 172.21.15.42:0/1492891225 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd500137030 0x7fd500139470 secure :-1 s=THROTTLE_DONE pgs=109 cs=0 l=1 rx=0x7fd4f80032b0 tx=0x7fd4f8010c90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:04.732 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:04.727+0000 7fd4ef7fe700 1 -- 172.21.15.42:0/1492891225 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) v1 ==== 94+0+2104 (secure 0 0 0) 0x7fd4f8032920 con 0x7fd500137030 2019-12-05T05:14:04.734 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 68 2019-12-05T05:14:04.734 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:14:04.734 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":68,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4841,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/3150831963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3150831963},{"type":"v1","addr":"172.21.15.40:6833","nonce":3150831963}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":68}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:13:13.391909+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5505":{"gid":5505,"name":"a","rank":0,"incarnation":64,"state":"up:active","state_seq":9,"addr":"172.21.15.42:6835/211479572","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":211479572},{"type":"v1","addr":"172.21.15.42:6835","nonce":211479572}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:14:04.737 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:04.731+0000 7fd50491c700 1 -- 172.21.15.42:0/1492891225 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd4e403b520 msgr2=0x7fd4e403da30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2019-12-05T05:14:04.737 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:04.731+0000 7fd50491c700 1 --2- 172.21.15.42:0/1492891225 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd4e403b520 0x7fd4e403da30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:14:04.737 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:04.731+0000 7fd50491c700 1 -- 172.21.15.42:0/1492891225 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd500137030 msgr2=0x7fd500139470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:14:04.737 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:04.731+0000 7fd50491c700 1 --2- 172.21.15.42:0/1492891225 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd500137030 0x7fd500139470 secure :-1 s=READY pgs=109 cs=0 l=1 rx=0x7fd4f80032b0 tx=0x7fd4f8010c90).stop 2019-12-05T05:14:04.738 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:04.731+0000 7fd50491c700 1 -- 172.21.15.42:0/1492891225 shutdown_connections 2019-12-05T05:14:04.738 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:04.731+0000 7fd50491c700 1 --2- 172.21.15.42:0/1492891225 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd4e403b520 0x7fd4e403da30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:14:04.738 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:04.731+0000 7fd50491c700 1 --2- 172.21.15.42:0/1492891225 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd500137030 0x7fd500139470 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:14:04.738 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:04.731+0000 7fd50491c700 1 -- 172.21.15.42:0/1492891225 shutdown_connections 2019-12-05T05:14:04.738 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:04.731+0000 7fd50491c700 1 -- 172.21.15.42:0/1492891225 wait complete. 2019-12-05T05:14:04.738 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:04.731+0000 7fd50491c700 1 -- 172.21.15.42:0/1492891225 >> 172.21.15.42:0/1492891225 conn(0x7fd50008fde0 msgr2=0x7fd500059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:14:04.748 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T05:14:05.500 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:14:05.504 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:14:06.751 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:14:06.860 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:06.847+0000 7fcc8b4a7700 1 Processor -- start 2019-12-05T05:14:06.860 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:06.847+0000 7fcc8b4a7700 1 -- start start 2019-12-05T05:14:06.861 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:06.847+0000 7fcc8b4a7700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fcc840a7740 0x7fcc8408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:14:06.861 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:06.847+0000 7fcc8b4a7700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fcc8409fae0 0x7fcc8407eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:14:06.861 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:06.847+0000 7fcc8b4a7700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc840a0710 con 0x7fcc8407e720 2019-12-05T05:14:06.861 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:06.847+0000 7fcc8b4a7700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc840a0b00 con 0x7fcc8409fae0 2019-12-05T05:14:06.861 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:06.847+0000 7fcc8b4a7700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc8408e5b0 con 0x7fcc840a7740 2019-12-05T05:14:06.861 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:06.847+0000 7fcc89a44700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fcc840a7740 0x7fcc8408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:14:06.861 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:06.847+0000 7fcc89a44700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fcc840a7740 0x7fcc8408df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:58402/0 (socket says 172.21.15.42:58402) 2019-12-05T05:14:06.862 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:06.847+0000 7fcc89a44700 1 -- 172.21.15.42:0/2904088467 learned_addr learned my addr 172.21.15.42:0/2904088467 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:14:06.862 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:06.847+0000 7fcc8a245700 1 --2- 172.21.15.42:0/2904088467 >> v2:172.21.15.40:3300/0 conn(0x7fcc8409fae0 0x7fcc8407eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:14:06.862 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:06.851+0000 7fcc88a42700 1 -- 172.21.15.42:0/2904088467 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 796738107 0 0) 0x7fcc840a0710 con 0x7fcc8407e720 2019-12-05T05:14:06.862 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:06.851+0000 7fcc88a42700 1 -- 172.21.15.42:0/2904088467 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc6c0030b0 con 0x7fcc8407e720 2019-12-05T05:14:06.862 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:06.851+0000 7fcc89a44700 1 -- 172.21.15.42:0/2904088467 >> v2:172.21.15.40:3300/0 conn(0x7fcc8409fae0 msgr2=0x7fcc8407eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:14:06.862 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:06.851+0000 7fcc89a44700 1 --2- 172.21.15.42:0/2904088467 >> v2:172.21.15.40:3300/0 conn(0x7fcc8409fae0 0x7fcc8407eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:14:06.862 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:06.851+0000 7fcc89a44700 1 -- 172.21.15.42:0/2904088467 >> v1:172.21.15.40:6789/0 conn(0x7fcc8407e720 legacy=0x7fcc8409f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:14:06.863 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:06.851+0000 7fcc89a44700 1 -- 172.21.15.42:0/2904088467 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc8407f0a0 con 0x7fcc840a7740 2019-12-05T05:14:06.863 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:06.851+0000 7fcc89a44700 1 --2- 172.21.15.42:0/2904088467 >> v2:172.21.15.42:3300/0 conn(0x7fcc840a7740 0x7fcc8408df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fcc800032b0 tx=0x7fcc800100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:06.863 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:06.851+0000 7fcc89a44700 1 --2- 172.21.15.42:0/2904088467 >> v2:172.21.15.42:3300/0 conn(0x7fcc840a7740 0x7fcc8408df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fcc800032b0 tx=0x7fcc800100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:06.863 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:06.851+0000 7fcc89a44700 1 --2- 172.21.15.42:0/2904088467 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcc840a7740 0x7fcc8408df80 secure :-1 s=READY pgs=526 cs=0 l=1 rx=0x7fcc800032b0 tx=0x7fcc800100b0).ready entity=mon.1 client_cookie=c0d4f0a1b4b38978 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:14:06.863 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:06.851+0000 7fcc89a44700 1 --2- 172.21.15.42:0/2904088467 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcc840a7740 0x7fcc8408df80 secure :-1 s=THROTTLE_DONE pgs=526 cs=0 l=1 rx=0x7fcc800032b0 tx=0x7fcc800100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:06.863 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:06.851+0000 7fcc88a42700 1 -- 172.21.15.42:0/2904088467 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcc80010d60 con 0x7fcc840a7740 2019-12-05T05:14:06.863 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:06.851+0000 7fcc89a44700 1 --2- 172.21.15.42:0/2904088467 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcc840a7740 0x7fcc8408df80 secure :-1 s=THROTTLE_DONE pgs=526 cs=0 l=1 rx=0x7fcc800032b0 tx=0x7fcc800100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:06.864 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:06.851+0000 7fcc89a44700 1 --2- 172.21.15.42:0/2904088467 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcc840a7740 0x7fcc8408df80 secure :-1 s=THROTTLE_DONE pgs=526 cs=0 l=1 rx=0x7fcc800032b0 tx=0x7fcc800100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:06.864 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:06.851+0000 7fcc88a42700 1 -- 172.21.15.42:0/2904088467 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc80010980 con 0x7fcc840a7740 2019-12-05T05:14:06.864 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:06.851+0000 7fcc88a42700 1 -- 172.21.15.42:0/2904088467 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcc800114a0 con 0x7fcc840a7740 2019-12-05T05:14:06.864 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:06.851+0000 7fcc8b4a7700 1 -- 172.21.15.42:0/2904088467 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcc840a7740 msgr2=0x7fcc8408df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:14:06.864 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:06.851+0000 7fcc8b4a7700 1 --2- 172.21.15.42:0/2904088467 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcc840a7740 0x7fcc8408df80 secure :-1 s=READY pgs=526 cs=0 l=1 rx=0x7fcc800032b0 tx=0x7fcc800100b0).stop 2019-12-05T05:14:06.864 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:06.851+0000 7fcc8b4a7700 1 -- 172.21.15.42:0/2904088467 shutdown_connections 2019-12-05T05:14:06.864 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:06.851+0000 7fcc8b4a7700 1 --2- 172.21.15.42:0/2904088467 >> v2:172.21.15.40:3300/0 conn(0x7fcc8409fae0 0x7fcc8407eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:14:06.865 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:06.851+0000 7fcc8b4a7700 1 --2- 172.21.15.42:0/2904088467 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcc840a7740 0x7fcc8408df80 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:14:06.865 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:06.851+0000 7fcc8b4a7700 1 -- 172.21.15.42:0/2904088467 shutdown_connections 2019-12-05T05:14:06.865 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:06.851+0000 7fcc8b4a7700 1 -- 172.21.15.42:0/2904088467 wait complete. 2019-12-05T05:14:06.865 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:06.851+0000 7fcc8b4a7700 1 -- 172.21.15.42:0/2904088467 >> 172.21.15.42:0/2904088467 conn(0x7fcc840aabe0 msgr2=0x7fcc840a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:14:06.865 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:06.855+0000 7fcc8b4a7700 1 Processor -- start 2019-12-05T05:14:06.865 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:06.855+0000 7fcc8b4a7700 1 -- start start 2019-12-05T05:14:06.868 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:06.855+0000 7fcc8b4a7700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fcc84123500 0x7fcc84122240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:14:06.869 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:06.855+0000 7fcc8b4a7700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcc840722c0 0x7fcc84074700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:14:06.869 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:06.855+0000 7fcc8b4a7700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc841250d0 con 0x7fcc840a5e10 2019-12-05T05:14:06.869 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:06.855+0000 7fcc8b4a7700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc84123a80 con 0x7fcc84123500 2019-12-05T05:14:06.869 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:06.855+0000 7fcc8b4a7700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc84123d20 con 0x7fcc840722c0 2019-12-05T05:14:06.869 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:06.855+0000 7fcc89243700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fcc84123500 0x7fcc84122240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:14:06.869 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:06.855+0000 7fcc89243700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fcc84123500 0x7fcc84122240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:58408/0 (socket says 172.21.15.42:58408) 2019-12-05T05:14:06.870 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:06.855+0000 7fcc89243700 1 -- 172.21.15.42:0/599587047 learned_addr learned my addr 172.21.15.42:0/599587047 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:14:06.870 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:06.855+0000 7fcc8a245700 1 --2- 172.21.15.42:0/599587047 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcc840722c0 0x7fcc84074700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:14:06.870 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:06.855+0000 7fcc7a7fc700 1 -- 172.21.15.42:0/599587047 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 237425761 0 0) 0x7fcc841250d0 con 0x7fcc840a5e10 2019-12-05T05:14:06.870 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:06.855+0000 7fcc7a7fc700 1 -- 172.21.15.42:0/599587047 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc640030b0 con 0x7fcc840a5e10 2019-12-05T05:14:06.870 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:06.855+0000 7fcc89243700 1 -- 172.21.15.42:0/599587047 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcc840722c0 msgr2=0x7fcc84074700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:14:06.870 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:06.855+0000 7fcc89243700 1 --2- 172.21.15.42:0/599587047 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcc840722c0 0x7fcc84074700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:14:06.870 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:06.855+0000 7fcc89243700 1 -- 172.21.15.42:0/599587047 >> v1:172.21.15.40:6789/0 conn(0x7fcc840a5e10 legacy=0x7fcc840a4bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:14:06.871 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:06.855+0000 7fcc89243700 1 -- 172.21.15.42:0/599587047 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc840a53d0 con 0x7fcc84123500 2019-12-05T05:14:06.871 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:06.855+0000 7fcc89243700 1 --2- 172.21.15.42:0/599587047 >> v2:172.21.15.42:3300/0 conn(0x7fcc84123500 0x7fcc84122240 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fcc74000e90 tx=0x7fcc740109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:06.871 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:06.855+0000 7fcc89243700 1 --2- 172.21.15.42:0/599587047 >> v2:172.21.15.42:3300/0 conn(0x7fcc84123500 0x7fcc84122240 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fcc74000e90 tx=0x7fcc740109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:06.871 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:06.859+0000 7fcc89243700 1 --2- 172.21.15.42:0/599587047 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcc84123500 0x7fcc84122240 secure :-1 s=READY pgs=527 cs=0 l=1 rx=0x7fcc74000e90 tx=0x7fcc740109a0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:14:06.871 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:06.859+0000 7fcc89243700 1 --2- 172.21.15.42:0/599587047 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcc84123500 0x7fcc84122240 secure :-1 s=THROTTLE_DONE pgs=527 cs=0 l=1 rx=0x7fcc74000e90 tx=0x7fcc740109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:06.871 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:06.859+0000 7fcc7a7fc700 1 -- 172.21.15.42:0/599587047 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcc7400caa0 con 0x7fcc84123500 2019-12-05T05:14:06.871 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:06.859+0000 7fcc89243700 1 --2- 172.21.15.42:0/599587047 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcc84123500 0x7fcc84122240 secure :-1 s=THROTTLE_DONE pgs=527 cs=0 l=1 rx=0x7fcc74000e90 tx=0x7fcc740109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:06.872 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:06.859+0000 7fcc89243700 1 --2- 172.21.15.42:0/599587047 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcc84123500 0x7fcc84122240 secure :-1 s=THROTTLE_DONE pgs=527 cs=0 l=1 rx=0x7fcc74000e90 tx=0x7fcc740109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:06.872 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:06.859+0000 7fcc7a7fc700 1 -- 172.21.15.42:0/599587047 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc7400c6c0 con 0x7fcc84123500 2019-12-05T05:14:06.872 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:06.859+0000 7fcc8b4a7700 1 -- 172.21.15.42:0/599587047 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc84074c50 con 0x7fcc84123500 2019-12-05T05:14:06.872 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:06.859+0000 7fcc7a7fc700 1 -- 172.21.15.42:0/599587047 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcc7400d6a0 con 0x7fcc84123500 2019-12-05T05:14:06.872 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:06.859+0000 7fcc8b4a7700 1 -- 172.21.15.42:0/599587047 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc84075200 con 0x7fcc84123500 2019-12-05T05:14:06.872 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:06.859+0000 7fcc89243700 1 --2- 172.21.15.42:0/599587047 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcc84123500 0x7fcc84122240 secure :-1 s=THROTTLE_DONE pgs=527 cs=0 l=1 rx=0x7fcc74000e90 tx=0x7fcc740109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:06.872 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:06.859+0000 7fcc7a7fc700 1 -- 172.21.15.42:0/599587047 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fcc7400d970 con 0x7fcc84123500 2019-12-05T05:14:06.873 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:06.859+0000 7fcc89243700 1 --2- 172.21.15.42:0/599587047 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcc84123500 0x7fcc84122240 secure :-1 s=THROTTLE_DONE pgs=527 cs=0 l=1 rx=0x7fcc74000e90 tx=0x7fcc740109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:06.873 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:06.859+0000 7fcc7a7fc700 1 --2- 172.21.15.42:0/599587047 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fcc6403b300 0x7fcc6403d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:14:06.873 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:06.859+0000 7fcc7a7fc700 1 -- 172.21.15.42:0/599587047 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6633+0+0 (secure 0 0 0) 0x7fcc7404b2a0 con 0x7fcc84123500 2019-12-05T05:14:06.873 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:06.859+0000 7fcc89a44700 1 --2- 172.21.15.42:0/599587047 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fcc6403b300 0x7fcc6403d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:14:06.873 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:06.859+0000 7fcc89a44700 1 --2- 172.21.15.42:0/599587047 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fcc6403b300 0x7fcc6403d810 crc :-1 s=READY pgs=242 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:14:06.874 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:06.863+0000 7fcc8b4a7700 1 -- 172.21.15.42:0/599587047 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc84123ee0 con 0x7fcc84123500 2019-12-05T05:14:06.878 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:06.871+0000 7fcc89243700 1 --2- 172.21.15.42:0/599587047 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcc84123500 0x7fcc84122240 secure :-1 s=THROTTLE_DONE pgs=527 cs=0 l=1 rx=0x7fcc74000e90 tx=0x7fcc740109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:06.878 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:06.871+0000 7fcc7a7fc700 1 -- 172.21.15.42:0/599587047 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fcc740322e0 con 0x7fcc84123500 2019-12-05T05:14:07.067 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:07.043+0000 7fcc8b4a7700 1 -- 172.21.15.42:0/599587047 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcc8405d110 con 0x7fcc84123500 2019-12-05T05:14:07.067 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:07.043+0000 7fcc89243700 1 --2- 172.21.15.42:0/599587047 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcc84123500 0x7fcc84122240 secure :-1 s=THROTTLE_DONE pgs=527 cs=0 l=1 rx=0x7fcc74000e90 tx=0x7fcc740109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:07.067 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:07.043+0000 7fcc7a7fc700 1 -- 172.21.15.42:0/599587047 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) v1 ==== 94+0+2104 (secure 0 0 0) 0x7fcc74032540 con 0x7fcc84123500 2019-12-05T05:14:07.067 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 68 2019-12-05T05:14:07.068 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:07.051+0000 7fcc8b4a7700 1 -- 172.21.15.42:0/599587047 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fcc6403b300 msgr2=0x7fcc6403d810 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:14:07.068 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:07.051+0000 7fcc8b4a7700 1 --2- 172.21.15.42:0/599587047 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fcc6403b300 0x7fcc6403d810 crc :-1 s=READY pgs=242 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:14:07.068 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:07.051+0000 7fcc8b4a7700 1 -- 172.21.15.42:0/599587047 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcc84123500 msgr2=0x7fcc84122240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:14:07.068 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:07.051+0000 7fcc8b4a7700 1 --2- 172.21.15.42:0/599587047 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcc84123500 0x7fcc84122240 secure :-1 s=READY pgs=527 cs=0 l=1 rx=0x7fcc74000e90 tx=0x7fcc740109a0).stop 2019-12-05T05:14:07.068 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:07.051+0000 7fcc8b4a7700 1 -- 172.21.15.42:0/599587047 shutdown_connections 2019-12-05T05:14:07.068 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:07.051+0000 7fcc8b4a7700 1 --2- 172.21.15.42:0/599587047 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fcc6403b300 0x7fcc6403d810 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:14:07.069 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:07.051+0000 7fcc8b4a7700 1 --2- 172.21.15.42:0/599587047 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcc840722c0 0x7fcc84074700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:14:07.069 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:07.051+0000 7fcc8b4a7700 1 --2- 172.21.15.42:0/599587047 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcc84123500 0x7fcc84122240 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:14:07.069 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:07.051+0000 7fcc8b4a7700 1 -- 172.21.15.42:0/599587047 shutdown_connections 2019-12-05T05:14:07.069 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:07.051+0000 7fcc8b4a7700 1 -- 172.21.15.42:0/599587047 wait complete. 2019-12-05T05:14:07.069 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:07.051+0000 7fcc8b4a7700 1 -- 172.21.15.42:0/599587047 >> 172.21.15.42:0/599587047 conn(0x7fcc840a9640 msgr2=0x7fcc8409f1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:14:07.069 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:14:07.070 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":68,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4841,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/3150831963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3150831963},{"type":"v1","addr":"172.21.15.40:6833","nonce":3150831963}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":68}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:13:13.391909+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5505":{"gid":5505,"name":"a","rank":0,"incarnation":64,"state":"up:active","state_seq":9,"addr":"172.21.15.42:6835/211479572","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":211479572},{"type":"v1","addr":"172.21.15.42:6835","nonce":211479572}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:14:07.071 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T05:14:09.072 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:14:09.178 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:09.167+0000 7f2f8d21a700 1 Processor -- start 2019-12-05T05:14:09.179 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:09.167+0000 7f2f8d21a700 1 -- start start 2019-12-05T05:14:09.179 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:09.167+0000 7f2f8d21a700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2f880a7740 0x7f2f8808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:14:09.179 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:09.167+0000 7f2f8d21a700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2f8809fae0 0x7f2f8807eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:14:09.179 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:09.167+0000 7f2f8d21a700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f880a0710 con 0x7f2f8807e720 2019-12-05T05:14:09.179 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:09.167+0000 7f2f8d21a700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f880a0b00 con 0x7f2f8809fae0 2019-12-05T05:14:09.180 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:09.167+0000 7f2f8d21a700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f8808e5b0 con 0x7f2f880a7740 2019-12-05T05:14:09.180 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:09.167+0000 7f2f8759e700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2f880a7740 0x7f2f8808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:14:09.180 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:09.167+0000 7f2f8ca19700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2f8809fae0 0x7f2f8807eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:14:09.180 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:09.167+0000 7f2f8759e700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2f880a7740 0x7f2f8808df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:58880/0 (socket says 172.21.15.42:58880) 2019-12-05T05:14:09.180 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:09.167+0000 7f2f8759e700 1 -- 172.21.15.42:0/3509711499 learned_addr learned my addr 172.21.15.42:0/3509711499 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:14:09.180 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:09.167+0000 7f2f8ca19700 1 -- 172.21.15.42:0/3509711499 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2f880a7740 msgr2=0x7f2f8808df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:14:09.180 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:09.167+0000 7f2f8ca19700 1 --2- 172.21.15.42:0/3509711499 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2f880a7740 0x7f2f8808df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:14:09.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:09.167+0000 7f2f8659c700 1 -- 172.21.15.42:0/3509711499 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3972657160 0 0) 0x7f2f880a0710 con 0x7f2f8807e720 2019-12-05T05:14:09.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:09.167+0000 7f2f8ca19700 1 -- 172.21.15.42:0/3509711499 >> v1:172.21.15.40:6789/0 conn(0x7f2f8807e720 legacy=0x7f2f8809f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:14:09.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:09.167+0000 7f2f8ca19700 1 -- 172.21.15.42:0/3509711499 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f8807f0a0 con 0x7f2f8809fae0 2019-12-05T05:14:09.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:09.167+0000 7f2f8759e700 1 --2- 172.21.15.42:0/3509711499 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2f880a7740 0x7f2f8808df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:14:09.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:09.171+0000 7f2f8ca19700 1 --2- 172.21.15.42:0/3509711499 >> v2:172.21.15.40:3300/0 conn(0x7f2f8809fae0 0x7f2f8807eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f2f78008b10 tx=0x7f2f7800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:09.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:09.171+0000 7f2f8ca19700 1 --2- 172.21.15.42:0/3509711499 >> v2:172.21.15.40:3300/0 conn(0x7f2f8809fae0 0x7f2f8807eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f2f78008b10 tx=0x7f2f7800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:09.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:09.171+0000 7f2f8ca19700 1 --2- 172.21.15.42:0/3509711499 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2f8809fae0 0x7f2f8807eb50 secure :-1 s=READY pgs=421 cs=0 l=1 rx=0x7f2f78008b10 tx=0x7f2f7800fff0).ready entity=mon.0 client_cookie=8c287276b6461510 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:14:09.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:09.171+0000 7f2f8ca19700 1 --2- 172.21.15.42:0/3509711499 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2f8809fae0 0x7f2f8807eb50 secure :-1 s=THROTTLE_DONE pgs=421 cs=0 l=1 rx=0x7f2f78008b10 tx=0x7f2f7800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:09.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:09.171+0000 7f2f8659c700 1 -- 172.21.15.42:0/3509711499 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2f78010900 con 0x7f2f8809fae0 2019-12-05T05:14:09.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:09.171+0000 7f2f8ca19700 1 --2- 172.21.15.42:0/3509711499 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2f8809fae0 0x7f2f8807eb50 secure :-1 s=THROTTLE_DONE pgs=421 cs=0 l=1 rx=0x7f2f78008b10 tx=0x7f2f7800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:09.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:09.171+0000 7f2f8ca19700 1 --2- 172.21.15.42:0/3509711499 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2f8809fae0 0x7f2f8807eb50 secure :-1 s=THROTTLE_DONE pgs=421 cs=0 l=1 rx=0x7f2f78008b10 tx=0x7f2f7800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:09.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:09.171+0000 7f2f8659c700 1 -- 172.21.15.42:0/3509711499 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f78008b50 con 0x7f2f8809fae0 2019-12-05T05:14:09.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:09.171+0000 7f2f8659c700 1 -- 172.21.15.42:0/3509711499 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2f780110c0 con 0x7f2f8809fae0 2019-12-05T05:14:09.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:09.171+0000 7f2f8d21a700 1 -- 172.21.15.42:0/3509711499 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2f8809fae0 msgr2=0x7f2f8807eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:14:09.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:09.171+0000 7f2f8d21a700 1 --2- 172.21.15.42:0/3509711499 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2f8809fae0 0x7f2f8807eb50 secure :-1 s=READY pgs=421 cs=0 l=1 rx=0x7f2f78008b10 tx=0x7f2f7800fff0).stop 2019-12-05T05:14:09.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:09.171+0000 7f2f8d21a700 1 -- 172.21.15.42:0/3509711499 shutdown_connections 2019-12-05T05:14:09.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:09.171+0000 7f2f8d21a700 1 --2- 172.21.15.42:0/3509711499 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2f8809fae0 0x7f2f8807eb50 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:14:09.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:09.171+0000 7f2f8d21a700 1 --2- 172.21.15.42:0/3509711499 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2f880a7740 0x7f2f8808df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:14:09.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:09.171+0000 7f2f8d21a700 1 -- 172.21.15.42:0/3509711499 shutdown_connections 2019-12-05T05:14:09.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:09.171+0000 7f2f8d21a700 1 -- 172.21.15.42:0/3509711499 wait complete. 2019-12-05T05:14:09.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:09.171+0000 7f2f8d21a700 1 -- 172.21.15.42:0/3509711499 >> 172.21.15.42:0/3509711499 conn(0x7f2f880aabe0 msgr2=0x7f2f880a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:14:09.187 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:09.175+0000 7f2f8d21a700 1 Processor -- start 2019-12-05T05:14:09.187 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:09.175+0000 7f2f8d21a700 1 -- start start 2019-12-05T05:14:09.187 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:09.175+0000 7f2f8d21a700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2f88116500 0x7f2f881221b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:14:09.187 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:09.175+0000 7f2f8d21a700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2f8811a4d0 0x7f2f88120be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:14:09.187 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:09.175+0000 7f2f8d21a700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f8808f370 con 0x7f2f88121130 2019-12-05T05:14:09.188 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:09.175+0000 7f2f8d21a700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f88122930 con 0x7f2f8811a4d0 2019-12-05T05:14:09.188 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:09.175+0000 7f2f8d21a700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f88122bd0 con 0x7f2f88116500 2019-12-05T05:14:09.188 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:09.175+0000 7f2f8ca19700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f2f88121130 0x7f2f8808eb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:58462/0 (socket says 172.21.15.42:58462) 2019-12-05T05:14:09.188 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:09.175+0000 7f2f8ca19700 1 -- 172.21.15.42:0/3702843320 learned_addr learned my addr 172.21.15.42:0/3702843320 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:14:09.188 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:09.175+0000 7f2f86d9d700 1 --2- 172.21.15.42:0/3702843320 >> v2:172.21.15.40:3300/0 conn(0x7f2f8811a4d0 0x7f2f88120be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:14:09.188 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:09.175+0000 7f2f8759e700 1 --2- 172.21.15.42:0/3702843320 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2f88116500 0x7f2f881221b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:14:09.188 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:09.175+0000 7f2f7ffff700 1 -- 172.21.15.42:0/3702843320 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2350690311 0 0) 0x7f2f8808f370 con 0x7f2f88121130 2019-12-05T05:14:09.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:09.175+0000 7f2f7ffff700 1 -- 172.21.15.42:0/3702843320 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f6c0030b0 con 0x7f2f88121130 2019-12-05T05:14:09.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:09.175+0000 7f2f86d9d700 1 -- 172.21.15.42:0/3702843320 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2f88116500 msgr2=0x7f2f881221b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:14:09.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:09.175+0000 7f2f86d9d700 1 --2- 172.21.15.42:0/3702843320 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2f88116500 0x7f2f881221b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:14:09.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:09.175+0000 7f2f86d9d700 1 -- 172.21.15.42:0/3702843320 >> v1:172.21.15.42:6789/0 conn(0x7f2f88121130 legacy=0x7f2f8808eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:14:09.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:09.175+0000 7f2f86d9d700 1 -- 172.21.15.42:0/3702843320 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f8808f5a0 con 0x7f2f8811a4d0 2019-12-05T05:14:09.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:09.175+0000 7f2f86d9d700 1 --2- 172.21.15.42:0/3702843320 >> v2:172.21.15.40:3300/0 conn(0x7f2f8811a4d0 0x7f2f88120be0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f2f88122af0 tx=0x7f2f700108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:09.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:09.175+0000 7f2f86d9d700 1 --2- 172.21.15.42:0/3702843320 >> v2:172.21.15.40:3300/0 conn(0x7f2f8811a4d0 0x7f2f88120be0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f2f88122af0 tx=0x7f2f700108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:09.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:09.175+0000 7f2f86d9d700 1 --2- 172.21.15.42:0/3702843320 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2f8811a4d0 0x7f2f88120be0 secure :-1 s=READY pgs=422 cs=0 l=1 rx=0x7f2f88122af0 tx=0x7f2f700108f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:14:09.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:09.175+0000 7f2f86d9d700 1 --2- 172.21.15.42:0/3702843320 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2f8811a4d0 0x7f2f88120be0 secure :-1 s=THROTTLE_DONE pgs=422 cs=0 l=1 rx=0x7f2f88122af0 tx=0x7f2f700108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:09.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:09.175+0000 7f2f86d9d700 1 --2- 172.21.15.42:0/3702843320 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2f8811a4d0 0x7f2f88120be0 secure :-1 s=THROTTLE_DONE pgs=422 cs=0 l=1 rx=0x7f2f88122af0 tx=0x7f2f700108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:09.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:09.175+0000 7f2f7ffff700 1 -- 172.21.15.42:0/3702843320 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2f7000ca80 con 0x7f2f8811a4d0 2019-12-05T05:14:09.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:09.175+0000 7f2f86d9d700 1 --2- 172.21.15.42:0/3702843320 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2f8811a4d0 0x7f2f88120be0 secure :-1 s=THROTTLE_DONE pgs=422 cs=0 l=1 rx=0x7f2f88122af0 tx=0x7f2f700108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:09.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:09.175+0000 7f2f7ffff700 1 -- 172.21.15.42:0/3702843320 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f7000c6a0 con 0x7f2f8811a4d0 2019-12-05T05:14:09.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:09.175+0000 7f2f8d21a700 1 -- 172.21.15.42:0/3702843320 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f8808f870 con 0x7f2f8811a4d0 2019-12-05T05:14:09.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:09.175+0000 7f2f7ffff700 1 -- 172.21.15.42:0/3702843320 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2f700110f0 con 0x7f2f8811a4d0 2019-12-05T05:14:09.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:09.175+0000 7f2f8d21a700 1 -- 172.21.15.42:0/3702843320 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f88077520 con 0x7f2f8811a4d0 2019-12-05T05:14:09.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:09.179+0000 7f2f86d9d700 1 --2- 172.21.15.42:0/3702843320 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2f8811a4d0 0x7f2f88120be0 secure :-1 s=THROTTLE_DONE pgs=422 cs=0 l=1 rx=0x7f2f88122af0 tx=0x7f2f700108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:09.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:09.179+0000 7f2f86d9d700 1 --2- 172.21.15.42:0/3702843320 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2f8811a4d0 0x7f2f88120be0 secure :-1 s=THROTTLE_DONE pgs=422 cs=0 l=1 rx=0x7f2f88122af0 tx=0x7f2f700108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:09.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:09.179+0000 7f2f7ffff700 1 -- 172.21.15.42:0/3702843320 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f2f700113c0 con 0x7f2f8811a4d0 2019-12-05T05:14:09.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:09.179+0000 7f2f7ffff700 1 --2- 172.21.15.42:0/3702843320 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2f6c03b2a0 0x7f2f6c03d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:14:09.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:09.179+0000 7f2f7ffff700 1 -- 172.21.15.42:0/3702843320 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6633+0+0 (secure 0 0 0) 0x7f2f7004b150 con 0x7f2f8811a4d0 2019-12-05T05:14:09.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:09.179+0000 7f2f8759e700 1 --2- 172.21.15.42:0/3702843320 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2f6c03b2a0 0x7f2f6c03d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:14:09.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:09.179+0000 7f2f8759e700 1 --2- 172.21.15.42:0/3702843320 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2f6c03b2a0 0x7f2f6c03d7b0 crc :-1 s=READY pgs=243 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:14:09.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:09.183+0000 7f2f8d21a700 1 -- 172.21.15.42:0/3702843320 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f8805d110 con 0x7f2f8811a4d0 2019-12-05T05:14:09.198 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:09.191+0000 7f2f86d9d700 1 --2- 172.21.15.42:0/3702843320 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2f8811a4d0 0x7f2f88120be0 secure :-1 s=THROTTLE_DONE pgs=422 cs=0 l=1 rx=0x7f2f88122af0 tx=0x7f2f700108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:09.198 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:09.191+0000 7f2f7ffff700 1 -- 172.21.15.42:0/3702843320 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f2f70032190 con 0x7f2f8811a4d0 2019-12-05T05:14:09.367 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:09.359+0000 7f2f8d21a700 1 -- 172.21.15.42:0/3702843320 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2f880a6350 con 0x7f2f8811a4d0 2019-12-05T05:14:09.368 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:09.367+0000 7f2f86d9d700 1 --2- 172.21.15.42:0/3702843320 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2f8811a4d0 0x7f2f88120be0 secure :-1 s=THROTTLE_DONE pgs=422 cs=0 l=1 rx=0x7f2f88122af0 tx=0x7f2f700108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:09.368 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:09.367+0000 7f2f7ffff700 1 -- 172.21.15.42:0/3702843320 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) v1 ==== 94+0+2104 (secure 0 0 0) 0x7f2f700323f0 con 0x7f2f8811a4d0 2019-12-05T05:14:09.370 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 68 2019-12-05T05:14:09.371 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:14:09.371 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":68,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4841,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/3150831963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3150831963},{"type":"v1","addr":"172.21.15.40:6833","nonce":3150831963}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":68}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:13:13.391909+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5505":{"gid":5505,"name":"a","rank":0,"incarnation":64,"state":"up:active","state_seq":9,"addr":"172.21.15.42:6835/211479572","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":211479572},{"type":"v1","addr":"172.21.15.42:6835","nonce":211479572}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:14:09.373 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:09.367+0000 7f2f8d21a700 1 -- 172.21.15.42:0/3702843320 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2f6c03b2a0 msgr2=0x7f2f6c03d7b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:14:09.373 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:09.367+0000 7f2f8d21a700 1 --2- 172.21.15.42:0/3702843320 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2f6c03b2a0 0x7f2f6c03d7b0 crc :-1 s=READY pgs=243 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:14:09.374 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:09.367+0000 7f2f8d21a700 1 -- 172.21.15.42:0/3702843320 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2f8811a4d0 msgr2=0x7f2f88120be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:14:09.374 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:09.367+0000 7f2f8d21a700 1 --2- 172.21.15.42:0/3702843320 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2f8811a4d0 0x7f2f88120be0 secure :-1 s=READY pgs=422 cs=0 l=1 rx=0x7f2f88122af0 tx=0x7f2f700108f0).stop 2019-12-05T05:14:09.374 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:09.367+0000 7f2f8d21a700 1 -- 172.21.15.42:0/3702843320 shutdown_connections 2019-12-05T05:14:09.374 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:09.367+0000 7f2f8d21a700 1 --2- 172.21.15.42:0/3702843320 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2f6c03b2a0 0x7f2f6c03d7b0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:14:09.374 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:09.367+0000 7f2f8d21a700 1 --2- 172.21.15.42:0/3702843320 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2f88116500 0x7f2f881221b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:14:09.374 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:09.367+0000 7f2f8d21a700 1 --2- 172.21.15.42:0/3702843320 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2f8811a4d0 0x7f2f88120be0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:14:09.374 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:09.367+0000 7f2f8d21a700 1 -- 172.21.15.42:0/3702843320 shutdown_connections 2019-12-05T05:14:09.375 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:09.367+0000 7f2f8d21a700 1 -- 172.21.15.42:0/3702843320 wait complete. 2019-12-05T05:14:09.375 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:09.367+0000 7f2f8d21a700 1 -- 172.21.15.42:0/3702843320 >> 172.21.15.42:0/3702843320 conn(0x7f2f880a82a0 msgr2=0x7f2f8807e5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:14:09.383 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T05:14:11.383 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:14:11.492 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:11.475+0000 7f39f70bb700 1 Processor -- start 2019-12-05T05:14:11.493 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:11.475+0000 7f39f70bb700 1 -- start start 2019-12-05T05:14:11.493 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:11.475+0000 7f39f70bb700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f39f00a7740 0x7f39f008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:14:11.493 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:11.475+0000 7f39f70bb700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39f00a0710 con 0x7f39f009fae0 2019-12-05T05:14:11.493 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:11.475+0000 7f39f70bb700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39f00a9ba0 con 0x7f39f007e720 2019-12-05T05:14:11.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:11.479+0000 7f39f70bb700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39f00a0b00 con 0x7f39f00a7740 2019-12-05T05:14:11.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:11.479+0000 7f39f4e57700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f39f007e720 0x7f39f009f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:58468/0 (socket says 172.21.15.42:58468) 2019-12-05T05:14:11.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:11.479+0000 7f39f4e57700 1 -- 172.21.15.42:0/120231505 learned_addr learned my addr 172.21.15.42:0/120231505 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:14:11.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:11.479+0000 7f39f5658700 1 --2- 172.21.15.42:0/120231505 >> v2:172.21.15.40:3300/0 conn(0x7f39f00a7740 0x7f39f008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:14:11.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:11.479+0000 7f39e7fff700 1 -- 172.21.15.42:0/120231505 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 693097534 0 0) 0x7f39f00a9ba0 con 0x7f39f007e720 2019-12-05T05:14:11.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:11.479+0000 7f39e7fff700 1 -- 172.21.15.42:0/120231505 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39dc0030b0 con 0x7f39f007e720 2019-12-05T05:14:11.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:11.479+0000 7f39e7fff700 1 -- 172.21.15.42:0/120231505 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 527421821 0 0) 0x7f39f00a0710 con 0x7f39f009fae0 2019-12-05T05:14:11.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:11.479+0000 7f39e7fff700 1 -- 172.21.15.42:0/120231505 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39f00a9ba0 con 0x7f39f009fae0 2019-12-05T05:14:11.496 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:11.479+0000 7f39e7fff700 1 -- 172.21.15.42:0/120231505 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2230607922 0 0) 0x7f39dc0030b0 con 0x7f39f007e720 2019-12-05T05:14:11.496 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:11.479+0000 7f39e7fff700 1 -- 172.21.15.42:0/120231505 >> v2:172.21.15.40:3300/0 conn(0x7f39f00a7740 msgr2=0x7f39f008df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:14:11.496 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:11.479+0000 7f39e7fff700 1 --2- 172.21.15.42:0/120231505 >> v2:172.21.15.40:3300/0 conn(0x7f39f00a7740 0x7f39f008df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:14:11.496 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:11.479+0000 7f39e7fff700 1 -- 172.21.15.42:0/120231505 >> v1:172.21.15.40:6789/0 conn(0x7f39f009fae0 legacy=0x7f39f014efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:14:11.497 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:11.479+0000 7f39e7fff700 1 -- 172.21.15.42:0/120231505 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39f007eb50 con 0x7f39f007e720 2019-12-05T05:14:11.497 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:11.479+0000 7f39e7fff700 1 -- 172.21.15.42:0/120231505 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7f39e0001750 con 0x7f39f007e720 2019-12-05T05:14:11.497 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:11.479+0000 7f39e7fff700 1 -- 172.21.15.42:0/120231505 >> v1:172.21.15.42:6789/0 conn(0x7f39f007e720 legacy=0x7f39f009f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:14:11.497 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:11.479+0000 7f39e7fff700 1 --2- 172.21.15.42:0/120231505 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f39dc003910 0x7f39dc003d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:14:11.498 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:11.479+0000 7f39e7fff700 1 --2- 172.21.15.42:0/120231505 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f39dc004420 0x7f39dc0048f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:14:11.498 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:11.479+0000 7f39e7fff700 1 --2- 172.21.15.42:0/120231505 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f39dc004fa0 0x7f39dc04f700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:14:11.498 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:11.479+0000 7f39e7fff700 1 -- 172.21.15.42:0/120231505 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39f00a0b00 con 0x7f39dc004420 2019-12-05T05:14:11.498 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:11.479+0000 7f39e7fff700 1 -- 172.21.15.42:0/120231505 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39dc04fec0 con 0x7f39dc003910 2019-12-05T05:14:11.499 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:11.479+0000 7f39e7fff700 1 -- 172.21.15.42:0/120231505 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39dc050100 con 0x7f39dc004fa0 2019-12-05T05:14:11.499 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:11.479+0000 7f39f5658700 1 --2- 172.21.15.42:0/120231505 >> v2:172.21.15.40:3300/0 conn(0x7f39f00a7740 0x7f39f008df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:14:11.499 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:11.479+0000 7f39f4e57700 1 --2- 172.21.15.42:0/120231505 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f39dc004420 0x7f39dc0048f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:14:11.499 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:11.479+0000 7f39f5658700 1 --2- 172.21.15.42:0/120231505 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f39dc003910 0x7f39dc003d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:14:11.499 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:11.479+0000 7f39f5e59700 1 --2- 172.21.15.42:0/120231505 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f39dc004fa0 0x7f39dc04f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:14:11.500 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:11.479+0000 7f39f5658700 1 -- 172.21.15.42:0/120231505 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f39dc004fa0 msgr2=0x7f39dc04f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:14:11.500 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:11.479+0000 7f39f5658700 1 --2- 172.21.15.42:0/120231505 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f39dc004fa0 0x7f39dc04f700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:14:11.501 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:11.479+0000 7f39f5658700 1 -- 172.21.15.42:0/120231505 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f39dc004420 msgr2=0x7f39dc0048f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:14:11.501 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:11.479+0000 7f39f5658700 1 --2- 172.21.15.42:0/120231505 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f39dc004420 0x7f39dc0048f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:14:11.501 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:11.479+0000 7f39f5e59700 1 -- 172.21.15.42:0/120231505 reap_dead start 2019-12-05T05:14:11.501 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:11.479+0000 7f39f5658700 1 -- 172.21.15.42:0/120231505 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39dc0503a0 con 0x7f39dc003910 2019-12-05T05:14:11.502 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:11.479+0000 7f39f5658700 1 --2- 172.21.15.42:0/120231505 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f39dc003910 0x7f39dc003d60 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f39ec002f30 tx=0x7f39ec00bed0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:11.502 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:11.479+0000 7f39f5658700 1 --2- 172.21.15.42:0/120231505 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f39dc003910 0x7f39dc003d60 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f39ec002f30 tx=0x7f39ec00bed0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:11.502 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:11.479+0000 7f39f5658700 1 --2- 172.21.15.42:0/120231505 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f39dc003910 0x7f39dc003d60 secure :-1 s=READY pgs=530 cs=0 l=1 rx=0x7f39ec002f30 tx=0x7f39ec00bed0).ready entity=mon.1 client_cookie=b85763740129e5cf server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:14:11.502 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:11.479+0000 7f39f5658700 1 --2- 172.21.15.42:0/120231505 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f39dc003910 0x7f39dc003d60 secure :-1 s=THROTTLE_DONE pgs=530 cs=0 l=1 rx=0x7f39ec002f30 tx=0x7f39ec00bed0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:11.502 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:11.479+0000 7f39e7fff700 1 -- 172.21.15.42:0/120231505 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f39ec011b00 con 0x7f39dc003910 2019-12-05T05:14:11.503 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:11.479+0000 7f39f5658700 1 --2- 172.21.15.42:0/120231505 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f39dc003910 0x7f39dc003d60 secure :-1 s=THROTTLE_DONE pgs=530 cs=0 l=1 rx=0x7f39ec002f30 tx=0x7f39ec00bed0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:11.503 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:11.479+0000 7f39f5658700 1 --2- 172.21.15.42:0/120231505 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f39dc003910 0x7f39dc003d60 secure :-1 s=THROTTLE_DONE pgs=530 cs=0 l=1 rx=0x7f39ec002f30 tx=0x7f39ec00bed0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:11.503 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:11.479+0000 7f39e7fff700 1 -- 172.21.15.42:0/120231505 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39ec007150 con 0x7f39dc003910 2019-12-05T05:14:11.503 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:11.479+0000 7f39e7fff700 1 -- 172.21.15.42:0/120231505 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f39ec005730 con 0x7f39dc003910 2019-12-05T05:14:11.503 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:11.479+0000 7f39f70bb700 1 -- 172.21.15.42:0/120231505 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f39dc003910 msgr2=0x7f39dc003d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:14:11.504 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:11.479+0000 7f39f70bb700 1 --2- 172.21.15.42:0/120231505 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f39dc003910 0x7f39dc003d60 secure :-1 s=READY pgs=530 cs=0 l=1 rx=0x7f39ec002f30 tx=0x7f39ec00bed0).stop 2019-12-05T05:14:11.504 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:11.483+0000 7f39f70bb700 1 -- 172.21.15.42:0/120231505 shutdown_connections 2019-12-05T05:14:11.504 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:11.483+0000 7f39f70bb700 1 --2- 172.21.15.42:0/120231505 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f39dc003910 0x7f39dc003d60 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:14:11.504 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:11.483+0000 7f39f70bb700 1 -- 172.21.15.42:0/120231505 shutdown_connections 2019-12-05T05:14:11.504 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:11.483+0000 7f39f70bb700 1 -- 172.21.15.42:0/120231505 wait complete. 2019-12-05T05:14:11.505 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:11.483+0000 7f39f70bb700 1 -- 172.21.15.42:0/120231505 >> 172.21.15.42:0/120231505 conn(0x7f39f00aabe0 msgr2=0x7f39f00a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:14:11.505 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:11.483+0000 7f39f70bb700 1 Processor -- start 2019-12-05T05:14:11.505 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:11.483+0000 7f39f70bb700 1 -- start start 2019-12-05T05:14:11.505 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:11.487+0000 7f39f70bb700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f39f00a3a20 0x7f39f00a2370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:14:11.506 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:11.487+0000 7f39f70bb700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f39f00a28c0 0x7f39f011aff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:14:11.506 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:11.487+0000 7f39f70bb700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39f00a4490 con 0x7f39f0145120 2019-12-05T05:14:11.506 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:11.487+0000 7f39f70bb700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39f00a3f50 con 0x7f39f00a3a20 2019-12-05T05:14:11.506 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:11.487+0000 7f39f70bb700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39f00a41f0 con 0x7f39f00a28c0 2019-12-05T05:14:11.506 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:11.487+0000 7f39f5658700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f39f0145120 0x7f39f011ca00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:58478/0 (socket says 172.21.15.42:58478) 2019-12-05T05:14:11.507 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:11.487+0000 7f39f5658700 1 -- 172.21.15.42:0/682713886 learned_addr learned my addr 172.21.15.42:0/682713886 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:14:11.507 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:11.487+0000 7f39f4e57700 1 --2- 172.21.15.42:0/682713886 >> v2:172.21.15.40:3300/0 conn(0x7f39f00a3a20 0x7f39f00a2370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:14:11.507 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:11.487+0000 7f39f5e59700 1 --2- 172.21.15.42:0/682713886 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f39f00a28c0 0x7f39f011aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:14:11.507 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:11.487+0000 7f39e5ffb700 1 -- 172.21.15.42:0/682713886 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1333643741 0 0) 0x7f39f00a4490 con 0x7f39f0145120 2019-12-05T05:14:11.508 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:11.487+0000 7f39e5ffb700 1 -- 172.21.15.42:0/682713886 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39d00030b0 con 0x7f39f0145120 2019-12-05T05:14:11.510 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:11.487+0000 7f39f4e57700 1 -- 172.21.15.42:0/682713886 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f39f00a28c0 msgr2=0x7f39f011aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:14:11.511 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:11.487+0000 7f39f4e57700 1 --2- 172.21.15.42:0/682713886 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f39f00a28c0 0x7f39f011aff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:14:11.511 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:11.487+0000 7f39f4e57700 1 -- 172.21.15.42:0/682713886 >> v1:172.21.15.42:6789/0 conn(0x7f39f0145120 legacy=0x7f39f011ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:14:11.511 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:11.487+0000 7f39e5ffb700 1 -- 172.21.15.42:0/682713886 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 2427174778 0 0) 0x7f39d00030b0 con 0x7f39f0145120 2019-12-05T05:14:11.511 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:11.487+0000 7f39f4e57700 1 -- 172.21.15.42:0/682713886 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39f00a2cf0 con 0x7f39f00a3a20 2019-12-05T05:14:11.511 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:11.487+0000 7f39f5e59700 1 --2- 172.21.15.42:0/682713886 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f39f00a28c0 0x7f39f011aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:14:11.511 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:11.487+0000 7f39f4e57700 1 --2- 172.21.15.42:0/682713886 >> v2:172.21.15.40:3300/0 conn(0x7f39f00a3a20 0x7f39f00a2370 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f39ec011f70 tx=0x7f39ec00c180).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:11.512 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:11.487+0000 7f39f4e57700 1 --2- 172.21.15.42:0/682713886 >> v2:172.21.15.40:3300/0 conn(0x7f39f00a3a20 0x7f39f00a2370 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f39ec011f70 tx=0x7f39ec00c180).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:11.512 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:11.487+0000 7f39f4e57700 1 --2- 172.21.15.42:0/682713886 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f39f00a3a20 0x7f39f00a2370 secure :-1 s=READY pgs=424 cs=0 l=1 rx=0x7f39ec011f70 tx=0x7f39ec00c180).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:14:11.512 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:11.487+0000 7f39f4e57700 1 --2- 172.21.15.42:0/682713886 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f39f00a3a20 0x7f39f00a2370 secure :-1 s=THROTTLE_DONE pgs=424 cs=0 l=1 rx=0x7f39ec011f70 tx=0x7f39ec00c180).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:11.512 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:11.487+0000 7f39e5ffb700 1 -- 172.21.15.42:0/682713886 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f39ec011b00 con 0x7f39f00a3a20 2019-12-05T05:14:11.512 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:11.487+0000 7f39f4e57700 1 --2- 172.21.15.42:0/682713886 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f39f00a3a20 0x7f39f00a2370 secure :-1 s=THROTTLE_DONE pgs=424 cs=0 l=1 rx=0x7f39ec011f70 tx=0x7f39ec00c180).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:11.512 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:11.487+0000 7f39f4e57700 1 --2- 172.21.15.42:0/682713886 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f39f00a3a20 0x7f39f00a2370 secure :-1 s=THROTTLE_DONE pgs=424 cs=0 l=1 rx=0x7f39ec011f70 tx=0x7f39ec00c180).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:11.512 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:11.487+0000 7f39e5ffb700 1 -- 172.21.15.42:0/682713886 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39ec011d00 con 0x7f39f00a3a20 2019-12-05T05:14:11.513 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:11.487+0000 7f39e5ffb700 1 -- 172.21.15.42:0/682713886 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f39e0014b60 con 0x7f39f00a3a20 2019-12-05T05:14:11.513 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:11.487+0000 7f39f70bb700 1 -- 172.21.15.42:0/682713886 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39f011b540 con 0x7f39f00a3a20 2019-12-05T05:14:11.513 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:11.487+0000 7f39f70bb700 1 -- 172.21.15.42:0/682713886 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39f011ba90 con 0x7f39f00a3a20 2019-12-05T05:14:11.513 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:11.487+0000 7f39f4e57700 1 --2- 172.21.15.42:0/682713886 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f39f00a3a20 0x7f39f00a2370 secure :-1 s=THROTTLE_DONE pgs=424 cs=0 l=1 rx=0x7f39ec011f70 tx=0x7f39ec00c180).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:11.513 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:11.491+0000 7f39e5ffb700 1 -- 172.21.15.42:0/682713886 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f39e0005410 con 0x7f39f00a3a20 2019-12-05T05:14:11.513 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:11.491+0000 7f39f4e57700 1 --2- 172.21.15.42:0/682713886 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f39f00a3a20 0x7f39f00a2370 secure :-1 s=THROTTLE_DONE pgs=424 cs=0 l=1 rx=0x7f39ec011f70 tx=0x7f39ec00c180).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:11.513 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:11.491+0000 7f39e5ffb700 1 --2- 172.21.15.42:0/682713886 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f39d003b190 0x7f39d003d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:14:11.514 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:11.491+0000 7f39e5ffb700 1 -- 172.21.15.42:0/682713886 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6633+0+0 (secure 0 0 0) 0x7f39e004cc30 con 0x7f39f00a3a20 2019-12-05T05:14:11.514 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:11.491+0000 7f39f5658700 1 --2- 172.21.15.42:0/682713886 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f39d003b190 0x7f39d003d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:14:11.514 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:11.491+0000 7f39f5658700 1 --2- 172.21.15.42:0/682713886 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f39d003b190 0x7f39d003d6a0 crc :-1 s=READY pgs=244 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:14:11.514 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:11.491+0000 7f39f70bb700 1 -- 172.21.15.42:0/682713886 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39f005d110 con 0x7f39f00a3a20 2019-12-05T05:14:11.514 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:11.499+0000 7f39f4e57700 1 --2- 172.21.15.42:0/682713886 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f39f00a3a20 0x7f39f00a2370 secure :-1 s=THROTTLE_DONE pgs=424 cs=0 l=1 rx=0x7f39ec011f70 tx=0x7f39ec00c180).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:11.514 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:11.499+0000 7f39e5ffb700 1 -- 172.21.15.42:0/682713886 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f39e0033f10 con 0x7f39f00a3a20 2019-12-05T05:14:11.680 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:11.667+0000 7f39f70bb700 1 -- 172.21.15.42:0/682713886 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f39f008f960 con 0x7f39f00a3a20 2019-12-05T05:14:11.680 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:11.667+0000 7f39f4e57700 1 --2- 172.21.15.42:0/682713886 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f39f00a3a20 0x7f39f00a2370 secure :-1 s=THROTTLE_DONE pgs=424 cs=0 l=1 rx=0x7f39ec011f70 tx=0x7f39ec00c180).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:11.681 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:11.667+0000 7f39e5ffb700 1 -- 172.21.15.42:0/682713886 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) v1 ==== 94+0+2104 (secure 0 0 0) 0x7f39e0001230 con 0x7f39f00a3a20 2019-12-05T05:14:11.681 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 68 2019-12-05T05:14:11.681 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:11.671+0000 7f39f70bb700 1 -- 172.21.15.42:0/682713886 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f39d003b190 msgr2=0x7f39d003d6a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:14:11.681 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:11.671+0000 7f39f70bb700 1 --2- 172.21.15.42:0/682713886 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f39d003b190 0x7f39d003d6a0 crc :-1 s=READY pgs=244 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:14:11.681 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:11.671+0000 7f39f70bb700 1 -- 172.21.15.42:0/682713886 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f39f00a3a20 msgr2=0x7f39f00a2370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:14:11.681 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:11.671+0000 7f39f70bb700 1 --2- 172.21.15.42:0/682713886 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f39f00a3a20 0x7f39f00a2370 secure :-1 s=READY pgs=424 cs=0 l=1 rx=0x7f39ec011f70 tx=0x7f39ec00c180).stop 2019-12-05T05:14:11.681 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:11.671+0000 7f39f70bb700 1 -- 172.21.15.42:0/682713886 shutdown_connections 2019-12-05T05:14:11.682 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:11.671+0000 7f39f70bb700 1 --2- 172.21.15.42:0/682713886 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f39d003b190 0x7f39d003d6a0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:14:11.682 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:11.671+0000 7f39f70bb700 1 --2- 172.21.15.42:0/682713886 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f39f00a28c0 0x7f39f011aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:14:11.682 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:11.671+0000 7f39f70bb700 1 --2- 172.21.15.42:0/682713886 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f39f00a3a20 0x7f39f00a2370 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:14:11.682 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:11.671+0000 7f39f70bb700 1 -- 172.21.15.42:0/682713886 shutdown_connections 2019-12-05T05:14:11.682 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:11.671+0000 7f39f70bb700 1 -- 172.21.15.42:0/682713886 wait complete. 2019-12-05T05:14:11.682 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:11.671+0000 7f39f70bb700 1 -- 172.21.15.42:0/682713886 >> 172.21.15.42:0/682713886 conn(0x7f39f007de90 msgr2=0x7f39f008eb70 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:14:11.683 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:14:11.683 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":68,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4841,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/3150831963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3150831963},{"type":"v1","addr":"172.21.15.40:6833","nonce":3150831963}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":68}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:13:13.391909+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5505":{"gid":5505,"name":"a","rank":0,"incarnation":64,"state":"up:active","state_seq":9,"addr":"172.21.15.42:6835/211479572","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":211479572},{"type":"v1","addr":"172.21.15.42:6835","nonce":211479572}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:14:11.690 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T05:14:13.691 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:14:13.798 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:13.783+0000 7f8d04ab6700 1 Processor -- start 2019-12-05T05:14:13.798 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:13.783+0000 7f8d04ab6700 1 -- start start 2019-12-05T05:14:13.798 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:13.787+0000 7f8d04ab6700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8d0008df80 0x7f8d0009f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:14:13.798 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:13.787+0000 7f8d04ab6700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d0009fc60 con 0x7f8d0009f830 2019-12-05T05:14:13.798 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:13.787+0000 7f8d04ab6700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d000a0710 con 0x7f8d000a7740 2019-12-05T05:14:13.799 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:13.787+0000 7f8d04ab6700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d000a0b00 con 0x7f8d0008df80 2019-12-05T05:14:13.799 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:13.787+0000 7f8cfed9d700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f8d000a7740 0x7f8d0007e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:58490/0 (socket says 172.21.15.42:58490) 2019-12-05T05:14:13.799 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:13.787+0000 7f8cfed9d700 1 -- 172.21.15.42:0/3584457381 learned_addr learned my addr 172.21.15.42:0/3584457381 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:14:13.799 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:13.787+0000 7f8cf7fff700 1 --2- 172.21.15.42:0/3584457381 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8d0008df80 0x7f8d0009f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:14:13.799 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:13.787+0000 7f8cfe59c700 1 -- 172.21.15.42:0/3584457381 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2516689525 0 0) 0x7f8d000a0710 con 0x7f8d000a7740 2019-12-05T05:14:13.799 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:13.787+0000 7f8cfe59c700 1 -- 172.21.15.42:0/3584457381 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8cec0030b0 con 0x7f8d000a7740 2019-12-05T05:14:13.799 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:13.787+0000 7f8cfe59c700 1 -- 172.21.15.42:0/3584457381 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1912940811 0 0) 0x7f8cec0030b0 con 0x7f8d000a7740 2019-12-05T05:14:13.800 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:13.787+0000 7f8cfe59c700 1 -- 172.21.15.42:0/3584457381 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8d0008df80 msgr2=0x7f8d0009f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:14:13.800 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:13.787+0000 7f8cfe59c700 1 --2- 172.21.15.42:0/3584457381 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8d0008df80 0x7f8d0009f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:14:13.800 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:13.787+0000 7f8cfe59c700 1 -- 172.21.15.42:0/3584457381 >> v1:172.21.15.40:6789/0 conn(0x7f8d0009f830 legacy=0x7f8d0014efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:14:13.800 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:13.787+0000 7f8cfe59c700 1 -- 172.21.15.42:0/3584457381 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d0007ef20 con 0x7f8d000a7740 2019-12-05T05:14:13.800 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:13.787+0000 7f8cfe59c700 1 -- 172.21.15.42:0/3584457381 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7f8cf0001750 con 0x7f8d000a7740 2019-12-05T05:14:13.800 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:13.787+0000 7f8cfe59c700 1 -- 172.21.15.42:0/3584457381 >> v1:172.21.15.42:6789/0 conn(0x7f8d000a7740 legacy=0x7f8d0007e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:14:13.800 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:13.787+0000 7f8cfe59c700 1 --2- 172.21.15.42:0/3584457381 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8cec003710 0x7f8cec003b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:14:13.801 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:13.787+0000 7f8cfe59c700 1 --2- 172.21.15.42:0/3584457381 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8cec004420 0x7f8cec004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:14:13.801 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:13.787+0000 7f8cfe59c700 1 --2- 172.21.15.42:0/3584457381 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8cec004da0 0x7f8cec00b670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:14:13.801 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:13.787+0000 7f8cfe59c700 1 -- 172.21.15.42:0/3584457381 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d000a0b00 con 0x7f8cec003710 2019-12-05T05:14:13.801 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:13.787+0000 7f8cff59e700 1 --2- 172.21.15.42:0/3584457381 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8cec004da0 0x7f8cec00b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:14:13.801 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:13.787+0000 7f8cfed9d700 1 --2- 172.21.15.42:0/3584457381 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8cec003710 0x7f8cec003b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:14:13.801 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:13.787+0000 7f8cfe59c700 1 -- 172.21.15.42:0/3584457381 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cec0051d0 con 0x7f8cec004da0 2019-12-05T05:14:13.801 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:13.787+0000 7f8cfe59c700 1 -- 172.21.15.42:0/3584457381 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cec04fbd0 con 0x7f8cec004420 2019-12-05T05:14:13.802 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:13.787+0000 7f8cf7fff700 1 --2- 172.21.15.42:0/3584457381 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8cec004420 0x7f8cec004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:14:13.802 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:13.787+0000 7f8cff59e700 1 -- 172.21.15.42:0/3584457381 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8cec004420 msgr2=0x7f8cec004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:14:13.802 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:13.787+0000 7f8cff59e700 1 --2- 172.21.15.42:0/3584457381 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8cec004420 0x7f8cec004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:14:13.802 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:13.787+0000 7f8cff59e700 1 -- 172.21.15.42:0/3584457381 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8cec003710 msgr2=0x7f8cec003b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:14:13.802 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:13.787+0000 7f8cff59e700 1 --2- 172.21.15.42:0/3584457381 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8cec003710 0x7f8cec003b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:14:13.802 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:13.787+0000 7f8cff59e700 1 -- 172.21.15.42:0/3584457381 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cec04fed0 con 0x7f8cec004da0 2019-12-05T05:14:13.802 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:13.787+0000 7f8cff59e700 1 --2- 172.21.15.42:0/3584457381 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8cec004da0 0x7f8cec00b670 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f8cf8001380 tx=0x7f8cf8010050).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:13.802 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:13.787+0000 7f8cff59e700 1 --2- 172.21.15.42:0/3584457381 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8cec004da0 0x7f8cec00b670 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f8cf8001380 tx=0x7f8cf8010050).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:13.803 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:13.787+0000 7f8cff59e700 1 --2- 172.21.15.42:0/3584457381 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8cec004da0 0x7f8cec00b670 secure :-1 s=READY pgs=533 cs=0 l=1 rx=0x7f8cf8001380 tx=0x7f8cf8010050).ready entity=mon.1 client_cookie=5f36fa4aa7fa22f2 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:14:13.803 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:13.791+0000 7f8cff59e700 1 --2- 172.21.15.42:0/3584457381 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8cec004da0 0x7f8cec00b670 secure :-1 s=THROTTLE_DONE pgs=533 cs=0 l=1 rx=0x7f8cf8001380 tx=0x7f8cf8010050).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:13.803 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:13.791+0000 7f8cff59e700 1 --2- 172.21.15.42:0/3584457381 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8cec004da0 0x7f8cec00b670 secure :-1 s=THROTTLE_DONE pgs=533 cs=0 l=1 rx=0x7f8cf8001380 tx=0x7f8cf8010050).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:13.803 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:13.791+0000 7f8cfe59c700 1 -- 172.21.15.42:0/3584457381 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8cf8006ac0 con 0x7f8cec004da0 2019-12-05T05:14:13.803 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:13.791+0000 7f8cff59e700 1 --2- 172.21.15.42:0/3584457381 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8cec004da0 0x7f8cec00b670 secure :-1 s=THROTTLE_DONE pgs=533 cs=0 l=1 rx=0x7f8cf8001380 tx=0x7f8cf8010050).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:13.803 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:13.791+0000 7f8cfe59c700 1 -- 172.21.15.42:0/3584457381 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cf80066e0 con 0x7f8cec004da0 2019-12-05T05:14:13.803 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:13.791+0000 7f8cfe59c700 1 -- 172.21.15.42:0/3584457381 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8cf8002730 con 0x7f8cec004da0 2019-12-05T05:14:13.804 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:13.791+0000 7f8d04ab6700 1 -- 172.21.15.42:0/3584457381 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8cec004da0 msgr2=0x7f8cec00b670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:14:13.804 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:13.791+0000 7f8d04ab6700 1 --2- 172.21.15.42:0/3584457381 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8cec004da0 0x7f8cec00b670 secure :-1 s=READY pgs=533 cs=0 l=1 rx=0x7f8cf8001380 tx=0x7f8cf8010050).stop 2019-12-05T05:14:13.804 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:13.791+0000 7f8cff59e700 1 -- 172.21.15.42:0/3584457381 reap_dead start 2019-12-05T05:14:13.804 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:13.791+0000 7f8d04ab6700 1 -- 172.21.15.42:0/3584457381 shutdown_connections 2019-12-05T05:14:13.804 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:13.791+0000 7f8d04ab6700 1 -- 172.21.15.42:0/3584457381 shutdown_connections 2019-12-05T05:14:13.804 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:13.791+0000 7f8d04ab6700 1 -- 172.21.15.42:0/3584457381 wait complete. 2019-12-05T05:14:13.804 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:13.791+0000 7f8d04ab6700 1 -- 172.21.15.42:0/3584457381 >> 172.21.15.42:0/3584457381 conn(0x7f8d000aabe0 msgr2=0x7f8d000a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:14:13.819 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:13.795+0000 7f8d04ab6700 1 Processor -- start 2019-12-05T05:14:13.819 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:13.795+0000 7f8d04ab6700 1 -- start start 2019-12-05T05:14:13.819 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:13.795+0000 7f8d04ab6700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f8d0011a770 0x7f8d0011ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:14:13.819 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:13.795+0000 7f8d04ab6700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8d0011d940 0x7f8d0011dd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:14:13.819 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:13.795+0000 7f8d04ab6700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d0011e2c0 con 0x7f8d000a3a60 2019-12-05T05:14:13.820 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:13.795+0000 7f8d04ab6700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d0011f480 con 0x7f8d0011a770 2019-12-05T05:14:13.820 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:13.795+0000 7f8d04ab6700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d0011f6e0 con 0x7f8d0011d940 2019-12-05T05:14:13.820 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:13.795+0000 7f8cfed9d700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f8d000a3a60 0x7f8d00077910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:58502/0 (socket says 172.21.15.42:58502) 2019-12-05T05:14:13.820 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:13.795+0000 7f8cfed9d700 1 -- 172.21.15.42:0/1998856607 learned_addr learned my addr 172.21.15.42:0/1998856607 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:14:13.820 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:13.795+0000 7f8cf7fff700 1 --2- 172.21.15.42:0/1998856607 >> v2:172.21.15.42:3300/0 conn(0x7f8d0011a770 0x7f8d0011ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:14:13.820 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:13.795+0000 7f8cff59e700 1 --2- 172.21.15.42:0/1998856607 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8d0011d940 0x7f8d0011dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:14:13.820 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:13.795+0000 7f8cf7fff700 1 -- 172.21.15.42:0/1998856607 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8d0011d940 msgr2=0x7f8d0011dd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:14:13.821 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:13.795+0000 7f8cf7fff700 1 --2- 172.21.15.42:0/1998856607 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8d0011d940 0x7f8d0011dd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:14:13.821 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:13.795+0000 7f8cf7fff700 1 -- 172.21.15.42:0/1998856607 >> v1:172.21.15.42:6789/0 conn(0x7f8d000a3a60 legacy=0x7f8d00077910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:14:13.821 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:13.795+0000 7f8cf7fff700 1 -- 172.21.15.42:0/1998856607 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d00078110 con 0x7f8d0011a770 2019-12-05T05:14:13.821 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:13.795+0000 7f8cf7fff700 1 --2- 172.21.15.42:0/1998856607 >> v2:172.21.15.42:3300/0 conn(0x7f8d0011a770 0x7f8d0011ef30 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f8ce800c280 tx=0x7f8ce8010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:13.821 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:13.795+0000 7f8cf7fff700 1 --2- 172.21.15.42:0/1998856607 >> v2:172.21.15.42:3300/0 conn(0x7f8d0011a770 0x7f8d0011ef30 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f8ce800c280 tx=0x7f8ce8010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:13.821 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:13.795+0000 7f8cf7fff700 1 --2- 172.21.15.42:0/1998856607 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8d0011a770 0x7f8d0011ef30 secure :-1 s=READY pgs=535 cs=0 l=1 rx=0x7f8ce800c280 tx=0x7f8ce8010b60).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:14:13.821 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:13.795+0000 7f8cf7fff700 1 --2- 172.21.15.42:0/1998856607 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8d0011a770 0x7f8d0011ef30 secure :-1 s=THROTTLE_DONE pgs=535 cs=0 l=1 rx=0x7f8ce800c280 tx=0x7f8ce8010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:13.822 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:13.795+0000 7f8cf77fe700 1 -- 172.21.15.42:0/1998856607 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8ce8005930 con 0x7f8d0011a770 2019-12-05T05:14:13.822 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:13.795+0000 7f8cf7fff700 1 --2- 172.21.15.42:0/1998856607 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8d0011a770 0x7f8d0011ef30 secure :-1 s=THROTTLE_DONE pgs=535 cs=0 l=1 rx=0x7f8ce800c280 tx=0x7f8ce8010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:13.822 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:13.795+0000 7f8cf7fff700 1 --2- 172.21.15.42:0/1998856607 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8d0011a770 0x7f8d0011ef30 secure :-1 s=THROTTLE_DONE pgs=535 cs=0 l=1 rx=0x7f8ce800c280 tx=0x7f8ce8010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:13.822 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:13.795+0000 7f8cf77fe700 1 -- 172.21.15.42:0/1998856607 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ce8007160 con 0x7f8d0011a770 2019-12-05T05:14:13.822 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:13.795+0000 7f8d04ab6700 1 -- 172.21.15.42:0/1998856607 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d001209c0 con 0x7f8d0011a770 2019-12-05T05:14:13.822 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:13.795+0000 7f8cf77fe700 1 -- 172.21.15.42:0/1998856607 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8ce8008260 con 0x7f8d0011a770 2019-12-05T05:14:13.822 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:13.795+0000 7f8d04ab6700 1 -- 172.21.15.42:0/1998856607 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d001357b0 con 0x7f8d0011a770 2019-12-05T05:14:13.822 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:13.795+0000 7f8cf7fff700 1 --2- 172.21.15.42:0/1998856607 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8d0011a770 0x7f8d0011ef30 secure :-1 s=THROTTLE_DONE pgs=535 cs=0 l=1 rx=0x7f8ce800c280 tx=0x7f8ce8010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:13.823 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:13.799+0000 7f8cf77fe700 1 -- 172.21.15.42:0/1998856607 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f8ce8008540 con 0x7f8d0011a770 2019-12-05T05:14:13.823 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:13.799+0000 7f8cf7fff700 1 --2- 172.21.15.42:0/1998856607 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8d0011a770 0x7f8d0011ef30 secure :-1 s=THROTTLE_DONE pgs=535 cs=0 l=1 rx=0x7f8ce800c280 tx=0x7f8ce8010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:13.823 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:13.799+0000 7f8cf77fe700 1 --2- 172.21.15.42:0/1998856607 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8ce403ad30 0x7f8ce403d240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:14:13.823 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:13.799+0000 7f8cf77fe700 1 -- 172.21.15.42:0/1998856607 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6719+0+0 (secure 0 0 0) 0x7f8ce804b760 con 0x7f8d0011a770 2019-12-05T05:14:13.823 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:13.799+0000 7f8cfed9d700 1 --2- 172.21.15.42:0/1998856607 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8ce403ad30 0x7f8ce403d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:14:13.823 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:13.799+0000 7f8cfed9d700 1 --2- 172.21.15.42:0/1998856607 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8ce403ad30 0x7f8ce403d240 crc :-1 s=READY pgs=246 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:14:13.823 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:13.799+0000 7f8d04ab6700 1 -- 172.21.15.42:0/1998856607 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d0005d110 con 0x7f8d0011a770 2019-12-05T05:14:13.824 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:13.803+0000 7f8cf7fff700 1 --2- 172.21.15.42:0/1998856607 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8d0011a770 0x7f8d0011ef30 secure :-1 s=THROTTLE_DONE pgs=535 cs=0 l=1 rx=0x7f8ce800c280 tx=0x7f8ce8010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:13.824 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:13.803+0000 7f8cf77fe700 1 -- 172.21.15.42:0/1998856607 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f8ce80327c0 con 0x7f8d0011a770 2019-12-05T05:14:13.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:13.967+0000 7f8d04ab6700 1 -- 172.21.15.42:0/1998856607 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8d0005b8c0 con 0x7f8d0011a770 2019-12-05T05:14:13.976 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:13.971+0000 7f8cf7fff700 1 --2- 172.21.15.42:0/1998856607 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8d0011a770 0x7f8d0011ef30 secure :-1 s=THROTTLE_DONE pgs=535 cs=0 l=1 rx=0x7f8ce800c280 tx=0x7f8ce8010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:13.976 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:13.971+0000 7f8cf77fe700 1 -- 172.21.15.42:0/1998856607 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 69 v69) v1 ==== 94+0+1771 (secure 0 0 0) 0x7f8ce8032a20 con 0x7f8d0011a770 2019-12-05T05:14:13.978 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 69 2019-12-05T05:14:13.978 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:14:13.978 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":69,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":69,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:14:13.366240+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4841},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4841":{"gid":4841,"name":"b","rank":0,"incarnation":69,"state":"up:replay","state_seq":2,"addr":"172.21.15.40:6833/3150831963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3150831963},{"type":"v1","addr":"172.21.15.40:6833","nonce":3150831963}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:14:13.981 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:13.975+0000 7f8d04ab6700 1 -- 172.21.15.42:0/1998856607 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8ce403ad30 msgr2=0x7f8ce403d240 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:14:13.981 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:13.975+0000 7f8d04ab6700 1 --2- 172.21.15.42:0/1998856607 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8ce403ad30 0x7f8ce403d240 crc :-1 s=READY pgs=246 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:14:13.981 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:13.975+0000 7f8d04ab6700 1 -- 172.21.15.42:0/1998856607 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8d0011a770 msgr2=0x7f8d0011ef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:14:13.981 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:13.975+0000 7f8d04ab6700 1 --2- 172.21.15.42:0/1998856607 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8d0011a770 0x7f8d0011ef30 secure :-1 s=READY pgs=535 cs=0 l=1 rx=0x7f8ce800c280 tx=0x7f8ce8010b60).stop 2019-12-05T05:14:13.982 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:13.975+0000 7f8d04ab6700 1 -- 172.21.15.42:0/1998856607 shutdown_connections 2019-12-05T05:14:13.982 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:13.975+0000 7f8d04ab6700 1 --2- 172.21.15.42:0/1998856607 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8ce403ad30 0x7f8ce403d240 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:14:13.982 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:13.975+0000 7f8d04ab6700 1 --2- 172.21.15.42:0/1998856607 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8d0011d940 0x7f8d0011dd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:14:13.982 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:13.975+0000 7f8d04ab6700 1 --2- 172.21.15.42:0/1998856607 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8d0011a770 0x7f8d0011ef30 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:14:13.983 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:13.975+0000 7f8d04ab6700 1 -- 172.21.15.42:0/1998856607 shutdown_connections 2019-12-05T05:14:13.983 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:13.975+0000 7f8d04ab6700 1 -- 172.21.15.42:0/1998856607 wait complete. 2019-12-05T05:14:13.983 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:13.975+0000 7f8d04ab6700 1 -- 172.21.15.42:0/1998856607 >> 172.21.15.42:0/1998856607 conn(0x7f8d000aabe0 msgr2=0x7f8d0007e450 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:14:13.991 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a down, removed from mdsmap 2019-12-05T05:14:13.991 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2019-12-05T05:14:13.992 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:14:14.093 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:14.079+0000 7f8623cca700 1 Processor -- start 2019-12-05T05:14:14.093 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:14.079+0000 7f8623cca700 1 -- start start 2019-12-05T05:14:14.093 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:14.079+0000 7f8623cca700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f861c0a7740 0x7f861c08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:14:14.094 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:14.079+0000 7f8623cca700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f861c09fae0 0x7f861c07eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:14:14.094 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:14.079+0000 7f8623cca700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f861c0a0710 con 0x7f861c07e720 2019-12-05T05:14:14.094 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:14.079+0000 7f8623cca700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f861c0a0b00 con 0x7f861c0a7740 2019-12-05T05:14:14.094 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:14.079+0000 7f8623cca700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f861c08e5b0 con 0x7f861c09fae0 2019-12-05T05:14:14.094 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:14.079+0000 7f8622267700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f861c0a7740 0x7f861c08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:14:14.095 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:14.079+0000 7f8622267700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f861c0a7740 0x7f861c08df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:58472/0 (socket says 172.21.15.42:58472) 2019-12-05T05:14:14.095 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:14.079+0000 7f8622267700 1 -- 172.21.15.42:0/1582176731 learned_addr learned my addr 172.21.15.42:0/1582176731 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:14:14.095 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:14.079+0000 7f8622a68700 1 --2- 172.21.15.42:0/1582176731 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f861c09fae0 0x7f861c07eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:14:14.095 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:14.079+0000 7f8622267700 1 -- 172.21.15.42:0/1582176731 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f861c09fae0 msgr2=0x7f861c07eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:14:14.095 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:14.079+0000 7f8622267700 1 --2- 172.21.15.42:0/1582176731 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f861c09fae0 0x7f861c07eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:14:14.095 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:14.079+0000 7f8622267700 1 -- 172.21.15.42:0/1582176731 >> v1:172.21.15.40:6789/0 conn(0x7f861c07e720 legacy=0x7f861c09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:14:14.096 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:14.079+0000 7f8622267700 1 -- 172.21.15.42:0/1582176731 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f861c07f0a0 con 0x7f861c0a7740 2019-12-05T05:14:14.096 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:14.083+0000 7f8622a68700 1 --2- 172.21.15.42:0/1582176731 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f861c09fae0 0x7f861c07eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T05:14:14.096 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:14.083+0000 7f8622267700 1 --2- 172.21.15.42:0/1582176731 >> v2:172.21.15.42:3300/0 conn(0x7f861c0a7740 0x7f861c08df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f8618008790 tx=0x7f861800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:14.096 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:14.083+0000 7f8622267700 1 --2- 172.21.15.42:0/1582176731 >> v2:172.21.15.42:3300/0 conn(0x7f861c0a7740 0x7f861c08df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f8618008790 tx=0x7f861800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:14.096 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:14.083+0000 7f8622267700 1 --2- 172.21.15.42:0/1582176731 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f861c0a7740 0x7f861c08df80 secure :-1 s=READY pgs=536 cs=0 l=1 rx=0x7f8618008790 tx=0x7f861800fd30).ready entity=mon.1 client_cookie=55dcf4a3e3100214 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:14:14.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:14.083+0000 7f8622267700 1 --2- 172.21.15.42:0/1582176731 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f861c0a7740 0x7f861c08df80 secure :-1 s=THROTTLE_DONE pgs=536 cs=0 l=1 rx=0x7f8618008790 tx=0x7f861800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:14.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:14.083+0000 7f8622267700 1 --2- 172.21.15.42:0/1582176731 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f861c0a7740 0x7f861c08df80 secure :-1 s=THROTTLE_DONE pgs=536 cs=0 l=1 rx=0x7f8618008790 tx=0x7f861800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:14.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:14.083+0000 7f8621265700 1 -- 172.21.15.42:0/1582176731 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8618008c90 con 0x7f861c0a7740 2019-12-05T05:14:14.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:14.083+0000 7f8622267700 1 --2- 172.21.15.42:0/1582176731 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f861c0a7740 0x7f861c08df80 secure :-1 s=THROTTLE_DONE pgs=536 cs=0 l=1 rx=0x7f8618008790 tx=0x7f861800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:14.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:14.083+0000 7f8621265700 1 -- 172.21.15.42:0/1582176731 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86180088b0 con 0x7f861c0a7740 2019-12-05T05:14:14.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:14.083+0000 7f8621265700 1 -- 172.21.15.42:0/1582176731 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8618011b10 con 0x7f861c0a7740 2019-12-05T05:14:14.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:14.083+0000 7f8623cca700 1 -- 172.21.15.42:0/1582176731 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f861c0a7740 msgr2=0x7f861c08df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:14:14.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:14.083+0000 7f8623cca700 1 --2- 172.21.15.42:0/1582176731 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f861c0a7740 0x7f861c08df80 secure :-1 s=READY pgs=536 cs=0 l=1 rx=0x7f8618008790 tx=0x7f861800fd30).stop 2019-12-05T05:14:14.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:14.083+0000 7f8623cca700 1 -- 172.21.15.42:0/1582176731 shutdown_connections 2019-12-05T05:14:14.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:14.083+0000 7f8623cca700 1 --2- 172.21.15.42:0/1582176731 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f861c09fae0 0x7f861c07eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:14:14.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:14.083+0000 7f8623cca700 1 --2- 172.21.15.42:0/1582176731 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f861c0a7740 0x7f861c08df80 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:14:14.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:14.083+0000 7f8623cca700 1 -- 172.21.15.42:0/1582176731 shutdown_connections 2019-12-05T05:14:14.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:14.083+0000 7f8623cca700 1 -- 172.21.15.42:0/1582176731 wait complete. 2019-12-05T05:14:14.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:14.083+0000 7f8623cca700 1 -- 172.21.15.42:0/1582176731 >> 172.21.15.42:0/1582176731 conn(0x7f861c0aabe0 msgr2=0x7f861c0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:14:14.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:14.087+0000 7f8623cca700 1 Processor -- start 2019-12-05T05:14:14.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:14.087+0000 7f8623cca700 1 -- start start 2019-12-05T05:14:14.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:14.087+0000 7f8623cca700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f861c12a800 0x7f861c12ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:14:14.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:14.087+0000 7f8623cca700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f861c126b80 con 0x7f861c11a060 2019-12-05T05:14:14.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:14.087+0000 7f8623cca700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f861c064470 con 0x7f861c064040 2019-12-05T05:14:14.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:14.087+0000 7f8623cca700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f861c0633c0 con 0x7f861c12a800 2019-12-05T05:14:14.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:14.087+0000 7f8621a66700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f861c12a800 0x7f861c12ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:14:14.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:14.087+0000 7f8622a68700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f861c064040 0x7f861c131590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:58524/0 (socket says 172.21.15.42:58524) 2019-12-05T05:14:14.103 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:14.087+0000 7f8621a66700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f861c12a800 0x7f861c12ac30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:58484/0 (socket says 172.21.15.42:58484) 2019-12-05T05:14:14.103 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:14.087+0000 7f8622267700 1 --1- 172.21.15.42:0/2603281010 >> v1:172.21.15.40:6789/0 conn(0x7f861c11a060 0x7f861c07c5b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:39728/0 (socket says 172.21.15.42:39728) 2019-12-05T05:14:14.103 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:14.087+0000 7f8622a68700 1 -- 172.21.15.42:0/2603281010 learned_addr learned my addr 172.21.15.42:0/2603281010 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:14:14.103 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:14.087+0000 7f8612ffd700 1 -- 172.21.15.42:0/2603281010 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 597816144 0 0) 0x7f861c064470 con 0x7f861c064040 2019-12-05T05:14:14.103 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:14.087+0000 7f8612ffd700 1 -- 172.21.15.42:0/2603281010 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86040030b0 con 0x7f861c064040 2019-12-05T05:14:14.103 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:14.087+0000 7f8612ffd700 1 -- 172.21.15.42:0/2603281010 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2888193782 0 0) 0x7f861c126b80 con 0x7f861c11a060 2019-12-05T05:14:14.103 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:14.087+0000 7f8612ffd700 1 -- 172.21.15.42:0/2603281010 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f861c064470 con 0x7f861c11a060 2019-12-05T05:14:14.104 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:14.087+0000 7f8612ffd700 1 -- 172.21.15.42:0/2603281010 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 2551105796 0 0) 0x7f86040030b0 con 0x7f861c064040 2019-12-05T05:14:14.104 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:14.087+0000 7f8621a66700 1 -- 172.21.15.42:0/2603281010 >> v1:172.21.15.42:6789/0 conn(0x7f861c064040 legacy=0x7f861c131590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:14:14.104 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:14.087+0000 7f8621a66700 1 -- 172.21.15.42:0/2603281010 >> v1:172.21.15.40:6789/0 conn(0x7f861c11a060 legacy=0x7f861c07c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:14:14.104 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:14.087+0000 7f8621a66700 1 -- 172.21.15.42:0/2603281010 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f861c0646a0 con 0x7f861c12a800 2019-12-05T05:14:14.104 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:14.091+0000 7f8621a66700 1 --2- 172.21.15.42:0/2603281010 >> v2:172.21.15.42:3300/0 conn(0x7f861c12a800 0x7f861c12ac30 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f860c001430 tx=0x7f86140032f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:14.104 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:14.091+0000 7f8621a66700 1 --2- 172.21.15.42:0/2603281010 >> v2:172.21.15.42:3300/0 conn(0x7f861c12a800 0x7f861c12ac30 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f860c001430 tx=0x7f86140032f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:14.104 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:14.091+0000 7f8621a66700 1 --2- 172.21.15.42:0/2603281010 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f861c12a800 0x7f861c12ac30 secure :-1 s=READY pgs=538 cs=0 l=1 rx=0x7f860c001430 tx=0x7f86140032f0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:14:14.105 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:14.091+0000 7f8621a66700 1 --2- 172.21.15.42:0/2603281010 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f861c12a800 0x7f861c12ac30 secure :-1 s=THROTTLE_DONE pgs=538 cs=0 l=1 rx=0x7f860c001430 tx=0x7f86140032f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:14.105 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:14.091+0000 7f8612ffd700 1 -- 172.21.15.42:0/2603281010 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8614001890 con 0x7f861c12a800 2019-12-05T05:14:14.105 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:14.091+0000 7f8621a66700 1 --2- 172.21.15.42:0/2603281010 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f861c12a800 0x7f861c12ac30 secure :-1 s=THROTTLE_DONE pgs=538 cs=0 l=1 rx=0x7f860c001430 tx=0x7f86140032f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:14.105 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:14.091+0000 7f8612ffd700 1 -- 172.21.15.42:0/2603281010 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86140016a0 con 0x7f861c12a800 2019-12-05T05:14:14.105 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:14.091+0000 7f8621a66700 1 --2- 172.21.15.42:0/2603281010 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f861c12a800 0x7f861c12ac30 secure :-1 s=THROTTLE_DONE pgs=538 cs=0 l=1 rx=0x7f860c001430 tx=0x7f86140032f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:14.105 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:14.091+0000 7f8612ffd700 1 -- 172.21.15.42:0/2603281010 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f860c00cb50 con 0x7f861c12a800 2019-12-05T05:14:14.105 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:14.091+0000 7f8623cca700 1 -- 172.21.15.42:0/2603281010 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f861c132da0 con 0x7f861c12a800 2019-12-05T05:14:14.106 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:14.091+0000 7f8623cca700 1 -- 172.21.15.42:0/2603281010 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f861c133350 con 0x7f861c12a800 2019-12-05T05:14:14.106 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:14.091+0000 7f8621a66700 1 --2- 172.21.15.42:0/2603281010 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f861c12a800 0x7f861c12ac30 secure :-1 s=THROTTLE_DONE pgs=538 cs=0 l=1 rx=0x7f860c001430 tx=0x7f86140032f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:14.106 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:14.091+0000 7f8612ffd700 1 -- 172.21.15.42:0/2603281010 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f860c006360 con 0x7f861c12a800 2019-12-05T05:14:14.106 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:14.091+0000 7f8621a66700 1 --2- 172.21.15.42:0/2603281010 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f861c12a800 0x7f861c12ac30 secure :-1 s=THROTTLE_DONE pgs=538 cs=0 l=1 rx=0x7f860c001430 tx=0x7f86140032f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:14.107 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:14.091+0000 7f8612ffd700 1 --2- 172.21.15.42:0/2603281010 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f860403b440 0x7f860403d950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:14:14.107 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:14.091+0000 7f8612ffd700 1 -- 172.21.15.42:0/2603281010 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6719+0+0 (secure 0 0 0) 0x7f860c04ab40 con 0x7f861c12a800 2019-12-05T05:14:14.107 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:14.091+0000 7f8622267700 1 --2- 172.21.15.42:0/2603281010 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f860403b440 0x7f860403d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:14:14.107 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:14.091+0000 7f8622267700 1 --2- 172.21.15.42:0/2603281010 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f860403b440 0x7f860403d950 crc :-1 s=READY pgs=247 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:14:14.107 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:14.095+0000 7f8623cca700 1 -- 172.21.15.42:0/2603281010 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f861c05d110 con 0x7f861c12a800 2019-12-05T05:14:14.108 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:14.095+0000 7f8621a66700 1 --2- 172.21.15.42:0/2603281010 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f861c12a800 0x7f861c12ac30 secure :-1 s=THROTTLE_DONE pgs=538 cs=0 l=1 rx=0x7f860c001430 tx=0x7f86140032f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:14.108 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:14.095+0000 7f8612ffd700 1 -- 172.21.15.42:0/2603281010 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f860c031ba0 con 0x7f861c12a800 2019-12-05T05:14:14.268 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:14.263+0000 7f8623cca700 1 -- 172.21.15.42:0/2603281010 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f861c05b8c0 con 0x7f861c12a800 2019-12-05T05:14:14.269 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:14.263+0000 7f8621a66700 1 --2- 172.21.15.42:0/2603281010 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f861c12a800 0x7f861c12ac30 secure :-1 s=THROTTLE_DONE pgs=538 cs=0 l=1 rx=0x7f860c001430 tx=0x7f86140032f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:14.269 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:14.263+0000 7f8612ffd700 1 -- 172.21.15.42:0/2603281010 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 69 v69) v1 ==== 94+0+1771 (secure 0 0 0) 0x7f860c031e00 con 0x7f861c12a800 2019-12-05T05:14:14.272 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 69 2019-12-05T05:14:14.272 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:14:14.272 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":69,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":69,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:14:13.366240+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4841},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4841":{"gid":4841,"name":"b","rank":0,"incarnation":69,"state":"up:replay","state_seq":2,"addr":"172.21.15.40:6833/3150831963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3150831963},{"type":"v1","addr":"172.21.15.40:6833","nonce":3150831963}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:14:14.276 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:14.271+0000 7f8623cca700 1 -- 172.21.15.42:0/2603281010 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f860403b440 msgr2=0x7f860403d950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:14:14.276 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:14.271+0000 7f8623cca700 1 --2- 172.21.15.42:0/2603281010 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f860403b440 0x7f860403d950 crc :-1 s=READY pgs=247 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:14:14.276 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:14.271+0000 7f8623cca700 1 -- 172.21.15.42:0/2603281010 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f861c12a800 msgr2=0x7f861c12ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:14:14.276 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:14.271+0000 7f8623cca700 1 --2- 172.21.15.42:0/2603281010 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f861c12a800 0x7f861c12ac30 secure :-1 s=READY pgs=538 cs=0 l=1 rx=0x7f860c001430 tx=0x7f86140032f0).stop 2019-12-05T05:14:14.276 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:14.271+0000 7f8623cca700 1 -- 172.21.15.42:0/2603281010 shutdown_connections 2019-12-05T05:14:14.277 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:14.271+0000 7f8623cca700 1 --2- 172.21.15.42:0/2603281010 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f860403b440 0x7f860403d950 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:14:14.277 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:14.271+0000 7f8623cca700 1 --2- 172.21.15.42:0/2603281010 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f861c12a800 0x7f861c12ac30 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:14:14.277 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:14.271+0000 7f8623cca700 1 -- 172.21.15.42:0/2603281010 shutdown_connections 2019-12-05T05:14:14.277 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:14.271+0000 7f8623cca700 1 -- 172.21.15.42:0/2603281010 wait complete. 2019-12-05T05:14:14.277 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:14.271+0000 7f8623cca700 1 -- 172.21.15.42:0/2603281010 >> 172.21.15.42:0/2603281010 conn(0x7f861c0a9720 msgr2=0x7f861c059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:14:14.289 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 69, "feature_flags": { "enable_multiple": false, "ever_enabled_multiple": false }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2019-12-05T05:03:35.604738+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 69, "ever_allowed_features": 2, "explicitly_allowed_features": 2, "failed": [], "flags": 18, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_4841": { "addr": "172.21.15.40:6833/3150831963", "addrs": { "addrvec": [ { "addr": "172.21.15.40:6832", "nonce": 3150831963, "type": "v2" }, { "addr": "172.21.15.40:6833", "nonce": 3150831963, "type": "v1" } ] }, "export_targets": [], "features": 4540138292836696063, "flags": 0, "gid": 4841, "incarnation": 69, "name": "b", "rank": 0, "state": "up:replay", "state_seq": 2 } }, "last_failure": 0, "last_failure_osd_epoch": 57, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 2, "min_compat_client": "0 (unknown)", "modified": "2019-12-05T05:14:13.366240+0000", "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4841 } } } ], "standbys": [] } 2019-12-05T05:14:16.290 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:14:16.396 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:16.383+0000 7f34274d1700 1 Processor -- start 2019-12-05T05:14:16.397 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:16.383+0000 7f34274d1700 1 -- start start 2019-12-05T05:14:16.397 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:16.387+0000 7f34274d1700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f342008df80 0x7f342009f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:14:16.397 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:16.387+0000 7f34274d1700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f342009f830 0x7f342014efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:14:16.397 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:16.387+0000 7f34274d1700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f342009fc60 con 0x7f34200a7740 2019-12-05T05:14:16.397 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:16.387+0000 7f34274d1700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34200a0b00 con 0x7f342008df80 2019-12-05T05:14:16.397 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:16.387+0000 7f34274d1700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f342008e5b0 con 0x7f342009f830 2019-12-05T05:14:16.398 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:16.387+0000 7f342526d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f342008df80 0x7f342009f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:14:16.398 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:16.387+0000 7f342526d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f342008df80 0x7f342009f2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:58500/0 (socket says 172.21.15.42:58500) 2019-12-05T05:14:16.398 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:16.387+0000 7f342526d700 1 -- 172.21.15.42:0/1018195376 learned_addr learned my addr 172.21.15.42:0/1018195376 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:14:16.398 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:16.387+0000 7f342626f700 1 --2- 172.21.15.42:0/1018195376 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f342009f830 0x7f342014efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:14:16.398 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:16.387+0000 7f342526d700 1 -- 172.21.15.42:0/1018195376 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f342009f830 msgr2=0x7f342014efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:14:16.398 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:16.387+0000 7f342526d700 1 --2- 172.21.15.42:0/1018195376 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f342009f830 0x7f342014efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:14:16.398 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:16.387+0000 7f342526d700 1 -- 172.21.15.42:0/1018195376 >> v1:172.21.15.42:6789/0 conn(0x7f34200a7740 legacy=0x7f342007e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:14:16.399 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:16.387+0000 7f342526d700 1 -- 172.21.15.42:0/1018195376 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34200a0710 con 0x7f342008df80 2019-12-05T05:14:16.399 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:16.387+0000 7f342526d700 1 --2- 172.21.15.42:0/1018195376 >> v2:172.21.15.42:3300/0 conn(0x7f342008df80 0x7f342009f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f341c008790 tx=0x7f341c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:16.399 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:16.387+0000 7f342526d700 1 --2- 172.21.15.42:0/1018195376 >> v2:172.21.15.42:3300/0 conn(0x7f342008df80 0x7f342009f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f341c008790 tx=0x7f341c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:16.399 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:16.387+0000 7f342526d700 1 --2- 172.21.15.42:0/1018195376 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f342008df80 0x7f342009f2e0 secure :-1 s=READY pgs=542 cs=0 l=1 rx=0x7f341c008790 tx=0x7f341c00fd30).ready entity=mon.1 client_cookie=323fa62118699f61 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:14:16.399 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:16.387+0000 7f342526d700 1 --2- 172.21.15.42:0/1018195376 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f342008df80 0x7f342009f2e0 secure :-1 s=THROTTLE_DONE pgs=542 cs=0 l=1 rx=0x7f341c008790 tx=0x7f341c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:16.399 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:16.387+0000 7f342526d700 1 --2- 172.21.15.42:0/1018195376 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f342008df80 0x7f342009f2e0 secure :-1 s=THROTTLE_DONE pgs=542 cs=0 l=1 rx=0x7f341c008790 tx=0x7f341c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:16.399 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:16.387+0000 7f3424a6c700 1 -- 172.21.15.42:0/1018195376 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f341c008c90 con 0x7f342008df80 2019-12-05T05:14:16.400 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:16.387+0000 7f342526d700 1 --2- 172.21.15.42:0/1018195376 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f342008df80 0x7f342009f2e0 secure :-1 s=THROTTLE_DONE pgs=542 cs=0 l=1 rx=0x7f341c008790 tx=0x7f341c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:16.400 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:16.387+0000 7f3424a6c700 1 -- 172.21.15.42:0/1018195376 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f341c0088b0 con 0x7f342008df80 2019-12-05T05:14:16.400 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:16.387+0000 7f3424a6c700 1 -- 172.21.15.42:0/1018195376 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f341c011b10 con 0x7f342008df80 2019-12-05T05:14:16.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:16.387+0000 7f34274d1700 1 -- 172.21.15.42:0/1018195376 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f342008df80 msgr2=0x7f342009f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:14:16.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:16.387+0000 7f34274d1700 1 --2- 172.21.15.42:0/1018195376 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f342008df80 0x7f342009f2e0 secure :-1 s=READY pgs=542 cs=0 l=1 rx=0x7f341c008790 tx=0x7f341c00fd30).stop 2019-12-05T05:14:16.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:16.387+0000 7f34274d1700 1 -- 172.21.15.42:0/1018195376 shutdown_connections 2019-12-05T05:14:16.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:16.387+0000 7f34274d1700 1 --2- 172.21.15.42:0/1018195376 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f342009f830 0x7f342014efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:14:16.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:16.387+0000 7f34274d1700 1 --2- 172.21.15.42:0/1018195376 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f342008df80 0x7f342009f2e0 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:14:16.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:16.387+0000 7f34274d1700 1 -- 172.21.15.42:0/1018195376 shutdown_connections 2019-12-05T05:14:16.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:16.387+0000 7f34274d1700 1 -- 172.21.15.42:0/1018195376 wait complete. 2019-12-05T05:14:16.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:16.387+0000 7f34274d1700 1 -- 172.21.15.42:0/1018195376 >> 172.21.15.42:0/1018195376 conn(0x7f34200aabe0 msgr2=0x7f34200a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:14:16.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:16.391+0000 7f34274d1700 1 Processor -- start 2019-12-05T05:14:16.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:16.391+0000 7f34274d1700 1 -- start start 2019-12-05T05:14:16.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:16.391+0000 7f34274d1700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f3420116360 0x7f342012a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:14:16.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:16.391+0000 7f34274d1700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3420126b50 con 0x7f3420126720 2019-12-05T05:14:16.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:16.391+0000 7f34274d1700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3420142a50 con 0x7f3420142620 2019-12-05T05:14:16.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:16.391+0000 7f34274d1700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3420116790 con 0x7f3420116360 2019-12-05T05:14:16.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:16.391+0000 7f342626f700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f3420142620 0x7f3420131130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:58546/0 (socket says 172.21.15.42:58546) 2019-12-05T05:14:16.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:16.391+0000 7f342626f700 1 -- 172.21.15.42:0/3974424766 learned_addr learned my addr 172.21.15.42:0/3974424766 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:14:16.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:16.391+0000 7f3425a6e700 1 --2- 172.21.15.42:0/3974424766 >> v2:172.21.15.40:3300/0 conn(0x7f3420116360 0x7f342012a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:14:16.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:16.391+0000 7f34127fc700 1 -- 172.21.15.42:0/3974424766 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 480549387 0 0) 0x7f3420142a50 con 0x7f3420142620 2019-12-05T05:14:16.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:16.391+0000 7f34127fc700 1 -- 172.21.15.42:0/3974424766 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34080030b0 con 0x7f3420142620 2019-12-05T05:14:16.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:16.391+0000 7f3425a6e700 1 -- 172.21.15.42:0/3974424766 >> v1:172.21.15.42:6789/0 conn(0x7f3420142620 legacy=0x7f3420131130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:14:16.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:16.391+0000 7f3425a6e700 1 -- 172.21.15.42:0/3974424766 >> v1:172.21.15.40:6789/0 conn(0x7f3420126720 legacy=0x7f3420127020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:14:16.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:16.391+0000 7f34127fc700 1 -- 172.21.15.42:0/3974424766 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4118500312 0 0) 0x7f3420126b50 con 0x7f3420126720 2019-12-05T05:14:16.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:16.395+0000 7f3425a6e700 1 -- 172.21.15.42:0/3974424766 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3420142c80 con 0x7f3420116360 2019-12-05T05:14:16.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:16.395+0000 7f3425a6e700 1 --2- 172.21.15.42:0/3974424766 >> v2:172.21.15.40:3300/0 conn(0x7f3420116360 0x7f342012a6f0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f3414001430 tx=0x7f341400c320).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:16.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:16.395+0000 7f3425a6e700 1 --2- 172.21.15.42:0/3974424766 >> v2:172.21.15.40:3300/0 conn(0x7f3420116360 0x7f342012a6f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f3414001430 tx=0x7f341400c320).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:16.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:16.395+0000 7f3425a6e700 1 --2- 172.21.15.42:0/3974424766 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3420116360 0x7f342012a6f0 secure :-1 s=READY pgs=431 cs=0 l=1 rx=0x7f3414001430 tx=0x7f341400c320).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:14:16.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:16.395+0000 7f3425a6e700 1 --2- 172.21.15.42:0/3974424766 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3420116360 0x7f342012a6f0 secure :-1 s=THROTTLE_DONE pgs=431 cs=0 l=1 rx=0x7f3414001430 tx=0x7f341400c320).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:16.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:16.395+0000 7f34127fc700 1 -- 172.21.15.42:0/3974424766 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f341400cb70 con 0x7f3420116360 2019-12-05T05:14:16.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:16.395+0000 7f3425a6e700 1 --2- 172.21.15.42:0/3974424766 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3420116360 0x7f342012a6f0 secure :-1 s=THROTTLE_DONE pgs=431 cs=0 l=1 rx=0x7f3414001430 tx=0x7f341400c320).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:16.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:16.395+0000 7f3425a6e700 1 --2- 172.21.15.42:0/3974424766 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3420116360 0x7f342012a6f0 secure :-1 s=THROTTLE_DONE pgs=431 cs=0 l=1 rx=0x7f3414001430 tx=0x7f341400c320).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:16.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:16.395+0000 7f34127fc700 1 -- 172.21.15.42:0/3974424766 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f341400c790 con 0x7f3420116360 2019-12-05T05:14:16.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:16.395+0000 7f34274d1700 1 -- 172.21.15.42:0/3974424766 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3420132940 con 0x7f3420116360 2019-12-05T05:14:16.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:16.395+0000 7f34127fc700 1 -- 172.21.15.42:0/3974424766 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3414011070 con 0x7f3420116360 2019-12-05T05:14:16.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:16.395+0000 7f34274d1700 1 -- 172.21.15.42:0/3974424766 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3420132ec0 con 0x7f3420116360 2019-12-05T05:14:16.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:16.395+0000 7f3425a6e700 1 --2- 172.21.15.42:0/3974424766 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3420116360 0x7f342012a6f0 secure :-1 s=THROTTLE_DONE pgs=431 cs=0 l=1 rx=0x7f3414001430 tx=0x7f341400c320).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:16.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:16.395+0000 7f34127fc700 1 -- 172.21.15.42:0/3974424766 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f3414011380 con 0x7f3420116360 2019-12-05T05:14:16.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:16.395+0000 7f3425a6e700 1 --2- 172.21.15.42:0/3974424766 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3420116360 0x7f342012a6f0 secure :-1 s=THROTTLE_DONE pgs=431 cs=0 l=1 rx=0x7f3414001430 tx=0x7f341400c320).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:16.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:16.395+0000 7f34127fc700 1 --2- 172.21.15.42:0/3974424766 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f340803b210 0x7f340803d720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:14:16.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:16.395+0000 7f34127fc700 1 -- 172.21.15.42:0/3974424766 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6719+0+0 (secure 0 0 0) 0x7f341404b1e0 con 0x7f3420116360 2019-12-05T05:14:16.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:16.399+0000 7f342526d700 1 --2- 172.21.15.42:0/3974424766 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f340803b210 0x7f340803d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:14:16.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:16.399+0000 7f342526d700 1 --2- 172.21.15.42:0/3974424766 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f340803b210 0x7f340803d720 crc :-1 s=READY pgs=248 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:14:16.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:16.399+0000 7f34274d1700 1 -- 172.21.15.42:0/3974424766 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f342005d110 con 0x7f3420116360 2019-12-05T05:14:16.414 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:16.407+0000 7f3425a6e700 1 --2- 172.21.15.42:0/3974424766 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3420116360 0x7f342012a6f0 secure :-1 s=THROTTLE_DONE pgs=431 cs=0 l=1 rx=0x7f3414001430 tx=0x7f341400c320).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:16.414 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:16.407+0000 7f34127fc700 1 -- 172.21.15.42:0/3974424766 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f3414032260 con 0x7f3420116360 2019-12-05T05:14:16.591 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:16.579+0000 7f34274d1700 1 -- 172.21.15.42:0/3974424766 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f342005b8c0 con 0x7f3420116360 2019-12-05T05:14:16.592 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:16.579+0000 7f3425a6e700 1 --2- 172.21.15.42:0/3974424766 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3420116360 0x7f342012a6f0 secure :-1 s=THROTTLE_DONE pgs=431 cs=0 l=1 rx=0x7f3414001430 tx=0x7f341400c320).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:16.592 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:16.579+0000 7f34127fc700 1 -- 172.21.15.42:0/3974424766 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) v1 ==== 94+0+1775 (secure 0 0 0) 0x7f34140324c0 con 0x7f3420116360 2019-12-05T05:14:16.592 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 70 2019-12-05T05:14:16.592 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:16.583+0000 7f34274d1700 1 -- 172.21.15.42:0/3974424766 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f340803b210 msgr2=0x7f340803d720 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:14:16.592 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:16.583+0000 7f34274d1700 1 --2- 172.21.15.42:0/3974424766 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f340803b210 0x7f340803d720 crc :-1 s=READY pgs=248 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:14:16.592 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:16.583+0000 7f34274d1700 1 -- 172.21.15.42:0/3974424766 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3420116360 msgr2=0x7f342012a6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:14:16.593 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:16.583+0000 7f34274d1700 1 --2- 172.21.15.42:0/3974424766 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3420116360 0x7f342012a6f0 secure :-1 s=READY pgs=431 cs=0 l=1 rx=0x7f3414001430 tx=0x7f341400c320).stop 2019-12-05T05:14:16.593 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:16.583+0000 7f34274d1700 1 -- 172.21.15.42:0/3974424766 shutdown_connections 2019-12-05T05:14:16.593 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:16.583+0000 7f34274d1700 1 --2- 172.21.15.42:0/3974424766 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f340803b210 0x7f340803d720 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:14:16.593 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:16.583+0000 7f34274d1700 1 --2- 172.21.15.42:0/3974424766 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3420116360 0x7f342012a6f0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:14:16.593 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:16.583+0000 7f34274d1700 1 -- 172.21.15.42:0/3974424766 shutdown_connections 2019-12-05T05:14:16.593 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:16.583+0000 7f34274d1700 1 -- 172.21.15.42:0/3974424766 wait complete. 2019-12-05T05:14:16.593 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:16.583+0000 7f34274d1700 1 -- 172.21.15.42:0/3974424766 >> 172.21.15.42:0/3974424766 conn(0x7f342008fde0 msgr2=0x7f34200a97e0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:14:16.595 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:14:16.595 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":70,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:14:15.882311+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4841},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4841":{"gid":4841,"name":"b","rank":0,"incarnation":69,"state":"up:reconnect","state_seq":17,"addr":"172.21.15.40:6833/3150831963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3150831963},{"type":"v1","addr":"172.21.15.40:6833","nonce":3150831963}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:14:16.601 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2019-12-05T05:14:18.605 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:14:18.718 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:18.707+0000 7fd35ebbc700 1 Processor -- start 2019-12-05T05:14:18.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:18.707+0000 7fd35ebbc700 1 -- start start 2019-12-05T05:14:18.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:18.707+0000 7fd35ebbc700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd3580a7730 0x7fd35808df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:14:18.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:18.707+0000 7fd35ebbc700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3580a0700 con 0x7fd35809fad0 2019-12-05T05:14:18.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:18.707+0000 7fd35ebbc700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3580a9b90 con 0x7fd35807e710 2019-12-05T05:14:18.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:18.707+0000 7fd35ebbc700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3580a0af0 con 0x7fd3580a7730 2019-12-05T05:14:18.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:18.707+0000 7fd35d159700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd3580a7730 0x7fd35808df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:14:18.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:18.707+0000 7fd35d159700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd3580a7730 0x7fd35808df70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:58512/0 (socket says 172.21.15.42:58512) 2019-12-05T05:14:18.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:18.707+0000 7fd35c958700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fd35807e710 0x7fd35809f2d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:58552/0 (socket says 172.21.15.42:58552) 2019-12-05T05:14:18.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:18.707+0000 7fd35d159700 1 -- 172.21.15.42:0/3421385809 learned_addr learned my addr 172.21.15.42:0/3421385809 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:14:18.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:18.707+0000 7fd35d159700 1 -- 172.21.15.42:0/3421385809 >> v1:172.21.15.42:6789/0 conn(0x7fd35807e710 legacy=0x7fd35809f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:14:18.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:18.707+0000 7fd34ffff700 1 -- 172.21.15.42:0/3421385809 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 317405062 0 0) 0x7fd3580a0700 con 0x7fd35809fad0 2019-12-05T05:14:18.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:18.707+0000 7fd35d159700 1 -- 172.21.15.42:0/3421385809 >> v1:172.21.15.40:6789/0 conn(0x7fd35809fad0 legacy=0x7fd35814efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:14:18.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:18.707+0000 7fd35d159700 1 -- 172.21.15.42:0/3421385809 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd35807eb40 con 0x7fd3580a7730 2019-12-05T05:14:18.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:18.707+0000 7fd35d159700 1 --2- 172.21.15.42:0/3421385809 >> v2:172.21.15.42:3300/0 conn(0x7fd3580a7730 0x7fd35808df70 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fd354008790 tx=0x7fd35400fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:18.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:18.711+0000 7fd35d159700 1 --2- 172.21.15.42:0/3421385809 >> v2:172.21.15.42:3300/0 conn(0x7fd3580a7730 0x7fd35808df70 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fd354008790 tx=0x7fd35400fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:18.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:18.711+0000 7fd35d159700 1 --2- 172.21.15.42:0/3421385809 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd3580a7730 0x7fd35808df70 secure :-1 s=READY pgs=545 cs=0 l=1 rx=0x7fd354008790 tx=0x7fd35400fcb0).ready entity=mon.1 client_cookie=4b1923d2129a2455 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:14:18.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:18.711+0000 7fd35d159700 1 --2- 172.21.15.42:0/3421385809 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd3580a7730 0x7fd35808df70 secure :-1 s=THROTTLE_DONE pgs=545 cs=0 l=1 rx=0x7fd354008790 tx=0x7fd35400fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:18.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:18.711+0000 7fd35d159700 1 --2- 172.21.15.42:0/3421385809 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd3580a7730 0x7fd35808df70 secure :-1 s=THROTTLE_DONE pgs=545 cs=0 l=1 rx=0x7fd354008790 tx=0x7fd35400fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:18.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:18.711+0000 7fd35d159700 1 --2- 172.21.15.42:0/3421385809 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd3580a7730 0x7fd35808df70 secure :-1 s=THROTTLE_DONE pgs=545 cs=0 l=1 rx=0x7fd354008790 tx=0x7fd35400fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:18.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:18.711+0000 7fd34ffff700 1 -- 172.21.15.42:0/3421385809 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd354011520 con 0x7fd3580a7730 2019-12-05T05:14:18.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:18.711+0000 7fd34ffff700 1 -- 172.21.15.42:0/3421385809 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd354008ca0 con 0x7fd3580a7730 2019-12-05T05:14:18.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:18.711+0000 7fd34ffff700 1 -- 172.21.15.42:0/3421385809 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd354011b00 con 0x7fd3580a7730 2019-12-05T05:14:18.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:18.711+0000 7fd35ebbc700 1 -- 172.21.15.42:0/3421385809 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd3580a7730 msgr2=0x7fd35808df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:14:18.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:18.711+0000 7fd35ebbc700 1 --2- 172.21.15.42:0/3421385809 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd3580a7730 0x7fd35808df70 secure :-1 s=READY pgs=545 cs=0 l=1 rx=0x7fd354008790 tx=0x7fd35400fcb0).stop 2019-12-05T05:14:18.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:18.711+0000 7fd35ebbc700 1 -- 172.21.15.42:0/3421385809 shutdown_connections 2019-12-05T05:14:18.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:18.711+0000 7fd35ebbc700 1 --2- 172.21.15.42:0/3421385809 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd3580a7730 0x7fd35808df70 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:14:18.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:18.711+0000 7fd35ebbc700 1 -- 172.21.15.42:0/3421385809 shutdown_connections 2019-12-05T05:14:18.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:18.711+0000 7fd35ebbc700 1 -- 172.21.15.42:0/3421385809 wait complete. 2019-12-05T05:14:18.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:18.711+0000 7fd35ebbc700 1 -- 172.21.15.42:0/3421385809 >> 172.21.15.42:0/3421385809 conn(0x7fd3580aabd0 msgr2=0x7fd3580a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:14:18.727 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:18.715+0000 7fd35ebbc700 1 Processor -- start 2019-12-05T05:14:18.727 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:18.715+0000 7fd35ebbc700 1 -- start start 2019-12-05T05:14:18.727 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:18.715+0000 7fd35ebbc700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd35811bf80 0x7fd3580a1500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:14:18.727 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:18.715+0000 7fd35ebbc700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd3580a1a50 0x7fd358137670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:14:18.728 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:18.715+0000 7fd35ebbc700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3580a1e80 con 0x7fd3580a2ef0 2019-12-05T05:14:18.728 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:18.715+0000 7fd35ebbc700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3580a29b0 con 0x7fd3580a1a50 2019-12-05T05:14:18.728 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:18.715+0000 7fd35ebbc700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3580a2c50 con 0x7fd35811bf80 2019-12-05T05:14:18.728 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:18.715+0000 7fd35d159700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fd3580a2ef0 0x7fd358130220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:58556/0 (socket says 172.21.15.42:58556) 2019-12-05T05:14:18.728 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:18.715+0000 7fd35d159700 1 -- 172.21.15.42:0/2783446860 learned_addr learned my addr 172.21.15.42:0/2783446860 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:14:18.728 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:18.715+0000 7fd35c958700 1 --2- 172.21.15.42:0/2783446860 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd35811bf80 0x7fd3580a1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:14:18.728 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:18.715+0000 7fd35d95a700 1 --2- 172.21.15.42:0/2783446860 >> v2:172.21.15.40:3300/0 conn(0x7fd3580a1a50 0x7fd358137670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:14:18.729 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:18.715+0000 7fd34dffb700 1 -- 172.21.15.42:0/2783446860 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1126755585 0 0) 0x7fd3580a1e80 con 0x7fd3580a2ef0 2019-12-05T05:14:18.729 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:18.715+0000 7fd34dffb700 1 -- 172.21.15.42:0/2783446860 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3380030b0 con 0x7fd3580a2ef0 2019-12-05T05:14:18.729 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:18.715+0000 7fd34dffb700 1 -- 172.21.15.42:0/2783446860 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 2226750720 0 0) 0x7fd3380030b0 con 0x7fd3580a2ef0 2019-12-05T05:14:18.729 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:18.715+0000 7fd34dffb700 1 -- 172.21.15.42:0/2783446860 --> v1:172.21.15.42:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd3580a1e80 con 0x7fd3580a2ef0 2019-12-05T05:14:18.729 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:18.715+0000 7fd34dffb700 1 -- 172.21.15.42:0/2783446860 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7fd354008470 con 0x7fd3580a2ef0 2019-12-05T05:14:18.729 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:18.715+0000 7fd35d95a700 1 -- 172.21.15.42:0/2783446860 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd35811bf80 msgr2=0x7fd3580a1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:14:18.729 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:18.715+0000 7fd35d95a700 1 --2- 172.21.15.42:0/2783446860 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd35811bf80 0x7fd3580a1500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:14:18.730 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:18.715+0000 7fd35d95a700 1 -- 172.21.15.42:0/2783446860 >> v1:172.21.15.42:6789/0 conn(0x7fd3580a2ef0 legacy=0x7fd358130220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:14:18.730 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:18.715+0000 7fd35d95a700 1 -- 172.21.15.42:0/2783446860 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd358138bd0 con 0x7fd3580a1a50 2019-12-05T05:14:18.730 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:18.715+0000 7fd35c958700 1 --2- 172.21.15.42:0/2783446860 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd35811bf80 0x7fd3580a1500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:14:18.730 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:18.715+0000 7fd35d95a700 1 --2- 172.21.15.42:0/2783446860 >> v2:172.21.15.40:3300/0 conn(0x7fd3580a1a50 0x7fd358137670 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fd3580a2b70 tx=0x7fd3540082d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:18.730 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:18.715+0000 7fd35d95a700 1 --2- 172.21.15.42:0/2783446860 >> v2:172.21.15.40:3300/0 conn(0x7fd3580a1a50 0x7fd358137670 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fd3580a2b70 tx=0x7fd3540082d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:18.730 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:18.715+0000 7fd35d95a700 1 --2- 172.21.15.42:0/2783446860 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd3580a1a50 0x7fd358137670 secure :-1 s=READY pgs=433 cs=0 l=1 rx=0x7fd3580a2b70 tx=0x7fd3540082d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:14:18.730 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:18.719+0000 7fd35d95a700 1 --2- 172.21.15.42:0/2783446860 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd3580a1a50 0x7fd358137670 secure :-1 s=THROTTLE_DONE pgs=433 cs=0 l=1 rx=0x7fd3580a2b70 tx=0x7fd3540082d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:18.731 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:18.719+0000 7fd34dffb700 1 -- 172.21.15.42:0/2783446860 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd350006bb0 con 0x7fd3580a1a50 2019-12-05T05:14:18.731 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:18.719+0000 7fd35d95a700 1 --2- 172.21.15.42:0/2783446860 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd3580a1a50 0x7fd358137670 secure :-1 s=THROTTLE_DONE pgs=433 cs=0 l=1 rx=0x7fd3580a2b70 tx=0x7fd3540082d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:18.731 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:18.719+0000 7fd35ebbc700 1 -- 172.21.15.42:0/2783446860 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd358137bc0 con 0x7fd3580a1a50 2019-12-05T05:14:18.731 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:18.719+0000 7fd34dffb700 1 -- 172.21.15.42:0/2783446860 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3500067d0 con 0x7fd3580a1a50 2019-12-05T05:14:18.731 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:18.719+0000 7fd35d95a700 1 --2- 172.21.15.42:0/2783446860 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd3580a1a50 0x7fd358137670 secure :-1 s=THROTTLE_DONE pgs=433 cs=0 l=1 rx=0x7fd3580a2b70 tx=0x7fd3540082d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:18.731 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:18.719+0000 7fd34dffb700 1 -- 172.21.15.42:0/2783446860 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd350002410 con 0x7fd3580a1a50 2019-12-05T05:14:18.731 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:18.719+0000 7fd35ebbc700 1 -- 172.21.15.42:0/2783446860 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3581380f0 con 0x7fd3580a1a50 2019-12-05T05:14:18.732 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:18.719+0000 7fd35d95a700 1 --2- 172.21.15.42:0/2783446860 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd3580a1a50 0x7fd358137670 secure :-1 s=THROTTLE_DONE pgs=433 cs=0 l=1 rx=0x7fd3580a2b70 tx=0x7fd3540082d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:18.732 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:18.719+0000 7fd34dffb700 1 -- 172.21.15.42:0/2783446860 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fd3500026f0 con 0x7fd3580a1a50 2019-12-05T05:14:18.732 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:18.719+0000 7fd35d95a700 1 --2- 172.21.15.42:0/2783446860 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd3580a1a50 0x7fd358137670 secure :-1 s=THROTTLE_DONE pgs=433 cs=0 l=1 rx=0x7fd3580a2b70 tx=0x7fd3540082d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:18.732 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:18.719+0000 7fd34dffb700 1 --2- 172.21.15.42:0/2783446860 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd33803b5f0 0x7fd33803db00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:14:18.732 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:18.719+0000 7fd34dffb700 1 -- 172.21.15.42:0/2783446860 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6719+0+0 (secure 0 0 0) 0x7fd35004b100 con 0x7fd3580a1a50 2019-12-05T05:14:18.732 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:18.719+0000 7fd35d159700 1 --2- 172.21.15.42:0/2783446860 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd33803b5f0 0x7fd33803db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:14:18.732 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:18.719+0000 7fd35d159700 1 --2- 172.21.15.42:0/2783446860 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd33803b5f0 0x7fd33803db00 crc :-1 s=READY pgs=249 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:14:18.733 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:18.723+0000 7fd35ebbc700 1 -- 172.21.15.42:0/2783446860 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd35805d100 con 0x7fd3580a1a50 2019-12-05T05:14:18.738 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:18.731+0000 7fd35d95a700 1 --2- 172.21.15.42:0/2783446860 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd3580a1a50 0x7fd358137670 secure :-1 s=THROTTLE_DONE pgs=433 cs=0 l=1 rx=0x7fd3580a2b70 tx=0x7fd3540082d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:18.738 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:18.731+0000 7fd34dffb700 1 -- 172.21.15.42:0/2783446860 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fd350032160 con 0x7fd3580a1a50 2019-12-05T05:14:18.909 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:18.903+0000 7fd35ebbc700 1 -- 172.21.15.42:0/2783446860 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd35805b8b0 con 0x7fd3580a1a50 2019-12-05T05:14:18.913 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:18.907+0000 7fd35d95a700 1 --2- 172.21.15.42:0/2783446860 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd3580a1a50 0x7fd358137670 secure :-1 s=THROTTLE_DONE pgs=433 cs=0 l=1 rx=0x7fd3580a2b70 tx=0x7fd3540082d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:18.914 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:18.907+0000 7fd34dffb700 1 -- 172.21.15.42:0/2783446860 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+1772 (secure 0 0 0) 0x7fd3500323c0 con 0x7fd3580a1a50 2019-12-05T05:14:18.916 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:14:18.916 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":72,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:14:17.896674+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4841},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4841":{"gid":4841,"name":"b","rank":0,"incarnation":69,"state":"up:active","state_seq":19,"addr":"172.21.15.40:6833/3150831963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3150831963},{"type":"v1","addr":"172.21.15.40:6833","nonce":3150831963}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:14:18.916 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 72 2019-12-05T05:14:18.919 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:18.911+0000 7fd35ebbc700 1 -- 172.21.15.42:0/2783446860 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd33803b5f0 msgr2=0x7fd33803db00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:14:18.919 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:18.911+0000 7fd35ebbc700 1 --2- 172.21.15.42:0/2783446860 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd33803b5f0 0x7fd33803db00 crc :-1 s=READY pgs=249 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:14:18.919 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:18.911+0000 7fd35ebbc700 1 -- 172.21.15.42:0/2783446860 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd3580a1a50 msgr2=0x7fd358137670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:14:18.919 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:18.911+0000 7fd35ebbc700 1 --2- 172.21.15.42:0/2783446860 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd3580a1a50 0x7fd358137670 secure :-1 s=READY pgs=433 cs=0 l=1 rx=0x7fd3580a2b70 tx=0x7fd3540082d0).stop 2019-12-05T05:14:18.920 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:18.911+0000 7fd35ebbc700 1 -- 172.21.15.42:0/2783446860 shutdown_connections 2019-12-05T05:14:18.920 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:18.911+0000 7fd35ebbc700 1 --2- 172.21.15.42:0/2783446860 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd33803b5f0 0x7fd33803db00 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:14:18.920 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:18.911+0000 7fd35ebbc700 1 --2- 172.21.15.42:0/2783446860 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd3580a1a50 0x7fd358137670 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:14:18.920 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:18.911+0000 7fd35ebbc700 1 --2- 172.21.15.42:0/2783446860 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd35811bf80 0x7fd3580a1500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:14:18.920 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:18.911+0000 7fd35ebbc700 1 -- 172.21.15.42:0/2783446860 shutdown_connections 2019-12-05T05:14:18.920 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:18.911+0000 7fd35ebbc700 1 -- 172.21.15.42:0/2783446860 wait complete. 2019-12-05T05:14:18.920 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:18.911+0000 7fd35ebbc700 1 -- 172.21.15.42:0/2783446860 >> 172.21.15.42:0/2783446860 conn(0x7fd3580a7f80 msgr2=0x7fd35807d860 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:14:18.930 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b has gained rank=0, replacing gid=5505 2019-12-05T05:14:18.930 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 4 secs before reviving mds.a 2019-12-05T05:14:22.939 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.a 2019-12-05T05:14:22.940 INFO:tasks.ceph.mds.a:Restarting daemon 2019-12-05T05:14:22.940 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2019-12-05T05:14:22.945 INFO:tasks.ceph.mds.a:Started 2019-12-05T05:14:22.946 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:14:23.011 INFO:tasks.ceph.mds.a.smithi042.stdout:starting mds.a at 2019-12-05T05:14:23.060 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:23.043+0000 7f6fe1735700 1 Processor -- start 2019-12-05T05:14:23.070 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:23.043+0000 7f6fe1735700 1 -- start start 2019-12-05T05:14:23.070 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:23.043+0000 7f6fe1735700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6fdc0a7740 0x7f6fdc08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:14:23.070 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:23.043+0000 7f6fe1735700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fdc0a0710 con 0x7f6fdc07e720 2019-12-05T05:14:23.070 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:23.043+0000 7f6fe1735700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fdc0a9ba0 con 0x7f6fdc09fae0 2019-12-05T05:14:23.070 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:23.043+0000 7f6fe1735700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fdc0a0b00 con 0x7f6fdc0a7740 2019-12-05T05:14:23.070 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:23.043+0000 7f6fdbfff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f6fdc09fae0 0x7f6fdc14efe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:58578/0 (socket says 172.21.15.42:58578) 2019-12-05T05:14:23.071 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:23.043+0000 7f6fdbfff700 1 -- 172.21.15.42:0/3258333767 learned_addr learned my addr 172.21.15.42:0/3258333767 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:14:23.071 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:23.043+0000 7f6fdb7fe700 1 --2- 172.21.15.42:0/3258333767 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6fdc0a7740 0x7f6fdc08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:14:23.071 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:23.043+0000 7f6fda7fc700 1 -- 172.21.15.42:0/3258333767 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3037017855 0 0) 0x7f6fdc0a9ba0 con 0x7f6fdc09fae0 2019-12-05T05:14:23.071 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:23.043+0000 7f6fda7fc700 1 -- 172.21.15.42:0/3258333767 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fc80030b0 con 0x7f6fdc09fae0 2019-12-05T05:14:23.071 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:23.043+0000 7f6fda7fc700 1 -- 172.21.15.42:0/3258333767 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2326540998 0 0) 0x7f6fdc0a0710 con 0x7f6fdc07e720 2019-12-05T05:14:23.071 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:23.043+0000 7f6fda7fc700 1 -- 172.21.15.42:0/3258333767 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fdc0a9ba0 con 0x7f6fdc07e720 2019-12-05T05:14:23.071 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:23.043+0000 7f6fdb7fe700 1 -- 172.21.15.42:0/3258333767 >> v1:172.21.15.42:6789/0 conn(0x7f6fdc09fae0 legacy=0x7f6fdc14efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:14:23.072 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:23.043+0000 7f6fdb7fe700 1 -- 172.21.15.42:0/3258333767 >> v1:172.21.15.40:6789/0 conn(0x7f6fdc07e720 legacy=0x7f6fdc09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:14:23.072 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:23.043+0000 7f6fdb7fe700 1 -- 172.21.15.42:0/3258333767 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fdc07eb50 con 0x7f6fdc0a7740 2019-12-05T05:14:23.072 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:23.043+0000 7f6fdb7fe700 1 --2- 172.21.15.42:0/3258333767 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6fdc0a7740 0x7f6fdc08df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f6fcc002f30 tx=0x7f6fcc00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:23.072 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:23.043+0000 7f6fdb7fe700 1 --2- 172.21.15.42:0/3258333767 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6fdc0a7740 0x7f6fdc08df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f6fcc002f30 tx=0x7f6fcc00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:23.072 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:23.043+0000 7f6fdb7fe700 1 --2- 172.21.15.42:0/3258333767 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6fdc0a7740 0x7f6fdc08df80 secure :-1 s=READY pgs=112 cs=0 l=1 rx=0x7f6fcc002f30 tx=0x7f6fcc00fbf0).ready entity=mon.2 client_cookie=20387d0fb853b954 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:14:23.072 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:23.043+0000 7f6fdb7fe700 1 --2- 172.21.15.42:0/3258333767 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6fdc0a7740 0x7f6fdc08df80 secure :-1 s=THROTTLE_DONE pgs=112 cs=0 l=1 rx=0x7f6fcc002f30 tx=0x7f6fcc00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:23.072 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:23.043+0000 7f6fda7fc700 1 -- 172.21.15.42:0/3258333767 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6fcc0112b0 con 0x7f6fdc0a7740 2019-12-05T05:14:23.073 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:23.043+0000 7f6fdb7fe700 1 --2- 172.21.15.42:0/3258333767 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6fdc0a7740 0x7f6fdc08df80 secure :-1 s=THROTTLE_DONE pgs=112 cs=0 l=1 rx=0x7f6fcc002f30 tx=0x7f6fcc00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:23.073 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:23.043+0000 7f6fda7fc700 1 -- 172.21.15.42:0/3258333767 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fcc008bc0 con 0x7f6fdc0a7740 2019-12-05T05:14:23.073 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:23.043+0000 7f6fdb7fe700 1 --2- 172.21.15.42:0/3258333767 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6fdc0a7740 0x7f6fdc08df80 secure :-1 s=THROTTLE_DONE pgs=112 cs=0 l=1 rx=0x7f6fcc002f30 tx=0x7f6fcc00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:23.073 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:23.043+0000 7f6fda7fc700 1 -- 172.21.15.42:0/3258333767 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6fcc011890 con 0x7f6fdc0a7740 2019-12-05T05:14:23.074 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:23.043+0000 7f6fe1735700 1 -- 172.21.15.42:0/3258333767 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6fdc0a7740 msgr2=0x7f6fdc08df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:14:23.074 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:23.043+0000 7f6fe1735700 1 --2- 172.21.15.42:0/3258333767 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6fdc0a7740 0x7f6fdc08df80 secure :-1 s=READY pgs=112 cs=0 l=1 rx=0x7f6fcc002f30 tx=0x7f6fcc00fbf0).stop 2019-12-05T05:14:23.074 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:23.047+0000 7f6fe1735700 1 -- 172.21.15.42:0/3258333767 shutdown_connections 2019-12-05T05:14:23.074 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:23.047+0000 7f6fe1735700 1 --2- 172.21.15.42:0/3258333767 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6fdc0a7740 0x7f6fdc08df80 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:14:23.074 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:23.047+0000 7f6fe1735700 1 -- 172.21.15.42:0/3258333767 shutdown_connections 2019-12-05T05:14:23.075 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:23.047+0000 7f6fe1735700 1 -- 172.21.15.42:0/3258333767 wait complete. 2019-12-05T05:14:23.075 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:23.047+0000 7f6fe1735700 1 -- 172.21.15.42:0/3258333767 >> 172.21.15.42:0/3258333767 conn(0x7f6fdc0aabe0 msgr2=0x7f6fdc0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:14:23.075 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:23.047+0000 7f6fe1735700 1 Processor -- start 2019-12-05T05:14:23.075 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:23.047+0000 7f6fe1735700 1 -- start start 2019-12-05T05:14:23.075 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:23.047+0000 7f6fe1735700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6fdc08f640 0x7f6fdc0a5d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:14:23.075 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:23.047+0000 7f6fe1735700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6fdc0a62b0 0x7f6fdc144620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:14:23.075 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:23.047+0000 7f6fe1735700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fdc144fa0 con 0x7f6fdc144b70 2019-12-05T05:14:23.076 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:23.047+0000 7f6fe1735700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fdc0a66e0 con 0x7f6fdc08f640 2019-12-05T05:14:23.076 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:23.047+0000 7f6fe1735700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fdc1451d0 con 0x7f6fdc0a62b0 2019-12-05T05:14:23.076 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:23.051+0000 7f6fdb7fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6fdc08f640 0x7f6fdc0a5d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:14:23.076 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:23.051+0000 7f6fdb7fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6fdc08f640 0x7f6fdc0a5d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:58544/0 (socket says 172.21.15.42:58544) 2019-12-05T05:14:23.076 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:23.051+0000 7f6fdb7fe700 1 -- 172.21.15.42:0/1692536555 learned_addr learned my addr 172.21.15.42:0/1692536555 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:14:23.076 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:23.051+0000 7f6fdaffd700 1 --2- 172.21.15.42:0/1692536555 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6fdc0a62b0 0x7f6fdc144620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:14:23.076 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:23.051+0000 7f6fdb7fe700 1 -- 172.21.15.42:0/1692536555 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6fdc0a62b0 msgr2=0x7f6fdc144620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:14:23.077 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:23.051+0000 7f6fdb7fe700 1 --2- 172.21.15.42:0/1692536555 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6fdc0a62b0 0x7f6fdc144620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:14:23.077 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:23.051+0000 7f6fdb7fe700 1 -- 172.21.15.42:0/1692536555 >> v1:172.21.15.42:6789/0 conn(0x7f6fdc144b70 legacy=0x7f6fdc07b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:14:23.077 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:23.051+0000 7f6fdb7fe700 1 -- 172.21.15.42:0/1692536555 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fdc07b900 con 0x7f6fdc08f640 2019-12-05T05:14:23.077 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:23.051+0000 7f6fdb7fe700 1 --2- 172.21.15.42:0/1692536555 >> v2:172.21.15.42:3300/0 conn(0x7f6fdc08f640 0x7f6fdc0a5d60 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f6fcc002f30 tx=0x7f6fcc008bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:23.077 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:23.051+0000 7f6fdb7fe700 1 --2- 172.21.15.42:0/1692536555 >> v2:172.21.15.42:3300/0 conn(0x7f6fdc08f640 0x7f6fdc0a5d60 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f6fcc002f30 tx=0x7f6fcc008bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:23.077 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:23.051+0000 7f6fdb7fe700 1 --2- 172.21.15.42:0/1692536555 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6fdc08f640 0x7f6fdc0a5d60 secure :-1 s=READY pgs=553 cs=0 l=1 rx=0x7f6fcc002f30 tx=0x7f6fcc008bc0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:14:23.077 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:23.051+0000 7f6fdb7fe700 1 --2- 172.21.15.42:0/1692536555 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6fdc08f640 0x7f6fdc0a5d60 secure :-1 s=THROTTLE_DONE pgs=553 cs=0 l=1 rx=0x7f6fcc002f30 tx=0x7f6fcc008bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:23.078 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:23.051+0000 7f6fd3fff700 1 -- 172.21.15.42:0/1692536555 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6fcc007150 con 0x7f6fdc08f640 2019-12-05T05:14:23.078 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:23.051+0000 7f6fdb7fe700 1 --2- 172.21.15.42:0/1692536555 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6fdc08f640 0x7f6fdc0a5d60 secure :-1 s=THROTTLE_DONE pgs=553 cs=0 l=1 rx=0x7f6fcc002f30 tx=0x7f6fcc008bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:23.078 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:23.051+0000 7f6fdb7fe700 1 --2- 172.21.15.42:0/1692536555 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6fdc08f640 0x7f6fdc0a5d60 secure :-1 s=THROTTLE_DONE pgs=553 cs=0 l=1 rx=0x7f6fcc002f30 tx=0x7f6fcc008bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:23.078 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:23.051+0000 7f6fd3fff700 1 -- 172.21.15.42:0/1692536555 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fcc0114b0 con 0x7f6fdc08f640 2019-12-05T05:14:23.078 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:23.051+0000 7f6fd3fff700 1 -- 172.21.15.42:0/1692536555 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6fcc00f550 con 0x7f6fdc08f640 2019-12-05T05:14:23.078 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:23.051+0000 7f6fe1735700 1 -- 172.21.15.42:0/1692536555 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fdc07bb70 con 0x7f6fdc08f640 2019-12-05T05:14:23.078 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:23.051+0000 7f6fe1735700 1 -- 172.21.15.42:0/1692536555 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fdc12f340 con 0x7f6fdc08f640 2019-12-05T05:14:23.079 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:23.051+0000 7f6fdb7fe700 1 --2- 172.21.15.42:0/1692536555 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6fdc08f640 0x7f6fdc0a5d60 secure :-1 s=THROTTLE_DONE pgs=553 cs=0 l=1 rx=0x7f6fcc002f30 tx=0x7f6fcc008bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:23.079 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:23.051+0000 7f6fd3fff700 1 -- 172.21.15.42:0/1692536555 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f6fcc005540 con 0x7f6fdc08f640 2019-12-05T05:14:23.079 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:23.051+0000 7f6fdb7fe700 1 --2- 172.21.15.42:0/1692536555 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6fdc08f640 0x7f6fdc0a5d60 secure :-1 s=THROTTLE_DONE pgs=553 cs=0 l=1 rx=0x7f6fcc002f30 tx=0x7f6fcc008bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:23.079 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:23.051+0000 7f6fd3fff700 1 --2- 172.21.15.42:0/1692536555 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6fc003ad60 0x7f6fc003d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:14:23.079 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:23.051+0000 7f6fd3fff700 1 -- 172.21.15.42:0/1692536555 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6719+0+0 (secure 0 0 0) 0x7f6fcc04b710 con 0x7f6fdc08f640 2019-12-05T05:14:23.079 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:23.051+0000 7f6fdaffd700 1 --2- 172.21.15.42:0/1692536555 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6fc003ad60 0x7f6fc003d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:14:23.079 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:23.051+0000 7f6fdaffd700 1 --2- 172.21.15.42:0/1692536555 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6fc003ad60 0x7f6fc003d270 crc :-1 s=READY pgs=250 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:14:23.080 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:23.059+0000 7f6fe1735700 1 -- 172.21.15.42:0/1692536555 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fdc05d110 con 0x7f6fdc08f640 2019-12-05T05:14:23.080 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:23.067+0000 7f6fdb7fe700 1 --2- 172.21.15.42:0/1692536555 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6fdc08f640 0x7f6fdc0a5d60 secure :-1 s=THROTTLE_DONE pgs=553 cs=0 l=1 rx=0x7f6fcc002f30 tx=0x7f6fcc008bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:23.080 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:23.067+0000 7f6fd3fff700 1 -- 172.21.15.42:0/1692536555 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f6fcc032790 con 0x7f6fdc08f640 2019-12-05T05:14:23.245 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:23.235+0000 7f6fe1735700 1 -- 172.21.15.42:0/1692536555 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6fdc05b8c0 con 0x7f6fdc08f640 2019-12-05T05:14:23.246 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:23.235+0000 7f6fdb7fe700 1 --2- 172.21.15.42:0/1692536555 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6fdc08f640 0x7f6fdc0a5d60 secure :-1 s=THROTTLE_DONE pgs=553 cs=0 l=1 rx=0x7f6fcc002f30 tx=0x7f6fcc008bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:23.246 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:23.235+0000 7f6fd3fff700 1 -- 172.21.15.42:0/1692536555 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+1772 (secure 0 0 0) 0x7f6fcc0335a0 con 0x7f6fdc08f640 2019-12-05T05:14:23.246 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 72 2019-12-05T05:14:23.246 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:14:23.246 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":72,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:14:17.896674+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4841},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4841":{"gid":4841,"name":"b","rank":0,"incarnation":69,"state":"up:active","state_seq":19,"addr":"172.21.15.40:6833/3150831963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3150831963},{"type":"v1","addr":"172.21.15.40:6833","nonce":3150831963}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:14:23.247 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:23.239+0000 7f6fe1735700 1 -- 172.21.15.42:0/1692536555 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6fc003ad60 msgr2=0x7f6fc003d270 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:14:23.247 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:23.239+0000 7f6fe1735700 1 --2- 172.21.15.42:0/1692536555 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6fc003ad60 0x7f6fc003d270 crc :-1 s=READY pgs=250 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:14:23.248 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:23.239+0000 7f6fe1735700 1 -- 172.21.15.42:0/1692536555 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6fdc08f640 msgr2=0x7f6fdc0a5d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:14:23.248 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:23.239+0000 7f6fe1735700 1 --2- 172.21.15.42:0/1692536555 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6fdc08f640 0x7f6fdc0a5d60 secure :-1 s=READY pgs=553 cs=0 l=1 rx=0x7f6fcc002f30 tx=0x7f6fcc008bc0).stop 2019-12-05T05:14:23.248 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:23.239+0000 7f6fe1735700 1 -- 172.21.15.42:0/1692536555 shutdown_connections 2019-12-05T05:14:23.248 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:23.239+0000 7f6fe1735700 1 --2- 172.21.15.42:0/1692536555 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6fc003ad60 0x7f6fc003d270 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:14:23.248 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:23.239+0000 7f6fe1735700 1 --2- 172.21.15.42:0/1692536555 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6fdc08f640 0x7f6fdc0a5d60 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:14:23.248 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:23.239+0000 7f6fe1735700 1 --2- 172.21.15.42:0/1692536555 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6fdc0a62b0 0x7f6fdc144620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:14:23.248 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:23.239+0000 7f6fe1735700 1 -- 172.21.15.42:0/1692536555 shutdown_connections 2019-12-05T05:14:23.249 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:23.239+0000 7f6fe1735700 1 -- 172.21.15.42:0/1692536555 wait complete. 2019-12-05T05:14:23.249 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:23.239+0000 7f6fe1735700 1 -- 172.21.15.42:0/1692536555 >> 172.21.15.42:0/1692536555 conn(0x7f6fdc08fde0 msgr2=0x7f6fdc07db50 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:14:23.261 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2019-12-05T05:14:25.262 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:14:25.370 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:25.355+0000 7fd0597c2700 1 Processor -- start 2019-12-05T05:14:25.370 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:25.355+0000 7fd0597c2700 1 -- start start 2019-12-05T05:14:25.370 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:25.359+0000 7fd0597c2700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd0540a7740 0x7fd05408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:14:25.370 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:25.359+0000 7fd0597c2700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd05409fae0 0x7fd05407eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:14:25.371 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:25.359+0000 7fd0597c2700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0540a0710 con 0x7fd05407e720 2019-12-05T05:14:25.371 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:25.359+0000 7fd0597c2700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0540a0b00 con 0x7fd05409fae0 2019-12-05T05:14:25.371 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:25.359+0000 7fd0597c2700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd05408e5b0 con 0x7fd0540a7740 2019-12-05T05:14:25.371 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:25.359+0000 7fd053fff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd05409fae0 0x7fd05407eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:14:25.372 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:25.359+0000 7fd052ffd700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fd05407e720 0x7fd05409f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:58592/0 (socket says 172.21.15.42:58592) 2019-12-05T05:14:25.373 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:25.359+0000 7fd052ffd700 1 -- 172.21.15.42:0/3458300835 learned_addr learned my addr 172.21.15.42:0/3458300835 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:14:25.373 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:25.359+0000 7fd0537fe700 1 --2- 172.21.15.42:0/3458300835 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd0540a7740 0x7fd05408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:14:25.374 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:25.359+0000 7fd0537fe700 1 -- 172.21.15.42:0/3458300835 >> v2:172.21.15.42:3300/0 conn(0x7fd05409fae0 msgr2=0x7fd05407eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:14:25.374 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:25.359+0000 7fd0537fe700 1 --2- 172.21.15.42:0/3458300835 >> v2:172.21.15.42:3300/0 conn(0x7fd05409fae0 0x7fd05407eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:14:25.374 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:25.359+0000 7fd0537fe700 1 -- 172.21.15.42:0/3458300835 >> v1:172.21.15.42:6789/0 conn(0x7fd05407e720 legacy=0x7fd05409f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:14:25.374 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:25.359+0000 7fd0537fe700 1 -- 172.21.15.42:0/3458300835 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd05407f0a0 con 0x7fd0540a7740 2019-12-05T05:14:25.374 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:25.359+0000 7fd0537fe700 1 --2- 172.21.15.42:0/3458300835 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd0540a7740 0x7fd05408df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fd044008b10 tx=0x7fd04400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:25.374 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:25.359+0000 7fd0537fe700 1 --2- 172.21.15.42:0/3458300835 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd0540a7740 0x7fd05408df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fd044008b10 tx=0x7fd04400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:25.375 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:25.359+0000 7fd0537fe700 1 --2- 172.21.15.42:0/3458300835 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd0540a7740 0x7fd05408df80 secure :-1 s=READY pgs=113 cs=0 l=1 rx=0x7fd044008b10 tx=0x7fd04400fff0).ready entity=mon.2 client_cookie=6987e1fe09ee01ea server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:14:25.375 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:25.359+0000 7fd0537fe700 1 --2- 172.21.15.42:0/3458300835 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd0540a7740 0x7fd05408df80 secure :-1 s=THROTTLE_DONE pgs=113 cs=0 l=1 rx=0x7fd044008b10 tx=0x7fd04400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:25.375 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:25.359+0000 7fd0537fe700 1 --2- 172.21.15.42:0/3458300835 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd0540a7740 0x7fd05408df80 secure :-1 s=THROTTLE_DONE pgs=113 cs=0 l=1 rx=0x7fd044008b10 tx=0x7fd04400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:25.375 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:25.359+0000 7fd0527fc700 1 -- 172.21.15.42:0/3458300835 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd044010900 con 0x7fd0540a7740 2019-12-05T05:14:25.375 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:25.359+0000 7fd0537fe700 1 --2- 172.21.15.42:0/3458300835 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd0540a7740 0x7fd05408df80 secure :-1 s=THROTTLE_DONE pgs=113 cs=0 l=1 rx=0x7fd044008b10 tx=0x7fd04400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:25.375 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:25.359+0000 7fd0527fc700 1 -- 172.21.15.42:0/3458300835 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd044008b50 con 0x7fd0540a7740 2019-12-05T05:14:25.376 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:25.359+0000 7fd0527fc700 1 -- 172.21.15.42:0/3458300835 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd0440110c0 con 0x7fd0540a7740 2019-12-05T05:14:25.376 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:25.359+0000 7fd0597c2700 1 -- 172.21.15.42:0/3458300835 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd0540a7740 msgr2=0x7fd05408df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:14:25.376 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:25.359+0000 7fd0597c2700 1 --2- 172.21.15.42:0/3458300835 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd0540a7740 0x7fd05408df80 secure :-1 s=READY pgs=113 cs=0 l=1 rx=0x7fd044008b10 tx=0x7fd04400fff0).stop 2019-12-05T05:14:25.376 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:25.359+0000 7fd0597c2700 1 -- 172.21.15.42:0/3458300835 shutdown_connections 2019-12-05T05:14:25.376 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:25.359+0000 7fd0597c2700 1 --2- 172.21.15.42:0/3458300835 >> v2:172.21.15.42:3300/0 conn(0x7fd05409fae0 0x7fd05407eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:14:25.377 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:25.359+0000 7fd0597c2700 1 --2- 172.21.15.42:0/3458300835 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd0540a7740 0x7fd05408df80 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:14:25.377 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:25.359+0000 7fd0597c2700 1 -- 172.21.15.42:0/3458300835 shutdown_connections 2019-12-05T05:14:25.377 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:25.359+0000 7fd0597c2700 1 -- 172.21.15.42:0/3458300835 wait complete. 2019-12-05T05:14:25.377 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:25.359+0000 7fd0597c2700 1 -- 172.21.15.42:0/3458300835 >> 172.21.15.42:0/3458300835 conn(0x7fd0540aabe0 msgr2=0x7fd0540a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:14:25.377 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:25.363+0000 7fd0597c2700 1 Processor -- start 2019-12-05T05:14:25.378 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:25.363+0000 7fd0597c2700 1 -- start start 2019-12-05T05:14:25.382 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:25.363+0000 7fd0597c2700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd054127170 0x7fd054114860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:14:25.383 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:25.363+0000 7fd0597c2700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd05411acb0 0x7fd05412e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:14:25.383 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:25.363+0000 7fd0597c2700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd054114db0 con 0x7fd0540a6c30 2019-12-05T05:14:25.383 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:25.363+0000 7fd0597c2700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0541243b0 con 0x7fd05411acb0 2019-12-05T05:14:25.383 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:25.363+0000 7fd0597c2700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd054127680 con 0x7fd054127170 2019-12-05T05:14:25.383 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:25.363+0000 7fd053fff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd05411acb0 0x7fd05412e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:14:25.383 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:25.363+0000 7fd053fff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd05411acb0 0x7fd05412e5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:58560/0 (socket says 172.21.15.42:58560) 2019-12-05T05:14:25.383 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:25.363+0000 7fd053fff700 1 -- 172.21.15.42:0/785697487 learned_addr learned my addr 172.21.15.42:0/785697487 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:14:25.384 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:25.367+0000 7fd0537fe700 1 --2- 172.21.15.42:0/785697487 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd054127170 0x7fd054114860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:14:25.384 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:25.367+0000 7fd053fff700 1 -- 172.21.15.42:0/785697487 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd054127170 msgr2=0x7fd054114860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:14:25.384 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:25.367+0000 7fd053fff700 1 --2- 172.21.15.42:0/785697487 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd054127170 0x7fd054114860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:14:25.384 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:25.367+0000 7fd053fff700 1 -- 172.21.15.42:0/785697487 >> v1:172.21.15.40:6789/0 conn(0x7fd0540a6c30 legacy=0x7fd054123bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:14:25.384 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:25.367+0000 7fd03bfff700 1 -- 172.21.15.42:0/785697487 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4294656265 0 0) 0x7fd054114db0 con 0x7fd0540a6c30 2019-12-05T05:14:25.384 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:25.367+0000 7fd053fff700 1 -- 172.21.15.42:0/785697487 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd05411b0e0 con 0x7fd05411acb0 2019-12-05T05:14:25.385 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:25.367+0000 7fd053fff700 1 --2- 172.21.15.42:0/785697487 >> v2:172.21.15.42:3300/0 conn(0x7fd05411acb0 0x7fd05412e5c0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fd054114fe0 tx=0x7fd04c010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:25.385 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:25.367+0000 7fd053fff700 1 --2- 172.21.15.42:0/785697487 >> v2:172.21.15.42:3300/0 conn(0x7fd05411acb0 0x7fd05412e5c0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fd054114fe0 tx=0x7fd04c010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:25.385 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:25.367+0000 7fd053fff700 1 --2- 172.21.15.42:0/785697487 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd05411acb0 0x7fd05412e5c0 secure :-1 s=READY pgs=555 cs=0 l=1 rx=0x7fd054114fe0 tx=0x7fd04c010ac0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:14:25.385 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:25.367+0000 7fd053fff700 1 --2- 172.21.15.42:0/785697487 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd05411acb0 0x7fd05412e5c0 secure :-1 s=THROTTLE_DONE pgs=555 cs=0 l=1 rx=0x7fd054114fe0 tx=0x7fd04c010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:25.385 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:25.367+0000 7fd03bfff700 1 -- 172.21.15.42:0/785697487 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd04c005540 con 0x7fd05411acb0 2019-12-05T05:14:25.385 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:25.367+0000 7fd053fff700 1 --2- 172.21.15.42:0/785697487 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd05411acb0 0x7fd05412e5c0 secure :-1 s=THROTTLE_DONE pgs=555 cs=0 l=1 rx=0x7fd054114fe0 tx=0x7fd04c010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:25.385 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:25.367+0000 7fd03bfff700 1 -- 172.21.15.42:0/785697487 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd04c007150 con 0x7fd05411acb0 2019-12-05T05:14:25.386 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:25.367+0000 7fd053fff700 1 --2- 172.21.15.42:0/785697487 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd05411acb0 0x7fd05412e5c0 secure :-1 s=THROTTLE_DONE pgs=555 cs=0 l=1 rx=0x7fd054114fe0 tx=0x7fd04c010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:25.386 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:25.367+0000 7fd0597c2700 1 -- 172.21.15.42:0/785697487 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd054127e60 con 0x7fd05411acb0 2019-12-05T05:14:25.386 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:25.367+0000 7fd03bfff700 1 -- 172.21.15.42:0/785697487 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd04c005b20 con 0x7fd05411acb0 2019-12-05T05:14:25.386 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:25.367+0000 7fd0597c2700 1 -- 172.21.15.42:0/785697487 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd05412eb10 con 0x7fd05411acb0 2019-12-05T05:14:25.386 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:25.367+0000 7fd053fff700 1 --2- 172.21.15.42:0/785697487 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd05411acb0 0x7fd05412e5c0 secure :-1 s=THROTTLE_DONE pgs=555 cs=0 l=1 rx=0x7fd054114fe0 tx=0x7fd04c010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:25.386 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:25.367+0000 7fd053fff700 1 --2- 172.21.15.42:0/785697487 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd05411acb0 0x7fd05412e5c0 secure :-1 s=THROTTLE_DONE pgs=555 cs=0 l=1 rx=0x7fd054114fe0 tx=0x7fd04c010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:25.386 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:25.367+0000 7fd03bfff700 1 -- 172.21.15.42:0/785697487 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fd04c012530 con 0x7fd05411acb0 2019-12-05T05:14:25.386 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:25.367+0000 7fd03bfff700 1 --2- 172.21.15.42:0/785697487 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd03403aca0 0x7fd03403d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:14:25.387 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:25.367+0000 7fd03bfff700 1 -- 172.21.15.42:0/785697487 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6719+0+0 (secure 0 0 0) 0x7fd04c04af40 con 0x7fd05411acb0 2019-12-05T05:14:25.387 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:25.367+0000 7fd0537fe700 1 --2- 172.21.15.42:0/785697487 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd03403aca0 0x7fd03403d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:14:25.387 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:25.367+0000 7fd0537fe700 1 --2- 172.21.15.42:0/785697487 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd03403aca0 0x7fd03403d1b0 crc :-1 s=READY pgs=251 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:14:25.387 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:25.371+0000 7fd0597c2700 1 -- 172.21.15.42:0/785697487 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd05405d110 con 0x7fd05411acb0 2019-12-05T05:14:25.388 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:25.379+0000 7fd053fff700 1 --2- 172.21.15.42:0/785697487 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd05411acb0 0x7fd05412e5c0 secure :-1 s=THROTTLE_DONE pgs=555 cs=0 l=1 rx=0x7fd054114fe0 tx=0x7fd04c010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:25.388 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:25.379+0000 7fd03bfff700 1 -- 172.21.15.42:0/785697487 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fd04c031f00 con 0x7fd05411acb0 2019-12-05T05:14:25.553 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:25.547+0000 7fd0597c2700 1 -- 172.21.15.42:0/785697487 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd05405b8c0 con 0x7fd05411acb0 2019-12-05T05:14:25.554 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:25.547+0000 7fd053fff700 1 --2- 172.21.15.42:0/785697487 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd05411acb0 0x7fd05412e5c0 secure :-1 s=THROTTLE_DONE pgs=555 cs=0 l=1 rx=0x7fd054114fe0 tx=0x7fd04c010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:25.554 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:25.547+0000 7fd03bfff700 1 -- 172.21.15.42:0/785697487 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+1772 (secure 0 0 0) 0x7fd04c005dc0 con 0x7fd05411acb0 2019-12-05T05:14:25.554 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 72 2019-12-05T05:14:25.555 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:14:25.555 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":72,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:14:17.896674+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4841},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4841":{"gid":4841,"name":"b","rank":0,"incarnation":69,"state":"up:active","state_seq":19,"addr":"172.21.15.40:6833/3150831963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3150831963},{"type":"v1","addr":"172.21.15.40:6833","nonce":3150831963}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:14:25.558 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:25.551+0000 7fd0597c2700 1 -- 172.21.15.42:0/785697487 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd03403aca0 msgr2=0x7fd03403d1b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:14:25.558 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:25.551+0000 7fd0597c2700 1 --2- 172.21.15.42:0/785697487 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd03403aca0 0x7fd03403d1b0 crc :-1 s=READY pgs=251 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:14:25.558 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:25.551+0000 7fd0597c2700 1 -- 172.21.15.42:0/785697487 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd05411acb0 msgr2=0x7fd05412e5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:14:25.559 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:25.551+0000 7fd0597c2700 1 --2- 172.21.15.42:0/785697487 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd05411acb0 0x7fd05412e5c0 secure :-1 s=READY pgs=555 cs=0 l=1 rx=0x7fd054114fe0 tx=0x7fd04c010ac0).stop 2019-12-05T05:14:25.559 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:25.551+0000 7fd0597c2700 1 -- 172.21.15.42:0/785697487 shutdown_connections 2019-12-05T05:14:25.560 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:25.551+0000 7fd0597c2700 1 --2- 172.21.15.42:0/785697487 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd03403aca0 0x7fd03403d1b0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:14:25.560 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:25.551+0000 7fd0597c2700 1 --2- 172.21.15.42:0/785697487 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd05411acb0 0x7fd05412e5c0 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:14:25.560 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:25.551+0000 7fd0597c2700 1 --2- 172.21.15.42:0/785697487 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd054127170 0x7fd054114860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:14:25.560 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:25.551+0000 7fd0597c2700 1 -- 172.21.15.42:0/785697487 shutdown_connections 2019-12-05T05:14:25.561 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:25.551+0000 7fd0597c2700 1 -- 172.21.15.42:0/785697487 wait complete. 2019-12-05T05:14:25.561 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:25.551+0000 7fd0597c2700 1 -- 172.21.15.42:0/785697487 >> 172.21.15.42:0/785697487 conn(0x7fd05408fde0 msgr2=0x7fd0540a9790 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:14:25.566 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2019-12-05T05:14:27.572 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:14:27.685 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:27.671+0000 7f2d771e0700 1 Processor -- start 2019-12-05T05:14:27.685 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:27.671+0000 7f2d771e0700 1 -- start start 2019-12-05T05:14:27.685 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:27.675+0000 7f2d771e0700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2d7009fae0 0x7f2d7014fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:14:27.685 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:27.675+0000 7f2d771e0700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d700a0710 con 0x7f2d700a7740 2019-12-05T05:14:27.685 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:27.675+0000 7f2d771e0700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d7007ef20 con 0x7f2d7008df80 2019-12-05T05:14:27.685 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:27.675+0000 7f2d771e0700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d700a0b00 con 0x7f2d7009fae0 2019-12-05T05:14:27.686 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:27.675+0000 7f2d74f7c700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f2d7008df80 0x7f2d7009f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:58608/0 (socket says 172.21.15.42:58608) 2019-12-05T05:14:27.686 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:27.675+0000 7f2d74f7c700 1 -- 172.21.15.42:0/1642983685 learned_addr learned my addr 172.21.15.42:0/1642983685 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:14:27.686 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:27.675+0000 7f2d75f7e700 1 --2- 172.21.15.42:0/1642983685 >> v2:172.21.15.40:3300/0 conn(0x7f2d7009fae0 0x7f2d7014fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:14:27.686 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:27.675+0000 7f2d63fff700 1 -- 172.21.15.42:0/1642983685 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2055074019 0 0) 0x7f2d7007ef20 con 0x7f2d7008df80 2019-12-05T05:14:27.686 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:27.675+0000 7f2d63fff700 1 -- 172.21.15.42:0/1642983685 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d5c0030b0 con 0x7f2d7008df80 2019-12-05T05:14:27.686 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:27.675+0000 7f2d63fff700 1 -- 172.21.15.42:0/1642983685 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2425142298 0 0) 0x7f2d700a0710 con 0x7f2d700a7740 2019-12-05T05:14:27.686 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:27.675+0000 7f2d63fff700 1 -- 172.21.15.42:0/1642983685 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d7007ef20 con 0x7f2d700a7740 2019-12-05T05:14:27.687 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:27.675+0000 7f2d75f7e700 1 -- 172.21.15.42:0/1642983685 >> v1:172.21.15.42:6789/0 conn(0x7f2d7008df80 legacy=0x7f2d7009f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:14:27.687 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:27.675+0000 7f2d75f7e700 1 -- 172.21.15.42:0/1642983685 >> v1:172.21.15.40:6789/0 conn(0x7f2d700a7740 legacy=0x7f2d7007e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:14:27.687 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:27.675+0000 7f2d75f7e700 1 -- 172.21.15.42:0/1642983685 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d700a9ba0 con 0x7f2d7009fae0 2019-12-05T05:14:27.687 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:27.675+0000 7f2d75f7e700 1 --2- 172.21.15.42:0/1642983685 >> v2:172.21.15.40:3300/0 conn(0x7f2d7009fae0 0x7f2d7014fff0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f2d68008b10 tx=0x7f2d6800ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:27.687 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:27.675+0000 7f2d75f7e700 1 --2- 172.21.15.42:0/1642983685 >> v2:172.21.15.40:3300/0 conn(0x7f2d7009fae0 0x7f2d7014fff0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f2d68008b10 tx=0x7f2d6800ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:27.687 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:27.675+0000 7f2d75f7e700 1 --2- 172.21.15.42:0/1642983685 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2d7009fae0 0x7f2d7014fff0 secure :-1 s=READY pgs=439 cs=0 l=1 rx=0x7f2d68008b10 tx=0x7f2d6800ff70).ready entity=mon.0 client_cookie=2a0c719105b6b558 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:14:27.687 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:27.675+0000 7f2d75f7e700 1 --2- 172.21.15.42:0/1642983685 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2d7009fae0 0x7f2d7014fff0 secure :-1 s=THROTTLE_DONE pgs=439 cs=0 l=1 rx=0x7f2d68008b10 tx=0x7f2d6800ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:27.688 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:27.675+0000 7f2d63fff700 1 -- 172.21.15.42:0/1642983685 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2d68010970 con 0x7f2d7009fae0 2019-12-05T05:14:27.688 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:27.675+0000 7f2d75f7e700 1 --2- 172.21.15.42:0/1642983685 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2d7009fae0 0x7f2d7014fff0 secure :-1 s=THROTTLE_DONE pgs=439 cs=0 l=1 rx=0x7f2d68008b10 tx=0x7f2d6800ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:27.688 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:27.675+0000 7f2d75f7e700 1 --2- 172.21.15.42:0/1642983685 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2d7009fae0 0x7f2d7014fff0 secure :-1 s=THROTTLE_DONE pgs=439 cs=0 l=1 rx=0x7f2d68008b10 tx=0x7f2d6800ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:27.688 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:27.675+0000 7f2d63fff700 1 -- 172.21.15.42:0/1642983685 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d68008b50 con 0x7f2d7009fae0 2019-12-05T05:14:27.692 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:27.675+0000 7f2d63fff700 1 -- 172.21.15.42:0/1642983685 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2d680110c0 con 0x7f2d7009fae0 2019-12-05T05:14:27.692 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:27.675+0000 7f2d771e0700 1 -- 172.21.15.42:0/1642983685 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2d7009fae0 msgr2=0x7f2d7014fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:14:27.693 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:27.675+0000 7f2d771e0700 1 --2- 172.21.15.42:0/1642983685 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2d7009fae0 0x7f2d7014fff0 secure :-1 s=READY pgs=439 cs=0 l=1 rx=0x7f2d68008b10 tx=0x7f2d6800ff70).stop 2019-12-05T05:14:27.693 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:27.675+0000 7f2d771e0700 1 -- 172.21.15.42:0/1642983685 shutdown_connections 2019-12-05T05:14:27.693 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:27.675+0000 7f2d771e0700 1 --2- 172.21.15.42:0/1642983685 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2d7009fae0 0x7f2d7014fff0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:14:27.693 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:27.675+0000 7f2d771e0700 1 -- 172.21.15.42:0/1642983685 shutdown_connections 2019-12-05T05:14:27.693 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:27.675+0000 7f2d771e0700 1 -- 172.21.15.42:0/1642983685 wait complete. 2019-12-05T05:14:27.694 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:27.675+0000 7f2d771e0700 1 -- 172.21.15.42:0/1642983685 >> 172.21.15.42:0/1642983685 conn(0x7f2d700aabe0 msgr2=0x7f2d700a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:14:27.694 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:27.679+0000 7f2d771e0700 1 Processor -- start 2019-12-05T05:14:27.694 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:27.679+0000 7f2d771e0700 1 -- start start 2019-12-05T05:14:27.699 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:27.679+0000 7f2d771e0700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2d70116990 0x7f2d70112d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:14:27.700 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:27.679+0000 7f2d771e0700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2d70111db0 0x7f2d7007d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:14:27.700 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:27.679+0000 7f2d771e0700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2d70126cc0 0x7f2d7007ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:14:27.700 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:27.679+0000 7f2d771e0700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d70112330 con 0x7f2d70116990 2019-12-05T05:14:27.700 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:27.679+0000 7f2d771e0700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d70112a80 con 0x7f2d70126cc0 2019-12-05T05:14:27.700 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:27.679+0000 7f2d771e0700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d70113270 con 0x7f2d70111db0 2019-12-05T05:14:27.701 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:27.679+0000 7f2d75f7e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2d70126cc0 0x7f2d7007ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:14:27.701 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:27.679+0000 7f2d75f7e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2d70126cc0 0x7f2d7007ddf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:58578/0 (socket says 172.21.15.42:58578) 2019-12-05T05:14:27.701 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:27.679+0000 7f2d75f7e700 1 -- 172.21.15.42:0/2382750705 learned_addr learned my addr 172.21.15.42:0/2382750705 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:14:27.701 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:27.679+0000 7f2d74f7c700 1 --2- 172.21.15.42:0/2382750705 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2d70111db0 0x7f2d7007d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:14:27.701 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:27.679+0000 7f2d7577d700 1 --2- 172.21.15.42:0/2382750705 >> v2:172.21.15.40:3300/0 conn(0x7f2d70116990 0x7f2d70112d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:14:27.701 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:27.679+0000 7f2d7577d700 1 -- 172.21.15.42:0/2382750705 >> v2:172.21.15.40:3300/0 conn(0x7f2d70116990 msgr2=0x7f2d70112d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2019-12-05T05:14:27.702 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:27.679+0000 7f2d7577d700 1 -- 172.21.15.42:0/2382750705 >> v2:172.21.15.40:3300/0 conn(0x7f2d70116990 msgr2=0x7f2d70112d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2019-12-05T05:14:27.702 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:27.679+0000 7f2d7577d700 1 --2- 172.21.15.42:0/2382750705 >> v2:172.21.15.40:3300/0 conn(0x7f2d70116990 0x7f2d70112d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-12-05T05:14:27.702 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:27.679+0000 7f2d7577d700 1 --2- 172.21.15.42:0/2382750705 >> v2:172.21.15.40:3300/0 conn(0x7f2d70116990 0x7f2d70112d20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rx=0 tx=0)._fault waiting 0.200000 2019-12-05T05:14:27.702 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:27.679+0000 7f2d75f7e700 1 -- 172.21.15.42:0/2382750705 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2d70111db0 msgr2=0x7f2d7007d8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:14:27.702 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:27.679+0000 7f2d75f7e700 1 --2- 172.21.15.42:0/2382750705 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2d70111db0 0x7f2d7007d8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:14:27.702 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:27.679+0000 7f2d75f7e700 1 -- 172.21.15.42:0/2382750705 >> v2:172.21.15.40:3300/0 conn(0x7f2d70116990 msgr2=0x7f2d70112d20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2019-12-05T05:14:27.702 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:27.679+0000 7f2d75f7e700 1 --2- 172.21.15.42:0/2382750705 >> v2:172.21.15.40:3300/0 conn(0x7f2d70116990 0x7f2d70112d20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:14:27.703 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:27.679+0000 7f2d75f7e700 1 -- 172.21.15.42:0/2382750705 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d70114bc0 con 0x7f2d70126cc0 2019-12-05T05:14:27.703 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:27.679+0000 7f2d75f7e700 1 --2- 172.21.15.42:0/2382750705 >> v2:172.21.15.42:3300/0 conn(0x7f2d70126cc0 0x7f2d7007ddf0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f2d68008b10 tx=0x7f2d68010e60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:27.703 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:27.679+0000 7f2d75f7e700 1 --2- 172.21.15.42:0/2382750705 >> v2:172.21.15.42:3300/0 conn(0x7f2d70126cc0 0x7f2d7007ddf0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f2d68008b10 tx=0x7f2d68010e60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:27.703 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:27.679+0000 7f2d75f7e700 1 --2- 172.21.15.42:0/2382750705 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2d70126cc0 0x7f2d7007ddf0 secure :-1 s=READY pgs=557 cs=0 l=1 rx=0x7f2d68008b10 tx=0x7f2d68010e60).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:14:27.703 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:27.679+0000 7f2d75f7e700 1 --2- 172.21.15.42:0/2382750705 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2d70126cc0 0x7f2d7007ddf0 secure :-1 s=THROTTLE_DONE pgs=557 cs=0 l=1 rx=0x7f2d68008b10 tx=0x7f2d68010e60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:27.703 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:27.679+0000 7f2d61ffb700 1 -- 172.21.15.42:0/2382750705 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2d68005ab0 con 0x7f2d70126cc0 2019-12-05T05:14:27.703 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:27.683+0000 7f2d75f7e700 1 --2- 172.21.15.42:0/2382750705 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2d70126cc0 0x7f2d7007ddf0 secure :-1 s=THROTTLE_DONE pgs=557 cs=0 l=1 rx=0x7f2d68008b10 tx=0x7f2d68010e60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:27.704 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:27.683+0000 7f2d61ffb700 1 -- 172.21.15.42:0/2382750705 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d68010b70 con 0x7f2d70126cc0 2019-12-05T05:14:27.704 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:27.683+0000 7f2d771e0700 1 -- 172.21.15.42:0/2382750705 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d700a4110 con 0x7f2d70126cc0 2019-12-05T05:14:27.704 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:27.683+0000 7f2d75f7e700 1 --2- 172.21.15.42:0/2382750705 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2d70126cc0 0x7f2d7007ddf0 secure :-1 s=THROTTLE_DONE pgs=557 cs=0 l=1 rx=0x7f2d68008b10 tx=0x7f2d68010e60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:27.704 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:27.683+0000 7f2d61ffb700 1 -- 172.21.15.42:0/2382750705 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2d6800f680 con 0x7f2d70126cc0 2019-12-05T05:14:27.704 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:27.683+0000 7f2d771e0700 1 -- 172.21.15.42:0/2382750705 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d7008ed10 con 0x7f2d70126cc0 2019-12-05T05:14:27.704 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:27.683+0000 7f2d75f7e700 1 --2- 172.21.15.42:0/2382750705 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2d70126cc0 0x7f2d7007ddf0 secure :-1 s=THROTTLE_DONE pgs=557 cs=0 l=1 rx=0x7f2d68008b10 tx=0x7f2d68010e60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:27.704 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:27.683+0000 7f2d61ffb700 1 -- 172.21.15.42:0/2382750705 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f2d68011390 con 0x7f2d70126cc0 2019-12-05T05:14:27.705 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:27.683+0000 7f2d75f7e700 1 --2- 172.21.15.42:0/2382750705 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2d70126cc0 0x7f2d7007ddf0 secure :-1 s=THROTTLE_DONE pgs=557 cs=0 l=1 rx=0x7f2d68008b10 tx=0x7f2d68010e60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:27.705 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:27.683+0000 7f2d61ffb700 1 --2- 172.21.15.42:0/2382750705 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2d5003ad70 0x7f2d5003d280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:14:27.705 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:27.683+0000 7f2d61ffb700 1 -- 172.21.15.42:0/2382750705 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6719+0+0 (secure 0 0 0) 0x7f2d6804aff0 con 0x7f2d70126cc0 2019-12-05T05:14:27.705 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:27.683+0000 7f2d7577d700 1 --2- 172.21.15.42:0/2382750705 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2d5003ad70 0x7f2d5003d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:14:27.705 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:27.683+0000 7f2d7577d700 1 --2- 172.21.15.42:0/2382750705 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2d5003ad70 0x7f2d5003d280 crc :-1 s=READY pgs=253 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:14:27.705 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:27.687+0000 7f2d771e0700 1 -- 172.21.15.42:0/2382750705 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d7005d110 con 0x7f2d70126cc0 2019-12-05T05:14:27.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:27.695+0000 7f2d75f7e700 1 --2- 172.21.15.42:0/2382750705 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2d70126cc0 0x7f2d7007ddf0 secure :-1 s=THROTTLE_DONE pgs=557 cs=0 l=1 rx=0x7f2d68008b10 tx=0x7f2d68010e60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:27.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:27.695+0000 7f2d61ffb700 1 -- 172.21.15.42:0/2382750705 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f2d68011c90 con 0x7f2d70126cc0 2019-12-05T05:14:27.887 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:27.859+0000 7f2d771e0700 1 -- 172.21.15.42:0/2382750705 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2d7005b8c0 con 0x7f2d70126cc0 2019-12-05T05:14:27.887 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:27.863+0000 7f2d75f7e700 1 --2- 172.21.15.42:0/2382750705 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2d70126cc0 0x7f2d7007ddf0 secure :-1 s=THROTTLE_DONE pgs=557 cs=0 l=1 rx=0x7f2d68008b10 tx=0x7f2d68010e60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:27.887 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:27.863+0000 7f2d61ffb700 1 -- 172.21.15.42:0/2382750705 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+2108 (secure 0 0 0) 0x7f2d6800f940 con 0x7f2d70126cc0 2019-12-05T05:14:27.888 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 73 2019-12-05T05:14:27.888 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:27.863+0000 7f2d771e0700 1 -- 172.21.15.42:0/2382750705 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2d5003ad70 msgr2=0x7f2d5003d280 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:14:27.888 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:27.863+0000 7f2d771e0700 1 --2- 172.21.15.42:0/2382750705 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2d5003ad70 0x7f2d5003d280 crc :-1 s=READY pgs=253 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:14:27.888 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:27.863+0000 7f2d771e0700 1 -- 172.21.15.42:0/2382750705 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2d70126cc0 msgr2=0x7f2d7007ddf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:14:27.888 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:27.863+0000 7f2d771e0700 1 --2- 172.21.15.42:0/2382750705 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2d70126cc0 0x7f2d7007ddf0 secure :-1 s=READY pgs=557 cs=0 l=1 rx=0x7f2d68008b10 tx=0x7f2d68010e60).stop 2019-12-05T05:14:27.888 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:27.863+0000 7f2d771e0700 1 -- 172.21.15.42:0/2382750705 shutdown_connections 2019-12-05T05:14:27.888 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:27.863+0000 7f2d771e0700 1 --2- 172.21.15.42:0/2382750705 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2d5003ad70 0x7f2d5003d280 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:14:27.889 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:27.863+0000 7f2d771e0700 1 --2- 172.21.15.42:0/2382750705 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2d70126cc0 0x7f2d7007ddf0 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:14:27.889 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:27.863+0000 7f2d771e0700 1 --2- 172.21.15.42:0/2382750705 >> v2:172.21.15.40:3300/0 conn(0x7f2d70116990 0x7f2d70112d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:14:27.889 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:27.863+0000 7f2d771e0700 1 --2- 172.21.15.42:0/2382750705 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2d70111db0 0x7f2d7007d8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:14:27.889 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:27.863+0000 7f2d771e0700 1 -- 172.21.15.42:0/2382750705 shutdown_connections 2019-12-05T05:14:27.889 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:27.863+0000 7f2d771e0700 1 -- 172.21.15.42:0/2382750705 wait complete. 2019-12-05T05:14:27.889 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:27.863+0000 7f2d771e0700 1 -- 172.21.15.42:0/2382750705 >> 172.21.15.42:0/2382750705 conn(0x7f2d700aabe0 msgr2=0x7f2d7009f510 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:14:27.889 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:14:27.890 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":73,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5629,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/3264829400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3264829400},{"type":"v1","addr":"172.21.15.42:6835","nonce":3264829400}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":73}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:14:17.896674+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4841},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4841":{"gid":4841,"name":"b","rank":0,"incarnation":69,"state":"up:active","state_seq":19,"addr":"172.21.15.40:6833/3150831963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3150831963},{"type":"v1","addr":"172.21.15.40:6833","nonce":3150831963}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:14:27.890 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a reported in up:standby state 2019-12-05T05:14:27.890 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 27 secs before thrashing 2019-12-05T05:14:35.548 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:14:35.552 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:14:54.892 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:14:55.000 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:54.987+0000 7f9479dc6700 1 Processor -- start 2019-12-05T05:14:55.000 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:54.987+0000 7f9479dc6700 1 -- start start 2019-12-05T05:14:55.000 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:54.991+0000 7f9479dc6700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f94740a7740 0x7f947408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:14:55.001 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:54.991+0000 7f9479dc6700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f947409fae0 0x7f947407eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:14:55.001 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:54.991+0000 7f9479dc6700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94740a0710 con 0x7f947407e720 2019-12-05T05:14:55.001 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:54.991+0000 7f9479dc6700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94740a0b00 con 0x7f94740a7740 2019-12-05T05:14:55.001 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:54.991+0000 7f9479dc6700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f947408e5b0 con 0x7f947409fae0 2019-12-05T05:14:55.001 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:54.991+0000 7f9478b64700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f947409fae0 0x7f947407eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:14:55.001 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:54.991+0000 7f94737fe700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f947407e720 0x7f947409f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:58646/0 (socket says 172.21.15.42:58646) 2019-12-05T05:14:55.002 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:54.991+0000 7f94737fe700 1 -- 172.21.15.42:0/978071248 learned_addr learned my addr 172.21.15.42:0/978071248 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:14:55.002 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:54.991+0000 7f9473fff700 1 --2- 172.21.15.42:0/978071248 >> v2:172.21.15.40:3300/0 conn(0x7f94740a7740 0x7f947408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:14:55.002 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:54.991+0000 7f9472ffd700 1 -- 172.21.15.42:0/978071248 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1242070785 0 0) 0x7f94740a0710 con 0x7f947407e720 2019-12-05T05:14:55.002 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:54.991+0000 7f9472ffd700 1 -- 172.21.15.42:0/978071248 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94600030b0 con 0x7f947407e720 2019-12-05T05:14:55.002 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:54.991+0000 7f9478b64700 1 -- 172.21.15.42:0/978071248 >> v2:172.21.15.40:3300/0 conn(0x7f94740a7740 msgr2=0x7f947408df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:14:55.002 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:54.991+0000 7f9478b64700 1 --2- 172.21.15.42:0/978071248 >> v2:172.21.15.40:3300/0 conn(0x7f94740a7740 0x7f947408df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:14:55.002 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:54.991+0000 7f9478b64700 1 -- 172.21.15.42:0/978071248 >> v1:172.21.15.42:6789/0 conn(0x7f947407e720 legacy=0x7f947409f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:14:55.003 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:54.991+0000 7f9478b64700 1 -- 172.21.15.42:0/978071248 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f947407f0a0 con 0x7f947409fae0 2019-12-05T05:14:55.003 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:54.991+0000 7f9478b64700 1 --2- 172.21.15.42:0/978071248 >> v2:172.21.15.42:3300/0 conn(0x7f947409fae0 0x7f947407eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f946c002f30 tx=0x7f946c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:55.003 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:54.991+0000 7f9478b64700 1 --2- 172.21.15.42:0/978071248 >> v2:172.21.15.42:3300/0 conn(0x7f947409fae0 0x7f947407eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f946c002f30 tx=0x7f946c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:55.003 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:54.991+0000 7f9478b64700 1 --2- 172.21.15.42:0/978071248 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f947409fae0 0x7f947407eb50 secure :-1 s=READY pgs=560 cs=0 l=1 rx=0x7f946c002f30 tx=0x7f946c00fc70).ready entity=mon.1 client_cookie=66a7bdac028d8943 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:14:55.003 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:54.991+0000 7f9478b64700 1 --2- 172.21.15.42:0/978071248 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f947409fae0 0x7f947407eb50 secure :-1 s=THROTTLE_DONE pgs=560 cs=0 l=1 rx=0x7f946c002f30 tx=0x7f946c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:55.003 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:54.991+0000 7f9478b64700 1 --2- 172.21.15.42:0/978071248 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f947409fae0 0x7f947407eb50 secure :-1 s=THROTTLE_DONE pgs=560 cs=0 l=1 rx=0x7f946c002f30 tx=0x7f946c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:55.003 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:54.991+0000 7f9472ffd700 1 -- 172.21.15.42:0/978071248 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f946c008bb0 con 0x7f947409fae0 2019-12-05T05:14:55.004 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:54.991+0000 7f9478b64700 1 --2- 172.21.15.42:0/978071248 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f947409fae0 0x7f947407eb50 secure :-1 s=THROTTLE_DONE pgs=560 cs=0 l=1 rx=0x7f946c002f30 tx=0x7f946c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:55.004 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:54.991+0000 7f9472ffd700 1 -- 172.21.15.42:0/978071248 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f946c0087d0 con 0x7f947409fae0 2019-12-05T05:14:55.004 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:54.991+0000 7f9472ffd700 1 -- 172.21.15.42:0/978071248 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f946c011820 con 0x7f947409fae0 2019-12-05T05:14:55.004 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:54.991+0000 7f9479dc6700 1 -- 172.21.15.42:0/978071248 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f947409fae0 msgr2=0x7f947407eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:14:55.004 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:54.991+0000 7f9479dc6700 1 --2- 172.21.15.42:0/978071248 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f947409fae0 0x7f947407eb50 secure :-1 s=READY pgs=560 cs=0 l=1 rx=0x7f946c002f30 tx=0x7f946c00fc70).stop 2019-12-05T05:14:55.004 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:54.991+0000 7f9479dc6700 1 -- 172.21.15.42:0/978071248 shutdown_connections 2019-12-05T05:14:55.005 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:54.991+0000 7f9479dc6700 1 --2- 172.21.15.42:0/978071248 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f947409fae0 0x7f947407eb50 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:14:55.005 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:54.991+0000 7f9479dc6700 1 --2- 172.21.15.42:0/978071248 >> v2:172.21.15.40:3300/0 conn(0x7f94740a7740 0x7f947408df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:14:55.005 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:54.991+0000 7f9479dc6700 1 -- 172.21.15.42:0/978071248 shutdown_connections 2019-12-05T05:14:55.005 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:54.991+0000 7f9479dc6700 1 -- 172.21.15.42:0/978071248 wait complete. 2019-12-05T05:14:55.005 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:54.991+0000 7f9479dc6700 1 -- 172.21.15.42:0/978071248 >> 172.21.15.42:0/978071248 conn(0x7f94740aabe0 msgr2=0x7f94740a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:14:55.008 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:54.995+0000 7f9479dc6700 1 Processor -- start 2019-12-05T05:14:55.008 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:54.995+0000 7f9479dc6700 1 -- start start 2019-12-05T05:14:55.008 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:54.995+0000 7f9479dc6700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f947407df50 0x7f9474117ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:14:55.008 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:54.999+0000 7f9479dc6700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f94740a6a70 0x7f94740a9790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:14:55.009 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:54.999+0000 7f9479dc6700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94740a9ce0 con 0x7f9474121fc0 2019-12-05T05:14:55.009 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:54.999+0000 7f9479dc6700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9474117c00 con 0x7f947407df50 2019-12-05T05:14:55.009 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:54.999+0000 7f9479dc6700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94741223f0 con 0x7f94740a6a70 2019-12-05T05:14:55.009 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:54.999+0000 7f9473fff700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f947407df50 0x7f9474117ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:14:55.009 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:54.999+0000 7f9478b64700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f94740a6a70 0x7f94740a9790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:14:55.009 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:54.999+0000 7f9473fff700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f947407df50 0x7f9474117ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.42:46428/0 (socket says 172.21.15.42:46428) 2019-12-05T05:14:55.009 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:54.999+0000 7f9473fff700 1 -- 172.21.15.42:0/3547361794 learned_addr learned my addr 172.21.15.42:0/3547361794 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:14:55.010 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:54.999+0000 7f9478b64700 1 -- 172.21.15.42:0/3547361794 >> v2:172.21.15.40:3300/0 conn(0x7f947407df50 msgr2=0x7f9474117ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:14:55.010 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:54.999+0000 7f9478b64700 1 --2- 172.21.15.42:0/3547361794 >> v2:172.21.15.40:3300/0 conn(0x7f947407df50 0x7f9474117ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:14:55.010 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:54.999+0000 7f9478b64700 1 -- 172.21.15.42:0/3547361794 >> v1:172.21.15.40:6789/0 conn(0x7f9474121fc0 legacy=0x7f9474120290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:14:55.010 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:54.999+0000 7f9478b64700 1 -- 172.21.15.42:0/3547361794 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94740a6ea0 con 0x7f94740a6a70 2019-12-05T05:14:55.010 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:54.999+0000 7f9478b64700 1 --2- 172.21.15.42:0/3547361794 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f94740a6a70 0x7f94740a9790 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f946c002f30 tx=0x7f946c011e60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:55.010 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:54.999+0000 7f9478b64700 1 --2- 172.21.15.42:0/3547361794 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f94740a6a70 0x7f94740a9790 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f946c002f30 tx=0x7f946c011e60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:55.011 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:54.999+0000 7f9478b64700 1 --2- 172.21.15.42:0/3547361794 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f94740a6a70 0x7f94740a9790 secure :-1 s=READY pgs=115 cs=0 l=1 rx=0x7f946c002f30 tx=0x7f946c011e60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:14:55.011 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:54.999+0000 7f9478b64700 1 --2- 172.21.15.42:0/3547361794 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f94740a6a70 0x7f94740a9790 secure :-1 s=THROTTLE_DONE pgs=115 cs=0 l=1 rx=0x7f946c002f30 tx=0x7f946c011e60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:55.011 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:54.999+0000 7f9478b64700 1 --2- 172.21.15.42:0/3547361794 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f94740a6a70 0x7f94740a9790 secure :-1 s=THROTTLE_DONE pgs=115 cs=0 l=1 rx=0x7f946c002f30 tx=0x7f946c011e60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:55.011 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:54.999+0000 7f9470ff9700 1 -- 172.21.15.42:0/3547361794 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f946c00f280 con 0x7f94740a6a70 2019-12-05T05:14:55.011 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:54.999+0000 7f9478b64700 1 --2- 172.21.15.42:0/3547361794 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f94740a6a70 0x7f94740a9790 secure :-1 s=THROTTLE_DONE pgs=115 cs=0 l=1 rx=0x7f946c002f30 tx=0x7f946c011e60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:55.011 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:54.999+0000 7f9470ff9700 1 -- 172.21.15.42:0/3547361794 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f946c008db0 con 0x7f94740a6a70 2019-12-05T05:14:55.011 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:54.999+0000 7f9479dc6700 1 -- 172.21.15.42:0/3547361794 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9474116430 con 0x7f94740a6a70 2019-12-05T05:14:55.012 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:54.999+0000 7f9470ff9700 1 -- 172.21.15.42:0/3547361794 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f946c005540 con 0x7f94740a6a70 2019-12-05T05:14:55.012 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:54.999+0000 7f9479dc6700 1 -- 172.21.15.42:0/3547361794 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9474116980 con 0x7f94740a6a70 2019-12-05T05:14:55.012 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:54.999+0000 7f9478b64700 1 --2- 172.21.15.42:0/3547361794 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f94740a6a70 0x7f94740a9790 secure :-1 s=THROTTLE_DONE pgs=115 cs=0 l=1 rx=0x7f946c002f30 tx=0x7f946c011e60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:55.012 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:55.003+0000 7f9470ff9700 1 -- 172.21.15.42:0/3547361794 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f946c005850 con 0x7f94740a6a70 2019-12-05T05:14:55.012 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:55.003+0000 7f9478b64700 1 --2- 172.21.15.42:0/3547361794 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f94740a6a70 0x7f94740a9790 secure :-1 s=THROTTLE_DONE pgs=115 cs=0 l=1 rx=0x7f946c002f30 tx=0x7f946c011e60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:55.013 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:55.003+0000 7f9470ff9700 1 --2- 172.21.15.42:0/3547361794 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f945803ad80 0x7f945803d290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:14:55.013 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:55.003+0000 7f9470ff9700 1 -- 172.21.15.42:0/3547361794 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6719+0+0 (secure 0 0 0) 0x7f946c08f7b0 con 0x7f94740a6a70 2019-12-05T05:14:55.013 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:55.003+0000 7f9473fff700 1 --2- 172.21.15.42:0/3547361794 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f945803ad80 0x7f945803d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:14:55.013 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:55.003+0000 7f9473fff700 1 --2- 172.21.15.42:0/3547361794 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f945803ad80 0x7f945803d290 crc :-1 s=READY pgs=254 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:14:55.014 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:55.003+0000 7f9479dc6700 1 -- 172.21.15.42:0/3547361794 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f947405d110 con 0x7f94740a6a70 2019-12-05T05:14:55.019 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:55.011+0000 7f9478b64700 1 --2- 172.21.15.42:0/3547361794 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f94740a6a70 0x7f94740a9790 secure :-1 s=THROTTLE_DONE pgs=115 cs=0 l=1 rx=0x7f946c002f30 tx=0x7f946c011e60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:55.019 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:55.011+0000 7f9470ff9700 1 -- 172.21.15.42:0/3547361794 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f946c076a90 con 0x7f94740a6a70 2019-12-05T05:14:55.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:55.183+0000 7f9479dc6700 1 -- 172.21.15.42:0/3547361794 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f947405b8c0 con 0x7f94740a6a70 2019-12-05T05:14:55.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:55.183+0000 7f9478b64700 1 --2- 172.21.15.42:0/3547361794 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f94740a6a70 0x7f94740a9790 secure :-1 s=THROTTLE_DONE pgs=115 cs=0 l=1 rx=0x7f946c002f30 tx=0x7f946c011e60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:55.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:55.183+0000 7f9470ff9700 1 -- 172.21.15.42:0/3547361794 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+2108 (secure 0 0 0) 0x7f946c005b60 con 0x7f94740a6a70 2019-12-05T05:14:55.192 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:14:55.192 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":73,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5629,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/3264829400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3264829400},{"type":"v1","addr":"172.21.15.42:6835","nonce":3264829400}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":73}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:14:17.896674+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4841},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4841":{"gid":4841,"name":"b","rank":0,"incarnation":69,"state":"up:active","state_seq":19,"addr":"172.21.15.40:6833/3150831963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3150831963},{"type":"v1","addr":"172.21.15.40:6833","nonce":3150831963}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:14:55.192 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 73 2019-12-05T05:14:55.194 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:55.187+0000 7f9479dc6700 1 -- 172.21.15.42:0/3547361794 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f945803ad80 msgr2=0x7f945803d290 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:14:55.194 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:55.187+0000 7f9479dc6700 1 --2- 172.21.15.42:0/3547361794 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f945803ad80 0x7f945803d290 crc :-1 s=READY pgs=254 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:14:55.195 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:55.187+0000 7f9479dc6700 1 -- 172.21.15.42:0/3547361794 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f94740a6a70 msgr2=0x7f94740a9790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:14:55.195 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:55.187+0000 7f9479dc6700 1 --2- 172.21.15.42:0/3547361794 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f94740a6a70 0x7f94740a9790 secure :-1 s=READY pgs=115 cs=0 l=1 rx=0x7f946c002f30 tx=0x7f946c011e60).stop 2019-12-05T05:14:55.195 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:55.187+0000 7f9479dc6700 1 -- 172.21.15.42:0/3547361794 shutdown_connections 2019-12-05T05:14:55.195 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:55.187+0000 7f9479dc6700 1 --2- 172.21.15.42:0/3547361794 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f945803ad80 0x7f945803d290 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:14:55.195 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:55.187+0000 7f9479dc6700 1 --2- 172.21.15.42:0/3547361794 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f94740a6a70 0x7f94740a9790 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:14:55.195 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:55.187+0000 7f9479dc6700 1 --2- 172.21.15.42:0/3547361794 >> v2:172.21.15.40:3300/0 conn(0x7f947407df50 0x7f9474117ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:14:55.196 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:55.187+0000 7f9479dc6700 1 -- 172.21.15.42:0/3547361794 shutdown_connections 2019-12-05T05:14:55.196 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:55.187+0000 7f9479dc6700 1 -- 172.21.15.42:0/3547361794 wait complete. 2019-12-05T05:14:55.196 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:55.187+0000 7f9479dc6700 1 -- 172.21.15.42:0/3547361794 >> 172.21.15.42:0/3547361794 conn(0x7f94740aabe0 msgr2=0x7f94741125a0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:14:55.204 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.b (rank=0) 2019-12-05T05:14:55.204 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2019-12-05T05:14:55.204 INFO:teuthology.orchestra.run:waiting for 300 2019-12-05T05:14:55.220 INFO:tasks.ceph.mds.b:Stopped 2019-12-05T05:14:55.221 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:14:55.323 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:55.307+0000 7fd55674e700 1 Processor -- start 2019-12-05T05:14:55.323 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:55.307+0000 7fd55674e700 1 -- start start 2019-12-05T05:14:55.323 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:55.311+0000 7fd55674e700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd55009fae0 0x7fd55014fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:14:55.324 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:55.311+0000 7fd55674e700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5500a0710 con 0x7fd55008df80 2019-12-05T05:14:55.324 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:55.311+0000 7fd55674e700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd55007ef20 con 0x7fd5500a7740 2019-12-05T05:14:55.324 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:55.311+0000 7fd55674e700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5500a0b00 con 0x7fd55009fae0 2019-12-05T05:14:55.324 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:55.311+0000 7fd554ceb700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fd5500a7740 0x7fd55007e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:58660/0 (socket says 172.21.15.42:58660) 2019-12-05T05:14:55.324 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:55.311+0000 7fd554ceb700 1 -- 172.21.15.42:0/1816839052 learned_addr learned my addr 172.21.15.42:0/1816839052 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:14:55.324 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:55.311+0000 7fd5554ec700 1 --2- 172.21.15.42:0/1816839052 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd55009fae0 0x7fd55014fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:14:55.324 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:55.311+0000 7fd54f7fe700 1 -- 172.21.15.42:0/1816839052 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1653079854 0 0) 0x7fd55007ef20 con 0x7fd5500a7740 2019-12-05T05:14:55.325 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:55.311+0000 7fd54f7fe700 1 -- 172.21.15.42:0/1816839052 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5400030b0 con 0x7fd5500a7740 2019-12-05T05:14:55.325 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:55.311+0000 7fd54f7fe700 1 -- 172.21.15.42:0/1816839052 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3794489012 0 0) 0x7fd5500a0710 con 0x7fd55008df80 2019-12-05T05:14:55.325 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:55.311+0000 7fd54f7fe700 1 -- 172.21.15.42:0/1816839052 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd55007ef20 con 0x7fd55008df80 2019-12-05T05:14:55.325 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:55.311+0000 7fd54f7fe700 1 -- 172.21.15.42:0/1816839052 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1746942687 0 0) 0x7fd5400030b0 con 0x7fd5500a7740 2019-12-05T05:14:55.325 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:55.311+0000 7fd54f7fe700 1 -- 172.21.15.42:0/1816839052 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd55009fae0 msgr2=0x7fd55014fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:14:55.325 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:55.311+0000 7fd54f7fe700 1 --2- 172.21.15.42:0/1816839052 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd55009fae0 0x7fd55014fff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:14:55.326 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:55.311+0000 7fd54f7fe700 1 -- 172.21.15.42:0/1816839052 >> v1:172.21.15.40:6789/0 conn(0x7fd55008df80 legacy=0x7fd55009f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:14:55.326 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:55.311+0000 7fd54f7fe700 1 -- 172.21.15.42:0/1816839052 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5500a9ba0 con 0x7fd5500a7740 2019-12-05T05:14:55.326 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:55.311+0000 7fd54f7fe700 1 -- 172.21.15.42:0/1816839052 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7fd544001750 con 0x7fd5500a7740 2019-12-05T05:14:55.326 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:55.311+0000 7fd54f7fe700 1 -- 172.21.15.42:0/1816839052 >> v1:172.21.15.42:6789/0 conn(0x7fd5500a7740 legacy=0x7fd55007e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:14:55.326 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:55.311+0000 7fd54f7fe700 1 --2- 172.21.15.42:0/1816839052 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd540003940 0x7fd540003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:14:55.326 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:55.311+0000 7fd54f7fe700 1 --2- 172.21.15.42:0/1816839052 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd540004420 0x7fd540004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:14:55.326 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:55.311+0000 7fd54f7fe700 1 --2- 172.21.15.42:0/1816839052 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd540004e90 0x7fd54004f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:14:55.327 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:55.311+0000 7fd54f7fe700 1 -- 172.21.15.42:0/1816839052 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5500a0b00 con 0x7fd540004420 2019-12-05T05:14:55.327 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:55.311+0000 7fd54f7fe700 1 -- 172.21.15.42:0/1816839052 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd54004fc30 con 0x7fd540004e90 2019-12-05T05:14:55.327 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:55.311+0000 7fd54f7fe700 1 -- 172.21.15.42:0/1816839052 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd54004fe70 con 0x7fd540003940 2019-12-05T05:14:55.327 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:55.311+0000 7fd554ceb700 1 --2- 172.21.15.42:0/1816839052 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd540003940 0x7fd540003d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:14:55.327 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:55.311+0000 7fd54ffff700 1 --2- 172.21.15.42:0/1816839052 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd540004420 0x7fd540004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:14:55.327 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:55.311+0000 7fd5554ec700 1 --2- 172.21.15.42:0/1816839052 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd540004e90 0x7fd54004f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:14:55.327 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:55.311+0000 7fd5554ec700 1 -- 172.21.15.42:0/1816839052 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd540003940 msgr2=0x7fd540003d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:14:55.328 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:55.311+0000 7fd5554ec700 1 --2- 172.21.15.42:0/1816839052 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd540003940 0x7fd540003d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:14:55.328 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:55.311+0000 7fd5554ec700 1 -- 172.21.15.42:0/1816839052 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd540004420 msgr2=0x7fd540004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:14:55.328 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:55.311+0000 7fd5554ec700 1 --2- 172.21.15.42:0/1816839052 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd540004420 0x7fd540004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:14:55.328 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:55.311+0000 7fd5554ec700 1 -- 172.21.15.42:0/1816839052 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd540050170 con 0x7fd540004e90 2019-12-05T05:14:55.328 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:55.311+0000 7fd554ceb700 1 --2- 172.21.15.42:0/1816839052 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd540003940 0x7fd540003d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T05:14:55.329 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:55.311+0000 7fd5554ec700 1 --2- 172.21.15.42:0/1816839052 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd540004e90 0x7fd54004f680 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fd5480032b0 tx=0x7fd548010300).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:55.329 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:55.311+0000 7fd5554ec700 1 --2- 172.21.15.42:0/1816839052 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd540004e90 0x7fd54004f680 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fd5480032b0 tx=0x7fd548010300).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:55.329 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:55.311+0000 7fd5554ec700 1 --2- 172.21.15.42:0/1816839052 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd540004e90 0x7fd54004f680 secure :-1 s=READY pgs=562 cs=0 l=1 rx=0x7fd5480032b0 tx=0x7fd548010300).ready entity=mon.1 client_cookie=1406ee2ad5e12fa0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:14:55.329 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:55.311+0000 7fd5554ec700 1 --2- 172.21.15.42:0/1816839052 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd540004e90 0x7fd54004f680 secure :-1 s=THROTTLE_DONE pgs=562 cs=0 l=1 rx=0x7fd5480032b0 tx=0x7fd548010300).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:55.329 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:55.311+0000 7fd5554ec700 1 --2- 172.21.15.42:0/1816839052 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd540004e90 0x7fd54004f680 secure :-1 s=THROTTLE_DONE pgs=562 cs=0 l=1 rx=0x7fd5480032b0 tx=0x7fd548010300).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:55.329 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:55.311+0000 7fd54f7fe700 1 -- 172.21.15.42:0/1816839052 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd548008d40 con 0x7fd540004e90 2019-12-05T05:14:55.329 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:55.311+0000 7fd5554ec700 1 --2- 172.21.15.42:0/1816839052 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd540004e90 0x7fd54004f680 secure :-1 s=THROTTLE_DONE pgs=562 cs=0 l=1 rx=0x7fd5480032b0 tx=0x7fd548010300).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:55.330 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:55.311+0000 7fd54f7fe700 1 -- 172.21.15.42:0/1816839052 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd548010cf0 con 0x7fd540004e90 2019-12-05T05:14:55.330 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:55.311+0000 7fd54f7fe700 1 -- 172.21.15.42:0/1816839052 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd548005ca0 con 0x7fd540004e90 2019-12-05T05:14:55.330 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:55.311+0000 7fd55674e700 1 -- 172.21.15.42:0/1816839052 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd540004e90 msgr2=0x7fd54004f680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:14:55.330 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:55.311+0000 7fd55674e700 1 --2- 172.21.15.42:0/1816839052 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd540004e90 0x7fd54004f680 secure :-1 s=READY pgs=562 cs=0 l=1 rx=0x7fd5480032b0 tx=0x7fd548010300).stop 2019-12-05T05:14:55.330 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:55.311+0000 7fd5554ec700 1 -- 172.21.15.42:0/1816839052 reap_dead start 2019-12-05T05:14:55.330 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:55.311+0000 7fd55674e700 1 -- 172.21.15.42:0/1816839052 shutdown_connections 2019-12-05T05:14:55.330 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:55.315+0000 7fd55674e700 1 -- 172.21.15.42:0/1816839052 shutdown_connections 2019-12-05T05:14:55.331 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:55.315+0000 7fd55674e700 1 -- 172.21.15.42:0/1816839052 wait complete. 2019-12-05T05:14:55.331 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:55.315+0000 7fd55674e700 1 -- 172.21.15.42:0/1816839052 >> 172.21.15.42:0/1816839052 conn(0x7fd5500aabe0 msgr2=0x7fd5500a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:14:55.331 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:55.315+0000 7fd55674e700 1 Processor -- start 2019-12-05T05:14:55.331 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:55.315+0000 7fd55674e700 1 -- start start 2019-12-05T05:14:55.331 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:55.315+0000 7fd55674e700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd55007d720 0x7fd55012d660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:14:55.331 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:55.315+0000 7fd55674e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd55012dbb0 0x7fd5500a5410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:14:55.331 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:55.315+0000 7fd55674e700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd5500a5960 0x7fd550130590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:14:55.332 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:55.315+0000 7fd55674e700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd550145370 con 0x7fd55007d720 2019-12-05T05:14:55.332 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:55.315+0000 7fd55674e700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd550142760 con 0x7fd55012dbb0 2019-12-05T05:14:55.332 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:55.315+0000 7fd55674e700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd550142a00 con 0x7fd5500a5960 2019-12-05T05:14:55.332 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:55.315+0000 7fd54ffff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd55012dbb0 0x7fd5500a5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:14:55.332 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:55.315+0000 7fd54ffff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd55012dbb0 0x7fd5500a5410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:58636/0 (socket says 172.21.15.42:58636) 2019-12-05T05:14:55.332 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:55.315+0000 7fd5554ec700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd5500a5960 0x7fd550130590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:14:55.332 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:55.315+0000 7fd54ffff700 1 -- 172.21.15.42:0/4238577546 learned_addr learned my addr 172.21.15.42:0/4238577546 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:14:55.333 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:55.315+0000 7fd554ceb700 1 --2- 172.21.15.42:0/4238577546 >> v2:172.21.15.40:3300/0 conn(0x7fd55007d720 0x7fd55012d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:14:55.335 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:55.319+0000 7fd5554ec700 1 -- 172.21.15.42:0/4238577546 >> v2:172.21.15.42:3300/0 conn(0x7fd55012dbb0 msgr2=0x7fd5500a5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:14:55.335 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:55.319+0000 7fd5554ec700 1 --2- 172.21.15.42:0/4238577546 >> v2:172.21.15.42:3300/0 conn(0x7fd55012dbb0 0x7fd5500a5410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:14:55.335 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:55.319+0000 7fd5554ec700 1 -- 172.21.15.42:0/4238577546 >> v2:172.21.15.40:3300/0 conn(0x7fd55007d720 msgr2=0x7fd55012d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:14:55.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:55.319+0000 7fd5554ec700 1 --2- 172.21.15.42:0/4238577546 >> v2:172.21.15.40:3300/0 conn(0x7fd55007d720 0x7fd55012d660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:14:55.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:55.319+0000 7fd5554ec700 1 -- 172.21.15.42:0/4238577546 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5500a7040 con 0x7fd5500a5960 2019-12-05T05:14:55.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:55.319+0000 7fd54ffff700 1 --2- 172.21.15.42:0/4238577546 >> v2:172.21.15.42:3300/0 conn(0x7fd55012dbb0 0x7fd5500a5410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:14:55.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:55.319+0000 7fd5554ec700 1 --2- 172.21.15.42:0/4238577546 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd5500a5960 0x7fd550130590 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fd550142920 tx=0x7fd548007b10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:55.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:55.319+0000 7fd5554ec700 1 --2- 172.21.15.42:0/4238577546 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd5500a5960 0x7fd550130590 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fd550142920 tx=0x7fd548007b10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:55.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:55.319+0000 7fd5554ec700 1 --2- 172.21.15.42:0/4238577546 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd5500a5960 0x7fd550130590 secure :-1 s=READY pgs=116 cs=0 l=1 rx=0x7fd550142920 tx=0x7fd548007b10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:14:55.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:55.319+0000 7fd5554ec700 1 --2- 172.21.15.42:0/4238577546 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd5500a5960 0x7fd550130590 secure :-1 s=THROTTLE_DONE pgs=116 cs=0 l=1 rx=0x7fd550142920 tx=0x7fd548007b10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:55.341 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:55.319+0000 7fd54d7fa700 1 -- 172.21.15.42:0/4238577546 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd548007160 con 0x7fd5500a5960 2019-12-05T05:14:55.341 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:55.319+0000 7fd5554ec700 1 --2- 172.21.15.42:0/4238577546 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd5500a5960 0x7fd550130590 secure :-1 s=THROTTLE_DONE pgs=116 cs=0 l=1 rx=0x7fd550142920 tx=0x7fd548007b10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:55.341 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:55.319+0000 7fd5554ec700 1 --2- 172.21.15.42:0/4238577546 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd5500a5960 0x7fd550130590 secure :-1 s=THROTTLE_DONE pgs=116 cs=0 l=1 rx=0x7fd550142920 tx=0x7fd548007b10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:55.341 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:55.319+0000 7fd54d7fa700 1 -- 172.21.15.42:0/4238577546 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5480058c0 con 0x7fd5500a5960 2019-12-05T05:14:55.341 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:55.319+0000 7fd54d7fa700 1 -- 172.21.15.42:0/4238577546 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd548001180 con 0x7fd5500a5960 2019-12-05T05:14:55.341 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:55.319+0000 7fd55674e700 1 -- 172.21.15.42:0/4238577546 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5500a5d90 con 0x7fd5500a5960 2019-12-05T05:14:55.341 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:55.319+0000 7fd55674e700 1 -- 172.21.15.42:0/4238577546 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5500a4d20 con 0x7fd5500a5960 2019-12-05T05:14:55.342 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:55.319+0000 7fd5554ec700 1 --2- 172.21.15.42:0/4238577546 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd5500a5960 0x7fd550130590 secure :-1 s=THROTTLE_DONE pgs=116 cs=0 l=1 rx=0x7fd550142920 tx=0x7fd548007b10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:55.342 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:55.319+0000 7fd54d7fa700 1 -- 172.21.15.42:0/4238577546 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fd548001380 con 0x7fd5500a5960 2019-12-05T05:14:55.342 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:55.319+0000 7fd5554ec700 1 --2- 172.21.15.42:0/4238577546 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd5500a5960 0x7fd550130590 secure :-1 s=THROTTLE_DONE pgs=116 cs=0 l=1 rx=0x7fd550142920 tx=0x7fd548007b10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:55.342 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:55.319+0000 7fd54d7fa700 1 --2- 172.21.15.42:0/4238577546 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd53003ada0 0x7fd53003d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:14:55.342 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:55.319+0000 7fd54d7fa700 1 -- 172.21.15.42:0/4238577546 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6719+0+0 (secure 0 0 0) 0x7fd54804ade0 con 0x7fd5500a5960 2019-12-05T05:14:55.342 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:55.319+0000 7fd554ceb700 1 --2- 172.21.15.42:0/4238577546 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd53003ada0 0x7fd53003d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:14:55.342 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:55.323+0000 7fd554ceb700 1 --2- 172.21.15.42:0/4238577546 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd53003ada0 0x7fd53003d2b0 crc :-1 s=READY pgs=255 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:14:55.343 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:55.323+0000 7fd55674e700 1 -- 172.21.15.42:0/4238577546 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd55005d110 con 0x7fd5500a5960 2019-12-05T05:14:55.343 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:55.331+0000 7fd5554ec700 1 --2- 172.21.15.42:0/4238577546 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd5500a5960 0x7fd550130590 secure :-1 s=THROTTLE_DONE pgs=116 cs=0 l=1 rx=0x7fd550142920 tx=0x7fd548007b10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:55.343 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:55.331+0000 7fd54d7fa700 1 -- 172.21.15.42:0/4238577546 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fd5480320c0 con 0x7fd5500a5960 2019-12-05T05:14:55.508 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:55.503+0000 7fd55674e700 1 -- 172.21.15.42:0/4238577546 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd55005b8c0 con 0x7fd5500a5960 2019-12-05T05:14:55.509 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:55.503+0000 7fd5554ec700 1 --2- 172.21.15.42:0/4238577546 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd5500a5960 0x7fd550130590 secure :-1 s=THROTTLE_DONE pgs=116 cs=0 l=1 rx=0x7fd550142920 tx=0x7fd548007b10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:55.509 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:55.503+0000 7fd54d7fa700 1 -- 172.21.15.42:0/4238577546 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+2108 (secure 0 0 0) 0x7fd54802fa80 con 0x7fd5500a5960 2019-12-05T05:14:55.511 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:14:55.511 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":73,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5629,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/3264829400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3264829400},{"type":"v1","addr":"172.21.15.42:6835","nonce":3264829400}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":73}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:14:17.896674+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4841},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4841":{"gid":4841,"name":"b","rank":0,"incarnation":69,"state":"up:active","state_seq":19,"addr":"172.21.15.40:6833/3150831963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3150831963},{"type":"v1","addr":"172.21.15.40:6833","nonce":3150831963}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:14:55.511 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 73 2019-12-05T05:14:55.514 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:55.507+0000 7fd55674e700 1 -- 172.21.15.42:0/4238577546 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd53003ada0 msgr2=0x7fd53003d2b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:14:55.514 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:55.507+0000 7fd55674e700 1 --2- 172.21.15.42:0/4238577546 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd53003ada0 0x7fd53003d2b0 crc :-1 s=READY pgs=255 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:14:55.514 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:55.507+0000 7fd55674e700 1 -- 172.21.15.42:0/4238577546 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd5500a5960 msgr2=0x7fd550130590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:14:55.514 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:55.507+0000 7fd55674e700 1 --2- 172.21.15.42:0/4238577546 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd5500a5960 0x7fd550130590 secure :-1 s=READY pgs=116 cs=0 l=1 rx=0x7fd550142920 tx=0x7fd548007b10).stop 2019-12-05T05:14:55.514 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:55.507+0000 7fd55674e700 1 -- 172.21.15.42:0/4238577546 shutdown_connections 2019-12-05T05:14:55.514 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:55.507+0000 7fd55674e700 1 --2- 172.21.15.42:0/4238577546 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd53003ada0 0x7fd53003d2b0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:14:55.515 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:55.507+0000 7fd55674e700 1 --2- 172.21.15.42:0/4238577546 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd5500a5960 0x7fd550130590 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:14:55.515 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:55.507+0000 7fd55674e700 1 --2- 172.21.15.42:0/4238577546 >> v2:172.21.15.40:3300/0 conn(0x7fd55007d720 0x7fd55012d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:14:55.515 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:55.507+0000 7fd55674e700 1 --2- 172.21.15.42:0/4238577546 >> v2:172.21.15.42:3300/0 conn(0x7fd55012dbb0 0x7fd5500a5410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:14:55.515 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:55.507+0000 7fd55674e700 1 -- 172.21.15.42:0/4238577546 shutdown_connections 2019-12-05T05:14:55.515 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:55.507+0000 7fd55674e700 1 -- 172.21.15.42:0/4238577546 wait complete. 2019-12-05T05:14:55.515 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:55.507+0000 7fd55674e700 1 -- 172.21.15.42:0/4238577546 >> 172.21.15.42:0/4238577546 conn(0x7fd55007e760 msgr2=0x7fd55011c4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:14:55.524 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T05:14:57.525 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:14:57.633 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:57.618+0000 7fd823b13700 1 Processor -- start 2019-12-05T05:14:57.634 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:57.618+0000 7fd823b13700 1 -- start start 2019-12-05T05:14:57.634 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:57.618+0000 7fd823b13700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd81c0a7740 0x7fd81c08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:14:57.634 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:57.618+0000 7fd823b13700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd81c07e720 0x7fd81c07eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:14:57.634 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:57.618+0000 7fd823b13700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd81c09f2e0 0x7fd81c09f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:14:57.634 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:57.618+0000 7fd823b13700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd81c0a0b00 con 0x7fd81c09f2e0 2019-12-05T05:14:57.634 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:57.618+0000 7fd823b13700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd81c08e5b0 con 0x7fd81c0a7740 2019-12-05T05:14:57.635 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:57.618+0000 7fd823b13700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd81c08e850 con 0x7fd81c07e720 2019-12-05T05:14:57.635 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:57.622+0000 7fd8220b0700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd81c0a7740 0x7fd81c08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:14:57.635 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:57.622+0000 7fd8220b0700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd81c0a7740 0x7fd81c08df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:58648/0 (socket says 172.21.15.42:58648) 2019-12-05T05:14:57.635 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:57.622+0000 7fd8220b0700 1 -- 172.21.15.42:0/2042665312 learned_addr learned my addr 172.21.15.42:0/2042665312 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:14:57.635 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:57.622+0000 7fd8218af700 1 --2- 172.21.15.42:0/2042665312 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd81c07e720 0x7fd81c07eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:14:57.635 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:57.622+0000 7fd8228b1700 1 --2- 172.21.15.42:0/2042665312 >> v2:172.21.15.40:3300/0 conn(0x7fd81c09f2e0 0x7fd81c09f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:14:57.636 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:57.622+0000 7fd8220b0700 1 -- 172.21.15.42:0/2042665312 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd81c07e720 msgr2=0x7fd81c07eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:14:57.636 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:57.622+0000 7fd8220b0700 1 --2- 172.21.15.42:0/2042665312 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd81c07e720 0x7fd81c07eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:14:57.636 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:57.622+0000 7fd8220b0700 1 -- 172.21.15.42:0/2042665312 >> v2:172.21.15.40:3300/0 conn(0x7fd81c09f2e0 msgr2=0x7fd81c09f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:14:57.636 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:57.622+0000 7fd8220b0700 1 --2- 172.21.15.42:0/2042665312 >> v2:172.21.15.40:3300/0 conn(0x7fd81c09f2e0 0x7fd81c09f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:14:57.636 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:57.622+0000 7fd8220b0700 1 -- 172.21.15.42:0/2042665312 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd81c07f0a0 con 0x7fd81c0a7740 2019-12-05T05:14:57.636 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:57.622+0000 7fd8220b0700 1 --2- 172.21.15.42:0/2042665312 >> v2:172.21.15.42:3300/0 conn(0x7fd81c0a7740 0x7fd81c08df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fd81c08e770 tx=0x7fd81800fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:57.636 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:57.622+0000 7fd8220b0700 1 --2- 172.21.15.42:0/2042665312 >> v2:172.21.15.42:3300/0 conn(0x7fd81c0a7740 0x7fd81c08df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fd81c08e770 tx=0x7fd81800fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:57.637 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:57.622+0000 7fd8220b0700 1 --2- 172.21.15.42:0/2042665312 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd81c0a7740 0x7fd81c08df80 secure :-1 s=READY pgs=563 cs=0 l=1 rx=0x7fd81c08e770 tx=0x7fd81800fdb0).ready entity=mon.1 client_cookie=4be26eb9ce1e428e server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:14:57.637 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:57.622+0000 7fd8220b0700 1 --2- 172.21.15.42:0/2042665312 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd81c0a7740 0x7fd81c08df80 secure :-1 s=THROTTLE_DONE pgs=563 cs=0 l=1 rx=0x7fd81c08e770 tx=0x7fd81800fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:57.637 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:57.622+0000 7fd8220b0700 1 --2- 172.21.15.42:0/2042665312 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd81c0a7740 0x7fd81c08df80 secure :-1 s=THROTTLE_DONE pgs=563 cs=0 l=1 rx=0x7fd81c08e770 tx=0x7fd81800fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:57.637 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:57.622+0000 7fd8210ae700 1 -- 172.21.15.42:0/2042665312 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd818008c90 con 0x7fd81c0a7740 2019-12-05T05:14:57.637 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:57.622+0000 7fd8220b0700 1 --2- 172.21.15.42:0/2042665312 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd81c0a7740 0x7fd81c08df80 secure :-1 s=THROTTLE_DONE pgs=563 cs=0 l=1 rx=0x7fd81c08e770 tx=0x7fd81800fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:57.637 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:57.622+0000 7fd8210ae700 1 -- 172.21.15.42:0/2042665312 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8180088b0 con 0x7fd81c0a7740 2019-12-05T05:14:57.638 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:57.622+0000 7fd8210ae700 1 -- 172.21.15.42:0/2042665312 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd818011b50 con 0x7fd81c0a7740 2019-12-05T05:14:57.638 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:57.622+0000 7fd823b13700 1 -- 172.21.15.42:0/2042665312 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd81c0a7740 msgr2=0x7fd81c08df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:14:57.638 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:57.622+0000 7fd823b13700 1 --2- 172.21.15.42:0/2042665312 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd81c0a7740 0x7fd81c08df80 secure :-1 s=READY pgs=563 cs=0 l=1 rx=0x7fd81c08e770 tx=0x7fd81800fdb0).stop 2019-12-05T05:14:57.638 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:57.622+0000 7fd823b13700 1 -- 172.21.15.42:0/2042665312 shutdown_connections 2019-12-05T05:14:57.638 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:57.622+0000 7fd823b13700 1 --2- 172.21.15.42:0/2042665312 >> v2:172.21.15.40:3300/0 conn(0x7fd81c09f2e0 0x7fd81c09f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:14:57.638 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:57.622+0000 7fd823b13700 1 --2- 172.21.15.42:0/2042665312 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd81c0a7740 0x7fd81c08df80 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:14:57.638 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:57.622+0000 7fd823b13700 1 --2- 172.21.15.42:0/2042665312 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd81c07e720 0x7fd81c07eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:14:57.639 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:57.622+0000 7fd823b13700 1 -- 172.21.15.42:0/2042665312 shutdown_connections 2019-12-05T05:14:57.639 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:57.622+0000 7fd823b13700 1 -- 172.21.15.42:0/2042665312 wait complete. 2019-12-05T05:14:57.639 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:57.622+0000 7fd823b13700 1 -- 172.21.15.42:0/2042665312 >> 172.21.15.42:0/2042665312 conn(0x7fd81c0aabe0 msgr2=0x7fd81c0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:14:57.639 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:57.626+0000 7fd823b13700 1 Processor -- start 2019-12-05T05:14:57.639 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:57.626+0000 7fd823b13700 1 -- start start 2019-12-05T05:14:57.642 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:57.626+0000 7fd823b13700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd81c11b940 0x7fd81c123ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:14:57.642 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:57.626+0000 7fd823b13700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd81c1241f0 0x7fd81c137660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:14:57.642 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:57.626+0000 7fd823b13700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd81c11bea0 con 0x7fd81c11a100 2019-12-05T05:14:57.643 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:57.626+0000 7fd823b13700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd81c123a40 con 0x7fd81c1241f0 2019-12-05T05:14:57.643 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:57.626+0000 7fd823b13700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd81c124620 con 0x7fd81c11b940 2019-12-05T05:14:57.643 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:57.626+0000 7fd8218af700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd81c11b940 0x7fd81c123ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:14:57.643 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:57.626+0000 7fd8218af700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd81c11b940 0x7fd81c123ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:58656/0 (socket says 172.21.15.42:58656) 2019-12-05T05:14:57.643 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:57.626+0000 7fd8218af700 1 -- 172.21.15.42:0/3600784928 learned_addr learned my addr 172.21.15.42:0/3600784928 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:14:57.643 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:57.626+0000 7fd8228b1700 1 --2- 172.21.15.42:0/3600784928 >> v2:172.21.15.40:3300/0 conn(0x7fd81c1241f0 0x7fd81c137660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:14:57.644 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:57.630+0000 7fd8218af700 1 -- 172.21.15.42:0/3600784928 >> v2:172.21.15.40:3300/0 conn(0x7fd81c1241f0 msgr2=0x7fd81c137660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:14:57.644 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:57.630+0000 7fd8218af700 1 --2- 172.21.15.42:0/3600784928 >> v2:172.21.15.40:3300/0 conn(0x7fd81c1241f0 0x7fd81c137660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:14:57.644 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:57.630+0000 7fd8218af700 1 -- 172.21.15.42:0/3600784928 >> v1:172.21.15.40:6789/0 conn(0x7fd81c11a100 legacy=0x7fd81c07da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:14:57.644 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:57.630+0000 7fd80effd700 1 -- 172.21.15.42:0/3600784928 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2405474261 0 0) 0x7fd81c11bea0 con 0x7fd81c11a100 2019-12-05T05:14:57.644 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:57.630+0000 7fd8218af700 1 -- 172.21.15.42:0/3600784928 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd81c122d40 con 0x7fd81c11b940 2019-12-05T05:14:57.644 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:57.630+0000 7fd8218af700 1 --2- 172.21.15.42:0/3600784928 >> v2:172.21.15.42:3300/0 conn(0x7fd81c11b940 0x7fd81c123ca0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fd810003070 tx=0x7fd810010bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:57.644 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:57.630+0000 7fd8218af700 1 --2- 172.21.15.42:0/3600784928 >> v2:172.21.15.42:3300/0 conn(0x7fd81c11b940 0x7fd81c123ca0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fd810003070 tx=0x7fd810010bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:57.645 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:57.630+0000 7fd8218af700 1 --2- 172.21.15.42:0/3600784928 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd81c11b940 0x7fd81c123ca0 secure :-1 s=READY pgs=564 cs=0 l=1 rx=0x7fd810003070 tx=0x7fd810010bc0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:14:57.645 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:57.630+0000 7fd8218af700 1 --2- 172.21.15.42:0/3600784928 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd81c11b940 0x7fd81c123ca0 secure :-1 s=THROTTLE_DONE pgs=564 cs=0 l=1 rx=0x7fd810003070 tx=0x7fd810010bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:57.645 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:57.630+0000 7fd8218af700 1 --2- 172.21.15.42:0/3600784928 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd81c11b940 0x7fd81c123ca0 secure :-1 s=THROTTLE_DONE pgs=564 cs=0 l=1 rx=0x7fd810003070 tx=0x7fd810010bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:57.645 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:57.630+0000 7fd80effd700 1 -- 172.21.15.42:0/3600784928 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd810005890 con 0x7fd81c11b940 2019-12-05T05:14:57.645 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:57.630+0000 7fd8218af700 1 --2- 172.21.15.42:0/3600784928 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd81c11b940 0x7fd81c123ca0 secure :-1 s=THROTTLE_DONE pgs=564 cs=0 l=1 rx=0x7fd810003070 tx=0x7fd810010bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:57.645 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:57.630+0000 7fd80effd700 1 -- 172.21.15.42:0/3600784928 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd810007160 con 0x7fd81c11b940 2019-12-05T05:14:57.645 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:57.630+0000 7fd823b13700 1 -- 172.21.15.42:0/3600784928 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd81c137bb0 con 0x7fd81c11b940 2019-12-05T05:14:57.645 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:57.630+0000 7fd80effd700 1 -- 172.21.15.42:0/3600784928 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd810012240 con 0x7fd81c11b940 2019-12-05T05:14:57.646 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:57.630+0000 7fd823b13700 1 -- 172.21.15.42:0/3600784928 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd81c138130 con 0x7fd81c11b940 2019-12-05T05:14:57.646 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:57.630+0000 7fd8218af700 1 --2- 172.21.15.42:0/3600784928 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd81c11b940 0x7fd81c123ca0 secure :-1 s=THROTTLE_DONE pgs=564 cs=0 l=1 rx=0x7fd810003070 tx=0x7fd810010bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:57.646 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:57.630+0000 7fd80effd700 1 -- 172.21.15.42:0/3600784928 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fd810012550 con 0x7fd81c11b940 2019-12-05T05:14:57.646 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:57.630+0000 7fd8218af700 1 --2- 172.21.15.42:0/3600784928 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd81c11b940 0x7fd81c123ca0 secure :-1 s=THROTTLE_DONE pgs=564 cs=0 l=1 rx=0x7fd810003070 tx=0x7fd810010bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:57.646 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:57.630+0000 7fd80effd700 1 --2- 172.21.15.42:0/3600784928 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd80403acc0 0x7fd80403d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:14:57.646 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:57.630+0000 7fd80effd700 1 -- 172.21.15.42:0/3600784928 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6719+0+0 (secure 0 0 0) 0x7fd81004b150 con 0x7fd81c11b940 2019-12-05T05:14:57.646 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:57.630+0000 7fd8220b0700 1 --2- 172.21.15.42:0/3600784928 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd80403acc0 0x7fd80403d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:14:57.647 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:57.634+0000 7fd8220b0700 1 --2- 172.21.15.42:0/3600784928 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd80403acc0 0x7fd80403d1d0 crc :-1 s=READY pgs=256 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:14:57.647 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:57.634+0000 7fd823b13700 1 -- 172.21.15.42:0/3600784928 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd81c130210 con 0x7fd81c11b940 2019-12-05T05:14:57.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:57.642+0000 7fd8218af700 1 --2- 172.21.15.42:0/3600784928 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd81c11b940 0x7fd81c123ca0 secure :-1 s=THROTTLE_DONE pgs=564 cs=0 l=1 rx=0x7fd810003070 tx=0x7fd810010bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:57.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:57.642+0000 7fd80effd700 1 -- 172.21.15.42:0/3600784928 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fd8100321d0 con 0x7fd81c11b940 2019-12-05T05:14:57.817 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:57.810+0000 7fd823b13700 1 -- 172.21.15.42:0/3600784928 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd81c05d110 con 0x7fd81c11b940 2019-12-05T05:14:57.818 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:57.810+0000 7fd8218af700 1 --2- 172.21.15.42:0/3600784928 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd81c11b940 0x7fd81c123ca0 secure :-1 s=THROTTLE_DONE pgs=564 cs=0 l=1 rx=0x7fd810003070 tx=0x7fd810010bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:57.819 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:57.810+0000 7fd80effd700 1 -- 172.21.15.42:0/3600784928 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+2108 (secure 0 0 0) 0x7fd810032430 con 0x7fd81c11b940 2019-12-05T05:14:57.819 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 73 2019-12-05T05:14:57.819 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:14:57.819 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":73,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5629,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/3264829400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3264829400},{"type":"v1","addr":"172.21.15.42:6835","nonce":3264829400}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":73}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:14:17.896674+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4841},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4841":{"gid":4841,"name":"b","rank":0,"incarnation":69,"state":"up:active","state_seq":19,"addr":"172.21.15.40:6833/3150831963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3150831963},{"type":"v1","addr":"172.21.15.40:6833","nonce":3150831963}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:14:57.821 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:57.814+0000 7fd823b13700 1 -- 172.21.15.42:0/3600784928 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd80403acc0 msgr2=0x7fd80403d1d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:14:57.821 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:57.814+0000 7fd823b13700 1 --2- 172.21.15.42:0/3600784928 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd80403acc0 0x7fd80403d1d0 crc :-1 s=READY pgs=256 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:14:57.821 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:57.814+0000 7fd823b13700 1 -- 172.21.15.42:0/3600784928 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd81c11b940 msgr2=0x7fd81c123ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:14:57.822 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:57.814+0000 7fd823b13700 1 --2- 172.21.15.42:0/3600784928 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd81c11b940 0x7fd81c123ca0 secure :-1 s=READY pgs=564 cs=0 l=1 rx=0x7fd810003070 tx=0x7fd810010bc0).stop 2019-12-05T05:14:57.822 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:57.814+0000 7fd823b13700 1 -- 172.21.15.42:0/3600784928 shutdown_connections 2019-12-05T05:14:57.822 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:57.814+0000 7fd823b13700 1 --2- 172.21.15.42:0/3600784928 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd80403acc0 0x7fd80403d1d0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:14:57.822 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:57.814+0000 7fd823b13700 1 --2- 172.21.15.42:0/3600784928 >> v2:172.21.15.40:3300/0 conn(0x7fd81c1241f0 0x7fd81c137660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:14:57.822 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:57.814+0000 7fd823b13700 1 --2- 172.21.15.42:0/3600784928 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd81c11b940 0x7fd81c123ca0 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:14:57.822 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:57.814+0000 7fd823b13700 1 -- 172.21.15.42:0/3600784928 shutdown_connections 2019-12-05T05:14:57.822 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:57.814+0000 7fd823b13700 1 -- 172.21.15.42:0/3600784928 wait complete. 2019-12-05T05:14:57.823 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:57.814+0000 7fd823b13700 1 -- 172.21.15.42:0/3600784928 >> 172.21.15.42:0/3600784928 conn(0x7fd81c0a69e0 msgr2=0x7fd81c09f3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:14:57.830 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T05:14:59.831 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:14:59.939 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:59.926+0000 7fcb47122700 1 Processor -- start 2019-12-05T05:14:59.940 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:59.926+0000 7fcb47122700 1 -- start start 2019-12-05T05:14:59.940 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:59.926+0000 7fcb47122700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcb400a7740 0x7fcb4008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:14:59.940 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:59.926+0000 7fcb47122700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb400a0710 con 0x7fcb4007e720 2019-12-05T05:14:59.940 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:59.926+0000 7fcb47122700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb400a9ba0 con 0x7fcb4009fae0 2019-12-05T05:14:59.940 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:59.926+0000 7fcb47122700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb400a0b00 con 0x7fcb400a7740 2019-12-05T05:14:59.940 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:59.926+0000 7fcb45ec0700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fcb4009fae0 0x7fcb4014efe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:58706/0 (socket says 172.21.15.42:58706) 2019-12-05T05:14:59.941 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:59.926+0000 7fcb45ec0700 1 -- 172.21.15.42:0/3742089567 learned_addr learned my addr 172.21.15.42:0/3742089567 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:14:59.941 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:59.926+0000 7fcb456bf700 1 --2- 172.21.15.42:0/3742089567 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcb400a7740 0x7fcb4008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:14:59.941 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:59.930+0000 7fcb37fff700 1 -- 172.21.15.42:0/3742089567 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3680964642 0 0) 0x7fcb400a9ba0 con 0x7fcb4009fae0 2019-12-05T05:14:59.941 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:59.930+0000 7fcb37fff700 1 -- 172.21.15.42:0/3742089567 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb2c0030b0 con 0x7fcb4009fae0 2019-12-05T05:14:59.941 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:59.930+0000 7fcb37fff700 1 -- 172.21.15.42:0/3742089567 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3836950649 0 0) 0x7fcb400a0710 con 0x7fcb4007e720 2019-12-05T05:14:59.941 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:59.930+0000 7fcb37fff700 1 -- 172.21.15.42:0/3742089567 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb400a9ba0 con 0x7fcb4007e720 2019-12-05T05:14:59.942 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:59.930+0000 7fcb37fff700 1 -- 172.21.15.42:0/3742089567 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3973992277 0 0) 0x7fcb2c0030b0 con 0x7fcb4009fae0 2019-12-05T05:14:59.942 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:59.930+0000 7fcb37fff700 1 -- 172.21.15.42:0/3742089567 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcb400a7740 msgr2=0x7fcb4008df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:14:59.942 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:59.930+0000 7fcb37fff700 1 --2- 172.21.15.42:0/3742089567 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcb400a7740 0x7fcb4008df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:14:59.942 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:59.930+0000 7fcb37fff700 1 -- 172.21.15.42:0/3742089567 >> v1:172.21.15.40:6789/0 conn(0x7fcb4007e720 legacy=0x7fcb4009f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:14:59.942 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:59.930+0000 7fcb37fff700 1 -- 172.21.15.42:0/3742089567 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb4007eb50 con 0x7fcb4009fae0 2019-12-05T05:14:59.942 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:59.930+0000 7fcb37fff700 1 -- 172.21.15.42:0/3742089567 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7fcb3c001750 con 0x7fcb4009fae0 2019-12-05T05:14:59.943 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:59.930+0000 7fcb37fff700 1 -- 172.21.15.42:0/3742089567 >> v1:172.21.15.42:6789/0 conn(0x7fcb4009fae0 legacy=0x7fcb4014efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:14:59.943 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:59.930+0000 7fcb37fff700 1 --2- 172.21.15.42:0/3742089567 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcb2c003940 0x7fcb2c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:14:59.943 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:59.930+0000 7fcb37fff700 1 --2- 172.21.15.42:0/3742089567 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcb2c004420 0x7fcb2c004870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:14:59.943 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:59.930+0000 7fcb37fff700 1 --2- 172.21.15.42:0/3742089567 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcb2c004f60 0x7fcb2c04f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:14:59.943 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:59.930+0000 7fcb44ebe700 1 --2- 172.21.15.42:0/3742089567 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcb2c004420 0x7fcb2c004870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:14:59.943 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:59.930+0000 7fcb37fff700 1 -- 172.21.15.42:0/3742089567 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb400a0b00 con 0x7fcb2c004f60 2019-12-05T05:14:59.944 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:59.930+0000 7fcb37fff700 1 -- 172.21.15.42:0/3742089567 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb2c04fe40 con 0x7fcb2c004420 2019-12-05T05:14:59.944 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:59.930+0000 7fcb37fff700 1 -- 172.21.15.42:0/3742089567 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb2c050080 con 0x7fcb2c003940 2019-12-05T05:14:59.944 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:59.930+0000 7fcb456bf700 1 --2- 172.21.15.42:0/3742089567 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcb400a7740 0x7fcb4008df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:14:59.944 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:59.930+0000 7fcb45ec0700 1 --2- 172.21.15.42:0/3742089567 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcb2c004f60 0x7fcb2c04f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:14:59.945 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:59.930+0000 7fcb44ebe700 1 -- 172.21.15.42:0/3742089567 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcb2c003940 msgr2=0x7fcb2c003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:14:59.945 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:59.930+0000 7fcb44ebe700 1 --2- 172.21.15.42:0/3742089567 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcb2c003940 0x7fcb2c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:14:59.945 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:59.930+0000 7fcb44ebe700 1 -- 172.21.15.42:0/3742089567 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcb2c004f60 msgr2=0x7fcb2c04f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:14:59.945 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:59.930+0000 7fcb44ebe700 1 --2- 172.21.15.42:0/3742089567 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcb2c004f60 0x7fcb2c04f680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:14:59.945 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:59.930+0000 7fcb44ebe700 1 -- 172.21.15.42:0/3742089567 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb2c050380 con 0x7fcb2c004420 2019-12-05T05:14:59.945 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:59.930+0000 7fcb45ec0700 1 --2- 172.21.15.42:0/3742089567 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcb2c004f60 0x7fcb2c04f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).send_auth_request state changed! 2019-12-05T05:14:59.946 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:59.930+0000 7fcb44ebe700 1 --2- 172.21.15.42:0/3742089567 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcb2c004420 0x7fcb2c004870 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fcb30001000 tx=0x7fcb3000be70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:59.946 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:59.930+0000 7fcb44ebe700 1 --2- 172.21.15.42:0/3742089567 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcb2c004420 0x7fcb2c004870 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fcb30001000 tx=0x7fcb3000be70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:59.946 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:59.930+0000 7fcb44ebe700 1 --2- 172.21.15.42:0/3742089567 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcb2c004420 0x7fcb2c004870 secure :-1 s=READY pgs=566 cs=0 l=1 rx=0x7fcb30001000 tx=0x7fcb3000be70).ready entity=mon.1 client_cookie=4a575da2b22c8b53 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:14:59.946 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:59.930+0000 7fcb44ebe700 1 --2- 172.21.15.42:0/3742089567 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcb2c004420 0x7fcb2c004870 secure :-1 s=THROTTLE_DONE pgs=566 cs=0 l=1 rx=0x7fcb30001000 tx=0x7fcb3000be70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:59.946 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:59.930+0000 7fcb37fff700 1 -- 172.21.15.42:0/3742089567 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcb30010be0 con 0x7fcb2c004420 2019-12-05T05:14:59.946 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:59.930+0000 7fcb44ebe700 1 --2- 172.21.15.42:0/3742089567 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcb2c004420 0x7fcb2c004870 secure :-1 s=THROTTLE_DONE pgs=566 cs=0 l=1 rx=0x7fcb30001000 tx=0x7fcb3000be70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:59.946 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:59.930+0000 7fcb37fff700 1 -- 172.21.15.42:0/3742089567 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb30010800 con 0x7fcb2c004420 2019-12-05T05:14:59.947 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:59.930+0000 7fcb44ebe700 1 --2- 172.21.15.42:0/3742089567 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcb2c004420 0x7fcb2c004870 secure :-1 s=THROTTLE_DONE pgs=566 cs=0 l=1 rx=0x7fcb30001000 tx=0x7fcb3000be70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:59.947 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:59.930+0000 7fcb37fff700 1 -- 172.21.15.42:0/3742089567 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcb3000c5b0 con 0x7fcb2c004420 2019-12-05T05:14:59.947 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:59.930+0000 7fcb47122700 1 -- 172.21.15.42:0/3742089567 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcb2c004420 msgr2=0x7fcb2c004870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:14:59.947 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:59.930+0000 7fcb47122700 1 --2- 172.21.15.42:0/3742089567 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcb2c004420 0x7fcb2c004870 secure :-1 s=READY pgs=566 cs=0 l=1 rx=0x7fcb30001000 tx=0x7fcb3000be70).stop 2019-12-05T05:14:59.947 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:59.930+0000 7fcb45ec0700 1 -- 172.21.15.42:0/3742089567 reap_dead start 2019-12-05T05:14:59.947 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:59.930+0000 7fcb47122700 1 -- 172.21.15.42:0/3742089567 shutdown_connections 2019-12-05T05:14:59.948 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:59.930+0000 7fcb47122700 1 -- 172.21.15.42:0/3742089567 shutdown_connections 2019-12-05T05:14:59.948 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:59.930+0000 7fcb47122700 1 -- 172.21.15.42:0/3742089567 wait complete. 2019-12-05T05:14:59.948 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:59.930+0000 7fcb47122700 1 -- 172.21.15.42:0/3742089567 >> 172.21.15.42:0/3742089567 conn(0x7fcb400aabe0 msgr2=0x7fcb400a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:14:59.951 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:59.934+0000 7fcb47122700 1 Processor -- start 2019-12-05T05:14:59.951 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:59.934+0000 7fcb47122700 1 -- start start 2019-12-05T05:14:59.951 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:59.934+0000 7fcb47122700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fcb40130e80 0x7fcb4011b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:14:59.952 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:59.934+0000 7fcb47122700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcb400a5850 0x7fcb4007a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:14:59.952 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:59.934+0000 7fcb47122700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb4011bf40 con 0x7fcb40116500 2019-12-05T05:14:59.952 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:59.934+0000 7fcb47122700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb4011ba00 con 0x7fcb40130e80 2019-12-05T05:14:59.952 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:59.934+0000 7fcb47122700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb4011bca0 con 0x7fcb400a5850 2019-12-05T05:14:59.952 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:59.934+0000 7fcb456bf700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fcb40116500 0x7fcb40130680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:39922/0 (socket says 172.21.15.42:39922) 2019-12-05T05:14:59.952 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:59.934+0000 7fcb456bf700 1 -- 172.21.15.42:0/1431233697 learned_addr learned my addr 172.21.15.42:0/1431233697 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:14:59.953 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:59.934+0000 7fcb45ec0700 1 --2- 172.21.15.42:0/1431233697 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcb400a5850 0x7fcb4007a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:14:59.953 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:59.934+0000 7fcb44ebe700 1 --2- 172.21.15.42:0/1431233697 >> v2:172.21.15.40:3300/0 conn(0x7fcb40130e80 0x7fcb4011b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:14:59.953 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:59.934+0000 7fcb35ffb700 1 -- 172.21.15.42:0/1431233697 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1279881234 0 0) 0x7fcb4011bf40 con 0x7fcb40116500 2019-12-05T05:14:59.953 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:59.934+0000 7fcb35ffb700 1 -- 172.21.15.42:0/1431233697 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb200030b0 con 0x7fcb40116500 2019-12-05T05:14:59.953 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:59.934+0000 7fcb45ec0700 1 -- 172.21.15.42:0/1431233697 >> v2:172.21.15.40:3300/0 conn(0x7fcb40130e80 msgr2=0x7fcb4011b410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:14:59.953 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:59.934+0000 7fcb45ec0700 1 --2- 172.21.15.42:0/1431233697 >> v2:172.21.15.40:3300/0 conn(0x7fcb40130e80 0x7fcb4011b410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:14:59.954 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:59.934+0000 7fcb45ec0700 1 -- 172.21.15.42:0/1431233697 >> v1:172.21.15.40:6789/0 conn(0x7fcb40116500 legacy=0x7fcb40130680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:14:59.954 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:59.934+0000 7fcb45ec0700 1 -- 172.21.15.42:0/1431233697 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb4007ab70 con 0x7fcb400a5850 2019-12-05T05:14:59.954 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:59.934+0000 7fcb44ebe700 1 --2- 172.21.15.42:0/1431233697 >> v2:172.21.15.40:3300/0 conn(0x7fcb40130e80 0x7fcb4011b410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:14:59.954 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:59.934+0000 7fcb45ec0700 1 --2- 172.21.15.42:0/1431233697 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcb400a5850 0x7fcb4007a620 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fcb300011d0 tx=0x7fcb3c0107a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:59.954 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:59.934+0000 7fcb45ec0700 1 --2- 172.21.15.42:0/1431233697 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcb400a5850 0x7fcb4007a620 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fcb300011d0 tx=0x7fcb3c0107a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:59.954 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:59.934+0000 7fcb45ec0700 1 --2- 172.21.15.42:0/1431233697 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcb400a5850 0x7fcb4007a620 secure :-1 s=READY pgs=117 cs=0 l=1 rx=0x7fcb300011d0 tx=0x7fcb3c0107a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:14:59.955 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:59.934+0000 7fcb45ec0700 1 --2- 172.21.15.42:0/1431233697 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcb400a5850 0x7fcb4007a620 secure :-1 s=THROTTLE_DONE pgs=117 cs=0 l=1 rx=0x7fcb300011d0 tx=0x7fcb3c0107a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:59.955 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:59.934+0000 7fcb35ffb700 1 -- 172.21.15.42:0/1431233697 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcb3c011320 con 0x7fcb400a5850 2019-12-05T05:14:59.955 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:59.934+0000 7fcb45ec0700 1 --2- 172.21.15.42:0/1431233697 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcb400a5850 0x7fcb4007a620 secure :-1 s=THROTTLE_DONE pgs=117 cs=0 l=1 rx=0x7fcb300011d0 tx=0x7fcb3c0107a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:59.955 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:59.934+0000 7fcb45ec0700 1 --2- 172.21.15.42:0/1431233697 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcb400a5850 0x7fcb4007a620 secure :-1 s=THROTTLE_DONE pgs=117 cs=0 l=1 rx=0x7fcb300011d0 tx=0x7fcb3c0107a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:59.955 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:59.934+0000 7fcb35ffb700 1 -- 172.21.15.42:0/1431233697 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb3c010f40 con 0x7fcb400a5850 2019-12-05T05:14:59.955 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:59.938+0000 7fcb47122700 1 -- 172.21.15.42:0/1431233697 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb4007ade0 con 0x7fcb400a5850 2019-12-05T05:14:59.956 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:59.938+0000 7fcb35ffb700 1 -- 172.21.15.42:0/1431233697 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcb3c011900 con 0x7fcb400a5850 2019-12-05T05:14:59.956 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:59.938+0000 7fcb47122700 1 -- 172.21.15.42:0/1431233697 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb4007b360 con 0x7fcb400a5850 2019-12-05T05:14:59.956 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:59.938+0000 7fcb45ec0700 1 --2- 172.21.15.42:0/1431233697 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcb400a5850 0x7fcb4007a620 secure :-1 s=THROTTLE_DONE pgs=117 cs=0 l=1 rx=0x7fcb300011d0 tx=0x7fcb3c0107a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:59.956 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:59.938+0000 7fcb45ec0700 1 --2- 172.21.15.42:0/1431233697 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcb400a5850 0x7fcb4007a620 secure :-1 s=THROTTLE_DONE pgs=117 cs=0 l=1 rx=0x7fcb300011d0 tx=0x7fcb3c0107a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:59.956 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:59.938+0000 7fcb35ffb700 1 -- 172.21.15.42:0/1431233697 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fcb3c007180 con 0x7fcb400a5850 2019-12-05T05:14:59.957 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:59.938+0000 7fcb35ffb700 1 --2- 172.21.15.42:0/1431233697 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fcb2003b2d0 0x7fcb2003d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:14:59.957 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:59.938+0000 7fcb35ffb700 1 -- 172.21.15.42:0/1431233697 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6719+0+0 (secure 0 0 0) 0x7fcb3c04a730 con 0x7fcb400a5850 2019-12-05T05:14:59.957 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:59.938+0000 7fcb456bf700 1 --2- 172.21.15.42:0/1431233697 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fcb2003b2d0 0x7fcb2003d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:14:59.957 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:59.938+0000 7fcb456bf700 1 --2- 172.21.15.42:0/1431233697 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fcb2003b2d0 0x7fcb2003d7e0 crc :-1 s=READY pgs=257 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:14:59.957 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:59.942+0000 7fcb47122700 1 -- 172.21.15.42:0/1431233697 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb4005d110 con 0x7fcb400a5850 2019-12-05T05:14:59.959 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:59.950+0000 7fcb45ec0700 1 --2- 172.21.15.42:0/1431233697 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcb400a5850 0x7fcb4007a620 secure :-1 s=THROTTLE_DONE pgs=117 cs=0 l=1 rx=0x7fcb300011d0 tx=0x7fcb3c0107a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:14:59.959 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:14:59.950+0000 7fcb35ffb700 1 -- 172.21.15.42:0/1431233697 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fcb3c011b00 con 0x7fcb400a5850 2019-12-05T05:15:00.129 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:00.122+0000 7fcb47122700 1 -- 172.21.15.42:0/1431233697 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcb4005b8c0 con 0x7fcb400a5850 2019-12-05T05:15:00.130 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:00.122+0000 7fcb45ec0700 1 --2- 172.21.15.42:0/1431233697 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcb400a5850 0x7fcb4007a620 secure :-1 s=THROTTLE_DONE pgs=117 cs=0 l=1 rx=0x7fcb300011d0 tx=0x7fcb3c0107a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:00.130 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:00.122+0000 7fcb35ffb700 1 -- 172.21.15.42:0/1431233697 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+2108 (secure 0 0 0) 0x7fcb3c031770 con 0x7fcb400a5850 2019-12-05T05:15:00.132 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 73 2019-12-05T05:15:00.132 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:15:00.132 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":73,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5629,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/3264829400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3264829400},{"type":"v1","addr":"172.21.15.42:6835","nonce":3264829400}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":73}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:14:17.896674+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4841},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4841":{"gid":4841,"name":"b","rank":0,"incarnation":69,"state":"up:active","state_seq":19,"addr":"172.21.15.40:6833/3150831963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3150831963},{"type":"v1","addr":"172.21.15.40:6833","nonce":3150831963}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:15:00.136 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:00.126+0000 7fcb47122700 1 -- 172.21.15.42:0/1431233697 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fcb2003b2d0 msgr2=0x7fcb2003d7e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:15:00.136 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:00.126+0000 7fcb47122700 1 --2- 172.21.15.42:0/1431233697 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fcb2003b2d0 0x7fcb2003d7e0 crc :-1 s=READY pgs=257 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:15:00.136 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:00.126+0000 7fcb47122700 1 -- 172.21.15.42:0/1431233697 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcb400a5850 msgr2=0x7fcb4007a620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:15:00.137 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:00.126+0000 7fcb47122700 1 --2- 172.21.15.42:0/1431233697 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcb400a5850 0x7fcb4007a620 secure :-1 s=READY pgs=117 cs=0 l=1 rx=0x7fcb300011d0 tx=0x7fcb3c0107a0).stop 2019-12-05T05:15:00.137 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:00.126+0000 7fcb47122700 1 -- 172.21.15.42:0/1431233697 shutdown_connections 2019-12-05T05:15:00.137 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:00.126+0000 7fcb47122700 1 --2- 172.21.15.42:0/1431233697 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fcb2003b2d0 0x7fcb2003d7e0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:15:00.137 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:00.126+0000 7fcb47122700 1 --2- 172.21.15.42:0/1431233697 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcb400a5850 0x7fcb4007a620 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:15:00.137 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:00.126+0000 7fcb47122700 1 --2- 172.21.15.42:0/1431233697 >> v2:172.21.15.40:3300/0 conn(0x7fcb40130e80 0x7fcb4011b410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:15:00.137 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:00.126+0000 7fcb47122700 1 -- 172.21.15.42:0/1431233697 shutdown_connections 2019-12-05T05:15:00.138 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:00.126+0000 7fcb47122700 1 -- 172.21.15.42:0/1431233697 wait complete. 2019-12-05T05:15:00.138 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:00.126+0000 7fcb47122700 1 -- 172.21.15.42:0/1431233697 >> 172.21.15.42:0/1431233697 conn(0x7fcb4007ee20 msgr2=0x7fcb40059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:15:00.146 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T05:15:02.147 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:15:02.251 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:02.238+0000 7f8e71a81700 1 Processor -- start 2019-12-05T05:15:02.252 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:02.238+0000 7f8e71a81700 1 -- start start 2019-12-05T05:15:02.252 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:02.238+0000 7f8e71a81700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8e6c08df80 0x7f8e6c09f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:15:02.252 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:02.238+0000 7f8e71a81700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8e6c09f830 0x7f8e6c14efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:15:02.252 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:02.238+0000 7f8e71a81700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e6c09fc60 con 0x7f8e6c0a7740 2019-12-05T05:15:02.252 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:02.238+0000 7f8e71a81700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e6c0a0b00 con 0x7f8e6c08df80 2019-12-05T05:15:02.252 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:02.238+0000 7f8e71a81700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e6c08e5b0 con 0x7f8e6c09f830 2019-12-05T05:15:02.252 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:02.238+0000 7f8e7081f700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8e6c09f830 0x7f8e6c14efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:15:02.253 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:02.238+0000 7f8e637fe700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8e6c08df80 0x7f8e6c09f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:15:02.253 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:02.238+0000 7f8e7081f700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8e6c09f830 0x7f8e6c14efe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:59160/0 (socket says 172.21.15.42:59160) 2019-12-05T05:15:02.253 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:02.238+0000 7f8e637fe700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8e6c08df80 0x7f8e6c09f2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.42:46508/0 (socket says 172.21.15.42:46508) 2019-12-05T05:15:02.253 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:02.238+0000 7f8e7081f700 1 -- 172.21.15.42:0/2951881449 learned_addr learned my addr 172.21.15.42:0/2951881449 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:15:02.253 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:02.242+0000 7f8e637fe700 1 -- 172.21.15.42:0/2951881449 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8e6c09f830 msgr2=0x7f8e6c14efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:15:02.253 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:02.242+0000 7f8e637fe700 1 --2- 172.21.15.42:0/2951881449 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8e6c09f830 0x7f8e6c14efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:15:02.253 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:02.242+0000 7f8e637fe700 1 -- 172.21.15.42:0/2951881449 >> v1:172.21.15.40:6789/0 conn(0x7f8e6c0a7740 legacy=0x7f8e6c07e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:15:02.254 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:02.242+0000 7f8e637fe700 1 -- 172.21.15.42:0/2951881449 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e6c0a0710 con 0x7f8e6c08df80 2019-12-05T05:15:02.254 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:02.242+0000 7f8e7081f700 1 --2- 172.21.15.42:0/2951881449 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8e6c09f830 0x7f8e6c14efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:15:02.254 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:02.242+0000 7f8e637fe700 1 --2- 172.21.15.42:0/2951881449 >> v2:172.21.15.40:3300/0 conn(0x7f8e6c08df80 0x7f8e6c09f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f8e54008790 tx=0x7f8e5c001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:02.254 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:02.242+0000 7f8e637fe700 1 --2- 172.21.15.42:0/2951881449 >> v2:172.21.15.40:3300/0 conn(0x7f8e6c08df80 0x7f8e6c09f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f8e54008790 tx=0x7f8e5c001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:02.254 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:02.242+0000 7f8e637fe700 1 --2- 172.21.15.42:0/2951881449 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e6c08df80 0x7f8e6c09f2e0 secure :-1 s=READY pgs=449 cs=0 l=1 rx=0x7f8e54008790 tx=0x7f8e5c001150).ready entity=mon.0 client_cookie=30b9933f8bfec422 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:15:02.254 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:02.242+0000 7f8e637fe700 1 --2- 172.21.15.42:0/2951881449 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e6c08df80 0x7f8e6c09f2e0 secure :-1 s=THROTTLE_DONE pgs=449 cs=0 l=1 rx=0x7f8e54008790 tx=0x7f8e5c001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:02.254 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:02.242+0000 7f8e637fe700 1 --2- 172.21.15.42:0/2951881449 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e6c08df80 0x7f8e6c09f2e0 secure :-1 s=THROTTLE_DONE pgs=449 cs=0 l=1 rx=0x7f8e54008790 tx=0x7f8e5c001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:02.255 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:02.242+0000 7f8e637fe700 1 --2- 172.21.15.42:0/2951881449 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e6c08df80 0x7f8e6c09f2e0 secure :-1 s=THROTTLE_DONE pgs=449 cs=0 l=1 rx=0x7f8e54008790 tx=0x7f8e5c001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:02.255 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:02.242+0000 7f8e6b7fe700 1 -- 172.21.15.42:0/2951881449 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8e54008bb0 con 0x7f8e6c08df80 2019-12-05T05:15:02.255 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:02.242+0000 7f8e6b7fe700 1 -- 172.21.15.42:0/2951881449 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e540087d0 con 0x7f8e6c08df80 2019-12-05T05:15:02.255 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:02.242+0000 7f8e6b7fe700 1 -- 172.21.15.42:0/2951881449 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8e54011760 con 0x7f8e6c08df80 2019-12-05T05:15:02.255 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:02.242+0000 7f8e71a81700 1 -- 172.21.15.42:0/2951881449 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e6c08df80 msgr2=0x7f8e6c09f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:15:02.255 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:02.242+0000 7f8e71a81700 1 --2- 172.21.15.42:0/2951881449 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e6c08df80 0x7f8e6c09f2e0 secure :-1 s=READY pgs=449 cs=0 l=1 rx=0x7f8e54008790 tx=0x7f8e5c001150).stop 2019-12-05T05:15:02.255 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:02.242+0000 7f8e71a81700 1 -- 172.21.15.42:0/2951881449 shutdown_connections 2019-12-05T05:15:02.256 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:02.242+0000 7f8e71a81700 1 --2- 172.21.15.42:0/2951881449 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8e6c09f830 0x7f8e6c14efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:15:02.256 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:02.242+0000 7f8e71a81700 1 --2- 172.21.15.42:0/2951881449 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e6c08df80 0x7f8e6c09f2e0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:15:02.256 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:02.242+0000 7f8e71a81700 1 -- 172.21.15.42:0/2951881449 shutdown_connections 2019-12-05T05:15:02.256 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:02.242+0000 7f8e71a81700 1 -- 172.21.15.42:0/2951881449 wait complete. 2019-12-05T05:15:02.256 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:02.242+0000 7f8e71a81700 1 -- 172.21.15.42:0/2951881449 >> 172.21.15.42:0/2951881449 conn(0x7f8e6c0aabe0 msgr2=0x7f8e6c0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:15:02.259 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:02.246+0000 7f8e71a81700 1 Processor -- start 2019-12-05T05:15:02.259 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:02.246+0000 7f8e71a81700 1 -- start start 2019-12-05T05:15:02.259 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:02.246+0000 7f8e71a81700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f8e6c0a3a60 0x7f8e6c0a6d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:15:02.260 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:02.246+0000 7f8e71a81700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8e6c0a59e0 0x7f8e6c07c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:15:02.260 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:02.246+0000 7f8e71a81700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e6c0a67a0 con 0x7f8e6c145c50 2019-12-05T05:15:02.260 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:02.246+0000 7f8e71a81700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e6c0a6210 con 0x7f8e6c0a59e0 2019-12-05T05:15:02.260 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:02.246+0000 7f8e71a81700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e6c0a6a70 con 0x7f8e6c0a3a60 2019-12-05T05:15:02.260 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:02.246+0000 7f8e6bfff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f8e6c0a3a60 0x7f8e6c0a6d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:15:02.260 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:02.246+0000 7f8e637fe700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f8e6c145c50 0x7f8e6c146750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:58736/0 (socket says 172.21.15.42:58736) 2019-12-05T05:15:02.261 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:02.246+0000 7f8e637fe700 1 -- 172.21.15.42:0/770418866 learned_addr learned my addr 172.21.15.42:0/770418866 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:15:02.261 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:02.246+0000 7f8e7081f700 1 --2- 172.21.15.42:0/770418866 >> v2:172.21.15.40:3300/0 conn(0x7f8e6c0a59e0 0x7f8e6c07c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:15:02.261 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:02.246+0000 7f8e6bfff700 1 -- 172.21.15.42:0/770418866 >> v2:172.21.15.40:3300/0 conn(0x7f8e6c0a59e0 msgr2=0x7f8e6c07c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:15:02.261 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:02.246+0000 7f8e6bfff700 1 --2- 172.21.15.42:0/770418866 >> v2:172.21.15.40:3300/0 conn(0x7f8e6c0a59e0 0x7f8e6c07c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:15:02.261 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:02.246+0000 7f8e697fa700 1 -- 172.21.15.42:0/770418866 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 18391362 0 0) 0x7f8e6c0a67a0 con 0x7f8e6c145c50 2019-12-05T05:15:02.261 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:02.246+0000 7f8e6bfff700 1 -- 172.21.15.42:0/770418866 >> v1:172.21.15.42:6789/0 conn(0x7f8e6c145c50 legacy=0x7f8e6c146750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:15:02.261 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:02.246+0000 7f8e6bfff700 1 -- 172.21.15.42:0/770418866 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e6c0a5e10 con 0x7f8e6c0a3a60 2019-12-05T05:15:02.262 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:02.246+0000 7f8e6bfff700 1 --2- 172.21.15.42:0/770418866 >> v2:172.21.15.42:3300/0 conn(0x7f8e6c0a3a60 0x7f8e6c0a6d10 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f8e5c000fc0 tx=0x7f8e5c010a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:02.262 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:02.250+0000 7f8e6bfff700 1 --2- 172.21.15.42:0/770418866 >> v2:172.21.15.42:3300/0 conn(0x7f8e6c0a3a60 0x7f8e6c0a6d10 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f8e5c000fc0 tx=0x7f8e5c010a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:02.262 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:02.250+0000 7f8e6bfff700 1 --2- 172.21.15.42:0/770418866 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8e6c0a3a60 0x7f8e6c0a6d10 secure :-1 s=READY pgs=568 cs=0 l=1 rx=0x7f8e5c000fc0 tx=0x7f8e5c010a20).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:15:02.262 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:02.250+0000 7f8e6bfff700 1 --2- 172.21.15.42:0/770418866 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8e6c0a3a60 0x7f8e6c0a6d10 secure :-1 s=THROTTLE_DONE pgs=568 cs=0 l=1 rx=0x7f8e5c000fc0 tx=0x7f8e5c010a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:02.262 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:02.250+0000 7f8e697fa700 1 -- 172.21.15.42:0/770418866 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8e5c00caa0 con 0x7f8e6c0a3a60 2019-12-05T05:15:02.262 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:02.250+0000 7f8e6bfff700 1 --2- 172.21.15.42:0/770418866 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8e6c0a3a60 0x7f8e6c0a6d10 secure :-1 s=THROTTLE_DONE pgs=568 cs=0 l=1 rx=0x7f8e5c000fc0 tx=0x7f8e5c010a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:02.263 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:02.250+0000 7f8e697fa700 1 -- 172.21.15.42:0/770418866 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e5c00c6c0 con 0x7f8e6c0a3a60 2019-12-05T05:15:02.263 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:02.250+0000 7f8e6bfff700 1 --2- 172.21.15.42:0/770418866 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8e6c0a3a60 0x7f8e6c0a6d10 secure :-1 s=THROTTLE_DONE pgs=568 cs=0 l=1 rx=0x7f8e5c000fc0 tx=0x7f8e5c010a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:02.263 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:02.250+0000 7f8e71a81700 1 -- 172.21.15.42:0/770418866 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e6c07ce60 con 0x7f8e6c0a3a60 2019-12-05T05:15:02.263 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:02.250+0000 7f8e697fa700 1 -- 172.21.15.42:0/770418866 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8e5c0111e0 con 0x7f8e6c0a3a60 2019-12-05T05:15:02.263 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:02.250+0000 7f8e71a81700 1 -- 172.21.15.42:0/770418866 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e6c07d1c0 con 0x7f8e6c0a3a60 2019-12-05T05:15:02.263 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:02.250+0000 7f8e6bfff700 1 --2- 172.21.15.42:0/770418866 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8e6c0a3a60 0x7f8e6c0a6d10 secure :-1 s=THROTTLE_DONE pgs=568 cs=0 l=1 rx=0x7f8e5c000fc0 tx=0x7f8e5c010a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:02.264 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:02.250+0000 7f8e697fa700 1 -- 172.21.15.42:0/770418866 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f8e5c011500 con 0x7f8e6c0a3a60 2019-12-05T05:15:02.264 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:02.250+0000 7f8e6bfff700 1 --2- 172.21.15.42:0/770418866 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8e6c0a3a60 0x7f8e6c0a6d10 secure :-1 s=THROTTLE_DONE pgs=568 cs=0 l=1 rx=0x7f8e5c000fc0 tx=0x7f8e5c010a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:02.264 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:02.250+0000 7f8e697fa700 1 --2- 172.21.15.42:0/770418866 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8e5003acd0 0x7f8e5003d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:15:02.264 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:02.250+0000 7f8e697fa700 1 -- 172.21.15.42:0/770418866 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6719+0+0 (secure 0 0 0) 0x7f8e5c04b2f0 con 0x7f8e6c0a3a60 2019-12-05T05:15:02.264 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:02.250+0000 7f8e637fe700 1 --2- 172.21.15.42:0/770418866 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8e5003acd0 0x7f8e5003d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:15:02.264 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:02.250+0000 7f8e637fe700 1 --2- 172.21.15.42:0/770418866 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8e5003acd0 0x7f8e5003d1e0 crc :-1 s=READY pgs=258 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:15:02.265 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:02.254+0000 7f8e71a81700 1 -- 172.21.15.42:0/770418866 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e6c05d110 con 0x7f8e6c0a3a60 2019-12-05T05:15:02.268 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:02.262+0000 7f8e6bfff700 1 --2- 172.21.15.42:0/770418866 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8e6c0a3a60 0x7f8e6c0a6d10 secure :-1 s=THROTTLE_DONE pgs=568 cs=0 l=1 rx=0x7f8e5c000fc0 tx=0x7f8e5c010a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:02.268 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:02.262+0000 7f8e697fa700 1 -- 172.21.15.42:0/770418866 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f8e5c032350 con 0x7f8e6c0a3a60 2019-12-05T05:15:02.438 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:02.430+0000 7f8e71a81700 1 -- 172.21.15.42:0/770418866 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8e6c05b8c0 con 0x7f8e6c0a3a60 2019-12-05T05:15:02.438 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:02.430+0000 7f8e6bfff700 1 --2- 172.21.15.42:0/770418866 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8e6c0a3a60 0x7f8e6c0a6d10 secure :-1 s=THROTTLE_DONE pgs=568 cs=0 l=1 rx=0x7f8e5c000fc0 tx=0x7f8e5c010a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:02.438 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:02.430+0000 7f8e697fa700 1 -- 172.21.15.42:0/770418866 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+2108 (secure 0 0 0) 0x7f8e5c0325b0 con 0x7f8e6c0a3a60 2019-12-05T05:15:02.439 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 73 2019-12-05T05:15:02.439 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:15:02.439 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":73,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5629,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/3264829400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3264829400},{"type":"v1","addr":"172.21.15.42:6835","nonce":3264829400}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":73}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:14:17.896674+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4841},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4841":{"gid":4841,"name":"b","rank":0,"incarnation":69,"state":"up:active","state_seq":19,"addr":"172.21.15.40:6833/3150831963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3150831963},{"type":"v1","addr":"172.21.15.40:6833","nonce":3150831963}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:15:02.442 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:02.434+0000 7f8e71a81700 1 -- 172.21.15.42:0/770418866 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8e5003acd0 msgr2=0x7f8e5003d1e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:15:02.442 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:02.434+0000 7f8e71a81700 1 --2- 172.21.15.42:0/770418866 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8e5003acd0 0x7f8e5003d1e0 crc :-1 s=READY pgs=258 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:15:02.443 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:02.434+0000 7f8e71a81700 1 -- 172.21.15.42:0/770418866 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8e6c0a3a60 msgr2=0x7f8e6c0a6d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:15:02.443 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:02.434+0000 7f8e71a81700 1 --2- 172.21.15.42:0/770418866 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8e6c0a3a60 0x7f8e6c0a6d10 secure :-1 s=READY pgs=568 cs=0 l=1 rx=0x7f8e5c000fc0 tx=0x7f8e5c010a20).stop 2019-12-05T05:15:02.443 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:02.434+0000 7f8e71a81700 1 -- 172.21.15.42:0/770418866 shutdown_connections 2019-12-05T05:15:02.443 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:02.434+0000 7f8e71a81700 1 --2- 172.21.15.42:0/770418866 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8e5003acd0 0x7f8e5003d1e0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:15:02.443 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:02.434+0000 7f8e71a81700 1 --2- 172.21.15.42:0/770418866 >> v2:172.21.15.40:3300/0 conn(0x7f8e6c0a59e0 0x7f8e6c07c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:15:02.443 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:02.434+0000 7f8e71a81700 1 --2- 172.21.15.42:0/770418866 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8e6c0a3a60 0x7f8e6c0a6d10 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:15:02.444 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:02.434+0000 7f8e71a81700 1 -- 172.21.15.42:0/770418866 shutdown_connections 2019-12-05T05:15:02.444 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:02.434+0000 7f8e71a81700 1 -- 172.21.15.42:0/770418866 wait complete. 2019-12-05T05:15:02.444 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:02.434+0000 7f8e71a81700 1 -- 172.21.15.42:0/770418866 >> 172.21.15.42:0/770418866 conn(0x7f8e6c0a82a0 msgr2=0x7f8e6c119440 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:15:02.456 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T05:15:04.457 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:15:04.561 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:04.546+0000 7f472b010700 1 Processor -- start 2019-12-05T05:15:04.562 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:04.550+0000 7f472b010700 1 -- start start 2019-12-05T05:15:04.562 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:04.550+0000 7f472b010700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f472409fad0 0x7f472414ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:15:04.562 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:04.550+0000 7f472b010700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47240a0700 con 0x7f47240a7730 2019-12-05T05:15:04.562 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:04.550+0000 7f472b010700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f472407ef10 con 0x7f472408df70 2019-12-05T05:15:04.562 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:04.550+0000 7f472b010700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47240a0af0 con 0x7f472409fad0 2019-12-05T05:15:04.562 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:04.550+0000 7f4728dac700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f472408df70 0x7f472409f2d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:58744/0 (socket says 172.21.15.42:58744) 2019-12-05T05:15:04.563 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:04.550+0000 7f4728dac700 1 -- 172.21.15.42:0/1210780994 learned_addr learned my addr 172.21.15.42:0/1210780994 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:15:04.563 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:04.550+0000 7f4729dae700 1 --2- 172.21.15.42:0/1210780994 >> v2:172.21.15.40:3300/0 conn(0x7f472409fad0 0x7f472414ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:15:04.563 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:04.550+0000 7f4723fff700 1 -- 172.21.15.42:0/1210780994 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 506974953 0 0) 0x7f472407ef10 con 0x7f472408df70 2019-12-05T05:15:04.563 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:04.550+0000 7f4723fff700 1 -- 172.21.15.42:0/1210780994 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f470c0030b0 con 0x7f472408df70 2019-12-05T05:15:04.563 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:04.550+0000 7f4729dae700 1 -- 172.21.15.42:0/1210780994 >> v1:172.21.15.42:6789/0 conn(0x7f472408df70 legacy=0x7f472409f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:15:04.563 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:04.550+0000 7f4729dae700 1 -- 172.21.15.42:0/1210780994 >> v1:172.21.15.40:6789/0 conn(0x7f47240a7730 legacy=0x7f472407e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:15:04.564 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:04.550+0000 7f4723fff700 1 -- 172.21.15.42:0/1210780994 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3258230006 0 0) 0x7f470c0030b0 con 0x7f472408df70 2019-12-05T05:15:04.564 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:04.550+0000 7f4729dae700 1 -- 172.21.15.42:0/1210780994 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47240a9b90 con 0x7f472409fad0 2019-12-05T05:15:04.564 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:04.550+0000 7f4729dae700 1 --2- 172.21.15.42:0/1210780994 >> v2:172.21.15.40:3300/0 conn(0x7f472409fad0 0x7f472414ffe0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f4714001110 tx=0x7f4718001820).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:04.564 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:04.550+0000 7f4729dae700 1 --2- 172.21.15.42:0/1210780994 >> v2:172.21.15.40:3300/0 conn(0x7f472409fad0 0x7f472414ffe0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f4714001110 tx=0x7f4718001820).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:04.564 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:04.550+0000 7f4729dae700 1 --2- 172.21.15.42:0/1210780994 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f472409fad0 0x7f472414ffe0 secure :-1 s=READY pgs=451 cs=0 l=1 rx=0x7f4714001110 tx=0x7f4718001820).ready entity=mon.0 client_cookie=aa28b22b82f27ece server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:15:04.565 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:04.550+0000 7f4729dae700 1 --2- 172.21.15.42:0/1210780994 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f472409fad0 0x7f472414ffe0 secure :-1 s=THROTTLE_DONE pgs=451 cs=0 l=1 rx=0x7f4714001110 tx=0x7f4718001820).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:04.565 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:04.550+0000 7f4723fff700 1 -- 172.21.15.42:0/1210780994 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4718001ad0 con 0x7f472409fad0 2019-12-05T05:15:04.565 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:04.550+0000 7f4729dae700 1 --2- 172.21.15.42:0/1210780994 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f472409fad0 0x7f472414ffe0 secure :-1 s=THROTTLE_DONE pgs=451 cs=0 l=1 rx=0x7f4714001110 tx=0x7f4718001820).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:04.565 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:04.550+0000 7f4729dae700 1 --2- 172.21.15.42:0/1210780994 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f472409fad0 0x7f472414ffe0 secure :-1 s=THROTTLE_DONE pgs=451 cs=0 l=1 rx=0x7f4714001110 tx=0x7f4718001820).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:04.566 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:04.554+0000 7f4723fff700 1 -- 172.21.15.42:0/1210780994 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47180018e0 con 0x7f472409fad0 2019-12-05T05:15:04.566 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:04.554+0000 7f4723fff700 1 -- 172.21.15.42:0/1210780994 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f471c008d60 con 0x7f472409fad0 2019-12-05T05:15:04.566 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:04.554+0000 7f472b010700 1 -- 172.21.15.42:0/1210780994 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f472409fad0 msgr2=0x7f472414ffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:15:04.566 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:04.554+0000 7f472b010700 1 --2- 172.21.15.42:0/1210780994 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f472409fad0 0x7f472414ffe0 secure :-1 s=READY pgs=451 cs=0 l=1 rx=0x7f4714001110 tx=0x7f4718001820).stop 2019-12-05T05:15:04.567 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:04.554+0000 7f472b010700 1 -- 172.21.15.42:0/1210780994 shutdown_connections 2019-12-05T05:15:04.567 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:04.554+0000 7f472b010700 1 --2- 172.21.15.42:0/1210780994 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f472409fad0 0x7f472414ffe0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:15:04.567 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:04.554+0000 7f472b010700 1 -- 172.21.15.42:0/1210780994 shutdown_connections 2019-12-05T05:15:04.567 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:04.554+0000 7f472b010700 1 -- 172.21.15.42:0/1210780994 wait complete. 2019-12-05T05:15:04.567 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:04.554+0000 7f472b010700 1 -- 172.21.15.42:0/1210780994 >> 172.21.15.42:0/1210780994 conn(0x7f47240aabd0 msgr2=0x7f47240a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:15:04.571 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:04.554+0000 7f472b010700 1 Processor -- start 2019-12-05T05:15:04.571 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:04.554+0000 7f472b010700 1 -- start start 2019-12-05T05:15:04.571 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:04.554+0000 7f472b010700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f472407cd60 0x7f47240a66a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:15:04.571 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:04.558+0000 7f472b010700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f472412dc60 0x7f472407bec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:15:04.571 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:04.558+0000 7f472b010700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47240a6bf0 con 0x7f472407ba90 2019-12-05T05:15:04.572 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:04.558+0000 7f472b010700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f472407d190 con 0x7f472407cd60 2019-12-05T05:15:04.572 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:04.558+0000 7f472b010700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f472407c610 con 0x7f472412dc60 2019-12-05T05:15:04.572 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:04.558+0000 7f4728dac700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f472407ba90 0x7f472412d460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:58750/0 (socket says 172.21.15.42:58750) 2019-12-05T05:15:04.572 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:04.558+0000 7f4728dac700 1 -- 172.21.15.42:0/188842879 learned_addr learned my addr 172.21.15.42:0/188842879 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:15:04.572 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:04.558+0000 7f4729dae700 1 --2- 172.21.15.42:0/188842879 >> v2:172.21.15.42:3300/0 conn(0x7f472412dc60 0x7f472407bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:15:04.573 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:04.558+0000 7f47295ad700 1 --2- 172.21.15.42:0/188842879 >> v2:172.21.15.40:3300/0 conn(0x7f472407cd60 0x7f47240a66a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:15:04.573 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:04.558+0000 7f4729dae700 1 -- 172.21.15.42:0/188842879 >> v2:172.21.15.40:3300/0 conn(0x7f472407cd60 msgr2=0x7f47240a66a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:15:04.573 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:04.558+0000 7f4729dae700 1 --2- 172.21.15.42:0/188842879 >> v2:172.21.15.40:3300/0 conn(0x7f472407cd60 0x7f47240a66a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:15:04.573 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:04.558+0000 7f4721ffb700 1 -- 172.21.15.42:0/188842879 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1211034028 0 0) 0x7f47240a6bf0 con 0x7f472407ba90 2019-12-05T05:15:04.573 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:04.558+0000 7f4729dae700 1 -- 172.21.15.42:0/188842879 >> v1:172.21.15.42:6789/0 conn(0x7f472407ba90 legacy=0x7f472412d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:15:04.573 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:04.558+0000 7f4729dae700 1 -- 172.21.15.42:0/188842879 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f472408eb60 con 0x7f472412dc60 2019-12-05T05:15:04.574 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:04.558+0000 7f4729dae700 1 --2- 172.21.15.42:0/188842879 >> v2:172.21.15.42:3300/0 conn(0x7f472412dc60 0x7f472407bec0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f47241220c0 tx=0x7f471c008b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:04.574 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:04.558+0000 7f4729dae700 1 --2- 172.21.15.42:0/188842879 >> v2:172.21.15.42:3300/0 conn(0x7f472412dc60 0x7f472407bec0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f47241220c0 tx=0x7f471c008b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:04.574 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:04.558+0000 7f4729dae700 1 --2- 172.21.15.42:0/188842879 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f472412dc60 0x7f472407bec0 secure :-1 s=READY pgs=571 cs=0 l=1 rx=0x7f47241220c0 tx=0x7f471c008b70).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:15:04.575 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:04.558+0000 7f4729dae700 1 --2- 172.21.15.42:0/188842879 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f472412dc60 0x7f472407bec0 secure :-1 s=THROTTLE_DONE pgs=571 cs=0 l=1 rx=0x7f47241220c0 tx=0x7f471c008b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:04.575 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:04.558+0000 7f4721ffb700 1 -- 172.21.15.42:0/188842879 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f471c016060 con 0x7f472412dc60 2019-12-05T05:15:04.575 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:04.558+0000 7f4729dae700 1 --2- 172.21.15.42:0/188842879 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f472412dc60 0x7f472407bec0 secure :-1 s=THROTTLE_DONE pgs=571 cs=0 l=1 rx=0x7f47241220c0 tx=0x7f471c008b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:04.575 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:04.558+0000 7f4729dae700 1 --2- 172.21.15.42:0/188842879 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f472412dc60 0x7f472407bec0 secure :-1 s=THROTTLE_DONE pgs=571 cs=0 l=1 rx=0x7f47241220c0 tx=0x7f471c008b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:04.576 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:04.558+0000 7f4721ffb700 1 -- 172.21.15.42:0/188842879 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f471c008980 con 0x7f472412dc60 2019-12-05T05:15:04.576 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:04.558+0000 7f4721ffb700 1 -- 172.21.15.42:0/188842879 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f471c0119e0 con 0x7f472412dc60 2019-12-05T05:15:04.576 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:04.558+0000 7f472b010700 1 -- 172.21.15.42:0/188842879 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f472408ee50 con 0x7f472412dc60 2019-12-05T05:15:04.576 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:04.558+0000 7f472b010700 1 -- 172.21.15.42:0/188842879 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f472408f320 con 0x7f472412dc60 2019-12-05T05:15:04.576 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:04.558+0000 7f4729dae700 1 --2- 172.21.15.42:0/188842879 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f472412dc60 0x7f472407bec0 secure :-1 s=THROTTLE_DONE pgs=571 cs=0 l=1 rx=0x7f47241220c0 tx=0x7f471c008b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:04.577 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:04.558+0000 7f4721ffb700 1 -- 172.21.15.42:0/188842879 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f471c011cb0 con 0x7f472412dc60 2019-12-05T05:15:04.577 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:04.558+0000 7f4729dae700 1 --2- 172.21.15.42:0/188842879 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f472412dc60 0x7f472407bec0 secure :-1 s=THROTTLE_DONE pgs=571 cs=0 l=1 rx=0x7f47241220c0 tx=0x7f471c008b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:04.577 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:04.562+0000 7f4721ffb700 1 --2- 172.21.15.42:0/188842879 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f470403ac10 0x7f470403d120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:15:04.577 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:04.562+0000 7f4721ffb700 1 -- 172.21.15.42:0/188842879 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6719+0+0 (secure 0 0 0) 0x7f471c04b010 con 0x7f472412dc60 2019-12-05T05:15:04.577 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:04.562+0000 7f47295ad700 1 --2- 172.21.15.42:0/188842879 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f470403ac10 0x7f470403d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:15:04.578 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:04.562+0000 7f47295ad700 1 --2- 172.21.15.42:0/188842879 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f470403ac10 0x7f470403d120 crc :-1 s=READY pgs=259 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:15:04.578 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:04.562+0000 7f472b010700 1 -- 172.21.15.42:0/188842879 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f472407c410 con 0x7f472412dc60 2019-12-05T05:15:04.579 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:04.570+0000 7f4729dae700 1 --2- 172.21.15.42:0/188842879 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f472412dc60 0x7f472407bec0 secure :-1 s=THROTTLE_DONE pgs=571 cs=0 l=1 rx=0x7f47241220c0 tx=0x7f471c008b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:04.579 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:04.570+0000 7f4721ffb700 1 -- 172.21.15.42:0/188842879 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f471c0322f0 con 0x7f472412dc60 2019-12-05T05:15:04.742 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:04.734+0000 7f472b010700 1 -- 172.21.15.42:0/188842879 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f472405d100 con 0x7f472412dc60 2019-12-05T05:15:04.743 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:04.734+0000 7f4729dae700 1 --2- 172.21.15.42:0/188842879 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f472412dc60 0x7f472407bec0 secure :-1 s=THROTTLE_DONE pgs=571 cs=0 l=1 rx=0x7f47241220c0 tx=0x7f471c008b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:04.743 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:04.734+0000 7f4721ffb700 1 -- 172.21.15.42:0/188842879 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+2108 (secure 0 0 0) 0x7f471c015b10 con 0x7f472412dc60 2019-12-05T05:15:04.744 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 73 2019-12-05T05:15:04.744 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:15:04.744 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":73,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5629,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/3264829400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3264829400},{"type":"v1","addr":"172.21.15.42:6835","nonce":3264829400}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":73}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:14:17.896674+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4841},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4841":{"gid":4841,"name":"b","rank":0,"incarnation":69,"state":"up:active","state_seq":19,"addr":"172.21.15.40:6833/3150831963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3150831963},{"type":"v1","addr":"172.21.15.40:6833","nonce":3150831963}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:15:04.746 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:04.738+0000 7f472b010700 1 -- 172.21.15.42:0/188842879 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f470403ac10 msgr2=0x7f470403d120 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:15:04.746 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:04.738+0000 7f472b010700 1 --2- 172.21.15.42:0/188842879 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f470403ac10 0x7f470403d120 crc :-1 s=READY pgs=259 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:15:04.746 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:04.738+0000 7f472b010700 1 -- 172.21.15.42:0/188842879 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f472412dc60 msgr2=0x7f472407bec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:15:04.746 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:04.738+0000 7f472b010700 1 --2- 172.21.15.42:0/188842879 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f472412dc60 0x7f472407bec0 secure :-1 s=READY pgs=571 cs=0 l=1 rx=0x7f47241220c0 tx=0x7f471c008b70).stop 2019-12-05T05:15:04.747 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:04.738+0000 7f472b010700 1 -- 172.21.15.42:0/188842879 shutdown_connections 2019-12-05T05:15:04.747 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:04.738+0000 7f472b010700 1 --2- 172.21.15.42:0/188842879 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f470403ac10 0x7f470403d120 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:15:04.747 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:04.738+0000 7f472b010700 1 --2- 172.21.15.42:0/188842879 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f472412dc60 0x7f472407bec0 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:15:04.747 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:04.738+0000 7f472b010700 1 --2- 172.21.15.42:0/188842879 >> v2:172.21.15.40:3300/0 conn(0x7f472407cd60 0x7f47240a66a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:15:04.747 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:04.738+0000 7f472b010700 1 -- 172.21.15.42:0/188842879 shutdown_connections 2019-12-05T05:15:04.747 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:04.738+0000 7f472b010700 1 -- 172.21.15.42:0/188842879 wait complete. 2019-12-05T05:15:04.748 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:04.738+0000 7f472b010700 1 -- 172.21.15.42:0/188842879 >> 172.21.15.42:0/188842879 conn(0x7f4724066fd0 msgr2=0x7f472411c800 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:15:04.756 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T05:15:05.589 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:15:05.596 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:15:06.757 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:15:06.861 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:06.846+0000 7f6228d90700 1 Processor -- start 2019-12-05T05:15:06.861 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:06.846+0000 7f6228d90700 1 -- start start 2019-12-05T05:15:06.861 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:06.850+0000 7f6228d90700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f62240a7740 0x7f622408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:15:06.861 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:06.850+0000 7f6228d90700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f622409fae0 0x7f622407eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:15:06.862 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:06.850+0000 7f6228d90700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62240a0710 con 0x7f622407e720 2019-12-05T05:15:06.862 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:06.850+0000 7f6228d90700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62240a0b00 con 0x7f62240a7740 2019-12-05T05:15:06.862 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:06.850+0000 7f6228d90700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f622408e5b0 con 0x7f622409fae0 2019-12-05T05:15:06.862 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:06.850+0000 7f622359e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f622409fae0 0x7f622407eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:15:06.862 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:06.850+0000 7f6222d9d700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f62240a7740 0x7f622408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:15:06.862 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:06.850+0000 7f6222d9d700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f62240a7740 0x7f622408df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.42:46546/0 (socket says 172.21.15.42:46546) 2019-12-05T05:15:06.862 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:06.850+0000 7f622359e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f622409fae0 0x7f622407eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:58734/0 (socket says 172.21.15.42:58734) 2019-12-05T05:15:06.863 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:06.850+0000 7f6222d9d700 1 -- 172.21.15.42:0/3411985186 learned_addr learned my addr 172.21.15.42:0/3411985186 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:15:06.863 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:06.850+0000 7f622359e700 1 -- 172.21.15.42:0/3411985186 >> v2:172.21.15.40:3300/0 conn(0x7f62240a7740 msgr2=0x7f622408df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:15:06.863 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:06.850+0000 7f622359e700 1 --2- 172.21.15.42:0/3411985186 >> v2:172.21.15.40:3300/0 conn(0x7f62240a7740 0x7f622408df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:15:06.863 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:06.850+0000 7f622359e700 1 -- 172.21.15.42:0/3411985186 >> v1:172.21.15.40:6789/0 conn(0x7f622407e720 legacy=0x7f622409f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:15:06.863 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:06.850+0000 7f622359e700 1 -- 172.21.15.42:0/3411985186 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f622407f0a0 con 0x7f622409fae0 2019-12-05T05:15:06.863 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:06.850+0000 7f622359e700 1 --2- 172.21.15.42:0/3411985186 >> v2:172.21.15.42:3300/0 conn(0x7f622409fae0 0x7f622407eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f6214002f30 tx=0x7f621400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:06.863 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:06.850+0000 7f622359e700 1 --2- 172.21.15.42:0/3411985186 >> v2:172.21.15.42:3300/0 conn(0x7f622409fae0 0x7f622407eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f6214002f30 tx=0x7f621400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:06.864 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:06.850+0000 7f622359e700 1 --2- 172.21.15.42:0/3411985186 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f622409fae0 0x7f622407eb50 secure :-1 s=READY pgs=572 cs=0 l=1 rx=0x7f6214002f30 tx=0x7f621400fc70).ready entity=mon.1 client_cookie=3cb5baad0b0f3e94 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:15:06.864 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:06.850+0000 7f622359e700 1 --2- 172.21.15.42:0/3411985186 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f622409fae0 0x7f622407eb50 secure :-1 s=THROTTLE_DONE pgs=572 cs=0 l=1 rx=0x7f6214002f30 tx=0x7f621400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:06.864 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:06.850+0000 7f622359e700 1 --2- 172.21.15.42:0/3411985186 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f622409fae0 0x7f622407eb50 secure :-1 s=THROTTLE_DONE pgs=572 cs=0 l=1 rx=0x7f6214002f30 tx=0x7f621400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:06.864 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:06.850+0000 7f6221d9b700 1 -- 172.21.15.42:0/3411985186 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6214008bb0 con 0x7f622409fae0 2019-12-05T05:15:06.864 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:06.850+0000 7f622359e700 1 --2- 172.21.15.42:0/3411985186 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f622409fae0 0x7f622407eb50 secure :-1 s=THROTTLE_DONE pgs=572 cs=0 l=1 rx=0x7f6214002f30 tx=0x7f621400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:06.864 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:06.850+0000 7f6221d9b700 1 -- 172.21.15.42:0/3411985186 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62140087d0 con 0x7f622409fae0 2019-12-05T05:15:06.864 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:06.850+0000 7f6221d9b700 1 -- 172.21.15.42:0/3411985186 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6214011820 con 0x7f622409fae0 2019-12-05T05:15:06.865 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:06.850+0000 7f6228d90700 1 -- 172.21.15.42:0/3411985186 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f622409fae0 msgr2=0x7f622407eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:15:06.865 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:06.850+0000 7f6228d90700 1 --2- 172.21.15.42:0/3411985186 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f622409fae0 0x7f622407eb50 secure :-1 s=READY pgs=572 cs=0 l=1 rx=0x7f6214002f30 tx=0x7f621400fc70).stop 2019-12-05T05:15:06.865 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:06.850+0000 7f6228d90700 1 -- 172.21.15.42:0/3411985186 shutdown_connections 2019-12-05T05:15:06.865 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:06.850+0000 7f6228d90700 1 --2- 172.21.15.42:0/3411985186 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f622409fae0 0x7f622407eb50 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:15:06.865 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:06.850+0000 7f6228d90700 1 --2- 172.21.15.42:0/3411985186 >> v2:172.21.15.40:3300/0 conn(0x7f62240a7740 0x7f622408df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:15:06.865 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:06.854+0000 7f6228d90700 1 -- 172.21.15.42:0/3411985186 shutdown_connections 2019-12-05T05:15:06.865 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:06.854+0000 7f6228d90700 1 -- 172.21.15.42:0/3411985186 wait complete. 2019-12-05T05:15:06.866 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:06.854+0000 7f6228d90700 1 -- 172.21.15.42:0/3411985186 >> 172.21.15.42:0/3411985186 conn(0x7f62240aabe0 msgr2=0x7f62240a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:15:06.869 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:06.854+0000 7f6228d90700 1 Processor -- start 2019-12-05T05:15:06.869 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:06.854+0000 7f6228d90700 1 -- start start 2019-12-05T05:15:06.869 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:06.854+0000 7f6228d90700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6224142870 0x7f62240a5b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:15:06.869 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:06.854+0000 7f6228d90700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6224142ca0 con 0x7f62241166c0 2019-12-05T05:15:06.869 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:06.854+0000 7f6228d90700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62240a5470 con 0x7f62240a4840 2019-12-05T05:15:06.870 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:06.854+0000 7f6228d90700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62241448e0 con 0x7f6224142870 2019-12-05T05:15:06.870 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:06.858+0000 7f622359e700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f62240a4840 0x7f62240a4c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:58778/0 (socket says 172.21.15.42:58778) 2019-12-05T05:15:06.870 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:06.858+0000 7f622359e700 1 -- 172.21.15.42:0/3131311498 learned_addr learned my addr 172.21.15.42:0/3131311498 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:15:06.870 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:06.858+0000 7f622259c700 1 --2- 172.21.15.42:0/3131311498 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6224142870 0x7f62240a5b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:15:06.870 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:06.858+0000 7f621b7fe700 1 -- 172.21.15.42:0/3131311498 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 924493446 0 0) 0x7f62240a5470 con 0x7f62240a4840 2019-12-05T05:15:06.870 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:06.858+0000 7f621b7fe700 1 -- 172.21.15.42:0/3131311498 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62080030b0 con 0x7f62240a4840 2019-12-05T05:15:06.870 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:06.858+0000 7f621b7fe700 1 -- 172.21.15.42:0/3131311498 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 440573167 0 0) 0x7f6224142ca0 con 0x7f62241166c0 2019-12-05T05:15:06.871 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:06.858+0000 7f621b7fe700 1 -- 172.21.15.42:0/3131311498 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62240a5470 con 0x7f62241166c0 2019-12-05T05:15:06.871 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:06.858+0000 7f622259c700 1 -- 172.21.15.42:0/3131311498 >> v1:172.21.15.42:6789/0 conn(0x7f62240a4840 legacy=0x7f62240a4c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:15:06.871 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:06.858+0000 7f622259c700 1 -- 172.21.15.42:0/3131311498 >> v1:172.21.15.40:6789/0 conn(0x7f62241166c0 legacy=0x7f6224144d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:15:06.871 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:06.858+0000 7f622259c700 1 -- 172.21.15.42:0/3131311498 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62240a56a0 con 0x7f6224142870 2019-12-05T05:15:06.871 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:06.858+0000 7f622259c700 1 --2- 172.21.15.42:0/3131311498 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6224142870 0x7f62240a5b70 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f62240a66e0 tx=0x7f620c0108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:06.871 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:06.858+0000 7f622259c700 1 --2- 172.21.15.42:0/3131311498 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6224142870 0x7f62240a5b70 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f62240a66e0 tx=0x7f620c0108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:06.871 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:06.858+0000 7f622259c700 1 --2- 172.21.15.42:0/3131311498 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6224142870 0x7f62240a5b70 secure :-1 s=READY pgs=118 cs=0 l=1 rx=0x7f62240a66e0 tx=0x7f620c0108f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:15:06.872 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:06.858+0000 7f622259c700 1 --2- 172.21.15.42:0/3131311498 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6224142870 0x7f62240a5b70 secure :-1 s=THROTTLE_DONE pgs=118 cs=0 l=1 rx=0x7f62240a66e0 tx=0x7f620c0108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:06.875 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:06.858+0000 7f621b7fe700 1 -- 172.21.15.42:0/3131311498 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f620c00cb60 con 0x7f6224142870 2019-12-05T05:15:06.875 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:06.858+0000 7f622259c700 1 --2- 172.21.15.42:0/3131311498 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6224142870 0x7f62240a5b70 secure :-1 s=THROTTLE_DONE pgs=118 cs=0 l=1 rx=0x7f62240a66e0 tx=0x7f620c0108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:06.875 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:06.858+0000 7f622259c700 1 --2- 172.21.15.42:0/3131311498 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6224142870 0x7f62240a5b70 secure :-1 s=THROTTLE_DONE pgs=118 cs=0 l=1 rx=0x7f62240a66e0 tx=0x7f620c0108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:06.875 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:06.858+0000 7f621b7fe700 1 -- 172.21.15.42:0/3131311498 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f620c00c780 con 0x7f6224142870 2019-12-05T05:15:06.876 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:06.858+0000 7f6228d90700 1 -- 172.21.15.42:0/3131311498 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f622408eb50 con 0x7f6224142870 2019-12-05T05:15:06.876 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:06.858+0000 7f621b7fe700 1 -- 172.21.15.42:0/3131311498 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f620c0110b0 con 0x7f6224142870 2019-12-05T05:15:06.876 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:06.858+0000 7f6228d90700 1 -- 172.21.15.42:0/3131311498 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f622408f100 con 0x7f6224142870 2019-12-05T05:15:06.876 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:06.858+0000 7f622259c700 1 --2- 172.21.15.42:0/3131311498 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6224142870 0x7f62240a5b70 secure :-1 s=THROTTLE_DONE pgs=118 cs=0 l=1 rx=0x7f62240a66e0 tx=0x7f620c0108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:06.876 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:06.862+0000 7f621b7fe700 1 -- 172.21.15.42:0/3131311498 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f620c011350 con 0x7f6224142870 2019-12-05T05:15:06.876 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:06.862+0000 7f622259c700 1 --2- 172.21.15.42:0/3131311498 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6224142870 0x7f62240a5b70 secure :-1 s=THROTTLE_DONE pgs=118 cs=0 l=1 rx=0x7f62240a66e0 tx=0x7f620c0108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:06.876 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:06.862+0000 7f621b7fe700 1 --2- 172.21.15.42:0/3131311498 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f620803b440 0x7f620803d950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:15:06.877 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:06.862+0000 7f621b7fe700 1 -- 172.21.15.42:0/3131311498 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6719+0+0 (secure 0 0 0) 0x7f620c04b070 con 0x7f6224142870 2019-12-05T05:15:06.877 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:06.862+0000 7f6222d9d700 1 --2- 172.21.15.42:0/3131311498 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f620803b440 0x7f620803d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:15:06.877 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:06.862+0000 7f6222d9d700 1 --2- 172.21.15.42:0/3131311498 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f620803b440 0x7f620803d950 crc :-1 s=READY pgs=260 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:15:06.877 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:06.862+0000 7f6228d90700 1 -- 172.21.15.42:0/3131311498 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f622405d110 con 0x7f6224142870 2019-12-05T05:15:06.879 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:06.870+0000 7f622259c700 1 --2- 172.21.15.42:0/3131311498 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6224142870 0x7f62240a5b70 secure :-1 s=THROTTLE_DONE pgs=118 cs=0 l=1 rx=0x7f62240a66e0 tx=0x7f620c0108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:06.879 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:06.870+0000 7f621b7fe700 1 -- 172.21.15.42:0/3131311498 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f62240a56a0 con 0x7f6224142870 2019-12-05T05:15:07.054 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:07.046+0000 7f6228d90700 1 -- 172.21.15.42:0/3131311498 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f622405b8c0 con 0x7f6224142870 2019-12-05T05:15:07.055 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:07.046+0000 7f622259c700 1 --2- 172.21.15.42:0/3131311498 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6224142870 0x7f62240a5b70 secure :-1 s=THROTTLE_DONE pgs=118 cs=0 l=1 rx=0x7f62240a66e0 tx=0x7f620c0108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:07.055 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:07.046+0000 7f621b7fe700 1 -- 172.21.15.42:0/3131311498 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+2108 (secure 0 0 0) 0x7f620c0320d0 con 0x7f6224142870 2019-12-05T05:15:07.056 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 73 2019-12-05T05:15:07.056 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:15:07.057 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":73,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5629,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/3264829400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3264829400},{"type":"v1","addr":"172.21.15.42:6835","nonce":3264829400}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":73}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:14:17.896674+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4841},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4841":{"gid":4841,"name":"b","rank":0,"incarnation":69,"state":"up:active","state_seq":19,"addr":"172.21.15.40:6833/3150831963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3150831963},{"type":"v1","addr":"172.21.15.40:6833","nonce":3150831963}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:15:07.059 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:07.050+0000 7f6228d90700 1 -- 172.21.15.42:0/3131311498 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f620803b440 msgr2=0x7f620803d950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:15:07.059 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:07.050+0000 7f6228d90700 1 --2- 172.21.15.42:0/3131311498 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f620803b440 0x7f620803d950 crc :-1 s=READY pgs=260 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:15:07.059 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:07.050+0000 7f6228d90700 1 -- 172.21.15.42:0/3131311498 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6224142870 msgr2=0x7f62240a5b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:15:07.059 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:07.050+0000 7f6228d90700 1 --2- 172.21.15.42:0/3131311498 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6224142870 0x7f62240a5b70 secure :-1 s=READY pgs=118 cs=0 l=1 rx=0x7f62240a66e0 tx=0x7f620c0108f0).stop 2019-12-05T05:15:07.063 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:07.050+0000 7f6228d90700 1 -- 172.21.15.42:0/3131311498 shutdown_connections 2019-12-05T05:15:07.063 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:07.050+0000 7f6228d90700 1 --2- 172.21.15.42:0/3131311498 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f620803b440 0x7f620803d950 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:15:07.063 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:07.050+0000 7f6228d90700 1 --2- 172.21.15.42:0/3131311498 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6224142870 0x7f62240a5b70 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:15:07.063 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:07.050+0000 7f6228d90700 1 -- 172.21.15.42:0/3131311498 shutdown_connections 2019-12-05T05:15:07.063 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:07.050+0000 7f6228d90700 1 -- 172.21.15.42:0/3131311498 wait complete. 2019-12-05T05:15:07.063 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:07.050+0000 7f6228d90700 1 -- 172.21.15.42:0/3131311498 >> 172.21.15.42:0/3131311498 conn(0x7f62240a9800 msgr2=0x7f6224059960 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:15:07.069 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T05:15:09.070 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:15:09.173 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.162+0000 7f323ea20700 1 Processor -- start 2019-12-05T05:15:09.173 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.162+0000 7f323ea20700 1 -- start start 2019-12-05T05:15:09.173 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.162+0000 7f323ea20700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f32380a7740 0x7f323808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:15:09.173 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.162+0000 7f323ea20700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f323809fae0 0x7f323807eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:15:09.174 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.162+0000 7f323ea20700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32380a0710 con 0x7f323807e720 2019-12-05T05:15:09.174 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.162+0000 7f323ea20700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32380a0b00 con 0x7f32380a7740 2019-12-05T05:15:09.174 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.162+0000 7f323ea20700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f323808e5b0 con 0x7f323809fae0 2019-12-05T05:15:09.174 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.162+0000 7f323d7be700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f323809fae0 0x7f323807eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:15:09.174 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.162+0000 7f323d7be700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f323809fae0 0x7f323807eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:58764/0 (socket says 172.21.15.42:58764) 2019-12-05T05:15:09.175 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.162+0000 7f323d7be700 1 -- 172.21.15.42:0/683019375 learned_addr learned my addr 172.21.15.42:0/683019375 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:15:09.175 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.162+0000 7f323d7be700 1 -- 172.21.15.42:0/683019375 >> v2:172.21.15.40:3300/0 conn(0x7f32380a7740 msgr2=0x7f323808df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:15:09.175 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.162+0000 7f323d7be700 1 --2- 172.21.15.42:0/683019375 >> v2:172.21.15.40:3300/0 conn(0x7f32380a7740 0x7f323808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:15:09.175 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.162+0000 7f323d7be700 1 -- 172.21.15.42:0/683019375 >> v1:172.21.15.40:6789/0 conn(0x7f323807e720 legacy=0x7f323809f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:15:09.175 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.162+0000 7f323d7be700 1 -- 172.21.15.42:0/683019375 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f323807f0a0 con 0x7f323809fae0 2019-12-05T05:15:09.175 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.162+0000 7f323d7be700 1 --2- 172.21.15.42:0/683019375 >> v2:172.21.15.42:3300/0 conn(0x7f323809fae0 0x7f323807eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f3228002f30 tx=0x7f322800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:09.175 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.162+0000 7f323d7be700 1 --2- 172.21.15.42:0/683019375 >> v2:172.21.15.42:3300/0 conn(0x7f323809fae0 0x7f323807eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f3228002f30 tx=0x7f322800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:09.176 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.162+0000 7f323d7be700 1 --2- 172.21.15.42:0/683019375 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f323809fae0 0x7f323807eb50 secure :-1 s=READY pgs=574 cs=0 l=1 rx=0x7f3228002f30 tx=0x7f322800fd30).ready entity=mon.1 client_cookie=d70e1f2b640bb228 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:15:09.176 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.162+0000 7f323d7be700 1 --2- 172.21.15.42:0/683019375 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f323809fae0 0x7f323807eb50 secure :-1 s=THROTTLE_DONE pgs=574 cs=0 l=1 rx=0x7f3228002f30 tx=0x7f322800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:09.176 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.162+0000 7f32377fe700 1 -- 172.21.15.42:0/683019375 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3228008c90 con 0x7f323809fae0 2019-12-05T05:15:09.176 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.162+0000 7f323d7be700 1 --2- 172.21.15.42:0/683019375 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f323809fae0 0x7f323807eb50 secure :-1 s=THROTTLE_DONE pgs=574 cs=0 l=1 rx=0x7f3228002f30 tx=0x7f322800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:09.176 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.162+0000 7f323d7be700 1 --2- 172.21.15.42:0/683019375 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f323809fae0 0x7f323807eb50 secure :-1 s=THROTTLE_DONE pgs=574 cs=0 l=1 rx=0x7f3228002f30 tx=0x7f322800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:09.176 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.162+0000 7f32377fe700 1 -- 172.21.15.42:0/683019375 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32280088b0 con 0x7f323809fae0 2019-12-05T05:15:09.177 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.162+0000 7f32377fe700 1 -- 172.21.15.42:0/683019375 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3228011ad0 con 0x7f323809fae0 2019-12-05T05:15:09.177 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.162+0000 7f323ea20700 1 -- 172.21.15.42:0/683019375 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f323809fae0 msgr2=0x7f323807eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:15:09.177 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.162+0000 7f323ea20700 1 --2- 172.21.15.42:0/683019375 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f323809fae0 0x7f323807eb50 secure :-1 s=READY pgs=574 cs=0 l=1 rx=0x7f3228002f30 tx=0x7f322800fd30).stop 2019-12-05T05:15:09.177 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.166+0000 7f323ea20700 1 -- 172.21.15.42:0/683019375 shutdown_connections 2019-12-05T05:15:09.177 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.166+0000 7f323ea20700 1 --2- 172.21.15.42:0/683019375 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f323809fae0 0x7f323807eb50 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:15:09.177 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.166+0000 7f323ea20700 1 --2- 172.21.15.42:0/683019375 >> v2:172.21.15.40:3300/0 conn(0x7f32380a7740 0x7f323808df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:15:09.177 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.166+0000 7f323ea20700 1 -- 172.21.15.42:0/683019375 shutdown_connections 2019-12-05T05:15:09.178 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.166+0000 7f323ea20700 1 -- 172.21.15.42:0/683019375 wait complete. 2019-12-05T05:15:09.178 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.166+0000 7f323ea20700 1 -- 172.21.15.42:0/683019375 >> 172.21.15.42:0/683019375 conn(0x7f32380aabe0 msgr2=0x7f32380a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:15:09.184 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.166+0000 7f323ea20700 1 Processor -- start 2019-12-05T05:15:09.184 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.166+0000 7f323ea20700 1 -- start start 2019-12-05T05:15:09.184 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.166+0000 7f323ea20700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f323811b4d0 0x7f323811b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:15:09.185 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.166+0000 7f323ea20700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3238120c80 con 0x7f3238118c20 2019-12-05T05:15:09.185 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.166+0000 7f323ea20700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3238112c70 con 0x7f32381327c0 2019-12-05T05:15:09.185 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.166+0000 7f323ea20700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3238112fe0 con 0x7f323811b4d0 2019-12-05T05:15:09.185 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.166+0000 7f323d7be700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f32381327c0 0x7f3238134c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:58808/0 (socket says 172.21.15.42:58808) 2019-12-05T05:15:09.185 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.166+0000 7f323d7be700 1 -- 172.21.15.42:0/2435887730 learned_addr learned my addr 172.21.15.42:0/2435887730 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:15:09.185 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.166+0000 7f3237fff700 1 --2- 172.21.15.42:0/2435887730 >> v2:172.21.15.40:3300/0 conn(0x7f323811b4d0 0x7f323811b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:15:09.185 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.166+0000 7f32357fa700 1 -- 172.21.15.42:0/2435887730 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2618747558 0 0) 0x7f3238112c70 con 0x7f32381327c0 2019-12-05T05:15:09.186 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.166+0000 7f32357fa700 1 -- 172.21.15.42:0/2435887730 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32180030b0 con 0x7f32381327c0 2019-12-05T05:15:09.186 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.166+0000 7f32357fa700 1 -- 172.21.15.42:0/2435887730 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 3056071120 0 0) 0x7f32180030b0 con 0x7f32381327c0 2019-12-05T05:15:09.186 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.166+0000 7f32357fa700 1 -- 172.21.15.42:0/2435887730 --> v1:172.21.15.42:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3238112c70 con 0x7f32381327c0 2019-12-05T05:15:09.186 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.166+0000 7f32357fa700 1 -- 172.21.15.42:0/2435887730 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7f3228008470 con 0x7f32381327c0 2019-12-05T05:15:09.186 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.166+0000 7f32357fa700 1 -- 172.21.15.42:0/2435887730 >> v2:172.21.15.40:3300/0 conn(0x7f323811b4d0 msgr2=0x7f323811b900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:15:09.187 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.166+0000 7f32357fa700 1 --2- 172.21.15.42:0/2435887730 >> v2:172.21.15.40:3300/0 conn(0x7f323811b4d0 0x7f323811b900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:15:09.187 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.166+0000 7f32357fa700 1 -- 172.21.15.42:0/2435887730 >> v1:172.21.15.42:6789/0 conn(0x7f32381327c0 legacy=0x7f3238134c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:15:09.187 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.166+0000 7f32357fa700 1 -- 172.21.15.42:0/2435887730 >> v1:172.21.15.40:6789/0 conn(0x7f3238118c20 legacy=0x7f32380a6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:15:09.187 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.166+0000 7f32357fa700 1 --2- 172.21.15.42:0/2435887730 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f32180038a0 0x7f3218003cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:15:09.187 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.166+0000 7f32357fa700 1 --2- 172.21.15.42:0/2435887730 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3218004420 0x7f3218004850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:15:09.187 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.166+0000 7f32357fa700 1 --2- 172.21.15.42:0/2435887730 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3218004e40 0x7f321800d880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:15:09.187 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.166+0000 7f32357fa700 1 -- 172.21.15.42:0/2435887730 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3238112fe0 con 0x7f32180038a0 2019-12-05T05:15:09.188 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.166+0000 7f32357fa700 1 -- 172.21.15.42:0/2435887730 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f321800de30 con 0x7f3218004420 2019-12-05T05:15:09.188 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.166+0000 7f32357fa700 1 -- 172.21.15.42:0/2435887730 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f321800e070 con 0x7f3218004e40 2019-12-05T05:15:09.188 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.166+0000 7f3237fff700 1 --2- 172.21.15.42:0/2435887730 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3218004420 0x7f3218004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:15:09.188 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.166+0000 7f323d7be700 1 --2- 172.21.15.42:0/2435887730 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3218004e40 0x7f321800d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:15:09.188 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.166+0000 7f323cfbd700 1 --2- 172.21.15.42:0/2435887730 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f32180038a0 0x7f3218003cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:15:09.188 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.166+0000 7f3237fff700 1 -- 172.21.15.42:0/2435887730 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3218004e40 msgr2=0x7f321800d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:15:09.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.166+0000 7f3237fff700 1 --2- 172.21.15.42:0/2435887730 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3218004e40 0x7f321800d880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:15:09.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.166+0000 7f3237fff700 1 -- 172.21.15.42:0/2435887730 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f32180038a0 msgr2=0x7f3218003cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:15:09.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.166+0000 7f3237fff700 1 --2- 172.21.15.42:0/2435887730 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f32180038a0 0x7f3218003cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:15:09.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.166+0000 7f3237fff700 1 -- 172.21.15.42:0/2435887730 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f323811a8c0 con 0x7f3218004420 2019-12-05T05:15:09.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.166+0000 7f323d7be700 1 -- 172.21.15.42:0/2435887730 reap_dead start 2019-12-05T05:15:09.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.166+0000 7f323ea20700 1 -- 172.21.15.42:0/2435887730 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3238118c20 con 0x7f3218004420 2019-12-05T05:15:09.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.166+0000 7f3237fff700 1 --2- 172.21.15.42:0/2435887730 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3218004420 0x7f3218004850 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f322c002ed0 tx=0x7f322c010d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:09.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.166+0000 7f323ea20700 1 -- 172.21.15.42:0/2435887730 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32380a6a90 con 0x7f3218004420 2019-12-05T05:15:09.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.170+0000 7f3237fff700 1 --2- 172.21.15.42:0/2435887730 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3218004420 0x7f3218004850 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f322c002ed0 tx=0x7f322c010d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:09.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.170+0000 7f3237fff700 1 --2- 172.21.15.42:0/2435887730 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3218004420 0x7f3218004850 secure :-1 s=READY pgs=576 cs=0 l=1 rx=0x7f322c002ed0 tx=0x7f322c010d50).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:15:09.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.170+0000 7f323cfbd700 1 --2- 172.21.15.42:0/2435887730 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f32180038a0 0x7f3218003cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).send_auth_request state changed! 2019-12-05T05:15:09.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.170+0000 7f3237fff700 1 --2- 172.21.15.42:0/2435887730 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3218004420 0x7f3218004850 secure :-1 s=THROTTLE_DONE pgs=576 cs=0 l=1 rx=0x7f322c002ed0 tx=0x7f322c010d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:09.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.170+0000 7f32357fa700 1 -- 172.21.15.42:0/2435887730 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f322c011a40 con 0x7f3218004420 2019-12-05T05:15:09.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.170+0000 7f3237fff700 1 --2- 172.21.15.42:0/2435887730 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3218004420 0x7f3218004850 secure :-1 s=THROTTLE_DONE pgs=576 cs=0 l=1 rx=0x7f322c002ed0 tx=0x7f322c010d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:09.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.170+0000 7f3237fff700 1 --2- 172.21.15.42:0/2435887730 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3218004420 0x7f3218004850 secure :-1 s=THROTTLE_DONE pgs=576 cs=0 l=1 rx=0x7f322c002ed0 tx=0x7f322c010d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:09.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.170+0000 7f32357fa700 1 -- 172.21.15.42:0/2435887730 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f322c011660 con 0x7f3218004420 2019-12-05T05:15:09.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.170+0000 7f32357fa700 1 -- 172.21.15.42:0/2435887730 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f322c00c770 con 0x7f3218004420 2019-12-05T05:15:09.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.170+0000 7f3237fff700 1 --2- 172.21.15.42:0/2435887730 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3218004420 0x7f3218004850 secure :-1 s=THROTTLE_DONE pgs=576 cs=0 l=1 rx=0x7f322c002ed0 tx=0x7f322c010d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:09.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.170+0000 7f32357fa700 1 -- 172.21.15.42:0/2435887730 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f322c00c970 con 0x7f3218004420 2019-12-05T05:15:09.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.170+0000 7f3237fff700 1 --2- 172.21.15.42:0/2435887730 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3218004420 0x7f3218004850 secure :-1 s=THROTTLE_DONE pgs=576 cs=0 l=1 rx=0x7f322c002ed0 tx=0x7f322c010d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:09.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.170+0000 7f32357fa700 1 --2- 172.21.15.42:0/2435887730 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3218040b80 0x7f3218043090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:15:09.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.170+0000 7f32357fa700 1 -- 172.21.15.42:0/2435887730 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6805+0+0 (secure 0 0 0) 0x7f322c04b0b0 con 0x7f3218004420 2019-12-05T05:15:09.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.170+0000 7f323cfbd700 1 --2- 172.21.15.42:0/2435887730 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3218040b80 0x7f3218043090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:15:09.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.170+0000 7f323cfbd700 1 --2- 172.21.15.42:0/2435887730 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3218040b80 0x7f3218043090 crc :-1 s=READY pgs=261 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:15:09.193 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.178+0000 7f323ea20700 1 -- 172.21.15.42:0/2435887730 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3210003200 con 0x7f3218004420 2019-12-05T05:15:09.193 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.182+0000 7f3237fff700 1 --2- 172.21.15.42:0/2435887730 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3218004420 0x7f3218004850 secure :-1 s=THROTTLE_DONE pgs=576 cs=0 l=1 rx=0x7f322c002ed0 tx=0x7f322c010d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:09.194 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.182+0000 7f32357fa700 1 -- 172.21.15.42:0/2435887730 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f322c032150 con 0x7f3218004420 2019-12-05T05:15:09.365 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.358+0000 7f323ea20700 1 -- 172.21.15.42:0/2435887730 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3210003630 con 0x7f3218004420 2019-12-05T05:15:09.366 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.358+0000 7f3237fff700 1 --2- 172.21.15.42:0/2435887730 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3218004420 0x7f3218004850 secure :-1 s=THROTTLE_DONE pgs=576 cs=0 l=1 rx=0x7f322c002ed0 tx=0x7f322c010d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:09.366 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.358+0000 7f32357fa700 1 -- 172.21.15.42:0/2435887730 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 74 v74) v1 ==== 94+0+1771 (secure 0 0 0) 0x7f322c0323b0 con 0x7f3218004420 2019-12-05T05:15:09.367 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 74 2019-12-05T05:15:09.367 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:15:09.367 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":74,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":74,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:15:08.382639+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5629},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5629":{"gid":5629,"name":"a","rank":0,"incarnation":74,"state":"up:replay","state_seq":2,"addr":"172.21.15.42:6835/3264829400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3264829400},{"type":"v1","addr":"172.21.15.42:6835","nonce":3264829400}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:15:09.369 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.362+0000 7f323ea20700 1 -- 172.21.15.42:0/2435887730 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3218040b80 msgr2=0x7f3218043090 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:15:09.369 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.362+0000 7f323ea20700 1 --2- 172.21.15.42:0/2435887730 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3218040b80 0x7f3218043090 crc :-1 s=READY pgs=261 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:15:09.370 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.362+0000 7f323ea20700 1 -- 172.21.15.42:0/2435887730 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3218004420 msgr2=0x7f3218004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:15:09.370 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.362+0000 7f323ea20700 1 --2- 172.21.15.42:0/2435887730 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3218004420 0x7f3218004850 secure :-1 s=READY pgs=576 cs=0 l=1 rx=0x7f322c002ed0 tx=0x7f322c010d50).stop 2019-12-05T05:15:09.370 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.362+0000 7f323ea20700 1 -- 172.21.15.42:0/2435887730 shutdown_connections 2019-12-05T05:15:09.371 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.362+0000 7f323ea20700 1 --2- 172.21.15.42:0/2435887730 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3218040b80 0x7f3218043090 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:15:09.371 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.362+0000 7f323ea20700 1 --2- 172.21.15.42:0/2435887730 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3218004420 0x7f3218004850 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:15:09.371 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.362+0000 7f323ea20700 1 -- 172.21.15.42:0/2435887730 shutdown_connections 2019-12-05T05:15:09.371 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.362+0000 7f323ea20700 1 -- 172.21.15.42:0/2435887730 wait complete. 2019-12-05T05:15:09.371 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.362+0000 7f323ea20700 1 -- 172.21.15.42:0/2435887730 >> 172.21.15.42:0/2435887730 conn(0x7f323811ca40 msgr2=0x7f323809f620 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:15:09.379 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b down, removed from mdsmap 2019-12-05T05:15:09.379 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2019-12-05T05:15:09.379 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:15:09.475 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.454+0000 7f26d880a700 1 Processor -- start 2019-12-05T05:15:09.476 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.454+0000 7f26d880a700 1 -- start start 2019-12-05T05:15:09.476 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.454+0000 7f26d880a700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f26d409fae0 0x7f26d414fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:15:09.476 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.454+0000 7f26d880a700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26d40a0710 con 0x7f26d408df80 2019-12-05T05:15:09.476 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.454+0000 7f26d880a700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26d407ef20 con 0x7f26d40a7740 2019-12-05T05:15:09.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.454+0000 7f26d880a700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26d40a0b00 con 0x7f26d409fae0 2019-12-05T05:15:09.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.454+0000 7f26d2d9d700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f26d40a7740 0x7f26d407e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:58818/0 (socket says 172.21.15.42:58818) 2019-12-05T05:15:09.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.454+0000 7f26d2d9d700 1 -- 172.21.15.42:0/3368749123 learned_addr learned my addr 172.21.15.42:0/3368749123 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:15:09.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.454+0000 7f26d359e700 1 --2- 172.21.15.42:0/3368749123 >> v2:172.21.15.40:3300/0 conn(0x7f26d409fae0 0x7f26d414fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:15:09.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.454+0000 7f26d1d9b700 1 -- 172.21.15.42:0/3368749123 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3276019749 0 0) 0x7f26d407ef20 con 0x7f26d40a7740 2019-12-05T05:15:09.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.454+0000 7f26d1d9b700 1 -- 172.21.15.42:0/3368749123 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26c00030b0 con 0x7f26d40a7740 2019-12-05T05:15:09.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.454+0000 7f26d1d9b700 1 -- 172.21.15.42:0/3368749123 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4273796673 0 0) 0x7f26c00030b0 con 0x7f26d40a7740 2019-12-05T05:15:09.482 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.454+0000 7f26d1d9b700 1 -- 172.21.15.42:0/3368749123 >> v2:172.21.15.40:3300/0 conn(0x7f26d409fae0 msgr2=0x7f26d414fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:15:09.482 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.454+0000 7f26d1d9b700 1 --2- 172.21.15.42:0/3368749123 >> v2:172.21.15.40:3300/0 conn(0x7f26d409fae0 0x7f26d414fff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:15:09.482 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.454+0000 7f26d1d9b700 1 -- 172.21.15.42:0/3368749123 >> v1:172.21.15.40:6789/0 conn(0x7f26d408df80 legacy=0x7f26d409f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:15:09.482 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.454+0000 7f26d1d9b700 1 -- 172.21.15.42:0/3368749123 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26d40a9ba0 con 0x7f26d40a7740 2019-12-05T05:15:09.482 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.454+0000 7f26d1d9b700 1 -- 172.21.15.42:0/3368749123 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7f26c4001ad0 con 0x7f26d40a7740 2019-12-05T05:15:09.482 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.454+0000 7f26d1d9b700 1 -- 172.21.15.42:0/3368749123 >> v1:172.21.15.42:6789/0 conn(0x7f26d40a7740 legacy=0x7f26d407e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:15:09.483 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.454+0000 7f26d1d9b700 1 --2- 172.21.15.42:0/3368749123 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f26c00036e0 0x7f26c0003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:15:09.483 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.454+0000 7f26d1d9b700 1 --2- 172.21.15.42:0/3368749123 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f26c0004420 0x7f26c0004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:15:09.483 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.458+0000 7f26d1d9b700 1 --2- 172.21.15.42:0/3368749123 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f26c0004da0 0x7f26c000b670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:15:09.483 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.458+0000 7f26d1d9b700 1 -- 172.21.15.42:0/3368749123 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26d40a0b00 con 0x7f26c00036e0 2019-12-05T05:15:09.483 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.458+0000 7f26d1d9b700 1 -- 172.21.15.42:0/3368749123 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26c004fc60 con 0x7f26c0004da0 2019-12-05T05:15:09.483 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.458+0000 7f26d1d9b700 1 -- 172.21.15.42:0/3368749123 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26c004fea0 con 0x7f26c0004420 2019-12-05T05:15:09.483 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.458+0000 7f26d359e700 1 --2- 172.21.15.42:0/3368749123 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f26c0004da0 0x7f26c000b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:15:09.484 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.458+0000 7f26d259c700 1 --2- 172.21.15.42:0/3368749123 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f26c0004420 0x7f26c0004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:15:09.484 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.458+0000 7f26d2d9d700 1 --2- 172.21.15.42:0/3368749123 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f26c00036e0 0x7f26c0003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:15:09.484 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.458+0000 7f26d359e700 1 -- 172.21.15.42:0/3368749123 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f26c0004420 msgr2=0x7f26c0004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:15:09.484 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.458+0000 7f26d359e700 1 --2- 172.21.15.42:0/3368749123 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f26c0004420 0x7f26c0004850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:15:09.484 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.458+0000 7f26d359e700 1 -- 172.21.15.42:0/3368749123 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f26c00036e0 msgr2=0x7f26c0003b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:15:09.484 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.458+0000 7f26d359e700 1 --2- 172.21.15.42:0/3368749123 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f26c00036e0 0x7f26c0003b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:15:09.484 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.458+0000 7f26d359e700 1 -- 172.21.15.42:0/3368749123 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26c00501a0 con 0x7f26c0004da0 2019-12-05T05:15:09.485 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.458+0000 7f26d359e700 1 --2- 172.21.15.42:0/3368749123 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f26c0004da0 0x7f26c000b670 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f26cc002f30 tx=0x7f26cc00ffa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:09.485 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.458+0000 7f26d359e700 1 -- 172.21.15.42:0/3368749123 reap_dead start 2019-12-05T05:15:09.485 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.458+0000 7f26d359e700 1 --2- 172.21.15.42:0/3368749123 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f26c0004da0 0x7f26c000b670 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f26cc002f30 tx=0x7f26cc00ffa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:09.485 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.458+0000 7f26d359e700 1 --2- 172.21.15.42:0/3368749123 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f26c0004da0 0x7f26c000b670 secure :-1 s=READY pgs=578 cs=0 l=1 rx=0x7f26cc002f30 tx=0x7f26cc00ffa0).ready entity=mon.1 client_cookie=801187377d4489ce server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:15:09.486 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.458+0000 7f26d359e700 1 --2- 172.21.15.42:0/3368749123 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f26c0004da0 0x7f26c000b670 secure :-1 s=THROTTLE_DONE pgs=578 cs=0 l=1 rx=0x7f26cc002f30 tx=0x7f26cc00ffa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:09.486 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.458+0000 7f26d1d9b700 1 -- 172.21.15.42:0/3368749123 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f26cc010cd0 con 0x7f26c0004da0 2019-12-05T05:15:09.486 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.458+0000 7f26d359e700 1 --2- 172.21.15.42:0/3368749123 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f26c0004da0 0x7f26c000b670 secure :-1 s=THROTTLE_DONE pgs=578 cs=0 l=1 rx=0x7f26cc002f30 tx=0x7f26cc00ffa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:09.486 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.458+0000 7f26d359e700 1 --2- 172.21.15.42:0/3368749123 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f26c0004da0 0x7f26c000b670 secure :-1 s=THROTTLE_DONE pgs=578 cs=0 l=1 rx=0x7f26cc002f30 tx=0x7f26cc00ffa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:09.486 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.458+0000 7f26d1d9b700 1 -- 172.21.15.42:0/3368749123 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26cc0108f0 con 0x7f26c0004da0 2019-12-05T05:15:09.486 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.458+0000 7f26d1d9b700 1 -- 172.21.15.42:0/3368749123 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f26cc008bb0 con 0x7f26c0004da0 2019-12-05T05:15:09.487 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.458+0000 7f26d880a700 1 -- 172.21.15.42:0/3368749123 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f26c0004da0 msgr2=0x7f26c000b670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:15:09.487 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.458+0000 7f26d880a700 1 --2- 172.21.15.42:0/3368749123 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f26c0004da0 0x7f26c000b670 secure :-1 s=READY pgs=578 cs=0 l=1 rx=0x7f26cc002f30 tx=0x7f26cc00ffa0).stop 2019-12-05T05:15:09.488 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.458+0000 7f26d880a700 1 -- 172.21.15.42:0/3368749123 shutdown_connections 2019-12-05T05:15:09.488 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.458+0000 7f26d880a700 1 --2- 172.21.15.42:0/3368749123 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f26c0004da0 0x7f26c000b670 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:15:09.488 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.458+0000 7f26d880a700 1 -- 172.21.15.42:0/3368749123 shutdown_connections 2019-12-05T05:15:09.488 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.458+0000 7f26d880a700 1 -- 172.21.15.42:0/3368749123 wait complete. 2019-12-05T05:15:09.488 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.458+0000 7f26d880a700 1 -- 172.21.15.42:0/3368749123 >> 172.21.15.42:0/3368749123 conn(0x7f26d40aabe0 msgr2=0x7f26d40a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:15:09.489 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.458+0000 7f26d880a700 1 Processor -- start 2019-12-05T05:15:09.489 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.458+0000 7f26d880a700 1 -- start start 2019-12-05T05:15:09.489 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.462+0000 7f26d880a700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f26d40771d0 0x7f26d40a6540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:15:09.489 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.462+0000 7f26d880a700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f26d4077be0 0x7f26d40a4840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:15:09.489 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.462+0000 7f26d880a700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26d407bd80 con 0x7f26d411ec40 2019-12-05T05:15:09.489 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.462+0000 7f26d880a700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26d40776a0 con 0x7f26d40771d0 2019-12-05T05:15:09.489 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.462+0000 7f26d880a700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26d4077940 con 0x7f26d4077be0 2019-12-05T05:15:09.490 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.462+0000 7f26d2d9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f26d40771d0 0x7f26d40a6540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:15:09.490 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.462+0000 7f26d2d9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f26d40771d0 0x7f26d40a6540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:58792/0 (socket says 172.21.15.42:58792) 2019-12-05T05:15:09.490 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.462+0000 7f26d2d9d700 1 -- 172.21.15.42:0/3127228680 learned_addr learned my addr 172.21.15.42:0/3127228680 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:15:09.490 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.462+0000 7f26d259c700 1 --2- 172.21.15.42:0/3127228680 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f26d4077be0 0x7f26d40a4840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:15:09.491 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.462+0000 7f26cb7fe700 1 -- 172.21.15.42:0/3127228680 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3973973827 0 0) 0x7f26d407bd80 con 0x7f26d411ec40 2019-12-05T05:15:09.491 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.462+0000 7f26d2d9d700 1 -- 172.21.15.42:0/3127228680 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f26d4077be0 msgr2=0x7f26d40a4840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:15:09.491 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.462+0000 7f26d2d9d700 1 --2- 172.21.15.42:0/3127228680 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f26d4077be0 0x7f26d40a4840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:15:09.491 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.462+0000 7f26d2d9d700 1 -- 172.21.15.42:0/3127228680 >> v1:172.21.15.42:6789/0 conn(0x7f26d411ec40 legacy=0x7f26d4141550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:15:09.491 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.462+0000 7f26d2d9d700 1 -- 172.21.15.42:0/3127228680 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26d40a4d90 con 0x7f26d40771d0 2019-12-05T05:15:09.491 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.462+0000 7f26d2d9d700 1 --2- 172.21.15.42:0/3127228680 >> v2:172.21.15.42:3300/0 conn(0x7f26d40771d0 0x7f26d40a6540 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f26d407bfb0 tx=0x7f26c4010c10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:09.491 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.462+0000 7f26d2d9d700 1 --2- 172.21.15.42:0/3127228680 >> v2:172.21.15.42:3300/0 conn(0x7f26d40771d0 0x7f26d40a6540 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f26d407bfb0 tx=0x7f26c4010c10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:09.492 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.462+0000 7f26d2d9d700 1 --2- 172.21.15.42:0/3127228680 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f26d40771d0 0x7f26d40a6540 secure :-1 s=READY pgs=580 cs=0 l=1 rx=0x7f26d407bfb0 tx=0x7f26c4010c10).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:15:09.492 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.462+0000 7f26d2d9d700 1 --2- 172.21.15.42:0/3127228680 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f26d40771d0 0x7f26d40a6540 secure :-1 s=THROTTLE_DONE pgs=580 cs=0 l=1 rx=0x7f26d407bfb0 tx=0x7f26c4010c10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:09.492 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.462+0000 7f26cb7fe700 1 -- 172.21.15.42:0/3127228680 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f26c40046f0 con 0x7f26d40771d0 2019-12-05T05:15:09.492 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.462+0000 7f26d2d9d700 1 --2- 172.21.15.42:0/3127228680 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f26d40771d0 0x7f26d40a6540 secure :-1 s=THROTTLE_DONE pgs=580 cs=0 l=1 rx=0x7f26d407bfb0 tx=0x7f26c4010c10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:09.492 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.462+0000 7f26cb7fe700 1 -- 172.21.15.42:0/3127228680 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26c400ca70 con 0x7f26d40771d0 2019-12-05T05:15:09.493 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.462+0000 7f26d2d9d700 1 --2- 172.21.15.42:0/3127228680 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f26d40771d0 0x7f26d40a6540 secure :-1 s=THROTTLE_DONE pgs=580 cs=0 l=1 rx=0x7f26d407bfb0 tx=0x7f26c4010c10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:09.493 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.462+0000 7f26d880a700 1 -- 172.21.15.42:0/3127228680 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26d4141d50 con 0x7f26d40771d0 2019-12-05T05:15:09.493 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.462+0000 7f26cb7fe700 1 -- 172.21.15.42:0/3127228680 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f26c4004cd0 con 0x7f26d40771d0 2019-12-05T05:15:09.493 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.462+0000 7f26d880a700 1 -- 172.21.15.42:0/3127228680 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26d412f2b0 con 0x7f26d40771d0 2019-12-05T05:15:09.493 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.462+0000 7f26d2d9d700 1 --2- 172.21.15.42:0/3127228680 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f26d40771d0 0x7f26d40a6540 secure :-1 s=THROTTLE_DONE pgs=580 cs=0 l=1 rx=0x7f26d407bfb0 tx=0x7f26c4010c10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:09.493 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.462+0000 7f26cb7fe700 1 -- 172.21.15.42:0/3127228680 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f26c4006180 con 0x7f26d40771d0 2019-12-05T05:15:09.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.462+0000 7f26d2d9d700 1 --2- 172.21.15.42:0/3127228680 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f26d40771d0 0x7f26d40a6540 secure :-1 s=THROTTLE_DONE pgs=580 cs=0 l=1 rx=0x7f26d407bfb0 tx=0x7f26c4010c10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:09.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.462+0000 7f26cb7fe700 1 --2- 172.21.15.42:0/3127228680 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f26b803ac40 0x7f26b803d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:15:09.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.462+0000 7f26cb7fe700 1 -- 172.21.15.42:0/3127228680 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6805+0+0 (secure 0 0 0) 0x7f26c404b050 con 0x7f26d40771d0 2019-12-05T05:15:09.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.462+0000 7f26d259c700 1 --2- 172.21.15.42:0/3127228680 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f26b803ac40 0x7f26b803d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:15:09.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.462+0000 7f26d259c700 1 --2- 172.21.15.42:0/3127228680 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f26b803ac40 0x7f26b803d150 crc :-1 s=READY pgs=262 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:15:09.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.470+0000 7f26d880a700 1 -- 172.21.15.42:0/3127228680 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26d405d110 con 0x7f26d40771d0 2019-12-05T05:15:09.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.474+0000 7f26d2d9d700 1 --2- 172.21.15.42:0/3127228680 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f26d40771d0 0x7f26d40a6540 secure :-1 s=THROTTLE_DONE pgs=580 cs=0 l=1 rx=0x7f26d407bfb0 tx=0x7f26c4010c10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:09.496 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.474+0000 7f26cb7fe700 1 -- 172.21.15.42:0/3127228680 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f26c40320b0 con 0x7f26d40771d0 2019-12-05T05:15:09.643 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.638+0000 7f26d880a700 1 -- 172.21.15.42:0/3127228680 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f26d405b8c0 con 0x7f26d40771d0 2019-12-05T05:15:09.645 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.638+0000 7f26d2d9d700 1 --2- 172.21.15.42:0/3127228680 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f26d40771d0 0x7f26d40a6540 secure :-1 s=THROTTLE_DONE pgs=580 cs=0 l=1 rx=0x7f26d407bfb0 tx=0x7f26c4010c10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:09.645 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.638+0000 7f26cb7fe700 1 -- 172.21.15.42:0/3127228680 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 74 v74) v1 ==== 94+0+1771 (secure 0 0 0) 0x7f26c4001530 con 0x7f26d40771d0 2019-12-05T05:15:09.645 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 74 2019-12-05T05:15:09.645 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:15:09.646 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":74,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":74,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:15:08.382639+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5629},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5629":{"gid":5629,"name":"a","rank":0,"incarnation":74,"state":"up:replay","state_seq":2,"addr":"172.21.15.42:6835/3264829400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3264829400},{"type":"v1","addr":"172.21.15.42:6835","nonce":3264829400}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:15:09.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.642+0000 7f26d880a700 1 -- 172.21.15.42:0/3127228680 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f26b803ac40 msgr2=0x7f26b803d150 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:15:09.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.642+0000 7f26d880a700 1 --2- 172.21.15.42:0/3127228680 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f26b803ac40 0x7f26b803d150 crc :-1 s=READY pgs=262 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:15:09.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.642+0000 7f26d880a700 1 -- 172.21.15.42:0/3127228680 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f26d40771d0 msgr2=0x7f26d40a6540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:15:09.650 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.642+0000 7f26d880a700 1 --2- 172.21.15.42:0/3127228680 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f26d40771d0 0x7f26d40a6540 secure :-1 s=READY pgs=580 cs=0 l=1 rx=0x7f26d407bfb0 tx=0x7f26c4010c10).stop 2019-12-05T05:15:09.650 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.642+0000 7f26d880a700 1 -- 172.21.15.42:0/3127228680 shutdown_connections 2019-12-05T05:15:09.650 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.642+0000 7f26d880a700 1 --2- 172.21.15.42:0/3127228680 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f26b803ac40 0x7f26b803d150 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:15:09.650 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.642+0000 7f26d880a700 1 --2- 172.21.15.42:0/3127228680 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f26d40771d0 0x7f26d40a6540 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:15:09.650 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.642+0000 7f26d880a700 1 --2- 172.21.15.42:0/3127228680 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f26d4077be0 0x7f26d40a4840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:15:09.650 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.642+0000 7f26d880a700 1 -- 172.21.15.42:0/3127228680 shutdown_connections 2019-12-05T05:15:09.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.642+0000 7f26d880a700 1 -- 172.21.15.42:0/3127228680 wait complete. 2019-12-05T05:15:09.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:09.642+0000 7f26d880a700 1 -- 172.21.15.42:0/3127228680 >> 172.21.15.42:0/3127228680 conn(0x7f26d407edd0 msgr2=0x7f26d4059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:15:09.659 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 74, "feature_flags": { "enable_multiple": false, "ever_enabled_multiple": false }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2019-12-05T05:03:35.604738+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 74, "ever_allowed_features": 2, "explicitly_allowed_features": 2, "failed": [], "flags": 18, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5629": { "addr": "172.21.15.42:6835/3264829400", "addrs": { "addrvec": [ { "addr": "172.21.15.42:6834", "nonce": 3264829400, "type": "v2" }, { "addr": "172.21.15.42:6835", "nonce": 3264829400, "type": "v1" } ] }, "export_targets": [], "features": 4540138292836696063, "flags": 0, "gid": 5629, "incarnation": 74, "name": "a", "rank": 0, "state": "up:replay", "state_seq": 2 } }, "last_failure": 0, "last_failure_osd_epoch": 58, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 2, "min_compat_client": "0 (unknown)", "modified": "2019-12-05T05:15:08.382639+0000", "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5629 } } } ], "standbys": [] } 2019-12-05T05:15:11.661 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:15:11.766 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:11.750+0000 7fcdbf8a0700 1 Processor -- start 2019-12-05T05:15:11.766 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:11.754+0000 7fcdbf8a0700 1 -- start start 2019-12-05T05:15:11.766 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:11.754+0000 7fcdbf8a0700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcdb80a7740 0x7fcdb808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:15:11.766 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:11.754+0000 7fcdbf8a0700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fcdb809fae0 0x7fcdb807eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:15:11.766 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:11.754+0000 7fcdbf8a0700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdb80a0710 con 0x7fcdb807e720 2019-12-05T05:15:11.767 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:11.754+0000 7fcdbf8a0700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcdb80a0b00 con 0x7fcdb809fae0 2019-12-05T05:15:11.767 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:11.754+0000 7fcdbf8a0700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdb808e5b0 con 0x7fcdb80a7740 2019-12-05T05:15:11.767 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:11.754+0000 7fcdbe63e700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fcdb809fae0 0x7fcdb807eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:15:11.767 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:11.754+0000 7fcdbde3d700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcdb80a7740 0x7fcdb808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:15:11.768 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:11.754+0000 7fcdbd63c700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fcdb807e720 0x7fcdb809f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:40050/0 (socket says 172.21.15.42:40050) 2019-12-05T05:15:11.768 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:11.754+0000 7fcdbd63c700 1 -- 172.21.15.42:0/623663434 learned_addr learned my addr 172.21.15.42:0/623663434 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:15:11.769 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:11.754+0000 7fcdbe63e700 1 -- 172.21.15.42:0/623663434 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcdb80a7740 msgr2=0x7fcdb808df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:15:11.769 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:11.754+0000 7fcdbe63e700 1 --2- 172.21.15.42:0/623663434 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcdb80a7740 0x7fcdb808df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:15:11.769 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:11.754+0000 7fcdbe63e700 1 -- 172.21.15.42:0/623663434 >> v1:172.21.15.40:6789/0 conn(0x7fcdb807e720 legacy=0x7fcdb809f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:15:11.769 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:11.754+0000 7fcdbe63e700 1 -- 172.21.15.42:0/623663434 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdb807f0a0 con 0x7fcdb809fae0 2019-12-05T05:15:11.769 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:11.754+0000 7fcdbe63e700 1 --2- 172.21.15.42:0/623663434 >> v2:172.21.15.40:3300/0 conn(0x7fcdb809fae0 0x7fcdb807eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fcdb0008b10 tx=0x7fcdb000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:11.769 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:11.754+0000 7fcdbe63e700 1 --2- 172.21.15.42:0/623663434 >> v2:172.21.15.40:3300/0 conn(0x7fcdb809fae0 0x7fcdb807eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fcdb0008b10 tx=0x7fcdb000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:11.770 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:11.754+0000 7fcdbe63e700 1 --2- 172.21.15.42:0/623663434 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcdb809fae0 0x7fcdb807eb50 secure :-1 s=READY pgs=457 cs=0 l=1 rx=0x7fcdb0008b10 tx=0x7fcdb000fff0).ready entity=mon.0 client_cookie=e9a424ee085c3897 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:15:11.770 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:11.754+0000 7fcdbe63e700 1 --2- 172.21.15.42:0/623663434 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcdb809fae0 0x7fcdb807eb50 secure :-1 s=THROTTLE_DONE pgs=457 cs=0 l=1 rx=0x7fcdb0008b10 tx=0x7fcdb000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:11.770 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:11.754+0000 7fcdbce3b700 1 -- 172.21.15.42:0/623663434 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcdb0010900 con 0x7fcdb809fae0 2019-12-05T05:15:11.770 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:11.754+0000 7fcdbe63e700 1 --2- 172.21.15.42:0/623663434 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcdb809fae0 0x7fcdb807eb50 secure :-1 s=THROTTLE_DONE pgs=457 cs=0 l=1 rx=0x7fcdb0008b10 tx=0x7fcdb000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:11.770 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:11.754+0000 7fcdbe63e700 1 --2- 172.21.15.42:0/623663434 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcdb809fae0 0x7fcdb807eb50 secure :-1 s=THROTTLE_DONE pgs=457 cs=0 l=1 rx=0x7fcdb0008b10 tx=0x7fcdb000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:11.770 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:11.754+0000 7fcdbce3b700 1 -- 172.21.15.42:0/623663434 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdb0008b50 con 0x7fcdb809fae0 2019-12-05T05:15:11.770 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:11.758+0000 7fcdbce3b700 1 -- 172.21.15.42:0/623663434 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcdb00110c0 con 0x7fcdb809fae0 2019-12-05T05:15:11.771 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:11.758+0000 7fcdbf8a0700 1 -- 172.21.15.42:0/623663434 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcdb809fae0 msgr2=0x7fcdb807eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:15:11.771 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:11.758+0000 7fcdbf8a0700 1 --2- 172.21.15.42:0/623663434 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcdb809fae0 0x7fcdb807eb50 secure :-1 s=READY pgs=457 cs=0 l=1 rx=0x7fcdb0008b10 tx=0x7fcdb000fff0).stop 2019-12-05T05:15:11.771 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:11.758+0000 7fcdbf8a0700 1 -- 172.21.15.42:0/623663434 shutdown_connections 2019-12-05T05:15:11.771 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:11.758+0000 7fcdbf8a0700 1 --2- 172.21.15.42:0/623663434 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcdb809fae0 0x7fcdb807eb50 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:15:11.771 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:11.758+0000 7fcdbf8a0700 1 --2- 172.21.15.42:0/623663434 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcdb80a7740 0x7fcdb808df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:15:11.771 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:11.758+0000 7fcdbf8a0700 1 -- 172.21.15.42:0/623663434 shutdown_connections 2019-12-05T05:15:11.771 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:11.758+0000 7fcdbf8a0700 1 -- 172.21.15.42:0/623663434 wait complete. 2019-12-05T05:15:11.772 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:11.758+0000 7fcdbf8a0700 1 -- 172.21.15.42:0/623663434 >> 172.21.15.42:0/623663434 conn(0x7fcdb80aabe0 msgr2=0x7fcdb80a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:15:11.775 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:11.762+0000 7fcdbf8a0700 1 Processor -- start 2019-12-05T05:15:11.775 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:11.762+0000 7fcdbf8a0700 1 -- start start 2019-12-05T05:15:11.775 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:11.762+0000 7fcdbf8a0700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcdb812d100 0x7fcdb812d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:15:11.775 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:11.762+0000 7fcdbf8a0700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdb812da80 con 0x7fcdb81182d0 2019-12-05T05:15:11.775 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:11.762+0000 7fcdbf8a0700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdb811c3f0 con 0x7fcdb8135b00 2019-12-05T05:15:11.775 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:11.762+0000 7fcdbf8a0700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdb811a260 con 0x7fcdb812d100 2019-12-05T05:15:11.776 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:11.762+0000 7fcdbe63e700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fcdb8135b00 0x7fcdb8137f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:58850/0 (socket says 172.21.15.42:58850) 2019-12-05T05:15:11.776 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:11.762+0000 7fcdbe63e700 1 -- 172.21.15.42:0/3173165218 learned_addr learned my addr 172.21.15.42:0/3173165218 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:15:11.776 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:11.762+0000 7fcdbd63c700 1 --2- 172.21.15.42:0/3173165218 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcdb812d100 0x7fcdb812d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:15:11.776 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:11.762+0000 7fcdae7fc700 1 -- 172.21.15.42:0/3173165218 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3499521071 0 0) 0x7fcdb811c3f0 con 0x7fcdb8135b00 2019-12-05T05:15:11.776 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:11.762+0000 7fcdae7fc700 1 -- 172.21.15.42:0/3173165218 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcda00030b0 con 0x7fcdb8135b00 2019-12-05T05:15:11.776 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:11.762+0000 7fcdae7fc700 1 -- 172.21.15.42:0/3173165218 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1466234475 0 0) 0x7fcdb812da80 con 0x7fcdb81182d0 2019-12-05T05:15:11.776 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:11.762+0000 7fcdae7fc700 1 -- 172.21.15.42:0/3173165218 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcdb811c3f0 con 0x7fcdb81182d0 2019-12-05T05:15:11.777 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:11.762+0000 7fcdbd63c700 1 -- 172.21.15.42:0/3173165218 >> v1:172.21.15.42:6789/0 conn(0x7fcdb8135b00 legacy=0x7fcdb8137f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:15:11.777 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:11.762+0000 7fcdbd63c700 1 -- 172.21.15.42:0/3173165218 >> v1:172.21.15.40:6789/0 conn(0x7fcdb81182d0 legacy=0x7fcdb811b770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:15:11.777 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:11.762+0000 7fcdbd63c700 1 -- 172.21.15.42:0/3173165218 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdb8118d40 con 0x7fcdb812d100 2019-12-05T05:15:11.777 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:11.762+0000 7fcdbd63c700 1 --2- 172.21.15.42:0/3173165218 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcdb812d100 0x7fcdb812d530 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fcdb807ca40 tx=0x7fcda8010890).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:11.777 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:11.762+0000 7fcdbd63c700 1 --2- 172.21.15.42:0/3173165218 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcdb812d100 0x7fcdb812d530 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fcdb807ca40 tx=0x7fcda8010890).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:11.777 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:11.762+0000 7fcdbd63c700 1 --2- 172.21.15.42:0/3173165218 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcdb812d100 0x7fcdb812d530 secure :-1 s=READY pgs=119 cs=0 l=1 rx=0x7fcdb807ca40 tx=0x7fcda8010890).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:15:11.778 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:11.762+0000 7fcdbd63c700 1 --2- 172.21.15.42:0/3173165218 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcdb812d100 0x7fcdb812d530 secure :-1 s=THROTTLE_DONE pgs=119 cs=0 l=1 rx=0x7fcdb807ca40 tx=0x7fcda8010890).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:11.779 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:11.762+0000 7fcdae7fc700 1 -- 172.21.15.42:0/3173165218 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcda8011580 con 0x7fcdb812d100 2019-12-05T05:15:11.779 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:11.766+0000 7fcdbd63c700 1 --2- 172.21.15.42:0/3173165218 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcdb812d100 0x7fcdb812d530 secure :-1 s=THROTTLE_DONE pgs=119 cs=0 l=1 rx=0x7fcdb807ca40 tx=0x7fcda8010890).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:11.779 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:11.766+0000 7fcdbd63c700 1 --2- 172.21.15.42:0/3173165218 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcdb812d100 0x7fcdb812d530 secure :-1 s=THROTTLE_DONE pgs=119 cs=0 l=1 rx=0x7fcdb807ca40 tx=0x7fcda8010890).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:11.779 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:11.766+0000 7fcdae7fc700 1 -- 172.21.15.42:0/3173165218 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcda80111a0 con 0x7fcdb812d100 2019-12-05T05:15:11.779 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:11.766+0000 7fcdbf8a0700 1 -- 172.21.15.42:0/3173165218 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdb808eb70 con 0x7fcdb812d100 2019-12-05T05:15:11.780 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:11.766+0000 7fcdae7fc700 1 -- 172.21.15.42:0/3173165218 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcda8011b60 con 0x7fcdb812d100 2019-12-05T05:15:11.780 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:11.766+0000 7fcdbf8a0700 1 -- 172.21.15.42:0/3173165218 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdb808f0a0 con 0x7fcdb812d100 2019-12-05T05:15:11.780 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:11.766+0000 7fcdbd63c700 1 --2- 172.21.15.42:0/3173165218 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcdb812d100 0x7fcdb812d530 secure :-1 s=THROTTLE_DONE pgs=119 cs=0 l=1 rx=0x7fcdb807ca40 tx=0x7fcda8010890).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:11.780 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:11.766+0000 7fcdae7fc700 1 -- 172.21.15.42:0/3173165218 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fcda800b330 con 0x7fcdb812d100 2019-12-05T05:15:11.780 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:11.766+0000 7fcdbd63c700 1 --2- 172.21.15.42:0/3173165218 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcdb812d100 0x7fcdb812d530 secure :-1 s=THROTTLE_DONE pgs=119 cs=0 l=1 rx=0x7fcdb807ca40 tx=0x7fcda8010890).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:11.780 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:11.766+0000 7fcdae7fc700 1 --2- 172.21.15.42:0/3173165218 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fcda003b470 0x7fcda003d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:15:11.781 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:11.766+0000 7fcdae7fc700 1 -- 172.21.15.42:0/3173165218 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6805+0+0 (secure 0 0 0) 0x7fcda804af80 con 0x7fcdb812d100 2019-12-05T05:15:11.781 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:11.766+0000 7fcdbde3d700 1 --2- 172.21.15.42:0/3173165218 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fcda003b470 0x7fcda003d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:15:11.781 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:11.766+0000 7fcdbde3d700 1 --2- 172.21.15.42:0/3173165218 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fcda003b470 0x7fcda003d980 crc :-1 s=READY pgs=263 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:15:11.782 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:11.770+0000 7fcdbf8a0700 1 -- 172.21.15.42:0/3173165218 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdb80aade0 con 0x7fcdb812d100 2019-12-05T05:15:11.784 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:11.778+0000 7fcdbd63c700 1 --2- 172.21.15.42:0/3173165218 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcdb812d100 0x7fcdb812d530 secure :-1 s=THROTTLE_DONE pgs=119 cs=0 l=1 rx=0x7fcdb807ca40 tx=0x7fcda8010890).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:11.784 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:11.778+0000 7fcdae7fc700 1 -- 172.21.15.42:0/3173165218 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fcda8031fe0 con 0x7fcdb812d100 2019-12-05T05:15:11.954 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:11.946+0000 7fcdbf8a0700 1 -- 172.21.15.42:0/3173165218 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcdb805d110 con 0x7fcdb812d100 2019-12-05T05:15:11.954 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:11.946+0000 7fcdbd63c700 1 --2- 172.21.15.42:0/3173165218 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcdb812d100 0x7fcdb812d530 secure :-1 s=THROTTLE_DONE pgs=119 cs=0 l=1 rx=0x7fcdb807ca40 tx=0x7fcda8010890).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:11.955 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:11.946+0000 7fcdae7fc700 1 -- 172.21.15.42:0/3173165218 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 75 v75) v1 ==== 94+0+1775 (secure 0 0 0) 0x7fcda8032240 con 0x7fcdb812d100 2019-12-05T05:15:11.957 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 75 2019-12-05T05:15:11.957 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:15:11.957 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":75,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":75,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:15:11.416720+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5629},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5629":{"gid":5629,"name":"a","rank":0,"incarnation":74,"state":"up:reconnect","state_seq":13,"addr":"172.21.15.42:6835/3264829400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3264829400},{"type":"v1","addr":"172.21.15.42:6835","nonce":3264829400}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:15:11.960 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:11.950+0000 7fcdbf8a0700 1 -- 172.21.15.42:0/3173165218 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fcda003b470 msgr2=0x7fcda003d980 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:15:11.960 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:11.950+0000 7fcdbf8a0700 1 --2- 172.21.15.42:0/3173165218 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fcda003b470 0x7fcda003d980 crc :-1 s=READY pgs=263 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:15:11.961 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:11.950+0000 7fcdbf8a0700 1 -- 172.21.15.42:0/3173165218 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcdb812d100 msgr2=0x7fcdb812d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:15:11.961 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:11.950+0000 7fcdbf8a0700 1 --2- 172.21.15.42:0/3173165218 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcdb812d100 0x7fcdb812d530 secure :-1 s=READY pgs=119 cs=0 l=1 rx=0x7fcdb807ca40 tx=0x7fcda8010890).stop 2019-12-05T05:15:11.961 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:11.950+0000 7fcdbf8a0700 1 -- 172.21.15.42:0/3173165218 shutdown_connections 2019-12-05T05:15:11.961 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:11.950+0000 7fcdbf8a0700 1 --2- 172.21.15.42:0/3173165218 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fcda003b470 0x7fcda003d980 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:15:11.961 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:11.950+0000 7fcdbf8a0700 1 --2- 172.21.15.42:0/3173165218 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcdb812d100 0x7fcdb812d530 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:15:11.961 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:11.950+0000 7fcdbf8a0700 1 -- 172.21.15.42:0/3173165218 shutdown_connections 2019-12-05T05:15:11.961 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:11.954+0000 7fcdbf8a0700 1 -- 172.21.15.42:0/3173165218 wait complete. 2019-12-05T05:15:11.962 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:11.954+0000 7fcdbf8a0700 1 -- 172.21.15.42:0/3173165218 >> 172.21.15.42:0/3173165218 conn(0x7fcdb80a9aa0 msgr2=0x7fcdb811c810 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:15:11.970 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2019-12-05T05:15:13.971 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:15:14.076 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:14.062+0000 7f318b26a700 1 Processor -- start 2019-12-05T05:15:14.076 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:14.062+0000 7f318b26a700 1 -- start start 2019-12-05T05:15:14.077 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:14.070+0000 7f318b26a700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f31840a7740 0x7f318408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:15:14.077 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:14.070+0000 7f318b26a700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31840a0710 con 0x7f318407e720 2019-12-05T05:15:14.077 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:14.070+0000 7f318b26a700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31840a9ba0 con 0x7f318409fae0 2019-12-05T05:15:14.077 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:14.070+0000 7f318b26a700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31840a0b00 con 0x7f31840a7740 2019-12-05T05:15:14.077 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:14.070+0000 7f3189807700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f31840a7740 0x7f318408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:15:14.078 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:14.070+0000 7f3189807700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f31840a7740 0x7f318408df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:58816/0 (socket says 172.21.15.42:58816) 2019-12-05T05:15:14.078 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:14.070+0000 7f3189807700 1 -- 172.21.15.42:0/383078078 learned_addr learned my addr 172.21.15.42:0/383078078 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:15:14.078 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:14.070+0000 7f3189807700 1 -- 172.21.15.42:0/383078078 >> v1:172.21.15.42:6789/0 conn(0x7f318409fae0 legacy=0x7f318414efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:15:14.078 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:14.070+0000 7f3189807700 1 -- 172.21.15.42:0/383078078 >> v1:172.21.15.40:6789/0 conn(0x7f318407e720 legacy=0x7f318409f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:15:14.078 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:14.070+0000 7f3189807700 1 -- 172.21.15.42:0/383078078 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f318407eb50 con 0x7f31840a7740 2019-12-05T05:15:14.078 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:14.070+0000 7f3189807700 1 --2- 172.21.15.42:0/383078078 >> v2:172.21.15.42:3300/0 conn(0x7f31840a7740 0x7f318408df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f3180008b10 tx=0x7f3180010030).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:14.078 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:14.070+0000 7f3189807700 1 --2- 172.21.15.42:0/383078078 >> v2:172.21.15.42:3300/0 conn(0x7f31840a7740 0x7f318408df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f3180008b10 tx=0x7f3180010030).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:14.079 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:14.070+0000 7f3189807700 1 --2- 172.21.15.42:0/383078078 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f31840a7740 0x7f318408df80 secure :-1 s=READY pgs=583 cs=0 l=1 rx=0x7f3180008b10 tx=0x7f3180010030).ready entity=mon.1 client_cookie=255ed451540f7c0b server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:15:14.079 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:14.070+0000 7f3189807700 1 --2- 172.21.15.42:0/383078078 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f31840a7740 0x7f318408df80 secure :-1 s=THROTTLE_DONE pgs=583 cs=0 l=1 rx=0x7f3180008b10 tx=0x7f3180010030).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:14.079 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:14.070+0000 7f3188805700 1 -- 172.21.15.42:0/383078078 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f318000a160 con 0x7f31840a7740 2019-12-05T05:15:14.079 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:14.070+0000 7f3189807700 1 --2- 172.21.15.42:0/383078078 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f31840a7740 0x7f318408df80 secure :-1 s=THROTTLE_DONE pgs=583 cs=0 l=1 rx=0x7f3180008b10 tx=0x7f3180010030).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:14.079 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:14.070+0000 7f3189807700 1 --2- 172.21.15.42:0/383078078 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f31840a7740 0x7f318408df80 secure :-1 s=THROTTLE_DONE pgs=583 cs=0 l=1 rx=0x7f3180008b10 tx=0x7f3180010030).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:14.079 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:14.070+0000 7f3188805700 1 -- 172.21.15.42:0/383078078 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3180011aa0 con 0x7f31840a7740 2019-12-05T05:15:14.079 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:14.070+0000 7f3188805700 1 -- 172.21.15.42:0/383078078 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f318000a740 con 0x7f31840a7740 2019-12-05T05:15:14.080 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:14.070+0000 7f318b26a700 1 -- 172.21.15.42:0/383078078 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f31840a7740 msgr2=0x7f318408df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:15:14.080 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:14.070+0000 7f318b26a700 1 --2- 172.21.15.42:0/383078078 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f31840a7740 0x7f318408df80 secure :-1 s=READY pgs=583 cs=0 l=1 rx=0x7f3180008b10 tx=0x7f3180010030).stop 2019-12-05T05:15:14.080 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:14.070+0000 7f318b26a700 1 -- 172.21.15.42:0/383078078 shutdown_connections 2019-12-05T05:15:14.080 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:14.070+0000 7f318b26a700 1 --2- 172.21.15.42:0/383078078 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f31840a7740 0x7f318408df80 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:15:14.080 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:14.070+0000 7f318b26a700 1 -- 172.21.15.42:0/383078078 shutdown_connections 2019-12-05T05:15:14.080 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:14.070+0000 7f318b26a700 1 -- 172.21.15.42:0/383078078 wait complete. 2019-12-05T05:15:14.080 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:14.070+0000 7f318b26a700 1 -- 172.21.15.42:0/383078078 >> 172.21.15.42:0/383078078 conn(0x7f31840aabe0 msgr2=0x7f31840a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:15:14.084 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:14.070+0000 7f318b26a700 1 Processor -- start 2019-12-05T05:15:14.084 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:14.070+0000 7f318b26a700 1 -- start start 2019-12-05T05:15:14.084 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:14.070+0000 7f318b26a700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f31841455d0 0x7f3184143000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:15:14.084 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:14.070+0000 7f318b26a700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3184147fc0 0x7f3184143550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:15:14.084 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:14.070+0000 7f318b26a700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f318408eb70 0x7f318408efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:15:14.084 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:14.074+0000 7f318b26a700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3184112590 con 0x7f31841455d0 2019-12-05T05:15:14.085 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:14.074+0000 7f318b26a700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3184112830 con 0x7f3184147fc0 2019-12-05T05:15:14.085 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:14.074+0000 7f318b26a700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3184112ad0 con 0x7f318408eb70 2019-12-05T05:15:14.085 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:14.074+0000 7f3189006700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3184147fc0 0x7f3184143550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:15:14.085 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:14.074+0000 7f3189006700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3184147fc0 0x7f3184143550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:58824/0 (socket says 172.21.15.42:58824) 2019-12-05T05:15:14.085 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:14.074+0000 7f3189006700 1 -- 172.21.15.42:0/3330981 learned_addr learned my addr 172.21.15.42:0/3330981 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:15:14.085 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:14.074+0000 7f3189807700 1 --2- 172.21.15.42:0/3330981 >> v2:172.21.15.40:3300/0 conn(0x7f31841455d0 0x7f3184143000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:15:14.085 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:14.074+0000 7f318a008700 1 --2- 172.21.15.42:0/3330981 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f318408eb70 0x7f318408efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:15:14.086 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:14.074+0000 7f3189006700 1 -- 172.21.15.42:0/3330981 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f318408eb70 msgr2=0x7f318408efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:15:14.086 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:14.074+0000 7f3189006700 1 --2- 172.21.15.42:0/3330981 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f318408eb70 0x7f318408efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:15:14.086 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:14.074+0000 7f3189006700 1 -- 172.21.15.42:0/3330981 >> v2:172.21.15.40:3300/0 conn(0x7f31841455d0 msgr2=0x7f3184143000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:15:14.086 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:14.074+0000 7f3189006700 1 --2- 172.21.15.42:0/3330981 >> v2:172.21.15.40:3300/0 conn(0x7f31841455d0 0x7f3184143000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:15:14.086 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:14.074+0000 7f3189006700 1 -- 172.21.15.42:0/3330981 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3184143aa0 con 0x7f3184147fc0 2019-12-05T05:15:14.086 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:14.074+0000 7f318a008700 1 --2- 172.21.15.42:0/3330981 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f318408eb70 0x7f318408efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T05:15:14.086 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:14.074+0000 7f3189006700 1 --2- 172.21.15.42:0/3330981 >> v2:172.21.15.42:3300/0 conn(0x7f3184147fc0 0x7f3184143550 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f3174002ef0 tx=0x7f31740109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:14.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:14.074+0000 7f3189006700 1 --2- 172.21.15.42:0/3330981 >> v2:172.21.15.42:3300/0 conn(0x7f3184147fc0 0x7f3184143550 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f3174002ef0 tx=0x7f31740109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:14.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:14.074+0000 7f3189006700 1 --2- 172.21.15.42:0/3330981 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3184147fc0 0x7f3184143550 secure :-1 s=READY pgs=584 cs=0 l=1 rx=0x7f3174002ef0 tx=0x7f31740109a0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:15:14.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:14.074+0000 7f3189006700 1 --2- 172.21.15.42:0/3330981 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3184147fc0 0x7f3184143550 secure :-1 s=THROTTLE_DONE pgs=584 cs=0 l=1 rx=0x7f3174002ef0 tx=0x7f31740109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:14.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:14.074+0000 7f317a7fc700 1 -- 172.21.15.42:0/3330981 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3174008810 con 0x7f3184147fc0 2019-12-05T05:15:14.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:14.074+0000 7f3189006700 1 --2- 172.21.15.42:0/3330981 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3184147fc0 0x7f3184143550 secure :-1 s=THROTTLE_DONE pgs=584 cs=0 l=1 rx=0x7f3174002ef0 tx=0x7f31740109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:14.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:14.074+0000 7f3189006700 1 --2- 172.21.15.42:0/3330981 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3184147fc0 0x7f3184143550 secure :-1 s=THROTTLE_DONE pgs=584 cs=0 l=1 rx=0x7f3174002ef0 tx=0x7f31740109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:14.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:14.074+0000 7f317a7fc700 1 -- 172.21.15.42:0/3330981 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3174007160 con 0x7f3184147fc0 2019-12-05T05:15:14.088 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:14.074+0000 7f318b26a700 1 -- 172.21.15.42:0/3330981 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3184143ec0 con 0x7f3184147fc0 2019-12-05T05:15:14.088 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:14.074+0000 7f317a7fc700 1 -- 172.21.15.42:0/3330981 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3174008df0 con 0x7f3184147fc0 2019-12-05T05:15:14.088 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:14.074+0000 7f318b26a700 1 -- 172.21.15.42:0/3330981 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f318408f4f0 con 0x7f3184147fc0 2019-12-05T05:15:14.088 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:14.074+0000 7f3189006700 1 --2- 172.21.15.42:0/3330981 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3184147fc0 0x7f3184143550 secure :-1 s=THROTTLE_DONE pgs=584 cs=0 l=1 rx=0x7f3174002ef0 tx=0x7f31740109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:14.088 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:14.074+0000 7f317a7fc700 1 -- 172.21.15.42:0/3330981 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f31740057e0 con 0x7f3184147fc0 2019-12-05T05:15:14.088 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:14.074+0000 7f3189006700 1 --2- 172.21.15.42:0/3330981 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3184147fc0 0x7f3184143550 secure :-1 s=THROTTLE_DONE pgs=584 cs=0 l=1 rx=0x7f3174002ef0 tx=0x7f31740109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:14.088 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:14.074+0000 7f317a7fc700 1 --2- 172.21.15.42:0/3330981 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f316c03add0 0x7f316c03d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:15:14.089 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:14.074+0000 7f317a7fc700 1 -- 172.21.15.42:0/3330981 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6805+0+0 (secure 0 0 0) 0x7f317404ad50 con 0x7f3184147fc0 2019-12-05T05:15:14.089 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:14.074+0000 7f3189807700 1 --2- 172.21.15.42:0/3330981 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f316c03add0 0x7f316c03d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:15:14.089 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:14.078+0000 7f3189807700 1 --2- 172.21.15.42:0/3330981 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f316c03add0 0x7f316c03d2e0 crc :-1 s=READY pgs=264 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:15:14.089 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:14.078+0000 7f318b26a700 1 -- 172.21.15.42:0/3330981 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f318405d110 con 0x7f3184147fc0 2019-12-05T05:15:14.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:14.086+0000 7f3189006700 1 --2- 172.21.15.42:0/3330981 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3184147fc0 0x7f3184143550 secure :-1 s=THROTTLE_DONE pgs=584 cs=0 l=1 rx=0x7f3174002ef0 tx=0x7f31740109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:14.100 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:14.086+0000 7f317a7fc700 1 -- 172.21.15.42:0/3330981 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f3174031dd0 con 0x7f3184147fc0 2019-12-05T05:15:14.279 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:14.270+0000 7f318b26a700 1 -- 172.21.15.42:0/3330981 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f31840aadf0 con 0x7f3184147fc0 2019-12-05T05:15:14.280 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:14.270+0000 7f3189006700 1 --2- 172.21.15.42:0/3330981 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3184147fc0 0x7f3184143550 secure :-1 s=THROTTLE_DONE pgs=584 cs=0 l=1 rx=0x7f3174002ef0 tx=0x7f31740109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:14.281 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:14.274+0000 7f317a7fc700 1 -- 172.21.15.42:0/3330981 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) v1 ==== 94+0+1772 (secure 0 0 0) 0x7f3174032030 con 0x7f3184147fc0 2019-12-05T05:15:14.282 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 77 2019-12-05T05:15:14.282 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:15:14.282 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":77,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":77,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:15:13.428215+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5629},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5629":{"gid":5629,"name":"a","rank":0,"incarnation":74,"state":"up:active","state_seq":15,"addr":"172.21.15.42:6835/3264829400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3264829400},{"type":"v1","addr":"172.21.15.42:6835","nonce":3264829400}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:15:14.286 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:14.278+0000 7f318b26a700 1 -- 172.21.15.42:0/3330981 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f316c03add0 msgr2=0x7f316c03d2e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:15:14.286 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:14.278+0000 7f318b26a700 1 --2- 172.21.15.42:0/3330981 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f316c03add0 0x7f316c03d2e0 crc :-1 s=READY pgs=264 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:15:14.286 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:14.278+0000 7f318b26a700 1 -- 172.21.15.42:0/3330981 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3184147fc0 msgr2=0x7f3184143550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:15:14.287 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:14.278+0000 7f318b26a700 1 --2- 172.21.15.42:0/3330981 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3184147fc0 0x7f3184143550 secure :-1 s=READY pgs=584 cs=0 l=1 rx=0x7f3174002ef0 tx=0x7f31740109a0).stop 2019-12-05T05:15:14.287 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:14.278+0000 7f318b26a700 1 -- 172.21.15.42:0/3330981 shutdown_connections 2019-12-05T05:15:14.287 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:14.278+0000 7f318b26a700 1 --2- 172.21.15.42:0/3330981 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f316c03add0 0x7f316c03d2e0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:15:14.287 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:14.278+0000 7f318b26a700 1 --2- 172.21.15.42:0/3330981 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f318408eb70 0x7f318408efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:15:14.288 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:14.278+0000 7f318b26a700 1 --2- 172.21.15.42:0/3330981 >> v2:172.21.15.40:3300/0 conn(0x7f31841455d0 0x7f3184143000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:15:14.288 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:14.278+0000 7f318b26a700 1 --2- 172.21.15.42:0/3330981 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3184147fc0 0x7f3184143550 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:15:14.288 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:14.278+0000 7f318b26a700 1 -- 172.21.15.42:0/3330981 shutdown_connections 2019-12-05T05:15:14.288 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:14.278+0000 7f318b26a700 1 -- 172.21.15.42:0/3330981 wait complete. 2019-12-05T05:15:14.288 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:14.278+0000 7f318b26a700 1 -- 172.21.15.42:0/3330981 >> 172.21.15.42:0/3330981 conn(0x7f318407e560 msgr2=0x7f318407c720 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:15:14.295 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a has gained rank=0, replacing gid=4841 2019-12-05T05:15:14.295 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 7 secs before reviving mds.b 2019-12-05T05:15:21.296 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.b 2019-12-05T05:15:21.296 INFO:tasks.ceph.mds.b:Restarting daemon 2019-12-05T05:15:21.296 INFO:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2019-12-05T05:15:21.300 INFO:tasks.ceph.mds.b:Started 2019-12-05T05:15:21.300 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:15:21.361 INFO:tasks.ceph.mds.b.smithi040.stdout:starting mds.b at 2019-12-05T05:15:21.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:21.394+0000 7f9909681700 1 Processor -- start 2019-12-05T05:15:21.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:21.394+0000 7f9909681700 1 -- start start 2019-12-05T05:15:21.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:21.398+0000 7f9909681700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f990408df80 0x7f990409f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:15:21.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:21.398+0000 7f9909681700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f990409f830 0x7f990414efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:15:21.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:21.398+0000 7f9909681700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f990409fc60 con 0x7f99040a7740 2019-12-05T05:15:21.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:21.398+0000 7f9909681700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99040a0b00 con 0x7f990409f830 2019-12-05T05:15:21.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:21.398+0000 7f9909681700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f990408e5b0 con 0x7f990408df80 2019-12-05T05:15:21.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:21.398+0000 7f99037fe700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f99040a7740 0x7f990407e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:58878/0 (socket says 172.21.15.42:58878) 2019-12-05T05:15:21.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:21.398+0000 7f99037fe700 1 -- 172.21.15.42:0/3715155005 learned_addr learned my addr 172.21.15.42:0/3715155005 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:15:21.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:21.398+0000 7f9903fff700 1 --2- 172.21.15.42:0/3715155005 >> v2:172.21.15.42:3300/0 conn(0x7f990409f830 0x7f990414efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:15:21.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:21.398+0000 7f9902ffd700 1 --2- 172.21.15.42:0/3715155005 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f990408df80 0x7f990409f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:15:21.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:21.398+0000 7f99027fc700 1 -- 172.21.15.42:0/3715155005 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 627992798 0 0) 0x7f990409fc60 con 0x7f99040a7740 2019-12-05T05:15:21.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:21.398+0000 7f99027fc700 1 -- 172.21.15.42:0/3715155005 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98ec0030b0 con 0x7f99040a7740 2019-12-05T05:15:21.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:21.398+0000 7f9903fff700 1 -- 172.21.15.42:0/3715155005 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f990408df80 msgr2=0x7f990409f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:15:21.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:21.398+0000 7f9903fff700 1 --2- 172.21.15.42:0/3715155005 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f990408df80 0x7f990409f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:15:21.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:21.398+0000 7f9903fff700 1 -- 172.21.15.42:0/3715155005 >> v1:172.21.15.42:6789/0 conn(0x7f99040a7740 legacy=0x7f990407e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:15:21.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:21.398+0000 7f9903fff700 1 -- 172.21.15.42:0/3715155005 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99040a0710 con 0x7f990409f830 2019-12-05T05:15:21.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:21.398+0000 7f9902ffd700 1 --2- 172.21.15.42:0/3715155005 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f990408df80 0x7f990409f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:15:21.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:21.398+0000 7f9903fff700 1 --2- 172.21.15.42:0/3715155005 >> v2:172.21.15.42:3300/0 conn(0x7f990409f830 0x7f990414efe0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f98fc008790 tx=0x7f98fc00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:21.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:21.398+0000 7f9903fff700 1 --2- 172.21.15.42:0/3715155005 >> v2:172.21.15.42:3300/0 conn(0x7f990409f830 0x7f990414efe0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f98fc008790 tx=0x7f98fc00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:21.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:21.398+0000 7f9903fff700 1 --2- 172.21.15.42:0/3715155005 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f990409f830 0x7f990414efe0 secure :-1 s=READY pgs=589 cs=0 l=1 rx=0x7f98fc008790 tx=0x7f98fc00fc70).ready entity=mon.1 client_cookie=37b6b8001af9ec72 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:15:21.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:21.398+0000 7f9903fff700 1 --2- 172.21.15.42:0/3715155005 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f990409f830 0x7f990414efe0 secure :-1 s=THROTTLE_DONE pgs=589 cs=0 l=1 rx=0x7f98fc008790 tx=0x7f98fc00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:21.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:21.398+0000 7f99027fc700 1 -- 172.21.15.42:0/3715155005 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f98fc008bb0 con 0x7f990409f830 2019-12-05T05:15:21.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:21.398+0000 7f9903fff700 1 --2- 172.21.15.42:0/3715155005 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f990409f830 0x7f990414efe0 secure :-1 s=THROTTLE_DONE pgs=589 cs=0 l=1 rx=0x7f98fc008790 tx=0x7f98fc00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:21.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:21.398+0000 7f9903fff700 1 --2- 172.21.15.42:0/3715155005 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f990409f830 0x7f990414efe0 secure :-1 s=THROTTLE_DONE pgs=589 cs=0 l=1 rx=0x7f98fc008790 tx=0x7f98fc00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:21.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:21.398+0000 7f99027fc700 1 -- 172.21.15.42:0/3715155005 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98fc0087d0 con 0x7f990409f830 2019-12-05T05:15:21.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:21.398+0000 7f99027fc700 1 -- 172.21.15.42:0/3715155005 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f98fc011860 con 0x7f990409f830 2019-12-05T05:15:21.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:21.398+0000 7f9909681700 1 -- 172.21.15.42:0/3715155005 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f990409f830 msgr2=0x7f990414efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:15:21.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:21.398+0000 7f9909681700 1 --2- 172.21.15.42:0/3715155005 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f990409f830 0x7f990414efe0 secure :-1 s=READY pgs=589 cs=0 l=1 rx=0x7f98fc008790 tx=0x7f98fc00fc70).stop 2019-12-05T05:15:21.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:21.398+0000 7f9909681700 1 -- 172.21.15.42:0/3715155005 shutdown_connections 2019-12-05T05:15:21.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:21.398+0000 7f9909681700 1 --2- 172.21.15.42:0/3715155005 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f990409f830 0x7f990414efe0 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:15:21.411 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:21.398+0000 7f9909681700 1 --2- 172.21.15.42:0/3715155005 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f990408df80 0x7f990409f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:15:21.411 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:21.398+0000 7f9909681700 1 -- 172.21.15.42:0/3715155005 shutdown_connections 2019-12-05T05:15:21.412 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:21.398+0000 7f9909681700 1 -- 172.21.15.42:0/3715155005 wait complete. 2019-12-05T05:15:21.412 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:21.398+0000 7f9909681700 1 -- 172.21.15.42:0/3715155005 >> 172.21.15.42:0/3715155005 conn(0x7f99040aabe0 msgr2=0x7f99040a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:15:21.415 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:21.402+0000 7f9909681700 1 Processor -- start 2019-12-05T05:15:21.415 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:21.402+0000 7f9909681700 1 -- start start 2019-12-05T05:15:21.415 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:21.406+0000 7f9909681700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f9904120c90 0x7f99040a3910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:15:21.416 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:21.406+0000 7f9909681700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f990412dee0 0x7f990412e310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:15:21.416 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:21.406+0000 7f9909681700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99041198d0 con 0x7f9904131400 2019-12-05T05:15:21.416 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:21.406+0000 7f9909681700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f990408f4c0 con 0x7f9904120c90 2019-12-05T05:15:21.416 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:21.406+0000 7f9909681700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f990408f760 con 0x7f990412dee0 2019-12-05T05:15:21.416 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:21.406+0000 7f9902ffd700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f990412dee0 0x7f990412e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:15:21.416 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:21.406+0000 7f9902ffd700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f990412dee0 0x7f990412e310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:58848/0 (socket says 172.21.15.42:58848) 2019-12-05T05:15:21.416 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:21.406+0000 7f9902ffd700 1 -- 172.21.15.42:0/527973322 learned_addr learned my addr 172.21.15.42:0/527973322 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:15:21.417 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:21.406+0000 7f99037fe700 1 --2- 172.21.15.42:0/527973322 >> v2:172.21.15.40:3300/0 conn(0x7f9904120c90 0x7f99040a3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:15:21.417 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:21.406+0000 7f9902ffd700 1 -- 172.21.15.42:0/527973322 >> v2:172.21.15.40:3300/0 conn(0x7f9904120c90 msgr2=0x7f99040a3910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:15:21.417 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:21.406+0000 7f9902ffd700 1 --2- 172.21.15.42:0/527973322 >> v2:172.21.15.40:3300/0 conn(0x7f9904120c90 0x7f99040a3910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:15:21.417 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:21.406+0000 7f9902ffd700 1 -- 172.21.15.42:0/527973322 >> v1:172.21.15.40:6789/0 conn(0x7f9904131400 legacy=0x7f9904131830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:15:21.417 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:21.406+0000 7f9902ffd700 1 -- 172.21.15.42:0/527973322 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9904119f30 con 0x7f990412dee0 2019-12-05T05:15:21.417 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:21.406+0000 7f9902ffd700 1 --2- 172.21.15.42:0/527973322 >> v2:172.21.15.42:3300/0 conn(0x7f990412dee0 0x7f990412e310 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f98f8002ef0 tx=0x7f98f8010db0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:21.417 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:21.406+0000 7f9902ffd700 1 --2- 172.21.15.42:0/527973322 >> v2:172.21.15.42:3300/0 conn(0x7f990412dee0 0x7f990412e310 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f98f8002ef0 tx=0x7f98f8010db0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:21.418 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:21.406+0000 7f9902ffd700 1 --2- 172.21.15.42:0/527973322 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f990412dee0 0x7f990412e310 secure :-1 s=READY pgs=590 cs=0 l=1 rx=0x7f98f8002ef0 tx=0x7f98f8010db0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:15:21.418 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:21.406+0000 7f9902ffd700 1 --2- 172.21.15.42:0/527973322 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f990412dee0 0x7f990412e310 secure :-1 s=THROTTLE_DONE pgs=590 cs=0 l=1 rx=0x7f98f8002ef0 tx=0x7f98f8010db0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:21.418 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:21.406+0000 7f98ebfff700 1 -- 172.21.15.42:0/527973322 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f98f80058e0 con 0x7f990412dee0 2019-12-05T05:15:21.418 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:21.406+0000 7f9902ffd700 1 --2- 172.21.15.42:0/527973322 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f990412dee0 0x7f990412e310 secure :-1 s=THROTTLE_DONE pgs=590 cs=0 l=1 rx=0x7f98f8002ef0 tx=0x7f98f8010db0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:21.418 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:21.406+0000 7f9902ffd700 1 --2- 172.21.15.42:0/527973322 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f990412dee0 0x7f990412e310 secure :-1 s=THROTTLE_DONE pgs=590 cs=0 l=1 rx=0x7f98f8002ef0 tx=0x7f98f8010db0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:21.418 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:21.406+0000 7f98ebfff700 1 -- 172.21.15.42:0/527973322 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98f8007160 con 0x7f990412dee0 2019-12-05T05:15:21.419 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:21.406+0000 7f9909681700 1 -- 172.21.15.42:0/527973322 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f990412e860 con 0x7f990412dee0 2019-12-05T05:15:21.419 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:21.406+0000 7f98ebfff700 1 -- 172.21.15.42:0/527973322 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f98f8012430 con 0x7f990412dee0 2019-12-05T05:15:21.419 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:21.406+0000 7f9909681700 1 -- 172.21.15.42:0/527973322 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9904132030 con 0x7f990412dee0 2019-12-05T05:15:21.419 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:21.406+0000 7f9902ffd700 1 --2- 172.21.15.42:0/527973322 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f990412dee0 0x7f990412e310 secure :-1 s=THROTTLE_DONE pgs=590 cs=0 l=1 rx=0x7f98f8002ef0 tx=0x7f98f8010db0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:21.420 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:21.410+0000 7f98ebfff700 1 -- 172.21.15.42:0/527973322 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f98f8010180 con 0x7f990412dee0 2019-12-05T05:15:21.420 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:21.410+0000 7f9902ffd700 1 --2- 172.21.15.42:0/527973322 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f990412dee0 0x7f990412e310 secure :-1 s=THROTTLE_DONE pgs=590 cs=0 l=1 rx=0x7f98f8002ef0 tx=0x7f98f8010db0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:21.420 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:21.410+0000 7f98ebfff700 1 --2- 172.21.15.42:0/527973322 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f98e403acf0 0x7f98e403d200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:15:21.420 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:21.410+0000 7f98ebfff700 1 -- 172.21.15.42:0/527973322 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6805+0+0 (secure 0 0 0) 0x7f98f804b0a0 con 0x7f990412dee0 2019-12-05T05:15:21.420 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:21.410+0000 7f99037fe700 1 --2- 172.21.15.42:0/527973322 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f98e403acf0 0x7f98e403d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:15:21.420 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:21.410+0000 7f99037fe700 1 --2- 172.21.15.42:0/527973322 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f98e403acf0 0x7f98e403d200 crc :-1 s=READY pgs=265 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:15:21.421 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:21.410+0000 7f9909681700 1 -- 172.21.15.42:0/527973322 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f990405d110 con 0x7f990412dee0 2019-12-05T05:15:21.422 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:21.414+0000 7f9902ffd700 1 --2- 172.21.15.42:0/527973322 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f990412dee0 0x7f990412e310 secure :-1 s=THROTTLE_DONE pgs=590 cs=0 l=1 rx=0x7f98f8002ef0 tx=0x7f98f8010db0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:21.422 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:21.414+0000 7f98ebfff700 1 -- 172.21.15.42:0/527973322 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f98f8032120 con 0x7f990412dee0 2019-12-05T05:15:21.581 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:21.578+0000 7f9909681700 1 -- 172.21.15.42:0/527973322 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f990405b8c0 con 0x7f990412dee0 2019-12-05T05:15:21.582 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:21.578+0000 7f9902ffd700 1 --2- 172.21.15.42:0/527973322 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f990412dee0 0x7f990412e310 secure :-1 s=THROTTLE_DONE pgs=590 cs=0 l=1 rx=0x7f98f8002ef0 tx=0x7f98f8010db0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:21.582 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:21.578+0000 7f98ebfff700 1 -- 172.21.15.42:0/527973322 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) v1 ==== 94+0+1772 (secure 0 0 0) 0x7f98f8032380 con 0x7f990412dee0 2019-12-05T05:15:21.584 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:15:21.585 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":77,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":77,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:15:13.428215+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5629},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5629":{"gid":5629,"name":"a","rank":0,"incarnation":74,"state":"up:active","state_seq":15,"addr":"172.21.15.42:6835/3264829400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3264829400},{"type":"v1","addr":"172.21.15.42:6835","nonce":3264829400}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:15:21.585 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 77 2019-12-05T05:15:21.587 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:21.582+0000 7f9909681700 1 -- 172.21.15.42:0/527973322 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f98e403acf0 msgr2=0x7f98e403d200 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:15:21.587 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:21.582+0000 7f9909681700 1 --2- 172.21.15.42:0/527973322 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f98e403acf0 0x7f98e403d200 crc :-1 s=READY pgs=265 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:15:21.587 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:21.582+0000 7f9909681700 1 -- 172.21.15.42:0/527973322 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f990412dee0 msgr2=0x7f990412e310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:15:21.587 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:21.582+0000 7f9909681700 1 --2- 172.21.15.42:0/527973322 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f990412dee0 0x7f990412e310 secure :-1 s=READY pgs=590 cs=0 l=1 rx=0x7f98f8002ef0 tx=0x7f98f8010db0).stop 2019-12-05T05:15:21.588 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:21.582+0000 7f9909681700 1 -- 172.21.15.42:0/527973322 shutdown_connections 2019-12-05T05:15:21.588 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:21.582+0000 7f9909681700 1 --2- 172.21.15.42:0/527973322 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f98e403acf0 0x7f98e403d200 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:15:21.588 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:21.582+0000 7f9909681700 1 --2- 172.21.15.42:0/527973322 >> v2:172.21.15.40:3300/0 conn(0x7f9904120c90 0x7f99040a3910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:15:21.588 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:21.582+0000 7f9909681700 1 --2- 172.21.15.42:0/527973322 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f990412dee0 0x7f990412e310 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:15:21.588 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:21.582+0000 7f9909681700 1 -- 172.21.15.42:0/527973322 shutdown_connections 2019-12-05T05:15:21.588 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:21.582+0000 7f9909681700 1 -- 172.21.15.42:0/527973322 wait complete. 2019-12-05T05:15:21.589 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:21.582+0000 7f9909681700 1 -- 172.21.15.42:0/527973322 >> 172.21.15.42:0/527973322 conn(0x7f9904067080 msgr2=0x7f99040a6b40 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:15:21.597 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2019-12-05T05:15:23.598 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:15:23.705 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:23.694+0000 7f6db3bc6700 1 Processor -- start 2019-12-05T05:15:23.705 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:23.694+0000 7f6db3bc6700 1 -- start start 2019-12-05T05:15:23.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:23.698+0000 7f6db3bc6700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6dac0a7740 0x7f6dac08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:15:23.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:23.698+0000 7f6db3bc6700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6dac0a0710 con 0x7f6dac09fae0 2019-12-05T05:15:23.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:23.698+0000 7f6db3bc6700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6dac0a9ba0 con 0x7f6dac07e720 2019-12-05T05:15:23.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:23.698+0000 7f6db3bc6700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6dac0a0b00 con 0x7f6dac0a7740 2019-12-05T05:15:23.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:23.698+0000 7f6db2163700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6dac0a7740 0x7f6dac08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:15:23.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:23.698+0000 7f6db1962700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f6dac07e720 0x7f6dac09f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:58894/0 (socket says 172.21.15.42:58894) 2019-12-05T05:15:23.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:23.698+0000 7f6db1962700 1 -- 172.21.15.42:0/911026164 learned_addr learned my addr 172.21.15.42:0/911026164 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:15:23.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:23.698+0000 7f6db1161700 1 -- 172.21.15.42:0/911026164 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2148847149 0 0) 0x7f6dac0a9ba0 con 0x7f6dac07e720 2019-12-05T05:15:23.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:23.698+0000 7f6db1161700 1 -- 172.21.15.42:0/911026164 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d980030b0 con 0x7f6dac07e720 2019-12-05T05:15:23.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:23.698+0000 7f6db1161700 1 -- 172.21.15.42:0/911026164 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3597318206 0 0) 0x7f6dac0a0710 con 0x7f6dac09fae0 2019-12-05T05:15:23.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:23.698+0000 7f6db1161700 1 -- 172.21.15.42:0/911026164 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6dac0a9ba0 con 0x7f6dac09fae0 2019-12-05T05:15:23.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:23.698+0000 7f6db1161700 1 -- 172.21.15.42:0/911026164 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3317561434 0 0) 0x7f6d980030b0 con 0x7f6dac07e720 2019-12-05T05:15:23.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:23.698+0000 7f6db2163700 1 -- 172.21.15.42:0/911026164 >> v1:172.21.15.42:6789/0 conn(0x7f6dac07e720 legacy=0x7f6dac09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:15:23.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:23.698+0000 7f6db2163700 1 -- 172.21.15.42:0/911026164 >> v1:172.21.15.40:6789/0 conn(0x7f6dac09fae0 legacy=0x7f6dac14efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:15:23.708 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:23.698+0000 7f6db2163700 1 -- 172.21.15.42:0/911026164 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dac07eb50 con 0x7f6dac0a7740 2019-12-05T05:15:23.708 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:23.698+0000 7f6db2163700 1 --2- 172.21.15.42:0/911026164 >> v2:172.21.15.42:3300/0 conn(0x7f6dac0a7740 0x7f6dac08df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f6da8008790 tx=0x7f6d9c001950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:23.708 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:23.698+0000 7f6db2163700 1 --2- 172.21.15.42:0/911026164 >> v2:172.21.15.42:3300/0 conn(0x7f6dac0a7740 0x7f6dac08df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f6da8008790 tx=0x7f6d9c001950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:23.708 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:23.698+0000 7f6db2163700 1 --2- 172.21.15.42:0/911026164 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6dac0a7740 0x7f6dac08df80 secure :-1 s=READY pgs=592 cs=0 l=1 rx=0x7f6da8008790 tx=0x7f6d9c001950).ready entity=mon.1 client_cookie=9ad0e7b4a6f703ab server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:15:23.708 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:23.698+0000 7f6db2163700 1 --2- 172.21.15.42:0/911026164 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6dac0a7740 0x7f6dac08df80 secure :-1 s=THROTTLE_DONE pgs=592 cs=0 l=1 rx=0x7f6da8008790 tx=0x7f6d9c001950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:23.708 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:23.698+0000 7f6db1161700 1 -- 172.21.15.42:0/911026164 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6d9c001750 con 0x7f6dac0a7740 2019-12-05T05:15:23.708 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:23.698+0000 7f6db2163700 1 --2- 172.21.15.42:0/911026164 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6dac0a7740 0x7f6dac08df80 secure :-1 s=THROTTLE_DONE pgs=592 cs=0 l=1 rx=0x7f6da8008790 tx=0x7f6d9c001950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:23.709 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:23.698+0000 7f6db2163700 1 --2- 172.21.15.42:0/911026164 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6dac0a7740 0x7f6dac08df80 secure :-1 s=THROTTLE_DONE pgs=592 cs=0 l=1 rx=0x7f6da8008790 tx=0x7f6d9c001950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:23.709 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:23.698+0000 7f6db1161700 1 -- 172.21.15.42:0/911026164 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d9c001560 con 0x7f6dac0a7740 2019-12-05T05:15:23.709 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:23.698+0000 7f6db1161700 1 -- 172.21.15.42:0/911026164 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6da8011360 con 0x7f6dac0a7740 2019-12-05T05:15:23.709 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:23.698+0000 7f6db3bc6700 1 -- 172.21.15.42:0/911026164 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6dac0a7740 msgr2=0x7f6dac08df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:15:23.709 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:23.698+0000 7f6db3bc6700 1 --2- 172.21.15.42:0/911026164 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6dac0a7740 0x7f6dac08df80 secure :-1 s=READY pgs=592 cs=0 l=1 rx=0x7f6da8008790 tx=0x7f6d9c001950).stop 2019-12-05T05:15:23.709 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:23.702+0000 7f6db3bc6700 1 -- 172.21.15.42:0/911026164 shutdown_connections 2019-12-05T05:15:23.709 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:23.702+0000 7f6db3bc6700 1 --2- 172.21.15.42:0/911026164 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6dac0a7740 0x7f6dac08df80 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:15:23.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:23.702+0000 7f6db3bc6700 1 -- 172.21.15.42:0/911026164 shutdown_connections 2019-12-05T05:15:23.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:23.702+0000 7f6db3bc6700 1 -- 172.21.15.42:0/911026164 wait complete. 2019-12-05T05:15:23.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:23.702+0000 7f6db3bc6700 1 -- 172.21.15.42:0/911026164 >> 172.21.15.42:0/911026164 conn(0x7f6dac0aabe0 msgr2=0x7f6dac0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:15:23.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:23.702+0000 7f6db3bc6700 1 Processor -- start 2019-12-05T05:15:23.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:23.702+0000 7f6db3bc6700 1 -- start start 2019-12-05T05:15:23.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:23.706+0000 7f6db3bc6700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6dac11ff50 0x7f6dac119350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:15:23.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:23.706+0000 7f6db3bc6700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6dac1357e0 0x7f6dac137c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:15:23.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:23.706+0000 7f6db3bc6700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6dac1198a0 con 0x7f6dac1185e0 2019-12-05T05:15:23.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:23.706+0000 7f6db3bc6700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6dac11a770 con 0x7f6dac11ff50 2019-12-05T05:15:23.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:23.706+0000 7f6db3bc6700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6dac11bfc0 con 0x7f6dac1357e0 2019-12-05T05:15:23.715 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:23.706+0000 7f6db1962700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6dac11ff50 0x7f6dac119350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:15:23.715 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:23.706+0000 7f6db1962700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6dac11ff50 0x7f6dac119350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:58860/0 (socket says 172.21.15.42:58860) 2019-12-05T05:15:23.715 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:23.706+0000 7f6db1962700 1 -- 172.21.15.42:0/3004970806 learned_addr learned my addr 172.21.15.42:0/3004970806 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:15:23.715 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:23.706+0000 7f6db2964700 1 --2- 172.21.15.42:0/3004970806 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6dac1357e0 0x7f6dac137c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:15:23.715 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:23.706+0000 7f6db1962700 1 -- 172.21.15.42:0/3004970806 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6dac1357e0 msgr2=0x7f6dac137c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:15:23.715 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:23.706+0000 7f6db1962700 1 --2- 172.21.15.42:0/3004970806 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6dac1357e0 0x7f6dac137c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:15:23.715 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:23.706+0000 7f6db1962700 1 -- 172.21.15.42:0/3004970806 >> v1:172.21.15.40:6789/0 conn(0x7f6dac1185e0 legacy=0x7f6dac1177b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:15:23.716 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:23.706+0000 7f6db1962700 1 -- 172.21.15.42:0/3004970806 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dac11c4d0 con 0x7f6dac11ff50 2019-12-05T05:15:23.716 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:23.706+0000 7f6db1962700 1 --2- 172.21.15.42:0/3004970806 >> v2:172.21.15.42:3300/0 conn(0x7f6dac11ff50 0x7f6dac119350 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f6dac119ad0 tx=0x7f6d9c010a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:23.716 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:23.706+0000 7f6db1962700 1 --2- 172.21.15.42:0/3004970806 >> v2:172.21.15.42:3300/0 conn(0x7f6dac11ff50 0x7f6dac119350 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f6dac119ad0 tx=0x7f6d9c010a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:23.716 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:23.706+0000 7f6db1962700 1 --2- 172.21.15.42:0/3004970806 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6dac11ff50 0x7f6dac119350 secure :-1 s=READY pgs=593 cs=0 l=1 rx=0x7f6dac119ad0 tx=0x7f6d9c010a90).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:15:23.716 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:23.706+0000 7f6db1962700 1 --2- 172.21.15.42:0/3004970806 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6dac11ff50 0x7f6dac119350 secure :-1 s=THROTTLE_DONE pgs=593 cs=0 l=1 rx=0x7f6dac119ad0 tx=0x7f6d9c010a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:23.716 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:23.706+0000 7f6da2ffd700 1 -- 172.21.15.42:0/3004970806 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6d9c00cb70 con 0x7f6dac11ff50 2019-12-05T05:15:23.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:23.706+0000 7f6db1962700 1 --2- 172.21.15.42:0/3004970806 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6dac11ff50 0x7f6dac119350 secure :-1 s=THROTTLE_DONE pgs=593 cs=0 l=1 rx=0x7f6dac119ad0 tx=0x7f6d9c010a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:23.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:23.706+0000 7f6db1962700 1 --2- 172.21.15.42:0/3004970806 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6dac11ff50 0x7f6dac119350 secure :-1 s=THROTTLE_DONE pgs=593 cs=0 l=1 rx=0x7f6dac119ad0 tx=0x7f6d9c010a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:23.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:23.706+0000 7f6da2ffd700 1 -- 172.21.15.42:0/3004970806 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d9c00c790 con 0x7f6dac11ff50 2019-12-05T05:15:23.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:23.706+0000 7f6db3bc6700 1 -- 172.21.15.42:0/3004970806 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6dac118dd0 con 0x7f6dac11ff50 2019-12-05T05:15:23.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:23.706+0000 7f6da2ffd700 1 -- 172.21.15.42:0/3004970806 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6d9c004620 con 0x7f6dac11ff50 2019-12-05T05:15:23.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:23.706+0000 7f6db3bc6700 1 -- 172.21.15.42:0/3004970806 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6dac07e100 con 0x7f6dac11ff50 2019-12-05T05:15:23.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:23.706+0000 7f6db1962700 1 --2- 172.21.15.42:0/3004970806 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6dac11ff50 0x7f6dac119350 secure :-1 s=THROTTLE_DONE pgs=593 cs=0 l=1 rx=0x7f6dac119ad0 tx=0x7f6d9c010a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:23.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:23.706+0000 7f6da2ffd700 1 -- 172.21.15.42:0/3004970806 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f6d9c004930 con 0x7f6dac11ff50 2019-12-05T05:15:23.718 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:23.706+0000 7f6db1962700 1 --2- 172.21.15.42:0/3004970806 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6dac11ff50 0x7f6dac119350 secure :-1 s=THROTTLE_DONE pgs=593 cs=0 l=1 rx=0x7f6dac119ad0 tx=0x7f6d9c010a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:23.718 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:23.710+0000 7f6da2ffd700 1 --2- 172.21.15.42:0/3004970806 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6d9403ae60 0x7f6d9403d370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:15:23.718 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:23.710+0000 7f6da2ffd700 1 -- 172.21.15.42:0/3004970806 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6805+0+0 (secure 0 0 0) 0x7f6d9c04ae20 con 0x7f6dac11ff50 2019-12-05T05:15:23.718 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:23.710+0000 7f6db2163700 1 --2- 172.21.15.42:0/3004970806 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6d9403ae60 0x7f6d9403d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:15:23.718 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:23.710+0000 7f6db2163700 1 --2- 172.21.15.42:0/3004970806 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6d9403ae60 0x7f6d9403d370 crc :-1 s=READY pgs=266 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:15:23.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:23.710+0000 7f6db3bc6700 1 -- 172.21.15.42:0/3004970806 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6dac11a930 con 0x7f6dac11ff50 2019-12-05T05:15:23.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:23.718+0000 7f6db1962700 1 --2- 172.21.15.42:0/3004970806 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6dac11ff50 0x7f6dac119350 secure :-1 s=THROTTLE_DONE pgs=593 cs=0 l=1 rx=0x7f6dac119ad0 tx=0x7f6d9c010a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:23.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:23.718+0000 7f6da2ffd700 1 -- 172.21.15.42:0/3004970806 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f6d9c031ea0 con 0x7f6dac11ff50 2019-12-05T05:15:23.885 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:23.882+0000 7f6db3bc6700 1 -- 172.21.15.42:0/3004970806 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6dac05d110 con 0x7f6dac11ff50 2019-12-05T05:15:23.886 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:23.882+0000 7f6db1962700 1 --2- 172.21.15.42:0/3004970806 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6dac11ff50 0x7f6dac119350 secure :-1 s=THROTTLE_DONE pgs=593 cs=0 l=1 rx=0x7f6dac119ad0 tx=0x7f6d9c010a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:23.886 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:23.882+0000 7f6da2ffd700 1 -- 172.21.15.42:0/3004970806 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) v1 ==== 94+0+1772 (secure 0 0 0) 0x7f6d9c032100 con 0x7f6dac11ff50 2019-12-05T05:15:23.886 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 77 2019-12-05T05:15:23.886 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:15:23.886 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":77,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":77,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:15:13.428215+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5629},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5629":{"gid":5629,"name":"a","rank":0,"incarnation":74,"state":"up:active","state_seq":15,"addr":"172.21.15.42:6835/3264829400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3264829400},{"type":"v1","addr":"172.21.15.42:6835","nonce":3264829400}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:15:23.889 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:23.886+0000 7f6db3bc6700 1 -- 172.21.15.42:0/3004970806 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6d9403ae60 msgr2=0x7f6d9403d370 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:15:23.889 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:23.886+0000 7f6db3bc6700 1 --2- 172.21.15.42:0/3004970806 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6d9403ae60 0x7f6d9403d370 crc :-1 s=READY pgs=266 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:15:23.889 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:23.886+0000 7f6db3bc6700 1 -- 172.21.15.42:0/3004970806 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6dac11ff50 msgr2=0x7f6dac119350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:15:23.889 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:23.886+0000 7f6db3bc6700 1 --2- 172.21.15.42:0/3004970806 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6dac11ff50 0x7f6dac119350 secure :-1 s=READY pgs=593 cs=0 l=1 rx=0x7f6dac119ad0 tx=0x7f6d9c010a90).stop 2019-12-05T05:15:23.889 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:23.886+0000 7f6db3bc6700 1 -- 172.21.15.42:0/3004970806 shutdown_connections 2019-12-05T05:15:23.889 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:23.886+0000 7f6db3bc6700 1 --2- 172.21.15.42:0/3004970806 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6d9403ae60 0x7f6d9403d370 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:15:23.890 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:23.886+0000 7f6db3bc6700 1 --2- 172.21.15.42:0/3004970806 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6dac1357e0 0x7f6dac137c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:15:23.890 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:23.886+0000 7f6db3bc6700 1 --2- 172.21.15.42:0/3004970806 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6dac11ff50 0x7f6dac119350 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:15:23.890 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:23.886+0000 7f6db3bc6700 1 -- 172.21.15.42:0/3004970806 shutdown_connections 2019-12-05T05:15:23.890 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:23.886+0000 7f6db3bc6700 1 -- 172.21.15.42:0/3004970806 wait complete. 2019-12-05T05:15:23.890 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:23.886+0000 7f6db3bc6700 1 -- 172.21.15.42:0/3004970806 >> 172.21.15.42:0/3004970806 conn(0x7f6dac07ee20 msgr2=0x7f6dac059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:15:23.899 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2019-12-05T05:15:25.900 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:15:26.007 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:25.998+0000 7fbcacceb700 1 Processor -- start 2019-12-05T05:15:26.007 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:25.998+0000 7fbcacceb700 1 -- start start 2019-12-05T05:15:26.008 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:25.998+0000 7fbcacceb700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fbca80a7730 0x7fbca808df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:15:26.008 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:25.998+0000 7fbcacceb700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbca807e710 0x7fbca807eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:15:26.008 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:25.998+0000 7fbcacceb700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbca80a0700 con 0x7fbca809f2d0 2019-12-05T05:15:26.008 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:25.998+0000 7fbcacceb700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbca80a0af0 con 0x7fbca80a7730 2019-12-05T05:15:26.009 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:25.998+0000 7fbcacceb700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbca808e5a0 con 0x7fbca807e710 2019-12-05T05:15:26.009 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:25.998+0000 7fbca759e700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fbca809f2d0 0x7fbca809f700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:58904/0 (socket says 172.21.15.42:58904) 2019-12-05T05:15:26.009 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:25.998+0000 7fbca759e700 1 -- 172.21.15.42:0/3613312023 learned_addr learned my addr 172.21.15.42:0/3613312023 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:15:26.009 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:25.998+0000 7fbc9e59c700 1 --2- 172.21.15.42:0/3613312023 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbca807e710 0x7fbca807eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:15:26.009 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:25.998+0000 7fbca6d9d700 1 --2- 172.21.15.42:0/3613312023 >> v2:172.21.15.40:3300/0 conn(0x7fbca80a7730 0x7fbca808df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:15:26.009 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:25.998+0000 7fbca659c700 1 -- 172.21.15.42:0/3613312023 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2354441162 0 0) 0x7fbca80a0700 con 0x7fbca809f2d0 2019-12-05T05:15:26.010 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:25.998+0000 7fbca659c700 1 -- 172.21.15.42:0/3613312023 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc940030b0 con 0x7fbca809f2d0 2019-12-05T05:15:26.010 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:25.998+0000 7fbc9e59c700 1 -- 172.21.15.42:0/3613312023 >> v2:172.21.15.40:3300/0 conn(0x7fbca80a7730 msgr2=0x7fbca808df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:15:26.010 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:26.002+0000 7fbc9e59c700 1 --2- 172.21.15.42:0/3613312023 >> v2:172.21.15.40:3300/0 conn(0x7fbca80a7730 0x7fbca808df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:15:26.011 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:26.002+0000 7fbc9e59c700 1 -- 172.21.15.42:0/3613312023 >> v1:172.21.15.42:6789/0 conn(0x7fbca809f2d0 legacy=0x7fbca809f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:15:26.011 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:26.002+0000 7fbc9e59c700 1 -- 172.21.15.42:0/3613312023 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbca807f090 con 0x7fbca807e710 2019-12-05T05:15:26.011 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:26.002+0000 7fbca6d9d700 1 --2- 172.21.15.42:0/3613312023 >> v2:172.21.15.40:3300/0 conn(0x7fbca80a7730 0x7fbca808df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:15:26.011 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:26.002+0000 7fbc9e59c700 1 --2- 172.21.15.42:0/3613312023 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbca807e710 0x7fbca807eb40 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fbc90008790 tx=0x7fbc9000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:26.011 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:26.002+0000 7fbc9e59c700 1 --2- 172.21.15.42:0/3613312023 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbca807e710 0x7fbca807eb40 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fbc90008790 tx=0x7fbc9000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:26.012 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:26.002+0000 7fbc9e59c700 1 --2- 172.21.15.42:0/3613312023 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbca807e710 0x7fbca807eb40 secure :-1 s=READY pgs=120 cs=0 l=1 rx=0x7fbc90008790 tx=0x7fbc9000fc70).ready entity=mon.2 client_cookie=e56f2d69366398d4 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:15:26.012 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:26.002+0000 7fbc9e59c700 1 --2- 172.21.15.42:0/3613312023 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbca807e710 0x7fbca807eb40 secure :-1 s=THROTTLE_DONE pgs=120 cs=0 l=1 rx=0x7fbc90008790 tx=0x7fbc9000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:26.012 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:26.002+0000 7fbc9e59c700 1 --2- 172.21.15.42:0/3613312023 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbca807e710 0x7fbca807eb40 secure :-1 s=THROTTLE_DONE pgs=120 cs=0 l=1 rx=0x7fbc90008790 tx=0x7fbc9000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:26.012 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:26.002+0000 7fbca659c700 1 -- 172.21.15.42:0/3613312023 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbc900112f0 con 0x7fbca807e710 2019-12-05T05:15:26.012 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:26.002+0000 7fbc9e59c700 1 --2- 172.21.15.42:0/3613312023 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbca807e710 0x7fbca807eb40 secure :-1 s=THROTTLE_DONE pgs=120 cs=0 l=1 rx=0x7fbc90008790 tx=0x7fbc9000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:26.012 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:26.002+0000 7fbca659c700 1 -- 172.21.15.42:0/3613312023 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc90008bc0 con 0x7fbca807e710 2019-12-05T05:15:26.012 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:26.002+0000 7fbca659c700 1 -- 172.21.15.42:0/3613312023 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbc900118d0 con 0x7fbca807e710 2019-12-05T05:15:26.013 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:26.002+0000 7fbcacceb700 1 -- 172.21.15.42:0/3613312023 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbca807e710 msgr2=0x7fbca807eb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:15:26.013 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:26.002+0000 7fbcacceb700 1 --2- 172.21.15.42:0/3613312023 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbca807e710 0x7fbca807eb40 secure :-1 s=READY pgs=120 cs=0 l=1 rx=0x7fbc90008790 tx=0x7fbc9000fc70).stop 2019-12-05T05:15:26.013 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:26.002+0000 7fbcacceb700 1 -- 172.21.15.42:0/3613312023 shutdown_connections 2019-12-05T05:15:26.013 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:26.002+0000 7fbcacceb700 1 --2- 172.21.15.42:0/3613312023 >> v2:172.21.15.40:3300/0 conn(0x7fbca80a7730 0x7fbca808df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:15:26.013 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:26.002+0000 7fbcacceb700 1 --2- 172.21.15.42:0/3613312023 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbca807e710 0x7fbca807eb40 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:15:26.014 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:26.002+0000 7fbcacceb700 1 -- 172.21.15.42:0/3613312023 shutdown_connections 2019-12-05T05:15:26.014 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:26.002+0000 7fbcacceb700 1 -- 172.21.15.42:0/3613312023 wait complete. 2019-12-05T05:15:26.014 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:26.002+0000 7fbcacceb700 1 -- 172.21.15.42:0/3613312023 >> 172.21.15.42:0/3613312023 conn(0x7fbca80aabd0 msgr2=0x7fbca80a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:15:26.014 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:26.002+0000 7fbcacceb700 1 Processor -- start 2019-12-05T05:15:26.014 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:26.002+0000 7fbcacceb700 1 -- start start 2019-12-05T05:15:26.018 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:26.006+0000 7fbcacceb700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbca811cf60 0x7fbca8121f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:15:26.018 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:26.006+0000 7fbcacceb700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fbca80742d0 0x7fbca8076710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:15:26.018 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:26.006+0000 7fbcacceb700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbca80a4a40 con 0x7fbca80a6bd0 2019-12-05T05:15:26.019 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:26.006+0000 7fbcacceb700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbca811d390 con 0x7fbca80742d0 2019-12-05T05:15:26.019 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:26.006+0000 7fbcacceb700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbca80a63b0 con 0x7fbca811cf60 2019-12-05T05:15:26.019 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:26.006+0000 7fbca759e700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fbca80742d0 0x7fbca8076710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:15:26.019 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:26.006+0000 7fbca759e700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fbca80742d0 0x7fbca8076710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.42:46684/0 (socket says 172.21.15.42:46684) 2019-12-05T05:15:26.019 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:26.006+0000 7fbca759e700 1 -- 172.21.15.42:0/612590613 learned_addr learned my addr 172.21.15.42:0/612590613 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:15:26.019 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:26.006+0000 7fbca6d9d700 1 --2- 172.21.15.42:0/612590613 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbca811cf60 0x7fbca8121f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:15:26.020 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:26.006+0000 7fbca759e700 1 -- 172.21.15.42:0/612590613 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbca811cf60 msgr2=0x7fbca8121f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:15:26.020 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:26.006+0000 7fbca759e700 1 --2- 172.21.15.42:0/612590613 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbca811cf60 0x7fbca8121f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:15:26.020 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:26.006+0000 7fbca759e700 1 -- 172.21.15.42:0/612590613 >> v1:172.21.15.40:6789/0 conn(0x7fbca80a6bd0 legacy=0x7fbca80a5aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:15:26.020 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:26.006+0000 7fbca759e700 1 -- 172.21.15.42:0/612590613 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbca80a6670 con 0x7fbca80742d0 2019-12-05T05:15:26.020 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:26.006+0000 7fbca759e700 1 --2- 172.21.15.42:0/612590613 >> v2:172.21.15.40:3300/0 conn(0x7fbca80742d0 0x7fbca8076710 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fbca0004720 tx=0x7fbca0010b90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:26.020 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:26.006+0000 7fbca759e700 1 --2- 172.21.15.42:0/612590613 >> v2:172.21.15.40:3300/0 conn(0x7fbca80742d0 0x7fbca8076710 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fbca0004720 tx=0x7fbca0010b90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:26.020 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:26.006+0000 7fbca759e700 1 --2- 172.21.15.42:0/612590613 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbca80742d0 0x7fbca8076710 secure :-1 s=READY pgs=470 cs=0 l=1 rx=0x7fbca0004720 tx=0x7fbca0010b90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:15:26.021 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:26.006+0000 7fbca759e700 1 --2- 172.21.15.42:0/612590613 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbca80742d0 0x7fbca8076710 secure :-1 s=THROTTLE_DONE pgs=470 cs=0 l=1 rx=0x7fbca0004720 tx=0x7fbca0010b90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:26.021 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:26.006+0000 7fbc9ffff700 1 -- 172.21.15.42:0/612590613 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbca0012250 con 0x7fbca80742d0 2019-12-05T05:15:26.021 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:26.006+0000 7fbca759e700 1 --2- 172.21.15.42:0/612590613 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbca80742d0 0x7fbca8076710 secure :-1 s=THROTTLE_DONE pgs=470 cs=0 l=1 rx=0x7fbca0004720 tx=0x7fbca0010b90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:26.021 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:26.006+0000 7fbcacceb700 1 -- 172.21.15.42:0/612590613 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbca80a7000 con 0x7fbca80742d0 2019-12-05T05:15:26.021 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:26.006+0000 7fbca759e700 1 --2- 172.21.15.42:0/612590613 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbca80742d0 0x7fbca8076710 secure :-1 s=THROTTLE_DONE pgs=470 cs=0 l=1 rx=0x7fbca0004720 tx=0x7fbca0010b90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:26.021 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:26.006+0000 7fbc9ffff700 1 -- 172.21.15.42:0/612590613 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbca0005ae0 con 0x7fbca80742d0 2019-12-05T05:15:26.022 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:26.006+0000 7fbc9ffff700 1 -- 172.21.15.42:0/612590613 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbca0012830 con 0x7fbca80742d0 2019-12-05T05:15:26.022 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:26.006+0000 7fbcacceb700 1 -- 172.21.15.42:0/612590613 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbca8076c60 con 0x7fbca80742d0 2019-12-05T05:15:26.022 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:26.006+0000 7fbca759e700 1 --2- 172.21.15.42:0/612590613 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbca80742d0 0x7fbca8076710 secure :-1 s=THROTTLE_DONE pgs=470 cs=0 l=1 rx=0x7fbca0004720 tx=0x7fbca0010b90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:26.022 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:26.010+0000 7fbca759e700 1 --2- 172.21.15.42:0/612590613 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbca80742d0 0x7fbca8076710 secure :-1 s=THROTTLE_DONE pgs=470 cs=0 l=1 rx=0x7fbca0004720 tx=0x7fbca0010b90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:26.022 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:26.010+0000 7fbc9ffff700 1 -- 172.21.15.42:0/612590613 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fbca0012b50 con 0x7fbca80742d0 2019-12-05T05:15:26.022 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:26.010+0000 7fbc9ffff700 1 --2- 172.21.15.42:0/612590613 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbc8c03ad20 0x7fbc8c03d230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:15:26.022 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:26.010+0000 7fbc9ffff700 1 -- 172.21.15.42:0/612590613 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6805+0+0 (secure 0 0 0) 0x7fbca004b640 con 0x7fbca80742d0 2019-12-05T05:15:26.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:26.010+0000 7fbca6d9d700 1 --2- 172.21.15.42:0/612590613 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbc8c03ad20 0x7fbc8c03d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:15:26.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:26.010+0000 7fbca6d9d700 1 --2- 172.21.15.42:0/612590613 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbc8c03ad20 0x7fbc8c03d230 crc :-1 s=READY pgs=268 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:15:26.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:26.014+0000 7fbcacceb700 1 -- 172.21.15.42:0/612590613 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbca805d100 con 0x7fbca80742d0 2019-12-05T05:15:26.028 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:26.022+0000 7fbca759e700 1 --2- 172.21.15.42:0/612590613 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbca80742d0 0x7fbca8076710 secure :-1 s=THROTTLE_DONE pgs=470 cs=0 l=1 rx=0x7fbca0004720 tx=0x7fbca0010b90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:26.028 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:26.022+0000 7fbc9ffff700 1 -- 172.21.15.42:0/612590613 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fbca80a6670 con 0x7fbca80742d0 2019-12-05T05:15:26.195 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:26.190+0000 7fbcacceb700 1 -- 172.21.15.42:0/612590613 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbca805b8b0 con 0x7fbca80742d0 2019-12-05T05:15:26.196 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:26.194+0000 7fbca759e700 1 --2- 172.21.15.42:0/612590613 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbca80742d0 0x7fbca8076710 secure :-1 s=THROTTLE_DONE pgs=470 cs=0 l=1 rx=0x7fbca0004720 tx=0x7fbca0010b90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:15:26.196 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:26.194+0000 7fbc9ffff700 1 -- 172.21.15.42:0/612590613 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 78 v78) v1 ==== 94+0+2108 (secure 0 0 0) 0x7fbca0032600 con 0x7fbca80742d0 2019-12-05T05:15:26.198 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:15:26.199 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":78,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5844,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/2698421979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2698421979},{"type":"v1","addr":"172.21.15.40:6833","nonce":2698421979}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":78}],"filesystems":[{"mdsmap":{"epoch":77,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:15:13.428215+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5629},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5629":{"gid":5629,"name":"a","rank":0,"incarnation":74,"state":"up:active","state_seq":15,"addr":"172.21.15.42:6835/3264829400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3264829400},{"type":"v1","addr":"172.21.15.42:6835","nonce":3264829400}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:15:26.199 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 78 2019-12-05T05:15:26.201 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:26.198+0000 7fbcacceb700 1 -- 172.21.15.42:0/612590613 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbc8c03ad20 msgr2=0x7fbc8c03d230 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:15:26.201 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:26.198+0000 7fbcacceb700 1 --2- 172.21.15.42:0/612590613 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbc8c03ad20 0x7fbc8c03d230 crc :-1 s=READY pgs=268 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:15:26.201 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:26.198+0000 7fbcacceb700 1 -- 172.21.15.42:0/612590613 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbca80742d0 msgr2=0x7fbca8076710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:15:26.201 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:26.198+0000 7fbcacceb700 1 --2- 172.21.15.42:0/612590613 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbca80742d0 0x7fbca8076710 secure :-1 s=READY pgs=470 cs=0 l=1 rx=0x7fbca0004720 tx=0x7fbca0010b90).stop 2019-12-05T05:15:26.202 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:26.198+0000 7fbcacceb700 1 -- 172.21.15.42:0/612590613 shutdown_connections 2019-12-05T05:15:26.202 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:26.198+0000 7fbcacceb700 1 --2- 172.21.15.42:0/612590613 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbc8c03ad20 0x7fbc8c03d230 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:15:26.202 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:26.198+0000 7fbcacceb700 1 --2- 172.21.15.42:0/612590613 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbca80742d0 0x7fbca8076710 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:15:26.202 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:26.198+0000 7fbcacceb700 1 --2- 172.21.15.42:0/612590613 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbca811cf60 0x7fbca8121f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:15:26.202 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:26.198+0000 7fbcacceb700 1 -- 172.21.15.42:0/612590613 shutdown_connections 2019-12-05T05:15:26.202 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:26.198+0000 7fbcacceb700 1 -- 172.21.15.42:0/612590613 wait complete. 2019-12-05T05:15:26.203 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:15:26.198+0000 7fbcacceb700 1 -- 172.21.15.42:0/612590613 >> 172.21.15.42:0/612590613 conn(0x7fbca807e620 msgr2=0x7fbca811d6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:15:26.211 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b reported in up:standby state 2019-12-05T05:15:26.211 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 93 secs before thrashing 2019-12-05T05:15:35.648 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:15:35.656 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:16:05.703 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:16:05.707 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:16:35.743 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:16:35.747 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:16:59.212 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:16:59.323 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.305+0000 7fb1088e5700 1 Processor -- start 2019-12-05T05:16:59.324 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.305+0000 7fb1088e5700 1 -- start start 2019-12-05T05:16:59.324 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.305+0000 7fb1088e5700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb1040a7730 0x7fb10408df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:16:59.324 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.305+0000 7fb1088e5700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb10409fad0 0x7fb10407eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:16:59.324 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.305+0000 7fb1088e5700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1040a0700 con 0x7fb10407e710 2019-12-05T05:16:59.325 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.305+0000 7fb1088e5700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1040a0af0 con 0x7fb1040a7730 2019-12-05T05:16:59.325 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.305+0000 7fb1088e5700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb10408e5a0 con 0x7fb10409fad0 2019-12-05T05:16:59.325 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.305+0000 7fb10259c700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fb10407e710 0x7fb10409f2d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:58954/0 (socket says 172.21.15.42:58954) 2019-12-05T05:16:59.325 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.305+0000 7fb10259c700 1 -- 172.21.15.42:0/438697250 learned_addr learned my addr 172.21.15.42:0/438697250 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:16:59.325 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.305+0000 7fb102d9d700 1 --2- 172.21.15.42:0/438697250 >> v2:172.21.15.40:3300/0 conn(0x7fb1040a7730 0x7fb10408df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:16:59.326 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.305+0000 7fb10359e700 1 --2- 172.21.15.42:0/438697250 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb10409fad0 0x7fb10407eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:16:59.326 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.305+0000 7fb101d9b700 1 -- 172.21.15.42:0/438697250 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1894204454 0 0) 0x7fb1040a0700 con 0x7fb10407e710 2019-12-05T05:16:59.326 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.305+0000 7fb101d9b700 1 -- 172.21.15.42:0/438697250 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0f00030b0 con 0x7fb10407e710 2019-12-05T05:16:59.326 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.305+0000 7fb10359e700 1 -- 172.21.15.42:0/438697250 >> v2:172.21.15.40:3300/0 conn(0x7fb1040a7730 msgr2=0x7fb10408df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:16:59.327 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.305+0000 7fb10359e700 1 --2- 172.21.15.42:0/438697250 >> v2:172.21.15.40:3300/0 conn(0x7fb1040a7730 0x7fb10408df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:16:59.327 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.305+0000 7fb10359e700 1 -- 172.21.15.42:0/438697250 >> v1:172.21.15.42:6789/0 conn(0x7fb10407e710 legacy=0x7fb10409f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:16:59.327 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.305+0000 7fb10359e700 1 -- 172.21.15.42:0/438697250 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb10407f090 con 0x7fb10409fad0 2019-12-05T05:16:59.327 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.305+0000 7fb102d9d700 1 --2- 172.21.15.42:0/438697250 >> v2:172.21.15.40:3300/0 conn(0x7fb1040a7730 0x7fb10408df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:16:59.327 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.305+0000 7fb10359e700 1 --2- 172.21.15.42:0/438697250 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb10409fad0 0x7fb10407eb40 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fb0fc008790 tx=0x7fb0fc00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:16:59.328 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.305+0000 7fb10359e700 1 --2- 172.21.15.42:0/438697250 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb10409fad0 0x7fb10407eb40 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fb0fc008790 tx=0x7fb0fc00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:16:59.328 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.305+0000 7fb10359e700 1 --2- 172.21.15.42:0/438697250 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb10409fad0 0x7fb10407eb40 secure :-1 s=READY pgs=121 cs=0 l=1 rx=0x7fb0fc008790 tx=0x7fb0fc00fc70).ready entity=mon.2 client_cookie=3a43fd84a2631683 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:16:59.328 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.305+0000 7fb10359e700 1 -- 172.21.15.42:0/438697250 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb10409fad0 msgr2=0x7fb10407eb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2019-12-05T05:16:59.329 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.305+0000 7fb10359e700 1 -- 172.21.15.42:0/438697250 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb10409fad0 msgr2=0x7fb10407eb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2019-12-05T05:16:59.329 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.305+0000 7fb10359e700 1 --2- 172.21.15.42:0/438697250 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb10409fad0 0x7fb10407eb40 secure :-1 s=READY pgs=121 cs=0 l=1 rx=0x7fb0fc008790 tx=0x7fb0fc00fc70).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-12-05T05:16:59.329 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.305+0000 7fb10359e700 1 --2- 172.21.15.42:0/438697250 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb10409fad0 0x7fb10407eb40 secure :-1 s=READY pgs=121 cs=0 l=1 rx=0x7fb0fc008790 tx=0x7fb0fc00fc70).stop 2019-12-05T05:16:59.329 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.305+0000 7fb101d9b700 1 -- 172.21.15.42:0/438697250 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb10409fad0 msgr2=0x7fb10407eb40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2019-12-05T05:16:59.329 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.305+0000 7fb101d9b700 1 --2- 172.21.15.42:0/438697250 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb10409fad0 0x7fb10407eb40 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:16:59.330 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.305+0000 7fb101d9b700 1 --2- 172.21.15.42:0/438697250 >> v2:172.21.15.42:3300/0 conn(0x7fb0f004b660 0x7fb0f004daa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:16:59.330 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.305+0000 7fb101d9b700 1 -- 172.21.15.42:0/438697250 --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1040a0700 con 0x7fb0f0003420 2019-12-05T05:16:59.330 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.305+0000 7fb101d9b700 1 -- 172.21.15.42:0/438697250 --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0f004f070 con 0x7fb0f004b660 2019-12-05T05:16:59.330 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.309+0000 7fb10259c700 1 --2- 172.21.15.42:0/438697250 >> v2:172.21.15.42:3300/0 conn(0x7fb0f004b660 0x7fb0f004daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:16:59.330 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.309+0000 7fb10259c700 1 -- 172.21.15.42:0/438697250 >> v1:172.21.15.40:6789/0 conn(0x7fb0f0003420 legacy=0x7fb0f0003880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:16:59.331 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.309+0000 7fb10259c700 1 -- 172.21.15.42:0/438697250 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0f004f2f0 con 0x7fb0f004b660 2019-12-05T05:16:59.331 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.309+0000 7fb10259c700 1 --2- 172.21.15.42:0/438697250 >> v2:172.21.15.42:3300/0 conn(0x7fb0f004b660 0x7fb0f004daa0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fb0f4001000 tx=0x7fb0f400bed0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:16:59.331 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.309+0000 7fb10259c700 1 --2- 172.21.15.42:0/438697250 >> v2:172.21.15.42:3300/0 conn(0x7fb0f004b660 0x7fb0f004daa0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fb0f4001000 tx=0x7fb0f400bed0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:16:59.331 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.309+0000 7fb10259c700 1 --2- 172.21.15.42:0/438697250 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb0f004b660 0x7fb0f004daa0 secure :-1 s=READY pgs=599 cs=0 l=1 rx=0x7fb0f4001000 tx=0x7fb0f400bed0).ready entity=mon.1 client_cookie=41ce08a173fa2ba2 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:16:59.331 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.309+0000 7fb10259c700 1 --2- 172.21.15.42:0/438697250 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb0f004b660 0x7fb0f004daa0 secure :-1 s=THROTTLE_DONE pgs=599 cs=0 l=1 rx=0x7fb0f4001000 tx=0x7fb0f400bed0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:16:59.332 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.309+0000 7fb101d9b700 1 -- 172.21.15.42:0/438697250 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb0f4010990 con 0x7fb0f004b660 2019-12-05T05:16:59.332 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.309+0000 7fb10259c700 1 --2- 172.21.15.42:0/438697250 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb0f004b660 0x7fb0f004daa0 secure :-1 s=THROTTLE_DONE pgs=599 cs=0 l=1 rx=0x7fb0f4001000 tx=0x7fb0f400bed0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:16:59.332 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.309+0000 7fb10259c700 1 --2- 172.21.15.42:0/438697250 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb0f004b660 0x7fb0f004daa0 secure :-1 s=THROTTLE_DONE pgs=599 cs=0 l=1 rx=0x7fb0f4001000 tx=0x7fb0f400bed0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:16:59.332 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.309+0000 7fb101d9b700 1 -- 172.21.15.42:0/438697250 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0f4010170 con 0x7fb0f004b660 2019-12-05T05:16:59.332 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.309+0000 7fb101d9b700 1 -- 172.21.15.42:0/438697250 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb0f400c680 con 0x7fb0f004b660 2019-12-05T05:16:59.333 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.309+0000 7fb1088e5700 1 -- 172.21.15.42:0/438697250 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb0f004b660 msgr2=0x7fb0f004daa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:16:59.333 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.309+0000 7fb1088e5700 1 --2- 172.21.15.42:0/438697250 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb0f004b660 0x7fb0f004daa0 secure :-1 s=READY pgs=599 cs=0 l=1 rx=0x7fb0f4001000 tx=0x7fb0f400bed0).stop 2019-12-05T05:16:59.333 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.309+0000 7fb10359e700 1 -- 172.21.15.42:0/438697250 reap_dead start 2019-12-05T05:16:59.333 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.309+0000 7fb1088e5700 1 -- 172.21.15.42:0/438697250 shutdown_connections 2019-12-05T05:16:59.333 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.309+0000 7fb1088e5700 1 --2- 172.21.15.42:0/438697250 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb0f004b660 0x7fb0f004daa0 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:16:59.334 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.309+0000 7fb1088e5700 1 -- 172.21.15.42:0/438697250 shutdown_connections 2019-12-05T05:16:59.334 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.309+0000 7fb1088e5700 1 -- 172.21.15.42:0/438697250 wait complete. 2019-12-05T05:16:59.334 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.309+0000 7fb1088e5700 1 -- 172.21.15.42:0/438697250 >> 172.21.15.42:0/438697250 conn(0x7fb1040aabd0 msgr2=0x7fb1040a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:16:59.337 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.309+0000 7fb1088e5700 1 Processor -- start 2019-12-05T05:16:59.337 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.309+0000 7fb1088e5700 1 -- start start 2019-12-05T05:16:59.337 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.313+0000 7fb1088e5700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb10411d380 0x7fb1040a5560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:16:59.337 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.313+0000 7fb1088e5700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb104142da0 0x7fb104134ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:16:59.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.313+0000 7fb1088e5700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb10411b0f0 con 0x7fb1041350f0 2019-12-05T05:16:59.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.313+0000 7fb1088e5700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb104143880 con 0x7fb104142da0 2019-12-05T05:16:59.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.313+0000 7fb1088e5700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb104143340 con 0x7fb10411d380 2019-12-05T05:16:59.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.313+0000 7fb10359e700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fb1041350f0 0x7fb104117e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:58968/0 (socket says 172.21.15.42:58968) 2019-12-05T05:16:59.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.313+0000 7fb10359e700 1 -- 172.21.15.42:0/1429169864 learned_addr learned my addr 172.21.15.42:0/1429169864 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:16:59.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.313+0000 7fb10259c700 1 --2- 172.21.15.42:0/1429169864 >> v2:172.21.15.40:3300/0 conn(0x7fb104142da0 0x7fb104134ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:16:59.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.313+0000 7fb102d9d700 1 --2- 172.21.15.42:0/1429169864 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb10411d380 0x7fb1040a5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:16:59.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.313+0000 7fb0fb7fe700 1 -- 172.21.15.42:0/1429169864 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3192511817 0 0) 0x7fb10411b0f0 con 0x7fb1041350f0 2019-12-05T05:16:59.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.313+0000 7fb0fb7fe700 1 -- 172.21.15.42:0/1429169864 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0e80030b0 con 0x7fb1041350f0 2019-12-05T05:16:59.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.313+0000 7fb10259c700 1 -- 172.21.15.42:0/1429169864 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb10411d380 msgr2=0x7fb1040a5560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:16:59.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.313+0000 7fb10259c700 1 --2- 172.21.15.42:0/1429169864 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb10411d380 0x7fb1040a5560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:16:59.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.313+0000 7fb10259c700 1 -- 172.21.15.42:0/1429169864 >> v1:172.21.15.42:6789/0 conn(0x7fb1041350f0 legacy=0x7fb104117e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:16:59.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.313+0000 7fb0fb7fe700 1 -- 172.21.15.42:0/1429169864 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 245910656 0 0) 0x7fb0e80030b0 con 0x7fb1041350f0 2019-12-05T05:16:59.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.313+0000 7fb10259c700 1 -- 172.21.15.42:0/1429169864 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb104135520 con 0x7fb104142da0 2019-12-05T05:16:59.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.313+0000 7fb102d9d700 1 --2- 172.21.15.42:0/1429169864 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb10411d380 0x7fb1040a5560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:16:59.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.313+0000 7fb10259c700 1 --2- 172.21.15.42:0/1429169864 >> v2:172.21.15.40:3300/0 conn(0x7fb104142da0 0x7fb104134ba0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fb0fc008de0 tx=0x7fb0fc0082d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:16:59.341 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.313+0000 7fb10259c700 1 --2- 172.21.15.42:0/1429169864 >> v2:172.21.15.40:3300/0 conn(0x7fb104142da0 0x7fb104134ba0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fb0fc008de0 tx=0x7fb0fc0082d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:16:59.341 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.313+0000 7fb10259c700 1 --2- 172.21.15.42:0/1429169864 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb104142da0 0x7fb104134ba0 secure :-1 s=READY pgs=474 cs=0 l=1 rx=0x7fb0fc008de0 tx=0x7fb0fc0082d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:16:59.341 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.313+0000 7fb10259c700 1 --2- 172.21.15.42:0/1429169864 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb104142da0 0x7fb104134ba0 secure :-1 s=THROTTLE_DONE pgs=474 cs=0 l=1 rx=0x7fb0fc008de0 tx=0x7fb0fc0082d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:16:59.341 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.313+0000 7fb0fb7fe700 1 -- 172.21.15.42:0/1429169864 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb0fc008be0 con 0x7fb104142da0 2019-12-05T05:16:59.341 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.313+0000 7fb10259c700 1 --2- 172.21.15.42:0/1429169864 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb104142da0 0x7fb104134ba0 secure :-1 s=THROTTLE_DONE pgs=474 cs=0 l=1 rx=0x7fb0fc008de0 tx=0x7fb0fc0082d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:16:59.341 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.313+0000 7fb10259c700 1 --2- 172.21.15.42:0/1429169864 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb104142da0 0x7fb104134ba0 secure :-1 s=THROTTLE_DONE pgs=474 cs=0 l=1 rx=0x7fb0fc008de0 tx=0x7fb0fc0082d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:16:59.341 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.313+0000 7fb0fb7fe700 1 -- 172.21.15.42:0/1429169864 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0fc008490 con 0x7fb104142da0 2019-12-05T05:16:59.342 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.313+0000 7fb1088e5700 1 -- 172.21.15.42:0/1429169864 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb104118690 con 0x7fb104142da0 2019-12-05T05:16:59.342 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.313+0000 7fb0fb7fe700 1 -- 172.21.15.42:0/1429169864 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb0f4004370 con 0x7fb104142da0 2019-12-05T05:16:59.342 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.313+0000 7fb1088e5700 1 -- 172.21.15.42:0/1429169864 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb104118c40 con 0x7fb104142da0 2019-12-05T05:16:59.342 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.317+0000 7fb10259c700 1 --2- 172.21.15.42:0/1429169864 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb104142da0 0x7fb104134ba0 secure :-1 s=THROTTLE_DONE pgs=474 cs=0 l=1 rx=0x7fb0fc008de0 tx=0x7fb0fc0082d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:16:59.342 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.317+0000 7fb0fb7fe700 1 -- 172.21.15.42:0/1429169864 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fb0f4004570 con 0x7fb104142da0 2019-12-05T05:16:59.342 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.317+0000 7fb10259c700 1 --2- 172.21.15.42:0/1429169864 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb104142da0 0x7fb104134ba0 secure :-1 s=THROTTLE_DONE pgs=474 cs=0 l=1 rx=0x7fb0fc008de0 tx=0x7fb0fc0082d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:16:59.342 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.317+0000 7fb0fb7fe700 1 --2- 172.21.15.42:0/1429169864 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb0e803b2d0 0x7fb0e803d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:16:59.343 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.317+0000 7fb0fb7fe700 1 -- 172.21.15.42:0/1429169864 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6805+0+0 (secure 0 0 0) 0x7fb0f404abb0 con 0x7fb104142da0 2019-12-05T05:16:59.343 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.317+0000 7fb102d9d700 1 --2- 172.21.15.42:0/1429169864 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb0e803b2d0 0x7fb0e803d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:16:59.343 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.317+0000 7fb102d9d700 1 --2- 172.21.15.42:0/1429169864 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb0e803b2d0 0x7fb0e803d7e0 crc :-1 s=READY pgs=269 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:16:59.344 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.321+0000 7fb1088e5700 1 -- 172.21.15.42:0/1429169864 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb10405d100 con 0x7fb104142da0 2019-12-05T05:16:59.344 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.333+0000 7fb10259c700 1 --2- 172.21.15.42:0/1429169864 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb104142da0 0x7fb104134ba0 secure :-1 s=THROTTLE_DONE pgs=474 cs=0 l=1 rx=0x7fb0fc008de0 tx=0x7fb0fc0082d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:16:59.344 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.333+0000 7fb0fb7fe700 1 -- 172.21.15.42:0/1429169864 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fb0f4031c30 con 0x7fb104142da0 2019-12-05T05:16:59.507 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.501+0000 7fb1088e5700 1 -- 172.21.15.42:0/1429169864 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb10405b8b0 con 0x7fb104142da0 2019-12-05T05:16:59.508 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.505+0000 7fb10259c700 1 --2- 172.21.15.42:0/1429169864 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb104142da0 0x7fb104134ba0 secure :-1 s=THROTTLE_DONE pgs=474 cs=0 l=1 rx=0x7fb0fc008de0 tx=0x7fb0fc0082d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:16:59.508 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.505+0000 7fb0fb7fe700 1 -- 172.21.15.42:0/1429169864 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 78 v78) v1 ==== 94+0+2108 (secure 0 0 0) 0x7fb0f4031e90 con 0x7fb104142da0 2019-12-05T05:16:59.510 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:16:59.510 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":78,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5844,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/2698421979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2698421979},{"type":"v1","addr":"172.21.15.40:6833","nonce":2698421979}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":78}],"filesystems":[{"mdsmap":{"epoch":77,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:15:13.428215+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5629},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5629":{"gid":5629,"name":"a","rank":0,"incarnation":74,"state":"up:active","state_seq":15,"addr":"172.21.15.42:6835/3264829400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3264829400},{"type":"v1","addr":"172.21.15.42:6835","nonce":3264829400}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:16:59.510 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 78 2019-12-05T05:16:59.512 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.509+0000 7fb1088e5700 1 -- 172.21.15.42:0/1429169864 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb0e803b2d0 msgr2=0x7fb0e803d7e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:16:59.513 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.509+0000 7fb1088e5700 1 --2- 172.21.15.42:0/1429169864 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb0e803b2d0 0x7fb0e803d7e0 crc :-1 s=READY pgs=269 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:16:59.513 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.509+0000 7fb1088e5700 1 -- 172.21.15.42:0/1429169864 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb104142da0 msgr2=0x7fb104134ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:16:59.513 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.509+0000 7fb1088e5700 1 --2- 172.21.15.42:0/1429169864 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb104142da0 0x7fb104134ba0 secure :-1 s=READY pgs=474 cs=0 l=1 rx=0x7fb0fc008de0 tx=0x7fb0fc0082d0).stop 2019-12-05T05:16:59.513 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.509+0000 7fb1088e5700 1 -- 172.21.15.42:0/1429169864 shutdown_connections 2019-12-05T05:16:59.513 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.509+0000 7fb1088e5700 1 --2- 172.21.15.42:0/1429169864 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb0e803b2d0 0x7fb0e803d7e0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:16:59.514 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.509+0000 7fb1088e5700 1 --2- 172.21.15.42:0/1429169864 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb10411d380 0x7fb1040a5560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:16:59.514 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.509+0000 7fb1088e5700 1 --2- 172.21.15.42:0/1429169864 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb104142da0 0x7fb104134ba0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:16:59.514 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.509+0000 7fb1088e5700 1 -- 172.21.15.42:0/1429169864 shutdown_connections 2019-12-05T05:16:59.515 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.509+0000 7fb1088e5700 1 -- 172.21.15.42:0/1429169864 wait complete. 2019-12-05T05:16:59.515 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.509+0000 7fb1088e5700 1 -- 172.21.15.42:0/1429169864 >> 172.21.15.42:0/1429169864 conn(0x7fb10411b8a0 msgr2=0x7fb10408ece0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:16:59.523 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.a (rank=0) 2019-12-05T05:16:59.523 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2019-12-05T05:16:59.523 INFO:teuthology.orchestra.run:waiting for 300 2019-12-05T05:16:59.532 INFO:tasks.ceph.mds.a:Stopped 2019-12-05T05:16:59.532 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:16:59.711 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.701+0000 7fd9552a3700 1 Processor -- start 2019-12-05T05:16:59.711 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.701+0000 7fd9552a3700 1 -- start start 2019-12-05T05:16:59.712 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.701+0000 7fd9552a3700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd95009fad0 0x7fd95014ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:16:59.712 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.701+0000 7fd9552a3700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9500a0700 con 0x7fd95008df70 2019-12-05T05:16:59.712 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.701+0000 7fd9552a3700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd95007ef10 con 0x7fd9500a7730 2019-12-05T05:16:59.712 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.701+0000 7fd9552a3700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9500a0af0 con 0x7fd95009fad0 2019-12-05T05:16:59.712 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.701+0000 7fd94f7fe700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fd9500a7730 0x7fd95007e710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:58974/0 (socket says 172.21.15.42:58974) 2019-12-05T05:16:59.715 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.701+0000 7fd94f7fe700 1 -- 172.21.15.42:0/1204615091 learned_addr learned my addr 172.21.15.42:0/1204615091 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:16:59.715 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.701+0000 7fd94ffff700 1 --2- 172.21.15.42:0/1204615091 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd95009fad0 0x7fd95014ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:16:59.715 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.701+0000 7fd94e7fc700 1 -- 172.21.15.42:0/1204615091 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3146651950 0 0) 0x7fd95007ef10 con 0x7fd9500a7730 2019-12-05T05:16:59.716 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.701+0000 7fd94e7fc700 1 -- 172.21.15.42:0/1204615091 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd93c0030b0 con 0x7fd9500a7730 2019-12-05T05:16:59.716 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.701+0000 7fd94e7fc700 1 -- 172.21.15.42:0/1204615091 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2634965606 0 0) 0x7fd9500a0700 con 0x7fd95008df70 2019-12-05T05:16:59.716 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.701+0000 7fd94e7fc700 1 -- 172.21.15.42:0/1204615091 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd95007ef10 con 0x7fd95008df70 2019-12-05T05:16:59.716 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.701+0000 7fd94e7fc700 1 -- 172.21.15.42:0/1204615091 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3889366289 0 0) 0x7fd93c0030b0 con 0x7fd9500a7730 2019-12-05T05:16:59.716 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.701+0000 7fd94e7fc700 1 -- 172.21.15.42:0/1204615091 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd95009fad0 msgr2=0x7fd95014ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:16:59.716 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.701+0000 7fd94e7fc700 1 --2- 172.21.15.42:0/1204615091 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd95009fad0 0x7fd95014ffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:16:59.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.701+0000 7fd94e7fc700 1 -- 172.21.15.42:0/1204615091 >> v1:172.21.15.40:6789/0 conn(0x7fd95008df70 legacy=0x7fd95009f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:16:59.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.701+0000 7fd94e7fc700 1 -- 172.21.15.42:0/1204615091 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9500a9b90 con 0x7fd9500a7730 2019-12-05T05:16:59.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.701+0000 7fd94e7fc700 1 -- 172.21.15.42:0/1204615091 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7fd948001750 con 0x7fd9500a7730 2019-12-05T05:16:59.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.701+0000 7fd94e7fc700 1 -- 172.21.15.42:0/1204615091 >> v1:172.21.15.42:6789/0 conn(0x7fd9500a7730 legacy=0x7fd95007e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:16:59.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.701+0000 7fd94e7fc700 1 --2- 172.21.15.42:0/1204615091 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd93c003940 0x7fd93c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:16:59.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.701+0000 7fd94e7fc700 1 --2- 172.21.15.42:0/1204615091 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd93c004420 0x7fd93c004870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:16:59.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.701+0000 7fd94e7fc700 1 --2- 172.21.15.42:0/1204615091 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd93c004ed0 0x7fd93c04f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:16:59.718 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.701+0000 7fd94e7fc700 1 -- 172.21.15.42:0/1204615091 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9500a0af0 con 0x7fd93c004ed0 2019-12-05T05:16:59.718 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.701+0000 7fd94e7fc700 1 -- 172.21.15.42:0/1204615091 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd93c04fdb0 con 0x7fd93c004420 2019-12-05T05:16:59.718 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.701+0000 7fd94e7fc700 1 -- 172.21.15.42:0/1204615091 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd93c04fff0 con 0x7fd93c003940 2019-12-05T05:16:59.718 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.701+0000 7fd94effd700 1 --2- 172.21.15.42:0/1204615091 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd93c004420 0x7fd93c004870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:16:59.718 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.705+0000 7fd94ffff700 1 --2- 172.21.15.42:0/1204615091 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd95009fad0 0x7fd95014ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:16:59.718 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.705+0000 7fd94f7fe700 1 --2- 172.21.15.42:0/1204615091 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd93c003940 0x7fd93c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:16:59.718 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.705+0000 7fd94effd700 1 -- 172.21.15.42:0/1204615091 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd93c003940 msgr2=0x7fd93c003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:16:59.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.705+0000 7fd94ffff700 1 --2- 172.21.15.42:0/1204615091 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd93c004ed0 0x7fd93c04f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:16:59.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.705+0000 7fd94effd700 1 --2- 172.21.15.42:0/1204615091 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd93c003940 0x7fd93c003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:16:59.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.705+0000 7fd94effd700 1 -- 172.21.15.42:0/1204615091 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd93c004ed0 msgr2=0x7fd93c04f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:16:59.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.705+0000 7fd94effd700 1 --2- 172.21.15.42:0/1204615091 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd93c004ed0 0x7fd93c04f680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:16:59.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.705+0000 7fd94effd700 1 -- 172.21.15.42:0/1204615091 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd93c0502f0 con 0x7fd93c004420 2019-12-05T05:16:59.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.705+0000 7fd94ffff700 1 --2- 172.21.15.42:0/1204615091 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd93c004ed0 0x7fd93c04f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).send_auth_request state changed! 2019-12-05T05:16:59.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.705+0000 7fd94effd700 1 --2- 172.21.15.42:0/1204615091 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd93c004420 0x7fd93c004870 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fd938001000 tx=0x7fd93800be70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:16:59.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.705+0000 7fd94effd700 1 --2- 172.21.15.42:0/1204615091 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd93c004420 0x7fd93c004870 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fd938001000 tx=0x7fd93800be70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:16:59.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.705+0000 7fd94effd700 1 --2- 172.21.15.42:0/1204615091 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd93c004420 0x7fd93c004870 secure :-1 s=READY pgs=602 cs=0 l=1 rx=0x7fd938001000 tx=0x7fd93800be70).ready entity=mon.1 client_cookie=b39c19654c210d44 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:16:59.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.705+0000 7fd94effd700 1 --2- 172.21.15.42:0/1204615091 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd93c004420 0x7fd93c004870 secure :-1 s=THROTTLE_DONE pgs=602 cs=0 l=1 rx=0x7fd938001000 tx=0x7fd93800be70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:16:59.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.705+0000 7fd94e7fc700 1 -- 172.21.15.42:0/1204615091 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd938010be0 con 0x7fd93c004420 2019-12-05T05:16:59.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.705+0000 7fd94effd700 1 --2- 172.21.15.42:0/1204615091 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd93c004420 0x7fd93c004870 secure :-1 s=THROTTLE_DONE pgs=602 cs=0 l=1 rx=0x7fd938001000 tx=0x7fd93800be70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:16:59.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.705+0000 7fd94effd700 1 --2- 172.21.15.42:0/1204615091 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd93c004420 0x7fd93c004870 secure :-1 s=THROTTLE_DONE pgs=602 cs=0 l=1 rx=0x7fd938001000 tx=0x7fd93800be70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:16:59.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.705+0000 7fd94e7fc700 1 -- 172.21.15.42:0/1204615091 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd938010800 con 0x7fd93c004420 2019-12-05T05:16:59.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.705+0000 7fd94e7fc700 1 -- 172.21.15.42:0/1204615091 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd93800c5b0 con 0x7fd93c004420 2019-12-05T05:16:59.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.705+0000 7fd9552a3700 1 -- 172.21.15.42:0/1204615091 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd93c004420 msgr2=0x7fd93c004870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:16:59.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.705+0000 7fd9552a3700 1 --2- 172.21.15.42:0/1204615091 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd93c004420 0x7fd93c004870 secure :-1 s=READY pgs=602 cs=0 l=1 rx=0x7fd938001000 tx=0x7fd93800be70).stop 2019-12-05T05:16:59.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.705+0000 7fd94ffff700 1 -- 172.21.15.42:0/1204615091 reap_dead start 2019-12-05T05:16:59.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.705+0000 7fd9552a3700 1 -- 172.21.15.42:0/1204615091 shutdown_connections 2019-12-05T05:16:59.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.705+0000 7fd9552a3700 1 -- 172.21.15.42:0/1204615091 shutdown_connections 2019-12-05T05:16:59.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.705+0000 7fd9552a3700 1 -- 172.21.15.42:0/1204615091 wait complete. 2019-12-05T05:16:59.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.705+0000 7fd9552a3700 1 -- 172.21.15.42:0/1204615091 >> 172.21.15.42:0/1204615091 conn(0x7fd9500aabd0 msgr2=0x7fd9500a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:16:59.726 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.709+0000 7fd9552a3700 1 Processor -- start 2019-12-05T05:16:59.726 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.709+0000 7fd9552a3700 1 -- start start 2019-12-05T05:16:59.726 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.709+0000 7fd9552a3700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd9501197c0 0x7fd95011a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:16:59.726 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.709+0000 7fd9552a3700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd950078180 0x7fd95011bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:16:59.726 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.709+0000 7fd9552a3700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd95012d560 con 0x7fd95011b7a0 2019-12-05T05:16:59.726 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.709+0000 7fd9552a3700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd95012d2c0 con 0x7fd950078180 2019-12-05T05:16:59.727 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.709+0000 7fd9552a3700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd95012d790 con 0x7fd9501197c0 2019-12-05T05:16:59.727 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.709+0000 7fd94ffff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd950078180 0x7fd95011bbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:16:59.727 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.709+0000 7fd94ffff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd950078180 0x7fd95011bbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:58950/0 (socket says 172.21.15.42:58950) 2019-12-05T05:16:59.727 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.709+0000 7fd94ffff700 1 -- 172.21.15.42:0/810846366 learned_addr learned my addr 172.21.15.42:0/810846366 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:16:59.728 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.709+0000 7fd94f7fe700 1 --2- 172.21.15.42:0/810846366 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd9501197c0 0x7fd95011a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:16:59.728 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.709+0000 7fd94ffff700 1 -- 172.21.15.42:0/810846366 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd9501197c0 msgr2=0x7fd95011a8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:16:59.728 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.709+0000 7fd94ffff700 1 --2- 172.21.15.42:0/810846366 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd9501197c0 0x7fd95011a8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:16:59.728 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.709+0000 7fd94ffff700 1 -- 172.21.15.42:0/810846366 >> v1:172.21.15.40:6789/0 conn(0x7fd95011b7a0 legacy=0x7fd950077980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:16:59.728 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.709+0000 7fd94ffff700 1 -- 172.21.15.42:0/810846366 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd95012d980 con 0x7fd950078180 2019-12-05T05:16:59.728 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.709+0000 7fd94ffff700 1 --2- 172.21.15.42:0/810846366 >> v2:172.21.15.42:3300/0 conn(0x7fd950078180 0x7fd95011bbd0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fd9380010d0 tx=0x7fd940010c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:16:59.729 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.709+0000 7fd94ffff700 1 --2- 172.21.15.42:0/810846366 >> v2:172.21.15.42:3300/0 conn(0x7fd950078180 0x7fd95011bbd0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fd9380010d0 tx=0x7fd940010c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:16:59.729 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.709+0000 7fd94ffff700 1 --2- 172.21.15.42:0/810846366 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd950078180 0x7fd95011bbd0 secure :-1 s=READY pgs=603 cs=0 l=1 rx=0x7fd9380010d0 tx=0x7fd940010c30).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:16:59.729 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.709+0000 7fd94ffff700 1 --2- 172.21.15.42:0/810846366 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd950078180 0x7fd95011bbd0 secure :-1 s=THROTTLE_DONE pgs=603 cs=0 l=1 rx=0x7fd9380010d0 tx=0x7fd940010c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:16:59.729 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.709+0000 7fd94ffff700 1 --2- 172.21.15.42:0/810846366 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd950078180 0x7fd95011bbd0 secure :-1 s=THROTTLE_DONE pgs=603 cs=0 l=1 rx=0x7fd9380010d0 tx=0x7fd940010c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:16:59.730 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.709+0000 7fd94ffff700 1 --2- 172.21.15.42:0/810846366 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd950078180 0x7fd95011bbd0 secure :-1 s=THROTTLE_DONE pgs=603 cs=0 l=1 rx=0x7fd9380010d0 tx=0x7fd940010c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:16:59.730 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.709+0000 7fd947fff700 1 -- 172.21.15.42:0/810846366 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd940007350 con 0x7fd950078180 2019-12-05T05:16:59.730 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.709+0000 7fd947fff700 1 -- 172.21.15.42:0/810846366 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd940005cb0 con 0x7fd950078180 2019-12-05T05:16:59.730 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.709+0000 7fd9552a3700 1 -- 172.21.15.42:0/810846366 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd95012dc50 con 0x7fd950078180 2019-12-05T05:16:59.731 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.709+0000 7fd947fff700 1 -- 172.21.15.42:0/810846366 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd940008780 con 0x7fd950078180 2019-12-05T05:16:59.731 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.709+0000 7fd9552a3700 1 -- 172.21.15.42:0/810846366 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd950134600 con 0x7fd950078180 2019-12-05T05:16:59.731 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.709+0000 7fd94ffff700 1 --2- 172.21.15.42:0/810846366 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd950078180 0x7fd95011bbd0 secure :-1 s=THROTTLE_DONE pgs=603 cs=0 l=1 rx=0x7fd9380010d0 tx=0x7fd940010c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:16:59.731 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.713+0000 7fd947fff700 1 -- 172.21.15.42:0/810846366 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fd94000b250 con 0x7fd950078180 2019-12-05T05:16:59.731 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.713+0000 7fd94ffff700 1 --2- 172.21.15.42:0/810846366 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd950078180 0x7fd95011bbd0 secure :-1 s=THROTTLE_DONE pgs=603 cs=0 l=1 rx=0x7fd9380010d0 tx=0x7fd940010c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:16:59.731 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.713+0000 7fd947fff700 1 --2- 172.21.15.42:0/810846366 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd93403ad20 0x7fd93403d230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:16:59.732 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.713+0000 7fd947fff700 1 -- 172.21.15.42:0/810846366 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6805+0+0 (secure 0 0 0) 0x7fd9400082d0 con 0x7fd950078180 2019-12-05T05:16:59.732 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.713+0000 7fd94f7fe700 1 --2- 172.21.15.42:0/810846366 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd93403ad20 0x7fd93403d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:16:59.732 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.713+0000 7fd94f7fe700 1 --2- 172.21.15.42:0/810846366 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd93403ad20 0x7fd93403d230 crc :-1 s=READY pgs=270 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:16:59.732 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.717+0000 7fd9552a3700 1 -- 172.21.15.42:0/810846366 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd95005d100 con 0x7fd950078180 2019-12-05T05:16:59.733 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.725+0000 7fd94ffff700 1 --2- 172.21.15.42:0/810846366 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd950078180 0x7fd95011bbd0 secure :-1 s=THROTTLE_DONE pgs=603 cs=0 l=1 rx=0x7fd9380010d0 tx=0x7fd940010c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:16:59.733 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.725+0000 7fd947fff700 1 -- 172.21.15.42:0/810846366 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fd940032030 con 0x7fd950078180 2019-12-05T05:16:59.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.889+0000 7fd9552a3700 1 -- 172.21.15.42:0/810846366 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd95005b8b0 con 0x7fd950078180 2019-12-05T05:16:59.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.889+0000 7fd94ffff700 1 --2- 172.21.15.42:0/810846366 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd950078180 0x7fd95011bbd0 secure :-1 s=THROTTLE_DONE pgs=603 cs=0 l=1 rx=0x7fd9380010d0 tx=0x7fd940010c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:16:59.897 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.889+0000 7fd947fff700 1 -- 172.21.15.42:0/810846366 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 78 v78) v1 ==== 94+0+2108 (secure 0 0 0) 0x7fd940032290 con 0x7fd950078180 2019-12-05T05:16:59.897 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 78 2019-12-05T05:16:59.898 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:16:59.900 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":78,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5844,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/2698421979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2698421979},{"type":"v1","addr":"172.21.15.40:6833","nonce":2698421979}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":78}],"filesystems":[{"mdsmap":{"epoch":77,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:15:13.428215+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5629},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5629":{"gid":5629,"name":"a","rank":0,"incarnation":74,"state":"up:active","state_seq":15,"addr":"172.21.15.42:6835/3264829400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3264829400},{"type":"v1","addr":"172.21.15.42:6835","nonce":3264829400}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:16:59.901 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.893+0000 7fd9552a3700 1 -- 172.21.15.42:0/810846366 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd93403ad20 msgr2=0x7fd93403d230 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:16:59.902 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.893+0000 7fd9552a3700 1 --2- 172.21.15.42:0/810846366 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd93403ad20 0x7fd93403d230 crc :-1 s=READY pgs=270 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:16:59.902 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.893+0000 7fd9552a3700 1 -- 172.21.15.42:0/810846366 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd950078180 msgr2=0x7fd95011bbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:16:59.902 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.893+0000 7fd9552a3700 1 --2- 172.21.15.42:0/810846366 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd950078180 0x7fd95011bbd0 secure :-1 s=READY pgs=603 cs=0 l=1 rx=0x7fd9380010d0 tx=0x7fd940010c30).stop 2019-12-05T05:16:59.902 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.893+0000 7fd9552a3700 1 -- 172.21.15.42:0/810846366 shutdown_connections 2019-12-05T05:16:59.902 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.893+0000 7fd9552a3700 1 --2- 172.21.15.42:0/810846366 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd93403ad20 0x7fd93403d230 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:16:59.902 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.893+0000 7fd9552a3700 1 --2- 172.21.15.42:0/810846366 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd950078180 0x7fd95011bbd0 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:16:59.903 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.893+0000 7fd9552a3700 1 --2- 172.21.15.42:0/810846366 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd9501197c0 0x7fd95011a8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:16:59.903 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.893+0000 7fd9552a3700 1 -- 172.21.15.42:0/810846366 shutdown_connections 2019-12-05T05:16:59.903 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.893+0000 7fd9552a3700 1 -- 172.21.15.42:0/810846366 wait complete. 2019-12-05T05:16:59.903 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:16:59.893+0000 7fd9552a3700 1 -- 172.21.15.42:0/810846366 >> 172.21.15.42:0/810846366 conn(0x7fd9500a8290 msgr2=0x7fd95011c530 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:16:59.907 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T05:17:01.908 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:17:02.019 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:02.009+0000 7fdcefae6700 1 Processor -- start 2019-12-05T05:17:02.020 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:02.009+0000 7fdcefae6700 1 -- start start 2019-12-05T05:17:02.020 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:02.009+0000 7fdcefae6700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fdce808df70 0x7fdce809f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:17:02.020 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:02.009+0000 7fdcefae6700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fdce809f820 0x7fdce814efd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:17:02.020 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:02.009+0000 7fdcefae6700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdce809fc50 con 0x7fdce80a7730 2019-12-05T05:17:02.020 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:02.009+0000 7fdcefae6700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdce80a0af0 con 0x7fdce808df70 2019-12-05T05:17:02.021 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:02.009+0000 7fdcefae6700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdce808e5a0 con 0x7fdce809f820 2019-12-05T05:17:02.021 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:02.009+0000 7fdcee884700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fdce809f820 0x7fdce814efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:17:02.021 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:02.009+0000 7fdcee083700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fdce80a7730 0x7fdce807e710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:59000/0 (socket says 172.21.15.42:59000) 2019-12-05T05:17:02.021 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:02.009+0000 7fdcee083700 1 -- 172.21.15.42:0/4112059823 learned_addr learned my addr 172.21.15.42:0/4112059823 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:17:02.021 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:02.009+0000 7fdced882700 1 --2- 172.21.15.42:0/4112059823 >> v2:172.21.15.40:3300/0 conn(0x7fdce808df70 0x7fdce809f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:17:02.022 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:02.009+0000 7fdced081700 1 -- 172.21.15.42:0/4112059823 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2469861467 0 0) 0x7fdce809fc50 con 0x7fdce80a7730 2019-12-05T05:17:02.022 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:02.009+0000 7fdced081700 1 -- 172.21.15.42:0/4112059823 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdcd40030b0 con 0x7fdce80a7730 2019-12-05T05:17:02.022 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:02.009+0000 7fdcee884700 1 -- 172.21.15.42:0/4112059823 >> v2:172.21.15.40:3300/0 conn(0x7fdce808df70 msgr2=0x7fdce809f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:17:02.022 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:02.009+0000 7fdcee884700 1 --2- 172.21.15.42:0/4112059823 >> v2:172.21.15.40:3300/0 conn(0x7fdce808df70 0x7fdce809f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:17:02.022 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:02.009+0000 7fdcee884700 1 -- 172.21.15.42:0/4112059823 >> v1:172.21.15.42:6789/0 conn(0x7fdce80a7730 legacy=0x7fdce807e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:17:02.022 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:02.009+0000 7fdcee884700 1 -- 172.21.15.42:0/4112059823 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdce80a0700 con 0x7fdce809f820 2019-12-05T05:17:02.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:02.009+0000 7fdced882700 1 --2- 172.21.15.42:0/4112059823 >> v2:172.21.15.40:3300/0 conn(0x7fdce808df70 0x7fdce809f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:17:02.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:02.013+0000 7fdcee884700 1 --2- 172.21.15.42:0/4112059823 >> v2:172.21.15.42:3300/0 conn(0x7fdce809f820 0x7fdce814efd0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fdce00032b0 tx=0x7fdce000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:02.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:02.013+0000 7fdcee884700 1 --2- 172.21.15.42:0/4112059823 >> v2:172.21.15.42:3300/0 conn(0x7fdce809f820 0x7fdce814efd0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fdce00032b0 tx=0x7fdce000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:02.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:02.013+0000 7fdcee884700 1 --2- 172.21.15.42:0/4112059823 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdce809f820 0x7fdce814efd0 secure :-1 s=READY pgs=605 cs=0 l=1 rx=0x7fdce00032b0 tx=0x7fdce000fff0).ready entity=mon.1 client_cookie=7673a705ebe81b7b server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:17:02.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:02.013+0000 7fdcee884700 1 --2- 172.21.15.42:0/4112059823 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdce809f820 0x7fdce814efd0 secure :-1 s=THROTTLE_DONE pgs=605 cs=0 l=1 rx=0x7fdce00032b0 tx=0x7fdce000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:02.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:02.013+0000 7fdced081700 1 -- 172.21.15.42:0/4112059823 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdce00108c0 con 0x7fdce809f820 2019-12-05T05:17:02.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:02.013+0000 7fdcee884700 1 --2- 172.21.15.42:0/4112059823 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdce809f820 0x7fdce814efd0 secure :-1 s=THROTTLE_DONE pgs=605 cs=0 l=1 rx=0x7fdce00032b0 tx=0x7fdce000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:02.024 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:02.013+0000 7fdcee884700 1 --2- 172.21.15.42:0/4112059823 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdce809f820 0x7fdce814efd0 secure :-1 s=THROTTLE_DONE pgs=605 cs=0 l=1 rx=0x7fdce00032b0 tx=0x7fdce000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:02.024 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:02.013+0000 7fdced081700 1 -- 172.21.15.42:0/4112059823 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdce0008b50 con 0x7fdce809f820 2019-12-05T05:17:02.024 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:02.013+0000 7fdced081700 1 -- 172.21.15.42:0/4112059823 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdce00110c0 con 0x7fdce809f820 2019-12-05T05:17:02.024 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:02.013+0000 7fdcefae6700 1 -- 172.21.15.42:0/4112059823 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdce809f820 msgr2=0x7fdce814efd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:17:02.024 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:02.013+0000 7fdcefae6700 1 --2- 172.21.15.42:0/4112059823 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdce809f820 0x7fdce814efd0 secure :-1 s=READY pgs=605 cs=0 l=1 rx=0x7fdce00032b0 tx=0x7fdce000fff0).stop 2019-12-05T05:17:02.024 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:02.013+0000 7fdcefae6700 1 -- 172.21.15.42:0/4112059823 shutdown_connections 2019-12-05T05:17:02.024 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:02.013+0000 7fdcefae6700 1 --2- 172.21.15.42:0/4112059823 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdce809f820 0x7fdce814efd0 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:17:02.025 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:02.013+0000 7fdcefae6700 1 --2- 172.21.15.42:0/4112059823 >> v2:172.21.15.40:3300/0 conn(0x7fdce808df70 0x7fdce809f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:17:02.025 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:02.013+0000 7fdcefae6700 1 -- 172.21.15.42:0/4112059823 shutdown_connections 2019-12-05T05:17:02.025 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:02.013+0000 7fdcefae6700 1 -- 172.21.15.42:0/4112059823 wait complete. 2019-12-05T05:17:02.025 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:02.013+0000 7fdcefae6700 1 -- 172.21.15.42:0/4112059823 >> 172.21.15.42:0/4112059823 conn(0x7fdce80aabd0 msgr2=0x7fdce80a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:17:02.029 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:02.017+0000 7fdcefae6700 1 Processor -- start 2019-12-05T05:17:02.029 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:02.017+0000 7fdcefae6700 1 -- start start 2019-12-05T05:17:02.029 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:02.017+0000 7fdcefae6700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdce807dbe0 0x7fdce811b400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:17:02.029 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:02.017+0000 7fdcefae6700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdce80a6fc0 con 0x7fdce80a5df0 2019-12-05T05:17:02.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:02.017+0000 7fdcefae6700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdce811a5d0 con 0x7fdce80a6220 2019-12-05T05:17:02.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:02.017+0000 7fdcefae6700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdce80a67e0 con 0x7fdce807dbe0 2019-12-05T05:17:02.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:02.017+0000 7fdcee884700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fdce80a6220 0x7fdce8136ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:59010/0 (socket says 172.21.15.42:59010) 2019-12-05T05:17:02.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:02.017+0000 7fdcee884700 1 -- 172.21.15.42:0/807244338 learned_addr learned my addr 172.21.15.42:0/807244338 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:17:02.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:02.017+0000 7fdcee083700 1 --2- 172.21.15.42:0/807244338 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdce807dbe0 0x7fdce811b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:17:02.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:02.017+0000 7fdcdeffd700 1 -- 172.21.15.42:0/807244338 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3139856783 0 0) 0x7fdce811a5d0 con 0x7fdce80a6220 2019-12-05T05:17:02.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:02.017+0000 7fdcdeffd700 1 -- 172.21.15.42:0/807244338 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdcd00030b0 con 0x7fdce80a6220 2019-12-05T05:17:02.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:02.017+0000 7fdcdeffd700 1 -- 172.21.15.42:0/807244338 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3865238880 0 0) 0x7fdce80a6fc0 con 0x7fdce80a5df0 2019-12-05T05:17:02.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:02.017+0000 7fdcdeffd700 1 -- 172.21.15.42:0/807244338 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdce811a5d0 con 0x7fdce80a5df0 2019-12-05T05:17:02.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:02.017+0000 7fdcdeffd700 1 -- 172.21.15.42:0/807244338 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 2978833843 0 0) 0x7fdcd00030b0 con 0x7fdce80a6220 2019-12-05T05:17:02.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:02.017+0000 7fdcdeffd700 1 -- 172.21.15.42:0/807244338 --> v1:172.21.15.42:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdce80a6fc0 con 0x7fdce80a6220 2019-12-05T05:17:02.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:02.017+0000 7fdcdeffd700 1 -- 172.21.15.42:0/807244338 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7fdce00108c0 con 0x7fdce80a6220 2019-12-05T05:17:02.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:02.017+0000 7fdcee083700 1 -- 172.21.15.42:0/807244338 >> v1:172.21.15.42:6789/0 conn(0x7fdce80a6220 legacy=0x7fdce8136ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:17:02.032 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:02.017+0000 7fdcee083700 1 -- 172.21.15.42:0/807244338 >> v1:172.21.15.40:6789/0 conn(0x7fdce80a5df0 legacy=0x7fdce8119dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:17:02.032 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:02.017+0000 7fdcee083700 1 -- 172.21.15.42:0/807244338 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdce811b950 con 0x7fdce807dbe0 2019-12-05T05:17:02.032 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:02.017+0000 7fdcee083700 1 --2- 172.21.15.42:0/807244338 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdce807dbe0 0x7fdce811b400 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fdce4004370 tx=0x7fdcd8007ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:02.032 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:02.021+0000 7fdcee083700 1 --2- 172.21.15.42:0/807244338 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdce807dbe0 0x7fdce811b400 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fdce4004370 tx=0x7fdcd8007ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:02.032 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:02.021+0000 7fdcee083700 1 --2- 172.21.15.42:0/807244338 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdce807dbe0 0x7fdce811b400 secure :-1 s=READY pgs=122 cs=0 l=1 rx=0x7fdce4004370 tx=0x7fdcd8007ad0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:17:02.032 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:02.021+0000 7fdcee083700 1 --2- 172.21.15.42:0/807244338 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdce807dbe0 0x7fdce811b400 secure :-1 s=THROTTLE_DONE pgs=122 cs=0 l=1 rx=0x7fdce4004370 tx=0x7fdcd8007ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:02.032 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:02.021+0000 7fdcdeffd700 1 -- 172.21.15.42:0/807244338 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdce4005a20 con 0x7fdce807dbe0 2019-12-05T05:17:02.033 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:02.021+0000 7fdcee083700 1 --2- 172.21.15.42:0/807244338 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdce807dbe0 0x7fdce811b400 secure :-1 s=THROTTLE_DONE pgs=122 cs=0 l=1 rx=0x7fdce4004370 tx=0x7fdcd8007ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:02.033 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:02.021+0000 7fdcdeffd700 1 -- 172.21.15.42:0/807244338 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdce4005640 con 0x7fdce807dbe0 2019-12-05T05:17:02.033 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:02.021+0000 7fdcefae6700 1 -- 172.21.15.42:0/807244338 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdce807e010 con 0x7fdce807dbe0 2019-12-05T05:17:02.033 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:02.021+0000 7fdcee083700 1 --2- 172.21.15.42:0/807244338 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdce807dbe0 0x7fdce811b400 secure :-1 s=THROTTLE_DONE pgs=122 cs=0 l=1 rx=0x7fdce4004370 tx=0x7fdcd8007ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:02.033 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:02.021+0000 7fdcdeffd700 1 -- 172.21.15.42:0/807244338 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdce40060c0 con 0x7fdce807dbe0 2019-12-05T05:17:02.033 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:02.021+0000 7fdcefae6700 1 -- 172.21.15.42:0/807244338 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdce80a9780 con 0x7fdce807dbe0 2019-12-05T05:17:02.033 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:02.021+0000 7fdcee083700 1 --2- 172.21.15.42:0/807244338 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdce807dbe0 0x7fdce811b400 secure :-1 s=THROTTLE_DONE pgs=122 cs=0 l=1 rx=0x7fdce4004370 tx=0x7fdcd8007ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:02.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:02.021+0000 7fdcdeffd700 1 -- 172.21.15.42:0/807244338 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fdce40063a0 con 0x7fdce807dbe0 2019-12-05T05:17:02.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:02.021+0000 7fdcee083700 1 --2- 172.21.15.42:0/807244338 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdce807dbe0 0x7fdce811b400 secure :-1 s=THROTTLE_DONE pgs=122 cs=0 l=1 rx=0x7fdce4004370 tx=0x7fdcd8007ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:02.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:02.021+0000 7fdcdeffd700 1 --2- 172.21.15.42:0/807244338 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fdcd003b840 0x7fdcd003dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:17:02.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:02.021+0000 7fdcdeffd700 1 -- 172.21.15.42:0/807244338 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6805+0+0 (secure 0 0 0) 0x7fdce404acf0 con 0x7fdce807dbe0 2019-12-05T05:17:02.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:02.021+0000 7fdced882700 1 --2- 172.21.15.42:0/807244338 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fdcd003b840 0x7fdcd003dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:17:02.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:02.021+0000 7fdced882700 1 --2- 172.21.15.42:0/807244338 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fdcd003b840 0x7fdcd003dd50 crc :-1 s=READY pgs=271 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:17:02.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:02.025+0000 7fdcefae6700 1 -- 172.21.15.42:0/807244338 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdce805d100 con 0x7fdce807dbe0 2019-12-05T05:17:02.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:02.033+0000 7fdcee083700 1 --2- 172.21.15.42:0/807244338 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdce807dbe0 0x7fdce811b400 secure :-1 s=THROTTLE_DONE pgs=122 cs=0 l=1 rx=0x7fdce4004370 tx=0x7fdcd8007ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:02.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:02.033+0000 7fdcdeffd700 1 -- 172.21.15.42:0/807244338 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fdce4031d50 con 0x7fdce807dbe0 2019-12-05T05:17:02.210 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:02.205+0000 7fdcefae6700 1 -- 172.21.15.42:0/807244338 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdce805b8b0 con 0x7fdce807dbe0 2019-12-05T05:17:02.211 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:02.205+0000 7fdcee083700 1 --2- 172.21.15.42:0/807244338 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdce807dbe0 0x7fdce811b400 secure :-1 s=THROTTLE_DONE pgs=122 cs=0 l=1 rx=0x7fdce4004370 tx=0x7fdcd8007ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:02.211 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:02.205+0000 7fdcdeffd700 1 -- 172.21.15.42:0/807244338 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 78 v78) v1 ==== 94+0+2108 (secure 0 0 0) 0x7fdce4031fb0 con 0x7fdce807dbe0 2019-12-05T05:17:02.213 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 78 2019-12-05T05:17:02.213 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:17:02.213 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":78,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5844,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/2698421979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2698421979},{"type":"v1","addr":"172.21.15.40:6833","nonce":2698421979}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":78}],"filesystems":[{"mdsmap":{"epoch":77,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:15:13.428215+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5629},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5629":{"gid":5629,"name":"a","rank":0,"incarnation":74,"state":"up:active","state_seq":15,"addr":"172.21.15.42:6835/3264829400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3264829400},{"type":"v1","addr":"172.21.15.42:6835","nonce":3264829400}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:17:02.215 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:02.209+0000 7fdcefae6700 1 -- 172.21.15.42:0/807244338 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fdcd003b840 msgr2=0x7fdcd003dd50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:17:02.216 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:02.209+0000 7fdcefae6700 1 --2- 172.21.15.42:0/807244338 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fdcd003b840 0x7fdcd003dd50 crc :-1 s=READY pgs=271 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:17:02.216 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:02.209+0000 7fdcefae6700 1 -- 172.21.15.42:0/807244338 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdce807dbe0 msgr2=0x7fdce811b400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:17:02.216 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:02.209+0000 7fdcefae6700 1 --2- 172.21.15.42:0/807244338 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdce807dbe0 0x7fdce811b400 secure :-1 s=READY pgs=122 cs=0 l=1 rx=0x7fdce4004370 tx=0x7fdcd8007ad0).stop 2019-12-05T05:17:02.216 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:02.209+0000 7fdcefae6700 1 -- 172.21.15.42:0/807244338 shutdown_connections 2019-12-05T05:17:02.216 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:02.209+0000 7fdcefae6700 1 --2- 172.21.15.42:0/807244338 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fdcd003b840 0x7fdcd003dd50 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:17:02.216 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:02.209+0000 7fdcefae6700 1 --2- 172.21.15.42:0/807244338 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdce807dbe0 0x7fdce811b400 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:17:02.217 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:02.209+0000 7fdcefae6700 1 -- 172.21.15.42:0/807244338 shutdown_connections 2019-12-05T05:17:02.217 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:02.209+0000 7fdcefae6700 1 -- 172.21.15.42:0/807244338 wait complete. 2019-12-05T05:17:02.217 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:02.213+0000 7fdcefae6700 1 -- 172.21.15.42:0/807244338 >> 172.21.15.42:0/807244338 conn(0x7fdce805a1b0 msgr2=0x7fdce8118870 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:17:02.235 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T05:17:04.236 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:17:04.341 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:04.329+0000 7f10d6f10700 1 Processor -- start 2019-12-05T05:17:04.341 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:04.329+0000 7f10d6f10700 1 -- start start 2019-12-05T05:17:04.341 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:04.333+0000 7f10d6f10700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f10d00a7740 0x7f10d008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:17:04.341 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:04.333+0000 7f10d6f10700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10d00a0710 con 0x7f10d007e720 2019-12-05T05:17:04.342 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:04.333+0000 7f10d6f10700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10d00a9ba0 con 0x7f10d009fae0 2019-12-05T05:17:04.342 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:04.333+0000 7f10d6f10700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10d00a0b00 con 0x7f10d00a7740 2019-12-05T05:17:04.342 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:04.333+0000 7f10d5cae700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f10d009fae0 0x7f10d014efe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:59020/0 (socket says 172.21.15.42:59020) 2019-12-05T05:17:04.342 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:04.333+0000 7f10d5cae700 1 -- 172.21.15.42:0/2668084827 learned_addr learned my addr 172.21.15.42:0/2668084827 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:17:04.342 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:04.333+0000 7f10d54ad700 1 --2- 172.21.15.42:0/2668084827 >> v2:172.21.15.40:3300/0 conn(0x7f10d00a7740 0x7f10d008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:17:04.342 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:04.333+0000 7f10cffff700 1 -- 172.21.15.42:0/2668084827 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1039601244 0 0) 0x7f10d00a9ba0 con 0x7f10d009fae0 2019-12-05T05:17:04.342 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:04.333+0000 7f10cffff700 1 -- 172.21.15.42:0/2668084827 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10b80030b0 con 0x7f10d009fae0 2019-12-05T05:17:04.343 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:04.333+0000 7f10cffff700 1 -- 172.21.15.42:0/2668084827 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 576684155 0 0) 0x7f10b80030b0 con 0x7f10d009fae0 2019-12-05T05:17:04.343 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:04.333+0000 7f10d54ad700 1 -- 172.21.15.42:0/2668084827 >> v1:172.21.15.42:6789/0 conn(0x7f10d009fae0 legacy=0x7f10d014efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:17:04.343 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:04.333+0000 7f10d54ad700 1 -- 172.21.15.42:0/2668084827 >> v1:172.21.15.40:6789/0 conn(0x7f10d007e720 legacy=0x7f10d009f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:17:04.343 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:04.333+0000 7f10d54ad700 1 -- 172.21.15.42:0/2668084827 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10d007eb50 con 0x7f10d00a7740 2019-12-05T05:17:04.343 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:04.333+0000 7f10d54ad700 1 --2- 172.21.15.42:0/2668084827 >> v2:172.21.15.40:3300/0 conn(0x7f10d00a7740 0x7f10d008df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f10c8008b10 tx=0x7f10c00014a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:04.343 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:04.333+0000 7f10d54ad700 1 --2- 172.21.15.42:0/2668084827 >> v2:172.21.15.40:3300/0 conn(0x7f10d00a7740 0x7f10d008df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f10c8008b10 tx=0x7f10c00014a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:04.343 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:04.333+0000 7f10d54ad700 1 --2- 172.21.15.42:0/2668084827 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f10d00a7740 0x7f10d008df80 secure :-1 s=READY pgs=480 cs=0 l=1 rx=0x7f10c8008b10 tx=0x7f10c00014a0).ready entity=mon.0 client_cookie=9b5a283d8aa9852c server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:17:04.344 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:04.333+0000 7f10d54ad700 1 --2- 172.21.15.42:0/2668084827 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f10d00a7740 0x7f10d008df80 secure :-1 s=THROTTLE_DONE pgs=480 cs=0 l=1 rx=0x7f10c8008b10 tx=0x7f10c00014a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:04.344 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:04.333+0000 7f10cffff700 1 -- 172.21.15.42:0/2668084827 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f10c0001750 con 0x7f10d00a7740 2019-12-05T05:17:04.344 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:04.333+0000 7f10d54ad700 1 --2- 172.21.15.42:0/2668084827 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f10d00a7740 0x7f10d008df80 secure :-1 s=THROTTLE_DONE pgs=480 cs=0 l=1 rx=0x7f10c8008b10 tx=0x7f10c00014a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:04.344 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:04.333+0000 7f10cffff700 1 -- 172.21.15.42:0/2668084827 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10c0001560 con 0x7f10d00a7740 2019-12-05T05:17:04.344 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:04.333+0000 7f10d54ad700 1 --2- 172.21.15.42:0/2668084827 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f10d00a7740 0x7f10d008df80 secure :-1 s=THROTTLE_DONE pgs=480 cs=0 l=1 rx=0x7f10c8008b10 tx=0x7f10c00014a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:04.344 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:04.333+0000 7f10cffff700 1 -- 172.21.15.42:0/2668084827 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f10c8011ac0 con 0x7f10d00a7740 2019-12-05T05:17:04.344 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:04.333+0000 7f10d6f10700 1 -- 172.21.15.42:0/2668084827 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f10d00a7740 msgr2=0x7f10d008df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:17:04.345 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:04.333+0000 7f10d6f10700 1 --2- 172.21.15.42:0/2668084827 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f10d00a7740 0x7f10d008df80 secure :-1 s=READY pgs=480 cs=0 l=1 rx=0x7f10c8008b10 tx=0x7f10c00014a0).stop 2019-12-05T05:17:04.345 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:04.333+0000 7f10d6f10700 1 -- 172.21.15.42:0/2668084827 shutdown_connections 2019-12-05T05:17:04.345 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:04.333+0000 7f10d6f10700 1 --2- 172.21.15.42:0/2668084827 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f10d00a7740 0x7f10d008df80 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:17:04.345 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:04.333+0000 7f10d6f10700 1 -- 172.21.15.42:0/2668084827 shutdown_connections 2019-12-05T05:17:04.345 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:04.333+0000 7f10d6f10700 1 -- 172.21.15.42:0/2668084827 wait complete. 2019-12-05T05:17:04.345 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:04.333+0000 7f10d6f10700 1 -- 172.21.15.42:0/2668084827 >> 172.21.15.42:0/2668084827 conn(0x7f10d00aabe0 msgr2=0x7f10d00a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:17:04.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:04.337+0000 7f10d6f10700 1 Processor -- start 2019-12-05T05:17:04.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:04.337+0000 7f10d6f10700 1 -- start start 2019-12-05T05:17:04.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:04.337+0000 7f10d6f10700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f10d011af80 0x7f10d011b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:17:04.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:04.337+0000 7f10d6f10700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f10d00732f0 0x7f10d0073720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:17:04.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:04.337+0000 7f10d6f10700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10d011b9f0 con 0x7f10d0116500 2019-12-05T05:17:04.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:04.337+0000 7f10d6f10700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10d0118a10 con 0x7f10d011af80 2019-12-05T05:17:04.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:04.337+0000 7f10d6f10700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10d01184d0 con 0x7f10d00732f0 2019-12-05T05:17:04.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:04.341+0000 7f10d4cac700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f10d011af80 0x7f10d011b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:17:04.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:04.341+0000 7f10d4cac700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f10d011af80 0x7f10d011b3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:58986/0 (socket says 172.21.15.42:58986) 2019-12-05T05:17:04.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:04.341+0000 7f10d4cac700 1 -- 172.21.15.42:0/2275854280 learned_addr learned my addr 172.21.15.42:0/2275854280 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:17:04.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:04.341+0000 7f10d54ad700 1 --1- 172.21.15.42:0/2275854280 >> v1:172.21.15.42:6789/0 conn(0x7f10d0116500 0x7f10d0119c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:59022/0 (socket says 172.21.15.42:59022) 2019-12-05T05:17:04.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:04.341+0000 7f10d5cae700 1 --2- 172.21.15.42:0/2275854280 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f10d00732f0 0x7f10d0073720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:17:04.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:04.341+0000 7f10d4cac700 1 -- 172.21.15.42:0/2275854280 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f10d00732f0 msgr2=0x7f10d0073720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:17:04.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:04.341+0000 7f10d4cac700 1 --2- 172.21.15.42:0/2275854280 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f10d00732f0 0x7f10d0073720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:17:04.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:04.341+0000 7f10d4cac700 1 -- 172.21.15.42:0/2275854280 >> v1:172.21.15.42:6789/0 conn(0x7f10d0116500 legacy=0x7f10d0119c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:17:04.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:04.341+0000 7f10d4cac700 1 -- 172.21.15.42:0/2275854280 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10d01180a0 con 0x7f10d011af80 2019-12-05T05:17:04.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:04.341+0000 7f10cdffb700 1 -- 172.21.15.42:0/2275854280 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2220720371 0 0) 0x7f10d011b9f0 con 0x7f10d0116500 2019-12-05T05:17:04.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:04.341+0000 7f10d4cac700 1 --2- 172.21.15.42:0/2275854280 >> v2:172.21.15.42:3300/0 conn(0x7f10d011af80 0x7f10d011b3b0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f10d0118690 tx=0x7f10c40109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:04.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:04.341+0000 7f10d4cac700 1 --2- 172.21.15.42:0/2275854280 >> v2:172.21.15.42:3300/0 conn(0x7f10d011af80 0x7f10d011b3b0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f10d0118690 tx=0x7f10c40109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:04.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:04.341+0000 7f10d4cac700 1 --2- 172.21.15.42:0/2275854280 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f10d011af80 0x7f10d011b3b0 secure :-1 s=READY pgs=609 cs=0 l=1 rx=0x7f10d0118690 tx=0x7f10c40109a0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:17:04.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:04.341+0000 7f10d4cac700 1 --2- 172.21.15.42:0/2275854280 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f10d011af80 0x7f10d011b3b0 secure :-1 s=THROTTLE_DONE pgs=609 cs=0 l=1 rx=0x7f10d0118690 tx=0x7f10c40109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:04.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:04.341+0000 7f10cdffb700 1 -- 172.21.15.42:0/2275854280 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f10c400caa0 con 0x7f10d011af80 2019-12-05T05:17:04.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:04.341+0000 7f10d4cac700 1 --2- 172.21.15.42:0/2275854280 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f10d011af80 0x7f10d011b3b0 secure :-1 s=THROTTLE_DONE pgs=609 cs=0 l=1 rx=0x7f10d0118690 tx=0x7f10c40109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:04.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:04.341+0000 7f10d4cac700 1 --2- 172.21.15.42:0/2275854280 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f10d011af80 0x7f10d011b3b0 secure :-1 s=THROTTLE_DONE pgs=609 cs=0 l=1 rx=0x7f10d0118690 tx=0x7f10c40109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:04.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:04.341+0000 7f10cdffb700 1 -- 172.21.15.42:0/2275854280 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10c400c6c0 con 0x7f10d011af80 2019-12-05T05:17:04.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:04.341+0000 7f10d6f10700 1 -- 172.21.15.42:0/2275854280 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10d0073c70 con 0x7f10d011af80 2019-12-05T05:17:04.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:04.341+0000 7f10cdffb700 1 -- 172.21.15.42:0/2275854280 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f10c4012210 con 0x7f10d011af80 2019-12-05T05:17:04.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:04.341+0000 7f10d6f10700 1 -- 172.21.15.42:0/2275854280 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10d007a600 con 0x7f10d011af80 2019-12-05T05:17:04.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:04.341+0000 7f10d4cac700 1 --2- 172.21.15.42:0/2275854280 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f10d011af80 0x7f10d011b3b0 secure :-1 s=THROTTLE_DONE pgs=609 cs=0 l=1 rx=0x7f10d0118690 tx=0x7f10c40109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:04.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:04.341+0000 7f10cdffb700 1 -- 172.21.15.42:0/2275854280 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f10c40124f0 con 0x7f10d011af80 2019-12-05T05:17:04.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:04.341+0000 7f10d4cac700 1 --2- 172.21.15.42:0/2275854280 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f10d011af80 0x7f10d011b3b0 secure :-1 s=THROTTLE_DONE pgs=609 cs=0 l=1 rx=0x7f10d0118690 tx=0x7f10c40109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:04.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:04.341+0000 7f10cdffb700 1 --2- 172.21.15.42:0/2275854280 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f10b003aca0 0x7f10b003d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:17:04.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:04.341+0000 7f10cdffb700 1 -- 172.21.15.42:0/2275854280 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6805+0+0 (secure 0 0 0) 0x7f10c404b250 con 0x7f10d011af80 2019-12-05T05:17:04.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:04.345+0000 7f10d54ad700 1 --2- 172.21.15.42:0/2275854280 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f10b003aca0 0x7f10b003d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:17:04.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:04.345+0000 7f10d54ad700 1 --2- 172.21.15.42:0/2275854280 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f10b003aca0 0x7f10b003d1b0 crc :-1 s=READY pgs=272 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:17:04.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:04.345+0000 7f10d6f10700 1 -- 172.21.15.42:0/2275854280 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10d005d110 con 0x7f10d011af80 2019-12-05T05:17:04.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:04.353+0000 7f10d4cac700 1 --2- 172.21.15.42:0/2275854280 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f10d011af80 0x7f10d011b3b0 secure :-1 s=THROTTLE_DONE pgs=609 cs=0 l=1 rx=0x7f10d0118690 tx=0x7f10c40109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:04.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:04.353+0000 7f10cdffb700 1 -- 172.21.15.42:0/2275854280 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f10c40322b0 con 0x7f10d011af80 2019-12-05T05:17:04.521 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:04.517+0000 7f10d6f10700 1 -- 172.21.15.42:0/2275854280 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f10d011ad60 con 0x7f10d011af80 2019-12-05T05:17:04.522 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:04.517+0000 7f10d4cac700 1 --2- 172.21.15.42:0/2275854280 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f10d011af80 0x7f10d011b3b0 secure :-1 s=THROTTLE_DONE pgs=609 cs=0 l=1 rx=0x7f10d0118690 tx=0x7f10c40109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:04.522 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:04.517+0000 7f10cdffb700 1 -- 172.21.15.42:0/2275854280 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 78 v78) v1 ==== 94+0+2108 (secure 0 0 0) 0x7f10c4032510 con 0x7f10d011af80 2019-12-05T05:17:04.522 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 78 2019-12-05T05:17:04.523 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:17:04.523 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":78,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5844,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/2698421979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2698421979},{"type":"v1","addr":"172.21.15.40:6833","nonce":2698421979}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":78}],"filesystems":[{"mdsmap":{"epoch":77,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:15:13.428215+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5629},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5629":{"gid":5629,"name":"a","rank":0,"incarnation":74,"state":"up:active","state_seq":15,"addr":"172.21.15.42:6835/3264829400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3264829400},{"type":"v1","addr":"172.21.15.42:6835","nonce":3264829400}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:17:04.525 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:04.521+0000 7f10d6f10700 1 -- 172.21.15.42:0/2275854280 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f10b003aca0 msgr2=0x7f10b003d1b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:17:04.525 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:04.521+0000 7f10d6f10700 1 --2- 172.21.15.42:0/2275854280 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f10b003aca0 0x7f10b003d1b0 crc :-1 s=READY pgs=272 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:17:04.525 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:04.521+0000 7f10d6f10700 1 -- 172.21.15.42:0/2275854280 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f10d011af80 msgr2=0x7f10d011b3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:17:04.525 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:04.521+0000 7f10d6f10700 1 --2- 172.21.15.42:0/2275854280 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f10d011af80 0x7f10d011b3b0 secure :-1 s=READY pgs=609 cs=0 l=1 rx=0x7f10d0118690 tx=0x7f10c40109a0).stop 2019-12-05T05:17:04.525 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:04.521+0000 7f10d6f10700 1 -- 172.21.15.42:0/2275854280 shutdown_connections 2019-12-05T05:17:04.526 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:04.521+0000 7f10d6f10700 1 --2- 172.21.15.42:0/2275854280 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f10b003aca0 0x7f10b003d1b0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:17:04.526 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:04.521+0000 7f10d6f10700 1 --2- 172.21.15.42:0/2275854280 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f10d00732f0 0x7f10d0073720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:17:04.526 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:04.521+0000 7f10d6f10700 1 --2- 172.21.15.42:0/2275854280 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f10d011af80 0x7f10d011b3b0 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:17:04.526 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:04.521+0000 7f10d6f10700 1 -- 172.21.15.42:0/2275854280 shutdown_connections 2019-12-05T05:17:04.526 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:04.521+0000 7f10d6f10700 1 -- 172.21.15.42:0/2275854280 wait complete. 2019-12-05T05:17:04.526 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:04.521+0000 7f10d6f10700 1 -- 172.21.15.42:0/2275854280 >> 172.21.15.42:0/2275854280 conn(0x7f10d008fde0 msgr2=0x7f10d00a9790 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:17:04.535 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T05:17:05.788 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:17:05.797 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:17:06.542 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:17:06.650 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:06.641+0000 7f34b55a3700 1 Processor -- start 2019-12-05T05:17:06.650 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:06.641+0000 7f34b55a3700 1 -- start start 2019-12-05T05:17:06.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:06.641+0000 7f34b55a3700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f34b008df80 0x7f34b009f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:17:06.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:06.641+0000 7f34b55a3700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f34b009f830 0x7f34b014efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:17:06.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:06.641+0000 7f34b55a3700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34b009fc60 con 0x7f34b00a7740 2019-12-05T05:17:06.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:06.641+0000 7f34b55a3700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34b00a0b00 con 0x7f34b008df80 2019-12-05T05:17:06.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:06.641+0000 7f34b55a3700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34b008e5b0 con 0x7f34b009f830 2019-12-05T05:17:06.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:06.641+0000 7f34affff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f34b009f830 0x7f34b014efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:17:06.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:06.641+0000 7f34af7fe700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f34b00a7740 0x7f34b007e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:59032/0 (socket says 172.21.15.42:59032) 2019-12-05T05:17:06.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:06.641+0000 7f34af7fe700 1 -- 172.21.15.42:0/1809458193 learned_addr learned my addr 172.21.15.42:0/1809458193 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:17:06.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:06.641+0000 7f34aeffd700 1 --2- 172.21.15.42:0/1809458193 >> v2:172.21.15.40:3300/0 conn(0x7f34b008df80 0x7f34b009f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:17:06.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:06.645+0000 7f34affff700 1 -- 172.21.15.42:0/1809458193 >> v2:172.21.15.40:3300/0 conn(0x7f34b008df80 msgr2=0x7f34b009f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:17:06.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:06.645+0000 7f34affff700 1 --2- 172.21.15.42:0/1809458193 >> v2:172.21.15.40:3300/0 conn(0x7f34b008df80 0x7f34b009f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:17:06.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:06.645+0000 7f34affff700 1 -- 172.21.15.42:0/1809458193 >> v1:172.21.15.42:6789/0 conn(0x7f34b00a7740 legacy=0x7f34b007e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:17:06.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:06.645+0000 7f34ae7fc700 1 -- 172.21.15.42:0/1809458193 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2900925178 0 0) 0x7f34b009fc60 con 0x7f34b00a7740 2019-12-05T05:17:06.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:06.645+0000 7f34affff700 1 -- 172.21.15.42:0/1809458193 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34b00a0710 con 0x7f34b009f830 2019-12-05T05:17:06.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:06.645+0000 7f34affff700 1 --2- 172.21.15.42:0/1809458193 >> v2:172.21.15.42:3300/0 conn(0x7f34b009f830 0x7f34b014efe0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f34a80032b0 tx=0x7f34a800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:06.654 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:06.645+0000 7f34affff700 1 --2- 172.21.15.42:0/1809458193 >> v2:172.21.15.42:3300/0 conn(0x7f34b009f830 0x7f34b014efe0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f34a80032b0 tx=0x7f34a800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:06.654 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:06.645+0000 7f34affff700 1 --2- 172.21.15.42:0/1809458193 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f34b009f830 0x7f34b014efe0 secure :-1 s=READY pgs=611 cs=0 l=1 rx=0x7f34a80032b0 tx=0x7f34a800fff0).ready entity=mon.1 client_cookie=2f4e607848064028 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:17:06.654 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:06.645+0000 7f34affff700 1 --2- 172.21.15.42:0/1809458193 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f34b009f830 0x7f34b014efe0 secure :-1 s=THROTTLE_DONE pgs=611 cs=0 l=1 rx=0x7f34a80032b0 tx=0x7f34a800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:06.654 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:06.645+0000 7f34affff700 1 --2- 172.21.15.42:0/1809458193 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f34b009f830 0x7f34b014efe0 secure :-1 s=THROTTLE_DONE pgs=611 cs=0 l=1 rx=0x7f34a80032b0 tx=0x7f34a800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:06.655 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:06.645+0000 7f34ae7fc700 1 -- 172.21.15.42:0/1809458193 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f34a80108c0 con 0x7f34b009f830 2019-12-05T05:17:06.655 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:06.645+0000 7f34affff700 1 --2- 172.21.15.42:0/1809458193 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f34b009f830 0x7f34b014efe0 secure :-1 s=THROTTLE_DONE pgs=611 cs=0 l=1 rx=0x7f34a80032b0 tx=0x7f34a800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:06.655 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:06.645+0000 7f34ae7fc700 1 -- 172.21.15.42:0/1809458193 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34a8008b50 con 0x7f34b009f830 2019-12-05T05:17:06.655 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:06.645+0000 7f34ae7fc700 1 -- 172.21.15.42:0/1809458193 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f34a80110c0 con 0x7f34b009f830 2019-12-05T05:17:06.672 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:06.645+0000 7f34b55a3700 1 -- 172.21.15.42:0/1809458193 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f34b009f830 msgr2=0x7f34b014efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:17:06.672 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:06.645+0000 7f34b55a3700 1 --2- 172.21.15.42:0/1809458193 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f34b009f830 0x7f34b014efe0 secure :-1 s=READY pgs=611 cs=0 l=1 rx=0x7f34a80032b0 tx=0x7f34a800fff0).stop 2019-12-05T05:17:06.672 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:06.645+0000 7f34b55a3700 1 -- 172.21.15.42:0/1809458193 shutdown_connections 2019-12-05T05:17:06.672 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:06.645+0000 7f34b55a3700 1 --2- 172.21.15.42:0/1809458193 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f34b009f830 0x7f34b014efe0 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:17:06.673 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:06.645+0000 7f34b55a3700 1 --2- 172.21.15.42:0/1809458193 >> v2:172.21.15.40:3300/0 conn(0x7f34b008df80 0x7f34b009f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:17:06.673 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:06.645+0000 7f34b55a3700 1 -- 172.21.15.42:0/1809458193 shutdown_connections 2019-12-05T05:17:06.673 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:06.645+0000 7f34b55a3700 1 -- 172.21.15.42:0/1809458193 wait complete. 2019-12-05T05:17:06.673 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:06.645+0000 7f34b55a3700 1 -- 172.21.15.42:0/1809458193 >> 172.21.15.42:0/1809458193 conn(0x7f34b00aabe0 msgr2=0x7f34b00a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:17:06.677 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:06.649+0000 7f34b55a3700 1 Processor -- start 2019-12-05T05:17:06.677 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:06.649+0000 7f34b55a3700 1 -- start start 2019-12-05T05:17:06.678 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:06.649+0000 7f34b55a3700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f34b0117050 0x7f34b0123f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:17:06.678 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:06.649+0000 7f34b55a3700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f34b0122250 0x7f34b0120830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:17:06.678 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:06.649+0000 7f34b55a3700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34b00a6c80 con 0x7f34b0118880 2019-12-05T05:17:06.678 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:06.649+0000 7f34b55a3700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34b0116b10 con 0x7f34b0122250 2019-12-05T05:17:06.678 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:06.649+0000 7f34b55a3700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34b0116db0 con 0x7f34b0117050 2019-12-05T05:17:06.678 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:06.649+0000 7f34af7fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f34b0117050 0x7f34b0123f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:17:06.678 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:06.649+0000 7f34affff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f34b0118880 0x7f34b0118cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:59042/0 (socket says 172.21.15.42:59042) 2019-12-05T05:17:06.679 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:06.649+0000 7f34affff700 1 -- 172.21.15.42:0/3025797960 learned_addr learned my addr 172.21.15.42:0/3025797960 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:17:06.679 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:06.649+0000 7f34aeffd700 1 --2- 172.21.15.42:0/3025797960 >> v2:172.21.15.40:3300/0 conn(0x7f34b0122250 0x7f34b0120830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:17:06.679 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:06.649+0000 7f3497fff700 1 -- 172.21.15.42:0/3025797960 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3204832768 0 0) 0x7f34b00a6c80 con 0x7f34b0118880 2019-12-05T05:17:06.679 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:06.649+0000 7f3497fff700 1 -- 172.21.15.42:0/3025797960 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34900030b0 con 0x7f34b0118880 2019-12-05T05:17:06.679 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:06.649+0000 7f34aeffd700 1 -- 172.21.15.42:0/3025797960 >> v2:172.21.15.42:3300/0 conn(0x7f34b0117050 msgr2=0x7f34b0123f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:17:06.679 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:06.649+0000 7f34aeffd700 1 --2- 172.21.15.42:0/3025797960 >> v2:172.21.15.42:3300/0 conn(0x7f34b0117050 0x7f34b0123f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:17:06.679 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:06.649+0000 7f34aeffd700 1 -- 172.21.15.42:0/3025797960 >> v1:172.21.15.42:6789/0 conn(0x7f34b0118880 legacy=0x7f34b0118cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:17:06.680 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:06.649+0000 7f34aeffd700 1 -- 172.21.15.42:0/3025797960 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34b0120d80 con 0x7f34b0122250 2019-12-05T05:17:06.680 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:06.649+0000 7f34af7fe700 1 --2- 172.21.15.42:0/3025797960 >> v2:172.21.15.42:3300/0 conn(0x7f34b0117050 0x7f34b0123f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:17:06.680 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:06.649+0000 7f34aeffd700 1 --2- 172.21.15.42:0/3025797960 >> v2:172.21.15.40:3300/0 conn(0x7f34b0122250 0x7f34b0120830 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f34b0116cd0 tx=0x7f34a4010b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:06.680 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:06.649+0000 7f34aeffd700 1 --2- 172.21.15.42:0/3025797960 >> v2:172.21.15.40:3300/0 conn(0x7f34b0122250 0x7f34b0120830 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f34b0116cd0 tx=0x7f34a4010b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:06.686 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:06.649+0000 7f34aeffd700 1 --2- 172.21.15.42:0/3025797960 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f34b0122250 0x7f34b0120830 secure :-1 s=READY pgs=481 cs=0 l=1 rx=0x7f34b0116cd0 tx=0x7f34a4010b70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:17:06.686 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:06.649+0000 7f34aeffd700 1 --2- 172.21.15.42:0/3025797960 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f34b0122250 0x7f34b0120830 secure :-1 s=THROTTLE_DONE pgs=481 cs=0 l=1 rx=0x7f34b0116cd0 tx=0x7f34a4010b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:06.687 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:06.649+0000 7f3497fff700 1 -- 172.21.15.42:0/3025797960 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f34a40125e0 con 0x7f34b0122250 2019-12-05T05:17:06.687 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:06.649+0000 7f34aeffd700 1 --2- 172.21.15.42:0/3025797960 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f34b0122250 0x7f34b0120830 secure :-1 s=THROTTLE_DONE pgs=481 cs=0 l=1 rx=0x7f34b0116cd0 tx=0x7f34a4010b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:06.687 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:06.649+0000 7f3497fff700 1 -- 172.21.15.42:0/3025797960 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34a4007150 con 0x7f34b0122250 2019-12-05T05:17:06.687 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:06.649+0000 7f34b55a3700 1 -- 172.21.15.42:0/3025797960 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34b01194b0 con 0x7f34b0122250 2019-12-05T05:17:06.687 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:06.649+0000 7f34aeffd700 1 --2- 172.21.15.42:0/3025797960 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f34b0122250 0x7f34b0120830 secure :-1 s=THROTTLE_DONE pgs=481 cs=0 l=1 rx=0x7f34b0116cd0 tx=0x7f34a4010b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:06.688 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:06.649+0000 7f3497fff700 1 -- 172.21.15.42:0/3025797960 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f34a4012c00 con 0x7f34b0122250 2019-12-05T05:17:06.688 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:06.649+0000 7f34b55a3700 1 -- 172.21.15.42:0/3025797960 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34b0122820 con 0x7f34b0122250 2019-12-05T05:17:06.688 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:06.653+0000 7f34aeffd700 1 --2- 172.21.15.42:0/3025797960 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f34b0122250 0x7f34b0120830 secure :-1 s=THROTTLE_DONE pgs=481 cs=0 l=1 rx=0x7f34b0116cd0 tx=0x7f34a4010b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:06.688 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:06.653+0000 7f3497fff700 1 -- 172.21.15.42:0/3025797960 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f34a4005540 con 0x7f34b0122250 2019-12-05T05:17:06.688 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:06.653+0000 7f34aeffd700 1 --2- 172.21.15.42:0/3025797960 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f34b0122250 0x7f34b0120830 secure :-1 s=THROTTLE_DONE pgs=481 cs=0 l=1 rx=0x7f34b0116cd0 tx=0x7f34a4010b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:06.688 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:06.653+0000 7f3497fff700 1 --2- 172.21.15.42:0/3025797960 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f349003b1e0 0x7f349003d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:17:06.689 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:06.653+0000 7f3497fff700 1 -- 172.21.15.42:0/3025797960 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6805+0+0 (secure 0 0 0) 0x7f34a404b100 con 0x7f34b0122250 2019-12-05T05:17:06.689 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:06.653+0000 7f34af7fe700 1 --2- 172.21.15.42:0/3025797960 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f349003b1e0 0x7f349003d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:17:06.689 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:06.653+0000 7f34af7fe700 1 --2- 172.21.15.42:0/3025797960 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f349003b1e0 0x7f349003d6f0 crc :-1 s=READY pgs=273 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:17:06.689 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:06.657+0000 7f34b55a3700 1 -- 172.21.15.42:0/3025797960 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34b005d110 con 0x7f34b0122250 2019-12-05T05:17:06.689 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:06.665+0000 7f34aeffd700 1 --2- 172.21.15.42:0/3025797960 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f34b0122250 0x7f34b0120830 secure :-1 s=THROTTLE_DONE pgs=481 cs=0 l=1 rx=0x7f34b0116cd0 tx=0x7f34a4010b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:06.690 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:06.665+0000 7f3497fff700 1 -- 172.21.15.42:0/3025797960 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f34a4013060 con 0x7f34b0122250 2019-12-05T05:17:06.839 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:06.833+0000 7f34b55a3700 1 -- 172.21.15.42:0/3025797960 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f34b005b8c0 con 0x7f34b0122250 2019-12-05T05:17:06.840 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:06.837+0000 7f34aeffd700 1 --2- 172.21.15.42:0/3025797960 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f34b0122250 0x7f34b0120830 secure :-1 s=THROTTLE_DONE pgs=481 cs=0 l=1 rx=0x7f34b0116cd0 tx=0x7f34a4010b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:06.840 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:06.837+0000 7f3497fff700 1 -- 172.21.15.42:0/3025797960 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 78 v78) v1 ==== 94+0+2108 (secure 0 0 0) 0x7f34a4032160 con 0x7f34b0122250 2019-12-05T05:17:06.842 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:17:06.842 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":78,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5844,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/2698421979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2698421979},{"type":"v1","addr":"172.21.15.40:6833","nonce":2698421979}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":78}],"filesystems":[{"mdsmap":{"epoch":77,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:15:13.428215+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5629},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5629":{"gid":5629,"name":"a","rank":0,"incarnation":74,"state":"up:active","state_seq":15,"addr":"172.21.15.42:6835/3264829400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3264829400},{"type":"v1","addr":"172.21.15.42:6835","nonce":3264829400}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:17:06.843 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 78 2019-12-05T05:17:06.845 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:06.841+0000 7f34b55a3700 1 -- 172.21.15.42:0/3025797960 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f349003b1e0 msgr2=0x7f349003d6f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:17:06.845 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:06.841+0000 7f34b55a3700 1 --2- 172.21.15.42:0/3025797960 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f349003b1e0 0x7f349003d6f0 crc :-1 s=READY pgs=273 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:17:06.846 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:06.841+0000 7f34b55a3700 1 -- 172.21.15.42:0/3025797960 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f34b0122250 msgr2=0x7f34b0120830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:17:06.846 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:06.841+0000 7f34b55a3700 1 --2- 172.21.15.42:0/3025797960 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f34b0122250 0x7f34b0120830 secure :-1 s=READY pgs=481 cs=0 l=1 rx=0x7f34b0116cd0 tx=0x7f34a4010b70).stop 2019-12-05T05:17:06.846 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:06.841+0000 7f34b55a3700 1 -- 172.21.15.42:0/3025797960 shutdown_connections 2019-12-05T05:17:06.846 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:06.841+0000 7f34b55a3700 1 --2- 172.21.15.42:0/3025797960 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f349003b1e0 0x7f349003d6f0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:17:06.846 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:06.841+0000 7f34b55a3700 1 --2- 172.21.15.42:0/3025797960 >> v2:172.21.15.42:3300/0 conn(0x7f34b0117050 0x7f34b0123f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:17:06.846 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:06.841+0000 7f34b55a3700 1 --2- 172.21.15.42:0/3025797960 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f34b0122250 0x7f34b0120830 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:17:06.847 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:06.841+0000 7f34b55a3700 1 -- 172.21.15.42:0/3025797960 shutdown_connections 2019-12-05T05:17:06.847 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:06.841+0000 7f34b55a3700 1 -- 172.21.15.42:0/3025797960 wait complete. 2019-12-05T05:17:06.847 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:06.841+0000 7f34b55a3700 1 -- 172.21.15.42:0/3025797960 >> 172.21.15.42:0/3025797960 conn(0x7f34b00a7f90 msgr2=0x7f34b0111db0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:17:06.856 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T05:17:08.857 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:17:08.962 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:08.953+0000 7fdff8e9a700 1 Processor -- start 2019-12-05T05:17:08.963 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:08.953+0000 7fdff8e9a700 1 -- start start 2019-12-05T05:17:08.963 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:08.953+0000 7fdff8e9a700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fdff40a7740 0x7fdff408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:17:08.963 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:08.953+0000 7fdff8e9a700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdff407e720 0x7fdff407eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:17:08.963 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:08.953+0000 7fdff8e9a700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdff40a0710 con 0x7fdff409f2e0 2019-12-05T05:17:08.964 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:08.953+0000 7fdff8e9a700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdff40a0b00 con 0x7fdff40a7740 2019-12-05T05:17:08.964 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:08.953+0000 7fdff8e9a700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdff408e5b0 con 0x7fdff407e720 2019-12-05T05:17:08.964 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:08.953+0000 7fdff2d9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fdff40a7740 0x7fdff408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:17:08.964 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:08.953+0000 7fdff2d9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fdff40a7740 0x7fdff408df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:59016/0 (socket says 172.21.15.42:59016) 2019-12-05T05:17:08.964 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:08.953+0000 7fdff2d9d700 1 -- 172.21.15.42:0/1631331935 learned_addr learned my addr 172.21.15.42:0/1631331935 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:17:08.964 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:08.953+0000 7fdff259c700 1 --2- 172.21.15.42:0/1631331935 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdff407e720 0x7fdff407eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:17:08.965 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:08.953+0000 7fdff2d9d700 1 -- 172.21.15.42:0/1631331935 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdff407e720 msgr2=0x7fdff407eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:17:08.965 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:08.953+0000 7fdff2d9d700 1 --2- 172.21.15.42:0/1631331935 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdff407e720 0x7fdff407eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:17:08.965 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:08.953+0000 7fdff2d9d700 1 -- 172.21.15.42:0/1631331935 >> v1:172.21.15.40:6789/0 conn(0x7fdff409f2e0 legacy=0x7fdff409f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:17:08.965 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:08.953+0000 7fdff2d9d700 1 -- 172.21.15.42:0/1631331935 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdff407f0a0 con 0x7fdff40a7740 2019-12-05T05:17:08.965 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:08.953+0000 7fdff2d9d700 1 --2- 172.21.15.42:0/1631331935 >> v2:172.21.15.42:3300/0 conn(0x7fdff40a7740 0x7fdff408df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fdfdc008790 tx=0x7fdfdc00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:08.965 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:08.953+0000 7fdff2d9d700 1 --2- 172.21.15.42:0/1631331935 >> v2:172.21.15.42:3300/0 conn(0x7fdff40a7740 0x7fdff408df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fdfdc008790 tx=0x7fdfdc00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:08.966 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:08.953+0000 7fdff2d9d700 1 --2- 172.21.15.42:0/1631331935 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdff40a7740 0x7fdff408df80 secure :-1 s=READY pgs=613 cs=0 l=1 rx=0x7fdfdc008790 tx=0x7fdfdc00fd30).ready entity=mon.1 client_cookie=c437743e9886ef53 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:17:08.966 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:08.953+0000 7fdff2d9d700 1 --2- 172.21.15.42:0/1631331935 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdff40a7740 0x7fdff408df80 secure :-1 s=THROTTLE_DONE pgs=613 cs=0 l=1 rx=0x7fdfdc008790 tx=0x7fdfdc00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:08.966 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:08.953+0000 7fdff2d9d700 1 --2- 172.21.15.42:0/1631331935 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdff40a7740 0x7fdff408df80 secure :-1 s=THROTTLE_DONE pgs=613 cs=0 l=1 rx=0x7fdfdc008790 tx=0x7fdfdc00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:08.966 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:08.953+0000 7fdff2d9d700 1 --2- 172.21.15.42:0/1631331935 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdff40a7740 0x7fdff408df80 secure :-1 s=THROTTLE_DONE pgs=613 cs=0 l=1 rx=0x7fdfdc008790 tx=0x7fdfdc00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:08.966 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:08.953+0000 7fdff1d9b700 1 -- 172.21.15.42:0/1631331935 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdfdc0113b0 con 0x7fdff40a7740 2019-12-05T05:17:08.966 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:08.953+0000 7fdff1d9b700 1 -- 172.21.15.42:0/1631331935 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfdc008bb0 con 0x7fdff40a7740 2019-12-05T05:17:08.966 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:08.957+0000 7fdff8e9a700 1 -- 172.21.15.42:0/1631331935 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdff40a7740 msgr2=0x7fdff408df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:17:08.967 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:08.957+0000 7fdff8e9a700 1 --2- 172.21.15.42:0/1631331935 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdff40a7740 0x7fdff408df80 secure :-1 s=READY pgs=613 cs=0 l=1 rx=0x7fdfdc008790 tx=0x7fdfdc00fd30).stop 2019-12-05T05:17:08.967 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:08.957+0000 7fdff1d9b700 1 -- 172.21.15.42:0/1631331935 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdfdc011990 con 0x7fdff40a7740 2019-12-05T05:17:08.967 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:08.957+0000 7fdff8e9a700 1 -- 172.21.15.42:0/1631331935 shutdown_connections 2019-12-05T05:17:08.967 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:08.957+0000 7fdff8e9a700 1 --2- 172.21.15.42:0/1631331935 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdff40a7740 0x7fdff408df80 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:17:08.967 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:08.957+0000 7fdff8e9a700 1 --2- 172.21.15.42:0/1631331935 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdff407e720 0x7fdff407eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:17:08.967 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:08.957+0000 7fdff8e9a700 1 -- 172.21.15.42:0/1631331935 shutdown_connections 2019-12-05T05:17:08.967 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:08.957+0000 7fdff8e9a700 1 -- 172.21.15.42:0/1631331935 wait complete. 2019-12-05T05:17:08.968 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:08.957+0000 7fdff8e9a700 1 -- 172.21.15.42:0/1631331935 >> 172.21.15.42:0/1631331935 conn(0x7fdff40aabe0 msgr2=0x7fdff40a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:17:08.968 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:08.957+0000 7fdff8e9a700 1 Processor -- start 2019-12-05T05:17:08.968 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:08.957+0000 7fdff8e9a700 1 -- start start 2019-12-05T05:17:08.971 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:08.957+0000 7fdff8e9a700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fdff40a6a40 0x7fdff41161c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:17:08.971 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:08.957+0000 7fdff8e9a700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fdff40a5ed0 0x7fdff411f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:17:08.971 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:08.957+0000 7fdff8e9a700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdff41170c0 con 0x7fdff408ed10 2019-12-05T05:17:08.971 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:08.957+0000 7fdff8e9a700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdff408f140 con 0x7fdff40a6a40 2019-12-05T05:17:08.971 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:08.957+0000 7fdff8e9a700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdff40a6fc0 con 0x7fdff40a5ed0 2019-12-05T05:17:08.972 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:08.957+0000 7fdff359e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fdff40a5ed0 0x7fdff411f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:17:08.972 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:08.957+0000 7fdff2d9d700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fdff40a6a40 0x7fdff41161c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:17:08.972 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:08.957+0000 7fdff2d9d700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fdff40a6a40 0x7fdff41161c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.42:46834/0 (socket says 172.21.15.42:46834) 2019-12-05T05:17:08.972 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:08.957+0000 7fdff2d9d700 1 -- 172.21.15.42:0/3954615529 learned_addr learned my addr 172.21.15.42:0/3954615529 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:17:08.972 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:08.961+0000 7fdff2d9d700 1 -- 172.21.15.42:0/3954615529 >> v2:172.21.15.42:3300/0 conn(0x7fdff40a5ed0 msgr2=0x7fdff411f2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:17:08.972 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:08.961+0000 7fdff2d9d700 1 --2- 172.21.15.42:0/3954615529 >> v2:172.21.15.42:3300/0 conn(0x7fdff40a5ed0 0x7fdff411f2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:17:08.973 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:08.961+0000 7fdff2d9d700 1 -- 172.21.15.42:0/3954615529 >> v1:172.21.15.40:6789/0 conn(0x7fdff408ed10 legacy=0x7fdff411eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:17:08.973 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:08.961+0000 7fdff2d9d700 1 -- 172.21.15.42:0/3954615529 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdff40a4a50 con 0x7fdff40a6a40 2019-12-05T05:17:08.973 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:08.961+0000 7fdff2d9d700 1 --2- 172.21.15.42:0/3954615529 >> v2:172.21.15.40:3300/0 conn(0x7fdff40a6a40 0x7fdff41161c0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fdfdc008790 tx=0x7fdfdc008bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:08.973 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:08.961+0000 7fdff2d9d700 1 --2- 172.21.15.42:0/3954615529 >> v2:172.21.15.40:3300/0 conn(0x7fdff40a6a40 0x7fdff41161c0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fdfdc008790 tx=0x7fdfdc008bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:08.973 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:08.961+0000 7fdff2d9d700 1 --2- 172.21.15.42:0/3954615529 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdff40a6a40 0x7fdff41161c0 secure :-1 s=READY pgs=484 cs=0 l=1 rx=0x7fdfdc008790 tx=0x7fdfdc008bb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:17:08.973 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:08.961+0000 7fdff2d9d700 1 --2- 172.21.15.42:0/3954615529 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdff40a6a40 0x7fdff41161c0 secure :-1 s=THROTTLE_DONE pgs=484 cs=0 l=1 rx=0x7fdfdc008790 tx=0x7fdfdc008bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:08.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:08.961+0000 7fdfeb7fe700 1 -- 172.21.15.42:0/3954615529 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdfdc007150 con 0x7fdff40a6a40 2019-12-05T05:17:08.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:08.961+0000 7fdff2d9d700 1 --2- 172.21.15.42:0/3954615529 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdff40a6a40 0x7fdff41161c0 secure :-1 s=THROTTLE_DONE pgs=484 cs=0 l=1 rx=0x7fdfdc008790 tx=0x7fdfdc008bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:08.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:08.961+0000 7fdff2d9d700 1 --2- 172.21.15.42:0/3954615529 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdff40a6a40 0x7fdff41161c0 secure :-1 s=THROTTLE_DONE pgs=484 cs=0 l=1 rx=0x7fdfdc008790 tx=0x7fdfdc008bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:08.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:08.961+0000 7fdfeb7fe700 1 -- 172.21.15.42:0/3954615529 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfdc0115b0 con 0x7fdff40a6a40 2019-12-05T05:17:08.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:08.961+0000 7fdfeb7fe700 1 -- 172.21.15.42:0/3954615529 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdfdc00f550 con 0x7fdff40a6a40 2019-12-05T05:17:08.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:08.961+0000 7fdff8e9a700 1 -- 172.21.15.42:0/3954615529 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdff4117b00 con 0x7fdff40a6a40 2019-12-05T05:17:08.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:08.961+0000 7fdff8e9a700 1 -- 172.21.15.42:0/3954615529 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdff411f840 con 0x7fdff40a6a40 2019-12-05T05:17:08.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:08.961+0000 7fdff2d9d700 1 --2- 172.21.15.42:0/3954615529 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdff40a6a40 0x7fdff41161c0 secure :-1 s=THROTTLE_DONE pgs=484 cs=0 l=1 rx=0x7fdfdc008790 tx=0x7fdfdc008bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:08.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:08.961+0000 7fdfeb7fe700 1 -- 172.21.15.42:0/3954615529 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fdfdc005540 con 0x7fdff40a6a40 2019-12-05T05:17:08.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:08.961+0000 7fdff2d9d700 1 --2- 172.21.15.42:0/3954615529 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdff40a6a40 0x7fdff41161c0 secure :-1 s=THROTTLE_DONE pgs=484 cs=0 l=1 rx=0x7fdfdc008790 tx=0x7fdfdc008bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:08.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:08.961+0000 7fdfeb7fe700 1 --2- 172.21.15.42:0/3954615529 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fdfcc03ae20 0x7fdfcc03d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:17:08.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:08.961+0000 7fdfeb7fe700 1 -- 172.21.15.42:0/3954615529 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6805+0+0 (secure 0 0 0) 0x7fdfdc04b3b0 con 0x7fdff40a6a40 2019-12-05T05:17:08.976 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:08.961+0000 7fdff259c700 1 --2- 172.21.15.42:0/3954615529 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fdfcc03ae20 0x7fdfcc03d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:17:08.976 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:08.961+0000 7fdff259c700 1 --2- 172.21.15.42:0/3954615529 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fdfcc03ae20 0x7fdfcc03d330 crc :-1 s=READY pgs=274 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:17:08.976 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:08.969+0000 7fdff8e9a700 1 -- 172.21.15.42:0/3954615529 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdff405d110 con 0x7fdff40a6a40 2019-12-05T05:17:08.983 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:08.981+0000 7fdff2d9d700 1 --2- 172.21.15.42:0/3954615529 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdff40a6a40 0x7fdff41161c0 secure :-1 s=THROTTLE_DONE pgs=484 cs=0 l=1 rx=0x7fdfdc008790 tx=0x7fdfdc008bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:08.984 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:08.981+0000 7fdfeb7fe700 1 -- 172.21.15.42:0/3954615529 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fdfdc032410 con 0x7fdff40a6a40 2019-12-05T05:17:09.150 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:09.145+0000 7fdff8e9a700 1 -- 172.21.15.42:0/3954615529 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdff40a5a10 con 0x7fdff40a6a40 2019-12-05T05:17:09.151 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:09.145+0000 7fdff2d9d700 1 --2- 172.21.15.42:0/3954615529 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdff40a6a40 0x7fdff41161c0 secure :-1 s=THROTTLE_DONE pgs=484 cs=0 l=1 rx=0x7fdfdc008790 tx=0x7fdfdc008bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:09.152 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:09.145+0000 7fdfeb7fe700 1 -- 172.21.15.42:0/3954615529 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 78 v78) v1 ==== 94+0+2108 (secure 0 0 0) 0x7fdfdc0321c0 con 0x7fdff40a6a40 2019-12-05T05:17:09.153 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:17:09.154 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":78,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5844,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/2698421979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2698421979},{"type":"v1","addr":"172.21.15.40:6833","nonce":2698421979}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":78}],"filesystems":[{"mdsmap":{"epoch":77,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:15:13.428215+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5629},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5629":{"gid":5629,"name":"a","rank":0,"incarnation":74,"state":"up:active","state_seq":15,"addr":"172.21.15.42:6835/3264829400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3264829400},{"type":"v1","addr":"172.21.15.42:6835","nonce":3264829400}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:17:09.154 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 78 2019-12-05T05:17:09.156 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:09.149+0000 7fdff8e9a700 1 -- 172.21.15.42:0/3954615529 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fdfcc03ae20 msgr2=0x7fdfcc03d330 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:17:09.156 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:09.149+0000 7fdff8e9a700 1 --2- 172.21.15.42:0/3954615529 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fdfcc03ae20 0x7fdfcc03d330 crc :-1 s=READY pgs=274 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:17:09.156 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:09.149+0000 7fdff8e9a700 1 -- 172.21.15.42:0/3954615529 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdff40a6a40 msgr2=0x7fdff41161c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:17:09.156 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:09.149+0000 7fdff8e9a700 1 --2- 172.21.15.42:0/3954615529 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdff40a6a40 0x7fdff41161c0 secure :-1 s=READY pgs=484 cs=0 l=1 rx=0x7fdfdc008790 tx=0x7fdfdc008bb0).stop 2019-12-05T05:17:09.157 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:09.153+0000 7fdff8e9a700 1 -- 172.21.15.42:0/3954615529 shutdown_connections 2019-12-05T05:17:09.157 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:09.153+0000 7fdff8e9a700 1 --2- 172.21.15.42:0/3954615529 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fdfcc03ae20 0x7fdfcc03d330 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:17:09.157 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:09.153+0000 7fdff8e9a700 1 --2- 172.21.15.42:0/3954615529 >> v2:172.21.15.42:3300/0 conn(0x7fdff40a5ed0 0x7fdff411f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:17:09.157 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:09.153+0000 7fdff8e9a700 1 --2- 172.21.15.42:0/3954615529 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdff40a6a40 0x7fdff41161c0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:17:09.157 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:09.153+0000 7fdff8e9a700 1 -- 172.21.15.42:0/3954615529 shutdown_connections 2019-12-05T05:17:09.157 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:09.153+0000 7fdff8e9a700 1 -- 172.21.15.42:0/3954615529 wait complete. 2019-12-05T05:17:09.157 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:09.153+0000 7fdff8e9a700 1 -- 172.21.15.42:0/3954615529 >> 172.21.15.42:0/3954615529 conn(0x7fdff40a8030 msgr2=0x7fdff4141400 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:17:09.166 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T05:17:11.167 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:17:11.274 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:11.265+0000 7f0cd85c4700 1 Processor -- start 2019-12-05T05:17:11.274 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:11.265+0000 7f0cd85c4700 1 -- start start 2019-12-05T05:17:11.274 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:11.265+0000 7f0cd85c4700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0cd00a7730 0x7f0cd008df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:17:11.275 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:11.265+0000 7f0cd85c4700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f0cd007e710 0x7f0cd007eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:17:11.275 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:11.265+0000 7f0cd85c4700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0cd009f2d0 0x7f0cd009f700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:17:11.275 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:11.265+0000 7f0cd85c4700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0cd00a0af0 con 0x7f0cd007e710 2019-12-05T05:17:11.275 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:11.265+0000 7f0cd85c4700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0cd008e5a0 con 0x7f0cd00a7730 2019-12-05T05:17:11.275 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:11.265+0000 7f0cd85c4700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cd008e840 con 0x7f0cd009f2d0 2019-12-05T05:17:11.275 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:11.265+0000 7f0cd6b61700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0cd00a7730 0x7f0cd008df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:17:11.275 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:11.265+0000 7f0cd6b61700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0cd00a7730 0x7f0cd008df70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:59026/0 (socket says 172.21.15.42:59026) 2019-12-05T05:17:11.276 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:11.265+0000 7f0cd6b61700 1 -- 172.21.15.42:0/2271563523 learned_addr learned my addr 172.21.15.42:0/2271563523 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:17:11.276 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:11.265+0000 7f0cd7362700 1 --2- 172.21.15.42:0/2271563523 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0cd009f2d0 0x7f0cd009f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:17:11.276 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:11.265+0000 7f0cd6360700 1 --2- 172.21.15.42:0/2271563523 >> v2:172.21.15.40:3300/0 conn(0x7f0cd007e710 0x7f0cd007eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:17:11.276 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:11.265+0000 7f0cd6b61700 1 -- 172.21.15.42:0/2271563523 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0cd009f2d0 msgr2=0x7f0cd009f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:17:11.276 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:11.265+0000 7f0cd6b61700 1 --2- 172.21.15.42:0/2271563523 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0cd009f2d0 0x7f0cd009f700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:17:11.277 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:11.265+0000 7f0cd6b61700 1 -- 172.21.15.42:0/2271563523 >> v2:172.21.15.40:3300/0 conn(0x7f0cd007e710 msgr2=0x7f0cd007eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:17:11.277 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:11.265+0000 7f0cd6b61700 1 --2- 172.21.15.42:0/2271563523 >> v2:172.21.15.40:3300/0 conn(0x7f0cd007e710 0x7f0cd007eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:17:11.277 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:11.265+0000 7f0cd6b61700 1 -- 172.21.15.42:0/2271563523 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cd007f090 con 0x7f0cd00a7730 2019-12-05T05:17:11.277 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:11.265+0000 7f0cd6360700 1 --2- 172.21.15.42:0/2271563523 >> v2:172.21.15.40:3300/0 conn(0x7f0cd007e710 0x7f0cd007eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:17:11.277 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:11.265+0000 7f0cd6b61700 1 --2- 172.21.15.42:0/2271563523 >> v2:172.21.15.42:3300/0 conn(0x7f0cd00a7730 0x7f0cd008df70 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f0cd00a0f40 tx=0x7f0ccc00fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:11.277 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:11.265+0000 7f0cd6b61700 1 --2- 172.21.15.42:0/2271563523 >> v2:172.21.15.42:3300/0 conn(0x7f0cd00a7730 0x7f0cd008df70 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f0cd00a0f40 tx=0x7f0ccc00fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:11.277 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:11.265+0000 7f0cd6b61700 1 --2- 172.21.15.42:0/2271563523 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0cd00a7730 0x7f0cd008df70 secure :-1 s=READY pgs=614 cs=0 l=1 rx=0x7f0cd00a0f40 tx=0x7f0ccc00fdb0).ready entity=mon.1 client_cookie=f86fcdf086b769fd server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:17:11.278 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:11.269+0000 7f0cd6b61700 1 --2- 172.21.15.42:0/2271563523 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0cd00a7730 0x7f0cd008df70 secure :-1 s=THROTTLE_DONE pgs=614 cs=0 l=1 rx=0x7f0cd00a0f40 tx=0x7f0ccc00fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:11.278 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:11.269+0000 7f0cd6b61700 1 --2- 172.21.15.42:0/2271563523 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0cd00a7730 0x7f0cd008df70 secure :-1 s=THROTTLE_DONE pgs=614 cs=0 l=1 rx=0x7f0cd00a0f40 tx=0x7f0ccc00fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:11.278 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:11.269+0000 7f0cd5b5f700 1 -- 172.21.15.42:0/2271563523 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0ccc008c90 con 0x7f0cd00a7730 2019-12-05T05:17:11.278 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:11.269+0000 7f0cd6b61700 1 --2- 172.21.15.42:0/2271563523 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0cd00a7730 0x7f0cd008df70 secure :-1 s=THROTTLE_DONE pgs=614 cs=0 l=1 rx=0x7f0cd00a0f40 tx=0x7f0ccc00fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:11.278 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:11.269+0000 7f0cd5b5f700 1 -- 172.21.15.42:0/2271563523 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ccc0088b0 con 0x7f0cd00a7730 2019-12-05T05:17:11.278 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:11.269+0000 7f0cd5b5f700 1 -- 172.21.15.42:0/2271563523 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0ccc011b50 con 0x7f0cd00a7730 2019-12-05T05:17:11.279 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:11.269+0000 7f0cd85c4700 1 -- 172.21.15.42:0/2271563523 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0cd00a7730 msgr2=0x7f0cd008df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:17:11.279 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:11.269+0000 7f0cd85c4700 1 --2- 172.21.15.42:0/2271563523 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0cd00a7730 0x7f0cd008df70 secure :-1 s=READY pgs=614 cs=0 l=1 rx=0x7f0cd00a0f40 tx=0x7f0ccc00fdb0).stop 2019-12-05T05:17:11.279 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:11.269+0000 7f0cd85c4700 1 -- 172.21.15.42:0/2271563523 shutdown_connections 2019-12-05T05:17:11.279 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:11.269+0000 7f0cd85c4700 1 --2- 172.21.15.42:0/2271563523 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0cd009f2d0 0x7f0cd009f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:17:11.279 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:11.269+0000 7f0cd85c4700 1 --2- 172.21.15.42:0/2271563523 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0cd00a7730 0x7f0cd008df70 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:17:11.279 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:11.269+0000 7f0cd85c4700 1 --2- 172.21.15.42:0/2271563523 >> v2:172.21.15.40:3300/0 conn(0x7f0cd007e710 0x7f0cd007eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:17:11.279 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:11.269+0000 7f0cd85c4700 1 -- 172.21.15.42:0/2271563523 shutdown_connections 2019-12-05T05:17:11.280 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:11.269+0000 7f0cd85c4700 1 -- 172.21.15.42:0/2271563523 wait complete. 2019-12-05T05:17:11.280 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:11.269+0000 7f0cd85c4700 1 -- 172.21.15.42:0/2271563523 >> 172.21.15.42:0/2271563523 conn(0x7f0cd00aabd0 msgr2=0x7f0cd00a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:17:11.284 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:11.269+0000 7f0cd85c4700 1 Processor -- start 2019-12-05T05:17:11.284 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:11.269+0000 7f0cd85c4700 1 -- start start 2019-12-05T05:17:11.284 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:11.273+0000 7f0cd85c4700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0cd01483b0 0x7f0cd0135fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:17:11.284 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:11.273+0000 7f0cd85c4700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cd0142b50 con 0x7f0cd0143400 2019-12-05T05:17:11.284 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:11.273+0000 7f0cd85c4700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cd0142520 con 0x7f0cd0129db0 2019-12-05T05:17:11.284 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:11.273+0000 7f0cd85c4700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cd0143bd0 con 0x7f0cd01483b0 2019-12-05T05:17:11.285 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:11.273+0000 7f0cd6b61700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f0cd0129db0 0x7f0cd0123c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:59070/0 (socket says 172.21.15.42:59070) 2019-12-05T05:17:11.285 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:11.273+0000 7f0cd6b61700 1 -- 172.21.15.42:0/2992867465 learned_addr learned my addr 172.21.15.42:0/2992867465 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:17:11.285 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:11.273+0000 7f0cd7362700 1 --2- 172.21.15.42:0/2992867465 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0cd01483b0 0x7f0cd0135fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:17:11.285 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:11.273+0000 7f0cd6360700 1 -- 172.21.15.42:0/2992867465 >> v1:172.21.15.40:6789/0 conn(0x7f0cd0143400 legacy=0x7f0cd01327b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2019-12-05T05:17:11.285 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:11.273+0000 7f0cd6360700 1 -- 172.21.15.42:0/2992867465 >> v1:172.21.15.40:6789/0 conn(0x7f0cd0143400 legacy=0x7f0cd01327b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2019-12-05T05:17:11.285 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:11.273+0000 7f0cd6360700 1 --1- 172.21.15.42:0/2992867465 >> v1:172.21.15.40:6789/0 conn(0x7f0cd0143400 0x7f0cd01327b0 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_1 read connect reply failed 2019-12-05T05:17:11.285 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:11.273+0000 7f0cd6360700 1 --1- 172.21.15.42:0/2992867465 >> v1:172.21.15.40:6789/0 conn(0x7f0cd0143400 0x7f0cd01327b0 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).fault on lossy channel, failing 2019-12-05T05:17:11.286 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:11.273+0000 7f0cc37fe700 1 -- 172.21.15.42:0/2992867465 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 218940233 0 0) 0x7f0cd0142520 con 0x7f0cd0129db0 2019-12-05T05:17:11.286 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:11.273+0000 7f0cc37fe700 1 -- 172.21.15.42:0/2992867465 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0cb80030b0 con 0x7f0cd0129db0 2019-12-05T05:17:11.286 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:11.273+0000 7f0cd7362700 1 -- 172.21.15.42:0/2992867465 >> v1:172.21.15.42:6789/0 conn(0x7f0cd0129db0 legacy=0x7f0cd0123c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:17:11.286 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:11.273+0000 7f0cd7362700 1 -- 172.21.15.42:0/2992867465 >> v1:172.21.15.40:6789/0 conn(0x7f0cd0143400 legacy=0x7f0cd01327b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2019-12-05T05:17:11.286 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:11.273+0000 7f0cd7362700 1 -- 172.21.15.42:0/2992867465 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cd0145180 con 0x7f0cd01483b0 2019-12-05T05:17:11.286 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:11.273+0000 7f0cd7362700 1 --2- 172.21.15.42:0/2992867465 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0cd01483b0 0x7f0cd0135fd0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f0cc80032b0 tx=0x7f0cc8010c80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:11.287 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:11.273+0000 7f0cd7362700 1 --2- 172.21.15.42:0/2992867465 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0cd01483b0 0x7f0cd0135fd0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f0cc80032b0 tx=0x7f0cc8010c80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:11.287 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:11.273+0000 7f0cd7362700 1 --2- 172.21.15.42:0/2992867465 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0cd01483b0 0x7f0cd0135fd0 secure :-1 s=READY pgs=123 cs=0 l=1 rx=0x7f0cc80032b0 tx=0x7f0cc8010c80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:17:11.287 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:11.273+0000 7f0cd7362700 1 --2- 172.21.15.42:0/2992867465 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0cd01483b0 0x7f0cd0135fd0 secure :-1 s=THROTTLE_DONE pgs=123 cs=0 l=1 rx=0x7f0cc80032b0 tx=0x7f0cc8010c80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:11.287 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:11.273+0000 7f0cc37fe700 1 -- 172.21.15.42:0/2992867465 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0cc8012a00 con 0x7f0cd01483b0 2019-12-05T05:17:11.287 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:11.273+0000 7f0cd7362700 1 --2- 172.21.15.42:0/2992867465 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0cd01483b0 0x7f0cd0135fd0 secure :-1 s=THROTTLE_DONE pgs=123 cs=0 l=1 rx=0x7f0cc80032b0 tx=0x7f0cc8010c80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:11.287 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:11.273+0000 7f0cd7362700 1 --2- 172.21.15.42:0/2992867465 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0cd01483b0 0x7f0cd0135fd0 secure :-1 s=THROTTLE_DONE pgs=123 cs=0 l=1 rx=0x7f0cc80032b0 tx=0x7f0cc8010c80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:11.287 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:11.273+0000 7f0cc37fe700 1 -- 172.21.15.42:0/2992867465 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cc8012620 con 0x7f0cd01483b0 2019-12-05T05:17:11.288 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:11.273+0000 7f0cd85c4700 1 -- 172.21.15.42:0/2992867465 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cd0124400 con 0x7f0cd01483b0 2019-12-05T05:17:11.288 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:11.273+0000 7f0cc37fe700 1 -- 172.21.15.42:0/2992867465 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0cc800e570 con 0x7f0cd01483b0 2019-12-05T05:17:11.288 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:11.273+0000 7f0cd85c4700 1 -- 172.21.15.42:0/2992867465 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cd012a1e0 con 0x7f0cd01483b0 2019-12-05T05:17:11.288 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:11.273+0000 7f0cd7362700 1 --2- 172.21.15.42:0/2992867465 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0cd01483b0 0x7f0cd0135fd0 secure :-1 s=THROTTLE_DONE pgs=123 cs=0 l=1 rx=0x7f0cc80032b0 tx=0x7f0cc8010c80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:11.288 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:11.277+0000 7f0cc37fe700 1 -- 172.21.15.42:0/2992867465 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f0cc8013060 con 0x7f0cd01483b0 2019-12-05T05:17:11.288 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:11.277+0000 7f0cd7362700 1 --2- 172.21.15.42:0/2992867465 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0cd01483b0 0x7f0cd0135fd0 secure :-1 s=THROTTLE_DONE pgs=123 cs=0 l=1 rx=0x7f0cc80032b0 tx=0x7f0cc8010c80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:11.288 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:11.277+0000 7f0cc37fe700 1 --2- 172.21.15.42:0/2992867465 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0cb803b210 0x7f0cb803d720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:17:11.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:11.277+0000 7f0cc37fe700 1 -- 172.21.15.42:0/2992867465 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6805+0+0 (secure 0 0 0) 0x7f0cc804b2a0 con 0x7f0cd01483b0 2019-12-05T05:17:11.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:11.277+0000 7f0cd6b61700 1 --2- 172.21.15.42:0/2992867465 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0cb803b210 0x7f0cb803d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:17:11.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:11.277+0000 7f0cd6b61700 1 --2- 172.21.15.42:0/2992867465 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0cb803b210 0x7f0cb803d720 crc :-1 s=READY pgs=275 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:17:11.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:11.281+0000 7f0cd85c4700 1 -- 172.21.15.42:0/2992867465 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cd005d100 con 0x7f0cd01483b0 2019-12-05T05:17:11.295 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:11.289+0000 7f0cd7362700 1 --2- 172.21.15.42:0/2992867465 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0cd01483b0 0x7f0cd0135fd0 secure :-1 s=THROTTLE_DONE pgs=123 cs=0 l=1 rx=0x7f0cc80032b0 tx=0x7f0cc8010c80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:11.296 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:11.289+0000 7f0cc37fe700 1 -- 172.21.15.42:0/2992867465 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f0cc8032320 con 0x7f0cd01483b0 2019-12-05T05:17:11.464 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:11.461+0000 7f0cd85c4700 1 -- 172.21.15.42:0/2992867465 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0cd005b8b0 con 0x7f0cd01483b0 2019-12-05T05:17:11.465 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:11.461+0000 7f0cd7362700 1 --2- 172.21.15.42:0/2992867465 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0cd01483b0 0x7f0cd0135fd0 secure :-1 s=THROTTLE_DONE pgs=123 cs=0 l=1 rx=0x7f0cc80032b0 tx=0x7f0cc8010c80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:11.465 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:11.461+0000 7f0cc37fe700 1 -- 172.21.15.42:0/2992867465 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 78 v78) v1 ==== 94+0+2108 (secure 0 0 0) 0x7f0cc8032580 con 0x7f0cd01483b0 2019-12-05T05:17:11.467 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:17:11.468 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":78,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5844,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/2698421979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2698421979},{"type":"v1","addr":"172.21.15.40:6833","nonce":2698421979}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":78}],"filesystems":[{"mdsmap":{"epoch":77,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:15:13.428215+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5629},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5629":{"gid":5629,"name":"a","rank":0,"incarnation":74,"state":"up:active","state_seq":15,"addr":"172.21.15.42:6835/3264829400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3264829400},{"type":"v1","addr":"172.21.15.42:6835","nonce":3264829400}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:17:11.468 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 78 2019-12-05T05:17:11.470 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:11.465+0000 7f0cd85c4700 1 -- 172.21.15.42:0/2992867465 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0cb803b210 msgr2=0x7f0cb803d720 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:17:11.470 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:11.465+0000 7f0cd85c4700 1 --2- 172.21.15.42:0/2992867465 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0cb803b210 0x7f0cb803d720 crc :-1 s=READY pgs=275 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:17:11.470 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:11.465+0000 7f0cd85c4700 1 -- 172.21.15.42:0/2992867465 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0cd01483b0 msgr2=0x7f0cd0135fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:17:11.470 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:11.465+0000 7f0cd85c4700 1 --2- 172.21.15.42:0/2992867465 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0cd01483b0 0x7f0cd0135fd0 secure :-1 s=READY pgs=123 cs=0 l=1 rx=0x7f0cc80032b0 tx=0x7f0cc8010c80).stop 2019-12-05T05:17:11.471 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:11.465+0000 7f0cd85c4700 1 -- 172.21.15.42:0/2992867465 shutdown_connections 2019-12-05T05:17:11.471 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:11.465+0000 7f0cd85c4700 1 --2- 172.21.15.42:0/2992867465 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0cb803b210 0x7f0cb803d720 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:17:11.471 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:11.465+0000 7f0cd85c4700 1 --2- 172.21.15.42:0/2992867465 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0cd01483b0 0x7f0cd0135fd0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:17:11.471 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:11.465+0000 7f0cd85c4700 1 -- 172.21.15.42:0/2992867465 shutdown_connections 2019-12-05T05:17:11.471 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:11.465+0000 7f0cd85c4700 1 -- 172.21.15.42:0/2992867465 wait complete. 2019-12-05T05:17:11.472 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:11.465+0000 7f0cd85c4700 1 -- 172.21.15.42:0/2992867465 >> 172.21.15.42:0/2992867465 conn(0x7f0cd00aabd0 msgr2=0x7f0cd007c710 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:17:11.479 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T05:17:13.481 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:17:13.589 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.581+0000 7fb61317e700 1 Processor -- start 2019-12-05T05:17:13.589 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.581+0000 7fb61317e700 1 -- start start 2019-12-05T05:17:13.589 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.581+0000 7fb61317e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb60c0a7740 0x7fb60c08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:17:13.589 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.581+0000 7fb61317e700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb60c07e720 0x7fb60c07eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:17:13.589 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.581+0000 7fb61317e700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb60c09f2e0 0x7fb60c09f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:17:13.589 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.581+0000 7fb61317e700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb60c0a0b00 con 0x7fb60c09f2e0 2019-12-05T05:17:13.590 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.581+0000 7fb61317e700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb60c08e5b0 con 0x7fb60c0a7740 2019-12-05T05:17:13.590 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.581+0000 7fb61317e700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb60c08e850 con 0x7fb60c07e720 2019-12-05T05:17:13.590 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.581+0000 7fb61171b700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb60c0a7740 0x7fb60c08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:17:13.590 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.581+0000 7fb610f1a700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb60c07e720 0x7fb60c07eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:17:13.590 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.581+0000 7fb61171b700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb60c0a7740 0x7fb60c08df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:59042/0 (socket says 172.21.15.42:59042) 2019-12-05T05:17:13.590 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.581+0000 7fb61171b700 1 -- 172.21.15.42:0/2653287799 learned_addr learned my addr 172.21.15.42:0/2653287799 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:17:13.590 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.581+0000 7fb611f1c700 1 --2- 172.21.15.42:0/2653287799 >> v2:172.21.15.40:3300/0 conn(0x7fb60c09f2e0 0x7fb60c09f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:17:13.591 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.581+0000 7fb611f1c700 1 -- 172.21.15.42:0/2653287799 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb60c07e720 msgr2=0x7fb60c07eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:17:13.591 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.581+0000 7fb611f1c700 1 --2- 172.21.15.42:0/2653287799 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb60c07e720 0x7fb60c07eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:17:13.591 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.581+0000 7fb611f1c700 1 -- 172.21.15.42:0/2653287799 >> v2:172.21.15.42:3300/0 conn(0x7fb60c0a7740 msgr2=0x7fb60c08df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:17:13.591 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.581+0000 7fb611f1c700 1 --2- 172.21.15.42:0/2653287799 >> v2:172.21.15.42:3300/0 conn(0x7fb60c0a7740 0x7fb60c08df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:17:13.591 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.581+0000 7fb611f1c700 1 -- 172.21.15.42:0/2653287799 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb60c07f0a0 con 0x7fb60c09f2e0 2019-12-05T05:17:13.591 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.581+0000 7fb61171b700 1 --2- 172.21.15.42:0/2653287799 >> v2:172.21.15.42:3300/0 conn(0x7fb60c0a7740 0x7fb60c08df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:17:13.592 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.581+0000 7fb611f1c700 1 --2- 172.21.15.42:0/2653287799 >> v2:172.21.15.40:3300/0 conn(0x7fb60c09f2e0 0x7fb60c09f710 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fb60c08e770 tx=0x7fb604010070).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:13.592 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.581+0000 7fb611f1c700 1 --2- 172.21.15.42:0/2653287799 >> v2:172.21.15.40:3300/0 conn(0x7fb60c09f2e0 0x7fb60c09f710 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fb60c08e770 tx=0x7fb604010070).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:13.592 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.581+0000 7fb611f1c700 1 --2- 172.21.15.42:0/2653287799 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb60c09f2e0 0x7fb60c09f710 secure :-1 s=READY pgs=486 cs=0 l=1 rx=0x7fb60c08e770 tx=0x7fb604010070).ready entity=mon.0 client_cookie=2fb36be59a145511 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:17:13.592 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.581+0000 7fb611f1c700 1 --2- 172.21.15.42:0/2653287799 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb60c09f2e0 0x7fb60c09f710 secure :-1 s=THROTTLE_DONE pgs=486 cs=0 l=1 rx=0x7fb60c08e770 tx=0x7fb604010070).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:13.592 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.581+0000 7fb611f1c700 1 --2- 172.21.15.42:0/2653287799 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb60c09f2e0 0x7fb60c09f710 secure :-1 s=THROTTLE_DONE pgs=486 cs=0 l=1 rx=0x7fb60c08e770 tx=0x7fb604010070).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:13.592 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.581+0000 7fb603fff700 1 -- 172.21.15.42:0/2653287799 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb604010940 con 0x7fb60c09f2e0 2019-12-05T05:17:13.592 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.581+0000 7fb611f1c700 1 --2- 172.21.15.42:0/2653287799 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb60c09f2e0 0x7fb60c09f710 secure :-1 s=THROTTLE_DONE pgs=486 cs=0 l=1 rx=0x7fb60c08e770 tx=0x7fb604010070).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:13.593 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.581+0000 7fb603fff700 1 -- 172.21.15.42:0/2653287799 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb604008b50 con 0x7fb60c09f2e0 2019-12-05T05:17:13.593 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.581+0000 7fb603fff700 1 -- 172.21.15.42:0/2653287799 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb6040110c0 con 0x7fb60c09f2e0 2019-12-05T05:17:13.593 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.581+0000 7fb61317e700 1 -- 172.21.15.42:0/2653287799 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb60c09f2e0 msgr2=0x7fb60c09f710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:17:13.593 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.581+0000 7fb61317e700 1 --2- 172.21.15.42:0/2653287799 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb60c09f2e0 0x7fb60c09f710 secure :-1 s=READY pgs=486 cs=0 l=1 rx=0x7fb60c08e770 tx=0x7fb604010070).stop 2019-12-05T05:17:13.593 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.585+0000 7fb61317e700 1 -- 172.21.15.42:0/2653287799 shutdown_connections 2019-12-05T05:17:13.594 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.585+0000 7fb61317e700 1 --2- 172.21.15.42:0/2653287799 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb60c09f2e0 0x7fb60c09f710 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:17:13.594 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.585+0000 7fb61317e700 1 --2- 172.21.15.42:0/2653287799 >> v2:172.21.15.42:3300/0 conn(0x7fb60c0a7740 0x7fb60c08df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:17:13.594 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.585+0000 7fb61317e700 1 --2- 172.21.15.42:0/2653287799 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb60c07e720 0x7fb60c07eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:17:13.594 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.585+0000 7fb61317e700 1 -- 172.21.15.42:0/2653287799 shutdown_connections 2019-12-05T05:17:13.595 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.585+0000 7fb61317e700 1 -- 172.21.15.42:0/2653287799 wait complete. 2019-12-05T05:17:13.595 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.585+0000 7fb61317e700 1 -- 172.21.15.42:0/2653287799 >> 172.21.15.42:0/2653287799 conn(0x7fb60c0aabe0 msgr2=0x7fb60c0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:17:13.599 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.585+0000 7fb61317e700 1 Processor -- start 2019-12-05T05:17:13.599 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.585+0000 7fb61317e700 1 -- start start 2019-12-05T05:17:13.599 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.585+0000 7fb61317e700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb60c0a5d10 0x7fb60c1387f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:17:13.599 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.585+0000 7fb61317e700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb60c11f890 con 0x7fb60c11cef0 2019-12-05T05:17:13.599 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.589+0000 7fb61317e700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb60c11f000 con 0x7fb60c117010 2019-12-05T05:17:13.600 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.589+0000 7fb61317e700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb60c1175c0 con 0x7fb60c0a5d10 2019-12-05T05:17:13.600 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.589+0000 7fb610f1a700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fb60c117010 0x7fb60c0a5510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:59088/0 (socket says 172.21.15.42:59088) 2019-12-05T05:17:13.600 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.589+0000 7fb610f1a700 1 -- 172.21.15.42:0/2517455770 learned_addr learned my addr 172.21.15.42:0/2517455770 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:17:13.600 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.589+0000 7fb611f1c700 1 --2- 172.21.15.42:0/2517455770 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb60c0a5d10 0x7fb60c1387f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:17:13.600 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.589+0000 7fb610f1a700 1 -- 172.21.15.42:0/2517455770 >> v1:172.21.15.42:6789/0 conn(0x7fb60c117010 legacy=0x7fb60c0a5510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2019-12-05T05:17:13.600 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.589+0000 7fb610f1a700 1 -- 172.21.15.42:0/2517455770 >> v1:172.21.15.42:6789/0 conn(0x7fb60c117010 legacy=0x7fb60c0a5510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2019-12-05T05:17:13.601 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.589+0000 7fb610f1a700 1 --1- 172.21.15.42:0/2517455770 >> v1:172.21.15.42:6789/0 conn(0x7fb60c117010 0x7fb60c0a5510 :-1 s=OPENED pgs=616 cs=1 l=1).handle_message read tag failed 2019-12-05T05:17:13.601 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.589+0000 7fb610f1a700 1 --1- 172.21.15.42:0/2517455770 >> v1:172.21.15.42:6789/0 conn(0x7fb60c117010 0x7fb60c0a5510 :-1 s=OPENED pgs=616 cs=1 l=1).fault on lossy channel, failing 2019-12-05T05:17:13.601 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.589+0000 7fb601ffb700 1 -- 172.21.15.42:0/2517455770 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3589776980 0 0) 0x7fb60c11f890 con 0x7fb60c11cef0 2019-12-05T05:17:13.601 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.589+0000 7fb601ffb700 1 -- 172.21.15.42:0/2517455770 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb5ec0030b0 con 0x7fb60c11cef0 2019-12-05T05:17:13.601 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.589+0000 7fb611f1c700 1 -- 172.21.15.42:0/2517455770 >> v1:172.21.15.42:6789/0 conn(0x7fb60c117010 legacy=0x7fb60c0a5510 unknown :-1 s=STATE_CLOSED l=1).mark_down 2019-12-05T05:17:13.601 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.589+0000 7fb611f1c700 1 -- 172.21.15.42:0/2517455770 >> v1:172.21.15.40:6789/0 conn(0x7fb60c11cef0 legacy=0x7fb60c07d8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:17:13.602 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.589+0000 7fb611f1c700 1 -- 172.21.15.42:0/2517455770 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb60c120a70 con 0x7fb60c0a5d10 2019-12-05T05:17:13.602 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.589+0000 7fb611f1c700 1 --2- 172.21.15.42:0/2517455770 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb60c0a5d10 0x7fb60c1387f0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fb60c08e770 tx=0x7fb604010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:13.602 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.589+0000 7fb611f1c700 1 --2- 172.21.15.42:0/2517455770 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb60c0a5d10 0x7fb60c1387f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fb60c08e770 tx=0x7fb604010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:13.602 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.589+0000 7fb611f1c700 1 --2- 172.21.15.42:0/2517455770 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb60c0a5d10 0x7fb60c1387f0 secure :-1 s=READY pgs=124 cs=0 l=1 rx=0x7fb60c08e770 tx=0x7fb604010ac0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:17:13.602 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.589+0000 7fb611f1c700 1 --2- 172.21.15.42:0/2517455770 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb60c0a5d10 0x7fb60c1387f0 secure :-1 s=THROTTLE_DONE pgs=124 cs=0 l=1 rx=0x7fb60c08e770 tx=0x7fb604010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:13.602 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.589+0000 7fb601ffb700 1 -- 172.21.15.42:0/2517455770 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb604005ab0 con 0x7fb60c0a5d10 2019-12-05T05:17:13.603 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.589+0000 7fb611f1c700 1 --2- 172.21.15.42:0/2517455770 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb60c0a5d10 0x7fb60c1387f0 secure :-1 s=THROTTLE_DONE pgs=124 cs=0 l=1 rx=0x7fb60c08e770 tx=0x7fb604010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:13.603 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.589+0000 7fb611f1c700 1 --2- 172.21.15.42:0/2517455770 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb60c0a5d10 0x7fb60c1387f0 secure :-1 s=THROTTLE_DONE pgs=124 cs=0 l=1 rx=0x7fb60c08e770 tx=0x7fb604010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:13.603 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.589+0000 7fb601ffb700 1 -- 172.21.15.42:0/2517455770 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb604010b40 con 0x7fb60c0a5d10 2019-12-05T05:17:13.603 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.589+0000 7fb601ffb700 1 -- 172.21.15.42:0/2517455770 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb60400f470 con 0x7fb60c0a5d10 2019-12-05T05:17:13.603 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.589+0000 7fb61317e700 1 -- 172.21.15.42:0/2517455770 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb60c138d40 con 0x7fb60c0a5d10 2019-12-05T05:17:13.604 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.589+0000 7fb61317e700 1 -- 172.21.15.42:0/2517455770 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb60c139290 con 0x7fb60c0a5d10 2019-12-05T05:17:13.604 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.589+0000 7fb611f1c700 1 --2- 172.21.15.42:0/2517455770 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb60c0a5d10 0x7fb60c1387f0 secure :-1 s=THROTTLE_DONE pgs=124 cs=0 l=1 rx=0x7fb60c08e770 tx=0x7fb604010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:13.604 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.589+0000 7fb601ffb700 1 -- 172.21.15.42:0/2517455770 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fb60400f670 con 0x7fb60c0a5d10 2019-12-05T05:17:13.604 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.589+0000 7fb611f1c700 1 --2- 172.21.15.42:0/2517455770 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb60c0a5d10 0x7fb60c1387f0 secure :-1 s=THROTTLE_DONE pgs=124 cs=0 l=1 rx=0x7fb60c08e770 tx=0x7fb604010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:13.604 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.589+0000 7fb601ffb700 1 --2- 172.21.15.42:0/2517455770 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb5ec03b230 0x7fb5ec03d740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:17:13.604 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.593+0000 7fb601ffb700 1 -- 172.21.15.42:0/2517455770 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6891+0+0 (secure 0 0 0) 0x7fb60408ede0 con 0x7fb60c0a5d10 2019-12-05T05:17:13.605 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.593+0000 7fb61171b700 1 --2- 172.21.15.42:0/2517455770 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb5ec03b230 0x7fb5ec03d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:17:13.605 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.593+0000 7fb61171b700 1 -- 172.21.15.42:0/2517455770 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb5ec03b230 msgr2=0x7fb5ec03d740 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2019-12-05T05:17:13.605 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.593+0000 7fb61171b700 1 -- 172.21.15.42:0/2517455770 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb5ec03b230 msgr2=0x7fb5ec03d740 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2019-12-05T05:17:13.605 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.593+0000 7fb61171b700 1 --2- 172.21.15.42:0/2517455770 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb5ec03b230 0x7fb5ec03d740 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-12-05T05:17:13.605 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.593+0000 7fb61171b700 1 --2- 172.21.15.42:0/2517455770 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb5ec03b230 0x7fb5ec03d740 crc :-1 s=START_CONNECT pgs=0 cs=0 l=1 rx=0 tx=0)._fault waiting 0.200000 2019-12-05T05:17:13.606 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.597+0000 7fb61317e700 1 -- 172.21.15.42:0/2517455770 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb60c117780 con 0x7fb60c0a5d10 2019-12-05T05:17:13.607 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.601+0000 7fb611f1c700 1 --2- 172.21.15.42:0/2517455770 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb60c0a5d10 0x7fb60c1387f0 secure :-1 s=THROTTLE_DONE pgs=124 cs=0 l=1 rx=0x7fb60c08e770 tx=0x7fb604010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:13.607 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.601+0000 7fb601ffb700 1 -- 172.21.15.42:0/2517455770 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fb604075e60 con 0x7fb60c0a5d10 2019-12-05T05:17:13.769 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.765+0000 7fb61317e700 1 -- 172.21.15.42:0/2517455770 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb60c05d110 con 0x7fb60c0a5d10 2019-12-05T05:17:13.769 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.765+0000 7fb611f1c700 1 --2- 172.21.15.42:0/2517455770 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb60c0a5d10 0x7fb60c1387f0 secure :-1 s=THROTTLE_DONE pgs=124 cs=0 l=1 rx=0x7fb60c08e770 tx=0x7fb604010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:13.770 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.765+0000 7fb601ffb700 1 -- 172.21.15.42:0/2517455770 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) v1 ==== 94+0+1771 (secure 0 0 0) 0x7fb6040082d0 con 0x7fb60c0a5d10 2019-12-05T05:17:13.770 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 79 2019-12-05T05:17:13.770 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:17:13.771 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":79,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":79,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:17:13.420087+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5844},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5844":{"gid":5844,"name":"b","rank":0,"incarnation":79,"state":"up:replay","state_seq":2,"addr":"172.21.15.40:6833/2698421979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2698421979},{"type":"v1","addr":"172.21.15.40:6833","nonce":2698421979}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:17:13.773 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.769+0000 7fb61317e700 1 -- 172.21.15.42:0/2517455770 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb5ec03b230 msgr2=0x7fb5ec03d740 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2019-12-05T05:17:13.773 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.769+0000 7fb61317e700 1 --2- 172.21.15.42:0/2517455770 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb5ec03b230 0x7fb5ec03d740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:17:13.773 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.769+0000 7fb61317e700 1 -- 172.21.15.42:0/2517455770 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb60c0a5d10 msgr2=0x7fb60c1387f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:17:13.773 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.769+0000 7fb61317e700 1 --2- 172.21.15.42:0/2517455770 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb60c0a5d10 0x7fb60c1387f0 secure :-1 s=READY pgs=124 cs=0 l=1 rx=0x7fb60c08e770 tx=0x7fb604010ac0).stop 2019-12-05T05:17:13.773 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.769+0000 7fb61317e700 1 -- 172.21.15.42:0/2517455770 shutdown_connections 2019-12-05T05:17:13.773 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.769+0000 7fb61317e700 1 --2- 172.21.15.42:0/2517455770 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb5ec03b230 0x7fb5ec03d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:17:13.774 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.769+0000 7fb61317e700 1 --2- 172.21.15.42:0/2517455770 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb60c0a5d10 0x7fb60c1387f0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:17:13.774 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.769+0000 7fb61317e700 1 -- 172.21.15.42:0/2517455770 shutdown_connections 2019-12-05T05:17:13.774 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.769+0000 7fb61317e700 1 -- 172.21.15.42:0/2517455770 wait complete. 2019-12-05T05:17:13.774 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.769+0000 7fb61317e700 1 -- 172.21.15.42:0/2517455770 >> 172.21.15.42:0/2517455770 conn(0x7fb60c0a9640 msgr2=0x7fb60c09f2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:17:13.783 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a down, removed from mdsmap 2019-12-05T05:17:13.783 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2019-12-05T05:17:13.783 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:17:13.889 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.877+0000 7fab96ea2700 1 Processor -- start 2019-12-05T05:17:13.889 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.877+0000 7fab96ea2700 1 -- start start 2019-12-05T05:17:13.889 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.877+0000 7fab96ea2700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fab900a7740 0x7fab9008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:17:13.889 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.877+0000 7fab96ea2700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fab9009fae0 0x7fab9007eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:17:13.889 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.877+0000 7fab96ea2700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab900a0710 con 0x7fab9007e720 2019-12-05T05:17:13.889 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.877+0000 7fab96ea2700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab900a0b00 con 0x7fab9009fae0 2019-12-05T05:17:13.890 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.877+0000 7fab96ea2700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab9008e5b0 con 0x7fab900a7740 2019-12-05T05:17:13.890 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.877+0000 7fab9543f700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fab900a7740 0x7fab9008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:17:13.890 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.877+0000 7fab9543f700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fab900a7740 0x7fab9008df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:59056/0 (socket says 172.21.15.42:59056) 2019-12-05T05:17:13.890 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.877+0000 7fab9543f700 1 -- 172.21.15.42:0/2161525285 learned_addr learned my addr 172.21.15.42:0/2161525285 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:17:13.891 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.877+0000 7fab95c40700 1 --2- 172.21.15.42:0/2161525285 >> v2:172.21.15.40:3300/0 conn(0x7fab9009fae0 0x7fab9007eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:17:13.891 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.881+0000 7fab9543f700 1 -- 172.21.15.42:0/2161525285 >> v2:172.21.15.40:3300/0 conn(0x7fab9009fae0 msgr2=0x7fab9007eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:17:13.891 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.881+0000 7fab9543f700 1 --2- 172.21.15.42:0/2161525285 >> v2:172.21.15.40:3300/0 conn(0x7fab9009fae0 0x7fab9007eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:17:13.891 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.881+0000 7fab9543f700 1 -- 172.21.15.42:0/2161525285 >> v1:172.21.15.40:6789/0 conn(0x7fab9007e720 legacy=0x7fab9009f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:17:13.891 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.881+0000 7fab9543f700 1 -- 172.21.15.42:0/2161525285 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab9007f0a0 con 0x7fab900a7740 2019-12-05T05:17:13.891 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.881+0000 7fab9543f700 1 --2- 172.21.15.42:0/2161525285 >> v2:172.21.15.42:3300/0 conn(0x7fab900a7740 0x7fab9008df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fab8c002f30 tx=0x7fab8c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:13.891 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.881+0000 7fab9543f700 1 --2- 172.21.15.42:0/2161525285 >> v2:172.21.15.42:3300/0 conn(0x7fab900a7740 0x7fab9008df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fab8c002f30 tx=0x7fab8c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:13.892 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.881+0000 7fab9543f700 1 --2- 172.21.15.42:0/2161525285 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fab900a7740 0x7fab9008df80 secure :-1 s=READY pgs=617 cs=0 l=1 rx=0x7fab8c002f30 tx=0x7fab8c00fd30).ready entity=mon.1 client_cookie=5b407f1761af277e server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:17:13.892 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.881+0000 7fab9543f700 1 --2- 172.21.15.42:0/2161525285 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fab900a7740 0x7fab9008df80 secure :-1 s=THROTTLE_DONE pgs=617 cs=0 l=1 rx=0x7fab8c002f30 tx=0x7fab8c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:13.892 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.881+0000 7fab9543f700 1 --2- 172.21.15.42:0/2161525285 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fab900a7740 0x7fab9008df80 secure :-1 s=THROTTLE_DONE pgs=617 cs=0 l=1 rx=0x7fab8c002f30 tx=0x7fab8c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:13.892 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.881+0000 7fab87fff700 1 -- 172.21.15.42:0/2161525285 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fab8c008c90 con 0x7fab900a7740 2019-12-05T05:17:13.892 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.881+0000 7fab9543f700 1 --2- 172.21.15.42:0/2161525285 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fab900a7740 0x7fab9008df80 secure :-1 s=THROTTLE_DONE pgs=617 cs=0 l=1 rx=0x7fab8c002f30 tx=0x7fab8c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:13.892 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.881+0000 7fab87fff700 1 -- 172.21.15.42:0/2161525285 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab8c0088b0 con 0x7fab900a7740 2019-12-05T05:17:13.892 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.881+0000 7fab87fff700 1 -- 172.21.15.42:0/2161525285 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fab8c011ad0 con 0x7fab900a7740 2019-12-05T05:17:13.893 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.881+0000 7fab96ea2700 1 -- 172.21.15.42:0/2161525285 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fab900a7740 msgr2=0x7fab9008df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:17:13.893 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.881+0000 7fab96ea2700 1 --2- 172.21.15.42:0/2161525285 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fab900a7740 0x7fab9008df80 secure :-1 s=READY pgs=617 cs=0 l=1 rx=0x7fab8c002f30 tx=0x7fab8c00fd30).stop 2019-12-05T05:17:13.893 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.881+0000 7fab96ea2700 1 -- 172.21.15.42:0/2161525285 shutdown_connections 2019-12-05T05:17:13.893 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.881+0000 7fab96ea2700 1 --2- 172.21.15.42:0/2161525285 >> v2:172.21.15.40:3300/0 conn(0x7fab9009fae0 0x7fab9007eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:17:13.893 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.881+0000 7fab96ea2700 1 --2- 172.21.15.42:0/2161525285 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fab900a7740 0x7fab9008df80 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:17:13.893 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.881+0000 7fab96ea2700 1 -- 172.21.15.42:0/2161525285 shutdown_connections 2019-12-05T05:17:13.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.881+0000 7fab96ea2700 1 -- 172.21.15.42:0/2161525285 wait complete. 2019-12-05T05:17:13.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.881+0000 7fab96ea2700 1 -- 172.21.15.42:0/2161525285 >> 172.21.15.42:0/2161525285 conn(0x7fab900aabe0 msgr2=0x7fab900a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:17:13.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.885+0000 7fab96ea2700 1 Processor -- start 2019-12-05T05:17:13.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.885+0000 7fab96ea2700 1 -- start start 2019-12-05T05:17:13.898 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.885+0000 7fab96ea2700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fab90124d80 0x7fab90125870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:17:13.898 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.885+0000 7fab96ea2700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fab9007b370 0x7fab9007b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:17:13.898 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.885+0000 7fab96ea2700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab9007bcf0 con 0x7fab901178b0 2019-12-05T05:17:13.898 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.885+0000 7fab96ea2700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab90125330 con 0x7fab9007b370 2019-12-05T05:17:13.898 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.885+0000 7fab96ea2700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab901255d0 con 0x7fab90124d80 2019-12-05T05:17:13.899 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.885+0000 7fab9543f700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fab90124d80 0x7fab90125870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:17:13.899 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.885+0000 7fab9543f700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fab90124d80 0x7fab90125870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:59064/0 (socket says 172.21.15.42:59064) 2019-12-05T05:17:13.899 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.885+0000 7fab95c40700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fab9007b370 0x7fab9007b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:17:13.899 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.885+0000 7fab9543f700 1 -- 172.21.15.42:0/1919690133 learned_addr learned my addr 172.21.15.42:0/1919690133 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:17:13.899 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.885+0000 7fab85ffb700 1 -- 172.21.15.42:0/1919690133 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 565261437 0 0) 0x7fab9007bcf0 con 0x7fab901178b0 2019-12-05T05:17:13.899 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.885+0000 7fab95c40700 1 -- 172.21.15.42:0/1919690133 >> v2:172.21.15.42:3300/0 conn(0x7fab90124d80 msgr2=0x7fab90125870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:17:13.900 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.885+0000 7fab95c40700 1 --2- 172.21.15.42:0/1919690133 >> v2:172.21.15.42:3300/0 conn(0x7fab90124d80 0x7fab90125870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:17:13.900 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.885+0000 7fab95c40700 1 -- 172.21.15.42:0/1919690133 >> v1:172.21.15.40:6789/0 conn(0x7fab901178b0 legacy=0x7fab900a1740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:17:13.901 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.885+0000 7fab95c40700 1 -- 172.21.15.42:0/1919690133 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab9007bf20 con 0x7fab9007b370 2019-12-05T05:17:13.901 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.885+0000 7fab9543f700 1 --2- 172.21.15.42:0/1919690133 >> v2:172.21.15.42:3300/0 conn(0x7fab90124d80 0x7fab90125870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:17:13.901 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.885+0000 7fab95c40700 1 --2- 172.21.15.42:0/1919690133 >> v2:172.21.15.40:3300/0 conn(0x7fab9007b370 0x7fab9007b7a0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fab900a7090 tx=0x7fab88010d00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:13.901 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.885+0000 7fab95c40700 1 --2- 172.21.15.42:0/1919690133 >> v2:172.21.15.40:3300/0 conn(0x7fab9007b370 0x7fab9007b7a0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fab900a7090 tx=0x7fab88010d00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:13.902 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.885+0000 7fab95c40700 1 --2- 172.21.15.42:0/1919690133 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab9007b370 0x7fab9007b7a0 secure :-1 s=READY pgs=490 cs=0 l=1 rx=0x7fab900a7090 tx=0x7fab88010d00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:17:13.902 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.885+0000 7fab95c40700 1 --2- 172.21.15.42:0/1919690133 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab9007b370 0x7fab9007b7a0 secure :-1 s=THROTTLE_DONE pgs=490 cs=0 l=1 rx=0x7fab900a7090 tx=0x7fab88010d00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:13.902 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.885+0000 7fab85ffb700 1 -- 172.21.15.42:0/1919690133 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fab880129b0 con 0x7fab9007b370 2019-12-05T05:17:13.902 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.885+0000 7fab95c40700 1 --2- 172.21.15.42:0/1919690133 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab9007b370 0x7fab9007b7a0 secure :-1 s=THROTTLE_DONE pgs=490 cs=0 l=1 rx=0x7fab900a7090 tx=0x7fab88010d00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:13.902 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.885+0000 7fab95c40700 1 --2- 172.21.15.42:0/1919690133 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab9007b370 0x7fab9007b7a0 secure :-1 s=THROTTLE_DONE pgs=490 cs=0 l=1 rx=0x7fab900a7090 tx=0x7fab88010d00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:13.902 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.885+0000 7fab85ffb700 1 -- 172.21.15.42:0/1919690133 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab880125d0 con 0x7fab9007b370 2019-12-05T05:17:13.903 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.885+0000 7fab96ea2700 1 -- 172.21.15.42:0/1919690133 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab901217c0 con 0x7fab9007b370 2019-12-05T05:17:13.903 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.885+0000 7fab85ffb700 1 -- 172.21.15.42:0/1919690133 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fab8800e570 con 0x7fab9007b370 2019-12-05T05:17:13.903 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.885+0000 7fab96ea2700 1 -- 172.21.15.42:0/1919690133 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab90121b50 con 0x7fab9007b370 2019-12-05T05:17:13.903 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.889+0000 7fab95c40700 1 --2- 172.21.15.42:0/1919690133 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab9007b370 0x7fab9007b7a0 secure :-1 s=THROTTLE_DONE pgs=490 cs=0 l=1 rx=0x7fab900a7090 tx=0x7fab88010d00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:13.903 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.889+0000 7fab85ffb700 1 -- 172.21.15.42:0/1919690133 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fab88013060 con 0x7fab9007b370 2019-12-05T05:17:13.903 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.889+0000 7fab95c40700 1 --2- 172.21.15.42:0/1919690133 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab9007b370 0x7fab9007b7a0 secure :-1 s=THROTTLE_DONE pgs=490 cs=0 l=1 rx=0x7fab900a7090 tx=0x7fab88010d00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:13.903 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.889+0000 7fab85ffb700 1 --2- 172.21.15.42:0/1919690133 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fab7003ac70 0x7fab7003d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:17:13.904 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.889+0000 7fab85ffb700 1 -- 172.21.15.42:0/1919690133 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6891+0+0 (secure 0 0 0) 0x7fab8804b360 con 0x7fab9007b370 2019-12-05T05:17:13.904 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.889+0000 7fab9543f700 1 --2- 172.21.15.42:0/1919690133 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fab7003ac70 0x7fab7003d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:17:13.904 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.889+0000 7fab9543f700 1 --2- 172.21.15.42:0/1919690133 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fab7003ac70 0x7fab7003d180 crc :-1 s=READY pgs=276 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:17:13.904 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.893+0000 7fab96ea2700 1 -- 172.21.15.42:0/1919690133 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab9005d110 con 0x7fab9007b370 2019-12-05T05:17:13.905 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.897+0000 7fab95c40700 1 --2- 172.21.15.42:0/1919690133 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab9007b370 0x7fab9007b7a0 secure :-1 s=THROTTLE_DONE pgs=490 cs=0 l=1 rx=0x7fab900a7090 tx=0x7fab88010d00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:13.905 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:13.897+0000 7fab85ffb700 1 -- 172.21.15.42:0/1919690133 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fab880323c0 con 0x7fab9007b370 2019-12-05T05:17:14.061 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:14.057+0000 7fab96ea2700 1 -- 172.21.15.42:0/1919690133 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fab9005b8c0 con 0x7fab9007b370 2019-12-05T05:17:14.062 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:14.057+0000 7fab95c40700 1 --2- 172.21.15.42:0/1919690133 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab9007b370 0x7fab9007b7a0 secure :-1 s=THROTTLE_DONE pgs=490 cs=0 l=1 rx=0x7fab900a7090 tx=0x7fab88010d00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:14.063 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:14.057+0000 7fab85ffb700 1 -- 172.21.15.42:0/1919690133 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) v1 ==== 94+0+1771 (secure 0 0 0) 0x7fab88032620 con 0x7fab9007b370 2019-12-05T05:17:14.065 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 79 2019-12-05T05:17:14.065 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:17:14.065 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":79,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":79,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:17:13.420087+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5844},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5844":{"gid":5844,"name":"b","rank":0,"incarnation":79,"state":"up:replay","state_seq":2,"addr":"172.21.15.40:6833/2698421979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2698421979},{"type":"v1","addr":"172.21.15.40:6833","nonce":2698421979}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:17:14.070 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:14.061+0000 7fab96ea2700 1 -- 172.21.15.42:0/1919690133 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fab7003ac70 msgr2=0x7fab7003d180 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:17:14.070 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:14.061+0000 7fab96ea2700 1 --2- 172.21.15.42:0/1919690133 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fab7003ac70 0x7fab7003d180 crc :-1 s=READY pgs=276 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:17:14.070 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:14.061+0000 7fab96ea2700 1 -- 172.21.15.42:0/1919690133 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab9007b370 msgr2=0x7fab9007b7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:17:14.070 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:14.061+0000 7fab96ea2700 1 --2- 172.21.15.42:0/1919690133 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab9007b370 0x7fab9007b7a0 secure :-1 s=READY pgs=490 cs=0 l=1 rx=0x7fab900a7090 tx=0x7fab88010d00).stop 2019-12-05T05:17:14.071 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:14.061+0000 7fab96ea2700 1 -- 172.21.15.42:0/1919690133 shutdown_connections 2019-12-05T05:17:14.071 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:14.061+0000 7fab96ea2700 1 --2- 172.21.15.42:0/1919690133 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fab7003ac70 0x7fab7003d180 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:17:14.072 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:14.061+0000 7fab96ea2700 1 --2- 172.21.15.42:0/1919690133 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab9007b370 0x7fab9007b7a0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:17:14.072 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:14.061+0000 7fab96ea2700 1 --2- 172.21.15.42:0/1919690133 >> v2:172.21.15.42:3300/0 conn(0x7fab90124d80 0x7fab90125870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:17:14.072 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:14.061+0000 7fab96ea2700 1 -- 172.21.15.42:0/1919690133 shutdown_connections 2019-12-05T05:17:14.072 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:14.061+0000 7fab96ea2700 1 -- 172.21.15.42:0/1919690133 wait complete. 2019-12-05T05:17:14.073 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:14.061+0000 7fab96ea2700 1 -- 172.21.15.42:0/1919690133 >> 172.21.15.42:0/1919690133 conn(0x7fab900aabe0 msgr2=0x7fab900a8040 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:17:14.078 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 79, "feature_flags": { "enable_multiple": false, "ever_enabled_multiple": false }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2019-12-05T05:03:35.604738+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 79, "ever_allowed_features": 2, "explicitly_allowed_features": 2, "failed": [], "flags": 18, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5844": { "addr": "172.21.15.40:6833/2698421979", "addrs": { "addrvec": [ { "addr": "172.21.15.40:6832", "nonce": 2698421979, "type": "v2" }, { "addr": "172.21.15.40:6833", "nonce": 2698421979, "type": "v1" } ] }, "export_targets": [], "features": 4540138292836696063, "flags": 0, "gid": 5844, "incarnation": 79, "name": "b", "rank": 0, "state": "up:replay", "state_seq": 2 } }, "last_failure": 0, "last_failure_osd_epoch": 59, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 2, "min_compat_client": "0 (unknown)", "modified": "2019-12-05T05:17:13.420087+0000", "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5844 } } } ], "standbys": [] } 2019-12-05T05:17:16.079 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:17:16.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:16.177+0000 7f4e1ff92700 1 Processor -- start 2019-12-05T05:17:16.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:16.177+0000 7f4e1ff92700 1 -- start start 2019-12-05T05:17:16.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:16.181+0000 7f4e1ff92700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f4e180a7740 0x7f4e1808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:17:16.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:16.181+0000 7f4e1ff92700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f4e1809fae0 0x7f4e1807eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:17:16.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:16.181+0000 7f4e1ff92700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e180a0710 con 0x7f4e1807e720 2019-12-05T05:17:16.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:16.181+0000 7f4e1ff92700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e180a0b00 con 0x7f4e180a7740 2019-12-05T05:17:16.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:16.181+0000 7f4e1ff92700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e1808e5b0 con 0x7f4e1809fae0 2019-12-05T05:17:16.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:16.181+0000 7f4e1ed30700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f4e1809fae0 0x7f4e1807eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:17:16.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:16.181+0000 7f4e1ed30700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f4e1809fae0 0x7f4e1807eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:59076/0 (socket says 172.21.15.42:59076) 2019-12-05T05:17:16.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:16.181+0000 7f4e1ed30700 1 -- 172.21.15.42:0/1438313933 learned_addr learned my addr 172.21.15.42:0/1438313933 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:17:16.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:16.181+0000 7f4e1e52f700 1 --2- 172.21.15.42:0/1438313933 >> v2:172.21.15.40:3300/0 conn(0x7f4e180a7740 0x7f4e1808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:17:16.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:16.181+0000 7f4e1ed30700 1 -- 172.21.15.42:0/1438313933 >> v2:172.21.15.40:3300/0 conn(0x7f4e180a7740 msgr2=0x7f4e1808df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:17:16.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:16.181+0000 7f4e1ed30700 1 --2- 172.21.15.42:0/1438313933 >> v2:172.21.15.40:3300/0 conn(0x7f4e180a7740 0x7f4e1808df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:17:16.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:16.181+0000 7f4e1ed30700 1 -- 172.21.15.42:0/1438313933 >> v1:172.21.15.40:6789/0 conn(0x7f4e1807e720 legacy=0x7f4e1809f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:17:16.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:16.181+0000 7f4e1ed30700 1 -- 172.21.15.42:0/1438313933 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e1807f0a0 con 0x7f4e1809fae0 2019-12-05T05:17:16.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:16.181+0000 7f4e1ed30700 1 --2- 172.21.15.42:0/1438313933 >> v2:172.21.15.42:3300/0 conn(0x7f4e1809fae0 0x7f4e1807eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f4e100032b0 tx=0x7f4e100100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:16.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:16.181+0000 7f4e1ed30700 1 --2- 172.21.15.42:0/1438313933 >> v2:172.21.15.42:3300/0 conn(0x7f4e1809fae0 0x7f4e1807eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f4e100032b0 tx=0x7f4e100100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:16.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:16.181+0000 7f4e1ed30700 1 --2- 172.21.15.42:0/1438313933 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4e1809fae0 0x7f4e1807eb50 secure :-1 s=READY pgs=618 cs=0 l=1 rx=0x7f4e100032b0 tx=0x7f4e100100b0).ready entity=mon.1 client_cookie=f7004f5b0eb650cd server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:17:16.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:16.181+0000 7f4e1ed30700 1 --2- 172.21.15.42:0/1438313933 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4e1809fae0 0x7f4e1807eb50 secure :-1 s=THROTTLE_DONE pgs=618 cs=0 l=1 rx=0x7f4e100032b0 tx=0x7f4e100100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:16.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:16.181+0000 7f4e1ed30700 1 --2- 172.21.15.42:0/1438313933 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4e1809fae0 0x7f4e1807eb50 secure :-1 s=THROTTLE_DONE pgs=618 cs=0 l=1 rx=0x7f4e100032b0 tx=0x7f4e100100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:16.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:16.181+0000 7f4e1ed30700 1 --2- 172.21.15.42:0/1438313933 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4e1809fae0 0x7f4e1807eb50 secure :-1 s=THROTTLE_DONE pgs=618 cs=0 l=1 rx=0x7f4e100032b0 tx=0x7f4e100100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:16.193 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:16.181+0000 7f4e1d52d700 1 -- 172.21.15.42:0/1438313933 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4e10010d60 con 0x7f4e1809fae0 2019-12-05T05:17:16.193 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:16.181+0000 7f4e1d52d700 1 -- 172.21.15.42:0/1438313933 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e10010980 con 0x7f4e1809fae0 2019-12-05T05:17:16.193 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:16.181+0000 7f4e1d52d700 1 -- 172.21.15.42:0/1438313933 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4e100114a0 con 0x7f4e1809fae0 2019-12-05T05:17:16.193 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:16.181+0000 7f4e1ff92700 1 -- 172.21.15.42:0/1438313933 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4e1809fae0 msgr2=0x7f4e1807eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:17:16.194 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:16.181+0000 7f4e1ff92700 1 --2- 172.21.15.42:0/1438313933 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4e1809fae0 0x7f4e1807eb50 secure :-1 s=READY pgs=618 cs=0 l=1 rx=0x7f4e100032b0 tx=0x7f4e100100b0).stop 2019-12-05T05:17:16.194 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:16.185+0000 7f4e1ff92700 1 -- 172.21.15.42:0/1438313933 shutdown_connections 2019-12-05T05:17:16.194 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:16.185+0000 7f4e1ff92700 1 --2- 172.21.15.42:0/1438313933 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4e1809fae0 0x7f4e1807eb50 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:17:16.195 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:16.185+0000 7f4e1ff92700 1 --2- 172.21.15.42:0/1438313933 >> v2:172.21.15.40:3300/0 conn(0x7f4e180a7740 0x7f4e1808df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:17:16.195 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:16.185+0000 7f4e1ff92700 1 -- 172.21.15.42:0/1438313933 shutdown_connections 2019-12-05T05:17:16.195 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:16.185+0000 7f4e1ff92700 1 -- 172.21.15.42:0/1438313933 wait complete. 2019-12-05T05:17:16.195 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:16.185+0000 7f4e1ff92700 1 -- 172.21.15.42:0/1438313933 >> 172.21.15.42:0/1438313933 conn(0x7f4e180aabe0 msgr2=0x7f4e180a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:17:16.204 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:16.185+0000 7f4e1ff92700 1 Processor -- start 2019-12-05T05:17:16.205 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:16.185+0000 7f4e1ff92700 1 -- start start 2019-12-05T05:17:16.205 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:16.189+0000 7f4e1ff92700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f4e180a4240 0x7f4e180a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:17:16.206 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:16.189+0000 7f4e1ff92700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4e18144b50 0x7f4e1812f480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:17:16.206 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:16.189+0000 7f4e1ff92700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e18128ea0 con 0x7f4e18113b00 2019-12-05T05:17:16.206 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:16.189+0000 7f4e1ff92700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e180a4e40 con 0x7f4e180a4240 2019-12-05T05:17:16.206 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:16.189+0000 7f4e1ff92700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e1808f5d0 con 0x7f4e18144b50 2019-12-05T05:17:16.207 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:16.189+0000 7f4e1dd2e700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f4e18113b00 0x7f4e181286a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:59118/0 (socket says 172.21.15.42:59118) 2019-12-05T05:17:16.207 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:16.189+0000 7f4e1dd2e700 1 -- 172.21.15.42:0/538717302 learned_addr learned my addr 172.21.15.42:0/538717302 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:17:16.207 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:16.189+0000 7f4e1e52f700 1 --2- 172.21.15.42:0/538717302 >> v2:172.21.15.40:3300/0 conn(0x7f4e180a4240 0x7f4e180a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:17:16.208 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:16.189+0000 7f4e1ed30700 1 --2- 172.21.15.42:0/538717302 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4e18144b50 0x7f4e1812f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:17:16.208 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:16.189+0000 7f4e0affd700 1 -- 172.21.15.42:0/538717302 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3773128705 0 0) 0x7f4e18128ea0 con 0x7f4e18113b00 2019-12-05T05:17:16.209 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:16.189+0000 7f4e0affd700 1 -- 172.21.15.42:0/538717302 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e000030b0 con 0x7f4e18113b00 2019-12-05T05:17:16.209 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:16.189+0000 7f4e1e52f700 1 -- 172.21.15.42:0/538717302 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4e18144b50 msgr2=0x7f4e1812f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:17:16.209 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:16.189+0000 7f4e1e52f700 1 --2- 172.21.15.42:0/538717302 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4e18144b50 0x7f4e1812f480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:17:16.209 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:16.189+0000 7f4e1e52f700 1 -- 172.21.15.42:0/538717302 >> v1:172.21.15.42:6789/0 conn(0x7f4e18113b00 legacy=0x7f4e181286a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:17:16.210 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:16.189+0000 7f4e1e52f700 1 -- 172.21.15.42:0/538717302 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e18128030 con 0x7f4e180a4240 2019-12-05T05:17:16.210 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:16.189+0000 7f4e1e52f700 1 --2- 172.21.15.42:0/538717302 >> v2:172.21.15.40:3300/0 conn(0x7f4e180a4240 0x7f4e180a50e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f4e0c002f30 tx=0x7f4e0c0109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:16.210 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:16.189+0000 7f4e1ed30700 1 --2- 172.21.15.42:0/538717302 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4e18144b50 0x7f4e1812f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:17:16.212 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:16.189+0000 7f4e1e52f700 1 --2- 172.21.15.42:0/538717302 >> v2:172.21.15.40:3300/0 conn(0x7f4e180a4240 0x7f4e180a50e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f4e0c002f30 tx=0x7f4e0c0109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:16.212 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:16.189+0000 7f4e1e52f700 1 --2- 172.21.15.42:0/538717302 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e180a4240 0x7f4e180a50e0 secure :-1 s=READY pgs=492 cs=0 l=1 rx=0x7f4e0c002f30 tx=0x7f4e0c0109e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:17:16.212 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:16.189+0000 7f4e1e52f700 1 --2- 172.21.15.42:0/538717302 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e180a4240 0x7f4e180a50e0 secure :-1 s=THROTTLE_DONE pgs=492 cs=0 l=1 rx=0x7f4e0c002f30 tx=0x7f4e0c0109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:16.212 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:16.189+0000 7f4e0affd700 1 -- 172.21.15.42:0/538717302 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4e0c005920 con 0x7f4e180a4240 2019-12-05T05:17:16.213 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:16.189+0000 7f4e1e52f700 1 --2- 172.21.15.42:0/538717302 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e180a4240 0x7f4e180a50e0 secure :-1 s=THROTTLE_DONE pgs=492 cs=0 l=1 rx=0x7f4e0c002f30 tx=0x7f4e0c0109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:16.213 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:16.189+0000 7f4e1e52f700 1 --2- 172.21.15.42:0/538717302 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e180a4240 0x7f4e180a50e0 secure :-1 s=THROTTLE_DONE pgs=492 cs=0 l=1 rx=0x7f4e0c002f30 tx=0x7f4e0c0109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:16.213 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:16.189+0000 7f4e0affd700 1 -- 172.21.15.42:0/538717302 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e0c005540 con 0x7f4e180a4240 2019-12-05T05:17:16.214 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:16.189+0000 7f4e1ff92700 1 -- 172.21.15.42:0/538717302 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e1808f8e0 con 0x7f4e180a4240 2019-12-05T05:17:16.214 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:16.189+0000 7f4e0affd700 1 -- 172.21.15.42:0/538717302 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4e0c012510 con 0x7f4e180a4240 2019-12-05T05:17:16.214 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:16.189+0000 7f4e1ff92700 1 -- 172.21.15.42:0/538717302 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e180a57d0 con 0x7f4e180a4240 2019-12-05T05:17:16.214 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:16.193+0000 7f4e1e52f700 1 --2- 172.21.15.42:0/538717302 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e180a4240 0x7f4e180a50e0 secure :-1 s=THROTTLE_DONE pgs=492 cs=0 l=1 rx=0x7f4e0c002f30 tx=0x7f4e0c0109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:16.215 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:16.193+0000 7f4e0affd700 1 -- 172.21.15.42:0/538717302 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f4e0c012820 con 0x7f4e180a4240 2019-12-05T05:17:16.215 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:16.193+0000 7f4e1e52f700 1 --2- 172.21.15.42:0/538717302 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e180a4240 0x7f4e180a50e0 secure :-1 s=THROTTLE_DONE pgs=492 cs=0 l=1 rx=0x7f4e0c002f30 tx=0x7f4e0c0109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:16.215 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:16.193+0000 7f4e0affd700 1 --2- 172.21.15.42:0/538717302 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4e0003b2a0 0x7f4e0003d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:17:16.215 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:16.193+0000 7f4e0affd700 1 -- 172.21.15.42:0/538717302 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6891+0+0 (secure 0 0 0) 0x7f4e0c04aaf0 con 0x7f4e180a4240 2019-12-05T05:17:16.215 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:16.193+0000 7f4e1dd2e700 1 --2- 172.21.15.42:0/538717302 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4e0003b2a0 0x7f4e0003d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:17:16.216 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:16.193+0000 7f4e1dd2e700 1 --2- 172.21.15.42:0/538717302 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4e0003b2a0 0x7f4e0003d7b0 crc :-1 s=READY pgs=277 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:17:16.216 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:16.193+0000 7f4e1ff92700 1 -- 172.21.15.42:0/538717302 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e1805d110 con 0x7f4e180a4240 2019-12-05T05:17:16.217 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:16.201+0000 7f4e1e52f700 1 --2- 172.21.15.42:0/538717302 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e180a4240 0x7f4e180a50e0 secure :-1 s=THROTTLE_DONE pgs=492 cs=0 l=1 rx=0x7f4e0c002f30 tx=0x7f4e0c0109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:16.217 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:16.201+0000 7f4e0affd700 1 -- 172.21.15.42:0/538717302 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f4e0c031b70 con 0x7f4e180a4240 2019-12-05T05:17:16.380 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:16.377+0000 7f4e1ff92700 1 -- 172.21.15.42:0/538717302 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4e1805b8c0 con 0x7f4e180a4240 2019-12-05T05:17:16.380 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:16.377+0000 7f4e1e52f700 1 --2- 172.21.15.42:0/538717302 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e180a4240 0x7f4e180a50e0 secure :-1 s=THROTTLE_DONE pgs=492 cs=0 l=1 rx=0x7f4e0c002f30 tx=0x7f4e0c0109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:16.381 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:16.377+0000 7f4e0affd700 1 -- 172.21.15.42:0/538717302 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) v1 ==== 94+0+1775 (secure 0 0 0) 0x7f4e0c031dd0 con 0x7f4e180a4240 2019-12-05T05:17:16.383 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 80 2019-12-05T05:17:16.383 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:17:16.383 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":80,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:17:15.935865+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5844},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5844":{"gid":5844,"name":"b","rank":0,"incarnation":79,"state":"up:reconnect","state_seq":30,"addr":"172.21.15.40:6833/2698421979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2698421979},{"type":"v1","addr":"172.21.15.40:6833","nonce":2698421979}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:17:16.386 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:16.381+0000 7f4e1ff92700 1 -- 172.21.15.42:0/538717302 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4e0003b2a0 msgr2=0x7f4e0003d7b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:17:16.386 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:16.381+0000 7f4e1ff92700 1 --2- 172.21.15.42:0/538717302 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4e0003b2a0 0x7f4e0003d7b0 crc :-1 s=READY pgs=277 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:17:16.386 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:16.381+0000 7f4e1ff92700 1 -- 172.21.15.42:0/538717302 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e180a4240 msgr2=0x7f4e180a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:17:16.386 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:16.381+0000 7f4e1ff92700 1 --2- 172.21.15.42:0/538717302 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e180a4240 0x7f4e180a50e0 secure :-1 s=READY pgs=492 cs=0 l=1 rx=0x7f4e0c002f30 tx=0x7f4e0c0109e0).stop 2019-12-05T05:17:16.387 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:16.381+0000 7f4e1ff92700 1 -- 172.21.15.42:0/538717302 shutdown_connections 2019-12-05T05:17:16.387 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:16.381+0000 7f4e1ff92700 1 --2- 172.21.15.42:0/538717302 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4e0003b2a0 0x7f4e0003d7b0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:17:16.387 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:16.381+0000 7f4e1ff92700 1 --2- 172.21.15.42:0/538717302 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4e18144b50 0x7f4e1812f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:17:16.387 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:16.381+0000 7f4e1ff92700 1 --2- 172.21.15.42:0/538717302 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e180a4240 0x7f4e180a50e0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:17:16.388 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:16.381+0000 7f4e1ff92700 1 -- 172.21.15.42:0/538717302 shutdown_connections 2019-12-05T05:17:16.388 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:16.381+0000 7f4e1ff92700 1 -- 172.21.15.42:0/538717302 wait complete. 2019-12-05T05:17:16.388 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:16.381+0000 7f4e1ff92700 1 -- 172.21.15.42:0/538717302 >> 172.21.15.42:0/538717302 conn(0x7f4e180a9990 msgr2=0x7f4e18059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:17:16.396 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2019-12-05T05:17:18.397 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:17:18.503 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:18.493+0000 7fc01a12a700 1 Processor -- start 2019-12-05T05:17:18.503 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:18.493+0000 7fc01a12a700 1 -- start start 2019-12-05T05:17:18.504 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:18.493+0000 7fc01a12a700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc0140a7740 0x7fc01408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:17:18.504 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:18.493+0000 7fc01a12a700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc01409fae0 0x7fc01407eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:17:18.504 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:18.493+0000 7fc01a12a700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0140a0710 con 0x7fc01407e720 2019-12-05T05:17:18.504 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:18.493+0000 7fc01a12a700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0140a0b00 con 0x7fc0140a7740 2019-12-05T05:17:18.504 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:18.493+0000 7fc01a12a700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc01408e5b0 con 0x7fc01409fae0 2019-12-05T05:17:18.505 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:18.497+0000 7fc0137fe700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fc01407e720 0x7fc01409f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:59128/0 (socket says 172.21.15.42:59128) 2019-12-05T05:17:18.505 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:18.497+0000 7fc0137fe700 1 -- 172.21.15.42:0/1871546980 learned_addr learned my addr 172.21.15.42:0/1871546980 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:17:18.505 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:18.497+0000 7fc013fff700 1 --2- 172.21.15.42:0/1871546980 >> v2:172.21.15.40:3300/0 conn(0x7fc0140a7740 0x7fc01408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:17:18.505 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:18.497+0000 7fc018ec8700 1 --2- 172.21.15.42:0/1871546980 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc01409fae0 0x7fc01407eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:17:18.505 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:18.497+0000 7fc012ffd700 1 -- 172.21.15.42:0/1871546980 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3081986278 0 0) 0x7fc0140a0710 con 0x7fc01407e720 2019-12-05T05:17:18.506 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:18.497+0000 7fc012ffd700 1 -- 172.21.15.42:0/1871546980 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0000030b0 con 0x7fc01407e720 2019-12-05T05:17:18.506 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:18.497+0000 7fc013fff700 1 -- 172.21.15.42:0/1871546980 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc01409fae0 msgr2=0x7fc01407eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:17:18.506 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:18.497+0000 7fc013fff700 1 --2- 172.21.15.42:0/1871546980 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc01409fae0 0x7fc01407eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:17:18.506 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:18.497+0000 7fc013fff700 1 -- 172.21.15.42:0/1871546980 >> v1:172.21.15.42:6789/0 conn(0x7fc01407e720 legacy=0x7fc01409f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:17:18.506 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:18.497+0000 7fc013fff700 1 -- 172.21.15.42:0/1871546980 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc01407f0a0 con 0x7fc0140a7740 2019-12-05T05:17:18.507 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:18.497+0000 7fc013fff700 1 --2- 172.21.15.42:0/1871546980 >> v2:172.21.15.40:3300/0 conn(0x7fc0140a7740 0x7fc01408df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fc00c008b10 tx=0x7fc00c00fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:18.507 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:18.497+0000 7fc013fff700 1 --2- 172.21.15.42:0/1871546980 >> v2:172.21.15.40:3300/0 conn(0x7fc0140a7740 0x7fc01408df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fc00c008b10 tx=0x7fc00c00fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:18.507 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:18.497+0000 7fc013fff700 1 --2- 172.21.15.42:0/1871546980 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc0140a7740 0x7fc01408df80 secure :-1 s=READY pgs=493 cs=0 l=1 rx=0x7fc00c008b10 tx=0x7fc00c00fff0).ready entity=mon.0 client_cookie=9397a28a42043e79 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:17:18.507 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:18.497+0000 7fc013fff700 1 --2- 172.21.15.42:0/1871546980 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc0140a7740 0x7fc01408df80 secure :-1 s=THROTTLE_DONE pgs=493 cs=0 l=1 rx=0x7fc00c008b10 tx=0x7fc00c00fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:18.507 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:18.497+0000 7fc012ffd700 1 -- 172.21.15.42:0/1871546980 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc00c010900 con 0x7fc0140a7740 2019-12-05T05:17:18.507 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:18.497+0000 7fc013fff700 1 --2- 172.21.15.42:0/1871546980 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc0140a7740 0x7fc01408df80 secure :-1 s=THROTTLE_DONE pgs=493 cs=0 l=1 rx=0x7fc00c008b10 tx=0x7fc00c00fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:18.508 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:18.497+0000 7fc013fff700 1 --2- 172.21.15.42:0/1871546980 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc0140a7740 0x7fc01408df80 secure :-1 s=THROTTLE_DONE pgs=493 cs=0 l=1 rx=0x7fc00c008b10 tx=0x7fc00c00fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:18.508 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:18.497+0000 7fc012ffd700 1 -- 172.21.15.42:0/1871546980 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc00c008b50 con 0x7fc0140a7740 2019-12-05T05:17:18.508 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:18.497+0000 7fc012ffd700 1 -- 172.21.15.42:0/1871546980 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc00c0110c0 con 0x7fc0140a7740 2019-12-05T05:17:18.508 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:18.497+0000 7fc01a12a700 1 -- 172.21.15.42:0/1871546980 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc0140a7740 msgr2=0x7fc01408df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:17:18.508 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:18.497+0000 7fc01a12a700 1 --2- 172.21.15.42:0/1871546980 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc0140a7740 0x7fc01408df80 secure :-1 s=READY pgs=493 cs=0 l=1 rx=0x7fc00c008b10 tx=0x7fc00c00fff0).stop 2019-12-05T05:17:18.509 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:18.497+0000 7fc01a12a700 1 -- 172.21.15.42:0/1871546980 shutdown_connections 2019-12-05T05:17:18.509 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:18.497+0000 7fc01a12a700 1 --2- 172.21.15.42:0/1871546980 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc01409fae0 0x7fc01407eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:17:18.509 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:18.497+0000 7fc01a12a700 1 --2- 172.21.15.42:0/1871546980 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc0140a7740 0x7fc01408df80 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:17:18.509 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:18.497+0000 7fc01a12a700 1 -- 172.21.15.42:0/1871546980 shutdown_connections 2019-12-05T05:17:18.509 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:18.497+0000 7fc01a12a700 1 -- 172.21.15.42:0/1871546980 wait complete. 2019-12-05T05:17:18.509 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:18.497+0000 7fc01a12a700 1 -- 172.21.15.42:0/1871546980 >> 172.21.15.42:0/1871546980 conn(0x7fc0140aabe0 msgr2=0x7fc0140a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:17:18.513 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:18.501+0000 7fc01a12a700 1 Processor -- start 2019-12-05T05:17:18.514 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:18.501+0000 7fc01a12a700 1 -- start start 2019-12-05T05:17:18.514 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:18.501+0000 7fc01a12a700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc014124c30 0x7fc014116f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:17:18.514 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:18.501+0000 7fc01a12a700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc0140a5660 0x7fc0140a5a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:17:18.514 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:18.501+0000 7fc01a12a700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc014122210 con 0x7fc014122ff0 2019-12-05T05:17:18.514 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:18.501+0000 7fc01a12a700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc014123420 con 0x7fc014124c30 2019-12-05T05:17:18.514 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:18.501+0000 7fc01a12a700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc014117560 con 0x7fc0140a5660 2019-12-05T05:17:18.515 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:18.501+0000 7fc013fff700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc014124c30 0x7fc014116f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:17:18.515 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:18.501+0000 7fc018ec8700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc0140a5660 0x7fc0140a5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:17:18.515 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:18.501+0000 7fc013fff700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc014124c30 0x7fc014116f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.42:46908/0 (socket says 172.21.15.42:46908) 2019-12-05T05:17:18.515 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:18.501+0000 7fc013fff700 1 -- 172.21.15.42:0/2923259356 learned_addr learned my addr 172.21.15.42:0/2923259356 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:17:18.515 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:18.501+0000 7fc013fff700 1 -- 172.21.15.42:0/2923259356 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc0140a5660 msgr2=0x7fc0140a5a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:17:18.515 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:18.501+0000 7fc013fff700 1 --2- 172.21.15.42:0/2923259356 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc0140a5660 0x7fc0140a5a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:17:18.516 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:18.501+0000 7fc013fff700 1 -- 172.21.15.42:0/2923259356 >> v1:172.21.15.40:6789/0 conn(0x7fc014122ff0 legacy=0x7fc01407d920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:17:18.516 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:18.501+0000 7fc010ff9700 1 -- 172.21.15.42:0/2923259356 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1211674459 0 0) 0x7fc0000030b0 con 0x7fc014122ff0 2019-12-05T05:17:18.516 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:18.501+0000 7fc013fff700 1 -- 172.21.15.42:0/2923259356 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0140a5fe0 con 0x7fc014124c30 2019-12-05T05:17:18.516 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:18.501+0000 7fc018ec8700 1 --2- 172.21.15.42:0/2923259356 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc0140a5660 0x7fc0140a5a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:17:18.516 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:18.501+0000 7fc013fff700 1 --2- 172.21.15.42:0/2923259356 >> v2:172.21.15.40:3300/0 conn(0x7fc014124c30 0x7fc014116f40 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fc00c008b10 tx=0x7fc00c007960).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:18.517 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:18.501+0000 7fc013fff700 1 --2- 172.21.15.42:0/2923259356 >> v2:172.21.15.40:3300/0 conn(0x7fc014124c30 0x7fc014116f40 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fc00c008b10 tx=0x7fc00c007960).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:18.517 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:18.501+0000 7fc013fff700 1 --2- 172.21.15.42:0/2923259356 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc014124c30 0x7fc014116f40 secure :-1 s=READY pgs=495 cs=0 l=1 rx=0x7fc00c008b10 tx=0x7fc00c007960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:17:18.517 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:18.501+0000 7fc013fff700 1 --2- 172.21.15.42:0/2923259356 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc014124c30 0x7fc014116f40 secure :-1 s=THROTTLE_DONE pgs=495 cs=0 l=1 rx=0x7fc00c008b10 tx=0x7fc00c007960).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:18.517 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:18.501+0000 7fc010ff9700 1 -- 172.21.15.42:0/2923259356 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc00c005ab0 con 0x7fc014124c30 2019-12-05T05:17:18.518 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:18.501+0000 7fc013fff700 1 --2- 172.21.15.42:0/2923259356 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc014124c30 0x7fc014116f40 secure :-1 s=THROTTLE_DONE pgs=495 cs=0 l=1 rx=0x7fc00c008b10 tx=0x7fc00c007960).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:18.518 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:18.501+0000 7fc013fff700 1 --2- 172.21.15.42:0/2923259356 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc014124c30 0x7fc014116f40 secure :-1 s=THROTTLE_DONE pgs=495 cs=0 l=1 rx=0x7fc00c008b10 tx=0x7fc00c007960).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:18.518 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:18.505+0000 7fc010ff9700 1 -- 172.21.15.42:0/2923259356 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc00c010b00 con 0x7fc014124c30 2019-12-05T05:17:18.518 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:18.505+0000 7fc010ff9700 1 -- 172.21.15.42:0/2923259356 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc00c00f470 con 0x7fc014124c30 2019-12-05T05:17:18.518 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:18.505+0000 7fc01a12a700 1 -- 172.21.15.42:0/2923259356 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc01413a6b0 con 0x7fc014124c30 2019-12-05T05:17:18.518 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:18.505+0000 7fc01a12a700 1 -- 172.21.15.42:0/2923259356 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc01413ac60 con 0x7fc014124c30 2019-12-05T05:17:18.519 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:18.505+0000 7fc013fff700 1 --2- 172.21.15.42:0/2923259356 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc014124c30 0x7fc014116f40 secure :-1 s=THROTTLE_DONE pgs=495 cs=0 l=1 rx=0x7fc00c008b10 tx=0x7fc00c007960).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:18.519 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:18.505+0000 7fc010ff9700 1 -- 172.21.15.42:0/2923259356 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fc00c00f700 con 0x7fc014124c30 2019-12-05T05:17:18.519 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:18.505+0000 7fc013fff700 1 --2- 172.21.15.42:0/2923259356 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc014124c30 0x7fc014116f40 secure :-1 s=THROTTLE_DONE pgs=495 cs=0 l=1 rx=0x7fc00c008b10 tx=0x7fc00c007960).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:18.519 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:18.505+0000 7fc010ff9700 1 --2- 172.21.15.42:0/2923259356 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbff803ace0 0x7fbff803d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:17:18.519 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:18.505+0000 7fc010ff9700 1 -- 172.21.15.42:0/2923259356 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6891+0+0 (secure 0 0 0) 0x7fc00c04afe0 con 0x7fc014124c30 2019-12-05T05:17:18.519 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:18.505+0000 7fc0137fe700 1 --2- 172.21.15.42:0/2923259356 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbff803ace0 0x7fbff803d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:17:18.520 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:18.505+0000 7fc0137fe700 1 --2- 172.21.15.42:0/2923259356 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbff803ace0 0x7fbff803d1f0 crc :-1 s=READY pgs=278 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:17:18.520 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:18.509+0000 7fc01a12a700 1 -- 172.21.15.42:0/2923259356 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc01405d110 con 0x7fc014124c30 2019-12-05T05:17:18.524 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:18.521+0000 7fc013fff700 1 --2- 172.21.15.42:0/2923259356 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc014124c30 0x7fc014116f40 secure :-1 s=THROTTLE_DONE pgs=495 cs=0 l=1 rx=0x7fc00c008b10 tx=0x7fc00c007960).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:18.524 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:18.521+0000 7fc010ff9700 1 -- 172.21.15.42:0/2923259356 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fc00c032060 con 0x7fc014124c30 2019-12-05T05:17:18.694 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:18.689+0000 7fc01a12a700 1 -- 172.21.15.42:0/2923259356 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc01405b8c0 con 0x7fc014124c30 2019-12-05T05:17:18.695 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:18.689+0000 7fc013fff700 1 --2- 172.21.15.42:0/2923259356 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc014124c30 0x7fc014116f40 secure :-1 s=THROTTLE_DONE pgs=495 cs=0 l=1 rx=0x7fc00c008b10 tx=0x7fc00c007960).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:18.695 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:18.689+0000 7fc010ff9700 1 -- 172.21.15.42:0/2923259356 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+1772 (secure 0 0 0) 0x7fc00c0322c0 con 0x7fc014124c30 2019-12-05T05:17:18.697 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 82 2019-12-05T05:17:18.697 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:17:18.699 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":82,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:17:17.949394+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5844},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5844":{"gid":5844,"name":"b","rank":0,"incarnation":79,"state":"up:active","state_seq":32,"addr":"172.21.15.40:6833/2698421979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2698421979},{"type":"v1","addr":"172.21.15.40:6833","nonce":2698421979}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:17:18.701 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:18.693+0000 7fc01a12a700 1 -- 172.21.15.42:0/2923259356 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbff803ace0 msgr2=0x7fbff803d1f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:17:18.701 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:18.693+0000 7fc01a12a700 1 --2- 172.21.15.42:0/2923259356 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbff803ace0 0x7fbff803d1f0 crc :-1 s=READY pgs=278 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:17:18.701 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:18.693+0000 7fc01a12a700 1 -- 172.21.15.42:0/2923259356 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc014124c30 msgr2=0x7fc014116f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:17:18.701 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:18.693+0000 7fc01a12a700 1 --2- 172.21.15.42:0/2923259356 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc014124c30 0x7fc014116f40 secure :-1 s=READY pgs=495 cs=0 l=1 rx=0x7fc00c008b10 tx=0x7fc00c007960).stop 2019-12-05T05:17:18.702 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:18.693+0000 7fc01a12a700 1 -- 172.21.15.42:0/2923259356 shutdown_connections 2019-12-05T05:17:18.702 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:18.693+0000 7fc01a12a700 1 --2- 172.21.15.42:0/2923259356 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbff803ace0 0x7fbff803d1f0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:17:18.703 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:18.693+0000 7fc01a12a700 1 --2- 172.21.15.42:0/2923259356 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc0140a5660 0x7fc0140a5a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:17:18.703 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:18.693+0000 7fc01a12a700 1 --2- 172.21.15.42:0/2923259356 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc014124c30 0x7fc014116f40 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:17:18.703 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:18.697+0000 7fc01a12a700 1 -- 172.21.15.42:0/2923259356 shutdown_connections 2019-12-05T05:17:18.703 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:18.697+0000 7fc01a12a700 1 -- 172.21.15.42:0/2923259356 wait complete. 2019-12-05T05:17:18.704 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:18.697+0000 7fc01a12a700 1 -- 172.21.15.42:0/2923259356 >> 172.21.15.42:0/2923259356 conn(0x7fc0140a7740 msgr2=0x7fc0140a9640 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:17:18.710 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b has gained rank=0, replacing gid=5629 2019-12-05T05:17:18.710 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 6 secs before reviving mds.a 2019-12-05T05:17:24.711 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.a 2019-12-05T05:17:24.711 INFO:tasks.ceph.mds.a:Restarting daemon 2019-12-05T05:17:24.712 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2019-12-05T05:17:24.715 INFO:tasks.ceph.mds.a:Started 2019-12-05T05:17:24.715 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:17:24.777 INFO:tasks.ceph.mds.a.smithi042.stdout:starting mds.a at 2019-12-05T05:17:24.817 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:24.805+0000 7ff5d4545700 1 Processor -- start 2019-12-05T05:17:24.818 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:24.809+0000 7ff5d4545700 1 -- start start 2019-12-05T05:17:24.818 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:24.809+0000 7ff5d4545700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff5cc09fae0 0x7ff5cc14fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:17:24.818 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:24.809+0000 7ff5d4545700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5cc0a0710 con 0x7ff5cc08df80 2019-12-05T05:17:24.818 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:24.809+0000 7ff5d4545700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5cc07ef20 con 0x7ff5cc0a7740 2019-12-05T05:17:24.818 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:24.809+0000 7ff5d4545700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5cc0a0b00 con 0x7ff5cc09fae0 2019-12-05T05:17:24.818 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:24.809+0000 7ff5d2ae2700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7ff5cc0a7740 0x7ff5cc07e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:59156/0 (socket says 172.21.15.42:59156) 2019-12-05T05:17:24.819 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:24.809+0000 7ff5d2ae2700 1 -- 172.21.15.42:0/2910808952 learned_addr learned my addr 172.21.15.42:0/2910808952 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:17:24.819 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:24.809+0000 7ff5d32e3700 1 --2- 172.21.15.42:0/2910808952 >> v2:172.21.15.40:3300/0 conn(0x7ff5cc09fae0 0x7ff5cc14fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:17:24.819 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:24.809+0000 7ff5d1ae0700 1 -- 172.21.15.42:0/2910808952 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1751969119 0 0) 0x7ff5cc07ef20 con 0x7ff5cc0a7740 2019-12-05T05:17:24.819 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:24.809+0000 7ff5d1ae0700 1 -- 172.21.15.42:0/2910808952 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5b40030b0 con 0x7ff5cc0a7740 2019-12-05T05:17:24.819 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:24.809+0000 7ff5d1ae0700 1 -- 172.21.15.42:0/2910808952 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3422158388 0 0) 0x7ff5b40030b0 con 0x7ff5cc0a7740 2019-12-05T05:17:24.819 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:24.809+0000 7ff5d1ae0700 1 -- 172.21.15.42:0/2910808952 >> v2:172.21.15.40:3300/0 conn(0x7ff5cc09fae0 msgr2=0x7ff5cc14fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:17:24.819 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:24.809+0000 7ff5d1ae0700 1 --2- 172.21.15.42:0/2910808952 >> v2:172.21.15.40:3300/0 conn(0x7ff5cc09fae0 0x7ff5cc14fff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:17:24.820 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:24.809+0000 7ff5d1ae0700 1 -- 172.21.15.42:0/2910808952 >> v1:172.21.15.40:6789/0 conn(0x7ff5cc08df80 legacy=0x7ff5cc09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:17:24.820 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:24.809+0000 7ff5d1ae0700 1 -- 172.21.15.42:0/2910808952 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5cc0a9ba0 con 0x7ff5cc0a7740 2019-12-05T05:17:24.820 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:24.809+0000 7ff5d1ae0700 1 -- 172.21.15.42:0/2910808952 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7ff5bc001ad0 con 0x7ff5cc0a7740 2019-12-05T05:17:24.820 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:24.809+0000 7ff5d1ae0700 1 -- 172.21.15.42:0/2910808952 >> v1:172.21.15.42:6789/0 conn(0x7ff5cc0a7740 legacy=0x7ff5cc07e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:17:24.820 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:24.809+0000 7ff5d1ae0700 1 --2- 172.21.15.42:0/2910808952 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff5b40036e0 0x7ff5b4003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:17:24.820 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:24.809+0000 7ff5d1ae0700 1 --2- 172.21.15.42:0/2910808952 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff5b4004420 0x7ff5b4004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:17:24.820 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:24.809+0000 7ff5d1ae0700 1 --2- 172.21.15.42:0/2910808952 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff5b4004da0 0x7ff5b400b670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:17:24.821 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:24.809+0000 7ff5d1ae0700 1 -- 172.21.15.42:0/2910808952 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5cc0a0b00 con 0x7ff5b40036e0 2019-12-05T05:17:24.821 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:24.809+0000 7ff5d1ae0700 1 -- 172.21.15.42:0/2910808952 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5b404fc60 con 0x7ff5b4004da0 2019-12-05T05:17:24.821 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:24.809+0000 7ff5d1ae0700 1 -- 172.21.15.42:0/2910808952 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5b404fea0 con 0x7ff5b4004420 2019-12-05T05:17:24.821 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:24.809+0000 7ff5d32e3700 1 --2- 172.21.15.42:0/2910808952 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff5b4004da0 0x7ff5b400b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:17:24.821 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:24.809+0000 7ff5d2ae2700 1 --2- 172.21.15.42:0/2910808952 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff5b40036e0 0x7ff5b4003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:17:24.821 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:24.809+0000 7ff5d22e1700 1 --2- 172.21.15.42:0/2910808952 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff5b4004420 0x7ff5b4004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:17:24.821 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:24.809+0000 7ff5d32e3700 1 -- 172.21.15.42:0/2910808952 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff5b4004420 msgr2=0x7ff5b4004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:17:24.822 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:24.809+0000 7ff5d32e3700 1 --2- 172.21.15.42:0/2910808952 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff5b4004420 0x7ff5b4004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:17:24.822 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:24.809+0000 7ff5d32e3700 1 -- 172.21.15.42:0/2910808952 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff5b40036e0 msgr2=0x7ff5b4003b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:17:24.822 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:24.809+0000 7ff5d32e3700 1 --2- 172.21.15.42:0/2910808952 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff5b40036e0 0x7ff5b4003b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:17:24.822 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:24.809+0000 7ff5d32e3700 1 -- 172.21.15.42:0/2910808952 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5b40501a0 con 0x7ff5b4004da0 2019-12-05T05:17:24.822 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:24.809+0000 7ff5d32e3700 1 --2- 172.21.15.42:0/2910808952 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff5b4004da0 0x7ff5b400b670 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7ff5c4002f30 tx=0x7ff5c400ffa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:24.822 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:24.809+0000 7ff5d32e3700 1 -- 172.21.15.42:0/2910808952 reap_dead start 2019-12-05T05:17:24.822 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:24.809+0000 7ff5d32e3700 1 --2- 172.21.15.42:0/2910808952 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff5b4004da0 0x7ff5b400b670 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7ff5c4002f30 tx=0x7ff5c400ffa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:24.823 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:24.809+0000 7ff5d32e3700 1 --2- 172.21.15.42:0/2910808952 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff5b4004da0 0x7ff5b400b670 secure :-1 s=READY pgs=624 cs=0 l=1 rx=0x7ff5c4002f30 tx=0x7ff5c400ffa0).ready entity=mon.1 client_cookie=a485a5c6f2797679 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:17:24.823 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:24.809+0000 7ff5d32e3700 1 --2- 172.21.15.42:0/2910808952 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff5b4004da0 0x7ff5b400b670 secure :-1 s=THROTTLE_DONE pgs=624 cs=0 l=1 rx=0x7ff5c4002f30 tx=0x7ff5c400ffa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:24.823 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:24.809+0000 7ff5d1ae0700 1 -- 172.21.15.42:0/2910808952 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff5c4010cd0 con 0x7ff5b4004da0 2019-12-05T05:17:24.823 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:24.809+0000 7ff5d32e3700 1 --2- 172.21.15.42:0/2910808952 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff5b4004da0 0x7ff5b400b670 secure :-1 s=THROTTLE_DONE pgs=624 cs=0 l=1 rx=0x7ff5c4002f30 tx=0x7ff5c400ffa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:24.823 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:24.809+0000 7ff5d32e3700 1 --2- 172.21.15.42:0/2910808952 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff5b4004da0 0x7ff5b400b670 secure :-1 s=THROTTLE_DONE pgs=624 cs=0 l=1 rx=0x7ff5c4002f30 tx=0x7ff5c400ffa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:24.823 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:24.809+0000 7ff5d1ae0700 1 -- 172.21.15.42:0/2910808952 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5c40108f0 con 0x7ff5b4004da0 2019-12-05T05:17:24.823 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:24.809+0000 7ff5d1ae0700 1 -- 172.21.15.42:0/2910808952 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff5c4008bb0 con 0x7ff5b4004da0 2019-12-05T05:17:24.824 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:24.809+0000 7ff5d4545700 1 -- 172.21.15.42:0/2910808952 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff5b4004da0 msgr2=0x7ff5b400b670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:17:24.824 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:24.809+0000 7ff5d4545700 1 --2- 172.21.15.42:0/2910808952 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff5b4004da0 0x7ff5b400b670 secure :-1 s=READY pgs=624 cs=0 l=1 rx=0x7ff5c4002f30 tx=0x7ff5c400ffa0).stop 2019-12-05T05:17:24.824 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:24.813+0000 7ff5d4545700 1 -- 172.21.15.42:0/2910808952 shutdown_connections 2019-12-05T05:17:24.824 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:24.813+0000 7ff5d4545700 1 --2- 172.21.15.42:0/2910808952 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff5b4004da0 0x7ff5b400b670 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:17:24.824 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:24.813+0000 7ff5d4545700 1 -- 172.21.15.42:0/2910808952 shutdown_connections 2019-12-05T05:17:24.824 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:24.813+0000 7ff5d4545700 1 -- 172.21.15.42:0/2910808952 wait complete. 2019-12-05T05:17:24.824 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:24.813+0000 7ff5d4545700 1 -- 172.21.15.42:0/2910808952 >> 172.21.15.42:0/2910808952 conn(0x7ff5cc0aabe0 msgr2=0x7ff5cc0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:17:24.827 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:24.813+0000 7ff5d4545700 1 Processor -- start 2019-12-05T05:17:24.827 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:24.813+0000 7ff5d4545700 1 -- start start 2019-12-05T05:17:24.828 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:24.817+0000 7ff5d4545700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff5cc0a2060 0x7ff5cc0a30b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:17:24.828 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:24.817+0000 7ff5d4545700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff5cc144a70 0x7ff5cc144ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:17:24.828 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:24.817+0000 7ff5d4545700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5cc0a4530 con 0x7ff5cc130390 2019-12-05T05:17:24.828 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:24.817+0000 7ff5d4545700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5cc0a4840 con 0x7ff5cc144a70 2019-12-05T05:17:24.828 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:24.817+0000 7ff5d4545700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5cc0a3cc0 con 0x7ff5cc0a2060 2019-12-05T05:17:24.828 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:24.817+0000 7ff5d2ae2700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff5cc0a2060 0x7ff5cc0a30b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:17:24.829 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:24.817+0000 7ff5d2ae2700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff5cc0a2060 0x7ff5cc0a30b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:59596/0 (socket says 172.21.15.42:59596) 2019-12-05T05:17:24.829 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:24.817+0000 7ff5d2ae2700 1 -- 172.21.15.42:0/3725690325 learned_addr learned my addr 172.21.15.42:0/3725690325 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:17:24.829 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:24.817+0000 7ff5d22e1700 1 --2- 172.21.15.42:0/3725690325 >> v2:172.21.15.40:3300/0 conn(0x7ff5cc144a70 0x7ff5cc144ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:17:24.829 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:24.817+0000 7ff5d2ae2700 1 -- 172.21.15.42:0/3725690325 >> v2:172.21.15.40:3300/0 conn(0x7ff5cc144a70 msgr2=0x7ff5cc144ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:17:24.829 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:24.817+0000 7ff5d2ae2700 1 --2- 172.21.15.42:0/3725690325 >> v2:172.21.15.40:3300/0 conn(0x7ff5cc144a70 0x7ff5cc144ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:17:24.829 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:24.817+0000 7ff5d2ae2700 1 -- 172.21.15.42:0/3725690325 >> v1:172.21.15.40:6789/0 conn(0x7ff5cc130390 legacy=0x7ff5cc1307c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:17:24.829 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:24.817+0000 7ff5d2ae2700 1 -- 172.21.15.42:0/3725690325 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5cc1453f0 con 0x7ff5cc0a2060 2019-12-05T05:17:24.830 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:24.817+0000 7ff5d2ae2700 1 --2- 172.21.15.42:0/3725690325 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff5cc0a2060 0x7ff5cc0a30b0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7ff5bc001d90 tx=0x7ff5bc010de0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:24.830 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:24.817+0000 7ff5d2ae2700 1 --2- 172.21.15.42:0/3725690325 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff5cc0a2060 0x7ff5cc0a30b0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7ff5bc001d90 tx=0x7ff5bc010de0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:24.830 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:24.817+0000 7ff5d2ae2700 1 --2- 172.21.15.42:0/3725690325 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff5cc0a2060 0x7ff5cc0a30b0 secure :-1 s=READY pgs=125 cs=0 l=1 rx=0x7ff5bc001d90 tx=0x7ff5bc010de0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:17:24.830 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:24.817+0000 7ff5d2ae2700 1 --2- 172.21.15.42:0/3725690325 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff5cc0a2060 0x7ff5cc0a30b0 secure :-1 s=THROTTLE_DONE pgs=125 cs=0 l=1 rx=0x7ff5bc001d90 tx=0x7ff5bc010de0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:24.830 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:24.817+0000 7ff5cb7fe700 1 -- 172.21.15.42:0/3725690325 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff5bc0046f0 con 0x7ff5cc0a2060 2019-12-05T05:17:24.830 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:24.817+0000 7ff5d2ae2700 1 --2- 172.21.15.42:0/3725690325 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff5cc0a2060 0x7ff5cc0a30b0 secure :-1 s=THROTTLE_DONE pgs=125 cs=0 l=1 rx=0x7ff5bc001d90 tx=0x7ff5bc010de0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:24.830 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:24.817+0000 7ff5d2ae2700 1 --2- 172.21.15.42:0/3725690325 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff5cc0a2060 0x7ff5cc0a30b0 secure :-1 s=THROTTLE_DONE pgs=125 cs=0 l=1 rx=0x7ff5bc001d90 tx=0x7ff5bc010de0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:24.831 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:24.817+0000 7ff5cb7fe700 1 -- 172.21.15.42:0/3725690325 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5bc00dbe0 con 0x7ff5cc0a2060 2019-12-05T05:17:24.831 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:24.817+0000 7ff5cb7fe700 1 -- 172.21.15.42:0/3725690325 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff5bc004cd0 con 0x7ff5cc0a2060 2019-12-05T05:17:24.831 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:24.817+0000 7ff5d4545700 1 -- 172.21.15.42:0/3725690325 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5cc130fc0 con 0x7ff5cc0a2060 2019-12-05T05:17:24.831 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:24.817+0000 7ff5d4545700 1 -- 172.21.15.42:0/3725690325 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5cc137ab0 con 0x7ff5cc0a2060 2019-12-05T05:17:24.831 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:24.821+0000 7ff5d2ae2700 1 --2- 172.21.15.42:0/3725690325 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff5cc0a2060 0x7ff5cc0a30b0 secure :-1 s=THROTTLE_DONE pgs=125 cs=0 l=1 rx=0x7ff5bc001d90 tx=0x7ff5bc010de0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:24.831 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:24.821+0000 7ff5cb7fe700 1 -- 172.21.15.42:0/3725690325 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7ff5bc0116f0 con 0x7ff5cc0a2060 2019-12-05T05:17:24.831 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:24.821+0000 7ff5d2ae2700 1 --2- 172.21.15.42:0/3725690325 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff5cc0a2060 0x7ff5cc0a30b0 secure :-1 s=THROTTLE_DONE pgs=125 cs=0 l=1 rx=0x7ff5bc001d90 tx=0x7ff5bc010de0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:24.832 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:24.821+0000 7ff5cb7fe700 1 --2- 172.21.15.42:0/3725690325 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff5ac03ae00 0x7ff5ac03d310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:17:24.832 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:24.821+0000 7ff5cb7fe700 1 -- 172.21.15.42:0/3725690325 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6891+0+0 (secure 0 0 0) 0x7ff5bc04adb0 con 0x7ff5cc0a2060 2019-12-05T05:17:24.832 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:24.821+0000 7ff5d22e1700 1 --2- 172.21.15.42:0/3725690325 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff5ac03ae00 0x7ff5ac03d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:17:24.832 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:24.821+0000 7ff5d22e1700 1 --2- 172.21.15.42:0/3725690325 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff5ac03ae00 0x7ff5ac03d310 crc :-1 s=READY pgs=280 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:17:24.832 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:24.821+0000 7ff5d4545700 1 -- 172.21.15.42:0/3725690325 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5cc05d110 con 0x7ff5cc0a2060 2019-12-05T05:17:24.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:24.833+0000 7ff5d2ae2700 1 --2- 172.21.15.42:0/3725690325 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff5cc0a2060 0x7ff5cc0a30b0 secure :-1 s=THROTTLE_DONE pgs=125 cs=0 l=1 rx=0x7ff5bc001d90 tx=0x7ff5bc010de0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:24.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:24.833+0000 7ff5cb7fe700 1 -- 172.21.15.42:0/3725690325 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7ff5bc031e30 con 0x7ff5cc0a2060 2019-12-05T05:17:25.007 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:25.001+0000 7ff5d4545700 1 -- 172.21.15.42:0/3725690325 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff5cc05b8c0 con 0x7ff5cc0a2060 2019-12-05T05:17:25.008 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:25.005+0000 7ff5d2ae2700 1 --2- 172.21.15.42:0/3725690325 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff5cc0a2060 0x7ff5cc0a30b0 secure :-1 s=THROTTLE_DONE pgs=125 cs=0 l=1 rx=0x7ff5bc001d90 tx=0x7ff5bc010de0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:25.008 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:25.005+0000 7ff5cb7fe700 1 -- 172.21.15.42:0/3725690325 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+1772 (secure 0 0 0) 0x7ff5bc032090 con 0x7ff5cc0a2060 2019-12-05T05:17:25.010 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:17:25.010 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":82,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:17:17.949394+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5844},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5844":{"gid":5844,"name":"b","rank":0,"incarnation":79,"state":"up:active","state_seq":32,"addr":"172.21.15.40:6833/2698421979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2698421979},{"type":"v1","addr":"172.21.15.40:6833","nonce":2698421979}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:17:25.010 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 82 2019-12-05T05:17:25.013 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:25.009+0000 7ff5d4545700 1 -- 172.21.15.42:0/3725690325 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff5ac03ae00 msgr2=0x7ff5ac03d310 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:17:25.013 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:25.009+0000 7ff5d4545700 1 --2- 172.21.15.42:0/3725690325 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff5ac03ae00 0x7ff5ac03d310 crc :-1 s=READY pgs=280 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:17:25.013 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:25.009+0000 7ff5d4545700 1 -- 172.21.15.42:0/3725690325 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff5cc0a2060 msgr2=0x7ff5cc0a30b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:17:25.013 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:25.009+0000 7ff5d4545700 1 --2- 172.21.15.42:0/3725690325 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff5cc0a2060 0x7ff5cc0a30b0 secure :-1 s=READY pgs=125 cs=0 l=1 rx=0x7ff5bc001d90 tx=0x7ff5bc010de0).stop 2019-12-05T05:17:25.013 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:25.009+0000 7ff5d4545700 1 -- 172.21.15.42:0/3725690325 shutdown_connections 2019-12-05T05:17:25.013 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:25.009+0000 7ff5d4545700 1 --2- 172.21.15.42:0/3725690325 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff5ac03ae00 0x7ff5ac03d310 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:17:25.014 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:25.009+0000 7ff5d4545700 1 --2- 172.21.15.42:0/3725690325 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff5cc0a2060 0x7ff5cc0a30b0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:17:25.014 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:25.009+0000 7ff5d4545700 1 --2- 172.21.15.42:0/3725690325 >> v2:172.21.15.40:3300/0 conn(0x7ff5cc144a70 0x7ff5cc144ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:17:25.014 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:25.009+0000 7ff5d4545700 1 -- 172.21.15.42:0/3725690325 shutdown_connections 2019-12-05T05:17:25.014 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:25.009+0000 7ff5d4545700 1 -- 172.21.15.42:0/3725690325 wait complete. 2019-12-05T05:17:25.014 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:25.009+0000 7ff5d4545700 1 -- 172.21.15.42:0/3725690325 >> 172.21.15.42:0/3725690325 conn(0x7ff5cc0a6ab0 msgr2=0x7ff5cc07e700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:17:25.027 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2019-12-05T05:17:27.028 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:17:27.133 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:27.121+0000 7fb21567e700 1 Processor -- start 2019-12-05T05:17:27.134 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:27.121+0000 7fb21567e700 1 -- start start 2019-12-05T05:17:27.134 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:27.125+0000 7fb21567e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb21008df80 0x7fb21009f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:17:27.134 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:27.125+0000 7fb21567e700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb21009fc60 con 0x7fb2100a7740 2019-12-05T05:17:27.134 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:27.125+0000 7fb21567e700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2100a0710 con 0x7fb21009f830 2019-12-05T05:17:27.135 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:27.125+0000 7fb21567e700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2100a0b00 con 0x7fb21008df80 2019-12-05T05:17:27.135 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:27.125+0000 7fb20effd700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb21008df80 0x7fb21009f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:17:27.135 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:27.125+0000 7fb20effd700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb21008df80 0x7fb21009f2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:59142/0 (socket says 172.21.15.42:59142) 2019-12-05T05:17:27.135 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:27.125+0000 7fb20ffff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fb21009f830 0x7fb21014efe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:59182/0 (socket says 172.21.15.42:59182) 2019-12-05T05:17:27.135 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:27.125+0000 7fb20effd700 1 -- 172.21.15.42:0/1538306373 learned_addr learned my addr 172.21.15.42:0/1538306373 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:17:27.135 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:27.125+0000 7fb20e7fc700 1 -- 172.21.15.42:0/1538306373 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 92030630 0 0) 0x7fb21009fc60 con 0x7fb2100a7740 2019-12-05T05:17:27.136 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:27.125+0000 7fb20e7fc700 1 -- 172.21.15.42:0/1538306373 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1f80030b0 con 0x7fb2100a7740 2019-12-05T05:17:27.136 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:27.125+0000 7fb20effd700 1 -- 172.21.15.42:0/1538306373 >> v1:172.21.15.42:6789/0 conn(0x7fb21009f830 legacy=0x7fb21014efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:17:27.136 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:27.125+0000 7fb20effd700 1 -- 172.21.15.42:0/1538306373 >> v1:172.21.15.40:6789/0 conn(0x7fb2100a7740 legacy=0x7fb21007e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:17:27.136 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:27.125+0000 7fb20effd700 1 -- 172.21.15.42:0/1538306373 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb21007ef20 con 0x7fb21008df80 2019-12-05T05:17:27.136 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:27.125+0000 7fb20effd700 1 --2- 172.21.15.42:0/1538306373 >> v2:172.21.15.42:3300/0 conn(0x7fb21008df80 0x7fb21009f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fb204008b10 tx=0x7fb204010030).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:27.136 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:27.125+0000 7fb20effd700 1 --2- 172.21.15.42:0/1538306373 >> v2:172.21.15.42:3300/0 conn(0x7fb21008df80 0x7fb21009f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fb204008b10 tx=0x7fb204010030).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:27.137 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:27.125+0000 7fb20effd700 1 --2- 172.21.15.42:0/1538306373 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb21008df80 0x7fb21009f2e0 secure :-1 s=READY pgs=626 cs=0 l=1 rx=0x7fb204008b10 tx=0x7fb204010030).ready entity=mon.1 client_cookie=1c9dbbedfcc37bcd server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:17:27.137 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:27.125+0000 7fb20effd700 1 --2- 172.21.15.42:0/1538306373 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb21008df80 0x7fb21009f2e0 secure :-1 s=THROTTLE_DONE pgs=626 cs=0 l=1 rx=0x7fb204008b10 tx=0x7fb204010030).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:27.137 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:27.125+0000 7fb20e7fc700 1 -- 172.21.15.42:0/1538306373 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb20400a160 con 0x7fb21008df80 2019-12-05T05:17:27.137 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:27.125+0000 7fb20effd700 1 --2- 172.21.15.42:0/1538306373 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb21008df80 0x7fb21009f2e0 secure :-1 s=THROTTLE_DONE pgs=626 cs=0 l=1 rx=0x7fb204008b10 tx=0x7fb204010030).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:27.137 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:27.125+0000 7fb20effd700 1 --2- 172.21.15.42:0/1538306373 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb21008df80 0x7fb21009f2e0 secure :-1 s=THROTTLE_DONE pgs=626 cs=0 l=1 rx=0x7fb204008b10 tx=0x7fb204010030).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:27.138 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:27.125+0000 7fb20e7fc700 1 -- 172.21.15.42:0/1538306373 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb204011aa0 con 0x7fb21008df80 2019-12-05T05:17:27.138 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:27.125+0000 7fb20e7fc700 1 -- 172.21.15.42:0/1538306373 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb20400a740 con 0x7fb21008df80 2019-12-05T05:17:27.138 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:27.125+0000 7fb21567e700 1 -- 172.21.15.42:0/1538306373 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb21008df80 msgr2=0x7fb21009f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:17:27.138 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:27.125+0000 7fb21567e700 1 --2- 172.21.15.42:0/1538306373 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb21008df80 0x7fb21009f2e0 secure :-1 s=READY pgs=626 cs=0 l=1 rx=0x7fb204008b10 tx=0x7fb204010030).stop 2019-12-05T05:17:27.139 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:27.129+0000 7fb21567e700 1 -- 172.21.15.42:0/1538306373 shutdown_connections 2019-12-05T05:17:27.139 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:27.129+0000 7fb21567e700 1 --2- 172.21.15.42:0/1538306373 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb21008df80 0x7fb21009f2e0 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:17:27.139 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:27.129+0000 7fb21567e700 1 -- 172.21.15.42:0/1538306373 shutdown_connections 2019-12-05T05:17:27.139 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:27.129+0000 7fb21567e700 1 -- 172.21.15.42:0/1538306373 wait complete. 2019-12-05T05:17:27.140 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:27.129+0000 7fb21567e700 1 -- 172.21.15.42:0/1538306373 >> 172.21.15.42:0/1538306373 conn(0x7fb2100aabe0 msgr2=0x7fb2100a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:17:27.143 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:27.129+0000 7fb21567e700 1 Processor -- start 2019-12-05T05:17:27.143 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:27.129+0000 7fb21567e700 1 -- start start 2019-12-05T05:17:27.143 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:27.133+0000 7fb21567e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb2101187d0 0x7fb21007d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:17:27.144 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:27.133+0000 7fb21567e700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb210128ee0 0x7fb210129310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:17:27.144 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:27.133+0000 7fb21567e700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb21007d920 con 0x7fb210063190 2019-12-05T05:17:27.144 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:27.133+0000 7fb21567e700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2100636a0 con 0x7fb210128ee0 2019-12-05T05:17:27.144 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:27.133+0000 7fb21567e700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb210128c40 con 0x7fb2101187d0 2019-12-05T05:17:27.144 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:27.133+0000 7fb20effd700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb2101187d0 0x7fb21007d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:17:27.144 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:27.133+0000 7fb20effd700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb2101187d0 0x7fb21007d3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:59148/0 (socket says 172.21.15.42:59148) 2019-12-05T05:17:27.145 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:27.133+0000 7fb20effd700 1 -- 172.21.15.42:0/1680338898 learned_addr learned my addr 172.21.15.42:0/1680338898 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:17:27.145 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:27.133+0000 7fb20ffff700 1 --2- 172.21.15.42:0/1680338898 >> v2:172.21.15.40:3300/0 conn(0x7fb210128ee0 0x7fb210129310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:17:27.145 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:27.133+0000 7fb20effd700 1 -- 172.21.15.42:0/1680338898 >> v2:172.21.15.40:3300/0 conn(0x7fb210128ee0 msgr2=0x7fb210129310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:17:27.145 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:27.133+0000 7fb20effd700 1 --2- 172.21.15.42:0/1680338898 >> v2:172.21.15.40:3300/0 conn(0x7fb210128ee0 0x7fb210129310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:17:27.145 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:27.133+0000 7fb20effd700 1 -- 172.21.15.42:0/1680338898 >> v1:172.21.15.40:6789/0 conn(0x7fb210063190 legacy=0x7fb21007aca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:17:27.145 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:27.133+0000 7fb20effd700 1 -- 172.21.15.42:0/1680338898 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb210129860 con 0x7fb2101187d0 2019-12-05T05:17:27.145 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:27.133+0000 7fb20effd700 1 --2- 172.21.15.42:0/1680338898 >> v2:172.21.15.42:3300/0 conn(0x7fb2101187d0 0x7fb21007d3d0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fb2100a7020 tx=0x7fb204011d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:27.146 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:27.133+0000 7fb20effd700 1 --2- 172.21.15.42:0/1680338898 >> v2:172.21.15.42:3300/0 conn(0x7fb2101187d0 0x7fb21007d3d0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fb2100a7020 tx=0x7fb204011d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:27.146 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:27.133+0000 7fb20effd700 1 --2- 172.21.15.42:0/1680338898 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb2101187d0 0x7fb21007d3d0 secure :-1 s=READY pgs=627 cs=0 l=1 rx=0x7fb2100a7020 tx=0x7fb204011d50).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:17:27.146 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:27.133+0000 7fb20effd700 1 --2- 172.21.15.42:0/1680338898 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb2101187d0 0x7fb21007d3d0 secure :-1 s=THROTTLE_DONE pgs=627 cs=0 l=1 rx=0x7fb2100a7020 tx=0x7fb204011d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:27.146 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:27.133+0000 7fb20effd700 1 --2- 172.21.15.42:0/1680338898 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb2101187d0 0x7fb21007d3d0 secure :-1 s=THROTTLE_DONE pgs=627 cs=0 l=1 rx=0x7fb2100a7020 tx=0x7fb204011d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:27.146 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:27.133+0000 7fb1f7fff700 1 -- 172.21.15.42:0/1680338898 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb2040058c0 con 0x7fb2101187d0 2019-12-05T05:17:27.146 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:27.133+0000 7fb20effd700 1 --2- 172.21.15.42:0/1680338898 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb2101187d0 0x7fb21007d3d0 secure :-1 s=THROTTLE_DONE pgs=627 cs=0 l=1 rx=0x7fb2100a7020 tx=0x7fb204011d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:27.146 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:27.133+0000 7fb1f7fff700 1 -- 172.21.15.42:0/1680338898 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb20400a360 con 0x7fb2101187d0 2019-12-05T05:17:27.147 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:27.133+0000 7fb1f7fff700 1 -- 172.21.15.42:0/1680338898 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb204007150 con 0x7fb2101187d0 2019-12-05T05:17:27.147 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:27.133+0000 7fb21567e700 1 -- 172.21.15.42:0/1680338898 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb21007b4a0 con 0x7fb2101187d0 2019-12-05T05:17:27.147 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:27.133+0000 7fb21567e700 1 -- 172.21.15.42:0/1680338898 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb21012ff10 con 0x7fb2101187d0 2019-12-05T05:17:27.147 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:27.133+0000 7fb20effd700 1 --2- 172.21.15.42:0/1680338898 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb2101187d0 0x7fb21007d3d0 secure :-1 s=THROTTLE_DONE pgs=627 cs=0 l=1 rx=0x7fb2100a7020 tx=0x7fb204011d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:27.147 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:27.137+0000 7fb1f7fff700 1 -- 172.21.15.42:0/1680338898 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fb204007460 con 0x7fb2101187d0 2019-12-05T05:17:27.147 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:27.137+0000 7fb20effd700 1 --2- 172.21.15.42:0/1680338898 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb2101187d0 0x7fb21007d3d0 secure :-1 s=THROTTLE_DONE pgs=627 cs=0 l=1 rx=0x7fb2100a7020 tx=0x7fb204011d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:27.148 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:27.137+0000 7fb1f7fff700 1 --2- 172.21.15.42:0/1680338898 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb1f003ada0 0x7fb1f003d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:17:27.148 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:27.137+0000 7fb1f7fff700 1 -- 172.21.15.42:0/1680338898 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6891+0+0 (secure 0 0 0) 0x7fb20404bc50 con 0x7fb2101187d0 2019-12-05T05:17:27.148 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:27.137+0000 7fb20f7fe700 1 --2- 172.21.15.42:0/1680338898 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb1f003ada0 0x7fb1f003d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:17:27.149 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:27.137+0000 7fb20f7fe700 1 --2- 172.21.15.42:0/1680338898 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb1f003ada0 0x7fb1f003d2b0 crc :-1 s=READY pgs=282 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:17:27.149 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:27.137+0000 7fb21567e700 1 -- 172.21.15.42:0/1680338898 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb21005d110 con 0x7fb2101187d0 2019-12-05T05:17:27.149 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:27.145+0000 7fb20effd700 1 --2- 172.21.15.42:0/1680338898 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb2101187d0 0x7fb21007d3d0 secure :-1 s=THROTTLE_DONE pgs=627 cs=0 l=1 rx=0x7fb2100a7020 tx=0x7fb204011d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:27.150 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:27.145+0000 7fb1f7fff700 1 -- 172.21.15.42:0/1680338898 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fb204030980 con 0x7fb2101187d0 2019-12-05T05:17:27.316 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:27.313+0000 7fb21567e700 1 -- 172.21.15.42:0/1680338898 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb21005b8c0 con 0x7fb2101187d0 2019-12-05T05:17:27.317 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:27.313+0000 7fb20effd700 1 --2- 172.21.15.42:0/1680338898 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb2101187d0 0x7fb21007d3d0 secure :-1 s=THROTTLE_DONE pgs=627 cs=0 l=1 rx=0x7fb2100a7020 tx=0x7fb204011d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:17:27.317 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:27.313+0000 7fb1f7fff700 1 -- 172.21.15.42:0/1680338898 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+2108 (secure 0 0 0) 0x7fb204033ae0 con 0x7fb2101187d0 2019-12-05T05:17:27.318 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 83 2019-12-05T05:17:27.318 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:17:27.318 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":83,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5833,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6835/1414371821","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1414371821},{"type":"v1","addr":"172.21.15.42:6835","nonce":1414371821}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":83}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:17:17.949394+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5844},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5844":{"gid":5844,"name":"b","rank":0,"incarnation":79,"state":"up:active","state_seq":32,"addr":"172.21.15.40:6833/2698421979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2698421979},{"type":"v1","addr":"172.21.15.40:6833","nonce":2698421979}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:17:27.320 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:27.313+0000 7fb21567e700 1 -- 172.21.15.42:0/1680338898 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb1f003ada0 msgr2=0x7fb1f003d2b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:17:27.320 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:27.313+0000 7fb21567e700 1 --2- 172.21.15.42:0/1680338898 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb1f003ada0 0x7fb1f003d2b0 crc :-1 s=READY pgs=282 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:17:27.321 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:27.313+0000 7fb21567e700 1 -- 172.21.15.42:0/1680338898 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb2101187d0 msgr2=0x7fb21007d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:17:27.321 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:27.313+0000 7fb21567e700 1 --2- 172.21.15.42:0/1680338898 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb2101187d0 0x7fb21007d3d0 secure :-1 s=READY pgs=627 cs=0 l=1 rx=0x7fb2100a7020 tx=0x7fb204011d50).stop 2019-12-05T05:17:27.321 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:27.313+0000 7fb21567e700 1 -- 172.21.15.42:0/1680338898 shutdown_connections 2019-12-05T05:17:27.321 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:27.313+0000 7fb21567e700 1 --2- 172.21.15.42:0/1680338898 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb1f003ada0 0x7fb1f003d2b0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:17:27.321 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:27.313+0000 7fb21567e700 1 --2- 172.21.15.42:0/1680338898 >> v2:172.21.15.40:3300/0 conn(0x7fb210128ee0 0x7fb210129310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:17:27.322 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:27.313+0000 7fb21567e700 1 --2- 172.21.15.42:0/1680338898 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb2101187d0 0x7fb21007d3d0 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:17:27.322 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:27.313+0000 7fb21567e700 1 -- 172.21.15.42:0/1680338898 shutdown_connections 2019-12-05T05:17:27.322 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:27.317+0000 7fb21567e700 1 -- 172.21.15.42:0/1680338898 wait complete. 2019-12-05T05:17:27.322 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:17:27.317+0000 7fb21567e700 1 -- 172.21.15.42:0/1680338898 >> 172.21.15.42:0/1680338898 conn(0x7fb2100aabe0 msgr2=0x7fb210059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:17:27.330 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a reported in up:standby state 2019-12-05T05:17:27.330 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 67 secs before thrashing 2019-12-05T05:17:35.906 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:17:35.910 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:18:05.955 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:18:05.959 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:18:34.332 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:18:34.448 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.437+0000 7f76ba3b4700 1 Processor -- start 2019-12-05T05:18:34.449 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.437+0000 7f76ba3b4700 1 -- start start 2019-12-05T05:18:34.449 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.441+0000 7f76ba3b4700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f76b40a7740 0x7f76b408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:18:34.449 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.441+0000 7f76ba3b4700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f76b407e720 0x7f76b407eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:18:34.449 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.441+0000 7f76ba3b4700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76b40a0710 con 0x7f76b409f2e0 2019-12-05T05:18:34.449 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.441+0000 7f76ba3b4700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76b40a0b00 con 0x7f76b407e720 2019-12-05T05:18:34.449 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.441+0000 7f76ba3b4700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76b408e5b0 con 0x7f76b40a7740 2019-12-05T05:18:34.449 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.441+0000 7f76b8951700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f76b40a7740 0x7f76b408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:18:34.450 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.441+0000 7f76b3fff700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f76b407e720 0x7f76b407eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:18:34.450 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.441+0000 7f76b8951700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f76b40a7740 0x7f76b408df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:59638/0 (socket says 172.21.15.42:59638) 2019-12-05T05:18:34.450 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.441+0000 7f76b8951700 1 -- 172.21.15.42:0/639459333 learned_addr learned my addr 172.21.15.42:0/639459333 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:18:34.450 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.441+0000 7f76b9152700 1 --1- 172.21.15.42:0/639459333 >> v1:172.21.15.40:6789/0 conn(0x7f76b409f2e0 0x7f76b409f710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:40422/0 (socket says 172.21.15.42:40422) 2019-12-05T05:18:34.450 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.441+0000 7f76b8951700 1 -- 172.21.15.42:0/639459333 >> v2:172.21.15.40:3300/0 conn(0x7f76b407e720 msgr2=0x7f76b407eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:18:34.450 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.441+0000 7f76b8951700 1 --2- 172.21.15.42:0/639459333 >> v2:172.21.15.40:3300/0 conn(0x7f76b407e720 0x7f76b407eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:18:34.450 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.441+0000 7f76b8951700 1 -- 172.21.15.42:0/639459333 >> v1:172.21.15.40:6789/0 conn(0x7f76b409f2e0 legacy=0x7f76b409f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:18:34.451 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.441+0000 7f76b8951700 1 -- 172.21.15.42:0/639459333 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76b407f0a0 con 0x7f76b40a7740 2019-12-05T05:18:34.451 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.441+0000 7f76b8951700 1 --2- 172.21.15.42:0/639459333 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f76b40a7740 0x7f76b408df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f76a4008b10 tx=0x7f76a40100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:34.451 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.441+0000 7f76b8951700 1 --2- 172.21.15.42:0/639459333 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f76b40a7740 0x7f76b408df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f76a4008b10 tx=0x7f76a40100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:34.451 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.441+0000 7f76b8951700 1 --2- 172.21.15.42:0/639459333 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f76b40a7740 0x7f76b408df80 secure :-1 s=READY pgs=126 cs=0 l=1 rx=0x7f76a4008b10 tx=0x7f76a40100b0).ready entity=mon.2 client_cookie=17c67f31f277b47e server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:18:34.451 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.441+0000 7f76b8951700 1 --2- 172.21.15.42:0/639459333 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f76b40a7740 0x7f76b408df80 secure :-1 s=THROTTLE_DONE pgs=126 cs=0 l=1 rx=0x7f76a4008b10 tx=0x7f76a40100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:34.451 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.441+0000 7f76b37fe700 1 -- 172.21.15.42:0/639459333 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f76a400a160 con 0x7f76b40a7740 2019-12-05T05:18:34.451 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.441+0000 7f76b8951700 1 --2- 172.21.15.42:0/639459333 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f76b40a7740 0x7f76b408df80 secure :-1 s=THROTTLE_DONE pgs=126 cs=0 l=1 rx=0x7f76a4008b10 tx=0x7f76a40100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:34.452 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.441+0000 7f76b37fe700 1 -- 172.21.15.42:0/639459333 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76a4011a30 con 0x7f76b40a7740 2019-12-05T05:18:34.452 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.441+0000 7f76b8951700 1 --2- 172.21.15.42:0/639459333 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f76b40a7740 0x7f76b408df80 secure :-1 s=THROTTLE_DONE pgs=126 cs=0 l=1 rx=0x7f76a4008b10 tx=0x7f76a40100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:34.452 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.441+0000 7f76b37fe700 1 -- 172.21.15.42:0/639459333 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f76a400a550 con 0x7f76b40a7740 2019-12-05T05:18:34.452 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.441+0000 7f76ba3b4700 1 -- 172.21.15.42:0/639459333 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f76b40a7740 msgr2=0x7f76b408df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:18:34.452 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.441+0000 7f76ba3b4700 1 --2- 172.21.15.42:0/639459333 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f76b40a7740 0x7f76b408df80 secure :-1 s=READY pgs=126 cs=0 l=1 rx=0x7f76a4008b10 tx=0x7f76a40100b0).stop 2019-12-05T05:18:34.452 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.441+0000 7f76ba3b4700 1 -- 172.21.15.42:0/639459333 shutdown_connections 2019-12-05T05:18:34.452 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.441+0000 7f76ba3b4700 1 --2- 172.21.15.42:0/639459333 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f76b40a7740 0x7f76b408df80 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:18:34.453 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.441+0000 7f76ba3b4700 1 --2- 172.21.15.42:0/639459333 >> v2:172.21.15.40:3300/0 conn(0x7f76b407e720 0x7f76b407eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:18:34.453 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.441+0000 7f76ba3b4700 1 -- 172.21.15.42:0/639459333 shutdown_connections 2019-12-05T05:18:34.453 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.441+0000 7f76ba3b4700 1 -- 172.21.15.42:0/639459333 wait complete. 2019-12-05T05:18:34.453 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.441+0000 7f76ba3b4700 1 -- 172.21.15.42:0/639459333 >> 172.21.15.42:0/639459333 conn(0x7f76b40aabe0 msgr2=0x7f76b40a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:18:34.458 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.445+0000 7f76ba3b4700 1 Processor -- start 2019-12-05T05:18:34.458 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.445+0000 7f76ba3b4700 1 -- start start 2019-12-05T05:18:34.458 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.445+0000 7f76ba3b4700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f76b412e570 0x7f76b4078340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:18:34.459 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.445+0000 7f76ba3b4700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f76b411ef00 0x7f76b408f230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:18:34.459 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.445+0000 7f76ba3b4700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f76b4119370 0x7f76b41197a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:18:34.459 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.445+0000 7f76ba3b4700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76b408ef90 con 0x7f76b4119370 2019-12-05T05:18:34.459 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.445+0000 7f76ba3b4700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76b4119cf0 con 0x7f76b412e570 2019-12-05T05:18:34.459 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.445+0000 7f76ba3b4700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76b4078890 con 0x7f76b411ef00 2019-12-05T05:18:34.461 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.449+0000 7f76b8951700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f76b412e570 0x7f76b4078340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:18:34.461 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.449+0000 7f76b8951700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f76b412e570 0x7f76b4078340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:59178/0 (socket says 172.21.15.42:59178) 2019-12-05T05:18:34.462 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.449+0000 7f76b8951700 1 -- 172.21.15.42:0/3910776142 learned_addr learned my addr 172.21.15.42:0/3910776142 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:18:34.462 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.449+0000 7f76b3fff700 1 --2- 172.21.15.42:0/3910776142 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f76b411ef00 0x7f76b408f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:18:34.462 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.449+0000 7f76b9152700 1 --2- 172.21.15.42:0/3910776142 >> v2:172.21.15.40:3300/0 conn(0x7f76b4119370 0x7f76b41197a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:18:34.462 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.449+0000 7f76b8951700 1 -- 172.21.15.42:0/3910776142 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f76b411ef00 msgr2=0x7f76b408f230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:18:34.462 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.449+0000 7f76b8951700 1 --2- 172.21.15.42:0/3910776142 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f76b411ef00 0x7f76b408f230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:18:34.463 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.449+0000 7f76b8951700 1 -- 172.21.15.42:0/3910776142 >> v2:172.21.15.40:3300/0 conn(0x7f76b4119370 msgr2=0x7f76b41197a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:18:34.463 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.449+0000 7f76b8951700 1 --2- 172.21.15.42:0/3910776142 >> v2:172.21.15.40:3300/0 conn(0x7f76b4119370 0x7f76b41197a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:18:34.463 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.449+0000 7f76b8951700 1 -- 172.21.15.42:0/3910776142 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76b408eb70 con 0x7f76b412e570 2019-12-05T05:18:34.463 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.449+0000 7f76b8951700 1 --2- 172.21.15.42:0/3910776142 >> v2:172.21.15.42:3300/0 conn(0x7f76b412e570 0x7f76b4078340 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f76b4118410 tx=0x7f76a400ab60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:34.463 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.449+0000 7f76b8951700 1 --2- 172.21.15.42:0/3910776142 >> v2:172.21.15.42:3300/0 conn(0x7f76b412e570 0x7f76b4078340 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f76b4118410 tx=0x7f76a400ab60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:34.463 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.449+0000 7f76b8951700 1 --2- 172.21.15.42:0/3910776142 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f76b412e570 0x7f76b4078340 secure :-1 s=READY pgs=628 cs=0 l=1 rx=0x7f76b4118410 tx=0x7f76a400ab60).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:18:34.463 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.449+0000 7f76b8951700 1 --2- 172.21.15.42:0/3910776142 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f76b412e570 0x7f76b4078340 secure :-1 s=THROTTLE_DONE pgs=628 cs=0 l=1 rx=0x7f76b4118410 tx=0x7f76a400ab60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:34.464 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.449+0000 7f76b17fa700 1 -- 172.21.15.42:0/3910776142 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f76a4005ab0 con 0x7f76b412e570 2019-12-05T05:18:34.464 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.449+0000 7f76b8951700 1 --2- 172.21.15.42:0/3910776142 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f76b412e570 0x7f76b4078340 secure :-1 s=THROTTLE_DONE pgs=628 cs=0 l=1 rx=0x7f76b4118410 tx=0x7f76a400ab60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:34.464 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.449+0000 7f76b8951700 1 --2- 172.21.15.42:0/3910776142 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f76b412e570 0x7f76b4078340 secure :-1 s=THROTTLE_DONE pgs=628 cs=0 l=1 rx=0x7f76b4118410 tx=0x7f76a400ab60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:34.464 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.449+0000 7f76b17fa700 1 -- 172.21.15.42:0/3910776142 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76a4011e10 con 0x7f76b412e570 2019-12-05T05:18:34.465 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.449+0000 7f76ba3b4700 1 -- 172.21.15.42:0/3910776142 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76b408f780 con 0x7f76b412e570 2019-12-05T05:18:34.465 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.449+0000 7f76b17fa700 1 -- 172.21.15.42:0/3910776142 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f76a4007340 con 0x7f76b412e570 2019-12-05T05:18:34.465 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.449+0000 7f76ba3b4700 1 -- 172.21.15.42:0/3910776142 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76b4120670 con 0x7f76b412e570 2019-12-05T05:18:34.465 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.453+0000 7f76b8951700 1 --2- 172.21.15.42:0/3910776142 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f76b412e570 0x7f76b4078340 secure :-1 s=THROTTLE_DONE pgs=628 cs=0 l=1 rx=0x7f76b4118410 tx=0x7f76a400ab60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:34.466 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.453+0000 7f76b17fa700 1 -- 172.21.15.42:0/3910776142 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f76a400f360 con 0x7f76b412e570 2019-12-05T05:18:34.466 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.453+0000 7f76b8951700 1 --2- 172.21.15.42:0/3910776142 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f76b412e570 0x7f76b4078340 secure :-1 s=THROTTLE_DONE pgs=628 cs=0 l=1 rx=0x7f76b4118410 tx=0x7f76a400ab60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:34.466 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.453+0000 7f76b17fa700 1 --2- 172.21.15.42:0/3910776142 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f769403adf0 0x7f769403d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:18:34.466 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.453+0000 7f76b17fa700 1 -- 172.21.15.42:0/3910776142 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6891+0+0 (secure 0 0 0) 0x7f76a404bde0 con 0x7f76b412e570 2019-12-05T05:18:34.466 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.453+0000 7f76b3fff700 1 --2- 172.21.15.42:0/3910776142 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f769403adf0 0x7f769403d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:18:34.466 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.453+0000 7f76b3fff700 1 --2- 172.21.15.42:0/3910776142 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f769403adf0 0x7f769403d300 crc :-1 s=READY pgs=286 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:18:34.466 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.453+0000 7f76ba3b4700 1 -- 172.21.15.42:0/3910776142 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76b405d110 con 0x7f76b412e570 2019-12-05T05:18:34.467 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.457+0000 7f76b8951700 1 --2- 172.21.15.42:0/3910776142 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f76b412e570 0x7f76b4078340 secure :-1 s=THROTTLE_DONE pgs=628 cs=0 l=1 rx=0x7f76b4118410 tx=0x7f76a400ab60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:34.467 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.457+0000 7f76b17fa700 1 -- 172.21.15.42:0/3910776142 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f76a4033c50 con 0x7f76b412e570 2019-12-05T05:18:34.629 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.625+0000 7f76ba3b4700 1 -- 172.21.15.42:0/3910776142 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f76b405b8c0 con 0x7f76b412e570 2019-12-05T05:18:34.630 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.625+0000 7f76b8951700 1 --2- 172.21.15.42:0/3910776142 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f76b412e570 0x7f76b4078340 secure :-1 s=THROTTLE_DONE pgs=628 cs=0 l=1 rx=0x7f76b4118410 tx=0x7f76a400ab60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:34.630 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.625+0000 7f76b17fa700 1 -- 172.21.15.42:0/3910776142 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+2108 (secure 0 0 0) 0x7f76a4031a30 con 0x7f76b412e570 2019-12-05T05:18:34.634 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:18:34.635 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":83,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5833,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6835/1414371821","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1414371821},{"type":"v1","addr":"172.21.15.42:6835","nonce":1414371821}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":83}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:17:17.949394+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5844},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5844":{"gid":5844,"name":"b","rank":0,"incarnation":79,"state":"up:active","state_seq":32,"addr":"172.21.15.40:6833/2698421979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2698421979},{"type":"v1","addr":"172.21.15.40:6833","nonce":2698421979}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:18:34.635 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 83 2019-12-05T05:18:34.637 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.629+0000 7f76ba3b4700 1 -- 172.21.15.42:0/3910776142 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f769403adf0 msgr2=0x7f769403d300 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:18:34.637 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.629+0000 7f76ba3b4700 1 --2- 172.21.15.42:0/3910776142 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f769403adf0 0x7f769403d300 crc :-1 s=READY pgs=286 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:18:34.638 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.629+0000 7f76ba3b4700 1 -- 172.21.15.42:0/3910776142 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f76b412e570 msgr2=0x7f76b4078340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:18:34.638 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.629+0000 7f76ba3b4700 1 --2- 172.21.15.42:0/3910776142 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f76b412e570 0x7f76b4078340 secure :-1 s=READY pgs=628 cs=0 l=1 rx=0x7f76b4118410 tx=0x7f76a400ab60).stop 2019-12-05T05:18:34.638 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.629+0000 7f76ba3b4700 1 -- 172.21.15.42:0/3910776142 shutdown_connections 2019-12-05T05:18:34.638 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.629+0000 7f76ba3b4700 1 --2- 172.21.15.42:0/3910776142 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f769403adf0 0x7f769403d300 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:18:34.638 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.629+0000 7f76ba3b4700 1 --2- 172.21.15.42:0/3910776142 >> v2:172.21.15.40:3300/0 conn(0x7f76b4119370 0x7f76b41197a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:18:34.638 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.629+0000 7f76ba3b4700 1 --2- 172.21.15.42:0/3910776142 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f76b412e570 0x7f76b4078340 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:18:34.639 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.629+0000 7f76ba3b4700 1 --2- 172.21.15.42:0/3910776142 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f76b411ef00 0x7f76b408f230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:18:34.639 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.629+0000 7f76ba3b4700 1 -- 172.21.15.42:0/3910776142 shutdown_connections 2019-12-05T05:18:34.639 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.629+0000 7f76ba3b4700 1 -- 172.21.15.42:0/3910776142 wait complete. 2019-12-05T05:18:34.639 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.629+0000 7f76ba3b4700 1 -- 172.21.15.42:0/3910776142 >> 172.21.15.42:0/3910776142 conn(0x7f76b4141a30 msgr2=0x7f76b4059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:18:34.657 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.b (rank=0) 2019-12-05T05:18:34.658 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2019-12-05T05:18:34.664 INFO:teuthology.orchestra.run:waiting for 300 2019-12-05T05:18:34.673 INFO:tasks.ceph.mds.b:Stopped 2019-12-05T05:18:34.674 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:18:34.777 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.765+0000 7f85e0a94700 1 Processor -- start 2019-12-05T05:18:34.777 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.765+0000 7f85e0a94700 1 -- start start 2019-12-05T05:18:34.777 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.765+0000 7f85e0a94700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f85dc09fae0 0x7f85dc14fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:18:34.777 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.765+0000 7f85e0a94700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85dc0a0710 con 0x7f85dc0a7740 2019-12-05T05:18:34.778 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.765+0000 7f85e0a94700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85dc07ef20 con 0x7f85dc08df80 2019-12-05T05:18:34.778 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.765+0000 7f85e0a94700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85dc0a0b00 con 0x7f85dc09fae0 2019-12-05T05:18:34.778 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.765+0000 7f85d259c700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f85dc08df80 0x7f85dc09f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:59228/0 (socket says 172.21.15.42:59228) 2019-12-05T05:18:34.778 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.765+0000 7f85d259c700 1 -- 172.21.15.42:0/271518487 learned_addr learned my addr 172.21.15.42:0/271518487 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:18:34.778 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.765+0000 7f85db59e700 1 --2- 172.21.15.42:0/271518487 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f85dc09fae0 0x7f85dc14fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:18:34.778 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.765+0000 7f85da59c700 1 -- 172.21.15.42:0/271518487 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2433512454 0 0) 0x7f85dc07ef20 con 0x7f85dc08df80 2019-12-05T05:18:34.779 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.765+0000 7f85da59c700 1 -- 172.21.15.42:0/271518487 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85c80030b0 con 0x7f85dc08df80 2019-12-05T05:18:34.779 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.765+0000 7f85da59c700 1 -- 172.21.15.42:0/271518487 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1712460188 0 0) 0x7f85dc0a0710 con 0x7f85dc0a7740 2019-12-05T05:18:34.779 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.765+0000 7f85da59c700 1 -- 172.21.15.42:0/271518487 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85dc07ef20 con 0x7f85dc0a7740 2019-12-05T05:18:34.779 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.765+0000 7f85db59e700 1 -- 172.21.15.42:0/271518487 >> v1:172.21.15.42:6789/0 conn(0x7f85dc08df80 legacy=0x7f85dc09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:18:34.779 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.765+0000 7f85db59e700 1 -- 172.21.15.42:0/271518487 >> v1:172.21.15.40:6789/0 conn(0x7f85dc0a7740 legacy=0x7f85dc07e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:18:34.780 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.769+0000 7f85db59e700 1 -- 172.21.15.42:0/271518487 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85dc0a9ba0 con 0x7f85dc09fae0 2019-12-05T05:18:34.780 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.769+0000 7f85db59e700 1 --2- 172.21.15.42:0/271518487 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f85dc09fae0 0x7f85dc14fff0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f85d40032b0 tx=0x7f85d400ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:34.780 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.769+0000 7f85db59e700 1 --2- 172.21.15.42:0/271518487 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f85dc09fae0 0x7f85dc14fff0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f85d40032b0 tx=0x7f85d400ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:34.780 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.769+0000 7f85db59e700 1 --2- 172.21.15.42:0/271518487 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f85dc09fae0 0x7f85dc14fff0 secure :-1 s=READY pgs=127 cs=0 l=1 rx=0x7f85d40032b0 tx=0x7f85d400ff70).ready entity=mon.2 client_cookie=e54cda6f1c01ae6d server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:18:34.780 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.769+0000 7f85db59e700 1 --2- 172.21.15.42:0/271518487 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f85dc09fae0 0x7f85dc14fff0 secure :-1 s=THROTTLE_DONE pgs=127 cs=0 l=1 rx=0x7f85d40032b0 tx=0x7f85d400ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:34.780 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.769+0000 7f85da59c700 1 -- 172.21.15.42:0/271518487 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f85d40109b0 con 0x7f85dc09fae0 2019-12-05T05:18:34.780 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.769+0000 7f85db59e700 1 --2- 172.21.15.42:0/271518487 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f85dc09fae0 0x7f85dc14fff0 secure :-1 s=THROTTLE_DONE pgs=127 cs=0 l=1 rx=0x7f85d40032b0 tx=0x7f85d400ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:34.781 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.769+0000 7f85da59c700 1 -- 172.21.15.42:0/271518487 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85d4008b50 con 0x7f85dc09fae0 2019-12-05T05:18:34.781 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.769+0000 7f85db59e700 1 --2- 172.21.15.42:0/271518487 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f85dc09fae0 0x7f85dc14fff0 secure :-1 s=THROTTLE_DONE pgs=127 cs=0 l=1 rx=0x7f85d40032b0 tx=0x7f85d400ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:34.781 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.769+0000 7f85da59c700 1 -- 172.21.15.42:0/271518487 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f85d40110c0 con 0x7f85dc09fae0 2019-12-05T05:18:34.781 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.769+0000 7f85e0a94700 1 -- 172.21.15.42:0/271518487 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f85dc09fae0 msgr2=0x7f85dc14fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:18:34.781 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.769+0000 7f85e0a94700 1 --2- 172.21.15.42:0/271518487 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f85dc09fae0 0x7f85dc14fff0 secure :-1 s=READY pgs=127 cs=0 l=1 rx=0x7f85d40032b0 tx=0x7f85d400ff70).stop 2019-12-05T05:18:34.781 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.769+0000 7f85e0a94700 1 -- 172.21.15.42:0/271518487 shutdown_connections 2019-12-05T05:18:34.781 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.769+0000 7f85e0a94700 1 --2- 172.21.15.42:0/271518487 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f85dc09fae0 0x7f85dc14fff0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:18:34.782 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.769+0000 7f85e0a94700 1 -- 172.21.15.42:0/271518487 shutdown_connections 2019-12-05T05:18:34.782 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.769+0000 7f85e0a94700 1 -- 172.21.15.42:0/271518487 wait complete. 2019-12-05T05:18:34.782 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.769+0000 7f85e0a94700 1 -- 172.21.15.42:0/271518487 >> 172.21.15.42:0/271518487 conn(0x7f85dc0aabe0 msgr2=0x7f85dc0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:18:34.782 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.773+0000 7f85e0a94700 1 Processor -- start 2019-12-05T05:18:34.782 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.773+0000 7f85e0a94700 1 -- start start 2019-12-05T05:18:34.786 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.773+0000 7f85e0a94700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f85dc122b40 0x7f85dc11ff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:18:34.786 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.773+0000 7f85e0a94700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f85dc120450 0x7f85dc123200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:18:34.786 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.773+0000 7f85e0a94700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f85dc123750 0x7f85dc123b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:18:34.786 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.773+0000 7f85e0a94700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85dc117830 con 0x7f85dc122b40 2019-12-05T05:18:34.786 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.773+0000 7f85e0a94700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85dc117ad0 con 0x7f85dc123750 2019-12-05T05:18:34.787 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.773+0000 7f85e0a94700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85dc117d70 con 0x7f85dc120450 2019-12-05T05:18:34.787 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.773+0000 7f85db59e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f85dc123750 0x7f85dc123b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:18:34.787 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.773+0000 7f85db59e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f85dc123750 0x7f85dc123b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:59196/0 (socket says 172.21.15.42:59196) 2019-12-05T05:18:34.787 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.773+0000 7f85db59e700 1 -- 172.21.15.42:0/2782043000 learned_addr learned my addr 172.21.15.42:0/2782043000 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:18:34.787 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.773+0000 7f85dad9d700 1 --2- 172.21.15.42:0/2782043000 >> v2:172.21.15.40:3300/0 conn(0x7f85dc122b40 0x7f85dc11ff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:18:34.787 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.773+0000 7f85d259c700 1 --2- 172.21.15.42:0/2782043000 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f85dc120450 0x7f85dc123200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:18:34.787 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.773+0000 7f85db59e700 1 -- 172.21.15.42:0/2782043000 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f85dc120450 msgr2=0x7f85dc123200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:18:34.788 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.773+0000 7f85db59e700 1 --2- 172.21.15.42:0/2782043000 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f85dc120450 0x7f85dc123200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:18:34.788 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.773+0000 7f85db59e700 1 -- 172.21.15.42:0/2782043000 >> v2:172.21.15.40:3300/0 conn(0x7f85dc122b40 msgr2=0x7f85dc11ff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:18:34.788 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.773+0000 7f85db59e700 1 --2- 172.21.15.42:0/2782043000 >> v2:172.21.15.40:3300/0 conn(0x7f85dc122b40 0x7f85dc11ff00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:18:34.788 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.773+0000 7f85db59e700 1 -- 172.21.15.42:0/2782043000 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85dc11ac70 con 0x7f85dc123750 2019-12-05T05:18:34.788 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.773+0000 7f85db59e700 1 --2- 172.21.15.42:0/2782043000 >> v2:172.21.15.42:3300/0 conn(0x7f85dc123750 0x7f85dc123b80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f85dc117c90 tx=0x7f85d4011140).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:34.788 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.773+0000 7f85db59e700 1 --2- 172.21.15.42:0/2782043000 >> v2:172.21.15.42:3300/0 conn(0x7f85dc123750 0x7f85dc123b80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f85dc117c90 tx=0x7f85d4011140).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:34.788 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.773+0000 7f85db59e700 1 --2- 172.21.15.42:0/2782043000 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f85dc123750 0x7f85dc123b80 secure :-1 s=READY pgs=630 cs=0 l=1 rx=0x7f85dc117c90 tx=0x7f85d4011140).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:18:34.789 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.773+0000 7f85db59e700 1 --2- 172.21.15.42:0/2782043000 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f85dc123750 0x7f85dc123b80 secure :-1 s=THROTTLE_DONE pgs=630 cs=0 l=1 rx=0x7f85dc117c90 tx=0x7f85d4011140).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:34.789 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.773+0000 7f85d3fff700 1 -- 172.21.15.42:0/2782043000 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f85d4005ab0 con 0x7f85dc123750 2019-12-05T05:18:34.789 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.773+0000 7f85db59e700 1 --2- 172.21.15.42:0/2782043000 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f85dc123750 0x7f85dc123b80 secure :-1 s=THROTTLE_DONE pgs=630 cs=0 l=1 rx=0x7f85dc117c90 tx=0x7f85d4011140).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:34.789 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.773+0000 7f85db59e700 1 --2- 172.21.15.42:0/2782043000 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f85dc123750 0x7f85dc123b80 secure :-1 s=THROTTLE_DONE pgs=630 cs=0 l=1 rx=0x7f85dc117c90 tx=0x7f85d4011140).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:34.789 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.773+0000 7f85d3fff700 1 -- 172.21.15.42:0/2782043000 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85d4010bb0 con 0x7f85dc123750 2019-12-05T05:18:34.789 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.773+0000 7f85e0a94700 1 -- 172.21.15.42:0/2782043000 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85dc120880 con 0x7f85dc123750 2019-12-05T05:18:34.789 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.773+0000 7f85d3fff700 1 -- 172.21.15.42:0/2782043000 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f85d400f470 con 0x7f85dc123750 2019-12-05T05:18:34.790 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.773+0000 7f85e0a94700 1 -- 172.21.15.42:0/2782043000 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85dc117220 con 0x7f85dc123750 2019-12-05T05:18:34.790 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.777+0000 7f85db59e700 1 --2- 172.21.15.42:0/2782043000 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f85dc123750 0x7f85dc123b80 secure :-1 s=THROTTLE_DONE pgs=630 cs=0 l=1 rx=0x7f85dc117c90 tx=0x7f85d4011140).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:34.790 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.777+0000 7f85d3fff700 1 -- 172.21.15.42:0/2782043000 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f85d400f670 con 0x7f85dc123750 2019-12-05T05:18:34.790 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.777+0000 7f85db59e700 1 --2- 172.21.15.42:0/2782043000 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f85dc123750 0x7f85dc123b80 secure :-1 s=THROTTLE_DONE pgs=630 cs=0 l=1 rx=0x7f85dc117c90 tx=0x7f85d4011140).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:34.790 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.777+0000 7f85d3fff700 1 --2- 172.21.15.42:0/2782043000 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f85c003ae40 0x7f85c003d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:18:34.790 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.777+0000 7f85d3fff700 1 -- 172.21.15.42:0/2782043000 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6891+0+0 (secure 0 0 0) 0x7f85d404af10 con 0x7f85dc123750 2019-12-05T05:18:34.790 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.777+0000 7f85dad9d700 1 --2- 172.21.15.42:0/2782043000 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f85c003ae40 0x7f85c003d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:18:34.791 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.777+0000 7f85dad9d700 1 --2- 172.21.15.42:0/2782043000 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f85c003ae40 0x7f85c003d350 crc :-1 s=READY pgs=287 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:18:34.791 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.781+0000 7f85e0a94700 1 -- 172.21.15.42:0/2782043000 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85dc05d110 con 0x7f85dc123750 2019-12-05T05:18:34.793 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.789+0000 7f85db59e700 1 --2- 172.21.15.42:0/2782043000 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f85dc123750 0x7f85dc123b80 secure :-1 s=THROTTLE_DONE pgs=630 cs=0 l=1 rx=0x7f85dc117c90 tx=0x7f85d4011140).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:34.794 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.789+0000 7f85d3fff700 1 -- 172.21.15.42:0/2782043000 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f85d400f980 con 0x7f85dc123750 2019-12-05T05:18:34.958 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.953+0000 7f85e0a94700 1 -- 172.21.15.42:0/2782043000 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f85dc05b8c0 con 0x7f85dc123750 2019-12-05T05:18:34.959 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.953+0000 7f85db59e700 1 --2- 172.21.15.42:0/2782043000 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f85dc123750 0x7f85dc123b80 secure :-1 s=THROTTLE_DONE pgs=630 cs=0 l=1 rx=0x7f85dc117c90 tx=0x7f85d4011140).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:34.959 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.953+0000 7f85d3fff700 1 -- 172.21.15.42:0/2782043000 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+2108 (secure 0 0 0) 0x7f85d4031f90 con 0x7f85dc123750 2019-12-05T05:18:34.959 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 83 2019-12-05T05:18:34.959 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:18:34.959 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":83,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5833,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6835/1414371821","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1414371821},{"type":"v1","addr":"172.21.15.42:6835","nonce":1414371821}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":83}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:17:17.949394+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5844},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5844":{"gid":5844,"name":"b","rank":0,"incarnation":79,"state":"up:active","state_seq":32,"addr":"172.21.15.40:6833/2698421979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2698421979},{"type":"v1","addr":"172.21.15.40:6833","nonce":2698421979}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:18:34.961 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.953+0000 7f85e0a94700 1 -- 172.21.15.42:0/2782043000 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f85c003ae40 msgr2=0x7f85c003d350 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:18:34.961 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.953+0000 7f85e0a94700 1 --2- 172.21.15.42:0/2782043000 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f85c003ae40 0x7f85c003d350 crc :-1 s=READY pgs=287 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:18:34.961 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.953+0000 7f85e0a94700 1 -- 172.21.15.42:0/2782043000 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f85dc123750 msgr2=0x7f85dc123b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:18:34.961 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.953+0000 7f85e0a94700 1 --2- 172.21.15.42:0/2782043000 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f85dc123750 0x7f85dc123b80 secure :-1 s=READY pgs=630 cs=0 l=1 rx=0x7f85dc117c90 tx=0x7f85d4011140).stop 2019-12-05T05:18:34.961 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.953+0000 7f85e0a94700 1 -- 172.21.15.42:0/2782043000 shutdown_connections 2019-12-05T05:18:34.961 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.953+0000 7f85e0a94700 1 --2- 172.21.15.42:0/2782043000 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f85c003ae40 0x7f85c003d350 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:18:34.961 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.953+0000 7f85e0a94700 1 --2- 172.21.15.42:0/2782043000 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f85dc123750 0x7f85dc123b80 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:18:34.962 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.953+0000 7f85e0a94700 1 --2- 172.21.15.42:0/2782043000 >> v2:172.21.15.40:3300/0 conn(0x7f85dc122b40 0x7f85dc11ff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:18:34.962 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.953+0000 7f85e0a94700 1 --2- 172.21.15.42:0/2782043000 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f85dc120450 0x7f85dc123200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:18:34.962 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.957+0000 7f85e0a94700 1 -- 172.21.15.42:0/2782043000 shutdown_connections 2019-12-05T05:18:34.962 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.957+0000 7f85e0a94700 1 -- 172.21.15.42:0/2782043000 wait complete. 2019-12-05T05:18:34.962 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:34.957+0000 7f85e0a94700 1 -- 172.21.15.42:0/2782043000 >> 172.21.15.42:0/2782043000 conn(0x7f85dc0a82a0 msgr2=0x7f85dc07e4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:18:34.969 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T05:18:36.003 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:18:36.008 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:18:36.974 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:18:37.088 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:37.077+0000 7f3eb30ff700 1 Processor -- start 2019-12-05T05:18:37.088 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:37.077+0000 7f3eb30ff700 1 -- start start 2019-12-05T05:18:37.089 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:37.077+0000 7f3eb30ff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3eac0a7740 0x7f3eac08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:18:37.089 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:37.077+0000 7f3eb30ff700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3eac0a0710 con 0x7f3eac07e720 2019-12-05T05:18:37.089 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:37.077+0000 7f3eb30ff700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3eac0a9ba0 con 0x7f3eac09fae0 2019-12-05T05:18:37.089 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:37.077+0000 7f3eb30ff700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3eac0a0b00 con 0x7f3eac0a7740 2019-12-05T05:18:37.089 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:37.077+0000 7f3eb169c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3eac0a7740 0x7f3eac08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:18:37.089 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:37.077+0000 7f3eb1e9d700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f3eac09fae0 0x7f3eac14efe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:59254/0 (socket says 172.21.15.42:59254) 2019-12-05T05:18:37.090 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:37.077+0000 7f3eb1e9d700 1 -- 172.21.15.42:0/768475131 learned_addr learned my addr 172.21.15.42:0/768475131 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:18:37.090 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:37.081+0000 7f3ea3fff700 1 -- 172.21.15.42:0/768475131 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3993124067 0 0) 0x7f3eac0a9ba0 con 0x7f3eac09fae0 2019-12-05T05:18:37.090 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:37.081+0000 7f3ea3fff700 1 -- 172.21.15.42:0/768475131 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e980030b0 con 0x7f3eac09fae0 2019-12-05T05:18:37.090 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:37.081+0000 7f3ea3fff700 1 -- 172.21.15.42:0/768475131 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 831139353 0 0) 0x7f3eac0a0710 con 0x7f3eac07e720 2019-12-05T05:18:37.090 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:37.081+0000 7f3ea3fff700 1 -- 172.21.15.42:0/768475131 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3eac0a9ba0 con 0x7f3eac07e720 2019-12-05T05:18:37.090 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:37.081+0000 7f3eb169c700 1 -- 172.21.15.42:0/768475131 >> v1:172.21.15.42:6789/0 conn(0x7f3eac09fae0 legacy=0x7f3eac14efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:18:37.090 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:37.081+0000 7f3eb169c700 1 -- 172.21.15.42:0/768475131 >> v1:172.21.15.40:6789/0 conn(0x7f3eac07e720 legacy=0x7f3eac09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:18:37.091 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:37.081+0000 7f3eb169c700 1 -- 172.21.15.42:0/768475131 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3eac07eb50 con 0x7f3eac0a7740 2019-12-05T05:18:37.091 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:37.081+0000 7f3eb169c700 1 --2- 172.21.15.42:0/768475131 >> v2:172.21.15.42:3300/0 conn(0x7f3eac0a7740 0x7f3eac08df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f3ea8008790 tx=0x7f3ea800fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:37.091 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:37.081+0000 7f3eb169c700 1 --2- 172.21.15.42:0/768475131 >> v2:172.21.15.42:3300/0 conn(0x7f3eac0a7740 0x7f3eac08df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f3ea8008790 tx=0x7f3ea800fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:37.091 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:37.081+0000 7f3eb169c700 1 --2- 172.21.15.42:0/768475131 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3eac0a7740 0x7f3eac08df80 secure :-1 s=READY pgs=632 cs=0 l=1 rx=0x7f3ea8008790 tx=0x7f3ea800fbf0).ready entity=mon.1 client_cookie=92df774d4ea7871a server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:18:37.091 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:37.081+0000 7f3eb169c700 1 --2- 172.21.15.42:0/768475131 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3eac0a7740 0x7f3eac08df80 secure :-1 s=THROTTLE_DONE pgs=632 cs=0 l=1 rx=0x7f3ea8008790 tx=0x7f3ea800fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:37.091 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:37.081+0000 7f3ea3fff700 1 -- 172.21.15.42:0/768475131 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3ea8011270 con 0x7f3eac0a7740 2019-12-05T05:18:37.092 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:37.081+0000 7f3eb169c700 1 --2- 172.21.15.42:0/768475131 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3eac0a7740 0x7f3eac08df80 secure :-1 s=THROTTLE_DONE pgs=632 cs=0 l=1 rx=0x7f3ea8008790 tx=0x7f3ea800fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:37.092 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:37.081+0000 7f3eb169c700 1 --2- 172.21.15.42:0/768475131 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3eac0a7740 0x7f3eac08df80 secure :-1 s=THROTTLE_DONE pgs=632 cs=0 l=1 rx=0x7f3ea8008790 tx=0x7f3ea800fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:37.092 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:37.081+0000 7f3ea3fff700 1 -- 172.21.15.42:0/768475131 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ea8008bc0 con 0x7f3eac0a7740 2019-12-05T05:18:37.092 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:37.081+0000 7f3ea3fff700 1 -- 172.21.15.42:0/768475131 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3ea8011850 con 0x7f3eac0a7740 2019-12-05T05:18:37.092 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:37.081+0000 7f3eb30ff700 1 -- 172.21.15.42:0/768475131 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3eac0a7740 msgr2=0x7f3eac08df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:18:37.092 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:37.081+0000 7f3eb30ff700 1 --2- 172.21.15.42:0/768475131 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3eac0a7740 0x7f3eac08df80 secure :-1 s=READY pgs=632 cs=0 l=1 rx=0x7f3ea8008790 tx=0x7f3ea800fbf0).stop 2019-12-05T05:18:37.092 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:37.081+0000 7f3eb30ff700 1 -- 172.21.15.42:0/768475131 shutdown_connections 2019-12-05T05:18:37.092 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:37.081+0000 7f3eb30ff700 1 --2- 172.21.15.42:0/768475131 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3eac0a7740 0x7f3eac08df80 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:18:37.093 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:37.081+0000 7f3eb30ff700 1 -- 172.21.15.42:0/768475131 shutdown_connections 2019-12-05T05:18:37.093 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:37.081+0000 7f3eb30ff700 1 -- 172.21.15.42:0/768475131 wait complete. 2019-12-05T05:18:37.093 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:37.081+0000 7f3eb30ff700 1 -- 172.21.15.42:0/768475131 >> 172.21.15.42:0/768475131 conn(0x7f3eac0aabe0 msgr2=0x7f3eac0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:18:37.096 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:37.085+0000 7f3eb30ff700 1 Processor -- start 2019-12-05T05:18:37.096 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:37.085+0000 7f3eb30ff700 1 -- start start 2019-12-05T05:18:37.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:37.085+0000 7f3eb30ff700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3eac1330d0 0x7f3eac135510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:18:37.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:37.085+0000 7f3eb30ff700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3eac0a4430 con 0x7f3eac145120 2019-12-05T05:18:37.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:37.085+0000 7f3eb30ff700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3eac07e150 con 0x7f3eac11a050 2019-12-05T05:18:37.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:37.085+0000 7f3eb30ff700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3eac1503b0 con 0x7f3eac1330d0 2019-12-05T05:18:37.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:37.085+0000 7f3eb169c700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f3eac11a050 0x7f3eac14fbb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:59256/0 (socket says 172.21.15.42:59256) 2019-12-05T05:18:37.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:37.085+0000 7f3eb169c700 1 -- 172.21.15.42:0/2140379655 learned_addr learned my addr 172.21.15.42:0/2140379655 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:18:37.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:37.085+0000 7f3eb1e9d700 1 --2- 172.21.15.42:0/2140379655 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3eac1330d0 0x7f3eac135510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:18:37.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:37.089+0000 7f3ea1ffb700 1 -- 172.21.15.42:0/2140379655 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1350908410 0 0) 0x7f3eac07e150 con 0x7f3eac11a050 2019-12-05T05:18:37.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:37.089+0000 7f3ea1ffb700 1 -- 172.21.15.42:0/2140379655 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e8c0030b0 con 0x7f3eac11a050 2019-12-05T05:18:37.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:37.089+0000 7f3ea1ffb700 1 -- 172.21.15.42:0/2140379655 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2527129432 0 0) 0x7f3eac0a4430 con 0x7f3eac145120 2019-12-05T05:18:37.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:37.089+0000 7f3ea1ffb700 1 -- 172.21.15.42:0/2140379655 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3eac07e150 con 0x7f3eac145120 2019-12-05T05:18:37.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:37.089+0000 7f3eb1e9d700 1 -- 172.21.15.42:0/2140379655 >> v1:172.21.15.42:6789/0 conn(0x7f3eac11a050 legacy=0x7f3eac14fbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:18:37.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:37.089+0000 7f3eb1e9d700 1 -- 172.21.15.42:0/2140379655 >> v1:172.21.15.40:6789/0 conn(0x7f3eac145120 legacy=0x7f3eac07d950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:18:37.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:37.089+0000 7f3eb1e9d700 1 -- 172.21.15.42:0/2140379655 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3eac144110 con 0x7f3eac1330d0 2019-12-05T05:18:37.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:37.089+0000 7f3eb1e9d700 1 --2- 172.21.15.42:0/2140379655 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3eac1330d0 0x7f3eac135510 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f3ea4004720 tx=0x7f3ea4010b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:37.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:37.089+0000 7f3eb1e9d700 1 --2- 172.21.15.42:0/2140379655 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3eac1330d0 0x7f3eac135510 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f3ea4004720 tx=0x7f3ea4010b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:37.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:37.089+0000 7f3eb1e9d700 1 --2- 172.21.15.42:0/2140379655 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3eac1330d0 0x7f3eac135510 secure :-1 s=READY pgs=129 cs=0 l=1 rx=0x7f3ea4004720 tx=0x7f3ea4010b30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:18:37.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:37.089+0000 7f3eb1e9d700 1 --2- 172.21.15.42:0/2140379655 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3eac1330d0 0x7f3eac135510 secure :-1 s=THROTTLE_DONE pgs=129 cs=0 l=1 rx=0x7f3ea4004720 tx=0x7f3ea4010b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:37.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:37.089+0000 7f3ea1ffb700 1 -- 172.21.15.42:0/2140379655 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3ea4005bc0 con 0x7f3eac1330d0 2019-12-05T05:18:37.100 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:37.089+0000 7f3eb1e9d700 1 --2- 172.21.15.42:0/2140379655 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3eac1330d0 0x7f3eac135510 secure :-1 s=THROTTLE_DONE pgs=129 cs=0 l=1 rx=0x7f3ea4004720 tx=0x7f3ea4010b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:37.100 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:37.089+0000 7f3eb1e9d700 1 --2- 172.21.15.42:0/2140379655 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3eac1330d0 0x7f3eac135510 secure :-1 s=THROTTLE_DONE pgs=129 cs=0 l=1 rx=0x7f3ea4004720 tx=0x7f3ea4010b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:37.100 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:37.089+0000 7f3ea1ffb700 1 -- 172.21.15.42:0/2140379655 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ea40057e0 con 0x7f3eac1330d0 2019-12-05T05:18:37.100 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:37.089+0000 7f3eb30ff700 1 -- 172.21.15.42:0/2140379655 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3eac0a3de0 con 0x7f3eac1330d0 2019-12-05T05:18:37.100 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:37.089+0000 7f3ea1ffb700 1 -- 172.21.15.42:0/2140379655 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3ea4012850 con 0x7f3eac1330d0 2019-12-05T05:18:37.100 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:37.089+0000 7f3eb30ff700 1 -- 172.21.15.42:0/2140379655 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3eac14e790 con 0x7f3eac1330d0 2019-12-05T05:18:37.100 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:37.089+0000 7f3eb1e9d700 1 --2- 172.21.15.42:0/2140379655 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3eac1330d0 0x7f3eac135510 secure :-1 s=THROTTLE_DONE pgs=129 cs=0 l=1 rx=0x7f3ea4004720 tx=0x7f3ea4010b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:37.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:37.089+0000 7f3ea1ffb700 1 -- 172.21.15.42:0/2140379655 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f3ea4012aa0 con 0x7f3eac1330d0 2019-12-05T05:18:37.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:37.089+0000 7f3eb1e9d700 1 --2- 172.21.15.42:0/2140379655 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3eac1330d0 0x7f3eac135510 secure :-1 s=THROTTLE_DONE pgs=129 cs=0 l=1 rx=0x7f3ea4004720 tx=0x7f3ea4010b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:37.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:37.089+0000 7f3ea1ffb700 1 --2- 172.21.15.42:0/2140379655 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3e8c03b420 0x7f3e8c03d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:18:37.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:37.089+0000 7f3ea1ffb700 1 -- 172.21.15.42:0/2140379655 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6891+0+0 (secure 0 0 0) 0x7f3ea404b030 con 0x7f3eac1330d0 2019-12-05T05:18:37.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:37.089+0000 7f3eb169c700 1 --2- 172.21.15.42:0/2140379655 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3e8c03b420 0x7f3e8c03d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:18:37.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:37.093+0000 7f3eb169c700 1 --2- 172.21.15.42:0/2140379655 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3e8c03b420 0x7f3e8c03d930 crc :-1 s=READY pgs=288 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:18:37.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:37.093+0000 7f3eb30ff700 1 -- 172.21.15.42:0/2140379655 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3eac05d110 con 0x7f3eac1330d0 2019-12-05T05:18:37.105 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:37.101+0000 7f3eb1e9d700 1 --2- 172.21.15.42:0/2140379655 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3eac1330d0 0x7f3eac135510 secure :-1 s=THROTTLE_DONE pgs=129 cs=0 l=1 rx=0x7f3ea4004720 tx=0x7f3ea4010b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:37.105 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:37.101+0000 7f3ea1ffb700 1 -- 172.21.15.42:0/2140379655 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f3ea4032090 con 0x7f3eac1330d0 2019-12-05T05:18:37.273 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:37.269+0000 7f3eb30ff700 1 -- 172.21.15.42:0/2140379655 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3eac05b8c0 con 0x7f3eac1330d0 2019-12-05T05:18:37.274 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:37.269+0000 7f3eb1e9d700 1 --2- 172.21.15.42:0/2140379655 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3eac1330d0 0x7f3eac135510 secure :-1 s=THROTTLE_DONE pgs=129 cs=0 l=1 rx=0x7f3ea4004720 tx=0x7f3ea4010b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:37.274 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:37.269+0000 7f3ea1ffb700 1 -- 172.21.15.42:0/2140379655 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+2108 (secure 0 0 0) 0x7f3ea40322f0 con 0x7f3eac1330d0 2019-12-05T05:18:37.276 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 83 2019-12-05T05:18:37.276 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:18:37.277 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":83,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5833,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6835/1414371821","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1414371821},{"type":"v1","addr":"172.21.15.42:6835","nonce":1414371821}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":83}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:17:17.949394+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5844},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5844":{"gid":5844,"name":"b","rank":0,"incarnation":79,"state":"up:active","state_seq":32,"addr":"172.21.15.40:6833/2698421979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2698421979},{"type":"v1","addr":"172.21.15.40:6833","nonce":2698421979}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:18:37.279 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:37.273+0000 7f3eb30ff700 1 -- 172.21.15.42:0/2140379655 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3e8c03b420 msgr2=0x7f3e8c03d930 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:18:37.279 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:37.273+0000 7f3eb30ff700 1 --2- 172.21.15.42:0/2140379655 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3e8c03b420 0x7f3e8c03d930 crc :-1 s=READY pgs=288 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:18:37.279 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:37.273+0000 7f3eb30ff700 1 -- 172.21.15.42:0/2140379655 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3eac1330d0 msgr2=0x7f3eac135510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:18:37.279 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:37.273+0000 7f3eb30ff700 1 --2- 172.21.15.42:0/2140379655 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3eac1330d0 0x7f3eac135510 secure :-1 s=READY pgs=129 cs=0 l=1 rx=0x7f3ea4004720 tx=0x7f3ea4010b30).stop 2019-12-05T05:18:37.279 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:37.273+0000 7f3eb30ff700 1 -- 172.21.15.42:0/2140379655 shutdown_connections 2019-12-05T05:18:37.280 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:37.273+0000 7f3eb30ff700 1 --2- 172.21.15.42:0/2140379655 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3e8c03b420 0x7f3e8c03d930 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:18:37.280 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:37.273+0000 7f3eb30ff700 1 --2- 172.21.15.42:0/2140379655 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3eac1330d0 0x7f3eac135510 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:18:37.280 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:37.273+0000 7f3eb30ff700 1 -- 172.21.15.42:0/2140379655 shutdown_connections 2019-12-05T05:18:37.280 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:37.273+0000 7f3eb30ff700 1 -- 172.21.15.42:0/2140379655 wait complete. 2019-12-05T05:18:37.280 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:37.273+0000 7f3eb30ff700 1 -- 172.21.15.42:0/2140379655 >> 172.21.15.42:0/2140379655 conn(0x7f3eac0aabe0 msgr2=0x7f3eac0a4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:18:37.289 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T05:18:39.290 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:18:39.397 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:39.385+0000 7fb6338af700 1 Processor -- start 2019-12-05T05:18:39.397 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:39.385+0000 7fb6338af700 1 -- start start 2019-12-05T05:18:39.397 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:39.389+0000 7fb6338af700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb62c08df70 0x7fb62c09f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:18:39.397 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:39.389+0000 7fb6338af700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb62c09f820 0x7fb62c14efd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:18:39.398 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:39.389+0000 7fb6338af700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb62c09fc50 con 0x7fb62c0a7730 2019-12-05T05:18:39.398 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:39.389+0000 7fb6338af700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb62c0a0af0 con 0x7fb62c09f820 2019-12-05T05:18:39.398 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:39.389+0000 7fb6338af700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb62c08e5a0 con 0x7fb62c08df70 2019-12-05T05:18:39.398 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:39.389+0000 7fb631e4c700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fb62c0a7730 0x7fb62c07e710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:40474/0 (socket says 172.21.15.42:40474) 2019-12-05T05:18:39.398 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:39.389+0000 7fb63164b700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb62c08df70 0x7fb62c09f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:18:39.398 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:39.389+0000 7fb631e4c700 1 -- 172.21.15.42:0/2136363796 learned_addr learned my addr 172.21.15.42:0/2136363796 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:18:39.398 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:39.389+0000 7fb63264d700 1 --2- 172.21.15.42:0/2136363796 >> v2:172.21.15.40:3300/0 conn(0x7fb62c09f820 0x7fb62c14efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:18:39.399 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:39.389+0000 7fb630e4a700 1 -- 172.21.15.42:0/2136363796 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 418266536 0 0) 0x7fb62c09fc50 con 0x7fb62c0a7730 2019-12-05T05:18:39.399 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:39.389+0000 7fb630e4a700 1 -- 172.21.15.42:0/2136363796 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6180030b0 con 0x7fb62c0a7730 2019-12-05T05:18:39.399 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:39.389+0000 7fb63164b700 1 -- 172.21.15.42:0/2136363796 >> v2:172.21.15.40:3300/0 conn(0x7fb62c09f820 msgr2=0x7fb62c14efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:18:39.399 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:39.389+0000 7fb63164b700 1 --2- 172.21.15.42:0/2136363796 >> v2:172.21.15.40:3300/0 conn(0x7fb62c09f820 0x7fb62c14efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:18:39.399 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:39.389+0000 7fb63164b700 1 -- 172.21.15.42:0/2136363796 >> v1:172.21.15.40:6789/0 conn(0x7fb62c0a7730 legacy=0x7fb62c07e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:18:39.399 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:39.389+0000 7fb63164b700 1 -- 172.21.15.42:0/2136363796 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb62c0a0700 con 0x7fb62c08df70 2019-12-05T05:18:39.399 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:39.389+0000 7fb63264d700 1 --2- 172.21.15.42:0/2136363796 >> v2:172.21.15.40:3300/0 conn(0x7fb62c09f820 0x7fb62c14efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:18:39.400 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:39.389+0000 7fb63164b700 1 --2- 172.21.15.42:0/2136363796 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb62c08df70 0x7fb62c09f2d0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fb620008790 tx=0x7fb62000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:39.400 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:39.389+0000 7fb63164b700 1 --2- 172.21.15.42:0/2136363796 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb62c08df70 0x7fb62c09f2d0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fb620008790 tx=0x7fb62000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:39.400 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:39.389+0000 7fb63164b700 1 --2- 172.21.15.42:0/2136363796 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb62c08df70 0x7fb62c09f2d0 secure :-1 s=READY pgs=130 cs=0 l=1 rx=0x7fb620008790 tx=0x7fb62000fc70).ready entity=mon.2 client_cookie=119eb99850db034f server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:18:39.400 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:39.389+0000 7fb63164b700 1 --2- 172.21.15.42:0/2136363796 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb62c08df70 0x7fb62c09f2d0 secure :-1 s=THROTTLE_DONE pgs=130 cs=0 l=1 rx=0x7fb620008790 tx=0x7fb62000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:39.400 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:39.389+0000 7fb63164b700 1 --2- 172.21.15.42:0/2136363796 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb62c08df70 0x7fb62c09f2d0 secure :-1 s=THROTTLE_DONE pgs=130 cs=0 l=1 rx=0x7fb620008790 tx=0x7fb62000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:39.400 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:39.389+0000 7fb630e4a700 1 -- 172.21.15.42:0/2136363796 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb620008bb0 con 0x7fb62c08df70 2019-12-05T05:18:39.400 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:39.389+0000 7fb63164b700 1 --2- 172.21.15.42:0/2136363796 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb62c08df70 0x7fb62c09f2d0 secure :-1 s=THROTTLE_DONE pgs=130 cs=0 l=1 rx=0x7fb620008790 tx=0x7fb62000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:39.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:39.389+0000 7fb630e4a700 1 -- 172.21.15.42:0/2136363796 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6200087d0 con 0x7fb62c08df70 2019-12-05T05:18:39.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:39.389+0000 7fb630e4a700 1 -- 172.21.15.42:0/2136363796 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb620011860 con 0x7fb62c08df70 2019-12-05T05:18:39.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:39.389+0000 7fb6338af700 1 -- 172.21.15.42:0/2136363796 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb62c08df70 msgr2=0x7fb62c09f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:18:39.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:39.389+0000 7fb6338af700 1 --2- 172.21.15.42:0/2136363796 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb62c08df70 0x7fb62c09f2d0 secure :-1 s=READY pgs=130 cs=0 l=1 rx=0x7fb620008790 tx=0x7fb62000fc70).stop 2019-12-05T05:18:39.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:39.389+0000 7fb6338af700 1 -- 172.21.15.42:0/2136363796 shutdown_connections 2019-12-05T05:18:39.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:39.389+0000 7fb6338af700 1 --2- 172.21.15.42:0/2136363796 >> v2:172.21.15.40:3300/0 conn(0x7fb62c09f820 0x7fb62c14efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:18:39.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:39.389+0000 7fb6338af700 1 --2- 172.21.15.42:0/2136363796 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb62c08df70 0x7fb62c09f2d0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:18:39.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:39.389+0000 7fb6338af700 1 -- 172.21.15.42:0/2136363796 shutdown_connections 2019-12-05T05:18:39.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:39.389+0000 7fb6338af700 1 -- 172.21.15.42:0/2136363796 wait complete. 2019-12-05T05:18:39.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:39.389+0000 7fb6338af700 1 -- 172.21.15.42:0/2136363796 >> 172.21.15.42:0/2136363796 conn(0x7fb62c0aabd0 msgr2=0x7fb62c0a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:18:39.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:39.393+0000 7fb6338af700 1 Processor -- start 2019-12-05T05:18:39.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:39.393+0000 7fb6338af700 1 -- start start 2019-12-05T05:18:39.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:39.397+0000 7fb6338af700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb62c07d150 0x7fb62c077750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:18:39.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:39.397+0000 7fb6338af700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb62c077ca0 0x7fb62c1172e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:18:39.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:39.397+0000 7fb6338af700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb62c117830 0x7fb62c1190d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:18:39.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:39.397+0000 7fb6338af700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb62c11b930 con 0x7fb62c07d150 2019-12-05T05:18:39.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:39.397+0000 7fb6338af700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb62c11bbd0 con 0x7fb62c077ca0 2019-12-05T05:18:39.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:39.397+0000 7fb6338af700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb62c118570 con 0x7fb62c117830 2019-12-05T05:18:39.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:39.397+0000 7fb63164b700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb62c077ca0 0x7fb62c1172e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:18:39.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:39.397+0000 7fb631e4c700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb62c07d150 0x7fb62c077750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:18:39.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:39.397+0000 7fb63264d700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb62c117830 0x7fb62c1190d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:18:39.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:39.397+0000 7fb631e4c700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb62c07d150 0x7fb62c077750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.42:47050/0 (socket says 172.21.15.42:47050) 2019-12-05T05:18:39.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:39.397+0000 7fb631e4c700 1 -- 172.21.15.42:0/4236822881 learned_addr learned my addr 172.21.15.42:0/4236822881 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:18:39.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:39.397+0000 7fb631e4c700 1 -- 172.21.15.42:0/4236822881 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb62c117830 msgr2=0x7fb62c1190d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:18:39.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:39.397+0000 7fb631e4c700 1 --2- 172.21.15.42:0/4236822881 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb62c117830 0x7fb62c1190d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:18:39.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:39.397+0000 7fb631e4c700 1 -- 172.21.15.42:0/4236822881 >> v2:172.21.15.42:3300/0 conn(0x7fb62c077ca0 msgr2=0x7fb62c1172e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:18:39.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:39.397+0000 7fb631e4c700 1 --2- 172.21.15.42:0/4236822881 >> v2:172.21.15.42:3300/0 conn(0x7fb62c077ca0 0x7fb62c1172e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:18:39.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:39.397+0000 7fb631e4c700 1 -- 172.21.15.42:0/4236822881 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb62c0780d0 con 0x7fb62c07d150 2019-12-05T05:18:39.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:39.397+0000 7fb631e4c700 1 --2- 172.21.15.42:0/4236822881 >> v2:172.21.15.40:3300/0 conn(0x7fb62c07d150 0x7fb62c077750 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fb628001430 tx=0x7fb628010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:39.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:39.397+0000 7fb631e4c700 1 --2- 172.21.15.42:0/4236822881 >> v2:172.21.15.40:3300/0 conn(0x7fb62c07d150 0x7fb62c077750 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fb628001430 tx=0x7fb628010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:39.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:39.397+0000 7fb631e4c700 1 --2- 172.21.15.42:0/4236822881 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb62c07d150 0x7fb62c077750 secure :-1 s=READY pgs=510 cs=0 l=1 rx=0x7fb628001430 tx=0x7fb628010ab0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:18:39.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:39.397+0000 7fb631e4c700 1 --2- 172.21.15.42:0/4236822881 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb62c07d150 0x7fb62c077750 secure :-1 s=THROTTLE_DONE pgs=510 cs=0 l=1 rx=0x7fb628001430 tx=0x7fb628010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:39.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:39.397+0000 7fb631e4c700 1 --2- 172.21.15.42:0/4236822881 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb62c07d150 0x7fb62c077750 secure :-1 s=THROTTLE_DONE pgs=510 cs=0 l=1 rx=0x7fb628001430 tx=0x7fb628010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:39.411 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:39.397+0000 7fb61e7fc700 1 -- 172.21.15.42:0/4236822881 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb62800c9a0 con 0x7fb62c07d150 2019-12-05T05:18:39.411 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:39.397+0000 7fb631e4c700 1 --2- 172.21.15.42:0/4236822881 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb62c07d150 0x7fb62c077750 secure :-1 s=THROTTLE_DONE pgs=510 cs=0 l=1 rx=0x7fb628001430 tx=0x7fb628010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:39.411 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:39.397+0000 7fb61e7fc700 1 -- 172.21.15.42:0/4236822881 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb628010250 con 0x7fb62c07d150 2019-12-05T05:18:39.411 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:39.397+0000 7fb61e7fc700 1 -- 172.21.15.42:0/4236822881 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb62800d0c0 con 0x7fb62c07d150 2019-12-05T05:18:39.411 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:39.397+0000 7fb6338af700 1 -- 172.21.15.42:0/4236822881 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb62c1180c0 con 0x7fb62c07d150 2019-12-05T05:18:39.411 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:39.397+0000 7fb6338af700 1 -- 172.21.15.42:0/4236822881 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb62c118cf0 con 0x7fb62c07d150 2019-12-05T05:18:39.411 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:39.397+0000 7fb631e4c700 1 --2- 172.21.15.42:0/4236822881 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb62c07d150 0x7fb62c077750 secure :-1 s=THROTTLE_DONE pgs=510 cs=0 l=1 rx=0x7fb628001430 tx=0x7fb628010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:39.411 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:39.401+0000 7fb61e7fc700 1 -- 172.21.15.42:0/4236822881 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fb62800d3d0 con 0x7fb62c07d150 2019-12-05T05:18:39.412 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:39.401+0000 7fb631e4c700 1 --2- 172.21.15.42:0/4236822881 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb62c07d150 0x7fb62c077750 secure :-1 s=THROTTLE_DONE pgs=510 cs=0 l=1 rx=0x7fb628001430 tx=0x7fb628010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:39.412 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:39.401+0000 7fb61e7fc700 1 --2- 172.21.15.42:0/4236822881 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb61403add0 0x7fb61403d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:18:39.412 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:39.401+0000 7fb61e7fc700 1 -- 172.21.15.42:0/4236822881 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6891+0+0 (secure 0 0 0) 0x7fb62804ac30 con 0x7fb62c07d150 2019-12-05T05:18:39.412 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:39.401+0000 7fb63164b700 1 --2- 172.21.15.42:0/4236822881 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb61403add0 0x7fb61403d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:18:39.413 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:39.401+0000 7fb63164b700 1 --2- 172.21.15.42:0/4236822881 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb61403add0 0x7fb61403d2e0 crc :-1 s=READY pgs=289 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:18:39.413 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:39.401+0000 7fb6338af700 1 -- 172.21.15.42:0/4236822881 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb62c05d100 con 0x7fb62c07d150 2019-12-05T05:18:39.415 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:39.409+0000 7fb631e4c700 1 --2- 172.21.15.42:0/4236822881 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb62c07d150 0x7fb62c077750 secure :-1 s=THROTTLE_DONE pgs=510 cs=0 l=1 rx=0x7fb628001430 tx=0x7fb628010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:39.415 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:39.409+0000 7fb61e7fc700 1 -- 172.21.15.42:0/4236822881 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fb628031cb0 con 0x7fb62c07d150 2019-12-05T05:18:39.582 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:39.577+0000 7fb6338af700 1 -- 172.21.15.42:0/4236822881 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb62c05b8b0 con 0x7fb62c07d150 2019-12-05T05:18:39.583 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:39.577+0000 7fb631e4c700 1 --2- 172.21.15.42:0/4236822881 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb62c07d150 0x7fb62c077750 secure :-1 s=THROTTLE_DONE pgs=510 cs=0 l=1 rx=0x7fb628001430 tx=0x7fb628010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:39.584 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:39.581+0000 7fb61e7fc700 1 -- 172.21.15.42:0/4236822881 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+2108 (secure 0 0 0) 0x7fb628031f10 con 0x7fb62c07d150 2019-12-05T05:18:39.586 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 83 2019-12-05T05:18:39.586 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:18:39.586 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":83,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5833,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6835/1414371821","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1414371821},{"type":"v1","addr":"172.21.15.42:6835","nonce":1414371821}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":83}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:17:17.949394+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5844},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5844":{"gid":5844,"name":"b","rank":0,"incarnation":79,"state":"up:active","state_seq":32,"addr":"172.21.15.40:6833/2698421979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2698421979},{"type":"v1","addr":"172.21.15.40:6833","nonce":2698421979}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:18:39.589 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:39.585+0000 7fb6338af700 1 -- 172.21.15.42:0/4236822881 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb61403add0 msgr2=0x7fb61403d2e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:18:39.589 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:39.585+0000 7fb6338af700 1 --2- 172.21.15.42:0/4236822881 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb61403add0 0x7fb61403d2e0 crc :-1 s=READY pgs=289 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:18:39.589 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:39.585+0000 7fb6338af700 1 -- 172.21.15.42:0/4236822881 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb62c07d150 msgr2=0x7fb62c077750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:18:39.589 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:39.585+0000 7fb6338af700 1 --2- 172.21.15.42:0/4236822881 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb62c07d150 0x7fb62c077750 secure :-1 s=READY pgs=510 cs=0 l=1 rx=0x7fb628001430 tx=0x7fb628010ab0).stop 2019-12-05T05:18:39.589 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:39.585+0000 7fb6338af700 1 -- 172.21.15.42:0/4236822881 shutdown_connections 2019-12-05T05:18:39.589 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:39.585+0000 7fb6338af700 1 --2- 172.21.15.42:0/4236822881 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb61403add0 0x7fb61403d2e0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:18:39.590 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:39.585+0000 7fb6338af700 1 --2- 172.21.15.42:0/4236822881 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb62c117830 0x7fb62c1190d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:18:39.590 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:39.585+0000 7fb6338af700 1 --2- 172.21.15.42:0/4236822881 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb62c07d150 0x7fb62c077750 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:18:39.590 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:39.585+0000 7fb6338af700 1 --2- 172.21.15.42:0/4236822881 >> v2:172.21.15.42:3300/0 conn(0x7fb62c077ca0 0x7fb62c1172e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:18:39.590 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:39.585+0000 7fb6338af700 1 -- 172.21.15.42:0/4236822881 shutdown_connections 2019-12-05T05:18:39.590 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:39.585+0000 7fb6338af700 1 -- 172.21.15.42:0/4236822881 wait complete. 2019-12-05T05:18:39.590 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:39.585+0000 7fb6338af700 1 -- 172.21.15.42:0/4236822881 >> 172.21.15.42:0/4236822881 conn(0x7fb62c11d1f0 msgr2=0x7fb62c0a9630 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:18:39.599 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T05:18:41.600 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:18:41.708 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:41.697+0000 7fb343791700 1 Processor -- start 2019-12-05T05:18:41.708 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:41.697+0000 7fb343791700 1 -- start start 2019-12-05T05:18:41.708 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:41.701+0000 7fb343791700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb33c08df80 0x7fb33c09f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:18:41.708 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:41.701+0000 7fb343791700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb33c09f830 0x7fb33c14efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:18:41.708 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:41.701+0000 7fb343791700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb33c09fc60 con 0x7fb33c0a7740 2019-12-05T05:18:41.708 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:41.701+0000 7fb343791700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb33c0a0b00 con 0x7fb33c08df80 2019-12-05T05:18:41.709 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:41.701+0000 7fb343791700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb33c08e5b0 con 0x7fb33c09f830 2019-12-05T05:18:41.709 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:41.701+0000 7fb34252f700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb33c09f830 0x7fb33c14efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:18:41.709 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:41.701+0000 7fb34252f700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb33c09f830 0x7fb33c14efe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:59246/0 (socket says 172.21.15.42:59246) 2019-12-05T05:18:41.709 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:41.701+0000 7fb34252f700 1 -- 172.21.15.42:0/1929008439 learned_addr learned my addr 172.21.15.42:0/1929008439 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:18:41.709 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:41.701+0000 7fb34152d700 1 --2- 172.21.15.42:0/1929008439 >> v2:172.21.15.40:3300/0 conn(0x7fb33c08df80 0x7fb33c09f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:18:41.709 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:41.701+0000 7fb34252f700 1 -- 172.21.15.42:0/1929008439 >> v2:172.21.15.40:3300/0 conn(0x7fb33c08df80 msgr2=0x7fb33c09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:18:41.709 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:41.701+0000 7fb34252f700 1 --2- 172.21.15.42:0/1929008439 >> v2:172.21.15.40:3300/0 conn(0x7fb33c08df80 0x7fb33c09f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:18:41.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:41.701+0000 7fb34252f700 1 -- 172.21.15.42:0/1929008439 >> v1:172.21.15.40:6789/0 conn(0x7fb33c0a7740 legacy=0x7fb33c07e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:18:41.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:41.701+0000 7fb34252f700 1 -- 172.21.15.42:0/1929008439 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb33c0a0710 con 0x7fb33c09f830 2019-12-05T05:18:41.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:41.701+0000 7fb34252f700 1 --2- 172.21.15.42:0/1929008439 >> v2:172.21.15.42:3300/0 conn(0x7fb33c09f830 0x7fb33c14efe0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fb3340032b0 tx=0x7fb3340100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:41.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:41.701+0000 7fb34252f700 1 --2- 172.21.15.42:0/1929008439 >> v2:172.21.15.42:3300/0 conn(0x7fb33c09f830 0x7fb33c14efe0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fb3340032b0 tx=0x7fb3340100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:41.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:41.701+0000 7fb34252f700 1 --2- 172.21.15.42:0/1929008439 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb33c09f830 0x7fb33c14efe0 secure :-1 s=READY pgs=634 cs=0 l=1 rx=0x7fb3340032b0 tx=0x7fb3340100b0).ready entity=mon.1 client_cookie=80197daa12aea015 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:18:41.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:41.701+0000 7fb34252f700 1 --2- 172.21.15.42:0/1929008439 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb33c09f830 0x7fb33c14efe0 secure :-1 s=THROTTLE_DONE pgs=634 cs=0 l=1 rx=0x7fb3340032b0 tx=0x7fb3340100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:41.711 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:41.701+0000 7fb340d2c700 1 -- 172.21.15.42:0/1929008439 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb334010d60 con 0x7fb33c09f830 2019-12-05T05:18:41.711 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:41.701+0000 7fb34252f700 1 --2- 172.21.15.42:0/1929008439 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb33c09f830 0x7fb33c14efe0 secure :-1 s=THROTTLE_DONE pgs=634 cs=0 l=1 rx=0x7fb3340032b0 tx=0x7fb3340100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:41.711 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:41.701+0000 7fb340d2c700 1 -- 172.21.15.42:0/1929008439 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb334010980 con 0x7fb33c09f830 2019-12-05T05:18:41.712 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:41.701+0000 7fb34252f700 1 --2- 172.21.15.42:0/1929008439 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb33c09f830 0x7fb33c14efe0 secure :-1 s=THROTTLE_DONE pgs=634 cs=0 l=1 rx=0x7fb3340032b0 tx=0x7fb3340100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:41.712 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:41.701+0000 7fb340d2c700 1 -- 172.21.15.42:0/1929008439 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb3340114a0 con 0x7fb33c09f830 2019-12-05T05:18:41.712 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:41.701+0000 7fb343791700 1 -- 172.21.15.42:0/1929008439 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb33c09f830 msgr2=0x7fb33c14efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:18:41.712 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:41.701+0000 7fb343791700 1 --2- 172.21.15.42:0/1929008439 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb33c09f830 0x7fb33c14efe0 secure :-1 s=READY pgs=634 cs=0 l=1 rx=0x7fb3340032b0 tx=0x7fb3340100b0).stop 2019-12-05T05:18:41.712 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:41.701+0000 7fb343791700 1 -- 172.21.15.42:0/1929008439 shutdown_connections 2019-12-05T05:18:41.712 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:41.701+0000 7fb343791700 1 --2- 172.21.15.42:0/1929008439 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb33c09f830 0x7fb33c14efe0 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:18:41.712 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:41.701+0000 7fb343791700 1 --2- 172.21.15.42:0/1929008439 >> v2:172.21.15.40:3300/0 conn(0x7fb33c08df80 0x7fb33c09f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:18:41.713 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:41.701+0000 7fb343791700 1 -- 172.21.15.42:0/1929008439 shutdown_connections 2019-12-05T05:18:41.713 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:41.701+0000 7fb343791700 1 -- 172.21.15.42:0/1929008439 wait complete. 2019-12-05T05:18:41.713 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:41.701+0000 7fb343791700 1 -- 172.21.15.42:0/1929008439 >> 172.21.15.42:0/1929008439 conn(0x7fb33c0aabe0 msgr2=0x7fb33c0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:18:41.716 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:41.705+0000 7fb343791700 1 Processor -- start 2019-12-05T05:18:41.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:41.705+0000 7fb343791700 1 -- start start 2019-12-05T05:18:41.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:41.709+0000 7fb343791700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb33c1175a0 0x7fb33c11b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:18:41.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:41.709+0000 7fb343791700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb33c07b9d0 0x7fb33c07be00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:18:41.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:41.709+0000 7fb343791700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb33c07c350 con 0x7fb33c0a6d90 2019-12-05T05:18:41.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:41.709+0000 7fb343791700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb33c11ad60 con 0x7fb33c1175a0 2019-12-05T05:18:41.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:41.709+0000 7fb343791700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb33c11b2e0 con 0x7fb33c07b9d0 2019-12-05T05:18:41.718 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:41.709+0000 7fb341d2e700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fb33c0a6d90 0x7fb33c11a4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:59288/0 (socket says 172.21.15.42:59288) 2019-12-05T05:18:41.718 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:41.709+0000 7fb341d2e700 1 -- 172.21.15.42:0/842864225 learned_addr learned my addr 172.21.15.42:0/842864225 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:18:41.718 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:41.709+0000 7fb34252f700 1 --2- 172.21.15.42:0/842864225 >> v2:172.21.15.42:3300/0 conn(0x7fb33c07b9d0 0x7fb33c07be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:18:41.718 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:41.709+0000 7fb34152d700 1 --2- 172.21.15.42:0/842864225 >> v2:172.21.15.40:3300/0 conn(0x7fb33c1175a0 0x7fb33c11b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:18:41.718 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:41.709+0000 7fb3327fc700 1 -- 172.21.15.42:0/842864225 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1822720034 0 0) 0x7fb33c07c350 con 0x7fb33c0a6d90 2019-12-05T05:18:41.718 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:41.709+0000 7fb3327fc700 1 -- 172.21.15.42:0/842864225 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3240030b0 con 0x7fb33c0a6d90 2019-12-05T05:18:41.718 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:41.709+0000 7fb34252f700 1 -- 172.21.15.42:0/842864225 >> v2:172.21.15.40:3300/0 conn(0x7fb33c1175a0 msgr2=0x7fb33c11b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:18:41.718 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:41.709+0000 7fb34252f700 1 --2- 172.21.15.42:0/842864225 >> v2:172.21.15.40:3300/0 conn(0x7fb33c1175a0 0x7fb33c11b580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:18:41.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:41.709+0000 7fb34252f700 1 -- 172.21.15.42:0/842864225 >> v1:172.21.15.42:6789/0 conn(0x7fb33c0a6d90 legacy=0x7fb33c11a4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:18:41.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:41.709+0000 7fb34252f700 1 -- 172.21.15.42:0/842864225 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb33c11bad0 con 0x7fb33c07b9d0 2019-12-05T05:18:41.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:41.709+0000 7fb34252f700 1 --2- 172.21.15.42:0/842864225 >> v2:172.21.15.42:3300/0 conn(0x7fb33c07b9d0 0x7fb33c07be00 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fb3340032b0 tx=0x7fb334010c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:41.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:41.709+0000 7fb34252f700 1 --2- 172.21.15.42:0/842864225 >> v2:172.21.15.42:3300/0 conn(0x7fb33c07b9d0 0x7fb33c07be00 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fb3340032b0 tx=0x7fb334010c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:41.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:41.709+0000 7fb34252f700 1 --2- 172.21.15.42:0/842864225 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb33c07b9d0 0x7fb33c07be00 secure :-1 s=READY pgs=636 cs=0 l=1 rx=0x7fb3340032b0 tx=0x7fb334010c30).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:18:41.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:41.709+0000 7fb34252f700 1 --2- 172.21.15.42:0/842864225 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb33c07b9d0 0x7fb33c07be00 secure :-1 s=THROTTLE_DONE pgs=636 cs=0 l=1 rx=0x7fb3340032b0 tx=0x7fb334010c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:41.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:41.709+0000 7fb3327fc700 1 -- 172.21.15.42:0/842864225 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb334011bc0 con 0x7fb33c07b9d0 2019-12-05T05:18:41.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:41.709+0000 7fb34252f700 1 --2- 172.21.15.42:0/842864225 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb33c07b9d0 0x7fb33c07be00 secure :-1 s=THROTTLE_DONE pgs=636 cs=0 l=1 rx=0x7fb3340032b0 tx=0x7fb334010c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:41.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:41.709+0000 7fb34252f700 1 --2- 172.21.15.42:0/842864225 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb33c07b9d0 0x7fb33c07be00 secure :-1 s=THROTTLE_DONE pgs=636 cs=0 l=1 rx=0x7fb3340032b0 tx=0x7fb334010c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:41.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:41.709+0000 7fb3327fc700 1 -- 172.21.15.42:0/842864225 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3340110c0 con 0x7fb33c07b9d0 2019-12-05T05:18:41.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:41.709+0000 7fb3327fc700 1 -- 172.21.15.42:0/842864225 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb334005ab0 con 0x7fb33c07b9d0 2019-12-05T05:18:41.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:41.709+0000 7fb343791700 1 -- 172.21.15.42:0/842864225 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb33c117090 con 0x7fb33c07b9d0 2019-12-05T05:18:41.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:41.709+0000 7fb343791700 1 -- 172.21.15.42:0/842864225 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb33c138c70 con 0x7fb33c07b9d0 2019-12-05T05:18:41.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:41.709+0000 7fb34252f700 1 --2- 172.21.15.42:0/842864225 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb33c07b9d0 0x7fb33c07be00 secure :-1 s=THROTTLE_DONE pgs=636 cs=0 l=1 rx=0x7fb3340032b0 tx=0x7fb334010c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:41.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:41.709+0000 7fb3327fc700 1 -- 172.21.15.42:0/842864225 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fb33400f280 con 0x7fb33c07b9d0 2019-12-05T05:18:41.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:41.709+0000 7fb34252f700 1 --2- 172.21.15.42:0/842864225 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb33c07b9d0 0x7fb33c07be00 secure :-1 s=THROTTLE_DONE pgs=636 cs=0 l=1 rx=0x7fb3340032b0 tx=0x7fb334010c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:41.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:41.709+0000 7fb3327fc700 1 --2- 172.21.15.42:0/842864225 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb32403b280 0x7fb32403d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:18:41.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:41.713+0000 7fb3327fc700 1 -- 172.21.15.42:0/842864225 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6891+0+0 (secure 0 0 0) 0x7fb33408f680 con 0x7fb33c07b9d0 2019-12-05T05:18:41.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:41.713+0000 7fb341d2e700 1 --2- 172.21.15.42:0/842864225 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb32403b280 0x7fb32403d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:18:41.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:41.713+0000 7fb341d2e700 1 --2- 172.21.15.42:0/842864225 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb32403b280 0x7fb32403d790 crc :-1 s=READY pgs=290 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:18:41.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:41.713+0000 7fb343791700 1 -- 172.21.15.42:0/842864225 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb33c05d110 con 0x7fb33c07b9d0 2019-12-05T05:18:41.725 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:41.721+0000 7fb34252f700 1 --2- 172.21.15.42:0/842864225 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb33c07b9d0 0x7fb33c07be00 secure :-1 s=THROTTLE_DONE pgs=636 cs=0 l=1 rx=0x7fb3340032b0 tx=0x7fb334010c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:41.725 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:41.721+0000 7fb3327fc700 1 -- 172.21.15.42:0/842864225 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fb334076940 con 0x7fb33c07b9d0 2019-12-05T05:18:41.893 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:41.889+0000 7fb343791700 1 -- 172.21.15.42:0/842864225 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb33c05b8c0 con 0x7fb33c07b9d0 2019-12-05T05:18:41.895 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:41.889+0000 7fb34252f700 1 --2- 172.21.15.42:0/842864225 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb33c07b9d0 0x7fb33c07be00 secure :-1 s=THROTTLE_DONE pgs=636 cs=0 l=1 rx=0x7fb3340032b0 tx=0x7fb334010c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:41.895 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:41.889+0000 7fb3327fc700 1 -- 172.21.15.42:0/842864225 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+2108 (secure 0 0 0) 0x7fb3340774f0 con 0x7fb33c07b9d0 2019-12-05T05:18:41.897 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 83 2019-12-05T05:18:41.897 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:18:41.897 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":83,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5833,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6835/1414371821","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1414371821},{"type":"v1","addr":"172.21.15.42:6835","nonce":1414371821}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":83}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:17:17.949394+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5844},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5844":{"gid":5844,"name":"b","rank":0,"incarnation":79,"state":"up:active","state_seq":32,"addr":"172.21.15.40:6833/2698421979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2698421979},{"type":"v1","addr":"172.21.15.40:6833","nonce":2698421979}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:18:41.900 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:41.893+0000 7fb343791700 1 -- 172.21.15.42:0/842864225 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb32403b280 msgr2=0x7fb32403d790 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:18:41.901 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:41.893+0000 7fb343791700 1 --2- 172.21.15.42:0/842864225 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb32403b280 0x7fb32403d790 crc :-1 s=READY pgs=290 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:18:41.901 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:41.893+0000 7fb343791700 1 -- 172.21.15.42:0/842864225 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb33c07b9d0 msgr2=0x7fb33c07be00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:18:41.901 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:41.893+0000 7fb343791700 1 --2- 172.21.15.42:0/842864225 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb33c07b9d0 0x7fb33c07be00 secure :-1 s=READY pgs=636 cs=0 l=1 rx=0x7fb3340032b0 tx=0x7fb334010c30).stop 2019-12-05T05:18:41.901 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:41.893+0000 7fb343791700 1 -- 172.21.15.42:0/842864225 shutdown_connections 2019-12-05T05:18:41.902 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:41.893+0000 7fb343791700 1 --2- 172.21.15.42:0/842864225 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb32403b280 0x7fb32403d790 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:18:41.902 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:41.893+0000 7fb343791700 1 --2- 172.21.15.42:0/842864225 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb33c07b9d0 0x7fb33c07be00 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:18:41.902 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:41.893+0000 7fb343791700 1 --2- 172.21.15.42:0/842864225 >> v2:172.21.15.40:3300/0 conn(0x7fb33c1175a0 0x7fb33c11b580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:18:41.902 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:41.893+0000 7fb343791700 1 -- 172.21.15.42:0/842864225 shutdown_connections 2019-12-05T05:18:41.903 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:41.893+0000 7fb343791700 1 -- 172.21.15.42:0/842864225 wait complete. 2019-12-05T05:18:41.903 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:41.897+0000 7fb343791700 1 -- 172.21.15.42:0/842864225 >> 172.21.15.42:0/842864225 conn(0x7fb33c05a2d0 msgr2=0x7fb33c11bf80 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:18:41.910 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T05:18:43.910 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:18:44.015 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:44.005+0000 7f975bed0700 1 Processor -- start 2019-12-05T05:18:44.016 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:44.005+0000 7f975bed0700 1 -- start start 2019-12-05T05:18:44.016 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:44.005+0000 7f975bed0700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f97540a7740 0x7f975408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:18:44.016 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:44.005+0000 7f975bed0700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f975407e720 0x7f975407eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:18:44.016 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:44.005+0000 7f975bed0700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f975409f2e0 0x7f975409f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:18:44.016 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:44.005+0000 7f975bed0700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97540a0b00 con 0x7f975407e720 2019-12-05T05:18:44.016 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:44.005+0000 7f975bed0700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f975408e5b0 con 0x7f975409f2e0 2019-12-05T05:18:44.017 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:44.005+0000 7f975bed0700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f975408e850 con 0x7f97540a7740 2019-12-05T05:18:44.017 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:44.009+0000 7f975a46d700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f97540a7740 0x7f975408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:18:44.017 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:44.009+0000 7f975ac6e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f975409f2e0 0x7f975409f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:18:44.017 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:44.009+0000 7f975a46d700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f97540a7740 0x7f975408df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:59724/0 (socket says 172.21.15.42:59724) 2019-12-05T05:18:44.017 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:44.009+0000 7f975a46d700 1 -- 172.21.15.42:0/2566041416 learned_addr learned my addr 172.21.15.42:0/2566041416 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:18:44.017 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:44.009+0000 7f9759c6c700 1 --2- 172.21.15.42:0/2566041416 >> v2:172.21.15.40:3300/0 conn(0x7f975407e720 0x7f975407eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:18:44.018 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:44.009+0000 7f975a46d700 1 -- 172.21.15.42:0/2566041416 >> v2:172.21.15.42:3300/0 conn(0x7f975409f2e0 msgr2=0x7f975409f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:18:44.018 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:44.009+0000 7f975a46d700 1 --2- 172.21.15.42:0/2566041416 >> v2:172.21.15.42:3300/0 conn(0x7f975409f2e0 0x7f975409f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:18:44.018 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:44.009+0000 7f975a46d700 1 -- 172.21.15.42:0/2566041416 >> v2:172.21.15.40:3300/0 conn(0x7f975407e720 msgr2=0x7f975407eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:18:44.018 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:44.009+0000 7f975a46d700 1 --2- 172.21.15.42:0/2566041416 >> v2:172.21.15.40:3300/0 conn(0x7f975407e720 0x7f975407eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:18:44.018 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:44.009+0000 7f975a46d700 1 -- 172.21.15.42:0/2566041416 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f975407f0a0 con 0x7f97540a7740 2019-12-05T05:18:44.018 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:44.009+0000 7f975a46d700 1 --2- 172.21.15.42:0/2566041416 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f97540a7740 0x7f975408df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f973c002f30 tx=0x7f973c00fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:44.019 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:44.009+0000 7f975a46d700 1 --2- 172.21.15.42:0/2566041416 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f97540a7740 0x7f975408df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f973c002f30 tx=0x7f973c00fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:44.019 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:44.009+0000 7f975a46d700 1 --2- 172.21.15.42:0/2566041416 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f97540a7740 0x7f975408df80 secure :-1 s=READY pgs=131 cs=0 l=1 rx=0x7f973c002f30 tx=0x7f973c00fdb0).ready entity=mon.2 client_cookie=201c829fdf19dda server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:18:44.019 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:44.009+0000 7f975a46d700 1 --2- 172.21.15.42:0/2566041416 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f97540a7740 0x7f975408df80 secure :-1 s=THROTTLE_DONE pgs=131 cs=0 l=1 rx=0x7f973c002f30 tx=0x7f973c00fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:44.019 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:44.009+0000 7f975946b700 1 -- 172.21.15.42:0/2566041416 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f973c008c90 con 0x7f97540a7740 2019-12-05T05:18:44.019 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:44.009+0000 7f975a46d700 1 --2- 172.21.15.42:0/2566041416 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f97540a7740 0x7f975408df80 secure :-1 s=THROTTLE_DONE pgs=131 cs=0 l=1 rx=0x7f973c002f30 tx=0x7f973c00fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:44.019 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:44.009+0000 7f975946b700 1 -- 172.21.15.42:0/2566041416 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f973c0088b0 con 0x7f97540a7740 2019-12-05T05:18:44.020 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:44.009+0000 7f975a46d700 1 --2- 172.21.15.42:0/2566041416 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f97540a7740 0x7f975408df80 secure :-1 s=THROTTLE_DONE pgs=131 cs=0 l=1 rx=0x7f973c002f30 tx=0x7f973c00fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:44.020 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:44.009+0000 7f975946b700 1 -- 172.21.15.42:0/2566041416 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f973c011b50 con 0x7f97540a7740 2019-12-05T05:18:44.020 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:44.009+0000 7f975bed0700 1 -- 172.21.15.42:0/2566041416 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f97540a7740 msgr2=0x7f975408df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:18:44.020 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:44.009+0000 7f975bed0700 1 --2- 172.21.15.42:0/2566041416 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f97540a7740 0x7f975408df80 secure :-1 s=READY pgs=131 cs=0 l=1 rx=0x7f973c002f30 tx=0x7f973c00fdb0).stop 2019-12-05T05:18:44.020 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:44.009+0000 7f975bed0700 1 -- 172.21.15.42:0/2566041416 shutdown_connections 2019-12-05T05:18:44.021 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:44.009+0000 7f975bed0700 1 --2- 172.21.15.42:0/2566041416 >> v2:172.21.15.42:3300/0 conn(0x7f975409f2e0 0x7f975409f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:18:44.021 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:44.009+0000 7f975bed0700 1 --2- 172.21.15.42:0/2566041416 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f97540a7740 0x7f975408df80 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:18:44.021 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:44.009+0000 7f975bed0700 1 --2- 172.21.15.42:0/2566041416 >> v2:172.21.15.40:3300/0 conn(0x7f975407e720 0x7f975407eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:18:44.021 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:44.009+0000 7f975bed0700 1 -- 172.21.15.42:0/2566041416 shutdown_connections 2019-12-05T05:18:44.022 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:44.009+0000 7f975bed0700 1 -- 172.21.15.42:0/2566041416 wait complete. 2019-12-05T05:18:44.022 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:44.009+0000 7f975bed0700 1 -- 172.21.15.42:0/2566041416 >> 172.21.15.42:0/2566041416 conn(0x7f97540aabe0 msgr2=0x7f97540a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:18:44.025 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:44.013+0000 7f975bed0700 1 Processor -- start 2019-12-05T05:18:44.025 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:44.013+0000 7f975bed0700 1 -- start start 2019-12-05T05:18:44.025 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:44.013+0000 7f975bed0700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f9754131850 0x7f9754131c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:18:44.025 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:44.013+0000 7f975bed0700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f97540722c0 0x7f9754074700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:18:44.025 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:44.013+0000 7f975bed0700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f975411f440 con 0x7f97541449b0 2019-12-05T05:18:44.026 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:44.013+0000 7f975bed0700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f975411beb0 con 0x7f9754131850 2019-12-05T05:18:44.026 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:44.013+0000 7f975bed0700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f975408fa80 con 0x7f97540722c0 2019-12-05T05:18:44.026 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:44.013+0000 7f975a46d700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f97541449b0 0x7f975411ec40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:59302/0 (socket says 172.21.15.42:59302) 2019-12-05T05:18:44.026 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:44.013+0000 7f975a46d700 1 -- 172.21.15.42:0/2891427968 learned_addr learned my addr 172.21.15.42:0/2891427968 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:18:44.026 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:44.013+0000 7f975ac6e700 1 --2- 172.21.15.42:0/2891427968 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f97540722c0 0x7f9754074700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:18:44.026 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:44.013+0000 7f9759c6c700 1 --2- 172.21.15.42:0/2891427968 >> v2:172.21.15.40:3300/0 conn(0x7f9754131850 0x7f9754131c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:18:44.027 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:44.017+0000 7f9752ffd700 1 -- 172.21.15.42:0/2891427968 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2545089602 0 0) 0x7f975411f440 con 0x7f97541449b0 2019-12-05T05:18:44.027 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:44.017+0000 7f9752ffd700 1 -- 172.21.15.42:0/2891427968 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97380030b0 con 0x7f97541449b0 2019-12-05T05:18:44.027 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:44.017+0000 7f975ac6e700 1 -- 172.21.15.42:0/2891427968 >> v2:172.21.15.40:3300/0 conn(0x7f9754131850 msgr2=0x7f9754131c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:18:44.028 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:44.017+0000 7f975ac6e700 1 --2- 172.21.15.42:0/2891427968 >> v2:172.21.15.40:3300/0 conn(0x7f9754131850 0x7f9754131c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:18:44.029 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:44.017+0000 7f975ac6e700 1 -- 172.21.15.42:0/2891427968 >> v1:172.21.15.42:6789/0 conn(0x7f97541449b0 legacy=0x7f975411ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:18:44.029 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:44.017+0000 7f975ac6e700 1 -- 172.21.15.42:0/2891427968 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9754144de0 con 0x7f97540722c0 2019-12-05T05:18:44.029 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:44.017+0000 7f9759c6c700 1 --2- 172.21.15.42:0/2891427968 >> v2:172.21.15.40:3300/0 conn(0x7f9754131850 0x7f9754131c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:18:44.029 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:44.017+0000 7f975ac6e700 1 --2- 172.21.15.42:0/2891427968 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f97540722c0 0x7f9754074700 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f974c002f30 tx=0x7f974c010980).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:44.029 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:44.017+0000 7f975ac6e700 1 --2- 172.21.15.42:0/2891427968 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f97540722c0 0x7f9754074700 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f974c002f30 tx=0x7f974c010980).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:44.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:44.017+0000 7f975ac6e700 1 --2- 172.21.15.42:0/2891427968 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f97540722c0 0x7f9754074700 secure :-1 s=READY pgs=132 cs=0 l=1 rx=0x7f974c002f30 tx=0x7f974c010980).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:18:44.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:44.017+0000 7f975ac6e700 1 --2- 172.21.15.42:0/2891427968 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f97540722c0 0x7f9754074700 secure :-1 s=THROTTLE_DONE pgs=132 cs=0 l=1 rx=0x7f974c002f30 tx=0x7f974c010980).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:44.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:44.017+0000 7f9752ffd700 1 -- 172.21.15.42:0/2891427968 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f974c002690 con 0x7f97540722c0 2019-12-05T05:18:44.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:44.017+0000 7f975ac6e700 1 --2- 172.21.15.42:0/2891427968 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f97540722c0 0x7f9754074700 secure :-1 s=THROTTLE_DONE pgs=132 cs=0 l=1 rx=0x7f974c002f30 tx=0x7f974c010980).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:44.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:44.017+0000 7f9752ffd700 1 -- 172.21.15.42:0/2891427968 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f974c0022b0 con 0x7f97540722c0 2019-12-05T05:18:44.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:44.017+0000 7f975ac6e700 1 --2- 172.21.15.42:0/2891427968 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f97540722c0 0x7f9754074700 secure :-1 s=THROTTLE_DONE pgs=132 cs=0 l=1 rx=0x7f974c002f30 tx=0x7f974c010980).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:44.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:44.017+0000 7f9752ffd700 1 -- 172.21.15.42:0/2891427968 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f974c002c70 con 0x7f97540722c0 2019-12-05T05:18:44.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:44.017+0000 7f975bed0700 1 -- 172.21.15.42:0/2891427968 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f975412dc40 con 0x7f97540722c0 2019-12-05T05:18:44.032 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:44.017+0000 7f975bed0700 1 -- 172.21.15.42:0/2891427968 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9754074c50 con 0x7f97540722c0 2019-12-05T05:18:44.032 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:44.017+0000 7f975ac6e700 1 --2- 172.21.15.42:0/2891427968 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f97540722c0 0x7f9754074700 secure :-1 s=THROTTLE_DONE pgs=132 cs=0 l=1 rx=0x7f974c002f30 tx=0x7f974c010980).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:44.032 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:44.017+0000 7f975ac6e700 1 --2- 172.21.15.42:0/2891427968 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f97540722c0 0x7f9754074700 secure :-1 s=THROTTLE_DONE pgs=132 cs=0 l=1 rx=0x7f974c002f30 tx=0x7f974c010980).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:44.032 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:44.017+0000 7f9752ffd700 1 -- 172.21.15.42:0/2891427968 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f974c007170 con 0x7f97540722c0 2019-12-05T05:18:44.032 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:44.017+0000 7f9752ffd700 1 --2- 172.21.15.42:0/2891427968 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f973803b210 0x7f973803d720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:18:44.032 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:44.017+0000 7f9752ffd700 1 -- 172.21.15.42:0/2891427968 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6891+0+0 (secure 0 0 0) 0x7f974c04abe0 con 0x7f97540722c0 2019-12-05T05:18:44.032 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:44.017+0000 7f975a46d700 1 --2- 172.21.15.42:0/2891427968 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f973803b210 0x7f973803d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:18:44.033 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:44.021+0000 7f975a46d700 1 --2- 172.21.15.42:0/2891427968 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f973803b210 0x7f973803d720 crc :-1 s=READY pgs=291 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:18:44.033 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:44.021+0000 7f975bed0700 1 -- 172.21.15.42:0/2891427968 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f975405d110 con 0x7f97540722c0 2019-12-05T05:18:44.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:44.029+0000 7f975ac6e700 1 --2- 172.21.15.42:0/2891427968 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f97540722c0 0x7f9754074700 secure :-1 s=THROTTLE_DONE pgs=132 cs=0 l=1 rx=0x7f974c002f30 tx=0x7f974c010980).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:44.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:44.029+0000 7f9752ffd700 1 -- 172.21.15.42:0/2891427968 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f974c031b40 con 0x7f97540722c0 2019-12-05T05:18:44.206 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:44.201+0000 7f975bed0700 1 -- 172.21.15.42:0/2891427968 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f975405b8c0 con 0x7f97540722c0 2019-12-05T05:18:44.207 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:44.201+0000 7f975ac6e700 1 --2- 172.21.15.42:0/2891427968 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f97540722c0 0x7f9754074700 secure :-1 s=THROTTLE_DONE pgs=132 cs=0 l=1 rx=0x7f974c002f30 tx=0x7f974c010980).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:44.207 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:44.201+0000 7f9752ffd700 1 -- 172.21.15.42:0/2891427968 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+2108 (secure 0 0 0) 0x7f974c031da0 con 0x7f97540722c0 2019-12-05T05:18:44.209 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 83 2019-12-05T05:18:44.210 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:18:44.210 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":83,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5833,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6835/1414371821","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1414371821},{"type":"v1","addr":"172.21.15.42:6835","nonce":1414371821}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":83}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:17:17.949394+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5844},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5844":{"gid":5844,"name":"b","rank":0,"incarnation":79,"state":"up:active","state_seq":32,"addr":"172.21.15.40:6833/2698421979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2698421979},{"type":"v1","addr":"172.21.15.40:6833","nonce":2698421979}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:18:44.212 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:44.209+0000 7f975bed0700 1 -- 172.21.15.42:0/2891427968 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f973803b210 msgr2=0x7f973803d720 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:18:44.212 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:44.209+0000 7f975bed0700 1 --2- 172.21.15.42:0/2891427968 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f973803b210 0x7f973803d720 crc :-1 s=READY pgs=291 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:18:44.213 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:44.209+0000 7f975bed0700 1 -- 172.21.15.42:0/2891427968 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f97540722c0 msgr2=0x7f9754074700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:18:44.213 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:44.209+0000 7f975bed0700 1 --2- 172.21.15.42:0/2891427968 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f97540722c0 0x7f9754074700 secure :-1 s=READY pgs=132 cs=0 l=1 rx=0x7f974c002f30 tx=0x7f974c010980).stop 2019-12-05T05:18:44.213 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:44.209+0000 7f975bed0700 1 -- 172.21.15.42:0/2891427968 shutdown_connections 2019-12-05T05:18:44.213 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:44.209+0000 7f975bed0700 1 --2- 172.21.15.42:0/2891427968 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f973803b210 0x7f973803d720 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:18:44.213 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:44.209+0000 7f975bed0700 1 --2- 172.21.15.42:0/2891427968 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f97540722c0 0x7f9754074700 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:18:44.213 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:44.209+0000 7f975bed0700 1 --2- 172.21.15.42:0/2891427968 >> v2:172.21.15.40:3300/0 conn(0x7f9754131850 0x7f9754131c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:18:44.213 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:44.209+0000 7f975bed0700 1 -- 172.21.15.42:0/2891427968 shutdown_connections 2019-12-05T05:18:44.214 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:44.209+0000 7f975bed0700 1 -- 172.21.15.42:0/2891427968 wait complete. 2019-12-05T05:18:44.214 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:44.209+0000 7f975bed0700 1 -- 172.21.15.42:0/2891427968 >> 172.21.15.42:0/2891427968 conn(0x7f975407e4f0 msgr2=0x7f97540a9790 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:18:44.222 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T05:18:46.223 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:18:46.337 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:46.328+0000 7fde58fbd700 1 Processor -- start 2019-12-05T05:18:46.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:46.328+0000 7fde58fbd700 1 -- start start 2019-12-05T05:18:46.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:46.328+0000 7fde58fbd700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fde540a7740 0x7fde5408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:18:46.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:46.328+0000 7fde58fbd700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fde5407e720 0x7fde5407eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:18:46.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:46.328+0000 7fde58fbd700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde540a0710 con 0x7fde5409f2e0 2019-12-05T05:18:46.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:46.328+0000 7fde58fbd700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde540a0b00 con 0x7fde540a7740 2019-12-05T05:18:46.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:46.328+0000 7fde58fbd700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde5408e5b0 con 0x7fde5407e720 2019-12-05T05:18:46.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:46.328+0000 7fde5259c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fde5407e720 0x7fde5407eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:18:46.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:46.328+0000 7fde5259c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fde5407e720 0x7fde5407eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:59272/0 (socket says 172.21.15.42:59272) 2019-12-05T05:18:46.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:46.328+0000 7fde5259c700 1 -- 172.21.15.42:0/3034479331 learned_addr learned my addr 172.21.15.42:0/3034479331 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:18:46.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:46.328+0000 7fde52d9d700 1 --2- 172.21.15.42:0/3034479331 >> v2:172.21.15.40:3300/0 conn(0x7fde540a7740 0x7fde5408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:18:46.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:46.328+0000 7fde5259c700 1 -- 172.21.15.42:0/3034479331 >> v2:172.21.15.40:3300/0 conn(0x7fde540a7740 msgr2=0x7fde5408df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:18:46.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:46.328+0000 7fde5259c700 1 --2- 172.21.15.42:0/3034479331 >> v2:172.21.15.40:3300/0 conn(0x7fde540a7740 0x7fde5408df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:18:46.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:46.328+0000 7fde5259c700 1 -- 172.21.15.42:0/3034479331 >> v1:172.21.15.40:6789/0 conn(0x7fde5409f2e0 legacy=0x7fde5409f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:18:46.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:46.328+0000 7fde5259c700 1 -- 172.21.15.42:0/3034479331 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde5407f0a0 con 0x7fde5407e720 2019-12-05T05:18:46.341 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:46.328+0000 7fde5259c700 1 --2- 172.21.15.42:0/3034479331 >> v2:172.21.15.42:3300/0 conn(0x7fde5407e720 0x7fde5407eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fde3c002f30 tx=0x7fde3c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:46.341 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:46.328+0000 7fde5259c700 1 --2- 172.21.15.42:0/3034479331 >> v2:172.21.15.42:3300/0 conn(0x7fde5407e720 0x7fde5407eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fde3c002f30 tx=0x7fde3c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:46.341 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:46.328+0000 7fde5259c700 1 --2- 172.21.15.42:0/3034479331 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fde5407e720 0x7fde5407eb50 secure :-1 s=READY pgs=638 cs=0 l=1 rx=0x7fde3c002f30 tx=0x7fde3c00fd30).ready entity=mon.1 client_cookie=e3c34704c3cd9090 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:18:46.341 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:46.328+0000 7fde5259c700 1 --2- 172.21.15.42:0/3034479331 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fde5407e720 0x7fde5407eb50 secure :-1 s=THROTTLE_DONE pgs=638 cs=0 l=1 rx=0x7fde3c002f30 tx=0x7fde3c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:46.341 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:46.328+0000 7fde5259c700 1 --2- 172.21.15.42:0/3034479331 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fde5407e720 0x7fde5407eb50 secure :-1 s=THROTTLE_DONE pgs=638 cs=0 l=1 rx=0x7fde3c002f30 tx=0x7fde3c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:46.341 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:46.328+0000 7fde51d9b700 1 -- 172.21.15.42:0/3034479331 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fde3c011370 con 0x7fde5407e720 2019-12-05T05:18:46.342 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:46.328+0000 7fde5259c700 1 --2- 172.21.15.42:0/3034479331 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fde5407e720 0x7fde5407eb50 secure :-1 s=THROTTLE_DONE pgs=638 cs=0 l=1 rx=0x7fde3c002f30 tx=0x7fde3c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:46.342 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:46.328+0000 7fde51d9b700 1 -- 172.21.15.42:0/3034479331 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde3c008bb0 con 0x7fde5407e720 2019-12-05T05:18:46.342 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:46.328+0000 7fde51d9b700 1 -- 172.21.15.42:0/3034479331 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fde3c011950 con 0x7fde5407e720 2019-12-05T05:18:46.342 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:46.328+0000 7fde58fbd700 1 -- 172.21.15.42:0/3034479331 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fde5407e720 msgr2=0x7fde5407eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:18:46.342 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:46.328+0000 7fde58fbd700 1 --2- 172.21.15.42:0/3034479331 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fde5407e720 0x7fde5407eb50 secure :-1 s=READY pgs=638 cs=0 l=1 rx=0x7fde3c002f30 tx=0x7fde3c00fd30).stop 2019-12-05T05:18:46.342 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:46.328+0000 7fde58fbd700 1 -- 172.21.15.42:0/3034479331 shutdown_connections 2019-12-05T05:18:46.342 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:46.328+0000 7fde58fbd700 1 --2- 172.21.15.42:0/3034479331 >> v2:172.21.15.40:3300/0 conn(0x7fde540a7740 0x7fde5408df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:18:46.343 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:46.328+0000 7fde58fbd700 1 --2- 172.21.15.42:0/3034479331 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fde5407e720 0x7fde5407eb50 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:18:46.343 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:46.328+0000 7fde58fbd700 1 -- 172.21.15.42:0/3034479331 shutdown_connections 2019-12-05T05:18:46.343 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:46.328+0000 7fde58fbd700 1 -- 172.21.15.42:0/3034479331 wait complete. 2019-12-05T05:18:46.343 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:46.328+0000 7fde58fbd700 1 -- 172.21.15.42:0/3034479331 >> 172.21.15.42:0/3034479331 conn(0x7fde540aabe0 msgr2=0x7fde540a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:18:46.343 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:46.332+0000 7fde58fbd700 1 Processor -- start 2019-12-05T05:18:46.344 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:46.332+0000 7fde58fbd700 1 -- start start 2019-12-05T05:18:46.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:46.332+0000 7fde58fbd700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fde5408ee90 0x7fde54121950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:18:46.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:46.332+0000 7fde58fbd700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fde541148d0 0x7fde54121ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:18:46.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:46.332+0000 7fde58fbd700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde541223f0 con 0x7fde541252c0 2019-12-05T05:18:46.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:46.332+0000 7fde58fbd700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde54114e20 con 0x7fde541148d0 2019-12-05T05:18:46.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:46.332+0000 7fde58fbd700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde54125cf0 con 0x7fde5408ee90 2019-12-05T05:18:46.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:46.332+0000 7fde52d9d700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fde541252c0 0x7fde541140d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:59318/0 (socket says 172.21.15.42:59318) 2019-12-05T05:18:46.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:46.332+0000 7fde5359e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fde541148d0 0x7fde54121ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:18:46.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:46.332+0000 7fde52d9d700 1 -- 172.21.15.42:0/1083918157 learned_addr learned my addr 172.21.15.42:0/1083918157 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:18:46.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:46.332+0000 7fde5259c700 1 --2- 172.21.15.42:0/1083918157 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fde5408ee90 0x7fde54121950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:18:46.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:46.332+0000 7fde4b7fe700 1 -- 172.21.15.42:0/1083918157 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2051348409 0 0) 0x7fde541223f0 con 0x7fde541252c0 2019-12-05T05:18:46.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:46.332+0000 7fde4b7fe700 1 -- 172.21.15.42:0/1083918157 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde380030b0 con 0x7fde541252c0 2019-12-05T05:18:46.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:46.336+0000 7fde5359e700 1 -- 172.21.15.42:0/1083918157 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fde5408ee90 msgr2=0x7fde54121950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:18:46.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:46.336+0000 7fde5359e700 1 --2- 172.21.15.42:0/1083918157 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fde5408ee90 0x7fde54121950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:18:46.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:46.336+0000 7fde5359e700 1 -- 172.21.15.42:0/1083918157 >> v1:172.21.15.42:6789/0 conn(0x7fde541252c0 legacy=0x7fde541140d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:18:46.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:46.336+0000 7fde5359e700 1 -- 172.21.15.42:0/1083918157 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde540a4150 con 0x7fde541148d0 2019-12-05T05:18:46.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:46.336+0000 7fde5259c700 1 --2- 172.21.15.42:0/1083918157 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fde5408ee90 0x7fde54121950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:18:46.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:46.336+0000 7fde5359e700 1 --2- 172.21.15.42:0/1083918157 >> v2:172.21.15.42:3300/0 conn(0x7fde541148d0 0x7fde54121ea0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fde4c001210 tx=0x7fde4c010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:46.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:46.336+0000 7fde5359e700 1 --2- 172.21.15.42:0/1083918157 >> v2:172.21.15.42:3300/0 conn(0x7fde541148d0 0x7fde54121ea0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fde4c001210 tx=0x7fde4c010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:46.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:46.336+0000 7fde5359e700 1 --2- 172.21.15.42:0/1083918157 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fde541148d0 0x7fde54121ea0 secure :-1 s=READY pgs=640 cs=0 l=1 rx=0x7fde4c001210 tx=0x7fde4c010b40).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:18:46.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:46.336+0000 7fde5359e700 1 --2- 172.21.15.42:0/1083918157 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fde541148d0 0x7fde54121ea0 secure :-1 s=THROTTLE_DONE pgs=640 cs=0 l=1 rx=0x7fde4c001210 tx=0x7fde4c010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:46.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:46.336+0000 7fde4b7fe700 1 -- 172.21.15.42:0/1083918157 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fde4c006dc0 con 0x7fde541148d0 2019-12-05T05:18:46.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:46.336+0000 7fde5359e700 1 --2- 172.21.15.42:0/1083918157 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fde541148d0 0x7fde54121ea0 secure :-1 s=THROTTLE_DONE pgs=640 cs=0 l=1 rx=0x7fde4c001210 tx=0x7fde4c010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:46.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:46.336+0000 7fde5359e700 1 --2- 172.21.15.42:0/1083918157 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fde541148d0 0x7fde54121ea0 secure :-1 s=THROTTLE_DONE pgs=640 cs=0 l=1 rx=0x7fde4c001210 tx=0x7fde4c010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:46.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:46.336+0000 7fde4b7fe700 1 -- 172.21.15.42:0/1083918157 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde4c0069e0 con 0x7fde541148d0 2019-12-05T05:18:46.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:46.336+0000 7fde58fbd700 1 -- 172.21.15.42:0/1083918157 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde5412c250 con 0x7fde541148d0 2019-12-05T05:18:46.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:46.336+0000 7fde4b7fe700 1 -- 172.21.15.42:0/1083918157 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fde4c002620 con 0x7fde541148d0 2019-12-05T05:18:46.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:46.336+0000 7fde58fbd700 1 -- 172.21.15.42:0/1083918157 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde5412c780 con 0x7fde541148d0 2019-12-05T05:18:46.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:46.336+0000 7fde5359e700 1 --2- 172.21.15.42:0/1083918157 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fde541148d0 0x7fde54121ea0 secure :-1 s=THROTTLE_DONE pgs=640 cs=0 l=1 rx=0x7fde4c001210 tx=0x7fde4c010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:46.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:46.336+0000 7fde4b7fe700 1 -- 172.21.15.42:0/1083918157 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fde4c0028f0 con 0x7fde541148d0 2019-12-05T05:18:46.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:46.336+0000 7fde5359e700 1 --2- 172.21.15.42:0/1083918157 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fde541148d0 0x7fde54121ea0 secure :-1 s=THROTTLE_DONE pgs=640 cs=0 l=1 rx=0x7fde4c001210 tx=0x7fde4c010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:46.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:46.336+0000 7fde4b7fe700 1 --2- 172.21.15.42:0/1083918157 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fde3803b270 0x7fde3803d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:18:46.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:46.336+0000 7fde4b7fe700 1 -- 172.21.15.42:0/1083918157 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6891+0+0 (secure 0 0 0) 0x7fde4c04b430 con 0x7fde541148d0 2019-12-05T05:18:46.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:46.336+0000 7fde52d9d700 1 --2- 172.21.15.42:0/1083918157 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fde3803b270 0x7fde3803d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:18:46.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:46.340+0000 7fde52d9d700 1 --2- 172.21.15.42:0/1083918157 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fde3803b270 0x7fde3803d780 crc :-1 s=READY pgs=292 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:18:46.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:46.344+0000 7fde58fbd700 1 -- 172.21.15.42:0/1083918157 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde5405d110 con 0x7fde541148d0 2019-12-05T05:18:46.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:46.348+0000 7fde5359e700 1 --2- 172.21.15.42:0/1083918157 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fde541148d0 0x7fde54121ea0 secure :-1 s=THROTTLE_DONE pgs=640 cs=0 l=1 rx=0x7fde4c001210 tx=0x7fde4c010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:46.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:46.348+0000 7fde4b7fe700 1 -- 172.21.15.42:0/1083918157 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fde4c0324b0 con 0x7fde541148d0 2019-12-05T05:18:46.519 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:46.512+0000 7fde58fbd700 1 -- 172.21.15.42:0/1083918157 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fde5405b8c0 con 0x7fde541148d0 2019-12-05T05:18:46.520 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:46.512+0000 7fde5359e700 1 --2- 172.21.15.42:0/1083918157 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fde541148d0 0x7fde54121ea0 secure :-1 s=THROTTLE_DONE pgs=640 cs=0 l=1 rx=0x7fde4c001210 tx=0x7fde4c010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:46.520 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:46.512+0000 7fde4b7fe700 1 -- 172.21.15.42:0/1083918157 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+2108 (secure 0 0 0) 0x7fde4c032710 con 0x7fde541148d0 2019-12-05T05:18:46.520 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 83 2019-12-05T05:18:46.521 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:18:46.521 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":83,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5833,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6835/1414371821","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1414371821},{"type":"v1","addr":"172.21.15.42:6835","nonce":1414371821}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":83}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:17:17.949394+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5844},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5844":{"gid":5844,"name":"b","rank":0,"incarnation":79,"state":"up:active","state_seq":32,"addr":"172.21.15.40:6833/2698421979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2698421979},{"type":"v1","addr":"172.21.15.40:6833","nonce":2698421979}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:18:46.530 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:46.516+0000 7fde58fbd700 1 -- 172.21.15.42:0/1083918157 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fde3803b270 msgr2=0x7fde3803d780 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:18:46.530 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:46.516+0000 7fde58fbd700 1 --2- 172.21.15.42:0/1083918157 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fde3803b270 0x7fde3803d780 crc :-1 s=READY pgs=292 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:18:46.530 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:46.516+0000 7fde58fbd700 1 -- 172.21.15.42:0/1083918157 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fde541148d0 msgr2=0x7fde54121ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:18:46.530 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:46.516+0000 7fde58fbd700 1 --2- 172.21.15.42:0/1083918157 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fde541148d0 0x7fde54121ea0 secure :-1 s=READY pgs=640 cs=0 l=1 rx=0x7fde4c001210 tx=0x7fde4c010b40).stop 2019-12-05T05:18:46.530 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:46.516+0000 7fde58fbd700 1 -- 172.21.15.42:0/1083918157 shutdown_connections 2019-12-05T05:18:46.531 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:46.516+0000 7fde58fbd700 1 --2- 172.21.15.42:0/1083918157 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fde3803b270 0x7fde3803d780 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:18:46.531 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:46.516+0000 7fde58fbd700 1 --2- 172.21.15.42:0/1083918157 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fde541148d0 0x7fde54121ea0 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:18:46.531 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:46.516+0000 7fde58fbd700 1 --2- 172.21.15.42:0/1083918157 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fde5408ee90 0x7fde54121950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:18:46.531 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:46.516+0000 7fde58fbd700 1 -- 172.21.15.42:0/1083918157 shutdown_connections 2019-12-05T05:18:46.531 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:46.516+0000 7fde58fbd700 1 -- 172.21.15.42:0/1083918157 wait complete. 2019-12-05T05:18:46.531 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:46.516+0000 7fde58fbd700 1 -- 172.21.15.42:0/1083918157 >> 172.21.15.42:0/1083918157 conn(0x7fde540a9c10 msgr2=0x7fde54141630 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:18:46.533 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T05:18:48.534 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:18:48.630 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.616+0000 7f6397a46700 1 Processor -- start 2019-12-05T05:18:48.630 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.616+0000 7f6397a46700 1 -- start start 2019-12-05T05:18:48.630 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.616+0000 7f6397a46700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f639008df80 0x7f639009f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:18:48.630 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.616+0000 7f6397a46700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f639009f830 0x7f639014efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:18:48.630 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.616+0000 7f6397a46700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f639009fc60 con 0x7f63900a7740 2019-12-05T05:18:48.631 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.616+0000 7f6397a46700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63900a0b00 con 0x7f639009f830 2019-12-05T05:18:48.631 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.616+0000 7f6397a46700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f639008e5b0 con 0x7f639008df80 2019-12-05T05:18:48.631 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.616+0000 7f63957e2700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f639008df80 0x7f639009f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:18:48.631 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.616+0000 7f6395fe3700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f63900a7740 0x7f639007e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:59346/0 (socket says 172.21.15.42:59346) 2019-12-05T05:18:48.632 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.616+0000 7f6395fe3700 1 -- 172.21.15.42:0/235352718 learned_addr learned my addr 172.21.15.42:0/235352718 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:18:48.632 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.616+0000 7f63967e4700 1 --2- 172.21.15.42:0/235352718 >> v2:172.21.15.40:3300/0 conn(0x7f639009f830 0x7f639014efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:18:48.632 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.620+0000 7f63957e2700 1 -- 172.21.15.42:0/235352718 >> v2:172.21.15.40:3300/0 conn(0x7f639009f830 msgr2=0x7f639014efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:18:48.632 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.620+0000 7f63957e2700 1 --2- 172.21.15.42:0/235352718 >> v2:172.21.15.40:3300/0 conn(0x7f639009f830 0x7f639014efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:18:48.632 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.620+0000 7f63957e2700 1 -- 172.21.15.42:0/235352718 >> v1:172.21.15.42:6789/0 conn(0x7f63900a7740 legacy=0x7f639007e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:18:48.633 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.620+0000 7f6394fe1700 1 -- 172.21.15.42:0/235352718 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4052969395 0 0) 0x7f639009fc60 con 0x7f63900a7740 2019-12-05T05:18:48.633 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.620+0000 7f63957e2700 1 -- 172.21.15.42:0/235352718 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63900a0710 con 0x7f639008df80 2019-12-05T05:18:48.633 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.620+0000 7f63957e2700 1 --2- 172.21.15.42:0/235352718 >> v2:172.21.15.42:3300/0 conn(0x7f639008df80 0x7f639009f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f6380002f30 tx=0x7f638000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:48.633 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.620+0000 7f63957e2700 1 --2- 172.21.15.42:0/235352718 >> v2:172.21.15.42:3300/0 conn(0x7f639008df80 0x7f639009f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f6380002f30 tx=0x7f638000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:48.633 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.620+0000 7f63957e2700 1 --2- 172.21.15.42:0/235352718 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f639008df80 0x7f639009f2e0 secure :-1 s=READY pgs=643 cs=0 l=1 rx=0x7f6380002f30 tx=0x7f638000fc70).ready entity=mon.1 client_cookie=daf05cebc90b1149 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:18:48.633 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.620+0000 7f63957e2700 1 --2- 172.21.15.42:0/235352718 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f639008df80 0x7f639009f2e0 secure :-1 s=THROTTLE_DONE pgs=643 cs=0 l=1 rx=0x7f6380002f30 tx=0x7f638000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:48.634 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.620+0000 7f6394fe1700 1 -- 172.21.15.42:0/235352718 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6380008bb0 con 0x7f639008df80 2019-12-05T05:18:48.634 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.620+0000 7f63957e2700 1 --2- 172.21.15.42:0/235352718 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f639008df80 0x7f639009f2e0 secure :-1 s=THROTTLE_DONE pgs=643 cs=0 l=1 rx=0x7f6380002f30 tx=0x7f638000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:48.634 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.620+0000 7f6394fe1700 1 -- 172.21.15.42:0/235352718 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63800087d0 con 0x7f639008df80 2019-12-05T05:18:48.634 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.620+0000 7f63957e2700 1 --2- 172.21.15.42:0/235352718 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f639008df80 0x7f639009f2e0 secure :-1 s=THROTTLE_DONE pgs=643 cs=0 l=1 rx=0x7f6380002f30 tx=0x7f638000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:48.634 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.620+0000 7f6397a46700 1 -- 172.21.15.42:0/235352718 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f639008df80 msgr2=0x7f639009f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:18:48.634 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.620+0000 7f6394fe1700 1 -- 172.21.15.42:0/235352718 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6380011820 con 0x7f639008df80 2019-12-05T05:18:48.635 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.620+0000 7f6397a46700 1 --2- 172.21.15.42:0/235352718 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f639008df80 0x7f639009f2e0 secure :-1 s=READY pgs=643 cs=0 l=1 rx=0x7f6380002f30 tx=0x7f638000fc70).stop 2019-12-05T05:18:48.635 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.620+0000 7f6397a46700 1 -- 172.21.15.42:0/235352718 shutdown_connections 2019-12-05T05:18:48.635 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.620+0000 7f6397a46700 1 --2- 172.21.15.42:0/235352718 >> v2:172.21.15.40:3300/0 conn(0x7f639009f830 0x7f639014efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:18:48.635 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.620+0000 7f6397a46700 1 --2- 172.21.15.42:0/235352718 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f639008df80 0x7f639009f2e0 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:18:48.635 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.620+0000 7f6397a46700 1 -- 172.21.15.42:0/235352718 shutdown_connections 2019-12-05T05:18:48.636 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.620+0000 7f6397a46700 1 -- 172.21.15.42:0/235352718 wait complete. 2019-12-05T05:18:48.636 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.620+0000 7f6397a46700 1 -- 172.21.15.42:0/235352718 >> 172.21.15.42:0/235352718 conn(0x7f63900aabe0 msgr2=0x7f63900a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:18:48.636 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.620+0000 7f6397a46700 1 Processor -- start 2019-12-05T05:18:48.636 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.620+0000 7f6397a46700 1 -- start start 2019-12-05T05:18:48.636 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.620+0000 7f6397a46700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f63901182b0 0x7f639011a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:18:48.637 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.620+0000 7f6397a46700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f63900a6eb0 0x7f6390138160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:18:48.637 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.620+0000 7f6397a46700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63901386b0 con 0x7f63900a6a80 2019-12-05T05:18:48.637 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.620+0000 7f6397a46700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63900a51c0 con 0x7f63900a6eb0 2019-12-05T05:18:48.637 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.620+0000 7f6397a46700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63900a5460 con 0x7f63901182b0 2019-12-05T05:18:48.637 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.620+0000 7f6395fe3700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f63901182b0 0x7f639011a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:18:48.638 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.620+0000 7f6395fe3700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f63901182b0 0x7f639011a5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:59780/0 (socket says 172.21.15.42:59780) 2019-12-05T05:18:48.638 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.620+0000 7f6395fe3700 1 -- 172.21.15.42:0/3346341040 learned_addr learned my addr 172.21.15.42:0/3346341040 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:18:48.638 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.620+0000 7f63967e4700 1 --2- 172.21.15.42:0/3346341040 >> v2:172.21.15.40:3300/0 conn(0x7f63900a6eb0 0x7f6390138160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:18:48.638 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.624+0000 7f6395fe3700 1 -- 172.21.15.42:0/3346341040 >> v2:172.21.15.40:3300/0 conn(0x7f63900a6eb0 msgr2=0x7f6390138160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:18:48.638 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.624+0000 7f6395fe3700 1 --2- 172.21.15.42:0/3346341040 >> v2:172.21.15.40:3300/0 conn(0x7f63900a6eb0 0x7f6390138160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:18:48.638 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.624+0000 7f6395fe3700 1 -- 172.21.15.42:0/3346341040 >> v1:172.21.15.40:6789/0 conn(0x7f63900a6a80 legacy=0x7f6390137960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:18:48.639 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.624+0000 7f6395fe3700 1 -- 172.21.15.42:0/3346341040 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63900a3aa0 con 0x7f63901182b0 2019-12-05T05:18:48.639 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.624+0000 7f6395fe3700 1 --2- 172.21.15.42:0/3346341040 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f63901182b0 0x7f639011a5c0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f638c0043a0 tx=0x7f638c010940).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:48.639 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.624+0000 7f6395fe3700 1 --2- 172.21.15.42:0/3346341040 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f63901182b0 0x7f639011a5c0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f638c0043a0 tx=0x7f638c010940).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:48.639 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.624+0000 7f6395fe3700 1 --2- 172.21.15.42:0/3346341040 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f63901182b0 0x7f639011a5c0 secure :-1 s=READY pgs=135 cs=0 l=1 rx=0x7f638c0043a0 tx=0x7f638c010940).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:18:48.639 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.624+0000 7f6395fe3700 1 --2- 172.21.15.42:0/3346341040 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f63901182b0 0x7f639011a5c0 secure :-1 s=THROTTLE_DONE pgs=135 cs=0 l=1 rx=0x7f638c0043a0 tx=0x7f638c010940).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:48.640 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.624+0000 7f63867fc700 1 -- 172.21.15.42:0/3346341040 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f638c011670 con 0x7f63901182b0 2019-12-05T05:18:48.640 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.624+0000 7f6395fe3700 1 --2- 172.21.15.42:0/3346341040 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f63901182b0 0x7f639011a5c0 secure :-1 s=THROTTLE_DONE pgs=135 cs=0 l=1 rx=0x7f638c0043a0 tx=0x7f638c010940).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:48.640 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.624+0000 7f6397a46700 1 -- 172.21.15.42:0/3346341040 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f639011ab10 con 0x7f63901182b0 2019-12-05T05:18:48.640 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.624+0000 7f63867fc700 1 -- 172.21.15.42:0/3346341040 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f638c011290 con 0x7f63901182b0 2019-12-05T05:18:48.640 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.624+0000 7f6395fe3700 1 --2- 172.21.15.42:0/3346341040 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f63901182b0 0x7f639011a5c0 secure :-1 s=THROTTLE_DONE pgs=135 cs=0 l=1 rx=0x7f638c0043a0 tx=0x7f638c010940).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:48.640 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.624+0000 7f63867fc700 1 -- 172.21.15.42:0/3346341040 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f638c011c50 con 0x7f63901182b0 2019-12-05T05:18:48.641 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.624+0000 7f6397a46700 1 -- 172.21.15.42:0/3346341040 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f639012f3f0 con 0x7f63901182b0 2019-12-05T05:18:48.641 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.624+0000 7f6395fe3700 1 --2- 172.21.15.42:0/3346341040 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f63901182b0 0x7f639011a5c0 secure :-1 s=THROTTLE_DONE pgs=135 cs=0 l=1 rx=0x7f638c0043a0 tx=0x7f638c010940).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:48.642 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.624+0000 7f63867fc700 1 -- 172.21.15.42:0/3346341040 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f638c00b330 con 0x7f63901182b0 2019-12-05T05:18:48.642 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.624+0000 7f6395fe3700 1 --2- 172.21.15.42:0/3346341040 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f63901182b0 0x7f639011a5c0 secure :-1 s=THROTTLE_DONE pgs=135 cs=0 l=1 rx=0x7f638c0043a0 tx=0x7f638c010940).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:48.643 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.624+0000 7f63867fc700 1 --2- 172.21.15.42:0/3346341040 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f637803ad00 0x7f637803d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:18:48.643 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.624+0000 7f63867fc700 1 -- 172.21.15.42:0/3346341040 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6977+0+0 (secure 0 0 0) 0x7f638c04b230 con 0x7f63901182b0 2019-12-05T05:18:48.643 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.624+0000 7f63957e2700 1 --2- 172.21.15.42:0/3346341040 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f637803ad00 0x7f637803d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:18:48.643 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.624+0000 7f63957e2700 1 --2- 172.21.15.42:0/3346341040 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f637803ad00 0x7f637803d210 crc :-1 s=READY pgs=293 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:18:48.643 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.632+0000 7f6397a46700 1 -- 172.21.15.42:0/3346341040 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f639005d110 con 0x7f63901182b0 2019-12-05T05:18:48.647 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.640+0000 7f6395fe3700 1 --2- 172.21.15.42:0/3346341040 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f63901182b0 0x7f639011a5c0 secure :-1 s=THROTTLE_DONE pgs=135 cs=0 l=1 rx=0x7f638c0043a0 tx=0x7f638c010940).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:48.647 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.640+0000 7f63867fc700 1 -- 172.21.15.42:0/3346341040 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f638c032290 con 0x7f63901182b0 2019-12-05T05:18:48.813 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.808+0000 7f6397a46700 1 -- 172.21.15.42:0/3346341040 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f639005b8c0 con 0x7f63901182b0 2019-12-05T05:18:48.814 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.808+0000 7f6395fe3700 1 --2- 172.21.15.42:0/3346341040 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f63901182b0 0x7f639011a5c0 secure :-1 s=THROTTLE_DONE pgs=135 cs=0 l=1 rx=0x7f638c0043a0 tx=0x7f638c010940).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:48.814 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.808+0000 7f63867fc700 1 -- 172.21.15.42:0/3346341040 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+1771 (secure 0 0 0) 0x7f638c0324f0 con 0x7f63901182b0 2019-12-05T05:18:48.816 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:18:48.817 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":84,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":84,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:18:48.448373+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5833},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5833":{"gid":5833,"name":"a","rank":0,"incarnation":84,"state":"up:replay","state_seq":1,"addr":"172.21.15.42:6835/1414371821","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1414371821},{"type":"v1","addr":"172.21.15.42:6835","nonce":1414371821}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:18:48.817 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 84 2019-12-05T05:18:48.819 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.812+0000 7f6397a46700 1 -- 172.21.15.42:0/3346341040 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f637803ad00 msgr2=0x7f637803d210 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:18:48.819 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.812+0000 7f6397a46700 1 --2- 172.21.15.42:0/3346341040 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f637803ad00 0x7f637803d210 crc :-1 s=READY pgs=293 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:18:48.819 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.812+0000 7f6397a46700 1 -- 172.21.15.42:0/3346341040 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f63901182b0 msgr2=0x7f639011a5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:18:48.820 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.812+0000 7f6397a46700 1 --2- 172.21.15.42:0/3346341040 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f63901182b0 0x7f639011a5c0 secure :-1 s=READY pgs=135 cs=0 l=1 rx=0x7f638c0043a0 tx=0x7f638c010940).stop 2019-12-05T05:18:48.820 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.812+0000 7f6397a46700 1 -- 172.21.15.42:0/3346341040 shutdown_connections 2019-12-05T05:18:48.820 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.812+0000 7f6397a46700 1 --2- 172.21.15.42:0/3346341040 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f637803ad00 0x7f637803d210 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:18:48.820 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.812+0000 7f6397a46700 1 --2- 172.21.15.42:0/3346341040 >> v2:172.21.15.40:3300/0 conn(0x7f63900a6eb0 0x7f6390138160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:18:48.820 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.812+0000 7f6397a46700 1 --2- 172.21.15.42:0/3346341040 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f63901182b0 0x7f639011a5c0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:18:48.821 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.812+0000 7f6397a46700 1 -- 172.21.15.42:0/3346341040 shutdown_connections 2019-12-05T05:18:48.821 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.812+0000 7f6397a46700 1 -- 172.21.15.42:0/3346341040 wait complete. 2019-12-05T05:18:48.821 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.812+0000 7f6397a46700 1 -- 172.21.15.42:0/3346341040 >> 172.21.15.42:0/3346341040 conn(0x7f63900a82a0 msgr2=0x7f639007e790 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:18:48.828 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b down, removed from mdsmap 2019-12-05T05:18:48.828 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2019-12-05T05:18:48.829 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:18:48.939 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.928+0000 7fe6cbf72700 1 Processor -- start 2019-12-05T05:18:48.939 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.928+0000 7fe6cbf72700 1 -- start start 2019-12-05T05:18:48.939 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.928+0000 7fe6cbf72700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe6c40a7740 0x7fe6c408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:18:48.940 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.928+0000 7fe6cbf72700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6c40a0710 con 0x7fe6c409fae0 2019-12-05T05:18:48.940 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.928+0000 7fe6cbf72700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6c40a9ba0 con 0x7fe6c407e720 2019-12-05T05:18:48.940 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.928+0000 7fe6cbf72700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6c40a0b00 con 0x7fe6c40a7740 2019-12-05T05:18:48.940 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.928+0000 7fe6c9d0e700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fe6c407e720 0x7fe6c409f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:59360/0 (socket says 172.21.15.42:59360) 2019-12-05T05:18:48.940 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.928+0000 7fe6c9d0e700 1 -- 172.21.15.42:0/2171183885 learned_addr learned my addr 172.21.15.42:0/2171183885 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:18:48.941 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.928+0000 7fe6ca50f700 1 --2- 172.21.15.42:0/2171183885 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe6c40a7740 0x7fe6c408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:18:48.941 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.928+0000 7fe6c950d700 1 -- 172.21.15.42:0/2171183885 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1639060053 0 0) 0x7fe6c40a9ba0 con 0x7fe6c407e720 2019-12-05T05:18:48.941 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.928+0000 7fe6c950d700 1 -- 172.21.15.42:0/2171183885 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6b00030b0 con 0x7fe6c407e720 2019-12-05T05:18:48.941 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.928+0000 7fe6c950d700 1 -- 172.21.15.42:0/2171183885 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 135694688 0 0) 0x7fe6b00030b0 con 0x7fe6c407e720 2019-12-05T05:18:48.941 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.928+0000 7fe6c950d700 1 -- 172.21.15.42:0/2171183885 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe6c40a7740 msgr2=0x7fe6c408df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:18:48.941 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.928+0000 7fe6c950d700 1 --2- 172.21.15.42:0/2171183885 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe6c40a7740 0x7fe6c408df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:18:48.941 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.928+0000 7fe6c950d700 1 -- 172.21.15.42:0/2171183885 >> v1:172.21.15.40:6789/0 conn(0x7fe6c409fae0 legacy=0x7fe6c414efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:18:48.942 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.928+0000 7fe6c950d700 1 -- 172.21.15.42:0/2171183885 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6c407eb50 con 0x7fe6c407e720 2019-12-05T05:18:48.942 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.928+0000 7fe6c950d700 1 -- 172.21.15.42:0/2171183885 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7fe6b4001750 con 0x7fe6c407e720 2019-12-05T05:18:48.942 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.928+0000 7fe6c950d700 1 -- 172.21.15.42:0/2171183885 >> v1:172.21.15.42:6789/0 conn(0x7fe6c407e720 legacy=0x7fe6c409f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:18:48.942 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.928+0000 7fe6c950d700 1 --2- 172.21.15.42:0/2171183885 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe6b0003710 0x7fe6b0003b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:18:48.942 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.928+0000 7fe6c950d700 1 --2- 172.21.15.42:0/2171183885 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe6b0004420 0x7fe6b0004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:18:48.942 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.928+0000 7fe6c950d700 1 --2- 172.21.15.42:0/2171183885 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe6b0004da0 0x7fe6b000b670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:18:48.942 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.928+0000 7fe6c950d700 1 -- 172.21.15.42:0/2171183885 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6c40a0b00 con 0x7fe6b0003710 2019-12-05T05:18:48.943 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.928+0000 7fe6c950d700 1 -- 172.21.15.42:0/2171183885 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6b00051d0 con 0x7fe6b0004420 2019-12-05T05:18:48.943 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.928+0000 7fe6c950d700 1 -- 172.21.15.42:0/2171183885 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6b004fbd0 con 0x7fe6b0004da0 2019-12-05T05:18:48.943 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.928+0000 7fe6c9d0e700 1 --2- 172.21.15.42:0/2171183885 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe6b0004420 0x7fe6b0004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:18:48.943 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.928+0000 7fe6ca50f700 1 --2- 172.21.15.42:0/2171183885 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe6b0003710 0x7fe6b0003b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:18:48.943 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.928+0000 7fe6c9d0e700 1 -- 172.21.15.42:0/2171183885 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe6b0004da0 msgr2=0x7fe6b000b670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:18:48.943 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.928+0000 7fe6c9d0e700 1 --2- 172.21.15.42:0/2171183885 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe6b0004da0 0x7fe6b000b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:18:48.943 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.928+0000 7fe6c9d0e700 1 -- 172.21.15.42:0/2171183885 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe6b0003710 msgr2=0x7fe6b0003b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:18:48.944 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.928+0000 7fe6c9d0e700 1 --2- 172.21.15.42:0/2171183885 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe6b0003710 0x7fe6b0003b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:18:48.944 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.928+0000 7fe6c9d0e700 1 -- 172.21.15.42:0/2171183885 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6b004fed0 con 0x7fe6b0004420 2019-12-05T05:18:48.944 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.928+0000 7fe6c9d0e700 1 --2- 172.21.15.42:0/2171183885 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe6b0004420 0x7fe6b0004850 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fe6b4001950 tx=0x7fe6b400c1e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:48.944 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.928+0000 7fe6c9d0e700 1 --2- 172.21.15.42:0/2171183885 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe6b0004420 0x7fe6b0004850 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fe6b4001950 tx=0x7fe6b400c1e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:48.944 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.928+0000 7fe6c9d0e700 1 --2- 172.21.15.42:0/2171183885 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe6b0004420 0x7fe6b0004850 secure :-1 s=READY pgs=645 cs=0 l=1 rx=0x7fe6b4001950 tx=0x7fe6b400c1e0).ready entity=mon.1 client_cookie=65859caeb7ce4652 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:18:48.944 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.928+0000 7fe6c9d0e700 1 --2- 172.21.15.42:0/2171183885 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe6b0004420 0x7fe6b0004850 secure :-1 s=THROTTLE_DONE pgs=645 cs=0 l=1 rx=0x7fe6b4001950 tx=0x7fe6b400c1e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:48.945 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.928+0000 7fe6c9d0e700 1 --2- 172.21.15.42:0/2171183885 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe6b0004420 0x7fe6b0004850 secure :-1 s=THROTTLE_DONE pgs=645 cs=0 l=1 rx=0x7fe6b4001950 tx=0x7fe6b400c1e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:48.945 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.928+0000 7fe6c950d700 1 -- 172.21.15.42:0/2171183885 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe6b400e940 con 0x7fe6b0004420 2019-12-05T05:18:48.945 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.928+0000 7fe6c9d0e700 1 --2- 172.21.15.42:0/2171183885 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe6b0004420 0x7fe6b0004850 secure :-1 s=THROTTLE_DONE pgs=645 cs=0 l=1 rx=0x7fe6b4001950 tx=0x7fe6b400c1e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:48.945 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.928+0000 7fe6c950d700 1 -- 172.21.15.42:0/2171183885 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6b4010b90 con 0x7fe6b0004420 2019-12-05T05:18:48.945 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.928+0000 7fe6c950d700 1 -- 172.21.15.42:0/2171183885 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe6b4011070 con 0x7fe6b0004420 2019-12-05T05:18:48.945 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.928+0000 7fe6cbf72700 1 -- 172.21.15.42:0/2171183885 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe6b0004420 msgr2=0x7fe6b0004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:18:48.945 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.928+0000 7fe6cbf72700 1 --2- 172.21.15.42:0/2171183885 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe6b0004420 0x7fe6b0004850 secure :-1 s=READY pgs=645 cs=0 l=1 rx=0x7fe6b4001950 tx=0x7fe6b400c1e0).stop 2019-12-05T05:18:48.946 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.928+0000 7fe6cad10700 1 -- 172.21.15.42:0/2171183885 reap_dead start 2019-12-05T05:18:48.946 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.928+0000 7fe6cbf72700 1 -- 172.21.15.42:0/2171183885 shutdown_connections 2019-12-05T05:18:48.946 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.928+0000 7fe6cbf72700 1 -- 172.21.15.42:0/2171183885 shutdown_connections 2019-12-05T05:18:48.946 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.928+0000 7fe6cbf72700 1 -- 172.21.15.42:0/2171183885 wait complete. 2019-12-05T05:18:48.946 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.928+0000 7fe6cbf72700 1 -- 172.21.15.42:0/2171183885 >> 172.21.15.42:0/2171183885 conn(0x7fe6c40aabe0 msgr2=0x7fe6c40a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:18:48.946 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.932+0000 7fe6cbf72700 1 Processor -- start 2019-12-05T05:18:48.946 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.932+0000 7fe6cbf72700 1 -- start start 2019-12-05T05:18:48.947 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.932+0000 7fe6cbf72700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe6c4124d90 0x7fe6c407d240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:18:48.947 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.932+0000 7fe6cbf72700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe6c407a2d0 0x7fe6c407a700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:18:48.947 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.932+0000 7fe6cbf72700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6c407c030 con 0x7fe6c411bce0 2019-12-05T05:18:48.947 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.932+0000 7fe6cbf72700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6c407ca60 con 0x7fe6c407a2d0 2019-12-05T05:18:48.947 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.932+0000 7fe6cbf72700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6c407cd00 con 0x7fe6c4124d90 2019-12-05T05:18:48.948 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.932+0000 7fe6ca50f700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe6c4124d90 0x7fe6c407d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:18:48.948 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.932+0000 7fe6ca50f700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe6c4124d90 0x7fe6c407d240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:59334/0 (socket says 172.21.15.42:59334) 2019-12-05T05:18:48.948 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.932+0000 7fe6ca50f700 1 -- 172.21.15.42:0/716749201 learned_addr learned my addr 172.21.15.42:0/716749201 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:18:48.948 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.932+0000 7fe6cad10700 1 --2- 172.21.15.42:0/716749201 >> v2:172.21.15.40:3300/0 conn(0x7fe6c407a2d0 0x7fe6c407a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:18:48.948 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.932+0000 7fe6ca50f700 1 -- 172.21.15.42:0/716749201 >> v2:172.21.15.40:3300/0 conn(0x7fe6c407a2d0 msgr2=0x7fe6c407a700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:18:48.948 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.932+0000 7fe6ca50f700 1 --2- 172.21.15.42:0/716749201 >> v2:172.21.15.40:3300/0 conn(0x7fe6c407a2d0 0x7fe6c407a700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:18:48.948 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.932+0000 7fe6ca50f700 1 -- 172.21.15.42:0/716749201 >> v1:172.21.15.40:6789/0 conn(0x7fe6c411bce0 legacy=0x7fe6c4142920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:18:48.949 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.932+0000 7fe6ca50f700 1 -- 172.21.15.42:0/716749201 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6c407ac50 con 0x7fe6c4124d90 2019-12-05T05:18:48.949 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.932+0000 7fe6ca50f700 1 --2- 172.21.15.42:0/716749201 >> v2:172.21.15.42:3300/0 conn(0x7fe6c4124d90 0x7fe6c407d240 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fe6c0002f30 tx=0x7fe6c0010c10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:48.949 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.932+0000 7fe6ca50f700 1 --2- 172.21.15.42:0/716749201 >> v2:172.21.15.42:3300/0 conn(0x7fe6c4124d90 0x7fe6c407d240 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fe6c0002f30 tx=0x7fe6c0010c10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:48.949 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.932+0000 7fe6ca50f700 1 --2- 172.21.15.42:0/716749201 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe6c4124d90 0x7fe6c407d240 secure :-1 s=READY pgs=646 cs=0 l=1 rx=0x7fe6c0002f30 tx=0x7fe6c0010c10).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:18:48.949 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.932+0000 7fe6ca50f700 1 --2- 172.21.15.42:0/716749201 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe6c4124d90 0x7fe6c407d240 secure :-1 s=THROTTLE_DONE pgs=646 cs=0 l=1 rx=0x7fe6c0002f30 tx=0x7fe6c0010c10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:48.949 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.932+0000 7fe6ca50f700 1 --2- 172.21.15.42:0/716749201 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe6c4124d90 0x7fe6c407d240 secure :-1 s=THROTTLE_DONE pgs=646 cs=0 l=1 rx=0x7fe6c0002f30 tx=0x7fe6c0010c10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:48.949 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.932+0000 7fe6baffd700 1 -- 172.21.15.42:0/716749201 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe6c0012b20 con 0x7fe6c4124d90 2019-12-05T05:18:48.950 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.932+0000 7fe6ca50f700 1 --2- 172.21.15.42:0/716749201 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe6c4124d90 0x7fe6c407d240 secure :-1 s=THROTTLE_DONE pgs=646 cs=0 l=1 rx=0x7fe6c0002f30 tx=0x7fe6c0010c10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:48.950 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.932+0000 7fe6baffd700 1 -- 172.21.15.42:0/716749201 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6c0012740 con 0x7fe6c4124d90 2019-12-05T05:18:48.950 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.932+0000 7fe6cbf72700 1 -- 172.21.15.42:0/716749201 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6c407af20 con 0x7fe6c4124d90 2019-12-05T05:18:48.950 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.932+0000 7fe6baffd700 1 -- 172.21.15.42:0/716749201 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe6c0011450 con 0x7fe6c4124d90 2019-12-05T05:18:48.950 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.932+0000 7fe6cbf72700 1 -- 172.21.15.42:0/716749201 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6c4137e20 con 0x7fe6c4124d90 2019-12-05T05:18:48.950 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.932+0000 7fe6ca50f700 1 --2- 172.21.15.42:0/716749201 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe6c4124d90 0x7fe6c407d240 secure :-1 s=THROTTLE_DONE pgs=646 cs=0 l=1 rx=0x7fe6c0002f30 tx=0x7fe6c0010c10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:48.950 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.932+0000 7fe6baffd700 1 -- 172.21.15.42:0/716749201 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fe6c0007170 con 0x7fe6c4124d90 2019-12-05T05:18:48.951 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.932+0000 7fe6baffd700 1 --2- 172.21.15.42:0/716749201 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe6ac03ad70 0x7fe6ac03d280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:18:48.951 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.932+0000 7fe6c9d0e700 1 --2- 172.21.15.42:0/716749201 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe6ac03ad70 0x7fe6ac03d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:18:48.951 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.932+0000 7fe6c9d0e700 1 --2- 172.21.15.42:0/716749201 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe6ac03ad70 0x7fe6ac03d280 crc :-1 s=READY pgs=294 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:18:48.952 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.936+0000 7fe6ca50f700 1 --2- 172.21.15.42:0/716749201 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe6c4124d90 0x7fe6c407d240 secure :-1 s=THROTTLE_DONE pgs=646 cs=0 l=1 rx=0x7fe6c0002f30 tx=0x7fe6c0010c10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:48.952 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.936+0000 7fe6cbf72700 1 -- 172.21.15.42:0/716749201 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6c405d110 con 0x7fe6c4124d90 2019-12-05T05:18:48.952 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.936+0000 7fe6baffd700 1 -- 172.21.15.42:0/716749201 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6977+0+0 (secure 0 0 0) 0x7fe6c0032430 con 0x7fe6c4124d90 2019-12-05T05:18:48.952 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.940+0000 7fe6ca50f700 1 --2- 172.21.15.42:0/716749201 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe6c4124d90 0x7fe6c407d240 secure :-1 s=THROTTLE_DONE pgs=646 cs=0 l=1 rx=0x7fe6c0002f30 tx=0x7fe6c0010c10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:48.952 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:48.940+0000 7fe6baffd700 1 -- 172.21.15.42:0/716749201 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fe6c000f0b0 con 0x7fe6c4124d90 2019-12-05T05:18:49.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:49.092+0000 7fe6cbf72700 1 -- 172.21.15.42:0/716749201 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe6c405ae30 con 0x7fe6c4124d90 2019-12-05T05:18:49.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:49.092+0000 7fe6ca50f700 1 --2- 172.21.15.42:0/716749201 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe6c4124d90 0x7fe6c407d240 secure :-1 s=THROTTLE_DONE pgs=646 cs=0 l=1 rx=0x7fe6c0002f30 tx=0x7fe6c0010c10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:49.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:49.092+0000 7fe6baffd700 1 -- 172.21.15.42:0/716749201 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+1771 (secure 0 0 0) 0x7fe6c0032690 con 0x7fe6c4124d90 2019-12-05T05:18:49.100 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 84 2019-12-05T05:18:49.100 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:18:49.100 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":84,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":84,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:18:48.448373+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5833},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5833":{"gid":5833,"name":"a","rank":0,"incarnation":84,"state":"up:replay","state_seq":1,"addr":"172.21.15.42:6835/1414371821","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1414371821},{"type":"v1","addr":"172.21.15.42:6835","nonce":1414371821}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:18:49.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:49.096+0000 7fe6cbf72700 1 -- 172.21.15.42:0/716749201 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe6ac03ad70 msgr2=0x7fe6ac03d280 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:18:49.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:49.096+0000 7fe6cbf72700 1 --2- 172.21.15.42:0/716749201 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe6ac03ad70 0x7fe6ac03d280 crc :-1 s=READY pgs=294 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:18:49.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:49.096+0000 7fe6cbf72700 1 -- 172.21.15.42:0/716749201 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe6c4124d90 msgr2=0x7fe6c407d240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:18:49.103 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:49.096+0000 7fe6cbf72700 1 --2- 172.21.15.42:0/716749201 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe6c4124d90 0x7fe6c407d240 secure :-1 s=READY pgs=646 cs=0 l=1 rx=0x7fe6c0002f30 tx=0x7fe6c0010c10).stop 2019-12-05T05:18:49.103 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:49.096+0000 7fe6cbf72700 1 -- 172.21.15.42:0/716749201 shutdown_connections 2019-12-05T05:18:49.103 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:49.096+0000 7fe6cbf72700 1 --2- 172.21.15.42:0/716749201 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe6ac03ad70 0x7fe6ac03d280 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:18:49.103 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:49.096+0000 7fe6cbf72700 1 --2- 172.21.15.42:0/716749201 >> v2:172.21.15.40:3300/0 conn(0x7fe6c407a2d0 0x7fe6c407a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:18:49.103 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:49.096+0000 7fe6cbf72700 1 --2- 172.21.15.42:0/716749201 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe6c4124d90 0x7fe6c407d240 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:18:49.103 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:49.096+0000 7fe6cbf72700 1 -- 172.21.15.42:0/716749201 shutdown_connections 2019-12-05T05:18:49.103 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:49.096+0000 7fe6cbf72700 1 -- 172.21.15.42:0/716749201 wait complete. 2019-12-05T05:18:49.104 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:49.096+0000 7fe6cbf72700 1 -- 172.21.15.42:0/716749201 >> 172.21.15.42:0/716749201 conn(0x7fe6c40a7740 msgr2=0x7fe6c408eb70 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:18:49.117 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 84, "feature_flags": { "enable_multiple": false, "ever_enabled_multiple": false }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2019-12-05T05:03:35.604738+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 84, "ever_allowed_features": 2, "explicitly_allowed_features": 2, "failed": [], "flags": 18, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5833": { "addr": "172.21.15.42:6835/1414371821", "addrs": { "addrvec": [ { "addr": "172.21.15.42:6834", "nonce": 1414371821, "type": "v2" }, { "addr": "172.21.15.42:6835", "nonce": 1414371821, "type": "v1" } ] }, "export_targets": [], "features": 4540138292836696063, "flags": 0, "gid": 5833, "incarnation": 84, "name": "a", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 60, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 2, "min_compat_client": "0 (unknown)", "modified": "2019-12-05T05:18:48.448373+0000", "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5833 } } } ], "standbys": [] } 2019-12-05T05:18:51.117 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:18:51.214 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:51.204+0000 7f10ff05c700 1 Processor -- start 2019-12-05T05:18:51.214 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:51.204+0000 7f10ff05c700 1 -- start start 2019-12-05T05:18:51.214 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:51.204+0000 7f10ff05c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f10f808df80 0x7f10f809f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:18:51.215 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:51.204+0000 7f10ff05c700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f10f809f830 0x7f10f814efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:18:51.215 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:51.204+0000 7f10ff05c700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10f809fc60 con 0x7f10f80a7740 2019-12-05T05:18:51.215 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:51.204+0000 7f10ff05c700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10f80a0b00 con 0x7f10f809f830 2019-12-05T05:18:51.215 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:51.204+0000 7f10ff05c700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10f808e5b0 con 0x7f10f808df80 2019-12-05T05:18:51.215 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:51.204+0000 7f10fcdf8700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f10f808df80 0x7f10f809f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:18:51.215 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:51.204+0000 7f10fcdf8700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f10f808df80 0x7f10f809f2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:59348/0 (socket says 172.21.15.42:59348) 2019-12-05T05:18:51.215 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:51.204+0000 7f10fcdf8700 1 -- 172.21.15.42:0/612478086 learned_addr learned my addr 172.21.15.42:0/612478086 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:18:51.216 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:51.204+0000 7f10fddfa700 1 --2- 172.21.15.42:0/612478086 >> v2:172.21.15.40:3300/0 conn(0x7f10f809f830 0x7f10f814efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:18:51.216 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:51.208+0000 7f10fcdf8700 1 -- 172.21.15.42:0/612478086 >> v2:172.21.15.40:3300/0 conn(0x7f10f809f830 msgr2=0x7f10f814efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:18:51.216 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:51.208+0000 7f10fcdf8700 1 --2- 172.21.15.42:0/612478086 >> v2:172.21.15.40:3300/0 conn(0x7f10f809f830 0x7f10f814efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:18:51.216 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:51.208+0000 7f10fcdf8700 1 -- 172.21.15.42:0/612478086 >> v1:172.21.15.40:6789/0 conn(0x7f10f80a7740 legacy=0x7f10f807e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:18:51.216 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:51.208+0000 7f10fcdf8700 1 -- 172.21.15.42:0/612478086 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10f80a0710 con 0x7f10f808df80 2019-12-05T05:18:51.216 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:51.208+0000 7f10fddfa700 1 --2- 172.21.15.42:0/612478086 >> v2:172.21.15.40:3300/0 conn(0x7f10f809f830 0x7f10f814efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T05:18:51.216 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:51.208+0000 7f10fcdf8700 1 --2- 172.21.15.42:0/612478086 >> v2:172.21.15.42:3300/0 conn(0x7f10f808df80 0x7f10f809f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f10e8002f30 tx=0x7f10e800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:51.217 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:51.208+0000 7f10fcdf8700 1 --2- 172.21.15.42:0/612478086 >> v2:172.21.15.42:3300/0 conn(0x7f10f808df80 0x7f10f809f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f10e8002f30 tx=0x7f10e800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:51.217 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:51.208+0000 7f10fcdf8700 1 --2- 172.21.15.42:0/612478086 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f10f808df80 0x7f10f809f2e0 secure :-1 s=READY pgs=649 cs=0 l=1 rx=0x7f10e8002f30 tx=0x7f10e800fd30).ready entity=mon.1 client_cookie=3f3209c05f751744 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:18:51.217 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:51.208+0000 7f10fcdf8700 1 --2- 172.21.15.42:0/612478086 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f10f808df80 0x7f10f809f2e0 secure :-1 s=THROTTLE_DONE pgs=649 cs=0 l=1 rx=0x7f10e8002f30 tx=0x7f10e800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:51.217 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:51.208+0000 7f10fcdf8700 1 --2- 172.21.15.42:0/612478086 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f10f808df80 0x7f10f809f2e0 secure :-1 s=THROTTLE_DONE pgs=649 cs=0 l=1 rx=0x7f10e8002f30 tx=0x7f10e800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:51.217 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:51.208+0000 7f10fcdf8700 1 --2- 172.21.15.42:0/612478086 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f10f808df80 0x7f10f809f2e0 secure :-1 s=THROTTLE_DONE pgs=649 cs=0 l=1 rx=0x7f10e8002f30 tx=0x7f10e800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:51.217 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:51.208+0000 7f10effff700 1 -- 172.21.15.42:0/612478086 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f10e8008c90 con 0x7f10f808df80 2019-12-05T05:18:51.217 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:51.208+0000 7f10effff700 1 -- 172.21.15.42:0/612478086 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10e80088b0 con 0x7f10f808df80 2019-12-05T05:18:51.218 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:51.208+0000 7f10effff700 1 -- 172.21.15.42:0/612478086 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f10e8011ad0 con 0x7f10f808df80 2019-12-05T05:18:51.218 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:51.208+0000 7f10ff05c700 1 -- 172.21.15.42:0/612478086 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f10f808df80 msgr2=0x7f10f809f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:18:51.218 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:51.208+0000 7f10ff05c700 1 --2- 172.21.15.42:0/612478086 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f10f808df80 0x7f10f809f2e0 secure :-1 s=READY pgs=649 cs=0 l=1 rx=0x7f10e8002f30 tx=0x7f10e800fd30).stop 2019-12-05T05:18:51.218 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:51.208+0000 7f10ff05c700 1 -- 172.21.15.42:0/612478086 shutdown_connections 2019-12-05T05:18:51.218 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:51.208+0000 7f10ff05c700 1 --2- 172.21.15.42:0/612478086 >> v2:172.21.15.40:3300/0 conn(0x7f10f809f830 0x7f10f814efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:18:51.218 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:51.208+0000 7f10ff05c700 1 --2- 172.21.15.42:0/612478086 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f10f808df80 0x7f10f809f2e0 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:18:51.218 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:51.208+0000 7f10ff05c700 1 -- 172.21.15.42:0/612478086 shutdown_connections 2019-12-05T05:18:51.219 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:51.208+0000 7f10ff05c700 1 -- 172.21.15.42:0/612478086 wait complete. 2019-12-05T05:18:51.219 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:51.208+0000 7f10ff05c700 1 -- 172.21.15.42:0/612478086 >> 172.21.15.42:0/612478086 conn(0x7f10f80aabe0 msgr2=0x7f10f80a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:18:51.220 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:51.212+0000 7f10ff05c700 1 Processor -- start 2019-12-05T05:18:51.221 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:51.212+0000 7f10ff05c700 1 -- start start 2019-12-05T05:18:51.221 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:51.212+0000 7f10ff05c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f10f8116530 0x7f10f8127ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:18:51.221 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:51.212+0000 7f10ff05c700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f10f8145080 0x7f10f8124500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:18:51.221 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:51.212+0000 7f10ff05c700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10f81454b0 con 0x7f10f8124a50 2019-12-05T05:18:51.221 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:51.212+0000 7f10ff05c700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10f81280b0 con 0x7f10f8116530 2019-12-05T05:18:51.221 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:51.212+0000 7f10ff05c700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10f8128350 con 0x7f10f8145080 2019-12-05T05:18:51.222 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:51.212+0000 7f10fd5f9700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f10f8116530 0x7f10f8127ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:18:51.222 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:51.212+0000 7f10fd5f9700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f10f8116530 0x7f10f8127ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:59352/0 (socket says 172.21.15.42:59352) 2019-12-05T05:18:51.222 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:51.212+0000 7f10fd5f9700 1 -- 172.21.15.42:0/2524039416 learned_addr learned my addr 172.21.15.42:0/2524039416 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:18:51.222 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:51.212+0000 7f10fcdf8700 1 --2- 172.21.15.42:0/2524039416 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f10f8145080 0x7f10f8124500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:18:51.222 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:51.212+0000 7f10fd5f9700 1 -- 172.21.15.42:0/2524039416 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f10f8145080 msgr2=0x7f10f8124500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:18:51.222 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:51.212+0000 7f10fd5f9700 1 --2- 172.21.15.42:0/2524039416 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f10f8145080 0x7f10f8124500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:18:51.222 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:51.212+0000 7f10fd5f9700 1 -- 172.21.15.42:0/2524039416 >> v1:172.21.15.40:6789/0 conn(0x7f10f8124a50 legacy=0x7f10f807bcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:18:51.222 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:51.212+0000 7f10fd5f9700 1 -- 172.21.15.42:0/2524039416 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10f8128890 con 0x7f10f8116530 2019-12-05T05:18:51.223 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:51.212+0000 7f10fd5f9700 1 --2- 172.21.15.42:0/2524039416 >> v2:172.21.15.42:3300/0 conn(0x7f10f8116530 0x7f10f8127ac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f10f807a500 tx=0x7f10f40109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:51.223 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:51.212+0000 7f10fd5f9700 1 -- 172.21.15.42:0/2524039416 >> v2:172.21.15.42:3300/0 conn(0x7f10f8116530 msgr2=0x7f10f8127ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2019-12-05T05:18:51.223 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:51.212+0000 7f10fd5f9700 1 -- 172.21.15.42:0/2524039416 >> v2:172.21.15.42:3300/0 conn(0x7f10f8116530 msgr2=0x7f10f8127ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2019-12-05T05:18:51.223 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:51.212+0000 7f10fd5f9700 1 --2- 172.21.15.42:0/2524039416 >> v2:172.21.15.42:3300/0 conn(0x7f10f8116530 0x7f10f8127ac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f10f807a500 tx=0x7f10f40109a0).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-12-05T05:18:51.223 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:51.212+0000 7f10fd5f9700 1 --2- 172.21.15.42:0/2524039416 >> v2:172.21.15.42:3300/0 conn(0x7f10f8116530 0x7f10f8127ac0 secure :-1 s=START_CONNECT pgs=0 cs=0 l=1 rx=0x7f10f807a500 tx=0x7f10f40109a0)._fault waiting 0.200000 2019-12-05T05:18:51.420 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:51.416+0000 7f10fd5f9700 1 --2- 172.21.15.42:0/2524039416 >> v2:172.21.15.42:3300/0 conn(0x7f10f8116530 0x7f10f8127ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:18:51.420 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:51.416+0000 7f10fd5f9700 0 --2- 172.21.15.42:0/2524039416 >> v2:172.21.15.42:3300/0 conn(0x7f10f8116530 0x7f10f8127ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2019-12-05T05:18:51.420 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:51.416+0000 7f10fd5f9700 1 --2- 172.21.15.42:0/2524039416 >> v2:172.21.15.42:3300/0 conn(0x7f10f8116530 0x7f10f8127ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:18:51.421 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:51.416+0000 7f10edffb700 1 -- 172.21.15.42:0/2524039416 >> v2:172.21.15.42:3300/0 conn(0x7f10f8116530 msgr2=0x7f10f8127ac0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2019-12-05T05:18:51.421 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:51.416+0000 7f10edffb700 1 --2- 172.21.15.42:0/2524039416 >> v2:172.21.15.42:3300/0 conn(0x7f10f8116530 0x7f10f8127ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:18:51.421 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:51.416+0000 7f10edffb700 1 --2- 172.21.15.42:0/2524039416 >> v2:172.21.15.40:3300/0 conn(0x7f10d8006fa0 0x7f10d80094b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:18:51.421 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:51.416+0000 7f10edffb700 1 -- 172.21.15.42:0/2524039416 --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10d8009bd0 con 0x7f10d8003090 2019-12-05T05:18:51.421 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:51.416+0000 7f10edffb700 1 -- 172.21.15.42:0/2524039416 --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10d800aea0 con 0x7f10d8006fa0 2019-12-05T05:18:51.421 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:51.416+0000 7f10fcdf8700 1 --2- 172.21.15.42:0/2524039416 >> v2:172.21.15.40:3300/0 conn(0x7f10d8006fa0 0x7f10d80094b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:18:51.423 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:51.416+0000 7f10edffb700 1 -- 172.21.15.42:0/2524039416 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 255062629 0 0) 0x7f10d8009bd0 con 0x7f10d8003090 2019-12-05T05:18:51.423 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:51.416+0000 7f10edffb700 1 -- 172.21.15.42:0/2524039416 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10d8009ee0 con 0x7f10d8003090 2019-12-05T05:18:51.423 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:51.416+0000 7f10fcdf8700 1 -- 172.21.15.42:0/2524039416 >> v1:172.21.15.42:6789/0 conn(0x7f10d8003090 legacy=0x7f10d8005560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:18:51.423 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:51.416+0000 7f10edffb700 1 -- 172.21.15.42:0/2524039416 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 4174425852 0 0) 0x7f10d8009ee0 con 0x7f10d8003090 2019-12-05T05:18:51.424 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:51.416+0000 7f10fcdf8700 1 -- 172.21.15.42:0/2524039416 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10d800b150 con 0x7f10d8006fa0 2019-12-05T05:18:51.424 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:51.416+0000 7f10fcdf8700 1 --2- 172.21.15.42:0/2524039416 >> v2:172.21.15.40:3300/0 conn(0x7f10d8006fa0 0x7f10d80094b0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f10f400ce10 tx=0x7f10f4001c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:51.424 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:51.416+0000 7f10fcdf8700 1 --2- 172.21.15.42:0/2524039416 >> v2:172.21.15.40:3300/0 conn(0x7f10d8006fa0 0x7f10d80094b0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f10f400ce10 tx=0x7f10f4001c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:51.424 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:51.416+0000 7f10fcdf8700 1 --2- 172.21.15.42:0/2524039416 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f10d8006fa0 0x7f10d80094b0 secure :-1 s=READY pgs=519 cs=0 l=1 rx=0x7f10f400ce10 tx=0x7f10f4001c30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:18:51.424 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:51.416+0000 7f10fcdf8700 1 --2- 172.21.15.42:0/2524039416 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f10d8006fa0 0x7f10d80094b0 secure :-1 s=THROTTLE_DONE pgs=519 cs=0 l=1 rx=0x7f10f400ce10 tx=0x7f10f4001c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:51.424 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:51.416+0000 7f10edffb700 1 -- 172.21.15.42:0/2524039416 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f10f400cbd0 con 0x7f10d8006fa0 2019-12-05T05:18:51.425 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:51.416+0000 7f10fcdf8700 1 --2- 172.21.15.42:0/2524039416 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f10d8006fa0 0x7f10d80094b0 secure :-1 s=THROTTLE_DONE pgs=519 cs=0 l=1 rx=0x7f10f400ce10 tx=0x7f10f4001c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:51.425 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:51.416+0000 7f10fcdf8700 1 --2- 172.21.15.42:0/2524039416 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f10d8006fa0 0x7f10d80094b0 secure :-1 s=THROTTLE_DONE pgs=519 cs=0 l=1 rx=0x7f10f400ce10 tx=0x7f10f4001c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:51.425 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:51.416+0000 7f10edffb700 1 -- 172.21.15.42:0/2524039416 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10f400c9e0 con 0x7f10d8006fa0 2019-12-05T05:18:51.425 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:51.416+0000 7f10ff05c700 1 -- 172.21.15.42:0/2524039416 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10f807c4c0 con 0x7f10d8006fa0 2019-12-05T05:18:51.425 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:51.416+0000 7f10edffb700 1 -- 172.21.15.42:0/2524039416 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f10e8005540 con 0x7f10d8006fa0 2019-12-05T05:18:51.425 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:51.416+0000 7f10ff05c700 1 -- 172.21.15.42:0/2524039416 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10f812ef10 con 0x7f10d8006fa0 2019-12-05T05:18:51.426 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:51.416+0000 7f10fcdf8700 1 --2- 172.21.15.42:0/2524039416 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f10d8006fa0 0x7f10d80094b0 secure :-1 s=THROTTLE_DONE pgs=519 cs=0 l=1 rx=0x7f10f400ce10 tx=0x7f10f4001c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:51.426 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:51.420+0000 7f10edffb700 1 -- 172.21.15.42:0/2524039416 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f10e80057e0 con 0x7f10d8006fa0 2019-12-05T05:18:51.426 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:51.420+0000 7f10fcdf8700 1 --2- 172.21.15.42:0/2524039416 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f10d8006fa0 0x7f10d80094b0 secure :-1 s=THROTTLE_DONE pgs=519 cs=0 l=1 rx=0x7f10f400ce10 tx=0x7f10f4001c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:51.427 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:51.420+0000 7f10edffb700 1 --2- 172.21.15.42:0/2524039416 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f10d80424b0 0x7f10d80449c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:18:51.427 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:51.420+0000 7f10edffb700 1 -- 172.21.15.42:0/2524039416 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6977+0+0 (secure 0 0 0) 0x7f10e804b710 con 0x7f10d8006fa0 2019-12-05T05:18:51.427 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:51.420+0000 7f10fd5f9700 1 --2- 172.21.15.42:0/2524039416 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f10d80424b0 0x7f10d80449c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:18:51.427 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:51.420+0000 7f10fd5f9700 1 --2- 172.21.15.42:0/2524039416 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f10d80424b0 0x7f10d80449c0 crc :-1 s=READY pgs=295 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:18:51.474 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:51.468+0000 7f10ff05c700 1 -- 172.21.15.42:0/2524039416 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10f805d110 con 0x7f10d8006fa0 2019-12-05T05:18:51.484 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:51.480+0000 7f10fcdf8700 1 --2- 172.21.15.42:0/2524039416 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f10d8006fa0 0x7f10d80094b0 secure :-1 s=THROTTLE_DONE pgs=519 cs=0 l=1 rx=0x7f10f400ce10 tx=0x7f10f4001c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:51.484 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:51.480+0000 7f10edffb700 1 -- 172.21.15.42:0/2524039416 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f10e80329d0 con 0x7f10d8006fa0 2019-12-05T05:18:51.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:51.644+0000 7f10ff05c700 1 -- 172.21.15.42:0/2524039416 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f10f805b8c0 con 0x7f10d8006fa0 2019-12-05T05:18:51.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:51.644+0000 7f10fcdf8700 1 --2- 172.21.15.42:0/2524039416 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f10d8006fa0 0x7f10d80094b0 secure :-1 s=THROTTLE_DONE pgs=519 cs=0 l=1 rx=0x7f10f400ce10 tx=0x7f10f4001c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:51.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:51.644+0000 7f10edffb700 1 -- 172.21.15.42:0/2524039416 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 85 v85) v1 ==== 94+0+1775 (secure 0 0 0) 0x7f10e8033580 con 0x7f10d8006fa0 2019-12-05T05:18:51.651 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 85 2019-12-05T05:18:51.651 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:18:51.651 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":85,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":85,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:18:50.950569+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5833},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5833":{"gid":5833,"name":"a","rank":0,"incarnation":84,"state":"up:reconnect","state_seq":23,"addr":"172.21.15.42:6835/1414371821","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1414371821},{"type":"v1","addr":"172.21.15.42:6835","nonce":1414371821}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:18:51.656 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:51.648+0000 7f10ff05c700 1 -- 172.21.15.42:0/2524039416 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f10d80424b0 msgr2=0x7f10d80449c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:18:51.656 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:51.648+0000 7f10ff05c700 1 --2- 172.21.15.42:0/2524039416 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f10d80424b0 0x7f10d80449c0 crc :-1 s=READY pgs=295 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:18:51.656 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:51.648+0000 7f10ff05c700 1 -- 172.21.15.42:0/2524039416 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f10d8006fa0 msgr2=0x7f10d80094b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:18:51.656 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:51.648+0000 7f10fddfa700 1 -- 172.21.15.42:0/2524039416 reap_dead start 2019-12-05T05:18:51.657 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:51.648+0000 7f10ff05c700 1 --2- 172.21.15.42:0/2524039416 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f10d8006fa0 0x7f10d80094b0 secure :-1 s=READY pgs=519 cs=0 l=1 rx=0x7f10f400ce10 tx=0x7f10f4001c30).stop 2019-12-05T05:18:51.657 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:51.648+0000 7f10ff05c700 1 -- 172.21.15.42:0/2524039416 shutdown_connections 2019-12-05T05:18:51.657 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:51.648+0000 7f10ff05c700 1 --2- 172.21.15.42:0/2524039416 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f10d8006fa0 0x7f10d80094b0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:18:51.657 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:51.648+0000 7f10ff05c700 1 -- 172.21.15.42:0/2524039416 shutdown_connections 2019-12-05T05:18:51.657 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:51.648+0000 7f10ff05c700 1 -- 172.21.15.42:0/2524039416 wait complete. 2019-12-05T05:18:51.657 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:51.648+0000 7f10ff05c700 1 -- 172.21.15.42:0/2524039416 >> 172.21.15.42:0/2524039416 conn(0x7f10f80a6f30 msgr2=0x7f10f809f380 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:18:51.668 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2019-12-05T05:18:53.669 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:18:53.774 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:53.764+0000 7ff21c2e6700 1 Processor -- start 2019-12-05T05:18:53.774 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:53.764+0000 7ff21c2e6700 1 -- start start 2019-12-05T05:18:53.775 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:53.764+0000 7ff21c2e6700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff2140a7730 0x7ff21408df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:18:53.775 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:53.764+0000 7ff21c2e6700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff21409fad0 0x7ff21407eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:18:53.775 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:53.764+0000 7ff21c2e6700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2140a0700 con 0x7ff21407e710 2019-12-05T05:18:53.775 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:53.764+0000 7ff21c2e6700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2140a0af0 con 0x7ff2140a7730 2019-12-05T05:18:53.775 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:53.764+0000 7ff21c2e6700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff21408e5a0 con 0x7ff21409fad0 2019-12-05T05:18:53.775 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:53.764+0000 7ff21b084700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff21409fad0 0x7ff21407eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:18:53.776 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:53.764+0000 7ff21b084700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff21409fad0 0x7ff21407eb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:59370/0 (socket says 172.21.15.42:59370) 2019-12-05T05:18:53.776 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:53.764+0000 7ff21a082700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7ff21407e710 0x7ff21409f2d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:59406/0 (socket says 172.21.15.42:59406) 2019-12-05T05:18:53.776 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:53.764+0000 7ff21b084700 1 -- 172.21.15.42:0/2977287504 learned_addr learned my addr 172.21.15.42:0/2977287504 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:18:53.776 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:53.764+0000 7ff21a883700 1 --2- 172.21.15.42:0/2977287504 >> v2:172.21.15.40:3300/0 conn(0x7ff2140a7730 0x7ff21408df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:18:53.776 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:53.768+0000 7ff21b084700 1 -- 172.21.15.42:0/2977287504 >> v2:172.21.15.40:3300/0 conn(0x7ff2140a7730 msgr2=0x7ff21408df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:18:53.776 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:53.768+0000 7ff21b084700 1 --2- 172.21.15.42:0/2977287504 >> v2:172.21.15.40:3300/0 conn(0x7ff2140a7730 0x7ff21408df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:18:53.776 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:53.768+0000 7ff21b084700 1 -- 172.21.15.42:0/2977287504 >> v1:172.21.15.42:6789/0 conn(0x7ff21407e710 legacy=0x7ff21409f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:18:53.777 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:53.768+0000 7ff21b084700 1 -- 172.21.15.42:0/2977287504 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff21407f090 con 0x7ff21409fad0 2019-12-05T05:18:53.777 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:53.768+0000 7ff21b084700 1 --2- 172.21.15.42:0/2977287504 >> v2:172.21.15.42:3300/0 conn(0x7ff21409fad0 0x7ff21407eb40 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7ff20c0032b0 tx=0x7ff20c0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:53.777 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:53.768+0000 7ff21b084700 1 --2- 172.21.15.42:0/2977287504 >> v2:172.21.15.42:3300/0 conn(0x7ff21409fad0 0x7ff21407eb40 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7ff20c0032b0 tx=0x7ff20c0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:53.777 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:53.768+0000 7ff21b084700 1 --2- 172.21.15.42:0/2977287504 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff21409fad0 0x7ff21407eb40 secure :-1 s=READY pgs=653 cs=0 l=1 rx=0x7ff20c0032b0 tx=0x7ff20c0100b0).ready entity=mon.1 client_cookie=d3a522a12e77cf1f server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:18:53.777 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:53.768+0000 7ff21b084700 1 --2- 172.21.15.42:0/2977287504 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff21409fad0 0x7ff21407eb40 secure :-1 s=THROTTLE_DONE pgs=653 cs=0 l=1 rx=0x7ff20c0032b0 tx=0x7ff20c0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:53.777 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:53.768+0000 7ff219881700 1 -- 172.21.15.42:0/2977287504 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff20c010d60 con 0x7ff21409fad0 2019-12-05T05:18:53.777 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:53.768+0000 7ff21b084700 1 --2- 172.21.15.42:0/2977287504 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff21409fad0 0x7ff21407eb40 secure :-1 s=THROTTLE_DONE pgs=653 cs=0 l=1 rx=0x7ff20c0032b0 tx=0x7ff20c0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:53.778 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:53.768+0000 7ff219881700 1 -- 172.21.15.42:0/2977287504 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff20c010980 con 0x7ff21409fad0 2019-12-05T05:18:53.778 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:53.768+0000 7ff21b084700 1 --2- 172.21.15.42:0/2977287504 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff21409fad0 0x7ff21407eb40 secure :-1 s=THROTTLE_DONE pgs=653 cs=0 l=1 rx=0x7ff20c0032b0 tx=0x7ff20c0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:53.778 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:53.768+0000 7ff219881700 1 -- 172.21.15.42:0/2977287504 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff20c0114a0 con 0x7ff21409fad0 2019-12-05T05:18:53.778 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:53.768+0000 7ff21c2e6700 1 -- 172.21.15.42:0/2977287504 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff21409fad0 msgr2=0x7ff21407eb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:18:53.778 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:53.768+0000 7ff21c2e6700 1 --2- 172.21.15.42:0/2977287504 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff21409fad0 0x7ff21407eb40 secure :-1 s=READY pgs=653 cs=0 l=1 rx=0x7ff20c0032b0 tx=0x7ff20c0100b0).stop 2019-12-05T05:18:53.779 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:53.768+0000 7ff21c2e6700 1 -- 172.21.15.42:0/2977287504 shutdown_connections 2019-12-05T05:18:53.779 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:53.768+0000 7ff21c2e6700 1 --2- 172.21.15.42:0/2977287504 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff21409fad0 0x7ff21407eb40 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:18:53.779 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:53.768+0000 7ff21c2e6700 1 --2- 172.21.15.42:0/2977287504 >> v2:172.21.15.40:3300/0 conn(0x7ff2140a7730 0x7ff21408df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:18:53.779 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:53.768+0000 7ff21c2e6700 1 -- 172.21.15.42:0/2977287504 shutdown_connections 2019-12-05T05:18:53.779 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:53.768+0000 7ff21c2e6700 1 -- 172.21.15.42:0/2977287504 wait complete. 2019-12-05T05:18:53.779 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:53.768+0000 7ff21c2e6700 1 -- 172.21.15.42:0/2977287504 >> 172.21.15.42:0/2977287504 conn(0x7ff2140aabd0 msgr2=0x7ff2140a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:18:53.783 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:53.772+0000 7ff21c2e6700 1 Processor -- start 2019-12-05T05:18:53.783 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:53.772+0000 7ff21c2e6700 1 -- start start 2019-12-05T05:18:53.783 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:53.772+0000 7ff21c2e6700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff214122820 0x7ff21406e290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:18:53.783 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:53.772+0000 7ff21c2e6700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff21406e7e0 0x7ff21406ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:18:53.784 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:53.772+0000 7ff21c2e6700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff214121170 con 0x7ff2140a51a0 2019-12-05T05:18:53.784 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:53.772+0000 7ff21c2e6700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff214123050 con 0x7ff214122820 2019-12-05T05:18:53.784 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:53.772+0000 7ff21c2e6700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff214123590 con 0x7ff21406e7e0 2019-12-05T05:18:53.784 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:53.772+0000 7ff21a082700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff214122820 0x7ff21406e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:18:53.784 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:53.772+0000 7ff21a082700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff214122820 0x7ff21406e290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:59374/0 (socket says 172.21.15.42:59374) 2019-12-05T05:18:53.784 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:53.772+0000 7ff21a883700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7ff2140a51a0 0x7ff214123830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:40618/0 (socket says 172.21.15.42:40618) 2019-12-05T05:18:53.784 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:53.772+0000 7ff21a082700 1 -- 172.21.15.42:0/3516463948 learned_addr learned my addr 172.21.15.42:0/3516463948 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:18:53.785 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:53.772+0000 7ff21b084700 1 --2- 172.21.15.42:0/3516463948 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff21406e7e0 0x7ff21406ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:18:53.785 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:53.772+0000 7ff21a082700 1 -- 172.21.15.42:0/3516463948 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff21406e7e0 msgr2=0x7ff21406ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:18:53.785 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:53.772+0000 7ff21a082700 1 --2- 172.21.15.42:0/3516463948 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff21406e7e0 0x7ff21406ec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:18:53.785 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:53.772+0000 7ff20b7fe700 1 -- 172.21.15.42:0/3516463948 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 84854834 0 0) 0x7ff214121170 con 0x7ff2140a51a0 2019-12-05T05:18:53.785 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:53.772+0000 7ff21a082700 1 -- 172.21.15.42:0/3516463948 >> v1:172.21.15.40:6789/0 conn(0x7ff2140a51a0 legacy=0x7ff214123830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:18:53.785 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:53.772+0000 7ff21a082700 1 -- 172.21.15.42:0/3516463948 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff214119c60 con 0x7ff214122820 2019-12-05T05:18:53.785 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:53.772+0000 7ff21a082700 1 --2- 172.21.15.42:0/3516463948 >> v2:172.21.15.42:3300/0 conn(0x7ff214122820 0x7ff21406e290 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7ff20400c3d0 tx=0x7ff2040109f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:53.786 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:53.772+0000 7ff21a082700 1 --2- 172.21.15.42:0/3516463948 >> v2:172.21.15.42:3300/0 conn(0x7ff214122820 0x7ff21406e290 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7ff20400c3d0 tx=0x7ff2040109f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:53.786 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:53.772+0000 7ff21a082700 1 --2- 172.21.15.42:0/3516463948 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff214122820 0x7ff21406e290 secure :-1 s=READY pgs=654 cs=0 l=1 rx=0x7ff20400c3d0 tx=0x7ff2040109f0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:18:53.786 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:53.772+0000 7ff21a082700 1 --2- 172.21.15.42:0/3516463948 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff214122820 0x7ff21406e290 secure :-1 s=THROTTLE_DONE pgs=654 cs=0 l=1 rx=0x7ff20400c3d0 tx=0x7ff2040109f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:53.788 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:53.772+0000 7ff20b7fe700 1 -- 172.21.15.42:0/3516463948 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff204011660 con 0x7ff214122820 2019-12-05T05:18:53.788 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:53.772+0000 7ff21a082700 1 --2- 172.21.15.42:0/3516463948 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff214122820 0x7ff21406e290 secure :-1 s=THROTTLE_DONE pgs=654 cs=0 l=1 rx=0x7ff20400c3d0 tx=0x7ff2040109f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:53.788 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:53.772+0000 7ff21a082700 1 --2- 172.21.15.42:0/3516463948 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff214122820 0x7ff21406e290 secure :-1 s=THROTTLE_DONE pgs=654 cs=0 l=1 rx=0x7ff20400c3d0 tx=0x7ff2040109f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:53.788 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:53.772+0000 7ff20b7fe700 1 -- 172.21.15.42:0/3516463948 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff204011280 con 0x7ff214122820 2019-12-05T05:18:53.789 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:53.772+0000 7ff21c2e6700 1 -- 172.21.15.42:0/3516463948 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff214124030 con 0x7ff214122820 2019-12-05T05:18:53.789 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:53.772+0000 7ff20b7fe700 1 -- 172.21.15.42:0/3516463948 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff204011c40 con 0x7ff214122820 2019-12-05T05:18:53.789 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:53.776+0000 7ff21c2e6700 1 -- 172.21.15.42:0/3516463948 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff214122c50 con 0x7ff214122820 2019-12-05T05:18:53.789 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:53.776+0000 7ff21a082700 1 --2- 172.21.15.42:0/3516463948 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff214122820 0x7ff21406e290 secure :-1 s=THROTTLE_DONE pgs=654 cs=0 l=1 rx=0x7ff20400c3d0 tx=0x7ff2040109f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:53.790 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:53.776+0000 7ff20b7fe700 1 -- 172.21.15.42:0/3516463948 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7ff204006460 con 0x7ff214122820 2019-12-05T05:18:53.790 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:53.776+0000 7ff21a082700 1 --2- 172.21.15.42:0/3516463948 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff214122820 0x7ff21406e290 secure :-1 s=THROTTLE_DONE pgs=654 cs=0 l=1 rx=0x7ff20400c3d0 tx=0x7ff2040109f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:53.790 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:53.776+0000 7ff20b7fe700 1 --2- 172.21.15.42:0/3516463948 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff1fc03ac10 0x7ff1fc03d120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:18:53.790 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:53.776+0000 7ff20b7fe700 1 -- 172.21.15.42:0/3516463948 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6977+0+0 (secure 0 0 0) 0x7ff20404ae00 con 0x7ff214122820 2019-12-05T05:18:53.791 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:53.776+0000 7ff21a883700 1 --2- 172.21.15.42:0/3516463948 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff1fc03ac10 0x7ff1fc03d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:18:53.791 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:53.776+0000 7ff21a883700 1 --2- 172.21.15.42:0/3516463948 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff1fc03ac10 0x7ff1fc03d120 crc :-1 s=READY pgs=296 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:18:53.792 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:53.780+0000 7ff21c2e6700 1 -- 172.21.15.42:0/3516463948 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff21405d100 con 0x7ff214122820 2019-12-05T05:18:53.792 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:53.784+0000 7ff21a082700 1 --2- 172.21.15.42:0/3516463948 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff214122820 0x7ff21406e290 secure :-1 s=THROTTLE_DONE pgs=654 cs=0 l=1 rx=0x7ff20400c3d0 tx=0x7ff2040109f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:53.793 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:53.784+0000 7ff20b7fe700 1 -- 172.21.15.42:0/3516463948 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7ff204031e80 con 0x7ff214122820 2019-12-05T05:18:53.956 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:53.952+0000 7ff21c2e6700 1 -- 172.21.15.42:0/3516463948 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff2140a6980 con 0x7ff214122820 2019-12-05T05:18:53.957 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:53.952+0000 7ff21a082700 1 --2- 172.21.15.42:0/3516463948 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff214122820 0x7ff21406e290 secure :-1 s=THROTTLE_DONE pgs=654 cs=0 l=1 rx=0x7ff20400c3d0 tx=0x7ff2040109f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:18:53.957 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:53.952+0000 7ff20b7fe700 1 -- 172.21.15.42:0/3516463948 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+1772 (secure 0 0 0) 0x7ff2040320e0 con 0x7ff214122820 2019-12-05T05:18:53.959 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:18:53.960 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":87,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:18:52.963069+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5833},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5833":{"gid":5833,"name":"a","rank":0,"incarnation":84,"state":"up:active","state_seq":25,"addr":"172.21.15.42:6835/1414371821","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1414371821},{"type":"v1","addr":"172.21.15.42:6835","nonce":1414371821}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:18:53.961 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 87 2019-12-05T05:18:53.962 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:53.956+0000 7ff21c2e6700 1 -- 172.21.15.42:0/3516463948 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff1fc03ac10 msgr2=0x7ff1fc03d120 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:18:53.962 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:53.956+0000 7ff21c2e6700 1 --2- 172.21.15.42:0/3516463948 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff1fc03ac10 0x7ff1fc03d120 crc :-1 s=READY pgs=296 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:18:53.962 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:53.956+0000 7ff21c2e6700 1 -- 172.21.15.42:0/3516463948 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff214122820 msgr2=0x7ff21406e290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:18:53.962 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:53.956+0000 7ff21c2e6700 1 --2- 172.21.15.42:0/3516463948 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff214122820 0x7ff21406e290 secure :-1 s=READY pgs=654 cs=0 l=1 rx=0x7ff20400c3d0 tx=0x7ff2040109f0).stop 2019-12-05T05:18:53.963 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:53.956+0000 7ff21c2e6700 1 -- 172.21.15.42:0/3516463948 shutdown_connections 2019-12-05T05:18:53.963 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:53.956+0000 7ff21c2e6700 1 --2- 172.21.15.42:0/3516463948 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff1fc03ac10 0x7ff1fc03d120 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:18:53.963 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:53.956+0000 7ff21c2e6700 1 --2- 172.21.15.42:0/3516463948 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff21406e7e0 0x7ff21406ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:18:53.963 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:53.956+0000 7ff21c2e6700 1 --2- 172.21.15.42:0/3516463948 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff214122820 0x7ff21406e290 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:18:53.964 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:53.956+0000 7ff21c2e6700 1 -- 172.21.15.42:0/3516463948 shutdown_connections 2019-12-05T05:18:53.964 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:53.956+0000 7ff21c2e6700 1 -- 172.21.15.42:0/3516463948 wait complete. 2019-12-05T05:18:53.964 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:18:53.956+0000 7ff21c2e6700 1 -- 172.21.15.42:0/3516463948 >> 172.21.15.42:0/3516463948 conn(0x7ff2140aabd0 msgr2=0x7ff2140596f0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:18:53.972 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a has gained rank=0, replacing gid=5844 2019-12-05T05:18:53.973 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 6 secs before reviving mds.b 2019-12-05T05:18:59.973 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.b 2019-12-05T05:18:59.974 INFO:tasks.ceph.mds.b:Restarting daemon 2019-12-05T05:18:59.974 INFO:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2019-12-05T05:18:59.978 INFO:tasks.ceph.mds.b:Started 2019-12-05T05:18:59.978 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:19:00.038 INFO:tasks.ceph.mds.b.smithi040.stdout:starting mds.b at 2019-12-05T05:19:00.086 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:00.076+0000 7fcf26f27700 1 Processor -- start 2019-12-05T05:19:00.086 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:00.076+0000 7fcf26f27700 1 -- start start 2019-12-05T05:19:00.086 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:00.076+0000 7fcf26f27700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fcf2008df80 0x7fcf2009f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:19:00.086 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:00.076+0000 7fcf26f27700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf2009fc60 con 0x7fcf200a7740 2019-12-05T05:19:00.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:00.076+0000 7fcf26f27700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf200a0710 con 0x7fcf2009f830 2019-12-05T05:19:00.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:00.076+0000 7fcf26f27700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf200a0b00 con 0x7fcf2008df80 2019-12-05T05:19:00.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:00.076+0000 7fcf24cc3700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fcf2008df80 0x7fcf2009f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:19:00.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:00.076+0000 7fcf24cc3700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fcf2008df80 0x7fcf2009f2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:59384/0 (socket says 172.21.15.42:59384) 2019-12-05T05:19:00.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:00.076+0000 7fcf25cc5700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fcf2009f830 0x7fcf2014efe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:59424/0 (socket says 172.21.15.42:59424) 2019-12-05T05:19:00.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:00.076+0000 7fcf24cc3700 1 -- 172.21.15.42:0/754350288 learned_addr learned my addr 172.21.15.42:0/754350288 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:19:00.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:00.076+0000 7fcf254c4700 1 --1- 172.21.15.42:0/754350288 >> v1:172.21.15.40:6789/0 conn(0x7fcf200a7740 0x7fcf2007e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:40628/0 (socket says 172.21.15.42:40628) 2019-12-05T05:19:00.088 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:00.080+0000 7fcf24cc3700 1 -- 172.21.15.42:0/754350288 >> v1:172.21.15.42:6789/0 conn(0x7fcf2009f830 legacy=0x7fcf2014efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:19:00.088 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:00.080+0000 7fcf1ffff700 1 -- 172.21.15.42:0/754350288 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 96896093 0 0) 0x7fcf2009fc60 con 0x7fcf200a7740 2019-12-05T05:19:00.088 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:00.080+0000 7fcf24cc3700 1 -- 172.21.15.42:0/754350288 >> v1:172.21.15.40:6789/0 conn(0x7fcf200a7740 legacy=0x7fcf2007e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:19:00.096 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:00.080+0000 7fcf24cc3700 1 -- 172.21.15.42:0/754350288 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf2007ef20 con 0x7fcf2008df80 2019-12-05T05:19:00.096 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:00.080+0000 7fcf24cc3700 1 --2- 172.21.15.42:0/754350288 >> v2:172.21.15.42:3300/0 conn(0x7fcf2008df80 0x7fcf2009f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fcf14008790 tx=0x7fcf10001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:19:00.096 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:00.080+0000 7fcf24cc3700 1 --2- 172.21.15.42:0/754350288 >> v2:172.21.15.42:3300/0 conn(0x7fcf2008df80 0x7fcf2009f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fcf14008790 tx=0x7fcf10001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:19:00.096 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:00.080+0000 7fcf24cc3700 1 --2- 172.21.15.42:0/754350288 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcf2008df80 0x7fcf2009f2e0 secure :-1 s=READY pgs=657 cs=0 l=1 rx=0x7fcf14008790 tx=0x7fcf10001150).ready entity=mon.1 client_cookie=45830ace2669d78 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:19:00.096 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:00.080+0000 7fcf24cc3700 1 --2- 172.21.15.42:0/754350288 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcf2008df80 0x7fcf2009f2e0 secure :-1 s=THROTTLE_DONE pgs=657 cs=0 l=1 rx=0x7fcf14008790 tx=0x7fcf10001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:19:00.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:00.080+0000 7fcf24cc3700 1 --2- 172.21.15.42:0/754350288 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcf2008df80 0x7fcf2009f2e0 secure :-1 s=THROTTLE_DONE pgs=657 cs=0 l=1 rx=0x7fcf14008790 tx=0x7fcf10001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:19:00.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:00.080+0000 7fcf1ffff700 1 -- 172.21.15.42:0/754350288 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcf14012060 con 0x7fcf2008df80 2019-12-05T05:19:00.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:00.080+0000 7fcf24cc3700 1 --2- 172.21.15.42:0/754350288 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcf2008df80 0x7fcf2009f2e0 secure :-1 s=THROTTLE_DONE pgs=657 cs=0 l=1 rx=0x7fcf14008790 tx=0x7fcf10001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:19:00.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:00.080+0000 7fcf1ffff700 1 -- 172.21.15.42:0/754350288 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf140110c0 con 0x7fcf2008df80 2019-12-05T05:19:00.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:00.080+0000 7fcf1ffff700 1 -- 172.21.15.42:0/754350288 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcf140116a0 con 0x7fcf2008df80 2019-12-05T05:19:00.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:00.080+0000 7fcf26f27700 1 -- 172.21.15.42:0/754350288 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcf2008df80 msgr2=0x7fcf2009f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:19:00.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:00.080+0000 7fcf26f27700 1 --2- 172.21.15.42:0/754350288 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcf2008df80 0x7fcf2009f2e0 secure :-1 s=READY pgs=657 cs=0 l=1 rx=0x7fcf14008790 tx=0x7fcf10001150).stop 2019-12-05T05:19:00.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:00.080+0000 7fcf26f27700 1 -- 172.21.15.42:0/754350288 shutdown_connections 2019-12-05T05:19:00.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:00.080+0000 7fcf26f27700 1 --2- 172.21.15.42:0/754350288 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcf2008df80 0x7fcf2009f2e0 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:19:00.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:00.080+0000 7fcf26f27700 1 -- 172.21.15.42:0/754350288 shutdown_connections 2019-12-05T05:19:00.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:00.080+0000 7fcf26f27700 1 -- 172.21.15.42:0/754350288 wait complete. 2019-12-05T05:19:00.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:00.080+0000 7fcf26f27700 1 -- 172.21.15.42:0/754350288 >> 172.21.15.42:0/754350288 conn(0x7fcf200aabe0 msgr2=0x7fcf200a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:19:00.113 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:00.080+0000 7fcf26f27700 1 Processor -- start 2019-12-05T05:19:00.113 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:00.080+0000 7fcf26f27700 1 -- start start 2019-12-05T05:19:00.113 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:00.084+0000 7fcf26f27700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fcf2007d0e0 0x7fcf200a68a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:19:00.114 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:00.084+0000 7fcf26f27700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcf200a2d10 0x7fcf20122120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:19:00.114 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:00.084+0000 7fcf26f27700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf201191a0 con 0x7fcf200a5850 2019-12-05T05:19:00.114 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:00.084+0000 7fcf26f27700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf200a2550 con 0x7fcf2007d0e0 2019-12-05T05:19:00.114 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:00.084+0000 7fcf26f27700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf200a27d0 con 0x7fcf200a2d10 2019-12-05T05:19:00.114 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:00.084+0000 7fcf254c4700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fcf2007d0e0 0x7fcf200a68a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:19:00.114 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:00.084+0000 7fcf254c4700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fcf2007d0e0 0x7fcf200a68a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.42:47204/0 (socket says 172.21.15.42:47204) 2019-12-05T05:19:00.114 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:00.084+0000 7fcf254c4700 1 -- 172.21.15.42:0/1838466953 learned_addr learned my addr 172.21.15.42:0/1838466953 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:19:00.115 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:00.084+0000 7fcf24cc3700 1 --2- 172.21.15.42:0/1838466953 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcf200a2d10 0x7fcf20122120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:19:00.115 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:00.084+0000 7fcf254c4700 1 -- 172.21.15.42:0/1838466953 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcf200a2d10 msgr2=0x7fcf20122120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:19:00.115 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:00.084+0000 7fcf254c4700 1 --2- 172.21.15.42:0/1838466953 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcf200a2d10 0x7fcf20122120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:19:00.115 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:00.084+0000 7fcf254c4700 1 -- 172.21.15.42:0/1838466953 >> v1:172.21.15.40:6789/0 conn(0x7fcf200a5850 legacy=0x7fcf201256e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:19:00.115 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:00.084+0000 7fcf254c4700 1 -- 172.21.15.42:0/1838466953 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf20122670 con 0x7fcf2007d0e0 2019-12-05T05:19:00.115 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:00.084+0000 7fcf254c4700 1 --2- 172.21.15.42:0/1838466953 >> v2:172.21.15.40:3300/0 conn(0x7fcf2007d0e0 0x7fcf200a68a0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fcf200a2990 tx=0x7fcf18010cc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:19:00.115 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:00.084+0000 7fcf254c4700 1 --2- 172.21.15.42:0/1838466953 >> v2:172.21.15.40:3300/0 conn(0x7fcf2007d0e0 0x7fcf200a68a0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fcf200a2990 tx=0x7fcf18010cc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:19:00.116 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:00.084+0000 7fcf254c4700 1 --2- 172.21.15.42:0/1838466953 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcf2007d0e0 0x7fcf200a68a0 secure :-1 s=READY pgs=524 cs=0 l=1 rx=0x7fcf200a2990 tx=0x7fcf18010cc0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:19:00.116 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:00.084+0000 7fcf254c4700 1 --2- 172.21.15.42:0/1838466953 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcf2007d0e0 0x7fcf200a68a0 secure :-1 s=THROTTLE_DONE pgs=524 cs=0 l=1 rx=0x7fcf200a2990 tx=0x7fcf18010cc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:19:00.116 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:00.084+0000 7fcf254c4700 1 --2- 172.21.15.42:0/1838466953 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcf2007d0e0 0x7fcf200a68a0 secure :-1 s=THROTTLE_DONE pgs=524 cs=0 l=1 rx=0x7fcf200a2990 tx=0x7fcf18010cc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:19:00.116 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:00.084+0000 7fcf254c4700 1 --2- 172.21.15.42:0/1838466953 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcf2007d0e0 0x7fcf200a68a0 secure :-1 s=THROTTLE_DONE pgs=524 cs=0 l=1 rx=0x7fcf200a2990 tx=0x7fcf18010cc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:19:00.116 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:00.084+0000 7fcf1dffb700 1 -- 172.21.15.42:0/1838466953 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcf1800cb50 con 0x7fcf2007d0e0 2019-12-05T05:19:00.116 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:00.084+0000 7fcf1dffb700 1 -- 172.21.15.42:0/1838466953 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf1800c770 con 0x7fcf2007d0e0 2019-12-05T05:19:00.117 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:00.084+0000 7fcf26f27700 1 -- 172.21.15.42:0/1838466953 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf200a1510 con 0x7fcf2007d0e0 2019-12-05T05:19:00.117 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:00.084+0000 7fcf1dffb700 1 -- 172.21.15.42:0/1838466953 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcf180048e0 con 0x7fcf2007d0e0 2019-12-05T05:19:00.117 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:00.084+0000 7fcf26f27700 1 -- 172.21.15.42:0/1838466953 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf200a5c80 con 0x7fcf2007d0e0 2019-12-05T05:19:00.117 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:00.084+0000 7fcf254c4700 1 --2- 172.21.15.42:0/1838466953 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcf2007d0e0 0x7fcf200a68a0 secure :-1 s=THROTTLE_DONE pgs=524 cs=0 l=1 rx=0x7fcf200a2990 tx=0x7fcf18010cc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:19:00.117 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:00.084+0000 7fcf1dffb700 1 -- 172.21.15.42:0/1838466953 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fcf18004ae0 con 0x7fcf2007d0e0 2019-12-05T05:19:00.117 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:00.084+0000 7fcf254c4700 1 --2- 172.21.15.42:0/1838466953 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcf2007d0e0 0x7fcf200a68a0 secure :-1 s=THROTTLE_DONE pgs=524 cs=0 l=1 rx=0x7fcf200a2990 tx=0x7fcf18010cc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:19:00.117 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:00.088+0000 7fcf1dffb700 1 --2- 172.21.15.42:0/1838466953 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fcf0003ad40 0x7fcf0003d250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:19:00.117 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:00.088+0000 7fcf1dffb700 1 -- 172.21.15.42:0/1838466953 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6977+0+0 (secure 0 0 0) 0x7fcf1804b210 con 0x7fcf2007d0e0 2019-12-05T05:19:00.118 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:00.088+0000 7fcf24cc3700 1 --2- 172.21.15.42:0/1838466953 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fcf0003ad40 0x7fcf0003d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:19:00.118 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:00.088+0000 7fcf24cc3700 1 --2- 172.21.15.42:0/1838466953 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fcf0003ad40 0x7fcf0003d250 crc :-1 s=READY pgs=297 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:19:00.118 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:00.092+0000 7fcf26f27700 1 -- 172.21.15.42:0/1838466953 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf2005d110 con 0x7fcf2007d0e0 2019-12-05T05:19:00.118 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:00.096+0000 7fcf254c4700 1 --2- 172.21.15.42:0/1838466953 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcf2007d0e0 0x7fcf200a68a0 secure :-1 s=THROTTLE_DONE pgs=524 cs=0 l=1 rx=0x7fcf200a2990 tx=0x7fcf18010cc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:19:00.118 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:00.096+0000 7fcf1dffb700 1 -- 172.21.15.42:0/1838466953 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fcf18032290 con 0x7fcf2007d0e0 2019-12-05T05:19:00.269 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:00.264+0000 7fcf26f27700 1 -- 172.21.15.42:0/1838466953 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcf2005b8c0 con 0x7fcf2007d0e0 2019-12-05T05:19:00.270 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:00.264+0000 7fcf254c4700 1 --2- 172.21.15.42:0/1838466953 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcf2007d0e0 0x7fcf200a68a0 secure :-1 s=THROTTLE_DONE pgs=524 cs=0 l=1 rx=0x7fcf200a2990 tx=0x7fcf18010cc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:19:00.271 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:00.264+0000 7fcf1dffb700 1 -- 172.21.15.42:0/1838466953 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+1772 (secure 0 0 0) 0x7fcf180324f0 con 0x7fcf2007d0e0 2019-12-05T05:19:00.272 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 87 2019-12-05T05:19:00.273 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:19:00.273 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":87,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:18:52.963069+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5833},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5833":{"gid":5833,"name":"a","rank":0,"incarnation":84,"state":"up:active","state_seq":25,"addr":"172.21.15.42:6835/1414371821","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1414371821},{"type":"v1","addr":"172.21.15.42:6835","nonce":1414371821}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:19:00.275 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:00.268+0000 7fcf26f27700 1 -- 172.21.15.42:0/1838466953 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fcf0003ad40 msgr2=0x7fcf0003d250 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:19:00.275 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:00.268+0000 7fcf26f27700 1 --2- 172.21.15.42:0/1838466953 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fcf0003ad40 0x7fcf0003d250 crc :-1 s=READY pgs=297 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:19:00.275 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:00.268+0000 7fcf26f27700 1 -- 172.21.15.42:0/1838466953 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcf2007d0e0 msgr2=0x7fcf200a68a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:19:00.276 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:00.268+0000 7fcf26f27700 1 --2- 172.21.15.42:0/1838466953 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcf2007d0e0 0x7fcf200a68a0 secure :-1 s=READY pgs=524 cs=0 l=1 rx=0x7fcf200a2990 tx=0x7fcf18010cc0).stop 2019-12-05T05:19:00.276 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:00.268+0000 7fcf26f27700 1 -- 172.21.15.42:0/1838466953 shutdown_connections 2019-12-05T05:19:00.276 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:00.268+0000 7fcf26f27700 1 --2- 172.21.15.42:0/1838466953 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fcf0003ad40 0x7fcf0003d250 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:19:00.276 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:00.268+0000 7fcf26f27700 1 --2- 172.21.15.42:0/1838466953 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcf2007d0e0 0x7fcf200a68a0 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:19:00.276 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:00.268+0000 7fcf26f27700 1 --2- 172.21.15.42:0/1838466953 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcf200a2d10 0x7fcf20122120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:19:00.276 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:00.268+0000 7fcf26f27700 1 -- 172.21.15.42:0/1838466953 shutdown_connections 2019-12-05T05:19:00.276 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:00.268+0000 7fcf26f27700 1 -- 172.21.15.42:0/1838466953 wait complete. 2019-12-05T05:19:00.277 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:00.268+0000 7fcf26f27700 1 -- 172.21.15.42:0/1838466953 >> 172.21.15.42:0/1838466953 conn(0x7fcf200a83c0 msgr2=0x7fcf201162b0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:19:00.285 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2019-12-05T05:19:02.286 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:19:02.395 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:02.384+0000 7f431dc79700 1 Processor -- start 2019-12-05T05:19:02.396 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:02.384+0000 7f431dc79700 1 -- start start 2019-12-05T05:19:02.396 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:02.384+0000 7f431dc79700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f43180a7740 0x7f431808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:19:02.396 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:02.384+0000 7f431dc79700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43180a0710 con 0x7f431809fae0 2019-12-05T05:19:02.396 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:02.384+0000 7f431dc79700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43180a9ba0 con 0x7f431807e720 2019-12-05T05:19:02.396 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:02.384+0000 7f431dc79700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43180a0b00 con 0x7f43180a7740 2019-12-05T05:19:02.396 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:02.384+0000 7f43177fe700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f431807e720 0x7f431809f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:59434/0 (socket says 172.21.15.42:59434) 2019-12-05T05:19:02.397 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:02.384+0000 7f43177fe700 1 -- 172.21.15.42:0/1694394945 learned_addr learned my addr 172.21.15.42:0/1694394945 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:19:02.397 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:02.384+0000 7f4317fff700 1 --2- 172.21.15.42:0/1694394945 >> v2:172.21.15.40:3300/0 conn(0x7f43180a7740 0x7f431808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:19:02.397 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:02.384+0000 7f4316ffd700 1 -- 172.21.15.42:0/1694394945 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1982223839 0 0) 0x7f43180a9ba0 con 0x7f431807e720 2019-12-05T05:19:02.397 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:02.384+0000 7f4316ffd700 1 -- 172.21.15.42:0/1694394945 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43040030b0 con 0x7f431807e720 2019-12-05T05:19:02.397 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:02.384+0000 7f4316ffd700 1 -- 172.21.15.42:0/1694394945 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2859357959 0 0) 0x7f43180a0710 con 0x7f431809fae0 2019-12-05T05:19:02.397 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:02.384+0000 7f4316ffd700 1 -- 172.21.15.42:0/1694394945 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43180a9ba0 con 0x7f431809fae0 2019-12-05T05:19:02.397 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:02.384+0000 7f4317fff700 1 -- 172.21.15.42:0/1694394945 >> v1:172.21.15.42:6789/0 conn(0x7f431807e720 legacy=0x7f431809f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:19:02.398 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:02.388+0000 7f4317fff700 1 -- 172.21.15.42:0/1694394945 >> v1:172.21.15.40:6789/0 conn(0x7f431809fae0 legacy=0x7f431814efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:19:02.398 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:02.388+0000 7f4317fff700 1 -- 172.21.15.42:0/1694394945 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f431807eb50 con 0x7f43180a7740 2019-12-05T05:19:02.398 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:02.388+0000 7f4317fff700 1 --2- 172.21.15.42:0/1694394945 >> v2:172.21.15.40:3300/0 conn(0x7f43180a7740 0x7f431808df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f4300008790 tx=0x7f430000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:19:02.398 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:02.388+0000 7f4317fff700 1 --2- 172.21.15.42:0/1694394945 >> v2:172.21.15.40:3300/0 conn(0x7f43180a7740 0x7f431808df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f4300008790 tx=0x7f430000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:19:02.398 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:02.388+0000 7f4317fff700 1 --2- 172.21.15.42:0/1694394945 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f43180a7740 0x7f431808df80 secure :-1 s=READY pgs=526 cs=0 l=1 rx=0x7f4300008790 tx=0x7f430000fbf0).ready entity=mon.0 client_cookie=1ec1de7698e1949a server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:19:02.398 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:02.388+0000 7f4317fff700 1 --2- 172.21.15.42:0/1694394945 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f43180a7740 0x7f431808df80 secure :-1 s=THROTTLE_DONE pgs=526 cs=0 l=1 rx=0x7f4300008790 tx=0x7f430000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:19:02.398 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:02.388+0000 7f4317fff700 1 --2- 172.21.15.42:0/1694394945 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f43180a7740 0x7f431808df80 secure :-1 s=THROTTLE_DONE pgs=526 cs=0 l=1 rx=0x7f4300008790 tx=0x7f430000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:19:02.399 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:02.388+0000 7f4316ffd700 1 -- 172.21.15.42:0/1694394945 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4300011270 con 0x7f43180a7740 2019-12-05T05:19:02.399 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:02.388+0000 7f4317fff700 1 --2- 172.21.15.42:0/1694394945 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f43180a7740 0x7f431808df80 secure :-1 s=THROTTLE_DONE pgs=526 cs=0 l=1 rx=0x7f4300008790 tx=0x7f430000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:19:02.399 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:02.388+0000 7f4316ffd700 1 -- 172.21.15.42:0/1694394945 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4300008bc0 con 0x7f43180a7740 2019-12-05T05:19:02.399 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:02.388+0000 7f4316ffd700 1 -- 172.21.15.42:0/1694394945 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4300011850 con 0x7f43180a7740 2019-12-05T05:19:02.399 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:02.388+0000 7f431dc79700 1 -- 172.21.15.42:0/1694394945 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f43180a7740 msgr2=0x7f431808df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:19:02.399 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:02.388+0000 7f431dc79700 1 --2- 172.21.15.42:0/1694394945 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f43180a7740 0x7f431808df80 secure :-1 s=READY pgs=526 cs=0 l=1 rx=0x7f4300008790 tx=0x7f430000fbf0).stop 2019-12-05T05:19:02.399 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:02.388+0000 7f431dc79700 1 -- 172.21.15.42:0/1694394945 shutdown_connections 2019-12-05T05:19:02.400 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:02.388+0000 7f431dc79700 1 --2- 172.21.15.42:0/1694394945 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f43180a7740 0x7f431808df80 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:19:02.400 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:02.388+0000 7f431dc79700 1 -- 172.21.15.42:0/1694394945 shutdown_connections 2019-12-05T05:19:02.400 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:02.388+0000 7f431dc79700 1 -- 172.21.15.42:0/1694394945 wait complete. 2019-12-05T05:19:02.400 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:02.388+0000 7f431dc79700 1 -- 172.21.15.42:0/1694394945 >> 172.21.15.42:0/1694394945 conn(0x7f43180aabe0 msgr2=0x7f43180a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:19:02.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:02.392+0000 7f431dc79700 1 Processor -- start 2019-12-05T05:19:02.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:02.392+0000 7f431dc79700 1 -- start start 2019-12-05T05:19:02.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:02.392+0000 7f431dc79700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f431807b390 0x7f4318126ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:19:02.404 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:02.392+0000 7f431dc79700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4318120e30 con 0x7f431807d3a0 2019-12-05T05:19:02.404 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:02.392+0000 7f431dc79700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f431812fc40 con 0x7f43181234d0 2019-12-05T05:19:02.404 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:02.392+0000 7f431dc79700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4318145410 con 0x7f431807b390 2019-12-05T05:19:02.404 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:02.392+0000 7f4317fff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f431807b390 0x7f4318126ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:19:02.404 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:02.392+0000 7f431ca17700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f43181234d0 0x7f4318123900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:59442/0 (socket says 172.21.15.42:59442) 2019-12-05T05:19:02.404 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:02.392+0000 7f431ca17700 1 -- 172.21.15.42:0/114081532 learned_addr learned my addr 172.21.15.42:0/114081532 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:19:02.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:02.392+0000 7f4314ff9700 1 -- 172.21.15.42:0/114081532 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3633451592 0 0) 0x7f4318120e30 con 0x7f431807d3a0 2019-12-05T05:19:02.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:02.392+0000 7f4317fff700 1 -- 172.21.15.42:0/114081532 >> v1:172.21.15.42:6789/0 conn(0x7f43181234d0 legacy=0x7f4318123900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:19:02.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:02.392+0000 7f4317fff700 1 -- 172.21.15.42:0/114081532 >> v1:172.21.15.40:6789/0 conn(0x7f431807d3a0 legacy=0x7f4318120630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:19:02.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:02.392+0000 7f4317fff700 1 -- 172.21.15.42:0/114081532 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4318130e80 con 0x7f431807b390 2019-12-05T05:19:02.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:02.392+0000 7f4317fff700 1 --2- 172.21.15.42:0/114081532 >> v2:172.21.15.42:3300/0 conn(0x7f431807b390 0x7f4318126ce0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f43100013e0 tx=0x7f43100014b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:19:02.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:02.392+0000 7f4317fff700 1 --2- 172.21.15.42:0/114081532 >> v2:172.21.15.42:3300/0 conn(0x7f431807b390 0x7f4318126ce0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f43100013e0 tx=0x7f43100014b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:19:02.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:02.392+0000 7f4317fff700 1 --2- 172.21.15.42:0/114081532 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f431807b390 0x7f4318126ce0 secure :-1 s=READY pgs=660 cs=0 l=1 rx=0x7f43100013e0 tx=0x7f43100014b0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:19:02.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:02.392+0000 7f4317fff700 1 --2- 172.21.15.42:0/114081532 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f431807b390 0x7f4318126ce0 secure :-1 s=THROTTLE_DONE pgs=660 cs=0 l=1 rx=0x7f43100013e0 tx=0x7f43100014b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:19:02.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:02.392+0000 7f4314ff9700 1 -- 172.21.15.42:0/114081532 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4300007340 con 0x7f431807b390 2019-12-05T05:19:02.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:02.392+0000 7f4317fff700 1 --2- 172.21.15.42:0/114081532 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f431807b390 0x7f4318126ce0 secure :-1 s=THROTTLE_DONE pgs=660 cs=0 l=1 rx=0x7f43100013e0 tx=0x7f43100014b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:19:02.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:02.392+0000 7f4317fff700 1 --2- 172.21.15.42:0/114081532 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f431807b390 0x7f4318126ce0 secure :-1 s=THROTTLE_DONE pgs=660 cs=0 l=1 rx=0x7f43100013e0 tx=0x7f43100014b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:19:02.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:02.392+0000 7f4314ff9700 1 -- 172.21.15.42:0/114081532 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4300011470 con 0x7f431807b390 2019-12-05T05:19:02.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:02.392+0000 7f4314ff9700 1 -- 172.21.15.42:0/114081532 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f430000f360 con 0x7f431807b390 2019-12-05T05:19:02.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:02.392+0000 7f431dc79700 1 -- 172.21.15.42:0/114081532 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f431812ec30 con 0x7f431807b390 2019-12-05T05:19:02.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:02.392+0000 7f431dc79700 1 -- 172.21.15.42:0/114081532 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f431812f1e0 con 0x7f431807b390 2019-12-05T05:19:02.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:02.396+0000 7f4317fff700 1 --2- 172.21.15.42:0/114081532 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f431807b390 0x7f4318126ce0 secure :-1 s=THROTTLE_DONE pgs=660 cs=0 l=1 rx=0x7f43100013e0 tx=0x7f43100014b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:19:02.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:02.396+0000 7f4314ff9700 1 -- 172.21.15.42:0/114081532 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f4300005540 con 0x7f431807b390 2019-12-05T05:19:02.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:02.396+0000 7f4317fff700 1 --2- 172.21.15.42:0/114081532 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f431807b390 0x7f4318126ce0 secure :-1 s=THROTTLE_DONE pgs=660 cs=0 l=1 rx=0x7f43100013e0 tx=0x7f43100014b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:19:02.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:02.396+0000 7f4314ff9700 1 --2- 172.21.15.42:0/114081532 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f42fc03ab90 0x7f42fc03d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:19:02.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:02.396+0000 7f4314ff9700 1 -- 172.21.15.42:0/114081532 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6977+0+0 (secure 0 0 0) 0x7f430004b7a0 con 0x7f431807b390 2019-12-05T05:19:02.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:02.396+0000 7f43177fe700 1 --2- 172.21.15.42:0/114081532 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f42fc03ab90 0x7f42fc03d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:19:02.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:02.396+0000 7f43177fe700 1 --2- 172.21.15.42:0/114081532 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f42fc03ab90 0x7f42fc03d0a0 crc :-1 s=READY pgs=298 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:19:02.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:02.400+0000 7f431dc79700 1 -- 172.21.15.42:0/114081532 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f431805d110 con 0x7f431807b390 2019-12-05T05:19:02.412 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:02.408+0000 7f4317fff700 1 --2- 172.21.15.42:0/114081532 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f431807b390 0x7f4318126ce0 secure :-1 s=THROTTLE_DONE pgs=660 cs=0 l=1 rx=0x7f43100013e0 tx=0x7f43100014b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:19:02.412 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:02.408+0000 7f4314ff9700 1 -- 172.21.15.42:0/114081532 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f430000f600 con 0x7f431807b390 2019-12-05T05:19:02.575 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:02.568+0000 7f431dc79700 1 -- 172.21.15.42:0/114081532 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f431805b8c0 con 0x7f431807b390 2019-12-05T05:19:02.576 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:02.572+0000 7f4317fff700 1 --2- 172.21.15.42:0/114081532 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f431807b390 0x7f4318126ce0 secure :-1 s=THROTTLE_DONE pgs=660 cs=0 l=1 rx=0x7f43100013e0 tx=0x7f43100014b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:19:02.576 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:02.572+0000 7f4314ff9700 1 -- 172.21.15.42:0/114081532 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+1772 (secure 0 0 0) 0x7f4300032a60 con 0x7f431807b390 2019-12-05T05:19:02.576 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 87 2019-12-05T05:19:02.577 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:19:02.577 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":87,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:18:52.963069+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5833},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5833":{"gid":5833,"name":"a","rank":0,"incarnation":84,"state":"up:active","state_seq":25,"addr":"172.21.15.42:6835/1414371821","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1414371821},{"type":"v1","addr":"172.21.15.42:6835","nonce":1414371821}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:19:02.580 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:02.576+0000 7f431dc79700 1 -- 172.21.15.42:0/114081532 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f42fc03ab90 msgr2=0x7f42fc03d0a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:19:02.580 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:02.576+0000 7f431dc79700 1 --2- 172.21.15.42:0/114081532 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f42fc03ab90 0x7f42fc03d0a0 crc :-1 s=READY pgs=298 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:19:02.580 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:02.576+0000 7f431dc79700 1 -- 172.21.15.42:0/114081532 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f431807b390 msgr2=0x7f4318126ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:19:02.581 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:02.576+0000 7f431dc79700 1 --2- 172.21.15.42:0/114081532 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f431807b390 0x7f4318126ce0 secure :-1 s=READY pgs=660 cs=0 l=1 rx=0x7f43100013e0 tx=0x7f43100014b0).stop 2019-12-05T05:19:02.581 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:02.576+0000 7f431dc79700 1 -- 172.21.15.42:0/114081532 shutdown_connections 2019-12-05T05:19:02.581 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:02.576+0000 7f431dc79700 1 --2- 172.21.15.42:0/114081532 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f42fc03ab90 0x7f42fc03d0a0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:19:02.581 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:02.576+0000 7f431dc79700 1 --2- 172.21.15.42:0/114081532 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f431807b390 0x7f4318126ce0 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:19:02.581 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:02.576+0000 7f431dc79700 1 -- 172.21.15.42:0/114081532 shutdown_connections 2019-12-05T05:19:02.581 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:02.576+0000 7f431dc79700 1 -- 172.21.15.42:0/114081532 wait complete. 2019-12-05T05:19:02.582 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:02.576+0000 7f431dc79700 1 -- 172.21.15.42:0/114081532 >> 172.21.15.42:0/114081532 conn(0x7f431807ee20 msgr2=0x7f43180a6a60 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:19:02.595 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2019-12-05T05:19:04.596 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:19:04.700 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:04.688+0000 7f25bff5d700 1 Processor -- start 2019-12-05T05:19:04.700 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:04.688+0000 7f25bff5d700 1 -- start start 2019-12-05T05:19:04.701 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:04.688+0000 7f25bff5d700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f25b80a7730 0x7f25b808df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:19:04.701 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:04.688+0000 7f25bff5d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f25b807e710 0x7f25b807eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:19:04.701 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:04.688+0000 7f25bff5d700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25b80a0700 con 0x7f25b809f2d0 2019-12-05T05:19:04.701 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:04.688+0000 7f25bff5d700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25b80a0af0 con 0x7f25b80a7730 2019-12-05T05:19:04.701 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:04.688+0000 7f25bff5d700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25b808e5a0 con 0x7f25b807e710 2019-12-05T05:19:04.702 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:04.688+0000 7f25bdcf9700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f25b807e710 0x7f25b807eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:19:04.702 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:04.688+0000 7f25bdcf9700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f25b807e710 0x7f25b807eb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:59412/0 (socket says 172.21.15.42:59412) 2019-12-05T05:19:04.702 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:04.688+0000 7f25becfb700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f25b809f2d0 0x7f25b809f700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:59452/0 (socket says 172.21.15.42:59452) 2019-12-05T05:19:04.702 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:04.688+0000 7f25bdcf9700 1 -- 172.21.15.42:0/3779274154 learned_addr learned my addr 172.21.15.42:0/3779274154 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:19:04.702 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:04.688+0000 7f25be4fa700 1 --2- 172.21.15.42:0/3779274154 >> v2:172.21.15.40:3300/0 conn(0x7f25b80a7730 0x7f25b808df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:19:04.702 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:04.692+0000 7f25be4fa700 1 -- 172.21.15.42:0/3779274154 >> v2:172.21.15.42:3300/0 conn(0x7f25b807e710 msgr2=0x7f25b807eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:19:04.702 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:04.692+0000 7f25be4fa700 1 --2- 172.21.15.42:0/3779274154 >> v2:172.21.15.42:3300/0 conn(0x7f25b807e710 0x7f25b807eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:19:04.703 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:04.692+0000 7f25be4fa700 1 -- 172.21.15.42:0/3779274154 >> v1:172.21.15.42:6789/0 conn(0x7f25b809f2d0 legacy=0x7f25b809f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:19:04.703 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:04.692+0000 7f25be4fa700 1 -- 172.21.15.42:0/3779274154 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25b807f090 con 0x7f25b80a7730 2019-12-05T05:19:04.703 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:04.692+0000 7f25bdcf9700 1 --2- 172.21.15.42:0/3779274154 >> v2:172.21.15.42:3300/0 conn(0x7f25b807e710 0x7f25b807eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:19:04.703 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:04.692+0000 7f25be4fa700 1 --2- 172.21.15.42:0/3779274154 >> v2:172.21.15.40:3300/0 conn(0x7f25b80a7730 0x7f25b808df70 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f25b4002f30 tx=0x7f25b400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:19:04.703 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:04.692+0000 7f25be4fa700 1 --2- 172.21.15.42:0/3779274154 >> v2:172.21.15.40:3300/0 conn(0x7f25b80a7730 0x7f25b808df70 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f25b4002f30 tx=0x7f25b400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:19:04.703 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:04.692+0000 7f25be4fa700 1 --2- 172.21.15.42:0/3779274154 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f25b80a7730 0x7f25b808df70 secure :-1 s=READY pgs=528 cs=0 l=1 rx=0x7f25b4002f30 tx=0x7f25b400fc70).ready entity=mon.0 client_cookie=c4a5d7bab558259c server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:19:04.703 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:04.692+0000 7f25be4fa700 1 --2- 172.21.15.42:0/3779274154 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f25b80a7730 0x7f25b808df70 secure :-1 s=THROTTLE_DONE pgs=528 cs=0 l=1 rx=0x7f25b4002f30 tx=0x7f25b400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:19:04.704 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:04.692+0000 7f25bd4f8700 1 -- 172.21.15.42:0/3779274154 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f25b40112b0 con 0x7f25b80a7730 2019-12-05T05:19:04.704 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:04.692+0000 7f25be4fa700 1 --2- 172.21.15.42:0/3779274154 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f25b80a7730 0x7f25b808df70 secure :-1 s=THROTTLE_DONE pgs=528 cs=0 l=1 rx=0x7f25b4002f30 tx=0x7f25b400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:19:04.704 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:04.692+0000 7f25be4fa700 1 --2- 172.21.15.42:0/3779274154 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f25b80a7730 0x7f25b808df70 secure :-1 s=THROTTLE_DONE pgs=528 cs=0 l=1 rx=0x7f25b4002f30 tx=0x7f25b400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:19:04.704 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:04.692+0000 7f25bd4f8700 1 -- 172.21.15.42:0/3779274154 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25b4008ad0 con 0x7f25b80a7730 2019-12-05T05:19:04.704 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:04.692+0000 7f25bd4f8700 1 -- 172.21.15.42:0/3779274154 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f25b4011890 con 0x7f25b80a7730 2019-12-05T05:19:04.705 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:04.692+0000 7f25bff5d700 1 -- 172.21.15.42:0/3779274154 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f25b80a7730 msgr2=0x7f25b808df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:19:04.705 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:04.692+0000 7f25bff5d700 1 --2- 172.21.15.42:0/3779274154 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f25b80a7730 0x7f25b808df70 secure :-1 s=READY pgs=528 cs=0 l=1 rx=0x7f25b4002f30 tx=0x7f25b400fc70).stop 2019-12-05T05:19:04.705 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:04.692+0000 7f25bff5d700 1 -- 172.21.15.42:0/3779274154 shutdown_connections 2019-12-05T05:19:04.705 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:04.692+0000 7f25bff5d700 1 --2- 172.21.15.42:0/3779274154 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f25b80a7730 0x7f25b808df70 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:19:04.705 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:04.692+0000 7f25bff5d700 1 --2- 172.21.15.42:0/3779274154 >> v2:172.21.15.42:3300/0 conn(0x7f25b807e710 0x7f25b807eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:19:04.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:04.692+0000 7f25bff5d700 1 -- 172.21.15.42:0/3779274154 shutdown_connections 2019-12-05T05:19:04.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:04.692+0000 7f25bff5d700 1 -- 172.21.15.42:0/3779274154 wait complete. 2019-12-05T05:19:04.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:04.692+0000 7f25bff5d700 1 -- 172.21.15.42:0/3779274154 >> 172.21.15.42:0/3779274154 conn(0x7f25b80aabd0 msgr2=0x7f25b80a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:19:04.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:04.696+0000 7f25bff5d700 1 Processor -- start 2019-12-05T05:19:04.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:04.696+0000 7f25bff5d700 1 -- start start 2019-12-05T05:19:04.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:04.696+0000 7f25bff5d700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f25b81193e0 0x7f25b8131d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:19:04.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:04.696+0000 7f25bff5d700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f25b8132270 0x7f25b8143730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:19:04.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:04.696+0000 7f25bff5d700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25b80a3b80 con 0x7f25b8143c80 2019-12-05T05:19:04.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:04.696+0000 7f25bff5d700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25b8119810 con 0x7f25b81193e0 2019-12-05T05:19:04.711 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:04.696+0000 7f25bff5d700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25b81317e0 con 0x7f25b8132270 2019-12-05T05:19:04.711 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:04.700+0000 7f25be4fa700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f25b81193e0 0x7f25b8131d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:19:04.711 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:04.700+0000 7f25bdcf9700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f25b8132270 0x7f25b8143730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:19:04.711 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:04.700+0000 7f25becfb700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f25b8143c80 0x7f25b808eb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:40666/0 (socket says 172.21.15.42:40666) 2019-12-05T05:19:04.711 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:04.700+0000 7f25becfb700 1 -- 172.21.15.42:0/984158301 learned_addr learned my addr 172.21.15.42:0/984158301 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:19:04.711 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:04.700+0000 7f25be4fa700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f25b81193e0 0x7f25b8131d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.42:47232/0 (socket says 172.21.15.42:47232) 2019-12-05T05:19:04.711 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:04.700+0000 7f25be4fa700 1 -- 172.21.15.42:0/984158301 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f25b8132270 msgr2=0x7f25b8143730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:19:04.712 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:04.700+0000 7f25be4fa700 1 --2- 172.21.15.42:0/984158301 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f25b8132270 0x7f25b8143730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:19:04.712 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:04.700+0000 7f25be4fa700 1 -- 172.21.15.42:0/984158301 >> v1:172.21.15.40:6789/0 conn(0x7f25b8143c80 legacy=0x7f25b808eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:19:04.712 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:04.700+0000 7f25be4fa700 1 -- 172.21.15.42:0/984158301 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25b812e0a0 con 0x7f25b81193e0 2019-12-05T05:19:04.712 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:04.700+0000 7f25bdcf9700 1 --2- 172.21.15.42:0/984158301 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f25b8132270 0x7f25b8143730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:19:04.712 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:04.700+0000 7f25be4fa700 1 --2- 172.21.15.42:0/984158301 >> v2:172.21.15.40:3300/0 conn(0x7f25b81193e0 0x7f25b8131d20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f25b81319a0 tx=0x7f25b4011e90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:19:04.713 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:04.700+0000 7f25be4fa700 1 --2- 172.21.15.42:0/984158301 >> v2:172.21.15.40:3300/0 conn(0x7f25b81193e0 0x7f25b8131d20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f25b81319a0 tx=0x7f25b4011e90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:19:04.713 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:04.700+0000 7f25be4fa700 1 --2- 172.21.15.42:0/984158301 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f25b81193e0 0x7f25b8131d20 secure :-1 s=READY pgs=530 cs=0 l=1 rx=0x7f25b81319a0 tx=0x7f25b4011e90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:19:04.713 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:04.700+0000 7f25be4fa700 1 --2- 172.21.15.42:0/984158301 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f25b81193e0 0x7f25b8131d20 secure :-1 s=THROTTLE_DONE pgs=530 cs=0 l=1 rx=0x7f25b81319a0 tx=0x7f25b4011e90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:19:04.713 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:04.700+0000 7f25aeffd700 1 -- 172.21.15.42:0/984158301 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f25b4007340 con 0x7f25b81193e0 2019-12-05T05:19:04.713 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:04.700+0000 7f25bff5d700 1 -- 172.21.15.42:0/984158301 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25b808f340 con 0x7f25b81193e0 2019-12-05T05:19:04.713 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:04.700+0000 7f25be4fa700 1 --2- 172.21.15.42:0/984158301 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f25b81193e0 0x7f25b8131d20 secure :-1 s=THROTTLE_DONE pgs=530 cs=0 l=1 rx=0x7f25b81319a0 tx=0x7f25b4011e90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:19:04.713 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:04.700+0000 7f25be4fa700 1 --2- 172.21.15.42:0/984158301 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f25b81193e0 0x7f25b8131d20 secure :-1 s=THROTTLE_DONE pgs=530 cs=0 l=1 rx=0x7f25b81319a0 tx=0x7f25b4011e90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:19:04.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:04.700+0000 7f25aeffd700 1 -- 172.21.15.42:0/984158301 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25b40114b0 con 0x7f25b81193e0 2019-12-05T05:19:04.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:04.700+0000 7f25aeffd700 1 -- 172.21.15.42:0/984158301 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f25b4005540 con 0x7f25b81193e0 2019-12-05T05:19:04.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:04.700+0000 7f25bff5d700 1 -- 172.21.15.42:0/984158301 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25b808f870 con 0x7f25b81193e0 2019-12-05T05:19:04.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:04.700+0000 7f25be4fa700 1 --2- 172.21.15.42:0/984158301 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f25b81193e0 0x7f25b8131d20 secure :-1 s=THROTTLE_DONE pgs=530 cs=0 l=1 rx=0x7f25b81319a0 tx=0x7f25b4011e90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:19:04.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:04.700+0000 7f25aeffd700 1 -- 172.21.15.42:0/984158301 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f25b40057e0 con 0x7f25b81193e0 2019-12-05T05:19:04.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:04.700+0000 7f25be4fa700 1 --2- 172.21.15.42:0/984158301 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f25b81193e0 0x7f25b8131d20 secure :-1 s=THROTTLE_DONE pgs=530 cs=0 l=1 rx=0x7f25b81319a0 tx=0x7f25b4011e90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:19:04.715 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:04.700+0000 7f25aeffd700 1 --2- 172.21.15.42:0/984158301 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f25a003ad50 0x7f25a003d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:19:04.715 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:04.700+0000 7f25aeffd700 1 -- 172.21.15.42:0/984158301 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6977+0+0 (secure 0 0 0) 0x7f25b404b610 con 0x7f25b81193e0 2019-12-05T05:19:04.715 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:04.700+0000 7f25bdcf9700 1 --2- 172.21.15.42:0/984158301 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f25a003ad50 0x7f25a003d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:19:04.715 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:04.704+0000 7f25bdcf9700 1 --2- 172.21.15.42:0/984158301 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f25a003ad50 0x7f25a003d260 crc :-1 s=READY pgs=299 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:19:04.715 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:04.704+0000 7f25bff5d700 1 -- 172.21.15.42:0/984158301 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25b805d100 con 0x7f25b81193e0 2019-12-05T05:19:04.718 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:04.712+0000 7f25be4fa700 1 --2- 172.21.15.42:0/984158301 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f25b81193e0 0x7f25b8131d20 secure :-1 s=THROTTLE_DONE pgs=530 cs=0 l=1 rx=0x7f25b81319a0 tx=0x7f25b4011e90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:19:04.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:04.712+0000 7f25aeffd700 1 -- 172.21.15.42:0/984158301 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f25b40328d0 con 0x7f25b81193e0 2019-12-05T05:19:04.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:04.884+0000 7f25bff5d700 1 -- 172.21.15.42:0/984158301 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f25b805b8b0 con 0x7f25b81193e0 2019-12-05T05:19:04.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:04.888+0000 7f25be4fa700 1 --2- 172.21.15.42:0/984158301 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f25b81193e0 0x7f25b8131d20 secure :-1 s=THROTTLE_DONE pgs=530 cs=0 l=1 rx=0x7f25b81319a0 tx=0x7f25b4011e90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:19:04.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:04.888+0000 7f25aeffd700 1 -- 172.21.15.42:0/984158301 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+1772 (secure 0 0 0) 0x7f25b4007f50 con 0x7f25b81193e0 2019-12-05T05:19:04.894 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 87 2019-12-05T05:19:04.895 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:19:04.895 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":87,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:18:52.963069+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5833},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5833":{"gid":5833,"name":"a","rank":0,"incarnation":84,"state":"up:active","state_seq":25,"addr":"172.21.15.42:6835/1414371821","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1414371821},{"type":"v1","addr":"172.21.15.42:6835","nonce":1414371821}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:19:04.897 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:04.892+0000 7f25bff5d700 1 -- 172.21.15.42:0/984158301 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f25a003ad50 msgr2=0x7f25a003d260 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:19:04.898 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:04.892+0000 7f25bff5d700 1 --2- 172.21.15.42:0/984158301 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f25a003ad50 0x7f25a003d260 crc :-1 s=READY pgs=299 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:19:04.898 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:04.892+0000 7f25bff5d700 1 -- 172.21.15.42:0/984158301 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f25b81193e0 msgr2=0x7f25b8131d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:19:04.898 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:04.892+0000 7f25bff5d700 1 --2- 172.21.15.42:0/984158301 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f25b81193e0 0x7f25b8131d20 secure :-1 s=READY pgs=530 cs=0 l=1 rx=0x7f25b81319a0 tx=0x7f25b4011e90).stop 2019-12-05T05:19:04.898 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:04.892+0000 7f25bff5d700 1 -- 172.21.15.42:0/984158301 shutdown_connections 2019-12-05T05:19:04.898 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:04.892+0000 7f25bff5d700 1 --2- 172.21.15.42:0/984158301 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f25a003ad50 0x7f25a003d260 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:19:04.898 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:04.892+0000 7f25bff5d700 1 --2- 172.21.15.42:0/984158301 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f25b81193e0 0x7f25b8131d20 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:19:04.899 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:04.892+0000 7f25bff5d700 1 --2- 172.21.15.42:0/984158301 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f25b8132270 0x7f25b8143730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:19:04.899 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:04.892+0000 7f25bff5d700 1 -- 172.21.15.42:0/984158301 shutdown_connections 2019-12-05T05:19:04.899 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:04.892+0000 7f25bff5d700 1 -- 172.21.15.42:0/984158301 wait complete. 2019-12-05T05:19:04.899 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:04.892+0000 7f25bff5d700 1 -- 172.21.15.42:0/984158301 >> 172.21.15.42:0/984158301 conn(0x7f25b80a97d0 msgr2=0x7f25b80a6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:19:04.908 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2019-12-05T05:19:06.050 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:19:06.064 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:19:06.909 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:19:07.018 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:07.008+0000 7fc9bc71f700 1 Processor -- start 2019-12-05T05:19:07.018 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:07.008+0000 7fc9bc71f700 1 -- start start 2019-12-05T05:19:07.019 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:07.008+0000 7fc9bc71f700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc9b40a7730 0x7fc9b408df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:19:07.019 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:07.008+0000 7fc9bc71f700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc9b409fad0 0x7fc9b407eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:19:07.019 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:07.008+0000 7fc9bc71f700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9b40a0700 con 0x7fc9b407e710 2019-12-05T05:19:07.019 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:07.008+0000 7fc9bc71f700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9b40a0af0 con 0x7fc9b409fad0 2019-12-05T05:19:07.019 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:07.008+0000 7fc9bc71f700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9b408e5a0 con 0x7fc9b40a7730 2019-12-05T05:19:07.019 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:07.008+0000 7fc9bacbc700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc9b40a7730 0x7fc9b408df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:19:07.020 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:07.008+0000 7fc9ba4bb700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fc9b407e710 0x7fc9b409f2d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:59476/0 (socket says 172.21.15.42:59476) 2019-12-05T05:19:07.020 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:07.008+0000 7fc9ba4bb700 1 -- 172.21.15.42:0/1945412611 learned_addr learned my addr 172.21.15.42:0/1945412611 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:19:07.020 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:07.008+0000 7fc9bb4bd700 1 --2- 172.21.15.42:0/1945412611 >> v2:172.21.15.40:3300/0 conn(0x7fc9b409fad0 0x7fc9b407eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:19:07.020 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:07.008+0000 7fc9b9cba700 1 -- 172.21.15.42:0/1945412611 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3889504742 0 0) 0x7fc9b40a0700 con 0x7fc9b407e710 2019-12-05T05:19:07.020 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:07.008+0000 7fc9b9cba700 1 -- 172.21.15.42:0/1945412611 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9a00030b0 con 0x7fc9b407e710 2019-12-05T05:19:07.021 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:07.008+0000 7fc9bacbc700 1 -- 172.21.15.42:0/1945412611 >> v2:172.21.15.40:3300/0 conn(0x7fc9b409fad0 msgr2=0x7fc9b407eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:19:07.021 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:07.008+0000 7fc9bacbc700 1 --2- 172.21.15.42:0/1945412611 >> v2:172.21.15.40:3300/0 conn(0x7fc9b409fad0 0x7fc9b407eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:19:07.021 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:07.008+0000 7fc9bacbc700 1 -- 172.21.15.42:0/1945412611 >> v1:172.21.15.42:6789/0 conn(0x7fc9b407e710 legacy=0x7fc9b409f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:19:07.021 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:07.008+0000 7fc9bacbc700 1 -- 172.21.15.42:0/1945412611 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9b407f090 con 0x7fc9b40a7730 2019-12-05T05:19:07.021 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:07.008+0000 7fc9bacbc700 1 --2- 172.21.15.42:0/1945412611 >> v2:172.21.15.42:3300/0 conn(0x7fc9b40a7730 0x7fc9b408df70 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fc9b0002f30 tx=0x7fc9b000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:19:07.021 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:07.012+0000 7fc9bacbc700 1 --2- 172.21.15.42:0/1945412611 >> v2:172.21.15.42:3300/0 conn(0x7fc9b40a7730 0x7fc9b408df70 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fc9b0002f30 tx=0x7fc9b000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:19:07.022 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:07.012+0000 7fc9bacbc700 1 --2- 172.21.15.42:0/1945412611 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc9b40a7730 0x7fc9b408df70 secure :-1 s=READY pgs=664 cs=0 l=1 rx=0x7fc9b0002f30 tx=0x7fc9b000fc70).ready entity=mon.1 client_cookie=697cb7054fb62ae4 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:19:07.022 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:07.012+0000 7fc9bacbc700 1 --2- 172.21.15.42:0/1945412611 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc9b40a7730 0x7fc9b408df70 secure :-1 s=THROTTLE_DONE pgs=664 cs=0 l=1 rx=0x7fc9b0002f30 tx=0x7fc9b000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:19:07.022 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:07.012+0000 7fc9b9cba700 1 -- 172.21.15.42:0/1945412611 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc9b0008bb0 con 0x7fc9b40a7730 2019-12-05T05:19:07.022 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:07.012+0000 7fc9bacbc700 1 --2- 172.21.15.42:0/1945412611 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc9b40a7730 0x7fc9b408df70 secure :-1 s=THROTTLE_DONE pgs=664 cs=0 l=1 rx=0x7fc9b0002f30 tx=0x7fc9b000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:19:07.022 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:07.012+0000 7fc9bacbc700 1 --2- 172.21.15.42:0/1945412611 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc9b40a7730 0x7fc9b408df70 secure :-1 s=THROTTLE_DONE pgs=664 cs=0 l=1 rx=0x7fc9b0002f30 tx=0x7fc9b000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:19:07.022 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:07.012+0000 7fc9b9cba700 1 -- 172.21.15.42:0/1945412611 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9b00087d0 con 0x7fc9b40a7730 2019-12-05T05:19:07.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:07.012+0000 7fc9b9cba700 1 -- 172.21.15.42:0/1945412611 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc9b0011820 con 0x7fc9b40a7730 2019-12-05T05:19:07.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:07.012+0000 7fc9bc71f700 1 -- 172.21.15.42:0/1945412611 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc9b40a7730 msgr2=0x7fc9b408df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:19:07.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:07.012+0000 7fc9bc71f700 1 --2- 172.21.15.42:0/1945412611 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc9b40a7730 0x7fc9b408df70 secure :-1 s=READY pgs=664 cs=0 l=1 rx=0x7fc9b0002f30 tx=0x7fc9b000fc70).stop 2019-12-05T05:19:07.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:07.012+0000 7fc9bc71f700 1 -- 172.21.15.42:0/1945412611 shutdown_connections 2019-12-05T05:19:07.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:07.012+0000 7fc9bc71f700 1 --2- 172.21.15.42:0/1945412611 >> v2:172.21.15.40:3300/0 conn(0x7fc9b409fad0 0x7fc9b407eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:19:07.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:07.012+0000 7fc9bc71f700 1 --2- 172.21.15.42:0/1945412611 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc9b40a7730 0x7fc9b408df70 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:19:07.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:07.012+0000 7fc9bc71f700 1 -- 172.21.15.42:0/1945412611 shutdown_connections 2019-12-05T05:19:07.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:07.012+0000 7fc9bc71f700 1 -- 172.21.15.42:0/1945412611 wait complete. 2019-12-05T05:19:07.024 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:07.012+0000 7fc9bc71f700 1 -- 172.21.15.42:0/1945412611 >> 172.21.15.42:0/1945412611 conn(0x7fc9b40aabd0 msgr2=0x7fc9b40a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:19:07.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:07.016+0000 7fc9bc71f700 1 Processor -- start 2019-12-05T05:19:07.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:07.016+0000 7fc9bc71f700 1 -- start start 2019-12-05T05:19:07.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:07.016+0000 7fc9bc71f700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc9b41192b0 0x7fc9b4130b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:19:07.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:07.016+0000 7fc9bc71f700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc9b412cce0 0x7fc9b407db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:19:07.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:07.016+0000 7fc9bc71f700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9b4131070 con 0x7fc9b4142900 2019-12-05T05:19:07.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:07.016+0000 7fc9bc71f700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9b4130340 con 0x7fc9b412cce0 2019-12-05T05:19:07.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:07.016+0000 7fc9bc71f700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9b41305e0 con 0x7fc9b41192b0 2019-12-05T05:19:07.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:07.016+0000 7fc9ba4bb700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc9b412cce0 0x7fc9b407db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:19:07.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:07.016+0000 7fc9bacbc700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc9b41192b0 0x7fc9b4130b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:19:07.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:07.016+0000 7fc9bb4bd700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fc9b4142900 0x7fc9b4116ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:40692/0 (socket says 172.21.15.42:40692) 2019-12-05T05:19:07.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:07.016+0000 7fc9ba4bb700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc9b412cce0 0x7fc9b407db40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.42:47264/0 (socket says 172.21.15.42:47264) 2019-12-05T05:19:07.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:07.016+0000 7fc9bb4bd700 1 -- 172.21.15.42:0/4254874902 learned_addr learned my addr 172.21.15.42:0/4254874902 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:19:07.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:07.016+0000 7fc9ba4bb700 1 -- 172.21.15.42:0/4254874902 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc9b41192b0 msgr2=0x7fc9b4130b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:19:07.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:07.016+0000 7fc9ba4bb700 1 --2- 172.21.15.42:0/4254874902 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc9b41192b0 0x7fc9b4130b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:19:07.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:07.016+0000 7fc9ab7fe700 1 -- 172.21.15.42:0/4254874902 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1002470906 0 0) 0x7fc9b4131070 con 0x7fc9b4142900 2019-12-05T05:19:07.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:07.016+0000 7fc9ba4bb700 1 -- 172.21.15.42:0/4254874902 >> v1:172.21.15.40:6789/0 conn(0x7fc9b4142900 legacy=0x7fc9b4116ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:19:07.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:07.016+0000 7fc9ba4bb700 1 -- 172.21.15.42:0/4254874902 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9b407e090 con 0x7fc9b412cce0 2019-12-05T05:19:07.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:07.016+0000 7fc9bacbc700 1 --2- 172.21.15.42:0/4254874902 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc9b41192b0 0x7fc9b4130b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:19:07.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:07.016+0000 7fc9ba4bb700 1 --2- 172.21.15.42:0/4254874902 >> v2:172.21.15.40:3300/0 conn(0x7fc9b412cce0 0x7fc9b407db40 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fc9a40043a0 tx=0x7fc9a40108b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:19:07.039 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:07.020+0000 7fc9ba4bb700 1 --2- 172.21.15.42:0/4254874902 >> v2:172.21.15.40:3300/0 conn(0x7fc9b412cce0 0x7fc9b407db40 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fc9a40043a0 tx=0x7fc9a40108b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:19:07.039 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:07.020+0000 7fc9ba4bb700 1 --2- 172.21.15.42:0/4254874902 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc9b412cce0 0x7fc9b407db40 secure :-1 s=READY pgs=534 cs=0 l=1 rx=0x7fc9a40043a0 tx=0x7fc9a40108b0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:19:07.039 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:07.020+0000 7fc9ba4bb700 1 --2- 172.21.15.42:0/4254874902 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc9b412cce0 0x7fc9b407db40 secure :-1 s=THROTTLE_DONE pgs=534 cs=0 l=1 rx=0x7fc9a40043a0 tx=0x7fc9a40108b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:19:07.039 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:07.020+0000 7fc9ba4bb700 1 --2- 172.21.15.42:0/4254874902 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc9b412cce0 0x7fc9b407db40 secure :-1 s=THROTTLE_DONE pgs=534 cs=0 l=1 rx=0x7fc9a40043a0 tx=0x7fc9a40108b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:19:07.039 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:07.020+0000 7fc9ab7fe700 1 -- 172.21.15.42:0/4254874902 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc9a4005b40 con 0x7fc9b412cce0 2019-12-05T05:19:07.039 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:07.020+0000 7fc9ba4bb700 1 --2- 172.21.15.42:0/4254874902 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc9b412cce0 0x7fc9b407db40 secure :-1 s=THROTTLE_DONE pgs=534 cs=0 l=1 rx=0x7fc9a40043a0 tx=0x7fc9a40108b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:19:07.039 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:07.020+0000 7fc9ab7fe700 1 -- 172.21.15.42:0/4254874902 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9a4005760 con 0x7fc9b412cce0 2019-12-05T05:19:07.052 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:07.020+0000 7fc9ab7fe700 1 -- 172.21.15.42:0/4254874902 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc9a40062b0 con 0x7fc9b412cce0 2019-12-05T05:19:07.053 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:07.020+0000 7fc9bc71f700 1 -- 172.21.15.42:0/4254874902 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9b4142d30 con 0x7fc9b412cce0 2019-12-05T05:19:07.053 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:07.020+0000 7fc9bc71f700 1 -- 172.21.15.42:0/4254874902 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9b4143260 con 0x7fc9b412cce0 2019-12-05T05:19:07.053 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:07.020+0000 7fc9ba4bb700 1 --2- 172.21.15.42:0/4254874902 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc9b412cce0 0x7fc9b407db40 secure :-1 s=THROTTLE_DONE pgs=534 cs=0 l=1 rx=0x7fc9a40043a0 tx=0x7fc9a40108b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:19:07.053 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:07.020+0000 7fc9ab7fe700 1 -- 172.21.15.42:0/4254874902 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fc9a4006580 con 0x7fc9b412cce0 2019-12-05T05:19:07.053 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:07.020+0000 7fc9ba4bb700 1 --2- 172.21.15.42:0/4254874902 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc9b412cce0 0x7fc9b407db40 secure :-1 s=THROTTLE_DONE pgs=534 cs=0 l=1 rx=0x7fc9a40043a0 tx=0x7fc9a40108b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:19:07.053 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:07.020+0000 7fc9ab7fe700 1 --2- 172.21.15.42:0/4254874902 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc99c03ad20 0x7fc99c03d230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:19:07.054 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:07.020+0000 7fc9ab7fe700 1 -- 172.21.15.42:0/4254874902 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6977+0+0 (secure 0 0 0) 0x7fc9a404aeb0 con 0x7fc9b412cce0 2019-12-05T05:19:07.054 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:07.020+0000 7fc9bacbc700 1 --2- 172.21.15.42:0/4254874902 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc99c03ad20 0x7fc99c03d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:19:07.054 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:07.020+0000 7fc9bacbc700 1 --2- 172.21.15.42:0/4254874902 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc99c03ad20 0x7fc99c03d230 crc :-1 s=READY pgs=301 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:19:07.054 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:07.024+0000 7fc9bc71f700 1 -- 172.21.15.42:0/4254874902 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9b405d100 con 0x7fc9b412cce0 2019-12-05T05:19:07.055 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:07.032+0000 7fc9ba4bb700 1 --2- 172.21.15.42:0/4254874902 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc9b412cce0 0x7fc9b407db40 secure :-1 s=THROTTLE_DONE pgs=534 cs=0 l=1 rx=0x7fc9a40043a0 tx=0x7fc9a40108b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:19:07.055 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:07.032+0000 7fc9ab7fe700 1 -- 172.21.15.42:0/4254874902 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fc9a4031f30 con 0x7fc9b412cce0 2019-12-05T05:19:07.206 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:07.200+0000 7fc9bc71f700 1 -- 172.21.15.42:0/4254874902 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc9b405b8b0 con 0x7fc9b412cce0 2019-12-05T05:19:07.208 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:07.200+0000 7fc9ba4bb700 1 --2- 172.21.15.42:0/4254874902 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc9b412cce0 0x7fc9b407db40 secure :-1 s=THROTTLE_DONE pgs=534 cs=0 l=1 rx=0x7fc9a40043a0 tx=0x7fc9a40108b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:19:07.208 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:07.200+0000 7fc9ab7fe700 1 -- 172.21.15.42:0/4254874902 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+2108 (secure 0 0 0) 0x7fc9a4032190 con 0x7fc9b412cce0 2019-12-05T05:19:07.209 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 88 2019-12-05T05:19:07.209 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:19:07.210 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":88,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5066,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/3061061478","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3061061478},{"type":"v1","addr":"172.21.15.40:6833","nonce":3061061478}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":88}],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:18:52.963069+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5833},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5833":{"gid":5833,"name":"a","rank":0,"incarnation":84,"state":"up:active","state_seq":25,"addr":"172.21.15.42:6835/1414371821","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1414371821},{"type":"v1","addr":"172.21.15.42:6835","nonce":1414371821}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:19:07.212 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:07.208+0000 7fc9bc71f700 1 -- 172.21.15.42:0/4254874902 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc99c03ad20 msgr2=0x7fc99c03d230 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:19:07.212 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:07.208+0000 7fc9bc71f700 1 --2- 172.21.15.42:0/4254874902 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc99c03ad20 0x7fc99c03d230 crc :-1 s=READY pgs=301 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:19:07.213 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:07.208+0000 7fc9bc71f700 1 -- 172.21.15.42:0/4254874902 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc9b412cce0 msgr2=0x7fc9b407db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:19:07.213 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:07.208+0000 7fc9bc71f700 1 --2- 172.21.15.42:0/4254874902 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc9b412cce0 0x7fc9b407db40 secure :-1 s=READY pgs=534 cs=0 l=1 rx=0x7fc9a40043a0 tx=0x7fc9a40108b0).stop 2019-12-05T05:19:07.213 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:07.208+0000 7fc9bc71f700 1 -- 172.21.15.42:0/4254874902 shutdown_connections 2019-12-05T05:19:07.213 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:07.208+0000 7fc9bc71f700 1 --2- 172.21.15.42:0/4254874902 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc99c03ad20 0x7fc99c03d230 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:19:07.213 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:07.208+0000 7fc9bc71f700 1 --2- 172.21.15.42:0/4254874902 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc9b41192b0 0x7fc9b4130b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:19:07.213 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:07.208+0000 7fc9bc71f700 1 --2- 172.21.15.42:0/4254874902 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc9b412cce0 0x7fc9b407db40 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:19:07.213 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:07.208+0000 7fc9bc71f700 1 -- 172.21.15.42:0/4254874902 shutdown_connections 2019-12-05T05:19:07.214 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:07.208+0000 7fc9bc71f700 1 -- 172.21.15.42:0/4254874902 wait complete. 2019-12-05T05:19:07.214 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:19:07.208+0000 7fc9bc71f700 1 -- 172.21.15.42:0/4254874902 >> 172.21.15.42:0/4254874902 conn(0x7fc9b40aabd0 msgr2=0x7fc9b411c530 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:19:07.222 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b reported in up:standby state 2019-12-05T05:19:07.223 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 80 secs before thrashing 2019-12-05T05:19:36.102 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:19:36.106 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:20:06.165 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:20:06.169 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:20:27.224 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:20:27.328 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.316+0000 7fac24f91700 1 Processor -- start 2019-12-05T05:20:27.329 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.316+0000 7fac24f91700 1 -- start start 2019-12-05T05:20:27.329 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.320+0000 7fac24f91700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fac200a7740 0x7fac2008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:20:27.329 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.320+0000 7fac24f91700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fac2009fae0 0x7fac2007eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:20:27.329 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.320+0000 7fac24f91700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac200a0710 con 0x7fac2007e720 2019-12-05T05:20:27.329 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.320+0000 7fac24f91700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac200a0b00 con 0x7fac2009fae0 2019-12-05T05:20:27.330 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.320+0000 7fac24f91700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac2008e5b0 con 0x7fac200a7740 2019-12-05T05:20:27.330 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.320+0000 7fac1e59c700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fac2007e720 0x7fac2009f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:59792/0 (socket says 172.21.15.42:59792) 2019-12-05T05:20:27.330 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.320+0000 7fac1e59c700 1 -- 172.21.15.42:0/633778047 learned_addr learned my addr 172.21.15.42:0/633778047 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:20:27.330 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.320+0000 7fac1ed9d700 1 --2- 172.21.15.42:0/633778047 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fac200a7740 0x7fac2008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:20:27.330 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.320+0000 7fac1f59e700 1 --2- 172.21.15.42:0/633778047 >> v2:172.21.15.40:3300/0 conn(0x7fac2009fae0 0x7fac2007eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:20:27.330 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.320+0000 7fac1dd9b700 1 -- 172.21.15.42:0/633778047 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1350951996 0 0) 0x7fac200a0710 con 0x7fac2007e720 2019-12-05T05:20:27.331 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.320+0000 7fac1dd9b700 1 -- 172.21.15.42:0/633778047 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac080030b0 con 0x7fac2007e720 2019-12-05T05:20:27.331 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.320+0000 7fac1ed9d700 1 -- 172.21.15.42:0/633778047 >> v2:172.21.15.40:3300/0 conn(0x7fac2009fae0 msgr2=0x7fac2007eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:20:27.331 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.320+0000 7fac1ed9d700 1 --2- 172.21.15.42:0/633778047 >> v2:172.21.15.40:3300/0 conn(0x7fac2009fae0 0x7fac2007eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:20:27.331 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.320+0000 7fac1ed9d700 1 -- 172.21.15.42:0/633778047 >> v1:172.21.15.42:6789/0 conn(0x7fac2007e720 legacy=0x7fac2009f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:20:27.331 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.320+0000 7fac1ed9d700 1 -- 172.21.15.42:0/633778047 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac2007f0a0 con 0x7fac200a7740 2019-12-05T05:20:27.331 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.320+0000 7fac1ed9d700 1 --2- 172.21.15.42:0/633778047 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fac200a7740 0x7fac2008df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fac10008b10 tx=0x7fac1000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:27.331 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.320+0000 7fac1ed9d700 1 --2- 172.21.15.42:0/633778047 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fac200a7740 0x7fac2008df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fac10008b10 tx=0x7fac1000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:27.332 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.320+0000 7fac1ed9d700 1 --2- 172.21.15.42:0/633778047 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fac200a7740 0x7fac2008df80 secure :-1 s=READY pgs=139 cs=0 l=1 rx=0x7fac10008b10 tx=0x7fac1000fff0).ready entity=mon.2 client_cookie=9a774384ae08fd6f server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:20:27.332 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.320+0000 7fac1ed9d700 1 --2- 172.21.15.42:0/633778047 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fac200a7740 0x7fac2008df80 secure :-1 s=THROTTLE_DONE pgs=139 cs=0 l=1 rx=0x7fac10008b10 tx=0x7fac1000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:27.332 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.320+0000 7fac1dd9b700 1 -- 172.21.15.42:0/633778047 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fac10010900 con 0x7fac200a7740 2019-12-05T05:20:27.332 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.320+0000 7fac1ed9d700 1 --2- 172.21.15.42:0/633778047 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fac200a7740 0x7fac2008df80 secure :-1 s=THROTTLE_DONE pgs=139 cs=0 l=1 rx=0x7fac10008b10 tx=0x7fac1000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:27.332 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.320+0000 7fac1dd9b700 1 -- 172.21.15.42:0/633778047 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac10008b50 con 0x7fac200a7740 2019-12-05T05:20:27.332 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.320+0000 7fac1ed9d700 1 --2- 172.21.15.42:0/633778047 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fac200a7740 0x7fac2008df80 secure :-1 s=THROTTLE_DONE pgs=139 cs=0 l=1 rx=0x7fac10008b10 tx=0x7fac1000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:27.333 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.320+0000 7fac1dd9b700 1 -- 172.21.15.42:0/633778047 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fac100110c0 con 0x7fac200a7740 2019-12-05T05:20:27.333 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.320+0000 7fac24f91700 1 -- 172.21.15.42:0/633778047 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fac200a7740 msgr2=0x7fac2008df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:20:27.333 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.320+0000 7fac24f91700 1 --2- 172.21.15.42:0/633778047 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fac200a7740 0x7fac2008df80 secure :-1 s=READY pgs=139 cs=0 l=1 rx=0x7fac10008b10 tx=0x7fac1000fff0).stop 2019-12-05T05:20:27.333 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.320+0000 7fac24f91700 1 -- 172.21.15.42:0/633778047 shutdown_connections 2019-12-05T05:20:27.333 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.320+0000 7fac24f91700 1 --2- 172.21.15.42:0/633778047 >> v2:172.21.15.40:3300/0 conn(0x7fac2009fae0 0x7fac2007eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:20:27.333 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.320+0000 7fac24f91700 1 --2- 172.21.15.42:0/633778047 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fac200a7740 0x7fac2008df80 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:20:27.333 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.320+0000 7fac24f91700 1 -- 172.21.15.42:0/633778047 shutdown_connections 2019-12-05T05:20:27.334 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.320+0000 7fac24f91700 1 -- 172.21.15.42:0/633778047 wait complete. 2019-12-05T05:20:27.334 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.320+0000 7fac24f91700 1 -- 172.21.15.42:0/633778047 >> 172.21.15.42:0/633778047 conn(0x7fac200aabe0 msgr2=0x7fac200a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:20:27.337 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.324+0000 7fac24f91700 1 Processor -- start 2019-12-05T05:20:27.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.324+0000 7fac24f91700 1 -- start start 2019-12-05T05:20:27.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.328+0000 7fac24f91700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fac20123f40 0x7fac20138500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:20:27.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.328+0000 7fac24f91700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac20124370 con 0x7fac2011a490 2019-12-05T05:20:27.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.328+0000 7fac24f91700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac2007dfe0 con 0x7fac201300a0 2019-12-05T05:20:27.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.328+0000 7fac24f91700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac20122890 con 0x7fac20123f40 2019-12-05T05:20:27.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.328+0000 7fac1ed9d700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fac201300a0 0x7fac20122b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:59796/0 (socket says 172.21.15.42:59796) 2019-12-05T05:20:27.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.328+0000 7fac1ed9d700 1 -- 172.21.15.42:0/1503029616 learned_addr learned my addr 172.21.15.42:0/1503029616 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:20:27.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.328+0000 7fac1f59e700 1 --2- 172.21.15.42:0/1503029616 >> v2:172.21.15.42:3300/0 conn(0x7fac20123f40 0x7fac20138500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:20:27.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.328+0000 7fac0f7fe700 1 -- 172.21.15.42:0/1503029616 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 272816464 0 0) 0x7fac2007dfe0 con 0x7fac201300a0 2019-12-05T05:20:27.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.328+0000 7fac0f7fe700 1 -- 172.21.15.42:0/1503029616 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac040030b0 con 0x7fac201300a0 2019-12-05T05:20:27.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.328+0000 7fac0f7fe700 1 -- 172.21.15.42:0/1503029616 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3182966069 0 0) 0x7fac20124370 con 0x7fac2011a490 2019-12-05T05:20:27.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.328+0000 7fac0f7fe700 1 -- 172.21.15.42:0/1503029616 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac2007dfe0 con 0x7fac2011a490 2019-12-05T05:20:27.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.328+0000 7fac1f59e700 1 -- 172.21.15.42:0/1503029616 >> v1:172.21.15.42:6789/0 conn(0x7fac201300a0 legacy=0x7fac20122b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:20:27.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.328+0000 7fac1f59e700 1 -- 172.21.15.42:0/1503029616 >> v1:172.21.15.40:6789/0 conn(0x7fac2011a490 legacy=0x7fac2011b680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:20:27.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.328+0000 7fac1f59e700 1 -- 172.21.15.42:0/1503029616 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac2011be80 con 0x7fac20123f40 2019-12-05T05:20:27.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.328+0000 7fac1f59e700 1 --2- 172.21.15.42:0/1503029616 >> v2:172.21.15.42:3300/0 conn(0x7fac20123f40 0x7fac20138500 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fac200a63f0 tx=0x7fac18010900).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:27.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.328+0000 7fac1f59e700 1 --2- 172.21.15.42:0/1503029616 >> v2:172.21.15.42:3300/0 conn(0x7fac20123f40 0x7fac20138500 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fac200a63f0 tx=0x7fac18010900).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:27.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.328+0000 7fac1f59e700 1 --2- 172.21.15.42:0/1503029616 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fac20123f40 0x7fac20138500 secure :-1 s=READY pgs=668 cs=0 l=1 rx=0x7fac200a63f0 tx=0x7fac18010900).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:20:27.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.328+0000 7fac1f59e700 1 --2- 172.21.15.42:0/1503029616 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fac20123f40 0x7fac20138500 secure :-1 s=THROTTLE_DONE pgs=668 cs=0 l=1 rx=0x7fac200a63f0 tx=0x7fac18010900).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:27.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.328+0000 7fac0f7fe700 1 -- 172.21.15.42:0/1503029616 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fac18002730 con 0x7fac20123f40 2019-12-05T05:20:27.341 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.328+0000 7fac1f59e700 1 --2- 172.21.15.42:0/1503029616 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fac20123f40 0x7fac20138500 secure :-1 s=THROTTLE_DONE pgs=668 cs=0 l=1 rx=0x7fac200a63f0 tx=0x7fac18010900).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:27.341 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.328+0000 7fac0f7fe700 1 -- 172.21.15.42:0/1503029616 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac18002350 con 0x7fac20123f40 2019-12-05T05:20:27.341 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.328+0000 7fac1f59e700 1 --2- 172.21.15.42:0/1503029616 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fac20123f40 0x7fac20138500 secure :-1 s=THROTTLE_DONE pgs=668 cs=0 l=1 rx=0x7fac200a63f0 tx=0x7fac18010900).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:27.341 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.328+0000 7fac24f91700 1 -- 172.21.15.42:0/1503029616 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac20138a50 con 0x7fac20123f40 2019-12-05T05:20:27.341 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.328+0000 7fac0f7fe700 1 -- 172.21.15.42:0/1503029616 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fac1800d160 con 0x7fac20123f40 2019-12-05T05:20:27.342 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.328+0000 7fac24f91700 1 -- 172.21.15.42:0/1503029616 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac20138fa0 con 0x7fac20123f40 2019-12-05T05:20:27.342 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.328+0000 7fac1f59e700 1 --2- 172.21.15.42:0/1503029616 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fac20123f40 0x7fac20138500 secure :-1 s=THROTTLE_DONE pgs=668 cs=0 l=1 rx=0x7fac200a63f0 tx=0x7fac18010900).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:27.342 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.328+0000 7fac0f7fe700 1 -- 172.21.15.42:0/1503029616 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fac18007170 con 0x7fac20123f40 2019-12-05T05:20:27.342 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.328+0000 7fac1f59e700 1 --2- 172.21.15.42:0/1503029616 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fac20123f40 0x7fac20138500 secure :-1 s=THROTTLE_DONE pgs=668 cs=0 l=1 rx=0x7fac200a63f0 tx=0x7fac18010900).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:27.342 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.332+0000 7fac0f7fe700 1 --2- 172.21.15.42:0/1503029616 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fac0403b420 0x7fac0403d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:20:27.342 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.332+0000 7fac0f7fe700 1 -- 172.21.15.42:0/1503029616 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6977+0+0 (secure 0 0 0) 0x7fac1804aa30 con 0x7fac20123f40 2019-12-05T05:20:27.342 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.332+0000 7fac1ed9d700 1 --2- 172.21.15.42:0/1503029616 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fac0403b420 0x7fac0403d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:20:27.343 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.332+0000 7fac1ed9d700 1 --2- 172.21.15.42:0/1503029616 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fac0403b420 0x7fac0403d930 crc :-1 s=READY pgs=303 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:20:27.343 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.332+0000 7fac24f91700 1 -- 172.21.15.42:0/1503029616 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac2011c0a0 con 0x7fac20123f40 2019-12-05T05:20:27.343 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.336+0000 7fac1f59e700 1 --2- 172.21.15.42:0/1503029616 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fac20123f40 0x7fac20138500 secure :-1 s=THROTTLE_DONE pgs=668 cs=0 l=1 rx=0x7fac200a63f0 tx=0x7fac18010900).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:27.344 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.336+0000 7fac0f7fe700 1 -- 172.21.15.42:0/1503029616 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fac2011be80 con 0x7fac20123f40 2019-12-05T05:20:27.506 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.500+0000 7fac24f91700 1 -- 172.21.15.42:0/1503029616 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fac2005d110 con 0x7fac20123f40 2019-12-05T05:20:27.508 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.500+0000 7fac1f59e700 1 --2- 172.21.15.42:0/1503029616 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fac20123f40 0x7fac20138500 secure :-1 s=THROTTLE_DONE pgs=668 cs=0 l=1 rx=0x7fac200a63f0 tx=0x7fac18010900).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:27.508 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.500+0000 7fac0f7fe700 1 -- 172.21.15.42:0/1503029616 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+2108 (secure 0 0 0) 0x7fac18031a90 con 0x7fac20123f40 2019-12-05T05:20:27.508 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 88 2019-12-05T05:20:27.508 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:20:27.509 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":88,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5066,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/3061061478","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3061061478},{"type":"v1","addr":"172.21.15.40:6833","nonce":3061061478}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":88}],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:18:52.963069+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5833},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5833":{"gid":5833,"name":"a","rank":0,"incarnation":84,"state":"up:active","state_seq":25,"addr":"172.21.15.42:6835/1414371821","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1414371821},{"type":"v1","addr":"172.21.15.42:6835","nonce":1414371821}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:20:27.510 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.504+0000 7fac24f91700 1 -- 172.21.15.42:0/1503029616 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fac0403b420 msgr2=0x7fac0403d930 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:20:27.510 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.504+0000 7fac24f91700 1 --2- 172.21.15.42:0/1503029616 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fac0403b420 0x7fac0403d930 crc :-1 s=READY pgs=303 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:20:27.510 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.504+0000 7fac24f91700 1 -- 172.21.15.42:0/1503029616 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fac20123f40 msgr2=0x7fac20138500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:20:27.511 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.504+0000 7fac24f91700 1 --2- 172.21.15.42:0/1503029616 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fac20123f40 0x7fac20138500 secure :-1 s=READY pgs=668 cs=0 l=1 rx=0x7fac200a63f0 tx=0x7fac18010900).stop 2019-12-05T05:20:27.511 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.504+0000 7fac24f91700 1 -- 172.21.15.42:0/1503029616 shutdown_connections 2019-12-05T05:20:27.511 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.504+0000 7fac24f91700 1 --2- 172.21.15.42:0/1503029616 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fac0403b420 0x7fac0403d930 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:20:27.511 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.504+0000 7fac24f91700 1 --2- 172.21.15.42:0/1503029616 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fac20123f40 0x7fac20138500 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:20:27.511 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.504+0000 7fac24f91700 1 -- 172.21.15.42:0/1503029616 shutdown_connections 2019-12-05T05:20:27.511 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.504+0000 7fac24f91700 1 -- 172.21.15.42:0/1503029616 wait complete. 2019-12-05T05:20:27.512 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.504+0000 7fac24f91700 1 -- 172.21.15.42:0/1503029616 >> 172.21.15.42:0/1503029616 conn(0x7fac200a7740 msgr2=0x7fac200a9790 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:20:27.519 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.a (rank=0) 2019-12-05T05:20:27.520 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2019-12-05T05:20:27.520 INFO:teuthology.orchestra.run:waiting for 300 2019-12-05T05:20:27.533 INFO:tasks.ceph.mds.a:Stopped 2019-12-05T05:20:27.533 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:20:27.704 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.692+0000 7f1d3709f700 1 Processor -- start 2019-12-05T05:20:27.704 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.692+0000 7f1d3709f700 1 -- start start 2019-12-05T05:20:27.704 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.692+0000 7f1d3709f700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1d300a7740 0x7f1d3008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:20:27.705 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.692+0000 7f1d3709f700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f1d3009fae0 0x7f1d3007eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:20:27.705 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.692+0000 7f1d3709f700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d300a0710 con 0x7f1d3007e720 2019-12-05T05:20:27.705 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.692+0000 7f1d3709f700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d300a0b00 con 0x7f1d3009fae0 2019-12-05T05:20:27.705 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.692+0000 7f1d3709f700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d3008e5b0 con 0x7f1d300a7740 2019-12-05T05:20:27.705 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.692+0000 7f1d35e3d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f1d3009fae0 0x7f1d3007eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:20:27.705 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.692+0000 7f1d35e3d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f1d3009fae0 0x7f1d3007eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:59772/0 (socket says 172.21.15.42:59772) 2019-12-05T05:20:27.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.692+0000 7f1d35e3d700 1 -- 172.21.15.42:0/2338188808 learned_addr learned my addr 172.21.15.42:0/2338188808 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:20:27.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.692+0000 7f1d3563c700 1 --2- 172.21.15.42:0/2338188808 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1d300a7740 0x7f1d3008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:20:27.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.696+0000 7f1d35e3d700 1 -- 172.21.15.42:0/2338188808 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1d300a7740 msgr2=0x7f1d3008df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:20:27.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.696+0000 7f1d35e3d700 1 --2- 172.21.15.42:0/2338188808 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1d300a7740 0x7f1d3008df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:20:27.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.696+0000 7f1d35e3d700 1 -- 172.21.15.42:0/2338188808 >> v1:172.21.15.40:6789/0 conn(0x7f1d3007e720 legacy=0x7f1d3009f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:20:27.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.696+0000 7f1d35e3d700 1 -- 172.21.15.42:0/2338188808 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d3007f0a0 con 0x7f1d3009fae0 2019-12-05T05:20:27.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.696+0000 7f1d35e3d700 1 --2- 172.21.15.42:0/2338188808 >> v2:172.21.15.42:3300/0 conn(0x7f1d3009fae0 0x7f1d3007eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f1d28008b10 tx=0x7f1d280100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:27.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.696+0000 7f1d35e3d700 1 --2- 172.21.15.42:0/2338188808 >> v2:172.21.15.42:3300/0 conn(0x7f1d3009fae0 0x7f1d3007eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f1d28008b10 tx=0x7f1d280100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:27.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.696+0000 7f1d35e3d700 1 --2- 172.21.15.42:0/2338188808 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1d3009fae0 0x7f1d3007eb50 secure :-1 s=READY pgs=669 cs=0 l=1 rx=0x7f1d28008b10 tx=0x7f1d280100b0).ready entity=mon.1 client_cookie=e676d3816c98a9d server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:20:27.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.696+0000 7f1d35e3d700 1 --2- 172.21.15.42:0/2338188808 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1d3009fae0 0x7f1d3007eb50 secure :-1 s=THROTTLE_DONE pgs=669 cs=0 l=1 rx=0x7f1d28008b10 tx=0x7f1d280100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:27.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.696+0000 7f1d35e3d700 1 --2- 172.21.15.42:0/2338188808 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1d3009fae0 0x7f1d3007eb50 secure :-1 s=THROTTLE_DONE pgs=669 cs=0 l=1 rx=0x7f1d28008b10 tx=0x7f1d280100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:27.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.696+0000 7f1d27fff700 1 -- 172.21.15.42:0/2338188808 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1d28010da0 con 0x7f1d3009fae0 2019-12-05T05:20:27.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.696+0000 7f1d35e3d700 1 --2- 172.21.15.42:0/2338188808 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1d3009fae0 0x7f1d3007eb50 secure :-1 s=THROTTLE_DONE pgs=669 cs=0 l=1 rx=0x7f1d28008b10 tx=0x7f1d280100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:27.708 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.696+0000 7f1d27fff700 1 -- 172.21.15.42:0/2338188808 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d280109c0 con 0x7f1d3009fae0 2019-12-05T05:20:27.708 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.696+0000 7f1d27fff700 1 -- 172.21.15.42:0/2338188808 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1d280114a0 con 0x7f1d3009fae0 2019-12-05T05:20:27.708 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.696+0000 7f1d3709f700 1 -- 172.21.15.42:0/2338188808 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1d3009fae0 msgr2=0x7f1d3007eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:20:27.708 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.696+0000 7f1d3709f700 1 --2- 172.21.15.42:0/2338188808 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1d3009fae0 0x7f1d3007eb50 secure :-1 s=READY pgs=669 cs=0 l=1 rx=0x7f1d28008b10 tx=0x7f1d280100b0).stop 2019-12-05T05:20:27.708 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.696+0000 7f1d3709f700 1 -- 172.21.15.42:0/2338188808 shutdown_connections 2019-12-05T05:20:27.708 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.696+0000 7f1d3709f700 1 --2- 172.21.15.42:0/2338188808 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1d3009fae0 0x7f1d3007eb50 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:20:27.708 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.696+0000 7f1d3709f700 1 --2- 172.21.15.42:0/2338188808 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1d300a7740 0x7f1d3008df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:20:27.709 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.696+0000 7f1d3709f700 1 -- 172.21.15.42:0/2338188808 shutdown_connections 2019-12-05T05:20:27.709 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.696+0000 7f1d3709f700 1 -- 172.21.15.42:0/2338188808 wait complete. 2019-12-05T05:20:27.709 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.696+0000 7f1d3709f700 1 -- 172.21.15.42:0/2338188808 >> 172.21.15.42:0/2338188808 conn(0x7f1d300aabe0 msgr2=0x7f1d300a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:20:27.712 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.700+0000 7f1d3709f700 1 Processor -- start 2019-12-05T05:20:27.712 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.700+0000 7f1d3709f700 1 -- start start 2019-12-05T05:20:27.713 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.700+0000 7f1d3709f700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f1d30122a80 0x7f1d30122eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:20:27.713 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.700+0000 7f1d3709f700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d3011e930 con 0x7f1d30116500 2019-12-05T05:20:27.713 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.700+0000 7f1d3709f700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d30124190 con 0x7f1d30126960 2019-12-05T05:20:27.713 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.700+0000 7f1d3709f700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d30125450 con 0x7f1d30122a80 2019-12-05T05:20:27.713 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.700+0000 7f1d35e3d700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f1d30126960 0x7f1d30126d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:59814/0 (socket says 172.21.15.42:59814) 2019-12-05T05:20:27.713 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.700+0000 7f1d35e3d700 1 -- 172.21.15.42:0/189040306 learned_addr learned my addr 172.21.15.42:0/189040306 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:20:27.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.700+0000 7f1d34e3b700 1 --2- 172.21.15.42:0/189040306 >> v2:172.21.15.40:3300/0 conn(0x7f1d30122a80 0x7f1d30122eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:20:27.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.700+0000 7f1d25ffb700 1 -- 172.21.15.42:0/189040306 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 214599915 0 0) 0x7f1d30124190 con 0x7f1d30126960 2019-12-05T05:20:27.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.700+0000 7f1d25ffb700 1 -- 172.21.15.42:0/189040306 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d100030b0 con 0x7f1d30126960 2019-12-05T05:20:27.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.700+0000 7f1d25ffb700 1 -- 172.21.15.42:0/189040306 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 3543341648 0 0) 0x7f1d100030b0 con 0x7f1d30126960 2019-12-05T05:20:27.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.700+0000 7f1d34e3b700 1 -- 172.21.15.42:0/189040306 >> v1:172.21.15.42:6789/0 conn(0x7f1d30126960 legacy=0x7f1d30126d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:20:27.715 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.700+0000 7f1d34e3b700 1 -- 172.21.15.42:0/189040306 >> v1:172.21.15.40:6789/0 conn(0x7f1d30116500 legacy=0x7f1d301175a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:20:27.715 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.700+0000 7f1d34e3b700 1 -- 172.21.15.42:0/189040306 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d30123400 con 0x7f1d30122a80 2019-12-05T05:20:27.715 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.700+0000 7f1d34e3b700 1 --2- 172.21.15.42:0/189040306 >> v2:172.21.15.40:3300/0 conn(0x7f1d30122a80 0x7f1d30122eb0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f1d28008a30 tx=0x7f1d28008650).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:27.715 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.700+0000 7f1d34e3b700 1 --2- 172.21.15.42:0/189040306 >> v2:172.21.15.40:3300/0 conn(0x7f1d30122a80 0x7f1d30122eb0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f1d28008a30 tx=0x7f1d28008650).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:27.715 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.700+0000 7f1d34e3b700 1 --2- 172.21.15.42:0/189040306 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1d30122a80 0x7f1d30122eb0 secure :-1 s=READY pgs=540 cs=0 l=1 rx=0x7f1d28008a30 tx=0x7f1d28008650).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:20:27.716 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.704+0000 7f1d34e3b700 1 --2- 172.21.15.42:0/189040306 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1d30122a80 0x7f1d30122eb0 secure :-1 s=THROTTLE_DONE pgs=540 cs=0 l=1 rx=0x7f1d28008a30 tx=0x7f1d28008650).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:27.716 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.704+0000 7f1d25ffb700 1 -- 172.21.15.42:0/189040306 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1d280087f0 con 0x7f1d30122a80 2019-12-05T05:20:27.716 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.704+0000 7f1d34e3b700 1 --2- 172.21.15.42:0/189040306 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1d30122a80 0x7f1d30122eb0 secure :-1 s=THROTTLE_DONE pgs=540 cs=0 l=1 rx=0x7f1d28008a30 tx=0x7f1d28008650).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:27.716 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.704+0000 7f1d34e3b700 1 --2- 172.21.15.42:0/189040306 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1d30122a80 0x7f1d30122eb0 secure :-1 s=THROTTLE_DONE pgs=540 cs=0 l=1 rx=0x7f1d28008a30 tx=0x7f1d28008650).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:27.716 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.704+0000 7f1d25ffb700 1 -- 172.21.15.42:0/189040306 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d280110c0 con 0x7f1d30122a80 2019-12-05T05:20:27.716 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.704+0000 7f1d3709f700 1 -- 172.21.15.42:0/189040306 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d30127590 con 0x7f1d30122a80 2019-12-05T05:20:27.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.704+0000 7f1d25ffb700 1 -- 172.21.15.42:0/189040306 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1d2000cd60 con 0x7f1d30122a80 2019-12-05T05:20:27.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.704+0000 7f1d3709f700 1 -- 172.21.15.42:0/189040306 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d3012ec80 con 0x7f1d30122a80 2019-12-05T05:20:27.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.704+0000 7f1d34e3b700 1 --2- 172.21.15.42:0/189040306 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1d30122a80 0x7f1d30122eb0 secure :-1 s=THROTTLE_DONE pgs=540 cs=0 l=1 rx=0x7f1d28008a30 tx=0x7f1d28008650).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:27.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.704+0000 7f1d25ffb700 1 -- 172.21.15.42:0/189040306 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f1d20006360 con 0x7f1d30122a80 2019-12-05T05:20:27.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.704+0000 7f1d34e3b700 1 --2- 172.21.15.42:0/189040306 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1d30122a80 0x7f1d30122eb0 secure :-1 s=THROTTLE_DONE pgs=540 cs=0 l=1 rx=0x7f1d28008a30 tx=0x7f1d28008650).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:27.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.704+0000 7f1d25ffb700 1 --2- 172.21.15.42:0/189040306 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f1d1003b210 0x7f1d1003d720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:20:27.718 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.704+0000 7f1d25ffb700 1 -- 172.21.15.42:0/189040306 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6977+0+0 (secure 0 0 0) 0x7f1d20011dc0 con 0x7f1d30122a80 2019-12-05T05:20:27.718 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.704+0000 7f1d3563c700 1 --2- 172.21.15.42:0/189040306 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f1d1003b210 0x7f1d1003d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:20:27.718 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.704+0000 7f1d3563c700 1 --2- 172.21.15.42:0/189040306 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f1d1003b210 0x7f1d1003d720 crc :-1 s=READY pgs=304 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:20:27.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.708+0000 7f1d3709f700 1 -- 172.21.15.42:0/189040306 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d3005d110 con 0x7f1d30122a80 2019-12-05T05:20:27.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.708+0000 7f1d34e3b700 1 -- 172.21.15.42:0/189040306 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1d30122a80 msgr2=0x7f1d30122eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2019-12-05T05:20:27.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.708+0000 7f1d34e3b700 1 -- 172.21.15.42:0/189040306 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1d30122a80 msgr2=0x7f1d30122eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2019-12-05T05:20:27.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.708+0000 7f1d34e3b700 1 --2- 172.21.15.42:0/189040306 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1d30122a80 0x7f1d30122eb0 secure :-1 s=READY pgs=540 cs=0 l=1 rx=0x7f1d28008a30 tx=0x7f1d28008650).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-12-05T05:20:27.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.708+0000 7f1d34e3b700 1 --2- 172.21.15.42:0/189040306 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1d30122a80 0x7f1d30122eb0 secure :-1 s=READY pgs=540 cs=0 l=1 rx=0x7f1d28008a30 tx=0x7f1d28008650).stop 2019-12-05T05:20:27.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.708+0000 7f1d25ffb700 1 -- 172.21.15.42:0/189040306 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1d30122a80 msgr2=0x7f1d30122eb0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2019-12-05T05:20:27.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.708+0000 7f1d25ffb700 1 --2- 172.21.15.42:0/189040306 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1d30122a80 0x7f1d30122eb0 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:20:27.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.708+0000 7f1d25ffb700 1 --2- 172.21.15.42:0/189040306 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1d100421c0 0x7f1d10044600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:20:27.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.708+0000 7f1d25ffb700 1 --2- 172.21.15.42:0/189040306 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1d10044d10 0x7f1d10047220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:20:27.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.708+0000 7f1d25ffb700 1 --2- 172.21.15.42:0/189040306 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1d10047a00 0x7f1d10049f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:20:27.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.708+0000 7f1d25ffb700 1 -- 172.21.15.42:0/189040306 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d1004a730 con 0x7f1d10047a00 2019-12-05T05:20:27.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.708+0000 7f1d25ffb700 1 -- 172.21.15.42:0/189040306 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d1004a920 con 0x7f1d10044d10 2019-12-05T05:20:27.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.708+0000 7f1d25ffb700 1 -- 172.21.15.42:0/189040306 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d1004ab60 con 0x7f1d100421c0 2019-12-05T05:20:27.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.708+0000 7f1d35e3d700 1 --2- 172.21.15.42:0/189040306 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1d10044d10 0x7f1d10047220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:20:27.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.708+0000 7f1d34e3b700 1 --2- 172.21.15.42:0/189040306 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1d100421c0 0x7f1d10044600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:20:27.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.708+0000 7f1d3563c700 1 --2- 172.21.15.42:0/189040306 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1d10047a00 0x7f1d10049f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:20:27.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.708+0000 7f1d35e3d700 1 -- 172.21.15.42:0/189040306 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1d100421c0 msgr2=0x7f1d10044600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:20:27.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.708+0000 7f1d35e3d700 1 --2- 172.21.15.42:0/189040306 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1d100421c0 0x7f1d10044600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:20:27.724 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.708+0000 7f1d35e3d700 1 -- 172.21.15.42:0/189040306 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1d10047a00 msgr2=0x7f1d10049f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:20:27.724 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.708+0000 7f1d35e3d700 1 --2- 172.21.15.42:0/189040306 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1d10047a00 0x7f1d10049f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:20:27.724 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.708+0000 7f1d35e3d700 1 -- 172.21.15.42:0/189040306 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f1d1004ae60 con 0x7f1d10044d10 2019-12-05T05:20:27.724 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.708+0000 7f1d35e3d700 1 -- 172.21.15.42:0/189040306 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d28011db0 con 0x7f1d10044d10 2019-12-05T05:20:27.724 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.708+0000 7f1d35e3d700 1 --2- 172.21.15.42:0/189040306 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1d10044d10 0x7f1d10047220 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f1d28008550 tx=0x7f1d2800f1a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:27.724 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.708+0000 7f1d35e3d700 1 -- 172.21.15.42:0/189040306 reap_dead start 2019-12-05T05:20:27.724 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.708+0000 7f1d35e3d700 1 --2- 172.21.15.42:0/189040306 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1d10044d10 0x7f1d10047220 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f1d28008550 tx=0x7f1d2800f1a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:27.725 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.708+0000 7f1d35e3d700 1 --2- 172.21.15.42:0/189040306 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1d10044d10 0x7f1d10047220 secure :-1 s=READY pgs=671 cs=0 l=1 rx=0x7f1d28008550 tx=0x7f1d2800f1a0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:20:27.725 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.708+0000 7f1d35e3d700 1 --2- 172.21.15.42:0/189040306 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1d10044d10 0x7f1d10047220 secure :-1 s=THROTTLE_DONE pgs=671 cs=0 l=1 rx=0x7f1d28008550 tx=0x7f1d2800f1a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:27.725 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.708+0000 7f1d35e3d700 1 --2- 172.21.15.42:0/189040306 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1d10044d10 0x7f1d10047220 secure :-1 s=THROTTLE_DONE pgs=671 cs=0 l=1 rx=0x7f1d28008550 tx=0x7f1d2800f1a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:27.725 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.708+0000 7f1d25ffb700 1 -- 172.21.15.42:0/189040306 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1d28011db0 con 0x7f1d10044d10 2019-12-05T05:20:27.725 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.708+0000 7f1d25ffb700 1 -- 172.21.15.42:0/189040306 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d28005ad0 con 0x7f1d10044d10 2019-12-05T05:20:27.726 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.708+0000 7f1d35e3d700 1 --2- 172.21.15.42:0/189040306 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1d10044d10 0x7f1d10047220 secure :-1 s=THROTTLE_DONE pgs=671 cs=0 l=1 rx=0x7f1d28008550 tx=0x7f1d2800f1a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:27.726 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.712+0000 7f1d25ffb700 1 -- 172.21.15.42:0/189040306 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f1d28007160 con 0x7f1d10044d10 2019-12-05T05:20:27.726 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.716+0000 7f1d35e3d700 1 --2- 172.21.15.42:0/189040306 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1d10044d10 0x7f1d10047220 secure :-1 s=THROTTLE_DONE pgs=671 cs=0 l=1 rx=0x7f1d28008550 tx=0x7f1d2800f1a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:27.726 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.716+0000 7f1d25ffb700 1 -- 172.21.15.42:0/189040306 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f1d28007470 con 0x7f1d10044d10 2019-12-05T05:20:27.892 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.888+0000 7f1d3709f700 1 -- 172.21.15.42:0/189040306 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1d3005b8c0 con 0x7f1d10044d10 2019-12-05T05:20:27.893 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.888+0000 7f1d35e3d700 1 --2- 172.21.15.42:0/189040306 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1d10044d10 0x7f1d10047220 secure :-1 s=THROTTLE_DONE pgs=671 cs=0 l=1 rx=0x7f1d28008550 tx=0x7f1d2800f1a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:27.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.888+0000 7f1d25ffb700 1 -- 172.21.15.42:0/189040306 <== mon.1 v2:172.21.15.42:3300/0 5 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+2108 (secure 0 0 0) 0x7f1d28007690 con 0x7f1d10044d10 2019-12-05T05:20:27.894 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 88 2019-12-05T05:20:27.894 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:20:27.895 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":88,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5066,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/3061061478","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3061061478},{"type":"v1","addr":"172.21.15.40:6833","nonce":3061061478}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":88}],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:18:52.963069+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5833},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5833":{"gid":5833,"name":"a","rank":0,"incarnation":84,"state":"up:active","state_seq":25,"addr":"172.21.15.42:6835/1414371821","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1414371821},{"type":"v1","addr":"172.21.15.42:6835","nonce":1414371821}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:20:27.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.888+0000 7f1d3709f700 1 -- 172.21.15.42:0/189040306 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f1d1003b210 msgr2=0x7f1d1003d720 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:20:27.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.888+0000 7f1d3709f700 1 --2- 172.21.15.42:0/189040306 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f1d1003b210 0x7f1d1003d720 crc :-1 s=READY pgs=304 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:20:27.897 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.888+0000 7f1d3709f700 1 -- 172.21.15.42:0/189040306 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1d10044d10 msgr2=0x7f1d10047220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:20:27.897 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.888+0000 7f1d3709f700 1 --2- 172.21.15.42:0/189040306 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1d10044d10 0x7f1d10047220 secure :-1 s=READY pgs=671 cs=0 l=1 rx=0x7f1d28008550 tx=0x7f1d2800f1a0).stop 2019-12-05T05:20:27.897 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.888+0000 7f1d3709f700 1 -- 172.21.15.42:0/189040306 shutdown_connections 2019-12-05T05:20:27.897 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.888+0000 7f1d3709f700 1 --2- 172.21.15.42:0/189040306 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1d30122a80 0x7f1d30122eb0 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:20:27.898 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.888+0000 7f1d3709f700 1 --2- 172.21.15.42:0/189040306 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f1d1003b210 0x7f1d1003d720 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:20:27.898 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.888+0000 7f1d3709f700 1 --2- 172.21.15.42:0/189040306 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1d10044d10 0x7f1d10047220 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:20:27.898 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.892+0000 7f1d3709f700 1 -- 172.21.15.42:0/189040306 shutdown_connections 2019-12-05T05:20:27.898 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.892+0000 7f1d3709f700 1 -- 172.21.15.42:0/189040306 wait complete. 2019-12-05T05:20:27.899 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:27.892+0000 7f1d3709f700 1 -- 172.21.15.42:0/189040306 >> 172.21.15.42:0/189040306 conn(0x7f1d300a98b0 msgr2=0x7f1d3011db90 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:20:27.907 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T05:20:29.908 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:20:30.013 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:30.000+0000 7f256b98f700 1 Processor -- start 2019-12-05T05:20:30.013 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:30.000+0000 7f256b98f700 1 -- start start 2019-12-05T05:20:30.013 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:30.004+0000 7f256b98f700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f25640a7740 0x7f256408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:20:30.013 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:30.004+0000 7f256b98f700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f256407e720 0x7f256407eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:20:30.013 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:30.004+0000 7f256b98f700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f256409f2e0 0x7f256409f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:20:30.014 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:30.004+0000 7f256b98f700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25640a0b00 con 0x7f256407e720 2019-12-05T05:20:30.014 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:30.004+0000 7f256b98f700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f256408e5b0 con 0x7f256409f2e0 2019-12-05T05:20:30.014 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:30.004+0000 7f256b98f700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f256408e850 con 0x7f25640a7740 2019-12-05T05:20:30.014 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:30.004+0000 7f256a72d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f256409f2e0 0x7f256409f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:20:30.014 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:30.004+0000 7f2569f2c700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f25640a7740 0x7f256408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:20:30.014 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:30.004+0000 7f256972b700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f256407e720 0x7f256407eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:20:30.014 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:30.004+0000 7f256a72d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f256409f2e0 0x7f256409f710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:59798/0 (socket says 172.21.15.42:59798) 2019-12-05T05:20:30.015 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:30.004+0000 7f2569f2c700 1 --2- 172.21.15.42:0/1470991896 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f25640a7740 0x7f256408df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:60260/0 (socket says 172.21.15.42:60260) 2019-12-05T05:20:30.015 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:30.004+0000 7f256a72d700 1 -- 172.21.15.42:0/1470991896 learned_addr learned my addr 172.21.15.42:0/1470991896 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:20:30.015 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:30.004+0000 7f256a72d700 1 -- 172.21.15.42:0/1470991896 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f25640a7740 msgr2=0x7f256408df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:20:30.015 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:30.004+0000 7f256a72d700 1 --2- 172.21.15.42:0/1470991896 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f25640a7740 0x7f256408df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:20:30.015 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:30.004+0000 7f256a72d700 1 -- 172.21.15.42:0/1470991896 >> v2:172.21.15.40:3300/0 conn(0x7f256407e720 msgr2=0x7f256407eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:20:30.015 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:30.004+0000 7f256a72d700 1 --2- 172.21.15.42:0/1470991896 >> v2:172.21.15.40:3300/0 conn(0x7f256407e720 0x7f256407eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:20:30.016 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:30.004+0000 7f256a72d700 1 -- 172.21.15.42:0/1470991896 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f256407f0a0 con 0x7f256409f2e0 2019-12-05T05:20:30.016 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:30.004+0000 7f256972b700 1 --2- 172.21.15.42:0/1470991896 >> v2:172.21.15.40:3300/0 conn(0x7f256407e720 0x7f256407eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:20:30.016 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:30.004+0000 7f256a72d700 1 --2- 172.21.15.42:0/1470991896 >> v2:172.21.15.42:3300/0 conn(0x7f256409f2e0 0x7f256409f710 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f256408e770 tx=0x7f255c00fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:30.016 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:30.004+0000 7f256a72d700 1 --2- 172.21.15.42:0/1470991896 >> v2:172.21.15.42:3300/0 conn(0x7f256409f2e0 0x7f256409f710 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f256408e770 tx=0x7f255c00fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:30.016 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:30.004+0000 7f256a72d700 1 --2- 172.21.15.42:0/1470991896 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f256409f2e0 0x7f256409f710 secure :-1 s=READY pgs=672 cs=0 l=1 rx=0x7f256408e770 tx=0x7f255c00fdb0).ready entity=mon.1 client_cookie=fc89bdd07bf664f2 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:20:30.016 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:30.004+0000 7f256a72d700 1 --2- 172.21.15.42:0/1470991896 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f256409f2e0 0x7f256409f710 secure :-1 s=THROTTLE_DONE pgs=672 cs=0 l=1 rx=0x7f256408e770 tx=0x7f255c00fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:30.017 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:30.004+0000 7f2568f2a700 1 -- 172.21.15.42:0/1470991896 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f255c008c90 con 0x7f256409f2e0 2019-12-05T05:20:30.017 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:30.004+0000 7f256a72d700 1 --2- 172.21.15.42:0/1470991896 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f256409f2e0 0x7f256409f710 secure :-1 s=THROTTLE_DONE pgs=672 cs=0 l=1 rx=0x7f256408e770 tx=0x7f255c00fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:30.017 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:30.004+0000 7f2568f2a700 1 -- 172.21.15.42:0/1470991896 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f255c0088b0 con 0x7f256409f2e0 2019-12-05T05:20:30.017 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:30.004+0000 7f256a72d700 1 --2- 172.21.15.42:0/1470991896 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f256409f2e0 0x7f256409f710 secure :-1 s=THROTTLE_DONE pgs=672 cs=0 l=1 rx=0x7f256408e770 tx=0x7f255c00fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:30.017 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:30.004+0000 7f2568f2a700 1 -- 172.21.15.42:0/1470991896 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f255c011b50 con 0x7f256409f2e0 2019-12-05T05:20:30.018 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:30.004+0000 7f256b98f700 1 -- 172.21.15.42:0/1470991896 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f256409f2e0 msgr2=0x7f256409f710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:20:30.018 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:30.004+0000 7f256b98f700 1 --2- 172.21.15.42:0/1470991896 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f256409f2e0 0x7f256409f710 secure :-1 s=READY pgs=672 cs=0 l=1 rx=0x7f256408e770 tx=0x7f255c00fdb0).stop 2019-12-05T05:20:30.018 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:30.004+0000 7f256b98f700 1 -- 172.21.15.42:0/1470991896 shutdown_connections 2019-12-05T05:20:30.018 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:30.004+0000 7f256b98f700 1 --2- 172.21.15.42:0/1470991896 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f256409f2e0 0x7f256409f710 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:20:30.018 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:30.004+0000 7f256b98f700 1 --2- 172.21.15.42:0/1470991896 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f25640a7740 0x7f256408df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:20:30.018 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:30.004+0000 7f256b98f700 1 --2- 172.21.15.42:0/1470991896 >> v2:172.21.15.40:3300/0 conn(0x7f256407e720 0x7f256407eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:20:30.019 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:30.004+0000 7f256b98f700 1 -- 172.21.15.42:0/1470991896 shutdown_connections 2019-12-05T05:20:30.019 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:30.004+0000 7f256b98f700 1 -- 172.21.15.42:0/1470991896 wait complete. 2019-12-05T05:20:30.019 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:30.004+0000 7f256b98f700 1 -- 172.21.15.42:0/1470991896 >> 172.21.15.42:0/1470991896 conn(0x7f25640aabe0 msgr2=0x7f25640a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:20:30.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:30.008+0000 7f256b98f700 1 Processor -- start 2019-12-05T05:20:30.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:30.008+0000 7f256b98f700 1 -- start start 2019-12-05T05:20:30.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:30.012+0000 7f256b98f700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f25641245a0 0x7f2564122140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:20:30.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:30.012+0000 7f256b98f700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f256411a8a0 con 0x7f256407d970 2019-12-05T05:20:30.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:30.012+0000 7f256b98f700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2564122690 con 0x7f25641234d0 2019-12-05T05:20:30.024 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:30.012+0000 7f256b98f700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2564123900 con 0x7f25641245a0 2019-12-05T05:20:30.024 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:30.012+0000 7f256a72d700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f25641234d0 0x7f25641378f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:59842/0 (socket says 172.21.15.42:59842) 2019-12-05T05:20:30.024 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:30.012+0000 7f256a72d700 1 -- 172.21.15.42:0/2553965745 learned_addr learned my addr 172.21.15.42:0/2553965745 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:20:30.024 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:30.012+0000 7f2569f2c700 1 --2- 172.21.15.42:0/2553965745 >> v2:172.21.15.40:3300/0 conn(0x7f25641245a0 0x7f2564122140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:20:30.024 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:30.012+0000 7f255a7fc700 1 -- 172.21.15.42:0/2553965745 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1391462023 0 0) 0x7f2564122690 con 0x7f25641234d0 2019-12-05T05:20:30.024 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:30.012+0000 7f255a7fc700 1 -- 172.21.15.42:0/2553965745 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f254c0030b0 con 0x7f25641234d0 2019-12-05T05:20:30.025 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:30.012+0000 7f2569f2c700 1 -- 172.21.15.42:0/2553965745 >> v1:172.21.15.42:6789/0 conn(0x7f25641234d0 legacy=0x7f25641378f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:20:30.025 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:30.012+0000 7f255a7fc700 1 -- 172.21.15.42:0/2553965745 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1813197001 0 0) 0x7f256411a8a0 con 0x7f256407d970 2019-12-05T05:20:30.025 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:30.012+0000 7f2569f2c700 1 -- 172.21.15.42:0/2553965745 >> v1:172.21.15.40:6789/0 conn(0x7f256407d970 legacy=0x7f2564122cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:20:30.025 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:30.012+0000 7f2569f2c700 1 -- 172.21.15.42:0/2553965745 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f256407dda0 con 0x7f25641245a0 2019-12-05T05:20:30.025 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:30.012+0000 7f2569f2c700 1 --2- 172.21.15.42:0/2553965745 >> v2:172.21.15.40:3300/0 conn(0x7f25641245a0 0x7f2564122140 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f25600032b0 tx=0x7f2560010ce0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:30.025 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:30.012+0000 7f2569f2c700 1 --2- 172.21.15.42:0/2553965745 >> v2:172.21.15.40:3300/0 conn(0x7f25641245a0 0x7f2564122140 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f25600032b0 tx=0x7f2560010ce0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:30.025 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:30.012+0000 7f2569f2c700 1 --2- 172.21.15.42:0/2553965745 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f25641245a0 0x7f2564122140 secure :-1 s=READY pgs=542 cs=0 l=1 rx=0x7f25600032b0 tx=0x7f2560010ce0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:20:30.026 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:30.012+0000 7f2569f2c700 1 --2- 172.21.15.42:0/2553965745 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f25641245a0 0x7f2564122140 secure :-1 s=THROTTLE_DONE pgs=542 cs=0 l=1 rx=0x7f25600032b0 tx=0x7f2560010ce0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:30.026 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:30.012+0000 7f255a7fc700 1 -- 172.21.15.42:0/2553965745 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2560012c90 con 0x7f25641245a0 2019-12-05T05:20:30.026 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:30.012+0000 7f2569f2c700 1 --2- 172.21.15.42:0/2553965745 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f25641245a0 0x7f2564122140 secure :-1 s=THROTTLE_DONE pgs=542 cs=0 l=1 rx=0x7f25600032b0 tx=0x7f2560010ce0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:30.026 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:30.012+0000 7f255a7fc700 1 -- 172.21.15.42:0/2553965745 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25600128b0 con 0x7f25641245a0 2019-12-05T05:20:30.026 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:30.012+0000 7f2569f2c700 1 --2- 172.21.15.42:0/2553965745 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f25641245a0 0x7f2564122140 secure :-1 s=THROTTLE_DONE pgs=542 cs=0 l=1 rx=0x7f25600032b0 tx=0x7f2560010ce0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:30.026 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:30.012+0000 7f256b98f700 1 -- 172.21.15.42:0/2553965745 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f256407e090 con 0x7f25641245a0 2019-12-05T05:20:30.027 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:30.012+0000 7f255a7fc700 1 -- 172.21.15.42:0/2553965745 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f256000e770 con 0x7f25641245a0 2019-12-05T05:20:30.027 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:30.012+0000 7f256b98f700 1 -- 172.21.15.42:0/2553965745 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2564139100 con 0x7f25641245a0 2019-12-05T05:20:30.027 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:30.012+0000 7f2569f2c700 1 --2- 172.21.15.42:0/2553965745 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f25641245a0 0x7f2564122140 secure :-1 s=THROTTLE_DONE pgs=542 cs=0 l=1 rx=0x7f25600032b0 tx=0x7f2560010ce0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:30.027 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:30.016+0000 7f255a7fc700 1 -- 172.21.15.42:0/2553965745 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f256000ea10 con 0x7f25641245a0 2019-12-05T05:20:30.027 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:30.016+0000 7f2569f2c700 1 --2- 172.21.15.42:0/2553965745 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f25641245a0 0x7f2564122140 secure :-1 s=THROTTLE_DONE pgs=542 cs=0 l=1 rx=0x7f25600032b0 tx=0x7f2560010ce0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:30.027 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:30.016+0000 7f255a7fc700 1 --2- 172.21.15.42:0/2553965745 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f254c03b1e0 0x7f254c03d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:20:30.027 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:30.016+0000 7f255a7fc700 1 -- 172.21.15.42:0/2553965745 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6977+0+0 (secure 0 0 0) 0x7f256004bad0 con 0x7f25641245a0 2019-12-05T05:20:30.028 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:30.016+0000 7f256972b700 1 --2- 172.21.15.42:0/2553965745 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f254c03b1e0 0x7f254c03d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:20:30.028 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:30.016+0000 7f256972b700 1 --2- 172.21.15.42:0/2553965745 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f254c03b1e0 0x7f254c03d6f0 crc :-1 s=READY pgs=305 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:20:30.028 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:30.016+0000 7f256b98f700 1 -- 172.21.15.42:0/2553965745 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f256405d110 con 0x7f25641245a0 2019-12-05T05:20:30.029 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:30.024+0000 7f2569f2c700 1 --2- 172.21.15.42:0/2553965745 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f25641245a0 0x7f2564122140 secure :-1 s=THROTTLE_DONE pgs=542 cs=0 l=1 rx=0x7f25600032b0 tx=0x7f2560010ce0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:30.029 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:30.024+0000 7f255a7fc700 1 -- 172.21.15.42:0/2553965745 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f256004bde0 con 0x7f25641245a0 2019-12-05T05:20:30.201 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:30.196+0000 7f256b98f700 1 -- 172.21.15.42:0/2553965745 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f256405b8c0 con 0x7f25641245a0 2019-12-05T05:20:30.202 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:30.196+0000 7f2569f2c700 1 --2- 172.21.15.42:0/2553965745 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f25641245a0 0x7f2564122140 secure :-1 s=THROTTLE_DONE pgs=542 cs=0 l=1 rx=0x7f25600032b0 tx=0x7f2560010ce0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:30.202 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:30.196+0000 7f255a7fc700 1 -- 172.21.15.42:0/2553965745 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+2108 (secure 0 0 0) 0x7f2560032b30 con 0x7f25641245a0 2019-12-05T05:20:30.204 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 88 2019-12-05T05:20:30.204 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:20:30.204 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":88,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5066,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/3061061478","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3061061478},{"type":"v1","addr":"172.21.15.40:6833","nonce":3061061478}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":88}],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:18:52.963069+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5833},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5833":{"gid":5833,"name":"a","rank":0,"incarnation":84,"state":"up:active","state_seq":25,"addr":"172.21.15.42:6835/1414371821","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1414371821},{"type":"v1","addr":"172.21.15.42:6835","nonce":1414371821}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:20:30.206 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:30.200+0000 7f256b98f700 1 -- 172.21.15.42:0/2553965745 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f254c03b1e0 msgr2=0x7f254c03d6f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:20:30.206 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:30.200+0000 7f256b98f700 1 --2- 172.21.15.42:0/2553965745 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f254c03b1e0 0x7f254c03d6f0 crc :-1 s=READY pgs=305 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:20:30.207 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:30.200+0000 7f256b98f700 1 -- 172.21.15.42:0/2553965745 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f25641245a0 msgr2=0x7f2564122140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:20:30.207 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:30.200+0000 7f256b98f700 1 --2- 172.21.15.42:0/2553965745 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f25641245a0 0x7f2564122140 secure :-1 s=READY pgs=542 cs=0 l=1 rx=0x7f25600032b0 tx=0x7f2560010ce0).stop 2019-12-05T05:20:30.207 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:30.200+0000 7f256b98f700 1 -- 172.21.15.42:0/2553965745 shutdown_connections 2019-12-05T05:20:30.207 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:30.200+0000 7f256b98f700 1 --2- 172.21.15.42:0/2553965745 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f254c03b1e0 0x7f254c03d6f0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:20:30.207 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:30.200+0000 7f256b98f700 1 --2- 172.21.15.42:0/2553965745 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f25641245a0 0x7f2564122140 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:20:30.207 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:30.200+0000 7f256b98f700 1 -- 172.21.15.42:0/2553965745 shutdown_connections 2019-12-05T05:20:30.208 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:30.200+0000 7f256b98f700 1 -- 172.21.15.42:0/2553965745 wait complete. 2019-12-05T05:20:30.208 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:30.200+0000 7f256b98f700 1 -- 172.21.15.42:0/2553965745 >> 172.21.15.42:0/2553965745 conn(0x7f256407e4f0 msgr2=0x7f25640a9790 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:20:30.217 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T05:20:32.218 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:20:32.321 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:32.312+0000 7f7027590700 1 Processor -- start 2019-12-05T05:20:32.322 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:32.312+0000 7f7027590700 1 -- start start 2019-12-05T05:20:32.322 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:32.312+0000 7f7027590700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f70200a7740 0x7f702008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:20:32.322 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:32.312+0000 7f7027590700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f702009fae0 0x7f702007eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:20:32.323 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:32.312+0000 7f7027590700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70200a0710 con 0x7f702007e720 2019-12-05T05:20:32.323 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:32.312+0000 7f7027590700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70200a0b00 con 0x7f70200a7740 2019-12-05T05:20:32.323 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:32.312+0000 7f7027590700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f702008e5b0 con 0x7f702009fae0 2019-12-05T05:20:32.324 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:32.312+0000 7f7025b2d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f70200a7740 0x7f702008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:20:32.324 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:32.312+0000 7f7025b2d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f70200a7740 0x7f702008df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:59814/0 (socket says 172.21.15.42:59814) 2019-12-05T05:20:32.324 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:32.312+0000 7f7025b2d700 1 -- 172.21.15.42:0/4218595929 learned_addr learned my addr 172.21.15.42:0/4218595929 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:20:32.324 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:32.312+0000 7f702632e700 1 --2- 172.21.15.42:0/4218595929 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f702009fae0 0x7f702007eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:20:32.324 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:32.312+0000 7f7025b2d700 1 -- 172.21.15.42:0/4218595929 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f702009fae0 msgr2=0x7f702007eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:20:32.325 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:32.312+0000 7f7025b2d700 1 --2- 172.21.15.42:0/4218595929 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f702009fae0 0x7f702007eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:20:32.325 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:32.312+0000 7f7025b2d700 1 -- 172.21.15.42:0/4218595929 >> v1:172.21.15.40:6789/0 conn(0x7f702007e720 legacy=0x7f702009f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:20:32.325 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:32.312+0000 7f7025b2d700 1 -- 172.21.15.42:0/4218595929 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f702007f0a0 con 0x7f70200a7740 2019-12-05T05:20:32.325 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:32.312+0000 7f7025b2d700 1 --2- 172.21.15.42:0/4218595929 >> v2:172.21.15.42:3300/0 conn(0x7f70200a7740 0x7f702008df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f701c008790 tx=0x7f701c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:32.326 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:32.312+0000 7f7025b2d700 1 --2- 172.21.15.42:0/4218595929 >> v2:172.21.15.42:3300/0 conn(0x7f70200a7740 0x7f702008df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f701c008790 tx=0x7f701c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:32.326 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:32.312+0000 7f7025b2d700 1 --2- 172.21.15.42:0/4218595929 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f70200a7740 0x7f702008df80 secure :-1 s=READY pgs=677 cs=0 l=1 rx=0x7f701c008790 tx=0x7f701c00fd30).ready entity=mon.1 client_cookie=a1f8b476163f3f71 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:20:32.326 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:32.312+0000 7f7025b2d700 1 --2- 172.21.15.42:0/4218595929 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f70200a7740 0x7f702008df80 secure :-1 s=THROTTLE_DONE pgs=677 cs=0 l=1 rx=0x7f701c008790 tx=0x7f701c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:32.327 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:32.312+0000 7f7024b2b700 1 -- 172.21.15.42:0/4218595929 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f701c008c90 con 0x7f70200a7740 2019-12-05T05:20:32.327 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:32.312+0000 7f7025b2d700 1 --2- 172.21.15.42:0/4218595929 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f70200a7740 0x7f702008df80 secure :-1 s=THROTTLE_DONE pgs=677 cs=0 l=1 rx=0x7f701c008790 tx=0x7f701c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:32.327 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:32.312+0000 7f7025b2d700 1 --2- 172.21.15.42:0/4218595929 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f70200a7740 0x7f702008df80 secure :-1 s=THROTTLE_DONE pgs=677 cs=0 l=1 rx=0x7f701c008790 tx=0x7f701c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:32.327 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:32.312+0000 7f7024b2b700 1 -- 172.21.15.42:0/4218595929 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f701c0088b0 con 0x7f70200a7740 2019-12-05T05:20:32.328 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:32.312+0000 7f7024b2b700 1 -- 172.21.15.42:0/4218595929 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f701c011b10 con 0x7f70200a7740 2019-12-05T05:20:32.328 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:32.312+0000 7f7027590700 1 -- 172.21.15.42:0/4218595929 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f70200a7740 msgr2=0x7f702008df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:20:32.328 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:32.312+0000 7f7027590700 1 --2- 172.21.15.42:0/4218595929 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f70200a7740 0x7f702008df80 secure :-1 s=READY pgs=677 cs=0 l=1 rx=0x7f701c008790 tx=0x7f701c00fd30).stop 2019-12-05T05:20:32.328 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:32.316+0000 7f7027590700 1 -- 172.21.15.42:0/4218595929 shutdown_connections 2019-12-05T05:20:32.328 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:32.316+0000 7f7027590700 1 --2- 172.21.15.42:0/4218595929 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f702009fae0 0x7f702007eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:20:32.329 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:32.316+0000 7f7027590700 1 --2- 172.21.15.42:0/4218595929 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f70200a7740 0x7f702008df80 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:20:32.329 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:32.316+0000 7f7027590700 1 -- 172.21.15.42:0/4218595929 shutdown_connections 2019-12-05T05:20:32.329 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:32.316+0000 7f7027590700 1 -- 172.21.15.42:0/4218595929 wait complete. 2019-12-05T05:20:32.329 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:32.316+0000 7f7027590700 1 -- 172.21.15.42:0/4218595929 >> 172.21.15.42:0/4218595929 conn(0x7f70200aabe0 msgr2=0x7f70200a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:20:32.335 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:32.316+0000 7f7027590700 1 Processor -- start 2019-12-05T05:20:32.335 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:32.316+0000 7f7027590700 1 -- start start 2019-12-05T05:20:32.335 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:32.316+0000 7f7027590700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f702008eeb0 0x7f70200658b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:20:32.335 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:32.316+0000 7f7027590700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7020063800 0x7f70200a3c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:20:32.336 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:32.316+0000 7f7027590700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7020063c30 con 0x7f70200a41d0 2019-12-05T05:20:32.336 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:32.316+0000 7f7027590700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7020063f80 con 0x7f7020063800 2019-12-05T05:20:32.336 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:32.316+0000 7f7027590700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7020064220 con 0x7f702008eeb0 2019-12-05T05:20:32.336 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:32.316+0000 7f7025b2d700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f702008eeb0 0x7f70200658b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:20:32.337 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:32.316+0000 7f702532c700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7020063800 0x7f70200a3c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:20:32.337 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:32.316+0000 7f7025b2d700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f702008eeb0 0x7f70200658b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:60286/0 (socket says 172.21.15.42:60286) 2019-12-05T05:20:32.337 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:32.316+0000 7f7025b2d700 1 -- 172.21.15.42:0/161998973 learned_addr learned my addr 172.21.15.42:0/161998973 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:20:32.337 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:32.320+0000 7f7025b2d700 1 -- 172.21.15.42:0/161998973 >> v2:172.21.15.40:3300/0 conn(0x7f7020063800 msgr2=0x7f70200a3c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:20:32.337 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:32.320+0000 7f7025b2d700 1 --2- 172.21.15.42:0/161998973 >> v2:172.21.15.40:3300/0 conn(0x7f7020063800 0x7f70200a3c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:20:32.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:32.320+0000 7f7025b2d700 1 -- 172.21.15.42:0/161998973 >> v1:172.21.15.40:6789/0 conn(0x7f70200a41d0 legacy=0x7f7020126e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:20:32.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:32.320+0000 7f7025b2d700 1 -- 172.21.15.42:0/161998973 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70200644c0 con 0x7f702008eeb0 2019-12-05T05:20:32.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:32.320+0000 7f7025b2d700 1 --2- 172.21.15.42:0/161998973 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f702008eeb0 0x7f70200658b0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f701c008790 tx=0x7f701c008b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:32.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:32.320+0000 7f7025b2d700 1 --2- 172.21.15.42:0/161998973 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f702008eeb0 0x7f70200658b0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f701c008790 tx=0x7f701c008b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:32.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:32.320+0000 7f7025b2d700 1 --2- 172.21.15.42:0/161998973 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f702008eeb0 0x7f70200658b0 secure :-1 s=READY pgs=140 cs=0 l=1 rx=0x7f701c008790 tx=0x7f701c008b60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:20:32.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:32.320+0000 7f7025b2d700 1 --2- 172.21.15.42:0/161998973 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f702008eeb0 0x7f70200658b0 secure :-1 s=THROTTLE_DONE pgs=140 cs=0 l=1 rx=0x7f701c008790 tx=0x7f701c008b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:32.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:32.320+0000 7f7025b2d700 1 --2- 172.21.15.42:0/161998973 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f702008eeb0 0x7f70200658b0 secure :-1 s=THROTTLE_DONE pgs=140 cs=0 l=1 rx=0x7f701c008790 tx=0x7f701c008b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:32.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:32.320+0000 7f70127fc700 1 -- 172.21.15.42:0/161998973 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f701c005730 con 0x7f702008eeb0 2019-12-05T05:20:32.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:32.320+0000 7f7025b2d700 1 --2- 172.21.15.42:0/161998973 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f702008eeb0 0x7f70200658b0 secure :-1 s=THROTTLE_DONE pgs=140 cs=0 l=1 rx=0x7f701c008790 tx=0x7f701c008b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:32.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:32.320+0000 7f70127fc700 1 -- 172.21.15.42:0/161998973 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f701c011730 con 0x7f702008eeb0 2019-12-05T05:20:32.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:32.320+0000 7f7027590700 1 -- 172.21.15.42:0/161998973 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7020127610 con 0x7f702008eeb0 2019-12-05T05:20:32.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:32.320+0000 7f70127fc700 1 -- 172.21.15.42:0/161998973 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f701c005d10 con 0x7f702008eeb0 2019-12-05T05:20:32.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:32.320+0000 7f7027590700 1 -- 172.21.15.42:0/161998973 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7020131920 con 0x7f702008eeb0 2019-12-05T05:20:32.341 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:32.320+0000 7f7025b2d700 1 --2- 172.21.15.42:0/161998973 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f702008eeb0 0x7f70200658b0 secure :-1 s=THROTTLE_DONE pgs=140 cs=0 l=1 rx=0x7f701c008790 tx=0x7f701c008b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:32.341 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:32.320+0000 7f7025b2d700 1 --2- 172.21.15.42:0/161998973 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f702008eeb0 0x7f70200658b0 secure :-1 s=THROTTLE_DONE pgs=140 cs=0 l=1 rx=0x7f701c008790 tx=0x7f701c008b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:32.341 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:32.320+0000 7f70127fc700 1 -- 172.21.15.42:0/161998973 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f701c0104d0 con 0x7f702008eeb0 2019-12-05T05:20:32.341 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:32.320+0000 7f70127fc700 1 --2- 172.21.15.42:0/161998973 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f700803ae00 0x7f700803d310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:20:32.342 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:32.324+0000 7f70127fc700 1 -- 172.21.15.42:0/161998973 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6977+0+0 (secure 0 0 0) 0x7f701c04b4f0 con 0x7f702008eeb0 2019-12-05T05:20:32.342 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:32.324+0000 7f702532c700 1 --2- 172.21.15.42:0/161998973 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f700803ae00 0x7f700803d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:20:32.342 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:32.324+0000 7f702532c700 1 --2- 172.21.15.42:0/161998973 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f700803ae00 0x7f700803d310 crc :-1 s=READY pgs=306 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:20:32.342 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:32.328+0000 7f7027590700 1 -- 172.21.15.42:0/161998973 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f702005d110 con 0x7f702008eeb0 2019-12-05T05:20:32.343 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:32.336+0000 7f7025b2d700 1 --2- 172.21.15.42:0/161998973 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f702008eeb0 0x7f70200658b0 secure :-1 s=THROTTLE_DONE pgs=140 cs=0 l=1 rx=0x7f701c008790 tx=0x7f701c008b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:32.343 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:32.336+0000 7f70127fc700 1 -- 172.21.15.42:0/161998973 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f701c010dd0 con 0x7f702008eeb0 2019-12-05T05:20:32.512 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:32.508+0000 7f7027590700 1 -- 172.21.15.42:0/161998973 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f702005b8c0 con 0x7f702008eeb0 2019-12-05T05:20:32.513 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:32.508+0000 7f7025b2d700 1 --2- 172.21.15.42:0/161998973 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f702008eeb0 0x7f70200658b0 secure :-1 s=THROTTLE_DONE pgs=140 cs=0 l=1 rx=0x7f701c008790 tx=0x7f701c008b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:32.513 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:32.508+0000 7f70127fc700 1 -- 172.21.15.42:0/161998973 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+2108 (secure 0 0 0) 0x7f701c032530 con 0x7f702008eeb0 2019-12-05T05:20:32.514 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 88 2019-12-05T05:20:32.514 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:20:32.514 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":88,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5066,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/3061061478","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3061061478},{"type":"v1","addr":"172.21.15.40:6833","nonce":3061061478}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":88}],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:18:52.963069+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5833},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5833":{"gid":5833,"name":"a","rank":0,"incarnation":84,"state":"up:active","state_seq":25,"addr":"172.21.15.42:6835/1414371821","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1414371821},{"type":"v1","addr":"172.21.15.42:6835","nonce":1414371821}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:20:32.515 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:32.508+0000 7f7027590700 1 -- 172.21.15.42:0/161998973 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f700803ae00 msgr2=0x7f700803d310 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:20:32.516 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:32.508+0000 7f7027590700 1 --2- 172.21.15.42:0/161998973 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f700803ae00 0x7f700803d310 crc :-1 s=READY pgs=306 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:20:32.516 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:32.508+0000 7f7027590700 1 -- 172.21.15.42:0/161998973 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f702008eeb0 msgr2=0x7f70200658b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:20:32.516 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:32.508+0000 7f7027590700 1 --2- 172.21.15.42:0/161998973 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f702008eeb0 0x7f70200658b0 secure :-1 s=READY pgs=140 cs=0 l=1 rx=0x7f701c008790 tx=0x7f701c008b60).stop 2019-12-05T05:20:32.517 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:32.512+0000 7f7027590700 1 -- 172.21.15.42:0/161998973 shutdown_connections 2019-12-05T05:20:32.517 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:32.512+0000 7f7027590700 1 --2- 172.21.15.42:0/161998973 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f700803ae00 0x7f700803d310 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:20:32.517 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:32.512+0000 7f7027590700 1 --2- 172.21.15.42:0/161998973 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f702008eeb0 0x7f70200658b0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:20:32.517 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:32.512+0000 7f7027590700 1 --2- 172.21.15.42:0/161998973 >> v2:172.21.15.40:3300/0 conn(0x7f7020063800 0x7f70200a3c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:20:32.517 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:32.512+0000 7f7027590700 1 -- 172.21.15.42:0/161998973 shutdown_connections 2019-12-05T05:20:32.517 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:32.512+0000 7f7027590700 1 -- 172.21.15.42:0/161998973 wait complete. 2019-12-05T05:20:32.518 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:32.512+0000 7f7027590700 1 -- 172.21.15.42:0/161998973 >> 172.21.15.42:0/161998973 conn(0x7f702009f040 msgr2=0x7f702009f580 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:20:32.532 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T05:20:34.532 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:20:34.638 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:34.628+0000 7feb27a8a700 1 Processor -- start 2019-12-05T05:20:34.639 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:34.628+0000 7feb27a8a700 1 -- start start 2019-12-05T05:20:34.639 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:34.628+0000 7feb27a8a700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7feb2009fae0 0x7feb2014fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:20:34.639 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:34.628+0000 7feb27a8a700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb200a0710 con 0x7feb2008df80 2019-12-05T05:20:34.640 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:34.628+0000 7feb27a8a700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb2007ef20 con 0x7feb200a7740 2019-12-05T05:20:34.640 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:34.628+0000 7feb27a8a700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb200a0b00 con 0x7feb2009fae0 2019-12-05T05:20:34.640 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:34.632+0000 7feb26828700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7feb2009fae0 0x7feb2014fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:20:34.640 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:34.632+0000 7feb26828700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7feb2009fae0 0x7feb2014fff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:59834/0 (socket says 172.21.15.42:59834) 2019-12-05T05:20:34.641 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:34.632+0000 7feb26828700 1 -- 172.21.15.42:0/2934935246 learned_addr learned my addr 172.21.15.42:0/2934935246 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:20:34.641 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:34.632+0000 7feb25826700 1 -- 172.21.15.42:0/2934935246 >> v1:172.21.15.40:6789/0 conn(0x7feb2008df80 legacy=0x7feb2009f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2019-12-05T05:20:34.641 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:34.632+0000 7feb25826700 1 -- 172.21.15.42:0/2934935246 >> v1:172.21.15.40:6789/0 conn(0x7feb2008df80 legacy=0x7feb2009f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2019-12-05T05:20:34.641 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:34.632+0000 7feb25826700 1 --1- 172.21.15.42:0/2934935246 >> v1:172.21.15.40:6789/0 conn(0x7feb2008df80 0x7feb2009f2e0 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).handle_connect_reply_1 read connect reply failed 2019-12-05T05:20:34.641 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:34.632+0000 7feb26828700 1 -- 172.21.15.42:0/2934935246 >> v1:172.21.15.42:6789/0 conn(0x7feb200a7740 legacy=0x7feb2007e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:20:34.642 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:34.632+0000 7feb26828700 1 -- 172.21.15.42:0/2934935246 >> v1:172.21.15.40:6789/0 conn(0x7feb2008df80 legacy=0x7feb2009f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:20:34.642 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:34.632+0000 7feb26828700 1 -- 172.21.15.42:0/2934935246 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb200a9ba0 con 0x7feb2009fae0 2019-12-05T05:20:34.642 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:34.632+0000 7feb26828700 1 --2- 172.21.15.42:0/2934935246 >> v2:172.21.15.42:3300/0 conn(0x7feb2009fae0 0x7feb2014fff0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7feb18008b10 tx=0x7feb18010030).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:34.642 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:34.632+0000 7feb26828700 1 --2- 172.21.15.42:0/2934935246 >> v2:172.21.15.42:3300/0 conn(0x7feb2009fae0 0x7feb2014fff0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7feb18008b10 tx=0x7feb18010030).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:34.643 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:34.632+0000 7feb26828700 1 --2- 172.21.15.42:0/2934935246 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feb2009fae0 0x7feb2014fff0 secure :-1 s=READY pgs=679 cs=0 l=1 rx=0x7feb18008b10 tx=0x7feb18010030).ready entity=mon.1 client_cookie=72a70c2e9f870eca server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:20:34.643 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:34.632+0000 7feb26828700 1 --2- 172.21.15.42:0/2934935246 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feb2009fae0 0x7feb2014fff0 secure :-1 s=THROTTLE_DONE pgs=679 cs=0 l=1 rx=0x7feb18008b10 tx=0x7feb18010030).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:34.643 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:34.632+0000 7feb25025700 1 -- 172.21.15.42:0/2934935246 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7feb180110c0 con 0x7feb2009fae0 2019-12-05T05:20:34.643 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:34.632+0000 7feb26828700 1 --2- 172.21.15.42:0/2934935246 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feb2009fae0 0x7feb2014fff0 secure :-1 s=THROTTLE_DONE pgs=679 cs=0 l=1 rx=0x7feb18008b10 tx=0x7feb18010030).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:34.644 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:34.632+0000 7feb26828700 1 --2- 172.21.15.42:0/2934935246 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feb2009fae0 0x7feb2014fff0 secure :-1 s=THROTTLE_DONE pgs=679 cs=0 l=1 rx=0x7feb18008b10 tx=0x7feb18010030).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:34.644 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:34.632+0000 7feb25025700 1 -- 172.21.15.42:0/2934935246 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb18010a30 con 0x7feb2009fae0 2019-12-05T05:20:34.644 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:34.632+0000 7feb25025700 1 -- 172.21.15.42:0/2934935246 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7feb180114b0 con 0x7feb2009fae0 2019-12-05T05:20:34.645 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:34.632+0000 7feb27a8a700 1 -- 172.21.15.42:0/2934935246 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feb2009fae0 msgr2=0x7feb2014fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:20:34.645 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:34.632+0000 7feb27a8a700 1 --2- 172.21.15.42:0/2934935246 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feb2009fae0 0x7feb2014fff0 secure :-1 s=READY pgs=679 cs=0 l=1 rx=0x7feb18008b10 tx=0x7feb18010030).stop 2019-12-05T05:20:34.645 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:34.632+0000 7feb27a8a700 1 -- 172.21.15.42:0/2934935246 shutdown_connections 2019-12-05T05:20:34.645 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:34.632+0000 7feb27a8a700 1 --2- 172.21.15.42:0/2934935246 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feb2009fae0 0x7feb2014fff0 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:20:34.645 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:34.632+0000 7feb27a8a700 1 -- 172.21.15.42:0/2934935246 shutdown_connections 2019-12-05T05:20:34.646 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:34.632+0000 7feb27a8a700 1 -- 172.21.15.42:0/2934935246 wait complete. 2019-12-05T05:20:34.646 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:34.632+0000 7feb27a8a700 1 -- 172.21.15.42:0/2934935246 >> 172.21.15.42:0/2934935246 conn(0x7feb200aabe0 msgr2=0x7feb200a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:20:34.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:34.632+0000 7feb27a8a700 1 Processor -- start 2019-12-05T05:20:34.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:34.636+0000 7feb27a8a700 1 -- start start 2019-12-05T05:20:34.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:34.636+0000 7feb27a8a700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7feb2011ba00 0x7feb2012d400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:20:34.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:34.636+0000 7feb27a8a700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7feb20135e00 0x7feb20138240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:20:34.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:34.636+0000 7feb27a8a700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb20142530 con 0x7feb2012d950 2019-12-05T05:20:34.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:34.636+0000 7feb27a8a700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb20142d40 con 0x7feb20135e00 2019-12-05T05:20:34.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:34.636+0000 7feb27a8a700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb20142fe0 con 0x7feb2011ba00 2019-12-05T05:20:34.654 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:34.636+0000 7feb25826700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7feb2012d950 0x7feb20144b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:59878/0 (socket says 172.21.15.42:59878) 2019-12-05T05:20:34.654 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:34.636+0000 7feb25826700 1 -- 172.21.15.42:0/1330013690 learned_addr learned my addr 172.21.15.42:0/1330013690 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:20:34.655 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:34.636+0000 7feb26828700 1 --2- 172.21.15.42:0/1330013690 >> v2:172.21.15.40:3300/0 conn(0x7feb20135e00 0x7feb20138240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:20:34.655 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:34.636+0000 7feb26027700 1 --2- 172.21.15.42:0/1330013690 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7feb2011ba00 0x7feb2012d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:20:34.655 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:34.636+0000 7feb16ffd700 1 -- 172.21.15.42:0/1330013690 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1018066623 0 0) 0x7feb20142530 con 0x7feb2012d950 2019-12-05T05:20:34.655 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:34.636+0000 7feb16ffd700 1 -- 172.21.15.42:0/1330013690 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb080030b0 con 0x7feb2012d950 2019-12-05T05:20:34.656 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:34.636+0000 7feb26828700 1 -- 172.21.15.42:0/1330013690 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7feb2011ba00 msgr2=0x7feb2012d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:20:34.656 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:34.636+0000 7feb26828700 1 --2- 172.21.15.42:0/1330013690 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7feb2011ba00 0x7feb2012d400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:20:34.656 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:34.636+0000 7feb26828700 1 -- 172.21.15.42:0/1330013690 >> v1:172.21.15.42:6789/0 conn(0x7feb2012d950 legacy=0x7feb20144b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:20:34.656 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:34.636+0000 7feb26828700 1 -- 172.21.15.42:0/1330013690 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb20145360 con 0x7feb20135e00 2019-12-05T05:20:34.656 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:34.636+0000 7feb26828700 1 --2- 172.21.15.42:0/1330013690 >> v2:172.21.15.40:3300/0 conn(0x7feb20135e00 0x7feb20138240 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7feb18008b10 tx=0x7feb18010ce0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:34.657 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:34.636+0000 7feb26828700 1 --2- 172.21.15.42:0/1330013690 >> v2:172.21.15.40:3300/0 conn(0x7feb20135e00 0x7feb20138240 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7feb18008b10 tx=0x7feb18010ce0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:34.657 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:34.636+0000 7feb26828700 1 --2- 172.21.15.42:0/1330013690 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb20135e00 0x7feb20138240 secure :-1 s=READY pgs=547 cs=0 l=1 rx=0x7feb18008b10 tx=0x7feb18010ce0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:20:34.657 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:34.636+0000 7feb26828700 1 --2- 172.21.15.42:0/1330013690 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb20135e00 0x7feb20138240 secure :-1 s=THROTTLE_DONE pgs=547 cs=0 l=1 rx=0x7feb18008b10 tx=0x7feb18010ce0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:34.657 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:34.636+0000 7feb16ffd700 1 -- 172.21.15.42:0/1330013690 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7feb18011cd0 con 0x7feb20135e00 2019-12-05T05:20:34.658 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:34.636+0000 7feb26828700 1 --2- 172.21.15.42:0/1330013690 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb20135e00 0x7feb20138240 secure :-1 s=THROTTLE_DONE pgs=547 cs=0 l=1 rx=0x7feb18008b10 tx=0x7feb18010ce0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:34.658 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:34.636+0000 7feb26828700 1 --2- 172.21.15.42:0/1330013690 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb20135e00 0x7feb20138240 secure :-1 s=THROTTLE_DONE pgs=547 cs=0 l=1 rx=0x7feb18008b10 tx=0x7feb18010ce0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:34.658 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:34.636+0000 7feb16ffd700 1 -- 172.21.15.42:0/1330013690 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb18010e10 con 0x7feb20135e00 2019-12-05T05:20:34.658 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:34.636+0000 7feb27a8a700 1 -- 172.21.15.42:0/1330013690 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb2011be30 con 0x7feb20135e00 2019-12-05T05:20:34.659 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:34.636+0000 7feb16ffd700 1 -- 172.21.15.42:0/1330013690 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7feb18005ca0 con 0x7feb20135e00 2019-12-05T05:20:34.659 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:34.636+0000 7feb27a8a700 1 -- 172.21.15.42:0/1330013690 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb201278e0 con 0x7feb20135e00 2019-12-05T05:20:34.659 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:34.640+0000 7feb26828700 1 --2- 172.21.15.42:0/1330013690 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb20135e00 0x7feb20138240 secure :-1 s=THROTTLE_DONE pgs=547 cs=0 l=1 rx=0x7feb18008b10 tx=0x7feb18010ce0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:34.659 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:34.640+0000 7feb26828700 1 --2- 172.21.15.42:0/1330013690 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb20135e00 0x7feb20138240 secure :-1 s=THROTTLE_DONE pgs=547 cs=0 l=1 rx=0x7feb18008b10 tx=0x7feb18010ce0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:34.659 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:34.640+0000 7feb16ffd700 1 -- 172.21.15.42:0/1330013690 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7feb1800f280 con 0x7feb20135e00 2019-12-05T05:20:34.660 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:34.640+0000 7feb16ffd700 1 --2- 172.21.15.42:0/1330013690 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7feb0803b2a0 0x7feb0803d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:20:34.660 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:34.640+0000 7feb16ffd700 1 -- 172.21.15.42:0/1330013690 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6977+0+0 (secure 0 0 0) 0x7feb1804b630 con 0x7feb20135e00 2019-12-05T05:20:34.660 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:34.640+0000 7feb26027700 1 --2- 172.21.15.42:0/1330013690 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7feb0803b2a0 0x7feb0803d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:20:34.660 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:34.640+0000 7feb26027700 1 --2- 172.21.15.42:0/1330013690 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7feb0803b2a0 0x7feb0803d7b0 crc :-1 s=READY pgs=307 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:20:34.661 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:34.644+0000 7feb27a8a700 1 -- 172.21.15.42:0/1330013690 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb2005d110 con 0x7feb20135e00 2019-12-05T05:20:34.661 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:34.652+0000 7feb26828700 1 --2- 172.21.15.42:0/1330013690 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb20135e00 0x7feb20138240 secure :-1 s=THROTTLE_DONE pgs=547 cs=0 l=1 rx=0x7feb18008b10 tx=0x7feb18010ce0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:34.662 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:34.652+0000 7feb16ffd700 1 -- 172.21.15.42:0/1330013690 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7feb180328d0 con 0x7feb20135e00 2019-12-05T05:20:34.829 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:34.824+0000 7feb27a8a700 1 -- 172.21.15.42:0/1330013690 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7feb2005b8c0 con 0x7feb20135e00 2019-12-05T05:20:34.830 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:34.824+0000 7feb26828700 1 --2- 172.21.15.42:0/1330013690 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb20135e00 0x7feb20138240 secure :-1 s=THROTTLE_DONE pgs=547 cs=0 l=1 rx=0x7feb18008b10 tx=0x7feb18010ce0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:34.830 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:34.824+0000 7feb16ffd700 1 -- 172.21.15.42:0/1330013690 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+2108 (secure 0 0 0) 0x7feb1804e950 con 0x7feb20135e00 2019-12-05T05:20:34.832 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 88 2019-12-05T05:20:34.832 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:20:34.832 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":88,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5066,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/3061061478","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3061061478},{"type":"v1","addr":"172.21.15.40:6833","nonce":3061061478}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":88}],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:18:52.963069+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5833},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5833":{"gid":5833,"name":"a","rank":0,"incarnation":84,"state":"up:active","state_seq":25,"addr":"172.21.15.42:6835/1414371821","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1414371821},{"type":"v1","addr":"172.21.15.42:6835","nonce":1414371821}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:20:34.834 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:34.828+0000 7feb27a8a700 1 -- 172.21.15.42:0/1330013690 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7feb0803b2a0 msgr2=0x7feb0803d7b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:20:34.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:34.828+0000 7feb27a8a700 1 --2- 172.21.15.42:0/1330013690 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7feb0803b2a0 0x7feb0803d7b0 crc :-1 s=READY pgs=307 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:20:34.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:34.828+0000 7feb27a8a700 1 -- 172.21.15.42:0/1330013690 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb20135e00 msgr2=0x7feb20138240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:20:34.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:34.828+0000 7feb27a8a700 1 --2- 172.21.15.42:0/1330013690 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb20135e00 0x7feb20138240 secure :-1 s=READY pgs=547 cs=0 l=1 rx=0x7feb18008b10 tx=0x7feb18010ce0).stop 2019-12-05T05:20:34.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:34.828+0000 7feb27a8a700 1 -- 172.21.15.42:0/1330013690 shutdown_connections 2019-12-05T05:20:34.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:34.828+0000 7feb27a8a700 1 --2- 172.21.15.42:0/1330013690 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7feb0803b2a0 0x7feb0803d7b0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:20:34.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:34.828+0000 7feb27a8a700 1 --2- 172.21.15.42:0/1330013690 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb20135e00 0x7feb20138240 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:20:34.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:34.828+0000 7feb27a8a700 1 --2- 172.21.15.42:0/1330013690 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7feb2011ba00 0x7feb2012d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:20:34.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:34.828+0000 7feb27a8a700 1 -- 172.21.15.42:0/1330013690 shutdown_connections 2019-12-05T05:20:34.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:34.828+0000 7feb27a8a700 1 -- 172.21.15.42:0/1330013690 wait complete. 2019-12-05T05:20:34.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:34.828+0000 7feb27a8a700 1 -- 172.21.15.42:0/1330013690 >> 172.21.15.42:0/1330013690 conn(0x7feb20119440 msgr2=0x7feb20059ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:20:34.850 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T05:20:36.572 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:20:36.575 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:20:36.851 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:20:36.957 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:36.944+0000 7f9ddb99e700 1 Processor -- start 2019-12-05T05:20:36.957 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:36.944+0000 7f9ddb99e700 1 -- start start 2019-12-05T05:20:36.957 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:36.944+0000 7f9ddb99e700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9dd408df80 0x7f9dd409f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:20:36.957 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:36.944+0000 7f9ddb99e700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9dd409fc60 con 0x7f9dd40a7740 2019-12-05T05:20:36.958 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:36.944+0000 7f9ddb99e700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9dd40a0710 con 0x7f9dd409f830 2019-12-05T05:20:36.958 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:36.944+0000 7f9ddb99e700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9dd40a0b00 con 0x7f9dd408df80 2019-12-05T05:20:36.958 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:36.944+0000 7f9dda73c700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f9dd409f830 0x7f9dd414efe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:59886/0 (socket says 172.21.15.42:59886) 2019-12-05T05:20:36.958 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:36.944+0000 7f9dda73c700 1 -- 172.21.15.42:0/1087792887 learned_addr learned my addr 172.21.15.42:0/1087792887 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:20:36.958 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:36.944+0000 7f9dd973a700 1 --2- 172.21.15.42:0/1087792887 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9dd408df80 0x7f9dd409f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:20:36.958 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:36.944+0000 7f9dd8f39700 1 -- 172.21.15.42:0/1087792887 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2251539527 0 0) 0x7f9dd40a0710 con 0x7f9dd409f830 2019-12-05T05:20:36.959 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:36.944+0000 7f9dd8f39700 1 -- 172.21.15.42:0/1087792887 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9dc00030b0 con 0x7f9dd409f830 2019-12-05T05:20:36.959 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:36.944+0000 7f9dd8f39700 1 -- 172.21.15.42:0/1087792887 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2028978209 0 0) 0x7f9dd409fc60 con 0x7f9dd40a7740 2019-12-05T05:20:36.959 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:36.944+0000 7f9dd8f39700 1 -- 172.21.15.42:0/1087792887 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9dd40a0710 con 0x7f9dd40a7740 2019-12-05T05:20:36.959 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:36.944+0000 7f9dd973a700 1 -- 172.21.15.42:0/1087792887 >> v1:172.21.15.42:6789/0 conn(0x7f9dd409f830 legacy=0x7f9dd414efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:20:36.959 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:36.944+0000 7f9dd8f39700 1 -- 172.21.15.42:0/1087792887 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3440796778 0 0) 0x7f9dc00030b0 con 0x7f9dd409f830 2019-12-05T05:20:36.959 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:36.944+0000 7f9dd973a700 1 -- 172.21.15.42:0/1087792887 >> v1:172.21.15.40:6789/0 conn(0x7f9dd40a7740 legacy=0x7f9dd407e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:20:36.960 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:36.944+0000 7f9dd973a700 1 -- 172.21.15.42:0/1087792887 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dd407ef20 con 0x7f9dd408df80 2019-12-05T05:20:36.960 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:36.944+0000 7f9dd973a700 1 --2- 172.21.15.42:0/1087792887 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9dd408df80 0x7f9dd409f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f9dc4008790 tx=0x7f9dcc001cd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:36.960 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:36.944+0000 7f9dd973a700 1 --2- 172.21.15.42:0/1087792887 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9dd408df80 0x7f9dd409f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f9dc4008790 tx=0x7f9dcc001cd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:36.960 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:36.944+0000 7f9dd973a700 1 --2- 172.21.15.42:0/1087792887 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9dd408df80 0x7f9dd409f2e0 secure :-1 s=READY pgs=141 cs=0 l=1 rx=0x7f9dc4008790 tx=0x7f9dcc001cd0).ready entity=mon.2 client_cookie=87806f01e3f0731d server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:20:36.960 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:36.948+0000 7f9dd973a700 1 --2- 172.21.15.42:0/1087792887 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9dd408df80 0x7f9dd409f2e0 secure :-1 s=THROTTLE_DONE pgs=141 cs=0 l=1 rx=0x7f9dc4008790 tx=0x7f9dcc001cd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:36.960 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:36.948+0000 7f9dd8f39700 1 -- 172.21.15.42:0/1087792887 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9dcc001ad0 con 0x7f9dd408df80 2019-12-05T05:20:36.960 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:36.948+0000 7f9dd973a700 1 --2- 172.21.15.42:0/1087792887 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9dd408df80 0x7f9dd409f2e0 secure :-1 s=THROTTLE_DONE pgs=141 cs=0 l=1 rx=0x7f9dc4008790 tx=0x7f9dcc001cd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:36.961 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:36.948+0000 7f9dd973a700 1 --2- 172.21.15.42:0/1087792887 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9dd408df80 0x7f9dd409f2e0 secure :-1 s=THROTTLE_DONE pgs=141 cs=0 l=1 rx=0x7f9dc4008790 tx=0x7f9dcc001cd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:36.961 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:36.948+0000 7f9dd8f39700 1 -- 172.21.15.42:0/1087792887 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9dcc0018e0 con 0x7f9dd408df80 2019-12-05T05:20:36.961 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:36.948+0000 7f9dd8f39700 1 -- 172.21.15.42:0/1087792887 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9dc4011360 con 0x7f9dd408df80 2019-12-05T05:20:36.961 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:36.948+0000 7f9ddb99e700 1 -- 172.21.15.42:0/1087792887 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9dd408df80 msgr2=0x7f9dd409f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:20:36.961 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:36.948+0000 7f9ddb99e700 1 --2- 172.21.15.42:0/1087792887 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9dd408df80 0x7f9dd409f2e0 secure :-1 s=READY pgs=141 cs=0 l=1 rx=0x7f9dc4008790 tx=0x7f9dcc001cd0).stop 2019-12-05T05:20:36.962 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:36.948+0000 7f9ddb99e700 1 -- 172.21.15.42:0/1087792887 shutdown_connections 2019-12-05T05:20:36.962 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:36.948+0000 7f9ddb99e700 1 --2- 172.21.15.42:0/1087792887 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9dd408df80 0x7f9dd409f2e0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:20:36.962 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:36.948+0000 7f9ddb99e700 1 -- 172.21.15.42:0/1087792887 shutdown_connections 2019-12-05T05:20:36.962 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:36.948+0000 7f9ddb99e700 1 -- 172.21.15.42:0/1087792887 wait complete. 2019-12-05T05:20:36.962 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:36.948+0000 7f9ddb99e700 1 -- 172.21.15.42:0/1087792887 >> 172.21.15.42:0/1087792887 conn(0x7f9dd40aabe0 msgr2=0x7f9dd40a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:20:36.962 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:36.952+0000 7f9ddb99e700 1 Processor -- start 2019-12-05T05:20:36.962 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:36.952+0000 7f9ddb99e700 1 -- start start 2019-12-05T05:20:36.966 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:36.952+0000 7f9ddb99e700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9dd4112ac0 0x7f9dd411ae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:20:36.967 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:36.952+0000 7f9ddb99e700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f9dd40a57a0 0x7f9dd40a6510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:20:36.967 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:36.952+0000 7f9ddb99e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f9dd40a6a60 0x7f9dd41189c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:20:36.967 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:36.952+0000 7f9ddb99e700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9dd40a5260 con 0x7f9dd40a57a0 2019-12-05T05:20:36.967 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:36.952+0000 7f9ddb99e700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9dd40a5500 con 0x7f9dd40a6a60 2019-12-05T05:20:36.967 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:36.952+0000 7f9ddb99e700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9dd40a6e90 con 0x7f9dd4112ac0 2019-12-05T05:20:36.967 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:36.952+0000 7f9dda73c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f9dd40a6a60 0x7f9dd41189c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:20:36.968 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:36.952+0000 7f9dda73c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f9dd40a6a60 0x7f9dd41189c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:59854/0 (socket says 172.21.15.42:59854) 2019-12-05T05:20:36.968 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:36.952+0000 7f9dda73c700 1 -- 172.21.15.42:0/230843866 learned_addr learned my addr 172.21.15.42:0/230843866 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:20:36.968 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:36.952+0000 7f9dd973a700 1 --2- 172.21.15.42:0/230843866 >> v2:172.21.15.40:3300/0 conn(0x7f9dd40a57a0 0x7f9dd40a6510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:20:36.968 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:36.952+0000 7f9dd9f3b700 1 --2- 172.21.15.42:0/230843866 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9dd4112ac0 0x7f9dd411ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:20:36.968 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:36.952+0000 7f9dda73c700 1 -- 172.21.15.42:0/230843866 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9dd4112ac0 msgr2=0x7f9dd411ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:20:36.969 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:36.952+0000 7f9dda73c700 1 --2- 172.21.15.42:0/230843866 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9dd4112ac0 0x7f9dd411ae40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:20:36.969 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:36.952+0000 7f9dda73c700 1 -- 172.21.15.42:0/230843866 >> v2:172.21.15.40:3300/0 conn(0x7f9dd40a57a0 msgr2=0x7f9dd40a6510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:20:36.969 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:36.952+0000 7f9dda73c700 1 --2- 172.21.15.42:0/230843866 >> v2:172.21.15.40:3300/0 conn(0x7f9dd40a57a0 0x7f9dd40a6510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:20:36.969 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:36.952+0000 7f9dda73c700 1 -- 172.21.15.42:0/230843866 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dd411b390 con 0x7f9dd40a6a60 2019-12-05T05:20:36.969 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:36.952+0000 7f9dda73c700 1 --2- 172.21.15.42:0/230843866 >> v2:172.21.15.42:3300/0 conn(0x7f9dd40a6a60 0x7f9dd41189c0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f9dcc001320 tx=0x7f9dcc010ce0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:36.969 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:36.956+0000 7f9dda73c700 1 --2- 172.21.15.42:0/230843866 >> v2:172.21.15.42:3300/0 conn(0x7f9dd40a6a60 0x7f9dd41189c0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f9dcc001320 tx=0x7f9dcc010ce0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:36.970 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:36.956+0000 7f9dda73c700 1 --2- 172.21.15.42:0/230843866 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9dd40a6a60 0x7f9dd41189c0 secure :-1 s=READY pgs=682 cs=0 l=1 rx=0x7f9dcc001320 tx=0x7f9dcc010ce0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:20:36.970 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:36.956+0000 7f9dda73c700 1 --2- 172.21.15.42:0/230843866 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9dd40a6a60 0x7f9dd41189c0 secure :-1 s=THROTTLE_DONE pgs=682 cs=0 l=1 rx=0x7f9dcc001320 tx=0x7f9dcc010ce0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:36.970 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:36.956+0000 7f9dda73c700 1 --2- 172.21.15.42:0/230843866 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9dd40a6a60 0x7f9dd41189c0 secure :-1 s=THROTTLE_DONE pgs=682 cs=0 l=1 rx=0x7f9dcc001320 tx=0x7f9dcc010ce0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:36.970 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:36.956+0000 7f9dca7fc700 1 -- 172.21.15.42:0/230843866 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9dcc004b90 con 0x7f9dd40a6a60 2019-12-05T05:20:36.970 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:36.956+0000 7f9dda73c700 1 --2- 172.21.15.42:0/230843866 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9dd40a6a60 0x7f9dd41189c0 secure :-1 s=THROTTLE_DONE pgs=682 cs=0 l=1 rx=0x7f9dcc001320 tx=0x7f9dcc010ce0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:36.970 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:36.956+0000 7f9dca7fc700 1 -- 172.21.15.42:0/230843866 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9dcc0047b0 con 0x7f9dd40a6a60 2019-12-05T05:20:36.970 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:36.956+0000 7f9dca7fc700 1 -- 172.21.15.42:0/230843866 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9dcc012980 con 0x7f9dd40a6a60 2019-12-05T05:20:36.971 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:36.956+0000 7f9ddb99e700 1 -- 172.21.15.42:0/230843866 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9dd40a4e80 con 0x7f9dd40a6a60 2019-12-05T05:20:36.971 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:36.956+0000 7f9ddb99e700 1 -- 172.21.15.42:0/230843866 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9dd41190b0 con 0x7f9dd40a6a60 2019-12-05T05:20:36.971 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:36.956+0000 7f9dda73c700 1 --2- 172.21.15.42:0/230843866 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9dd40a6a60 0x7f9dd41189c0 secure :-1 s=THROTTLE_DONE pgs=682 cs=0 l=1 rx=0x7f9dcc001320 tx=0x7f9dcc010ce0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:36.971 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:36.956+0000 7f9dca7fc700 1 -- 172.21.15.42:0/230843866 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f9dcc012c50 con 0x7f9dd40a6a60 2019-12-05T05:20:36.971 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:36.956+0000 7f9dda73c700 1 --2- 172.21.15.42:0/230843866 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9dd40a6a60 0x7f9dd41189c0 secure :-1 s=THROTTLE_DONE pgs=682 cs=0 l=1 rx=0x7f9dcc001320 tx=0x7f9dcc010ce0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:36.971 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:36.956+0000 7f9dca7fc700 1 --2- 172.21.15.42:0/230843866 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9dbc03ad10 0x7f9dbc03d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:20:36.972 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:36.956+0000 7f9dca7fc700 1 -- 172.21.15.42:0/230843866 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6977+0+0 (secure 0 0 0) 0x7f9dcc04b670 con 0x7f9dd40a6a60 2019-12-05T05:20:36.972 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:36.956+0000 7f9dd9f3b700 1 --2- 172.21.15.42:0/230843866 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9dbc03ad10 0x7f9dbc03d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:20:36.972 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:36.956+0000 7f9dd9f3b700 1 --2- 172.21.15.42:0/230843866 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9dbc03ad10 0x7f9dbc03d220 crc :-1 s=READY pgs=308 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:20:36.972 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:36.960+0000 7f9ddb99e700 1 -- 172.21.15.42:0/230843866 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9dd41197d0 con 0x7f9dd40a6a60 2019-12-05T05:20:36.972 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:36.968+0000 7f9dda73c700 1 --2- 172.21.15.42:0/230843866 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9dd40a6a60 0x7f9dd41189c0 secure :-1 s=THROTTLE_DONE pgs=682 cs=0 l=1 rx=0x7f9dcc001320 tx=0x7f9dcc010ce0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:36.973 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:36.968+0000 7f9dca7fc700 1 -- 172.21.15.42:0/230843866 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f9dcc0326f0 con 0x7f9dd40a6a60 2019-12-05T05:20:37.142 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:37.136+0000 7f9ddb99e700 1 -- 172.21.15.42:0/230843866 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9dd405b8c0 con 0x7f9dd40a6a60 2019-12-05T05:20:37.143 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:37.136+0000 7f9dda73c700 1 --2- 172.21.15.42:0/230843866 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9dd40a6a60 0x7f9dd41189c0 secure :-1 s=THROTTLE_DONE pgs=682 cs=0 l=1 rx=0x7f9dcc001320 tx=0x7f9dcc010ce0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:37.143 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:37.136+0000 7f9dca7fc700 1 -- 172.21.15.42:0/230843866 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+2108 (secure 0 0 0) 0x7f9dcc032950 con 0x7f9dd40a6a60 2019-12-05T05:20:37.144 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:20:37.144 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":88,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5066,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/3061061478","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3061061478},{"type":"v1","addr":"172.21.15.40:6833","nonce":3061061478}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":88}],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:18:52.963069+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5833},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5833":{"gid":5833,"name":"a","rank":0,"incarnation":84,"state":"up:active","state_seq":25,"addr":"172.21.15.42:6835/1414371821","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1414371821},{"type":"v1","addr":"172.21.15.42:6835","nonce":1414371821}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:20:37.144 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 88 2019-12-05T05:20:37.146 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:37.140+0000 7f9ddb99e700 1 -- 172.21.15.42:0/230843866 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9dbc03ad10 msgr2=0x7f9dbc03d220 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:20:37.146 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:37.140+0000 7f9ddb99e700 1 --2- 172.21.15.42:0/230843866 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9dbc03ad10 0x7f9dbc03d220 crc :-1 s=READY pgs=308 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:20:37.146 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:37.140+0000 7f9ddb99e700 1 -- 172.21.15.42:0/230843866 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9dd40a6a60 msgr2=0x7f9dd41189c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:20:37.146 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:37.140+0000 7f9ddb99e700 1 --2- 172.21.15.42:0/230843866 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9dd40a6a60 0x7f9dd41189c0 secure :-1 s=READY pgs=682 cs=0 l=1 rx=0x7f9dcc001320 tx=0x7f9dcc010ce0).stop 2019-12-05T05:20:37.147 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:37.140+0000 7f9ddb99e700 1 -- 172.21.15.42:0/230843866 shutdown_connections 2019-12-05T05:20:37.147 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:37.140+0000 7f9ddb99e700 1 --2- 172.21.15.42:0/230843866 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9dbc03ad10 0x7f9dbc03d220 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:20:37.147 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:37.140+0000 7f9ddb99e700 1 --2- 172.21.15.42:0/230843866 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9dd40a6a60 0x7f9dd41189c0 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:20:37.147 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:37.140+0000 7f9ddb99e700 1 --2- 172.21.15.42:0/230843866 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9dd4112ac0 0x7f9dd411ae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:20:37.147 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:37.140+0000 7f9ddb99e700 1 --2- 172.21.15.42:0/230843866 >> v2:172.21.15.40:3300/0 conn(0x7f9dd40a57a0 0x7f9dd40a6510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:20:37.147 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:37.140+0000 7f9ddb99e700 1 -- 172.21.15.42:0/230843866 shutdown_connections 2019-12-05T05:20:37.148 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:37.140+0000 7f9ddb99e700 1 -- 172.21.15.42:0/230843866 wait complete. 2019-12-05T05:20:37.148 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:37.140+0000 7f9ddb99e700 1 -- 172.21.15.42:0/230843866 >> 172.21.15.42:0/230843866 conn(0x7f9dd407e4f0 msgr2=0x7f9dd411bf80 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:20:37.156 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T05:20:39.157 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:20:39.267 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:39.256+0000 7fca3f35a700 1 Processor -- start 2019-12-05T05:20:39.267 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:39.256+0000 7fca3f35a700 1 -- start start 2019-12-05T05:20:39.267 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:39.256+0000 7fca3f35a700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fca380a7740 0x7fca3808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:20:39.267 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:39.256+0000 7fca3f35a700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fca3807e720 0x7fca3807eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:20:39.267 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:39.256+0000 7fca3f35a700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca380a0710 con 0x7fca3809f2e0 2019-12-05T05:20:39.268 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:39.256+0000 7fca3f35a700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca380a0b00 con 0x7fca380a7740 2019-12-05T05:20:39.268 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:39.256+0000 7fca3f35a700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca3808e5b0 con 0x7fca3807e720 2019-12-05T05:20:39.268 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:39.256+0000 7fca3e0f8700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fca3809f2e0 0x7fca3809f710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:59896/0 (socket says 172.21.15.42:59896) 2019-12-05T05:20:39.268 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:39.256+0000 7fca3e0f8700 1 -- 172.21.15.42:0/1335901904 learned_addr learned my addr 172.21.15.42:0/1335901904 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:20:39.268 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:39.256+0000 7fca3d8f7700 1 --2- 172.21.15.42:0/1335901904 >> v2:172.21.15.40:3300/0 conn(0x7fca380a7740 0x7fca3808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:20:39.268 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:39.256+0000 7fca3d0f6700 1 --2- 172.21.15.42:0/1335901904 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fca3807e720 0x7fca3807eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:20:39.268 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:39.256+0000 7fca3c8f5700 1 -- 172.21.15.42:0/1335901904 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 308648993 0 0) 0x7fca380a0710 con 0x7fca3809f2e0 2019-12-05T05:20:39.269 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:39.256+0000 7fca3c8f5700 1 -- 172.21.15.42:0/1335901904 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca240030b0 con 0x7fca3809f2e0 2019-12-05T05:20:39.269 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:39.260+0000 7fca3d0f6700 1 -- 172.21.15.42:0/1335901904 >> v2:172.21.15.40:3300/0 conn(0x7fca380a7740 msgr2=0x7fca3808df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:20:39.269 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:39.260+0000 7fca3d0f6700 1 --2- 172.21.15.42:0/1335901904 >> v2:172.21.15.40:3300/0 conn(0x7fca380a7740 0x7fca3808df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:20:39.269 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:39.260+0000 7fca3d0f6700 1 -- 172.21.15.42:0/1335901904 >> v1:172.21.15.42:6789/0 conn(0x7fca3809f2e0 legacy=0x7fca3809f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:20:39.269 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:39.260+0000 7fca3c8f5700 1 -- 172.21.15.42:0/1335901904 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1913663988 0 0) 0x7fca240030b0 con 0x7fca3809f2e0 2019-12-05T05:20:39.269 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:39.260+0000 7fca3d0f6700 1 -- 172.21.15.42:0/1335901904 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca3807f0a0 con 0x7fca3807e720 2019-12-05T05:20:39.270 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:39.260+0000 7fca3d8f7700 1 --2- 172.21.15.42:0/1335901904 >> v2:172.21.15.40:3300/0 conn(0x7fca380a7740 0x7fca3808df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:20:39.270 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:39.260+0000 7fca3d0f6700 1 --2- 172.21.15.42:0/1335901904 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fca3807e720 0x7fca3807eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fca2c008790 tx=0x7fca30001d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:39.270 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:39.260+0000 7fca3d0f6700 1 --2- 172.21.15.42:0/1335901904 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fca3807e720 0x7fca3807eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fca2c008790 tx=0x7fca30001d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:39.270 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:39.260+0000 7fca3d0f6700 1 --2- 172.21.15.42:0/1335901904 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fca3807e720 0x7fca3807eb50 secure :-1 s=READY pgs=142 cs=0 l=1 rx=0x7fca2c008790 tx=0x7fca30001d50).ready entity=mon.2 client_cookie=769d601351a9820d server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:20:39.270 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:39.260+0000 7fca3d0f6700 1 --2- 172.21.15.42:0/1335901904 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fca3807e720 0x7fca3807eb50 secure :-1 s=THROTTLE_DONE pgs=142 cs=0 l=1 rx=0x7fca2c008790 tx=0x7fca30001d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:39.270 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:39.260+0000 7fca3c8f5700 1 -- 172.21.15.42:0/1335901904 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fca30001ad0 con 0x7fca3807e720 2019-12-05T05:20:39.270 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:39.260+0000 7fca3d0f6700 1 --2- 172.21.15.42:0/1335901904 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fca3807e720 0x7fca3807eb50 secure :-1 s=THROTTLE_DONE pgs=142 cs=0 l=1 rx=0x7fca2c008790 tx=0x7fca30001d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:39.271 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:39.260+0000 7fca3d0f6700 1 --2- 172.21.15.42:0/1335901904 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fca3807e720 0x7fca3807eb50 secure :-1 s=THROTTLE_DONE pgs=142 cs=0 l=1 rx=0x7fca2c008790 tx=0x7fca30001d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:39.271 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:39.260+0000 7fca3c8f5700 1 -- 172.21.15.42:0/1335901904 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca300018e0 con 0x7fca3807e720 2019-12-05T05:20:39.271 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:39.260+0000 7fca3c8f5700 1 -- 172.21.15.42:0/1335901904 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fca2c0113e0 con 0x7fca3807e720 2019-12-05T05:20:39.271 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:39.260+0000 7fca3f35a700 1 -- 172.21.15.42:0/1335901904 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fca3807e720 msgr2=0x7fca3807eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:20:39.271 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:39.260+0000 7fca3f35a700 1 --2- 172.21.15.42:0/1335901904 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fca3807e720 0x7fca3807eb50 secure :-1 s=READY pgs=142 cs=0 l=1 rx=0x7fca2c008790 tx=0x7fca30001d50).stop 2019-12-05T05:20:39.271 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:39.260+0000 7fca3f35a700 1 -- 172.21.15.42:0/1335901904 shutdown_connections 2019-12-05T05:20:39.271 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:39.260+0000 7fca3f35a700 1 --2- 172.21.15.42:0/1335901904 >> v2:172.21.15.40:3300/0 conn(0x7fca380a7740 0x7fca3808df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:20:39.272 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:39.260+0000 7fca3f35a700 1 --2- 172.21.15.42:0/1335901904 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fca3807e720 0x7fca3807eb50 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:20:39.272 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:39.260+0000 7fca3f35a700 1 -- 172.21.15.42:0/1335901904 shutdown_connections 2019-12-05T05:20:39.272 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:39.260+0000 7fca3f35a700 1 -- 172.21.15.42:0/1335901904 wait complete. 2019-12-05T05:20:39.272 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:39.260+0000 7fca3f35a700 1 -- 172.21.15.42:0/1335901904 >> 172.21.15.42:0/1335901904 conn(0x7fca380aabe0 msgr2=0x7fca380a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:20:39.276 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:39.264+0000 7fca3f35a700 1 Processor -- start 2019-12-05T05:20:39.276 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:39.264+0000 7fca3f35a700 1 -- start start 2019-12-05T05:20:39.277 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:39.264+0000 7fca3f35a700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fca3811feb0 0x7fca3811b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:20:39.277 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:39.264+0000 7fca3f35a700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fca3811b920 0x7fca3811bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:20:39.277 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:39.264+0000 7fca3f35a700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca38131200 con 0x7fca381197c0 2019-12-05T05:20:39.277 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:39.264+0000 7fca3f35a700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca38119cc0 con 0x7fca3811b920 2019-12-05T05:20:39.277 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:39.264+0000 7fca3f35a700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca381218c0 con 0x7fca3811feb0 2019-12-05T05:20:39.278 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:39.264+0000 7fca3d8f7700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fca381197c0 0x7fca38118350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:59902/0 (socket says 172.21.15.42:59902) 2019-12-05T05:20:39.278 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:39.264+0000 7fca3d8f7700 1 -- 172.21.15.42:0/2751398579 learned_addr learned my addr 172.21.15.42:0/2751398579 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:20:39.278 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:39.264+0000 7fca3e0f8700 1 --2- 172.21.15.42:0/2751398579 >> v2:172.21.15.42:3300/0 conn(0x7fca3811b920 0x7fca3811bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:20:39.278 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:39.264+0000 7fca3d0f6700 1 --2- 172.21.15.42:0/2751398579 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fca3811feb0 0x7fca3811b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:20:39.279 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:39.264+0000 7fca2a7fc700 1 -- 172.21.15.42:0/2751398579 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 609887183 0 0) 0x7fca38131200 con 0x7fca381197c0 2019-12-05T05:20:39.279 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:39.264+0000 7fca2a7fc700 1 -- 172.21.15.42:0/2751398579 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca200030b0 con 0x7fca381197c0 2019-12-05T05:20:39.279 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:39.268+0000 7fca3e0f8700 1 -- 172.21.15.42:0/2751398579 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fca3811feb0 msgr2=0x7fca3811b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:20:39.279 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:39.268+0000 7fca3e0f8700 1 --2- 172.21.15.42:0/2751398579 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fca3811feb0 0x7fca3811b3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:20:39.279 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:39.268+0000 7fca3e0f8700 1 -- 172.21.15.42:0/2751398579 >> v1:172.21.15.42:6789/0 conn(0x7fca381197c0 legacy=0x7fca38118350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:20:39.279 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:39.268+0000 7fca3e0f8700 1 -- 172.21.15.42:0/2751398579 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca38116ed0 con 0x7fca3811b920 2019-12-05T05:20:39.281 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:39.268+0000 7fca3d0f6700 1 --2- 172.21.15.42:0/2751398579 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fca3811feb0 0x7fca3811b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:20:39.281 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:39.268+0000 7fca3e0f8700 1 --2- 172.21.15.42:0/2751398579 >> v2:172.21.15.42:3300/0 conn(0x7fca3811b920 0x7fca3811bd50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fca30001dd0 tx=0x7fca30010c40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:39.281 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:39.268+0000 7fca3e0f8700 1 --2- 172.21.15.42:0/2751398579 >> v2:172.21.15.42:3300/0 conn(0x7fca3811b920 0x7fca3811bd50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fca30001dd0 tx=0x7fca30010c40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:39.281 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:39.268+0000 7fca3e0f8700 1 --2- 172.21.15.42:0/2751398579 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fca3811b920 0x7fca3811bd50 secure :-1 s=READY pgs=685 cs=0 l=1 rx=0x7fca30001dd0 tx=0x7fca30010c40).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:20:39.281 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:39.268+0000 7fca3e0f8700 1 --2- 172.21.15.42:0/2751398579 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fca3811b920 0x7fca3811bd50 secure :-1 s=THROTTLE_DONE pgs=685 cs=0 l=1 rx=0x7fca30001dd0 tx=0x7fca30010c40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:39.282 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:39.268+0000 7fca3e0f8700 1 --2- 172.21.15.42:0/2751398579 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fca3811b920 0x7fca3811bd50 secure :-1 s=THROTTLE_DONE pgs=685 cs=0 l=1 rx=0x7fca30001dd0 tx=0x7fca30010c40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:39.282 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:39.268+0000 7fca2a7fc700 1 -- 172.21.15.42:0/2751398579 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fca30004b90 con 0x7fca3811b920 2019-12-05T05:20:39.282 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:39.268+0000 7fca3e0f8700 1 --2- 172.21.15.42:0/2751398579 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fca3811b920 0x7fca3811bd50 secure :-1 s=THROTTLE_DONE pgs=685 cs=0 l=1 rx=0x7fca30001dd0 tx=0x7fca30010c40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:39.282 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:39.268+0000 7fca2a7fc700 1 -- 172.21.15.42:0/2751398579 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca300047b0 con 0x7fca3811b920 2019-12-05T05:20:39.282 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:39.268+0000 7fca3f35a700 1 -- 172.21.15.42:0/2751398579 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca38118b50 con 0x7fca3811b920 2019-12-05T05:20:39.282 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:39.268+0000 7fca2a7fc700 1 -- 172.21.15.42:0/2751398579 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fca300127f0 con 0x7fca3811b920 2019-12-05T05:20:39.282 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:39.268+0000 7fca3f35a700 1 -- 172.21.15.42:0/2751398579 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca38120c50 con 0x7fca3811b920 2019-12-05T05:20:39.283 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:39.268+0000 7fca3e0f8700 1 --2- 172.21.15.42:0/2751398579 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fca3811b920 0x7fca3811bd50 secure :-1 s=THROTTLE_DONE pgs=685 cs=0 l=1 rx=0x7fca30001dd0 tx=0x7fca30010c40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:39.283 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:39.268+0000 7fca2a7fc700 1 -- 172.21.15.42:0/2751398579 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fca30012ac0 con 0x7fca3811b920 2019-12-05T05:20:39.286 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:39.268+0000 7fca3e0f8700 1 --2- 172.21.15.42:0/2751398579 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fca3811b920 0x7fca3811bd50 secure :-1 s=THROTTLE_DONE pgs=685 cs=0 l=1 rx=0x7fca30001dd0 tx=0x7fca30010c40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:39.286 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:39.268+0000 7fca2a7fc700 1 --2- 172.21.15.42:0/2751398579 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fca2003b270 0x7fca2003d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:20:39.286 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:39.268+0000 7fca2a7fc700 1 -- 172.21.15.42:0/2751398579 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6977+0+0 (secure 0 0 0) 0x7fca3004b450 con 0x7fca3811b920 2019-12-05T05:20:39.286 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:39.268+0000 7fca3d8f7700 1 --2- 172.21.15.42:0/2751398579 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fca2003b270 0x7fca2003d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:20:39.286 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:39.272+0000 7fca3d8f7700 1 --2- 172.21.15.42:0/2751398579 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fca2003b270 0x7fca2003d780 crc :-1 s=READY pgs=309 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:20:39.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:39.272+0000 7fca3f35a700 1 -- 172.21.15.42:0/2751398579 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca3805d110 con 0x7fca3811b920 2019-12-05T05:20:39.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:39.280+0000 7fca3e0f8700 1 --2- 172.21.15.42:0/2751398579 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fca3811b920 0x7fca3811bd50 secure :-1 s=THROTTLE_DONE pgs=685 cs=0 l=1 rx=0x7fca30001dd0 tx=0x7fca30010c40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:39.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:39.280+0000 7fca2a7fc700 1 -- 172.21.15.42:0/2751398579 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fca30012dd0 con 0x7fca3811b920 2019-12-05T05:20:39.461 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:39.456+0000 7fca3f35a700 1 -- 172.21.15.42:0/2751398579 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fca3805b8c0 con 0x7fca3811b920 2019-12-05T05:20:39.463 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:39.456+0000 7fca3e0f8700 1 --2- 172.21.15.42:0/2751398579 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fca3811b920 0x7fca3811bd50 secure :-1 s=THROTTLE_DONE pgs=685 cs=0 l=1 rx=0x7fca30001dd0 tx=0x7fca30010c40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:39.463 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:39.456+0000 7fca2a7fc700 1 -- 172.21.15.42:0/2751398579 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+2108 (secure 0 0 0) 0x7fca300324d0 con 0x7fca3811b920 2019-12-05T05:20:39.463 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 88 2019-12-05T05:20:39.464 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:20:39.464 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":88,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5066,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/3061061478","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3061061478},{"type":"v1","addr":"172.21.15.40:6833","nonce":3061061478}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":88}],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:18:52.963069+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5833},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5833":{"gid":5833,"name":"a","rank":0,"incarnation":84,"state":"up:active","state_seq":25,"addr":"172.21.15.42:6835/1414371821","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1414371821},{"type":"v1","addr":"172.21.15.42:6835","nonce":1414371821}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:20:39.466 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:39.460+0000 7fca3f35a700 1 -- 172.21.15.42:0/2751398579 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fca2003b270 msgr2=0x7fca2003d780 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:20:39.466 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:39.460+0000 7fca3f35a700 1 --2- 172.21.15.42:0/2751398579 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fca2003b270 0x7fca2003d780 crc :-1 s=READY pgs=309 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:20:39.466 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:39.460+0000 7fca3f35a700 1 -- 172.21.15.42:0/2751398579 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fca3811b920 msgr2=0x7fca3811bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:20:39.466 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:39.460+0000 7fca3f35a700 1 --2- 172.21.15.42:0/2751398579 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fca3811b920 0x7fca3811bd50 secure :-1 s=READY pgs=685 cs=0 l=1 rx=0x7fca30001dd0 tx=0x7fca30010c40).stop 2019-12-05T05:20:39.467 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:39.460+0000 7fca3f35a700 1 -- 172.21.15.42:0/2751398579 shutdown_connections 2019-12-05T05:20:39.467 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:39.460+0000 7fca3f35a700 1 --2- 172.21.15.42:0/2751398579 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fca2003b270 0x7fca2003d780 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:20:39.467 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:39.460+0000 7fca3f35a700 1 --2- 172.21.15.42:0/2751398579 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fca3811b920 0x7fca3811bd50 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:20:39.467 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:39.460+0000 7fca3f35a700 1 --2- 172.21.15.42:0/2751398579 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fca3811feb0 0x7fca3811b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:20:39.473 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:39.460+0000 7fca3f35a700 1 -- 172.21.15.42:0/2751398579 shutdown_connections 2019-12-05T05:20:39.473 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:39.460+0000 7fca3f35a700 1 -- 172.21.15.42:0/2751398579 wait complete. 2019-12-05T05:20:39.473 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:39.460+0000 7fca3f35a700 1 -- 172.21.15.42:0/2751398579 >> 172.21.15.42:0/2751398579 conn(0x7fca3807e4f0 msgr2=0x7fca380a9640 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:20:39.475 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T05:20:41.475 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:20:41.583 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:41.571+0000 7ff89903e700 1 Processor -- start 2019-12-05T05:20:41.583 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:41.571+0000 7ff89903e700 1 -- start start 2019-12-05T05:20:41.583 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:41.571+0000 7ff89903e700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff89409fae0 0x7ff89414fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:20:41.584 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:41.571+0000 7ff89903e700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8940a0710 con 0x7ff8940a7740 2019-12-05T05:20:41.584 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:41.571+0000 7ff89903e700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff89407ef20 con 0x7ff89408df80 2019-12-05T05:20:41.584 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:41.571+0000 7ff89903e700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8940a0b00 con 0x7ff89409fae0 2019-12-05T05:20:41.584 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:41.571+0000 7ff88ad9d700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7ff89408df80 0x7ff89409f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:59914/0 (socket says 172.21.15.42:59914) 2019-12-05T05:20:41.584 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:41.575+0000 7ff88ad9d700 1 -- 172.21.15.42:0/3863746932 learned_addr learned my addr 172.21.15.42:0/3863746932 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:20:41.584 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:41.575+0000 7ff89883d700 1 --2- 172.21.15.42:0/3863746932 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff89409fae0 0x7ff89414fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:20:41.584 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:41.575+0000 7ff892d9d700 1 -- 172.21.15.42:0/3863746932 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1717999260 0 0) 0x7ff89407ef20 con 0x7ff89408df80 2019-12-05T05:20:41.585 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:41.575+0000 7ff892d9d700 1 -- 172.21.15.42:0/3863746932 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8800030b0 con 0x7ff89408df80 2019-12-05T05:20:41.585 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:41.575+0000 7ff892d9d700 1 -- 172.21.15.42:0/3863746932 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2098681759 0 0) 0x7ff8940a0710 con 0x7ff8940a7740 2019-12-05T05:20:41.585 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:41.575+0000 7ff89883d700 1 -- 172.21.15.42:0/3863746932 >> v1:172.21.15.42:6789/0 conn(0x7ff89408df80 legacy=0x7ff89409f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:20:41.585 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:41.575+0000 7ff89883d700 1 -- 172.21.15.42:0/3863746932 >> v1:172.21.15.40:6789/0 conn(0x7ff8940a7740 legacy=0x7ff89407e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:20:41.585 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:41.575+0000 7ff89883d700 1 -- 172.21.15.42:0/3863746932 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8940a9ba0 con 0x7ff89409fae0 2019-12-05T05:20:41.585 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:41.575+0000 7ff89883d700 1 --2- 172.21.15.42:0/3863746932 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff89409fae0 0x7ff89414fff0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7ff88c0032b0 tx=0x7ff88c00ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:41.585 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:41.575+0000 7ff89883d700 1 --2- 172.21.15.42:0/3863746932 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff89409fae0 0x7ff89414fff0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7ff88c0032b0 tx=0x7ff88c00ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:41.586 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:41.575+0000 7ff89883d700 1 --2- 172.21.15.42:0/3863746932 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff89409fae0 0x7ff89414fff0 secure :-1 s=READY pgs=143 cs=0 l=1 rx=0x7ff88c0032b0 tx=0x7ff88c00ff70).ready entity=mon.2 client_cookie=457914d10aa84546 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:20:41.586 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:41.575+0000 7ff89883d700 1 --2- 172.21.15.42:0/3863746932 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff89409fae0 0x7ff89414fff0 secure :-1 s=THROTTLE_DONE pgs=143 cs=0 l=1 rx=0x7ff88c0032b0 tx=0x7ff88c00ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:41.586 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:41.575+0000 7ff892d9d700 1 -- 172.21.15.42:0/3863746932 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff88c0109b0 con 0x7ff89409fae0 2019-12-05T05:20:41.586 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:41.575+0000 7ff89883d700 1 --2- 172.21.15.42:0/3863746932 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff89409fae0 0x7ff89414fff0 secure :-1 s=THROTTLE_DONE pgs=143 cs=0 l=1 rx=0x7ff88c0032b0 tx=0x7ff88c00ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:41.586 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:41.575+0000 7ff892d9d700 1 -- 172.21.15.42:0/3863746932 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff88c008b50 con 0x7ff89409fae0 2019-12-05T05:20:41.586 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:41.575+0000 7ff89883d700 1 --2- 172.21.15.42:0/3863746932 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff89409fae0 0x7ff89414fff0 secure :-1 s=THROTTLE_DONE pgs=143 cs=0 l=1 rx=0x7ff88c0032b0 tx=0x7ff88c00ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:41.586 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:41.575+0000 7ff892d9d700 1 -- 172.21.15.42:0/3863746932 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff88c0110c0 con 0x7ff89409fae0 2019-12-05T05:20:41.587 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:41.575+0000 7ff89903e700 1 -- 172.21.15.42:0/3863746932 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff89409fae0 msgr2=0x7ff89414fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:20:41.587 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:41.575+0000 7ff89903e700 1 --2- 172.21.15.42:0/3863746932 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff89409fae0 0x7ff89414fff0 secure :-1 s=READY pgs=143 cs=0 l=1 rx=0x7ff88c0032b0 tx=0x7ff88c00ff70).stop 2019-12-05T05:20:41.587 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:41.575+0000 7ff89903e700 1 -- 172.21.15.42:0/3863746932 shutdown_connections 2019-12-05T05:20:41.587 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:41.575+0000 7ff89903e700 1 --2- 172.21.15.42:0/3863746932 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff89409fae0 0x7ff89414fff0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:20:41.587 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:41.575+0000 7ff89903e700 1 -- 172.21.15.42:0/3863746932 shutdown_connections 2019-12-05T05:20:41.587 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:41.575+0000 7ff89903e700 1 -- 172.21.15.42:0/3863746932 wait complete. 2019-12-05T05:20:41.587 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:41.575+0000 7ff89903e700 1 -- 172.21.15.42:0/3863746932 >> 172.21.15.42:0/3863746932 conn(0x7ff8940aabe0 msgr2=0x7ff8940a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:20:41.591 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:41.579+0000 7ff89903e700 1 Processor -- start 2019-12-05T05:20:41.592 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:41.579+0000 7ff89903e700 1 -- start start 2019-12-05T05:20:41.592 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:41.579+0000 7ff89903e700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff89412e830 0x7ff8940722c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:20:41.592 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:41.579+0000 7ff89903e700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8941213b0 con 0x7ff894144c30 2019-12-05T05:20:41.592 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:41.579+0000 7ff89903e700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff894120a40 con 0x7ff894072810 2019-12-05T05:20:41.592 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:41.579+0000 7ff89903e700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff894120e70 con 0x7ff89412e830 2019-12-05T05:20:41.592 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:41.579+0000 7ff89883d700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7ff894072810 0x7ff894074c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:59922/0 (socket says 172.21.15.42:59922) 2019-12-05T05:20:41.593 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:41.579+0000 7ff89883d700 1 -- 172.21.15.42:0/835561325 learned_addr learned my addr 172.21.15.42:0/835561325 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:20:41.593 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:41.579+0000 7ff88ad9d700 1 --2- 172.21.15.42:0/835561325 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff89412e830 0x7ff8940722c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:20:41.593 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:41.583+0000 7ff890d99700 1 -- 172.21.15.42:0/835561325 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4120469989 0 0) 0x7ff894120a40 con 0x7ff894072810 2019-12-05T05:20:41.593 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:41.583+0000 7ff890d99700 1 -- 172.21.15.42:0/835561325 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8780030b0 con 0x7ff894072810 2019-12-05T05:20:41.593 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:41.583+0000 7ff890d99700 1 -- 172.21.15.42:0/835561325 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1340866755 0 0) 0x7ff8941213b0 con 0x7ff894144c30 2019-12-05T05:20:41.593 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:41.583+0000 7ff890d99700 1 -- 172.21.15.42:0/835561325 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff894120a40 con 0x7ff894144c30 2019-12-05T05:20:41.593 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:41.583+0000 7ff88ad9d700 1 -- 172.21.15.42:0/835561325 >> v1:172.21.15.42:6789/0 conn(0x7ff894072810 legacy=0x7ff894074c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:20:41.594 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:41.583+0000 7ff88ad9d700 1 -- 172.21.15.42:0/835561325 >> v1:172.21.15.40:6789/0 conn(0x7ff894144c30 legacy=0x7ff89412e030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:20:41.594 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:41.583+0000 7ff88ad9d700 1 -- 172.21.15.42:0/835561325 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff894119c30 con 0x7ff89412e830 2019-12-05T05:20:41.594 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:41.583+0000 7ff88ad9d700 1 --2- 172.21.15.42:0/835561325 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff89412e830 0x7ff8940722c0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7ff89409f140 tx=0x7ff87c010840).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:41.594 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:41.583+0000 7ff88ad9d700 1 --2- 172.21.15.42:0/835561325 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff89412e830 0x7ff8940722c0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7ff89409f140 tx=0x7ff87c010840).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:41.594 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:41.583+0000 7ff88ad9d700 1 --2- 172.21.15.42:0/835561325 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff89412e830 0x7ff8940722c0 secure :-1 s=READY pgs=144 cs=0 l=1 rx=0x7ff89409f140 tx=0x7ff87c010840).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:20:41.594 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:41.583+0000 7ff88ad9d700 1 --2- 172.21.15.42:0/835561325 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff89412e830 0x7ff8940722c0 secure :-1 s=THROTTLE_DONE pgs=144 cs=0 l=1 rx=0x7ff89409f140 tx=0x7ff87c010840).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:41.597 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:41.583+0000 7ff88ad9d700 1 --2- 172.21.15.42:0/835561325 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff89412e830 0x7ff8940722c0 secure :-1 s=THROTTLE_DONE pgs=144 cs=0 l=1 rx=0x7ff89409f140 tx=0x7ff87c010840).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:41.597 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:41.583+0000 7ff890d99700 1 -- 172.21.15.42:0/835561325 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff87c005c30 con 0x7ff89412e830 2019-12-05T05:20:41.598 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:41.583+0000 7ff88ad9d700 1 --2- 172.21.15.42:0/835561325 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff89412e830 0x7ff8940722c0 secure :-1 s=THROTTLE_DONE pgs=144 cs=0 l=1 rx=0x7ff89409f140 tx=0x7ff87c010840).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:41.598 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:41.583+0000 7ff890d99700 1 -- 172.21.15.42:0/835561325 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff87c005850 con 0x7ff89412e830 2019-12-05T05:20:41.598 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:41.583+0000 7ff890d99700 1 -- 172.21.15.42:0/835561325 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff87c0064a0 con 0x7ff89412e830 2019-12-05T05:20:41.598 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:41.583+0000 7ff89903e700 1 -- 172.21.15.42:0/835561325 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff89408f850 con 0x7ff89412e830 2019-12-05T05:20:41.598 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:41.583+0000 7ff89903e700 1 -- 172.21.15.42:0/835561325 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff894076460 con 0x7ff89412e830 2019-12-05T05:20:41.598 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:41.583+0000 7ff88ad9d700 1 --2- 172.21.15.42:0/835561325 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff89412e830 0x7ff8940722c0 secure :-1 s=THROTTLE_DONE pgs=144 cs=0 l=1 rx=0x7ff89409f140 tx=0x7ff87c010840).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:41.598 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:41.583+0000 7ff890d99700 1 -- 172.21.15.42:0/835561325 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7ff87c0067b0 con 0x7ff89412e830 2019-12-05T05:20:41.599 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:41.583+0000 7ff88ad9d700 1 --2- 172.21.15.42:0/835561325 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff89412e830 0x7ff8940722c0 secure :-1 s=THROTTLE_DONE pgs=144 cs=0 l=1 rx=0x7ff89409f140 tx=0x7ff87c010840).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:41.599 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:41.583+0000 7ff890d99700 1 --2- 172.21.15.42:0/835561325 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff87803b4c0 0x7ff87803d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:20:41.599 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:41.587+0000 7ff890d99700 1 -- 172.21.15.42:0/835561325 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6977+0+0 (secure 0 0 0) 0x7ff87c04af50 con 0x7ff89412e830 2019-12-05T05:20:41.599 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:41.587+0000 7ff89359e700 1 --2- 172.21.15.42:0/835561325 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff87803b4c0 0x7ff87803d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:20:41.600 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:41.587+0000 7ff89359e700 1 --2- 172.21.15.42:0/835561325 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff87803b4c0 0x7ff87803d9d0 crc :-1 s=READY pgs=310 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:20:41.600 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:41.587+0000 7ff89903e700 1 -- 172.21.15.42:0/835561325 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff894120c70 con 0x7ff89412e830 2019-12-05T05:20:41.600 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:41.591+0000 7ff88ad9d700 1 --2- 172.21.15.42:0/835561325 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff89412e830 0x7ff8940722c0 secure :-1 s=THROTTLE_DONE pgs=144 cs=0 l=1 rx=0x7ff89409f140 tx=0x7ff87c010840).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:41.600 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:41.591+0000 7ff890d99700 1 -- 172.21.15.42:0/835561325 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7ff87c031fd0 con 0x7ff89412e830 2019-12-05T05:20:41.764 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:41.759+0000 7ff89903e700 1 -- 172.21.15.42:0/835561325 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff89405d110 con 0x7ff89412e830 2019-12-05T05:20:41.768 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:41.759+0000 7ff88ad9d700 1 --2- 172.21.15.42:0/835561325 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff89412e830 0x7ff8940722c0 secure :-1 s=THROTTLE_DONE pgs=144 cs=0 l=1 rx=0x7ff89409f140 tx=0x7ff87c010840).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:41.768 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:41.759+0000 7ff890d99700 1 -- 172.21.15.42:0/835561325 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+2108 (secure 0 0 0) 0x7ff87c032230 con 0x7ff89412e830 2019-12-05T05:20:41.768 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 88 2019-12-05T05:20:41.768 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:20:41.769 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":88,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5066,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/3061061478","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3061061478},{"type":"v1","addr":"172.21.15.40:6833","nonce":3061061478}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":88}],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:18:52.963069+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5833},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5833":{"gid":5833,"name":"a","rank":0,"incarnation":84,"state":"up:active","state_seq":25,"addr":"172.21.15.42:6835/1414371821","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1414371821},{"type":"v1","addr":"172.21.15.42:6835","nonce":1414371821}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:20:41.771 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:41.763+0000 7ff89903e700 1 -- 172.21.15.42:0/835561325 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff87803b4c0 msgr2=0x7ff87803d9d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:20:41.772 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:41.763+0000 7ff89903e700 1 --2- 172.21.15.42:0/835561325 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff87803b4c0 0x7ff87803d9d0 crc :-1 s=READY pgs=310 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:20:41.773 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:41.763+0000 7ff89903e700 1 -- 172.21.15.42:0/835561325 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff89412e830 msgr2=0x7ff8940722c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:20:41.773 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:41.763+0000 7ff89903e700 1 --2- 172.21.15.42:0/835561325 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff89412e830 0x7ff8940722c0 secure :-1 s=READY pgs=144 cs=0 l=1 rx=0x7ff89409f140 tx=0x7ff87c010840).stop 2019-12-05T05:20:41.774 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:41.763+0000 7ff89903e700 1 -- 172.21.15.42:0/835561325 shutdown_connections 2019-12-05T05:20:41.775 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:41.763+0000 7ff89903e700 1 --2- 172.21.15.42:0/835561325 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff87803b4c0 0x7ff87803d9d0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:20:41.777 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:41.763+0000 7ff89903e700 1 --2- 172.21.15.42:0/835561325 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff89412e830 0x7ff8940722c0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:20:41.777 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:41.763+0000 7ff89903e700 1 -- 172.21.15.42:0/835561325 shutdown_connections 2019-12-05T05:20:41.778 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:41.763+0000 7ff89903e700 1 -- 172.21.15.42:0/835561325 wait complete. 2019-12-05T05:20:41.778 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:41.763+0000 7ff89903e700 1 -- 172.21.15.42:0/835561325 >> 172.21.15.42:0/835561325 conn(0x7ff8940a8f60 msgr2=0x7ff89411c520 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:20:41.781 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T05:20:43.782 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:20:43.893 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:43.879+0000 7f4a10a39700 1 Processor -- start 2019-12-05T05:20:43.893 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:43.879+0000 7f4a10a39700 1 -- start start 2019-12-05T05:20:43.893 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:43.879+0000 7f4a10a39700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4a0c0a7740 0x7f4a0c08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:20:43.893 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:43.879+0000 7f4a10a39700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f4a0c07e720 0x7f4a0c07eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:20:43.893 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:43.879+0000 7f4a10a39700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a0c0a0710 con 0x7f4a0c09f2e0 2019-12-05T05:20:43.893 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:43.879+0000 7f4a10a39700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a0c0a0b00 con 0x7f4a0c07e720 2019-12-05T05:20:43.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:43.879+0000 7f4a10a39700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a0c08e5b0 con 0x7f4a0c0a7740 2019-12-05T05:20:43.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:43.879+0000 7f4a0a59c700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f4a0c07e720 0x7f4a0c07eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:20:43.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:43.879+0000 7f4a0ad9d700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4a0c0a7740 0x7f4a0c08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:20:43.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:43.879+0000 7f4a0b59e700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f4a0c09f2e0 0x7f4a0c09f710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:41138/0 (socket says 172.21.15.42:41138) 2019-12-05T05:20:43.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:43.879+0000 7f4a0b59e700 1 -- 172.21.15.42:0/100136509 learned_addr learned my addr 172.21.15.42:0/100136509 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:20:43.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:43.879+0000 7f4a0ad9d700 1 -- 172.21.15.42:0/100136509 >> v2:172.21.15.40:3300/0 conn(0x7f4a0c07e720 msgr2=0x7f4a0c07eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:20:43.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:43.879+0000 7f4a0ad9d700 1 --2- 172.21.15.42:0/100136509 >> v2:172.21.15.40:3300/0 conn(0x7f4a0c07e720 0x7f4a0c07eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:20:43.895 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:43.879+0000 7f4a0ad9d700 1 -- 172.21.15.42:0/100136509 >> v1:172.21.15.40:6789/0 conn(0x7f4a0c09f2e0 legacy=0x7f4a0c09f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:20:43.895 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:43.879+0000 7f4a0ad9d700 1 -- 172.21.15.42:0/100136509 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a0c07f0a0 con 0x7f4a0c0a7740 2019-12-05T05:20:43.895 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:43.879+0000 7f4a0ad9d700 1 --2- 172.21.15.42:0/100136509 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4a0c0a7740 0x7f4a0c08df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f4a04008b10 tx=0x7f4a0400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:43.895 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:43.879+0000 7f4a0ad9d700 1 --2- 172.21.15.42:0/100136509 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4a0c0a7740 0x7f4a0c08df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f4a04008b10 tx=0x7f4a0400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:43.895 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:43.879+0000 7f4a0ad9d700 1 --2- 172.21.15.42:0/100136509 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4a0c0a7740 0x7f4a0c08df80 secure :-1 s=READY pgs=145 cs=0 l=1 rx=0x7f4a04008b10 tx=0x7f4a0400fff0).ready entity=mon.2 client_cookie=6d9914ad03d639ac server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:20:43.895 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:43.879+0000 7f4a0ad9d700 1 --2- 172.21.15.42:0/100136509 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4a0c0a7740 0x7f4a0c08df80 secure :-1 s=THROTTLE_DONE pgs=145 cs=0 l=1 rx=0x7f4a04008b10 tx=0x7f4a0400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:43.895 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:43.879+0000 7f4a0ad9d700 1 --2- 172.21.15.42:0/100136509 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4a0c0a7740 0x7f4a0c08df80 secure :-1 s=THROTTLE_DONE pgs=145 cs=0 l=1 rx=0x7f4a04008b10 tx=0x7f4a0400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:43.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:43.879+0000 7f4a09d9b700 1 -- 172.21.15.42:0/100136509 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4a04011a50 con 0x7f4a0c0a7740 2019-12-05T05:20:43.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:43.879+0000 7f4a0ad9d700 1 --2- 172.21.15.42:0/100136509 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4a0c0a7740 0x7f4a0c08df80 secure :-1 s=THROTTLE_DONE pgs=145 cs=0 l=1 rx=0x7f4a04008b10 tx=0x7f4a0400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:43.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:43.879+0000 7f4a09d9b700 1 -- 172.21.15.42:0/100136509 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a04011670 con 0x7f4a0c0a7740 2019-12-05T05:20:43.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:43.879+0000 7f4a09d9b700 1 -- 172.21.15.42:0/100136509 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4a0400a350 con 0x7f4a0c0a7740 2019-12-05T05:20:43.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:43.879+0000 7f4a10a39700 1 -- 172.21.15.42:0/100136509 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4a0c0a7740 msgr2=0x7f4a0c08df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:20:43.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:43.879+0000 7f4a10a39700 1 --2- 172.21.15.42:0/100136509 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4a0c0a7740 0x7f4a0c08df80 secure :-1 s=READY pgs=145 cs=0 l=1 rx=0x7f4a04008b10 tx=0x7f4a0400fff0).stop 2019-12-05T05:20:43.897 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:43.879+0000 7f4a10a39700 1 -- 172.21.15.42:0/100136509 shutdown_connections 2019-12-05T05:20:43.897 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:43.879+0000 7f4a10a39700 1 --2- 172.21.15.42:0/100136509 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4a0c0a7740 0x7f4a0c08df80 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:20:43.897 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:43.879+0000 7f4a10a39700 1 --2- 172.21.15.42:0/100136509 >> v2:172.21.15.40:3300/0 conn(0x7f4a0c07e720 0x7f4a0c07eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:20:43.897 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:43.879+0000 7f4a10a39700 1 -- 172.21.15.42:0/100136509 shutdown_connections 2019-12-05T05:20:43.897 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:43.879+0000 7f4a10a39700 1 -- 172.21.15.42:0/100136509 wait complete. 2019-12-05T05:20:43.897 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:43.879+0000 7f4a10a39700 1 -- 172.21.15.42:0/100136509 >> 172.21.15.42:0/100136509 conn(0x7f4a0c0aabe0 msgr2=0x7f4a0c0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:20:43.897 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:43.883+0000 7f4a10a39700 1 Processor -- start 2019-12-05T05:20:43.898 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:43.883+0000 7f4a10a39700 1 -- start start 2019-12-05T05:20:43.898 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:43.883+0000 7f4a10a39700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f4a0c130f80 0x7f4a0c1171a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:20:43.898 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:43.883+0000 7f4a10a39700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a0c116500 con 0x7f4a0c11a690 2019-12-05T05:20:43.898 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:43.883+0000 7f4a10a39700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a0c116730 con 0x7f4a0c11c350 2019-12-05T05:20:43.898 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:43.883+0000 7f4a10a39700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a0c117920 con 0x7f4a0c130f80 2019-12-05T05:20:43.898 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:43.883+0000 7f4a0a59c700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f4a0c11c350 0x7f4a0c130780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:59932/0 (socket says 172.21.15.42:59932) 2019-12-05T05:20:43.898 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:43.883+0000 7f4a0a59c700 1 -- 172.21.15.42:0/742992729 learned_addr learned my addr 172.21.15.42:0/742992729 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:20:43.899 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:43.883+0000 7f4a0b59e700 1 --2- 172.21.15.42:0/742992729 >> v2:172.21.15.40:3300/0 conn(0x7f4a0c130f80 0x7f4a0c1171a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:20:43.899 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:43.883+0000 7f49fb7fe700 1 -- 172.21.15.42:0/742992729 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 494397182 0 0) 0x7f4a0c116730 con 0x7f4a0c11c350 2019-12-05T05:20:43.899 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:43.883+0000 7f49fb7fe700 1 -- 172.21.15.42:0/742992729 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49f00030b0 con 0x7f4a0c11c350 2019-12-05T05:20:43.899 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:43.883+0000 7f49fb7fe700 1 -- 172.21.15.42:0/742992729 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1828642040 0 0) 0x7f4a0c116500 con 0x7f4a0c11a690 2019-12-05T05:20:43.899 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:43.883+0000 7f49fb7fe700 1 -- 172.21.15.42:0/742992729 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a0c116730 con 0x7f4a0c11a690 2019-12-05T05:20:43.899 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:43.883+0000 7f4a0b59e700 1 -- 172.21.15.42:0/742992729 >> v1:172.21.15.42:6789/0 conn(0x7f4a0c11c350 legacy=0x7f4a0c130780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:20:43.899 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:43.883+0000 7f49fb7fe700 1 -- 172.21.15.42:0/742992729 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 108235818 0 0) 0x7f49f00030b0 con 0x7f4a0c11c350 2019-12-05T05:20:43.900 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:43.883+0000 7f4a0b59e700 1 -- 172.21.15.42:0/742992729 >> v1:172.21.15.40:6789/0 conn(0x7f4a0c11a690 legacy=0x7f4a0c11bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:20:43.900 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:43.883+0000 7f4a0b59e700 1 -- 172.21.15.42:0/742992729 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a0c13ab80 con 0x7f4a0c130f80 2019-12-05T05:20:43.900 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:43.883+0000 7f4a0b59e700 1 --2- 172.21.15.42:0/742992729 >> v2:172.21.15.40:3300/0 conn(0x7f4a0c130f80 0x7f4a0c1171a0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f4a00007900 tx=0x7f4a00002f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:43.900 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:43.883+0000 7f4a0b59e700 1 --2- 172.21.15.42:0/742992729 >> v2:172.21.15.40:3300/0 conn(0x7f4a0c130f80 0x7f4a0c1171a0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f4a00007900 tx=0x7f4a00002f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:43.900 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:43.883+0000 7f4a0b59e700 1 --2- 172.21.15.42:0/742992729 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4a0c130f80 0x7f4a0c1171a0 secure :-1 s=READY pgs=555 cs=0 l=1 rx=0x7f4a00007900 tx=0x7f4a00002f70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:20:43.900 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:43.883+0000 7f4a0b59e700 1 --2- 172.21.15.42:0/742992729 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4a0c130f80 0x7f4a0c1171a0 secure :-1 s=THROTTLE_DONE pgs=555 cs=0 l=1 rx=0x7f4a00007900 tx=0x7f4a00002f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:43.901 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:43.887+0000 7f49fb7fe700 1 -- 172.21.15.42:0/742992729 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4a00001490 con 0x7f4a0c130f80 2019-12-05T05:20:43.901 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:43.887+0000 7f4a0b59e700 1 --2- 172.21.15.42:0/742992729 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4a0c130f80 0x7f4a0c1171a0 secure :-1 s=THROTTLE_DONE pgs=555 cs=0 l=1 rx=0x7f4a00007900 tx=0x7f4a00002f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:43.901 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:43.887+0000 7f4a10a39700 1 -- 172.21.15.42:0/742992729 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a0c138b60 con 0x7f4a0c130f80 2019-12-05T05:20:43.901 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:43.887+0000 7f4a0b59e700 1 --2- 172.21.15.42:0/742992729 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4a0c130f80 0x7f4a0c1171a0 secure :-1 s=THROTTLE_DONE pgs=555 cs=0 l=1 rx=0x7f4a00007900 tx=0x7f4a00002f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:43.901 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:43.887+0000 7f49fb7fe700 1 -- 172.21.15.42:0/742992729 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a000012a0 con 0x7f4a0c130f80 2019-12-05T05:20:43.901 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:43.887+0000 7f49fb7fe700 1 -- 172.21.15.42:0/742992729 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f49fc0116a0 con 0x7f4a0c130f80 2019-12-05T05:20:43.902 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:43.887+0000 7f4a10a39700 1 -- 172.21.15.42:0/742992729 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a0c139060 con 0x7f4a0c130f80 2019-12-05T05:20:43.902 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:43.887+0000 7f4a0b59e700 1 --2- 172.21.15.42:0/742992729 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4a0c130f80 0x7f4a0c1171a0 secure :-1 s=THROTTLE_DONE pgs=555 cs=0 l=1 rx=0x7f4a00007900 tx=0x7f4a00002f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:43.904 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:43.887+0000 7f49fb7fe700 1 -- 172.21.15.42:0/742992729 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f49fc011cf0 con 0x7f4a0c130f80 2019-12-05T05:20:43.905 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:43.887+0000 7f4a0b59e700 1 --2- 172.21.15.42:0/742992729 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4a0c130f80 0x7f4a0c1171a0 secure :-1 s=THROTTLE_DONE pgs=555 cs=0 l=1 rx=0x7f4a00007900 tx=0x7f4a00002f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:43.905 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:43.887+0000 7f49fb7fe700 1 --2- 172.21.15.42:0/742992729 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f49f003b400 0x7f49f003d910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:20:43.906 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:43.887+0000 7f49fb7fe700 1 -- 172.21.15.42:0/742992729 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 7063+0+0 (secure 0 0 0) 0x7f49fc04a930 con 0x7f4a0c130f80 2019-12-05T05:20:43.906 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:43.887+0000 7f4a0ad9d700 1 --2- 172.21.15.42:0/742992729 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f49f003b400 0x7f49f003d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:20:43.906 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:43.887+0000 7f4a0ad9d700 1 --2- 172.21.15.42:0/742992729 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f49f003b400 0x7f49f003d910 crc :-1 s=READY pgs=311 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:20:43.907 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:43.895+0000 7f4a10a39700 1 -- 172.21.15.42:0/742992729 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a0c05d110 con 0x7f4a0c130f80 2019-12-05T05:20:43.909 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:43.899+0000 7f4a0b59e700 1 --2- 172.21.15.42:0/742992729 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4a0c130f80 0x7f4a0c1171a0 secure :-1 s=THROTTLE_DONE pgs=555 cs=0 l=1 rx=0x7f4a00007900 tx=0x7f4a00002f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:43.909 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:43.899+0000 7f49fb7fe700 1 -- 172.21.15.42:0/742992729 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f49fc031990 con 0x7f4a0c130f80 2019-12-05T05:20:44.072 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:44.067+0000 7f4a10a39700 1 -- 172.21.15.42:0/742992729 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4a0c05b8c0 con 0x7f4a0c130f80 2019-12-05T05:20:44.073 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:44.067+0000 7f4a0b59e700 1 --2- 172.21.15.42:0/742992729 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4a0c130f80 0x7f4a0c1171a0 secure :-1 s=THROTTLE_DONE pgs=555 cs=0 l=1 rx=0x7f4a00007900 tx=0x7f4a00002f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:44.073 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:44.067+0000 7f49fb7fe700 1 -- 172.21.15.42:0/742992729 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+1771 (secure 0 0 0) 0x7f49fc031bf0 con 0x7f4a0c130f80 2019-12-05T05:20:44.073 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 89 2019-12-05T05:20:44.074 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:20:44.074 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":89,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:20:43.496004+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5066},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5066":{"gid":5066,"name":"b","rank":0,"incarnation":89,"state":"up:replay","state_seq":2,"addr":"172.21.15.40:6833/3061061478","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3061061478},{"type":"v1","addr":"172.21.15.40:6833","nonce":3061061478}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:20:44.076 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:44.067+0000 7f4a10a39700 1 -- 172.21.15.42:0/742992729 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f49f003b400 msgr2=0x7f49f003d910 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:20:44.076 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:44.067+0000 7f4a10a39700 1 --2- 172.21.15.42:0/742992729 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f49f003b400 0x7f49f003d910 crc :-1 s=READY pgs=311 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:20:44.076 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:44.067+0000 7f4a10a39700 1 -- 172.21.15.42:0/742992729 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4a0c130f80 msgr2=0x7f4a0c1171a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:20:44.076 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:44.067+0000 7f4a10a39700 1 --2- 172.21.15.42:0/742992729 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4a0c130f80 0x7f4a0c1171a0 secure :-1 s=READY pgs=555 cs=0 l=1 rx=0x7f4a00007900 tx=0x7f4a00002f70).stop 2019-12-05T05:20:44.077 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:44.071+0000 7f4a10a39700 1 -- 172.21.15.42:0/742992729 shutdown_connections 2019-12-05T05:20:44.077 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:44.071+0000 7f4a10a39700 1 --2- 172.21.15.42:0/742992729 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f49f003b400 0x7f49f003d910 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:20:44.077 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:44.071+0000 7f4a10a39700 1 --2- 172.21.15.42:0/742992729 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4a0c130f80 0x7f4a0c1171a0 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:20:44.077 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:44.071+0000 7f4a10a39700 1 -- 172.21.15.42:0/742992729 shutdown_connections 2019-12-05T05:20:44.077 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:44.071+0000 7f4a10a39700 1 -- 172.21.15.42:0/742992729 wait complete. 2019-12-05T05:20:44.077 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:44.071+0000 7f4a10a39700 1 -- 172.21.15.42:0/742992729 >> 172.21.15.42:0/742992729 conn(0x7f4a0c066fe0 msgr2=0x7f4a0c0a9800 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:20:44.086 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a down, removed from mdsmap 2019-12-05T05:20:44.086 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2019-12-05T05:20:44.086 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:20:44.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:44.175+0000 7fdac101f700 1 Processor -- start 2019-12-05T05:20:44.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:44.175+0000 7fdac101f700 1 -- start start 2019-12-05T05:20:44.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:44.175+0000 7fdac101f700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fdabc09fae0 0x7fdabc14fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:20:44.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:44.175+0000 7fdac101f700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdabc0a0710 con 0x7fdabc08df80 2019-12-05T05:20:44.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:44.175+0000 7fdac101f700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdabc07ef20 con 0x7fdabc0a7740 2019-12-05T05:20:44.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:44.175+0000 7fdac101f700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdabc0a0b00 con 0x7fdabc09fae0 2019-12-05T05:20:44.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:44.175+0000 7fdabb59e700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fdabc0a7740 0x7fdabc07e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:59940/0 (socket says 172.21.15.42:59940) 2019-12-05T05:20:44.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:44.175+0000 7fdabb59e700 1 -- 172.21.15.42:0/854472878 learned_addr learned my addr 172.21.15.42:0/854472878 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:20:44.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:44.179+0000 7fdac081e700 1 --2- 172.21.15.42:0/854472878 >> v2:172.21.15.40:3300/0 conn(0x7fdabc09fae0 0x7fdabc14fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:20:44.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:44.179+0000 7fdabad9d700 1 -- 172.21.15.42:0/854472878 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2067069135 0 0) 0x7fdabc07ef20 con 0x7fdabc0a7740 2019-12-05T05:20:44.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:44.179+0000 7fdabad9d700 1 -- 172.21.15.42:0/854472878 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdaa80030b0 con 0x7fdabc0a7740 2019-12-05T05:20:44.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:44.179+0000 7fdabad9d700 1 -- 172.21.15.42:0/854472878 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 68855509 0 0) 0x7fdabc0a0710 con 0x7fdabc08df80 2019-12-05T05:20:44.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:44.179+0000 7fdabad9d700 1 -- 172.21.15.42:0/854472878 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdabc07ef20 con 0x7fdabc08df80 2019-12-05T05:20:44.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:44.179+0000 7fdac081e700 1 -- 172.21.15.42:0/854472878 >> v1:172.21.15.42:6789/0 conn(0x7fdabc0a7740 legacy=0x7fdabc07e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:20:44.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:44.179+0000 7fdac081e700 1 -- 172.21.15.42:0/854472878 >> v1:172.21.15.40:6789/0 conn(0x7fdabc08df80 legacy=0x7fdabc09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:20:44.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:44.179+0000 7fdac081e700 1 -- 172.21.15.42:0/854472878 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdabc0a9ba0 con 0x7fdabc09fae0 2019-12-05T05:20:44.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:44.179+0000 7fdac081e700 1 --2- 172.21.15.42:0/854472878 >> v2:172.21.15.40:3300/0 conn(0x7fdabc09fae0 0x7fdabc14fff0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fdab4008b10 tx=0x7fdab400ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:44.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:44.179+0000 7fdac081e700 1 --2- 172.21.15.42:0/854472878 >> v2:172.21.15.40:3300/0 conn(0x7fdabc09fae0 0x7fdabc14fff0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fdab4008b10 tx=0x7fdab400ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:44.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:44.179+0000 7fdac081e700 1 --2- 172.21.15.42:0/854472878 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdabc09fae0 0x7fdabc14fff0 secure :-1 s=READY pgs=557 cs=0 l=1 rx=0x7fdab4008b10 tx=0x7fdab400ff70).ready entity=mon.0 client_cookie=d19f62c627a146be server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:20:44.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:44.179+0000 7fdac081e700 1 --2- 172.21.15.42:0/854472878 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdabc09fae0 0x7fdabc14fff0 secure :-1 s=THROTTLE_DONE pgs=557 cs=0 l=1 rx=0x7fdab4008b10 tx=0x7fdab400ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:44.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:44.179+0000 7fdabad9d700 1 -- 172.21.15.42:0/854472878 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdab4010970 con 0x7fdabc09fae0 2019-12-05T05:20:44.193 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:44.179+0000 7fdac081e700 1 --2- 172.21.15.42:0/854472878 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdabc09fae0 0x7fdabc14fff0 secure :-1 s=THROTTLE_DONE pgs=557 cs=0 l=1 rx=0x7fdab4008b10 tx=0x7fdab400ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:44.193 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:44.179+0000 7fdabad9d700 1 -- 172.21.15.42:0/854472878 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdab4008b50 con 0x7fdabc09fae0 2019-12-05T05:20:44.194 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:44.179+0000 7fdac081e700 1 --2- 172.21.15.42:0/854472878 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdabc09fae0 0x7fdabc14fff0 secure :-1 s=THROTTLE_DONE pgs=557 cs=0 l=1 rx=0x7fdab4008b10 tx=0x7fdab400ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:44.194 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:44.179+0000 7fdabad9d700 1 -- 172.21.15.42:0/854472878 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdab40110c0 con 0x7fdabc09fae0 2019-12-05T05:20:44.195 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:44.179+0000 7fdac101f700 1 -- 172.21.15.42:0/854472878 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdabc09fae0 msgr2=0x7fdabc14fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:20:44.195 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:44.179+0000 7fdac101f700 1 --2- 172.21.15.42:0/854472878 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdabc09fae0 0x7fdabc14fff0 secure :-1 s=READY pgs=557 cs=0 l=1 rx=0x7fdab4008b10 tx=0x7fdab400ff70).stop 2019-12-05T05:20:44.195 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:44.179+0000 7fdac101f700 1 -- 172.21.15.42:0/854472878 shutdown_connections 2019-12-05T05:20:44.195 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:44.179+0000 7fdac101f700 1 --2- 172.21.15.42:0/854472878 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdabc09fae0 0x7fdabc14fff0 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:20:44.195 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:44.179+0000 7fdac101f700 1 -- 172.21.15.42:0/854472878 shutdown_connections 2019-12-05T05:20:44.195 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:44.179+0000 7fdac101f700 1 -- 172.21.15.42:0/854472878 wait complete. 2019-12-05T05:20:44.196 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:44.179+0000 7fdac101f700 1 -- 172.21.15.42:0/854472878 >> 172.21.15.42:0/854472878 conn(0x7fdabc0aabe0 msgr2=0x7fdabc0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:20:44.196 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:44.183+0000 7fdac101f700 1 Processor -- start 2019-12-05T05:20:44.196 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:44.183+0000 7fdac101f700 1 -- start start 2019-12-05T05:20:44.200 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:44.183+0000 7fdac101f700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fdabc118cd0 0x7fdabc07da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:20:44.200 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:44.183+0000 7fdac101f700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fdabc124810 0x7fdabc142a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:20:44.200 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:44.183+0000 7fdac101f700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdabc142f70 0x7fdabc117100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:20:44.201 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:44.183+0000 7fdac101f700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdabc11b2e0 con 0x7fdabc124810 2019-12-05T05:20:44.201 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:44.183+0000 7fdac101f700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdabc11b580 con 0x7fdabc118cd0 2019-12-05T05:20:44.201 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:44.183+0000 7fdac101f700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdabc11b820 con 0x7fdabc142f70 2019-12-05T05:20:44.201 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:44.183+0000 7fdabb59e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fdabc118cd0 0x7fdabc07da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:20:44.201 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:44.183+0000 7fdabb59e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fdabc118cd0 0x7fdabc07da60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:59908/0 (socket says 172.21.15.42:59908) 2019-12-05T05:20:44.201 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:44.183+0000 7fdab3fff700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fdabc124810 0x7fdabc142a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:20:44.202 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:44.183+0000 7fdac081e700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdabc142f70 0x7fdabc117100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:20:44.203 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:44.183+0000 7fdabb59e700 1 -- 172.21.15.42:0/999091280 learned_addr learned my addr 172.21.15.42:0/999091280 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:20:44.203 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:44.183+0000 7fdac081e700 1 -- 172.21.15.42:0/999091280 >> v2:172.21.15.42:3300/0 conn(0x7fdabc118cd0 msgr2=0x7fdabc07da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:20:44.203 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:44.183+0000 7fdac081e700 1 --2- 172.21.15.42:0/999091280 >> v2:172.21.15.42:3300/0 conn(0x7fdabc118cd0 0x7fdabc07da60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:20:44.204 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:44.183+0000 7fdac081e700 1 -- 172.21.15.42:0/999091280 >> v2:172.21.15.40:3300/0 conn(0x7fdabc124810 msgr2=0x7fdabc142a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:20:44.204 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:44.183+0000 7fdac081e700 1 --2- 172.21.15.42:0/999091280 >> v2:172.21.15.40:3300/0 conn(0x7fdabc124810 0x7fdabc142a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:20:44.204 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:44.183+0000 7fdac081e700 1 -- 172.21.15.42:0/999091280 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdabc1433a0 con 0x7fdabc142f70 2019-12-05T05:20:44.204 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:44.183+0000 7fdac081e700 1 --2- 172.21.15.42:0/999091280 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdabc142f70 0x7fdabc117100 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fdabc0a5ed0 tx=0x7fdab4010e60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:44.204 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:44.183+0000 7fdac081e700 1 --2- 172.21.15.42:0/999091280 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdabc142f70 0x7fdabc117100 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fdabc0a5ed0 tx=0x7fdab4010e60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:44.205 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:44.183+0000 7fdac081e700 1 --2- 172.21.15.42:0/999091280 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdabc142f70 0x7fdabc117100 secure :-1 s=READY pgs=146 cs=0 l=1 rx=0x7fdabc0a5ed0 tx=0x7fdab4010e60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:20:44.206 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:44.187+0000 7fdac081e700 1 --2- 172.21.15.42:0/999091280 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdabc142f70 0x7fdabc117100 secure :-1 s=THROTTLE_DONE pgs=146 cs=0 l=1 rx=0x7fdabc0a5ed0 tx=0x7fdab4010e60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:44.206 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:44.187+0000 7fdab8d99700 1 -- 172.21.15.42:0/999091280 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdab4005ab0 con 0x7fdabc142f70 2019-12-05T05:20:44.206 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:44.187+0000 7fdac081e700 1 --2- 172.21.15.42:0/999091280 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdabc142f70 0x7fdabc117100 secure :-1 s=THROTTLE_DONE pgs=146 cs=0 l=1 rx=0x7fdabc0a5ed0 tx=0x7fdab4010e60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:44.206 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:44.187+0000 7fdac081e700 1 --2- 172.21.15.42:0/999091280 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdabc142f70 0x7fdabc117100 secure :-1 s=THROTTLE_DONE pgs=146 cs=0 l=1 rx=0x7fdabc0a5ed0 tx=0x7fdab4010e60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:44.206 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:44.187+0000 7fdab8d99700 1 -- 172.21.15.42:0/999091280 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdab4010b70 con 0x7fdabc142f70 2019-12-05T05:20:44.207 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:44.187+0000 7fdab8d99700 1 -- 172.21.15.42:0/999091280 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdab4007350 con 0x7fdabc142f70 2019-12-05T05:20:44.207 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:44.187+0000 7fdac101f700 1 -- 172.21.15.42:0/999091280 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdabc11af60 con 0x7fdabc142f70 2019-12-05T05:20:44.208 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:44.187+0000 7fdac101f700 1 -- 172.21.15.42:0/999091280 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdabc07dfb0 con 0x7fdabc142f70 2019-12-05T05:20:44.209 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:44.187+0000 7fdac081e700 1 --2- 172.21.15.42:0/999091280 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdabc142f70 0x7fdabc117100 secure :-1 s=THROTTLE_DONE pgs=146 cs=0 l=1 rx=0x7fdabc0a5ed0 tx=0x7fdab4010e60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:44.209 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:44.187+0000 7fdac081e700 1 --2- 172.21.15.42:0/999091280 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdabc142f70 0x7fdabc117100 secure :-1 s=THROTTLE_DONE pgs=146 cs=0 l=1 rx=0x7fdabc0a5ed0 tx=0x7fdab4010e60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:44.209 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:44.187+0000 7fdab8d99700 1 -- 172.21.15.42:0/999091280 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fdab4007550 con 0x7fdabc142f70 2019-12-05T05:20:44.209 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:44.187+0000 7fdab8d99700 1 --2- 172.21.15.42:0/999091280 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fdaa003aec0 0x7fdaa003d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:20:44.209 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:44.187+0000 7fdab8d99700 1 -- 172.21.15.42:0/999091280 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 7063+0+0 (secure 0 0 0) 0x7fdab404d4b0 con 0x7fdabc142f70 2019-12-05T05:20:44.209 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:44.187+0000 7fdabb59e700 1 --2- 172.21.15.42:0/999091280 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fdaa003aec0 0x7fdaa003d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:20:44.210 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:44.187+0000 7fdabb59e700 1 --2- 172.21.15.42:0/999091280 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fdaa003aec0 0x7fdaa003d3d0 crc :-1 s=READY pgs=312 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:20:44.210 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:44.191+0000 7fdac101f700 1 -- 172.21.15.42:0/999091280 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdabc05d110 con 0x7fdabc142f70 2019-12-05T05:20:44.213 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:44.203+0000 7fdac081e700 1 --2- 172.21.15.42:0/999091280 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdabc142f70 0x7fdabc117100 secure :-1 s=THROTTLE_DONE pgs=146 cs=0 l=1 rx=0x7fdabc0a5ed0 tx=0x7fdab4010e60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:44.213 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:44.203+0000 7fdab8d99700 1 -- 172.21.15.42:0/999091280 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fdab4007b10 con 0x7fdabc142f70 2019-12-05T05:20:44.374 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:44.367+0000 7fdac101f700 1 -- 172.21.15.42:0/999091280 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdabc05b8c0 con 0x7fdabc142f70 2019-12-05T05:20:44.374 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:44.367+0000 7fdac081e700 1 --2- 172.21.15.42:0/999091280 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdabc142f70 0x7fdabc117100 secure :-1 s=THROTTLE_DONE pgs=146 cs=0 l=1 rx=0x7fdabc0a5ed0 tx=0x7fdab4010e60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:44.375 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:44.367+0000 7fdab8d99700 1 -- 172.21.15.42:0/999091280 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+1771 (secure 0 0 0) 0x7fdab4032130 con 0x7fdabc142f70 2019-12-05T05:20:44.375 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 89 2019-12-05T05:20:44.375 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:20:44.376 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":89,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:20:43.496004+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5066},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5066":{"gid":5066,"name":"b","rank":0,"incarnation":89,"state":"up:replay","state_seq":2,"addr":"172.21.15.40:6833/3061061478","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3061061478},{"type":"v1","addr":"172.21.15.40:6833","nonce":3061061478}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:20:44.378 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:44.371+0000 7fdac101f700 1 -- 172.21.15.42:0/999091280 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fdaa003aec0 msgr2=0x7fdaa003d3d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:20:44.378 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:44.371+0000 7fdac101f700 1 --2- 172.21.15.42:0/999091280 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fdaa003aec0 0x7fdaa003d3d0 crc :-1 s=READY pgs=312 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:20:44.378 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:44.371+0000 7fdac101f700 1 -- 172.21.15.42:0/999091280 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdabc142f70 msgr2=0x7fdabc117100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:20:44.378 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:44.371+0000 7fdac101f700 1 --2- 172.21.15.42:0/999091280 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdabc142f70 0x7fdabc117100 secure :-1 s=READY pgs=146 cs=0 l=1 rx=0x7fdabc0a5ed0 tx=0x7fdab4010e60).stop 2019-12-05T05:20:44.379 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:44.371+0000 7fdac101f700 1 -- 172.21.15.42:0/999091280 shutdown_connections 2019-12-05T05:20:44.379 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:44.371+0000 7fdac101f700 1 --2- 172.21.15.42:0/999091280 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fdaa003aec0 0x7fdaa003d3d0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:20:44.379 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:44.371+0000 7fdac101f700 1 --2- 172.21.15.42:0/999091280 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdabc142f70 0x7fdabc117100 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:20:44.379 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:44.371+0000 7fdac101f700 1 --2- 172.21.15.42:0/999091280 >> v2:172.21.15.42:3300/0 conn(0x7fdabc118cd0 0x7fdabc07da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:20:44.379 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:44.371+0000 7fdac101f700 1 --2- 172.21.15.42:0/999091280 >> v2:172.21.15.40:3300/0 conn(0x7fdabc124810 0x7fdabc142a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:20:44.379 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:44.371+0000 7fdac101f700 1 -- 172.21.15.42:0/999091280 shutdown_connections 2019-12-05T05:20:44.379 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:44.371+0000 7fdac101f700 1 -- 172.21.15.42:0/999091280 wait complete. 2019-12-05T05:20:44.380 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:44.371+0000 7fdac101f700 1 -- 172.21.15.42:0/999091280 >> 172.21.15.42:0/999091280 conn(0x7fdabc08fde0 msgr2=0x7fdabc07e590 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:20:44.388 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 89, "feature_flags": { "enable_multiple": false, "ever_enabled_multiple": false }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2019-12-05T05:03:35.604738+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 89, "ever_allowed_features": 2, "explicitly_allowed_features": 2, "failed": [], "flags": 18, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5066": { "addr": "172.21.15.40:6833/3061061478", "addrs": { "addrvec": [ { "addr": "172.21.15.40:6832", "nonce": 3061061478, "type": "v2" }, { "addr": "172.21.15.40:6833", "nonce": 3061061478, "type": "v1" } ] }, "export_targets": [], "features": 4540138292836696063, "flags": 0, "gid": 5066, "incarnation": 89, "name": "b", "rank": 0, "state": "up:replay", "state_seq": 2 } }, "last_failure": 0, "last_failure_osd_epoch": 61, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 2, "min_compat_client": "0 (unknown)", "modified": "2019-12-05T05:20:43.496004+0000", "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5066 } } } ], "standbys": [] } 2019-12-05T05:20:46.390 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:20:46.496 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:46.483+0000 7f61c76b0700 1 Processor -- start 2019-12-05T05:20:46.496 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:46.483+0000 7f61c76b0700 1 -- start start 2019-12-05T05:20:46.496 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:46.483+0000 7f61c76b0700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f61c00a7740 0x7f61c008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:20:46.497 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:46.483+0000 7f61c76b0700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f61c007e720 0x7f61c007eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:20:46.497 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:46.483+0000 7f61c76b0700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61c00a0710 con 0x7f61c009f2e0 2019-12-05T05:20:46.497 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:46.483+0000 7f61c76b0700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61c00a0b00 con 0x7f61c00a7740 2019-12-05T05:20:46.497 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:46.483+0000 7f61c76b0700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61c008e5b0 con 0x7f61c007e720 2019-12-05T05:20:46.497 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:46.483+0000 7f61c644e700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f61c009f2e0 0x7f61c009f710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:59960/0 (socket says 172.21.15.42:59960) 2019-12-05T05:20:46.497 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:46.483+0000 7f61c644e700 1 -- 172.21.15.42:0/2222243618 learned_addr learned my addr 172.21.15.42:0/2222243618 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:20:46.498 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:46.483+0000 7f61c544c700 1 --2- 172.21.15.42:0/2222243618 >> v2:172.21.15.42:3300/0 conn(0x7f61c007e720 0x7f61c007eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:20:46.498 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:46.487+0000 7f61c5c4d700 1 --2- 172.21.15.42:0/2222243618 >> v2:172.21.15.40:3300/0 conn(0x7f61c00a7740 0x7f61c008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:20:46.498 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:46.487+0000 7f61c4c4b700 1 -- 172.21.15.42:0/2222243618 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3563775039 0 0) 0x7f61c00a0710 con 0x7f61c009f2e0 2019-12-05T05:20:46.498 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:46.487+0000 7f61c4c4b700 1 -- 172.21.15.42:0/2222243618 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61ac0030b0 con 0x7f61c009f2e0 2019-12-05T05:20:46.498 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:46.487+0000 7f61c544c700 1 -- 172.21.15.42:0/2222243618 >> v2:172.21.15.40:3300/0 conn(0x7f61c00a7740 msgr2=0x7f61c008df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:20:46.498 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:46.487+0000 7f61c544c700 1 --2- 172.21.15.42:0/2222243618 >> v2:172.21.15.40:3300/0 conn(0x7f61c00a7740 0x7f61c008df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:20:46.498 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:46.487+0000 7f61c544c700 1 -- 172.21.15.42:0/2222243618 >> v1:172.21.15.42:6789/0 conn(0x7f61c009f2e0 legacy=0x7f61c009f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:20:46.499 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:46.487+0000 7f61c544c700 1 -- 172.21.15.42:0/2222243618 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61c007f0a0 con 0x7f61c007e720 2019-12-05T05:20:46.499 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:46.487+0000 7f61c5c4d700 1 --2- 172.21.15.42:0/2222243618 >> v2:172.21.15.40:3300/0 conn(0x7f61c00a7740 0x7f61c008df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:20:46.499 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:46.487+0000 7f61c544c700 1 --2- 172.21.15.42:0/2222243618 >> v2:172.21.15.42:3300/0 conn(0x7f61c007e720 0x7f61c007eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f61b0002f30 tx=0x7f61b000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:46.499 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:46.487+0000 7f61c544c700 1 --2- 172.21.15.42:0/2222243618 >> v2:172.21.15.42:3300/0 conn(0x7f61c007e720 0x7f61c007eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f61b0002f30 tx=0x7f61b000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:46.499 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:46.487+0000 7f61c544c700 1 --2- 172.21.15.42:0/2222243618 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f61c007e720 0x7f61c007eb50 secure :-1 s=READY pgs=691 cs=0 l=1 rx=0x7f61b0002f30 tx=0x7f61b000fc70).ready entity=mon.1 client_cookie=d7465ee7bceb945f server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:20:46.499 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:46.487+0000 7f61c544c700 1 --2- 172.21.15.42:0/2222243618 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f61c007e720 0x7f61c007eb50 secure :-1 s=THROTTLE_DONE pgs=691 cs=0 l=1 rx=0x7f61b0002f30 tx=0x7f61b000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:46.499 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:46.487+0000 7f61c4c4b700 1 -- 172.21.15.42:0/2222243618 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f61b00112b0 con 0x7f61c007e720 2019-12-05T05:20:46.500 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:46.487+0000 7f61c544c700 1 --2- 172.21.15.42:0/2222243618 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f61c007e720 0x7f61c007eb50 secure :-1 s=THROTTLE_DONE pgs=691 cs=0 l=1 rx=0x7f61b0002f30 tx=0x7f61b000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:46.500 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:46.487+0000 7f61c544c700 1 --2- 172.21.15.42:0/2222243618 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f61c007e720 0x7f61c007eb50 secure :-1 s=THROTTLE_DONE pgs=691 cs=0 l=1 rx=0x7f61b0002f30 tx=0x7f61b000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:46.500 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:46.487+0000 7f61c4c4b700 1 -- 172.21.15.42:0/2222243618 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61b0008ad0 con 0x7f61c007e720 2019-12-05T05:20:46.500 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:46.487+0000 7f61c4c4b700 1 -- 172.21.15.42:0/2222243618 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f61b0011890 con 0x7f61c007e720 2019-12-05T05:20:46.500 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:46.487+0000 7f61c76b0700 1 -- 172.21.15.42:0/2222243618 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f61c007e720 msgr2=0x7f61c007eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:20:46.500 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:46.487+0000 7f61c76b0700 1 --2- 172.21.15.42:0/2222243618 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f61c007e720 0x7f61c007eb50 secure :-1 s=READY pgs=691 cs=0 l=1 rx=0x7f61b0002f30 tx=0x7f61b000fc70).stop 2019-12-05T05:20:46.501 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:46.487+0000 7f61c76b0700 1 -- 172.21.15.42:0/2222243618 shutdown_connections 2019-12-05T05:20:46.501 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:46.487+0000 7f61c76b0700 1 --2- 172.21.15.42:0/2222243618 >> v2:172.21.15.40:3300/0 conn(0x7f61c00a7740 0x7f61c008df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:20:46.502 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:46.487+0000 7f61c76b0700 1 --2- 172.21.15.42:0/2222243618 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f61c007e720 0x7f61c007eb50 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:20:46.502 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:46.487+0000 7f61c76b0700 1 -- 172.21.15.42:0/2222243618 shutdown_connections 2019-12-05T05:20:46.502 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:46.487+0000 7f61c76b0700 1 -- 172.21.15.42:0/2222243618 wait complete. 2019-12-05T05:20:46.502 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:46.487+0000 7f61c76b0700 1 -- 172.21.15.42:0/2222243618 >> 172.21.15.42:0/2222243618 conn(0x7f61c00aabe0 msgr2=0x7f61c00a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:20:46.506 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:46.491+0000 7f61c76b0700 1 Processor -- start 2019-12-05T05:20:46.506 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:46.491+0000 7f61c76b0700 1 -- start start 2019-12-05T05:20:46.506 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:46.491+0000 7f61c76b0700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f61c011b0a0 0x7f61c007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:20:46.506 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:46.491+0000 7f61c76b0700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61c007c980 con 0x7f61c011bfd0 2019-12-05T05:20:46.507 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:46.491+0000 7f61c76b0700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61c01397c0 con 0x7f61c007ac70 2019-12-05T05:20:46.507 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:46.491+0000 7f61c76b0700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61c007cda0 con 0x7f61c011b0a0 2019-12-05T05:20:46.507 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:46.495+0000 7f61c644e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f61c011b0a0 0x7f61c007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:20:46.507 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:46.495+0000 7f61c5c4d700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f61c011bfd0 0x7f61c0118050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:41170/0 (socket says 172.21.15.42:41170) 2019-12-05T05:20:46.507 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:46.495+0000 7f61c644e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f61c011b0a0 0x7f61c007b0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:59928/0 (socket says 172.21.15.42:59928) 2019-12-05T05:20:46.507 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:46.495+0000 7f61c5c4d700 1 -- 172.21.15.42:0/1375820862 learned_addr learned my addr 172.21.15.42:0/1375820862 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:20:46.508 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:46.495+0000 7f61c544c700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f61c007ac70 0x7f61c011a8a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:59964/0 (socket says 172.21.15.42:59964) 2019-12-05T05:20:46.508 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:46.495+0000 7f61b67fc700 1 -- 172.21.15.42:0/1375820862 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 347350041 0 0) 0x7f61c007c980 con 0x7f61c011bfd0 2019-12-05T05:20:46.508 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:46.495+0000 7f61b67fc700 1 -- 172.21.15.42:0/1375820862 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61a80030b0 con 0x7f61c011bfd0 2019-12-05T05:20:46.508 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:46.495+0000 7f61c644e700 1 -- 172.21.15.42:0/1375820862 >> v1:172.21.15.42:6789/0 conn(0x7f61c007ac70 legacy=0x7f61c011a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:20:46.508 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:46.495+0000 7f61c644e700 1 -- 172.21.15.42:0/1375820862 >> v1:172.21.15.40:6789/0 conn(0x7f61c011bfd0 legacy=0x7f61c0118050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:20:46.509 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:46.495+0000 7f61c644e700 1 -- 172.21.15.42:0/1375820862 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61c013aa00 con 0x7f61c011b0a0 2019-12-05T05:20:46.509 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:46.495+0000 7f61c644e700 1 --2- 172.21.15.42:0/1375820862 >> v2:172.21.15.42:3300/0 conn(0x7f61c011b0a0 0x7f61c007b0a0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f61b8004720 tx=0x7f61b8010b10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:46.509 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:46.495+0000 7f61c644e700 1 --2- 172.21.15.42:0/1375820862 >> v2:172.21.15.42:3300/0 conn(0x7f61c011b0a0 0x7f61c007b0a0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f61b8004720 tx=0x7f61b8010b10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:46.509 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:46.495+0000 7f61c644e700 1 --2- 172.21.15.42:0/1375820862 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f61c011b0a0 0x7f61c007b0a0 secure :-1 s=READY pgs=693 cs=0 l=1 rx=0x7f61b8004720 tx=0x7f61b8010b10).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:20:46.509 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:46.495+0000 7f61c644e700 1 --2- 172.21.15.42:0/1375820862 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f61c011b0a0 0x7f61c007b0a0 secure :-1 s=THROTTLE_DONE pgs=693 cs=0 l=1 rx=0x7f61b8004720 tx=0x7f61b8010b10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:46.509 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:46.495+0000 7f61c644e700 1 --2- 172.21.15.42:0/1375820862 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f61c011b0a0 0x7f61c007b0a0 secure :-1 s=THROTTLE_DONE pgs=693 cs=0 l=1 rx=0x7f61b8004720 tx=0x7f61b8010b10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:46.509 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:46.495+0000 7f61b67fc700 1 -- 172.21.15.42:0/1375820862 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f61b8005bc0 con 0x7f61c011b0a0 2019-12-05T05:20:46.510 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:46.495+0000 7f61c644e700 1 --2- 172.21.15.42:0/1375820862 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f61c011b0a0 0x7f61c007b0a0 secure :-1 s=THROTTLE_DONE pgs=693 cs=0 l=1 rx=0x7f61b8004720 tx=0x7f61b8010b10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:46.510 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:46.495+0000 7f61b67fc700 1 -- 172.21.15.42:0/1375820862 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61b80057e0 con 0x7f61c011b0a0 2019-12-05T05:20:46.510 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:46.495+0000 7f61c76b0700 1 -- 172.21.15.42:0/1375820862 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61c01399f0 con 0x7f61c011b0a0 2019-12-05T05:20:46.510 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:46.495+0000 7f61b67fc700 1 -- 172.21.15.42:0/1375820862 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f61b8012830 con 0x7f61c011b0a0 2019-12-05T05:20:46.510 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:46.495+0000 7f61c76b0700 1 -- 172.21.15.42:0/1375820862 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61c0139f70 con 0x7f61c011b0a0 2019-12-05T05:20:46.510 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:46.495+0000 7f61c644e700 1 --2- 172.21.15.42:0/1375820862 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f61c011b0a0 0x7f61c007b0a0 secure :-1 s=THROTTLE_DONE pgs=693 cs=0 l=1 rx=0x7f61b8004720 tx=0x7f61b8010b10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:46.510 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:46.495+0000 7f61b67fc700 1 -- 172.21.15.42:0/1375820862 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f61b8012a30 con 0x7f61c011b0a0 2019-12-05T05:20:46.511 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:46.495+0000 7f61c644e700 1 --2- 172.21.15.42:0/1375820862 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f61c011b0a0 0x7f61c007b0a0 secure :-1 s=THROTTLE_DONE pgs=693 cs=0 l=1 rx=0x7f61b8004720 tx=0x7f61b8010b10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:46.511 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:46.495+0000 7f61b67fc700 1 --2- 172.21.15.42:0/1375820862 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f61a803b260 0x7f61a803d770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:20:46.511 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:46.495+0000 7f61b67fc700 1 -- 172.21.15.42:0/1375820862 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 7063+0+0 (secure 0 0 0) 0x7f61b804afa0 con 0x7f61c011b0a0 2019-12-05T05:20:46.511 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:46.495+0000 7f61c5c4d700 1 --2- 172.21.15.42:0/1375820862 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f61a803b260 0x7f61a803d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:20:46.511 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:46.499+0000 7f61c5c4d700 1 --2- 172.21.15.42:0/1375820862 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f61a803b260 0x7f61a803d770 crc :-1 s=READY pgs=313 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:20:46.511 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:46.499+0000 7f61c76b0700 1 -- 172.21.15.42:0/1375820862 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61c005d110 con 0x7f61c011b0a0 2019-12-05T05:20:46.514 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:46.507+0000 7f61c644e700 1 --2- 172.21.15.42:0/1375820862 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f61c011b0a0 0x7f61c007b0a0 secure :-1 s=THROTTLE_DONE pgs=693 cs=0 l=1 rx=0x7f61b8004720 tx=0x7f61b8010b10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:46.515 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:46.507+0000 7f61b67fc700 1 -- 172.21.15.42:0/1375820862 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f61b8032020 con 0x7f61c011b0a0 2019-12-05T05:20:46.686 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:46.679+0000 7f61c76b0700 1 -- 172.21.15.42:0/1375820862 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f61c011ba70 con 0x7f61c011b0a0 2019-12-05T05:20:46.687 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:46.679+0000 7f61c644e700 1 --2- 172.21.15.42:0/1375820862 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f61c011b0a0 0x7f61c007b0a0 secure :-1 s=THROTTLE_DONE pgs=693 cs=0 l=1 rx=0x7f61b8004720 tx=0x7f61b8010b10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:46.687 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:46.679+0000 7f61b67fc700 1 -- 172.21.15.42:0/1375820862 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 90 v90) v1 ==== 94+0+1775 (secure 0 0 0) 0x7f61b8032280 con 0x7f61c011b0a0 2019-12-05T05:20:46.687 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 90 2019-12-05T05:20:46.687 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:20:46.688 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":90,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":90,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:20:45.945478+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5066},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5066":{"gid":5066,"name":"b","rank":0,"incarnation":89,"state":"up:reconnect","state_seq":28,"addr":"172.21.15.40:6833/3061061478","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3061061478},{"type":"v1","addr":"172.21.15.40:6833","nonce":3061061478}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:20:46.690 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:46.679+0000 7f61c76b0700 1 -- 172.21.15.42:0/1375820862 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f61a803b260 msgr2=0x7f61a803d770 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:20:46.690 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:46.679+0000 7f61c76b0700 1 --2- 172.21.15.42:0/1375820862 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f61a803b260 0x7f61a803d770 crc :-1 s=READY pgs=313 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:20:46.690 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:46.679+0000 7f61c76b0700 1 -- 172.21.15.42:0/1375820862 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f61c011b0a0 msgr2=0x7f61c007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:20:46.690 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:46.679+0000 7f61c76b0700 1 --2- 172.21.15.42:0/1375820862 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f61c011b0a0 0x7f61c007b0a0 secure :-1 s=READY pgs=693 cs=0 l=1 rx=0x7f61b8004720 tx=0x7f61b8010b10).stop 2019-12-05T05:20:46.690 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:46.679+0000 7f61c76b0700 1 -- 172.21.15.42:0/1375820862 shutdown_connections 2019-12-05T05:20:46.691 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:46.679+0000 7f61c76b0700 1 --2- 172.21.15.42:0/1375820862 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f61a803b260 0x7f61a803d770 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:20:46.691 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:46.679+0000 7f61c76b0700 1 --2- 172.21.15.42:0/1375820862 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f61c011b0a0 0x7f61c007b0a0 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:20:46.691 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:46.679+0000 7f61c76b0700 1 -- 172.21.15.42:0/1375820862 shutdown_connections 2019-12-05T05:20:46.691 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:46.679+0000 7f61c76b0700 1 -- 172.21.15.42:0/1375820862 wait complete. 2019-12-05T05:20:46.691 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:46.679+0000 7f61c76b0700 1 -- 172.21.15.42:0/1375820862 >> 172.21.15.42:0/1375820862 conn(0x7f61c009f1c0 msgr2=0x7f61c0059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:20:46.697 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2019-12-05T05:20:48.697 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:20:48.797 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:48.787+0000 7f0c49887700 1 Processor -- start 2019-12-05T05:20:48.797 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:48.787+0000 7f0c49887700 1 -- start start 2019-12-05T05:20:48.798 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:48.787+0000 7f0c49887700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f0c440a7730 0x7f0c4408df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:20:48.798 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:48.787+0000 7f0c49887700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0c4407e710 0x7f0c4407eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:20:48.798 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:48.787+0000 7f0c49887700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c440a0700 con 0x7f0c4409f2d0 2019-12-05T05:20:48.798 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:48.787+0000 7f0c49887700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c440a0af0 con 0x7f0c440a7730 2019-12-05T05:20:48.798 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:48.787+0000 7f0c49887700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c4408e5a0 con 0x7f0c4407e710 2019-12-05T05:20:48.798 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:48.787+0000 7f0c42ffd700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0c4407e710 0x7f0c4407eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:20:48.799 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:48.787+0000 7f0c437fe700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f0c440a7730 0x7f0c4408df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:20:48.799 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:48.787+0000 7f0c42ffd700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0c4407e710 0x7f0c4407eb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:59938/0 (socket says 172.21.15.42:59938) 2019-12-05T05:20:48.799 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:48.787+0000 7f0c42ffd700 1 -- 172.21.15.42:0/987175611 learned_addr learned my addr 172.21.15.42:0/987175611 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:20:48.799 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:48.787+0000 7f0c42ffd700 1 -- 172.21.15.42:0/987175611 >> v2:172.21.15.40:3300/0 conn(0x7f0c440a7730 msgr2=0x7f0c4408df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:20:48.799 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:48.787+0000 7f0c42ffd700 1 --2- 172.21.15.42:0/987175611 >> v2:172.21.15.40:3300/0 conn(0x7f0c440a7730 0x7f0c4408df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:20:48.800 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:48.787+0000 7f0c42ffd700 1 -- 172.21.15.42:0/987175611 >> v1:172.21.15.40:6789/0 conn(0x7f0c4409f2d0 legacy=0x7f0c4409f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:20:48.800 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:48.787+0000 7f0c42ffd700 1 -- 172.21.15.42:0/987175611 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c4407f090 con 0x7f0c4407e710 2019-12-05T05:20:48.800 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:48.787+0000 7f0c437fe700 1 --2- 172.21.15.42:0/987175611 >> v2:172.21.15.40:3300/0 conn(0x7f0c440a7730 0x7f0c4408df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:20:48.800 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:48.787+0000 7f0c42ffd700 1 --2- 172.21.15.42:0/987175611 >> v2:172.21.15.42:3300/0 conn(0x7f0c4407e710 0x7f0c4407eb40 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f0c38002f30 tx=0x7f0c3800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:48.800 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:48.787+0000 7f0c42ffd700 1 --2- 172.21.15.42:0/987175611 >> v2:172.21.15.42:3300/0 conn(0x7f0c4407e710 0x7f0c4407eb40 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f0c38002f30 tx=0x7f0c3800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:48.800 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:48.787+0000 7f0c42ffd700 1 --2- 172.21.15.42:0/987175611 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0c4407e710 0x7f0c4407eb40 secure :-1 s=READY pgs=694 cs=0 l=1 rx=0x7f0c38002f30 tx=0x7f0c3800fd30).ready entity=mon.1 client_cookie=2b88cad3155549f2 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:20:48.801 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:48.787+0000 7f0c42ffd700 1 --2- 172.21.15.42:0/987175611 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0c4407e710 0x7f0c4407eb40 secure :-1 s=THROTTLE_DONE pgs=694 cs=0 l=1 rx=0x7f0c38002f30 tx=0x7f0c3800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:48.801 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:48.787+0000 7f0c427fc700 1 -- 172.21.15.42:0/987175611 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0c38011370 con 0x7f0c4407e710 2019-12-05T05:20:48.801 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:48.787+0000 7f0c42ffd700 1 --2- 172.21.15.42:0/987175611 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0c4407e710 0x7f0c4407eb40 secure :-1 s=THROTTLE_DONE pgs=694 cs=0 l=1 rx=0x7f0c38002f30 tx=0x7f0c3800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:48.801 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:48.787+0000 7f0c42ffd700 1 --2- 172.21.15.42:0/987175611 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0c4407e710 0x7f0c4407eb40 secure :-1 s=THROTTLE_DONE pgs=694 cs=0 l=1 rx=0x7f0c38002f30 tx=0x7f0c3800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:48.801 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:48.787+0000 7f0c427fc700 1 -- 172.21.15.42:0/987175611 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c38008bb0 con 0x7f0c4407e710 2019-12-05T05:20:48.801 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:48.787+0000 7f0c427fc700 1 -- 172.21.15.42:0/987175611 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0c38011950 con 0x7f0c4407e710 2019-12-05T05:20:48.802 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:48.787+0000 7f0c49887700 1 -- 172.21.15.42:0/987175611 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0c4407e710 msgr2=0x7f0c4407eb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:20:48.802 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:48.787+0000 7f0c49887700 1 --2- 172.21.15.42:0/987175611 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0c4407e710 0x7f0c4407eb40 secure :-1 s=READY pgs=694 cs=0 l=1 rx=0x7f0c38002f30 tx=0x7f0c3800fd30).stop 2019-12-05T05:20:48.802 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:48.791+0000 7f0c49887700 1 -- 172.21.15.42:0/987175611 shutdown_connections 2019-12-05T05:20:48.802 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:48.791+0000 7f0c49887700 1 --2- 172.21.15.42:0/987175611 >> v2:172.21.15.40:3300/0 conn(0x7f0c440a7730 0x7f0c4408df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:20:48.802 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:48.791+0000 7f0c49887700 1 --2- 172.21.15.42:0/987175611 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0c4407e710 0x7f0c4407eb40 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:20:48.802 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:48.791+0000 7f0c49887700 1 -- 172.21.15.42:0/987175611 shutdown_connections 2019-12-05T05:20:48.802 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:48.791+0000 7f0c49887700 1 -- 172.21.15.42:0/987175611 wait complete. 2019-12-05T05:20:48.803 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:48.791+0000 7f0c49887700 1 -- 172.21.15.42:0/987175611 >> 172.21.15.42:0/987175611 conn(0x7f0c440aabd0 msgr2=0x7f0c440a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:20:48.806 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:48.791+0000 7f0c49887700 1 Processor -- start 2019-12-05T05:20:48.806 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:48.791+0000 7f0c49887700 1 -- start start 2019-12-05T05:20:48.806 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:48.795+0000 7f0c49887700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0c44146560 0x7f0c44147800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:20:48.806 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:48.795+0000 7f0c49887700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f0c440a3c30 0x7f0c441150d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:20:48.806 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:48.795+0000 7f0c49887700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c440a4060 con 0x7f0c44145760 2019-12-05T05:20:48.807 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:48.795+0000 7f0c49887700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c44146ac0 con 0x7f0c440a3c30 2019-12-05T05:20:48.807 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:48.795+0000 7f0c49887700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c44146d60 con 0x7f0c44146560 2019-12-05T05:20:48.807 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:48.795+0000 7f0c42ffd700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0c44146560 0x7f0c44147800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:20:48.807 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:48.795+0000 7f0c42ffd700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0c44146560 0x7f0c44147800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:59944/0 (socket says 172.21.15.42:59944) 2019-12-05T05:20:48.807 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:48.795+0000 7f0c42ffd700 1 -- 172.21.15.42:0/598404815 learned_addr learned my addr 172.21.15.42:0/598404815 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:20:48.807 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:48.795+0000 7f0c43fff700 1 --2- 172.21.15.42:0/598404815 >> v2:172.21.15.40:3300/0 conn(0x7f0c440a3c30 0x7f0c441150d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:20:48.807 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:48.795+0000 7f0c42ffd700 1 -- 172.21.15.42:0/598404815 >> v2:172.21.15.40:3300/0 conn(0x7f0c440a3c30 msgr2=0x7f0c441150d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:20:48.808 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:48.795+0000 7f0c42ffd700 1 --2- 172.21.15.42:0/598404815 >> v2:172.21.15.40:3300/0 conn(0x7f0c440a3c30 0x7f0c441150d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:20:48.808 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:48.795+0000 7f0c42ffd700 1 -- 172.21.15.42:0/598404815 >> v1:172.21.15.42:6789/0 conn(0x7f0c44145760 legacy=0x7f0c44147000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:20:48.808 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:48.795+0000 7f0c42ffd700 1 -- 172.21.15.42:0/598404815 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c44115620 con 0x7f0c44146560 2019-12-05T05:20:48.808 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:48.795+0000 7f0c42ffd700 1 --2- 172.21.15.42:0/598404815 >> v2:172.21.15.42:3300/0 conn(0x7f0c44146560 0x7f0c44147800 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f0c38002f30 tx=0x7f0c38011f60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:48.808 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:48.795+0000 7f0c42ffd700 1 --2- 172.21.15.42:0/598404815 >> v2:172.21.15.42:3300/0 conn(0x7f0c44146560 0x7f0c44147800 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f0c38002f30 tx=0x7f0c38011f60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:48.808 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:48.795+0000 7f0c42ffd700 1 --2- 172.21.15.42:0/598404815 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0c44146560 0x7f0c44147800 secure :-1 s=READY pgs=696 cs=0 l=1 rx=0x7f0c38002f30 tx=0x7f0c38011f60).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:20:48.809 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:48.795+0000 7f0c42ffd700 1 --2- 172.21.15.42:0/598404815 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0c44146560 0x7f0c44147800 secure :-1 s=THROTTLE_DONE pgs=696 cs=0 l=1 rx=0x7f0c38002f30 tx=0x7f0c38011f60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:48.809 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:48.795+0000 7f0c42ffd700 1 --2- 172.21.15.42:0/598404815 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0c44146560 0x7f0c44147800 secure :-1 s=THROTTLE_DONE pgs=696 cs=0 l=1 rx=0x7f0c38002f30 tx=0x7f0c38011f60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:48.809 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:48.795+0000 7f0c2bfff700 1 -- 172.21.15.42:0/598404815 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0c38007340 con 0x7f0c44146560 2019-12-05T05:20:48.809 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:48.795+0000 7f0c42ffd700 1 --2- 172.21.15.42:0/598404815 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0c44146560 0x7f0c44147800 secure :-1 s=THROTTLE_DONE pgs=696 cs=0 l=1 rx=0x7f0c38002f30 tx=0x7f0c38011f60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:48.809 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:48.795+0000 7f0c2bfff700 1 -- 172.21.15.42:0/598404815 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c38011570 con 0x7f0c44146560 2019-12-05T05:20:48.809 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:48.795+0000 7f0c49887700 1 -- 172.21.15.42:0/598404815 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c440a3130 con 0x7f0c44146560 2019-12-05T05:20:48.811 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:48.795+0000 7f0c2bfff700 1 -- 172.21.15.42:0/598404815 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0c38005540 con 0x7f0c44146560 2019-12-05T05:20:48.811 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:48.795+0000 7f0c49887700 1 -- 172.21.15.42:0/598404815 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c4412e5c0 con 0x7f0c44146560 2019-12-05T05:20:48.811 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:48.795+0000 7f0c42ffd700 1 --2- 172.21.15.42:0/598404815 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0c44146560 0x7f0c44147800 secure :-1 s=THROTTLE_DONE pgs=696 cs=0 l=1 rx=0x7f0c38002f30 tx=0x7f0c38011f60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:48.811 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:48.799+0000 7f0c2bfff700 1 -- 172.21.15.42:0/598404815 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f0c38005850 con 0x7f0c44146560 2019-12-05T05:20:48.811 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:48.799+0000 7f0c42ffd700 1 --2- 172.21.15.42:0/598404815 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0c44146560 0x7f0c44147800 secure :-1 s=THROTTLE_DONE pgs=696 cs=0 l=1 rx=0x7f0c38002f30 tx=0x7f0c38011f60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:48.811 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:48.799+0000 7f0c2bfff700 1 --2- 172.21.15.42:0/598404815 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0c2403ad80 0x7f0c2403d290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:20:48.811 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:48.799+0000 7f0c2bfff700 1 -- 172.21.15.42:0/598404815 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 7063+0+0 (secure 0 0 0) 0x7f0c3804b160 con 0x7f0c44146560 2019-12-05T05:20:48.812 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:48.799+0000 7f0c437fe700 1 --2- 172.21.15.42:0/598404815 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0c2403ad80 0x7f0c2403d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:20:48.812 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:48.799+0000 7f0c437fe700 1 --2- 172.21.15.42:0/598404815 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0c2403ad80 0x7f0c2403d290 crc :-1 s=READY pgs=314 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:20:48.813 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:48.799+0000 7f0c49887700 1 -- 172.21.15.42:0/598404815 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c4405d100 con 0x7f0c44146560 2019-12-05T05:20:48.815 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:48.811+0000 7f0c42ffd700 1 --2- 172.21.15.42:0/598404815 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0c44146560 0x7f0c44147800 secure :-1 s=THROTTLE_DONE pgs=696 cs=0 l=1 rx=0x7f0c38002f30 tx=0x7f0c38011f60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:48.816 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:48.811+0000 7f0c2bfff700 1 -- 172.21.15.42:0/598404815 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f0c380321e0 con 0x7f0c44146560 2019-12-05T05:20:48.984 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:48.979+0000 7f0c49887700 1 -- 172.21.15.42:0/598404815 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0c4405b8b0 con 0x7f0c44146560 2019-12-05T05:20:48.985 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:48.979+0000 7f0c42ffd700 1 --2- 172.21.15.42:0/598404815 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0c44146560 0x7f0c44147800 secure :-1 s=THROTTLE_DONE pgs=696 cs=0 l=1 rx=0x7f0c38002f30 tx=0x7f0c38011f60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:48.985 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:48.979+0000 7f0c2bfff700 1 -- 172.21.15.42:0/598404815 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 92 v92) v1 ==== 94+0+1772 (secure 0 0 0) 0x7f0c38007f50 con 0x7f0c44146560 2019-12-05T05:20:48.985 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 92 2019-12-05T05:20:48.986 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:20:48.986 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":92,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":92,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:20:47.959813+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5066},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5066":{"gid":5066,"name":"b","rank":0,"incarnation":89,"state":"up:active","state_seq":30,"addr":"172.21.15.40:6833/3061061478","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3061061478},{"type":"v1","addr":"172.21.15.40:6833","nonce":3061061478}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:20:48.988 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:48.979+0000 7f0c49887700 1 -- 172.21.15.42:0/598404815 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0c2403ad80 msgr2=0x7f0c2403d290 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:20:48.988 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:48.979+0000 7f0c49887700 1 --2- 172.21.15.42:0/598404815 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0c2403ad80 0x7f0c2403d290 crc :-1 s=READY pgs=314 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:20:48.988 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:48.979+0000 7f0c49887700 1 -- 172.21.15.42:0/598404815 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0c44146560 msgr2=0x7f0c44147800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:20:48.989 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:48.979+0000 7f0c49887700 1 --2- 172.21.15.42:0/598404815 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0c44146560 0x7f0c44147800 secure :-1 s=READY pgs=696 cs=0 l=1 rx=0x7f0c38002f30 tx=0x7f0c38011f60).stop 2019-12-05T05:20:48.989 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:48.979+0000 7f0c49887700 1 -- 172.21.15.42:0/598404815 shutdown_connections 2019-12-05T05:20:48.989 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:48.979+0000 7f0c49887700 1 --2- 172.21.15.42:0/598404815 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0c2403ad80 0x7f0c2403d290 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:20:48.989 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:48.979+0000 7f0c49887700 1 --2- 172.21.15.42:0/598404815 >> v2:172.21.15.40:3300/0 conn(0x7f0c440a3c30 0x7f0c441150d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:20:48.989 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:48.979+0000 7f0c49887700 1 --2- 172.21.15.42:0/598404815 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0c44146560 0x7f0c44147800 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:20:48.989 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:48.983+0000 7f0c49887700 1 -- 172.21.15.42:0/598404815 shutdown_connections 2019-12-05T05:20:48.989 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:48.983+0000 7f0c49887700 1 -- 172.21.15.42:0/598404815 wait complete. 2019-12-05T05:20:48.990 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:48.983+0000 7f0c49887700 1 -- 172.21.15.42:0/598404815 >> 172.21.15.42:0/598404815 conn(0x7f0c440a7f80 msgr2=0x7f0c441162c0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:20:48.998 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b has gained rank=0, replacing gid=5833 2019-12-05T05:20:48.998 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 1 secs before reviving mds.a 2019-12-05T05:20:49.999 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.a 2019-12-05T05:20:49.999 INFO:tasks.ceph.mds.a:Restarting daemon 2019-12-05T05:20:49.999 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2019-12-05T05:20:50.003 INFO:tasks.ceph.mds.a:Started 2019-12-05T05:20:50.003 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:20:50.064 INFO:tasks.ceph.mds.a.smithi042.stdout:starting mds.a at 2019-12-05T05:20:50.092 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:50.083+0000 7f6b8214c700 1 Processor -- start 2019-12-05T05:20:50.092 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:50.083+0000 7f6b8214c700 1 -- start start 2019-12-05T05:20:50.092 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:50.083+0000 7f6b8214c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6b7c0a7740 0x7f6b7c08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:20:50.092 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:50.083+0000 7f6b8214c700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6b7c07e720 0x7f6b7c07eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:20:50.092 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:50.083+0000 7f6b8214c700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b7c0a0710 con 0x7f6b7c09f2e0 2019-12-05T05:20:50.093 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:50.083+0000 7f6b8214c700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b7c0a0b00 con 0x7f6b7c0a7740 2019-12-05T05:20:50.093 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:50.083+0000 7f6b8214c700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b7c08e5b0 con 0x7f6b7c07e720 2019-12-05T05:20:50.093 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:50.083+0000 7f6b7bfff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6b7c0a7740 0x7f6b7c08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:20:50.093 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:50.083+0000 7f6b7bfff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6b7c0a7740 0x7f6b7c08df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:59966/0 (socket says 172.21.15.42:59966) 2019-12-05T05:20:50.093 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:50.083+0000 7f6b7b7fe700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6b7c07e720 0x7f6b7c07eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:20:50.093 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:50.083+0000 7f6b7bfff700 1 -- 172.21.15.42:0/3383667330 learned_addr learned my addr 172.21.15.42:0/3383667330 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:20:50.094 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:50.083+0000 7f6b80eea700 1 --1- 172.21.15.42:0/3383667330 >> v1:172.21.15.40:6789/0 conn(0x7f6b7c09f2e0 0x7f6b7c09f710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:41216/0 (socket says 172.21.15.42:41216) 2019-12-05T05:20:50.094 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:50.083+0000 7f6b7bfff700 1 -- 172.21.15.42:0/3383667330 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6b7c07e720 msgr2=0x7f6b7c07eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:20:50.094 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:50.083+0000 7f6b7bfff700 1 --2- 172.21.15.42:0/3383667330 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6b7c07e720 0x7f6b7c07eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:20:50.094 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:50.083+0000 7f6b7bfff700 1 -- 172.21.15.42:0/3383667330 >> v1:172.21.15.40:6789/0 conn(0x7f6b7c09f2e0 legacy=0x7f6b7c09f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:20:50.094 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:50.083+0000 7f6b7bfff700 1 -- 172.21.15.42:0/3383667330 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b7c07f0a0 con 0x7f6b7c0a7740 2019-12-05T05:20:50.094 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:50.083+0000 7f6b7affd700 1 -- 172.21.15.42:0/3383667330 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3506937963 0 0) 0x7f6b7c0a0710 con 0x7f6b7c09f2e0 2019-12-05T05:20:50.095 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:50.083+0000 7f6b7bfff700 1 --2- 172.21.15.42:0/3383667330 >> v2:172.21.15.42:3300/0 conn(0x7f6b7c0a7740 0x7f6b7c08df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f6b6c008790 tx=0x7f6b6c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:50.095 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:50.083+0000 7f6b7bfff700 1 --2- 172.21.15.42:0/3383667330 >> v2:172.21.15.42:3300/0 conn(0x7f6b7c0a7740 0x7f6b7c08df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f6b6c008790 tx=0x7f6b6c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:50.095 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:50.083+0000 7f6b7bfff700 1 --2- 172.21.15.42:0/3383667330 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6b7c0a7740 0x7f6b7c08df80 secure :-1 s=READY pgs=698 cs=0 l=1 rx=0x7f6b6c008790 tx=0x7f6b6c00fd30).ready entity=mon.1 client_cookie=983dadf711dc161a server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:20:50.095 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:50.083+0000 7f6b7bfff700 1 --2- 172.21.15.42:0/3383667330 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6b7c0a7740 0x7f6b7c08df80 secure :-1 s=THROTTLE_DONE pgs=698 cs=0 l=1 rx=0x7f6b6c008790 tx=0x7f6b6c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:50.095 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:50.083+0000 7f6b7bfff700 1 --2- 172.21.15.42:0/3383667330 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6b7c0a7740 0x7f6b7c08df80 secure :-1 s=THROTTLE_DONE pgs=698 cs=0 l=1 rx=0x7f6b6c008790 tx=0x7f6b6c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:50.095 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:50.083+0000 7f6b7affd700 1 -- 172.21.15.42:0/3383667330 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6b6c0113b0 con 0x7f6b7c0a7740 2019-12-05T05:20:50.095 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:50.083+0000 7f6b7bfff700 1 --2- 172.21.15.42:0/3383667330 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6b7c0a7740 0x7f6b7c08df80 secure :-1 s=THROTTLE_DONE pgs=698 cs=0 l=1 rx=0x7f6b6c008790 tx=0x7f6b6c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:50.096 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:50.083+0000 7f6b7affd700 1 -- 172.21.15.42:0/3383667330 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b6c008bb0 con 0x7f6b7c0a7740 2019-12-05T05:20:50.096 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:50.083+0000 7f6b7affd700 1 -- 172.21.15.42:0/3383667330 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6b6c011990 con 0x7f6b7c0a7740 2019-12-05T05:20:50.096 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:50.083+0000 7f6b8214c700 1 -- 172.21.15.42:0/3383667330 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6b7c0a7740 msgr2=0x7f6b7c08df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:20:50.096 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:50.083+0000 7f6b8214c700 1 --2- 172.21.15.42:0/3383667330 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6b7c0a7740 0x7f6b7c08df80 secure :-1 s=READY pgs=698 cs=0 l=1 rx=0x7f6b6c008790 tx=0x7f6b6c00fd30).stop 2019-12-05T05:20:50.096 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:50.083+0000 7f6b8214c700 1 -- 172.21.15.42:0/3383667330 shutdown_connections 2019-12-05T05:20:50.096 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:50.083+0000 7f6b8214c700 1 --2- 172.21.15.42:0/3383667330 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6b7c0a7740 0x7f6b7c08df80 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:20:50.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:50.083+0000 7f6b8214c700 1 --2- 172.21.15.42:0/3383667330 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6b7c07e720 0x7f6b7c07eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:20:50.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:50.083+0000 7f6b8214c700 1 -- 172.21.15.42:0/3383667330 shutdown_connections 2019-12-05T05:20:50.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:50.083+0000 7f6b8214c700 1 -- 172.21.15.42:0/3383667330 wait complete. 2019-12-05T05:20:50.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:50.083+0000 7f6b8214c700 1 -- 172.21.15.42:0/3383667330 >> 172.21.15.42:0/3383667330 conn(0x7f6b7c0aabe0 msgr2=0x7f6b7c0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:20:50.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:50.087+0000 7f6b8214c700 1 Processor -- start 2019-12-05T05:20:50.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:50.087+0000 7f6b8214c700 1 -- start start 2019-12-05T05:20:50.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:50.087+0000 7f6b8214c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6b7c124a20 0x7f6b7c0a37a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:20:50.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:50.087+0000 7f6b8214c700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6b7c0a3cf0 0x7f6b7c0a4120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:20:50.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:50.087+0000 7f6b8214c700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b7c1205e0 con 0x7f6b7c11be90 2019-12-05T05:20:50.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:50.087+0000 7f6b8214c700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b7c119350 con 0x7f6b7c124a20 2019-12-05T05:20:50.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:50.087+0000 7f6b8214c700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b7c1195f0 con 0x7f6b7c0a3cf0 2019-12-05T05:20:50.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:50.087+0000 7f6b7bfff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f6b7c11be90 0x7f6b7c11a470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:60010/0 (socket says 172.21.15.42:60010) 2019-12-05T05:20:50.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:50.087+0000 7f6b7bfff700 1 -- 172.21.15.42:0/3761055601 learned_addr learned my addr 172.21.15.42:0/3761055601 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:20:50.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:50.087+0000 7f6b7b7fe700 1 --2- 172.21.15.42:0/3761055601 >> v2:172.21.15.42:3300/0 conn(0x7f6b7c124a20 0x7f6b7c0a37a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:20:50.103 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:50.087+0000 7f6b80eea700 1 --2- 172.21.15.42:0/3761055601 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6b7c0a3cf0 0x7f6b7c0a4120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:20:50.103 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:50.087+0000 7f6b78ff9700 1 -- 172.21.15.42:0/3761055601 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3777828444 0 0) 0x7f6b7c1205e0 con 0x7f6b7c11be90 2019-12-05T05:20:50.104 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:50.087+0000 7f6b78ff9700 1 -- 172.21.15.42:0/3761055601 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b5c0030b0 con 0x7f6b7c11be90 2019-12-05T05:20:50.104 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:50.087+0000 7f6b80eea700 1 -- 172.21.15.42:0/3761055601 >> v2:172.21.15.42:3300/0 conn(0x7f6b7c124a20 msgr2=0x7f6b7c0a37a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:20:50.104 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:50.087+0000 7f6b80eea700 1 --2- 172.21.15.42:0/3761055601 >> v2:172.21.15.42:3300/0 conn(0x7f6b7c124a20 0x7f6b7c0a37a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:20:50.104 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:50.087+0000 7f6b80eea700 1 -- 172.21.15.42:0/3761055601 >> v1:172.21.15.42:6789/0 conn(0x7f6b7c11be90 legacy=0x7f6b7c11a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:20:50.105 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:50.087+0000 7f6b80eea700 1 -- 172.21.15.42:0/3761055601 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b7c131290 con 0x7f6b7c0a3cf0 2019-12-05T05:20:50.105 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:50.087+0000 7f6b7b7fe700 1 --2- 172.21.15.42:0/3761055601 >> v2:172.21.15.42:3300/0 conn(0x7f6b7c124a20 0x7f6b7c0a37a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:20:50.105 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:50.087+0000 7f6b80eea700 1 --2- 172.21.15.42:0/3761055601 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6b7c0a3cf0 0x7f6b7c0a4120 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f6b7c0a8f90 tx=0x7f6b74010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:50.105 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:50.087+0000 7f6b80eea700 1 --2- 172.21.15.42:0/3761055601 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6b7c0a3cf0 0x7f6b7c0a4120 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f6b7c0a8f90 tx=0x7f6b74010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:50.105 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:50.087+0000 7f6b80eea700 1 --2- 172.21.15.42:0/3761055601 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6b7c0a3cf0 0x7f6b7c0a4120 secure :-1 s=READY pgs=149 cs=0 l=1 rx=0x7f6b7c0a8f90 tx=0x7f6b74010b60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:20:50.105 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:50.087+0000 7f6b80eea700 1 --2- 172.21.15.42:0/3761055601 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6b7c0a3cf0 0x7f6b7c0a4120 secure :-1 s=THROTTLE_DONE pgs=149 cs=0 l=1 rx=0x7f6b7c0a8f90 tx=0x7f6b74010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:50.105 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:50.087+0000 7f6b78ff9700 1 -- 172.21.15.42:0/3761055601 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6b74004ad0 con 0x7f6b7c0a3cf0 2019-12-05T05:20:50.106 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:50.087+0000 7f6b80eea700 1 --2- 172.21.15.42:0/3761055601 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6b7c0a3cf0 0x7f6b7c0a4120 secure :-1 s=THROTTLE_DONE pgs=149 cs=0 l=1 rx=0x7f6b7c0a8f90 tx=0x7f6b74010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:50.106 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:50.087+0000 7f6b78ff9700 1 -- 172.21.15.42:0/3761055601 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b740046f0 con 0x7f6b7c0a3cf0 2019-12-05T05:20:50.106 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:50.087+0000 7f6b80eea700 1 --2- 172.21.15.42:0/3761055601 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6b7c0a3cf0 0x7f6b7c0a4120 secure :-1 s=THROTTLE_DONE pgs=149 cs=0 l=1 rx=0x7f6b7c0a8f90 tx=0x7f6b74010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:50.106 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:50.087+0000 7f6b8214c700 1 -- 172.21.15.42:0/3761055601 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b7c118eb0 con 0x7f6b7c0a3cf0 2019-12-05T05:20:50.106 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:50.087+0000 7f6b78ff9700 1 -- 172.21.15.42:0/3761055601 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6b740124c0 con 0x7f6b7c0a3cf0 2019-12-05T05:20:50.106 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:50.087+0000 7f6b8214c700 1 -- 172.21.15.42:0/3761055601 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b7c078310 con 0x7f6b7c0a3cf0 2019-12-05T05:20:50.106 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:50.087+0000 7f6b80eea700 1 --2- 172.21.15.42:0/3761055601 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6b7c0a3cf0 0x7f6b7c0a4120 secure :-1 s=THROTTLE_DONE pgs=149 cs=0 l=1 rx=0x7f6b7c0a8f90 tx=0x7f6b74010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:50.107 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:50.087+0000 7f6b78ff9700 1 -- 172.21.15.42:0/3761055601 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f6b740127e0 con 0x7f6b7c0a3cf0 2019-12-05T05:20:50.107 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:50.087+0000 7f6b80eea700 1 --2- 172.21.15.42:0/3761055601 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6b7c0a3cf0 0x7f6b7c0a4120 secure :-1 s=THROTTLE_DONE pgs=149 cs=0 l=1 rx=0x7f6b7c0a8f90 tx=0x7f6b74010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:50.107 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:50.091+0000 7f6b78ff9700 1 --2- 172.21.15.42:0/3761055601 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6b5c03b250 0x7f6b5c03d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:20:50.107 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:50.091+0000 7f6b78ff9700 1 -- 172.21.15.42:0/3761055601 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 7063+0+0 (secure 0 0 0) 0x7f6b7404b0d0 con 0x7f6b7c0a3cf0 2019-12-05T05:20:50.107 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:50.091+0000 7f6b7bfff700 1 --2- 172.21.15.42:0/3761055601 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6b5c03b250 0x7f6b5c03d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:20:50.107 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:50.091+0000 7f6b7bfff700 1 --2- 172.21.15.42:0/3761055601 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6b5c03b250 0x7f6b5c03d760 crc :-1 s=READY pgs=315 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:20:50.108 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:50.099+0000 7f6b8214c700 1 -- 172.21.15.42:0/3761055601 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b7c05d110 con 0x7f6b7c0a3cf0 2019-12-05T05:20:50.110 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:50.103+0000 7f6b80eea700 1 --2- 172.21.15.42:0/3761055601 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6b7c0a3cf0 0x7f6b7c0a4120 secure :-1 s=THROTTLE_DONE pgs=149 cs=0 l=1 rx=0x7f6b7c0a8f90 tx=0x7f6b74010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:50.110 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:50.103+0000 7f6b78ff9700 1 -- 172.21.15.42:0/3761055601 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f6b74032130 con 0x7f6b7c0a3cf0 2019-12-05T05:20:50.268 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:50.263+0000 7f6b8214c700 1 -- 172.21.15.42:0/3761055601 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6b7c05b8c0 con 0x7f6b7c0a3cf0 2019-12-05T05:20:50.269 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:50.263+0000 7f6b80eea700 1 --2- 172.21.15.42:0/3761055601 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6b7c0a3cf0 0x7f6b7c0a4120 secure :-1 s=THROTTLE_DONE pgs=149 cs=0 l=1 rx=0x7f6b7c0a8f90 tx=0x7f6b74010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:50.269 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:50.263+0000 7f6b78ff9700 1 -- 172.21.15.42:0/3761055601 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 92 v92) v1 ==== 94+0+1772 (secure 0 0 0) 0x7f6b74032390 con 0x7f6b7c0a3cf0 2019-12-05T05:20:50.271 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 92 2019-12-05T05:20:50.272 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:20:50.272 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":92,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":92,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:20:47.959813+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5066},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5066":{"gid":5066,"name":"b","rank":0,"incarnation":89,"state":"up:active","state_seq":30,"addr":"172.21.15.40:6833/3061061478","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3061061478},{"type":"v1","addr":"172.21.15.40:6833","nonce":3061061478}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:20:50.275 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:50.267+0000 7f6b8214c700 1 -- 172.21.15.42:0/3761055601 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6b5c03b250 msgr2=0x7f6b5c03d760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:20:50.275 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:50.267+0000 7f6b8214c700 1 --2- 172.21.15.42:0/3761055601 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6b5c03b250 0x7f6b5c03d760 crc :-1 s=READY pgs=315 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:20:50.275 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:50.267+0000 7f6b8214c700 1 -- 172.21.15.42:0/3761055601 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6b7c0a3cf0 msgr2=0x7f6b7c0a4120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:20:50.275 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:50.267+0000 7f6b8214c700 1 --2- 172.21.15.42:0/3761055601 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6b7c0a3cf0 0x7f6b7c0a4120 secure :-1 s=READY pgs=149 cs=0 l=1 rx=0x7f6b7c0a8f90 tx=0x7f6b74010b60).stop 2019-12-05T05:20:50.276 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:50.267+0000 7f6b8214c700 1 -- 172.21.15.42:0/3761055601 shutdown_connections 2019-12-05T05:20:50.276 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:50.267+0000 7f6b8214c700 1 --2- 172.21.15.42:0/3761055601 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6b5c03b250 0x7f6b5c03d760 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:20:50.276 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:50.267+0000 7f6b8214c700 1 --2- 172.21.15.42:0/3761055601 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6b7c0a3cf0 0x7f6b7c0a4120 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:20:50.276 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:50.267+0000 7f6b8214c700 1 --2- 172.21.15.42:0/3761055601 >> v2:172.21.15.42:3300/0 conn(0x7f6b7c124a20 0x7f6b7c0a37a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:20:50.276 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:50.267+0000 7f6b8214c700 1 -- 172.21.15.42:0/3761055601 shutdown_connections 2019-12-05T05:20:50.276 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:50.267+0000 7f6b8214c700 1 -- 172.21.15.42:0/3761055601 wait complete. 2019-12-05T05:20:50.277 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:50.267+0000 7f6b8214c700 1 -- 172.21.15.42:0/3761055601 >> 172.21.15.42:0/3761055601 conn(0x7f6b7c0aabe0 msgr2=0x7f6b7c11c470 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:20:50.286 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2019-12-05T05:20:52.286 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:20:52.395 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:52.383+0000 7fac9d369700 1 Processor -- start 2019-12-05T05:20:52.395 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:52.383+0000 7fac9d369700 1 -- start start 2019-12-05T05:20:52.396 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:52.383+0000 7fac9d369700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fac980a7740 0x7fac9808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:20:52.396 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:52.383+0000 7fac9d369700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fac9807e720 0x7fac9807eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:20:52.396 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:52.383+0000 7fac9d369700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fac9809f2e0 0x7fac9809f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:20:52.396 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:52.383+0000 7fac9d369700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac980a0b00 con 0x7fac9809f2e0 2019-12-05T05:20:52.396 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:52.383+0000 7fac9d369700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac9808e5b0 con 0x7fac9807e720 2019-12-05T05:20:52.396 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:52.383+0000 7fac9d369700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac9808e850 con 0x7fac980a7740 2019-12-05T05:20:52.396 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:52.383+0000 7fac96ffd700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fac9807e720 0x7fac9807eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:20:52.397 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:52.383+0000 7fac96ffd700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fac9807e720 0x7fac9807eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:59994/0 (socket says 172.21.15.42:59994) 2019-12-05T05:20:52.397 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:52.383+0000 7fac96ffd700 1 -- 172.21.15.42:0/3855280716 learned_addr learned my addr 172.21.15.42:0/3855280716 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:20:52.397 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:52.383+0000 7fac977fe700 1 --2- 172.21.15.42:0/3855280716 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fac980a7740 0x7fac9808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:20:52.397 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:52.383+0000 7fac97fff700 1 --2- 172.21.15.42:0/3855280716 >> v2:172.21.15.40:3300/0 conn(0x7fac9809f2e0 0x7fac9809f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:20:52.397 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:52.383+0000 7fac96ffd700 1 -- 172.21.15.42:0/3855280716 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fac980a7740 msgr2=0x7fac9808df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:20:52.397 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:52.383+0000 7fac96ffd700 1 --2- 172.21.15.42:0/3855280716 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fac980a7740 0x7fac9808df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:20:52.397 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:52.383+0000 7fac96ffd700 1 -- 172.21.15.42:0/3855280716 >> v2:172.21.15.40:3300/0 conn(0x7fac9809f2e0 msgr2=0x7fac9809f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:20:52.398 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:52.383+0000 7fac96ffd700 1 --2- 172.21.15.42:0/3855280716 >> v2:172.21.15.40:3300/0 conn(0x7fac9809f2e0 0x7fac9809f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:20:52.398 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:52.383+0000 7fac96ffd700 1 -- 172.21.15.42:0/3855280716 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac9807f0a0 con 0x7fac9807e720 2019-12-05T05:20:52.398 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:52.383+0000 7fac96ffd700 1 --2- 172.21.15.42:0/3855280716 >> v2:172.21.15.42:3300/0 conn(0x7fac9807e720 0x7fac9807eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fac9808e770 tx=0x7fac8c00fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:52.398 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:52.383+0000 7fac96ffd700 1 --2- 172.21.15.42:0/3855280716 >> v2:172.21.15.42:3300/0 conn(0x7fac9807e720 0x7fac9807eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fac9808e770 tx=0x7fac8c00fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:52.398 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:52.383+0000 7fac96ffd700 1 --2- 172.21.15.42:0/3855280716 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fac9807e720 0x7fac9807eb50 secure :-1 s=READY pgs=700 cs=0 l=1 rx=0x7fac9808e770 tx=0x7fac8c00fdb0).ready entity=mon.1 client_cookie=f764bd88901f72db server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:20:52.399 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:52.387+0000 7fac96ffd700 1 --2- 172.21.15.42:0/3855280716 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fac9807e720 0x7fac9807eb50 secure :-1 s=THROTTLE_DONE pgs=700 cs=0 l=1 rx=0x7fac9808e770 tx=0x7fac8c00fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:52.399 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:52.387+0000 7fac967fc700 1 -- 172.21.15.42:0/3855280716 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fac8c008c90 con 0x7fac9807e720 2019-12-05T05:20:52.399 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:52.387+0000 7fac96ffd700 1 --2- 172.21.15.42:0/3855280716 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fac9807e720 0x7fac9807eb50 secure :-1 s=THROTTLE_DONE pgs=700 cs=0 l=1 rx=0x7fac9808e770 tx=0x7fac8c00fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:52.399 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:52.387+0000 7fac96ffd700 1 --2- 172.21.15.42:0/3855280716 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fac9807e720 0x7fac9807eb50 secure :-1 s=THROTTLE_DONE pgs=700 cs=0 l=1 rx=0x7fac9808e770 tx=0x7fac8c00fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:52.399 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:52.387+0000 7fac967fc700 1 -- 172.21.15.42:0/3855280716 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac8c0088b0 con 0x7fac9807e720 2019-12-05T05:20:52.399 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:52.387+0000 7fac967fc700 1 -- 172.21.15.42:0/3855280716 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fac8c011b50 con 0x7fac9807e720 2019-12-05T05:20:52.400 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:52.387+0000 7fac9d369700 1 -- 172.21.15.42:0/3855280716 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fac9807e720 msgr2=0x7fac9807eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:20:52.400 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:52.387+0000 7fac9d369700 1 --2- 172.21.15.42:0/3855280716 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fac9807e720 0x7fac9807eb50 secure :-1 s=READY pgs=700 cs=0 l=1 rx=0x7fac9808e770 tx=0x7fac8c00fdb0).stop 2019-12-05T05:20:52.400 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:52.387+0000 7fac9d369700 1 -- 172.21.15.42:0/3855280716 shutdown_connections 2019-12-05T05:20:52.400 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:52.387+0000 7fac9d369700 1 --2- 172.21.15.42:0/3855280716 >> v2:172.21.15.40:3300/0 conn(0x7fac9809f2e0 0x7fac9809f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:20:52.400 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:52.387+0000 7fac9d369700 1 --2- 172.21.15.42:0/3855280716 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fac980a7740 0x7fac9808df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:20:52.400 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:52.387+0000 7fac9d369700 1 --2- 172.21.15.42:0/3855280716 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fac9807e720 0x7fac9807eb50 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:20:52.400 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:52.387+0000 7fac9d369700 1 -- 172.21.15.42:0/3855280716 shutdown_connections 2019-12-05T05:20:52.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:52.387+0000 7fac9d369700 1 -- 172.21.15.42:0/3855280716 wait complete. 2019-12-05T05:20:52.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:52.387+0000 7fac9d369700 1 -- 172.21.15.42:0/3855280716 >> 172.21.15.42:0/3855280716 conn(0x7fac980aabe0 msgr2=0x7fac980a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:20:52.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:52.387+0000 7fac9d369700 1 Processor -- start 2019-12-05T05:20:52.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:52.387+0000 7fac9d369700 1 -- start start 2019-12-05T05:20:52.404 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:52.391+0000 7fac9d369700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fac980a4840 0x7fac980a3d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:20:52.404 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:52.391+0000 7fac9d369700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac980a4250 con 0x7fac980a29c0 2019-12-05T05:20:52.404 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:52.391+0000 7fac9d369700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac980a4c70 con 0x7fac981280e0 2019-12-05T05:20:52.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:52.391+0000 7fac9d369700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac980a21e0 con 0x7fac980a4840 2019-12-05T05:20:52.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:52.391+0000 7fac96ffd700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fac980a4840 0x7fac980a3d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:20:52.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:52.391+0000 7fac97fff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fac981280e0 0x7fac98128510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:60040/0 (socket says 172.21.15.42:60040) 2019-12-05T05:20:52.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:52.391+0000 7fac97fff700 1 -- 172.21.15.42:0/1166206254 learned_addr learned my addr 172.21.15.42:0/1166206254 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:20:52.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:52.391+0000 7fac7ffff700 1 -- 172.21.15.42:0/1166206254 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 53487514 0 0) 0x7fac980a4c70 con 0x7fac981280e0 2019-12-05T05:20:52.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:52.391+0000 7fac7ffff700 1 -- 172.21.15.42:0/1166206254 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac780030b0 con 0x7fac981280e0 2019-12-05T05:20:52.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:52.391+0000 7fac96ffd700 1 -- 172.21.15.42:0/1166206254 >> v1:172.21.15.42:6789/0 conn(0x7fac981280e0 legacy=0x7fac98128510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:20:52.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:52.391+0000 7fac96ffd700 1 -- 172.21.15.42:0/1166206254 >> v1:172.21.15.40:6789/0 conn(0x7fac980a29c0 legacy=0x7fac980a17b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:20:52.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:52.391+0000 7fac7ffff700 1 -- 172.21.15.42:0/1166206254 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1112218341 0 0) 0x7fac980a4250 con 0x7fac980a29c0 2019-12-05T05:20:52.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:52.391+0000 7fac96ffd700 1 -- 172.21.15.42:0/1166206254 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac98128d10 con 0x7fac980a4840 2019-12-05T05:20:52.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:52.391+0000 7fac96ffd700 1 --2- 172.21.15.42:0/1166206254 >> v2:172.21.15.40:3300/0 conn(0x7fac980a4840 0x7fac980a3d00 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fac9808e770 tx=0x7fac8c008ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:52.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:52.391+0000 7fac96ffd700 1 --2- 172.21.15.42:0/1166206254 >> v2:172.21.15.40:3300/0 conn(0x7fac980a4840 0x7fac980a3d00 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fac9808e770 tx=0x7fac8c008ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:52.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:52.391+0000 7fac96ffd700 1 --2- 172.21.15.42:0/1166206254 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fac980a4840 0x7fac980a3d00 secure :-1 s=READY pgs=563 cs=0 l=1 rx=0x7fac9808e770 tx=0x7fac8c008ae0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:20:52.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:52.391+0000 7fac96ffd700 1 --2- 172.21.15.42:0/1166206254 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fac980a4840 0x7fac980a3d00 secure :-1 s=THROTTLE_DONE pgs=563 cs=0 l=1 rx=0x7fac9808e770 tx=0x7fac8c008ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:52.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:52.391+0000 7fac7ffff700 1 -- 172.21.15.42:0/1166206254 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fac8c00f470 con 0x7fac980a4840 2019-12-05T05:20:52.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:52.391+0000 7fac96ffd700 1 --2- 172.21.15.42:0/1166206254 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fac980a4840 0x7fac980a3d00 secure :-1 s=THROTTLE_DONE pgs=563 cs=0 l=1 rx=0x7fac9808e770 tx=0x7fac8c008ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:52.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:52.391+0000 7fac7ffff700 1 -- 172.21.15.42:0/1166206254 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac8c011770 con 0x7fac980a4840 2019-12-05T05:20:52.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:52.391+0000 7fac96ffd700 1 --2- 172.21.15.42:0/1166206254 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fac980a4840 0x7fac980a3d00 secure :-1 s=THROTTLE_DONE pgs=563 cs=0 l=1 rx=0x7fac9808e770 tx=0x7fac8c008ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:52.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:52.391+0000 7fac9d369700 1 -- 172.21.15.42:0/1166206254 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac98130430 con 0x7fac980a4840 2019-12-05T05:20:52.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:52.391+0000 7fac7ffff700 1 -- 172.21.15.42:0/1166206254 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fac8c005730 con 0x7fac980a4840 2019-12-05T05:20:52.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:52.391+0000 7fac9d369700 1 -- 172.21.15.42:0/1166206254 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac981309e0 con 0x7fac980a4840 2019-12-05T05:20:52.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:52.391+0000 7fac96ffd700 1 --2- 172.21.15.42:0/1166206254 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fac980a4840 0x7fac980a3d00 secure :-1 s=THROTTLE_DONE pgs=563 cs=0 l=1 rx=0x7fac9808e770 tx=0x7fac8c008ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:52.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:52.391+0000 7fac7ffff700 1 -- 172.21.15.42:0/1166206254 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fac8c005a10 con 0x7fac980a4840 2019-12-05T05:20:52.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:52.391+0000 7fac96ffd700 1 --2- 172.21.15.42:0/1166206254 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fac980a4840 0x7fac980a3d00 secure :-1 s=THROTTLE_DONE pgs=563 cs=0 l=1 rx=0x7fac9808e770 tx=0x7fac8c008ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:52.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:52.395+0000 7fac7ffff700 1 --2- 172.21.15.42:0/1166206254 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fac7803b1d0 0x7fac7803d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:20:52.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:52.395+0000 7fac7ffff700 1 -- 172.21.15.42:0/1166206254 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 7063+0+0 (secure 0 0 0) 0x7fac8c08f380 con 0x7fac980a4840 2019-12-05T05:20:52.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:52.395+0000 7fac977fe700 1 --2- 172.21.15.42:0/1166206254 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fac7803b1d0 0x7fac7803d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:20:52.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:52.395+0000 7fac977fe700 1 --2- 172.21.15.42:0/1166206254 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fac7803b1d0 0x7fac7803d6e0 crc :-1 s=READY pgs=316 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:20:52.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:52.399+0000 7fac9d369700 1 -- 172.21.15.42:0/1166206254 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac9805d110 con 0x7fac980a4840 2019-12-05T05:20:52.415 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:52.407+0000 7fac96ffd700 1 --2- 172.21.15.42:0/1166206254 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fac980a4840 0x7fac980a3d00 secure :-1 s=THROTTLE_DONE pgs=563 cs=0 l=1 rx=0x7fac9808e770 tx=0x7fac8c008ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:52.416 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:52.407+0000 7fac7ffff700 1 -- 172.21.15.42:0/1166206254 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fac8c0763e0 con 0x7fac980a4840 2019-12-05T05:20:52.584 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:52.579+0000 7fac9d369700 1 -- 172.21.15.42:0/1166206254 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fac9805b8c0 con 0x7fac980a4840 2019-12-05T05:20:52.585 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:52.579+0000 7fac96ffd700 1 --2- 172.21.15.42:0/1166206254 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fac980a4840 0x7fac980a3d00 secure :-1 s=THROTTLE_DONE pgs=563 cs=0 l=1 rx=0x7fac9808e770 tx=0x7fac8c008ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:52.585 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:52.579+0000 7fac7ffff700 1 -- 172.21.15.42:0/1166206254 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 92 v92) v1 ==== 94+0+1772 (secure 0 0 0) 0x7fac8c005d20 con 0x7fac980a4840 2019-12-05T05:20:52.585 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 92 2019-12-05T05:20:52.585 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:20:52.585 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":92,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":92,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:20:47.959813+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5066},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5066":{"gid":5066,"name":"b","rank":0,"incarnation":89,"state":"up:active","state_seq":30,"addr":"172.21.15.40:6833/3061061478","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3061061478},{"type":"v1","addr":"172.21.15.40:6833","nonce":3061061478}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:20:52.587 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:52.579+0000 7fac9d369700 1 -- 172.21.15.42:0/1166206254 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fac7803b1d0 msgr2=0x7fac7803d6e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:20:52.587 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:52.579+0000 7fac9d369700 1 --2- 172.21.15.42:0/1166206254 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fac7803b1d0 0x7fac7803d6e0 crc :-1 s=READY pgs=316 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:20:52.588 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:52.579+0000 7fac9d369700 1 -- 172.21.15.42:0/1166206254 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fac980a4840 msgr2=0x7fac980a3d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:20:52.588 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:52.579+0000 7fac9d369700 1 --2- 172.21.15.42:0/1166206254 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fac980a4840 0x7fac980a3d00 secure :-1 s=READY pgs=563 cs=0 l=1 rx=0x7fac9808e770 tx=0x7fac8c008ae0).stop 2019-12-05T05:20:52.588 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:52.579+0000 7fac9d369700 1 -- 172.21.15.42:0/1166206254 shutdown_connections 2019-12-05T05:20:52.588 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:52.579+0000 7fac9d369700 1 --2- 172.21.15.42:0/1166206254 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fac7803b1d0 0x7fac7803d6e0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:20:52.588 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:52.579+0000 7fac9d369700 1 --2- 172.21.15.42:0/1166206254 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fac980a4840 0x7fac980a3d00 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:20:52.588 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:52.579+0000 7fac9d369700 1 -- 172.21.15.42:0/1166206254 shutdown_connections 2019-12-05T05:20:52.589 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:52.579+0000 7fac9d369700 1 -- 172.21.15.42:0/1166206254 wait complete. 2019-12-05T05:20:52.589 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:52.579+0000 7fac9d369700 1 -- 172.21.15.42:0/1166206254 >> 172.21.15.42:0/1166206254 conn(0x7fac980a9010 msgr2=0x7fac9811b320 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:20:52.597 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2019-12-05T05:20:54.598 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:20:54.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:54.707+0000 7f52a6d17700 1 Processor -- start 2019-12-05T05:20:54.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:54.707+0000 7f52a6d17700 1 -- start start 2019-12-05T05:20:54.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:54.707+0000 7f52a6d17700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f52a00a7740 0x7f52a008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:20:54.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:54.707+0000 7f52a6d17700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f52a009fae0 0x7f52a007eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:20:54.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:54.707+0000 7f52a6d17700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52a00a0710 con 0x7f52a007e720 2019-12-05T05:20:54.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:54.707+0000 7f52a6d17700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52a00a0b00 con 0x7f52a00a7740 2019-12-05T05:20:54.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:54.707+0000 7f52a6d17700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52a008e5b0 con 0x7f52a009fae0 2019-12-05T05:20:54.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:54.707+0000 7f52a5ab5700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f52a009fae0 0x7f52a007eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:20:54.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:54.707+0000 7f52a5ab5700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f52a009fae0 0x7f52a007eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:60016/0 (socket says 172.21.15.42:60016) 2019-12-05T05:20:54.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:54.707+0000 7f52a5ab5700 1 -- 172.21.15.42:0/792428500 learned_addr learned my addr 172.21.15.42:0/792428500 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:20:54.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:54.707+0000 7f52a4ab3700 1 --1- 172.21.15.42:0/792428500 >> v1:172.21.15.42:6789/0 conn(0x7f52a007e720 0x7f52a009f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:60052/0 (socket says 172.21.15.42:60052) 2019-12-05T05:20:54.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:54.711+0000 7f52a52b4700 1 --2- 172.21.15.42:0/792428500 >> v2:172.21.15.40:3300/0 conn(0x7f52a00a7740 0x7f52a008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:20:54.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:54.711+0000 7f52a5ab5700 1 -- 172.21.15.42:0/792428500 >> v2:172.21.15.40:3300/0 conn(0x7f52a00a7740 msgr2=0x7f52a008df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:20:54.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:54.711+0000 7f52a5ab5700 1 --2- 172.21.15.42:0/792428500 >> v2:172.21.15.40:3300/0 conn(0x7f52a00a7740 0x7f52a008df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:20:54.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:54.711+0000 7f52a5ab5700 1 -- 172.21.15.42:0/792428500 >> v1:172.21.15.42:6789/0 conn(0x7f52a007e720 legacy=0x7f52a009f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:20:54.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:54.711+0000 7f52a5ab5700 1 -- 172.21.15.42:0/792428500 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52a007f0a0 con 0x7f52a009fae0 2019-12-05T05:20:54.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:54.711+0000 7f5297fff700 1 -- 172.21.15.42:0/792428500 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2282163401 0 0) 0x7f52a00a0710 con 0x7f52a007e720 2019-12-05T05:20:54.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:54.711+0000 7f52a5ab5700 1 --2- 172.21.15.42:0/792428500 >> v2:172.21.15.42:3300/0 conn(0x7f52a009fae0 0x7f52a007eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f52980032b0 tx=0x7f52980100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:54.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:54.711+0000 7f52a5ab5700 1 --2- 172.21.15.42:0/792428500 >> v2:172.21.15.42:3300/0 conn(0x7f52a009fae0 0x7f52a007eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f52980032b0 tx=0x7f52980100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:54.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:54.711+0000 7f52a5ab5700 1 --2- 172.21.15.42:0/792428500 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f52a009fae0 0x7f52a007eb50 secure :-1 s=READY pgs=703 cs=0 l=1 rx=0x7f52980032b0 tx=0x7f52980100b0).ready entity=mon.1 client_cookie=9817c6d7a1a269fe server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:20:54.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:54.711+0000 7f52a5ab5700 1 --2- 172.21.15.42:0/792428500 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f52a009fae0 0x7f52a007eb50 secure :-1 s=THROTTLE_DONE pgs=703 cs=0 l=1 rx=0x7f52980032b0 tx=0x7f52980100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:54.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:54.711+0000 7f5297fff700 1 -- 172.21.15.42:0/792428500 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5298010d60 con 0x7f52a009fae0 2019-12-05T05:20:54.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:54.711+0000 7f52a5ab5700 1 --2- 172.21.15.42:0/792428500 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f52a009fae0 0x7f52a007eb50 secure :-1 s=THROTTLE_DONE pgs=703 cs=0 l=1 rx=0x7f52980032b0 tx=0x7f52980100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:54.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:54.711+0000 7f52a5ab5700 1 --2- 172.21.15.42:0/792428500 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f52a009fae0 0x7f52a007eb50 secure :-1 s=THROTTLE_DONE pgs=703 cs=0 l=1 rx=0x7f52980032b0 tx=0x7f52980100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:54.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:54.711+0000 7f5297fff700 1 -- 172.21.15.42:0/792428500 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5298010980 con 0x7f52a009fae0 2019-12-05T05:20:54.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:54.711+0000 7f5297fff700 1 -- 172.21.15.42:0/792428500 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f52980114a0 con 0x7f52a009fae0 2019-12-05T05:20:54.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:54.711+0000 7f52a6d17700 1 -- 172.21.15.42:0/792428500 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f52a009fae0 msgr2=0x7f52a007eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:20:54.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:54.711+0000 7f52a6d17700 1 --2- 172.21.15.42:0/792428500 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f52a009fae0 0x7f52a007eb50 secure :-1 s=READY pgs=703 cs=0 l=1 rx=0x7f52980032b0 tx=0x7f52980100b0).stop 2019-12-05T05:20:54.724 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:54.711+0000 7f52a6d17700 1 -- 172.21.15.42:0/792428500 shutdown_connections 2019-12-05T05:20:54.724 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:54.711+0000 7f52a6d17700 1 --2- 172.21.15.42:0/792428500 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f52a009fae0 0x7f52a007eb50 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:20:54.724 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:54.711+0000 7f52a6d17700 1 --2- 172.21.15.42:0/792428500 >> v2:172.21.15.40:3300/0 conn(0x7f52a00a7740 0x7f52a008df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:20:54.724 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:54.711+0000 7f52a6d17700 1 -- 172.21.15.42:0/792428500 shutdown_connections 2019-12-05T05:20:54.724 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:54.711+0000 7f52a6d17700 1 -- 172.21.15.42:0/792428500 wait complete. 2019-12-05T05:20:54.724 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:54.711+0000 7f52a6d17700 1 -- 172.21.15.42:0/792428500 >> 172.21.15.42:0/792428500 conn(0x7f52a00aabe0 msgr2=0x7f52a00a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:20:54.728 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:54.715+0000 7f52a6d17700 1 Processor -- start 2019-12-05T05:20:54.728 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:54.715+0000 7f52a6d17700 1 -- start start 2019-12-05T05:20:54.729 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:54.715+0000 7f52a6d17700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f52a012e140 0x7f52a012e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:20:54.729 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:54.715+0000 7f52a6d17700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52a0139b60 con 0x7f52a01239d0 2019-12-05T05:20:54.729 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:54.715+0000 7f52a6d17700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52a013ada0 con 0x7f52a011a6f0 2019-12-05T05:20:54.729 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:54.715+0000 7f52a6d17700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52a011a490 con 0x7f52a012e140 2019-12-05T05:20:54.729 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:54.715+0000 7f52a5ab5700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f52a011a6f0 0x7f52a011ab20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:60060/0 (socket says 172.21.15.42:60060) 2019-12-05T05:20:54.730 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:54.715+0000 7f52a5ab5700 1 -- 172.21.15.42:0/3227611383 learned_addr learned my addr 172.21.15.42:0/3227611383 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:20:54.731 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:54.715+0000 7f52a4ab3700 1 --2- 172.21.15.42:0/3227611383 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f52a012e140 0x7f52a012e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:20:54.731 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:54.715+0000 7f5295ffb700 1 -- 172.21.15.42:0/3227611383 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1690124364 0 0) 0x7f52a013ada0 con 0x7f52a011a6f0 2019-12-05T05:20:54.731 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:54.715+0000 7f5295ffb700 1 -- 172.21.15.42:0/3227611383 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52800030b0 con 0x7f52a011a6f0 2019-12-05T05:20:54.731 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:54.715+0000 7f5295ffb700 1 -- 172.21.15.42:0/3227611383 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3117070533 0 0) 0x7f52a0139b60 con 0x7f52a01239d0 2019-12-05T05:20:54.731 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:54.715+0000 7f5295ffb700 1 -- 172.21.15.42:0/3227611383 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52a013ada0 con 0x7f52a01239d0 2019-12-05T05:20:54.731 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:54.715+0000 7f52a4ab3700 1 -- 172.21.15.42:0/3227611383 >> v1:172.21.15.42:6789/0 conn(0x7f52a011a6f0 legacy=0x7f52a011ab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:20:54.731 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:54.715+0000 7f5295ffb700 1 -- 172.21.15.42:0/3227611383 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 3849961591 0 0) 0x7f52800030b0 con 0x7f52a011a6f0 2019-12-05T05:20:54.732 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:54.715+0000 7f52a4ab3700 1 -- 172.21.15.42:0/3227611383 >> v1:172.21.15.40:6789/0 conn(0x7f52a01239d0 legacy=0x7f52a011b950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:20:54.732 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:54.715+0000 7f52a4ab3700 1 -- 172.21.15.42:0/3227611383 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52a013bfe0 con 0x7f52a012e140 2019-12-05T05:20:54.732 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:54.715+0000 7f52a4ab3700 1 --2- 172.21.15.42:0/3227611383 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f52a012e140 0x7f52a012e570 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f5298008510 tx=0x7f5298010580).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:54.732 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:54.715+0000 7f52a4ab3700 1 --2- 172.21.15.42:0/3227611383 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f52a012e140 0x7f52a012e570 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f5298008510 tx=0x7f5298010580).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:54.732 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:54.715+0000 7f52a4ab3700 1 --2- 172.21.15.42:0/3227611383 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f52a012e140 0x7f52a012e570 secure :-1 s=READY pgs=150 cs=0 l=1 rx=0x7f5298008510 tx=0x7f5298010580).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:20:54.732 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:54.715+0000 7f52a4ab3700 1 --2- 172.21.15.42:0/3227611383 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f52a012e140 0x7f52a012e570 secure :-1 s=THROTTLE_DONE pgs=150 cs=0 l=1 rx=0x7f5298008510 tx=0x7f5298010580).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:54.733 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:54.715+0000 7f5295ffb700 1 -- 172.21.15.42:0/3227611383 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f52980087f0 con 0x7f52a012e140 2019-12-05T05:20:54.733 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:54.715+0000 7f52a4ab3700 1 --2- 172.21.15.42:0/3227611383 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f52a012e140 0x7f52a012e570 secure :-1 s=THROTTLE_DONE pgs=150 cs=0 l=1 rx=0x7f5298008510 tx=0x7f5298010580).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:54.733 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:54.715+0000 7f52a4ab3700 1 --2- 172.21.15.42:0/3227611383 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f52a012e140 0x7f52a012e570 secure :-1 s=THROTTLE_DONE pgs=150 cs=0 l=1 rx=0x7f5298008510 tx=0x7f5298010580).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:54.733 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:54.715+0000 7f5295ffb700 1 -- 172.21.15.42:0/3227611383 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52980110c0 con 0x7f52a012e140 2019-12-05T05:20:54.733 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:54.715+0000 7f52a6d17700 1 -- 172.21.15.42:0/3227611383 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52a0139d90 con 0x7f52a012e140 2019-12-05T05:20:54.733 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:54.715+0000 7f5295ffb700 1 -- 172.21.15.42:0/3227611383 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5290006a40 con 0x7f52a012e140 2019-12-05T05:20:54.733 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:54.715+0000 7f52a6d17700 1 -- 172.21.15.42:0/3227611383 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52a013a310 con 0x7f52a012e140 2019-12-05T05:20:54.733 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:54.719+0000 7f52a4ab3700 1 --2- 172.21.15.42:0/3227611383 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f52a012e140 0x7f52a012e570 secure :-1 s=THROTTLE_DONE pgs=150 cs=0 l=1 rx=0x7f5298008510 tx=0x7f5298010580).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:54.734 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:54.719+0000 7f5295ffb700 1 -- 172.21.15.42:0/3227611383 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f52900110f0 con 0x7f52a012e140 2019-12-05T05:20:54.734 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:54.719+0000 7f52a4ab3700 1 --2- 172.21.15.42:0/3227611383 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f52a012e140 0x7f52a012e570 secure :-1 s=THROTTLE_DONE pgs=150 cs=0 l=1 rx=0x7f5298008510 tx=0x7f5298010580).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:54.734 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:54.719+0000 7f5295ffb700 1 --2- 172.21.15.42:0/3227611383 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f528003b440 0x7f528003d950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:20:54.734 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:54.719+0000 7f5295ffb700 1 -- 172.21.15.42:0/3227611383 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 7063+0+0 (secure 0 0 0) 0x7f529004aad0 con 0x7f52a012e140 2019-12-05T05:20:54.734 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:54.719+0000 7f52a52b4700 1 --2- 172.21.15.42:0/3227611383 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f528003b440 0x7f528003d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:20:54.734 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:54.719+0000 7f52a52b4700 1 --2- 172.21.15.42:0/3227611383 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f528003b440 0x7f528003d950 crc :-1 s=READY pgs=317 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:20:54.735 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:54.723+0000 7f52a6d17700 1 -- 172.21.15.42:0/3227611383 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52a005d110 con 0x7f52a012e140 2019-12-05T05:20:54.738 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:54.731+0000 7f52a4ab3700 1 --2- 172.21.15.42:0/3227611383 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f52a012e140 0x7f52a012e570 secure :-1 s=THROTTLE_DONE pgs=150 cs=0 l=1 rx=0x7f5298008510 tx=0x7f5298010580).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:54.738 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:54.731+0000 7f5295ffb700 1 -- 172.21.15.42:0/3227611383 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f529004ade0 con 0x7f52a012e140 2019-12-05T05:20:54.909 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:54.903+0000 7f52a6d17700 1 -- 172.21.15.42:0/3227611383 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f52a011d4b0 con 0x7f52a012e140 2019-12-05T05:20:54.910 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:54.903+0000 7f52a4ab3700 1 --2- 172.21.15.42:0/3227611383 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f52a012e140 0x7f52a012e570 secure :-1 s=THROTTLE_DONE pgs=150 cs=0 l=1 rx=0x7f5298008510 tx=0x7f5298010580).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:54.910 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:54.903+0000 7f5295ffb700 1 -- 172.21.15.42:0/3227611383 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 92 v92) v1 ==== 94+0+1772 (secure 0 0 0) 0x7f5290031b50 con 0x7f52a012e140 2019-12-05T05:20:54.912 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 92 2019-12-05T05:20:54.912 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:20:54.912 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":92,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":92,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:20:47.959813+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5066},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5066":{"gid":5066,"name":"b","rank":0,"incarnation":89,"state":"up:active","state_seq":30,"addr":"172.21.15.40:6833/3061061478","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3061061478},{"type":"v1","addr":"172.21.15.40:6833","nonce":3061061478}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:20:54.913 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:54.907+0000 7f52a6d17700 1 -- 172.21.15.42:0/3227611383 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f528003b440 msgr2=0x7f528003d950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:20:54.914 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:54.907+0000 7f52a6d17700 1 --2- 172.21.15.42:0/3227611383 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f528003b440 0x7f528003d950 crc :-1 s=READY pgs=317 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:20:54.914 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:54.907+0000 7f52a6d17700 1 -- 172.21.15.42:0/3227611383 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f52a012e140 msgr2=0x7f52a012e570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:20:54.914 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:54.907+0000 7f52a6d17700 1 --2- 172.21.15.42:0/3227611383 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f52a012e140 0x7f52a012e570 secure :-1 s=READY pgs=150 cs=0 l=1 rx=0x7f5298008510 tx=0x7f5298010580).stop 2019-12-05T05:20:54.914 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:54.907+0000 7f52a6d17700 1 -- 172.21.15.42:0/3227611383 shutdown_connections 2019-12-05T05:20:54.914 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:54.907+0000 7f52a6d17700 1 --2- 172.21.15.42:0/3227611383 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f528003b440 0x7f528003d950 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:20:54.914 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:54.907+0000 7f52a6d17700 1 --2- 172.21.15.42:0/3227611383 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f52a012e140 0x7f52a012e570 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:20:54.914 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:54.907+0000 7f52a6d17700 1 -- 172.21.15.42:0/3227611383 shutdown_connections 2019-12-05T05:20:54.915 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:54.907+0000 7f52a6d17700 1 -- 172.21.15.42:0/3227611383 wait complete. 2019-12-05T05:20:54.915 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:54.907+0000 7f52a6d17700 1 -- 172.21.15.42:0/3227611383 >> 172.21.15.42:0/3227611383 conn(0x7f52a011cf10 msgr2=0x7f52a007e560 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:20:54.923 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2019-12-05T05:20:56.925 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:20:57.033 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:57.019+0000 7ff0f42a6700 1 Processor -- start 2019-12-05T05:20:57.033 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:57.019+0000 7ff0f42a6700 1 -- start start 2019-12-05T05:20:57.033 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:57.023+0000 7ff0f42a6700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff0ec08df80 0x7ff0ec09f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:20:57.033 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:57.023+0000 7ff0f42a6700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0ec09fc60 con 0x7ff0ec0a7740 2019-12-05T05:20:57.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:57.023+0000 7ff0f42a6700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0ec0a0710 con 0x7ff0ec09f830 2019-12-05T05:20:57.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:57.023+0000 7ff0f42a6700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0ec0a0b00 con 0x7ff0ec08df80 2019-12-05T05:20:57.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:57.023+0000 7ff0f2843700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7ff0ec0a7740 0x7ff0ec07e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:41294/0 (socket says 172.21.15.42:41294) 2019-12-05T05:20:57.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:57.023+0000 7ff0f2843700 1 -- 172.21.15.42:0/2917626937 learned_addr learned my addr 172.21.15.42:0/2917626937 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:20:57.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:57.023+0000 7ff0f2042700 1 --2- 172.21.15.42:0/2917626937 >> v2:172.21.15.40:3300/0 conn(0x7ff0ec08df80 0x7ff0ec09f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:20:57.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:57.023+0000 7ff0f1841700 1 -- 172.21.15.42:0/2917626937 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3926642533 0 0) 0x7ff0ec09fc60 con 0x7ff0ec0a7740 2019-12-05T05:20:57.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:57.023+0000 7ff0f1841700 1 -- 172.21.15.42:0/2917626937 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0d80030b0 con 0x7ff0ec0a7740 2019-12-05T05:20:57.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:57.023+0000 7ff0f1841700 1 -- 172.21.15.42:0/2917626937 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4167297310 0 0) 0x7ff0ec0a0710 con 0x7ff0ec09f830 2019-12-05T05:20:57.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:57.023+0000 7ff0f1841700 1 -- 172.21.15.42:0/2917626937 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0ec09fc60 con 0x7ff0ec09f830 2019-12-05T05:20:57.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:57.023+0000 7ff0f2042700 1 -- 172.21.15.42:0/2917626937 >> v1:172.21.15.42:6789/0 conn(0x7ff0ec09f830 legacy=0x7ff0ec14efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:20:57.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:57.023+0000 7ff0f2042700 1 -- 172.21.15.42:0/2917626937 >> v1:172.21.15.40:6789/0 conn(0x7ff0ec0a7740 legacy=0x7ff0ec07e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:20:57.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:57.023+0000 7ff0f2042700 1 -- 172.21.15.42:0/2917626937 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0ec07ef20 con 0x7ff0ec08df80 2019-12-05T05:20:57.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:57.023+0000 7ff0f2042700 1 --2- 172.21.15.42:0/2917626937 >> v2:172.21.15.40:3300/0 conn(0x7ff0ec08df80 0x7ff0ec09f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7ff0dc008790 tx=0x7ff0dc00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:57.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:57.023+0000 7ff0f2042700 1 --2- 172.21.15.42:0/2917626937 >> v2:172.21.15.40:3300/0 conn(0x7ff0ec08df80 0x7ff0ec09f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7ff0dc008790 tx=0x7ff0dc00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:57.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:57.023+0000 7ff0f2042700 1 --2- 172.21.15.42:0/2917626937 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff0ec08df80 0x7ff0ec09f2e0 secure :-1 s=READY pgs=566 cs=0 l=1 rx=0x7ff0dc008790 tx=0x7ff0dc00fbf0).ready entity=mon.0 client_cookie=a08702e27d832d2d server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:20:57.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:57.023+0000 7ff0f2042700 1 --2- 172.21.15.42:0/2917626937 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff0ec08df80 0x7ff0ec09f2e0 secure :-1 s=THROTTLE_DONE pgs=566 cs=0 l=1 rx=0x7ff0dc008790 tx=0x7ff0dc00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:57.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:57.023+0000 7ff0f1841700 1 -- 172.21.15.42:0/2917626937 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff0dc011270 con 0x7ff0ec08df80 2019-12-05T05:20:57.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:57.023+0000 7ff0f2042700 1 --2- 172.21.15.42:0/2917626937 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff0ec08df80 0x7ff0ec09f2e0 secure :-1 s=THROTTLE_DONE pgs=566 cs=0 l=1 rx=0x7ff0dc008790 tx=0x7ff0dc00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:57.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:57.023+0000 7ff0f2042700 1 --2- 172.21.15.42:0/2917626937 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff0ec08df80 0x7ff0ec09f2e0 secure :-1 s=THROTTLE_DONE pgs=566 cs=0 l=1 rx=0x7ff0dc008790 tx=0x7ff0dc00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:57.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:57.023+0000 7ff0f1841700 1 -- 172.21.15.42:0/2917626937 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0dc008bc0 con 0x7ff0ec08df80 2019-12-05T05:20:57.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:57.023+0000 7ff0f1841700 1 -- 172.21.15.42:0/2917626937 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff0dc011850 con 0x7ff0ec08df80 2019-12-05T05:20:57.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:57.023+0000 7ff0f42a6700 1 -- 172.21.15.42:0/2917626937 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff0ec08df80 msgr2=0x7ff0ec09f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:20:57.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:57.023+0000 7ff0f42a6700 1 --2- 172.21.15.42:0/2917626937 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff0ec08df80 0x7ff0ec09f2e0 secure :-1 s=READY pgs=566 cs=0 l=1 rx=0x7ff0dc008790 tx=0x7ff0dc00fbf0).stop 2019-12-05T05:20:57.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:57.027+0000 7ff0f42a6700 1 -- 172.21.15.42:0/2917626937 shutdown_connections 2019-12-05T05:20:57.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:57.027+0000 7ff0f42a6700 1 --2- 172.21.15.42:0/2917626937 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff0ec08df80 0x7ff0ec09f2e0 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:20:57.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:57.027+0000 7ff0f42a6700 1 -- 172.21.15.42:0/2917626937 shutdown_connections 2019-12-05T05:20:57.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:57.027+0000 7ff0f42a6700 1 -- 172.21.15.42:0/2917626937 wait complete. 2019-12-05T05:20:57.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:57.027+0000 7ff0f42a6700 1 -- 172.21.15.42:0/2917626937 >> 172.21.15.42:0/2917626937 conn(0x7ff0ec0aabe0 msgr2=0x7ff0ec0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:20:57.042 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:57.027+0000 7ff0f42a6700 1 Processor -- start 2019-12-05T05:20:57.042 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:57.027+0000 7ff0f42a6700 1 -- start start 2019-12-05T05:20:57.042 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:57.031+0000 7ff0f42a6700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff0ec12cd20 0x7ff0ec12f160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:20:57.042 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:57.031+0000 7ff0f42a6700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0ec0a1510 con 0x7ff0ec142530 2019-12-05T05:20:57.042 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:57.031+0000 7ff0f42a6700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0ec1268d0 con 0x7ff0ec114330 2019-12-05T05:20:57.043 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:57.031+0000 7ff0f42a6700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0ec143b80 con 0x7ff0ec12cd20 2019-12-05T05:20:57.043 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:57.031+0000 7ff0f2042700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7ff0ec142530 0x7ff0ec126e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:41302/0 (socket says 172.21.15.42:41302) 2019-12-05T05:20:57.043 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:57.031+0000 7ff0f2042700 1 -- 172.21.15.42:0/474034876 learned_addr learned my addr 172.21.15.42:0/474034876 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:20:57.043 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:57.031+0000 7ff0f3044700 1 --2- 172.21.15.42:0/474034876 >> v2:172.21.15.40:3300/0 conn(0x7ff0ec12cd20 0x7ff0ec12f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:20:57.043 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:57.031+0000 7ff0e37fe700 1 -- 172.21.15.42:0/474034876 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2321061962 0 0) 0x7ff0ec0a1510 con 0x7ff0ec142530 2019-12-05T05:20:57.043 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:57.031+0000 7ff0e37fe700 1 -- 172.21.15.42:0/474034876 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0d40030b0 con 0x7ff0ec142530 2019-12-05T05:20:57.043 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:57.031+0000 7ff0f3044700 1 -- 172.21.15.42:0/474034876 >> v1:172.21.15.42:6789/0 conn(0x7ff0ec114330 legacy=0x7ff0ec143380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:20:57.044 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:57.031+0000 7ff0f3044700 1 -- 172.21.15.42:0/474034876 >> v1:172.21.15.40:6789/0 conn(0x7ff0ec142530 legacy=0x7ff0ec126e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:20:57.044 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:57.031+0000 7ff0f3044700 1 -- 172.21.15.42:0/474034876 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0ec127610 con 0x7ff0ec12cd20 2019-12-05T05:20:57.044 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:57.031+0000 7ff0f3044700 1 --2- 172.21.15.42:0/474034876 >> v2:172.21.15.40:3300/0 conn(0x7ff0ec12cd20 0x7ff0ec12f160 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7ff0ec1167f0 tx=0x7ff0e4010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:57.044 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:57.031+0000 7ff0f3044700 1 --2- 172.21.15.42:0/474034876 >> v2:172.21.15.40:3300/0 conn(0x7ff0ec12cd20 0x7ff0ec12f160 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7ff0ec1167f0 tx=0x7ff0e4010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:57.044 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:57.031+0000 7ff0f3044700 1 --2- 172.21.15.42:0/474034876 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff0ec12cd20 0x7ff0ec12f160 secure :-1 s=READY pgs=568 cs=0 l=1 rx=0x7ff0ec1167f0 tx=0x7ff0e4010aa0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:20:57.044 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:57.031+0000 7ff0f3044700 1 --2- 172.21.15.42:0/474034876 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff0ec12cd20 0x7ff0ec12f160 secure :-1 s=THROTTLE_DONE pgs=568 cs=0 l=1 rx=0x7ff0ec1167f0 tx=0x7ff0e4010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:57.044 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:57.031+0000 7ff0e37fe700 1 -- 172.21.15.42:0/474034876 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff0e4008160 con 0x7ff0ec12cd20 2019-12-05T05:20:57.045 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:57.031+0000 7ff0f42a6700 1 -- 172.21.15.42:0/474034876 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0ec145350 con 0x7ff0ec12cd20 2019-12-05T05:20:57.045 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:57.031+0000 7ff0f3044700 1 --2- 172.21.15.42:0/474034876 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff0ec12cd20 0x7ff0ec12f160 secure :-1 s=THROTTLE_DONE pgs=568 cs=0 l=1 rx=0x7ff0ec1167f0 tx=0x7ff0e4010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:57.045 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:57.031+0000 7ff0f42a6700 1 -- 172.21.15.42:0/474034876 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0ec12f6b0 con 0x7ff0ec12cd20 2019-12-05T05:20:57.045 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:57.031+0000 7ff0e37fe700 1 -- 172.21.15.42:0/474034876 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0e4006ad0 con 0x7ff0ec12cd20 2019-12-05T05:20:57.045 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:57.031+0000 7ff0f3044700 1 --2- 172.21.15.42:0/474034876 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff0ec12cd20 0x7ff0ec12f160 secure :-1 s=THROTTLE_DONE pgs=568 cs=0 l=1 rx=0x7ff0ec1167f0 tx=0x7ff0e4010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:57.045 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:57.031+0000 7ff0e37fe700 1 -- 172.21.15.42:0/474034876 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff0e4008780 con 0x7ff0ec12cd20 2019-12-05T05:20:57.045 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:57.031+0000 7ff0f3044700 1 --2- 172.21.15.42:0/474034876 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff0ec12cd20 0x7ff0ec12f160 secure :-1 s=THROTTLE_DONE pgs=568 cs=0 l=1 rx=0x7ff0ec1167f0 tx=0x7ff0e4010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:57.046 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:57.035+0000 7ff0e37fe700 1 -- 172.21.15.42:0/474034876 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7ff0e4008ae0 con 0x7ff0ec12cd20 2019-12-05T05:20:57.046 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:57.035+0000 7ff0f3044700 1 --2- 172.21.15.42:0/474034876 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff0ec12cd20 0x7ff0ec12f160 secure :-1 s=THROTTLE_DONE pgs=568 cs=0 l=1 rx=0x7ff0ec1167f0 tx=0x7ff0e4010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:57.046 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:57.035+0000 7ff0e37fe700 1 --2- 172.21.15.42:0/474034876 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff0d403b1e0 0x7ff0d403d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:20:57.046 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:57.035+0000 7ff0e37fe700 1 -- 172.21.15.42:0/474034876 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 7063+0+0 (secure 0 0 0) 0x7ff0e404b480 con 0x7ff0ec12cd20 2019-12-05T05:20:57.047 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:57.035+0000 7ff0f2843700 1 --2- 172.21.15.42:0/474034876 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff0d403b1e0 0x7ff0d403d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:20:57.047 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:57.035+0000 7ff0f2843700 1 --2- 172.21.15.42:0/474034876 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff0d403b1e0 0x7ff0d403d6f0 crc :-1 s=READY pgs=319 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:20:57.047 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:57.035+0000 7ff0f42a6700 1 -- 172.21.15.42:0/474034876 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0ec05d110 con 0x7ff0ec12cd20 2019-12-05T05:20:57.049 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:57.043+0000 7ff0f3044700 1 --2- 172.21.15.42:0/474034876 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff0ec12cd20 0x7ff0ec12f160 secure :-1 s=THROTTLE_DONE pgs=568 cs=0 l=1 rx=0x7ff0ec1167f0 tx=0x7ff0e4010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:57.049 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:57.043+0000 7ff0e37fe700 1 -- 172.21.15.42:0/474034876 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7ff0e40324e0 con 0x7ff0ec12cd20 2019-12-05T05:20:57.221 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:57.215+0000 7ff0f42a6700 1 -- 172.21.15.42:0/474034876 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff0ec05b8c0 con 0x7ff0ec12cd20 2019-12-05T05:20:57.222 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:57.215+0000 7ff0f3044700 1 --2- 172.21.15.42:0/474034876 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff0ec12cd20 0x7ff0ec12f160 secure :-1 s=THROTTLE_DONE pgs=568 cs=0 l=1 rx=0x7ff0ec1167f0 tx=0x7ff0e4010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:20:57.222 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:57.215+0000 7ff0e37fe700 1 -- 172.21.15.42:0/474034876 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 93 v93) v1 ==== 94+0+2108 (secure 0 0 0) 0x7ff0e4032740 con 0x7ff0ec12cd20 2019-12-05T05:20:57.224 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:20:57.224 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":93,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5123,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/1214659805","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1214659805},{"type":"v1","addr":"172.21.15.42:6835","nonce":1214659805}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":93}],"filesystems":[{"mdsmap":{"epoch":92,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:20:47.959813+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5066},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5066":{"gid":5066,"name":"b","rank":0,"incarnation":89,"state":"up:active","state_seq":30,"addr":"172.21.15.40:6833/3061061478","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3061061478},{"type":"v1","addr":"172.21.15.40:6833","nonce":3061061478}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:20:57.224 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 93 2019-12-05T05:20:57.226 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:57.219+0000 7ff0f42a6700 1 -- 172.21.15.42:0/474034876 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff0d403b1e0 msgr2=0x7ff0d403d6f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:20:57.227 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:57.219+0000 7ff0f42a6700 1 --2- 172.21.15.42:0/474034876 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff0d403b1e0 0x7ff0d403d6f0 crc :-1 s=READY pgs=319 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:20:57.227 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:57.219+0000 7ff0f42a6700 1 -- 172.21.15.42:0/474034876 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff0ec12cd20 msgr2=0x7ff0ec12f160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:20:57.227 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:57.219+0000 7ff0f42a6700 1 --2- 172.21.15.42:0/474034876 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff0ec12cd20 0x7ff0ec12f160 secure :-1 s=READY pgs=568 cs=0 l=1 rx=0x7ff0ec1167f0 tx=0x7ff0e4010aa0).stop 2019-12-05T05:20:57.227 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:57.219+0000 7ff0f42a6700 1 -- 172.21.15.42:0/474034876 shutdown_connections 2019-12-05T05:20:57.227 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:57.219+0000 7ff0f42a6700 1 --2- 172.21.15.42:0/474034876 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff0d403b1e0 0x7ff0d403d6f0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:20:57.228 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:57.219+0000 7ff0f42a6700 1 --2- 172.21.15.42:0/474034876 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff0ec12cd20 0x7ff0ec12f160 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:20:57.228 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:57.219+0000 7ff0f42a6700 1 -- 172.21.15.42:0/474034876 shutdown_connections 2019-12-05T05:20:57.228 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:57.219+0000 7ff0f42a6700 1 -- 172.21.15.42:0/474034876 wait complete. 2019-12-05T05:20:57.228 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:20:57.219+0000 7ff0f42a6700 1 -- 172.21.15.42:0/474034876 >> 172.21.15.42:0/474034876 conn(0x7ff0ec08fde0 msgr2=0x7ff0ec0a9810 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:20:57.237 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a reported in up:standby state 2019-12-05T05:20:57.237 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 112 secs before thrashing 2019-12-05T05:21:07.495 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:21:07.499 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:21:37.936 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:21:37.940 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:22:07.997 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:22:08.002 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:22:38.056 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:22:38.060 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:22:49.238 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:22:49.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.326+0000 7f8c07205700 1 Processor -- start 2019-12-05T05:22:49.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.326+0000 7f8c07205700 1 -- start start 2019-12-05T05:22:49.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.326+0000 7f8c07205700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8c000a7740 0x7f8c0008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:22:49.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.326+0000 7f8c07205700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f8c0007e720 0x7f8c0007eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:22:49.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.326+0000 7f8c07205700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c000a0710 con 0x7f8c0009f2e0 2019-12-05T05:22:49.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.326+0000 7f8c07205700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c000a0b00 con 0x7f8c0007e720 2019-12-05T05:22:49.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.326+0000 7f8c07205700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c0008e5b0 con 0x7f8c000a7740 2019-12-05T05:22:49.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.326+0000 7f8c04fa1700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f8c0007e720 0x7f8c0007eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:22:49.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.326+0000 7f8c04fa1700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f8c0007e720 0x7f8c0007eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:60514/0 (socket says 172.21.15.42:60514) 2019-12-05T05:22:49.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.326+0000 7f8c04fa1700 1 -- 172.21.15.42:0/3025101852 learned_addr learned my addr 172.21.15.42:0/3025101852 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:22:49.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.326+0000 7f8c057a2700 1 --2- 172.21.15.42:0/3025101852 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8c000a7740 0x7f8c0008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:22:49.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.326+0000 7f8c04fa1700 1 -- 172.21.15.42:0/3025101852 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8c000a7740 msgr2=0x7f8c0008df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:22:49.341 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.326+0000 7f8c04fa1700 1 --2- 172.21.15.42:0/3025101852 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8c000a7740 0x7f8c0008df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:22:49.341 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.326+0000 7f8c04fa1700 1 -- 172.21.15.42:0/3025101852 >> v1:172.21.15.40:6789/0 conn(0x7f8c0009f2e0 legacy=0x7f8c0009f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:22:49.341 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.326+0000 7f8bfffff700 1 -- 172.21.15.42:0/3025101852 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2637605702 0 0) 0x7f8c000a0710 con 0x7f8c0009f2e0 2019-12-05T05:22:49.341 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.326+0000 7f8c04fa1700 1 -- 172.21.15.42:0/3025101852 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c0007f0a0 con 0x7f8c0007e720 2019-12-05T05:22:49.341 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.326+0000 7f8c04fa1700 1 --2- 172.21.15.42:0/3025101852 >> v2:172.21.15.42:3300/0 conn(0x7f8c0007e720 0x7f8c0007eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f8bf0008790 tx=0x7f8bf000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:22:49.341 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.326+0000 7f8c04fa1700 1 --2- 172.21.15.42:0/3025101852 >> v2:172.21.15.42:3300/0 conn(0x7f8c0007e720 0x7f8c0007eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f8bf0008790 tx=0x7f8bf000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:22:49.342 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.326+0000 7f8c04fa1700 1 --2- 172.21.15.42:0/3025101852 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8c0007e720 0x7f8c0007eb50 secure :-1 s=READY pgs=715 cs=0 l=1 rx=0x7f8bf0008790 tx=0x7f8bf000fd30).ready entity=mon.1 client_cookie=82bc1117071e074b server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:22:49.342 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.330+0000 7f8c04fa1700 1 --2- 172.21.15.42:0/3025101852 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8c0007e720 0x7f8c0007eb50 secure :-1 s=THROTTLE_DONE pgs=715 cs=0 l=1 rx=0x7f8bf0008790 tx=0x7f8bf000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:22:49.342 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.330+0000 7f8bfffff700 1 -- 172.21.15.42:0/3025101852 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8bf00113b0 con 0x7f8c0007e720 2019-12-05T05:22:49.342 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.330+0000 7f8c04fa1700 1 --2- 172.21.15.42:0/3025101852 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8c0007e720 0x7f8c0007eb50 secure :-1 s=THROTTLE_DONE pgs=715 cs=0 l=1 rx=0x7f8bf0008790 tx=0x7f8bf000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:22:49.342 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.330+0000 7f8c04fa1700 1 --2- 172.21.15.42:0/3025101852 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8c0007e720 0x7f8c0007eb50 secure :-1 s=THROTTLE_DONE pgs=715 cs=0 l=1 rx=0x7f8bf0008790 tx=0x7f8bf000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:22:49.342 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.330+0000 7f8bfffff700 1 -- 172.21.15.42:0/3025101852 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bf0008bb0 con 0x7f8c0007e720 2019-12-05T05:22:49.343 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.330+0000 7f8bfffff700 1 -- 172.21.15.42:0/3025101852 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8bf0011990 con 0x7f8c0007e720 2019-12-05T05:22:49.343 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.330+0000 7f8c07205700 1 -- 172.21.15.42:0/3025101852 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8c0007e720 msgr2=0x7f8c0007eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:22:49.343 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.330+0000 7f8c07205700 1 --2- 172.21.15.42:0/3025101852 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8c0007e720 0x7f8c0007eb50 secure :-1 s=READY pgs=715 cs=0 l=1 rx=0x7f8bf0008790 tx=0x7f8bf000fd30).stop 2019-12-05T05:22:49.343 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.330+0000 7f8c07205700 1 -- 172.21.15.42:0/3025101852 shutdown_connections 2019-12-05T05:22:49.343 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.330+0000 7f8c07205700 1 --2- 172.21.15.42:0/3025101852 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8c000a7740 0x7f8c0008df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:22:49.343 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.330+0000 7f8c07205700 1 --2- 172.21.15.42:0/3025101852 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8c0007e720 0x7f8c0007eb50 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:22:49.343 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.330+0000 7f8c07205700 1 -- 172.21.15.42:0/3025101852 shutdown_connections 2019-12-05T05:22:49.344 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.330+0000 7f8c07205700 1 -- 172.21.15.42:0/3025101852 wait complete. 2019-12-05T05:22:49.344 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.330+0000 7f8c07205700 1 -- 172.21.15.42:0/3025101852 >> 172.21.15.42:0/3025101852 conn(0x7f8c000aabe0 msgr2=0x7f8c000a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:22:49.344 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.330+0000 7f8c07205700 1 Processor -- start 2019-12-05T05:22:49.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.330+0000 7f8c07205700 1 -- start start 2019-12-05T05:22:49.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.334+0000 7f8c07205700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8c00142f70 0x7f8c00064740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:22:49.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.334+0000 7f8c07205700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8c00059f50 0x7f8c0007b830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:22:49.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.334+0000 7f8c07205700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f8c0007bd80 0x7f8c00063430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:22:49.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.334+0000 7f8c07205700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c00064ec0 con 0x7f8c00142f70 2019-12-05T05:22:49.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.334+0000 7f8c07205700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c00065160 con 0x7f8c0007bd80 2019-12-05T05:22:49.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.334+0000 7f8c07205700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c00065400 con 0x7f8c00059f50 2019-12-05T05:22:49.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.334+0000 7f8c05fa3700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f8c0007bd80 0x7f8c00063430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:22:49.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.334+0000 7f8c05fa3700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f8c0007bd80 0x7f8c00063430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:60520/0 (socket says 172.21.15.42:60520) 2019-12-05T05:22:49.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.334+0000 7f8c05fa3700 1 -- 172.21.15.42:0/2714450405 learned_addr learned my addr 172.21.15.42:0/2714450405 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:22:49.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.334+0000 7f8c04fa1700 1 --2- 172.21.15.42:0/2714450405 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8c00059f50 0x7f8c0007b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:22:49.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.334+0000 7f8c057a2700 1 --2- 172.21.15.42:0/2714450405 >> v2:172.21.15.40:3300/0 conn(0x7f8c00142f70 0x7f8c00064740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:22:49.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.334+0000 7f8c05fa3700 1 -- 172.21.15.42:0/2714450405 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8c00059f50 msgr2=0x7f8c0007b830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:22:49.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.334+0000 7f8c05fa3700 1 --2- 172.21.15.42:0/2714450405 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8c00059f50 0x7f8c0007b830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:22:49.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.334+0000 7f8c05fa3700 1 -- 172.21.15.42:0/2714450405 >> v2:172.21.15.40:3300/0 conn(0x7f8c00142f70 msgr2=0x7f8c00064740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:22:49.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.334+0000 7f8c05fa3700 1 --2- 172.21.15.42:0/2714450405 >> v2:172.21.15.40:3300/0 conn(0x7f8c00142f70 0x7f8c00064740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:22:49.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.334+0000 7f8c05fa3700 1 -- 172.21.15.42:0/2714450405 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c0007c1b0 con 0x7f8c0007bd80 2019-12-05T05:22:49.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.334+0000 7f8c05fa3700 1 --2- 172.21.15.42:0/2714450405 >> v2:172.21.15.42:3300/0 conn(0x7f8c0007bd80 0x7f8c00063430 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f8bf8001210 tx=0x7f8bf8010bd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:22:49.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.334+0000 7f8c05fa3700 1 --2- 172.21.15.42:0/2714450405 >> v2:172.21.15.42:3300/0 conn(0x7f8c0007bd80 0x7f8c00063430 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f8bf8001210 tx=0x7f8bf8010bd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:22:49.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.334+0000 7f8c05fa3700 1 --2- 172.21.15.42:0/2714450405 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8c0007bd80 0x7f8c00063430 secure :-1 s=READY pgs=716 cs=0 l=1 rx=0x7f8bf8001210 tx=0x7f8bf8010bd0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:22:49.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.334+0000 7f8c05fa3700 1 --2- 172.21.15.42:0/2714450405 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8c0007bd80 0x7f8c00063430 secure :-1 s=THROTTLE_DONE pgs=716 cs=0 l=1 rx=0x7f8bf8001210 tx=0x7f8bf8010bd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:22:49.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.334+0000 7f8c05fa3700 1 --2- 172.21.15.42:0/2714450405 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8c0007bd80 0x7f8c00063430 secure :-1 s=THROTTLE_DONE pgs=716 cs=0 l=1 rx=0x7f8bf8001210 tx=0x7f8bf8010bd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:22:49.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.334+0000 7f8c05fa3700 1 --2- 172.21.15.42:0/2714450405 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8c0007bd80 0x7f8c00063430 secure :-1 s=THROTTLE_DONE pgs=716 cs=0 l=1 rx=0x7f8bf8001210 tx=0x7f8bf8010bd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:22:49.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.334+0000 7f8bfdffb700 1 -- 172.21.15.42:0/2714450405 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8bf8006dc0 con 0x7f8c0007bd80 2019-12-05T05:22:49.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.334+0000 7f8bfdffb700 1 -- 172.21.15.42:0/2714450405 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bf80069e0 con 0x7f8c0007bd80 2019-12-05T05:22:49.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.334+0000 7f8c07205700 1 -- 172.21.15.42:0/2714450405 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c00063980 con 0x7f8c0007bd80 2019-12-05T05:22:49.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.334+0000 7f8bfdffb700 1 -- 172.21.15.42:0/2714450405 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8bf8012670 con 0x7f8c0007bd80 2019-12-05T05:22:49.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.334+0000 7f8c07205700 1 -- 172.21.15.42:0/2714450405 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c00063f00 con 0x7f8c0007bd80 2019-12-05T05:22:49.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.334+0000 7f8c05fa3700 1 --2- 172.21.15.42:0/2714450405 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8c0007bd80 0x7f8c00063430 secure :-1 s=THROTTLE_DONE pgs=716 cs=0 l=1 rx=0x7f8bf8001210 tx=0x7f8bf8010bd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:22:49.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.334+0000 7f8bfdffb700 1 -- 172.21.15.42:0/2714450405 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f8bf8012980 con 0x7f8c0007bd80 2019-12-05T05:22:49.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.334+0000 7f8c05fa3700 1 --2- 172.21.15.42:0/2714450405 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8c0007bd80 0x7f8c00063430 secure :-1 s=THROTTLE_DONE pgs=716 cs=0 l=1 rx=0x7f8bf8001210 tx=0x7f8bf8010bd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:22:49.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.338+0000 7f8bfdffb700 1 --2- 172.21.15.42:0/2714450405 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8be003add0 0x7f8be003d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:22:49.358 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.338+0000 7f8bfdffb700 1 -- 172.21.15.42:0/2714450405 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 7063+0+0 (secure 0 0 0) 0x7f8bf804b600 con 0x7f8c0007bd80 2019-12-05T05:22:49.358 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.338+0000 7f8c057a2700 1 --2- 172.21.15.42:0/2714450405 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8be003add0 0x7f8be003d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:22:49.358 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.338+0000 7f8c057a2700 1 --2- 172.21.15.42:0/2714450405 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8be003add0 0x7f8be003d2e0 crc :-1 s=READY pgs=321 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:22:49.358 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.342+0000 7f8c07205700 1 -- 172.21.15.42:0/2714450405 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c0005d110 con 0x7f8c0007bd80 2019-12-05T05:22:49.359 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.346+0000 7f8c05fa3700 1 --2- 172.21.15.42:0/2714450405 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8c0007bd80 0x7f8c00063430 secure :-1 s=THROTTLE_DONE pgs=716 cs=0 l=1 rx=0x7f8bf8001210 tx=0x7f8bf8010bd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:22:49.359 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.346+0000 7f8bfdffb700 1 -- 172.21.15.42:0/2714450405 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f8bf8032680 con 0x7f8c0007bd80 2019-12-05T05:22:49.519 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.510+0000 7f8c07205700 1 -- 172.21.15.42:0/2714450405 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8c0008fa10 con 0x7f8c0007bd80 2019-12-05T05:22:49.520 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.514+0000 7f8c05fa3700 1 --2- 172.21.15.42:0/2714450405 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8c0007bd80 0x7f8c00063430 secure :-1 s=THROTTLE_DONE pgs=716 cs=0 l=1 rx=0x7f8bf8001210 tx=0x7f8bf8010bd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:22:49.520 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.514+0000 7f8bfdffb700 1 -- 172.21.15.42:0/2714450405 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 93 v93) v1 ==== 94+0+2108 (secure 0 0 0) 0x7f8bf80328e0 con 0x7f8c0007bd80 2019-12-05T05:22:49.520 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 93 2019-12-05T05:22:49.521 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:22:49.521 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":93,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5123,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/1214659805","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1214659805},{"type":"v1","addr":"172.21.15.42:6835","nonce":1214659805}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":93}],"filesystems":[{"mdsmap":{"epoch":92,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:20:47.959813+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5066},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5066":{"gid":5066,"name":"b","rank":0,"incarnation":89,"state":"up:active","state_seq":30,"addr":"172.21.15.40:6833/3061061478","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3061061478},{"type":"v1","addr":"172.21.15.40:6833","nonce":3061061478}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:22:49.522 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.514+0000 7f8c07205700 1 -- 172.21.15.42:0/2714450405 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8be003add0 msgr2=0x7f8be003d2e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:22:49.522 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.514+0000 7f8c07205700 1 --2- 172.21.15.42:0/2714450405 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8be003add0 0x7f8be003d2e0 crc :-1 s=READY pgs=321 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:22:49.522 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.514+0000 7f8c07205700 1 -- 172.21.15.42:0/2714450405 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8c0007bd80 msgr2=0x7f8c00063430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:22:49.523 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.514+0000 7f8c07205700 1 --2- 172.21.15.42:0/2714450405 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8c0007bd80 0x7f8c00063430 secure :-1 s=READY pgs=716 cs=0 l=1 rx=0x7f8bf8001210 tx=0x7f8bf8010bd0).stop 2019-12-05T05:22:49.523 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.514+0000 7f8c07205700 1 -- 172.21.15.42:0/2714450405 shutdown_connections 2019-12-05T05:22:49.524 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.514+0000 7f8c07205700 1 --2- 172.21.15.42:0/2714450405 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8be003add0 0x7f8be003d2e0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:22:49.524 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.514+0000 7f8c07205700 1 --2- 172.21.15.42:0/2714450405 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8c0007bd80 0x7f8c00063430 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:22:49.524 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.514+0000 7f8c07205700 1 --2- 172.21.15.42:0/2714450405 >> v2:172.21.15.40:3300/0 conn(0x7f8c00142f70 0x7f8c00064740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:22:49.524 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.514+0000 7f8c07205700 1 --2- 172.21.15.42:0/2714450405 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8c00059f50 0x7f8c0007b830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:22:49.524 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.514+0000 7f8c07205700 1 -- 172.21.15.42:0/2714450405 shutdown_connections 2019-12-05T05:22:49.524 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.514+0000 7f8c07205700 1 -- 172.21.15.42:0/2714450405 wait complete. 2019-12-05T05:22:49.525 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.514+0000 7f8c07205700 1 -- 172.21.15.42:0/2714450405 >> 172.21.15.42:0/2714450405 conn(0x7f8c0007e4f0 msgr2=0x7f8c000a9790 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:22:49.532 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.b (rank=0) 2019-12-05T05:22:49.533 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2019-12-05T05:22:49.533 INFO:teuthology.orchestra.run:waiting for 300 2019-12-05T05:22:49.550 INFO:tasks.ceph.mds.b:Stopped 2019-12-05T05:22:49.551 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:22:49.650 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.638+0000 7f8494ea3700 1 Processor -- start 2019-12-05T05:22:49.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.638+0000 7f8494ea3700 1 -- start start 2019-12-05T05:22:49.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.638+0000 7f8494ea3700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f849008df80 0x7f849009f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:22:49.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.638+0000 7f8494ea3700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f849009f830 0x7f849014efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:22:49.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.638+0000 7f8494ea3700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f849009fc60 con 0x7f84900a7740 2019-12-05T05:22:49.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.638+0000 7f8494ea3700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84900a0b00 con 0x7f849009f830 2019-12-05T05:22:49.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.638+0000 7f8494ea3700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f849008e5b0 con 0x7f849008df80 2019-12-05T05:22:49.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.638+0000 7f848e59c700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f849008df80 0x7f849009f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:22:49.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.638+0000 7f848f59e700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f849009f830 0x7f849014efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:22:49.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.638+0000 7f848e59c700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f849008df80 0x7f849009f2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:60996/0 (socket says 172.21.15.42:60996) 2019-12-05T05:22:49.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.638+0000 7f848e59c700 1 -- 172.21.15.42:0/1342795566 learned_addr learned my addr 172.21.15.42:0/1342795566 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:22:49.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.638+0000 7f848ed9d700 1 --1- 172.21.15.42:0/1342795566 >> v1:172.21.15.40:6789/0 conn(0x7f84900a7740 0x7f849007e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:41774/0 (socket says 172.21.15.42:41774) 2019-12-05T05:22:49.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.638+0000 7f848f59e700 1 -- 172.21.15.42:0/1342795566 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f849008df80 msgr2=0x7f849009f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:22:49.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.638+0000 7f848f59e700 1 --2- 172.21.15.42:0/1342795566 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f849008df80 0x7f849009f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:22:49.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.642+0000 7f848f59e700 1 -- 172.21.15.42:0/1342795566 >> v1:172.21.15.40:6789/0 conn(0x7f84900a7740 legacy=0x7f849007e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:22:49.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.642+0000 7f848f59e700 1 -- 172.21.15.42:0/1342795566 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84900a0710 con 0x7f849009f830 2019-12-05T05:22:49.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.642+0000 7f848f59e700 1 --2- 172.21.15.42:0/1342795566 >> v2:172.21.15.40:3300/0 conn(0x7f849009f830 0x7f849014efe0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f8488008b10 tx=0x7f848800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:22:49.654 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.642+0000 7f848f59e700 1 --2- 172.21.15.42:0/1342795566 >> v2:172.21.15.40:3300/0 conn(0x7f849009f830 0x7f849014efe0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f8488008b10 tx=0x7f848800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:22:49.654 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.642+0000 7f848f59e700 1 --2- 172.21.15.42:0/1342795566 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f849009f830 0x7f849014efe0 secure :-1 s=READY pgs=577 cs=0 l=1 rx=0x7f8488008b10 tx=0x7f848800fff0).ready entity=mon.0 client_cookie=ab1eb70aae8e0540 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:22:49.654 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.642+0000 7f848f59e700 1 --2- 172.21.15.42:0/1342795566 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f849009f830 0x7f849014efe0 secure :-1 s=THROTTLE_DONE pgs=577 cs=0 l=1 rx=0x7f8488008b10 tx=0x7f848800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:22:49.654 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.642+0000 7f848f59e700 1 --2- 172.21.15.42:0/1342795566 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f849009f830 0x7f849014efe0 secure :-1 s=THROTTLE_DONE pgs=577 cs=0 l=1 rx=0x7f8488008b10 tx=0x7f848800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:22:49.654 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.642+0000 7f848dd9b700 1 -- 172.21.15.42:0/1342795566 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8488010900 con 0x7f849009f830 2019-12-05T05:22:49.655 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.642+0000 7f848f59e700 1 --2- 172.21.15.42:0/1342795566 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f849009f830 0x7f849014efe0 secure :-1 s=THROTTLE_DONE pgs=577 cs=0 l=1 rx=0x7f8488008b10 tx=0x7f848800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:22:49.655 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.642+0000 7f848dd9b700 1 -- 172.21.15.42:0/1342795566 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8488008b50 con 0x7f849009f830 2019-12-05T05:22:49.655 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.642+0000 7f848dd9b700 1 -- 172.21.15.42:0/1342795566 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f84880110c0 con 0x7f849009f830 2019-12-05T05:22:49.655 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.642+0000 7f8494ea3700 1 -- 172.21.15.42:0/1342795566 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f849009f830 msgr2=0x7f849014efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:22:49.655 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.642+0000 7f8494ea3700 1 --2- 172.21.15.42:0/1342795566 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f849009f830 0x7f849014efe0 secure :-1 s=READY pgs=577 cs=0 l=1 rx=0x7f8488008b10 tx=0x7f848800fff0).stop 2019-12-05T05:22:49.655 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.642+0000 7f8494ea3700 1 -- 172.21.15.42:0/1342795566 shutdown_connections 2019-12-05T05:22:49.656 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.642+0000 7f8494ea3700 1 --2- 172.21.15.42:0/1342795566 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f849009f830 0x7f849014efe0 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:22:49.656 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.642+0000 7f8494ea3700 1 --2- 172.21.15.42:0/1342795566 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f849008df80 0x7f849009f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:22:49.656 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.642+0000 7f8494ea3700 1 -- 172.21.15.42:0/1342795566 shutdown_connections 2019-12-05T05:22:49.656 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.642+0000 7f8494ea3700 1 -- 172.21.15.42:0/1342795566 wait complete. 2019-12-05T05:22:49.656 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.642+0000 7f8494ea3700 1 -- 172.21.15.42:0/1342795566 >> 172.21.15.42:0/1342795566 conn(0x7f84900aabe0 msgr2=0x7f84900a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:22:49.660 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.646+0000 7f8494ea3700 1 Processor -- start 2019-12-05T05:22:49.660 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.646+0000 7f8494ea3700 1 -- start start 2019-12-05T05:22:49.660 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.646+0000 7f8494ea3700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8490065050 0x7f8490142d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:22:49.661 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.646+0000 7f8494ea3700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f849008eb70 0x7f8490063950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:22:49.661 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.646+0000 7f8494ea3700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84900642d0 con 0x7f8490063ea0 2019-12-05T05:22:49.661 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.646+0000 7f8494ea3700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8490065b40 con 0x7f8490065050 2019-12-05T05:22:49.661 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.646+0000 7f8494ea3700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84901433a0 con 0x7f849008eb70 2019-12-05T05:22:49.661 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.646+0000 7f848e59c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f849008eb70 0x7f8490063950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:22:49.661 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.646+0000 7f848e59c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f849008eb70 0x7f8490063950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:60536/0 (socket says 172.21.15.42:60536) 2019-12-05T05:22:49.662 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.646+0000 7f848ed9d700 1 --2- 172.21.15.42:0/248295459 >> v2:172.21.15.40:3300/0 conn(0x7f8490065050 0x7f8490142d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:22:49.662 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.646+0000 7f848e59c700 1 -- 172.21.15.42:0/248295459 learned_addr learned my addr 172.21.15.42:0/248295459 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:22:49.662 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.646+0000 7f848e59c700 1 -- 172.21.15.42:0/248295459 >> v2:172.21.15.40:3300/0 conn(0x7f8490065050 msgr2=0x7f8490142d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:22:49.662 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.646+0000 7f848e59c700 1 --2- 172.21.15.42:0/248295459 >> v2:172.21.15.40:3300/0 conn(0x7f8490065050 0x7f8490142d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:22:49.662 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.646+0000 7f848e59c700 1 -- 172.21.15.42:0/248295459 >> v1:172.21.15.42:6789/0 conn(0x7f8490063ea0 legacy=0x7f84901172a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:22:49.662 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.646+0000 7f848e59c700 1 -- 172.21.15.42:0/248295459 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f849008efa0 con 0x7f849008eb70 2019-12-05T05:22:49.663 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.646+0000 7f848e59c700 1 --2- 172.21.15.42:0/248295459 >> v2:172.21.15.42:3300/0 conn(0x7f849008eb70 0x7f8490063950 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f8484002f30 tx=0x7f8484010cb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:22:49.663 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.646+0000 7f848e59c700 1 --2- 172.21.15.42:0/248295459 >> v2:172.21.15.42:3300/0 conn(0x7f849008eb70 0x7f8490063950 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f8484002f30 tx=0x7f8484010cb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:22:49.663 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.646+0000 7f848e59c700 1 --2- 172.21.15.42:0/248295459 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f849008eb70 0x7f8490063950 secure :-1 s=READY pgs=718 cs=0 l=1 rx=0x7f8484002f30 tx=0x7f8484010cb0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:22:49.663 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.646+0000 7f848e59c700 1 --2- 172.21.15.42:0/248295459 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f849008eb70 0x7f8490063950 secure :-1 s=THROTTLE_DONE pgs=718 cs=0 l=1 rx=0x7f8484002f30 tx=0x7f8484010cb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:22:49.663 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.646+0000 7f847f7fe700 1 -- 172.21.15.42:0/248295459 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f84840088b0 con 0x7f849008eb70 2019-12-05T05:22:49.663 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.646+0000 7f848e59c700 1 --2- 172.21.15.42:0/248295459 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f849008eb70 0x7f8490063950 secure :-1 s=THROTTLE_DONE pgs=718 cs=0 l=1 rx=0x7f8484002f30 tx=0x7f8484010cb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:22:49.664 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.646+0000 7f847f7fe700 1 -- 172.21.15.42:0/248295459 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8484007160 con 0x7f849008eb70 2019-12-05T05:22:49.664 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.646+0000 7f8494ea3700 1 -- 172.21.15.42:0/248295459 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f849012a810 con 0x7f849008eb70 2019-12-05T05:22:49.664 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.646+0000 7f848e59c700 1 --2- 172.21.15.42:0/248295459 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f849008eb70 0x7f8490063950 secure :-1 s=THROTTLE_DONE pgs=718 cs=0 l=1 rx=0x7f8484002f30 tx=0x7f8484010cb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:22:49.664 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.646+0000 7f847f7fe700 1 -- 172.21.15.42:0/248295459 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8484005840 con 0x7f849008eb70 2019-12-05T05:22:49.664 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.646+0000 7f8494ea3700 1 -- 172.21.15.42:0/248295459 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8490117aa0 con 0x7f849008eb70 2019-12-05T05:22:49.664 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.650+0000 7f848e59c700 1 --2- 172.21.15.42:0/248295459 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f849008eb70 0x7f8490063950 secure :-1 s=THROTTLE_DONE pgs=718 cs=0 l=1 rx=0x7f8484002f30 tx=0x7f8484010cb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:22:49.665 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.650+0000 7f847f7fe700 1 -- 172.21.15.42:0/248295459 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f8484005ae0 con 0x7f849008eb70 2019-12-05T05:22:49.665 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.650+0000 7f848e59c700 1 --2- 172.21.15.42:0/248295459 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f849008eb70 0x7f8490063950 secure :-1 s=THROTTLE_DONE pgs=718 cs=0 l=1 rx=0x7f8484002f30 tx=0x7f8484010cb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:22:49.665 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.650+0000 7f847f7fe700 1 --2- 172.21.15.42:0/248295459 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f847403adc0 0x7f847403d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:22:49.665 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.650+0000 7f847f7fe700 1 -- 172.21.15.42:0/248295459 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 7063+0+0 (secure 0 0 0) 0x7f848404b430 con 0x7f849008eb70 2019-12-05T05:22:49.665 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.650+0000 7f848ed9d700 1 --2- 172.21.15.42:0/248295459 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f847403adc0 0x7f847403d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:22:49.665 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.650+0000 7f848ed9d700 1 --2- 172.21.15.42:0/248295459 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f847403adc0 0x7f847403d2d0 crc :-1 s=READY pgs=322 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:22:49.666 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.654+0000 7f8494ea3700 1 -- 172.21.15.42:0/248295459 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f849005d110 con 0x7f849008eb70 2019-12-05T05:22:49.671 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.658+0000 7f848e59c700 1 --2- 172.21.15.42:0/248295459 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f849008eb70 0x7f8490063950 secure :-1 s=THROTTLE_DONE pgs=718 cs=0 l=1 rx=0x7f8484002f30 tx=0x7f8484010cb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:22:49.672 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.658+0000 7f847f7fe700 1 -- 172.21.15.42:0/248295459 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f8484032490 con 0x7f849008eb70 2019-12-05T05:22:49.846 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.830+0000 7f8494ea3700 1 -- 172.21.15.42:0/248295459 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f849005b8c0 con 0x7f849008eb70 2019-12-05T05:22:49.846 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.830+0000 7f848e59c700 1 --2- 172.21.15.42:0/248295459 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f849008eb70 0x7f8490063950 secure :-1 s=THROTTLE_DONE pgs=718 cs=0 l=1 rx=0x7f8484002f30 tx=0x7f8484010cb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:22:49.846 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.830+0000 7f847f7fe700 1 -- 172.21.15.42:0/248295459 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 93 v93) v1 ==== 94+0+2108 (secure 0 0 0) 0x7f84840326f0 con 0x7f849008eb70 2019-12-05T05:22:49.847 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 93 2019-12-05T05:22:49.847 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.838+0000 7f8494ea3700 1 -- 172.21.15.42:0/248295459 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f847403adc0 msgr2=0x7f847403d2d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:22:49.847 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.838+0000 7f8494ea3700 1 --2- 172.21.15.42:0/248295459 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f847403adc0 0x7f847403d2d0 crc :-1 s=READY pgs=322 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:22:49.847 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.838+0000 7f8494ea3700 1 -- 172.21.15.42:0/248295459 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f849008eb70 msgr2=0x7f8490063950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:22:49.847 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.838+0000 7f8494ea3700 1 --2- 172.21.15.42:0/248295459 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f849008eb70 0x7f8490063950 secure :-1 s=READY pgs=718 cs=0 l=1 rx=0x7f8484002f30 tx=0x7f8484010cb0).stop 2019-12-05T05:22:49.847 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.838+0000 7f8494ea3700 1 -- 172.21.15.42:0/248295459 shutdown_connections 2019-12-05T05:22:49.848 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.838+0000 7f8494ea3700 1 --2- 172.21.15.42:0/248295459 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f847403adc0 0x7f847403d2d0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:22:49.848 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.838+0000 7f8494ea3700 1 --2- 172.21.15.42:0/248295459 >> v2:172.21.15.40:3300/0 conn(0x7f8490065050 0x7f8490142d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:22:49.848 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.838+0000 7f8494ea3700 1 --2- 172.21.15.42:0/248295459 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f849008eb70 0x7f8490063950 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:22:49.848 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.838+0000 7f8494ea3700 1 -- 172.21.15.42:0/248295459 shutdown_connections 2019-12-05T05:22:49.849 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.838+0000 7f8494ea3700 1 -- 172.21.15.42:0/248295459 wait complete. 2019-12-05T05:22:49.849 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:49.838+0000 7f8494ea3700 1 -- 172.21.15.42:0/248295459 >> 172.21.15.42:0/248295459 conn(0x7f84900a82a0 msgr2=0x7f8490115e80 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:22:49.849 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:22:49.849 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":93,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5123,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/1214659805","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1214659805},{"type":"v1","addr":"172.21.15.42:6835","nonce":1214659805}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":93}],"filesystems":[{"mdsmap":{"epoch":92,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:20:47.959813+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5066},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5066":{"gid":5066,"name":"b","rank":0,"incarnation":89,"state":"up:active","state_seq":30,"addr":"172.21.15.40:6833/3061061478","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3061061478},{"type":"v1","addr":"172.21.15.40:6833","nonce":3061061478}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:22:49.857 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T05:22:51.859 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:22:51.962 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:51.950+0000 7f5503cd6700 1 Processor -- start 2019-12-05T05:22:51.963 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:51.950+0000 7f5503cd6700 1 -- start start 2019-12-05T05:22:51.963 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:51.950+0000 7f5503cd6700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f54fc0a7740 0x7f54fc08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:22:51.963 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:51.950+0000 7f5503cd6700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f54fc07e720 0x7f54fc07eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:22:51.963 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:51.950+0000 7f5503cd6700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f54fc09f2e0 0x7f54fc09f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:22:51.963 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:51.950+0000 7f5503cd6700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54fc0a0b00 con 0x7f54fc0a7740 2019-12-05T05:22:51.963 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:51.950+0000 7f5503cd6700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54fc08e5b0 con 0x7f54fc09f2e0 2019-12-05T05:22:51.964 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:51.950+0000 7f5503cd6700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54fc08e850 con 0x7f54fc07e720 2019-12-05T05:22:51.964 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:51.950+0000 7f5502a74700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f54fc09f2e0 0x7f54fc09f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:22:51.964 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:51.954+0000 7f5502a74700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f54fc09f2e0 0x7f54fc09f710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:60552/0 (socket says 172.21.15.42:60552) 2019-12-05T05:22:51.964 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:51.954+0000 7f5502a74700 1 -- 172.21.15.42:0/752695861 learned_addr learned my addr 172.21.15.42:0/752695861 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:22:51.964 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:51.954+0000 7f5502273700 1 --2- 172.21.15.42:0/752695861 >> v2:172.21.15.40:3300/0 conn(0x7f54fc0a7740 0x7f54fc08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:22:51.964 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:51.954+0000 7f5501a72700 1 --2- 172.21.15.42:0/752695861 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f54fc07e720 0x7f54fc07eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:22:51.964 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:51.954+0000 7f5502a74700 1 -- 172.21.15.42:0/752695861 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f54fc07e720 msgr2=0x7f54fc07eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:22:51.965 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:51.954+0000 7f5502a74700 1 --2- 172.21.15.42:0/752695861 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f54fc07e720 0x7f54fc07eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:22:51.965 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:51.954+0000 7f5502a74700 1 -- 172.21.15.42:0/752695861 >> v2:172.21.15.40:3300/0 conn(0x7f54fc0a7740 msgr2=0x7f54fc08df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:22:51.965 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:51.954+0000 7f5502a74700 1 --2- 172.21.15.42:0/752695861 >> v2:172.21.15.40:3300/0 conn(0x7f54fc0a7740 0x7f54fc08df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:22:51.965 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:51.954+0000 7f5502a74700 1 -- 172.21.15.42:0/752695861 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54fc07f0a0 con 0x7f54fc09f2e0 2019-12-05T05:22:51.965 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:51.954+0000 7f5502a74700 1 --2- 172.21.15.42:0/752695861 >> v2:172.21.15.42:3300/0 conn(0x7f54fc09f2e0 0x7f54fc09f710 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f54fc08e770 tx=0x7f54f4010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:22:51.965 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:51.954+0000 7f5502a74700 1 --2- 172.21.15.42:0/752695861 >> v2:172.21.15.42:3300/0 conn(0x7f54fc09f2e0 0x7f54fc09f710 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f54fc08e770 tx=0x7f54f4010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:22:51.965 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:51.954+0000 7f5502a74700 1 --2- 172.21.15.42:0/752695861 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f54fc09f2e0 0x7f54fc09f710 secure :-1 s=READY pgs=719 cs=0 l=1 rx=0x7f54fc08e770 tx=0x7f54f4010130).ready entity=mon.1 client_cookie=8c79a799dbf09c27 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:22:51.966 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:51.954+0000 7f5502a74700 1 --2- 172.21.15.42:0/752695861 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f54fc09f2e0 0x7f54fc09f710 secure :-1 s=THROTTLE_DONE pgs=719 cs=0 l=1 rx=0x7f54fc08e770 tx=0x7f54f4010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:22:51.966 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:51.954+0000 7f5502a74700 1 --2- 172.21.15.42:0/752695861 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f54fc09f2e0 0x7f54fc09f710 secure :-1 s=THROTTLE_DONE pgs=719 cs=0 l=1 rx=0x7f54fc08e770 tx=0x7f54f4010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:22:51.966 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:51.954+0000 7f5501271700 1 -- 172.21.15.42:0/752695861 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f54f4010de0 con 0x7f54fc09f2e0 2019-12-05T05:22:51.966 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:51.954+0000 7f5502a74700 1 --2- 172.21.15.42:0/752695861 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f54fc09f2e0 0x7f54fc09f710 secure :-1 s=THROTTLE_DONE pgs=719 cs=0 l=1 rx=0x7f54fc08e770 tx=0x7f54f4010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:22:51.966 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:51.954+0000 7f5501271700 1 -- 172.21.15.42:0/752695861 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54f4010a00 con 0x7f54fc09f2e0 2019-12-05T05:22:51.966 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:51.954+0000 7f5501271700 1 -- 172.21.15.42:0/752695861 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f54f40114a0 con 0x7f54fc09f2e0 2019-12-05T05:22:51.966 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:51.954+0000 7f5503cd6700 1 -- 172.21.15.42:0/752695861 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f54fc09f2e0 msgr2=0x7f54fc09f710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:22:51.967 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:51.954+0000 7f5503cd6700 1 --2- 172.21.15.42:0/752695861 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f54fc09f2e0 0x7f54fc09f710 secure :-1 s=READY pgs=719 cs=0 l=1 rx=0x7f54fc08e770 tx=0x7f54f4010130).stop 2019-12-05T05:22:51.967 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:51.954+0000 7f5503cd6700 1 -- 172.21.15.42:0/752695861 shutdown_connections 2019-12-05T05:22:51.967 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:51.954+0000 7f5503cd6700 1 --2- 172.21.15.42:0/752695861 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f54fc09f2e0 0x7f54fc09f710 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:22:51.967 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:51.954+0000 7f5503cd6700 1 --2- 172.21.15.42:0/752695861 >> v2:172.21.15.40:3300/0 conn(0x7f54fc0a7740 0x7f54fc08df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:22:51.967 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:51.954+0000 7f5503cd6700 1 --2- 172.21.15.42:0/752695861 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f54fc07e720 0x7f54fc07eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:22:51.967 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:51.954+0000 7f5503cd6700 1 -- 172.21.15.42:0/752695861 shutdown_connections 2019-12-05T05:22:51.967 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:51.954+0000 7f5503cd6700 1 -- 172.21.15.42:0/752695861 wait complete. 2019-12-05T05:22:51.968 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:51.954+0000 7f5503cd6700 1 -- 172.21.15.42:0/752695861 >> 172.21.15.42:0/752695861 conn(0x7f54fc0aabe0 msgr2=0x7f54fc0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:22:51.970 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:51.958+0000 7f5503cd6700 1 Processor -- start 2019-12-05T05:22:51.970 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:51.958+0000 7f5503cd6700 1 -- start start 2019-12-05T05:22:51.970 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:51.958+0000 7f5503cd6700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f54fc117cc0 0x7f54fc0a9640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:22:51.971 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:51.958+0000 7f5503cd6700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f54fc12df20 0x7f54fc12e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:22:51.971 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:51.958+0000 7f5503cd6700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54fc11b470 con 0x7f54fc0a6a50 2019-12-05T05:22:51.971 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:51.958+0000 7f5503cd6700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54fc0a7080 con 0x7f54fc12df20 2019-12-05T05:22:51.971 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:51.958+0000 7f5503cd6700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54fc11a9d0 con 0x7f54fc117cc0 2019-12-05T05:22:51.971 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:51.958+0000 7f5501a72700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f54fc117cc0 0x7f54fc0a9640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:22:51.971 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:51.958+0000 7f5501a72700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f54fc117cc0 0x7f54fc0a9640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:60554/0 (socket says 172.21.15.42:60554) 2019-12-05T05:22:51.971 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:51.958+0000 7f5501a72700 1 -- 172.21.15.42:0/20872955 learned_addr learned my addr 172.21.15.42:0/20872955 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:22:51.972 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:51.958+0000 7f5502a74700 1 --2- 172.21.15.42:0/20872955 >> v2:172.21.15.40:3300/0 conn(0x7f54fc12df20 0x7f54fc12e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:22:51.972 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:51.958+0000 7f5501a72700 1 -- 172.21.15.42:0/20872955 >> v2:172.21.15.40:3300/0 conn(0x7f54fc12df20 msgr2=0x7f54fc12e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:22:51.972 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:51.958+0000 7f5501a72700 1 --2- 172.21.15.42:0/20872955 >> v2:172.21.15.40:3300/0 conn(0x7f54fc12df20 0x7f54fc12e350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:22:51.972 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:51.958+0000 7f5501a72700 1 -- 172.21.15.42:0/20872955 >> v1:172.21.15.42:6789/0 conn(0x7f54fc0a6a50 legacy=0x7f54fc11ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:22:51.972 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:51.958+0000 7f5501a72700 1 -- 172.21.15.42:0/20872955 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54fc0a9b90 con 0x7f54fc117cc0 2019-12-05T05:22:51.972 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:51.958+0000 7f5501a72700 1 --2- 172.21.15.42:0/20872955 >> v2:172.21.15.42:3300/0 conn(0x7f54fc117cc0 0x7f54fc0a9640 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f54ec003070 tx=0x7f54ec010bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:22:51.972 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:51.958+0000 7f5501a72700 1 --2- 172.21.15.42:0/20872955 >> v2:172.21.15.42:3300/0 conn(0x7f54fc117cc0 0x7f54fc0a9640 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f54ec003070 tx=0x7f54ec010bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:22:51.973 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:51.958+0000 7f5501a72700 1 --2- 172.21.15.42:0/20872955 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f54fc117cc0 0x7f54fc0a9640 secure :-1 s=READY pgs=721 cs=0 l=1 rx=0x7f54ec003070 tx=0x7f54ec010bc0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:22:51.973 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:51.958+0000 7f5501a72700 1 --2- 172.21.15.42:0/20872955 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f54fc117cc0 0x7f54fc0a9640 secure :-1 s=THROTTLE_DONE pgs=721 cs=0 l=1 rx=0x7f54ec003070 tx=0x7f54ec010bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:22:51.973 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:51.958+0000 7f54f2ffd700 1 -- 172.21.15.42:0/20872955 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f54ec005890 con 0x7f54fc117cc0 2019-12-05T05:22:51.973 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:51.958+0000 7f5501a72700 1 --2- 172.21.15.42:0/20872955 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f54fc117cc0 0x7f54fc0a9640 secure :-1 s=THROTTLE_DONE pgs=721 cs=0 l=1 rx=0x7f54ec003070 tx=0x7f54ec010bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:22:51.973 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:51.958+0000 7f54f2ffd700 1 -- 172.21.15.42:0/20872955 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54ec007160 con 0x7f54fc117cc0 2019-12-05T05:22:51.973 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:51.958+0000 7f5501a72700 1 --2- 172.21.15.42:0/20872955 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f54fc117cc0 0x7f54fc0a9640 secure :-1 s=THROTTLE_DONE pgs=721 cs=0 l=1 rx=0x7f54ec003070 tx=0x7f54ec010bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:22:51.973 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:51.958+0000 7f5503cd6700 1 -- 172.21.15.42:0/20872955 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54fc12db40 con 0x7f54fc117cc0 2019-12-05T05:22:51.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:51.958+0000 7f54f2ffd700 1 -- 172.21.15.42:0/20872955 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f54ec012240 con 0x7f54fc117cc0 2019-12-05T05:22:51.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:51.958+0000 7f5503cd6700 1 -- 172.21.15.42:0/20872955 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54fc12ea40 con 0x7f54fc117cc0 2019-12-05T05:22:51.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:51.962+0000 7f5501a72700 1 --2- 172.21.15.42:0/20872955 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f54fc117cc0 0x7f54fc0a9640 secure :-1 s=THROTTLE_DONE pgs=721 cs=0 l=1 rx=0x7f54ec003070 tx=0x7f54ec010bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:22:51.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:51.962+0000 7f54f2ffd700 1 -- 172.21.15.42:0/20872955 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f54ec012560 con 0x7f54fc117cc0 2019-12-05T05:22:51.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:51.962+0000 7f5501a72700 1 --2- 172.21.15.42:0/20872955 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f54fc117cc0 0x7f54fc0a9640 secure :-1 s=THROTTLE_DONE pgs=721 cs=0 l=1 rx=0x7f54ec003070 tx=0x7f54ec010bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:22:51.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:51.962+0000 7f54f2ffd700 1 --2- 172.21.15.42:0/20872955 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f54e403ad60 0x7f54e403d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:22:51.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:51.962+0000 7f54f2ffd700 1 -- 172.21.15.42:0/20872955 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 7063+0+0 (secure 0 0 0) 0x7f54ec04b240 con 0x7f54fc117cc0 2019-12-05T05:22:51.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:51.962+0000 7f5502273700 1 --2- 172.21.15.42:0/20872955 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f54e403ad60 0x7f54e403d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:22:51.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:51.962+0000 7f5502273700 1 --2- 172.21.15.42:0/20872955 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f54e403ad60 0x7f54e403d270 crc :-1 s=READY pgs=323 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:22:51.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:51.966+0000 7f5503cd6700 1 -- 172.21.15.42:0/20872955 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54fc0a6e80 con 0x7f54fc117cc0 2019-12-05T05:22:51.980 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:51.974+0000 7f5501a72700 1 --2- 172.21.15.42:0/20872955 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f54fc117cc0 0x7f54fc0a9640 secure :-1 s=THROTTLE_DONE pgs=721 cs=0 l=1 rx=0x7f54ec003070 tx=0x7f54ec010bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:22:51.980 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:51.974+0000 7f54f2ffd700 1 -- 172.21.15.42:0/20872955 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f54ec0322a0 con 0x7f54fc117cc0 2019-12-05T05:22:52.145 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:52.138+0000 7f5503cd6700 1 -- 172.21.15.42:0/20872955 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f54fc05d110 con 0x7f54fc117cc0 2019-12-05T05:22:52.146 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:52.138+0000 7f5501a72700 1 --2- 172.21.15.42:0/20872955 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f54fc117cc0 0x7f54fc0a9640 secure :-1 s=THROTTLE_DONE pgs=721 cs=0 l=1 rx=0x7f54ec003070 tx=0x7f54ec010bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:22:52.146 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:52.138+0000 7f54f2ffd700 1 -- 172.21.15.42:0/20872955 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 93 v93) v1 ==== 94+0+2108 (secure 0 0 0) 0x7f54ec032500 con 0x7f54fc117cc0 2019-12-05T05:22:52.148 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 93 2019-12-05T05:22:52.148 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:22:52.148 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":93,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5123,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/1214659805","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1214659805},{"type":"v1","addr":"172.21.15.42:6835","nonce":1214659805}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":93}],"filesystems":[{"mdsmap":{"epoch":92,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:20:47.959813+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5066},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5066":{"gid":5066,"name":"b","rank":0,"incarnation":89,"state":"up:active","state_seq":30,"addr":"172.21.15.40:6833/3061061478","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3061061478},{"type":"v1","addr":"172.21.15.40:6833","nonce":3061061478}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:22:52.151 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:52.142+0000 7f5503cd6700 1 -- 172.21.15.42:0/20872955 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f54e403ad60 msgr2=0x7f54e403d270 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:22:52.151 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:52.142+0000 7f5503cd6700 1 --2- 172.21.15.42:0/20872955 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f54e403ad60 0x7f54e403d270 crc :-1 s=READY pgs=323 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:22:52.151 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:52.142+0000 7f5503cd6700 1 -- 172.21.15.42:0/20872955 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f54fc117cc0 msgr2=0x7f54fc0a9640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:22:52.151 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:52.142+0000 7f5503cd6700 1 --2- 172.21.15.42:0/20872955 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f54fc117cc0 0x7f54fc0a9640 secure :-1 s=READY pgs=721 cs=0 l=1 rx=0x7f54ec003070 tx=0x7f54ec010bc0).stop 2019-12-05T05:22:52.151 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:52.142+0000 7f5503cd6700 1 -- 172.21.15.42:0/20872955 shutdown_connections 2019-12-05T05:22:52.151 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:52.142+0000 7f5503cd6700 1 --2- 172.21.15.42:0/20872955 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f54e403ad60 0x7f54e403d270 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:22:52.152 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:52.142+0000 7f5503cd6700 1 --2- 172.21.15.42:0/20872955 >> v2:172.21.15.40:3300/0 conn(0x7f54fc12df20 0x7f54fc12e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:22:52.152 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:52.142+0000 7f5503cd6700 1 --2- 172.21.15.42:0/20872955 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f54fc117cc0 0x7f54fc0a9640 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:22:52.152 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:52.142+0000 7f5503cd6700 1 -- 172.21.15.42:0/20872955 shutdown_connections 2019-12-05T05:22:52.152 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:52.142+0000 7f5503cd6700 1 -- 172.21.15.42:0/20872955 wait complete. 2019-12-05T05:22:52.152 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:52.142+0000 7f5503cd6700 1 -- 172.21.15.42:0/20872955 >> 172.21.15.42:0/20872955 conn(0x7f54fc0a82a0 msgr2=0x7f54fc07e4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:22:52.161 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T05:22:54.162 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:22:54.270 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:54.258+0000 7ff9f3444700 1 Processor -- start 2019-12-05T05:22:54.270 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:54.258+0000 7ff9f3444700 1 -- start start 2019-12-05T05:22:54.270 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:54.258+0000 7ff9f3444700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff9ec0a7740 0x7ff9ec08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:22:54.271 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:54.258+0000 7ff9f3444700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff9ec09fae0 0x7ff9ec07eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:22:54.271 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:54.258+0000 7ff9f3444700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9ec0a0710 con 0x7ff9ec07e720 2019-12-05T05:22:54.271 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:54.258+0000 7ff9f3444700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9ec0a0b00 con 0x7ff9ec0a7740 2019-12-05T05:22:54.271 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:54.258+0000 7ff9f3444700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9ec08e5b0 con 0x7ff9ec09fae0 2019-12-05T05:22:54.271 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:54.258+0000 7ff9f21e2700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff9ec09fae0 0x7ff9ec07eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:22:54.271 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:54.258+0000 7ff9f21e2700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff9ec09fae0 0x7ff9ec07eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:60566/0 (socket says 172.21.15.42:60566) 2019-12-05T05:22:54.271 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:54.258+0000 7ff9f21e2700 1 -- 172.21.15.42:0/1176226259 learned_addr learned my addr 172.21.15.42:0/1176226259 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:22:54.272 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:54.258+0000 7ff9f19e1700 1 --2- 172.21.15.42:0/1176226259 >> v2:172.21.15.40:3300/0 conn(0x7ff9ec0a7740 0x7ff9ec08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:22:54.272 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:54.262+0000 7ff9f21e2700 1 -- 172.21.15.42:0/1176226259 >> v2:172.21.15.40:3300/0 conn(0x7ff9ec0a7740 msgr2=0x7ff9ec08df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:22:54.272 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:54.262+0000 7ff9f21e2700 1 --2- 172.21.15.42:0/1176226259 >> v2:172.21.15.40:3300/0 conn(0x7ff9ec0a7740 0x7ff9ec08df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:22:54.272 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:54.262+0000 7ff9f21e2700 1 -- 172.21.15.42:0/1176226259 >> v1:172.21.15.40:6789/0 conn(0x7ff9ec07e720 legacy=0x7ff9ec09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:22:54.272 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:54.262+0000 7ff9f21e2700 1 -- 172.21.15.42:0/1176226259 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9ec07f0a0 con 0x7ff9ec09fae0 2019-12-05T05:22:54.272 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:54.262+0000 7ff9f21e2700 1 --2- 172.21.15.42:0/1176226259 >> v2:172.21.15.42:3300/0 conn(0x7ff9ec09fae0 0x7ff9ec07eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7ff9e40032b0 tx=0x7ff9e40100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:22:54.272 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:54.262+0000 7ff9f21e2700 1 --2- 172.21.15.42:0/1176226259 >> v2:172.21.15.42:3300/0 conn(0x7ff9ec09fae0 0x7ff9ec07eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7ff9e40032b0 tx=0x7ff9e40100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:22:54.273 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:54.262+0000 7ff9f21e2700 1 --2- 172.21.15.42:0/1176226259 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff9ec09fae0 0x7ff9ec07eb50 secure :-1 s=READY pgs=722 cs=0 l=1 rx=0x7ff9e40032b0 tx=0x7ff9e40100b0).ready entity=mon.1 client_cookie=b8e82bfcc1a1f8fe server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:22:54.273 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:54.262+0000 7ff9f21e2700 1 --2- 172.21.15.42:0/1176226259 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff9ec09fae0 0x7ff9ec07eb50 secure :-1 s=THROTTLE_DONE pgs=722 cs=0 l=1 rx=0x7ff9e40032b0 tx=0x7ff9e40100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:22:54.273 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:54.262+0000 7ff9f09df700 1 -- 172.21.15.42:0/1176226259 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff9e4010d60 con 0x7ff9ec09fae0 2019-12-05T05:22:54.273 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:54.262+0000 7ff9f21e2700 1 --2- 172.21.15.42:0/1176226259 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff9ec09fae0 0x7ff9ec07eb50 secure :-1 s=THROTTLE_DONE pgs=722 cs=0 l=1 rx=0x7ff9e40032b0 tx=0x7ff9e40100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:22:54.273 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:54.262+0000 7ff9f09df700 1 -- 172.21.15.42:0/1176226259 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9e4010980 con 0x7ff9ec09fae0 2019-12-05T05:22:54.273 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:54.262+0000 7ff9f21e2700 1 --2- 172.21.15.42:0/1176226259 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff9ec09fae0 0x7ff9ec07eb50 secure :-1 s=THROTTLE_DONE pgs=722 cs=0 l=1 rx=0x7ff9e40032b0 tx=0x7ff9e40100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:22:54.273 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:54.262+0000 7ff9f3444700 1 -- 172.21.15.42:0/1176226259 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff9ec09fae0 msgr2=0x7ff9ec07eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:22:54.274 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:54.262+0000 7ff9f09df700 1 -- 172.21.15.42:0/1176226259 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff9e40114a0 con 0x7ff9ec09fae0 2019-12-05T05:22:54.274 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:54.262+0000 7ff9f3444700 1 --2- 172.21.15.42:0/1176226259 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff9ec09fae0 0x7ff9ec07eb50 secure :-1 s=READY pgs=722 cs=0 l=1 rx=0x7ff9e40032b0 tx=0x7ff9e40100b0).stop 2019-12-05T05:22:54.274 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:54.262+0000 7ff9f3444700 1 -- 172.21.15.42:0/1176226259 shutdown_connections 2019-12-05T05:22:54.274 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:54.262+0000 7ff9f3444700 1 --2- 172.21.15.42:0/1176226259 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff9ec09fae0 0x7ff9ec07eb50 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:22:54.274 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:54.262+0000 7ff9f3444700 1 --2- 172.21.15.42:0/1176226259 >> v2:172.21.15.40:3300/0 conn(0x7ff9ec0a7740 0x7ff9ec08df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:22:54.274 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:54.262+0000 7ff9f3444700 1 -- 172.21.15.42:0/1176226259 shutdown_connections 2019-12-05T05:22:54.274 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:54.262+0000 7ff9f3444700 1 -- 172.21.15.42:0/1176226259 wait complete. 2019-12-05T05:22:54.275 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:54.262+0000 7ff9f3444700 1 -- 172.21.15.42:0/1176226259 >> 172.21.15.42:0/1176226259 conn(0x7ff9ec0aabe0 msgr2=0x7ff9ec0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:22:54.278 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:54.262+0000 7ff9f3444700 1 Processor -- start 2019-12-05T05:22:54.278 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:54.262+0000 7ff9f3444700 1 -- start start 2019-12-05T05:22:54.278 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:54.262+0000 7ff9f3444700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff9ec119560 0x7ff9ec1222c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:22:54.278 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:54.262+0000 7ff9f3444700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff9ec1237a0 0x7ff9ec123bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:22:54.278 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:54.262+0000 7ff9f3444700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9ec0a4e00 con 0x7ff9ec08eb50 2019-12-05T05:22:54.278 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:54.262+0000 7ff9f3444700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9ec124120 con 0x7ff9ec119560 2019-12-05T05:22:54.278 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:54.262+0000 7ff9f3444700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9ec0a51d0 con 0x7ff9ec1237a0 2019-12-05T05:22:54.279 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:54.266+0000 7ff9f19e1700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff9ec119560 0x7ff9ec1222c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:22:54.279 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:54.266+0000 7ff9f19e1700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff9ec119560 0x7ff9ec1222c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:60572/0 (socket says 172.21.15.42:60572) 2019-12-05T05:22:54.279 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:54.266+0000 7ff9f19e1700 1 -- 172.21.15.42:0/3544095952 learned_addr learned my addr 172.21.15.42:0/3544095952 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:22:54.279 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:54.266+0000 7ff9f11e0700 1 --2- 172.21.15.42:0/3544095952 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff9ec1237a0 0x7ff9ec123bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:22:54.279 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:54.266+0000 7ff9de7fc700 1 -- 172.21.15.42:0/3544095952 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 97153130 0 0) 0x7ff9ec0a4e00 con 0x7ff9ec08eb50 2019-12-05T05:22:54.279 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:54.266+0000 7ff9f19e1700 1 -- 172.21.15.42:0/3544095952 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff9ec1237a0 msgr2=0x7ff9ec123bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:22:54.279 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:54.266+0000 7ff9f19e1700 1 --2- 172.21.15.42:0/3544095952 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff9ec1237a0 0x7ff9ec123bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:22:54.280 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:54.266+0000 7ff9f19e1700 1 -- 172.21.15.42:0/3544095952 >> v1:172.21.15.40:6789/0 conn(0x7ff9ec08eb50 legacy=0x7ff9ec08ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:22:54.280 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:54.266+0000 7ff9f19e1700 1 -- 172.21.15.42:0/3544095952 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9ec08f780 con 0x7ff9ec119560 2019-12-05T05:22:54.280 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:54.266+0000 7ff9f19e1700 1 --2- 172.21.15.42:0/3544095952 >> v2:172.21.15.42:3300/0 conn(0x7ff9ec119560 0x7ff9ec1222c0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7ff9ec0a5390 tx=0x7ff9e0010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:22:54.280 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:54.266+0000 7ff9f19e1700 1 --2- 172.21.15.42:0/3544095952 >> v2:172.21.15.42:3300/0 conn(0x7ff9ec119560 0x7ff9ec1222c0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7ff9ec0a5390 tx=0x7ff9e0010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:22:54.280 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:54.266+0000 7ff9f19e1700 1 --2- 172.21.15.42:0/3544095952 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff9ec119560 0x7ff9ec1222c0 secure :-1 s=READY pgs=723 cs=0 l=1 rx=0x7ff9ec0a5390 tx=0x7ff9e0010b60).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:22:54.280 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:54.266+0000 7ff9f19e1700 1 --2- 172.21.15.42:0/3544095952 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff9ec119560 0x7ff9ec1222c0 secure :-1 s=THROTTLE_DONE pgs=723 cs=0 l=1 rx=0x7ff9ec0a5390 tx=0x7ff9e0010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:22:54.280 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:54.266+0000 7ff9de7fc700 1 -- 172.21.15.42:0/3544095952 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff9e0012590 con 0x7ff9ec119560 2019-12-05T05:22:54.281 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:54.266+0000 7ff9f19e1700 1 --2- 172.21.15.42:0/3544095952 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff9ec119560 0x7ff9ec1222c0 secure :-1 s=THROTTLE_DONE pgs=723 cs=0 l=1 rx=0x7ff9ec0a5390 tx=0x7ff9e0010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:22:54.281 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:54.266+0000 7ff9f19e1700 1 --2- 172.21.15.42:0/3544095952 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff9ec119560 0x7ff9ec1222c0 secure :-1 s=THROTTLE_DONE pgs=723 cs=0 l=1 rx=0x7ff9ec0a5390 tx=0x7ff9e0010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:22:54.281 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:54.266+0000 7ff9de7fc700 1 -- 172.21.15.42:0/3544095952 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9e0007150 con 0x7ff9ec119560 2019-12-05T05:22:54.281 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:54.266+0000 7ff9f3444700 1 -- 172.21.15.42:0/3544095952 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9ec122810 con 0x7ff9ec119560 2019-12-05T05:22:54.281 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:54.266+0000 7ff9de7fc700 1 -- 172.21.15.42:0/3544095952 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff9e0012b70 con 0x7ff9ec119560 2019-12-05T05:22:54.281 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:54.266+0000 7ff9f3444700 1 -- 172.21.15.42:0/3544095952 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9ec0785e0 con 0x7ff9ec119560 2019-12-05T05:22:54.281 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:54.266+0000 7ff9f19e1700 1 --2- 172.21.15.42:0/3544095952 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff9ec119560 0x7ff9ec1222c0 secure :-1 s=THROTTLE_DONE pgs=723 cs=0 l=1 rx=0x7ff9ec0a5390 tx=0x7ff9e0010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:22:54.282 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:54.266+0000 7ff9de7fc700 1 -- 172.21.15.42:0/3544095952 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7ff9e0005540 con 0x7ff9ec119560 2019-12-05T05:22:54.282 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:54.266+0000 7ff9f19e1700 1 --2- 172.21.15.42:0/3544095952 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff9ec119560 0x7ff9ec1222c0 secure :-1 s=THROTTLE_DONE pgs=723 cs=0 l=1 rx=0x7ff9ec0a5390 tx=0x7ff9e0010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:22:54.282 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:54.266+0000 7ff9de7fc700 1 --2- 172.21.15.42:0/3544095952 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff9d403ac60 0x7ff9d403d170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:22:54.282 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:54.266+0000 7ff9de7fc700 1 -- 172.21.15.42:0/3544095952 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 7063+0+0 (secure 0 0 0) 0x7ff9e004af20 con 0x7ff9ec119560 2019-12-05T05:22:54.282 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:54.266+0000 7ff9f11e0700 1 --2- 172.21.15.42:0/3544095952 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff9d403ac60 0x7ff9d403d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:22:54.282 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:54.270+0000 7ff9f11e0700 1 --2- 172.21.15.42:0/3544095952 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff9d403ac60 0x7ff9d403d170 crc :-1 s=READY pgs=324 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:22:54.283 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:54.274+0000 7ff9f3444700 1 -- 172.21.15.42:0/3544095952 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9ec05d110 con 0x7ff9ec119560 2019-12-05T05:22:54.288 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:54.282+0000 7ff9f19e1700 1 --2- 172.21.15.42:0/3544095952 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff9ec119560 0x7ff9ec1222c0 secure :-1 s=THROTTLE_DONE pgs=723 cs=0 l=1 rx=0x7ff9ec0a5390 tx=0x7ff9e0010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:22:54.288 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:54.282+0000 7ff9de7fc700 1 -- 172.21.15.42:0/3544095952 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7ff9e0031fa0 con 0x7ff9ec119560 2019-12-05T05:22:54.453 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:54.446+0000 7ff9f3444700 1 -- 172.21.15.42:0/3544095952 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff9ec05b8c0 con 0x7ff9ec119560 2019-12-05T05:22:54.454 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:54.446+0000 7ff9f19e1700 1 --2- 172.21.15.42:0/3544095952 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff9ec119560 0x7ff9ec1222c0 secure :-1 s=THROTTLE_DONE pgs=723 cs=0 l=1 rx=0x7ff9ec0a5390 tx=0x7ff9e0010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:22:54.454 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:54.446+0000 7ff9de7fc700 1 -- 172.21.15.42:0/3544095952 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 93 v93) v1 ==== 94+0+2108 (secure 0 0 0) 0x7ff9e0032200 con 0x7ff9ec119560 2019-12-05T05:22:54.454 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:22:54.454 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":93,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5123,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/1214659805","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1214659805},{"type":"v1","addr":"172.21.15.42:6835","nonce":1214659805}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":93}],"filesystems":[{"mdsmap":{"epoch":92,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:20:47.959813+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5066},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5066":{"gid":5066,"name":"b","rank":0,"incarnation":89,"state":"up:active","state_seq":30,"addr":"172.21.15.40:6833/3061061478","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3061061478},{"type":"v1","addr":"172.21.15.40:6833","nonce":3061061478}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:22:54.455 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 93 2019-12-05T05:22:54.456 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:54.450+0000 7ff9f3444700 1 -- 172.21.15.42:0/3544095952 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff9d403ac60 msgr2=0x7ff9d403d170 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:22:54.457 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:54.450+0000 7ff9f3444700 1 --2- 172.21.15.42:0/3544095952 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff9d403ac60 0x7ff9d403d170 crc :-1 s=READY pgs=324 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:22:54.457 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:54.450+0000 7ff9f3444700 1 -- 172.21.15.42:0/3544095952 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff9ec119560 msgr2=0x7ff9ec1222c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:22:54.457 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:54.450+0000 7ff9f3444700 1 --2- 172.21.15.42:0/3544095952 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff9ec119560 0x7ff9ec1222c0 secure :-1 s=READY pgs=723 cs=0 l=1 rx=0x7ff9ec0a5390 tx=0x7ff9e0010b60).stop 2019-12-05T05:22:54.457 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:54.450+0000 7ff9f3444700 1 -- 172.21.15.42:0/3544095952 shutdown_connections 2019-12-05T05:22:54.457 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:54.450+0000 7ff9f3444700 1 --2- 172.21.15.42:0/3544095952 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff9d403ac60 0x7ff9d403d170 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:22:54.457 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:54.450+0000 7ff9f3444700 1 --2- 172.21.15.42:0/3544095952 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff9ec119560 0x7ff9ec1222c0 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:22:54.458 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:54.450+0000 7ff9f3444700 1 --2- 172.21.15.42:0/3544095952 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff9ec1237a0 0x7ff9ec123bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:22:54.458 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:54.450+0000 7ff9f3444700 1 -- 172.21.15.42:0/3544095952 shutdown_connections 2019-12-05T05:22:54.458 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:54.450+0000 7ff9f3444700 1 -- 172.21.15.42:0/3544095952 wait complete. 2019-12-05T05:22:54.458 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:54.450+0000 7ff9f3444700 1 -- 172.21.15.42:0/3544095952 >> 172.21.15.42:0/3544095952 conn(0x7ff9ec11d170 msgr2=0x7ff9ec11c6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:22:54.466 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T05:22:56.468 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:22:56.575 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:56.562+0000 7f989134b700 1 Processor -- start 2019-12-05T05:22:56.575 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:56.562+0000 7f989134b700 1 -- start start 2019-12-05T05:22:56.575 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:56.562+0000 7f989134b700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f988c0a7740 0x7f988c08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:22:56.575 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:56.562+0000 7f989134b700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f988c07e720 0x7f988c07eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:22:56.576 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:56.562+0000 7f989134b700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f988c09f2e0 0x7f988c09f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:22:56.576 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:56.562+0000 7f989134b700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f988c0a0b00 con 0x7f988c09f2e0 2019-12-05T05:22:56.576 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:56.562+0000 7f989134b700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f988c08e5b0 con 0x7f988c07e720 2019-12-05T05:22:56.576 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:56.562+0000 7f989134b700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f988c08e850 con 0x7f988c0a7740 2019-12-05T05:22:56.576 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:56.562+0000 7f988affd700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f988c07e720 0x7f988c07eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:22:56.576 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:56.562+0000 7f988affd700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f988c07e720 0x7f988c07eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:60586/0 (socket says 172.21.15.42:60586) 2019-12-05T05:22:56.576 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:56.562+0000 7f988affd700 1 -- 172.21.15.42:0/1305935357 learned_addr learned my addr 172.21.15.42:0/1305935357 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:22:56.577 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:56.562+0000 7f988b7fe700 1 --2- 172.21.15.42:0/1305935357 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f988c0a7740 0x7f988c08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:22:56.578 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:56.562+0000 7f988bfff700 1 --2- 172.21.15.42:0/1305935357 >> v2:172.21.15.40:3300/0 conn(0x7f988c09f2e0 0x7f988c09f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:22:56.578 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:56.562+0000 7f988affd700 1 -- 172.21.15.42:0/1305935357 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f988c0a7740 msgr2=0x7f988c08df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:22:56.578 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:56.562+0000 7f988affd700 1 --2- 172.21.15.42:0/1305935357 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f988c0a7740 0x7f988c08df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:22:56.578 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:56.562+0000 7f988affd700 1 -- 172.21.15.42:0/1305935357 >> v2:172.21.15.40:3300/0 conn(0x7f988c09f2e0 msgr2=0x7f988c09f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:22:56.578 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:56.562+0000 7f988affd700 1 --2- 172.21.15.42:0/1305935357 >> v2:172.21.15.40:3300/0 conn(0x7f988c09f2e0 0x7f988c09f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:22:56.578 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:56.562+0000 7f988affd700 1 -- 172.21.15.42:0/1305935357 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f988c07f0a0 con 0x7f988c07e720 2019-12-05T05:22:56.578 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:56.562+0000 7f988bfff700 1 --2- 172.21.15.42:0/1305935357 >> v2:172.21.15.40:3300/0 conn(0x7f988c09f2e0 0x7f988c09f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T05:22:56.579 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:56.562+0000 7f988b7fe700 1 --2- 172.21.15.42:0/1305935357 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f988c0a7740 0x7f988c08df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T05:22:56.579 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:56.562+0000 7f988affd700 1 --2- 172.21.15.42:0/1305935357 >> v2:172.21.15.42:3300/0 conn(0x7f988c07e720 0x7f988c07eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f988c08e770 tx=0x7f988000fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:22:56.579 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:56.562+0000 7f988affd700 1 --2- 172.21.15.42:0/1305935357 >> v2:172.21.15.42:3300/0 conn(0x7f988c07e720 0x7f988c07eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f988c08e770 tx=0x7f988000fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:22:56.580 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:56.562+0000 7f988affd700 1 --2- 172.21.15.42:0/1305935357 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f988c07e720 0x7f988c07eb50 secure :-1 s=READY pgs=724 cs=0 l=1 rx=0x7f988c08e770 tx=0x7f988000fdb0).ready entity=mon.1 client_cookie=3a697142d770c595 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:22:56.580 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:56.566+0000 7f988affd700 1 --2- 172.21.15.42:0/1305935357 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f988c07e720 0x7f988c07eb50 secure :-1 s=THROTTLE_DONE pgs=724 cs=0 l=1 rx=0x7f988c08e770 tx=0x7f988000fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:22:56.580 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:56.566+0000 7f988affd700 1 --2- 172.21.15.42:0/1305935357 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f988c07e720 0x7f988c07eb50 secure :-1 s=THROTTLE_DONE pgs=724 cs=0 l=1 rx=0x7f988c08e770 tx=0x7f988000fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:22:56.580 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:56.566+0000 7f988a7fc700 1 -- 172.21.15.42:0/1305935357 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9880008c90 con 0x7f988c07e720 2019-12-05T05:22:56.580 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:56.566+0000 7f988affd700 1 --2- 172.21.15.42:0/1305935357 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f988c07e720 0x7f988c07eb50 secure :-1 s=THROTTLE_DONE pgs=724 cs=0 l=1 rx=0x7f988c08e770 tx=0x7f988000fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:22:56.581 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:56.566+0000 7f988a7fc700 1 -- 172.21.15.42:0/1305935357 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98800088b0 con 0x7f988c07e720 2019-12-05T05:22:56.581 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:56.566+0000 7f988a7fc700 1 -- 172.21.15.42:0/1305935357 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9880011b50 con 0x7f988c07e720 2019-12-05T05:22:56.581 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:56.566+0000 7f989134b700 1 -- 172.21.15.42:0/1305935357 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f988c07e720 msgr2=0x7f988c07eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:22:56.581 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:56.566+0000 7f989134b700 1 --2- 172.21.15.42:0/1305935357 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f988c07e720 0x7f988c07eb50 secure :-1 s=READY pgs=724 cs=0 l=1 rx=0x7f988c08e770 tx=0x7f988000fdb0).stop 2019-12-05T05:22:56.581 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:56.566+0000 7f989134b700 1 -- 172.21.15.42:0/1305935357 shutdown_connections 2019-12-05T05:22:56.581 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:56.566+0000 7f989134b700 1 --2- 172.21.15.42:0/1305935357 >> v2:172.21.15.40:3300/0 conn(0x7f988c09f2e0 0x7f988c09f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:22:56.582 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:56.566+0000 7f989134b700 1 --2- 172.21.15.42:0/1305935357 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f988c0a7740 0x7f988c08df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:22:56.582 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:56.566+0000 7f989134b700 1 --2- 172.21.15.42:0/1305935357 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f988c07e720 0x7f988c07eb50 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:22:56.582 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:56.566+0000 7f989134b700 1 -- 172.21.15.42:0/1305935357 shutdown_connections 2019-12-05T05:22:56.582 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:56.566+0000 7f989134b700 1 -- 172.21.15.42:0/1305935357 wait complete. 2019-12-05T05:22:56.582 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:56.566+0000 7f989134b700 1 -- 172.21.15.42:0/1305935357 >> 172.21.15.42:0/1305935357 conn(0x7f988c0aabe0 msgr2=0x7f988c0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:22:56.586 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:56.570+0000 7f989134b700 1 Processor -- start 2019-12-05T05:22:56.586 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:56.570+0000 7f989134b700 1 -- start start 2019-12-05T05:22:56.586 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:56.570+0000 7f989134b700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f988c11a920 0x7f988c117750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:22:56.586 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:56.570+0000 7f989134b700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f988c1315d0 con 0x7f988c0a38b0 2019-12-05T05:22:56.586 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:56.570+0000 7f989134b700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f988c11b640 con 0x7f988c11b210 2019-12-05T05:22:56.587 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:56.570+0000 7f989134b700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f988c0a3ce0 con 0x7f988c11a920 2019-12-05T05:22:56.587 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:56.570+0000 7f988b7fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f988c11a920 0x7f988c117750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:22:56.587 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:56.570+0000 7f988bfff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f988c11b210 0x7f988c0752e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:60632/0 (socket says 172.21.15.42:60632) 2019-12-05T05:22:56.587 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:56.570+0000 7f988bfff700 1 -- 172.21.15.42:0/1413557091 learned_addr learned my addr 172.21.15.42:0/1413557091 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:22:56.588 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:56.570+0000 7f988b7fe700 1 --2- 172.21.15.42:0/1413557091 >> v2:172.21.15.42:3300/0 conn(0x7f988c11a920 0x7f988c117750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:60590/0 (socket says 172.21.15.42:60590) 2019-12-05T05:22:56.588 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:56.570+0000 7f988b7fe700 1 -- 172.21.15.42:0/1413557091 >> v1:172.21.15.42:6789/0 conn(0x7f988c11b210 legacy=0x7f988c0752e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:22:56.588 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:56.570+0000 7f988b7fe700 1 -- 172.21.15.42:0/1413557091 >> v1:172.21.15.40:6789/0 conn(0x7f988c0a38b0 legacy=0x7f988c119790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:22:56.588 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:56.570+0000 7f9873fff700 1 -- 172.21.15.42:0/1413557091 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2152880242 0 0) 0x7f988c11b640 con 0x7f988c11b210 2019-12-05T05:22:56.588 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:56.570+0000 7f988b7fe700 1 -- 172.21.15.42:0/1413557091 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f988c11b870 con 0x7f988c11a920 2019-12-05T05:22:56.588 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:56.570+0000 7f988b7fe700 1 --2- 172.21.15.42:0/1413557091 >> v2:172.21.15.42:3300/0 conn(0x7f988c11a920 0x7f988c117750 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f988c124690 tx=0x7f987c010970).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:22:56.588 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:56.570+0000 7f988b7fe700 1 --2- 172.21.15.42:0/1413557091 >> v2:172.21.15.42:3300/0 conn(0x7f988c11a920 0x7f988c117750 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f988c124690 tx=0x7f987c010970).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:22:56.589 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:56.570+0000 7f988b7fe700 1 --2- 172.21.15.42:0/1413557091 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f988c11a920 0x7f988c117750 secure :-1 s=READY pgs=726 cs=0 l=1 rx=0x7f988c124690 tx=0x7f987c010970).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:22:56.589 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:56.570+0000 7f988b7fe700 1 --2- 172.21.15.42:0/1413557091 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f988c11a920 0x7f988c117750 secure :-1 s=THROTTLE_DONE pgs=726 cs=0 l=1 rx=0x7f988c124690 tx=0x7f987c010970).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:22:56.589 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:56.570+0000 7f9873fff700 1 -- 172.21.15.42:0/1413557091 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f987c005930 con 0x7f988c11a920 2019-12-05T05:22:56.589 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:56.570+0000 7f988b7fe700 1 --2- 172.21.15.42:0/1413557091 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f988c11a920 0x7f988c117750 secure :-1 s=THROTTLE_DONE pgs=726 cs=0 l=1 rx=0x7f988c124690 tx=0x7f987c010970).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:22:56.590 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:56.570+0000 7f989134b700 1 -- 172.21.15.42:0/1413557091 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f988c076af0 con 0x7f988c11a920 2019-12-05T05:22:56.590 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:56.570+0000 7f988b7fe700 1 --2- 172.21.15.42:0/1413557091 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f988c11a920 0x7f988c117750 secure :-1 s=THROTTLE_DONE pgs=726 cs=0 l=1 rx=0x7f988c124690 tx=0x7f987c010970).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:22:56.590 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:56.570+0000 7f9873fff700 1 -- 172.21.15.42:0/1413557091 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f987c007160 con 0x7f988c11a920 2019-12-05T05:22:56.590 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:56.570+0000 7f9873fff700 1 -- 172.21.15.42:0/1413557091 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f987c0060c0 con 0x7f988c11a920 2019-12-05T05:22:56.590 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:56.570+0000 7f989134b700 1 -- 172.21.15.42:0/1413557091 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f988c077070 con 0x7f988c11a920 2019-12-05T05:22:56.590 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:56.570+0000 7f988b7fe700 1 --2- 172.21.15.42:0/1413557091 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f988c11a920 0x7f988c117750 secure :-1 s=THROTTLE_DONE pgs=726 cs=0 l=1 rx=0x7f988c124690 tx=0x7f987c010970).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:22:56.591 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:56.574+0000 7f9873fff700 1 -- 172.21.15.42:0/1413557091 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f987c0063a0 con 0x7f988c11a920 2019-12-05T05:22:56.591 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:56.574+0000 7f988b7fe700 1 --2- 172.21.15.42:0/1413557091 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f988c11a920 0x7f988c117750 secure :-1 s=THROTTLE_DONE pgs=726 cs=0 l=1 rx=0x7f988c124690 tx=0x7f987c010970).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:22:56.591 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:56.574+0000 7f9873fff700 1 --2- 172.21.15.42:0/1413557091 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f986c03ac50 0x7f986c03d160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:22:56.591 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:56.574+0000 7f9873fff700 1 -- 172.21.15.42:0/1413557091 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 7063+0+0 (secure 0 0 0) 0x7f987c04aed0 con 0x7f988c11a920 2019-12-05T05:22:56.591 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:56.574+0000 7f988affd700 1 --2- 172.21.15.42:0/1413557091 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f986c03ac50 0x7f986c03d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:22:56.591 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:56.574+0000 7f988affd700 1 --2- 172.21.15.42:0/1413557091 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f986c03ac50 0x7f986c03d160 crc :-1 s=READY pgs=325 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:22:56.591 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:56.578+0000 7f989134b700 1 -- 172.21.15.42:0/1413557091 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f988c124c60 con 0x7f988c11a920 2019-12-05T05:22:56.592 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:56.582+0000 7f988b7fe700 1 --2- 172.21.15.42:0/1413557091 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f988c11a920 0x7f988c117750 secure :-1 s=THROTTLE_DONE pgs=726 cs=0 l=1 rx=0x7f988c124690 tx=0x7f987c010970).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:22:56.592 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:56.582+0000 7f9873fff700 1 -- 172.21.15.42:0/1413557091 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f987c031f30 con 0x7f988c11a920 2019-12-05T05:22:56.765 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:56.758+0000 7f989134b700 1 -- 172.21.15.42:0/1413557091 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f988c05d110 con 0x7f988c11a920 2019-12-05T05:22:56.766 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:56.758+0000 7f988b7fe700 1 --2- 172.21.15.42:0/1413557091 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f988c11a920 0x7f988c117750 secure :-1 s=THROTTLE_DONE pgs=726 cs=0 l=1 rx=0x7f988c124690 tx=0x7f987c010970).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:22:56.766 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:56.758+0000 7f9873fff700 1 -- 172.21.15.42:0/1413557091 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 93 v93) v1 ==== 94+0+2108 (secure 0 0 0) 0x7f987c032190 con 0x7f988c11a920 2019-12-05T05:22:56.768 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:22:56.769 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":93,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5123,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/1214659805","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1214659805},{"type":"v1","addr":"172.21.15.42:6835","nonce":1214659805}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":93}],"filesystems":[{"mdsmap":{"epoch":92,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:20:47.959813+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5066},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5066":{"gid":5066,"name":"b","rank":0,"incarnation":89,"state":"up:active","state_seq":30,"addr":"172.21.15.40:6833/3061061478","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3061061478},{"type":"v1","addr":"172.21.15.40:6833","nonce":3061061478}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:22:56.769 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 93 2019-12-05T05:22:56.771 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:56.766+0000 7f989134b700 1 -- 172.21.15.42:0/1413557091 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f986c03ac50 msgr2=0x7f986c03d160 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:22:56.771 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:56.766+0000 7f989134b700 1 --2- 172.21.15.42:0/1413557091 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f986c03ac50 0x7f986c03d160 crc :-1 s=READY pgs=325 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:22:56.771 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:56.766+0000 7f989134b700 1 -- 172.21.15.42:0/1413557091 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f988c11a920 msgr2=0x7f988c117750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:22:56.771 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:56.766+0000 7f989134b700 1 --2- 172.21.15.42:0/1413557091 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f988c11a920 0x7f988c117750 secure :-1 s=READY pgs=726 cs=0 l=1 rx=0x7f988c124690 tx=0x7f987c010970).stop 2019-12-05T05:22:56.771 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:56.766+0000 7f989134b700 1 -- 172.21.15.42:0/1413557091 shutdown_connections 2019-12-05T05:22:56.772 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:56.766+0000 7f989134b700 1 --2- 172.21.15.42:0/1413557091 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f986c03ac50 0x7f986c03d160 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:22:56.772 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:56.766+0000 7f989134b700 1 --2- 172.21.15.42:0/1413557091 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f988c11a920 0x7f988c117750 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:22:56.772 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:56.766+0000 7f989134b700 1 -- 172.21.15.42:0/1413557091 shutdown_connections 2019-12-05T05:22:56.772 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:56.766+0000 7f989134b700 1 -- 172.21.15.42:0/1413557091 wait complete. 2019-12-05T05:22:56.772 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:56.766+0000 7f989134b700 1 -- 172.21.15.42:0/1413557091 >> 172.21.15.42:0/1413557091 conn(0x7f988c09fa10 msgr2=0x7f988c07e4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:22:56.781 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T05:22:58.783 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:22:58.892 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:58.882+0000 7fc130c9f700 1 Processor -- start 2019-12-05T05:22:58.892 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:58.882+0000 7fc130c9f700 1 -- start start 2019-12-05T05:22:58.892 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:58.882+0000 7fc130c9f700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc12c0a7740 0x7fc12c08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:22:58.893 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:58.886+0000 7fc130c9f700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc12c09fae0 0x7fc12c07eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:22:58.893 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:58.886+0000 7fc130c9f700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc12c0a0710 con 0x7fc12c07e720 2019-12-05T05:22:58.893 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:58.886+0000 7fc130c9f700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc12c0a0b00 con 0x7fc12c0a7740 2019-12-05T05:22:58.893 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:58.886+0000 7fc130c9f700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc12c08e5b0 con 0x7fc12c09fae0 2019-12-05T05:22:58.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:58.886+0000 7fc12b59e700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc12c09fae0 0x7fc12c07eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:22:58.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:58.886+0000 7fc12ad9d700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc12c0a7740 0x7fc12c08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:22:58.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:58.886+0000 7fc12b59e700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc12c09fae0 0x7fc12c07eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:32836/0 (socket says 172.21.15.42:32836) 2019-12-05T05:22:58.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:58.886+0000 7fc12b59e700 1 -- 172.21.15.42:0/1068990262 learned_addr learned my addr 172.21.15.42:0/1068990262 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:22:58.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:58.886+0000 7fc12b59e700 1 -- 172.21.15.42:0/1068990262 >> v2:172.21.15.40:3300/0 conn(0x7fc12c0a7740 msgr2=0x7fc12c08df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:22:58.895 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:58.886+0000 7fc12b59e700 1 --2- 172.21.15.42:0/1068990262 >> v2:172.21.15.40:3300/0 conn(0x7fc12c0a7740 0x7fc12c08df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:22:58.895 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:58.886+0000 7fc12b59e700 1 -- 172.21.15.42:0/1068990262 >> v1:172.21.15.40:6789/0 conn(0x7fc12c07e720 legacy=0x7fc12c09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:22:58.895 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:58.886+0000 7fc12b59e700 1 -- 172.21.15.42:0/1068990262 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc12c07f0a0 con 0x7fc12c09fae0 2019-12-05T05:22:58.895 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:58.886+0000 7fc12b59e700 1 --2- 172.21.15.42:0/1068990262 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc12c09fae0 0x7fc12c07eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fc11c008790 tx=0x7fc11c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:22:58.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:58.886+0000 7fc12b59e700 1 --2- 172.21.15.42:0/1068990262 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc12c09fae0 0x7fc12c07eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fc11c008790 tx=0x7fc11c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:22:58.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:58.886+0000 7fc12b59e700 1 --2- 172.21.15.42:0/1068990262 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc12c09fae0 0x7fc12c07eb50 secure :-1 s=READY pgs=153 cs=0 l=1 rx=0x7fc11c008790 tx=0x7fc11c00fd30).ready entity=mon.2 client_cookie=8f329868c8a027c8 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:22:58.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:58.886+0000 7fc12b59e700 1 --2- 172.21.15.42:0/1068990262 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc12c09fae0 0x7fc12c07eb50 secure :-1 s=THROTTLE_DONE pgs=153 cs=0 l=1 rx=0x7fc11c008790 tx=0x7fc11c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:22:58.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:58.886+0000 7fc12b59e700 1 --2- 172.21.15.42:0/1068990262 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc12c09fae0 0x7fc12c07eb50 secure :-1 s=THROTTLE_DONE pgs=153 cs=0 l=1 rx=0x7fc11c008790 tx=0x7fc11c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:22:58.897 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:58.886+0000 7fc12b59e700 1 --2- 172.21.15.42:0/1068990262 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc12c09fae0 0x7fc12c07eb50 secure :-1 s=THROTTLE_DONE pgs=153 cs=0 l=1 rx=0x7fc11c008790 tx=0x7fc11c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:22:58.897 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:58.886+0000 7fc129d9b700 1 -- 172.21.15.42:0/1068990262 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc11c008c90 con 0x7fc12c09fae0 2019-12-05T05:22:58.897 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:58.886+0000 7fc129d9b700 1 -- 172.21.15.42:0/1068990262 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc11c0088b0 con 0x7fc12c09fae0 2019-12-05T05:22:58.897 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:58.886+0000 7fc129d9b700 1 -- 172.21.15.42:0/1068990262 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc11c011b10 con 0x7fc12c09fae0 2019-12-05T05:22:58.898 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:58.886+0000 7fc130c9f700 1 -- 172.21.15.42:0/1068990262 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc12c09fae0 msgr2=0x7fc12c07eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:22:58.898 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:58.886+0000 7fc130c9f700 1 --2- 172.21.15.42:0/1068990262 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc12c09fae0 0x7fc12c07eb50 secure :-1 s=READY pgs=153 cs=0 l=1 rx=0x7fc11c008790 tx=0x7fc11c00fd30).stop 2019-12-05T05:22:58.898 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:58.886+0000 7fc130c9f700 1 -- 172.21.15.42:0/1068990262 shutdown_connections 2019-12-05T05:22:58.898 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:58.886+0000 7fc130c9f700 1 --2- 172.21.15.42:0/1068990262 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc12c09fae0 0x7fc12c07eb50 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:22:58.899 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:58.886+0000 7fc130c9f700 1 --2- 172.21.15.42:0/1068990262 >> v2:172.21.15.40:3300/0 conn(0x7fc12c0a7740 0x7fc12c08df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:22:58.899 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:58.886+0000 7fc130c9f700 1 -- 172.21.15.42:0/1068990262 shutdown_connections 2019-12-05T05:22:58.900 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:58.886+0000 7fc130c9f700 1 -- 172.21.15.42:0/1068990262 wait complete. 2019-12-05T05:22:58.900 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:58.886+0000 7fc130c9f700 1 -- 172.21.15.42:0/1068990262 >> 172.21.15.42:0/1068990262 conn(0x7fc12c0aabe0 msgr2=0x7fc12c0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:22:58.905 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:58.886+0000 7fc130c9f700 1 Processor -- start 2019-12-05T05:22:58.906 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:58.886+0000 7fc130c9f700 1 -- start start 2019-12-05T05:22:58.906 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:58.886+0000 7fc130c9f700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc12c0a3a60 0x7fc12c11c150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:22:58.906 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:58.886+0000 7fc130c9f700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc12c06eed0 con 0x7fc12c06e2a0 2019-12-05T05:22:58.906 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:58.886+0000 7fc130c9f700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc12c11b410 con 0x7fc12c117c90 2019-12-05T05:22:58.907 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:58.886+0000 7fc130c9f700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc12c1180c0 con 0x7fc12c0a3a60 2019-12-05T05:22:58.907 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:58.886+0000 7fc12a59c700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fc12c117c90 0x7fc12c11ac10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:60644/0 (socket says 172.21.15.42:60644) 2019-12-05T05:22:58.907 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:58.886+0000 7fc12a59c700 1 -- 172.21.15.42:0/3834997265 learned_addr learned my addr 172.21.15.42:0/3834997265 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:22:58.907 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:58.886+0000 7fc12ad9d700 1 --2- 172.21.15.42:0/3834997265 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc12c0a3a60 0x7fc12c11c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:22:58.908 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:58.890+0000 7fc1237fe700 1 -- 172.21.15.42:0/3834997265 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 608556241 0 0) 0x7fc12c11b410 con 0x7fc12c117c90 2019-12-05T05:22:58.908 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:58.890+0000 7fc1237fe700 1 -- 172.21.15.42:0/3834997265 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1100030b0 con 0x7fc12c117c90 2019-12-05T05:22:58.908 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:58.890+0000 7fc12ad9d700 1 -- 172.21.15.42:0/3834997265 >> v1:172.21.15.42:6789/0 conn(0x7fc12c117c90 legacy=0x7fc12c11ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:22:58.908 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:58.890+0000 7fc12ad9d700 1 -- 172.21.15.42:0/3834997265 >> v1:172.21.15.40:6789/0 conn(0x7fc12c06e2a0 legacy=0x7fc12c06e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:22:58.908 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:58.890+0000 7fc1237fe700 1 -- 172.21.15.42:0/3834997265 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2364653656 0 0) 0x7fc12c06eed0 con 0x7fc12c06e2a0 2019-12-05T05:22:58.909 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:58.890+0000 7fc12ad9d700 1 -- 172.21.15.42:0/3834997265 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc12c0785e0 con 0x7fc12c0a3a60 2019-12-05T05:22:58.909 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:58.890+0000 7fc12ad9d700 1 --2- 172.21.15.42:0/3834997265 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc12c0a3a60 0x7fc12c11c150 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fc1240032b0 tx=0x7fc124010d10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:22:58.910 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:58.890+0000 7fc12ad9d700 1 --2- 172.21.15.42:0/3834997265 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc12c0a3a60 0x7fc12c11c150 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fc1240032b0 tx=0x7fc124010d10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:22:58.910 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:58.890+0000 7fc12ad9d700 1 --2- 172.21.15.42:0/3834997265 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc12c0a3a60 0x7fc12c11c150 secure :-1 s=READY pgs=154 cs=0 l=1 rx=0x7fc1240032b0 tx=0x7fc124010d10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:22:58.910 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:58.890+0000 7fc12ad9d700 1 --2- 172.21.15.42:0/3834997265 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc12c0a3a60 0x7fc12c11c150 secure :-1 s=THROTTLE_DONE pgs=154 cs=0 l=1 rx=0x7fc1240032b0 tx=0x7fc124010d10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:22:58.910 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:58.890+0000 7fc12ad9d700 1 --2- 172.21.15.42:0/3834997265 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc12c0a3a60 0x7fc12c11c150 secure :-1 s=THROTTLE_DONE pgs=154 cs=0 l=1 rx=0x7fc1240032b0 tx=0x7fc124010d10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:22:58.911 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:58.890+0000 7fc1237fe700 1 -- 172.21.15.42:0/3834997265 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc124012c70 con 0x7fc12c0a3a60 2019-12-05T05:22:58.911 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:58.890+0000 7fc12ad9d700 1 --2- 172.21.15.42:0/3834997265 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc12c0a3a60 0x7fc12c11c150 secure :-1 s=THROTTLE_DONE pgs=154 cs=0 l=1 rx=0x7fc1240032b0 tx=0x7fc124010d10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:22:58.911 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:58.890+0000 7fc1237fe700 1 -- 172.21.15.42:0/3834997265 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc124012890 con 0x7fc12c0a3a60 2019-12-05T05:22:58.911 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:58.890+0000 7fc130c9f700 1 -- 172.21.15.42:0/3834997265 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc12c0765c0 con 0x7fc12c0a3a60 2019-12-05T05:22:58.912 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:58.890+0000 7fc1237fe700 1 -- 172.21.15.42:0/3834997265 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc12400e7c0 con 0x7fc12c0a3a60 2019-12-05T05:22:58.912 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:58.890+0000 7fc130c9f700 1 -- 172.21.15.42:0/3834997265 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc12c076a90 con 0x7fc12c0a3a60 2019-12-05T05:22:58.913 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:58.890+0000 7fc12ad9d700 1 --2- 172.21.15.42:0/3834997265 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc12c0a3a60 0x7fc12c11c150 secure :-1 s=THROTTLE_DONE pgs=154 cs=0 l=1 rx=0x7fc1240032b0 tx=0x7fc124010d10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:22:58.913 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:58.890+0000 7fc1237fe700 1 -- 172.21.15.42:0/3834997265 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fc12400ea50 con 0x7fc12c0a3a60 2019-12-05T05:22:58.913 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:58.890+0000 7fc12ad9d700 1 --2- 172.21.15.42:0/3834997265 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc12c0a3a60 0x7fc12c11c150 secure :-1 s=THROTTLE_DONE pgs=154 cs=0 l=1 rx=0x7fc1240032b0 tx=0x7fc124010d10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:22:58.913 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:58.890+0000 7fc1237fe700 1 --2- 172.21.15.42:0/3834997265 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc11003b210 0x7fc11003d720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:22:58.914 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:58.890+0000 7fc1237fe700 1 -- 172.21.15.42:0/3834997265 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 7063+0+0 (secure 0 0 0) 0x7fc12404bad0 con 0x7fc12c0a3a60 2019-12-05T05:22:58.914 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:58.890+0000 7fc12a59c700 1 --2- 172.21.15.42:0/3834997265 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc11003b210 0x7fc11003d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:22:58.914 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:58.894+0000 7fc12a59c700 1 --2- 172.21.15.42:0/3834997265 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc11003b210 0x7fc11003d720 crc :-1 s=READY pgs=326 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:22:58.914 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:58.894+0000 7fc130c9f700 1 -- 172.21.15.42:0/3834997265 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc12c05d110 con 0x7fc12c0a3a60 2019-12-05T05:22:58.915 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:58.906+0000 7fc12ad9d700 1 --2- 172.21.15.42:0/3834997265 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc12c0a3a60 0x7fc12c11c150 secure :-1 s=THROTTLE_DONE pgs=154 cs=0 l=1 rx=0x7fc1240032b0 tx=0x7fc124010d10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:22:58.915 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:58.906+0000 7fc1237fe700 1 -- 172.21.15.42:0/3834997265 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fc12404bde0 con 0x7fc12c0a3a60 2019-12-05T05:22:59.077 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:59.070+0000 7fc130c9f700 1 -- 172.21.15.42:0/3834997265 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc12c05b8c0 con 0x7fc12c0a3a60 2019-12-05T05:22:59.078 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:59.070+0000 7fc12ad9d700 1 --2- 172.21.15.42:0/3834997265 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc12c0a3a60 0x7fc12c11c150 secure :-1 s=THROTTLE_DONE pgs=154 cs=0 l=1 rx=0x7fc1240032b0 tx=0x7fc124010d10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:22:59.078 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:59.070+0000 7fc1237fe700 1 -- 172.21.15.42:0/3834997265 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 93 v93) v1 ==== 94+0+2108 (secure 0 0 0) 0x7fc124032b50 con 0x7fc12c0a3a60 2019-12-05T05:22:59.080 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 93 2019-12-05T05:22:59.081 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:22:59.081 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":93,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5123,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/1214659805","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1214659805},{"type":"v1","addr":"172.21.15.42:6835","nonce":1214659805}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":93}],"filesystems":[{"mdsmap":{"epoch":92,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:20:47.959813+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5066},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5066":{"gid":5066,"name":"b","rank":0,"incarnation":89,"state":"up:active","state_seq":30,"addr":"172.21.15.40:6833/3061061478","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3061061478},{"type":"v1","addr":"172.21.15.40:6833","nonce":3061061478}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:22:59.083 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:59.078+0000 7fc130c9f700 1 -- 172.21.15.42:0/3834997265 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc11003b210 msgr2=0x7fc11003d720 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:22:59.083 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:59.078+0000 7fc130c9f700 1 --2- 172.21.15.42:0/3834997265 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc11003b210 0x7fc11003d720 crc :-1 s=READY pgs=326 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:22:59.083 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:59.078+0000 7fc130c9f700 1 -- 172.21.15.42:0/3834997265 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc12c0a3a60 msgr2=0x7fc12c11c150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:22:59.084 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:59.078+0000 7fc130c9f700 1 --2- 172.21.15.42:0/3834997265 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc12c0a3a60 0x7fc12c11c150 secure :-1 s=READY pgs=154 cs=0 l=1 rx=0x7fc1240032b0 tx=0x7fc124010d10).stop 2019-12-05T05:22:59.084 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:59.078+0000 7fc130c9f700 1 -- 172.21.15.42:0/3834997265 shutdown_connections 2019-12-05T05:22:59.084 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:59.078+0000 7fc130c9f700 1 --2- 172.21.15.42:0/3834997265 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc11003b210 0x7fc11003d720 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:22:59.084 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:59.078+0000 7fc130c9f700 1 --2- 172.21.15.42:0/3834997265 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc12c0a3a60 0x7fc12c11c150 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:22:59.085 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:59.078+0000 7fc130c9f700 1 -- 172.21.15.42:0/3834997265 shutdown_connections 2019-12-05T05:22:59.085 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:59.078+0000 7fc130c9f700 1 -- 172.21.15.42:0/3834997265 wait complete. 2019-12-05T05:22:59.085 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:22:59.078+0000 7fc130c9f700 1 -- 172.21.15.42:0/3834997265 >> 172.21.15.42:0/3834997265 conn(0x7fc12c0aabe0 msgr2=0x7fc12c0a6a40 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:22:59.093 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T05:23:01.094 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:23:01.201 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:01.194+0000 7fb0d4a45700 1 Processor -- start 2019-12-05T05:23:01.202 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:01.194+0000 7fb0d4a45700 1 -- start start 2019-12-05T05:23:01.202 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:01.194+0000 7fb0d4a45700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb0d00a7740 0x7fb0d008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:23:01.202 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:01.194+0000 7fb0d4a45700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb0d007e720 0x7fb0d007eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:23:01.202 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:01.194+0000 7fb0d4a45700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0d00a0710 con 0x7fb0d009f2e0 2019-12-05T05:23:01.202 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:01.194+0000 7fb0d4a45700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0d00a0b00 con 0x7fb0d00a7740 2019-12-05T05:23:01.202 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:01.194+0000 7fb0d4a45700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0d008e5b0 con 0x7fb0d007e720 2019-12-05T05:23:01.203 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:01.194+0000 7fb0ce59c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb0d007e720 0x7fb0d007eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:23:01.203 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:01.194+0000 7fb0ce59c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb0d007e720 0x7fb0d007eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:60616/0 (socket says 172.21.15.42:60616) 2019-12-05T05:23:01.203 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:01.194+0000 7fb0ce59c700 1 -- 172.21.15.42:0/848933633 learned_addr learned my addr 172.21.15.42:0/848933633 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:23:01.203 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:01.194+0000 7fb0ced9d700 1 --2- 172.21.15.42:0/848933633 >> v2:172.21.15.40:3300/0 conn(0x7fb0d00a7740 0x7fb0d008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:23:01.203 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:01.194+0000 7fb0ce59c700 1 -- 172.21.15.42:0/848933633 >> v2:172.21.15.40:3300/0 conn(0x7fb0d00a7740 msgr2=0x7fb0d008df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:23:01.203 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:01.194+0000 7fb0ce59c700 1 --2- 172.21.15.42:0/848933633 >> v2:172.21.15.40:3300/0 conn(0x7fb0d00a7740 0x7fb0d008df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:23:01.203 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:01.194+0000 7fb0ce59c700 1 -- 172.21.15.42:0/848933633 >> v1:172.21.15.40:6789/0 conn(0x7fb0d009f2e0 legacy=0x7fb0d009f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:23:01.204 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:01.194+0000 7fb0ce59c700 1 -- 172.21.15.42:0/848933633 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0d007f0a0 con 0x7fb0d007e720 2019-12-05T05:23:01.204 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:01.194+0000 7fb0ce59c700 1 --2- 172.21.15.42:0/848933633 >> v2:172.21.15.42:3300/0 conn(0x7fb0d007e720 0x7fb0d007eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fb0b8002f30 tx=0x7fb0b800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:01.204 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:01.194+0000 7fb0ce59c700 1 --2- 172.21.15.42:0/848933633 >> v2:172.21.15.42:3300/0 conn(0x7fb0d007e720 0x7fb0d007eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fb0b8002f30 tx=0x7fb0b800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:01.204 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:01.194+0000 7fb0ce59c700 1 --2- 172.21.15.42:0/848933633 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb0d007e720 0x7fb0d007eb50 secure :-1 s=READY pgs=728 cs=0 l=1 rx=0x7fb0b8002f30 tx=0x7fb0b800fd30).ready entity=mon.1 client_cookie=19e8b2cc90c2b0d7 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:23:01.204 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:01.194+0000 7fb0ce59c700 1 --2- 172.21.15.42:0/848933633 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb0d007e720 0x7fb0d007eb50 secure :-1 s=THROTTLE_DONE pgs=728 cs=0 l=1 rx=0x7fb0b8002f30 tx=0x7fb0b800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:01.204 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:01.194+0000 7fb0ce59c700 1 --2- 172.21.15.42:0/848933633 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb0d007e720 0x7fb0d007eb50 secure :-1 s=THROTTLE_DONE pgs=728 cs=0 l=1 rx=0x7fb0b8002f30 tx=0x7fb0b800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:01.204 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:01.194+0000 7fb0cdd9b700 1 -- 172.21.15.42:0/848933633 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb0b8011370 con 0x7fb0d007e720 2019-12-05T05:23:01.205 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:01.194+0000 7fb0ce59c700 1 --2- 172.21.15.42:0/848933633 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb0d007e720 0x7fb0d007eb50 secure :-1 s=THROTTLE_DONE pgs=728 cs=0 l=1 rx=0x7fb0b8002f30 tx=0x7fb0b800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:01.205 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:01.194+0000 7fb0cdd9b700 1 -- 172.21.15.42:0/848933633 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0b8008bb0 con 0x7fb0d007e720 2019-12-05T05:23:01.205 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:01.194+0000 7fb0cdd9b700 1 -- 172.21.15.42:0/848933633 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb0b8011950 con 0x7fb0d007e720 2019-12-05T05:23:01.205 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:01.194+0000 7fb0d4a45700 1 -- 172.21.15.42:0/848933633 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb0d007e720 msgr2=0x7fb0d007eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:01.205 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:01.194+0000 7fb0d4a45700 1 --2- 172.21.15.42:0/848933633 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb0d007e720 0x7fb0d007eb50 secure :-1 s=READY pgs=728 cs=0 l=1 rx=0x7fb0b8002f30 tx=0x7fb0b800fd30).stop 2019-12-05T05:23:01.205 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:01.198+0000 7fb0d4a45700 1 -- 172.21.15.42:0/848933633 shutdown_connections 2019-12-05T05:23:01.205 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:01.198+0000 7fb0d4a45700 1 --2- 172.21.15.42:0/848933633 >> v2:172.21.15.40:3300/0 conn(0x7fb0d00a7740 0x7fb0d008df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:23:01.206 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:01.198+0000 7fb0d4a45700 1 --2- 172.21.15.42:0/848933633 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb0d007e720 0x7fb0d007eb50 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:23:01.206 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:01.198+0000 7fb0d4a45700 1 -- 172.21.15.42:0/848933633 shutdown_connections 2019-12-05T05:23:01.206 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:01.198+0000 7fb0d4a45700 1 -- 172.21.15.42:0/848933633 wait complete. 2019-12-05T05:23:01.206 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:01.198+0000 7fb0d4a45700 1 -- 172.21.15.42:0/848933633 >> 172.21.15.42:0/848933633 conn(0x7fb0d00aabe0 msgr2=0x7fb0d00a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:23:01.209 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:01.198+0000 7fb0d4a45700 1 Processor -- start 2019-12-05T05:23:01.209 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:01.198+0000 7fb0d4a45700 1 -- start start 2019-12-05T05:23:01.209 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:01.202+0000 7fb0d4a45700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb0d007b120 0x7fb0d00a5ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:23:01.209 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:01.202+0000 7fb0d4a45700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0d007a090 con 0x7fb0d00a3a60 2019-12-05T05:23:01.209 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:01.202+0000 7fb0d4a45700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0d00a60f0 con 0x7fb0d00a6e50 2019-12-05T05:23:01.210 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:01.202+0000 7fb0d4a45700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0d007a2c0 con 0x7fb0d007b120 2019-12-05T05:23:01.210 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:01.202+0000 7fb0cf59e700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fb0d00a6e50 0x7fb0d008f350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:60664/0 (socket says 172.21.15.42:60664) 2019-12-05T05:23:01.210 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:01.202+0000 7fb0cf59e700 1 -- 172.21.15.42:0/1662054527 learned_addr learned my addr 172.21.15.42:0/1662054527 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:23:01.210 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:01.202+0000 7fb0ced9d700 1 --2- 172.21.15.42:0/1662054527 >> v2:172.21.15.40:3300/0 conn(0x7fb0d007b120 0x7fb0d00a5ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:23:01.210 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:01.202+0000 7fb0c77fe700 1 -- 172.21.15.42:0/1662054527 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3764748226 0 0) 0x7fb0d00a60f0 con 0x7fb0d00a6e50 2019-12-05T05:23:01.210 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:01.202+0000 7fb0c77fe700 1 -- 172.21.15.42:0/1662054527 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0b40030b0 con 0x7fb0d00a6e50 2019-12-05T05:23:01.211 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:01.202+0000 7fb0ced9d700 1 -- 172.21.15.42:0/1662054527 >> v1:172.21.15.42:6789/0 conn(0x7fb0d00a6e50 legacy=0x7fb0d008f350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:01.211 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:01.202+0000 7fb0c77fe700 1 -- 172.21.15.42:0/1662054527 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 2124416228 0 0) 0x7fb0b40030b0 con 0x7fb0d00a6e50 2019-12-05T05:23:01.211 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:01.202+0000 7fb0ced9d700 1 -- 172.21.15.42:0/1662054527 >> v1:172.21.15.40:6789/0 conn(0x7fb0d00a3a60 legacy=0x7fb0d008eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:01.211 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:01.202+0000 7fb0ced9d700 1 -- 172.21.15.42:0/1662054527 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0d00a6320 con 0x7fb0d007b120 2019-12-05T05:23:01.212 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:01.202+0000 7fb0ced9d700 1 --2- 172.21.15.42:0/1662054527 >> v2:172.21.15.40:3300/0 conn(0x7fb0d007b120 0x7fb0d00a5ba0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fb0b8008370 tx=0x7fb0c8001980).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:01.212 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:01.202+0000 7fb0ced9d700 1 --2- 172.21.15.42:0/1662054527 >> v2:172.21.15.40:3300/0 conn(0x7fb0d007b120 0x7fb0d00a5ba0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fb0b8008370 tx=0x7fb0c8001980).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:01.212 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:01.202+0000 7fb0ced9d700 1 --2- 172.21.15.42:0/1662054527 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb0d007b120 0x7fb0d00a5ba0 secure :-1 s=READY pgs=585 cs=0 l=1 rx=0x7fb0b8008370 tx=0x7fb0c8001980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:23:01.212 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:01.202+0000 7fb0ced9d700 1 --2- 172.21.15.42:0/1662054527 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb0d007b120 0x7fb0d00a5ba0 secure :-1 s=THROTTLE_DONE pgs=585 cs=0 l=1 rx=0x7fb0b8008370 tx=0x7fb0c8001980).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:01.212 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:01.202+0000 7fb0c77fe700 1 -- 172.21.15.42:0/1662054527 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb0c8001c30 con 0x7fb0d007b120 2019-12-05T05:23:01.213 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:01.202+0000 7fb0ced9d700 1 --2- 172.21.15.42:0/1662054527 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb0d007b120 0x7fb0d00a5ba0 secure :-1 s=THROTTLE_DONE pgs=585 cs=0 l=1 rx=0x7fb0b8008370 tx=0x7fb0c8001980).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:01.213 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:01.202+0000 7fb0d4a45700 1 -- 172.21.15.42:0/1662054527 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0d0130860 con 0x7fb0d007b120 2019-12-05T05:23:01.213 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:01.202+0000 7fb0c77fe700 1 -- 172.21.15.42:0/1662054527 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0c8001a40 con 0x7fb0d007b120 2019-12-05T05:23:01.213 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:01.202+0000 7fb0ced9d700 1 --2- 172.21.15.42:0/1662054527 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb0d007b120 0x7fb0d00a5ba0 secure :-1 s=THROTTLE_DONE pgs=585 cs=0 l=1 rx=0x7fb0b8008370 tx=0x7fb0c8001980).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:01.213 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:01.202+0000 7fb0c77fe700 1 -- 172.21.15.42:0/1662054527 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb0c0005b20 con 0x7fb0d007b120 2019-12-05T05:23:01.213 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:01.202+0000 7fb0d4a45700 1 -- 172.21.15.42:0/1662054527 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0d0130e10 con 0x7fb0d007b120 2019-12-05T05:23:01.214 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:01.206+0000 7fb0ced9d700 1 --2- 172.21.15.42:0/1662054527 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb0d007b120 0x7fb0d00a5ba0 secure :-1 s=THROTTLE_DONE pgs=585 cs=0 l=1 rx=0x7fb0b8008370 tx=0x7fb0c8001980).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:01.214 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:01.206+0000 7fb0ced9d700 1 --2- 172.21.15.42:0/1662054527 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb0d007b120 0x7fb0d00a5ba0 secure :-1 s=THROTTLE_DONE pgs=585 cs=0 l=1 rx=0x7fb0b8008370 tx=0x7fb0c8001980).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:01.214 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:01.206+0000 7fb0c77fe700 1 -- 172.21.15.42:0/1662054527 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fb0c0007920 con 0x7fb0d007b120 2019-12-05T05:23:01.215 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:01.206+0000 7fb0c77fe700 1 --2- 172.21.15.42:0/1662054527 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb0b403b200 0x7fb0b403d710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:23:01.215 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:01.206+0000 7fb0c77fe700 1 -- 172.21.15.42:0/1662054527 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 7063+0+0 (secure 0 0 0) 0x7fb0c004a7b0 con 0x7fb0d007b120 2019-12-05T05:23:01.215 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:01.206+0000 7fb0ce59c700 1 --2- 172.21.15.42:0/1662054527 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb0b403b200 0x7fb0b403d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:23:01.216 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:01.206+0000 7fb0ce59c700 1 --2- 172.21.15.42:0/1662054527 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb0b403b200 0x7fb0b403d710 crc :-1 s=READY pgs=327 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:23:01.216 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:01.210+0000 7fb0d4a45700 1 -- 172.21.15.42:0/1662054527 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0d0128130 con 0x7fb0d007b120 2019-12-05T05:23:01.222 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:01.218+0000 7fb0ced9d700 1 --2- 172.21.15.42:0/1662054527 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb0d007b120 0x7fb0d00a5ba0 secure :-1 s=THROTTLE_DONE pgs=585 cs=0 l=1 rx=0x7fb0b8008370 tx=0x7fb0c8001980).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:01.222 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:01.218+0000 7fb0c77fe700 1 -- 172.21.15.42:0/1662054527 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fb0c00317f0 con 0x7fb0d007b120 2019-12-05T05:23:01.394 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:01.390+0000 7fb0d4a45700 1 -- 172.21.15.42:0/1662054527 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb0d00aae10 con 0x7fb0d007b120 2019-12-05T05:23:01.395 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:01.390+0000 7fb0ced9d700 1 --2- 172.21.15.42:0/1662054527 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb0d007b120 0x7fb0d00a5ba0 secure :-1 s=THROTTLE_DONE pgs=585 cs=0 l=1 rx=0x7fb0b8008370 tx=0x7fb0c8001980).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:01.395 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:01.390+0000 7fb0c77fe700 1 -- 172.21.15.42:0/1662054527 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 93 v93) v1 ==== 94+0+2108 (secure 0 0 0) 0x7fb0c0031a50 con 0x7fb0d007b120 2019-12-05T05:23:01.397 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:23:01.397 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":93,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5123,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/1214659805","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1214659805},{"type":"v1","addr":"172.21.15.42:6835","nonce":1214659805}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":93}],"filesystems":[{"mdsmap":{"epoch":92,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:20:47.959813+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5066},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5066":{"gid":5066,"name":"b","rank":0,"incarnation":89,"state":"up:active","state_seq":30,"addr":"172.21.15.40:6833/3061061478","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3061061478},{"type":"v1","addr":"172.21.15.40:6833","nonce":3061061478}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:23:01.397 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 93 2019-12-05T05:23:01.400 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:01.394+0000 7fb0d4a45700 1 -- 172.21.15.42:0/1662054527 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb0b403b200 msgr2=0x7fb0b403d710 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:01.400 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:01.394+0000 7fb0d4a45700 1 --2- 172.21.15.42:0/1662054527 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb0b403b200 0x7fb0b403d710 crc :-1 s=READY pgs=327 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:23:01.400 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:01.394+0000 7fb0d4a45700 1 -- 172.21.15.42:0/1662054527 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb0d007b120 msgr2=0x7fb0d00a5ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:01.400 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:01.394+0000 7fb0d4a45700 1 --2- 172.21.15.42:0/1662054527 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb0d007b120 0x7fb0d00a5ba0 secure :-1 s=READY pgs=585 cs=0 l=1 rx=0x7fb0b8008370 tx=0x7fb0c8001980).stop 2019-12-05T05:23:01.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:01.394+0000 7fb0d4a45700 1 -- 172.21.15.42:0/1662054527 shutdown_connections 2019-12-05T05:23:01.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:01.394+0000 7fb0d4a45700 1 --2- 172.21.15.42:0/1662054527 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb0b403b200 0x7fb0b403d710 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:23:01.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:01.398+0000 7fb0d4a45700 1 --2- 172.21.15.42:0/1662054527 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb0d007b120 0x7fb0d00a5ba0 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:23:01.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:01.398+0000 7fb0d4a45700 1 -- 172.21.15.42:0/1662054527 shutdown_connections 2019-12-05T05:23:01.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:01.398+0000 7fb0d4a45700 1 -- 172.21.15.42:0/1662054527 wait complete. 2019-12-05T05:23:01.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:01.398+0000 7fb0d4a45700 1 -- 172.21.15.42:0/1662054527 >> 172.21.15.42:0/1662054527 conn(0x7fb0d0116dd0 msgr2=0x7fb0d0059c00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:23:01.411 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T05:23:03.412 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:23:03.523 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.514+0000 7f337a39c700 1 Processor -- start 2019-12-05T05:23:03.523 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.514+0000 7f337a39c700 1 -- start start 2019-12-05T05:23:03.523 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.514+0000 7f337a39c700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f337408df70 0x7f337409f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:23:03.524 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.514+0000 7f337a39c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f337409f820 0x7f337414efd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:23:03.524 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.514+0000 7f337a39c700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f337409fc50 con 0x7f33740a7730 2019-12-05T05:23:03.524 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.514+0000 7f337a39c700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33740a0af0 con 0x7f337408df70 2019-12-05T05:23:03.524 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.514+0000 7f337a39c700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f337408e5a0 con 0x7f337409f820 2019-12-05T05:23:03.524 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.514+0000 7f337913a700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f337409f820 0x7f337414efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:23:03.524 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.514+0000 7f3378939700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f33740a7730 0x7f337407e710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:60670/0 (socket says 172.21.15.42:60670) 2019-12-05T05:23:03.524 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.514+0000 7f3378939700 1 -- 172.21.15.42:0/1511397012 learned_addr learned my addr 172.21.15.42:0/1511397012 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:23:03.525 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.514+0000 7f3373fff700 1 --2- 172.21.15.42:0/1511397012 >> v2:172.21.15.40:3300/0 conn(0x7f337408df70 0x7f337409f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:23:03.525 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.518+0000 7f337913a700 1 -- 172.21.15.42:0/1511397012 >> v2:172.21.15.40:3300/0 conn(0x7f337408df70 msgr2=0x7f337409f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:23:03.525 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.518+0000 7f337913a700 1 --2- 172.21.15.42:0/1511397012 >> v2:172.21.15.40:3300/0 conn(0x7f337408df70 0x7f337409f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:23:03.525 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.518+0000 7f337913a700 1 -- 172.21.15.42:0/1511397012 >> v1:172.21.15.42:6789/0 conn(0x7f33740a7730 legacy=0x7f337407e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:03.525 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.518+0000 7f337913a700 1 -- 172.21.15.42:0/1511397012 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33740a0700 con 0x7f337409f820 2019-12-05T05:23:03.526 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.518+0000 7f337913a700 1 --2- 172.21.15.42:0/1511397012 >> v2:172.21.15.42:3300/0 conn(0x7f337409f820 0x7f337414efd0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f336c0032b0 tx=0x7f336c00fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:03.526 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.518+0000 7f337913a700 1 --2- 172.21.15.42:0/1511397012 >> v2:172.21.15.42:3300/0 conn(0x7f337409f820 0x7f337414efd0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f336c0032b0 tx=0x7f336c00fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:03.526 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.518+0000 7f337913a700 1 --2- 172.21.15.42:0/1511397012 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f337409f820 0x7f337414efd0 secure :-1 s=READY pgs=731 cs=0 l=1 rx=0x7f336c0032b0 tx=0x7f336c00fff0).ready entity=mon.1 client_cookie=1f4eb6845426f191 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:23:03.526 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.518+0000 7f337913a700 1 --2- 172.21.15.42:0/1511397012 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f337409f820 0x7f337414efd0 secure :-1 s=THROTTLE_DONE pgs=731 cs=0 l=1 rx=0x7f336c0032b0 tx=0x7f336c00fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:03.526 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.518+0000 7f33737fe700 1 -- 172.21.15.42:0/1511397012 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f336c0108c0 con 0x7f337409f820 2019-12-05T05:23:03.526 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.518+0000 7f337913a700 1 --2- 172.21.15.42:0/1511397012 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f337409f820 0x7f337414efd0 secure :-1 s=THROTTLE_DONE pgs=731 cs=0 l=1 rx=0x7f336c0032b0 tx=0x7f336c00fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:03.527 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.518+0000 7f33737fe700 1 -- 172.21.15.42:0/1511397012 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f336c008b50 con 0x7f337409f820 2019-12-05T05:23:03.527 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.518+0000 7f337913a700 1 --2- 172.21.15.42:0/1511397012 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f337409f820 0x7f337414efd0 secure :-1 s=THROTTLE_DONE pgs=731 cs=0 l=1 rx=0x7f336c0032b0 tx=0x7f336c00fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:03.527 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.518+0000 7f33737fe700 1 -- 172.21.15.42:0/1511397012 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f336c0110c0 con 0x7f337409f820 2019-12-05T05:23:03.527 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.518+0000 7f337a39c700 1 -- 172.21.15.42:0/1511397012 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f337409f820 msgr2=0x7f337414efd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:03.527 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.518+0000 7f337a39c700 1 --2- 172.21.15.42:0/1511397012 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f337409f820 0x7f337414efd0 secure :-1 s=READY pgs=731 cs=0 l=1 rx=0x7f336c0032b0 tx=0x7f336c00fff0).stop 2019-12-05T05:23:03.527 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.518+0000 7f337a39c700 1 -- 172.21.15.42:0/1511397012 shutdown_connections 2019-12-05T05:23:03.527 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.518+0000 7f337a39c700 1 --2- 172.21.15.42:0/1511397012 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f337409f820 0x7f337414efd0 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:23:03.528 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.518+0000 7f337a39c700 1 --2- 172.21.15.42:0/1511397012 >> v2:172.21.15.40:3300/0 conn(0x7f337408df70 0x7f337409f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:23:03.528 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.518+0000 7f337a39c700 1 -- 172.21.15.42:0/1511397012 shutdown_connections 2019-12-05T05:23:03.528 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.518+0000 7f337a39c700 1 -- 172.21.15.42:0/1511397012 wait complete. 2019-12-05T05:23:03.528 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.518+0000 7f337a39c700 1 -- 172.21.15.42:0/1511397012 >> 172.21.15.42:0/1511397012 conn(0x7f33740aabd0 msgr2=0x7f33740a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:23:03.530 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.522+0000 7f337a39c700 1 Processor -- start 2019-12-05T05:23:03.531 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.522+0000 7f337a39c700 1 -- start start 2019-12-05T05:23:03.531 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.522+0000 7f337a39c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f33740a4130 0x7f3374116350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:23:03.531 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.522+0000 7f337a39c700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f3374129f00 0x7f33740a3520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:23:03.531 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.522+0000 7f337a39c700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f33740a3a70 0x7f3374126940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:23:03.531 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.522+0000 7f337a39c700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f337412a480 con 0x7f3374129f00 2019-12-05T05:23:03.531 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.522+0000 7f337a39c700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f337412a720 con 0x7f33740a4130 2019-12-05T05:23:03.531 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.522+0000 7f337a39c700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f337412a9c0 con 0x7f33740a3a70 2019-12-05T05:23:03.532 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.522+0000 7f3378939700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f33740a4130 0x7f3374116350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:23:03.532 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.522+0000 7f3378939700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f33740a4130 0x7f3374116350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:60638/0 (socket says 172.21.15.42:60638) 2019-12-05T05:23:03.532 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.522+0000 7f3378939700 1 -- 172.21.15.42:0/990559439 learned_addr learned my addr 172.21.15.42:0/990559439 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:23:03.532 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.522+0000 7f337913a700 1 --2- 172.21.15.42:0/990559439 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f33740a3a70 0x7f3374126940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:23:03.532 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.522+0000 7f3373fff700 1 --2- 172.21.15.42:0/990559439 >> v2:172.21.15.40:3300/0 conn(0x7f3374129f00 0x7f33740a3520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:23:03.532 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.526+0000 7f3378939700 1 -- 172.21.15.42:0/990559439 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f33740a3a70 msgr2=0x7f3374126940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:03.532 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.526+0000 7f3378939700 1 --2- 172.21.15.42:0/990559439 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f33740a3a70 0x7f3374126940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:23:03.533 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.526+0000 7f3378939700 1 -- 172.21.15.42:0/990559439 >> v2:172.21.15.40:3300/0 conn(0x7f3374129f00 msgr2=0x7f33740a3520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:03.533 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.526+0000 7f3378939700 1 --2- 172.21.15.42:0/990559439 >> v2:172.21.15.40:3300/0 conn(0x7f3374129f00 0x7f33740a3520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:23:03.533 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.526+0000 7f3378939700 1 -- 172.21.15.42:0/990559439 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33740a71a0 con 0x7f33740a4130 2019-12-05T05:23:03.533 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.526+0000 7f3373fff700 1 --2- 172.21.15.42:0/990559439 >> v2:172.21.15.40:3300/0 conn(0x7f3374129f00 0x7f33740a3520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:23:03.533 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.526+0000 7f3378939700 1 --2- 172.21.15.42:0/990559439 >> v2:172.21.15.42:3300/0 conn(0x7f33740a4130 0x7f3374116350 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f337412a8e0 tx=0x7f3364010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:03.533 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.526+0000 7f3378939700 1 --2- 172.21.15.42:0/990559439 >> v2:172.21.15.42:3300/0 conn(0x7f33740a4130 0x7f3374116350 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f337412a8e0 tx=0x7f3364010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:03.533 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.526+0000 7f3378939700 1 --2- 172.21.15.42:0/990559439 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f33740a4130 0x7f3374116350 secure :-1 s=READY pgs=732 cs=0 l=1 rx=0x7f337412a8e0 tx=0x7f3364010ab0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:23:03.534 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.526+0000 7f3378939700 1 --2- 172.21.15.42:0/990559439 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f33740a4130 0x7f3374116350 secure :-1 s=THROTTLE_DONE pgs=732 cs=0 l=1 rx=0x7f337412a8e0 tx=0x7f3364010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:03.534 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.526+0000 7f3378939700 1 --2- 172.21.15.42:0/990559439 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f33740a4130 0x7f3374116350 secure :-1 s=THROTTLE_DONE pgs=732 cs=0 l=1 rx=0x7f337412a8e0 tx=0x7f3364010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:03.534 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.526+0000 7f3378939700 1 --2- 172.21.15.42:0/990559439 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f33740a4130 0x7f3374116350 secure :-1 s=THROTTLE_DONE pgs=732 cs=0 l=1 rx=0x7f337412a8e0 tx=0x7f3364010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:03.534 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.526+0000 7f33717fa700 1 -- 172.21.15.42:0/990559439 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f336400c8b0 con 0x7f33740a4130 2019-12-05T05:23:03.534 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.526+0000 7f33717fa700 1 -- 172.21.15.42:0/990559439 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f336400c4d0 con 0x7f33740a4130 2019-12-05T05:23:03.534 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.526+0000 7f33717fa700 1 -- 172.21.15.42:0/990559439 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f33640124a0 con 0x7f33740a4130 2019-12-05T05:23:03.534 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.526+0000 7f337a39c700 1 -- 172.21.15.42:0/990559439 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f337412ac60 con 0x7f33740a4130 2019-12-05T05:23:03.535 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.526+0000 7f337a39c700 1 -- 172.21.15.42:0/990559439 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33741270b0 con 0x7f33740a4130 2019-12-05T05:23:03.535 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.526+0000 7f3378939700 1 --2- 172.21.15.42:0/990559439 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f33740a4130 0x7f3374116350 secure :-1 s=THROTTLE_DONE pgs=732 cs=0 l=1 rx=0x7f337412a8e0 tx=0x7f3364010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:03.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.526+0000 7f3378939700 1 --2- 172.21.15.42:0/990559439 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f33740a4130 0x7f3374116350 secure :-1 s=THROTTLE_DONE pgs=732 cs=0 l=1 rx=0x7f337412a8e0 tx=0x7f3364010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:03.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.526+0000 7f33717fa700 1 -- 172.21.15.42:0/990559439 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f33640126a0 con 0x7f33740a4130 2019-12-05T05:23:03.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.526+0000 7f33717fa700 1 --2- 172.21.15.42:0/990559439 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f335403ad70 0x7f335403d280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:23:03.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.526+0000 7f33717fa700 1 -- 172.21.15.42:0/990559439 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 7063+0+0 (secure 0 0 0) 0x7f336404ad10 con 0x7f33740a4130 2019-12-05T05:23:03.539 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.526+0000 7f3373fff700 1 --2- 172.21.15.42:0/990559439 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f335403ad70 0x7f335403d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:23:03.539 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.530+0000 7f3373fff700 1 --2- 172.21.15.42:0/990559439 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f335403ad70 0x7f335403d280 crc :-1 s=READY pgs=328 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:23:03.539 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.530+0000 7f337a39c700 1 -- 172.21.15.42:0/990559439 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f337405d100 con 0x7f33740a4130 2019-12-05T05:23:03.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.534+0000 7f3378939700 1 --2- 172.21.15.42:0/990559439 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f33740a4130 0x7f3374116350 secure :-1 s=THROTTLE_DONE pgs=732 cs=0 l=1 rx=0x7f337412a8e0 tx=0x7f3364010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:03.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.534+0000 7f33717fa700 1 -- 172.21.15.42:0/990559439 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f3364031d50 con 0x7f33740a4130 2019-12-05T05:23:03.712 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.710+0000 7f337a39c700 1 -- 172.21.15.42:0/990559439 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f337405b8b0 con 0x7f33740a4130 2019-12-05T05:23:03.712 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.710+0000 7f3378939700 1 --2- 172.21.15.42:0/990559439 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f33740a4130 0x7f3374116350 secure :-1 s=THROTTLE_DONE pgs=732 cs=0 l=1 rx=0x7f337412a8e0 tx=0x7f3364010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:03.713 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.710+0000 7f33717fa700 1 -- 172.21.15.42:0/990559439 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+1771 (secure 0 0 0) 0x7f3364031fb0 con 0x7f33740a4130 2019-12-05T05:23:03.713 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 94 2019-12-05T05:23:03.713 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:23:03.713 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":94,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:23:03.538339+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5123},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5123":{"gid":5123,"name":"a","rank":0,"incarnation":94,"state":"up:replay","state_seq":2,"addr":"172.21.15.42:6835/1214659805","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1214659805},{"type":"v1","addr":"172.21.15.42:6835","nonce":1214659805}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:23:03.715 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.710+0000 7f337a39c700 1 -- 172.21.15.42:0/990559439 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f335403ad70 msgr2=0x7f335403d280 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:03.716 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.710+0000 7f337a39c700 1 --2- 172.21.15.42:0/990559439 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f335403ad70 0x7f335403d280 crc :-1 s=READY pgs=328 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:23:03.716 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.710+0000 7f337a39c700 1 -- 172.21.15.42:0/990559439 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f33740a4130 msgr2=0x7f3374116350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:03.716 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.710+0000 7f337a39c700 1 --2- 172.21.15.42:0/990559439 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f33740a4130 0x7f3374116350 secure :-1 s=READY pgs=732 cs=0 l=1 rx=0x7f337412a8e0 tx=0x7f3364010ab0).stop 2019-12-05T05:23:03.716 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.710+0000 7f337a39c700 1 -- 172.21.15.42:0/990559439 shutdown_connections 2019-12-05T05:23:03.716 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.710+0000 7f337a39c700 1 --2- 172.21.15.42:0/990559439 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f335403ad70 0x7f335403d280 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:23:03.716 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.710+0000 7f337a39c700 1 --2- 172.21.15.42:0/990559439 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f33740a3a70 0x7f3374126940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:23:03.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.710+0000 7f337a39c700 1 --2- 172.21.15.42:0/990559439 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f33740a4130 0x7f3374116350 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:23:03.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.710+0000 7f337a39c700 1 --2- 172.21.15.42:0/990559439 >> v2:172.21.15.40:3300/0 conn(0x7f3374129f00 0x7f33740a3520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:23:03.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.710+0000 7f337a39c700 1 -- 172.21.15.42:0/990559439 shutdown_connections 2019-12-05T05:23:03.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.710+0000 7f337a39c700 1 -- 172.21.15.42:0/990559439 wait complete. 2019-12-05T05:23:03.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.710+0000 7f337a39c700 1 -- 172.21.15.42:0/990559439 >> 172.21.15.42:0/990559439 conn(0x7f33740a98a0 msgr2=0x7f337407cd50 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:23:03.725 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b down, removed from mdsmap 2019-12-05T05:23:03.725 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2019-12-05T05:23:03.725 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:23:03.827 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.798+0000 7feb1ab84700 1 Processor -- start 2019-12-05T05:23:03.827 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.798+0000 7feb1ab84700 1 -- start start 2019-12-05T05:23:03.827 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.798+0000 7feb1ab84700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7feb140a7740 0x7feb1408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:23:03.827 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.798+0000 7feb1ab84700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb140a0710 con 0x7feb1409fae0 2019-12-05T05:23:03.827 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.798+0000 7feb1ab84700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb140a9ba0 con 0x7feb1407e720 2019-12-05T05:23:03.828 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.798+0000 7feb1ab84700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb140a0b00 con 0x7feb140a7740 2019-12-05T05:23:03.828 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.798+0000 7feb18920700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7feb1407e720 0x7feb1409f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:60700/0 (socket says 172.21.15.42:60700) 2019-12-05T05:23:03.828 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.798+0000 7feb18920700 1 -- 172.21.15.42:0/3781411612 learned_addr learned my addr 172.21.15.42:0/3781411612 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:23:03.828 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.798+0000 7feb19121700 1 --2- 172.21.15.42:0/3781411612 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7feb140a7740 0x7feb1408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:23:03.828 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.798+0000 7feb0bfff700 1 -- 172.21.15.42:0/3781411612 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3449415755 0 0) 0x7feb140a9ba0 con 0x7feb1407e720 2019-12-05T05:23:03.828 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.798+0000 7feb0bfff700 1 -- 172.21.15.42:0/3781411612 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb000030b0 con 0x7feb1407e720 2019-12-05T05:23:03.828 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.798+0000 7feb0bfff700 1 -- 172.21.15.42:0/3781411612 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3168243714 0 0) 0x7feb000030b0 con 0x7feb1407e720 2019-12-05T05:23:03.829 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.798+0000 7feb0bfff700 1 -- 172.21.15.42:0/3781411612 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7feb140a7740 msgr2=0x7feb1408df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:23:03.829 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.798+0000 7feb0bfff700 1 --2- 172.21.15.42:0/3781411612 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7feb140a7740 0x7feb1408df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:23:03.829 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.798+0000 7feb0bfff700 1 -- 172.21.15.42:0/3781411612 >> v1:172.21.15.40:6789/0 conn(0x7feb1409fae0 legacy=0x7feb1414efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:03.829 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.798+0000 7feb0bfff700 1 -- 172.21.15.42:0/3781411612 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb1407eb50 con 0x7feb1407e720 2019-12-05T05:23:03.829 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.798+0000 7feb0bfff700 1 -- 172.21.15.42:0/3781411612 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7feb04001750 con 0x7feb1407e720 2019-12-05T05:23:03.829 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.798+0000 7feb0bfff700 1 -- 172.21.15.42:0/3781411612 >> v1:172.21.15.42:6789/0 conn(0x7feb1407e720 legacy=0x7feb1409f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:03.829 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.798+0000 7feb0bfff700 1 --2- 172.21.15.42:0/3781411612 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb00003710 0x7feb00003b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:23:03.830 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.798+0000 7feb0bfff700 1 --2- 172.21.15.42:0/3781411612 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7feb00004420 0x7feb00004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:23:03.830 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.798+0000 7feb0bfff700 1 --2- 172.21.15.42:0/3781411612 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feb00004da0 0x7feb0000b670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:23:03.830 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.798+0000 7feb0bfff700 1 -- 172.21.15.42:0/3781411612 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb140a0b00 con 0x7feb00003710 2019-12-05T05:23:03.830 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.798+0000 7feb0bfff700 1 -- 172.21.15.42:0/3781411612 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb000051d0 con 0x7feb00004da0 2019-12-05T05:23:03.830 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.798+0000 7feb0bfff700 1 -- 172.21.15.42:0/3781411612 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb0004fbd0 con 0x7feb00004420 2019-12-05T05:23:03.830 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.798+0000 7feb19922700 1 --2- 172.21.15.42:0/3781411612 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feb00004da0 0x7feb0000b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:23:03.831 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.798+0000 7feb19121700 1 --2- 172.21.15.42:0/3781411612 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb00003710 0x7feb00003b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:23:03.831 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.798+0000 7feb19922700 1 -- 172.21.15.42:0/3781411612 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7feb00004420 msgr2=0x7feb00004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:23:03.831 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.798+0000 7feb19922700 1 --2- 172.21.15.42:0/3781411612 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7feb00004420 0x7feb00004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:23:03.831 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.798+0000 7feb19922700 1 -- 172.21.15.42:0/3781411612 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb00003710 msgr2=0x7feb00003b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:23:03.831 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.798+0000 7feb19922700 1 --2- 172.21.15.42:0/3781411612 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb00003710 0x7feb00003b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:23:03.831 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.798+0000 7feb19922700 1 -- 172.21.15.42:0/3781411612 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb0004fed0 con 0x7feb00004da0 2019-12-05T05:23:03.831 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.802+0000 7feb19922700 1 --2- 172.21.15.42:0/3781411612 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feb00004da0 0x7feb0000b670 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7feb0c001380 tx=0x7feb0c010020).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:03.832 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.802+0000 7feb19922700 1 --2- 172.21.15.42:0/3781411612 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feb00004da0 0x7feb0000b670 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7feb0c001380 tx=0x7feb0c010020).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:03.832 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.802+0000 7feb19922700 1 --2- 172.21.15.42:0/3781411612 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feb00004da0 0x7feb0000b670 secure :-1 s=READY pgs=734 cs=0 l=1 rx=0x7feb0c001380 tx=0x7feb0c010020).ready entity=mon.1 client_cookie=3befa50af4bb16f5 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:23:03.832 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.802+0000 7feb19922700 1 --2- 172.21.15.42:0/3781411612 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feb00004da0 0x7feb0000b670 secure :-1 s=THROTTLE_DONE pgs=734 cs=0 l=1 rx=0x7feb0c001380 tx=0x7feb0c010020).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:03.832 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.802+0000 7feb0bfff700 1 -- 172.21.15.42:0/3781411612 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7feb0c006ac0 con 0x7feb00004da0 2019-12-05T05:23:03.832 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.802+0000 7feb19922700 1 --2- 172.21.15.42:0/3781411612 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feb00004da0 0x7feb0000b670 secure :-1 s=THROTTLE_DONE pgs=734 cs=0 l=1 rx=0x7feb0c001380 tx=0x7feb0c010020).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:03.832 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.802+0000 7feb19922700 1 --2- 172.21.15.42:0/3781411612 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feb00004da0 0x7feb0000b670 secure :-1 s=THROTTLE_DONE pgs=734 cs=0 l=1 rx=0x7feb0c001380 tx=0x7feb0c010020).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:03.833 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.802+0000 7feb0bfff700 1 -- 172.21.15.42:0/3781411612 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb0c0066e0 con 0x7feb00004da0 2019-12-05T05:23:03.833 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.802+0000 7feb0bfff700 1 -- 172.21.15.42:0/3781411612 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7feb0c002730 con 0x7feb00004da0 2019-12-05T05:23:03.833 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.802+0000 7feb1ab84700 1 -- 172.21.15.42:0/3781411612 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feb00004da0 msgr2=0x7feb0000b670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:03.833 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.802+0000 7feb1ab84700 1 --2- 172.21.15.42:0/3781411612 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feb00004da0 0x7feb0000b670 secure :-1 s=READY pgs=734 cs=0 l=1 rx=0x7feb0c001380 tx=0x7feb0c010020).stop 2019-12-05T05:23:03.834 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.802+0000 7feb19922700 1 -- 172.21.15.42:0/3781411612 reap_dead start 2019-12-05T05:23:03.834 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.802+0000 7feb1ab84700 1 -- 172.21.15.42:0/3781411612 shutdown_connections 2019-12-05T05:23:03.834 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.802+0000 7feb1ab84700 1 -- 172.21.15.42:0/3781411612 shutdown_connections 2019-12-05T05:23:03.834 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.802+0000 7feb1ab84700 1 -- 172.21.15.42:0/3781411612 wait complete. 2019-12-05T05:23:03.834 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.802+0000 7feb1ab84700 1 -- 172.21.15.42:0/3781411612 >> 172.21.15.42:0/3781411612 conn(0x7feb140aabe0 msgr2=0x7feb140a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:23:03.834 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.802+0000 7feb1ab84700 1 Processor -- start 2019-12-05T05:23:03.834 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.802+0000 7feb1ab84700 1 -- start start 2019-12-05T05:23:03.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.802+0000 7feb1ab84700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7feb14116b40 0x7feb1411b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:23:03.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.802+0000 7feb1ab84700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb14117ac0 con 0x7feb14117690 2019-12-05T05:23:03.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.802+0000 7feb1ab84700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb1411cc00 con 0x7feb1411a500 2019-12-05T05:23:03.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.802+0000 7feb1ab84700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb1411bec0 con 0x7feb14116b40 2019-12-05T05:23:03.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.802+0000 7feb19121700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7feb1411a500 0x7feb14116340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:60710/0 (socket says 172.21.15.42:60710) 2019-12-05T05:23:03.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.802+0000 7feb19121700 1 -- 172.21.15.42:0/366111627 learned_addr learned my addr 172.21.15.42:0/366111627 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:23:03.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.802+0000 7feb18920700 1 --2- 172.21.15.42:0/366111627 >> v2:172.21.15.40:3300/0 conn(0x7feb14116b40 0x7feb1411b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:23:03.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.802+0000 7feb09ffb700 1 -- 172.21.15.42:0/366111627 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3455801233 0 0) 0x7feb1411cc00 con 0x7feb1411a500 2019-12-05T05:23:03.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.802+0000 7feb09ffb700 1 -- 172.21.15.42:0/366111627 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feaf40030b0 con 0x7feb1411a500 2019-12-05T05:23:03.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.802+0000 7feb09ffb700 1 -- 172.21.15.42:0/366111627 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 2963735474 0 0) 0x7feaf40030b0 con 0x7feb1411a500 2019-12-05T05:23:03.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.802+0000 7feb09ffb700 1 -- 172.21.15.42:0/366111627 --> v1:172.21.15.42:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feb1411cc00 con 0x7feb1411a500 2019-12-05T05:23:03.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.802+0000 7feb09ffb700 1 -- 172.21.15.42:0/366111627 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7feb10005b40 con 0x7feb1411a500 2019-12-05T05:23:03.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.802+0000 7feb09ffb700 1 -- 172.21.15.42:0/366111627 >> v2:172.21.15.40:3300/0 conn(0x7feb14116b40 msgr2=0x7feb1411b8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:03.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.802+0000 7feb09ffb700 1 --2- 172.21.15.42:0/366111627 >> v2:172.21.15.40:3300/0 conn(0x7feb14116b40 0x7feb1411b8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:23:03.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.802+0000 7feb09ffb700 1 -- 172.21.15.42:0/366111627 >> v1:172.21.15.42:6789/0 conn(0x7feb1411a500 legacy=0x7feb14116340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:03.837 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.802+0000 7feb09ffb700 1 -- 172.21.15.42:0/366111627 >> v1:172.21.15.40:6789/0 conn(0x7feb14117690 legacy=0x7feb1411c400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:03.837 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.802+0000 7feb09ffb700 1 --2- 172.21.15.42:0/366111627 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7feaf40038a0 0x7feaf4003cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:23:03.837 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.802+0000 7feb09ffb700 1 --2- 172.21.15.42:0/366111627 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feaf4004420 0x7feaf4004850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:23:03.837 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.802+0000 7feb09ffb700 1 --2- 172.21.15.42:0/366111627 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feaf4004ec0 0x7feaf400b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:23:03.837 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.802+0000 7feb09ffb700 1 -- 172.21.15.42:0/366111627 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb1411bec0 con 0x7feaf4004ec0 2019-12-05T05:23:03.837 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.802+0000 7feb09ffb700 1 -- 172.21.15.42:0/366111627 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feaf400bcc0 con 0x7feaf4004420 2019-12-05T05:23:03.837 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.802+0000 7feb09ffb700 1 -- 172.21.15.42:0/366111627 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feaf400bf00 con 0x7feaf40038a0 2019-12-05T05:23:03.838 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.802+0000 7feb18920700 1 --2- 172.21.15.42:0/366111627 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feaf4004420 0x7feaf4004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:23:03.838 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.806+0000 7feb19121700 1 --2- 172.21.15.42:0/366111627 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7feaf40038a0 0x7feaf4003cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:23:03.838 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.806+0000 7feb18920700 1 -- 172.21.15.42:0/366111627 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7feaf40038a0 msgr2=0x7feaf4003cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:03.838 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.806+0000 7feb18920700 1 --2- 172.21.15.42:0/366111627 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7feaf40038a0 0x7feaf4003cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:23:03.838 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.806+0000 7feb18920700 1 -- 172.21.15.42:0/366111627 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feaf4004ec0 msgr2=0x7feaf400b670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:03.838 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.806+0000 7feb18920700 1 --2- 172.21.15.42:0/366111627 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feaf4004ec0 0x7feaf400b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:23:03.838 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.806+0000 7feb19922700 1 -- 172.21.15.42:0/366111627 reap_dead start 2019-12-05T05:23:03.839 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.806+0000 7feb18920700 1 -- 172.21.15.42:0/366111627 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb1411ce30 con 0x7feaf4004420 2019-12-05T05:23:03.839 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.806+0000 7feb1ab84700 1 -- 172.21.15.42:0/366111627 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb14117690 con 0x7feaf4004420 2019-12-05T05:23:03.839 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.806+0000 7feb18920700 1 --2- 172.21.15.42:0/366111627 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feaf4004420 0x7feaf4004850 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7feb04001460 tx=0x7feb04001bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:03.839 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.806+0000 7feb1ab84700 1 -- 172.21.15.42:0/366111627 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb141178a0 con 0x7feaf4004420 2019-12-05T05:23:03.839 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.806+0000 7feb18920700 1 --2- 172.21.15.42:0/366111627 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feaf4004420 0x7feaf4004850 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7feb04001460 tx=0x7feb04001bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:03.840 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.806+0000 7feb18920700 1 --2- 172.21.15.42:0/366111627 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feaf4004420 0x7feaf4004850 secure :-1 s=READY pgs=736 cs=0 l=1 rx=0x7feb04001460 tx=0x7feb04001bb0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:23:03.840 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.806+0000 7feb18920700 1 --2- 172.21.15.42:0/366111627 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feaf4004420 0x7feaf4004850 secure :-1 s=THROTTLE_DONE pgs=736 cs=0 l=1 rx=0x7feb04001460 tx=0x7feb04001bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:03.840 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.806+0000 7feb18920700 1 --2- 172.21.15.42:0/366111627 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feaf4004420 0x7feaf4004850 secure :-1 s=THROTTLE_DONE pgs=736 cs=0 l=1 rx=0x7feb04001460 tx=0x7feb04001bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:03.840 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.806+0000 7feb09ffb700 1 -- 172.21.15.42:0/366111627 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7feb0400ca70 con 0x7feaf4004420 2019-12-05T05:23:03.840 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.806+0000 7feb18920700 1 --2- 172.21.15.42:0/366111627 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feaf4004420 0x7feaf4004850 secure :-1 s=THROTTLE_DONE pgs=736 cs=0 l=1 rx=0x7feb04001460 tx=0x7feb04001bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:03.840 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.806+0000 7feb09ffb700 1 -- 172.21.15.42:0/366111627 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb0400c690 con 0x7feaf4004420 2019-12-05T05:23:03.841 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.806+0000 7feb09ffb700 1 -- 172.21.15.42:0/366111627 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7feb040045f0 con 0x7feaf4004420 2019-12-05T05:23:03.841 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.806+0000 7feb18920700 1 --2- 172.21.15.42:0/366111627 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feaf4004420 0x7feaf4004850 secure :-1 s=THROTTLE_DONE pgs=736 cs=0 l=1 rx=0x7feb04001460 tx=0x7feb04001bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:03.841 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.806+0000 7feb09ffb700 1 -- 172.21.15.42:0/366111627 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7feb040047f0 con 0x7feaf4004420 2019-12-05T05:23:03.841 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.806+0000 7feb18920700 1 --2- 172.21.15.42:0/366111627 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feaf4004420 0x7feaf4004850 secure :-1 s=THROTTLE_DONE pgs=736 cs=0 l=1 rx=0x7feb04001460 tx=0x7feb04001bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:03.841 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.806+0000 7feb09ffb700 1 --2- 172.21.15.42:0/366111627 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7feaf403e9b0 0x7feaf4040ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:23:03.841 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.806+0000 7feb09ffb700 1 -- 172.21.15.42:0/366111627 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 7149+0+0 (secure 0 0 0) 0x7feb0404ac20 con 0x7feaf4004420 2019-12-05T05:23:03.841 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.806+0000 7feb19121700 1 --2- 172.21.15.42:0/366111627 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7feaf403e9b0 0x7feaf4040ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:23:03.842 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.806+0000 7feb19121700 1 --2- 172.21.15.42:0/366111627 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7feaf403e9b0 0x7feaf4040ec0 crc :-1 s=READY pgs=329 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:23:03.842 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.814+0000 7feb1ab84700 1 -- 172.21.15.42:0/366111627 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feaec003200 con 0x7feaf4004420 2019-12-05T05:23:03.842 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.822+0000 7feb18920700 1 --2- 172.21.15.42:0/366111627 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feaf4004420 0x7feaf4004850 secure :-1 s=THROTTLE_DONE pgs=736 cs=0 l=1 rx=0x7feb04001460 tx=0x7feb04001bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:03.842 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.822+0000 7feb09ffb700 1 -- 172.21.15.42:0/366111627 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7feb040120c0 con 0x7feaf4004420 2019-12-05T05:23:03.985 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.982+0000 7feb1ab84700 1 -- 172.21.15.42:0/366111627 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7feaec003630 con 0x7feaf4004420 2019-12-05T05:23:03.986 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.982+0000 7feb18920700 1 --2- 172.21.15.42:0/366111627 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feaf4004420 0x7feaf4004850 secure :-1 s=THROTTLE_DONE pgs=736 cs=0 l=1 rx=0x7feb04001460 tx=0x7feb04001bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:03.986 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.982+0000 7feb09ffb700 1 -- 172.21.15.42:0/366111627 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+1771 (secure 0 0 0) 0x7feb141178a0 con 0x7feaf4004420 2019-12-05T05:23:03.987 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 94 2019-12-05T05:23:03.987 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:23:03.987 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":94,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:23:03.538339+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5123},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5123":{"gid":5123,"name":"a","rank":0,"incarnation":94,"state":"up:replay","state_seq":2,"addr":"172.21.15.42:6835/1214659805","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1214659805},{"type":"v1","addr":"172.21.15.42:6835","nonce":1214659805}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:23:03.990 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.986+0000 7feb1ab84700 1 -- 172.21.15.42:0/366111627 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7feaf403e9b0 msgr2=0x7feaf4040ec0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:03.990 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.986+0000 7feb1ab84700 1 --2- 172.21.15.42:0/366111627 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7feaf403e9b0 0x7feaf4040ec0 crc :-1 s=READY pgs=329 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:23:03.990 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.986+0000 7feb1ab84700 1 -- 172.21.15.42:0/366111627 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feaf4004420 msgr2=0x7feaf4004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:03.991 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.986+0000 7feb1ab84700 1 --2- 172.21.15.42:0/366111627 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feaf4004420 0x7feaf4004850 secure :-1 s=READY pgs=736 cs=0 l=1 rx=0x7feb04001460 tx=0x7feb04001bb0).stop 2019-12-05T05:23:03.991 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.986+0000 7feb1ab84700 1 -- 172.21.15.42:0/366111627 shutdown_connections 2019-12-05T05:23:03.991 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.986+0000 7feb1ab84700 1 --2- 172.21.15.42:0/366111627 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7feaf403e9b0 0x7feaf4040ec0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:23:03.991 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.986+0000 7feb1ab84700 1 --2- 172.21.15.42:0/366111627 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feaf4004420 0x7feaf4004850 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:23:03.991 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.986+0000 7feb1ab84700 1 -- 172.21.15.42:0/366111627 shutdown_connections 2019-12-05T05:23:03.991 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.986+0000 7feb1ab84700 1 -- 172.21.15.42:0/366111627 wait complete. 2019-12-05T05:23:03.991 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:03.986+0000 7feb1ab84700 1 -- 172.21.15.42:0/366111627 >> 172.21.15.42:0/366111627 conn(0x7feb140a7740 msgr2=0x7feb1407db30 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:23:03.999 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 94, "feature_flags": { "enable_multiple": false, "ever_enabled_multiple": false }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2019-12-05T05:03:35.604738+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 94, "ever_allowed_features": 2, "explicitly_allowed_features": 2, "failed": [], "flags": 18, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5123": { "addr": "172.21.15.42:6835/1214659805", "addrs": { "addrvec": [ { "addr": "172.21.15.42:6834", "nonce": 1214659805, "type": "v2" }, { "addr": "172.21.15.42:6835", "nonce": 1214659805, "type": "v1" } ] }, "export_targets": [], "features": 4540138292836696063, "flags": 0, "gid": 5123, "incarnation": 94, "name": "a", "rank": 0, "state": "up:replay", "state_seq": 2 } }, "last_failure": 0, "last_failure_osd_epoch": 62, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 2, "min_compat_client": "0 (unknown)", "modified": "2019-12-05T05:23:03.538339+0000", "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5123 } } } ], "standbys": [] } 2019-12-05T05:23:05.999 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:23:06.115 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:06.090+0000 7f86af0c7700 1 Processor -- start 2019-12-05T05:23:06.115 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:06.090+0000 7f86af0c7700 1 -- start start 2019-12-05T05:23:06.115 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:06.090+0000 7f86af0c7700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f86a809fae0 0x7f86a814fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:23:06.115 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:06.090+0000 7f86af0c7700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86a80a0710 con 0x7f86a80a7740 2019-12-05T05:23:06.115 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:06.090+0000 7f86af0c7700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86a807ef20 con 0x7f86a808df80 2019-12-05T05:23:06.116 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:06.090+0000 7f86af0c7700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86a80a0b00 con 0x7f86a809fae0 2019-12-05T05:23:06.116 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:06.090+0000 7f86ade65700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f86a809fae0 0x7f86a814fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:23:06.116 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:06.090+0000 7f86ace63700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f86a808df80 0x7f86a809f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:60728/0 (socket says 172.21.15.42:60728) 2019-12-05T05:23:06.116 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:06.090+0000 7f86ace63700 1 -- 172.21.15.42:0/3543789045 learned_addr learned my addr 172.21.15.42:0/3543789045 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:23:06.118 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:06.090+0000 7f86ade65700 1 -- 172.21.15.42:0/3543789045 >> v1:172.21.15.42:6789/0 conn(0x7f86a808df80 legacy=0x7f86a809f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:06.118 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:06.090+0000 7f86ade65700 1 -- 172.21.15.42:0/3543789045 >> v1:172.21.15.40:6789/0 conn(0x7f86a80a7740 legacy=0x7f86a807e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:06.118 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:06.090+0000 7f86ade65700 1 -- 172.21.15.42:0/3543789045 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86a80a9ba0 con 0x7f86a809fae0 2019-12-05T05:23:06.118 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:06.090+0000 7f86ade65700 1 --2- 172.21.15.42:0/3543789045 >> v2:172.21.15.42:3300/0 conn(0x7f86a809fae0 0x7f86a814fff0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f86a0008b10 tx=0x7f86a000ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:06.118 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:06.090+0000 7f86ade65700 1 --2- 172.21.15.42:0/3543789045 >> v2:172.21.15.42:3300/0 conn(0x7f86a809fae0 0x7f86a814fff0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f86a0008b10 tx=0x7f86a000ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:06.119 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:06.090+0000 7f86ade65700 1 --2- 172.21.15.42:0/3543789045 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f86a809fae0 0x7f86a814fff0 secure :-1 s=READY pgs=738 cs=0 l=1 rx=0x7f86a0008b10 tx=0x7f86a000ff70).ready entity=mon.1 client_cookie=e46148cfcc64f6e0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:23:06.119 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:06.094+0000 7f86ade65700 1 --2- 172.21.15.42:0/3543789045 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f86a809fae0 0x7f86a814fff0 secure :-1 s=THROTTLE_DONE pgs=738 cs=0 l=1 rx=0x7f86a0008b10 tx=0x7f86a000ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:06.119 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:06.094+0000 7f869bfff700 1 -- 172.21.15.42:0/3543789045 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f86a0010970 con 0x7f86a809fae0 2019-12-05T05:23:06.119 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:06.094+0000 7f86ade65700 1 --2- 172.21.15.42:0/3543789045 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f86a809fae0 0x7f86a814fff0 secure :-1 s=THROTTLE_DONE pgs=738 cs=0 l=1 rx=0x7f86a0008b10 tx=0x7f86a000ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:06.119 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:06.094+0000 7f869bfff700 1 -- 172.21.15.42:0/3543789045 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86a0008b50 con 0x7f86a809fae0 2019-12-05T05:23:06.119 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:06.094+0000 7f86ade65700 1 --2- 172.21.15.42:0/3543789045 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f86a809fae0 0x7f86a814fff0 secure :-1 s=THROTTLE_DONE pgs=738 cs=0 l=1 rx=0x7f86a0008b10 tx=0x7f86a000ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:06.119 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:06.094+0000 7f86af0c7700 1 -- 172.21.15.42:0/3543789045 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f86a809fae0 msgr2=0x7f86a814fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:06.120 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:06.094+0000 7f869bfff700 1 -- 172.21.15.42:0/3543789045 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f86a00110c0 con 0x7f86a809fae0 2019-12-05T05:23:06.120 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:06.094+0000 7f86af0c7700 1 --2- 172.21.15.42:0/3543789045 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f86a809fae0 0x7f86a814fff0 secure :-1 s=READY pgs=738 cs=0 l=1 rx=0x7f86a0008b10 tx=0x7f86a000ff70).stop 2019-12-05T05:23:06.120 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:06.094+0000 7f86af0c7700 1 -- 172.21.15.42:0/3543789045 shutdown_connections 2019-12-05T05:23:06.120 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:06.094+0000 7f86af0c7700 1 --2- 172.21.15.42:0/3543789045 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f86a809fae0 0x7f86a814fff0 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:23:06.121 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:06.094+0000 7f86af0c7700 1 -- 172.21.15.42:0/3543789045 shutdown_connections 2019-12-05T05:23:06.121 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:06.094+0000 7f86af0c7700 1 -- 172.21.15.42:0/3543789045 wait complete. 2019-12-05T05:23:06.121 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:06.094+0000 7f86af0c7700 1 -- 172.21.15.42:0/3543789045 >> 172.21.15.42:0/3543789045 conn(0x7f86a80aabe0 msgr2=0x7f86a80a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:23:06.121 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:06.094+0000 7f86af0c7700 1 Processor -- start 2019-12-05T05:23:06.121 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:06.094+0000 7f86af0c7700 1 -- start start 2019-12-05T05:23:06.122 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:06.094+0000 7f86af0c7700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f86a8120410 0x7f86a8120b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:23:06.122 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:06.094+0000 7f86af0c7700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86a80a6ce0 con 0x7f86a81210d0 2019-12-05T05:23:06.122 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:06.094+0000 7f86af0c7700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86a8124680 con 0x7f86a807cca0 2019-12-05T05:23:06.122 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:06.094+0000 7f86af0c7700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86a81208e0 con 0x7f86a8120410 2019-12-05T05:23:06.122 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:06.094+0000 7f86ad664700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f86a8120410 0x7f86a8120b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:23:06.122 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:06.098+0000 7f86ad664700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f86a8120410 0x7f86a8120b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:60696/0 (socket says 172.21.15.42:60696) 2019-12-05T05:23:06.122 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:06.098+0000 7f86ad664700 1 -- 172.21.15.42:0/178139333 learned_addr learned my addr 172.21.15.42:0/178139333 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:23:06.123 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:06.098+0000 7f86ad664700 1 -- 172.21.15.42:0/178139333 >> v1:172.21.15.42:6789/0 conn(0x7f86a807cca0 legacy=0x7f86a80a64e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:06.123 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:06.098+0000 7f86ad664700 1 -- 172.21.15.42:0/178139333 >> v1:172.21.15.40:6789/0 conn(0x7f86a81210d0 legacy=0x7f86a8123e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:06.123 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:06.098+0000 7f86ad664700 1 -- 172.21.15.42:0/178139333 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86a8143390 con 0x7f86a8120410 2019-12-05T05:23:06.123 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:06.098+0000 7f86ad664700 1 --2- 172.21.15.42:0/178139333 >> v2:172.21.15.42:3300/0 conn(0x7f86a8120410 0x7f86a8120b80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f869c000e90 tx=0x7f869c0108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:06.123 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:06.098+0000 7f86ad664700 1 --2- 172.21.15.42:0/178139333 >> v2:172.21.15.42:3300/0 conn(0x7f86a8120410 0x7f86a8120b80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f869c000e90 tx=0x7f869c0108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:06.123 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:06.098+0000 7f86ad664700 1 --2- 172.21.15.42:0/178139333 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f86a8120410 0x7f86a8120b80 secure :-1 s=READY pgs=740 cs=0 l=1 rx=0x7f869c000e90 tx=0x7f869c0108f0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:23:06.123 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:06.098+0000 7f86ad664700 1 --2- 172.21.15.42:0/178139333 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f86a8120410 0x7f86a8120b80 secure :-1 s=THROTTLE_DONE pgs=740 cs=0 l=1 rx=0x7f869c000e90 tx=0x7f869c0108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:06.124 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:06.098+0000 7f8699ffb700 1 -- 172.21.15.42:0/178139333 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f869c00caa0 con 0x7f86a8120410 2019-12-05T05:23:06.124 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:06.098+0000 7f86ad664700 1 --2- 172.21.15.42:0/178139333 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f86a8120410 0x7f86a8120b80 secure :-1 s=THROTTLE_DONE pgs=740 cs=0 l=1 rx=0x7f869c000e90 tx=0x7f869c0108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:06.124 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:06.098+0000 7f86ad664700 1 --2- 172.21.15.42:0/178139333 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f86a8120410 0x7f86a8120b80 secure :-1 s=THROTTLE_DONE pgs=740 cs=0 l=1 rx=0x7f869c000e90 tx=0x7f869c0108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:06.124 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:06.098+0000 7f8699ffb700 1 -- 172.21.15.42:0/178139333 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f869c00c6c0 con 0x7f86a8120410 2019-12-05T05:23:06.124 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:06.098+0000 7f8699ffb700 1 -- 172.21.15.42:0/178139333 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f869c011130 con 0x7f86a8120410 2019-12-05T05:23:06.124 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:06.098+0000 7f86af0c7700 1 -- 172.21.15.42:0/178139333 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86a812be30 con 0x7f86a8120410 2019-12-05T05:23:06.125 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:06.098+0000 7f86af0c7700 1 -- 172.21.15.42:0/178139333 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86a812c330 con 0x7f86a8120410 2019-12-05T05:23:06.125 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:06.098+0000 7f86ad664700 1 --2- 172.21.15.42:0/178139333 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f86a8120410 0x7f86a8120b80 secure :-1 s=THROTTLE_DONE pgs=740 cs=0 l=1 rx=0x7f869c000e90 tx=0x7f869c0108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:06.125 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:06.098+0000 7f8699ffb700 1 -- 172.21.15.42:0/178139333 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f869c011440 con 0x7f86a8120410 2019-12-05T05:23:06.125 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:06.098+0000 7f86ad664700 1 --2- 172.21.15.42:0/178139333 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f86a8120410 0x7f86a8120b80 secure :-1 s=THROTTLE_DONE pgs=740 cs=0 l=1 rx=0x7f869c000e90 tx=0x7f869c0108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:06.125 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:06.098+0000 7f8699ffb700 1 --2- 172.21.15.42:0/178139333 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f868803ad90 0x7f868803d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:23:06.125 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:06.098+0000 7f8699ffb700 1 -- 172.21.15.42:0/178139333 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 7149+0+0 (secure 0 0 0) 0x7f869c04b1b0 con 0x7f86a8120410 2019-12-05T05:23:06.125 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:06.098+0000 7f86ace63700 1 --2- 172.21.15.42:0/178139333 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f868803ad90 0x7f868803d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:23:06.125 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:06.098+0000 7f86ace63700 1 --2- 172.21.15.42:0/178139333 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f868803ad90 0x7f868803d2a0 crc :-1 s=READY pgs=330 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:23:06.126 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:06.106+0000 7f86af0c7700 1 -- 172.21.15.42:0/178139333 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86a805d110 con 0x7f86a8120410 2019-12-05T05:23:06.126 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:06.114+0000 7f86ad664700 1 --2- 172.21.15.42:0/178139333 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f86a8120410 0x7f86a8120b80 secure :-1 s=THROTTLE_DONE pgs=740 cs=0 l=1 rx=0x7f869c000e90 tx=0x7f869c0108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:06.127 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:06.114+0000 7f8699ffb700 1 -- 172.21.15.42:0/178139333 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f869c032230 con 0x7f86a8120410 2019-12-05T05:23:06.281 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:06.278+0000 7f86af0c7700 1 -- 172.21.15.42:0/178139333 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f86a805b8c0 con 0x7f86a8120410 2019-12-05T05:23:06.281 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:06.278+0000 7f86ad664700 1 --2- 172.21.15.42:0/178139333 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f86a8120410 0x7f86a8120b80 secure :-1 s=THROTTLE_DONE pgs=740 cs=0 l=1 rx=0x7f869c000e90 tx=0x7f869c0108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:06.282 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:06.278+0000 7f8699ffb700 1 -- 172.21.15.42:0/178139333 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 95 v95) v1 ==== 94+0+1775 (secure 0 0 0) 0x7f869c032490 con 0x7f86a8120410 2019-12-05T05:23:06.284 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 95 2019-12-05T05:23:06.284 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:23:06.284 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":95,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":95,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:23:05.976778+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5123},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5123":{"gid":5123,"name":"a","rank":0,"incarnation":94,"state":"up:reconnect","state_seq":35,"addr":"172.21.15.42:6835/1214659805","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1214659805},{"type":"v1","addr":"172.21.15.42:6835","nonce":1214659805}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:23:06.286 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:06.282+0000 7f86af0c7700 1 -- 172.21.15.42:0/178139333 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f868803ad90 msgr2=0x7f868803d2a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:06.286 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:06.282+0000 7f86af0c7700 1 --2- 172.21.15.42:0/178139333 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f868803ad90 0x7f868803d2a0 crc :-1 s=READY pgs=330 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:23:06.287 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:06.282+0000 7f86af0c7700 1 -- 172.21.15.42:0/178139333 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f86a8120410 msgr2=0x7f86a8120b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:06.287 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:06.282+0000 7f86af0c7700 1 --2- 172.21.15.42:0/178139333 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f86a8120410 0x7f86a8120b80 secure :-1 s=READY pgs=740 cs=0 l=1 rx=0x7f869c000e90 tx=0x7f869c0108f0).stop 2019-12-05T05:23:06.287 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:06.282+0000 7f86af0c7700 1 -- 172.21.15.42:0/178139333 shutdown_connections 2019-12-05T05:23:06.287 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:06.282+0000 7f86af0c7700 1 --2- 172.21.15.42:0/178139333 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f868803ad90 0x7f868803d2a0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:23:06.287 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:06.282+0000 7f86af0c7700 1 --2- 172.21.15.42:0/178139333 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f86a8120410 0x7f86a8120b80 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:23:06.287 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:06.282+0000 7f86af0c7700 1 -- 172.21.15.42:0/178139333 shutdown_connections 2019-12-05T05:23:06.288 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:06.282+0000 7f86af0c7700 1 -- 172.21.15.42:0/178139333 wait complete. 2019-12-05T05:23:06.295 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:06.282+0000 7f86af0c7700 1 -- 172.21.15.42:0/178139333 >> 172.21.15.42:0/178139333 conn(0x7f86a807e910 msgr2=0x7f86a81162d0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:23:06.296 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2019-12-05T05:23:08.105 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:23:08.111 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:23:08.297 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:23:08.399 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:08.390+0000 7f43f182e700 1 Processor -- start 2019-12-05T05:23:08.399 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:08.390+0000 7f43f182e700 1 -- start start 2019-12-05T05:23:08.399 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:08.390+0000 7f43f182e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f43ec0a7740 0x7f43ec08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:23:08.399 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:08.390+0000 7f43f182e700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f43ec09fae0 0x7f43ec07eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:23:08.400 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:08.390+0000 7f43f182e700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43ec0a0710 con 0x7f43ec07e720 2019-12-05T05:23:08.400 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:08.390+0000 7f43f182e700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43ec0a0b00 con 0x7f43ec0a7740 2019-12-05T05:23:08.400 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:08.390+0000 7f43f182e700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43ec08e5b0 con 0x7f43ec09fae0 2019-12-05T05:23:08.400 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:08.394+0000 7f43eb7fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f43ec0a7740 0x7f43ec08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:23:08.400 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:08.394+0000 7f43eb7fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f43ec0a7740 0x7f43ec08df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:60710/0 (socket says 172.21.15.42:60710) 2019-12-05T05:23:08.400 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:08.394+0000 7f43eb7fe700 1 -- 172.21.15.42:0/2306734394 learned_addr learned my addr 172.21.15.42:0/2306734394 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:23:08.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:08.394+0000 7f43ebfff700 1 --2- 172.21.15.42:0/2306734394 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f43ec09fae0 0x7f43ec07eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:23:08.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:08.394+0000 7f43eb7fe700 1 -- 172.21.15.42:0/2306734394 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f43ec09fae0 msgr2=0x7f43ec07eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:23:08.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:08.394+0000 7f43eb7fe700 1 --2- 172.21.15.42:0/2306734394 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f43ec09fae0 0x7f43ec07eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:23:08.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:08.394+0000 7f43eb7fe700 1 -- 172.21.15.42:0/2306734394 >> v1:172.21.15.40:6789/0 conn(0x7f43ec07e720 legacy=0x7f43ec09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:08.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:08.394+0000 7f43eb7fe700 1 -- 172.21.15.42:0/2306734394 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43ec07f0a0 con 0x7f43ec0a7740 2019-12-05T05:23:08.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:08.394+0000 7f43ebfff700 1 --2- 172.21.15.42:0/2306734394 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f43ec09fae0 0x7f43ec07eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T05:23:08.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:08.394+0000 7f43eb7fe700 1 --2- 172.21.15.42:0/2306734394 >> v2:172.21.15.42:3300/0 conn(0x7f43ec0a7740 0x7f43ec08df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f43dc008790 tx=0x7f43dc00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:08.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:08.394+0000 7f43eb7fe700 1 --2- 172.21.15.42:0/2306734394 >> v2:172.21.15.42:3300/0 conn(0x7f43ec0a7740 0x7f43ec08df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f43dc008790 tx=0x7f43dc00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:08.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:08.394+0000 7f43eb7fe700 1 --2- 172.21.15.42:0/2306734394 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f43ec0a7740 0x7f43ec08df80 secure :-1 s=READY pgs=741 cs=0 l=1 rx=0x7f43dc008790 tx=0x7f43dc00fd30).ready entity=mon.1 client_cookie=ea977c665475ea40 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:23:08.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:08.394+0000 7f43eb7fe700 1 --2- 172.21.15.42:0/2306734394 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f43ec0a7740 0x7f43ec08df80 secure :-1 s=THROTTLE_DONE pgs=741 cs=0 l=1 rx=0x7f43dc008790 tx=0x7f43dc00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:08.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:08.394+0000 7f43eb7fe700 1 --2- 172.21.15.42:0/2306734394 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f43ec0a7740 0x7f43ec08df80 secure :-1 s=THROTTLE_DONE pgs=741 cs=0 l=1 rx=0x7f43dc008790 tx=0x7f43dc00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:08.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:08.394+0000 7f43ea7fc700 1 -- 172.21.15.42:0/2306734394 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f43dc008c90 con 0x7f43ec0a7740 2019-12-05T05:23:08.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:08.394+0000 7f43eb7fe700 1 --2- 172.21.15.42:0/2306734394 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f43ec0a7740 0x7f43ec08df80 secure :-1 s=THROTTLE_DONE pgs=741 cs=0 l=1 rx=0x7f43dc008790 tx=0x7f43dc00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:08.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:08.394+0000 7f43ea7fc700 1 -- 172.21.15.42:0/2306734394 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43dc0088b0 con 0x7f43ec0a7740 2019-12-05T05:23:08.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:08.394+0000 7f43ea7fc700 1 -- 172.21.15.42:0/2306734394 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f43dc011b10 con 0x7f43ec0a7740 2019-12-05T05:23:08.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:08.394+0000 7f43f182e700 1 -- 172.21.15.42:0/2306734394 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f43ec0a7740 msgr2=0x7f43ec08df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:08.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:08.394+0000 7f43f182e700 1 --2- 172.21.15.42:0/2306734394 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f43ec0a7740 0x7f43ec08df80 secure :-1 s=READY pgs=741 cs=0 l=1 rx=0x7f43dc008790 tx=0x7f43dc00fd30).stop 2019-12-05T05:23:08.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:08.394+0000 7f43f182e700 1 -- 172.21.15.42:0/2306734394 shutdown_connections 2019-12-05T05:23:08.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:08.394+0000 7f43f182e700 1 --2- 172.21.15.42:0/2306734394 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f43ec09fae0 0x7f43ec07eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:23:08.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:08.394+0000 7f43f182e700 1 --2- 172.21.15.42:0/2306734394 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f43ec0a7740 0x7f43ec08df80 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:23:08.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:08.394+0000 7f43f182e700 1 -- 172.21.15.42:0/2306734394 shutdown_connections 2019-12-05T05:23:08.404 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:08.394+0000 7f43f182e700 1 -- 172.21.15.42:0/2306734394 wait complete. 2019-12-05T05:23:08.404 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:08.394+0000 7f43f182e700 1 -- 172.21.15.42:0/2306734394 >> 172.21.15.42:0/2306734394 conn(0x7f43ec0aabe0 msgr2=0x7f43ec0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:23:08.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:08.398+0000 7f43f182e700 1 Processor -- start 2019-12-05T05:23:08.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:08.398+0000 7f43f182e700 1 -- start start 2019-12-05T05:23:08.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:08.398+0000 7f43f182e700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f43ec079350 0x7f43ec079780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:23:08.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:08.398+0000 7f43f182e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f43ec128200 0x7f43ec128630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:23:08.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:08.398+0000 7f43f182e700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43ec079cd0 con 0x7f43ec11a060 2019-12-05T05:23:08.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:08.398+0000 7f43f182e700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43ec07c130 con 0x7f43ec128200 2019-12-05T05:23:08.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:08.398+0000 7f43f182e700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43ec07c3d0 con 0x7f43ec079350 2019-12-05T05:23:08.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:08.398+0000 7f43ebfff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f43ec128200 0x7f43ec128630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:23:08.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:08.398+0000 7f43ebfff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f43ec128200 0x7f43ec128630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:60718/0 (socket says 172.21.15.42:60718) 2019-12-05T05:23:08.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:08.398+0000 7f43ebfff700 1 -- 172.21.15.42:0/3093839971 learned_addr learned my addr 172.21.15.42:0/3093839971 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:23:08.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:08.398+0000 7f43eaffd700 1 --2- 172.21.15.42:0/3093839971 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f43ec079350 0x7f43ec079780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:23:08.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:08.398+0000 7f43ebfff700 1 -- 172.21.15.42:0/3093839971 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f43ec079350 msgr2=0x7f43ec079780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:08.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:08.398+0000 7f43ebfff700 1 --2- 172.21.15.42:0/3093839971 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f43ec079350 0x7f43ec079780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:23:08.412 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:08.398+0000 7f43ebfff700 1 -- 172.21.15.42:0/3093839971 >> v1:172.21.15.40:6789/0 conn(0x7f43ec11a060 legacy=0x7f43ec07b890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:08.412 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:08.398+0000 7f43ebfff700 1 -- 172.21.15.42:0/3093839971 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43ec07ab80 con 0x7f43ec128200 2019-12-05T05:23:08.413 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:08.398+0000 7f43ebfff700 1 --2- 172.21.15.42:0/3093839971 >> v2:172.21.15.42:3300/0 conn(0x7f43ec128200 0x7f43ec128630 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f43ec07c2f0 tx=0x7f43e4010bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:08.413 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:08.398+0000 7f43ebfff700 1 --2- 172.21.15.42:0/3093839971 >> v2:172.21.15.42:3300/0 conn(0x7f43ec128200 0x7f43ec128630 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f43ec07c2f0 tx=0x7f43e4010bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:08.414 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:08.398+0000 7f43ebfff700 1 --2- 172.21.15.42:0/3093839971 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f43ec128200 0x7f43ec128630 secure :-1 s=READY pgs=742 cs=0 l=1 rx=0x7f43ec07c2f0 tx=0x7f43e4010bb0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:23:08.414 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:08.398+0000 7f43ebfff700 1 --2- 172.21.15.42:0/3093839971 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f43ec128200 0x7f43ec128630 secure :-1 s=THROTTLE_DONE pgs=742 cs=0 l=1 rx=0x7f43ec07c2f0 tx=0x7f43e4010bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:08.414 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:08.398+0000 7f43ebfff700 1 --2- 172.21.15.42:0/3093839971 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f43ec128200 0x7f43ec128630 secure :-1 s=THROTTLE_DONE pgs=742 cs=0 l=1 rx=0x7f43ec07c2f0 tx=0x7f43e4010bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:08.414 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:08.398+0000 7f43d3fff700 1 -- 172.21.15.42:0/3093839971 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f43e4007350 con 0x7f43ec128200 2019-12-05T05:23:08.415 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:08.398+0000 7f43ebfff700 1 --2- 172.21.15.42:0/3093839971 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f43ec128200 0x7f43ec128630 secure :-1 s=THROTTLE_DONE pgs=742 cs=0 l=1 rx=0x7f43ec07c2f0 tx=0x7f43e4010bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:08.415 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:08.398+0000 7f43d3fff700 1 -- 172.21.15.42:0/3093839971 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43e4005cb0 con 0x7f43ec128200 2019-12-05T05:23:08.415 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:08.398+0000 7f43f182e700 1 -- 172.21.15.42:0/3093839971 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43ec128b80 con 0x7f43ec128200 2019-12-05T05:23:08.415 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:08.398+0000 7f43d3fff700 1 -- 172.21.15.42:0/3093839971 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f43e4012420 con 0x7f43ec128200 2019-12-05T05:23:08.415 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:08.398+0000 7f43f182e700 1 -- 172.21.15.42:0/3093839971 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43ec12f420 con 0x7f43ec128200 2019-12-05T05:23:08.415 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:08.398+0000 7f43ebfff700 1 --2- 172.21.15.42:0/3093839971 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f43ec128200 0x7f43ec128630 secure :-1 s=THROTTLE_DONE pgs=742 cs=0 l=1 rx=0x7f43ec07c2f0 tx=0x7f43e4010bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:08.416 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:08.402+0000 7f43d3fff700 1 -- 172.21.15.42:0/3093839971 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f43e4012730 con 0x7f43ec128200 2019-12-05T05:23:08.416 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:08.402+0000 7f43ebfff700 1 --2- 172.21.15.42:0/3093839971 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f43ec128200 0x7f43ec128630 secure :-1 s=THROTTLE_DONE pgs=742 cs=0 l=1 rx=0x7f43ec07c2f0 tx=0x7f43e4010bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:08.416 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:08.402+0000 7f43d3fff700 1 --2- 172.21.15.42:0/3093839971 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f43cc03ae70 0x7f43cc03d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:23:08.416 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:08.402+0000 7f43d3fff700 1 -- 172.21.15.42:0/3093839971 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 7149+0+0 (secure 0 0 0) 0x7f43e404b260 con 0x7f43ec128200 2019-12-05T05:23:08.416 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:08.402+0000 7f43eb7fe700 1 --2- 172.21.15.42:0/3093839971 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f43cc03ae70 0x7f43cc03d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:23:08.417 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:08.402+0000 7f43eb7fe700 1 --2- 172.21.15.42:0/3093839971 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f43cc03ae70 0x7f43cc03d380 crc :-1 s=READY pgs=331 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:23:08.417 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:08.406+0000 7f43f182e700 1 -- 172.21.15.42:0/3093839971 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43ec05d110 con 0x7f43ec128200 2019-12-05T05:23:08.418 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:08.414+0000 7f43ebfff700 1 --2- 172.21.15.42:0/3093839971 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f43ec128200 0x7f43ec128630 secure :-1 s=THROTTLE_DONE pgs=742 cs=0 l=1 rx=0x7f43ec07c2f0 tx=0x7f43e4010bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:08.418 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:08.414+0000 7f43d3fff700 1 -- 172.21.15.42:0/3093839971 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f43e40322e0 con 0x7f43ec128200 2019-12-05T05:23:08.604 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:08.602+0000 7f43f182e700 1 -- 172.21.15.42:0/3093839971 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f43ec05b8c0 con 0x7f43ec128200 2019-12-05T05:23:08.605 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:08.602+0000 7f43ebfff700 1 --2- 172.21.15.42:0/3093839971 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f43ec128200 0x7f43ec128630 secure :-1 s=THROTTLE_DONE pgs=742 cs=0 l=1 rx=0x7f43ec07c2f0 tx=0x7f43e4010bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:08.605 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:08.602+0000 7f43d3fff700 1 -- 172.21.15.42:0/3093839971 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) v1 ==== 94+0+1772 (secure 0 0 0) 0x7f43e4032540 con 0x7f43ec128200 2019-12-05T05:23:08.605 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 97 2019-12-05T05:23:08.605 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:23:08.605 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":97,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":97,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:23:07.988779+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5123},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5123":{"gid":5123,"name":"a","rank":0,"incarnation":94,"state":"up:active","state_seq":37,"addr":"172.21.15.42:6835/1214659805","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1214659805},{"type":"v1","addr":"172.21.15.42:6835","nonce":1214659805}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:23:08.608 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:08.602+0000 7f43f182e700 1 -- 172.21.15.42:0/3093839971 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f43cc03ae70 msgr2=0x7f43cc03d380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:08.608 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:08.602+0000 7f43f182e700 1 --2- 172.21.15.42:0/3093839971 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f43cc03ae70 0x7f43cc03d380 crc :-1 s=READY pgs=331 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:23:08.608 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:08.602+0000 7f43f182e700 1 -- 172.21.15.42:0/3093839971 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f43ec128200 msgr2=0x7f43ec128630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:08.608 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:08.602+0000 7f43f182e700 1 --2- 172.21.15.42:0/3093839971 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f43ec128200 0x7f43ec128630 secure :-1 s=READY pgs=742 cs=0 l=1 rx=0x7f43ec07c2f0 tx=0x7f43e4010bb0).stop 2019-12-05T05:23:08.608 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:08.602+0000 7f43f182e700 1 -- 172.21.15.42:0/3093839971 shutdown_connections 2019-12-05T05:23:08.609 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:08.602+0000 7f43f182e700 1 --2- 172.21.15.42:0/3093839971 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f43cc03ae70 0x7f43cc03d380 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:23:08.609 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:08.602+0000 7f43f182e700 1 --2- 172.21.15.42:0/3093839971 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f43ec128200 0x7f43ec128630 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:23:08.609 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:08.602+0000 7f43f182e700 1 --2- 172.21.15.42:0/3093839971 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f43ec079350 0x7f43ec079780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:23:08.609 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:08.606+0000 7f43f182e700 1 -- 172.21.15.42:0/3093839971 shutdown_connections 2019-12-05T05:23:08.609 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:08.606+0000 7f43f182e700 1 -- 172.21.15.42:0/3093839971 wait complete. 2019-12-05T05:23:08.610 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:08.606+0000 7f43f182e700 1 -- 172.21.15.42:0/3093839971 >> 172.21.15.42:0/3093839971 conn(0x7f43ec0a9720 msgr2=0x7f43ec059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:23:08.617 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a has gained rank=0, replacing gid=5066 2019-12-05T05:23:08.617 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 6 secs before reviving mds.b 2019-12-05T05:23:14.618 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.b 2019-12-05T05:23:14.618 INFO:tasks.ceph.mds.b:Restarting daemon 2019-12-05T05:23:14.619 INFO:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2019-12-05T05:23:14.622 INFO:tasks.ceph.mds.b:Started 2019-12-05T05:23:14.622 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:23:14.682 INFO:tasks.ceph.mds.b.smithi040.stdout:starting mds.b at 2019-12-05T05:23:14.729 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:14.722+0000 7fbb1bdf1700 1 Processor -- start 2019-12-05T05:23:14.729 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:14.722+0000 7fbb1bdf1700 1 -- start start 2019-12-05T05:23:14.729 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:14.722+0000 7fbb1bdf1700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fbb1408df80 0x7fbb1409f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:23:14.730 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:14.722+0000 7fbb1bdf1700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb1409fc60 con 0x7fbb1409f830 2019-12-05T05:23:14.730 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:14.722+0000 7fbb1bdf1700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb140a0710 con 0x7fbb140a7740 2019-12-05T05:23:14.730 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:14.722+0000 7fbb1bdf1700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb140a0b00 con 0x7fbb1408df80 2019-12-05T05:23:14.730 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:14.722+0000 7fbb19b8d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fbb1408df80 0x7fbb1409f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:23:14.730 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:14.722+0000 7fbb19b8d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fbb1408df80 0x7fbb1409f2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:60762/0 (socket says 172.21.15.42:60762) 2019-12-05T05:23:14.731 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:14.722+0000 7fbb19b8d700 1 -- 172.21.15.42:0/1365946748 learned_addr learned my addr 172.21.15.42:0/1365946748 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:23:14.731 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:14.722+0000 7fbb19b8d700 1 -- 172.21.15.42:0/1365946748 >> v1:172.21.15.42:6789/0 conn(0x7fbb140a7740 legacy=0x7fbb1407e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:14.731 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:14.722+0000 7fbb19b8d700 1 -- 172.21.15.42:0/1365946748 >> v1:172.21.15.40:6789/0 conn(0x7fbb1409f830 legacy=0x7fbb1414efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:14.731 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:14.722+0000 7fbb19b8d700 1 -- 172.21.15.42:0/1365946748 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb1407ef20 con 0x7fbb1408df80 2019-12-05T05:23:14.731 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:14.722+0000 7fbb1938c700 1 -- 172.21.15.42:0/1365946748 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1920184404 0 0) 0x7fbb1409fc60 con 0x7fbb1409f830 2019-12-05T05:23:14.731 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:14.722+0000 7fbb19b8d700 1 --2- 172.21.15.42:0/1365946748 >> v2:172.21.15.42:3300/0 conn(0x7fbb1408df80 0x7fbb1409f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fbb10008790 tx=0x7fbb1000fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:14.731 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:14.722+0000 7fbb19b8d700 1 --2- 172.21.15.42:0/1365946748 >> v2:172.21.15.42:3300/0 conn(0x7fbb1408df80 0x7fbb1409f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fbb10008790 tx=0x7fbb1000fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:14.732 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:14.722+0000 7fbb19b8d700 1 --2- 172.21.15.42:0/1365946748 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbb1408df80 0x7fbb1409f2e0 secure :-1 s=READY pgs=747 cs=0 l=1 rx=0x7fbb10008790 tx=0x7fbb1000fcb0).ready entity=mon.1 client_cookie=b7d132b52f71a83 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:23:14.732 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:14.722+0000 7fbb19b8d700 1 --2- 172.21.15.42:0/1365946748 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbb1408df80 0x7fbb1409f2e0 secure :-1 s=THROTTLE_DONE pgs=747 cs=0 l=1 rx=0x7fbb10008790 tx=0x7fbb1000fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:14.732 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:14.722+0000 7fbb19b8d700 1 --2- 172.21.15.42:0/1365946748 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbb1408df80 0x7fbb1409f2e0 secure :-1 s=THROTTLE_DONE pgs=747 cs=0 l=1 rx=0x7fbb10008790 tx=0x7fbb1000fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:14.732 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:14.722+0000 7fbb19b8d700 1 --2- 172.21.15.42:0/1365946748 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbb1408df80 0x7fbb1409f2e0 secure :-1 s=THROTTLE_DONE pgs=747 cs=0 l=1 rx=0x7fbb10008790 tx=0x7fbb1000fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:14.732 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:14.722+0000 7fbb1938c700 1 -- 172.21.15.42:0/1365946748 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbb10011520 con 0x7fbb1408df80 2019-12-05T05:23:14.732 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:14.722+0000 7fbb1938c700 1 -- 172.21.15.42:0/1365946748 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb10008ca0 con 0x7fbb1408df80 2019-12-05T05:23:14.732 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:14.722+0000 7fbb1938c700 1 -- 172.21.15.42:0/1365946748 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbb10011b00 con 0x7fbb1408df80 2019-12-05T05:23:14.733 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:14.722+0000 7fbb1bdf1700 1 -- 172.21.15.42:0/1365946748 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbb1408df80 msgr2=0x7fbb1409f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:14.733 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:14.722+0000 7fbb1bdf1700 1 --2- 172.21.15.42:0/1365946748 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbb1408df80 0x7fbb1409f2e0 secure :-1 s=READY pgs=747 cs=0 l=1 rx=0x7fbb10008790 tx=0x7fbb1000fcb0).stop 2019-12-05T05:23:14.733 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:14.726+0000 7fbb1bdf1700 1 -- 172.21.15.42:0/1365946748 shutdown_connections 2019-12-05T05:23:14.733 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:14.726+0000 7fbb1bdf1700 1 --2- 172.21.15.42:0/1365946748 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbb1408df80 0x7fbb1409f2e0 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:23:14.733 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:14.726+0000 7fbb1bdf1700 1 -- 172.21.15.42:0/1365946748 shutdown_connections 2019-12-05T05:23:14.733 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:14.726+0000 7fbb1bdf1700 1 -- 172.21.15.42:0/1365946748 wait complete. 2019-12-05T05:23:14.733 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:14.726+0000 7fbb1bdf1700 1 -- 172.21.15.42:0/1365946748 >> 172.21.15.42:0/1365946748 conn(0x7fbb140aabe0 msgr2=0x7fbb140a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:23:14.737 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:14.726+0000 7fbb1bdf1700 1 Processor -- start 2019-12-05T05:23:14.737 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:14.726+0000 7fbb1bdf1700 1 -- start start 2019-12-05T05:23:14.737 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:14.726+0000 7fbb1bdf1700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbb1411c250 0x7fbb14123ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:23:14.737 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:14.726+0000 7fbb1bdf1700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fbb14134d80 0x7fbb141351b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:23:14.737 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:14.726+0000 7fbb1bdf1700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb14123750 con 0x7fbb1407c8d0 2019-12-05T05:23:14.737 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:14.726+0000 7fbb1bdf1700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb14124250 con 0x7fbb14134d80 2019-12-05T05:23:14.738 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:14.726+0000 7fbb1bdf1700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb141247d0 con 0x7fbb1411c250 2019-12-05T05:23:14.738 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:14.730+0000 7fbb19b8d700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fbb1407c8d0 0x7fbb14124a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:42014/0 (socket says 172.21.15.42:42014) 2019-12-05T05:23:14.738 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:14.730+0000 7fbb19b8d700 1 -- 172.21.15.42:0/869366211 learned_addr learned my addr 172.21.15.42:0/869366211 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:23:14.738 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:14.730+0000 7fbb1ab8f700 1 --2- 172.21.15.42:0/869366211 >> v2:172.21.15.42:3300/0 conn(0x7fbb14134d80 0x7fbb141351b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:23:14.738 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:14.730+0000 7fbb1a38e700 1 --2- 172.21.15.42:0/869366211 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbb1411c250 0x7fbb14123ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:23:14.738 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:14.730+0000 7fbb06ffd700 1 -- 172.21.15.42:0/869366211 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 491601057 0 0) 0x7fbb14123750 con 0x7fbb1407c8d0 2019-12-05T05:23:14.738 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:14.730+0000 7fbb06ffd700 1 -- 172.21.15.42:0/869366211 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbafc0030b0 con 0x7fbb1407c8d0 2019-12-05T05:23:14.738 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:14.730+0000 7fbb1ab8f700 1 -- 172.21.15.42:0/869366211 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbb1411c250 msgr2=0x7fbb14123ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:14.739 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:14.730+0000 7fbb1ab8f700 1 --2- 172.21.15.42:0/869366211 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbb1411c250 0x7fbb14123ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:23:14.739 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:14.730+0000 7fbb1ab8f700 1 -- 172.21.15.42:0/869366211 >> v1:172.21.15.40:6789/0 conn(0x7fbb1407c8d0 legacy=0x7fbb14124a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:14.739 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:14.730+0000 7fbb1ab8f700 1 -- 172.21.15.42:0/869366211 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb14122090 con 0x7fbb14134d80 2019-12-05T05:23:14.739 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:14.730+0000 7fbb1a38e700 1 --2- 172.21.15.42:0/869366211 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbb1411c250 0x7fbb14123ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:23:14.739 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:14.730+0000 7fbb1ab8f700 1 --2- 172.21.15.42:0/869366211 >> v2:172.21.15.42:3300/0 conn(0x7fbb14134d80 0x7fbb141351b0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fbb14124410 tx=0x7fbb0c010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:14.739 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:14.730+0000 7fbb1ab8f700 1 --2- 172.21.15.42:0/869366211 >> v2:172.21.15.42:3300/0 conn(0x7fbb14134d80 0x7fbb141351b0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fbb14124410 tx=0x7fbb0c010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:14.739 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:14.730+0000 7fbb1ab8f700 1 --2- 172.21.15.42:0/869366211 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbb14134d80 0x7fbb141351b0 secure :-1 s=READY pgs=748 cs=0 l=1 rx=0x7fbb14124410 tx=0x7fbb0c010b50).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:23:14.740 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:14.730+0000 7fbb1ab8f700 1 --2- 172.21.15.42:0/869366211 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbb14134d80 0x7fbb141351b0 secure :-1 s=THROTTLE_DONE pgs=748 cs=0 l=1 rx=0x7fbb14124410 tx=0x7fbb0c010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:14.740 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:14.730+0000 7fbb1ab8f700 1 --2- 172.21.15.42:0/869366211 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbb14134d80 0x7fbb141351b0 secure :-1 s=THROTTLE_DONE pgs=748 cs=0 l=1 rx=0x7fbb14124410 tx=0x7fbb0c010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:14.740 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:14.730+0000 7fbb06ffd700 1 -- 172.21.15.42:0/869366211 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbb0c006ac0 con 0x7fbb14134d80 2019-12-05T05:23:14.740 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:14.730+0000 7fbb1ab8f700 1 --2- 172.21.15.42:0/869366211 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbb14134d80 0x7fbb141351b0 secure :-1 s=THROTTLE_DONE pgs=748 cs=0 l=1 rx=0x7fbb14124410 tx=0x7fbb0c010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:14.740 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:14.730+0000 7fbb06ffd700 1 -- 172.21.15.42:0/869366211 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb0c0066e0 con 0x7fbb14134d80 2019-12-05T05:23:14.740 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:14.730+0000 7fbb1bdf1700 1 -- 172.21.15.42:0/869366211 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb14135700 con 0x7fbb14134d80 2019-12-05T05:23:14.740 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:14.730+0000 7fbb06ffd700 1 -- 172.21.15.42:0/869366211 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbb0c002730 con 0x7fbb14134d80 2019-12-05T05:23:14.741 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:14.730+0000 7fbb1bdf1700 1 -- 172.21.15.42:0/869366211 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb1413c140 con 0x7fbb14134d80 2019-12-05T05:23:14.741 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:14.730+0000 7fbb1ab8f700 1 --2- 172.21.15.42:0/869366211 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbb14134d80 0x7fbb141351b0 secure :-1 s=THROTTLE_DONE pgs=748 cs=0 l=1 rx=0x7fbb14124410 tx=0x7fbb0c010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:14.741 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:14.730+0000 7fbb06ffd700 1 -- 172.21.15.42:0/869366211 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fbb0c002a40 con 0x7fbb14134d80 2019-12-05T05:23:14.741 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:14.734+0000 7fbb1ab8f700 1 --2- 172.21.15.42:0/869366211 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbb14134d80 0x7fbb141351b0 secure :-1 s=THROTTLE_DONE pgs=748 cs=0 l=1 rx=0x7fbb14124410 tx=0x7fbb0c010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:14.741 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:14.734+0000 7fbb06ffd700 1 --2- 172.21.15.42:0/869366211 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbafc03b270 0x7fbafc03d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:23:14.741 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:14.734+0000 7fbb06ffd700 1 -- 172.21.15.42:0/869366211 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 7149+0+0 (secure 0 0 0) 0x7fbb0c04b020 con 0x7fbb14134d80 2019-12-05T05:23:14.741 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:14.734+0000 7fbb1a38e700 1 --2- 172.21.15.42:0/869366211 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbafc03b270 0x7fbafc03d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:23:14.742 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:14.734+0000 7fbb1a38e700 1 --2- 172.21.15.42:0/869366211 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbafc03b270 0x7fbafc03d780 crc :-1 s=READY pgs=332 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:23:14.742 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:14.738+0000 7fbb1bdf1700 1 -- 172.21.15.42:0/869366211 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb1405d110 con 0x7fbb14134d80 2019-12-05T05:23:14.751 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:14.746+0000 7fbb1ab8f700 1 --2- 172.21.15.42:0/869366211 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbb14134d80 0x7fbb141351b0 secure :-1 s=THROTTLE_DONE pgs=748 cs=0 l=1 rx=0x7fbb14124410 tx=0x7fbb0c010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:14.752 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:14.746+0000 7fbb06ffd700 1 -- 172.21.15.42:0/869366211 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fbb0c0320a0 con 0x7fbb14134d80 2019-12-05T05:23:14.922 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:14.918+0000 7fbb1bdf1700 1 -- 172.21.15.42:0/869366211 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbb1405b8c0 con 0x7fbb14134d80 2019-12-05T05:23:14.923 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:14.918+0000 7fbb1ab8f700 1 --2- 172.21.15.42:0/869366211 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbb14134d80 0x7fbb141351b0 secure :-1 s=THROTTLE_DONE pgs=748 cs=0 l=1 rx=0x7fbb14124410 tx=0x7fbb0c010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:14.923 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:14.918+0000 7fbb06ffd700 1 -- 172.21.15.42:0/869366211 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) v1 ==== 94+0+1772 (secure 0 0 0) 0x7fbb0c032300 con 0x7fbb14134d80 2019-12-05T05:23:14.924 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 97 2019-12-05T05:23:14.924 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:23:14.924 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":97,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":97,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:23:07.988779+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5123},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5123":{"gid":5123,"name":"a","rank":0,"incarnation":94,"state":"up:active","state_seq":37,"addr":"172.21.15.42:6835/1214659805","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1214659805},{"type":"v1","addr":"172.21.15.42:6835","nonce":1214659805}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:23:14.926 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:14.922+0000 7fbb1bdf1700 1 -- 172.21.15.42:0/869366211 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbafc03b270 msgr2=0x7fbafc03d780 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:14.926 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:14.922+0000 7fbb1bdf1700 1 --2- 172.21.15.42:0/869366211 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbafc03b270 0x7fbafc03d780 crc :-1 s=READY pgs=332 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:23:14.927 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:14.922+0000 7fbb1bdf1700 1 -- 172.21.15.42:0/869366211 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbb14134d80 msgr2=0x7fbb141351b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:14.927 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:14.922+0000 7fbb1bdf1700 1 --2- 172.21.15.42:0/869366211 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbb14134d80 0x7fbb141351b0 secure :-1 s=READY pgs=748 cs=0 l=1 rx=0x7fbb14124410 tx=0x7fbb0c010b50).stop 2019-12-05T05:23:14.927 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:14.922+0000 7fbb1bdf1700 1 -- 172.21.15.42:0/869366211 shutdown_connections 2019-12-05T05:23:14.927 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:14.922+0000 7fbb1bdf1700 1 --2- 172.21.15.42:0/869366211 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbafc03b270 0x7fbafc03d780 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:23:14.927 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:14.922+0000 7fbb1bdf1700 1 --2- 172.21.15.42:0/869366211 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbb14134d80 0x7fbb141351b0 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:23:14.927 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:14.922+0000 7fbb1bdf1700 1 --2- 172.21.15.42:0/869366211 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbb1411c250 0x7fbb14123ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:23:14.928 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:14.922+0000 7fbb1bdf1700 1 -- 172.21.15.42:0/869366211 shutdown_connections 2019-12-05T05:23:14.928 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:14.922+0000 7fbb1bdf1700 1 -- 172.21.15.42:0/869366211 wait complete. 2019-12-05T05:23:14.928 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:14.922+0000 7fbb1bdf1700 1 -- 172.21.15.42:0/869366211 >> 172.21.15.42:0/869366211 conn(0x7fbb1408fde0 msgr2=0x7fbb14059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:23:14.934 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2019-12-05T05:23:16.936 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:23:17.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:17.026+0000 7fc09d479700 1 Processor -- start 2019-12-05T05:23:17.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:17.026+0000 7fc09d479700 1 -- start start 2019-12-05T05:23:17.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:17.030+0000 7fc09d479700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc0980a7730 0x7fc09808df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:23:17.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:17.030+0000 7fc09d479700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0980a0700 con 0x7fc09807e710 2019-12-05T05:23:17.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:17.030+0000 7fc09d479700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0980a9b90 con 0x7fc09809fad0 2019-12-05T05:23:17.039 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:17.030+0000 7fc09d479700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0980a0af0 con 0x7fc0980a7730 2019-12-05T05:23:17.039 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:17.030+0000 7fc097fff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fc09809fad0 0x7fc09814efd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:60820/0 (socket says 172.21.15.42:60820) 2019-12-05T05:23:17.039 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:17.030+0000 7fc097fff700 1 -- 172.21.15.42:0/1830434535 learned_addr learned my addr 172.21.15.42:0/1830434535 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:23:17.039 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:17.030+0000 7fc0977fe700 1 --2- 172.21.15.42:0/1830434535 >> v2:172.21.15.40:3300/0 conn(0x7fc0980a7730 0x7fc09808df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:23:17.039 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:17.030+0000 7fc0967fc700 1 -- 172.21.15.42:0/1830434535 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1389149154 0 0) 0x7fc0980a9b90 con 0x7fc09809fad0 2019-12-05T05:23:17.039 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:17.030+0000 7fc0967fc700 1 -- 172.21.15.42:0/1830434535 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0840030b0 con 0x7fc09809fad0 2019-12-05T05:23:17.039 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:17.030+0000 7fc0977fe700 1 -- 172.21.15.42:0/1830434535 >> v1:172.21.15.42:6789/0 conn(0x7fc09809fad0 legacy=0x7fc09814efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:17.040 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:17.030+0000 7fc0977fe700 1 -- 172.21.15.42:0/1830434535 >> v1:172.21.15.40:6789/0 conn(0x7fc09807e710 legacy=0x7fc09809f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:17.040 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:17.030+0000 7fc0977fe700 1 -- 172.21.15.42:0/1830434535 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc09807eb40 con 0x7fc0980a7730 2019-12-05T05:23:17.040 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:17.030+0000 7fc0977fe700 1 --2- 172.21.15.42:0/1830434535 >> v2:172.21.15.40:3300/0 conn(0x7fc0980a7730 0x7fc09808df70 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fc080008790 tx=0x7fc08000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:17.041 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:17.030+0000 7fc0977fe700 1 --2- 172.21.15.42:0/1830434535 >> v2:172.21.15.40:3300/0 conn(0x7fc0980a7730 0x7fc09808df70 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fc080008790 tx=0x7fc08000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:17.041 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:17.030+0000 7fc0977fe700 1 --2- 172.21.15.42:0/1830434535 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc0980a7730 0x7fc09808df70 secure :-1 s=READY pgs=601 cs=0 l=1 rx=0x7fc080008790 tx=0x7fc08000fbf0).ready entity=mon.0 client_cookie=2b6f512904b620b4 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:23:17.041 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:17.030+0000 7fc0977fe700 1 --2- 172.21.15.42:0/1830434535 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc0980a7730 0x7fc09808df70 secure :-1 s=THROTTLE_DONE pgs=601 cs=0 l=1 rx=0x7fc080008790 tx=0x7fc08000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:17.041 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:17.030+0000 7fc0977fe700 1 --2- 172.21.15.42:0/1830434535 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc0980a7730 0x7fc09808df70 secure :-1 s=THROTTLE_DONE pgs=601 cs=0 l=1 rx=0x7fc080008790 tx=0x7fc08000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:17.041 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:17.030+0000 7fc0967fc700 1 -- 172.21.15.42:0/1830434535 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc080011270 con 0x7fc0980a7730 2019-12-05T05:23:17.041 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:17.030+0000 7fc0977fe700 1 --2- 172.21.15.42:0/1830434535 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc0980a7730 0x7fc09808df70 secure :-1 s=THROTTLE_DONE pgs=601 cs=0 l=1 rx=0x7fc080008790 tx=0x7fc08000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:17.042 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:17.030+0000 7fc0967fc700 1 -- 172.21.15.42:0/1830434535 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc080008bc0 con 0x7fc0980a7730 2019-12-05T05:23:17.042 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:17.030+0000 7fc0967fc700 1 -- 172.21.15.42:0/1830434535 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc080011850 con 0x7fc0980a7730 2019-12-05T05:23:17.042 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:17.030+0000 7fc09d479700 1 -- 172.21.15.42:0/1830434535 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc0980a7730 msgr2=0x7fc09808df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:17.042 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:17.030+0000 7fc09d479700 1 --2- 172.21.15.42:0/1830434535 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc0980a7730 0x7fc09808df70 secure :-1 s=READY pgs=601 cs=0 l=1 rx=0x7fc080008790 tx=0x7fc08000fbf0).stop 2019-12-05T05:23:17.042 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:17.034+0000 7fc09d479700 1 -- 172.21.15.42:0/1830434535 shutdown_connections 2019-12-05T05:23:17.042 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:17.034+0000 7fc09d479700 1 --2- 172.21.15.42:0/1830434535 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc0980a7730 0x7fc09808df70 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:23:17.042 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:17.034+0000 7fc09d479700 1 -- 172.21.15.42:0/1830434535 shutdown_connections 2019-12-05T05:23:17.043 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:17.034+0000 7fc09d479700 1 -- 172.21.15.42:0/1830434535 wait complete. 2019-12-05T05:23:17.043 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:17.034+0000 7fc09d479700 1 -- 172.21.15.42:0/1830434535 >> 172.21.15.42:0/1830434535 conn(0x7fc0980aabd0 msgr2=0x7fc0980a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:23:17.054 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:17.034+0000 7fc09d479700 1 Processor -- start 2019-12-05T05:23:17.054 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:17.034+0000 7fc09d479700 1 -- start start 2019-12-05T05:23:17.054 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:17.038+0000 7fc09d479700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc098121090 0x7fc0980a3a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:23:17.054 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:17.038+0000 7fc09d479700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc0980a3f90 0x7fc098134440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:23:17.055 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:17.038+0000 7fc09d479700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0980a2710 con 0x7fc09811c0c0 2019-12-05T05:23:17.055 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:17.038+0000 7fc09d479700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0980a21d0 con 0x7fc098121090 2019-12-05T05:23:17.055 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:17.038+0000 7fc09d479700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0980a2470 con 0x7fc0980a3f90 2019-12-05T05:23:17.055 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:17.038+0000 7fc0977fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc098121090 0x7fc0980a3a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:23:17.055 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:17.038+0000 7fc0977fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc098121090 0x7fc0980a3a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:60784/0 (socket says 172.21.15.42:60784) 2019-12-05T05:23:17.056 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:17.038+0000 7fc0977fe700 1 -- 172.21.15.42:0/3763941062 learned_addr learned my addr 172.21.15.42:0/3763941062 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:23:17.056 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:17.038+0000 7fc097fff700 1 --2- 172.21.15.42:0/3763941062 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc0980a3f90 0x7fc098134440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:23:17.056 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:17.038+0000 7fc0977fe700 1 -- 172.21.15.42:0/3763941062 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc0980a3f90 msgr2=0x7fc098134440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:17.056 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:17.038+0000 7fc0977fe700 1 --2- 172.21.15.42:0/3763941062 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc0980a3f90 0x7fc098134440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:23:17.056 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:17.038+0000 7fc0977fe700 1 -- 172.21.15.42:0/3763941062 >> v1:172.21.15.42:6789/0 conn(0x7fc09811c0c0 legacy=0x7fc09807d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:17.056 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:17.038+0000 7fc0977fe700 1 -- 172.21.15.42:0/3763941062 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc09812cff0 con 0x7fc098121090 2019-12-05T05:23:17.057 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:17.038+0000 7fc0977fe700 1 --2- 172.21.15.42:0/3763941062 >> v2:172.21.15.42:3300/0 conn(0x7fc098121090 0x7fc0980a3a40 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fc080008790 tx=0x7fc080008ea0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:17.057 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:17.038+0000 7fc0977fe700 1 --2- 172.21.15.42:0/3763941062 >> v2:172.21.15.42:3300/0 conn(0x7fc098121090 0x7fc0980a3a40 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fc080008790 tx=0x7fc080008ea0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:17.057 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:17.038+0000 7fc0977fe700 1 --2- 172.21.15.42:0/3763941062 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc098121090 0x7fc0980a3a40 secure :-1 s=READY pgs=751 cs=0 l=1 rx=0x7fc080008790 tx=0x7fc080008ea0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:23:17.057 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:17.038+0000 7fc0977fe700 1 --2- 172.21.15.42:0/3763941062 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc098121090 0x7fc0980a3a40 secure :-1 s=THROTTLE_DONE pgs=751 cs=0 l=1 rx=0x7fc080008790 tx=0x7fc080008ea0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:17.057 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:17.038+0000 7fc08ffff700 1 -- 172.21.15.42:0/3763941062 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc080007340 con 0x7fc098121090 2019-12-05T05:23:17.057 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:17.038+0000 7fc0977fe700 1 --2- 172.21.15.42:0/3763941062 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc098121090 0x7fc0980a3a40 secure :-1 s=THROTTLE_DONE pgs=751 cs=0 l=1 rx=0x7fc080008790 tx=0x7fc080008ea0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:17.057 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:17.038+0000 7fc0977fe700 1 --2- 172.21.15.42:0/3763941062 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc098121090 0x7fc0980a3a40 secure :-1 s=THROTTLE_DONE pgs=751 cs=0 l=1 rx=0x7fc080008790 tx=0x7fc080008ea0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:17.058 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:17.038+0000 7fc08ffff700 1 -- 172.21.15.42:0/3763941062 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc080011470 con 0x7fc098121090 2019-12-05T05:23:17.058 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:17.038+0000 7fc08ffff700 1 -- 172.21.15.42:0/3763941062 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc080005540 con 0x7fc098121090 2019-12-05T05:23:17.058 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:17.038+0000 7fc09d479700 1 -- 172.21.15.42:0/3763941062 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0980a1890 con 0x7fc098121090 2019-12-05T05:23:17.058 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:17.038+0000 7fc09d479700 1 -- 172.21.15.42:0/3763941062 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc09807e040 con 0x7fc098121090 2019-12-05T05:23:17.058 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:17.038+0000 7fc0977fe700 1 --2- 172.21.15.42:0/3763941062 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc098121090 0x7fc0980a3a40 secure :-1 s=THROTTLE_DONE pgs=751 cs=0 l=1 rx=0x7fc080008790 tx=0x7fc080008ea0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:17.058 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:17.042+0000 7fc08ffff700 1 -- 172.21.15.42:0/3763941062 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fc080005850 con 0x7fc098121090 2019-12-05T05:23:17.059 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:17.042+0000 7fc0977fe700 1 --2- 172.21.15.42:0/3763941062 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc098121090 0x7fc0980a3a40 secure :-1 s=THROTTLE_DONE pgs=751 cs=0 l=1 rx=0x7fc080008790 tx=0x7fc080008ea0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:17.059 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:17.042+0000 7fc08ffff700 1 --2- 172.21.15.42:0/3763941062 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc07c03ace0 0x7fc07c03d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:23:17.059 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:17.042+0000 7fc08ffff700 1 -- 172.21.15.42:0/3763941062 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 7149+0+0 (secure 0 0 0) 0x7fc08004bbb0 con 0x7fc098121090 2019-12-05T05:23:17.059 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:17.042+0000 7fc096ffd700 1 --2- 172.21.15.42:0/3763941062 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc07c03ace0 0x7fc07c03d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:23:17.059 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:17.042+0000 7fc096ffd700 1 --2- 172.21.15.42:0/3763941062 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc07c03ace0 0x7fc07c03d1f0 crc :-1 s=READY pgs=333 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:23:17.060 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:17.042+0000 7fc09d479700 1 -- 172.21.15.42:0/3763941062 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc09805d100 con 0x7fc098121090 2019-12-05T05:23:17.060 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:17.050+0000 7fc0977fe700 1 --2- 172.21.15.42:0/3763941062 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc098121090 0x7fc0980a3a40 secure :-1 s=THROTTLE_DONE pgs=751 cs=0 l=1 rx=0x7fc080008790 tx=0x7fc080008ea0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:17.060 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:17.050+0000 7fc08ffff700 1 -- 172.21.15.42:0/3763941062 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fc080032e90 con 0x7fc098121090 2019-12-05T05:23:17.219 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:17.214+0000 7fc09d479700 1 -- 172.21.15.42:0/3763941062 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc09805b8b0 con 0x7fc098121090 2019-12-05T05:23:17.219 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:17.214+0000 7fc0977fe700 1 --2- 172.21.15.42:0/3763941062 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc098121090 0x7fc0980a3a40 secure :-1 s=THROTTLE_DONE pgs=751 cs=0 l=1 rx=0x7fc080008790 tx=0x7fc080008ea0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:17.219 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:17.214+0000 7fc08ffff700 1 -- 172.21.15.42:0/3763941062 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) v1 ==== 94+0+1772 (secure 0 0 0) 0x7fc080030840 con 0x7fc098121090 2019-12-05T05:23:17.220 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 97 2019-12-05T05:23:17.220 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:23:17.220 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":97,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":97,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:23:07.988779+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5123},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5123":{"gid":5123,"name":"a","rank":0,"incarnation":94,"state":"up:active","state_seq":37,"addr":"172.21.15.42:6835/1214659805","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1214659805},{"type":"v1","addr":"172.21.15.42:6835","nonce":1214659805}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:23:17.222 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:17.218+0000 7fc09d479700 1 -- 172.21.15.42:0/3763941062 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc07c03ace0 msgr2=0x7fc07c03d1f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:17.222 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:17.218+0000 7fc09d479700 1 --2- 172.21.15.42:0/3763941062 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc07c03ace0 0x7fc07c03d1f0 crc :-1 s=READY pgs=333 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:23:17.222 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:17.218+0000 7fc09d479700 1 -- 172.21.15.42:0/3763941062 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc098121090 msgr2=0x7fc0980a3a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:17.223 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:17.218+0000 7fc09d479700 1 --2- 172.21.15.42:0/3763941062 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc098121090 0x7fc0980a3a40 secure :-1 s=READY pgs=751 cs=0 l=1 rx=0x7fc080008790 tx=0x7fc080008ea0).stop 2019-12-05T05:23:17.223 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:17.218+0000 7fc09d479700 1 -- 172.21.15.42:0/3763941062 shutdown_connections 2019-12-05T05:23:17.223 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:17.218+0000 7fc09d479700 1 --2- 172.21.15.42:0/3763941062 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc07c03ace0 0x7fc07c03d1f0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:23:17.223 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:17.218+0000 7fc09d479700 1 --2- 172.21.15.42:0/3763941062 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc0980a3f90 0x7fc098134440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:23:17.223 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:17.218+0000 7fc09d479700 1 --2- 172.21.15.42:0/3763941062 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc098121090 0x7fc0980a3a40 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:23:17.223 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:17.218+0000 7fc09d479700 1 -- 172.21.15.42:0/3763941062 shutdown_connections 2019-12-05T05:23:17.223 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:17.218+0000 7fc09d479700 1 -- 172.21.15.42:0/3763941062 wait complete. 2019-12-05T05:23:17.224 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:17.218+0000 7fc09d479700 1 -- 172.21.15.42:0/3763941062 >> 172.21.15.42:0/3763941062 conn(0x7fc09808fdd0 msgr2=0x7fc09811c600 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:23:17.232 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2019-12-05T05:23:19.232 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:23:19.335 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:19.326+0000 7f55b623a700 1 Processor -- start 2019-12-05T05:23:19.335 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:19.326+0000 7f55b623a700 1 -- start start 2019-12-05T05:23:19.336 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:19.326+0000 7f55b623a700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f55b008df80 0x7f55b009f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:23:19.336 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:19.326+0000 7f55b623a700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55b009fc60 con 0x7f55b00a7740 2019-12-05T05:23:19.336 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:19.326+0000 7f55b623a700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55b00a0710 con 0x7f55b009f830 2019-12-05T05:23:19.336 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:19.326+0000 7f55b623a700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55b00a0b00 con 0x7f55b008df80 2019-12-05T05:23:19.336 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:19.330+0000 7f55b4fd8700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f55b009f830 0x7f55b014efe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:60848/0 (socket says 172.21.15.42:60848) 2019-12-05T05:23:19.336 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:19.330+0000 7f55b4fd8700 1 -- 172.21.15.42:0/4284526395 learned_addr learned my addr 172.21.15.42:0/4284526395 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:23:19.337 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:19.330+0000 7f55af7fe700 1 --2- 172.21.15.42:0/4284526395 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f55b008df80 0x7f55b009f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:23:19.337 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:19.330+0000 7f55aeffd700 1 -- 172.21.15.42:0/4284526395 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3048937833 0 0) 0x7f55b009fc60 con 0x7f55b00a7740 2019-12-05T05:23:19.337 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:19.330+0000 7f55aeffd700 1 -- 172.21.15.42:0/4284526395 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55900030b0 con 0x7f55b00a7740 2019-12-05T05:23:19.337 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:19.330+0000 7f55aeffd700 1 -- 172.21.15.42:0/4284526395 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1755199676 0 0) 0x7f55b00a0710 con 0x7f55b009f830 2019-12-05T05:23:19.337 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:19.330+0000 7f55aeffd700 1 -- 172.21.15.42:0/4284526395 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55b009fc60 con 0x7f55b009f830 2019-12-05T05:23:19.337 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:19.330+0000 7f55af7fe700 1 -- 172.21.15.42:0/4284526395 >> v1:172.21.15.42:6789/0 conn(0x7f55b009f830 legacy=0x7f55b014efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:19.337 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:19.330+0000 7f55af7fe700 1 -- 172.21.15.42:0/4284526395 >> v1:172.21.15.40:6789/0 conn(0x7f55b00a7740 legacy=0x7f55b007e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:19.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:19.330+0000 7f55af7fe700 1 -- 172.21.15.42:0/4284526395 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55b007ef20 con 0x7f55b008df80 2019-12-05T05:23:19.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:19.330+0000 7f55af7fe700 1 --2- 172.21.15.42:0/4284526395 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f55b008df80 0x7f55b009f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f5598002f30 tx=0x7f559800fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:19.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:19.330+0000 7f55af7fe700 1 --2- 172.21.15.42:0/4284526395 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f55b008df80 0x7f55b009f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f5598002f30 tx=0x7f559800fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:19.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:19.330+0000 7f55af7fe700 1 --2- 172.21.15.42:0/4284526395 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f55b008df80 0x7f55b009f2e0 secure :-1 s=READY pgs=156 cs=0 l=1 rx=0x7f5598002f30 tx=0x7f559800fbf0).ready entity=mon.2 client_cookie=18450f075aa78937 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:23:19.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:19.330+0000 7f55af7fe700 1 --2- 172.21.15.42:0/4284526395 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f55b008df80 0x7f55b009f2e0 secure :-1 s=THROTTLE_DONE pgs=156 cs=0 l=1 rx=0x7f5598002f30 tx=0x7f559800fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:19.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:19.330+0000 7f55aeffd700 1 -- 172.21.15.42:0/4284526395 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f55980112b0 con 0x7f55b008df80 2019-12-05T05:23:19.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:19.330+0000 7f55af7fe700 1 --2- 172.21.15.42:0/4284526395 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f55b008df80 0x7f55b009f2e0 secure :-1 s=THROTTLE_DONE pgs=156 cs=0 l=1 rx=0x7f5598002f30 tx=0x7f559800fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:19.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:19.330+0000 7f55af7fe700 1 --2- 172.21.15.42:0/4284526395 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f55b008df80 0x7f55b009f2e0 secure :-1 s=THROTTLE_DONE pgs=156 cs=0 l=1 rx=0x7f5598002f30 tx=0x7f559800fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:19.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:19.330+0000 7f55aeffd700 1 -- 172.21.15.42:0/4284526395 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5598008bc0 con 0x7f55b008df80 2019-12-05T05:23:19.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:19.330+0000 7f55aeffd700 1 -- 172.21.15.42:0/4284526395 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5598011890 con 0x7f55b008df80 2019-12-05T05:23:19.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:19.330+0000 7f55b623a700 1 -- 172.21.15.42:0/4284526395 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f55b008df80 msgr2=0x7f55b009f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:19.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:19.330+0000 7f55b623a700 1 --2- 172.21.15.42:0/4284526395 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f55b008df80 0x7f55b009f2e0 secure :-1 s=READY pgs=156 cs=0 l=1 rx=0x7f5598002f30 tx=0x7f559800fbf0).stop 2019-12-05T05:23:19.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:19.330+0000 7f55b623a700 1 -- 172.21.15.42:0/4284526395 shutdown_connections 2019-12-05T05:23:19.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:19.330+0000 7f55b623a700 1 --2- 172.21.15.42:0/4284526395 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f55b008df80 0x7f55b009f2e0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:23:19.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:19.330+0000 7f55b623a700 1 -- 172.21.15.42:0/4284526395 shutdown_connections 2019-12-05T05:23:19.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:19.330+0000 7f55b623a700 1 -- 172.21.15.42:0/4284526395 wait complete. 2019-12-05T05:23:19.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:19.330+0000 7f55b623a700 1 -- 172.21.15.42:0/4284526395 >> 172.21.15.42:0/4284526395 conn(0x7f55b00aabe0 msgr2=0x7f55b00a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:23:19.343 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:19.334+0000 7f55b623a700 1 Processor -- start 2019-12-05T05:23:19.343 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:19.334+0000 7f55b623a700 1 -- start start 2019-12-05T05:23:19.343 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:19.334+0000 7f55b623a700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f55b0079590 0x7f55b007a930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:23:19.344 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:19.334+0000 7f55b623a700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f55b01227e0 0x7f55b00a2180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:23:19.344 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:19.334+0000 7f55b623a700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f55b00a26d0 0x7f55b0125440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:23:19.344 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:19.334+0000 7f55b623a700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55b00a1900 con 0x7f55b00a26d0 2019-12-05T05:23:19.344 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:19.334+0000 7f55b623a700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55b01222a0 con 0x7f55b01227e0 2019-12-05T05:23:19.344 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:19.334+0000 7f55b623a700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55b0122540 con 0x7f55b0079590 2019-12-05T05:23:19.344 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:19.334+0000 7f55af7fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f55b01227e0 0x7f55b00a2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:23:19.344 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:19.334+0000 7f55af7fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f55b01227e0 0x7f55b00a2180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:60814/0 (socket says 172.21.15.42:60814) 2019-12-05T05:23:19.345 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:19.334+0000 7f55af7fe700 1 -- 172.21.15.42:0/123295731 learned_addr learned my addr 172.21.15.42:0/123295731 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:23:19.345 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:19.334+0000 7f55affff700 1 --2- 172.21.15.42:0/123295731 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f55b0079590 0x7f55b007a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:23:19.345 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:19.334+0000 7f55b4fd8700 1 --2- 172.21.15.42:0/123295731 >> v2:172.21.15.40:3300/0 conn(0x7f55b00a26d0 0x7f55b0125440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:23:19.345 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:19.334+0000 7f55af7fe700 1 -- 172.21.15.42:0/123295731 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f55b0079590 msgr2=0x7f55b007a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:19.345 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:19.334+0000 7f55af7fe700 1 --2- 172.21.15.42:0/123295731 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f55b0079590 0x7f55b007a930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:23:19.346 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:19.334+0000 7f55af7fe700 1 -- 172.21.15.42:0/123295731 >> v2:172.21.15.40:3300/0 conn(0x7f55b00a26d0 msgr2=0x7f55b0125440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:19.346 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:19.334+0000 7f55af7fe700 1 --2- 172.21.15.42:0/123295731 >> v2:172.21.15.40:3300/0 conn(0x7f55b00a26d0 0x7f55b0125440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:23:19.346 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:19.334+0000 7f55af7fe700 1 -- 172.21.15.42:0/123295731 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55b007ae80 con 0x7f55b01227e0 2019-12-05T05:23:19.346 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:19.334+0000 7f55affff700 1 --2- 172.21.15.42:0/123295731 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f55b0079590 0x7f55b007a930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T05:23:19.346 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:19.334+0000 7f55af7fe700 1 --2- 172.21.15.42:0/123295731 >> v2:172.21.15.42:3300/0 conn(0x7f55b01227e0 0x7f55b00a2180 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f55b0078ea0 tx=0x7f5598008c40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:19.346 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:19.334+0000 7f55af7fe700 1 --2- 172.21.15.42:0/123295731 >> v2:172.21.15.42:3300/0 conn(0x7f55b01227e0 0x7f55b00a2180 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f55b0078ea0 tx=0x7f5598008c40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:19.347 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:19.334+0000 7f55af7fe700 1 --2- 172.21.15.42:0/123295731 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f55b01227e0 0x7f55b00a2180 secure :-1 s=READY pgs=753 cs=0 l=1 rx=0x7f55b0078ea0 tx=0x7f5598008c40).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:23:19.347 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:19.334+0000 7f55af7fe700 1 --2- 172.21.15.42:0/123295731 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f55b01227e0 0x7f55b00a2180 secure :-1 s=THROTTLE_DONE pgs=753 cs=0 l=1 rx=0x7f55b0078ea0 tx=0x7f5598008c40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:19.347 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:19.334+0000 7f55acff9700 1 -- 172.21.15.42:0/123295731 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5598007150 con 0x7f55b01227e0 2019-12-05T05:23:19.347 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:19.334+0000 7f55af7fe700 1 --2- 172.21.15.42:0/123295731 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f55b01227e0 0x7f55b00a2180 secure :-1 s=THROTTLE_DONE pgs=753 cs=0 l=1 rx=0x7f55b0078ea0 tx=0x7f5598008c40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:19.347 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:19.334+0000 7f55af7fe700 1 --2- 172.21.15.42:0/123295731 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f55b01227e0 0x7f55b00a2180 secure :-1 s=THROTTLE_DONE pgs=753 cs=0 l=1 rx=0x7f55b0078ea0 tx=0x7f5598008c40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:19.347 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:19.334+0000 7f55acff9700 1 -- 172.21.15.42:0/123295731 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55980114b0 con 0x7f55b01227e0 2019-12-05T05:23:19.347 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:19.334+0000 7f55b623a700 1 -- 172.21.15.42:0/123295731 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55b0121e80 con 0x7f55b01227e0 2019-12-05T05:23:19.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:19.334+0000 7f55acff9700 1 -- 172.21.15.42:0/123295731 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f559800f550 con 0x7f55b01227e0 2019-12-05T05:23:19.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:19.334+0000 7f55b623a700 1 -- 172.21.15.42:0/123295731 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55b007a660 con 0x7f55b01227e0 2019-12-05T05:23:19.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:19.338+0000 7f55af7fe700 1 --2- 172.21.15.42:0/123295731 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f55b01227e0 0x7f55b00a2180 secure :-1 s=THROTTLE_DONE pgs=753 cs=0 l=1 rx=0x7f55b0078ea0 tx=0x7f5598008c40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:19.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:19.338+0000 7f55acff9700 1 -- 172.21.15.42:0/123295731 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f5598005540 con 0x7f55b01227e0 2019-12-05T05:23:19.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:19.338+0000 7f55af7fe700 1 --2- 172.21.15.42:0/123295731 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f55b01227e0 0x7f55b00a2180 secure :-1 s=THROTTLE_DONE pgs=753 cs=0 l=1 rx=0x7f55b0078ea0 tx=0x7f5598008c40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:19.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:19.338+0000 7f55acff9700 1 --2- 172.21.15.42:0/123295731 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f558803ae00 0x7f558803d310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:23:19.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:19.338+0000 7f55acff9700 1 -- 172.21.15.42:0/123295731 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 7149+0+0 (secure 0 0 0) 0x7f559804b700 con 0x7f55b01227e0 2019-12-05T05:23:19.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:19.338+0000 7f55affff700 1 --2- 172.21.15.42:0/123295731 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f558803ae00 0x7f558803d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:23:19.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:19.338+0000 7f55affff700 1 --2- 172.21.15.42:0/123295731 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f558803ae00 0x7f558803d310 crc :-1 s=READY pgs=334 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:23:19.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:19.342+0000 7f55b623a700 1 -- 172.21.15.42:0/123295731 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55b005d110 con 0x7f55b01227e0 2019-12-05T05:23:19.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:19.350+0000 7f55af7fe700 1 --2- 172.21.15.42:0/123295731 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f55b01227e0 0x7f55b00a2180 secure :-1 s=THROTTLE_DONE pgs=753 cs=0 l=1 rx=0x7f55b0078ea0 tx=0x7f5598008c40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:19.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:19.350+0000 7f55acff9700 1 -- 172.21.15.42:0/123295731 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f55b007a660 con 0x7f55b01227e0 2019-12-05T05:23:19.521 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:19.518+0000 7f55b623a700 1 -- 172.21.15.42:0/123295731 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f55b005b8c0 con 0x7f55b01227e0 2019-12-05T05:23:19.522 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:19.518+0000 7f55af7fe700 1 --2- 172.21.15.42:0/123295731 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f55b01227e0 0x7f55b00a2180 secure :-1 s=THROTTLE_DONE pgs=753 cs=0 l=1 rx=0x7f55b0078ea0 tx=0x7f5598008c40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:19.522 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:19.518+0000 7f55acff9700 1 -- 172.21.15.42:0/123295731 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) v1 ==== 94+0+1772 (secure 0 0 0) 0x7f5598033590 con 0x7f55b01227e0 2019-12-05T05:23:19.523 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 97 2019-12-05T05:23:19.523 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:23:19.523 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":97,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":97,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:23:07.988779+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5123},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5123":{"gid":5123,"name":"a","rank":0,"incarnation":94,"state":"up:active","state_seq":37,"addr":"172.21.15.42:6835/1214659805","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1214659805},{"type":"v1","addr":"172.21.15.42:6835","nonce":1214659805}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:23:19.531 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:19.522+0000 7f55b623a700 1 -- 172.21.15.42:0/123295731 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f558803ae00 msgr2=0x7f558803d310 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:19.531 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:19.522+0000 7f55b623a700 1 --2- 172.21.15.42:0/123295731 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f558803ae00 0x7f558803d310 crc :-1 s=READY pgs=334 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:23:19.532 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:19.522+0000 7f55b623a700 1 -- 172.21.15.42:0/123295731 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f55b01227e0 msgr2=0x7f55b00a2180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:19.532 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:19.522+0000 7f55b623a700 1 --2- 172.21.15.42:0/123295731 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f55b01227e0 0x7f55b00a2180 secure :-1 s=READY pgs=753 cs=0 l=1 rx=0x7f55b0078ea0 tx=0x7f5598008c40).stop 2019-12-05T05:23:19.532 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:19.522+0000 7f55b623a700 1 -- 172.21.15.42:0/123295731 shutdown_connections 2019-12-05T05:23:19.532 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:19.522+0000 7f55b623a700 1 --2- 172.21.15.42:0/123295731 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f558803ae00 0x7f558803d310 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:23:19.532 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:19.522+0000 7f55b623a700 1 --2- 172.21.15.42:0/123295731 >> v2:172.21.15.40:3300/0 conn(0x7f55b00a26d0 0x7f55b0125440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:23:19.532 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:19.522+0000 7f55b623a700 1 --2- 172.21.15.42:0/123295731 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f55b0079590 0x7f55b007a930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:23:19.533 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:19.522+0000 7f55b623a700 1 --2- 172.21.15.42:0/123295731 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f55b01227e0 0x7f55b00a2180 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:23:19.533 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:19.522+0000 7f55b623a700 1 -- 172.21.15.42:0/123295731 shutdown_connections 2019-12-05T05:23:19.539 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:19.522+0000 7f55b623a700 1 -- 172.21.15.42:0/123295731 wait complete. 2019-12-05T05:23:19.539 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:19.522+0000 7f55b623a700 1 -- 172.21.15.42:0/123295731 >> 172.21.15.42:0/123295731 conn(0x7f55b00a82a0 msgr2=0x7f55b014ffb0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:23:19.541 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2019-12-05T05:23:21.541 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:23:21.647 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:21.638+0000 7f4800813700 1 Processor -- start 2019-12-05T05:23:21.647 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:21.638+0000 7f4800813700 1 -- start start 2019-12-05T05:23:21.647 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:21.642+0000 7f4800813700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f47fc0a7740 0x7f47fc08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:23:21.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:21.642+0000 7f4800813700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f47fc07e720 0x7f47fc07eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:23:21.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:21.642+0000 7f4800813700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47fc0a0710 con 0x7f47fc09f2e0 2019-12-05T05:23:21.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:21.642+0000 7f4800813700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47fc0a0b00 con 0x7f47fc07e720 2019-12-05T05:23:21.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:21.642+0000 7f4800813700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47fc08e5b0 con 0x7f47fc0a7740 2019-12-05T05:23:21.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:21.642+0000 7f47fa59c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f47fc07e720 0x7f47fc07eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:23:21.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:21.642+0000 7f47fb59e700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f47fc09f2e0 0x7f47fc09f710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:60870/0 (socket says 172.21.15.42:60870) 2019-12-05T05:23:21.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:21.642+0000 7f47fb59e700 1 -- 172.21.15.42:0/4243918245 learned_addr learned my addr 172.21.15.42:0/4243918245 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:23:21.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:21.642+0000 7f47fa59c700 1 --2- 172.21.15.42:0/4243918245 >> v2:172.21.15.42:3300/0 conn(0x7f47fc07e720 0x7f47fc07eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:60834/0 (socket says 172.21.15.42:60834) 2019-12-05T05:23:21.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:21.642+0000 7f47fad9d700 1 --2- 172.21.15.42:0/4243918245 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f47fc0a7740 0x7f47fc08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:23:21.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:21.642+0000 7f47fa59c700 1 -- 172.21.15.42:0/4243918245 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f47fc0a7740 msgr2=0x7f47fc08df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:23:21.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:21.642+0000 7f47fa59c700 1 --2- 172.21.15.42:0/4243918245 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f47fc0a7740 0x7f47fc08df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:23:21.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:21.642+0000 7f47fa59c700 1 -- 172.21.15.42:0/4243918245 >> v1:172.21.15.42:6789/0 conn(0x7f47fc09f2e0 legacy=0x7f47fc09f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:21.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:21.642+0000 7f47f9d9b700 1 -- 172.21.15.42:0/4243918245 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2522838898 0 0) 0x7f47fc0a0710 con 0x7f47fc09f2e0 2019-12-05T05:23:21.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:21.642+0000 7f47fa59c700 1 -- 172.21.15.42:0/4243918245 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47fc07f0a0 con 0x7f47fc07e720 2019-12-05T05:23:21.650 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:21.642+0000 7f47fa59c700 1 --2- 172.21.15.42:0/4243918245 >> v2:172.21.15.42:3300/0 conn(0x7f47fc07e720 0x7f47fc07eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f47f0008790 tx=0x7f47f000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:21.650 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:21.642+0000 7f47fa59c700 1 --2- 172.21.15.42:0/4243918245 >> v2:172.21.15.42:3300/0 conn(0x7f47fc07e720 0x7f47fc07eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f47f0008790 tx=0x7f47f000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:21.650 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:21.642+0000 7f47fa59c700 1 --2- 172.21.15.42:0/4243918245 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f47fc07e720 0x7f47fc07eb50 secure :-1 s=READY pgs=758 cs=0 l=1 rx=0x7f47f0008790 tx=0x7f47f000fc70).ready entity=mon.1 client_cookie=5517bce4be872714 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:23:21.650 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:21.642+0000 7f47fa59c700 1 --2- 172.21.15.42:0/4243918245 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f47fc07e720 0x7f47fc07eb50 secure :-1 s=THROTTLE_DONE pgs=758 cs=0 l=1 rx=0x7f47f0008790 tx=0x7f47f000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:21.650 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:21.642+0000 7f47f9d9b700 1 -- 172.21.15.42:0/4243918245 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f47f00112f0 con 0x7f47fc07e720 2019-12-05T05:23:21.650 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:21.642+0000 7f47fa59c700 1 --2- 172.21.15.42:0/4243918245 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f47fc07e720 0x7f47fc07eb50 secure :-1 s=THROTTLE_DONE pgs=758 cs=0 l=1 rx=0x7f47f0008790 tx=0x7f47f000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:21.650 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:21.642+0000 7f47fa59c700 1 --2- 172.21.15.42:0/4243918245 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f47fc07e720 0x7f47fc07eb50 secure :-1 s=THROTTLE_DONE pgs=758 cs=0 l=1 rx=0x7f47f0008790 tx=0x7f47f000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:21.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:21.642+0000 7f47f9d9b700 1 -- 172.21.15.42:0/4243918245 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47f0008ad0 con 0x7f47fc07e720 2019-12-05T05:23:21.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:21.642+0000 7f47f9d9b700 1 -- 172.21.15.42:0/4243918245 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f47f00118d0 con 0x7f47fc07e720 2019-12-05T05:23:21.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:21.642+0000 7f4800813700 1 -- 172.21.15.42:0/4243918245 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f47fc07e720 msgr2=0x7f47fc07eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:21.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:21.642+0000 7f4800813700 1 --2- 172.21.15.42:0/4243918245 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f47fc07e720 0x7f47fc07eb50 secure :-1 s=READY pgs=758 cs=0 l=1 rx=0x7f47f0008790 tx=0x7f47f000fc70).stop 2019-12-05T05:23:21.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:21.642+0000 7f4800813700 1 -- 172.21.15.42:0/4243918245 shutdown_connections 2019-12-05T05:23:21.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:21.642+0000 7f4800813700 1 --2- 172.21.15.42:0/4243918245 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f47fc0a7740 0x7f47fc08df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:23:21.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:21.642+0000 7f4800813700 1 --2- 172.21.15.42:0/4243918245 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f47fc07e720 0x7f47fc07eb50 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:23:21.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:21.642+0000 7f4800813700 1 -- 172.21.15.42:0/4243918245 shutdown_connections 2019-12-05T05:23:21.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:21.642+0000 7f4800813700 1 -- 172.21.15.42:0/4243918245 wait complete. 2019-12-05T05:23:21.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:21.642+0000 7f4800813700 1 -- 172.21.15.42:0/4243918245 >> 172.21.15.42:0/4243918245 conn(0x7f47fc0aabe0 msgr2=0x7f47fc0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:23:21.656 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:21.646+0000 7f4800813700 1 Processor -- start 2019-12-05T05:23:21.656 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:21.646+0000 7f4800813700 1 -- start start 2019-12-05T05:23:21.656 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:21.646+0000 7f4800813700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f47fc122070 0x7f47fc11abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:23:21.656 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:21.646+0000 7f4800813700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f47fc1336f0 0x7f47fc135b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:23:21.656 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:21.646+0000 7f4800813700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47fc11a710 con 0x7f47fc12f080 2019-12-05T05:23:21.657 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:21.646+0000 7f4800813700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47fc119cf0 con 0x7f47fc1336f0 2019-12-05T05:23:21.657 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:21.646+0000 7f4800813700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47fc118bc0 con 0x7f47fc122070 2019-12-05T05:23:21.657 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:21.646+0000 7f47fb59e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f47fc1336f0 0x7f47fc135b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:23:21.657 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:21.646+0000 7f47fad9d700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f47fc12f080 0x7f47fc1194f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:60874/0 (socket says 172.21.15.42:60874) 2019-12-05T05:23:21.657 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:21.646+0000 7f47fad9d700 1 -- 172.21.15.42:0/4143813742 learned_addr learned my addr 172.21.15.42:0/4143813742 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:23:21.657 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:21.646+0000 7f47fa59c700 1 --2- 172.21.15.42:0/4143813742 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f47fc122070 0x7f47fc11abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:23:21.657 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:21.646+0000 7f47eb7fe700 1 -- 172.21.15.42:0/4143813742 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3020472246 0 0) 0x7f47fc11a710 con 0x7f47fc12f080 2019-12-05T05:23:21.658 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:21.646+0000 7f47eb7fe700 1 -- 172.21.15.42:0/4143813742 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47e00030b0 con 0x7f47fc12f080 2019-12-05T05:23:21.658 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:21.646+0000 7f47fb59e700 1 -- 172.21.15.42:0/4143813742 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f47fc122070 msgr2=0x7f47fc11abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:21.658 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:21.646+0000 7f47fb59e700 1 --2- 172.21.15.42:0/4143813742 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f47fc122070 0x7f47fc11abe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:23:21.658 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:21.646+0000 7f47fb59e700 1 -- 172.21.15.42:0/4143813742 >> v1:172.21.15.42:6789/0 conn(0x7f47fc12f080 legacy=0x7f47fc1194f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:21.658 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:21.646+0000 7f47fb59e700 1 -- 172.21.15.42:0/4143813742 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47fc11b130 con 0x7f47fc1336f0 2019-12-05T05:23:21.658 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:21.646+0000 7f47fb59e700 1 --2- 172.21.15.42:0/4143813742 >> v2:172.21.15.42:3300/0 conn(0x7f47fc1336f0 0x7f47fc135b30 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f47f4004720 tx=0x7f47f4010be0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:21.658 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:21.646+0000 7f47fb59e700 1 --2- 172.21.15.42:0/4143813742 >> v2:172.21.15.42:3300/0 conn(0x7f47fc1336f0 0x7f47fc135b30 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f47f4004720 tx=0x7f47f4010be0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:21.659 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:21.646+0000 7f47fb59e700 1 --2- 172.21.15.42:0/4143813742 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f47fc1336f0 0x7f47fc135b30 secure :-1 s=READY pgs=760 cs=0 l=1 rx=0x7f47f4004720 tx=0x7f47f4010be0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:23:21.659 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:21.646+0000 7f47fb59e700 1 --2- 172.21.15.42:0/4143813742 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f47fc1336f0 0x7f47fc135b30 secure :-1 s=THROTTLE_DONE pgs=760 cs=0 l=1 rx=0x7f47f4004720 tx=0x7f47f4010be0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:21.659 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:21.646+0000 7f47eb7fe700 1 -- 172.21.15.42:0/4143813742 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f47f4004b90 con 0x7f47fc1336f0 2019-12-05T05:23:21.659 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:21.646+0000 7f47fb59e700 1 --2- 172.21.15.42:0/4143813742 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f47fc1336f0 0x7f47fc135b30 secure :-1 s=THROTTLE_DONE pgs=760 cs=0 l=1 rx=0x7f47f4004720 tx=0x7f47f4010be0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:21.659 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:21.646+0000 7f47eb7fe700 1 -- 172.21.15.42:0/4143813742 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47f40047b0 con 0x7f47fc1336f0 2019-12-05T05:23:21.659 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:21.646+0000 7f47fb59e700 1 --2- 172.21.15.42:0/4143813742 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f47fc1336f0 0x7f47fc135b30 secure :-1 s=THROTTLE_DONE pgs=760 cs=0 l=1 rx=0x7f47f4004720 tx=0x7f47f4010be0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:21.659 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:21.646+0000 7f4800813700 1 -- 172.21.15.42:0/4143813742 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47fc1224a0 con 0x7f47fc1336f0 2019-12-05T05:23:21.660 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:21.646+0000 7f47eb7fe700 1 -- 172.21.15.42:0/4143813742 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f47f4012790 con 0x7f47fc1336f0 2019-12-05T05:23:21.660 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:21.646+0000 7f4800813700 1 -- 172.21.15.42:0/4143813742 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47fc08f800 con 0x7f47fc1336f0 2019-12-05T05:23:21.660 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:21.650+0000 7f47fb59e700 1 --2- 172.21.15.42:0/4143813742 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f47fc1336f0 0x7f47fc135b30 secure :-1 s=THROTTLE_DONE pgs=760 cs=0 l=1 rx=0x7f47f4004720 tx=0x7f47f4010be0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:21.660 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:21.650+0000 7f47eb7fe700 1 -- 172.21.15.42:0/4143813742 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f47f4012ab0 con 0x7f47fc1336f0 2019-12-05T05:23:21.660 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:21.650+0000 7f47fb59e700 1 --2- 172.21.15.42:0/4143813742 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f47fc1336f0 0x7f47fc135b30 secure :-1 s=THROTTLE_DONE pgs=760 cs=0 l=1 rx=0x7f47f4004720 tx=0x7f47f4010be0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:21.660 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:21.650+0000 7f47eb7fe700 1 --2- 172.21.15.42:0/4143813742 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f47e003b220 0x7f47e003d730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:23:21.660 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:21.650+0000 7f47eb7fe700 1 -- 172.21.15.42:0/4143813742 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 7149+0+0 (secure 0 0 0) 0x7f47f404b450 con 0x7f47fc1336f0 2019-12-05T05:23:21.661 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:21.650+0000 7f47fad9d700 1 --2- 172.21.15.42:0/4143813742 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f47e003b220 0x7f47e003d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:23:21.661 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:21.650+0000 7f47fad9d700 1 --2- 172.21.15.42:0/4143813742 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f47e003b220 0x7f47e003d730 crc :-1 s=READY pgs=336 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:23:21.661 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:21.654+0000 7f4800813700 1 -- 172.21.15.42:0/4143813742 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47fc05d110 con 0x7f47fc1336f0 2019-12-05T05:23:21.662 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:21.658+0000 7f47fb59e700 1 --2- 172.21.15.42:0/4143813742 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f47fc1336f0 0x7f47fc135b30 secure :-1 s=THROTTLE_DONE pgs=760 cs=0 l=1 rx=0x7f47f4004720 tx=0x7f47f4010be0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:21.662 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:21.658+0000 7f47eb7fe700 1 -- 172.21.15.42:0/4143813742 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f47f40324b0 con 0x7f47fc1336f0 2019-12-05T05:23:21.842 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:21.838+0000 7f4800813700 1 -- 172.21.15.42:0/4143813742 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f47fc05b8c0 con 0x7f47fc1336f0 2019-12-05T05:23:21.843 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:21.838+0000 7f47fb59e700 1 --2- 172.21.15.42:0/4143813742 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f47fc1336f0 0x7f47fc135b30 secure :-1 s=THROTTLE_DONE pgs=760 cs=0 l=1 rx=0x7f47f4004720 tx=0x7f47f4010be0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:21.844 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:21.838+0000 7f47eb7fe700 1 -- 172.21.15.42:0/4143813742 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 98 v98) v1 ==== 94+0+2108 (secure 0 0 0) 0x7f47f4032710 con 0x7f47fc1336f0 2019-12-05T05:23:21.844 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 98 2019-12-05T05:23:21.844 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:23:21.844 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":98,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5177,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/3208416000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3208416000},{"type":"v1","addr":"172.21.15.40:6833","nonce":3208416000}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":98}],"filesystems":[{"mdsmap":{"epoch":97,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:23:07.988779+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5123},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5123":{"gid":5123,"name":"a","rank":0,"incarnation":94,"state":"up:active","state_seq":37,"addr":"172.21.15.42:6835/1214659805","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1214659805},{"type":"v1","addr":"172.21.15.42:6835","nonce":1214659805}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:23:21.846 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:21.842+0000 7f4800813700 1 -- 172.21.15.42:0/4143813742 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f47e003b220 msgr2=0x7f47e003d730 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:21.846 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:21.842+0000 7f4800813700 1 --2- 172.21.15.42:0/4143813742 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f47e003b220 0x7f47e003d730 crc :-1 s=READY pgs=336 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:23:21.846 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:21.842+0000 7f4800813700 1 -- 172.21.15.42:0/4143813742 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f47fc1336f0 msgr2=0x7f47fc135b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:21.846 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:21.842+0000 7f4800813700 1 --2- 172.21.15.42:0/4143813742 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f47fc1336f0 0x7f47fc135b30 secure :-1 s=READY pgs=760 cs=0 l=1 rx=0x7f47f4004720 tx=0x7f47f4010be0).stop 2019-12-05T05:23:21.846 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:21.842+0000 7f4800813700 1 -- 172.21.15.42:0/4143813742 shutdown_connections 2019-12-05T05:23:21.847 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:21.842+0000 7f4800813700 1 --2- 172.21.15.42:0/4143813742 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f47e003b220 0x7f47e003d730 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:23:21.847 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:21.842+0000 7f4800813700 1 --2- 172.21.15.42:0/4143813742 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f47fc1336f0 0x7f47fc135b30 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:23:21.847 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:21.842+0000 7f4800813700 1 --2- 172.21.15.42:0/4143813742 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f47fc122070 0x7f47fc11abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:23:21.847 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:21.842+0000 7f4800813700 1 -- 172.21.15.42:0/4143813742 shutdown_connections 2019-12-05T05:23:21.847 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:21.842+0000 7f4800813700 1 -- 172.21.15.42:0/4143813742 wait complete. 2019-12-05T05:23:21.847 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:21.842+0000 7f4800813700 1 -- 172.21.15.42:0/4143813742 >> 172.21.15.42:0/4143813742 conn(0x7f47fc0a82a0 msgr2=0x7f47fc09f380 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:23:21.856 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b reported in up:standby state 2019-12-05T05:23:21.856 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 13 secs before thrashing 2019-12-05T05:23:34.857 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:23:34.965 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:34.954+0000 7f4f8a0a5700 1 Processor -- start 2019-12-05T05:23:34.969 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:34.954+0000 7f4f8a0a5700 1 -- start start 2019-12-05T05:23:34.969 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:34.958+0000 7f4f8a0a5700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f4f840a7740 0x7f4f8408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:23:34.969 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:34.958+0000 7f4f8a0a5700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f840a0710 con 0x7f4f8409fae0 2019-12-05T05:23:34.969 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:34.958+0000 7f4f8a0a5700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f840a9ba0 con 0x7f4f8407e720 2019-12-05T05:23:34.969 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:34.958+0000 7f4f8a0a5700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f840a0b00 con 0x7f4f840a7740 2019-12-05T05:23:34.969 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:34.958+0000 7f4f837fe700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f4f8407e720 0x7f4f8409f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:60934/0 (socket says 172.21.15.42:60934) 2019-12-05T05:23:34.970 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:34.958+0000 7f4f837fe700 1 -- 172.21.15.42:0/1392812596 learned_addr learned my addr 172.21.15.42:0/1392812596 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:23:34.970 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:34.958+0000 7f4f83fff700 1 --2- 172.21.15.42:0/1392812596 >> v2:172.21.15.40:3300/0 conn(0x7f4f840a7740 0x7f4f8408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:23:34.970 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:34.958+0000 7f4f82ffd700 1 -- 172.21.15.42:0/1392812596 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1564205739 0 0) 0x7f4f840a9ba0 con 0x7f4f8407e720 2019-12-05T05:23:34.970 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:34.958+0000 7f4f82ffd700 1 -- 172.21.15.42:0/1392812596 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f700030b0 con 0x7f4f8407e720 2019-12-05T05:23:34.970 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:34.958+0000 7f4f82ffd700 1 -- 172.21.15.42:0/1392812596 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3725356614 0 0) 0x7f4f840a0710 con 0x7f4f8409fae0 2019-12-05T05:23:34.970 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:34.958+0000 7f4f82ffd700 1 -- 172.21.15.42:0/1392812596 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f840a9ba0 con 0x7f4f8409fae0 2019-12-05T05:23:34.971 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:34.958+0000 7f4f83fff700 1 -- 172.21.15.42:0/1392812596 >> v1:172.21.15.42:6789/0 conn(0x7f4f8407e720 legacy=0x7f4f8409f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:34.971 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:34.958+0000 7f4f83fff700 1 -- 172.21.15.42:0/1392812596 >> v1:172.21.15.40:6789/0 conn(0x7f4f8409fae0 legacy=0x7f4f8414efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:34.971 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:34.958+0000 7f4f83fff700 1 -- 172.21.15.42:0/1392812596 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f8407eb50 con 0x7f4f840a7740 2019-12-05T05:23:34.971 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:34.958+0000 7f4f83fff700 1 --2- 172.21.15.42:0/1392812596 >> v2:172.21.15.40:3300/0 conn(0x7f4f840a7740 0x7f4f8408df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f4f74008b10 tx=0x7f4f7400ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:34.971 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:34.958+0000 7f4f83fff700 1 --2- 172.21.15.42:0/1392812596 >> v2:172.21.15.40:3300/0 conn(0x7f4f840a7740 0x7f4f8408df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f4f74008b10 tx=0x7f4f7400ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:34.971 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:34.958+0000 7f4f83fff700 1 --2- 172.21.15.42:0/1392812596 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4f840a7740 0x7f4f8408df80 secure :-1 s=READY pgs=608 cs=0 l=1 rx=0x7f4f74008b10 tx=0x7f4f7400ff70).ready entity=mon.0 client_cookie=abb318c0477224cf server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:23:34.972 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:34.958+0000 7f4f83fff700 1 --2- 172.21.15.42:0/1392812596 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4f840a7740 0x7f4f8408df80 secure :-1 s=THROTTLE_DONE pgs=608 cs=0 l=1 rx=0x7f4f74008b10 tx=0x7f4f7400ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:34.972 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:34.958+0000 7f4f83fff700 1 --2- 172.21.15.42:0/1392812596 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4f840a7740 0x7f4f8408df80 secure :-1 s=THROTTLE_DONE pgs=608 cs=0 l=1 rx=0x7f4f74008b10 tx=0x7f4f7400ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:34.972 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:34.958+0000 7f4f82ffd700 1 -- 172.21.15.42:0/1392812596 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4f740119d0 con 0x7f4f840a7740 2019-12-05T05:23:34.972 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:34.958+0000 7f4f83fff700 1 --2- 172.21.15.42:0/1392812596 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4f840a7740 0x7f4f8408df80 secure :-1 s=THROTTLE_DONE pgs=608 cs=0 l=1 rx=0x7f4f74008b10 tx=0x7f4f7400ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:34.972 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:34.958+0000 7f4f82ffd700 1 -- 172.21.15.42:0/1392812596 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f740115f0 con 0x7f4f840a7740 2019-12-05T05:23:34.972 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:34.958+0000 7f4f82ffd700 1 -- 172.21.15.42:0/1392812596 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4f7400a160 con 0x7f4f840a7740 2019-12-05T05:23:34.973 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:34.958+0000 7f4f8a0a5700 1 -- 172.21.15.42:0/1392812596 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4f840a7740 msgr2=0x7f4f8408df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:34.973 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:34.958+0000 7f4f8a0a5700 1 --2- 172.21.15.42:0/1392812596 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4f840a7740 0x7f4f8408df80 secure :-1 s=READY pgs=608 cs=0 l=1 rx=0x7f4f74008b10 tx=0x7f4f7400ff70).stop 2019-12-05T05:23:34.973 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:34.958+0000 7f4f8a0a5700 1 -- 172.21.15.42:0/1392812596 shutdown_connections 2019-12-05T05:23:34.973 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:34.958+0000 7f4f8a0a5700 1 --2- 172.21.15.42:0/1392812596 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4f840a7740 0x7f4f8408df80 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:23:34.973 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:34.958+0000 7f4f8a0a5700 1 -- 172.21.15.42:0/1392812596 shutdown_connections 2019-12-05T05:23:34.973 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:34.958+0000 7f4f8a0a5700 1 -- 172.21.15.42:0/1392812596 wait complete. 2019-12-05T05:23:34.973 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:34.958+0000 7f4f8a0a5700 1 -- 172.21.15.42:0/1392812596 >> 172.21.15.42:0/1392812596 conn(0x7f4f840aabe0 msgr2=0x7f4f840a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:23:34.981 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:34.962+0000 7f4f8a0a5700 1 Processor -- start 2019-12-05T05:23:34.982 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:34.962+0000 7f4f8a0a5700 1 -- start start 2019-12-05T05:23:34.982 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:34.966+0000 7f4f8a0a5700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f4f8411c430 0x7f4f84133560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:23:34.982 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:34.966+0000 7f4f8a0a5700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f4f84116500 0x7f4f8411b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:23:34.982 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:34.966+0000 7f4f8a0a5700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f84119dc0 con 0x7f4f8411b820 2019-12-05T05:23:34.983 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:34.966+0000 7f4f8a0a5700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f8411bef0 con 0x7f4f84116500 2019-12-05T05:23:34.983 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:34.966+0000 7f4f8a0a5700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f8411c190 con 0x7f4f8411c430 2019-12-05T05:23:34.983 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:34.966+0000 7f4f83fff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f4f8411c430 0x7f4f84133560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:23:34.983 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:34.966+0000 7f4f83fff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f4f8411c430 0x7f4f84133560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:60902/0 (socket says 172.21.15.42:60902) 2019-12-05T05:23:34.983 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:34.966+0000 7f4f83fff700 1 -- 172.21.15.42:0/195559915 learned_addr learned my addr 172.21.15.42:0/195559915 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:23:34.984 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:34.966+0000 7f4f837fe700 1 --2- 172.21.15.42:0/195559915 >> v2:172.21.15.40:3300/0 conn(0x7f4f84116500 0x7f4f8411b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:23:34.984 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:34.966+0000 7f4f83fff700 1 -- 172.21.15.42:0/195559915 >> v2:172.21.15.40:3300/0 conn(0x7f4f84116500 msgr2=0x7f4f8411b2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:34.984 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:34.966+0000 7f4f83fff700 1 --2- 172.21.15.42:0/195559915 >> v2:172.21.15.40:3300/0 conn(0x7f4f84116500 0x7f4f8411b2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:23:34.984 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:34.966+0000 7f4f83fff700 1 -- 172.21.15.42:0/195559915 >> v1:172.21.15.40:6789/0 conn(0x7f4f8411b820 legacy=0x7f4f8408eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:34.985 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:34.966+0000 7f4f83fff700 1 -- 172.21.15.42:0/195559915 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f84133ab0 con 0x7f4f8411c430 2019-12-05T05:23:34.985 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:34.966+0000 7f4f83fff700 1 --2- 172.21.15.42:0/195559915 >> v2:172.21.15.42:3300/0 conn(0x7f4f8411c430 0x7f4f84133560 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f4f74008b10 tx=0x7f4f74011dc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:34.985 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:34.966+0000 7f4f83fff700 1 --2- 172.21.15.42:0/195559915 >> v2:172.21.15.42:3300/0 conn(0x7f4f8411c430 0x7f4f84133560 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f4f74008b10 tx=0x7f4f74011dc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:34.985 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:34.966+0000 7f4f83fff700 1 --2- 172.21.15.42:0/195559915 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4f8411c430 0x7f4f84133560 secure :-1 s=READY pgs=766 cs=0 l=1 rx=0x7f4f74008b10 tx=0x7f4f74011dc0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:23:34.985 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:34.966+0000 7f4f83fff700 1 --2- 172.21.15.42:0/195559915 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4f8411c430 0x7f4f84133560 secure :-1 s=THROTTLE_DONE pgs=766 cs=0 l=1 rx=0x7f4f74008b10 tx=0x7f4f74011dc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:34.986 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:34.966+0000 7f4f83fff700 1 --2- 172.21.15.42:0/195559915 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4f8411c430 0x7f4f84133560 secure :-1 s=THROTTLE_DONE pgs=766 cs=0 l=1 rx=0x7f4f74008b10 tx=0x7f4f74011dc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:34.986 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:34.966+0000 7f4f80ff9700 1 -- 172.21.15.42:0/195559915 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4f7400f780 con 0x7f4f8411c430 2019-12-05T05:23:34.986 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:34.966+0000 7f4f83fff700 1 --2- 172.21.15.42:0/195559915 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4f8411c430 0x7f4f84133560 secure :-1 s=THROTTLE_DONE pgs=766 cs=0 l=1 rx=0x7f4f74008b10 tx=0x7f4f74011dc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:34.986 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:34.966+0000 7f4f80ff9700 1 -- 172.21.15.42:0/195559915 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f74011bd0 con 0x7f4f8411c430 2019-12-05T05:23:34.986 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:34.966+0000 7f4f8a0a5700 1 -- 172.21.15.42:0/195559915 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f8408f370 con 0x7f4f8411c430 2019-12-05T05:23:34.987 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:34.966+0000 7f4f80ff9700 1 -- 172.21.15.42:0/195559915 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4f74005ab0 con 0x7f4f8411c430 2019-12-05T05:23:34.987 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:34.966+0000 7f4f8a0a5700 1 -- 172.21.15.42:0/195559915 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f8408f8f0 con 0x7f4f8411c430 2019-12-05T05:23:34.987 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:34.966+0000 7f4f83fff700 1 --2- 172.21.15.42:0/195559915 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4f8411c430 0x7f4f84133560 secure :-1 s=THROTTLE_DONE pgs=766 cs=0 l=1 rx=0x7f4f74008b10 tx=0x7f4f74011dc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:34.987 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:34.970+0000 7f4f80ff9700 1 -- 172.21.15.42:0/195559915 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f4f74007150 con 0x7f4f8411c430 2019-12-05T05:23:34.987 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:34.970+0000 7f4f83fff700 1 --2- 172.21.15.42:0/195559915 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4f8411c430 0x7f4f84133560 secure :-1 s=THROTTLE_DONE pgs=766 cs=0 l=1 rx=0x7f4f74008b10 tx=0x7f4f74011dc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:34.988 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:34.970+0000 7f4f80ff9700 1 --2- 172.21.15.42:0/195559915 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4f6803ad70 0x7f4f6803d280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:23:34.988 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:34.970+0000 7f4f80ff9700 1 -- 172.21.15.42:0/195559915 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 7149+0+0 (secure 0 0 0) 0x7f4f7404b470 con 0x7f4f8411c430 2019-12-05T05:23:34.988 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:34.970+0000 7f4f837fe700 1 --2- 172.21.15.42:0/195559915 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4f6803ad70 0x7f4f6803d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:23:34.988 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:34.970+0000 7f4f837fe700 1 --2- 172.21.15.42:0/195559915 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4f6803ad70 0x7f4f6803d280 crc :-1 s=READY pgs=337 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:23:34.988 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:34.970+0000 7f4f8a0a5700 1 -- 172.21.15.42:0/195559915 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f8405d110 con 0x7f4f8411c430 2019-12-05T05:23:34.988 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:34.974+0000 7f4f83fff700 1 --2- 172.21.15.42:0/195559915 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4f8411c430 0x7f4f84133560 secure :-1 s=THROTTLE_DONE pgs=766 cs=0 l=1 rx=0x7f4f74008b10 tx=0x7f4f74011dc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:34.989 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:34.978+0000 7f4f80ff9700 1 -- 172.21.15.42:0/195559915 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f4f74005dc0 con 0x7f4f8411c430 2019-12-05T05:23:35.142 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:35.138+0000 7f4f8a0a5700 1 -- 172.21.15.42:0/195559915 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4f8405b8c0 con 0x7f4f8411c430 2019-12-05T05:23:35.143 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:35.138+0000 7f4f83fff700 1 --2- 172.21.15.42:0/195559915 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4f8411c430 0x7f4f84133560 secure :-1 s=THROTTLE_DONE pgs=766 cs=0 l=1 rx=0x7f4f74008b10 tx=0x7f4f74011dc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:35.143 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:35.138+0000 7f4f80ff9700 1 -- 172.21.15.42:0/195559915 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 98 v98) v1 ==== 94+0+2108 (secure 0 0 0) 0x7f4f74032750 con 0x7f4f8411c430 2019-12-05T05:23:35.145 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:23:35.145 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":98,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5177,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/3208416000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3208416000},{"type":"v1","addr":"172.21.15.40:6833","nonce":3208416000}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":98}],"filesystems":[{"mdsmap":{"epoch":97,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:23:07.988779+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5123},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5123":{"gid":5123,"name":"a","rank":0,"incarnation":94,"state":"up:active","state_seq":37,"addr":"172.21.15.42:6835/1214659805","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1214659805},{"type":"v1","addr":"172.21.15.42:6835","nonce":1214659805}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:23:35.146 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 98 2019-12-05T05:23:35.148 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:35.142+0000 7f4f8a0a5700 1 -- 172.21.15.42:0/195559915 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4f6803ad70 msgr2=0x7f4f6803d280 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:35.148 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:35.142+0000 7f4f8a0a5700 1 --2- 172.21.15.42:0/195559915 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4f6803ad70 0x7f4f6803d280 crc :-1 s=READY pgs=337 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:23:35.149 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:35.142+0000 7f4f8a0a5700 1 -- 172.21.15.42:0/195559915 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4f8411c430 msgr2=0x7f4f84133560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:35.167 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:35.142+0000 7f4f8a0a5700 1 --2- 172.21.15.42:0/195559915 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4f8411c430 0x7f4f84133560 secure :-1 s=READY pgs=766 cs=0 l=1 rx=0x7f4f74008b10 tx=0x7f4f74011dc0).stop 2019-12-05T05:23:35.167 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:35.142+0000 7f4f8a0a5700 1 -- 172.21.15.42:0/195559915 shutdown_connections 2019-12-05T05:23:35.167 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:35.142+0000 7f4f8a0a5700 1 --2- 172.21.15.42:0/195559915 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4f6803ad70 0x7f4f6803d280 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:23:35.167 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:35.142+0000 7f4f8a0a5700 1 --2- 172.21.15.42:0/195559915 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4f8411c430 0x7f4f84133560 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:23:35.168 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:35.142+0000 7f4f8a0a5700 1 --2- 172.21.15.42:0/195559915 >> v2:172.21.15.40:3300/0 conn(0x7f4f84116500 0x7f4f8411b2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:23:35.168 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:35.146+0000 7f4f8a0a5700 1 -- 172.21.15.42:0/195559915 shutdown_connections 2019-12-05T05:23:35.168 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:35.146+0000 7f4f8a0a5700 1 -- 172.21.15.42:0/195559915 wait complete. 2019-12-05T05:23:35.168 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:35.146+0000 7f4f8a0a5700 1 -- 172.21.15.42:0/195559915 >> 172.21.15.42:0/195559915 conn(0x7f4f840a9770 msgr2=0x7f4f84059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:23:35.171 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.a (rank=0) 2019-12-05T05:23:35.171 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2019-12-05T05:23:35.172 INFO:teuthology.orchestra.run:waiting for 300 2019-12-05T05:23:35.186 INFO:tasks.ceph.mds.a:Stopped 2019-12-05T05:23:35.186 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:23:35.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:35.350+0000 7fd71a1c0700 1 Processor -- start 2019-12-05T05:23:35.358 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:35.350+0000 7fd71a1c0700 1 -- start start 2019-12-05T05:23:35.358 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:35.350+0000 7fd71a1c0700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd71408df80 0x7fd71409f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:23:35.358 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:35.350+0000 7fd71a1c0700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd71409f830 0x7fd71414efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:23:35.358 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:35.350+0000 7fd71a1c0700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd71409fc60 con 0x7fd7140a7740 2019-12-05T05:23:35.359 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:35.350+0000 7fd71a1c0700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7140a0b00 con 0x7fd71409f830 2019-12-05T05:23:35.359 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:35.350+0000 7fd71a1c0700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd71408e5b0 con 0x7fd71408df80 2019-12-05T05:23:35.359 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:35.350+0000 7fd713fff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fd7140a7740 0x7fd71407e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:60950/0 (socket says 172.21.15.42:60950) 2019-12-05T05:23:35.359 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:35.350+0000 7fd713fff700 1 -- 172.21.15.42:0/3355471716 learned_addr learned my addr 172.21.15.42:0/3355471716 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:23:35.359 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:35.350+0000 7fd7137fe700 1 --2- 172.21.15.42:0/3355471716 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd71408df80 0x7fd71409f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:23:35.359 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:35.350+0000 7fd718f5e700 1 --2- 172.21.15.42:0/3355471716 >> v2:172.21.15.40:3300/0 conn(0x7fd71409f830 0x7fd71414efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:23:35.360 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:35.350+0000 7fd712ffd700 1 -- 172.21.15.42:0/3355471716 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1526823407 0 0) 0x7fd71409fc60 con 0x7fd7140a7740 2019-12-05T05:23:35.360 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:35.350+0000 7fd712ffd700 1 -- 172.21.15.42:0/3355471716 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7000030b0 con 0x7fd7140a7740 2019-12-05T05:23:35.360 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:35.350+0000 7fd7137fe700 1 -- 172.21.15.42:0/3355471716 >> v2:172.21.15.40:3300/0 conn(0x7fd71409f830 msgr2=0x7fd71414efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:23:35.360 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:35.350+0000 7fd7137fe700 1 --2- 172.21.15.42:0/3355471716 >> v2:172.21.15.40:3300/0 conn(0x7fd71409f830 0x7fd71414efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:23:35.360 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:35.350+0000 7fd7137fe700 1 -- 172.21.15.42:0/3355471716 >> v1:172.21.15.42:6789/0 conn(0x7fd7140a7740 legacy=0x7fd71407e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:35.361 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:35.350+0000 7fd7137fe700 1 -- 172.21.15.42:0/3355471716 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7140a0710 con 0x7fd71408df80 2019-12-05T05:23:35.361 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:35.350+0000 7fd7137fe700 1 --2- 172.21.15.42:0/3355471716 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd71408df80 0x7fd71409f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fd6fc008790 tx=0x7fd6fc00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:35.361 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:35.350+0000 7fd7137fe700 1 --2- 172.21.15.42:0/3355471716 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd71408df80 0x7fd71409f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fd6fc008790 tx=0x7fd6fc00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:35.362 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:35.350+0000 7fd7137fe700 1 --2- 172.21.15.42:0/3355471716 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd71408df80 0x7fd71409f2e0 secure :-1 s=READY pgs=157 cs=0 l=1 rx=0x7fd6fc008790 tx=0x7fd6fc00fc70).ready entity=mon.2 client_cookie=660e530d54a559af server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:23:35.362 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:35.350+0000 7fd7137fe700 1 --2- 172.21.15.42:0/3355471716 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd71408df80 0x7fd71409f2e0 secure :-1 s=THROTTLE_DONE pgs=157 cs=0 l=1 rx=0x7fd6fc008790 tx=0x7fd6fc00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:35.362 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:35.350+0000 7fd712ffd700 1 -- 172.21.15.42:0/3355471716 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd6fc008bb0 con 0x7fd71408df80 2019-12-05T05:23:35.362 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:35.350+0000 7fd7137fe700 1 --2- 172.21.15.42:0/3355471716 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd71408df80 0x7fd71409f2e0 secure :-1 s=THROTTLE_DONE pgs=157 cs=0 l=1 rx=0x7fd6fc008790 tx=0x7fd6fc00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:35.363 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:35.350+0000 7fd7137fe700 1 --2- 172.21.15.42:0/3355471716 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd71408df80 0x7fd71409f2e0 secure :-1 s=THROTTLE_DONE pgs=157 cs=0 l=1 rx=0x7fd6fc008790 tx=0x7fd6fc00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:35.363 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:35.350+0000 7fd712ffd700 1 -- 172.21.15.42:0/3355471716 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6fc0087d0 con 0x7fd71408df80 2019-12-05T05:23:35.363 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:35.350+0000 7fd712ffd700 1 -- 172.21.15.42:0/3355471716 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd6fc011860 con 0x7fd71408df80 2019-12-05T05:23:35.363 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:35.350+0000 7fd71a1c0700 1 -- 172.21.15.42:0/3355471716 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd71408df80 msgr2=0x7fd71409f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:35.363 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:35.350+0000 7fd71a1c0700 1 --2- 172.21.15.42:0/3355471716 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd71408df80 0x7fd71409f2e0 secure :-1 s=READY pgs=157 cs=0 l=1 rx=0x7fd6fc008790 tx=0x7fd6fc00fc70).stop 2019-12-05T05:23:35.364 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:35.354+0000 7fd71a1c0700 1 -- 172.21.15.42:0/3355471716 shutdown_connections 2019-12-05T05:23:35.364 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:35.354+0000 7fd71a1c0700 1 --2- 172.21.15.42:0/3355471716 >> v2:172.21.15.40:3300/0 conn(0x7fd71409f830 0x7fd71414efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:23:35.364 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:35.354+0000 7fd71a1c0700 1 --2- 172.21.15.42:0/3355471716 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd71408df80 0x7fd71409f2e0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:23:35.364 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:35.354+0000 7fd71a1c0700 1 -- 172.21.15.42:0/3355471716 shutdown_connections 2019-12-05T05:23:35.364 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:35.354+0000 7fd71a1c0700 1 -- 172.21.15.42:0/3355471716 wait complete. 2019-12-05T05:23:35.365 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:35.354+0000 7fd71a1c0700 1 -- 172.21.15.42:0/3355471716 >> 172.21.15.42:0/3355471716 conn(0x7fd7140aabe0 msgr2=0x7fd7140a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:23:35.368 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:35.354+0000 7fd71a1c0700 1 Processor -- start 2019-12-05T05:23:35.368 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:35.354+0000 7fd71a1c0700 1 -- start start 2019-12-05T05:23:35.369 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:35.358+0000 7fd71a1c0700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd714120cf0 0x7fd71411b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:23:35.369 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:35.358+0000 7fd71a1c0700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd71411dac0 con 0x7fd71411e5f0 2019-12-05T05:23:35.369 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:35.358+0000 7fd71a1c0700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd71411c260 con 0x7fd71411be30 2019-12-05T05:23:35.369 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:35.358+0000 7fd71a1c0700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd71411ffb0 con 0x7fd714120cf0 2019-12-05T05:23:35.369 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:35.358+0000 7fd718f5e700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fd71411be30 0x7fd714137620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:60960/0 (socket says 172.21.15.42:60960) 2019-12-05T05:23:35.370 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:35.358+0000 7fd718f5e700 1 -- 172.21.15.42:0/2184500175 learned_addr learned my addr 172.21.15.42:0/2184500175 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:23:35.370 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:35.358+0000 7fd7137fe700 1 --2- 172.21.15.42:0/2184500175 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd714120cf0 0x7fd71411b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:23:35.370 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:35.358+0000 7fd710ff9700 1 -- 172.21.15.42:0/2184500175 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3433532402 0 0) 0x7fd71411c260 con 0x7fd71411be30 2019-12-05T05:23:35.370 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:35.358+0000 7fd710ff9700 1 -- 172.21.15.42:0/2184500175 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6f80030b0 con 0x7fd71411be30 2019-12-05T05:23:35.370 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:35.358+0000 7fd710ff9700 1 -- 172.21.15.42:0/2184500175 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 849602075 0 0) 0x7fd71411dac0 con 0x7fd71411e5f0 2019-12-05T05:23:35.370 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:35.358+0000 7fd710ff9700 1 -- 172.21.15.42:0/2184500175 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd71411c260 con 0x7fd71411e5f0 2019-12-05T05:23:35.370 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:35.358+0000 7fd7137fe700 1 -- 172.21.15.42:0/2184500175 >> v1:172.21.15.42:6789/0 conn(0x7fd71411be30 legacy=0x7fd714137620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:35.371 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:35.358+0000 7fd7137fe700 1 -- 172.21.15.42:0/2184500175 >> v1:172.21.15.40:6789/0 conn(0x7fd71411e5f0 legacy=0x7fd71407d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:35.371 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:35.358+0000 7fd7137fe700 1 -- 172.21.15.42:0/2184500175 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd71411fbf0 con 0x7fd714120cf0 2019-12-05T05:23:35.371 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:35.358+0000 7fd7137fe700 1 --2- 172.21.15.42:0/2184500175 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd714120cf0 0x7fd71411b8e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fd6fc008790 tx=0x7fd6fc0118a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:35.371 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:35.358+0000 7fd7137fe700 1 --2- 172.21.15.42:0/2184500175 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd714120cf0 0x7fd71411b8e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fd6fc008790 tx=0x7fd6fc0118a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:35.371 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:35.358+0000 7fd7137fe700 1 --2- 172.21.15.42:0/2184500175 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd714120cf0 0x7fd71411b8e0 secure :-1 s=READY pgs=158 cs=0 l=1 rx=0x7fd6fc008790 tx=0x7fd6fc0118a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:23:35.372 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:35.358+0000 7fd7137fe700 1 --2- 172.21.15.42:0/2184500175 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd714120cf0 0x7fd71411b8e0 secure :-1 s=THROTTLE_DONE pgs=158 cs=0 l=1 rx=0x7fd6fc008790 tx=0x7fd6fc0118a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:35.372 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:35.358+0000 7fd710ff9700 1 -- 172.21.15.42:0/2184500175 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd6fc0087d0 con 0x7fd714120cf0 2019-12-05T05:23:35.372 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:35.358+0000 7fd7137fe700 1 --2- 172.21.15.42:0/2184500175 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd714120cf0 0x7fd71411b8e0 secure :-1 s=THROTTLE_DONE pgs=158 cs=0 l=1 rx=0x7fd6fc008790 tx=0x7fd6fc0118a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:35.372 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:35.358+0000 7fd7137fe700 1 --2- 172.21.15.42:0/2184500175 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd714120cf0 0x7fd71411b8e0 secure :-1 s=THROTTLE_DONE pgs=158 cs=0 l=1 rx=0x7fd6fc008790 tx=0x7fd6fc0118a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:35.372 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:35.358+0000 7fd710ff9700 1 -- 172.21.15.42:0/2184500175 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6fc008db0 con 0x7fd714120cf0 2019-12-05T05:23:35.372 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:35.358+0000 7fd71a1c0700 1 -- 172.21.15.42:0/2184500175 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd714138e30 con 0x7fd714120cf0 2019-12-05T05:23:35.373 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:35.358+0000 7fd710ff9700 1 -- 172.21.15.42:0/2184500175 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd6fc005540 con 0x7fd714120cf0 2019-12-05T05:23:35.373 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:35.358+0000 7fd71a1c0700 1 -- 172.21.15.42:0/2184500175 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7141393e0 con 0x7fd714120cf0 2019-12-05T05:23:35.373 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:35.358+0000 7fd7137fe700 1 --2- 172.21.15.42:0/2184500175 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd714120cf0 0x7fd71411b8e0 secure :-1 s=THROTTLE_DONE pgs=158 cs=0 l=1 rx=0x7fd6fc008790 tx=0x7fd6fc0118a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:35.373 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:35.362+0000 7fd710ff9700 1 -- 172.21.15.42:0/2184500175 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fd6fc005790 con 0x7fd714120cf0 2019-12-05T05:23:35.373 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:35.362+0000 7fd7137fe700 1 --2- 172.21.15.42:0/2184500175 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd714120cf0 0x7fd71411b8e0 secure :-1 s=THROTTLE_DONE pgs=158 cs=0 l=1 rx=0x7fd6fc008790 tx=0x7fd6fc0118a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:35.373 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:35.362+0000 7fd710ff9700 1 --2- 172.21.15.42:0/2184500175 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd6f803b4b0 0x7fd6f803d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:23:35.373 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:35.362+0000 7fd710ff9700 1 -- 172.21.15.42:0/2184500175 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 7149+0+0 (secure 0 0 0) 0x7fd6fc04b4f0 con 0x7fd714120cf0 2019-12-05T05:23:35.374 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:35.362+0000 7fd713fff700 1 --2- 172.21.15.42:0/2184500175 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd6f803b4b0 0x7fd6f803d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:23:35.374 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:35.362+0000 7fd713fff700 1 --2- 172.21.15.42:0/2184500175 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd6f803b4b0 0x7fd6f803d9c0 crc :-1 s=READY pgs=338 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:23:35.374 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:35.366+0000 7fd71a1c0700 1 -- 172.21.15.42:0/2184500175 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd71405d110 con 0x7fd714120cf0 2019-12-05T05:23:35.378 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:35.374+0000 7fd7137fe700 1 --2- 172.21.15.42:0/2184500175 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd714120cf0 0x7fd71411b8e0 secure :-1 s=THROTTLE_DONE pgs=158 cs=0 l=1 rx=0x7fd6fc008790 tx=0x7fd6fc0118a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:35.378 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:35.374+0000 7fd710ff9700 1 -- 172.21.15.42:0/2184500175 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fd6fc0327d0 con 0x7fd714120cf0 2019-12-05T05:23:35.546 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:35.542+0000 7fd71a1c0700 1 -- 172.21.15.42:0/2184500175 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd71405b8c0 con 0x7fd714120cf0 2019-12-05T05:23:35.547 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:35.542+0000 7fd7137fe700 1 --2- 172.21.15.42:0/2184500175 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd714120cf0 0x7fd71411b8e0 secure :-1 s=THROTTLE_DONE pgs=158 cs=0 l=1 rx=0x7fd6fc008790 tx=0x7fd6fc0118a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:35.547 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:35.542+0000 7fd710ff9700 1 -- 172.21.15.42:0/2184500175 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 98 v98) v1 ==== 94+0+2108 (secure 0 0 0) 0x7fd6fc033380 con 0x7fd714120cf0 2019-12-05T05:23:35.549 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:23:35.549 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":98,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5177,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/3208416000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3208416000},{"type":"v1","addr":"172.21.15.40:6833","nonce":3208416000}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":98}],"filesystems":[{"mdsmap":{"epoch":97,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:23:07.988779+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5123},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5123":{"gid":5123,"name":"a","rank":0,"incarnation":94,"state":"up:active","state_seq":37,"addr":"172.21.15.42:6835/1214659805","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1214659805},{"type":"v1","addr":"172.21.15.42:6835","nonce":1214659805}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:23:35.550 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 98 2019-12-05T05:23:35.552 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:35.546+0000 7fd71a1c0700 1 -- 172.21.15.42:0/2184500175 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd6f803b4b0 msgr2=0x7fd6f803d9c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:35.552 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:35.546+0000 7fd71a1c0700 1 --2- 172.21.15.42:0/2184500175 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd6f803b4b0 0x7fd6f803d9c0 crc :-1 s=READY pgs=338 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:23:35.552 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:35.546+0000 7fd71a1c0700 1 -- 172.21.15.42:0/2184500175 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd714120cf0 msgr2=0x7fd71411b8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:35.553 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:35.546+0000 7fd71a1c0700 1 --2- 172.21.15.42:0/2184500175 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd714120cf0 0x7fd71411b8e0 secure :-1 s=READY pgs=158 cs=0 l=1 rx=0x7fd6fc008790 tx=0x7fd6fc0118a0).stop 2019-12-05T05:23:35.553 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:35.550+0000 7fd71a1c0700 1 -- 172.21.15.42:0/2184500175 shutdown_connections 2019-12-05T05:23:35.553 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:35.550+0000 7fd71a1c0700 1 --2- 172.21.15.42:0/2184500175 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd6f803b4b0 0x7fd6f803d9c0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:23:35.553 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:35.550+0000 7fd71a1c0700 1 --2- 172.21.15.42:0/2184500175 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd714120cf0 0x7fd71411b8e0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:23:35.553 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:35.550+0000 7fd71a1c0700 1 -- 172.21.15.42:0/2184500175 shutdown_connections 2019-12-05T05:23:35.553 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:35.550+0000 7fd71a1c0700 1 -- 172.21.15.42:0/2184500175 wait complete. 2019-12-05T05:23:35.553 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:35.550+0000 7fd71a1c0700 1 -- 172.21.15.42:0/2184500175 >> 172.21.15.42:0/2184500175 conn(0x7fd7140a6d10 msgr2=0x7fd71411c610 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:23:35.563 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T05:23:37.564 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:23:37.670 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:37.662+0000 7f9422194700 1 Processor -- start 2019-12-05T05:23:37.670 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:37.662+0000 7f9422194700 1 -- start start 2019-12-05T05:23:37.670 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:37.662+0000 7f9422194700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f941c0a7740 0x7f941c08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:23:37.670 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:37.662+0000 7f9422194700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f941c09fae0 0x7f941c07eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:23:37.671 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:37.662+0000 7f9422194700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f941c0a0710 con 0x7f941c07e720 2019-12-05T05:23:37.671 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:37.662+0000 7f9422194700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f941c0a0b00 con 0x7f941c09fae0 2019-12-05T05:23:37.671 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:37.662+0000 7f9422194700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f941c08e5b0 con 0x7f941c0a7740 2019-12-05T05:23:37.671 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:37.662+0000 7f941bfff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f941c0a7740 0x7f941c08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:23:37.671 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:37.662+0000 7f9420f32700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f941c09fae0 0x7f941c07eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:23:37.671 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:37.662+0000 7f941bfff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f941c0a7740 0x7f941c08df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:60936/0 (socket says 172.21.15.42:60936) 2019-12-05T05:23:37.672 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:37.662+0000 7f941bfff700 1 -- 172.21.15.42:0/1600980690 learned_addr learned my addr 172.21.15.42:0/1600980690 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:23:37.672 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:37.662+0000 7f941bfff700 1 -- 172.21.15.42:0/1600980690 >> v2:172.21.15.40:3300/0 conn(0x7f941c09fae0 msgr2=0x7f941c07eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:23:37.672 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:37.662+0000 7f941bfff700 1 --2- 172.21.15.42:0/1600980690 >> v2:172.21.15.40:3300/0 conn(0x7f941c09fae0 0x7f941c07eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:23:37.672 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:37.662+0000 7f941bfff700 1 -- 172.21.15.42:0/1600980690 >> v1:172.21.15.40:6789/0 conn(0x7f941c07e720 legacy=0x7f941c09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:37.672 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:37.662+0000 7f941bfff700 1 -- 172.21.15.42:0/1600980690 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f941c07f0a0 con 0x7f941c0a7740 2019-12-05T05:23:37.672 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:37.662+0000 7f941bfff700 1 --2- 172.21.15.42:0/1600980690 >> v2:172.21.15.42:3300/0 conn(0x7f941c0a7740 0x7f941c08df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f940c002f30 tx=0x7f940c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:37.672 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:37.662+0000 7f941bfff700 1 --2- 172.21.15.42:0/1600980690 >> v2:172.21.15.42:3300/0 conn(0x7f941c0a7740 0x7f941c08df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f940c002f30 tx=0x7f940c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:37.673 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:37.662+0000 7f941bfff700 1 --2- 172.21.15.42:0/1600980690 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f941c0a7740 0x7f941c08df80 secure :-1 s=READY pgs=769 cs=0 l=1 rx=0x7f940c002f30 tx=0x7f940c00fd30).ready entity=mon.1 client_cookie=d7e6958ee732d649 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:23:37.673 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:37.666+0000 7f941bfff700 1 --2- 172.21.15.42:0/1600980690 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f941c0a7740 0x7f941c08df80 secure :-1 s=THROTTLE_DONE pgs=769 cs=0 l=1 rx=0x7f940c002f30 tx=0x7f940c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:37.673 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:37.666+0000 7f941bfff700 1 --2- 172.21.15.42:0/1600980690 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f941c0a7740 0x7f941c08df80 secure :-1 s=THROTTLE_DONE pgs=769 cs=0 l=1 rx=0x7f940c002f30 tx=0x7f940c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:37.673 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:37.666+0000 7f941affd700 1 -- 172.21.15.42:0/1600980690 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f940c008c90 con 0x7f941c0a7740 2019-12-05T05:23:37.673 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:37.666+0000 7f941bfff700 1 --2- 172.21.15.42:0/1600980690 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f941c0a7740 0x7f941c08df80 secure :-1 s=THROTTLE_DONE pgs=769 cs=0 l=1 rx=0x7f940c002f30 tx=0x7f940c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:37.673 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:37.666+0000 7f941affd700 1 -- 172.21.15.42:0/1600980690 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f940c0088b0 con 0x7f941c0a7740 2019-12-05T05:23:37.674 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:37.666+0000 7f941affd700 1 -- 172.21.15.42:0/1600980690 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f940c011ad0 con 0x7f941c0a7740 2019-12-05T05:23:37.674 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:37.666+0000 7f9422194700 1 -- 172.21.15.42:0/1600980690 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f941c0a7740 msgr2=0x7f941c08df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:37.674 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:37.666+0000 7f9422194700 1 --2- 172.21.15.42:0/1600980690 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f941c0a7740 0x7f941c08df80 secure :-1 s=READY pgs=769 cs=0 l=1 rx=0x7f940c002f30 tx=0x7f940c00fd30).stop 2019-12-05T05:23:37.674 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:37.666+0000 7f9422194700 1 -- 172.21.15.42:0/1600980690 shutdown_connections 2019-12-05T05:23:37.674 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:37.666+0000 7f9422194700 1 --2- 172.21.15.42:0/1600980690 >> v2:172.21.15.40:3300/0 conn(0x7f941c09fae0 0x7f941c07eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:23:37.674 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:37.666+0000 7f9422194700 1 --2- 172.21.15.42:0/1600980690 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f941c0a7740 0x7f941c08df80 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:23:37.674 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:37.666+0000 7f9422194700 1 -- 172.21.15.42:0/1600980690 shutdown_connections 2019-12-05T05:23:37.675 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:37.666+0000 7f9422194700 1 -- 172.21.15.42:0/1600980690 wait complete. 2019-12-05T05:23:37.675 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:37.666+0000 7f9422194700 1 -- 172.21.15.42:0/1600980690 >> 172.21.15.42:0/1600980690 conn(0x7f941c0aabe0 msgr2=0x7f941c0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:23:37.678 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:37.666+0000 7f9422194700 1 Processor -- start 2019-12-05T05:23:37.678 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:37.666+0000 7f9422194700 1 -- start start 2019-12-05T05:23:37.678 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:37.670+0000 7f9422194700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f941c118840 0x7f941c07bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:23:37.679 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:37.670+0000 7f9422194700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f941c07d240 con 0x7f941c07aff0 2019-12-05T05:23:37.679 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:37.670+0000 7f9422194700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f941c0a7040 con 0x7f941c130d10 2019-12-05T05:23:37.679 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:37.670+0000 7f9422194700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f941c07d670 con 0x7f941c118840 2019-12-05T05:23:37.679 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:37.670+0000 7f9420f32700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f941c130d10 0x7f941c1377e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:60984/0 (socket says 172.21.15.42:60984) 2019-12-05T05:23:37.680 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:37.670+0000 7f9420f32700 1 -- 172.21.15.42:0/3878869481 learned_addr learned my addr 172.21.15.42:0/3878869481 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:23:37.680 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:37.670+0000 7f941bfff700 1 --2- 172.21.15.42:0/3878869481 >> v2:172.21.15.40:3300/0 conn(0x7f941c118840 0x7f941c07bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:23:37.680 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:37.670+0000 7f9418ff9700 1 -- 172.21.15.42:0/3878869481 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3055272584 0 0) 0x7f941c0a7040 con 0x7f941c130d10 2019-12-05T05:23:37.680 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:37.670+0000 7f9418ff9700 1 -- 172.21.15.42:0/3878869481 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93fc0030b0 con 0x7f941c130d10 2019-12-05T05:23:37.680 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:37.670+0000 7f941bfff700 1 -- 172.21.15.42:0/3878869481 >> v1:172.21.15.42:6789/0 conn(0x7f941c130d10 legacy=0x7f941c1377e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:37.680 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:37.670+0000 7f9418ff9700 1 -- 172.21.15.42:0/3878869481 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 3639593446 0 0) 0x7f93fc0030b0 con 0x7f941c130d10 2019-12-05T05:23:37.681 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:37.670+0000 7f941bfff700 1 -- 172.21.15.42:0/3878869481 >> v1:172.21.15.40:6789/0 conn(0x7f941c07aff0 legacy=0x7f941c130510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:37.681 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:37.670+0000 7f941bfff700 1 -- 172.21.15.42:0/3878869481 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f941c07b420 con 0x7f941c118840 2019-12-05T05:23:37.681 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:37.670+0000 7f941bfff700 1 --2- 172.21.15.42:0/3878869481 >> v2:172.21.15.40:3300/0 conn(0x7f941c118840 0x7f941c07bf70 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f94100010d0 tx=0x7f9410001110).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:37.681 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:37.670+0000 7f941bfff700 1 --2- 172.21.15.42:0/3878869481 >> v2:172.21.15.40:3300/0 conn(0x7f941c118840 0x7f941c07bf70 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f94100010d0 tx=0x7f9410001110).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:37.681 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:37.670+0000 7f941bfff700 1 --2- 172.21.15.42:0/3878869481 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f941c118840 0x7f941c07bf70 secure :-1 s=READY pgs=613 cs=0 l=1 rx=0x7f94100010d0 tx=0x7f9410001110).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:23:37.682 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:37.670+0000 7f941bfff700 1 --2- 172.21.15.42:0/3878869481 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f941c118840 0x7f941c07bf70 secure :-1 s=THROTTLE_DONE pgs=613 cs=0 l=1 rx=0x7f94100010d0 tx=0x7f9410001110).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:37.682 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:37.670+0000 7f9418ff9700 1 -- 172.21.15.42:0/3878869481 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9414001810 con 0x7f941c118840 2019-12-05T05:23:37.682 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:37.670+0000 7f941bfff700 1 --2- 172.21.15.42:0/3878869481 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f941c118840 0x7f941c07bf70 secure :-1 s=THROTTLE_DONE pgs=613 cs=0 l=1 rx=0x7f94100010d0 tx=0x7f9410001110).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:37.682 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:37.670+0000 7f9422194700 1 -- 172.21.15.42:0/3878869481 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f941c07b6f0 con 0x7f941c118840 2019-12-05T05:23:37.682 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:37.670+0000 7f9418ff9700 1 -- 172.21.15.42:0/3878869481 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9414001620 con 0x7f941c118840 2019-12-05T05:23:37.682 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:37.670+0000 7f941bfff700 1 --2- 172.21.15.42:0/3878869481 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f941c118840 0x7f941c07bf70 secure :-1 s=THROTTLE_DONE pgs=613 cs=0 l=1 rx=0x7f94100010d0 tx=0x7f9410001110).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:37.682 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:37.670+0000 7f9418ff9700 1 -- 172.21.15.42:0/3878869481 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f940c0075e0 con 0x7f941c118840 2019-12-05T05:23:37.683 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:37.670+0000 7f9422194700 1 -- 172.21.15.42:0/3878869481 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f941c138ff0 con 0x7f941c118840 2019-12-05T05:23:37.683 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:37.674+0000 7f941bfff700 1 --2- 172.21.15.42:0/3878869481 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f941c118840 0x7f941c07bf70 secure :-1 s=THROTTLE_DONE pgs=613 cs=0 l=1 rx=0x7f94100010d0 tx=0x7f9410001110).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:37.683 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:37.674+0000 7f9418ff9700 1 -- 172.21.15.42:0/3878869481 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f940c00f280 con 0x7f941c118840 2019-12-05T05:23:37.683 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:37.674+0000 7f941bfff700 1 --2- 172.21.15.42:0/3878869481 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f941c118840 0x7f941c07bf70 secure :-1 s=THROTTLE_DONE pgs=613 cs=0 l=1 rx=0x7f94100010d0 tx=0x7f9410001110).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:37.683 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:37.674+0000 7f9418ff9700 1 --2- 172.21.15.42:0/3878869481 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f93fc03b1a0 0x7f93fc03d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:23:37.683 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:37.674+0000 7f9418ff9700 1 -- 172.21.15.42:0/3878869481 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 7149+0+0 (secure 0 0 0) 0x7f940c04b040 con 0x7f941c118840 2019-12-05T05:23:37.684 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:37.674+0000 7f941b7fe700 1 --2- 172.21.15.42:0/3878869481 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f93fc03b1a0 0x7f93fc03d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:23:37.684 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:37.674+0000 7f941b7fe700 1 --2- 172.21.15.42:0/3878869481 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f93fc03b1a0 0x7f93fc03d6b0 crc :-1 s=READY pgs=339 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:23:37.684 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:37.674+0000 7f9422194700 1 -- 172.21.15.42:0/3878869481 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f941c07d470 con 0x7f941c118840 2019-12-05T05:23:37.688 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:37.686+0000 7f941bfff700 1 --2- 172.21.15.42:0/3878869481 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f941c118840 0x7f941c07bf70 secure :-1 s=THROTTLE_DONE pgs=613 cs=0 l=1 rx=0x7f94100010d0 tx=0x7f9410001110).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:37.688 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:37.686+0000 7f9418ff9700 1 -- 172.21.15.42:0/3878869481 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f940c0320a0 con 0x7f941c118840 2019-12-05T05:23:37.857 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:37.854+0000 7f9422194700 1 -- 172.21.15.42:0/3878869481 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f941c05d110 con 0x7f941c118840 2019-12-05T05:23:37.858 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:37.854+0000 7f941bfff700 1 --2- 172.21.15.42:0/3878869481 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f941c118840 0x7f941c07bf70 secure :-1 s=THROTTLE_DONE pgs=613 cs=0 l=1 rx=0x7f94100010d0 tx=0x7f9410001110).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:37.858 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:37.854+0000 7f9418ff9700 1 -- 172.21.15.42:0/3878869481 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 98 v98) v1 ==== 94+0+2108 (secure 0 0 0) 0x7f940c032300 con 0x7f941c118840 2019-12-05T05:23:37.860 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 98 2019-12-05T05:23:37.861 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:23:37.861 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":98,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5177,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/3208416000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3208416000},{"type":"v1","addr":"172.21.15.40:6833","nonce":3208416000}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":98}],"filesystems":[{"mdsmap":{"epoch":97,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:23:07.988779+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5123},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5123":{"gid":5123,"name":"a","rank":0,"incarnation":94,"state":"up:active","state_seq":37,"addr":"172.21.15.42:6835/1214659805","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1214659805},{"type":"v1","addr":"172.21.15.42:6835","nonce":1214659805}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:23:37.863 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:37.858+0000 7f9422194700 1 -- 172.21.15.42:0/3878869481 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f93fc03b1a0 msgr2=0x7f93fc03d6b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:37.863 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:37.858+0000 7f9422194700 1 --2- 172.21.15.42:0/3878869481 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f93fc03b1a0 0x7f93fc03d6b0 crc :-1 s=READY pgs=339 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:23:37.864 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:37.858+0000 7f9422194700 1 -- 172.21.15.42:0/3878869481 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f941c118840 msgr2=0x7f941c07bf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:37.864 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:37.858+0000 7f9422194700 1 --2- 172.21.15.42:0/3878869481 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f941c118840 0x7f941c07bf70 secure :-1 s=READY pgs=613 cs=0 l=1 rx=0x7f94100010d0 tx=0x7f9410001110).stop 2019-12-05T05:23:37.864 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:37.858+0000 7f9422194700 1 -- 172.21.15.42:0/3878869481 shutdown_connections 2019-12-05T05:23:37.864 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:37.858+0000 7f9422194700 1 --2- 172.21.15.42:0/3878869481 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f93fc03b1a0 0x7f93fc03d6b0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:23:37.864 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:37.858+0000 7f9422194700 1 --2- 172.21.15.42:0/3878869481 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f941c118840 0x7f941c07bf70 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:23:37.864 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:37.858+0000 7f9422194700 1 -- 172.21.15.42:0/3878869481 shutdown_connections 2019-12-05T05:23:37.865 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:37.858+0000 7f9422194700 1 -- 172.21.15.42:0/3878869481 wait complete. 2019-12-05T05:23:37.865 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:37.858+0000 7f9422194700 1 -- 172.21.15.42:0/3878869481 >> 172.21.15.42:0/3878869481 conn(0x7f941c08fda0 msgr2=0x7f941c09f620 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:23:37.873 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T05:23:38.162 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:23:38.174 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:23:39.874 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:23:39.983 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:39.974+0000 7f80e780c700 1 Processor -- start 2019-12-05T05:23:39.983 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:39.974+0000 7f80e780c700 1 -- start start 2019-12-05T05:23:39.983 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:39.974+0000 7f80e780c700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f80e00a7740 0x7f80e008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:23:39.984 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:39.974+0000 7f80e780c700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f80e007e720 0x7f80e007eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:23:39.984 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:39.974+0000 7f80e780c700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80e00a0710 con 0x7f80e009f2e0 2019-12-05T05:23:39.984 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:39.974+0000 7f80e780c700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80e00a0b00 con 0x7f80e007e720 2019-12-05T05:23:39.984 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:39.974+0000 7f80e780c700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80e008e5b0 con 0x7f80e00a7740 2019-12-05T05:23:39.984 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:39.974+0000 7f80e65aa700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f80e009f2e0 0x7f80e009f710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:60994/0 (socket says 172.21.15.42:60994) 2019-12-05T05:23:39.984 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:39.974+0000 7f80e65aa700 1 -- 172.21.15.42:0/42343538 learned_addr learned my addr 172.21.15.42:0/42343538 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:23:39.985 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:39.974+0000 7f80e5da9700 1 --2- 172.21.15.42:0/42343538 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f80e00a7740 0x7f80e008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:23:39.985 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:39.974+0000 7f80e55a8700 1 --2- 172.21.15.42:0/42343538 >> v2:172.21.15.40:3300/0 conn(0x7f80e007e720 0x7f80e007eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:23:39.985 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:39.978+0000 7f80e4da7700 1 -- 172.21.15.42:0/42343538 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 893597488 0 0) 0x7f80e00a0710 con 0x7f80e009f2e0 2019-12-05T05:23:39.985 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:39.978+0000 7f80e4da7700 1 -- 172.21.15.42:0/42343538 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80cc0030b0 con 0x7f80e009f2e0 2019-12-05T05:23:39.986 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:39.978+0000 7f80e5da9700 1 -- 172.21.15.42:0/42343538 >> v2:172.21.15.40:3300/0 conn(0x7f80e007e720 msgr2=0x7f80e007eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:23:39.986 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:39.978+0000 7f80e5da9700 1 --2- 172.21.15.42:0/42343538 >> v2:172.21.15.40:3300/0 conn(0x7f80e007e720 0x7f80e007eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:23:39.986 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:39.978+0000 7f80e5da9700 1 -- 172.21.15.42:0/42343538 >> v1:172.21.15.42:6789/0 conn(0x7f80e009f2e0 legacy=0x7f80e009f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:39.986 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:39.978+0000 7f80e5da9700 1 -- 172.21.15.42:0/42343538 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80e007f0a0 con 0x7f80e00a7740 2019-12-05T05:23:39.986 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:39.978+0000 7f80e5da9700 1 --2- 172.21.15.42:0/42343538 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f80e00a7740 0x7f80e008df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f80d0008790 tx=0x7f80d000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:39.986 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:39.978+0000 7f80e5da9700 1 --2- 172.21.15.42:0/42343538 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f80e00a7740 0x7f80e008df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f80d0008790 tx=0x7f80d000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:39.987 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:39.978+0000 7f80e5da9700 1 --2- 172.21.15.42:0/42343538 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f80e00a7740 0x7f80e008df80 secure :-1 s=READY pgs=159 cs=0 l=1 rx=0x7f80d0008790 tx=0x7f80d000fc70).ready entity=mon.2 client_cookie=9c1e36a896b87f2 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:23:39.987 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:39.978+0000 7f80e5da9700 1 --2- 172.21.15.42:0/42343538 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f80e00a7740 0x7f80e008df80 secure :-1 s=THROTTLE_DONE pgs=159 cs=0 l=1 rx=0x7f80d0008790 tx=0x7f80d000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:39.987 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:39.978+0000 7f80e4da7700 1 -- 172.21.15.42:0/42343538 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f80d00112f0 con 0x7f80e00a7740 2019-12-05T05:23:39.987 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:39.978+0000 7f80e5da9700 1 --2- 172.21.15.42:0/42343538 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f80e00a7740 0x7f80e008df80 secure :-1 s=THROTTLE_DONE pgs=159 cs=0 l=1 rx=0x7f80d0008790 tx=0x7f80d000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:39.987 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:39.978+0000 7f80e4da7700 1 -- 172.21.15.42:0/42343538 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80d0008ad0 con 0x7f80e00a7740 2019-12-05T05:23:39.987 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:39.978+0000 7f80e5da9700 1 --2- 172.21.15.42:0/42343538 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f80e00a7740 0x7f80e008df80 secure :-1 s=THROTTLE_DONE pgs=159 cs=0 l=1 rx=0x7f80d0008790 tx=0x7f80d000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:39.988 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:39.978+0000 7f80e4da7700 1 -- 172.21.15.42:0/42343538 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f80d00118d0 con 0x7f80e00a7740 2019-12-05T05:23:39.988 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:39.978+0000 7f80e780c700 1 -- 172.21.15.42:0/42343538 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f80e00a7740 msgr2=0x7f80e008df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:39.988 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:39.978+0000 7f80e780c700 1 --2- 172.21.15.42:0/42343538 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f80e00a7740 0x7f80e008df80 secure :-1 s=READY pgs=159 cs=0 l=1 rx=0x7f80d0008790 tx=0x7f80d000fc70).stop 2019-12-05T05:23:39.988 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:39.978+0000 7f80e780c700 1 -- 172.21.15.42:0/42343538 shutdown_connections 2019-12-05T05:23:39.988 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:39.978+0000 7f80e780c700 1 --2- 172.21.15.42:0/42343538 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f80e00a7740 0x7f80e008df80 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:23:39.991 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:39.978+0000 7f80e780c700 1 --2- 172.21.15.42:0/42343538 >> v2:172.21.15.40:3300/0 conn(0x7f80e007e720 0x7f80e007eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:23:39.991 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:39.978+0000 7f80e780c700 1 -- 172.21.15.42:0/42343538 shutdown_connections 2019-12-05T05:23:39.991 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:39.978+0000 7f80e780c700 1 -- 172.21.15.42:0/42343538 wait complete. 2019-12-05T05:23:39.992 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:39.978+0000 7f80e780c700 1 -- 172.21.15.42:0/42343538 >> 172.21.15.42:0/42343538 conn(0x7f80e00aabe0 msgr2=0x7f80e00a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:23:39.995 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:39.982+0000 7f80e780c700 1 Processor -- start 2019-12-05T05:23:39.995 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:39.982+0000 7f80e780c700 1 -- start start 2019-12-05T05:23:39.995 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:39.982+0000 7f80e780c700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f80e0118f80 0x7f80e0131dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:23:39.995 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:39.982+0000 7f80e780c700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80e0121150 con 0x7f80e00a3a20 2019-12-05T05:23:39.996 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:39.982+0000 7f80e780c700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80e0076af0 con 0x7f80e0120d20 2019-12-05T05:23:39.996 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:39.982+0000 7f80e780c700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80e0119570 con 0x7f80e0118f80 2019-12-05T05:23:39.996 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:39.982+0000 7f80e5da9700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f80e00a3a20 0x7f80e01315c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:42204/0 (socket says 172.21.15.42:42204) 2019-12-05T05:23:39.996 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:39.982+0000 7f80e65aa700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f80e0120d20 0x7f80e00742d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:32768/0 (socket says 172.21.15.42:32768) 2019-12-05T05:23:39.997 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:39.982+0000 7f80e5da9700 1 -- 172.21.15.42:0/203781442 learned_addr learned my addr 172.21.15.42:0/203781442 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:23:39.997 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:39.982+0000 7f80e55a8700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f80e0118f80 0x7f80e0131dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:23:39.997 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:39.982+0000 7f80d67fc700 1 -- 172.21.15.42:0/203781442 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 34781139 0 0) 0x7f80e0076af0 con 0x7f80e0120d20 2019-12-05T05:23:39.997 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:39.982+0000 7f80d67fc700 1 -- 172.21.15.42:0/203781442 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80c80030b0 con 0x7f80e0120d20 2019-12-05T05:23:39.997 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:39.982+0000 7f80d67fc700 1 -- 172.21.15.42:0/203781442 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 294781137 0 0) 0x7f80e0121150 con 0x7f80e00a3a20 2019-12-05T05:23:39.997 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:39.982+0000 7f80d67fc700 1 -- 172.21.15.42:0/203781442 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80e0076af0 con 0x7f80e00a3a20 2019-12-05T05:23:39.998 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:39.982+0000 7f80e55a8700 1 -- 172.21.15.42:0/203781442 >> v1:172.21.15.42:6789/0 conn(0x7f80e0120d20 legacy=0x7f80e00742d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:39.998 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:39.982+0000 7f80e55a8700 1 -- 172.21.15.42:0/203781442 >> v1:172.21.15.40:6789/0 conn(0x7f80e00a3a20 legacy=0x7f80e01315c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:39.998 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:39.986+0000 7f80e55a8700 1 -- 172.21.15.42:0/203781442 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80e0077d30 con 0x7f80e0118f80 2019-12-05T05:23:39.998 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:39.986+0000 7f80e55a8700 1 --2- 172.21.15.42:0/203781442 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f80e0118f80 0x7f80e0131dc0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f80dc002f30 tx=0x7f80dc010b20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:39.998 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:39.986+0000 7f80e55a8700 1 --2- 172.21.15.42:0/203781442 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f80e0118f80 0x7f80e0131dc0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f80dc002f30 tx=0x7f80dc010b20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:39.998 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:39.986+0000 7f80e55a8700 1 --2- 172.21.15.42:0/203781442 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f80e0118f80 0x7f80e0131dc0 secure :-1 s=READY pgs=160 cs=0 l=1 rx=0x7f80dc002f30 tx=0x7f80dc010b20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:23:39.999 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:39.986+0000 7f80e55a8700 1 --2- 172.21.15.42:0/203781442 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f80e0118f80 0x7f80e0131dc0 secure :-1 s=THROTTLE_DONE pgs=160 cs=0 l=1 rx=0x7f80dc002f30 tx=0x7f80dc010b20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:39.999 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:39.986+0000 7f80d67fc700 1 -- 172.21.15.42:0/203781442 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f80dc005540 con 0x7f80e0118f80 2019-12-05T05:23:39.999 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:39.986+0000 7f80e780c700 1 -- 172.21.15.42:0/203781442 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80e0076d20 con 0x7f80e0118f80 2019-12-05T05:23:39.999 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:39.986+0000 7f80e55a8700 1 --2- 172.21.15.42:0/203781442 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f80e0118f80 0x7f80e0131dc0 secure :-1 s=THROTTLE_DONE pgs=160 cs=0 l=1 rx=0x7f80dc002f30 tx=0x7f80dc010b20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:39.999 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:39.986+0000 7f80d67fc700 1 -- 172.21.15.42:0/203781442 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80dc007150 con 0x7f80e0118f80 2019-12-05T05:23:40.000 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:39.986+0000 7f80e55a8700 1 --2- 172.21.15.42:0/203781442 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f80e0118f80 0x7f80e0131dc0 secure :-1 s=THROTTLE_DONE pgs=160 cs=0 l=1 rx=0x7f80dc002f30 tx=0x7f80dc010b20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:40.000 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:39.986+0000 7f80d67fc700 1 -- 172.21.15.42:0/203781442 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f80dc005b20 con 0x7f80e0118f80 2019-12-05T05:23:40.000 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:39.986+0000 7f80e780c700 1 -- 172.21.15.42:0/203781442 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80e00772d0 con 0x7f80e0118f80 2019-12-05T05:23:40.000 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:39.986+0000 7f80e55a8700 1 --2- 172.21.15.42:0/203781442 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f80e0118f80 0x7f80e0131dc0 secure :-1 s=THROTTLE_DONE pgs=160 cs=0 l=1 rx=0x7f80dc002f30 tx=0x7f80dc010b20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:40.001 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:39.986+0000 7f80d67fc700 1 -- 172.21.15.42:0/203781442 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f80dc0123c0 con 0x7f80e0118f80 2019-12-05T05:23:40.002 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:39.986+0000 7f80e55a8700 1 --2- 172.21.15.42:0/203781442 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f80e0118f80 0x7f80e0131dc0 secure :-1 s=THROTTLE_DONE pgs=160 cs=0 l=1 rx=0x7f80dc002f30 tx=0x7f80dc010b20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:40.002 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:39.986+0000 7f80d67fc700 1 --2- 172.21.15.42:0/203781442 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f80c803b4c0 0x7f80c803d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:23:40.002 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:39.986+0000 7f80d67fc700 1 -- 172.21.15.42:0/203781442 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 7149+0+0 (secure 0 0 0) 0x7f80dc04ae10 con 0x7f80e0118f80 2019-12-05T05:23:40.002 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:39.986+0000 7f80e5da9700 1 --2- 172.21.15.42:0/203781442 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f80c803b4c0 0x7f80c803d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:23:40.003 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:39.986+0000 7f80e5da9700 1 --2- 172.21.15.42:0/203781442 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f80c803b4c0 0x7f80c803d9d0 crc :-1 s=READY pgs=340 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:23:40.003 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:39.990+0000 7f80e780c700 1 -- 172.21.15.42:0/203781442 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80e005d110 con 0x7f80e0118f80 2019-12-05T05:23:40.004 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:40.002+0000 7f80e55a8700 1 --2- 172.21.15.42:0/203781442 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f80e0118f80 0x7f80e0131dc0 secure :-1 s=THROTTLE_DONE pgs=160 cs=0 l=1 rx=0x7f80dc002f30 tx=0x7f80dc010b20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:40.004 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:40.002+0000 7f80d67fc700 1 -- 172.21.15.42:0/203781442 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f80dc031e70 con 0x7f80e0118f80 2019-12-05T05:23:40.172 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:40.170+0000 7f80e780c700 1 -- 172.21.15.42:0/203781442 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f80e011d3c0 con 0x7f80e0118f80 2019-12-05T05:23:40.173 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:40.170+0000 7f80e55a8700 1 --2- 172.21.15.42:0/203781442 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f80e0118f80 0x7f80e0131dc0 secure :-1 s=THROTTLE_DONE pgs=160 cs=0 l=1 rx=0x7f80dc002f30 tx=0x7f80dc010b20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:40.173 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:40.170+0000 7f80d67fc700 1 -- 172.21.15.42:0/203781442 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 98 v98) v1 ==== 94+0+2108 (secure 0 0 0) 0x7f80dc0320d0 con 0x7f80e0118f80 2019-12-05T05:23:40.175 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 98 2019-12-05T05:23:40.175 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:23:40.175 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":98,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5177,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/3208416000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3208416000},{"type":"v1","addr":"172.21.15.40:6833","nonce":3208416000}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":98}],"filesystems":[{"mdsmap":{"epoch":97,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:23:07.988779+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5123},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5123":{"gid":5123,"name":"a","rank":0,"incarnation":94,"state":"up:active","state_seq":37,"addr":"172.21.15.42:6835/1214659805","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1214659805},{"type":"v1","addr":"172.21.15.42:6835","nonce":1214659805}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:23:40.179 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:40.174+0000 7f80e780c700 1 -- 172.21.15.42:0/203781442 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f80c803b4c0 msgr2=0x7f80c803d9d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:40.180 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:40.174+0000 7f80e780c700 1 --2- 172.21.15.42:0/203781442 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f80c803b4c0 0x7f80c803d9d0 crc :-1 s=READY pgs=340 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:23:40.180 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:40.174+0000 7f80e780c700 1 -- 172.21.15.42:0/203781442 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f80e0118f80 msgr2=0x7f80e0131dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:40.180 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:40.174+0000 7f80e780c700 1 --2- 172.21.15.42:0/203781442 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f80e0118f80 0x7f80e0131dc0 secure :-1 s=READY pgs=160 cs=0 l=1 rx=0x7f80dc002f30 tx=0x7f80dc010b20).stop 2019-12-05T05:23:40.180 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:40.174+0000 7f80e780c700 1 -- 172.21.15.42:0/203781442 shutdown_connections 2019-12-05T05:23:40.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:40.174+0000 7f80e780c700 1 --2- 172.21.15.42:0/203781442 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f80c803b4c0 0x7f80c803d9d0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:23:40.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:40.174+0000 7f80e780c700 1 --2- 172.21.15.42:0/203781442 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f80e0118f80 0x7f80e0131dc0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:23:40.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:40.174+0000 7f80e780c700 1 -- 172.21.15.42:0/203781442 shutdown_connections 2019-12-05T05:23:40.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:40.174+0000 7f80e780c700 1 -- 172.21.15.42:0/203781442 wait complete. 2019-12-05T05:23:40.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:40.174+0000 7f80e780c700 1 -- 172.21.15.42:0/203781442 >> 172.21.15.42:0/203781442 conn(0x7f80e00a82a0 msgr2=0x7f80e00a9790 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:23:40.189 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T05:23:42.190 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:23:42.297 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:42.286+0000 7fc6515b8700 1 Processor -- start 2019-12-05T05:23:42.298 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:42.286+0000 7fc6515b8700 1 -- start start 2019-12-05T05:23:42.298 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:42.290+0000 7fc6515b8700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc64c0a7740 0x7fc64c08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:23:42.298 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:42.290+0000 7fc6515b8700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc64c07e720 0x7fc64c07eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:23:42.298 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:42.290+0000 7fc6515b8700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc64c09f2e0 0x7fc64c09f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:23:42.298 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:42.290+0000 7fc6515b8700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc64c0a0b00 con 0x7fc64c0a7740 2019-12-05T05:23:42.298 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:42.290+0000 7fc6515b8700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc64c08e5b0 con 0x7fc64c07e720 2019-12-05T05:23:42.298 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:42.290+0000 7fc6515b8700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc64c08e850 con 0x7fc64c09f2e0 2019-12-05T05:23:42.299 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:42.290+0000 7fc64affd700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc64c07e720 0x7fc64c07eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:23:42.299 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:42.290+0000 7fc64affd700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc64c07e720 0x7fc64c07eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:60970/0 (socket says 172.21.15.42:60970) 2019-12-05T05:23:42.299 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:42.290+0000 7fc64affd700 1 -- 172.21.15.42:0/441149483 learned_addr learned my addr 172.21.15.42:0/441149483 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:23:42.299 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:42.290+0000 7fc64b7fe700 1 --2- 172.21.15.42:0/441149483 >> v2:172.21.15.40:3300/0 conn(0x7fc64c0a7740 0x7fc64c08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:23:42.299 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:42.290+0000 7fc64bfff700 1 --2- 172.21.15.42:0/441149483 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc64c09f2e0 0x7fc64c09f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:23:42.299 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:42.290+0000 7fc64affd700 1 -- 172.21.15.42:0/441149483 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc64c09f2e0 msgr2=0x7fc64c09f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:23:42.299 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:42.290+0000 7fc64affd700 1 --2- 172.21.15.42:0/441149483 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc64c09f2e0 0x7fc64c09f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:23:42.300 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:42.290+0000 7fc64affd700 1 -- 172.21.15.42:0/441149483 >> v2:172.21.15.40:3300/0 conn(0x7fc64c0a7740 msgr2=0x7fc64c08df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:23:42.300 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:42.290+0000 7fc64affd700 1 --2- 172.21.15.42:0/441149483 >> v2:172.21.15.40:3300/0 conn(0x7fc64c0a7740 0x7fc64c08df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:23:42.300 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:42.290+0000 7fc64affd700 1 -- 172.21.15.42:0/441149483 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc64c07f0a0 con 0x7fc64c07e720 2019-12-05T05:23:42.300 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:42.290+0000 7fc64b7fe700 1 --2- 172.21.15.42:0/441149483 >> v2:172.21.15.40:3300/0 conn(0x7fc64c0a7740 0x7fc64c08df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:23:42.301 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:42.290+0000 7fc64affd700 1 --2- 172.21.15.42:0/441149483 >> v2:172.21.15.42:3300/0 conn(0x7fc64c07e720 0x7fc64c07eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fc64c0a0f50 tx=0x7fc64000fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:42.301 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:42.290+0000 7fc64affd700 1 --2- 172.21.15.42:0/441149483 >> v2:172.21.15.42:3300/0 conn(0x7fc64c07e720 0x7fc64c07eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fc64c0a0f50 tx=0x7fc64000fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:42.301 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:42.290+0000 7fc64affd700 1 --2- 172.21.15.42:0/441149483 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc64c07e720 0x7fc64c07eb50 secure :-1 s=READY pgs=773 cs=0 l=1 rx=0x7fc64c0a0f50 tx=0x7fc64000fdb0).ready entity=mon.1 client_cookie=9ea8cfb0bf160a74 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:23:42.301 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:42.290+0000 7fc64affd700 1 --2- 172.21.15.42:0/441149483 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc64c07e720 0x7fc64c07eb50 secure :-1 s=THROTTLE_DONE pgs=773 cs=0 l=1 rx=0x7fc64c0a0f50 tx=0x7fc64000fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:42.301 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:42.290+0000 7fc64affd700 1 --2- 172.21.15.42:0/441149483 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc64c07e720 0x7fc64c07eb50 secure :-1 s=THROTTLE_DONE pgs=773 cs=0 l=1 rx=0x7fc64c0a0f50 tx=0x7fc64000fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:42.301 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:42.290+0000 7fc64affd700 1 --2- 172.21.15.42:0/441149483 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc64c07e720 0x7fc64c07eb50 secure :-1 s=THROTTLE_DONE pgs=773 cs=0 l=1 rx=0x7fc64c0a0f50 tx=0x7fc64000fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:42.301 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:42.290+0000 7fc64a7fc700 1 -- 172.21.15.42:0/441149483 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc640008c90 con 0x7fc64c07e720 2019-12-05T05:23:42.302 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:42.290+0000 7fc64a7fc700 1 -- 172.21.15.42:0/441149483 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6400088b0 con 0x7fc64c07e720 2019-12-05T05:23:42.302 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:42.290+0000 7fc64a7fc700 1 -- 172.21.15.42:0/441149483 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc640011b50 con 0x7fc64c07e720 2019-12-05T05:23:42.302 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:42.290+0000 7fc6515b8700 1 -- 172.21.15.42:0/441149483 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc64c07e720 msgr2=0x7fc64c07eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:42.302 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:42.290+0000 7fc6515b8700 1 --2- 172.21.15.42:0/441149483 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc64c07e720 0x7fc64c07eb50 secure :-1 s=READY pgs=773 cs=0 l=1 rx=0x7fc64c0a0f50 tx=0x7fc64000fdb0).stop 2019-12-05T05:23:42.302 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:42.290+0000 7fc6515b8700 1 -- 172.21.15.42:0/441149483 shutdown_connections 2019-12-05T05:23:42.302 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:42.290+0000 7fc6515b8700 1 --2- 172.21.15.42:0/441149483 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc64c09f2e0 0x7fc64c09f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:23:42.302 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:42.290+0000 7fc6515b8700 1 --2- 172.21.15.42:0/441149483 >> v2:172.21.15.40:3300/0 conn(0x7fc64c0a7740 0x7fc64c08df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:23:42.303 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:42.290+0000 7fc6515b8700 1 --2- 172.21.15.42:0/441149483 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc64c07e720 0x7fc64c07eb50 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:23:42.303 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:42.290+0000 7fc6515b8700 1 -- 172.21.15.42:0/441149483 shutdown_connections 2019-12-05T05:23:42.303 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:42.294+0000 7fc6515b8700 1 -- 172.21.15.42:0/441149483 wait complete. 2019-12-05T05:23:42.303 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:42.294+0000 7fc6515b8700 1 -- 172.21.15.42:0/441149483 >> 172.21.15.42:0/441149483 conn(0x7fc64c0aabe0 msgr2=0x7fc64c0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:23:42.307 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:42.294+0000 7fc6515b8700 1 Processor -- start 2019-12-05T05:23:42.307 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:42.294+0000 7fc6515b8700 1 -- start start 2019-12-05T05:23:42.307 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:42.294+0000 7fc6515b8700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc64c07bd40 0x7fc64c07c170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:23:42.307 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:42.294+0000 7fc6515b8700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc64c121f60 0x7fc64c122390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:23:42.307 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:42.294+0000 7fc6515b8700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc64c123e80 con 0x7fc64c08eeb0 2019-12-05T05:23:42.307 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:42.294+0000 7fc6515b8700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc64c11b710 con 0x7fc64c121f60 2019-12-05T05:23:42.308 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:42.294+0000 7fc6515b8700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc64c11af30 con 0x7fc64c07bd40 2019-12-05T05:23:42.308 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:42.298+0000 7fc64bfff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc64c121f60 0x7fc64c122390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:23:42.308 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:42.298+0000 7fc64affd700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc64c07bd40 0x7fc64c07c170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:23:42.308 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:42.298+0000 7fc64b7fe700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fc64c08eeb0 0x7fc64c123680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:42220/0 (socket says 172.21.15.42:42220) 2019-12-05T05:23:42.308 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:42.298+0000 7fc64b7fe700 1 -- 172.21.15.42:0/1465301360 learned_addr learned my addr 172.21.15.42:0/1465301360 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:23:42.308 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:42.298+0000 7fc633fff700 1 -- 172.21.15.42:0/1465301360 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3748269571 0 0) 0x7fc64c123e80 con 0x7fc64c08eeb0 2019-12-05T05:23:42.309 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:42.298+0000 7fc633fff700 1 -- 172.21.15.42:0/1465301360 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc62c0030b0 con 0x7fc64c08eeb0 2019-12-05T05:23:42.309 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:42.298+0000 7fc64bfff700 1 -- 172.21.15.42:0/1465301360 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc64c07bd40 msgr2=0x7fc64c07c170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:42.309 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:42.298+0000 7fc64bfff700 1 --2- 172.21.15.42:0/1465301360 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc64c07bd40 0x7fc64c07c170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:23:42.309 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:42.298+0000 7fc64bfff700 1 -- 172.21.15.42:0/1465301360 >> v1:172.21.15.40:6789/0 conn(0x7fc64c08eeb0 legacy=0x7fc64c123680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:42.309 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:42.298+0000 7fc64bfff700 1 -- 172.21.15.42:0/1465301360 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc64c07c6c0 con 0x7fc64c121f60 2019-12-05T05:23:42.309 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:42.298+0000 7fc64affd700 1 --2- 172.21.15.42:0/1465301360 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc64c07bd40 0x7fc64c07c170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:23:42.309 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:42.298+0000 7fc64bfff700 1 --2- 172.21.15.42:0/1465301360 >> v2:172.21.15.42:3300/0 conn(0x7fc64c121f60 0x7fc64c122390 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fc6440032b0 tx=0x7fc644010d10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:42.310 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:42.298+0000 7fc64bfff700 1 --2- 172.21.15.42:0/1465301360 >> v2:172.21.15.42:3300/0 conn(0x7fc64c121f60 0x7fc64c122390 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fc6440032b0 tx=0x7fc644010d10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:42.310 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:42.298+0000 7fc64bfff700 1 --2- 172.21.15.42:0/1465301360 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc64c121f60 0x7fc64c122390 secure :-1 s=READY pgs=774 cs=0 l=1 rx=0x7fc6440032b0 tx=0x7fc644010d10).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:23:42.310 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:42.298+0000 7fc64bfff700 1 --2- 172.21.15.42:0/1465301360 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc64c121f60 0x7fc64c122390 secure :-1 s=THROTTLE_DONE pgs=774 cs=0 l=1 rx=0x7fc6440032b0 tx=0x7fc644010d10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:42.310 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:42.298+0000 7fc633fff700 1 -- 172.21.15.42:0/1465301360 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc644007350 con 0x7fc64c121f60 2019-12-05T05:23:42.314 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:42.298+0000 7fc64bfff700 1 --2- 172.21.15.42:0/1465301360 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc64c121f60 0x7fc64c122390 secure :-1 s=THROTTLE_DONE pgs=774 cs=0 l=1 rx=0x7fc6440032b0 tx=0x7fc644010d10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:42.315 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:42.298+0000 7fc633fff700 1 -- 172.21.15.42:0/1465301360 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc644005cb0 con 0x7fc64c121f60 2019-12-05T05:23:42.315 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:42.298+0000 7fc64bfff700 1 --2- 172.21.15.42:0/1465301360 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc64c121f60 0x7fc64c122390 secure :-1 s=THROTTLE_DONE pgs=774 cs=0 l=1 rx=0x7fc6440032b0 tx=0x7fc644010d10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:42.315 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:42.298+0000 7fc6515b8700 1 -- 172.21.15.42:0/1465301360 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc64c1228e0 con 0x7fc64c121f60 2019-12-05T05:23:42.315 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:42.298+0000 7fc633fff700 1 -- 172.21.15.42:0/1465301360 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc6440123d0 con 0x7fc64c121f60 2019-12-05T05:23:42.315 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:42.298+0000 7fc6515b8700 1 -- 172.21.15.42:0/1465301360 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc64c12d780 con 0x7fc64c121f60 2019-12-05T05:23:42.315 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:42.298+0000 7fc64bfff700 1 --2- 172.21.15.42:0/1465301360 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc64c121f60 0x7fc64c122390 secure :-1 s=THROTTLE_DONE pgs=774 cs=0 l=1 rx=0x7fc6440032b0 tx=0x7fc644010d10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:42.316 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:42.298+0000 7fc633fff700 1 -- 172.21.15.42:0/1465301360 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fc6440126f0 con 0x7fc64c121f60 2019-12-05T05:23:42.316 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:42.298+0000 7fc64bfff700 1 --2- 172.21.15.42:0/1465301360 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc64c121f60 0x7fc64c122390 secure :-1 s=THROTTLE_DONE pgs=774 cs=0 l=1 rx=0x7fc6440032b0 tx=0x7fc644010d10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:42.316 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:42.298+0000 7fc633fff700 1 --2- 172.21.15.42:0/1465301360 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc62c03b260 0x7fc62c03d770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:23:42.316 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:42.298+0000 7fc633fff700 1 -- 172.21.15.42:0/1465301360 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 7149+0+0 (secure 0 0 0) 0x7fc64404b8a0 con 0x7fc64c121f60 2019-12-05T05:23:42.316 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:42.302+0000 7fc64b7fe700 1 --2- 172.21.15.42:0/1465301360 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc62c03b260 0x7fc62c03d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:23:42.316 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:42.302+0000 7fc64b7fe700 1 --2- 172.21.15.42:0/1465301360 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc62c03b260 0x7fc62c03d770 crc :-1 s=READY pgs=341 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:23:42.316 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:42.302+0000 7fc6515b8700 1 -- 172.21.15.42:0/1465301360 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc64c05d110 con 0x7fc64c121f60 2019-12-05T05:23:42.317 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:42.314+0000 7fc64bfff700 1 --2- 172.21.15.42:0/1465301360 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc64c121f60 0x7fc64c122390 secure :-1 s=THROTTLE_DONE pgs=774 cs=0 l=1 rx=0x7fc6440032b0 tx=0x7fc644010d10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:42.317 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:42.314+0000 7fc633fff700 1 -- 172.21.15.42:0/1465301360 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fc644032900 con 0x7fc64c121f60 2019-12-05T05:23:42.484 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:42.482+0000 7fc6515b8700 1 -- 172.21.15.42:0/1465301360 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc64c05b8c0 con 0x7fc64c121f60 2019-12-05T05:23:42.485 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:42.482+0000 7fc64bfff700 1 --2- 172.21.15.42:0/1465301360 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc64c121f60 0x7fc64c122390 secure :-1 s=THROTTLE_DONE pgs=774 cs=0 l=1 rx=0x7fc6440032b0 tx=0x7fc644010d10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:42.485 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:42.482+0000 7fc633fff700 1 -- 172.21.15.42:0/1465301360 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 98 v98) v1 ==== 94+0+2108 (secure 0 0 0) 0x7fc644032b60 con 0x7fc64c121f60 2019-12-05T05:23:42.487 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:23:42.487 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":98,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5177,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/3208416000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3208416000},{"type":"v1","addr":"172.21.15.40:6833","nonce":3208416000}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":98}],"filesystems":[{"mdsmap":{"epoch":97,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:23:07.988779+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5123},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5123":{"gid":5123,"name":"a","rank":0,"incarnation":94,"state":"up:active","state_seq":37,"addr":"172.21.15.42:6835/1214659805","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1214659805},{"type":"v1","addr":"172.21.15.42:6835","nonce":1214659805}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:23:42.487 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 98 2019-12-05T05:23:42.490 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:42.486+0000 7fc6515b8700 1 -- 172.21.15.42:0/1465301360 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc62c03b260 msgr2=0x7fc62c03d770 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:42.490 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:42.486+0000 7fc6515b8700 1 --2- 172.21.15.42:0/1465301360 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc62c03b260 0x7fc62c03d770 crc :-1 s=READY pgs=341 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:23:42.490 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:42.486+0000 7fc6515b8700 1 -- 172.21.15.42:0/1465301360 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc64c121f60 msgr2=0x7fc64c122390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:42.490 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:42.486+0000 7fc6515b8700 1 --2- 172.21.15.42:0/1465301360 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc64c121f60 0x7fc64c122390 secure :-1 s=READY pgs=774 cs=0 l=1 rx=0x7fc6440032b0 tx=0x7fc644010d10).stop 2019-12-05T05:23:42.490 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:42.486+0000 7fc6515b8700 1 -- 172.21.15.42:0/1465301360 shutdown_connections 2019-12-05T05:23:42.490 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:42.486+0000 7fc6515b8700 1 --2- 172.21.15.42:0/1465301360 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc62c03b260 0x7fc62c03d770 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:23:42.490 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:42.486+0000 7fc6515b8700 1 --2- 172.21.15.42:0/1465301360 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc64c121f60 0x7fc64c122390 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:23:42.491 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:42.486+0000 7fc6515b8700 1 --2- 172.21.15.42:0/1465301360 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc64c07bd40 0x7fc64c07c170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:23:42.491 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:42.486+0000 7fc6515b8700 1 -- 172.21.15.42:0/1465301360 shutdown_connections 2019-12-05T05:23:42.491 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:42.486+0000 7fc6515b8700 1 -- 172.21.15.42:0/1465301360 wait complete. 2019-12-05T05:23:42.491 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:42.486+0000 7fc6515b8700 1 -- 172.21.15.42:0/1465301360 >> 172.21.15.42:0/1465301360 conn(0x7fc64c11ce90 msgr2=0x7fc64c09f040 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:23:42.500 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T05:23:44.501 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:23:44.608 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:44.598+0000 7fea6114d700 1 Processor -- start 2019-12-05T05:23:44.608 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:44.598+0000 7fea6114d700 1 -- start start 2019-12-05T05:23:44.608 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:44.602+0000 7fea6114d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fea5c08df80 0x7fea5c09f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:23:44.608 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:44.602+0000 7fea6114d700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea5c09fc60 con 0x7fea5c09f830 2019-12-05T05:23:44.608 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:44.602+0000 7fea6114d700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea5c0a0710 con 0x7fea5c0a7740 2019-12-05T05:23:44.609 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:44.602+0000 7fea6114d700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea5c0a0b00 con 0x7fea5c08df80 2019-12-05T05:23:44.609 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:44.602+0000 7fea5b59e700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fea5c0a7740 0x7fea5c07e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:32788/0 (socket says 172.21.15.42:32788) 2019-12-05T05:23:44.609 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:44.602+0000 7fea5b59e700 1 -- 172.21.15.42:0/3654884327 learned_addr learned my addr 172.21.15.42:0/3654884327 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:23:44.609 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:44.602+0000 7fea5ad9d700 1 --2- 172.21.15.42:0/3654884327 >> v2:172.21.15.42:3300/0 conn(0x7fea5c08df80 0x7fea5c09f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:23:44.609 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:44.602+0000 7fea5a59c700 1 -- 172.21.15.42:0/3654884327 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1791347182 0 0) 0x7fea5c0a0710 con 0x7fea5c0a7740 2019-12-05T05:23:44.609 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:44.602+0000 7fea5a59c700 1 -- 172.21.15.42:0/3654884327 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea440030b0 con 0x7fea5c0a7740 2019-12-05T05:23:44.609 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:44.602+0000 7fea5ad9d700 1 -- 172.21.15.42:0/3654884327 >> v1:172.21.15.42:6789/0 conn(0x7fea5c0a7740 legacy=0x7fea5c07e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:44.610 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:44.602+0000 7fea5ad9d700 1 -- 172.21.15.42:0/3654884327 >> v1:172.21.15.40:6789/0 conn(0x7fea5c09f830 legacy=0x7fea5c14efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:44.610 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:44.602+0000 7fea5ad9d700 1 -- 172.21.15.42:0/3654884327 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea5c07ef20 con 0x7fea5c08df80 2019-12-05T05:23:44.610 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:44.602+0000 7fea5ad9d700 1 --2- 172.21.15.42:0/3654884327 >> v2:172.21.15.42:3300/0 conn(0x7fea5c08df80 0x7fea5c09f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fea50008790 tx=0x7fea5000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:44.610 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:44.602+0000 7fea5ad9d700 1 --2- 172.21.15.42:0/3654884327 >> v2:172.21.15.42:3300/0 conn(0x7fea5c08df80 0x7fea5c09f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fea50008790 tx=0x7fea5000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:44.610 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:44.602+0000 7fea5ad9d700 1 --2- 172.21.15.42:0/3654884327 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fea5c08df80 0x7fea5c09f2e0 secure :-1 s=READY pgs=776 cs=0 l=1 rx=0x7fea50008790 tx=0x7fea5000fbf0).ready entity=mon.1 client_cookie=27cd8867ae37108b server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:23:44.610 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:44.602+0000 7fea5ad9d700 1 --2- 172.21.15.42:0/3654884327 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fea5c08df80 0x7fea5c09f2e0 secure :-1 s=THROTTLE_DONE pgs=776 cs=0 l=1 rx=0x7fea50008790 tx=0x7fea5000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:44.610 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:44.602+0000 7fea5a59c700 1 -- 172.21.15.42:0/3654884327 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fea50011270 con 0x7fea5c08df80 2019-12-05T05:23:44.611 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:44.602+0000 7fea5ad9d700 1 --2- 172.21.15.42:0/3654884327 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fea5c08df80 0x7fea5c09f2e0 secure :-1 s=THROTTLE_DONE pgs=776 cs=0 l=1 rx=0x7fea50008790 tx=0x7fea5000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:44.611 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:44.602+0000 7fea5ad9d700 1 --2- 172.21.15.42:0/3654884327 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fea5c08df80 0x7fea5c09f2e0 secure :-1 s=THROTTLE_DONE pgs=776 cs=0 l=1 rx=0x7fea50008790 tx=0x7fea5000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:44.611 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:44.602+0000 7fea5a59c700 1 -- 172.21.15.42:0/3654884327 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea50008bc0 con 0x7fea5c08df80 2019-12-05T05:23:44.611 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:44.602+0000 7fea5a59c700 1 -- 172.21.15.42:0/3654884327 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fea50011850 con 0x7fea5c08df80 2019-12-05T05:23:44.611 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:44.602+0000 7fea6114d700 1 -- 172.21.15.42:0/3654884327 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fea5c08df80 msgr2=0x7fea5c09f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:44.611 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:44.602+0000 7fea6114d700 1 --2- 172.21.15.42:0/3654884327 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fea5c08df80 0x7fea5c09f2e0 secure :-1 s=READY pgs=776 cs=0 l=1 rx=0x7fea50008790 tx=0x7fea5000fbf0).stop 2019-12-05T05:23:44.611 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:44.602+0000 7fea6114d700 1 -- 172.21.15.42:0/3654884327 shutdown_connections 2019-12-05T05:23:44.612 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:44.602+0000 7fea6114d700 1 --2- 172.21.15.42:0/3654884327 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fea5c08df80 0x7fea5c09f2e0 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:23:44.612 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:44.602+0000 7fea6114d700 1 -- 172.21.15.42:0/3654884327 shutdown_connections 2019-12-05T05:23:44.612 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:44.602+0000 7fea6114d700 1 -- 172.21.15.42:0/3654884327 wait complete. 2019-12-05T05:23:44.612 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:44.602+0000 7fea6114d700 1 -- 172.21.15.42:0/3654884327 >> 172.21.15.42:0/3654884327 conn(0x7fea5c0aabe0 msgr2=0x7fea5c0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:23:44.615 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:44.606+0000 7fea6114d700 1 Processor -- start 2019-12-05T05:23:44.616 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:44.606+0000 7fea6114d700 1 -- start start 2019-12-05T05:23:44.616 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:44.606+0000 7fea6114d700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fea5c1213f0 0x7fea5c1190e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:23:44.616 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:44.606+0000 7fea6114d700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fea5c134270 0x7fea5c0702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:23:44.616 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:44.606+0000 7fea6114d700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea5c119d20 con 0x7fea5c116500 2019-12-05T05:23:44.616 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:44.606+0000 7fea6114d700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea5c119f50 con 0x7fea5c134270 2019-12-05T05:23:44.616 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:44.606+0000 7fea6114d700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea5c121ba0 con 0x7fea5c1213f0 2019-12-05T05:23:44.617 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:44.610+0000 7fea5ad9d700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fea5c116500 0x7fea5c133a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:32796/0 (socket says 172.21.15.42:32796) 2019-12-05T05:23:44.617 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:44.610+0000 7fea5ad9d700 1 -- 172.21.15.42:0/2623357892 learned_addr learned my addr 172.21.15.42:0/2623357892 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:23:44.617 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:44.610+0000 7fea5b59e700 1 --2- 172.21.15.42:0/2623357892 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fea5c1213f0 0x7fea5c1190e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:23:44.617 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:44.610+0000 7fea6094c700 1 --2- 172.21.15.42:0/2623357892 >> v2:172.21.15.40:3300/0 conn(0x7fea5c134270 0x7fea5c0702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:23:44.617 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:44.610+0000 7fea43fff700 1 -- 172.21.15.42:0/2623357892 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3773390890 0 0) 0x7fea5c119d20 con 0x7fea5c116500 2019-12-05T05:23:44.617 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:44.610+0000 7fea43fff700 1 -- 172.21.15.42:0/2623357892 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea3c0030b0 con 0x7fea5c116500 2019-12-05T05:23:44.617 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:44.610+0000 7fea6094c700 1 -- 172.21.15.42:0/2623357892 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fea5c1213f0 msgr2=0x7fea5c1190e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:44.618 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:44.610+0000 7fea6094c700 1 --2- 172.21.15.42:0/2623357892 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fea5c1213f0 0x7fea5c1190e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:23:44.618 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:44.610+0000 7fea6094c700 1 -- 172.21.15.42:0/2623357892 >> v1:172.21.15.42:6789/0 conn(0x7fea5c116500 legacy=0x7fea5c133a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:44.618 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:44.610+0000 7fea6094c700 1 -- 172.21.15.42:0/2623357892 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea5c070810 con 0x7fea5c134270 2019-12-05T05:23:44.618 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:44.610+0000 7fea5b59e700 1 --2- 172.21.15.42:0/2623357892 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fea5c1213f0 0x7fea5c1190e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:23:44.618 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:44.610+0000 7fea6094c700 1 --2- 172.21.15.42:0/2623357892 >> v2:172.21.15.40:3300/0 conn(0x7fea5c134270 0x7fea5c0702c0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fea54001210 tx=0x7fea54010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:44.618 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:44.610+0000 7fea6094c700 1 --2- 172.21.15.42:0/2623357892 >> v2:172.21.15.40:3300/0 conn(0x7fea5c134270 0x7fea5c0702c0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fea54001210 tx=0x7fea54010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:44.618 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:44.610+0000 7fea6094c700 1 --2- 172.21.15.42:0/2623357892 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fea5c134270 0x7fea5c0702c0 secure :-1 s=READY pgs=617 cs=0 l=1 rx=0x7fea54001210 tx=0x7fea54010b40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:23:44.619 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:44.610+0000 7fea6094c700 1 --2- 172.21.15.42:0/2623357892 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fea5c134270 0x7fea5c0702c0 secure :-1 s=THROTTLE_DONE pgs=617 cs=0 l=1 rx=0x7fea54001210 tx=0x7fea54010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:44.619 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:44.610+0000 7fea43fff700 1 -- 172.21.15.42:0/2623357892 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fea54006dc0 con 0x7fea5c134270 2019-12-05T05:23:44.619 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:44.610+0000 7fea6094c700 1 --2- 172.21.15.42:0/2623357892 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fea5c134270 0x7fea5c0702c0 secure :-1 s=THROTTLE_DONE pgs=617 cs=0 l=1 rx=0x7fea54001210 tx=0x7fea54010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:44.619 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:44.610+0000 7fea43fff700 1 -- 172.21.15.42:0/2623357892 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea540069e0 con 0x7fea5c134270 2019-12-05T05:23:44.619 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:44.610+0000 7fea6094c700 1 --2- 172.21.15.42:0/2623357892 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fea5c134270 0x7fea5c0702c0 secure :-1 s=THROTTLE_DONE pgs=617 cs=0 l=1 rx=0x7fea54001210 tx=0x7fea54010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:44.619 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:44.610+0000 7fea6114d700 1 -- 172.21.15.42:0/2623357892 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea5c070ae0 con 0x7fea5c134270 2019-12-05T05:23:44.619 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:44.610+0000 7fea43fff700 1 -- 172.21.15.42:0/2623357892 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fea54002620 con 0x7fea5c134270 2019-12-05T05:23:44.620 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:44.610+0000 7fea6114d700 1 -- 172.21.15.42:0/2623357892 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea5c071090 con 0x7fea5c134270 2019-12-05T05:23:44.620 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:44.610+0000 7fea6094c700 1 --2- 172.21.15.42:0/2623357892 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fea5c134270 0x7fea5c0702c0 secure :-1 s=THROTTLE_DONE pgs=617 cs=0 l=1 rx=0x7fea54001210 tx=0x7fea54010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:44.620 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:44.610+0000 7fea43fff700 1 -- 172.21.15.42:0/2623357892 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fea54002980 con 0x7fea5c134270 2019-12-05T05:23:44.620 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:44.610+0000 7fea6094c700 1 --2- 172.21.15.42:0/2623357892 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fea5c134270 0x7fea5c0702c0 secure :-1 s=THROTTLE_DONE pgs=617 cs=0 l=1 rx=0x7fea54001210 tx=0x7fea54010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:44.621 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:44.610+0000 7fea43fff700 1 --2- 172.21.15.42:0/2623357892 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fea3c03b200 0x7fea3c03d710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:23:44.621 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:44.610+0000 7fea43fff700 1 -- 172.21.15.42:0/2623357892 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 7149+0+0 (secure 0 0 0) 0x7fea5404b540 con 0x7fea5c134270 2019-12-05T05:23:44.621 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:44.614+0000 7fea5b59e700 1 --2- 172.21.15.42:0/2623357892 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fea3c03b200 0x7fea3c03d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:23:44.622 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:44.614+0000 7fea5b59e700 1 --2- 172.21.15.42:0/2623357892 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fea3c03b200 0x7fea3c03d710 crc :-1 s=READY pgs=343 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:23:44.622 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:44.614+0000 7fea6114d700 1 -- 172.21.15.42:0/2623357892 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea5c05d110 con 0x7fea5c134270 2019-12-05T05:23:44.626 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:44.622+0000 7fea6094c700 1 --2- 172.21.15.42:0/2623357892 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fea5c134270 0x7fea5c0702c0 secure :-1 s=THROTTLE_DONE pgs=617 cs=0 l=1 rx=0x7fea54001210 tx=0x7fea54010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:44.626 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:44.622+0000 7fea43fff700 1 -- 172.21.15.42:0/2623357892 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fea540325a0 con 0x7fea5c134270 2019-12-05T05:23:44.795 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:44.790+0000 7fea6114d700 1 -- 172.21.15.42:0/2623357892 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fea5c05b8c0 con 0x7fea5c134270 2019-12-05T05:23:44.795 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:44.794+0000 7fea6094c700 1 --2- 172.21.15.42:0/2623357892 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fea5c134270 0x7fea5c0702c0 secure :-1 s=THROTTLE_DONE pgs=617 cs=0 l=1 rx=0x7fea54001210 tx=0x7fea54010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:44.796 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:44.794+0000 7fea43fff700 1 -- 172.21.15.42:0/2623357892 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 98 v98) v1 ==== 94+0+2108 (secure 0 0 0) 0x7fea54032800 con 0x7fea5c134270 2019-12-05T05:23:44.798 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:23:44.798 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":98,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5177,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/3208416000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3208416000},{"type":"v1","addr":"172.21.15.40:6833","nonce":3208416000}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":98}],"filesystems":[{"mdsmap":{"epoch":97,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:23:07.988779+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5123},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5123":{"gid":5123,"name":"a","rank":0,"incarnation":94,"state":"up:active","state_seq":37,"addr":"172.21.15.42:6835/1214659805","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1214659805},{"type":"v1","addr":"172.21.15.42:6835","nonce":1214659805}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:23:44.798 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 98 2019-12-05T05:23:44.800 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:44.798+0000 7fea6114d700 1 -- 172.21.15.42:0/2623357892 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fea3c03b200 msgr2=0x7fea3c03d710 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:44.801 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:44.798+0000 7fea6114d700 1 --2- 172.21.15.42:0/2623357892 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fea3c03b200 0x7fea3c03d710 crc :-1 s=READY pgs=343 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:23:44.801 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:44.798+0000 7fea6114d700 1 -- 172.21.15.42:0/2623357892 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fea5c134270 msgr2=0x7fea5c0702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:44.801 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:44.798+0000 7fea6114d700 1 --2- 172.21.15.42:0/2623357892 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fea5c134270 0x7fea5c0702c0 secure :-1 s=READY pgs=617 cs=0 l=1 rx=0x7fea54001210 tx=0x7fea54010b40).stop 2019-12-05T05:23:44.801 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:44.798+0000 7fea6114d700 1 -- 172.21.15.42:0/2623357892 shutdown_connections 2019-12-05T05:23:44.801 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:44.798+0000 7fea6114d700 1 --2- 172.21.15.42:0/2623357892 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fea3c03b200 0x7fea3c03d710 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:23:44.801 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:44.798+0000 7fea6114d700 1 --2- 172.21.15.42:0/2623357892 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fea5c134270 0x7fea5c0702c0 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:23:44.802 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:44.798+0000 7fea6114d700 1 --2- 172.21.15.42:0/2623357892 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fea5c1213f0 0x7fea5c1190e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:23:44.802 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:44.798+0000 7fea6114d700 1 -- 172.21.15.42:0/2623357892 shutdown_connections 2019-12-05T05:23:44.802 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:44.798+0000 7fea6114d700 1 -- 172.21.15.42:0/2623357892 wait complete. 2019-12-05T05:23:44.802 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:44.798+0000 7fea6114d700 1 -- 172.21.15.42:0/2623357892 >> 172.21.15.42:0/2623357892 conn(0x7fea5c0a9770 msgr2=0x7fea5c0a6b90 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:23:44.810 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T05:23:46.812 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:23:46.923 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:46.914+0000 7f55d0e33700 1 Processor -- start 2019-12-05T05:23:46.924 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:46.914+0000 7f55d0e33700 1 -- start start 2019-12-05T05:23:46.924 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:46.914+0000 7f55d0e33700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f55cc0a7740 0x7f55cc08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:23:46.924 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:46.914+0000 7f55d0e33700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f55cc07e720 0x7f55cc07eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:23:46.924 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:46.914+0000 7f55d0e33700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55cc0a0710 con 0x7f55cc09f2e0 2019-12-05T05:23:46.924 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:46.914+0000 7f55d0e33700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55cc0a0b00 con 0x7f55cc07e720 2019-12-05T05:23:46.925 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:46.914+0000 7f55d0e33700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55cc08e5b0 con 0x7f55cc0a7740 2019-12-05T05:23:46.925 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:46.918+0000 7f55cad9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f55cc0a7740 0x7f55cc08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:23:46.925 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:46.918+0000 7f55cb59e700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f55cc09f2e0 0x7f55cc09f710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:32804/0 (socket says 172.21.15.42:32804) 2019-12-05T05:23:46.925 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:46.918+0000 7f55cb59e700 1 -- 172.21.15.42:0/4234430881 learned_addr learned my addr 172.21.15.42:0/4234430881 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:23:46.925 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:46.918+0000 7f55c259c700 1 --2- 172.21.15.42:0/4234430881 >> v2:172.21.15.40:3300/0 conn(0x7f55cc07e720 0x7f55cc07eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:23:46.925 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:46.918+0000 7f55cad9d700 1 -- 172.21.15.42:0/4234430881 >> v2:172.21.15.40:3300/0 conn(0x7f55cc07e720 msgr2=0x7f55cc07eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:23:46.925 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:46.918+0000 7f55cad9d700 1 --2- 172.21.15.42:0/4234430881 >> v2:172.21.15.40:3300/0 conn(0x7f55cc07e720 0x7f55cc07eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:23:46.926 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:46.918+0000 7f55cad9d700 1 -- 172.21.15.42:0/4234430881 >> v1:172.21.15.42:6789/0 conn(0x7f55cc09f2e0 legacy=0x7f55cc09f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:46.926 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:46.918+0000 7f55cad9d700 1 -- 172.21.15.42:0/4234430881 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55cc07f0a0 con 0x7f55cc0a7740 2019-12-05T05:23:46.926 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:46.918+0000 7f55cad9d700 1 --2- 172.21.15.42:0/4234430881 >> v2:172.21.15.42:3300/0 conn(0x7f55cc0a7740 0x7f55cc08df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f55c4001490 tx=0x7f55c40014d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:46.926 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:46.918+0000 7f55cad9d700 1 --2- 172.21.15.42:0/4234430881 >> v2:172.21.15.42:3300/0 conn(0x7f55cc0a7740 0x7f55cc08df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f55c4001490 tx=0x7f55c40014d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:46.926 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:46.918+0000 7f55cad9d700 1 --2- 172.21.15.42:0/4234430881 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f55cc0a7740 0x7f55cc08df80 secure :-1 s=READY pgs=779 cs=0 l=1 rx=0x7f55c4001490 tx=0x7f55c40014d0).ready entity=mon.1 client_cookie=cc1c1862e09d1779 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:23:46.926 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:46.918+0000 7f55cad9d700 1 --2- 172.21.15.42:0/4234430881 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f55cc0a7740 0x7f55cc08df80 secure :-1 s=THROTTLE_DONE pgs=779 cs=0 l=1 rx=0x7f55c4001490 tx=0x7f55c40014d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:46.926 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:46.918+0000 7f55ca59c700 1 -- 172.21.15.42:0/4234430881 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f55bc008d50 con 0x7f55cc0a7740 2019-12-05T05:23:46.926 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:46.918+0000 7f55cad9d700 1 --2- 172.21.15.42:0/4234430881 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f55cc0a7740 0x7f55cc08df80 secure :-1 s=THROTTLE_DONE pgs=779 cs=0 l=1 rx=0x7f55c4001490 tx=0x7f55c40014d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:46.927 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:46.918+0000 7f55ca59c700 1 -- 172.21.15.42:0/4234430881 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55bc008970 con 0x7f55cc0a7740 2019-12-05T05:23:46.927 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:46.918+0000 7f55cad9d700 1 --2- 172.21.15.42:0/4234430881 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f55cc0a7740 0x7f55cc08df80 secure :-1 s=THROTTLE_DONE pgs=779 cs=0 l=1 rx=0x7f55c4001490 tx=0x7f55c40014d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:46.927 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:46.918+0000 7f55ca59c700 1 -- 172.21.15.42:0/4234430881 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f55bc011860 con 0x7f55cc0a7740 2019-12-05T05:23:46.927 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:46.918+0000 7f55d0e33700 1 -- 172.21.15.42:0/4234430881 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f55cc0a7740 msgr2=0x7f55cc08df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:46.927 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:46.918+0000 7f55d0e33700 1 --2- 172.21.15.42:0/4234430881 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f55cc0a7740 0x7f55cc08df80 secure :-1 s=READY pgs=779 cs=0 l=1 rx=0x7f55c4001490 tx=0x7f55c40014d0).stop 2019-12-05T05:23:46.927 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:46.918+0000 7f55d0e33700 1 -- 172.21.15.42:0/4234430881 shutdown_connections 2019-12-05T05:23:46.927 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:46.918+0000 7f55d0e33700 1 --2- 172.21.15.42:0/4234430881 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f55cc0a7740 0x7f55cc08df80 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:23:46.928 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:46.918+0000 7f55d0e33700 1 --2- 172.21.15.42:0/4234430881 >> v2:172.21.15.40:3300/0 conn(0x7f55cc07e720 0x7f55cc07eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:23:46.928 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:46.918+0000 7f55d0e33700 1 -- 172.21.15.42:0/4234430881 shutdown_connections 2019-12-05T05:23:46.928 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:46.918+0000 7f55d0e33700 1 -- 172.21.15.42:0/4234430881 wait complete. 2019-12-05T05:23:46.928 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:46.918+0000 7f55d0e33700 1 -- 172.21.15.42:0/4234430881 >> 172.21.15.42:0/4234430881 conn(0x7f55cc0aabe0 msgr2=0x7f55cc0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:23:46.932 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:46.922+0000 7f55d0e33700 1 Processor -- start 2019-12-05T05:23:46.932 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:46.922+0000 7f55d0e33700 1 -- start start 2019-12-05T05:23:46.932 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:46.922+0000 7f55d0e33700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f55cc0a59e0 0x7f55cc0a5e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:23:46.933 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:46.922+0000 7f55d0e33700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f55cc07c7c0 0x7f55cc07cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:23:46.933 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:46.922+0000 7f55d0e33700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55cc07d140 con 0x7f55cc116500 2019-12-05T05:23:46.933 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:46.922+0000 7f55d0e33700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55cc146c50 con 0x7f55cc07c7c0 2019-12-05T05:23:46.933 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:46.922+0000 7f55d0e33700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55cc146ef0 con 0x7f55cc0a59e0 2019-12-05T05:23:46.933 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:46.922+0000 7f55cad9d700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f55cc116500 0x7f55cc0a6b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:32806/0 (socket says 172.21.15.42:32806) 2019-12-05T05:23:46.933 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:46.922+0000 7f55cad9d700 1 -- 172.21.15.42:0/130820940 learned_addr learned my addr 172.21.15.42:0/130820940 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:23:46.933 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:46.922+0000 7f55c259c700 1 --2- 172.21.15.42:0/130820940 >> v2:172.21.15.42:3300/0 conn(0x7f55cc0a59e0 0x7f55cc0a5e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:23:46.934 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:46.926+0000 7f55cb59e700 1 --2- 172.21.15.42:0/130820940 >> v2:172.21.15.40:3300/0 conn(0x7f55cc07c7c0 0x7f55cc07cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:23:46.934 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:46.926+0000 7f55c3fff700 1 -- 172.21.15.42:0/130820940 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3977996998 0 0) 0x7f55cc07d140 con 0x7f55cc116500 2019-12-05T05:23:46.934 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:46.926+0000 7f55c259c700 1 -- 172.21.15.42:0/130820940 >> v2:172.21.15.40:3300/0 conn(0x7f55cc07c7c0 msgr2=0x7f55cc07cbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:46.934 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:46.926+0000 7f55c259c700 1 --2- 172.21.15.42:0/130820940 >> v2:172.21.15.40:3300/0 conn(0x7f55cc07c7c0 0x7f55cc07cbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:23:46.934 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:46.926+0000 7f55c259c700 1 -- 172.21.15.42:0/130820940 >> v1:172.21.15.42:6789/0 conn(0x7f55cc116500 legacy=0x7f55cc0a6b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:46.934 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:46.926+0000 7f55c259c700 1 -- 172.21.15.42:0/130820940 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55cc147430 con 0x7f55cc0a59e0 2019-12-05T05:23:46.934 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:46.926+0000 7f55c259c700 1 --2- 172.21.15.42:0/130820940 >> v2:172.21.15.42:3300/0 conn(0x7f55cc0a59e0 0x7f55cc0a5e10 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f55b4002f30 tx=0x7f55b4010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:46.935 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:46.926+0000 7f55c259c700 1 --2- 172.21.15.42:0/130820940 >> v2:172.21.15.42:3300/0 conn(0x7f55cc0a59e0 0x7f55cc0a5e10 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f55b4002f30 tx=0x7f55b4010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:46.935 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:46.926+0000 7f55c259c700 1 --2- 172.21.15.42:0/130820940 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f55cc0a59e0 0x7f55cc0a5e10 secure :-1 s=READY pgs=781 cs=0 l=1 rx=0x7f55b4002f30 tx=0x7f55b4010ac0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:23:46.935 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:46.926+0000 7f55c259c700 1 --2- 172.21.15.42:0/130820940 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f55cc0a59e0 0x7f55cc0a5e10 secure :-1 s=THROTTLE_DONE pgs=781 cs=0 l=1 rx=0x7f55b4002f30 tx=0x7f55b4010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:46.935 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:46.926+0000 7f55c3fff700 1 -- 172.21.15.42:0/130820940 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f55b4012950 con 0x7f55cc0a59e0 2019-12-05T05:23:46.935 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:46.926+0000 7f55c259c700 1 --2- 172.21.15.42:0/130820940 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f55cc0a59e0 0x7f55cc0a5e10 secure :-1 s=THROTTLE_DONE pgs=781 cs=0 l=1 rx=0x7f55b4002f30 tx=0x7f55b4010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:46.935 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:46.926+0000 7f55c3fff700 1 -- 172.21.15.42:0/130820940 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55b4012570 con 0x7f55cc0a59e0 2019-12-05T05:23:46.935 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:46.926+0000 7f55c259c700 1 --2- 172.21.15.42:0/130820940 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f55cc0a59e0 0x7f55cc0a5e10 secure :-1 s=THROTTLE_DONE pgs=781 cs=0 l=1 rx=0x7f55b4002f30 tx=0x7f55b4010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:46.935 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:46.926+0000 7f55d0e33700 1 -- 172.21.15.42:0/130820940 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55cc11c430 con 0x7f55cc0a59e0 2019-12-05T05:23:46.936 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:46.926+0000 7f55c3fff700 1 -- 172.21.15.42:0/130820940 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f55b4005540 con 0x7f55cc0a59e0 2019-12-05T05:23:46.936 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:46.926+0000 7f55d0e33700 1 -- 172.21.15.42:0/130820940 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55cc137cb0 con 0x7f55cc0a59e0 2019-12-05T05:23:46.936 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:46.926+0000 7f55c259c700 1 --2- 172.21.15.42:0/130820940 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f55cc0a59e0 0x7f55cc0a5e10 secure :-1 s=THROTTLE_DONE pgs=781 cs=0 l=1 rx=0x7f55b4002f30 tx=0x7f55b4010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:46.936 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:46.926+0000 7f55c3fff700 1 -- 172.21.15.42:0/130820940 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f55b4007180 con 0x7f55cc0a59e0 2019-12-05T05:23:46.936 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:46.926+0000 7f55c259c700 1 --2- 172.21.15.42:0/130820940 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f55cc0a59e0 0x7f55cc0a5e10 secure :-1 s=THROTTLE_DONE pgs=781 cs=0 l=1 rx=0x7f55b4002f30 tx=0x7f55b4010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:46.936 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:46.926+0000 7f55c3fff700 1 --2- 172.21.15.42:0/130820940 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f55b003aca0 0x7f55b003d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:23:46.936 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:46.926+0000 7f55c3fff700 1 -- 172.21.15.42:0/130820940 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 7149+0+0 (secure 0 0 0) 0x7f55b404afa0 con 0x7f55cc0a59e0 2019-12-05T05:23:46.937 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:46.926+0000 7f55cad9d700 1 --2- 172.21.15.42:0/130820940 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f55b003aca0 0x7f55b003d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:23:46.937 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:46.926+0000 7f55cad9d700 1 --2- 172.21.15.42:0/130820940 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f55b003aca0 0x7f55b003d1b0 crc :-1 s=READY pgs=344 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:23:46.937 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:46.930+0000 7f55d0e33700 1 -- 172.21.15.42:0/130820940 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55cc05d110 con 0x7f55cc0a59e0 2019-12-05T05:23:46.941 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:46.938+0000 7f55c259c700 1 --2- 172.21.15.42:0/130820940 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f55cc0a59e0 0x7f55cc0a5e10 secure :-1 s=THROTTLE_DONE pgs=781 cs=0 l=1 rx=0x7f55b4002f30 tx=0x7f55b4010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:46.945 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:46.938+0000 7f55c3fff700 1 -- 172.21.15.42:0/130820940 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f55b4032000 con 0x7f55cc0a59e0 2019-12-05T05:23:47.111 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:47.106+0000 7f55d0e33700 1 -- 172.21.15.42:0/130820940 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f55cc05b8c0 con 0x7f55cc0a59e0 2019-12-05T05:23:47.112 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:47.110+0000 7f55c259c700 1 --2- 172.21.15.42:0/130820940 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f55cc0a59e0 0x7f55cc0a5e10 secure :-1 s=THROTTLE_DONE pgs=781 cs=0 l=1 rx=0x7f55b4002f30 tx=0x7f55b4010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:47.112 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:47.110+0000 7f55c3fff700 1 -- 172.21.15.42:0/130820940 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 98 v98) v1 ==== 94+0+2108 (secure 0 0 0) 0x7f55b4032260 con 0x7f55cc0a59e0 2019-12-05T05:23:47.113 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 98 2019-12-05T05:23:47.113 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:23:47.113 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":98,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5177,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/3208416000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3208416000},{"type":"v1","addr":"172.21.15.40:6833","nonce":3208416000}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":98}],"filesystems":[{"mdsmap":{"epoch":97,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:23:07.988779+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5123},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5123":{"gid":5123,"name":"a","rank":0,"incarnation":94,"state":"up:active","state_seq":37,"addr":"172.21.15.42:6835/1214659805","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1214659805},{"type":"v1","addr":"172.21.15.42:6835","nonce":1214659805}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:23:47.114 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:47.110+0000 7f55d0e33700 1 -- 172.21.15.42:0/130820940 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f55b003aca0 msgr2=0x7f55b003d1b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:47.115 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:47.110+0000 7f55d0e33700 1 --2- 172.21.15.42:0/130820940 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f55b003aca0 0x7f55b003d1b0 crc :-1 s=READY pgs=344 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:23:47.115 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:47.110+0000 7f55d0e33700 1 -- 172.21.15.42:0/130820940 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f55cc0a59e0 msgr2=0x7f55cc0a5e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:47.115 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:47.110+0000 7f55d0e33700 1 --2- 172.21.15.42:0/130820940 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f55cc0a59e0 0x7f55cc0a5e10 secure :-1 s=READY pgs=781 cs=0 l=1 rx=0x7f55b4002f30 tx=0x7f55b4010ac0).stop 2019-12-05T05:23:47.116 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:47.110+0000 7f55d0e33700 1 -- 172.21.15.42:0/130820940 shutdown_connections 2019-12-05T05:23:47.116 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:47.110+0000 7f55d0e33700 1 --2- 172.21.15.42:0/130820940 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f55b003aca0 0x7f55b003d1b0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:23:47.116 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:47.110+0000 7f55d0e33700 1 --2- 172.21.15.42:0/130820940 >> v2:172.21.15.40:3300/0 conn(0x7f55cc07c7c0 0x7f55cc07cbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:23:47.116 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:47.110+0000 7f55d0e33700 1 --2- 172.21.15.42:0/130820940 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f55cc0a59e0 0x7f55cc0a5e10 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:23:47.116 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:47.110+0000 7f55d0e33700 1 -- 172.21.15.42:0/130820940 shutdown_connections 2019-12-05T05:23:47.117 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:47.110+0000 7f55d0e33700 1 -- 172.21.15.42:0/130820940 wait complete. 2019-12-05T05:23:47.117 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:47.110+0000 7f55d0e33700 1 -- 172.21.15.42:0/130820940 >> 172.21.15.42:0/130820940 conn(0x7f55cc0aabe0 msgr2=0x7f55cc09f620 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:23:47.125 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T05:23:49.126 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:23:49.241 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:49.234+0000 7fc90a918700 1 Processor -- start 2019-12-05T05:23:49.241 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:49.234+0000 7fc90a918700 1 -- start start 2019-12-05T05:23:49.241 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:49.234+0000 7fc90a918700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc90408df80 0x7fc90409f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:23:49.241 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:49.234+0000 7fc90a918700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc90409f830 0x7fc90414efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:23:49.242 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:49.234+0000 7fc90a918700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc90409fc60 con 0x7fc9040a7740 2019-12-05T05:23:49.242 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:49.234+0000 7fc90a918700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9040a0b00 con 0x7fc90408df80 2019-12-05T05:23:49.242 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:49.234+0000 7fc90a918700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc90408e5b0 con 0x7fc90409f830 2019-12-05T05:23:49.242 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:49.234+0000 7fc9096b6700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc90409f830 0x7fc90414efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:23:49.242 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:49.234+0000 7fc9096b6700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc90409f830 0x7fc90414efe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:32778/0 (socket says 172.21.15.42:32778) 2019-12-05T05:23:49.242 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:49.234+0000 7fc9096b6700 1 -- 172.21.15.42:0/4050391298 learned_addr learned my addr 172.21.15.42:0/4050391298 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:23:49.243 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:49.234+0000 7fc903fff700 1 --2- 172.21.15.42:0/4050391298 >> v2:172.21.15.40:3300/0 conn(0x7fc90408df80 0x7fc90409f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:23:49.243 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:49.234+0000 7fc9096b6700 1 -- 172.21.15.42:0/4050391298 >> v2:172.21.15.40:3300/0 conn(0x7fc90408df80 msgr2=0x7fc90409f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:23:49.243 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:49.234+0000 7fc9096b6700 1 --2- 172.21.15.42:0/4050391298 >> v2:172.21.15.40:3300/0 conn(0x7fc90408df80 0x7fc90409f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:23:49.243 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:49.234+0000 7fc9096b6700 1 -- 172.21.15.42:0/4050391298 >> v1:172.21.15.40:6789/0 conn(0x7fc9040a7740 legacy=0x7fc90407e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:49.243 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:49.234+0000 7fc9096b6700 1 -- 172.21.15.42:0/4050391298 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9040a0710 con 0x7fc90409f830 2019-12-05T05:23:49.243 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:49.234+0000 7fc9096b6700 1 --2- 172.21.15.42:0/4050391298 >> v2:172.21.15.42:3300/0 conn(0x7fc90409f830 0x7fc90414efe0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fc8fc0032b0 tx=0x7fc8fc0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:49.243 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:49.234+0000 7fc9096b6700 1 --2- 172.21.15.42:0/4050391298 >> v2:172.21.15.42:3300/0 conn(0x7fc90409f830 0x7fc90414efe0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fc8fc0032b0 tx=0x7fc8fc0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:49.243 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:49.234+0000 7fc9096b6700 1 --2- 172.21.15.42:0/4050391298 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc90409f830 0x7fc90414efe0 secure :-1 s=READY pgs=782 cs=0 l=1 rx=0x7fc8fc0032b0 tx=0x7fc8fc0100b0).ready entity=mon.1 client_cookie=430ababfdeab4538 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:23:49.244 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:49.234+0000 7fc9096b6700 1 --2- 172.21.15.42:0/4050391298 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc90409f830 0x7fc90414efe0 secure :-1 s=THROTTLE_DONE pgs=782 cs=0 l=1 rx=0x7fc8fc0032b0 tx=0x7fc8fc0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:49.244 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:49.234+0000 7fc9096b6700 1 --2- 172.21.15.42:0/4050391298 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc90409f830 0x7fc90414efe0 secure :-1 s=THROTTLE_DONE pgs=782 cs=0 l=1 rx=0x7fc8fc0032b0 tx=0x7fc8fc0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:49.244 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:49.234+0000 7fc9037fe700 1 -- 172.21.15.42:0/4050391298 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc8fc010d60 con 0x7fc90409f830 2019-12-05T05:23:49.244 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:49.234+0000 7fc9096b6700 1 --2- 172.21.15.42:0/4050391298 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc90409f830 0x7fc90414efe0 secure :-1 s=THROTTLE_DONE pgs=782 cs=0 l=1 rx=0x7fc8fc0032b0 tx=0x7fc8fc0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:49.244 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:49.234+0000 7fc9037fe700 1 -- 172.21.15.42:0/4050391298 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8fc010980 con 0x7fc90409f830 2019-12-05T05:23:49.244 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:49.234+0000 7fc9037fe700 1 -- 172.21.15.42:0/4050391298 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc8fc0114a0 con 0x7fc90409f830 2019-12-05T05:23:49.244 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:49.234+0000 7fc90a918700 1 -- 172.21.15.42:0/4050391298 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc90409f830 msgr2=0x7fc90414efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:49.245 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:49.234+0000 7fc90a918700 1 --2- 172.21.15.42:0/4050391298 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc90409f830 0x7fc90414efe0 secure :-1 s=READY pgs=782 cs=0 l=1 rx=0x7fc8fc0032b0 tx=0x7fc8fc0100b0).stop 2019-12-05T05:23:49.245 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:49.238+0000 7fc90a918700 1 -- 172.21.15.42:0/4050391298 shutdown_connections 2019-12-05T05:23:49.245 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:49.238+0000 7fc90a918700 1 --2- 172.21.15.42:0/4050391298 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc90409f830 0x7fc90414efe0 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:23:49.245 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:49.238+0000 7fc90a918700 1 --2- 172.21.15.42:0/4050391298 >> v2:172.21.15.40:3300/0 conn(0x7fc90408df80 0x7fc90409f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:23:49.245 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:49.238+0000 7fc90a918700 1 -- 172.21.15.42:0/4050391298 shutdown_connections 2019-12-05T05:23:49.245 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:49.238+0000 7fc90a918700 1 -- 172.21.15.42:0/4050391298 wait complete. 2019-12-05T05:23:49.245 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:49.238+0000 7fc90a918700 1 -- 172.21.15.42:0/4050391298 >> 172.21.15.42:0/4050391298 conn(0x7fc9040aabe0 msgr2=0x7fc9040a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:23:49.249 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:49.238+0000 7fc90a918700 1 Processor -- start 2019-12-05T05:23:49.249 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:49.238+0000 7fc90a918700 1 -- start start 2019-12-05T05:23:49.249 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:49.238+0000 7fc90a918700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc904063e70 0x7fc904112af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:23:49.250 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:49.238+0000 7fc90a918700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc904064880 0x7fc90411f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:23:49.250 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:49.238+0000 7fc90a918700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc904065bf0 con 0x7fc904125940 2019-12-05T05:23:49.250 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:49.238+0000 7fc90a918700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc904064340 con 0x7fc904064880 2019-12-05T05:23:49.250 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:49.238+0000 7fc90a918700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9040645e0 con 0x7fc904063e70 2019-12-05T05:23:49.250 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:49.238+0000 7fc908eb5700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc904063e70 0x7fc904112af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:23:49.250 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:49.242+0000 7fc903fff700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc904064880 0x7fc90411f8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:23:49.251 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:49.242+0000 7fc9096b6700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fc904125940 0x7fc9040653f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:42264/0 (socket says 172.21.15.42:42264) 2019-12-05T05:23:49.251 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:49.242+0000 7fc9096b6700 1 -- 172.21.15.42:0/970687493 learned_addr learned my addr 172.21.15.42:0/970687493 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:23:49.251 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:49.242+0000 7fc9017fa700 1 -- 172.21.15.42:0/970687493 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1849782578 0 0) 0x7fc904065bf0 con 0x7fc904125940 2019-12-05T05:23:49.251 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:49.242+0000 7fc9017fa700 1 -- 172.21.15.42:0/970687493 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8e40030b0 con 0x7fc904125940 2019-12-05T05:23:49.251 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:49.242+0000 7fc903fff700 1 -- 172.21.15.42:0/970687493 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc904063e70 msgr2=0x7fc904112af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:49.251 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:49.242+0000 7fc903fff700 1 --2- 172.21.15.42:0/970687493 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc904063e70 0x7fc904112af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:23:49.251 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:49.242+0000 7fc903fff700 1 -- 172.21.15.42:0/970687493 >> v1:172.21.15.40:6789/0 conn(0x7fc904125940 legacy=0x7fc9040653f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:49.252 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:49.242+0000 7fc903fff700 1 -- 172.21.15.42:0/970687493 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc904113040 con 0x7fc904064880 2019-12-05T05:23:49.252 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:49.242+0000 7fc908eb5700 1 --2- 172.21.15.42:0/970687493 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc904063e70 0x7fc904112af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:23:49.252 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:49.242+0000 7fc903fff700 1 --2- 172.21.15.42:0/970687493 >> v2:172.21.15.40:3300/0 conn(0x7fc904064880 0x7fc90411f8f0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fc904064500 tx=0x7fc8f0010ba0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:49.252 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:49.242+0000 7fc903fff700 1 --2- 172.21.15.42:0/970687493 >> v2:172.21.15.40:3300/0 conn(0x7fc904064880 0x7fc90411f8f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fc904064500 tx=0x7fc8f0010ba0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:49.252 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:49.242+0000 7fc903fff700 1 --2- 172.21.15.42:0/970687493 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc904064880 0x7fc90411f8f0 secure :-1 s=READY pgs=620 cs=0 l=1 rx=0x7fc904064500 tx=0x7fc8f0010ba0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:23:49.252 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:49.242+0000 7fc903fff700 1 --2- 172.21.15.42:0/970687493 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc904064880 0x7fc90411f8f0 secure :-1 s=THROTTLE_DONE pgs=620 cs=0 l=1 rx=0x7fc904064500 tx=0x7fc8f0010ba0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:49.252 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:49.242+0000 7fc9017fa700 1 -- 172.21.15.42:0/970687493 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc8f0012610 con 0x7fc904064880 2019-12-05T05:23:49.253 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:49.242+0000 7fc903fff700 1 --2- 172.21.15.42:0/970687493 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc904064880 0x7fc90411f8f0 secure :-1 s=THROTTLE_DONE pgs=620 cs=0 l=1 rx=0x7fc904064500 tx=0x7fc8f0010ba0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:49.253 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:49.242+0000 7fc9017fa700 1 -- 172.21.15.42:0/970687493 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8f0007150 con 0x7fc904064880 2019-12-05T05:23:49.253 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:49.242+0000 7fc90a918700 1 -- 172.21.15.42:0/970687493 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc90411fe40 con 0x7fc904064880 2019-12-05T05:23:49.253 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:49.242+0000 7fc903fff700 1 --2- 172.21.15.42:0/970687493 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc904064880 0x7fc90411f8f0 secure :-1 s=THROTTLE_DONE pgs=620 cs=0 l=1 rx=0x7fc904064500 tx=0x7fc8f0010ba0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:49.253 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:49.242+0000 7fc9017fa700 1 -- 172.21.15.42:0/970687493 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc8f0012bf0 con 0x7fc904064880 2019-12-05T05:23:49.253 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:49.242+0000 7fc90a918700 1 -- 172.21.15.42:0/970687493 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc904125f90 con 0x7fc904064880 2019-12-05T05:23:49.253 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:49.242+0000 7fc903fff700 1 --2- 172.21.15.42:0/970687493 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc904064880 0x7fc90411f8f0 secure :-1 s=THROTTLE_DONE pgs=620 cs=0 l=1 rx=0x7fc904064500 tx=0x7fc8f0010ba0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:49.254 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:49.246+0000 7fc9017fa700 1 -- 172.21.15.42:0/970687493 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fc8f0005540 con 0x7fc904064880 2019-12-05T05:23:49.254 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:49.246+0000 7fc903fff700 1 --2- 172.21.15.42:0/970687493 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc904064880 0x7fc90411f8f0 secure :-1 s=THROTTLE_DONE pgs=620 cs=0 l=1 rx=0x7fc904064500 tx=0x7fc8f0010ba0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:49.254 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:49.246+0000 7fc9017fa700 1 --2- 172.21.15.42:0/970687493 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc8e403b210 0x7fc8e403d720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:23:49.254 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:49.246+0000 7fc9017fa700 1 -- 172.21.15.42:0/970687493 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 7149+0+0 (secure 0 0 0) 0x7fc8f004b150 con 0x7fc904064880 2019-12-05T05:23:49.254 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:49.246+0000 7fc908eb5700 1 --2- 172.21.15.42:0/970687493 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc8e403b210 0x7fc8e403d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:23:49.254 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:49.246+0000 7fc908eb5700 1 --2- 172.21.15.42:0/970687493 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc8e403b210 0x7fc8e403d720 crc :-1 s=READY pgs=345 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:23:49.255 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:49.250+0000 7fc90a918700 1 -- 172.21.15.42:0/970687493 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc90405d110 con 0x7fc904064880 2019-12-05T05:23:49.262 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:49.258+0000 7fc903fff700 1 --2- 172.21.15.42:0/970687493 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc904064880 0x7fc90411f8f0 secure :-1 s=THROTTLE_DONE pgs=620 cs=0 l=1 rx=0x7fc904064500 tx=0x7fc8f0010ba0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:49.262 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:49.258+0000 7fc9017fa700 1 -- 172.21.15.42:0/970687493 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fc8f00321b0 con 0x7fc904064880 2019-12-05T05:23:49.431 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:49.426+0000 7fc90a918700 1 -- 172.21.15.42:0/970687493 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc90405b8c0 con 0x7fc904064880 2019-12-05T05:23:49.432 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:49.430+0000 7fc903fff700 1 --2- 172.21.15.42:0/970687493 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc904064880 0x7fc90411f8f0 secure :-1 s=THROTTLE_DONE pgs=620 cs=0 l=1 rx=0x7fc904064500 tx=0x7fc8f0010ba0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:49.432 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:49.430+0000 7fc9017fa700 1 -- 172.21.15.42:0/970687493 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 98 v98) v1 ==== 94+0+2108 (secure 0 0 0) 0x7fc8f0032410 con 0x7fc904064880 2019-12-05T05:23:49.435 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 98 2019-12-05T05:23:49.435 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:23:49.435 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":98,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5177,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/3208416000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3208416000},{"type":"v1","addr":"172.21.15.40:6833","nonce":3208416000}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":98}],"filesystems":[{"mdsmap":{"epoch":97,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:23:07.988779+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5123},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5123":{"gid":5123,"name":"a","rank":0,"incarnation":94,"state":"up:active","state_seq":37,"addr":"172.21.15.42:6835/1214659805","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1214659805},{"type":"v1","addr":"172.21.15.42:6835","nonce":1214659805}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:23:49.438 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:49.434+0000 7fc90a918700 1 -- 172.21.15.42:0/970687493 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc8e403b210 msgr2=0x7fc8e403d720 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:49.438 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:49.434+0000 7fc90a918700 1 --2- 172.21.15.42:0/970687493 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc8e403b210 0x7fc8e403d720 crc :-1 s=READY pgs=345 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:23:49.438 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:49.434+0000 7fc90a918700 1 -- 172.21.15.42:0/970687493 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc904064880 msgr2=0x7fc90411f8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:49.438 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:49.434+0000 7fc90a918700 1 --2- 172.21.15.42:0/970687493 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc904064880 0x7fc90411f8f0 secure :-1 s=READY pgs=620 cs=0 l=1 rx=0x7fc904064500 tx=0x7fc8f0010ba0).stop 2019-12-05T05:23:49.439 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:49.434+0000 7fc90a918700 1 -- 172.21.15.42:0/970687493 shutdown_connections 2019-12-05T05:23:49.439 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:49.434+0000 7fc90a918700 1 --2- 172.21.15.42:0/970687493 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc8e403b210 0x7fc8e403d720 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:23:49.439 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:49.434+0000 7fc90a918700 1 --2- 172.21.15.42:0/970687493 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc904063e70 0x7fc904112af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:23:49.439 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:49.434+0000 7fc90a918700 1 --2- 172.21.15.42:0/970687493 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc904064880 0x7fc90411f8f0 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:23:49.439 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:49.434+0000 7fc90a918700 1 -- 172.21.15.42:0/970687493 shutdown_connections 2019-12-05T05:23:49.440 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:49.434+0000 7fc90a918700 1 -- 172.21.15.42:0/970687493 wait complete. 2019-12-05T05:23:49.440 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:49.434+0000 7fc90a918700 1 -- 172.21.15.42:0/970687493 >> 172.21.15.42:0/970687493 conn(0x7fc90408fda0 msgr2=0x7fc9040a9820 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:23:49.448 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T05:23:51.449 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:23:51.555 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:51.546+0000 7f52cfb1c700 1 Processor -- start 2019-12-05T05:23:51.555 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:51.546+0000 7f52cfb1c700 1 -- start start 2019-12-05T05:23:51.556 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:51.550+0000 7f52cfb1c700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f52c808df80 0x7f52c809f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:23:51.556 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:51.550+0000 7f52cfb1c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f52c809f830 0x7f52c814efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:23:51.556 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:51.550+0000 7f52cfb1c700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52c809fc60 con 0x7f52c80a7740 2019-12-05T05:23:51.556 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:51.550+0000 7f52cfb1c700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52c80a0b00 con 0x7f52c808df80 2019-12-05T05:23:51.556 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:51.550+0000 7f52cfb1c700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52c808e5b0 con 0x7f52c809f830 2019-12-05T05:23:51.556 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:51.550+0000 7f52ce0b9700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f52c80a7740 0x7f52c807e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:32830/0 (socket says 172.21.15.42:32830) 2019-12-05T05:23:51.556 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:51.550+0000 7f52ce0b9700 1 -- 172.21.15.42:0/3753546936 learned_addr learned my addr 172.21.15.42:0/3753546936 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:23:51.557 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:51.550+0000 7f52ce8ba700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f52c809f830 0x7f52c814efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:23:51.557 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:51.550+0000 7f52cd8b8700 1 --2- 172.21.15.42:0/3753546936 >> v2:172.21.15.40:3300/0 conn(0x7f52c808df80 0x7f52c809f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:23:51.557 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:51.550+0000 7f52cd0b7700 1 -- 172.21.15.42:0/3753546936 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4028303130 0 0) 0x7f52c809fc60 con 0x7f52c80a7740 2019-12-05T05:23:51.557 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:51.550+0000 7f52cd0b7700 1 -- 172.21.15.42:0/3753546936 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52b40030b0 con 0x7f52c80a7740 2019-12-05T05:23:51.557 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:51.550+0000 7f52ce8ba700 1 -- 172.21.15.42:0/3753546936 >> v2:172.21.15.40:3300/0 conn(0x7f52c808df80 msgr2=0x7f52c809f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:23:51.557 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:51.550+0000 7f52ce8ba700 1 --2- 172.21.15.42:0/3753546936 >> v2:172.21.15.40:3300/0 conn(0x7f52c808df80 0x7f52c809f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:23:51.557 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:51.550+0000 7f52ce8ba700 1 -- 172.21.15.42:0/3753546936 >> v1:172.21.15.42:6789/0 conn(0x7f52c80a7740 legacy=0x7f52c807e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:51.558 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:51.550+0000 7f52ce8ba700 1 -- 172.21.15.42:0/3753546936 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52c80a0710 con 0x7f52c809f830 2019-12-05T05:23:51.558 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:51.550+0000 7f52ce8ba700 1 --2- 172.21.15.42:0/3753546936 >> v2:172.21.15.42:3300/0 conn(0x7f52c809f830 0x7f52c814efe0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f52c00032b0 tx=0x7f52c000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:51.558 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:51.550+0000 7f52ce8ba700 1 --2- 172.21.15.42:0/3753546936 >> v2:172.21.15.42:3300/0 conn(0x7f52c809f830 0x7f52c814efe0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f52c00032b0 tx=0x7f52c000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:51.558 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:51.550+0000 7f52ce8ba700 1 --2- 172.21.15.42:0/3753546936 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f52c809f830 0x7f52c814efe0 secure :-1 s=READY pgs=784 cs=0 l=1 rx=0x7f52c00032b0 tx=0x7f52c000fff0).ready entity=mon.1 client_cookie=259f40cf5e3352b9 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:23:51.558 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:51.550+0000 7f52ce8ba700 1 --2- 172.21.15.42:0/3753546936 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f52c809f830 0x7f52c814efe0 secure :-1 s=THROTTLE_DONE pgs=784 cs=0 l=1 rx=0x7f52c00032b0 tx=0x7f52c000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:51.559 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:51.550+0000 7f52cd0b7700 1 -- 172.21.15.42:0/3753546936 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f52c00108c0 con 0x7f52c809f830 2019-12-05T05:23:51.559 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:51.550+0000 7f52ce8ba700 1 --2- 172.21.15.42:0/3753546936 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f52c809f830 0x7f52c814efe0 secure :-1 s=THROTTLE_DONE pgs=784 cs=0 l=1 rx=0x7f52c00032b0 tx=0x7f52c000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:51.559 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:51.550+0000 7f52ce8ba700 1 --2- 172.21.15.42:0/3753546936 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f52c809f830 0x7f52c814efe0 secure :-1 s=THROTTLE_DONE pgs=784 cs=0 l=1 rx=0x7f52c00032b0 tx=0x7f52c000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:51.559 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:51.550+0000 7f52cd0b7700 1 -- 172.21.15.42:0/3753546936 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52c0008b50 con 0x7f52c809f830 2019-12-05T05:23:51.559 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:51.550+0000 7f52cd0b7700 1 -- 172.21.15.42:0/3753546936 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f52c00110c0 con 0x7f52c809f830 2019-12-05T05:23:51.560 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:51.550+0000 7f52cfb1c700 1 -- 172.21.15.42:0/3753546936 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f52c809f830 msgr2=0x7f52c814efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:51.560 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:51.550+0000 7f52cfb1c700 1 --2- 172.21.15.42:0/3753546936 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f52c809f830 0x7f52c814efe0 secure :-1 s=READY pgs=784 cs=0 l=1 rx=0x7f52c00032b0 tx=0x7f52c000fff0).stop 2019-12-05T05:23:51.560 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:51.550+0000 7f52cfb1c700 1 -- 172.21.15.42:0/3753546936 shutdown_connections 2019-12-05T05:23:51.560 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:51.550+0000 7f52cfb1c700 1 --2- 172.21.15.42:0/3753546936 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f52c809f830 0x7f52c814efe0 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:23:51.560 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:51.550+0000 7f52cfb1c700 1 --2- 172.21.15.42:0/3753546936 >> v2:172.21.15.40:3300/0 conn(0x7f52c808df80 0x7f52c809f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:23:51.560 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:51.550+0000 7f52cfb1c700 1 -- 172.21.15.42:0/3753546936 shutdown_connections 2019-12-05T05:23:51.561 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:51.550+0000 7f52cfb1c700 1 -- 172.21.15.42:0/3753546936 wait complete. 2019-12-05T05:23:51.561 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:51.550+0000 7f52cfb1c700 1 -- 172.21.15.42:0/3753546936 >> 172.21.15.42:0/3753546936 conn(0x7f52c80aabe0 msgr2=0x7f52c80a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:23:51.564 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:51.554+0000 7f52cfb1c700 1 Processor -- start 2019-12-05T05:23:51.564 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:51.554+0000 7f52cfb1c700 1 -- start start 2019-12-05T05:23:51.565 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:51.554+0000 7f52cfb1c700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f52c8116500 0x7f52c807d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:23:51.565 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:51.554+0000 7f52cfb1c700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f52c8118500 0x7f52c811dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:23:51.565 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:51.554+0000 7f52cfb1c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f52c808eb70 0x7f52c808efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:23:51.565 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:51.554+0000 7f52cfb1c700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52c807d360 con 0x7f52c8118500 2019-12-05T05:23:51.565 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:51.554+0000 7f52cfb1c700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52c8118c80 con 0x7f52c808eb70 2019-12-05T05:23:51.565 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:51.554+0000 7f52cfb1c700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52c8118930 con 0x7f52c8116500 2019-12-05T05:23:51.565 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:51.554+0000 7f52ce8ba700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f52c808eb70 0x7f52c808efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:23:51.566 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:51.554+0000 7f52ce8ba700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f52c808eb70 0x7f52c808efa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:32802/0 (socket says 172.21.15.42:32802) 2019-12-05T05:23:51.566 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:51.554+0000 7f52ce8ba700 1 -- 172.21.15.42:0/1468113878 learned_addr learned my addr 172.21.15.42:0/1468113878 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:23:51.566 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:51.554+0000 7f52ce0b9700 1 --2- 172.21.15.42:0/1468113878 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f52c8116500 0x7f52c807d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:23:51.566 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:51.554+0000 7f52cd8b8700 1 --2- 172.21.15.42:0/1468113878 >> v2:172.21.15.40:3300/0 conn(0x7f52c8118500 0x7f52c811dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:23:51.566 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:51.554+0000 7f52ce8ba700 1 -- 172.21.15.42:0/1468113878 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f52c8116500 msgr2=0x7f52c807d600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:51.566 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:51.554+0000 7f52ce8ba700 1 --2- 172.21.15.42:0/1468113878 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f52c8116500 0x7f52c807d600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:23:51.566 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:51.554+0000 7f52ce8ba700 1 -- 172.21.15.42:0/1468113878 >> v2:172.21.15.40:3300/0 conn(0x7f52c8118500 msgr2=0x7f52c811dcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:51.567 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:51.554+0000 7f52ce8ba700 1 --2- 172.21.15.42:0/1468113878 >> v2:172.21.15.40:3300/0 conn(0x7f52c8118500 0x7f52c811dcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:23:51.567 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:51.554+0000 7f52ce8ba700 1 -- 172.21.15.42:0/1468113878 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52c80a6c80 con 0x7f52c808eb70 2019-12-05T05:23:51.567 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:51.554+0000 7f52ce8ba700 1 --2- 172.21.15.42:0/1468113878 >> v2:172.21.15.42:3300/0 conn(0x7f52c808eb70 0x7f52c808efa0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f52c807c5d0 tx=0x7f52c00088f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:51.567 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:51.554+0000 7f52ce8ba700 1 --2- 172.21.15.42:0/1468113878 >> v2:172.21.15.42:3300/0 conn(0x7f52c808eb70 0x7f52c808efa0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f52c807c5d0 tx=0x7f52c00088f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:51.567 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:51.554+0000 7f52ce8ba700 1 --2- 172.21.15.42:0/1468113878 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f52c808eb70 0x7f52c808efa0 secure :-1 s=READY pgs=785 cs=0 l=1 rx=0x7f52c807c5d0 tx=0x7f52c00088f0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:23:51.567 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:51.554+0000 7f52ce8ba700 1 --2- 172.21.15.42:0/1468113878 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f52c808eb70 0x7f52c808efa0 secure :-1 s=THROTTLE_DONE pgs=785 cs=0 l=1 rx=0x7f52c807c5d0 tx=0x7f52c00088f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:51.567 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:51.554+0000 7f52beffd700 1 -- 172.21.15.42:0/1468113878 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f52c0005ab0 con 0x7f52c808eb70 2019-12-05T05:23:51.568 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:51.554+0000 7f52ce8ba700 1 --2- 172.21.15.42:0/1468113878 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f52c808eb70 0x7f52c808efa0 secure :-1 s=THROTTLE_DONE pgs=785 cs=0 l=1 rx=0x7f52c807c5d0 tx=0x7f52c00088f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:51.568 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:51.554+0000 7f52cfb1c700 1 -- 172.21.15.42:0/1468113878 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52c808f4f0 con 0x7f52c808eb70 2019-12-05T05:23:51.568 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:51.554+0000 7f52beffd700 1 -- 172.21.15.42:0/1468113878 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52c0010ac0 con 0x7f52c808eb70 2019-12-05T05:23:51.568 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:51.554+0000 7f52ce8ba700 1 --2- 172.21.15.42:0/1468113878 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f52c808eb70 0x7f52c808efa0 secure :-1 s=THROTTLE_DONE pgs=785 cs=0 l=1 rx=0x7f52c807c5d0 tx=0x7f52c00088f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:51.568 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:51.554+0000 7f52beffd700 1 -- 172.21.15.42:0/1468113878 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f52c000f470 con 0x7f52c808eb70 2019-12-05T05:23:51.568 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:51.554+0000 7f52cfb1c700 1 -- 172.21.15.42:0/1468113878 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52c8077560 con 0x7f52c808eb70 2019-12-05T05:23:51.568 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:51.558+0000 7f52ce8ba700 1 --2- 172.21.15.42:0/1468113878 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f52c808eb70 0x7f52c808efa0 secure :-1 s=THROTTLE_DONE pgs=785 cs=0 l=1 rx=0x7f52c807c5d0 tx=0x7f52c00088f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:51.569 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:51.558+0000 7f52beffd700 1 -- 172.21.15.42:0/1468113878 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f52c000f700 con 0x7f52c808eb70 2019-12-05T05:23:51.569 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:51.558+0000 7f52ce8ba700 1 --2- 172.21.15.42:0/1468113878 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f52c808eb70 0x7f52c808efa0 secure :-1 s=THROTTLE_DONE pgs=785 cs=0 l=1 rx=0x7f52c807c5d0 tx=0x7f52c00088f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:51.569 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:51.558+0000 7f52beffd700 1 --2- 172.21.15.42:0/1468113878 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f52b003ae10 0x7f52b003d320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:23:51.569 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:51.558+0000 7f52beffd700 1 -- 172.21.15.42:0/1468113878 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 7149+0+0 (secure 0 0 0) 0x7f52c004b0d0 con 0x7f52c808eb70 2019-12-05T05:23:51.569 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:51.558+0000 7f52ce0b9700 1 --2- 172.21.15.42:0/1468113878 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f52b003ae10 0x7f52b003d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:23:51.569 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:51.558+0000 7f52ce0b9700 1 --2- 172.21.15.42:0/1468113878 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f52b003ae10 0x7f52b003d320 crc :-1 s=READY pgs=346 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:23:51.570 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:51.562+0000 7f52cfb1c700 1 -- 172.21.15.42:0/1468113878 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52c805d110 con 0x7f52c808eb70 2019-12-05T05:23:51.573 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:51.570+0000 7f52ce8ba700 1 --2- 172.21.15.42:0/1468113878 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f52c808eb70 0x7f52c808efa0 secure :-1 s=THROTTLE_DONE pgs=785 cs=0 l=1 rx=0x7f52c807c5d0 tx=0x7f52c00088f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:51.573 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:51.570+0000 7f52beffd700 1 -- 172.21.15.42:0/1468113878 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f52c0032090 con 0x7f52c808eb70 2019-12-05T05:23:51.743 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:51.738+0000 7f52cfb1c700 1 -- 172.21.15.42:0/1468113878 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f52c805b8c0 con 0x7f52c808eb70 2019-12-05T05:23:51.744 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:51.742+0000 7f52ce8ba700 1 --2- 172.21.15.42:0/1468113878 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f52c808eb70 0x7f52c808efa0 secure :-1 s=THROTTLE_DONE pgs=785 cs=0 l=1 rx=0x7f52c807c5d0 tx=0x7f52c00088f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:51.745 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:51.742+0000 7f52beffd700 1 -- 172.21.15.42:0/1468113878 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 98 v98) v1 ==== 94+0+2108 (secure 0 0 0) 0x7f52c00322f0 con 0x7f52c808eb70 2019-12-05T05:23:51.745 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 98 2019-12-05T05:23:51.745 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:23:51.745 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":98,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5177,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/3208416000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3208416000},{"type":"v1","addr":"172.21.15.40:6833","nonce":3208416000}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":98}],"filesystems":[{"mdsmap":{"epoch":97,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:23:07.988779+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5123},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5123":{"gid":5123,"name":"a","rank":0,"incarnation":94,"state":"up:active","state_seq":37,"addr":"172.21.15.42:6835/1214659805","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1214659805},{"type":"v1","addr":"172.21.15.42:6835","nonce":1214659805}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:23:51.747 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:51.742+0000 7f52cfb1c700 1 -- 172.21.15.42:0/1468113878 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f52b003ae10 msgr2=0x7f52b003d320 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:51.747 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:51.742+0000 7f52cfb1c700 1 --2- 172.21.15.42:0/1468113878 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f52b003ae10 0x7f52b003d320 crc :-1 s=READY pgs=346 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:23:51.747 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:51.742+0000 7f52cfb1c700 1 -- 172.21.15.42:0/1468113878 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f52c808eb70 msgr2=0x7f52c808efa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:51.748 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:51.742+0000 7f52cfb1c700 1 --2- 172.21.15.42:0/1468113878 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f52c808eb70 0x7f52c808efa0 secure :-1 s=READY pgs=785 cs=0 l=1 rx=0x7f52c807c5d0 tx=0x7f52c00088f0).stop 2019-12-05T05:23:51.748 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:51.742+0000 7f52cfb1c700 1 -- 172.21.15.42:0/1468113878 shutdown_connections 2019-12-05T05:23:51.748 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:51.742+0000 7f52cfb1c700 1 --2- 172.21.15.42:0/1468113878 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f52b003ae10 0x7f52b003d320 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:23:51.748 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:51.742+0000 7f52cfb1c700 1 --2- 172.21.15.42:0/1468113878 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f52c808eb70 0x7f52c808efa0 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:23:51.748 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:51.742+0000 7f52cfb1c700 1 --2- 172.21.15.42:0/1468113878 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f52c8116500 0x7f52c807d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:23:51.748 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:51.742+0000 7f52cfb1c700 1 --2- 172.21.15.42:0/1468113878 >> v2:172.21.15.40:3300/0 conn(0x7f52c8118500 0x7f52c811dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:23:51.749 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:51.742+0000 7f52cfb1c700 1 -- 172.21.15.42:0/1468113878 shutdown_connections 2019-12-05T05:23:51.749 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:51.742+0000 7f52cfb1c700 1 -- 172.21.15.42:0/1468113878 wait complete. 2019-12-05T05:23:51.749 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:51.742+0000 7f52cfb1c700 1 -- 172.21.15.42:0/1468113878 >> 172.21.15.42:0/1468113878 conn(0x7f52c811d150 msgr2=0x7f52c807e5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:23:51.757 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T05:23:53.759 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:23:53.865 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:53.858+0000 7fdbea8ae700 1 Processor -- start 2019-12-05T05:23:53.865 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:53.858+0000 7fdbea8ae700 1 -- start start 2019-12-05T05:23:53.865 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:53.858+0000 7fdbea8ae700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdbe40a7740 0x7fdbe408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:23:53.865 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:53.858+0000 7fdbea8ae700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fdbe409fae0 0x7fdbe407eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:23:53.865 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:53.858+0000 7fdbea8ae700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbe40a0710 con 0x7fdbe407e720 2019-12-05T05:23:53.865 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:53.858+0000 7fdbea8ae700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdbe40a0b00 con 0x7fdbe409fae0 2019-12-05T05:23:53.866 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:53.858+0000 7fdbea8ae700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbe408e5b0 con 0x7fdbe40a7740 2019-12-05T05:23:53.866 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:53.858+0000 7fdbe964c700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fdbe409fae0 0x7fdbe407eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:23:53.866 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:53.858+0000 7fdbe8e4b700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdbe40a7740 0x7fdbe408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:23:53.866 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:53.858+0000 7fdbe3fff700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fdbe407e720 0x7fdbe409f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:42284/0 (socket says 172.21.15.42:42284) 2019-12-05T05:23:53.866 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:53.858+0000 7fdbe3fff700 1 -- 172.21.15.42:0/3945551762 learned_addr learned my addr 172.21.15.42:0/3945551762 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:23:53.866 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:53.858+0000 7fdbe964c700 1 -- 172.21.15.42:0/3945551762 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdbe40a7740 msgr2=0x7fdbe408df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:23:53.867 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:53.858+0000 7fdbe964c700 1 --2- 172.21.15.42:0/3945551762 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdbe40a7740 0x7fdbe408df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:23:53.867 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:53.858+0000 7fdbe964c700 1 -- 172.21.15.42:0/3945551762 >> v1:172.21.15.40:6789/0 conn(0x7fdbe407e720 legacy=0x7fdbe409f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:53.867 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:53.858+0000 7fdbe964c700 1 -- 172.21.15.42:0/3945551762 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbe407f0a0 con 0x7fdbe409fae0 2019-12-05T05:23:53.867 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:53.858+0000 7fdbe964c700 1 --2- 172.21.15.42:0/3945551762 >> v2:172.21.15.40:3300/0 conn(0x7fdbe409fae0 0x7fdbe407eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fdbdc008b10 tx=0x7fdbdc00fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:53.867 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:53.858+0000 7fdbe964c700 1 --2- 172.21.15.42:0/3945551762 >> v2:172.21.15.40:3300/0 conn(0x7fdbe409fae0 0x7fdbe407eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fdbdc008b10 tx=0x7fdbdc00fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:53.867 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:53.858+0000 7fdbe964c700 1 --2- 172.21.15.42:0/3945551762 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdbe409fae0 0x7fdbe407eb50 secure :-1 s=READY pgs=622 cs=0 l=1 rx=0x7fdbdc008b10 tx=0x7fdbdc00fff0).ready entity=mon.0 client_cookie=bcabf565084a6cea server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:23:53.867 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:53.858+0000 7fdbe964c700 1 --2- 172.21.15.42:0/3945551762 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdbe409fae0 0x7fdbe407eb50 secure :-1 s=THROTTLE_DONE pgs=622 cs=0 l=1 rx=0x7fdbdc008b10 tx=0x7fdbdc00fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:53.868 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:53.858+0000 7fdbe964c700 1 --2- 172.21.15.42:0/3945551762 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdbe409fae0 0x7fdbe407eb50 secure :-1 s=THROTTLE_DONE pgs=622 cs=0 l=1 rx=0x7fdbdc008b10 tx=0x7fdbdc00fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:53.868 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:53.858+0000 7fdbe964c700 1 --2- 172.21.15.42:0/3945551762 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdbe409fae0 0x7fdbe407eb50 secure :-1 s=THROTTLE_DONE pgs=622 cs=0 l=1 rx=0x7fdbdc008b10 tx=0x7fdbdc00fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:53.868 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:53.858+0000 7fdbe37fe700 1 -- 172.21.15.42:0/3945551762 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdbdc010900 con 0x7fdbe409fae0 2019-12-05T05:23:53.868 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:53.858+0000 7fdbe37fe700 1 -- 172.21.15.42:0/3945551762 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbdc008b50 con 0x7fdbe409fae0 2019-12-05T05:23:53.868 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:53.858+0000 7fdbe37fe700 1 -- 172.21.15.42:0/3945551762 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdbdc0110c0 con 0x7fdbe409fae0 2019-12-05T05:23:53.868 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:53.858+0000 7fdbea8ae700 1 -- 172.21.15.42:0/3945551762 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdbe409fae0 msgr2=0x7fdbe407eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:53.868 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:53.858+0000 7fdbea8ae700 1 --2- 172.21.15.42:0/3945551762 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdbe409fae0 0x7fdbe407eb50 secure :-1 s=READY pgs=622 cs=0 l=1 rx=0x7fdbdc008b10 tx=0x7fdbdc00fff0).stop 2019-12-05T05:23:53.869 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:53.862+0000 7fdbea8ae700 1 -- 172.21.15.42:0/3945551762 shutdown_connections 2019-12-05T05:23:53.869 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:53.862+0000 7fdbea8ae700 1 --2- 172.21.15.42:0/3945551762 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdbe409fae0 0x7fdbe407eb50 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:23:53.869 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:53.862+0000 7fdbea8ae700 1 --2- 172.21.15.42:0/3945551762 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdbe40a7740 0x7fdbe408df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:23:53.869 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:53.862+0000 7fdbea8ae700 1 -- 172.21.15.42:0/3945551762 shutdown_connections 2019-12-05T05:23:53.869 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:53.862+0000 7fdbea8ae700 1 -- 172.21.15.42:0/3945551762 wait complete. 2019-12-05T05:23:53.869 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:53.862+0000 7fdbea8ae700 1 -- 172.21.15.42:0/3945551762 >> 172.21.15.42:0/3945551762 conn(0x7fdbe40aabe0 msgr2=0x7fdbe40a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:23:53.872 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:53.862+0000 7fdbea8ae700 1 Processor -- start 2019-12-05T05:23:53.873 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:53.862+0000 7fdbea8ae700 1 -- start start 2019-12-05T05:23:53.873 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:53.862+0000 7fdbea8ae700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdbe411ac50 0x7fdbe4119300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:23:53.873 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:53.862+0000 7fdbea8ae700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fdbe41238d0 0x7fdbe4123d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:23:53.873 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:53.862+0000 7fdbea8ae700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbe411b080 con 0x7fdbe412e5c0 2019-12-05T05:23:53.873 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:53.862+0000 7fdbea8ae700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdbe412e0c0 con 0x7fdbe41238d0 2019-12-05T05:23:53.873 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:53.862+0000 7fdbea8ae700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbe412e320 con 0x7fdbe411ac50 2019-12-05T05:23:53.873 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:53.862+0000 7fdbe964c700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fdbe412e5c0 0x7fdbe4131500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:32854/0 (socket says 172.21.15.42:32854) 2019-12-05T05:23:53.874 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:53.862+0000 7fdbe964c700 1 -- 172.21.15.42:0/1685165957 learned_addr learned my addr 172.21.15.42:0/1685165957 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:23:53.874 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:53.862+0000 7fdbe8e4b700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdbe411ac50 0x7fdbe4119300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:23:53.874 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:53.862+0000 7fdbe3fff700 1 --2- 172.21.15.42:0/1685165957 >> v2:172.21.15.40:3300/0 conn(0x7fdbe41238d0 0x7fdbe4123d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:23:53.874 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:53.866+0000 7fdbe17fa700 1 -- 172.21.15.42:0/1685165957 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 774779701 0 0) 0x7fdbdc0087f0 con 0x7fdbe412e5c0 2019-12-05T05:23:53.874 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:53.866+0000 7fdbe8e4b700 1 -- 172.21.15.42:0/1685165957 >> v2:172.21.15.40:3300/0 conn(0x7fdbe41238d0 msgr2=0x7fdbe4123d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:53.874 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:53.866+0000 7fdbe8e4b700 1 --2- 172.21.15.42:0/1685165957 >> v2:172.21.15.40:3300/0 conn(0x7fdbe41238d0 0x7fdbe4123d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:23:53.875 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:53.866+0000 7fdbe8e4b700 1 -- 172.21.15.42:0/1685165957 >> v1:172.21.15.42:6789/0 conn(0x7fdbe412e5c0 legacy=0x7fdbe4131500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:53.875 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:53.866+0000 7fdbe8e4b700 1 -- 172.21.15.42:0/1685165957 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbe407c8f0 con 0x7fdbe411ac50 2019-12-05T05:23:53.875 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:53.866+0000 7fdbe8e4b700 1 --2- 172.21.15.42:0/1685165957 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdbe411ac50 0x7fdbe4119300 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fdbcc002f30 tx=0x7fdbcc0109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:53.875 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:53.866+0000 7fdbe8e4b700 1 --2- 172.21.15.42:0/1685165957 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdbe411ac50 0x7fdbe4119300 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fdbcc002f30 tx=0x7fdbcc0109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:53.875 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:53.866+0000 7fdbe8e4b700 1 --2- 172.21.15.42:0/1685165957 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdbe411ac50 0x7fdbe4119300 secure :-1 s=READY pgs=161 cs=0 l=1 rx=0x7fdbcc002f30 tx=0x7fdbcc0109e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:23:53.875 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:53.866+0000 7fdbe8e4b700 1 --2- 172.21.15.42:0/1685165957 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdbe411ac50 0x7fdbe4119300 secure :-1 s=THROTTLE_DONE pgs=161 cs=0 l=1 rx=0x7fdbcc002f30 tx=0x7fdbcc0109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:53.876 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:53.866+0000 7fdbe17fa700 1 -- 172.21.15.42:0/1685165957 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdbcc005920 con 0x7fdbe411ac50 2019-12-05T05:23:53.876 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:53.866+0000 7fdbe8e4b700 1 --2- 172.21.15.42:0/1685165957 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdbe411ac50 0x7fdbe4119300 secure :-1 s=THROTTLE_DONE pgs=161 cs=0 l=1 rx=0x7fdbcc002f30 tx=0x7fdbcc0109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:53.876 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:53.866+0000 7fdbe17fa700 1 -- 172.21.15.42:0/1685165957 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbcc005540 con 0x7fdbe411ac50 2019-12-05T05:23:53.876 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:53.866+0000 7fdbe8e4b700 1 --2- 172.21.15.42:0/1685165957 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdbe411ac50 0x7fdbe4119300 secure :-1 s=THROTTLE_DONE pgs=161 cs=0 l=1 rx=0x7fdbcc002f30 tx=0x7fdbcc0109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:53.876 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:53.866+0000 7fdbe17fa700 1 -- 172.21.15.42:0/1685165957 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdbcc012510 con 0x7fdbe411ac50 2019-12-05T05:23:53.876 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:53.866+0000 7fdbea8ae700 1 -- 172.21.15.42:0/1685165957 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbe4119850 con 0x7fdbe411ac50 2019-12-05T05:23:53.876 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:53.866+0000 7fdbea8ae700 1 -- 172.21.15.42:0/1685165957 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbe412ec10 con 0x7fdbe411ac50 2019-12-05T05:23:53.877 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:53.866+0000 7fdbe8e4b700 1 --2- 172.21.15.42:0/1685165957 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdbe411ac50 0x7fdbe4119300 secure :-1 s=THROTTLE_DONE pgs=161 cs=0 l=1 rx=0x7fdbcc002f30 tx=0x7fdbcc0109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:53.877 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:53.866+0000 7fdbe17fa700 1 -- 172.21.15.42:0/1685165957 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fdbcc0127f0 con 0x7fdbe411ac50 2019-12-05T05:23:53.877 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:53.866+0000 7fdbe8e4b700 1 --2- 172.21.15.42:0/1685165957 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdbe411ac50 0x7fdbe4119300 secure :-1 s=THROTTLE_DONE pgs=161 cs=0 l=1 rx=0x7fdbcc002f30 tx=0x7fdbcc0109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:53.877 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:53.866+0000 7fdbe17fa700 1 --2- 172.21.15.42:0/1685165957 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fdbc803ac90 0x7fdbc803d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:23:53.877 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:53.866+0000 7fdbe17fa700 1 -- 172.21.15.42:0/1685165957 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 7235+0+0 (secure 0 0 0) 0x7fdbcc04ab50 con 0x7fdbe411ac50 2019-12-05T05:23:53.878 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:53.866+0000 7fdbe3fff700 1 --2- 172.21.15.42:0/1685165957 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fdbc803ac90 0x7fdbc803d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:23:53.878 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:53.870+0000 7fdbe3fff700 1 --2- 172.21.15.42:0/1685165957 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fdbc803ac90 0x7fdbc803d1a0 crc :-1 s=READY pgs=347 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:23:53.878 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:53.874+0000 7fdbea8ae700 1 -- 172.21.15.42:0/1685165957 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbe405d110 con 0x7fdbe411ac50 2019-12-05T05:23:53.883 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:53.878+0000 7fdbe8e4b700 1 --2- 172.21.15.42:0/1685165957 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdbe411ac50 0x7fdbe4119300 secure :-1 s=THROTTLE_DONE pgs=161 cs=0 l=1 rx=0x7fdbcc002f30 tx=0x7fdbcc0109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:53.883 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:53.878+0000 7fdbe17fa700 1 -- 172.21.15.42:0/1685165957 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fdbcc031bb0 con 0x7fdbe411ac50 2019-12-05T05:23:54.057 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:54.054+0000 7fdbea8ae700 1 -- 172.21.15.42:0/1685165957 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdbe4123600 con 0x7fdbe411ac50 2019-12-05T05:23:54.058 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:54.054+0000 7fdbe8e4b700 1 --2- 172.21.15.42:0/1685165957 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdbe411ac50 0x7fdbe4119300 secure :-1 s=THROTTLE_DONE pgs=161 cs=0 l=1 rx=0x7fdbcc002f30 tx=0x7fdbcc0109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:54.058 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:54.054+0000 7fdbe17fa700 1 -- 172.21.15.42:0/1685165957 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+1771 (secure 0 0 0) 0x7fdbcc031e10 con 0x7fdbe411ac50 2019-12-05T05:23:54.059 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 99 2019-12-05T05:23:54.059 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:23:54.059 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":99,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:23:53.553120+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5177":{"gid":5177,"name":"b","rank":0,"incarnation":99,"state":"up:replay","state_seq":2,"addr":"172.21.15.40:6833/3208416000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3208416000},{"type":"v1","addr":"172.21.15.40:6833","nonce":3208416000}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:23:54.061 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:54.058+0000 7fdbea8ae700 1 -- 172.21.15.42:0/1685165957 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fdbc803ac90 msgr2=0x7fdbc803d1a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:54.061 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:54.058+0000 7fdbea8ae700 1 --2- 172.21.15.42:0/1685165957 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fdbc803ac90 0x7fdbc803d1a0 crc :-1 s=READY pgs=347 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:23:54.061 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:54.058+0000 7fdbea8ae700 1 -- 172.21.15.42:0/1685165957 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdbe411ac50 msgr2=0x7fdbe4119300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:54.063 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:54.058+0000 7fdbea8ae700 1 --2- 172.21.15.42:0/1685165957 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdbe411ac50 0x7fdbe4119300 secure :-1 s=READY pgs=161 cs=0 l=1 rx=0x7fdbcc002f30 tx=0x7fdbcc0109e0).stop 2019-12-05T05:23:54.063 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:54.058+0000 7fdbea8ae700 1 -- 172.21.15.42:0/1685165957 shutdown_connections 2019-12-05T05:23:54.063 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:54.058+0000 7fdbea8ae700 1 --2- 172.21.15.42:0/1685165957 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fdbc803ac90 0x7fdbc803d1a0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:23:54.064 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:54.058+0000 7fdbea8ae700 1 --2- 172.21.15.42:0/1685165957 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdbe411ac50 0x7fdbe4119300 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:23:54.064 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:54.058+0000 7fdbea8ae700 1 --2- 172.21.15.42:0/1685165957 >> v2:172.21.15.40:3300/0 conn(0x7fdbe41238d0 0x7fdbe4123d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:23:54.065 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:54.058+0000 7fdbea8ae700 1 -- 172.21.15.42:0/1685165957 shutdown_connections 2019-12-05T05:23:54.065 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:54.058+0000 7fdbea8ae700 1 -- 172.21.15.42:0/1685165957 wait complete. 2019-12-05T05:23:54.065 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:54.058+0000 7fdbea8ae700 1 -- 172.21.15.42:0/1685165957 >> 172.21.15.42:0/1685165957 conn(0x7fdbe40a6d30 msgr2=0x7fdbe40a9800 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:23:54.071 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a down, removed from mdsmap 2019-12-05T05:23:54.071 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2019-12-05T05:23:54.071 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:23:54.176 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:54.162+0000 7fe60b93b700 1 Processor -- start 2019-12-05T05:23:54.176 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:54.162+0000 7fe60b93b700 1 -- start start 2019-12-05T05:23:54.177 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:54.162+0000 7fe60b93b700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe60408df80 0x7fe60409f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:23:54.177 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:54.162+0000 7fe60b93b700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe60409fc60 con 0x7fe6040a7740 2019-12-05T05:23:54.177 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:54.162+0000 7fe60b93b700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6040a0710 con 0x7fe60409f830 2019-12-05T05:23:54.177 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:54.162+0000 7fe60b93b700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6040a0b00 con 0x7fe60408df80 2019-12-05T05:23:54.177 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:54.162+0000 7fe60a6d9700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fe60409f830 0x7fe60414efe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:32860/0 (socket says 172.21.15.42:32860) 2019-12-05T05:23:54.178 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:54.162+0000 7fe60a6d9700 1 -- 172.21.15.42:0/2982970017 learned_addr learned my addr 172.21.15.42:0/2982970017 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:23:54.178 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:54.162+0000 7fe6096d7700 1 --2- 172.21.15.42:0/2982970017 >> v2:172.21.15.40:3300/0 conn(0x7fe60408df80 0x7fe60409f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:23:54.178 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:54.162+0000 7fe608ed6700 1 -- 172.21.15.42:0/2982970017 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2847779266 0 0) 0x7fe6040a0710 con 0x7fe60409f830 2019-12-05T05:23:54.178 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:54.162+0000 7fe608ed6700 1 -- 172.21.15.42:0/2982970017 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5f00030b0 con 0x7fe60409f830 2019-12-05T05:23:54.178 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:54.162+0000 7fe6096d7700 1 -- 172.21.15.42:0/2982970017 >> v1:172.21.15.42:6789/0 conn(0x7fe60409f830 legacy=0x7fe60414efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:54.179 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:54.162+0000 7fe608ed6700 1 -- 172.21.15.42:0/2982970017 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2113478225 0 0) 0x7fe60409fc60 con 0x7fe6040a7740 2019-12-05T05:23:54.179 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:54.162+0000 7fe6096d7700 1 -- 172.21.15.42:0/2982970017 >> v1:172.21.15.40:6789/0 conn(0x7fe6040a7740 legacy=0x7fe60407e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:54.179 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:54.162+0000 7fe6096d7700 1 -- 172.21.15.42:0/2982970017 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe60407ef20 con 0x7fe60408df80 2019-12-05T05:23:54.179 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:54.162+0000 7fe6096d7700 1 --2- 172.21.15.42:0/2982970017 >> v2:172.21.15.40:3300/0 conn(0x7fe60408df80 0x7fe60409f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fe5f4008790 tx=0x7fe5f400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:54.179 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:54.162+0000 7fe6096d7700 1 --2- 172.21.15.42:0/2982970017 >> v2:172.21.15.40:3300/0 conn(0x7fe60408df80 0x7fe60409f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fe5f4008790 tx=0x7fe5f400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:54.180 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:54.162+0000 7fe6096d7700 1 --2- 172.21.15.42:0/2982970017 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe60408df80 0x7fe60409f2e0 secure :-1 s=READY pgs=624 cs=0 l=1 rx=0x7fe5f4008790 tx=0x7fe5f400fbf0).ready entity=mon.0 client_cookie=a7ebf413c81d4192 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:23:54.180 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:54.162+0000 7fe6096d7700 1 --2- 172.21.15.42:0/2982970017 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe60408df80 0x7fe60409f2e0 secure :-1 s=THROTTLE_DONE pgs=624 cs=0 l=1 rx=0x7fe5f4008790 tx=0x7fe5f400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:54.180 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:54.162+0000 7fe608ed6700 1 -- 172.21.15.42:0/2982970017 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe5f4011270 con 0x7fe60408df80 2019-12-05T05:23:54.180 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:54.162+0000 7fe6096d7700 1 --2- 172.21.15.42:0/2982970017 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe60408df80 0x7fe60409f2e0 secure :-1 s=THROTTLE_DONE pgs=624 cs=0 l=1 rx=0x7fe5f4008790 tx=0x7fe5f400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:54.180 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:54.162+0000 7fe608ed6700 1 -- 172.21.15.42:0/2982970017 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5f4008bc0 con 0x7fe60408df80 2019-12-05T05:23:54.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:54.162+0000 7fe6096d7700 1 --2- 172.21.15.42:0/2982970017 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe60408df80 0x7fe60409f2e0 secure :-1 s=THROTTLE_DONE pgs=624 cs=0 l=1 rx=0x7fe5f4008790 tx=0x7fe5f400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:54.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:54.162+0000 7fe60b93b700 1 -- 172.21.15.42:0/2982970017 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe60408df80 msgr2=0x7fe60409f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:54.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:54.162+0000 7fe608ed6700 1 -- 172.21.15.42:0/2982970017 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe5f4011850 con 0x7fe60408df80 2019-12-05T05:23:54.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:54.162+0000 7fe60b93b700 1 --2- 172.21.15.42:0/2982970017 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe60408df80 0x7fe60409f2e0 secure :-1 s=READY pgs=624 cs=0 l=1 rx=0x7fe5f4008790 tx=0x7fe5f400fbf0).stop 2019-12-05T05:23:54.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:54.166+0000 7fe60b93b700 1 -- 172.21.15.42:0/2982970017 shutdown_connections 2019-12-05T05:23:54.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:54.166+0000 7fe60b93b700 1 --2- 172.21.15.42:0/2982970017 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe60408df80 0x7fe60409f2e0 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:23:54.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:54.166+0000 7fe60b93b700 1 -- 172.21.15.42:0/2982970017 shutdown_connections 2019-12-05T05:23:54.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:54.166+0000 7fe60b93b700 1 -- 172.21.15.42:0/2982970017 wait complete. 2019-12-05T05:23:54.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:54.166+0000 7fe60b93b700 1 -- 172.21.15.42:0/2982970017 >> 172.21.15.42:0/2982970017 conn(0x7fe6040aabe0 msgr2=0x7fe6040a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:23:54.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:54.166+0000 7fe60b93b700 1 Processor -- start 2019-12-05T05:23:54.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:54.166+0000 7fe60b93b700 1 -- start start 2019-12-05T05:23:54.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:54.166+0000 7fe60b93b700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe6041446e0 0x7fe604123f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:23:54.184 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:54.166+0000 7fe60b93b700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe60411fe90 0x7fe604116610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:23:54.184 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:54.166+0000 7fe60b93b700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe604123450 con 0x7fe6041153c0 2019-12-05T05:23:54.184 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:54.166+0000 7fe60b93b700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe604123760 con 0x7fe6041446e0 2019-12-05T05:23:54.185 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:54.166+0000 7fe60b93b700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe604123a00 con 0x7fe60411fe90 2019-12-05T05:23:54.185 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:54.166+0000 7fe6096d7700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe60411fe90 0x7fe604116610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:23:54.185 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:54.166+0000 7fe6096d7700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe60411fe90 0x7fe604116610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:32826/0 (socket says 172.21.15.42:32826) 2019-12-05T05:23:54.185 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:54.166+0000 7fe60a6d9700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fe6041153c0 0x7fe604142d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:42308/0 (socket says 172.21.15.42:42308) 2019-12-05T05:23:54.186 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:54.166+0000 7fe6096d7700 1 -- 172.21.15.42:0/3278333025 learned_addr learned my addr 172.21.15.42:0/3278333025 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:23:54.186 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:54.166+0000 7fe609ed8700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe6041446e0 0x7fe604123f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:23:54.186 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:54.170+0000 7fe5fa7fc700 1 -- 172.21.15.42:0/3278333025 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 364413703 0 0) 0x7fe604123450 con 0x7fe6041153c0 2019-12-05T05:23:54.187 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:54.170+0000 7fe5fa7fc700 1 -- 172.21.15.42:0/3278333025 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5ec0030b0 con 0x7fe6041153c0 2019-12-05T05:23:54.187 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:54.170+0000 7fe609ed8700 1 -- 172.21.15.42:0/3278333025 >> v2:172.21.15.42:3300/0 conn(0x7fe60411fe90 msgr2=0x7fe604116610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:54.188 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:54.170+0000 7fe609ed8700 1 --2- 172.21.15.42:0/3278333025 >> v2:172.21.15.42:3300/0 conn(0x7fe60411fe90 0x7fe604116610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:23:54.188 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:54.170+0000 7fe609ed8700 1 -- 172.21.15.42:0/3278333025 >> v1:172.21.15.40:6789/0 conn(0x7fe6041153c0 legacy=0x7fe604142d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:54.188 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:54.170+0000 7fe609ed8700 1 -- 172.21.15.42:0/3278333025 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe604116b60 con 0x7fe6041446e0 2019-12-05T05:23:54.188 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:54.170+0000 7fe6096d7700 1 --2- 172.21.15.42:0/3278333025 >> v2:172.21.15.42:3300/0 conn(0x7fe60411fe90 0x7fe604116610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:23:54.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:54.170+0000 7fe609ed8700 1 --2- 172.21.15.42:0/3278333025 >> v2:172.21.15.40:3300/0 conn(0x7fe6041446e0 0x7fe604123f40 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fe604123680 tx=0x7fe600010840).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:54.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:54.170+0000 7fe609ed8700 1 --2- 172.21.15.42:0/3278333025 >> v2:172.21.15.40:3300/0 conn(0x7fe6041446e0 0x7fe604123f40 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fe604123680 tx=0x7fe600010840).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:54.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:54.170+0000 7fe609ed8700 1 --2- 172.21.15.42:0/3278333025 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe6041446e0 0x7fe604123f40 secure :-1 s=READY pgs=626 cs=0 l=1 rx=0x7fe604123680 tx=0x7fe600010840).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:23:54.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:54.170+0000 7fe609ed8700 1 --2- 172.21.15.42:0/3278333025 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe6041446e0 0x7fe604123f40 secure :-1 s=THROTTLE_DONE pgs=626 cs=0 l=1 rx=0x7fe604123680 tx=0x7fe600010840).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:54.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:54.170+0000 7fe5fa7fc700 1 -- 172.21.15.42:0/3278333025 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe600006a40 con 0x7fe6041446e0 2019-12-05T05:23:54.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:54.170+0000 7fe609ed8700 1 --2- 172.21.15.42:0/3278333025 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe6041446e0 0x7fe604123f40 secure :-1 s=THROTTLE_DONE pgs=626 cs=0 l=1 rx=0x7fe604123680 tx=0x7fe600010840).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:54.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:54.170+0000 7fe60b93b700 1 -- 172.21.15.42:0/3278333025 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6041202c0 con 0x7fe6041446e0 2019-12-05T05:23:54.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:54.170+0000 7fe5fa7fc700 1 -- 172.21.15.42:0/3278333025 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe600006660 con 0x7fe6041446e0 2019-12-05T05:23:54.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:54.170+0000 7fe609ed8700 1 --2- 172.21.15.42:0/3278333025 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe6041446e0 0x7fe604123f40 secure :-1 s=THROTTLE_DONE pgs=626 cs=0 l=1 rx=0x7fe604123680 tx=0x7fe600010840).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:54.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:54.170+0000 7fe5fa7fc700 1 -- 172.21.15.42:0/3278333025 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe600011040 con 0x7fe6041446e0 2019-12-05T05:23:54.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:54.170+0000 7fe60b93b700 1 -- 172.21.15.42:0/3278333025 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6041157f0 con 0x7fe6041446e0 2019-12-05T05:23:54.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:54.170+0000 7fe609ed8700 1 --2- 172.21.15.42:0/3278333025 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe6041446e0 0x7fe604123f40 secure :-1 s=THROTTLE_DONE pgs=626 cs=0 l=1 rx=0x7fe604123680 tx=0x7fe600010840).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:54.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:54.170+0000 7fe5fa7fc700 1 -- 172.21.15.42:0/3278333025 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fe600011360 con 0x7fe6041446e0 2019-12-05T05:23:54.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:54.170+0000 7fe609ed8700 1 --2- 172.21.15.42:0/3278333025 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe6041446e0 0x7fe604123f40 secure :-1 s=THROTTLE_DONE pgs=626 cs=0 l=1 rx=0x7fe604123680 tx=0x7fe600010840).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:54.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:54.170+0000 7fe5fa7fc700 1 --2- 172.21.15.42:0/3278333025 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe5ec03b260 0x7fe5ec03d770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:23:54.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:54.170+0000 7fe5fa7fc700 1 -- 172.21.15.42:0/3278333025 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 7235+0+0 (secure 0 0 0) 0x7fe60004b0b0 con 0x7fe6041446e0 2019-12-05T05:23:54.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:54.170+0000 7fe6096d7700 1 --2- 172.21.15.42:0/3278333025 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe5ec03b260 0x7fe5ec03d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:23:54.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:54.174+0000 7fe6096d7700 1 --2- 172.21.15.42:0/3278333025 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe5ec03b260 0x7fe5ec03d770 crc :-1 s=READY pgs=348 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:23:54.193 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:54.178+0000 7fe60b93b700 1 -- 172.21.15.42:0/3278333025 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe60405d110 con 0x7fe6041446e0 2019-12-05T05:23:54.194 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:54.190+0000 7fe609ed8700 1 --2- 172.21.15.42:0/3278333025 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe6041446e0 0x7fe604123f40 secure :-1 s=THROTTLE_DONE pgs=626 cs=0 l=1 rx=0x7fe604123680 tx=0x7fe600010840).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:54.194 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:54.190+0000 7fe5fa7fc700 1 -- 172.21.15.42:0/3278333025 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fe600032110 con 0x7fe6041446e0 2019-12-05T05:23:54.359 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:54.354+0000 7fe60b93b700 1 -- 172.21.15.42:0/3278333025 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe6041505d0 con 0x7fe6041446e0 2019-12-05T05:23:54.361 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:54.358+0000 7fe609ed8700 1 --2- 172.21.15.42:0/3278333025 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe6041446e0 0x7fe604123f40 secure :-1 s=THROTTLE_DONE pgs=626 cs=0 l=1 rx=0x7fe604123680 tx=0x7fe600010840).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:54.361 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:54.358+0000 7fe5fa7fc700 1 -- 172.21.15.42:0/3278333025 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+1771 (secure 0 0 0) 0x7fe600032370 con 0x7fe6041446e0 2019-12-05T05:23:54.362 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 99 2019-12-05T05:23:54.362 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:23:54.363 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":99,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:23:53.553120+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5177":{"gid":5177,"name":"b","rank":0,"incarnation":99,"state":"up:replay","state_seq":2,"addr":"172.21.15.40:6833/3208416000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3208416000},{"type":"v1","addr":"172.21.15.40:6833","nonce":3208416000}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:23:54.365 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:54.362+0000 7fe60b93b700 1 -- 172.21.15.42:0/3278333025 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe5ec03b260 msgr2=0x7fe5ec03d770 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:54.365 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:54.362+0000 7fe60b93b700 1 --2- 172.21.15.42:0/3278333025 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe5ec03b260 0x7fe5ec03d770 crc :-1 s=READY pgs=348 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:23:54.365 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:54.362+0000 7fe60b93b700 1 -- 172.21.15.42:0/3278333025 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe6041446e0 msgr2=0x7fe604123f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:54.366 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:54.362+0000 7fe60b93b700 1 --2- 172.21.15.42:0/3278333025 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe6041446e0 0x7fe604123f40 secure :-1 s=READY pgs=626 cs=0 l=1 rx=0x7fe604123680 tx=0x7fe600010840).stop 2019-12-05T05:23:54.366 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:54.362+0000 7fe60b93b700 1 -- 172.21.15.42:0/3278333025 shutdown_connections 2019-12-05T05:23:54.366 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:54.362+0000 7fe60b93b700 1 --2- 172.21.15.42:0/3278333025 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe5ec03b260 0x7fe5ec03d770 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:23:54.367 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:54.362+0000 7fe60b93b700 1 --2- 172.21.15.42:0/3278333025 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe6041446e0 0x7fe604123f40 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:23:54.367 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:54.362+0000 7fe60b93b700 1 --2- 172.21.15.42:0/3278333025 >> v2:172.21.15.42:3300/0 conn(0x7fe60411fe90 0x7fe604116610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:23:54.367 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:54.362+0000 7fe60b93b700 1 -- 172.21.15.42:0/3278333025 shutdown_connections 2019-12-05T05:23:54.367 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:54.362+0000 7fe60b93b700 1 -- 172.21.15.42:0/3278333025 wait complete. 2019-12-05T05:23:54.368 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:54.362+0000 7fe60b93b700 1 -- 172.21.15.42:0/3278333025 >> 172.21.15.42:0/3278333025 conn(0x7fe6040a8230 msgr2=0x7fe6040a6a60 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:23:54.375 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 99, "feature_flags": { "enable_multiple": false, "ever_enabled_multiple": false }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2019-12-05T05:03:35.604738+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 99, "ever_allowed_features": 2, "explicitly_allowed_features": 2, "failed": [], "flags": 18, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5177": { "addr": "172.21.15.40:6833/3208416000", "addrs": { "addrvec": [ { "addr": "172.21.15.40:6832", "nonce": 3208416000, "type": "v2" }, { "addr": "172.21.15.40:6833", "nonce": 3208416000, "type": "v1" } ] }, "export_targets": [], "features": 4540138292836696063, "flags": 0, "gid": 5177, "incarnation": 99, "name": "b", "rank": 0, "state": "up:replay", "state_seq": 2 } }, "last_failure": 0, "last_failure_osd_epoch": 63, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 2, "min_compat_client": "0 (unknown)", "modified": "2019-12-05T05:23:53.553120+0000", "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5177 } } } ], "standbys": [] } 2019-12-05T05:23:56.377 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:23:56.483 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:56.474+0000 7fb3f8c02700 1 Processor -- start 2019-12-05T05:23:56.483 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:56.474+0000 7fb3f8c02700 1 -- start start 2019-12-05T05:23:56.483 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:56.474+0000 7fb3f8c02700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb3f408df80 0x7fb3f409f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:23:56.484 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:56.474+0000 7fb3f8c02700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3f409fc60 con 0x7fb3f409f830 2019-12-05T05:23:56.484 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:56.474+0000 7fb3f8c02700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3f40a0710 con 0x7fb3f40a7740 2019-12-05T05:23:56.484 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:56.474+0000 7fb3f8c02700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3f40a0b00 con 0x7fb3f408df80 2019-12-05T05:23:56.484 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:56.474+0000 7fb3f2d9d700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fb3f40a7740 0x7fb3f407e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:32874/0 (socket says 172.21.15.42:32874) 2019-12-05T05:23:56.484 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:56.474+0000 7fb3f2d9d700 1 -- 172.21.15.42:0/4180128405 learned_addr learned my addr 172.21.15.42:0/4180128405 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:23:56.485 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:56.474+0000 7fb3ea59c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb3f408df80 0x7fb3f409f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:23:56.485 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:56.478+0000 7fb3f259c700 1 -- 172.21.15.42:0/4180128405 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1817972003 0 0) 0x7fb3f40a0710 con 0x7fb3f40a7740 2019-12-05T05:23:56.485 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:56.478+0000 7fb3f259c700 1 -- 172.21.15.42:0/4180128405 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3e00030b0 con 0x7fb3f40a7740 2019-12-05T05:23:56.485 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:56.478+0000 7fb3f259c700 1 -- 172.21.15.42:0/4180128405 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2265953003 0 0) 0x7fb3f409fc60 con 0x7fb3f409f830 2019-12-05T05:23:56.485 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:56.478+0000 7fb3f259c700 1 -- 172.21.15.42:0/4180128405 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3f40a0710 con 0x7fb3f409f830 2019-12-05T05:23:56.486 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:56.478+0000 7fb3ea59c700 1 -- 172.21.15.42:0/4180128405 >> v1:172.21.15.42:6789/0 conn(0x7fb3f40a7740 legacy=0x7fb3f407e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:56.486 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:56.478+0000 7fb3ea59c700 1 -- 172.21.15.42:0/4180128405 >> v1:172.21.15.40:6789/0 conn(0x7fb3f409f830 legacy=0x7fb3f414efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:56.486 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:56.478+0000 7fb3ea59c700 1 -- 172.21.15.42:0/4180128405 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3f407ef20 con 0x7fb3f408df80 2019-12-05T05:23:56.486 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:56.478+0000 7fb3ea59c700 1 --2- 172.21.15.42:0/4180128405 >> v2:172.21.15.42:3300/0 conn(0x7fb3f408df80 0x7fb3f409f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fb3dc008790 tx=0x7fb3dc00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:56.486 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:56.478+0000 7fb3ea59c700 1 --2- 172.21.15.42:0/4180128405 >> v2:172.21.15.42:3300/0 conn(0x7fb3f408df80 0x7fb3f409f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fb3dc008790 tx=0x7fb3dc00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:56.486 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:56.478+0000 7fb3ea59c700 1 --2- 172.21.15.42:0/4180128405 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb3f408df80 0x7fb3f409f2e0 secure :-1 s=READY pgs=789 cs=0 l=1 rx=0x7fb3dc008790 tx=0x7fb3dc00fbf0).ready entity=mon.1 client_cookie=5b1a8b7aeb0da69a server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:23:56.486 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:56.478+0000 7fb3ea59c700 1 --2- 172.21.15.42:0/4180128405 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb3f408df80 0x7fb3f409f2e0 secure :-1 s=THROTTLE_DONE pgs=789 cs=0 l=1 rx=0x7fb3dc008790 tx=0x7fb3dc00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:56.487 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:56.478+0000 7fb3f259c700 1 -- 172.21.15.42:0/4180128405 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb3dc011270 con 0x7fb3f408df80 2019-12-05T05:23:56.487 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:56.478+0000 7fb3ea59c700 1 --2- 172.21.15.42:0/4180128405 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb3f408df80 0x7fb3f409f2e0 secure :-1 s=THROTTLE_DONE pgs=789 cs=0 l=1 rx=0x7fb3dc008790 tx=0x7fb3dc00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:56.487 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:56.478+0000 7fb3f259c700 1 -- 172.21.15.42:0/4180128405 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3dc008bc0 con 0x7fb3f408df80 2019-12-05T05:23:56.487 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:56.478+0000 7fb3ea59c700 1 --2- 172.21.15.42:0/4180128405 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb3f408df80 0x7fb3f409f2e0 secure :-1 s=THROTTLE_DONE pgs=789 cs=0 l=1 rx=0x7fb3dc008790 tx=0x7fb3dc00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:56.487 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:56.478+0000 7fb3f8c02700 1 -- 172.21.15.42:0/4180128405 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb3f408df80 msgr2=0x7fb3f409f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:56.487 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:56.478+0000 7fb3f259c700 1 -- 172.21.15.42:0/4180128405 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb3dc011850 con 0x7fb3f408df80 2019-12-05T05:23:56.489 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:56.478+0000 7fb3f8c02700 1 --2- 172.21.15.42:0/4180128405 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb3f408df80 0x7fb3f409f2e0 secure :-1 s=READY pgs=789 cs=0 l=1 rx=0x7fb3dc008790 tx=0x7fb3dc00fbf0).stop 2019-12-05T05:23:56.489 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:56.478+0000 7fb3f8c02700 1 -- 172.21.15.42:0/4180128405 shutdown_connections 2019-12-05T05:23:56.490 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:56.478+0000 7fb3f8c02700 1 --2- 172.21.15.42:0/4180128405 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb3f408df80 0x7fb3f409f2e0 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:23:56.490 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:56.478+0000 7fb3f8c02700 1 -- 172.21.15.42:0/4180128405 shutdown_connections 2019-12-05T05:23:56.490 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:56.478+0000 7fb3f8c02700 1 -- 172.21.15.42:0/4180128405 wait complete. 2019-12-05T05:23:56.490 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:56.478+0000 7fb3f8c02700 1 -- 172.21.15.42:0/4180128405 >> 172.21.15.42:0/4180128405 conn(0x7fb3f40aabe0 msgr2=0x7fb3f40a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:23:56.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:56.482+0000 7fb3f8c02700 1 Processor -- start 2019-12-05T05:23:56.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:56.482+0000 7fb3f8c02700 1 -- start start 2019-12-05T05:23:56.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:56.482+0000 7fb3f8c02700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb3f414a1c0 0x7fb3f40a3c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:23:56.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:56.482+0000 7fb3f8c02700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb3f40a2640 0x7fb3f407aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:23:56.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:56.482+0000 7fb3f8c02700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3f40a66e0 con 0x7fb3f40a62b0 2019-12-05T05:23:56.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:56.482+0000 7fb3f8c02700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3f40a69c0 con 0x7fb3f414a1c0 2019-12-05T05:23:56.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:56.482+0000 7fb3f8c02700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3f40a6c60 con 0x7fb3f40a2640 2019-12-05T05:23:56.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:56.482+0000 7fb3f359e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb3f40a2640 0x7fb3f407aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:23:56.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:56.482+0000 7fb3f359e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb3f40a2640 0x7fb3f407aa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:32844/0 (socket says 172.21.15.42:32844) 2019-12-05T05:23:56.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:56.482+0000 7fb3f2d9d700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb3f414a1c0 0x7fb3f40a3c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:23:56.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:56.482+0000 7fb3f359e700 1 -- 172.21.15.42:0/2938785625 learned_addr learned my addr 172.21.15.42:0/2938785625 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:23:56.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:56.482+0000 7fb3ea59c700 1 --1- 172.21.15.42:0/2938785625 >> v1:172.21.15.42:6789/0 conn(0x7fb3f40a62b0 0x7fb3f40a1e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:32884/0 (socket says 172.21.15.42:32884) 2019-12-05T05:23:56.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:56.482+0000 7fb3f2d9d700 1 -- 172.21.15.42:0/2938785625 >> v2:172.21.15.42:3300/0 conn(0x7fb3f40a2640 msgr2=0x7fb3f407aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:56.496 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:56.482+0000 7fb3f2d9d700 1 --2- 172.21.15.42:0/2938785625 >> v2:172.21.15.42:3300/0 conn(0x7fb3f40a2640 0x7fb3f407aa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:23:56.496 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:56.482+0000 7fb3f2d9d700 1 -- 172.21.15.42:0/2938785625 >> v1:172.21.15.42:6789/0 conn(0x7fb3f40a62b0 legacy=0x7fb3f40a1e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:56.496 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:56.482+0000 7fb3f2d9d700 1 -- 172.21.15.42:0/2938785625 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3f407af80 con 0x7fb3f414a1c0 2019-12-05T05:23:56.496 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:56.482+0000 7fb3ebfff700 1 -- 172.21.15.42:0/2938785625 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1718626625 0 0) 0x7fb3dc008490 con 0x7fb3f40a62b0 2019-12-05T05:23:56.496 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:56.482+0000 7fb3f359e700 1 --2- 172.21.15.42:0/2938785625 >> v2:172.21.15.42:3300/0 conn(0x7fb3f40a2640 0x7fb3f407aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:23:56.496 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:56.482+0000 7fb3f2d9d700 1 --2- 172.21.15.42:0/2938785625 >> v2:172.21.15.40:3300/0 conn(0x7fb3f414a1c0 0x7fb3f40a3c00 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fb3f40a6b80 tx=0x7fb3e40108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:56.496 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:56.482+0000 7fb3f2d9d700 1 --2- 172.21.15.42:0/2938785625 >> v2:172.21.15.40:3300/0 conn(0x7fb3f414a1c0 0x7fb3f40a3c00 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fb3f40a6b80 tx=0x7fb3e40108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:56.497 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:56.482+0000 7fb3f2d9d700 1 --2- 172.21.15.42:0/2938785625 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb3f414a1c0 0x7fb3f40a3c00 secure :-1 s=READY pgs=629 cs=0 l=1 rx=0x7fb3f40a6b80 tx=0x7fb3e40108f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:23:56.497 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:56.482+0000 7fb3f2d9d700 1 --2- 172.21.15.42:0/2938785625 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb3f414a1c0 0x7fb3f40a3c00 secure :-1 s=THROTTLE_DONE pgs=629 cs=0 l=1 rx=0x7fb3f40a6b80 tx=0x7fb3e40108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:56.497 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:56.482+0000 7fb3ebfff700 1 -- 172.21.15.42:0/2938785625 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb3e4005af0 con 0x7fb3f414a1c0 2019-12-05T05:23:56.497 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:56.482+0000 7fb3f2d9d700 1 --2- 172.21.15.42:0/2938785625 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb3f414a1c0 0x7fb3f40a3c00 secure :-1 s=THROTTLE_DONE pgs=629 cs=0 l=1 rx=0x7fb3f40a6b80 tx=0x7fb3e40108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:56.497 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:56.482+0000 7fb3f8c02700 1 -- 172.21.15.42:0/2938785625 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3f407b1f0 con 0x7fb3f414a1c0 2019-12-05T05:23:56.497 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:56.482+0000 7fb3ebfff700 1 -- 172.21.15.42:0/2938785625 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3e4005710 con 0x7fb3f414a1c0 2019-12-05T05:23:56.497 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:56.482+0000 7fb3f2d9d700 1 --2- 172.21.15.42:0/2938785625 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb3f414a1c0 0x7fb3f40a3c00 secure :-1 s=THROTTLE_DONE pgs=629 cs=0 l=1 rx=0x7fb3f40a6b80 tx=0x7fb3e40108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:56.498 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:56.482+0000 7fb3ebfff700 1 -- 172.21.15.42:0/2938785625 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb3e40062b0 con 0x7fb3f414a1c0 2019-12-05T05:23:56.498 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:56.482+0000 7fb3f8c02700 1 -- 172.21.15.42:0/2938785625 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3f407b720 con 0x7fb3f414a1c0 2019-12-05T05:23:56.498 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:56.486+0000 7fb3f2d9d700 1 --2- 172.21.15.42:0/2938785625 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb3f414a1c0 0x7fb3f40a3c00 secure :-1 s=THROTTLE_DONE pgs=629 cs=0 l=1 rx=0x7fb3f40a6b80 tx=0x7fb3e40108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:56.498 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:56.486+0000 7fb3ebfff700 1 -- 172.21.15.42:0/2938785625 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fb3e4006500 con 0x7fb3f414a1c0 2019-12-05T05:23:56.498 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:56.486+0000 7fb3f2d9d700 1 --2- 172.21.15.42:0/2938785625 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb3f414a1c0 0x7fb3f40a3c00 secure :-1 s=THROTTLE_DONE pgs=629 cs=0 l=1 rx=0x7fb3f40a6b80 tx=0x7fb3e40108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:56.498 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:56.486+0000 7fb3ebfff700 1 --2- 172.21.15.42:0/2938785625 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb3d803ad00 0x7fb3d803d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:23:56.499 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:56.486+0000 7fb3ebfff700 1 -- 172.21.15.42:0/2938785625 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 7235+0+0 (secure 0 0 0) 0x7fb3e404af30 con 0x7fb3f414a1c0 2019-12-05T05:23:56.499 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:56.486+0000 7fb3ea59c700 1 --2- 172.21.15.42:0/2938785625 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb3d803ad00 0x7fb3d803d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:23:56.499 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:56.486+0000 7fb3ea59c700 1 --2- 172.21.15.42:0/2938785625 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb3d803ad00 0x7fb3d803d210 crc :-1 s=READY pgs=349 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:23:56.500 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:56.490+0000 7fb3f8c02700 1 -- 172.21.15.42:0/2938785625 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3f405d110 con 0x7fb3f414a1c0 2019-12-05T05:23:56.501 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:56.498+0000 7fb3f2d9d700 1 --2- 172.21.15.42:0/2938785625 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb3f414a1c0 0x7fb3f40a3c00 secure :-1 s=THROTTLE_DONE pgs=629 cs=0 l=1 rx=0x7fb3f40a6b80 tx=0x7fb3e40108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:56.501 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:56.498+0000 7fb3ebfff700 1 -- 172.21.15.42:0/2938785625 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fb3e4031f90 con 0x7fb3f414a1c0 2019-12-05T05:23:56.673 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:56.670+0000 7fb3f8c02700 1 -- 172.21.15.42:0/2938785625 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb3f408e4d0 con 0x7fb3f414a1c0 2019-12-05T05:23:56.674 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:56.670+0000 7fb3f2d9d700 1 --2- 172.21.15.42:0/2938785625 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb3f414a1c0 0x7fb3f40a3c00 secure :-1 s=THROTTLE_DONE pgs=629 cs=0 l=1 rx=0x7fb3f40a6b80 tx=0x7fb3e40108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:56.674 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:56.670+0000 7fb3ebfff700 1 -- 172.21.15.42:0/2938785625 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+1777 (secure 0 0 0) 0x7fb3e40321f0 con 0x7fb3f414a1c0 2019-12-05T05:23:56.676 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 100 2019-12-05T05:23:56.676 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:23:56.677 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":100,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:23:56.075994+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5177":{"gid":5177,"name":"b","rank":0,"incarnation":99,"state":"up:reconnect","state_seq":12,"addr":"172.21.15.40:6833/3208416000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3208416000},{"type":"v1","addr":"172.21.15.40:6833","nonce":3208416000}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:23:56.680 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:56.678+0000 7fb3f8c02700 1 -- 172.21.15.42:0/2938785625 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb3d803ad00 msgr2=0x7fb3d803d210 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:56.681 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:56.678+0000 7fb3f8c02700 1 --2- 172.21.15.42:0/2938785625 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb3d803ad00 0x7fb3d803d210 crc :-1 s=READY pgs=349 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:23:56.681 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:56.678+0000 7fb3f8c02700 1 -- 172.21.15.42:0/2938785625 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb3f414a1c0 msgr2=0x7fb3f40a3c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:56.681 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:56.678+0000 7fb3f8c02700 1 --2- 172.21.15.42:0/2938785625 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb3f414a1c0 0x7fb3f40a3c00 secure :-1 s=READY pgs=629 cs=0 l=1 rx=0x7fb3f40a6b80 tx=0x7fb3e40108f0).stop 2019-12-05T05:23:56.681 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:56.678+0000 7fb3f8c02700 1 -- 172.21.15.42:0/2938785625 shutdown_connections 2019-12-05T05:23:56.681 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:56.678+0000 7fb3f8c02700 1 --2- 172.21.15.42:0/2938785625 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb3d803ad00 0x7fb3d803d210 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:23:56.681 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:56.678+0000 7fb3f8c02700 1 --2- 172.21.15.42:0/2938785625 >> v2:172.21.15.42:3300/0 conn(0x7fb3f40a2640 0x7fb3f407aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:23:56.681 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:56.678+0000 7fb3f8c02700 1 --2- 172.21.15.42:0/2938785625 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb3f414a1c0 0x7fb3f40a3c00 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:23:56.682 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:56.678+0000 7fb3f8c02700 1 -- 172.21.15.42:0/2938785625 shutdown_connections 2019-12-05T05:23:56.682 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:56.678+0000 7fb3f8c02700 1 -- 172.21.15.42:0/2938785625 wait complete. 2019-12-05T05:23:56.682 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:56.678+0000 7fb3f8c02700 1 -- 172.21.15.42:0/2938785625 >> 172.21.15.42:0/2938785625 conn(0x7fb3f4059700 msgr2=0x7fb3f41162d0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:23:56.694 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2019-12-05T05:23:58.695 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:23:58.797 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:58.790+0000 7ff73031d700 1 Processor -- start 2019-12-05T05:23:58.798 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:58.790+0000 7ff73031d700 1 -- start start 2019-12-05T05:23:58.798 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:58.790+0000 7ff73031d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff7280a7740 0x7ff72808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:23:58.798 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:58.790+0000 7ff73031d700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff72807e720 0x7ff72807eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:23:58.798 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:58.790+0000 7ff73031d700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7280a0710 con 0x7ff72809f2e0 2019-12-05T05:23:58.799 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:58.790+0000 7ff73031d700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7280a0b00 con 0x7ff7280a7740 2019-12-05T05:23:58.799 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:58.790+0000 7ff73031d700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff72808e5b0 con 0x7ff72807e720 2019-12-05T05:23:58.799 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:58.790+0000 7ff72e8ba700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff7280a7740 0x7ff72808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:23:58.799 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:58.790+0000 7ff72e8ba700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff7280a7740 0x7ff72808df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:32854/0 (socket says 172.21.15.42:32854) 2019-12-05T05:23:58.799 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:58.790+0000 7ff72e8ba700 1 -- 172.21.15.42:0/2633601367 learned_addr learned my addr 172.21.15.42:0/2633601367 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:23:58.799 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:58.790+0000 7ff72e0b9700 1 --2- 172.21.15.42:0/2633601367 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff72807e720 0x7ff72807eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:23:58.800 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:58.790+0000 7ff72e8ba700 1 -- 172.21.15.42:0/2633601367 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff72807e720 msgr2=0x7ff72807eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:23:58.800 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:58.790+0000 7ff72e8ba700 1 --2- 172.21.15.42:0/2633601367 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff72807e720 0x7ff72807eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:23:58.800 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:58.790+0000 7ff72e8ba700 1 -- 172.21.15.42:0/2633601367 >> v1:172.21.15.40:6789/0 conn(0x7ff72809f2e0 legacy=0x7ff72809f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:58.800 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:58.790+0000 7ff72e8ba700 1 -- 172.21.15.42:0/2633601367 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff72807f0a0 con 0x7ff7280a7740 2019-12-05T05:23:58.801 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:58.790+0000 7ff72e8ba700 1 --2- 172.21.15.42:0/2633601367 >> v2:172.21.15.42:3300/0 conn(0x7ff7280a7740 0x7ff72808df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7ff71c008b10 tx=0x7ff71c0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:58.801 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:58.790+0000 7ff72e8ba700 1 --2- 172.21.15.42:0/2633601367 >> v2:172.21.15.42:3300/0 conn(0x7ff7280a7740 0x7ff72808df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7ff71c008b10 tx=0x7ff71c0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:58.801 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:58.790+0000 7ff72e8ba700 1 --2- 172.21.15.42:0/2633601367 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff7280a7740 0x7ff72808df80 secure :-1 s=READY pgs=791 cs=0 l=1 rx=0x7ff71c008b10 tx=0x7ff71c0100b0).ready entity=mon.1 client_cookie=a994ae401a27d9a3 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:23:58.801 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:58.790+0000 7ff72e8ba700 1 --2- 172.21.15.42:0/2633601367 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff7280a7740 0x7ff72808df80 secure :-1 s=THROTTLE_DONE pgs=791 cs=0 l=1 rx=0x7ff71c008b10 tx=0x7ff71c0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:58.801 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:58.790+0000 7ff72e8ba700 1 --2- 172.21.15.42:0/2633601367 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff7280a7740 0x7ff72808df80 secure :-1 s=THROTTLE_DONE pgs=791 cs=0 l=1 rx=0x7ff71c008b10 tx=0x7ff71c0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:58.802 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:58.790+0000 7ff72d8b8700 1 -- 172.21.15.42:0/2633601367 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff71c00a160 con 0x7ff7280a7740 2019-12-05T05:23:58.802 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:58.790+0000 7ff72e8ba700 1 --2- 172.21.15.42:0/2633601367 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff7280a7740 0x7ff72808df80 secure :-1 s=THROTTLE_DONE pgs=791 cs=0 l=1 rx=0x7ff71c008b10 tx=0x7ff71c0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:58.802 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:58.794+0000 7ff72d8b8700 1 -- 172.21.15.42:0/2633601367 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff71c011a30 con 0x7ff7280a7740 2019-12-05T05:23:58.802 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:58.794+0000 7ff72d8b8700 1 -- 172.21.15.42:0/2633601367 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff71c00a550 con 0x7ff7280a7740 2019-12-05T05:23:58.803 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:58.794+0000 7ff73031d700 1 -- 172.21.15.42:0/2633601367 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff7280a7740 msgr2=0x7ff72808df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:58.803 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:58.794+0000 7ff73031d700 1 --2- 172.21.15.42:0/2633601367 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff7280a7740 0x7ff72808df80 secure :-1 s=READY pgs=791 cs=0 l=1 rx=0x7ff71c008b10 tx=0x7ff71c0100b0).stop 2019-12-05T05:23:58.803 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:58.794+0000 7ff73031d700 1 -- 172.21.15.42:0/2633601367 shutdown_connections 2019-12-05T05:23:58.803 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:58.794+0000 7ff73031d700 1 --2- 172.21.15.42:0/2633601367 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff7280a7740 0x7ff72808df80 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:23:58.803 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:58.794+0000 7ff73031d700 1 --2- 172.21.15.42:0/2633601367 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff72807e720 0x7ff72807eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:23:58.804 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:58.794+0000 7ff73031d700 1 -- 172.21.15.42:0/2633601367 shutdown_connections 2019-12-05T05:23:58.804 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:58.794+0000 7ff73031d700 1 -- 172.21.15.42:0/2633601367 wait complete. 2019-12-05T05:23:58.804 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:58.794+0000 7ff73031d700 1 -- 172.21.15.42:0/2633601367 >> 172.21.15.42:0/2633601367 conn(0x7ff7280aabe0 msgr2=0x7ff7280a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:23:58.808 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:58.794+0000 7ff73031d700 1 Processor -- start 2019-12-05T05:23:58.809 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:58.794+0000 7ff73031d700 1 -- start start 2019-12-05T05:23:58.809 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:58.798+0000 7ff73031d700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff728143900 0x7ff728142530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:23:58.809 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:58.798+0000 7ff73031d700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff72808ec10 0x7ff72808f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:23:58.810 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:58.798+0000 7ff73031d700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff728144350 con 0x7ff728115cc0 2019-12-05T05:23:58.810 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:58.798+0000 7ff73031d700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff728143e10 con 0x7ff728143900 2019-12-05T05:23:58.810 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:58.798+0000 7ff73031d700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7281440b0 con 0x7ff72808ec10 2019-12-05T05:23:58.810 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:58.798+0000 7ff72e0b9700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7ff728115cc0 0x7ff728078800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:32900/0 (socket says 172.21.15.42:32900) 2019-12-05T05:23:58.810 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:58.798+0000 7ff72e0b9700 1 -- 172.21.15.42:0/2754708450 learned_addr learned my addr 172.21.15.42:0/2754708450 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:23:58.811 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:58.798+0000 7ff72e8ba700 1 --2- 172.21.15.42:0/2754708450 >> v2:172.21.15.40:3300/0 conn(0x7ff728143900 0x7ff728142530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:23:58.812 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:58.798+0000 7ff72f0bb700 1 --2- 172.21.15.42:0/2754708450 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff72808ec10 0x7ff72808f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:23:58.812 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:58.798+0000 7ff7277fe700 1 -- 172.21.15.42:0/2754708450 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3750904206 0 0) 0x7ff728144350 con 0x7ff728115cc0 2019-12-05T05:23:58.812 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:58.798+0000 7ff7277fe700 1 -- 172.21.15.42:0/2754708450 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7100030b0 con 0x7ff728115cc0 2019-12-05T05:23:58.812 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:58.798+0000 7ff72e8ba700 1 -- 172.21.15.42:0/2754708450 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff72808ec10 msgr2=0x7ff72808f040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:58.812 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:58.798+0000 7ff72e8ba700 1 --2- 172.21.15.42:0/2754708450 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff72808ec10 0x7ff72808f040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:23:58.813 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:58.798+0000 7ff72e8ba700 1 -- 172.21.15.42:0/2754708450 >> v1:172.21.15.42:6789/0 conn(0x7ff728115cc0 legacy=0x7ff728078800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:58.813 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:58.798+0000 7ff72e8ba700 1 -- 172.21.15.42:0/2754708450 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff728079000 con 0x7ff728143900 2019-12-05T05:23:58.813 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:58.798+0000 7ff72e8ba700 1 --2- 172.21.15.42:0/2754708450 >> v2:172.21.15.40:3300/0 conn(0x7ff728143900 0x7ff728142530 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7ff728125bd0 tx=0x7ff71c011ce0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:58.813 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:58.798+0000 7ff72e8ba700 1 --2- 172.21.15.42:0/2754708450 >> v2:172.21.15.40:3300/0 conn(0x7ff728143900 0x7ff728142530 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7ff728125bd0 tx=0x7ff71c011ce0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:58.813 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:58.798+0000 7ff72e8ba700 1 --2- 172.21.15.42:0/2754708450 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff728143900 0x7ff728142530 secure :-1 s=READY pgs=631 cs=0 l=1 rx=0x7ff728125bd0 tx=0x7ff71c011ce0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:23:58.813 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:58.798+0000 7ff72e8ba700 1 --2- 172.21.15.42:0/2754708450 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff728143900 0x7ff728142530 secure :-1 s=THROTTLE_DONE pgs=631 cs=0 l=1 rx=0x7ff728125bd0 tx=0x7ff71c011ce0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:58.814 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:58.798+0000 7ff7277fe700 1 -- 172.21.15.42:0/2754708450 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff71c00ad60 con 0x7ff728143900 2019-12-05T05:23:58.814 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:58.798+0000 7ff72e8ba700 1 --2- 172.21.15.42:0/2754708450 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff728143900 0x7ff728142530 secure :-1 s=THROTTLE_DONE pgs=631 cs=0 l=1 rx=0x7ff728125bd0 tx=0x7ff71c011ce0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:58.814 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:58.798+0000 7ff7277fe700 1 -- 172.21.15.42:0/2754708450 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff71c011e10 con 0x7ff728143900 2019-12-05T05:23:58.814 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:58.798+0000 7ff72e8ba700 1 --2- 172.21.15.42:0/2754708450 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff728143900 0x7ff728142530 secure :-1 s=THROTTLE_DONE pgs=631 cs=0 l=1 rx=0x7ff728125bd0 tx=0x7ff71c011ce0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:58.815 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:58.798+0000 7ff73031d700 1 -- 172.21.15.42:0/2754708450 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff72808f590 con 0x7ff728143900 2019-12-05T05:23:58.815 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:58.798+0000 7ff7277fe700 1 -- 172.21.15.42:0/2754708450 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff71c005ca0 con 0x7ff728143900 2019-12-05T05:23:58.815 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:58.798+0000 7ff73031d700 1 -- 172.21.15.42:0/2754708450 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff72812f4b0 con 0x7ff728143900 2019-12-05T05:23:58.815 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:58.798+0000 7ff72e8ba700 1 --2- 172.21.15.42:0/2754708450 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff728143900 0x7ff728142530 secure :-1 s=THROTTLE_DONE pgs=631 cs=0 l=1 rx=0x7ff728125bd0 tx=0x7ff71c011ce0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:58.815 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:58.802+0000 7ff7277fe700 1 -- 172.21.15.42:0/2754708450 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7ff71c007150 con 0x7ff728143900 2019-12-05T05:23:58.816 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:58.802+0000 7ff72e8ba700 1 --2- 172.21.15.42:0/2754708450 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff728143900 0x7ff728142530 secure :-1 s=THROTTLE_DONE pgs=631 cs=0 l=1 rx=0x7ff728125bd0 tx=0x7ff71c011ce0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:58.816 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:58.802+0000 7ff7277fe700 1 --2- 172.21.15.42:0/2754708450 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff71003b280 0x7ff71003d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:23:58.816 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:58.802+0000 7ff7277fe700 1 -- 172.21.15.42:0/2754708450 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 7235+0+0 (secure 0 0 0) 0x7ff71c04bc30 con 0x7ff728143900 2019-12-05T05:23:58.816 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:58.802+0000 7ff72e0b9700 1 --2- 172.21.15.42:0/2754708450 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff71003b280 0x7ff71003d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:23:58.816 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:58.802+0000 7ff72e0b9700 1 --2- 172.21.15.42:0/2754708450 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff71003b280 0x7ff71003d790 crc :-1 s=READY pgs=350 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:23:58.817 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:58.806+0000 7ff73031d700 1 -- 172.21.15.42:0/2754708450 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff72805d110 con 0x7ff728143900 2019-12-05T05:23:58.818 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:58.814+0000 7ff72e8ba700 1 --2- 172.21.15.42:0/2754708450 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff728143900 0x7ff728142530 secure :-1 s=THROTTLE_DONE pgs=631 cs=0 l=1 rx=0x7ff728125bd0 tx=0x7ff71c011ce0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:58.818 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:58.814+0000 7ff7277fe700 1 -- 172.21.15.42:0/2754708450 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7ff71c032c90 con 0x7ff728143900 2019-12-05T05:23:58.998 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:58.994+0000 7ff73031d700 1 -- 172.21.15.42:0/2754708450 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff72805b8c0 con 0x7ff728143900 2019-12-05T05:23:58.999 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:58.994+0000 7ff72e8ba700 1 --2- 172.21.15.42:0/2754708450 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff728143900 0x7ff728142530 secure :-1 s=THROTTLE_DONE pgs=631 cs=0 l=1 rx=0x7ff728125bd0 tx=0x7ff71c011ce0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:23:59.000 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:58.994+0000 7ff7277fe700 1 -- 172.21.15.42:0/2754708450 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) v1 ==== 95+0+1774 (secure 0 0 0) 0x7ff71c032ef0 con 0x7ff728143900 2019-12-05T05:23:59.000 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 102 2019-12-05T05:23:59.000 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:23:59.000 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":102,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":102,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:23:58.091182+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5177":{"gid":5177,"name":"b","rank":0,"incarnation":99,"state":"up:active","state_seq":14,"addr":"172.21.15.40:6833/3208416000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3208416000},{"type":"v1","addr":"172.21.15.40:6833","nonce":3208416000}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:23:59.003 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:58.998+0000 7ff73031d700 1 -- 172.21.15.42:0/2754708450 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff71003b280 msgr2=0x7ff71003d790 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:59.003 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:58.998+0000 7ff73031d700 1 --2- 172.21.15.42:0/2754708450 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff71003b280 0x7ff71003d790 crc :-1 s=READY pgs=350 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:23:59.004 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:58.998+0000 7ff73031d700 1 -- 172.21.15.42:0/2754708450 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff728143900 msgr2=0x7ff728142530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:23:59.004 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:58.998+0000 7ff73031d700 1 --2- 172.21.15.42:0/2754708450 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff728143900 0x7ff728142530 secure :-1 s=READY pgs=631 cs=0 l=1 rx=0x7ff728125bd0 tx=0x7ff71c011ce0).stop 2019-12-05T05:23:59.004 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:58.998+0000 7ff73031d700 1 -- 172.21.15.42:0/2754708450 shutdown_connections 2019-12-05T05:23:59.004 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:58.998+0000 7ff73031d700 1 --2- 172.21.15.42:0/2754708450 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff71003b280 0x7ff71003d790 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:23:59.005 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:58.998+0000 7ff73031d700 1 --2- 172.21.15.42:0/2754708450 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff72808ec10 0x7ff72808f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:23:59.005 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:58.998+0000 7ff73031d700 1 --2- 172.21.15.42:0/2754708450 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff728143900 0x7ff728142530 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:23:59.005 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:58.998+0000 7ff73031d700 1 -- 172.21.15.42:0/2754708450 shutdown_connections 2019-12-05T05:23:59.005 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:58.998+0000 7ff73031d700 1 -- 172.21.15.42:0/2754708450 wait complete. 2019-12-05T05:23:59.005 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:23:58.998+0000 7ff73031d700 1 -- 172.21.15.42:0/2754708450 >> 172.21.15.42:0/2754708450 conn(0x7ff7280aabe0 msgr2=0x7ff72809f430 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:23:59.012 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b has gained rank=0, replacing gid=5123 2019-12-05T05:23:59.012 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 3 secs before reviving mds.a 2019-12-05T05:24:02.014 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.a 2019-12-05T05:24:02.014 INFO:tasks.ceph.mds.a:Restarting daemon 2019-12-05T05:24:02.014 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2019-12-05T05:24:02.018 INFO:tasks.ceph.mds.a:Started 2019-12-05T05:24:02.018 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:24:02.079 INFO:tasks.ceph.mds.a.smithi042.stdout:starting mds.a at 2019-12-05T05:24:02.126 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:02.118+0000 7f88bb124700 1 Processor -- start 2019-12-05T05:24:02.126 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:02.118+0000 7f88bb124700 1 -- start start 2019-12-05T05:24:02.126 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:02.118+0000 7f88bb124700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f88b40a7740 0x7f88b408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:24:02.126 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:02.118+0000 7f88bb124700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f88b409fae0 0x7f88b407eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:24:02.126 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:02.118+0000 7f88bb124700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88b40a0710 con 0x7f88b407e720 2019-12-05T05:24:02.127 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:02.118+0000 7f88bb124700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88b40a0b00 con 0x7f88b409fae0 2019-12-05T05:24:02.127 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:02.118+0000 7f88bb124700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88b408e5b0 con 0x7f88b40a7740 2019-12-05T05:24:02.127 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:02.118+0000 7f88b96c1700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f88b40a7740 0x7f88b408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:24:02.127 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:02.118+0000 7f88b96c1700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f88b40a7740 0x7f88b408df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:32900/0 (socket says 172.21.15.42:32900) 2019-12-05T05:24:02.127 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:02.118+0000 7f88b96c1700 1 -- 172.21.15.42:0/2547891189 learned_addr learned my addr 172.21.15.42:0/2547891189 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:24:02.127 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:02.118+0000 7f88b9ec2700 1 --2- 172.21.15.42:0/2547891189 >> v2:172.21.15.40:3300/0 conn(0x7f88b409fae0 0x7f88b407eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:24:02.127 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:02.118+0000 7f88b96c1700 1 -- 172.21.15.42:0/2547891189 >> v2:172.21.15.40:3300/0 conn(0x7f88b409fae0 msgr2=0x7f88b407eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:24:02.128 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:02.118+0000 7f88b96c1700 1 --2- 172.21.15.42:0/2547891189 >> v2:172.21.15.40:3300/0 conn(0x7f88b409fae0 0x7f88b407eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:24:02.128 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:02.118+0000 7f88b96c1700 1 -- 172.21.15.42:0/2547891189 >> v1:172.21.15.42:6789/0 conn(0x7f88b407e720 legacy=0x7f88b409f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:24:02.128 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:02.118+0000 7f88b96c1700 1 -- 172.21.15.42:0/2547891189 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88b407f0a0 con 0x7f88b40a7740 2019-12-05T05:24:02.128 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:02.118+0000 7f88b96c1700 1 --2- 172.21.15.42:0/2547891189 >> v2:172.21.15.42:3300/0 conn(0x7f88b40a7740 0x7f88b408df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f88b0002f30 tx=0x7f88b000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:24:02.128 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:02.122+0000 7f88b96c1700 1 --2- 172.21.15.42:0/2547891189 >> v2:172.21.15.42:3300/0 conn(0x7f88b40a7740 0x7f88b408df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f88b0002f30 tx=0x7f88b000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:24:02.128 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:02.122+0000 7f88b96c1700 1 --2- 172.21.15.42:0/2547891189 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f88b40a7740 0x7f88b408df80 secure :-1 s=READY pgs=797 cs=0 l=1 rx=0x7f88b0002f30 tx=0x7f88b000fd30).ready entity=mon.1 client_cookie=b65a8feb8dd5e4d0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:24:02.128 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:02.122+0000 7f88b96c1700 1 --2- 172.21.15.42:0/2547891189 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f88b40a7740 0x7f88b408df80 secure :-1 s=THROTTLE_DONE pgs=797 cs=0 l=1 rx=0x7f88b0002f30 tx=0x7f88b000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:24:02.129 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:02.122+0000 7f88abfff700 1 -- 172.21.15.42:0/2547891189 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f88b0008c90 con 0x7f88b40a7740 2019-12-05T05:24:02.129 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:02.122+0000 7f88b96c1700 1 --2- 172.21.15.42:0/2547891189 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f88b40a7740 0x7f88b408df80 secure :-1 s=THROTTLE_DONE pgs=797 cs=0 l=1 rx=0x7f88b0002f30 tx=0x7f88b000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:24:02.130 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:02.122+0000 7f88b96c1700 1 --2- 172.21.15.42:0/2547891189 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f88b40a7740 0x7f88b408df80 secure :-1 s=THROTTLE_DONE pgs=797 cs=0 l=1 rx=0x7f88b0002f30 tx=0x7f88b000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:24:02.130 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:02.122+0000 7f88abfff700 1 -- 172.21.15.42:0/2547891189 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88b00088b0 con 0x7f88b40a7740 2019-12-05T05:24:02.130 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:02.122+0000 7f88abfff700 1 -- 172.21.15.42:0/2547891189 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f88b0011ad0 con 0x7f88b40a7740 2019-12-05T05:24:02.130 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:02.122+0000 7f88bb124700 1 -- 172.21.15.42:0/2547891189 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f88b40a7740 msgr2=0x7f88b408df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:24:02.130 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:02.122+0000 7f88bb124700 1 --2- 172.21.15.42:0/2547891189 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f88b40a7740 0x7f88b408df80 secure :-1 s=READY pgs=797 cs=0 l=1 rx=0x7f88b0002f30 tx=0x7f88b000fd30).stop 2019-12-05T05:24:02.131 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:02.122+0000 7f88bb124700 1 -- 172.21.15.42:0/2547891189 shutdown_connections 2019-12-05T05:24:02.131 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:02.122+0000 7f88bb124700 1 --2- 172.21.15.42:0/2547891189 >> v2:172.21.15.40:3300/0 conn(0x7f88b409fae0 0x7f88b407eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:24:02.131 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:02.122+0000 7f88bb124700 1 --2- 172.21.15.42:0/2547891189 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f88b40a7740 0x7f88b408df80 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:24:02.131 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:02.122+0000 7f88bb124700 1 -- 172.21.15.42:0/2547891189 shutdown_connections 2019-12-05T05:24:02.131 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:02.122+0000 7f88bb124700 1 -- 172.21.15.42:0/2547891189 wait complete. 2019-12-05T05:24:02.131 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:02.122+0000 7f88bb124700 1 -- 172.21.15.42:0/2547891189 >> 172.21.15.42:0/2547891189 conn(0x7f88b40aabe0 msgr2=0x7f88b40a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:24:02.134 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:02.122+0000 7f88bb124700 1 Processor -- start 2019-12-05T05:24:02.135 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:02.122+0000 7f88bb124700 1 -- start start 2019-12-05T05:24:02.135 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:02.126+0000 7f88bb124700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f88b4063520 0x7f88b4065430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:24:02.135 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:02.126+0000 7f88bb124700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f88b408ecf0 0x7f88b411b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:24:02.135 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:02.126+0000 7f88bb124700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f88b411b910 0x7f88b4126e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:24:02.135 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:02.126+0000 7f88bb124700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88b4063950 con 0x7f88b4063520 2019-12-05T05:24:02.135 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:02.126+0000 7f88bb124700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88b4063bc0 con 0x7f88b411b910 2019-12-05T05:24:02.136 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:02.126+0000 7f88bb124700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88b4063e60 con 0x7f88b408ecf0 2019-12-05T05:24:02.136 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:02.126+0000 7f88b9ec2700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f88b411b910 0x7f88b4126e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:24:02.136 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:02.126+0000 7f88b8ec0700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f88b408ecf0 0x7f88b411b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:24:02.136 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:02.126+0000 7f88b9ec2700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f88b411b910 0x7f88b4126e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:32910/0 (socket says 172.21.15.42:32910) 2019-12-05T05:24:02.136 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:02.126+0000 7f88b96c1700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f88b4063520 0x7f88b4065430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:24:02.136 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:02.126+0000 7f88b9ec2700 1 -- 172.21.15.42:0/437564376 learned_addr learned my addr 172.21.15.42:0/437564376 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:24:02.136 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:02.126+0000 7f88b9ec2700 1 -- 172.21.15.42:0/437564376 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f88b408ecf0 msgr2=0x7f88b411b3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:24:02.137 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:02.126+0000 7f88b9ec2700 1 --2- 172.21.15.42:0/437564376 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f88b408ecf0 0x7f88b411b3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:24:02.137 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:02.126+0000 7f88b9ec2700 1 -- 172.21.15.42:0/437564376 >> v2:172.21.15.40:3300/0 conn(0x7f88b4063520 msgr2=0x7f88b4065430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:24:02.138 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:02.126+0000 7f88b9ec2700 1 --2- 172.21.15.42:0/437564376 >> v2:172.21.15.40:3300/0 conn(0x7f88b4063520 0x7f88b4065430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:24:02.138 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:02.126+0000 7f88b9ec2700 1 -- 172.21.15.42:0/437564376 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88b408f120 con 0x7f88b411b910 2019-12-05T05:24:02.138 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:02.126+0000 7f88b9ec2700 1 --2- 172.21.15.42:0/437564376 >> v2:172.21.15.42:3300/0 conn(0x7f88b411b910 0x7f88b4126e30 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f88b4064020 tx=0x7f88ac010d80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:24:02.138 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:02.126+0000 7f88b96c1700 1 --2- 172.21.15.42:0/437564376 >> v2:172.21.15.40:3300/0 conn(0x7f88b4063520 0x7f88b4065430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:24:02.138 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:02.126+0000 7f88b9ec2700 1 --2- 172.21.15.42:0/437564376 >> v2:172.21.15.42:3300/0 conn(0x7f88b411b910 0x7f88b4126e30 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f88b4064020 tx=0x7f88ac010d80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:24:02.139 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:02.126+0000 7f88b9ec2700 1 --2- 172.21.15.42:0/437564376 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f88b411b910 0x7f88b4126e30 secure :-1 s=READY pgs=798 cs=0 l=1 rx=0x7f88b4064020 tx=0x7f88ac010d80).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:24:02.139 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:02.126+0000 7f88b9ec2700 1 --2- 172.21.15.42:0/437564376 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f88b411b910 0x7f88b4126e30 secure :-1 s=THROTTLE_DONE pgs=798 cs=0 l=1 rx=0x7f88b4064020 tx=0x7f88ac010d80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:24:02.139 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:02.126+0000 7f88b9ec2700 1 --2- 172.21.15.42:0/437564376 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f88b411b910 0x7f88b4126e30 secure :-1 s=THROTTLE_DONE pgs=798 cs=0 l=1 rx=0x7f88b4064020 tx=0x7f88ac010d80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:24:02.140 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:02.126+0000 7f88a9ffb700 1 -- 172.21.15.42:0/437564376 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f88ac012a70 con 0x7f88b411b910 2019-12-05T05:24:02.140 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:02.126+0000 7f88b9ec2700 1 --2- 172.21.15.42:0/437564376 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f88b411b910 0x7f88b4126e30 secure :-1 s=THROTTLE_DONE pgs=798 cs=0 l=1 rx=0x7f88b4064020 tx=0x7f88ac010d80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:24:02.140 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:02.126+0000 7f88a9ffb700 1 -- 172.21.15.42:0/437564376 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88ac012690 con 0x7f88b411b910 2019-12-05T05:24:02.140 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:02.126+0000 7f88a9ffb700 1 -- 172.21.15.42:0/437564376 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f88ac00e760 con 0x7f88b411b910 2019-12-05T05:24:02.140 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:02.126+0000 7f88bb124700 1 -- 172.21.15.42:0/437564376 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88b4127380 con 0x7f88b411b910 2019-12-05T05:24:02.141 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:02.126+0000 7f88bb124700 1 -- 172.21.15.42:0/437564376 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88b4127930 con 0x7f88b411b910 2019-12-05T05:24:02.141 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:02.126+0000 7f88b9ec2700 1 --2- 172.21.15.42:0/437564376 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f88b411b910 0x7f88b4126e30 secure :-1 s=THROTTLE_DONE pgs=798 cs=0 l=1 rx=0x7f88b4064020 tx=0x7f88ac010d80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:24:02.141 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:02.126+0000 7f88a9ffb700 1 -- 172.21.15.42:0/437564376 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f88ac00ea70 con 0x7f88b411b910 2019-12-05T05:24:02.141 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:02.126+0000 7f88b9ec2700 1 --2- 172.21.15.42:0/437564376 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f88b411b910 0x7f88b4126e30 secure :-1 s=THROTTLE_DONE pgs=798 cs=0 l=1 rx=0x7f88b4064020 tx=0x7f88ac010d80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:24:02.141 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:02.130+0000 7f88a9ffb700 1 --2- 172.21.15.42:0/437564376 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f889403ae60 0x7f889403d370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:24:02.141 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:02.130+0000 7f88a9ffb700 1 -- 172.21.15.42:0/437564376 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 7235+0+0 (secure 0 0 0) 0x7f88ac04b900 con 0x7f88b411b910 2019-12-05T05:24:02.142 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:02.130+0000 7f88b96c1700 1 --2- 172.21.15.42:0/437564376 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f889403ae60 0x7f889403d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:24:02.142 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:02.130+0000 7f88b96c1700 1 --2- 172.21.15.42:0/437564376 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f889403ae60 0x7f889403d370 crc :-1 s=READY pgs=351 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:24:02.143 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:02.130+0000 7f88bb124700 1 -- 172.21.15.42:0/437564376 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88b405d110 con 0x7f88b411b910 2019-12-05T05:24:02.143 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:02.138+0000 7f88b9ec2700 1 --2- 172.21.15.42:0/437564376 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f88b411b910 0x7f88b4126e30 secure :-1 s=THROTTLE_DONE pgs=798 cs=0 l=1 rx=0x7f88b4064020 tx=0x7f88ac010d80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:24:02.143 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:02.138+0000 7f88a9ffb700 1 -- 172.21.15.42:0/437564376 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f88ac032980 con 0x7f88b411b910 2019-12-05T05:24:02.315 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:02.310+0000 7f88bb124700 1 -- 172.21.15.42:0/437564376 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f88b405b8c0 con 0x7f88b411b910 2019-12-05T05:24:02.316 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:02.314+0000 7f88b9ec2700 1 --2- 172.21.15.42:0/437564376 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f88b411b910 0x7f88b4126e30 secure :-1 s=THROTTLE_DONE pgs=798 cs=0 l=1 rx=0x7f88b4064020 tx=0x7f88ac010d80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:24:02.316 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:02.314+0000 7f88a9ffb700 1 -- 172.21.15.42:0/437564376 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) v1 ==== 95+0+1774 (secure 0 0 0) 0x7f88ac032be0 con 0x7f88b411b910 2019-12-05T05:24:02.316 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 102 2019-12-05T05:24:02.317 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:24:02.317 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":102,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":102,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:23:58.091182+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5177":{"gid":5177,"name":"b","rank":0,"incarnation":99,"state":"up:active","state_seq":14,"addr":"172.21.15.40:6833/3208416000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3208416000},{"type":"v1","addr":"172.21.15.40:6833","nonce":3208416000}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:24:02.319 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:02.314+0000 7f88bb124700 1 -- 172.21.15.42:0/437564376 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f889403ae60 msgr2=0x7f889403d370 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:24:02.319 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:02.314+0000 7f88bb124700 1 --2- 172.21.15.42:0/437564376 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f889403ae60 0x7f889403d370 crc :-1 s=READY pgs=351 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:24:02.320 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:02.314+0000 7f88bb124700 1 -- 172.21.15.42:0/437564376 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f88b411b910 msgr2=0x7f88b4126e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:24:02.320 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:02.314+0000 7f88bb124700 1 --2- 172.21.15.42:0/437564376 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f88b411b910 0x7f88b4126e30 secure :-1 s=READY pgs=798 cs=0 l=1 rx=0x7f88b4064020 tx=0x7f88ac010d80).stop 2019-12-05T05:24:02.320 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:02.314+0000 7f88bb124700 1 -- 172.21.15.42:0/437564376 shutdown_connections 2019-12-05T05:24:02.320 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:02.314+0000 7f88bb124700 1 --2- 172.21.15.42:0/437564376 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f889403ae60 0x7f889403d370 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:24:02.320 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:02.314+0000 7f88bb124700 1 --2- 172.21.15.42:0/437564376 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f88b411b910 0x7f88b4126e30 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:24:02.320 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:02.314+0000 7f88bb124700 1 --2- 172.21.15.42:0/437564376 >> v2:172.21.15.40:3300/0 conn(0x7f88b4063520 0x7f88b4065430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:24:02.321 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:02.314+0000 7f88bb124700 1 --2- 172.21.15.42:0/437564376 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f88b408ecf0 0x7f88b411b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:24:02.321 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:02.314+0000 7f88bb124700 1 -- 172.21.15.42:0/437564376 shutdown_connections 2019-12-05T05:24:02.321 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:02.314+0000 7f88bb124700 1 -- 172.21.15.42:0/437564376 wait complete. 2019-12-05T05:24:02.321 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:02.314+0000 7f88bb124700 1 -- 172.21.15.42:0/437564376 >> 172.21.15.42:0/437564376 conn(0x7f88b40a89d0 msgr2=0x7f88b409f620 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:24:02.333 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2019-12-05T05:24:04.334 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:24:04.436 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:04.426+0000 7fb2d5d4d700 1 Processor -- start 2019-12-05T05:24:04.437 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:04.426+0000 7fb2d5d4d700 1 -- start start 2019-12-05T05:24:04.437 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:04.430+0000 7fb2d5d4d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb2d00a7740 0x7fb2d008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:24:04.437 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:04.430+0000 7fb2d5d4d700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb2d007e720 0x7fb2d007eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:24:04.437 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:04.430+0000 7fb2d5d4d700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2d00a0710 con 0x7fb2d009f2e0 2019-12-05T05:24:04.437 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:04.430+0000 7fb2d5d4d700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2d00a0b00 con 0x7fb2d00a7740 2019-12-05T05:24:04.437 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:04.430+0000 7fb2d5d4d700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2d008e5b0 con 0x7fb2d007e720 2019-12-05T05:24:04.438 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:04.430+0000 7fb2cffff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb2d00a7740 0x7fb2d008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:24:04.438 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:04.430+0000 7fb2d4aeb700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fb2d009f2e0 0x7fb2d009f710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:32956/0 (socket says 172.21.15.42:32956) 2019-12-05T05:24:04.438 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:04.430+0000 7fb2d4aeb700 1 -- 172.21.15.42:0/2700046386 learned_addr learned my addr 172.21.15.42:0/2700046386 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:24:04.438 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:04.430+0000 7fb2cf7fe700 1 --2- 172.21.15.42:0/2700046386 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb2d007e720 0x7fb2d007eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:24:04.438 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:04.430+0000 7fb2cffff700 1 -- 172.21.15.42:0/2700046386 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb2d007e720 msgr2=0x7fb2d007eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:24:04.438 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:04.430+0000 7fb2cffff700 1 --2- 172.21.15.42:0/2700046386 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb2d007e720 0x7fb2d007eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:24:04.438 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:04.430+0000 7fb2cffff700 1 -- 172.21.15.42:0/2700046386 >> v1:172.21.15.42:6789/0 conn(0x7fb2d009f2e0 legacy=0x7fb2d009f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:24:04.439 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:04.430+0000 7fb2cffff700 1 -- 172.21.15.42:0/2700046386 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2d007f0a0 con 0x7fb2d00a7740 2019-12-05T05:24:04.439 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:04.430+0000 7fb2cffff700 1 --2- 172.21.15.42:0/2700046386 >> v2:172.21.15.42:3300/0 conn(0x7fb2d00a7740 0x7fb2d008df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fb2c8008790 tx=0x7fb2c00014d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:24:04.439 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:04.430+0000 7fb2cffff700 1 --2- 172.21.15.42:0/2700046386 >> v2:172.21.15.42:3300/0 conn(0x7fb2d00a7740 0x7fb2d008df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fb2c8008790 tx=0x7fb2c00014d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:24:04.439 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:04.430+0000 7fb2cffff700 1 --2- 172.21.15.42:0/2700046386 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb2d00a7740 0x7fb2d008df80 secure :-1 s=READY pgs=800 cs=0 l=1 rx=0x7fb2c8008790 tx=0x7fb2c00014d0).ready entity=mon.1 client_cookie=7fa7ac5eb3ae3ba6 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:24:04.439 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:04.430+0000 7fb2cffff700 1 --2- 172.21.15.42:0/2700046386 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb2d00a7740 0x7fb2d008df80 secure :-1 s=THROTTLE_DONE pgs=800 cs=0 l=1 rx=0x7fb2c8008790 tx=0x7fb2c00014d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:24:04.439 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:04.430+0000 7fb2ceffd700 1 -- 172.21.15.42:0/2700046386 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb2c8008bb0 con 0x7fb2d00a7740 2019-12-05T05:24:04.439 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:04.430+0000 7fb2cffff700 1 --2- 172.21.15.42:0/2700046386 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb2d00a7740 0x7fb2d008df80 secure :-1 s=THROTTLE_DONE pgs=800 cs=0 l=1 rx=0x7fb2c8008790 tx=0x7fb2c00014d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:24:04.440 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:04.430+0000 7fb2cffff700 1 --2- 172.21.15.42:0/2700046386 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb2d00a7740 0x7fb2d008df80 secure :-1 s=THROTTLE_DONE pgs=800 cs=0 l=1 rx=0x7fb2c8008790 tx=0x7fb2c00014d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:24:04.440 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:04.430+0000 7fb2ceffd700 1 -- 172.21.15.42:0/2700046386 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2c80087d0 con 0x7fb2d00a7740 2019-12-05T05:24:04.440 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:04.430+0000 7fb2ceffd700 1 -- 172.21.15.42:0/2700046386 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb2c8011760 con 0x7fb2d00a7740 2019-12-05T05:24:04.440 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:04.430+0000 7fb2d5d4d700 1 -- 172.21.15.42:0/2700046386 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb2d00a7740 msgr2=0x7fb2d008df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:24:04.440 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:04.430+0000 7fb2d5d4d700 1 --2- 172.21.15.42:0/2700046386 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb2d00a7740 0x7fb2d008df80 secure :-1 s=READY pgs=800 cs=0 l=1 rx=0x7fb2c8008790 tx=0x7fb2c00014d0).stop 2019-12-05T05:24:04.440 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:04.430+0000 7fb2d5d4d700 1 -- 172.21.15.42:0/2700046386 shutdown_connections 2019-12-05T05:24:04.440 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:04.430+0000 7fb2d5d4d700 1 --2- 172.21.15.42:0/2700046386 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb2d00a7740 0x7fb2d008df80 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:24:04.441 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:04.430+0000 7fb2d5d4d700 1 --2- 172.21.15.42:0/2700046386 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb2d007e720 0x7fb2d007eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:24:04.441 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:04.434+0000 7fb2d5d4d700 1 -- 172.21.15.42:0/2700046386 shutdown_connections 2019-12-05T05:24:04.441 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:04.434+0000 7fb2d5d4d700 1 -- 172.21.15.42:0/2700046386 wait complete. 2019-12-05T05:24:04.441 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:04.434+0000 7fb2d5d4d700 1 -- 172.21.15.42:0/2700046386 >> 172.21.15.42:0/2700046386 conn(0x7fb2d00aabe0 msgr2=0x7fb2d00a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:24:04.443 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:04.434+0000 7fb2d5d4d700 1 Processor -- start 2019-12-05T05:24:04.443 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:04.434+0000 7fb2d5d4d700 1 -- start start 2019-12-05T05:24:04.443 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:04.438+0000 7fb2d5d4d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb2d007b7c0 0x7fb2d0120900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:24:04.444 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:04.438+0000 7fb2d5d4d700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb2d012cef0 0x7fb2d012d320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:24:04.444 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:04.438+0000 7fb2d5d4d700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2d0120e50 con 0x7fb2d0130410 2019-12-05T05:24:04.444 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:04.438+0000 7fb2d5d4d700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2d007bff0 con 0x7fb2d012cef0 2019-12-05T05:24:04.444 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:04.438+0000 7fb2d5d4d700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2d007c290 con 0x7fb2d007b7c0 2019-12-05T05:24:04.444 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:04.438+0000 7fb2cffff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb2d007b7c0 0x7fb2d0120900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:24:04.444 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:04.438+0000 7fb2cffff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb2d007b7c0 0x7fb2d0120900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:32920/0 (socket says 172.21.15.42:32920) 2019-12-05T05:24:04.444 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:04.438+0000 7fb2d4aeb700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fb2d0130410 0x7fb2d0130840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:32962/0 (socket says 172.21.15.42:32962) 2019-12-05T05:24:04.445 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:04.438+0000 7fb2cffff700 1 -- 172.21.15.42:0/4035147186 learned_addr learned my addr 172.21.15.42:0/4035147186 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:24:04.445 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:04.438+0000 7fb2cf7fe700 1 --2- 172.21.15.42:0/4035147186 >> v2:172.21.15.40:3300/0 conn(0x7fb2d012cef0 0x7fb2d012d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:24:04.445 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:04.438+0000 7fb2cffff700 1 -- 172.21.15.42:0/4035147186 >> v2:172.21.15.40:3300/0 conn(0x7fb2d012cef0 msgr2=0x7fb2d012d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:24:04.445 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:04.438+0000 7fb2cffff700 1 --2- 172.21.15.42:0/4035147186 >> v2:172.21.15.40:3300/0 conn(0x7fb2d012cef0 0x7fb2d012d320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:24:04.445 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:04.438+0000 7fb2cffff700 1 -- 172.21.15.42:0/4035147186 >> v1:172.21.15.42:6789/0 conn(0x7fb2d0130410 legacy=0x7fb2d0130840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:24:04.445 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:04.438+0000 7fb2ccff9700 1 -- 172.21.15.42:0/4035147186 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2055453385 0 0) 0x7fb2d0120e50 con 0x7fb2d0130410 2019-12-05T05:24:04.445 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:04.438+0000 7fb2cffff700 1 -- 172.21.15.42:0/4035147186 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2d011ba10 con 0x7fb2d007b7c0 2019-12-05T05:24:04.446 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:04.438+0000 7fb2cffff700 1 --2- 172.21.15.42:0/4035147186 >> v2:172.21.15.42:3300/0 conn(0x7fb2d007b7c0 0x7fb2d0120900 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fb2c8008790 tx=0x7fb2c8008690).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:24:04.446 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:04.438+0000 7fb2cffff700 1 --2- 172.21.15.42:0/4035147186 >> v2:172.21.15.42:3300/0 conn(0x7fb2d007b7c0 0x7fb2d0120900 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fb2c8008790 tx=0x7fb2c8008690).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:24:04.446 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:04.438+0000 7fb2cffff700 1 --2- 172.21.15.42:0/4035147186 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb2d007b7c0 0x7fb2d0120900 secure :-1 s=READY pgs=802 cs=0 l=1 rx=0x7fb2c8008790 tx=0x7fb2c8008690).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:24:04.446 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:04.438+0000 7fb2cffff700 1 --2- 172.21.15.42:0/4035147186 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb2d007b7c0 0x7fb2d0120900 secure :-1 s=THROTTLE_DONE pgs=802 cs=0 l=1 rx=0x7fb2c8008790 tx=0x7fb2c8008690).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:24:04.446 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:04.438+0000 7fb2cffff700 1 --2- 172.21.15.42:0/4035147186 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb2d007b7c0 0x7fb2d0120900 secure :-1 s=THROTTLE_DONE pgs=802 cs=0 l=1 rx=0x7fb2c8008790 tx=0x7fb2c8008690).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:24:04.447 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:04.438+0000 7fb2ccff9700 1 -- 172.21.15.42:0/4035147186 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb2c800f280 con 0x7fb2d007b7c0 2019-12-05T05:24:04.447 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:04.438+0000 7fb2cffff700 1 --2- 172.21.15.42:0/4035147186 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb2d007b7c0 0x7fb2d0120900 secure :-1 s=THROTTLE_DONE pgs=802 cs=0 l=1 rx=0x7fb2c8008790 tx=0x7fb2c8008690).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:24:04.447 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:04.438+0000 7fb2ccff9700 1 -- 172.21.15.42:0/4035147186 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2c8008db0 con 0x7fb2d007b7c0 2019-12-05T05:24:04.447 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:04.438+0000 7fb2d5d4d700 1 -- 172.21.15.42:0/4035147186 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2d007c530 con 0x7fb2d007b7c0 2019-12-05T05:24:04.447 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:04.438+0000 7fb2ccff9700 1 -- 172.21.15.42:0/4035147186 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb2c8005540 con 0x7fb2d007b7c0 2019-12-05T05:24:04.448 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:04.438+0000 7fb2d5d4d700 1 -- 172.21.15.42:0/4035147186 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2d012d870 con 0x7fb2d007b7c0 2019-12-05T05:24:04.448 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:04.438+0000 7fb2cffff700 1 --2- 172.21.15.42:0/4035147186 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb2d007b7c0 0x7fb2d0120900 secure :-1 s=THROTTLE_DONE pgs=802 cs=0 l=1 rx=0x7fb2c8008790 tx=0x7fb2c8008690).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:24:04.449 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:04.442+0000 7fb2ccff9700 1 -- 172.21.15.42:0/4035147186 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fb2c8005850 con 0x7fb2d007b7c0 2019-12-05T05:24:04.449 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:04.442+0000 7fb2cffff700 1 --2- 172.21.15.42:0/4035147186 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb2d007b7c0 0x7fb2d0120900 secure :-1 s=THROTTLE_DONE pgs=802 cs=0 l=1 rx=0x7fb2c8008790 tx=0x7fb2c8008690).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:24:04.449 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:04.442+0000 7fb2ccff9700 1 --2- 172.21.15.42:0/4035147186 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb2b403ad10 0x7fb2b403d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:24:04.449 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:04.442+0000 7fb2ccff9700 1 -- 172.21.15.42:0/4035147186 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 7235+0+0 (secure 0 0 0) 0x7fb2c804b9f0 con 0x7fb2d007b7c0 2019-12-05T05:24:04.450 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:04.442+0000 7fb2cf7fe700 1 --2- 172.21.15.42:0/4035147186 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb2b403ad10 0x7fb2b403d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:24:04.450 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:04.442+0000 7fb2cf7fe700 1 --2- 172.21.15.42:0/4035147186 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb2b403ad10 0x7fb2b403d220 crc :-1 s=READY pgs=352 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:24:04.450 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:04.442+0000 7fb2d5d4d700 1 -- 172.21.15.42:0/4035147186 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2d005d110 con 0x7fb2d007b7c0 2019-12-05T05:24:04.453 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:04.450+0000 7fb2cffff700 1 --2- 172.21.15.42:0/4035147186 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb2d007b7c0 0x7fb2d0120900 secure :-1 s=THROTTLE_DONE pgs=802 cs=0 l=1 rx=0x7fb2c8008790 tx=0x7fb2c8008690).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:24:04.453 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:04.450+0000 7fb2ccff9700 1 -- 172.21.15.42:0/4035147186 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fb2c80306b0 con 0x7fb2d007b7c0 2019-12-05T05:24:04.614 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:04.610+0000 7fb2d5d4d700 1 -- 172.21.15.42:0/4035147186 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb2d005b8c0 con 0x7fb2d007b7c0 2019-12-05T05:24:04.615 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:04.610+0000 7fb2cffff700 1 --2- 172.21.15.42:0/4035147186 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb2d007b7c0 0x7fb2d0120900 secure :-1 s=THROTTLE_DONE pgs=802 cs=0 l=1 rx=0x7fb2c8008790 tx=0x7fb2c8008690).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:24:04.615 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:04.610+0000 7fb2ccff9700 1 -- 172.21.15.42:0/4035147186 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) v1 ==== 95+0+1774 (secure 0 0 0) 0x7fb2c8005b60 con 0x7fb2d007b7c0 2019-12-05T05:24:04.615 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 102 2019-12-05T05:24:04.616 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:24:04.616 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":102,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":102,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:23:58.091182+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5177":{"gid":5177,"name":"b","rank":0,"incarnation":99,"state":"up:active","state_seq":14,"addr":"172.21.15.40:6833/3208416000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3208416000},{"type":"v1","addr":"172.21.15.40:6833","nonce":3208416000}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:24:04.618 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:04.614+0000 7fb2d5d4d700 1 -- 172.21.15.42:0/4035147186 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb2b403ad10 msgr2=0x7fb2b403d220 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:24:04.618 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:04.614+0000 7fb2d5d4d700 1 --2- 172.21.15.42:0/4035147186 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb2b403ad10 0x7fb2b403d220 crc :-1 s=READY pgs=352 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:24:04.618 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:04.614+0000 7fb2d5d4d700 1 -- 172.21.15.42:0/4035147186 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb2d007b7c0 msgr2=0x7fb2d0120900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:24:04.618 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:04.614+0000 7fb2d5d4d700 1 --2- 172.21.15.42:0/4035147186 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb2d007b7c0 0x7fb2d0120900 secure :-1 s=READY pgs=802 cs=0 l=1 rx=0x7fb2c8008790 tx=0x7fb2c8008690).stop 2019-12-05T05:24:04.618 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:04.614+0000 7fb2d5d4d700 1 -- 172.21.15.42:0/4035147186 shutdown_connections 2019-12-05T05:24:04.619 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:04.614+0000 7fb2d5d4d700 1 --2- 172.21.15.42:0/4035147186 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb2b403ad10 0x7fb2b403d220 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:24:04.619 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:04.614+0000 7fb2d5d4d700 1 --2- 172.21.15.42:0/4035147186 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb2d007b7c0 0x7fb2d0120900 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:24:04.619 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:04.614+0000 7fb2d5d4d700 1 --2- 172.21.15.42:0/4035147186 >> v2:172.21.15.40:3300/0 conn(0x7fb2d012cef0 0x7fb2d012d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:24:04.619 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:04.614+0000 7fb2d5d4d700 1 -- 172.21.15.42:0/4035147186 shutdown_connections 2019-12-05T05:24:04.619 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:04.614+0000 7fb2d5d4d700 1 -- 172.21.15.42:0/4035147186 wait complete. 2019-12-05T05:24:04.619 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:04.614+0000 7fb2d5d4d700 1 -- 172.21.15.42:0/4035147186 >> 172.21.15.42:0/4035147186 conn(0x7fb2d00aabe0 msgr2=0x7fb2d0059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:24:04.628 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2019-12-05T05:24:06.629 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:24:06.739 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:06.726+0000 7fe267219700 1 Processor -- start 2019-12-05T05:24:06.739 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:06.726+0000 7fe267219700 1 -- start start 2019-12-05T05:24:06.739 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:06.730+0000 7fe267219700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe26009fad0 0x7fe26014ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:24:06.739 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:06.730+0000 7fe267219700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2600a0700 con 0x7fe2600a7730 2019-12-05T05:24:06.740 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:06.730+0000 7fe267219700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe26007ef10 con 0x7fe26008df70 2019-12-05T05:24:06.740 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:06.730+0000 7fe267219700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2600a0af0 con 0x7fe26009fad0 2019-12-05T05:24:06.740 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:06.730+0000 7fe264fb5700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fe26008df70 0x7fe26009f2d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:32982/0 (socket says 172.21.15.42:32982) 2019-12-05T05:24:06.740 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:06.730+0000 7fe264fb5700 1 -- 172.21.15.42:0/1964740957 learned_addr learned my addr 172.21.15.42:0/1964740957 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:24:06.740 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:06.730+0000 7fe265fb7700 1 --2- 172.21.15.42:0/1964740957 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe26009fad0 0x7fe26014ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:24:06.741 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:06.730+0000 7fe257fff700 1 -- 172.21.15.42:0/1964740957 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2825720612 0 0) 0x7fe26007ef10 con 0x7fe26008df70 2019-12-05T05:24:06.741 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:06.730+0000 7fe257fff700 1 -- 172.21.15.42:0/1964740957 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe24c0030b0 con 0x7fe26008df70 2019-12-05T05:24:06.741 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:06.730+0000 7fe257fff700 1 -- 172.21.15.42:0/1964740957 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1947218147 0 0) 0x7fe2600a0700 con 0x7fe2600a7730 2019-12-05T05:24:06.741 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:06.730+0000 7fe257fff700 1 -- 172.21.15.42:0/1964740957 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe26007ef10 con 0x7fe2600a7730 2019-12-05T05:24:06.741 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:06.730+0000 7fe265fb7700 1 -- 172.21.15.42:0/1964740957 >> v1:172.21.15.42:6789/0 conn(0x7fe26008df70 legacy=0x7fe26009f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:24:06.741 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:06.730+0000 7fe265fb7700 1 -- 172.21.15.42:0/1964740957 >> v1:172.21.15.40:6789/0 conn(0x7fe2600a7730 legacy=0x7fe26007e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:24:06.741 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:06.730+0000 7fe265fb7700 1 -- 172.21.15.42:0/1964740957 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2600a9b90 con 0x7fe26009fad0 2019-12-05T05:24:06.742 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:06.730+0000 7fe265fb7700 1 --2- 172.21.15.42:0/1964740957 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe26009fad0 0x7fe26014ffe0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fe2580032b0 tx=0x7fe25800ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:24:06.742 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:06.730+0000 7fe265fb7700 1 --2- 172.21.15.42:0/1964740957 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe26009fad0 0x7fe26014ffe0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fe2580032b0 tx=0x7fe25800ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:24:06.742 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:06.730+0000 7fe265fb7700 1 --2- 172.21.15.42:0/1964740957 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe26009fad0 0x7fe26014ffe0 secure :-1 s=READY pgs=163 cs=0 l=1 rx=0x7fe2580032b0 tx=0x7fe25800ff70).ready entity=mon.2 client_cookie=519e72c3ac889154 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:24:06.742 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:06.730+0000 7fe265fb7700 1 --2- 172.21.15.42:0/1964740957 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe26009fad0 0x7fe26014ffe0 secure :-1 s=THROTTLE_DONE pgs=163 cs=0 l=1 rx=0x7fe2580032b0 tx=0x7fe25800ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:24:06.742 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:06.730+0000 7fe265fb7700 1 --2- 172.21.15.42:0/1964740957 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe26009fad0 0x7fe26014ffe0 secure :-1 s=THROTTLE_DONE pgs=163 cs=0 l=1 rx=0x7fe2580032b0 tx=0x7fe25800ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:24:06.742 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:06.730+0000 7fe257fff700 1 -- 172.21.15.42:0/1964740957 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe2580109b0 con 0x7fe26009fad0 2019-12-05T05:24:06.742 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:06.730+0000 7fe265fb7700 1 --2- 172.21.15.42:0/1964740957 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe26009fad0 0x7fe26014ffe0 secure :-1 s=THROTTLE_DONE pgs=163 cs=0 l=1 rx=0x7fe2580032b0 tx=0x7fe25800ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:24:06.743 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:06.730+0000 7fe257fff700 1 -- 172.21.15.42:0/1964740957 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe258008b50 con 0x7fe26009fad0 2019-12-05T05:24:06.743 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:06.730+0000 7fe257fff700 1 -- 172.21.15.42:0/1964740957 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe2580110c0 con 0x7fe26009fad0 2019-12-05T05:24:06.743 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:06.730+0000 7fe267219700 1 -- 172.21.15.42:0/1964740957 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe26009fad0 msgr2=0x7fe26014ffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:24:06.743 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:06.730+0000 7fe267219700 1 --2- 172.21.15.42:0/1964740957 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe26009fad0 0x7fe26014ffe0 secure :-1 s=READY pgs=163 cs=0 l=1 rx=0x7fe2580032b0 tx=0x7fe25800ff70).stop 2019-12-05T05:24:06.743 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:06.730+0000 7fe267219700 1 -- 172.21.15.42:0/1964740957 shutdown_connections 2019-12-05T05:24:06.743 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:06.730+0000 7fe267219700 1 --2- 172.21.15.42:0/1964740957 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe26009fad0 0x7fe26014ffe0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:24:06.743 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:06.734+0000 7fe267219700 1 -- 172.21.15.42:0/1964740957 shutdown_connections 2019-12-05T05:24:06.744 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:06.734+0000 7fe267219700 1 -- 172.21.15.42:0/1964740957 wait complete. 2019-12-05T05:24:06.744 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:06.734+0000 7fe267219700 1 -- 172.21.15.42:0/1964740957 >> 172.21.15.42:0/1964740957 conn(0x7fe2600aabd0 msgr2=0x7fe2600a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:24:06.744 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:06.734+0000 7fe267219700 1 Processor -- start 2019-12-05T05:24:06.744 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:06.734+0000 7fe267219700 1 -- start start 2019-12-05T05:24:06.747 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:06.738+0000 7fe267219700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe2601303b0 0x7fe2601307e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:24:06.747 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:06.738+0000 7fe267219700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe26011b8c0 0x7fe26011bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:24:06.748 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:06.738+0000 7fe267219700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2601222f0 con 0x7fe260116350 2019-12-05T05:24:06.748 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:06.738+0000 7fe267219700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe260116780 con 0x7fe2601303b0 2019-12-05T05:24:06.749 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:06.738+0000 7fe267219700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe260122600 con 0x7fe26011b8c0 2019-12-05T05:24:06.749 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:06.738+0000 7fe264fb5700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe2601303b0 0x7fe2601307e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:24:06.749 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:06.738+0000 7fe264fb5700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe2601303b0 0x7fe2601307e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:32952/0 (socket says 172.21.15.42:32952) 2019-12-05T05:24:06.749 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:06.738+0000 7fe264fb5700 1 -- 172.21.15.42:0/666001243 learned_addr learned my addr 172.21.15.42:0/666001243 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:24:06.749 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:06.738+0000 7fe265fb7700 1 --2- 172.21.15.42:0/666001243 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe26011b8c0 0x7fe26011bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:24:06.749 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:06.738+0000 7fe264fb5700 1 -- 172.21.15.42:0/666001243 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe26011b8c0 msgr2=0x7fe26011bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:24:06.749 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:06.738+0000 7fe264fb5700 1 --2- 172.21.15.42:0/666001243 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe26011b8c0 0x7fe26011bcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:24:06.750 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:06.738+0000 7fe264fb5700 1 -- 172.21.15.42:0/666001243 >> v1:172.21.15.40:6789/0 conn(0x7fe260116350 legacy=0x7fe26008f410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:24:06.750 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:06.738+0000 7fe264fb5700 1 -- 172.21.15.42:0/666001243 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe260122b40 con 0x7fe2601303b0 2019-12-05T05:24:06.750 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:06.738+0000 7fe264fb5700 1 --2- 172.21.15.42:0/666001243 >> v2:172.21.15.42:3300/0 conn(0x7fe2601303b0 0x7fe2601307e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fe250000b80 tx=0x7fe250010a00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:24:06.750 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:06.738+0000 7fe264fb5700 1 --2- 172.21.15.42:0/666001243 >> v2:172.21.15.42:3300/0 conn(0x7fe2601303b0 0x7fe2601307e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fe250000b80 tx=0x7fe250010a00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:24:06.750 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:06.738+0000 7fe264fb5700 1 --2- 172.21.15.42:0/666001243 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe2601303b0 0x7fe2601307e0 secure :-1 s=READY pgs=804 cs=0 l=1 rx=0x7fe250000b80 tx=0x7fe250010a00).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:24:06.750 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:06.738+0000 7fe264fb5700 1 --2- 172.21.15.42:0/666001243 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe2601303b0 0x7fe2601307e0 secure :-1 s=THROTTLE_DONE pgs=804 cs=0 l=1 rx=0x7fe250000b80 tx=0x7fe250010a00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:24:06.750 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:06.738+0000 7fe255ffb700 1 -- 172.21.15.42:0/666001243 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe25000c9a0 con 0x7fe2601303b0 2019-12-05T05:24:06.751 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:06.738+0000 7fe264fb5700 1 --2- 172.21.15.42:0/666001243 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe2601303b0 0x7fe2601307e0 secure :-1 s=THROTTLE_DONE pgs=804 cs=0 l=1 rx=0x7fe250000b80 tx=0x7fe250010a00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:24:06.751 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:06.738+0000 7fe264fb5700 1 --2- 172.21.15.42:0/666001243 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe2601303b0 0x7fe2601307e0 secure :-1 s=THROTTLE_DONE pgs=804 cs=0 l=1 rx=0x7fe250000b80 tx=0x7fe250010a00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:24:06.751 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:06.738+0000 7fe255ffb700 1 -- 172.21.15.42:0/666001243 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe250010250 con 0x7fe2601303b0 2019-12-05T05:24:06.751 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:06.738+0000 7fe267219700 1 -- 172.21.15.42:0/666001243 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe260130d30 con 0x7fe2601303b0 2019-12-05T05:24:06.751 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:06.738+0000 7fe255ffb700 1 -- 172.21.15.42:0/666001243 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe25000d0c0 con 0x7fe2601303b0 2019-12-05T05:24:06.751 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:06.738+0000 7fe267219700 1 -- 172.21.15.42:0/666001243 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe26011c240 con 0x7fe2601303b0 2019-12-05T05:24:06.751 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:06.738+0000 7fe264fb5700 1 --2- 172.21.15.42:0/666001243 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe2601303b0 0x7fe2601307e0 secure :-1 s=THROTTLE_DONE pgs=804 cs=0 l=1 rx=0x7fe250000b80 tx=0x7fe250010a00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:24:06.752 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:06.738+0000 7fe255ffb700 1 -- 172.21.15.42:0/666001243 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fe250005460 con 0x7fe2601303b0 2019-12-05T05:24:06.752 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:06.738+0000 7fe264fb5700 1 --2- 172.21.15.42:0/666001243 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe2601303b0 0x7fe2601307e0 secure :-1 s=THROTTLE_DONE pgs=804 cs=0 l=1 rx=0x7fe250000b80 tx=0x7fe250010a00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:24:06.752 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:06.738+0000 7fe255ffb700 1 --2- 172.21.15.42:0/666001243 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe24003ad40 0x7fe24003d250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:24:06.752 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:06.738+0000 7fe255ffb700 1 -- 172.21.15.42:0/666001243 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 7235+0+0 (secure 0 0 0) 0x7fe25004ab50 con 0x7fe2601303b0 2019-12-05T05:24:06.752 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:06.742+0000 7fe2657b6700 1 --2- 172.21.15.42:0/666001243 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe24003ad40 0x7fe24003d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:24:06.752 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:06.742+0000 7fe2657b6700 1 --2- 172.21.15.42:0/666001243 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe24003ad40 0x7fe24003d250 crc :-1 s=READY pgs=354 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:24:06.756 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:06.742+0000 7fe267219700 1 -- 172.21.15.42:0/666001243 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe26005d100 con 0x7fe2601303b0 2019-12-05T05:24:06.756 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:06.746+0000 7fe264fb5700 1 --2- 172.21.15.42:0/666001243 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe2601303b0 0x7fe2601307e0 secure :-1 s=THROTTLE_DONE pgs=804 cs=0 l=1 rx=0x7fe250000b80 tx=0x7fe250010a00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:24:06.757 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:06.746+0000 7fe255ffb700 1 -- 172.21.15.42:0/666001243 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fe250031bb0 con 0x7fe2601303b0 2019-12-05T05:24:06.918 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:06.914+0000 7fe267219700 1 -- 172.21.15.42:0/666001243 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe26005b8b0 con 0x7fe2601303b0 2019-12-05T05:24:06.919 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:06.914+0000 7fe264fb5700 1 --2- 172.21.15.42:0/666001243 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe2601303b0 0x7fe2601307e0 secure :-1 s=THROTTLE_DONE pgs=804 cs=0 l=1 rx=0x7fe250000b80 tx=0x7fe250010a00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:24:06.919 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:06.914+0000 7fe255ffb700 1 -- 172.21.15.42:0/666001243 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 103 v103) v1 ==== 95+0+2111 (secure 0 0 0) 0x7fe250031e10 con 0x7fe2601303b0 2019-12-05T05:24:06.920 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 103 2019-12-05T05:24:06.921 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:24:06.921 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":103,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5240,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/1455108684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1455108684},{"type":"v1","addr":"172.21.15.42:6835","nonce":1455108684}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":103}],"filesystems":[{"mdsmap":{"epoch":102,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:23:58.091182+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5177":{"gid":5177,"name":"b","rank":0,"incarnation":99,"state":"up:active","state_seq":14,"addr":"172.21.15.40:6833/3208416000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3208416000},{"type":"v1","addr":"172.21.15.40:6833","nonce":3208416000}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:24:06.923 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:06.918+0000 7fe267219700 1 -- 172.21.15.42:0/666001243 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe24003ad40 msgr2=0x7fe24003d250 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:24:06.923 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:06.918+0000 7fe267219700 1 --2- 172.21.15.42:0/666001243 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe24003ad40 0x7fe24003d250 crc :-1 s=READY pgs=354 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:24:06.923 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:06.918+0000 7fe267219700 1 -- 172.21.15.42:0/666001243 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe2601303b0 msgr2=0x7fe2601307e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:24:06.924 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:06.918+0000 7fe267219700 1 --2- 172.21.15.42:0/666001243 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe2601303b0 0x7fe2601307e0 secure :-1 s=READY pgs=804 cs=0 l=1 rx=0x7fe250000b80 tx=0x7fe250010a00).stop 2019-12-05T05:24:06.924 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:06.918+0000 7fe267219700 1 -- 172.21.15.42:0/666001243 shutdown_connections 2019-12-05T05:24:06.924 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:06.918+0000 7fe267219700 1 --2- 172.21.15.42:0/666001243 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe24003ad40 0x7fe24003d250 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:24:06.924 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:06.918+0000 7fe267219700 1 --2- 172.21.15.42:0/666001243 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe26011b8c0 0x7fe26011bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:24:06.924 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:06.918+0000 7fe267219700 1 --2- 172.21.15.42:0/666001243 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe2601303b0 0x7fe2601307e0 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:24:06.925 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:06.918+0000 7fe267219700 1 -- 172.21.15.42:0/666001243 shutdown_connections 2019-12-05T05:24:06.925 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:06.918+0000 7fe267219700 1 -- 172.21.15.42:0/666001243 wait complete. 2019-12-05T05:24:06.925 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:24:06.918+0000 7fe267219700 1 -- 172.21.15.42:0/666001243 >> 172.21.15.42:0/666001243 conn(0x7fe2600a7730 msgr2=0x7fe26011daf0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:24:06.935 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a reported in up:standby state 2019-12-05T05:24:06.935 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 70 secs before thrashing 2019-12-05T05:24:08.228 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:24:08.232 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:24:38.278 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:24:38.282 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:25:08.336 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:25:08.340 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:25:16.937 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:25:17.052 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.029+0000 7ff162dd4700 1 Processor -- start 2019-12-05T05:25:17.053 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.029+0000 7ff162dd4700 1 -- start start 2019-12-05T05:25:17.053 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.033+0000 7ff162dd4700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff15c0a7740 0x7ff15c08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:25:17.053 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.033+0000 7ff162dd4700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff15c09fae0 0x7ff15c07eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:25:17.053 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.033+0000 7ff162dd4700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff15c0a0710 con 0x7ff15c07e720 2019-12-05T05:25:17.053 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.033+0000 7ff162dd4700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff15c0a0b00 con 0x7ff15c0a7740 2019-12-05T05:25:17.053 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.033+0000 7ff162dd4700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff15c08e5b0 con 0x7ff15c09fae0 2019-12-05T05:25:17.053 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.033+0000 7ff161b72700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff15c09fae0 0x7ff15c07eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:25:17.054 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.033+0000 7ff161371700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff15c0a7740 0x7ff15c08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:25:17.054 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.033+0000 7ff161371700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff15c0a7740 0x7ff15c08df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.42:49258/0 (socket says 172.21.15.42:49258) 2019-12-05T05:25:17.054 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.033+0000 7ff161371700 1 -- 172.21.15.42:0/3408806437 learned_addr learned my addr 172.21.15.42:0/3408806437 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:25:17.054 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.033+0000 7ff160b70700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7ff15c07e720 0x7ff15c09f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:42688/0 (socket says 172.21.15.42:42688) 2019-12-05T05:25:17.054 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.033+0000 7ff161371700 1 -- 172.21.15.42:0/3408806437 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff15c09fae0 msgr2=0x7ff15c07eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:25:17.054 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.033+0000 7ff161371700 1 --2- 172.21.15.42:0/3408806437 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff15c09fae0 0x7ff15c07eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:25:17.054 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.033+0000 7ff161371700 1 -- 172.21.15.42:0/3408806437 >> v1:172.21.15.40:6789/0 conn(0x7ff15c07e720 legacy=0x7ff15c09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:25:17.055 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.033+0000 7ff161371700 1 -- 172.21.15.42:0/3408806437 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff15c07f0a0 con 0x7ff15c0a7740 2019-12-05T05:25:17.055 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.033+0000 7ff161371700 1 --2- 172.21.15.42:0/3408806437 >> v2:172.21.15.40:3300/0 conn(0x7ff15c0a7740 0x7ff15c08df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7ff158008790 tx=0x7ff15800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:17.061 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.033+0000 7ff161371700 1 --2- 172.21.15.42:0/3408806437 >> v2:172.21.15.40:3300/0 conn(0x7ff15c0a7740 0x7ff15c08df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7ff158008790 tx=0x7ff15800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:17.061 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.033+0000 7ff161371700 1 --2- 172.21.15.42:0/3408806437 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff15c0a7740 0x7ff15c08df80 secure :-1 s=READY pgs=638 cs=0 l=1 rx=0x7ff158008790 tx=0x7ff15800fd30).ready entity=mon.0 client_cookie=6c7b22c6de659a0b server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:25:17.061 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.033+0000 7ff161371700 1 --2- 172.21.15.42:0/3408806437 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff15c0a7740 0x7ff15c08df80 secure :-1 s=THROTTLE_DONE pgs=638 cs=0 l=1 rx=0x7ff158008790 tx=0x7ff15800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:17.061 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.033+0000 7ff161371700 1 --2- 172.21.15.42:0/3408806437 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff15c0a7740 0x7ff15c08df80 secure :-1 s=THROTTLE_DONE pgs=638 cs=0 l=1 rx=0x7ff158008790 tx=0x7ff15800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:17.061 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.033+0000 7ff161371700 1 --2- 172.21.15.42:0/3408806437 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff15c0a7740 0x7ff15c08df80 secure :-1 s=THROTTLE_DONE pgs=638 cs=0 l=1 rx=0x7ff158008790 tx=0x7ff15800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:17.062 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.033+0000 7ff14ffff700 1 -- 172.21.15.42:0/3408806437 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff158008c90 con 0x7ff15c0a7740 2019-12-05T05:25:17.062 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.033+0000 7ff14ffff700 1 -- 172.21.15.42:0/3408806437 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1580088b0 con 0x7ff15c0a7740 2019-12-05T05:25:17.062 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.033+0000 7ff14ffff700 1 -- 172.21.15.42:0/3408806437 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff158011b10 con 0x7ff15c0a7740 2019-12-05T05:25:17.062 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.033+0000 7ff162dd4700 1 -- 172.21.15.42:0/3408806437 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff15c0a7740 msgr2=0x7ff15c08df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:25:17.062 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.033+0000 7ff162dd4700 1 --2- 172.21.15.42:0/3408806437 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff15c0a7740 0x7ff15c08df80 secure :-1 s=READY pgs=638 cs=0 l=1 rx=0x7ff158008790 tx=0x7ff15800fd30).stop 2019-12-05T05:25:17.063 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.037+0000 7ff162dd4700 1 -- 172.21.15.42:0/3408806437 shutdown_connections 2019-12-05T05:25:17.063 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.037+0000 7ff162dd4700 1 --2- 172.21.15.42:0/3408806437 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff15c09fae0 0x7ff15c07eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:25:17.063 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.037+0000 7ff162dd4700 1 --2- 172.21.15.42:0/3408806437 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff15c0a7740 0x7ff15c08df80 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:25:17.063 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.037+0000 7ff162dd4700 1 -- 172.21.15.42:0/3408806437 shutdown_connections 2019-12-05T05:25:17.063 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.037+0000 7ff162dd4700 1 -- 172.21.15.42:0/3408806437 wait complete. 2019-12-05T05:25:17.064 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.037+0000 7ff162dd4700 1 -- 172.21.15.42:0/3408806437 >> 172.21.15.42:0/3408806437 conn(0x7ff15c0aabe0 msgr2=0x7ff15c0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:25:17.064 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.037+0000 7ff162dd4700 1 Processor -- start 2019-12-05T05:25:17.064 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.037+0000 7ff162dd4700 1 -- start start 2019-12-05T05:25:17.064 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.041+0000 7ff162dd4700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff15c1135d0 0x7ff15c1155e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:25:17.065 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.041+0000 7ff162dd4700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff15c11a520 0x7ff15c127550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:25:17.066 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.041+0000 7ff162dd4700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff15c126df0 con 0x7ff15c112a10 2019-12-05T05:25:17.066 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.041+0000 7ff162dd4700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff15c115e80 con 0x7ff15c1135d0 2019-12-05T05:25:17.066 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.041+0000 7ff162dd4700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff15c112770 con 0x7ff15c11a520 2019-12-05T05:25:17.066 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.041+0000 7ff161371700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff15c1135d0 0x7ff15c1155e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:25:17.066 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.041+0000 7ff160b70700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff15c11a520 0x7ff15c127550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:25:17.066 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.041+0000 7ff161371700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff15c1135d0 0x7ff15c1155e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:33216/0 (socket says 172.21.15.42:33216) 2019-12-05T05:25:17.067 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.041+0000 7ff161371700 1 -- 172.21.15.42:0/1708985535 learned_addr learned my addr 172.21.15.42:0/1708985535 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:25:17.067 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.041+0000 7ff14dffb700 1 -- 172.21.15.42:0/1708985535 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 863985254 0 0) 0x7ff15c126df0 con 0x7ff15c112a10 2019-12-05T05:25:17.067 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.041+0000 7ff14dffb700 1 -- 172.21.15.42:0/1708985535 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff13c0030b0 con 0x7ff15c112a10 2019-12-05T05:25:17.067 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.041+0000 7ff161371700 1 -- 172.21.15.42:0/1708985535 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff15c11a520 msgr2=0x7ff15c127550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:25:17.067 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.041+0000 7ff161371700 1 --2- 172.21.15.42:0/1708985535 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff15c11a520 0x7ff15c127550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:25:17.067 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.041+0000 7ff161371700 1 -- 172.21.15.42:0/1708985535 >> v1:172.21.15.40:6789/0 conn(0x7ff15c112a10 legacy=0x7ff15c11ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:25:17.068 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.041+0000 7ff161371700 1 -- 172.21.15.42:0/1708985535 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff15c115b30 con 0x7ff15c1135d0 2019-12-05T05:25:17.068 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.041+0000 7ff160b70700 1 --2- 172.21.15.42:0/1708985535 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff15c11a520 0x7ff15c127550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:25:17.068 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.041+0000 7ff161371700 1 --2- 172.21.15.42:0/1708985535 >> v2:172.21.15.42:3300/0 conn(0x7ff15c1135d0 0x7ff15c1155e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7ff15c116040 tx=0x7ff158008b90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:17.069 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.041+0000 7ff161371700 1 --2- 172.21.15.42:0/1708985535 >> v2:172.21.15.42:3300/0 conn(0x7ff15c1135d0 0x7ff15c1155e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7ff15c116040 tx=0x7ff158008b90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:17.069 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.041+0000 7ff161371700 1 --2- 172.21.15.42:0/1708985535 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff15c1135d0 0x7ff15c1155e0 secure :-1 s=READY pgs=805 cs=0 l=1 rx=0x7ff15c116040 tx=0x7ff158008b90).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:25:17.069 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.041+0000 7ff161371700 1 --2- 172.21.15.42:0/1708985535 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff15c1135d0 0x7ff15c1155e0 secure :-1 s=THROTTLE_DONE pgs=805 cs=0 l=1 rx=0x7ff15c116040 tx=0x7ff158008b90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:17.070 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.041+0000 7ff14dffb700 1 -- 172.21.15.42:0/1708985535 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff15800f490 con 0x7ff15c1135d0 2019-12-05T05:25:17.070 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.041+0000 7ff161371700 1 --2- 172.21.15.42:0/1708985535 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff15c1135d0 0x7ff15c1155e0 secure :-1 s=THROTTLE_DONE pgs=805 cs=0 l=1 rx=0x7ff15c116040 tx=0x7ff158008b90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:17.070 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.041+0000 7ff14dffb700 1 -- 172.21.15.42:0/1708985535 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff158011730 con 0x7ff15c1135d0 2019-12-05T05:25:17.070 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.041+0000 7ff162dd4700 1 -- 172.21.15.42:0/1708985535 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff15c127aa0 con 0x7ff15c1135d0 2019-12-05T05:25:17.071 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.041+0000 7ff161371700 1 --2- 172.21.15.42:0/1708985535 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff15c1135d0 0x7ff15c1155e0 secure :-1 s=THROTTLE_DONE pgs=805 cs=0 l=1 rx=0x7ff15c116040 tx=0x7ff158008b90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:17.071 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.041+0000 7ff14dffb700 1 -- 172.21.15.42:0/1708985535 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff158005730 con 0x7ff15c1135d0 2019-12-05T05:25:17.071 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.041+0000 7ff162dd4700 1 -- 172.21.15.42:0/1708985535 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff15c112fe0 con 0x7ff15c1135d0 2019-12-05T05:25:17.071 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.045+0000 7ff161371700 1 --2- 172.21.15.42:0/1708985535 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff15c1135d0 0x7ff15c1155e0 secure :-1 s=THROTTLE_DONE pgs=805 cs=0 l=1 rx=0x7ff15c116040 tx=0x7ff158008b90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:17.072 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.045+0000 7ff14dffb700 1 -- 172.21.15.42:0/1708985535 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7ff1580059d0 con 0x7ff15c1135d0 2019-12-05T05:25:17.072 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.045+0000 7ff161371700 1 --2- 172.21.15.42:0/1708985535 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff15c1135d0 0x7ff15c1155e0 secure :-1 s=THROTTLE_DONE pgs=805 cs=0 l=1 rx=0x7ff15c116040 tx=0x7ff158008b90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:17.072 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.045+0000 7ff14dffb700 1 --2- 172.21.15.42:0/1708985535 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff13c03b2a0 0x7ff13c03d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:25:17.072 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.045+0000 7ff14dffb700 1 -- 172.21.15.42:0/1708985535 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 7235+0+0 (secure 0 0 0) 0x7ff15804b7f0 con 0x7ff15c1135d0 2019-12-05T05:25:17.072 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.045+0000 7ff160b70700 1 --2- 172.21.15.42:0/1708985535 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff13c03b2a0 0x7ff13c03d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:25:17.072 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.045+0000 7ff160b70700 1 --2- 172.21.15.42:0/1708985535 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff13c03b2a0 0x7ff13c03d7b0 crc :-1 s=READY pgs=355 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:25:17.072 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.045+0000 7ff162dd4700 1 -- 172.21.15.42:0/1708985535 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff15c05d110 con 0x7ff15c1135d0 2019-12-05T05:25:17.073 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.053+0000 7ff161371700 1 --2- 172.21.15.42:0/1708985535 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff15c1135d0 0x7ff15c1155e0 secure :-1 s=THROTTLE_DONE pgs=805 cs=0 l=1 rx=0x7ff15c116040 tx=0x7ff158008b90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:17.073 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.053+0000 7ff14dffb700 1 -- 172.21.15.42:0/1708985535 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7ff158032a10 con 0x7ff15c1135d0 2019-12-05T05:25:17.230 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.217+0000 7ff162dd4700 1 -- 172.21.15.42:0/1708985535 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff15c05b8c0 con 0x7ff15c1135d0 2019-12-05T05:25:17.231 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.221+0000 7ff161371700 1 --2- 172.21.15.42:0/1708985535 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff15c1135d0 0x7ff15c1155e0 secure :-1 s=THROTTLE_DONE pgs=805 cs=0 l=1 rx=0x7ff15c116040 tx=0x7ff158008b90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:17.231 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.221+0000 7ff14dffb700 1 -- 172.21.15.42:0/1708985535 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 103 v103) v1 ==== 95+0+2111 (secure 0 0 0) 0x7ff158005ce0 con 0x7ff15c1135d0 2019-12-05T05:25:17.231 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 103 2019-12-05T05:25:17.231 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.221+0000 7ff162dd4700 1 -- 172.21.15.42:0/1708985535 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff13c03b2a0 msgr2=0x7ff13c03d7b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:25:17.231 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.221+0000 7ff162dd4700 1 --2- 172.21.15.42:0/1708985535 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff13c03b2a0 0x7ff13c03d7b0 crc :-1 s=READY pgs=355 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:25:17.231 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.221+0000 7ff162dd4700 1 -- 172.21.15.42:0/1708985535 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff15c1135d0 msgr2=0x7ff15c1155e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:25:17.232 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.221+0000 7ff162dd4700 1 --2- 172.21.15.42:0/1708985535 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff15c1135d0 0x7ff15c1155e0 secure :-1 s=READY pgs=805 cs=0 l=1 rx=0x7ff15c116040 tx=0x7ff158008b90).stop 2019-12-05T05:25:17.232 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.221+0000 7ff162dd4700 1 -- 172.21.15.42:0/1708985535 shutdown_connections 2019-12-05T05:25:17.232 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.221+0000 7ff162dd4700 1 --2- 172.21.15.42:0/1708985535 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff13c03b2a0 0x7ff13c03d7b0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:25:17.232 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.221+0000 7ff162dd4700 1 --2- 172.21.15.42:0/1708985535 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff15c1135d0 0x7ff15c1155e0 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:25:17.249 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.221+0000 7ff162dd4700 1 --2- 172.21.15.42:0/1708985535 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff15c11a520 0x7ff15c127550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:25:17.250 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.221+0000 7ff162dd4700 1 -- 172.21.15.42:0/1708985535 shutdown_connections 2019-12-05T05:25:17.250 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.221+0000 7ff162dd4700 1 -- 172.21.15.42:0/1708985535 wait complete. 2019-12-05T05:25:17.250 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.225+0000 7ff162dd4700 1 -- 172.21.15.42:0/1708985535 >> 172.21.15.42:0/1708985535 conn(0x7ff15c08fda0 msgr2=0x7ff15c09f4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:25:17.250 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:25:17.250 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":103,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5240,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/1455108684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1455108684},{"type":"v1","addr":"172.21.15.42:6835","nonce":1455108684}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":103}],"filesystems":[{"mdsmap":{"epoch":102,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:23:58.091182+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5177":{"gid":5177,"name":"b","rank":0,"incarnation":99,"state":"up:active","state_seq":14,"addr":"172.21.15.40:6833/3208416000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3208416000},{"type":"v1","addr":"172.21.15.40:6833","nonce":3208416000}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:25:17.252 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.b (rank=0) 2019-12-05T05:25:17.252 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2019-12-05T05:25:17.253 INFO:teuthology.orchestra.run:waiting for 300 2019-12-05T05:25:17.270 INFO:tasks.ceph.mds.b:Stopped 2019-12-05T05:25:17.270 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:25:17.369 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.361+0000 7f1f8b15d700 1 Processor -- start 2019-12-05T05:25:17.370 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.361+0000 7f1f8b15d700 1 -- start start 2019-12-05T05:25:17.370 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.361+0000 7f1f8b15d700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1f8408df70 0x7f1f8409f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:25:17.371 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.361+0000 7f1f8b15d700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f8409fc50 con 0x7f1f840a7730 2019-12-05T05:25:17.371 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.361+0000 7f1f8b15d700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f840a0700 con 0x7f1f8409f820 2019-12-05T05:25:17.371 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.361+0000 7f1f8b15d700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f840a0af0 con 0x7f1f8408df70 2019-12-05T05:25:17.371 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.361+0000 7f1f89efb700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f1f8409f820 0x7f1f8414efd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:33268/0 (socket says 172.21.15.42:33268) 2019-12-05T05:25:17.371 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.361+0000 7f1f89efb700 1 -- 172.21.15.42:0/2528192509 learned_addr learned my addr 172.21.15.42:0/2528192509 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:25:17.371 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.361+0000 7f1f88ef9700 1 --2- 172.21.15.42:0/2528192509 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1f8408df70 0x7f1f8409f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:25:17.372 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.361+0000 7f1f7bfff700 1 -- 172.21.15.42:0/2528192509 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4131295360 0 0) 0x7f1f840a0700 con 0x7f1f8409f820 2019-12-05T05:25:17.372 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.361+0000 7f1f7bfff700 1 -- 172.21.15.42:0/2528192509 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f6c0030b0 con 0x7f1f8409f820 2019-12-05T05:25:17.372 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.361+0000 7f1f7bfff700 1 -- 172.21.15.42:0/2528192509 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1798990018 0 0) 0x7f1f8409fc50 con 0x7f1f840a7730 2019-12-05T05:25:17.372 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.361+0000 7f1f7bfff700 1 -- 172.21.15.42:0/2528192509 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f840a0700 con 0x7f1f840a7730 2019-12-05T05:25:17.372 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.361+0000 7f1f88ef9700 1 -- 172.21.15.42:0/2528192509 >> v1:172.21.15.42:6789/0 conn(0x7f1f8409f820 legacy=0x7f1f8414efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:25:17.373 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.361+0000 7f1f7bfff700 1 -- 172.21.15.42:0/2528192509 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2449518621 0 0) 0x7f1f6c0030b0 con 0x7f1f8409f820 2019-12-05T05:25:17.373 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.361+0000 7f1f88ef9700 1 -- 172.21.15.42:0/2528192509 >> v1:172.21.15.40:6789/0 conn(0x7f1f840a7730 legacy=0x7f1f8407e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:25:17.373 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.361+0000 7f1f88ef9700 1 -- 172.21.15.42:0/2528192509 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f8407ef10 con 0x7f1f8408df70 2019-12-05T05:25:17.373 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.361+0000 7f1f88ef9700 1 --2- 172.21.15.42:0/2528192509 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1f8408df70 0x7f1f8409f2d0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f1f80008790 tx=0x7f1f7c001950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:17.373 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.361+0000 7f1f88ef9700 1 --2- 172.21.15.42:0/2528192509 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1f8408df70 0x7f1f8409f2d0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f1f80008790 tx=0x7f1f7c001950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:17.374 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.361+0000 7f1f88ef9700 1 --2- 172.21.15.42:0/2528192509 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1f8408df70 0x7f1f8409f2d0 secure :-1 s=READY pgs=167 cs=0 l=1 rx=0x7f1f80008790 tx=0x7f1f7c001950).ready entity=mon.2 client_cookie=af8f600a1d99a229 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:25:17.374 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.361+0000 7f1f88ef9700 1 --2- 172.21.15.42:0/2528192509 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1f8408df70 0x7f1f8409f2d0 secure :-1 s=THROTTLE_DONE pgs=167 cs=0 l=1 rx=0x7f1f80008790 tx=0x7f1f7c001950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:17.374 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.361+0000 7f1f88ef9700 1 --2- 172.21.15.42:0/2528192509 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1f8408df70 0x7f1f8409f2d0 secure :-1 s=THROTTLE_DONE pgs=167 cs=0 l=1 rx=0x7f1f80008790 tx=0x7f1f7c001950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:17.374 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.361+0000 7f1f7bfff700 1 -- 172.21.15.42:0/2528192509 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1f7c001750 con 0x7f1f8408df70 2019-12-05T05:25:17.374 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.361+0000 7f1f88ef9700 1 --2- 172.21.15.42:0/2528192509 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1f8408df70 0x7f1f8409f2d0 secure :-1 s=THROTTLE_DONE pgs=167 cs=0 l=1 rx=0x7f1f80008790 tx=0x7f1f7c001950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:17.374 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.361+0000 7f1f7bfff700 1 -- 172.21.15.42:0/2528192509 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f7c001560 con 0x7f1f8408df70 2019-12-05T05:25:17.375 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.361+0000 7f1f7bfff700 1 -- 172.21.15.42:0/2528192509 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1f80011360 con 0x7f1f8408df70 2019-12-05T05:25:17.375 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.361+0000 7f1f8b15d700 1 -- 172.21.15.42:0/2528192509 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1f8408df70 msgr2=0x7f1f8409f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:25:17.375 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.361+0000 7f1f8b15d700 1 --2- 172.21.15.42:0/2528192509 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1f8408df70 0x7f1f8409f2d0 secure :-1 s=READY pgs=167 cs=0 l=1 rx=0x7f1f80008790 tx=0x7f1f7c001950).stop 2019-12-05T05:25:17.375 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.365+0000 7f1f8b15d700 1 -- 172.21.15.42:0/2528192509 shutdown_connections 2019-12-05T05:25:17.375 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.365+0000 7f1f8b15d700 1 --2- 172.21.15.42:0/2528192509 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1f8408df70 0x7f1f8409f2d0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:25:17.375 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.365+0000 7f1f8b15d700 1 -- 172.21.15.42:0/2528192509 shutdown_connections 2019-12-05T05:25:17.376 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.365+0000 7f1f8b15d700 1 -- 172.21.15.42:0/2528192509 wait complete. 2019-12-05T05:25:17.376 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.365+0000 7f1f8b15d700 1 -- 172.21.15.42:0/2528192509 >> 172.21.15.42:0/2528192509 conn(0x7f1f840aabd0 msgr2=0x7f1f840a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:25:17.379 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.365+0000 7f1f8b15d700 1 Processor -- start 2019-12-05T05:25:17.379 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.365+0000 7f1f8b15d700 1 -- start start 2019-12-05T05:25:17.379 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.369+0000 7f1f8b15d700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1f841433f0 0x7f1f84066fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:25:17.380 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.369+0000 7f1f8b15d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f1f840a5e90 0x7f1f84139b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:25:17.380 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.369+0000 7f1f8b15d700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f84067520 con 0x7f1f840a5a60 2019-12-05T05:25:17.380 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.369+0000 7f1f8b15d700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f841316f0 con 0x7f1f840a5e90 2019-12-05T05:25:17.380 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.369+0000 7f1f8b15d700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f841438f0 con 0x7f1f841433f0 2019-12-05T05:25:17.382 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.369+0000 7f1f89efb700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f1f840a5e90 0x7f1f84139b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:25:17.382 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.369+0000 7f1f89efb700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f1f840a5e90 0x7f1f84139b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:33236/0 (socket says 172.21.15.42:33236) 2019-12-05T05:25:17.382 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.369+0000 7f1f89efb700 1 -- 172.21.15.42:0/48008939 learned_addr learned my addr 172.21.15.42:0/48008939 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:25:17.382 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.369+0000 7f1f896fa700 1 --2- 172.21.15.42:0/48008939 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1f841433f0 0x7f1f84066fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:25:17.382 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.369+0000 7f1f79ffb700 1 -- 172.21.15.42:0/48008939 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 938103435 0 0) 0x7f1f84067520 con 0x7f1f840a5a60 2019-12-05T05:25:17.383 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.369+0000 7f1f89efb700 1 -- 172.21.15.42:0/48008939 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1f841433f0 msgr2=0x7f1f84066fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:25:17.383 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.369+0000 7f1f89efb700 1 --2- 172.21.15.42:0/48008939 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1f841433f0 0x7f1f84066fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:25:17.383 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.369+0000 7f1f89efb700 1 -- 172.21.15.42:0/48008939 >> v1:172.21.15.40:6789/0 conn(0x7f1f840a5a60 legacy=0x7f1f840a6850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:25:17.383 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.369+0000 7f1f89efb700 1 -- 172.21.15.42:0/48008939 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f840a7050 con 0x7f1f840a5e90 2019-12-05T05:25:17.383 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.369+0000 7f1f89efb700 1 --2- 172.21.15.42:0/48008939 >> v2:172.21.15.42:3300/0 conn(0x7f1f840a5e90 0x7f1f84139b50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f1f84067750 tx=0x7f1f7c010850).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:17.383 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.369+0000 7f1f89efb700 1 --2- 172.21.15.42:0/48008939 >> v2:172.21.15.42:3300/0 conn(0x7f1f840a5e90 0x7f1f84139b50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f1f84067750 tx=0x7f1f7c010850).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:17.383 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.369+0000 7f1f89efb700 1 --2- 172.21.15.42:0/48008939 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1f840a5e90 0x7f1f84139b50 secure :-1 s=READY pgs=807 cs=0 l=1 rx=0x7f1f84067750 tx=0x7f1f7c010850).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:25:17.384 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.369+0000 7f1f89efb700 1 --2- 172.21.15.42:0/48008939 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1f840a5e90 0x7f1f84139b50 secure :-1 s=THROTTLE_DONE pgs=807 cs=0 l=1 rx=0x7f1f84067750 tx=0x7f1f7c010850).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:17.384 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.369+0000 7f1f89efb700 1 --2- 172.21.15.42:0/48008939 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1f840a5e90 0x7f1f84139b50 secure :-1 s=THROTTLE_DONE pgs=807 cs=0 l=1 rx=0x7f1f84067750 tx=0x7f1f7c010850).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:17.384 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.369+0000 7f1f79ffb700 1 -- 172.21.15.42:0/48008939 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1f7c011540 con 0x7f1f840a5e90 2019-12-05T05:25:17.384 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.369+0000 7f1f89efb700 1 --2- 172.21.15.42:0/48008939 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1f840a5e90 0x7f1f84139b50 secure :-1 s=THROTTLE_DONE pgs=807 cs=0 l=1 rx=0x7f1f84067750 tx=0x7f1f7c010850).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:17.384 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.369+0000 7f1f79ffb700 1 -- 172.21.15.42:0/48008939 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f7c011160 con 0x7f1f840a5e90 2019-12-05T05:25:17.384 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.369+0000 7f1f79ffb700 1 -- 172.21.15.42:0/48008939 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1f7c011b20 con 0x7f1f840a5e90 2019-12-05T05:25:17.384 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.369+0000 7f1f8b15d700 1 -- 172.21.15.42:0/48008939 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f8413a0a0 con 0x7f1f840a5e90 2019-12-05T05:25:17.385 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.369+0000 7f1f8b15d700 1 -- 172.21.15.42:0/48008939 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f8413a650 con 0x7f1f840a5e90 2019-12-05T05:25:17.385 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.369+0000 7f1f89efb700 1 --2- 172.21.15.42:0/48008939 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1f840a5e90 0x7f1f84139b50 secure :-1 s=THROTTLE_DONE pgs=807 cs=0 l=1 rx=0x7f1f84067750 tx=0x7f1f7c010850).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:17.385 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.373+0000 7f1f79ffb700 1 -- 172.21.15.42:0/48008939 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f1f7c00b330 con 0x7f1f840a5e90 2019-12-05T05:25:17.385 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.373+0000 7f1f89efb700 1 --2- 172.21.15.42:0/48008939 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1f840a5e90 0x7f1f84139b50 secure :-1 s=THROTTLE_DONE pgs=807 cs=0 l=1 rx=0x7f1f84067750 tx=0x7f1f7c010850).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:17.385 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.373+0000 7f1f79ffb700 1 --2- 172.21.15.42:0/48008939 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f1f6403ac20 0x7f1f6403d130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:25:17.385 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.373+0000 7f1f79ffb700 1 -- 172.21.15.42:0/48008939 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 7235+0+0 (secure 0 0 0) 0x7f1f7c04b180 con 0x7f1f840a5e90 2019-12-05T05:25:17.386 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.373+0000 7f1f896fa700 1 --2- 172.21.15.42:0/48008939 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f1f6403ac20 0x7f1f6403d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:25:17.386 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.373+0000 7f1f896fa700 1 --2- 172.21.15.42:0/48008939 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f1f6403ac20 0x7f1f6403d130 crc :-1 s=READY pgs=356 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:25:17.386 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.377+0000 7f1f8b15d700 1 -- 172.21.15.42:0/48008939 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f8405d100 con 0x7f1f840a5e90 2019-12-05T05:25:17.388 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.385+0000 7f1f89efb700 1 --2- 172.21.15.42:0/48008939 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1f840a5e90 0x7f1f84139b50 secure :-1 s=THROTTLE_DONE pgs=807 cs=0 l=1 rx=0x7f1f84067750 tx=0x7f1f7c010850).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:17.388 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.385+0000 7f1f79ffb700 1 -- 172.21.15.42:0/48008939 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f1f7c0321c0 con 0x7f1f840a5e90 2019-12-05T05:25:17.558 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.553+0000 7f1f8b15d700 1 -- 172.21.15.42:0/48008939 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1f8405b8b0 con 0x7f1f840a5e90 2019-12-05T05:25:17.559 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.553+0000 7f1f89efb700 1 --2- 172.21.15.42:0/48008939 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1f840a5e90 0x7f1f84139b50 secure :-1 s=THROTTLE_DONE pgs=807 cs=0 l=1 rx=0x7f1f84067750 tx=0x7f1f7c010850).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:17.561 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.553+0000 7f1f79ffb700 1 -- 172.21.15.42:0/48008939 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 103 v103) v1 ==== 95+0+2111 (secure 0 0 0) 0x7f1f7c032420 con 0x7f1f840a5e90 2019-12-05T05:25:17.561 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 103 2019-12-05T05:25:17.562 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:25:17.562 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":103,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5240,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/1455108684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1455108684},{"type":"v1","addr":"172.21.15.42:6835","nonce":1455108684}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":103}],"filesystems":[{"mdsmap":{"epoch":102,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:23:58.091182+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5177":{"gid":5177,"name":"b","rank":0,"incarnation":99,"state":"up:active","state_seq":14,"addr":"172.21.15.40:6833/3208416000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3208416000},{"type":"v1","addr":"172.21.15.40:6833","nonce":3208416000}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:25:17.563 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.557+0000 7f1f8b15d700 1 -- 172.21.15.42:0/48008939 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f1f6403ac20 msgr2=0x7f1f6403d130 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:25:17.563 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.557+0000 7f1f8b15d700 1 --2- 172.21.15.42:0/48008939 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f1f6403ac20 0x7f1f6403d130 crc :-1 s=READY pgs=356 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:25:17.563 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.557+0000 7f1f8b15d700 1 -- 172.21.15.42:0/48008939 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1f840a5e90 msgr2=0x7f1f84139b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:25:17.564 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.557+0000 7f1f8b15d700 1 --2- 172.21.15.42:0/48008939 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1f840a5e90 0x7f1f84139b50 secure :-1 s=READY pgs=807 cs=0 l=1 rx=0x7f1f84067750 tx=0x7f1f7c010850).stop 2019-12-05T05:25:17.564 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.557+0000 7f1f8b15d700 1 -- 172.21.15.42:0/48008939 shutdown_connections 2019-12-05T05:25:17.564 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.557+0000 7f1f8b15d700 1 --2- 172.21.15.42:0/48008939 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f1f6403ac20 0x7f1f6403d130 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:25:17.564 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.557+0000 7f1f8b15d700 1 --2- 172.21.15.42:0/48008939 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1f840a5e90 0x7f1f84139b50 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:25:17.564 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.557+0000 7f1f8b15d700 1 --2- 172.21.15.42:0/48008939 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1f841433f0 0x7f1f84066fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:25:17.564 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.557+0000 7f1f8b15d700 1 -- 172.21.15.42:0/48008939 shutdown_connections 2019-12-05T05:25:17.564 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.557+0000 7f1f8b15d700 1 -- 172.21.15.42:0/48008939 wait complete. 2019-12-05T05:25:17.565 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:17.557+0000 7f1f8b15d700 1 -- 172.21.15.42:0/48008939 >> 172.21.15.42:0/48008939 conn(0x7f1f840a8290 msgr2=0x7f1f8411ced0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:25:17.572 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T05:25:19.572 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:25:19.676 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:19.665+0000 7f0b12532700 1 Processor -- start 2019-12-05T05:25:19.677 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:19.665+0000 7f0b12532700 1 -- start start 2019-12-05T05:25:19.677 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:19.669+0000 7f0b12532700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0b0c09fae0 0x7f0b0c14fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:25:19.677 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:19.669+0000 7f0b12532700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b0c0a0710 con 0x7f0b0c0a7740 2019-12-05T05:25:19.677 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:19.669+0000 7f0b12532700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b0c07ef20 con 0x7f0b0c08df80 2019-12-05T05:25:19.677 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:19.669+0000 7f0b12532700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b0c0a0b00 con 0x7f0b0c09fae0 2019-12-05T05:25:19.677 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:19.669+0000 7f0b0bfff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f0b0c08df80 0x7f0b0c09f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:33286/0 (socket says 172.21.15.42:33286) 2019-12-05T05:25:19.678 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:19.669+0000 7f0b0bfff700 1 -- 172.21.15.42:0/3019195402 learned_addr learned my addr 172.21.15.42:0/3019195402 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:25:19.678 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:19.669+0000 7f0b112d0700 1 --2- 172.21.15.42:0/3019195402 >> v2:172.21.15.42:3300/0 conn(0x7f0b0c09fae0 0x7f0b0c14fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:25:19.678 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:19.669+0000 7f0b0b7fe700 1 -- 172.21.15.42:0/3019195402 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 647326872 0 0) 0x7f0b0c07ef20 con 0x7f0b0c08df80 2019-12-05T05:25:19.678 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:19.669+0000 7f0b0b7fe700 1 -- 172.21.15.42:0/3019195402 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0af40030b0 con 0x7f0b0c08df80 2019-12-05T05:25:19.678 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:19.669+0000 7f0b0b7fe700 1 -- 172.21.15.42:0/3019195402 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3165855499 0 0) 0x7f0b0c0a0710 con 0x7f0b0c0a7740 2019-12-05T05:25:19.678 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:19.669+0000 7f0b0b7fe700 1 -- 172.21.15.42:0/3019195402 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b0c07ef20 con 0x7f0b0c0a7740 2019-12-05T05:25:19.679 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:19.669+0000 7f0b0b7fe700 1 -- 172.21.15.42:0/3019195402 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3773174319 0 0) 0x7f0af40030b0 con 0x7f0b0c08df80 2019-12-05T05:25:19.679 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:19.669+0000 7f0b112d0700 1 -- 172.21.15.42:0/3019195402 >> v1:172.21.15.42:6789/0 conn(0x7f0b0c08df80 legacy=0x7f0b0c09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:25:19.679 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:19.669+0000 7f0b112d0700 1 -- 172.21.15.42:0/3019195402 >> v1:172.21.15.40:6789/0 conn(0x7f0b0c0a7740 legacy=0x7f0b0c07e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:25:19.679 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:19.669+0000 7f0b112d0700 1 -- 172.21.15.42:0/3019195402 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b0c0a9ba0 con 0x7f0b0c09fae0 2019-12-05T05:25:19.679 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:19.669+0000 7f0b112d0700 1 --2- 172.21.15.42:0/3019195402 >> v2:172.21.15.42:3300/0 conn(0x7f0b0c09fae0 0x7f0b0c14fff0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f0b04008790 tx=0x7f0b000014a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:19.680 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:19.669+0000 7f0b112d0700 1 --2- 172.21.15.42:0/3019195402 >> v2:172.21.15.42:3300/0 conn(0x7f0b0c09fae0 0x7f0b0c14fff0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f0b04008790 tx=0x7f0b000014a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:19.680 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:19.669+0000 7f0b112d0700 1 --2- 172.21.15.42:0/3019195402 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0b0c09fae0 0x7f0b0c14fff0 secure :-1 s=READY pgs=809 cs=0 l=1 rx=0x7f0b04008790 tx=0x7f0b000014a0).ready entity=mon.1 client_cookie=15c6fe451fa326ef server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:25:19.680 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:19.669+0000 7f0b112d0700 1 --2- 172.21.15.42:0/3019195402 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0b0c09fae0 0x7f0b0c14fff0 secure :-1 s=THROTTLE_DONE pgs=809 cs=0 l=1 rx=0x7f0b04008790 tx=0x7f0b000014a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:19.680 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:19.669+0000 7f0b112d0700 1 --2- 172.21.15.42:0/3019195402 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0b0c09fae0 0x7f0b0c14fff0 secure :-1 s=THROTTLE_DONE pgs=809 cs=0 l=1 rx=0x7f0b04008790 tx=0x7f0b000014a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:19.680 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:19.669+0000 7f0b0b7fe700 1 -- 172.21.15.42:0/3019195402 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0b00001750 con 0x7f0b0c09fae0 2019-12-05T05:25:19.680 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:19.669+0000 7f0b112d0700 1 --2- 172.21.15.42:0/3019195402 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0b0c09fae0 0x7f0b0c14fff0 secure :-1 s=THROTTLE_DONE pgs=809 cs=0 l=1 rx=0x7f0b04008790 tx=0x7f0b000014a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:19.680 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:19.669+0000 7f0b0b7fe700 1 -- 172.21.15.42:0/3019195402 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b00001560 con 0x7f0b0c09fae0 2019-12-05T05:25:19.681 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:19.669+0000 7f0b0b7fe700 1 -- 172.21.15.42:0/3019195402 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0b04008da0 con 0x7f0b0c09fae0 2019-12-05T05:25:19.681 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:19.669+0000 7f0b12532700 1 -- 172.21.15.42:0/3019195402 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0b0c09fae0 msgr2=0x7f0b0c14fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:25:19.681 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:19.669+0000 7f0b12532700 1 --2- 172.21.15.42:0/3019195402 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0b0c09fae0 0x7f0b0c14fff0 secure :-1 s=READY pgs=809 cs=0 l=1 rx=0x7f0b04008790 tx=0x7f0b000014a0).stop 2019-12-05T05:25:19.681 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:19.669+0000 7f0b12532700 1 -- 172.21.15.42:0/3019195402 shutdown_connections 2019-12-05T05:25:19.681 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:19.669+0000 7f0b12532700 1 --2- 172.21.15.42:0/3019195402 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0b0c09fae0 0x7f0b0c14fff0 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:25:19.681 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:19.669+0000 7f0b12532700 1 -- 172.21.15.42:0/3019195402 shutdown_connections 2019-12-05T05:25:19.681 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:19.669+0000 7f0b12532700 1 -- 172.21.15.42:0/3019195402 wait complete. 2019-12-05T05:25:19.682 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:19.669+0000 7f0b12532700 1 -- 172.21.15.42:0/3019195402 >> 172.21.15.42:0/3019195402 conn(0x7f0b0c0aabe0 msgr2=0x7f0b0c0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:25:19.685 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:19.673+0000 7f0b12532700 1 Processor -- start 2019-12-05T05:25:19.685 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:19.673+0000 7f0b12532700 1 -- start start 2019-12-05T05:25:19.685 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:19.673+0000 7f0b12532700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f0b0c11ffc0 0x7f0b0c119bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:25:19.685 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:19.673+0000 7f0b12532700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0b0c1185f0 0x7f0b0c118a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:25:19.686 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:19.673+0000 7f0b12532700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0b0c11b8f0 0x7f0b0c11bd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:25:19.686 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:19.673+0000 7f0b12532700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b0c117fc0 con 0x7f0b0c11ffc0 2019-12-05T05:25:19.686 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:19.673+0000 7f0b12532700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b0c1203f0 con 0x7f0b0c1185f0 2019-12-05T05:25:19.686 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:19.673+0000 7f0b12532700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b0c1173b0 con 0x7f0b0c11b8f0 2019-12-05T05:25:19.686 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:19.677+0000 7f0b0bfff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0b0c1185f0 0x7f0b0c118a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:25:19.686 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:19.677+0000 7f0b0bfff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0b0c1185f0 0x7f0b0c118a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:33256/0 (socket says 172.21.15.42:33256) 2019-12-05T05:25:19.686 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:19.677+0000 7f0b0bfff700 1 -- 172.21.15.42:0/452941459 learned_addr learned my addr 172.21.15.42:0/452941459 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:25:19.687 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:19.677+0000 7f0b112d0700 1 --2- 172.21.15.42:0/452941459 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0b0c11b8f0 0x7f0b0c11bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:25:19.687 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:19.677+0000 7f0b10acf700 1 --2- 172.21.15.42:0/452941459 >> v2:172.21.15.40:3300/0 conn(0x7f0b0c11ffc0 0x7f0b0c119bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:25:19.687 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:19.677+0000 7f0b0bfff700 1 -- 172.21.15.42:0/452941459 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0b0c11b8f0 msgr2=0x7f0b0c11bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:25:19.687 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:19.677+0000 7f0b0bfff700 1 --2- 172.21.15.42:0/452941459 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0b0c11b8f0 0x7f0b0c11bd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:25:19.687 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:19.677+0000 7f0b0bfff700 1 -- 172.21.15.42:0/452941459 >> v2:172.21.15.40:3300/0 conn(0x7f0b0c11ffc0 msgr2=0x7f0b0c119bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:25:19.687 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:19.677+0000 7f0b0bfff700 1 --2- 172.21.15.42:0/452941459 >> v2:172.21.15.40:3300/0 conn(0x7f0b0c11ffc0 0x7f0b0c119bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:25:19.687 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:19.677+0000 7f0b0bfff700 1 -- 172.21.15.42:0/452941459 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b0c1210e0 con 0x7f0b0c1185f0 2019-12-05T05:25:19.688 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:19.677+0000 7f0b0bfff700 1 --2- 172.21.15.42:0/452941459 >> v2:172.21.15.42:3300/0 conn(0x7f0b0c1185f0 0x7f0b0c118a20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f0b0c05ad00 tx=0x7f0b00010a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:19.688 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:19.677+0000 7f0b0bfff700 1 --2- 172.21.15.42:0/452941459 >> v2:172.21.15.42:3300/0 conn(0x7f0b0c1185f0 0x7f0b0c118a20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f0b0c05ad00 tx=0x7f0b00010a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:19.688 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:19.677+0000 7f0b0bfff700 1 --2- 172.21.15.42:0/452941459 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0b0c1185f0 0x7f0b0c118a20 secure :-1 s=READY pgs=810 cs=0 l=1 rx=0x7f0b0c05ad00 tx=0x7f0b00010a90).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:25:19.688 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:19.677+0000 7f0b0bfff700 1 --2- 172.21.15.42:0/452941459 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0b0c1185f0 0x7f0b0c118a20 secure :-1 s=THROTTLE_DONE pgs=810 cs=0 l=1 rx=0x7f0b0c05ad00 tx=0x7f0b00010a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:19.688 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:19.677+0000 7f0b0bfff700 1 --2- 172.21.15.42:0/452941459 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0b0c1185f0 0x7f0b0c118a20 secure :-1 s=THROTTLE_DONE pgs=810 cs=0 l=1 rx=0x7f0b0c05ad00 tx=0x7f0b00010a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:19.688 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:19.677+0000 7f0b097fa700 1 -- 172.21.15.42:0/452941459 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0b0000cb60 con 0x7f0b0c1185f0 2019-12-05T05:25:19.689 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:19.677+0000 7f0b0bfff700 1 --2- 172.21.15.42:0/452941459 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0b0c1185f0 0x7f0b0c118a20 secure :-1 s=THROTTLE_DONE pgs=810 cs=0 l=1 rx=0x7f0b0c05ad00 tx=0x7f0b00010a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:19.689 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:19.677+0000 7f0b097fa700 1 -- 172.21.15.42:0/452941459 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b0000c780 con 0x7f0b0c1185f0 2019-12-05T05:25:19.689 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:19.677+0000 7f0b12532700 1 -- 172.21.15.42:0/452941459 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b0c1316e0 con 0x7f0b0c1185f0 2019-12-05T05:25:19.689 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:19.677+0000 7f0b097fa700 1 -- 172.21.15.42:0/452941459 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0b00004620 con 0x7f0b0c1185f0 2019-12-05T05:25:19.689 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:19.677+0000 7f0b12532700 1 -- 172.21.15.42:0/452941459 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b0c11a110 con 0x7f0b0c1185f0 2019-12-05T05:25:19.689 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:19.677+0000 7f0b0bfff700 1 --2- 172.21.15.42:0/452941459 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0b0c1185f0 0x7f0b0c118a20 secure :-1 s=THROTTLE_DONE pgs=810 cs=0 l=1 rx=0x7f0b0c05ad00 tx=0x7f0b00010a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:19.689 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:19.677+0000 7f0b097fa700 1 -- 172.21.15.42:0/452941459 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f0b00004930 con 0x7f0b0c1185f0 2019-12-05T05:25:19.690 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:19.677+0000 7f0b0bfff700 1 --2- 172.21.15.42:0/452941459 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0b0c1185f0 0x7f0b0c118a20 secure :-1 s=THROTTLE_DONE pgs=810 cs=0 l=1 rx=0x7f0b0c05ad00 tx=0x7f0b00010a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:19.690 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:19.677+0000 7f0b097fa700 1 --2- 172.21.15.42:0/452941459 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0aec03ae00 0x7f0aec03d310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:25:19.690 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:19.677+0000 7f0b097fa700 1 -- 172.21.15.42:0/452941459 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 7235+0+0 (secure 0 0 0) 0x7f0b0004ad30 con 0x7f0b0c1185f0 2019-12-05T05:25:19.690 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:19.677+0000 7f0b10acf700 1 --2- 172.21.15.42:0/452941459 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0aec03ae00 0x7f0aec03d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:25:19.690 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:19.677+0000 7f0b10acf700 1 --2- 172.21.15.42:0/452941459 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0aec03ae00 0x7f0aec03d310 crc :-1 s=READY pgs=357 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:25:19.691 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:19.681+0000 7f0b12532700 1 -- 172.21.15.42:0/452941459 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b0c05d110 con 0x7f0b0c1185f0 2019-12-05T05:25:19.691 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:19.685+0000 7f0b0bfff700 1 --2- 172.21.15.42:0/452941459 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0b0c1185f0 0x7f0b0c118a20 secure :-1 s=THROTTLE_DONE pgs=810 cs=0 l=1 rx=0x7f0b0c05ad00 tx=0x7f0b00010a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:19.691 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:19.685+0000 7f0b097fa700 1 -- 172.21.15.42:0/452941459 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f0b00031db0 con 0x7f0b0c1185f0 2019-12-05T05:25:19.857 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:19.853+0000 7f0b12532700 1 -- 172.21.15.42:0/452941459 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0b0c05b8c0 con 0x7f0b0c1185f0 2019-12-05T05:25:19.857 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:19.853+0000 7f0b0bfff700 1 --2- 172.21.15.42:0/452941459 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0b0c1185f0 0x7f0b0c118a20 secure :-1 s=THROTTLE_DONE pgs=810 cs=0 l=1 rx=0x7f0b0c05ad00 tx=0x7f0b00010a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:19.858 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:19.853+0000 7f0b097fa700 1 -- 172.21.15.42:0/452941459 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 103 v103) v1 ==== 95+0+2111 (secure 0 0 0) 0x7f0b00032010 con 0x7f0b0c1185f0 2019-12-05T05:25:19.860 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:25:19.860 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":103,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5240,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/1455108684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1455108684},{"type":"v1","addr":"172.21.15.42:6835","nonce":1455108684}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":103}],"filesystems":[{"mdsmap":{"epoch":102,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:23:58.091182+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5177":{"gid":5177,"name":"b","rank":0,"incarnation":99,"state":"up:active","state_seq":14,"addr":"172.21.15.40:6833/3208416000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3208416000},{"type":"v1","addr":"172.21.15.40:6833","nonce":3208416000}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:25:19.860 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 103 2019-12-05T05:25:19.865 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:19.857+0000 7f0b12532700 1 -- 172.21.15.42:0/452941459 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0aec03ae00 msgr2=0x7f0aec03d310 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:25:19.865 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:19.857+0000 7f0b12532700 1 --2- 172.21.15.42:0/452941459 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0aec03ae00 0x7f0aec03d310 crc :-1 s=READY pgs=357 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:25:19.865 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:19.857+0000 7f0b12532700 1 -- 172.21.15.42:0/452941459 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0b0c1185f0 msgr2=0x7f0b0c118a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:25:19.866 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:19.857+0000 7f0b12532700 1 --2- 172.21.15.42:0/452941459 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0b0c1185f0 0x7f0b0c118a20 secure :-1 s=READY pgs=810 cs=0 l=1 rx=0x7f0b0c05ad00 tx=0x7f0b00010a90).stop 2019-12-05T05:25:19.866 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:19.857+0000 7f0b12532700 1 -- 172.21.15.42:0/452941459 shutdown_connections 2019-12-05T05:25:19.866 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:19.857+0000 7f0b12532700 1 --2- 172.21.15.42:0/452941459 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0aec03ae00 0x7f0aec03d310 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:25:19.866 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:19.857+0000 7f0b12532700 1 --2- 172.21.15.42:0/452941459 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0b0c11b8f0 0x7f0b0c11bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:25:19.866 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:19.857+0000 7f0b12532700 1 --2- 172.21.15.42:0/452941459 >> v2:172.21.15.40:3300/0 conn(0x7f0b0c11ffc0 0x7f0b0c119bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:25:19.866 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:19.857+0000 7f0b12532700 1 --2- 172.21.15.42:0/452941459 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0b0c1185f0 0x7f0b0c118a20 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:25:19.866 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:19.857+0000 7f0b12532700 1 -- 172.21.15.42:0/452941459 shutdown_connections 2019-12-05T05:25:19.867 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:19.857+0000 7f0b12532700 1 -- 172.21.15.42:0/452941459 wait complete. 2019-12-05T05:25:19.867 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:19.857+0000 7f0b12532700 1 -- 172.21.15.42:0/452941459 >> 172.21.15.42:0/452941459 conn(0x7f0b0c0a7f90 msgr2=0x7f0b0c07e4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:25:19.873 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T05:25:21.874 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:25:21.978 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:21.965+0000 7f4da45d8700 1 Processor -- start 2019-12-05T05:25:21.978 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:21.965+0000 7f4da45d8700 1 -- start start 2019-12-05T05:25:21.978 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:21.969+0000 7f4da45d8700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f4d9c0a7730 0x7f4d9c08df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:25:21.978 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:21.969+0000 7f4da45d8700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d9c0a0700 con 0x7f4d9c07e710 2019-12-05T05:25:21.979 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:21.969+0000 7f4da45d8700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d9c0a9b90 con 0x7f4d9c09fad0 2019-12-05T05:25:21.979 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:21.969+0000 7f4da45d8700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d9c0a0af0 con 0x7f4d9c0a7730 2019-12-05T05:25:21.979 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:21.969+0000 7f4da3376700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f4d9c09fad0 0x7f4d9c14efd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:33304/0 (socket says 172.21.15.42:33304) 2019-12-05T05:25:21.979 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:21.969+0000 7f4da3376700 1 -- 172.21.15.42:0/1366019373 learned_addr learned my addr 172.21.15.42:0/1366019373 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:25:21.979 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:21.969+0000 7f4da2b75700 1 --2- 172.21.15.42:0/1366019373 >> v2:172.21.15.40:3300/0 conn(0x7f4d9c0a7730 0x7f4d9c08df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:25:21.979 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:21.969+0000 7f4da1b73700 1 -- 172.21.15.42:0/1366019373 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3772763305 0 0) 0x7f4d9c0a9b90 con 0x7f4d9c09fad0 2019-12-05T05:25:21.980 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:21.969+0000 7f4da1b73700 1 -- 172.21.15.42:0/1366019373 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d880030b0 con 0x7f4d9c09fad0 2019-12-05T05:25:21.980 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:21.969+0000 7f4da2b75700 1 -- 172.21.15.42:0/1366019373 >> v1:172.21.15.42:6789/0 conn(0x7f4d9c09fad0 legacy=0x7f4d9c14efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:25:21.980 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:21.969+0000 7f4da2b75700 1 -- 172.21.15.42:0/1366019373 >> v1:172.21.15.40:6789/0 conn(0x7f4d9c07e710 legacy=0x7f4d9c09f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:25:21.980 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:21.969+0000 7f4da1b73700 1 -- 172.21.15.42:0/1366019373 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2110233730 0 0) 0x7f4d9c0a0700 con 0x7f4d9c07e710 2019-12-05T05:25:21.980 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:21.969+0000 7f4da2b75700 1 -- 172.21.15.42:0/1366019373 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d9c07eb40 con 0x7f4d9c0a7730 2019-12-05T05:25:21.981 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:21.969+0000 7f4da2b75700 1 --2- 172.21.15.42:0/1366019373 >> v2:172.21.15.40:3300/0 conn(0x7f4d9c0a7730 0x7f4d9c08df70 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f4d98008790 tx=0x7f4d9800fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:21.981 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:21.969+0000 7f4da2b75700 1 --2- 172.21.15.42:0/1366019373 >> v2:172.21.15.40:3300/0 conn(0x7f4d9c0a7730 0x7f4d9c08df70 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f4d98008790 tx=0x7f4d9800fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:21.981 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:21.969+0000 7f4da2b75700 1 --2- 172.21.15.42:0/1366019373 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4d9c0a7730 0x7f4d9c08df70 secure :-1 s=READY pgs=644 cs=0 l=1 rx=0x7f4d98008790 tx=0x7f4d9800fbf0).ready entity=mon.0 client_cookie=da14a0f19855cc24 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:25:21.981 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:21.969+0000 7f4da2b75700 1 --2- 172.21.15.42:0/1366019373 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4d9c0a7730 0x7f4d9c08df70 secure :-1 s=THROTTLE_DONE pgs=644 cs=0 l=1 rx=0x7f4d98008790 tx=0x7f4d9800fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:21.981 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:21.969+0000 7f4da1b73700 1 -- 172.21.15.42:0/1366019373 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4d98011270 con 0x7f4d9c0a7730 2019-12-05T05:25:21.981 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:21.969+0000 7f4da2b75700 1 --2- 172.21.15.42:0/1366019373 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4d9c0a7730 0x7f4d9c08df70 secure :-1 s=THROTTLE_DONE pgs=644 cs=0 l=1 rx=0x7f4d98008790 tx=0x7f4d9800fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:21.981 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:21.969+0000 7f4da1b73700 1 -- 172.21.15.42:0/1366019373 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d98008bc0 con 0x7f4d9c0a7730 2019-12-05T05:25:21.982 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:21.969+0000 7f4da2b75700 1 --2- 172.21.15.42:0/1366019373 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4d9c0a7730 0x7f4d9c08df70 secure :-1 s=THROTTLE_DONE pgs=644 cs=0 l=1 rx=0x7f4d98008790 tx=0x7f4d9800fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:21.982 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:21.969+0000 7f4da1b73700 1 -- 172.21.15.42:0/1366019373 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4d98011850 con 0x7f4d9c0a7730 2019-12-05T05:25:21.982 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:21.969+0000 7f4da45d8700 1 -- 172.21.15.42:0/1366019373 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4d9c0a7730 msgr2=0x7f4d9c08df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:25:21.982 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:21.969+0000 7f4da45d8700 1 --2- 172.21.15.42:0/1366019373 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4d9c0a7730 0x7f4d9c08df70 secure :-1 s=READY pgs=644 cs=0 l=1 rx=0x7f4d98008790 tx=0x7f4d9800fbf0).stop 2019-12-05T05:25:21.982 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:21.973+0000 7f4da45d8700 1 -- 172.21.15.42:0/1366019373 shutdown_connections 2019-12-05T05:25:21.982 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:21.973+0000 7f4da45d8700 1 --2- 172.21.15.42:0/1366019373 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4d9c0a7730 0x7f4d9c08df70 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:25:21.982 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:21.973+0000 7f4da45d8700 1 -- 172.21.15.42:0/1366019373 shutdown_connections 2019-12-05T05:25:21.983 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:21.973+0000 7f4da45d8700 1 -- 172.21.15.42:0/1366019373 wait complete. 2019-12-05T05:25:21.983 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:21.973+0000 7f4da45d8700 1 -- 172.21.15.42:0/1366019373 >> 172.21.15.42:0/1366019373 conn(0x7f4d9c0aabd0 msgr2=0x7f4d9c0a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:25:21.986 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:21.973+0000 7f4da45d8700 1 Processor -- start 2019-12-05T05:25:21.987 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:21.973+0000 7f4da45d8700 1 -- start start 2019-12-05T05:25:21.987 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:21.977+0000 7f4da45d8700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f4d9c0a38b0 0x7f4d9c11a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:25:21.987 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:21.977+0000 7f4da45d8700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f4d9c117ec0 0x7f4d9c1182f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:25:21.987 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:21.977+0000 7f4da45d8700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d9c1313c0 con 0x7f4d9c08eb60 2019-12-05T05:25:21.987 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:21.977+0000 7f4da45d8700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d9c0a3ce0 con 0x7f4d9c117ec0 2019-12-05T05:25:21.987 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:21.977+0000 7f4da45d8700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d9c11b160 con 0x7f4d9c0a38b0 2019-12-05T05:25:21.987 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:21.977+0000 7f4da2b75700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f4d9c0a38b0 0x7f4d9c11a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:25:21.988 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:21.977+0000 7f4da2b75700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f4d9c0a38b0 0x7f4d9c11a890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:33268/0 (socket says 172.21.15.42:33268) 2019-12-05T05:25:21.988 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:21.977+0000 7f4da2b75700 1 -- 172.21.15.42:0/1465218791 learned_addr learned my addr 172.21.15.42:0/1465218791 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:25:21.988 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:21.977+0000 7f4da2374700 1 --2- 172.21.15.42:0/1465218791 >> v2:172.21.15.40:3300/0 conn(0x7f4d9c117ec0 0x7f4d9c1182f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:25:21.988 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:21.977+0000 7f4da2b75700 1 -- 172.21.15.42:0/1465218791 >> v2:172.21.15.40:3300/0 conn(0x7f4d9c117ec0 msgr2=0x7f4d9c1182f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:25:21.988 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:21.977+0000 7f4da2b75700 1 --2- 172.21.15.42:0/1465218791 >> v2:172.21.15.40:3300/0 conn(0x7f4d9c117ec0 0x7f4d9c1182f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:25:21.988 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:21.977+0000 7f4da2b75700 1 -- 172.21.15.42:0/1465218791 >> v1:172.21.15.40:6789/0 conn(0x7f4d9c08eb60 legacy=0x7f4d9c08ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:25:21.989 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:21.977+0000 7f4da2b75700 1 -- 172.21.15.42:0/1465218791 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d9c11ade0 con 0x7f4d9c0a38b0 2019-12-05T05:25:21.989 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:21.977+0000 7f4da2374700 1 --2- 172.21.15.42:0/1465218791 >> v2:172.21.15.40:3300/0 conn(0x7f4d9c117ec0 0x7f4d9c1182f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T05:25:21.989 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:21.977+0000 7f4da2b75700 1 --2- 172.21.15.42:0/1465218791 >> v2:172.21.15.42:3300/0 conn(0x7f4d9c0a38b0 0x7f4d9c11a890 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f4d98008790 tx=0x7f4d98008ed0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:21.989 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:21.977+0000 7f4da2b75700 1 --2- 172.21.15.42:0/1465218791 >> v2:172.21.15.42:3300/0 conn(0x7f4d9c0a38b0 0x7f4d9c11a890 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f4d98008790 tx=0x7f4d98008ed0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:21.989 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:21.977+0000 7f4da2b75700 1 --2- 172.21.15.42:0/1465218791 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4d9c0a38b0 0x7f4d9c11a890 secure :-1 s=READY pgs=812 cs=0 l=1 rx=0x7f4d98008790 tx=0x7f4d98008ed0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:25:21.989 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:21.977+0000 7f4da2b75700 1 --2- 172.21.15.42:0/1465218791 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4d9c0a38b0 0x7f4d9c11a890 secure :-1 s=THROTTLE_DONE pgs=812 cs=0 l=1 rx=0x7f4d98008790 tx=0x7f4d98008ed0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:21.989 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:21.977+0000 7f4da2b75700 1 --2- 172.21.15.42:0/1465218791 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4d9c0a38b0 0x7f4d9c11a890 secure :-1 s=THROTTLE_DONE pgs=812 cs=0 l=1 rx=0x7f4d98008790 tx=0x7f4d98008ed0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:21.990 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:21.977+0000 7f4da2b75700 1 --2- 172.21.15.42:0/1465218791 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4d9c0a38b0 0x7f4d9c11a890 secure :-1 s=THROTTLE_DONE pgs=812 cs=0 l=1 rx=0x7f4d98008790 tx=0x7f4d98008ed0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:21.990 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:21.977+0000 7f4d8f7fe700 1 -- 172.21.15.42:0/1465218791 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4d98007340 con 0x7f4d9c0a38b0 2019-12-05T05:25:21.990 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:21.977+0000 7f4d8f7fe700 1 -- 172.21.15.42:0/1465218791 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d98011470 con 0x7f4d9c0a38b0 2019-12-05T05:25:21.990 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:21.977+0000 7f4d8f7fe700 1 -- 172.21.15.42:0/1465218791 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4d98005540 con 0x7f4d9c0a38b0 2019-12-05T05:25:21.990 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:21.977+0000 7f4da45d8700 1 -- 172.21.15.42:0/1465218791 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d9c08f790 con 0x7f4d9c0a38b0 2019-12-05T05:25:21.990 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:21.977+0000 7f4da45d8700 1 -- 172.21.15.42:0/1465218791 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d9c119de0 con 0x7f4d9c0a38b0 2019-12-05T05:25:21.991 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:21.977+0000 7f4da2b75700 1 --2- 172.21.15.42:0/1465218791 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4d9c0a38b0 0x7f4d9c11a890 secure :-1 s=THROTTLE_DONE pgs=812 cs=0 l=1 rx=0x7f4d98008790 tx=0x7f4d98008ed0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:21.991 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:21.981+0000 7f4d8f7fe700 1 -- 172.21.15.42:0/1465218791 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f4d98005810 con 0x7f4d9c0a38b0 2019-12-05T05:25:21.991 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:21.981+0000 7f4da2b75700 1 --2- 172.21.15.42:0/1465218791 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4d9c0a38b0 0x7f4d9c11a890 secure :-1 s=THROTTLE_DONE pgs=812 cs=0 l=1 rx=0x7f4d98008790 tx=0x7f4d98008ed0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:21.991 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:21.981+0000 7f4d8f7fe700 1 --2- 172.21.15.42:0/1465218791 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4d8403ae40 0x7f4d8403d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:25:21.991 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:21.981+0000 7f4d8f7fe700 1 -- 172.21.15.42:0/1465218791 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 7235+0+0 (secure 0 0 0) 0x7f4d9804bd80 con 0x7f4d9c0a38b0 2019-12-05T05:25:21.992 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:21.981+0000 7f4da2374700 1 --2- 172.21.15.42:0/1465218791 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4d8403ae40 0x7f4d8403d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:25:21.992 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:21.981+0000 7f4da2374700 1 --2- 172.21.15.42:0/1465218791 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4d8403ae40 0x7f4d8403d350 crc :-1 s=READY pgs=358 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:25:21.992 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:21.981+0000 7f4da45d8700 1 -- 172.21.15.42:0/1465218791 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d9c05d100 con 0x7f4d9c0a38b0 2019-12-05T05:25:21.995 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:21.989+0000 7f4da2b75700 1 --2- 172.21.15.42:0/1465218791 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4d9c0a38b0 0x7f4d9c11a890 secure :-1 s=THROTTLE_DONE pgs=812 cs=0 l=1 rx=0x7f4d98008790 tx=0x7f4d98008ed0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:21.995 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:21.989+0000 7f4d8f7fe700 1 -- 172.21.15.42:0/1465218791 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f4d98032e00 con 0x7f4d9c0a38b0 2019-12-05T05:25:22.157 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:22.153+0000 7f4da45d8700 1 -- 172.21.15.42:0/1465218791 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4d9c0aabd0 con 0x7f4d9c0a38b0 2019-12-05T05:25:22.158 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:22.153+0000 7f4da2b75700 1 --2- 172.21.15.42:0/1465218791 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4d9c0a38b0 0x7f4d9c11a890 secure :-1 s=THROTTLE_DONE pgs=812 cs=0 l=1 rx=0x7f4d98008790 tx=0x7f4d98008ed0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:22.158 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:22.153+0000 7f4d8f7fe700 1 -- 172.21.15.42:0/1465218791 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 103 v103) v1 ==== 95+0+2111 (secure 0 0 0) 0x7f4d98033060 con 0x7f4d9c0a38b0 2019-12-05T05:25:22.159 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 103 2019-12-05T05:25:22.160 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:25:22.160 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":103,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5240,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/1455108684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1455108684},{"type":"v1","addr":"172.21.15.42:6835","nonce":1455108684}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":103}],"filesystems":[{"mdsmap":{"epoch":102,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:23:58.091182+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5177":{"gid":5177,"name":"b","rank":0,"incarnation":99,"state":"up:active","state_seq":14,"addr":"172.21.15.40:6833/3208416000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3208416000},{"type":"v1","addr":"172.21.15.40:6833","nonce":3208416000}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:25:22.162 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:22.157+0000 7f4da45d8700 1 -- 172.21.15.42:0/1465218791 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4d8403ae40 msgr2=0x7f4d8403d350 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:25:22.162 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:22.157+0000 7f4da45d8700 1 --2- 172.21.15.42:0/1465218791 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4d8403ae40 0x7f4d8403d350 crc :-1 s=READY pgs=358 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:25:22.162 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:22.157+0000 7f4da45d8700 1 -- 172.21.15.42:0/1465218791 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4d9c0a38b0 msgr2=0x7f4d9c11a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:25:22.162 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:22.157+0000 7f4da45d8700 1 --2- 172.21.15.42:0/1465218791 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4d9c0a38b0 0x7f4d9c11a890 secure :-1 s=READY pgs=812 cs=0 l=1 rx=0x7f4d98008790 tx=0x7f4d98008ed0).stop 2019-12-05T05:25:22.162 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:22.157+0000 7f4da45d8700 1 -- 172.21.15.42:0/1465218791 shutdown_connections 2019-12-05T05:25:22.162 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:22.157+0000 7f4da45d8700 1 --2- 172.21.15.42:0/1465218791 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4d8403ae40 0x7f4d8403d350 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:25:22.163 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:22.157+0000 7f4da45d8700 1 --2- 172.21.15.42:0/1465218791 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4d9c0a38b0 0x7f4d9c11a890 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:25:22.163 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:22.157+0000 7f4da45d8700 1 --2- 172.21.15.42:0/1465218791 >> v2:172.21.15.40:3300/0 conn(0x7f4d9c117ec0 0x7f4d9c1182f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:25:22.163 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:22.157+0000 7f4da45d8700 1 -- 172.21.15.42:0/1465218791 shutdown_connections 2019-12-05T05:25:22.163 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:22.157+0000 7f4da45d8700 1 -- 172.21.15.42:0/1465218791 wait complete. 2019-12-05T05:25:22.163 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:22.157+0000 7f4da45d8700 1 -- 172.21.15.42:0/1465218791 >> 172.21.15.42:0/1465218791 conn(0x7f4d9c11c690 msgr2=0x7f4d9c0a6b50 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:25:22.170 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T05:25:24.171 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:25:24.279 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:24.269+0000 7fceb2517700 1 Processor -- start 2019-12-05T05:25:24.279 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:24.269+0000 7fceb2517700 1 -- start start 2019-12-05T05:25:24.280 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:24.269+0000 7fceb2517700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fceac0a7740 0x7fceac08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:25:24.280 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:24.269+0000 7fceb2517700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fceac09fae0 0x7fceac07eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:25:24.280 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:24.269+0000 7fceb2517700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fceac0a0710 con 0x7fceac07e720 2019-12-05T05:25:24.280 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:24.269+0000 7fceb2517700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fceac0a0b00 con 0x7fceac0a7740 2019-12-05T05:25:24.280 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:24.269+0000 7fceb2517700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fceac08e5b0 con 0x7fceac09fae0 2019-12-05T05:25:24.280 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:24.269+0000 7fceb12b5700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fceac09fae0 0x7fceac07eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:25:24.280 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:24.273+0000 7fceabfff700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fceac07e720 0x7fceac09f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:42762/0 (socket says 172.21.15.42:42762) 2019-12-05T05:25:24.281 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:24.273+0000 7fceabfff700 1 -- 172.21.15.42:0/1267095071 learned_addr learned my addr 172.21.15.42:0/1267095071 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:25:24.281 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:24.273+0000 7fceb0ab4700 1 --2- 172.21.15.42:0/1267095071 >> v2:172.21.15.40:3300/0 conn(0x7fceac0a7740 0x7fceac08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:25:24.281 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:24.273+0000 7fceab7fe700 1 -- 172.21.15.42:0/1267095071 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2177652485 0 0) 0x7fceac0a0710 con 0x7fceac07e720 2019-12-05T05:25:24.281 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:24.273+0000 7fceab7fe700 1 -- 172.21.15.42:0/1267095071 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce980030b0 con 0x7fceac07e720 2019-12-05T05:25:24.281 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:24.273+0000 7fceb12b5700 1 -- 172.21.15.42:0/1267095071 >> v2:172.21.15.40:3300/0 conn(0x7fceac0a7740 msgr2=0x7fceac08df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:25:24.281 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:24.273+0000 7fceb12b5700 1 --2- 172.21.15.42:0/1267095071 >> v2:172.21.15.40:3300/0 conn(0x7fceac0a7740 0x7fceac08df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:25:24.281 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:24.273+0000 7fceb12b5700 1 -- 172.21.15.42:0/1267095071 >> v1:172.21.15.40:6789/0 conn(0x7fceac07e720 legacy=0x7fceac09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:25:24.282 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:24.273+0000 7fceb12b5700 1 -- 172.21.15.42:0/1267095071 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fceac07f0a0 con 0x7fceac09fae0 2019-12-05T05:25:24.282 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:24.273+0000 7fceb12b5700 1 --2- 172.21.15.42:0/1267095071 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fceac09fae0 0x7fceac07eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fcea4008b10 tx=0x7fcea400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:24.282 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:24.273+0000 7fceb12b5700 1 --2- 172.21.15.42:0/1267095071 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fceac09fae0 0x7fceac07eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fcea4008b10 tx=0x7fcea400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:24.282 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:24.273+0000 7fceb12b5700 1 --2- 172.21.15.42:0/1267095071 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fceac09fae0 0x7fceac07eb50 secure :-1 s=READY pgs=168 cs=0 l=1 rx=0x7fcea4008b10 tx=0x7fcea400fff0).ready entity=mon.2 client_cookie=c69eb4b6a05cfc7a server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:25:24.282 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:24.273+0000 7fceb12b5700 1 --2- 172.21.15.42:0/1267095071 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fceac09fae0 0x7fceac07eb50 secure :-1 s=THROTTLE_DONE pgs=168 cs=0 l=1 rx=0x7fcea4008b10 tx=0x7fcea400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:24.282 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:24.273+0000 7fceab7fe700 1 -- 172.21.15.42:0/1267095071 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcea4010900 con 0x7fceac09fae0 2019-12-05T05:25:24.282 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:24.273+0000 7fceb12b5700 1 --2- 172.21.15.42:0/1267095071 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fceac09fae0 0x7fceac07eb50 secure :-1 s=THROTTLE_DONE pgs=168 cs=0 l=1 rx=0x7fcea4008b10 tx=0x7fcea400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:24.283 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:24.273+0000 7fceb12b5700 1 --2- 172.21.15.42:0/1267095071 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fceac09fae0 0x7fceac07eb50 secure :-1 s=THROTTLE_DONE pgs=168 cs=0 l=1 rx=0x7fcea4008b10 tx=0x7fcea400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:24.283 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:24.273+0000 7fceab7fe700 1 -- 172.21.15.42:0/1267095071 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcea4008b50 con 0x7fceac09fae0 2019-12-05T05:25:24.283 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:24.273+0000 7fceab7fe700 1 -- 172.21.15.42:0/1267095071 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcea40110c0 con 0x7fceac09fae0 2019-12-05T05:25:24.283 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:24.273+0000 7fceb2517700 1 -- 172.21.15.42:0/1267095071 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fceac09fae0 msgr2=0x7fceac07eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:25:24.283 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:24.273+0000 7fceb2517700 1 --2- 172.21.15.42:0/1267095071 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fceac09fae0 0x7fceac07eb50 secure :-1 s=READY pgs=168 cs=0 l=1 rx=0x7fcea4008b10 tx=0x7fcea400fff0).stop 2019-12-05T05:25:24.283 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:24.273+0000 7fceb2517700 1 -- 172.21.15.42:0/1267095071 shutdown_connections 2019-12-05T05:25:24.283 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:24.273+0000 7fceb2517700 1 --2- 172.21.15.42:0/1267095071 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fceac09fae0 0x7fceac07eb50 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:25:24.284 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:24.273+0000 7fceb2517700 1 --2- 172.21.15.42:0/1267095071 >> v2:172.21.15.40:3300/0 conn(0x7fceac0a7740 0x7fceac08df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:25:24.284 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:24.273+0000 7fceb2517700 1 -- 172.21.15.42:0/1267095071 shutdown_connections 2019-12-05T05:25:24.284 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:24.273+0000 7fceb2517700 1 -- 172.21.15.42:0/1267095071 wait complete. 2019-12-05T05:25:24.284 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:24.273+0000 7fceb2517700 1 -- 172.21.15.42:0/1267095071 >> 172.21.15.42:0/1267095071 conn(0x7fceac0aabe0 msgr2=0x7fceac0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:25:24.287 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:24.277+0000 7fceb2517700 1 Processor -- start 2019-12-05T05:25:24.287 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:24.277+0000 7fceb2517700 1 -- start start 2019-12-05T05:25:24.287 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:24.277+0000 7fceb2517700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fceac0a5eb0 0x7fceac07ad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:25:24.287 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:24.277+0000 7fceb2517700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fceac07b280 0x7fceac1335c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:25:24.287 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:24.277+0000 7fceb2517700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fceac0a62e0 con 0x7fceac116500 2019-12-05T05:25:24.287 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:24.277+0000 7fceb2517700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fceac07d7f0 con 0x7fceac07b280 2019-12-05T05:25:24.288 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:24.277+0000 7fceb2517700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fceac0a6730 con 0x7fceac0a5eb0 2019-12-05T05:25:24.288 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:24.281+0000 7fceb12b5700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fceac07b280 0x7fceac1335c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:25:24.288 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:24.281+0000 7fceb0ab4700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fceac116500 0x7fceac0a6b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:42766/0 (socket says 172.21.15.42:42766) 2019-12-05T05:25:24.288 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:24.281+0000 7fceb12b5700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fceac07b280 0x7fceac1335c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:33292/0 (socket says 172.21.15.42:33292) 2019-12-05T05:25:24.288 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:24.281+0000 7fceb0ab4700 1 -- 172.21.15.42:0/3905500267 learned_addr learned my addr 172.21.15.42:0/3905500267 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:25:24.288 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:24.281+0000 7fceabfff700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fceac0a5eb0 0x7fceac07ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:25:24.288 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:24.281+0000 7fcea97fa700 1 -- 172.21.15.42:0/3905500267 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 815714432 0 0) 0x7fceac0a62e0 con 0x7fceac116500 2019-12-05T05:25:24.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:24.281+0000 7fcea97fa700 1 -- 172.21.15.42:0/3905500267 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce900030b0 con 0x7fceac116500 2019-12-05T05:25:24.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:24.281+0000 7fceb12b5700 1 -- 172.21.15.42:0/3905500267 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fceac0a5eb0 msgr2=0x7fceac07ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:25:24.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:24.281+0000 7fceb12b5700 1 --2- 172.21.15.42:0/3905500267 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fceac0a5eb0 0x7fceac07ad30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:25:24.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:24.281+0000 7fceb12b5700 1 -- 172.21.15.42:0/3905500267 >> v1:172.21.15.40:6789/0 conn(0x7fceac116500 legacy=0x7fceac0a6b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:25:24.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:24.281+0000 7fceb12b5700 1 -- 172.21.15.42:0/3905500267 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fceac133b10 con 0x7fceac07b280 2019-12-05T05:25:24.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:24.281+0000 7fceb12b5700 1 --2- 172.21.15.42:0/3905500267 >> v2:172.21.15.42:3300/0 conn(0x7fceac07b280 0x7fceac1335c0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fcea4008b10 tx=0x7fcea4010900).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:24.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:24.281+0000 7fceb12b5700 1 --2- 172.21.15.42:0/3905500267 >> v2:172.21.15.42:3300/0 conn(0x7fceac07b280 0x7fceac1335c0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fcea4008b10 tx=0x7fcea4010900).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:24.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:24.281+0000 7fceb12b5700 1 --2- 172.21.15.42:0/3905500267 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fceac07b280 0x7fceac1335c0 secure :-1 s=READY pgs=813 cs=0 l=1 rx=0x7fcea4008b10 tx=0x7fcea4010900).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:25:24.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:24.281+0000 7fceb12b5700 1 --2- 172.21.15.42:0/3905500267 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fceac07b280 0x7fceac1335c0 secure :-1 s=THROTTLE_DONE pgs=813 cs=0 l=1 rx=0x7fcea4008b10 tx=0x7fcea4010900).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:24.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:24.281+0000 7fceb12b5700 1 --2- 172.21.15.42:0/3905500267 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fceac07b280 0x7fceac1335c0 secure :-1 s=THROTTLE_DONE pgs=813 cs=0 l=1 rx=0x7fcea4008b10 tx=0x7fcea4010900).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:24.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:24.281+0000 7fcea97fa700 1 -- 172.21.15.42:0/3905500267 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcea4005ab0 con 0x7fceac07b280 2019-12-05T05:25:24.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:24.281+0000 7fceb12b5700 1 --2- 172.21.15.42:0/3905500267 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fceac07b280 0x7fceac1335c0 secure :-1 s=THROTTLE_DONE pgs=813 cs=0 l=1 rx=0x7fcea4008b10 tx=0x7fcea4010900).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:24.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:24.281+0000 7fcea97fa700 1 -- 172.21.15.42:0/3905500267 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcea4010b00 con 0x7fceac07b280 2019-12-05T05:25:24.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:24.281+0000 7fcea97fa700 1 -- 172.21.15.42:0/3905500267 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcea400f470 con 0x7fceac07b280 2019-12-05T05:25:24.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:24.281+0000 7fceb2517700 1 -- 172.21.15.42:0/3905500267 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fceac133de0 con 0x7fceac07b280 2019-12-05T05:25:24.291 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:24.281+0000 7fceb2517700 1 -- 172.21.15.42:0/3905500267 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fceac13ac30 con 0x7fceac07b280 2019-12-05T05:25:24.291 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:24.281+0000 7fceb12b5700 1 --2- 172.21.15.42:0/3905500267 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fceac07b280 0x7fceac1335c0 secure :-1 s=THROTTLE_DONE pgs=813 cs=0 l=1 rx=0x7fcea4008b10 tx=0x7fcea4010900).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:24.291 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:24.281+0000 7fcea97fa700 1 -- 172.21.15.42:0/3905500267 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fcea400f700 con 0x7fceac07b280 2019-12-05T05:25:24.291 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:24.281+0000 7fceb12b5700 1 --2- 172.21.15.42:0/3905500267 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fceac07b280 0x7fceac1335c0 secure :-1 s=THROTTLE_DONE pgs=813 cs=0 l=1 rx=0x7fcea4008b10 tx=0x7fcea4010900).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:24.292 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:24.281+0000 7fcea97fa700 1 --2- 172.21.15.42:0/3905500267 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fce9003b2a0 0x7fce9003d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:25:24.292 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:24.281+0000 7fcea97fa700 1 -- 172.21.15.42:0/3905500267 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 7235+0+0 (secure 0 0 0) 0x7fcea404aee0 con 0x7fceac07b280 2019-12-05T05:25:24.292 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:24.281+0000 7fceb0ab4700 1 --2- 172.21.15.42:0/3905500267 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fce9003b2a0 0x7fce9003d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:25:24.292 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:24.285+0000 7fceb0ab4700 1 --2- 172.21.15.42:0/3905500267 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fce9003b2a0 0x7fce9003d7b0 crc :-1 s=READY pgs=359 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:25:24.293 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:24.285+0000 7fceb2517700 1 -- 172.21.15.42:0/3905500267 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fceac0a68f0 con 0x7fceac07b280 2019-12-05T05:25:24.296 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:24.293+0000 7fceb12b5700 1 --2- 172.21.15.42:0/3905500267 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fceac07b280 0x7fceac1335c0 secure :-1 s=THROTTLE_DONE pgs=813 cs=0 l=1 rx=0x7fcea4008b10 tx=0x7fcea4010900).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:24.296 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:24.293+0000 7fcea97fa700 1 -- 172.21.15.42:0/3905500267 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fcea40321c0 con 0x7fceac07b280 2019-12-05T05:25:24.460 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:24.457+0000 7fceb2517700 1 -- 172.21.15.42:0/3905500267 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fceac05d110 con 0x7fceac07b280 2019-12-05T05:25:24.461 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:24.457+0000 7fceb12b5700 1 --2- 172.21.15.42:0/3905500267 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fceac07b280 0x7fceac1335c0 secure :-1 s=THROTTLE_DONE pgs=813 cs=0 l=1 rx=0x7fcea4008b10 tx=0x7fcea4010900).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:24.461 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:24.457+0000 7fcea97fa700 1 -- 172.21.15.42:0/3905500267 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 103 v103) v1 ==== 95+0+2111 (secure 0 0 0) 0x7fcea4032d70 con 0x7fceac07b280 2019-12-05T05:25:24.463 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 103 2019-12-05T05:25:24.463 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:25:24.463 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":103,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5240,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/1455108684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1455108684},{"type":"v1","addr":"172.21.15.42:6835","nonce":1455108684}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":103}],"filesystems":[{"mdsmap":{"epoch":102,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:23:58.091182+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5177":{"gid":5177,"name":"b","rank":0,"incarnation":99,"state":"up:active","state_seq":14,"addr":"172.21.15.40:6833/3208416000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3208416000},{"type":"v1","addr":"172.21.15.40:6833","nonce":3208416000}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:25:24.465 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:24.461+0000 7fceb2517700 1 -- 172.21.15.42:0/3905500267 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fce9003b2a0 msgr2=0x7fce9003d7b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:25:24.466 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:24.461+0000 7fceb2517700 1 --2- 172.21.15.42:0/3905500267 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fce9003b2a0 0x7fce9003d7b0 crc :-1 s=READY pgs=359 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:25:24.466 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:24.461+0000 7fceb2517700 1 -- 172.21.15.42:0/3905500267 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fceac07b280 msgr2=0x7fceac1335c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:25:24.466 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:24.461+0000 7fceb2517700 1 --2- 172.21.15.42:0/3905500267 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fceac07b280 0x7fceac1335c0 secure :-1 s=READY pgs=813 cs=0 l=1 rx=0x7fcea4008b10 tx=0x7fcea4010900).stop 2019-12-05T05:25:24.466 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:24.461+0000 7fceb2517700 1 -- 172.21.15.42:0/3905500267 shutdown_connections 2019-12-05T05:25:24.466 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:24.461+0000 7fceb2517700 1 --2- 172.21.15.42:0/3905500267 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fce9003b2a0 0x7fce9003d7b0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:25:24.466 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:24.461+0000 7fceb2517700 1 --2- 172.21.15.42:0/3905500267 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fceac07b280 0x7fceac1335c0 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:25:24.466 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:24.461+0000 7fceb2517700 1 --2- 172.21.15.42:0/3905500267 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fceac0a5eb0 0x7fceac07ad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:25:24.467 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:24.461+0000 7fceb2517700 1 -- 172.21.15.42:0/3905500267 shutdown_connections 2019-12-05T05:25:24.467 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:24.461+0000 7fceb2517700 1 -- 172.21.15.42:0/3905500267 wait complete. 2019-12-05T05:25:24.467 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:24.461+0000 7fceb2517700 1 -- 172.21.15.42:0/3905500267 >> 172.21.15.42:0/3905500267 conn(0x7fceac11c6b0 msgr2=0x7fceac09f380 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:25:24.476 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T05:25:26.480 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:25:26.598 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:26.589+0000 7f39a921e700 1 Processor -- start 2019-12-05T05:25:26.598 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:26.589+0000 7f39a921e700 1 -- start start 2019-12-05T05:25:26.599 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:26.589+0000 7f39a921e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f39a40a7740 0x7f39a408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:25:26.599 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:26.589+0000 7f39a921e700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f39a409fae0 0x7f39a407eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:25:26.599 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:26.589+0000 7f39a921e700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39a40a0710 con 0x7f39a407e720 2019-12-05T05:25:26.599 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:26.589+0000 7f39a921e700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39a40a0b00 con 0x7f39a409fae0 2019-12-05T05:25:26.599 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:26.589+0000 7f39a921e700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39a408e5b0 con 0x7f39a40a7740 2019-12-05T05:25:26.600 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:26.589+0000 7f39a359e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f39a40a7740 0x7f39a408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:25:26.600 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:26.589+0000 7f39a359e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f39a40a7740 0x7f39a408df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:33298/0 (socket says 172.21.15.42:33298) 2019-12-05T05:25:26.600 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:26.589+0000 7f39a359e700 1 -- 172.21.15.42:0/4120433440 learned_addr learned my addr 172.21.15.42:0/4120433440 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:25:26.600 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:26.589+0000 7f39a8a1d700 1 --2- 172.21.15.42:0/4120433440 >> v2:172.21.15.40:3300/0 conn(0x7f39a409fae0 0x7f39a407eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:25:26.600 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:26.589+0000 7f39a359e700 1 -- 172.21.15.42:0/4120433440 >> v2:172.21.15.40:3300/0 conn(0x7f39a409fae0 msgr2=0x7f39a407eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:25:26.600 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:26.589+0000 7f39a359e700 1 --2- 172.21.15.42:0/4120433440 >> v2:172.21.15.40:3300/0 conn(0x7f39a409fae0 0x7f39a407eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:25:26.601 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:26.589+0000 7f39a359e700 1 -- 172.21.15.42:0/4120433440 >> v1:172.21.15.40:6789/0 conn(0x7f39a407e720 legacy=0x7f39a409f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:25:26.601 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:26.589+0000 7f39a359e700 1 -- 172.21.15.42:0/4120433440 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39a407f0a0 con 0x7f39a40a7740 2019-12-05T05:25:26.601 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:26.589+0000 7f39a259c700 1 -- 172.21.15.42:0/4120433440 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3173130130 0 0) 0x7f39a40a0710 con 0x7f39a407e720 2019-12-05T05:25:26.601 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:26.589+0000 7f39a8a1d700 1 --2- 172.21.15.42:0/4120433440 >> v2:172.21.15.40:3300/0 conn(0x7f39a409fae0 0x7f39a407eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T05:25:26.601 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:26.589+0000 7f39a359e700 1 --2- 172.21.15.42:0/4120433440 >> v2:172.21.15.42:3300/0 conn(0x7f39a40a7740 0x7f39a408df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f398c002f30 tx=0x7f398c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:26.601 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:26.589+0000 7f39a359e700 1 --2- 172.21.15.42:0/4120433440 >> v2:172.21.15.42:3300/0 conn(0x7f39a40a7740 0x7f39a408df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f398c002f30 tx=0x7f398c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:26.601 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:26.589+0000 7f39a359e700 1 --2- 172.21.15.42:0/4120433440 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f39a40a7740 0x7f39a408df80 secure :-1 s=READY pgs=814 cs=0 l=1 rx=0x7f398c002f30 tx=0x7f398c00fd30).ready entity=mon.1 client_cookie=58d7898ef9b35210 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:25:26.602 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:26.589+0000 7f39a359e700 1 --2- 172.21.15.42:0/4120433440 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f39a40a7740 0x7f39a408df80 secure :-1 s=THROTTLE_DONE pgs=814 cs=0 l=1 rx=0x7f398c002f30 tx=0x7f398c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:26.602 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:26.589+0000 7f39a259c700 1 -- 172.21.15.42:0/4120433440 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f398c008c90 con 0x7f39a40a7740 2019-12-05T05:25:26.602 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:26.589+0000 7f39a359e700 1 --2- 172.21.15.42:0/4120433440 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f39a40a7740 0x7f39a408df80 secure :-1 s=THROTTLE_DONE pgs=814 cs=0 l=1 rx=0x7f398c002f30 tx=0x7f398c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:26.602 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:26.589+0000 7f39a359e700 1 --2- 172.21.15.42:0/4120433440 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f39a40a7740 0x7f39a408df80 secure :-1 s=THROTTLE_DONE pgs=814 cs=0 l=1 rx=0x7f398c002f30 tx=0x7f398c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:26.602 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:26.589+0000 7f39a259c700 1 -- 172.21.15.42:0/4120433440 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f398c0088b0 con 0x7f39a40a7740 2019-12-05T05:25:26.602 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:26.593+0000 7f39a259c700 1 -- 172.21.15.42:0/4120433440 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f398c011ad0 con 0x7f39a40a7740 2019-12-05T05:25:26.603 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:26.593+0000 7f39a921e700 1 -- 172.21.15.42:0/4120433440 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f39a40a7740 msgr2=0x7f39a408df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:25:26.603 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:26.593+0000 7f39a921e700 1 --2- 172.21.15.42:0/4120433440 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f39a40a7740 0x7f39a408df80 secure :-1 s=READY pgs=814 cs=0 l=1 rx=0x7f398c002f30 tx=0x7f398c00fd30).stop 2019-12-05T05:25:26.603 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:26.593+0000 7f39a921e700 1 -- 172.21.15.42:0/4120433440 shutdown_connections 2019-12-05T05:25:26.603 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:26.593+0000 7f39a921e700 1 --2- 172.21.15.42:0/4120433440 >> v2:172.21.15.40:3300/0 conn(0x7f39a409fae0 0x7f39a407eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:25:26.603 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:26.593+0000 7f39a921e700 1 --2- 172.21.15.42:0/4120433440 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f39a40a7740 0x7f39a408df80 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:25:26.603 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:26.593+0000 7f39a921e700 1 -- 172.21.15.42:0/4120433440 shutdown_connections 2019-12-05T05:25:26.603 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:26.593+0000 7f39a921e700 1 -- 172.21.15.42:0/4120433440 wait complete. 2019-12-05T05:25:26.603 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:26.593+0000 7f39a921e700 1 -- 172.21.15.42:0/4120433440 >> 172.21.15.42:0/4120433440 conn(0x7f39a40aabe0 msgr2=0x7f39a40a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:25:26.607 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:26.593+0000 7f39a921e700 1 Processor -- start 2019-12-05T05:25:26.607 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:26.593+0000 7f39a921e700 1 -- start start 2019-12-05T05:25:26.607 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:26.597+0000 7f39a921e700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f39a41243f0 0x7f39a411f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:25:26.607 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:26.597+0000 7f39a921e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f39a4121650 0x7f39a40762f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:25:26.607 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:26.597+0000 7f39a921e700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39a41202b0 con 0x7f39a4121220 2019-12-05T05:25:26.607 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:26.597+0000 7f39a921e700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39a4121a80 con 0x7f39a41243f0 2019-12-05T05:25:26.608 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:26.597+0000 7f39a921e700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39a4119960 con 0x7f39a4121650 2019-12-05T05:25:26.608 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:26.597+0000 7f39a2d9d700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f39a4121220 0x7f39a4119160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:33342/0 (socket says 172.21.15.42:33342) 2019-12-05T05:25:26.608 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:26.597+0000 7f39a2d9d700 1 -- 172.21.15.42:0/3656586059 learned_addr learned my addr 172.21.15.42:0/3656586059 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:25:26.608 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:26.597+0000 7f39a8a1d700 1 --2- 172.21.15.42:0/3656586059 >> v2:172.21.15.42:3300/0 conn(0x7f39a4121650 0x7f39a40762f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:25:26.608 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:26.597+0000 7f39a359e700 1 --2- 172.21.15.42:0/3656586059 >> v2:172.21.15.40:3300/0 conn(0x7f39a41243f0 0x7f39a411f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:25:26.609 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:26.597+0000 7f39a8a1d700 1 -- 172.21.15.42:0/3656586059 >> v2:172.21.15.40:3300/0 conn(0x7f39a41243f0 msgr2=0x7f39a411f8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:25:26.609 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:26.597+0000 7f399bfff700 1 -- 172.21.15.42:0/3656586059 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2063432493 0 0) 0x7f39a41202b0 con 0x7f39a4121220 2019-12-05T05:25:26.609 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:26.597+0000 7f39a8a1d700 1 --2- 172.21.15.42:0/3656586059 >> v2:172.21.15.40:3300/0 conn(0x7f39a41243f0 0x7f39a411f8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:25:26.609 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:26.597+0000 7f39a8a1d700 1 -- 172.21.15.42:0/3656586059 >> v1:172.21.15.42:6789/0 conn(0x7f39a4121220 legacy=0x7f39a4119160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:25:26.609 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:26.597+0000 7f39a8a1d700 1 -- 172.21.15.42:0/3656586059 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39a411fe00 con 0x7f39a4121650 2019-12-05T05:25:26.609 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:26.597+0000 7f39a359e700 1 --2- 172.21.15.42:0/3656586059 >> v2:172.21.15.40:3300/0 conn(0x7f39a41243f0 0x7f39a411f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:25:26.609 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:26.597+0000 7f39a8a1d700 1 --2- 172.21.15.42:0/3656586059 >> v2:172.21.15.42:3300/0 conn(0x7f39a4121650 0x7f39a40762f0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f399c002f30 tx=0x7f399c010880).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:26.610 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:26.597+0000 7f39a8a1d700 1 --2- 172.21.15.42:0/3656586059 >> v2:172.21.15.42:3300/0 conn(0x7f39a4121650 0x7f39a40762f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f399c002f30 tx=0x7f399c010880).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:26.610 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:26.597+0000 7f39a8a1d700 1 --2- 172.21.15.42:0/3656586059 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f39a4121650 0x7f39a40762f0 secure :-1 s=READY pgs=816 cs=0 l=1 rx=0x7f399c002f30 tx=0x7f399c010880).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:25:26.610 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:26.597+0000 7f39a8a1d700 1 --2- 172.21.15.42:0/3656586059 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f39a4121650 0x7f39a40762f0 secure :-1 s=THROTTLE_DONE pgs=816 cs=0 l=1 rx=0x7f399c002f30 tx=0x7f399c010880).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:26.610 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:26.597+0000 7f399bfff700 1 -- 172.21.15.42:0/3656586059 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f399c005b20 con 0x7f39a4121650 2019-12-05T05:25:26.610 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:26.597+0000 7f39a8a1d700 1 --2- 172.21.15.42:0/3656586059 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f39a4121650 0x7f39a40762f0 secure :-1 s=THROTTLE_DONE pgs=816 cs=0 l=1 rx=0x7f399c002f30 tx=0x7f399c010880).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:26.610 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:26.597+0000 7f399bfff700 1 -- 172.21.15.42:0/3656586059 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f399c007200 con 0x7f39a4121650 2019-12-05T05:25:26.610 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:26.597+0000 7f39a8a1d700 1 --2- 172.21.15.42:0/3656586059 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f39a4121650 0x7f39a40762f0 secure :-1 s=THROTTLE_DONE pgs=816 cs=0 l=1 rx=0x7f399c002f30 tx=0x7f399c010880).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:26.611 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:26.597+0000 7f39a921e700 1 -- 172.21.15.42:0/3656586059 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39a4076840 con 0x7f39a4121650 2019-12-05T05:25:26.611 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:26.597+0000 7f399bfff700 1 -- 172.21.15.42:0/3656586059 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f399c0062b0 con 0x7f39a4121650 2019-12-05T05:25:26.611 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:26.597+0000 7f39a921e700 1 -- 172.21.15.42:0/3656586059 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39a4076c20 con 0x7f39a4121650 2019-12-05T05:25:26.611 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:26.597+0000 7f39a8a1d700 1 --2- 172.21.15.42:0/3656586059 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f39a4121650 0x7f39a40762f0 secure :-1 s=THROTTLE_DONE pgs=816 cs=0 l=1 rx=0x7f399c002f30 tx=0x7f399c010880).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:26.611 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:26.601+0000 7f399bfff700 1 -- 172.21.15.42:0/3656586059 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f399c007bb0 con 0x7f39a4121650 2019-12-05T05:25:26.611 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:26.601+0000 7f39a8a1d700 1 --2- 172.21.15.42:0/3656586059 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f39a4121650 0x7f39a40762f0 secure :-1 s=THROTTLE_DONE pgs=816 cs=0 l=1 rx=0x7f399c002f30 tx=0x7f399c010880).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:26.611 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:26.601+0000 7f399bfff700 1 --2- 172.21.15.42:0/3656586059 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f397c03ac20 0x7f397c03d130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:25:26.611 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:26.601+0000 7f399bfff700 1 -- 172.21.15.42:0/3656586059 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 7235+0+0 (secure 0 0 0) 0x7f399c04acb0 con 0x7f39a4121650 2019-12-05T05:25:26.612 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:26.601+0000 7f39a359e700 1 --2- 172.21.15.42:0/3656586059 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f397c03ac20 0x7f397c03d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:25:26.612 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:26.601+0000 7f39a359e700 1 --2- 172.21.15.42:0/3656586059 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f397c03ac20 0x7f397c03d130 crc :-1 s=READY pgs=360 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:25:26.612 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:26.605+0000 7f39a921e700 1 -- 172.21.15.42:0/3656586059 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39a405d110 con 0x7f39a4121650 2019-12-05T05:25:26.618 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:26.613+0000 7f39a8a1d700 1 --2- 172.21.15.42:0/3656586059 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f39a4121650 0x7f39a40762f0 secure :-1 s=THROTTLE_DONE pgs=816 cs=0 l=1 rx=0x7f399c002f30 tx=0x7f399c010880).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:26.618 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:26.613+0000 7f399bfff700 1 -- 172.21.15.42:0/3656586059 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f399c031d10 con 0x7f39a4121650 2019-12-05T05:25:26.784 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:26.781+0000 7f39a921e700 1 -- 172.21.15.42:0/3656586059 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f39a405b8c0 con 0x7f39a4121650 2019-12-05T05:25:26.785 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:26.781+0000 7f39a8a1d700 1 --2- 172.21.15.42:0/3656586059 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f39a4121650 0x7f39a40762f0 secure :-1 s=THROTTLE_DONE pgs=816 cs=0 l=1 rx=0x7f399c002f30 tx=0x7f399c010880).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:26.785 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:26.781+0000 7f399bfff700 1 -- 172.21.15.42:0/3656586059 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 103 v103) v1 ==== 95+0+2111 (secure 0 0 0) 0x7f399c031f70 con 0x7f39a4121650 2019-12-05T05:25:26.785 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 103 2019-12-05T05:25:26.786 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:25:26.786 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":103,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5240,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/1455108684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1455108684},{"type":"v1","addr":"172.21.15.42:6835","nonce":1455108684}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":103}],"filesystems":[{"mdsmap":{"epoch":102,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:23:58.091182+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5177":{"gid":5177,"name":"b","rank":0,"incarnation":99,"state":"up:active","state_seq":14,"addr":"172.21.15.40:6833/3208416000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3208416000},{"type":"v1","addr":"172.21.15.40:6833","nonce":3208416000}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:25:26.788 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:26.781+0000 7f39a921e700 1 -- 172.21.15.42:0/3656586059 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f397c03ac20 msgr2=0x7f397c03d130 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:25:26.788 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:26.781+0000 7f39a921e700 1 --2- 172.21.15.42:0/3656586059 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f397c03ac20 0x7f397c03d130 crc :-1 s=READY pgs=360 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:25:26.788 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:26.781+0000 7f39a921e700 1 -- 172.21.15.42:0/3656586059 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f39a4121650 msgr2=0x7f39a40762f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:25:26.788 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:26.781+0000 7f39a921e700 1 --2- 172.21.15.42:0/3656586059 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f39a4121650 0x7f39a40762f0 secure :-1 s=READY pgs=816 cs=0 l=1 rx=0x7f399c002f30 tx=0x7f399c010880).stop 2019-12-05T05:25:26.788 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:26.781+0000 7f39a921e700 1 -- 172.21.15.42:0/3656586059 shutdown_connections 2019-12-05T05:25:26.788 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:26.785+0000 7f39a921e700 1 --2- 172.21.15.42:0/3656586059 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f397c03ac20 0x7f397c03d130 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:25:26.789 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:26.785+0000 7f39a921e700 1 --2- 172.21.15.42:0/3656586059 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f39a4121650 0x7f39a40762f0 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:25:26.789 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:26.785+0000 7f39a921e700 1 --2- 172.21.15.42:0/3656586059 >> v2:172.21.15.40:3300/0 conn(0x7f39a41243f0 0x7f39a411f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:25:26.789 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:26.785+0000 7f39a921e700 1 -- 172.21.15.42:0/3656586059 shutdown_connections 2019-12-05T05:25:26.789 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:26.785+0000 7f39a921e700 1 -- 172.21.15.42:0/3656586059 wait complete. 2019-12-05T05:25:26.789 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:26.785+0000 7f39a921e700 1 -- 172.21.15.42:0/3656586059 >> 172.21.15.42:0/3656586059 conn(0x7f39a40a7740 msgr2=0x7f39a409f240 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:25:26.801 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T05:25:28.804 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:25:28.910 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:28.901+0000 7fd7001ad700 1 Processor -- start 2019-12-05T05:25:28.911 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:28.901+0000 7fd7001ad700 1 -- start start 2019-12-05T05:25:28.911 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:28.901+0000 7fd7001ad700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd6f80a7740 0x7fd6f808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:25:28.911 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:28.901+0000 7fd7001ad700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd6f809fae0 0x7fd6f807eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:25:28.911 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:28.901+0000 7fd7001ad700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6f80a0710 con 0x7fd6f807e720 2019-12-05T05:25:28.912 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:28.901+0000 7fd7001ad700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6f80a0b00 con 0x7fd6f809fae0 2019-12-05T05:25:28.912 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:28.905+0000 7fd7001ad700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6f808e5b0 con 0x7fd6f80a7740 2019-12-05T05:25:28.912 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:28.905+0000 7fd6fef4b700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd6f809fae0 0x7fd6f807eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:25:28.912 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:28.905+0000 7fd6fef4b700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd6f809fae0 0x7fd6f807eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:33322/0 (socket says 172.21.15.42:33322) 2019-12-05T05:25:28.913 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:28.905+0000 7fd6fef4b700 1 -- 172.21.15.42:0/4055975500 learned_addr learned my addr 172.21.15.42:0/4055975500 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:25:28.913 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:28.905+0000 7fd6fe74a700 1 --2- 172.21.15.42:0/4055975500 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd6f80a7740 0x7fd6f808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:25:28.913 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:28.905+0000 7fd6fef4b700 1 -- 172.21.15.42:0/4055975500 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd6f80a7740 msgr2=0x7fd6f808df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:25:28.913 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:28.905+0000 7fd6fef4b700 1 --2- 172.21.15.42:0/4055975500 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd6f80a7740 0x7fd6f808df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:25:28.913 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:28.905+0000 7fd6fef4b700 1 -- 172.21.15.42:0/4055975500 >> v1:172.21.15.40:6789/0 conn(0x7fd6f807e720 legacy=0x7fd6f809f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:25:28.913 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:28.905+0000 7fd6fef4b700 1 -- 172.21.15.42:0/4055975500 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6f807f0a0 con 0x7fd6f809fae0 2019-12-05T05:25:28.914 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:28.905+0000 7fd6fef4b700 1 --2- 172.21.15.42:0/4055975500 >> v2:172.21.15.42:3300/0 conn(0x7fd6f809fae0 0x7fd6f807eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fd6f0008790 tx=0x7fd6f000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:28.914 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:28.905+0000 7fd6fef4b700 1 --2- 172.21.15.42:0/4055975500 >> v2:172.21.15.42:3300/0 conn(0x7fd6f809fae0 0x7fd6f807eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fd6f0008790 tx=0x7fd6f000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:28.914 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:28.905+0000 7fd6fef4b700 1 --2- 172.21.15.42:0/4055975500 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd6f809fae0 0x7fd6f807eb50 secure :-1 s=READY pgs=817 cs=0 l=1 rx=0x7fd6f0008790 tx=0x7fd6f000fd30).ready entity=mon.1 client_cookie=c8875922b9bfc9a7 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:25:28.914 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:28.905+0000 7fd6fef4b700 1 --2- 172.21.15.42:0/4055975500 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd6f809fae0 0x7fd6f807eb50 secure :-1 s=THROTTLE_DONE pgs=817 cs=0 l=1 rx=0x7fd6f0008790 tx=0x7fd6f000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:28.914 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:28.905+0000 7fd6fd748700 1 -- 172.21.15.42:0/4055975500 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd6f0008c90 con 0x7fd6f809fae0 2019-12-05T05:25:28.915 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:28.905+0000 7fd6fef4b700 1 --2- 172.21.15.42:0/4055975500 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd6f809fae0 0x7fd6f807eb50 secure :-1 s=THROTTLE_DONE pgs=817 cs=0 l=1 rx=0x7fd6f0008790 tx=0x7fd6f000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:28.915 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:28.905+0000 7fd6fef4b700 1 --2- 172.21.15.42:0/4055975500 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd6f809fae0 0x7fd6f807eb50 secure :-1 s=THROTTLE_DONE pgs=817 cs=0 l=1 rx=0x7fd6f0008790 tx=0x7fd6f000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:28.915 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:28.905+0000 7fd6fd748700 1 -- 172.21.15.42:0/4055975500 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6f00088b0 con 0x7fd6f809fae0 2019-12-05T05:25:28.915 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:28.905+0000 7fd6fd748700 1 -- 172.21.15.42:0/4055975500 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd6f0011b10 con 0x7fd6f809fae0 2019-12-05T05:25:28.915 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:28.905+0000 7fd7001ad700 1 -- 172.21.15.42:0/4055975500 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd6f809fae0 msgr2=0x7fd6f807eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:25:28.915 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:28.905+0000 7fd7001ad700 1 --2- 172.21.15.42:0/4055975500 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd6f809fae0 0x7fd6f807eb50 secure :-1 s=READY pgs=817 cs=0 l=1 rx=0x7fd6f0008790 tx=0x7fd6f000fd30).stop 2019-12-05T05:25:28.916 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:28.905+0000 7fd7001ad700 1 -- 172.21.15.42:0/4055975500 shutdown_connections 2019-12-05T05:25:28.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:28.905+0000 7fd7001ad700 1 --2- 172.21.15.42:0/4055975500 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd6f809fae0 0x7fd6f807eb50 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:25:28.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:28.905+0000 7fd7001ad700 1 --2- 172.21.15.42:0/4055975500 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd6f80a7740 0x7fd6f808df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:25:28.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:28.905+0000 7fd7001ad700 1 -- 172.21.15.42:0/4055975500 shutdown_connections 2019-12-05T05:25:28.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:28.905+0000 7fd7001ad700 1 -- 172.21.15.42:0/4055975500 wait complete. 2019-12-05T05:25:28.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:28.905+0000 7fd7001ad700 1 -- 172.21.15.42:0/4055975500 >> 172.21.15.42:0/4055975500 conn(0x7fd6f80aabe0 msgr2=0x7fd6f80a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:25:28.921 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:28.909+0000 7fd7001ad700 1 Processor -- start 2019-12-05T05:25:28.922 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:28.909+0000 7fd7001ad700 1 -- start start 2019-12-05T05:25:28.922 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:28.909+0000 7fd7001ad700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd6f8122af0 0x7fd6f8122f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:25:28.922 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:28.909+0000 7fd7001ad700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6f8121600 con 0x7fd6f80752f0 2019-12-05T05:25:28.922 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:28.909+0000 7fd7001ad700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6f811b000 con 0x7fd6f80a9930 2019-12-05T05:25:28.922 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:28.909+0000 7fd7001ad700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6f80a9d60 con 0x7fd6f8122af0 2019-12-05T05:25:28.923 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:28.909+0000 7fd6fe74a700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fd6f80a9930 0x7fd6f811a800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:33362/0 (socket says 172.21.15.42:33362) 2019-12-05T05:25:28.923 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:28.909+0000 7fd6fe74a700 1 -- 172.21.15.42:0/3949427572 learned_addr learned my addr 172.21.15.42:0/3949427572 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:25:28.923 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:28.909+0000 7fd6fdf49700 1 --2- 172.21.15.42:0/3949427572 >> v2:172.21.15.40:3300/0 conn(0x7fd6f8122af0 0x7fd6f8122f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:25:28.923 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:28.909+0000 7fd6eeffd700 1 -- 172.21.15.42:0/3949427572 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 255628053 0 0) 0x7fd6f811b000 con 0x7fd6f80a9930 2019-12-05T05:25:28.923 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:28.909+0000 7fd6eeffd700 1 -- 172.21.15.42:0/3949427572 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6e00030b0 con 0x7fd6f80a9930 2019-12-05T05:25:28.924 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:28.909+0000 7fd6eeffd700 1 -- 172.21.15.42:0/3949427572 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 2779925059 0 0) 0x7fd6e00030b0 con 0x7fd6f80a9930 2019-12-05T05:25:28.924 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:28.909+0000 7fd6eeffd700 1 -- 172.21.15.42:0/3949427572 --> v1:172.21.15.42:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd6f811b000 con 0x7fd6f80a9930 2019-12-05T05:25:28.924 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:28.909+0000 7fd6eeffd700 1 -- 172.21.15.42:0/3949427572 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7fd6f4001810 con 0x7fd6f80a9930 2019-12-05T05:25:28.924 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:28.913+0000 7fd6fdf49700 1 -- 172.21.15.42:0/3949427572 >> v1:172.21.15.42:6789/0 conn(0x7fd6f80a9930 legacy=0x7fd6f811a800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:25:28.924 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:28.913+0000 7fd6fdf49700 1 -- 172.21.15.42:0/3949427572 >> v1:172.21.15.40:6789/0 conn(0x7fd6f80752f0 legacy=0x7fd6f8077730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:25:28.925 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:28.913+0000 7fd6fdf49700 1 -- 172.21.15.42:0/3949427572 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6f80a1d00 con 0x7fd6f8122af0 2019-12-05T05:25:28.925 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:28.913+0000 7fd6fdf49700 1 --2- 172.21.15.42:0/3949427572 >> v2:172.21.15.40:3300/0 conn(0x7fd6f8122af0 0x7fd6f8122f20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fd6e8001430 tx=0x7fd6f000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:28.925 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:28.913+0000 7fd6fdf49700 1 --2- 172.21.15.42:0/3949427572 >> v2:172.21.15.40:3300/0 conn(0x7fd6f8122af0 0x7fd6f8122f20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fd6e8001430 tx=0x7fd6f000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:28.925 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:28.913+0000 7fd6fdf49700 1 --2- 172.21.15.42:0/3949427572 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd6f8122af0 0x7fd6f8122f20 secure :-1 s=READY pgs=653 cs=0 l=1 rx=0x7fd6e8001430 tx=0x7fd6f000fd30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:25:28.925 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:28.913+0000 7fd6fdf49700 1 --2- 172.21.15.42:0/3949427572 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd6f8122af0 0x7fd6f8122f20 secure :-1 s=THROTTLE_DONE pgs=653 cs=0 l=1 rx=0x7fd6e8001430 tx=0x7fd6f000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:28.926 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:28.913+0000 7fd6eeffd700 1 -- 172.21.15.42:0/3949427572 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd6e800c960 con 0x7fd6f8122af0 2019-12-05T05:25:28.926 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:28.913+0000 7fd6fdf49700 1 --2- 172.21.15.42:0/3949427572 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd6f8122af0 0x7fd6f8122f20 secure :-1 s=THROTTLE_DONE pgs=653 cs=0 l=1 rx=0x7fd6e8001430 tx=0x7fd6f000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:28.926 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:28.913+0000 7fd7001ad700 1 -- 172.21.15.42:0/3949427572 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6f8078f40 con 0x7fd6f8122af0 2019-12-05T05:25:28.927 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:28.913+0000 7fd6eeffd700 1 -- 172.21.15.42:0/3949427572 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6e800c580 con 0x7fd6f8122af0 2019-12-05T05:25:28.927 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:28.913+0000 7fd6fdf49700 1 --2- 172.21.15.42:0/3949427572 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd6f8122af0 0x7fd6f8122f20 secure :-1 s=THROTTLE_DONE pgs=653 cs=0 l=1 rx=0x7fd6e8001430 tx=0x7fd6f000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:28.927 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:28.913+0000 7fd6eeffd700 1 -- 172.21.15.42:0/3949427572 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd6e8010d00 con 0x7fd6f8122af0 2019-12-05T05:25:28.927 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:28.913+0000 7fd7001ad700 1 -- 172.21.15.42:0/3949427572 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6f8079410 con 0x7fd6f8122af0 2019-12-05T05:25:28.927 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:28.913+0000 7fd6fdf49700 1 --2- 172.21.15.42:0/3949427572 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd6f8122af0 0x7fd6f8122f20 secure :-1 s=THROTTLE_DONE pgs=653 cs=0 l=1 rx=0x7fd6e8001430 tx=0x7fd6f000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:28.928 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:28.913+0000 7fd6eeffd700 1 -- 172.21.15.42:0/3949427572 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fd6e8007430 con 0x7fd6f8122af0 2019-12-05T05:25:28.928 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:28.913+0000 7fd6fdf49700 1 --2- 172.21.15.42:0/3949427572 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd6f8122af0 0x7fd6f8122f20 secure :-1 s=THROTTLE_DONE pgs=653 cs=0 l=1 rx=0x7fd6e8001430 tx=0x7fd6f000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:28.928 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:28.913+0000 7fd6eeffd700 1 --2- 172.21.15.42:0/3949427572 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd6e003b620 0x7fd6e003db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:25:28.929 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:28.913+0000 7fd6eeffd700 1 -- 172.21.15.42:0/3949427572 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 7235+0+0 (secure 0 0 0) 0x7fd6e804af30 con 0x7fd6f8122af0 2019-12-05T05:25:28.929 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:28.913+0000 7fd6fe74a700 1 --2- 172.21.15.42:0/3949427572 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd6e003b620 0x7fd6e003db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:25:28.930 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:28.913+0000 7fd6fe74a700 1 --2- 172.21.15.42:0/3949427572 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd6e003b620 0x7fd6e003db30 crc :-1 s=READY pgs=361 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:25:28.930 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:28.917+0000 7fd7001ad700 1 -- 172.21.15.42:0/3949427572 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6f805d110 con 0x7fd6f8122af0 2019-12-05T05:25:28.931 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:28.925+0000 7fd6fdf49700 1 --2- 172.21.15.42:0/3949427572 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd6f8122af0 0x7fd6f8122f20 secure :-1 s=THROTTLE_DONE pgs=653 cs=0 l=1 rx=0x7fd6e8001430 tx=0x7fd6f000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:28.931 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:28.925+0000 7fd6eeffd700 1 -- 172.21.15.42:0/3949427572 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fd6e8031f10 con 0x7fd6f8122af0 2019-12-05T05:25:29.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:29.097+0000 7fd7001ad700 1 -- 172.21.15.42:0/3949427572 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd6f805b8c0 con 0x7fd6f8122af0 2019-12-05T05:25:29.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:29.097+0000 7fd6fdf49700 1 --2- 172.21.15.42:0/3949427572 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd6f8122af0 0x7fd6f8122f20 secure :-1 s=THROTTLE_DONE pgs=653 cs=0 l=1 rx=0x7fd6e8001430 tx=0x7fd6f000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:29.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:29.097+0000 7fd6eeffd700 1 -- 172.21.15.42:0/3949427572 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 103 v103) v1 ==== 95+0+2111 (secure 0 0 0) 0x7fd6e8032170 con 0x7fd6f8122af0 2019-12-05T05:25:29.104 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:25:29.104 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":103,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5240,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/1455108684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1455108684},{"type":"v1","addr":"172.21.15.42:6835","nonce":1455108684}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":103}],"filesystems":[{"mdsmap":{"epoch":102,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:23:58.091182+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5177":{"gid":5177,"name":"b","rank":0,"incarnation":99,"state":"up:active","state_seq":14,"addr":"172.21.15.40:6833/3208416000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3208416000},{"type":"v1","addr":"172.21.15.40:6833","nonce":3208416000}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:25:29.104 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 103 2019-12-05T05:25:29.106 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:29.101+0000 7fd7001ad700 1 -- 172.21.15.42:0/3949427572 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd6e003b620 msgr2=0x7fd6e003db30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:25:29.107 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:29.101+0000 7fd7001ad700 1 --2- 172.21.15.42:0/3949427572 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd6e003b620 0x7fd6e003db30 crc :-1 s=READY pgs=361 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:25:29.107 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:29.101+0000 7fd7001ad700 1 -- 172.21.15.42:0/3949427572 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd6f8122af0 msgr2=0x7fd6f8122f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:25:29.107 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:29.101+0000 7fd7001ad700 1 --2- 172.21.15.42:0/3949427572 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd6f8122af0 0x7fd6f8122f20 secure :-1 s=READY pgs=653 cs=0 l=1 rx=0x7fd6e8001430 tx=0x7fd6f000fd30).stop 2019-12-05T05:25:29.107 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:29.101+0000 7fd7001ad700 1 -- 172.21.15.42:0/3949427572 shutdown_connections 2019-12-05T05:25:29.107 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:29.101+0000 7fd7001ad700 1 --2- 172.21.15.42:0/3949427572 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd6e003b620 0x7fd6e003db30 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:25:29.107 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:29.101+0000 7fd7001ad700 1 --2- 172.21.15.42:0/3949427572 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd6f8122af0 0x7fd6f8122f20 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:25:29.108 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:29.101+0000 7fd7001ad700 1 -- 172.21.15.42:0/3949427572 shutdown_connections 2019-12-05T05:25:29.109 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:29.101+0000 7fd7001ad700 1 -- 172.21.15.42:0/3949427572 wait complete. 2019-12-05T05:25:29.109 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:29.101+0000 7fd7001ad700 1 -- 172.21.15.42:0/3949427572 >> 172.21.15.42:0/3949427572 conn(0x7fd6f80a82a0 msgr2=0x7fd6f8059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:25:29.117 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T05:25:31.118 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:25:31.233 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:31.221+0000 7fbe75189700 1 Processor -- start 2019-12-05T05:25:31.233 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:31.221+0000 7fbe75189700 1 -- start start 2019-12-05T05:25:31.234 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:31.225+0000 7fbe75189700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbe7008df70 0x7fbe7009f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:25:31.234 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:31.225+0000 7fbe75189700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fbe7009f820 0x7fbe7014efd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:25:31.234 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:31.225+0000 7fbe75189700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe7009fc50 con 0x7fbe700a7730 2019-12-05T05:25:31.234 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:31.225+0000 7fbe75189700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe700a0af0 con 0x7fbe7009f820 2019-12-05T05:25:31.235 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:31.225+0000 7fbe75189700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe7008e5a0 con 0x7fbe7008df70 2019-12-05T05:25:31.235 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:31.225+0000 7fbe74988700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fbe7009f820 0x7fbe7014efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:25:31.235 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:31.225+0000 7fbe6ed9d700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbe7008df70 0x7fbe7009f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:25:31.236 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:31.225+0000 7fbe6ed9d700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbe7008df70 0x7fbe7009f2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:33802/0 (socket says 172.21.15.42:33802) 2019-12-05T05:25:31.236 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:31.225+0000 7fbe74988700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fbe7009f820 0x7fbe7014efd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:33338/0 (socket says 172.21.15.42:33338) 2019-12-05T05:25:31.236 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:31.225+0000 7fbe6ed9d700 1 -- 172.21.15.42:0/3441454335 learned_addr learned my addr 172.21.15.42:0/3441454335 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:25:31.236 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:31.225+0000 7fbe6f59e700 1 --1- 172.21.15.42:0/3441454335 >> v1:172.21.15.40:6789/0 conn(0x7fbe700a7730 0x7fbe7007e710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:42812/0 (socket says 172.21.15.42:42812) 2019-12-05T05:25:31.237 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:31.225+0000 7fbe74988700 1 -- 172.21.15.42:0/3441454335 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbe7008df70 msgr2=0x7fbe7009f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:25:31.237 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:31.225+0000 7fbe74988700 1 --2- 172.21.15.42:0/3441454335 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbe7008df70 0x7fbe7009f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:25:31.237 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:31.225+0000 7fbe74988700 1 -- 172.21.15.42:0/3441454335 >> v1:172.21.15.40:6789/0 conn(0x7fbe700a7730 legacy=0x7fbe7007e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:25:31.238 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:31.225+0000 7fbe6e59c700 1 -- 172.21.15.42:0/3441454335 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2692323697 0 0) 0x7fbe7009fc50 con 0x7fbe700a7730 2019-12-05T05:25:31.238 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:31.225+0000 7fbe74988700 1 -- 172.21.15.42:0/3441454335 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe700a0700 con 0x7fbe7009f820 2019-12-05T05:25:31.238 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:31.225+0000 7fbe6ed9d700 1 --2- 172.21.15.42:0/3441454335 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbe7008df70 0x7fbe7009f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:25:31.238 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:31.225+0000 7fbe74988700 1 --2- 172.21.15.42:0/3441454335 >> v2:172.21.15.42:3300/0 conn(0x7fbe7009f820 0x7fbe7014efd0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fbe68008b10 tx=0x7fbe6800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:31.238 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:31.225+0000 7fbe74988700 1 --2- 172.21.15.42:0/3441454335 >> v2:172.21.15.42:3300/0 conn(0x7fbe7009f820 0x7fbe7014efd0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fbe68008b10 tx=0x7fbe6800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:31.239 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:31.225+0000 7fbe74988700 1 --2- 172.21.15.42:0/3441454335 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbe7009f820 0x7fbe7014efd0 secure :-1 s=READY pgs=819 cs=0 l=1 rx=0x7fbe68008b10 tx=0x7fbe6800fff0).ready entity=mon.1 client_cookie=3afc153e8bbddf5e server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:25:31.239 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:31.225+0000 7fbe74988700 1 --2- 172.21.15.42:0/3441454335 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbe7009f820 0x7fbe7014efd0 secure :-1 s=THROTTLE_DONE pgs=819 cs=0 l=1 rx=0x7fbe68008b10 tx=0x7fbe6800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:31.239 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:31.225+0000 7fbe6e59c700 1 -- 172.21.15.42:0/3441454335 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbe68010900 con 0x7fbe7009f820 2019-12-05T05:25:31.239 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:31.225+0000 7fbe74988700 1 --2- 172.21.15.42:0/3441454335 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbe7009f820 0x7fbe7014efd0 secure :-1 s=THROTTLE_DONE pgs=819 cs=0 l=1 rx=0x7fbe68008b10 tx=0x7fbe6800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:31.240 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:31.225+0000 7fbe6e59c700 1 -- 172.21.15.42:0/3441454335 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe68008b50 con 0x7fbe7009f820 2019-12-05T05:25:31.240 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:31.225+0000 7fbe74988700 1 --2- 172.21.15.42:0/3441454335 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbe7009f820 0x7fbe7014efd0 secure :-1 s=THROTTLE_DONE pgs=819 cs=0 l=1 rx=0x7fbe68008b10 tx=0x7fbe6800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:31.240 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:31.225+0000 7fbe6e59c700 1 -- 172.21.15.42:0/3441454335 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbe680110c0 con 0x7fbe7009f820 2019-12-05T05:25:31.240 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:31.225+0000 7fbe75189700 1 -- 172.21.15.42:0/3441454335 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbe7009f820 msgr2=0x7fbe7014efd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:25:31.241 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:31.225+0000 7fbe75189700 1 --2- 172.21.15.42:0/3441454335 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbe7009f820 0x7fbe7014efd0 secure :-1 s=READY pgs=819 cs=0 l=1 rx=0x7fbe68008b10 tx=0x7fbe6800fff0).stop 2019-12-05T05:25:31.241 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:31.229+0000 7fbe75189700 1 -- 172.21.15.42:0/3441454335 shutdown_connections 2019-12-05T05:25:31.241 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:31.229+0000 7fbe75189700 1 --2- 172.21.15.42:0/3441454335 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbe7009f820 0x7fbe7014efd0 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:25:31.241 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:31.229+0000 7fbe75189700 1 --2- 172.21.15.42:0/3441454335 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbe7008df70 0x7fbe7009f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:25:31.241 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:31.229+0000 7fbe75189700 1 -- 172.21.15.42:0/3441454335 shutdown_connections 2019-12-05T05:25:31.242 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:31.229+0000 7fbe75189700 1 -- 172.21.15.42:0/3441454335 wait complete. 2019-12-05T05:25:31.242 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:31.229+0000 7fbe75189700 1 -- 172.21.15.42:0/3441454335 >> 172.21.15.42:0/3441454335 conn(0x7fbe700aabd0 msgr2=0x7fbe700a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:25:31.247 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:31.229+0000 7fbe75189700 1 Processor -- start 2019-12-05T05:25:31.247 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:31.229+0000 7fbe75189700 1 -- start start 2019-12-05T05:25:31.248 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:31.233+0000 7fbe75189700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fbe700a3710 0x7fbe7008f490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:25:31.248 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:31.233+0000 7fbe75189700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fbe7011f290 0x7fbe70124970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:25:31.248 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:31.233+0000 7fbe75189700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe7011eb90 con 0x7fbe7011ee60 2019-12-05T05:25:31.248 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:31.233+0000 7fbe75189700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe70119c50 con 0x7fbe7011f290 2019-12-05T05:25:31.249 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:31.233+0000 7fbe75189700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe70119ef0 con 0x7fbe700a3710 2019-12-05T05:25:31.249 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:31.233+0000 7fbe6f59e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fbe700a3710 0x7fbe7008f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:25:31.249 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:31.233+0000 7fbe6f59e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fbe700a3710 0x7fbe7008f490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:33340/0 (socket says 172.21.15.42:33340) 2019-12-05T05:25:31.249 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:31.233+0000 7fbe6f59e700 1 -- 172.21.15.42:0/2851711320 learned_addr learned my addr 172.21.15.42:0/2851711320 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:25:31.249 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:31.233+0000 7fbe74988700 1 --2- 172.21.15.42:0/2851711320 >> v2:172.21.15.40:3300/0 conn(0x7fbe7011f290 0x7fbe70124970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:25:31.250 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:31.233+0000 7fbe6f59e700 1 -- 172.21.15.42:0/2851711320 >> v2:172.21.15.40:3300/0 conn(0x7fbe7011f290 msgr2=0x7fbe70124970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:25:31.250 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:31.233+0000 7fbe6f59e700 1 --2- 172.21.15.42:0/2851711320 >> v2:172.21.15.40:3300/0 conn(0x7fbe7011f290 0x7fbe70124970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:25:31.250 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:31.233+0000 7fbe6f59e700 1 -- 172.21.15.42:0/2851711320 >> v1:172.21.15.40:6789/0 conn(0x7fbe7011ee60 legacy=0x7fbe70124170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:25:31.250 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:31.233+0000 7fbe6f59e700 1 -- 172.21.15.42:0/2851711320 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe7008f9e0 con 0x7fbe700a3710 2019-12-05T05:25:31.251 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:31.233+0000 7fbe6f59e700 1 --2- 172.21.15.42:0/2851711320 >> v2:172.21.15.42:3300/0 conn(0x7fbe700a3710 0x7fbe7008f490 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fbe60000e90 tx=0x7fbe6000c390).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:31.251 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:31.233+0000 7fbe6f59e700 1 --2- 172.21.15.42:0/2851711320 >> v2:172.21.15.42:3300/0 conn(0x7fbe700a3710 0x7fbe7008f490 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fbe60000e90 tx=0x7fbe6000c390).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:31.251 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:31.233+0000 7fbe6f59e700 1 --2- 172.21.15.42:0/2851711320 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbe700a3710 0x7fbe7008f490 secure :-1 s=READY pgs=820 cs=0 l=1 rx=0x7fbe60000e90 tx=0x7fbe6000c390).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:25:31.251 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:31.233+0000 7fbe6f59e700 1 --2- 172.21.15.42:0/2851711320 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbe700a3710 0x7fbe7008f490 secure :-1 s=THROTTLE_DONE pgs=820 cs=0 l=1 rx=0x7fbe60000e90 tx=0x7fbe6000c390).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:31.251 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:31.233+0000 7fbe6f59e700 1 --2- 172.21.15.42:0/2851711320 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbe700a3710 0x7fbe7008f490 secure :-1 s=THROTTLE_DONE pgs=820 cs=0 l=1 rx=0x7fbe60000e90 tx=0x7fbe6000c390).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:31.252 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:31.233+0000 7fbe67fff700 1 -- 172.21.15.42:0/2851711320 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbe60011660 con 0x7fbe700a3710 2019-12-05T05:25:31.252 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:31.233+0000 7fbe6f59e700 1 --2- 172.21.15.42:0/2851711320 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbe700a3710 0x7fbe7008f490 secure :-1 s=THROTTLE_DONE pgs=820 cs=0 l=1 rx=0x7fbe60000e90 tx=0x7fbe6000c390).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:31.252 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:31.233+0000 7fbe67fff700 1 -- 172.21.15.42:0/2851711320 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe60011280 con 0x7fbe700a3710 2019-12-05T05:25:31.252 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:31.233+0000 7fbe75189700 1 -- 172.21.15.42:0/2851711320 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe70124ec0 con 0x7fbe700a3710 2019-12-05T05:25:31.253 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:31.233+0000 7fbe67fff700 1 -- 172.21.15.42:0/2851711320 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbe60011c40 con 0x7fbe700a3710 2019-12-05T05:25:31.253 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:31.233+0000 7fbe75189700 1 -- 172.21.15.42:0/2851711320 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe700a3b40 con 0x7fbe700a3710 2019-12-05T05:25:31.254 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:31.233+0000 7fbe6f59e700 1 --2- 172.21.15.42:0/2851711320 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbe700a3710 0x7fbe7008f490 secure :-1 s=THROTTLE_DONE pgs=820 cs=0 l=1 rx=0x7fbe60000e90 tx=0x7fbe6000c390).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:31.254 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:31.237+0000 7fbe67fff700 1 -- 172.21.15.42:0/2851711320 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fbe60006460 con 0x7fbe700a3710 2019-12-05T05:25:31.254 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:31.237+0000 7fbe6f59e700 1 --2- 172.21.15.42:0/2851711320 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbe700a3710 0x7fbe7008f490 secure :-1 s=THROTTLE_DONE pgs=820 cs=0 l=1 rx=0x7fbe60000e90 tx=0x7fbe6000c390).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:31.254 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:31.237+0000 7fbe67fff700 1 --2- 172.21.15.42:0/2851711320 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbe5403ad30 0x7fbe5403d240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:25:31.255 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:31.237+0000 7fbe67fff700 1 -- 172.21.15.42:0/2851711320 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 7235+0+0 (secure 0 0 0) 0x7fbe6004af30 con 0x7fbe700a3710 2019-12-05T05:25:31.255 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:31.237+0000 7fbe6ed9d700 1 --2- 172.21.15.42:0/2851711320 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbe5403ad30 0x7fbe5403d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:25:31.255 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:31.237+0000 7fbe6ed9d700 1 --2- 172.21.15.42:0/2851711320 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbe5403ad30 0x7fbe5403d240 crc :-1 s=READY pgs=362 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:25:31.256 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:31.237+0000 7fbe75189700 1 -- 172.21.15.42:0/2851711320 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe7005d100 con 0x7fbe700a3710 2019-12-05T05:25:31.257 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:31.245+0000 7fbe6f59e700 1 --2- 172.21.15.42:0/2851711320 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbe700a3710 0x7fbe7008f490 secure :-1 s=THROTTLE_DONE pgs=820 cs=0 l=1 rx=0x7fbe60000e90 tx=0x7fbe6000c390).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:31.257 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:31.245+0000 7fbe67fff700 1 -- 172.21.15.42:0/2851711320 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fbe60031fb0 con 0x7fbe700a3710 2019-12-05T05:25:31.417 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:31.413+0000 7fbe75189700 1 -- 172.21.15.42:0/2851711320 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbe7005b8b0 con 0x7fbe700a3710 2019-12-05T05:25:31.418 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:31.413+0000 7fbe6f59e700 1 --2- 172.21.15.42:0/2851711320 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbe700a3710 0x7fbe7008f490 secure :-1 s=THROTTLE_DONE pgs=820 cs=0 l=1 rx=0x7fbe60000e90 tx=0x7fbe6000c390).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:31.418 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:31.413+0000 7fbe67fff700 1 -- 172.21.15.42:0/2851711320 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 103 v103) v1 ==== 95+0+2111 (secure 0 0 0) 0x7fbe60032210 con 0x7fbe700a3710 2019-12-05T05:25:31.418 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 103 2019-12-05T05:25:31.419 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:25:31.419 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":103,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5240,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/1455108684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1455108684},{"type":"v1","addr":"172.21.15.42:6835","nonce":1455108684}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":103}],"filesystems":[{"mdsmap":{"epoch":102,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:23:58.091182+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5177},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5177":{"gid":5177,"name":"b","rank":0,"incarnation":99,"state":"up:active","state_seq":14,"addr":"172.21.15.40:6833/3208416000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3208416000},{"type":"v1","addr":"172.21.15.40:6833","nonce":3208416000}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:25:31.421 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:31.417+0000 7fbe75189700 1 -- 172.21.15.42:0/2851711320 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbe5403ad30 msgr2=0x7fbe5403d240 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:25:31.421 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:31.417+0000 7fbe75189700 1 --2- 172.21.15.42:0/2851711320 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbe5403ad30 0x7fbe5403d240 crc :-1 s=READY pgs=362 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:25:31.421 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:31.417+0000 7fbe75189700 1 -- 172.21.15.42:0/2851711320 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbe700a3710 msgr2=0x7fbe7008f490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:25:31.421 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:31.417+0000 7fbe75189700 1 --2- 172.21.15.42:0/2851711320 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbe700a3710 0x7fbe7008f490 secure :-1 s=READY pgs=820 cs=0 l=1 rx=0x7fbe60000e90 tx=0x7fbe6000c390).stop 2019-12-05T05:25:31.421 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:31.417+0000 7fbe75189700 1 -- 172.21.15.42:0/2851711320 shutdown_connections 2019-12-05T05:25:31.421 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:31.417+0000 7fbe75189700 1 --2- 172.21.15.42:0/2851711320 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbe5403ad30 0x7fbe5403d240 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:25:31.422 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:31.417+0000 7fbe75189700 1 --2- 172.21.15.42:0/2851711320 >> v2:172.21.15.40:3300/0 conn(0x7fbe7011f290 0x7fbe70124970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:25:31.422 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:31.417+0000 7fbe75189700 1 --2- 172.21.15.42:0/2851711320 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbe700a3710 0x7fbe7008f490 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:25:31.422 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:31.417+0000 7fbe75189700 1 -- 172.21.15.42:0/2851711320 shutdown_connections 2019-12-05T05:25:31.422 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:31.417+0000 7fbe75189700 1 -- 172.21.15.42:0/2851711320 wait complete. 2019-12-05T05:25:31.422 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:31.417+0000 7fbe75189700 1 -- 172.21.15.42:0/2851711320 >> 172.21.15.42:0/2851711320 conn(0x7fbe700aabd0 msgr2=0x7fbe700a9630 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:25:31.430 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T05:25:33.431 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:25:33.554 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.525+0000 7fb851d9a700 1 Processor -- start 2019-12-05T05:25:33.555 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.525+0000 7fb851d9a700 1 -- start start 2019-12-05T05:25:33.555 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.525+0000 7fb851d9a700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb84c0a7740 0x7fb84c08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:25:33.555 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.525+0000 7fb851d9a700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb84c07e720 0x7fb84c07eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:25:33.555 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.525+0000 7fb851d9a700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb84c09f2e0 0x7fb84c09f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:25:33.556 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.525+0000 7fb851d9a700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb84c0a0b00 con 0x7fb84c0a7740 2019-12-05T05:25:33.556 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.525+0000 7fb851d9a700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb84c08e5b0 con 0x7fb84c09f2e0 2019-12-05T05:25:33.556 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.525+0000 7fb851d9a700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb84c08e850 con 0x7fb84c07e720 2019-12-05T05:25:33.556 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.525+0000 7fb850b38700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb84c09f2e0 0x7fb84c09f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:25:33.557 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.525+0000 7fb850b38700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb84c09f2e0 0x7fb84c09f710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:33352/0 (socket says 172.21.15.42:33352) 2019-12-05T05:25:33.557 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.525+0000 7fb850b38700 1 -- 172.21.15.42:0/622854966 learned_addr learned my addr 172.21.15.42:0/622854966 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:25:33.557 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.529+0000 7fb8437fe700 1 --2- 172.21.15.42:0/622854966 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb84c07e720 0x7fb84c07eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:25:33.559 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.529+0000 7fb84bfff700 1 --2- 172.21.15.42:0/622854966 >> v2:172.21.15.40:3300/0 conn(0x7fb84c0a7740 0x7fb84c08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:25:33.559 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.529+0000 7fb850b38700 1 -- 172.21.15.42:0/622854966 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb84c07e720 msgr2=0x7fb84c07eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:25:33.559 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.529+0000 7fb850b38700 1 --2- 172.21.15.42:0/622854966 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb84c07e720 0x7fb84c07eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:25:33.559 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.529+0000 7fb850b38700 1 -- 172.21.15.42:0/622854966 >> v2:172.21.15.40:3300/0 conn(0x7fb84c0a7740 msgr2=0x7fb84c08df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:25:33.559 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.529+0000 7fb850b38700 1 --2- 172.21.15.42:0/622854966 >> v2:172.21.15.40:3300/0 conn(0x7fb84c0a7740 0x7fb84c08df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:25:33.560 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.529+0000 7fb850b38700 1 -- 172.21.15.42:0/622854966 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb84c07f0a0 con 0x7fb84c09f2e0 2019-12-05T05:25:33.560 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.529+0000 7fb850b38700 1 --2- 172.21.15.42:0/622854966 >> v2:172.21.15.42:3300/0 conn(0x7fb84c09f2e0 0x7fb84c09f710 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fb84c08e770 tx=0x7fb844010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:33.560 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.529+0000 7fb850b38700 1 --2- 172.21.15.42:0/622854966 >> v2:172.21.15.42:3300/0 conn(0x7fb84c09f2e0 0x7fb84c09f710 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fb84c08e770 tx=0x7fb844010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:33.560 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.529+0000 7fb850b38700 1 --2- 172.21.15.42:0/622854966 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb84c09f2e0 0x7fb84c09f710 secure :-1 s=READY pgs=821 cs=0 l=1 rx=0x7fb84c08e770 tx=0x7fb844010130).ready entity=mon.1 client_cookie=12eb85967d70f610 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:25:33.560 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.529+0000 7fb850b38700 1 --2- 172.21.15.42:0/622854966 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb84c09f2e0 0x7fb84c09f710 secure :-1 s=THROTTLE_DONE pgs=821 cs=0 l=1 rx=0x7fb84c08e770 tx=0x7fb844010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:33.560 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.529+0000 7fb84b7fe700 1 -- 172.21.15.42:0/622854966 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb844010de0 con 0x7fb84c09f2e0 2019-12-05T05:25:33.560 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.529+0000 7fb850b38700 1 --2- 172.21.15.42:0/622854966 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb84c09f2e0 0x7fb84c09f710 secure :-1 s=THROTTLE_DONE pgs=821 cs=0 l=1 rx=0x7fb84c08e770 tx=0x7fb844010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:33.561 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.529+0000 7fb850b38700 1 --2- 172.21.15.42:0/622854966 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb84c09f2e0 0x7fb84c09f710 secure :-1 s=THROTTLE_DONE pgs=821 cs=0 l=1 rx=0x7fb84c08e770 tx=0x7fb844010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:33.561 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.529+0000 7fb84b7fe700 1 -- 172.21.15.42:0/622854966 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb844010a00 con 0x7fb84c09f2e0 2019-12-05T05:25:33.561 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.529+0000 7fb84b7fe700 1 -- 172.21.15.42:0/622854966 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb8440114a0 con 0x7fb84c09f2e0 2019-12-05T05:25:33.561 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.529+0000 7fb851d9a700 1 -- 172.21.15.42:0/622854966 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb84c09f2e0 msgr2=0x7fb84c09f710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:25:33.561 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.529+0000 7fb851d9a700 1 --2- 172.21.15.42:0/622854966 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb84c09f2e0 0x7fb84c09f710 secure :-1 s=READY pgs=821 cs=0 l=1 rx=0x7fb84c08e770 tx=0x7fb844010130).stop 2019-12-05T05:25:33.561 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.529+0000 7fb851d9a700 1 -- 172.21.15.42:0/622854966 shutdown_connections 2019-12-05T05:25:33.561 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.529+0000 7fb851d9a700 1 --2- 172.21.15.42:0/622854966 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb84c09f2e0 0x7fb84c09f710 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:25:33.562 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.529+0000 7fb851d9a700 1 --2- 172.21.15.42:0/622854966 >> v2:172.21.15.40:3300/0 conn(0x7fb84c0a7740 0x7fb84c08df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:25:33.562 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.529+0000 7fb851d9a700 1 --2- 172.21.15.42:0/622854966 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb84c07e720 0x7fb84c07eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:25:33.573 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.529+0000 7fb851d9a700 1 -- 172.21.15.42:0/622854966 shutdown_connections 2019-12-05T05:25:33.573 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.529+0000 7fb851d9a700 1 -- 172.21.15.42:0/622854966 wait complete. 2019-12-05T05:25:33.573 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.529+0000 7fb851d9a700 1 -- 172.21.15.42:0/622854966 >> 172.21.15.42:0/622854966 conn(0x7fb84c0aabe0 msgr2=0x7fb84c0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:25:33.574 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.533+0000 7fb851d9a700 1 Processor -- start 2019-12-05T05:25:33.574 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.533+0000 7fb851d9a700 1 -- start start 2019-12-05T05:25:33.574 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.533+0000 7fb851d9a700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb84c07cdc0 0x7fb84c1233a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:25:33.574 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.533+0000 7fb851d9a700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb84c13b9f0 0x7fb84c13de30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:25:33.574 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.533+0000 7fb851d9a700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb84c07bd50 con 0x7fb84c0a5ab0 2019-12-05T05:25:33.574 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.533+0000 7fb851d9a700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb84c0a4a50 con 0x7fb84c07cdc0 2019-12-05T05:25:33.574 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.533+0000 7fb851d9a700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb84c0a6ca0 con 0x7fb84c13b9f0 2019-12-05T05:25:33.575 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.533+0000 7fb850b38700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb84c13b9f0 0x7fb84c13de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:25:33.575 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.533+0000 7fb84bfff700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb84c07cdc0 0x7fb84c1233a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:25:33.575 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.533+0000 7fb850b38700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb84c13b9f0 0x7fb84c13de30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:33824/0 (socket says 172.21.15.42:33824) 2019-12-05T05:25:33.575 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.533+0000 7fb850b38700 1 -- 172.21.15.42:0/3097395732 learned_addr learned my addr 172.21.15.42:0/3097395732 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:25:33.575 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.533+0000 7fb8437fe700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fb84c0a5ab0 0x7fb84c0a5ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:42834/0 (socket says 172.21.15.42:42834) 2019-12-05T05:25:33.575 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.537+0000 7fb84bfff700 1 -- 172.21.15.42:0/3097395732 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb84c13b9f0 msgr2=0x7fb84c13de30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:25:33.575 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.537+0000 7fb84bfff700 1 --2- 172.21.15.42:0/3097395732 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb84c13b9f0 0x7fb84c13de30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:25:33.576 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.537+0000 7fb84bfff700 1 -- 172.21.15.42:0/3097395732 >> v1:172.21.15.40:6789/0 conn(0x7fb84c0a5ab0 legacy=0x7fb84c0a5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:25:33.576 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.537+0000 7fb84bfff700 1 -- 172.21.15.42:0/3097395732 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb84c1238f0 con 0x7fb84c07cdc0 2019-12-05T05:25:33.576 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.537+0000 7fb8497fa700 1 -- 172.21.15.42:0/3097395732 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 537390647 0 0) 0x7fb84c07bd50 con 0x7fb84c0a5ab0 2019-12-05T05:25:33.576 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.537+0000 7fb850b38700 1 --2- 172.21.15.42:0/3097395732 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb84c13b9f0 0x7fb84c13de30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:25:33.576 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.537+0000 7fb84bfff700 1 --2- 172.21.15.42:0/3097395732 >> v2:172.21.15.40:3300/0 conn(0x7fb84c07cdc0 0x7fb84c1233a0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fb84c0a4c10 tx=0x7fb83c010ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:33.576 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.537+0000 7fb84bfff700 1 --2- 172.21.15.42:0/3097395732 >> v2:172.21.15.40:3300/0 conn(0x7fb84c07cdc0 0x7fb84c1233a0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fb84c0a4c10 tx=0x7fb83c010ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:33.576 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.537+0000 7fb84bfff700 1 --2- 172.21.15.42:0/3097395732 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb84c07cdc0 0x7fb84c1233a0 secure :-1 s=READY pgs=657 cs=0 l=1 rx=0x7fb84c0a4c10 tx=0x7fb83c010ae0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:25:33.577 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.537+0000 7fb84bfff700 1 --2- 172.21.15.42:0/3097395732 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb84c07cdc0 0x7fb84c1233a0 secure :-1 s=THROTTLE_DONE pgs=657 cs=0 l=1 rx=0x7fb84c0a4c10 tx=0x7fb83c010ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:33.577 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.537+0000 7fb8497fa700 1 -- 172.21.15.42:0/3097395732 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb83c0129b0 con 0x7fb84c07cdc0 2019-12-05T05:25:33.577 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.537+0000 7fb84bfff700 1 --2- 172.21.15.42:0/3097395732 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb84c07cdc0 0x7fb84c1233a0 secure :-1 s=THROTTLE_DONE pgs=657 cs=0 l=1 rx=0x7fb84c0a4c10 tx=0x7fb83c010ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:33.577 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.537+0000 7fb84bfff700 1 --2- 172.21.15.42:0/3097395732 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb84c07cdc0 0x7fb84c1233a0 secure :-1 s=THROTTLE_DONE pgs=657 cs=0 l=1 rx=0x7fb84c0a4c10 tx=0x7fb83c010ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:33.577 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.537+0000 7fb8497fa700 1 -- 172.21.15.42:0/3097395732 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb83c0125d0 con 0x7fb84c07cdc0 2019-12-05T05:25:33.577 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.537+0000 7fb8497fa700 1 -- 172.21.15.42:0/3097395732 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb83c005540 con 0x7fb84c07cdc0 2019-12-05T05:25:33.577 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.537+0000 7fb851d9a700 1 -- 172.21.15.42:0/3097395732 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb84c07c0f0 con 0x7fb84c07cdc0 2019-12-05T05:25:33.578 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.537+0000 7fb851d9a700 1 -- 172.21.15.42:0/3097395732 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb84c07b340 con 0x7fb84c07cdc0 2019-12-05T05:25:33.578 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.537+0000 7fb84bfff700 1 --2- 172.21.15.42:0/3097395732 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb84c07cdc0 0x7fb84c1233a0 secure :-1 s=THROTTLE_DONE pgs=657 cs=0 l=1 rx=0x7fb84c0a4c10 tx=0x7fb83c010ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:33.578 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.537+0000 7fb84bfff700 1 --2- 172.21.15.42:0/3097395732 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb84c07cdc0 0x7fb84c1233a0 secure :-1 s=THROTTLE_DONE pgs=657 cs=0 l=1 rx=0x7fb84c0a4c10 tx=0x7fb83c010ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:33.578 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.537+0000 7fb8497fa700 1 -- 172.21.15.42:0/3097395732 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fb83c007140 con 0x7fb84c07cdc0 2019-12-05T05:25:33.578 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.537+0000 7fb8497fa700 1 --2- 172.21.15.42:0/3097395732 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb83003ac90 0x7fb83003d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:25:33.578 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.537+0000 7fb8497fa700 1 -- 172.21.15.42:0/3097395732 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 7235+0+0 (secure 0 0 0) 0x7fb83c04ae40 con 0x7fb84c07cdc0 2019-12-05T05:25:33.578 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.537+0000 7fb8437fe700 1 --2- 172.21.15.42:0/3097395732 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb83003ac90 0x7fb83003d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:25:33.579 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.537+0000 7fb8437fe700 1 --2- 172.21.15.42:0/3097395732 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb83003ac90 0x7fb83003d1a0 crc :-1 s=READY pgs=363 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:25:33.579 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.541+0000 7fb851d9a700 1 -- 172.21.15.42:0/3097395732 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb84c05d110 con 0x7fb84c07cdc0 2019-12-05T05:25:33.579 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.549+0000 7fb84bfff700 1 --2- 172.21.15.42:0/3097395732 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb84c07cdc0 0x7fb84c1233a0 secure :-1 s=THROTTLE_DONE pgs=657 cs=0 l=1 rx=0x7fb84c0a4c10 tx=0x7fb83c010ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:33.579 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.549+0000 7fb8497fa700 1 -- 172.21.15.42:0/3097395732 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fb83c031e80 con 0x7fb84c07cdc0 2019-12-05T05:25:33.747 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.741+0000 7fb851d9a700 1 -- 172.21.15.42:0/3097395732 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb84c05b8c0 con 0x7fb84c07cdc0 2019-12-05T05:25:33.748 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.745+0000 7fb84bfff700 1 --2- 172.21.15.42:0/3097395732 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb84c07cdc0 0x7fb84c1233a0 secure :-1 s=THROTTLE_DONE pgs=657 cs=0 l=1 rx=0x7fb84c0a4c10 tx=0x7fb83c010ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:33.748 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.745+0000 7fb8497fa700 1 -- 172.21.15.42:0/3097395732 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 104 v104) v1 ==== 95+0+1774 (secure 0 0 0) 0x7fb83c0320e0 con 0x7fb84c07cdc0 2019-12-05T05:25:33.750 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:25:33.750 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":104,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:25:33.583844+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5240},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5240":{"gid":5240,"name":"a","rank":0,"incarnation":104,"state":"up:replay","state_seq":2,"addr":"172.21.15.42:6835/1455108684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1455108684},{"type":"v1","addr":"172.21.15.42:6835","nonce":1455108684}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:25:33.751 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 104 2019-12-05T05:25:33.753 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.749+0000 7fb851d9a700 1 -- 172.21.15.42:0/3097395732 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb83003ac90 msgr2=0x7fb83003d1a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:25:33.753 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.749+0000 7fb851d9a700 1 --2- 172.21.15.42:0/3097395732 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb83003ac90 0x7fb83003d1a0 crc :-1 s=READY pgs=363 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:25:33.753 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.749+0000 7fb851d9a700 1 -- 172.21.15.42:0/3097395732 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb84c07cdc0 msgr2=0x7fb84c1233a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:25:33.753 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.749+0000 7fb851d9a700 1 --2- 172.21.15.42:0/3097395732 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb84c07cdc0 0x7fb84c1233a0 secure :-1 s=READY pgs=657 cs=0 l=1 rx=0x7fb84c0a4c10 tx=0x7fb83c010ae0).stop 2019-12-05T05:25:33.754 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.749+0000 7fb851d9a700 1 -- 172.21.15.42:0/3097395732 shutdown_connections 2019-12-05T05:25:33.754 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.749+0000 7fb851d9a700 1 --2- 172.21.15.42:0/3097395732 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb83003ac90 0x7fb83003d1a0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:25:33.754 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.749+0000 7fb851d9a700 1 --2- 172.21.15.42:0/3097395732 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb84c13b9f0 0x7fb84c13de30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:25:33.754 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.749+0000 7fb851d9a700 1 --2- 172.21.15.42:0/3097395732 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb84c07cdc0 0x7fb84c1233a0 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:25:33.754 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.749+0000 7fb851d9a700 1 -- 172.21.15.42:0/3097395732 shutdown_connections 2019-12-05T05:25:33.754 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.749+0000 7fb851d9a700 1 -- 172.21.15.42:0/3097395732 wait complete. 2019-12-05T05:25:33.754 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.749+0000 7fb851d9a700 1 -- 172.21.15.42:0/3097395732 >> 172.21.15.42:0/3097395732 conn(0x7fb84c09fa10 msgr2=0x7fb84c059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:25:33.763 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b down, removed from mdsmap 2019-12-05T05:25:33.763 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2019-12-05T05:25:33.763 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:25:33.877 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.869+0000 7fd4e1900700 1 Processor -- start 2019-12-05T05:25:33.878 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.869+0000 7fd4e1900700 1 -- start start 2019-12-05T05:25:33.878 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.869+0000 7fd4e1900700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd4dc08df70 0x7fd4dc09f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:25:33.878 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.869+0000 7fd4e1900700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd4dc09f820 0x7fd4dc14efd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:25:33.878 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.869+0000 7fd4e1900700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4dc09fc50 con 0x7fd4dc0a7730 2019-12-05T05:25:33.878 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.869+0000 7fd4e1900700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4dc0a0af0 con 0x7fd4dc08df70 2019-12-05T05:25:33.878 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.869+0000 7fd4e1900700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4dc08e5a0 con 0x7fd4dc09f820 2019-12-05T05:25:33.879 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.869+0000 7fd4db7fe700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fd4dc0a7730 0x7fd4dc07e710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:33414/0 (socket says 172.21.15.42:33414) 2019-12-05T05:25:33.879 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.869+0000 7fd4db7fe700 1 -- 172.21.15.42:0/3924696262 learned_addr learned my addr 172.21.15.42:0/3924696262 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:25:33.879 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.869+0000 7fd4dbfff700 1 --2- 172.21.15.42:0/3924696262 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd4dc09f820 0x7fd4dc14efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:25:33.879 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.869+0000 7fd4d2ffd700 1 --2- 172.21.15.42:0/3924696262 >> v2:172.21.15.40:3300/0 conn(0x7fd4dc08df70 0x7fd4dc09f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:25:33.879 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.869+0000 7fd4daffd700 1 -- 172.21.15.42:0/3924696262 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2874917185 0 0) 0x7fd4dc09fc50 con 0x7fd4dc0a7730 2019-12-05T05:25:33.879 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.869+0000 7fd4daffd700 1 -- 172.21.15.42:0/3924696262 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4c80030b0 con 0x7fd4dc0a7730 2019-12-05T05:25:33.881 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.869+0000 7fd4daffd700 1 -- 172.21.15.42:0/3924696262 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2386805179 0 0) 0x7fd4c80030b0 con 0x7fd4dc0a7730 2019-12-05T05:25:33.881 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.869+0000 7fd4d2ffd700 1 -- 172.21.15.42:0/3924696262 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd4dc09f820 msgr2=0x7fd4dc14efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:25:33.881 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.869+0000 7fd4d2ffd700 1 --2- 172.21.15.42:0/3924696262 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd4dc09f820 0x7fd4dc14efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:25:33.881 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.869+0000 7fd4d2ffd700 1 -- 172.21.15.42:0/3924696262 >> v1:172.21.15.42:6789/0 conn(0x7fd4dc0a7730 legacy=0x7fd4dc07e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:25:33.881 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.869+0000 7fd4d2ffd700 1 -- 172.21.15.42:0/3924696262 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4dc0a0700 con 0x7fd4dc08df70 2019-12-05T05:25:33.882 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.869+0000 7fd4d2ffd700 1 --2- 172.21.15.42:0/3924696262 >> v2:172.21.15.40:3300/0 conn(0x7fd4dc08df70 0x7fd4dc09f2d0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fd4c4008790 tx=0x7fd4cc0019d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:33.882 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.869+0000 7fd4d2ffd700 1 --2- 172.21.15.42:0/3924696262 >> v2:172.21.15.40:3300/0 conn(0x7fd4dc08df70 0x7fd4dc09f2d0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fd4c4008790 tx=0x7fd4cc0019d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:33.882 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.869+0000 7fd4d2ffd700 1 --2- 172.21.15.42:0/3924696262 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd4dc08df70 0x7fd4dc09f2d0 secure :-1 s=READY pgs=658 cs=0 l=1 rx=0x7fd4c4008790 tx=0x7fd4cc0019d0).ready entity=mon.0 client_cookie=6100a04fdc38b139 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:25:33.882 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.869+0000 7fd4d2ffd700 1 --2- 172.21.15.42:0/3924696262 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd4dc08df70 0x7fd4dc09f2d0 secure :-1 s=THROTTLE_DONE pgs=658 cs=0 l=1 rx=0x7fd4c4008790 tx=0x7fd4cc0019d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:33.882 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.869+0000 7fd4daffd700 1 -- 172.21.15.42:0/3924696262 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd4cc001750 con 0x7fd4dc08df70 2019-12-05T05:25:33.882 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.869+0000 7fd4d2ffd700 1 --2- 172.21.15.42:0/3924696262 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd4dc08df70 0x7fd4dc09f2d0 secure :-1 s=THROTTLE_DONE pgs=658 cs=0 l=1 rx=0x7fd4c4008790 tx=0x7fd4cc0019d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:33.883 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.869+0000 7fd4d2ffd700 1 --2- 172.21.15.42:0/3924696262 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd4dc08df70 0x7fd4dc09f2d0 secure :-1 s=THROTTLE_DONE pgs=658 cs=0 l=1 rx=0x7fd4c4008790 tx=0x7fd4cc0019d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:33.883 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.869+0000 7fd4daffd700 1 -- 172.21.15.42:0/3924696262 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4cc001560 con 0x7fd4dc08df70 2019-12-05T05:25:33.883 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.869+0000 7fd4daffd700 1 -- 172.21.15.42:0/3924696262 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd4c4008da0 con 0x7fd4dc08df70 2019-12-05T05:25:33.883 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.869+0000 7fd4e1900700 1 -- 172.21.15.42:0/3924696262 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd4dc08df70 msgr2=0x7fd4dc09f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:25:33.887 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.869+0000 7fd4e1900700 1 --2- 172.21.15.42:0/3924696262 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd4dc08df70 0x7fd4dc09f2d0 secure :-1 s=READY pgs=658 cs=0 l=1 rx=0x7fd4c4008790 tx=0x7fd4cc0019d0).stop 2019-12-05T05:25:33.887 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.869+0000 7fd4e1900700 1 -- 172.21.15.42:0/3924696262 shutdown_connections 2019-12-05T05:25:33.887 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.869+0000 7fd4e1900700 1 --2- 172.21.15.42:0/3924696262 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd4dc09f820 0x7fd4dc14efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:25:33.888 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.869+0000 7fd4e1900700 1 --2- 172.21.15.42:0/3924696262 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd4dc08df70 0x7fd4dc09f2d0 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:25:33.888 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.873+0000 7fd4e1900700 1 -- 172.21.15.42:0/3924696262 shutdown_connections 2019-12-05T05:25:33.888 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.873+0000 7fd4e1900700 1 -- 172.21.15.42:0/3924696262 wait complete. 2019-12-05T05:25:33.888 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.873+0000 7fd4e1900700 1 -- 172.21.15.42:0/3924696262 >> 172.21.15.42:0/3924696262 conn(0x7fd4dc0aabd0 msgr2=0x7fd4dc0a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:25:33.889 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.873+0000 7fd4e1900700 1 Processor -- start 2019-12-05T05:25:33.889 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.873+0000 7fd4e1900700 1 -- start start 2019-12-05T05:25:33.892 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.873+0000 7fd4e1900700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd4dc11bb30 0x7fd4dc132130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:25:33.892 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.873+0000 7fd4e1900700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4dc131640 con 0x7fd4dc0a5580 2019-12-05T05:25:33.892 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.873+0000 7fd4e1900700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4dc119c00 con 0x7fd4dc12e080 2019-12-05T05:25:33.892 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.873+0000 7fd4e1900700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4dc131950 con 0x7fd4dc11bb30 2019-12-05T05:25:33.893 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.873+0000 7fd4db7fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd4dc11bb30 0x7fd4dc132130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:25:33.893 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.873+0000 7fd4db7fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd4dc11bb30 0x7fd4dc132130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:33382/0 (socket says 172.21.15.42:33382) 2019-12-05T05:25:33.893 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.873+0000 7fd4db7fe700 1 -- 172.21.15.42:0/2742154699 learned_addr learned my addr 172.21.15.42:0/2742154699 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:25:33.893 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.877+0000 7fd4db7fe700 1 -- 172.21.15.42:0/2742154699 >> v1:172.21.15.42:6789/0 conn(0x7fd4dc12e080 legacy=0x7fd4dc13bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:25:33.893 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.877+0000 7fd4db7fe700 1 -- 172.21.15.42:0/2742154699 >> v1:172.21.15.40:6789/0 conn(0x7fd4dc0a5580 legacy=0x7fd4dc1183b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:25:33.893 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.877+0000 7fd4db7fe700 1 -- 172.21.15.42:0/2742154699 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4dc0a59b0 con 0x7fd4dc11bb30 2019-12-05T05:25:33.893 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.877+0000 7fd4db7fe700 1 --2- 172.21.15.42:0/2742154699 >> v2:172.21.15.42:3300/0 conn(0x7fd4dc11bb30 0x7fd4dc132130 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fd4cc001a50 tx=0x7fd4cc0109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:33.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.877+0000 7fd4db7fe700 1 --2- 172.21.15.42:0/2742154699 >> v2:172.21.15.42:3300/0 conn(0x7fd4dc11bb30 0x7fd4dc132130 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fd4cc001a50 tx=0x7fd4cc0109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:33.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.877+0000 7fd4db7fe700 1 --2- 172.21.15.42:0/2742154699 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd4dc11bb30 0x7fd4dc132130 secure :-1 s=READY pgs=824 cs=0 l=1 rx=0x7fd4cc001a50 tx=0x7fd4cc0109e0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:25:33.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.877+0000 7fd4db7fe700 1 --2- 172.21.15.42:0/2742154699 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd4dc11bb30 0x7fd4dc132130 secure :-1 s=THROTTLE_DONE pgs=824 cs=0 l=1 rx=0x7fd4cc001a50 tx=0x7fd4cc0109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:33.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.877+0000 7fd4db7fe700 1 --2- 172.21.15.42:0/2742154699 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd4dc11bb30 0x7fd4dc132130 secure :-1 s=THROTTLE_DONE pgs=824 cs=0 l=1 rx=0x7fd4cc001a50 tx=0x7fd4cc0109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:33.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.877+0000 7fd4db7fe700 1 --2- 172.21.15.42:0/2742154699 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd4dc11bb30 0x7fd4dc132130 secure :-1 s=THROTTLE_DONE pgs=824 cs=0 l=1 rx=0x7fd4cc001a50 tx=0x7fd4cc0109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:33.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.877+0000 7fd4d8ff9700 1 -- 172.21.15.42:0/2742154699 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd4cc00c890 con 0x7fd4dc11bb30 2019-12-05T05:25:33.895 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.877+0000 7fd4d8ff9700 1 -- 172.21.15.42:0/2742154699 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4cc00c4b0 con 0x7fd4dc11bb30 2019-12-05T05:25:33.895 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.877+0000 7fd4e1900700 1 -- 172.21.15.42:0/2742154699 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4dc12e4b0 con 0x7fd4dc11bb30 2019-12-05T05:25:33.895 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.877+0000 7fd4d8ff9700 1 -- 172.21.15.42:0/2742154699 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd4cc012450 con 0x7fd4dc11bb30 2019-12-05T05:25:33.895 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.877+0000 7fd4e1900700 1 -- 172.21.15.42:0/2742154699 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4dc13d2d0 con 0x7fd4dc11bb30 2019-12-05T05:25:33.895 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.877+0000 7fd4db7fe700 1 --2- 172.21.15.42:0/2742154699 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd4dc11bb30 0x7fd4dc132130 secure :-1 s=THROTTLE_DONE pgs=824 cs=0 l=1 rx=0x7fd4cc001a50 tx=0x7fd4cc0109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:33.895 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.877+0000 7fd4d8ff9700 1 -- 172.21.15.42:0/2742154699 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fd4cc012760 con 0x7fd4dc11bb30 2019-12-05T05:25:33.895 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.877+0000 7fd4db7fe700 1 --2- 172.21.15.42:0/2742154699 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd4dc11bb30 0x7fd4dc132130 secure :-1 s=THROTTLE_DONE pgs=824 cs=0 l=1 rx=0x7fd4cc001a50 tx=0x7fd4cc0109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:33.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.877+0000 7fd4d8ff9700 1 --2- 172.21.15.42:0/2742154699 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd4c003ad90 0x7fd4c003d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:25:33.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.877+0000 7fd4d8ff9700 1 -- 172.21.15.42:0/2742154699 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7321+0+0 (secure 0 0 0) 0x7fd4cc04ad70 con 0x7fd4dc11bb30 2019-12-05T05:25:33.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.877+0000 7fd4d2ffd700 1 --2- 172.21.15.42:0/2742154699 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd4c003ad90 0x7fd4c003d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:25:33.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.877+0000 7fd4d2ffd700 1 --2- 172.21.15.42:0/2742154699 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd4c003ad90 0x7fd4c003d2a0 crc :-1 s=READY pgs=364 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:25:33.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.881+0000 7fd4e1900700 1 -- 172.21.15.42:0/2742154699 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4dc05d100 con 0x7fd4dc11bb30 2019-12-05T05:25:33.902 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.885+0000 7fd4db7fe700 1 --2- 172.21.15.42:0/2742154699 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd4dc11bb30 0x7fd4dc132130 secure :-1 s=THROTTLE_DONE pgs=824 cs=0 l=1 rx=0x7fd4cc001a50 tx=0x7fd4cc0109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:33.902 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:33.885+0000 7fd4d8ff9700 1 -- 172.21.15.42:0/2742154699 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fd4cc031df0 con 0x7fd4dc11bb30 2019-12-05T05:25:34.045 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:34.041+0000 7fd4e1900700 1 -- 172.21.15.42:0/2742154699 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd4dc05b8b0 con 0x7fd4dc11bb30 2019-12-05T05:25:34.046 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:34.041+0000 7fd4db7fe700 1 --2- 172.21.15.42:0/2742154699 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd4dc11bb30 0x7fd4dc132130 secure :-1 s=THROTTLE_DONE pgs=824 cs=0 l=1 rx=0x7fd4cc001a50 tx=0x7fd4cc0109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:34.046 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:34.041+0000 7fd4d8ff9700 1 -- 172.21.15.42:0/2742154699 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 104 v104) v1 ==== 95+0+1774 (secure 0 0 0) 0x7fd4cc032050 con 0x7fd4dc11bb30 2019-12-05T05:25:34.046 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 104 2019-12-05T05:25:34.046 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:25:34.047 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":104,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:25:33.583844+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5240},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5240":{"gid":5240,"name":"a","rank":0,"incarnation":104,"state":"up:replay","state_seq":2,"addr":"172.21.15.42:6835/1455108684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1455108684},{"type":"v1","addr":"172.21.15.42:6835","nonce":1455108684}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:25:34.049 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:34.045+0000 7fd4e1900700 1 -- 172.21.15.42:0/2742154699 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd4c003ad90 msgr2=0x7fd4c003d2a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:25:34.049 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:34.045+0000 7fd4e1900700 1 --2- 172.21.15.42:0/2742154699 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd4c003ad90 0x7fd4c003d2a0 crc :-1 s=READY pgs=364 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:25:34.050 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:34.045+0000 7fd4e1900700 1 -- 172.21.15.42:0/2742154699 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd4dc11bb30 msgr2=0x7fd4dc132130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:25:34.050 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:34.045+0000 7fd4e1900700 1 --2- 172.21.15.42:0/2742154699 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd4dc11bb30 0x7fd4dc132130 secure :-1 s=READY pgs=824 cs=0 l=1 rx=0x7fd4cc001a50 tx=0x7fd4cc0109e0).stop 2019-12-05T05:25:34.050 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:34.045+0000 7fd4e1900700 1 -- 172.21.15.42:0/2742154699 shutdown_connections 2019-12-05T05:25:34.050 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:34.045+0000 7fd4e1900700 1 --2- 172.21.15.42:0/2742154699 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd4c003ad90 0x7fd4c003d2a0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:25:34.050 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:34.045+0000 7fd4e1900700 1 --2- 172.21.15.42:0/2742154699 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd4dc11bb30 0x7fd4dc132130 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:25:34.051 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:34.045+0000 7fd4e1900700 1 -- 172.21.15.42:0/2742154699 shutdown_connections 2019-12-05T05:25:34.051 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:34.045+0000 7fd4e1900700 1 -- 172.21.15.42:0/2742154699 wait complete. 2019-12-05T05:25:34.051 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:34.045+0000 7fd4e1900700 1 -- 172.21.15.42:0/2742154699 >> 172.21.15.42:0/2742154699 conn(0x7fd4dc0a7730 msgr2=0x7fd4dc0596f0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:25:34.059 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 104, "feature_flags": { "enable_multiple": false, "ever_enabled_multiple": false }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2019-12-05T05:03:35.604738+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 104, "ever_allowed_features": 2, "explicitly_allowed_features": 2, "failed": [], "flags": 18, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5240": { "addr": "172.21.15.42:6835/1455108684", "addrs": { "addrvec": [ { "addr": "172.21.15.42:6834", "nonce": 1455108684, "type": "v2" }, { "addr": "172.21.15.42:6835", "nonce": 1455108684, "type": "v1" } ] }, "export_targets": [], "features": 4540138292836696063, "flags": 0, "gid": 5240, "incarnation": 104, "name": "a", "rank": 0, "state": "up:replay", "state_seq": 2 } }, "last_failure": 0, "last_failure_osd_epoch": 64, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 2, "min_compat_client": "0 (unknown)", "modified": "2019-12-05T05:25:33.583844+0000", "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5240 } } } ], "standbys": [] } 2019-12-05T05:25:36.060 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:25:36.152 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:36.141+0000 7f75822d3700 1 Processor -- start 2019-12-05T05:25:36.153 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:36.141+0000 7f75822d3700 1 -- start start 2019-12-05T05:25:36.153 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:36.145+0000 7f75822d3700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f757c0a7740 0x7f757c08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:25:36.153 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:36.145+0000 7f75822d3700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f757c0a0710 con 0x7f757c07e720 2019-12-05T05:25:36.153 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:36.145+0000 7f75822d3700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f757c0a9ba0 con 0x7f757c09fae0 2019-12-05T05:25:36.153 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:36.145+0000 7f75822d3700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f757c0a0b00 con 0x7f757c0a7740 2019-12-05T05:25:36.153 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:36.145+0000 7f7581071700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f757c09fae0 0x7f757c14efe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:33436/0 (socket says 172.21.15.42:33436) 2019-12-05T05:25:36.154 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:36.145+0000 7f7581071700 1 -- 172.21.15.42:0/4126801806 learned_addr learned my addr 172.21.15.42:0/4126801806 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:25:36.154 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:36.145+0000 7f7580870700 1 --2- 172.21.15.42:0/4126801806 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f757c0a7740 0x7f757c08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:25:36.154 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:36.145+0000 7f757b7fe700 1 -- 172.21.15.42:0/4126801806 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1958298433 0 0) 0x7f757c0a9ba0 con 0x7f757c09fae0 2019-12-05T05:25:36.154 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:36.145+0000 7f757b7fe700 1 -- 172.21.15.42:0/4126801806 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f755c0030b0 con 0x7f757c09fae0 2019-12-05T05:25:36.154 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:36.145+0000 7f757b7fe700 1 -- 172.21.15.42:0/4126801806 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3326006355 0 0) 0x7f757c0a0710 con 0x7f757c07e720 2019-12-05T05:25:36.154 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:36.145+0000 7f757b7fe700 1 -- 172.21.15.42:0/4126801806 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f757c0a9ba0 con 0x7f757c07e720 2019-12-05T05:25:36.154 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:36.145+0000 7f757b7fe700 1 -- 172.21.15.42:0/4126801806 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1301664790 0 0) 0x7f755c0030b0 con 0x7f757c09fae0 2019-12-05T05:25:36.155 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:36.145+0000 7f757b7fe700 1 -- 172.21.15.42:0/4126801806 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f757c0a7740 msgr2=0x7f757c08df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:25:36.155 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:36.145+0000 7f757b7fe700 1 --2- 172.21.15.42:0/4126801806 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f757c0a7740 0x7f757c08df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:25:36.155 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:36.145+0000 7f757b7fe700 1 -- 172.21.15.42:0/4126801806 >> v1:172.21.15.40:6789/0 conn(0x7f757c07e720 legacy=0x7f757c09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:25:36.155 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:36.145+0000 7f757b7fe700 1 -- 172.21.15.42:0/4126801806 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f757c07eb50 con 0x7f757c09fae0 2019-12-05T05:25:36.155 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:36.145+0000 7f757b7fe700 1 -- 172.21.15.42:0/4126801806 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7f756c001ad0 con 0x7f757c09fae0 2019-12-05T05:25:36.155 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:36.145+0000 7f757b7fe700 1 -- 172.21.15.42:0/4126801806 >> v1:172.21.15.42:6789/0 conn(0x7f757c09fae0 legacy=0x7f757c14efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:25:36.155 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:36.145+0000 7f757b7fe700 1 --2- 172.21.15.42:0/4126801806 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f755c003940 0x7f755c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:25:36.155 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:36.145+0000 7f757b7fe700 1 --2- 172.21.15.42:0/4126801806 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f755c004420 0x7f755c0048d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:25:36.156 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:36.145+0000 7f757b7fe700 1 --2- 172.21.15.42:0/4126801806 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f755c004fc0 0x7f755c04f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:25:36.156 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:36.145+0000 7f757b7fe700 1 -- 172.21.15.42:0/4126801806 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f757c0a0b00 con 0x7f755c004420 2019-12-05T05:25:36.156 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:36.145+0000 7f7580870700 1 --2- 172.21.15.42:0/4126801806 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f755c003940 0x7f755c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:25:36.156 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:36.145+0000 7f757b7fe700 1 -- 172.21.15.42:0/4126801806 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f755c04fc00 con 0x7f755c003940 2019-12-05T05:25:36.156 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:36.145+0000 7f757b7fe700 1 -- 172.21.15.42:0/4126801806 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f755c04fe40 con 0x7f755c004fc0 2019-12-05T05:25:36.156 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:36.145+0000 7f757bfff700 1 --2- 172.21.15.42:0/4126801806 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f755c004420 0x7f755c0048d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:25:36.156 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:36.145+0000 7f7581071700 1 --2- 172.21.15.42:0/4126801806 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f755c004fc0 0x7f755c04f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:25:36.157 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:36.145+0000 7f7580870700 1 -- 172.21.15.42:0/4126801806 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f755c004fc0 msgr2=0x7f755c04f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:25:36.157 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:36.145+0000 7f7580870700 1 --2- 172.21.15.42:0/4126801806 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f755c004fc0 0x7f755c04f680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:25:36.157 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:36.145+0000 7f7580870700 1 -- 172.21.15.42:0/4126801806 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f755c004420 msgr2=0x7f755c0048d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:25:36.157 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:36.145+0000 7f7580870700 1 --2- 172.21.15.42:0/4126801806 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f755c004420 0x7f755c0048d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:25:36.157 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:36.145+0000 7f7580870700 1 -- 172.21.15.42:0/4126801806 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f755c050140 con 0x7f755c003940 2019-12-05T05:25:36.157 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:36.145+0000 7f7580870700 1 --2- 172.21.15.42:0/4126801806 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f755c003940 0x7f755c003d90 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f7574002f30 tx=0x7f757400ffe0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:36.157 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:36.145+0000 7f7580870700 1 --2- 172.21.15.42:0/4126801806 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f755c003940 0x7f755c003d90 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f7574002f30 tx=0x7f757400ffe0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:36.158 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:36.145+0000 7f7580870700 1 --2- 172.21.15.42:0/4126801806 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f755c003940 0x7f755c003d90 secure :-1 s=READY pgs=826 cs=0 l=1 rx=0x7f7574002f30 tx=0x7f757400ffe0).ready entity=mon.1 client_cookie=565ee0288422e466 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:25:36.158 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:36.145+0000 7f7580870700 1 --2- 172.21.15.42:0/4126801806 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f755c003940 0x7f755c003d90 secure :-1 s=THROTTLE_DONE pgs=826 cs=0 l=1 rx=0x7f7574002f30 tx=0x7f757400ffe0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:36.158 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:36.145+0000 7f757b7fe700 1 -- 172.21.15.42:0/4126801806 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7574010d10 con 0x7f755c003940 2019-12-05T05:25:36.158 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:36.145+0000 7f7580870700 1 --2- 172.21.15.42:0/4126801806 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f755c003940 0x7f755c003d90 secure :-1 s=THROTTLE_DONE pgs=826 cs=0 l=1 rx=0x7f7574002f30 tx=0x7f757400ffe0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:36.158 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:36.145+0000 7f7580870700 1 --2- 172.21.15.42:0/4126801806 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f755c003940 0x7f755c003d90 secure :-1 s=THROTTLE_DONE pgs=826 cs=0 l=1 rx=0x7f7574002f30 tx=0x7f757400ffe0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:36.158 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:36.145+0000 7f757b7fe700 1 -- 172.21.15.42:0/4126801806 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7574010930 con 0x7f755c003940 2019-12-05T05:25:36.158 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:36.145+0000 7f757b7fe700 1 -- 172.21.15.42:0/4126801806 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7574008bb0 con 0x7f755c003940 2019-12-05T05:25:36.159 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:36.145+0000 7f75822d3700 1 -- 172.21.15.42:0/4126801806 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f755c003940 msgr2=0x7f755c003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:25:36.159 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:36.145+0000 7f75822d3700 1 --2- 172.21.15.42:0/4126801806 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f755c003940 0x7f755c003d90 secure :-1 s=READY pgs=826 cs=0 l=1 rx=0x7f7574002f30 tx=0x7f757400ffe0).stop 2019-12-05T05:25:36.159 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:36.145+0000 7f7581071700 1 -- 172.21.15.42:0/4126801806 reap_dead start 2019-12-05T05:25:36.159 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:36.145+0000 7f75822d3700 1 -- 172.21.15.42:0/4126801806 shutdown_connections 2019-12-05T05:25:36.159 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:36.145+0000 7f75822d3700 1 -- 172.21.15.42:0/4126801806 shutdown_connections 2019-12-05T05:25:36.159 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:36.145+0000 7f75822d3700 1 -- 172.21.15.42:0/4126801806 wait complete. 2019-12-05T05:25:36.159 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:36.145+0000 7f75822d3700 1 -- 172.21.15.42:0/4126801806 >> 172.21.15.42:0/4126801806 conn(0x7f757c0aabe0 msgr2=0x7f757c0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:25:36.163 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:36.149+0000 7f75822d3700 1 Processor -- start 2019-12-05T05:25:36.163 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:36.149+0000 7f75822d3700 1 -- start start 2019-12-05T05:25:36.163 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:36.153+0000 7f75822d3700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f757c118f30 0x7f757c1432f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:25:36.163 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:36.153+0000 7f75822d3700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f757c0a5040 0x7f757c07b370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:25:36.163 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:36.153+0000 7f75822d3700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f757c142530 con 0x7f757c0a5970 2019-12-05T05:25:36.164 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:36.153+0000 7f75822d3700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f757c0a6c70 con 0x7f757c0a5040 2019-12-05T05:25:36.164 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:36.153+0000 7f75822d3700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f757c0a5f20 con 0x7f757c118f30 2019-12-05T05:25:36.164 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:36.153+0000 7f7580870700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f757c118f30 0x7f757c1432f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:25:36.164 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:36.153+0000 7f7580870700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f757c118f30 0x7f757c1432f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:33874/0 (socket says 172.21.15.42:33874) 2019-12-05T05:25:36.164 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:36.153+0000 7f7580870700 1 -- 172.21.15.42:0/408904815 learned_addr learned my addr 172.21.15.42:0/408904815 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:25:36.164 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:36.153+0000 7f7581071700 1 --2- 172.21.15.42:0/408904815 >> v2:172.21.15.40:3300/0 conn(0x7f757c0a5040 0x7f757c07b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:25:36.164 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:36.153+0000 7f75797fa700 1 -- 172.21.15.42:0/408904815 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 290793617 0 0) 0x7f757c142530 con 0x7f757c0a5970 2019-12-05T05:25:36.165 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:36.153+0000 7f75797fa700 1 -- 172.21.15.42:0/408904815 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75540030b0 con 0x7f757c0a5970 2019-12-05T05:25:36.165 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:36.153+0000 7f7580870700 1 -- 172.21.15.42:0/408904815 >> v2:172.21.15.40:3300/0 conn(0x7f757c0a5040 msgr2=0x7f757c07b370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:25:36.165 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:36.153+0000 7f7580870700 1 --2- 172.21.15.42:0/408904815 >> v2:172.21.15.40:3300/0 conn(0x7f757c0a5040 0x7f757c07b370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:25:36.165 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:36.153+0000 7f7580870700 1 -- 172.21.15.42:0/408904815 >> v1:172.21.15.40:6789/0 conn(0x7f757c0a5970 legacy=0x7f757c0a4840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:25:36.165 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:36.153+0000 7f7580870700 1 -- 172.21.15.42:0/408904815 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f757c0a6460 con 0x7f757c118f30 2019-12-05T05:25:36.165 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:36.153+0000 7f7580870700 1 --2- 172.21.15.42:0/408904815 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f757c118f30 0x7f757c1432f0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f7574002f30 tx=0x7f7574008f40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:36.165 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:36.153+0000 7f7580870700 1 --2- 172.21.15.42:0/408904815 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f757c118f30 0x7f757c1432f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f7574002f30 tx=0x7f7574008f40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:36.166 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:36.153+0000 7f7580870700 1 --2- 172.21.15.42:0/408904815 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f757c118f30 0x7f757c1432f0 secure :-1 s=READY pgs=170 cs=0 l=1 rx=0x7f7574002f30 tx=0x7f7574008f40).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:25:36.166 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:36.153+0000 7f7580870700 1 --2- 172.21.15.42:0/408904815 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f757c118f30 0x7f757c1432f0 secure :-1 s=THROTTLE_DONE pgs=170 cs=0 l=1 rx=0x7f7574002f30 tx=0x7f7574008f40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:36.166 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:36.153+0000 7f75797fa700 1 -- 172.21.15.42:0/408904815 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f757400f1f0 con 0x7f757c118f30 2019-12-05T05:25:36.166 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:36.153+0000 7f7580870700 1 --2- 172.21.15.42:0/408904815 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f757c118f30 0x7f757c1432f0 secure :-1 s=THROTTLE_DONE pgs=170 cs=0 l=1 rx=0x7f7574002f30 tx=0x7f7574008f40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:36.166 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:36.153+0000 7f75797fa700 1 -- 172.21.15.42:0/408904815 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75740087d0 con 0x7f757c118f30 2019-12-05T05:25:36.166 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:36.153+0000 7f75822d3700 1 -- 172.21.15.42:0/408904815 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f757c09f2e0 con 0x7f757c118f30 2019-12-05T05:25:36.166 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:36.153+0000 7f7580870700 1 --2- 172.21.15.42:0/408904815 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f757c118f30 0x7f757c1432f0 secure :-1 s=THROTTLE_DONE pgs=170 cs=0 l=1 rx=0x7f7574002f30 tx=0x7f7574008f40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:36.167 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:36.153+0000 7f75797fa700 1 -- 172.21.15.42:0/408904815 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f757400f7d0 con 0x7f757c118f30 2019-12-05T05:25:36.167 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:36.153+0000 7f75822d3700 1 -- 172.21.15.42:0/408904815 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f757c09f890 con 0x7f757c118f30 2019-12-05T05:25:36.167 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:36.153+0000 7f7580870700 1 --2- 172.21.15.42:0/408904815 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f757c118f30 0x7f757c1432f0 secure :-1 s=THROTTLE_DONE pgs=170 cs=0 l=1 rx=0x7f7574002f30 tx=0x7f7574008f40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:36.167 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:36.157+0000 7f7580870700 1 --2- 172.21.15.42:0/408904815 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f757c118f30 0x7f757c1432f0 secure :-1 s=THROTTLE_DONE pgs=170 cs=0 l=1 rx=0x7f7574002f30 tx=0x7f7574008f40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:36.167 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:36.157+0000 7f75797fa700 1 -- 172.21.15.42:0/408904815 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f7574005540 con 0x7f757c118f30 2019-12-05T05:25:36.167 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:36.157+0000 7f75797fa700 1 --2- 172.21.15.42:0/408904815 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f755403b270 0x7f755403d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:25:36.167 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:36.157+0000 7f75797fa700 1 -- 172.21.15.42:0/408904815 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7321+0+0 (secure 0 0 0) 0x7f757404b0b0 con 0x7f757c118f30 2019-12-05T05:25:36.168 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:36.157+0000 7f757bfff700 1 --2- 172.21.15.42:0/408904815 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f755403b270 0x7f755403d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:25:36.168 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:36.157+0000 7f757bfff700 1 --2- 172.21.15.42:0/408904815 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f755403b270 0x7f755403d780 crc :-1 s=READY pgs=365 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:25:36.168 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:36.157+0000 7f75822d3700 1 -- 172.21.15.42:0/408904815 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f757c05d110 con 0x7f757c118f30 2019-12-05T05:25:36.172 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:36.169+0000 7f7580870700 1 --2- 172.21.15.42:0/408904815 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f757c118f30 0x7f757c1432f0 secure :-1 s=THROTTLE_DONE pgs=170 cs=0 l=1 rx=0x7f7574002f30 tx=0x7f7574008f40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:36.173 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:36.169+0000 7f75797fa700 1 -- 172.21.15.42:0/408904815 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f757c0a6460 con 0x7f757c118f30 2019-12-05T05:25:36.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:36.333+0000 7f75822d3700 1 -- 172.21.15.42:0/408904815 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f757c05b8c0 con 0x7f757c118f30 2019-12-05T05:25:36.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:36.333+0000 7f7580870700 1 --2- 172.21.15.42:0/408904815 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f757c118f30 0x7f757c1432f0 secure :-1 s=THROTTLE_DONE pgs=170 cs=0 l=1 rx=0x7f7574002f30 tx=0x7f7574008f40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:36.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:36.333+0000 7f75797fa700 1 -- 172.21.15.42:0/408904815 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) v1 ==== 95+0+1778 (secure 0 0 0) 0x7f757402fd30 con 0x7f757c118f30 2019-12-05T05:25:36.341 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 105 2019-12-05T05:25:36.341 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:25:36.342 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":105,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:25:36.075103+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5240},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5240":{"gid":5240,"name":"a","rank":0,"incarnation":104,"state":"up:reconnect","state_seq":25,"addr":"172.21.15.42:6835/1455108684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1455108684},{"type":"v1","addr":"172.21.15.42:6835","nonce":1455108684}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:25:36.344 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:36.337+0000 7f75822d3700 1 -- 172.21.15.42:0/408904815 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f755403b270 msgr2=0x7f755403d780 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:25:36.344 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:36.337+0000 7f75822d3700 1 --2- 172.21.15.42:0/408904815 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f755403b270 0x7f755403d780 crc :-1 s=READY pgs=365 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:25:36.344 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:36.337+0000 7f75822d3700 1 -- 172.21.15.42:0/408904815 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f757c118f30 msgr2=0x7f757c1432f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:25:36.344 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:36.337+0000 7f75822d3700 1 --2- 172.21.15.42:0/408904815 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f757c118f30 0x7f757c1432f0 secure :-1 s=READY pgs=170 cs=0 l=1 rx=0x7f7574002f30 tx=0x7f7574008f40).stop 2019-12-05T05:25:36.345 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:36.341+0000 7f75822d3700 1 -- 172.21.15.42:0/408904815 shutdown_connections 2019-12-05T05:25:36.345 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:36.341+0000 7f75822d3700 1 --2- 172.21.15.42:0/408904815 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f755403b270 0x7f755403d780 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:25:36.345 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:36.341+0000 7f75822d3700 1 --2- 172.21.15.42:0/408904815 >> v2:172.21.15.40:3300/0 conn(0x7f757c0a5040 0x7f757c07b370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:25:36.345 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:36.341+0000 7f75822d3700 1 --2- 172.21.15.42:0/408904815 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f757c118f30 0x7f757c1432f0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:25:36.346 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:36.341+0000 7f75822d3700 1 -- 172.21.15.42:0/408904815 shutdown_connections 2019-12-05T05:25:36.346 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:36.341+0000 7f75822d3700 1 -- 172.21.15.42:0/408904815 wait complete. 2019-12-05T05:25:36.346 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:36.341+0000 7f75822d3700 1 -- 172.21.15.42:0/408904815 >> 172.21.15.42:0/408904815 conn(0x7f757c08e6b0 msgr2=0x7f757c059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:25:36.354 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2019-12-05T05:25:38.355 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:25:38.396 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:25:38.400 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:25:38.458 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:38.449+0000 7f5d4b259700 1 Processor -- start 2019-12-05T05:25:38.459 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:38.449+0000 7f5d4b259700 1 -- start start 2019-12-05T05:25:38.459 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:38.449+0000 7f5d4b259700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5d440a7740 0x7f5d4408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:25:38.459 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:38.449+0000 7f5d4b259700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5d4407e720 0x7f5d4407eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:25:38.462 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:38.449+0000 7f5d4b259700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d440a0710 con 0x7f5d4409f2e0 2019-12-05T05:25:38.462 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:38.449+0000 7f5d4b259700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d440a0b00 con 0x7f5d4407e720 2019-12-05T05:25:38.462 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:38.449+0000 7f5d4b259700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d4408e5b0 con 0x7f5d440a7740 2019-12-05T05:25:38.462 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:38.449+0000 7f5d48ff5700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5d4407e720 0x7f5d4407eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:25:38.463 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:38.449+0000 7f5d48ff5700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5d4407e720 0x7f5d4407eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:33424/0 (socket says 172.21.15.42:33424) 2019-12-05T05:25:38.463 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:38.449+0000 7f5d48ff5700 1 -- 172.21.15.42:0/2966457379 learned_addr learned my addr 172.21.15.42:0/2966457379 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:25:38.463 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:38.449+0000 7f5d497f6700 1 --2- 172.21.15.42:0/2966457379 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5d440a7740 0x7f5d4408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:25:38.463 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:38.449+0000 7f5d48ff5700 1 -- 172.21.15.42:0/2966457379 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5d440a7740 msgr2=0x7f5d4408df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:25:38.463 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:38.449+0000 7f5d48ff5700 1 --2- 172.21.15.42:0/2966457379 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5d440a7740 0x7f5d4408df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:25:38.463 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:38.449+0000 7f5d48ff5700 1 -- 172.21.15.42:0/2966457379 >> v1:172.21.15.40:6789/0 conn(0x7f5d4409f2e0 legacy=0x7f5d4409f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:25:38.463 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:38.453+0000 7f5d3bfff700 1 -- 172.21.15.42:0/2966457379 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1806366658 0 0) 0x7f5d440a0710 con 0x7f5d4409f2e0 2019-12-05T05:25:38.464 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:38.453+0000 7f5d48ff5700 1 -- 172.21.15.42:0/2966457379 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d4407f0a0 con 0x7f5d4407e720 2019-12-05T05:25:38.464 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:38.453+0000 7f5d497f6700 1 --2- 172.21.15.42:0/2966457379 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5d440a7740 0x7f5d4408df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T05:25:38.464 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:38.453+0000 7f5d48ff5700 1 --2- 172.21.15.42:0/2966457379 >> v2:172.21.15.42:3300/0 conn(0x7f5d4407e720 0x7f5d4407eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f5d40008790 tx=0x7f5d4000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:38.464 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:38.453+0000 7f5d48ff5700 1 --2- 172.21.15.42:0/2966457379 >> v2:172.21.15.42:3300/0 conn(0x7f5d4407e720 0x7f5d4407eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f5d40008790 tx=0x7f5d4000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:38.464 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:38.453+0000 7f5d48ff5700 1 --2- 172.21.15.42:0/2966457379 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5d4407e720 0x7f5d4407eb50 secure :-1 s=READY pgs=827 cs=0 l=1 rx=0x7f5d40008790 tx=0x7f5d4000fd30).ready entity=mon.1 client_cookie=484e33226ac6c03d server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:25:38.464 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:38.453+0000 7f5d48ff5700 1 --2- 172.21.15.42:0/2966457379 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5d4407e720 0x7f5d4407eb50 secure :-1 s=THROTTLE_DONE pgs=827 cs=0 l=1 rx=0x7f5d40008790 tx=0x7f5d4000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:38.464 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:38.453+0000 7f5d3bfff700 1 -- 172.21.15.42:0/2966457379 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5d400113b0 con 0x7f5d4407e720 2019-12-05T05:25:38.465 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:38.453+0000 7f5d48ff5700 1 --2- 172.21.15.42:0/2966457379 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5d4407e720 0x7f5d4407eb50 secure :-1 s=THROTTLE_DONE pgs=827 cs=0 l=1 rx=0x7f5d40008790 tx=0x7f5d4000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:38.465 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:38.453+0000 7f5d48ff5700 1 --2- 172.21.15.42:0/2966457379 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5d4407e720 0x7f5d4407eb50 secure :-1 s=THROTTLE_DONE pgs=827 cs=0 l=1 rx=0x7f5d40008790 tx=0x7f5d4000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:38.465 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:38.453+0000 7f5d3bfff700 1 -- 172.21.15.42:0/2966457379 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d40008bb0 con 0x7f5d4407e720 2019-12-05T05:25:38.465 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:38.453+0000 7f5d3bfff700 1 -- 172.21.15.42:0/2966457379 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5d40011990 con 0x7f5d4407e720 2019-12-05T05:25:38.465 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:38.453+0000 7f5d4b259700 1 -- 172.21.15.42:0/2966457379 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5d4407e720 msgr2=0x7f5d4407eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:25:38.465 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:38.453+0000 7f5d4b259700 1 --2- 172.21.15.42:0/2966457379 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5d4407e720 0x7f5d4407eb50 secure :-1 s=READY pgs=827 cs=0 l=1 rx=0x7f5d40008790 tx=0x7f5d4000fd30).stop 2019-12-05T05:25:38.465 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:38.453+0000 7f5d4b259700 1 -- 172.21.15.42:0/2966457379 shutdown_connections 2019-12-05T05:25:38.466 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:38.453+0000 7f5d4b259700 1 --2- 172.21.15.42:0/2966457379 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5d440a7740 0x7f5d4408df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:25:38.466 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:38.453+0000 7f5d4b259700 1 --2- 172.21.15.42:0/2966457379 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5d4407e720 0x7f5d4407eb50 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:25:38.466 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:38.453+0000 7f5d4b259700 1 -- 172.21.15.42:0/2966457379 shutdown_connections 2019-12-05T05:25:38.466 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:38.453+0000 7f5d4b259700 1 -- 172.21.15.42:0/2966457379 wait complete. 2019-12-05T05:25:38.466 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:38.453+0000 7f5d4b259700 1 -- 172.21.15.42:0/2966457379 >> 172.21.15.42:0/2966457379 conn(0x7f5d440aabe0 msgr2=0x7f5d440a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:25:38.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:38.453+0000 7f5d4b259700 1 Processor -- start 2019-12-05T05:25:38.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:38.453+0000 7f5d4b259700 1 -- start start 2019-12-05T05:25:38.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:38.457+0000 7f5d4b259700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5d44123f70 0x7f5d44077b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:25:38.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:38.457+0000 7f5d4b259700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d44064f00 con 0x7f5d44133380 2019-12-05T05:25:38.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:38.457+0000 7f5d4b259700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d44064bf0 con 0x7f5d44065750 2019-12-05T05:25:38.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:38.457+0000 7f5d4b259700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d44065b80 con 0x7f5d44123f70 2019-12-05T05:25:38.482 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:38.457+0000 7f5d49ff7700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5d44123f70 0x7f5d44077b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:25:38.482 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:38.457+0000 7f5d497f6700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f5d44065750 0x7f5d440643f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:33466/0 (socket says 172.21.15.42:33466) 2019-12-05T05:25:38.483 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:38.457+0000 7f5d497f6700 1 -- 172.21.15.42:0/174108475 learned_addr learned my addr 172.21.15.42:0/174108475 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:25:38.483 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:38.457+0000 7f5d49ff7700 1 -- 172.21.15.42:0/174108475 >> v1:172.21.15.42:6789/0 conn(0x7f5d44065750 legacy=0x7f5d440643f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:25:38.483 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:38.457+0000 7f5d49ff7700 1 -- 172.21.15.42:0/174108475 >> v1:172.21.15.40:6789/0 conn(0x7f5d44133380 legacy=0x7f5d440742e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:25:38.483 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:38.457+0000 7f5d39ffb700 1 -- 172.21.15.42:0/174108475 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1726091515 0 0) 0x7f5d44064bf0 con 0x7f5d44065750 2019-12-05T05:25:38.483 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:38.457+0000 7f5d49ff7700 1 -- 172.21.15.42:0/174108475 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d441337b0 con 0x7f5d44123f70 2019-12-05T05:25:38.483 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:38.457+0000 7f5d49ff7700 1 --2- 172.21.15.42:0/174108475 >> v2:172.21.15.42:3300/0 conn(0x7f5d44123f70 0x7f5d44077b00 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f5d34000b80 tx=0x7f5d34010700).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:38.484 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:38.457+0000 7f5d49ff7700 1 --2- 172.21.15.42:0/174108475 >> v2:172.21.15.42:3300/0 conn(0x7f5d44123f70 0x7f5d44077b00 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f5d34000b80 tx=0x7f5d34010700).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:38.484 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:38.457+0000 7f5d49ff7700 1 --2- 172.21.15.42:0/174108475 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5d44123f70 0x7f5d44077b00 secure :-1 s=READY pgs=829 cs=0 l=1 rx=0x7f5d34000b80 tx=0x7f5d34010700).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:25:38.484 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:38.457+0000 7f5d49ff7700 1 --2- 172.21.15.42:0/174108475 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5d44123f70 0x7f5d44077b00 secure :-1 s=THROTTLE_DONE pgs=829 cs=0 l=1 rx=0x7f5d34000b80 tx=0x7f5d34010700).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:38.484 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:38.457+0000 7f5d39ffb700 1 -- 172.21.15.42:0/174108475 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5d34006b30 con 0x7f5d44123f70 2019-12-05T05:25:38.484 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:38.457+0000 7f5d49ff7700 1 --2- 172.21.15.42:0/174108475 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5d44123f70 0x7f5d44077b00 secure :-1 s=THROTTLE_DONE pgs=829 cs=0 l=1 rx=0x7f5d34000b80 tx=0x7f5d34010700).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:38.484 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:38.457+0000 7f5d49ff7700 1 --2- 172.21.15.42:0/174108475 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5d44123f70 0x7f5d44077b00 secure :-1 s=THROTTLE_DONE pgs=829 cs=0 l=1 rx=0x7f5d34000b80 tx=0x7f5d34010700).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:38.485 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:38.457+0000 7f5d39ffb700 1 -- 172.21.15.42:0/174108475 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d34006750 con 0x7f5d44123f70 2019-12-05T05:25:38.485 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:38.457+0000 7f5d4b259700 1 -- 172.21.15.42:0/174108475 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d441243a0 con 0x7f5d44123f70 2019-12-05T05:25:38.485 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:38.457+0000 7f5d39ffb700 1 -- 172.21.15.42:0/174108475 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5d34010f40 con 0x7f5d44123f70 2019-12-05T05:25:38.486 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:38.457+0000 7f5d4b259700 1 -- 172.21.15.42:0/174108475 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d44078050 con 0x7f5d44123f70 2019-12-05T05:25:38.486 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:38.457+0000 7f5d49ff7700 1 --2- 172.21.15.42:0/174108475 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5d44123f70 0x7f5d44077b00 secure :-1 s=THROTTLE_DONE pgs=829 cs=0 l=1 rx=0x7f5d34000b80 tx=0x7f5d34010700).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:38.486 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:38.461+0000 7f5d39ffb700 1 -- 172.21.15.42:0/174108475 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f5d34011250 con 0x7f5d44123f70 2019-12-05T05:25:38.486 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:38.461+0000 7f5d49ff7700 1 --2- 172.21.15.42:0/174108475 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5d44123f70 0x7f5d44077b00 secure :-1 s=THROTTLE_DONE pgs=829 cs=0 l=1 rx=0x7f5d34000b80 tx=0x7f5d34010700).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:38.487 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:38.461+0000 7f5d39ffb700 1 --2- 172.21.15.42:0/174108475 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5d2403ac20 0x7f5d2403d130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:25:38.487 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:38.461+0000 7f5d39ffb700 1 -- 172.21.15.42:0/174108475 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7321+0+0 (secure 0 0 0) 0x7f5d34011dc0 con 0x7f5d44123f70 2019-12-05T05:25:38.487 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:38.461+0000 7f5d497f6700 1 --2- 172.21.15.42:0/174108475 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5d2403ac20 0x7f5d2403d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:25:38.487 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:38.461+0000 7f5d497f6700 1 --2- 172.21.15.42:0/174108475 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5d2403ac20 0x7f5d2403d130 crc :-1 s=READY pgs=366 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:25:38.487 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:38.465+0000 7f5d4b259700 1 -- 172.21.15.42:0/174108475 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d4405d110 con 0x7f5d44123f70 2019-12-05T05:25:38.487 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:38.473+0000 7f5d49ff7700 1 --2- 172.21.15.42:0/174108475 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5d44123f70 0x7f5d44077b00 secure :-1 s=THROTTLE_DONE pgs=829 cs=0 l=1 rx=0x7f5d34000b80 tx=0x7f5d34010700).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:38.487 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:38.473+0000 7f5d39ffb700 1 -- 172.21.15.42:0/174108475 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f5d34032140 con 0x7f5d44123f70 2019-12-05T05:25:38.641 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:38.637+0000 7f5d4b259700 1 -- 172.21.15.42:0/174108475 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5d4405b8c0 con 0x7f5d44123f70 2019-12-05T05:25:38.642 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:38.637+0000 7f5d49ff7700 1 --2- 172.21.15.42:0/174108475 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5d44123f70 0x7f5d44077b00 secure :-1 s=THROTTLE_DONE pgs=829 cs=0 l=1 rx=0x7f5d34000b80 tx=0x7f5d34010700).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:38.642 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:38.637+0000 7f5d39ffb700 1 -- 172.21.15.42:0/174108475 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+1775 (secure 0 0 0) 0x7f5d340323a0 con 0x7f5d44123f70 2019-12-05T05:25:38.642 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 107 2019-12-05T05:25:38.642 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:25:38.643 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":107,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:25:38.087763+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5240},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5240":{"gid":5240,"name":"a","rank":0,"incarnation":104,"state":"up:active","state_seq":27,"addr":"172.21.15.42:6835/1455108684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1455108684},{"type":"v1","addr":"172.21.15.42:6835","nonce":1455108684}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:25:38.645 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:38.641+0000 7f5d4b259700 1 -- 172.21.15.42:0/174108475 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5d2403ac20 msgr2=0x7f5d2403d130 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:25:38.645 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:38.641+0000 7f5d4b259700 1 --2- 172.21.15.42:0/174108475 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5d2403ac20 0x7f5d2403d130 crc :-1 s=READY pgs=366 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:25:38.645 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:38.641+0000 7f5d4b259700 1 -- 172.21.15.42:0/174108475 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5d44123f70 msgr2=0x7f5d44077b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:25:38.645 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:38.641+0000 7f5d4b259700 1 --2- 172.21.15.42:0/174108475 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5d44123f70 0x7f5d44077b00 secure :-1 s=READY pgs=829 cs=0 l=1 rx=0x7f5d34000b80 tx=0x7f5d34010700).stop 2019-12-05T05:25:38.646 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:38.641+0000 7f5d4b259700 1 -- 172.21.15.42:0/174108475 shutdown_connections 2019-12-05T05:25:38.646 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:38.641+0000 7f5d4b259700 1 --2- 172.21.15.42:0/174108475 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5d2403ac20 0x7f5d2403d130 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:25:38.646 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:38.641+0000 7f5d4b259700 1 --2- 172.21.15.42:0/174108475 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5d44123f70 0x7f5d44077b00 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:25:38.646 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:38.641+0000 7f5d4b259700 1 -- 172.21.15.42:0/174108475 shutdown_connections 2019-12-05T05:25:38.646 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:38.641+0000 7f5d4b259700 1 -- 172.21.15.42:0/174108475 wait complete. 2019-12-05T05:25:38.646 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:38.641+0000 7f5d4b259700 1 -- 172.21.15.42:0/174108475 >> 172.21.15.42:0/174108475 conn(0x7f5d440a6bb0 msgr2=0x7f5d4407e5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:25:38.655 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a has gained rank=0, replacing gid=5177 2019-12-05T05:25:38.655 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 6 secs before reviving mds.b 2019-12-05T05:25:44.656 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.b 2019-12-05T05:25:44.656 INFO:tasks.ceph.mds.b:Restarting daemon 2019-12-05T05:25:44.657 INFO:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2019-12-05T05:25:44.661 INFO:tasks.ceph.mds.b:Started 2019-12-05T05:25:44.662 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:25:44.722 INFO:tasks.ceph.mds.b.smithi040.stdout:starting mds.b at 2019-12-05T05:25:44.764 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:44.753+0000 7fdf518a9700 1 Processor -- start 2019-12-05T05:25:44.764 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:44.753+0000 7fdf518a9700 1 -- start start 2019-12-05T05:25:44.765 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:44.753+0000 7fdf518a9700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fdf4c0a7730 0x7fdf4c08df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:25:44.766 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:44.753+0000 7fdf518a9700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf4c0a0700 con 0x7fdf4c09fad0 2019-12-05T05:25:44.766 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:44.753+0000 7fdf518a9700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf4c0a9b90 con 0x7fdf4c07e710 2019-12-05T05:25:44.766 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:44.753+0000 7fdf518a9700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf4c0a0af0 con 0x7fdf4c0a7730 2019-12-05T05:25:44.766 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:44.753+0000 7fdf4affd700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fdf4c07e710 0x7fdf4c09f2d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:33498/0 (socket says 172.21.15.42:33498) 2019-12-05T05:25:44.766 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:44.753+0000 7fdf4b7fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fdf4c0a7730 0x7fdf4c08df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:25:44.767 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:44.753+0000 7fdf4affd700 1 -- 172.21.15.42:0/3901695222 learned_addr learned my addr 172.21.15.42:0/3901695222 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:25:44.767 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:44.753+0000 7fdf4a7fc700 1 -- 172.21.15.42:0/3901695222 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 5995454 0 0) 0x7fdf4c0a9b90 con 0x7fdf4c07e710 2019-12-05T05:25:44.767 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:44.753+0000 7fdf4a7fc700 1 -- 172.21.15.42:0/3901695222 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf340030b0 con 0x7fdf4c07e710 2019-12-05T05:25:44.767 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:44.753+0000 7fdf4a7fc700 1 -- 172.21.15.42:0/3901695222 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2980686091 0 0) 0x7fdf4c0a0700 con 0x7fdf4c09fad0 2019-12-05T05:25:44.767 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:44.753+0000 7fdf4a7fc700 1 -- 172.21.15.42:0/3901695222 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf4c0a9b90 con 0x7fdf4c09fad0 2019-12-05T05:25:44.768 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:44.753+0000 7fdf4a7fc700 1 -- 172.21.15.42:0/3901695222 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1014438271 0 0) 0x7fdf340030b0 con 0x7fdf4c07e710 2019-12-05T05:25:44.768 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:44.753+0000 7fdf4b7fe700 1 -- 172.21.15.42:0/3901695222 >> v1:172.21.15.42:6789/0 conn(0x7fdf4c07e710 legacy=0x7fdf4c09f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:25:44.768 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:44.753+0000 7fdf4b7fe700 1 -- 172.21.15.42:0/3901695222 >> v1:172.21.15.40:6789/0 conn(0x7fdf4c09fad0 legacy=0x7fdf4c14efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:25:44.768 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:44.757+0000 7fdf4b7fe700 1 -- 172.21.15.42:0/3901695222 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf4c07eb40 con 0x7fdf4c0a7730 2019-12-05T05:25:44.768 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:44.757+0000 7fdf4b7fe700 1 --2- 172.21.15.42:0/3901695222 >> v2:172.21.15.42:3300/0 conn(0x7fdf4c0a7730 0x7fdf4c08df70 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fdf3c008790 tx=0x7fdf40001950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:44.768 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:44.757+0000 7fdf4b7fe700 1 --2- 172.21.15.42:0/3901695222 >> v2:172.21.15.42:3300/0 conn(0x7fdf4c0a7730 0x7fdf4c08df70 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fdf3c008790 tx=0x7fdf40001950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:44.769 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:44.757+0000 7fdf4b7fe700 1 --2- 172.21.15.42:0/3901695222 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdf4c0a7730 0x7fdf4c08df70 secure :-1 s=READY pgs=834 cs=0 l=1 rx=0x7fdf3c008790 tx=0x7fdf40001950).ready entity=mon.1 client_cookie=23d50c4b60e096bb server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:25:44.769 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:44.757+0000 7fdf4b7fe700 1 --2- 172.21.15.42:0/3901695222 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdf4c0a7730 0x7fdf4c08df70 secure :-1 s=THROTTLE_DONE pgs=834 cs=0 l=1 rx=0x7fdf3c008790 tx=0x7fdf40001950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:44.769 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:44.757+0000 7fdf4a7fc700 1 -- 172.21.15.42:0/3901695222 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdf40001750 con 0x7fdf4c0a7730 2019-12-05T05:25:44.769 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:44.757+0000 7fdf4b7fe700 1 --2- 172.21.15.42:0/3901695222 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdf4c0a7730 0x7fdf4c08df70 secure :-1 s=THROTTLE_DONE pgs=834 cs=0 l=1 rx=0x7fdf3c008790 tx=0x7fdf40001950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:44.769 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:44.757+0000 7fdf4b7fe700 1 --2- 172.21.15.42:0/3901695222 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdf4c0a7730 0x7fdf4c08df70 secure :-1 s=THROTTLE_DONE pgs=834 cs=0 l=1 rx=0x7fdf3c008790 tx=0x7fdf40001950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:44.769 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:44.757+0000 7fdf4a7fc700 1 -- 172.21.15.42:0/3901695222 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf40001560 con 0x7fdf4c0a7730 2019-12-05T05:25:44.770 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:44.757+0000 7fdf4a7fc700 1 -- 172.21.15.42:0/3901695222 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdf3c011360 con 0x7fdf4c0a7730 2019-12-05T05:25:44.770 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:44.757+0000 7fdf518a9700 1 -- 172.21.15.42:0/3901695222 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdf4c0a7730 msgr2=0x7fdf4c08df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:25:44.770 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:44.757+0000 7fdf518a9700 1 --2- 172.21.15.42:0/3901695222 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdf4c0a7730 0x7fdf4c08df70 secure :-1 s=READY pgs=834 cs=0 l=1 rx=0x7fdf3c008790 tx=0x7fdf40001950).stop 2019-12-05T05:25:44.770 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:44.757+0000 7fdf518a9700 1 -- 172.21.15.42:0/3901695222 shutdown_connections 2019-12-05T05:25:44.771 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:44.757+0000 7fdf518a9700 1 --2- 172.21.15.42:0/3901695222 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdf4c0a7730 0x7fdf4c08df70 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:25:44.771 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:44.757+0000 7fdf518a9700 1 -- 172.21.15.42:0/3901695222 shutdown_connections 2019-12-05T05:25:44.771 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:44.757+0000 7fdf518a9700 1 -- 172.21.15.42:0/3901695222 wait complete. 2019-12-05T05:25:44.771 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:44.757+0000 7fdf518a9700 1 -- 172.21.15.42:0/3901695222 >> 172.21.15.42:0/3901695222 conn(0x7fdf4c0aabd0 msgr2=0x7fdf4c0a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:25:44.775 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:44.761+0000 7fdf518a9700 1 Processor -- start 2019-12-05T05:25:44.776 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:44.761+0000 7fdf518a9700 1 -- start start 2019-12-05T05:25:44.776 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:44.761+0000 7fdf518a9700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fdf4c1244f0 0x7fdf4c120b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:25:44.776 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:44.761+0000 7fdf518a9700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf4c1190d0 con 0x7fdf4c11ef30 2019-12-05T05:25:44.777 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:44.761+0000 7fdf518a9700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf4c121060 con 0x7fdf4c0722b0 2019-12-05T05:25:44.777 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:44.761+0000 7fdf518a9700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf4c124920 con 0x7fdf4c1244f0 2019-12-05T05:25:44.777 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:44.761+0000 7fdf4b7fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fdf4c1244f0 0x7fdf4c120b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:25:44.777 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:44.761+0000 7fdf4b7fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fdf4c1244f0 0x7fdf4c120b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:33464/0 (socket says 172.21.15.42:33464) 2019-12-05T05:25:44.777 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:44.761+0000 7fdf4b7fe700 1 -- 172.21.15.42:0/409007527 learned_addr learned my addr 172.21.15.42:0/409007527 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:25:44.778 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:44.761+0000 7fdf4bfff700 1 --1- 172.21.15.42:0/409007527 >> v1:172.21.15.42:6789/0 conn(0x7fdf4c0722b0 0x7fdf4c0746f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:33506/0 (socket says 172.21.15.42:33506) 2019-12-05T05:25:44.778 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:44.761+0000 7fdf33fff700 1 -- 172.21.15.42:0/409007527 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2159475969 0 0) 0x7fdf4c1190d0 con 0x7fdf4c11ef30 2019-12-05T05:25:44.778 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:44.761+0000 7fdf33fff700 1 -- 172.21.15.42:0/409007527 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf2c0030b0 con 0x7fdf4c11ef30 2019-12-05T05:25:44.778 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:44.761+0000 7fdf4b7fe700 1 -- 172.21.15.42:0/409007527 >> v1:172.21.15.42:6789/0 conn(0x7fdf4c0722b0 legacy=0x7fdf4c0746f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:25:44.778 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:44.761+0000 7fdf4b7fe700 1 -- 172.21.15.42:0/409007527 >> v1:172.21.15.40:6789/0 conn(0x7fdf4c11ef30 legacy=0x7fdf4c11f360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:25:44.778 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:44.761+0000 7fdf4b7fe700 1 -- 172.21.15.42:0/409007527 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf4c077f20 con 0x7fdf4c1244f0 2019-12-05T05:25:44.779 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:44.761+0000 7fdf4b7fe700 1 --2- 172.21.15.42:0/409007527 >> v2:172.21.15.42:3300/0 conn(0x7fdf4c1244f0 0x7fdf4c120b10 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fdf3c008790 tx=0x7fdf3c0086a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:44.779 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:44.765+0000 7fdf4b7fe700 1 --2- 172.21.15.42:0/409007527 >> v2:172.21.15.42:3300/0 conn(0x7fdf4c1244f0 0x7fdf4c120b10 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fdf3c008790 tx=0x7fdf3c0086a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:44.779 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:44.765+0000 7fdf4b7fe700 1 --2- 172.21.15.42:0/409007527 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdf4c1244f0 0x7fdf4c120b10 secure :-1 s=READY pgs=836 cs=0 l=1 rx=0x7fdf3c008790 tx=0x7fdf3c0086a0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:25:44.779 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:44.765+0000 7fdf4b7fe700 1 --2- 172.21.15.42:0/409007527 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdf4c1244f0 0x7fdf4c120b10 secure :-1 s=THROTTLE_DONE pgs=836 cs=0 l=1 rx=0x7fdf3c008790 tx=0x7fdf3c0086a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:44.779 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:44.765+0000 7fdf33fff700 1 -- 172.21.15.42:0/409007527 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdf3c007150 con 0x7fdf4c1244f0 2019-12-05T05:25:44.779 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:44.765+0000 7fdf4b7fe700 1 --2- 172.21.15.42:0/409007527 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdf4c1244f0 0x7fdf4c120b10 secure :-1 s=THROTTLE_DONE pgs=836 cs=0 l=1 rx=0x7fdf3c008790 tx=0x7fdf3c0086a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:44.780 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:44.765+0000 7fdf4b7fe700 1 --2- 172.21.15.42:0/409007527 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdf4c1244f0 0x7fdf4c120b10 secure :-1 s=THROTTLE_DONE pgs=836 cs=0 l=1 rx=0x7fdf3c008790 tx=0x7fdf3c0086a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:44.780 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:44.765+0000 7fdf33fff700 1 -- 172.21.15.42:0/409007527 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf3c008db0 con 0x7fdf4c1244f0 2019-12-05T05:25:44.780 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:44.765+0000 7fdf518a9700 1 -- 172.21.15.42:0/409007527 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf4c075f00 con 0x7fdf4c1244f0 2019-12-05T05:25:44.780 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:44.765+0000 7fdf33fff700 1 -- 172.21.15.42:0/409007527 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdf3c011860 con 0x7fdf4c1244f0 2019-12-05T05:25:44.781 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:44.765+0000 7fdf518a9700 1 -- 172.21.15.42:0/409007527 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf4c076450 con 0x7fdf4c1244f0 2019-12-05T05:25:44.781 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:44.765+0000 7fdf4b7fe700 1 --2- 172.21.15.42:0/409007527 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdf4c1244f0 0x7fdf4c120b10 secure :-1 s=THROTTLE_DONE pgs=836 cs=0 l=1 rx=0x7fdf3c008790 tx=0x7fdf3c0086a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:44.781 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:44.765+0000 7fdf33fff700 1 -- 172.21.15.42:0/409007527 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fdf3c011b70 con 0x7fdf4c1244f0 2019-12-05T05:25:44.781 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:44.765+0000 7fdf4b7fe700 1 --2- 172.21.15.42:0/409007527 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdf4c1244f0 0x7fdf4c120b10 secure :-1 s=THROTTLE_DONE pgs=836 cs=0 l=1 rx=0x7fdf3c008790 tx=0x7fdf3c0086a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:44.781 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:44.765+0000 7fdf33fff700 1 --2- 172.21.15.42:0/409007527 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fdf2c03b260 0x7fdf2c03d770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:25:44.782 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:44.765+0000 7fdf33fff700 1 -- 172.21.15.42:0/409007527 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7321+0+0 (secure 0 0 0) 0x7fdf3c04b1b0 con 0x7fdf4c1244f0 2019-12-05T05:25:44.782 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:44.765+0000 7fdf4affd700 1 --2- 172.21.15.42:0/409007527 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fdf2c03b260 0x7fdf2c03d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:25:44.782 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:44.765+0000 7fdf4affd700 1 --2- 172.21.15.42:0/409007527 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fdf2c03b260 0x7fdf2c03d770 crc :-1 s=READY pgs=367 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:25:44.782 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:44.769+0000 7fdf518a9700 1 -- 172.21.15.42:0/409007527 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf4c05d100 con 0x7fdf4c1244f0 2019-12-05T05:25:44.783 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:44.773+0000 7fdf4b7fe700 1 --2- 172.21.15.42:0/409007527 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdf4c1244f0 0x7fdf4c120b10 secure :-1 s=THROTTLE_DONE pgs=836 cs=0 l=1 rx=0x7fdf3c008790 tx=0x7fdf3c0086a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:44.783 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:44.773+0000 7fdf33fff700 1 -- 172.21.15.42:0/409007527 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fdf3c032490 con 0x7fdf4c1244f0 2019-12-05T05:25:44.962 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:44.957+0000 7fdf518a9700 1 -- 172.21.15.42:0/409007527 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdf4c05b8b0 con 0x7fdf4c1244f0 2019-12-05T05:25:44.963 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:44.957+0000 7fdf4b7fe700 1 --2- 172.21.15.42:0/409007527 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdf4c1244f0 0x7fdf4c120b10 secure :-1 s=THROTTLE_DONE pgs=836 cs=0 l=1 rx=0x7fdf3c008790 tx=0x7fdf3c0086a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:44.963 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:44.957+0000 7fdf33fff700 1 -- 172.21.15.42:0/409007527 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+1775 (secure 0 0 0) 0x7fdf3c033040 con 0x7fdf4c1244f0 2019-12-05T05:25:44.964 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 107 2019-12-05T05:25:44.964 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:25:44.964 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":107,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:25:38.087763+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5240},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5240":{"gid":5240,"name":"a","rank":0,"incarnation":104,"state":"up:active","state_seq":27,"addr":"172.21.15.42:6835/1455108684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1455108684},{"type":"v1","addr":"172.21.15.42:6835","nonce":1455108684}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:25:44.966 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:44.961+0000 7fdf518a9700 1 -- 172.21.15.42:0/409007527 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fdf2c03b260 msgr2=0x7fdf2c03d770 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:25:44.966 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:44.961+0000 7fdf518a9700 1 --2- 172.21.15.42:0/409007527 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fdf2c03b260 0x7fdf2c03d770 crc :-1 s=READY pgs=367 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:25:44.967 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:44.961+0000 7fdf518a9700 1 -- 172.21.15.42:0/409007527 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdf4c1244f0 msgr2=0x7fdf4c120b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:25:44.967 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:44.961+0000 7fdf518a9700 1 --2- 172.21.15.42:0/409007527 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdf4c1244f0 0x7fdf4c120b10 secure :-1 s=READY pgs=836 cs=0 l=1 rx=0x7fdf3c008790 tx=0x7fdf3c0086a0).stop 2019-12-05T05:25:44.967 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:44.961+0000 7fdf518a9700 1 -- 172.21.15.42:0/409007527 shutdown_connections 2019-12-05T05:25:44.967 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:44.961+0000 7fdf518a9700 1 --2- 172.21.15.42:0/409007527 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fdf2c03b260 0x7fdf2c03d770 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:25:44.967 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:44.961+0000 7fdf518a9700 1 --2- 172.21.15.42:0/409007527 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdf4c1244f0 0x7fdf4c120b10 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:25:44.968 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:44.961+0000 7fdf518a9700 1 -- 172.21.15.42:0/409007527 shutdown_connections 2019-12-05T05:25:44.968 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:44.961+0000 7fdf518a9700 1 -- 172.21.15.42:0/409007527 wait complete. 2019-12-05T05:25:44.968 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:44.961+0000 7fdf518a9700 1 -- 172.21.15.42:0/409007527 >> 172.21.15.42:0/409007527 conn(0x7fdf4c07ee10 msgr2=0x7fdf4c09f3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:25:44.976 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2019-12-05T05:25:46.977 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:25:47.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:47.077+0000 7fe1f9713700 1 Processor -- start 2019-12-05T05:25:47.088 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:47.077+0000 7fe1f9713700 1 -- start start 2019-12-05T05:25:47.088 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:47.077+0000 7fe1f9713700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe1f408df80 0x7fe1f409f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:25:47.088 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:47.077+0000 7fe1f9713700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe1f409f830 0x7fe1f414efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:25:47.088 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:47.077+0000 7fe1f9713700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1f409fc60 con 0x7fe1f40a7740 2019-12-05T05:25:47.088 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:47.077+0000 7fe1f9713700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1f40a0b00 con 0x7fe1f409f830 2019-12-05T05:25:47.088 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:47.077+0000 7fe1f9713700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1f408e5b0 con 0x7fe1f408df80 2019-12-05T05:25:47.088 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:47.077+0000 7fe1f37fe700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fe1f40a7740 0x7fe1f407e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:33522/0 (socket says 172.21.15.42:33522) 2019-12-05T05:25:47.089 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:47.077+0000 7fe1f37fe700 1 -- 172.21.15.42:0/3880954960 learned_addr learned my addr 172.21.15.42:0/3880954960 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:25:47.089 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:47.077+0000 7fe1f2ffd700 1 --2- 172.21.15.42:0/3880954960 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe1f408df80 0x7fe1f409f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:25:47.089 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:47.077+0000 7fe1f3fff700 1 --2- 172.21.15.42:0/3880954960 >> v2:172.21.15.40:3300/0 conn(0x7fe1f409f830 0x7fe1f414efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:25:47.089 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:47.081+0000 7fe1f27fc700 1 -- 172.21.15.42:0/3880954960 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1456822518 0 0) 0x7fe1f409fc60 con 0x7fe1f40a7740 2019-12-05T05:25:47.089 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:47.081+0000 7fe1f27fc700 1 -- 172.21.15.42:0/3880954960 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1dc0030b0 con 0x7fe1f40a7740 2019-12-05T05:25:47.089 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:47.081+0000 7fe1f2ffd700 1 -- 172.21.15.42:0/3880954960 >> v2:172.21.15.40:3300/0 conn(0x7fe1f409f830 msgr2=0x7fe1f414efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:25:47.089 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:47.081+0000 7fe1f2ffd700 1 --2- 172.21.15.42:0/3880954960 >> v2:172.21.15.40:3300/0 conn(0x7fe1f409f830 0x7fe1f414efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:25:47.090 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:47.081+0000 7fe1f2ffd700 1 -- 172.21.15.42:0/3880954960 >> v1:172.21.15.42:6789/0 conn(0x7fe1f40a7740 legacy=0x7fe1f407e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:25:47.090 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:47.081+0000 7fe1f2ffd700 1 -- 172.21.15.42:0/3880954960 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1f40a0710 con 0x7fe1f408df80 2019-12-05T05:25:47.090 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:47.081+0000 7fe1f2ffd700 1 --2- 172.21.15.42:0/3880954960 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe1f408df80 0x7fe1f409f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fe1e8008790 tx=0x7fe1e800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:47.090 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:47.081+0000 7fe1f2ffd700 1 --2- 172.21.15.42:0/3880954960 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe1f408df80 0x7fe1f409f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fe1e8008790 tx=0x7fe1e800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:47.090 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:47.081+0000 7fe1f2ffd700 1 --2- 172.21.15.42:0/3880954960 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe1f408df80 0x7fe1f409f2e0 secure :-1 s=READY pgs=172 cs=0 l=1 rx=0x7fe1e8008790 tx=0x7fe1e800fc70).ready entity=mon.2 client_cookie=7b12932d14b09a24 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:25:47.090 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:47.081+0000 7fe1f2ffd700 1 --2- 172.21.15.42:0/3880954960 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe1f408df80 0x7fe1f409f2e0 secure :-1 s=THROTTLE_DONE pgs=172 cs=0 l=1 rx=0x7fe1e8008790 tx=0x7fe1e800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:47.090 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:47.081+0000 7fe1f2ffd700 1 --2- 172.21.15.42:0/3880954960 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe1f408df80 0x7fe1f409f2e0 secure :-1 s=THROTTLE_DONE pgs=172 cs=0 l=1 rx=0x7fe1e8008790 tx=0x7fe1e800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:47.091 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:47.081+0000 7fe1f27fc700 1 -- 172.21.15.42:0/3880954960 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe1e8008bb0 con 0x7fe1f408df80 2019-12-05T05:25:47.091 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:47.081+0000 7fe1f2ffd700 1 --2- 172.21.15.42:0/3880954960 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe1f408df80 0x7fe1f409f2e0 secure :-1 s=THROTTLE_DONE pgs=172 cs=0 l=1 rx=0x7fe1e8008790 tx=0x7fe1e800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:47.091 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:47.081+0000 7fe1f27fc700 1 -- 172.21.15.42:0/3880954960 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1e80087d0 con 0x7fe1f408df80 2019-12-05T05:25:47.091 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:47.081+0000 7fe1f27fc700 1 -- 172.21.15.42:0/3880954960 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe1e8011860 con 0x7fe1f408df80 2019-12-05T05:25:47.092 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:47.081+0000 7fe1f9713700 1 -- 172.21.15.42:0/3880954960 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe1f408df80 msgr2=0x7fe1f409f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:25:47.092 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:47.081+0000 7fe1f9713700 1 --2- 172.21.15.42:0/3880954960 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe1f408df80 0x7fe1f409f2e0 secure :-1 s=READY pgs=172 cs=0 l=1 rx=0x7fe1e8008790 tx=0x7fe1e800fc70).stop 2019-12-05T05:25:47.092 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:47.081+0000 7fe1f9713700 1 -- 172.21.15.42:0/3880954960 shutdown_connections 2019-12-05T05:25:47.092 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:47.081+0000 7fe1f9713700 1 --2- 172.21.15.42:0/3880954960 >> v2:172.21.15.40:3300/0 conn(0x7fe1f409f830 0x7fe1f414efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:25:47.093 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:47.081+0000 7fe1f9713700 1 --2- 172.21.15.42:0/3880954960 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe1f408df80 0x7fe1f409f2e0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:25:47.093 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:47.081+0000 7fe1f9713700 1 -- 172.21.15.42:0/3880954960 shutdown_connections 2019-12-05T05:25:47.093 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:47.081+0000 7fe1f9713700 1 -- 172.21.15.42:0/3880954960 wait complete. 2019-12-05T05:25:47.093 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:47.081+0000 7fe1f9713700 1 -- 172.21.15.42:0/3880954960 >> 172.21.15.42:0/3880954960 conn(0x7fe1f40aabe0 msgr2=0x7fe1f40a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:25:47.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:47.085+0000 7fe1f9713700 1 Processor -- start 2019-12-05T05:25:47.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:47.085+0000 7fe1f9713700 1 -- start start 2019-12-05T05:25:47.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:47.085+0000 7fe1f9713700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe1f40a9640 0x7fe1f40a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:25:47.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:47.085+0000 7fe1f9713700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe1f40a5620 0x7fe1f411a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:25:47.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:47.085+0000 7fe1f9713700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1f40a4d50 con 0x7fe1f411ae00 2019-12-05T05:25:47.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:47.085+0000 7fe1f9713700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1f4142150 con 0x7fe1f40a9640 2019-12-05T05:25:47.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:47.085+0000 7fe1f9713700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1f4119ab0 con 0x7fe1f40a5620 2019-12-05T05:25:47.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:47.085+0000 7fe1f2ffd700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe1f40a5620 0x7fe1f411a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:25:47.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:47.085+0000 7fe1f37fe700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe1f40a9640 0x7fe1f40a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:25:47.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:47.085+0000 7fe1f2ffd700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe1f40a5620 0x7fe1f411a8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:33958/0 (socket says 172.21.15.42:33958) 2019-12-05T05:25:47.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:47.085+0000 7fe1f2ffd700 1 -- 172.21.15.42:0/4097288055 learned_addr learned my addr 172.21.15.42:0/4097288055 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:25:47.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:47.085+0000 7fe1dbfff700 1 -- 172.21.15.42:0/4097288055 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3974993275 0 0) 0x7fe1f40a4d50 con 0x7fe1f411ae00 2019-12-05T05:25:47.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:47.085+0000 7fe1dbfff700 1 -- 172.21.15.42:0/4097288055 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1d40030b0 con 0x7fe1f411ae00 2019-12-05T05:25:47.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:47.085+0000 7fe1f2ffd700 1 -- 172.21.15.42:0/4097288055 >> v2:172.21.15.40:3300/0 conn(0x7fe1f40a9640 msgr2=0x7fe1f40a50d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:25:47.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:47.085+0000 7fe1f2ffd700 1 --2- 172.21.15.42:0/4097288055 >> v2:172.21.15.40:3300/0 conn(0x7fe1f40a9640 0x7fe1f40a50d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:25:47.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:47.085+0000 7fe1f2ffd700 1 -- 172.21.15.42:0/4097288055 >> v1:172.21.15.40:6789/0 conn(0x7fe1f411ae00 legacy=0x7fe1f4116520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:25:47.100 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:47.085+0000 7fe1f2ffd700 1 -- 172.21.15.42:0/4097288055 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1f411a290 con 0x7fe1f40a5620 2019-12-05T05:25:47.100 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:47.085+0000 7fe1f2ffd700 1 --2- 172.21.15.42:0/4097288055 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe1f40a5620 0x7fe1f411a8b0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fe1e8008790 tx=0x7fe1e8011980).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:47.100 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:47.085+0000 7fe1f2ffd700 1 --2- 172.21.15.42:0/4097288055 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe1f40a5620 0x7fe1f411a8b0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fe1e8008790 tx=0x7fe1e8011980).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:47.100 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:47.085+0000 7fe1f2ffd700 1 --2- 172.21.15.42:0/4097288055 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe1f40a5620 0x7fe1f411a8b0 secure :-1 s=READY pgs=173 cs=0 l=1 rx=0x7fe1e8008790 tx=0x7fe1e8011980).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:25:47.100 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:47.089+0000 7fe1f2ffd700 1 --2- 172.21.15.42:0/4097288055 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe1f40a5620 0x7fe1f411a8b0 secure :-1 s=THROTTLE_DONE pgs=173 cs=0 l=1 rx=0x7fe1e8008790 tx=0x7fe1e8011980).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:47.100 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:47.089+0000 7fe1dbfff700 1 -- 172.21.15.42:0/4097288055 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe1e800f280 con 0x7fe1f40a5620 2019-12-05T05:25:47.100 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:47.089+0000 7fe1f2ffd700 1 --2- 172.21.15.42:0/4097288055 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe1f40a5620 0x7fe1f411a8b0 secure :-1 s=THROTTLE_DONE pgs=173 cs=0 l=1 rx=0x7fe1e8008790 tx=0x7fe1e8011980).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:47.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:47.089+0000 7fe1f2ffd700 1 --2- 172.21.15.42:0/4097288055 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe1f40a5620 0x7fe1f411a8b0 secure :-1 s=THROTTLE_DONE pgs=173 cs=0 l=1 rx=0x7fe1e8008790 tx=0x7fe1e8011980).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:47.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:47.089+0000 7fe1dbfff700 1 -- 172.21.15.42:0/4097288055 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1e8008db0 con 0x7fe1f40a5620 2019-12-05T05:25:47.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:47.089+0000 7fe1f9713700 1 -- 172.21.15.42:0/4097288055 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1f40a9a70 con 0x7fe1f40a5620 2019-12-05T05:25:47.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:47.089+0000 7fe1dbfff700 1 -- 172.21.15.42:0/4097288055 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe1e8005540 con 0x7fe1f40a5620 2019-12-05T05:25:47.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:47.089+0000 7fe1f9713700 1 -- 172.21.15.42:0/4097288055 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1f4116d20 con 0x7fe1f40a5620 2019-12-05T05:25:47.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:47.089+0000 7fe1f2ffd700 1 --2- 172.21.15.42:0/4097288055 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe1f40a5620 0x7fe1f411a8b0 secure :-1 s=THROTTLE_DONE pgs=173 cs=0 l=1 rx=0x7fe1e8008790 tx=0x7fe1e8011980).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:47.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:47.089+0000 7fe1dbfff700 1 -- 172.21.15.42:0/4097288055 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fe1e8005850 con 0x7fe1f40a5620 2019-12-05T05:25:47.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:47.089+0000 7fe1f2ffd700 1 --2- 172.21.15.42:0/4097288055 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe1f40a5620 0x7fe1f411a8b0 secure :-1 s=THROTTLE_DONE pgs=173 cs=0 l=1 rx=0x7fe1e8008790 tx=0x7fe1e8011980).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:47.104 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:47.089+0000 7fe1dbfff700 1 --2- 172.21.15.42:0/4097288055 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe1d403b2b0 0x7fe1d403d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:25:47.104 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:47.089+0000 7fe1dbfff700 1 -- 172.21.15.42:0/4097288055 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7321+0+0 (secure 0 0 0) 0x7fe1e804b6b0 con 0x7fe1f40a5620 2019-12-05T05:25:47.104 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:47.089+0000 7fe1f37fe700 1 --2- 172.21.15.42:0/4097288055 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe1d403b2b0 0x7fe1d403d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:25:47.105 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:47.089+0000 7fe1f37fe700 1 --2- 172.21.15.42:0/4097288055 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe1d403b2b0 0x7fe1d403d7c0 crc :-1 s=READY pgs=368 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:25:47.105 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:47.093+0000 7fe1f9713700 1 -- 172.21.15.42:0/4097288055 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1f405d110 con 0x7fe1f40a5620 2019-12-05T05:25:47.106 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:47.101+0000 7fe1f2ffd700 1 --2- 172.21.15.42:0/4097288055 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe1f40a5620 0x7fe1f411a8b0 secure :-1 s=THROTTLE_DONE pgs=173 cs=0 l=1 rx=0x7fe1e8008790 tx=0x7fe1e8011980).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:47.107 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:47.101+0000 7fe1dbfff700 1 -- 172.21.15.42:0/4097288055 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fe1e8032990 con 0x7fe1f40a5620 2019-12-05T05:25:47.278 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:47.273+0000 7fe1f9713700 1 -- 172.21.15.42:0/4097288055 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe1f405b8c0 con 0x7fe1f40a5620 2019-12-05T05:25:47.278 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:47.273+0000 7fe1f2ffd700 1 --2- 172.21.15.42:0/4097288055 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe1f40a5620 0x7fe1f411a8b0 secure :-1 s=THROTTLE_DONE pgs=173 cs=0 l=1 rx=0x7fe1e8008790 tx=0x7fe1e8011980).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:47.279 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:47.273+0000 7fe1dbfff700 1 -- 172.21.15.42:0/4097288055 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+1775 (secure 0 0 0) 0x7fe1e8033540 con 0x7fe1f40a5620 2019-12-05T05:25:47.282 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 107 2019-12-05T05:25:47.282 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:25:47.282 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":107,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:25:38.087763+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5240},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5240":{"gid":5240,"name":"a","rank":0,"incarnation":104,"state":"up:active","state_seq":27,"addr":"172.21.15.42:6835/1455108684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1455108684},{"type":"v1","addr":"172.21.15.42:6835","nonce":1455108684}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:25:47.288 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:47.277+0000 7fe1f9713700 1 -- 172.21.15.42:0/4097288055 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe1d403b2b0 msgr2=0x7fe1d403d7c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:25:47.288 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:47.277+0000 7fe1f9713700 1 --2- 172.21.15.42:0/4097288055 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe1d403b2b0 0x7fe1d403d7c0 crc :-1 s=READY pgs=368 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:25:47.288 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:47.277+0000 7fe1f9713700 1 -- 172.21.15.42:0/4097288055 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe1f40a5620 msgr2=0x7fe1f411a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:25:47.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:47.277+0000 7fe1f9713700 1 --2- 172.21.15.42:0/4097288055 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe1f40a5620 0x7fe1f411a8b0 secure :-1 s=READY pgs=173 cs=0 l=1 rx=0x7fe1e8008790 tx=0x7fe1e8011980).stop 2019-12-05T05:25:47.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:47.277+0000 7fe1f9713700 1 -- 172.21.15.42:0/4097288055 shutdown_connections 2019-12-05T05:25:47.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:47.277+0000 7fe1f9713700 1 --2- 172.21.15.42:0/4097288055 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe1d403b2b0 0x7fe1d403d7c0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:25:47.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:47.277+0000 7fe1f9713700 1 --2- 172.21.15.42:0/4097288055 >> v2:172.21.15.40:3300/0 conn(0x7fe1f40a9640 0x7fe1f40a50d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:25:47.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:47.277+0000 7fe1f9713700 1 --2- 172.21.15.42:0/4097288055 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe1f40a5620 0x7fe1f411a8b0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:25:47.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:47.277+0000 7fe1f9713700 1 -- 172.21.15.42:0/4097288055 shutdown_connections 2019-12-05T05:25:47.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:47.277+0000 7fe1f9713700 1 -- 172.21.15.42:0/4097288055 wait complete. 2019-12-05T05:25:47.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:47.277+0000 7fe1f9713700 1 -- 172.21.15.42:0/4097288055 >> 172.21.15.42:0/4097288055 conn(0x7fe1f409ffb0 msgr2=0x7fe1f41185d0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:25:47.293 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2019-12-05T05:25:49.294 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:25:49.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:49.393+0000 7fb15c186700 1 Processor -- start 2019-12-05T05:25:49.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:49.393+0000 7fb15c186700 1 -- start start 2019-12-05T05:25:49.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:49.393+0000 7fb15c186700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb15408df80 0x7fb15409f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:25:49.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:49.393+0000 7fb15c186700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb15409f830 0x7fb15414efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:25:49.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:49.393+0000 7fb15c186700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb15409fc60 con 0x7fb1540a7740 2019-12-05T05:25:49.404 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:49.393+0000 7fb15c186700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1540a0b00 con 0x7fb15409f830 2019-12-05T05:25:49.404 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:49.393+0000 7fb15c186700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb15408e5b0 con 0x7fb15408df80 2019-12-05T05:25:49.404 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:49.393+0000 7fb15a723700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fb1540a7740 0x7fb15407e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:33546/0 (socket says 172.21.15.42:33546) 2019-12-05T05:25:49.404 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:49.393+0000 7fb15a723700 1 -- 172.21.15.42:0/147376100 learned_addr learned my addr 172.21.15.42:0/147376100 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:25:49.404 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:49.393+0000 7fb159f22700 1 --2- 172.21.15.42:0/147376100 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb15408df80 0x7fb15409f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:25:49.404 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:49.393+0000 7fb15af24700 1 --2- 172.21.15.42:0/147376100 >> v2:172.21.15.42:3300/0 conn(0x7fb15409f830 0x7fb15414efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:25:49.404 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:49.393+0000 7fb159721700 1 -- 172.21.15.42:0/147376100 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2357378911 0 0) 0x7fb15409fc60 con 0x7fb1540a7740 2019-12-05T05:25:49.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:49.393+0000 7fb159721700 1 -- 172.21.15.42:0/147376100 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1400030b0 con 0x7fb1540a7740 2019-12-05T05:25:49.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:49.393+0000 7fb159f22700 1 -- 172.21.15.42:0/147376100 >> v2:172.21.15.42:3300/0 conn(0x7fb15409f830 msgr2=0x7fb15414efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:25:49.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:49.393+0000 7fb159f22700 1 --2- 172.21.15.42:0/147376100 >> v2:172.21.15.42:3300/0 conn(0x7fb15409f830 0x7fb15414efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:25:49.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:49.393+0000 7fb159f22700 1 -- 172.21.15.42:0/147376100 >> v1:172.21.15.42:6789/0 conn(0x7fb1540a7740 legacy=0x7fb15407e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:25:49.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:49.393+0000 7fb159f22700 1 -- 172.21.15.42:0/147376100 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1540a0710 con 0x7fb15408df80 2019-12-05T05:25:49.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:49.393+0000 7fb15af24700 1 --2- 172.21.15.42:0/147376100 >> v2:172.21.15.42:3300/0 conn(0x7fb15409f830 0x7fb15414efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:25:49.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:49.393+0000 7fb159f22700 1 --2- 172.21.15.42:0/147376100 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb15408df80 0x7fb15409f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fb148008b10 tx=0x7fb14800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:49.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:49.397+0000 7fb159f22700 1 --2- 172.21.15.42:0/147376100 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb15408df80 0x7fb15409f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fb148008b10 tx=0x7fb14800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:49.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:49.397+0000 7fb159f22700 1 --2- 172.21.15.42:0/147376100 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb15408df80 0x7fb15409f2e0 secure :-1 s=READY pgs=174 cs=0 l=1 rx=0x7fb148008b10 tx=0x7fb14800fff0).ready entity=mon.2 client_cookie=fb5f7081802c9dea server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:25:49.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:49.397+0000 7fb159f22700 1 --2- 172.21.15.42:0/147376100 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb15408df80 0x7fb15409f2e0 secure :-1 s=THROTTLE_DONE pgs=174 cs=0 l=1 rx=0x7fb148008b10 tx=0x7fb14800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:49.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:49.397+0000 7fb159721700 1 -- 172.21.15.42:0/147376100 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb148010900 con 0x7fb15408df80 2019-12-05T05:25:49.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:49.397+0000 7fb159f22700 1 --2- 172.21.15.42:0/147376100 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb15408df80 0x7fb15409f2e0 secure :-1 s=THROTTLE_DONE pgs=174 cs=0 l=1 rx=0x7fb148008b10 tx=0x7fb14800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:49.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:49.397+0000 7fb159f22700 1 --2- 172.21.15.42:0/147376100 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb15408df80 0x7fb15409f2e0 secure :-1 s=THROTTLE_DONE pgs=174 cs=0 l=1 rx=0x7fb148008b10 tx=0x7fb14800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:49.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:49.397+0000 7fb159721700 1 -- 172.21.15.42:0/147376100 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb148008b50 con 0x7fb15408df80 2019-12-05T05:25:49.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:49.397+0000 7fb159721700 1 -- 172.21.15.42:0/147376100 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb1480110c0 con 0x7fb15408df80 2019-12-05T05:25:49.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:49.397+0000 7fb15c186700 1 -- 172.21.15.42:0/147376100 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb15408df80 msgr2=0x7fb15409f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:25:49.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:49.397+0000 7fb15c186700 1 --2- 172.21.15.42:0/147376100 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb15408df80 0x7fb15409f2e0 secure :-1 s=READY pgs=174 cs=0 l=1 rx=0x7fb148008b10 tx=0x7fb14800fff0).stop 2019-12-05T05:25:49.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:49.397+0000 7fb15c186700 1 -- 172.21.15.42:0/147376100 shutdown_connections 2019-12-05T05:25:49.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:49.397+0000 7fb15c186700 1 --2- 172.21.15.42:0/147376100 >> v2:172.21.15.42:3300/0 conn(0x7fb15409f830 0x7fb15414efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:25:49.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:49.397+0000 7fb15c186700 1 --2- 172.21.15.42:0/147376100 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb15408df80 0x7fb15409f2e0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:25:49.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:49.397+0000 7fb15c186700 1 -- 172.21.15.42:0/147376100 shutdown_connections 2019-12-05T05:25:49.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:49.397+0000 7fb15c186700 1 -- 172.21.15.42:0/147376100 wait complete. 2019-12-05T05:25:49.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:49.397+0000 7fb15c186700 1 -- 172.21.15.42:0/147376100 >> 172.21.15.42:0/147376100 conn(0x7fb1540aabe0 msgr2=0x7fb1540a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:25:49.411 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:49.397+0000 7fb15c186700 1 Processor -- start 2019-12-05T05:25:49.411 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:49.401+0000 7fb15c186700 1 -- start start 2019-12-05T05:25:49.411 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:49.401+0000 7fb15c186700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb15411cff0 0x7fb15407ba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:25:49.411 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:49.401+0000 7fb15c186700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb1540a5eb0 0x7fb154135ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:25:49.412 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:49.401+0000 7fb15c186700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb154142530 con 0x7fb1540a4b30 2019-12-05T05:25:49.412 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:49.401+0000 7fb15c186700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb154142840 con 0x7fb15411cff0 2019-12-05T05:25:49.412 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:49.401+0000 7fb15c186700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb154142ae0 con 0x7fb1540a5eb0 2019-12-05T05:25:49.412 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:49.401+0000 7fb15af24700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb1540a5eb0 0x7fb154135ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:25:49.412 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:49.401+0000 7fb15a723700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb15411cff0 0x7fb15407ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:25:49.412 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:49.401+0000 7fb15af24700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb1540a5eb0 0x7fb154135ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:33518/0 (socket says 172.21.15.42:33518) 2019-12-05T05:25:49.412 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:49.401+0000 7fb15af24700 1 -- 172.21.15.42:0/2079396005 learned_addr learned my addr 172.21.15.42:0/2079396005 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:25:49.413 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:49.401+0000 7fb15af24700 1 -- 172.21.15.42:0/2079396005 >> v2:172.21.15.40:3300/0 conn(0x7fb15411cff0 msgr2=0x7fb15407ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:25:49.413 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:49.401+0000 7fb15af24700 1 --2- 172.21.15.42:0/2079396005 >> v2:172.21.15.40:3300/0 conn(0x7fb15411cff0 0x7fb15407ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:25:49.413 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:49.401+0000 7fb15af24700 1 -- 172.21.15.42:0/2079396005 >> v1:172.21.15.40:6789/0 conn(0x7fb1540a4b30 legacy=0x7fb1540a56b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:25:49.413 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:49.401+0000 7fb15af24700 1 -- 172.21.15.42:0/2079396005 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb15407bfb0 con 0x7fb1540a5eb0 2019-12-05T05:25:49.413 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:49.401+0000 7fb15af24700 1 --2- 172.21.15.42:0/2079396005 >> v2:172.21.15.42:3300/0 conn(0x7fb1540a5eb0 0x7fb154135ac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fb14c002f30 tx=0x7fb14c010b80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:49.413 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:49.401+0000 7fb15af24700 1 --2- 172.21.15.42:0/2079396005 >> v2:172.21.15.42:3300/0 conn(0x7fb1540a5eb0 0x7fb154135ac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fb14c002f30 tx=0x7fb14c010b80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:49.413 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:49.401+0000 7fb15af24700 1 --2- 172.21.15.42:0/2079396005 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb1540a5eb0 0x7fb154135ac0 secure :-1 s=READY pgs=839 cs=0 l=1 rx=0x7fb14c002f30 tx=0x7fb14c010b80).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:25:49.414 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:49.401+0000 7fb15af24700 1 --2- 172.21.15.42:0/2079396005 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb1540a5eb0 0x7fb154135ac0 secure :-1 s=THROTTLE_DONE pgs=839 cs=0 l=1 rx=0x7fb14c002f30 tx=0x7fb14c010b80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:49.414 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:49.401+0000 7fb15af24700 1 --2- 172.21.15.42:0/2079396005 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb1540a5eb0 0x7fb154135ac0 secure :-1 s=THROTTLE_DONE pgs=839 cs=0 l=1 rx=0x7fb14c002f30 tx=0x7fb14c010b80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:49.414 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:49.401+0000 7fb146ffd700 1 -- 172.21.15.42:0/2079396005 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb14c00d0c0 con 0x7fb1540a5eb0 2019-12-05T05:25:49.414 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:49.401+0000 7fb15af24700 1 --2- 172.21.15.42:0/2079396005 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb1540a5eb0 0x7fb154135ac0 secure :-1 s=THROTTLE_DONE pgs=839 cs=0 l=1 rx=0x7fb14c002f30 tx=0x7fb14c010b80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:49.414 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:49.401+0000 7fb146ffd700 1 -- 172.21.15.42:0/2079396005 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb14c007150 con 0x7fb1540a5eb0 2019-12-05T05:25:49.414 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:49.401+0000 7fb15c186700 1 -- 172.21.15.42:0/2079396005 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1540a4f60 con 0x7fb1540a5eb0 2019-12-05T05:25:49.414 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:49.401+0000 7fb146ffd700 1 -- 172.21.15.42:0/2079396005 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb14c0129b0 con 0x7fb1540a5eb0 2019-12-05T05:25:49.414 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:49.401+0000 7fb15c186700 1 -- 172.21.15.42:0/2079396005 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb154136010 con 0x7fb1540a5eb0 2019-12-05T05:25:49.415 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:49.405+0000 7fb15af24700 1 --2- 172.21.15.42:0/2079396005 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb1540a5eb0 0x7fb154135ac0 secure :-1 s=THROTTLE_DONE pgs=839 cs=0 l=1 rx=0x7fb14c002f30 tx=0x7fb14c010b80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:49.415 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:49.405+0000 7fb146ffd700 1 -- 172.21.15.42:0/2079396005 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fb14c012cc0 con 0x7fb1540a5eb0 2019-12-05T05:25:49.415 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:49.405+0000 7fb15af24700 1 --2- 172.21.15.42:0/2079396005 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb1540a5eb0 0x7fb154135ac0 secure :-1 s=THROTTLE_DONE pgs=839 cs=0 l=1 rx=0x7fb14c002f30 tx=0x7fb14c010b80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:49.415 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:49.405+0000 7fb146ffd700 1 --2- 172.21.15.42:0/2079396005 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb13c03aed0 0x7fb13c03d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:25:49.415 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:49.405+0000 7fb146ffd700 1 -- 172.21.15.42:0/2079396005 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7321+0+0 (secure 0 0 0) 0x7fb14c04b190 con 0x7fb1540a5eb0 2019-12-05T05:25:49.415 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:49.405+0000 7fb15a723700 1 --2- 172.21.15.42:0/2079396005 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb13c03aed0 0x7fb13c03d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:25:49.415 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:49.405+0000 7fb15a723700 1 --2- 172.21.15.42:0/2079396005 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb13c03aed0 0x7fb13c03d3e0 crc :-1 s=READY pgs=370 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:25:49.416 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:49.409+0000 7fb15c186700 1 -- 172.21.15.42:0/2079396005 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb15405d110 con 0x7fb1540a5eb0 2019-12-05T05:25:49.422 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:49.417+0000 7fb15af24700 1 --2- 172.21.15.42:0/2079396005 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb1540a5eb0 0x7fb154135ac0 secure :-1 s=THROTTLE_DONE pgs=839 cs=0 l=1 rx=0x7fb14c002f30 tx=0x7fb14c010b80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:49.422 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:49.417+0000 7fb146ffd700 1 -- 172.21.15.42:0/2079396005 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fb14c032210 con 0x7fb1540a5eb0 2019-12-05T05:25:49.593 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:49.589+0000 7fb15c186700 1 -- 172.21.15.42:0/2079396005 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb1540aadf0 con 0x7fb1540a5eb0 2019-12-05T05:25:49.594 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:49.589+0000 7fb15af24700 1 --2- 172.21.15.42:0/2079396005 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb1540a5eb0 0x7fb154135ac0 secure :-1 s=THROTTLE_DONE pgs=839 cs=0 l=1 rx=0x7fb14c002f30 tx=0x7fb14c010b80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:49.594 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:49.589+0000 7fb146ffd700 1 -- 172.21.15.42:0/2079396005 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+1775 (secure 0 0 0) 0x7fb14c032470 con 0x7fb1540a5eb0 2019-12-05T05:25:49.594 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 107 2019-12-05T05:25:49.594 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:25:49.594 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":107,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:25:38.087763+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5240},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5240":{"gid":5240,"name":"a","rank":0,"incarnation":104,"state":"up:active","state_seq":27,"addr":"172.21.15.42:6835/1455108684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1455108684},{"type":"v1","addr":"172.21.15.42:6835","nonce":1455108684}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:25:49.599 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:49.593+0000 7fb15c186700 1 -- 172.21.15.42:0/2079396005 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb13c03aed0 msgr2=0x7fb13c03d3e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:25:49.599 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:49.593+0000 7fb15c186700 1 --2- 172.21.15.42:0/2079396005 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb13c03aed0 0x7fb13c03d3e0 crc :-1 s=READY pgs=370 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:25:49.599 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:49.593+0000 7fb15c186700 1 -- 172.21.15.42:0/2079396005 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb1540a5eb0 msgr2=0x7fb154135ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:25:49.599 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:49.593+0000 7fb15c186700 1 --2- 172.21.15.42:0/2079396005 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb1540a5eb0 0x7fb154135ac0 secure :-1 s=READY pgs=839 cs=0 l=1 rx=0x7fb14c002f30 tx=0x7fb14c010b80).stop 2019-12-05T05:25:49.599 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:49.593+0000 7fb15c186700 1 -- 172.21.15.42:0/2079396005 shutdown_connections 2019-12-05T05:25:49.599 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:49.593+0000 7fb15c186700 1 --2- 172.21.15.42:0/2079396005 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb13c03aed0 0x7fb13c03d3e0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:25:49.600 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:49.593+0000 7fb15c186700 1 --2- 172.21.15.42:0/2079396005 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb1540a5eb0 0x7fb154135ac0 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:25:49.600 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:49.593+0000 7fb15c186700 1 --2- 172.21.15.42:0/2079396005 >> v2:172.21.15.40:3300/0 conn(0x7fb15411cff0 0x7fb15407ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:25:49.600 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:49.593+0000 7fb15c186700 1 -- 172.21.15.42:0/2079396005 shutdown_connections 2019-12-05T05:25:49.600 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:49.593+0000 7fb15c186700 1 -- 172.21.15.42:0/2079396005 wait complete. 2019-12-05T05:25:49.600 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:49.593+0000 7fb15c186700 1 -- 172.21.15.42:0/2079396005 >> 172.21.15.42:0/2079396005 conn(0x7fb1540a7f90 msgr2=0x7fb15411d780 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:25:49.606 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2019-12-05T05:25:51.607 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:25:51.711 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:51.701+0000 7fad7f4f0700 1 Processor -- start 2019-12-05T05:25:51.711 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:51.701+0000 7fad7f4f0700 1 -- start start 2019-12-05T05:25:51.712 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:51.701+0000 7fad7f4f0700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fad7808df80 0x7fad7809f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:25:51.712 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:51.701+0000 7fad7f4f0700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fad7809f830 0x7fad7814efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:25:51.712 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:51.701+0000 7fad7f4f0700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad7809fc60 con 0x7fad780a7740 2019-12-05T05:25:51.712 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:51.701+0000 7fad7f4f0700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad780a0b00 con 0x7fad7809f830 2019-12-05T05:25:51.712 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:51.701+0000 7fad7f4f0700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad7808e5b0 con 0x7fad7808df80 2019-12-05T05:25:51.712 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:51.701+0000 7fad7d28c700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fad7808df80 0x7fad7809f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:25:51.712 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:51.701+0000 7fad7e28e700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fad7809f830 0x7fad7814efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:25:51.713 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:51.701+0000 7fad7d28c700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fad7808df80 0x7fad7809f2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:34000/0 (socket says 172.21.15.42:34000) 2019-12-05T05:25:51.713 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:51.701+0000 7fad7d28c700 1 -- 172.21.15.42:0/1135512181 learned_addr learned my addr 172.21.15.42:0/1135512181 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:25:51.713 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:51.701+0000 7fad7e28e700 1 -- 172.21.15.42:0/1135512181 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fad7808df80 msgr2=0x7fad7809f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:25:51.713 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:51.701+0000 7fad7e28e700 1 --2- 172.21.15.42:0/1135512181 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fad7808df80 0x7fad7809f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:25:51.713 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:51.701+0000 7fad7e28e700 1 -- 172.21.15.42:0/1135512181 >> v1:172.21.15.40:6789/0 conn(0x7fad780a7740 legacy=0x7fad7807e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:25:51.713 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:51.701+0000 7fad7e28e700 1 -- 172.21.15.42:0/1135512181 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad780a0710 con 0x7fad7809f830 2019-12-05T05:25:51.713 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:51.701+0000 7fad7e28e700 1 --2- 172.21.15.42:0/1135512181 >> v2:172.21.15.40:3300/0 conn(0x7fad7809f830 0x7fad7814efe0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fad70008b10 tx=0x7fad7000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:51.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:51.701+0000 7fad7e28e700 1 --2- 172.21.15.42:0/1135512181 >> v2:172.21.15.40:3300/0 conn(0x7fad7809f830 0x7fad7814efe0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fad70008b10 tx=0x7fad7000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:51.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:51.701+0000 7fad7e28e700 1 --2- 172.21.15.42:0/1135512181 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fad7809f830 0x7fad7814efe0 secure :-1 s=READY pgs=671 cs=0 l=1 rx=0x7fad70008b10 tx=0x7fad7000fff0).ready entity=mon.0 client_cookie=a94381e678ff3af1 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:25:51.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:51.701+0000 7fad7e28e700 1 --2- 172.21.15.42:0/1135512181 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fad7809f830 0x7fad7814efe0 secure :-1 s=THROTTLE_DONE pgs=671 cs=0 l=1 rx=0x7fad70008b10 tx=0x7fad7000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:51.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:51.701+0000 7fad7ca8b700 1 -- 172.21.15.42:0/1135512181 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fad70010900 con 0x7fad7809f830 2019-12-05T05:25:51.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:51.701+0000 7fad7e28e700 1 --2- 172.21.15.42:0/1135512181 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fad7809f830 0x7fad7814efe0 secure :-1 s=THROTTLE_DONE pgs=671 cs=0 l=1 rx=0x7fad70008b10 tx=0x7fad7000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:51.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:51.701+0000 7fad7ca8b700 1 -- 172.21.15.42:0/1135512181 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad70008b50 con 0x7fad7809f830 2019-12-05T05:25:51.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:51.701+0000 7fad7e28e700 1 --2- 172.21.15.42:0/1135512181 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fad7809f830 0x7fad7814efe0 secure :-1 s=THROTTLE_DONE pgs=671 cs=0 l=1 rx=0x7fad70008b10 tx=0x7fad7000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:51.715 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:51.705+0000 7fad7f4f0700 1 -- 172.21.15.42:0/1135512181 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fad7809f830 msgr2=0x7fad7814efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:25:51.715 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:51.705+0000 7fad7ca8b700 1 -- 172.21.15.42:0/1135512181 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fad700110c0 con 0x7fad7809f830 2019-12-05T05:25:51.715 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:51.705+0000 7fad7f4f0700 1 --2- 172.21.15.42:0/1135512181 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fad7809f830 0x7fad7814efe0 secure :-1 s=READY pgs=671 cs=0 l=1 rx=0x7fad70008b10 tx=0x7fad7000fff0).stop 2019-12-05T05:25:51.715 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:51.705+0000 7fad7f4f0700 1 -- 172.21.15.42:0/1135512181 shutdown_connections 2019-12-05T05:25:51.715 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:51.705+0000 7fad7f4f0700 1 --2- 172.21.15.42:0/1135512181 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fad7809f830 0x7fad7814efe0 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:25:51.715 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:51.705+0000 7fad7f4f0700 1 --2- 172.21.15.42:0/1135512181 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fad7808df80 0x7fad7809f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:25:51.715 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:51.705+0000 7fad7f4f0700 1 -- 172.21.15.42:0/1135512181 shutdown_connections 2019-12-05T05:25:51.716 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:51.705+0000 7fad7f4f0700 1 -- 172.21.15.42:0/1135512181 wait complete. 2019-12-05T05:25:51.716 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:51.705+0000 7fad7f4f0700 1 -- 172.21.15.42:0/1135512181 >> 172.21.15.42:0/1135512181 conn(0x7fad780aabe0 msgr2=0x7fad780a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:25:51.716 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:51.705+0000 7fad7f4f0700 1 Processor -- start 2019-12-05T05:25:51.716 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:51.705+0000 7fad7f4f0700 1 -- start start 2019-12-05T05:25:51.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:51.709+0000 7fad7f4f0700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fad78121a70 0x7fad7812ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:25:51.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:51.709+0000 7fad7f4f0700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fad78118260 0x7fad78118690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:25:51.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:51.709+0000 7fad7f4f0700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad7812f350 con 0x7fad7811b750 2019-12-05T05:25:51.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:51.709+0000 7fad7f4f0700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad78118e80 con 0x7fad78118260 2019-12-05T05:25:51.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:51.709+0000 7fad7f4f0700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad78119120 con 0x7fad78121a70 2019-12-05T05:25:51.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:51.709+0000 7fad7da8d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fad78121a70 0x7fad7812ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:25:51.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:51.709+0000 7fad7da8d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fad78121a70 0x7fad7812ee00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:33538/0 (socket says 172.21.15.42:33538) 2019-12-05T05:25:51.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:51.709+0000 7fad7da8d700 1 -- 172.21.15.42:0/1867621962 learned_addr learned my addr 172.21.15.42:0/1867621962 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:25:51.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:51.709+0000 7fad7d28c700 1 --2- 172.21.15.42:0/1867621962 >> v2:172.21.15.40:3300/0 conn(0x7fad78118260 0x7fad78118690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:25:51.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:51.709+0000 7fad7da8d700 1 -- 172.21.15.42:0/1867621962 >> v2:172.21.15.40:3300/0 conn(0x7fad78118260 msgr2=0x7fad78118690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:25:51.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:51.709+0000 7fad7da8d700 1 --2- 172.21.15.42:0/1867621962 >> v2:172.21.15.40:3300/0 conn(0x7fad78118260 0x7fad78118690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:25:51.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:51.709+0000 7fad7da8d700 1 -- 172.21.15.42:0/1867621962 >> v1:172.21.15.40:6789/0 conn(0x7fad7811b750 legacy=0x7fad7811bb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:25:51.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:51.709+0000 7fad7da8d700 1 -- 172.21.15.42:0/1867621962 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad7811b150 con 0x7fad78121a70 2019-12-05T05:25:51.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:51.709+0000 7fad7da8d700 1 --2- 172.21.15.42:0/1867621962 >> v2:172.21.15.42:3300/0 conn(0x7fad78121a70 0x7fad7812ee00 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fad7811a910 tx=0x7fad740109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:51.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:51.709+0000 7fad7da8d700 1 --2- 172.21.15.42:0/1867621962 >> v2:172.21.15.42:3300/0 conn(0x7fad78121a70 0x7fad7812ee00 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fad7811a910 tx=0x7fad740109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:51.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:51.709+0000 7fad7da8d700 1 --2- 172.21.15.42:0/1867621962 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fad78121a70 0x7fad7812ee00 secure :-1 s=READY pgs=840 cs=0 l=1 rx=0x7fad7811a910 tx=0x7fad740109a0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:25:51.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:51.709+0000 7fad7da8d700 1 --2- 172.21.15.42:0/1867621962 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fad78121a70 0x7fad7812ee00 secure :-1 s=THROTTLE_DONE pgs=840 cs=0 l=1 rx=0x7fad7811a910 tx=0x7fad740109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:51.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:51.709+0000 7fad6e7fc700 1 -- 172.21.15.42:0/1867621962 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fad7400caa0 con 0x7fad78121a70 2019-12-05T05:25:51.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:51.709+0000 7fad7da8d700 1 --2- 172.21.15.42:0/1867621962 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fad78121a70 0x7fad7812ee00 secure :-1 s=THROTTLE_DONE pgs=840 cs=0 l=1 rx=0x7fad7811a910 tx=0x7fad740109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:51.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:51.709+0000 7fad7da8d700 1 --2- 172.21.15.42:0/1867621962 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fad78121a70 0x7fad7812ee00 secure :-1 s=THROTTLE_DONE pgs=840 cs=0 l=1 rx=0x7fad7811a910 tx=0x7fad740109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:51.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:51.709+0000 7fad6e7fc700 1 -- 172.21.15.42:0/1867621962 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad7400c6c0 con 0x7fad78121a70 2019-12-05T05:25:51.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:51.709+0000 7fad7f4f0700 1 -- 172.21.15.42:0/1867621962 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad78135910 con 0x7fad78121a70 2019-12-05T05:25:51.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:51.709+0000 7fad6e7fc700 1 -- 172.21.15.42:0/1867621962 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fad74011160 con 0x7fad78121a70 2019-12-05T05:25:51.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:51.709+0000 7fad7f4f0700 1 -- 172.21.15.42:0/1867621962 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad78135e40 con 0x7fad78121a70 2019-12-05T05:25:51.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:51.709+0000 7fad7da8d700 1 --2- 172.21.15.42:0/1867621962 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fad78121a70 0x7fad7812ee00 secure :-1 s=THROTTLE_DONE pgs=840 cs=0 l=1 rx=0x7fad7811a910 tx=0x7fad740109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:51.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:51.709+0000 7fad6e7fc700 1 -- 172.21.15.42:0/1867621962 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fad74011470 con 0x7fad78121a70 2019-12-05T05:25:51.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:51.709+0000 7fad7da8d700 1 --2- 172.21.15.42:0/1867621962 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fad78121a70 0x7fad7812ee00 secure :-1 s=THROTTLE_DONE pgs=840 cs=0 l=1 rx=0x7fad7811a910 tx=0x7fad740109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:51.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:51.709+0000 7fad6e7fc700 1 --2- 172.21.15.42:0/1867621962 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fad6003adc0 0x7fad6003d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:25:51.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:51.709+0000 7fad6e7fc700 1 -- 172.21.15.42:0/1867621962 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7321+0+0 (secure 0 0 0) 0x7fad7404b1e0 con 0x7fad78121a70 2019-12-05T05:25:51.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:51.709+0000 7fad7d28c700 1 --2- 172.21.15.42:0/1867621962 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fad6003adc0 0x7fad6003d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:25:51.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:51.713+0000 7fad7d28c700 1 --2- 172.21.15.42:0/1867621962 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fad6003adc0 0x7fad6003d2d0 crc :-1 s=READY pgs=372 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:25:51.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:51.717+0000 7fad7f4f0700 1 -- 172.21.15.42:0/1867621962 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad7805d110 con 0x7fad78121a70 2019-12-05T05:25:51.727 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:51.721+0000 7fad7da8d700 1 --2- 172.21.15.42:0/1867621962 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fad78121a70 0x7fad7812ee00 secure :-1 s=THROTTLE_DONE pgs=840 cs=0 l=1 rx=0x7fad7811a910 tx=0x7fad740109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:51.727 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:51.721+0000 7fad6e7fc700 1 -- 172.21.15.42:0/1867621962 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fad74032260 con 0x7fad78121a70 2019-12-05T05:25:51.900 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:51.897+0000 7fad7f4f0700 1 -- 172.21.15.42:0/1867621962 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fad7805b8c0 con 0x7fad78121a70 2019-12-05T05:25:51.901 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:51.897+0000 7fad7da8d700 1 --2- 172.21.15.42:0/1867621962 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fad78121a70 0x7fad7812ee00 secure :-1 s=THROTTLE_DONE pgs=840 cs=0 l=1 rx=0x7fad7811a910 tx=0x7fad740109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:25:51.901 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:51.897+0000 7fad6e7fc700 1 -- 172.21.15.42:0/1867621962 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+2112 (secure 0 0 0) 0x7fad740324c0 con 0x7fad78121a70 2019-12-05T05:25:51.902 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 108 2019-12-05T05:25:51.902 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:25:51.902 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6546,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/1149478696","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1149478696},{"type":"v1","addr":"172.21.15.40:6833","nonce":1149478696}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":108}],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:25:38.087763+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5240},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5240":{"gid":5240,"name":"a","rank":0,"incarnation":104,"state":"up:active","state_seq":27,"addr":"172.21.15.42:6835/1455108684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1455108684},{"type":"v1","addr":"172.21.15.42:6835","nonce":1455108684}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:25:51.906 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:51.901+0000 7fad7f4f0700 1 -- 172.21.15.42:0/1867621962 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fad6003adc0 msgr2=0x7fad6003d2d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:25:51.906 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:51.901+0000 7fad7f4f0700 1 --2- 172.21.15.42:0/1867621962 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fad6003adc0 0x7fad6003d2d0 crc :-1 s=READY pgs=372 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:25:51.906 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:51.901+0000 7fad7f4f0700 1 -- 172.21.15.42:0/1867621962 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fad78121a70 msgr2=0x7fad7812ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:25:51.906 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:51.901+0000 7fad7f4f0700 1 --2- 172.21.15.42:0/1867621962 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fad78121a70 0x7fad7812ee00 secure :-1 s=READY pgs=840 cs=0 l=1 rx=0x7fad7811a910 tx=0x7fad740109a0).stop 2019-12-05T05:25:51.906 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:51.901+0000 7fad7f4f0700 1 -- 172.21.15.42:0/1867621962 shutdown_connections 2019-12-05T05:25:51.906 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:51.901+0000 7fad7f4f0700 1 --2- 172.21.15.42:0/1867621962 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fad6003adc0 0x7fad6003d2d0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:25:51.906 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:51.901+0000 7fad7f4f0700 1 --2- 172.21.15.42:0/1867621962 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fad78121a70 0x7fad7812ee00 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:25:51.907 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:51.901+0000 7fad7f4f0700 1 --2- 172.21.15.42:0/1867621962 >> v2:172.21.15.40:3300/0 conn(0x7fad78118260 0x7fad78118690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:25:51.907 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:51.901+0000 7fad7f4f0700 1 -- 172.21.15.42:0/1867621962 shutdown_connections 2019-12-05T05:25:51.907 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:51.901+0000 7fad7f4f0700 1 -- 172.21.15.42:0/1867621962 wait complete. 2019-12-05T05:25:51.907 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:25:51.901+0000 7fad7f4f0700 1 -- 172.21.15.42:0/1867621962 >> 172.21.15.42:0/1867621962 conn(0x7fad78141360 msgr2=0x7fad780a97c0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:25:51.917 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b reported in up:standby state 2019-12-05T05:25:51.918 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 58 secs before thrashing 2019-12-05T05:26:08.441 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:26:08.456 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:26:38.491 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:26:38.496 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:26:49.919 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:26:50.045 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.032+0000 7f4f3588d700 1 Processor -- start 2019-12-05T05:26:50.045 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.032+0000 7f4f3588d700 1 -- start start 2019-12-05T05:26:50.045 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.032+0000 7f4f3588d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f4f300a7730 0x7f4f3008df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:26:50.046 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.032+0000 7f4f3588d700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4f3007e710 0x7f4f3007eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:26:50.046 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.036+0000 7f4f3588d700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f300a0700 con 0x7f4f3009f2d0 2019-12-05T05:26:50.046 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.036+0000 7f4f3588d700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f300a0af0 con 0x7f4f300a7730 2019-12-05T05:26:50.046 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.036+0000 7f4f3588d700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f3008e5a0 con 0x7f4f3007e710 2019-12-05T05:26:50.047 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.036+0000 7f4f2f7fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f4f300a7730 0x7f4f3008df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:26:50.047 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.036+0000 7f4f2f7fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f4f300a7730 0x7f4f3008df70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:33812/0 (socket says 172.21.15.42:33812) 2019-12-05T05:26:50.047 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.036+0000 7f4f2f7fe700 1 -- 172.21.15.42:0/919239314 learned_addr learned my addr 172.21.15.42:0/919239314 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:26:50.047 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.036+0000 7f4f2effd700 1 --2- 172.21.15.42:0/919239314 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4f3007e710 0x7f4f3007eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:26:50.047 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.036+0000 7f4f2f7fe700 1 -- 172.21.15.42:0/919239314 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4f3007e710 msgr2=0x7f4f3007eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:26:50.048 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.036+0000 7f4f2f7fe700 1 --2- 172.21.15.42:0/919239314 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4f3007e710 0x7f4f3007eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:26:50.048 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.036+0000 7f4f2f7fe700 1 -- 172.21.15.42:0/919239314 >> v1:172.21.15.40:6789/0 conn(0x7f4f3009f2d0 legacy=0x7f4f3009f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:26:50.048 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.036+0000 7f4f2f7fe700 1 -- 172.21.15.42:0/919239314 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f3007f090 con 0x7f4f300a7730 2019-12-05T05:26:50.048 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.036+0000 7f4f2f7fe700 1 --2- 172.21.15.42:0/919239314 >> v2:172.21.15.42:3300/0 conn(0x7f4f300a7730 0x7f4f3008df70 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f4f20008790 tx=0x7f4f2000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:26:50.048 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.036+0000 7f4f2f7fe700 1 --2- 172.21.15.42:0/919239314 >> v2:172.21.15.42:3300/0 conn(0x7f4f300a7730 0x7f4f3008df70 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f4f20008790 tx=0x7f4f2000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:26:50.048 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.036+0000 7f4f2f7fe700 1 --2- 172.21.15.42:0/919239314 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4f300a7730 0x7f4f3008df70 secure :-1 s=READY pgs=849 cs=0 l=1 rx=0x7f4f20008790 tx=0x7f4f2000fd30).ready entity=mon.1 client_cookie=13a4c254fd12015f server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:26:50.049 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.036+0000 7f4f2f7fe700 1 --2- 172.21.15.42:0/919239314 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4f300a7730 0x7f4f3008df70 secure :-1 s=THROTTLE_DONE pgs=849 cs=0 l=1 rx=0x7f4f20008790 tx=0x7f4f2000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:26:50.049 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.036+0000 7f4f2e7fc700 1 -- 172.21.15.42:0/919239314 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4f200113b0 con 0x7f4f300a7730 2019-12-05T05:26:50.049 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.036+0000 7f4f2f7fe700 1 --2- 172.21.15.42:0/919239314 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4f300a7730 0x7f4f3008df70 secure :-1 s=THROTTLE_DONE pgs=849 cs=0 l=1 rx=0x7f4f20008790 tx=0x7f4f2000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:26:50.049 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.036+0000 7f4f2f7fe700 1 --2- 172.21.15.42:0/919239314 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4f300a7730 0x7f4f3008df70 secure :-1 s=THROTTLE_DONE pgs=849 cs=0 l=1 rx=0x7f4f20008790 tx=0x7f4f2000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:26:50.049 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.036+0000 7f4f2e7fc700 1 -- 172.21.15.42:0/919239314 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f20008bb0 con 0x7f4f300a7730 2019-12-05T05:26:50.049 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.036+0000 7f4f2e7fc700 1 -- 172.21.15.42:0/919239314 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4f20011990 con 0x7f4f300a7730 2019-12-05T05:26:50.050 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.036+0000 7f4f3588d700 1 -- 172.21.15.42:0/919239314 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4f300a7730 msgr2=0x7f4f3008df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:26:50.050 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.036+0000 7f4f3588d700 1 --2- 172.21.15.42:0/919239314 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4f300a7730 0x7f4f3008df70 secure :-1 s=READY pgs=849 cs=0 l=1 rx=0x7f4f20008790 tx=0x7f4f2000fd30).stop 2019-12-05T05:26:50.050 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.036+0000 7f4f3588d700 1 -- 172.21.15.42:0/919239314 shutdown_connections 2019-12-05T05:26:50.050 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.036+0000 7f4f3588d700 1 --2- 172.21.15.42:0/919239314 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4f300a7730 0x7f4f3008df70 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:26:50.051 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.036+0000 7f4f3588d700 1 --2- 172.21.15.42:0/919239314 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4f3007e710 0x7f4f3007eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:26:50.051 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.036+0000 7f4f3588d700 1 -- 172.21.15.42:0/919239314 shutdown_connections 2019-12-05T05:26:50.051 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.036+0000 7f4f3588d700 1 -- 172.21.15.42:0/919239314 wait complete. 2019-12-05T05:26:50.051 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.036+0000 7f4f3588d700 1 -- 172.21.15.42:0/919239314 >> 172.21.15.42:0/919239314 conn(0x7f4f300aabd0 msgr2=0x7f4f300a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:26:50.069 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.040+0000 7f4f3588d700 1 Processor -- start 2019-12-05T05:26:50.069 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.040+0000 7f4f3588d700 1 -- start start 2019-12-05T05:26:50.069 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.040+0000 7f4f3588d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f4f30119db0 0x7f4f30115a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:26:50.069 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.040+0000 7f4f3588d700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4f30113380 0x7f4f301137b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:26:50.069 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.040+0000 7f4f3588d700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f30115fc0 con 0x7f4f30127b50 2019-12-05T05:26:50.070 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.040+0000 7f4f3588d700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f3011a1e0 con 0x7f4f30119db0 2019-12-05T05:26:50.070 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.040+0000 7f4f3588d700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f30127370 con 0x7f4f30113380 2019-12-05T05:26:50.071 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.040+0000 7f4f2f7fe700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f4f30127b50 0x7f4f30123910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:33856/0 (socket says 172.21.15.42:33856) 2019-12-05T05:26:50.071 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.040+0000 7f4f2effd700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f4f30119db0 0x7f4f30115a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:26:50.071 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.040+0000 7f4f2f7fe700 1 -- 172.21.15.42:0/3260096383 learned_addr learned my addr 172.21.15.42:0/3260096383 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:26:50.071 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.040+0000 7f4f2ffff700 1 --2- 172.21.15.42:0/3260096383 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4f30113380 0x7f4f301137b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:26:50.071 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.044+0000 7f4f2effd700 1 -- 172.21.15.42:0/3260096383 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4f30113380 msgr2=0x7f4f301137b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:26:50.071 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.044+0000 7f4f2effd700 1 --2- 172.21.15.42:0/3260096383 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4f30113380 0x7f4f301137b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:26:50.072 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.044+0000 7f4f2effd700 1 -- 172.21.15.42:0/3260096383 >> v1:172.21.15.42:6789/0 conn(0x7f4f30127b50 legacy=0x7f4f30123910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:26:50.072 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.044+0000 7f4f17fff700 1 -- 172.21.15.42:0/3260096383 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1184931885 0 0) 0x7f4f200078f0 con 0x7f4f30127b50 2019-12-05T05:26:50.072 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.044+0000 7f4f2effd700 1 -- 172.21.15.42:0/3260096383 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f30126ed0 con 0x7f4f30119db0 2019-12-05T05:26:50.073 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.044+0000 7f4f2effd700 1 --2- 172.21.15.42:0/3260096383 >> v2:172.21.15.42:3300/0 conn(0x7f4f30119db0 0x7f4f30115a70 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f4f24002f30 tx=0x7f4f24012d00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:26:50.073 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.044+0000 7f4f2effd700 1 --2- 172.21.15.42:0/3260096383 >> v2:172.21.15.42:3300/0 conn(0x7f4f30119db0 0x7f4f30115a70 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f4f24002f30 tx=0x7f4f24012d00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:26:50.073 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.044+0000 7f4f2effd700 1 --2- 172.21.15.42:0/3260096383 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4f30119db0 0x7f4f30115a70 secure :-1 s=READY pgs=851 cs=0 l=1 rx=0x7f4f24002f30 tx=0x7f4f24012d00).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:26:50.073 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.044+0000 7f4f2effd700 1 --2- 172.21.15.42:0/3260096383 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4f30119db0 0x7f4f30115a70 secure :-1 s=THROTTLE_DONE pgs=851 cs=0 l=1 rx=0x7f4f24002f30 tx=0x7f4f24012d00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:26:50.073 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.044+0000 7f4f17fff700 1 -- 172.21.15.42:0/3260096383 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4f24014b90 con 0x7f4f30119db0 2019-12-05T05:26:50.074 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.044+0000 7f4f2effd700 1 --2- 172.21.15.42:0/3260096383 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4f30119db0 0x7f4f30115a70 secure :-1 s=THROTTLE_DONE pgs=851 cs=0 l=1 rx=0x7f4f24002f30 tx=0x7f4f24012d00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:26:50.074 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.044+0000 7f4f2effd700 1 --2- 172.21.15.42:0/3260096383 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4f30119db0 0x7f4f30115a70 secure :-1 s=THROTTLE_DONE pgs=851 cs=0 l=1 rx=0x7f4f24002f30 tx=0x7f4f24012d00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:26:50.074 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.044+0000 7f4f17fff700 1 -- 172.21.15.42:0/3260096383 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f240147b0 con 0x7f4f30119db0 2019-12-05T05:26:50.074 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.044+0000 7f4f3588d700 1 -- 172.21.15.42:0/3260096383 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f3008eb60 con 0x7f4f30119db0 2019-12-05T05:26:50.074 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.044+0000 7f4f17fff700 1 -- 172.21.15.42:0/3260096383 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4f24014d90 con 0x7f4f30119db0 2019-12-05T05:26:50.074 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.044+0000 7f4f3588d700 1 -- 172.21.15.42:0/3260096383 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f3008f110 con 0x7f4f30119db0 2019-12-05T05:26:50.075 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.044+0000 7f4f2effd700 1 --2- 172.21.15.42:0/3260096383 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4f30119db0 0x7f4f30115a70 secure :-1 s=THROTTLE_DONE pgs=851 cs=0 l=1 rx=0x7f4f24002f30 tx=0x7f4f24012d00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:26:50.075 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.044+0000 7f4f17fff700 1 -- 172.21.15.42:0/3260096383 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f4f24007180 con 0x7f4f30119db0 2019-12-05T05:26:50.075 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.044+0000 7f4f2effd700 1 --2- 172.21.15.42:0/3260096383 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4f30119db0 0x7f4f30115a70 secure :-1 s=THROTTLE_DONE pgs=851 cs=0 l=1 rx=0x7f4f24002f30 tx=0x7f4f24012d00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:26:50.075 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.044+0000 7f4f17fff700 1 --2- 172.21.15.42:0/3260096383 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4f1003ac50 0x7f4f1003d160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:26:50.075 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.044+0000 7f4f17fff700 1 -- 172.21.15.42:0/3260096383 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7321+0+0 (secure 0 0 0) 0x7f4f2404d3e0 con 0x7f4f30119db0 2019-12-05T05:26:50.075 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.044+0000 7f4f2f7fe700 1 --2- 172.21.15.42:0/3260096383 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4f1003ac50 0x7f4f1003d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:26:50.076 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.048+0000 7f4f2f7fe700 1 --2- 172.21.15.42:0/3260096383 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4f1003ac50 0x7f4f1003d160 crc :-1 s=READY pgs=375 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:26:50.076 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.048+0000 7f4f3588d700 1 -- 172.21.15.42:0/3260096383 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f3005d100 con 0x7f4f30119db0 2019-12-05T05:26:50.076 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.056+0000 7f4f2effd700 1 --2- 172.21.15.42:0/3260096383 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4f30119db0 0x7f4f30115a70 secure :-1 s=THROTTLE_DONE pgs=851 cs=0 l=1 rx=0x7f4f24002f30 tx=0x7f4f24012d00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:26:50.076 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.056+0000 7f4f17fff700 1 -- 172.21.15.42:0/3260096383 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f4f24034440 con 0x7f4f30119db0 2019-12-05T05:26:50.227 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.220+0000 7f4f3588d700 1 -- 172.21.15.42:0/3260096383 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4f3005b8b0 con 0x7f4f30119db0 2019-12-05T05:26:50.228 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.220+0000 7f4f2effd700 1 --2- 172.21.15.42:0/3260096383 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4f30119db0 0x7f4f30115a70 secure :-1 s=THROTTLE_DONE pgs=851 cs=0 l=1 rx=0x7f4f24002f30 tx=0x7f4f24012d00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:26:50.228 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.220+0000 7f4f17fff700 1 -- 172.21.15.42:0/3260096383 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f4f240346a0 con 0x7f4f30119db0 2019-12-05T05:26:50.228 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 108 2019-12-05T05:26:50.228 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:26:50.228 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6546,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/1149478696","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1149478696},{"type":"v1","addr":"172.21.15.40:6833","nonce":1149478696}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":108}],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:25:38.087763+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5240},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5240":{"gid":5240,"name":"a","rank":0,"incarnation":104,"state":"up:active","state_seq":27,"addr":"172.21.15.42:6835/1455108684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1455108684},{"type":"v1","addr":"172.21.15.42:6835","nonce":1455108684}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:26:50.231 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.224+0000 7f4f3588d700 1 -- 172.21.15.42:0/3260096383 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4f1003ac50 msgr2=0x7f4f1003d160 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:26:50.231 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.224+0000 7f4f3588d700 1 --2- 172.21.15.42:0/3260096383 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4f1003ac50 0x7f4f1003d160 crc :-1 s=READY pgs=375 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:26:50.231 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.224+0000 7f4f3588d700 1 -- 172.21.15.42:0/3260096383 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4f30119db0 msgr2=0x7f4f30115a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:26:50.231 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.224+0000 7f4f3588d700 1 --2- 172.21.15.42:0/3260096383 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4f30119db0 0x7f4f30115a70 secure :-1 s=READY pgs=851 cs=0 l=1 rx=0x7f4f24002f30 tx=0x7f4f24012d00).stop 2019-12-05T05:26:50.231 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.224+0000 7f4f3588d700 1 -- 172.21.15.42:0/3260096383 shutdown_connections 2019-12-05T05:26:50.233 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.224+0000 7f4f3588d700 1 --2- 172.21.15.42:0/3260096383 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4f1003ac50 0x7f4f1003d160 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:26:50.233 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.224+0000 7f4f3588d700 1 --2- 172.21.15.42:0/3260096383 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4f30113380 0x7f4f301137b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:26:50.233 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.224+0000 7f4f3588d700 1 --2- 172.21.15.42:0/3260096383 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4f30119db0 0x7f4f30115a70 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:26:50.234 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.224+0000 7f4f3588d700 1 -- 172.21.15.42:0/3260096383 shutdown_connections 2019-12-05T05:26:50.234 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.224+0000 7f4f3588d700 1 -- 172.21.15.42:0/3260096383 wait complete. 2019-12-05T05:26:50.234 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.224+0000 7f4f3588d700 1 -- 172.21.15.42:0/3260096383 >> 172.21.15.42:0/3260096383 conn(0x7f4f300a98e0 msgr2=0x7f4f3009f230 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:26:50.239 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.a (rank=0) 2019-12-05T05:26:50.239 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2019-12-05T05:26:50.240 INFO:teuthology.orchestra.run:waiting for 300 2019-12-05T05:26:50.249 INFO:tasks.ceph.mds.a:Stopped 2019-12-05T05:26:50.249 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:26:50.429 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.416+0000 7f8de3f9a700 1 Processor -- start 2019-12-05T05:26:50.429 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.416+0000 7f8de3f9a700 1 -- start start 2019-12-05T05:26:50.429 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.420+0000 7f8de3f9a700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8ddc08df80 0x7f8ddc09f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:26:50.430 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.420+0000 7f8de3f9a700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8ddc09f830 0x7f8ddc14efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:26:50.430 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.420+0000 7f8de3f9a700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ddc09fc60 con 0x7f8ddc0a7740 2019-12-05T05:26:50.430 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.420+0000 7f8de3f9a700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ddc0a0b00 con 0x7f8ddc08df80 2019-12-05T05:26:50.430 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.420+0000 7f8de3f9a700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ddc08e5b0 con 0x7f8ddc09f830 2019-12-05T05:26:50.430 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.420+0000 7f8de2537700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f8ddc0a7740 0x7f8ddc07e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:33866/0 (socket says 172.21.15.42:33866) 2019-12-05T05:26:50.430 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.420+0000 7f8de2537700 1 -- 172.21.15.42:0/1516087808 learned_addr learned my addr 172.21.15.42:0/1516087808 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:26:50.431 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.420+0000 7f8de1d36700 1 --2- 172.21.15.42:0/1516087808 >> v2:172.21.15.40:3300/0 conn(0x7f8ddc08df80 0x7f8ddc09f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:26:50.431 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.420+0000 7f8de2d38700 1 --2- 172.21.15.42:0/1516087808 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8ddc09f830 0x7f8ddc14efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:26:50.431 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.420+0000 7f8de1535700 1 -- 172.21.15.42:0/1516087808 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2510797844 0 0) 0x7f8ddc09fc60 con 0x7f8ddc0a7740 2019-12-05T05:26:50.431 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.420+0000 7f8de1535700 1 -- 172.21.15.42:0/1516087808 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8dc80030b0 con 0x7f8ddc0a7740 2019-12-05T05:26:50.432 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.420+0000 7f8de1d36700 1 -- 172.21.15.42:0/1516087808 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8ddc09f830 msgr2=0x7f8ddc14efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:26:50.432 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.420+0000 7f8de1d36700 1 --2- 172.21.15.42:0/1516087808 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8ddc09f830 0x7f8ddc14efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:26:50.432 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.420+0000 7f8de1535700 1 -- 172.21.15.42:0/1516087808 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1381203715 0 0) 0x7f8dc80030b0 con 0x7f8ddc0a7740 2019-12-05T05:26:50.432 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.420+0000 7f8de1d36700 1 -- 172.21.15.42:0/1516087808 >> v1:172.21.15.42:6789/0 conn(0x7f8ddc0a7740 legacy=0x7f8ddc07e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:26:50.432 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.420+0000 7f8de1d36700 1 -- 172.21.15.42:0/1516087808 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ddc0a0710 con 0x7f8ddc08df80 2019-12-05T05:26:50.432 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.420+0000 7f8de1d36700 1 --2- 172.21.15.42:0/1516087808 >> v2:172.21.15.40:3300/0 conn(0x7f8ddc08df80 0x7f8ddc09f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f8dcc008790 tx=0x7f8dd8001820).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:26:50.433 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.420+0000 7f8de1d36700 1 --2- 172.21.15.42:0/1516087808 >> v2:172.21.15.40:3300/0 conn(0x7f8ddc08df80 0x7f8ddc09f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f8dcc008790 tx=0x7f8dd8001820).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:26:50.433 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.420+0000 7f8de1d36700 1 --2- 172.21.15.42:0/1516087808 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8ddc08df80 0x7f8ddc09f2e0 secure :-1 s=READY pgs=679 cs=0 l=1 rx=0x7f8dcc008790 tx=0x7f8dd8001820).ready entity=mon.0 client_cookie=1a7d1e7f2ec66c2e server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:26:50.433 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.420+0000 7f8de1d36700 1 --2- 172.21.15.42:0/1516087808 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8ddc08df80 0x7f8ddc09f2e0 secure :-1 s=THROTTLE_DONE pgs=679 cs=0 l=1 rx=0x7f8dcc008790 tx=0x7f8dd8001820).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:26:50.433 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.420+0000 7f8de1535700 1 -- 172.21.15.42:0/1516087808 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8dd8001ad0 con 0x7f8ddc08df80 2019-12-05T05:26:50.433 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.420+0000 7f8de1d36700 1 --2- 172.21.15.42:0/1516087808 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8ddc08df80 0x7f8ddc09f2e0 secure :-1 s=THROTTLE_DONE pgs=679 cs=0 l=1 rx=0x7f8dcc008790 tx=0x7f8dd8001820).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:26:50.434 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.420+0000 7f8de1d36700 1 --2- 172.21.15.42:0/1516087808 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8ddc08df80 0x7f8ddc09f2e0 secure :-1 s=THROTTLE_DONE pgs=679 cs=0 l=1 rx=0x7f8dcc008790 tx=0x7f8dd8001820).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:26:50.434 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.420+0000 7f8de1535700 1 -- 172.21.15.42:0/1516087808 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8dd80018e0 con 0x7f8ddc08df80 2019-12-05T05:26:50.434 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.420+0000 7f8de1535700 1 -- 172.21.15.42:0/1516087808 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8dcc008da0 con 0x7f8ddc08df80 2019-12-05T05:26:50.434 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.420+0000 7f8de3f9a700 1 -- 172.21.15.42:0/1516087808 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8ddc08df80 msgr2=0x7f8ddc09f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:26:50.434 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.420+0000 7f8de3f9a700 1 --2- 172.21.15.42:0/1516087808 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8ddc08df80 0x7f8ddc09f2e0 secure :-1 s=READY pgs=679 cs=0 l=1 rx=0x7f8dcc008790 tx=0x7f8dd8001820).stop 2019-12-05T05:26:50.434 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.420+0000 7f8de3f9a700 1 -- 172.21.15.42:0/1516087808 shutdown_connections 2019-12-05T05:26:50.435 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.420+0000 7f8de3f9a700 1 --2- 172.21.15.42:0/1516087808 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8ddc09f830 0x7f8ddc14efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:26:50.435 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.420+0000 7f8de3f9a700 1 --2- 172.21.15.42:0/1516087808 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8ddc08df80 0x7f8ddc09f2e0 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:26:50.435 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.424+0000 7f8de3f9a700 1 -- 172.21.15.42:0/1516087808 shutdown_connections 2019-12-05T05:26:50.435 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.424+0000 7f8de3f9a700 1 -- 172.21.15.42:0/1516087808 wait complete. 2019-12-05T05:26:50.435 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.424+0000 7f8de3f9a700 1 -- 172.21.15.42:0/1516087808 >> 172.21.15.42:0/1516087808 conn(0x7f8ddc0aabe0 msgr2=0x7f8ddc0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:26:50.438 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.424+0000 7f8de3f9a700 1 Processor -- start 2019-12-05T05:26:50.439 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.424+0000 7f8de3f9a700 1 -- start start 2019-12-05T05:26:50.439 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.428+0000 7f8de3f9a700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f8ddc07d340 0x7f8ddc119220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:26:50.439 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.428+0000 7f8de3f9a700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ddc07b9d0 con 0x7f8ddc119770 2019-12-05T05:26:50.439 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.428+0000 7f8de3f9a700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ddc07b5d0 con 0x7f8ddc119ba0 2019-12-05T05:26:50.439 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.428+0000 7f8de3f9a700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ddc07bd50 con 0x7f8ddc07d340 2019-12-05T05:26:50.439 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.428+0000 7f8de2537700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f8ddc07d340 0x7f8ddc119220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:26:50.439 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.428+0000 7f8de2d38700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f8ddc119ba0 0x7f8ddc11b780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:33876/0 (socket says 172.21.15.42:33876) 2019-12-05T05:26:50.440 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.428+0000 7f8de2d38700 1 -- 172.21.15.42:0/276147148 learned_addr learned my addr 172.21.15.42:0/276147148 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:26:50.440 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.428+0000 7f8de2537700 1 -- 172.21.15.42:0/276147148 >> v1:172.21.15.42:6789/0 conn(0x7f8ddc119ba0 legacy=0x7f8ddc11b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:26:50.440 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.428+0000 7f8de2537700 1 -- 172.21.15.42:0/276147148 >> v1:172.21.15.40:6789/0 conn(0x7f8ddc119770 legacy=0x7f8ddc11af80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:26:50.440 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.428+0000 7f8de2537700 1 -- 172.21.15.42:0/276147148 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ddc07d770 con 0x7f8ddc07d340 2019-12-05T05:26:50.440 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.428+0000 7f8dd2ffd700 1 -- 172.21.15.42:0/276147148 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3672970792 0 0) 0x7f8ddc07b5d0 con 0x7f8ddc119ba0 2019-12-05T05:26:50.440 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.428+0000 7f8de2537700 1 --2- 172.21.15.42:0/276147148 >> v2:172.21.15.42:3300/0 conn(0x7f8ddc07d340 0x7f8ddc119220 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f8ddc11c400 tx=0x7f8dd8001820).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:26:50.441 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.428+0000 7f8de2537700 1 --2- 172.21.15.42:0/276147148 >> v2:172.21.15.42:3300/0 conn(0x7f8ddc07d340 0x7f8ddc119220 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f8ddc11c400 tx=0x7f8dd8001820).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:26:50.441 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.428+0000 7f8de2537700 1 --2- 172.21.15.42:0/276147148 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8ddc07d340 0x7f8ddc119220 secure :-1 s=READY pgs=854 cs=0 l=1 rx=0x7f8ddc11c400 tx=0x7f8dd8001820).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:26:50.441 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.428+0000 7f8de2537700 1 --2- 172.21.15.42:0/276147148 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8ddc07d340 0x7f8ddc119220 secure :-1 s=THROTTLE_DONE pgs=854 cs=0 l=1 rx=0x7f8ddc11c400 tx=0x7f8dd8001820).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:26:50.441 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.428+0000 7f8dd2ffd700 1 -- 172.21.15.42:0/276147148 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8dd8013060 con 0x7f8ddc07d340 2019-12-05T05:26:50.441 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.428+0000 7f8de2537700 1 --2- 172.21.15.42:0/276147148 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8ddc07d340 0x7f8ddc119220 secure :-1 s=THROTTLE_DONE pgs=854 cs=0 l=1 rx=0x7f8ddc11c400 tx=0x7f8dd8001820).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:26:50.442 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.428+0000 7f8de2537700 1 --2- 172.21.15.42:0/276147148 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8ddc07d340 0x7f8ddc119220 secure :-1 s=THROTTLE_DONE pgs=854 cs=0 l=1 rx=0x7f8ddc11c400 tx=0x7f8dd8001820).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:26:50.442 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.428+0000 7f8dd2ffd700 1 -- 172.21.15.42:0/276147148 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8dd800cc00 con 0x7f8ddc07d340 2019-12-05T05:26:50.442 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.428+0000 7f8de3f9a700 1 -- 172.21.15.42:0/276147148 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ddc118970 con 0x7f8ddc07d340 2019-12-05T05:26:50.442 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.428+0000 7f8dd2ffd700 1 -- 172.21.15.42:0/276147148 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8dd8004b90 con 0x7f8ddc07d340 2019-12-05T05:26:50.442 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.428+0000 7f8de3f9a700 1 -- 172.21.15.42:0/276147148 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ddc0a4840 con 0x7f8ddc07d340 2019-12-05T05:26:50.442 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.428+0000 7f8de2537700 1 --2- 172.21.15.42:0/276147148 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8ddc07d340 0x7f8ddc119220 secure :-1 s=THROTTLE_DONE pgs=854 cs=0 l=1 rx=0x7f8ddc11c400 tx=0x7f8dd8001820).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:26:50.442 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.428+0000 7f8dd2ffd700 1 -- 172.21.15.42:0/276147148 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f8dd8006180 con 0x7f8ddc07d340 2019-12-05T05:26:50.443 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.428+0000 7f8de2537700 1 --2- 172.21.15.42:0/276147148 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8ddc07d340 0x7f8ddc119220 secure :-1 s=THROTTLE_DONE pgs=854 cs=0 l=1 rx=0x7f8ddc11c400 tx=0x7f8dd8001820).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:26:50.443 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.428+0000 7f8dd2ffd700 1 --2- 172.21.15.42:0/276147148 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8dc403f000 0x7f8dc4041510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:26:50.443 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.428+0000 7f8dd2ffd700 1 -- 172.21.15.42:0/276147148 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7321+0+0 (secure 0 0 0) 0x7f8dd804ae40 con 0x7f8ddc07d340 2019-12-05T05:26:50.443 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.432+0000 7f8de1d36700 1 --2- 172.21.15.42:0/276147148 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8dc403f000 0x7f8dc4041510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:26:50.443 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.432+0000 7f8de1d36700 1 --2- 172.21.15.42:0/276147148 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8dc403f000 0x7f8dc4041510 crc :-1 s=READY pgs=376 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:26:50.443 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.432+0000 7f8de3f9a700 1 -- 172.21.15.42:0/276147148 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ddc05d110 con 0x7f8ddc07d340 2019-12-05T05:26:50.446 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.440+0000 7f8de2537700 1 --2- 172.21.15.42:0/276147148 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8ddc07d340 0x7f8ddc119220 secure :-1 s=THROTTLE_DONE pgs=854 cs=0 l=1 rx=0x7f8ddc11c400 tx=0x7f8dd8001820).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:26:50.446 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.440+0000 7f8dd2ffd700 1 -- 172.21.15.42:0/276147148 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f8dd8031ea0 con 0x7f8ddc07d340 2019-12-05T05:26:50.618 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.612+0000 7f8de3f9a700 1 -- 172.21.15.42:0/276147148 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8ddc05b8c0 con 0x7f8ddc07d340 2019-12-05T05:26:50.619 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.612+0000 7f8de2537700 1 --2- 172.21.15.42:0/276147148 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8ddc07d340 0x7f8ddc119220 secure :-1 s=THROTTLE_DONE pgs=854 cs=0 l=1 rx=0x7f8ddc11c400 tx=0x7f8dd8001820).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:26:50.619 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.612+0000 7f8dd2ffd700 1 -- 172.21.15.42:0/276147148 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f8dd8032100 con 0x7f8ddc07d340 2019-12-05T05:26:50.619 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 108 2019-12-05T05:26:50.620 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:26:50.620 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6546,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/1149478696","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1149478696},{"type":"v1","addr":"172.21.15.40:6833","nonce":1149478696}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":108}],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:25:38.087763+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5240},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5240":{"gid":5240,"name":"a","rank":0,"incarnation":104,"state":"up:active","state_seq":27,"addr":"172.21.15.42:6835/1455108684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1455108684},{"type":"v1","addr":"172.21.15.42:6835","nonce":1455108684}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:26:50.622 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.612+0000 7f8de3f9a700 1 -- 172.21.15.42:0/276147148 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8dc403f000 msgr2=0x7f8dc4041510 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:26:50.622 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.612+0000 7f8de3f9a700 1 --2- 172.21.15.42:0/276147148 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8dc403f000 0x7f8dc4041510 crc :-1 s=READY pgs=376 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:26:50.622 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.612+0000 7f8de3f9a700 1 -- 172.21.15.42:0/276147148 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8ddc07d340 msgr2=0x7f8ddc119220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:26:50.622 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.612+0000 7f8de3f9a700 1 --2- 172.21.15.42:0/276147148 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8ddc07d340 0x7f8ddc119220 secure :-1 s=READY pgs=854 cs=0 l=1 rx=0x7f8ddc11c400 tx=0x7f8dd8001820).stop 2019-12-05T05:26:50.623 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.616+0000 7f8de3f9a700 1 -- 172.21.15.42:0/276147148 shutdown_connections 2019-12-05T05:26:50.623 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.616+0000 7f8de3f9a700 1 --2- 172.21.15.42:0/276147148 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8dc403f000 0x7f8dc4041510 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:26:50.623 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.616+0000 7f8de3f9a700 1 --2- 172.21.15.42:0/276147148 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8ddc07d340 0x7f8ddc119220 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:26:50.623 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.616+0000 7f8de3f9a700 1 -- 172.21.15.42:0/276147148 shutdown_connections 2019-12-05T05:26:50.624 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.616+0000 7f8de3f9a700 1 -- 172.21.15.42:0/276147148 wait complete. 2019-12-05T05:26:50.624 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:50.616+0000 7f8de3f9a700 1 -- 172.21.15.42:0/276147148 >> 172.21.15.42:0/276147148 conn(0x7f8ddc0a7740 msgr2=0x7f8ddc059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:26:50.631 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T05:26:52.636 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:26:52.761 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:52.748+0000 7fd549ecf700 1 Processor -- start 2019-12-05T05:26:52.762 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:52.748+0000 7fd549ecf700 1 -- start start 2019-12-05T05:26:52.762 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:52.752+0000 7fd549ecf700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd5440a7740 0x7fd54408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:26:52.762 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:52.752+0000 7fd549ecf700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd54407e720 0x7fd54407eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:26:52.762 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:52.752+0000 7fd549ecf700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5440a0710 con 0x7fd54409f2e0 2019-12-05T05:26:52.762 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:52.752+0000 7fd549ecf700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5440a0b00 con 0x7fd5440a7740 2019-12-05T05:26:52.762 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:52.752+0000 7fd549ecf700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd54408e5b0 con 0x7fd54407e720 2019-12-05T05:26:52.762 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:52.752+0000 7fd5437fe700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd54407e720 0x7fd54407eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:26:52.763 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:52.752+0000 7fd543fff700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd5440a7740 0x7fd54408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:26:52.763 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:52.752+0000 7fd5437fe700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd54407e720 0x7fd54407eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:34322/0 (socket says 172.21.15.42:34322) 2019-12-05T05:26:52.763 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:52.752+0000 7fd5437fe700 1 -- 172.21.15.42:0/1800503322 learned_addr learned my addr 172.21.15.42:0/1800503322 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:26:52.763 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:52.752+0000 7fd5437fe700 1 -- 172.21.15.42:0/1800503322 >> v2:172.21.15.40:3300/0 conn(0x7fd5440a7740 msgr2=0x7fd54408df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:26:52.763 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:52.752+0000 7fd5437fe700 1 --2- 172.21.15.42:0/1800503322 >> v2:172.21.15.40:3300/0 conn(0x7fd5440a7740 0x7fd54408df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:26:52.763 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:52.752+0000 7fd5437fe700 1 -- 172.21.15.42:0/1800503322 >> v1:172.21.15.40:6789/0 conn(0x7fd54409f2e0 legacy=0x7fd54409f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:26:52.763 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:52.752+0000 7fd5437fe700 1 -- 172.21.15.42:0/1800503322 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd54407f0a0 con 0x7fd54407e720 2019-12-05T05:26:52.764 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:52.752+0000 7fd5437fe700 1 --2- 172.21.15.42:0/1800503322 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd54407e720 0x7fd54407eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fd52c008790 tx=0x7fd52c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:26:52.764 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:52.752+0000 7fd5437fe700 1 --2- 172.21.15.42:0/1800503322 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd54407e720 0x7fd54407eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fd52c008790 tx=0x7fd52c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:26:52.764 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:52.752+0000 7fd5437fe700 1 --2- 172.21.15.42:0/1800503322 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd54407e720 0x7fd54407eb50 secure :-1 s=READY pgs=177 cs=0 l=1 rx=0x7fd52c008790 tx=0x7fd52c00fd30).ready entity=mon.2 client_cookie=32b0ef6531a6997d server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:26:52.764 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:52.752+0000 7fd5437fe700 1 --2- 172.21.15.42:0/1800503322 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd54407e720 0x7fd54407eb50 secure :-1 s=THROTTLE_DONE pgs=177 cs=0 l=1 rx=0x7fd52c008790 tx=0x7fd52c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:26:52.764 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:52.752+0000 7fd5437fe700 1 --2- 172.21.15.42:0/1800503322 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd54407e720 0x7fd54407eb50 secure :-1 s=THROTTLE_DONE pgs=177 cs=0 l=1 rx=0x7fd52c008790 tx=0x7fd52c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:26:52.764 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:52.752+0000 7fd542ffd700 1 -- 172.21.15.42:0/1800503322 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd52c0113b0 con 0x7fd54407e720 2019-12-05T05:26:52.765 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:52.752+0000 7fd5437fe700 1 --2- 172.21.15.42:0/1800503322 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd54407e720 0x7fd54407eb50 secure :-1 s=THROTTLE_DONE pgs=177 cs=0 l=1 rx=0x7fd52c008790 tx=0x7fd52c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:26:52.765 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:52.752+0000 7fd542ffd700 1 -- 172.21.15.42:0/1800503322 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd52c008bb0 con 0x7fd54407e720 2019-12-05T05:26:52.765 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:52.752+0000 7fd542ffd700 1 -- 172.21.15.42:0/1800503322 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd52c011990 con 0x7fd54407e720 2019-12-05T05:26:52.765 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:52.752+0000 7fd549ecf700 1 -- 172.21.15.42:0/1800503322 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd54407e720 msgr2=0x7fd54407eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:26:52.765 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:52.752+0000 7fd549ecf700 1 --2- 172.21.15.42:0/1800503322 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd54407e720 0x7fd54407eb50 secure :-1 s=READY pgs=177 cs=0 l=1 rx=0x7fd52c008790 tx=0x7fd52c00fd30).stop 2019-12-05T05:26:52.765 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:52.756+0000 7fd549ecf700 1 -- 172.21.15.42:0/1800503322 shutdown_connections 2019-12-05T05:26:52.766 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:52.756+0000 7fd549ecf700 1 --2- 172.21.15.42:0/1800503322 >> v2:172.21.15.40:3300/0 conn(0x7fd5440a7740 0x7fd54408df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:26:52.766 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:52.756+0000 7fd549ecf700 1 --2- 172.21.15.42:0/1800503322 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd54407e720 0x7fd54407eb50 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:26:52.766 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:52.756+0000 7fd549ecf700 1 -- 172.21.15.42:0/1800503322 shutdown_connections 2019-12-05T05:26:52.766 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:52.756+0000 7fd549ecf700 1 -- 172.21.15.42:0/1800503322 wait complete. 2019-12-05T05:26:52.767 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:52.756+0000 7fd549ecf700 1 -- 172.21.15.42:0/1800503322 >> 172.21.15.42:0/1800503322 conn(0x7fd5440aabe0 msgr2=0x7fd5440a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:26:52.772 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:52.756+0000 7fd549ecf700 1 Processor -- start 2019-12-05T05:26:52.773 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:52.756+0000 7fd549ecf700 1 -- start start 2019-12-05T05:26:52.773 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:52.760+0000 7fd549ecf700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd544116500 0x7fd544122230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:26:52.773 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:52.760+0000 7fd549ecf700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd544123700 0x7fd544117560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:26:52.774 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:52.760+0000 7fd549ecf700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd544122780 con 0x7fd544117ab0 2019-12-05T05:26:52.774 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:52.760+0000 7fd549ecf700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5441231c0 con 0x7fd544123700 2019-12-05T05:26:52.774 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:52.760+0000 7fd549ecf700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd544123460 con 0x7fd544116500 2019-12-05T05:26:52.774 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:52.760+0000 7fd543fff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd544116500 0x7fd544122230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:26:52.774 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:52.760+0000 7fd543fff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd544116500 0x7fd544122230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:33860/0 (socket says 172.21.15.42:33860) 2019-12-05T05:26:52.775 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:52.760+0000 7fd543fff700 1 -- 172.21.15.42:0/476369885 learned_addr learned my addr 172.21.15.42:0/476369885 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:26:52.775 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:52.760+0000 7fd5437fe700 1 --2- 172.21.15.42:0/476369885 >> v2:172.21.15.40:3300/0 conn(0x7fd544123700 0x7fd544117560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:26:52.775 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:52.760+0000 7fd543fff700 1 -- 172.21.15.42:0/476369885 >> v2:172.21.15.40:3300/0 conn(0x7fd544123700 msgr2=0x7fd544117560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:26:52.775 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:52.760+0000 7fd543fff700 1 --2- 172.21.15.42:0/476369885 >> v2:172.21.15.40:3300/0 conn(0x7fd544123700 0x7fd544117560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:26:52.776 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:52.760+0000 7fd543fff700 1 -- 172.21.15.42:0/476369885 >> v1:172.21.15.42:6789/0 conn(0x7fd544117ab0 legacy=0x7fd54408eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:26:52.776 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:52.760+0000 7fd543fff700 1 -- 172.21.15.42:0/476369885 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd544119f00 con 0x7fd544116500 2019-12-05T05:26:52.776 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:52.760+0000 7fd543fff700 1 --2- 172.21.15.42:0/476369885 >> v2:172.21.15.42:3300/0 conn(0x7fd544116500 0x7fd544122230 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fd5340033f0 tx=0x7fd534010ee0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:26:52.776 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:52.760+0000 7fd543fff700 1 --2- 172.21.15.42:0/476369885 >> v2:172.21.15.42:3300/0 conn(0x7fd544116500 0x7fd544122230 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fd5340033f0 tx=0x7fd534010ee0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:26:52.777 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:52.760+0000 7fd543fff700 1 --2- 172.21.15.42:0/476369885 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd544116500 0x7fd544122230 secure :-1 s=READY pgs=856 cs=0 l=1 rx=0x7fd5340033f0 tx=0x7fd534010ee0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:26:52.777 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:52.760+0000 7fd543fff700 1 --2- 172.21.15.42:0/476369885 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd544116500 0x7fd544122230 secure :-1 s=THROTTLE_DONE pgs=856 cs=0 l=1 rx=0x7fd5340033f0 tx=0x7fd534010ee0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:26:52.777 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:52.760+0000 7fd540ff9700 1 -- 172.21.15.42:0/476369885 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd534012b40 con 0x7fd544116500 2019-12-05T05:26:52.777 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:52.760+0000 7fd543fff700 1 --2- 172.21.15.42:0/476369885 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd544116500 0x7fd544122230 secure :-1 s=THROTTLE_DONE pgs=856 cs=0 l=1 rx=0x7fd5340033f0 tx=0x7fd534010ee0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:26:52.777 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:52.760+0000 7fd543fff700 1 --2- 172.21.15.42:0/476369885 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd544116500 0x7fd544122230 secure :-1 s=THROTTLE_DONE pgs=856 cs=0 l=1 rx=0x7fd5340033f0 tx=0x7fd534010ee0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:26:52.778 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:52.760+0000 7fd540ff9700 1 -- 172.21.15.42:0/476369885 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd53400e160 con 0x7fd544116500 2019-12-05T05:26:52.778 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:52.760+0000 7fd549ecf700 1 -- 172.21.15.42:0/476369885 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd54408f370 con 0x7fd544116500 2019-12-05T05:26:52.778 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:52.760+0000 7fd540ff9700 1 -- 172.21.15.42:0/476369885 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd534005ab0 con 0x7fd544116500 2019-12-05T05:26:52.778 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:52.760+0000 7fd549ecf700 1 -- 172.21.15.42:0/476369885 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd54408f920 con 0x7fd544116500 2019-12-05T05:26:52.779 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:52.764+0000 7fd543fff700 1 --2- 172.21.15.42:0/476369885 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd544116500 0x7fd544122230 secure :-1 s=THROTTLE_DONE pgs=856 cs=0 l=1 rx=0x7fd5340033f0 tx=0x7fd534010ee0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:26:52.779 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:52.764+0000 7fd540ff9700 1 -- 172.21.15.42:0/476369885 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fd534007180 con 0x7fd544116500 2019-12-05T05:26:52.779 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:52.764+0000 7fd543fff700 1 --2- 172.21.15.42:0/476369885 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd544116500 0x7fd544122230 secure :-1 s=THROTTLE_DONE pgs=856 cs=0 l=1 rx=0x7fd5340033f0 tx=0x7fd534010ee0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:26:52.779 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:52.764+0000 7fd540ff9700 1 --2- 172.21.15.42:0/476369885 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd52803ad20 0x7fd52803d230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:26:52.779 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:52.764+0000 7fd540ff9700 1 -- 172.21.15.42:0/476369885 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7321+0+0 (secure 0 0 0) 0x7fd53404afa0 con 0x7fd544116500 2019-12-05T05:26:52.779 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:52.764+0000 7fd5437fe700 1 --2- 172.21.15.42:0/476369885 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd52803ad20 0x7fd52803d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:26:52.780 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:52.764+0000 7fd5437fe700 1 --2- 172.21.15.42:0/476369885 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd52803ad20 0x7fd52803d230 crc :-1 s=READY pgs=378 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:26:52.780 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:52.764+0000 7fd549ecf700 1 -- 172.21.15.42:0/476369885 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd54405d110 con 0x7fd544116500 2019-12-05T05:26:52.781 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:52.772+0000 7fd543fff700 1 --2- 172.21.15.42:0/476369885 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd544116500 0x7fd544122230 secure :-1 s=THROTTLE_DONE pgs=856 cs=0 l=1 rx=0x7fd5340033f0 tx=0x7fd534010ee0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:26:52.781 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:52.772+0000 7fd540ff9700 1 -- 172.21.15.42:0/476369885 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fd534032020 con 0x7fd544116500 2019-12-05T05:26:52.944 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:52.940+0000 7fd549ecf700 1 -- 172.21.15.42:0/476369885 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd54405b8c0 con 0x7fd544116500 2019-12-05T05:26:52.945 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:52.940+0000 7fd543fff700 1 --2- 172.21.15.42:0/476369885 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd544116500 0x7fd544122230 secure :-1 s=THROTTLE_DONE pgs=856 cs=0 l=1 rx=0x7fd5340033f0 tx=0x7fd534010ee0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:26:52.945 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:52.940+0000 7fd540ff9700 1 -- 172.21.15.42:0/476369885 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+2112 (secure 0 0 0) 0x7fd534032280 con 0x7fd544116500 2019-12-05T05:26:52.945 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 108 2019-12-05T05:26:52.946 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:26:52.946 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6546,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/1149478696","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1149478696},{"type":"v1","addr":"172.21.15.40:6833","nonce":1149478696}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":108}],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:25:38.087763+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5240},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5240":{"gid":5240,"name":"a","rank":0,"incarnation":104,"state":"up:active","state_seq":27,"addr":"172.21.15.42:6835/1455108684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1455108684},{"type":"v1","addr":"172.21.15.42:6835","nonce":1455108684}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:26:52.948 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:52.944+0000 7fd549ecf700 1 -- 172.21.15.42:0/476369885 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd52803ad20 msgr2=0x7fd52803d230 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:26:52.948 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:52.944+0000 7fd549ecf700 1 --2- 172.21.15.42:0/476369885 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd52803ad20 0x7fd52803d230 crc :-1 s=READY pgs=378 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:26:52.948 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:52.944+0000 7fd549ecf700 1 -- 172.21.15.42:0/476369885 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd544116500 msgr2=0x7fd544122230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:26:52.948 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:52.944+0000 7fd549ecf700 1 --2- 172.21.15.42:0/476369885 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd544116500 0x7fd544122230 secure :-1 s=READY pgs=856 cs=0 l=1 rx=0x7fd5340033f0 tx=0x7fd534010ee0).stop 2019-12-05T05:26:52.949 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:52.944+0000 7fd549ecf700 1 -- 172.21.15.42:0/476369885 shutdown_connections 2019-12-05T05:26:52.949 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:52.944+0000 7fd549ecf700 1 --2- 172.21.15.42:0/476369885 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd52803ad20 0x7fd52803d230 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:26:52.949 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:52.944+0000 7fd549ecf700 1 --2- 172.21.15.42:0/476369885 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd544116500 0x7fd544122230 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:26:52.949 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:52.944+0000 7fd549ecf700 1 --2- 172.21.15.42:0/476369885 >> v2:172.21.15.40:3300/0 conn(0x7fd544123700 0x7fd544117560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:26:52.949 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:52.944+0000 7fd549ecf700 1 -- 172.21.15.42:0/476369885 shutdown_connections 2019-12-05T05:26:52.949 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:52.944+0000 7fd549ecf700 1 -- 172.21.15.42:0/476369885 wait complete. 2019-12-05T05:26:52.949 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:52.944+0000 7fd549ecf700 1 -- 172.21.15.42:0/476369885 >> 172.21.15.42:0/476369885 conn(0x7fd5440a9640 msgr2=0x7fd54409f620 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:26:52.964 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T05:26:54.964 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:26:55.070 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:55.060+0000 7fce59440700 1 Processor -- start 2019-12-05T05:26:55.071 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:55.060+0000 7fce59440700 1 -- start start 2019-12-05T05:26:55.071 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:55.060+0000 7fce59440700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fce540a7740 0x7fce5408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:26:55.071 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:55.060+0000 7fce59440700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fce5407e720 0x7fce5407eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:26:55.072 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:55.060+0000 7fce59440700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fce5409f2e0 0x7fce5409f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:26:55.072 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:55.060+0000 7fce59440700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce540a0b00 con 0x7fce540a7740 2019-12-05T05:26:55.072 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:55.060+0000 7fce59440700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce5408e5b0 con 0x7fce5409f2e0 2019-12-05T05:26:55.072 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:55.060+0000 7fce59440700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce5408e850 con 0x7fce5407e720 2019-12-05T05:26:55.073 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:55.060+0000 7fce53fff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fce5409f2e0 0x7fce5409f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:26:55.073 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:55.060+0000 7fce52ffd700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fce5407e720 0x7fce5407eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:26:55.073 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:55.060+0000 7fce53fff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fce5409f2e0 0x7fce5409f710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:33874/0 (socket says 172.21.15.42:33874) 2019-12-05T05:26:55.073 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:55.060+0000 7fce537fe700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fce540a7740 0x7fce5408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:26:55.073 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:55.060+0000 7fce53fff700 1 -- 172.21.15.42:0/2458849631 learned_addr learned my addr 172.21.15.42:0/2458849631 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:26:55.074 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:55.060+0000 7fce53fff700 1 -- 172.21.15.42:0/2458849631 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fce5407e720 msgr2=0x7fce5407eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:26:55.074 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:55.060+0000 7fce53fff700 1 --2- 172.21.15.42:0/2458849631 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fce5407e720 0x7fce5407eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:26:55.074 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:55.060+0000 7fce53fff700 1 -- 172.21.15.42:0/2458849631 >> v2:172.21.15.40:3300/0 conn(0x7fce540a7740 msgr2=0x7fce5408df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:26:55.074 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:55.060+0000 7fce53fff700 1 --2- 172.21.15.42:0/2458849631 >> v2:172.21.15.40:3300/0 conn(0x7fce540a7740 0x7fce5408df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:26:55.074 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:55.060+0000 7fce53fff700 1 -- 172.21.15.42:0/2458849631 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce5407f0a0 con 0x7fce5409f2e0 2019-12-05T05:26:55.074 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:55.060+0000 7fce53fff700 1 --2- 172.21.15.42:0/2458849631 >> v2:172.21.15.42:3300/0 conn(0x7fce5409f2e0 0x7fce5409f710 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fce5408e770 tx=0x7fce4c010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:26:55.074 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:55.060+0000 7fce53fff700 1 --2- 172.21.15.42:0/2458849631 >> v2:172.21.15.42:3300/0 conn(0x7fce5409f2e0 0x7fce5409f710 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fce5408e770 tx=0x7fce4c010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:26:55.075 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:55.060+0000 7fce53fff700 1 --2- 172.21.15.42:0/2458849631 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fce5409f2e0 0x7fce5409f710 secure :-1 s=READY pgs=857 cs=0 l=1 rx=0x7fce5408e770 tx=0x7fce4c010130).ready entity=mon.1 client_cookie=ea0f8a2e9c377f2a server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:26:55.075 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:55.064+0000 7fce53fff700 1 --2- 172.21.15.42:0/2458849631 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fce5409f2e0 0x7fce5409f710 secure :-1 s=THROTTLE_DONE pgs=857 cs=0 l=1 rx=0x7fce5408e770 tx=0x7fce4c010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:26:55.075 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:55.064+0000 7fce53fff700 1 --2- 172.21.15.42:0/2458849631 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fce5409f2e0 0x7fce5409f710 secure :-1 s=THROTTLE_DONE pgs=857 cs=0 l=1 rx=0x7fce5408e770 tx=0x7fce4c010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:26:55.075 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:55.064+0000 7fce53fff700 1 --2- 172.21.15.42:0/2458849631 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fce5409f2e0 0x7fce5409f710 secure :-1 s=THROTTLE_DONE pgs=857 cs=0 l=1 rx=0x7fce5408e770 tx=0x7fce4c010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:26:55.075 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:55.064+0000 7fce527fc700 1 -- 172.21.15.42:0/2458849631 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fce4c010de0 con 0x7fce5409f2e0 2019-12-05T05:26:55.075 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:55.064+0000 7fce527fc700 1 -- 172.21.15.42:0/2458849631 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce4c010a00 con 0x7fce5409f2e0 2019-12-05T05:26:55.075 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:55.064+0000 7fce527fc700 1 -- 172.21.15.42:0/2458849631 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fce4c0114a0 con 0x7fce5409f2e0 2019-12-05T05:26:55.076 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:55.064+0000 7fce59440700 1 -- 172.21.15.42:0/2458849631 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fce5409f2e0 msgr2=0x7fce5409f710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:26:55.076 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:55.064+0000 7fce59440700 1 --2- 172.21.15.42:0/2458849631 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fce5409f2e0 0x7fce5409f710 secure :-1 s=READY pgs=857 cs=0 l=1 rx=0x7fce5408e770 tx=0x7fce4c010130).stop 2019-12-05T05:26:55.076 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:55.064+0000 7fce59440700 1 -- 172.21.15.42:0/2458849631 shutdown_connections 2019-12-05T05:26:55.076 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:55.064+0000 7fce59440700 1 --2- 172.21.15.42:0/2458849631 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fce5409f2e0 0x7fce5409f710 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:26:55.076 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:55.064+0000 7fce59440700 1 --2- 172.21.15.42:0/2458849631 >> v2:172.21.15.40:3300/0 conn(0x7fce540a7740 0x7fce5408df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:26:55.076 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:55.064+0000 7fce59440700 1 --2- 172.21.15.42:0/2458849631 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fce5407e720 0x7fce5407eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:26:55.076 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:55.064+0000 7fce59440700 1 -- 172.21.15.42:0/2458849631 shutdown_connections 2019-12-05T05:26:55.077 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:55.064+0000 7fce59440700 1 -- 172.21.15.42:0/2458849631 wait complete. 2019-12-05T05:26:55.077 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:55.064+0000 7fce59440700 1 -- 172.21.15.42:0/2458849631 >> 172.21.15.42:0/2458849631 conn(0x7fce540aabe0 msgr2=0x7fce540a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:26:55.081 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:55.068+0000 7fce59440700 1 Processor -- start 2019-12-05T05:26:55.081 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:55.068+0000 7fce59440700 1 -- start start 2019-12-05T05:26:55.081 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:55.068+0000 7fce59440700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fce540a6a50 0x7fce54117cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:26:55.081 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:55.068+0000 7fce59440700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce5411b470 con 0x7fce5412df20 2019-12-05T05:26:55.081 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:55.068+0000 7fce59440700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce5412db40 con 0x7fce540a9640 2019-12-05T05:26:55.081 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:55.068+0000 7fce59440700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce540a7080 con 0x7fce540a6a50 2019-12-05T05:26:55.081 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:55.068+0000 7fce52ffd700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fce540a9640 0x7fce5411ac70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:33918/0 (socket says 172.21.15.42:33918) 2019-12-05T05:26:55.082 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:55.068+0000 7fce52ffd700 1 -- 172.21.15.42:0/2945609492 learned_addr learned my addr 172.21.15.42:0/2945609492 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:26:55.082 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:55.068+0000 7fce537fe700 1 --2- 172.21.15.42:0/2945609492 >> v2:172.21.15.40:3300/0 conn(0x7fce540a6a50 0x7fce54117cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:26:55.082 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:55.068+0000 7fce3bfff700 1 -- 172.21.15.42:0/2945609492 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 826299153 0 0) 0x7fce5412db40 con 0x7fce540a9640 2019-12-05T05:26:55.082 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:55.068+0000 7fce3bfff700 1 -- 172.21.15.42:0/2945609492 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce340030b0 con 0x7fce540a9640 2019-12-05T05:26:55.082 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:55.068+0000 7fce3bfff700 1 -- 172.21.15.42:0/2945609492 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1731970460 0 0) 0x7fce5411b470 con 0x7fce5412df20 2019-12-05T05:26:55.082 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:55.068+0000 7fce537fe700 1 -- 172.21.15.42:0/2945609492 >> v1:172.21.15.42:6789/0 conn(0x7fce540a9640 legacy=0x7fce5411ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:26:55.082 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:55.068+0000 7fce537fe700 1 -- 172.21.15.42:0/2945609492 >> v1:172.21.15.40:6789/0 conn(0x7fce5412df20 legacy=0x7fce5412e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:26:55.083 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:55.068+0000 7fce537fe700 1 -- 172.21.15.42:0/2945609492 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce540a9a70 con 0x7fce540a6a50 2019-12-05T05:26:55.083 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:55.068+0000 7fce537fe700 1 --2- 172.21.15.42:0/2945609492 >> v2:172.21.15.40:3300/0 conn(0x7fce540a6a50 0x7fce54117cc0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fce44002f30 tx=0x7fce440109b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:26:55.083 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:55.068+0000 7fce537fe700 1 --2- 172.21.15.42:0/2945609492 >> v2:172.21.15.40:3300/0 conn(0x7fce540a6a50 0x7fce54117cc0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fce44002f30 tx=0x7fce440109b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:26:55.083 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:55.068+0000 7fce537fe700 1 --2- 172.21.15.42:0/2945609492 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fce540a6a50 0x7fce54117cc0 secure :-1 s=READY pgs=683 cs=0 l=1 rx=0x7fce44002f30 tx=0x7fce440109b0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:26:55.083 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:55.072+0000 7fce537fe700 1 --2- 172.21.15.42:0/2945609492 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fce540a6a50 0x7fce54117cc0 secure :-1 s=THROTTLE_DONE pgs=683 cs=0 l=1 rx=0x7fce44002f30 tx=0x7fce440109b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:26:55.083 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:55.072+0000 7fce3bfff700 1 -- 172.21.15.42:0/2945609492 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fce44005930 con 0x7fce540a6a50 2019-12-05T05:26:55.083 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:55.072+0000 7fce537fe700 1 --2- 172.21.15.42:0/2945609492 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fce540a6a50 0x7fce54117cc0 secure :-1 s=THROTTLE_DONE pgs=683 cs=0 l=1 rx=0x7fce44002f30 tx=0x7fce440109b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:26:55.084 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:55.072+0000 7fce59440700 1 -- 172.21.15.42:0/2945609492 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce54141e10 con 0x7fce540a6a50 2019-12-05T05:26:55.084 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:55.072+0000 7fce3bfff700 1 -- 172.21.15.42:0/2945609492 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce44007160 con 0x7fce540a6a50 2019-12-05T05:26:55.084 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:55.072+0000 7fce537fe700 1 --2- 172.21.15.42:0/2945609492 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fce540a6a50 0x7fce54117cc0 secure :-1 s=THROTTLE_DONE pgs=683 cs=0 l=1 rx=0x7fce44002f30 tx=0x7fce440109b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:26:55.084 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:55.072+0000 7fce3bfff700 1 -- 172.21.15.42:0/2945609492 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fce440060c0 con 0x7fce540a6a50 2019-12-05T05:26:55.084 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:55.072+0000 7fce59440700 1 -- 172.21.15.42:0/2945609492 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce540762f0 con 0x7fce540a6a50 2019-12-05T05:26:55.084 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:55.072+0000 7fce537fe700 1 --2- 172.21.15.42:0/2945609492 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fce540a6a50 0x7fce54117cc0 secure :-1 s=THROTTLE_DONE pgs=683 cs=0 l=1 rx=0x7fce44002f30 tx=0x7fce440109b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:26:55.084 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:55.072+0000 7fce3bfff700 1 -- 172.21.15.42:0/2945609492 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fce440063e0 con 0x7fce540a6a50 2019-12-05T05:26:55.085 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:55.072+0000 7fce537fe700 1 --2- 172.21.15.42:0/2945609492 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fce540a6a50 0x7fce54117cc0 secure :-1 s=THROTTLE_DONE pgs=683 cs=0 l=1 rx=0x7fce44002f30 tx=0x7fce440109b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:26:55.085 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:55.072+0000 7fce3bfff700 1 --2- 172.21.15.42:0/2945609492 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fce3403b1c0 0x7fce3403d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:26:55.085 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:55.072+0000 7fce3bfff700 1 -- 172.21.15.42:0/2945609492 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7321+0+0 (secure 0 0 0) 0x7fce4404adc0 con 0x7fce540a6a50 2019-12-05T05:26:55.085 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:55.072+0000 7fce52ffd700 1 --2- 172.21.15.42:0/2945609492 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fce3403b1c0 0x7fce3403d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:26:55.085 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:55.072+0000 7fce52ffd700 1 --2- 172.21.15.42:0/2945609492 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fce3403b1c0 0x7fce3403d6d0 crc :-1 s=READY pgs=379 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:26:55.085 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:55.076+0000 7fce59440700 1 -- 172.21.15.42:0/2945609492 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce540a6e80 con 0x7fce540a6a50 2019-12-05T05:26:55.088 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:55.084+0000 7fce537fe700 1 --2- 172.21.15.42:0/2945609492 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fce540a6a50 0x7fce54117cc0 secure :-1 s=THROTTLE_DONE pgs=683 cs=0 l=1 rx=0x7fce44002f30 tx=0x7fce440109b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:26:55.088 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:55.084+0000 7fce3bfff700 1 -- 172.21.15.42:0/2945609492 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fce44031e20 con 0x7fce540a6a50 2019-12-05T05:26:55.261 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:55.256+0000 7fce59440700 1 -- 172.21.15.42:0/2945609492 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fce5405d110 con 0x7fce540a6a50 2019-12-05T05:26:55.261 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:55.256+0000 7fce537fe700 1 --2- 172.21.15.42:0/2945609492 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fce540a6a50 0x7fce54117cc0 secure :-1 s=THROTTLE_DONE pgs=683 cs=0 l=1 rx=0x7fce44002f30 tx=0x7fce440109b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:26:55.261 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:55.256+0000 7fce3bfff700 1 -- 172.21.15.42:0/2945609492 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+2112 (secure 0 0 0) 0x7fce44032080 con 0x7fce540a6a50 2019-12-05T05:26:55.264 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 108 2019-12-05T05:26:55.264 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:26:55.264 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6546,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/1149478696","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1149478696},{"type":"v1","addr":"172.21.15.40:6833","nonce":1149478696}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":108}],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:25:38.087763+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5240},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5240":{"gid":5240,"name":"a","rank":0,"incarnation":104,"state":"up:active","state_seq":27,"addr":"172.21.15.42:6835/1455108684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1455108684},{"type":"v1","addr":"172.21.15.42:6835","nonce":1455108684}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:26:55.267 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:55.260+0000 7fce59440700 1 -- 172.21.15.42:0/2945609492 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fce3403b1c0 msgr2=0x7fce3403d6d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:26:55.267 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:55.260+0000 7fce59440700 1 --2- 172.21.15.42:0/2945609492 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fce3403b1c0 0x7fce3403d6d0 crc :-1 s=READY pgs=379 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:26:55.267 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:55.260+0000 7fce59440700 1 -- 172.21.15.42:0/2945609492 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fce540a6a50 msgr2=0x7fce54117cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:26:55.267 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:55.260+0000 7fce59440700 1 --2- 172.21.15.42:0/2945609492 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fce540a6a50 0x7fce54117cc0 secure :-1 s=READY pgs=683 cs=0 l=1 rx=0x7fce44002f30 tx=0x7fce440109b0).stop 2019-12-05T05:26:55.268 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:55.260+0000 7fce59440700 1 -- 172.21.15.42:0/2945609492 shutdown_connections 2019-12-05T05:26:55.268 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:55.260+0000 7fce59440700 1 --2- 172.21.15.42:0/2945609492 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fce3403b1c0 0x7fce3403d6d0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:26:55.268 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:55.260+0000 7fce59440700 1 --2- 172.21.15.42:0/2945609492 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fce540a6a50 0x7fce54117cc0 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:26:55.269 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:55.260+0000 7fce59440700 1 -- 172.21.15.42:0/2945609492 shutdown_connections 2019-12-05T05:26:55.269 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:55.260+0000 7fce59440700 1 -- 172.21.15.42:0/2945609492 wait complete. 2019-12-05T05:26:55.270 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:55.260+0000 7fce59440700 1 -- 172.21.15.42:0/2945609492 >> 172.21.15.42:0/2945609492 conn(0x7fce540a82a0 msgr2=0x7fce5407e4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:26:55.283 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T05:26:57.285 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:26:57.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:57.380+0000 7fe1389ab700 1 Processor -- start 2019-12-05T05:26:57.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:57.380+0000 7fe1389ab700 1 -- start start 2019-12-05T05:26:57.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:57.380+0000 7fe1389ab700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe1340a7740 0x7fe13408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:26:57.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:57.380+0000 7fe1389ab700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe13407e720 0x7fe13407eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:26:57.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:57.380+0000 7fe1389ab700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe13409f2e0 0x7fe13409f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:26:57.422 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:57.380+0000 7fe1389ab700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1340a0b00 con 0x7fe13409f2e0 2019-12-05T05:26:57.422 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:57.380+0000 7fe1389ab700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe13408e5b0 con 0x7fe13407e720 2019-12-05T05:26:57.422 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:57.380+0000 7fe1389ab700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe13408e850 con 0x7fe1340a7740 2019-12-05T05:26:57.422 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:57.380+0000 7fe13259c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe13407e720 0x7fe13407eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:26:57.423 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:57.380+0000 7fe13359e700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe13409f2e0 0x7fe13409f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:26:57.423 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:57.380+0000 7fe13259c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe13407e720 0x7fe13407eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:33890/0 (socket says 172.21.15.42:33890) 2019-12-05T05:26:57.423 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:57.380+0000 7fe13259c700 1 -- 172.21.15.42:0/1346502084 learned_addr learned my addr 172.21.15.42:0/1346502084 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:26:57.423 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:57.380+0000 7fe132d9d700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe1340a7740 0x7fe13408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:26:57.423 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:57.384+0000 7fe13259c700 1 -- 172.21.15.42:0/1346502084 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe1340a7740 msgr2=0x7fe13408df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:26:57.424 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:57.384+0000 7fe13259c700 1 --2- 172.21.15.42:0/1346502084 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe1340a7740 0x7fe13408df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:26:57.424 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:57.384+0000 7fe13259c700 1 -- 172.21.15.42:0/1346502084 >> v2:172.21.15.40:3300/0 conn(0x7fe13409f2e0 msgr2=0x7fe13409f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:26:57.424 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:57.384+0000 7fe13259c700 1 --2- 172.21.15.42:0/1346502084 >> v2:172.21.15.40:3300/0 conn(0x7fe13409f2e0 0x7fe13409f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:26:57.424 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:57.384+0000 7fe13259c700 1 -- 172.21.15.42:0/1346502084 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe13407f0a0 con 0x7fe13407e720 2019-12-05T05:26:57.424 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:57.384+0000 7fe13259c700 1 --2- 172.21.15.42:0/1346502084 >> v2:172.21.15.42:3300/0 conn(0x7fe13407e720 0x7fe13407eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fe13408e770 tx=0x7fe12800fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:26:57.425 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:57.384+0000 7fe13259c700 1 --2- 172.21.15.42:0/1346502084 >> v2:172.21.15.42:3300/0 conn(0x7fe13407e720 0x7fe13407eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fe13408e770 tx=0x7fe12800fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:26:57.425 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:57.384+0000 7fe13259c700 1 --2- 172.21.15.42:0/1346502084 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe13407e720 0x7fe13407eb50 secure :-1 s=READY pgs=859 cs=0 l=1 rx=0x7fe13408e770 tx=0x7fe12800fdb0).ready entity=mon.1 client_cookie=3e27cc36d501eff4 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:26:57.425 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:57.384+0000 7fe13259c700 1 --2- 172.21.15.42:0/1346502084 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe13407e720 0x7fe13407eb50 secure :-1 s=THROTTLE_DONE pgs=859 cs=0 l=1 rx=0x7fe13408e770 tx=0x7fe12800fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:26:57.425 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:57.384+0000 7fe13259c700 1 --2- 172.21.15.42:0/1346502084 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe13407e720 0x7fe13407eb50 secure :-1 s=THROTTLE_DONE pgs=859 cs=0 l=1 rx=0x7fe13408e770 tx=0x7fe12800fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:26:57.425 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:57.384+0000 7fe13259c700 1 --2- 172.21.15.42:0/1346502084 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe13407e720 0x7fe13407eb50 secure :-1 s=THROTTLE_DONE pgs=859 cs=0 l=1 rx=0x7fe13408e770 tx=0x7fe12800fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:26:57.425 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:57.384+0000 7fe131d9b700 1 -- 172.21.15.42:0/1346502084 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe128008c90 con 0x7fe13407e720 2019-12-05T05:26:57.426 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:57.384+0000 7fe131d9b700 1 -- 172.21.15.42:0/1346502084 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1280088b0 con 0x7fe13407e720 2019-12-05T05:26:57.426 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:57.384+0000 7fe131d9b700 1 -- 172.21.15.42:0/1346502084 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe128011c40 con 0x7fe13407e720 2019-12-05T05:26:57.426 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:57.384+0000 7fe1389ab700 1 -- 172.21.15.42:0/1346502084 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe13407e720 msgr2=0x7fe13407eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:26:57.426 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:57.384+0000 7fe1389ab700 1 --2- 172.21.15.42:0/1346502084 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe13407e720 0x7fe13407eb50 secure :-1 s=READY pgs=859 cs=0 l=1 rx=0x7fe13408e770 tx=0x7fe12800fdb0).stop 2019-12-05T05:26:57.426 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:57.384+0000 7fe1389ab700 1 -- 172.21.15.42:0/1346502084 shutdown_connections 2019-12-05T05:26:57.426 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:57.384+0000 7fe1389ab700 1 --2- 172.21.15.42:0/1346502084 >> v2:172.21.15.40:3300/0 conn(0x7fe13409f2e0 0x7fe13409f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:26:57.426 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:57.384+0000 7fe1389ab700 1 --2- 172.21.15.42:0/1346502084 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe1340a7740 0x7fe13408df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:26:57.427 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:57.384+0000 7fe1389ab700 1 --2- 172.21.15.42:0/1346502084 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe13407e720 0x7fe13407eb50 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:26:57.427 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:57.384+0000 7fe1389ab700 1 -- 172.21.15.42:0/1346502084 shutdown_connections 2019-12-05T05:26:57.427 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:57.384+0000 7fe1389ab700 1 -- 172.21.15.42:0/1346502084 wait complete. 2019-12-05T05:26:57.427 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:57.384+0000 7fe1389ab700 1 -- 172.21.15.42:0/1346502084 >> 172.21.15.42:0/1346502084 conn(0x7fe1340aabe0 msgr2=0x7fe1340a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:26:57.427 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:57.388+0000 7fe1389ab700 1 Processor -- start 2019-12-05T05:26:57.427 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:57.388+0000 7fe1389ab700 1 -- start start 2019-12-05T05:26:57.427 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:57.388+0000 7fe1389ab700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe1341161c0 0x7fe134117500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:26:57.428 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:57.388+0000 7fe1389ab700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1340a5a80 con 0x7fe13408eeb0 2019-12-05T05:26:57.428 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:57.388+0000 7fe1389ab700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1340a6ab0 con 0x7fe134140860 2019-12-05T05:26:57.428 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:57.388+0000 7fe1389ab700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1340a6e30 con 0x7fe1341161c0 2019-12-05T05:26:57.428 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:57.388+0000 7fe13359e700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fe134140860 0x7fe1340752e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:33934/0 (socket says 172.21.15.42:33934) 2019-12-05T05:26:57.429 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:57.388+0000 7fe13359e700 1 -- 172.21.15.42:0/3758714285 learned_addr learned my addr 172.21.15.42:0/3758714285 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:26:57.429 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:57.388+0000 7fe132d9d700 1 --2- 172.21.15.42:0/3758714285 >> v2:172.21.15.40:3300/0 conn(0x7fe1341161c0 0x7fe134117500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:26:57.429 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:57.388+0000 7fe1237fe700 1 -- 172.21.15.42:0/3758714285 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 223725913 0 0) 0x7fe1340a6ab0 con 0x7fe134140860 2019-12-05T05:26:57.429 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:57.388+0000 7fe1237fe700 1 -- 172.21.15.42:0/3758714285 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1180030b0 con 0x7fe134140860 2019-12-05T05:26:57.429 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:57.388+0000 7fe132d9d700 1 -- 172.21.15.42:0/3758714285 >> v2:172.21.15.40:3300/0 conn(0x7fe1341161c0 msgr2=0x7fe134117500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2019-12-05T05:26:57.430 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:57.388+0000 7fe132d9d700 1 -- 172.21.15.42:0/3758714285 >> v2:172.21.15.40:3300/0 conn(0x7fe1341161c0 msgr2=0x7fe134117500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2019-12-05T05:26:57.430 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:57.388+0000 7fe132d9d700 1 --2- 172.21.15.42:0/3758714285 >> v2:172.21.15.40:3300/0 conn(0x7fe1341161c0 0x7fe134117500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-12-05T05:26:57.430 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:57.388+0000 7fe1237fe700 1 -- 172.21.15.42:0/3758714285 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1751071964 0 0) 0x7fe1340a5a80 con 0x7fe13408eeb0 2019-12-05T05:26:57.430 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:57.388+0000 7fe1237fe700 1 -- 172.21.15.42:0/3758714285 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1340a6ab0 con 0x7fe13408eeb0 2019-12-05T05:26:57.430 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:57.388+0000 7fe132d9d700 1 --2- 172.21.15.42:0/3758714285 >> v2:172.21.15.40:3300/0 conn(0x7fe1341161c0 0x7fe134117500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rx=0 tx=0)._fault waiting 0.200000 2019-12-05T05:26:57.430 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:57.388+0000 7fe1237fe700 1 -- 172.21.15.42:0/3758714285 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 4225977683 0 0) 0x7fe1180030b0 con 0x7fe134140860 2019-12-05T05:26:57.431 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:57.388+0000 7fe1237fe700 1 -- 172.21.15.42:0/3758714285 --> v1:172.21.15.42:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe1340a5a80 con 0x7fe134140860 2019-12-05T05:26:57.431 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:57.388+0000 7fe1237fe700 1 -- 172.21.15.42:0/3758714285 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7fe12c001810 con 0x7fe134140860 2019-12-05T05:26:57.431 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:57.388+0000 7fe1237fe700 1 -- 172.21.15.42:0/3758714285 >> v2:172.21.15.40:3300/0 conn(0x7fe1341161c0 msgr2=0x7fe134117500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2019-12-05T05:26:57.431 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:57.388+0000 7fe1237fe700 1 --2- 172.21.15.42:0/3758714285 >> v2:172.21.15.40:3300/0 conn(0x7fe1341161c0 0x7fe134117500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:26:57.431 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:57.388+0000 7fe1237fe700 1 -- 172.21.15.42:0/3758714285 >> v1:172.21.15.42:6789/0 conn(0x7fe134140860 legacy=0x7fe1340752e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:26:57.431 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:57.388+0000 7fe1237fe700 1 -- 172.21.15.42:0/3758714285 >> v1:172.21.15.40:6789/0 conn(0x7fe13408eeb0 legacy=0x7fe1340a5dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:26:57.431 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:57.388+0000 7fe1237fe700 1 --2- 172.21.15.42:0/3758714285 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe118003ab0 0x7fe118004420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:26:57.432 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:57.388+0000 7fe1237fe700 1 --2- 172.21.15.42:0/3758714285 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe118004970 0x7fe118004da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:26:57.432 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:57.388+0000 7fe1237fe700 1 --2- 172.21.15.42:0/3758714285 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe118009660 0x7fe11800baa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:26:57.432 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:57.388+0000 7fe1237fe700 1 -- 172.21.15.42:0/3758714285 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1340a6e30 con 0x7fe118004970 2019-12-05T05:26:57.432 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:57.388+0000 7fe1237fe700 1 -- 172.21.15.42:0/3758714285 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe11800bff0 con 0x7fe118003ab0 2019-12-05T05:26:57.432 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:57.388+0000 7fe1237fe700 1 -- 172.21.15.42:0/3758714285 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe11800c1b0 con 0x7fe118009660 2019-12-05T05:26:57.432 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:57.388+0000 7fe132d9d700 1 --2- 172.21.15.42:0/3758714285 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe118003ab0 0x7fe118004420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:26:57.433 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:57.388+0000 7fe13259c700 1 --2- 172.21.15.42:0/3758714285 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe118004970 0x7fe118004da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:26:57.433 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:57.388+0000 7fe13359e700 1 --2- 172.21.15.42:0/3758714285 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe118009660 0x7fe11800baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:26:57.433 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:57.388+0000 7fe132d9d700 1 -- 172.21.15.42:0/3758714285 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe118009660 msgr2=0x7fe11800baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:26:57.433 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:57.388+0000 7fe132d9d700 1 --2- 172.21.15.42:0/3758714285 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe118009660 0x7fe11800baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:26:57.433 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:57.388+0000 7fe132d9d700 1 -- 172.21.15.42:0/3758714285 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe118004970 msgr2=0x7fe118004da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:26:57.433 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:57.388+0000 7fe132d9d700 1 --2- 172.21.15.42:0/3758714285 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe118004970 0x7fe118004da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:26:57.433 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:57.388+0000 7fe132d9d700 1 -- 172.21.15.42:0/3758714285 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe134117a50 con 0x7fe118003ab0 2019-12-05T05:26:57.434 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:57.388+0000 7fe13359e700 1 -- 172.21.15.42:0/3758714285 reap_dead start 2019-12-05T05:26:57.434 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:57.388+0000 7fe1389ab700 1 -- 172.21.15.42:0/3758714285 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe134140c90 con 0x7fe118003ab0 2019-12-05T05:26:57.434 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:57.388+0000 7fe132d9d700 1 --2- 172.21.15.42:0/3758714285 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe118003ab0 0x7fe118004420 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fe1180043b0 tx=0x7fe124007420).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:26:57.435 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:57.388+0000 7fe13259c700 1 --2- 172.21.15.42:0/3758714285 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe118004970 0x7fe118004da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:26:57.435 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:57.388+0000 7fe1389ab700 1 -- 172.21.15.42:0/3758714285 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1341161c0 con 0x7fe118003ab0 2019-12-05T05:26:57.435 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:57.392+0000 7fe132d9d700 1 --2- 172.21.15.42:0/3758714285 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe118003ab0 0x7fe118004420 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fe1180043b0 tx=0x7fe124007420).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:26:57.435 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:57.392+0000 7fe132d9d700 1 --2- 172.21.15.42:0/3758714285 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe118003ab0 0x7fe118004420 secure :-1 s=READY pgs=861 cs=0 l=1 rx=0x7fe1180043b0 tx=0x7fe124007420).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:26:57.435 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:57.392+0000 7fe132d9d700 1 --2- 172.21.15.42:0/3758714285 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe118003ab0 0x7fe118004420 secure :-1 s=THROTTLE_DONE pgs=861 cs=0 l=1 rx=0x7fe1180043b0 tx=0x7fe124007420).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:26:57.435 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:57.392+0000 7fe132d9d700 1 --2- 172.21.15.42:0/3758714285 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe118003ab0 0x7fe118004420 secure :-1 s=THROTTLE_DONE pgs=861 cs=0 l=1 rx=0x7fe1180043b0 tx=0x7fe124007420).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:26:57.435 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:57.392+0000 7fe132d9d700 1 --2- 172.21.15.42:0/3758714285 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe118003ab0 0x7fe118004420 secure :-1 s=THROTTLE_DONE pgs=861 cs=0 l=1 rx=0x7fe1180043b0 tx=0x7fe124007420).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:26:57.436 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:57.392+0000 7fe1237fe700 1 -- 172.21.15.42:0/3758714285 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe124011b30 con 0x7fe118003ab0 2019-12-05T05:26:57.436 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:57.392+0000 7fe1237fe700 1 -- 172.21.15.42:0/3758714285 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe124011750 con 0x7fe118003ab0 2019-12-05T05:26:57.436 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:57.392+0000 7fe1237fe700 1 -- 172.21.15.42:0/3758714285 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe12400c370 con 0x7fe118003ab0 2019-12-05T05:26:57.436 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:57.392+0000 7fe132d9d700 1 --2- 172.21.15.42:0/3758714285 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe118003ab0 0x7fe118004420 secure :-1 s=THROTTLE_DONE pgs=861 cs=0 l=1 rx=0x7fe1180043b0 tx=0x7fe124007420).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:26:57.436 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:57.392+0000 7fe1237fe700 1 -- 172.21.15.42:0/3758714285 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fe12400c570 con 0x7fe118003ab0 2019-12-05T05:26:57.436 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:57.392+0000 7fe132d9d700 1 --2- 172.21.15.42:0/3758714285 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe118003ab0 0x7fe118004420 secure :-1 s=THROTTLE_DONE pgs=861 cs=0 l=1 rx=0x7fe1180043b0 tx=0x7fe124007420).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:26:57.437 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:57.392+0000 7fe1237fe700 1 --2- 172.21.15.42:0/3758714285 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe11803ec60 0x7fe118041170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:26:57.437 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:57.392+0000 7fe1237fe700 1 -- 172.21.15.42:0/3758714285 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7321+0+0 (secure 0 0 0) 0x7fe12404ada0 con 0x7fe118003ab0 2019-12-05T05:26:57.437 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:57.392+0000 7fe13259c700 1 --2- 172.21.15.42:0/3758714285 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe11803ec60 0x7fe118041170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:26:57.437 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:57.392+0000 7fe13259c700 1 --2- 172.21.15.42:0/3758714285 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe11803ec60 0x7fe118041170 crc :-1 s=READY pgs=380 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:26:57.437 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:57.396+0000 7fe1389ab700 1 -- 172.21.15.42:0/3758714285 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe10c003200 con 0x7fe118003ab0 2019-12-05T05:26:57.437 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:57.400+0000 7fe132d9d700 1 --2- 172.21.15.42:0/3758714285 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe118003ab0 0x7fe118004420 secure :-1 s=THROTTLE_DONE pgs=861 cs=0 l=1 rx=0x7fe1180043b0 tx=0x7fe124007420).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:26:57.438 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:57.400+0000 7fe1237fe700 1 -- 172.21.15.42:0/3758714285 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fe124031e20 con 0x7fe118003ab0 2019-12-05T05:26:57.569 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:57.564+0000 7fe1389ab700 1 -- 172.21.15.42:0/3758714285 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe10c003630 con 0x7fe118003ab0 2019-12-05T05:26:57.570 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:57.564+0000 7fe132d9d700 1 --2- 172.21.15.42:0/3758714285 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe118003ab0 0x7fe118004420 secure :-1 s=THROTTLE_DONE pgs=861 cs=0 l=1 rx=0x7fe1180043b0 tx=0x7fe124007420).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:26:57.571 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:57.564+0000 7fe1237fe700 1 -- 172.21.15.42:0/3758714285 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+2112 (secure 0 0 0) 0x7fe124032080 con 0x7fe118003ab0 2019-12-05T05:26:57.571 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 108 2019-12-05T05:26:57.571 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:26:57.571 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6546,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/1149478696","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1149478696},{"type":"v1","addr":"172.21.15.40:6833","nonce":1149478696}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":108}],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:25:38.087763+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5240},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5240":{"gid":5240,"name":"a","rank":0,"incarnation":104,"state":"up:active","state_seq":27,"addr":"172.21.15.42:6835/1455108684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1455108684},{"type":"v1","addr":"172.21.15.42:6835","nonce":1455108684}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:26:57.574 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:57.568+0000 7fe1389ab700 1 -- 172.21.15.42:0/3758714285 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe11803ec60 msgr2=0x7fe118041170 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:26:57.574 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:57.568+0000 7fe1389ab700 1 --2- 172.21.15.42:0/3758714285 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe11803ec60 0x7fe118041170 crc :-1 s=READY pgs=380 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:26:57.574 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:57.568+0000 7fe1389ab700 1 -- 172.21.15.42:0/3758714285 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe118003ab0 msgr2=0x7fe118004420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:26:57.574 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:57.568+0000 7fe1389ab700 1 --2- 172.21.15.42:0/3758714285 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe118003ab0 0x7fe118004420 secure :-1 s=READY pgs=861 cs=0 l=1 rx=0x7fe1180043b0 tx=0x7fe124007420).stop 2019-12-05T05:26:57.575 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:57.568+0000 7fe1389ab700 1 -- 172.21.15.42:0/3758714285 shutdown_connections 2019-12-05T05:26:57.575 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:57.568+0000 7fe1389ab700 1 --2- 172.21.15.42:0/3758714285 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe11803ec60 0x7fe118041170 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:26:57.575 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:57.568+0000 7fe1389ab700 1 --2- 172.21.15.42:0/3758714285 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe118003ab0 0x7fe118004420 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:26:57.576 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:57.568+0000 7fe1389ab700 1 -- 172.21.15.42:0/3758714285 shutdown_connections 2019-12-05T05:26:57.576 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:57.568+0000 7fe1389ab700 1 -- 172.21.15.42:0/3758714285 wait complete. 2019-12-05T05:26:57.576 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:57.568+0000 7fe1389ab700 1 -- 172.21.15.42:0/3758714285 >> 172.21.15.42:0/3758714285 conn(0x7fe1340a9790 msgr2=0x7fe13409f380 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:26:57.582 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T05:26:59.583 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:26:59.695 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:59.684+0000 7f9b6eb96700 1 Processor -- start 2019-12-05T05:26:59.695 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:59.684+0000 7f9b6eb96700 1 -- start start 2019-12-05T05:26:59.695 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:59.688+0000 7f9b6eb96700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f9b680a7740 0x7f9b6808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:26:59.695 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:59.688+0000 7f9b6eb96700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9b6807e720 0x7f9b6807eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:26:59.695 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:59.688+0000 7f9b6eb96700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f9b6809f2e0 0x7f9b6809f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:26:59.696 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:59.688+0000 7f9b6eb96700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b680a0b00 con 0x7f9b6809f2e0 2019-12-05T05:26:59.696 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:59.688+0000 7f9b6eb96700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b6808e5b0 con 0x7f9b680a7740 2019-12-05T05:26:59.696 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:59.688+0000 7f9b6eb96700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b6808e850 con 0x7f9b6807e720 2019-12-05T05:26:59.696 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:59.688+0000 7f9b6d133700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f9b680a7740 0x7f9b6808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:26:59.696 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:59.688+0000 7f9b6d133700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f9b680a7740 0x7f9b6808df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:33908/0 (socket says 172.21.15.42:33908) 2019-12-05T05:26:59.696 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:59.688+0000 7f9b6d133700 1 -- 172.21.15.42:0/996733721 learned_addr learned my addr 172.21.15.42:0/996733721 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:26:59.697 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:59.688+0000 7f9b6d934700 1 --2- 172.21.15.42:0/996733721 >> v2:172.21.15.40:3300/0 conn(0x7f9b6809f2e0 0x7f9b6809f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:26:59.697 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:59.688+0000 7f9b6c932700 1 --2- 172.21.15.42:0/996733721 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9b6807e720 0x7f9b6807eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:26:59.697 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:59.688+0000 7f9b6d133700 1 -- 172.21.15.42:0/996733721 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9b6807e720 msgr2=0x7f9b6807eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:26:59.697 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:59.688+0000 7f9b6d133700 1 --2- 172.21.15.42:0/996733721 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9b6807e720 0x7f9b6807eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:26:59.697 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:59.688+0000 7f9b6d133700 1 -- 172.21.15.42:0/996733721 >> v2:172.21.15.40:3300/0 conn(0x7f9b6809f2e0 msgr2=0x7f9b6809f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:26:59.697 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:59.688+0000 7f9b6d133700 1 --2- 172.21.15.42:0/996733721 >> v2:172.21.15.40:3300/0 conn(0x7f9b6809f2e0 0x7f9b6809f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:26:59.697 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:59.688+0000 7f9b6d133700 1 -- 172.21.15.42:0/996733721 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b6807f0a0 con 0x7f9b680a7740 2019-12-05T05:26:59.698 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:59.688+0000 7f9b6d133700 1 --2- 172.21.15.42:0/996733721 >> v2:172.21.15.42:3300/0 conn(0x7f9b680a7740 0x7f9b6808df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f9b6808e770 tx=0x7f9b5c00fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:26:59.698 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:59.688+0000 7f9b6d133700 1 --2- 172.21.15.42:0/996733721 >> v2:172.21.15.42:3300/0 conn(0x7f9b680a7740 0x7f9b6808df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f9b6808e770 tx=0x7f9b5c00fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:26:59.699 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:59.688+0000 7f9b6d133700 1 --2- 172.21.15.42:0/996733721 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9b680a7740 0x7f9b6808df80 secure :-1 s=READY pgs=862 cs=0 l=1 rx=0x7f9b6808e770 tx=0x7f9b5c00fdb0).ready entity=mon.1 client_cookie=14d034ab0f23116e server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:26:59.699 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:59.688+0000 7f9b6d133700 1 --2- 172.21.15.42:0/996733721 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9b680a7740 0x7f9b6808df80 secure :-1 s=THROTTLE_DONE pgs=862 cs=0 l=1 rx=0x7f9b6808e770 tx=0x7f9b5c00fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:26:59.699 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:59.688+0000 7f9b6d133700 1 --2- 172.21.15.42:0/996733721 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9b680a7740 0x7f9b6808df80 secure :-1 s=THROTTLE_DONE pgs=862 cs=0 l=1 rx=0x7f9b6808e770 tx=0x7f9b5c00fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:26:59.700 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:59.688+0000 7f9b5bfff700 1 -- 172.21.15.42:0/996733721 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9b5c008c90 con 0x7f9b680a7740 2019-12-05T05:26:59.700 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:59.688+0000 7f9b6d133700 1 --2- 172.21.15.42:0/996733721 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9b680a7740 0x7f9b6808df80 secure :-1 s=THROTTLE_DONE pgs=862 cs=0 l=1 rx=0x7f9b6808e770 tx=0x7f9b5c00fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:26:59.700 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:59.688+0000 7f9b5bfff700 1 -- 172.21.15.42:0/996733721 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b5c0088b0 con 0x7f9b680a7740 2019-12-05T05:26:59.700 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:59.688+0000 7f9b5bfff700 1 -- 172.21.15.42:0/996733721 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9b5c011b50 con 0x7f9b680a7740 2019-12-05T05:26:59.700 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:59.688+0000 7f9b6eb96700 1 -- 172.21.15.42:0/996733721 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9b680a7740 msgr2=0x7f9b6808df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:26:59.700 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:59.688+0000 7f9b6eb96700 1 --2- 172.21.15.42:0/996733721 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9b680a7740 0x7f9b6808df80 secure :-1 s=READY pgs=862 cs=0 l=1 rx=0x7f9b6808e770 tx=0x7f9b5c00fdb0).stop 2019-12-05T05:26:59.700 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:59.688+0000 7f9b6eb96700 1 -- 172.21.15.42:0/996733721 shutdown_connections 2019-12-05T05:26:59.701 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:59.688+0000 7f9b6eb96700 1 --2- 172.21.15.42:0/996733721 >> v2:172.21.15.40:3300/0 conn(0x7f9b6809f2e0 0x7f9b6809f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:26:59.701 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:59.688+0000 7f9b6eb96700 1 --2- 172.21.15.42:0/996733721 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9b680a7740 0x7f9b6808df80 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:26:59.701 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:59.688+0000 7f9b6eb96700 1 --2- 172.21.15.42:0/996733721 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9b6807e720 0x7f9b6807eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:26:59.701 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:59.688+0000 7f9b6eb96700 1 -- 172.21.15.42:0/996733721 shutdown_connections 2019-12-05T05:26:59.701 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:59.688+0000 7f9b6eb96700 1 -- 172.21.15.42:0/996733721 wait complete. 2019-12-05T05:26:59.701 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:59.688+0000 7f9b6eb96700 1 -- 172.21.15.42:0/996733721 >> 172.21.15.42:0/996733721 conn(0x7f9b680aabe0 msgr2=0x7f9b680a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:26:59.705 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:59.692+0000 7f9b6eb96700 1 Processor -- start 2019-12-05T05:26:59.705 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:59.692+0000 7f9b6eb96700 1 -- start start 2019-12-05T05:26:59.705 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:59.692+0000 7f9b6eb96700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9b6811a3a0 0x7f9b68124360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:26:59.705 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:59.692+0000 7f9b6eb96700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f9b681230d0 0x7f9b680762f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:26:59.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:59.692+0000 7f9b6eb96700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b6811b0f0 con 0x7f9b68121550 2019-12-05T05:26:59.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:59.692+0000 7f9b6eb96700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b68123500 con 0x7f9b681230d0 2019-12-05T05:26:59.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:59.692+0000 7f9b6eb96700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b68121980 con 0x7f9b6811a3a0 2019-12-05T05:26:59.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:59.692+0000 7f9b6d934700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f9b681230d0 0x7f9b680762f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:26:59.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:59.692+0000 7f9b6c932700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f9b68121550 0x7f9b681228d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:43392/0 (socket says 172.21.15.42:43392) 2019-12-05T05:26:59.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:59.692+0000 7f9b6c932700 1 -- 172.21.15.42:0/235411678 learned_addr learned my addr 172.21.15.42:0/235411678 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:26:59.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:59.692+0000 7f9b6d133700 1 --2- 172.21.15.42:0/235411678 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9b6811a3a0 0x7f9b68124360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:26:59.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:59.692+0000 7f9b6d934700 1 -- 172.21.15.42:0/235411678 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9b6811a3a0 msgr2=0x7f9b68124360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:26:59.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:59.692+0000 7f9b6d934700 1 --2- 172.21.15.42:0/235411678 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9b6811a3a0 0x7f9b68124360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:26:59.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:59.692+0000 7f9b6d934700 1 -- 172.21.15.42:0/235411678 >> v1:172.21.15.40:6789/0 conn(0x7f9b68121550 legacy=0x7f9b681228d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:26:59.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:59.692+0000 7f9b59ffb700 1 -- 172.21.15.42:0/235411678 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4244967110 0 0) 0x7f9b6811b0f0 con 0x7f9b68121550 2019-12-05T05:26:59.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:59.692+0000 7f9b6d934700 1 -- 172.21.15.42:0/235411678 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b68077850 con 0x7f9b681230d0 2019-12-05T05:26:59.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:59.692+0000 7f9b6d934700 1 --2- 172.21.15.42:0/235411678 >> v2:172.21.15.42:3300/0 conn(0x7f9b681230d0 0x7f9b680762f0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f9b600032b0 tx=0x7f9b60010e40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:26:59.708 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:59.692+0000 7f9b6d934700 1 --2- 172.21.15.42:0/235411678 >> v2:172.21.15.42:3300/0 conn(0x7f9b681230d0 0x7f9b680762f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f9b600032b0 tx=0x7f9b60010e40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:26:59.708 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:59.692+0000 7f9b6d934700 1 --2- 172.21.15.42:0/235411678 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9b681230d0 0x7f9b680762f0 secure :-1 s=READY pgs=863 cs=0 l=1 rx=0x7f9b600032b0 tx=0x7f9b60010e40).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:26:59.708 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:59.692+0000 7f9b6d934700 1 --2- 172.21.15.42:0/235411678 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9b681230d0 0x7f9b680762f0 secure :-1 s=THROTTLE_DONE pgs=863 cs=0 l=1 rx=0x7f9b600032b0 tx=0x7f9b60010e40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:26:59.708 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:59.692+0000 7f9b59ffb700 1 -- 172.21.15.42:0/235411678 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9b60012ae0 con 0x7f9b681230d0 2019-12-05T05:26:59.712 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:59.692+0000 7f9b6d934700 1 --2- 172.21.15.42:0/235411678 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9b681230d0 0x7f9b680762f0 secure :-1 s=THROTTLE_DONE pgs=863 cs=0 l=1 rx=0x7f9b600032b0 tx=0x7f9b60010e40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:26:59.712 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:59.692+0000 7f9b6d934700 1 --2- 172.21.15.42:0/235411678 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9b681230d0 0x7f9b680762f0 secure :-1 s=THROTTLE_DONE pgs=863 cs=0 l=1 rx=0x7f9b600032b0 tx=0x7f9b60010e40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:26:59.712 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:59.692+0000 7f9b59ffb700 1 -- 172.21.15.42:0/235411678 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b6000e160 con 0x7f9b681230d0 2019-12-05T05:26:59.712 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:59.692+0000 7f9b6eb96700 1 -- 172.21.15.42:0/235411678 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b68076840 con 0x7f9b681230d0 2019-12-05T05:26:59.712 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:59.692+0000 7f9b59ffb700 1 -- 172.21.15.42:0/235411678 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9b60005ab0 con 0x7f9b681230d0 2019-12-05T05:26:59.713 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:59.692+0000 7f9b6eb96700 1 -- 172.21.15.42:0/235411678 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b68076d90 con 0x7f9b681230d0 2019-12-05T05:26:59.713 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:59.692+0000 7f9b6d934700 1 --2- 172.21.15.42:0/235411678 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9b681230d0 0x7f9b680762f0 secure :-1 s=THROTTLE_DONE pgs=863 cs=0 l=1 rx=0x7f9b600032b0 tx=0x7f9b60010e40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:26:59.713 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:59.696+0000 7f9b6d934700 1 --2- 172.21.15.42:0/235411678 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9b681230d0 0x7f9b680762f0 secure :-1 s=THROTTLE_DONE pgs=863 cs=0 l=1 rx=0x7f9b600032b0 tx=0x7f9b60010e40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:26:59.713 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:59.696+0000 7f9b59ffb700 1 -- 172.21.15.42:0/235411678 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f9b60007180 con 0x7f9b681230d0 2019-12-05T05:26:59.713 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:59.696+0000 7f9b59ffb700 1 --2- 172.21.15.42:0/235411678 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9b4803ac90 0x7f9b4803d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:26:59.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:59.696+0000 7f9b59ffb700 1 -- 172.21.15.42:0/235411678 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7321+0+0 (secure 0 0 0) 0x7f9b6004af30 con 0x7f9b681230d0 2019-12-05T05:26:59.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:59.696+0000 7f9b6d133700 1 --2- 172.21.15.42:0/235411678 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9b4803ac90 0x7f9b4803d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:26:59.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:59.696+0000 7f9b6d133700 1 --2- 172.21.15.42:0/235411678 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9b4803ac90 0x7f9b4803d1a0 crc :-1 s=READY pgs=381 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:26:59.715 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:59.700+0000 7f9b6eb96700 1 -- 172.21.15.42:0/235411678 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b6805d110 con 0x7f9b681230d0 2019-12-05T05:26:59.715 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:59.708+0000 7f9b6d934700 1 --2- 172.21.15.42:0/235411678 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9b681230d0 0x7f9b680762f0 secure :-1 s=THROTTLE_DONE pgs=863 cs=0 l=1 rx=0x7f9b600032b0 tx=0x7f9b60010e40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:26:59.715 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:59.708+0000 7f9b59ffb700 1 -- 172.21.15.42:0/235411678 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f9b60031f70 con 0x7f9b681230d0 2019-12-05T05:26:59.887 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:59.880+0000 7f9b6eb96700 1 -- 172.21.15.42:0/235411678 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9b6805b8c0 con 0x7f9b681230d0 2019-12-05T05:26:59.888 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:59.884+0000 7f9b6d934700 1 --2- 172.21.15.42:0/235411678 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9b681230d0 0x7f9b680762f0 secure :-1 s=THROTTLE_DONE pgs=863 cs=0 l=1 rx=0x7f9b600032b0 tx=0x7f9b60010e40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:26:59.888 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:59.884+0000 7f9b59ffb700 1 -- 172.21.15.42:0/235411678 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f9b60005dc0 con 0x7f9b681230d0 2019-12-05T05:26:59.890 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:26:59.891 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6546,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/1149478696","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1149478696},{"type":"v1","addr":"172.21.15.40:6833","nonce":1149478696}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":108}],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:25:38.087763+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5240},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5240":{"gid":5240,"name":"a","rank":0,"incarnation":104,"state":"up:active","state_seq":27,"addr":"172.21.15.42:6835/1455108684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1455108684},{"type":"v1","addr":"172.21.15.42:6835","nonce":1455108684}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:26:59.891 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 108 2019-12-05T05:26:59.892 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:59.888+0000 7f9b6eb96700 1 -- 172.21.15.42:0/235411678 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9b4803ac90 msgr2=0x7f9b4803d1a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:26:59.893 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:59.888+0000 7f9b6eb96700 1 --2- 172.21.15.42:0/235411678 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9b4803ac90 0x7f9b4803d1a0 crc :-1 s=READY pgs=381 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:26:59.893 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:59.888+0000 7f9b6eb96700 1 -- 172.21.15.42:0/235411678 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9b681230d0 msgr2=0x7f9b680762f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:26:59.893 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:59.888+0000 7f9b6eb96700 1 --2- 172.21.15.42:0/235411678 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9b681230d0 0x7f9b680762f0 secure :-1 s=READY pgs=863 cs=0 l=1 rx=0x7f9b600032b0 tx=0x7f9b60010e40).stop 2019-12-05T05:26:59.893 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:59.888+0000 7f9b6eb96700 1 -- 172.21.15.42:0/235411678 shutdown_connections 2019-12-05T05:26:59.893 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:59.888+0000 7f9b6eb96700 1 --2- 172.21.15.42:0/235411678 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9b4803ac90 0x7f9b4803d1a0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:26:59.893 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:59.888+0000 7f9b6eb96700 1 --2- 172.21.15.42:0/235411678 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9b681230d0 0x7f9b680762f0 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:26:59.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:59.888+0000 7f9b6eb96700 1 --2- 172.21.15.42:0/235411678 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9b6811a3a0 0x7f9b68124360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:26:59.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:59.888+0000 7f9b6eb96700 1 -- 172.21.15.42:0/235411678 shutdown_connections 2019-12-05T05:26:59.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:59.888+0000 7f9b6eb96700 1 -- 172.21.15.42:0/235411678 wait complete. 2019-12-05T05:26:59.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:26:59.888+0000 7f9b6eb96700 1 -- 172.21.15.42:0/235411678 >> 172.21.15.42:0/235411678 conn(0x7f9b680a69e0 msgr2=0x7f9b6809f3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:26:59.903 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T05:27:01.905 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:27:02.021 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:02.008+0000 7f6d8000d700 1 Processor -- start 2019-12-05T05:27:02.021 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:02.008+0000 7f6d8000d700 1 -- start start 2019-12-05T05:27:02.021 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:02.012+0000 7f6d8000d700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f6d7808df70 0x7f6d7809f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:27:02.022 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:02.012+0000 7f6d8000d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6d7809f820 0x7f6d7814efd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:27:02.022 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:02.012+0000 7f6d8000d700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d7809fc50 con 0x7f6d780a7730 2019-12-05T05:27:02.022 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:02.012+0000 7f6d8000d700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d780a0af0 con 0x7f6d7808df70 2019-12-05T05:27:02.022 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:02.012+0000 7f6d8000d700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d7808e5a0 con 0x7f6d7809f820 2019-12-05T05:27:02.022 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:02.012+0000 7f6d7e5aa700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f6d780a7730 0x7f6d7807e710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:33960/0 (socket says 172.21.15.42:33960) 2019-12-05T05:27:02.022 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:02.012+0000 7f6d7edab700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6d7809f820 0x7f6d7814efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:27:02.022 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:02.012+0000 7f6d7e5aa700 1 -- 172.21.15.42:0/266706873 learned_addr learned my addr 172.21.15.42:0/266706873 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:27:02.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:02.012+0000 7f6d7dda9700 1 --2- 172.21.15.42:0/266706873 >> v2:172.21.15.40:3300/0 conn(0x7f6d7808df70 0x7f6d7809f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:27:02.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:02.012+0000 7f6d7edab700 1 -- 172.21.15.42:0/266706873 >> v2:172.21.15.40:3300/0 conn(0x7f6d7808df70 msgr2=0x7f6d7809f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:27:02.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:02.012+0000 7f6d7edab700 1 --2- 172.21.15.42:0/266706873 >> v2:172.21.15.40:3300/0 conn(0x7f6d7808df70 0x7f6d7809f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:27:02.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:02.012+0000 7f6d7edab700 1 -- 172.21.15.42:0/266706873 >> v1:172.21.15.42:6789/0 conn(0x7f6d780a7730 legacy=0x7f6d7807e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:27:02.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:02.012+0000 7f6d7edab700 1 -- 172.21.15.42:0/266706873 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d780a0700 con 0x7f6d7809f820 2019-12-05T05:27:02.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:02.012+0000 7f6d7d5a8700 1 -- 172.21.15.42:0/266706873 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1525828441 0 0) 0x7f6d7809fc50 con 0x7f6d780a7730 2019-12-05T05:27:02.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:02.012+0000 7f6d7edab700 1 --2- 172.21.15.42:0/266706873 >> v2:172.21.15.42:3300/0 conn(0x7f6d7809f820 0x7f6d7814efd0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f6d700032b0 tx=0x7f6d7000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:02.024 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:02.012+0000 7f6d7edab700 1 --2- 172.21.15.42:0/266706873 >> v2:172.21.15.42:3300/0 conn(0x7f6d7809f820 0x7f6d7814efd0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f6d700032b0 tx=0x7f6d7000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:02.024 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:02.012+0000 7f6d7edab700 1 --2- 172.21.15.42:0/266706873 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6d7809f820 0x7f6d7814efd0 secure :-1 s=READY pgs=866 cs=0 l=1 rx=0x7f6d700032b0 tx=0x7f6d7000fff0).ready entity=mon.1 client_cookie=8084fda2b43dc7d8 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:27:02.024 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:02.012+0000 7f6d7edab700 1 --2- 172.21.15.42:0/266706873 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6d7809f820 0x7f6d7814efd0 secure :-1 s=THROTTLE_DONE pgs=866 cs=0 l=1 rx=0x7f6d700032b0 tx=0x7f6d7000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:02.024 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:02.012+0000 7f6d7d5a8700 1 -- 172.21.15.42:0/266706873 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6d700108c0 con 0x7f6d7809f820 2019-12-05T05:27:02.024 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:02.012+0000 7f6d7edab700 1 --2- 172.21.15.42:0/266706873 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6d7809f820 0x7f6d7814efd0 secure :-1 s=THROTTLE_DONE pgs=866 cs=0 l=1 rx=0x7f6d700032b0 tx=0x7f6d7000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:02.024 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:02.012+0000 7f6d7edab700 1 --2- 172.21.15.42:0/266706873 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6d7809f820 0x7f6d7814efd0 secure :-1 s=THROTTLE_DONE pgs=866 cs=0 l=1 rx=0x7f6d700032b0 tx=0x7f6d7000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:02.024 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:02.012+0000 7f6d7d5a8700 1 -- 172.21.15.42:0/266706873 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d70008b50 con 0x7f6d7809f820 2019-12-05T05:27:02.025 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:02.012+0000 7f6d7d5a8700 1 -- 172.21.15.42:0/266706873 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6d700110c0 con 0x7f6d7809f820 2019-12-05T05:27:02.025 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:02.012+0000 7f6d8000d700 1 -- 172.21.15.42:0/266706873 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6d7809f820 msgr2=0x7f6d7814efd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:27:02.025 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:02.012+0000 7f6d8000d700 1 --2- 172.21.15.42:0/266706873 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6d7809f820 0x7f6d7814efd0 secure :-1 s=READY pgs=866 cs=0 l=1 rx=0x7f6d700032b0 tx=0x7f6d7000fff0).stop 2019-12-05T05:27:02.025 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:02.016+0000 7f6d8000d700 1 -- 172.21.15.42:0/266706873 shutdown_connections 2019-12-05T05:27:02.025 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:02.016+0000 7f6d8000d700 1 --2- 172.21.15.42:0/266706873 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6d7809f820 0x7f6d7814efd0 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:27:02.025 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:02.016+0000 7f6d8000d700 1 --2- 172.21.15.42:0/266706873 >> v2:172.21.15.40:3300/0 conn(0x7f6d7808df70 0x7f6d7809f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:27:02.025 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:02.016+0000 7f6d8000d700 1 -- 172.21.15.42:0/266706873 shutdown_connections 2019-12-05T05:27:02.026 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:02.016+0000 7f6d8000d700 1 -- 172.21.15.42:0/266706873 wait complete. 2019-12-05T05:27:02.043 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:02.016+0000 7f6d8000d700 1 -- 172.21.15.42:0/266706873 >> 172.21.15.42:0/266706873 conn(0x7f6d780aabd0 msgr2=0x7f6d780a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:27:02.043 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:02.016+0000 7f6d8000d700 1 Processor -- start 2019-12-05T05:27:02.043 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:02.016+0000 7f6d8000d700 1 -- start start 2019-12-05T05:27:02.043 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:02.020+0000 7f6d8000d700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6d7811a300 0x7f6d78121cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:27:02.043 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:02.020+0000 7f6d8000d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6d781367b0 0x7f6d78138bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:27:02.043 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:02.020+0000 7f6d8000d700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d7811a730 con 0x7f6d7812dec0 2019-12-05T05:27:02.044 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:02.020+0000 7f6d8000d700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d7811dd00 con 0x7f6d781367b0 2019-12-05T05:27:02.044 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:02.020+0000 7f6d8000d700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d7811dfa0 con 0x7f6d7811a300 2019-12-05T05:27:02.044 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:02.020+0000 7f6d7dda9700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f6d7812dec0 0x7f6d7812e2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:33968/0 (socket says 172.21.15.42:33968) 2019-12-05T05:27:02.044 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:02.020+0000 7f6d7dda9700 1 -- 172.21.15.42:0/1990219554 learned_addr learned my addr 172.21.15.42:0/1990219554 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:27:02.045 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:02.020+0000 7f6d7edab700 1 --2- 172.21.15.42:0/1990219554 >> v2:172.21.15.42:3300/0 conn(0x7f6d781367b0 0x7f6d78138bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:27:02.045 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:02.020+0000 7f6d7e5aa700 1 --2- 172.21.15.42:0/1990219554 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6d7811a300 0x7f6d78121cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:27:02.045 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:02.020+0000 7f6d7edab700 1 -- 172.21.15.42:0/1990219554 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6d7811a300 msgr2=0x7f6d78121cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:27:02.045 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:02.020+0000 7f6d7edab700 1 --2- 172.21.15.42:0/1990219554 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6d7811a300 0x7f6d78121cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:27:02.046 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:02.020+0000 7f6d7edab700 1 -- 172.21.15.42:0/1990219554 >> v1:172.21.15.42:6789/0 conn(0x7f6d7812dec0 legacy=0x7f6d7812e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:27:02.046 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:02.020+0000 7f6d6effd700 1 -- 172.21.15.42:0/1990219554 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3054761592 0 0) 0x7f6d68001ec0 con 0x7f6d7812dec0 2019-12-05T05:27:02.046 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:02.020+0000 7f6d7edab700 1 -- 172.21.15.42:0/1990219554 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d78122200 con 0x7f6d781367b0 2019-12-05T05:27:02.046 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:02.020+0000 7f6d7edab700 1 --2- 172.21.15.42:0/1990219554 >> v2:172.21.15.42:3300/0 conn(0x7f6d781367b0 0x7f6d78138bf0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f6d7811dec0 tx=0x7f6d70010d30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:02.046 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:02.020+0000 7f6d7edab700 1 --2- 172.21.15.42:0/1990219554 >> v2:172.21.15.42:3300/0 conn(0x7f6d781367b0 0x7f6d78138bf0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f6d7811dec0 tx=0x7f6d70010d30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:02.046 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:02.020+0000 7f6d7edab700 1 --2- 172.21.15.42:0/1990219554 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6d781367b0 0x7f6d78138bf0 secure :-1 s=READY pgs=868 cs=0 l=1 rx=0x7f6d7811dec0 tx=0x7f6d70010d30).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:27:02.047 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:02.020+0000 7f6d7edab700 1 --2- 172.21.15.42:0/1990219554 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6d781367b0 0x7f6d78138bf0 secure :-1 s=THROTTLE_DONE pgs=868 cs=0 l=1 rx=0x7f6d7811dec0 tx=0x7f6d70010d30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:02.047 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:02.020+0000 7f6d6effd700 1 -- 172.21.15.42:0/1990219554 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6d70005ab0 con 0x7f6d781367b0 2019-12-05T05:27:02.047 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:02.020+0000 7f6d7edab700 1 --2- 172.21.15.42:0/1990219554 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6d781367b0 0x7f6d78138bf0 secure :-1 s=THROTTLE_DONE pgs=868 cs=0 l=1 rx=0x7f6d7811dec0 tx=0x7f6d70010d30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:02.047 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:02.020+0000 7f6d7edab700 1 --2- 172.21.15.42:0/1990219554 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6d781367b0 0x7f6d78138bf0 secure :-1 s=THROTTLE_DONE pgs=868 cs=0 l=1 rx=0x7f6d7811dec0 tx=0x7f6d70010d30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:02.047 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:02.020+0000 7f6d6effd700 1 -- 172.21.15.42:0/1990219554 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d70010ac0 con 0x7f6d781367b0 2019-12-05T05:27:02.047 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:02.020+0000 7f6d8000d700 1 -- 172.21.15.42:0/1990219554 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d7807c740 con 0x7f6d781367b0 2019-12-05T05:27:02.047 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:02.020+0000 7f6d6effd700 1 -- 172.21.15.42:0/1990219554 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6d7000f6a0 con 0x7f6d781367b0 2019-12-05T05:27:02.048 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:02.020+0000 7f6d8000d700 1 -- 172.21.15.42:0/1990219554 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d78139140 con 0x7f6d781367b0 2019-12-05T05:27:02.048 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:02.020+0000 7f6d7edab700 1 --2- 172.21.15.42:0/1990219554 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6d781367b0 0x7f6d78138bf0 secure :-1 s=THROTTLE_DONE pgs=868 cs=0 l=1 rx=0x7f6d7811dec0 tx=0x7f6d70010d30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:02.048 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:02.024+0000 7f6d6effd700 1 -- 172.21.15.42:0/1990219554 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f6d700113d0 con 0x7f6d781367b0 2019-12-05T05:27:02.048 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:02.024+0000 7f6d7edab700 1 --2- 172.21.15.42:0/1990219554 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6d781367b0 0x7f6d78138bf0 secure :-1 s=THROTTLE_DONE pgs=868 cs=0 l=1 rx=0x7f6d7811dec0 tx=0x7f6d70010d30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:02.048 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:02.024+0000 7f6d6effd700 1 --2- 172.21.15.42:0/1990219554 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6d6003aca0 0x7f6d6003d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:27:02.048 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:02.024+0000 7f6d6effd700 1 -- 172.21.15.42:0/1990219554 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7321+0+0 (secure 0 0 0) 0x7f6d7004b070 con 0x7f6d781367b0 2019-12-05T05:27:02.048 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:02.024+0000 7f6d7e5aa700 1 --2- 172.21.15.42:0/1990219554 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6d6003aca0 0x7f6d6003d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:27:02.049 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:02.024+0000 7f6d7e5aa700 1 --2- 172.21.15.42:0/1990219554 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6d6003aca0 0x7f6d6003d1b0 crc :-1 s=READY pgs=382 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:27:02.049 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:02.024+0000 7f6d8000d700 1 -- 172.21.15.42:0/1990219554 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d7805d100 con 0x7f6d781367b0 2019-12-05T05:27:02.049 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:02.032+0000 7f6d7edab700 1 --2- 172.21.15.42:0/1990219554 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6d781367b0 0x7f6d78138bf0 secure :-1 s=THROTTLE_DONE pgs=868 cs=0 l=1 rx=0x7f6d7811dec0 tx=0x7f6d70010d30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:02.049 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:02.032+0000 7f6d6effd700 1 -- 172.21.15.42:0/1990219554 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f6d7000f940 con 0x7f6d781367b0 2019-12-05T05:27:02.198 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:02.192+0000 7f6d8000d700 1 -- 172.21.15.42:0/1990219554 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6d7805b8b0 con 0x7f6d781367b0 2019-12-05T05:27:02.199 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:02.192+0000 7f6d7edab700 1 --2- 172.21.15.42:0/1990219554 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6d781367b0 0x7f6d78138bf0 secure :-1 s=THROTTLE_DONE pgs=868 cs=0 l=1 rx=0x7f6d7811dec0 tx=0x7f6d70010d30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:02.199 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:02.192+0000 7f6d6effd700 1 -- 172.21.15.42:0/1990219554 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f6d70032ee0 con 0x7f6d781367b0 2019-12-05T05:27:02.199 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 108 2019-12-05T05:27:02.199 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:27:02.200 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6546,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/1149478696","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1149478696},{"type":"v1","addr":"172.21.15.40:6833","nonce":1149478696}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":108}],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:25:38.087763+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5240},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5240":{"gid":5240,"name":"a","rank":0,"incarnation":104,"state":"up:active","state_seq":27,"addr":"172.21.15.42:6835/1455108684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1455108684},{"type":"v1","addr":"172.21.15.42:6835","nonce":1455108684}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:27:02.213 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:02.196+0000 7f6d8000d700 1 -- 172.21.15.42:0/1990219554 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6d6003aca0 msgr2=0x7f6d6003d1b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:27:02.213 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:02.196+0000 7f6d8000d700 1 --2- 172.21.15.42:0/1990219554 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6d6003aca0 0x7f6d6003d1b0 crc :-1 s=READY pgs=382 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:27:02.214 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:02.196+0000 7f6d8000d700 1 -- 172.21.15.42:0/1990219554 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6d781367b0 msgr2=0x7f6d78138bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:27:02.214 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:02.196+0000 7f6d8000d700 1 --2- 172.21.15.42:0/1990219554 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6d781367b0 0x7f6d78138bf0 secure :-1 s=READY pgs=868 cs=0 l=1 rx=0x7f6d7811dec0 tx=0x7f6d70010d30).stop 2019-12-05T05:27:02.214 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:02.196+0000 7f6d8000d700 1 -- 172.21.15.42:0/1990219554 shutdown_connections 2019-12-05T05:27:02.214 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:02.196+0000 7f6d8000d700 1 --2- 172.21.15.42:0/1990219554 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6d6003aca0 0x7f6d6003d1b0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:27:02.214 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:02.196+0000 7f6d8000d700 1 --2- 172.21.15.42:0/1990219554 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6d781367b0 0x7f6d78138bf0 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:27:02.214 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:02.196+0000 7f6d8000d700 1 --2- 172.21.15.42:0/1990219554 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6d7811a300 0x7f6d78121cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:27:02.214 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:02.196+0000 7f6d8000d700 1 -- 172.21.15.42:0/1990219554 shutdown_connections 2019-12-05T05:27:02.215 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:02.196+0000 7f6d8000d700 1 -- 172.21.15.42:0/1990219554 wait complete. 2019-12-05T05:27:02.215 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:02.196+0000 7f6d8000d700 1 -- 172.21.15.42:0/1990219554 >> 172.21.15.42:0/1990219554 conn(0x7f6d780a9780 msgr2=0x7f6d7811cb50 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:27:02.215 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T05:27:04.217 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:27:04.335 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:04.316+0000 7f51844c0700 1 Processor -- start 2019-12-05T05:27:04.336 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:04.316+0000 7f51844c0700 1 -- start start 2019-12-05T05:27:04.336 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:04.316+0000 7f51844c0700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f517c08df80 0x7f517c09f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:27:04.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:04.316+0000 7f51844c0700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f517c09fc60 con 0x7f517c09f830 2019-12-05T05:27:04.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:04.316+0000 7f51844c0700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f517c0a0710 con 0x7f517c0a7740 2019-12-05T05:27:04.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:04.316+0000 7f51844c0700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f517c0a0b00 con 0x7f517c08df80 2019-12-05T05:27:04.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:04.316+0000 7f518225c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f517c08df80 0x7f517c09f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:27:04.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:04.316+0000 7f518225c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f517c08df80 0x7f517c09f2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:33938/0 (socket says 172.21.15.42:33938) 2019-12-05T05:27:04.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:04.316+0000 7f518225c700 1 -- 172.21.15.42:0/3448474831 learned_addr learned my addr 172.21.15.42:0/3448474831 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:27:04.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:04.320+0000 7f518225c700 1 -- 172.21.15.42:0/3448474831 >> v1:172.21.15.42:6789/0 conn(0x7f517c0a7740 legacy=0x7f517c07e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:27:04.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:04.320+0000 7f518225c700 1 -- 172.21.15.42:0/3448474831 >> v1:172.21.15.40:6789/0 conn(0x7f517c09f830 legacy=0x7f517c14efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:27:04.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:04.320+0000 7f518225c700 1 -- 172.21.15.42:0/3448474831 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f517c07ef20 con 0x7f517c08df80 2019-12-05T05:27:04.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:04.320+0000 7f518225c700 1 --2- 172.21.15.42:0/3448474831 >> v2:172.21.15.42:3300/0 conn(0x7f517c08df80 0x7f517c09f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f516c008b10 tx=0x7f516c010030).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:04.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:04.320+0000 7f518225c700 1 --2- 172.21.15.42:0/3448474831 >> v2:172.21.15.42:3300/0 conn(0x7f517c08df80 0x7f517c09f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f516c008b10 tx=0x7f516c010030).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:04.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:04.320+0000 7f518225c700 1 --2- 172.21.15.42:0/3448474831 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f517c08df80 0x7f517c09f2e0 secure :-1 s=READY pgs=870 cs=0 l=1 rx=0x7f516c008b10 tx=0x7f516c010030).ready entity=mon.1 client_cookie=e90268c096508a4a server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:27:04.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:04.320+0000 7f518225c700 1 --2- 172.21.15.42:0/3448474831 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f517c08df80 0x7f517c09f2e0 secure :-1 s=THROTTLE_DONE pgs=870 cs=0 l=1 rx=0x7f516c008b10 tx=0x7f516c010030).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:04.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:04.320+0000 7f5181a5b700 1 -- 172.21.15.42:0/3448474831 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f516c00a160 con 0x7f517c08df80 2019-12-05T05:27:04.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:04.320+0000 7f518225c700 1 --2- 172.21.15.42:0/3448474831 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f517c08df80 0x7f517c09f2e0 secure :-1 s=THROTTLE_DONE pgs=870 cs=0 l=1 rx=0x7f516c008b10 tx=0x7f516c010030).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:04.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:04.320+0000 7f518225c700 1 --2- 172.21.15.42:0/3448474831 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f517c08df80 0x7f517c09f2e0 secure :-1 s=THROTTLE_DONE pgs=870 cs=0 l=1 rx=0x7f516c008b10 tx=0x7f516c010030).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:04.341 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:04.320+0000 7f5181a5b700 1 -- 172.21.15.42:0/3448474831 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f516c011aa0 con 0x7f517c08df80 2019-12-05T05:27:04.341 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:04.320+0000 7f5181a5b700 1 -- 172.21.15.42:0/3448474831 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f516c00a740 con 0x7f517c08df80 2019-12-05T05:27:04.341 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:04.320+0000 7f51844c0700 1 -- 172.21.15.42:0/3448474831 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f517c08df80 msgr2=0x7f517c09f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:27:04.341 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:04.320+0000 7f51844c0700 1 --2- 172.21.15.42:0/3448474831 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f517c08df80 0x7f517c09f2e0 secure :-1 s=READY pgs=870 cs=0 l=1 rx=0x7f516c008b10 tx=0x7f516c010030).stop 2019-12-05T05:27:04.341 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:04.320+0000 7f51844c0700 1 -- 172.21.15.42:0/3448474831 shutdown_connections 2019-12-05T05:27:04.341 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:04.320+0000 7f51844c0700 1 --2- 172.21.15.42:0/3448474831 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f517c08df80 0x7f517c09f2e0 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:27:04.341 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:04.320+0000 7f51844c0700 1 -- 172.21.15.42:0/3448474831 shutdown_connections 2019-12-05T05:27:04.342 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:04.320+0000 7f51844c0700 1 -- 172.21.15.42:0/3448474831 wait complete. 2019-12-05T05:27:04.342 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:04.320+0000 7f51844c0700 1 -- 172.21.15.42:0/3448474831 >> 172.21.15.42:0/3448474831 conn(0x7f517c0aabe0 msgr2=0x7f517c0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:27:04.342 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:04.324+0000 7f51844c0700 1 Processor -- start 2019-12-05T05:27:04.342 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:04.324+0000 7f51844c0700 1 -- start start 2019-12-05T05:27:04.342 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:04.324+0000 7f51844c0700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f517c118a40 0x7f517c0a6bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:27:04.342 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:04.324+0000 7f51844c0700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f517c143600 0x7f517c077cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:27:04.342 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:04.324+0000 7f51844c0700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f517c0a7110 con 0x7f517c078240 2019-12-05T05:27:04.343 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:04.324+0000 7f51844c0700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f517c142fc0 con 0x7f517c118a40 2019-12-05T05:27:04.343 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:04.324+0000 7f51844c0700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f517c143b70 con 0x7f517c143600 2019-12-05T05:27:04.343 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:04.324+0000 7f518325e700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f517c078240 0x7f517c0a5a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:33986/0 (socket says 172.21.15.42:33986) 2019-12-05T05:27:04.343 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:04.324+0000 7f518325e700 1 -- 172.21.15.42:0/3729742022 learned_addr learned my addr 172.21.15.42:0/3729742022 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:27:04.343 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:04.324+0000 7f5182a5d700 1 --2- 172.21.15.42:0/3729742022 >> v2:172.21.15.40:3300/0 conn(0x7f517c118a40 0x7f517c0a6bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:27:04.343 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:04.324+0000 7f518225c700 1 --2- 172.21.15.42:0/3729742022 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f517c143600 0x7f517c077cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:27:04.343 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:04.324+0000 7f51737fe700 1 -- 172.21.15.42:0/3729742022 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2874329517 0 0) 0x7f517c0a7110 con 0x7f517c078240 2019-12-05T05:27:04.344 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:04.324+0000 7f51737fe700 1 -- 172.21.15.42:0/3729742022 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f515c0030b0 con 0x7f517c078240 2019-12-05T05:27:04.344 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:04.328+0000 7f518225c700 1 -- 172.21.15.42:0/3729742022 >> v2:172.21.15.40:3300/0 conn(0x7f517c118a40 msgr2=0x7f517c0a6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:27:04.344 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:04.328+0000 7f518225c700 1 --2- 172.21.15.42:0/3729742022 >> v2:172.21.15.40:3300/0 conn(0x7f517c118a40 0x7f517c0a6bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:27:04.344 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:04.328+0000 7f51737fe700 1 -- 172.21.15.42:0/3729742022 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 3031570374 0 0) 0x7f515c0030b0 con 0x7f517c078240 2019-12-05T05:27:04.344 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:04.328+0000 7f518225c700 1 -- 172.21.15.42:0/3729742022 >> v1:172.21.15.42:6789/0 conn(0x7f517c078240 legacy=0x7f517c0a5a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:27:04.344 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:04.328+0000 7f518225c700 1 -- 172.21.15.42:0/3729742022 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f517c1440b0 con 0x7f517c143600 2019-12-05T05:27:04.345 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:04.328+0000 7f518225c700 1 --2- 172.21.15.42:0/3729742022 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f517c143600 0x7f517c077cf0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f5174001b60 tx=0x7f5174001ba0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:04.347 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:04.328+0000 7f518225c700 1 --2- 172.21.15.42:0/3729742022 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f517c143600 0x7f517c077cf0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f5174001b60 tx=0x7f5174001ba0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:04.347 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:04.328+0000 7f518225c700 1 --2- 172.21.15.42:0/3729742022 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f517c143600 0x7f517c077cf0 secure :-1 s=READY pgs=178 cs=0 l=1 rx=0x7f5174001b60 tx=0x7f5174001ba0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:27:04.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:04.328+0000 7f518225c700 1 --2- 172.21.15.42:0/3729742022 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f517c143600 0x7f517c077cf0 secure :-1 s=THROTTLE_DONE pgs=178 cs=0 l=1 rx=0x7f5174001b60 tx=0x7f5174001ba0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:04.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:04.328+0000 7f51737fe700 1 -- 172.21.15.42:0/3729742022 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5174001920 con 0x7f517c143600 2019-12-05T05:27:04.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:04.328+0000 7f518225c700 1 --2- 172.21.15.42:0/3729742022 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f517c143600 0x7f517c077cf0 secure :-1 s=THROTTLE_DONE pgs=178 cs=0 l=1 rx=0x7f5174001b60 tx=0x7f5174001ba0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:04.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:04.328+0000 7f518225c700 1 --2- 172.21.15.42:0/3729742022 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f517c143600 0x7f517c077cf0 secure :-1 s=THROTTLE_DONE pgs=178 cs=0 l=1 rx=0x7f5174001b60 tx=0x7f5174001ba0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:04.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:04.328+0000 7f51737fe700 1 -- 172.21.15.42:0/3729742022 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5174001730 con 0x7f517c143600 2019-12-05T05:27:04.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:04.328+0000 7f51844c0700 1 -- 172.21.15.42:0/3729742022 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f517c0a6220 con 0x7f517c143600 2019-12-05T05:27:04.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:04.328+0000 7f51737fe700 1 -- 172.21.15.42:0/3729742022 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f516c005ab0 con 0x7f517c143600 2019-12-05T05:27:04.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:04.328+0000 7f518225c700 1 -- 172.21.15.42:0/3729742022 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f517c143600 msgr2=0x7f517c077cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2019-12-05T05:27:04.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:04.328+0000 7f518225c700 1 -- 172.21.15.42:0/3729742022 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f517c143600 msgr2=0x7f517c077cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2019-12-05T05:27:04.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:04.328+0000 7f51844c0700 1 -- 172.21.15.42:0/3729742022 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f517c12df90 con 0x7f517c143600 2019-12-05T05:27:04.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:04.328+0000 7f518225c700 1 --2- 172.21.15.42:0/3729742022 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f517c143600 0x7f517c077cf0 secure :-1 s=READY pgs=178 cs=0 l=1 rx=0x7f5174001b60 tx=0x7f5174001ba0).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-12-05T05:27:04.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:04.328+0000 7f518225c700 1 --2- 172.21.15.42:0/3729742022 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f517c143600 0x7f517c077cf0 secure :-1 s=READY pgs=178 cs=0 l=1 rx=0x7f5174001b60 tx=0x7f5174001ba0).stop 2019-12-05T05:27:04.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:04.328+0000 7f51737fe700 1 -- 172.21.15.42:0/3729742022 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f517c143600 msgr2=0x7f517c077cf0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2019-12-05T05:27:04.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:04.328+0000 7f51737fe700 1 --2- 172.21.15.42:0/3729742022 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f517c143600 0x7f517c077cf0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:27:04.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:04.328+0000 7f51737fe700 1 --2- 172.21.15.42:0/3729742022 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f515c0033c0 0x7f515c0037f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:27:04.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:04.328+0000 7f51737fe700 1 --2- 172.21.15.42:0/3729742022 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f515c003d40 0x7f515c004420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:27:04.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:04.328+0000 7f51737fe700 1 --2- 172.21.15.42:0/3729742022 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f515c004970 0x7f515c004da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:27:04.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:04.328+0000 7f51737fe700 1 -- 172.21.15.42:0/3729742022 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f515c00b670 con 0x7f515c0033c0 2019-12-05T05:27:04.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:04.328+0000 7f51737fe700 1 -- 172.21.15.42:0/3729742022 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f515c00b860 con 0x7f515c003d40 2019-12-05T05:27:04.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:04.328+0000 7f51737fe700 1 -- 172.21.15.42:0/3729742022 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f515c00baa0 con 0x7f515c004970 2019-12-05T05:27:04.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:04.328+0000 7f518225c700 1 --2- 172.21.15.42:0/3729742022 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f515c003d40 0x7f515c004420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:27:04.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:04.328+0000 7f5182a5d700 1 --2- 172.21.15.42:0/3729742022 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f515c0033c0 0x7f515c0037f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:27:04.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:04.328+0000 7f518325e700 1 --2- 172.21.15.42:0/3729742022 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f515c004970 0x7f515c004da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:27:04.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:04.328+0000 7f518225c700 1 -- 172.21.15.42:0/3729742022 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f515c004970 msgr2=0x7f515c004da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:27:04.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:04.328+0000 7f518225c700 1 --2- 172.21.15.42:0/3729742022 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f515c004970 0x7f515c004da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:27:04.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:04.328+0000 7f518225c700 1 -- 172.21.15.42:0/3729742022 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f515c0033c0 msgr2=0x7f515c0037f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:27:04.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:04.328+0000 7f518225c700 1 --2- 172.21.15.42:0/3729742022 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f515c0033c0 0x7f515c0037f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:27:04.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:04.328+0000 7f518225c700 1 -- 172.21.15.42:0/3729742022 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+,osdmap=0}) v3 -- 0x7f515c00be60 con 0x7f515c003d40 2019-12-05T05:27:04.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:04.328+0000 7f5182a5d700 1 --2- 172.21.15.42:0/3729742022 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f515c0033c0 0x7f515c0037f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:27:04.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:04.328+0000 7f518225c700 1 --2- 172.21.15.42:0/3729742022 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f515c003d40 0x7f515c004420 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f5174001b60 tx=0x7f516c0075b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:04.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:04.328+0000 7f518225c700 1 --2- 172.21.15.42:0/3729742022 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f515c003d40 0x7f515c004420 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f5174001b60 tx=0x7f516c0075b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:04.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:04.328+0000 7f518225c700 1 --2- 172.21.15.42:0/3729742022 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f515c003d40 0x7f515c004420 secure :-1 s=READY pgs=872 cs=0 l=1 rx=0x7f5174001b60 tx=0x7f516c0075b0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:27:04.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:04.328+0000 7f518225c700 1 --2- 172.21.15.42:0/3729742022 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f515c003d40 0x7f515c004420 secure :-1 s=THROTTLE_DONE pgs=872 cs=0 l=1 rx=0x7f5174001b60 tx=0x7f516c0075b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:04.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:04.328+0000 7f51737fe700 1 -- 172.21.15.42:0/3729742022 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f516c00f760 con 0x7f515c003d40 2019-12-05T05:27:04.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:04.328+0000 7f518225c700 1 --2- 172.21.15.42:0/3729742022 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f515c003d40 0x7f515c004420 secure :-1 s=THROTTLE_DONE pgs=872 cs=0 l=1 rx=0x7f5174001b60 tx=0x7f516c0075b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:04.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:04.328+0000 7f51737fe700 1 -- 172.21.15.42:0/3729742022 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f516c00ac60 con 0x7f515c003d40 2019-12-05T05:27:04.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:04.332+0000 7f518225c700 1 --2- 172.21.15.42:0/3729742022 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f515c003d40 0x7f515c004420 secure :-1 s=THROTTLE_DONE pgs=872 cs=0 l=1 rx=0x7f5174001b60 tx=0x7f516c0075b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:04.361 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:04.332+0000 7f51737fe700 1 -- 172.21.15.42:0/3729742022 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f516c00f960 con 0x7f515c003d40 2019-12-05T05:27:04.362 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:04.332+0000 7f518225c700 1 --2- 172.21.15.42:0/3729742022 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f515c003d40 0x7f515c004420 secure :-1 s=THROTTLE_DONE pgs=872 cs=0 l=1 rx=0x7f5174001b60 tx=0x7f516c0075b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:04.362 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:04.332+0000 7f51737fe700 1 --2- 172.21.15.42:0/3729742022 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f515c043ca0 0x7f515c0461b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:27:04.362 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:04.332+0000 7f51737fe700 1 -- 172.21.15.42:0/3729742022 <== mon.1 v2:172.21.15.42:3300/0 4 ==== osd_map(64..64 src has 1..64) v4 ==== 7321+0+0 (secure 0 0 0) 0x7f516c04cde0 con 0x7f515c003d40 2019-12-05T05:27:04.363 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:04.332+0000 7f5182a5d700 1 --2- 172.21.15.42:0/3729742022 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f515c043ca0 0x7f515c0461b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:27:04.363 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:04.332+0000 7f5182a5d700 1 --2- 172.21.15.42:0/3729742022 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f515c043ca0 0x7f515c0461b0 crc :-1 s=READY pgs=383 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:27:04.363 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:04.332+0000 7f51844c0700 1 -- 172.21.15.42:0/3729742022 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f517c05d110 con 0x7f515c003d40 2019-12-05T05:27:04.364 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:04.340+0000 7f518225c700 1 --2- 172.21.15.42:0/3729742022 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f515c003d40 0x7f515c004420 secure :-1 s=THROTTLE_DONE pgs=872 cs=0 l=1 rx=0x7f5174001b60 tx=0x7f516c0075b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:04.364 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:04.340+0000 7f51737fe700 1 -- 172.21.15.42:0/3729742022 <== mon.1 v2:172.21.15.42:3300/0 5 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f516c035750 con 0x7f515c003d40 2019-12-05T05:27:04.510 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:04.504+0000 7f51844c0700 1 -- 172.21.15.42:0/3729742022 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f517c07e210 con 0x7f515c003d40 2019-12-05T05:27:04.511 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:04.504+0000 7f518225c700 1 --2- 172.21.15.42:0/3729742022 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f515c003d40 0x7f515c004420 secure :-1 s=THROTTLE_DONE pgs=872 cs=0 l=1 rx=0x7f5174001b60 tx=0x7f516c0075b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:04.511 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:04.504+0000 7f51737fe700 1 -- 172.21.15.42:0/3729742022 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f516c034be0 con 0x7f515c003d40 2019-12-05T05:27:04.513 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:27:04.513 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6546,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/1149478696","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1149478696},{"type":"v1","addr":"172.21.15.40:6833","nonce":1149478696}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":108}],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:25:38.087763+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5240},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5240":{"gid":5240,"name":"a","rank":0,"incarnation":104,"state":"up:active","state_seq":27,"addr":"172.21.15.42:6835/1455108684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1455108684},{"type":"v1","addr":"172.21.15.42:6835","nonce":1455108684}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:27:04.514 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 108 2019-12-05T05:27:04.515 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:04.508+0000 7f51844c0700 1 -- 172.21.15.42:0/3729742022 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f515c043ca0 msgr2=0x7f515c0461b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:27:04.515 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:04.508+0000 7f51844c0700 1 --2- 172.21.15.42:0/3729742022 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f515c043ca0 0x7f515c0461b0 crc :-1 s=READY pgs=383 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:27:04.516 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:04.508+0000 7f51844c0700 1 -- 172.21.15.42:0/3729742022 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f515c003d40 msgr2=0x7f515c004420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:27:04.516 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:04.508+0000 7f51844c0700 1 --2- 172.21.15.42:0/3729742022 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f515c003d40 0x7f515c004420 secure :-1 s=READY pgs=872 cs=0 l=1 rx=0x7f5174001b60 tx=0x7f516c0075b0).stop 2019-12-05T05:27:04.516 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:04.508+0000 7f518325e700 1 -- 172.21.15.42:0/3729742022 reap_dead start 2019-12-05T05:27:04.516 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:04.508+0000 7f518325e700 1 -- 172.21.15.42:0/3729742022 reap_dead start 2019-12-05T05:27:04.517 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:04.508+0000 7f51844c0700 1 -- 172.21.15.42:0/3729742022 shutdown_connections 2019-12-05T05:27:04.517 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:04.508+0000 7f51844c0700 1 -- 172.21.15.42:0/3729742022 shutdown_connections 2019-12-05T05:27:04.517 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:04.508+0000 7f51844c0700 1 -- 172.21.15.42:0/3729742022 wait complete. 2019-12-05T05:27:04.517 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:04.508+0000 7f51844c0700 1 -- 172.21.15.42:0/3729742022 >> 172.21.15.42:0/3729742022 conn(0x7f517c08fda0 msgr2=0x7f517c07e4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:27:04.529 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T05:27:06.531 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:27:06.638 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:06.624+0000 7efe8d3f7700 1 Processor -- start 2019-12-05T05:27:06.638 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:06.624+0000 7efe8d3f7700 1 -- start start 2019-12-05T05:27:06.638 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:06.628+0000 7efe8d3f7700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efe8808df80 0x7efe8809f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:27:06.638 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:06.628+0000 7efe8d3f7700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe8809fc60 con 0x7efe880a7740 2019-12-05T05:27:06.639 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:06.628+0000 7efe8d3f7700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe880a0710 con 0x7efe8809f830 2019-12-05T05:27:06.639 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:06.628+0000 7efe8d3f7700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe880a0b00 con 0x7efe8808df80 2019-12-05T05:27:06.639 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:06.628+0000 7efe87fff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7efe8809f830 0x7efe8814efe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:34000/0 (socket says 172.21.15.42:34000) 2019-12-05T05:27:06.639 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:06.628+0000 7efe87fff700 1 -- 172.21.15.42:0/3321774701 learned_addr learned my addr 172.21.15.42:0/3321774701 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:27:06.639 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:06.628+0000 7efe86ffd700 1 --2- 172.21.15.42:0/3321774701 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efe8808df80 0x7efe8809f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:27:06.639 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:06.628+0000 7efe867fc700 1 -- 172.21.15.42:0/3321774701 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1437891890 0 0) 0x7efe880a0710 con 0x7efe8809f830 2019-12-05T05:27:06.640 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:06.628+0000 7efe867fc700 1 -- 172.21.15.42:0/3321774701 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe700030b0 con 0x7efe8809f830 2019-12-05T05:27:06.640 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:06.628+0000 7efe867fc700 1 -- 172.21.15.42:0/3321774701 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 828437761 0 0) 0x7efe8809fc60 con 0x7efe880a7740 2019-12-05T05:27:06.640 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:06.628+0000 7efe867fc700 1 -- 172.21.15.42:0/3321774701 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe880a0710 con 0x7efe880a7740 2019-12-05T05:27:06.640 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:06.628+0000 7efe86ffd700 1 -- 172.21.15.42:0/3321774701 >> v1:172.21.15.42:6789/0 conn(0x7efe8809f830 legacy=0x7efe8814efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:27:06.640 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:06.628+0000 7efe867fc700 1 -- 172.21.15.42:0/3321774701 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 262695282 0 0) 0x7efe700030b0 con 0x7efe8809f830 2019-12-05T05:27:06.640 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:06.628+0000 7efe86ffd700 1 -- 172.21.15.42:0/3321774701 >> v1:172.21.15.40:6789/0 conn(0x7efe880a7740 legacy=0x7efe8807e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:27:06.640 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:06.628+0000 7efe86ffd700 1 -- 172.21.15.42:0/3321774701 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe8807ef20 con 0x7efe8808df80 2019-12-05T05:27:06.641 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:06.628+0000 7efe86ffd700 1 --2- 172.21.15.42:0/3321774701 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efe8808df80 0x7efe8809f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7efe7c008790 tx=0x7efe80001cd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:06.641 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:06.628+0000 7efe86ffd700 1 --2- 172.21.15.42:0/3321774701 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efe8808df80 0x7efe8809f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7efe7c008790 tx=0x7efe80001cd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:06.641 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:06.628+0000 7efe86ffd700 1 --2- 172.21.15.42:0/3321774701 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efe8808df80 0x7efe8809f2e0 secure :-1 s=READY pgs=179 cs=0 l=1 rx=0x7efe7c008790 tx=0x7efe80001cd0).ready entity=mon.2 client_cookie=6d57ea2d9840b20e server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:27:06.641 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:06.628+0000 7efe86ffd700 1 --2- 172.21.15.42:0/3321774701 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efe8808df80 0x7efe8809f2e0 secure :-1 s=THROTTLE_DONE pgs=179 cs=0 l=1 rx=0x7efe7c008790 tx=0x7efe80001cd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:06.641 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:06.628+0000 7efe867fc700 1 -- 172.21.15.42:0/3321774701 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efe80001ad0 con 0x7efe8808df80 2019-12-05T05:27:06.641 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:06.628+0000 7efe86ffd700 1 --2- 172.21.15.42:0/3321774701 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efe8808df80 0x7efe8809f2e0 secure :-1 s=THROTTLE_DONE pgs=179 cs=0 l=1 rx=0x7efe7c008790 tx=0x7efe80001cd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:06.641 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:06.628+0000 7efe86ffd700 1 --2- 172.21.15.42:0/3321774701 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efe8808df80 0x7efe8809f2e0 secure :-1 s=THROTTLE_DONE pgs=179 cs=0 l=1 rx=0x7efe7c008790 tx=0x7efe80001cd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:06.642 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:06.628+0000 7efe867fc700 1 -- 172.21.15.42:0/3321774701 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe800018e0 con 0x7efe8808df80 2019-12-05T05:27:06.642 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:06.628+0000 7efe867fc700 1 -- 172.21.15.42:0/3321774701 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efe7c011360 con 0x7efe8808df80 2019-12-05T05:27:06.642 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:06.628+0000 7efe8d3f7700 1 -- 172.21.15.42:0/3321774701 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efe8808df80 msgr2=0x7efe8809f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:27:06.642 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:06.628+0000 7efe8d3f7700 1 --2- 172.21.15.42:0/3321774701 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efe8808df80 0x7efe8809f2e0 secure :-1 s=READY pgs=179 cs=0 l=1 rx=0x7efe7c008790 tx=0x7efe80001cd0).stop 2019-12-05T05:27:06.642 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:06.632+0000 7efe8d3f7700 1 -- 172.21.15.42:0/3321774701 shutdown_connections 2019-12-05T05:27:06.642 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:06.632+0000 7efe8d3f7700 1 --2- 172.21.15.42:0/3321774701 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efe8808df80 0x7efe8809f2e0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:27:06.642 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:06.632+0000 7efe8d3f7700 1 -- 172.21.15.42:0/3321774701 shutdown_connections 2019-12-05T05:27:06.643 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:06.632+0000 7efe8d3f7700 1 -- 172.21.15.42:0/3321774701 wait complete. 2019-12-05T05:27:06.643 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:06.632+0000 7efe8d3f7700 1 -- 172.21.15.42:0/3321774701 >> 172.21.15.42:0/3321774701 conn(0x7efe880aabe0 msgr2=0x7efe880a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:27:06.646 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:06.632+0000 7efe8d3f7700 1 Processor -- start 2019-12-05T05:27:06.646 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:06.632+0000 7efe8d3f7700 1 -- start start 2019-12-05T05:27:06.647 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:06.636+0000 7efe8d3f7700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7efe8807c620 0x7efe88143570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:27:06.647 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:06.636+0000 7efe8d3f7700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efe8807adf0 0x7efe8812c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:27:06.647 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:06.636+0000 7efe8d3f7700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe88144650 con 0x7efe8807a9c0 2019-12-05T05:27:06.647 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:06.636+0000 7efe8d3f7700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe88144960 con 0x7efe8807c620 2019-12-05T05:27:06.647 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:06.636+0000 7efe8d3f7700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe88144c00 con 0x7efe8807adf0 2019-12-05T05:27:06.647 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:06.636+0000 7efe877fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7efe8807c620 0x7efe88143570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:27:06.647 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:06.636+0000 7efe877fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7efe8807c620 0x7efe88143570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:33964/0 (socket says 172.21.15.42:33964) 2019-12-05T05:27:06.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:06.636+0000 7efe877fe700 1 -- 172.21.15.42:0/1925770182 learned_addr learned my addr 172.21.15.42:0/1925770182 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:27:06.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:06.636+0000 7efe87fff700 1 --2- 172.21.15.42:0/1925770182 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efe8807adf0 0x7efe8812c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:27:06.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:06.636+0000 7efe877fe700 1 -- 172.21.15.42:0/1925770182 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efe8807adf0 msgr2=0x7efe8812c4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:27:06.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:06.636+0000 7efe877fe700 1 --2- 172.21.15.42:0/1925770182 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efe8807adf0 0x7efe8812c4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:27:06.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:06.636+0000 7efe877fe700 1 -- 172.21.15.42:0/1925770182 >> v1:172.21.15.40:6789/0 conn(0x7efe8807a9c0 legacy=0x7efe881250a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:27:06.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:06.636+0000 7efe877fe700 1 -- 172.21.15.42:0/1925770182 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe881258a0 con 0x7efe8807c620 2019-12-05T05:27:06.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:06.636+0000 7efe877fe700 1 --2- 172.21.15.42:0/1925770182 >> v2:172.21.15.42:3300/0 conn(0x7efe8807c620 0x7efe88143570 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7efe88144880 tx=0x7efe780109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:06.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:06.636+0000 7efe877fe700 1 --2- 172.21.15.42:0/1925770182 >> v2:172.21.15.42:3300/0 conn(0x7efe8807c620 0x7efe88143570 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7efe88144880 tx=0x7efe780109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:06.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:06.636+0000 7efe877fe700 1 --2- 172.21.15.42:0/1925770182 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efe8807c620 0x7efe88143570 secure :-1 s=READY pgs=874 cs=0 l=1 rx=0x7efe88144880 tx=0x7efe780109a0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:27:06.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:06.636+0000 7efe877fe700 1 --2- 172.21.15.42:0/1925770182 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efe8807c620 0x7efe88143570 secure :-1 s=THROTTLE_DONE pgs=874 cs=0 l=1 rx=0x7efe88144880 tx=0x7efe780109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:06.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:06.636+0000 7efe877fe700 1 --2- 172.21.15.42:0/1925770182 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efe8807c620 0x7efe88143570 secure :-1 s=THROTTLE_DONE pgs=874 cs=0 l=1 rx=0x7efe88144880 tx=0x7efe780109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:06.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:06.636+0000 7efe6ffff700 1 -- 172.21.15.42:0/1925770182 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efe7800caa0 con 0x7efe8807c620 2019-12-05T05:27:06.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:06.636+0000 7efe877fe700 1 --2- 172.21.15.42:0/1925770182 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efe8807c620 0x7efe88143570 secure :-1 s=THROTTLE_DONE pgs=874 cs=0 l=1 rx=0x7efe88144880 tx=0x7efe780109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:06.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:06.636+0000 7efe6ffff700 1 -- 172.21.15.42:0/1925770182 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe7800c6c0 con 0x7efe8807c620 2019-12-05T05:27:06.650 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:06.636+0000 7efe8d3f7700 1 -- 172.21.15.42:0/1925770182 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe88143ac0 con 0x7efe8807c620 2019-12-05T05:27:06.650 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:06.636+0000 7efe6ffff700 1 -- 172.21.15.42:0/1925770182 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efe7800d6a0 con 0x7efe8807c620 2019-12-05T05:27:06.650 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:06.636+0000 7efe8d3f7700 1 -- 172.21.15.42:0/1925770182 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe88145140 con 0x7efe8807c620 2019-12-05T05:27:06.650 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:06.640+0000 7efe877fe700 1 --2- 172.21.15.42:0/1925770182 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efe8807c620 0x7efe88143570 secure :-1 s=THROTTLE_DONE pgs=874 cs=0 l=1 rx=0x7efe88144880 tx=0x7efe780109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:06.650 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:06.640+0000 7efe6ffff700 1 -- 172.21.15.42:0/1925770182 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7efe7800d8a0 con 0x7efe8807c620 2019-12-05T05:27:06.650 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:06.640+0000 7efe877fe700 1 --2- 172.21.15.42:0/1925770182 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efe8807c620 0x7efe88143570 secure :-1 s=THROTTLE_DONE pgs=874 cs=0 l=1 rx=0x7efe88144880 tx=0x7efe780109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:06.650 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:06.640+0000 7efe6ffff700 1 --2- 172.21.15.42:0/1925770182 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7efe6803acf0 0x7efe6803d200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:27:06.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:06.640+0000 7efe6ffff700 1 -- 172.21.15.42:0/1925770182 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7321+0+0 (secure 0 0 0) 0x7efe7804b270 con 0x7efe8807c620 2019-12-05T05:27:06.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:06.640+0000 7efe86ffd700 1 --2- 172.21.15.42:0/1925770182 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7efe6803acf0 0x7efe6803d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:27:06.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:06.640+0000 7efe86ffd700 1 --2- 172.21.15.42:0/1925770182 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7efe6803acf0 0x7efe6803d200 crc :-1 s=READY pgs=384 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:27:06.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:06.640+0000 7efe8d3f7700 1 -- 172.21.15.42:0/1925770182 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe8805d110 con 0x7efe8807c620 2019-12-05T05:27:06.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:06.648+0000 7efe877fe700 1 --2- 172.21.15.42:0/1925770182 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efe8807c620 0x7efe88143570 secure :-1 s=THROTTLE_DONE pgs=874 cs=0 l=1 rx=0x7efe88144880 tx=0x7efe780109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:06.654 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:06.648+0000 7efe6ffff700 1 -- 172.21.15.42:0/1925770182 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7efe78032550 con 0x7efe8807c620 2019-12-05T05:27:06.817 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:06.812+0000 7efe8d3f7700 1 -- 172.21.15.42:0/1925770182 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efe880a71e0 con 0x7efe8807c620 2019-12-05T05:27:06.818 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:06.812+0000 7efe877fe700 1 --2- 172.21.15.42:0/1925770182 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efe8807c620 0x7efe88143570 secure :-1 s=THROTTLE_DONE pgs=874 cs=0 l=1 rx=0x7efe88144880 tx=0x7efe780109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:06.819 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:06.812+0000 7efe6ffff700 1 -- 172.21.15.42:0/1925770182 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+2112 (secure 0 0 0) 0x7efe78033100 con 0x7efe8807c620 2019-12-05T05:27:06.819 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 108 2019-12-05T05:27:06.819 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:27:06.819 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6546,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/1149478696","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1149478696},{"type":"v1","addr":"172.21.15.40:6833","nonce":1149478696}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":108}],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:25:38.087763+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5240},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5240":{"gid":5240,"name":"a","rank":0,"incarnation":104,"state":"up:active","state_seq":27,"addr":"172.21.15.42:6835/1455108684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1455108684},{"type":"v1","addr":"172.21.15.42:6835","nonce":1455108684}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:27:06.821 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:06.816+0000 7efe8d3f7700 1 -- 172.21.15.42:0/1925770182 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7efe6803acf0 msgr2=0x7efe6803d200 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:27:06.821 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:06.816+0000 7efe8d3f7700 1 --2- 172.21.15.42:0/1925770182 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7efe6803acf0 0x7efe6803d200 crc :-1 s=READY pgs=384 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:27:06.822 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:06.816+0000 7efe8d3f7700 1 -- 172.21.15.42:0/1925770182 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efe8807c620 msgr2=0x7efe88143570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:27:06.822 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:06.816+0000 7efe8d3f7700 1 --2- 172.21.15.42:0/1925770182 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efe8807c620 0x7efe88143570 secure :-1 s=READY pgs=874 cs=0 l=1 rx=0x7efe88144880 tx=0x7efe780109a0).stop 2019-12-05T05:27:06.822 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:06.816+0000 7efe8d3f7700 1 -- 172.21.15.42:0/1925770182 shutdown_connections 2019-12-05T05:27:06.822 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:06.816+0000 7efe8d3f7700 1 --2- 172.21.15.42:0/1925770182 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7efe6803acf0 0x7efe6803d200 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:27:06.822 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:06.816+0000 7efe8d3f7700 1 --2- 172.21.15.42:0/1925770182 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efe8807adf0 0x7efe8812c4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:27:06.823 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:06.816+0000 7efe8d3f7700 1 --2- 172.21.15.42:0/1925770182 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efe8807c620 0x7efe88143570 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:27:06.823 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:06.816+0000 7efe8d3f7700 1 -- 172.21.15.42:0/1925770182 shutdown_connections 2019-12-05T05:27:06.823 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:06.816+0000 7efe8d3f7700 1 -- 172.21.15.42:0/1925770182 wait complete. 2019-12-05T05:27:06.823 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:06.816+0000 7efe8d3f7700 1 -- 172.21.15.42:0/1925770182 >> 172.21.15.42:0/1925770182 conn(0x7efe8807de90 msgr2=0x7efe88059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:27:06.831 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T05:27:08.536 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:27:08.541 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:27:08.832 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:27:08.935 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:08.924+0000 7f26294a5700 1 Processor -- start 2019-12-05T05:27:08.936 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:08.924+0000 7f26294a5700 1 -- start start 2019-12-05T05:27:08.936 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:08.928+0000 7f26294a5700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f26240a7730 0x7f262408df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:27:08.936 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:08.928+0000 7f26294a5700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f262407e710 0x7f262407eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:27:08.936 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:08.928+0000 7f26294a5700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26240a0700 con 0x7f262409f2d0 2019-12-05T05:27:08.936 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:08.928+0000 7f26294a5700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26240a0af0 con 0x7f262407e710 2019-12-05T05:27:08.936 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:08.928+0000 7f26294a5700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f262408e5a0 con 0x7f26240a7730 2019-12-05T05:27:08.937 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:08.928+0000 7f2623fff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f262409f2d0 0x7f262409f700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:34014/0 (socket says 172.21.15.42:34014) 2019-12-05T05:27:08.937 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:08.928+0000 7f2623fff700 1 -- 172.21.15.42:0/531342808 learned_addr learned my addr 172.21.15.42:0/531342808 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:27:08.937 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:08.928+0000 7f26237fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f26240a7730 0x7f262408df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:27:08.937 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:08.928+0000 7f2622ffd700 1 --2- 172.21.15.42:0/531342808 >> v2:172.21.15.40:3300/0 conn(0x7f262407e710 0x7f262407eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:27:08.937 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:08.928+0000 7f26227fc700 1 -- 172.21.15.42:0/531342808 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4017901893 0 0) 0x7f26240a0700 con 0x7f262409f2d0 2019-12-05T05:27:08.937 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:08.928+0000 7f26227fc700 1 -- 172.21.15.42:0/531342808 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26100030b0 con 0x7f262409f2d0 2019-12-05T05:27:08.937 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:08.928+0000 7f2622ffd700 1 -- 172.21.15.42:0/531342808 >> v2:172.21.15.42:3300/0 conn(0x7f26240a7730 msgr2=0x7f262408df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:27:08.938 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:08.928+0000 7f2622ffd700 1 --2- 172.21.15.42:0/531342808 >> v2:172.21.15.42:3300/0 conn(0x7f26240a7730 0x7f262408df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:27:08.938 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:08.928+0000 7f2622ffd700 1 -- 172.21.15.42:0/531342808 >> v1:172.21.15.42:6789/0 conn(0x7f262409f2d0 legacy=0x7f262409f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:27:08.938 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:08.928+0000 7f2622ffd700 1 -- 172.21.15.42:0/531342808 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f262407f090 con 0x7f262407e710 2019-12-05T05:27:08.938 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:08.928+0000 7f2622ffd700 1 --2- 172.21.15.42:0/531342808 >> v2:172.21.15.40:3300/0 conn(0x7f262407e710 0x7f262407eb40 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f2614002f30 tx=0x7f261400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:08.938 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:08.928+0000 7f2622ffd700 1 --2- 172.21.15.42:0/531342808 >> v2:172.21.15.40:3300/0 conn(0x7f262407e710 0x7f262407eb40 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f2614002f30 tx=0x7f261400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:08.938 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:08.928+0000 7f2622ffd700 1 --2- 172.21.15.42:0/531342808 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f262407e710 0x7f262407eb40 secure :-1 s=READY pgs=689 cs=0 l=1 rx=0x7f2614002f30 tx=0x7f261400fc70).ready entity=mon.0 client_cookie=6264dea08f162911 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:27:08.938 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:08.928+0000 7f2622ffd700 1 --2- 172.21.15.42:0/531342808 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f262407e710 0x7f262407eb40 secure :-1 s=THROTTLE_DONE pgs=689 cs=0 l=1 rx=0x7f2614002f30 tx=0x7f261400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:08.939 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:08.928+0000 7f26227fc700 1 -- 172.21.15.42:0/531342808 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f26140112b0 con 0x7f262407e710 2019-12-05T05:27:08.939 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:08.928+0000 7f2622ffd700 1 --2- 172.21.15.42:0/531342808 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f262407e710 0x7f262407eb40 secure :-1 s=THROTTLE_DONE pgs=689 cs=0 l=1 rx=0x7f2614002f30 tx=0x7f261400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:08.939 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:08.928+0000 7f2622ffd700 1 --2- 172.21.15.42:0/531342808 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f262407e710 0x7f262407eb40 secure :-1 s=THROTTLE_DONE pgs=689 cs=0 l=1 rx=0x7f2614002f30 tx=0x7f261400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:08.939 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:08.928+0000 7f26227fc700 1 -- 172.21.15.42:0/531342808 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2614008ad0 con 0x7f262407e710 2019-12-05T05:27:08.940 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:08.928+0000 7f26227fc700 1 -- 172.21.15.42:0/531342808 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2614011890 con 0x7f262407e710 2019-12-05T05:27:08.940 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:08.928+0000 7f26294a5700 1 -- 172.21.15.42:0/531342808 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f262407e710 msgr2=0x7f262407eb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:27:08.940 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:08.928+0000 7f26294a5700 1 --2- 172.21.15.42:0/531342808 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f262407e710 0x7f262407eb40 secure :-1 s=READY pgs=689 cs=0 l=1 rx=0x7f2614002f30 tx=0x7f261400fc70).stop 2019-12-05T05:27:08.940 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:08.928+0000 7f26294a5700 1 -- 172.21.15.42:0/531342808 shutdown_connections 2019-12-05T05:27:08.940 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:08.928+0000 7f26294a5700 1 --2- 172.21.15.42:0/531342808 >> v2:172.21.15.42:3300/0 conn(0x7f26240a7730 0x7f262408df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:27:08.940 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:08.928+0000 7f26294a5700 1 --2- 172.21.15.42:0/531342808 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f262407e710 0x7f262407eb40 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:27:08.940 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:08.928+0000 7f26294a5700 1 -- 172.21.15.42:0/531342808 shutdown_connections 2019-12-05T05:27:08.941 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:08.928+0000 7f26294a5700 1 -- 172.21.15.42:0/531342808 wait complete. 2019-12-05T05:27:08.941 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:08.928+0000 7f26294a5700 1 -- 172.21.15.42:0/531342808 >> 172.21.15.42:0/531342808 conn(0x7f26240aabd0 msgr2=0x7f26240a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:27:08.958 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:08.932+0000 7f26294a5700 1 Processor -- start 2019-12-05T05:27:08.958 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:08.932+0000 7f26294a5700 1 -- start start 2019-12-05T05:27:08.959 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:08.932+0000 7f26294a5700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f262407d170 0x7f2624138c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:27:08.959 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:08.932+0000 7f26294a5700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f262407d5a0 con 0x7f262411b710 2019-12-05T05:27:08.959 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:08.932+0000 7f26294a5700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26240a7160 con 0x7f262407cd40 2019-12-05T05:27:08.959 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:08.932+0000 7f26294a5700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26240a68a0 con 0x7f262407d170 2019-12-05T05:27:08.959 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:08.932+0000 7f2622ffd700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f262407cd40 0x7f26241353f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:34018/0 (socket says 172.21.15.42:34018) 2019-12-05T05:27:08.960 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:08.932+0000 7f2622ffd700 1 -- 172.21.15.42:0/169222707 learned_addr learned my addr 172.21.15.42:0/169222707 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:27:08.960 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:08.932+0000 7f2623fff700 1 --2- 172.21.15.42:0/169222707 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f262407d170 0x7f2624138c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:27:08.961 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:08.932+0000 7f261bfff700 1 -- 172.21.15.42:0/169222707 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2201109980 0 0) 0x7f26240a7160 con 0x7f262407cd40 2019-12-05T05:27:08.961 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:08.932+0000 7f261bfff700 1 -- 172.21.15.42:0/169222707 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26080030b0 con 0x7f262407cd40 2019-12-05T05:27:08.961 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:08.932+0000 7f261bfff700 1 -- 172.21.15.42:0/169222707 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3336861917 0 0) 0x7f260c001ec0 con 0x7f262411b710 2019-12-05T05:27:08.962 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:08.932+0000 7f261bfff700 1 -- 172.21.15.42:0/169222707 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26240a7160 con 0x7f262411b710 2019-12-05T05:27:08.962 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:08.932+0000 7f261bfff700 1 -- 172.21.15.42:0/169222707 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 3842219935 0 0) 0x7f26080030b0 con 0x7f262407cd40 2019-12-05T05:27:08.962 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:08.932+0000 7f261bfff700 1 -- 172.21.15.42:0/169222707 --> v1:172.21.15.42:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f260c001ec0 con 0x7f262407cd40 2019-12-05T05:27:08.962 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:08.932+0000 7f261bfff700 1 -- 172.21.15.42:0/169222707 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7f26140112b0 con 0x7f262407cd40 2019-12-05T05:27:08.963 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:08.932+0000 7f261bfff700 1 -- 172.21.15.42:0/169222707 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f262407d170 msgr2=0x7f2624138c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:27:08.964 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:08.932+0000 7f261bfff700 1 --2- 172.21.15.42:0/169222707 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f262407d170 0x7f2624138c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:27:08.972 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:08.932+0000 7f261bfff700 1 -- 172.21.15.42:0/169222707 >> v1:172.21.15.42:6789/0 conn(0x7f262407cd40 legacy=0x7f26241353f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:27:08.972 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:08.932+0000 7f261bfff700 1 -- 172.21.15.42:0/169222707 >> v1:172.21.15.40:6789/0 conn(0x7f262411b710 legacy=0x7f262407bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:27:08.972 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:08.932+0000 7f261bfff700 1 --2- 172.21.15.42:0/169222707 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2608003ab0 0x7f2608004420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:27:08.973 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:08.932+0000 7f261bfff700 1 --2- 172.21.15.42:0/169222707 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2608003ee0 0x7f2608004970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:27:08.973 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:08.932+0000 7f261bfff700 1 --2- 172.21.15.42:0/169222707 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2608004ec0 0x7f260800d880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:27:08.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:08.932+0000 7f261bfff700 1 -- 172.21.15.42:0/169222707 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26240a68a0 con 0x7f2608004ec0 2019-12-05T05:27:08.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:08.932+0000 7f261bfff700 1 -- 172.21.15.42:0/169222707 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2608009690 con 0x7f2608003ee0 2019-12-05T05:27:08.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:08.932+0000 7f261bfff700 1 -- 172.21.15.42:0/169222707 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26080098d0 con 0x7f2608003ab0 2019-12-05T05:27:08.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:08.932+0000 7f2622ffd700 1 --2- 172.21.15.42:0/169222707 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2608003ee0 0x7f2608004970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:27:08.976 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:08.932+0000 7f2623fff700 1 --2- 172.21.15.42:0/169222707 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2608004ec0 0x7f260800d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:27:08.976 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:08.932+0000 7f26237fe700 1 --2- 172.21.15.42:0/169222707 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2608003ab0 0x7f2608004420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:27:08.976 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:08.936+0000 7f2623fff700 1 -- 172.21.15.42:0/169222707 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2608003ab0 msgr2=0x7f2608004420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:27:08.976 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:08.936+0000 7f2623fff700 1 --2- 172.21.15.42:0/169222707 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2608003ab0 0x7f2608004420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:27:08.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:08.936+0000 7f2623fff700 1 -- 172.21.15.42:0/169222707 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2608003ee0 msgr2=0x7f2608004970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:27:08.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:08.936+0000 7f2623fff700 1 --2- 172.21.15.42:0/169222707 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2608003ee0 0x7f2608004970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:27:08.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:08.936+0000 7f2623fff700 1 -- 172.21.15.42:0/169222707 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f262407b660 con 0x7f2608004ec0 2019-12-05T05:27:08.978 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:08.936+0000 7f26294a5700 1 -- 172.21.15.42:0/169222707 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26240a6b40 con 0x7f2608004ec0 2019-12-05T05:27:08.978 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:08.936+0000 7f2622ffd700 1 --2- 172.21.15.42:0/169222707 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2608003ee0 0x7f2608004970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:27:08.978 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:08.936+0000 7f26294a5700 1 -- 172.21.15.42:0/169222707 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26240a4e50 con 0x7f2608004ec0 2019-12-05T05:27:08.978 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:08.936+0000 7f2623fff700 1 --2- 172.21.15.42:0/169222707 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2608004ec0 0x7f260800d880 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f261c004720 tx=0x7f261c010f10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:08.979 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:08.936+0000 7f2623fff700 1 --2- 172.21.15.42:0/169222707 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2608004ec0 0x7f260800d880 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f261c004720 tx=0x7f261c010f10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:08.979 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:08.936+0000 7f2623fff700 1 --2- 172.21.15.42:0/169222707 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2608004ec0 0x7f260800d880 secure :-1 s=READY pgs=691 cs=0 l=1 rx=0x7f261c004720 tx=0x7f261c010f10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:27:08.980 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:08.936+0000 7f2623fff700 1 --2- 172.21.15.42:0/169222707 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2608004ec0 0x7f260800d880 secure :-1 s=THROTTLE_DONE pgs=691 cs=0 l=1 rx=0x7f261c004720 tx=0x7f261c010f10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:08.980 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:08.936+0000 7f2623fff700 1 --2- 172.21.15.42:0/169222707 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2608004ec0 0x7f260800d880 secure :-1 s=THROTTLE_DONE pgs=691 cs=0 l=1 rx=0x7f261c004720 tx=0x7f261c010f10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:08.980 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:08.936+0000 7f261bfff700 1 -- 172.21.15.42:0/169222707 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f261c012c70 con 0x7f2608004ec0 2019-12-05T05:27:08.981 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:08.936+0000 7f2623fff700 1 --2- 172.21.15.42:0/169222707 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2608004ec0 0x7f260800d880 secure :-1 s=THROTTLE_DONE pgs=691 cs=0 l=1 rx=0x7f261c004720 tx=0x7f261c010f10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:08.982 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:08.936+0000 7f261bfff700 1 -- 172.21.15.42:0/169222707 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f261c012890 con 0x7f2608004ec0 2019-12-05T05:27:08.982 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:08.936+0000 7f261bfff700 1 -- 172.21.15.42:0/169222707 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f261c00cc10 con 0x7f2608004ec0 2019-12-05T05:27:08.982 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:08.936+0000 7f2623fff700 1 --2- 172.21.15.42:0/169222707 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2608004ec0 0x7f260800d880 secure :-1 s=THROTTLE_DONE pgs=691 cs=0 l=1 rx=0x7f261c004720 tx=0x7f261c010f10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:08.982 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:08.936+0000 7f261bfff700 1 -- 172.21.15.42:0/169222707 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f261c0047b0 con 0x7f2608004ec0 2019-12-05T05:27:08.982 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:08.936+0000 7f2623fff700 1 --2- 172.21.15.42:0/169222707 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2608004ec0 0x7f260800d880 secure :-1 s=THROTTLE_DONE pgs=691 cs=0 l=1 rx=0x7f261c004720 tx=0x7f261c010f10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:08.983 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:08.936+0000 7f261bfff700 1 --2- 172.21.15.42:0/169222707 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2608048200 0x7f260804a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:27:08.983 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:08.936+0000 7f261bfff700 1 -- 172.21.15.42:0/169222707 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7407+0+0 (secure 0 0 0) 0x7f261c04ad40 con 0x7f2608004ec0 2019-12-05T05:27:08.983 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:08.936+0000 7f26237fe700 1 --2- 172.21.15.42:0/169222707 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2608048200 0x7f260804a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:27:08.983 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:08.936+0000 7f26237fe700 1 --2- 172.21.15.42:0/169222707 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2608048200 0x7f260804a710 crc :-1 s=READY pgs=385 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:27:08.983 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:08.940+0000 7f26294a5700 1 -- 172.21.15.42:0/169222707 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25fc003200 con 0x7f2608004ec0 2019-12-05T05:27:08.983 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:08.948+0000 7f2623fff700 1 --2- 172.21.15.42:0/169222707 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2608004ec0 0x7f260800d880 secure :-1 s=THROTTLE_DONE pgs=691 cs=0 l=1 rx=0x7f261c004720 tx=0x7f261c010f10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:08.983 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:08.948+0000 7f261bfff700 1 -- 172.21.15.42:0/169222707 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f261c031dc0 con 0x7f2608004ec0 2019-12-05T05:27:09.118 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:09.112+0000 7f26294a5700 1 -- 172.21.15.42:0/169222707 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f25fc003630 con 0x7f2608004ec0 2019-12-05T05:27:09.122 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:09.116+0000 7f2623fff700 1 --2- 172.21.15.42:0/169222707 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2608004ec0 0x7f260800d880 secure :-1 s=THROTTLE_DONE pgs=691 cs=0 l=1 rx=0x7f261c004720 tx=0x7f261c010f10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:09.122 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:09.116+0000 7f261bfff700 1 -- 172.21.15.42:0/169222707 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 109 v109) v1 ==== 95+0+1774 (secure 0 0 0) 0x7f261c032020 con 0x7f2608004ec0 2019-12-05T05:27:09.123 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 109 2019-12-05T05:27:09.123 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:27:09.123 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":109,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":109,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:27:08.612859+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6546},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6546":{"gid":6546,"name":"b","rank":0,"incarnation":109,"state":"up:replay","state_seq":2,"addr":"172.21.15.40:6833/1149478696","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1149478696},{"type":"v1","addr":"172.21.15.40:6833","nonce":1149478696}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:27:09.124 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:09.116+0000 7f26294a5700 1 -- 172.21.15.42:0/169222707 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2608048200 msgr2=0x7f260804a710 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:27:09.125 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:09.116+0000 7f26294a5700 1 --2- 172.21.15.42:0/169222707 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2608048200 0x7f260804a710 crc :-1 s=READY pgs=385 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:27:09.125 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:09.116+0000 7f26294a5700 1 -- 172.21.15.42:0/169222707 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2608004ec0 msgr2=0x7f260800d880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:27:09.125 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:09.116+0000 7f26294a5700 1 --2- 172.21.15.42:0/169222707 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2608004ec0 0x7f260800d880 secure :-1 s=READY pgs=691 cs=0 l=1 rx=0x7f261c004720 tx=0x7f261c010f10).stop 2019-12-05T05:27:09.125 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:09.116+0000 7f2623fff700 1 -- 172.21.15.42:0/169222707 reap_dead start 2019-12-05T05:27:09.126 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:09.116+0000 7f2623fff700 1 -- 172.21.15.42:0/169222707 reap_dead start 2019-12-05T05:27:09.126 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:09.116+0000 7f26294a5700 1 -- 172.21.15.42:0/169222707 shutdown_connections 2019-12-05T05:27:09.126 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:09.120+0000 7f26294a5700 1 -- 172.21.15.42:0/169222707 shutdown_connections 2019-12-05T05:27:09.126 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:09.120+0000 7f26294a5700 1 -- 172.21.15.42:0/169222707 wait complete. 2019-12-05T05:27:09.126 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:09.120+0000 7f26294a5700 1 -- 172.21.15.42:0/169222707 >> 172.21.15.42:0/169222707 conn(0x7f262409f210 msgr2=0x7f26240596f0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:27:09.133 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a down, removed from mdsmap 2019-12-05T05:27:09.134 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2019-12-05T05:27:09.134 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:27:09.238 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:09.224+0000 7f818fab1700 1 Processor -- start 2019-12-05T05:27:09.238 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:09.224+0000 7f818fab1700 1 -- start start 2019-12-05T05:27:09.238 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:09.228+0000 7f818fab1700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f818808df80 0x7f818809f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:27:09.238 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:09.228+0000 7f818fab1700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f818809f830 0x7f818814efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:27:09.239 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:09.228+0000 7f818fab1700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f818809fc60 con 0x7f81880a7740 2019-12-05T05:27:09.239 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:09.228+0000 7f818fab1700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81880a0b00 con 0x7f818808df80 2019-12-05T05:27:09.239 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:09.228+0000 7f818fab1700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f818808e5b0 con 0x7f818809f830 2019-12-05T05:27:09.239 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:09.228+0000 7f818d84d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f818808df80 0x7f818809f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:27:09.239 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:09.228+0000 7f818e04e700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f81880a7740 0x7f818807e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:34032/0 (socket says 172.21.15.42:34032) 2019-12-05T05:27:09.239 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:09.228+0000 7f818e04e700 1 -- 172.21.15.42:0/1612110103 learned_addr learned my addr 172.21.15.42:0/1612110103 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:27:09.239 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:09.228+0000 7f818d84d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f818808df80 0x7f818809f2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:33992/0 (socket says 172.21.15.42:33992) 2019-12-05T05:27:09.240 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:09.228+0000 7f818e84f700 1 --2- 172.21.15.42:0/1612110103 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f818809f830 0x7f818814efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:27:09.240 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:09.228+0000 7f818d84d700 1 -- 172.21.15.42:0/1612110103 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f818809f830 msgr2=0x7f818814efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:27:09.240 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:09.228+0000 7f818d84d700 1 --2- 172.21.15.42:0/1612110103 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f818809f830 0x7f818814efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:27:09.240 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:09.228+0000 7f818d04c700 1 -- 172.21.15.42:0/1612110103 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 696469473 0 0) 0x7f818809fc60 con 0x7f81880a7740 2019-12-05T05:27:09.241 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:09.228+0000 7f818d84d700 1 -- 172.21.15.42:0/1612110103 >> v1:172.21.15.42:6789/0 conn(0x7f81880a7740 legacy=0x7f818807e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:27:09.242 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:09.228+0000 7f818d84d700 1 -- 172.21.15.42:0/1612110103 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81880a0710 con 0x7f818808df80 2019-12-05T05:27:09.242 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:09.228+0000 7f818d84d700 1 --2- 172.21.15.42:0/1612110103 >> v2:172.21.15.42:3300/0 conn(0x7f818808df80 0x7f818809f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f817c008790 tx=0x7f817c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:09.242 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:09.228+0000 7f818d84d700 1 --2- 172.21.15.42:0/1612110103 >> v2:172.21.15.42:3300/0 conn(0x7f818808df80 0x7f818809f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f817c008790 tx=0x7f817c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:09.242 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:09.228+0000 7f818d84d700 1 --2- 172.21.15.42:0/1612110103 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f818808df80 0x7f818809f2e0 secure :-1 s=READY pgs=878 cs=0 l=1 rx=0x7f817c008790 tx=0x7f817c00fc70).ready entity=mon.1 client_cookie=a9161461acd28350 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:27:09.242 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:09.228+0000 7f818d84d700 1 --2- 172.21.15.42:0/1612110103 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f818808df80 0x7f818809f2e0 secure :-1 s=THROTTLE_DONE pgs=878 cs=0 l=1 rx=0x7f817c008790 tx=0x7f817c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:09.242 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:09.228+0000 7f818d04c700 1 -- 172.21.15.42:0/1612110103 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f817c008bb0 con 0x7f818808df80 2019-12-05T05:27:09.243 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:09.228+0000 7f818d84d700 1 --2- 172.21.15.42:0/1612110103 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f818808df80 0x7f818809f2e0 secure :-1 s=THROTTLE_DONE pgs=878 cs=0 l=1 rx=0x7f817c008790 tx=0x7f817c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:09.243 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:09.228+0000 7f818d84d700 1 --2- 172.21.15.42:0/1612110103 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f818808df80 0x7f818809f2e0 secure :-1 s=THROTTLE_DONE pgs=878 cs=0 l=1 rx=0x7f817c008790 tx=0x7f817c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:09.243 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:09.228+0000 7f818d04c700 1 -- 172.21.15.42:0/1612110103 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f817c0087d0 con 0x7f818808df80 2019-12-05T05:27:09.243 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:09.228+0000 7f818d04c700 1 -- 172.21.15.42:0/1612110103 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f817c011860 con 0x7f818808df80 2019-12-05T05:27:09.243 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:09.228+0000 7f818fab1700 1 -- 172.21.15.42:0/1612110103 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f818808df80 msgr2=0x7f818809f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:27:09.243 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:09.228+0000 7f818fab1700 1 --2- 172.21.15.42:0/1612110103 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f818808df80 0x7f818809f2e0 secure :-1 s=READY pgs=878 cs=0 l=1 rx=0x7f817c008790 tx=0x7f817c00fc70).stop 2019-12-05T05:27:09.243 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:09.228+0000 7f818fab1700 1 -- 172.21.15.42:0/1612110103 shutdown_connections 2019-12-05T05:27:09.244 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:09.228+0000 7f818fab1700 1 --2- 172.21.15.42:0/1612110103 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f818809f830 0x7f818814efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:27:09.244 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:09.228+0000 7f818fab1700 1 --2- 172.21.15.42:0/1612110103 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f818808df80 0x7f818809f2e0 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:27:09.245 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:09.228+0000 7f818fab1700 1 -- 172.21.15.42:0/1612110103 shutdown_connections 2019-12-05T05:27:09.245 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:09.228+0000 7f818fab1700 1 -- 172.21.15.42:0/1612110103 wait complete. 2019-12-05T05:27:09.245 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:09.228+0000 7f818fab1700 1 -- 172.21.15.42:0/1612110103 >> 172.21.15.42:0/1612110103 conn(0x7f81880aabe0 msgr2=0x7f81880a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:27:09.245 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:09.232+0000 7f818fab1700 1 Processor -- start 2019-12-05T05:27:09.245 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:09.232+0000 7f818fab1700 1 -- start start 2019-12-05T05:27:09.245 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:09.232+0000 7f818fab1700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f818811dc70 0x7f8188142950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:27:09.245 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:09.232+0000 7f818fab1700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f8188142ea0 0x7f8188132510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:27:09.246 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:09.232+0000 7f818fab1700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81881448d0 con 0x7f8188132a60 2019-12-05T05:27:09.246 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:09.232+0000 7f818fab1700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81881440f0 con 0x7f8188142ea0 2019-12-05T05:27:09.246 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:09.232+0000 7f818fab1700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8188144390 con 0x7f818811dc70 2019-12-05T05:27:09.246 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:09.232+0000 7f818d84d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f8188142ea0 0x7f8188132510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:27:09.246 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:09.232+0000 7f818e84f700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f8188132a60 0x7f81880a5bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:34040/0 (socket says 172.21.15.42:34040) 2019-12-05T05:27:09.246 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:09.232+0000 7f818e84f700 1 -- 172.21.15.42:0/2825490176 learned_addr learned my addr 172.21.15.42:0/2825490176 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:27:09.246 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:09.232+0000 7f818e04e700 1 --2- 172.21.15.42:0/2825490176 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f818811dc70 0x7f8188142950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:27:09.248 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:09.232+0000 7f818d84d700 1 -- 172.21.15.42:0/2825490176 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f818811dc70 msgr2=0x7f8188142950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:27:09.248 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:09.232+0000 7f818d84d700 1 --2- 172.21.15.42:0/2825490176 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f818811dc70 0x7f8188142950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:27:09.248 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:09.232+0000 7f8186ffd700 1 -- 172.21.15.42:0/2825490176 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1911434935 0 0) 0x7f81881448d0 con 0x7f8188132a60 2019-12-05T05:27:09.248 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:09.232+0000 7f818d84d700 1 -- 172.21.15.42:0/2825490176 >> v1:172.21.15.42:6789/0 conn(0x7f8188132a60 legacy=0x7f81880a5bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:27:09.248 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:09.232+0000 7f818d84d700 1 -- 172.21.15.42:0/2825490176 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8188143cd0 con 0x7f8188142ea0 2019-12-05T05:27:09.248 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:09.232+0000 7f818d84d700 1 --2- 172.21.15.42:0/2825490176 >> v2:172.21.15.42:3300/0 conn(0x7f8188142ea0 0x7f8188132510 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f81881442b0 tx=0x7f817c008a80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:09.249 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:09.232+0000 7f818d84d700 1 -- 172.21.15.42:0/2825490176 >> v2:172.21.15.42:3300/0 conn(0x7f8188142ea0 msgr2=0x7f8188132510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2019-12-05T05:27:09.249 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:09.232+0000 7f818d84d700 1 -- 172.21.15.42:0/2825490176 >> v2:172.21.15.42:3300/0 conn(0x7f8188142ea0 msgr2=0x7f8188132510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2019-12-05T05:27:09.249 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:09.232+0000 7f818d84d700 1 --2- 172.21.15.42:0/2825490176 >> v2:172.21.15.42:3300/0 conn(0x7f8188142ea0 0x7f8188132510 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f81881442b0 tx=0x7f817c008a80).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-12-05T05:27:09.249 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:09.232+0000 7f818d84d700 1 --2- 172.21.15.42:0/2825490176 >> v2:172.21.15.42:3300/0 conn(0x7f8188142ea0 0x7f8188132510 secure :-1 s=START_CONNECT pgs=0 cs=0 l=1 rx=0x7f81881442b0 tx=0x7f817c008a80)._fault waiting 0.200000 2019-12-05T05:27:09.440 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:09.436+0000 7f818d84d700 1 --2- 172.21.15.42:0/2825490176 >> v2:172.21.15.42:3300/0 conn(0x7f8188142ea0 0x7f8188132510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:27:09.441 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:09.436+0000 7f818d84d700 0 --2- 172.21.15.42:0/2825490176 >> v2:172.21.15.42:3300/0 conn(0x7f8188142ea0 0x7f8188132510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2019-12-05T05:27:09.441 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:09.436+0000 7f818d84d700 1 --2- 172.21.15.42:0/2825490176 >> v2:172.21.15.42:3300/0 conn(0x7f8188142ea0 0x7f8188132510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:27:09.441 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:09.436+0000 7f8186ffd700 1 -- 172.21.15.42:0/2825490176 >> v2:172.21.15.42:3300/0 conn(0x7f8188142ea0 msgr2=0x7f8188132510 unknown :-1 s=STATE_CLOSED l=1).mark_down 2019-12-05T05:27:09.441 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:09.436+0000 7f8186ffd700 1 --2- 172.21.15.42:0/2825490176 >> v2:172.21.15.42:3300/0 conn(0x7f8188142ea0 0x7f8188132510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:27:09.441 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:09.436+0000 7f8186ffd700 1 --2- 172.21.15.42:0/2825490176 >> v2:172.21.15.40:3300/0 conn(0x7f8168003050 0x7f8168005520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:27:09.441 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:09.436+0000 7f8186ffd700 1 -- 172.21.15.42:0/2825490176 --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81881448d0 con 0x7f8168005c10 2019-12-05T05:27:09.442 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:09.436+0000 7f8186ffd700 1 -- 172.21.15.42:0/2825490176 --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f816800aba0 con 0x7f8168003050 2019-12-05T05:27:09.442 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:09.436+0000 7f818e04e700 1 --2- 172.21.15.42:0/2825490176 >> v2:172.21.15.40:3300/0 conn(0x7f8168003050 0x7f8168005520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:27:09.444 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:09.436+0000 7f8186ffd700 1 -- 172.21.15.42:0/2825490176 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 716020369 0 0) 0x7f81881448d0 con 0x7f8168005c10 2019-12-05T05:27:09.444 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:09.436+0000 7f8186ffd700 1 -- 172.21.15.42:0/2825490176 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8168009be0 con 0x7f8168005c10 2019-12-05T05:27:09.445 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:09.436+0000 7f818e04e700 1 -- 172.21.15.42:0/2825490176 >> v1:172.21.15.40:6789/0 conn(0x7f8168005c10 legacy=0x7f8168008120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:27:09.445 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:09.436+0000 7f818e04e700 1 -- 172.21.15.42:0/2825490176 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f816800ae50 con 0x7f8168003050 2019-12-05T05:27:09.445 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:09.436+0000 7f818e04e700 1 --2- 172.21.15.42:0/2825490176 >> v2:172.21.15.40:3300/0 conn(0x7f8168003050 0x7f8168005520 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f81780043a0 tx=0x7f8178010bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:09.445 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:09.436+0000 7f818e04e700 1 --2- 172.21.15.42:0/2825490176 >> v2:172.21.15.40:3300/0 conn(0x7f8168003050 0x7f8168005520 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f81780043a0 tx=0x7f8178010bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:09.445 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:09.436+0000 7f818e04e700 1 --2- 172.21.15.42:0/2825490176 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8168003050 0x7f8168005520 secure :-1 s=READY pgs=693 cs=0 l=1 rx=0x7f81780043a0 tx=0x7f8178010bc0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:27:09.445 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:09.436+0000 7f818e04e700 1 --2- 172.21.15.42:0/2825490176 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8168003050 0x7f8168005520 secure :-1 s=THROTTLE_DONE pgs=693 cs=0 l=1 rx=0x7f81780043a0 tx=0x7f8178010bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:09.445 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:09.436+0000 7f818e04e700 1 --2- 172.21.15.42:0/2825490176 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8168003050 0x7f8168005520 secure :-1 s=THROTTLE_DONE pgs=693 cs=0 l=1 rx=0x7f81780043a0 tx=0x7f8178010bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:09.446 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:09.436+0000 7f8186ffd700 1 -- 172.21.15.42:0/2825490176 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8178012a00 con 0x7f8168003050 2019-12-05T05:27:09.446 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:09.436+0000 7f818e04e700 1 --2- 172.21.15.42:0/2825490176 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8168003050 0x7f8168005520 secure :-1 s=THROTTLE_DONE pgs=693 cs=0 l=1 rx=0x7f81780043a0 tx=0x7f8178010bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:09.446 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:09.436+0000 7f8186ffd700 1 -- 172.21.15.42:0/2825490176 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8178012620 con 0x7f8168003050 2019-12-05T05:27:09.446 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:09.436+0000 7f818fab1700 1 -- 172.21.15.42:0/2825490176 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f818811e0a0 con 0x7f8168003050 2019-12-05T05:27:09.446 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:09.436+0000 7f8186ffd700 1 -- 172.21.15.42:0/2825490176 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f817800c4b0 con 0x7f8168003050 2019-12-05T05:27:09.446 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:09.436+0000 7f818fab1700 1 -- 172.21.15.42:0/2825490176 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81880a63d0 con 0x7f8168003050 2019-12-05T05:27:09.447 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:09.436+0000 7f818e04e700 1 --2- 172.21.15.42:0/2825490176 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8168003050 0x7f8168005520 secure :-1 s=THROTTLE_DONE pgs=693 cs=0 l=1 rx=0x7f81780043a0 tx=0x7f8178010bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:09.447 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:09.440+0000 7f8186ffd700 1 -- 172.21.15.42:0/2825490176 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f817800c780 con 0x7f8168003050 2019-12-05T05:27:09.447 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:09.440+0000 7f818e04e700 1 --2- 172.21.15.42:0/2825490176 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8168003050 0x7f8168005520 secure :-1 s=THROTTLE_DONE pgs=693 cs=0 l=1 rx=0x7f81780043a0 tx=0x7f8178010bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:09.448 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:09.440+0000 7f8186ffd700 1 --2- 172.21.15.42:0/2825490176 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8168042180 0x7f8168044690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:27:09.448 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:09.440+0000 7f8186ffd700 1 -- 172.21.15.42:0/2825490176 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7407+0+0 (secure 0 0 0) 0x7f817804b020 con 0x7f8168003050 2019-12-05T05:27:09.448 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:09.440+0000 7f818d84d700 1 --2- 172.21.15.42:0/2825490176 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8168042180 0x7f8168044690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:27:09.449 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:09.440+0000 7f818d84d700 1 --2- 172.21.15.42:0/2825490176 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8168042180 0x7f8168044690 crc :-1 s=READY pgs=386 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:27:09.449 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:09.440+0000 7f818fab1700 1 -- 172.21.15.42:0/2825490176 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f818805d110 con 0x7f8168003050 2019-12-05T05:27:09.454 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:09.448+0000 7f818e04e700 1 --2- 172.21.15.42:0/2825490176 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8168003050 0x7f8168005520 secure :-1 s=THROTTLE_DONE pgs=693 cs=0 l=1 rx=0x7f81780043a0 tx=0x7f8178010bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:09.454 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:09.448+0000 7f8186ffd700 1 -- 172.21.15.42:0/2825490176 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f81780320a0 con 0x7f8168003050 2019-12-05T05:27:09.621 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:09.608+0000 7f818fab1700 1 -- 172.21.15.42:0/2825490176 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f818805b8c0 con 0x7f8168003050 2019-12-05T05:27:09.621 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:09.612+0000 7f818e04e700 1 --2- 172.21.15.42:0/2825490176 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8168003050 0x7f8168005520 secure :-1 s=THROTTLE_DONE pgs=693 cs=0 l=1 rx=0x7f81780043a0 tx=0x7f8178010bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:09.621 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:09.612+0000 7f8186ffd700 1 -- 172.21.15.42:0/2825490176 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 109 v109) v1 ==== 95+0+1774 (secure 0 0 0) 0x7f8178032300 con 0x7f8168003050 2019-12-05T05:27:09.621 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 109 2019-12-05T05:27:09.622 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:09.612+0000 7f818fab1700 1 -- 172.21.15.42:0/2825490176 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8168042180 msgr2=0x7f8168044690 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:27:09.622 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:09.612+0000 7f818fab1700 1 --2- 172.21.15.42:0/2825490176 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8168042180 0x7f8168044690 crc :-1 s=READY pgs=386 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:27:09.622 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:09.612+0000 7f818fab1700 1 -- 172.21.15.42:0/2825490176 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8168003050 msgr2=0x7f8168005520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:27:09.622 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:09.612+0000 7f818e84f700 1 -- 172.21.15.42:0/2825490176 reap_dead start 2019-12-05T05:27:09.622 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:09.612+0000 7f818fab1700 1 --2- 172.21.15.42:0/2825490176 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8168003050 0x7f8168005520 secure :-1 s=READY pgs=693 cs=0 l=1 rx=0x7f81780043a0 tx=0x7f8178010bc0).stop 2019-12-05T05:27:09.622 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:09.612+0000 7f818fab1700 1 -- 172.21.15.42:0/2825490176 shutdown_connections 2019-12-05T05:27:09.622 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:09.612+0000 7f818fab1700 1 --2- 172.21.15.42:0/2825490176 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8168003050 0x7f8168005520 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:27:09.623 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:09.612+0000 7f818fab1700 1 -- 172.21.15.42:0/2825490176 shutdown_connections 2019-12-05T05:27:09.623 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:09.612+0000 7f818fab1700 1 -- 172.21.15.42:0/2825490176 wait complete. 2019-12-05T05:27:09.623 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:09.612+0000 7f818fab1700 1 -- 172.21.15.42:0/2825490176 >> 172.21.15.42:0/2825490176 conn(0x7f818808fde0 msgr2=0x7f81880a9790 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:27:09.623 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:27:09.623 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":109,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":109,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:27:08.612859+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6546},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6546":{"gid":6546,"name":"b","rank":0,"incarnation":109,"state":"up:replay","state_seq":2,"addr":"172.21.15.40:6833/1149478696","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1149478696},{"type":"v1","addr":"172.21.15.40:6833","nonce":1149478696}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:27:09.629 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 109, "feature_flags": { "enable_multiple": false, "ever_enabled_multiple": false }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2019-12-05T05:03:35.604738+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 109, "ever_allowed_features": 2, "explicitly_allowed_features": 2, "failed": [], "flags": 18, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_6546": { "addr": "172.21.15.40:6833/1149478696", "addrs": { "addrvec": [ { "addr": "172.21.15.40:6832", "nonce": 1149478696, "type": "v2" }, { "addr": "172.21.15.40:6833", "nonce": 1149478696, "type": "v1" } ] }, "export_targets": [], "features": 4540138292836696063, "flags": 0, "gid": 6546, "incarnation": 109, "name": "b", "rank": 0, "state": "up:replay", "state_seq": 2 } }, "last_failure": 0, "last_failure_osd_epoch": 65, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 2, "min_compat_client": "0 (unknown)", "modified": "2019-12-05T05:27:08.612859+0000", "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6546 } } } ], "standbys": [] } 2019-12-05T05:27:11.631 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:27:11.740 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:11.728+0000 7f9d288e0700 1 Processor -- start 2019-12-05T05:27:11.740 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:11.728+0000 7f9d288e0700 1 -- start start 2019-12-05T05:27:11.740 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:11.728+0000 7f9d288e0700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f9d240a7730 0x7f9d2408df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:27:11.741 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:11.728+0000 7f9d288e0700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d240a0700 con 0x7f9d2409fad0 2019-12-05T05:27:11.741 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:11.728+0000 7f9d288e0700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d240a9b90 con 0x7f9d2407e710 2019-12-05T05:27:11.741 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:11.728+0000 7f9d288e0700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d240a0af0 con 0x7f9d240a7730 2019-12-05T05:27:11.741 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:11.728+0000 7f9d2259c700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f9d2407e710 0x7f9d2409f2d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:34052/0 (socket says 172.21.15.42:34052) 2019-12-05T05:27:11.741 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:11.728+0000 7f9d2259c700 1 -- 172.21.15.42:0/4075263315 learned_addr learned my addr 172.21.15.42:0/4075263315 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:27:11.741 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:11.728+0000 7f9d22d9d700 1 --2- 172.21.15.42:0/4075263315 >> v2:172.21.15.40:3300/0 conn(0x7f9d240a7730 0x7f9d2408df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:27:11.742 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:11.732+0000 7f9d21d9b700 1 -- 172.21.15.42:0/4075263315 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3417956283 0 0) 0x7f9d240a9b90 con 0x7f9d2407e710 2019-12-05T05:27:11.742 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:11.732+0000 7f9d22d9d700 1 -- 172.21.15.42:0/4075263315 >> v1:172.21.15.42:6789/0 conn(0x7f9d2407e710 legacy=0x7f9d2409f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:27:11.742 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:11.732+0000 7f9d22d9d700 1 -- 172.21.15.42:0/4075263315 >> v1:172.21.15.40:6789/0 conn(0x7f9d2409fad0 legacy=0x7f9d2414efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:27:11.742 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:11.732+0000 7f9d22d9d700 1 -- 172.21.15.42:0/4075263315 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d2407eb40 con 0x7f9d240a7730 2019-12-05T05:27:11.742 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:11.732+0000 7f9d22d9d700 1 --2- 172.21.15.42:0/4075263315 >> v2:172.21.15.40:3300/0 conn(0x7f9d240a7730 0x7f9d2408df70 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f9d14008b10 tx=0x7f9d1c001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:11.742 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:11.732+0000 7f9d22d9d700 1 --2- 172.21.15.42:0/4075263315 >> v2:172.21.15.40:3300/0 conn(0x7f9d240a7730 0x7f9d2408df70 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f9d14008b10 tx=0x7f9d1c001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:11.742 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:11.732+0000 7f9d22d9d700 1 --2- 172.21.15.42:0/4075263315 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9d240a7730 0x7f9d2408df70 secure :-1 s=READY pgs=695 cs=0 l=1 rx=0x7f9d14008b10 tx=0x7f9d1c001150).ready entity=mon.0 client_cookie=8e273d652bc4cb8d server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:27:11.743 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:11.732+0000 7f9d22d9d700 1 --2- 172.21.15.42:0/4075263315 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9d240a7730 0x7f9d2408df70 secure :-1 s=THROTTLE_DONE pgs=695 cs=0 l=1 rx=0x7f9d14008b10 tx=0x7f9d1c001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:11.743 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:11.732+0000 7f9d21d9b700 1 -- 172.21.15.42:0/4075263315 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9d14010870 con 0x7f9d240a7730 2019-12-05T05:27:11.743 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:11.732+0000 7f9d22d9d700 1 --2- 172.21.15.42:0/4075263315 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9d240a7730 0x7f9d2408df70 secure :-1 s=THROTTLE_DONE pgs=695 cs=0 l=1 rx=0x7f9d14008b10 tx=0x7f9d1c001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:11.743 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:11.732+0000 7f9d21d9b700 1 -- 172.21.15.42:0/4075263315 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d14008b50 con 0x7f9d240a7730 2019-12-05T05:27:11.743 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:11.732+0000 7f9d22d9d700 1 --2- 172.21.15.42:0/4075263315 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9d240a7730 0x7f9d2408df70 secure :-1 s=THROTTLE_DONE pgs=695 cs=0 l=1 rx=0x7f9d14008b10 tx=0x7f9d1c001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:11.743 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:11.732+0000 7f9d21d9b700 1 -- 172.21.15.42:0/4075263315 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9d140110c0 con 0x7f9d240a7730 2019-12-05T05:27:11.744 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:11.732+0000 7f9d288e0700 1 -- 172.21.15.42:0/4075263315 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9d240a7730 msgr2=0x7f9d2408df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:27:11.744 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:11.732+0000 7f9d288e0700 1 --2- 172.21.15.42:0/4075263315 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9d240a7730 0x7f9d2408df70 secure :-1 s=READY pgs=695 cs=0 l=1 rx=0x7f9d14008b10 tx=0x7f9d1c001150).stop 2019-12-05T05:27:11.744 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:11.732+0000 7f9d288e0700 1 -- 172.21.15.42:0/4075263315 shutdown_connections 2019-12-05T05:27:11.745 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:11.732+0000 7f9d288e0700 1 --2- 172.21.15.42:0/4075263315 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9d240a7730 0x7f9d2408df70 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:27:11.745 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:11.732+0000 7f9d288e0700 1 -- 172.21.15.42:0/4075263315 shutdown_connections 2019-12-05T05:27:11.745 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:11.732+0000 7f9d288e0700 1 -- 172.21.15.42:0/4075263315 wait complete. 2019-12-05T05:27:11.745 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:11.732+0000 7f9d288e0700 1 -- 172.21.15.42:0/4075263315 >> 172.21.15.42:0/4075263315 conn(0x7f9d240aabd0 msgr2=0x7f9d240a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:27:11.748 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:11.736+0000 7f9d288e0700 1 Processor -- start 2019-12-05T05:27:11.749 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:11.736+0000 7f9d288e0700 1 -- start start 2019-12-05T05:27:11.749 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:11.736+0000 7f9d288e0700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9d24119630 0x7f9d241426d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:27:11.749 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:11.736+0000 7f9d288e0700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f9d2411b770 0x7f9d2407dab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:27:11.749 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:11.736+0000 7f9d288e0700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d24142c20 con 0x7f9d2412ce40 2019-12-05T05:27:11.749 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:11.736+0000 7f9d288e0700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d24142ef0 con 0x7f9d2411b770 2019-12-05T05:27:11.749 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:11.736+0000 7f9d288e0700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d24118760 con 0x7f9d24119630 2019-12-05T05:27:11.750 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:11.736+0000 7f9d2359e700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f9d2412ce40 0x7f9d24118a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:34062/0 (socket says 172.21.15.42:34062) 2019-12-05T05:27:11.750 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:11.736+0000 7f9d22d9d700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9d24119630 0x7f9d241426d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:27:11.750 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:11.736+0000 7f9d2359e700 1 -- 172.21.15.42:0/3969657115 learned_addr learned my addr 172.21.15.42:0/3969657115 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:27:11.750 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:11.736+0000 7f9d2259c700 1 --2- 172.21.15.42:0/3969657115 >> v2:172.21.15.42:3300/0 conn(0x7f9d2411b770 0x7f9d2407dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:27:11.750 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:11.740+0000 7f9d1b7fe700 1 -- 172.21.15.42:0/3969657115 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2690931719 0 0) 0x7f9d24142c20 con 0x7f9d2412ce40 2019-12-05T05:27:11.751 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:11.740+0000 7f9d1b7fe700 1 -- 172.21.15.42:0/3969657115 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9cfc0030b0 con 0x7f9d2412ce40 2019-12-05T05:27:11.751 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:11.740+0000 7f9d22d9d700 1 -- 172.21.15.42:0/3969657115 >> v2:172.21.15.42:3300/0 conn(0x7f9d2411b770 msgr2=0x7f9d2407dab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:27:11.751 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:11.740+0000 7f9d22d9d700 1 --2- 172.21.15.42:0/3969657115 >> v2:172.21.15.42:3300/0 conn(0x7f9d2411b770 0x7f9d2407dab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:27:11.751 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:11.740+0000 7f9d22d9d700 1 -- 172.21.15.42:0/3969657115 >> v1:172.21.15.42:6789/0 conn(0x7f9d2412ce40 legacy=0x7f9d24118a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:27:11.751 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:11.740+0000 7f9d22d9d700 1 -- 172.21.15.42:0/3969657115 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d2407e000 con 0x7f9d24119630 2019-12-05T05:27:11.751 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:11.740+0000 7f9d2259c700 1 --2- 172.21.15.42:0/3969657115 >> v2:172.21.15.42:3300/0 conn(0x7f9d2411b770 0x7f9d2407dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:27:11.752 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:11.740+0000 7f9d22d9d700 1 --2- 172.21.15.42:0/3969657115 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9d24119630 0x7f9d241426d0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f9d14008b10 tx=0x7f9d14010ce0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:11.752 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:11.740+0000 7f9d22d9d700 1 --2- 172.21.15.42:0/3969657115 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9d24119630 0x7f9d241426d0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f9d14008b10 tx=0x7f9d14010ce0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:11.752 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:11.740+0000 7f9d22d9d700 1 --2- 172.21.15.42:0/3969657115 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9d24119630 0x7f9d241426d0 secure :-1 s=READY pgs=180 cs=0 l=1 rx=0x7f9d14008b10 tx=0x7f9d14010ce0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:27:11.752 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:11.740+0000 7f9d22d9d700 1 --2- 172.21.15.42:0/3969657115 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9d24119630 0x7f9d241426d0 secure :-1 s=THROTTLE_DONE pgs=180 cs=0 l=1 rx=0x7f9d14008b10 tx=0x7f9d14010ce0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:11.752 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:11.740+0000 7f9d1b7fe700 1 -- 172.21.15.42:0/3969657115 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9d14005ab0 con 0x7f9d24119630 2019-12-05T05:27:11.752 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:11.740+0000 7f9d22d9d700 1 --2- 172.21.15.42:0/3969657115 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9d24119630 0x7f9d241426d0 secure :-1 s=THROTTLE_DONE pgs=180 cs=0 l=1 rx=0x7f9d14008b10 tx=0x7f9d14010ce0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:11.752 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:11.740+0000 7f9d22d9d700 1 --2- 172.21.15.42:0/3969657115 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9d24119630 0x7f9d241426d0 secure :-1 s=THROTTLE_DONE pgs=180 cs=0 l=1 rx=0x7f9d14008b10 tx=0x7f9d14010ce0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:11.753 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:11.740+0000 7f9d1b7fe700 1 -- 172.21.15.42:0/3969657115 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d14010a70 con 0x7f9d24119630 2019-12-05T05:27:11.753 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:11.740+0000 7f9d288e0700 1 -- 172.21.15.42:0/3969657115 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d24137850 con 0x7f9d24119630 2019-12-05T05:27:11.753 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:11.740+0000 7f9d1b7fe700 1 -- 172.21.15.42:0/3969657115 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9d1400f680 con 0x7f9d24119630 2019-12-05T05:27:11.753 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:11.740+0000 7f9d288e0700 1 -- 172.21.15.42:0/3969657115 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d24137d80 con 0x7f9d24119630 2019-12-05T05:27:11.753 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:11.740+0000 7f9d22d9d700 1 --2- 172.21.15.42:0/3969657115 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9d24119630 0x7f9d241426d0 secure :-1 s=THROTTLE_DONE pgs=180 cs=0 l=1 rx=0x7f9d14008b10 tx=0x7f9d14010ce0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:11.753 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:11.740+0000 7f9d1b7fe700 1 -- 172.21.15.42:0/3969657115 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f9d14011360 con 0x7f9d24119630 2019-12-05T05:27:11.753 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:11.740+0000 7f9d22d9d700 1 --2- 172.21.15.42:0/3969657115 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9d24119630 0x7f9d241426d0 secure :-1 s=THROTTLE_DONE pgs=180 cs=0 l=1 rx=0x7f9d14008b10 tx=0x7f9d14010ce0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:11.754 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:11.744+0000 7f9d1b7fe700 1 --2- 172.21.15.42:0/3969657115 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9cfc03b210 0x7f9cfc03d720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:27:11.754 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:11.744+0000 7f9d1b7fe700 1 -- 172.21.15.42:0/3969657115 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7407+0+0 (secure 0 0 0) 0x7f9d1404b100 con 0x7f9d24119630 2019-12-05T05:27:11.754 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:11.744+0000 7f9d2259c700 1 --2- 172.21.15.42:0/3969657115 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9cfc03b210 0x7f9cfc03d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:27:11.759 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:11.744+0000 7f9d2259c700 1 --2- 172.21.15.42:0/3969657115 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9cfc03b210 0x7f9cfc03d720 crc :-1 s=READY pgs=387 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:27:11.760 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:11.744+0000 7f9d288e0700 1 -- 172.21.15.42:0/3969657115 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d2405d100 con 0x7f9d24119630 2019-12-05T05:27:11.761 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:11.752+0000 7f9d22d9d700 1 --2- 172.21.15.42:0/3969657115 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9d24119630 0x7f9d241426d0 secure :-1 s=THROTTLE_DONE pgs=180 cs=0 l=1 rx=0x7f9d14008b10 tx=0x7f9d14010ce0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:11.761 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:11.752+0000 7f9d1b7fe700 1 -- 172.21.15.42:0/3969657115 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f9d1400f990 con 0x7f9d24119630 2019-12-05T05:27:11.933 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:11.924+0000 7f9d288e0700 1 -- 172.21.15.42:0/3969657115 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9d2405b8b0 con 0x7f9d24119630 2019-12-05T05:27:11.933 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:11.924+0000 7f9d22d9d700 1 --2- 172.21.15.42:0/3969657115 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9d24119630 0x7f9d241426d0 secure :-1 s=THROTTLE_DONE pgs=180 cs=0 l=1 rx=0x7f9d14008b10 tx=0x7f9d14010ce0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:11.933 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:11.924+0000 7f9d1b7fe700 1 -- 172.21.15.42:0/3969657115 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 110 v110) v1 ==== 95+0+1778 (secure 0 0 0) 0x7f9d14011c60 con 0x7f9d24119630 2019-12-05T05:27:11.933 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 110 2019-12-05T05:27:11.933 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:27:11.934 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":110,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":110,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:27:11.308887+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6546},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6546":{"gid":6546,"name":"b","rank":0,"incarnation":109,"state":"up:reconnect","state_seq":23,"addr":"172.21.15.40:6833/1149478696","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1149478696},{"type":"v1","addr":"172.21.15.40:6833","nonce":1149478696}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:27:11.944 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:11.928+0000 7f9d288e0700 1 -- 172.21.15.42:0/3969657115 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9cfc03b210 msgr2=0x7f9cfc03d720 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:27:11.944 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:11.928+0000 7f9d288e0700 1 --2- 172.21.15.42:0/3969657115 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9cfc03b210 0x7f9cfc03d720 crc :-1 s=READY pgs=387 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:27:11.945 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:11.928+0000 7f9d288e0700 1 -- 172.21.15.42:0/3969657115 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9d24119630 msgr2=0x7f9d241426d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:27:11.945 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:11.928+0000 7f9d288e0700 1 --2- 172.21.15.42:0/3969657115 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9d24119630 0x7f9d241426d0 secure :-1 s=READY pgs=180 cs=0 l=1 rx=0x7f9d14008b10 tx=0x7f9d14010ce0).stop 2019-12-05T05:27:11.945 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:11.928+0000 7f9d288e0700 1 -- 172.21.15.42:0/3969657115 shutdown_connections 2019-12-05T05:27:11.945 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:11.928+0000 7f9d288e0700 1 --2- 172.21.15.42:0/3969657115 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9cfc03b210 0x7f9cfc03d720 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:27:11.945 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:11.928+0000 7f9d288e0700 1 --2- 172.21.15.42:0/3969657115 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9d24119630 0x7f9d241426d0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:27:11.945 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:11.928+0000 7f9d288e0700 1 --2- 172.21.15.42:0/3969657115 >> v2:172.21.15.42:3300/0 conn(0x7f9d2411b770 0x7f9d2407dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:27:11.945 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:11.928+0000 7f9d288e0700 1 -- 172.21.15.42:0/3969657115 shutdown_connections 2019-12-05T05:27:11.946 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:11.932+0000 7f9d288e0700 1 -- 172.21.15.42:0/3969657115 wait complete. 2019-12-05T05:27:11.946 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:11.932+0000 7f9d288e0700 1 -- 172.21.15.42:0/3969657115 >> 172.21.15.42:0/3969657115 conn(0x7f9d240a7730 msgr2=0x7f9d240596f0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:27:11.947 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2019-12-05T05:27:13.952 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:27:14.077 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:14.060+0000 7f1db33f9700 1 Processor -- start 2019-12-05T05:27:14.077 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:14.060+0000 7f1db33f9700 1 -- start start 2019-12-05T05:27:14.077 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:14.064+0000 7f1db33f9700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f1dac08df80 0x7f1dac09f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:27:14.077 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:14.064+0000 7f1db33f9700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1dac09fc60 con 0x7f1dac0a7740 2019-12-05T05:27:14.078 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:14.064+0000 7f1db33f9700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1dac0a0710 con 0x7f1dac09f830 2019-12-05T05:27:14.078 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:14.064+0000 7f1db33f9700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1dac0a0b00 con 0x7f1dac08df80 2019-12-05T05:27:14.078 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:14.064+0000 7f1db2197700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f1dac09f830 0x7f1dac14efe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:34078/0 (socket says 172.21.15.42:34078) 2019-12-05T05:27:14.078 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:14.064+0000 7f1db2197700 1 -- 172.21.15.42:0/2687536389 learned_addr learned my addr 172.21.15.42:0/2687536389 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:27:14.078 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:14.064+0000 7f1db1195700 1 --2- 172.21.15.42:0/2687536389 >> v2:172.21.15.40:3300/0 conn(0x7f1dac08df80 0x7f1dac09f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:27:14.078 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:14.064+0000 7f1db0994700 1 -- 172.21.15.42:0/2687536389 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4158510450 0 0) 0x7f1dac0a0710 con 0x7f1dac09f830 2019-12-05T05:27:14.078 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:14.064+0000 7f1db0994700 1 -- 172.21.15.42:0/2687536389 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d940030b0 con 0x7f1dac09f830 2019-12-05T05:27:14.079 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:14.064+0000 7f1db1195700 1 -- 172.21.15.42:0/2687536389 >> v1:172.21.15.42:6789/0 conn(0x7f1dac09f830 legacy=0x7f1dac14efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:27:14.079 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:14.064+0000 7f1db0994700 1 -- 172.21.15.42:0/2687536389 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3175323753 0 0) 0x7f1dac09fc60 con 0x7f1dac0a7740 2019-12-05T05:27:14.079 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:14.064+0000 7f1db1195700 1 -- 172.21.15.42:0/2687536389 >> v1:172.21.15.40:6789/0 conn(0x7f1dac0a7740 legacy=0x7f1dac07e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:27:14.079 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:14.064+0000 7f1db1195700 1 -- 172.21.15.42:0/2687536389 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dac07ef20 con 0x7f1dac08df80 2019-12-05T05:27:14.079 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:14.064+0000 7f1db1195700 1 --2- 172.21.15.42:0/2687536389 >> v2:172.21.15.40:3300/0 conn(0x7f1dac08df80 0x7f1dac09f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f1da0008790 tx=0x7f1da000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:14.079 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:14.064+0000 7f1db1195700 1 --2- 172.21.15.42:0/2687536389 >> v2:172.21.15.40:3300/0 conn(0x7f1dac08df80 0x7f1dac09f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f1da0008790 tx=0x7f1da000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:14.080 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:14.064+0000 7f1db1195700 1 --2- 172.21.15.42:0/2687536389 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1dac08df80 0x7f1dac09f2e0 secure :-1 s=READY pgs=697 cs=0 l=1 rx=0x7f1da0008790 tx=0x7f1da000fbf0).ready entity=mon.0 client_cookie=fceb1bbf6b2cd96c server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:27:14.080 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:14.064+0000 7f1db1195700 1 --2- 172.21.15.42:0/2687536389 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1dac08df80 0x7f1dac09f2e0 secure :-1 s=THROTTLE_DONE pgs=697 cs=0 l=1 rx=0x7f1da0008790 tx=0x7f1da000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:14.080 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:14.064+0000 7f1db0994700 1 -- 172.21.15.42:0/2687536389 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1da0011270 con 0x7f1dac08df80 2019-12-05T05:27:14.080 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:14.064+0000 7f1db1195700 1 --2- 172.21.15.42:0/2687536389 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1dac08df80 0x7f1dac09f2e0 secure :-1 s=THROTTLE_DONE pgs=697 cs=0 l=1 rx=0x7f1da0008790 tx=0x7f1da000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:14.080 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:14.064+0000 7f1db1195700 1 --2- 172.21.15.42:0/2687536389 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1dac08df80 0x7f1dac09f2e0 secure :-1 s=THROTTLE_DONE pgs=697 cs=0 l=1 rx=0x7f1da0008790 tx=0x7f1da000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:14.080 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:14.064+0000 7f1db0994700 1 -- 172.21.15.42:0/2687536389 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1da0008bc0 con 0x7f1dac08df80 2019-12-05T05:27:14.080 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:14.064+0000 7f1db0994700 1 -- 172.21.15.42:0/2687536389 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1da0011850 con 0x7f1dac08df80 2019-12-05T05:27:14.081 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:14.064+0000 7f1db33f9700 1 -- 172.21.15.42:0/2687536389 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1dac08df80 msgr2=0x7f1dac09f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:27:14.081 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:14.064+0000 7f1db33f9700 1 --2- 172.21.15.42:0/2687536389 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1dac08df80 0x7f1dac09f2e0 secure :-1 s=READY pgs=697 cs=0 l=1 rx=0x7f1da0008790 tx=0x7f1da000fbf0).stop 2019-12-05T05:27:14.081 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:14.068+0000 7f1db33f9700 1 -- 172.21.15.42:0/2687536389 shutdown_connections 2019-12-05T05:27:14.081 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:14.068+0000 7f1db33f9700 1 --2- 172.21.15.42:0/2687536389 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1dac08df80 0x7f1dac09f2e0 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:27:14.081 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:14.068+0000 7f1db33f9700 1 -- 172.21.15.42:0/2687536389 shutdown_connections 2019-12-05T05:27:14.081 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:14.068+0000 7f1db33f9700 1 -- 172.21.15.42:0/2687536389 wait complete. 2019-12-05T05:27:14.082 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:14.068+0000 7f1db33f9700 1 -- 172.21.15.42:0/2687536389 >> 172.21.15.42:0/2687536389 conn(0x7f1dac0aabe0 msgr2=0x7f1dac0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:27:14.084 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:14.068+0000 7f1db33f9700 1 Processor -- start 2019-12-05T05:27:14.084 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:14.068+0000 7f1db33f9700 1 -- start start 2019-12-05T05:27:14.084 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:14.068+0000 7f1db33f9700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f1dac119c70 0x7f1dac064cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:27:14.085 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:14.068+0000 7f1db33f9700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1dac0a3d70 0x7f1dac07d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:27:14.085 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:14.068+0000 7f1db33f9700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1dac07d900 con 0x7f1dac0636f0 2019-12-05T05:27:14.085 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:14.068+0000 7f1db33f9700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1dac0652c0 con 0x7f1dac119c70 2019-12-05T05:27:14.085 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:14.068+0000 7f1db33f9700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1dac065560 con 0x7f1dac0a3d70 2019-12-05T05:27:14.085 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:14.072+0000 7f1db1996700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f1dac119c70 0x7f1dac064cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:27:14.085 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:14.072+0000 7f1db1996700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f1dac119c70 0x7f1dac064cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:34042/0 (socket says 172.21.15.42:34042) 2019-12-05T05:27:14.085 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:14.072+0000 7f1db2197700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1dac0a3d70 0x7f1dac07d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:27:14.086 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:14.072+0000 7f1db1996700 1 -- 172.21.15.42:0/3886006118 learned_addr learned my addr 172.21.15.42:0/3886006118 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:27:14.086 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:14.072+0000 7f1db1996700 1 -- 172.21.15.42:0/3886006118 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1dac0a3d70 msgr2=0x7f1dac07d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:27:14.086 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:14.072+0000 7f1db1996700 1 --2- 172.21.15.42:0/3886006118 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1dac0a3d70 0x7f1dac07d3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:27:14.086 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:14.072+0000 7f1db1996700 1 -- 172.21.15.42:0/3886006118 >> v1:172.21.15.40:6789/0 conn(0x7f1dac0636f0 legacy=0x7f1dac0a3570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:27:14.086 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:14.072+0000 7f1db1996700 1 -- 172.21.15.42:0/3886006118 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dac063b20 con 0x7f1dac119c70 2019-12-05T05:27:14.086 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:14.072+0000 7f1db1996700 1 --2- 172.21.15.42:0/3886006118 >> v2:172.21.15.42:3300/0 conn(0x7f1dac119c70 0x7f1dac064cd0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f1dac065480 tx=0x7f1da4010d20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:14.086 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:14.072+0000 7f1db1996700 1 --2- 172.21.15.42:0/3886006118 >> v2:172.21.15.42:3300/0 conn(0x7f1dac119c70 0x7f1dac064cd0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f1dac065480 tx=0x7f1da4010d20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:14.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:14.072+0000 7f1db1996700 1 --2- 172.21.15.42:0/3886006118 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1dac119c70 0x7f1dac064cd0 secure :-1 s=READY pgs=884 cs=0 l=1 rx=0x7f1dac065480 tx=0x7f1da4010d20).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:27:14.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:14.072+0000 7f1db1996700 1 --2- 172.21.15.42:0/3886006118 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1dac119c70 0x7f1dac064cd0 secure :-1 s=THROTTLE_DONE pgs=884 cs=0 l=1 rx=0x7f1dac065480 tx=0x7f1da4010d20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:14.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:14.072+0000 7f1db1996700 1 --2- 172.21.15.42:0/3886006118 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1dac119c70 0x7f1dac064cd0 secure :-1 s=THROTTLE_DONE pgs=884 cs=0 l=1 rx=0x7f1dac065480 tx=0x7f1da4010d20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:14.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:14.072+0000 7f1daa7fc700 1 -- 172.21.15.42:0/3886006118 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1da4010550 con 0x7f1dac119c70 2019-12-05T05:27:14.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:14.072+0000 7f1db1996700 1 --2- 172.21.15.42:0/3886006118 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1dac119c70 0x7f1dac064cd0 secure :-1 s=THROTTLE_DONE pgs=884 cs=0 l=1 rx=0x7f1dac065480 tx=0x7f1da4010d20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:14.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:14.072+0000 7f1daa7fc700 1 -- 172.21.15.42:0/3886006118 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1da4010170 con 0x7f1dac119c70 2019-12-05T05:27:14.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:14.072+0000 7f1db33f9700 1 -- 172.21.15.42:0/3886006118 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1dac07db30 con 0x7f1dac119c70 2019-12-05T05:27:14.088 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:14.072+0000 7f1daa7fc700 1 -- 172.21.15.42:0/3886006118 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1da400cb20 con 0x7f1dac119c70 2019-12-05T05:27:14.089 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:14.072+0000 7f1db33f9700 1 -- 172.21.15.42:0/3886006118 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1dac07dec0 con 0x7f1dac119c70 2019-12-05T05:27:14.089 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:14.072+0000 7f1db1996700 1 --2- 172.21.15.42:0/3886006118 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1dac119c70 0x7f1dac064cd0 secure :-1 s=THROTTLE_DONE pgs=884 cs=0 l=1 rx=0x7f1dac065480 tx=0x7f1da4010d20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:14.089 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:14.072+0000 7f1daa7fc700 1 -- 172.21.15.42:0/3886006118 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f1da40066e0 con 0x7f1dac119c70 2019-12-05T05:27:14.089 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:14.072+0000 7f1db1996700 1 --2- 172.21.15.42:0/3886006118 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1dac119c70 0x7f1dac064cd0 secure :-1 s=THROTTLE_DONE pgs=884 cs=0 l=1 rx=0x7f1dac065480 tx=0x7f1da4010d20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:14.090 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:14.072+0000 7f1daa7fc700 1 --2- 172.21.15.42:0/3886006118 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f1d8c03ad20 0x7f1d8c03d230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:27:14.090 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:14.072+0000 7f1daa7fc700 1 -- 172.21.15.42:0/3886006118 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7407+0+0 (secure 0 0 0) 0x7f1da404b2f0 con 0x7f1dac119c70 2019-12-05T05:27:14.090 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:14.072+0000 7f1db1195700 1 --2- 172.21.15.42:0/3886006118 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f1d8c03ad20 0x7f1d8c03d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:27:14.090 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:14.076+0000 7f1db1195700 1 --2- 172.21.15.42:0/3886006118 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f1d8c03ad20 0x7f1d8c03d230 crc :-1 s=READY pgs=388 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:27:14.090 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:14.076+0000 7f1db33f9700 1 -- 172.21.15.42:0/3886006118 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1dac05d110 con 0x7f1dac119c70 2019-12-05T05:27:14.093 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:14.084+0000 7f1db1996700 1 --2- 172.21.15.42:0/3886006118 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1dac119c70 0x7f1dac064cd0 secure :-1 s=THROTTLE_DONE pgs=884 cs=0 l=1 rx=0x7f1dac065480 tx=0x7f1da4010d20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:14.093 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:14.084+0000 7f1daa7fc700 1 -- 172.21.15.42:0/3886006118 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f1da4032370 con 0x7f1dac119c70 2019-12-05T05:27:14.253 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:14.248+0000 7f1db33f9700 1 -- 172.21.15.42:0/3886006118 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1dac05b8c0 con 0x7f1dac119c70 2019-12-05T05:27:14.254 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:14.248+0000 7f1db1996700 1 --2- 172.21.15.42:0/3886006118 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1dac119c70 0x7f1dac064cd0 secure :-1 s=THROTTLE_DONE pgs=884 cs=0 l=1 rx=0x7f1dac065480 tx=0x7f1da4010d20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:14.254 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:14.248+0000 7f1daa7fc700 1 -- 172.21.15.42:0/3886006118 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 112 v112) v1 ==== 95+0+1775 (secure 0 0 0) 0x7f1da40325d0 con 0x7f1dac119c70 2019-12-05T05:27:14.254 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 112 2019-12-05T05:27:14.255 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:27:14.255 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":112,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":112,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:27:13.322209+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6546},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6546":{"gid":6546,"name":"b","rank":0,"incarnation":109,"state":"up:active","state_seq":25,"addr":"172.21.15.40:6833/1149478696","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1149478696},{"type":"v1","addr":"172.21.15.40:6833","nonce":1149478696}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:27:14.257 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:14.252+0000 7f1db33f9700 1 -- 172.21.15.42:0/3886006118 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f1d8c03ad20 msgr2=0x7f1d8c03d230 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:27:14.257 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:14.252+0000 7f1db33f9700 1 --2- 172.21.15.42:0/3886006118 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f1d8c03ad20 0x7f1d8c03d230 crc :-1 s=READY pgs=388 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:27:14.257 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:14.252+0000 7f1db33f9700 1 -- 172.21.15.42:0/3886006118 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1dac119c70 msgr2=0x7f1dac064cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:27:14.257 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:14.252+0000 7f1db33f9700 1 --2- 172.21.15.42:0/3886006118 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1dac119c70 0x7f1dac064cd0 secure :-1 s=READY pgs=884 cs=0 l=1 rx=0x7f1dac065480 tx=0x7f1da4010d20).stop 2019-12-05T05:27:14.258 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:14.252+0000 7f1db33f9700 1 -- 172.21.15.42:0/3886006118 shutdown_connections 2019-12-05T05:27:14.258 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:14.252+0000 7f1db33f9700 1 --2- 172.21.15.42:0/3886006118 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f1d8c03ad20 0x7f1d8c03d230 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:27:14.258 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:14.252+0000 7f1db33f9700 1 --2- 172.21.15.42:0/3886006118 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1dac0a3d70 0x7f1dac07d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:27:14.258 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:14.252+0000 7f1db33f9700 1 --2- 172.21.15.42:0/3886006118 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1dac119c70 0x7f1dac064cd0 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:27:14.258 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:14.252+0000 7f1db33f9700 1 -- 172.21.15.42:0/3886006118 shutdown_connections 2019-12-05T05:27:14.258 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:14.252+0000 7f1db33f9700 1 -- 172.21.15.42:0/3886006118 wait complete. 2019-12-05T05:27:14.258 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:14.252+0000 7f1db33f9700 1 -- 172.21.15.42:0/3886006118 >> 172.21.15.42:0/3886006118 conn(0x7f1dac0a7f90 msgr2=0x7f1dac0a9770 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:27:14.268 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b has gained rank=0, replacing gid=5240 2019-12-05T05:27:14.269 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 8 secs before reviving mds.a 2019-12-05T05:27:22.269 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.a 2019-12-05T05:27:22.270 INFO:tasks.ceph.mds.a:Restarting daemon 2019-12-05T05:27:22.270 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2019-12-05T05:27:22.274 INFO:tasks.ceph.mds.a:Started 2019-12-05T05:27:22.274 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:27:22.330 INFO:tasks.ceph.mds.a.smithi042.stdout:starting mds.a at 2019-12-05T05:27:22.379 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:22.368+0000 7f42988c5700 1 Processor -- start 2019-12-05T05:27:22.379 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:22.368+0000 7f42988c5700 1 -- start start 2019-12-05T05:27:22.379 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:22.368+0000 7f42988c5700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f429408df80 0x7f429409f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:27:22.380 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:22.368+0000 7f42988c5700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f429409f830 0x7f429414efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:27:22.381 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:22.368+0000 7f42988c5700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f429409fc60 con 0x7f42940a7740 2019-12-05T05:27:22.381 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:22.368+0000 7f42988c5700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42940a0b00 con 0x7f429409f830 2019-12-05T05:27:22.391 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:22.368+0000 7f42988c5700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f429408e5b0 con 0x7f429408df80 2019-12-05T05:27:22.391 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:22.368+0000 7f4292d9d700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f42940a7740 0x7f429407e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:34130/0 (socket says 172.21.15.42:34130) 2019-12-05T05:27:22.391 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:22.368+0000 7f4292d9d700 1 -- 172.21.15.42:0/2312633125 learned_addr learned my addr 172.21.15.42:0/2312633125 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:27:22.391 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:22.368+0000 7f429259c700 1 --2- 172.21.15.42:0/2312633125 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f429408df80 0x7f429409f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:27:22.392 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:22.368+0000 7f429359e700 1 --2- 172.21.15.42:0/2312633125 >> v2:172.21.15.40:3300/0 conn(0x7f429409f830 0x7f429414efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:27:22.392 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:22.368+0000 7f4291d9b700 1 -- 172.21.15.42:0/2312633125 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3278752215 0 0) 0x7f429409fc60 con 0x7f42940a7740 2019-12-05T05:27:22.392 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:22.372+0000 7f4291d9b700 1 -- 172.21.15.42:0/2312633125 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42740030b0 con 0x7f42940a7740 2019-12-05T05:27:22.392 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:22.372+0000 7f429259c700 1 -- 172.21.15.42:0/2312633125 >> v2:172.21.15.40:3300/0 conn(0x7f429409f830 msgr2=0x7f429414efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:27:22.393 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:22.372+0000 7f429259c700 1 --2- 172.21.15.42:0/2312633125 >> v2:172.21.15.40:3300/0 conn(0x7f429409f830 0x7f429414efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:27:22.393 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:22.372+0000 7f429259c700 1 -- 172.21.15.42:0/2312633125 >> v1:172.21.15.42:6789/0 conn(0x7f42940a7740 legacy=0x7f429407e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:27:22.393 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:22.372+0000 7f4291d9b700 1 -- 172.21.15.42:0/2312633125 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1392356463 0 0) 0x7f42740030b0 con 0x7f42940a7740 2019-12-05T05:27:22.393 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:22.372+0000 7f429259c700 1 -- 172.21.15.42:0/2312633125 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42940a0710 con 0x7f429408df80 2019-12-05T05:27:22.393 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:22.372+0000 7f429259c700 1 --2- 172.21.15.42:0/2312633125 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f429408df80 0x7f429409f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f4284008b10 tx=0x7f427c0014a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:22.394 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:22.372+0000 7f429259c700 1 --2- 172.21.15.42:0/2312633125 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f429408df80 0x7f429409f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f4284008b10 tx=0x7f427c0014a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:22.394 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:22.372+0000 7f429259c700 1 --2- 172.21.15.42:0/2312633125 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f429408df80 0x7f429409f2e0 secure :-1 s=READY pgs=181 cs=0 l=1 rx=0x7f4284008b10 tx=0x7f427c0014a0).ready entity=mon.2 client_cookie=90f552ed705be163 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:27:22.394 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:22.372+0000 7f429259c700 1 --2- 172.21.15.42:0/2312633125 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f429408df80 0x7f429409f2e0 secure :-1 s=THROTTLE_DONE pgs=181 cs=0 l=1 rx=0x7f4284008b10 tx=0x7f427c0014a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:22.394 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:22.372+0000 7f4291d9b700 1 -- 172.21.15.42:0/2312633125 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f427c001750 con 0x7f429408df80 2019-12-05T05:27:22.394 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:22.372+0000 7f429259c700 1 --2- 172.21.15.42:0/2312633125 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f429408df80 0x7f429409f2e0 secure :-1 s=THROTTLE_DONE pgs=181 cs=0 l=1 rx=0x7f4284008b10 tx=0x7f427c0014a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:22.394 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:22.372+0000 7f429259c700 1 --2- 172.21.15.42:0/2312633125 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f429408df80 0x7f429409f2e0 secure :-1 s=THROTTLE_DONE pgs=181 cs=0 l=1 rx=0x7f4284008b10 tx=0x7f427c0014a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:22.395 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:22.372+0000 7f4291d9b700 1 -- 172.21.15.42:0/2312633125 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f427c001560 con 0x7f429408df80 2019-12-05T05:27:22.395 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:22.372+0000 7f4291d9b700 1 -- 172.21.15.42:0/2312633125 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f42840109f0 con 0x7f429408df80 2019-12-05T05:27:22.395 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:22.372+0000 7f42988c5700 1 -- 172.21.15.42:0/2312633125 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f429408df80 msgr2=0x7f429409f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:27:22.395 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:22.372+0000 7f42988c5700 1 --2- 172.21.15.42:0/2312633125 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f429408df80 0x7f429409f2e0 secure :-1 s=READY pgs=181 cs=0 l=1 rx=0x7f4284008b10 tx=0x7f427c0014a0).stop 2019-12-05T05:27:22.395 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:22.372+0000 7f42988c5700 1 -- 172.21.15.42:0/2312633125 shutdown_connections 2019-12-05T05:27:22.395 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:22.372+0000 7f42988c5700 1 --2- 172.21.15.42:0/2312633125 >> v2:172.21.15.40:3300/0 conn(0x7f429409f830 0x7f429414efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:27:22.395 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:22.372+0000 7f42988c5700 1 --2- 172.21.15.42:0/2312633125 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f429408df80 0x7f429409f2e0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:27:22.396 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:22.372+0000 7f42988c5700 1 -- 172.21.15.42:0/2312633125 shutdown_connections 2019-12-05T05:27:22.396 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:22.372+0000 7f42988c5700 1 -- 172.21.15.42:0/2312633125 wait complete. 2019-12-05T05:27:22.396 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:22.372+0000 7f42988c5700 1 -- 172.21.15.42:0/2312633125 >> 172.21.15.42:0/2312633125 conn(0x7f42940aabe0 msgr2=0x7f42940a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:27:22.400 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:22.376+0000 7f42988c5700 1 Processor -- start 2019-12-05T05:27:22.400 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:22.376+0000 7f42988c5700 1 -- start start 2019-12-05T05:27:22.400 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:22.376+0000 7f42988c5700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f429411bd60 0x7f4294123910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:27:22.400 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:22.376+0000 7f42988c5700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f429407b9d0 0x7f429411a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:27:22.400 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:22.376+0000 7f42988c5700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4294123030 con 0x7f429411acc0 2019-12-05T05:27:22.400 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:22.376+0000 7f42988c5700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f429407d7d0 con 0x7f429411bd60 2019-12-05T05:27:22.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:22.376+0000 7f42988c5700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f429411a230 con 0x7f429407b9d0 2019-12-05T05:27:22.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:22.376+0000 7f4292d9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f429411bd60 0x7f4294123910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:27:22.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:22.376+0000 7f4292d9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f429411bd60 0x7f4294123910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:34098/0 (socket says 172.21.15.42:34098) 2019-12-05T05:27:22.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:22.376+0000 7f4292d9d700 1 -- 172.21.15.42:0/2722398494 learned_addr learned my addr 172.21.15.42:0/2722398494 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:27:22.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:22.376+0000 7f429259c700 1 --2- 172.21.15.42:0/2722398494 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f429407b9d0 0x7f429411a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:27:22.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:22.376+0000 7f4292d9d700 1 -- 172.21.15.42:0/2722398494 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f429407b9d0 msgr2=0x7f429411a770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:27:22.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:22.376+0000 7f4292d9d700 1 --2- 172.21.15.42:0/2722398494 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f429407b9d0 0x7f429411a770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:27:22.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:22.376+0000 7f4292d9d700 1 -- 172.21.15.42:0/2722398494 >> v1:172.21.15.42:6789/0 conn(0x7f429411acc0 legacy=0x7f4294130290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:27:22.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:22.376+0000 7f4292d9d700 1 -- 172.21.15.42:0/2722398494 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f429407d9f0 con 0x7f429411bd60 2019-12-05T05:27:22.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:22.376+0000 7f4292d9d700 1 --2- 172.21.15.42:0/2722398494 >> v2:172.21.15.42:3300/0 conn(0x7f429411bd60 0x7f4294123910 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f429407d990 tx=0x7f427c010a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:22.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:22.376+0000 7f4292d9d700 1 --2- 172.21.15.42:0/2722398494 >> v2:172.21.15.42:3300/0 conn(0x7f429411bd60 0x7f4294123910 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f429407d990 tx=0x7f427c010a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:22.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:22.376+0000 7f4292d9d700 1 --2- 172.21.15.42:0/2722398494 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f429411bd60 0x7f4294123910 secure :-1 s=READY pgs=890 cs=0 l=1 rx=0x7f429407d990 tx=0x7f427c010a90).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:27:22.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:22.376+0000 7f4292d9d700 1 --2- 172.21.15.42:0/2722398494 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f429411bd60 0x7f4294123910 secure :-1 s=THROTTLE_DONE pgs=890 cs=0 l=1 rx=0x7f429407d990 tx=0x7f427c010a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:22.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:22.376+0000 7f428b7fe700 1 -- 172.21.15.42:0/2722398494 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f427c00c890 con 0x7f429411bd60 2019-12-05T05:27:22.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:22.376+0000 7f4292d9d700 1 --2- 172.21.15.42:0/2722398494 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f429411bd60 0x7f4294123910 secure :-1 s=THROTTLE_DONE pgs=890 cs=0 l=1 rx=0x7f429407d990 tx=0x7f427c010a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:22.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:22.376+0000 7f4292d9d700 1 --2- 172.21.15.42:0/2722398494 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f429411bd60 0x7f4294123910 secure :-1 s=THROTTLE_DONE pgs=890 cs=0 l=1 rx=0x7f429407d990 tx=0x7f427c010a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:22.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:22.376+0000 7f428b7fe700 1 -- 172.21.15.42:0/2722398494 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f427c00c4b0 con 0x7f429411bd60 2019-12-05T05:27:22.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:22.376+0000 7f42988c5700 1 -- 172.21.15.42:0/2722398494 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4294123e60 con 0x7f429411bd60 2019-12-05T05:27:22.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:22.376+0000 7f428b7fe700 1 -- 172.21.15.42:0/2722398494 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f427c004430 con 0x7f429411bd60 2019-12-05T05:27:22.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:22.376+0000 7f42988c5700 1 -- 172.21.15.42:0/2722398494 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f429407d330 con 0x7f429411bd60 2019-12-05T05:27:22.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:22.380+0000 7f4292d9d700 1 --2- 172.21.15.42:0/2722398494 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f429411bd60 0x7f4294123910 secure :-1 s=THROTTLE_DONE pgs=890 cs=0 l=1 rx=0x7f429407d990 tx=0x7f427c010a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:22.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:22.380+0000 7f428b7fe700 1 -- 172.21.15.42:0/2722398494 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f427c0046d0 con 0x7f429411bd60 2019-12-05T05:27:22.404 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:22.380+0000 7f4292d9d700 1 --2- 172.21.15.42:0/2722398494 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f429411bd60 0x7f4294123910 secure :-1 s=THROTTLE_DONE pgs=890 cs=0 l=1 rx=0x7f429407d990 tx=0x7f427c010a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:22.404 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:22.380+0000 7f428b7fe700 1 --2- 172.21.15.42:0/2722398494 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f426c03ae20 0x7f426c03d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:27:22.404 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:22.380+0000 7f428b7fe700 1 -- 172.21.15.42:0/2722398494 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7407+0+0 (secure 0 0 0) 0x7f427c04ae00 con 0x7f429411bd60 2019-12-05T05:27:22.404 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:22.380+0000 7f429259c700 1 --2- 172.21.15.42:0/2722398494 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f426c03ae20 0x7f426c03d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:27:22.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:22.380+0000 7f429259c700 1 --2- 172.21.15.42:0/2722398494 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f426c03ae20 0x7f426c03d330 crc :-1 s=READY pgs=389 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:27:22.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:22.384+0000 7f42988c5700 1 -- 172.21.15.42:0/2722398494 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f429405d110 con 0x7f429411bd60 2019-12-05T05:27:22.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:22.388+0000 7f4292d9d700 1 --2- 172.21.15.42:0/2722398494 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f429411bd60 0x7f4294123910 secure :-1 s=THROTTLE_DONE pgs=890 cs=0 l=1 rx=0x7f429407d990 tx=0x7f427c010a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:22.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:22.388+0000 7f428b7fe700 1 -- 172.21.15.42:0/2722398494 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f427c031dc0 con 0x7f429411bd60 2019-12-05T05:27:22.561 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:22.552+0000 7f42988c5700 1 -- 172.21.15.42:0/2722398494 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f429405b8c0 con 0x7f429411bd60 2019-12-05T05:27:22.561 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:22.552+0000 7f4292d9d700 1 --2- 172.21.15.42:0/2722398494 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f429411bd60 0x7f4294123910 secure :-1 s=THROTTLE_DONE pgs=890 cs=0 l=1 rx=0x7f429407d990 tx=0x7f427c010a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:22.561 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:22.552+0000 7f428b7fe700 1 -- 172.21.15.42:0/2722398494 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 112 v112) v1 ==== 95+0+1775 (secure 0 0 0) 0x7f427c032020 con 0x7f429411bd60 2019-12-05T05:27:22.562 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 112 2019-12-05T05:27:22.562 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:22.552+0000 7f42988c5700 1 -- 172.21.15.42:0/2722398494 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f426c03ae20 msgr2=0x7f426c03d330 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:27:22.562 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:22.552+0000 7f42988c5700 1 --2- 172.21.15.42:0/2722398494 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f426c03ae20 0x7f426c03d330 crc :-1 s=READY pgs=389 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:27:22.562 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:22.552+0000 7f42988c5700 1 -- 172.21.15.42:0/2722398494 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f429411bd60 msgr2=0x7f4294123910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:27:22.562 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:22.552+0000 7f42988c5700 1 --2- 172.21.15.42:0/2722398494 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f429411bd60 0x7f4294123910 secure :-1 s=READY pgs=890 cs=0 l=1 rx=0x7f429407d990 tx=0x7f427c010a90).stop 2019-12-05T05:27:22.562 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:22.556+0000 7f42988c5700 1 -- 172.21.15.42:0/2722398494 shutdown_connections 2019-12-05T05:27:22.562 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:22.556+0000 7f42988c5700 1 --2- 172.21.15.42:0/2722398494 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f426c03ae20 0x7f426c03d330 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:27:22.563 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:22.556+0000 7f42988c5700 1 --2- 172.21.15.42:0/2722398494 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f429411bd60 0x7f4294123910 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:27:22.563 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:22.556+0000 7f42988c5700 1 --2- 172.21.15.42:0/2722398494 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f429407b9d0 0x7f429411a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:27:22.563 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:22.556+0000 7f42988c5700 1 -- 172.21.15.42:0/2722398494 shutdown_connections 2019-12-05T05:27:22.563 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:22.556+0000 7f42988c5700 1 -- 172.21.15.42:0/2722398494 wait complete. 2019-12-05T05:27:22.563 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:22.556+0000 7f42988c5700 1 -- 172.21.15.42:0/2722398494 >> 172.21.15.42:0/2722398494 conn(0x7f429411cef0 msgr2=0x7f4294059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:27:22.563 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:27:22.564 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":112,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":112,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:27:13.322209+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6546},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6546":{"gid":6546,"name":"b","rank":0,"incarnation":109,"state":"up:active","state_seq":25,"addr":"172.21.15.40:6833/1149478696","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1149478696},{"type":"v1","addr":"172.21.15.40:6833","nonce":1149478696}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:27:22.569 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2019-12-05T05:27:24.570 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:27:24.681 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:24.672+0000 7f7a1a627700 1 Processor -- start 2019-12-05T05:27:24.681 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:24.672+0000 7f7a1a627700 1 -- start start 2019-12-05T05:27:24.681 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:24.672+0000 7f7a1a627700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f7a1409fad0 0x7f7a1414ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:27:24.681 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:24.672+0000 7f7a1a627700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a140a0700 con 0x7f7a140a7730 2019-12-05T05:27:24.682 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:24.672+0000 7f7a1a627700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a1407ef10 con 0x7f7a1408df70 2019-12-05T05:27:24.682 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:24.672+0000 7f7a1a627700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a140a0af0 con 0x7f7a1409fad0 2019-12-05T05:27:24.682 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:24.672+0000 7f7a0bfff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f7a1408df70 0x7f7a1409f2d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:34162/0 (socket says 172.21.15.42:34162) 2019-12-05T05:27:24.682 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:24.672+0000 7f7a0bfff700 1 -- 172.21.15.42:0/1180855890 learned_addr learned my addr 172.21.15.42:0/1180855890 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:27:24.682 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:24.672+0000 7f7a193c5700 1 --2- 172.21.15.42:0/1180855890 >> v2:172.21.15.42:3300/0 conn(0x7f7a1409fad0 0x7f7a1414ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:27:24.682 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:24.672+0000 7f7a0b7fe700 1 -- 172.21.15.42:0/1180855890 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 639028643 0 0) 0x7f7a1407ef10 con 0x7f7a1408df70 2019-12-05T05:27:24.682 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:24.672+0000 7f7a0b7fe700 1 -- 172.21.15.42:0/1180855890 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79fc0030b0 con 0x7f7a1408df70 2019-12-05T05:27:24.683 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:24.672+0000 7f7a0b7fe700 1 -- 172.21.15.42:0/1180855890 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1505301960 0 0) 0x7f7a140a0700 con 0x7f7a140a7730 2019-12-05T05:27:24.683 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:24.672+0000 7f7a0b7fe700 1 -- 172.21.15.42:0/1180855890 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a1407ef10 con 0x7f7a140a7730 2019-12-05T05:27:24.683 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:24.672+0000 7f7a193c5700 1 -- 172.21.15.42:0/1180855890 >> v1:172.21.15.42:6789/0 conn(0x7f7a1408df70 legacy=0x7f7a1409f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:27:24.683 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:24.672+0000 7f7a193c5700 1 -- 172.21.15.42:0/1180855890 >> v1:172.21.15.40:6789/0 conn(0x7f7a140a7730 legacy=0x7f7a1407e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:27:24.683 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:24.672+0000 7f7a193c5700 1 -- 172.21.15.42:0/1180855890 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a140a9b90 con 0x7f7a1409fad0 2019-12-05T05:27:24.683 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:24.672+0000 7f7a193c5700 1 --2- 172.21.15.42:0/1180855890 >> v2:172.21.15.42:3300/0 conn(0x7f7a1409fad0 0x7f7a1414ffe0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f7a0c008790 tx=0x7f7a0c00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:24.683 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:24.672+0000 7f7a193c5700 1 --2- 172.21.15.42:0/1180855890 >> v2:172.21.15.42:3300/0 conn(0x7f7a1409fad0 0x7f7a1414ffe0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f7a0c008790 tx=0x7f7a0c00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:24.684 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:24.672+0000 7f7a193c5700 1 --2- 172.21.15.42:0/1180855890 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7a1409fad0 0x7f7a1414ffe0 secure :-1 s=READY pgs=892 cs=0 l=1 rx=0x7f7a0c008790 tx=0x7f7a0c00fbf0).ready entity=mon.1 client_cookie=68da31f8f6a690c6 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:27:24.684 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:24.672+0000 7f7a193c5700 1 --2- 172.21.15.42:0/1180855890 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7a1409fad0 0x7f7a1414ffe0 secure :-1 s=THROTTLE_DONE pgs=892 cs=0 l=1 rx=0x7f7a0c008790 tx=0x7f7a0c00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:24.684 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:24.672+0000 7f7a0b7fe700 1 -- 172.21.15.42:0/1180855890 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7a0c008bb0 con 0x7f7a1409fad0 2019-12-05T05:27:24.684 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:24.672+0000 7f7a193c5700 1 --2- 172.21.15.42:0/1180855890 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7a1409fad0 0x7f7a1414ffe0 secure :-1 s=THROTTLE_DONE pgs=892 cs=0 l=1 rx=0x7f7a0c008790 tx=0x7f7a0c00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:24.684 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:24.672+0000 7f7a193c5700 1 --2- 172.21.15.42:0/1180855890 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7a1409fad0 0x7f7a1414ffe0 secure :-1 s=THROTTLE_DONE pgs=892 cs=0 l=1 rx=0x7f7a0c008790 tx=0x7f7a0c00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:24.685 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:24.672+0000 7f7a0b7fe700 1 -- 172.21.15.42:0/1180855890 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a0c0087d0 con 0x7f7a1409fad0 2019-12-05T05:27:24.685 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:24.672+0000 7f7a0b7fe700 1 -- 172.21.15.42:0/1180855890 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7a0c0118d0 con 0x7f7a1409fad0 2019-12-05T05:27:24.685 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:24.672+0000 7f7a1a627700 1 -- 172.21.15.42:0/1180855890 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7a1409fad0 msgr2=0x7f7a1414ffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:27:24.685 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:24.672+0000 7f7a1a627700 1 --2- 172.21.15.42:0/1180855890 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7a1409fad0 0x7f7a1414ffe0 secure :-1 s=READY pgs=892 cs=0 l=1 rx=0x7f7a0c008790 tx=0x7f7a0c00fbf0).stop 2019-12-05T05:27:24.685 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:24.676+0000 7f7a1a627700 1 -- 172.21.15.42:0/1180855890 shutdown_connections 2019-12-05T05:27:24.685 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:24.676+0000 7f7a1a627700 1 --2- 172.21.15.42:0/1180855890 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7a1409fad0 0x7f7a1414ffe0 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:27:24.685 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:24.676+0000 7f7a1a627700 1 -- 172.21.15.42:0/1180855890 shutdown_connections 2019-12-05T05:27:24.685 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:24.676+0000 7f7a1a627700 1 -- 172.21.15.42:0/1180855890 wait complete. 2019-12-05T05:27:24.686 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:24.676+0000 7f7a1a627700 1 -- 172.21.15.42:0/1180855890 >> 172.21.15.42:0/1180855890 conn(0x7f7a140aabd0 msgr2=0x7f7a140a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:27:24.689 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:24.676+0000 7f7a1a627700 1 Processor -- start 2019-12-05T05:27:24.689 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:24.676+0000 7f7a1a627700 1 -- start start 2019-12-05T05:27:24.689 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:24.680+0000 7f7a1a627700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7a1411b3e0 0x7f7a1407c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:27:24.690 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:24.680+0000 7f7a1a627700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f7a14129540 0x7f7a14129970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:27:24.690 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:24.680+0000 7f7a1a627700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a14119020 con 0x7f7a141164f0 2019-12-05T05:27:24.690 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:24.680+0000 7f7a1a627700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a1411bf80 con 0x7f7a1411b3e0 2019-12-05T05:27:24.690 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:24.680+0000 7f7a1a627700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a1407cc20 con 0x7f7a14129540 2019-12-05T05:27:24.690 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:24.680+0000 7f7a193c5700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f7a14129540 0x7f7a14129970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:27:24.690 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:24.680+0000 7f7a193c5700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f7a14129540 0x7f7a14129970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:34132/0 (socket says 172.21.15.42:34132) 2019-12-05T05:27:24.690 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:24.680+0000 7f7a193c5700 1 -- 172.21.15.42:0/1494049016 learned_addr learned my addr 172.21.15.42:0/1494049016 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:27:24.691 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:24.680+0000 7f7a0bfff700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7a1411b3e0 0x7f7a1407c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:27:24.691 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:24.680+0000 7f7a193c5700 1 -- 172.21.15.42:0/1494049016 >> v2:172.21.15.40:3300/0 conn(0x7f7a1411b3e0 msgr2=0x7f7a1407c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:27:24.691 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:24.680+0000 7f7a193c5700 1 --2- 172.21.15.42:0/1494049016 >> v2:172.21.15.40:3300/0 conn(0x7f7a1411b3e0 0x7f7a1407c510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:27:24.691 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:24.680+0000 7f7a193c5700 1 -- 172.21.15.42:0/1494049016 >> v1:172.21.15.40:6789/0 conn(0x7f7a141164f0 legacy=0x7f7a1411abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:27:24.691 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:24.680+0000 7f7a193c5700 1 -- 172.21.15.42:0/1494049016 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a14118c50 con 0x7f7a14129540 2019-12-05T05:27:24.691 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:24.680+0000 7f7a193c5700 1 --2- 172.21.15.42:0/1494049016 >> v2:172.21.15.42:3300/0 conn(0x7f7a14129540 0x7f7a14129970 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f7a0c008790 tx=0x7f7a0c011990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:24.692 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:24.680+0000 7f7a193c5700 1 --2- 172.21.15.42:0/1494049016 >> v2:172.21.15.42:3300/0 conn(0x7f7a14129540 0x7f7a14129970 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f7a0c008790 tx=0x7f7a0c011990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:24.692 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:24.680+0000 7f7a193c5700 1 --2- 172.21.15.42:0/1494049016 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7a14129540 0x7f7a14129970 secure :-1 s=READY pgs=893 cs=0 l=1 rx=0x7f7a0c008790 tx=0x7f7a0c011990).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:27:24.692 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:24.680+0000 7f7a193c5700 1 --2- 172.21.15.42:0/1494049016 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7a14129540 0x7f7a14129970 secure :-1 s=THROTTLE_DONE pgs=893 cs=0 l=1 rx=0x7f7a0c008790 tx=0x7f7a0c011990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:24.692 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:24.680+0000 7f7a193c5700 1 --2- 172.21.15.42:0/1494049016 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7a14129540 0x7f7a14129970 secure :-1 s=THROTTLE_DONE pgs=893 cs=0 l=1 rx=0x7f7a0c008790 tx=0x7f7a0c011990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:24.692 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:24.680+0000 7f7a193c5700 1 --2- 172.21.15.42:0/1494049016 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7a14129540 0x7f7a14129970 secure :-1 s=THROTTLE_DONE pgs=893 cs=0 l=1 rx=0x7f7a0c008790 tx=0x7f7a0c011990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:24.692 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:24.680+0000 7f7a097fa700 1 -- 172.21.15.42:0/1494049016 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7a0c0087d0 con 0x7f7a14129540 2019-12-05T05:27:24.692 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:24.680+0000 7f7a097fa700 1 -- 172.21.15.42:0/1494049016 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a0c008db0 con 0x7f7a14129540 2019-12-05T05:27:24.693 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:24.680+0000 7f7a097fa700 1 -- 172.21.15.42:0/1494049016 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7a0c005540 con 0x7f7a14129540 2019-12-05T05:27:24.693 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:24.680+0000 7f7a1a627700 1 -- 172.21.15.42:0/1494049016 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a14129ec0 con 0x7f7a14129540 2019-12-05T05:27:24.693 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:24.680+0000 7f7a1a627700 1 -- 172.21.15.42:0/1494049016 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a14130900 con 0x7f7a14129540 2019-12-05T05:27:24.693 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:24.684+0000 7f7a193c5700 1 --2- 172.21.15.42:0/1494049016 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7a14129540 0x7f7a14129970 secure :-1 s=THROTTLE_DONE pgs=893 cs=0 l=1 rx=0x7f7a0c008790 tx=0x7f7a0c011990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:24.693 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:24.684+0000 7f7a097fa700 1 -- 172.21.15.42:0/1494049016 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f7a0c005850 con 0x7f7a14129540 2019-12-05T05:27:24.693 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:24.684+0000 7f7a193c5700 1 --2- 172.21.15.42:0/1494049016 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7a14129540 0x7f7a14129970 secure :-1 s=THROTTLE_DONE pgs=893 cs=0 l=1 rx=0x7f7a0c008790 tx=0x7f7a0c011990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:24.693 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:24.684+0000 7f7a097fa700 1 --2- 172.21.15.42:0/1494049016 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f79f403add0 0x7f79f403d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:27:24.694 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:24.684+0000 7f7a097fa700 1 -- 172.21.15.42:0/1494049016 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7407+0+0 (secure 0 0 0) 0x7f7a0c012060 con 0x7f7a14129540 2019-12-05T05:27:24.694 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:24.684+0000 7f7a18bc4700 1 --2- 172.21.15.42:0/1494049016 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f79f403add0 0x7f79f403d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:27:24.695 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:24.684+0000 7f7a18bc4700 1 --2- 172.21.15.42:0/1494049016 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f79f403add0 0x7f79f403d2e0 crc :-1 s=READY pgs=390 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:27:24.695 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:24.688+0000 7f7a1a627700 1 -- 172.21.15.42:0/1494049016 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a1405d100 con 0x7f7a14129540 2019-12-05T05:27:24.698 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:24.692+0000 7f7a193c5700 1 --2- 172.21.15.42:0/1494049016 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7a14129540 0x7f7a14129970 secure :-1 s=THROTTLE_DONE pgs=893 cs=0 l=1 rx=0x7f7a0c008790 tx=0x7f7a0c011990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:24.698 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:24.692+0000 7f7a097fa700 1 -- 172.21.15.42:0/1494049016 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f7a0c032930 con 0x7f7a14129540 2019-12-05T05:27:24.886 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:24.872+0000 7f7a1a627700 1 -- 172.21.15.42:0/1494049016 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7a1405b8b0 con 0x7f7a14129540 2019-12-05T05:27:24.886 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:24.872+0000 7f7a193c5700 1 --2- 172.21.15.42:0/1494049016 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7a14129540 0x7f7a14129970 secure :-1 s=THROTTLE_DONE pgs=893 cs=0 l=1 rx=0x7f7a0c008790 tx=0x7f7a0c011990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:24.887 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:24.872+0000 7f7a097fa700 1 -- 172.21.15.42:0/1494049016 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 112 v112) v1 ==== 95+0+1775 (secure 0 0 0) 0x7f7a0c0334e0 con 0x7f7a14129540 2019-12-05T05:27:24.887 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 112 2019-12-05T05:27:24.887 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:24.876+0000 7f7a1a627700 1 -- 172.21.15.42:0/1494049016 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f79f403add0 msgr2=0x7f79f403d2e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:27:24.887 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:24.876+0000 7f7a1a627700 1 --2- 172.21.15.42:0/1494049016 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f79f403add0 0x7f79f403d2e0 crc :-1 s=READY pgs=390 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:27:24.887 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:24.876+0000 7f7a1a627700 1 -- 172.21.15.42:0/1494049016 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7a14129540 msgr2=0x7f7a14129970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:27:24.887 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:24.876+0000 7f7a1a627700 1 --2- 172.21.15.42:0/1494049016 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7a14129540 0x7f7a14129970 secure :-1 s=READY pgs=893 cs=0 l=1 rx=0x7f7a0c008790 tx=0x7f7a0c011990).stop 2019-12-05T05:27:24.887 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:24.876+0000 7f7a1a627700 1 -- 172.21.15.42:0/1494049016 shutdown_connections 2019-12-05T05:27:24.888 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:24.876+0000 7f7a1a627700 1 --2- 172.21.15.42:0/1494049016 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f79f403add0 0x7f79f403d2e0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:27:24.888 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:24.876+0000 7f7a1a627700 1 --2- 172.21.15.42:0/1494049016 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7a14129540 0x7f7a14129970 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:27:24.888 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:24.876+0000 7f7a1a627700 1 --2- 172.21.15.42:0/1494049016 >> v2:172.21.15.40:3300/0 conn(0x7f7a1411b3e0 0x7f7a1407c510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:27:24.888 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:24.876+0000 7f7a1a627700 1 -- 172.21.15.42:0/1494049016 shutdown_connections 2019-12-05T05:27:24.888 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:24.876+0000 7f7a1a627700 1 -- 172.21.15.42:0/1494049016 wait complete. 2019-12-05T05:27:24.888 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:24.876+0000 7f7a1a627700 1 -- 172.21.15.42:0/1494049016 >> 172.21.15.42:0/1494049016 conn(0x7f7a140aabd0 msgr2=0x7f7a140596f0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:27:24.889 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:27:24.889 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":112,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":112,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:27:13.322209+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6546},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6546":{"gid":6546,"name":"b","rank":0,"incarnation":109,"state":"up:active","state_seq":25,"addr":"172.21.15.40:6833/1149478696","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1149478696},{"type":"v1","addr":"172.21.15.40:6833","nonce":1149478696}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:27:24.913 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2019-12-05T05:27:26.915 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:27:27.027 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:27.016+0000 7f7f2ab99700 1 Processor -- start 2019-12-05T05:27:27.027 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:27.016+0000 7f7f2ab99700 1 -- start start 2019-12-05T05:27:27.027 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:27.016+0000 7f7f2ab99700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7f240a7730 0x7f7f2408df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:27:27.027 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:27.016+0000 7f7f2ab99700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7f2407e710 0x7f7f2407eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:27:27.028 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:27.016+0000 7f7f2ab99700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f240a0700 con 0x7f7f2409f2d0 2019-12-05T05:27:27.028 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:27.016+0000 7f7f2ab99700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f240a0af0 con 0x7f7f240a7730 2019-12-05T05:27:27.028 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:27.016+0000 7f7f2ab99700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f2408e5a0 con 0x7f7f2407e710 2019-12-05T05:27:27.028 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:27.020+0000 7f7f29937700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f7f2409f2d0 0x7f7f2409f700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:34182/0 (socket says 172.21.15.42:34182) 2019-12-05T05:27:27.028 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:27.020+0000 7f7f29937700 1 -- 172.21.15.42:0/1482539685 learned_addr learned my addr 172.21.15.42:0/1482539685 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:27:27.028 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:27.020+0000 7f7f28935700 1 --2- 172.21.15.42:0/1482539685 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7f2407e710 0x7f7f2407eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:27:27.028 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:27.020+0000 7f7f29136700 1 --2- 172.21.15.42:0/1482539685 >> v2:172.21.15.40:3300/0 conn(0x7f7f240a7730 0x7f7f2408df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:27:27.029 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:27.020+0000 7f7f1bfff700 1 -- 172.21.15.42:0/1482539685 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2819964265 0 0) 0x7f7f240a0700 con 0x7f7f2409f2d0 2019-12-05T05:27:27.029 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:27.020+0000 7f7f1bfff700 1 -- 172.21.15.42:0/1482539685 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f100030b0 con 0x7f7f2409f2d0 2019-12-05T05:27:27.029 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:27.020+0000 7f7f28935700 1 -- 172.21.15.42:0/1482539685 >> v2:172.21.15.40:3300/0 conn(0x7f7f240a7730 msgr2=0x7f7f2408df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:27:27.029 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:27.020+0000 7f7f28935700 1 --2- 172.21.15.42:0/1482539685 >> v2:172.21.15.40:3300/0 conn(0x7f7f240a7730 0x7f7f2408df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:27:27.029 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:27.020+0000 7f7f28935700 1 -- 172.21.15.42:0/1482539685 >> v1:172.21.15.42:6789/0 conn(0x7f7f2409f2d0 legacy=0x7f7f2409f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:27:27.029 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:27.020+0000 7f7f28935700 1 -- 172.21.15.42:0/1482539685 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f2407f090 con 0x7f7f2407e710 2019-12-05T05:27:27.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:27.020+0000 7f7f28935700 1 --2- 172.21.15.42:0/1482539685 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7f2407e710 0x7f7f2407eb40 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f7f14008790 tx=0x7f7f1400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:27.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:27.020+0000 7f7f28935700 1 --2- 172.21.15.42:0/1482539685 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7f2407e710 0x7f7f2407eb40 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f7f14008790 tx=0x7f7f1400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:27.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:27.020+0000 7f7f28935700 1 --2- 172.21.15.42:0/1482539685 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7f2407e710 0x7f7f2407eb40 secure :-1 s=READY pgs=182 cs=0 l=1 rx=0x7f7f14008790 tx=0x7f7f1400fc70).ready entity=mon.2 client_cookie=37da5a14e13785a9 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:27:27.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:27.020+0000 7f7f28935700 1 --2- 172.21.15.42:0/1482539685 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7f2407e710 0x7f7f2407eb40 secure :-1 s=THROTTLE_DONE pgs=182 cs=0 l=1 rx=0x7f7f14008790 tx=0x7f7f1400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:27.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:27.020+0000 7f7f1bfff700 1 -- 172.21.15.42:0/1482539685 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7f140112f0 con 0x7f7f2407e710 2019-12-05T05:27:27.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:27.020+0000 7f7f28935700 1 --2- 172.21.15.42:0/1482539685 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7f2407e710 0x7f7f2407eb40 secure :-1 s=THROTTLE_DONE pgs=182 cs=0 l=1 rx=0x7f7f14008790 tx=0x7f7f1400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:27.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:27.020+0000 7f7f28935700 1 --2- 172.21.15.42:0/1482539685 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7f2407e710 0x7f7f2407eb40 secure :-1 s=THROTTLE_DONE pgs=182 cs=0 l=1 rx=0x7f7f14008790 tx=0x7f7f1400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:27.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:27.020+0000 7f7f1bfff700 1 -- 172.21.15.42:0/1482539685 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f14008ad0 con 0x7f7f2407e710 2019-12-05T05:27:27.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:27.020+0000 7f7f1bfff700 1 -- 172.21.15.42:0/1482539685 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7f140118d0 con 0x7f7f2407e710 2019-12-05T05:27:27.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:27.020+0000 7f7f2ab99700 1 -- 172.21.15.42:0/1482539685 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7f2407e710 msgr2=0x7f7f2407eb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:27:27.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:27.020+0000 7f7f2ab99700 1 --2- 172.21.15.42:0/1482539685 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7f2407e710 0x7f7f2407eb40 secure :-1 s=READY pgs=182 cs=0 l=1 rx=0x7f7f14008790 tx=0x7f7f1400fc70).stop 2019-12-05T05:27:27.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:27.020+0000 7f7f2ab99700 1 -- 172.21.15.42:0/1482539685 shutdown_connections 2019-12-05T05:27:27.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:27.020+0000 7f7f2ab99700 1 --2- 172.21.15.42:0/1482539685 >> v2:172.21.15.40:3300/0 conn(0x7f7f240a7730 0x7f7f2408df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:27:27.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:27.020+0000 7f7f2ab99700 1 --2- 172.21.15.42:0/1482539685 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7f2407e710 0x7f7f2407eb40 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:27:27.032 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:27.020+0000 7f7f2ab99700 1 -- 172.21.15.42:0/1482539685 shutdown_connections 2019-12-05T05:27:27.032 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:27.020+0000 7f7f2ab99700 1 -- 172.21.15.42:0/1482539685 wait complete. 2019-12-05T05:27:27.032 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:27.020+0000 7f7f2ab99700 1 -- 172.21.15.42:0/1482539685 >> 172.21.15.42:0/1482539685 conn(0x7f7f240aabd0 msgr2=0x7f7f240a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:27:27.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:27.024+0000 7f7f2ab99700 1 Processor -- start 2019-12-05T05:27:27.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:27.024+0000 7f7f2ab99700 1 -- start start 2019-12-05T05:27:27.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:27.024+0000 7f7f2ab99700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7f2408ef30 0x7f7f240a1500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:27:27.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:27.024+0000 7f7f2ab99700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f240a2d50 con 0x7f7f24145170 2019-12-05T05:27:27.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:27.024+0000 7f7f2ab99700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f240a2f80 con 0x7f7f24142520 2019-12-05T05:27:27.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:27.024+0000 7f7f2ab99700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f240a4410 con 0x7f7f2408ef30 2019-12-05T05:27:27.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:27.024+0000 7f7f29136700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f7f24142520 0x7f7f240a59b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:34184/0 (socket says 172.21.15.42:34184) 2019-12-05T05:27:27.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:27.024+0000 7f7f29136700 1 -- 172.21.15.42:0/3855366333 learned_addr learned my addr 172.21.15.42:0/3855366333 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:27:27.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:27.024+0000 7f7f28935700 1 --2- 172.21.15.42:0/3855366333 >> v2:172.21.15.40:3300/0 conn(0x7f7f2408ef30 0x7f7f240a1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:27:27.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:27.024+0000 7f7f19ffb700 1 -- 172.21.15.42:0/3855366333 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1050554968 0 0) 0x7f7f240a2f80 con 0x7f7f24142520 2019-12-05T05:27:27.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:27.024+0000 7f7f19ffb700 1 -- 172.21.15.42:0/3855366333 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f040030b0 con 0x7f7f24142520 2019-12-05T05:27:27.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:27.024+0000 7f7f19ffb700 1 -- 172.21.15.42:0/3855366333 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 509831537 0 0) 0x7f7f240a2d50 con 0x7f7f24145170 2019-12-05T05:27:27.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:27.024+0000 7f7f19ffb700 1 -- 172.21.15.42:0/3855366333 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f240a2f80 con 0x7f7f24145170 2019-12-05T05:27:27.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:27.024+0000 7f7f28935700 1 -- 172.21.15.42:0/3855366333 >> v1:172.21.15.42:6789/0 conn(0x7f7f24142520 legacy=0x7f7f240a59b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:27:27.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:27.024+0000 7f7f28935700 1 -- 172.21.15.42:0/3855366333 >> v1:172.21.15.40:6789/0 conn(0x7f7f24145170 legacy=0x7f7f240a61b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:27:27.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:27.024+0000 7f7f28935700 1 -- 172.21.15.42:0/3855366333 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f240a31b0 con 0x7f7f2408ef30 2019-12-05T05:27:27.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:27.024+0000 7f7f28935700 1 --2- 172.21.15.42:0/3855366333 >> v2:172.21.15.40:3300/0 conn(0x7f7f2408ef30 0x7f7f240a1500 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f7f14008790 tx=0x7f7f140118f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:27.039 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:27.024+0000 7f7f28935700 1 --2- 172.21.15.42:0/3855366333 >> v2:172.21.15.40:3300/0 conn(0x7f7f2408ef30 0x7f7f240a1500 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f7f14008790 tx=0x7f7f140118f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:27.039 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:27.024+0000 7f7f28935700 1 --2- 172.21.15.42:0/3855366333 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7f2408ef30 0x7f7f240a1500 secure :-1 s=READY pgs=704 cs=0 l=1 rx=0x7f7f14008790 tx=0x7f7f140118f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:27:27.039 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:27.024+0000 7f7f28935700 1 --2- 172.21.15.42:0/3855366333 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7f2408ef30 0x7f7f240a1500 secure :-1 s=THROTTLE_DONE pgs=704 cs=0 l=1 rx=0x7f7f14008790 tx=0x7f7f140118f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:27.039 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:27.024+0000 7f7f19ffb700 1 -- 172.21.15.42:0/3855366333 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7f14007150 con 0x7f7f2408ef30 2019-12-05T05:27:27.039 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:27.024+0000 7f7f28935700 1 --2- 172.21.15.42:0/3855366333 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7f2408ef30 0x7f7f240a1500 secure :-1 s=THROTTLE_DONE pgs=704 cs=0 l=1 rx=0x7f7f14008790 tx=0x7f7f140118f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:27.039 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:27.024+0000 7f7f19ffb700 1 -- 172.21.15.42:0/3855366333 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f140114f0 con 0x7f7f2408ef30 2019-12-05T05:27:27.039 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:27.024+0000 7f7f28935700 1 --2- 172.21.15.42:0/3855366333 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7f2408ef30 0x7f7f240a1500 secure :-1 s=THROTTLE_DONE pgs=704 cs=0 l=1 rx=0x7f7f14008790 tx=0x7f7f140118f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:27.039 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:27.024+0000 7f7f2ab99700 1 -- 172.21.15.42:0/3855366333 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f241369f0 con 0x7f7f2408ef30 2019-12-05T05:27:27.040 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:27.024+0000 7f7f19ffb700 1 -- 172.21.15.42:0/3855366333 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7f14008cc0 con 0x7f7f2408ef30 2019-12-05T05:27:27.040 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:27.024+0000 7f7f2ab99700 1 -- 172.21.15.42:0/3855366333 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f24136ec0 con 0x7f7f2408ef30 2019-12-05T05:27:27.040 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:27.028+0000 7f7f28935700 1 --2- 172.21.15.42:0/3855366333 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7f2408ef30 0x7f7f240a1500 secure :-1 s=THROTTLE_DONE pgs=704 cs=0 l=1 rx=0x7f7f14008790 tx=0x7f7f140118f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:27.040 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:27.028+0000 7f7f19ffb700 1 -- 172.21.15.42:0/3855366333 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f7f14005540 con 0x7f7f2408ef30 2019-12-05T05:27:27.040 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:27.028+0000 7f7f28935700 1 --2- 172.21.15.42:0/3855366333 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7f2408ef30 0x7f7f240a1500 secure :-1 s=THROTTLE_DONE pgs=704 cs=0 l=1 rx=0x7f7f14008790 tx=0x7f7f140118f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:27.040 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:27.028+0000 7f7f19ffb700 1 --2- 172.21.15.42:0/3855366333 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f7f0403b470 0x7f7f0403d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:27:27.040 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:27.028+0000 7f7f19ffb700 1 -- 172.21.15.42:0/3855366333 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7407+0+0 (secure 0 0 0) 0x7f7f1408f670 con 0x7f7f2408ef30 2019-12-05T05:27:27.041 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:27.028+0000 7f7f29136700 1 --2- 172.21.15.42:0/3855366333 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f7f0403b470 0x7f7f0403d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:27:27.041 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:27.028+0000 7f7f29136700 1 --2- 172.21.15.42:0/3855366333 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f7f0403b470 0x7f7f0403d980 crc :-1 s=READY pgs=392 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:27:27.041 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:27.032+0000 7f7f2ab99700 1 -- 172.21.15.42:0/3855366333 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f2405d100 con 0x7f7f2408ef30 2019-12-05T05:27:27.045 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:27.040+0000 7f7f28935700 1 --2- 172.21.15.42:0/3855366333 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7f2408ef30 0x7f7f240a1500 secure :-1 s=THROTTLE_DONE pgs=704 cs=0 l=1 rx=0x7f7f14008790 tx=0x7f7f140118f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:27.045 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:27.040+0000 7f7f19ffb700 1 -- 172.21.15.42:0/3855366333 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f7f14005850 con 0x7f7f2408ef30 2019-12-05T05:27:27.232 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:27.228+0000 7f7f2ab99700 1 -- 172.21.15.42:0/3855366333 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7f2405b8b0 con 0x7f7f2408ef30 2019-12-05T05:27:27.233 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:27.228+0000 7f7f28935700 1 --2- 172.21.15.42:0/3855366333 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7f2408ef30 0x7f7f240a1500 secure :-1 s=THROTTLE_DONE pgs=704 cs=0 l=1 rx=0x7f7f14008790 tx=0x7f7f140118f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:27.234 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:27.228+0000 7f7f19ffb700 1 -- 172.21.15.42:0/3855366333 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 113 v113) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f7f14056c40 con 0x7f7f2408ef30 2019-12-05T05:27:27.235 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 113 2019-12-05T05:27:27.236 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:27:27.236 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":113,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6666,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/2544447110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2544447110},{"type":"v1","addr":"172.21.15.42:6835","nonce":2544447110}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":113}],"filesystems":[{"mdsmap":{"epoch":112,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:27:13.322209+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6546},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6546":{"gid":6546,"name":"b","rank":0,"incarnation":109,"state":"up:active","state_seq":25,"addr":"172.21.15.40:6833/1149478696","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1149478696},{"type":"v1","addr":"172.21.15.40:6833","nonce":1149478696}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:27:27.238 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:27.232+0000 7f7f2ab99700 1 -- 172.21.15.42:0/3855366333 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f7f0403b470 msgr2=0x7f7f0403d980 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:27:27.238 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:27.232+0000 7f7f2ab99700 1 --2- 172.21.15.42:0/3855366333 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f7f0403b470 0x7f7f0403d980 crc :-1 s=READY pgs=392 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:27:27.238 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:27.232+0000 7f7f2ab99700 1 -- 172.21.15.42:0/3855366333 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7f2408ef30 msgr2=0x7f7f240a1500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:27:27.239 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:27.232+0000 7f7f2ab99700 1 --2- 172.21.15.42:0/3855366333 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7f2408ef30 0x7f7f240a1500 secure :-1 s=READY pgs=704 cs=0 l=1 rx=0x7f7f14008790 tx=0x7f7f140118f0).stop 2019-12-05T05:27:27.239 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:27.232+0000 7f7f2ab99700 1 -- 172.21.15.42:0/3855366333 shutdown_connections 2019-12-05T05:27:27.239 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:27.232+0000 7f7f2ab99700 1 --2- 172.21.15.42:0/3855366333 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f7f0403b470 0x7f7f0403d980 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:27:27.239 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:27.232+0000 7f7f2ab99700 1 --2- 172.21.15.42:0/3855366333 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7f2408ef30 0x7f7f240a1500 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:27:27.239 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:27.232+0000 7f7f2ab99700 1 -- 172.21.15.42:0/3855366333 shutdown_connections 2019-12-05T05:27:27.239 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:27.232+0000 7f7f2ab99700 1 -- 172.21.15.42:0/3855366333 wait complete. 2019-12-05T05:27:27.239 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:27.232+0000 7f7f2ab99700 1 -- 172.21.15.42:0/3855366333 >> 172.21.15.42:0/3855366333 conn(0x7f7f2407de80 msgr2=0x7f7f2407e4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:27:27.248 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a reported in up:standby state 2019-12-05T05:27:27.248 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 32 secs before thrashing 2019-12-05T05:27:38.608 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:27:38.611 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:27:59.249 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:27:59.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.340+0000 7f0ce1548700 1 Processor -- start 2019-12-05T05:27:59.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.340+0000 7f0ce1548700 1 -- start start 2019-12-05T05:27:59.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.344+0000 7f0ce1548700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0cdc0a7740 0x7f0cdc08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:27:59.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.344+0000 7f0ce1548700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0cdc09fae0 0x7f0cdc07eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:27:59.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.344+0000 7f0ce1548700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cdc0a0710 con 0x7f0cdc07e720 2019-12-05T05:27:59.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.344+0000 7f0ce1548700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0cdc0a0b00 con 0x7f0cdc0a7740 2019-12-05T05:27:59.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.344+0000 7f0ce1548700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cdc08e5b0 con 0x7f0cdc09fae0 2019-12-05T05:27:59.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.344+0000 7f0cdb7fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0cdc0a7740 0x7f0cdc08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:27:59.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.344+0000 7f0cdb7fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0cdc0a7740 0x7f0cdc08df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:34240/0 (socket says 172.21.15.42:34240) 2019-12-05T05:27:59.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.344+0000 7f0cdb7fe700 1 -- 172.21.15.42:0/1334917927 learned_addr learned my addr 172.21.15.42:0/1334917927 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:27:59.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.344+0000 7f0cdbfff700 1 --2- 172.21.15.42:0/1334917927 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0cdc09fae0 0x7f0cdc07eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:27:59.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.344+0000 7f0cdb7fe700 1 -- 172.21.15.42:0/1334917927 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0cdc09fae0 msgr2=0x7f0cdc07eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:27:59.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.344+0000 7f0cdb7fe700 1 --2- 172.21.15.42:0/1334917927 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0cdc09fae0 0x7f0cdc07eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:27:59.362 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.344+0000 7f0cdb7fe700 1 -- 172.21.15.42:0/1334917927 >> v1:172.21.15.42:6789/0 conn(0x7f0cdc07e720 legacy=0x7f0cdc09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:27:59.362 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.344+0000 7f0cdb7fe700 1 -- 172.21.15.42:0/1334917927 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cdc07f0a0 con 0x7f0cdc0a7740 2019-12-05T05:27:59.363 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.344+0000 7f0cdb7fe700 1 --2- 172.21.15.42:0/1334917927 >> v2:172.21.15.42:3300/0 conn(0x7f0cdc0a7740 0x7f0cdc08df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f0ccc008b10 tx=0x7f0ccc0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:59.363 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.344+0000 7f0cdb7fe700 1 --2- 172.21.15.42:0/1334917927 >> v2:172.21.15.42:3300/0 conn(0x7f0cdc0a7740 0x7f0cdc08df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f0ccc008b10 tx=0x7f0ccc0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:59.363 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.344+0000 7f0cdb7fe700 1 --2- 172.21.15.42:0/1334917927 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0cdc0a7740 0x7f0cdc08df80 secure :-1 s=READY pgs=897 cs=0 l=1 rx=0x7f0ccc008b10 tx=0x7f0ccc0100b0).ready entity=mon.1 client_cookie=e8ff1f48e6c5273c server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:27:59.363 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.344+0000 7f0cdb7fe700 1 --2- 172.21.15.42:0/1334917927 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0cdc0a7740 0x7f0cdc08df80 secure :-1 s=THROTTLE_DONE pgs=897 cs=0 l=1 rx=0x7f0ccc008b10 tx=0x7f0ccc0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:59.364 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.344+0000 7f0cda7fc700 1 -- 172.21.15.42:0/1334917927 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0ccc010da0 con 0x7f0cdc0a7740 2019-12-05T05:27:59.364 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.344+0000 7f0cdb7fe700 1 --2- 172.21.15.42:0/1334917927 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0cdc0a7740 0x7f0cdc08df80 secure :-1 s=THROTTLE_DONE pgs=897 cs=0 l=1 rx=0x7f0ccc008b10 tx=0x7f0ccc0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:59.364 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.344+0000 7f0cda7fc700 1 -- 172.21.15.42:0/1334917927 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ccc0109c0 con 0x7f0cdc0a7740 2019-12-05T05:27:59.364 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.344+0000 7f0cdb7fe700 1 --2- 172.21.15.42:0/1334917927 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0cdc0a7740 0x7f0cdc08df80 secure :-1 s=THROTTLE_DONE pgs=897 cs=0 l=1 rx=0x7f0ccc008b10 tx=0x7f0ccc0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:59.365 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.344+0000 7f0cda7fc700 1 -- 172.21.15.42:0/1334917927 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0ccc0114a0 con 0x7f0cdc0a7740 2019-12-05T05:27:59.365 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.344+0000 7f0ce1548700 1 -- 172.21.15.42:0/1334917927 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0cdc0a7740 msgr2=0x7f0cdc08df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:27:59.365 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.344+0000 7f0ce1548700 1 --2- 172.21.15.42:0/1334917927 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0cdc0a7740 0x7f0cdc08df80 secure :-1 s=READY pgs=897 cs=0 l=1 rx=0x7f0ccc008b10 tx=0x7f0ccc0100b0).stop 2019-12-05T05:27:59.365 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.348+0000 7f0ce1548700 1 -- 172.21.15.42:0/1334917927 shutdown_connections 2019-12-05T05:27:59.366 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.348+0000 7f0ce1548700 1 --2- 172.21.15.42:0/1334917927 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0cdc09fae0 0x7f0cdc07eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:27:59.366 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.348+0000 7f0ce1548700 1 --2- 172.21.15.42:0/1334917927 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0cdc0a7740 0x7f0cdc08df80 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:27:59.366 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.348+0000 7f0ce1548700 1 -- 172.21.15.42:0/1334917927 shutdown_connections 2019-12-05T05:27:59.366 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.348+0000 7f0ce1548700 1 -- 172.21.15.42:0/1334917927 wait complete. 2019-12-05T05:27:59.367 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.348+0000 7f0ce1548700 1 -- 172.21.15.42:0/1334917927 >> 172.21.15.42:0/1334917927 conn(0x7f0cdc0aabe0 msgr2=0x7f0cdc0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:27:59.395 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.348+0000 7f0ce1548700 1 Processor -- start 2019-12-05T05:27:59.395 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.348+0000 7f0ce1548700 1 -- start start 2019-12-05T05:27:59.395 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.352+0000 7f0ce1548700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0cdc0a6eb0 0x7f0cdc07d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:27:59.396 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.352+0000 7f0ce1548700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0cdc0a4ae0 0x7f0cdc0a5a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:27:59.396 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.352+0000 7f0ce1548700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cdc11bdb0 con 0x7f0cdc0a5f90 2019-12-05T05:27:59.396 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.352+0000 7f0ce1548700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0cdc11bb10 con 0x7f0cdc0a4ae0 2019-12-05T05:27:59.396 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.352+0000 7f0ce1548700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cdc0a57a0 con 0x7f0cdc0a6eb0 2019-12-05T05:27:59.396 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.352+0000 7f0cdaffd700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0cdc0a4ae0 0x7f0cdc0a5a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:27:59.397 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.352+0000 7f0cdaffd700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0cdc0a4ae0 0x7f0cdc0a5a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:34246/0 (socket says 172.21.15.42:34246) 2019-12-05T05:27:59.397 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.352+0000 7f0cdaffd700 1 -- 172.21.15.42:0/1462476 learned_addr learned my addr 172.21.15.42:0/1462476 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:27:59.397 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.352+0000 7f0cdb7fe700 1 --2- 172.21.15.42:0/1462476 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0cdc0a6eb0 0x7f0cdc07d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:27:59.397 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.352+0000 7f0cdaffd700 1 -- 172.21.15.42:0/1462476 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0cdc0a6eb0 msgr2=0x7f0cdc07d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:27:59.398 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.352+0000 7f0cdaffd700 1 --2- 172.21.15.42:0/1462476 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0cdc0a6eb0 0x7f0cdc07d540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:27:59.398 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.352+0000 7f0cdaffd700 1 -- 172.21.15.42:0/1462476 >> v1:172.21.15.40:6789/0 conn(0x7f0cdc0a5f90 legacy=0x7f0cdc07bb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:27:59.398 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.352+0000 7f0cdaffd700 1 -- 172.21.15.42:0/1462476 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cdc07da90 con 0x7f0cdc0a4ae0 2019-12-05T05:27:59.399 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.352+0000 7f0cdaffd700 1 --2- 172.21.15.42:0/1462476 >> v2:172.21.15.42:3300/0 conn(0x7f0cdc0a4ae0 0x7f0cdc0a5a40 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f0cdc11bcd0 tx=0x7f0cc4012bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:59.399 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.352+0000 7f0cdaffd700 1 --2- 172.21.15.42:0/1462476 >> v2:172.21.15.42:3300/0 conn(0x7f0cdc0a4ae0 0x7f0cdc0a5a40 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f0cdc11bcd0 tx=0x7f0cc4012bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:59.399 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.352+0000 7f0cdaffd700 1 --2- 172.21.15.42:0/1462476 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0cdc0a4ae0 0x7f0cdc0a5a40 secure :-1 s=READY pgs=898 cs=0 l=1 rx=0x7f0cdc11bcd0 tx=0x7f0cc4012bb0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:27:59.400 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.352+0000 7f0cdaffd700 1 -- 172.21.15.42:0/1462476 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0cdc0a4ae0 msgr2=0x7f0cdc0a5a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2019-12-05T05:27:59.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.352+0000 7f0cdaffd700 1 -- 172.21.15.42:0/1462476 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0cdc0a4ae0 msgr2=0x7f0cdc0a5a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2019-12-05T05:27:59.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.352+0000 7f0cdaffd700 1 --2- 172.21.15.42:0/1462476 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0cdc0a4ae0 0x7f0cdc0a5a40 secure :-1 s=READY pgs=898 cs=0 l=1 rx=0x7f0cdc11bcd0 tx=0x7f0cc4012bb0).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-12-05T05:27:59.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.352+0000 7f0cdaffd700 1 --2- 172.21.15.42:0/1462476 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0cdc0a4ae0 0x7f0cdc0a5a40 secure :-1 s=READY pgs=898 cs=0 l=1 rx=0x7f0cdc11bcd0 tx=0x7f0cc4012bb0).stop 2019-12-05T05:27:59.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.352+0000 7f0cd3fff700 1 -- 172.21.15.42:0/1462476 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0cdc0a4ae0 msgr2=0x7f0cdc0a5a40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2019-12-05T05:27:59.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.352+0000 7f0cd3fff700 1 --2- 172.21.15.42:0/1462476 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0cdc0a4ae0 0x7f0cdc0a5a40 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:27:59.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.352+0000 7f0cd3fff700 1 --2- 172.21.15.42:0/1462476 >> v2:172.21.15.40:3300/0 conn(0x7f0cc0006fa0 0x7f0cc00094b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:27:59.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.352+0000 7f0cd3fff700 1 -- 172.21.15.42:0/1462476 --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cc0009bd0 con 0x7f0cc0003090 2019-12-05T05:27:59.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.352+0000 7f0cd3fff700 1 -- 172.21.15.42:0/1462476 --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0cc000aea0 con 0x7f0cc0006fa0 2019-12-05T05:27:59.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.352+0000 7f0cdaffd700 1 --2- 172.21.15.42:0/1462476 >> v2:172.21.15.40:3300/0 conn(0x7f0cc0006fa0 0x7f0cc00094b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:27:59.412 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.352+0000 7f0cd3fff700 1 -- 172.21.15.42:0/1462476 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2677637966 0 0) 0x7f0cc0009bd0 con 0x7f0cc0003090 2019-12-05T05:27:59.412 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.352+0000 7f0cd3fff700 1 -- 172.21.15.42:0/1462476 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0cc0009ee0 con 0x7f0cc0003090 2019-12-05T05:27:59.412 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.352+0000 7f0cdaffd700 1 -- 172.21.15.42:0/1462476 >> v1:172.21.15.42:6789/0 conn(0x7f0cc0003090 legacy=0x7f0cc0005560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:27:59.412 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.352+0000 7f0cdaffd700 1 -- 172.21.15.42:0/1462476 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cc000b0f0 con 0x7f0cc0006fa0 2019-12-05T05:27:59.412 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.356+0000 7f0cdaffd700 1 --2- 172.21.15.42:0/1462476 >> v2:172.21.15.40:3300/0 conn(0x7f0cc0006fa0 0x7f0cc00094b0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f0cdc11bcd0 tx=0x7f0cc4014730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:59.413 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.356+0000 7f0cdaffd700 1 --2- 172.21.15.42:0/1462476 >> v2:172.21.15.40:3300/0 conn(0x7f0cc0006fa0 0x7f0cc00094b0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f0cdc11bcd0 tx=0x7f0cc4014730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:59.413 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.356+0000 7f0cdaffd700 1 --2- 172.21.15.42:0/1462476 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0cc0006fa0 0x7f0cc00094b0 secure :-1 s=READY pgs=708 cs=0 l=1 rx=0x7f0cdc11bcd0 tx=0x7f0cc4014730).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:27:59.413 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.356+0000 7f0cdaffd700 1 --2- 172.21.15.42:0/1462476 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0cc0006fa0 0x7f0cc00094b0 secure :-1 s=THROTTLE_DONE pgs=708 cs=0 l=1 rx=0x7f0cdc11bcd0 tx=0x7f0cc4014730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:59.413 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.356+0000 7f0cd3fff700 1 -- 172.21.15.42:0/1462476 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0cc4003110 con 0x7f0cc0006fa0 2019-12-05T05:27:59.413 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.356+0000 7f0cdaffd700 1 --2- 172.21.15.42:0/1462476 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0cc0006fa0 0x7f0cc00094b0 secure :-1 s=THROTTLE_DONE pgs=708 cs=0 l=1 rx=0x7f0cdc11bcd0 tx=0x7f0cc4014730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:59.413 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.356+0000 7f0ce1548700 1 -- 172.21.15.42:0/1462476 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cdc11b790 con 0x7f0cc0006fa0 2019-12-05T05:27:59.413 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.356+0000 7f0cdaffd700 1 --2- 172.21.15.42:0/1462476 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0cc0006fa0 0x7f0cc00094b0 secure :-1 s=THROTTLE_DONE pgs=708 cs=0 l=1 rx=0x7f0cdc11bcd0 tx=0x7f0cc4014730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:59.414 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.356+0000 7f0cd3fff700 1 -- 172.21.15.42:0/1462476 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cc4014bc0 con 0x7f0cc0006fa0 2019-12-05T05:27:59.414 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.356+0000 7f0ce1548700 1 -- 172.21.15.42:0/1462476 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cdc1312e0 con 0x7f0cc0006fa0 2019-12-05T05:27:59.414 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.356+0000 7f0cd3fff700 1 -- 172.21.15.42:0/1462476 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0cc40036f0 con 0x7f0cc0006fa0 2019-12-05T05:27:59.414 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.356+0000 7f0cdaffd700 1 --2- 172.21.15.42:0/1462476 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0cc0006fa0 0x7f0cc00094b0 secure :-1 s=THROTTLE_DONE pgs=708 cs=0 l=1 rx=0x7f0cdc11bcd0 tx=0x7f0cc4014730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:59.414 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.356+0000 7f0cd3fff700 1 -- 172.21.15.42:0/1462476 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f0cc4003a00 con 0x7f0cc0006fa0 2019-12-05T05:27:59.414 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.356+0000 7f0cdaffd700 1 --2- 172.21.15.42:0/1462476 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0cc0006fa0 0x7f0cc00094b0 secure :-1 s=THROTTLE_DONE pgs=708 cs=0 l=1 rx=0x7f0cdc11bcd0 tx=0x7f0cc4014730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:59.414 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.356+0000 7f0cd3fff700 1 --2- 172.21.15.42:0/1462476 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0cc0042440 0x7f0cc0044950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:27:59.415 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.356+0000 7f0cd3fff700 1 -- 172.21.15.42:0/1462476 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7407+0+0 (secure 0 0 0) 0x7f0cc404f090 con 0x7f0cc0006fa0 2019-12-05T05:27:59.415 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.356+0000 7f0cdb7fe700 1 --2- 172.21.15.42:0/1462476 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0cc0042440 0x7f0cc0044950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:27:59.415 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.356+0000 7f0cdb7fe700 1 --2- 172.21.15.42:0/1462476 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0cc0042440 0x7f0cc0044950 crc :-1 s=READY pgs=393 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:27:59.415 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.372+0000 7f0ce1548700 1 -- 172.21.15.42:0/1462476 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cdc05d110 con 0x7f0cc0006fa0 2019-12-05T05:27:59.415 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.384+0000 7f0cdaffd700 1 --2- 172.21.15.42:0/1462476 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0cc0006fa0 0x7f0cc00094b0 secure :-1 s=THROTTLE_DONE pgs=708 cs=0 l=1 rx=0x7f0cdc11bcd0 tx=0x7f0cc4014730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:59.415 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.384+0000 7f0cd3fff700 1 -- 172.21.15.42:0/1462476 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f0cc4036110 con 0x7f0cc0006fa0 2019-12-05T05:27:59.587 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.560+0000 7f0ce1548700 1 -- 172.21.15.42:0/1462476 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0cdc05b8c0 con 0x7f0cc0006fa0 2019-12-05T05:27:59.587 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.560+0000 7f0cdaffd700 1 --2- 172.21.15.42:0/1462476 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0cc0006fa0 0x7f0cc00094b0 secure :-1 s=THROTTLE_DONE pgs=708 cs=0 l=1 rx=0x7f0cdc11bcd0 tx=0x7f0cc4014730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:59.587 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.560+0000 7f0cd3fff700 1 -- 172.21.15.42:0/1462476 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 113 v113) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f0cc4036370 con 0x7f0cc0006fa0 2019-12-05T05:27:59.588 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 113 2019-12-05T05:27:59.588 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.564+0000 7f0ce1548700 1 -- 172.21.15.42:0/1462476 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0cc0042440 msgr2=0x7f0cc0044950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:27:59.588 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.564+0000 7f0ce1548700 1 --2- 172.21.15.42:0/1462476 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0cc0042440 0x7f0cc0044950 crc :-1 s=READY pgs=393 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:27:59.588 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.564+0000 7f0ce1548700 1 -- 172.21.15.42:0/1462476 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0cc0006fa0 msgr2=0x7f0cc00094b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:27:59.588 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.564+0000 7f0ce1548700 1 --2- 172.21.15.42:0/1462476 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0cc0006fa0 0x7f0cc00094b0 secure :-1 s=READY pgs=708 cs=0 l=1 rx=0x7f0cdc11bcd0 tx=0x7f0cc4014730).stop 2019-12-05T05:27:59.588 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.564+0000 7f0cdbfff700 1 -- 172.21.15.42:0/1462476 reap_dead start 2019-12-05T05:27:59.588 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.564+0000 7f0cdbfff700 1 -- 172.21.15.42:0/1462476 reap_dead start 2019-12-05T05:27:59.589 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.564+0000 7f0ce1548700 1 -- 172.21.15.42:0/1462476 shutdown_connections 2019-12-05T05:27:59.589 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.564+0000 7f0ce1548700 1 --2- 172.21.15.42:0/1462476 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0cc0006fa0 0x7f0cc00094b0 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:27:59.589 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.564+0000 7f0ce1548700 1 --2- 172.21.15.42:0/1462476 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0cdc0a4ae0 0x7f0cdc0a5a40 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:27:59.589 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.564+0000 7f0ce1548700 1 -- 172.21.15.42:0/1462476 shutdown_connections 2019-12-05T05:27:59.589 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.564+0000 7f0ce1548700 1 -- 172.21.15.42:0/1462476 wait complete. 2019-12-05T05:27:59.589 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.564+0000 7f0ce1548700 1 -- 172.21.15.42:0/1462476 >> 172.21.15.42:0/1462476 conn(0x7f0cdc0a9640 msgr2=0x7f0cdc09f580 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:27:59.590 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:27:59.590 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":113,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6666,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/2544447110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2544447110},{"type":"v1","addr":"172.21.15.42:6835","nonce":2544447110}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":113}],"filesystems":[{"mdsmap":{"epoch":112,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:27:13.322209+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6546},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6546":{"gid":6546,"name":"b","rank":0,"incarnation":109,"state":"up:active","state_seq":25,"addr":"172.21.15.40:6833/1149478696","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1149478696},{"type":"v1","addr":"172.21.15.40:6833","nonce":1149478696}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:27:59.590 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.b (rank=0) 2019-12-05T05:27:59.590 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2019-12-05T05:27:59.600 INFO:teuthology.orchestra.run:waiting for 300 2019-12-05T05:27:59.608 INFO:tasks.ceph.mds.b:Stopped 2019-12-05T05:27:59.608 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:27:59.711 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.700+0000 7fb0d3c81700 1 Processor -- start 2019-12-05T05:27:59.711 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.700+0000 7fb0d3c81700 1 -- start start 2019-12-05T05:27:59.711 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.704+0000 7fb0d3c81700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb0cc08df70 0x7fb0cc09f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:27:59.711 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.704+0000 7fb0d3c81700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0cc09fc50 con 0x7fb0cc0a7730 2019-12-05T05:27:59.712 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.704+0000 7fb0d3c81700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0cc0a0700 con 0x7fb0cc09f820 2019-12-05T05:27:59.712 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.704+0000 7fb0d3c81700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0cc0a0af0 con 0x7fb0cc08df70 2019-12-05T05:27:59.712 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.704+0000 7fb0d1a1d700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb0cc08df70 0x7fb0cc09f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:27:59.712 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.704+0000 7fb0d2a1f700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fb0cc09f820 0x7fb0cc14efd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:34302/0 (socket says 172.21.15.42:34302) 2019-12-05T05:27:59.712 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.704+0000 7fb0d2a1f700 1 -- 172.21.15.42:0/4077540801 learned_addr learned my addr 172.21.15.42:0/4077540801 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:27:59.712 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.704+0000 7fb0d121c700 1 -- 172.21.15.42:0/4077540801 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 649740486 0 0) 0x7fb0cc0a0700 con 0x7fb0cc09f820 2019-12-05T05:27:59.713 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.704+0000 7fb0d121c700 1 -- 172.21.15.42:0/4077540801 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0b80030b0 con 0x7fb0cc09f820 2019-12-05T05:27:59.713 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.704+0000 7fb0d1a1d700 1 -- 172.21.15.42:0/4077540801 >> v1:172.21.15.42:6789/0 conn(0x7fb0cc09f820 legacy=0x7fb0cc14efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:27:59.713 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.704+0000 7fb0d1a1d700 1 -- 172.21.15.42:0/4077540801 >> v1:172.21.15.40:6789/0 conn(0x7fb0cc0a7730 legacy=0x7fb0cc07e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:27:59.713 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.704+0000 7fb0d1a1d700 1 -- 172.21.15.42:0/4077540801 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0cc07ef10 con 0x7fb0cc08df70 2019-12-05T05:27:59.713 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.704+0000 7fb0d1a1d700 1 --2- 172.21.15.42:0/4077540801 >> v2:172.21.15.40:3300/0 conn(0x7fb0cc08df70 0x7fb0cc09f2d0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fb0c0008790 tx=0x7fb0c000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:59.713 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.704+0000 7fb0d1a1d700 1 --2- 172.21.15.42:0/4077540801 >> v2:172.21.15.40:3300/0 conn(0x7fb0cc08df70 0x7fb0cc09f2d0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fb0c0008790 tx=0x7fb0c000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:59.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.704+0000 7fb0d1a1d700 1 --2- 172.21.15.42:0/4077540801 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb0cc08df70 0x7fb0cc09f2d0 secure :-1 s=READY pgs=710 cs=0 l=1 rx=0x7fb0c0008790 tx=0x7fb0c000fbf0).ready entity=mon.0 client_cookie=b3b22b481dab2d9 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:27:59.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.704+0000 7fb0d1a1d700 1 --2- 172.21.15.42:0/4077540801 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb0cc08df70 0x7fb0cc09f2d0 secure :-1 s=THROTTLE_DONE pgs=710 cs=0 l=1 rx=0x7fb0c0008790 tx=0x7fb0c000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:59.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.704+0000 7fb0d121c700 1 -- 172.21.15.42:0/4077540801 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb0c0011270 con 0x7fb0cc08df70 2019-12-05T05:27:59.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.704+0000 7fb0d1a1d700 1 --2- 172.21.15.42:0/4077540801 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb0cc08df70 0x7fb0cc09f2d0 secure :-1 s=THROTTLE_DONE pgs=710 cs=0 l=1 rx=0x7fb0c0008790 tx=0x7fb0c000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:59.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.704+0000 7fb0d1a1d700 1 --2- 172.21.15.42:0/4077540801 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb0cc08df70 0x7fb0cc09f2d0 secure :-1 s=THROTTLE_DONE pgs=710 cs=0 l=1 rx=0x7fb0c0008790 tx=0x7fb0c000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:59.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.704+0000 7fb0d121c700 1 -- 172.21.15.42:0/4077540801 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0c0008bc0 con 0x7fb0cc08df70 2019-12-05T05:27:59.715 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.704+0000 7fb0d121c700 1 -- 172.21.15.42:0/4077540801 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb0c0011850 con 0x7fb0cc08df70 2019-12-05T05:27:59.715 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.704+0000 7fb0d3c81700 1 -- 172.21.15.42:0/4077540801 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb0cc08df70 msgr2=0x7fb0cc09f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:27:59.715 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.704+0000 7fb0d3c81700 1 --2- 172.21.15.42:0/4077540801 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb0cc08df70 0x7fb0cc09f2d0 secure :-1 s=READY pgs=710 cs=0 l=1 rx=0x7fb0c0008790 tx=0x7fb0c000fbf0).stop 2019-12-05T05:27:59.715 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.704+0000 7fb0d3c81700 1 -- 172.21.15.42:0/4077540801 shutdown_connections 2019-12-05T05:27:59.715 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.704+0000 7fb0d3c81700 1 --2- 172.21.15.42:0/4077540801 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb0cc08df70 0x7fb0cc09f2d0 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:27:59.716 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.704+0000 7fb0d3c81700 1 -- 172.21.15.42:0/4077540801 shutdown_connections 2019-12-05T05:27:59.716 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.704+0000 7fb0d3c81700 1 -- 172.21.15.42:0/4077540801 wait complete. 2019-12-05T05:27:59.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.704+0000 7fb0d3c81700 1 -- 172.21.15.42:0/4077540801 >> 172.21.15.42:0/4077540801 conn(0x7fb0cc0aabd0 msgr2=0x7fb0cc0a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:27:59.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.708+0000 7fb0d3c81700 1 Processor -- start 2019-12-05T05:27:59.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.708+0000 7fb0d3c81700 1 -- start start 2019-12-05T05:27:59.724 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.708+0000 7fb0d3c81700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb0cc11a7c0 0x7fb0cc07da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:27:59.724 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.708+0000 7fb0d3c81700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb0cc119580 0x7fb0cc137910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:27:59.724 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.708+0000 7fb0d3c81700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0cc0a1680 con 0x7fb0cc07dfc0 2019-12-05T05:27:59.724 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.708+0000 7fb0d3c81700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0cc11b620 con 0x7fb0cc119580 2019-12-05T05:27:59.724 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.708+0000 7fb0d3c81700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0cc11b8c0 con 0x7fb0cc11a7c0 2019-12-05T05:27:59.725 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.708+0000 7fb0d221e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb0cc11a7c0 0x7fb0cc07da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:27:59.725 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.708+0000 7fb0d221e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb0cc11a7c0 0x7fb0cc07da70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:34266/0 (socket says 172.21.15.42:34266) 2019-12-05T05:27:59.725 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.708+0000 7fb0d221e700 1 -- 172.21.15.42:0/936695648 learned_addr learned my addr 172.21.15.42:0/936695648 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:27:59.725 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.708+0000 7fb0d221e700 1 -- 172.21.15.42:0/936695648 >> v2:172.21.15.42:3300/0 conn(0x7fb0cc11a7c0 msgr2=0x7fb0cc07da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2019-12-05T05:27:59.725 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.708+0000 7fb0d221e700 1 -- 172.21.15.42:0/936695648 >> v2:172.21.15.42:3300/0 conn(0x7fb0cc11a7c0 msgr2=0x7fb0cc07da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2019-12-05T05:27:59.725 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.708+0000 7fb0d221e700 1 --2- 172.21.15.42:0/936695648 >> v2:172.21.15.42:3300/0 conn(0x7fb0cc11a7c0 0x7fb0cc07da70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-12-05T05:27:59.725 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.708+0000 7fb0d221e700 1 --2- 172.21.15.42:0/936695648 >> v2:172.21.15.42:3300/0 conn(0x7fb0cc11a7c0 0x7fb0cc07da70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rx=0 tx=0)._fault waiting 0.200000 2019-12-05T05:27:59.726 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.708+0000 7fb0d2a1f700 1 --2- 172.21.15.42:0/936695648 >> v2:172.21.15.40:3300/0 conn(0x7fb0cc119580 0x7fb0cc137910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:27:59.726 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.708+0000 7fb0beffd700 1 -- 172.21.15.42:0/936695648 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2518085753 0 0) 0x7fb0cc0a1680 con 0x7fb0cc07dfc0 2019-12-05T05:27:59.726 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.712+0000 7fb0beffd700 1 -- 172.21.15.42:0/936695648 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0b40030b0 con 0x7fb0cc07dfc0 2019-12-05T05:27:59.726 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.712+0000 7fb0beffd700 1 -- 172.21.15.42:0/936695648 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 2744015020 0 0) 0x7fb0b40030b0 con 0x7fb0cc07dfc0 2019-12-05T05:27:59.726 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.712+0000 7fb0beffd700 1 -- 172.21.15.42:0/936695648 --> v1:172.21.15.42:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb0cc0a1680 con 0x7fb0cc07dfc0 2019-12-05T05:27:59.726 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.712+0000 7fb0beffd700 1 -- 172.21.15.42:0/936695648 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7fb0c0011270 con 0x7fb0cc07dfc0 2019-12-05T05:27:59.727 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.712+0000 7fb0beffd700 1 -- 172.21.15.42:0/936695648 >> v2:172.21.15.42:3300/0 conn(0x7fb0cc11a7c0 msgr2=0x7fb0cc07da70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2019-12-05T05:27:59.727 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.712+0000 7fb0beffd700 1 --2- 172.21.15.42:0/936695648 >> v2:172.21.15.42:3300/0 conn(0x7fb0cc11a7c0 0x7fb0cc07da70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:27:59.727 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.712+0000 7fb0beffd700 1 -- 172.21.15.42:0/936695648 >> v2:172.21.15.40:3300/0 conn(0x7fb0cc119580 msgr2=0x7fb0cc137910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:27:59.727 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.712+0000 7fb0beffd700 1 --2- 172.21.15.42:0/936695648 >> v2:172.21.15.40:3300/0 conn(0x7fb0cc119580 0x7fb0cc137910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:27:59.727 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.712+0000 7fb0beffd700 1 -- 172.21.15.42:0/936695648 >> v1:172.21.15.42:6789/0 conn(0x7fb0cc07dfc0 legacy=0x7fb0cc0a1950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:27:59.727 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.712+0000 7fb0beffd700 1 --2- 172.21.15.42:0/936695648 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb0b40038a0 0x7fb0b4003cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:27:59.728 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.712+0000 7fb0beffd700 1 --2- 172.21.15.42:0/936695648 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb0b4004420 0x7fb0b4004850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:27:59.728 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.712+0000 7fb0beffd700 1 --2- 172.21.15.42:0/936695648 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb0b4004e90 0x7fb0b400b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:27:59.728 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.712+0000 7fb0beffd700 1 -- 172.21.15.42:0/936695648 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0cc11b620 con 0x7fb0b4004420 2019-12-05T05:27:59.728 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.712+0000 7fb0d221e700 1 --2- 172.21.15.42:0/936695648 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb0b40038a0 0x7fb0b4003cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:27:59.728 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.712+0000 7fb0beffd700 1 -- 172.21.15.42:0/936695648 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0cc11b8c0 con 0x7fb0b40038a0 2019-12-05T05:27:59.729 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.712+0000 7fb0beffd700 1 -- 172.21.15.42:0/936695648 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0b400bd50 con 0x7fb0b4004e90 2019-12-05T05:27:59.730 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.712+0000 7fb0d2a1f700 1 --2- 172.21.15.42:0/936695648 >> v2:172.21.15.40:3300/0 conn(0x7fb0cc119580 0x7fb0cc137910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:27:59.730 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.712+0000 7fb0d1a1d700 1 --2- 172.21.15.42:0/936695648 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb0b4004420 0x7fb0b4004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:27:59.730 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.712+0000 7fb0d2a1f700 1 --2- 172.21.15.42:0/936695648 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb0b4004e90 0x7fb0b400b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:27:59.730 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.712+0000 7fb0d221e700 1 -- 172.21.15.42:0/936695648 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb0b4004e90 msgr2=0x7fb0b400b670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:27:59.730 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.712+0000 7fb0d221e700 1 --2- 172.21.15.42:0/936695648 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb0b4004e90 0x7fb0b400b670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:27:59.730 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.712+0000 7fb0d221e700 1 -- 172.21.15.42:0/936695648 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb0b4004420 msgr2=0x7fb0b4004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:27:59.731 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.712+0000 7fb0d221e700 1 --2- 172.21.15.42:0/936695648 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb0b4004420 0x7fb0b4004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:27:59.731 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.712+0000 7fb0d221e700 1 -- 172.21.15.42:0/936695648 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0cc118ec0 con 0x7fb0b40038a0 2019-12-05T05:27:59.731 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.712+0000 7fb0d3c81700 1 -- 172.21.15.42:0/936695648 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0cc117190 con 0x7fb0b40038a0 2019-12-05T05:27:59.731 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.712+0000 7fb0d221e700 1 --2- 172.21.15.42:0/936695648 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb0b40038a0 0x7fb0b4003cf0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fb0c8000e90 tx=0x7fb0c8012f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:59.731 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.712+0000 7fb0d2a1f700 1 --2- 172.21.15.42:0/936695648 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb0b4004e90 0x7fb0b400b670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).send_auth_request state changed! 2019-12-05T05:27:59.731 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.712+0000 7fb0d2a1f700 1 -- 172.21.15.42:0/936695648 reap_dead start 2019-12-05T05:27:59.732 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.712+0000 7fb0d3c81700 1 -- 172.21.15.42:0/936695648 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0cc11b160 con 0x7fb0b40038a0 2019-12-05T05:27:59.732 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.712+0000 7fb0d221e700 1 --2- 172.21.15.42:0/936695648 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb0b40038a0 0x7fb0b4003cf0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fb0c8000e90 tx=0x7fb0c8012f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:59.732 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.712+0000 7fb0d221e700 1 --2- 172.21.15.42:0/936695648 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb0b40038a0 0x7fb0b4003cf0 secure :-1 s=READY pgs=902 cs=0 l=1 rx=0x7fb0c8000e90 tx=0x7fb0c8012f70).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:27:59.732 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.712+0000 7fb0d221e700 1 --2- 172.21.15.42:0/936695648 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb0b40038a0 0x7fb0b4003cf0 secure :-1 s=THROTTLE_DONE pgs=902 cs=0 l=1 rx=0x7fb0c8000e90 tx=0x7fb0c8012f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:59.732 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.712+0000 7fb0d221e700 1 --2- 172.21.15.42:0/936695648 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb0b40038a0 0x7fb0b4003cf0 secure :-1 s=THROTTLE_DONE pgs=902 cs=0 l=1 rx=0x7fb0c8000e90 tx=0x7fb0c8012f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:59.732 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.712+0000 7fb0beffd700 1 -- 172.21.15.42:0/936695648 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb0c8008950 con 0x7fb0b40038a0 2019-12-05T05:27:59.733 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.712+0000 7fb0d221e700 1 --2- 172.21.15.42:0/936695648 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb0b40038a0 0x7fb0b4003cf0 secure :-1 s=THROTTLE_DONE pgs=902 cs=0 l=1 rx=0x7fb0c8000e90 tx=0x7fb0c8012f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:59.733 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.712+0000 7fb0beffd700 1 -- 172.21.15.42:0/936695648 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0c8008570 con 0x7fb0b40038a0 2019-12-05T05:27:59.733 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.712+0000 7fb0beffd700 1 -- 172.21.15.42:0/936695648 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb0c8003160 con 0x7fb0b40038a0 2019-12-05T05:27:59.733 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.712+0000 7fb0d221e700 1 --2- 172.21.15.42:0/936695648 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb0b40038a0 0x7fb0b4003cf0 secure :-1 s=THROTTLE_DONE pgs=902 cs=0 l=1 rx=0x7fb0c8000e90 tx=0x7fb0c8012f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:59.733 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.712+0000 7fb0beffd700 1 -- 172.21.15.42:0/936695648 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fb0c8003360 con 0x7fb0b40038a0 2019-12-05T05:27:59.733 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.712+0000 7fb0d221e700 1 --2- 172.21.15.42:0/936695648 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb0b40038a0 0x7fb0b4003cf0 secure :-1 s=THROTTLE_DONE pgs=902 cs=0 l=1 rx=0x7fb0c8000e90 tx=0x7fb0c8012f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:59.734 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.716+0000 7fb0beffd700 1 --2- 172.21.15.42:0/936695648 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb0b403e8b0 0x7fb0b4040dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:27:59.734 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.716+0000 7fb0beffd700 1 -- 172.21.15.42:0/936695648 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7407+0+0 (secure 0 0 0) 0x7fb0c804cd00 con 0x7fb0b40038a0 2019-12-05T05:27:59.734 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.716+0000 7fb0d1a1d700 1 --2- 172.21.15.42:0/936695648 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb0b403e8b0 0x7fb0b4040dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:27:59.734 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.716+0000 7fb0d1a1d700 1 --2- 172.21.15.42:0/936695648 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb0b403e8b0 0x7fb0b4040dc0 crc :-1 s=READY pgs=394 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:27:59.734 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.716+0000 7fb0d3c81700 1 -- 172.21.15.42:0/936695648 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0a8003200 con 0x7fb0b40038a0 2019-12-05T05:27:59.735 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.720+0000 7fb0d221e700 1 --2- 172.21.15.42:0/936695648 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb0b40038a0 0x7fb0b4003cf0 secure :-1 s=THROTTLE_DONE pgs=902 cs=0 l=1 rx=0x7fb0c8000e90 tx=0x7fb0c8012f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:59.735 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.720+0000 7fb0beffd700 1 -- 172.21.15.42:0/936695648 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fb0c8033d80 con 0x7fb0b40038a0 2019-12-05T05:27:59.889 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.884+0000 7fb0d3c81700 1 -- 172.21.15.42:0/936695648 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb0a8003630 con 0x7fb0b40038a0 2019-12-05T05:27:59.890 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.884+0000 7fb0d221e700 1 --2- 172.21.15.42:0/936695648 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb0b40038a0 0x7fb0b4003cf0 secure :-1 s=THROTTLE_DONE pgs=902 cs=0 l=1 rx=0x7fb0c8000e90 tx=0x7fb0c8012f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:27:59.890 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.884+0000 7fb0beffd700 1 -- 172.21.15.42:0/936695648 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 113 v113) v1 ==== 95+0+2112 (secure 0 0 0) 0x7fb0c8033fe0 con 0x7fb0b40038a0 2019-12-05T05:27:59.890 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 113 2019-12-05T05:27:59.890 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:27:59.890 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":113,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6666,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/2544447110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2544447110},{"type":"v1","addr":"172.21.15.42:6835","nonce":2544447110}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":113}],"filesystems":[{"mdsmap":{"epoch":112,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:27:13.322209+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6546},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6546":{"gid":6546,"name":"b","rank":0,"incarnation":109,"state":"up:active","state_seq":25,"addr":"172.21.15.40:6833/1149478696","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1149478696},{"type":"v1","addr":"172.21.15.40:6833","nonce":1149478696}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:27:59.893 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.884+0000 7fb0d3c81700 1 -- 172.21.15.42:0/936695648 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb0b403e8b0 msgr2=0x7fb0b4040dc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:27:59.893 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.884+0000 7fb0d3c81700 1 --2- 172.21.15.42:0/936695648 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb0b403e8b0 0x7fb0b4040dc0 crc :-1 s=READY pgs=394 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:27:59.893 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.884+0000 7fb0d3c81700 1 -- 172.21.15.42:0/936695648 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb0b40038a0 msgr2=0x7fb0b4003cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:27:59.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.884+0000 7fb0d3c81700 1 --2- 172.21.15.42:0/936695648 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb0b40038a0 0x7fb0b4003cf0 secure :-1 s=READY pgs=902 cs=0 l=1 rx=0x7fb0c8000e90 tx=0x7fb0c8012f70).stop 2019-12-05T05:27:59.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.888+0000 7fb0d3c81700 1 -- 172.21.15.42:0/936695648 shutdown_connections 2019-12-05T05:27:59.895 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.888+0000 7fb0d3c81700 1 --2- 172.21.15.42:0/936695648 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb0b403e8b0 0x7fb0b4040dc0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:27:59.895 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.888+0000 7fb0d3c81700 1 --2- 172.21.15.42:0/936695648 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb0b40038a0 0x7fb0b4003cf0 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:27:59.895 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.888+0000 7fb0d3c81700 1 -- 172.21.15.42:0/936695648 shutdown_connections 2019-12-05T05:27:59.895 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.888+0000 7fb0d3c81700 1 -- 172.21.15.42:0/936695648 wait complete. 2019-12-05T05:27:59.911 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:27:59.888+0000 7fb0d3c81700 1 -- 172.21.15.42:0/936695648 >> 172.21.15.42:0/936695648 conn(0x7fb0cc09f490 msgr2=0x7fb0cc07e550 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:27:59.931 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T05:28:01.932 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:28:02.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:02.028+0000 7f9e192d8700 1 Processor -- start 2019-12-05T05:28:02.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:02.028+0000 7f9e192d8700 1 -- start start 2019-12-05T05:28:02.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:02.028+0000 7f9e192d8700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9e140a7740 0x7f9e1408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:28:02.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:02.028+0000 7f9e192d8700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f9e1409fae0 0x7f9e1407eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:28:02.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:02.028+0000 7f9e192d8700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e140a0710 con 0x7f9e1407e720 2019-12-05T05:28:02.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:02.028+0000 7f9e192d8700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e140a0b00 con 0x7f9e1409fae0 2019-12-05T05:28:02.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:02.028+0000 7f9e192d8700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e1408e5b0 con 0x7f9e140a7740 2019-12-05T05:28:02.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:02.028+0000 7f9e12ffd700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f9e1407e720 0x7f9e1409f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:34326/0 (socket says 172.21.15.42:34326) 2019-12-05T05:28:02.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:02.028+0000 7f9e12ffd700 1 -- 172.21.15.42:0/1067545595 learned_addr learned my addr 172.21.15.42:0/1067545595 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:28:02.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:02.028+0000 7f9e137fe700 1 --2- 172.21.15.42:0/1067545595 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9e140a7740 0x7f9e1408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:28:02.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:02.028+0000 7f9e13fff700 1 --2- 172.21.15.42:0/1067545595 >> v2:172.21.15.40:3300/0 conn(0x7f9e1409fae0 0x7f9e1407eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:28:02.039 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:02.028+0000 7f9e127fc700 1 -- 172.21.15.42:0/1067545595 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2742721043 0 0) 0x7f9e140a0710 con 0x7f9e1407e720 2019-12-05T05:28:02.039 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:02.028+0000 7f9e127fc700 1 -- 172.21.15.42:0/1067545595 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e000030b0 con 0x7f9e1407e720 2019-12-05T05:28:02.039 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:02.028+0000 7f9e137fe700 1 -- 172.21.15.42:0/1067545595 >> v2:172.21.15.40:3300/0 conn(0x7f9e1409fae0 msgr2=0x7f9e1407eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:28:02.039 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:02.028+0000 7f9e137fe700 1 --2- 172.21.15.42:0/1067545595 >> v2:172.21.15.40:3300/0 conn(0x7f9e1409fae0 0x7f9e1407eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:28:02.039 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:02.028+0000 7f9e137fe700 1 -- 172.21.15.42:0/1067545595 >> v1:172.21.15.42:6789/0 conn(0x7f9e1407e720 legacy=0x7f9e1409f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:28:02.039 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:02.028+0000 7f9e137fe700 1 -- 172.21.15.42:0/1067545595 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e1407f0a0 con 0x7f9e140a7740 2019-12-05T05:28:02.039 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:02.028+0000 7f9e137fe700 1 --2- 172.21.15.42:0/1067545595 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9e140a7740 0x7f9e1408df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f9e0c008b10 tx=0x7f9e0c00fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:02.040 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:02.028+0000 7f9e137fe700 1 --2- 172.21.15.42:0/1067545595 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9e140a7740 0x7f9e1408df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f9e0c008b10 tx=0x7f9e0c00fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:02.040 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:02.028+0000 7f9e137fe700 1 --2- 172.21.15.42:0/1067545595 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9e140a7740 0x7f9e1408df80 secure :-1 s=READY pgs=183 cs=0 l=1 rx=0x7f9e0c008b10 tx=0x7f9e0c00fff0).ready entity=mon.2 client_cookie=d54436b60dba5ddb server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:28:02.040 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:02.028+0000 7f9e137fe700 1 --2- 172.21.15.42:0/1067545595 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9e140a7740 0x7f9e1408df80 secure :-1 s=THROTTLE_DONE pgs=183 cs=0 l=1 rx=0x7f9e0c008b10 tx=0x7f9e0c00fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:02.041 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:02.028+0000 7f9e127fc700 1 -- 172.21.15.42:0/1067545595 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9e0c010900 con 0x7f9e140a7740 2019-12-05T05:28:02.041 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:02.028+0000 7f9e137fe700 1 --2- 172.21.15.42:0/1067545595 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9e140a7740 0x7f9e1408df80 secure :-1 s=THROTTLE_DONE pgs=183 cs=0 l=1 rx=0x7f9e0c008b10 tx=0x7f9e0c00fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:02.041 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:02.028+0000 7f9e137fe700 1 --2- 172.21.15.42:0/1067545595 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9e140a7740 0x7f9e1408df80 secure :-1 s=THROTTLE_DONE pgs=183 cs=0 l=1 rx=0x7f9e0c008b10 tx=0x7f9e0c00fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:02.041 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:02.028+0000 7f9e127fc700 1 -- 172.21.15.42:0/1067545595 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e0c008b50 con 0x7f9e140a7740 2019-12-05T05:28:02.041 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:02.028+0000 7f9e127fc700 1 -- 172.21.15.42:0/1067545595 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9e0c0110c0 con 0x7f9e140a7740 2019-12-05T05:28:02.041 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:02.028+0000 7f9e192d8700 1 -- 172.21.15.42:0/1067545595 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9e140a7740 msgr2=0x7f9e1408df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:28:02.041 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:02.028+0000 7f9e192d8700 1 --2- 172.21.15.42:0/1067545595 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9e140a7740 0x7f9e1408df80 secure :-1 s=READY pgs=183 cs=0 l=1 rx=0x7f9e0c008b10 tx=0x7f9e0c00fff0).stop 2019-12-05T05:28:02.042 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:02.032+0000 7f9e192d8700 1 -- 172.21.15.42:0/1067545595 shutdown_connections 2019-12-05T05:28:02.042 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:02.032+0000 7f9e192d8700 1 --2- 172.21.15.42:0/1067545595 >> v2:172.21.15.40:3300/0 conn(0x7f9e1409fae0 0x7f9e1407eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:28:02.042 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:02.032+0000 7f9e192d8700 1 --2- 172.21.15.42:0/1067545595 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9e140a7740 0x7f9e1408df80 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:28:02.042 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:02.032+0000 7f9e192d8700 1 -- 172.21.15.42:0/1067545595 shutdown_connections 2019-12-05T05:28:02.042 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:02.032+0000 7f9e192d8700 1 -- 172.21.15.42:0/1067545595 wait complete. 2019-12-05T05:28:02.042 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:02.032+0000 7f9e192d8700 1 -- 172.21.15.42:0/1067545595 >> 172.21.15.42:0/1067545595 conn(0x7f9e140aabe0 msgr2=0x7f9e140a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:28:02.046 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:02.032+0000 7f9e192d8700 1 Processor -- start 2019-12-05T05:28:02.046 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:02.032+0000 7f9e192d8700 1 -- start start 2019-12-05T05:28:02.046 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:02.036+0000 7f9e192d8700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f9e14124410 0x7f9e1407d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:28:02.047 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:02.036+0000 7f9e192d8700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f9e1411b480 0x7f9e1411b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:28:02.047 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:02.036+0000 7f9e192d8700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9e14117610 0x7f9e14117a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:28:02.047 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:02.036+0000 7f9e192d8700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e1411c2d0 con 0x7f9e14124410 2019-12-05T05:28:02.047 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:02.036+0000 7f9e192d8700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e1411c030 con 0x7f9e1411b480 2019-12-05T05:28:02.047 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:02.036+0000 7f9e192d8700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e14117370 con 0x7f9e14117610 2019-12-05T05:28:02.047 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:02.036+0000 7f9e12ffd700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f9e1411b480 0x7f9e1411b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:28:02.047 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:02.036+0000 7f9e12ffd700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f9e1411b480 0x7f9e1411b8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:34292/0 (socket says 172.21.15.42:34292) 2019-12-05T05:28:02.048 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:02.036+0000 7f9e12ffd700 1 -- 172.21.15.42:0/1528600832 learned_addr learned my addr 172.21.15.42:0/1528600832 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:28:02.048 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:02.036+0000 7f9e13fff700 1 --2- 172.21.15.42:0/1528600832 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9e14117610 0x7f9e14117a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:28:02.048 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:02.036+0000 7f9e137fe700 1 --2- 172.21.15.42:0/1528600832 >> v2:172.21.15.40:3300/0 conn(0x7f9e14124410 0x7f9e1407d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:28:02.048 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:02.036+0000 7f9e12ffd700 1 -- 172.21.15.42:0/1528600832 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9e14117610 msgr2=0x7f9e14117a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:28:02.048 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:02.036+0000 7f9e12ffd700 1 --2- 172.21.15.42:0/1528600832 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9e14117610 0x7f9e14117a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:28:02.048 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:02.036+0000 7f9e12ffd700 1 -- 172.21.15.42:0/1528600832 >> v2:172.21.15.40:3300/0 conn(0x7f9e14124410 msgr2=0x7f9e1407d920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:28:02.048 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:02.036+0000 7f9e12ffd700 1 --2- 172.21.15.42:0/1528600832 >> v2:172.21.15.40:3300/0 conn(0x7f9e14124410 0x7f9e1407d920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:28:02.049 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:02.036+0000 7f9e12ffd700 1 -- 172.21.15.42:0/1528600832 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e1411add0 con 0x7f9e1411b480 2019-12-05T05:28:02.049 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:02.036+0000 7f9e12ffd700 1 --2- 172.21.15.42:0/1528600832 >> v2:172.21.15.42:3300/0 conn(0x7f9e1411b480 0x7f9e1411b8b0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f9e1411c1f0 tx=0x7f9dfc010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:02.049 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:02.036+0000 7f9e12ffd700 1 --2- 172.21.15.42:0/1528600832 >> v2:172.21.15.42:3300/0 conn(0x7f9e1411b480 0x7f9e1411b8b0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f9e1411c1f0 tx=0x7f9dfc010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:02.049 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:02.036+0000 7f9e12ffd700 1 --2- 172.21.15.42:0/1528600832 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9e1411b480 0x7f9e1411b8b0 secure :-1 s=READY pgs=904 cs=0 l=1 rx=0x7f9e1411c1f0 tx=0x7f9dfc010ab0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:28:02.049 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:02.036+0000 7f9e12ffd700 1 --2- 172.21.15.42:0/1528600832 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9e1411b480 0x7f9e1411b8b0 secure :-1 s=THROTTLE_DONE pgs=904 cs=0 l=1 rx=0x7f9e1411c1f0 tx=0x7f9dfc010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:02.049 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:02.036+0000 7f9e0bfff700 1 -- 172.21.15.42:0/1528600832 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9dfc00cab0 con 0x7f9e1411b480 2019-12-05T05:28:02.049 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:02.036+0000 7f9e12ffd700 1 --2- 172.21.15.42:0/1528600832 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9e1411b480 0x7f9e1411b8b0 secure :-1 s=THROTTLE_DONE pgs=904 cs=0 l=1 rx=0x7f9e1411c1f0 tx=0x7f9dfc010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:02.050 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:02.036+0000 7f9e12ffd700 1 --2- 172.21.15.42:0/1528600832 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9e1411b480 0x7f9e1411b8b0 secure :-1 s=THROTTLE_DONE pgs=904 cs=0 l=1 rx=0x7f9e1411c1f0 tx=0x7f9dfc010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:02.050 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:02.036+0000 7f9e0bfff700 1 -- 172.21.15.42:0/1528600832 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9dfc010250 con 0x7f9e1411b480 2019-12-05T05:28:02.050 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:02.036+0000 7f9e192d8700 1 -- 172.21.15.42:0/1528600832 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e14117f90 con 0x7f9e1411b480 2019-12-05T05:28:02.050 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:02.036+0000 7f9e0bfff700 1 -- 172.21.15.42:0/1528600832 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9dfc00d2b0 con 0x7f9e1411b480 2019-12-05T05:28:02.050 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:02.036+0000 7f9e192d8700 1 -- 172.21.15.42:0/1528600832 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e14118f80 con 0x7f9e1411b480 2019-12-05T05:28:02.053 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:02.036+0000 7f9e12ffd700 1 --2- 172.21.15.42:0/1528600832 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9e1411b480 0x7f9e1411b8b0 secure :-1 s=THROTTLE_DONE pgs=904 cs=0 l=1 rx=0x7f9e1411c1f0 tx=0x7f9dfc010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:02.053 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:02.040+0000 7f9e0bfff700 1 -- 172.21.15.42:0/1528600832 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f9dfc00d4b0 con 0x7f9e1411b480 2019-12-05T05:28:02.053 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:02.040+0000 7f9e12ffd700 1 --2- 172.21.15.42:0/1528600832 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9e1411b480 0x7f9e1411b8b0 secure :-1 s=THROTTLE_DONE pgs=904 cs=0 l=1 rx=0x7f9e1411c1f0 tx=0x7f9dfc010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:02.053 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:02.040+0000 7f9e0bfff700 1 --2- 172.21.15.42:0/1528600832 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9df803ae60 0x7f9df803d370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:28:02.053 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:02.040+0000 7f9e0bfff700 1 -- 172.21.15.42:0/1528600832 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7407+0+0 (secure 0 0 0) 0x7f9dfc04ac70 con 0x7f9e1411b480 2019-12-05T05:28:02.054 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:02.040+0000 7f9e137fe700 1 --2- 172.21.15.42:0/1528600832 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9df803ae60 0x7f9df803d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:28:02.054 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:02.040+0000 7f9e137fe700 1 --2- 172.21.15.42:0/1528600832 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9df803ae60 0x7f9df803d370 crc :-1 s=READY pgs=395 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:28:02.055 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:02.044+0000 7f9e192d8700 1 -- 172.21.15.42:0/1528600832 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e1405d110 con 0x7f9e1411b480 2019-12-05T05:28:02.055 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:02.048+0000 7f9e12ffd700 1 --2- 172.21.15.42:0/1528600832 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9e1411b480 0x7f9e1411b8b0 secure :-1 s=THROTTLE_DONE pgs=904 cs=0 l=1 rx=0x7f9e1411c1f0 tx=0x7f9dfc010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:02.055 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:02.048+0000 7f9e0bfff700 1 -- 172.21.15.42:0/1528600832 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f9dfc031c90 con 0x7f9e1411b480 2019-12-05T05:28:02.221 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:02.216+0000 7f9e192d8700 1 -- 172.21.15.42:0/1528600832 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9e1405b8c0 con 0x7f9e1411b480 2019-12-05T05:28:02.223 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:02.216+0000 7f9e12ffd700 1 --2- 172.21.15.42:0/1528600832 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9e1411b480 0x7f9e1411b8b0 secure :-1 s=THROTTLE_DONE pgs=904 cs=0 l=1 rx=0x7f9e1411c1f0 tx=0x7f9dfc010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:02.223 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:02.216+0000 7f9e0bfff700 1 -- 172.21.15.42:0/1528600832 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 113 v113) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f9dfc031ef0 con 0x7f9e1411b480 2019-12-05T05:28:02.223 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 113 2019-12-05T05:28:02.223 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:28:02.223 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":113,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6666,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/2544447110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2544447110},{"type":"v1","addr":"172.21.15.42:6835","nonce":2544447110}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":113}],"filesystems":[{"mdsmap":{"epoch":112,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:27:13.322209+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6546},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6546":{"gid":6546,"name":"b","rank":0,"incarnation":109,"state":"up:active","state_seq":25,"addr":"172.21.15.40:6833/1149478696","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1149478696},{"type":"v1","addr":"172.21.15.40:6833","nonce":1149478696}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:28:02.225 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:02.220+0000 7f9e192d8700 1 -- 172.21.15.42:0/1528600832 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9df803ae60 msgr2=0x7f9df803d370 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:28:02.225 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:02.220+0000 7f9e192d8700 1 --2- 172.21.15.42:0/1528600832 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9df803ae60 0x7f9df803d370 crc :-1 s=READY pgs=395 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:28:02.225 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:02.220+0000 7f9e192d8700 1 -- 172.21.15.42:0/1528600832 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9e1411b480 msgr2=0x7f9e1411b8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:28:02.226 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:02.220+0000 7f9e192d8700 1 --2- 172.21.15.42:0/1528600832 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9e1411b480 0x7f9e1411b8b0 secure :-1 s=READY pgs=904 cs=0 l=1 rx=0x7f9e1411c1f0 tx=0x7f9dfc010ab0).stop 2019-12-05T05:28:02.226 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:02.220+0000 7f9e192d8700 1 -- 172.21.15.42:0/1528600832 shutdown_connections 2019-12-05T05:28:02.226 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:02.220+0000 7f9e192d8700 1 --2- 172.21.15.42:0/1528600832 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9df803ae60 0x7f9df803d370 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:28:02.226 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:02.220+0000 7f9e192d8700 1 --2- 172.21.15.42:0/1528600832 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9e14117610 0x7f9e14117a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:28:02.226 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:02.220+0000 7f9e192d8700 1 --2- 172.21.15.42:0/1528600832 >> v2:172.21.15.40:3300/0 conn(0x7f9e14124410 0x7f9e1407d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:28:02.226 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:02.220+0000 7f9e192d8700 1 --2- 172.21.15.42:0/1528600832 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9e1411b480 0x7f9e1411b8b0 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:28:02.226 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:02.220+0000 7f9e192d8700 1 -- 172.21.15.42:0/1528600832 shutdown_connections 2019-12-05T05:28:02.227 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:02.220+0000 7f9e192d8700 1 -- 172.21.15.42:0/1528600832 wait complete. 2019-12-05T05:28:02.227 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:02.220+0000 7f9e192d8700 1 -- 172.21.15.42:0/1528600832 >> 172.21.15.42:0/1528600832 conn(0x7f9e140a9790 msgr2=0x7f9e1409f380 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:28:02.234 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T05:28:04.236 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:28:04.342 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:04.332+0000 7fa1c2b94700 1 Processor -- start 2019-12-05T05:28:04.342 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:04.332+0000 7fa1c2b94700 1 -- start start 2019-12-05T05:28:04.343 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:04.332+0000 7fa1c2b94700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa1bc0a7740 0x7fa1bc08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:28:04.343 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:04.332+0000 7fa1c2b94700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1bc0a0710 con 0x7fa1bc07e720 2019-12-05T05:28:04.343 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:04.332+0000 7fa1c2b94700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1bc0a9ba0 con 0x7fa1bc09fae0 2019-12-05T05:28:04.343 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:04.332+0000 7fa1c2b94700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1bc0a0b00 con 0x7fa1bc0a7740 2019-12-05T05:28:04.343 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:04.332+0000 7fa1c1932700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fa1bc09fae0 0x7fa1bc14efe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:34344/0 (socket says 172.21.15.42:34344) 2019-12-05T05:28:04.343 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:04.332+0000 7fa1c1932700 1 -- 172.21.15.42:0/2557019902 learned_addr learned my addr 172.21.15.42:0/2557019902 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:28:04.344 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:04.332+0000 7fa1c1131700 1 --2- 172.21.15.42:0/2557019902 >> v2:172.21.15.40:3300/0 conn(0x7fa1bc0a7740 0x7fa1bc08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:28:04.344 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:04.332+0000 7fa1affff700 1 -- 172.21.15.42:0/2557019902 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4095748601 0 0) 0x7fa1bc0a9ba0 con 0x7fa1bc09fae0 2019-12-05T05:28:04.344 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:04.332+0000 7fa1affff700 1 -- 172.21.15.42:0/2557019902 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1a80030b0 con 0x7fa1bc09fae0 2019-12-05T05:28:04.345 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:04.332+0000 7fa1affff700 1 -- 172.21.15.42:0/2557019902 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 931897078 0 0) 0x7fa1bc0a0710 con 0x7fa1bc07e720 2019-12-05T05:28:04.345 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:04.332+0000 7fa1affff700 1 -- 172.21.15.42:0/2557019902 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1bc0a9ba0 con 0x7fa1bc07e720 2019-12-05T05:28:04.345 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:04.332+0000 7fa1c1131700 1 -- 172.21.15.42:0/2557019902 >> v1:172.21.15.42:6789/0 conn(0x7fa1bc09fae0 legacy=0x7fa1bc14efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:28:04.345 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:04.332+0000 7fa1affff700 1 -- 172.21.15.42:0/2557019902 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1942200914 0 0) 0x7fa1a80030b0 con 0x7fa1bc09fae0 2019-12-05T05:28:04.345 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:04.332+0000 7fa1c1131700 1 -- 172.21.15.42:0/2557019902 >> v1:172.21.15.40:6789/0 conn(0x7fa1bc07e720 legacy=0x7fa1bc09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:28:04.345 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:04.332+0000 7fa1c1131700 1 -- 172.21.15.42:0/2557019902 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1bc07eb50 con 0x7fa1bc0a7740 2019-12-05T05:28:04.346 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:04.336+0000 7fa1c1131700 1 --2- 172.21.15.42:0/2557019902 >> v2:172.21.15.40:3300/0 conn(0x7fa1bc0a7740 0x7fa1bc08df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fa1b8008790 tx=0x7fa1b4001950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:04.346 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:04.336+0000 7fa1c1131700 1 --2- 172.21.15.42:0/2557019902 >> v2:172.21.15.40:3300/0 conn(0x7fa1bc0a7740 0x7fa1bc08df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fa1b8008790 tx=0x7fa1b4001950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:04.346 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:04.336+0000 7fa1c1131700 1 --2- 172.21.15.42:0/2557019902 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa1bc0a7740 0x7fa1bc08df80 secure :-1 s=READY pgs=714 cs=0 l=1 rx=0x7fa1b8008790 tx=0x7fa1b4001950).ready entity=mon.0 client_cookie=cf9b31606d25552 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:28:04.346 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:04.336+0000 7fa1c1131700 1 --2- 172.21.15.42:0/2557019902 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa1bc0a7740 0x7fa1bc08df80 secure :-1 s=THROTTLE_DONE pgs=714 cs=0 l=1 rx=0x7fa1b8008790 tx=0x7fa1b4001950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:04.346 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:04.336+0000 7fa1affff700 1 -- 172.21.15.42:0/2557019902 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa1b4001750 con 0x7fa1bc0a7740 2019-12-05T05:28:04.346 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:04.336+0000 7fa1c1131700 1 --2- 172.21.15.42:0/2557019902 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa1bc0a7740 0x7fa1bc08df80 secure :-1 s=THROTTLE_DONE pgs=714 cs=0 l=1 rx=0x7fa1b8008790 tx=0x7fa1b4001950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:04.346 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:04.336+0000 7fa1affff700 1 -- 172.21.15.42:0/2557019902 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1b4001560 con 0x7fa1bc0a7740 2019-12-05T05:28:04.347 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:04.336+0000 7fa1c1131700 1 --2- 172.21.15.42:0/2557019902 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa1bc0a7740 0x7fa1bc08df80 secure :-1 s=THROTTLE_DONE pgs=714 cs=0 l=1 rx=0x7fa1b8008790 tx=0x7fa1b4001950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:04.347 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:04.336+0000 7fa1affff700 1 -- 172.21.15.42:0/2557019902 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa1b8011360 con 0x7fa1bc0a7740 2019-12-05T05:28:04.347 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:04.336+0000 7fa1c2b94700 1 -- 172.21.15.42:0/2557019902 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa1bc0a7740 msgr2=0x7fa1bc08df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:28:04.347 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:04.336+0000 7fa1c2b94700 1 --2- 172.21.15.42:0/2557019902 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa1bc0a7740 0x7fa1bc08df80 secure :-1 s=READY pgs=714 cs=0 l=1 rx=0x7fa1b8008790 tx=0x7fa1b4001950).stop 2019-12-05T05:28:04.347 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:04.336+0000 7fa1c2b94700 1 -- 172.21.15.42:0/2557019902 shutdown_connections 2019-12-05T05:28:04.347 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:04.336+0000 7fa1c2b94700 1 --2- 172.21.15.42:0/2557019902 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa1bc0a7740 0x7fa1bc08df80 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:28:04.347 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:04.336+0000 7fa1c2b94700 1 -- 172.21.15.42:0/2557019902 shutdown_connections 2019-12-05T05:28:04.347 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:04.336+0000 7fa1c2b94700 1 -- 172.21.15.42:0/2557019902 wait complete. 2019-12-05T05:28:04.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:04.336+0000 7fa1c2b94700 1 -- 172.21.15.42:0/2557019902 >> 172.21.15.42:0/2557019902 conn(0x7fa1bc0aabe0 msgr2=0x7fa1bc0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:28:04.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:04.340+0000 7fa1c2b94700 1 Processor -- start 2019-12-05T05:28:04.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:04.340+0000 7fa1c2b94700 1 -- start start 2019-12-05T05:28:04.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:04.340+0000 7fa1c2b94700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fa1bc0a68e0 0x7fa1bc11b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:28:04.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:04.340+0000 7fa1c2b94700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa1bc119cc0 0x7fa1bc11a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:28:04.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:04.340+0000 7fa1c2b94700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1bc11ad00 con 0x7fa1bc0a5f50 2019-12-05T05:28:04.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:04.340+0000 7fa1c2b94700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1bc0a5770 con 0x7fa1bc119cc0 2019-12-05T05:28:04.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:04.340+0000 7fa1c2b94700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1bc0a5a10 con 0x7fa1bc0a68e0 2019-12-05T05:28:04.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:04.340+0000 7fa1c1131700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fa1bc0a68e0 0x7fa1bc11b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:28:04.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:04.340+0000 7fa1c1131700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fa1bc0a68e0 0x7fa1bc11b4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:34308/0 (socket says 172.21.15.42:34308) 2019-12-05T05:28:04.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:04.340+0000 7fa1c1131700 1 -- 172.21.15.42:0/3003495634 learned_addr learned my addr 172.21.15.42:0/3003495634 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:28:04.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:04.340+0000 7fa1c1932700 1 --2- 172.21.15.42:0/3003495634 >> v2:172.21.15.40:3300/0 conn(0x7fa1bc119cc0 0x7fa1bc11a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:28:04.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:04.340+0000 7fa1adffb700 1 -- 172.21.15.42:0/3003495634 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3302332186 0 0) 0x7fa1bc11ad00 con 0x7fa1bc0a5f50 2019-12-05T05:28:04.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:04.340+0000 7fa1adffb700 1 -- 172.21.15.42:0/3003495634 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa19c0030b0 con 0x7fa1bc0a5f50 2019-12-05T05:28:04.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:04.340+0000 7fa1c1131700 1 -- 172.21.15.42:0/3003495634 >> v2:172.21.15.40:3300/0 conn(0x7fa1bc119cc0 msgr2=0x7fa1bc11a0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:28:04.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:04.340+0000 7fa1c1131700 1 --2- 172.21.15.42:0/3003495634 >> v2:172.21.15.40:3300/0 conn(0x7fa1bc119cc0 0x7fa1bc11a0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:28:04.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:04.340+0000 7fa1c1131700 1 -- 172.21.15.42:0/3003495634 >> v1:172.21.15.40:6789/0 conn(0x7fa1bc0a5f50 legacy=0x7fa1bc07db50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:28:04.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:04.340+0000 7fa1c1131700 1 -- 172.21.15.42:0/3003495634 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1bc0a6380 con 0x7fa1bc0a68e0 2019-12-05T05:28:04.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:04.340+0000 7fa1c1131700 1 --2- 172.21.15.42:0/3003495634 >> v2:172.21.15.42:3300/0 conn(0x7fa1bc0a68e0 0x7fa1bc11b4a0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fa1b8008790 tx=0x7fa1b80111f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:04.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:04.344+0000 7fa1c1131700 1 --2- 172.21.15.42:0/3003495634 >> v2:172.21.15.42:3300/0 conn(0x7fa1bc0a68e0 0x7fa1bc11b4a0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fa1b8008790 tx=0x7fa1b80111f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:04.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:04.344+0000 7fa1c1131700 1 --2- 172.21.15.42:0/3003495634 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa1bc0a68e0 0x7fa1bc11b4a0 secure :-1 s=READY pgs=906 cs=0 l=1 rx=0x7fa1b8008790 tx=0x7fa1b80111f0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:28:04.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:04.344+0000 7fa1c1131700 1 --2- 172.21.15.42:0/3003495634 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa1bc0a68e0 0x7fa1bc11b4a0 secure :-1 s=THROTTLE_DONE pgs=906 cs=0 l=1 rx=0x7fa1b8008790 tx=0x7fa1b80111f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:04.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:04.344+0000 7fa1adffb700 1 -- 172.21.15.42:0/3003495634 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa1b8007150 con 0x7fa1bc0a68e0 2019-12-05T05:28:04.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:04.344+0000 7fa1c1131700 1 --2- 172.21.15.42:0/3003495634 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa1bc0a68e0 0x7fa1bc11b4a0 secure :-1 s=THROTTLE_DONE pgs=906 cs=0 l=1 rx=0x7fa1b8008790 tx=0x7fa1b80111f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:04.358 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:04.344+0000 7fa1adffb700 1 -- 172.21.15.42:0/3003495634 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1b8008db0 con 0x7fa1bc0a68e0 2019-12-05T05:28:04.359 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:04.344+0000 7fa1c1131700 1 --2- 172.21.15.42:0/3003495634 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa1bc0a68e0 0x7fa1bc11b4a0 secure :-1 s=THROTTLE_DONE pgs=906 cs=0 l=1 rx=0x7fa1b8008790 tx=0x7fa1b80111f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:04.360 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:04.344+0000 7fa1c2b94700 1 -- 172.21.15.42:0/3003495634 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1bc09f2e0 con 0x7fa1bc0a68e0 2019-12-05T05:28:04.360 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:04.344+0000 7fa1adffb700 1 -- 172.21.15.42:0/3003495634 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa1b8005730 con 0x7fa1bc0a68e0 2019-12-05T05:28:04.360 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:04.344+0000 7fa1c2b94700 1 -- 172.21.15.42:0/3003495634 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1bc09f890 con 0x7fa1bc0a68e0 2019-12-05T05:28:04.360 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:04.344+0000 7fa1c1131700 1 --2- 172.21.15.42:0/3003495634 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa1bc0a68e0 0x7fa1bc11b4a0 secure :-1 s=THROTTLE_DONE pgs=906 cs=0 l=1 rx=0x7fa1b8008790 tx=0x7fa1b80111f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:04.360 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:04.344+0000 7fa1adffb700 1 -- 172.21.15.42:0/3003495634 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fa1b80059d0 con 0x7fa1bc0a68e0 2019-12-05T05:28:04.360 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:04.344+0000 7fa1c1131700 1 --2- 172.21.15.42:0/3003495634 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa1bc0a68e0 0x7fa1bc11b4a0 secure :-1 s=THROTTLE_DONE pgs=906 cs=0 l=1 rx=0x7fa1b8008790 tx=0x7fa1b80111f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:04.361 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:04.344+0000 7fa1adffb700 1 --2- 172.21.15.42:0/3003495634 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa19c03b250 0x7fa19c03d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:28:04.361 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:04.344+0000 7fa1adffb700 1 -- 172.21.15.42:0/3003495634 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7407+0+0 (secure 0 0 0) 0x7fa1b80901e0 con 0x7fa1bc0a68e0 2019-12-05T05:28:04.361 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:04.344+0000 7fa1c0930700 1 --2- 172.21.15.42:0/3003495634 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa19c03b250 0x7fa19c03d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:28:04.361 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:04.344+0000 7fa1c0930700 1 --2- 172.21.15.42:0/3003495634 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa19c03b250 0x7fa19c03d760 crc :-1 s=READY pgs=396 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:28:04.367 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:04.348+0000 7fa1c2b94700 1 -- 172.21.15.42:0/3003495634 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1bc05d110 con 0x7fa1bc0a68e0 2019-12-05T05:28:04.367 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:04.356+0000 7fa1c1131700 1 --2- 172.21.15.42:0/3003495634 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa1bc0a68e0 0x7fa1bc11b4a0 secure :-1 s=THROTTLE_DONE pgs=906 cs=0 l=1 rx=0x7fa1b8008790 tx=0x7fa1b80111f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:04.367 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:04.356+0000 7fa1adffb700 1 -- 172.21.15.42:0/3003495634 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fa1bc0a6380 con 0x7fa1bc0a68e0 2019-12-05T05:28:04.531 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:04.524+0000 7fa1c2b94700 1 -- 172.21.15.42:0/3003495634 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa1bc05b8c0 con 0x7fa1bc0a68e0 2019-12-05T05:28:04.531 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:04.524+0000 7fa1c1131700 1 --2- 172.21.15.42:0/3003495634 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa1bc0a68e0 0x7fa1bc11b4a0 secure :-1 s=THROTTLE_DONE pgs=906 cs=0 l=1 rx=0x7fa1b8008790 tx=0x7fa1b80111f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:04.531 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:04.524+0000 7fa1adffb700 1 -- 172.21.15.42:0/3003495634 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 113 v113) v1 ==== 95+0+2112 (secure 0 0 0) 0x7fa1b8077240 con 0x7fa1bc0a68e0 2019-12-05T05:28:04.531 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 113 2019-12-05T05:28:04.532 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:28:04.533 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":113,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6666,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/2544447110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2544447110},{"type":"v1","addr":"172.21.15.42:6835","nonce":2544447110}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":113}],"filesystems":[{"mdsmap":{"epoch":112,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:27:13.322209+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6546},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6546":{"gid":6546,"name":"b","rank":0,"incarnation":109,"state":"up:active","state_seq":25,"addr":"172.21.15.40:6833/1149478696","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1149478696},{"type":"v1","addr":"172.21.15.40:6833","nonce":1149478696}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:28:04.534 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:04.528+0000 7fa1c2b94700 1 -- 172.21.15.42:0/3003495634 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa19c03b250 msgr2=0x7fa19c03d760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:28:04.535 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:04.528+0000 7fa1c2b94700 1 --2- 172.21.15.42:0/3003495634 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa19c03b250 0x7fa19c03d760 crc :-1 s=READY pgs=396 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:28:04.535 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:04.528+0000 7fa1c2b94700 1 -- 172.21.15.42:0/3003495634 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa1bc0a68e0 msgr2=0x7fa1bc11b4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:28:04.535 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:04.528+0000 7fa1c2b94700 1 --2- 172.21.15.42:0/3003495634 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa1bc0a68e0 0x7fa1bc11b4a0 secure :-1 s=READY pgs=906 cs=0 l=1 rx=0x7fa1b8008790 tx=0x7fa1b80111f0).stop 2019-12-05T05:28:04.536 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:04.528+0000 7fa1c2b94700 1 -- 172.21.15.42:0/3003495634 shutdown_connections 2019-12-05T05:28:04.536 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:04.528+0000 7fa1c2b94700 1 --2- 172.21.15.42:0/3003495634 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa19c03b250 0x7fa19c03d760 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:28:04.536 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:04.528+0000 7fa1c2b94700 1 --2- 172.21.15.42:0/3003495634 >> v2:172.21.15.40:3300/0 conn(0x7fa1bc119cc0 0x7fa1bc11a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:28:04.536 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:04.528+0000 7fa1c2b94700 1 --2- 172.21.15.42:0/3003495634 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa1bc0a68e0 0x7fa1bc11b4a0 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:28:04.536 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:04.528+0000 7fa1c2b94700 1 -- 172.21.15.42:0/3003495634 shutdown_connections 2019-12-05T05:28:04.536 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:04.528+0000 7fa1c2b94700 1 -- 172.21.15.42:0/3003495634 wait complete. 2019-12-05T05:28:04.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:04.528+0000 7fa1c2b94700 1 -- 172.21.15.42:0/3003495634 >> 172.21.15.42:0/3003495634 conn(0x7fa1bc0aabe0 msgr2=0x7fa1bc08eb70 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:28:04.543 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T05:28:06.544 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:28:06.650 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:06.636+0000 7ffaa3a8e700 1 Processor -- start 2019-12-05T05:28:06.650 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:06.636+0000 7ffaa3a8e700 1 -- start start 2019-12-05T05:28:06.650 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:06.640+0000 7ffaa3a8e700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ffa9c09fae0 0x7ffa9c14fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:28:06.650 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:06.640+0000 7ffaa3a8e700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa9c0a0710 con 0x7ffa9c0a7740 2019-12-05T05:28:06.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:06.640+0000 7ffaa3a8e700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa9c07ef20 con 0x7ffa9c08df80 2019-12-05T05:28:06.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:06.640+0000 7ffaa3a8e700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa9c0a0b00 con 0x7ffa9c09fae0 2019-12-05T05:28:06.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:06.640+0000 7ffaa182a700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7ffa9c08df80 0x7ffa9c09f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:34356/0 (socket says 172.21.15.42:34356) 2019-12-05T05:28:06.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:06.640+0000 7ffaa182a700 1 -- 172.21.15.42:0/2122845912 learned_addr learned my addr 172.21.15.42:0/2122845912 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:28:06.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:06.640+0000 7ffaa282c700 1 --2- 172.21.15.42:0/2122845912 >> v2:172.21.15.40:3300/0 conn(0x7ffa9c09fae0 0x7ffa9c14fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:28:06.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:06.640+0000 7ffaa1029700 1 -- 172.21.15.42:0/2122845912 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3034196468 0 0) 0x7ffa9c07ef20 con 0x7ffa9c08df80 2019-12-05T05:28:06.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:06.640+0000 7ffaa1029700 1 -- 172.21.15.42:0/2122845912 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa880030b0 con 0x7ffa9c08df80 2019-12-05T05:28:06.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:06.640+0000 7ffaa1029700 1 -- 172.21.15.42:0/2122845912 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 344655614 0 0) 0x7ffa880030b0 con 0x7ffa9c08df80 2019-12-05T05:28:06.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:06.640+0000 7ffaa1029700 1 -- 172.21.15.42:0/2122845912 >> v2:172.21.15.40:3300/0 conn(0x7ffa9c09fae0 msgr2=0x7ffa9c14fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:28:06.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:06.640+0000 7ffaa1029700 1 --2- 172.21.15.42:0/2122845912 >> v2:172.21.15.40:3300/0 conn(0x7ffa9c09fae0 0x7ffa9c14fff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:28:06.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:06.640+0000 7ffaa1029700 1 -- 172.21.15.42:0/2122845912 >> v1:172.21.15.40:6789/0 conn(0x7ffa9c0a7740 legacy=0x7ffa9c07e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:28:06.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:06.640+0000 7ffaa1029700 1 -- 172.21.15.42:0/2122845912 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa9c0a9ba0 con 0x7ffa9c08df80 2019-12-05T05:28:06.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:06.640+0000 7ffaa1029700 1 -- 172.21.15.42:0/2122845912 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7ffa8c001750 con 0x7ffa9c08df80 2019-12-05T05:28:06.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:06.640+0000 7ffaa282c700 1 --2- 172.21.15.42:0/2122845912 >> v2:172.21.15.40:3300/0 conn(0x7ffa9c09fae0 0x7ffa9c14fff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:28:06.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:06.640+0000 7ffaa1029700 1 -- 172.21.15.42:0/2122845912 >> v1:172.21.15.42:6789/0 conn(0x7ffa9c08df80 legacy=0x7ffa9c09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:28:06.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:06.640+0000 7ffaa1029700 1 --2- 172.21.15.42:0/2122845912 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ffa880036e0 0x7ffa88003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:28:06.654 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:06.640+0000 7ffaa1029700 1 --2- 172.21.15.42:0/2122845912 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ffa88004420 0x7ffa88004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:28:06.654 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:06.640+0000 7ffaa1029700 1 --2- 172.21.15.42:0/2122845912 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffa88004da0 0x7ffa8800b670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:28:06.655 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:06.640+0000 7ffaa202b700 1 --2- 172.21.15.42:0/2122845912 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ffa880036e0 0x7ffa88003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:28:06.655 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:06.640+0000 7ffaa1029700 1 -- 172.21.15.42:0/2122845912 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa9c0a0b00 con 0x7ffa88004da0 2019-12-05T05:28:06.655 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:06.640+0000 7ffaa1029700 1 -- 172.21.15.42:0/2122845912 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa8804fc60 con 0x7ffa880036e0 2019-12-05T05:28:06.655 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:06.640+0000 7ffaa1029700 1 -- 172.21.15.42:0/2122845912 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa8804fea0 con 0x7ffa88004420 2019-12-05T05:28:06.655 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:06.640+0000 7ffaa282c700 1 --2- 172.21.15.42:0/2122845912 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffa88004da0 0x7ffa8800b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:28:06.656 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:06.640+0000 7ffaa182a700 1 --2- 172.21.15.42:0/2122845912 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ffa88004420 0x7ffa88004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:28:06.656 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:06.640+0000 7ffaa202b700 1 -- 172.21.15.42:0/2122845912 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ffa88004420 msgr2=0x7ffa88004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:28:06.656 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:06.640+0000 7ffaa202b700 1 --2- 172.21.15.42:0/2122845912 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ffa88004420 0x7ffa88004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:28:06.656 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:06.640+0000 7ffaa202b700 1 -- 172.21.15.42:0/2122845912 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffa88004da0 msgr2=0x7ffa8800b670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:28:06.656 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:06.640+0000 7ffaa202b700 1 --2- 172.21.15.42:0/2122845912 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffa88004da0 0x7ffa8800b670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:28:06.656 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:06.640+0000 7ffaa202b700 1 -- 172.21.15.42:0/2122845912 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa880501a0 con 0x7ffa880036e0 2019-12-05T05:28:06.657 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:06.640+0000 7ffaa282c700 1 -- 172.21.15.42:0/2122845912 reap_dead start 2019-12-05T05:28:06.657 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:06.640+0000 7ffaa182a700 1 --2- 172.21.15.42:0/2122845912 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ffa88004420 0x7ffa88004850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T05:28:06.657 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:06.640+0000 7ffaa202b700 1 --2- 172.21.15.42:0/2122845912 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ffa880036e0 0x7ffa88003b50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7ffa98001380 tx=0x7ffa98010050).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:06.657 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:06.640+0000 7ffaa202b700 1 --2- 172.21.15.42:0/2122845912 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ffa880036e0 0x7ffa88003b50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7ffa98001380 tx=0x7ffa98010050).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:06.658 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:06.640+0000 7ffaa202b700 1 --2- 172.21.15.42:0/2122845912 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ffa880036e0 0x7ffa88003b50 secure :-1 s=READY pgs=908 cs=0 l=1 rx=0x7ffa98001380 tx=0x7ffa98010050).ready entity=mon.1 client_cookie=61fec511b27e9852 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:28:06.658 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:06.644+0000 7ffaa202b700 1 --2- 172.21.15.42:0/2122845912 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ffa880036e0 0x7ffa88003b50 secure :-1 s=THROTTLE_DONE pgs=908 cs=0 l=1 rx=0x7ffa98001380 tx=0x7ffa98010050).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:06.658 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:06.644+0000 7ffaa1029700 1 -- 172.21.15.42:0/2122845912 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ffa98006ac0 con 0x7ffa880036e0 2019-12-05T05:28:06.659 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:06.644+0000 7ffaa202b700 1 --2- 172.21.15.42:0/2122845912 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ffa880036e0 0x7ffa88003b50 secure :-1 s=THROTTLE_DONE pgs=908 cs=0 l=1 rx=0x7ffa98001380 tx=0x7ffa98010050).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:06.659 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:06.644+0000 7ffaa202b700 1 --2- 172.21.15.42:0/2122845912 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ffa880036e0 0x7ffa88003b50 secure :-1 s=THROTTLE_DONE pgs=908 cs=0 l=1 rx=0x7ffa98001380 tx=0x7ffa98010050).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:06.659 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:06.644+0000 7ffaa1029700 1 -- 172.21.15.42:0/2122845912 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa980066e0 con 0x7ffa880036e0 2019-12-05T05:28:06.659 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:06.644+0000 7ffaa1029700 1 -- 172.21.15.42:0/2122845912 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ffa98002730 con 0x7ffa880036e0 2019-12-05T05:28:06.659 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:06.644+0000 7ffaa3a8e700 1 -- 172.21.15.42:0/2122845912 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ffa880036e0 msgr2=0x7ffa88003b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:28:06.660 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:06.644+0000 7ffaa3a8e700 1 --2- 172.21.15.42:0/2122845912 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ffa880036e0 0x7ffa88003b50 secure :-1 s=READY pgs=908 cs=0 l=1 rx=0x7ffa98001380 tx=0x7ffa98010050).stop 2019-12-05T05:28:06.660 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:06.644+0000 7ffaa3a8e700 1 -- 172.21.15.42:0/2122845912 shutdown_connections 2019-12-05T05:28:06.660 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:06.644+0000 7ffaa3a8e700 1 --2- 172.21.15.42:0/2122845912 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ffa880036e0 0x7ffa88003b50 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:28:06.660 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:06.644+0000 7ffaa3a8e700 1 -- 172.21.15.42:0/2122845912 shutdown_connections 2019-12-05T05:28:06.660 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:06.644+0000 7ffaa3a8e700 1 -- 172.21.15.42:0/2122845912 wait complete. 2019-12-05T05:28:06.660 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:06.644+0000 7ffaa3a8e700 1 -- 172.21.15.42:0/2122845912 >> 172.21.15.42:0/2122845912 conn(0x7ffa9c0aabe0 msgr2=0x7ffa9c0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:28:06.666 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:06.648+0000 7ffaa3a8e700 1 Processor -- start 2019-12-05T05:28:06.666 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:06.648+0000 7ffaa3a8e700 1 -- start start 2019-12-05T05:28:06.666 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:06.648+0000 7ffaa3a8e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ffa9c0a4840 0x7ffa9c126480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:28:06.667 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:06.648+0000 7ffaa3a8e700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ffa9c130a10 0x7ffa9c1343a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:28:06.667 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:06.648+0000 7ffaa3a8e700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa9c0a71b0 con 0x7ffa9c1305e0 2019-12-05T05:28:06.667 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:06.648+0000 7ffaa3a8e700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa9c11a820 con 0x7ffa9c0a4840 2019-12-05T05:28:06.667 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:06.648+0000 7ffaa3a8e700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa9c11aac0 con 0x7ffa9c130a10 2019-12-05T05:28:06.667 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:06.648+0000 7ffaa202b700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ffa9c0a4840 0x7ffa9c126480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:28:06.667 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:06.648+0000 7ffaa202b700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ffa9c0a4840 0x7ffa9c126480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:34330/0 (socket says 172.21.15.42:34330) 2019-12-05T05:28:06.667 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:06.648+0000 7ffaa202b700 1 -- 172.21.15.42:0/2185992106 learned_addr learned my addr 172.21.15.42:0/2185992106 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:28:06.668 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:06.648+0000 7ffaa282c700 1 --2- 172.21.15.42:0/2185992106 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ffa9c130a10 0x7ffa9c1343a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:28:06.668 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:06.648+0000 7ffaa202b700 1 -- 172.21.15.42:0/2185992106 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ffa9c130a10 msgr2=0x7ffa9c1343a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:28:06.668 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:06.648+0000 7ffaa202b700 1 --2- 172.21.15.42:0/2185992106 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ffa9c130a10 0x7ffa9c1343a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:28:06.669 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:06.648+0000 7ffaa202b700 1 -- 172.21.15.42:0/2185992106 >> v1:172.21.15.42:6789/0 conn(0x7ffa9c1305e0 legacy=0x7ffa9c133ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:28:06.669 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:06.648+0000 7ffaa202b700 1 -- 172.21.15.42:0/2185992106 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa9c130e40 con 0x7ffa9c0a4840 2019-12-05T05:28:06.669 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:06.648+0000 7ffaa202b700 1 --2- 172.21.15.42:0/2185992106 >> v2:172.21.15.42:3300/0 conn(0x7ffa9c0a4840 0x7ffa9c126480 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7ffa9c11a9e0 tx=0x7ffa9800f670).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:06.670 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:06.648+0000 7ffaa202b700 1 --2- 172.21.15.42:0/2185992106 >> v2:172.21.15.42:3300/0 conn(0x7ffa9c0a4840 0x7ffa9c126480 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7ffa9c11a9e0 tx=0x7ffa9800f670).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:06.670 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:06.648+0000 7ffaa202b700 1 --2- 172.21.15.42:0/2185992106 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ffa9c0a4840 0x7ffa9c126480 secure :-1 s=READY pgs=910 cs=0 l=1 rx=0x7ffa9c11a9e0 tx=0x7ffa9800f670).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:28:06.670 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:06.648+0000 7ffaa202b700 1 --2- 172.21.15.42:0/2185992106 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ffa9c0a4840 0x7ffa9c126480 secure :-1 s=THROTTLE_DONE pgs=910 cs=0 l=1 rx=0x7ffa9c11a9e0 tx=0x7ffa9800f670).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:06.670 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:06.648+0000 7ffaa202b700 1 --2- 172.21.15.42:0/2185992106 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ffa9c0a4840 0x7ffa9c126480 secure :-1 s=THROTTLE_DONE pgs=910 cs=0 l=1 rx=0x7ffa9c11a9e0 tx=0x7ffa9800f670).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:06.670 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:06.648+0000 7ffa92ffd700 1 -- 172.21.15.42:0/2185992106 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ffa98011b30 con 0x7ffa9c0a4840 2019-12-05T05:28:06.671 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:06.648+0000 7ffaa202b700 1 --2- 172.21.15.42:0/2185992106 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ffa9c0a4840 0x7ffa9c126480 secure :-1 s=THROTTLE_DONE pgs=910 cs=0 l=1 rx=0x7ffa9c11a9e0 tx=0x7ffa9800f670).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:06.671 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:06.648+0000 7ffa92ffd700 1 -- 172.21.15.42:0/2185992106 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa98006cc0 con 0x7ffa9c0a4840 2019-12-05T05:28:06.671 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:06.648+0000 7ffaa3a8e700 1 -- 172.21.15.42:0/2185992106 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa9c1269d0 con 0x7ffa9c0a4840 2019-12-05T05:28:06.671 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:06.648+0000 7ffa92ffd700 1 -- 172.21.15.42:0/2185992106 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ffa98002730 con 0x7ffa9c0a4840 2019-12-05T05:28:06.671 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:06.648+0000 7ffaa3a8e700 1 -- 172.21.15.42:0/2185992106 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa9c141fb0 con 0x7ffa9c0a4840 2019-12-05T05:28:06.671 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:06.652+0000 7ffaa202b700 1 --2- 172.21.15.42:0/2185992106 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ffa9c0a4840 0x7ffa9c126480 secure :-1 s=THROTTLE_DONE pgs=910 cs=0 l=1 rx=0x7ffa9c11a9e0 tx=0x7ffa9800f670).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:06.672 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:06.652+0000 7ffa92ffd700 1 -- 172.21.15.42:0/2185992106 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7ffa98002a40 con 0x7ffa9c0a4840 2019-12-05T05:28:06.672 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:06.652+0000 7ffaa202b700 1 --2- 172.21.15.42:0/2185992106 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ffa9c0a4840 0x7ffa9c126480 secure :-1 s=THROTTLE_DONE pgs=910 cs=0 l=1 rx=0x7ffa9c11a9e0 tx=0x7ffa9800f670).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:06.673 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:06.652+0000 7ffa92ffd700 1 --2- 172.21.15.42:0/2185992106 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ffa8403ad10 0x7ffa8403d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:28:06.673 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:06.652+0000 7ffa92ffd700 1 -- 172.21.15.42:0/2185992106 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7407+0+0 (secure 0 0 0) 0x7ffa9804d930 con 0x7ffa9c0a4840 2019-12-05T05:28:06.673 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:06.652+0000 7ffaa182a700 1 --2- 172.21.15.42:0/2185992106 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ffa8403ad10 0x7ffa8403d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:28:06.673 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:06.652+0000 7ffaa182a700 1 --2- 172.21.15.42:0/2185992106 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ffa8403ad10 0x7ffa8403d220 crc :-1 s=READY pgs=397 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:28:06.673 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:06.652+0000 7ffaa3a8e700 1 -- 172.21.15.42:0/2185992106 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa9c05d110 con 0x7ffa9c0a4840 2019-12-05T05:28:06.673 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:06.660+0000 7ffaa202b700 1 --2- 172.21.15.42:0/2185992106 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ffa9c0a4840 0x7ffa9c126480 secure :-1 s=THROTTLE_DONE pgs=910 cs=0 l=1 rx=0x7ffa9c11a9e0 tx=0x7ffa9800f670).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:06.674 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:06.660+0000 7ffa92ffd700 1 -- 172.21.15.42:0/2185992106 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7ffa980349b0 con 0x7ffa9c0a4840 2019-12-05T05:28:06.833 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:06.828+0000 7ffaa3a8e700 1 -- 172.21.15.42:0/2185992106 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffa9c05b8c0 con 0x7ffa9c0a4840 2019-12-05T05:28:06.834 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:06.828+0000 7ffaa202b700 1 --2- 172.21.15.42:0/2185992106 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ffa9c0a4840 0x7ffa9c126480 secure :-1 s=THROTTLE_DONE pgs=910 cs=0 l=1 rx=0x7ffa9c11a9e0 tx=0x7ffa9800f670).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:06.834 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:06.828+0000 7ffa92ffd700 1 -- 172.21.15.42:0/2185992106 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 113 v113) v1 ==== 95+0+2112 (secure 0 0 0) 0x7ffa98002d50 con 0x7ffa9c0a4840 2019-12-05T05:28:06.834 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 113 2019-12-05T05:28:06.835 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:28:06.835 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":113,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6666,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/2544447110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2544447110},{"type":"v1","addr":"172.21.15.42:6835","nonce":2544447110}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":113}],"filesystems":[{"mdsmap":{"epoch":112,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:27:13.322209+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6546},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6546":{"gid":6546,"name":"b","rank":0,"incarnation":109,"state":"up:active","state_seq":25,"addr":"172.21.15.40:6833/1149478696","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1149478696},{"type":"v1","addr":"172.21.15.40:6833","nonce":1149478696}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:28:06.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:06.832+0000 7ffaa3a8e700 1 -- 172.21.15.42:0/2185992106 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ffa8403ad10 msgr2=0x7ffa8403d220 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:28:06.837 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:06.832+0000 7ffaa3a8e700 1 --2- 172.21.15.42:0/2185992106 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ffa8403ad10 0x7ffa8403d220 crc :-1 s=READY pgs=397 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:28:06.837 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:06.832+0000 7ffaa3a8e700 1 -- 172.21.15.42:0/2185992106 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ffa9c0a4840 msgr2=0x7ffa9c126480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:28:06.837 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:06.832+0000 7ffaa3a8e700 1 --2- 172.21.15.42:0/2185992106 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ffa9c0a4840 0x7ffa9c126480 secure :-1 s=READY pgs=910 cs=0 l=1 rx=0x7ffa9c11a9e0 tx=0x7ffa9800f670).stop 2019-12-05T05:28:06.838 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:06.832+0000 7ffaa3a8e700 1 -- 172.21.15.42:0/2185992106 shutdown_connections 2019-12-05T05:28:06.838 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:06.832+0000 7ffaa3a8e700 1 --2- 172.21.15.42:0/2185992106 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ffa8403ad10 0x7ffa8403d220 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:28:06.838 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:06.832+0000 7ffaa3a8e700 1 --2- 172.21.15.42:0/2185992106 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ffa9c130a10 0x7ffa9c1343a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:28:06.838 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:06.832+0000 7ffaa3a8e700 1 --2- 172.21.15.42:0/2185992106 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ffa9c0a4840 0x7ffa9c126480 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:28:06.838 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:06.832+0000 7ffaa3a8e700 1 -- 172.21.15.42:0/2185992106 shutdown_connections 2019-12-05T05:28:06.839 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:06.832+0000 7ffaa3a8e700 1 -- 172.21.15.42:0/2185992106 wait complete. 2019-12-05T05:28:06.842 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:06.832+0000 7ffaa3a8e700 1 -- 172.21.15.42:0/2185992106 >> 172.21.15.42:0/2185992106 conn(0x7ffa9c066fe0 msgr2=0x7ffa9c11c540 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:28:06.846 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T05:28:08.656 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:28:08.666 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:28:08.847 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:28:08.957 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:08.944+0000 7fbfe339f700 1 Processor -- start 2019-12-05T05:28:08.957 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:08.944+0000 7fbfe339f700 1 -- start start 2019-12-05T05:28:08.957 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:08.944+0000 7fbfe339f700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fbfdc0a7740 0x7fbfdc08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:28:08.957 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:08.944+0000 7fbfe339f700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbfdc07e720 0x7fbfdc07eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:28:08.958 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:08.944+0000 7fbfe339f700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fbfdc09f2e0 0x7fbfdc09f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:28:08.958 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:08.944+0000 7fbfe339f700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfdc0a0b00 con 0x7fbfdc09f2e0 2019-12-05T05:28:08.958 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:08.944+0000 7fbfe339f700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfdc08e5b0 con 0x7fbfdc0a7740 2019-12-05T05:28:08.958 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:08.944+0000 7fbfe339f700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfdc08e850 con 0x7fbfdc07e720 2019-12-05T05:28:08.958 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:08.944+0000 7fbfe193c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fbfdc0a7740 0x7fbfdc08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:28:08.958 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:08.944+0000 7fbfe113b700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbfdc07e720 0x7fbfdc07eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:28:08.958 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:08.948+0000 7fbfe113b700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbfdc07e720 0x7fbfdc07eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:34806/0 (socket says 172.21.15.42:34806) 2019-12-05T05:28:08.959 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:08.948+0000 7fbfe113b700 1 -- 172.21.15.42:0/467114386 learned_addr learned my addr 172.21.15.42:0/467114386 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:28:08.959 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:08.948+0000 7fbfe213d700 1 --2- 172.21.15.42:0/467114386 >> v2:172.21.15.40:3300/0 conn(0x7fbfdc09f2e0 0x7fbfdc09f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:28:08.959 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:08.948+0000 7fbfe193c700 1 -- 172.21.15.42:0/467114386 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbfdc07e720 msgr2=0x7fbfdc07eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:28:08.959 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:08.948+0000 7fbfe193c700 1 --2- 172.21.15.42:0/467114386 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbfdc07e720 0x7fbfdc07eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:28:08.959 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:08.948+0000 7fbfe193c700 1 -- 172.21.15.42:0/467114386 >> v2:172.21.15.40:3300/0 conn(0x7fbfdc09f2e0 msgr2=0x7fbfdc09f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:28:08.959 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:08.948+0000 7fbfe193c700 1 --2- 172.21.15.42:0/467114386 >> v2:172.21.15.40:3300/0 conn(0x7fbfdc09f2e0 0x7fbfdc09f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:28:08.959 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:08.948+0000 7fbfe193c700 1 -- 172.21.15.42:0/467114386 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfdc07f0a0 con 0x7fbfdc0a7740 2019-12-05T05:28:08.960 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:08.948+0000 7fbfe113b700 1 --2- 172.21.15.42:0/467114386 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbfdc07e720 0x7fbfdc07eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:28:08.960 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:08.948+0000 7fbfe193c700 1 --2- 172.21.15.42:0/467114386 >> v2:172.21.15.42:3300/0 conn(0x7fbfdc0a7740 0x7fbfdc08df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fbfdc08e770 tx=0x7fbfd800fcf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:08.960 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:08.948+0000 7fbfe193c700 1 --2- 172.21.15.42:0/467114386 >> v2:172.21.15.42:3300/0 conn(0x7fbfdc0a7740 0x7fbfdc08df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fbfdc08e770 tx=0x7fbfd800fcf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:08.960 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:08.948+0000 7fbfe193c700 1 --2- 172.21.15.42:0/467114386 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbfdc0a7740 0x7fbfdc08df80 secure :-1 s=READY pgs=911 cs=0 l=1 rx=0x7fbfdc08e770 tx=0x7fbfd800fcf0).ready entity=mon.1 client_cookie=4a4eb3cb40c22c7f server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:28:08.960 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:08.948+0000 7fbfe193c700 1 --2- 172.21.15.42:0/467114386 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbfdc0a7740 0x7fbfdc08df80 secure :-1 s=THROTTLE_DONE pgs=911 cs=0 l=1 rx=0x7fbfdc08e770 tx=0x7fbfd800fcf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:08.960 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:08.948+0000 7fbfe093a700 1 -- 172.21.15.42:0/467114386 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbfd8008bb0 con 0x7fbfdc0a7740 2019-12-05T05:28:08.961 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:08.948+0000 7fbfe193c700 1 --2- 172.21.15.42:0/467114386 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbfdc0a7740 0x7fbfdc08df80 secure :-1 s=THROTTLE_DONE pgs=911 cs=0 l=1 rx=0x7fbfdc08e770 tx=0x7fbfd800fcf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:08.961 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:08.948+0000 7fbfe193c700 1 --2- 172.21.15.42:0/467114386 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbfdc0a7740 0x7fbfdc08df80 secure :-1 s=THROTTLE_DONE pgs=911 cs=0 l=1 rx=0x7fbfdc08e770 tx=0x7fbfd800fcf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:08.961 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:08.948+0000 7fbfe093a700 1 -- 172.21.15.42:0/467114386 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfd80087d0 con 0x7fbfdc0a7740 2019-12-05T05:28:08.961 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:08.948+0000 7fbfe093a700 1 -- 172.21.15.42:0/467114386 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbfd80118a0 con 0x7fbfdc0a7740 2019-12-05T05:28:08.961 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:08.948+0000 7fbfe339f700 1 -- 172.21.15.42:0/467114386 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbfdc0a7740 msgr2=0x7fbfdc08df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:28:08.961 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:08.948+0000 7fbfe339f700 1 --2- 172.21.15.42:0/467114386 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbfdc0a7740 0x7fbfdc08df80 secure :-1 s=READY pgs=911 cs=0 l=1 rx=0x7fbfdc08e770 tx=0x7fbfd800fcf0).stop 2019-12-05T05:28:08.962 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:08.948+0000 7fbfe339f700 1 -- 172.21.15.42:0/467114386 shutdown_connections 2019-12-05T05:28:08.962 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:08.948+0000 7fbfe339f700 1 --2- 172.21.15.42:0/467114386 >> v2:172.21.15.40:3300/0 conn(0x7fbfdc09f2e0 0x7fbfdc09f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:28:08.962 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:08.948+0000 7fbfe339f700 1 --2- 172.21.15.42:0/467114386 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbfdc0a7740 0x7fbfdc08df80 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:28:08.962 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:08.948+0000 7fbfe339f700 1 --2- 172.21.15.42:0/467114386 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbfdc07e720 0x7fbfdc07eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:28:08.962 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:08.948+0000 7fbfe339f700 1 -- 172.21.15.42:0/467114386 shutdown_connections 2019-12-05T05:28:08.962 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:08.948+0000 7fbfe339f700 1 -- 172.21.15.42:0/467114386 wait complete. 2019-12-05T05:28:08.962 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:08.948+0000 7fbfe339f700 1 -- 172.21.15.42:0/467114386 >> 172.21.15.42:0/467114386 conn(0x7fbfdc0aabe0 msgr2=0x7fbfdc0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:28:08.963 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:08.952+0000 7fbfe339f700 1 Processor -- start 2019-12-05T05:28:08.963 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:08.952+0000 7fbfe339f700 1 -- start start 2019-12-05T05:28:08.966 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:08.952+0000 7fbfe339f700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbfdc116500 0x7fbfdc1427a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:28:08.966 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:08.952+0000 7fbfe339f700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fbfdc133580 0x7fbfdc0762f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:28:08.966 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:08.952+0000 7fbfe339f700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfdc076840 con 0x7fbfdc144c40 2019-12-05T05:28:08.966 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:08.952+0000 7fbfe339f700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfdc143cf0 con 0x7fbfdc133580 2019-12-05T05:28:08.966 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:08.952+0000 7fbfe339f700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfdc145460 con 0x7fbfdc116500 2019-12-05T05:28:08.966 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:08.952+0000 7fbfe213d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fbfdc133580 0x7fbfdc0762f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:28:08.966 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:08.952+0000 7fbfe113b700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fbfdc144c40 0x7fbfdc143f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:34384/0 (socket says 172.21.15.42:34384) 2019-12-05T05:28:08.967 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:08.952+0000 7fbfe113b700 1 -- 172.21.15.42:0/767593359 learned_addr learned my addr 172.21.15.42:0/767593359 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:28:08.967 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:08.952+0000 7fbfe193c700 1 --2- 172.21.15.42:0/767593359 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbfdc116500 0x7fbfdc1427a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:28:08.967 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:08.952+0000 7fbfe213d700 1 -- 172.21.15.42:0/767593359 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbfdc116500 msgr2=0x7fbfdc1427a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:28:08.967 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:08.952+0000 7fbfe213d700 1 --2- 172.21.15.42:0/767593359 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbfdc116500 0x7fbfdc1427a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:28:08.967 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:08.952+0000 7fbfd27fc700 1 -- 172.21.15.42:0/767593359 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2379473424 0 0) 0x7fbfdc076840 con 0x7fbfdc144c40 2019-12-05T05:28:08.967 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:08.952+0000 7fbfe213d700 1 -- 172.21.15.42:0/767593359 >> v1:172.21.15.42:6789/0 conn(0x7fbfdc144c40 legacy=0x7fbfdc143f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:28:08.967 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:08.952+0000 7fbfe213d700 1 -- 172.21.15.42:0/767593359 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfdc077a80 con 0x7fbfdc133580 2019-12-05T05:28:08.968 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:08.952+0000 7fbfe213d700 1 --2- 172.21.15.42:0/767593359 >> v2:172.21.15.42:3300/0 conn(0x7fbfdc133580 0x7fbfdc0762f0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fbfdc142e80 tx=0x7fbfd4010e40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:08.968 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:08.952+0000 7fbfe213d700 1 --2- 172.21.15.42:0/767593359 >> v2:172.21.15.42:3300/0 conn(0x7fbfdc133580 0x7fbfdc0762f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fbfdc142e80 tx=0x7fbfd4010e40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:08.968 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:08.952+0000 7fbfe213d700 1 --2- 172.21.15.42:0/767593359 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbfdc133580 0x7fbfdc0762f0 secure :-1 s=READY pgs=913 cs=0 l=1 rx=0x7fbfdc142e80 tx=0x7fbfd4010e40).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:28:08.968 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:08.952+0000 7fbfe213d700 1 --2- 172.21.15.42:0/767593359 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbfdc133580 0x7fbfdc0762f0 secure :-1 s=THROTTLE_DONE pgs=913 cs=0 l=1 rx=0x7fbfdc142e80 tx=0x7fbfd4010e40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:08.968 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:08.952+0000 7fbfd27fc700 1 -- 172.21.15.42:0/767593359 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbfd4005ca0 con 0x7fbfdc133580 2019-12-05T05:28:08.968 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:08.952+0000 7fbfe213d700 1 --2- 172.21.15.42:0/767593359 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbfdc133580 0x7fbfdc0762f0 secure :-1 s=THROTTLE_DONE pgs=913 cs=0 l=1 rx=0x7fbfdc142e80 tx=0x7fbfd4010e40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:08.969 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:08.952+0000 7fbfe213d700 1 --2- 172.21.15.42:0/767593359 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbfdc133580 0x7fbfdc0762f0 secure :-1 s=THROTTLE_DONE pgs=913 cs=0 l=1 rx=0x7fbfdc142e80 tx=0x7fbfd4010e40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:08.969 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:08.952+0000 7fbfd27fc700 1 -- 172.21.15.42:0/767593359 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfd40058c0 con 0x7fbfdc133580 2019-12-05T05:28:08.969 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:08.952+0000 7fbfd27fc700 1 -- 172.21.15.42:0/767593359 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbfd4011c20 con 0x7fbfdc133580 2019-12-05T05:28:08.969 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:08.952+0000 7fbfe339f700 1 -- 172.21.15.42:0/767593359 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfdc076a70 con 0x7fbfdc133580 2019-12-05T05:28:08.969 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:08.956+0000 7fbfe339f700 1 -- 172.21.15.42:0/767593359 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfdc076fa0 con 0x7fbfdc133580 2019-12-05T05:28:08.969 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:08.956+0000 7fbfe213d700 1 --2- 172.21.15.42:0/767593359 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbfdc133580 0x7fbfdc0762f0 secure :-1 s=THROTTLE_DONE pgs=913 cs=0 l=1 rx=0x7fbfdc142e80 tx=0x7fbfd4010e40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:08.969 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:08.956+0000 7fbfd27fc700 1 -- 172.21.15.42:0/767593359 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fbfd4007180 con 0x7fbfdc133580 2019-12-05T05:28:08.970 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:08.956+0000 7fbfe213d700 1 --2- 172.21.15.42:0/767593359 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbfdc133580 0x7fbfdc0762f0 secure :-1 s=THROTTLE_DONE pgs=913 cs=0 l=1 rx=0x7fbfdc142e80 tx=0x7fbfd4010e40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:08.970 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:08.956+0000 7fbfd27fc700 1 --2- 172.21.15.42:0/767593359 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbfc403ac50 0x7fbfc403d160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:28:08.970 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:08.956+0000 7fbfd27fc700 1 -- 172.21.15.42:0/767593359 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7407+0+0 (secure 0 0 0) 0x7fbfd404ad70 con 0x7fbfdc133580 2019-12-05T05:28:08.970 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:08.956+0000 7fbfe193c700 1 --2- 172.21.15.42:0/767593359 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbfc403ac50 0x7fbfc403d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:28:08.970 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:08.956+0000 7fbfe193c700 1 --2- 172.21.15.42:0/767593359 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbfc403ac50 0x7fbfc403d160 crc :-1 s=READY pgs=398 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:28:08.970 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:08.960+0000 7fbfe339f700 1 -- 172.21.15.42:0/767593359 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfdc05d110 con 0x7fbfdc133580 2019-12-05T05:28:08.971 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:08.964+0000 7fbfe213d700 1 --2- 172.21.15.42:0/767593359 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbfdc133580 0x7fbfdc0762f0 secure :-1 s=THROTTLE_DONE pgs=913 cs=0 l=1 rx=0x7fbfdc142e80 tx=0x7fbfd4010e40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:08.971 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:08.964+0000 7fbfd27fc700 1 -- 172.21.15.42:0/767593359 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fbfd4031dd0 con 0x7fbfdc133580 2019-12-05T05:28:09.135 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:09.128+0000 7fbfe339f700 1 -- 172.21.15.42:0/767593359 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbfdc05b8c0 con 0x7fbfdc133580 2019-12-05T05:28:09.136 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:09.132+0000 7fbfe213d700 1 --2- 172.21.15.42:0/767593359 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbfdc133580 0x7fbfdc0762f0 secure :-1 s=THROTTLE_DONE pgs=913 cs=0 l=1 rx=0x7fbfdc142e80 tx=0x7fbfd4010e40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:09.136 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:09.132+0000 7fbfd27fc700 1 -- 172.21.15.42:0/767593359 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 113 v113) v1 ==== 95+0+2112 (secure 0 0 0) 0x7fbfd4032030 con 0x7fbfdc133580 2019-12-05T05:28:09.136 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 113 2019-12-05T05:28:09.136 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:28:09.137 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":113,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6666,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/2544447110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2544447110},{"type":"v1","addr":"172.21.15.42:6835","nonce":2544447110}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":113}],"filesystems":[{"mdsmap":{"epoch":112,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:27:13.322209+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6546},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6546":{"gid":6546,"name":"b","rank":0,"incarnation":109,"state":"up:active","state_seq":25,"addr":"172.21.15.40:6833/1149478696","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1149478696},{"type":"v1","addr":"172.21.15.40:6833","nonce":1149478696}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:28:09.149 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:09.132+0000 7fbfe339f700 1 -- 172.21.15.42:0/767593359 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbfc403ac50 msgr2=0x7fbfc403d160 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:28:09.149 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:09.132+0000 7fbfe339f700 1 --2- 172.21.15.42:0/767593359 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbfc403ac50 0x7fbfc403d160 crc :-1 s=READY pgs=398 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:28:09.149 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:09.132+0000 7fbfe339f700 1 -- 172.21.15.42:0/767593359 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbfdc133580 msgr2=0x7fbfdc0762f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:28:09.150 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:09.132+0000 7fbfe339f700 1 --2- 172.21.15.42:0/767593359 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbfdc133580 0x7fbfdc0762f0 secure :-1 s=READY pgs=913 cs=0 l=1 rx=0x7fbfdc142e80 tx=0x7fbfd4010e40).stop 2019-12-05T05:28:09.150 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:09.132+0000 7fbfe339f700 1 -- 172.21.15.42:0/767593359 shutdown_connections 2019-12-05T05:28:09.150 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:09.132+0000 7fbfe339f700 1 --2- 172.21.15.42:0/767593359 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbfc403ac50 0x7fbfc403d160 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:28:09.150 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:09.132+0000 7fbfe339f700 1 --2- 172.21.15.42:0/767593359 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbfdc133580 0x7fbfdc0762f0 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:28:09.150 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:09.132+0000 7fbfe339f700 1 --2- 172.21.15.42:0/767593359 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbfdc116500 0x7fbfdc1427a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:28:09.150 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:09.132+0000 7fbfe339f700 1 -- 172.21.15.42:0/767593359 shutdown_connections 2019-12-05T05:28:09.150 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:09.132+0000 7fbfe339f700 1 -- 172.21.15.42:0/767593359 wait complete. 2019-12-05T05:28:09.151 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:09.132+0000 7fbfe339f700 1 -- 172.21.15.42:0/767593359 >> 172.21.15.42:0/767593359 conn(0x7fbfdc07e840 msgr2=0x7fbfdc11c6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:28:09.151 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T05:28:11.152 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:28:11.262 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:11.248+0000 7f997cfaa700 1 Processor -- start 2019-12-05T05:28:11.263 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:11.248+0000 7f997cfaa700 1 -- start start 2019-12-05T05:28:11.263 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:11.248+0000 7f997cfaa700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f99780a7740 0x7f997808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:28:11.263 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:11.248+0000 7f997cfaa700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f997807e720 0x7f997807eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:28:11.263 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:11.248+0000 7f997cfaa700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f997809f2e0 0x7f997809f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:28:11.263 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:11.248+0000 7f997cfaa700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99780a0b00 con 0x7f99780a7740 2019-12-05T05:28:11.264 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:11.248+0000 7f997cfaa700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f997808e5b0 con 0x7f997809f2e0 2019-12-05T05:28:11.264 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:11.248+0000 7f997cfaa700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f997808e850 con 0x7f997807e720 2019-12-05T05:28:11.264 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:11.248+0000 7f997759e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f997809f2e0 0x7f997809f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:28:11.264 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:11.248+0000 7f997659c700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f997807e720 0x7f997807eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:28:11.264 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:11.248+0000 7f9976d9d700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f99780a7740 0x7f997808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:28:11.264 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:11.248+0000 7f997759e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f997809f2e0 0x7f997809f710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:34354/0 (socket says 172.21.15.42:34354) 2019-12-05T05:28:11.264 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:11.248+0000 7f997759e700 1 -- 172.21.15.42:0/3339359940 learned_addr learned my addr 172.21.15.42:0/3339359940 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:28:11.265 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:11.252+0000 7f997759e700 1 -- 172.21.15.42:0/3339359940 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f997807e720 msgr2=0x7f997807eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:28:11.265 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:11.252+0000 7f997759e700 1 --2- 172.21.15.42:0/3339359940 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f997807e720 0x7f997807eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:28:11.265 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:11.252+0000 7f997759e700 1 -- 172.21.15.42:0/3339359940 >> v2:172.21.15.40:3300/0 conn(0x7f99780a7740 msgr2=0x7f997808df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:28:11.265 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:11.252+0000 7f997759e700 1 --2- 172.21.15.42:0/3339359940 >> v2:172.21.15.40:3300/0 conn(0x7f99780a7740 0x7f997808df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:28:11.265 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:11.252+0000 7f997759e700 1 -- 172.21.15.42:0/3339359940 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f997807f0a0 con 0x7f997809f2e0 2019-12-05T05:28:11.265 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:11.252+0000 7f997759e700 1 --2- 172.21.15.42:0/3339359940 >> v2:172.21.15.42:3300/0 conn(0x7f997809f2e0 0x7f997809f710 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f997808e770 tx=0x7f9970010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:11.265 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:11.252+0000 7f997759e700 1 --2- 172.21.15.42:0/3339359940 >> v2:172.21.15.42:3300/0 conn(0x7f997809f2e0 0x7f997809f710 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f997808e770 tx=0x7f9970010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:11.265 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:11.252+0000 7f997759e700 1 --2- 172.21.15.42:0/3339359940 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f997809f2e0 0x7f997809f710 secure :-1 s=READY pgs=914 cs=0 l=1 rx=0x7f997808e770 tx=0x7f9970010130).ready entity=mon.1 client_cookie=1682c2fceb942ce0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:28:11.266 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:11.252+0000 7f997759e700 1 --2- 172.21.15.42:0/3339359940 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f997809f2e0 0x7f997809f710 secure :-1 s=THROTTLE_DONE pgs=914 cs=0 l=1 rx=0x7f997808e770 tx=0x7f9970010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:11.266 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:11.252+0000 7f9975d9b700 1 -- 172.21.15.42:0/3339359940 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9970010de0 con 0x7f997809f2e0 2019-12-05T05:28:11.266 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:11.252+0000 7f997759e700 1 --2- 172.21.15.42:0/3339359940 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f997809f2e0 0x7f997809f710 secure :-1 s=THROTTLE_DONE pgs=914 cs=0 l=1 rx=0x7f997808e770 tx=0x7f9970010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:11.266 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:11.252+0000 7f997759e700 1 --2- 172.21.15.42:0/3339359940 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f997809f2e0 0x7f997809f710 secure :-1 s=THROTTLE_DONE pgs=914 cs=0 l=1 rx=0x7f997808e770 tx=0x7f9970010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:11.266 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:11.252+0000 7f9975d9b700 1 -- 172.21.15.42:0/3339359940 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9970010a00 con 0x7f997809f2e0 2019-12-05T05:28:11.266 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:11.252+0000 7f9975d9b700 1 -- 172.21.15.42:0/3339359940 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f99700114a0 con 0x7f997809f2e0 2019-12-05T05:28:11.266 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:11.252+0000 7f997cfaa700 1 -- 172.21.15.42:0/3339359940 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f997809f2e0 msgr2=0x7f997809f710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:28:11.267 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:11.252+0000 7f997cfaa700 1 --2- 172.21.15.42:0/3339359940 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f997809f2e0 0x7f997809f710 secure :-1 s=READY pgs=914 cs=0 l=1 rx=0x7f997808e770 tx=0x7f9970010130).stop 2019-12-05T05:28:11.267 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:11.252+0000 7f997cfaa700 1 -- 172.21.15.42:0/3339359940 shutdown_connections 2019-12-05T05:28:11.267 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:11.252+0000 7f997cfaa700 1 --2- 172.21.15.42:0/3339359940 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f997809f2e0 0x7f997809f710 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:28:11.267 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:11.252+0000 7f997cfaa700 1 --2- 172.21.15.42:0/3339359940 >> v2:172.21.15.40:3300/0 conn(0x7f99780a7740 0x7f997808df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:28:11.267 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:11.252+0000 7f997cfaa700 1 --2- 172.21.15.42:0/3339359940 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f997807e720 0x7f997807eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:28:11.267 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:11.252+0000 7f997cfaa700 1 -- 172.21.15.42:0/3339359940 shutdown_connections 2019-12-05T05:28:11.267 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:11.252+0000 7f997cfaa700 1 -- 172.21.15.42:0/3339359940 wait complete. 2019-12-05T05:28:11.268 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:11.252+0000 7f997cfaa700 1 -- 172.21.15.42:0/3339359940 >> 172.21.15.42:0/3339359940 conn(0x7f99780aabe0 msgr2=0x7f99780a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:28:11.268 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:11.256+0000 7f997cfaa700 1 Processor -- start 2019-12-05T05:28:11.268 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:11.256+0000 7f997cfaa700 1 -- start start 2019-12-05T05:28:11.268 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:11.256+0000 7f997cfaa700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f99780a62b0 0x7f99781233a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:28:11.268 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:11.256+0000 7f997cfaa700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f997813b9f0 0x7f997813de30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:28:11.281 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:11.256+0000 7f997cfaa700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f997807bd50 con 0x7f997807cdc0 2019-12-05T05:28:11.281 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:11.256+0000 7f997cfaa700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99780a4a50 con 0x7f99780a62b0 2019-12-05T05:28:11.281 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:11.256+0000 7f997cfaa700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99780a6ca0 con 0x7f997813b9f0 2019-12-05T05:28:11.281 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:11.256+0000 7f9976d9d700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f997807cdc0 0x7f99780a5ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:34394/0 (socket says 172.21.15.42:34394) 2019-12-05T05:28:11.281 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:11.256+0000 7f9976d9d700 1 -- 172.21.15.42:0/2313330121 learned_addr learned my addr 172.21.15.42:0/2313330121 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:28:11.282 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:11.256+0000 7f997659c700 1 --2- 172.21.15.42:0/2313330121 >> v2:172.21.15.40:3300/0 conn(0x7f99780a62b0 0x7f99781233a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:28:11.282 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:11.256+0000 7f997759e700 1 --2- 172.21.15.42:0/2313330121 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f997813b9f0 0x7f997813de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:28:11.282 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:11.256+0000 7f996f7fe700 1 -- 172.21.15.42:0/2313330121 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2320828336 0 0) 0x7f997807bd50 con 0x7f997807cdc0 2019-12-05T05:28:11.282 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:11.256+0000 7f996f7fe700 1 -- 172.21.15.42:0/2313330121 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f995c0030b0 con 0x7f997807cdc0 2019-12-05T05:28:11.285 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:11.256+0000 7f997759e700 1 -- 172.21.15.42:0/2313330121 >> v2:172.21.15.40:3300/0 conn(0x7f99780a62b0 msgr2=0x7f99781233a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:28:11.285 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:11.256+0000 7f997759e700 1 --2- 172.21.15.42:0/2313330121 >> v2:172.21.15.40:3300/0 conn(0x7f99780a62b0 0x7f99781233a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:28:11.285 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:11.256+0000 7f996f7fe700 1 -- 172.21.15.42:0/2313330121 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 2719344129 0 0) 0x7f995c0030b0 con 0x7f997807cdc0 2019-12-05T05:28:11.285 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:11.256+0000 7f997759e700 1 -- 172.21.15.42:0/2313330121 >> v1:172.21.15.42:6789/0 conn(0x7f997807cdc0 legacy=0x7f99780a5ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:28:11.285 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:11.256+0000 7f997759e700 1 -- 172.21.15.42:0/2313330121 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99781238f0 con 0x7f997813b9f0 2019-12-05T05:28:11.286 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:11.256+0000 7f997659c700 1 --2- 172.21.15.42:0/2313330121 >> v2:172.21.15.40:3300/0 conn(0x7f99780a62b0 0x7f99781233a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:28:11.286 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:11.256+0000 7f997759e700 1 --2- 172.21.15.42:0/2313330121 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f997813b9f0 0x7f997813de30 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f99600016d0 tx=0x7f9960002f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:11.286 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:11.260+0000 7f997759e700 1 --2- 172.21.15.42:0/2313330121 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f997813b9f0 0x7f997813de30 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f99600016d0 tx=0x7f9960002f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:11.287 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:11.260+0000 7f997759e700 1 --2- 172.21.15.42:0/2313330121 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f997813b9f0 0x7f997813de30 secure :-1 s=READY pgs=184 cs=0 l=1 rx=0x7f99600016d0 tx=0x7f9960002f70).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:28:11.287 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:11.260+0000 7f997759e700 1 --2- 172.21.15.42:0/2313330121 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f997813b9f0 0x7f997813de30 secure :-1 s=THROTTLE_DONE pgs=184 cs=0 l=1 rx=0x7f99600016d0 tx=0x7f9960002f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:11.287 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:11.260+0000 7f996f7fe700 1 -- 172.21.15.42:0/2313330121 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9960001490 con 0x7f997813b9f0 2019-12-05T05:28:11.287 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:11.260+0000 7f997759e700 1 --2- 172.21.15.42:0/2313330121 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f997813b9f0 0x7f997813de30 secure :-1 s=THROTTLE_DONE pgs=184 cs=0 l=1 rx=0x7f99600016d0 tx=0x7f9960002f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:11.287 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:11.260+0000 7f997759e700 1 --2- 172.21.15.42:0/2313330121 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f997813b9f0 0x7f997813de30 secure :-1 s=THROTTLE_DONE pgs=184 cs=0 l=1 rx=0x7f99600016d0 tx=0x7f9960002f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:11.288 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:11.260+0000 7f996f7fe700 1 -- 172.21.15.42:0/2313330121 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99600012a0 con 0x7f997813b9f0 2019-12-05T05:28:11.288 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:11.260+0000 7f996f7fe700 1 -- 172.21.15.42:0/2313330121 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9970011d60 con 0x7f997813b9f0 2019-12-05T05:28:11.288 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:11.260+0000 7f997cfaa700 1 -- 172.21.15.42:0/2313330121 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f997807c0f0 con 0x7f997813b9f0 2019-12-05T05:28:11.288 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:11.260+0000 7f997cfaa700 1 -- 172.21.15.42:0/2313330121 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f997807b340 con 0x7f997813b9f0 2019-12-05T05:28:11.288 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:11.260+0000 7f997759e700 1 --2- 172.21.15.42:0/2313330121 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f997813b9f0 0x7f997813de30 secure :-1 s=THROTTLE_DONE pgs=184 cs=0 l=1 rx=0x7f99600016d0 tx=0x7f9960002f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:11.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:11.260+0000 7f996f7fe700 1 -- 172.21.15.42:0/2313330121 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f99700058c0 con 0x7f997813b9f0 2019-12-05T05:28:11.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:11.260+0000 7f997759e700 1 --2- 172.21.15.42:0/2313330121 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f997813b9f0 0x7f997813de30 secure :-1 s=THROTTLE_DONE pgs=184 cs=0 l=1 rx=0x7f99600016d0 tx=0x7f9960002f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:11.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:11.260+0000 7f996f7fe700 1 --2- 172.21.15.42:0/2313330121 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f995c03b230 0x7f995c03d740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:28:11.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:11.260+0000 7f996f7fe700 1 -- 172.21.15.42:0/2313330121 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7407+0+0 (secure 0 0 0) 0x7f997008f0c0 con 0x7f997813b9f0 2019-12-05T05:28:11.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:11.260+0000 7f9976d9d700 1 --2- 172.21.15.42:0/2313330121 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f995c03b230 0x7f995c03d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:28:11.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:11.260+0000 7f9976d9d700 1 --2- 172.21.15.42:0/2313330121 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f995c03b230 0x7f995c03d740 crc :-1 s=READY pgs=399 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:28:11.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:11.264+0000 7f997cfaa700 1 -- 172.21.15.42:0/2313330121 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f997805d110 con 0x7f997813b9f0 2019-12-05T05:28:11.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:11.272+0000 7f997759e700 1 --2- 172.21.15.42:0/2313330121 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f997813b9f0 0x7f997813de30 secure :-1 s=THROTTLE_DONE pgs=184 cs=0 l=1 rx=0x7f99600016d0 tx=0x7f9960002f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:11.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:11.276+0000 7f996f7fe700 1 -- 172.21.15.42:0/2313330121 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f9970007a90 con 0x7f997813b9f0 2019-12-05T05:28:11.447 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:11.440+0000 7f997cfaa700 1 -- 172.21.15.42:0/2313330121 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f997805b8c0 con 0x7f997813b9f0 2019-12-05T05:28:11.448 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:11.440+0000 7f997759e700 1 --2- 172.21.15.42:0/2313330121 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f997813b9f0 0x7f997813de30 secure :-1 s=THROTTLE_DONE pgs=184 cs=0 l=1 rx=0x7f99600016d0 tx=0x7f9960002f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:11.448 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:11.444+0000 7f996f7fe700 1 -- 172.21.15.42:0/2313330121 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 113 v113) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f9970076120 con 0x7f997813b9f0 2019-12-05T05:28:11.453 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:28:11.453 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":113,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6666,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/2544447110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2544447110},{"type":"v1","addr":"172.21.15.42:6835","nonce":2544447110}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":113}],"filesystems":[{"mdsmap":{"epoch":112,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:27:13.322209+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6546},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6546":{"gid":6546,"name":"b","rank":0,"incarnation":109,"state":"up:active","state_seq":25,"addr":"172.21.15.40:6833/1149478696","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1149478696},{"type":"v1","addr":"172.21.15.40:6833","nonce":1149478696}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:28:11.453 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 113 2019-12-05T05:28:11.453 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:11.448+0000 7f997cfaa700 1 -- 172.21.15.42:0/2313330121 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f995c03b230 msgr2=0x7f995c03d740 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:28:11.454 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:11.448+0000 7f997cfaa700 1 --2- 172.21.15.42:0/2313330121 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f995c03b230 0x7f995c03d740 crc :-1 s=READY pgs=399 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:28:11.454 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:11.448+0000 7f997cfaa700 1 -- 172.21.15.42:0/2313330121 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f997813b9f0 msgr2=0x7f997813de30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:28:11.454 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:11.448+0000 7f997cfaa700 1 --2- 172.21.15.42:0/2313330121 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f997813b9f0 0x7f997813de30 secure :-1 s=READY pgs=184 cs=0 l=1 rx=0x7f99600016d0 tx=0x7f9960002f70).stop 2019-12-05T05:28:11.454 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:11.448+0000 7f997cfaa700 1 -- 172.21.15.42:0/2313330121 shutdown_connections 2019-12-05T05:28:11.454 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:11.448+0000 7f997cfaa700 1 --2- 172.21.15.42:0/2313330121 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f995c03b230 0x7f995c03d740 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:28:11.454 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:11.448+0000 7f997cfaa700 1 --2- 172.21.15.42:0/2313330121 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f997813b9f0 0x7f997813de30 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:28:11.455 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:11.448+0000 7f997cfaa700 1 --2- 172.21.15.42:0/2313330121 >> v2:172.21.15.40:3300/0 conn(0x7f99780a62b0 0x7f99781233a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:28:11.456 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:11.448+0000 7f997cfaa700 1 -- 172.21.15.42:0/2313330121 shutdown_connections 2019-12-05T05:28:11.456 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:11.448+0000 7f997cfaa700 1 -- 172.21.15.42:0/2313330121 wait complete. 2019-12-05T05:28:11.456 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:11.448+0000 7f997cfaa700 1 -- 172.21.15.42:0/2313330121 >> 172.21.15.42:0/2313330121 conn(0x7f997809fa10 msgr2=0x7f9978059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:28:11.463 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T05:28:13.465 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:28:13.571 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.560+0000 7ff42019f700 1 Processor -- start 2019-12-05T05:28:13.571 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.560+0000 7ff42019f700 1 -- start start 2019-12-05T05:28:13.571 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.560+0000 7ff42019f700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff4180a7730 0x7ff41808df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:28:13.571 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.560+0000 7ff42019f700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff41807e710 0x7ff41807eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:28:13.572 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.560+0000 7ff42019f700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff41809f2d0 0x7ff41809f700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:28:13.572 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.560+0000 7ff42019f700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4180a0af0 con 0x7ff4180a7730 2019-12-05T05:28:13.572 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.560+0000 7ff42019f700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff41808e5a0 con 0x7ff41807e710 2019-12-05T05:28:13.572 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.560+0000 7ff42019f700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff41808e840 con 0x7ff41809f2d0 2019-12-05T05:28:13.572 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.560+0000 7ff41df3b700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff41807e710 0x7ff41807eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:28:13.572 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.560+0000 7ff41df3b700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff41807e710 0x7ff41807eb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:34368/0 (socket says 172.21.15.42:34368) 2019-12-05T05:28:13.573 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.560+0000 7ff41df3b700 1 -- 172.21.15.42:0/4217061898 learned_addr learned my addr 172.21.15.42:0/4217061898 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:28:13.573 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.560+0000 7ff41e73c700 1 --2- 172.21.15.42:0/4217061898 >> v2:172.21.15.40:3300/0 conn(0x7ff4180a7730 0x7ff41808df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:28:13.573 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.560+0000 7ff41ef3d700 1 --2- 172.21.15.42:0/4217061898 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff41809f2d0 0x7ff41809f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:28:13.573 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.564+0000 7ff41df3b700 1 -- 172.21.15.42:0/4217061898 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff41809f2d0 msgr2=0x7ff41809f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:28:13.573 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.564+0000 7ff41df3b700 1 --2- 172.21.15.42:0/4217061898 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff41809f2d0 0x7ff41809f700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:28:13.574 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.564+0000 7ff41df3b700 1 -- 172.21.15.42:0/4217061898 >> v2:172.21.15.40:3300/0 conn(0x7ff4180a7730 msgr2=0x7ff41808df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:28:13.574 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.564+0000 7ff41df3b700 1 --2- 172.21.15.42:0/4217061898 >> v2:172.21.15.40:3300/0 conn(0x7ff4180a7730 0x7ff41808df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:28:13.574 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.564+0000 7ff41df3b700 1 -- 172.21.15.42:0/4217061898 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff41807f090 con 0x7ff41807e710 2019-12-05T05:28:13.574 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.564+0000 7ff41df3b700 1 --2- 172.21.15.42:0/4217061898 >> v2:172.21.15.42:3300/0 conn(0x7ff41807e710 0x7ff41807eb40 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7ff4180a0f40 tx=0x7ff40c00fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:13.574 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.564+0000 7ff41df3b700 1 --2- 172.21.15.42:0/4217061898 >> v2:172.21.15.42:3300/0 conn(0x7ff41807e710 0x7ff41807eb40 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7ff4180a0f40 tx=0x7ff40c00fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:13.574 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.564+0000 7ff41df3b700 1 --2- 172.21.15.42:0/4217061898 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff41807e710 0x7ff41807eb40 secure :-1 s=READY pgs=916 cs=0 l=1 rx=0x7ff4180a0f40 tx=0x7ff40c00fdb0).ready entity=mon.1 client_cookie=7dac33da725195d0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:28:13.575 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.564+0000 7ff41df3b700 1 --2- 172.21.15.42:0/4217061898 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff41807e710 0x7ff41807eb40 secure :-1 s=THROTTLE_DONE pgs=916 cs=0 l=1 rx=0x7ff4180a0f40 tx=0x7ff40c00fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:13.575 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.564+0000 7ff41df3b700 1 --2- 172.21.15.42:0/4217061898 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff41807e710 0x7ff41807eb40 secure :-1 s=THROTTLE_DONE pgs=916 cs=0 l=1 rx=0x7ff4180a0f40 tx=0x7ff40c00fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:13.575 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.564+0000 7ff41df3b700 1 --2- 172.21.15.42:0/4217061898 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff41807e710 0x7ff41807eb40 secure :-1 s=THROTTLE_DONE pgs=916 cs=0 l=1 rx=0x7ff4180a0f40 tx=0x7ff40c00fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:13.575 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.564+0000 7ff41d73a700 1 -- 172.21.15.42:0/4217061898 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff40c008c90 con 0x7ff41807e710 2019-12-05T05:28:13.575 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.564+0000 7ff41d73a700 1 -- 172.21.15.42:0/4217061898 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff40c0088b0 con 0x7ff41807e710 2019-12-05T05:28:13.575 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.564+0000 7ff41d73a700 1 -- 172.21.15.42:0/4217061898 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff40c011b50 con 0x7ff41807e710 2019-12-05T05:28:13.576 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.564+0000 7ff42019f700 1 -- 172.21.15.42:0/4217061898 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff41807e710 msgr2=0x7ff41807eb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:28:13.576 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.564+0000 7ff42019f700 1 --2- 172.21.15.42:0/4217061898 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff41807e710 0x7ff41807eb40 secure :-1 s=READY pgs=916 cs=0 l=1 rx=0x7ff4180a0f40 tx=0x7ff40c00fdb0).stop 2019-12-05T05:28:13.576 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.564+0000 7ff42019f700 1 -- 172.21.15.42:0/4217061898 shutdown_connections 2019-12-05T05:28:13.576 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.564+0000 7ff42019f700 1 --2- 172.21.15.42:0/4217061898 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff41809f2d0 0x7ff41809f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:28:13.577 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.564+0000 7ff42019f700 1 --2- 172.21.15.42:0/4217061898 >> v2:172.21.15.40:3300/0 conn(0x7ff4180a7730 0x7ff41808df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:28:13.577 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.564+0000 7ff42019f700 1 --2- 172.21.15.42:0/4217061898 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff41807e710 0x7ff41807eb40 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:28:13.577 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.564+0000 7ff42019f700 1 -- 172.21.15.42:0/4217061898 shutdown_connections 2019-12-05T05:28:13.577 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.564+0000 7ff42019f700 1 -- 172.21.15.42:0/4217061898 wait complete. 2019-12-05T05:28:13.577 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.564+0000 7ff42019f700 1 -- 172.21.15.42:0/4217061898 >> 172.21.15.42:0/4217061898 conn(0x7ff4180aabd0 msgr2=0x7ff4180a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:28:13.588 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.568+0000 7ff42019f700 1 Processor -- start 2019-12-05T05:28:13.588 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.568+0000 7ff42019f700 1 -- start start 2019-12-05T05:28:13.589 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.568+0000 7ff42019f700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff41807b9c0 0x7ff418121460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:28:13.589 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.568+0000 7ff42019f700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff41811add0 0x7ff41807d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:28:13.589 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.568+0000 7ff42019f700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff41811b3e0 con 0x7ff41807d640 2019-12-05T05:28:13.589 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.568+0000 7ff42019f700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff41807c250 con 0x7ff41811add0 2019-12-05T05:28:13.589 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.568+0000 7ff42019f700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff41811bf10 con 0x7ff41807b9c0 2019-12-05T05:28:13.589 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.568+0000 7ff41e73c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff41807b9c0 0x7ff418121460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:28:13.590 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.568+0000 7ff41e73c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff41807b9c0 0x7ff418121460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:34372/0 (socket says 172.21.15.42:34372) 2019-12-05T05:28:13.590 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.568+0000 7ff41e73c700 1 -- 172.21.15.42:0/3957876378 learned_addr learned my addr 172.21.15.42:0/3957876378 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:28:13.590 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.568+0000 7ff41df3b700 1 --2- 172.21.15.42:0/3957876378 >> v2:172.21.15.40:3300/0 conn(0x7ff41811add0 0x7ff41807d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:28:13.590 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.568+0000 7ff40affd700 1 -- 172.21.15.42:0/3957876378 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4253476607 0 0) 0x7ff41811b3e0 con 0x7ff41807d640 2019-12-05T05:28:13.591 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.568+0000 7ff40affd700 1 -- 172.21.15.42:0/3957876378 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4000030b0 con 0x7ff41807d640 2019-12-05T05:28:13.591 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.568+0000 7ff41e73c700 1 -- 172.21.15.42:0/3957876378 >> v2:172.21.15.40:3300/0 conn(0x7ff41811add0 msgr2=0x7ff41807d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:28:13.591 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.568+0000 7ff41e73c700 1 --2- 172.21.15.42:0/3957876378 >> v2:172.21.15.40:3300/0 conn(0x7ff41811add0 0x7ff41807d0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:28:13.591 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.568+0000 7ff41e73c700 1 -- 172.21.15.42:0/3957876378 >> v1:172.21.15.40:6789/0 conn(0x7ff41807d640 legacy=0x7ff418130120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:28:13.591 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.568+0000 7ff41e73c700 1 -- 172.21.15.42:0/3957876378 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4181219b0 con 0x7ff41807b9c0 2019-12-05T05:28:13.592 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.568+0000 7ff41df3b700 1 --2- 172.21.15.42:0/3957876378 >> v2:172.21.15.40:3300/0 conn(0x7ff41811add0 0x7ff41807d0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:28:13.592 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.568+0000 7ff41e73c700 1 --2- 172.21.15.42:0/3957876378 >> v2:172.21.15.42:3300/0 conn(0x7ff41807b9c0 0x7ff418121460 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7ff414003070 tx=0x7ff414010ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:13.592 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.568+0000 7ff41e73c700 1 --2- 172.21.15.42:0/3957876378 >> v2:172.21.15.42:3300/0 conn(0x7ff41807b9c0 0x7ff418121460 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7ff414003070 tx=0x7ff414010ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:13.592 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.568+0000 7ff41e73c700 1 --2- 172.21.15.42:0/3957876378 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff41807b9c0 0x7ff418121460 secure :-1 s=READY pgs=917 cs=0 l=1 rx=0x7ff414003070 tx=0x7ff414010ae0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:28:13.592 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.568+0000 7ff41e73c700 1 --2- 172.21.15.42:0/3957876378 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff41807b9c0 0x7ff418121460 secure :-1 s=THROTTLE_DONE pgs=917 cs=0 l=1 rx=0x7ff414003070 tx=0x7ff414010ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:13.592 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.568+0000 7ff40affd700 1 -- 172.21.15.42:0/3957876378 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff414005930 con 0x7ff41807b9c0 2019-12-05T05:28:13.593 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.568+0000 7ff41e73c700 1 --2- 172.21.15.42:0/3957876378 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff41807b9c0 0x7ff418121460 secure :-1 s=THROTTLE_DONE pgs=917 cs=0 l=1 rx=0x7ff414003070 tx=0x7ff414010ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:13.593 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.568+0000 7ff41e73c700 1 --2- 172.21.15.42:0/3957876378 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff41807b9c0 0x7ff418121460 secure :-1 s=THROTTLE_DONE pgs=917 cs=0 l=1 rx=0x7ff414003070 tx=0x7ff414010ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:13.593 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.568+0000 7ff40affd700 1 -- 172.21.15.42:0/3957876378 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff414007160 con 0x7ff41807b9c0 2019-12-05T05:28:13.593 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.568+0000 7ff42019f700 1 -- 172.21.15.42:0/3957876378 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff418120af0 con 0x7ff41807b9c0 2019-12-05T05:28:13.593 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.568+0000 7ff40affd700 1 -- 172.21.15.42:0/3957876378 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff414012160 con 0x7ff41807b9c0 2019-12-05T05:28:13.593 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.568+0000 7ff42019f700 1 -- 172.21.15.42:0/3957876378 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff41811a0d0 con 0x7ff41807b9c0 2019-12-05T05:28:13.593 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.572+0000 7ff41e73c700 1 --2- 172.21.15.42:0/3957876378 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff41807b9c0 0x7ff418121460 secure :-1 s=THROTTLE_DONE pgs=917 cs=0 l=1 rx=0x7ff414003070 tx=0x7ff414010ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:13.594 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.572+0000 7ff41e73c700 1 --2- 172.21.15.42:0/3957876378 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff41807b9c0 0x7ff418121460 secure :-1 s=THROTTLE_DONE pgs=917 cs=0 l=1 rx=0x7ff414003070 tx=0x7ff414010ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:13.594 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.572+0000 7ff40affd700 1 -- 172.21.15.42:0/3957876378 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7ff414012470 con 0x7ff41807b9c0 2019-12-05T05:28:13.594 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.572+0000 7ff40affd700 1 --2- 172.21.15.42:0/3957876378 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff40003b2d0 0x7ff40003d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:28:13.594 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.572+0000 7ff40affd700 1 -- 172.21.15.42:0/3957876378 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7407+0+0 (secure 0 0 0) 0x7ff41404ae60 con 0x7ff41807b9c0 2019-12-05T05:28:13.594 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.572+0000 7ff41df3b700 1 --2- 172.21.15.42:0/3957876378 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff40003b2d0 0x7ff40003d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:28:13.595 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.572+0000 7ff41df3b700 1 --2- 172.21.15.42:0/3957876378 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff40003b2d0 0x7ff40003d7e0 crc :-1 s=READY pgs=400 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:28:13.595 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.576+0000 7ff42019f700 1 -- 172.21.15.42:0/3957876378 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff41805d100 con 0x7ff41807b9c0 2019-12-05T05:28:13.595 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.584+0000 7ff41e73c700 1 --2- 172.21.15.42:0/3957876378 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff41807b9c0 0x7ff418121460 secure :-1 s=THROTTLE_DONE pgs=917 cs=0 l=1 rx=0x7ff414003070 tx=0x7ff414010ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:13.596 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.584+0000 7ff40affd700 1 -- 172.21.15.42:0/3957876378 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7ff414031ea0 con 0x7ff41807b9c0 2019-12-05T05:28:13.814 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.808+0000 7ff42019f700 1 -- 172.21.15.42:0/3957876378 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff41805b8b0 con 0x7ff41807b9c0 2019-12-05T05:28:13.815 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.808+0000 7ff41e73c700 1 --2- 172.21.15.42:0/3957876378 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff41807b9c0 0x7ff418121460 secure :-1 s=THROTTLE_DONE pgs=917 cs=0 l=1 rx=0x7ff414003070 tx=0x7ff414010ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:13.815 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.808+0000 7ff40affd700 1 -- 172.21.15.42:0/3957876378 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 114 v114) v1 ==== 95+0+1774 (secure 0 0 0) 0x7ff414032100 con 0x7ff41807b9c0 2019-12-05T05:28:13.817 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 114 2019-12-05T05:28:13.817 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:28:13.817 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":114,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":114,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:28:13.632148+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6666},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6666":{"gid":6666,"name":"a","rank":0,"incarnation":114,"state":"up:replay","state_seq":2,"addr":"172.21.15.42:6835/2544447110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2544447110},{"type":"v1","addr":"172.21.15.42:6835","nonce":2544447110}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:28:13.818 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.812+0000 7ff42019f700 1 -- 172.21.15.42:0/3957876378 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff40003b2d0 msgr2=0x7ff40003d7e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:28:13.819 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.812+0000 7ff42019f700 1 --2- 172.21.15.42:0/3957876378 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff40003b2d0 0x7ff40003d7e0 crc :-1 s=READY pgs=400 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:28:13.819 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.812+0000 7ff42019f700 1 -- 172.21.15.42:0/3957876378 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff41807b9c0 msgr2=0x7ff418121460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:28:13.819 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.812+0000 7ff42019f700 1 --2- 172.21.15.42:0/3957876378 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff41807b9c0 0x7ff418121460 secure :-1 s=READY pgs=917 cs=0 l=1 rx=0x7ff414003070 tx=0x7ff414010ae0).stop 2019-12-05T05:28:13.819 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.812+0000 7ff42019f700 1 -- 172.21.15.42:0/3957876378 shutdown_connections 2019-12-05T05:28:13.819 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.812+0000 7ff42019f700 1 --2- 172.21.15.42:0/3957876378 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff40003b2d0 0x7ff40003d7e0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:28:13.820 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.812+0000 7ff42019f700 1 --2- 172.21.15.42:0/3957876378 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff41807b9c0 0x7ff418121460 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:28:13.820 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.812+0000 7ff42019f700 1 --2- 172.21.15.42:0/3957876378 >> v2:172.21.15.40:3300/0 conn(0x7ff41811add0 0x7ff41807d0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:28:13.820 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.812+0000 7ff42019f700 1 -- 172.21.15.42:0/3957876378 shutdown_connections 2019-12-05T05:28:13.820 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.812+0000 7ff42019f700 1 -- 172.21.15.42:0/3957876378 wait complete. 2019-12-05T05:28:13.820 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.812+0000 7ff42019f700 1 -- 172.21.15.42:0/3957876378 >> 172.21.15.42:0/3957876378 conn(0x7ff41808fd90 msgr2=0x7ff4180596f0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:28:13.830 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b down, removed from mdsmap 2019-12-05T05:28:13.830 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2019-12-05T05:28:13.830 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:28:13.919 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.908+0000 7fe04f581700 1 Processor -- start 2019-12-05T05:28:13.920 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.908+0000 7fe04f581700 1 -- start start 2019-12-05T05:28:13.920 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.912+0000 7fe04f581700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe0480a7740 0x7fe04808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:28:13.920 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.912+0000 7fe04f581700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe04809fae0 0x7fe04807eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:28:13.920 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.912+0000 7fe04f581700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0480a0710 con 0x7fe04807e720 2019-12-05T05:28:13.920 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.912+0000 7fe04f581700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0480a0b00 con 0x7fe04809fae0 2019-12-05T05:28:13.920 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.912+0000 7fe04f581700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe04808e5b0 con 0x7fe0480a7740 2019-12-05T05:28:13.921 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.912+0000 7fe04db1e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe0480a7740 0x7fe04808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:28:13.921 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.912+0000 7fe04db1e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe0480a7740 0x7fe04808df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:34394/0 (socket says 172.21.15.42:34394) 2019-12-05T05:28:13.921 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.912+0000 7fe04db1e700 1 -- 172.21.15.42:0/152846908 learned_addr learned my addr 172.21.15.42:0/152846908 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:28:13.921 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.912+0000 7fe04d31d700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fe04807e720 0x7fe04809f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:34434/0 (socket says 172.21.15.42:34434) 2019-12-05T05:28:13.921 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.912+0000 7fe04e31f700 1 --2- 172.21.15.42:0/152846908 >> v2:172.21.15.40:3300/0 conn(0x7fe04809fae0 0x7fe04807eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:28:13.922 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.912+0000 7fe04db1e700 1 -- 172.21.15.42:0/152846908 >> v2:172.21.15.40:3300/0 conn(0x7fe04809fae0 msgr2=0x7fe04807eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:28:13.922 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.912+0000 7fe04db1e700 1 --2- 172.21.15.42:0/152846908 >> v2:172.21.15.40:3300/0 conn(0x7fe04809fae0 0x7fe04807eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:28:13.922 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.912+0000 7fe04db1e700 1 -- 172.21.15.42:0/152846908 >> v1:172.21.15.42:6789/0 conn(0x7fe04807e720 legacy=0x7fe04809f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:28:13.922 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.912+0000 7fe04db1e700 1 -- 172.21.15.42:0/152846908 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe04807f0a0 con 0x7fe0480a7740 2019-12-05T05:28:13.922 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.912+0000 7fe04db1e700 1 --2- 172.21.15.42:0/152846908 >> v2:172.21.15.42:3300/0 conn(0x7fe0480a7740 0x7fe04808df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fe0400032b0 tx=0x7fe0400100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:13.922 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.912+0000 7fe04db1e700 1 --2- 172.21.15.42:0/152846908 >> v2:172.21.15.42:3300/0 conn(0x7fe0480a7740 0x7fe04808df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fe0400032b0 tx=0x7fe0400100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:13.922 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.912+0000 7fe04db1e700 1 --2- 172.21.15.42:0/152846908 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe0480a7740 0x7fe04808df80 secure :-1 s=READY pgs=919 cs=0 l=1 rx=0x7fe0400032b0 tx=0x7fe0400100b0).ready entity=mon.1 client_cookie=3a2d600989435e38 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:28:13.923 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.912+0000 7fe04db1e700 1 --2- 172.21.15.42:0/152846908 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe0480a7740 0x7fe04808df80 secure :-1 s=THROTTLE_DONE pgs=919 cs=0 l=1 rx=0x7fe0400032b0 tx=0x7fe0400100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:13.923 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.912+0000 7fe04db1e700 1 --2- 172.21.15.42:0/152846908 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe0480a7740 0x7fe04808df80 secure :-1 s=THROTTLE_DONE pgs=919 cs=0 l=1 rx=0x7fe0400032b0 tx=0x7fe0400100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:13.924 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.912+0000 7fe04cb1c700 1 -- 172.21.15.42:0/152846908 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe040010d60 con 0x7fe0480a7740 2019-12-05T05:28:13.924 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.912+0000 7fe04db1e700 1 --2- 172.21.15.42:0/152846908 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe0480a7740 0x7fe04808df80 secure :-1 s=THROTTLE_DONE pgs=919 cs=0 l=1 rx=0x7fe0400032b0 tx=0x7fe0400100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:13.924 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.912+0000 7fe04cb1c700 1 -- 172.21.15.42:0/152846908 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe040010980 con 0x7fe0480a7740 2019-12-05T05:28:13.924 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.912+0000 7fe04cb1c700 1 -- 172.21.15.42:0/152846908 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe0400114a0 con 0x7fe0480a7740 2019-12-05T05:28:13.924 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.912+0000 7fe04f581700 1 -- 172.21.15.42:0/152846908 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe0480a7740 msgr2=0x7fe04808df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:28:13.924 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.912+0000 7fe04f581700 1 --2- 172.21.15.42:0/152846908 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe0480a7740 0x7fe04808df80 secure :-1 s=READY pgs=919 cs=0 l=1 rx=0x7fe0400032b0 tx=0x7fe0400100b0).stop 2019-12-05T05:28:13.924 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.912+0000 7fe04f581700 1 -- 172.21.15.42:0/152846908 shutdown_connections 2019-12-05T05:28:13.925 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.912+0000 7fe04f581700 1 --2- 172.21.15.42:0/152846908 >> v2:172.21.15.40:3300/0 conn(0x7fe04809fae0 0x7fe04807eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:28:13.925 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.912+0000 7fe04f581700 1 --2- 172.21.15.42:0/152846908 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe0480a7740 0x7fe04808df80 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:28:13.925 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.912+0000 7fe04f581700 1 -- 172.21.15.42:0/152846908 shutdown_connections 2019-12-05T05:28:13.925 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.912+0000 7fe04f581700 1 -- 172.21.15.42:0/152846908 wait complete. 2019-12-05T05:28:13.925 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.912+0000 7fe04f581700 1 -- 172.21.15.42:0/152846908 >> 172.21.15.42:0/152846908 conn(0x7fe0480aabe0 msgr2=0x7fe0480a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:28:13.925 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.912+0000 7fe04f581700 1 Processor -- start 2019-12-05T05:28:13.925 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.912+0000 7fe04f581700 1 -- start start 2019-12-05T05:28:13.928 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.916+0000 7fe04f581700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe048123c30 0x7fe048079f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:28:13.928 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.916+0000 7fe04f581700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe0480a51e0 0x7fe0480a5610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:28:13.928 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.916+0000 7fe04f581700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0480a33e0 con 0x7fe04807a4a0 2019-12-05T05:28:13.929 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.916+0000 7fe04f581700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe048124460 con 0x7fe0480a51e0 2019-12-05T05:28:13.929 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.916+0000 7fe04f581700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0480a4f40 con 0x7fe048123c30 2019-12-05T05:28:13.929 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.916+0000 7fe04e31f700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe0480a51e0 0x7fe0480a5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:28:13.929 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.916+0000 7fe04e31f700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe0480a51e0 0x7fe0480a5610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:34404/0 (socket says 172.21.15.42:34404) 2019-12-05T05:28:13.929 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.916+0000 7fe04e31f700 1 -- 172.21.15.42:0/3142650483 learned_addr learned my addr 172.21.15.42:0/3142650483 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:28:13.929 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.916+0000 7fe04db1e700 1 --2- 172.21.15.42:0/3142650483 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe048123c30 0x7fe048079f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:28:13.929 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.916+0000 7fe04e31f700 1 -- 172.21.15.42:0/3142650483 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe048123c30 msgr2=0x7fe048079f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:28:13.930 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.916+0000 7fe04e31f700 1 --2- 172.21.15.42:0/3142650483 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe048123c30 0x7fe048079f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:28:13.930 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.916+0000 7fe04e31f700 1 -- 172.21.15.42:0/3142650483 >> v1:172.21.15.40:6789/0 conn(0x7fe04807a4a0 legacy=0x7fe0480a2be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:28:13.930 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.916+0000 7fe0467fc700 1 -- 172.21.15.42:0/3142650483 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3799425934 0 0) 0x7fe0480a33e0 con 0x7fe04807a4a0 2019-12-05T05:28:13.930 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.916+0000 7fe04e31f700 1 -- 172.21.15.42:0/3142650483 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe048124060 con 0x7fe0480a51e0 2019-12-05T05:28:13.930 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.916+0000 7fe04e31f700 1 --2- 172.21.15.42:0/3142650483 >> v2:172.21.15.42:3300/0 conn(0x7fe0480a51e0 0x7fe0480a5610 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fe048124620 tx=0x7fe038010950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:13.930 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.916+0000 7fe04e31f700 1 --2- 172.21.15.42:0/3142650483 >> v2:172.21.15.42:3300/0 conn(0x7fe0480a51e0 0x7fe0480a5610 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fe048124620 tx=0x7fe038010950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:13.930 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.916+0000 7fe04e31f700 1 --2- 172.21.15.42:0/3142650483 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe0480a51e0 0x7fe0480a5610 secure :-1 s=READY pgs=920 cs=0 l=1 rx=0x7fe048124620 tx=0x7fe038010950).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:28:13.931 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.916+0000 7fe04e31f700 1 --2- 172.21.15.42:0/3142650483 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe0480a51e0 0x7fe0480a5610 secure :-1 s=THROTTLE_DONE pgs=920 cs=0 l=1 rx=0x7fe048124620 tx=0x7fe038010950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:13.931 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.916+0000 7fe0467fc700 1 -- 172.21.15.42:0/3142650483 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe038005920 con 0x7fe0480a51e0 2019-12-05T05:28:13.931 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.916+0000 7fe04e31f700 1 --2- 172.21.15.42:0/3142650483 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe0480a51e0 0x7fe0480a5610 secure :-1 s=THROTTLE_DONE pgs=920 cs=0 l=1 rx=0x7fe048124620 tx=0x7fe038010950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:13.931 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.916+0000 7fe04f581700 1 -- 172.21.15.42:0/3142650483 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0480a5b60 con 0x7fe0480a51e0 2019-12-05T05:28:13.931 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.916+0000 7fe0467fc700 1 -- 172.21.15.42:0/3142650483 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe038005540 con 0x7fe0480a51e0 2019-12-05T05:28:13.931 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.916+0000 7fe04e31f700 1 --2- 172.21.15.42:0/3142650483 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe0480a51e0 0x7fe0480a5610 secure :-1 s=THROTTLE_DONE pgs=920 cs=0 l=1 rx=0x7fe048124620 tx=0x7fe038010950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:13.931 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.916+0000 7fe0467fc700 1 -- 172.21.15.42:0/3142650483 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe038001280 con 0x7fe0480a51e0 2019-12-05T05:28:13.931 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.916+0000 7fe04f581700 1 -- 172.21.15.42:0/3142650483 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe04812e2d0 con 0x7fe0480a51e0 2019-12-05T05:28:13.932 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.916+0000 7fe04e31f700 1 --2- 172.21.15.42:0/3142650483 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe0480a51e0 0x7fe0480a5610 secure :-1 s=THROTTLE_DONE pgs=920 cs=0 l=1 rx=0x7fe048124620 tx=0x7fe038010950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:13.932 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.916+0000 7fe0467fc700 1 -- 172.21.15.42:0/3142650483 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fe038001520 con 0x7fe0480a51e0 2019-12-05T05:28:13.932 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.916+0000 7fe04e31f700 1 --2- 172.21.15.42:0/3142650483 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe0480a51e0 0x7fe0480a5610 secure :-1 s=THROTTLE_DONE pgs=920 cs=0 l=1 rx=0x7fe048124620 tx=0x7fe038010950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:13.932 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.916+0000 7fe0467fc700 1 --2- 172.21.15.42:0/3142650483 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe02803aca0 0x7fe02803d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:28:13.932 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.916+0000 7fe0467fc700 1 -- 172.21.15.42:0/3142650483 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 7493+0+0 (secure 0 0 0) 0x7fe03804ad20 con 0x7fe0480a51e0 2019-12-05T05:28:13.932 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.916+0000 7fe04db1e700 1 --2- 172.21.15.42:0/3142650483 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe02803aca0 0x7fe02803d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:28:13.933 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.920+0000 7fe04db1e700 1 --2- 172.21.15.42:0/3142650483 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe02803aca0 0x7fe02803d1b0 crc :-1 s=READY pgs=401 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:28:13.933 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.924+0000 7fe04f581700 1 -- 172.21.15.42:0/3142650483 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe04805d110 con 0x7fe0480a51e0 2019-12-05T05:28:13.936 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.932+0000 7fe04e31f700 1 --2- 172.21.15.42:0/3142650483 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe0480a51e0 0x7fe0480a5610 secure :-1 s=THROTTLE_DONE pgs=920 cs=0 l=1 rx=0x7fe048124620 tx=0x7fe038010950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:13.936 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:13.932+0000 7fe0467fc700 1 -- 172.21.15.42:0/3142650483 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fe038031d80 con 0x7fe0480a51e0 2019-12-05T05:28:14.096 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:14.092+0000 7fe04f581700 1 -- 172.21.15.42:0/3142650483 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe04805b8c0 con 0x7fe0480a51e0 2019-12-05T05:28:14.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:14.092+0000 7fe04e31f700 1 --2- 172.21.15.42:0/3142650483 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe0480a51e0 0x7fe0480a5610 secure :-1 s=THROTTLE_DONE pgs=920 cs=0 l=1 rx=0x7fe048124620 tx=0x7fe038010950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:14.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:14.092+0000 7fe0467fc700 1 -- 172.21.15.42:0/3142650483 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 114 v114) v1 ==== 95+0+1774 (secure 0 0 0) 0x7fe038031fe0 con 0x7fe0480a51e0 2019-12-05T05:28:14.099 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 114 2019-12-05T05:28:14.099 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:28:14.099 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":114,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":114,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:28:13.632148+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6666},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6666":{"gid":6666,"name":"a","rank":0,"incarnation":114,"state":"up:replay","state_seq":2,"addr":"172.21.15.42:6835/2544447110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2544447110},{"type":"v1","addr":"172.21.15.42:6835","nonce":2544447110}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:28:14.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:14.096+0000 7fe04f581700 1 -- 172.21.15.42:0/3142650483 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe02803aca0 msgr2=0x7fe02803d1b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:28:14.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:14.096+0000 7fe04f581700 1 --2- 172.21.15.42:0/3142650483 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe02803aca0 0x7fe02803d1b0 crc :-1 s=READY pgs=401 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:28:14.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:14.096+0000 7fe04f581700 1 -- 172.21.15.42:0/3142650483 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe0480a51e0 msgr2=0x7fe0480a5610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:28:14.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:14.096+0000 7fe04f581700 1 --2- 172.21.15.42:0/3142650483 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe0480a51e0 0x7fe0480a5610 secure :-1 s=READY pgs=920 cs=0 l=1 rx=0x7fe048124620 tx=0x7fe038010950).stop 2019-12-05T05:28:14.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:14.096+0000 7fe04f581700 1 -- 172.21.15.42:0/3142650483 shutdown_connections 2019-12-05T05:28:14.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:14.096+0000 7fe04f581700 1 --2- 172.21.15.42:0/3142650483 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe02803aca0 0x7fe02803d1b0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:28:14.103 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:14.096+0000 7fe04f581700 1 --2- 172.21.15.42:0/3142650483 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe0480a51e0 0x7fe0480a5610 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:28:14.103 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:14.096+0000 7fe04f581700 1 --2- 172.21.15.42:0/3142650483 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe048123c30 0x7fe048079f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:28:14.103 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:14.096+0000 7fe04f581700 1 -- 172.21.15.42:0/3142650483 shutdown_connections 2019-12-05T05:28:14.103 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:14.096+0000 7fe04f581700 1 -- 172.21.15.42:0/3142650483 wait complete. 2019-12-05T05:28:14.103 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:14.096+0000 7fe04f581700 1 -- 172.21.15.42:0/3142650483 >> 172.21.15.42:0/3142650483 conn(0x7fe0480aabe0 msgr2=0x7fe048059d50 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:28:14.111 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 114, "feature_flags": { "enable_multiple": false, "ever_enabled_multiple": false }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2019-12-05T05:03:35.604738+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 114, "ever_allowed_features": 2, "explicitly_allowed_features": 2, "failed": [], "flags": 18, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_6666": { "addr": "172.21.15.42:6835/2544447110", "addrs": { "addrvec": [ { "addr": "172.21.15.42:6834", "nonce": 2544447110, "type": "v2" }, { "addr": "172.21.15.42:6835", "nonce": 2544447110, "type": "v1" } ] }, "export_targets": [], "features": 4540138292836696063, "flags": 0, "gid": 6666, "incarnation": 114, "name": "a", "rank": 0, "state": "up:replay", "state_seq": 2 } }, "last_failure": 0, "last_failure_osd_epoch": 66, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 2, "min_compat_client": "0 (unknown)", "modified": "2019-12-05T05:28:13.632148+0000", "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6666 } } } ], "standbys": [] } 2019-12-05T05:28:16.112 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:28:16.208 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:16.192+0000 7fa6d49c2700 1 Processor -- start 2019-12-05T05:28:16.208 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:16.196+0000 7fa6d49c2700 1 -- start start 2019-12-05T05:28:16.209 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:16.196+0000 7fa6d49c2700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa6d00a7740 0x7fa6d008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:28:16.209 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:16.196+0000 7fa6d49c2700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6d00a0710 con 0x7fa6d009fae0 2019-12-05T05:28:16.209 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:16.196+0000 7fa6d49c2700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6d00a9ba0 con 0x7fa6d007e720 2019-12-05T05:28:16.209 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:16.196+0000 7fa6d49c2700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6d00a0b00 con 0x7fa6d00a7740 2019-12-05T05:28:16.210 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:16.196+0000 7fa6ce59c700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fa6d007e720 0x7fa6d009f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:34448/0 (socket says 172.21.15.42:34448) 2019-12-05T05:28:16.210 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:16.196+0000 7fa6ce59c700 1 -- 172.21.15.42:0/1536457402 learned_addr learned my addr 172.21.15.42:0/1536457402 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:28:16.210 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:16.196+0000 7fa6ced9d700 1 --2- 172.21.15.42:0/1536457402 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa6d00a7740 0x7fa6d008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:28:16.210 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:16.196+0000 7fa6cdd9b700 1 -- 172.21.15.42:0/1536457402 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2555962954 0 0) 0x7fa6d00a9ba0 con 0x7fa6d007e720 2019-12-05T05:28:16.210 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:16.196+0000 7fa6cdd9b700 1 -- 172.21.15.42:0/1536457402 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6b80030b0 con 0x7fa6d007e720 2019-12-05T05:28:16.211 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:16.196+0000 7fa6cdd9b700 1 -- 172.21.15.42:0/1536457402 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 535307015 0 0) 0x7fa6d00a0710 con 0x7fa6d009fae0 2019-12-05T05:28:16.211 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:16.196+0000 7fa6cdd9b700 1 -- 172.21.15.42:0/1536457402 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6d00a9ba0 con 0x7fa6d009fae0 2019-12-05T05:28:16.211 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:16.196+0000 7fa6ced9d700 1 -- 172.21.15.42:0/1536457402 >> v1:172.21.15.42:6789/0 conn(0x7fa6d007e720 legacy=0x7fa6d009f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:28:16.211 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:16.196+0000 7fa6ced9d700 1 -- 172.21.15.42:0/1536457402 >> v1:172.21.15.40:6789/0 conn(0x7fa6d009fae0 legacy=0x7fa6d014efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:28:16.212 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:16.196+0000 7fa6ced9d700 1 -- 172.21.15.42:0/1536457402 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6d007eb50 con 0x7fa6d00a7740 2019-12-05T05:28:16.212 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:16.196+0000 7fa6ced9d700 1 --2- 172.21.15.42:0/1536457402 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa6d00a7740 0x7fa6d008df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fa6c00032b0 tx=0x7fa6c000ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:16.212 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:16.196+0000 7fa6ced9d700 1 --2- 172.21.15.42:0/1536457402 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa6d00a7740 0x7fa6d008df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fa6c00032b0 tx=0x7fa6c000ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:16.213 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:16.196+0000 7fa6ced9d700 1 --2- 172.21.15.42:0/1536457402 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa6d00a7740 0x7fa6d008df80 secure :-1 s=READY pgs=188 cs=0 l=1 rx=0x7fa6c00032b0 tx=0x7fa6c000ff70).ready entity=mon.2 client_cookie=285c62eff1561613 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:28:16.213 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:16.196+0000 7fa6ced9d700 1 --2- 172.21.15.42:0/1536457402 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa6d00a7740 0x7fa6d008df80 secure :-1 s=THROTTLE_DONE pgs=188 cs=0 l=1 rx=0x7fa6c00032b0 tx=0x7fa6c000ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:16.213 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:16.196+0000 7fa6cdd9b700 1 -- 172.21.15.42:0/1536457402 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa6c0011a10 con 0x7fa6d00a7740 2019-12-05T05:28:16.213 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:16.196+0000 7fa6ced9d700 1 --2- 172.21.15.42:0/1536457402 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa6d00a7740 0x7fa6d008df80 secure :-1 s=THROTTLE_DONE pgs=188 cs=0 l=1 rx=0x7fa6c00032b0 tx=0x7fa6c000ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:16.214 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:16.196+0000 7fa6cdd9b700 1 -- 172.21.15.42:0/1536457402 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6c0011630 con 0x7fa6d00a7740 2019-12-05T05:28:16.214 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:16.196+0000 7fa6ced9d700 1 --2- 172.21.15.42:0/1536457402 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa6d00a7740 0x7fa6d008df80 secure :-1 s=THROTTLE_DONE pgs=188 cs=0 l=1 rx=0x7fa6c00032b0 tx=0x7fa6c000ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:16.214 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:16.196+0000 7fa6cdd9b700 1 -- 172.21.15.42:0/1536457402 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa6c000a160 con 0x7fa6d00a7740 2019-12-05T05:28:16.214 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:16.196+0000 7fa6d49c2700 1 -- 172.21.15.42:0/1536457402 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa6d00a7740 msgr2=0x7fa6d008df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:28:16.214 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:16.196+0000 7fa6d49c2700 1 --2- 172.21.15.42:0/1536457402 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa6d00a7740 0x7fa6d008df80 secure :-1 s=READY pgs=188 cs=0 l=1 rx=0x7fa6c00032b0 tx=0x7fa6c000ff70).stop 2019-12-05T05:28:16.215 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:16.200+0000 7fa6d49c2700 1 -- 172.21.15.42:0/1536457402 shutdown_connections 2019-12-05T05:28:16.215 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:16.200+0000 7fa6d49c2700 1 --2- 172.21.15.42:0/1536457402 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa6d00a7740 0x7fa6d008df80 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:28:16.215 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:16.200+0000 7fa6d49c2700 1 -- 172.21.15.42:0/1536457402 shutdown_connections 2019-12-05T05:28:16.215 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:16.200+0000 7fa6d49c2700 1 -- 172.21.15.42:0/1536457402 wait complete. 2019-12-05T05:28:16.216 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:16.200+0000 7fa6d49c2700 1 -- 172.21.15.42:0/1536457402 >> 172.21.15.42:0/1536457402 conn(0x7fa6d00aabe0 msgr2=0x7fa6d00a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:28:16.216 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:16.200+0000 7fa6d49c2700 1 Processor -- start 2019-12-05T05:28:16.216 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:16.200+0000 7fa6d49c2700 1 -- start start 2019-12-05T05:28:16.221 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:16.204+0000 7fa6d49c2700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa6d007cfb0 0x7fa6d014a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:28:16.222 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:16.204+0000 7fa6d49c2700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa6d014aa40 0x7fa6d008f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:28:16.222 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:16.204+0000 7fa6d49c2700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fa6d00a5b20 0x7fa6d01184a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:28:16.222 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:16.204+0000 7fa6d49c2700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6d014ae70 con 0x7fa6d014aa40 2019-12-05T05:28:16.222 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:16.204+0000 7fa6d49c2700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6d014a6a0 con 0x7fa6d00a5b20 2019-12-05T05:28:16.223 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:16.204+0000 7fa6d49c2700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6d008f090 con 0x7fa6d007cfb0 2019-12-05T05:28:16.223 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:16.204+0000 7fa6cf59e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fa6d00a5b20 0x7fa6d01184a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:28:16.223 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:16.204+0000 7fa6ced9d700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa6d007cfb0 0x7fa6d014a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:28:16.223 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:16.204+0000 7fa6cf59e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fa6d00a5b20 0x7fa6d01184a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:34420/0 (socket says 172.21.15.42:34420) 2019-12-05T05:28:16.224 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:16.204+0000 7fa6cf59e700 1 -- 172.21.15.42:0/1548551476 learned_addr learned my addr 172.21.15.42:0/1548551476 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:28:16.224 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:16.204+0000 7fa6ce59c700 1 --2- 172.21.15.42:0/1548551476 >> v2:172.21.15.40:3300/0 conn(0x7fa6d014aa40 0x7fa6d008f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:28:16.224 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:16.204+0000 7fa6cf59e700 1 -- 172.21.15.42:0/1548551476 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa6d007cfb0 msgr2=0x7fa6d014a070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:28:16.224 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:16.204+0000 7fa6cf59e700 1 --2- 172.21.15.42:0/1548551476 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa6d007cfb0 0x7fa6d014a070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:28:16.224 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:16.204+0000 7fa6cf59e700 1 -- 172.21.15.42:0/1548551476 >> v2:172.21.15.40:3300/0 conn(0x7fa6d014aa40 msgr2=0x7fa6d008f5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:28:16.225 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:16.204+0000 7fa6cf59e700 1 --2- 172.21.15.42:0/1548551476 >> v2:172.21.15.40:3300/0 conn(0x7fa6d014aa40 0x7fa6d008f5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:28:16.225 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:16.204+0000 7fa6cf59e700 1 -- 172.21.15.42:0/1548551476 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6d00a5f50 con 0x7fa6d00a5b20 2019-12-05T05:28:16.225 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:16.204+0000 7fa6cf59e700 1 --2- 172.21.15.42:0/1548551476 >> v2:172.21.15.42:3300/0 conn(0x7fa6d00a5b20 0x7fa6d01184a0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fa6d008f250 tx=0x7fa6c8010820).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:16.225 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:16.204+0000 7fa6cf59e700 1 --2- 172.21.15.42:0/1548551476 >> v2:172.21.15.42:3300/0 conn(0x7fa6d00a5b20 0x7fa6d01184a0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fa6d008f250 tx=0x7fa6c8010820).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:16.226 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:16.204+0000 7fa6cf59e700 1 --2- 172.21.15.42:0/1548551476 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa6d00a5b20 0x7fa6d01184a0 secure :-1 s=READY pgs=922 cs=0 l=1 rx=0x7fa6d008f250 tx=0x7fa6c8010820).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:28:16.226 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:16.204+0000 7fa6cf59e700 1 --2- 172.21.15.42:0/1548551476 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa6d00a5b20 0x7fa6d01184a0 secure :-1 s=THROTTLE_DONE pgs=922 cs=0 l=1 rx=0x7fa6d008f250 tx=0x7fa6c8010820).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:16.226 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:16.204+0000 7fa6bf7fe700 1 -- 172.21.15.42:0/1548551476 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa6c8006a40 con 0x7fa6d00a5b20 2019-12-05T05:28:16.226 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:16.204+0000 7fa6cf59e700 1 --2- 172.21.15.42:0/1548551476 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa6d00a5b20 0x7fa6d01184a0 secure :-1 s=THROTTLE_DONE pgs=922 cs=0 l=1 rx=0x7fa6d008f250 tx=0x7fa6c8010820).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:16.226 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:16.204+0000 7fa6cf59e700 1 --2- 172.21.15.42:0/1548551476 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa6d00a5b20 0x7fa6d01184a0 secure :-1 s=THROTTLE_DONE pgs=922 cs=0 l=1 rx=0x7fa6d008f250 tx=0x7fa6c8010820).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:16.227 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:16.204+0000 7fa6bf7fe700 1 -- 172.21.15.42:0/1548551476 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6c8006660 con 0x7fa6d00a5b20 2019-12-05T05:28:16.227 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:16.204+0000 7fa6d49c2700 1 -- 172.21.15.42:0/1548551476 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6d01189f0 con 0x7fa6d00a5b20 2019-12-05T05:28:16.227 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:16.204+0000 7fa6bf7fe700 1 -- 172.21.15.42:0/1548551476 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa6c8007640 con 0x7fa6d00a5b20 2019-12-05T05:28:16.227 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:16.204+0000 7fa6d49c2700 1 -- 172.21.15.42:0/1548551476 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6d0118f70 con 0x7fa6d00a5b20 2019-12-05T05:28:16.228 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:16.204+0000 7fa6cf59e700 1 --2- 172.21.15.42:0/1548551476 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa6d00a5b20 0x7fa6d01184a0 secure :-1 s=THROTTLE_DONE pgs=922 cs=0 l=1 rx=0x7fa6d008f250 tx=0x7fa6c8010820).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:16.228 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:16.208+0000 7fa6bf7fe700 1 -- 172.21.15.42:0/1548551476 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fa6c8007960 con 0x7fa6d00a5b20 2019-12-05T05:28:16.229 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:16.208+0000 7fa6cf59e700 1 --2- 172.21.15.42:0/1548551476 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa6d00a5b20 0x7fa6d01184a0 secure :-1 s=THROTTLE_DONE pgs=922 cs=0 l=1 rx=0x7fa6d008f250 tx=0x7fa6c8010820).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:16.229 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:16.208+0000 7fa6bf7fe700 1 --2- 172.21.15.42:0/1548551476 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa6b403ad20 0x7fa6b403d230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:28:16.229 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:16.208+0000 7fa6bf7fe700 1 -- 172.21.15.42:0/1548551476 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 7493+0+0 (secure 0 0 0) 0x7fa6c804b0d0 con 0x7fa6d00a5b20 2019-12-05T05:28:16.229 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:16.208+0000 7fa6ced9d700 1 --2- 172.21.15.42:0/1548551476 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa6b403ad20 0x7fa6b403d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:28:16.230 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:16.208+0000 7fa6ced9d700 1 --2- 172.21.15.42:0/1548551476 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa6b403ad20 0x7fa6b403d230 crc :-1 s=READY pgs=402 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:28:16.230 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:16.208+0000 7fa6d49c2700 1 -- 172.21.15.42:0/1548551476 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6d005d110 con 0x7fa6d00a5b20 2019-12-05T05:28:16.231 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:16.216+0000 7fa6cf59e700 1 --2- 172.21.15.42:0/1548551476 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa6d00a5b20 0x7fa6d01184a0 secure :-1 s=THROTTLE_DONE pgs=922 cs=0 l=1 rx=0x7fa6d008f250 tx=0x7fa6c8010820).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:16.231 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:16.216+0000 7fa6bf7fe700 1 -- 172.21.15.42:0/1548551476 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fa6c8032130 con 0x7fa6d00a5b20 2019-12-05T05:28:16.391 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:16.384+0000 7fa6d49c2700 1 -- 172.21.15.42:0/1548551476 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa6d005b8c0 con 0x7fa6d00a5b20 2019-12-05T05:28:16.392 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:16.388+0000 7fa6cf59e700 1 --2- 172.21.15.42:0/1548551476 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa6d00a5b20 0x7fa6d01184a0 secure :-1 s=THROTTLE_DONE pgs=922 cs=0 l=1 rx=0x7fa6d008f250 tx=0x7fa6c8010820).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:16.392 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:16.388+0000 7fa6bf7fe700 1 -- 172.21.15.42:0/1548551476 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 115 v115) v1 ==== 95+0+1778 (secure 0 0 0) 0x7fa6c8032390 con 0x7fa6d00a5b20 2019-12-05T05:28:16.392 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 115 2019-12-05T05:28:16.393 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:28:16.393 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":115,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":115,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:28:16.116906+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6666},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6666":{"gid":6666,"name":"a","rank":0,"incarnation":114,"state":"up:reconnect","state_seq":15,"addr":"172.21.15.42:6835/2544447110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2544447110},{"type":"v1","addr":"172.21.15.42:6835","nonce":2544447110}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:28:16.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:16.388+0000 7fa6d49c2700 1 -- 172.21.15.42:0/1548551476 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa6b403ad20 msgr2=0x7fa6b403d230 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:28:16.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:16.388+0000 7fa6d49c2700 1 --2- 172.21.15.42:0/1548551476 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa6b403ad20 0x7fa6b403d230 crc :-1 s=READY pgs=402 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:28:16.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:16.388+0000 7fa6d49c2700 1 -- 172.21.15.42:0/1548551476 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa6d00a5b20 msgr2=0x7fa6d01184a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:28:16.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:16.388+0000 7fa6d49c2700 1 --2- 172.21.15.42:0/1548551476 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa6d00a5b20 0x7fa6d01184a0 secure :-1 s=READY pgs=922 cs=0 l=1 rx=0x7fa6d008f250 tx=0x7fa6c8010820).stop 2019-12-05T05:28:16.404 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:16.388+0000 7fa6d49c2700 1 -- 172.21.15.42:0/1548551476 shutdown_connections 2019-12-05T05:28:16.404 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:16.388+0000 7fa6d49c2700 1 --2- 172.21.15.42:0/1548551476 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa6b403ad20 0x7fa6b403d230 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:28:16.404 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:16.388+0000 7fa6d49c2700 1 --2- 172.21.15.42:0/1548551476 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa6d00a5b20 0x7fa6d01184a0 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:28:16.404 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:16.388+0000 7fa6d49c2700 1 --2- 172.21.15.42:0/1548551476 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa6d007cfb0 0x7fa6d014a070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:28:16.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:16.388+0000 7fa6d49c2700 1 --2- 172.21.15.42:0/1548551476 >> v2:172.21.15.40:3300/0 conn(0x7fa6d014aa40 0x7fa6d008f5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:28:16.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:16.388+0000 7fa6d49c2700 1 -- 172.21.15.42:0/1548551476 shutdown_connections 2019-12-05T05:28:16.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:16.388+0000 7fa6d49c2700 1 -- 172.21.15.42:0/1548551476 wait complete. 2019-12-05T05:28:16.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:16.388+0000 7fa6d49c2700 1 -- 172.21.15.42:0/1548551476 >> 172.21.15.42:0/1548551476 conn(0x7fa6d00a7740 msgr2=0x7fa6d0059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:28:16.407 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2019-12-05T05:28:18.408 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:28:18.531 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:18.516+0000 7fd403e24700 1 Processor -- start 2019-12-05T05:28:18.531 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:18.516+0000 7fd403e24700 1 -- start start 2019-12-05T05:28:18.531 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:18.516+0000 7fd403e24700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd3fc09fae0 0x7fd3fc14fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:28:18.531 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:18.516+0000 7fd403e24700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3fc0a0710 con 0x7fd3fc0a7740 2019-12-05T05:28:18.532 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:18.516+0000 7fd403e24700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3fc07ef20 con 0x7fd3fc08df80 2019-12-05T05:28:18.532 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:18.516+0000 7fd403e24700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3fc0a0b00 con 0x7fd3fc09fae0 2019-12-05T05:28:18.532 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:18.516+0000 7fd401bc0700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fd3fc08df80 0x7fd3fc09f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:34464/0 (socket says 172.21.15.42:34464) 2019-12-05T05:28:18.532 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:18.516+0000 7fd401bc0700 1 -- 172.21.15.42:0/4219751351 learned_addr learned my addr 172.21.15.42:0/4219751351 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:28:18.532 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:18.516+0000 7fd402bc2700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd3fc09fae0 0x7fd3fc14fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:28:18.532 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:18.516+0000 7fd402bc2700 1 -- 172.21.15.42:0/4219751351 >> v1:172.21.15.42:6789/0 conn(0x7fd3fc08df80 legacy=0x7fd3fc09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:28:18.533 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:18.516+0000 7fd402bc2700 1 -- 172.21.15.42:0/4219751351 >> v1:172.21.15.40:6789/0 conn(0x7fd3fc0a7740 legacy=0x7fd3fc07e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:28:18.533 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:18.516+0000 7fd4013bf700 1 -- 172.21.15.42:0/4219751351 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1689489984 0 0) 0x7fd3fc0a0710 con 0x7fd3fc0a7740 2019-12-05T05:28:18.533 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:18.516+0000 7fd402bc2700 1 -- 172.21.15.42:0/4219751351 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3fc0a9ba0 con 0x7fd3fc09fae0 2019-12-05T05:28:18.533 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:18.516+0000 7fd402bc2700 1 --2- 172.21.15.42:0/4219751351 >> v2:172.21.15.42:3300/0 conn(0x7fd3fc09fae0 0x7fd3fc14fff0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fd3f4008b10 tx=0x7fd3f8001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:18.533 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:18.516+0000 7fd402bc2700 1 --2- 172.21.15.42:0/4219751351 >> v2:172.21.15.42:3300/0 conn(0x7fd3fc09fae0 0x7fd3fc14fff0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fd3f4008b10 tx=0x7fd3f8001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:18.533 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:18.516+0000 7fd402bc2700 1 --2- 172.21.15.42:0/4219751351 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd3fc09fae0 0x7fd3fc14fff0 secure :-1 s=READY pgs=924 cs=0 l=1 rx=0x7fd3f4008b10 tx=0x7fd3f8001150).ready entity=mon.1 client_cookie=2fc2bed6f5f3e370 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:28:18.534 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:18.516+0000 7fd402bc2700 1 --2- 172.21.15.42:0/4219751351 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd3fc09fae0 0x7fd3fc14fff0 secure :-1 s=THROTTLE_DONE pgs=924 cs=0 l=1 rx=0x7fd3f4008b10 tx=0x7fd3f8001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:18.534 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:18.516+0000 7fd4013bf700 1 -- 172.21.15.42:0/4219751351 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd3f4010870 con 0x7fd3fc09fae0 2019-12-05T05:28:18.534 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:18.516+0000 7fd402bc2700 1 --2- 172.21.15.42:0/4219751351 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd3fc09fae0 0x7fd3fc14fff0 secure :-1 s=THROTTLE_DONE pgs=924 cs=0 l=1 rx=0x7fd3f4008b10 tx=0x7fd3f8001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:18.534 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:18.516+0000 7fd402bc2700 1 --2- 172.21.15.42:0/4219751351 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd3fc09fae0 0x7fd3fc14fff0 secure :-1 s=THROTTLE_DONE pgs=924 cs=0 l=1 rx=0x7fd3f4008b10 tx=0x7fd3f8001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:18.534 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:18.516+0000 7fd4013bf700 1 -- 172.21.15.42:0/4219751351 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3f4008b50 con 0x7fd3fc09fae0 2019-12-05T05:28:18.534 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:18.520+0000 7fd4013bf700 1 -- 172.21.15.42:0/4219751351 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd3f40110c0 con 0x7fd3fc09fae0 2019-12-05T05:28:18.535 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:18.520+0000 7fd403e24700 1 -- 172.21.15.42:0/4219751351 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd3fc09fae0 msgr2=0x7fd3fc14fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:28:18.535 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:18.520+0000 7fd403e24700 1 --2- 172.21.15.42:0/4219751351 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd3fc09fae0 0x7fd3fc14fff0 secure :-1 s=READY pgs=924 cs=0 l=1 rx=0x7fd3f4008b10 tx=0x7fd3f8001150).stop 2019-12-05T05:28:18.535 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:18.520+0000 7fd403e24700 1 -- 172.21.15.42:0/4219751351 shutdown_connections 2019-12-05T05:28:18.535 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:18.520+0000 7fd403e24700 1 --2- 172.21.15.42:0/4219751351 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd3fc09fae0 0x7fd3fc14fff0 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:28:18.535 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:18.520+0000 7fd403e24700 1 -- 172.21.15.42:0/4219751351 shutdown_connections 2019-12-05T05:28:18.535 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:18.520+0000 7fd403e24700 1 -- 172.21.15.42:0/4219751351 wait complete. 2019-12-05T05:28:18.535 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:18.520+0000 7fd403e24700 1 -- 172.21.15.42:0/4219751351 >> 172.21.15.42:0/4219751351 conn(0x7fd3fc0aabe0 msgr2=0x7fd3fc0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:28:18.536 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:18.520+0000 7fd403e24700 1 Processor -- start 2019-12-05T05:28:18.536 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:18.520+0000 7fd403e24700 1 -- start start 2019-12-05T05:28:18.536 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:18.520+0000 7fd403e24700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd3fc122d00 0x7fd3fc11db20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:28:18.536 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:18.520+0000 7fd403e24700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd3fc12e210 0x7fd3fc11bd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:28:18.536 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:18.520+0000 7fd403e24700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3fc1221a0 con 0x7fd3fc11a3c0 2019-12-05T05:28:18.536 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:18.520+0000 7fd403e24700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3fc122520 con 0x7fd3fc122d00 2019-12-05T05:28:18.536 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:18.520+0000 7fd403e24700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3fc1227c0 con 0x7fd3fc12e210 2019-12-05T05:28:18.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:18.524+0000 7fd402bc2700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fd3fc11a3c0 0x7fd3fc11a7f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:34474/0 (socket says 172.21.15.42:34474) 2019-12-05T05:28:18.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:18.524+0000 7fd402bc2700 1 -- 172.21.15.42:0/2288351268 learned_addr learned my addr 172.21.15.42:0/2288351268 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:28:18.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:18.524+0000 7fd401bc0700 1 --2- 172.21.15.42:0/2288351268 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd3fc12e210 0x7fd3fc11bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:28:18.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:18.524+0000 7fd4023c1700 1 --2- 172.21.15.42:0/2288351268 >> v2:172.21.15.40:3300/0 conn(0x7fd3fc122d00 0x7fd3fc11db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:28:18.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:18.524+0000 7fd3f2ffd700 1 -- 172.21.15.42:0/2288351268 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1064595223 0 0) 0x7fd3fc1221a0 con 0x7fd3fc11a3c0 2019-12-05T05:28:18.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:18.524+0000 7fd3f2ffd700 1 -- 172.21.15.42:0/2288351268 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3e40030b0 con 0x7fd3fc11a3c0 2019-12-05T05:28:18.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:18.524+0000 7fd402bc2700 1 -- 172.21.15.42:0/2288351268 >> v1:172.21.15.42:6789/0 conn(0x7fd3fc11a3c0 legacy=0x7fd3fc11a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2019-12-05T05:28:18.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:18.524+0000 7fd402bc2700 1 -- 172.21.15.42:0/2288351268 >> v1:172.21.15.42:6789/0 conn(0x7fd3fc11a3c0 legacy=0x7fd3fc11a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2019-12-05T05:28:18.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:18.524+0000 7fd402bc2700 1 --1- 172.21.15.42:0/2288351268 >> v1:172.21.15.42:6789/0 conn(0x7fd3fc11a3c0 0x7fd3fc11a7f0 :-1 s=OPENED pgs=925 cs=1 l=1).handle_message read tag failed 2019-12-05T05:28:18.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:18.524+0000 7fd402bc2700 1 --1- 172.21.15.42:0/2288351268 >> v1:172.21.15.42:6789/0 conn(0x7fd3fc11a3c0 0x7fd3fc11a7f0 :-1 s=OPENED pgs=925 cs=1 l=1).fault on lossy channel, failing 2019-12-05T05:28:18.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:18.524+0000 7fd4023c1700 1 -- 172.21.15.42:0/2288351268 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd3fc12e210 msgr2=0x7fd3fc11bd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:28:18.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:18.524+0000 7fd4023c1700 1 --2- 172.21.15.42:0/2288351268 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd3fc12e210 0x7fd3fc11bd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:28:18.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:18.524+0000 7fd4023c1700 1 -- 172.21.15.42:0/2288351268 >> v1:172.21.15.42:6789/0 conn(0x7fd3fc11a3c0 legacy=0x7fd3fc11a7f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2019-12-05T05:28:18.539 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:18.524+0000 7fd4023c1700 1 -- 172.21.15.42:0/2288351268 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3fc11c2d0 con 0x7fd3fc122d00 2019-12-05T05:28:18.539 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:18.524+0000 7fd401bc0700 1 --2- 172.21.15.42:0/2288351268 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd3fc12e210 0x7fd3fc11bd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:28:18.539 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:18.524+0000 7fd4023c1700 1 --2- 172.21.15.42:0/2288351268 >> v2:172.21.15.40:3300/0 conn(0x7fd3fc122d00 0x7fd3fc11db20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fd3fc1226e0 tx=0x7fd3f800c320).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:18.539 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:18.524+0000 7fd4023c1700 1 --2- 172.21.15.42:0/2288351268 >> v2:172.21.15.40:3300/0 conn(0x7fd3fc122d00 0x7fd3fc11db20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fd3fc1226e0 tx=0x7fd3f800c320).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:18.539 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:18.524+0000 7fd4023c1700 1 --2- 172.21.15.42:0/2288351268 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd3fc122d00 0x7fd3fc11db20 secure :-1 s=READY pgs=722 cs=0 l=1 rx=0x7fd3fc1226e0 tx=0x7fd3f800c320).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:28:18.539 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:18.524+0000 7fd4023c1700 1 --2- 172.21.15.42:0/2288351268 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd3fc122d00 0x7fd3fc11db20 secure :-1 s=THROTTLE_DONE pgs=722 cs=0 l=1 rx=0x7fd3fc1226e0 tx=0x7fd3f800c320).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:18.539 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:18.524+0000 7fd3f2ffd700 1 -- 172.21.15.42:0/2288351268 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd3f800ca80 con 0x7fd3fc122d00 2019-12-05T05:28:18.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:18.524+0000 7fd4023c1700 1 --2- 172.21.15.42:0/2288351268 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd3fc122d00 0x7fd3fc11db20 secure :-1 s=THROTTLE_DONE pgs=722 cs=0 l=1 rx=0x7fd3fc1226e0 tx=0x7fd3f800c320).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:18.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:18.524+0000 7fd3f2ffd700 1 -- 172.21.15.42:0/2288351268 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3f800c6a0 con 0x7fd3fc122d00 2019-12-05T05:28:18.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:18.524+0000 7fd403e24700 1 -- 172.21.15.42:0/2288351268 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3fc11aff0 con 0x7fd3fc122d00 2019-12-05T05:28:18.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:18.524+0000 7fd4023c1700 1 --2- 172.21.15.42:0/2288351268 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd3fc122d00 0x7fd3fc11db20 secure :-1 s=THROTTLE_DONE pgs=722 cs=0 l=1 rx=0x7fd3fc1226e0 tx=0x7fd3f800c320).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:18.541 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:18.524+0000 7fd3f2ffd700 1 -- 172.21.15.42:0/2288351268 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd3f80110e0 con 0x7fd3fc122d00 2019-12-05T05:28:18.541 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:18.524+0000 7fd403e24700 1 -- 172.21.15.42:0/2288351268 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3fc12e640 con 0x7fd3fc122d00 2019-12-05T05:28:18.541 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:18.524+0000 7fd4023c1700 1 --2- 172.21.15.42:0/2288351268 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd3fc122d00 0x7fd3fc11db20 secure :-1 s=THROTTLE_DONE pgs=722 cs=0 l=1 rx=0x7fd3fc1226e0 tx=0x7fd3f800c320).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:18.542 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:18.528+0000 7fd3f2ffd700 1 -- 172.21.15.42:0/2288351268 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fd3f80113c0 con 0x7fd3fc122d00 2019-12-05T05:28:18.543 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:18.528+0000 7fd4023c1700 1 --2- 172.21.15.42:0/2288351268 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd3fc122d00 0x7fd3fc11db20 secure :-1 s=THROTTLE_DONE pgs=722 cs=0 l=1 rx=0x7fd3fc1226e0 tx=0x7fd3f800c320).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:18.543 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:18.528+0000 7fd3f2ffd700 1 --2- 172.21.15.42:0/2288351268 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd3e403b230 0x7fd3e403d740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:28:18.543 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:18.528+0000 7fd3f2ffd700 1 -- 172.21.15.42:0/2288351268 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 7493+0+0 (secure 0 0 0) 0x7fd3f804b1b0 con 0x7fd3fc122d00 2019-12-05T05:28:18.543 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:18.528+0000 7fd401bc0700 1 --2- 172.21.15.42:0/2288351268 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd3e403b230 0x7fd3e403d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:28:18.544 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:18.528+0000 7fd401bc0700 1 --2- 172.21.15.42:0/2288351268 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd3e403b230 0x7fd3e403d740 crc :-1 s=READY pgs=403 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:28:18.544 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:18.532+0000 7fd403e24700 1 -- 172.21.15.42:0/2288351268 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3fc05d110 con 0x7fd3fc122d00 2019-12-05T05:28:18.547 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:18.540+0000 7fd4023c1700 1 --2- 172.21.15.42:0/2288351268 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd3fc122d00 0x7fd3fc11db20 secure :-1 s=THROTTLE_DONE pgs=722 cs=0 l=1 rx=0x7fd3fc1226e0 tx=0x7fd3f800c320).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:18.547 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:18.540+0000 7fd3f2ffd700 1 -- 172.21.15.42:0/2288351268 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fd3f80321f0 con 0x7fd3fc122d00 2019-12-05T05:28:18.724 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:18.720+0000 7fd403e24700 1 -- 172.21.15.42:0/2288351268 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd3fc05b8c0 con 0x7fd3fc122d00 2019-12-05T05:28:18.725 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:18.720+0000 7fd4023c1700 1 --2- 172.21.15.42:0/2288351268 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd3fc122d00 0x7fd3fc11db20 secure :-1 s=THROTTLE_DONE pgs=722 cs=0 l=1 rx=0x7fd3fc1226e0 tx=0x7fd3f800c320).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:18.725 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:18.720+0000 7fd3f2ffd700 1 -- 172.21.15.42:0/2288351268 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) v1 ==== 95+0+1775 (secure 0 0 0) 0x7fd3f8032450 con 0x7fd3fc122d00 2019-12-05T05:28:18.727 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 117 2019-12-05T05:28:18.727 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:28:18.727 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":117,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":117,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:28:18.130626+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6666},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6666":{"gid":6666,"name":"a","rank":0,"incarnation":114,"state":"up:active","state_seq":17,"addr":"172.21.15.42:6835/2544447110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2544447110},{"type":"v1","addr":"172.21.15.42:6835","nonce":2544447110}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:28:18.730 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:18.724+0000 7fd403e24700 1 -- 172.21.15.42:0/2288351268 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd3e403b230 msgr2=0x7fd3e403d740 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:28:18.730 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:18.724+0000 7fd403e24700 1 --2- 172.21.15.42:0/2288351268 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd3e403b230 0x7fd3e403d740 crc :-1 s=READY pgs=403 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:28:18.730 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:18.724+0000 7fd403e24700 1 -- 172.21.15.42:0/2288351268 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd3fc122d00 msgr2=0x7fd3fc11db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:28:18.730 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:18.724+0000 7fd403e24700 1 --2- 172.21.15.42:0/2288351268 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd3fc122d00 0x7fd3fc11db20 secure :-1 s=READY pgs=722 cs=0 l=1 rx=0x7fd3fc1226e0 tx=0x7fd3f800c320).stop 2019-12-05T05:28:18.731 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:18.724+0000 7fd403e24700 1 -- 172.21.15.42:0/2288351268 shutdown_connections 2019-12-05T05:28:18.731 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:18.724+0000 7fd403e24700 1 --2- 172.21.15.42:0/2288351268 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd3e403b230 0x7fd3e403d740 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:28:18.731 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:18.724+0000 7fd403e24700 1 --2- 172.21.15.42:0/2288351268 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd3fc122d00 0x7fd3fc11db20 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:28:18.731 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:18.724+0000 7fd403e24700 1 --2- 172.21.15.42:0/2288351268 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd3fc12e210 0x7fd3fc11bd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:28:18.731 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:18.724+0000 7fd403e24700 1 -- 172.21.15.42:0/2288351268 shutdown_connections 2019-12-05T05:28:18.731 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:18.724+0000 7fd403e24700 1 -- 172.21.15.42:0/2288351268 wait complete. 2019-12-05T05:28:18.731 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:18.724+0000 7fd403e24700 1 -- 172.21.15.42:0/2288351268 >> 172.21.15.42:0/2288351268 conn(0x7fd3fc07e4f0 msgr2=0x7fd3fc08eb70 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:28:18.742 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a has gained rank=0, replacing gid=6546 2019-12-05T05:28:18.742 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 6 secs before reviving mds.b 2019-12-05T05:28:24.743 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.b 2019-12-05T05:28:24.744 INFO:tasks.ceph.mds.b:Restarting daemon 2019-12-05T05:28:24.744 INFO:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2019-12-05T05:28:24.756 INFO:tasks.ceph.mds.b:Started 2019-12-05T05:28:24.756 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:28:24.817 INFO:tasks.ceph.mds.b.smithi040.stdout:starting mds.b at 2019-12-05T05:28:24.863 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:24.852+0000 7f32ce9a6700 1 Processor -- start 2019-12-05T05:28:24.864 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:24.852+0000 7f32ce9a6700 1 -- start start 2019-12-05T05:28:24.864 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:24.852+0000 7f32ce9a6700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f32c808df80 0x7f32c809f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:28:24.864 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:24.852+0000 7f32ce9a6700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32c809fc60 con 0x7f32c809f830 2019-12-05T05:28:24.864 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:24.852+0000 7f32ce9a6700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32c80a0710 con 0x7f32c80a7740 2019-12-05T05:28:24.864 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:24.852+0000 7f32ce9a6700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32c80a0b00 con 0x7f32c808df80 2019-12-05T05:28:24.864 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:24.852+0000 7f32ccf43700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f32c80a7740 0x7f32c807e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:34500/0 (socket says 172.21.15.42:34500) 2019-12-05T05:28:24.865 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:24.852+0000 7f32ccf43700 1 -- 172.21.15.42:0/696902073 learned_addr learned my addr 172.21.15.42:0/696902073 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:28:24.865 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:24.852+0000 7f32c7fff700 1 --2- 172.21.15.42:0/696902073 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f32c808df80 0x7f32c809f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:28:24.865 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:24.852+0000 7f32c77fe700 1 -- 172.21.15.42:0/696902073 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3347013493 0 0) 0x7f32c80a0710 con 0x7f32c80a7740 2019-12-05T05:28:24.865 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:24.852+0000 7f32c77fe700 1 -- 172.21.15.42:0/696902073 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32a80030b0 con 0x7f32c80a7740 2019-12-05T05:28:24.866 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:24.852+0000 7f32c77fe700 1 -- 172.21.15.42:0/696902073 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1338289507 0 0) 0x7f32c809fc60 con 0x7f32c809f830 2019-12-05T05:28:24.866 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:24.852+0000 7f32c7fff700 1 -- 172.21.15.42:0/696902073 >> v1:172.21.15.42:6789/0 conn(0x7f32c80a7740 legacy=0x7f32c807e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:28:24.866 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:24.852+0000 7f32c7fff700 1 -- 172.21.15.42:0/696902073 >> v1:172.21.15.40:6789/0 conn(0x7f32c809f830 legacy=0x7f32c814efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:28:24.866 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:24.852+0000 7f32c7fff700 1 -- 172.21.15.42:0/696902073 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32c807ef20 con 0x7f32c808df80 2019-12-05T05:28:24.866 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:24.852+0000 7f32c7fff700 1 --2- 172.21.15.42:0/696902073 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f32c808df80 0x7f32c809f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f32b0002f30 tx=0x7f32b000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:24.866 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:24.852+0000 7f32c7fff700 1 --2- 172.21.15.42:0/696902073 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f32c808df80 0x7f32c809f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f32b0002f30 tx=0x7f32b000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:24.866 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:24.852+0000 7f32c7fff700 1 --2- 172.21.15.42:0/696902073 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f32c808df80 0x7f32c809f2e0 secure :-1 s=READY pgs=190 cs=0 l=1 rx=0x7f32b0002f30 tx=0x7f32b000fbf0).ready entity=mon.2 client_cookie=4ea66df1665fb395 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:28:24.867 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:24.852+0000 7f32c7fff700 1 --2- 172.21.15.42:0/696902073 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f32c808df80 0x7f32c809f2e0 secure :-1 s=THROTTLE_DONE pgs=190 cs=0 l=1 rx=0x7f32b0002f30 tx=0x7f32b000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:24.867 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:24.852+0000 7f32c77fe700 1 -- 172.21.15.42:0/696902073 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f32b00112b0 con 0x7f32c808df80 2019-12-05T05:28:24.878 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:24.852+0000 7f32c7fff700 1 --2- 172.21.15.42:0/696902073 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f32c808df80 0x7f32c809f2e0 secure :-1 s=THROTTLE_DONE pgs=190 cs=0 l=1 rx=0x7f32b0002f30 tx=0x7f32b000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:24.878 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:24.852+0000 7f32c77fe700 1 -- 172.21.15.42:0/696902073 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32b0008bc0 con 0x7f32c808df80 2019-12-05T05:28:24.878 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:24.852+0000 7f32ce9a6700 1 -- 172.21.15.42:0/696902073 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f32c808df80 msgr2=0x7f32c809f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:28:24.878 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:24.852+0000 7f32c7fff700 1 --2- 172.21.15.42:0/696902073 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f32c808df80 0x7f32c809f2e0 secure :-1 s=THROTTLE_DONE pgs=190 cs=0 l=1 rx=0x7f32b0002f30 tx=0x7f32b000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:24.878 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:24.852+0000 7f32c77fe700 1 -- 172.21.15.42:0/696902073 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f32b0011890 con 0x7f32c808df80 2019-12-05T05:28:24.878 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:24.852+0000 7f32ce9a6700 1 --2- 172.21.15.42:0/696902073 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f32c808df80 0x7f32c809f2e0 secure :-1 s=READY pgs=190 cs=0 l=1 rx=0x7f32b0002f30 tx=0x7f32b000fbf0).stop 2019-12-05T05:28:24.879 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:24.856+0000 7f32ce9a6700 1 -- 172.21.15.42:0/696902073 shutdown_connections 2019-12-05T05:28:24.879 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:24.856+0000 7f32ce9a6700 1 --2- 172.21.15.42:0/696902073 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f32c808df80 0x7f32c809f2e0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:28:24.879 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:24.856+0000 7f32ce9a6700 1 -- 172.21.15.42:0/696902073 shutdown_connections 2019-12-05T05:28:24.879 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:24.856+0000 7f32ce9a6700 1 -- 172.21.15.42:0/696902073 wait complete. 2019-12-05T05:28:24.879 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:24.856+0000 7f32ce9a6700 1 -- 172.21.15.42:0/696902073 >> 172.21.15.42:0/696902073 conn(0x7f32c80aabe0 msgr2=0x7f32c80a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:28:24.879 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:24.856+0000 7f32ce9a6700 1 Processor -- start 2019-12-05T05:28:24.880 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:24.856+0000 7f32ce9a6700 1 -- start start 2019-12-05T05:28:24.883 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:24.860+0000 7f32ce9a6700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f32c8142db0 0x7f32c80a1a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:28:24.883 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:24.860+0000 7f32ce9a6700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f32c80648a0 0x7f32c80a2d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:28:24.884 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:24.860+0000 7f32ce9a6700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f32c80a3290 0x7f32c80a59e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:28:24.884 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:24.860+0000 7f32ce9a6700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32c80a21c0 con 0x7f32c80a3290 2019-12-05T05:28:24.884 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:24.860+0000 7f32ce9a6700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32c80a2460 con 0x7f32c8142db0 2019-12-05T05:28:24.884 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:24.860+0000 7f32ce9a6700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32c807e1d0 con 0x7f32c80648a0 2019-12-05T05:28:24.893 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:24.860+0000 7f32ccf43700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f32c8142db0 0x7f32c80a1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:28:24.893 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:24.860+0000 7f32ccf43700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f32c8142db0 0x7f32c80a1a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:34468/0 (socket says 172.21.15.42:34468) 2019-12-05T05:28:24.893 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:24.860+0000 7f32ccf43700 1 -- 172.21.15.42:0/3547527552 learned_addr learned my addr 172.21.15.42:0/3547527552 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:28:24.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:24.860+0000 7f32c7fff700 1 --2- 172.21.15.42:0/3547527552 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f32c80648a0 0x7f32c80a2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:28:24.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:24.860+0000 7f32cd744700 1 --2- 172.21.15.42:0/3547527552 >> v2:172.21.15.40:3300/0 conn(0x7f32c80a3290 0x7f32c80a59e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:28:24.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:24.860+0000 7f32ccf43700 1 -- 172.21.15.42:0/3547527552 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f32c80648a0 msgr2=0x7f32c80a2d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:28:24.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:24.860+0000 7f32ccf43700 1 --2- 172.21.15.42:0/3547527552 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f32c80648a0 0x7f32c80a2d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:28:24.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:24.860+0000 7f32ccf43700 1 -- 172.21.15.42:0/3547527552 >> v2:172.21.15.40:3300/0 conn(0x7f32c80a3290 msgr2=0x7f32c80a59e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:28:24.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:24.860+0000 7f32ccf43700 1 --2- 172.21.15.42:0/3547527552 >> v2:172.21.15.40:3300/0 conn(0x7f32c80a3290 0x7f32c80a59e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:28:24.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:24.860+0000 7f32ccf43700 1 -- 172.21.15.42:0/3547527552 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32c8064cd0 con 0x7f32c8142db0 2019-12-05T05:28:24.895 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:24.860+0000 7f32cd744700 1 --2- 172.21.15.42:0/3547527552 >> v2:172.21.15.40:3300/0 conn(0x7f32c80a3290 0x7f32c80a59e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:28:24.895 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:24.860+0000 7f32ccf43700 1 --2- 172.21.15.42:0/3547527552 >> v2:172.21.15.42:3300/0 conn(0x7f32c8142db0 0x7f32c80a1a40 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f32b80043a0 tx=0x7f32b800c450).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:24.895 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:24.860+0000 7f32ccf43700 1 --2- 172.21.15.42:0/3547527552 >> v2:172.21.15.42:3300/0 conn(0x7f32c8142db0 0x7f32c80a1a40 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f32b80043a0 tx=0x7f32b800c450).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:24.895 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:24.860+0000 7f32ccf43700 1 --2- 172.21.15.42:0/3547527552 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f32c8142db0 0x7f32c80a1a40 secure :-1 s=READY pgs=929 cs=0 l=1 rx=0x7f32b80043a0 tx=0x7f32b800c450).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:28:24.895 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:24.860+0000 7f32ccf43700 1 --2- 172.21.15.42:0/3547527552 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f32c8142db0 0x7f32c80a1a40 secure :-1 s=THROTTLE_DONE pgs=929 cs=0 l=1 rx=0x7f32b80043a0 tx=0x7f32b800c450).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:24.895 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:24.860+0000 7f32c57fa700 1 -- 172.21.15.42:0/3547527552 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f32b800c6c0 con 0x7f32c8142db0 2019-12-05T05:28:24.895 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:24.860+0000 7f32ccf43700 1 --2- 172.21.15.42:0/3547527552 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f32c8142db0 0x7f32c80a1a40 secure :-1 s=THROTTLE_DONE pgs=929 cs=0 l=1 rx=0x7f32b80043a0 tx=0x7f32b800c450).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:24.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:24.860+0000 7f32ccf43700 1 --2- 172.21.15.42:0/3547527552 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f32c8142db0 0x7f32c80a1a40 secure :-1 s=THROTTLE_DONE pgs=929 cs=0 l=1 rx=0x7f32b80043a0 tx=0x7f32b800c450).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:24.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:24.860+0000 7f32c57fa700 1 -- 172.21.15.42:0/3547527552 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32b8010170 con 0x7f32c8142db0 2019-12-05T05:28:24.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:24.860+0000 7f32ce9a6700 1 -- 172.21.15.42:0/3547527552 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32c80a36c0 con 0x7f32c8142db0 2019-12-05T05:28:24.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:24.860+0000 7f32c57fa700 1 -- 172.21.15.42:0/3547527552 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f32b800cca0 con 0x7f32c8142db0 2019-12-05T05:28:24.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:24.860+0000 7f32ce9a6700 1 -- 172.21.15.42:0/3547527552 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32c80a5f30 con 0x7f32c8142db0 2019-12-05T05:28:24.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:24.860+0000 7f32ccf43700 1 --2- 172.21.15.42:0/3547527552 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f32c8142db0 0x7f32c80a1a40 secure :-1 s=THROTTLE_DONE pgs=929 cs=0 l=1 rx=0x7f32b80043a0 tx=0x7f32b800c450).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:24.897 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:24.864+0000 7f32c57fa700 1 -- 172.21.15.42:0/3547527552 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f32b8005460 con 0x7f32c8142db0 2019-12-05T05:28:24.897 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:24.864+0000 7f32ccf43700 1 --2- 172.21.15.42:0/3547527552 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f32c8142db0 0x7f32c80a1a40 secure :-1 s=THROTTLE_DONE pgs=929 cs=0 l=1 rx=0x7f32b80043a0 tx=0x7f32b800c450).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:24.897 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:24.864+0000 7f32c57fa700 1 --2- 172.21.15.42:0/3547527552 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f32a003ae30 0x7f32a003d340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:28:24.897 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:24.864+0000 7f32c57fa700 1 -- 172.21.15.42:0/3547527552 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 7493+0+0 (secure 0 0 0) 0x7f32b804aa40 con 0x7f32c8142db0 2019-12-05T05:28:24.897 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:24.864+0000 7f32c7fff700 1 --2- 172.21.15.42:0/3547527552 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f32a003ae30 0x7f32a003d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:28:24.897 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:24.864+0000 7f32c7fff700 1 --2- 172.21.15.42:0/3547527552 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f32a003ae30 0x7f32a003d340 crc :-1 s=READY pgs=404 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:28:24.897 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:24.868+0000 7f32ce9a6700 1 -- 172.21.15.42:0/3547527552 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32c805d110 con 0x7f32c8142db0 2019-12-05T05:28:24.898 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:24.872+0000 7f32ccf43700 1 --2- 172.21.15.42:0/3547527552 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f32c8142db0 0x7f32c80a1a40 secure :-1 s=THROTTLE_DONE pgs=929 cs=0 l=1 rx=0x7f32b80043a0 tx=0x7f32b800c450).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:24.898 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:24.872+0000 7f32c57fa700 1 -- 172.21.15.42:0/3547527552 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f32b8031ac0 con 0x7f32c8142db0 2019-12-05T05:28:25.044 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:25.040+0000 7f32ce9a6700 1 -- 172.21.15.42:0/3547527552 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f32c805b8c0 con 0x7f32c8142db0 2019-12-05T05:28:25.045 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:25.040+0000 7f32ccf43700 1 --2- 172.21.15.42:0/3547527552 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f32c8142db0 0x7f32c80a1a40 secure :-1 s=THROTTLE_DONE pgs=929 cs=0 l=1 rx=0x7f32b80043a0 tx=0x7f32b800c450).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:25.045 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:25.040+0000 7f32c57fa700 1 -- 172.21.15.42:0/3547527552 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) v1 ==== 95+0+1775 (secure 0 0 0) 0x7f32b8031d20 con 0x7f32c8142db0 2019-12-05T05:28:25.047 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 117 2019-12-05T05:28:25.047 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:28:25.047 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":117,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":117,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:28:18.130626+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6666},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6666":{"gid":6666,"name":"a","rank":0,"incarnation":114,"state":"up:active","state_seq":17,"addr":"172.21.15.42:6835/2544447110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2544447110},{"type":"v1","addr":"172.21.15.42:6835","nonce":2544447110}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:28:25.050 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:25.044+0000 7f32ce9a6700 1 -- 172.21.15.42:0/3547527552 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f32a003ae30 msgr2=0x7f32a003d340 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:28:25.050 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:25.044+0000 7f32ce9a6700 1 --2- 172.21.15.42:0/3547527552 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f32a003ae30 0x7f32a003d340 crc :-1 s=READY pgs=404 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:28:25.050 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:25.044+0000 7f32ce9a6700 1 -- 172.21.15.42:0/3547527552 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f32c8142db0 msgr2=0x7f32c80a1a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:28:25.050 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:25.044+0000 7f32ce9a6700 1 --2- 172.21.15.42:0/3547527552 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f32c8142db0 0x7f32c80a1a40 secure :-1 s=READY pgs=929 cs=0 l=1 rx=0x7f32b80043a0 tx=0x7f32b800c450).stop 2019-12-05T05:28:25.050 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:25.044+0000 7f32ce9a6700 1 -- 172.21.15.42:0/3547527552 shutdown_connections 2019-12-05T05:28:25.050 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:25.044+0000 7f32ce9a6700 1 --2- 172.21.15.42:0/3547527552 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f32a003ae30 0x7f32a003d340 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:28:25.051 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:25.044+0000 7f32ce9a6700 1 --2- 172.21.15.42:0/3547527552 >> v2:172.21.15.40:3300/0 conn(0x7f32c80a3290 0x7f32c80a59e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:28:25.051 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:25.044+0000 7f32ce9a6700 1 --2- 172.21.15.42:0/3547527552 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f32c8142db0 0x7f32c80a1a40 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:28:25.051 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:25.044+0000 7f32ce9a6700 1 --2- 172.21.15.42:0/3547527552 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f32c80648a0 0x7f32c80a2d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:28:25.051 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:25.044+0000 7f32ce9a6700 1 -- 172.21.15.42:0/3547527552 shutdown_connections 2019-12-05T05:28:25.051 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:25.044+0000 7f32ce9a6700 1 -- 172.21.15.42:0/3547527552 wait complete. 2019-12-05T05:28:25.051 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:25.044+0000 7f32ce9a6700 1 -- 172.21.15.42:0/3547527552 >> 172.21.15.42:0/3547527552 conn(0x7f32c807e4f0 msgr2=0x7f32c80a7f90 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:28:25.060 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2019-12-05T05:28:27.061 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:28:27.166 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:27.156+0000 7f9ae7ffa700 1 Processor -- start 2019-12-05T05:28:27.167 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:27.156+0000 7f9ae7ffa700 1 -- start start 2019-12-05T05:28:27.167 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:27.156+0000 7f9ae7ffa700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f9ae008df80 0x7f9ae009f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:28:27.167 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:27.156+0000 7f9ae7ffa700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f9ae009f830 0x7f9ae014efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:28:27.167 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:27.156+0000 7f9ae7ffa700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ae009fc60 con 0x7f9ae00a7740 2019-12-05T05:28:27.167 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:27.156+0000 7f9ae7ffa700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ae00a0b00 con 0x7f9ae008df80 2019-12-05T05:28:27.167 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:27.156+0000 7f9ae7ffa700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ae008e5b0 con 0x7f9ae009f830 2019-12-05T05:28:27.167 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:27.156+0000 7f9ae6597700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f9ae00a7740 0x7f9ae007e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:34530/0 (socket says 172.21.15.42:34530) 2019-12-05T05:28:27.168 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:27.156+0000 7f9ae6597700 1 -- 172.21.15.42:0/3341800493 learned_addr learned my addr 172.21.15.42:0/3341800493 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:28:27.168 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:27.156+0000 7f9ae6d98700 1 --2- 172.21.15.42:0/3341800493 >> v2:172.21.15.42:3300/0 conn(0x7f9ae009f830 0x7f9ae014efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:28:27.168 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:27.156+0000 7f9ae5d96700 1 --2- 172.21.15.42:0/3341800493 >> v2:172.21.15.40:3300/0 conn(0x7f9ae008df80 0x7f9ae009f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:28:27.168 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:27.156+0000 7f9ae5595700 1 -- 172.21.15.42:0/3341800493 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1652527004 0 0) 0x7f9ae009fc60 con 0x7f9ae00a7740 2019-12-05T05:28:27.168 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:27.156+0000 7f9ae5595700 1 -- 172.21.15.42:0/3341800493 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9acc0030b0 con 0x7f9ae00a7740 2019-12-05T05:28:27.168 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:27.156+0000 7f9ae6d98700 1 -- 172.21.15.42:0/3341800493 >> v2:172.21.15.40:3300/0 conn(0x7f9ae008df80 msgr2=0x7f9ae009f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:28:27.168 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:27.156+0000 7f9ae6d98700 1 --2- 172.21.15.42:0/3341800493 >> v2:172.21.15.40:3300/0 conn(0x7f9ae008df80 0x7f9ae009f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:28:27.169 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:27.156+0000 7f9ae6d98700 1 -- 172.21.15.42:0/3341800493 >> v1:172.21.15.42:6789/0 conn(0x7f9ae00a7740 legacy=0x7f9ae007e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:28:27.169 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:27.156+0000 7f9ae6d98700 1 -- 172.21.15.42:0/3341800493 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ae00a0710 con 0x7f9ae009f830 2019-12-05T05:28:27.169 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:27.156+0000 7f9ae6d98700 1 --2- 172.21.15.42:0/3341800493 >> v2:172.21.15.42:3300/0 conn(0x7f9ae009f830 0x7f9ae014efe0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f9ad80032b0 tx=0x7f9ad800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:27.169 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:27.156+0000 7f9ae6d98700 1 --2- 172.21.15.42:0/3341800493 >> v2:172.21.15.42:3300/0 conn(0x7f9ae009f830 0x7f9ae014efe0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f9ad80032b0 tx=0x7f9ad800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:27.169 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:27.156+0000 7f9ae6d98700 1 --2- 172.21.15.42:0/3341800493 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9ae009f830 0x7f9ae014efe0 secure :-1 s=READY pgs=931 cs=0 l=1 rx=0x7f9ad80032b0 tx=0x7f9ad800fff0).ready entity=mon.1 client_cookie=551bcbc85351df5b server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:28:27.169 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:27.156+0000 7f9ae6d98700 1 --2- 172.21.15.42:0/3341800493 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9ae009f830 0x7f9ae014efe0 secure :-1 s=THROTTLE_DONE pgs=931 cs=0 l=1 rx=0x7f9ad80032b0 tx=0x7f9ad800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:27.169 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:27.156+0000 7f9ae5595700 1 -- 172.21.15.42:0/3341800493 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9ad80108c0 con 0x7f9ae009f830 2019-12-05T05:28:27.170 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:27.156+0000 7f9ae6d98700 1 --2- 172.21.15.42:0/3341800493 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9ae009f830 0x7f9ae014efe0 secure :-1 s=THROTTLE_DONE pgs=931 cs=0 l=1 rx=0x7f9ad80032b0 tx=0x7f9ad800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:27.170 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:27.156+0000 7f9ae6d98700 1 --2- 172.21.15.42:0/3341800493 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9ae009f830 0x7f9ae014efe0 secure :-1 s=THROTTLE_DONE pgs=931 cs=0 l=1 rx=0x7f9ad80032b0 tx=0x7f9ad800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:27.170 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:27.156+0000 7f9ae5595700 1 -- 172.21.15.42:0/3341800493 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ad8008b50 con 0x7f9ae009f830 2019-12-05T05:28:27.170 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:27.156+0000 7f9ae5595700 1 -- 172.21.15.42:0/3341800493 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9ad80110c0 con 0x7f9ae009f830 2019-12-05T05:28:27.170 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:27.156+0000 7f9ae7ffa700 1 -- 172.21.15.42:0/3341800493 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9ae009f830 msgr2=0x7f9ae014efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:28:27.170 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:27.156+0000 7f9ae7ffa700 1 --2- 172.21.15.42:0/3341800493 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9ae009f830 0x7f9ae014efe0 secure :-1 s=READY pgs=931 cs=0 l=1 rx=0x7f9ad80032b0 tx=0x7f9ad800fff0).stop 2019-12-05T05:28:27.170 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:27.160+0000 7f9ae7ffa700 1 -- 172.21.15.42:0/3341800493 shutdown_connections 2019-12-05T05:28:27.171 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:27.160+0000 7f9ae7ffa700 1 --2- 172.21.15.42:0/3341800493 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9ae009f830 0x7f9ae014efe0 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:28:27.171 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:27.160+0000 7f9ae7ffa700 1 --2- 172.21.15.42:0/3341800493 >> v2:172.21.15.40:3300/0 conn(0x7f9ae008df80 0x7f9ae009f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:28:27.171 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:27.160+0000 7f9ae7ffa700 1 -- 172.21.15.42:0/3341800493 shutdown_connections 2019-12-05T05:28:27.171 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:27.160+0000 7f9ae7ffa700 1 -- 172.21.15.42:0/3341800493 wait complete. 2019-12-05T05:28:27.171 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:27.160+0000 7f9ae7ffa700 1 -- 172.21.15.42:0/3341800493 >> 172.21.15.42:0/3341800493 conn(0x7f9ae00aabe0 msgr2=0x7f9ae00a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:28:27.171 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:27.160+0000 7f9ae7ffa700 1 Processor -- start 2019-12-05T05:28:27.171 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:27.160+0000 7f9ae7ffa700 1 -- start start 2019-12-05T05:28:27.175 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:27.164+0000 7f9ae7ffa700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f9ae0116500 0x7f9ae007cf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:28:27.175 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:27.164+0000 7f9ae7ffa700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9ae007af60 0x7f9ae0133670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:28:27.175 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:27.164+0000 7f9ae7ffa700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f9ae007b390 0x7f9ae0133bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:28:27.175 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:27.164+0000 7f9ae7ffa700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ae00a7230 con 0x7f9ae007b390 2019-12-05T05:28:27.176 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:27.164+0000 7f9ae7ffa700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ae00a5290 con 0x7f9ae0116500 2019-12-05T05:28:27.176 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:27.164+0000 7f9ae7ffa700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ae00a5530 con 0x7f9ae007af60 2019-12-05T05:28:27.176 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:27.164+0000 7f9ae6597700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f9ae0116500 0x7f9ae007cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:28:27.176 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:27.164+0000 7f9ae6597700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f9ae0116500 0x7f9ae007cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:34500/0 (socket says 172.21.15.42:34500) 2019-12-05T05:28:27.176 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:27.164+0000 7f9ae6597700 1 -- 172.21.15.42:0/290126608 learned_addr learned my addr 172.21.15.42:0/290126608 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:28:27.176 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:27.164+0000 7f9ae6d98700 1 --2- 172.21.15.42:0/290126608 >> v2:172.21.15.40:3300/0 conn(0x7f9ae007b390 0x7f9ae0133bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:28:27.176 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:27.164+0000 7f9ae5d96700 1 --2- 172.21.15.42:0/290126608 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9ae007af60 0x7f9ae0133670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:28:27.177 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:27.164+0000 7f9ae6597700 1 -- 172.21.15.42:0/290126608 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9ae007af60 msgr2=0x7f9ae0133670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:28:27.177 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:27.164+0000 7f9ae6597700 1 --2- 172.21.15.42:0/290126608 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9ae007af60 0x7f9ae0133670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:28:27.177 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:27.164+0000 7f9ae6597700 1 -- 172.21.15.42:0/290126608 >> v2:172.21.15.40:3300/0 conn(0x7f9ae007b390 msgr2=0x7f9ae0133bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:28:27.177 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:27.164+0000 7f9ae6597700 1 --2- 172.21.15.42:0/290126608 >> v2:172.21.15.40:3300/0 conn(0x7f9ae007b390 0x7f9ae0133bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:28:27.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:27.164+0000 7f9ae6597700 1 -- 172.21.15.42:0/290126608 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ae00a4fd0 con 0x7f9ae0116500 2019-12-05T05:28:27.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:27.164+0000 7f9ae6597700 1 --2- 172.21.15.42:0/290126608 >> v2:172.21.15.42:3300/0 conn(0x7f9ae0116500 0x7f9ae007cf90 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f9ae00a56f0 tx=0x7f9adc00c450).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:27.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:27.164+0000 7f9ae6597700 1 --2- 172.21.15.42:0/290126608 >> v2:172.21.15.42:3300/0 conn(0x7f9ae0116500 0x7f9ae007cf90 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f9ae00a56f0 tx=0x7f9adc00c450).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:27.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:27.164+0000 7f9ae6597700 1 --2- 172.21.15.42:0/290126608 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9ae0116500 0x7f9ae007cf90 secure :-1 s=READY pgs=932 cs=0 l=1 rx=0x7f9ae00a56f0 tx=0x7f9adc00c450).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:28:27.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:27.164+0000 7f9ae6597700 1 --2- 172.21.15.42:0/290126608 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9ae0116500 0x7f9ae007cf90 secure :-1 s=THROTTLE_DONE pgs=932 cs=0 l=1 rx=0x7f9ae00a56f0 tx=0x7f9adc00c450).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:27.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:27.164+0000 7f9ae6597700 1 --2- 172.21.15.42:0/290126608 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9ae0116500 0x7f9ae007cf90 secure :-1 s=THROTTLE_DONE pgs=932 cs=0 l=1 rx=0x7f9ae00a56f0 tx=0x7f9adc00c450).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:27.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:27.164+0000 7f9ae6597700 1 --2- 172.21.15.42:0/290126608 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9ae0116500 0x7f9ae007cf90 secure :-1 s=THROTTLE_DONE pgs=932 cs=0 l=1 rx=0x7f9ae00a56f0 tx=0x7f9adc00c450).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:27.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:27.164+0000 7f9ad6ffd700 1 -- 172.21.15.42:0/290126608 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9adc00c6c0 con 0x7f9ae0116500 2019-12-05T05:28:27.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:27.164+0000 7f9ad6ffd700 1 -- 172.21.15.42:0/290126608 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9adc010170 con 0x7f9ae0116500 2019-12-05T05:28:27.184 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:27.164+0000 7f9ae7ffa700 1 -- 172.21.15.42:0/290126608 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ae00a5a70 con 0x7f9ae0116500 2019-12-05T05:28:27.184 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:27.164+0000 7f9ad6ffd700 1 -- 172.21.15.42:0/290126608 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9adc00cca0 con 0x7f9ae0116500 2019-12-05T05:28:27.184 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:27.164+0000 7f9ae7ffa700 1 -- 172.21.15.42:0/290126608 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ae011bec0 con 0x7f9ae0116500 2019-12-05T05:28:27.184 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:27.164+0000 7f9ae6597700 1 --2- 172.21.15.42:0/290126608 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9ae0116500 0x7f9ae007cf90 secure :-1 s=THROTTLE_DONE pgs=932 cs=0 l=1 rx=0x7f9ae00a56f0 tx=0x7f9adc00c450).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:27.184 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:27.164+0000 7f9ad6ffd700 1 -- 172.21.15.42:0/290126608 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f9adc005460 con 0x7f9ae0116500 2019-12-05T05:28:27.184 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:27.164+0000 7f9ae6597700 1 --2- 172.21.15.42:0/290126608 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9ae0116500 0x7f9ae007cf90 secure :-1 s=THROTTLE_DONE pgs=932 cs=0 l=1 rx=0x7f9ae00a56f0 tx=0x7f9adc00c450).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:27.185 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:27.164+0000 7f9ad6ffd700 1 --2- 172.21.15.42:0/290126608 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9ac803aeb0 0x7f9ac803d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:28:27.185 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:27.164+0000 7f9ad6ffd700 1 -- 172.21.15.42:0/290126608 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 7493+0+0 (secure 0 0 0) 0x7f9adc04ab30 con 0x7f9ae0116500 2019-12-05T05:28:27.185 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:27.164+0000 7f9ae5d96700 1 --2- 172.21.15.42:0/290126608 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9ac803aeb0 0x7f9ac803d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:28:27.185 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:27.168+0000 7f9ae5d96700 1 --2- 172.21.15.42:0/290126608 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9ac803aeb0 0x7f9ac803d3c0 crc :-1 s=READY pgs=406 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:28:27.186 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:27.172+0000 7f9ae7ffa700 1 -- 172.21.15.42:0/290126608 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ae005d110 con 0x7f9ae0116500 2019-12-05T05:28:27.186 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:27.176+0000 7f9ae6597700 1 --2- 172.21.15.42:0/290126608 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9ae0116500 0x7f9ae007cf90 secure :-1 s=THROTTLE_DONE pgs=932 cs=0 l=1 rx=0x7f9ae00a56f0 tx=0x7f9adc00c450).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:27.186 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:27.176+0000 7f9ad6ffd700 1 -- 172.21.15.42:0/290126608 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f9ae00a4fd0 con 0x7f9ae0116500 2019-12-05T05:28:27.380 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:27.376+0000 7f9ae7ffa700 1 -- 172.21.15.42:0/290126608 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9ae005b8c0 con 0x7f9ae0116500 2019-12-05T05:28:27.381 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:27.376+0000 7f9ae6597700 1 --2- 172.21.15.42:0/290126608 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9ae0116500 0x7f9ae007cf90 secure :-1 s=THROTTLE_DONE pgs=932 cs=0 l=1 rx=0x7f9ae00a56f0 tx=0x7f9adc00c450).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:28:27.381 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:27.376+0000 7f9ad6ffd700 1 -- 172.21.15.42:0/290126608 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 118 v118) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f9adc031af0 con 0x7f9ae0116500 2019-12-05T05:28:27.381 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 118 2019-12-05T05:28:27.382 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:28:27.382 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":118,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5426,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/3168718372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3168718372},{"type":"v1","addr":"172.21.15.40:6833","nonce":3168718372}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":118}],"filesystems":[{"mdsmap":{"epoch":117,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:28:18.130626+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6666},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6666":{"gid":6666,"name":"a","rank":0,"incarnation":114,"state":"up:active","state_seq":17,"addr":"172.21.15.42:6835/2544447110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2544447110},{"type":"v1","addr":"172.21.15.42:6835","nonce":2544447110}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:28:27.383 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:27.376+0000 7f9ae7ffa700 1 -- 172.21.15.42:0/290126608 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9ac803aeb0 msgr2=0x7f9ac803d3c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:28:27.383 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:27.376+0000 7f9ae7ffa700 1 --2- 172.21.15.42:0/290126608 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9ac803aeb0 0x7f9ac803d3c0 crc :-1 s=READY pgs=406 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:28:27.383 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:27.376+0000 7f9ae7ffa700 1 -- 172.21.15.42:0/290126608 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9ae0116500 msgr2=0x7f9ae007cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:28:27.383 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:27.376+0000 7f9ae7ffa700 1 --2- 172.21.15.42:0/290126608 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9ae0116500 0x7f9ae007cf90 secure :-1 s=READY pgs=932 cs=0 l=1 rx=0x7f9ae00a56f0 tx=0x7f9adc00c450).stop 2019-12-05T05:28:27.384 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:27.380+0000 7f9ae7ffa700 1 -- 172.21.15.42:0/290126608 shutdown_connections 2019-12-05T05:28:27.385 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:27.380+0000 7f9ae7ffa700 1 --2- 172.21.15.42:0/290126608 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9ac803aeb0 0x7f9ac803d3c0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:28:27.385 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:27.380+0000 7f9ae7ffa700 1 --2- 172.21.15.42:0/290126608 >> v2:172.21.15.40:3300/0 conn(0x7f9ae007b390 0x7f9ae0133bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:28:27.385 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:27.380+0000 7f9ae7ffa700 1 --2- 172.21.15.42:0/290126608 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9ae0116500 0x7f9ae007cf90 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:28:27.385 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:27.380+0000 7f9ae7ffa700 1 --2- 172.21.15.42:0/290126608 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9ae007af60 0x7f9ae0133670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:28:27.385 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:27.380+0000 7f9ae7ffa700 1 -- 172.21.15.42:0/290126608 shutdown_connections 2019-12-05T05:28:27.385 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:27.380+0000 7f9ae7ffa700 1 -- 172.21.15.42:0/290126608 wait complete. 2019-12-05T05:28:27.386 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:28:27.380+0000 7f9ae7ffa700 1 -- 172.21.15.42:0/290126608 >> 172.21.15.42:0/290126608 conn(0x7f9ae00a9790 msgr2=0x7f9ae0059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:28:27.396 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b reported in up:standby state 2019-12-05T05:28:27.396 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 35 secs before thrashing 2019-12-05T05:28:38.723 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:28:38.743 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:29:02.401 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:29:02.515 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.503+0000 7f6387b55700 1 Processor -- start 2019-12-05T05:29:02.516 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.503+0000 7f6387b55700 1 -- start start 2019-12-05T05:29:02.516 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.503+0000 7f6387b55700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f63800a7740 0x7f638008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:29:02.516 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.503+0000 7f6387b55700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63800a0710 con 0x7f638007e720 2019-12-05T05:29:02.516 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.503+0000 7f6387b55700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63800a9ba0 con 0x7f638009fae0 2019-12-05T05:29:02.516 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.503+0000 7f6387b55700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63800a0b00 con 0x7f63800a7740 2019-12-05T05:29:02.517 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.503+0000 7f63868f3700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f638009fae0 0x7f638014efe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:34668/0 (socket says 172.21.15.42:34668) 2019-12-05T05:29:02.517 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.503+0000 7f63868f3700 1 -- 172.21.15.42:0/1581701952 learned_addr learned my addr 172.21.15.42:0/1581701952 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:29:02.517 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.503+0000 7f63860f2700 1 --2- 172.21.15.42:0/1581701952 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f63800a7740 0x7f638008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:29:02.517 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.503+0000 7f63850f0700 1 -- 172.21.15.42:0/1581701952 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 651658916 0 0) 0x7f63800a9ba0 con 0x7f638009fae0 2019-12-05T05:29:02.517 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.503+0000 7f63850f0700 1 -- 172.21.15.42:0/1581701952 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f636c0030b0 con 0x7f638009fae0 2019-12-05T05:29:02.517 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.503+0000 7f63850f0700 1 -- 172.21.15.42:0/1581701952 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1746534373 0 0) 0x7f63800a0710 con 0x7f638007e720 2019-12-05T05:29:02.518 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.503+0000 7f63850f0700 1 -- 172.21.15.42:0/1581701952 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63800a9ba0 con 0x7f638007e720 2019-12-05T05:29:02.518 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.507+0000 7f63860f2700 1 -- 172.21.15.42:0/1581701952 >> v1:172.21.15.42:6789/0 conn(0x7f638009fae0 legacy=0x7f638014efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:29:02.518 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.507+0000 7f63860f2700 1 -- 172.21.15.42:0/1581701952 >> v1:172.21.15.40:6789/0 conn(0x7f638007e720 legacy=0x7f638009f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:29:02.518 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.507+0000 7f63860f2700 1 -- 172.21.15.42:0/1581701952 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f638007eb50 con 0x7f63800a7740 2019-12-05T05:29:02.518 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.507+0000 7f63860f2700 1 --2- 172.21.15.42:0/1581701952 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f63800a7740 0x7f638008df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f637c002f30 tx=0x7f637c00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:02.518 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.507+0000 7f63860f2700 1 --2- 172.21.15.42:0/1581701952 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f63800a7740 0x7f638008df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f637c002f30 tx=0x7f637c00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:02.519 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.507+0000 7f63860f2700 1 --2- 172.21.15.42:0/1581701952 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f63800a7740 0x7f638008df80 secure :-1 s=READY pgs=191 cs=0 l=1 rx=0x7f637c002f30 tx=0x7f637c00fbf0).ready entity=mon.2 client_cookie=b6fafc9553d04c99 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:29:02.519 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.507+0000 7f63860f2700 1 --2- 172.21.15.42:0/1581701952 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f63800a7740 0x7f638008df80 secure :-1 s=THROTTLE_DONE pgs=191 cs=0 l=1 rx=0x7f637c002f30 tx=0x7f637c00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:02.519 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.507+0000 7f63860f2700 1 --2- 172.21.15.42:0/1581701952 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f63800a7740 0x7f638008df80 secure :-1 s=THROTTLE_DONE pgs=191 cs=0 l=1 rx=0x7f637c002f30 tx=0x7f637c00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:02.519 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.507+0000 7f63850f0700 1 -- 172.21.15.42:0/1581701952 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f637c0112b0 con 0x7f63800a7740 2019-12-05T05:29:02.520 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.507+0000 7f63860f2700 1 --2- 172.21.15.42:0/1581701952 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f63800a7740 0x7f638008df80 secure :-1 s=THROTTLE_DONE pgs=191 cs=0 l=1 rx=0x7f637c002f30 tx=0x7f637c00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:02.520 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.507+0000 7f63850f0700 1 -- 172.21.15.42:0/1581701952 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f637c008bc0 con 0x7f63800a7740 2019-12-05T05:29:02.520 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.507+0000 7f63850f0700 1 -- 172.21.15.42:0/1581701952 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f637c011890 con 0x7f63800a7740 2019-12-05T05:29:02.520 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.507+0000 7f6387b55700 1 -- 172.21.15.42:0/1581701952 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f63800a7740 msgr2=0x7f638008df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:29:02.520 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.507+0000 7f6387b55700 1 --2- 172.21.15.42:0/1581701952 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f63800a7740 0x7f638008df80 secure :-1 s=READY pgs=191 cs=0 l=1 rx=0x7f637c002f30 tx=0x7f637c00fbf0).stop 2019-12-05T05:29:02.521 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.507+0000 7f6387b55700 1 -- 172.21.15.42:0/1581701952 shutdown_connections 2019-12-05T05:29:02.521 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.507+0000 7f6387b55700 1 --2- 172.21.15.42:0/1581701952 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f63800a7740 0x7f638008df80 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:29:02.521 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.507+0000 7f6387b55700 1 -- 172.21.15.42:0/1581701952 shutdown_connections 2019-12-05T05:29:02.521 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.507+0000 7f6387b55700 1 -- 172.21.15.42:0/1581701952 wait complete. 2019-12-05T05:29:02.521 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.507+0000 7f6387b55700 1 -- 172.21.15.42:0/1581701952 >> 172.21.15.42:0/1581701952 conn(0x7f63800aabe0 msgr2=0x7f63800a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:29:02.525 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.511+0000 7f6387b55700 1 Processor -- start 2019-12-05T05:29:02.525 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.511+0000 7f6387b55700 1 -- start start 2019-12-05T05:29:02.525 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.511+0000 7f6387b55700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f638007c530 0x7f6380130630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:29:02.525 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.511+0000 7f6387b55700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f638007c960 con 0x7f63800a5d40 2019-12-05T05:29:02.525 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.511+0000 7f6387b55700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63800a6770 con 0x7f6380118b70 2019-12-05T05:29:02.526 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.511+0000 7f6387b55700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6380130b80 con 0x7f638007c530 2019-12-05T05:29:02.526 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.511+0000 7f63858f1700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f638007c530 0x7f6380130630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:29:02.526 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.511+0000 7f63858f1700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f638007c530 0x7f6380130630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:34634/0 (socket says 172.21.15.42:34634) 2019-12-05T05:29:02.526 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.511+0000 7f63860f2700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f6380118b70 0x7f638011bed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:34670/0 (socket says 172.21.15.42:34670) 2019-12-05T05:29:02.526 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.511+0000 7f63858f1700 1 -- 172.21.15.42:0/3677864889 learned_addr learned my addr 172.21.15.42:0/3677864889 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:29:02.526 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.511+0000 7f6376ffd700 1 -- 172.21.15.42:0/3677864889 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3288658978 0 0) 0x7f638007c960 con 0x7f63800a5d40 2019-12-05T05:29:02.527 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.511+0000 7f63858f1700 1 -- 172.21.15.42:0/3677864889 >> v1:172.21.15.42:6789/0 conn(0x7f6380118b70 legacy=0x7f638011bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:29:02.527 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.515+0000 7f63858f1700 1 -- 172.21.15.42:0/3677864889 >> v1:172.21.15.40:6789/0 conn(0x7f63800a5d40 legacy=0x7f6380137a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:29:02.527 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.515+0000 7f63858f1700 1 -- 172.21.15.42:0/3677864889 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63800a6170 con 0x7f638007c530 2019-12-05T05:29:02.527 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.515+0000 7f63858f1700 1 --2- 172.21.15.42:0/3677864889 >> v2:172.21.15.42:3300/0 conn(0x7f638007c530 0x7f6380130630 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f637c008370 tx=0x7f637c00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:02.527 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.515+0000 7f63858f1700 1 --2- 172.21.15.42:0/3677864889 >> v2:172.21.15.42:3300/0 conn(0x7f638007c530 0x7f6380130630 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f637c008370 tx=0x7f637c00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:02.527 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.515+0000 7f63858f1700 1 --2- 172.21.15.42:0/3677864889 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f638007c530 0x7f6380130630 secure :-1 s=READY pgs=935 cs=0 l=1 rx=0x7f637c008370 tx=0x7f637c00fbf0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:29:02.527 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.515+0000 7f63858f1700 1 --2- 172.21.15.42:0/3677864889 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f638007c530 0x7f6380130630 secure :-1 s=THROTTLE_DONE pgs=935 cs=0 l=1 rx=0x7f637c008370 tx=0x7f637c00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:02.528 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.515+0000 7f6376ffd700 1 -- 172.21.15.42:0/3677864889 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f637000c8b0 con 0x7f638007c530 2019-12-05T05:29:02.528 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.515+0000 7f63858f1700 1 --2- 172.21.15.42:0/3677864889 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f638007c530 0x7f6380130630 secure :-1 s=THROTTLE_DONE pgs=935 cs=0 l=1 rx=0x7f637c008370 tx=0x7f637c00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:02.528 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.515+0000 7f63858f1700 1 --2- 172.21.15.42:0/3677864889 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f638007c530 0x7f6380130630 secure :-1 s=THROTTLE_DONE pgs=935 cs=0 l=1 rx=0x7f637c008370 tx=0x7f637c00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:02.528 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.515+0000 7f6376ffd700 1 -- 172.21.15.42:0/3677864889 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6370011e10 con 0x7f638007c530 2019-12-05T05:29:02.528 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.515+0000 7f6387b55700 1 -- 172.21.15.42:0/3677864889 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63800a6440 con 0x7f638007c530 2019-12-05T05:29:02.528 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.515+0000 7f6376ffd700 1 -- 172.21.15.42:0/3677864889 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f637000d4b0 con 0x7f638007c530 2019-12-05T05:29:02.528 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.515+0000 7f6387b55700 1 -- 172.21.15.42:0/3677864889 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6380139290 con 0x7f638007c530 2019-12-05T05:29:02.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.515+0000 7f63858f1700 1 --2- 172.21.15.42:0/3677864889 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f638007c530 0x7f6380130630 secure :-1 s=THROTTLE_DONE pgs=935 cs=0 l=1 rx=0x7f637c008370 tx=0x7f637c00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:02.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.515+0000 7f6376ffd700 1 -- 172.21.15.42:0/3677864889 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f637000d780 con 0x7f638007c530 2019-12-05T05:29:02.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.515+0000 7f63858f1700 1 --2- 172.21.15.42:0/3677864889 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f638007c530 0x7f6380130630 secure :-1 s=THROTTLE_DONE pgs=935 cs=0 l=1 rx=0x7f637c008370 tx=0x7f637c00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:02.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.515+0000 7f6376ffd700 1 --2- 172.21.15.42:0/3677864889 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f636803acd0 0x7f636803d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:29:02.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.515+0000 7f6376ffd700 1 -- 172.21.15.42:0/3677864889 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 7493+0+0 (secure 0 0 0) 0x7f637004b020 con 0x7f638007c530 2019-12-05T05:29:02.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.515+0000 7f63860f2700 1 --2- 172.21.15.42:0/3677864889 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f636803acd0 0x7f636803d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:29:02.530 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.519+0000 7f63860f2700 1 --2- 172.21.15.42:0/3677864889 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f636803acd0 0x7f636803d1e0 crc :-1 s=READY pgs=407 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:29:02.530 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.519+0000 7f6387b55700 1 -- 172.21.15.42:0/3677864889 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f638005d110 con 0x7f638007c530 2019-12-05T05:29:02.533 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.527+0000 7f63858f1700 1 --2- 172.21.15.42:0/3677864889 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f638007c530 0x7f6380130630 secure :-1 s=THROTTLE_DONE pgs=935 cs=0 l=1 rx=0x7f637c008370 tx=0x7f637c00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:02.533 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.527+0000 7f6376ffd700 1 -- 172.21.15.42:0/3677864889 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f63700320a0 con 0x7f638007c530 2019-12-05T05:29:02.709 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.703+0000 7f6387b55700 1 -- 172.21.15.42:0/3677864889 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f638005b8c0 con 0x7f638007c530 2019-12-05T05:29:02.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.703+0000 7f63858f1700 1 --2- 172.21.15.42:0/3677864889 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f638007c530 0x7f6380130630 secure :-1 s=THROTTLE_DONE pgs=935 cs=0 l=1 rx=0x7f637c008370 tx=0x7f637c00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:02.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.703+0000 7f6376ffd700 1 -- 172.21.15.42:0/3677864889 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 118 v118) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f6370032300 con 0x7f638007c530 2019-12-05T05:29:02.710 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 118 2019-12-05T05:29:02.711 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:29:02.711 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":118,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5426,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/3168718372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3168718372},{"type":"v1","addr":"172.21.15.40:6833","nonce":3168718372}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":118}],"filesystems":[{"mdsmap":{"epoch":117,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:28:18.130626+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6666},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6666":{"gid":6666,"name":"a","rank":0,"incarnation":114,"state":"up:active","state_seq":17,"addr":"172.21.15.42:6835/2544447110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2544447110},{"type":"v1","addr":"172.21.15.42:6835","nonce":2544447110}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:29:02.713 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.707+0000 7f6387b55700 1 -- 172.21.15.42:0/3677864889 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f636803acd0 msgr2=0x7f636803d1e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:29:02.713 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.707+0000 7f6387b55700 1 --2- 172.21.15.42:0/3677864889 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f636803acd0 0x7f636803d1e0 crc :-1 s=READY pgs=407 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:29:02.713 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.707+0000 7f6387b55700 1 -- 172.21.15.42:0/3677864889 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f638007c530 msgr2=0x7f6380130630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:29:02.713 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.707+0000 7f6387b55700 1 --2- 172.21.15.42:0/3677864889 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f638007c530 0x7f6380130630 secure :-1 s=READY pgs=935 cs=0 l=1 rx=0x7f637c008370 tx=0x7f637c00fbf0).stop 2019-12-05T05:29:02.713 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.707+0000 7f6387b55700 1 -- 172.21.15.42:0/3677864889 shutdown_connections 2019-12-05T05:29:02.713 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.707+0000 7f6387b55700 1 --2- 172.21.15.42:0/3677864889 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f636803acd0 0x7f636803d1e0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:29:02.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.707+0000 7f6387b55700 1 --2- 172.21.15.42:0/3677864889 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f638007c530 0x7f6380130630 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:29:02.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.707+0000 7f6387b55700 1 -- 172.21.15.42:0/3677864889 shutdown_connections 2019-12-05T05:29:02.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.707+0000 7f6387b55700 1 -- 172.21.15.42:0/3677864889 wait complete. 2019-12-05T05:29:02.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.707+0000 7f6387b55700 1 -- 172.21.15.42:0/3677864889 >> 172.21.15.42:0/3677864889 conn(0x7f638011d260 msgr2=0x7f6380059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:29:02.723 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.a (rank=0) 2019-12-05T05:29:02.723 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2019-12-05T05:29:02.723 INFO:teuthology.orchestra.run:waiting for 300 2019-12-05T05:29:02.737 INFO:tasks.ceph.mds.a:Stopped 2019-12-05T05:29:02.737 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:29:02.913 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.891+0000 7fa2fec9f700 1 Processor -- start 2019-12-05T05:29:02.913 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.891+0000 7fa2fec9f700 1 -- start start 2019-12-05T05:29:02.913 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.891+0000 7fa2fec9f700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa2f808df80 0x7fa2f809f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:29:02.913 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.891+0000 7fa2fec9f700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2f809fc60 con 0x7fa2f809f830 2019-12-05T05:29:02.913 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.891+0000 7fa2fec9f700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2f80a0710 con 0x7fa2f80a7740 2019-12-05T05:29:02.914 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.891+0000 7fa2fec9f700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2f80a0b00 con 0x7fa2f808df80 2019-12-05T05:29:02.914 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.891+0000 7fa2fd23c700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fa2f80a7740 0x7fa2f807e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:34682/0 (socket says 172.21.15.42:34682) 2019-12-05T05:29:02.914 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.891+0000 7fa2fd23c700 1 -- 172.21.15.42:0/1546373614 learned_addr learned my addr 172.21.15.42:0/1546373614 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:29:02.914 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.891+0000 7fa2fca3b700 1 --2- 172.21.15.42:0/1546373614 >> v2:172.21.15.40:3300/0 conn(0x7fa2f808df80 0x7fa2f809f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:29:02.914 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.891+0000 7fa2ebfff700 1 -- 172.21.15.42:0/1546373614 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2463426008 0 0) 0x7fa2f80a0710 con 0x7fa2f80a7740 2019-12-05T05:29:02.914 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.891+0000 7fa2ebfff700 1 -- 172.21.15.42:0/1546373614 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2e40030b0 con 0x7fa2f80a7740 2019-12-05T05:29:02.914 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.891+0000 7fa2ebfff700 1 -- 172.21.15.42:0/1546373614 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1995501237 0 0) 0x7fa2f809fc60 con 0x7fa2f809f830 2019-12-05T05:29:02.915 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.895+0000 7fa2ebfff700 1 -- 172.21.15.42:0/1546373614 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2f80a0710 con 0x7fa2f809f830 2019-12-05T05:29:02.915 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.895+0000 7fa2ebfff700 1 -- 172.21.15.42:0/1546373614 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3761455561 0 0) 0x7fa2e40030b0 con 0x7fa2f80a7740 2019-12-05T05:29:02.915 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.895+0000 7fa2fca3b700 1 -- 172.21.15.42:0/1546373614 >> v1:172.21.15.42:6789/0 conn(0x7fa2f80a7740 legacy=0x7fa2f807e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:29:02.915 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.895+0000 7fa2fca3b700 1 -- 172.21.15.42:0/1546373614 >> v1:172.21.15.40:6789/0 conn(0x7fa2f809f830 legacy=0x7fa2f814efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:29:02.915 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.895+0000 7fa2fca3b700 1 -- 172.21.15.42:0/1546373614 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2f807ef20 con 0x7fa2f808df80 2019-12-05T05:29:02.916 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.895+0000 7fa2fca3b700 1 --2- 172.21.15.42:0/1546373614 >> v2:172.21.15.40:3300/0 conn(0x7fa2f808df80 0x7fa2f809f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fa2ec008b10 tx=0x7fa2f4001950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:02.916 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.895+0000 7fa2fca3b700 1 --2- 172.21.15.42:0/1546373614 >> v2:172.21.15.40:3300/0 conn(0x7fa2f808df80 0x7fa2f809f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fa2ec008b10 tx=0x7fa2f4001950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:02.916 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.895+0000 7fa2fca3b700 1 --2- 172.21.15.42:0/1546373614 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa2f808df80 0x7fa2f809f2e0 secure :-1 s=READY pgs=730 cs=0 l=1 rx=0x7fa2ec008b10 tx=0x7fa2f4001950).ready entity=mon.0 client_cookie=d70164f2dd12f446 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:29:02.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.895+0000 7fa2fca3b700 1 --2- 172.21.15.42:0/1546373614 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa2f808df80 0x7fa2f809f2e0 secure :-1 s=THROTTLE_DONE pgs=730 cs=0 l=1 rx=0x7fa2ec008b10 tx=0x7fa2f4001950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:02.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.895+0000 7fa2fca3b700 1 --2- 172.21.15.42:0/1546373614 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa2f808df80 0x7fa2f809f2e0 secure :-1 s=THROTTLE_DONE pgs=730 cs=0 l=1 rx=0x7fa2ec008b10 tx=0x7fa2f4001950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:02.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.895+0000 7fa2ebfff700 1 -- 172.21.15.42:0/1546373614 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa2f4001750 con 0x7fa2f808df80 2019-12-05T05:29:02.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.895+0000 7fa2fca3b700 1 --2- 172.21.15.42:0/1546373614 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa2f808df80 0x7fa2f809f2e0 secure :-1 s=THROTTLE_DONE pgs=730 cs=0 l=1 rx=0x7fa2ec008b10 tx=0x7fa2f4001950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:02.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.895+0000 7fa2ebfff700 1 -- 172.21.15.42:0/1546373614 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2f4001560 con 0x7fa2f808df80 2019-12-05T05:29:02.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.895+0000 7fa2ebfff700 1 -- 172.21.15.42:0/1546373614 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa2ec011ac0 con 0x7fa2f808df80 2019-12-05T05:29:02.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.895+0000 7fa2fec9f700 1 -- 172.21.15.42:0/1546373614 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa2f808df80 msgr2=0x7fa2f809f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:29:02.918 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.895+0000 7fa2fec9f700 1 --2- 172.21.15.42:0/1546373614 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa2f808df80 0x7fa2f809f2e0 secure :-1 s=READY pgs=730 cs=0 l=1 rx=0x7fa2ec008b10 tx=0x7fa2f4001950).stop 2019-12-05T05:29:02.918 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.895+0000 7fa2fec9f700 1 -- 172.21.15.42:0/1546373614 shutdown_connections 2019-12-05T05:29:02.918 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.895+0000 7fa2fec9f700 1 --2- 172.21.15.42:0/1546373614 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa2f808df80 0x7fa2f809f2e0 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:29:02.918 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.895+0000 7fa2fec9f700 1 -- 172.21.15.42:0/1546373614 shutdown_connections 2019-12-05T05:29:02.918 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.895+0000 7fa2fec9f700 1 -- 172.21.15.42:0/1546373614 wait complete. 2019-12-05T05:29:02.918 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.895+0000 7fa2fec9f700 1 -- 172.21.15.42:0/1546373614 >> 172.21.15.42:0/1546373614 conn(0x7fa2f80aabe0 msgr2=0x7fa2f80a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:29:02.918 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.895+0000 7fa2fec9f700 1 Processor -- start 2019-12-05T05:29:02.918 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.895+0000 7fa2fec9f700 1 -- start start 2019-12-05T05:29:02.919 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.899+0000 7fa2fec9f700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fa2f80a6df0 0x7fa2f81161c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:29:02.919 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.899+0000 7fa2fec9f700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa2f80742e0 0x7fa2f8076720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:29:02.919 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.899+0000 7fa2fec9f700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2f8116710 con 0x7fa2f808eeb0 2019-12-05T05:29:02.919 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.899+0000 7fa2fec9f700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2f80a7220 con 0x7fa2f80a6df0 2019-12-05T05:29:02.919 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.899+0000 7fa2fec9f700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2f80a54f0 con 0x7fa2f80742e0 2019-12-05T05:29:02.919 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.899+0000 7fa2fca3b700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fa2f80a6df0 0x7fa2f81161c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:29:02.919 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.899+0000 7fa2fd23c700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fa2f808eeb0 0x7fa2f80a5b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:34686/0 (socket says 172.21.15.42:34686) 2019-12-05T05:29:02.920 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.899+0000 7fa2fd23c700 1 -- 172.21.15.42:0/2758044867 learned_addr learned my addr 172.21.15.42:0/2758044867 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:29:02.920 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.899+0000 7fa2fda3d700 1 --2- 172.21.15.42:0/2758044867 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa2f80742e0 0x7fa2f8076720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:29:02.920 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.899+0000 7fa2fca3b700 1 -- 172.21.15.42:0/2758044867 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa2f80742e0 msgr2=0x7fa2f8076720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:29:02.920 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.899+0000 7fa2fca3b700 1 --2- 172.21.15.42:0/2758044867 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa2f80742e0 0x7fa2f8076720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:29:02.920 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.899+0000 7fa2fca3b700 1 -- 172.21.15.42:0/2758044867 >> v1:172.21.15.42:6789/0 conn(0x7fa2f808eeb0 legacy=0x7fa2f80a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:29:02.920 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.899+0000 7fa2e9ffb700 1 -- 172.21.15.42:0/2758044867 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4268690180 0 0) 0x7fa2f8116710 con 0x7fa2f808eeb0 2019-12-05T05:29:02.920 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.899+0000 7fa2fca3b700 1 -- 172.21.15.42:0/2758044867 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2f8077c80 con 0x7fa2f80a6df0 2019-12-05T05:29:02.921 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.899+0000 7fa2fca3b700 1 --2- 172.21.15.42:0/2758044867 >> v2:172.21.15.42:3300/0 conn(0x7fa2f80a6df0 0x7fa2f81161c0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fa2ec008b10 tx=0x7fa2ec008870).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:02.921 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.899+0000 7fa2fca3b700 1 --2- 172.21.15.42:0/2758044867 >> v2:172.21.15.42:3300/0 conn(0x7fa2f80a6df0 0x7fa2f81161c0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fa2ec008b10 tx=0x7fa2ec008870).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:02.921 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.899+0000 7fa2fca3b700 1 --2- 172.21.15.42:0/2758044867 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa2f80a6df0 0x7fa2f81161c0 secure :-1 s=READY pgs=938 cs=0 l=1 rx=0x7fa2ec008b10 tx=0x7fa2ec008870).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:29:02.921 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.899+0000 7fa2fca3b700 1 --2- 172.21.15.42:0/2758044867 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa2f80a6df0 0x7fa2f81161c0 secure :-1 s=THROTTLE_DONE pgs=938 cs=0 l=1 rx=0x7fa2ec008b10 tx=0x7fa2ec008870).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:02.921 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.899+0000 7fa2e9ffb700 1 -- 172.21.15.42:0/2758044867 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa2ec00f700 con 0x7fa2f80a6df0 2019-12-05T05:29:02.921 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.899+0000 7fa2fca3b700 1 --2- 172.21.15.42:0/2758044867 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa2f80a6df0 0x7fa2f81161c0 secure :-1 s=THROTTLE_DONE pgs=938 cs=0 l=1 rx=0x7fa2ec008b10 tx=0x7fa2ec008870).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:02.921 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.899+0000 7fa2fca3b700 1 --2- 172.21.15.42:0/2758044867 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa2f80a6df0 0x7fa2f81161c0 secure :-1 s=THROTTLE_DONE pgs=938 cs=0 l=1 rx=0x7fa2ec008b10 tx=0x7fa2ec008870).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:02.922 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.899+0000 7fa2e9ffb700 1 -- 172.21.15.42:0/2758044867 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2ec0116e0 con 0x7fa2f80a6df0 2019-12-05T05:29:02.922 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.899+0000 7fa2e9ffb700 1 -- 172.21.15.42:0/2758044867 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa2ec005ab0 con 0x7fa2f80a6df0 2019-12-05T05:29:02.922 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.899+0000 7fa2fec9f700 1 -- 172.21.15.42:0/2758044867 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2f8076c70 con 0x7fa2f80a6df0 2019-12-05T05:29:02.922 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.899+0000 7fa2fec9f700 1 -- 172.21.15.42:0/2758044867 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2f8077140 con 0x7fa2f80a6df0 2019-12-05T05:29:02.922 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.899+0000 7fa2fca3b700 1 --2- 172.21.15.42:0/2758044867 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa2f80a6df0 0x7fa2f81161c0 secure :-1 s=THROTTLE_DONE pgs=938 cs=0 l=1 rx=0x7fa2ec008b10 tx=0x7fa2ec008870).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:02.922 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.903+0000 7fa2e9ffb700 1 -- 172.21.15.42:0/2758044867 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fa2ec007150 con 0x7fa2f80a6df0 2019-12-05T05:29:02.922 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.903+0000 7fa2fca3b700 1 --2- 172.21.15.42:0/2758044867 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa2f80a6df0 0x7fa2f81161c0 secure :-1 s=THROTTLE_DONE pgs=938 cs=0 l=1 rx=0x7fa2ec008b10 tx=0x7fa2ec008870).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:02.923 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.903+0000 7fa2e9ffb700 1 --2- 172.21.15.42:0/2758044867 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa2d803aca0 0x7fa2d803d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:29:02.923 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.903+0000 7fa2e9ffb700 1 -- 172.21.15.42:0/2758044867 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 7493+0+0 (secure 0 0 0) 0x7fa2ec04b330 con 0x7fa2f80a6df0 2019-12-05T05:29:02.923 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.903+0000 7fa2fd23c700 1 --2- 172.21.15.42:0/2758044867 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa2d803aca0 0x7fa2d803d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:29:02.923 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.903+0000 7fa2fd23c700 1 --2- 172.21.15.42:0/2758044867 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa2d803aca0 0x7fa2d803d1b0 crc :-1 s=READY pgs=408 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:29:02.924 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.907+0000 7fa2fec9f700 1 -- 172.21.15.42:0/2758044867 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2f8117b90 con 0x7fa2f80a6df0 2019-12-05T05:29:02.924 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.915+0000 7fa2fca3b700 1 --2- 172.21.15.42:0/2758044867 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa2f80a6df0 0x7fa2f81161c0 secure :-1 s=THROTTLE_DONE pgs=938 cs=0 l=1 rx=0x7fa2ec008b10 tx=0x7fa2ec008870).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:02.924 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:02.915+0000 7fa2e9ffb700 1 -- 172.21.15.42:0/2758044867 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fa2ec0325f0 con 0x7fa2f80a6df0 2019-12-05T05:29:03.089 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:03.083+0000 7fa2fec9f700 1 -- 172.21.15.42:0/2758044867 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa2f805d110 con 0x7fa2f80a6df0 2019-12-05T05:29:03.089 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:03.083+0000 7fa2fca3b700 1 --2- 172.21.15.42:0/2758044867 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa2f80a6df0 0x7fa2f81161c0 secure :-1 s=THROTTLE_DONE pgs=938 cs=0 l=1 rx=0x7fa2ec008b10 tx=0x7fa2ec008870).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:03.089 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:03.083+0000 7fa2e9ffb700 1 -- 172.21.15.42:0/2758044867 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 118 v118) v1 ==== 95+0+2112 (secure 0 0 0) 0x7fa2ec005d50 con 0x7fa2f80a6df0 2019-12-05T05:29:03.090 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 118 2019-12-05T05:29:03.090 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:29:03.090 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":118,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5426,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/3168718372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3168718372},{"type":"v1","addr":"172.21.15.40:6833","nonce":3168718372}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":118}],"filesystems":[{"mdsmap":{"epoch":117,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:28:18.130626+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6666},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6666":{"gid":6666,"name":"a","rank":0,"incarnation":114,"state":"up:active","state_seq":17,"addr":"172.21.15.42:6835/2544447110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2544447110},{"type":"v1","addr":"172.21.15.42:6835","nonce":2544447110}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:29:03.094 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:03.087+0000 7fa2fec9f700 1 -- 172.21.15.42:0/2758044867 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa2d803aca0 msgr2=0x7fa2d803d1b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:29:03.094 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:03.087+0000 7fa2fec9f700 1 --2- 172.21.15.42:0/2758044867 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa2d803aca0 0x7fa2d803d1b0 crc :-1 s=READY pgs=408 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:29:03.095 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:03.087+0000 7fa2fec9f700 1 -- 172.21.15.42:0/2758044867 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa2f80a6df0 msgr2=0x7fa2f81161c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:29:03.095 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:03.087+0000 7fa2fec9f700 1 --2- 172.21.15.42:0/2758044867 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa2f80a6df0 0x7fa2f81161c0 secure :-1 s=READY pgs=938 cs=0 l=1 rx=0x7fa2ec008b10 tx=0x7fa2ec008870).stop 2019-12-05T05:29:03.095 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:03.087+0000 7fa2fec9f700 1 -- 172.21.15.42:0/2758044867 shutdown_connections 2019-12-05T05:29:03.095 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:03.087+0000 7fa2fec9f700 1 --2- 172.21.15.42:0/2758044867 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa2d803aca0 0x7fa2d803d1b0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:29:03.095 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:03.087+0000 7fa2fec9f700 1 --2- 172.21.15.42:0/2758044867 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa2f80742e0 0x7fa2f8076720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:29:03.095 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:03.087+0000 7fa2fec9f700 1 --2- 172.21.15.42:0/2758044867 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa2f80a6df0 0x7fa2f81161c0 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:29:03.095 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:03.087+0000 7fa2fec9f700 1 -- 172.21.15.42:0/2758044867 shutdown_connections 2019-12-05T05:29:03.096 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:03.087+0000 7fa2fec9f700 1 -- 172.21.15.42:0/2758044867 wait complete. 2019-12-05T05:29:03.096 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:03.087+0000 7fa2fec9f700 1 -- 172.21.15.42:0/2758044867 >> 172.21.15.42:0/2758044867 conn(0x7fa2f80aabe0 msgr2=0x7fa2f80a7f90 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:29:03.106 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T05:29:05.107 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:29:05.214 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:05.199+0000 7f11c0ec5700 1 Processor -- start 2019-12-05T05:29:05.214 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:05.199+0000 7f11c0ec5700 1 -- start start 2019-12-05T05:29:05.214 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:05.199+0000 7f11c0ec5700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f11bc08df70 0x7f11bc09f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:29:05.214 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:05.199+0000 7f11c0ec5700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f11bc09f820 0x7f11bc14efd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:29:05.215 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:05.199+0000 7f11c0ec5700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11bc09fc50 con 0x7f11bc0a7730 2019-12-05T05:29:05.215 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:05.199+0000 7f11c0ec5700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11bc0a0af0 con 0x7f11bc08df70 2019-12-05T05:29:05.215 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:05.199+0000 7f11c0ec5700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11bc08e5a0 con 0x7f11bc09f820 2019-12-05T05:29:05.215 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:05.199+0000 7f11bb59e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f11bc09f820 0x7f11bc14efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:29:05.215 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:05.199+0000 7f11bb59e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f11bc09f820 0x7f11bc14efd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:34666/0 (socket says 172.21.15.42:34666) 2019-12-05T05:29:05.215 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:05.199+0000 7f11ba59c700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f11bc08df70 0x7f11bc09f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:29:05.216 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:05.199+0000 7f11bb59e700 1 -- 172.21.15.42:0/3416563536 learned_addr learned my addr 172.21.15.42:0/3416563536 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:29:05.216 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:05.199+0000 7f11bb59e700 1 -- 172.21.15.42:0/3416563536 >> v2:172.21.15.40:3300/0 conn(0x7f11bc08df70 msgr2=0x7f11bc09f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:29:05.216 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:05.199+0000 7f11bb59e700 1 --2- 172.21.15.42:0/3416563536 >> v2:172.21.15.40:3300/0 conn(0x7f11bc08df70 0x7f11bc09f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:29:05.216 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:05.199+0000 7f11bb59e700 1 -- 172.21.15.42:0/3416563536 >> v1:172.21.15.40:6789/0 conn(0x7f11bc0a7730 legacy=0x7f11bc07e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:29:05.216 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:05.199+0000 7f11bb59e700 1 -- 172.21.15.42:0/3416563536 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11bc0a0700 con 0x7f11bc09f820 2019-12-05T05:29:05.216 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:05.199+0000 7f11bb59e700 1 --2- 172.21.15.42:0/3416563536 >> v2:172.21.15.42:3300/0 conn(0x7f11bc09f820 0x7f11bc14efd0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f11ac002f30 tx=0x7f11ac00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:05.217 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:05.199+0000 7f11bb59e700 1 --2- 172.21.15.42:0/3416563536 >> v2:172.21.15.42:3300/0 conn(0x7f11bc09f820 0x7f11bc14efd0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f11ac002f30 tx=0x7f11ac00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:05.217 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:05.199+0000 7f11bb59e700 1 --2- 172.21.15.42:0/3416563536 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f11bc09f820 0x7f11bc14efd0 secure :-1 s=READY pgs=939 cs=0 l=1 rx=0x7f11ac002f30 tx=0x7f11ac00fd30).ready entity=mon.1 client_cookie=59bcddbd16113247 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:29:05.217 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:05.199+0000 7f11bb59e700 1 --2- 172.21.15.42:0/3416563536 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f11bc09f820 0x7f11bc14efd0 secure :-1 s=THROTTLE_DONE pgs=939 cs=0 l=1 rx=0x7f11ac002f30 tx=0x7f11ac00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:05.217 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:05.199+0000 7f11bb59e700 1 --2- 172.21.15.42:0/3416563536 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f11bc09f820 0x7f11bc14efd0 secure :-1 s=THROTTLE_DONE pgs=939 cs=0 l=1 rx=0x7f11ac002f30 tx=0x7f11ac00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:05.217 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:05.199+0000 7f11b9d9b700 1 -- 172.21.15.42:0/3416563536 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f11ac008c90 con 0x7f11bc09f820 2019-12-05T05:29:05.217 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:05.199+0000 7f11bb59e700 1 --2- 172.21.15.42:0/3416563536 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f11bc09f820 0x7f11bc14efd0 secure :-1 s=THROTTLE_DONE pgs=939 cs=0 l=1 rx=0x7f11ac002f30 tx=0x7f11ac00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:05.217 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:05.199+0000 7f11b9d9b700 1 -- 172.21.15.42:0/3416563536 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11ac0088b0 con 0x7f11bc09f820 2019-12-05T05:29:05.218 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:05.199+0000 7f11b9d9b700 1 -- 172.21.15.42:0/3416563536 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f11ac011ad0 con 0x7f11bc09f820 2019-12-05T05:29:05.218 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:05.199+0000 7f11c0ec5700 1 -- 172.21.15.42:0/3416563536 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f11bc09f820 msgr2=0x7f11bc14efd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:29:05.218 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:05.199+0000 7f11c0ec5700 1 --2- 172.21.15.42:0/3416563536 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f11bc09f820 0x7f11bc14efd0 secure :-1 s=READY pgs=939 cs=0 l=1 rx=0x7f11ac002f30 tx=0x7f11ac00fd30).stop 2019-12-05T05:29:05.218 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:05.203+0000 7f11c0ec5700 1 -- 172.21.15.42:0/3416563536 shutdown_connections 2019-12-05T05:29:05.218 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:05.203+0000 7f11c0ec5700 1 --2- 172.21.15.42:0/3416563536 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f11bc09f820 0x7f11bc14efd0 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:29:05.218 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:05.203+0000 7f11c0ec5700 1 --2- 172.21.15.42:0/3416563536 >> v2:172.21.15.40:3300/0 conn(0x7f11bc08df70 0x7f11bc09f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:29:05.218 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:05.203+0000 7f11c0ec5700 1 -- 172.21.15.42:0/3416563536 shutdown_connections 2019-12-05T05:29:05.219 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:05.203+0000 7f11c0ec5700 1 -- 172.21.15.42:0/3416563536 wait complete. 2019-12-05T05:29:05.219 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:05.203+0000 7f11c0ec5700 1 -- 172.21.15.42:0/3416563536 >> 172.21.15.42:0/3416563536 conn(0x7f11bc0aabd0 msgr2=0x7f11bc0a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:29:05.219 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:05.203+0000 7f11c0ec5700 1 Processor -- start 2019-12-05T05:29:05.219 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:05.203+0000 7f11c0ec5700 1 -- start start 2019-12-05T05:29:05.220 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:05.207+0000 7f11c0ec5700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f11bc11a6b0 0x7f11bc11bad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:29:05.220 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:05.207+0000 7f11c0ec5700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f11bc119450 0x7f11bc119880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:29:05.220 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:05.207+0000 7f11c0ec5700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11bc11d670 con 0x7f11bc12f6e0 2019-12-05T05:29:05.220 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:05.207+0000 7f11c0ec5700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11bc11c6f0 con 0x7f11bc119450 2019-12-05T05:29:05.221 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:05.207+0000 7f11c0ec5700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11bc11aae0 con 0x7f11bc11a6b0 2019-12-05T05:29:05.221 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:05.207+0000 7f11bb59e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f11bc119450 0x7f11bc119880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:29:05.221 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:05.207+0000 7f11bb59e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f11bc119450 0x7f11bc119880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:34672/0 (socket says 172.21.15.42:34672) 2019-12-05T05:29:05.221 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:05.207+0000 7f11bb59e700 1 -- 172.21.15.42:0/326988827 learned_addr learned my addr 172.21.15.42:0/326988827 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:29:05.221 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:05.207+0000 7f11bad9d700 1 --2- 172.21.15.42:0/326988827 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f11bc11a6b0 0x7f11bc11bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:29:05.221 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:05.207+0000 7f11b37fe700 1 -- 172.21.15.42:0/326988827 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3265694138 0 0) 0x7f11bc11d670 con 0x7f11bc12f6e0 2019-12-05T05:29:05.222 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:05.207+0000 7f11bb59e700 1 -- 172.21.15.42:0/326988827 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f11bc11a6b0 msgr2=0x7f11bc11bad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:29:05.222 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:05.207+0000 7f11bb59e700 1 --2- 172.21.15.42:0/326988827 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f11bc11a6b0 0x7f11bc11bad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:29:05.222 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:05.207+0000 7f11bb59e700 1 -- 172.21.15.42:0/326988827 >> v1:172.21.15.40:6789/0 conn(0x7f11bc12f6e0 legacy=0x7f11bc12fb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:29:05.222 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:05.207+0000 7f11bb59e700 1 -- 172.21.15.42:0/326988827 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11bc11c020 con 0x7f11bc119450 2019-12-05T05:29:05.222 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:05.207+0000 7f11bb59e700 1 --2- 172.21.15.42:0/326988827 >> v2:172.21.15.42:3300/0 conn(0x7f11bc119450 0x7f11bc119880 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f11ac002f30 tx=0x7f11ac008b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:05.222 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:05.207+0000 7f11bb59e700 1 --2- 172.21.15.42:0/326988827 >> v2:172.21.15.42:3300/0 conn(0x7f11bc119450 0x7f11bc119880 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f11ac002f30 tx=0x7f11ac008b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:05.222 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:05.207+0000 7f11bb59e700 1 --2- 172.21.15.42:0/326988827 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f11bc119450 0x7f11bc119880 secure :-1 s=READY pgs=940 cs=0 l=1 rx=0x7f11ac002f30 tx=0x7f11ac008b60).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:29:05.223 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:05.207+0000 7f11bb59e700 1 --2- 172.21.15.42:0/326988827 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f11bc119450 0x7f11bc119880 secure :-1 s=THROTTLE_DONE pgs=940 cs=0 l=1 rx=0x7f11ac002f30 tx=0x7f11ac008b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:05.223 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:05.207+0000 7f11b37fe700 1 -- 172.21.15.42:0/326988827 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f11ac00f680 con 0x7f11bc119450 2019-12-05T05:29:05.223 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:05.207+0000 7f11bb59e700 1 --2- 172.21.15.42:0/326988827 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f11bc119450 0x7f11bc119880 secure :-1 s=THROTTLE_DONE pgs=940 cs=0 l=1 rx=0x7f11ac002f30 tx=0x7f11ac008b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:05.223 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:05.207+0000 7f11bb59e700 1 --2- 172.21.15.42:0/326988827 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f11bc119450 0x7f11bc119880 secure :-1 s=THROTTLE_DONE pgs=940 cs=0 l=1 rx=0x7f11ac002f30 tx=0x7f11ac008b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:05.223 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:05.207+0000 7f11b37fe700 1 -- 172.21.15.42:0/326988827 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11ac0116f0 con 0x7f11bc119450 2019-12-05T05:29:05.223 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:05.207+0000 7f11c0ec5700 1 -- 172.21.15.42:0/326988827 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11bc119dd0 con 0x7f11bc119450 2019-12-05T05:29:05.223 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:05.207+0000 7f11b37fe700 1 -- 172.21.15.42:0/326988827 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f11ac005920 con 0x7f11bc119450 2019-12-05T05:29:05.224 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:05.207+0000 7f11c0ec5700 1 -- 172.21.15.42:0/326988827 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11bc0772f0 con 0x7f11bc119450 2019-12-05T05:29:05.225 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:05.207+0000 7f11bb59e700 1 --2- 172.21.15.42:0/326988827 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f11bc119450 0x7f11bc119880 secure :-1 s=THROTTLE_DONE pgs=940 cs=0 l=1 rx=0x7f11ac002f30 tx=0x7f11ac008b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:05.225 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:05.207+0000 7f11b37fe700 1 -- 172.21.15.42:0/326988827 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f11ac005bc0 con 0x7f11bc119450 2019-12-05T05:29:05.225 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:05.207+0000 7f11bb59e700 1 --2- 172.21.15.42:0/326988827 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f11bc119450 0x7f11bc119880 secure :-1 s=THROTTLE_DONE pgs=940 cs=0 l=1 rx=0x7f11ac002f30 tx=0x7f11ac008b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:05.226 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:05.207+0000 7f11b37fe700 1 --2- 172.21.15.42:0/326988827 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f11a003ac60 0x7f11a003d170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:29:05.226 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:05.207+0000 7f11b37fe700 1 -- 172.21.15.42:0/326988827 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 7493+0+0 (secure 0 0 0) 0x7f11ac04b940 con 0x7f11bc119450 2019-12-05T05:29:05.226 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:05.211+0000 7f11bad9d700 1 --2- 172.21.15.42:0/326988827 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f11a003ac60 0x7f11a003d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:29:05.226 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:05.211+0000 7f11bad9d700 1 --2- 172.21.15.42:0/326988827 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f11a003ac60 0x7f11a003d170 crc :-1 s=READY pgs=409 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:29:05.226 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:05.215+0000 7f11c0ec5700 1 -- 172.21.15.42:0/326988827 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11bc05d100 con 0x7f11bc119450 2019-12-05T05:29:05.226 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:05.219+0000 7f11bb59e700 1 --2- 172.21.15.42:0/326988827 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f11bc119450 0x7f11bc119880 secure :-1 s=THROTTLE_DONE pgs=940 cs=0 l=1 rx=0x7f11ac002f30 tx=0x7f11ac008b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:05.227 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:05.219+0000 7f11b37fe700 1 -- 172.21.15.42:0/326988827 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f11ac0329a0 con 0x7f11bc119450 2019-12-05T05:29:05.388 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:05.383+0000 7f11c0ec5700 1 -- 172.21.15.42:0/326988827 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f11bc05b8b0 con 0x7f11bc119450 2019-12-05T05:29:05.390 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:05.383+0000 7f11bb59e700 1 --2- 172.21.15.42:0/326988827 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f11bc119450 0x7f11bc119880 secure :-1 s=THROTTLE_DONE pgs=940 cs=0 l=1 rx=0x7f11ac002f30 tx=0x7f11ac008b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:05.390 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:05.383+0000 7f11b37fe700 1 -- 172.21.15.42:0/326988827 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 118 v118) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f11ac007660 con 0x7f11bc119450 2019-12-05T05:29:05.391 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 118 2019-12-05T05:29:05.392 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:29:05.392 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":118,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5426,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/3168718372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3168718372},{"type":"v1","addr":"172.21.15.40:6833","nonce":3168718372}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":118}],"filesystems":[{"mdsmap":{"epoch":117,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:28:18.130626+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6666},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6666":{"gid":6666,"name":"a","rank":0,"incarnation":114,"state":"up:active","state_seq":17,"addr":"172.21.15.42:6835/2544447110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2544447110},{"type":"v1","addr":"172.21.15.42:6835","nonce":2544447110}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:29:05.399 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:05.387+0000 7f11c0ec5700 1 -- 172.21.15.42:0/326988827 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f11a003ac60 msgr2=0x7f11a003d170 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:29:05.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:05.387+0000 7f11c0ec5700 1 --2- 172.21.15.42:0/326988827 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f11a003ac60 0x7f11a003d170 crc :-1 s=READY pgs=409 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:29:05.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:05.387+0000 7f11c0ec5700 1 -- 172.21.15.42:0/326988827 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f11bc119450 msgr2=0x7f11bc119880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:29:05.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:05.387+0000 7f11c0ec5700 1 --2- 172.21.15.42:0/326988827 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f11bc119450 0x7f11bc119880 secure :-1 s=READY pgs=940 cs=0 l=1 rx=0x7f11ac002f30 tx=0x7f11ac008b60).stop 2019-12-05T05:29:05.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:05.387+0000 7f11c0ec5700 1 -- 172.21.15.42:0/326988827 shutdown_connections 2019-12-05T05:29:05.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:05.387+0000 7f11c0ec5700 1 --2- 172.21.15.42:0/326988827 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f11a003ac60 0x7f11a003d170 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:29:05.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:05.387+0000 7f11c0ec5700 1 --2- 172.21.15.42:0/326988827 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f11bc119450 0x7f11bc119880 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:29:05.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:05.387+0000 7f11c0ec5700 1 --2- 172.21.15.42:0/326988827 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f11bc11a6b0 0x7f11bc11bad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:29:05.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:05.387+0000 7f11c0ec5700 1 -- 172.21.15.42:0/326988827 shutdown_connections 2019-12-05T05:29:05.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:05.387+0000 7f11c0ec5700 1 -- 172.21.15.42:0/326988827 wait complete. 2019-12-05T05:29:05.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:05.387+0000 7f11c0ec5700 1 -- 172.21.15.42:0/326988827 >> 172.21.15.42:0/326988827 conn(0x7f11bc08fd90 msgr2=0x7f11bc11dad0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:29:05.404 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T05:29:07.409 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:29:07.516 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:07.503+0000 7fd309cf4700 1 Processor -- start 2019-12-05T05:29:07.516 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:07.503+0000 7fd309cf4700 1 -- start start 2019-12-05T05:29:07.517 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:07.503+0000 7fd309cf4700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd30408df80 0x7fd30409f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:29:07.517 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:07.503+0000 7fd309cf4700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd30409fc60 con 0x7fd30409f830 2019-12-05T05:29:07.517 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:07.503+0000 7fd309cf4700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3040a0710 con 0x7fd3040a7740 2019-12-05T05:29:07.517 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:07.503+0000 7fd309cf4700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3040a0b00 con 0x7fd30408df80 2019-12-05T05:29:07.517 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:07.503+0000 7fd303fff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fd3040a7740 0x7fd30407e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:34716/0 (socket says 172.21.15.42:34716) 2019-12-05T05:29:07.517 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:07.503+0000 7fd303fff700 1 -- 172.21.15.42:0/3238639990 learned_addr learned my addr 172.21.15.42:0/3238639990 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:29:07.518 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:07.503+0000 7fd2fb7fe700 1 --2- 172.21.15.42:0/3238639990 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd30408df80 0x7fd30409f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:29:07.518 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:07.507+0000 7fd3037fe700 1 -- 172.21.15.42:0/3238639990 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4050222106 0 0) 0x7fd3040a0710 con 0x7fd3040a7740 2019-12-05T05:29:07.518 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:07.507+0000 7fd3037fe700 1 -- 172.21.15.42:0/3238639990 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2f00030b0 con 0x7fd3040a7740 2019-12-05T05:29:07.518 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:07.507+0000 7fd3037fe700 1 -- 172.21.15.42:0/3238639990 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1725775874 0 0) 0x7fd30409fc60 con 0x7fd30409f830 2019-12-05T05:29:07.518 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:07.507+0000 7fd3037fe700 1 -- 172.21.15.42:0/3238639990 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3040a0710 con 0x7fd30409f830 2019-12-05T05:29:07.519 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:07.507+0000 7fd3037fe700 1 -- 172.21.15.42:0/3238639990 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1686423787 0 0) 0x7fd2f00030b0 con 0x7fd3040a7740 2019-12-05T05:29:07.519 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:07.507+0000 7fd3037fe700 1 -- 172.21.15.42:0/3238639990 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd30408df80 msgr2=0x7fd30409f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:29:07.519 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:07.507+0000 7fd3037fe700 1 --2- 172.21.15.42:0/3238639990 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd30408df80 0x7fd30409f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:29:07.519 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:07.507+0000 7fd3037fe700 1 -- 172.21.15.42:0/3238639990 >> v1:172.21.15.40:6789/0 conn(0x7fd30409f830 legacy=0x7fd30414efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:29:07.519 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:07.507+0000 7fd3037fe700 1 -- 172.21.15.42:0/3238639990 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd30407ef20 con 0x7fd3040a7740 2019-12-05T05:29:07.520 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:07.507+0000 7fd3037fe700 1 -- 172.21.15.42:0/3238639990 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7fd2f4001750 con 0x7fd3040a7740 2019-12-05T05:29:07.520 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:07.507+0000 7fd3037fe700 1 -- 172.21.15.42:0/3238639990 >> v1:172.21.15.42:6789/0 conn(0x7fd3040a7740 legacy=0x7fd30407e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:29:07.520 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:07.507+0000 7fd3037fe700 1 --2- 172.21.15.42:0/3238639990 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd2f0003940 0x7fd2f0003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:29:07.520 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:07.507+0000 7fd3037fe700 1 --2- 172.21.15.42:0/3238639990 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd2f0004420 0x7fd2f0004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:29:07.528 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:07.507+0000 7fd3037fe700 1 --2- 172.21.15.42:0/3238639990 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd2f0004e40 0x7fd2f004f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:29:07.528 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:07.507+0000 7fd3037fe700 1 -- 172.21.15.42:0/3238639990 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3040a0b00 con 0x7fd2f0004e40 2019-12-05T05:29:07.528 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:07.507+0000 7fd3037fe700 1 -- 172.21.15.42:0/3238639990 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2f004fc00 con 0x7fd2f0004420 2019-12-05T05:29:07.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:07.507+0000 7fd3037fe700 1 -- 172.21.15.42:0/3238639990 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2f004fe40 con 0x7fd2f0003940 2019-12-05T05:29:07.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:07.507+0000 7fd2fb7fe700 1 --2- 172.21.15.42:0/3238639990 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd2f0004420 0x7fd2f0004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:29:07.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:07.507+0000 7fd308a92700 1 --2- 172.21.15.42:0/3238639990 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd2f0004e40 0x7fd2f004f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:29:07.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:07.507+0000 7fd303fff700 1 --2- 172.21.15.42:0/3238639990 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd2f0003940 0x7fd2f0003d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:29:07.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:07.507+0000 7fd2fb7fe700 1 -- 172.21.15.42:0/3238639990 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd2f0003940 msgr2=0x7fd2f0003d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:29:07.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:07.507+0000 7fd2fb7fe700 1 --2- 172.21.15.42:0/3238639990 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd2f0003940 0x7fd2f0003d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:29:07.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:07.507+0000 7fd2fb7fe700 1 -- 172.21.15.42:0/3238639990 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd2f0004e40 msgr2=0x7fd2f004f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:29:07.530 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:07.507+0000 7fd2fb7fe700 1 --2- 172.21.15.42:0/3238639990 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd2f0004e40 0x7fd2f004f680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:29:07.530 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:07.507+0000 7fd2fb7fe700 1 -- 172.21.15.42:0/3238639990 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2f0050140 con 0x7fd2f0004420 2019-12-05T05:29:07.530 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:07.507+0000 7fd2fb7fe700 1 --2- 172.21.15.42:0/3238639990 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd2f0004420 0x7fd2f0004850 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fd2ec002f30 tx=0x7fd2ec00ffd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:07.530 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:07.507+0000 7fd2fb7fe700 1 --2- 172.21.15.42:0/3238639990 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd2f0004420 0x7fd2f0004850 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fd2ec002f30 tx=0x7fd2ec00ffd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:07.530 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:07.507+0000 7fd2fb7fe700 1 --2- 172.21.15.42:0/3238639990 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd2f0004420 0x7fd2f0004850 secure :-1 s=READY pgs=942 cs=0 l=1 rx=0x7fd2ec002f30 tx=0x7fd2ec00ffd0).ready entity=mon.1 client_cookie=3e0280c3b49e4790 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:29:07.530 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:07.507+0000 7fd2fb7fe700 1 --2- 172.21.15.42:0/3238639990 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd2f0004420 0x7fd2f0004850 secure :-1 s=THROTTLE_DONE pgs=942 cs=0 l=1 rx=0x7fd2ec002f30 tx=0x7fd2ec00ffd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:07.530 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:07.507+0000 7fd3037fe700 1 -- 172.21.15.42:0/3238639990 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd2ec010da0 con 0x7fd2f0004420 2019-12-05T05:29:07.531 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:07.507+0000 7fd2fb7fe700 1 --2- 172.21.15.42:0/3238639990 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd2f0004420 0x7fd2f0004850 secure :-1 s=THROTTLE_DONE pgs=942 cs=0 l=1 rx=0x7fd2ec002f30 tx=0x7fd2ec00ffd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:07.531 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:07.507+0000 7fd3037fe700 1 -- 172.21.15.42:0/3238639990 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2ec0109c0 con 0x7fd2f0004420 2019-12-05T05:29:07.531 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:07.507+0000 7fd2fb7fe700 1 --2- 172.21.15.42:0/3238639990 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd2f0004420 0x7fd2f0004850 secure :-1 s=THROTTLE_DONE pgs=942 cs=0 l=1 rx=0x7fd2ec002f30 tx=0x7fd2ec00ffd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:07.531 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:07.507+0000 7fd309cf4700 1 -- 172.21.15.42:0/3238639990 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd2f0004420 msgr2=0x7fd2f0004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:29:07.531 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:07.507+0000 7fd3037fe700 1 -- 172.21.15.42:0/3238639990 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd2ec008bb0 con 0x7fd2f0004420 2019-12-05T05:29:07.531 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:07.507+0000 7fd309cf4700 1 --2- 172.21.15.42:0/3238639990 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd2f0004420 0x7fd2f0004850 secure :-1 s=READY pgs=942 cs=0 l=1 rx=0x7fd2ec002f30 tx=0x7fd2ec00ffd0).stop 2019-12-05T05:29:07.531 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:07.507+0000 7fd308a92700 1 -- 172.21.15.42:0/3238639990 reap_dead start 2019-12-05T05:29:07.532 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:07.507+0000 7fd309cf4700 1 -- 172.21.15.42:0/3238639990 shutdown_connections 2019-12-05T05:29:07.532 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:07.507+0000 7fd309cf4700 1 -- 172.21.15.42:0/3238639990 shutdown_connections 2019-12-05T05:29:07.532 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:07.507+0000 7fd309cf4700 1 -- 172.21.15.42:0/3238639990 wait complete. 2019-12-05T05:29:07.532 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:07.507+0000 7fd309cf4700 1 -- 172.21.15.42:0/3238639990 >> 172.21.15.42:0/3238639990 conn(0x7fd3040aabe0 msgr2=0x7fd3040a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:29:07.553 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:07.511+0000 7fd309cf4700 1 Processor -- start 2019-12-05T05:29:07.554 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:07.511+0000 7fd309cf4700 1 -- start start 2019-12-05T05:29:07.554 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:07.511+0000 7fd309cf4700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd304144cc0 0x7fd3040722c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:29:07.554 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:07.511+0000 7fd309cf4700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd304072810 0x7fd304074c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:29:07.554 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:07.511+0000 7fd309cf4700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3041454a0 con 0x7fd3040a4e60 2019-12-05T05:29:07.554 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:07.511+0000 7fd309cf4700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd304143b70 con 0x7fd304144cc0 2019-12-05T05:29:07.554 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:07.511+0000 7fd309cf4700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3040a4840 con 0x7fd304072810 2019-12-05T05:29:07.555 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:07.511+0000 7fd2fb7fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd304144cc0 0x7fd3040722c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:29:07.555 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:07.511+0000 7fd2fb7fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd304144cc0 0x7fd3040722c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:34694/0 (socket says 172.21.15.42:34694) 2019-12-05T05:29:07.555 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:07.511+0000 7fd2fb7fe700 1 -- 172.21.15.42:0/3878378330 learned_addr learned my addr 172.21.15.42:0/3878378330 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:29:07.555 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:07.511+0000 7fd308a92700 1 --2- 172.21.15.42:0/3878378330 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd304072810 0x7fd304074c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:29:07.555 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:07.515+0000 7fd2fb7fe700 1 -- 172.21.15.42:0/3878378330 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd304072810 msgr2=0x7fd304074c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:29:07.555 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:07.515+0000 7fd2fb7fe700 1 --2- 172.21.15.42:0/3878378330 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd304072810 0x7fd304074c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:29:07.555 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:07.515+0000 7fd2fb7fe700 1 -- 172.21.15.42:0/3878378330 >> v1:172.21.15.40:6789/0 conn(0x7fd3040a4e60 legacy=0x7fd304143dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:29:07.556 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:07.515+0000 7fd2fb7fe700 1 -- 172.21.15.42:0/3878378330 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd304142a40 con 0x7fd304144cc0 2019-12-05T05:29:07.556 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:07.515+0000 7fd2fb7fe700 1 --2- 172.21.15.42:0/3878378330 >> v2:172.21.15.42:3300/0 conn(0x7fd304144cc0 0x7fd3040722c0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fd304144a70 tx=0x7fd2ec010c80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:07.556 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:07.515+0000 7fd2fb7fe700 1 --2- 172.21.15.42:0/3878378330 >> v2:172.21.15.42:3300/0 conn(0x7fd304144cc0 0x7fd3040722c0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fd304144a70 tx=0x7fd2ec010c80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:07.556 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:07.515+0000 7fd2fb7fe700 1 --2- 172.21.15.42:0/3878378330 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd304144cc0 0x7fd3040722c0 secure :-1 s=READY pgs=943 cs=0 l=1 rx=0x7fd304144a70 tx=0x7fd2ec010c80).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:29:07.564 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:07.515+0000 7fd2fb7fe700 1 --2- 172.21.15.42:0/3878378330 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd304144cc0 0x7fd3040722c0 secure :-1 s=THROTTLE_DONE pgs=943 cs=0 l=1 rx=0x7fd304144a70 tx=0x7fd2ec010c80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:07.564 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:07.515+0000 7fd3017fa700 1 -- 172.21.15.42:0/3878378330 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd2ec008dc0 con 0x7fd304144cc0 2019-12-05T05:29:07.564 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:07.515+0000 7fd2fb7fe700 1 --2- 172.21.15.42:0/3878378330 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd304144cc0 0x7fd3040722c0 secure :-1 s=THROTTLE_DONE pgs=943 cs=0 l=1 rx=0x7fd304144a70 tx=0x7fd2ec010c80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:07.564 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:07.515+0000 7fd3017fa700 1 -- 172.21.15.42:0/3878378330 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2ec0087d0 con 0x7fd304144cc0 2019-12-05T05:29:07.564 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:07.515+0000 7fd2fb7fe700 1 --2- 172.21.15.42:0/3878378330 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd304144cc0 0x7fd3040722c0 secure :-1 s=THROTTLE_DONE pgs=943 cs=0 l=1 rx=0x7fd304144a70 tx=0x7fd2ec010c80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:07.565 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:07.515+0000 7fd309cf4700 1 -- 172.21.15.42:0/3878378330 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3041445d0 con 0x7fd304144cc0 2019-12-05T05:29:07.565 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:07.515+0000 7fd3017fa700 1 -- 172.21.15.42:0/3878378330 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd2ec00f5d0 con 0x7fd304144cc0 2019-12-05T05:29:07.565 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:07.515+0000 7fd309cf4700 1 -- 172.21.15.42:0/3878378330 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3040751a0 con 0x7fd304144cc0 2019-12-05T05:29:07.565 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:07.515+0000 7fd2fb7fe700 1 --2- 172.21.15.42:0/3878378330 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd304144cc0 0x7fd3040722c0 secure :-1 s=THROTTLE_DONE pgs=943 cs=0 l=1 rx=0x7fd304144a70 tx=0x7fd2ec010c80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:07.565 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:07.515+0000 7fd3017fa700 1 -- 172.21.15.42:0/3878378330 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fd2ec005540 con 0x7fd304144cc0 2019-12-05T05:29:07.565 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:07.515+0000 7fd2fb7fe700 1 --2- 172.21.15.42:0/3878378330 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd304144cc0 0x7fd3040722c0 secure :-1 s=THROTTLE_DONE pgs=943 cs=0 l=1 rx=0x7fd304144a70 tx=0x7fd2ec010c80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:07.565 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:07.515+0000 7fd3017fa700 1 --2- 172.21.15.42:0/3878378330 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd2e803acc0 0x7fd2e803d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:29:07.566 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:07.515+0000 7fd3017fa700 1 -- 172.21.15.42:0/3878378330 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 7493+0+0 (secure 0 0 0) 0x7fd2ec04adc0 con 0x7fd304144cc0 2019-12-05T05:29:07.566 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:07.515+0000 7fd303fff700 1 --2- 172.21.15.42:0/3878378330 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd2e803acc0 0x7fd2e803d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:29:07.566 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:07.519+0000 7fd303fff700 1 --2- 172.21.15.42:0/3878378330 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd2e803acc0 0x7fd2e803d1d0 crc :-1 s=READY pgs=410 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:29:07.566 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:07.519+0000 7fd309cf4700 1 -- 172.21.15.42:0/3878378330 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd30405d110 con 0x7fd304144cc0 2019-12-05T05:29:07.566 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:07.527+0000 7fd2fb7fe700 1 --2- 172.21.15.42:0/3878378330 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd304144cc0 0x7fd3040722c0 secure :-1 s=THROTTLE_DONE pgs=943 cs=0 l=1 rx=0x7fd304144a70 tx=0x7fd2ec010c80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:07.566 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:07.527+0000 7fd3017fa700 1 -- 172.21.15.42:0/3878378330 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fd2ec00f870 con 0x7fd304144cc0 2019-12-05T05:29:07.702 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:07.695+0000 7fd309cf4700 1 -- 172.21.15.42:0/3878378330 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd304065a80 con 0x7fd304144cc0 2019-12-05T05:29:07.703 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:07.695+0000 7fd2fb7fe700 1 --2- 172.21.15.42:0/3878378330 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd304144cc0 0x7fd3040722c0 secure :-1 s=THROTTLE_DONE pgs=943 cs=0 l=1 rx=0x7fd304144a70 tx=0x7fd2ec010c80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:07.703 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:07.695+0000 7fd3017fa700 1 -- 172.21.15.42:0/3878378330 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 118 v118) v1 ==== 95+0+2112 (secure 0 0 0) 0x7fd2ec032080 con 0x7fd304144cc0 2019-12-05T05:29:07.703 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 118 2019-12-05T05:29:07.703 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:29:07.704 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":118,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5426,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/3168718372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3168718372},{"type":"v1","addr":"172.21.15.40:6833","nonce":3168718372}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":118}],"filesystems":[{"mdsmap":{"epoch":117,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:28:18.130626+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6666},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6666":{"gid":6666,"name":"a","rank":0,"incarnation":114,"state":"up:active","state_seq":17,"addr":"172.21.15.42:6835/2544447110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2544447110},{"type":"v1","addr":"172.21.15.42:6835","nonce":2544447110}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:29:07.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:07.699+0000 7fd309cf4700 1 -- 172.21.15.42:0/3878378330 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd2e803acc0 msgr2=0x7fd2e803d1d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:29:07.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:07.699+0000 7fd309cf4700 1 --2- 172.21.15.42:0/3878378330 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd2e803acc0 0x7fd2e803d1d0 crc :-1 s=READY pgs=410 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:29:07.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:07.699+0000 7fd309cf4700 1 -- 172.21.15.42:0/3878378330 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd304144cc0 msgr2=0x7fd3040722c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:29:07.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:07.699+0000 7fd309cf4700 1 --2- 172.21.15.42:0/3878378330 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd304144cc0 0x7fd3040722c0 secure :-1 s=READY pgs=943 cs=0 l=1 rx=0x7fd304144a70 tx=0x7fd2ec010c80).stop 2019-12-05T05:29:07.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:07.699+0000 7fd309cf4700 1 -- 172.21.15.42:0/3878378330 shutdown_connections 2019-12-05T05:29:07.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:07.699+0000 7fd309cf4700 1 --2- 172.21.15.42:0/3878378330 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd2e803acc0 0x7fd2e803d1d0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:29:07.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:07.699+0000 7fd309cf4700 1 --2- 172.21.15.42:0/3878378330 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd304072810 0x7fd304074c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:29:07.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:07.699+0000 7fd309cf4700 1 --2- 172.21.15.42:0/3878378330 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd304144cc0 0x7fd3040722c0 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:29:07.708 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:07.699+0000 7fd309cf4700 1 -- 172.21.15.42:0/3878378330 shutdown_connections 2019-12-05T05:29:07.708 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:07.699+0000 7fd309cf4700 1 -- 172.21.15.42:0/3878378330 wait complete. 2019-12-05T05:29:07.708 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:07.699+0000 7fd309cf4700 1 -- 172.21.15.42:0/3878378330 >> 172.21.15.42:0/3878378330 conn(0x7fd3040a7740 msgr2=0x7fd3040a6bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:29:07.715 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T05:29:08.777 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:29:08.781 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:29:09.716 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:29:09.830 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:09.815+0000 7f975c112700 1 Processor -- start 2019-12-05T05:29:09.830 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:09.815+0000 7f975c112700 1 -- start start 2019-12-05T05:29:09.831 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:09.819+0000 7f975c112700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f97540a7740 0x7f975408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:29:09.831 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:09.819+0000 7f975c112700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f975407e720 0x7f975407eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:29:09.831 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:09.819+0000 7f975c112700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97540a0710 con 0x7f975409f2e0 2019-12-05T05:29:09.831 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:09.819+0000 7f975c112700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97540a0b00 con 0x7f97540a7740 2019-12-05T05:29:09.831 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:09.819+0000 7f975c112700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f975408e5b0 con 0x7f975407e720 2019-12-05T05:29:09.831 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:09.819+0000 7f9759eae700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f975407e720 0x7f975407eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:29:09.831 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:09.819+0000 7f975a6af700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f97540a7740 0x7f975408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:29:09.832 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:09.819+0000 7f9759eae700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f975407e720 0x7f975407eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:34716/0 (socket says 172.21.15.42:34716) 2019-12-05T05:29:09.832 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:09.819+0000 7f9759eae700 1 -- 172.21.15.42:0/2674657513 learned_addr learned my addr 172.21.15.42:0/2674657513 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:29:09.832 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:09.819+0000 7f97596ad700 1 -- 172.21.15.42:0/2674657513 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2779820647 0 0) 0x7f97540a0710 con 0x7f975409f2e0 2019-12-05T05:29:09.832 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:09.819+0000 7f97596ad700 1 -- 172.21.15.42:0/2674657513 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97400030b0 con 0x7f975409f2e0 2019-12-05T05:29:09.832 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:09.819+0000 7f975a6af700 1 -- 172.21.15.42:0/2674657513 >> v2:172.21.15.42:3300/0 conn(0x7f975407e720 msgr2=0x7f975407eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:29:09.832 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:09.819+0000 7f975a6af700 1 --2- 172.21.15.42:0/2674657513 >> v2:172.21.15.42:3300/0 conn(0x7f975407e720 0x7f975407eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:29:09.832 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:09.819+0000 7f975a6af700 1 -- 172.21.15.42:0/2674657513 >> v1:172.21.15.42:6789/0 conn(0x7f975409f2e0 legacy=0x7f975409f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:29:09.833 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:09.819+0000 7f975a6af700 1 -- 172.21.15.42:0/2674657513 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f975407f0a0 con 0x7f97540a7740 2019-12-05T05:29:09.833 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:09.819+0000 7f975a6af700 1 --2- 172.21.15.42:0/2674657513 >> v2:172.21.15.40:3300/0 conn(0x7f97540a7740 0x7f975408df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f9750002f30 tx=0x7f975000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:09.833 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:09.819+0000 7f975a6af700 1 --2- 172.21.15.42:0/2674657513 >> v2:172.21.15.40:3300/0 conn(0x7f97540a7740 0x7f975408df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f9750002f30 tx=0x7f975000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:09.833 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:09.819+0000 7f975a6af700 1 --2- 172.21.15.42:0/2674657513 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f97540a7740 0x7f975408df80 secure :-1 s=READY pgs=737 cs=0 l=1 rx=0x7f9750002f30 tx=0x7f975000fc70).ready entity=mon.0 client_cookie=f1957ba6d37ffd37 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:29:09.833 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:09.819+0000 7f975a6af700 1 --2- 172.21.15.42:0/2674657513 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f97540a7740 0x7f975408df80 secure :-1 s=THROTTLE_DONE pgs=737 cs=0 l=1 rx=0x7f9750002f30 tx=0x7f975000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:09.833 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:09.819+0000 7f975a6af700 1 --2- 172.21.15.42:0/2674657513 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f97540a7740 0x7f975408df80 secure :-1 s=THROTTLE_DONE pgs=737 cs=0 l=1 rx=0x7f9750002f30 tx=0x7f975000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:09.833 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:09.819+0000 7f97596ad700 1 -- 172.21.15.42:0/2674657513 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f97500112b0 con 0x7f97540a7740 2019-12-05T05:29:09.834 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:09.819+0000 7f975a6af700 1 --2- 172.21.15.42:0/2674657513 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f97540a7740 0x7f975408df80 secure :-1 s=THROTTLE_DONE pgs=737 cs=0 l=1 rx=0x7f9750002f30 tx=0x7f975000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:09.834 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:09.819+0000 7f97596ad700 1 -- 172.21.15.42:0/2674657513 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9750008ad0 con 0x7f97540a7740 2019-12-05T05:29:09.834 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:09.819+0000 7f97596ad700 1 -- 172.21.15.42:0/2674657513 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9750011890 con 0x7f97540a7740 2019-12-05T05:29:09.834 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:09.819+0000 7f975c112700 1 -- 172.21.15.42:0/2674657513 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f97540a7740 msgr2=0x7f975408df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:29:09.834 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:09.819+0000 7f975c112700 1 --2- 172.21.15.42:0/2674657513 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f97540a7740 0x7f975408df80 secure :-1 s=READY pgs=737 cs=0 l=1 rx=0x7f9750002f30 tx=0x7f975000fc70).stop 2019-12-05T05:29:09.834 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:09.819+0000 7f975c112700 1 -- 172.21.15.42:0/2674657513 shutdown_connections 2019-12-05T05:29:09.834 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:09.819+0000 7f975c112700 1 --2- 172.21.15.42:0/2674657513 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f97540a7740 0x7f975408df80 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:29:09.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:09.819+0000 7f975c112700 1 --2- 172.21.15.42:0/2674657513 >> v2:172.21.15.42:3300/0 conn(0x7f975407e720 0x7f975407eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:29:09.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:09.819+0000 7f975c112700 1 -- 172.21.15.42:0/2674657513 shutdown_connections 2019-12-05T05:29:09.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:09.819+0000 7f975c112700 1 -- 172.21.15.42:0/2674657513 wait complete. 2019-12-05T05:29:09.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:09.819+0000 7f975c112700 1 -- 172.21.15.42:0/2674657513 >> 172.21.15.42:0/2674657513 conn(0x7f97540aabe0 msgr2=0x7f97540a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:29:09.839 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:09.823+0000 7f975c112700 1 Processor -- start 2019-12-05T05:29:09.840 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:09.823+0000 7f975c112700 1 -- start start 2019-12-05T05:29:09.840 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:09.827+0000 7f975c112700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f97541333a0 0x7f9754141c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:29:09.840 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:09.827+0000 7f975c112700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f97540a9770 0x7f97540762f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:29:09.840 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:09.827+0000 7f975c112700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97540a5a00 con 0x7f97540a55d0 2019-12-05T05:29:09.840 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:09.827+0000 7f975c112700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9754130600 con 0x7f97540a9770 2019-12-05T05:29:09.841 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:09.827+0000 7f975c112700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97541338b0 con 0x7f97541333a0 2019-12-05T05:29:09.841 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:09.827+0000 7f9759eae700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f97540a55d0 0x7f975412fe00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:34758/0 (socket says 172.21.15.42:34758) 2019-12-05T05:29:09.841 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:09.827+0000 7f9759eae700 1 -- 172.21.15.42:0/2825397112 learned_addr learned my addr 172.21.15.42:0/2825397112 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:29:09.841 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:09.827+0000 7f975aeb0700 1 --2- 172.21.15.42:0/2825397112 >> v2:172.21.15.42:3300/0 conn(0x7f97540a9770 0x7f97540762f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:29:09.841 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:09.827+0000 7f975a6af700 1 --2- 172.21.15.42:0/2825397112 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f97541333a0 0x7f9754141c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:29:09.841 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:09.827+0000 7f9746ffd700 1 -- 172.21.15.42:0/2825397112 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3223135221 0 0) 0x7f9748007f50 con 0x7f97540a55d0 2019-12-05T05:29:09.841 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:09.827+0000 7f9746ffd700 1 -- 172.21.15.42:0/2825397112 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f973c0030b0 con 0x7f97540a55d0 2019-12-05T05:29:09.842 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:09.827+0000 7f975aeb0700 1 -- 172.21.15.42:0/2825397112 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f97541333a0 msgr2=0x7f9754141c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:29:09.842 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:09.827+0000 7f975aeb0700 1 --2- 172.21.15.42:0/2825397112 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f97541333a0 0x7f9754141c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:29:09.842 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:09.827+0000 7f975aeb0700 1 -- 172.21.15.42:0/2825397112 >> v1:172.21.15.42:6789/0 conn(0x7f97540a55d0 legacy=0x7f975412fe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:29:09.842 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:09.827+0000 7f9746ffd700 1 -- 172.21.15.42:0/2825397112 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 172827978 0 0) 0x7f973c0030b0 con 0x7f97540a55d0 2019-12-05T05:29:09.842 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:09.827+0000 7f975aeb0700 1 -- 172.21.15.42:0/2825397112 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9754134090 con 0x7f97540a9770 2019-12-05T05:29:09.843 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:09.827+0000 7f975a6af700 1 --2- 172.21.15.42:0/2825397112 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f97541333a0 0x7f9754141c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:29:09.843 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:09.827+0000 7f975aeb0700 1 --2- 172.21.15.42:0/2825397112 >> v2:172.21.15.42:3300/0 conn(0x7f97540a9770 0x7f97540762f0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f97480012c0 tx=0x7f9748002f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:09.843 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:09.827+0000 7f975aeb0700 1 --2- 172.21.15.42:0/2825397112 >> v2:172.21.15.42:3300/0 conn(0x7f97540a9770 0x7f97540762f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f97480012c0 tx=0x7f9748002f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:09.843 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:09.827+0000 7f975aeb0700 1 --2- 172.21.15.42:0/2825397112 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f97540a9770 0x7f97540762f0 secure :-1 s=READY pgs=947 cs=0 l=1 rx=0x7f97480012c0 tx=0x7f9748002f70).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:29:09.843 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:09.827+0000 7f975aeb0700 1 --2- 172.21.15.42:0/2825397112 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f97540a9770 0x7f97540762f0 secure :-1 s=THROTTLE_DONE pgs=947 cs=0 l=1 rx=0x7f97480012c0 tx=0x7f9748002f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:09.843 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:09.827+0000 7f9746ffd700 1 -- 172.21.15.42:0/2825397112 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9748001080 con 0x7f97540a9770 2019-12-05T05:29:09.843 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:09.827+0000 7f975aeb0700 1 --2- 172.21.15.42:0/2825397112 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f97540a9770 0x7f97540762f0 secure :-1 s=THROTTLE_DONE pgs=947 cs=0 l=1 rx=0x7f97480012c0 tx=0x7f9748002f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:09.844 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:09.827+0000 7f975aeb0700 1 --2- 172.21.15.42:0/2825397112 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f97540a9770 0x7f97540762f0 secure :-1 s=THROTTLE_DONE pgs=947 cs=0 l=1 rx=0x7f97480012c0 tx=0x7f9748002f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:09.844 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:09.827+0000 7f9746ffd700 1 -- 172.21.15.42:0/2825397112 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9748000e90 con 0x7f97540a9770 2019-12-05T05:29:09.844 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:09.827+0000 7f975c112700 1 -- 172.21.15.42:0/2825397112 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97540a9ba0 con 0x7f97540a9770 2019-12-05T05:29:09.844 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:09.827+0000 7f9746ffd700 1 -- 172.21.15.42:0/2825397112 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f974c006cb0 con 0x7f97540a9770 2019-12-05T05:29:09.844 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:09.827+0000 7f975c112700 1 -- 172.21.15.42:0/2825397112 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9754076840 con 0x7f97540a9770 2019-12-05T05:29:09.844 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:09.827+0000 7f975aeb0700 1 --2- 172.21.15.42:0/2825397112 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f97540a9770 0x7f97540762f0 secure :-1 s=THROTTLE_DONE pgs=947 cs=0 l=1 rx=0x7f97480012c0 tx=0x7f9748002f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:09.845 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:09.827+0000 7f9746ffd700 1 -- 172.21.15.42:0/2825397112 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f974c012460 con 0x7f97540a9770 2019-12-05T05:29:09.845 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:09.827+0000 7f975aeb0700 1 --2- 172.21.15.42:0/2825397112 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f97540a9770 0x7f97540762f0 secure :-1 s=THROTTLE_DONE pgs=947 cs=0 l=1 rx=0x7f97480012c0 tx=0x7f9748002f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:09.845 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:09.831+0000 7f9746ffd700 1 --2- 172.21.15.42:0/2825397112 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f973c03b210 0x7f973c03d720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:29:09.845 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:09.831+0000 7f9746ffd700 1 -- 172.21.15.42:0/2825397112 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 7493+0+0 (secure 0 0 0) 0x7f974c04a9d0 con 0x7f97540a9770 2019-12-05T05:29:09.845 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:09.831+0000 7f975a6af700 1 --2- 172.21.15.42:0/2825397112 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f973c03b210 0x7f973c03d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:29:09.845 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:09.831+0000 7f975a6af700 1 --2- 172.21.15.42:0/2825397112 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f973c03b210 0x7f973c03d720 crc :-1 s=READY pgs=411 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:29:09.845 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:09.831+0000 7f975c112700 1 -- 172.21.15.42:0/2825397112 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f975405d110 con 0x7f97540a9770 2019-12-05T05:29:09.847 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:09.839+0000 7f975aeb0700 1 --2- 172.21.15.42:0/2825397112 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f97540a9770 0x7f97540762f0 secure :-1 s=THROTTLE_DONE pgs=947 cs=0 l=1 rx=0x7f97480012c0 tx=0x7f9748002f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:09.847 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:09.839+0000 7f9746ffd700 1 -- 172.21.15.42:0/2825397112 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f974c031a30 con 0x7f97540a9770 2019-12-05T05:29:10.014 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:10.007+0000 7f975c112700 1 -- 172.21.15.42:0/2825397112 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f975407e180 con 0x7f97540a9770 2019-12-05T05:29:10.015 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:10.007+0000 7f975aeb0700 1 --2- 172.21.15.42:0/2825397112 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f97540a9770 0x7f97540762f0 secure :-1 s=THROTTLE_DONE pgs=947 cs=0 l=1 rx=0x7f97480012c0 tx=0x7f9748002f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:10.015 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:10.007+0000 7f9746ffd700 1 -- 172.21.15.42:0/2825397112 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 118 v118) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f974c031c90 con 0x7f97540a9770 2019-12-05T05:29:10.015 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 118 2019-12-05T05:29:10.016 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:29:10.016 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":118,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5426,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/3168718372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3168718372},{"type":"v1","addr":"172.21.15.40:6833","nonce":3168718372}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":118}],"filesystems":[{"mdsmap":{"epoch":117,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:28:18.130626+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6666},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6666":{"gid":6666,"name":"a","rank":0,"incarnation":114,"state":"up:active","state_seq":17,"addr":"172.21.15.42:6835/2544447110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2544447110},{"type":"v1","addr":"172.21.15.42:6835","nonce":2544447110}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:29:10.022 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:10.011+0000 7f975c112700 1 -- 172.21.15.42:0/2825397112 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f973c03b210 msgr2=0x7f973c03d720 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:29:10.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:10.011+0000 7f975c112700 1 --2- 172.21.15.42:0/2825397112 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f973c03b210 0x7f973c03d720 crc :-1 s=READY pgs=411 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:29:10.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:10.011+0000 7f975c112700 1 -- 172.21.15.42:0/2825397112 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f97540a9770 msgr2=0x7f97540762f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:29:10.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:10.011+0000 7f975c112700 1 --2- 172.21.15.42:0/2825397112 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f97540a9770 0x7f97540762f0 secure :-1 s=READY pgs=947 cs=0 l=1 rx=0x7f97480012c0 tx=0x7f9748002f70).stop 2019-12-05T05:29:10.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:10.011+0000 7f975c112700 1 -- 172.21.15.42:0/2825397112 shutdown_connections 2019-12-05T05:29:10.024 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:10.011+0000 7f975c112700 1 --2- 172.21.15.42:0/2825397112 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f973c03b210 0x7f973c03d720 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:29:10.024 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:10.011+0000 7f975c112700 1 --2- 172.21.15.42:0/2825397112 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f97540a9770 0x7f97540762f0 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:29:10.024 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:10.011+0000 7f975c112700 1 --2- 172.21.15.42:0/2825397112 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f97541333a0 0x7f9754141c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:29:10.024 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:10.011+0000 7f975c112700 1 -- 172.21.15.42:0/2825397112 shutdown_connections 2019-12-05T05:29:10.024 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:10.011+0000 7f975c112700 1 -- 172.21.15.42:0/2825397112 wait complete. 2019-12-05T05:29:10.024 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:10.011+0000 7f975c112700 1 -- 172.21.15.42:0/2825397112 >> 172.21.15.42:0/2825397112 conn(0x7f975408fda0 msgr2=0x7f975411bb90 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:29:10.028 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T05:29:12.029 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:29:12.144 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:12.127+0000 7f460f416700 1 Processor -- start 2019-12-05T05:29:12.148 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:12.127+0000 7f460f416700 1 -- start start 2019-12-05T05:29:12.148 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:12.127+0000 7f460f416700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f460808df80 0x7f460809f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:29:12.148 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:12.127+0000 7f460f416700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f460809f830 0x7f460814efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:29:12.148 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:12.127+0000 7f460f416700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f460809fc60 con 0x7f46080a7740 2019-12-05T05:29:12.149 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:12.127+0000 7f460f416700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46080a0b00 con 0x7f460809f830 2019-12-05T05:29:12.149 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:12.127+0000 7f460f416700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f460808e5b0 con 0x7f460808df80 2019-12-05T05:29:12.149 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:12.127+0000 7f460d9b3700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f46080a7740 0x7f460807e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:34766/0 (socket says 172.21.15.42:34766) 2019-12-05T05:29:12.149 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:12.127+0000 7f460d9b3700 1 -- 172.21.15.42:0/68890653 learned_addr learned my addr 172.21.15.42:0/68890653 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:29:12.150 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:12.127+0000 7f460d1b2700 1 --2- 172.21.15.42:0/68890653 >> v2:172.21.15.42:3300/0 conn(0x7f460808df80 0x7f460809f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:29:12.150 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:12.127+0000 7f460e1b4700 1 --2- 172.21.15.42:0/68890653 >> v2:172.21.15.40:3300/0 conn(0x7f460809f830 0x7f460814efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:29:12.151 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:12.127+0000 7f460d1b2700 1 -- 172.21.15.42:0/68890653 >> v2:172.21.15.40:3300/0 conn(0x7f460809f830 msgr2=0x7f460814efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:29:12.151 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:12.127+0000 7f460d1b2700 1 --2- 172.21.15.42:0/68890653 >> v2:172.21.15.40:3300/0 conn(0x7f460809f830 0x7f460814efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:29:12.151 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:12.127+0000 7f460d1b2700 1 -- 172.21.15.42:0/68890653 >> v1:172.21.15.42:6789/0 conn(0x7f46080a7740 legacy=0x7f460807e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:29:12.151 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:12.127+0000 7f460d1b2700 1 -- 172.21.15.42:0/68890653 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46080a0710 con 0x7f460808df80 2019-12-05T05:29:12.152 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:12.127+0000 7f460d1b2700 1 --2- 172.21.15.42:0/68890653 >> v2:172.21.15.42:3300/0 conn(0x7f460808df80 0x7f460809f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f46000032b0 tx=0x7f460000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:12.152 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:12.127+0000 7f460d1b2700 1 --2- 172.21.15.42:0/68890653 >> v2:172.21.15.42:3300/0 conn(0x7f460808df80 0x7f460809f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f46000032b0 tx=0x7f460000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:12.152 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:12.127+0000 7f460d1b2700 1 --2- 172.21.15.42:0/68890653 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f460808df80 0x7f460809f2e0 secure :-1 s=READY pgs=949 cs=0 l=1 rx=0x7f46000032b0 tx=0x7f460000fff0).ready entity=mon.1 client_cookie=2e85448ba49bb17 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:29:12.152 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:12.127+0000 7f460d1b2700 1 --2- 172.21.15.42:0/68890653 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f460808df80 0x7f460809f2e0 secure :-1 s=THROTTLE_DONE pgs=949 cs=0 l=1 rx=0x7f46000032b0 tx=0x7f460000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:12.152 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:12.127+0000 7f460d1b2700 1 --2- 172.21.15.42:0/68890653 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f460808df80 0x7f460809f2e0 secure :-1 s=THROTTLE_DONE pgs=949 cs=0 l=1 rx=0x7f46000032b0 tx=0x7f460000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:12.153 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:12.127+0000 7f460d1b2700 1 --2- 172.21.15.42:0/68890653 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f460808df80 0x7f460809f2e0 secure :-1 s=THROTTLE_DONE pgs=949 cs=0 l=1 rx=0x7f46000032b0 tx=0x7f460000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:12.153 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:12.127+0000 7f460c9b1700 1 -- 172.21.15.42:0/68890653 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f46000108c0 con 0x7f460808df80 2019-12-05T05:29:12.153 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:12.127+0000 7f460c9b1700 1 -- 172.21.15.42:0/68890653 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4600008b50 con 0x7f460808df80 2019-12-05T05:29:12.153 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:12.127+0000 7f460c9b1700 1 -- 172.21.15.42:0/68890653 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f46000110c0 con 0x7f460808df80 2019-12-05T05:29:12.153 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:12.127+0000 7f460f416700 1 -- 172.21.15.42:0/68890653 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f460808df80 msgr2=0x7f460809f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:29:12.153 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:12.127+0000 7f460f416700 1 --2- 172.21.15.42:0/68890653 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f460808df80 0x7f460809f2e0 secure :-1 s=READY pgs=949 cs=0 l=1 rx=0x7f46000032b0 tx=0x7f460000fff0).stop 2019-12-05T05:29:12.153 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:12.131+0000 7f460f416700 1 -- 172.21.15.42:0/68890653 shutdown_connections 2019-12-05T05:29:12.154 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:12.131+0000 7f460f416700 1 --2- 172.21.15.42:0/68890653 >> v2:172.21.15.40:3300/0 conn(0x7f460809f830 0x7f460814efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:29:12.154 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:12.131+0000 7f460f416700 1 --2- 172.21.15.42:0/68890653 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f460808df80 0x7f460809f2e0 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:29:12.154 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:12.131+0000 7f460f416700 1 -- 172.21.15.42:0/68890653 shutdown_connections 2019-12-05T05:29:12.154 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:12.131+0000 7f460f416700 1 -- 172.21.15.42:0/68890653 wait complete. 2019-12-05T05:29:12.154 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:12.131+0000 7f460f416700 1 -- 172.21.15.42:0/68890653 >> 172.21.15.42:0/68890653 conn(0x7f46080aabe0 msgr2=0x7f46080a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:29:12.154 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:12.131+0000 7f460f416700 1 Processor -- start 2019-12-05T05:29:12.154 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:12.131+0000 7f460f416700 1 -- start start 2019-12-05T05:29:12.155 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:12.135+0000 7f460f416700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f46080a6380 0x7f46080a67b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:29:12.155 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:12.135+0000 7f460f416700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f460811a8f0 0x7f460811ad20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:29:12.155 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:12.135+0000 7f460f416700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46080a6d00 con 0x7f46080a5690 2019-12-05T05:29:12.155 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:12.135+0000 7f460f416700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46080a5ba0 con 0x7f46080a6380 2019-12-05T05:29:12.155 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:12.135+0000 7f460f416700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46080a5e40 con 0x7f460811a8f0 2019-12-05T05:29:12.155 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:12.135+0000 7f460d9b3700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f46080a5690 0x7f460811c6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:34772/0 (socket says 172.21.15.42:34772) 2019-12-05T05:29:12.155 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:12.135+0000 7f460d9b3700 1 -- 172.21.15.42:0/3269885956 learned_addr learned my addr 172.21.15.42:0/3269885956 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:29:12.156 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:12.135+0000 7f460e1b4700 1 --2- 172.21.15.42:0/3269885956 >> v2:172.21.15.42:3300/0 conn(0x7f460811a8f0 0x7f460811ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:29:12.156 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:12.135+0000 7f460d1b2700 1 --2- 172.21.15.42:0/3269885956 >> v2:172.21.15.40:3300/0 conn(0x7f46080a6380 0x7f46080a67b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:29:12.156 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:12.135+0000 7f46067fc700 1 -- 172.21.15.42:0/3269885956 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1814428715 0 0) 0x7f46080a6d00 con 0x7f46080a5690 2019-12-05T05:29:12.156 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:12.135+0000 7f46067fc700 1 -- 172.21.15.42:0/3269885956 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45e80030b0 con 0x7f46080a5690 2019-12-05T05:29:12.156 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:12.135+0000 7f460e1b4700 1 -- 172.21.15.42:0/3269885956 >> v2:172.21.15.40:3300/0 conn(0x7f46080a6380 msgr2=0x7f46080a67b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:29:12.156 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:12.135+0000 7f460e1b4700 1 --2- 172.21.15.42:0/3269885956 >> v2:172.21.15.40:3300/0 conn(0x7f46080a6380 0x7f46080a67b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:29:12.156 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:12.135+0000 7f460e1b4700 1 -- 172.21.15.42:0/3269885956 >> v1:172.21.15.42:6789/0 conn(0x7f46080a5690 legacy=0x7f460811c6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:29:12.157 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:12.135+0000 7f460e1b4700 1 -- 172.21.15.42:0/3269885956 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46080a9770 con 0x7f460811a8f0 2019-12-05T05:29:12.157 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:12.135+0000 7f460e1b4700 1 --2- 172.21.15.42:0/3269885956 >> v2:172.21.15.42:3300/0 conn(0x7f460811a8f0 0x7f460811ad20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f45fc002f30 tx=0x7f45fc0108c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:12.157 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:12.135+0000 7f460e1b4700 1 --2- 172.21.15.42:0/3269885956 >> v2:172.21.15.42:3300/0 conn(0x7f460811a8f0 0x7f460811ad20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f45fc002f30 tx=0x7f45fc0108c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:12.157 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:12.135+0000 7f460e1b4700 1 --2- 172.21.15.42:0/3269885956 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f460811a8f0 0x7f460811ad20 secure :-1 s=READY pgs=951 cs=0 l=1 rx=0x7f45fc002f30 tx=0x7f45fc0108c0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:29:12.157 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:12.135+0000 7f460e1b4700 1 --2- 172.21.15.42:0/3269885956 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f460811a8f0 0x7f460811ad20 secure :-1 s=THROTTLE_DONE pgs=951 cs=0 l=1 rx=0x7f45fc002f30 tx=0x7f45fc0108c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:12.158 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:12.135+0000 7f46067fc700 1 -- 172.21.15.42:0/3269885956 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f45fc005930 con 0x7f460811a8f0 2019-12-05T05:29:12.158 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:12.135+0000 7f460e1b4700 1 --2- 172.21.15.42:0/3269885956 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f460811a8f0 0x7f460811ad20 secure :-1 s=THROTTLE_DONE pgs=951 cs=0 l=1 rx=0x7f45fc002f30 tx=0x7f45fc0108c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:12.158 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:12.135+0000 7f46067fc700 1 -- 172.21.15.42:0/3269885956 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45fc007160 con 0x7f460811a8f0 2019-12-05T05:29:12.158 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:12.135+0000 7f460e1b4700 1 --2- 172.21.15.42:0/3269885956 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f460811a8f0 0x7f460811ad20 secure :-1 s=THROTTLE_DONE pgs=951 cs=0 l=1 rx=0x7f45fc002f30 tx=0x7f45fc0108c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:12.158 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:12.135+0000 7f460f416700 1 -- 172.21.15.42:0/3269885956 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46080a9a40 con 0x7f460811a8f0 2019-12-05T05:29:12.158 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:12.135+0000 7f46067fc700 1 -- 172.21.15.42:0/3269885956 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f45fc0060c0 con 0x7f460811a8f0 2019-12-05T05:29:12.159 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:12.135+0000 7f460f416700 1 -- 172.21.15.42:0/3269885956 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f460811b270 con 0x7f460811a8f0 2019-12-05T05:29:12.159 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:12.139+0000 7f460e1b4700 1 --2- 172.21.15.42:0/3269885956 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f460811a8f0 0x7f460811ad20 secure :-1 s=THROTTLE_DONE pgs=951 cs=0 l=1 rx=0x7f45fc002f30 tx=0x7f45fc0108c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:12.161 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:12.139+0000 7f46067fc700 1 -- 172.21.15.42:0/3269885956 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f45fc007ae0 con 0x7f460811a8f0 2019-12-05T05:29:12.161 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:12.139+0000 7f460e1b4700 1 --2- 172.21.15.42:0/3269885956 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f460811a8f0 0x7f460811ad20 secure :-1 s=THROTTLE_DONE pgs=951 cs=0 l=1 rx=0x7f45fc002f30 tx=0x7f45fc0108c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:12.161 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:12.139+0000 7f46067fc700 1 --2- 172.21.15.42:0/3269885956 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f45e803b230 0x7f45e803d740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:29:12.161 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:12.139+0000 7f46067fc700 1 -- 172.21.15.42:0/3269885956 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 7493+0+0 (secure 0 0 0) 0x7f45fc04aa70 con 0x7f460811a8f0 2019-12-05T05:29:12.161 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:12.139+0000 7f460d9b3700 1 --2- 172.21.15.42:0/3269885956 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f45e803b230 0x7f45e803d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:29:12.162 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:12.139+0000 7f460d9b3700 1 --2- 172.21.15.42:0/3269885956 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f45e803b230 0x7f45e803d740 crc :-1 s=READY pgs=412 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:29:12.162 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:12.143+0000 7f460f416700 1 -- 172.21.15.42:0/3269885956 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f460805d110 con 0x7f460811a8f0 2019-12-05T05:29:12.162 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:12.147+0000 7f460e1b4700 1 --2- 172.21.15.42:0/3269885956 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f460811a8f0 0x7f460811ad20 secure :-1 s=THROTTLE_DONE pgs=951 cs=0 l=1 rx=0x7f45fc002f30 tx=0x7f45fc0108c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:12.162 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:12.147+0000 7f46067fc700 1 -- 172.21.15.42:0/3269885956 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f45fc031ad0 con 0x7f460811a8f0 2019-12-05T05:29:12.318 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:12.311+0000 7f460f416700 1 -- 172.21.15.42:0/3269885956 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f460805b8c0 con 0x7f460811a8f0 2019-12-05T05:29:12.320 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:12.311+0000 7f460e1b4700 1 --2- 172.21.15.42:0/3269885956 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f460811a8f0 0x7f460811ad20 secure :-1 s=THROTTLE_DONE pgs=951 cs=0 l=1 rx=0x7f45fc002f30 tx=0x7f45fc0108c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:12.320 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:12.311+0000 7f46067fc700 1 -- 172.21.15.42:0/3269885956 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 118 v118) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f45fc031d30 con 0x7f460811a8f0 2019-12-05T05:29:12.320 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 118 2019-12-05T05:29:12.320 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:29:12.320 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":118,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5426,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/3168718372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3168718372},{"type":"v1","addr":"172.21.15.40:6833","nonce":3168718372}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":118}],"filesystems":[{"mdsmap":{"epoch":117,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:28:18.130626+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6666},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6666":{"gid":6666,"name":"a","rank":0,"incarnation":114,"state":"up:active","state_seq":17,"addr":"172.21.15.42:6835/2544447110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2544447110},{"type":"v1","addr":"172.21.15.42:6835","nonce":2544447110}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:29:12.321 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:12.315+0000 7f460f416700 1 -- 172.21.15.42:0/3269885956 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f45e803b230 msgr2=0x7f45e803d740 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:29:12.321 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:12.315+0000 7f460f416700 1 --2- 172.21.15.42:0/3269885956 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f45e803b230 0x7f45e803d740 crc :-1 s=READY pgs=412 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:29:12.321 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:12.315+0000 7f460f416700 1 -- 172.21.15.42:0/3269885956 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f460811a8f0 msgr2=0x7f460811ad20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:29:12.321 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:12.315+0000 7f460f416700 1 --2- 172.21.15.42:0/3269885956 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f460811a8f0 0x7f460811ad20 secure :-1 s=READY pgs=951 cs=0 l=1 rx=0x7f45fc002f30 tx=0x7f45fc0108c0).stop 2019-12-05T05:29:12.322 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:12.315+0000 7f460f416700 1 -- 172.21.15.42:0/3269885956 shutdown_connections 2019-12-05T05:29:12.322 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:12.315+0000 7f460f416700 1 --2- 172.21.15.42:0/3269885956 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f45e803b230 0x7f45e803d740 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:29:12.322 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:12.315+0000 7f460f416700 1 --2- 172.21.15.42:0/3269885956 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f460811a8f0 0x7f460811ad20 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:29:12.322 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:12.315+0000 7f460f416700 1 --2- 172.21.15.42:0/3269885956 >> v2:172.21.15.40:3300/0 conn(0x7f46080a6380 0x7f46080a67b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:29:12.323 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:12.315+0000 7f460f416700 1 -- 172.21.15.42:0/3269885956 shutdown_connections 2019-12-05T05:29:12.323 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:12.315+0000 7f460f416700 1 -- 172.21.15.42:0/3269885956 wait complete. 2019-12-05T05:29:12.323 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:12.315+0000 7f460f416700 1 -- 172.21.15.42:0/3269885956 >> 172.21.15.42:0/3269885956 conn(0x7f460805a0f0 msgr2=0x7f4608059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:29:12.332 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T05:29:14.333 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:29:14.444 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:14.431+0000 7f704f694700 1 Processor -- start 2019-12-05T05:29:14.444 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:14.431+0000 7f704f694700 1 -- start start 2019-12-05T05:29:14.444 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:14.431+0000 7f704f694700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f70480a7740 0x7f704808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:29:14.444 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:14.431+0000 7f704f694700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f704807e720 0x7f704807eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:29:14.445 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:14.431+0000 7f704f694700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70480a0710 con 0x7f704809f2e0 2019-12-05T05:29:14.445 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:14.431+0000 7f704f694700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70480a0b00 con 0x7f70480a7740 2019-12-05T05:29:14.445 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:14.431+0000 7f704f694700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f704808e5b0 con 0x7f704807e720 2019-12-05T05:29:14.445 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:14.431+0000 7f704d430700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f704807e720 0x7f704807eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:29:14.445 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:14.431+0000 7f704dc31700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f70480a7740 0x7f704808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:29:14.445 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:14.431+0000 7f704d430700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f704807e720 0x7f704807eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:35210/0 (socket says 172.21.15.42:35210) 2019-12-05T05:29:14.445 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:14.431+0000 7f704d430700 1 -- 172.21.15.42:0/3493032450 learned_addr learned my addr 172.21.15.42:0/3493032450 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:29:14.446 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:14.431+0000 7f704d430700 1 -- 172.21.15.42:0/3493032450 >> v2:172.21.15.40:3300/0 conn(0x7f70480a7740 msgr2=0x7f704808df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:29:14.446 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:14.431+0000 7f704d430700 1 --2- 172.21.15.42:0/3493032450 >> v2:172.21.15.40:3300/0 conn(0x7f70480a7740 0x7f704808df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:29:14.446 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:14.431+0000 7f704d430700 1 -- 172.21.15.42:0/3493032450 >> v1:172.21.15.40:6789/0 conn(0x7f704809f2e0 legacy=0x7f704809f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:29:14.446 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:14.431+0000 7f704d430700 1 -- 172.21.15.42:0/3493032450 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f704807f0a0 con 0x7f704807e720 2019-12-05T05:29:14.446 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:14.431+0000 7f704d430700 1 --2- 172.21.15.42:0/3493032450 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f704807e720 0x7f704807eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f703c008790 tx=0x7f703c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:14.446 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:14.431+0000 7f704d430700 1 --2- 172.21.15.42:0/3493032450 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f704807e720 0x7f704807eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f703c008790 tx=0x7f703c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:14.446 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:14.431+0000 7f704d430700 1 --2- 172.21.15.42:0/3493032450 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f704807e720 0x7f704807eb50 secure :-1 s=READY pgs=194 cs=0 l=1 rx=0x7f703c008790 tx=0x7f703c00fd30).ready entity=mon.2 client_cookie=91b49f788bd32355 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:29:14.447 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:14.431+0000 7f704d430700 1 -- 172.21.15.42:0/3493032450 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f704807e720 msgr2=0x7f704807eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2019-12-05T05:29:14.447 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:14.431+0000 7f704d430700 1 -- 172.21.15.42:0/3493032450 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f704807e720 msgr2=0x7f704807eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2019-12-05T05:29:14.447 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:14.431+0000 7f704d430700 1 --2- 172.21.15.42:0/3493032450 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f704807e720 0x7f704807eb50 secure :-1 s=READY pgs=194 cs=0 l=1 rx=0x7f703c008790 tx=0x7f703c00fd30).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-12-05T05:29:14.447 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:14.431+0000 7f704d430700 1 --2- 172.21.15.42:0/3493032450 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f704807e720 0x7f704807eb50 secure :-1 s=READY pgs=194 cs=0 l=1 rx=0x7f703c008790 tx=0x7f703c00fd30).stop 2019-12-05T05:29:14.447 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:14.431+0000 7f704cc2f700 1 -- 172.21.15.42:0/3493032450 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f704807e720 msgr2=0x7f704807eb50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2019-12-05T05:29:14.447 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:14.431+0000 7f704cc2f700 1 --2- 172.21.15.42:0/3493032450 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f704807e720 0x7f704807eb50 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:29:14.447 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:14.431+0000 7f704cc2f700 1 --2- 172.21.15.42:0/3493032450 >> v2:172.21.15.42:3300/0 conn(0x7f703004afb0 0x7f703004d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:29:14.448 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:14.431+0000 7f704cc2f700 1 -- 172.21.15.42:0/3493032450 --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f703004dbe0 con 0x7f7030003090 2019-12-05T05:29:14.448 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:14.431+0000 7f704cc2f700 1 -- 172.21.15.42:0/3493032450 --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f703004eeb0 con 0x7f703004afb0 2019-12-05T05:29:14.448 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:14.431+0000 7f704d430700 1 --2- 172.21.15.42:0/3493032450 >> v2:172.21.15.42:3300/0 conn(0x7f703004afb0 0x7f703004d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:29:14.448 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:14.431+0000 7f704d430700 1 -- 172.21.15.42:0/3493032450 >> v1:172.21.15.42:6789/0 conn(0x7f7030003090 legacy=0x7f7030005560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:29:14.539 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:14.431+0000 7f704cc2f700 1 -- 172.21.15.42:0/3493032450 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2155602365 0 0) 0x7f703004dbe0 con 0x7f7030003090 2019-12-05T05:29:14.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:14.431+0000 7f704d430700 1 -- 172.21.15.42:0/3493032450 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f703004f160 con 0x7f703004afb0 2019-12-05T05:29:14.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:14.431+0000 7f704d430700 1 --2- 172.21.15.42:0/3493032450 >> v2:172.21.15.42:3300/0 conn(0x7f703004afb0 0x7f703004d4c0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f703c008790 tx=0x7f703c008f50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:14.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:14.435+0000 7f704d430700 1 --2- 172.21.15.42:0/3493032450 >> v2:172.21.15.42:3300/0 conn(0x7f703004afb0 0x7f703004d4c0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f703c008790 tx=0x7f703c008f50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:14.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:14.435+0000 7f704d430700 1 --2- 172.21.15.42:0/3493032450 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f703004afb0 0x7f703004d4c0 secure :-1 s=READY pgs=953 cs=0 l=1 rx=0x7f703c008790 tx=0x7f703c008f50).ready entity=mon.1 client_cookie=fa189208ddd8a446 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:29:14.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:14.435+0000 7f704d430700 1 --2- 172.21.15.42:0/3493032450 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f703004afb0 0x7f703004d4c0 secure :-1 s=THROTTLE_DONE pgs=953 cs=0 l=1 rx=0x7f703c008790 tx=0x7f703c008f50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:14.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:14.435+0000 7f704cc2f700 1 -- 172.21.15.42:0/3493032450 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f703c012630 con 0x7f703004afb0 2019-12-05T05:29:14.541 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:14.435+0000 7f704d430700 1 --2- 172.21.15.42:0/3493032450 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f703004afb0 0x7f703004d4c0 secure :-1 s=THROTTLE_DONE pgs=953 cs=0 l=1 rx=0x7f703c008790 tx=0x7f703c008f50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:14.541 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:14.435+0000 7f704d430700 1 --2- 172.21.15.42:0/3493032450 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f703004afb0 0x7f703004d4c0 secure :-1 s=THROTTLE_DONE pgs=953 cs=0 l=1 rx=0x7f703c008790 tx=0x7f703c008f50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:14.541 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:14.435+0000 7f704cc2f700 1 -- 172.21.15.42:0/3493032450 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f703c00f5c0 con 0x7f703004afb0 2019-12-05T05:29:14.541 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:14.435+0000 7f704cc2f700 1 -- 172.21.15.42:0/3493032450 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f703c012c10 con 0x7f703004afb0 2019-12-05T05:29:14.541 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:14.435+0000 7f704f694700 1 -- 172.21.15.42:0/3493032450 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f703004afb0 msgr2=0x7f703004d4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:29:14.541 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:14.435+0000 7f704f694700 1 --2- 172.21.15.42:0/3493032450 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f703004afb0 0x7f703004d4c0 secure :-1 s=READY pgs=953 cs=0 l=1 rx=0x7f703c008790 tx=0x7f703c008f50).stop 2019-12-05T05:29:14.541 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:14.435+0000 7f704e432700 1 -- 172.21.15.42:0/3493032450 reap_dead start 2019-12-05T05:29:14.542 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:14.435+0000 7f704f694700 1 -- 172.21.15.42:0/3493032450 shutdown_connections 2019-12-05T05:29:14.542 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:14.435+0000 7f704f694700 1 --2- 172.21.15.42:0/3493032450 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f703004afb0 0x7f703004d4c0 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:29:14.542 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:14.435+0000 7f704f694700 1 -- 172.21.15.42:0/3493032450 shutdown_connections 2019-12-05T05:29:14.542 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:14.435+0000 7f704f694700 1 -- 172.21.15.42:0/3493032450 wait complete. 2019-12-05T05:29:14.542 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:14.435+0000 7f704f694700 1 -- 172.21.15.42:0/3493032450 >> 172.21.15.42:0/3493032450 conn(0x7f70480aabe0 msgr2=0x7f70480a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:29:14.542 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:14.439+0000 7f704f694700 1 Processor -- start 2019-12-05T05:29:14.543 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:14.439+0000 7f704f694700 1 -- start start 2019-12-05T05:29:14.547 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:14.439+0000 7f704f694700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7048124110 0x7f704807d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:29:14.547 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:14.439+0000 7f704f694700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f7048130970 0x7f70480a5c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:29:14.548 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:14.439+0000 7f704f694700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7048130da0 0x7f70480a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:29:14.548 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:14.439+0000 7f704f694700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f704807d070 con 0x7f7048130da0 2019-12-05T05:29:14.548 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:14.439+0000 7f704f694700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f704807d310 con 0x7f7048130970 2019-12-05T05:29:14.548 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:14.439+0000 7f704f694700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f704807db00 con 0x7f7048124110 2019-12-05T05:29:14.548 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:14.439+0000 7f704d430700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f7048130970 0x7f70480a5c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:29:14.553 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:14.439+0000 7f704d430700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f7048130970 0x7f70480a5c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:34756/0 (socket says 172.21.15.42:34756) 2019-12-05T05:29:14.553 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:14.439+0000 7f704d430700 1 -- 172.21.15.42:0/1187859816 learned_addr learned my addr 172.21.15.42:0/1187859816 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:29:14.553 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:14.439+0000 7f704dc31700 1 --2- 172.21.15.42:0/1187859816 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7048124110 0x7f704807d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:29:14.554 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:14.439+0000 7f704e432700 1 --2- 172.21.15.42:0/1187859816 >> v2:172.21.15.40:3300/0 conn(0x7f7048130da0 0x7f70480a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:29:14.554 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:14.439+0000 7f704d430700 1 -- 172.21.15.42:0/1187859816 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7048124110 msgr2=0x7f704807d5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:29:14.554 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:14.439+0000 7f704d430700 1 --2- 172.21.15.42:0/1187859816 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7048124110 0x7f704807d5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:29:14.554 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:14.439+0000 7f704d430700 1 -- 172.21.15.42:0/1187859816 >> v2:172.21.15.40:3300/0 conn(0x7f7048130da0 msgr2=0x7f70480a6180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:29:14.555 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:14.439+0000 7f704d430700 1 --2- 172.21.15.42:0/1187859816 >> v2:172.21.15.40:3300/0 conn(0x7f7048130da0 0x7f70480a6180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:29:14.555 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:14.439+0000 7f704d430700 1 -- 172.21.15.42:0/1187859816 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f704807ccd0 con 0x7f7048130970 2019-12-05T05:29:14.555 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:14.439+0000 7f704e432700 1 --2- 172.21.15.42:0/1187859816 >> v2:172.21.15.40:3300/0 conn(0x7f7048130da0 0x7f70480a6180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T05:29:14.555 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:14.439+0000 7f704d430700 1 --2- 172.21.15.42:0/1187859816 >> v2:172.21.15.42:3300/0 conn(0x7f7048130970 0x7f70480a5c30 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f703c008790 tx=0x7f703c00f8a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:14.556 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:14.439+0000 7f704d430700 1 --2- 172.21.15.42:0/1187859816 >> v2:172.21.15.42:3300/0 conn(0x7f7048130970 0x7f70480a5c30 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f703c008790 tx=0x7f703c00f8a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:14.556 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:14.439+0000 7f704d430700 1 --2- 172.21.15.42:0/1187859816 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7048130970 0x7f70480a5c30 secure :-1 s=READY pgs=954 cs=0 l=1 rx=0x7f703c008790 tx=0x7f703c00f8a0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:29:14.566 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:14.439+0000 7f704d430700 1 --2- 172.21.15.42:0/1187859816 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7048130970 0x7f70480a5c30 secure :-1 s=THROTTLE_DONE pgs=954 cs=0 l=1 rx=0x7f703c008790 tx=0x7f703c00f8a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:14.567 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:14.439+0000 7f704d430700 1 --2- 172.21.15.42:0/1187859816 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7048130970 0x7f70480a5c30 secure :-1 s=THROTTLE_DONE pgs=954 cs=0 l=1 rx=0x7f703c008790 tx=0x7f703c00f8a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:14.567 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:14.439+0000 7f70467fc700 1 -- 172.21.15.42:0/1187859816 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f703c011810 con 0x7f7048130970 2019-12-05T05:29:14.567 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:14.439+0000 7f704d430700 1 --2- 172.21.15.42:0/1187859816 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7048130970 0x7f70480a5c30 secure :-1 s=THROTTLE_DONE pgs=954 cs=0 l=1 rx=0x7f703c008790 tx=0x7f703c00f8a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:14.567 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:14.439+0000 7f70467fc700 1 -- 172.21.15.42:0/1187859816 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f703c012830 con 0x7f7048130970 2019-12-05T05:29:14.568 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:14.439+0000 7f704f694700 1 -- 172.21.15.42:0/1187859816 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70481238f0 con 0x7f7048130970 2019-12-05T05:29:14.568 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:14.439+0000 7f70467fc700 1 -- 172.21.15.42:0/1187859816 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f703c011df0 con 0x7f7048130970 2019-12-05T05:29:14.568 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:14.439+0000 7f704f694700 1 -- 172.21.15.42:0/1187859816 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f704811d260 con 0x7f7048130970 2019-12-05T05:29:14.568 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:14.443+0000 7f704d430700 1 --2- 172.21.15.42:0/1187859816 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7048130970 0x7f70480a5c30 secure :-1 s=THROTTLE_DONE pgs=954 cs=0 l=1 rx=0x7f703c008790 tx=0x7f703c00f8a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:14.569 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:14.443+0000 7f70467fc700 1 -- 172.21.15.42:0/1187859816 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f703c005540 con 0x7f7048130970 2019-12-05T05:29:14.569 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:14.443+0000 7f704d430700 1 --2- 172.21.15.42:0/1187859816 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7048130970 0x7f70480a5c30 secure :-1 s=THROTTLE_DONE pgs=954 cs=0 l=1 rx=0x7f703c008790 tx=0x7f703c00f8a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:14.569 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:14.443+0000 7f70467fc700 1 --2- 172.21.15.42:0/1187859816 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f702803ae40 0x7f702803d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:29:14.569 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:14.443+0000 7f70467fc700 1 -- 172.21.15.42:0/1187859816 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 7493+0+0 (secure 0 0 0) 0x7f703c04bd60 con 0x7f7048130970 2019-12-05T05:29:14.569 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:14.443+0000 7f704dc31700 1 --2- 172.21.15.42:0/1187859816 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f702803ae40 0x7f702803d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:29:14.570 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:14.443+0000 7f704dc31700 1 --2- 172.21.15.42:0/1187859816 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f702803ae40 0x7f702803d350 crc :-1 s=READY pgs=414 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:29:14.570 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:14.447+0000 7f704f694700 1 -- 172.21.15.42:0/1187859816 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f704805d110 con 0x7f7048130970 2019-12-05T05:29:14.570 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:14.451+0000 7f704d430700 1 --2- 172.21.15.42:0/1187859816 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7048130970 0x7f70480a5c30 secure :-1 s=THROTTLE_DONE pgs=954 cs=0 l=1 rx=0x7f703c008790 tx=0x7f703c00f8a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:14.570 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:14.451+0000 7f70467fc700 1 -- 172.21.15.42:0/1187859816 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f703c007f50 con 0x7f7048130970 2019-12-05T05:29:14.630 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:14.623+0000 7f704f694700 1 -- 172.21.15.42:0/1187859816 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f704805b8c0 con 0x7f7048130970 2019-12-05T05:29:14.631 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:14.627+0000 7f704d430700 1 --2- 172.21.15.42:0/1187859816 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7048130970 0x7f70480a5c30 secure :-1 s=THROTTLE_DONE pgs=954 cs=0 l=1 rx=0x7f703c008790 tx=0x7f703c00f8a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:14.632 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:14.627+0000 7f70467fc700 1 -- 172.21.15.42:0/1187859816 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 118 v118) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f703c033bf0 con 0x7f7048130970 2019-12-05T05:29:14.634 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 118 2019-12-05T05:29:14.634 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:29:14.634 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":118,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5426,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/3168718372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3168718372},{"type":"v1","addr":"172.21.15.40:6833","nonce":3168718372}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":118}],"filesystems":[{"mdsmap":{"epoch":117,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:28:18.130626+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6666},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6666":{"gid":6666,"name":"a","rank":0,"incarnation":114,"state":"up:active","state_seq":17,"addr":"172.21.15.42:6835/2544447110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2544447110},{"type":"v1","addr":"172.21.15.42:6835","nonce":2544447110}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:29:14.637 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:14.631+0000 7f704f694700 1 -- 172.21.15.42:0/1187859816 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f702803ae40 msgr2=0x7f702803d350 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:29:14.637 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:14.631+0000 7f704f694700 1 --2- 172.21.15.42:0/1187859816 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f702803ae40 0x7f702803d350 crc :-1 s=READY pgs=414 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:29:14.637 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:14.631+0000 7f704f694700 1 -- 172.21.15.42:0/1187859816 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7048130970 msgr2=0x7f70480a5c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:29:14.638 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:14.631+0000 7f704f694700 1 --2- 172.21.15.42:0/1187859816 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7048130970 0x7f70480a5c30 secure :-1 s=READY pgs=954 cs=0 l=1 rx=0x7f703c008790 tx=0x7f703c00f8a0).stop 2019-12-05T05:29:14.638 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:14.631+0000 7f704f694700 1 -- 172.21.15.42:0/1187859816 shutdown_connections 2019-12-05T05:29:14.638 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:14.631+0000 7f704f694700 1 --2- 172.21.15.42:0/1187859816 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f702803ae40 0x7f702803d350 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:29:14.638 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:14.631+0000 7f704f694700 1 --2- 172.21.15.42:0/1187859816 >> v2:172.21.15.40:3300/0 conn(0x7f7048130da0 0x7f70480a6180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:29:14.639 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:14.631+0000 7f704f694700 1 --2- 172.21.15.42:0/1187859816 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7048124110 0x7f704807d5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:29:14.639 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:14.631+0000 7f704f694700 1 --2- 172.21.15.42:0/1187859816 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7048130970 0x7f70480a5c30 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:29:14.639 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:14.631+0000 7f704f694700 1 -- 172.21.15.42:0/1187859816 shutdown_connections 2019-12-05T05:29:14.639 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:14.631+0000 7f704f694700 1 -- 172.21.15.42:0/1187859816 wait complete. 2019-12-05T05:29:14.640 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:14.631+0000 7f704f694700 1 -- 172.21.15.42:0/1187859816 >> 172.21.15.42:0/1187859816 conn(0x7f70480a9a30 msgr2=0x7f704808eb70 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:29:14.648 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T05:29:16.649 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:29:16.763 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:16.747+0000 7f19dd655700 1 Processor -- start 2019-12-05T05:29:16.764 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:16.747+0000 7f19dd655700 1 -- start start 2019-12-05T05:29:16.764 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:16.751+0000 7f19dd655700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f19d809fae0 0x7f19d814fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:29:16.764 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:16.751+0000 7f19dd655700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19d80a0710 con 0x7f19d80a7740 2019-12-05T05:29:16.764 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:16.751+0000 7f19dd655700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19d807ef20 con 0x7f19d808df80 2019-12-05T05:29:16.765 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:16.751+0000 7f19dd655700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19d80a0b00 con 0x7f19d809fae0 2019-12-05T05:29:16.765 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:16.751+0000 7f19d6ffd700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f19d808df80 0x7f19d809f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:34808/0 (socket says 172.21.15.42:34808) 2019-12-05T05:29:16.765 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:16.751+0000 7f19d6ffd700 1 -- 172.21.15.42:0/515091084 learned_addr learned my addr 172.21.15.42:0/515091084 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:29:16.766 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:16.751+0000 7f19d7fff700 1 --2- 172.21.15.42:0/515091084 >> v2:172.21.15.40:3300/0 conn(0x7f19d809fae0 0x7f19d814fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:29:16.766 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:16.751+0000 7f19d67fc700 1 -- 172.21.15.42:0/515091084 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3721324768 0 0) 0x7f19d807ef20 con 0x7f19d808df80 2019-12-05T05:29:16.766 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:16.751+0000 7f19d67fc700 1 -- 172.21.15.42:0/515091084 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19c00030b0 con 0x7f19d808df80 2019-12-05T05:29:16.766 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:16.751+0000 7f19d67fc700 1 -- 172.21.15.42:0/515091084 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2218540259 0 0) 0x7f19c00030b0 con 0x7f19d808df80 2019-12-05T05:29:16.766 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:16.751+0000 7f19d67fc700 1 -- 172.21.15.42:0/515091084 >> v2:172.21.15.40:3300/0 conn(0x7f19d809fae0 msgr2=0x7f19d814fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:29:16.767 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:16.751+0000 7f19d67fc700 1 --2- 172.21.15.42:0/515091084 >> v2:172.21.15.40:3300/0 conn(0x7f19d809fae0 0x7f19d814fff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:29:16.767 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:16.751+0000 7f19d67fc700 1 -- 172.21.15.42:0/515091084 >> v1:172.21.15.40:6789/0 conn(0x7f19d80a7740 legacy=0x7f19d807e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:29:16.767 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:16.751+0000 7f19d67fc700 1 -- 172.21.15.42:0/515091084 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19d80a9ba0 con 0x7f19d808df80 2019-12-05T05:29:16.767 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:16.751+0000 7f19d67fc700 1 -- 172.21.15.42:0/515091084 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7f19cc001750 con 0x7f19d808df80 2019-12-05T05:29:16.767 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:16.751+0000 7f19d67fc700 1 -- 172.21.15.42:0/515091084 >> v1:172.21.15.42:6789/0 conn(0x7f19d808df80 legacy=0x7f19d809f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:29:16.767 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:16.751+0000 7f19d67fc700 1 --2- 172.21.15.42:0/515091084 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f19c00036e0 0x7f19c0003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:29:16.768 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:16.751+0000 7f19d67fc700 1 --2- 172.21.15.42:0/515091084 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f19c0004420 0x7f19c0004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:29:16.768 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:16.751+0000 7f19d67fc700 1 --2- 172.21.15.42:0/515091084 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f19c0004da0 0x7f19c000b670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:29:16.768 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:16.751+0000 7f19d67fc700 1 -- 172.21.15.42:0/515091084 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19d80a0b00 con 0x7f19c00036e0 2019-12-05T05:29:16.768 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:16.751+0000 7f19d67fc700 1 -- 172.21.15.42:0/515091084 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19c004fc60 con 0x7f19c0004da0 2019-12-05T05:29:16.768 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:16.751+0000 7f19d7fff700 1 --2- 172.21.15.42:0/515091084 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f19c0004da0 0x7f19c000b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:29:16.768 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:16.751+0000 7f19d67fc700 1 -- 172.21.15.42:0/515091084 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19c004fea0 con 0x7f19c0004420 2019-12-05T05:29:16.769 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:16.751+0000 7f19d6ffd700 1 --2- 172.21.15.42:0/515091084 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f19c0004420 0x7f19c0004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:29:16.769 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:16.751+0000 7f19d77fe700 1 --2- 172.21.15.42:0/515091084 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f19c00036e0 0x7f19c0003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:29:16.769 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:16.751+0000 7f19d7fff700 1 -- 172.21.15.42:0/515091084 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f19c0004420 msgr2=0x7f19c0004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:29:16.769 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:16.751+0000 7f19d7fff700 1 --2- 172.21.15.42:0/515091084 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f19c0004420 0x7f19c0004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:29:16.769 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:16.751+0000 7f19d7fff700 1 -- 172.21.15.42:0/515091084 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f19c00036e0 msgr2=0x7f19c0003b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:29:16.769 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:16.751+0000 7f19d7fff700 1 --2- 172.21.15.42:0/515091084 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f19c00036e0 0x7f19c0003b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:29:16.770 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:16.751+0000 7f19d7fff700 1 -- 172.21.15.42:0/515091084 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19c00501a0 con 0x7f19c0004da0 2019-12-05T05:29:16.770 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:16.751+0000 7f19d7fff700 1 --2- 172.21.15.42:0/515091084 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f19c0004da0 0x7f19c000b670 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f19d00032b0 tx=0x7f19d0010320).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:16.770 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:16.751+0000 7f19d7fff700 1 -- 172.21.15.42:0/515091084 reap_dead start 2019-12-05T05:29:16.770 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:16.751+0000 7f19d7fff700 1 --2- 172.21.15.42:0/515091084 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f19c0004da0 0x7f19c000b670 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f19d00032b0 tx=0x7f19d0010320).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:16.771 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:16.751+0000 7f19d7fff700 1 --2- 172.21.15.42:0/515091084 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f19c0004da0 0x7f19c000b670 secure :-1 s=READY pgs=956 cs=0 l=1 rx=0x7f19d00032b0 tx=0x7f19d0010320).ready entity=mon.1 client_cookie=ff114d7d95abb832 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:29:16.771 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:16.751+0000 7f19d7fff700 1 --2- 172.21.15.42:0/515091084 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f19c0004da0 0x7f19c000b670 secure :-1 s=THROTTLE_DONE pgs=956 cs=0 l=1 rx=0x7f19d00032b0 tx=0x7f19d0010320).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:16.771 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:16.751+0000 7f19d67fc700 1 -- 172.21.15.42:0/515091084 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f19d0008d40 con 0x7f19c0004da0 2019-12-05T05:29:16.771 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:16.751+0000 7f19d7fff700 1 --2- 172.21.15.42:0/515091084 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f19c0004da0 0x7f19c000b670 secure :-1 s=THROTTLE_DONE pgs=956 cs=0 l=1 rx=0x7f19d00032b0 tx=0x7f19d0010320).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:16.771 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:16.751+0000 7f19d7fff700 1 --2- 172.21.15.42:0/515091084 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f19c0004da0 0x7f19c000b670 secure :-1 s=THROTTLE_DONE pgs=956 cs=0 l=1 rx=0x7f19d00032b0 tx=0x7f19d0010320).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:16.771 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:16.751+0000 7f19d67fc700 1 -- 172.21.15.42:0/515091084 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19d0010c70 con 0x7f19c0004da0 2019-12-05T05:29:16.772 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:16.751+0000 7f19d67fc700 1 -- 172.21.15.42:0/515091084 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f19d0005ca0 con 0x7f19c0004da0 2019-12-05T05:29:16.772 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:16.751+0000 7f19dd655700 1 -- 172.21.15.42:0/515091084 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f19c0004da0 msgr2=0x7f19c000b670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:29:16.772 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:16.751+0000 7f19dd655700 1 --2- 172.21.15.42:0/515091084 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f19c0004da0 0x7f19c000b670 secure :-1 s=READY pgs=956 cs=0 l=1 rx=0x7f19d00032b0 tx=0x7f19d0010320).stop 2019-12-05T05:29:16.772 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:16.755+0000 7f19dd655700 1 -- 172.21.15.42:0/515091084 shutdown_connections 2019-12-05T05:29:16.773 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:16.755+0000 7f19dd655700 1 --2- 172.21.15.42:0/515091084 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f19c0004da0 0x7f19c000b670 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:29:16.773 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:16.755+0000 7f19dd655700 1 -- 172.21.15.42:0/515091084 shutdown_connections 2019-12-05T05:29:16.773 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:16.755+0000 7f19dd655700 1 -- 172.21.15.42:0/515091084 wait complete. 2019-12-05T05:29:16.773 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:16.755+0000 7f19dd655700 1 -- 172.21.15.42:0/515091084 >> 172.21.15.42:0/515091084 conn(0x7f19d80aabe0 msgr2=0x7f19d80a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:29:16.774 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:16.755+0000 7f19dd655700 1 Processor -- start 2019-12-05T05:29:16.774 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:16.755+0000 7f19dd655700 1 -- start start 2019-12-05T05:29:16.777 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:16.759+0000 7f19dd655700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f19d807d430 0x7f19d80a2030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:29:16.777 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:16.759+0000 7f19dd655700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f19d80a4080 0x7f19d8135b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:29:16.778 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:16.759+0000 7f19dd655700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f19d8064c70 0x7f19d80a59e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:29:16.778 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:16.759+0000 7f19dd655700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19d80a1af0 con 0x7f19d8064c70 2019-12-05T05:29:16.778 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:16.759+0000 7f19dd655700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19d80a1d90 con 0x7f19d80a4080 2019-12-05T05:29:16.778 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:16.759+0000 7f19dd655700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19d8064880 con 0x7f19d807d430 2019-12-05T05:29:16.778 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:16.759+0000 7f19d6ffd700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f19d80a4080 0x7f19d8135b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:29:16.779 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:16.759+0000 7f19d6ffd700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f19d80a4080 0x7f19d8135b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:34782/0 (socket says 172.21.15.42:34782) 2019-12-05T05:29:16.779 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:16.759+0000 7f19d6ffd700 1 -- 172.21.15.42:0/3618855755 learned_addr learned my addr 172.21.15.42:0/3618855755 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:29:16.779 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:16.759+0000 7f19d77fe700 1 --2- 172.21.15.42:0/3618855755 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f19d807d430 0x7f19d80a2030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:29:16.779 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:16.759+0000 7f19d7fff700 1 --2- 172.21.15.42:0/3618855755 >> v2:172.21.15.40:3300/0 conn(0x7f19d8064c70 0x7f19d80a59e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:29:16.779 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:16.759+0000 7f19d6ffd700 1 -- 172.21.15.42:0/3618855755 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f19d807d430 msgr2=0x7f19d80a2030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:29:16.780 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:16.759+0000 7f19d6ffd700 1 --2- 172.21.15.42:0/3618855755 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f19d807d430 0x7f19d80a2030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:29:16.780 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:16.759+0000 7f19d6ffd700 1 -- 172.21.15.42:0/3618855755 >> v2:172.21.15.40:3300/0 conn(0x7f19d8064c70 msgr2=0x7f19d80a59e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:29:16.780 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:16.759+0000 7f19d6ffd700 1 --2- 172.21.15.42:0/3618855755 >> v2:172.21.15.40:3300/0 conn(0x7f19d8064c70 0x7f19d80a59e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:29:16.780 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:16.759+0000 7f19d6ffd700 1 -- 172.21.15.42:0/3618855755 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19d80a5f30 con 0x7f19d80a4080 2019-12-05T05:29:16.781 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:16.759+0000 7f19d6ffd700 1 --2- 172.21.15.42:0/3618855755 >> v2:172.21.15.42:3300/0 conn(0x7f19d80a4080 0x7f19d8135b60 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f19cc001950 tx=0x7f19cc010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:16.781 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:16.759+0000 7f19d6ffd700 1 --2- 172.21.15.42:0/3618855755 >> v2:172.21.15.42:3300/0 conn(0x7f19d80a4080 0x7f19d8135b60 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f19cc001950 tx=0x7f19cc010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:16.781 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:16.759+0000 7f19d6ffd700 1 --2- 172.21.15.42:0/3618855755 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f19d80a4080 0x7f19d8135b60 secure :-1 s=READY pgs=957 cs=0 l=1 rx=0x7f19cc001950 tx=0x7f19cc010b60).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:29:16.781 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:16.759+0000 7f19d6ffd700 1 --2- 172.21.15.42:0/3618855755 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f19d80a4080 0x7f19d8135b60 secure :-1 s=THROTTLE_DONE pgs=957 cs=0 l=1 rx=0x7f19cc001950 tx=0x7f19cc010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:16.782 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:16.759+0000 7f19d6ffd700 1 --2- 172.21.15.42:0/3618855755 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f19d80a4080 0x7f19d8135b60 secure :-1 s=THROTTLE_DONE pgs=957 cs=0 l=1 rx=0x7f19cc001950 tx=0x7f19cc010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:16.782 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:16.759+0000 7f19d6ffd700 1 --2- 172.21.15.42:0/3618855755 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f19d80a4080 0x7f19d8135b60 secure :-1 s=THROTTLE_DONE pgs=957 cs=0 l=1 rx=0x7f19cc001950 tx=0x7f19cc010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:16.783 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:16.759+0000 7f19bffff700 1 -- 172.21.15.42:0/3618855755 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f19cc0129f0 con 0x7f19d80a4080 2019-12-05T05:29:16.783 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:16.759+0000 7f19bffff700 1 -- 172.21.15.42:0/3618855755 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19cc012610 con 0x7f19d80a4080 2019-12-05T05:29:16.783 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:16.759+0000 7f19dd655700 1 -- 172.21.15.42:0/3618855755 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19d80a6260 con 0x7f19d80a4080 2019-12-05T05:29:16.783 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:16.759+0000 7f19bffff700 1 -- 172.21.15.42:0/3618855755 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f19cc008570 con 0x7f19d80a4080 2019-12-05T05:29:16.783 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:16.759+0000 7f19dd655700 1 -- 172.21.15.42:0/3618855755 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19d80a6790 con 0x7f19d80a4080 2019-12-05T05:29:16.783 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:16.759+0000 7f19d6ffd700 1 --2- 172.21.15.42:0/3618855755 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f19d80a4080 0x7f19d8135b60 secure :-1 s=THROTTLE_DONE pgs=957 cs=0 l=1 rx=0x7f19cc001950 tx=0x7f19cc010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:16.784 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:16.763+0000 7f19d6ffd700 1 --2- 172.21.15.42:0/3618855755 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f19d80a4080 0x7f19d8135b60 secure :-1 s=THROTTLE_DONE pgs=957 cs=0 l=1 rx=0x7f19cc001950 tx=0x7f19cc010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:16.784 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:16.763+0000 7f19bffff700 1 -- 172.21.15.42:0/3618855755 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f19cc00b220 con 0x7f19d80a4080 2019-12-05T05:29:16.784 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:16.763+0000 7f19bffff700 1 --2- 172.21.15.42:0/3618855755 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f19b803ad70 0x7f19b803d280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:29:16.784 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:16.763+0000 7f19bffff700 1 -- 172.21.15.42:0/3618855755 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 7493+0+0 (secure 0 0 0) 0x7f19cc04b0d0 con 0x7f19d80a4080 2019-12-05T05:29:16.785 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:16.763+0000 7f19d77fe700 1 --2- 172.21.15.42:0/3618855755 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f19b803ad70 0x7f19b803d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:29:16.785 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:16.763+0000 7f19d77fe700 1 --2- 172.21.15.42:0/3618855755 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f19b803ad70 0x7f19b803d280 crc :-1 s=READY pgs=415 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:29:16.785 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:16.763+0000 7f19dd655700 1 -- 172.21.15.42:0/3618855755 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19d805d110 con 0x7f19d80a4080 2019-12-05T05:29:16.786 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:16.771+0000 7f19d6ffd700 1 --2- 172.21.15.42:0/3618855755 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f19d80a4080 0x7f19d8135b60 secure :-1 s=THROTTLE_DONE pgs=957 cs=0 l=1 rx=0x7f19cc001950 tx=0x7f19cc010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:16.786 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:16.771+0000 7f19bffff700 1 -- 172.21.15.42:0/3618855755 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f19cc032110 con 0x7f19d80a4080 2019-12-05T05:29:16.939 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:16.931+0000 7f19dd655700 1 -- 172.21.15.42:0/3618855755 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f19d80aadf0 con 0x7f19d80a4080 2019-12-05T05:29:16.940 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:16.931+0000 7f19d6ffd700 1 --2- 172.21.15.42:0/3618855755 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f19d80a4080 0x7f19d8135b60 secure :-1 s=THROTTLE_DONE pgs=957 cs=0 l=1 rx=0x7f19cc001950 tx=0x7f19cc010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:16.940 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:16.931+0000 7f19bffff700 1 -- 172.21.15.42:0/3618855755 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 118 v118) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f19cc032370 con 0x7f19d80a4080 2019-12-05T05:29:16.940 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 118 2019-12-05T05:29:16.940 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:29:16.941 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":118,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5426,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/3168718372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3168718372},{"type":"v1","addr":"172.21.15.40:6833","nonce":3168718372}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":118}],"filesystems":[{"mdsmap":{"epoch":117,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:28:18.130626+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6666},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6666":{"gid":6666,"name":"a","rank":0,"incarnation":114,"state":"up:active","state_seq":17,"addr":"172.21.15.42:6835/2544447110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2544447110},{"type":"v1","addr":"172.21.15.42:6835","nonce":2544447110}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:29:16.943 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:16.935+0000 7f19dd655700 1 -- 172.21.15.42:0/3618855755 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f19b803ad70 msgr2=0x7f19b803d280 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:29:16.943 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:16.935+0000 7f19dd655700 1 --2- 172.21.15.42:0/3618855755 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f19b803ad70 0x7f19b803d280 crc :-1 s=READY pgs=415 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:29:16.944 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:16.935+0000 7f19dd655700 1 -- 172.21.15.42:0/3618855755 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f19d80a4080 msgr2=0x7f19d8135b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:29:16.944 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:16.935+0000 7f19dd655700 1 --2- 172.21.15.42:0/3618855755 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f19d80a4080 0x7f19d8135b60 secure :-1 s=READY pgs=957 cs=0 l=1 rx=0x7f19cc001950 tx=0x7f19cc010b60).stop 2019-12-05T05:29:16.944 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:16.935+0000 7f19dd655700 1 -- 172.21.15.42:0/3618855755 shutdown_connections 2019-12-05T05:29:16.944 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:16.935+0000 7f19dd655700 1 --2- 172.21.15.42:0/3618855755 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f19b803ad70 0x7f19b803d280 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:29:16.945 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:16.935+0000 7f19dd655700 1 --2- 172.21.15.42:0/3618855755 >> v2:172.21.15.40:3300/0 conn(0x7f19d8064c70 0x7f19d80a59e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:29:16.945 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:16.935+0000 7f19dd655700 1 --2- 172.21.15.42:0/3618855755 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f19d807d430 0x7f19d80a2030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:29:16.945 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:16.935+0000 7f19dd655700 1 --2- 172.21.15.42:0/3618855755 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f19d80a4080 0x7f19d8135b60 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:29:16.945 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:16.935+0000 7f19dd655700 1 -- 172.21.15.42:0/3618855755 shutdown_connections 2019-12-05T05:29:16.945 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:16.935+0000 7f19dd655700 1 -- 172.21.15.42:0/3618855755 wait complete. 2019-12-05T05:29:16.946 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:16.935+0000 7f19dd655700 1 -- 172.21.15.42:0/3618855755 >> 172.21.15.42:0/3618855755 conn(0x7f19d80a8940 msgr2=0x7f19d808eb70 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:29:16.951 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T05:29:18.954 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:29:19.064 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.051+0000 7f6e6b7d4700 1 Processor -- start 2019-12-05T05:29:19.065 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.051+0000 7f6e6b7d4700 1 -- start start 2019-12-05T05:29:19.065 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.055+0000 7f6e6b7d4700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6e6409fae0 0x7f6e6414fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:29:19.065 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.055+0000 7f6e6b7d4700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e640a0710 con 0x7f6e6408df80 2019-12-05T05:29:19.065 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.055+0000 7f6e6b7d4700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e6407ef20 con 0x7f6e640a7740 2019-12-05T05:29:19.066 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.055+0000 7f6e6b7d4700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e640a0b00 con 0x7f6e6409fae0 2019-12-05T05:29:19.066 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.055+0000 7f6e69d71700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f6e640a7740 0x7f6e6407e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:34826/0 (socket says 172.21.15.42:34826) 2019-12-05T05:29:19.067 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.055+0000 7f6e69d71700 1 -- 172.21.15.42:0/30910411 learned_addr learned my addr 172.21.15.42:0/30910411 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:29:19.067 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.055+0000 7f6e6a572700 1 --2- 172.21.15.42:0/30910411 >> v2:172.21.15.42:3300/0 conn(0x7f6e6409fae0 0x7f6e6414fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:29:19.067 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.055+0000 7f6e6a572700 1 -- 172.21.15.42:0/30910411 >> v1:172.21.15.42:6789/0 conn(0x7f6e640a7740 legacy=0x7f6e6407e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:29:19.067 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.055+0000 7f6e6a572700 1 -- 172.21.15.42:0/30910411 >> v1:172.21.15.40:6789/0 conn(0x7f6e6408df80 legacy=0x7f6e6409f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:29:19.068 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.055+0000 7f6e6a572700 1 -- 172.21.15.42:0/30910411 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e640a9ba0 con 0x7f6e6409fae0 2019-12-05T05:29:19.068 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.055+0000 7f6e6a572700 1 --2- 172.21.15.42:0/30910411 >> v2:172.21.15.42:3300/0 conn(0x7f6e6409fae0 0x7f6e6414fff0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f6e5c008b10 tx=0x7f6e5c00ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:19.068 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.055+0000 7f6e6a572700 1 --2- 172.21.15.42:0/30910411 >> v2:172.21.15.42:3300/0 conn(0x7f6e6409fae0 0x7f6e6414fff0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f6e5c008b10 tx=0x7f6e5c00ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:19.068 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.055+0000 7f6e6a572700 1 --2- 172.21.15.42:0/30910411 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6e6409fae0 0x7f6e6414fff0 secure :-1 s=READY pgs=959 cs=0 l=1 rx=0x7f6e5c008b10 tx=0x7f6e5c00ff70).ready entity=mon.1 client_cookie=3307ad74adfb78d1 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:29:19.069 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.055+0000 7f6e6a572700 1 --2- 172.21.15.42:0/30910411 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6e6409fae0 0x7f6e6414fff0 secure :-1 s=THROTTLE_DONE pgs=959 cs=0 l=1 rx=0x7f6e5c008b10 tx=0x7f6e5c00ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:19.069 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.055+0000 7f6e68d6f700 1 -- 172.21.15.42:0/30910411 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6e5c010970 con 0x7f6e6409fae0 2019-12-05T05:29:19.070 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.055+0000 7f6e6a572700 1 --2- 172.21.15.42:0/30910411 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6e6409fae0 0x7f6e6414fff0 secure :-1 s=THROTTLE_DONE pgs=959 cs=0 l=1 rx=0x7f6e5c008b10 tx=0x7f6e5c00ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:19.070 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.055+0000 7f6e68d6f700 1 -- 172.21.15.42:0/30910411 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e5c008b50 con 0x7f6e6409fae0 2019-12-05T05:29:19.070 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.055+0000 7f6e6a572700 1 --2- 172.21.15.42:0/30910411 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6e6409fae0 0x7f6e6414fff0 secure :-1 s=THROTTLE_DONE pgs=959 cs=0 l=1 rx=0x7f6e5c008b10 tx=0x7f6e5c00ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:19.070 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.055+0000 7f6e68d6f700 1 -- 172.21.15.42:0/30910411 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6e5c0110c0 con 0x7f6e6409fae0 2019-12-05T05:29:19.071 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.055+0000 7f6e6b7d4700 1 -- 172.21.15.42:0/30910411 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6e6409fae0 msgr2=0x7f6e6414fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:29:19.071 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.055+0000 7f6e6b7d4700 1 --2- 172.21.15.42:0/30910411 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6e6409fae0 0x7f6e6414fff0 secure :-1 s=READY pgs=959 cs=0 l=1 rx=0x7f6e5c008b10 tx=0x7f6e5c00ff70).stop 2019-12-05T05:29:19.071 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.055+0000 7f6e6b7d4700 1 -- 172.21.15.42:0/30910411 shutdown_connections 2019-12-05T05:29:19.071 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.055+0000 7f6e6b7d4700 1 --2- 172.21.15.42:0/30910411 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6e6409fae0 0x7f6e6414fff0 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:29:19.072 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.055+0000 7f6e6b7d4700 1 -- 172.21.15.42:0/30910411 shutdown_connections 2019-12-05T05:29:19.072 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.055+0000 7f6e6b7d4700 1 -- 172.21.15.42:0/30910411 wait complete. 2019-12-05T05:29:19.074 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.055+0000 7f6e6b7d4700 1 -- 172.21.15.42:0/30910411 >> 172.21.15.42:0/30910411 conn(0x7f6e640aabe0 msgr2=0x7f6e640a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:29:19.081 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.059+0000 7f6e6b7d4700 1 Processor -- start 2019-12-05T05:29:19.082 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.059+0000 7f6e6b7d4700 1 -- start start 2019-12-05T05:29:19.082 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.059+0000 7f6e6b7d4700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f6e640658f0 0x7f6e64114820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:29:19.082 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.059+0000 7f6e6b7d4700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6e640637c0 0x7f6e64063bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:29:19.082 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.059+0000 7f6e6b7d4700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e640632b0 con 0x7f6e640a65e0 2019-12-05T05:29:19.083 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.059+0000 7f6e6b7d4700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e640a6d50 con 0x7f6e640658f0 2019-12-05T05:29:19.083 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.059+0000 7f6e6b7d4700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e640a6a10 con 0x7f6e640637c0 2019-12-05T05:29:19.085 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.059+0000 7f6e69d71700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f6e640658f0 0x7f6e64114820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:29:19.085 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.059+0000 7f6e6a572700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6e640637c0 0x7f6e64063bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:29:19.085 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.059+0000 7f6e69d71700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f6e640658f0 0x7f6e64114820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.42:50842/0 (socket says 172.21.15.42:50842) 2019-12-05T05:29:19.085 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.059+0000 7f6e69570700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f6e640a65e0 0x7f6e641295c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:44274/0 (socket says 172.21.15.42:44274) 2019-12-05T05:29:19.086 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.059+0000 7f6e6a572700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6e640637c0 0x7f6e64063bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:35264/0 (socket says 172.21.15.42:35264) 2019-12-05T05:29:19.086 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.059+0000 7f6e69d71700 1 -- 172.21.15.42:0/3332089873 learned_addr learned my addr 172.21.15.42:0/3332089873 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:29:19.086 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.063+0000 7f6e6a572700 1 -- 172.21.15.42:0/3332089873 >> v2:172.21.15.40:3300/0 conn(0x7f6e640658f0 msgr2=0x7f6e64114820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:29:19.086 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.063+0000 7f6e6a572700 1 --2- 172.21.15.42:0/3332089873 >> v2:172.21.15.40:3300/0 conn(0x7f6e640658f0 0x7f6e64114820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:29:19.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.063+0000 7f6e6a572700 1 -- 172.21.15.42:0/3332089873 >> v1:172.21.15.40:6789/0 conn(0x7f6e640a65e0 legacy=0x7f6e641295c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:29:19.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.063+0000 7f6e6a572700 1 -- 172.21.15.42:0/3332089873 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e64064140 con 0x7f6e640637c0 2019-12-05T05:29:19.088 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.063+0000 7f6e6a572700 1 --2- 172.21.15.42:0/3332089873 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6e640637c0 0x7f6e64063bf0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f6e5c008b10 tx=0x7f6e5c008870).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:19.088 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.063+0000 7f6e6a572700 1 --2- 172.21.15.42:0/3332089873 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6e640637c0 0x7f6e64063bf0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f6e5c008b10 tx=0x7f6e5c008870).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:19.088 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.063+0000 7f6e6a572700 1 --2- 172.21.15.42:0/3332089873 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6e640637c0 0x7f6e64063bf0 secure :-1 s=READY pgs=195 cs=0 l=1 rx=0x7f6e5c008b10 tx=0x7f6e5c008870).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:29:19.089 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.063+0000 7f6e6a572700 1 --2- 172.21.15.42:0/3332089873 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6e640637c0 0x7f6e64063bf0 secure :-1 s=THROTTLE_DONE pgs=195 cs=0 l=1 rx=0x7f6e5c008b10 tx=0x7f6e5c008870).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:19.089 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.063+0000 7f6e6a572700 1 --2- 172.21.15.42:0/3332089873 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6e640637c0 0x7f6e64063bf0 secure :-1 s=THROTTLE_DONE pgs=195 cs=0 l=1 rx=0x7f6e5c008b10 tx=0x7f6e5c008870).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:19.089 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.063+0000 7f6e6a572700 1 --2- 172.21.15.42:0/3332089873 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6e640637c0 0x7f6e64063bf0 secure :-1 s=THROTTLE_DONE pgs=195 cs=0 l=1 rx=0x7f6e5c008b10 tx=0x7f6e5c008870).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:19.089 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.063+0000 7f6e5a7fc700 1 -- 172.21.15.42:0/3332089873 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6e5c005cc0 con 0x7f6e640637c0 2019-12-05T05:29:19.090 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.063+0000 7f6e5a7fc700 1 -- 172.21.15.42:0/3332089873 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e5c010b70 con 0x7f6e640637c0 2019-12-05T05:29:19.090 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.063+0000 7f6e6b7d4700 1 -- 172.21.15.42:0/3332089873 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e64065060 con 0x7f6e640637c0 2019-12-05T05:29:19.090 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.063+0000 7f6e5a7fc700 1 -- 172.21.15.42:0/3332089873 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6e5c007540 con 0x7f6e640637c0 2019-12-05T05:29:19.091 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.063+0000 7f6e6b7d4700 1 -- 172.21.15.42:0/3332089873 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e64130410 con 0x7f6e640637c0 2019-12-05T05:29:19.092 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.063+0000 7f6e6a572700 1 --2- 172.21.15.42:0/3332089873 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6e640637c0 0x7f6e64063bf0 secure :-1 s=THROTTLE_DONE pgs=195 cs=0 l=1 rx=0x7f6e5c008b10 tx=0x7f6e5c008870).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:19.092 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.063+0000 7f6e5a7fc700 1 -- 172.21.15.42:0/3332089873 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f6e5c00f220 con 0x7f6e640637c0 2019-12-05T05:29:19.092 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.063+0000 7f6e6a572700 1 --2- 172.21.15.42:0/3332089873 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6e640637c0 0x7f6e64063bf0 secure :-1 s=THROTTLE_DONE pgs=195 cs=0 l=1 rx=0x7f6e5c008b10 tx=0x7f6e5c008870).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:19.093 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.063+0000 7f6e5a7fc700 1 --2- 172.21.15.42:0/3332089873 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6e4c03adb0 0x7f6e4c03d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:29:19.093 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.063+0000 7f6e5a7fc700 1 -- 172.21.15.42:0/3332089873 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7579+0+0 (secure 0 0 0) 0x7f6e5c04b0c0 con 0x7f6e640637c0 2019-12-05T05:29:19.093 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.063+0000 7f6e69d71700 1 --2- 172.21.15.42:0/3332089873 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6e4c03adb0 0x7f6e4c03d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:29:19.093 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.067+0000 7f6e69d71700 1 --2- 172.21.15.42:0/3332089873 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6e4c03adb0 0x7f6e4c03d2c0 crc :-1 s=READY pgs=416 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:29:19.093 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.067+0000 7f6e6b7d4700 1 -- 172.21.15.42:0/3332089873 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e6405d110 con 0x7f6e640637c0 2019-12-05T05:29:19.094 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.075+0000 7f6e6a572700 1 --2- 172.21.15.42:0/3332089873 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6e640637c0 0x7f6e64063bf0 secure :-1 s=THROTTLE_DONE pgs=195 cs=0 l=1 rx=0x7f6e5c008b10 tx=0x7f6e5c008870).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:19.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.075+0000 7f6e5a7fc700 1 -- 172.21.15.42:0/3332089873 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f6e5c032140 con 0x7f6e640637c0 2019-12-05T05:29:19.245 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.239+0000 7f6e6b7d4700 1 -- 172.21.15.42:0/3332089873 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6e6405b8c0 con 0x7f6e640637c0 2019-12-05T05:29:19.245 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.239+0000 7f6e6a572700 1 --2- 172.21.15.42:0/3332089873 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6e640637c0 0x7f6e64063bf0 secure :-1 s=THROTTLE_DONE pgs=195 cs=0 l=1 rx=0x7f6e5c008b10 tx=0x7f6e5c008870).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:19.245 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.239+0000 7f6e5a7fc700 1 -- 172.21.15.42:0/3332089873 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 119 v119) v1 ==== 95+0+1774 (secure 0 0 0) 0x7f6e5c0323a0 con 0x7f6e640637c0 2019-12-05T05:29:19.247 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 119 2019-12-05T05:29:19.247 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:29:19.247 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":119,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":119,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:29:18.667642+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5426},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5426":{"gid":5426,"name":"b","rank":0,"incarnation":119,"state":"up:replay","state_seq":1,"addr":"172.21.15.40:6833/3168718372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3168718372},{"type":"v1","addr":"172.21.15.40:6833","nonce":3168718372}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:29:19.250 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.243+0000 7f6e6b7d4700 1 -- 172.21.15.42:0/3332089873 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6e4c03adb0 msgr2=0x7f6e4c03d2c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:29:19.250 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.243+0000 7f6e6b7d4700 1 --2- 172.21.15.42:0/3332089873 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6e4c03adb0 0x7f6e4c03d2c0 crc :-1 s=READY pgs=416 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:29:19.250 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.243+0000 7f6e6b7d4700 1 -- 172.21.15.42:0/3332089873 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6e640637c0 msgr2=0x7f6e64063bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:29:19.250 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.243+0000 7f6e6b7d4700 1 --2- 172.21.15.42:0/3332089873 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6e640637c0 0x7f6e64063bf0 secure :-1 s=READY pgs=195 cs=0 l=1 rx=0x7f6e5c008b10 tx=0x7f6e5c008870).stop 2019-12-05T05:29:19.250 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.243+0000 7f6e6b7d4700 1 -- 172.21.15.42:0/3332089873 shutdown_connections 2019-12-05T05:29:19.251 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.243+0000 7f6e6b7d4700 1 --2- 172.21.15.42:0/3332089873 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6e4c03adb0 0x7f6e4c03d2c0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:29:19.251 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.243+0000 7f6e6b7d4700 1 --2- 172.21.15.42:0/3332089873 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6e640637c0 0x7f6e64063bf0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:29:19.251 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.243+0000 7f6e6b7d4700 1 --2- 172.21.15.42:0/3332089873 >> v2:172.21.15.40:3300/0 conn(0x7f6e640658f0 0x7f6e64114820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:29:19.251 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.243+0000 7f6e6b7d4700 1 -- 172.21.15.42:0/3332089873 shutdown_connections 2019-12-05T05:29:19.251 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.243+0000 7f6e6b7d4700 1 -- 172.21.15.42:0/3332089873 wait complete. 2019-12-05T05:29:19.251 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.243+0000 7f6e6b7d4700 1 -- 172.21.15.42:0/3332089873 >> 172.21.15.42:0/3332089873 conn(0x7f6e64115170 msgr2=0x7f6e6414fd20 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:29:19.260 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a down, removed from mdsmap 2019-12-05T05:29:19.260 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2019-12-05T05:29:19.261 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:29:19.358 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.347+0000 7fa5d47c6700 1 Processor -- start 2019-12-05T05:29:19.359 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.347+0000 7fa5d47c6700 1 -- start start 2019-12-05T05:29:19.359 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.347+0000 7fa5d47c6700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa5cc08df80 0x7fa5cc09f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:29:19.359 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.347+0000 7fa5d47c6700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fa5cc09f830 0x7fa5cc14efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:29:19.359 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.347+0000 7fa5d47c6700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5cc09fc60 con 0x7fa5cc0a7740 2019-12-05T05:29:19.359 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.347+0000 7fa5d47c6700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5cc0a0b00 con 0x7fa5cc09f830 2019-12-05T05:29:19.359 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.347+0000 7fa5d47c6700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5cc08e5b0 con 0x7fa5cc08df80 2019-12-05T05:29:19.359 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.347+0000 7fa5d3564700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fa5cc09f830 0x7fa5cc14efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:29:19.360 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.347+0000 7fa5d2d63700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fa5cc0a7740 0x7fa5cc07e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:44280/0 (socket says 172.21.15.42:44280) 2019-12-05T05:29:19.360 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.347+0000 7fa5d2d63700 1 -- 172.21.15.42:0/2620688813 learned_addr learned my addr 172.21.15.42:0/2620688813 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:29:19.360 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.347+0000 7fa5d2562700 1 --2- 172.21.15.42:0/2620688813 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa5cc08df80 0x7fa5cc09f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:29:19.360 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.347+0000 7fa5d3564700 1 -- 172.21.15.42:0/2620688813 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa5cc08df80 msgr2=0x7fa5cc09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:29:19.361 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.347+0000 7fa5d3564700 1 --2- 172.21.15.42:0/2620688813 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa5cc08df80 0x7fa5cc09f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:29:19.361 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.347+0000 7fa5d1d61700 1 -- 172.21.15.42:0/2620688813 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1757480421 0 0) 0x7fa5cc09fc60 con 0x7fa5cc0a7740 2019-12-05T05:29:19.361 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.347+0000 7fa5d3564700 1 -- 172.21.15.42:0/2620688813 >> v1:172.21.15.40:6789/0 conn(0x7fa5cc0a7740 legacy=0x7fa5cc07e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:29:19.361 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.347+0000 7fa5d3564700 1 -- 172.21.15.42:0/2620688813 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5cc0a0710 con 0x7fa5cc09f830 2019-12-05T05:29:19.361 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.347+0000 7fa5d3564700 1 --2- 172.21.15.42:0/2620688813 >> v2:172.21.15.42:3300/0 conn(0x7fa5cc09f830 0x7fa5cc14efe0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fa5c4008b10 tx=0x7fa5c400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:19.361 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.347+0000 7fa5d3564700 1 --2- 172.21.15.42:0/2620688813 >> v2:172.21.15.42:3300/0 conn(0x7fa5cc09f830 0x7fa5cc14efe0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fa5c4008b10 tx=0x7fa5c400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:19.362 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.347+0000 7fa5d3564700 1 --2- 172.21.15.42:0/2620688813 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa5cc09f830 0x7fa5cc14efe0 secure :-1 s=READY pgs=960 cs=0 l=1 rx=0x7fa5c4008b10 tx=0x7fa5c400fff0).ready entity=mon.1 client_cookie=2dfa302ed0faad0c server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:29:19.362 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.347+0000 7fa5d3564700 1 --2- 172.21.15.42:0/2620688813 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa5cc09f830 0x7fa5cc14efe0 secure :-1 s=THROTTLE_DONE pgs=960 cs=0 l=1 rx=0x7fa5c4008b10 tx=0x7fa5c400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:19.362 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.347+0000 7fa5d3564700 1 --2- 172.21.15.42:0/2620688813 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa5cc09f830 0x7fa5cc14efe0 secure :-1 s=THROTTLE_DONE pgs=960 cs=0 l=1 rx=0x7fa5c4008b10 tx=0x7fa5c400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:19.362 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.347+0000 7fa5d1d61700 1 -- 172.21.15.42:0/2620688813 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa5c4010900 con 0x7fa5cc09f830 2019-12-05T05:29:19.362 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.347+0000 7fa5d3564700 1 --2- 172.21.15.42:0/2620688813 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa5cc09f830 0x7fa5cc14efe0 secure :-1 s=THROTTLE_DONE pgs=960 cs=0 l=1 rx=0x7fa5c4008b10 tx=0x7fa5c400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:19.362 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.347+0000 7fa5d1d61700 1 -- 172.21.15.42:0/2620688813 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5c4008b50 con 0x7fa5cc09f830 2019-12-05T05:29:19.363 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.347+0000 7fa5d1d61700 1 -- 172.21.15.42:0/2620688813 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa5c40110c0 con 0x7fa5cc09f830 2019-12-05T05:29:19.363 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.347+0000 7fa5d47c6700 1 -- 172.21.15.42:0/2620688813 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa5cc09f830 msgr2=0x7fa5cc14efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:29:19.363 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.347+0000 7fa5d47c6700 1 --2- 172.21.15.42:0/2620688813 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa5cc09f830 0x7fa5cc14efe0 secure :-1 s=READY pgs=960 cs=0 l=1 rx=0x7fa5c4008b10 tx=0x7fa5c400fff0).stop 2019-12-05T05:29:19.363 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.351+0000 7fa5d47c6700 1 -- 172.21.15.42:0/2620688813 shutdown_connections 2019-12-05T05:29:19.363 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.351+0000 7fa5d47c6700 1 --2- 172.21.15.42:0/2620688813 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa5cc09f830 0x7fa5cc14efe0 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:29:19.363 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.351+0000 7fa5d47c6700 1 --2- 172.21.15.42:0/2620688813 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa5cc08df80 0x7fa5cc09f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:29:19.363 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.351+0000 7fa5d47c6700 1 -- 172.21.15.42:0/2620688813 shutdown_connections 2019-12-05T05:29:19.364 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.351+0000 7fa5d47c6700 1 -- 172.21.15.42:0/2620688813 wait complete. 2019-12-05T05:29:19.364 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.351+0000 7fa5d47c6700 1 -- 172.21.15.42:0/2620688813 >> 172.21.15.42:0/2620688813 conn(0x7fa5cc0aabe0 msgr2=0x7fa5cc0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:29:19.368 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.355+0000 7fa5d47c6700 1 Processor -- start 2019-12-05T05:29:19.368 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.355+0000 7fa5d47c6700 1 -- start start 2019-12-05T05:29:19.368 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.355+0000 7fa5d47c6700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa5cc146f60 0x7fa5cc145d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:29:19.368 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.355+0000 7fa5d47c6700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa5cc0a5d00 0x7fa5cc148050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:29:19.368 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.355+0000 7fa5d47c6700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5cc11c030 con 0x7fa5cc1462a0 2019-12-05T05:29:19.369 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.355+0000 7fa5d47c6700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5cc146a20 con 0x7fa5cc0a5d00 2019-12-05T05:29:19.369 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.355+0000 7fa5d47c6700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5cc146cc0 con 0x7fa5cc146f60 2019-12-05T05:29:19.369 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.355+0000 7fa5d3564700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fa5cc1462a0 0x7fa5cc07d7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:34850/0 (socket says 172.21.15.42:34850) 2019-12-05T05:29:19.369 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.355+0000 7fa5d3564700 1 -- 172.21.15.42:0/2880440593 learned_addr learned my addr 172.21.15.42:0/2880440593 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:29:19.369 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.355+0000 7fa5d2d63700 1 --2- 172.21.15.42:0/2880440593 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa5cc146f60 0x7fa5cc145d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:29:19.369 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.355+0000 7fa5d2562700 1 --2- 172.21.15.42:0/2880440593 >> v2:172.21.15.40:3300/0 conn(0x7fa5cc0a5d00 0x7fa5cc148050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:29:19.370 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.355+0000 7fa5c37fe700 1 -- 172.21.15.42:0/2880440593 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 737259532 0 0) 0x7fa5cc11c030 con 0x7fa5cc1462a0 2019-12-05T05:29:19.370 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.355+0000 7fa5c37fe700 1 -- 172.21.15.42:0/2880440593 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5b40030b0 con 0x7fa5cc1462a0 2019-12-05T05:29:19.370 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.355+0000 7fa5d2562700 1 -- 172.21.15.42:0/2880440593 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa5cc146f60 msgr2=0x7fa5cc145d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:29:19.370 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.355+0000 7fa5d2562700 1 --2- 172.21.15.42:0/2880440593 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa5cc146f60 0x7fa5cc145d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:29:19.370 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.355+0000 7fa5d2562700 1 -- 172.21.15.42:0/2880440593 >> v1:172.21.15.42:6789/0 conn(0x7fa5cc1462a0 legacy=0x7fa5cc07d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:29:19.370 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.355+0000 7fa5d2562700 1 -- 172.21.15.42:0/2880440593 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5cc07dfc0 con 0x7fa5cc0a5d00 2019-12-05T05:29:19.370 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.355+0000 7fa5d2d63700 1 --2- 172.21.15.42:0/2880440593 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa5cc146f60 0x7fa5cc145d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:29:19.370 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.355+0000 7fa5d2562700 1 --2- 172.21.15.42:0/2880440593 >> v2:172.21.15.40:3300/0 conn(0x7fa5cc0a5d00 0x7fa5cc148050 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fa5bc002f30 tx=0x7fa5bc010af0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:19.371 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.355+0000 7fa5d2562700 1 --2- 172.21.15.42:0/2880440593 >> v2:172.21.15.40:3300/0 conn(0x7fa5cc0a5d00 0x7fa5cc148050 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fa5bc002f30 tx=0x7fa5bc010af0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:19.371 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.355+0000 7fa5d2562700 1 --2- 172.21.15.42:0/2880440593 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa5cc0a5d00 0x7fa5cc148050 secure :-1 s=READY pgs=743 cs=0 l=1 rx=0x7fa5bc002f30 tx=0x7fa5bc010af0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:29:19.371 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.355+0000 7fa5d2562700 1 --2- 172.21.15.42:0/2880440593 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa5cc0a5d00 0x7fa5cc148050 secure :-1 s=THROTTLE_DONE pgs=743 cs=0 l=1 rx=0x7fa5bc002f30 tx=0x7fa5bc010af0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:19.371 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.355+0000 7fa5c37fe700 1 -- 172.21.15.42:0/2880440593 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa5bc012980 con 0x7fa5cc0a5d00 2019-12-05T05:29:19.372 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.355+0000 7fa5d2562700 1 --2- 172.21.15.42:0/2880440593 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa5cc0a5d00 0x7fa5cc148050 secure :-1 s=THROTTLE_DONE pgs=743 cs=0 l=1 rx=0x7fa5bc002f30 tx=0x7fa5bc010af0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:19.372 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.355+0000 7fa5d2562700 1 --2- 172.21.15.42:0/2880440593 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa5cc0a5d00 0x7fa5cc148050 secure :-1 s=THROTTLE_DONE pgs=743 cs=0 l=1 rx=0x7fa5bc002f30 tx=0x7fa5bc010af0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:19.372 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.355+0000 7fa5c37fe700 1 -- 172.21.15.42:0/2880440593 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5bc0125a0 con 0x7fa5cc0a5d00 2019-12-05T05:29:19.372 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.355+0000 7fa5d47c6700 1 -- 172.21.15.42:0/2880440593 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5cc1485a0 con 0x7fa5cc0a5d00 2019-12-05T05:29:19.372 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.355+0000 7fa5c37fe700 1 -- 172.21.15.42:0/2880440593 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa5bc005540 con 0x7fa5cc0a5d00 2019-12-05T05:29:19.372 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.355+0000 7fa5d47c6700 1 -- 172.21.15.42:0/2880440593 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5cc134760 con 0x7fa5cc0a5d00 2019-12-05T05:29:19.372 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.359+0000 7fa5d2562700 1 --2- 172.21.15.42:0/2880440593 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa5cc0a5d00 0x7fa5cc148050 secure :-1 s=THROTTLE_DONE pgs=743 cs=0 l=1 rx=0x7fa5bc002f30 tx=0x7fa5bc010af0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:19.373 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.359+0000 7fa5c37fe700 1 -- 172.21.15.42:0/2880440593 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fa5bc007180 con 0x7fa5cc0a5d00 2019-12-05T05:29:19.373 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.359+0000 7fa5d2562700 1 --2- 172.21.15.42:0/2880440593 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa5cc0a5d00 0x7fa5cc148050 secure :-1 s=THROTTLE_DONE pgs=743 cs=0 l=1 rx=0x7fa5bc002f30 tx=0x7fa5bc010af0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:19.373 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.359+0000 7fa5c37fe700 1 --2- 172.21.15.42:0/2880440593 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa5b403b280 0x7fa5b403d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:29:19.373 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.359+0000 7fa5c37fe700 1 -- 172.21.15.42:0/2880440593 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7579+0+0 (secure 0 0 0) 0x7fa5bc04aee0 con 0x7fa5cc0a5d00 2019-12-05T05:29:19.373 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.359+0000 7fa5d2d63700 1 --2- 172.21.15.42:0/2880440593 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa5b403b280 0x7fa5b403d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:29:19.373 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.359+0000 7fa5d2d63700 1 --2- 172.21.15.42:0/2880440593 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa5b403b280 0x7fa5b403d790 crc :-1 s=READY pgs=417 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:29:19.373 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.359+0000 7fa5d47c6700 1 -- 172.21.15.42:0/2880440593 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5cc05d110 con 0x7fa5cc0a5d00 2019-12-05T05:29:19.377 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.371+0000 7fa5d2562700 1 --2- 172.21.15.42:0/2880440593 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa5cc0a5d00 0x7fa5cc148050 secure :-1 s=THROTTLE_DONE pgs=743 cs=0 l=1 rx=0x7fa5bc002f30 tx=0x7fa5bc010af0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:19.377 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.371+0000 7fa5c37fe700 1 -- 172.21.15.42:0/2880440593 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fa5bc031f60 con 0x7fa5cc0a5d00 2019-12-05T05:29:19.549 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.539+0000 7fa5d47c6700 1 -- 172.21.15.42:0/2880440593 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa5cc0a71e0 con 0x7fa5cc0a5d00 2019-12-05T05:29:19.549 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.539+0000 7fa5d2562700 1 --2- 172.21.15.42:0/2880440593 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa5cc0a5d00 0x7fa5cc148050 secure :-1 s=THROTTLE_DONE pgs=743 cs=0 l=1 rx=0x7fa5bc002f30 tx=0x7fa5bc010af0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:19.549 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.539+0000 7fa5c37fe700 1 -- 172.21.15.42:0/2880440593 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 119 v119) v1 ==== 95+0+1774 (secure 0 0 0) 0x7fa5bc0321c0 con 0x7fa5cc0a5d00 2019-12-05T05:29:19.550 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 119 2019-12-05T05:29:19.550 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:29:19.550 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":119,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":119,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:29:18.667642+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5426},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5426":{"gid":5426,"name":"b","rank":0,"incarnation":119,"state":"up:replay","state_seq":1,"addr":"172.21.15.40:6833/3168718372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3168718372},{"type":"v1","addr":"172.21.15.40:6833","nonce":3168718372}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:29:19.552 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.543+0000 7fa5d47c6700 1 -- 172.21.15.42:0/2880440593 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa5b403b280 msgr2=0x7fa5b403d790 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:29:19.552 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.543+0000 7fa5d47c6700 1 --2- 172.21.15.42:0/2880440593 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa5b403b280 0x7fa5b403d790 crc :-1 s=READY pgs=417 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:29:19.552 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.543+0000 7fa5d47c6700 1 -- 172.21.15.42:0/2880440593 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa5cc0a5d00 msgr2=0x7fa5cc148050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:29:19.552 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.543+0000 7fa5d47c6700 1 --2- 172.21.15.42:0/2880440593 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa5cc0a5d00 0x7fa5cc148050 secure :-1 s=READY pgs=743 cs=0 l=1 rx=0x7fa5bc002f30 tx=0x7fa5bc010af0).stop 2019-12-05T05:29:19.552 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.543+0000 7fa5d47c6700 1 -- 172.21.15.42:0/2880440593 shutdown_connections 2019-12-05T05:29:19.552 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.543+0000 7fa5d47c6700 1 --2- 172.21.15.42:0/2880440593 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa5b403b280 0x7fa5b403d790 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:29:19.552 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.543+0000 7fa5d47c6700 1 --2- 172.21.15.42:0/2880440593 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa5cc146f60 0x7fa5cc145d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:29:19.553 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.543+0000 7fa5d47c6700 1 --2- 172.21.15.42:0/2880440593 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa5cc0a5d00 0x7fa5cc148050 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:29:19.567 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.543+0000 7fa5d47c6700 1 -- 172.21.15.42:0/2880440593 shutdown_connections 2019-12-05T05:29:19.567 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.543+0000 7fa5d47c6700 1 -- 172.21.15.42:0/2880440593 wait complete. 2019-12-05T05:29:19.567 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:19.543+0000 7fa5d47c6700 1 -- 172.21.15.42:0/2880440593 >> 172.21.15.42:0/2880440593 conn(0x7fa5cc0a98e0 msgr2=0x7fa5cc07e600 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:29:19.569 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 119, "feature_flags": { "enable_multiple": false, "ever_enabled_multiple": false }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2019-12-05T05:03:35.604738+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 119, "ever_allowed_features": 2, "explicitly_allowed_features": 2, "failed": [], "flags": 18, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5426": { "addr": "172.21.15.40:6833/3168718372", "addrs": { "addrvec": [ { "addr": "172.21.15.40:6832", "nonce": 3168718372, "type": "v2" }, { "addr": "172.21.15.40:6833", "nonce": 3168718372, "type": "v1" } ] }, "export_targets": [], "features": 4540138292836696063, "flags": 0, "gid": 5426, "incarnation": 119, "name": "b", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 67, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 2, "min_compat_client": "0 (unknown)", "modified": "2019-12-05T05:29:18.667642+0000", "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5426 } } } ], "standbys": [] } 2019-12-05T05:29:21.570 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:29:21.691 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:21.667+0000 7fbfb3652700 1 Processor -- start 2019-12-05T05:29:21.692 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:21.667+0000 7fbfb3652700 1 -- start start 2019-12-05T05:29:21.692 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:21.667+0000 7fbfb3652700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbfac0a7740 0x7fbfac08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:29:21.693 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:21.667+0000 7fbfb3652700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fbfac07e720 0x7fbfac07eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:29:21.693 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:21.667+0000 7fbfb3652700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfac0a0710 con 0x7fbfac09f2e0 2019-12-05T05:29:21.693 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:21.667+0000 7fbfb3652700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfac0a0b00 con 0x7fbfac07e720 2019-12-05T05:29:21.693 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:21.667+0000 7fbfb3652700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfac08e5b0 con 0x7fbfac0a7740 2019-12-05T05:29:21.694 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:21.667+0000 7fbfb13ee700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fbfac07e720 0x7fbfac07eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:29:21.694 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:21.667+0000 7fbfb13ee700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fbfac07e720 0x7fbfac07eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:34816/0 (socket says 172.21.15.42:34816) 2019-12-05T05:29:21.694 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:21.667+0000 7fbfb13ee700 1 -- 172.21.15.42:0/778117000 learned_addr learned my addr 172.21.15.42:0/778117000 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:29:21.694 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:21.667+0000 7fbfb1bef700 1 --2- 172.21.15.42:0/778117000 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbfac0a7740 0x7fbfac08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:29:21.695 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:21.671+0000 7fbfb13ee700 1 -- 172.21.15.42:0/778117000 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbfac0a7740 msgr2=0x7fbfac08df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:29:21.695 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:21.671+0000 7fbfb13ee700 1 --2- 172.21.15.42:0/778117000 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbfac0a7740 0x7fbfac08df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:29:21.695 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:21.671+0000 7fbfb13ee700 1 -- 172.21.15.42:0/778117000 >> v1:172.21.15.42:6789/0 conn(0x7fbfac09f2e0 legacy=0x7fbfac09f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:29:21.695 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:21.671+0000 7fbfb13ee700 1 -- 172.21.15.42:0/778117000 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfac07f0a0 con 0x7fbfac07e720 2019-12-05T05:29:21.695 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:21.671+0000 7fbfb1bef700 1 --2- 172.21.15.42:0/778117000 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbfac0a7740 0x7fbfac08df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:29:21.696 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:21.671+0000 7fbfb13ee700 1 --2- 172.21.15.42:0/778117000 >> v2:172.21.15.42:3300/0 conn(0x7fbfac07e720 0x7fbfac07eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fbf9c008790 tx=0x7fbf9c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:21.696 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:21.671+0000 7fbfb13ee700 1 --2- 172.21.15.42:0/778117000 >> v2:172.21.15.42:3300/0 conn(0x7fbfac07e720 0x7fbfac07eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fbf9c008790 tx=0x7fbf9c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:21.696 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:21.671+0000 7fbfb13ee700 1 --2- 172.21.15.42:0/778117000 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbfac07e720 0x7fbfac07eb50 secure :-1 s=READY pgs=963 cs=0 l=1 rx=0x7fbf9c008790 tx=0x7fbf9c00fd30).ready entity=mon.1 client_cookie=ef532319b901f97e server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:29:21.696 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:21.671+0000 7fbfb13ee700 1 -- 172.21.15.42:0/778117000 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbfac07e720 msgr2=0x7fbfac07eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2019-12-05T05:29:21.696 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:21.671+0000 7fbfb13ee700 1 -- 172.21.15.42:0/778117000 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbfac07e720 msgr2=0x7fbfac07eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2019-12-05T05:29:21.697 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:21.671+0000 7fbfb13ee700 1 --2- 172.21.15.42:0/778117000 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbfac07e720 0x7fbfac07eb50 secure :-1 s=READY pgs=963 cs=0 l=1 rx=0x7fbf9c008790 tx=0x7fbf9c00fd30).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-12-05T05:29:21.697 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:21.671+0000 7fbfb13ee700 1 --2- 172.21.15.42:0/778117000 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbfac07e720 0x7fbfac07eb50 secure :-1 s=READY pgs=963 cs=0 l=1 rx=0x7fbf9c008790 tx=0x7fbf9c00fd30).stop 2019-12-05T05:29:21.697 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:21.671+0000 7fbfb0bed700 1 -- 172.21.15.42:0/778117000 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbfac07e720 msgr2=0x7fbfac07eb50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2019-12-05T05:29:21.697 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:21.671+0000 7fbfb0bed700 1 --2- 172.21.15.42:0/778117000 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbfac07e720 0x7fbfac07eb50 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:29:21.697 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:21.671+0000 7fbfb0bed700 1 --2- 172.21.15.42:0/778117000 >> v2:172.21.15.40:3300/0 conn(0x7fbf9404afb0 0x7fbf9404d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:29:21.697 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:21.671+0000 7fbfb0bed700 1 -- 172.21.15.42:0/778117000 --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf9404dbe0 con 0x7fbf940470a0 2019-12-05T05:29:21.698 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:21.671+0000 7fbfb0bed700 1 -- 172.21.15.42:0/778117000 --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf9404eeb0 con 0x7fbf9404afb0 2019-12-05T05:29:21.699 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:21.671+0000 7fbfb13ee700 1 --2- 172.21.15.42:0/778117000 >> v2:172.21.15.40:3300/0 conn(0x7fbf9404afb0 0x7fbf9404d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:29:21.699 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:21.671+0000 7fbfb0bed700 1 -- 172.21.15.42:0/778117000 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1775633016 0 0) 0x7fbf9404dbe0 con 0x7fbf940470a0 2019-12-05T05:29:21.699 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:21.671+0000 7fbfb0bed700 1 -- 172.21.15.42:0/778117000 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf9404def0 con 0x7fbf940470a0 2019-12-05T05:29:21.700 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:21.671+0000 7fbfb13ee700 1 -- 172.21.15.42:0/778117000 >> v1:172.21.15.40:6789/0 conn(0x7fbf940470a0 legacy=0x7fbf94049570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:29:21.700 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:21.671+0000 7fbfb13ee700 1 -- 172.21.15.42:0/778117000 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf9404f160 con 0x7fbf9404afb0 2019-12-05T05:29:21.700 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:21.671+0000 7fbfb13ee700 1 --2- 172.21.15.42:0/778117000 >> v2:172.21.15.40:3300/0 conn(0x7fbf9404afb0 0x7fbf9404d4c0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fbf9c008790 tx=0x7fbf9c008f80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:21.700 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:21.671+0000 7fbfb13ee700 1 --2- 172.21.15.42:0/778117000 >> v2:172.21.15.40:3300/0 conn(0x7fbf9404afb0 0x7fbf9404d4c0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fbf9c008790 tx=0x7fbf9c008f80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:21.700 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:21.671+0000 7fbfb13ee700 1 --2- 172.21.15.42:0/778117000 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbf9404afb0 0x7fbf9404d4c0 secure :-1 s=READY pgs=745 cs=0 l=1 rx=0x7fbf9c008790 tx=0x7fbf9c008f80).ready entity=mon.0 client_cookie=effcd6d4c108e0e0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:29:21.700 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:21.671+0000 7fbfb13ee700 1 --2- 172.21.15.42:0/778117000 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbf9404afb0 0x7fbf9404d4c0 secure :-1 s=THROTTLE_DONE pgs=745 cs=0 l=1 rx=0x7fbf9c008790 tx=0x7fbf9c008f80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:21.701 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:21.671+0000 7fbfb13ee700 1 --2- 172.21.15.42:0/778117000 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbf9404afb0 0x7fbf9404d4c0 secure :-1 s=THROTTLE_DONE pgs=745 cs=0 l=1 rx=0x7fbf9c008790 tx=0x7fbf9c008f80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:21.701 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:21.671+0000 7fbfb0bed700 1 -- 172.21.15.42:0/778117000 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbf9c012440 con 0x7fbf9404afb0 2019-12-05T05:29:21.701 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:21.671+0000 7fbfb13ee700 1 --2- 172.21.15.42:0/778117000 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbf9404afb0 0x7fbf9404d4c0 secure :-1 s=THROTTLE_DONE pgs=745 cs=0 l=1 rx=0x7fbf9c008790 tx=0x7fbf9c008f80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:21.701 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:21.671+0000 7fbfb0bed700 1 -- 172.21.15.42:0/778117000 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf9c00f490 con 0x7fbf9404afb0 2019-12-05T05:29:21.701 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:21.671+0000 7fbfb0bed700 1 -- 172.21.15.42:0/778117000 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbf9c012a20 con 0x7fbf9404afb0 2019-12-05T05:29:21.701 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:21.671+0000 7fbfb3652700 1 -- 172.21.15.42:0/778117000 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbf9404afb0 msgr2=0x7fbf9404d4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:29:21.701 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:21.671+0000 7fbfb3652700 1 --2- 172.21.15.42:0/778117000 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbf9404afb0 0x7fbf9404d4c0 secure :-1 s=READY pgs=745 cs=0 l=1 rx=0x7fbf9c008790 tx=0x7fbf9c008f80).stop 2019-12-05T05:29:21.702 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:21.671+0000 7fbfb23f0700 1 -- 172.21.15.42:0/778117000 reap_dead start 2019-12-05T05:29:21.702 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:21.671+0000 7fbfb3652700 1 -- 172.21.15.42:0/778117000 shutdown_connections 2019-12-05T05:29:21.702 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:21.671+0000 7fbfb3652700 1 --2- 172.21.15.42:0/778117000 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbf9404afb0 0x7fbf9404d4c0 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:29:21.702 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:21.671+0000 7fbfb3652700 1 --2- 172.21.15.42:0/778117000 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbfac07e720 0x7fbfac07eb50 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:29:21.702 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:21.671+0000 7fbfb3652700 1 -- 172.21.15.42:0/778117000 shutdown_connections 2019-12-05T05:29:21.702 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:21.671+0000 7fbfb3652700 1 -- 172.21.15.42:0/778117000 wait complete. 2019-12-05T05:29:21.702 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:21.671+0000 7fbfb3652700 1 -- 172.21.15.42:0/778117000 >> 172.21.15.42:0/778117000 conn(0x7fbfac0aabe0 msgr2=0x7fbfac0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:29:21.703 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:21.675+0000 7fbfb3652700 1 Processor -- start 2019-12-05T05:29:21.703 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:21.675+0000 7fbfb3652700 1 -- start start 2019-12-05T05:29:21.703 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:21.675+0000 7fbfb3652700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fbfac119930 0x7fbfac116300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:29:21.703 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:21.679+0000 7fbfb3652700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfac117ba0 con 0x7fbfac11ab50 2019-12-05T05:29:21.704 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:21.679+0000 7fbfb3652700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfac1297a0 con 0x7fbfac11b8b0 2019-12-05T05:29:21.704 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:21.679+0000 7fbfb3652700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfac11a110 con 0x7fbfac119930 2019-12-05T05:29:21.704 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:21.679+0000 7fbfb13ee700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fbfac119930 0x7fbfac116300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:29:21.704 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:21.679+0000 7fbfb23f0700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fbfac11b8b0 0x7fbfac136a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:34868/0 (socket says 172.21.15.42:34868) 2019-12-05T05:29:21.704 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:21.679+0000 7fbfb23f0700 1 -- 172.21.15.42:0/1529880374 learned_addr learned my addr 172.21.15.42:0/1529880374 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:29:21.705 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:21.679+0000 7fbfb13ee700 1 -- 172.21.15.42:0/1529880374 >> v1:172.21.15.42:6789/0 conn(0x7fbfac11b8b0 legacy=0x7fbfac136a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:29:21.705 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:21.679+0000 7fbfb13ee700 1 -- 172.21.15.42:0/1529880374 >> v1:172.21.15.40:6789/0 conn(0x7fbfac11ab50 legacy=0x7fbfac128fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:29:21.705 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:21.679+0000 7fbfa27fc700 1 -- 172.21.15.42:0/1529880374 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 634805689 0 0) 0x7fbfac117ba0 con 0x7fbfac11ab50 2019-12-05T05:29:21.705 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:21.679+0000 7fbfb13ee700 1 -- 172.21.15.42:0/1529880374 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfac116850 con 0x7fbfac119930 2019-12-05T05:29:21.705 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:21.679+0000 7fbfb13ee700 1 --2- 172.21.15.42:0/1529880374 >> v2:172.21.15.42:3300/0 conn(0x7fbfac119930 0x7fbfac116300 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fbf9c008790 tx=0x7fbf9c00f670).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:21.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:21.679+0000 7fbfb13ee700 1 --2- 172.21.15.42:0/1529880374 >> v2:172.21.15.42:3300/0 conn(0x7fbfac119930 0x7fbfac116300 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fbf9c008790 tx=0x7fbf9c00f670).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:21.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:21.679+0000 7fbfb13ee700 1 --2- 172.21.15.42:0/1529880374 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbfac119930 0x7fbfac116300 secure :-1 s=READY pgs=965 cs=0 l=1 rx=0x7fbf9c008790 tx=0x7fbf9c00f670).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:29:21.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:21.679+0000 7fbfb13ee700 1 --2- 172.21.15.42:0/1529880374 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbfac119930 0x7fbfac116300 secure :-1 s=THROTTLE_DONE pgs=965 cs=0 l=1 rx=0x7fbf9c008790 tx=0x7fbf9c00f670).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:21.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:21.679+0000 7fbfa27fc700 1 -- 172.21.15.42:0/1529880374 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbf9c0117b0 con 0x7fbfac119930 2019-12-05T05:29:21.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:21.679+0000 7fbfb3652700 1 -- 172.21.15.42:0/1529880374 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfac116b20 con 0x7fbfac119930 2019-12-05T05:29:21.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:21.679+0000 7fbfb13ee700 1 --2- 172.21.15.42:0/1529880374 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbfac119930 0x7fbfac116300 secure :-1 s=THROTTLE_DONE pgs=965 cs=0 l=1 rx=0x7fbf9c008790 tx=0x7fbf9c00f670).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:21.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:21.679+0000 7fbfb3652700 1 -- 172.21.15.42:0/1529880374 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfac138280 con 0x7fbfac119930 2019-12-05T05:29:21.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:21.679+0000 7fbfa27fc700 1 -- 172.21.15.42:0/1529880374 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf9c012640 con 0x7fbfac119930 2019-12-05T05:29:21.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:21.679+0000 7fbfb13ee700 1 --2- 172.21.15.42:0/1529880374 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbfac119930 0x7fbfac116300 secure :-1 s=THROTTLE_DONE pgs=965 cs=0 l=1 rx=0x7fbf9c008790 tx=0x7fbf9c00f670).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:21.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:21.679+0000 7fbfa27fc700 1 -- 172.21.15.42:0/1529880374 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbf9c011d90 con 0x7fbfac119930 2019-12-05T05:29:21.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:21.679+0000 7fbfb13ee700 1 --2- 172.21.15.42:0/1529880374 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbfac119930 0x7fbfac116300 secure :-1 s=THROTTLE_DONE pgs=965 cs=0 l=1 rx=0x7fbf9c008790 tx=0x7fbf9c00f670).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:21.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:21.679+0000 7fbfa27fc700 1 -- 172.21.15.42:0/1529880374 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fbf9c005540 con 0x7fbfac119930 2019-12-05T05:29:21.708 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:21.679+0000 7fbfb13ee700 1 --2- 172.21.15.42:0/1529880374 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbfac119930 0x7fbfac116300 secure :-1 s=THROTTLE_DONE pgs=965 cs=0 l=1 rx=0x7fbf9c008790 tx=0x7fbf9c00f670).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:21.708 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:21.683+0000 7fbfa27fc700 1 --2- 172.21.15.42:0/1529880374 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbf8c03ac50 0x7fbf8c03d160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:29:21.708 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:21.683+0000 7fbfa27fc700 1 -- 172.21.15.42:0/1529880374 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7579+0+0 (secure 0 0 0) 0x7fbf9c04b990 con 0x7fbfac119930 2019-12-05T05:29:21.708 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:21.683+0000 7fbfb1bef700 1 --2- 172.21.15.42:0/1529880374 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbf8c03ac50 0x7fbf8c03d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:29:21.709 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:21.683+0000 7fbfb1bef700 1 --2- 172.21.15.42:0/1529880374 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbf8c03ac50 0x7fbf8c03d160 crc :-1 s=READY pgs=418 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:29:21.709 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:21.683+0000 7fbfb3652700 1 -- 172.21.15.42:0/1529880374 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfac05d110 con 0x7fbfac119930 2019-12-05T05:29:21.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:21.691+0000 7fbfb13ee700 1 --2- 172.21.15.42:0/1529880374 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbfac119930 0x7fbfac116300 secure :-1 s=THROTTLE_DONE pgs=965 cs=0 l=1 rx=0x7fbf9c008790 tx=0x7fbf9c00f670).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:21.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:21.691+0000 7fbfa27fc700 1 -- 172.21.15.42:0/1529880374 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fbf9c007f50 con 0x7fbfac119930 2019-12-05T05:29:21.861 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:21.855+0000 7fbfb3652700 1 -- 172.21.15.42:0/1529880374 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbfac116d30 con 0x7fbfac119930 2019-12-05T05:29:21.862 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:21.855+0000 7fbfb13ee700 1 --2- 172.21.15.42:0/1529880374 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbfac119930 0x7fbfac116300 secure :-1 s=THROTTLE_DONE pgs=965 cs=0 l=1 rx=0x7fbf9c008790 tx=0x7fbf9c00f670).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:21.862 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:21.855+0000 7fbfa27fc700 1 -- 172.21.15.42:0/1529880374 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 119 v119) v1 ==== 95+0+1774 (secure 0 0 0) 0x7fbf9c005850 con 0x7fbfac119930 2019-12-05T05:29:21.862 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 119 2019-12-05T05:29:21.863 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:29:21.863 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":119,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":119,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:29:18.667642+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5426},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5426":{"gid":5426,"name":"b","rank":0,"incarnation":119,"state":"up:replay","state_seq":1,"addr":"172.21.15.40:6833/3168718372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3168718372},{"type":"v1","addr":"172.21.15.40:6833","nonce":3168718372}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:29:21.865 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:21.859+0000 7fbfb3652700 1 -- 172.21.15.42:0/1529880374 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbf8c03ac50 msgr2=0x7fbf8c03d160 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:29:21.865 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:21.859+0000 7fbfb3652700 1 --2- 172.21.15.42:0/1529880374 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbf8c03ac50 0x7fbf8c03d160 crc :-1 s=READY pgs=418 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:29:21.865 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:21.859+0000 7fbfb3652700 1 -- 172.21.15.42:0/1529880374 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbfac119930 msgr2=0x7fbfac116300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:29:21.865 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:21.859+0000 7fbfb3652700 1 --2- 172.21.15.42:0/1529880374 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbfac119930 0x7fbfac116300 secure :-1 s=READY pgs=965 cs=0 l=1 rx=0x7fbf9c008790 tx=0x7fbf9c00f670).stop 2019-12-05T05:29:21.866 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:21.859+0000 7fbfb3652700 1 -- 172.21.15.42:0/1529880374 shutdown_connections 2019-12-05T05:29:21.866 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:21.859+0000 7fbfb3652700 1 --2- 172.21.15.42:0/1529880374 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbf8c03ac50 0x7fbf8c03d160 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:29:21.866 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:21.859+0000 7fbfb3652700 1 --2- 172.21.15.42:0/1529880374 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbfac119930 0x7fbfac116300 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:29:21.866 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:21.859+0000 7fbfb3652700 1 -- 172.21.15.42:0/1529880374 shutdown_connections 2019-12-05T05:29:21.866 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:21.859+0000 7fbfb3652700 1 -- 172.21.15.42:0/1529880374 wait complete. 2019-12-05T05:29:21.866 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:21.859+0000 7fbfb3652700 1 -- 172.21.15.42:0/1529880374 >> 172.21.15.42:0/1529880374 conn(0x7fbfac0a7740 msgr2=0x7fbfac08eb70 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:29:21.875 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2019-12-05T05:29:23.875 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:29:23.979 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:23.967+0000 7fa662c13700 1 Processor -- start 2019-12-05T05:29:23.980 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:23.967+0000 7fa662c13700 1 -- start start 2019-12-05T05:29:23.980 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:23.971+0000 7fa662c13700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fa65c08df80 0x7fa65c09f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:29:23.980 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:23.971+0000 7fa662c13700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa65c09f830 0x7fa65c14efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:29:23.980 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:23.971+0000 7fa662c13700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa65c09fc60 con 0x7fa65c0a7740 2019-12-05T05:29:23.981 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:23.971+0000 7fa662c13700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa65c0a0b00 con 0x7fa65c09f830 2019-12-05T05:29:23.981 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:23.971+0000 7fa662c13700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa65c08e5b0 con 0x7fa65c08df80 2019-12-05T05:29:23.981 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:23.971+0000 7fa6609af700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fa65c08df80 0x7fa65c09f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:29:23.981 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:23.971+0000 7fa6609af700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fa65c08df80 0x7fa65c09f2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:34842/0 (socket says 172.21.15.42:34842) 2019-12-05T05:29:23.982 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:23.971+0000 7fa6609af700 1 -- 172.21.15.42:0/3296298304 learned_addr learned my addr 172.21.15.42:0/3296298304 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:29:23.982 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:23.971+0000 7fa6619b1700 1 --2- 172.21.15.42:0/3296298304 >> v2:172.21.15.40:3300/0 conn(0x7fa65c09f830 0x7fa65c14efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:29:23.983 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:23.971+0000 7fa6609af700 1 -- 172.21.15.42:0/3296298304 >> v2:172.21.15.40:3300/0 conn(0x7fa65c09f830 msgr2=0x7fa65c14efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:29:23.983 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:23.971+0000 7fa6609af700 1 --2- 172.21.15.42:0/3296298304 >> v2:172.21.15.40:3300/0 conn(0x7fa65c09f830 0x7fa65c14efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:29:23.983 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:23.971+0000 7fa6609af700 1 -- 172.21.15.42:0/3296298304 >> v1:172.21.15.40:6789/0 conn(0x7fa65c0a7740 legacy=0x7fa65c07e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:29:23.983 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:23.971+0000 7fa6609af700 1 -- 172.21.15.42:0/3296298304 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa65c0a0710 con 0x7fa65c08df80 2019-12-05T05:29:23.984 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:23.971+0000 7fa6609af700 1 --2- 172.21.15.42:0/3296298304 >> v2:172.21.15.42:3300/0 conn(0x7fa65c08df80 0x7fa65c09f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fa64c002f30 tx=0x7fa64c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:23.984 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:23.971+0000 7fa6609af700 1 --2- 172.21.15.42:0/3296298304 >> v2:172.21.15.42:3300/0 conn(0x7fa65c08df80 0x7fa65c09f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fa64c002f30 tx=0x7fa64c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:23.984 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:23.971+0000 7fa6609af700 1 --2- 172.21.15.42:0/3296298304 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa65c08df80 0x7fa65c09f2e0 secure :-1 s=READY pgs=966 cs=0 l=1 rx=0x7fa64c002f30 tx=0x7fa64c00fd30).ready entity=mon.1 client_cookie=495b9bcd1536421a server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:29:23.985 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:23.971+0000 7fa6609af700 1 --2- 172.21.15.42:0/3296298304 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa65c08df80 0x7fa65c09f2e0 secure :-1 s=THROTTLE_DONE pgs=966 cs=0 l=1 rx=0x7fa64c002f30 tx=0x7fa64c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:23.985 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:23.971+0000 7fa653fff700 1 -- 172.21.15.42:0/3296298304 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa64c008c90 con 0x7fa65c08df80 2019-12-05T05:29:23.985 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:23.971+0000 7fa6609af700 1 --2- 172.21.15.42:0/3296298304 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa65c08df80 0x7fa65c09f2e0 secure :-1 s=THROTTLE_DONE pgs=966 cs=0 l=1 rx=0x7fa64c002f30 tx=0x7fa64c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:23.985 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:23.971+0000 7fa653fff700 1 -- 172.21.15.42:0/3296298304 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa64c0088b0 con 0x7fa65c08df80 2019-12-05T05:29:23.986 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:23.971+0000 7fa6609af700 1 --2- 172.21.15.42:0/3296298304 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa65c08df80 0x7fa65c09f2e0 secure :-1 s=THROTTLE_DONE pgs=966 cs=0 l=1 rx=0x7fa64c002f30 tx=0x7fa64c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:23.986 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:23.971+0000 7fa653fff700 1 -- 172.21.15.42:0/3296298304 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa64c011ad0 con 0x7fa65c08df80 2019-12-05T05:29:23.986 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:23.971+0000 7fa662c13700 1 -- 172.21.15.42:0/3296298304 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa65c08df80 msgr2=0x7fa65c09f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:29:23.986 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:23.971+0000 7fa662c13700 1 --2- 172.21.15.42:0/3296298304 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa65c08df80 0x7fa65c09f2e0 secure :-1 s=READY pgs=966 cs=0 l=1 rx=0x7fa64c002f30 tx=0x7fa64c00fd30).stop 2019-12-05T05:29:23.987 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:23.971+0000 7fa662c13700 1 -- 172.21.15.42:0/3296298304 shutdown_connections 2019-12-05T05:29:23.988 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:23.971+0000 7fa662c13700 1 --2- 172.21.15.42:0/3296298304 >> v2:172.21.15.40:3300/0 conn(0x7fa65c09f830 0x7fa65c14efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:29:23.988 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:23.971+0000 7fa662c13700 1 --2- 172.21.15.42:0/3296298304 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa65c08df80 0x7fa65c09f2e0 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:29:23.988 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:23.971+0000 7fa662c13700 1 -- 172.21.15.42:0/3296298304 shutdown_connections 2019-12-05T05:29:23.989 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:23.971+0000 7fa662c13700 1 -- 172.21.15.42:0/3296298304 wait complete. 2019-12-05T05:29:23.989 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:23.971+0000 7fa662c13700 1 -- 172.21.15.42:0/3296298304 >> 172.21.15.42:0/3296298304 conn(0x7fa65c0aabe0 msgr2=0x7fa65c0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:29:23.995 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:23.975+0000 7fa662c13700 1 Processor -- start 2019-12-05T05:29:23.996 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:23.975+0000 7fa662c13700 1 -- start start 2019-12-05T05:29:23.996 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:23.975+0000 7fa662c13700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa65c131370 0x7fa65c145020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:29:23.996 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:23.975+0000 7fa662c13700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fa65c060360 0x7fa65c1444b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:29:23.996 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:23.975+0000 7fa662c13700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa65c060790 con 0x7fa65c0a3a60 2019-12-05T05:29:23.997 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:23.975+0000 7fa662c13700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa65c060a70 con 0x7fa65c060360 2019-12-05T05:29:23.997 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:23.975+0000 7fa662c13700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa65c060d10 con 0x7fa65c131370 2019-12-05T05:29:23.997 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:23.975+0000 7fa6619b1700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fa65c060360 0x7fa65c1444b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:29:23.997 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:23.975+0000 7fa6609af700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fa65c0a3a60 0x7fa65c0655f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:34886/0 (socket says 172.21.15.42:34886) 2019-12-05T05:29:23.998 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:23.975+0000 7fa6609af700 1 -- 172.21.15.42:0/2680186606 learned_addr learned my addr 172.21.15.42:0/2680186606 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:29:23.999 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:23.975+0000 7fa6611b0700 1 --2- 172.21.15.42:0/2680186606 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa65c131370 0x7fa65c145020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:29:23.999 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:23.975+0000 7fa6619b1700 1 -- 172.21.15.42:0/2680186606 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa65c131370 msgr2=0x7fa65c145020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:29:23.999 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:23.975+0000 7fa6619b1700 1 --2- 172.21.15.42:0/2680186606 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa65c131370 0x7fa65c145020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:29:24.000 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:23.975+0000 7fa6619b1700 1 -- 172.21.15.42:0/2680186606 >> v1:172.21.15.42:6789/0 conn(0x7fa65c0a3a60 legacy=0x7fa65c0655f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:29:24.000 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:23.975+0000 7fa6619b1700 1 -- 172.21.15.42:0/2680186606 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa65c144a00 con 0x7fa65c060360 2019-12-05T05:29:24.000 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:23.975+0000 7fa651ffb700 1 -- 172.21.15.42:0/2680186606 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1147954639 0 0) 0x7fa64c008470 con 0x7fa65c0a3a60 2019-12-05T05:29:24.000 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:23.975+0000 7fa6619b1700 1 --2- 172.21.15.42:0/2680186606 >> v2:172.21.15.42:3300/0 conn(0x7fa65c060360 0x7fa65c1444b0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fa65c090380 tx=0x7fa654010d40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:24.001 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:23.975+0000 7fa6619b1700 1 --2- 172.21.15.42:0/2680186606 >> v2:172.21.15.42:3300/0 conn(0x7fa65c060360 0x7fa65c1444b0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fa65c090380 tx=0x7fa654010d40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:24.001 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:23.975+0000 7fa6619b1700 1 --2- 172.21.15.42:0/2680186606 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa65c060360 0x7fa65c1444b0 secure :-1 s=READY pgs=968 cs=0 l=1 rx=0x7fa65c090380 tx=0x7fa654010d40).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:29:24.001 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:23.975+0000 7fa6619b1700 1 --2- 172.21.15.42:0/2680186606 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa65c060360 0x7fa65c1444b0 secure :-1 s=THROTTLE_DONE pgs=968 cs=0 l=1 rx=0x7fa65c090380 tx=0x7fa654010d40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:24.001 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:23.975+0000 7fa651ffb700 1 -- 172.21.15.42:0/2680186606 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa654012a30 con 0x7fa65c060360 2019-12-05T05:29:24.002 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:23.975+0000 7fa6619b1700 1 --2- 172.21.15.42:0/2680186606 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa65c060360 0x7fa65c1444b0 secure :-1 s=THROTTLE_DONE pgs=968 cs=0 l=1 rx=0x7fa65c090380 tx=0x7fa654010d40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:24.002 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:23.975+0000 7fa6619b1700 1 --2- 172.21.15.42:0/2680186606 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa65c060360 0x7fa65c1444b0 secure :-1 s=THROTTLE_DONE pgs=968 cs=0 l=1 rx=0x7fa65c090380 tx=0x7fa654010d40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:24.002 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:23.975+0000 7fa651ffb700 1 -- 172.21.15.42:0/2680186606 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa654012650 con 0x7fa65c060360 2019-12-05T05:29:24.002 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:23.975+0000 7fa662c13700 1 -- 172.21.15.42:0/2680186606 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa65c1387c0 con 0x7fa65c060360 2019-12-05T05:29:24.003 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:23.975+0000 7fa651ffb700 1 -- 172.21.15.42:0/2680186606 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa65400e760 con 0x7fa65c060360 2019-12-05T05:29:24.003 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:23.975+0000 7fa662c13700 1 -- 172.21.15.42:0/2680186606 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa65c138cf0 con 0x7fa65c060360 2019-12-05T05:29:24.003 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:23.979+0000 7fa6619b1700 1 --2- 172.21.15.42:0/2680186606 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa65c060360 0x7fa65c1444b0 secure :-1 s=THROTTLE_DONE pgs=968 cs=0 l=1 rx=0x7fa65c090380 tx=0x7fa654010d40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:24.003 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:23.979+0000 7fa651ffb700 1 -- 172.21.15.42:0/2680186606 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fa65400ea70 con 0x7fa65c060360 2019-12-05T05:29:24.004 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:23.979+0000 7fa6619b1700 1 --2- 172.21.15.42:0/2680186606 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa65c060360 0x7fa65c1444b0 secure :-1 s=THROTTLE_DONE pgs=968 cs=0 l=1 rx=0x7fa65c090380 tx=0x7fa654010d40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:24.004 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:23.979+0000 7fa651ffb700 1 --2- 172.21.15.42:0/2680186606 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa63c03ac30 0x7fa63c03d140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:29:24.004 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:23.979+0000 7fa651ffb700 1 -- 172.21.15.42:0/2680186606 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7579+0+0 (secure 0 0 0) 0x7fa65404b680 con 0x7fa65c060360 2019-12-05T05:29:24.004 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:23.979+0000 7fa6611b0700 1 --2- 172.21.15.42:0/2680186606 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa63c03ac30 0x7fa63c03d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:29:24.005 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:23.979+0000 7fa6611b0700 1 --2- 172.21.15.42:0/2680186606 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa63c03ac30 0x7fa63c03d140 crc :-1 s=READY pgs=419 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:29:24.005 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:23.983+0000 7fa662c13700 1 -- 172.21.15.42:0/2680186606 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa65c05d110 con 0x7fa65c060360 2019-12-05T05:29:24.006 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:23.991+0000 7fa6619b1700 1 --2- 172.21.15.42:0/2680186606 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa65c060360 0x7fa65c1444b0 secure :-1 s=THROTTLE_DONE pgs=968 cs=0 l=1 rx=0x7fa65c090380 tx=0x7fa654010d40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:24.006 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:23.991+0000 7fa651ffb700 1 -- 172.21.15.42:0/2680186606 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fa654032700 con 0x7fa65c060360 2019-12-05T05:29:24.167 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:24.163+0000 7fa662c13700 1 -- 172.21.15.42:0/2680186606 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa65c05b8c0 con 0x7fa65c060360 2019-12-05T05:29:24.168 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:24.163+0000 7fa6619b1700 1 --2- 172.21.15.42:0/2680186606 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa65c060360 0x7fa65c1444b0 secure :-1 s=THROTTLE_DONE pgs=968 cs=0 l=1 rx=0x7fa65c090380 tx=0x7fa654010d40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:24.168 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:24.163+0000 7fa651ffb700 1 -- 172.21.15.42:0/2680186606 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 121 v121) v1 ==== 95+0+1775 (secure 0 0 0) 0x7fa654032960 con 0x7fa65c060360 2019-12-05T05:29:24.169 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 121 2019-12-05T05:29:24.169 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:29:24.170 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":121,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":121,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:29:23.924545+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5426},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5426":{"gid":5426,"name":"b","rank":0,"incarnation":119,"state":"up:rejoin","state_seq":17,"addr":"172.21.15.40:6833/3168718372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3168718372},{"type":"v1","addr":"172.21.15.40:6833","nonce":3168718372}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:29:24.171 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:24.163+0000 7fa662c13700 1 -- 172.21.15.42:0/2680186606 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa63c03ac30 msgr2=0x7fa63c03d140 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:29:24.171 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:24.163+0000 7fa662c13700 1 --2- 172.21.15.42:0/2680186606 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa63c03ac30 0x7fa63c03d140 crc :-1 s=READY pgs=419 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:29:24.171 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:24.163+0000 7fa662c13700 1 -- 172.21.15.42:0/2680186606 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa65c060360 msgr2=0x7fa65c1444b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:29:24.171 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:24.163+0000 7fa662c13700 1 --2- 172.21.15.42:0/2680186606 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa65c060360 0x7fa65c1444b0 secure :-1 s=READY pgs=968 cs=0 l=1 rx=0x7fa65c090380 tx=0x7fa654010d40).stop 2019-12-05T05:29:24.172 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:24.163+0000 7fa662c13700 1 -- 172.21.15.42:0/2680186606 shutdown_connections 2019-12-05T05:29:24.172 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:24.163+0000 7fa662c13700 1 --2- 172.21.15.42:0/2680186606 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa63c03ac30 0x7fa63c03d140 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:29:24.172 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:24.163+0000 7fa662c13700 1 --2- 172.21.15.42:0/2680186606 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa65c060360 0x7fa65c1444b0 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:29:24.173 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:24.163+0000 7fa662c13700 1 --2- 172.21.15.42:0/2680186606 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa65c131370 0x7fa65c145020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:29:24.173 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:24.163+0000 7fa662c13700 1 -- 172.21.15.42:0/2680186606 shutdown_connections 2019-12-05T05:29:24.173 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:24.163+0000 7fa662c13700 1 -- 172.21.15.42:0/2680186606 wait complete. 2019-12-05T05:29:24.173 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:24.163+0000 7fa662c13700 1 -- 172.21.15.42:0/2680186606 >> 172.21.15.42:0/2680186606 conn(0x7fa65c05a120 msgr2=0x7fa65c0a6a60 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:29:24.182 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2019-12-05T05:29:26.183 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:29:26.286 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:26.275+0000 7efe52f31700 1 Processor -- start 2019-12-05T05:29:26.287 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:26.275+0000 7efe52f31700 1 -- start start 2019-12-05T05:29:26.288 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:26.275+0000 7efe52f31700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7efe4c08df80 0x7efe4c09f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:29:26.288 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:26.275+0000 7efe52f31700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efe4c09f830 0x7efe4c14efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:29:26.288 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:26.275+0000 7efe52f31700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe4c09fc60 con 0x7efe4c0a7740 2019-12-05T05:29:26.288 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:26.275+0000 7efe52f31700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe4c0a0b00 con 0x7efe4c08df80 2019-12-05T05:29:26.288 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:26.275+0000 7efe52f31700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe4c08e5b0 con 0x7efe4c09f830 2019-12-05T05:29:26.288 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:26.275+0000 7efe514ce700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7efe4c0a7740 0x7efe4c07e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:34896/0 (socket says 172.21.15.42:34896) 2019-12-05T05:29:26.288 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:26.275+0000 7efe514ce700 1 -- 172.21.15.42:0/3584337009 learned_addr learned my addr 172.21.15.42:0/3584337009 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:29:26.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:26.275+0000 7efe50ccd700 1 --2- 172.21.15.42:0/3584337009 >> v2:172.21.15.40:3300/0 conn(0x7efe4c08df80 0x7efe4c09f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:29:26.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:26.275+0000 7efe51ccf700 1 --2- 172.21.15.42:0/3584337009 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efe4c09f830 0x7efe4c14efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:29:26.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:26.275+0000 7efe4bfff700 1 -- 172.21.15.42:0/3584337009 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2284534724 0 0) 0x7efe4c09fc60 con 0x7efe4c0a7740 2019-12-05T05:29:26.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:26.275+0000 7efe4bfff700 1 -- 172.21.15.42:0/3584337009 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe3c0030b0 con 0x7efe4c0a7740 2019-12-05T05:29:26.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:26.279+0000 7efe50ccd700 1 -- 172.21.15.42:0/3584337009 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efe4c09f830 msgr2=0x7efe4c14efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:29:26.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:26.279+0000 7efe50ccd700 1 --2- 172.21.15.42:0/3584337009 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efe4c09f830 0x7efe4c14efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:29:26.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:26.279+0000 7efe4bfff700 1 -- 172.21.15.42:0/3584337009 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2701345291 0 0) 0x7efe3c0030b0 con 0x7efe4c0a7740 2019-12-05T05:29:26.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:26.279+0000 7efe50ccd700 1 -- 172.21.15.42:0/3584337009 >> v1:172.21.15.42:6789/0 conn(0x7efe4c0a7740 legacy=0x7efe4c07e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:29:26.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:26.279+0000 7efe50ccd700 1 -- 172.21.15.42:0/3584337009 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe4c0a0710 con 0x7efe4c08df80 2019-12-05T05:29:26.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:26.279+0000 7efe50ccd700 1 --2- 172.21.15.42:0/3584337009 >> v2:172.21.15.40:3300/0 conn(0x7efe4c08df80 0x7efe4c09f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7efe38008790 tx=0x7efe40001820).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:26.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:26.279+0000 7efe50ccd700 1 --2- 172.21.15.42:0/3584337009 >> v2:172.21.15.40:3300/0 conn(0x7efe4c08df80 0x7efe4c09f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7efe38008790 tx=0x7efe40001820).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:26.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:26.279+0000 7efe50ccd700 1 --2- 172.21.15.42:0/3584337009 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efe4c08df80 0x7efe4c09f2e0 secure :-1 s=READY pgs=748 cs=0 l=1 rx=0x7efe38008790 tx=0x7efe40001820).ready entity=mon.0 client_cookie=7e71f12d5b5bc999 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:29:26.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:26.279+0000 7efe50ccd700 1 --2- 172.21.15.42:0/3584337009 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efe4c08df80 0x7efe4c09f2e0 secure :-1 s=THROTTLE_DONE pgs=748 cs=0 l=1 rx=0x7efe38008790 tx=0x7efe40001820).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:26.291 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:26.279+0000 7efe4bfff700 1 -- 172.21.15.42:0/3584337009 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efe40001ad0 con 0x7efe4c08df80 2019-12-05T05:29:26.291 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:26.279+0000 7efe50ccd700 1 --2- 172.21.15.42:0/3584337009 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efe4c08df80 0x7efe4c09f2e0 secure :-1 s=THROTTLE_DONE pgs=748 cs=0 l=1 rx=0x7efe38008790 tx=0x7efe40001820).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:26.291 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:26.279+0000 7efe4bfff700 1 -- 172.21.15.42:0/3584337009 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe400018e0 con 0x7efe4c08df80 2019-12-05T05:29:26.291 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:26.279+0000 7efe50ccd700 1 --2- 172.21.15.42:0/3584337009 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efe4c08df80 0x7efe4c09f2e0 secure :-1 s=THROTTLE_DONE pgs=748 cs=0 l=1 rx=0x7efe38008790 tx=0x7efe40001820).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:26.291 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:26.279+0000 7efe52f31700 1 -- 172.21.15.42:0/3584337009 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efe4c08df80 msgr2=0x7efe4c09f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:29:26.291 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:26.279+0000 7efe52f31700 1 --2- 172.21.15.42:0/3584337009 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efe4c08df80 0x7efe4c09f2e0 secure :-1 s=READY pgs=748 cs=0 l=1 rx=0x7efe38008790 tx=0x7efe40001820).stop 2019-12-05T05:29:26.291 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:26.279+0000 7efe4bfff700 1 -- 172.21.15.42:0/3584337009 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efe38008da0 con 0x7efe4c08df80 2019-12-05T05:29:26.292 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:26.279+0000 7efe52f31700 1 -- 172.21.15.42:0/3584337009 shutdown_connections 2019-12-05T05:29:26.292 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:26.279+0000 7efe52f31700 1 --2- 172.21.15.42:0/3584337009 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efe4c09f830 0x7efe4c14efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:29:26.292 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:26.279+0000 7efe52f31700 1 --2- 172.21.15.42:0/3584337009 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efe4c08df80 0x7efe4c09f2e0 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:29:26.292 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:26.279+0000 7efe52f31700 1 -- 172.21.15.42:0/3584337009 shutdown_connections 2019-12-05T05:29:26.292 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:26.279+0000 7efe52f31700 1 -- 172.21.15.42:0/3584337009 wait complete. 2019-12-05T05:29:26.292 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:26.279+0000 7efe52f31700 1 -- 172.21.15.42:0/3584337009 >> 172.21.15.42:0/3584337009 conn(0x7efe4c0aabe0 msgr2=0x7efe4c0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:29:26.296 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:26.279+0000 7efe52f31700 1 Processor -- start 2019-12-05T05:29:26.296 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:26.279+0000 7efe52f31700 1 -- start start 2019-12-05T05:29:26.296 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:26.283+0000 7efe52f31700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efe4c1302c0 0x7efe4c12cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:29:26.297 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:26.283+0000 7efe52f31700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7efe4c130c20 0x7efe4c1183e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:29:26.297 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:26.283+0000 7efe52f31700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe4c118930 con 0x7efe4c07d000 2019-12-05T05:29:26.298 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:26.283+0000 7efe52f31700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe4c1306f0 con 0x7efe4c130c20 2019-12-05T05:29:26.298 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:26.283+0000 7efe52f31700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe4c130980 con 0x7efe4c1302c0 2019-12-05T05:29:26.298 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:26.283+0000 7efe50ccd700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7efe4c130c20 0x7efe4c1183e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:29:26.299 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:26.283+0000 7efe514ce700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efe4c1302c0 0x7efe4c12cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:29:26.299 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:26.283+0000 7efe514ce700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efe4c1302c0 0x7efe4c12cd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:35330/0 (socket says 172.21.15.42:35330) 2019-12-05T05:29:26.299 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:26.283+0000 7efe50ccd700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7efe4c130c20 0x7efe4c1183e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:34866/0 (socket says 172.21.15.42:34866) 2019-12-05T05:29:26.299 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:26.283+0000 7efe514ce700 1 -- 172.21.15.42:0/1017397970 learned_addr learned my addr 172.21.15.42:0/1017397970 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:29:26.299 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:26.283+0000 7efe50ccd700 1 -- 172.21.15.42:0/1017397970 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efe4c1302c0 msgr2=0x7efe4c12cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:29:26.299 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:26.283+0000 7efe50ccd700 1 --2- 172.21.15.42:0/1017397970 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efe4c1302c0 0x7efe4c12cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:29:26.300 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:26.283+0000 7efe50ccd700 1 -- 172.21.15.42:0/1017397970 >> v1:172.21.15.40:6789/0 conn(0x7efe4c07d000 legacy=0x7efe4c118f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:29:26.300 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:26.283+0000 7efe50ccd700 1 -- 172.21.15.42:0/1017397970 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe4c131050 con 0x7efe4c130c20 2019-12-05T05:29:26.300 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:26.283+0000 7efe50ccd700 1 --2- 172.21.15.42:0/1017397970 >> v2:172.21.15.42:3300/0 conn(0x7efe4c130c20 0x7efe4c1183e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7efe38008790 tx=0x7efe38008c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:26.300 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:26.283+0000 7efe50ccd700 1 --2- 172.21.15.42:0/1017397970 >> v2:172.21.15.42:3300/0 conn(0x7efe4c130c20 0x7efe4c1183e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7efe38008790 tx=0x7efe38008c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:26.300 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:26.283+0000 7efe50ccd700 1 --2- 172.21.15.42:0/1017397970 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efe4c130c20 0x7efe4c1183e0 secure :-1 s=READY pgs=970 cs=0 l=1 rx=0x7efe38008790 tx=0x7efe38008c30).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:29:26.300 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:26.283+0000 7efe50ccd700 1 --2- 172.21.15.42:0/1017397970 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efe4c130c20 0x7efe4c1183e0 secure :-1 s=THROTTLE_DONE pgs=970 cs=0 l=1 rx=0x7efe38008790 tx=0x7efe38008c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:26.301 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:26.283+0000 7efe50ccd700 1 --2- 172.21.15.42:0/1017397970 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efe4c130c20 0x7efe4c1183e0 secure :-1 s=THROTTLE_DONE pgs=970 cs=0 l=1 rx=0x7efe38008790 tx=0x7efe38008c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:26.301 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:26.283+0000 7efe49ffb700 1 -- 172.21.15.42:0/1017397970 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efe38007350 con 0x7efe4c130c20 2019-12-05T05:29:26.301 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:26.283+0000 7efe50ccd700 1 --2- 172.21.15.42:0/1017397970 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efe4c130c20 0x7efe4c1183e0 secure :-1 s=THROTTLE_DONE pgs=970 cs=0 l=1 rx=0x7efe38008790 tx=0x7efe38008c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:26.301 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:26.283+0000 7efe49ffb700 1 -- 172.21.15.42:0/1017397970 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe380089c0 con 0x7efe4c130c20 2019-12-05T05:29:26.302 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:26.283+0000 7efe52f31700 1 -- 172.21.15.42:0/1017397970 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe4c0a7000 con 0x7efe4c130c20 2019-12-05T05:29:26.302 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:26.283+0000 7efe49ffb700 1 -- 172.21.15.42:0/1017397970 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efe38011b60 con 0x7efe4c130c20 2019-12-05T05:29:26.302 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:26.283+0000 7efe52f31700 1 -- 172.21.15.42:0/1017397970 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe4c07d650 con 0x7efe4c130c20 2019-12-05T05:29:26.302 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:26.283+0000 7efe50ccd700 1 --2- 172.21.15.42:0/1017397970 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efe4c130c20 0x7efe4c1183e0 secure :-1 s=THROTTLE_DONE pgs=970 cs=0 l=1 rx=0x7efe38008790 tx=0x7efe38008c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:26.302 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:26.287+0000 7efe49ffb700 1 -- 172.21.15.42:0/1017397970 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7efe38005540 con 0x7efe4c130c20 2019-12-05T05:29:26.302 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:26.287+0000 7efe50ccd700 1 --2- 172.21.15.42:0/1017397970 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efe4c130c20 0x7efe4c1183e0 secure :-1 s=THROTTLE_DONE pgs=970 cs=0 l=1 rx=0x7efe38008790 tx=0x7efe38008c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:26.302 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:26.287+0000 7efe49ffb700 1 --2- 172.21.15.42:0/1017397970 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7efe2c03ae10 0x7efe2c03d320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:29:26.303 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:26.287+0000 7efe49ffb700 1 -- 172.21.15.42:0/1017397970 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7579+0+0 (secure 0 0 0) 0x7efe3804b460 con 0x7efe4c130c20 2019-12-05T05:29:26.303 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:26.287+0000 7efe514ce700 1 --2- 172.21.15.42:0/1017397970 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7efe2c03ae10 0x7efe2c03d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:29:26.303 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:26.287+0000 7efe514ce700 1 --2- 172.21.15.42:0/1017397970 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7efe2c03ae10 0x7efe2c03d320 crc :-1 s=READY pgs=420 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:29:26.304 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:26.291+0000 7efe52f31700 1 -- 172.21.15.42:0/1017397970 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe4c05d110 con 0x7efe4c130c20 2019-12-05T05:29:26.304 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:26.299+0000 7efe50ccd700 1 --2- 172.21.15.42:0/1017397970 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efe4c130c20 0x7efe4c1183e0 secure :-1 s=THROTTLE_DONE pgs=970 cs=0 l=1 rx=0x7efe38008790 tx=0x7efe38008c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:26.304 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:26.299+0000 7efe49ffb700 1 -- 172.21.15.42:0/1017397970 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7efe380324e0 con 0x7efe4c130c20 2019-12-05T05:29:26.482 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:26.475+0000 7efe52f31700 1 -- 172.21.15.42:0/1017397970 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efe4c05b8c0 con 0x7efe4c130c20 2019-12-05T05:29:26.483 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:26.475+0000 7efe50ccd700 1 --2- 172.21.15.42:0/1017397970 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efe4c130c20 0x7efe4c1183e0 secure :-1 s=THROTTLE_DONE pgs=970 cs=0 l=1 rx=0x7efe38008790 tx=0x7efe38008c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:26.483 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:26.475+0000 7efe49ffb700 1 -- 172.21.15.42:0/1017397970 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) v1 ==== 95+0+1775 (secure 0 0 0) 0x7efe38032740 con 0x7efe4c130c20 2019-12-05T05:29:26.483 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 122 2019-12-05T05:29:26.483 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:29:26.483 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":122,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":122,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:29:24.937492+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5426},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5426":{"gid":5426,"name":"b","rank":0,"incarnation":119,"state":"up:active","state_seq":18,"addr":"172.21.15.40:6833/3168718372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3168718372},{"type":"v1","addr":"172.21.15.40:6833","nonce":3168718372}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:29:26.486 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:26.479+0000 7efe52f31700 1 -- 172.21.15.42:0/1017397970 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7efe2c03ae10 msgr2=0x7efe2c03d320 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:29:26.486 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:26.479+0000 7efe52f31700 1 --2- 172.21.15.42:0/1017397970 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7efe2c03ae10 0x7efe2c03d320 crc :-1 s=READY pgs=420 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:29:26.487 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:26.479+0000 7efe52f31700 1 -- 172.21.15.42:0/1017397970 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efe4c130c20 msgr2=0x7efe4c1183e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:29:26.487 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:26.479+0000 7efe52f31700 1 --2- 172.21.15.42:0/1017397970 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efe4c130c20 0x7efe4c1183e0 secure :-1 s=READY pgs=970 cs=0 l=1 rx=0x7efe38008790 tx=0x7efe38008c30).stop 2019-12-05T05:29:26.487 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:26.479+0000 7efe52f31700 1 -- 172.21.15.42:0/1017397970 shutdown_connections 2019-12-05T05:29:26.487 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:26.479+0000 7efe52f31700 1 --2- 172.21.15.42:0/1017397970 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7efe2c03ae10 0x7efe2c03d320 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:29:26.487 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:26.479+0000 7efe52f31700 1 --2- 172.21.15.42:0/1017397970 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efe4c1302c0 0x7efe4c12cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:29:26.487 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:26.479+0000 7efe52f31700 1 --2- 172.21.15.42:0/1017397970 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efe4c130c20 0x7efe4c1183e0 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:29:26.487 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:26.479+0000 7efe52f31700 1 -- 172.21.15.42:0/1017397970 shutdown_connections 2019-12-05T05:29:26.488 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:26.479+0000 7efe52f31700 1 -- 172.21.15.42:0/1017397970 wait complete. 2019-12-05T05:29:26.488 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:26.479+0000 7efe52f31700 1 -- 172.21.15.42:0/1017397970 >> 172.21.15.42:0/1017397970 conn(0x7efe4c08fdc0 msgr2=0x7efe4c059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:29:26.496 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b has gained rank=0, replacing gid=6666 2019-12-05T05:29:26.496 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 7 secs before reviving mds.a 2019-12-05T05:29:33.497 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.a 2019-12-05T05:29:33.497 INFO:tasks.ceph.mds.a:Restarting daemon 2019-12-05T05:29:33.499 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2019-12-05T05:29:33.504 INFO:tasks.ceph.mds.a:Started 2019-12-05T05:29:33.505 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:29:33.574 INFO:tasks.ceph.mds.a.smithi042.stdout:starting mds.a at 2019-12-05T05:29:33.608 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:33.595+0000 7f6ed3e94700 1 Processor -- start 2019-12-05T05:29:33.608 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:33.595+0000 7f6ed3e94700 1 -- start start 2019-12-05T05:29:33.608 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:33.599+0000 7f6ed3e94700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6ecc08df80 0x7f6ecc09f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:29:33.608 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:33.599+0000 7f6ed3e94700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6ecc09f830 0x7f6ecc14efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:29:33.608 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:33.599+0000 7f6ed3e94700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ecc09fc60 con 0x7f6ecc0a7740 2019-12-05T05:29:33.608 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:33.599+0000 7f6ed3e94700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ecc0a0b00 con 0x7f6ecc09f830 2019-12-05T05:29:33.609 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:33.599+0000 7f6ed3e94700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ecc08e5b0 con 0x7f6ecc08df80 2019-12-05T05:29:33.609 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:33.599+0000 7f6ed2c32700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6ecc09f830 0x7f6ecc14efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:29:33.609 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:33.599+0000 7f6ed2c32700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6ecc09f830 0x7f6ecc14efe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:34928/0 (socket says 172.21.15.42:34928) 2019-12-05T05:29:33.609 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:33.599+0000 7f6ed2c32700 1 -- 172.21.15.42:0/1994484945 learned_addr learned my addr 172.21.15.42:0/1994484945 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:29:33.609 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:33.599+0000 7f6ed1c30700 1 --2- 172.21.15.42:0/1994484945 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6ecc08df80 0x7f6ecc09f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:29:33.610 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:33.599+0000 7f6ed2c32700 1 -- 172.21.15.42:0/1994484945 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6ecc08df80 msgr2=0x7f6ecc09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:29:33.610 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:33.599+0000 7f6ed2c32700 1 --2- 172.21.15.42:0/1994484945 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6ecc08df80 0x7f6ecc09f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:29:33.610 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:33.599+0000 7f6ed2c32700 1 -- 172.21.15.42:0/1994484945 >> v1:172.21.15.40:6789/0 conn(0x7f6ecc0a7740 legacy=0x7f6ecc07e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:29:33.610 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:33.599+0000 7f6ed2c32700 1 -- 172.21.15.42:0/1994484945 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ecc0a0710 con 0x7f6ecc09f830 2019-12-05T05:29:33.610 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:33.599+0000 7f6ed1c30700 1 --2- 172.21.15.42:0/1994484945 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6ecc08df80 0x7f6ecc09f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T05:29:33.610 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:33.599+0000 7f6ed2c32700 1 --2- 172.21.15.42:0/1994484945 >> v2:172.21.15.42:3300/0 conn(0x7f6ecc09f830 0x7f6ecc14efe0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f6ec4008b10 tx=0x7f6ec40100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:33.610 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:33.599+0000 7f6ed2c32700 1 --2- 172.21.15.42:0/1994484945 >> v2:172.21.15.42:3300/0 conn(0x7f6ecc09f830 0x7f6ecc14efe0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f6ec4008b10 tx=0x7f6ec40100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:33.611 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:33.599+0000 7f6ed2c32700 1 --2- 172.21.15.42:0/1994484945 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6ecc09f830 0x7f6ecc14efe0 secure :-1 s=READY pgs=975 cs=0 l=1 rx=0x7f6ec4008b10 tx=0x7f6ec40100b0).ready entity=mon.1 client_cookie=27439c1a726d5abc server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:29:33.611 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:33.599+0000 7f6ed2c32700 1 --2- 172.21.15.42:0/1994484945 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6ecc09f830 0x7f6ecc14efe0 secure :-1 s=THROTTLE_DONE pgs=975 cs=0 l=1 rx=0x7f6ec4008b10 tx=0x7f6ec40100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:33.611 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:33.599+0000 7f6ed2c32700 1 --2- 172.21.15.42:0/1994484945 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6ecc09f830 0x7f6ecc14efe0 secure :-1 s=THROTTLE_DONE pgs=975 cs=0 l=1 rx=0x7f6ec4008b10 tx=0x7f6ec40100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:33.611 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:33.599+0000 7f6ed142f700 1 -- 172.21.15.42:0/1994484945 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6ec4010da0 con 0x7f6ecc09f830 2019-12-05T05:29:33.611 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:33.599+0000 7f6ed2c32700 1 --2- 172.21.15.42:0/1994484945 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6ecc09f830 0x7f6ecc14efe0 secure :-1 s=THROTTLE_DONE pgs=975 cs=0 l=1 rx=0x7f6ec4008b10 tx=0x7f6ec40100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:33.611 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:33.599+0000 7f6ed142f700 1 -- 172.21.15.42:0/1994484945 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ec40109c0 con 0x7f6ecc09f830 2019-12-05T05:29:33.611 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:33.599+0000 7f6ed142f700 1 -- 172.21.15.42:0/1994484945 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6ec40114a0 con 0x7f6ecc09f830 2019-12-05T05:29:33.612 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:33.599+0000 7f6ed3e94700 1 -- 172.21.15.42:0/1994484945 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6ecc09f830 msgr2=0x7f6ecc14efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:29:33.612 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:33.599+0000 7f6ed3e94700 1 --2- 172.21.15.42:0/1994484945 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6ecc09f830 0x7f6ecc14efe0 secure :-1 s=READY pgs=975 cs=0 l=1 rx=0x7f6ec4008b10 tx=0x7f6ec40100b0).stop 2019-12-05T05:29:33.612 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:33.599+0000 7f6ed3e94700 1 -- 172.21.15.42:0/1994484945 shutdown_connections 2019-12-05T05:29:33.612 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:33.599+0000 7f6ed3e94700 1 --2- 172.21.15.42:0/1994484945 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6ecc09f830 0x7f6ecc14efe0 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:29:33.612 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:33.599+0000 7f6ed3e94700 1 --2- 172.21.15.42:0/1994484945 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6ecc08df80 0x7f6ecc09f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:29:33.612 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:33.599+0000 7f6ed3e94700 1 -- 172.21.15.42:0/1994484945 shutdown_connections 2019-12-05T05:29:33.612 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:33.599+0000 7f6ed3e94700 1 -- 172.21.15.42:0/1994484945 wait complete. 2019-12-05T05:29:33.613 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:33.599+0000 7f6ed3e94700 1 -- 172.21.15.42:0/1994484945 >> 172.21.15.42:0/1994484945 conn(0x7f6ecc0aabe0 msgr2=0x7f6ecc0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:29:33.613 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:33.603+0000 7f6ed3e94700 1 Processor -- start 2019-12-05T05:29:33.613 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:33.603+0000 7f6ed3e94700 1 -- start start 2019-12-05T05:29:33.616 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:33.603+0000 7f6ed3e94700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6ecc144810 0x7f6ecc12cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:29:33.617 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:33.603+0000 7f6ed3e94700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f6ecc12d320 0x7f6ecc12f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:29:33.617 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:33.603+0000 7f6ed3e94700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ecc0a1720 con 0x7f6ecc127990 2019-12-05T05:29:33.617 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:33.603+0000 7f6ed3e94700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ecc0a4470 con 0x7f6ecc12d320 2019-12-05T05:29:33.617 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:33.603+0000 7f6ed3e94700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ecc0a2330 con 0x7f6ecc144810 2019-12-05T05:29:33.617 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:33.603+0000 7f6ed1c30700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6ecc144810 0x7f6ecc12cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:29:33.617 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:33.603+0000 7f6ed1c30700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6ecc144810 0x7f6ecc12cdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:35400/0 (socket says 172.21.15.42:35400) 2019-12-05T05:29:33.618 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:33.603+0000 7f6ed2c32700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f6ecc12d320 0x7f6ecc12f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:29:33.618 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:33.603+0000 7f6ed1c30700 1 -- 172.21.15.42:0/3558378131 learned_addr learned my addr 172.21.15.42:0/3558378131 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:29:33.618 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:33.603+0000 7f6ed1c30700 1 -- 172.21.15.42:0/3558378131 >> v2:172.21.15.40:3300/0 conn(0x7f6ecc12d320 msgr2=0x7f6ecc12f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:29:33.618 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:33.603+0000 7f6ed1c30700 1 --2- 172.21.15.42:0/3558378131 >> v2:172.21.15.40:3300/0 conn(0x7f6ecc12d320 0x7f6ecc12f760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:29:33.618 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:33.603+0000 7f6ed1c30700 1 -- 172.21.15.42:0/3558378131 >> v1:172.21.15.40:6789/0 conn(0x7f6ecc127990 legacy=0x7f6ecc0a25d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:29:33.618 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:33.603+0000 7f6ed1c30700 1 -- 172.21.15.42:0/3558378131 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ecc0a4080 con 0x7f6ecc144810 2019-12-05T05:29:33.618 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:33.603+0000 7f6ed2c32700 1 --2- 172.21.15.42:0/3558378131 >> v2:172.21.15.40:3300/0 conn(0x7f6ecc12d320 0x7f6ecc12f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:29:33.619 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:33.603+0000 7f6ed1c30700 1 --2- 172.21.15.42:0/3558378131 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6ecc144810 0x7f6ecc12cdd0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f6ebc0077a0 tx=0x7f6ebc010b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:33.619 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:33.603+0000 7f6ed1c30700 1 --2- 172.21.15.42:0/3558378131 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6ecc144810 0x7f6ecc12cdd0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f6ebc0077a0 tx=0x7f6ebc010b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:33.619 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:33.603+0000 7f6ed1c30700 1 --2- 172.21.15.42:0/3558378131 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6ecc144810 0x7f6ecc12cdd0 secure :-1 s=READY pgs=196 cs=0 l=1 rx=0x7f6ebc0077a0 tx=0x7f6ebc010b00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:29:33.619 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:33.603+0000 7f6ed1c30700 1 --2- 172.21.15.42:0/3558378131 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6ecc144810 0x7f6ecc12cdd0 secure :-1 s=THROTTLE_DONE pgs=196 cs=0 l=1 rx=0x7f6ebc0077a0 tx=0x7f6ebc010b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:33.620 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:33.603+0000 7f6ec2ffd700 1 -- 172.21.15.42:0/3558378131 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6ebc005930 con 0x7f6ecc144810 2019-12-05T05:29:33.620 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:33.603+0000 7f6ed1c30700 1 --2- 172.21.15.42:0/3558378131 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6ecc144810 0x7f6ecc12cdd0 secure :-1 s=THROTTLE_DONE pgs=196 cs=0 l=1 rx=0x7f6ebc0077a0 tx=0x7f6ebc010b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:33.620 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:33.603+0000 7f6ec2ffd700 1 -- 172.21.15.42:0/3558378131 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ebc007160 con 0x7f6ecc144810 2019-12-05T05:29:33.620 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:33.603+0000 7f6ed3e94700 1 -- 172.21.15.42:0/3558378131 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ecc144c40 con 0x7f6ecc144810 2019-12-05T05:29:33.620 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:33.603+0000 7f6ed1c30700 1 --2- 172.21.15.42:0/3558378131 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6ecc144810 0x7f6ecc12cdd0 secure :-1 s=THROTTLE_DONE pgs=196 cs=0 l=1 rx=0x7f6ebc0077a0 tx=0x7f6ebc010b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:33.620 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:33.603+0000 7f6ec2ffd700 1 -- 172.21.15.42:0/3558378131 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6ebc012180 con 0x7f6ecc144810 2019-12-05T05:29:33.620 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:33.603+0000 7f6ed3e94700 1 -- 172.21.15.42:0/3558378131 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ecc08f630 con 0x7f6ecc144810 2019-12-05T05:29:33.621 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:33.607+0000 7f6ed1c30700 1 --2- 172.21.15.42:0/3558378131 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6ecc144810 0x7f6ecc12cdd0 secure :-1 s=THROTTLE_DONE pgs=196 cs=0 l=1 rx=0x7f6ebc0077a0 tx=0x7f6ebc010b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:33.621 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:33.607+0000 7f6ec2ffd700 1 -- 172.21.15.42:0/3558378131 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f6ebc012420 con 0x7f6ecc144810 2019-12-05T05:29:33.621 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:33.607+0000 7f6ed1c30700 1 --2- 172.21.15.42:0/3558378131 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6ecc144810 0x7f6ecc12cdd0 secure :-1 s=THROTTLE_DONE pgs=196 cs=0 l=1 rx=0x7f6ebc0077a0 tx=0x7f6ebc010b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:33.621 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:33.607+0000 7f6ec2ffd700 1 --2- 172.21.15.42:0/3558378131 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6eb403adf0 0x7f6eb403d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:29:33.621 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:33.607+0000 7f6ec2ffd700 1 -- 172.21.15.42:0/3558378131 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7579+0+0 (secure 0 0 0) 0x7f6ebc04b1d0 con 0x7f6ecc144810 2019-12-05T05:29:33.621 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:33.607+0000 7f6ed2431700 1 --2- 172.21.15.42:0/3558378131 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6eb403adf0 0x7f6eb403d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:29:33.621 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:33.607+0000 7f6ed2431700 1 --2- 172.21.15.42:0/3558378131 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6eb403adf0 0x7f6eb403d300 crc :-1 s=READY pgs=421 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:29:33.622 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:33.611+0000 7f6ed3e94700 1 -- 172.21.15.42:0/3558378131 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ecc05d110 con 0x7f6ecc144810 2019-12-05T05:29:33.629 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:33.623+0000 7f6ed1c30700 1 --2- 172.21.15.42:0/3558378131 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6ecc144810 0x7f6ecc12cdd0 secure :-1 s=THROTTLE_DONE pgs=196 cs=0 l=1 rx=0x7f6ebc0077a0 tx=0x7f6ebc010b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:33.629 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:33.623+0000 7f6ec2ffd700 1 -- 172.21.15.42:0/3558378131 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f6ebc030de0 con 0x7f6ecc144810 2019-12-05T05:29:33.802 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:33.795+0000 7f6ed3e94700 1 -- 172.21.15.42:0/3558378131 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6ecc08fa60 con 0x7f6ecc144810 2019-12-05T05:29:33.803 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:33.795+0000 7f6ed1c30700 1 --2- 172.21.15.42:0/3558378131 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6ecc144810 0x7f6ecc12cdd0 secure :-1 s=THROTTLE_DONE pgs=196 cs=0 l=1 rx=0x7f6ebc0077a0 tx=0x7f6ebc010b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:33.803 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:33.795+0000 7f6ec2ffd700 1 -- 172.21.15.42:0/3558378131 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) v1 ==== 95+0+1775 (secure 0 0 0) 0x7f6ebc032230 con 0x7f6ecc144810 2019-12-05T05:29:33.803 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 122 2019-12-05T05:29:33.804 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:29:33.805 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":122,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":122,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:29:24.937492+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5426},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5426":{"gid":5426,"name":"b","rank":0,"incarnation":119,"state":"up:active","state_seq":18,"addr":"172.21.15.40:6833/3168718372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3168718372},{"type":"v1","addr":"172.21.15.40:6833","nonce":3168718372}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:29:33.806 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:33.799+0000 7f6ed3e94700 1 -- 172.21.15.42:0/3558378131 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6eb403adf0 msgr2=0x7f6eb403d300 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:29:33.806 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:33.799+0000 7f6ed3e94700 1 --2- 172.21.15.42:0/3558378131 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6eb403adf0 0x7f6eb403d300 crc :-1 s=READY pgs=421 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:29:33.806 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:33.799+0000 7f6ed3e94700 1 -- 172.21.15.42:0/3558378131 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6ecc144810 msgr2=0x7f6ecc12cdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:29:33.806 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:33.799+0000 7f6ed3e94700 1 --2- 172.21.15.42:0/3558378131 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6ecc144810 0x7f6ecc12cdd0 secure :-1 s=READY pgs=196 cs=0 l=1 rx=0x7f6ebc0077a0 tx=0x7f6ebc010b00).stop 2019-12-05T05:29:33.806 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:33.799+0000 7f6ed3e94700 1 -- 172.21.15.42:0/3558378131 shutdown_connections 2019-12-05T05:29:33.806 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:33.799+0000 7f6ed3e94700 1 --2- 172.21.15.42:0/3558378131 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6eb403adf0 0x7f6eb403d300 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:29:33.807 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:33.799+0000 7f6ed3e94700 1 --2- 172.21.15.42:0/3558378131 >> v2:172.21.15.40:3300/0 conn(0x7f6ecc12d320 0x7f6ecc12f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:29:33.807 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:33.799+0000 7f6ed3e94700 1 --2- 172.21.15.42:0/3558378131 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6ecc144810 0x7f6ecc12cdd0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:29:33.807 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:33.799+0000 7f6ed3e94700 1 -- 172.21.15.42:0/3558378131 shutdown_connections 2019-12-05T05:29:33.807 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:33.799+0000 7f6ed3e94700 1 -- 172.21.15.42:0/3558378131 wait complete. 2019-12-05T05:29:33.807 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:33.799+0000 7f6ed3e94700 1 -- 172.21.15.42:0/3558378131 >> 172.21.15.42:0/3558378131 conn(0x7f6ecc0a7f90 msgr2=0x7f6ecc115f50 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:29:33.818 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2019-12-05T05:29:35.831 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:29:35.941 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:35.923+0000 7f1678f02700 1 Processor -- start 2019-12-05T05:29:35.941 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:35.923+0000 7f1678f02700 1 -- start start 2019-12-05T05:29:35.942 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:35.927+0000 7f1678f02700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f16740a7740 0x7f167408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:29:35.942 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:35.927+0000 7f1678f02700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16740a0710 con 0x7f167407e720 2019-12-05T05:29:35.942 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:35.927+0000 7f1678f02700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16740a9ba0 con 0x7f167409fae0 2019-12-05T05:29:35.942 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:35.927+0000 7f1678f02700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16740a0b00 con 0x7f16740a7740 2019-12-05T05:29:35.942 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:35.927+0000 7f167359e700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f167409fae0 0x7f167414efe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:34990/0 (socket says 172.21.15.42:34990) 2019-12-05T05:29:35.942 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:35.927+0000 7f167359e700 1 -- 172.21.15.42:0/1031979570 learned_addr learned my addr 172.21.15.42:0/1031979570 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:29:35.943 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:35.927+0000 7f1672d9d700 1 --2- 172.21.15.42:0/1031979570 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f16740a7740 0x7f167408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:29:35.943 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:35.927+0000 7f1671d9b700 1 -- 172.21.15.42:0/1031979570 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2029236780 0 0) 0x7f16740a9ba0 con 0x7f167409fae0 2019-12-05T05:29:35.943 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:35.927+0000 7f1671d9b700 1 -- 172.21.15.42:0/1031979570 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f165c0030b0 con 0x7f167409fae0 2019-12-05T05:29:35.943 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:35.927+0000 7f1671d9b700 1 -- 172.21.15.42:0/1031979570 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 268853288 0 0) 0x7f16740a0710 con 0x7f167407e720 2019-12-05T05:29:35.943 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:35.927+0000 7f1671d9b700 1 -- 172.21.15.42:0/1031979570 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16740a9ba0 con 0x7f167407e720 2019-12-05T05:29:35.943 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:35.927+0000 7f1672d9d700 1 -- 172.21.15.42:0/1031979570 >> v1:172.21.15.42:6789/0 conn(0x7f167409fae0 legacy=0x7f167414efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:29:35.943 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:35.927+0000 7f1672d9d700 1 -- 172.21.15.42:0/1031979570 >> v1:172.21.15.40:6789/0 conn(0x7f167407e720 legacy=0x7f167409f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:29:35.944 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:35.927+0000 7f1672d9d700 1 -- 172.21.15.42:0/1031979570 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f167407eb50 con 0x7f16740a7740 2019-12-05T05:29:35.944 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:35.927+0000 7f1672d9d700 1 --2- 172.21.15.42:0/1031979570 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f16740a7740 0x7f167408df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f16640032b0 tx=0x7f166400ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:35.944 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:35.927+0000 7f1672d9d700 1 --2- 172.21.15.42:0/1031979570 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f16740a7740 0x7f167408df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f16640032b0 tx=0x7f166400ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:35.944 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:35.927+0000 7f1672d9d700 1 --2- 172.21.15.42:0/1031979570 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f16740a7740 0x7f167408df80 secure :-1 s=READY pgs=197 cs=0 l=1 rx=0x7f16640032b0 tx=0x7f166400ff70).ready entity=mon.2 client_cookie=6497e6fb4f1ce4fb server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:29:35.944 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:35.927+0000 7f1672d9d700 1 --2- 172.21.15.42:0/1031979570 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f16740a7740 0x7f167408df80 secure :-1 s=THROTTLE_DONE pgs=197 cs=0 l=1 rx=0x7f16640032b0 tx=0x7f166400ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:35.944 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:35.927+0000 7f1671d9b700 1 -- 172.21.15.42:0/1031979570 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1664011a10 con 0x7f16740a7740 2019-12-05T05:29:35.944 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:35.927+0000 7f1672d9d700 1 --2- 172.21.15.42:0/1031979570 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f16740a7740 0x7f167408df80 secure :-1 s=THROTTLE_DONE pgs=197 cs=0 l=1 rx=0x7f16640032b0 tx=0x7f166400ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:35.945 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:35.927+0000 7f1672d9d700 1 --2- 172.21.15.42:0/1031979570 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f16740a7740 0x7f167408df80 secure :-1 s=THROTTLE_DONE pgs=197 cs=0 l=1 rx=0x7f16640032b0 tx=0x7f166400ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:35.945 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:35.927+0000 7f1671d9b700 1 -- 172.21.15.42:0/1031979570 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1664011630 con 0x7f16740a7740 2019-12-05T05:29:35.945 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:35.927+0000 7f1671d9b700 1 -- 172.21.15.42:0/1031979570 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f166400a160 con 0x7f16740a7740 2019-12-05T05:29:35.945 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:35.927+0000 7f1678f02700 1 -- 172.21.15.42:0/1031979570 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f16740a7740 msgr2=0x7f167408df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:29:35.945 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:35.927+0000 7f1678f02700 1 --2- 172.21.15.42:0/1031979570 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f16740a7740 0x7f167408df80 secure :-1 s=READY pgs=197 cs=0 l=1 rx=0x7f16640032b0 tx=0x7f166400ff70).stop 2019-12-05T05:29:35.945 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:35.931+0000 7f1678f02700 1 -- 172.21.15.42:0/1031979570 shutdown_connections 2019-12-05T05:29:35.945 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:35.931+0000 7f1678f02700 1 --2- 172.21.15.42:0/1031979570 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f16740a7740 0x7f167408df80 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:29:35.946 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:35.931+0000 7f1678f02700 1 -- 172.21.15.42:0/1031979570 shutdown_connections 2019-12-05T05:29:35.946 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:35.931+0000 7f1678f02700 1 -- 172.21.15.42:0/1031979570 wait complete. 2019-12-05T05:29:35.946 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:35.931+0000 7f1678f02700 1 -- 172.21.15.42:0/1031979570 >> 172.21.15.42:0/1031979570 conn(0x7f16740aabe0 msgr2=0x7f16740a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:29:35.946 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:35.931+0000 7f1678f02700 1 Processor -- start 2019-12-05T05:29:35.946 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:35.931+0000 7f1678f02700 1 -- start start 2019-12-05T05:29:35.946 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:35.931+0000 7f1678f02700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f167412ed30 0x7f1674131170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:29:35.946 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:35.931+0000 7f1678f02700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1674122ea0 con 0x7f1674142530 2019-12-05T05:29:35.947 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:35.931+0000 7f1678f02700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1674064d80 con 0x7f167408eeb0 2019-12-05T05:29:35.947 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:35.931+0000 7f1678f02700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1674142960 con 0x7f167412ed30 2019-12-05T05:29:35.947 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:35.931+0000 7f167359e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f167412ed30 0x7f1674131170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:29:35.947 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:35.931+0000 7f167359e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f167412ed30 0x7f1674131170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:34958/0 (socket says 172.21.15.42:34958) 2019-12-05T05:29:35.947 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:35.931+0000 7f167359e700 1 -- 172.21.15.42:0/331704648 learned_addr learned my addr 172.21.15.42:0/331704648 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:29:35.947 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:35.931+0000 7f167359e700 1 -- 172.21.15.42:0/331704648 >> v1:172.21.15.42:6789/0 conn(0x7f167408eeb0 legacy=0x7f1674063190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:29:35.947 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:35.931+0000 7f167359e700 1 -- 172.21.15.42:0/331704648 >> v1:172.21.15.40:6789/0 conn(0x7f1674142530 legacy=0x7f16741226a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:29:35.948 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:35.931+0000 7f16637fe700 1 -- 172.21.15.42:0/331704648 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2661965879 0 0) 0x7f1674064d80 con 0x7f167408eeb0 2019-12-05T05:29:35.948 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:35.935+0000 7f167359e700 1 -- 172.21.15.42:0/331704648 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16740655d0 con 0x7f167412ed30 2019-12-05T05:29:35.948 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:35.935+0000 7f167359e700 1 --2- 172.21.15.42:0/331704648 >> v2:172.21.15.42:3300/0 conn(0x7f167412ed30 0x7f1674131170 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f166c0043a0 tx=0x7f166c0107b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:35.948 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:35.935+0000 7f167359e700 1 --2- 172.21.15.42:0/331704648 >> v2:172.21.15.42:3300/0 conn(0x7f167412ed30 0x7f1674131170 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f166c0043a0 tx=0x7f166c0107b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:35.948 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:35.935+0000 7f167359e700 1 --2- 172.21.15.42:0/331704648 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f167412ed30 0x7f1674131170 secure :-1 s=READY pgs=978 cs=0 l=1 rx=0x7f166c0043a0 tx=0x7f166c0107b0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:29:35.948 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:35.935+0000 7f167359e700 1 --2- 172.21.15.42:0/331704648 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f167412ed30 0x7f1674131170 secure :-1 s=THROTTLE_DONE pgs=978 cs=0 l=1 rx=0x7f166c0043a0 tx=0x7f166c0107b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:35.949 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:35.935+0000 7f16637fe700 1 -- 172.21.15.42:0/331704648 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f166c005c30 con 0x7f167412ed30 2019-12-05T05:29:35.949 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:35.935+0000 7f167359e700 1 --2- 172.21.15.42:0/331704648 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f167412ed30 0x7f1674131170 secure :-1 s=THROTTLE_DONE pgs=978 cs=0 l=1 rx=0x7f166c0043a0 tx=0x7f166c0107b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:35.958 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:35.935+0000 7f1678f02700 1 -- 172.21.15.42:0/331704648 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16741316c0 con 0x7f167412ed30 2019-12-05T05:29:35.959 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:35.935+0000 7f16637fe700 1 -- 172.21.15.42:0/331704648 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f166c005850 con 0x7f167412ed30 2019-12-05T05:29:35.959 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:35.935+0000 7f167359e700 1 --2- 172.21.15.42:0/331704648 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f167412ed30 0x7f1674131170 secure :-1 s=THROTTLE_DONE pgs=978 cs=0 l=1 rx=0x7f166c0043a0 tx=0x7f166c0107b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:35.959 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:35.935+0000 7f16637fe700 1 -- 172.21.15.42:0/331704648 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f166c0064a0 con 0x7f167412ed30 2019-12-05T05:29:35.959 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:35.935+0000 7f1678f02700 1 -- 172.21.15.42:0/331704648 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1674131bf0 con 0x7f167412ed30 2019-12-05T05:29:35.959 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:35.935+0000 7f167359e700 1 --2- 172.21.15.42:0/331704648 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f167412ed30 0x7f1674131170 secure :-1 s=THROTTLE_DONE pgs=978 cs=0 l=1 rx=0x7f166c0043a0 tx=0x7f166c0107b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:35.961 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:35.935+0000 7f16637fe700 1 -- 172.21.15.42:0/331704648 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f166c0067c0 con 0x7f167412ed30 2019-12-05T05:29:35.961 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:35.935+0000 7f167359e700 1 --2- 172.21.15.42:0/331704648 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f167412ed30 0x7f1674131170 secure :-1 s=THROTTLE_DONE pgs=978 cs=0 l=1 rx=0x7f166c0043a0 tx=0x7f166c0107b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:35.961 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:35.935+0000 7f16637fe700 1 --2- 172.21.15.42:0/331704648 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f165803ac80 0x7f165803d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:29:35.961 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:35.935+0000 7f16637fe700 1 -- 172.21.15.42:0/331704648 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7579+0+0 (secure 0 0 0) 0x7f166c04b070 con 0x7f167412ed30 2019-12-05T05:29:35.961 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:35.935+0000 7f1672d9d700 1 --2- 172.21.15.42:0/331704648 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f165803ac80 0x7f165803d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:29:35.961 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:35.935+0000 7f1672d9d700 1 --2- 172.21.15.42:0/331704648 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f165803ac80 0x7f165803d190 crc :-1 s=READY pgs=422 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:29:35.962 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:35.939+0000 7f1678f02700 1 -- 172.21.15.42:0/331704648 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f167405d110 con 0x7f167412ed30 2019-12-05T05:29:35.962 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:35.951+0000 7f167359e700 1 --2- 172.21.15.42:0/331704648 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f167412ed30 0x7f1674131170 secure :-1 s=THROTTLE_DONE pgs=978 cs=0 l=1 rx=0x7f166c0043a0 tx=0x7f166c0107b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:35.962 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:35.951+0000 7f16637fe700 1 -- 172.21.15.42:0/331704648 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f166c0320d0 con 0x7f167412ed30 2019-12-05T05:29:36.123 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:36.115+0000 7f1678f02700 1 -- 172.21.15.42:0/331704648 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f167405b8c0 con 0x7f167412ed30 2019-12-05T05:29:36.124 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:36.119+0000 7f167359e700 1 --2- 172.21.15.42:0/331704648 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f167412ed30 0x7f1674131170 secure :-1 s=THROTTLE_DONE pgs=978 cs=0 l=1 rx=0x7f166c0043a0 tx=0x7f166c0107b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:36.124 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:36.119+0000 7f16637fe700 1 -- 172.21.15.42:0/331704648 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) v1 ==== 95+0+1775 (secure 0 0 0) 0x7f166c032330 con 0x7f167412ed30 2019-12-05T05:29:36.124 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 122 2019-12-05T05:29:36.124 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:29:36.124 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":122,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":122,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:29:24.937492+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5426},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5426":{"gid":5426,"name":"b","rank":0,"incarnation":119,"state":"up:active","state_seq":18,"addr":"172.21.15.40:6833/3168718372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3168718372},{"type":"v1","addr":"172.21.15.40:6833","nonce":3168718372}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:29:36.126 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:36.119+0000 7f1678f02700 1 -- 172.21.15.42:0/331704648 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f165803ac80 msgr2=0x7f165803d190 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:29:36.126 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:36.119+0000 7f1678f02700 1 --2- 172.21.15.42:0/331704648 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f165803ac80 0x7f165803d190 crc :-1 s=READY pgs=422 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:29:36.126 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:36.119+0000 7f1678f02700 1 -- 172.21.15.42:0/331704648 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f167412ed30 msgr2=0x7f1674131170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:29:36.126 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:36.119+0000 7f1678f02700 1 --2- 172.21.15.42:0/331704648 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f167412ed30 0x7f1674131170 secure :-1 s=READY pgs=978 cs=0 l=1 rx=0x7f166c0043a0 tx=0x7f166c0107b0).stop 2019-12-05T05:29:36.127 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:36.119+0000 7f1678f02700 1 -- 172.21.15.42:0/331704648 shutdown_connections 2019-12-05T05:29:36.127 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:36.119+0000 7f1678f02700 1 --2- 172.21.15.42:0/331704648 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f165803ac80 0x7f165803d190 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:29:36.127 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:36.119+0000 7f1678f02700 1 --2- 172.21.15.42:0/331704648 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f167412ed30 0x7f1674131170 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:29:36.127 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:36.119+0000 7f1678f02700 1 -- 172.21.15.42:0/331704648 shutdown_connections 2019-12-05T05:29:36.128 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:36.119+0000 7f1678f02700 1 -- 172.21.15.42:0/331704648 wait complete. 2019-12-05T05:29:36.128 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:36.119+0000 7f1678f02700 1 -- 172.21.15.42:0/331704648 >> 172.21.15.42:0/331704648 conn(0x7f16740a82a0 msgr2=0x7f167409f580 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:29:36.136 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2019-12-05T05:29:38.137 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:29:38.248 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:38.235+0000 7fab54d86700 1 Processor -- start 2019-12-05T05:29:38.248 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:38.235+0000 7fab54d86700 1 -- start start 2019-12-05T05:29:38.248 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:38.239+0000 7fab54d86700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fab5009fae0 0x7fab5014fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:29:38.248 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:38.239+0000 7fab54d86700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab500a0710 con 0x7fab5008df80 2019-12-05T05:29:38.249 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:38.239+0000 7fab54d86700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab5007ef20 con 0x7fab500a7740 2019-12-05T05:29:38.249 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:38.239+0000 7fab54d86700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab500a0b00 con 0x7fab5009fae0 2019-12-05T05:29:38.249 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:38.239+0000 7fab4ed9d700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fab500a7740 0x7fab5007e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:35012/0 (socket says 172.21.15.42:35012) 2019-12-05T05:29:38.249 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:38.239+0000 7fab4ed9d700 1 -- 172.21.15.42:0/2751462278 learned_addr learned my addr 172.21.15.42:0/2751462278 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:29:38.249 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:38.239+0000 7fab4f59e700 1 --2- 172.21.15.42:0/2751462278 >> v2:172.21.15.40:3300/0 conn(0x7fab5009fae0 0x7fab5014fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:29:38.249 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:38.239+0000 7fab4e59c700 1 -- 172.21.15.42:0/2751462278 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1145749721 0 0) 0x7fab5007ef20 con 0x7fab500a7740 2019-12-05T05:29:38.250 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:38.239+0000 7fab4e59c700 1 -- 172.21.15.42:0/2751462278 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab3c0030b0 con 0x7fab500a7740 2019-12-05T05:29:38.250 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:38.239+0000 7fab4f59e700 1 -- 172.21.15.42:0/2751462278 >> v1:172.21.15.42:6789/0 conn(0x7fab500a7740 legacy=0x7fab5007e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:29:38.250 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:38.239+0000 7fab4f59e700 1 -- 172.21.15.42:0/2751462278 >> v1:172.21.15.40:6789/0 conn(0x7fab5008df80 legacy=0x7fab5009f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:29:38.250 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:38.239+0000 7fab4e59c700 1 -- 172.21.15.42:0/2751462278 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 802910551 0 0) 0x7fab500a0710 con 0x7fab5008df80 2019-12-05T05:29:38.250 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:38.239+0000 7fab4f59e700 1 -- 172.21.15.42:0/2751462278 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab500a9ba0 con 0x7fab5009fae0 2019-12-05T05:29:38.250 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:38.239+0000 7fab4f59e700 1 --2- 172.21.15.42:0/2751462278 >> v2:172.21.15.40:3300/0 conn(0x7fab5009fae0 0x7fab5014fff0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fab48008b10 tx=0x7fab4800ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:38.250 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:38.239+0000 7fab4f59e700 1 --2- 172.21.15.42:0/2751462278 >> v2:172.21.15.40:3300/0 conn(0x7fab5009fae0 0x7fab5014fff0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fab48008b10 tx=0x7fab4800ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:38.251 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:38.239+0000 7fab4f59e700 1 --2- 172.21.15.42:0/2751462278 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab5009fae0 0x7fab5014fff0 secure :-1 s=READY pgs=761 cs=0 l=1 rx=0x7fab48008b10 tx=0x7fab4800ff70).ready entity=mon.0 client_cookie=607e939daa21ef40 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:29:38.252 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:38.239+0000 7fab4f59e700 1 --2- 172.21.15.42:0/2751462278 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab5009fae0 0x7fab5014fff0 secure :-1 s=THROTTLE_DONE pgs=761 cs=0 l=1 rx=0x7fab48008b10 tx=0x7fab4800ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:38.252 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:38.239+0000 7fab4f59e700 1 --2- 172.21.15.42:0/2751462278 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab5009fae0 0x7fab5014fff0 secure :-1 s=THROTTLE_DONE pgs=761 cs=0 l=1 rx=0x7fab48008b10 tx=0x7fab4800ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:38.252 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:38.239+0000 7fab4e59c700 1 -- 172.21.15.42:0/2751462278 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fab48010970 con 0x7fab5009fae0 2019-12-05T05:29:38.252 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:38.239+0000 7fab4f59e700 1 --2- 172.21.15.42:0/2751462278 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab5009fae0 0x7fab5014fff0 secure :-1 s=THROTTLE_DONE pgs=761 cs=0 l=1 rx=0x7fab48008b10 tx=0x7fab4800ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:38.253 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:38.239+0000 7fab4e59c700 1 -- 172.21.15.42:0/2751462278 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab48008b50 con 0x7fab5009fae0 2019-12-05T05:29:38.253 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:38.239+0000 7fab4e59c700 1 -- 172.21.15.42:0/2751462278 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fab480110c0 con 0x7fab5009fae0 2019-12-05T05:29:38.253 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:38.239+0000 7fab54d86700 1 -- 172.21.15.42:0/2751462278 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab5009fae0 msgr2=0x7fab5014fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:29:38.253 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:38.239+0000 7fab54d86700 1 --2- 172.21.15.42:0/2751462278 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab5009fae0 0x7fab5014fff0 secure :-1 s=READY pgs=761 cs=0 l=1 rx=0x7fab48008b10 tx=0x7fab4800ff70).stop 2019-12-05T05:29:38.253 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:38.239+0000 7fab54d86700 1 -- 172.21.15.42:0/2751462278 shutdown_connections 2019-12-05T05:29:38.253 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:38.239+0000 7fab54d86700 1 --2- 172.21.15.42:0/2751462278 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab5009fae0 0x7fab5014fff0 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:29:38.254 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:38.239+0000 7fab54d86700 1 -- 172.21.15.42:0/2751462278 shutdown_connections 2019-12-05T05:29:38.254 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:38.239+0000 7fab54d86700 1 -- 172.21.15.42:0/2751462278 wait complete. 2019-12-05T05:29:38.254 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:38.243+0000 7fab54d86700 1 -- 172.21.15.42:0/2751462278 >> 172.21.15.42:0/2751462278 conn(0x7fab500aabe0 msgr2=0x7fab500a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:29:38.258 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:38.243+0000 7fab54d86700 1 Processor -- start 2019-12-05T05:29:38.258 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:38.243+0000 7fab54d86700 1 -- start start 2019-12-05T05:29:38.258 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:38.243+0000 7fab54d86700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fab50118f00 0x7fab501258b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:29:38.258 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:38.243+0000 7fab54d86700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fab50116950 0x7fab5007a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:29:38.258 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:38.243+0000 7fab54d86700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab500a5f00 con 0x7fab5007af10 2019-12-05T05:29:38.258 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:38.243+0000 7fab54d86700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab50118720 con 0x7fab50118f00 2019-12-05T05:29:38.259 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:38.243+0000 7fab54d86700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab501189c0 con 0x7fab50116950 2019-12-05T05:29:38.259 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:38.243+0000 7fab4ed9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fab50118f00 0x7fab501258b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:29:38.259 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:38.243+0000 7fab4f59e700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fab5007af10 0x7fab500a5700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:35020/0 (socket says 172.21.15.42:35020) 2019-12-05T05:29:38.259 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:38.243+0000 7fab4f59e700 1 -- 172.21.15.42:0/3740036586 learned_addr learned my addr 172.21.15.42:0/3740036586 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:29:38.259 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:38.243+0000 7fab4659c700 1 --2- 172.21.15.42:0/3740036586 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fab50116950 0x7fab5007a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:29:38.259 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:38.247+0000 7fab47fff700 1 -- 172.21.15.42:0/3740036586 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3324301160 0 0) 0x7fab500a5f00 con 0x7fab5007af10 2019-12-05T05:29:38.259 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:38.247+0000 7fab47fff700 1 -- 172.21.15.42:0/3740036586 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab340030b0 con 0x7fab5007af10 2019-12-05T05:29:38.260 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:38.247+0000 7fab4ed9d700 1 -- 172.21.15.42:0/3740036586 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fab50116950 msgr2=0x7fab5007a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:29:38.260 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:38.247+0000 7fab4ed9d700 1 --2- 172.21.15.42:0/3740036586 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fab50116950 0x7fab5007a9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:29:38.260 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:38.247+0000 7fab4ed9d700 1 -- 172.21.15.42:0/3740036586 >> v1:172.21.15.42:6789/0 conn(0x7fab5007af10 legacy=0x7fab500a5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:29:38.260 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:38.247+0000 7fab4ed9d700 1 -- 172.21.15.42:0/3740036586 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab50125e00 con 0x7fab50118f00 2019-12-05T05:29:38.260 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:38.247+0000 7fab4ed9d700 1 --2- 172.21.15.42:0/3740036586 >> v2:172.21.15.42:3300/0 conn(0x7fab50118f00 0x7fab501258b0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fab501188e0 tx=0x7fab40012aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:38.260 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:38.247+0000 7fab4ed9d700 1 --2- 172.21.15.42:0/3740036586 >> v2:172.21.15.42:3300/0 conn(0x7fab50118f00 0x7fab501258b0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fab501188e0 tx=0x7fab40012aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:38.261 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:38.247+0000 7fab4ed9d700 1 --2- 172.21.15.42:0/3740036586 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fab50118f00 0x7fab501258b0 secure :-1 s=READY pgs=981 cs=0 l=1 rx=0x7fab501188e0 tx=0x7fab40012aa0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:29:38.261 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:38.247+0000 7fab4ed9d700 1 --2- 172.21.15.42:0/3740036586 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fab50118f00 0x7fab501258b0 secure :-1 s=THROTTLE_DONE pgs=981 cs=0 l=1 rx=0x7fab501188e0 tx=0x7fab40012aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:38.261 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:38.247+0000 7fab47fff700 1 -- 172.21.15.42:0/3740036586 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fab4000aa80 con 0x7fab50118f00 2019-12-05T05:29:38.261 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:38.247+0000 7fab4ed9d700 1 --2- 172.21.15.42:0/3740036586 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fab50118f00 0x7fab501258b0 secure :-1 s=THROTTLE_DONE pgs=981 cs=0 l=1 rx=0x7fab501188e0 tx=0x7fab40012aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:38.261 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:38.247+0000 7fab54d86700 1 -- 172.21.15.42:0/3740036586 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab5008eb70 con 0x7fab50118f00 2019-12-05T05:29:38.261 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:38.247+0000 7fab47fff700 1 -- 172.21.15.42:0/3740036586 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab4000a6a0 con 0x7fab50118f00 2019-12-05T05:29:38.261 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:38.247+0000 7fab4ed9d700 1 --2- 172.21.15.42:0/3740036586 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fab50118f00 0x7fab501258b0 secure :-1 s=THROTTLE_DONE pgs=981 cs=0 l=1 rx=0x7fab501188e0 tx=0x7fab40012aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:38.261 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:38.247+0000 7fab47fff700 1 -- 172.21.15.42:0/3740036586 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fab40005650 con 0x7fab50118f00 2019-12-05T05:29:38.262 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:38.247+0000 7fab54d86700 1 -- 172.21.15.42:0/3740036586 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab5008f0a0 con 0x7fab50118f00 2019-12-05T05:29:38.263 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:38.247+0000 7fab4ed9d700 1 --2- 172.21.15.42:0/3740036586 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fab50118f00 0x7fab501258b0 secure :-1 s=THROTTLE_DONE pgs=981 cs=0 l=1 rx=0x7fab501188e0 tx=0x7fab40012aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:38.263 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:38.247+0000 7fab47fff700 1 -- 172.21.15.42:0/3740036586 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fab400058f0 con 0x7fab50118f00 2019-12-05T05:29:38.263 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:38.247+0000 7fab4ed9d700 1 --2- 172.21.15.42:0/3740036586 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fab50118f00 0x7fab501258b0 secure :-1 s=THROTTLE_DONE pgs=981 cs=0 l=1 rx=0x7fab501188e0 tx=0x7fab40012aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:38.263 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:38.247+0000 7fab47fff700 1 --2- 172.21.15.42:0/3740036586 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fab3403b280 0x7fab3403d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:29:38.264 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:38.251+0000 7fab47fff700 1 -- 172.21.15.42:0/3740036586 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7579+0+0 (secure 0 0 0) 0x7fab4004cd60 con 0x7fab50118f00 2019-12-05T05:29:38.264 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:38.251+0000 7fab4659c700 1 --2- 172.21.15.42:0/3740036586 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fab3403b280 0x7fab3403d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:29:38.264 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:38.251+0000 7fab4659c700 1 --2- 172.21.15.42:0/3740036586 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fab3403b280 0x7fab3403d790 crc :-1 s=READY pgs=424 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:29:38.264 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:38.251+0000 7fab54d86700 1 -- 172.21.15.42:0/3740036586 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab50115340 con 0x7fab50118f00 2019-12-05T05:29:38.265 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:38.259+0000 7fab4ed9d700 1 --2- 172.21.15.42:0/3740036586 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fab50118f00 0x7fab501258b0 secure :-1 s=THROTTLE_DONE pgs=981 cs=0 l=1 rx=0x7fab501188e0 tx=0x7fab40012aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:38.265 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:38.259+0000 7fab47fff700 1 -- 172.21.15.42:0/3740036586 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fab40033dc0 con 0x7fab50118f00 2019-12-05T05:29:38.433 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:38.427+0000 7fab54d86700 1 -- 172.21.15.42:0/3740036586 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fab5005d110 con 0x7fab50118f00 2019-12-05T05:29:38.433 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:38.427+0000 7fab4ed9d700 1 --2- 172.21.15.42:0/3740036586 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fab50118f00 0x7fab501258b0 secure :-1 s=THROTTLE_DONE pgs=981 cs=0 l=1 rx=0x7fab501188e0 tx=0x7fab40012aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:29:38.434 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:38.427+0000 7fab47fff700 1 -- 172.21.15.42:0/3740036586 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 123 v123) v1 ==== 95+0+2112 (secure 0 0 0) 0x7fab40034020 con 0x7fab50118f00 2019-12-05T05:29:38.435 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:29:38.435 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":123,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6778,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/3028851337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3028851337},{"type":"v1","addr":"172.21.15.42:6835","nonce":3028851337}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":123}],"filesystems":[{"mdsmap":{"epoch":122,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:29:24.937492+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5426},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5426":{"gid":5426,"name":"b","rank":0,"incarnation":119,"state":"up:active","state_seq":18,"addr":"172.21.15.40:6833/3168718372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3168718372},{"type":"v1","addr":"172.21.15.40:6833","nonce":3168718372}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:29:38.435 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 123 2019-12-05T05:29:38.437 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:38.431+0000 7fab54d86700 1 -- 172.21.15.42:0/3740036586 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fab3403b280 msgr2=0x7fab3403d790 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:29:38.437 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:38.431+0000 7fab54d86700 1 --2- 172.21.15.42:0/3740036586 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fab3403b280 0x7fab3403d790 crc :-1 s=READY pgs=424 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:29:38.437 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:38.431+0000 7fab54d86700 1 -- 172.21.15.42:0/3740036586 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fab50118f00 msgr2=0x7fab501258b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:29:38.437 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:38.431+0000 7fab54d86700 1 --2- 172.21.15.42:0/3740036586 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fab50118f00 0x7fab501258b0 secure :-1 s=READY pgs=981 cs=0 l=1 rx=0x7fab501188e0 tx=0x7fab40012aa0).stop 2019-12-05T05:29:38.438 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:38.431+0000 7fab54d86700 1 -- 172.21.15.42:0/3740036586 shutdown_connections 2019-12-05T05:29:38.438 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:38.431+0000 7fab54d86700 1 --2- 172.21.15.42:0/3740036586 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fab3403b280 0x7fab3403d790 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:29:38.438 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:38.431+0000 7fab54d86700 1 --2- 172.21.15.42:0/3740036586 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fab50118f00 0x7fab501258b0 unknown :-1 s=CLOSED pgs=981 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:29:38.438 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:38.431+0000 7fab54d86700 1 --2- 172.21.15.42:0/3740036586 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fab50116950 0x7fab5007a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:29:38.439 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:38.431+0000 7fab54d86700 1 -- 172.21.15.42:0/3740036586 shutdown_connections 2019-12-05T05:29:38.439 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:38.431+0000 7fab54d86700 1 -- 172.21.15.42:0/3740036586 wait complete. 2019-12-05T05:29:38.439 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:29:38.431+0000 7fab54d86700 1 -- 172.21.15.42:0/3740036586 >> 172.21.15.42:0/3740036586 conn(0x7fab5007dee0 msgr2=0x7fab5009f240 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:29:38.447 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a reported in up:standby state 2019-12-05T05:29:38.447 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 57 secs before thrashing 2019-12-05T05:29:38.832 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:29:38.844 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:30:08.884 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:30:08.888 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:30:35.449 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:30:35.555 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.543+0000 7fb8ec785700 1 Processor -- start 2019-12-05T05:30:35.556 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.543+0000 7fb8ec785700 1 -- start start 2019-12-05T05:30:35.556 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.543+0000 7fb8ec785700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb8e40a7740 0x7fb8e408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:30:35.556 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.543+0000 7fb8ec785700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8e40a0710 con 0x7fb8e409fae0 2019-12-05T05:30:35.556 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.543+0000 7fb8ec785700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8e40a9ba0 con 0x7fb8e407e720 2019-12-05T05:30:35.556 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.543+0000 7fb8ec785700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8e40a0b00 con 0x7fb8e40a7740 2019-12-05T05:30:35.556 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.543+0000 7fb8ea521700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fb8e407e720 0x7fb8e409f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:35228/0 (socket says 172.21.15.42:35228) 2019-12-05T05:30:35.556 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.543+0000 7fb8ea521700 1 -- 172.21.15.42:0/2951138880 learned_addr learned my addr 172.21.15.42:0/2951138880 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:30:35.557 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.543+0000 7fb8ead22700 1 --2- 172.21.15.42:0/2951138880 >> v2:172.21.15.40:3300/0 conn(0x7fb8e40a7740 0x7fb8e408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:30:35.557 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.543+0000 7fb8e9d20700 1 -- 172.21.15.42:0/2951138880 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 809299147 0 0) 0x7fb8e40a9ba0 con 0x7fb8e407e720 2019-12-05T05:30:35.557 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.543+0000 7fb8e9d20700 1 -- 172.21.15.42:0/2951138880 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8cc0030b0 con 0x7fb8e407e720 2019-12-05T05:30:35.557 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.543+0000 7fb8e9d20700 1 -- 172.21.15.42:0/2951138880 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2713749470 0 0) 0x7fb8e40a0710 con 0x7fb8e409fae0 2019-12-05T05:30:35.557 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.543+0000 7fb8e9d20700 1 -- 172.21.15.42:0/2951138880 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8e40a9ba0 con 0x7fb8e409fae0 2019-12-05T05:30:35.557 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.543+0000 7fb8e9d20700 1 -- 172.21.15.42:0/2951138880 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4171519946 0 0) 0x7fb8cc0030b0 con 0x7fb8e407e720 2019-12-05T05:30:35.557 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.543+0000 7fb8e9d20700 1 -- 172.21.15.42:0/2951138880 >> v2:172.21.15.40:3300/0 conn(0x7fb8e40a7740 msgr2=0x7fb8e408df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:30:35.558 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.543+0000 7fb8e9d20700 1 --2- 172.21.15.42:0/2951138880 >> v2:172.21.15.40:3300/0 conn(0x7fb8e40a7740 0x7fb8e408df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:30:35.559 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.543+0000 7fb8e9d20700 1 -- 172.21.15.42:0/2951138880 >> v1:172.21.15.40:6789/0 conn(0x7fb8e409fae0 legacy=0x7fb8e414efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:30:35.560 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.543+0000 7fb8e9d20700 1 -- 172.21.15.42:0/2951138880 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8e407eb50 con 0x7fb8e407e720 2019-12-05T05:30:35.560 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.543+0000 7fb8e9d20700 1 -- 172.21.15.42:0/2951138880 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7fb8d8001750 con 0x7fb8e407e720 2019-12-05T05:30:35.560 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.543+0000 7fb8e9d20700 1 -- 172.21.15.42:0/2951138880 >> v1:172.21.15.42:6789/0 conn(0x7fb8e407e720 legacy=0x7fb8e409f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:30:35.560 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.543+0000 7fb8e9d20700 1 --2- 172.21.15.42:0/2951138880 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb8cc003960 0x7fb8cc003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:30:35.560 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.543+0000 7fb8e9d20700 1 --2- 172.21.15.42:0/2951138880 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb8cc004470 0x7fb8cc004940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:30:35.560 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.543+0000 7fb8e9d20700 1 --2- 172.21.15.42:0/2951138880 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb8cc004fa0 0x7fb8cc04f700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:30:35.561 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.543+0000 7fb8e9d20700 1 -- 172.21.15.42:0/2951138880 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8e40a0b00 con 0x7fb8cc004470 2019-12-05T05:30:35.561 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.543+0000 7fb8e9d20700 1 -- 172.21.15.42:0/2951138880 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8cc04fec0 con 0x7fb8cc003960 2019-12-05T05:30:35.561 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.543+0000 7fb8e9d20700 1 -- 172.21.15.42:0/2951138880 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8cc050100 con 0x7fb8cc004fa0 2019-12-05T05:30:35.561 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.543+0000 7fb8ead22700 1 --2- 172.21.15.42:0/2951138880 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb8cc003960 0x7fb8cc003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:30:35.561 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.543+0000 7fb8ea521700 1 --2- 172.21.15.42:0/2951138880 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb8cc004470 0x7fb8cc004940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:30:35.562 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.543+0000 7fb8eb523700 1 --2- 172.21.15.42:0/2951138880 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb8cc004fa0 0x7fb8cc04f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:30:35.562 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.543+0000 7fb8ead22700 1 -- 172.21.15.42:0/2951138880 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb8cc004fa0 msgr2=0x7fb8cc04f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:30:35.562 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.543+0000 7fb8ead22700 1 --2- 172.21.15.42:0/2951138880 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb8cc004fa0 0x7fb8cc04f700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:30:35.562 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.543+0000 7fb8ead22700 1 -- 172.21.15.42:0/2951138880 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb8cc004470 msgr2=0x7fb8cc004940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:30:35.562 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.543+0000 7fb8ead22700 1 --2- 172.21.15.42:0/2951138880 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb8cc004470 0x7fb8cc004940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:30:35.562 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.543+0000 7fb8eb523700 1 -- 172.21.15.42:0/2951138880 reap_dead start 2019-12-05T05:30:35.562 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.543+0000 7fb8ead22700 1 -- 172.21.15.42:0/2951138880 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8cc050400 con 0x7fb8cc003960 2019-12-05T05:30:35.563 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.543+0000 7fb8ead22700 1 --2- 172.21.15.42:0/2951138880 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb8cc003960 0x7fb8cc003db0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fb8dc0032b0 tx=0x7fb8dc010360).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:35.563 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.543+0000 7fb8ead22700 1 --2- 172.21.15.42:0/2951138880 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb8cc003960 0x7fb8cc003db0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fb8dc0032b0 tx=0x7fb8dc010360).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:35.563 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.543+0000 7fb8ead22700 1 --2- 172.21.15.42:0/2951138880 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb8cc003960 0x7fb8cc003db0 secure :-1 s=READY pgs=990 cs=0 l=1 rx=0x7fb8dc0032b0 tx=0x7fb8dc010360).ready entity=mon.1 client_cookie=a261a56f67e452d6 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:30:35.563 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.547+0000 7fb8ead22700 1 --2- 172.21.15.42:0/2951138880 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb8cc003960 0x7fb8cc003db0 secure :-1 s=THROTTLE_DONE pgs=990 cs=0 l=1 rx=0x7fb8dc0032b0 tx=0x7fb8dc010360).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:35.563 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.547+0000 7fb8ead22700 1 --2- 172.21.15.42:0/2951138880 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb8cc003960 0x7fb8cc003db0 secure :-1 s=THROTTLE_DONE pgs=990 cs=0 l=1 rx=0x7fb8dc0032b0 tx=0x7fb8dc010360).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:35.563 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.547+0000 7fb8e9d20700 1 -- 172.21.15.42:0/2951138880 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb8dc008d60 con 0x7fb8cc003960 2019-12-05T05:30:35.563 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.547+0000 7fb8ead22700 1 --2- 172.21.15.42:0/2951138880 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb8cc003960 0x7fb8cc003db0 secure :-1 s=THROTTLE_DONE pgs=990 cs=0 l=1 rx=0x7fb8dc0032b0 tx=0x7fb8dc010360).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:35.564 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.547+0000 7fb8e9d20700 1 -- 172.21.15.42:0/2951138880 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8dc010cb0 con 0x7fb8cc003960 2019-12-05T05:30:35.564 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.547+0000 7fb8e9d20700 1 -- 172.21.15.42:0/2951138880 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb8dc005ca0 con 0x7fb8cc003960 2019-12-05T05:30:35.565 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.547+0000 7fb8ec785700 1 -- 172.21.15.42:0/2951138880 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb8cc003960 msgr2=0x7fb8cc003db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:30:35.565 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.547+0000 7fb8ec785700 1 --2- 172.21.15.42:0/2951138880 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb8cc003960 0x7fb8cc003db0 secure :-1 s=READY pgs=990 cs=0 l=1 rx=0x7fb8dc0032b0 tx=0x7fb8dc010360).stop 2019-12-05T05:30:35.565 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.547+0000 7fb8ec785700 1 -- 172.21.15.42:0/2951138880 shutdown_connections 2019-12-05T05:30:35.566 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.547+0000 7fb8ec785700 1 --2- 172.21.15.42:0/2951138880 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb8cc003960 0x7fb8cc003db0 unknown :-1 s=CLOSED pgs=990 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:30:35.566 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.547+0000 7fb8ec785700 1 -- 172.21.15.42:0/2951138880 shutdown_connections 2019-12-05T05:30:35.566 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.547+0000 7fb8ec785700 1 -- 172.21.15.42:0/2951138880 wait complete. 2019-12-05T05:30:35.567 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.547+0000 7fb8ec785700 1 -- 172.21.15.42:0/2951138880 >> 172.21.15.42:0/2951138880 conn(0x7fb8e40aabe0 msgr2=0x7fb8e40a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:30:35.567 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.547+0000 7fb8ec785700 1 Processor -- start 2019-12-05T05:30:35.567 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.551+0000 7fb8ec785700 1 -- start start 2019-12-05T05:30:35.571 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.551+0000 7fb8ec785700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb8e40a6420 0x7fb8e405f290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:30:35.571 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.551+0000 7fb8ec785700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb8e40a3160 0x7fb8e40a3590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:30:35.571 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.551+0000 7fb8ec785700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8e40a3ae0 con 0x7fb8e406f2b0 2019-12-05T05:30:35.571 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.551+0000 7fb8ec785700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8e4060c90 con 0x7fb8e40a3160 2019-12-05T05:30:35.571 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.551+0000 7fb8ec785700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8e405fdf0 con 0x7fb8e40a6420 2019-12-05T05:30:35.571 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.551+0000 7fb8ead22700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb8e40a6420 0x7fb8e405f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:30:35.572 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.551+0000 7fb8ead22700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb8e40a6420 0x7fb8e405f290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:35202/0 (socket says 172.21.15.42:35202) 2019-12-05T05:30:35.572 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.551+0000 7fb8ead22700 1 -- 172.21.15.42:0/1618083535 learned_addr learned my addr 172.21.15.42:0/1618083535 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:30:35.572 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.551+0000 7fb8ea521700 1 --2- 172.21.15.42:0/1618083535 >> v2:172.21.15.40:3300/0 conn(0x7fb8e40a3160 0x7fb8e40a3590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:30:35.572 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.551+0000 7fb8ead22700 1 -- 172.21.15.42:0/1618083535 >> v2:172.21.15.40:3300/0 conn(0x7fb8e40a3160 msgr2=0x7fb8e40a3590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:30:35.572 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.551+0000 7fb8ead22700 1 --2- 172.21.15.42:0/1618083535 >> v2:172.21.15.40:3300/0 conn(0x7fb8e40a3160 0x7fb8e40a3590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:30:35.572 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.551+0000 7fb8ead22700 1 -- 172.21.15.42:0/1618083535 >> v1:172.21.15.42:6789/0 conn(0x7fb8e406f2b0 legacy=0x7fb8e406f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:30:35.572 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.551+0000 7fb8e37fe700 1 -- 172.21.15.42:0/1618083535 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1169915619 0 0) 0x7fb8e40a3ae0 con 0x7fb8e406f2b0 2019-12-05T05:30:35.573 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.551+0000 7fb8ead22700 1 -- 172.21.15.42:0/1618083535 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8e4060180 con 0x7fb8e40a6420 2019-12-05T05:30:35.573 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.551+0000 7fb8ead22700 1 --2- 172.21.15.42:0/1618083535 >> v2:172.21.15.42:3300/0 conn(0x7fb8e40a6420 0x7fb8e405f290 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fb8e40a2550 tx=0x7fb8dc010eb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:35.573 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.551+0000 7fb8ead22700 1 --2- 172.21.15.42:0/1618083535 >> v2:172.21.15.42:3300/0 conn(0x7fb8e40a6420 0x7fb8e405f290 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fb8e40a2550 tx=0x7fb8dc010eb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:35.573 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.551+0000 7fb8ead22700 1 --2- 172.21.15.42:0/1618083535 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb8e40a6420 0x7fb8e405f290 secure :-1 s=READY pgs=992 cs=0 l=1 rx=0x7fb8e40a2550 tx=0x7fb8dc010eb0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:30:35.573 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.551+0000 7fb8ead22700 1 --2- 172.21.15.42:0/1618083535 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb8e40a6420 0x7fb8e405f290 secure :-1 s=THROTTLE_DONE pgs=992 cs=0 l=1 rx=0x7fb8e40a2550 tx=0x7fb8dc010eb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:35.573 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.551+0000 7fb8e37fe700 1 -- 172.21.15.42:0/1618083535 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb8dc007580 con 0x7fb8e40a6420 2019-12-05T05:30:35.573 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.551+0000 7fb8ead22700 1 --2- 172.21.15.42:0/1618083535 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb8e40a6420 0x7fb8e405f290 secure :-1 s=THROTTLE_DONE pgs=992 cs=0 l=1 rx=0x7fb8e40a2550 tx=0x7fb8dc010eb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:35.574 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.551+0000 7fb8ead22700 1 --2- 172.21.15.42:0/1618083535 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb8e40a6420 0x7fb8e405f290 secure :-1 s=THROTTLE_DONE pgs=992 cs=0 l=1 rx=0x7fb8e40a2550 tx=0x7fb8dc010eb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:35.574 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.551+0000 7fb8e37fe700 1 -- 172.21.15.42:0/1618083535 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8dc0058c0 con 0x7fb8e40a6420 2019-12-05T05:30:35.576 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.551+0000 7fb8ec785700 1 -- 172.21.15.42:0/1618083535 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8e40a3e40 con 0x7fb8e40a6420 2019-12-05T05:30:35.576 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.551+0000 7fb8e37fe700 1 -- 172.21.15.42:0/1618083535 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb8dc00f5d0 con 0x7fb8e40a6420 2019-12-05T05:30:35.576 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.551+0000 7fb8ec785700 1 -- 172.21.15.42:0/1618083535 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8e406fee0 con 0x7fb8e40a6420 2019-12-05T05:30:35.576 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.551+0000 7fb8ead22700 1 --2- 172.21.15.42:0/1618083535 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb8e40a6420 0x7fb8e405f290 secure :-1 s=THROTTLE_DONE pgs=992 cs=0 l=1 rx=0x7fb8e40a2550 tx=0x7fb8dc010eb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:35.576 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.555+0000 7fb8e37fe700 1 -- 172.21.15.42:0/1618083535 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fb8dc00f7d0 con 0x7fb8e40a6420 2019-12-05T05:30:35.576 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.555+0000 7fb8ead22700 1 --2- 172.21.15.42:0/1618083535 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb8e40a6420 0x7fb8e405f290 secure :-1 s=THROTTLE_DONE pgs=992 cs=0 l=1 rx=0x7fb8e40a2550 tx=0x7fb8dc010eb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:35.576 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.555+0000 7fb8e37fe700 1 --2- 172.21.15.42:0/1618083535 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb8c403ac60 0x7fb8c403d170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:30:35.577 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.555+0000 7fb8e37fe700 1 -- 172.21.15.42:0/1618083535 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7579+0+0 (secure 0 0 0) 0x7fb8dc04b3e0 con 0x7fb8e40a6420 2019-12-05T05:30:35.577 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.555+0000 7fb8ea521700 1 --2- 172.21.15.42:0/1618083535 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb8c403ac60 0x7fb8c403d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:30:35.577 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.555+0000 7fb8ea521700 1 --2- 172.21.15.42:0/1618083535 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb8c403ac60 0x7fb8c403d170 crc :-1 s=READY pgs=425 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:30:35.577 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.559+0000 7fb8ec785700 1 -- 172.21.15.42:0/1618083535 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8e405d110 con 0x7fb8e40a6420 2019-12-05T05:30:35.577 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.563+0000 7fb8ead22700 1 --2- 172.21.15.42:0/1618083535 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb8e40a6420 0x7fb8e405f290 secure :-1 s=THROTTLE_DONE pgs=992 cs=0 l=1 rx=0x7fb8e40a2550 tx=0x7fb8dc010eb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:35.577 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.563+0000 7fb8e37fe700 1 -- 172.21.15.42:0/1618083535 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fb8dc032460 con 0x7fb8e40a6420 2019-12-05T05:30:35.747 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.735+0000 7fb8ec785700 1 -- 172.21.15.42:0/1618083535 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb8e405b8c0 con 0x7fb8e40a6420 2019-12-05T05:30:35.747 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.739+0000 7fb8ead22700 1 --2- 172.21.15.42:0/1618083535 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb8e40a6420 0x7fb8e405f290 secure :-1 s=THROTTLE_DONE pgs=992 cs=0 l=1 rx=0x7fb8e40a2550 tx=0x7fb8dc010eb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:35.747 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.739+0000 7fb8e37fe700 1 -- 172.21.15.42:0/1618083535 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 123 v123) v1 ==== 95+0+2112 (secure 0 0 0) 0x7fb8dc0326c0 con 0x7fb8e40a6420 2019-12-05T05:30:35.747 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 123 2019-12-05T05:30:35.748 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:30:35.748 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":123,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6778,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/3028851337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3028851337},{"type":"v1","addr":"172.21.15.42:6835","nonce":3028851337}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":123}],"filesystems":[{"mdsmap":{"epoch":122,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:29:24.937492+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5426},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5426":{"gid":5426,"name":"b","rank":0,"incarnation":119,"state":"up:active","state_seq":18,"addr":"172.21.15.40:6833/3168718372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3168718372},{"type":"v1","addr":"172.21.15.40:6833","nonce":3168718372}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:30:35.750 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.743+0000 7fb8ec785700 1 -- 172.21.15.42:0/1618083535 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb8c403ac60 msgr2=0x7fb8c403d170 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:30:35.750 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.743+0000 7fb8ec785700 1 --2- 172.21.15.42:0/1618083535 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb8c403ac60 0x7fb8c403d170 crc :-1 s=READY pgs=425 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:30:35.751 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.743+0000 7fb8ec785700 1 -- 172.21.15.42:0/1618083535 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb8e40a6420 msgr2=0x7fb8e405f290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:30:35.752 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.743+0000 7fb8ec785700 1 --2- 172.21.15.42:0/1618083535 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb8e40a6420 0x7fb8e405f290 secure :-1 s=READY pgs=992 cs=0 l=1 rx=0x7fb8e40a2550 tx=0x7fb8dc010eb0).stop 2019-12-05T05:30:35.752 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.743+0000 7fb8ec785700 1 -- 172.21.15.42:0/1618083535 shutdown_connections 2019-12-05T05:30:35.752 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.743+0000 7fb8ec785700 1 --2- 172.21.15.42:0/1618083535 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb8c403ac60 0x7fb8c403d170 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:30:35.753 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.743+0000 7fb8ec785700 1 --2- 172.21.15.42:0/1618083535 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb8e40a6420 0x7fb8e405f290 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:30:35.753 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.743+0000 7fb8ec785700 1 --2- 172.21.15.42:0/1618083535 >> v2:172.21.15.40:3300/0 conn(0x7fb8e40a3160 0x7fb8e40a3590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:30:35.754 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.743+0000 7fb8ec785700 1 -- 172.21.15.42:0/1618083535 shutdown_connections 2019-12-05T05:30:35.754 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.743+0000 7fb8ec785700 1 -- 172.21.15.42:0/1618083535 wait complete. 2019-12-05T05:30:35.754 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.743+0000 7fb8ec785700 1 -- 172.21.15.42:0/1618083535 >> 172.21.15.42:0/1618083535 conn(0x7fb8e407e5e0 msgr2=0x7fb8e408eb70 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:30:35.759 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.b (rank=0) 2019-12-05T05:30:35.759 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2019-12-05T05:30:35.759 INFO:teuthology.orchestra.run:waiting for 300 2019-12-05T05:30:35.776 INFO:tasks.ceph.mds.b:Stopped 2019-12-05T05:30:35.776 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:30:35.877 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.863+0000 7faf0f333700 1 Processor -- start 2019-12-05T05:30:35.877 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.863+0000 7faf0f333700 1 -- start start 2019-12-05T05:30:35.877 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.867+0000 7faf0f333700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7faf0808df80 0x7faf0809f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:30:35.877 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.867+0000 7faf0f333700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf0809fc60 con 0x7faf0809f830 2019-12-05T05:30:35.878 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.867+0000 7faf0f333700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf080a0710 con 0x7faf080a7740 2019-12-05T05:30:35.878 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.867+0000 7faf0f333700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf080a0b00 con 0x7faf0808df80 2019-12-05T05:30:35.878 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.867+0000 7faf0d8d0700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7faf080a7740 0x7faf0807e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:35250/0 (socket says 172.21.15.42:35250) 2019-12-05T05:30:35.879 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.867+0000 7faf0d8d0700 1 -- 172.21.15.42:0/752039651 learned_addr learned my addr 172.21.15.42:0/752039651 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:30:35.879 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.867+0000 7faf0d0cf700 1 --2- 172.21.15.42:0/752039651 >> v2:172.21.15.40:3300/0 conn(0x7faf0808df80 0x7faf0809f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:30:35.879 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.867+0000 7faf0c8ce700 1 -- 172.21.15.42:0/752039651 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1818391019 0 0) 0x7faf080a0710 con 0x7faf080a7740 2019-12-05T05:30:35.879 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.867+0000 7faf0c8ce700 1 -- 172.21.15.42:0/752039651 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faef00030b0 con 0x7faf080a7740 2019-12-05T05:30:35.879 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.867+0000 7faf0d0cf700 1 -- 172.21.15.42:0/752039651 >> v1:172.21.15.42:6789/0 conn(0x7faf080a7740 legacy=0x7faf0807e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:30:35.880 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.867+0000 7faf0c8ce700 1 -- 172.21.15.42:0/752039651 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1073219679 0 0) 0x7faf0809fc60 con 0x7faf0809f830 2019-12-05T05:30:35.880 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.867+0000 7faf0d0cf700 1 -- 172.21.15.42:0/752039651 >> v1:172.21.15.40:6789/0 conn(0x7faf0809f830 legacy=0x7faf0814efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:30:35.880 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.867+0000 7faf0d0cf700 1 -- 172.21.15.42:0/752039651 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf0807ef20 con 0x7faf0808df80 2019-12-05T05:30:35.880 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.867+0000 7faf0d0cf700 1 --2- 172.21.15.42:0/752039651 >> v2:172.21.15.40:3300/0 conn(0x7faf0808df80 0x7faf0809f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7faefc008b10 tx=0x7faf000014a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:35.881 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.867+0000 7faf0d0cf700 1 --2- 172.21.15.42:0/752039651 >> v2:172.21.15.40:3300/0 conn(0x7faf0808df80 0x7faf0809f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7faefc008b10 tx=0x7faf000014a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:35.881 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.867+0000 7faf0d0cf700 1 --2- 172.21.15.42:0/752039651 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faf0808df80 0x7faf0809f2e0 secure :-1 s=READY pgs=768 cs=0 l=1 rx=0x7faefc008b10 tx=0x7faf000014a0).ready entity=mon.0 client_cookie=5de98f2fa9ca5ace server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:30:35.881 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.867+0000 7faf0d0cf700 1 --2- 172.21.15.42:0/752039651 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faf0808df80 0x7faf0809f2e0 secure :-1 s=THROTTLE_DONE pgs=768 cs=0 l=1 rx=0x7faefc008b10 tx=0x7faf000014a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:35.881 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.867+0000 7faf0c8ce700 1 -- 172.21.15.42:0/752039651 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faefc0116e0 con 0x7faf0808df80 2019-12-05T05:30:35.882 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.867+0000 7faf0d0cf700 1 --2- 172.21.15.42:0/752039651 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faf0808df80 0x7faf0809f2e0 secure :-1 s=THROTTLE_DONE pgs=768 cs=0 l=1 rx=0x7faefc008b10 tx=0x7faf000014a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:35.883 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.867+0000 7faf0c8ce700 1 -- 172.21.15.42:0/752039651 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf00001230 con 0x7faf0808df80 2019-12-05T05:30:35.883 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.867+0000 7faf0d0cf700 1 --2- 172.21.15.42:0/752039651 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faf0808df80 0x7faf0809f2e0 secure :-1 s=THROTTLE_DONE pgs=768 cs=0 l=1 rx=0x7faefc008b10 tx=0x7faf000014a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:35.883 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.867+0000 7faf0c8ce700 1 -- 172.21.15.42:0/752039651 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faefc011cc0 con 0x7faf0808df80 2019-12-05T05:30:35.883 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.867+0000 7faf0f333700 1 -- 172.21.15.42:0/752039651 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faf0808df80 msgr2=0x7faf0809f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:30:35.884 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.867+0000 7faf0f333700 1 --2- 172.21.15.42:0/752039651 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faf0808df80 0x7faf0809f2e0 secure :-1 s=READY pgs=768 cs=0 l=1 rx=0x7faefc008b10 tx=0x7faf000014a0).stop 2019-12-05T05:30:35.884 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.867+0000 7faf0f333700 1 -- 172.21.15.42:0/752039651 shutdown_connections 2019-12-05T05:30:35.884 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.867+0000 7faf0f333700 1 --2- 172.21.15.42:0/752039651 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faf0808df80 0x7faf0809f2e0 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:30:35.884 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.867+0000 7faf0f333700 1 -- 172.21.15.42:0/752039651 shutdown_connections 2019-12-05T05:30:35.884 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.867+0000 7faf0f333700 1 -- 172.21.15.42:0/752039651 wait complete. 2019-12-05T05:30:35.885 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.867+0000 7faf0f333700 1 -- 172.21.15.42:0/752039651 >> 172.21.15.42:0/752039651 conn(0x7faf080aabe0 msgr2=0x7faf080a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:30:35.891 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.871+0000 7faf0f333700 1 Processor -- start 2019-12-05T05:30:35.892 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.871+0000 7faf0f333700 1 -- start start 2019-12-05T05:30:35.892 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.875+0000 7faf0f333700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7faf08119420 0x7faf0807d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:30:35.892 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.875+0000 7faf0f333700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7faf08118be0 0x7faf081373a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:30:35.893 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.875+0000 7faf0f333700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf0811a880 con 0x7faf0807ddf0 2019-12-05T05:30:35.893 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.875+0000 7faf0f333700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf0811c360 con 0x7faf08118be0 2019-12-05T05:30:35.893 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.875+0000 7faf0f333700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf08142ab0 con 0x7faf08119420 2019-12-05T05:30:35.893 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.875+0000 7faf0d8d0700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7faf08119420 0x7faf0807d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:30:35.893 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.875+0000 7faf0d8d0700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7faf08119420 0x7faf0807d8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:35218/0 (socket says 172.21.15.42:35218) 2019-12-05T05:30:35.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.875+0000 7faf0d8d0700 1 -- 172.21.15.42:0/41379298 learned_addr learned my addr 172.21.15.42:0/41379298 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:30:35.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.875+0000 7faf0e0d1700 1 --2- 172.21.15.42:0/41379298 >> v2:172.21.15.40:3300/0 conn(0x7faf08118be0 0x7faf081373a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:30:35.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.875+0000 7faf0d8d0700 1 -- 172.21.15.42:0/41379298 >> v2:172.21.15.40:3300/0 conn(0x7faf08118be0 msgr2=0x7faf081373a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:30:35.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.875+0000 7faf0d8d0700 1 --2- 172.21.15.42:0/41379298 >> v2:172.21.15.40:3300/0 conn(0x7faf08118be0 0x7faf081373a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:30:35.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.875+0000 7faf0d8d0700 1 -- 172.21.15.42:0/41379298 >> v1:172.21.15.40:6789/0 conn(0x7faf0807ddf0 legacy=0x7faf081183e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:30:35.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.875+0000 7faf067fc700 1 -- 172.21.15.42:0/41379298 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 330468398 0 0) 0x7faf0811a880 con 0x7faf0807ddf0 2019-12-05T05:30:35.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.875+0000 7faf0d8d0700 1 -- 172.21.15.42:0/41379298 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf0811a520 con 0x7faf08119420 2019-12-05T05:30:35.895 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.875+0000 7faf0d8d0700 1 --2- 172.21.15.42:0/41379298 >> v2:172.21.15.42:3300/0 conn(0x7faf08119420 0x7faf0807d8a0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7faf000043a0 tx=0x7faf00010a00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:35.895 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.875+0000 7faf0d8d0700 1 --2- 172.21.15.42:0/41379298 >> v2:172.21.15.42:3300/0 conn(0x7faf08119420 0x7faf0807d8a0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7faf000043a0 tx=0x7faf00010a00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:35.895 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.875+0000 7faf0d8d0700 1 --2- 172.21.15.42:0/41379298 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf08119420 0x7faf0807d8a0 secure :-1 s=READY pgs=994 cs=0 l=1 rx=0x7faf000043a0 tx=0x7faf00010a00).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:30:35.895 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.875+0000 7faf0d8d0700 1 --2- 172.21.15.42:0/41379298 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf08119420 0x7faf0807d8a0 secure :-1 s=THROTTLE_DONE pgs=994 cs=0 l=1 rx=0x7faf000043a0 tx=0x7faf00010a00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:35.895 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.875+0000 7faf0d8d0700 1 --2- 172.21.15.42:0/41379298 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf08119420 0x7faf0807d8a0 secure :-1 s=THROTTLE_DONE pgs=994 cs=0 l=1 rx=0x7faf000043a0 tx=0x7faf00010a00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:35.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.875+0000 7faf067fc700 1 -- 172.21.15.42:0/41379298 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faf0000c6c0 con 0x7faf08119420 2019-12-05T05:30:35.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.875+0000 7faf0d8d0700 1 --2- 172.21.15.42:0/41379298 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf08119420 0x7faf0807d8a0 secure :-1 s=THROTTLE_DONE pgs=994 cs=0 l=1 rx=0x7faf000043a0 tx=0x7faf00010a00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:35.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.875+0000 7faf067fc700 1 -- 172.21.15.42:0/41379298 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf00010170 con 0x7faf08119420 2019-12-05T05:30:35.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.875+0000 7faf0f333700 1 -- 172.21.15.42:0/41379298 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf08142530 con 0x7faf08119420 2019-12-05T05:30:35.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.875+0000 7faf067fc700 1 -- 172.21.15.42:0/41379298 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faf0000cca0 con 0x7faf08119420 2019-12-05T05:30:35.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.875+0000 7faf0f333700 1 -- 172.21.15.42:0/41379298 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf081378f0 con 0x7faf08119420 2019-12-05T05:30:35.897 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.875+0000 7faf0d8d0700 1 --2- 172.21.15.42:0/41379298 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf08119420 0x7faf0807d8a0 secure :-1 s=THROTTLE_DONE pgs=994 cs=0 l=1 rx=0x7faf000043a0 tx=0x7faf00010a00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:35.897 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.875+0000 7faf067fc700 1 -- 172.21.15.42:0/41379298 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7faf00005460 con 0x7faf08119420 2019-12-05T05:30:35.897 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.879+0000 7faf0d8d0700 1 --2- 172.21.15.42:0/41379298 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf08119420 0x7faf0807d8a0 secure :-1 s=THROTTLE_DONE pgs=994 cs=0 l=1 rx=0x7faf000043a0 tx=0x7faf00010a00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:35.897 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.879+0000 7faf067fc700 1 --2- 172.21.15.42:0/41379298 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7faee803ac90 0x7faee803d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:30:35.897 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.879+0000 7faf067fc700 1 -- 172.21.15.42:0/41379298 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7579+0+0 (secure 0 0 0) 0x7faf0004aaf0 con 0x7faf08119420 2019-12-05T05:30:35.898 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.879+0000 7faf0d0cf700 1 --2- 172.21.15.42:0/41379298 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7faee803ac90 0x7faee803d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:30:35.898 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.879+0000 7faf0d0cf700 1 --2- 172.21.15.42:0/41379298 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7faee803ac90 0x7faee803d1a0 crc :-1 s=READY pgs=426 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:30:35.898 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.879+0000 7faf0f333700 1 -- 172.21.15.42:0/41379298 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf0805d110 con 0x7faf08119420 2019-12-05T05:30:35.898 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.887+0000 7faf0d8d0700 1 --2- 172.21.15.42:0/41379298 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf08119420 0x7faf0807d8a0 secure :-1 s=THROTTLE_DONE pgs=994 cs=0 l=1 rx=0x7faf000043a0 tx=0x7faf00010a00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:35.898 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:35.887+0000 7faf067fc700 1 -- 172.21.15.42:0/41379298 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7faf00031b70 con 0x7faf08119420 2019-12-05T05:30:36.057 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:36.051+0000 7faf0f333700 1 -- 172.21.15.42:0/41379298 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faf0805b8c0 con 0x7faf08119420 2019-12-05T05:30:36.057 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:36.051+0000 7faf0d8d0700 1 --2- 172.21.15.42:0/41379298 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf08119420 0x7faf0807d8a0 secure :-1 s=THROTTLE_DONE pgs=994 cs=0 l=1 rx=0x7faf000043a0 tx=0x7faf00010a00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:36.057 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:36.051+0000 7faf067fc700 1 -- 172.21.15.42:0/41379298 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 123 v123) v1 ==== 95+0+2112 (secure 0 0 0) 0x7faf00031dd0 con 0x7faf08119420 2019-12-05T05:30:36.057 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 123 2019-12-05T05:30:36.058 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:30:36.058 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":123,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6778,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/3028851337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3028851337},{"type":"v1","addr":"172.21.15.42:6835","nonce":3028851337}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":123}],"filesystems":[{"mdsmap":{"epoch":122,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:29:24.937492+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5426},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5426":{"gid":5426,"name":"b","rank":0,"incarnation":119,"state":"up:active","state_seq":18,"addr":"172.21.15.40:6833/3168718372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3168718372},{"type":"v1","addr":"172.21.15.40:6833","nonce":3168718372}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:30:36.059 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:36.051+0000 7faf0f333700 1 -- 172.21.15.42:0/41379298 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7faee803ac90 msgr2=0x7faee803d1a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:30:36.059 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:36.051+0000 7faf0f333700 1 --2- 172.21.15.42:0/41379298 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7faee803ac90 0x7faee803d1a0 crc :-1 s=READY pgs=426 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:30:36.060 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:36.051+0000 7faf0f333700 1 -- 172.21.15.42:0/41379298 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf08119420 msgr2=0x7faf0807d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:30:36.060 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:36.051+0000 7faf0f333700 1 --2- 172.21.15.42:0/41379298 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf08119420 0x7faf0807d8a0 secure :-1 s=READY pgs=994 cs=0 l=1 rx=0x7faf000043a0 tx=0x7faf00010a00).stop 2019-12-05T05:30:36.060 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:36.051+0000 7faf0f333700 1 -- 172.21.15.42:0/41379298 shutdown_connections 2019-12-05T05:30:36.060 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:36.051+0000 7faf0f333700 1 --2- 172.21.15.42:0/41379298 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7faee803ac90 0x7faee803d1a0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:30:36.060 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:36.051+0000 7faf0f333700 1 --2- 172.21.15.42:0/41379298 >> v2:172.21.15.40:3300/0 conn(0x7faf08118be0 0x7faf081373a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:30:36.061 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:36.051+0000 7faf0f333700 1 --2- 172.21.15.42:0/41379298 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf08119420 0x7faf0807d8a0 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:30:36.061 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:36.055+0000 7faf0f333700 1 -- 172.21.15.42:0/41379298 shutdown_connections 2019-12-05T05:30:36.061 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:36.055+0000 7faf0f333700 1 -- 172.21.15.42:0/41379298 wait complete. 2019-12-05T05:30:36.061 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:36.055+0000 7faf0f333700 1 -- 172.21.15.42:0/41379298 >> 172.21.15.42:0/41379298 conn(0x7faf080aabe0 msgr2=0x7faf0811c6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:30:36.069 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T05:30:38.070 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:30:38.177 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:38.163+0000 7fbf8e4dd700 1 Processor -- start 2019-12-05T05:30:38.177 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:38.163+0000 7fbf8e4dd700 1 -- start start 2019-12-05T05:30:38.177 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:38.167+0000 7fbf8e4dd700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fbf880a7740 0x7fbf8808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:30:38.177 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:38.167+0000 7fbf8e4dd700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbf8807e720 0x7fbf8807eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:30:38.178 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:38.167+0000 7fbf8e4dd700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf880a0710 con 0x7fbf8809f2e0 2019-12-05T05:30:38.179 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:38.167+0000 7fbf8e4dd700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf880a0b00 con 0x7fbf880a7740 2019-12-05T05:30:38.179 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:38.167+0000 7fbf8e4dd700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf8808e5b0 con 0x7fbf8807e720 2019-12-05T05:30:38.180 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:38.167+0000 7fbf8ca7a700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fbf880a7740 0x7fbf8808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:30:38.180 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:38.167+0000 7fbf8d27b700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fbf8809f2e0 0x7fbf8809f710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:35274/0 (socket says 172.21.15.42:35274) 2019-12-05T05:30:38.180 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:38.167+0000 7fbf8d27b700 1 -- 172.21.15.42:0/780861882 learned_addr learned my addr 172.21.15.42:0/780861882 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:30:38.180 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:38.167+0000 7fbf87fff700 1 --2- 172.21.15.42:0/780861882 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbf8807e720 0x7fbf8807eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:30:38.180 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:38.167+0000 7fbf8ca7a700 1 -- 172.21.15.42:0/780861882 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbf8807e720 msgr2=0x7fbf8807eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:30:38.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:38.167+0000 7fbf8ca7a700 1 --2- 172.21.15.42:0/780861882 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbf8807e720 0x7fbf8807eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:30:38.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:38.167+0000 7fbf8ca7a700 1 -- 172.21.15.42:0/780861882 >> v1:172.21.15.42:6789/0 conn(0x7fbf8809f2e0 legacy=0x7fbf8809f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:30:38.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:38.167+0000 7fbf877fe700 1 -- 172.21.15.42:0/780861882 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 461061159 0 0) 0x7fbf880a0710 con 0x7fbf8809f2e0 2019-12-05T05:30:38.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:38.167+0000 7fbf8ca7a700 1 -- 172.21.15.42:0/780861882 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf8807f0a0 con 0x7fbf880a7740 2019-12-05T05:30:38.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:38.167+0000 7fbf8ca7a700 1 --2- 172.21.15.42:0/780861882 >> v2:172.21.15.40:3300/0 conn(0x7fbf880a7740 0x7fbf8808df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fbf7c008790 tx=0x7fbf7c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:38.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:38.167+0000 7fbf8ca7a700 1 --2- 172.21.15.42:0/780861882 >> v2:172.21.15.40:3300/0 conn(0x7fbf880a7740 0x7fbf8808df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fbf7c008790 tx=0x7fbf7c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:38.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:38.167+0000 7fbf8ca7a700 1 --2- 172.21.15.42:0/780861882 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbf880a7740 0x7fbf8808df80 secure :-1 s=READY pgs=770 cs=0 l=1 rx=0x7fbf7c008790 tx=0x7fbf7c00fc70).ready entity=mon.0 client_cookie=ef8ef22ae541c94b server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:30:38.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:38.167+0000 7fbf8ca7a700 1 --2- 172.21.15.42:0/780861882 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbf880a7740 0x7fbf8808df80 secure :-1 s=THROTTLE_DONE pgs=770 cs=0 l=1 rx=0x7fbf7c008790 tx=0x7fbf7c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:38.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:38.167+0000 7fbf8ca7a700 1 --2- 172.21.15.42:0/780861882 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbf880a7740 0x7fbf8808df80 secure :-1 s=THROTTLE_DONE pgs=770 cs=0 l=1 rx=0x7fbf7c008790 tx=0x7fbf7c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:38.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:38.167+0000 7fbf877fe700 1 -- 172.21.15.42:0/780861882 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbf7c0112f0 con 0x7fbf880a7740 2019-12-05T05:30:38.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:38.167+0000 7fbf8ca7a700 1 --2- 172.21.15.42:0/780861882 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbf880a7740 0x7fbf8808df80 secure :-1 s=THROTTLE_DONE pgs=770 cs=0 l=1 rx=0x7fbf7c008790 tx=0x7fbf7c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:38.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:38.167+0000 7fbf877fe700 1 -- 172.21.15.42:0/780861882 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf7c008ad0 con 0x7fbf880a7740 2019-12-05T05:30:38.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:38.167+0000 7fbf877fe700 1 -- 172.21.15.42:0/780861882 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbf7c0118d0 con 0x7fbf880a7740 2019-12-05T05:30:38.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:38.167+0000 7fbf8e4dd700 1 -- 172.21.15.42:0/780861882 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbf880a7740 msgr2=0x7fbf8808df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:30:38.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:38.167+0000 7fbf8e4dd700 1 --2- 172.21.15.42:0/780861882 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbf880a7740 0x7fbf8808df80 secure :-1 s=READY pgs=770 cs=0 l=1 rx=0x7fbf7c008790 tx=0x7fbf7c00fc70).stop 2019-12-05T05:30:38.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:38.167+0000 7fbf8e4dd700 1 -- 172.21.15.42:0/780861882 shutdown_connections 2019-12-05T05:30:38.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:38.167+0000 7fbf8e4dd700 1 --2- 172.21.15.42:0/780861882 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbf880a7740 0x7fbf8808df80 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:30:38.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:38.167+0000 7fbf8e4dd700 1 --2- 172.21.15.42:0/780861882 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbf8807e720 0x7fbf8807eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:30:38.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:38.167+0000 7fbf8e4dd700 1 -- 172.21.15.42:0/780861882 shutdown_connections 2019-12-05T05:30:38.184 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:38.167+0000 7fbf8e4dd700 1 -- 172.21.15.42:0/780861882 wait complete. 2019-12-05T05:30:38.184 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:38.167+0000 7fbf8e4dd700 1 -- 172.21.15.42:0/780861882 >> 172.21.15.42:0/780861882 conn(0x7fbf880aabe0 msgr2=0x7fbf880a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:30:38.187 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:38.171+0000 7fbf8e4dd700 1 Processor -- start 2019-12-05T05:30:38.187 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:38.171+0000 7fbf8e4dd700 1 -- start start 2019-12-05T05:30:38.187 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:38.175+0000 7fbf8e4dd700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fbf8811b930 0x7fbf880a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:30:38.187 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:38.175+0000 7fbf8e4dd700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fbf880742e0 0x7fbf88076720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:30:38.188 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:38.175+0000 7fbf8e4dd700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf88117010 con 0x7fbf881187a0 2019-12-05T05:30:38.188 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:38.175+0000 7fbf8e4dd700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf88119ec0 con 0x7fbf880742e0 2019-12-05T05:30:38.188 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:38.175+0000 7fbf8e4dd700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf88118e90 con 0x7fbf8811b930 2019-12-05T05:30:38.188 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:38.175+0000 7fbf8ca7a700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fbf8811b930 0x7fbf880a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:30:38.188 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:38.175+0000 7fbf8ca7a700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fbf8811b930 0x7fbf880a6070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:35238/0 (socket says 172.21.15.42:35238) 2019-12-05T05:30:38.188 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:38.175+0000 7fbf8ca7a700 1 -- 172.21.15.42:0/3171841607 learned_addr learned my addr 172.21.15.42:0/3171841607 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:30:38.188 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:38.175+0000 7fbf8d27b700 1 --2- 172.21.15.42:0/3171841607 >> v2:172.21.15.40:3300/0 conn(0x7fbf880742e0 0x7fbf88076720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:30:38.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:38.175+0000 7fbf8ca7a700 1 -- 172.21.15.42:0/3171841607 >> v2:172.21.15.40:3300/0 conn(0x7fbf880742e0 msgr2=0x7fbf88076720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:30:38.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:38.175+0000 7fbf8ca7a700 1 --2- 172.21.15.42:0/3171841607 >> v2:172.21.15.40:3300/0 conn(0x7fbf880742e0 0x7fbf88076720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:30:38.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:38.175+0000 7fbf8ca7a700 1 -- 172.21.15.42:0/3171841607 >> v1:172.21.15.40:6789/0 conn(0x7fbf881187a0 legacy=0x7fbf881196c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:30:38.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:38.175+0000 7fbf8ca7a700 1 -- 172.21.15.42:0/3171841607 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf88118bd0 con 0x7fbf8811b930 2019-12-05T05:30:38.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:38.175+0000 7fbf8ca7a700 1 --2- 172.21.15.42:0/3171841607 >> v2:172.21.15.42:3300/0 conn(0x7fbf8811b930 0x7fbf880a6070 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fbf7c008790 tx=0x7fbf7c011a10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:38.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:38.175+0000 7fbf8ca7a700 1 --2- 172.21.15.42:0/3171841607 >> v2:172.21.15.42:3300/0 conn(0x7fbf8811b930 0x7fbf880a6070 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fbf7c008790 tx=0x7fbf7c011a10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:38.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:38.175+0000 7fbf8ca7a700 1 --2- 172.21.15.42:0/3171841607 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbf8811b930 0x7fbf880a6070 secure :-1 s=READY pgs=996 cs=0 l=1 rx=0x7fbf7c008790 tx=0x7fbf7c011a10).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:30:38.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:38.175+0000 7fbf8ca7a700 1 --2- 172.21.15.42:0/3171841607 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbf8811b930 0x7fbf880a6070 secure :-1 s=THROTTLE_DONE pgs=996 cs=0 l=1 rx=0x7fbf7c008790 tx=0x7fbf7c011a10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:38.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:38.175+0000 7fbf8ca7a700 1 --2- 172.21.15.42:0/3171841607 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbf8811b930 0x7fbf880a6070 secure :-1 s=THROTTLE_DONE pgs=996 cs=0 l=1 rx=0x7fbf7c008790 tx=0x7fbf7c011a10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:38.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:38.175+0000 7fbf857fa700 1 -- 172.21.15.42:0/3171841607 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbf7c007150 con 0x7fbf8811b930 2019-12-05T05:30:38.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:38.175+0000 7fbf8ca7a700 1 --2- 172.21.15.42:0/3171841607 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbf8811b930 0x7fbf880a6070 secure :-1 s=THROTTLE_DONE pgs=996 cs=0 l=1 rx=0x7fbf7c008790 tx=0x7fbf7c011a10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:38.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:38.175+0000 7fbf857fa700 1 -- 172.21.15.42:0/3171841607 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf7c0114f0 con 0x7fbf8811b930 2019-12-05T05:30:38.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:38.175+0000 7fbf8e4dd700 1 -- 172.21.15.42:0/3171841607 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf88076c70 con 0x7fbf8811b930 2019-12-05T05:30:38.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:38.175+0000 7fbf857fa700 1 -- 172.21.15.42:0/3171841607 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbf7c00f550 con 0x7fbf8811b930 2019-12-05T05:30:38.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:38.175+0000 7fbf8e4dd700 1 -- 172.21.15.42:0/3171841607 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf88077170 con 0x7fbf8811b930 2019-12-05T05:30:38.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:38.175+0000 7fbf8ca7a700 1 --2- 172.21.15.42:0/3171841607 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbf8811b930 0x7fbf880a6070 secure :-1 s=THROTTLE_DONE pgs=996 cs=0 l=1 rx=0x7fbf7c008790 tx=0x7fbf7c011a10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:38.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:38.175+0000 7fbf857fa700 1 -- 172.21.15.42:0/3171841607 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fbf7c005540 con 0x7fbf8811b930 2019-12-05T05:30:38.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:38.175+0000 7fbf8ca7a700 1 --2- 172.21.15.42:0/3171841607 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbf8811b930 0x7fbf880a6070 secure :-1 s=THROTTLE_DONE pgs=996 cs=0 l=1 rx=0x7fbf7c008790 tx=0x7fbf7c011a10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:38.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:38.175+0000 7fbf857fa700 1 --2- 172.21.15.42:0/3171841607 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbf6803ada0 0x7fbf6803d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:30:38.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:38.175+0000 7fbf857fa700 1 -- 172.21.15.42:0/3171841607 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7579+0+0 (secure 0 0 0) 0x7fbf7c04b920 con 0x7fbf8811b930 2019-12-05T05:30:38.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:38.175+0000 7fbf87fff700 1 --2- 172.21.15.42:0/3171841607 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbf6803ada0 0x7fbf6803d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:30:38.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:38.179+0000 7fbf87fff700 1 --2- 172.21.15.42:0/3171841607 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbf6803ada0 0x7fbf6803d2b0 crc :-1 s=READY pgs=427 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:30:38.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:38.179+0000 7fbf8e4dd700 1 -- 172.21.15.42:0/3171841607 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf8805d110 con 0x7fbf8811b930 2019-12-05T05:30:38.193 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:38.187+0000 7fbf8ca7a700 1 --2- 172.21.15.42:0/3171841607 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbf8811b930 0x7fbf880a6070 secure :-1 s=THROTTLE_DONE pgs=996 cs=0 l=1 rx=0x7fbf7c008790 tx=0x7fbf7c011a10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:38.193 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:38.187+0000 7fbf857fa700 1 -- 172.21.15.42:0/3171841607 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fbf88118bd0 con 0x7fbf8811b930 2019-12-05T05:30:38.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:38.347+0000 7fbf8e4dd700 1 -- 172.21.15.42:0/3171841607 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbf8805b8c0 con 0x7fbf8811b930 2019-12-05T05:30:38.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:38.347+0000 7fbf8ca7a700 1 --2- 172.21.15.42:0/3171841607 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbf8811b930 0x7fbf880a6070 secure :-1 s=THROTTLE_DONE pgs=996 cs=0 l=1 rx=0x7fbf7c008790 tx=0x7fbf7c011a10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:38.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:38.347+0000 7fbf857fa700 1 -- 172.21.15.42:0/3171841607 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 123 v123) v1 ==== 95+0+2112 (secure 0 0 0) 0x7fbf7c0305d0 con 0x7fbf8811b930 2019-12-05T05:30:38.357 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 123 2019-12-05T05:30:38.357 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:30:38.357 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":123,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6778,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/3028851337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3028851337},{"type":"v1","addr":"172.21.15.42:6835","nonce":3028851337}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":123}],"filesystems":[{"mdsmap":{"epoch":122,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:29:24.937492+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5426},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5426":{"gid":5426,"name":"b","rank":0,"incarnation":119,"state":"up:active","state_seq":18,"addr":"172.21.15.40:6833/3168718372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3168718372},{"type":"v1","addr":"172.21.15.40:6833","nonce":3168718372}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:30:38.358 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:38.351+0000 7fbf8e4dd700 1 -- 172.21.15.42:0/3171841607 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbf6803ada0 msgr2=0x7fbf6803d2b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:30:38.359 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:38.351+0000 7fbf8e4dd700 1 --2- 172.21.15.42:0/3171841607 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbf6803ada0 0x7fbf6803d2b0 crc :-1 s=READY pgs=427 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:30:38.359 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:38.351+0000 7fbf8e4dd700 1 -- 172.21.15.42:0/3171841607 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbf8811b930 msgr2=0x7fbf880a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:30:38.359 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:38.351+0000 7fbf8e4dd700 1 --2- 172.21.15.42:0/3171841607 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbf8811b930 0x7fbf880a6070 secure :-1 s=READY pgs=996 cs=0 l=1 rx=0x7fbf7c008790 tx=0x7fbf7c011a10).stop 2019-12-05T05:30:38.359 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:38.351+0000 7fbf8e4dd700 1 -- 172.21.15.42:0/3171841607 shutdown_connections 2019-12-05T05:30:38.360 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:38.351+0000 7fbf8e4dd700 1 --2- 172.21.15.42:0/3171841607 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbf6803ada0 0x7fbf6803d2b0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:30:38.360 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:38.351+0000 7fbf8e4dd700 1 --2- 172.21.15.42:0/3171841607 >> v2:172.21.15.40:3300/0 conn(0x7fbf880742e0 0x7fbf88076720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:30:38.360 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:38.351+0000 7fbf8e4dd700 1 --2- 172.21.15.42:0/3171841607 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbf8811b930 0x7fbf880a6070 unknown :-1 s=CLOSED pgs=996 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:30:38.360 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:38.351+0000 7fbf8e4dd700 1 -- 172.21.15.42:0/3171841607 shutdown_connections 2019-12-05T05:30:38.360 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:38.351+0000 7fbf8e4dd700 1 -- 172.21.15.42:0/3171841607 wait complete. 2019-12-05T05:30:38.360 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:38.351+0000 7fbf8e4dd700 1 -- 172.21.15.42:0/3171841607 >> 172.21.15.42:0/3171841607 conn(0x7fbf8811c610 msgr2=0x7fbf8809f380 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:30:38.374 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T05:30:38.952 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:30:38.960 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:30:40.375 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:30:40.482 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:40.471+0000 7fc279290700 1 Processor -- start 2019-12-05T05:30:40.482 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:40.471+0000 7fc279290700 1 -- start start 2019-12-05T05:30:40.482 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:40.471+0000 7fc279290700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc2740a7740 0x7fc27408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:30:40.483 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:40.471+0000 7fc279290700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc27407e720 0x7fc27407eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:30:40.483 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:40.471+0000 7fc279290700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2740a0710 con 0x7fc27409f2e0 2019-12-05T05:30:40.483 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:40.471+0000 7fc279290700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2740a0b00 con 0x7fc27407e720 2019-12-05T05:30:40.483 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:40.471+0000 7fc279290700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc27408e5b0 con 0x7fc2740a7740 2019-12-05T05:30:40.483 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:40.471+0000 7fc272ffd700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc27407e720 0x7fc27407eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:30:40.483 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:40.471+0000 7fc272ffd700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc27407e720 0x7fc27407eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:35248/0 (socket says 172.21.15.42:35248) 2019-12-05T05:30:40.483 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:40.471+0000 7fc272ffd700 1 -- 172.21.15.42:0/1475272859 learned_addr learned my addr 172.21.15.42:0/1475272859 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:30:40.484 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:40.471+0000 7fc2737fe700 1 --2- 172.21.15.42:0/1475272859 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc2740a7740 0x7fc27408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:30:40.484 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:40.471+0000 7fc272ffd700 1 -- 172.21.15.42:0/1475272859 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc2740a7740 msgr2=0x7fc27408df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:30:40.484 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:40.471+0000 7fc272ffd700 1 --2- 172.21.15.42:0/1475272859 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc2740a7740 0x7fc27408df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:30:40.484 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:40.471+0000 7fc272ffd700 1 -- 172.21.15.42:0/1475272859 >> v1:172.21.15.42:6789/0 conn(0x7fc27409f2e0 legacy=0x7fc27409f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:30:40.484 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:40.471+0000 7fc272ffd700 1 -- 172.21.15.42:0/1475272859 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc27407f0a0 con 0x7fc27407e720 2019-12-05T05:30:40.484 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:40.471+0000 7fc272ffd700 1 --2- 172.21.15.42:0/1475272859 >> v2:172.21.15.42:3300/0 conn(0x7fc27407e720 0x7fc27407eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fc25c008790 tx=0x7fc25c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:40.485 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:40.471+0000 7fc272ffd700 1 --2- 172.21.15.42:0/1475272859 >> v2:172.21.15.42:3300/0 conn(0x7fc27407e720 0x7fc27407eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fc25c008790 tx=0x7fc25c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:40.485 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:40.471+0000 7fc272ffd700 1 --2- 172.21.15.42:0/1475272859 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc27407e720 0x7fc27407eb50 secure :-1 s=READY pgs=998 cs=0 l=1 rx=0x7fc25c008790 tx=0x7fc25c00fd30).ready entity=mon.1 client_cookie=b7fceac51b4175dc server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:30:40.485 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:40.471+0000 7fc272ffd700 1 --2- 172.21.15.42:0/1475272859 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc27407e720 0x7fc27407eb50 secure :-1 s=THROTTLE_DONE pgs=998 cs=0 l=1 rx=0x7fc25c008790 tx=0x7fc25c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:40.485 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:40.471+0000 7fc272ffd700 1 --2- 172.21.15.42:0/1475272859 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc27407e720 0x7fc27407eb50 secure :-1 s=THROTTLE_DONE pgs=998 cs=0 l=1 rx=0x7fc25c008790 tx=0x7fc25c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:40.485 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:40.471+0000 7fc2727fc700 1 -- 172.21.15.42:0/1475272859 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc25c0113b0 con 0x7fc27407e720 2019-12-05T05:30:40.485 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:40.471+0000 7fc272ffd700 1 --2- 172.21.15.42:0/1475272859 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc27407e720 0x7fc27407eb50 secure :-1 s=THROTTLE_DONE pgs=998 cs=0 l=1 rx=0x7fc25c008790 tx=0x7fc25c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:40.485 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:40.471+0000 7fc2727fc700 1 -- 172.21.15.42:0/1475272859 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc25c008bb0 con 0x7fc27407e720 2019-12-05T05:30:40.486 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:40.471+0000 7fc2727fc700 1 -- 172.21.15.42:0/1475272859 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc25c011990 con 0x7fc27407e720 2019-12-05T05:30:40.486 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:40.471+0000 7fc279290700 1 -- 172.21.15.42:0/1475272859 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc27407e720 msgr2=0x7fc27407eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:30:40.486 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:40.471+0000 7fc279290700 1 --2- 172.21.15.42:0/1475272859 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc27407e720 0x7fc27407eb50 secure :-1 s=READY pgs=998 cs=0 l=1 rx=0x7fc25c008790 tx=0x7fc25c00fd30).stop 2019-12-05T05:30:40.486 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:40.475+0000 7fc279290700 1 -- 172.21.15.42:0/1475272859 shutdown_connections 2019-12-05T05:30:40.486 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:40.475+0000 7fc279290700 1 --2- 172.21.15.42:0/1475272859 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc2740a7740 0x7fc27408df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:30:40.487 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:40.475+0000 7fc279290700 1 --2- 172.21.15.42:0/1475272859 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc27407e720 0x7fc27407eb50 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:30:40.487 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:40.475+0000 7fc279290700 1 -- 172.21.15.42:0/1475272859 shutdown_connections 2019-12-05T05:30:40.487 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:40.475+0000 7fc279290700 1 -- 172.21.15.42:0/1475272859 wait complete. 2019-12-05T05:30:40.487 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:40.475+0000 7fc279290700 1 -- 172.21.15.42:0/1475272859 >> 172.21.15.42:0/1475272859 conn(0x7fc2740aabe0 msgr2=0x7fc2740a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:30:40.491 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:40.475+0000 7fc279290700 1 Processor -- start 2019-12-05T05:30:40.491 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:40.475+0000 7fc279290700 1 -- start start 2019-12-05T05:30:40.491 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:40.479+0000 7fc279290700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc274113760 0x7fc274117f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:30:40.491 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:40.479+0000 7fc279290700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc2740734c0 0x7fc2740738f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:30:40.492 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:40.479+0000 7fc279290700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2740726b0 con 0x7fc27408eb70 2019-12-05T05:30:40.492 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:40.479+0000 7fc279290700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc274073e40 con 0x7fc274113760 2019-12-05T05:30:40.492 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:40.479+0000 7fc279290700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc274073040 con 0x7fc2740734c0 2019-12-05T05:30:40.492 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:40.479+0000 7fc272ffd700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc2740734c0 0x7fc2740738f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:30:40.492 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:40.479+0000 7fc272ffd700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc2740734c0 0x7fc2740738f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:35258/0 (socket says 172.21.15.42:35258) 2019-12-05T05:30:40.493 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:40.479+0000 7fc272ffd700 1 -- 172.21.15.42:0/3470769499 learned_addr learned my addr 172.21.15.42:0/3470769499 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:30:40.493 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:40.479+0000 7fc273fff700 1 --1- 172.21.15.42:0/3470769499 >> v1:172.21.15.42:6789/0 conn(0x7fc27408eb70 0x7fc27408efa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:35294/0 (socket says 172.21.15.42:35294) 2019-12-05T05:30:40.493 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:40.479+0000 7fc2737fe700 1 --2- 172.21.15.42:0/3470769499 >> v2:172.21.15.40:3300/0 conn(0x7fc274113760 0x7fc274117f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:30:40.493 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:40.479+0000 7fc272ffd700 1 -- 172.21.15.42:0/3470769499 >> v2:172.21.15.40:3300/0 conn(0x7fc274113760 msgr2=0x7fc274117f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:30:40.493 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:40.479+0000 7fc272ffd700 1 --2- 172.21.15.42:0/3470769499 >> v2:172.21.15.40:3300/0 conn(0x7fc274113760 0x7fc274117f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:30:40.493 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:40.479+0000 7fc272ffd700 1 -- 172.21.15.42:0/3470769499 >> v1:172.21.15.42:6789/0 conn(0x7fc27408eb70 legacy=0x7fc27408efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:30:40.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:40.479+0000 7fc272ffd700 1 -- 172.21.15.42:0/3470769499 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc27408f7a0 con 0x7fc2740734c0 2019-12-05T05:30:40.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:40.479+0000 7fc26bfff700 1 -- 172.21.15.42:0/3470769499 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2531367180 0 0) 0x7fc2740726b0 con 0x7fc27408eb70 2019-12-05T05:30:40.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:40.479+0000 7fc272ffd700 1 --2- 172.21.15.42:0/3470769499 >> v2:172.21.15.42:3300/0 conn(0x7fc2740734c0 0x7fc2740738f0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fc2740a7390 tx=0x7fc25c008bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:40.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:40.479+0000 7fc272ffd700 1 --2- 172.21.15.42:0/3470769499 >> v2:172.21.15.42:3300/0 conn(0x7fc2740734c0 0x7fc2740738f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fc2740a7390 tx=0x7fc25c008bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:40.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:40.479+0000 7fc272ffd700 1 --2- 172.21.15.42:0/3470769499 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc2740734c0 0x7fc2740738f0 secure :-1 s=READY pgs=1000 cs=0 l=1 rx=0x7fc2740a7390 tx=0x7fc25c008bb0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:30:40.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:40.479+0000 7fc272ffd700 1 --2- 172.21.15.42:0/3470769499 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc2740734c0 0x7fc2740738f0 secure :-1 s=THROTTLE_DONE pgs=1000 cs=0 l=1 rx=0x7fc2740a7390 tx=0x7fc25c008bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:40.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:40.479+0000 7fc26bfff700 1 -- 172.21.15.42:0/3470769499 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc25c007340 con 0x7fc2740734c0 2019-12-05T05:30:40.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:40.479+0000 7fc272ffd700 1 --2- 172.21.15.42:0/3470769499 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc2740734c0 0x7fc2740738f0 secure :-1 s=THROTTLE_DONE pgs=1000 cs=0 l=1 rx=0x7fc2740a7390 tx=0x7fc25c008bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:40.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:40.479+0000 7fc272ffd700 1 --2- 172.21.15.42:0/3470769499 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc2740734c0 0x7fc2740738f0 secure :-1 s=THROTTLE_DONE pgs=1000 cs=0 l=1 rx=0x7fc2740a7390 tx=0x7fc25c008bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:40.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:40.479+0000 7fc26bfff700 1 -- 172.21.15.42:0/3470769499 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc25c0115b0 con 0x7fc2740734c0 2019-12-05T05:30:40.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:40.479+0000 7fc26bfff700 1 -- 172.21.15.42:0/3470769499 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc25c005540 con 0x7fc2740734c0 2019-12-05T05:30:40.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:40.479+0000 7fc279290700 1 -- 172.21.15.42:0/3470769499 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2741318e0 con 0x7fc2740734c0 2019-12-05T05:30:40.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:40.479+0000 7fc279290700 1 -- 172.21.15.42:0/3470769499 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc274131de0 con 0x7fc2740734c0 2019-12-05T05:30:40.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:40.479+0000 7fc272ffd700 1 --2- 172.21.15.42:0/3470769499 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc2740734c0 0x7fc2740738f0 secure :-1 s=THROTTLE_DONE pgs=1000 cs=0 l=1 rx=0x7fc2740a7390 tx=0x7fc25c008bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:40.496 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:40.479+0000 7fc26bfff700 1 -- 172.21.15.42:0/3470769499 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fc25c0057d0 con 0x7fc2740734c0 2019-12-05T05:30:40.496 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:40.479+0000 7fc272ffd700 1 --2- 172.21.15.42:0/3470769499 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc2740734c0 0x7fc2740738f0 secure :-1 s=THROTTLE_DONE pgs=1000 cs=0 l=1 rx=0x7fc2740a7390 tx=0x7fc25c008bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:40.496 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:40.479+0000 7fc26bfff700 1 --2- 172.21.15.42:0/3470769499 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc25803ac80 0x7fc25803d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:30:40.496 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:40.483+0000 7fc26bfff700 1 -- 172.21.15.42:0/3470769499 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7579+0+0 (secure 0 0 0) 0x7fc25c04b400 con 0x7fc2740734c0 2019-12-05T05:30:40.496 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:40.483+0000 7fc2737fe700 1 --2- 172.21.15.42:0/3470769499 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc25803ac80 0x7fc25803d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:30:40.496 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:40.483+0000 7fc2737fe700 1 --2- 172.21.15.42:0/3470769499 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc25803ac80 0x7fc25803d190 crc :-1 s=READY pgs=428 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:30:40.496 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:40.483+0000 7fc279290700 1 -- 172.21.15.42:0/3470769499 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc27405d110 con 0x7fc2740734c0 2019-12-05T05:30:40.500 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:40.495+0000 7fc272ffd700 1 --2- 172.21.15.42:0/3470769499 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc2740734c0 0x7fc2740738f0 secure :-1 s=THROTTLE_DONE pgs=1000 cs=0 l=1 rx=0x7fc2740a7390 tx=0x7fc25c008bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:40.501 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:40.495+0000 7fc26bfff700 1 -- 172.21.15.42:0/3470769499 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fc25c0326e0 con 0x7fc2740734c0 2019-12-05T05:30:40.675 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:40.663+0000 7fc279290700 1 -- 172.21.15.42:0/3470769499 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc27405b8c0 con 0x7fc2740734c0 2019-12-05T05:30:40.675 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:40.663+0000 7fc272ffd700 1 --2- 172.21.15.42:0/3470769499 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc2740734c0 0x7fc2740738f0 secure :-1 s=THROTTLE_DONE pgs=1000 cs=0 l=1 rx=0x7fc2740a7390 tx=0x7fc25c008bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:40.675 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:40.663+0000 7fc26bfff700 1 -- 172.21.15.42:0/3470769499 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 123 v123) v1 ==== 95+0+2112 (secure 0 0 0) 0x7fc25c033290 con 0x7fc2740734c0 2019-12-05T05:30:40.675 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 123 2019-12-05T05:30:40.675 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:40.667+0000 7fc279290700 1 -- 172.21.15.42:0/3470769499 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc25803ac80 msgr2=0x7fc25803d190 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:30:40.676 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:40.667+0000 7fc279290700 1 --2- 172.21.15.42:0/3470769499 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc25803ac80 0x7fc25803d190 crc :-1 s=READY pgs=428 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:30:40.676 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:40.667+0000 7fc279290700 1 -- 172.21.15.42:0/3470769499 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc2740734c0 msgr2=0x7fc2740738f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:30:40.676 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:40.667+0000 7fc279290700 1 --2- 172.21.15.42:0/3470769499 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc2740734c0 0x7fc2740738f0 secure :-1 s=READY pgs=1000 cs=0 l=1 rx=0x7fc2740a7390 tx=0x7fc25c008bb0).stop 2019-12-05T05:30:40.676 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:40.667+0000 7fc279290700 1 -- 172.21.15.42:0/3470769499 shutdown_connections 2019-12-05T05:30:40.676 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:40.667+0000 7fc279290700 1 --2- 172.21.15.42:0/3470769499 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc25803ac80 0x7fc25803d190 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:30:40.676 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:40.667+0000 7fc279290700 1 --2- 172.21.15.42:0/3470769499 >> v2:172.21.15.40:3300/0 conn(0x7fc274113760 0x7fc274117f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:30:40.677 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:40.667+0000 7fc279290700 1 --2- 172.21.15.42:0/3470769499 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc2740734c0 0x7fc2740738f0 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:30:40.677 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:40.667+0000 7fc279290700 1 -- 172.21.15.42:0/3470769499 shutdown_connections 2019-12-05T05:30:40.677 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:40.667+0000 7fc279290700 1 -- 172.21.15.42:0/3470769499 wait complete. 2019-12-05T05:30:40.677 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:40.667+0000 7fc279290700 1 -- 172.21.15.42:0/3470769499 >> 172.21.15.42:0/3470769499 conn(0x7fc2740a8230 msgr2=0x7fc2740a9870 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:30:40.677 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:30:40.677 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":123,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6778,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/3028851337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3028851337},{"type":"v1","addr":"172.21.15.42:6835","nonce":3028851337}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":123}],"filesystems":[{"mdsmap":{"epoch":122,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:29:24.937492+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5426},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5426":{"gid":5426,"name":"b","rank":0,"incarnation":119,"state":"up:active","state_seq":18,"addr":"172.21.15.40:6833/3168718372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3168718372},{"type":"v1","addr":"172.21.15.40:6833","nonce":3168718372}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:30:40.684 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T05:30:42.685 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:30:42.790 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:42.779+0000 7fce541a5700 1 Processor -- start 2019-12-05T05:30:42.790 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:42.779+0000 7fce541a5700 1 -- start start 2019-12-05T05:30:42.791 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:42.779+0000 7fce541a5700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fce4c09fae0 0x7fce4c14fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:30:42.791 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:42.779+0000 7fce541a5700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce4c0a0710 con 0x7fce4c08df80 2019-12-05T05:30:42.791 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:42.779+0000 7fce541a5700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce4c07ef20 con 0x7fce4c0a7740 2019-12-05T05:30:42.791 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:42.779+0000 7fce541a5700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce4c0a0b00 con 0x7fce4c09fae0 2019-12-05T05:30:42.791 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:42.779+0000 7fce52742700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fce4c0a7740 0x7fce4c07e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:35302/0 (socket says 172.21.15.42:35302) 2019-12-05T05:30:42.791 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:42.779+0000 7fce52742700 1 -- 172.21.15.42:0/3380226869 learned_addr learned my addr 172.21.15.42:0/3380226869 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:30:42.792 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:42.779+0000 7fce52f43700 1 --2- 172.21.15.42:0/3380226869 >> v2:172.21.15.40:3300/0 conn(0x7fce4c09fae0 0x7fce4c14fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:30:42.792 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:42.779+0000 7fce51740700 1 -- 172.21.15.42:0/3380226869 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1385750487 0 0) 0x7fce4c07ef20 con 0x7fce4c0a7740 2019-12-05T05:30:42.792 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:42.779+0000 7fce51740700 1 -- 172.21.15.42:0/3380226869 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce380030b0 con 0x7fce4c0a7740 2019-12-05T05:30:42.792 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:42.779+0000 7fce52f43700 1 -- 172.21.15.42:0/3380226869 >> v1:172.21.15.42:6789/0 conn(0x7fce4c0a7740 legacy=0x7fce4c07e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:30:42.792 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:42.783+0000 7fce52f43700 1 -- 172.21.15.42:0/3380226869 >> v1:172.21.15.40:6789/0 conn(0x7fce4c08df80 legacy=0x7fce4c09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:30:42.792 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:42.783+0000 7fce52f43700 1 -- 172.21.15.42:0/3380226869 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce4c0a9ba0 con 0x7fce4c09fae0 2019-12-05T05:30:42.792 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:42.783+0000 7fce52f43700 1 --2- 172.21.15.42:0/3380226869 >> v2:172.21.15.40:3300/0 conn(0x7fce4c09fae0 0x7fce4c14fff0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fce44008b10 tx=0x7fce4400ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:42.793 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:42.783+0000 7fce52f43700 1 --2- 172.21.15.42:0/3380226869 >> v2:172.21.15.40:3300/0 conn(0x7fce4c09fae0 0x7fce4c14fff0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fce44008b10 tx=0x7fce4400ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:42.793 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:42.783+0000 7fce52f43700 1 --2- 172.21.15.42:0/3380226869 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fce4c09fae0 0x7fce4c14fff0 secure :-1 s=READY pgs=775 cs=0 l=1 rx=0x7fce44008b10 tx=0x7fce4400ff70).ready entity=mon.0 client_cookie=47996ea135f7ad94 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:30:42.793 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:42.783+0000 7fce52f43700 1 --2- 172.21.15.42:0/3380226869 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fce4c09fae0 0x7fce4c14fff0 secure :-1 s=THROTTLE_DONE pgs=775 cs=0 l=1 rx=0x7fce44008b10 tx=0x7fce4400ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:42.793 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:42.783+0000 7fce51740700 1 -- 172.21.15.42:0/3380226869 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fce44010970 con 0x7fce4c09fae0 2019-12-05T05:30:42.793 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:42.783+0000 7fce52f43700 1 --2- 172.21.15.42:0/3380226869 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fce4c09fae0 0x7fce4c14fff0 secure :-1 s=THROTTLE_DONE pgs=775 cs=0 l=1 rx=0x7fce44008b10 tx=0x7fce4400ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:42.793 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:42.783+0000 7fce51740700 1 -- 172.21.15.42:0/3380226869 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce44008b50 con 0x7fce4c09fae0 2019-12-05T05:30:42.793 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:42.783+0000 7fce52f43700 1 --2- 172.21.15.42:0/3380226869 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fce4c09fae0 0x7fce4c14fff0 secure :-1 s=THROTTLE_DONE pgs=775 cs=0 l=1 rx=0x7fce44008b10 tx=0x7fce4400ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:42.794 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:42.783+0000 7fce541a5700 1 -- 172.21.15.42:0/3380226869 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fce4c09fae0 msgr2=0x7fce4c14fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:30:42.794 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:42.783+0000 7fce541a5700 1 --2- 172.21.15.42:0/3380226869 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fce4c09fae0 0x7fce4c14fff0 secure :-1 s=READY pgs=775 cs=0 l=1 rx=0x7fce44008b10 tx=0x7fce4400ff70).stop 2019-12-05T05:30:42.794 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:42.783+0000 7fce51740700 1 -- 172.21.15.42:0/3380226869 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fce440110c0 con 0x7fce4c09fae0 2019-12-05T05:30:42.794 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:42.783+0000 7fce541a5700 1 -- 172.21.15.42:0/3380226869 shutdown_connections 2019-12-05T05:30:42.794 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:42.783+0000 7fce541a5700 1 --2- 172.21.15.42:0/3380226869 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fce4c09fae0 0x7fce4c14fff0 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:30:42.794 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:42.783+0000 7fce541a5700 1 -- 172.21.15.42:0/3380226869 shutdown_connections 2019-12-05T05:30:42.794 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:42.783+0000 7fce541a5700 1 -- 172.21.15.42:0/3380226869 wait complete. 2019-12-05T05:30:42.794 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:42.783+0000 7fce541a5700 1 -- 172.21.15.42:0/3380226869 >> 172.21.15.42:0/3380226869 conn(0x7fce4c0aabe0 msgr2=0x7fce4c0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:30:42.797 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:42.787+0000 7fce541a5700 1 Processor -- start 2019-12-05T05:30:42.797 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:42.787+0000 7fce541a5700 1 -- start start 2019-12-05T05:30:42.798 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:42.787+0000 7fce541a5700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fce4c11bd00 0x7fce4c0a1510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:30:42.798 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:42.787+0000 7fce541a5700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fce4c135830 0x7fce4c137c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:30:42.798 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:42.787+0000 7fce541a5700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce4c143860 con 0x7fce4c142720 2019-12-05T05:30:42.798 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:42.787+0000 7fce541a5700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce4c143db0 con 0x7fce4c135830 2019-12-05T05:30:42.798 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:42.787+0000 7fce541a5700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce4c144050 con 0x7fce4c11bd00 2019-12-05T05:30:42.798 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:42.787+0000 7fce52f43700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fce4c135830 0x7fce4c137c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:30:42.798 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:42.787+0000 7fce52742700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fce4c11bd00 0x7fce4c0a1510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:30:42.799 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:42.787+0000 7fce52f43700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fce4c135830 0x7fce4c137c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:35274/0 (socket says 172.21.15.42:35274) 2019-12-05T05:30:42.799 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:42.787+0000 7fce52f43700 1 -- 172.21.15.42:0/855151177 learned_addr learned my addr 172.21.15.42:0/855151177 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:30:42.799 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:42.787+0000 7fce42ffd700 1 -- 172.21.15.42:0/855151177 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3829143486 0 0) 0x7fce4c143860 con 0x7fce4c142720 2019-12-05T05:30:42.799 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:42.787+0000 7fce52f43700 1 -- 172.21.15.42:0/855151177 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fce4c11bd00 msgr2=0x7fce4c0a1510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:30:42.799 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:42.787+0000 7fce52f43700 1 --2- 172.21.15.42:0/855151177 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fce4c11bd00 0x7fce4c0a1510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:30:42.799 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:42.787+0000 7fce52f43700 1 -- 172.21.15.42:0/855151177 >> v1:172.21.15.40:6789/0 conn(0x7fce4c142720 legacy=0x7fce4c142b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:30:42.799 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:42.787+0000 7fce52f43700 1 -- 172.21.15.42:0/855151177 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce4c0a1a60 con 0x7fce4c135830 2019-12-05T05:30:42.800 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:42.787+0000 7fce52742700 1 --2- 172.21.15.42:0/855151177 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fce4c11bd00 0x7fce4c0a1510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:30:42.800 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:42.787+0000 7fce52f43700 1 --2- 172.21.15.42:0/855151177 >> v2:172.21.15.42:3300/0 conn(0x7fce4c135830 0x7fce4c137c70 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fce4c143f70 tx=0x7fce440112f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:42.800 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:42.787+0000 7fce52f43700 1 --2- 172.21.15.42:0/855151177 >> v2:172.21.15.42:3300/0 conn(0x7fce4c135830 0x7fce4c137c70 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fce4c143f70 tx=0x7fce440112f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:42.800 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:42.787+0000 7fce52f43700 1 --2- 172.21.15.42:0/855151177 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fce4c135830 0x7fce4c137c70 secure :-1 s=READY pgs=1002 cs=0 l=1 rx=0x7fce4c143f70 tx=0x7fce440112f0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:30:42.800 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:42.791+0000 7fce52f43700 1 --2- 172.21.15.42:0/855151177 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fce4c135830 0x7fce4c137c70 secure :-1 s=THROTTLE_DONE pgs=1002 cs=0 l=1 rx=0x7fce4c143f70 tx=0x7fce440112f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:42.800 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:42.791+0000 7fce42ffd700 1 -- 172.21.15.42:0/855151177 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fce44005ab0 con 0x7fce4c135830 2019-12-05T05:30:42.800 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:42.791+0000 7fce52f43700 1 --2- 172.21.15.42:0/855151177 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fce4c135830 0x7fce4c137c70 secure :-1 s=THROTTLE_DONE pgs=1002 cs=0 l=1 rx=0x7fce4c143f70 tx=0x7fce440112f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:42.801 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:42.791+0000 7fce541a5700 1 -- 172.21.15.42:0/855151177 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce4c11c130 con 0x7fce4c135830 2019-12-05T05:30:42.801 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:42.791+0000 7fce42ffd700 1 -- 172.21.15.42:0/855151177 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce44010b70 con 0x7fce4c135830 2019-12-05T05:30:42.801 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:42.791+0000 7fce52f43700 1 --2- 172.21.15.42:0/855151177 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fce4c135830 0x7fce4c137c70 secure :-1 s=THROTTLE_DONE pgs=1002 cs=0 l=1 rx=0x7fce4c143f70 tx=0x7fce440112f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:42.801 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:42.791+0000 7fce541a5700 1 -- 172.21.15.42:0/855151177 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce4c1381c0 con 0x7fce4c135830 2019-12-05T05:30:42.801 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:42.791+0000 7fce42ffd700 1 -- 172.21.15.42:0/855151177 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fce4400f680 con 0x7fce4c135830 2019-12-05T05:30:42.801 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:42.791+0000 7fce52f43700 1 --2- 172.21.15.42:0/855151177 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fce4c135830 0x7fce4c137c70 secure :-1 s=THROTTLE_DONE pgs=1002 cs=0 l=1 rx=0x7fce4c143f70 tx=0x7fce440112f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:42.801 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:42.791+0000 7fce42ffd700 1 -- 172.21.15.42:0/855151177 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fce4400f8d0 con 0x7fce4c135830 2019-12-05T05:30:42.802 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:42.791+0000 7fce52f43700 1 --2- 172.21.15.42:0/855151177 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fce4c135830 0x7fce4c137c70 secure :-1 s=THROTTLE_DONE pgs=1002 cs=0 l=1 rx=0x7fce4c143f70 tx=0x7fce440112f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:42.802 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:42.791+0000 7fce42ffd700 1 --2- 172.21.15.42:0/855151177 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fce3403acd0 0x7fce3403d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:30:42.803 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:42.791+0000 7fce42ffd700 1 -- 172.21.15.42:0/855151177 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7579+0+0 (secure 0 0 0) 0x7fce4404b080 con 0x7fce4c135830 2019-12-05T05:30:42.803 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:42.795+0000 7fce52742700 1 --2- 172.21.15.42:0/855151177 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fce3403acd0 0x7fce3403d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:30:42.803 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:42.795+0000 7fce52742700 1 --2- 172.21.15.42:0/855151177 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fce3403acd0 0x7fce3403d1e0 crc :-1 s=READY pgs=429 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:30:42.803 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:42.795+0000 7fce541a5700 1 -- 172.21.15.42:0/855151177 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce4c05d110 con 0x7fce4c135830 2019-12-05T05:30:42.808 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:42.803+0000 7fce52f43700 1 --2- 172.21.15.42:0/855151177 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fce4c135830 0x7fce4c137c70 secure :-1 s=THROTTLE_DONE pgs=1002 cs=0 l=1 rx=0x7fce4c143f70 tx=0x7fce440112f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:42.808 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:42.803+0000 7fce42ffd700 1 -- 172.21.15.42:0/855151177 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fce44032ef0 con 0x7fce4c135830 2019-12-05T05:30:42.980 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:42.975+0000 7fce541a5700 1 -- 172.21.15.42:0/855151177 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fce4c05b8c0 con 0x7fce4c135830 2019-12-05T05:30:42.980 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:42.975+0000 7fce52f43700 1 --2- 172.21.15.42:0/855151177 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fce4c135830 0x7fce4c137c70 secure :-1 s=THROTTLE_DONE pgs=1002 cs=0 l=1 rx=0x7fce4c143f70 tx=0x7fce440112f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:42.980 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:42.975+0000 7fce42ffd700 1 -- 172.21.15.42:0/855151177 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 123 v123) v1 ==== 95+0+2112 (secure 0 0 0) 0x7fce440339d0 con 0x7fce4c135830 2019-12-05T05:30:42.981 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 123 2019-12-05T05:30:42.981 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:30:42.981 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":123,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6778,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/3028851337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3028851337},{"type":"v1","addr":"172.21.15.42:6835","nonce":3028851337}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":123}],"filesystems":[{"mdsmap":{"epoch":122,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:29:24.937492+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5426},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5426":{"gid":5426,"name":"b","rank":0,"incarnation":119,"state":"up:active","state_seq":18,"addr":"172.21.15.40:6833/3168718372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3168718372},{"type":"v1","addr":"172.21.15.40:6833","nonce":3168718372}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:30:42.982 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:42.975+0000 7fce541a5700 1 -- 172.21.15.42:0/855151177 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fce3403acd0 msgr2=0x7fce3403d1e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:30:42.983 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:42.975+0000 7fce541a5700 1 --2- 172.21.15.42:0/855151177 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fce3403acd0 0x7fce3403d1e0 crc :-1 s=READY pgs=429 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:30:42.983 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:42.975+0000 7fce541a5700 1 -- 172.21.15.42:0/855151177 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fce4c135830 msgr2=0x7fce4c137c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:30:42.983 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:42.975+0000 7fce541a5700 1 --2- 172.21.15.42:0/855151177 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fce4c135830 0x7fce4c137c70 secure :-1 s=READY pgs=1002 cs=0 l=1 rx=0x7fce4c143f70 tx=0x7fce440112f0).stop 2019-12-05T05:30:42.983 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:42.975+0000 7fce541a5700 1 -- 172.21.15.42:0/855151177 shutdown_connections 2019-12-05T05:30:42.983 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:42.975+0000 7fce541a5700 1 --2- 172.21.15.42:0/855151177 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fce3403acd0 0x7fce3403d1e0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:30:42.984 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:42.975+0000 7fce541a5700 1 --2- 172.21.15.42:0/855151177 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fce4c135830 0x7fce4c137c70 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:30:42.985 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:42.975+0000 7fce541a5700 1 --2- 172.21.15.42:0/855151177 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fce4c11bd00 0x7fce4c0a1510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:30:42.985 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:42.975+0000 7fce541a5700 1 -- 172.21.15.42:0/855151177 shutdown_connections 2019-12-05T05:30:42.985 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:42.975+0000 7fce541a5700 1 -- 172.21.15.42:0/855151177 wait complete. 2019-12-05T05:30:42.985 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:42.975+0000 7fce541a5700 1 -- 172.21.15.42:0/855151177 >> 172.21.15.42:0/855151177 conn(0x7fce4c119230 msgr2=0x7fce4c059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:30:42.993 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T05:30:44.996 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:30:45.113 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:45.091+0000 7f068b9ce700 1 Processor -- start 2019-12-05T05:30:45.113 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:45.091+0000 7f068b9ce700 1 -- start start 2019-12-05T05:30:45.114 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:45.095+0000 7f068b9ce700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f06840a7740 0x7f068408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:30:45.114 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:45.095+0000 7f068b9ce700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06840a0710 con 0x7f068407e720 2019-12-05T05:30:45.114 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:45.095+0000 7f068b9ce700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06840a9ba0 con 0x7f068409fae0 2019-12-05T05:30:45.114 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:45.095+0000 7f068b9ce700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06840a0b00 con 0x7f06840a7740 2019-12-05T05:30:45.114 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:45.095+0000 7f068a76c700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f068409fae0 0x7f068414efe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:35320/0 (socket says 172.21.15.42:35320) 2019-12-05T05:30:45.114 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:45.095+0000 7f068a76c700 1 -- 172.21.15.42:0/1289945586 learned_addr learned my addr 172.21.15.42:0/1289945586 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:30:45.114 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:45.095+0000 7f0689f6b700 1 --2- 172.21.15.42:0/1289945586 >> v2:172.21.15.40:3300/0 conn(0x7f06840a7740 0x7f068408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:30:45.115 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:45.095+0000 7f0688f69700 1 -- 172.21.15.42:0/1289945586 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1688158655 0 0) 0x7f06840a9ba0 con 0x7f068409fae0 2019-12-05T05:30:45.115 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:45.095+0000 7f0688f69700 1 -- 172.21.15.42:0/1289945586 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06700030b0 con 0x7f068409fae0 2019-12-05T05:30:45.115 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:45.095+0000 7f0689f6b700 1 -- 172.21.15.42:0/1289945586 >> v1:172.21.15.42:6789/0 conn(0x7f068409fae0 legacy=0x7f068414efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:30:45.115 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:45.095+0000 7f0689f6b700 1 -- 172.21.15.42:0/1289945586 >> v1:172.21.15.40:6789/0 conn(0x7f068407e720 legacy=0x7f068409f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:30:45.115 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:45.095+0000 7f0689f6b700 1 -- 172.21.15.42:0/1289945586 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f068407eb50 con 0x7f06840a7740 2019-12-05T05:30:45.115 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:45.095+0000 7f0689f6b700 1 --2- 172.21.15.42:0/1289945586 >> v2:172.21.15.40:3300/0 conn(0x7f06840a7740 0x7f068408df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f0680008790 tx=0x7f068000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:45.115 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:45.095+0000 7f0689f6b700 1 --2- 172.21.15.42:0/1289945586 >> v2:172.21.15.40:3300/0 conn(0x7f06840a7740 0x7f068408df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f0680008790 tx=0x7f068000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:45.116 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:45.095+0000 7f0689f6b700 1 --2- 172.21.15.42:0/1289945586 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f06840a7740 0x7f068408df80 secure :-1 s=READY pgs=778 cs=0 l=1 rx=0x7f0680008790 tx=0x7f068000fbf0).ready entity=mon.0 client_cookie=ff943ee2bd0c2eea server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:30:45.116 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:45.095+0000 7f0689f6b700 1 --2- 172.21.15.42:0/1289945586 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f06840a7740 0x7f068408df80 secure :-1 s=THROTTLE_DONE pgs=778 cs=0 l=1 rx=0x7f0680008790 tx=0x7f068000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:45.117 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:45.095+0000 7f0689f6b700 1 --2- 172.21.15.42:0/1289945586 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f06840a7740 0x7f068408df80 secure :-1 s=THROTTLE_DONE pgs=778 cs=0 l=1 rx=0x7f0680008790 tx=0x7f068000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:45.117 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:45.095+0000 7f0689f6b700 1 --2- 172.21.15.42:0/1289945586 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f06840a7740 0x7f068408df80 secure :-1 s=THROTTLE_DONE pgs=778 cs=0 l=1 rx=0x7f0680008790 tx=0x7f068000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:45.117 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:45.095+0000 7f0688f69700 1 -- 172.21.15.42:0/1289945586 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0680011270 con 0x7f06840a7740 2019-12-05T05:30:45.117 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:45.095+0000 7f0688f69700 1 -- 172.21.15.42:0/1289945586 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0680008bc0 con 0x7f06840a7740 2019-12-05T05:30:45.117 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:45.095+0000 7f0688f69700 1 -- 172.21.15.42:0/1289945586 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0680011850 con 0x7f06840a7740 2019-12-05T05:30:45.117 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:45.095+0000 7f068b9ce700 1 -- 172.21.15.42:0/1289945586 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f06840a7740 msgr2=0x7f068408df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:30:45.118 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:45.095+0000 7f068b9ce700 1 --2- 172.21.15.42:0/1289945586 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f06840a7740 0x7f068408df80 secure :-1 s=READY pgs=778 cs=0 l=1 rx=0x7f0680008790 tx=0x7f068000fbf0).stop 2019-12-05T05:30:45.118 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:45.095+0000 7f068b9ce700 1 -- 172.21.15.42:0/1289945586 shutdown_connections 2019-12-05T05:30:45.121 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:45.095+0000 7f068b9ce700 1 --2- 172.21.15.42:0/1289945586 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f06840a7740 0x7f068408df80 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:30:45.122 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:45.095+0000 7f068b9ce700 1 -- 172.21.15.42:0/1289945586 shutdown_connections 2019-12-05T05:30:45.122 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:45.095+0000 7f068b9ce700 1 -- 172.21.15.42:0/1289945586 wait complete. 2019-12-05T05:30:45.122 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:45.095+0000 7f068b9ce700 1 -- 172.21.15.42:0/1289945586 >> 172.21.15.42:0/1289945586 conn(0x7f06840aabe0 msgr2=0x7f06840a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:30:45.122 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:45.099+0000 7f068b9ce700 1 Processor -- start 2019-12-05T05:30:45.122 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:45.099+0000 7f068b9ce700 1 -- start start 2019-12-05T05:30:45.122 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:45.099+0000 7f068b9ce700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0684142530 0x7f068414ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:30:45.123 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:45.099+0000 7f068b9ce700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f068408fa50 con 0x7f0684116a10 2019-12-05T05:30:45.123 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:45.099+0000 7f068b9ce700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0684142960 con 0x7f068414f280 2019-12-05T05:30:45.123 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:45.099+0000 7f068b9ce700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06840a59c0 con 0x7f0684142530 2019-12-05T05:30:45.123 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:45.099+0000 7f068a76c700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f068414f280 0x7f068412b0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:35326/0 (socket says 172.21.15.42:35326) 2019-12-05T05:30:45.123 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:45.099+0000 7f068a76c700 1 -- 172.21.15.42:0/1743611562 learned_addr learned my addr 172.21.15.42:0/1743611562 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:30:45.123 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:45.099+0000 7f068976a700 1 --2- 172.21.15.42:0/1743611562 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0684142530 0x7f068414ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:30:45.123 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:45.099+0000 7f06767fc700 1 -- 172.21.15.42:0/1743611562 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3476690522 0 0) 0x7f0684142960 con 0x7f068414f280 2019-12-05T05:30:45.123 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:45.103+0000 7f06767fc700 1 -- 172.21.15.42:0/1743611562 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f066c0030b0 con 0x7f068414f280 2019-12-05T05:30:45.124 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:45.103+0000 7f06767fc700 1 -- 172.21.15.42:0/1743611562 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1421701990 0 0) 0x7f068408fa50 con 0x7f0684116a10 2019-12-05T05:30:45.124 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:45.103+0000 7f06767fc700 1 -- 172.21.15.42:0/1743611562 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0684142960 con 0x7f0684116a10 2019-12-05T05:30:45.124 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:45.103+0000 7f068976a700 1 -- 172.21.15.42:0/1743611562 >> v1:172.21.15.42:6789/0 conn(0x7f068414f280 legacy=0x7f068412b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:30:45.124 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:45.103+0000 7f068976a700 1 -- 172.21.15.42:0/1743611562 >> v1:172.21.15.40:6789/0 conn(0x7f0684116a10 legacy=0x7f06841238f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:30:45.124 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:45.103+0000 7f068976a700 1 -- 172.21.15.42:0/1743611562 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06841240f0 con 0x7f0684142530 2019-12-05T05:30:45.124 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:45.103+0000 7f068976a700 1 --2- 172.21.15.42:0/1743611562 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0684142530 0x7f068414ed30 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f0678000f00 tx=0x7f067800c670).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:45.124 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:45.103+0000 7f068976a700 1 --2- 172.21.15.42:0/1743611562 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0684142530 0x7f068414ed30 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f0678000f00 tx=0x7f067800c670).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:45.125 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:45.103+0000 7f068976a700 1 --2- 172.21.15.42:0/1743611562 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0684142530 0x7f068414ed30 secure :-1 s=READY pgs=200 cs=0 l=1 rx=0x7f0678000f00 tx=0x7f067800c670).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:30:45.125 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:45.103+0000 7f068976a700 1 --2- 172.21.15.42:0/1743611562 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0684142530 0x7f068414ed30 secure :-1 s=THROTTLE_DONE pgs=200 cs=0 l=1 rx=0x7f0678000f00 tx=0x7f067800c670).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:45.125 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:45.103+0000 7f068976a700 1 --2- 172.21.15.42:0/1743611562 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0684142530 0x7f068414ed30 secure :-1 s=THROTTLE_DONE pgs=200 cs=0 l=1 rx=0x7f0678000f00 tx=0x7f067800c670).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:45.125 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:45.103+0000 7f068976a700 1 --2- 172.21.15.42:0/1743611562 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0684142530 0x7f068414ed30 secure :-1 s=THROTTLE_DONE pgs=200 cs=0 l=1 rx=0x7f0678000f00 tx=0x7f067800c670).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:45.125 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:45.103+0000 7f06767fc700 1 -- 172.21.15.42:0/1743611562 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f06780101c0 con 0x7f0684142530 2019-12-05T05:30:45.125 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:45.103+0000 7f06767fc700 1 -- 172.21.15.42:0/1743611562 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f067800cb30 con 0x7f0684142530 2019-12-05T05:30:45.125 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:45.103+0000 7f068b9ce700 1 -- 172.21.15.42:0/1743611562 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f068414f6b0 con 0x7f0684142530 2019-12-05T05:30:45.126 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:45.103+0000 7f06767fc700 1 -- 172.21.15.42:0/1743611562 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0678012460 con 0x7f0684142530 2019-12-05T05:30:45.126 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:45.103+0000 7f068b9ce700 1 -- 172.21.15.42:0/1743611562 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f068412c8d0 con 0x7f0684142530 2019-12-05T05:30:45.126 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:45.103+0000 7f068976a700 1 --2- 172.21.15.42:0/1743611562 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0684142530 0x7f068414ed30 secure :-1 s=THROTTLE_DONE pgs=200 cs=0 l=1 rx=0x7f0678000f00 tx=0x7f067800c670).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:45.126 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:45.103+0000 7f06767fc700 1 -- 172.21.15.42:0/1743611562 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f06780066e0 con 0x7f0684142530 2019-12-05T05:30:45.126 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:45.103+0000 7f068976a700 1 --2- 172.21.15.42:0/1743611562 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0684142530 0x7f068414ed30 secure :-1 s=THROTTLE_DONE pgs=200 cs=0 l=1 rx=0x7f0678000f00 tx=0x7f067800c670).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:45.126 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:45.103+0000 7f06767fc700 1 --2- 172.21.15.42:0/1743611562 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f066c03b500 0x7f066c03da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:30:45.126 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:45.103+0000 7f06767fc700 1 -- 172.21.15.42:0/1743611562 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7579+0+0 (secure 0 0 0) 0x7f067804b0f0 con 0x7f0684142530 2019-12-05T05:30:45.127 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:45.103+0000 7f0689f6b700 1 --2- 172.21.15.42:0/1743611562 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f066c03b500 0x7f066c03da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:30:45.127 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:45.107+0000 7f0689f6b700 1 --2- 172.21.15.42:0/1743611562 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f066c03b500 0x7f066c03da10 crc :-1 s=READY pgs=430 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:30:45.127 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:45.107+0000 7f068b9ce700 1 -- 172.21.15.42:0/1743611562 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f068405d110 con 0x7f0684142530 2019-12-05T05:30:45.128 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:45.119+0000 7f068976a700 1 --2- 172.21.15.42:0/1743611562 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0684142530 0x7f068414ed30 secure :-1 s=THROTTLE_DONE pgs=200 cs=0 l=1 rx=0x7f0678000f00 tx=0x7f067800c670).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:45.128 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:45.119+0000 7f06767fc700 1 -- 172.21.15.42:0/1743611562 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f0678032130 con 0x7f0684142530 2019-12-05T05:30:45.295 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:45.287+0000 7f068b9ce700 1 -- 172.21.15.42:0/1743611562 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f068405b8c0 con 0x7f0684142530 2019-12-05T05:30:45.296 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:45.295+0000 7f068976a700 1 --2- 172.21.15.42:0/1743611562 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0684142530 0x7f068414ed30 secure :-1 s=THROTTLE_DONE pgs=200 cs=0 l=1 rx=0x7f0678000f00 tx=0x7f067800c670).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:45.297 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:45.295+0000 7f06767fc700 1 -- 172.21.15.42:0/1743611562 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 123 v123) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f0678032390 con 0x7f0684142530 2019-12-05T05:30:45.298 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 123 2019-12-05T05:30:45.298 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:30:45.299 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":123,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6778,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/3028851337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3028851337},{"type":"v1","addr":"172.21.15.42:6835","nonce":3028851337}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":123}],"filesystems":[{"mdsmap":{"epoch":122,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:29:24.937492+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5426},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5426":{"gid":5426,"name":"b","rank":0,"incarnation":119,"state":"up:active","state_seq":18,"addr":"172.21.15.40:6833/3168718372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3168718372},{"type":"v1","addr":"172.21.15.40:6833","nonce":3168718372}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:30:45.301 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:45.295+0000 7f068b9ce700 1 -- 172.21.15.42:0/1743611562 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f066c03b500 msgr2=0x7f066c03da10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:30:45.301 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:45.295+0000 7f068b9ce700 1 --2- 172.21.15.42:0/1743611562 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f066c03b500 0x7f066c03da10 crc :-1 s=READY pgs=430 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:30:45.301 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:45.295+0000 7f068b9ce700 1 -- 172.21.15.42:0/1743611562 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0684142530 msgr2=0x7f068414ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:30:45.302 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:45.295+0000 7f068b9ce700 1 --2- 172.21.15.42:0/1743611562 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0684142530 0x7f068414ed30 secure :-1 s=READY pgs=200 cs=0 l=1 rx=0x7f0678000f00 tx=0x7f067800c670).stop 2019-12-05T05:30:45.302 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:45.295+0000 7f068b9ce700 1 -- 172.21.15.42:0/1743611562 shutdown_connections 2019-12-05T05:30:45.302 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:45.295+0000 7f068b9ce700 1 --2- 172.21.15.42:0/1743611562 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f066c03b500 0x7f066c03da10 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:30:45.302 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:45.295+0000 7f068b9ce700 1 --2- 172.21.15.42:0/1743611562 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0684142530 0x7f068414ed30 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:30:45.302 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:45.295+0000 7f068b9ce700 1 -- 172.21.15.42:0/1743611562 shutdown_connections 2019-12-05T05:30:45.302 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:45.295+0000 7f068b9ce700 1 -- 172.21.15.42:0/1743611562 wait complete. 2019-12-05T05:30:45.302 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:45.295+0000 7f068b9ce700 1 -- 172.21.15.42:0/1743611562 >> 172.21.15.42:0/1743611562 conn(0x7f068408fde0 msgr2=0x7f06840a8040 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:30:45.311 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T05:30:47.312 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:30:47.423 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:47.411+0000 7f5008f2c700 1 Processor -- start 2019-12-05T05:30:47.424 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:47.411+0000 7f5008f2c700 1 -- start start 2019-12-05T05:30:47.424 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:47.411+0000 7f5008f2c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f50040a7740 0x7f500408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:30:47.424 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:47.411+0000 7f5008f2c700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f500407e720 0x7f500407eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:30:47.424 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:47.411+0000 7f5008f2c700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50040a0710 con 0x7f500409f2e0 2019-12-05T05:30:47.424 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:47.411+0000 7f5008f2c700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50040a0b00 con 0x7f500407e720 2019-12-05T05:30:47.424 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:47.411+0000 7f5008f2c700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f500408e5b0 con 0x7f50040a7740 2019-12-05T05:30:47.424 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:47.415+0000 7f5002d9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f50040a7740 0x7f500408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:30:47.425 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:47.415+0000 7f500259c700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f500407e720 0x7f500407eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:30:47.425 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:47.415+0000 7f5002d9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f50040a7740 0x7f500408df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:35296/0 (socket says 172.21.15.42:35296) 2019-12-05T05:30:47.425 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:47.415+0000 7f5002d9d700 1 -- 172.21.15.42:0/941048565 learned_addr learned my addr 172.21.15.42:0/941048565 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:30:47.425 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:47.415+0000 7f500259c700 1 -- 172.21.15.42:0/941048565 >> v2:172.21.15.42:3300/0 conn(0x7f50040a7740 msgr2=0x7f500408df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:30:47.425 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:47.415+0000 7f500259c700 1 --2- 172.21.15.42:0/941048565 >> v2:172.21.15.42:3300/0 conn(0x7f50040a7740 0x7f500408df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:30:47.425 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:47.415+0000 7f500259c700 1 -- 172.21.15.42:0/941048565 >> v1:172.21.15.42:6789/0 conn(0x7f500409f2e0 legacy=0x7f500409f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:30:47.425 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:47.415+0000 7f500259c700 1 -- 172.21.15.42:0/941048565 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f500407f0a0 con 0x7f500407e720 2019-12-05T05:30:47.426 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:47.415+0000 7f500259c700 1 --2- 172.21.15.42:0/941048565 >> v2:172.21.15.40:3300/0 conn(0x7f500407e720 0x7f500407eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f4fec002f30 tx=0x7f4fec00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:47.426 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:47.415+0000 7f500259c700 1 --2- 172.21.15.42:0/941048565 >> v2:172.21.15.40:3300/0 conn(0x7f500407e720 0x7f500407eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f4fec002f30 tx=0x7f4fec00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:47.426 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:47.415+0000 7f500259c700 1 --2- 172.21.15.42:0/941048565 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f500407e720 0x7f500407eb50 secure :-1 s=READY pgs=780 cs=0 l=1 rx=0x7f4fec002f30 tx=0x7f4fec00fc70).ready entity=mon.0 client_cookie=fa5e8c2641100995 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:30:47.426 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:47.415+0000 7f500259c700 1 --2- 172.21.15.42:0/941048565 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f500407e720 0x7f500407eb50 secure :-1 s=THROTTLE_DONE pgs=780 cs=0 l=1 rx=0x7f4fec002f30 tx=0x7f4fec00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:47.426 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:47.415+0000 7f500259c700 1 --2- 172.21.15.42:0/941048565 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f500407e720 0x7f500407eb50 secure :-1 s=THROTTLE_DONE pgs=780 cs=0 l=1 rx=0x7f4fec002f30 tx=0x7f4fec00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:47.426 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:47.415+0000 7f500259c700 1 --2- 172.21.15.42:0/941048565 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f500407e720 0x7f500407eb50 secure :-1 s=THROTTLE_DONE pgs=780 cs=0 l=1 rx=0x7f4fec002f30 tx=0x7f4fec00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:47.426 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:47.415+0000 7f5001d9b700 1 -- 172.21.15.42:0/941048565 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4fec0112b0 con 0x7f500407e720 2019-12-05T05:30:47.427 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:47.415+0000 7f5001d9b700 1 -- 172.21.15.42:0/941048565 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fec008ad0 con 0x7f500407e720 2019-12-05T05:30:47.427 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:47.415+0000 7f5001d9b700 1 -- 172.21.15.42:0/941048565 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4fec011890 con 0x7f500407e720 2019-12-05T05:30:47.427 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:47.415+0000 7f5008f2c700 1 -- 172.21.15.42:0/941048565 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f500407e720 msgr2=0x7f500407eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:30:47.427 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:47.415+0000 7f5008f2c700 1 --2- 172.21.15.42:0/941048565 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f500407e720 0x7f500407eb50 secure :-1 s=READY pgs=780 cs=0 l=1 rx=0x7f4fec002f30 tx=0x7f4fec00fc70).stop 2019-12-05T05:30:47.427 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:47.415+0000 7f5008f2c700 1 -- 172.21.15.42:0/941048565 shutdown_connections 2019-12-05T05:30:47.427 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:47.415+0000 7f5008f2c700 1 --2- 172.21.15.42:0/941048565 >> v2:172.21.15.42:3300/0 conn(0x7f50040a7740 0x7f500408df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:30:47.427 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:47.415+0000 7f5008f2c700 1 --2- 172.21.15.42:0/941048565 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f500407e720 0x7f500407eb50 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:30:47.428 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:47.415+0000 7f5008f2c700 1 -- 172.21.15.42:0/941048565 shutdown_connections 2019-12-05T05:30:47.428 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:47.415+0000 7f5008f2c700 1 -- 172.21.15.42:0/941048565 wait complete. 2019-12-05T05:30:47.428 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:47.415+0000 7f5008f2c700 1 -- 172.21.15.42:0/941048565 >> 172.21.15.42:0/941048565 conn(0x7f50040aabe0 msgr2=0x7f50040a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:30:47.435 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:47.419+0000 7f5008f2c700 1 Processor -- start 2019-12-05T05:30:47.435 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:47.419+0000 7f5008f2c700 1 -- start start 2019-12-05T05:30:47.436 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:47.419+0000 7f5008f2c700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f500408ee90 0x7f50040653c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:30:47.436 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:47.419+0000 7f5008f2c700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f500411a470 con 0x7f5004065910 2019-12-05T05:30:47.436 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:47.419+0000 7f5008f2c700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50040a4f60 con 0x7f5004122a30 2019-12-05T05:30:47.436 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:47.419+0000 7f5008f2c700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f500411bf60 con 0x7f500408ee90 2019-12-05T05:30:47.436 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:47.419+0000 7f500359e700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f5004122a30 0x7f5004077680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:35344/0 (socket says 172.21.15.42:35344) 2019-12-05T05:30:47.437 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:47.419+0000 7f500359e700 1 -- 172.21.15.42:0/1817501587 learned_addr learned my addr 172.21.15.42:0/1817501587 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:30:47.437 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:47.419+0000 7f5002d9d700 1 --2- 172.21.15.42:0/1817501587 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f500408ee90 0x7f50040653c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:30:47.437 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:47.419+0000 7f4ffb7fe700 1 -- 172.21.15.42:0/1817501587 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1298560269 0 0) 0x7f50040a4f60 con 0x7f5004122a30 2019-12-05T05:30:47.437 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:47.419+0000 7f4ffb7fe700 1 -- 172.21.15.42:0/1817501587 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4fe80030b0 con 0x7f5004122a30 2019-12-05T05:30:47.437 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:47.419+0000 7f4ffb7fe700 1 -- 172.21.15.42:0/1817501587 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2409249752 0 0) 0x7f500411a470 con 0x7f5004065910 2019-12-05T05:30:47.437 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:47.419+0000 7f4ffb7fe700 1 -- 172.21.15.42:0/1817501587 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50040a4f60 con 0x7f5004065910 2019-12-05T05:30:47.437 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:47.419+0000 7f5002d9d700 1 -- 172.21.15.42:0/1817501587 >> v1:172.21.15.42:6789/0 conn(0x7f5004122a30 legacy=0x7f5004077680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:30:47.438 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:47.419+0000 7f5002d9d700 1 -- 172.21.15.42:0/1817501587 >> v1:172.21.15.40:6789/0 conn(0x7f5004065910 legacy=0x7f5004122230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:30:47.438 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:47.419+0000 7f5002d9d700 1 -- 172.21.15.42:0/1817501587 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f500411b1e0 con 0x7f500408ee90 2019-12-05T05:30:47.438 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:47.419+0000 7f5002d9d700 1 --2- 172.21.15.42:0/1817501587 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f500408ee90 0x7f50040653c0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f4ff4002f30 tx=0x7f4ff40109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:47.438 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:47.419+0000 7f5002d9d700 1 --2- 172.21.15.42:0/1817501587 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f500408ee90 0x7f50040653c0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f4ff4002f30 tx=0x7f4ff40109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:47.438 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:47.419+0000 7f5002d9d700 1 --2- 172.21.15.42:0/1817501587 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f500408ee90 0x7f50040653c0 secure :-1 s=READY pgs=201 cs=0 l=1 rx=0x7f4ff4002f30 tx=0x7f4ff40109e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:30:47.438 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:47.423+0000 7f5002d9d700 1 --2- 172.21.15.42:0/1817501587 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f500408ee90 0x7f50040653c0 secure :-1 s=THROTTLE_DONE pgs=201 cs=0 l=1 rx=0x7f4ff4002f30 tx=0x7f4ff40109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:47.438 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:47.423+0000 7f4ffb7fe700 1 -- 172.21.15.42:0/1817501587 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4ff4005930 con 0x7f500408ee90 2019-12-05T05:30:47.439 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:47.423+0000 7f5002d9d700 1 --2- 172.21.15.42:0/1817501587 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f500408ee90 0x7f50040653c0 secure :-1 s=THROTTLE_DONE pgs=201 cs=0 l=1 rx=0x7f4ff4002f30 tx=0x7f4ff40109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:47.439 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:47.423+0000 7f5008f2c700 1 -- 172.21.15.42:0/1817501587 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5004122e60 con 0x7f500408ee90 2019-12-05T05:30:47.439 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:47.423+0000 7f4ffb7fe700 1 -- 172.21.15.42:0/1817501587 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ff4007160 con 0x7f500408ee90 2019-12-05T05:30:47.439 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:47.423+0000 7f5002d9d700 1 --2- 172.21.15.42:0/1817501587 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f500408ee90 0x7f50040653c0 secure :-1 s=THROTTLE_DONE pgs=201 cs=0 l=1 rx=0x7f4ff4002f30 tx=0x7f4ff40109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:47.440 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:47.423+0000 7f4ffb7fe700 1 -- 172.21.15.42:0/1817501587 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4ff40088b0 con 0x7f500408ee90 2019-12-05T05:30:47.440 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:47.423+0000 7f5008f2c700 1 -- 172.21.15.42:0/1817501587 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f500411d540 con 0x7f500408ee90 2019-12-05T05:30:47.440 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:47.423+0000 7f5002d9d700 1 --2- 172.21.15.42:0/1817501587 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f500408ee90 0x7f50040653c0 secure :-1 s=THROTTLE_DONE pgs=201 cs=0 l=1 rx=0x7f4ff4002f30 tx=0x7f4ff40109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:47.440 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:47.423+0000 7f4ffb7fe700 1 -- 172.21.15.42:0/1817501587 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f4ff4008b90 con 0x7f500408ee90 2019-12-05T05:30:47.440 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:47.423+0000 7f5002d9d700 1 --2- 172.21.15.42:0/1817501587 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f500408ee90 0x7f50040653c0 secure :-1 s=THROTTLE_DONE pgs=201 cs=0 l=1 rx=0x7f4ff4002f30 tx=0x7f4ff40109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:47.440 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:47.423+0000 7f4ffb7fe700 1 --2- 172.21.15.42:0/1817501587 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4fe803b3b0 0x7f4fe803d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:30:47.440 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:47.423+0000 7f4ffb7fe700 1 -- 172.21.15.42:0/1817501587 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7579+0+0 (secure 0 0 0) 0x7f4ff404adc0 con 0x7f500408ee90 2019-12-05T05:30:47.441 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:47.423+0000 7f500259c700 1 --2- 172.21.15.42:0/1817501587 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4fe803b3b0 0x7f4fe803d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:30:47.441 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:47.423+0000 7f500259c700 1 --2- 172.21.15.42:0/1817501587 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4fe803b3b0 0x7f4fe803d8c0 crc :-1 s=READY pgs=431 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:30:47.441 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:47.427+0000 7f5008f2c700 1 -- 172.21.15.42:0/1817501587 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f500405d110 con 0x7f500408ee90 2019-12-05T05:30:47.443 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:47.435+0000 7f5002d9d700 1 --2- 172.21.15.42:0/1817501587 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f500408ee90 0x7f50040653c0 secure :-1 s=THROTTLE_DONE pgs=201 cs=0 l=1 rx=0x7f4ff4002f30 tx=0x7f4ff40109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:47.443 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:47.435+0000 7f4ffb7fe700 1 -- 172.21.15.42:0/1817501587 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f4ff4031e00 con 0x7f500408ee90 2019-12-05T05:30:47.614 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:47.607+0000 7f5008f2c700 1 -- 172.21.15.42:0/1817501587 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f500405b8c0 con 0x7f500408ee90 2019-12-05T05:30:47.615 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:47.607+0000 7f5002d9d700 1 --2- 172.21.15.42:0/1817501587 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f500408ee90 0x7f50040653c0 secure :-1 s=THROTTLE_DONE pgs=201 cs=0 l=1 rx=0x7f4ff4002f30 tx=0x7f4ff40109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:47.615 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:47.607+0000 7f4ffb7fe700 1 -- 172.21.15.42:0/1817501587 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 123 v123) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f4ff4032060 con 0x7f500408ee90 2019-12-05T05:30:47.617 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 123 2019-12-05T05:30:47.617 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:30:47.618 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":123,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6778,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/3028851337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3028851337},{"type":"v1","addr":"172.21.15.42:6835","nonce":3028851337}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":123}],"filesystems":[{"mdsmap":{"epoch":122,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:29:24.937492+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5426},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5426":{"gid":5426,"name":"b","rank":0,"incarnation":119,"state":"up:active","state_seq":18,"addr":"172.21.15.40:6833/3168718372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3168718372},{"type":"v1","addr":"172.21.15.40:6833","nonce":3168718372}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:30:47.621 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:47.611+0000 7f5008f2c700 1 -- 172.21.15.42:0/1817501587 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4fe803b3b0 msgr2=0x7f4fe803d8c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:30:47.621 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:47.611+0000 7f5008f2c700 1 --2- 172.21.15.42:0/1817501587 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4fe803b3b0 0x7f4fe803d8c0 crc :-1 s=READY pgs=431 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:30:47.621 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:47.611+0000 7f5008f2c700 1 -- 172.21.15.42:0/1817501587 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f500408ee90 msgr2=0x7f50040653c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:30:47.622 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:47.611+0000 7f5008f2c700 1 --2- 172.21.15.42:0/1817501587 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f500408ee90 0x7f50040653c0 secure :-1 s=READY pgs=201 cs=0 l=1 rx=0x7f4ff4002f30 tx=0x7f4ff40109e0).stop 2019-12-05T05:30:47.622 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:47.611+0000 7f5008f2c700 1 -- 172.21.15.42:0/1817501587 shutdown_connections 2019-12-05T05:30:47.622 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:47.611+0000 7f5008f2c700 1 --2- 172.21.15.42:0/1817501587 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4fe803b3b0 0x7f4fe803d8c0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:30:47.622 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:47.611+0000 7f5008f2c700 1 --2- 172.21.15.42:0/1817501587 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f500408ee90 0x7f50040653c0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:30:47.623 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:47.611+0000 7f5008f2c700 1 -- 172.21.15.42:0/1817501587 shutdown_connections 2019-12-05T05:30:47.623 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:47.615+0000 7f5008f2c700 1 -- 172.21.15.42:0/1817501587 wait complete. 2019-12-05T05:30:47.623 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:47.615+0000 7f5008f2c700 1 -- 172.21.15.42:0/1817501587 >> 172.21.15.42:0/1817501587 conn(0x7f50040aabe0 msgr2=0x7f500411da30 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:30:47.630 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T05:30:49.632 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:30:49.734 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:49.718+0000 7fe6edf00700 1 Processor -- start 2019-12-05T05:30:49.734 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:49.718+0000 7fe6edf00700 1 -- start start 2019-12-05T05:30:49.734 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:49.718+0000 7fe6edf00700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe6e809fae0 0x7fe6e814fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:30:49.734 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:49.718+0000 7fe6edf00700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6e80a0710 con 0x7fe6e80a7740 2019-12-05T05:30:49.735 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:49.718+0000 7fe6edf00700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6e807ef20 con 0x7fe6e808df80 2019-12-05T05:30:49.735 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:49.718+0000 7fe6edf00700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6e80a0b00 con 0x7fe6e809fae0 2019-12-05T05:30:49.735 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:49.718+0000 7fe6e77fe700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fe6e808df80 0x7fe6e809f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:35366/0 (socket says 172.21.15.42:35366) 2019-12-05T05:30:49.735 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:49.718+0000 7fe6e77fe700 1 -- 172.21.15.42:0/3058300688 learned_addr learned my addr 172.21.15.42:0/3058300688 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:30:49.735 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:49.718+0000 7fe6ecc9e700 1 --2- 172.21.15.42:0/3058300688 >> v2:172.21.15.40:3300/0 conn(0x7fe6e809fae0 0x7fe6e814fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:30:49.735 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:49.722+0000 7fe6e6ffd700 1 -- 172.21.15.42:0/3058300688 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2229374095 0 0) 0x7fe6e807ef20 con 0x7fe6e808df80 2019-12-05T05:30:49.735 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:49.722+0000 7fe6e6ffd700 1 -- 172.21.15.42:0/3058300688 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6d00030b0 con 0x7fe6e808df80 2019-12-05T05:30:49.736 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:49.722+0000 7fe6e6ffd700 1 -- 172.21.15.42:0/3058300688 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2726367968 0 0) 0x7fe6d00030b0 con 0x7fe6e808df80 2019-12-05T05:30:49.736 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:49.722+0000 7fe6e6ffd700 1 -- 172.21.15.42:0/3058300688 >> v2:172.21.15.40:3300/0 conn(0x7fe6e809fae0 msgr2=0x7fe6e814fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:30:49.736 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:49.722+0000 7fe6e6ffd700 1 --2- 172.21.15.42:0/3058300688 >> v2:172.21.15.40:3300/0 conn(0x7fe6e809fae0 0x7fe6e814fff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:30:49.736 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:49.722+0000 7fe6e6ffd700 1 -- 172.21.15.42:0/3058300688 >> v1:172.21.15.40:6789/0 conn(0x7fe6e80a7740 legacy=0x7fe6e807e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:30:49.736 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:49.722+0000 7fe6e6ffd700 1 -- 172.21.15.42:0/3058300688 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6e80a9ba0 con 0x7fe6e808df80 2019-12-05T05:30:49.736 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:49.722+0000 7fe6e6ffd700 1 -- 172.21.15.42:0/3058300688 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7fe6dc001750 con 0x7fe6e808df80 2019-12-05T05:30:49.736 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:49.722+0000 7fe6e6ffd700 1 -- 172.21.15.42:0/3058300688 >> v1:172.21.15.42:6789/0 conn(0x7fe6e808df80 legacy=0x7fe6e809f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:30:49.737 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:49.722+0000 7fe6e6ffd700 1 --2- 172.21.15.42:0/3058300688 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe6d00036e0 0x7fe6d0003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:30:49.737 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:49.722+0000 7fe6e6ffd700 1 --2- 172.21.15.42:0/3058300688 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe6d0004420 0x7fe6d0004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:30:49.737 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:49.722+0000 7fe6e6ffd700 1 --2- 172.21.15.42:0/3058300688 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe6d0004da0 0x7fe6d000b670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:30:49.737 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:49.722+0000 7fe6e6ffd700 1 -- 172.21.15.42:0/3058300688 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6e80a0b00 con 0x7fe6d0004da0 2019-12-05T05:30:49.737 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:49.722+0000 7fe6e6ffd700 1 -- 172.21.15.42:0/3058300688 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6d004fc60 con 0x7fe6d00036e0 2019-12-05T05:30:49.737 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:49.722+0000 7fe6e6ffd700 1 -- 172.21.15.42:0/3058300688 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6d004fea0 con 0x7fe6d0004420 2019-12-05T05:30:49.738 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:49.722+0000 7fe6e7fff700 1 --2- 172.21.15.42:0/3058300688 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe6d00036e0 0x7fe6d0003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:30:49.738 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:49.722+0000 7fe6ecc9e700 1 --2- 172.21.15.42:0/3058300688 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe6d0004da0 0x7fe6d000b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:30:49.738 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:49.722+0000 7fe6e77fe700 1 --2- 172.21.15.42:0/3058300688 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe6d0004420 0x7fe6d0004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:30:49.738 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:49.722+0000 7fe6e7fff700 1 -- 172.21.15.42:0/3058300688 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe6d0004420 msgr2=0x7fe6d0004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:30:49.738 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:49.722+0000 7fe6e7fff700 1 --2- 172.21.15.42:0/3058300688 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe6d0004420 0x7fe6d0004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:30:49.738 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:49.722+0000 7fe6e7fff700 1 -- 172.21.15.42:0/3058300688 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe6d0004da0 msgr2=0x7fe6d000b670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:30:49.738 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:49.722+0000 7fe6e7fff700 1 --2- 172.21.15.42:0/3058300688 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe6d0004da0 0x7fe6d000b670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:30:49.739 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:49.722+0000 7fe6ecc9e700 1 -- 172.21.15.42:0/3058300688 reap_dead start 2019-12-05T05:30:49.739 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:49.722+0000 7fe6e7fff700 1 -- 172.21.15.42:0/3058300688 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6d00501a0 con 0x7fe6d00036e0 2019-12-05T05:30:49.739 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:49.722+0000 7fe6e7fff700 1 --2- 172.21.15.42:0/3058300688 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe6d00036e0 0x7fe6d0003b50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fe6d8001000 tx=0x7fe6d800fcd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:49.739 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:49.722+0000 7fe6e7fff700 1 --2- 172.21.15.42:0/3058300688 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe6d00036e0 0x7fe6d0003b50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fe6d8001000 tx=0x7fe6d800fcd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:49.740 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:49.722+0000 7fe6e7fff700 1 --2- 172.21.15.42:0/3058300688 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe6d00036e0 0x7fe6d0003b50 secure :-1 s=READY pgs=1009 cs=0 l=1 rx=0x7fe6d8001000 tx=0x7fe6d800fcd0).ready entity=mon.1 client_cookie=57971daba5def82a server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:30:49.740 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:49.722+0000 7fe6e7fff700 1 --2- 172.21.15.42:0/3058300688 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe6d00036e0 0x7fe6d0003b50 secure :-1 s=THROTTLE_DONE pgs=1009 cs=0 l=1 rx=0x7fe6d8001000 tx=0x7fe6d800fcd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:49.740 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:49.722+0000 7fe6e6ffd700 1 -- 172.21.15.42:0/3058300688 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe6d8003540 con 0x7fe6d00036e0 2019-12-05T05:30:49.740 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:49.722+0000 7fe6e7fff700 1 --2- 172.21.15.42:0/3058300688 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe6d00036e0 0x7fe6d0003b50 secure :-1 s=THROTTLE_DONE pgs=1009 cs=0 l=1 rx=0x7fe6d8001000 tx=0x7fe6d800fcd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:49.740 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:49.722+0000 7fe6e6ffd700 1 -- 172.21.15.42:0/3058300688 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6d8003160 con 0x7fe6d00036e0 2019-12-05T05:30:49.740 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:49.722+0000 7fe6e7fff700 1 --2- 172.21.15.42:0/3058300688 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe6d00036e0 0x7fe6d0003b50 secure :-1 s=THROTTLE_DONE pgs=1009 cs=0 l=1 rx=0x7fe6d8001000 tx=0x7fe6d800fcd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:49.741 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:49.722+0000 7fe6edf00700 1 -- 172.21.15.42:0/3058300688 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe6d00036e0 msgr2=0x7fe6d0003b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:30:49.741 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:49.722+0000 7fe6e6ffd700 1 -- 172.21.15.42:0/3058300688 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe6d8003b20 con 0x7fe6d00036e0 2019-12-05T05:30:49.741 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:49.722+0000 7fe6edf00700 1 --2- 172.21.15.42:0/3058300688 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe6d00036e0 0x7fe6d0003b50 secure :-1 s=READY pgs=1009 cs=0 l=1 rx=0x7fe6d8001000 tx=0x7fe6d800fcd0).stop 2019-12-05T05:30:49.741 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:49.722+0000 7fe6edf00700 1 -- 172.21.15.42:0/3058300688 shutdown_connections 2019-12-05T05:30:49.741 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:49.722+0000 7fe6edf00700 1 --2- 172.21.15.42:0/3058300688 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe6d00036e0 0x7fe6d0003b50 unknown :-1 s=CLOSED pgs=1009 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:30:49.741 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:49.722+0000 7fe6edf00700 1 -- 172.21.15.42:0/3058300688 shutdown_connections 2019-12-05T05:30:49.741 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:49.722+0000 7fe6edf00700 1 -- 172.21.15.42:0/3058300688 wait complete. 2019-12-05T05:30:49.742 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:49.722+0000 7fe6edf00700 1 -- 172.21.15.42:0/3058300688 >> 172.21.15.42:0/3058300688 conn(0x7fe6e80aabe0 msgr2=0x7fe6e80a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:30:49.742 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:49.722+0000 7fe6edf00700 1 Processor -- start 2019-12-05T05:30:49.742 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:49.722+0000 7fe6edf00700 1 -- start start 2019-12-05T05:30:49.742 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:49.726+0000 7fe6edf00700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe6e8143070 0x7fe6e80a22d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:30:49.742 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:49.726+0000 7fe6edf00700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe6e8144ab0 0x7fe6e8077e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:30:49.742 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:49.726+0000 7fe6edf00700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6e81434a0 con 0x7fe6e80a4100 2019-12-05T05:30:49.743 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:49.726+0000 7fe6edf00700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6e80a4840 con 0x7fe6e8144ab0 2019-12-05T05:30:49.743 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:49.726+0000 7fe6edf00700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6e8144200 con 0x7fe6e8143070 2019-12-05T05:30:49.743 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:49.726+0000 7fe6ecc9e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe6e8144ab0 0x7fe6e8077e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:30:49.743 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:49.726+0000 7fe6ecc9e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe6e8144ab0 0x7fe6e8077e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:35342/0 (socket says 172.21.15.42:35342) 2019-12-05T05:30:49.743 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:49.726+0000 7fe6ecc9e700 1 -- 172.21.15.42:0/888786035 learned_addr learned my addr 172.21.15.42:0/888786035 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:30:49.743 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:49.726+0000 7fe6e77fe700 1 --2- 172.21.15.42:0/888786035 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe6e8143070 0x7fe6e80a22d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:30:49.744 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:49.726+0000 7fe6ecc9e700 1 -- 172.21.15.42:0/888786035 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe6e8143070 msgr2=0x7fe6e80a22d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:30:49.744 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:49.726+0000 7fe6ecc9e700 1 --2- 172.21.15.42:0/888786035 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe6e8143070 0x7fe6e80a22d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:30:49.744 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:49.726+0000 7fe6ecc9e700 1 -- 172.21.15.42:0/888786035 >> v1:172.21.15.40:6789/0 conn(0x7fe6e80a4100 legacy=0x7fe6e8077610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:30:49.744 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:49.726+0000 7fe6ecc9e700 1 -- 172.21.15.42:0/888786035 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6e8078360 con 0x7fe6e8144ab0 2019-12-05T05:30:49.744 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:49.726+0000 7fe6ecc9e700 1 --2- 172.21.15.42:0/888786035 >> v2:172.21.15.42:3300/0 conn(0x7fe6e8144ab0 0x7fe6e8077e10 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fe6e81443c0 tx=0x7fe6e000c630).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:49.744 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:49.726+0000 7fe6ecc9e700 1 --2- 172.21.15.42:0/888786035 >> v2:172.21.15.42:3300/0 conn(0x7fe6e8144ab0 0x7fe6e8077e10 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fe6e81443c0 tx=0x7fe6e000c630).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:49.745 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:49.726+0000 7fe6ecc9e700 1 --2- 172.21.15.42:0/888786035 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe6e8144ab0 0x7fe6e8077e10 secure :-1 s=READY pgs=1010 cs=0 l=1 rx=0x7fe6e81443c0 tx=0x7fe6e000c630).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:30:49.745 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:49.726+0000 7fe6ecc9e700 1 --2- 172.21.15.42:0/888786035 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe6e8144ab0 0x7fe6e8077e10 secure :-1 s=THROTTLE_DONE pgs=1010 cs=0 l=1 rx=0x7fe6e81443c0 tx=0x7fe6e000c630).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:49.745 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:49.726+0000 7fe6ecc9e700 1 --2- 172.21.15.42:0/888786035 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe6e8144ab0 0x7fe6e8077e10 secure :-1 s=THROTTLE_DONE pgs=1010 cs=0 l=1 rx=0x7fe6e81443c0 tx=0x7fe6e000c630).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:49.745 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:49.726+0000 7fe6e4ff9700 1 -- 172.21.15.42:0/888786035 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe6e0005ca0 con 0x7fe6e8144ab0 2019-12-05T05:30:49.745 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:49.726+0000 7fe6ecc9e700 1 --2- 172.21.15.42:0/888786035 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe6e8144ab0 0x7fe6e8077e10 secure :-1 s=THROTTLE_DONE pgs=1010 cs=0 l=1 rx=0x7fe6e81443c0 tx=0x7fe6e000c630).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:49.745 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:49.726+0000 7fe6e4ff9700 1 -- 172.21.15.42:0/888786035 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6e00058c0 con 0x7fe6e8144ab0 2019-12-05T05:30:49.745 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:49.726+0000 7fe6edf00700 1 -- 172.21.15.42:0/888786035 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6e8144ee0 con 0x7fe6e8144ab0 2019-12-05T05:30:49.746 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:49.726+0000 7fe6e4ff9700 1 -- 172.21.15.42:0/888786035 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe6e0014af0 con 0x7fe6e8144ab0 2019-12-05T05:30:49.746 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:49.726+0000 7fe6edf00700 1 -- 172.21.15.42:0/888786035 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6e80a1a80 con 0x7fe6e8144ab0 2019-12-05T05:30:49.746 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:49.726+0000 7fe6ecc9e700 1 --2- 172.21.15.42:0/888786035 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe6e8144ab0 0x7fe6e8077e10 secure :-1 s=THROTTLE_DONE pgs=1010 cs=0 l=1 rx=0x7fe6e81443c0 tx=0x7fe6e000c630).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:49.747 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:49.726+0000 7fe6e4ff9700 1 -- 172.21.15.42:0/888786035 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fe6e0014cf0 con 0x7fe6e8144ab0 2019-12-05T05:30:49.747 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:49.726+0000 7fe6ecc9e700 1 --2- 172.21.15.42:0/888786035 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe6e8144ab0 0x7fe6e8077e10 secure :-1 s=THROTTLE_DONE pgs=1010 cs=0 l=1 rx=0x7fe6e81443c0 tx=0x7fe6e000c630).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:49.748 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:49.726+0000 7fe6e4ff9700 1 --2- 172.21.15.42:0/888786035 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe6c803acf0 0x7fe6c803d200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:30:49.748 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:49.726+0000 7fe6e4ff9700 1 -- 172.21.15.42:0/888786035 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 7665+0+0 (secure 0 0 0) 0x7fe6e004d230 con 0x7fe6e8144ab0 2019-12-05T05:30:49.748 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:49.726+0000 7fe6e7fff700 1 --2- 172.21.15.42:0/888786035 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe6c803acf0 0x7fe6c803d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:30:49.748 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:49.726+0000 7fe6e7fff700 1 --2- 172.21.15.42:0/888786035 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe6c803acf0 0x7fe6c803d200 crc :-1 s=READY pgs=432 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:30:49.748 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:49.734+0000 7fe6edf00700 1 -- 172.21.15.42:0/888786035 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6e805d110 con 0x7fe6e8144ab0 2019-12-05T05:30:49.748 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:49.738+0000 7fe6ecc9e700 1 --2- 172.21.15.42:0/888786035 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe6e8144ab0 0x7fe6e8077e10 secure :-1 s=THROTTLE_DONE pgs=1010 cs=0 l=1 rx=0x7fe6e81443c0 tx=0x7fe6e000c630).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:49.748 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:49.738+0000 7fe6e4ff9700 1 -- 172.21.15.42:0/888786035 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fe6e00342b0 con 0x7fe6e8144ab0 2019-12-05T05:30:49.922 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:49.902+0000 7fe6edf00700 1 -- 172.21.15.42:0/888786035 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe6e805b8c0 con 0x7fe6e8144ab0 2019-12-05T05:30:49.927 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:49.902+0000 7fe6ecc9e700 1 --2- 172.21.15.42:0/888786035 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe6e8144ab0 0x7fe6e8077e10 secure :-1 s=THROTTLE_DONE pgs=1010 cs=0 l=1 rx=0x7fe6e81443c0 tx=0x7fe6e000c630).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:49.927 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:49.902+0000 7fe6e4ff9700 1 -- 172.21.15.42:0/888786035 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 124 v124) v1 ==== 95+0+1774 (secure 0 0 0) 0x7fe6e0034510 con 0x7fe6e8144ab0 2019-12-05T05:30:49.927 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 124 2019-12-05T05:30:49.927 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:49.906+0000 7fe6edf00700 1 -- 172.21.15.42:0/888786035 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe6c803acf0 msgr2=0x7fe6c803d200 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:30:49.928 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:49.906+0000 7fe6edf00700 1 --2- 172.21.15.42:0/888786035 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe6c803acf0 0x7fe6c803d200 crc :-1 s=READY pgs=432 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:30:49.928 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:49.906+0000 7fe6edf00700 1 -- 172.21.15.42:0/888786035 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe6e8144ab0 msgr2=0x7fe6e8077e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:30:49.934 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:49.906+0000 7fe6edf00700 1 --2- 172.21.15.42:0/888786035 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe6e8144ab0 0x7fe6e8077e10 secure :-1 s=READY pgs=1010 cs=0 l=1 rx=0x7fe6e81443c0 tx=0x7fe6e000c630).stop 2019-12-05T05:30:49.934 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:49.906+0000 7fe6edf00700 1 -- 172.21.15.42:0/888786035 shutdown_connections 2019-12-05T05:30:49.934 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:49.906+0000 7fe6edf00700 1 --2- 172.21.15.42:0/888786035 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe6c803acf0 0x7fe6c803d200 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:30:49.934 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:49.906+0000 7fe6edf00700 1 --2- 172.21.15.42:0/888786035 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe6e8144ab0 0x7fe6e8077e10 unknown :-1 s=CLOSED pgs=1010 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:30:49.934 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:49.906+0000 7fe6edf00700 1 --2- 172.21.15.42:0/888786035 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe6e8143070 0x7fe6e80a22d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:30:49.935 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:49.906+0000 7fe6edf00700 1 -- 172.21.15.42:0/888786035 shutdown_connections 2019-12-05T05:30:49.935 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:49.906+0000 7fe6edf00700 1 -- 172.21.15.42:0/888786035 wait complete. 2019-12-05T05:30:49.935 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:49.906+0000 7fe6edf00700 1 -- 172.21.15.42:0/888786035 >> 172.21.15.42:0/888786035 conn(0x7fe6e80a8940 msgr2=0x7fe6e808ec50 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:30:49.935 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:30:49.935 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":124,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":124,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:30:48.694163+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6778},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6778":{"gid":6778,"name":"a","rank":0,"incarnation":124,"state":"up:replay","state_seq":2,"addr":"172.21.15.42:6835/3028851337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3028851337},{"type":"v1","addr":"172.21.15.42:6835","nonce":3028851337}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:30:49.944 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b down, removed from mdsmap 2019-12-05T05:30:49.944 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2019-12-05T05:30:49.945 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:30:50.051 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:50.034+0000 7f5b0bae4700 1 Processor -- start 2019-12-05T05:30:50.051 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:50.034+0000 7f5b0bae4700 1 -- start start 2019-12-05T05:30:50.052 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:50.038+0000 7f5b0bae4700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5b0408df70 0x7f5b0409f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:30:50.052 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:50.038+0000 7f5b0bae4700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b0409fc50 con 0x7f5b040a7730 2019-12-05T05:30:50.052 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:50.038+0000 7f5b0bae4700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b040a0700 con 0x7f5b0409f820 2019-12-05T05:30:50.052 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:50.038+0000 7f5b0bae4700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b040a0af0 con 0x7f5b0408df70 2019-12-05T05:30:50.053 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:50.038+0000 7f5b0a882700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f5b0409f820 0x7f5b0414efd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:35388/0 (socket says 172.21.15.42:35388) 2019-12-05T05:30:50.053 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:50.038+0000 7f5b0a882700 1 -- 172.21.15.42:0/2872748763 learned_addr learned my addr 172.21.15.42:0/2872748763 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:30:50.053 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:50.038+0000 7f5b09880700 1 --2- 172.21.15.42:0/2872748763 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5b0408df70 0x7f5b0409f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:30:50.053 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:50.038+0000 7f5b0907f700 1 -- 172.21.15.42:0/2872748763 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4224983652 0 0) 0x7f5b040a0700 con 0x7f5b0409f820 2019-12-05T05:30:50.053 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:50.038+0000 7f5b0907f700 1 -- 172.21.15.42:0/2872748763 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5af00030b0 con 0x7f5b0409f820 2019-12-05T05:30:50.053 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:50.038+0000 7f5b0907f700 1 -- 172.21.15.42:0/2872748763 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3682358813 0 0) 0x7f5af00030b0 con 0x7f5b0409f820 2019-12-05T05:30:50.053 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:50.038+0000 7f5b0907f700 1 -- 172.21.15.42:0/2872748763 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5b0408df70 msgr2=0x7f5b0409f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:30:50.054 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:50.038+0000 7f5b0907f700 1 --2- 172.21.15.42:0/2872748763 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5b0408df70 0x7f5b0409f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:30:50.054 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:50.038+0000 7f5b0907f700 1 -- 172.21.15.42:0/2872748763 >> v1:172.21.15.40:6789/0 conn(0x7f5b040a7730 legacy=0x7f5b0407e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:30:50.054 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:50.038+0000 7f5b0907f700 1 -- 172.21.15.42:0/2872748763 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b0407ef10 con 0x7f5b0409f820 2019-12-05T05:30:50.054 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:50.038+0000 7f5b0907f700 1 -- 172.21.15.42:0/2872748763 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7f5afc001750 con 0x7f5b0409f820 2019-12-05T05:30:50.054 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:50.038+0000 7f5b0907f700 1 -- 172.21.15.42:0/2872748763 >> v1:172.21.15.42:6789/0 conn(0x7f5b0409f820 legacy=0x7f5b0414efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:30:50.054 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:50.038+0000 7f5b0907f700 1 --2- 172.21.15.42:0/2872748763 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5af0003710 0x7f5af0003b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:30:50.054 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:50.038+0000 7f5b0907f700 1 --2- 172.21.15.42:0/2872748763 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5af0004420 0x7f5af0004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:30:50.055 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:50.038+0000 7f5b0907f700 1 --2- 172.21.15.42:0/2872748763 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5af0004da0 0x7f5af000b670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:30:50.055 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:50.038+0000 7f5b0907f700 1 -- 172.21.15.42:0/2872748763 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b040a0af0 con 0x7f5af0004da0 2019-12-05T05:30:50.055 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:50.038+0000 7f5b09880700 1 --2- 172.21.15.42:0/2872748763 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5af0004420 0x7f5af0004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:30:50.055 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:50.038+0000 7f5b0907f700 1 -- 172.21.15.42:0/2872748763 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5af004fbd0 con 0x7f5af0004420 2019-12-05T05:30:50.055 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:50.038+0000 7f5b0907f700 1 -- 172.21.15.42:0/2872748763 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5af004fd90 con 0x7f5af0003710 2019-12-05T05:30:50.055 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:50.038+0000 7f5b0a081700 1 --2- 172.21.15.42:0/2872748763 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5af0003710 0x7f5af0003b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:30:50.056 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:50.038+0000 7f5b0a882700 1 --2- 172.21.15.42:0/2872748763 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5af0004da0 0x7f5af000b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:30:50.056 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:50.038+0000 7f5b09880700 1 -- 172.21.15.42:0/2872748763 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5af0003710 msgr2=0x7f5af0003b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:30:50.056 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:50.038+0000 7f5b09880700 1 --2- 172.21.15.42:0/2872748763 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5af0003710 0x7f5af0003b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:30:50.056 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:50.038+0000 7f5b09880700 1 -- 172.21.15.42:0/2872748763 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5af0004da0 msgr2=0x7f5af000b670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:30:50.056 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:50.038+0000 7f5b09880700 1 --2- 172.21.15.42:0/2872748763 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5af0004da0 0x7f5af000b670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:30:50.056 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:50.038+0000 7f5b09880700 1 -- 172.21.15.42:0/2872748763 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5af004ffb0 con 0x7f5af0004420 2019-12-05T05:30:50.057 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:50.038+0000 7f5b09880700 1 --2- 172.21.15.42:0/2872748763 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5af0004420 0x7f5af0004850 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f5af4002f30 tx=0x7f5af400ffe0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:50.057 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:50.038+0000 7f5b09880700 1 --2- 172.21.15.42:0/2872748763 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5af0004420 0x7f5af0004850 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f5af4002f30 tx=0x7f5af400ffe0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:50.057 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:50.038+0000 7f5b09880700 1 --2- 172.21.15.42:0/2872748763 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5af0004420 0x7f5af0004850 secure :-1 s=READY pgs=1012 cs=0 l=1 rx=0x7f5af4002f30 tx=0x7f5af400ffe0).ready entity=mon.1 client_cookie=f99858421b375ce2 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:30:50.057 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:50.038+0000 7f5b09880700 1 --2- 172.21.15.42:0/2872748763 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5af0004420 0x7f5af0004850 secure :-1 s=THROTTLE_DONE pgs=1012 cs=0 l=1 rx=0x7f5af4002f30 tx=0x7f5af400ffe0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:50.057 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:50.038+0000 7f5b0907f700 1 -- 172.21.15.42:0/2872748763 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5af4010d10 con 0x7f5af0004420 2019-12-05T05:30:50.057 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:50.038+0000 7f5b09880700 1 --2- 172.21.15.42:0/2872748763 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5af0004420 0x7f5af0004850 secure :-1 s=THROTTLE_DONE pgs=1012 cs=0 l=1 rx=0x7f5af4002f30 tx=0x7f5af400ffe0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:50.058 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:50.038+0000 7f5b09880700 1 --2- 172.21.15.42:0/2872748763 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5af0004420 0x7f5af0004850 secure :-1 s=THROTTLE_DONE pgs=1012 cs=0 l=1 rx=0x7f5af4002f30 tx=0x7f5af400ffe0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:50.058 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:50.038+0000 7f5b0907f700 1 -- 172.21.15.42:0/2872748763 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5af4010930 con 0x7f5af0004420 2019-12-05T05:30:50.058 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:50.038+0000 7f5b0907f700 1 -- 172.21.15.42:0/2872748763 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5af4008bb0 con 0x7f5af0004420 2019-12-05T05:30:50.058 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:50.038+0000 7f5b0bae4700 1 -- 172.21.15.42:0/2872748763 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5af0004420 msgr2=0x7f5af0004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:30:50.058 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:50.038+0000 7f5b0bae4700 1 --2- 172.21.15.42:0/2872748763 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5af0004420 0x7f5af0004850 secure :-1 s=READY pgs=1012 cs=0 l=1 rx=0x7f5af4002f30 tx=0x7f5af400ffe0).stop 2019-12-05T05:30:50.059 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:50.038+0000 7f5b0a882700 1 -- 172.21.15.42:0/2872748763 reap_dead start 2019-12-05T05:30:50.059 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:50.038+0000 7f5b0bae4700 1 -- 172.21.15.42:0/2872748763 shutdown_connections 2019-12-05T05:30:50.059 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:50.038+0000 7f5b0bae4700 1 -- 172.21.15.42:0/2872748763 shutdown_connections 2019-12-05T05:30:50.059 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:50.038+0000 7f5b0bae4700 1 -- 172.21.15.42:0/2872748763 wait complete. 2019-12-05T05:30:50.060 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:50.038+0000 7f5b0bae4700 1 -- 172.21.15.42:0/2872748763 >> 172.21.15.42:0/2872748763 conn(0x7f5b040aabd0 msgr2=0x7f5b040a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:30:50.060 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:50.042+0000 7f5b0bae4700 1 Processor -- start 2019-12-05T05:30:50.060 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:50.042+0000 7f5b0bae4700 1 -- start start 2019-12-05T05:30:50.060 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:50.042+0000 7f5b0bae4700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5b040a1800 0x7f5b040a2840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:30:50.060 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:50.042+0000 7f5b0bae4700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5b040a2d90 0x7f5b041413e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:30:50.060 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:50.042+0000 7f5b0bae4700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b0412f8e0 con 0x7f5b04141930 2019-12-05T05:30:50.060 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:50.042+0000 7f5b0bae4700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b0407dea0 con 0x7f5b040a1800 2019-12-05T05:30:50.061 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:50.042+0000 7f5b0bae4700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b0407e140 con 0x7f5b040a2d90 2019-12-05T05:30:50.061 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:50.042+0000 7f5b0a081700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5b040a1800 0x7f5b040a2840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:30:50.061 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:50.042+0000 7f5b0a081700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5b040a1800 0x7f5b040a2840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.42:51406/0 (socket says 172.21.15.42:51406) 2019-12-05T05:30:50.061 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:50.042+0000 7f5b0a081700 1 -- 172.21.15.42:0/3403190971 learned_addr learned my addr 172.21.15.42:0/3403190971 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:30:50.061 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:50.042+0000 7f5b09880700 1 --2- 172.21.15.42:0/3403190971 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5b040a2d90 0x7f5b041413e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:30:50.061 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:50.042+0000 7f5b0a081700 1 -- 172.21.15.42:0/3403190971 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5b040a2d90 msgr2=0x7f5b041413e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:30:50.061 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:50.042+0000 7f5b0a081700 1 --2- 172.21.15.42:0/3403190971 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5b040a2d90 0x7f5b041413e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:30:50.062 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:50.042+0000 7f5b0a081700 1 -- 172.21.15.42:0/3403190971 >> v1:172.21.15.40:6789/0 conn(0x7f5b04141930 legacy=0x7f5b0413af70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:30:50.062 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:50.042+0000 7f5b0a081700 1 -- 172.21.15.42:0/3403190971 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b040a1c30 con 0x7f5b040a1800 2019-12-05T05:30:50.062 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:50.042+0000 7f5b0a081700 1 --2- 172.21.15.42:0/3403190971 >> v2:172.21.15.40:3300/0 conn(0x7f5b040a1800 0x7f5b040a2840 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f5b0407e060 tx=0x7f5b00010da0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:50.062 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:50.042+0000 7f5b0a081700 1 --2- 172.21.15.42:0/3403190971 >> v2:172.21.15.40:3300/0 conn(0x7f5b040a1800 0x7f5b040a2840 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f5b0407e060 tx=0x7f5b00010da0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:50.062 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:50.042+0000 7f5b0a081700 1 --2- 172.21.15.42:0/3403190971 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b040a1800 0x7f5b040a2840 secure :-1 s=READY pgs=786 cs=0 l=1 rx=0x7f5b0407e060 tx=0x7f5b00010da0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:30:50.062 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:50.042+0000 7f5b0a081700 1 --2- 172.21.15.42:0/3403190971 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b040a1800 0x7f5b040a2840 secure :-1 s=THROTTLE_DONE pgs=786 cs=0 l=1 rx=0x7f5b0407e060 tx=0x7f5b00010da0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:50.062 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:50.042+0000 7f5afaffd700 1 -- 172.21.15.42:0/3403190971 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5b0000ca40 con 0x7f5b040a1800 2019-12-05T05:30:50.063 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:50.042+0000 7f5b0a081700 1 --2- 172.21.15.42:0/3403190971 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b040a1800 0x7f5b040a2840 secure :-1 s=THROTTLE_DONE pgs=786 cs=0 l=1 rx=0x7f5b0407e060 tx=0x7f5b00010da0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:50.063 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:50.042+0000 7f5b0bae4700 1 -- 172.21.15.42:0/3403190971 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b040a31c0 con 0x7f5b040a1800 2019-12-05T05:30:50.063 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:50.042+0000 7f5b0a081700 1 --2- 172.21.15.42:0/3403190971 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b040a1800 0x7f5b040a2840 secure :-1 s=THROTTLE_DONE pgs=786 cs=0 l=1 rx=0x7f5b0407e060 tx=0x7f5b00010da0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:50.063 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:50.042+0000 7f5afaffd700 1 -- 172.21.15.42:0/3403190971 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b000101c0 con 0x7f5b040a1800 2019-12-05T05:30:50.063 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:50.042+0000 7f5afaffd700 1 -- 172.21.15.42:0/3403190971 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5b000126d0 con 0x7f5b040a1800 2019-12-05T05:30:50.063 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:50.042+0000 7f5b0bae4700 1 -- 172.21.15.42:0/3403190971 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b0405ef10 con 0x7f5b040a1800 2019-12-05T05:30:50.063 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:50.042+0000 7f5b0a081700 1 --2- 172.21.15.42:0/3403190971 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b040a1800 0x7f5b040a2840 secure :-1 s=THROTTLE_DONE pgs=786 cs=0 l=1 rx=0x7f5b0407e060 tx=0x7f5b00010da0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:50.065 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:50.046+0000 7f5afaffd700 1 -- 172.21.15.42:0/3403190971 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f5b000129b0 con 0x7f5b040a1800 2019-12-05T05:30:50.065 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:50.046+0000 7f5b0a081700 1 --2- 172.21.15.42:0/3403190971 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b040a1800 0x7f5b040a2840 secure :-1 s=THROTTLE_DONE pgs=786 cs=0 l=1 rx=0x7f5b0407e060 tx=0x7f5b00010da0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:50.065 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:50.046+0000 7f5afaffd700 1 --2- 172.21.15.42:0/3403190971 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5aec03acf0 0x7f5aec03d200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:30:50.065 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:50.046+0000 7f5afaffd700 1 -- 172.21.15.42:0/3403190971 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 7665+0+0 (secure 0 0 0) 0x7f5b0004b290 con 0x7f5b040a1800 2019-12-05T05:30:50.065 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:50.046+0000 7f5b09880700 1 --2- 172.21.15.42:0/3403190971 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5aec03acf0 0x7f5aec03d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:30:50.065 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:50.046+0000 7f5b09880700 1 --2- 172.21.15.42:0/3403190971 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5aec03acf0 0x7f5aec03d200 crc :-1 s=READY pgs=433 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:30:50.065 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:50.050+0000 7f5b0bae4700 1 -- 172.21.15.42:0/3403190971 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b0405f120 con 0x7f5b040a1800 2019-12-05T05:30:50.069 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:50.062+0000 7f5b0a081700 1 --2- 172.21.15.42:0/3403190971 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b040a1800 0x7f5b040a2840 secure :-1 s=THROTTLE_DONE pgs=786 cs=0 l=1 rx=0x7f5b0407e060 tx=0x7f5b00010da0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:50.069 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:50.062+0000 7f5afaffd700 1 -- 172.21.15.42:0/3403190971 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f5b000322f0 con 0x7f5b040a1800 2019-12-05T05:30:50.227 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:50.218+0000 7f5b0bae4700 1 -- 172.21.15.42:0/3403190971 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5b0405d100 con 0x7f5b040a1800 2019-12-05T05:30:50.228 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:50.222+0000 7f5b0a081700 1 --2- 172.21.15.42:0/3403190971 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b040a1800 0x7f5b040a2840 secure :-1 s=THROTTLE_DONE pgs=786 cs=0 l=1 rx=0x7f5b0407e060 tx=0x7f5b00010da0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:50.228 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:50.222+0000 7f5afaffd700 1 -- 172.21.15.42:0/3403190971 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 124 v124) v1 ==== 95+0+1774 (secure 0 0 0) 0x7f5b00032550 con 0x7f5b040a1800 2019-12-05T05:30:50.228 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 124 2019-12-05T05:30:50.229 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:30:50.229 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":124,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":124,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:30:48.694163+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6778},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6778":{"gid":6778,"name":"a","rank":0,"incarnation":124,"state":"up:replay","state_seq":2,"addr":"172.21.15.42:6835/3028851337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3028851337},{"type":"v1","addr":"172.21.15.42:6835","nonce":3028851337}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:30:50.231 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:50.222+0000 7f5b0bae4700 1 -- 172.21.15.42:0/3403190971 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5aec03acf0 msgr2=0x7f5aec03d200 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:30:50.231 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:50.222+0000 7f5b0bae4700 1 --2- 172.21.15.42:0/3403190971 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5aec03acf0 0x7f5aec03d200 crc :-1 s=READY pgs=433 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:30:50.231 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:50.222+0000 7f5b0bae4700 1 -- 172.21.15.42:0/3403190971 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b040a1800 msgr2=0x7f5b040a2840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:30:50.231 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:50.222+0000 7f5b0bae4700 1 --2- 172.21.15.42:0/3403190971 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b040a1800 0x7f5b040a2840 secure :-1 s=READY pgs=786 cs=0 l=1 rx=0x7f5b0407e060 tx=0x7f5b00010da0).stop 2019-12-05T05:30:50.231 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:50.222+0000 7f5b0bae4700 1 -- 172.21.15.42:0/3403190971 shutdown_connections 2019-12-05T05:30:50.232 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:50.222+0000 7f5b0bae4700 1 --2- 172.21.15.42:0/3403190971 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5aec03acf0 0x7f5aec03d200 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:30:50.232 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:50.222+0000 7f5b0bae4700 1 --2- 172.21.15.42:0/3403190971 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b040a1800 0x7f5b040a2840 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:30:50.232 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:50.222+0000 7f5b0bae4700 1 --2- 172.21.15.42:0/3403190971 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5b040a2d90 0x7f5b041413e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:30:50.232 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:50.222+0000 7f5b0bae4700 1 -- 172.21.15.42:0/3403190971 shutdown_connections 2019-12-05T05:30:50.232 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:50.222+0000 7f5b0bae4700 1 -- 172.21.15.42:0/3403190971 wait complete. 2019-12-05T05:30:50.232 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:50.222+0000 7f5b0bae4700 1 -- 172.21.15.42:0/3403190971 >> 172.21.15.42:0/3403190971 conn(0x7f5b040a9770 msgr2=0x7f5b0411b310 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:30:50.240 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 124, "feature_flags": { "enable_multiple": false, "ever_enabled_multiple": false }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2019-12-05T05:03:35.604738+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 124, "ever_allowed_features": 2, "explicitly_allowed_features": 2, "failed": [], "flags": 18, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_6778": { "addr": "172.21.15.42:6835/3028851337", "addrs": { "addrvec": [ { "addr": "172.21.15.42:6834", "nonce": 3028851337, "type": "v2" }, { "addr": "172.21.15.42:6835", "nonce": 3028851337, "type": "v1" } ] }, "export_targets": [], "features": 4540138292836696063, "flags": 0, "gid": 6778, "incarnation": 124, "name": "a", "rank": 0, "state": "up:replay", "state_seq": 2 } }, "last_failure": 0, "last_failure_osd_epoch": 68, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 2, "min_compat_client": "0 (unknown)", "modified": "2019-12-05T05:30:48.694163+0000", "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6778 } } } ], "standbys": [] } 2019-12-05T05:30:52.242 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:30:52.347 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:52.334+0000 7f28704aa700 1 Processor -- start 2019-12-05T05:30:52.347 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:52.334+0000 7f28704aa700 1 -- start start 2019-12-05T05:30:52.347 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:52.334+0000 7f28704aa700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f28680a7740 0x7f286808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:30:52.347 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:52.334+0000 7f28704aa700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28680a0710 con 0x7f286809fae0 2019-12-05T05:30:52.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:52.334+0000 7f28704aa700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28680a9ba0 con 0x7f286807e720 2019-12-05T05:30:52.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:52.334+0000 7f28704aa700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28680a0b00 con 0x7f28680a7740 2019-12-05T05:30:52.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:52.334+0000 7f286e246700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f286807e720 0x7f286809f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:35406/0 (socket says 172.21.15.42:35406) 2019-12-05T05:30:52.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:52.334+0000 7f286e246700 1 -- 172.21.15.42:0/2826503646 learned_addr learned my addr 172.21.15.42:0/2826503646 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:30:52.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:52.334+0000 7f286ea47700 1 --2- 172.21.15.42:0/2826503646 >> v2:172.21.15.40:3300/0 conn(0x7f28680a7740 0x7f286808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:30:52.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:52.334+0000 7f286da45700 1 -- 172.21.15.42:0/2826503646 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2177483959 0 0) 0x7f28680a9ba0 con 0x7f286807e720 2019-12-05T05:30:52.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:52.334+0000 7f286da45700 1 -- 172.21.15.42:0/2826503646 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28540030b0 con 0x7f286807e720 2019-12-05T05:30:52.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:52.334+0000 7f286da45700 1 -- 172.21.15.42:0/2826503646 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3635161748 0 0) 0x7f28540030b0 con 0x7f286807e720 2019-12-05T05:30:52.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:52.334+0000 7f286da45700 1 -- 172.21.15.42:0/2826503646 >> v2:172.21.15.40:3300/0 conn(0x7f28680a7740 msgr2=0x7f286808df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:30:52.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:52.334+0000 7f286da45700 1 --2- 172.21.15.42:0/2826503646 >> v2:172.21.15.40:3300/0 conn(0x7f28680a7740 0x7f286808df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:30:52.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:52.334+0000 7f286da45700 1 -- 172.21.15.42:0/2826503646 >> v1:172.21.15.40:6789/0 conn(0x7f286809fae0 legacy=0x7f286814efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:30:52.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:52.334+0000 7f286da45700 1 -- 172.21.15.42:0/2826503646 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f286807eb50 con 0x7f286807e720 2019-12-05T05:30:52.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:52.334+0000 7f286da45700 1 -- 172.21.15.42:0/2826503646 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7f285c001ad0 con 0x7f286807e720 2019-12-05T05:30:52.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:52.334+0000 7f286da45700 1 -- 172.21.15.42:0/2826503646 >> v1:172.21.15.42:6789/0 conn(0x7f286807e720 legacy=0x7f286809f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:30:52.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:52.334+0000 7f286da45700 1 --2- 172.21.15.42:0/2826503646 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f28540036e0 0x7f2854003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:30:52.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:52.334+0000 7f286da45700 1 --2- 172.21.15.42:0/2826503646 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2854004420 0x7f2854004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:30:52.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:52.334+0000 7f286da45700 1 --2- 172.21.15.42:0/2826503646 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2854004da0 0x7f285400b670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:30:52.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:52.334+0000 7f286da45700 1 -- 172.21.15.42:0/2826503646 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28680a0b00 con 0x7f2854004420 2019-12-05T05:30:52.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:52.334+0000 7f286da45700 1 -- 172.21.15.42:0/2826503646 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f285404fc60 con 0x7f28540036e0 2019-12-05T05:30:52.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:52.334+0000 7f286da45700 1 -- 172.21.15.42:0/2826503646 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f285404fea0 con 0x7f2854004da0 2019-12-05T05:30:52.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:52.334+0000 7f286ea47700 1 --2- 172.21.15.42:0/2826503646 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f28540036e0 0x7f2854003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:30:52.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:52.334+0000 7f286e246700 1 --2- 172.21.15.42:0/2826503646 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2854004420 0x7f2854004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:30:52.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:52.334+0000 7f286f248700 1 --2- 172.21.15.42:0/2826503646 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2854004da0 0x7f285400b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:30:52.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:52.334+0000 7f286ea47700 1 -- 172.21.15.42:0/2826503646 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2854004da0 msgr2=0x7f285400b670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:30:52.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:52.334+0000 7f286ea47700 1 --2- 172.21.15.42:0/2826503646 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2854004da0 0x7f285400b670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:30:52.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:52.334+0000 7f286ea47700 1 -- 172.21.15.42:0/2826503646 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2854004420 msgr2=0x7f2854004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:30:52.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:52.334+0000 7f286ea47700 1 --2- 172.21.15.42:0/2826503646 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2854004420 0x7f2854004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:30:52.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:52.334+0000 7f286ea47700 1 -- 172.21.15.42:0/2826503646 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28540501a0 con 0x7f28540036e0 2019-12-05T05:30:52.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:52.334+0000 7f286f248700 1 --2- 172.21.15.42:0/2826503646 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2854004da0 0x7f285400b670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).send_auth_request state changed! 2019-12-05T05:30:52.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:52.334+0000 7f286e246700 1 --2- 172.21.15.42:0/2826503646 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2854004420 0x7f2854004850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T05:30:52.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:52.334+0000 7f286f248700 1 -- 172.21.15.42:0/2826503646 reap_dead start 2019-12-05T05:30:52.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:52.334+0000 7f286ea47700 1 --2- 172.21.15.42:0/2826503646 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f28540036e0 0x7f2854003b50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f2864002f30 tx=0x7f286400ffe0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:52.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:52.334+0000 7f286ea47700 1 --2- 172.21.15.42:0/2826503646 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f28540036e0 0x7f2854003b50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f2864002f30 tx=0x7f286400ffe0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:52.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:52.334+0000 7f286ea47700 1 --2- 172.21.15.42:0/2826503646 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f28540036e0 0x7f2854003b50 secure :-1 s=READY pgs=1014 cs=0 l=1 rx=0x7f2864002f30 tx=0x7f286400ffe0).ready entity=mon.1 client_cookie=3ba16d13d77b843f server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:30:52.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:52.338+0000 7f286ea47700 1 --2- 172.21.15.42:0/2826503646 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f28540036e0 0x7f2854003b50 secure :-1 s=THROTTLE_DONE pgs=1014 cs=0 l=1 rx=0x7f2864002f30 tx=0x7f286400ffe0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:52.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:52.338+0000 7f286da45700 1 -- 172.21.15.42:0/2826503646 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2864010d10 con 0x7f28540036e0 2019-12-05T05:30:52.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:52.338+0000 7f286ea47700 1 --2- 172.21.15.42:0/2826503646 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f28540036e0 0x7f2854003b50 secure :-1 s=THROTTLE_DONE pgs=1014 cs=0 l=1 rx=0x7f2864002f30 tx=0x7f286400ffe0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:52.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:52.338+0000 7f286da45700 1 -- 172.21.15.42:0/2826503646 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2864010930 con 0x7f28540036e0 2019-12-05T05:30:52.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:52.338+0000 7f286ea47700 1 --2- 172.21.15.42:0/2826503646 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f28540036e0 0x7f2854003b50 secure :-1 s=THROTTLE_DONE pgs=1014 cs=0 l=1 rx=0x7f2864002f30 tx=0x7f286400ffe0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:52.358 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:52.338+0000 7f286da45700 1 -- 172.21.15.42:0/2826503646 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2864008bb0 con 0x7f28540036e0 2019-12-05T05:30:52.358 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:52.338+0000 7f28704aa700 1 -- 172.21.15.42:0/2826503646 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f28540036e0 msgr2=0x7f2854003b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:30:52.358 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:52.338+0000 7f28704aa700 1 --2- 172.21.15.42:0/2826503646 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f28540036e0 0x7f2854003b50 secure :-1 s=READY pgs=1014 cs=0 l=1 rx=0x7f2864002f30 tx=0x7f286400ffe0).stop 2019-12-05T05:30:52.358 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:52.338+0000 7f28704aa700 1 -- 172.21.15.42:0/2826503646 shutdown_connections 2019-12-05T05:30:52.358 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:52.338+0000 7f28704aa700 1 --2- 172.21.15.42:0/2826503646 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f28540036e0 0x7f2854003b50 unknown :-1 s=CLOSED pgs=1014 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:30:52.359 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:52.338+0000 7f28704aa700 1 -- 172.21.15.42:0/2826503646 shutdown_connections 2019-12-05T05:30:52.359 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:52.338+0000 7f28704aa700 1 -- 172.21.15.42:0/2826503646 wait complete. 2019-12-05T05:30:52.359 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:52.338+0000 7f28704aa700 1 -- 172.21.15.42:0/2826503646 >> 172.21.15.42:0/2826503646 conn(0x7f28680aabe0 msgr2=0x7f28680a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:30:52.360 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:52.338+0000 7f28704aa700 1 Processor -- start 2019-12-05T05:30:52.360 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:52.338+0000 7f28704aa700 1 -- start start 2019-12-05T05:30:52.363 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:52.342+0000 7f28704aa700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f286807ba00 0x7f286807be30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:30:52.363 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:52.342+0000 7f28704aa700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2868074800 0x7f2868074c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:30:52.363 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:52.342+0000 7f28704aa700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28680a4b70 con 0x7f286811c8b0 2019-12-05T05:30:52.363 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:52.342+0000 7f28704aa700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f286807c380 con 0x7f286807ba00 2019-12-05T05:30:52.364 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:52.342+0000 7f28704aa700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f286812b2a0 con 0x7f2868074800 2019-12-05T05:30:52.364 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:52.342+0000 7f286f248700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2868074800 0x7f2868074c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:30:52.364 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:52.342+0000 7f286f248700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2868074800 0x7f2868074c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:35384/0 (socket says 172.21.15.42:35384) 2019-12-05T05:30:52.364 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:52.342+0000 7f286e246700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f286807ba00 0x7f286807be30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:30:52.364 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:52.342+0000 7f286f248700 1 -- 172.21.15.42:0/3326981548 learned_addr learned my addr 172.21.15.42:0/3326981548 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:30:52.364 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:52.342+0000 7f286e246700 1 -- 172.21.15.42:0/3326981548 >> v2:172.21.15.42:3300/0 conn(0x7f2868074800 msgr2=0x7f2868074c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:30:52.365 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:52.342+0000 7f286e246700 1 --2- 172.21.15.42:0/3326981548 >> v2:172.21.15.42:3300/0 conn(0x7f2868074800 0x7f2868074c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:30:52.365 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:52.342+0000 7f286e246700 1 -- 172.21.15.42:0/3326981548 >> v1:172.21.15.40:6789/0 conn(0x7f286811c8b0 legacy=0x7f28680a5170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:30:52.365 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:52.342+0000 7f286e246700 1 -- 172.21.15.42:0/3326981548 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2868075180 con 0x7f286807ba00 2019-12-05T05:30:52.365 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:52.342+0000 7f285b7fe700 1 -- 172.21.15.42:0/3326981548 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2533595481 0 0) 0x7f28680a4b70 con 0x7f286811c8b0 2019-12-05T05:30:52.365 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:52.342+0000 7f286f248700 1 --2- 172.21.15.42:0/3326981548 >> v2:172.21.15.42:3300/0 conn(0x7f2868074800 0x7f2868074c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:30:52.365 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:52.342+0000 7f286e246700 1 --2- 172.21.15.42:0/3326981548 >> v2:172.21.15.40:3300/0 conn(0x7f286807ba00 0x7f286807be30 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f2868141d00 tx=0x7f285c010e20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:52.365 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:52.342+0000 7f286e246700 1 --2- 172.21.15.42:0/3326981548 >> v2:172.21.15.40:3300/0 conn(0x7f286807ba00 0x7f286807be30 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f2868141d00 tx=0x7f285c010e20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:52.366 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:52.342+0000 7f286e246700 1 --2- 172.21.15.42:0/3326981548 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f286807ba00 0x7f286807be30 secure :-1 s=READY pgs=789 cs=0 l=1 rx=0x7f2868141d00 tx=0x7f285c010e20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:30:52.366 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:52.342+0000 7f286e246700 1 --2- 172.21.15.42:0/3326981548 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f286807ba00 0x7f286807be30 secure :-1 s=THROTTLE_DONE pgs=789 cs=0 l=1 rx=0x7f2868141d00 tx=0x7f285c010e20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:52.366 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:52.342+0000 7f285b7fe700 1 -- 172.21.15.42:0/3326981548 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f285c0090c0 con 0x7f286807ba00 2019-12-05T05:30:52.366 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:52.342+0000 7f286e246700 1 --2- 172.21.15.42:0/3326981548 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f286807ba00 0x7f286807be30 secure :-1 s=THROTTLE_DONE pgs=789 cs=0 l=1 rx=0x7f2868141d00 tx=0x7f285c010e20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:52.366 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:52.342+0000 7f286e246700 1 --2- 172.21.15.42:0/3326981548 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f286807ba00 0x7f286807be30 secure :-1 s=THROTTLE_DONE pgs=789 cs=0 l=1 rx=0x7f2868141d00 tx=0x7f285c010e20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:52.367 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:52.342+0000 7f285b7fe700 1 -- 172.21.15.42:0/3326981548 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f285c008b00 con 0x7f286807ba00 2019-12-05T05:30:52.367 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:52.342+0000 7f28704aa700 1 -- 172.21.15.42:0/3326981548 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2868075450 con 0x7f286807ba00 2019-12-05T05:30:52.367 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:52.342+0000 7f285b7fe700 1 -- 172.21.15.42:0/3326981548 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f285c0096a0 con 0x7f286807ba00 2019-12-05T05:30:52.367 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:52.342+0000 7f28704aa700 1 -- 172.21.15.42:0/3326981548 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28681329c0 con 0x7f286807ba00 2019-12-05T05:30:52.368 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:52.342+0000 7f286e246700 1 --2- 172.21.15.42:0/3326981548 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f286807ba00 0x7f286807be30 secure :-1 s=THROTTLE_DONE pgs=789 cs=0 l=1 rx=0x7f2868141d00 tx=0x7f285c010e20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:52.368 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:52.342+0000 7f285b7fe700 1 -- 172.21.15.42:0/3326981548 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f285c0099b0 con 0x7f286807ba00 2019-12-05T05:30:52.368 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:52.342+0000 7f286e246700 1 --2- 172.21.15.42:0/3326981548 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f286807ba00 0x7f286807be30 secure :-1 s=THROTTLE_DONE pgs=789 cs=0 l=1 rx=0x7f2868141d00 tx=0x7f285c010e20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:52.368 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:52.346+0000 7f285b7fe700 1 --2- 172.21.15.42:0/3326981548 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f285003ac90 0x7f285003d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:30:52.368 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:52.346+0000 7f285b7fe700 1 -- 172.21.15.42:0/3326981548 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 7665+0+0 (secure 0 0 0) 0x7f285c04b710 con 0x7f286807ba00 2019-12-05T05:30:52.369 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:52.346+0000 7f286ea47700 1 --2- 172.21.15.42:0/3326981548 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f285003ac90 0x7f285003d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:30:52.369 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:52.346+0000 7f286ea47700 1 --2- 172.21.15.42:0/3326981548 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f285003ac90 0x7f285003d1a0 crc :-1 s=READY pgs=434 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:30:52.369 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:52.350+0000 7f28704aa700 1 -- 172.21.15.42:0/3326981548 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f286805d110 con 0x7f286807ba00 2019-12-05T05:30:52.370 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:52.358+0000 7f286e246700 1 --2- 172.21.15.42:0/3326981548 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f286807ba00 0x7f286807be30 secure :-1 s=THROTTLE_DONE pgs=789 cs=0 l=1 rx=0x7f2868141d00 tx=0x7f285c010e20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:52.370 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:52.358+0000 7f285b7fe700 1 -- 172.21.15.42:0/3326981548 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f285c032790 con 0x7f286807ba00 2019-12-05T05:30:52.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:52.526+0000 7f28704aa700 1 -- 172.21.15.42:0/3326981548 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f286805b8c0 con 0x7f286807ba00 2019-12-05T05:30:52.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:52.526+0000 7f286e246700 1 --2- 172.21.15.42:0/3326981548 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f286807ba00 0x7f286807be30 secure :-1 s=THROTTLE_DONE pgs=789 cs=0 l=1 rx=0x7f2868141d00 tx=0x7f285c010e20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:52.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:52.526+0000 7f285b7fe700 1 -- 172.21.15.42:0/3326981548 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 125 v125) v1 ==== 95+0+1778 (secure 0 0 0) 0x7f285c0329f0 con 0x7f286807ba00 2019-12-05T05:30:52.538 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 125 2019-12-05T05:30:52.538 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:30:52.538 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":125,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":125,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:30:51.726682+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6778},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6778":{"gid":6778,"name":"a","rank":0,"incarnation":124,"state":"up:reconnect","state_seq":21,"addr":"172.21.15.42:6835/3028851337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3028851337},{"type":"v1","addr":"172.21.15.42:6835","nonce":3028851337}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:30:52.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:52.530+0000 7f28704aa700 1 -- 172.21.15.42:0/3326981548 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f285003ac90 msgr2=0x7f285003d1a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:30:52.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:52.530+0000 7f28704aa700 1 --2- 172.21.15.42:0/3326981548 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f285003ac90 0x7f285003d1a0 crc :-1 s=READY pgs=434 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:30:52.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:52.530+0000 7f28704aa700 1 -- 172.21.15.42:0/3326981548 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f286807ba00 msgr2=0x7f286807be30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:30:52.541 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:52.530+0000 7f28704aa700 1 --2- 172.21.15.42:0/3326981548 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f286807ba00 0x7f286807be30 secure :-1 s=READY pgs=789 cs=0 l=1 rx=0x7f2868141d00 tx=0x7f285c010e20).stop 2019-12-05T05:30:52.541 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:52.530+0000 7f28704aa700 1 -- 172.21.15.42:0/3326981548 shutdown_connections 2019-12-05T05:30:52.541 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:52.530+0000 7f28704aa700 1 --2- 172.21.15.42:0/3326981548 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f285003ac90 0x7f285003d1a0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:30:52.541 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:52.530+0000 7f28704aa700 1 --2- 172.21.15.42:0/3326981548 >> v2:172.21.15.42:3300/0 conn(0x7f2868074800 0x7f2868074c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:30:52.541 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:52.530+0000 7f28704aa700 1 --2- 172.21.15.42:0/3326981548 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f286807ba00 0x7f286807be30 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:30:52.541 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:52.530+0000 7f28704aa700 1 -- 172.21.15.42:0/3326981548 shutdown_connections 2019-12-05T05:30:52.542 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:52.530+0000 7f28704aa700 1 -- 172.21.15.42:0/3326981548 wait complete. 2019-12-05T05:30:52.542 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:52.530+0000 7f28704aa700 1 -- 172.21.15.42:0/3326981548 >> 172.21.15.42:0/3326981548 conn(0x7f28680aabe0 msgr2=0x7f2868141360 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:30:52.548 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2019-12-05T05:30:54.549 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:30:54.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:54.638+0000 7fc7139c5700 1 Processor -- start 2019-12-05T05:30:54.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:54.638+0000 7fc7139c5700 1 -- start start 2019-12-05T05:30:54.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:54.638+0000 7fc7139c5700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc70c0a7740 0x7fc70c08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:30:54.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:54.638+0000 7fc7139c5700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc70c09fae0 0x7fc70c07eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:30:54.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:54.638+0000 7fc7139c5700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc70c0a0710 con 0x7fc70c07e720 2019-12-05T05:30:54.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:54.638+0000 7fc7139c5700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc70c0a0b00 con 0x7fc70c0a7740 2019-12-05T05:30:54.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:54.638+0000 7fc7139c5700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc70c08e5b0 con 0x7fc70c09fae0 2019-12-05T05:30:54.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:54.638+0000 7fc711761700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fc70c07e720 0x7fc70c09f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:35434/0 (socket says 172.21.15.42:35434) 2019-12-05T05:30:54.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:54.638+0000 7fc711761700 1 -- 172.21.15.42:0/2517947174 learned_addr learned my addr 172.21.15.42:0/2517947174 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:30:54.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:54.638+0000 7fc711f62700 1 --2- 172.21.15.42:0/2517947174 >> v2:172.21.15.40:3300/0 conn(0x7fc70c0a7740 0x7fc70c08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:30:54.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:54.638+0000 7fc712763700 1 --2- 172.21.15.42:0/2517947174 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc70c09fae0 0x7fc70c07eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:30:54.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:54.642+0000 7fc710f60700 1 -- 172.21.15.42:0/2517947174 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1188718624 0 0) 0x7fc70c0a0710 con 0x7fc70c07e720 2019-12-05T05:30:54.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:54.642+0000 7fc710f60700 1 -- 172.21.15.42:0/2517947174 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6f80030b0 con 0x7fc70c07e720 2019-12-05T05:30:54.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:54.642+0000 7fc712763700 1 -- 172.21.15.42:0/2517947174 >> v2:172.21.15.40:3300/0 conn(0x7fc70c0a7740 msgr2=0x7fc70c08df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:30:54.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:54.642+0000 7fc712763700 1 --2- 172.21.15.42:0/2517947174 >> v2:172.21.15.40:3300/0 conn(0x7fc70c0a7740 0x7fc70c08df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:30:54.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:54.642+0000 7fc712763700 1 -- 172.21.15.42:0/2517947174 >> v1:172.21.15.42:6789/0 conn(0x7fc70c07e720 legacy=0x7fc70c09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:30:54.654 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:54.642+0000 7fc712763700 1 -- 172.21.15.42:0/2517947174 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc70c07f0a0 con 0x7fc70c09fae0 2019-12-05T05:30:54.654 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:54.642+0000 7fc712763700 1 --2- 172.21.15.42:0/2517947174 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc70c09fae0 0x7fc70c07eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fc704008b10 tx=0x7fc70400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:54.654 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:54.642+0000 7fc712763700 1 --2- 172.21.15.42:0/2517947174 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc70c09fae0 0x7fc70c07eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fc704008b10 tx=0x7fc70400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:54.654 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:54.642+0000 7fc712763700 1 --2- 172.21.15.42:0/2517947174 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc70c09fae0 0x7fc70c07eb50 secure :-1 s=READY pgs=204 cs=0 l=1 rx=0x7fc704008b10 tx=0x7fc70400fff0).ready entity=mon.2 client_cookie=40e0e8eb5834de13 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:30:54.655 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:54.642+0000 7fc712763700 1 --2- 172.21.15.42:0/2517947174 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc70c09fae0 0x7fc70c07eb50 secure :-1 s=THROTTLE_DONE pgs=204 cs=0 l=1 rx=0x7fc704008b10 tx=0x7fc70400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:54.655 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:54.642+0000 7fc710f60700 1 -- 172.21.15.42:0/2517947174 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc704010900 con 0x7fc70c09fae0 2019-12-05T05:30:54.655 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:54.642+0000 7fc712763700 1 --2- 172.21.15.42:0/2517947174 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc70c09fae0 0x7fc70c07eb50 secure :-1 s=THROTTLE_DONE pgs=204 cs=0 l=1 rx=0x7fc704008b10 tx=0x7fc70400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:54.655 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:54.642+0000 7fc710f60700 1 -- 172.21.15.42:0/2517947174 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc704008b50 con 0x7fc70c09fae0 2019-12-05T05:30:54.655 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:54.642+0000 7fc712763700 1 --2- 172.21.15.42:0/2517947174 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc70c09fae0 0x7fc70c07eb50 secure :-1 s=THROTTLE_DONE pgs=204 cs=0 l=1 rx=0x7fc704008b10 tx=0x7fc70400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:54.656 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:54.642+0000 7fc710f60700 1 -- 172.21.15.42:0/2517947174 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc7040110c0 con 0x7fc70c09fae0 2019-12-05T05:30:54.656 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:54.642+0000 7fc7139c5700 1 -- 172.21.15.42:0/2517947174 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc70c09fae0 msgr2=0x7fc70c07eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:30:54.656 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:54.642+0000 7fc7139c5700 1 --2- 172.21.15.42:0/2517947174 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc70c09fae0 0x7fc70c07eb50 secure :-1 s=READY pgs=204 cs=0 l=1 rx=0x7fc704008b10 tx=0x7fc70400fff0).stop 2019-12-05T05:30:54.656 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:54.642+0000 7fc7139c5700 1 -- 172.21.15.42:0/2517947174 shutdown_connections 2019-12-05T05:30:54.656 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:54.642+0000 7fc7139c5700 1 --2- 172.21.15.42:0/2517947174 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc70c09fae0 0x7fc70c07eb50 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:30:54.657 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:54.642+0000 7fc7139c5700 1 --2- 172.21.15.42:0/2517947174 >> v2:172.21.15.40:3300/0 conn(0x7fc70c0a7740 0x7fc70c08df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:30:54.657 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:54.642+0000 7fc7139c5700 1 -- 172.21.15.42:0/2517947174 shutdown_connections 2019-12-05T05:30:54.657 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:54.642+0000 7fc7139c5700 1 -- 172.21.15.42:0/2517947174 wait complete. 2019-12-05T05:30:54.657 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:54.642+0000 7fc7139c5700 1 -- 172.21.15.42:0/2517947174 >> 172.21.15.42:0/2517947174 conn(0x7fc70c0aabe0 msgr2=0x7fc70c0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:30:54.660 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:54.646+0000 7fc7139c5700 1 Processor -- start 2019-12-05T05:30:54.661 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:54.646+0000 7fc7139c5700 1 -- start start 2019-12-05T05:30:54.661 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:54.646+0000 7fc7139c5700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc70c11bd90 0x7fc70c130db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:30:54.661 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:54.646+0000 7fc7139c5700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc70c0a6720 0x7fc70c1378a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:30:54.661 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:54.646+0000 7fc7139c5700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc70c11c1c0 con 0x7fc70c07c830 2019-12-05T05:30:54.661 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:54.646+0000 7fc7139c5700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc70c07cc60 con 0x7fc70c0a6720 2019-12-05T05:30:54.661 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:54.646+0000 7fc7139c5700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc70c130870 con 0x7fc70c11bd90 2019-12-05T05:30:54.661 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:54.646+0000 7fc711761700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc70c11bd90 0x7fc70c130db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:30:54.662 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:54.646+0000 7fc711f62700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fc70c07c830 0x7fc70c144550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:35438/0 (socket says 172.21.15.42:35438) 2019-12-05T05:30:54.662 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:54.646+0000 7fc711f62700 1 -- 172.21.15.42:0/2510693461 learned_addr learned my addr 172.21.15.42:0/2510693461 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:30:54.662 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:54.646+0000 7fc711761700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc70c11bd90 0x7fc70c130db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:35402/0 (socket says 172.21.15.42:35402) 2019-12-05T05:30:54.662 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:54.646+0000 7fc712763700 1 --2- 172.21.15.42:0/2510693461 >> v2:172.21.15.40:3300/0 conn(0x7fc70c0a6720 0x7fc70c1378a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:30:54.662 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:54.650+0000 7fc711761700 1 -- 172.21.15.42:0/2510693461 >> v2:172.21.15.40:3300/0 conn(0x7fc70c0a6720 msgr2=0x7fc70c1378a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:30:54.662 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:54.650+0000 7fc711761700 1 --2- 172.21.15.42:0/2510693461 >> v2:172.21.15.40:3300/0 conn(0x7fc70c0a6720 0x7fc70c1378a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:30:54.663 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:54.650+0000 7fc7027fc700 1 -- 172.21.15.42:0/2510693461 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1241078628 0 0) 0x7fc70c11c1c0 con 0x7fc70c07c830 2019-12-05T05:30:54.671 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:54.650+0000 7fc711761700 1 -- 172.21.15.42:0/2510693461 >> v1:172.21.15.42:6789/0 conn(0x7fc70c07c830 legacy=0x7fc70c144550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:30:54.672 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:54.650+0000 7fc711761700 1 -- 172.21.15.42:0/2510693461 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc70c130450 con 0x7fc70c11bd90 2019-12-05T05:30:54.672 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:54.650+0000 7fc711761700 1 --2- 172.21.15.42:0/2510693461 >> v2:172.21.15.42:3300/0 conn(0x7fc70c11bd90 0x7fc70c130db0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fc6fc0043a0 tx=0x7fc6fc0108b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:54.672 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:54.650+0000 7fc711761700 1 --2- 172.21.15.42:0/2510693461 >> v2:172.21.15.42:3300/0 conn(0x7fc70c11bd90 0x7fc70c130db0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fc6fc0043a0 tx=0x7fc6fc0108b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:54.672 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:54.650+0000 7fc711761700 1 --2- 172.21.15.42:0/2510693461 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc70c11bd90 0x7fc70c130db0 secure :-1 s=READY pgs=1019 cs=0 l=1 rx=0x7fc6fc0043a0 tx=0x7fc6fc0108b0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:30:54.672 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:54.650+0000 7fc711761700 1 --2- 172.21.15.42:0/2510693461 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc70c11bd90 0x7fc70c130db0 secure :-1 s=THROTTLE_DONE pgs=1019 cs=0 l=1 rx=0x7fc6fc0043a0 tx=0x7fc6fc0108b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:54.672 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:54.650+0000 7fc7027fc700 1 -- 172.21.15.42:0/2510693461 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc6fc005b40 con 0x7fc70c11bd90 2019-12-05T05:30:54.672 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:54.650+0000 7fc711761700 1 --2- 172.21.15.42:0/2510693461 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc70c11bd90 0x7fc70c130db0 secure :-1 s=THROTTLE_DONE pgs=1019 cs=0 l=1 rx=0x7fc6fc0043a0 tx=0x7fc6fc0108b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:54.673 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:54.650+0000 7fc7139c5700 1 -- 172.21.15.42:0/2510693461 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc70c0a6b50 con 0x7fc70c11bd90 2019-12-05T05:30:54.673 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:54.650+0000 7fc7027fc700 1 -- 172.21.15.42:0/2510693461 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6fc005760 con 0x7fc70c11bd90 2019-12-05T05:30:54.673 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:54.650+0000 7fc711761700 1 --2- 172.21.15.42:0/2510693461 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc70c11bd90 0x7fc70c130db0 secure :-1 s=THROTTLE_DONE pgs=1019 cs=0 l=1 rx=0x7fc6fc0043a0 tx=0x7fc6fc0108b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:54.673 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:54.650+0000 7fc7027fc700 1 -- 172.21.15.42:0/2510693461 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc6fc0062b0 con 0x7fc70c11bd90 2019-12-05T05:30:54.673 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:54.650+0000 7fc7139c5700 1 -- 172.21.15.42:0/2510693461 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc70c137df0 con 0x7fc70c11bd90 2019-12-05T05:30:54.673 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:54.650+0000 7fc711761700 1 --2- 172.21.15.42:0/2510693461 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc70c11bd90 0x7fc70c130db0 secure :-1 s=THROTTLE_DONE pgs=1019 cs=0 l=1 rx=0x7fc6fc0043a0 tx=0x7fc6fc0108b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:54.673 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:54.650+0000 7fc7027fc700 1 -- 172.21.15.42:0/2510693461 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fc6fc006590 con 0x7fc70c11bd90 2019-12-05T05:30:54.674 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:54.650+0000 7fc711761700 1 --2- 172.21.15.42:0/2510693461 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc70c11bd90 0x7fc70c130db0 secure :-1 s=THROTTLE_DONE pgs=1019 cs=0 l=1 rx=0x7fc6fc0043a0 tx=0x7fc6fc0108b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:54.674 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:54.650+0000 7fc7027fc700 1 --2- 172.21.15.42:0/2510693461 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc6f403aca0 0x7fc6f403d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:30:54.674 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:54.650+0000 7fc7027fc700 1 -- 172.21.15.42:0/2510693461 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 7665+0+0 (secure 0 0 0) 0x7fc6fc04af60 con 0x7fc70c11bd90 2019-12-05T05:30:54.674 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:54.650+0000 7fc711f62700 1 --2- 172.21.15.42:0/2510693461 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc6f403aca0 0x7fc6f403d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:30:54.674 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:54.650+0000 7fc711f62700 1 --2- 172.21.15.42:0/2510693461 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc6f403aca0 0x7fc6f403d1b0 crc :-1 s=READY pgs=435 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:30:54.674 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:54.654+0000 7fc7139c5700 1 -- 172.21.15.42:0/2510693461 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc70c05d110 con 0x7fc70c11bd90 2019-12-05T05:30:54.675 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:54.662+0000 7fc711761700 1 --2- 172.21.15.42:0/2510693461 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc70c11bd90 0x7fc70c130db0 secure :-1 s=THROTTLE_DONE pgs=1019 cs=0 l=1 rx=0x7fc6fc0043a0 tx=0x7fc6fc0108b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:54.675 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:54.662+0000 7fc7027fc700 1 -- 172.21.15.42:0/2510693461 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fc6fc031fc0 con 0x7fc70c11bd90 2019-12-05T05:30:54.851 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:54.842+0000 7fc7139c5700 1 -- 172.21.15.42:0/2510693461 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc70c05b8c0 con 0x7fc70c11bd90 2019-12-05T05:30:54.852 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:54.846+0000 7fc711761700 1 --2- 172.21.15.42:0/2510693461 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc70c11bd90 0x7fc70c130db0 secure :-1 s=THROTTLE_DONE pgs=1019 cs=0 l=1 rx=0x7fc6fc0043a0 tx=0x7fc6fc0108b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:30:54.852 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:54.846+0000 7fc7027fc700 1 -- 172.21.15.42:0/2510693461 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 127 v127) v1 ==== 95+0+1775 (secure 0 0 0) 0x7fc6fc032220 con 0x7fc70c11bd90 2019-12-05T05:30:54.852 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 127 2019-12-05T05:30:54.852 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:30:54.852 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":127,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":127,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:30:53.739046+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6778},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6778":{"gid":6778,"name":"a","rank":0,"incarnation":124,"state":"up:active","state_seq":23,"addr":"172.21.15.42:6835/3028851337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3028851337},{"type":"v1","addr":"172.21.15.42:6835","nonce":3028851337}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:30:54.855 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:54.846+0000 7fc7139c5700 1 -- 172.21.15.42:0/2510693461 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc6f403aca0 msgr2=0x7fc6f403d1b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:30:54.855 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:54.846+0000 7fc7139c5700 1 --2- 172.21.15.42:0/2510693461 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc6f403aca0 0x7fc6f403d1b0 crc :-1 s=READY pgs=435 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:30:54.855 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:54.846+0000 7fc7139c5700 1 -- 172.21.15.42:0/2510693461 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc70c11bd90 msgr2=0x7fc70c130db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:30:54.855 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:54.846+0000 7fc7139c5700 1 --2- 172.21.15.42:0/2510693461 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc70c11bd90 0x7fc70c130db0 secure :-1 s=READY pgs=1019 cs=0 l=1 rx=0x7fc6fc0043a0 tx=0x7fc6fc0108b0).stop 2019-12-05T05:30:54.855 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:54.846+0000 7fc7139c5700 1 -- 172.21.15.42:0/2510693461 shutdown_connections 2019-12-05T05:30:54.855 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:54.846+0000 7fc7139c5700 1 --2- 172.21.15.42:0/2510693461 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc6f403aca0 0x7fc6f403d1b0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:30:54.856 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:54.846+0000 7fc7139c5700 1 --2- 172.21.15.42:0/2510693461 >> v2:172.21.15.40:3300/0 conn(0x7fc70c0a6720 0x7fc70c1378a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:30:54.856 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:54.846+0000 7fc7139c5700 1 --2- 172.21.15.42:0/2510693461 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc70c11bd90 0x7fc70c130db0 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:30:54.856 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:54.846+0000 7fc7139c5700 1 -- 172.21.15.42:0/2510693461 shutdown_connections 2019-12-05T05:30:54.856 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:54.846+0000 7fc7139c5700 1 -- 172.21.15.42:0/2510693461 wait complete. 2019-12-05T05:30:54.856 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:30:54.846+0000 7fc7139c5700 1 -- 172.21.15.42:0/2510693461 >> 172.21.15.42:0/2510693461 conn(0x7fc70c0a99c0 msgr2=0x7fc70c059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:30:54.869 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a has gained rank=0, replacing gid=5426 2019-12-05T05:30:54.869 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 9 secs before reviving mds.b 2019-12-05T05:31:03.870 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.b 2019-12-05T05:31:03.870 INFO:tasks.ceph.mds.b:Restarting daemon 2019-12-05T05:31:03.870 INFO:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2019-12-05T05:31:03.877 INFO:tasks.ceph.mds.b:Started 2019-12-05T05:31:03.878 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:31:03.938 INFO:tasks.ceph.mds.b.smithi040.stdout:starting mds.b at 2019-12-05T05:31:03.988 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:03.978+0000 7f5fb222f700 1 Processor -- start 2019-12-05T05:31:03.988 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:03.978+0000 7f5fb222f700 1 -- start start 2019-12-05T05:31:03.988 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:03.978+0000 7f5fb222f700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5fac0a7730 0x7f5fac08df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:31:03.989 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:03.978+0000 7f5fb222f700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fac0a0700 con 0x7f5fac09fad0 2019-12-05T05:31:03.989 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:03.978+0000 7f5fb222f700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fac0a9b90 con 0x7f5fac07e710 2019-12-05T05:31:03.989 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:03.978+0000 7f5fb222f700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5fac0a0af0 con 0x7f5fac0a7730 2019-12-05T05:31:03.989 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:03.978+0000 7f5fabfff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5fac0a7730 0x7f5fac08df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:31:03.989 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:03.978+0000 7f5fab7fe700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f5fac07e710 0x7f5fac09f2d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:35514/0 (socket says 172.21.15.42:35514) 2019-12-05T05:31:03.989 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:03.978+0000 7f5fb0fcd700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f5fac09fad0 0x7f5fac14efd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:44952/0 (socket says 172.21.15.42:44952) 2019-12-05T05:31:03.990 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:03.978+0000 7f5fab7fe700 1 -- 172.21.15.42:0/692498511 learned_addr learned my addr 172.21.15.42:0/692498511 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:31:03.990 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:03.978+0000 7f5fabfff700 1 --2- 172.21.15.42:0/692498511 >> v2:172.21.15.42:3300/0 conn(0x7f5fac0a7730 0x7f5fac08df70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:35472/0 (socket says 172.21.15.42:35472) 2019-12-05T05:31:03.990 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:03.978+0000 7f5fabfff700 1 -- 172.21.15.42:0/692498511 >> v1:172.21.15.42:6789/0 conn(0x7f5fac07e710 legacy=0x7f5fac09f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:31:03.990 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:03.978+0000 7f5faaffd700 1 -- 172.21.15.42:0/692498511 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 605682743 0 0) 0x7f5fac0a0700 con 0x7f5fac09fad0 2019-12-05T05:31:03.990 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:03.978+0000 7f5fabfff700 1 -- 172.21.15.42:0/692498511 >> v1:172.21.15.40:6789/0 conn(0x7f5fac09fad0 legacy=0x7f5fac14efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:31:03.990 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:03.978+0000 7f5fabfff700 1 -- 172.21.15.42:0/692498511 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fac07eb40 con 0x7f5fac0a7730 2019-12-05T05:31:03.990 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:03.978+0000 7f5fabfff700 1 --2- 172.21.15.42:0/692498511 >> v2:172.21.15.42:3300/0 conn(0x7f5fac0a7730 0x7f5fac08df70 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f5f9c008790 tx=0x7f5f9c00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:03.991 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:03.978+0000 7f5fabfff700 1 --2- 172.21.15.42:0/692498511 >> v2:172.21.15.42:3300/0 conn(0x7f5fac0a7730 0x7f5fac08df70 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f5f9c008790 tx=0x7f5f9c00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:03.991 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:03.978+0000 7f5fabfff700 1 --2- 172.21.15.42:0/692498511 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5fac0a7730 0x7f5fac08df70 secure :-1 s=READY pgs=1021 cs=0 l=1 rx=0x7f5f9c008790 tx=0x7f5f9c00fbf0).ready entity=mon.1 client_cookie=de93d1417ad2763b server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:31:03.991 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:03.978+0000 7f5fabfff700 1 --2- 172.21.15.42:0/692498511 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5fac0a7730 0x7f5fac08df70 secure :-1 s=THROTTLE_DONE pgs=1021 cs=0 l=1 rx=0x7f5f9c008790 tx=0x7f5f9c00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:03.991 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:03.978+0000 7f5faaffd700 1 -- 172.21.15.42:0/692498511 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5f9c011270 con 0x7f5fac0a7730 2019-12-05T05:31:03.991 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:03.978+0000 7f5fabfff700 1 --2- 172.21.15.42:0/692498511 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5fac0a7730 0x7f5fac08df70 secure :-1 s=THROTTLE_DONE pgs=1021 cs=0 l=1 rx=0x7f5f9c008790 tx=0x7f5f9c00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:03.991 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:03.978+0000 7f5faaffd700 1 -- 172.21.15.42:0/692498511 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f9c008bc0 con 0x7f5fac0a7730 2019-12-05T05:31:03.991 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:03.978+0000 7f5fabfff700 1 --2- 172.21.15.42:0/692498511 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5fac0a7730 0x7f5fac08df70 secure :-1 s=THROTTLE_DONE pgs=1021 cs=0 l=1 rx=0x7f5f9c008790 tx=0x7f5f9c00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:03.991 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:03.978+0000 7f5faaffd700 1 -- 172.21.15.42:0/692498511 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5f9c011850 con 0x7f5fac0a7730 2019-12-05T05:31:03.992 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:03.978+0000 7f5fb222f700 1 -- 172.21.15.42:0/692498511 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5fac0a7730 msgr2=0x7f5fac08df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:31:03.992 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:03.978+0000 7f5fb222f700 1 --2- 172.21.15.42:0/692498511 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5fac0a7730 0x7f5fac08df70 secure :-1 s=READY pgs=1021 cs=0 l=1 rx=0x7f5f9c008790 tx=0x7f5f9c00fbf0).stop 2019-12-05T05:31:03.992 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:03.978+0000 7f5fb222f700 1 -- 172.21.15.42:0/692498511 shutdown_connections 2019-12-05T05:31:03.992 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:03.978+0000 7f5fb222f700 1 --2- 172.21.15.42:0/692498511 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5fac0a7730 0x7f5fac08df70 unknown :-1 s=CLOSED pgs=1021 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:31:03.992 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:03.978+0000 7f5fb222f700 1 -- 172.21.15.42:0/692498511 shutdown_connections 2019-12-05T05:31:03.992 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:03.982+0000 7f5fb222f700 1 -- 172.21.15.42:0/692498511 wait complete. 2019-12-05T05:31:03.992 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:03.982+0000 7f5fb222f700 1 -- 172.21.15.42:0/692498511 >> 172.21.15.42:0/692498511 conn(0x7f5fac0aabd0 msgr2=0x7f5fac0a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:31:03.996 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:03.982+0000 7f5fb222f700 1 Processor -- start 2019-12-05T05:31:03.996 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:03.982+0000 7f5fb222f700 1 -- start start 2019-12-05T05:31:03.996 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:03.986+0000 7f5fb222f700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5fac07a760 0x7f5fac1480b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:31:03.996 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:03.986+0000 7f5fb222f700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5fac125700 0x7f5fac125b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:31:03.996 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:03.986+0000 7f5fb222f700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fac075fc0 con 0x7f5fac07d000 2019-12-05T05:31:03.997 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:03.986+0000 7f5fb222f700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5fac148600 con 0x7f5fac125700 2019-12-05T05:31:03.997 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:03.986+0000 7f5fb222f700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fac07ace0 con 0x7f5fac07a760 2019-12-05T05:31:03.997 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:03.986+0000 7f5fb0fcd700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5fac125700 0x7f5fac125b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:31:03.997 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:03.986+0000 7f5fb0fcd700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5fac125700 0x7f5fac125b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:35482/0 (socket says 172.21.15.42:35482) 2019-12-05T05:31:03.997 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:03.986+0000 7f5fb0fcd700 1 -- 172.21.15.42:0/2651065360 learned_addr learned my addr 172.21.15.42:0/2651065360 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:31:03.997 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:03.986+0000 7f5fabfff700 1 --2- 172.21.15.42:0/2651065360 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5fac07a760 0x7f5fac1480b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:31:03.998 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:03.986+0000 7f5fb0fcd700 1 -- 172.21.15.42:0/2651065360 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5fac07a760 msgr2=0x7f5fac1480b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:31:03.998 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:03.986+0000 7f5fb0fcd700 1 --2- 172.21.15.42:0/2651065360 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5fac07a760 0x7f5fac1480b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:31:03.998 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:03.986+0000 7f5fb0fcd700 1 -- 172.21.15.42:0/2651065360 >> v1:172.21.15.42:6789/0 conn(0x7f5fac07d000 legacy=0x7f5fac0757c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:31:03.998 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:03.986+0000 7f5fb0fcd700 1 -- 172.21.15.42:0/2651065360 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fac126080 con 0x7f5fac125700 2019-12-05T05:31:03.998 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:03.986+0000 7f5fb0fcd700 1 --2- 172.21.15.42:0/2651065360 >> v2:172.21.15.42:3300/0 conn(0x7f5fac125700 0x7f5fac125b30 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f5fac07aea0 tx=0x7f5fa4010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:03.998 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:03.986+0000 7f5fb0fcd700 1 --2- 172.21.15.42:0/2651065360 >> v2:172.21.15.42:3300/0 conn(0x7f5fac125700 0x7f5fac125b30 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f5fac07aea0 tx=0x7f5fa4010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:03.998 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:03.986+0000 7f5fb0fcd700 1 --2- 172.21.15.42:0/2651065360 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5fac125700 0x7f5fac125b30 secure :-1 s=READY pgs=1023 cs=0 l=1 rx=0x7f5fac07aea0 tx=0x7f5fa4010b50).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:31:03.999 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:03.986+0000 7f5fb0fcd700 1 --2- 172.21.15.42:0/2651065360 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5fac125700 0x7f5fac125b30 secure :-1 s=THROTTLE_DONE pgs=1023 cs=0 l=1 rx=0x7f5fac07aea0 tx=0x7f5fa4010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:03.999 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:03.986+0000 7f5fb0fcd700 1 --2- 172.21.15.42:0/2651065360 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5fac125700 0x7f5fac125b30 secure :-1 s=THROTTLE_DONE pgs=1023 cs=0 l=1 rx=0x7f5fac07aea0 tx=0x7f5fa4010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:03.999 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:03.986+0000 7f5fb0fcd700 1 --2- 172.21.15.42:0/2651065360 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5fac125700 0x7f5fac125b30 secure :-1 s=THROTTLE_DONE pgs=1023 cs=0 l=1 rx=0x7f5fac07aea0 tx=0x7f5fa4010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:03.999 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:03.986+0000 7f5fa8ff9700 1 -- 172.21.15.42:0/2651065360 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5fa400d0c0 con 0x7f5fac125700 2019-12-05T05:31:03.999 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:03.986+0000 7f5fa8ff9700 1 -- 172.21.15.42:0/2651065360 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fa4011500 con 0x7f5fac125700 2019-12-05T05:31:03.999 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:03.986+0000 7f5fb222f700 1 -- 172.21.15.42:0/2651065360 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fac12cab0 con 0x7f5fac125700 2019-12-05T05:31:03.999 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:03.986+0000 7f5fa8ff9700 1 -- 172.21.15.42:0/2651065360 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5fa400d6a0 con 0x7f5fac125700 2019-12-05T05:31:04.000 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:03.986+0000 7f5fb222f700 1 -- 172.21.15.42:0/2651065360 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fac12cfe0 con 0x7f5fac125700 2019-12-05T05:31:04.000 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:03.986+0000 7f5fb0fcd700 1 --2- 172.21.15.42:0/2651065360 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5fac125700 0x7f5fac125b30 secure :-1 s=THROTTLE_DONE pgs=1023 cs=0 l=1 rx=0x7f5fac07aea0 tx=0x7f5fa4010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:04.000 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:03.986+0000 7f5fa8ff9700 1 -- 172.21.15.42:0/2651065360 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f5fa400d9b0 con 0x7f5fac125700 2019-12-05T05:31:04.000 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:03.986+0000 7f5fb0fcd700 1 --2- 172.21.15.42:0/2651065360 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5fac125700 0x7f5fac125b30 secure :-1 s=THROTTLE_DONE pgs=1023 cs=0 l=1 rx=0x7f5fac07aea0 tx=0x7f5fa4010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:04.000 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:03.986+0000 7f5fa8ff9700 1 --2- 172.21.15.42:0/2651065360 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5f9003ad70 0x7f5f9003d280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:31:04.000 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:03.986+0000 7f5fa8ff9700 1 -- 172.21.15.42:0/2651065360 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 7665+0+0 (secure 0 0 0) 0x7f5fa404ae80 con 0x7f5fac125700 2019-12-05T05:31:04.000 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:03.986+0000 7f5fabfff700 1 --2- 172.21.15.42:0/2651065360 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5f9003ad70 0x7f5f9003d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:31:04.001 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:03.990+0000 7f5fabfff700 1 --2- 172.21.15.42:0/2651065360 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5f9003ad70 0x7f5f9003d280 crc :-1 s=READY pgs=436 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:31:04.001 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:03.994+0000 7f5fb222f700 1 -- 172.21.15.42:0/2651065360 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fac05d100 con 0x7f5fac125700 2019-12-05T05:31:04.013 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:04.002+0000 7f5fb0fcd700 1 --2- 172.21.15.42:0/2651065360 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5fac125700 0x7f5fac125b30 secure :-1 s=THROTTLE_DONE pgs=1023 cs=0 l=1 rx=0x7f5fac07aea0 tx=0x7f5fa4010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:04.013 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:04.002+0000 7f5fa8ff9700 1 -- 172.21.15.42:0/2651065360 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f5fa4031f00 con 0x7f5fac125700 2019-12-05T05:31:04.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:04.182+0000 7f5fb222f700 1 -- 172.21.15.42:0/2651065360 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5fac1505c0 con 0x7f5fac125700 2019-12-05T05:31:04.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:04.182+0000 7f5fb0fcd700 1 --2- 172.21.15.42:0/2651065360 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5fac125700 0x7f5fac125b30 secure :-1 s=THROTTLE_DONE pgs=1023 cs=0 l=1 rx=0x7f5fac07aea0 tx=0x7f5fa4010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:04.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:04.182+0000 7f5fa8ff9700 1 -- 172.21.15.42:0/2651065360 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 127 v127) v1 ==== 95+0+1775 (secure 0 0 0) 0x7f5fa4032160 con 0x7f5fac125700 2019-12-05T05:31:04.190 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 127 2019-12-05T05:31:04.190 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:31:04.190 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":127,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":127,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:30:53.739046+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6778},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6778":{"gid":6778,"name":"a","rank":0,"incarnation":124,"state":"up:active","state_seq":23,"addr":"172.21.15.42:6835/3028851337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3028851337},{"type":"v1","addr":"172.21.15.42:6835","nonce":3028851337}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:31:04.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:04.182+0000 7f5fb222f700 1 -- 172.21.15.42:0/2651065360 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5f9003ad70 msgr2=0x7f5f9003d280 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:31:04.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:04.182+0000 7f5fb222f700 1 --2- 172.21.15.42:0/2651065360 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5f9003ad70 0x7f5f9003d280 crc :-1 s=READY pgs=436 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:31:04.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:04.182+0000 7f5fb222f700 1 -- 172.21.15.42:0/2651065360 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5fac125700 msgr2=0x7f5fac125b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:31:04.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:04.182+0000 7f5fb222f700 1 --2- 172.21.15.42:0/2651065360 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5fac125700 0x7f5fac125b30 secure :-1 s=READY pgs=1023 cs=0 l=1 rx=0x7f5fac07aea0 tx=0x7f5fa4010b50).stop 2019-12-05T05:31:04.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:04.182+0000 7f5fb222f700 1 -- 172.21.15.42:0/2651065360 shutdown_connections 2019-12-05T05:31:04.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:04.182+0000 7f5fb222f700 1 --2- 172.21.15.42:0/2651065360 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5f9003ad70 0x7f5f9003d280 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:31:04.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:04.182+0000 7f5fb222f700 1 --2- 172.21.15.42:0/2651065360 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5fac125700 0x7f5fac125b30 unknown :-1 s=CLOSED pgs=1023 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:31:04.193 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:04.182+0000 7f5fb222f700 1 --2- 172.21.15.42:0/2651065360 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5fac07a760 0x7f5fac1480b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:31:04.193 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:04.182+0000 7f5fb222f700 1 -- 172.21.15.42:0/2651065360 shutdown_connections 2019-12-05T05:31:04.193 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:04.182+0000 7f5fb222f700 1 -- 172.21.15.42:0/2651065360 wait complete. 2019-12-05T05:31:04.193 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:04.182+0000 7f5fb222f700 1 -- 172.21.15.42:0/2651065360 >> 172.21.15.42:0/2651065360 conn(0x7f5fac077e40 msgr2=0x7f5fac0a6840 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:31:04.205 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2019-12-05T05:31:06.206 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:31:06.317 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:06.306+0000 7fed203cf700 1 Processor -- start 2019-12-05T05:31:06.317 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:06.306+0000 7fed203cf700 1 -- start start 2019-12-05T05:31:06.317 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:06.306+0000 7fed203cf700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fed1808df80 0x7fed1809f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:31:06.317 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:06.306+0000 7fed203cf700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fed1809f830 0x7fed1814efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:31:06.318 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:06.306+0000 7fed203cf700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed1809fc60 con 0x7fed180a7740 2019-12-05T05:31:06.318 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:06.306+0000 7fed203cf700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed180a0b00 con 0x7fed1809f830 2019-12-05T05:31:06.318 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:06.306+0000 7fed203cf700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed1808e5b0 con 0x7fed1808df80 2019-12-05T05:31:06.318 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:06.306+0000 7fed1e96c700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fed180a7740 0x7fed1807e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:35530/0 (socket says 172.21.15.42:35530) 2019-12-05T05:31:06.318 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:06.306+0000 7fed1e96c700 1 -- 172.21.15.42:0/1653177911 learned_addr learned my addr 172.21.15.42:0/1653177911 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:31:06.318 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:06.306+0000 7fed1f16d700 1 --2- 172.21.15.42:0/1653177911 >> v2:172.21.15.42:3300/0 conn(0x7fed1809f830 0x7fed1814efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:31:06.319 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:06.306+0000 7fed1e16b700 1 --2- 172.21.15.42:0/1653177911 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fed1808df80 0x7fed1809f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:31:06.319 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:06.306+0000 7fed1d96a700 1 -- 172.21.15.42:0/1653177911 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1651018549 0 0) 0x7fed1809fc60 con 0x7fed180a7740 2019-12-05T05:31:06.319 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:06.306+0000 7fed1d96a700 1 -- 172.21.15.42:0/1653177911 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed040030b0 con 0x7fed180a7740 2019-12-05T05:31:06.319 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:06.306+0000 7fed1f16d700 1 -- 172.21.15.42:0/1653177911 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fed1808df80 msgr2=0x7fed1809f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:31:06.319 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:06.306+0000 7fed1f16d700 1 --2- 172.21.15.42:0/1653177911 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fed1808df80 0x7fed1809f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:31:06.319 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:06.306+0000 7fed1f16d700 1 -- 172.21.15.42:0/1653177911 >> v1:172.21.15.42:6789/0 conn(0x7fed180a7740 legacy=0x7fed1807e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:31:06.319 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:06.306+0000 7fed1f16d700 1 -- 172.21.15.42:0/1653177911 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed180a0710 con 0x7fed1809f830 2019-12-05T05:31:06.320 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:06.306+0000 7fed1f16d700 1 --2- 172.21.15.42:0/1653177911 >> v2:172.21.15.42:3300/0 conn(0x7fed1809f830 0x7fed1814efe0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fed10008b10 tx=0x7fed1000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:06.320 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:06.306+0000 7fed1f16d700 1 --2- 172.21.15.42:0/1653177911 >> v2:172.21.15.42:3300/0 conn(0x7fed1809f830 0x7fed1814efe0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fed10008b10 tx=0x7fed1000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:06.320 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:06.306+0000 7fed1f16d700 1 --2- 172.21.15.42:0/1653177911 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fed1809f830 0x7fed1814efe0 secure :-1 s=READY pgs=1025 cs=0 l=1 rx=0x7fed10008b10 tx=0x7fed1000fff0).ready entity=mon.1 client_cookie=50606cf113a029b5 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:31:06.320 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:06.306+0000 7fed1f16d700 1 --2- 172.21.15.42:0/1653177911 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fed1809f830 0x7fed1814efe0 secure :-1 s=THROTTLE_DONE pgs=1025 cs=0 l=1 rx=0x7fed10008b10 tx=0x7fed1000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:06.320 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:06.306+0000 7fed1d96a700 1 -- 172.21.15.42:0/1653177911 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fed10010900 con 0x7fed1809f830 2019-12-05T05:31:06.320 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:06.306+0000 7fed1f16d700 1 --2- 172.21.15.42:0/1653177911 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fed1809f830 0x7fed1814efe0 secure :-1 s=THROTTLE_DONE pgs=1025 cs=0 l=1 rx=0x7fed10008b10 tx=0x7fed1000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:06.321 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:06.306+0000 7fed1f16d700 1 --2- 172.21.15.42:0/1653177911 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fed1809f830 0x7fed1814efe0 secure :-1 s=THROTTLE_DONE pgs=1025 cs=0 l=1 rx=0x7fed10008b10 tx=0x7fed1000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:06.321 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:06.306+0000 7fed1d96a700 1 -- 172.21.15.42:0/1653177911 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed10008b50 con 0x7fed1809f830 2019-12-05T05:31:06.321 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:06.306+0000 7fed1d96a700 1 -- 172.21.15.42:0/1653177911 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fed100110c0 con 0x7fed1809f830 2019-12-05T05:31:06.321 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:06.306+0000 7fed203cf700 1 -- 172.21.15.42:0/1653177911 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fed1809f830 msgr2=0x7fed1814efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:31:06.321 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:06.306+0000 7fed203cf700 1 --2- 172.21.15.42:0/1653177911 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fed1809f830 0x7fed1814efe0 secure :-1 s=READY pgs=1025 cs=0 l=1 rx=0x7fed10008b10 tx=0x7fed1000fff0).stop 2019-12-05T05:31:06.322 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:06.306+0000 7fed203cf700 1 -- 172.21.15.42:0/1653177911 shutdown_connections 2019-12-05T05:31:06.322 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:06.306+0000 7fed203cf700 1 --2- 172.21.15.42:0/1653177911 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fed1809f830 0x7fed1814efe0 unknown :-1 s=CLOSED pgs=1025 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:31:06.322 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:06.306+0000 7fed203cf700 1 --2- 172.21.15.42:0/1653177911 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fed1808df80 0x7fed1809f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:31:06.322 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:06.306+0000 7fed203cf700 1 -- 172.21.15.42:0/1653177911 shutdown_connections 2019-12-05T05:31:06.322 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:06.306+0000 7fed203cf700 1 -- 172.21.15.42:0/1653177911 wait complete. 2019-12-05T05:31:06.322 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:06.306+0000 7fed203cf700 1 -- 172.21.15.42:0/1653177911 >> 172.21.15.42:0/1653177911 conn(0x7fed180aabe0 msgr2=0x7fed180a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:31:06.326 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:06.310+0000 7fed203cf700 1 Processor -- start 2019-12-05T05:31:06.326 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:06.310+0000 7fed203cf700 1 -- start start 2019-12-05T05:31:06.326 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:06.310+0000 7fed203cf700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fed18117dd0 0x7fed18145b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:31:06.327 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:06.310+0000 7fed203cf700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fed18116700 0x7fed18116b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:31:06.327 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:06.310+0000 7fed203cf700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed18127550 con 0x7fed1808eeb0 2019-12-05T05:31:06.328 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:06.310+0000 7fed203cf700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed18146450 con 0x7fed18117dd0 2019-12-05T05:31:06.328 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:06.310+0000 7fed203cf700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed181161c0 con 0x7fed18116700 2019-12-05T05:31:06.328 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:06.310+0000 7fed1e16b700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fed1808eeb0 0x7fed18126d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:35536/0 (socket says 172.21.15.42:35536) 2019-12-05T05:31:06.328 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:06.310+0000 7fed1e16b700 1 -- 172.21.15.42:0/1236212540 learned_addr learned my addr 172.21.15.42:0/1236212540 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:31:06.328 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:06.310+0000 7fed1f16d700 1 --2- 172.21.15.42:0/1236212540 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fed18116700 0x7fed18116b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:31:06.328 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:06.310+0000 7fed1e96c700 1 --2- 172.21.15.42:0/1236212540 >> v2:172.21.15.40:3300/0 conn(0x7fed18117dd0 0x7fed18145b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:31:06.328 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:06.310+0000 7fed0f7fe700 1 -- 172.21.15.42:0/1236212540 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2150941018 0 0) 0x7fed18127550 con 0x7fed1808eeb0 2019-12-05T05:31:06.329 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:06.314+0000 7fed0f7fe700 1 -- 172.21.15.42:0/1236212540 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed000030b0 con 0x7fed1808eeb0 2019-12-05T05:31:06.329 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:06.314+0000 7fed1f16d700 1 -- 172.21.15.42:0/1236212540 >> v2:172.21.15.40:3300/0 conn(0x7fed18117dd0 msgr2=0x7fed18145b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:31:06.329 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:06.314+0000 7fed1f16d700 1 --2- 172.21.15.42:0/1236212540 >> v2:172.21.15.40:3300/0 conn(0x7fed18117dd0 0x7fed18145b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:31:06.329 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:06.314+0000 7fed1f16d700 1 -- 172.21.15.42:0/1236212540 >> v1:172.21.15.42:6789/0 conn(0x7fed1808eeb0 legacy=0x7fed18126d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:31:06.329 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:06.314+0000 7fed1f16d700 1 -- 172.21.15.42:0/1236212540 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed181304c0 con 0x7fed18116700 2019-12-05T05:31:06.329 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:06.314+0000 7fed1f16d700 1 --2- 172.21.15.42:0/1236212540 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fed18116700 0x7fed18116b30 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fed10008b10 tx=0x7fed100087f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:06.329 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:06.314+0000 7fed1f16d700 1 --2- 172.21.15.42:0/1236212540 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fed18116700 0x7fed18116b30 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fed10008b10 tx=0x7fed100087f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:06.330 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:06.314+0000 7fed1f16d700 1 --2- 172.21.15.42:0/1236212540 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fed18116700 0x7fed18116b30 secure :-1 s=READY pgs=209 cs=0 l=1 rx=0x7fed10008b10 tx=0x7fed100087f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:31:06.330 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:06.314+0000 7fed1f16d700 1 --2- 172.21.15.42:0/1236212540 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fed18116700 0x7fed18116b30 secure :-1 s=THROTTLE_DONE pgs=209 cs=0 l=1 rx=0x7fed10008b10 tx=0x7fed100087f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:06.330 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:06.314+0000 7fed0f7fe700 1 -- 172.21.15.42:0/1236212540 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fed10005ab0 con 0x7fed18116700 2019-12-05T05:31:06.330 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:06.314+0000 7fed1f16d700 1 --2- 172.21.15.42:0/1236212540 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fed18116700 0x7fed18116b30 secure :-1 s=THROTTLE_DONE pgs=209 cs=0 l=1 rx=0x7fed10008b10 tx=0x7fed100087f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:06.330 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:06.314+0000 7fed203cf700 1 -- 172.21.15.42:0/1236212540 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed181460d0 con 0x7fed18116700 2019-12-05T05:31:06.330 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:06.314+0000 7fed0f7fe700 1 -- 172.21.15.42:0/1236212540 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed10010b00 con 0x7fed18116700 2019-12-05T05:31:06.330 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:06.314+0000 7fed1f16d700 1 --2- 172.21.15.42:0/1236212540 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fed18116700 0x7fed18116b30 secure :-1 s=THROTTLE_DONE pgs=209 cs=0 l=1 rx=0x7fed10008b10 tx=0x7fed100087f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:06.331 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:06.314+0000 7fed0f7fe700 1 -- 172.21.15.42:0/1236212540 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fed1000f470 con 0x7fed18116700 2019-12-05T05:31:06.331 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:06.314+0000 7fed203cf700 1 -- 172.21.15.42:0/1236212540 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed18137910 con 0x7fed18116700 2019-12-05T05:31:06.331 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:06.314+0000 7fed1f16d700 1 --2- 172.21.15.42:0/1236212540 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fed18116700 0x7fed18116b30 secure :-1 s=THROTTLE_DONE pgs=209 cs=0 l=1 rx=0x7fed10008b10 tx=0x7fed100087f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:06.331 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:06.314+0000 7fed0f7fe700 1 -- 172.21.15.42:0/1236212540 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fed1000f670 con 0x7fed18116700 2019-12-05T05:31:06.331 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:06.314+0000 7fed1f16d700 1 --2- 172.21.15.42:0/1236212540 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fed18116700 0x7fed18116b30 secure :-1 s=THROTTLE_DONE pgs=209 cs=0 l=1 rx=0x7fed10008b10 tx=0x7fed100087f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:06.331 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:06.314+0000 7fed0f7fe700 1 --2- 172.21.15.42:0/1236212540 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fed0003b2e0 0x7fed0003d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:31:06.331 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:06.314+0000 7fed0f7fe700 1 -- 172.21.15.42:0/1236212540 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 7665+0+0 (secure 0 0 0) 0x7fed1008f0a0 con 0x7fed18116700 2019-12-05T05:31:06.332 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:06.314+0000 7fed1e96c700 1 --2- 172.21.15.42:0/1236212540 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fed0003b2e0 0x7fed0003d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:31:06.332 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:06.318+0000 7fed1e96c700 1 --2- 172.21.15.42:0/1236212540 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fed0003b2e0 0x7fed0003d7f0 crc :-1 s=READY pgs=438 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:31:06.332 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:06.318+0000 7fed203cf700 1 -- 172.21.15.42:0/1236212540 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed1805d110 con 0x7fed18116700 2019-12-05T05:31:06.335 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:06.330+0000 7fed1f16d700 1 --2- 172.21.15.42:0/1236212540 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fed18116700 0x7fed18116b30 secure :-1 s=THROTTLE_DONE pgs=209 cs=0 l=1 rx=0x7fed10008b10 tx=0x7fed100087f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:06.336 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:06.330+0000 7fed0f7fe700 1 -- 172.21.15.42:0/1236212540 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fed10076100 con 0x7fed18116700 2019-12-05T05:31:06.504 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:06.498+0000 7fed203cf700 1 -- 172.21.15.42:0/1236212540 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fed1805b8c0 con 0x7fed18116700 2019-12-05T05:31:06.505 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:06.498+0000 7fed1f16d700 1 --2- 172.21.15.42:0/1236212540 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fed18116700 0x7fed18116b30 secure :-1 s=THROTTLE_DONE pgs=209 cs=0 l=1 rx=0x7fed10008b10 tx=0x7fed100087f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:06.505 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:06.498+0000 7fed0f7fe700 1 -- 172.21.15.42:0/1236212540 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 128 v128) v1 ==== 95+0+2112 (secure 0 0 0) 0x7fed1000f980 con 0x7fed18116700 2019-12-05T05:31:06.507 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 128 2019-12-05T05:31:06.507 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:31:06.507 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":128,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6984,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/3543033403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3543033403},{"type":"v1","addr":"172.21.15.40:6833","nonce":3543033403}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":128}],"filesystems":[{"mdsmap":{"epoch":127,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:30:53.739046+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6778},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6778":{"gid":6778,"name":"a","rank":0,"incarnation":124,"state":"up:active","state_seq":23,"addr":"172.21.15.42:6835/3028851337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3028851337},{"type":"v1","addr":"172.21.15.42:6835","nonce":3028851337}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:31:06.511 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:06.502+0000 7fed203cf700 1 -- 172.21.15.42:0/1236212540 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fed0003b2e0 msgr2=0x7fed0003d7f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:31:06.511 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:06.502+0000 7fed203cf700 1 --2- 172.21.15.42:0/1236212540 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fed0003b2e0 0x7fed0003d7f0 crc :-1 s=READY pgs=438 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:31:06.511 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:06.502+0000 7fed203cf700 1 -- 172.21.15.42:0/1236212540 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fed18116700 msgr2=0x7fed18116b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:31:06.511 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:06.502+0000 7fed203cf700 1 --2- 172.21.15.42:0/1236212540 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fed18116700 0x7fed18116b30 secure :-1 s=READY pgs=209 cs=0 l=1 rx=0x7fed10008b10 tx=0x7fed100087f0).stop 2019-12-05T05:31:06.512 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:06.502+0000 7fed203cf700 1 -- 172.21.15.42:0/1236212540 shutdown_connections 2019-12-05T05:31:06.512 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:06.502+0000 7fed203cf700 1 --2- 172.21.15.42:0/1236212540 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fed0003b2e0 0x7fed0003d7f0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:31:06.512 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:06.502+0000 7fed203cf700 1 --2- 172.21.15.42:0/1236212540 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fed18116700 0x7fed18116b30 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:31:06.512 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:06.502+0000 7fed203cf700 1 --2- 172.21.15.42:0/1236212540 >> v2:172.21.15.40:3300/0 conn(0x7fed18117dd0 0x7fed18145b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:31:06.512 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:06.502+0000 7fed203cf700 1 -- 172.21.15.42:0/1236212540 shutdown_connections 2019-12-05T05:31:06.512 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:06.502+0000 7fed203cf700 1 -- 172.21.15.42:0/1236212540 wait complete. 2019-12-05T05:31:06.513 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:06.502+0000 7fed203cf700 1 -- 172.21.15.42:0/1236212540 >> 172.21.15.42:0/1236212540 conn(0x7fed180aabe0 msgr2=0x7fed1811c470 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:31:06.525 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b reported in up:standby state 2019-12-05T05:31:06.526 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 26 secs before thrashing 2019-12-05T05:31:09.015 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:31:09.019 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:31:32.526 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:31:32.636 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:32.626+0000 7f2dc3259700 1 Processor -- start 2019-12-05T05:31:32.636 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:32.626+0000 7f2dc3259700 1 -- start start 2019-12-05T05:31:32.636 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:32.626+0000 7f2dc3259700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2dbc09fae0 0x7f2dbc14fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:31:32.636 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:32.626+0000 7f2dc3259700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dbc0a0710 con 0x7f2dbc0a7740 2019-12-05T05:31:32.636 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:32.626+0000 7f2dc3259700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dbc07ef20 con 0x7f2dbc08df80 2019-12-05T05:31:32.637 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:32.626+0000 7f2dc3259700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2dbc0a0b00 con 0x7f2dbc09fae0 2019-12-05T05:31:32.637 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:32.626+0000 7f2dc0ff5700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f2dbc08df80 0x7f2dbc09f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:35632/0 (socket says 172.21.15.42:35632) 2019-12-05T05:31:32.637 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:32.626+0000 7f2dc0ff5700 1 -- 172.21.15.42:0/617047548 learned_addr learned my addr 172.21.15.42:0/617047548 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:31:32.637 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:32.626+0000 7f2dc1ff7700 1 --2- 172.21.15.42:0/617047548 >> v2:172.21.15.40:3300/0 conn(0x7f2dbc09fae0 0x7f2dbc14fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:31:32.637 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:32.630+0000 7f2db3fff700 1 -- 172.21.15.42:0/617047548 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4178407582 0 0) 0x7f2dbc07ef20 con 0x7f2dbc08df80 2019-12-05T05:31:32.638 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:32.630+0000 7f2db3fff700 1 -- 172.21.15.42:0/617047548 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2da80030b0 con 0x7f2dbc08df80 2019-12-05T05:31:32.638 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:32.630+0000 7f2db3fff700 1 -- 172.21.15.42:0/617047548 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3233537741 0 0) 0x7f2dbc0a0710 con 0x7f2dbc0a7740 2019-12-05T05:31:32.638 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:32.630+0000 7f2db3fff700 1 -- 172.21.15.42:0/617047548 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2dbc07ef20 con 0x7f2dbc0a7740 2019-12-05T05:31:32.638 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:32.630+0000 7f2dc1ff7700 1 -- 172.21.15.42:0/617047548 >> v1:172.21.15.42:6789/0 conn(0x7f2dbc08df80 legacy=0x7f2dbc09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:31:32.638 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:32.630+0000 7f2dc1ff7700 1 -- 172.21.15.42:0/617047548 >> v1:172.21.15.40:6789/0 conn(0x7f2dbc0a7740 legacy=0x7f2dbc07e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:31:32.639 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:32.630+0000 7f2dc1ff7700 1 -- 172.21.15.42:0/617047548 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dbc0a9ba0 con 0x7f2dbc09fae0 2019-12-05T05:31:32.639 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:32.630+0000 7f2dc1ff7700 1 --2- 172.21.15.42:0/617047548 >> v2:172.21.15.40:3300/0 conn(0x7f2dbc09fae0 0x7f2dbc14fff0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f2db4008b10 tx=0x7f2db400ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:32.639 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:32.630+0000 7f2dc1ff7700 1 --2- 172.21.15.42:0/617047548 >> v2:172.21.15.40:3300/0 conn(0x7f2dbc09fae0 0x7f2dbc14fff0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f2db4008b10 tx=0x7f2db400ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:32.639 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:32.630+0000 7f2dc1ff7700 1 --2- 172.21.15.42:0/617047548 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2dbc09fae0 0x7f2dbc14fff0 secure :-1 s=READY pgs=798 cs=0 l=1 rx=0x7f2db4008b10 tx=0x7f2db400ff70).ready entity=mon.0 client_cookie=c5d4dad19b83d079 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:31:32.639 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:32.630+0000 7f2dc1ff7700 1 --2- 172.21.15.42:0/617047548 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2dbc09fae0 0x7f2dbc14fff0 secure :-1 s=THROTTLE_DONE pgs=798 cs=0 l=1 rx=0x7f2db4008b10 tx=0x7f2db400ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:32.639 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:32.630+0000 7f2dc1ff7700 1 --2- 172.21.15.42:0/617047548 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2dbc09fae0 0x7f2dbc14fff0 secure :-1 s=THROTTLE_DONE pgs=798 cs=0 l=1 rx=0x7f2db4008b10 tx=0x7f2db400ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:32.640 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:32.630+0000 7f2db3fff700 1 -- 172.21.15.42:0/617047548 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2db4010970 con 0x7f2dbc09fae0 2019-12-05T05:31:32.640 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:32.630+0000 7f2dc1ff7700 1 --2- 172.21.15.42:0/617047548 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2dbc09fae0 0x7f2dbc14fff0 secure :-1 s=THROTTLE_DONE pgs=798 cs=0 l=1 rx=0x7f2db4008b10 tx=0x7f2db400ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:32.640 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:32.630+0000 7f2db3fff700 1 -- 172.21.15.42:0/617047548 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2db4008b50 con 0x7f2dbc09fae0 2019-12-05T05:31:32.640 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:32.630+0000 7f2db3fff700 1 -- 172.21.15.42:0/617047548 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2db40110c0 con 0x7f2dbc09fae0 2019-12-05T05:31:32.640 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:32.630+0000 7f2dc3259700 1 -- 172.21.15.42:0/617047548 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2dbc09fae0 msgr2=0x7f2dbc14fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:31:32.640 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:32.630+0000 7f2dc3259700 1 --2- 172.21.15.42:0/617047548 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2dbc09fae0 0x7f2dbc14fff0 secure :-1 s=READY pgs=798 cs=0 l=1 rx=0x7f2db4008b10 tx=0x7f2db400ff70).stop 2019-12-05T05:31:32.641 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:32.630+0000 7f2dc3259700 1 -- 172.21.15.42:0/617047548 shutdown_connections 2019-12-05T05:31:32.641 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:32.630+0000 7f2dc3259700 1 --2- 172.21.15.42:0/617047548 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2dbc09fae0 0x7f2dbc14fff0 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:31:32.641 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:32.630+0000 7f2dc3259700 1 -- 172.21.15.42:0/617047548 shutdown_connections 2019-12-05T05:31:32.641 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:32.630+0000 7f2dc3259700 1 -- 172.21.15.42:0/617047548 wait complete. 2019-12-05T05:31:32.641 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:32.630+0000 7f2dc3259700 1 -- 172.21.15.42:0/617047548 >> 172.21.15.42:0/617047548 conn(0x7f2dbc0aabe0 msgr2=0x7f2dbc0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:31:32.641 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:32.634+0000 7f2dc3259700 1 Processor -- start 2019-12-05T05:31:32.642 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:32.634+0000 7f2dc3259700 1 -- start start 2019-12-05T05:31:32.645 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:32.634+0000 7f2dc3259700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2dbc119b40 0x7f2dbc064a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:31:32.646 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:32.634+0000 7f2dc3259700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2dbc064000 0x7f2dbc07d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:31:32.646 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:32.634+0000 7f2dc3259700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dbc065570 con 0x7f2dbc113da0 2019-12-05T05:31:32.646 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:32.634+0000 7f2dc3259700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2dbc065030 con 0x7f2dbc064000 2019-12-05T05:31:32.646 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:32.634+0000 7f2dc3259700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2dbc0652d0 con 0x7f2dbc119b40 2019-12-05T05:31:32.646 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:32.634+0000 7f2dc17f6700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2dbc119b40 0x7f2dbc064a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:31:32.647 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:32.634+0000 7f2dc17f6700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2dbc119b40 0x7f2dbc064a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:35596/0 (socket says 172.21.15.42:35596) 2019-12-05T05:31:32.647 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:32.634+0000 7f2dc17f6700 1 -- 172.21.15.42:0/3828025538 learned_addr learned my addr 172.21.15.42:0/3828025538 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:31:32.647 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:32.634+0000 7f2dc1ff7700 1 --2- 172.21.15.42:0/3828025538 >> v2:172.21.15.40:3300/0 conn(0x7f2dbc064000 0x7f2dbc07d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:31:32.647 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:32.634+0000 7f2dc17f6700 1 -- 172.21.15.42:0/3828025538 >> v2:172.21.15.40:3300/0 conn(0x7f2dbc064000 msgr2=0x7f2dbc07d390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:31:32.647 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:32.634+0000 7f2dc17f6700 1 --2- 172.21.15.42:0/3828025538 >> v2:172.21.15.40:3300/0 conn(0x7f2dbc064000 0x7f2dbc07d390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:31:32.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:32.634+0000 7f2dc17f6700 1 -- 172.21.15.42:0/3828025538 >> v1:172.21.15.42:6789/0 conn(0x7f2dbc113da0 legacy=0x7f2dbc063800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:31:32.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:32.634+0000 7f2db1ffb700 1 -- 172.21.15.42:0/3828025538 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3170645551 0 0) 0x7f2dbc065570 con 0x7f2dbc113da0 2019-12-05T05:31:32.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:32.634+0000 7f2dc17f6700 1 -- 172.21.15.42:0/3828025538 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dbc07d8e0 con 0x7f2dbc119b40 2019-12-05T05:31:32.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:32.634+0000 7f2dc17f6700 1 --2- 172.21.15.42:0/3828025538 >> v2:172.21.15.42:3300/0 conn(0x7f2dbc119b40 0x7f2dbc064a00 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f2db8000e90 tx=0x7f2db80109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:32.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:32.634+0000 7f2dc17f6700 1 --2- 172.21.15.42:0/3828025538 >> v2:172.21.15.42:3300/0 conn(0x7f2dbc119b40 0x7f2dbc064a00 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f2db8000e90 tx=0x7f2db80109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:32.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:32.634+0000 7f2dc17f6700 1 --2- 172.21.15.42:0/3828025538 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2dbc119b40 0x7f2dbc064a00 secure :-1 s=READY pgs=1029 cs=0 l=1 rx=0x7f2db8000e90 tx=0x7f2db80109a0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:31:32.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:32.634+0000 7f2dc17f6700 1 --2- 172.21.15.42:0/3828025538 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2dbc119b40 0x7f2dbc064a00 secure :-1 s=THROTTLE_DONE pgs=1029 cs=0 l=1 rx=0x7f2db8000e90 tx=0x7f2db80109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:32.650 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:32.634+0000 7f2db1ffb700 1 -- 172.21.15.42:0/3828025538 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2db800caa0 con 0x7f2dbc119b40 2019-12-05T05:31:32.650 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:32.634+0000 7f2dc17f6700 1 --2- 172.21.15.42:0/3828025538 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2dbc119b40 0x7f2dbc064a00 secure :-1 s=THROTTLE_DONE pgs=1029 cs=0 l=1 rx=0x7f2db8000e90 tx=0x7f2db80109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:32.650 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:32.634+0000 7f2dc17f6700 1 --2- 172.21.15.42:0/3828025538 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2dbc119b40 0x7f2dbc064a00 secure :-1 s=THROTTLE_DONE pgs=1029 cs=0 l=1 rx=0x7f2db8000e90 tx=0x7f2db80109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:32.650 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:32.634+0000 7f2db1ffb700 1 -- 172.21.15.42:0/3828025538 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2db800c6c0 con 0x7f2dbc119b40 2019-12-05T05:31:32.650 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:32.634+0000 7f2dc3259700 1 -- 172.21.15.42:0/3828025538 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2dbc07db50 con 0x7f2dbc119b40 2019-12-05T05:31:32.650 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:32.634+0000 7f2db1ffb700 1 -- 172.21.15.42:0/3828025538 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2db800d6a0 con 0x7f2dbc119b40 2019-12-05T05:31:32.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:32.634+0000 7f2dc3259700 1 -- 172.21.15.42:0/3828025538 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2dbc07e0d0 con 0x7f2dbc119b40 2019-12-05T05:31:32.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:32.638+0000 7f2dc17f6700 1 --2- 172.21.15.42:0/3828025538 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2dbc119b40 0x7f2dbc064a00 secure :-1 s=THROTTLE_DONE pgs=1029 cs=0 l=1 rx=0x7f2db8000e90 tx=0x7f2db80109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:32.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:32.638+0000 7f2db1ffb700 1 -- 172.21.15.42:0/3828025538 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f2db800d9b0 con 0x7f2dbc119b40 2019-12-05T05:31:32.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:32.638+0000 7f2dc17f6700 1 --2- 172.21.15.42:0/3828025538 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2dbc119b40 0x7f2dbc064a00 secure :-1 s=THROTTLE_DONE pgs=1029 cs=0 l=1 rx=0x7f2db8000e90 tx=0x7f2db80109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:32.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:32.638+0000 7f2db1ffb700 1 --2- 172.21.15.42:0/3828025538 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2d9c03acb0 0x7f2d9c03d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:31:32.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:32.638+0000 7f2db1ffb700 1 -- 172.21.15.42:0/3828025538 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 7665+0+0 (secure 0 0 0) 0x7f2db804b1d0 con 0x7f2dbc119b40 2019-12-05T05:31:32.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:32.638+0000 7f2dc0ff5700 1 --2- 172.21.15.42:0/3828025538 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2d9c03acb0 0x7f2d9c03d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:31:32.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:32.638+0000 7f2dc0ff5700 1 --2- 172.21.15.42:0/3828025538 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2d9c03acb0 0x7f2d9c03d1c0 crc :-1 s=READY pgs=439 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:31:32.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:32.642+0000 7f2dc3259700 1 -- 172.21.15.42:0/3828025538 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2dbc05d110 con 0x7f2dbc119b40 2019-12-05T05:31:32.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:32.646+0000 7f2dc17f6700 1 --2- 172.21.15.42:0/3828025538 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2dbc119b40 0x7f2dbc064a00 secure :-1 s=THROTTLE_DONE pgs=1029 cs=0 l=1 rx=0x7f2db8000e90 tx=0x7f2db80109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:32.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:32.646+0000 7f2db1ffb700 1 -- 172.21.15.42:0/3828025538 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f2db8032250 con 0x7f2dbc119b40 2019-12-05T05:31:32.816 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:32.814+0000 7f2dc3259700 1 -- 172.21.15.42:0/3828025538 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2dbc05b8c0 con 0x7f2dbc119b40 2019-12-05T05:31:32.817 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:32.814+0000 7f2dc17f6700 1 --2- 172.21.15.42:0/3828025538 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2dbc119b40 0x7f2dbc064a00 secure :-1 s=THROTTLE_DONE pgs=1029 cs=0 l=1 rx=0x7f2db8000e90 tx=0x7f2db80109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:32.817 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:32.814+0000 7f2db1ffb700 1 -- 172.21.15.42:0/3828025538 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 128 v128) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f2db80324b0 con 0x7f2dbc119b40 2019-12-05T05:31:32.817 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 128 2019-12-05T05:31:32.817 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:31:32.818 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":128,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6984,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/3543033403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3543033403},{"type":"v1","addr":"172.21.15.40:6833","nonce":3543033403}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":128}],"filesystems":[{"mdsmap":{"epoch":127,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:30:53.739046+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6778},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6778":{"gid":6778,"name":"a","rank":0,"incarnation":124,"state":"up:active","state_seq":23,"addr":"172.21.15.42:6835/3028851337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3028851337},{"type":"v1","addr":"172.21.15.42:6835","nonce":3028851337}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:31:32.819 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:32.814+0000 7f2dc3259700 1 -- 172.21.15.42:0/3828025538 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2d9c03acb0 msgr2=0x7f2d9c03d1c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:31:32.820 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:32.814+0000 7f2dc3259700 1 --2- 172.21.15.42:0/3828025538 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2d9c03acb0 0x7f2d9c03d1c0 crc :-1 s=READY pgs=439 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:31:32.820 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:32.814+0000 7f2dc3259700 1 -- 172.21.15.42:0/3828025538 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2dbc119b40 msgr2=0x7f2dbc064a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:31:32.820 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:32.814+0000 7f2dc3259700 1 --2- 172.21.15.42:0/3828025538 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2dbc119b40 0x7f2dbc064a00 secure :-1 s=READY pgs=1029 cs=0 l=1 rx=0x7f2db8000e90 tx=0x7f2db80109a0).stop 2019-12-05T05:31:32.820 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:32.814+0000 7f2dc3259700 1 -- 172.21.15.42:0/3828025538 shutdown_connections 2019-12-05T05:31:32.820 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:32.814+0000 7f2dc3259700 1 --2- 172.21.15.42:0/3828025538 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2d9c03acb0 0x7f2d9c03d1c0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:31:32.820 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:32.814+0000 7f2dc3259700 1 --2- 172.21.15.42:0/3828025538 >> v2:172.21.15.40:3300/0 conn(0x7f2dbc064000 0x7f2dbc07d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:31:32.821 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:32.814+0000 7f2dc3259700 1 --2- 172.21.15.42:0/3828025538 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2dbc119b40 0x7f2dbc064a00 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:31:32.821 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:32.818+0000 7f2dc3259700 1 -- 172.21.15.42:0/3828025538 shutdown_connections 2019-12-05T05:31:32.821 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:32.818+0000 7f2dc3259700 1 -- 172.21.15.42:0/3828025538 wait complete. 2019-12-05T05:31:32.821 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:32.818+0000 7f2dc3259700 1 -- 172.21.15.42:0/3828025538 >> 172.21.15.42:0/3828025538 conn(0x7f2dbc0a7740 msgr2=0x7f2dbc09f460 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:31:32.829 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.a (rank=0) 2019-12-05T05:31:32.830 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2019-12-05T05:31:32.830 INFO:teuthology.orchestra.run:waiting for 300 2019-12-05T05:31:32.843 INFO:tasks.ceph.mds.a:Stopped 2019-12-05T05:31:32.843 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:31:33.010 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:33.002+0000 7f3bde5f5700 1 Processor -- start 2019-12-05T05:31:33.010 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:33.002+0000 7f3bde5f5700 1 -- start start 2019-12-05T05:31:33.010 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:33.002+0000 7f3bde5f5700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3bd80a7740 0x7f3bd808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:31:33.011 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:33.002+0000 7f3bde5f5700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f3bd809fae0 0x7f3bd807eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:31:33.011 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:33.002+0000 7f3bde5f5700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bd80a0710 con 0x7f3bd807e720 2019-12-05T05:31:33.011 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:33.002+0000 7f3bde5f5700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3bd80a0b00 con 0x7f3bd809fae0 2019-12-05T05:31:33.011 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:33.002+0000 7f3bde5f5700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3bd808e5b0 con 0x7f3bd80a7740 2019-12-05T05:31:33.011 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:33.002+0000 7f3bdcb92700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3bd80a7740 0x7f3bd808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:31:33.011 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:33.002+0000 7f3bdcb92700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3bd80a7740 0x7f3bd808df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:35608/0 (socket says 172.21.15.42:35608) 2019-12-05T05:31:33.012 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:33.002+0000 7f3bdcb92700 1 -- 172.21.15.42:0/130981425 learned_addr learned my addr 172.21.15.42:0/130981425 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:31:33.012 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:33.002+0000 7f3bdd393700 1 --2- 172.21.15.42:0/130981425 >> v2:172.21.15.40:3300/0 conn(0x7f3bd809fae0 0x7f3bd807eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:31:33.012 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:33.002+0000 7f3bdcb92700 1 -- 172.21.15.42:0/130981425 >> v2:172.21.15.40:3300/0 conn(0x7f3bd809fae0 msgr2=0x7f3bd807eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:31:33.012 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:33.002+0000 7f3bdcb92700 1 --2- 172.21.15.42:0/130981425 >> v2:172.21.15.40:3300/0 conn(0x7f3bd809fae0 0x7f3bd807eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:31:33.012 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:33.002+0000 7f3bd77fe700 1 -- 172.21.15.42:0/130981425 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1791202149 0 0) 0x7f3bd80a0710 con 0x7f3bd807e720 2019-12-05T05:31:33.013 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:33.002+0000 7f3bdcb92700 1 -- 172.21.15.42:0/130981425 >> v1:172.21.15.40:6789/0 conn(0x7f3bd807e720 legacy=0x7f3bd809f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:31:33.013 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:33.002+0000 7f3bdcb92700 1 -- 172.21.15.42:0/130981425 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bd807f0a0 con 0x7f3bd80a7740 2019-12-05T05:31:33.013 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:33.002+0000 7f3bdcb92700 1 --2- 172.21.15.42:0/130981425 >> v2:172.21.15.42:3300/0 conn(0x7f3bd80a7740 0x7f3bd808df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f3bc4002f30 tx=0x7f3bc400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:33.013 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:33.002+0000 7f3bdcb92700 1 --2- 172.21.15.42:0/130981425 >> v2:172.21.15.42:3300/0 conn(0x7f3bd80a7740 0x7f3bd808df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f3bc4002f30 tx=0x7f3bc400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:33.013 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:33.002+0000 7f3bdcb92700 1 --2- 172.21.15.42:0/130981425 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3bd80a7740 0x7f3bd808df80 secure :-1 s=READY pgs=1030 cs=0 l=1 rx=0x7f3bc4002f30 tx=0x7f3bc400fd30).ready entity=mon.1 client_cookie=75bab6e8939399b6 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:31:33.013 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:33.002+0000 7f3bdcb92700 1 --2- 172.21.15.42:0/130981425 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3bd80a7740 0x7f3bd808df80 secure :-1 s=THROTTLE_DONE pgs=1030 cs=0 l=1 rx=0x7f3bc4002f30 tx=0x7f3bc400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:33.014 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:33.002+0000 7f3bd77fe700 1 -- 172.21.15.42:0/130981425 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3bc4008c90 con 0x7f3bd80a7740 2019-12-05T05:31:33.014 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:33.002+0000 7f3bdcb92700 1 --2- 172.21.15.42:0/130981425 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3bd80a7740 0x7f3bd808df80 secure :-1 s=THROTTLE_DONE pgs=1030 cs=0 l=1 rx=0x7f3bc4002f30 tx=0x7f3bc400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:33.014 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:33.002+0000 7f3bd77fe700 1 -- 172.21.15.42:0/130981425 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bc40088b0 con 0x7f3bd80a7740 2019-12-05T05:31:33.014 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:33.002+0000 7f3bdcb92700 1 --2- 172.21.15.42:0/130981425 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3bd80a7740 0x7f3bd808df80 secure :-1 s=THROTTLE_DONE pgs=1030 cs=0 l=1 rx=0x7f3bc4002f30 tx=0x7f3bc400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:33.015 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:33.002+0000 7f3bde5f5700 1 -- 172.21.15.42:0/130981425 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3bd80a7740 msgr2=0x7f3bd808df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:31:33.015 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:33.002+0000 7f3bde5f5700 1 --2- 172.21.15.42:0/130981425 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3bd80a7740 0x7f3bd808df80 secure :-1 s=READY pgs=1030 cs=0 l=1 rx=0x7f3bc4002f30 tx=0x7f3bc400fd30).stop 2019-12-05T05:31:33.015 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:33.002+0000 7f3bd77fe700 1 -- 172.21.15.42:0/130981425 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3bc4011ad0 con 0x7f3bd80a7740 2019-12-05T05:31:33.015 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:33.006+0000 7f3bde5f5700 1 -- 172.21.15.42:0/130981425 shutdown_connections 2019-12-05T05:31:33.015 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:33.006+0000 7f3bde5f5700 1 --2- 172.21.15.42:0/130981425 >> v2:172.21.15.40:3300/0 conn(0x7f3bd809fae0 0x7f3bd807eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:31:33.015 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:33.006+0000 7f3bde5f5700 1 --2- 172.21.15.42:0/130981425 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3bd80a7740 0x7f3bd808df80 unknown :-1 s=CLOSED pgs=1030 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:31:33.015 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:33.006+0000 7f3bde5f5700 1 -- 172.21.15.42:0/130981425 shutdown_connections 2019-12-05T05:31:33.016 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:33.006+0000 7f3bde5f5700 1 -- 172.21.15.42:0/130981425 wait complete. 2019-12-05T05:31:33.016 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:33.006+0000 7f3bde5f5700 1 -- 172.21.15.42:0/130981425 >> 172.21.15.42:0/130981425 conn(0x7f3bd80aabe0 msgr2=0x7f3bd80a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:31:33.016 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:33.006+0000 7f3bde5f5700 1 Processor -- start 2019-12-05T05:31:33.016 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:33.006+0000 7f3bde5f5700 1 -- start start 2019-12-05T05:31:33.025 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:33.010+0000 7f3bde5f5700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3bd81145b0 0x7f3bd80a22d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:31:33.025 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:33.010+0000 7f3bde5f5700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3bd8123ee0 0x7f3bd812e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:31:33.025 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:33.010+0000 7f3bde5f5700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bd81149e0 con 0x7f3bd81236d0 2019-12-05T05:31:33.025 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:33.010+0000 7f3bde5f5700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3bd81150d0 con 0x7f3bd81145b0 2019-12-05T05:31:33.026 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:33.010+0000 7f3bde5f5700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bd8115370 con 0x7f3bd8123ee0 2019-12-05T05:31:33.026 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:33.010+0000 7f3bd7fff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3bd81145b0 0x7f3bd80a22d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:31:33.026 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:33.010+0000 7f3bd7fff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3bd81145b0 0x7f3bd80a22d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:35614/0 (socket says 172.21.15.42:35614) 2019-12-05T05:31:33.026 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:33.010+0000 7f3bdcb92700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f3bd81236d0 0x7f3bd8115610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:45090/0 (socket says 172.21.15.42:45090) 2019-12-05T05:31:33.026 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:33.010+0000 7f3bd7fff700 1 -- 172.21.15.42:0/2277756396 learned_addr learned my addr 172.21.15.42:0/2277756396 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:31:33.026 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:33.010+0000 7f3bdd393700 1 --2- 172.21.15.42:0/2277756396 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3bd8123ee0 0x7f3bd812e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:31:33.027 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:33.010+0000 7f3bd7fff700 1 -- 172.21.15.42:0/2277756396 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3bd8123ee0 msgr2=0x7f3bd812e0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:31:33.027 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:33.010+0000 7f3bd7fff700 1 --2- 172.21.15.42:0/2277756396 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3bd8123ee0 0x7f3bd812e0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:31:33.027 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:33.010+0000 7f3bd7fff700 1 -- 172.21.15.42:0/2277756396 >> v1:172.21.15.40:6789/0 conn(0x7f3bd81236d0 legacy=0x7f3bd8115610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:31:33.027 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:33.010+0000 7f3bd57fa700 1 -- 172.21.15.42:0/2277756396 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3712351377 0 0) 0x7f3bd81149e0 con 0x7f3bd81236d0 2019-12-05T05:31:33.028 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:33.010+0000 7f3bd7fff700 1 -- 172.21.15.42:0/2277756396 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bd80a2820 con 0x7f3bd81145b0 2019-12-05T05:31:33.028 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:33.010+0000 7f3bd7fff700 1 --2- 172.21.15.42:0/2277756396 >> v2:172.21.15.42:3300/0 conn(0x7f3bd81145b0 0x7f3bd80a22d0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f3bc8001460 tx=0x7f3bc80109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:33.028 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:33.010+0000 7f3bd7fff700 1 --2- 172.21.15.42:0/2277756396 >> v2:172.21.15.42:3300/0 conn(0x7f3bd81145b0 0x7f3bd80a22d0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f3bc8001460 tx=0x7f3bc80109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:33.028 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:33.010+0000 7f3bd7fff700 1 --2- 172.21.15.42:0/2277756396 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3bd81145b0 0x7f3bd80a22d0 secure :-1 s=READY pgs=1031 cs=0 l=1 rx=0x7f3bc8001460 tx=0x7f3bc80109a0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:31:33.028 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:33.010+0000 7f3bd7fff700 1 -- 172.21.15.42:0/2277756396 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3bd81145b0 msgr2=0x7f3bd80a22d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2019-12-05T05:31:33.028 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:33.010+0000 7f3bd7fff700 1 -- 172.21.15.42:0/2277756396 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3bd81145b0 msgr2=0x7f3bd80a22d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2019-12-05T05:31:33.029 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:33.010+0000 7f3bd7fff700 1 --2- 172.21.15.42:0/2277756396 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3bd81145b0 0x7f3bd80a22d0 secure :-1 s=READY pgs=1031 cs=0 l=1 rx=0x7f3bc8001460 tx=0x7f3bc80109a0).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-12-05T05:31:33.029 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:33.010+0000 7f3bd7fff700 1 --2- 172.21.15.42:0/2277756396 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3bd81145b0 0x7f3bd80a22d0 secure :-1 s=READY pgs=1031 cs=0 l=1 rx=0x7f3bc8001460 tx=0x7f3bc80109a0).stop 2019-12-05T05:31:33.029 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:33.010+0000 7f3bd57fa700 1 -- 172.21.15.42:0/2277756396 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3bd81145b0 msgr2=0x7f3bd80a22d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2019-12-05T05:31:33.029 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:33.010+0000 7f3bd57fa700 1 --2- 172.21.15.42:0/2277756396 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3bd81145b0 0x7f3bd80a22d0 unknown :-1 s=CLOSED pgs=1031 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:31:33.029 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:33.010+0000 7f3bd57fa700 1 --2- 172.21.15.42:0/2277756396 >> v2:172.21.15.40:3300/0 conn(0x7f3bb4003050 0x7f3bb4005520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:31:33.029 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:33.010+0000 7f3bd57fa700 1 -- 172.21.15.42:0/2277756396 --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bd81149e0 con 0x7f3bb4005c10 2019-12-05T05:31:33.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:33.010+0000 7f3bd57fa700 1 -- 172.21.15.42:0/2277756396 --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3bb400aba0 con 0x7f3bb4003050 2019-12-05T05:31:33.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:33.010+0000 7f3bdcb92700 1 --2- 172.21.15.42:0/2277756396 >> v2:172.21.15.40:3300/0 conn(0x7f3bb4003050 0x7f3bb4005520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:31:33.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:33.010+0000 7f3bd57fa700 1 -- 172.21.15.42:0/2277756396 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3610741407 0 0) 0x7f3bd81149e0 con 0x7f3bb4005c10 2019-12-05T05:31:33.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:33.010+0000 7f3bd57fa700 1 -- 172.21.15.42:0/2277756396 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3bb4009be0 con 0x7f3bb4005c10 2019-12-05T05:31:33.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:33.010+0000 7f3bdcb92700 1 -- 172.21.15.42:0/2277756396 >> v1:172.21.15.42:6789/0 conn(0x7f3bb4005c10 legacy=0x7f3bb4008120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:31:33.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:33.010+0000 7f3bdcb92700 1 -- 172.21.15.42:0/2277756396 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bb400adf0 con 0x7f3bb4003050 2019-12-05T05:31:33.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:33.010+0000 7f3bdcb92700 1 --2- 172.21.15.42:0/2277756396 >> v2:172.21.15.40:3300/0 conn(0x7f3bb4003050 0x7f3bb4005520 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f3bc4007d80 tx=0x7f3bc40118e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:33.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:33.010+0000 7f3bdcb92700 1 --2- 172.21.15.42:0/2277756396 >> v2:172.21.15.40:3300/0 conn(0x7f3bb4003050 0x7f3bb4005520 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f3bc4007d80 tx=0x7f3bc40118e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:33.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:33.010+0000 7f3bdcb92700 1 --2- 172.21.15.42:0/2277756396 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3bb4003050 0x7f3bb4005520 secure :-1 s=READY pgs=801 cs=0 l=1 rx=0x7f3bc4007d80 tx=0x7f3bc40118e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:31:33.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:33.010+0000 7f3bdcb92700 1 -- 172.21.15.42:0/2277756396 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3bb4003050 msgr2=0x7f3bb4005520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2019-12-05T05:31:33.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:33.010+0000 7f3bdcb92700 1 -- 172.21.15.42:0/2277756396 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3bb4003050 msgr2=0x7f3bb4005520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2019-12-05T05:31:33.032 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:33.010+0000 7f3bdcb92700 1 --2- 172.21.15.42:0/2277756396 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3bb4003050 0x7f3bb4005520 secure :-1 s=READY pgs=801 cs=0 l=1 rx=0x7f3bc4007d80 tx=0x7f3bc40118e0).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-12-05T05:31:33.032 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:33.010+0000 7f3bdcb92700 1 --2- 172.21.15.42:0/2277756396 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3bb4003050 0x7f3bb4005520 secure :-1 s=READY pgs=801 cs=0 l=1 rx=0x7f3bc4007d80 tx=0x7f3bc40118e0).stop 2019-12-05T05:31:33.032 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:33.010+0000 7f3bdd393700 1 -- 172.21.15.42:0/2277756396 reap_dead start 2019-12-05T05:31:33.032 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:33.010+0000 7f3bd57fa700 1 -- 172.21.15.42:0/2277756396 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3bb4003050 msgr2=0x7f3bb4005520 unknown :-1 s=STATE_CLOSED l=1).mark_down 2019-12-05T05:31:33.032 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:33.010+0000 7f3bd57fa700 1 --2- 172.21.15.42:0/2277756396 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3bb4003050 0x7f3bb4005520 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:31:33.032 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:33.010+0000 7f3bd57fa700 1 -- 172.21.15.42:0/2277756396 --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bd81149e0 con 0x7f3bb4005c10 2019-12-05T05:31:33.033 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:33.010+0000 7f3bd57fa700 1 -- 172.21.15.42:0/2277756396 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bb400a1c0 con 0x7f3bb4003050 2019-12-05T05:31:33.033 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:33.010+0000 7f3bd57fa700 1 -- 172.21.15.42:0/2277756396 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bb400a1c0 con 0x7f3bd81145b0 2019-12-05T05:31:33.033 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:33.014+0000 7f3bd57fa700 1 -- 172.21.15.42:0/2277756396 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2016183477 0 0) 0x7f3bd81149e0 con 0x7f3bb4005c10 2019-12-05T05:31:33.033 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:33.014+0000 7f3bd57fa700 1 -- 172.21.15.42:0/2277756396 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3bb400a380 con 0x7f3bb4005c10 2019-12-05T05:31:33.033 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:33.014+0000 7f3bd57fa700 1 -- 172.21.15.42:0/2277756396 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 3984591790 0 0) 0x7f3bb400a380 con 0x7f3bb4005c10 2019-12-05T05:31:33.033 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:33.014+0000 7f3bd57fa700 1 -- 172.21.15.42:0/2277756396 --> v1:172.21.15.42:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3bd81149e0 con 0x7f3bb4005c10 2019-12-05T05:31:33.033 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:33.014+0000 7f3bd57fa700 1 -- 172.21.15.42:0/2277756396 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7f3bc4005540 con 0x7f3bb4005c10 2019-12-05T05:31:33.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:33.014+0000 7f3bd57fa700 1 -- 172.21.15.42:0/2277756396 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3bd81145b0 msgr2=0x7f3bd80a22d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2019-12-05T05:31:33.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:33.014+0000 7f3bd57fa700 1 --2- 172.21.15.42:0/2277756396 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3bd81145b0 0x7f3bd80a22d0 unknown :-1 s=CLOSED pgs=1031 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:31:33.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:33.014+0000 7f3bd57fa700 1 -- 172.21.15.42:0/2277756396 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3bb4003050 msgr2=0x7f3bb4005520 unknown :-1 s=STATE_CLOSED l=1).mark_down 2019-12-05T05:31:33.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:33.014+0000 7f3bd57fa700 1 --2- 172.21.15.42:0/2277756396 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3bb4003050 0x7f3bb4005520 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:31:33.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:33.014+0000 7f3bd57fa700 1 -- 172.21.15.42:0/2277756396 >> v1:172.21.15.42:6789/0 conn(0x7f3bb4005c10 legacy=0x7f3bb40080a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:31:33.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:33.014+0000 7f3bd57fa700 1 --2- 172.21.15.42:0/2277756396 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3bb400b0c0 0x7f3bb400d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:31:33.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:33.014+0000 7f3bd57fa700 1 --2- 172.21.15.42:0/2277756396 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3bb400da50 0x7f3bb400fe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:31:33.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:33.014+0000 7f3bd57fa700 1 --2- 172.21.15.42:0/2277756396 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3bb4010430 0x7f3bb40128f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:31:33.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:33.014+0000 7f3bd57fa700 1 -- 172.21.15.42:0/2277756396 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bb400a1c0 con 0x7f3bb400da50 2019-12-05T05:31:33.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:33.014+0000 7f3bd57fa700 1 -- 172.21.15.42:0/2277756396 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bb4012f90 con 0x7f3bb4010430 2019-12-05T05:31:33.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:33.014+0000 7f3bd57fa700 1 -- 172.21.15.42:0/2277756396 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bb40131d0 con 0x7f3bb400b0c0 2019-12-05T05:31:33.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:33.014+0000 7f3bdd393700 1 --2- 172.21.15.42:0/2277756396 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3bb4010430 0x7f3bb40128f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:31:33.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:33.014+0000 7f3bdcb92700 1 --2- 172.21.15.42:0/2277756396 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3bb400b0c0 0x7f3bb400d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:31:33.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:33.014+0000 7f3bd7fff700 1 --2- 172.21.15.42:0/2277756396 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3bb400da50 0x7f3bb400fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:31:33.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:33.014+0000 7f3bdcb92700 1 -- 172.21.15.42:0/2277756396 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3bb4010430 msgr2=0x7f3bb40128f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:31:33.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:33.014+0000 7f3bdcb92700 1 --2- 172.21.15.42:0/2277756396 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3bb4010430 0x7f3bb40128f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:31:33.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:33.014+0000 7f3bdcb92700 1 -- 172.21.15.42:0/2277756396 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3bb400da50 msgr2=0x7f3bb400fe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:31:33.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:33.014+0000 7f3bdcb92700 1 --2- 172.21.15.42:0/2277756396 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3bb400da50 0x7f3bb400fe90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:31:33.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:33.014+0000 7f3bdcb92700 1 -- 172.21.15.42:0/2277756396 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bc800b890 con 0x7f3bb400b0c0 2019-12-05T05:31:33.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:33.014+0000 7f3bde5f5700 1 -- 172.21.15.42:0/2277756396 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bd81236d0 con 0x7f3bb400b0c0 2019-12-05T05:31:33.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:33.014+0000 7f3bdcb92700 1 --2- 172.21.15.42:0/2277756396 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3bb400b0c0 0x7f3bb400d500 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f3bc40078c0 tx=0x7f3bc4015520).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:33.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:33.014+0000 7f3bde5f5700 1 -- 172.21.15.42:0/2277756396 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bd8115610 con 0x7f3bb400b0c0 2019-12-05T05:31:33.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:33.014+0000 7f3bdcb92700 1 --2- 172.21.15.42:0/2277756396 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3bb400b0c0 0x7f3bb400d500 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f3bc40078c0 tx=0x7f3bc4015520).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:33.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:33.014+0000 7f3bdcb92700 1 --2- 172.21.15.42:0/2277756396 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3bb400b0c0 0x7f3bb400d500 secure :-1 s=READY pgs=210 cs=0 l=1 rx=0x7f3bc40078c0 tx=0x7f3bc4015520).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:31:33.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:33.014+0000 7f3bdcb92700 1 --2- 172.21.15.42:0/2277756396 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3bb400b0c0 0x7f3bb400d500 secure :-1 s=THROTTLE_DONE pgs=210 cs=0 l=1 rx=0x7f3bc40078c0 tx=0x7f3bc4015520).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:33.039 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:33.014+0000 7f3bd57fa700 1 -- 172.21.15.42:0/2277756396 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3bc4015cd0 con 0x7f3bb400b0c0 2019-12-05T05:31:33.039 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:33.014+0000 7f3bdcb92700 1 --2- 172.21.15.42:0/2277756396 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3bb400b0c0 0x7f3bb400d500 secure :-1 s=THROTTLE_DONE pgs=210 cs=0 l=1 rx=0x7f3bc40078c0 tx=0x7f3bc4015520).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:33.039 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:33.014+0000 7f3bd57fa700 1 -- 172.21.15.42:0/2277756396 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bc4015180 con 0x7f3bb400b0c0 2019-12-05T05:31:33.039 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:33.014+0000 7f3bdcb92700 1 --2- 172.21.15.42:0/2277756396 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3bb400b0c0 0x7f3bb400d500 secure :-1 s=THROTTLE_DONE pgs=210 cs=0 l=1 rx=0x7f3bc40078c0 tx=0x7f3bc4015520).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:33.039 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:33.014+0000 7f3bd57fa700 1 -- 172.21.15.42:0/2277756396 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3bc4013860 con 0x7f3bb400b0c0 2019-12-05T05:31:33.039 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:33.014+0000 7f3bdcb92700 1 --2- 172.21.15.42:0/2277756396 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3bb400b0c0 0x7f3bb400d500 secure :-1 s=THROTTLE_DONE pgs=210 cs=0 l=1 rx=0x7f3bc40078c0 tx=0x7f3bc4015520).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:33.040 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:33.014+0000 7f3bd57fa700 1 -- 172.21.15.42:0/2277756396 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f3bc4013a60 con 0x7f3bb400b0c0 2019-12-05T05:31:33.040 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:33.014+0000 7f3bdcb92700 1 --2- 172.21.15.42:0/2277756396 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3bb400b0c0 0x7f3bb400d500 secure :-1 s=THROTTLE_DONE pgs=210 cs=0 l=1 rx=0x7f3bc40078c0 tx=0x7f3bc4015520).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:33.040 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:33.018+0000 7f3bd57fa700 1 --2- 172.21.15.42:0/2277756396 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3bb404ae50 0x7f3bb404d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:31:33.040 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:33.018+0000 7f3bd57fa700 1 -- 172.21.15.42:0/2277756396 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 7665+0+0 (secure 0 0 0) 0x7f3bc404d840 con 0x7f3bb400b0c0 2019-12-05T05:31:33.040 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:33.018+0000 7f3bd7fff700 1 --2- 172.21.15.42:0/2277756396 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3bb404ae50 0x7f3bb404d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:31:33.041 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:33.018+0000 7f3bde5f5700 1 -- 172.21.15.42:0/2277756396 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bb8003200 con 0x7f3bb400b0c0 2019-12-05T05:31:33.041 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:33.018+0000 7f3bd7fff700 1 --2- 172.21.15.42:0/2277756396 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3bb404ae50 0x7f3bb404d360 crc :-1 s=READY pgs=440 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:31:33.042 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:33.026+0000 7f3bdcb92700 1 --2- 172.21.15.42:0/2277756396 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3bb400b0c0 0x7f3bb400d500 secure :-1 s=THROTTLE_DONE pgs=210 cs=0 l=1 rx=0x7f3bc40078c0 tx=0x7f3bc4015520).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:33.042 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:33.026+0000 7f3bd57fa700 1 -- 172.21.15.42:0/2277756396 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f3bc40348a0 con 0x7f3bb400b0c0 2019-12-05T05:31:33.200 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:33.198+0000 7f3bde5f5700 1 -- 172.21.15.42:0/2277756396 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3bb8003630 con 0x7f3bb400b0c0 2019-12-05T05:31:33.201 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:33.198+0000 7f3bdcb92700 1 --2- 172.21.15.42:0/2277756396 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3bb400b0c0 0x7f3bb400d500 secure :-1 s=THROTTLE_DONE pgs=210 cs=0 l=1 rx=0x7f3bc40078c0 tx=0x7f3bc4015520).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:33.201 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:33.198+0000 7f3bd57fa700 1 -- 172.21.15.42:0/2277756396 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 128 v128) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f3bc4033dc0 con 0x7f3bb400b0c0 2019-12-05T05:31:33.204 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:31:33.204 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":128,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6984,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/3543033403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3543033403},{"type":"v1","addr":"172.21.15.40:6833","nonce":3543033403}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":128}],"filesystems":[{"mdsmap":{"epoch":127,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:30:53.739046+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6778},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6778":{"gid":6778,"name":"a","rank":0,"incarnation":124,"state":"up:active","state_seq":23,"addr":"172.21.15.42:6835/3028851337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3028851337},{"type":"v1","addr":"172.21.15.42:6835","nonce":3028851337}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:31:33.204 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 128 2019-12-05T05:31:33.206 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:33.202+0000 7f3bde5f5700 1 -- 172.21.15.42:0/2277756396 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3bb404ae50 msgr2=0x7f3bb404d360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:31:33.206 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:33.202+0000 7f3bde5f5700 1 --2- 172.21.15.42:0/2277756396 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3bb404ae50 0x7f3bb404d360 crc :-1 s=READY pgs=440 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:31:33.207 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:33.202+0000 7f3bde5f5700 1 -- 172.21.15.42:0/2277756396 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3bb400b0c0 msgr2=0x7f3bb400d500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:31:33.207 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:33.202+0000 7f3bde5f5700 1 --2- 172.21.15.42:0/2277756396 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3bb400b0c0 0x7f3bb400d500 secure :-1 s=READY pgs=210 cs=0 l=1 rx=0x7f3bc40078c0 tx=0x7f3bc4015520).stop 2019-12-05T05:31:33.207 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:33.202+0000 7f3bdd393700 1 -- 172.21.15.42:0/2277756396 reap_dead start 2019-12-05T05:31:33.207 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:33.202+0000 7f3bde5f5700 1 -- 172.21.15.42:0/2277756396 shutdown_connections 2019-12-05T05:31:33.207 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:33.202+0000 7f3bde5f5700 1 --2- 172.21.15.42:0/2277756396 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3bd81145b0 0x7f3bd80a22d0 unknown :-1 s=CLOSED pgs=1031 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:31:33.225 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:33.202+0000 7f3bde5f5700 1 --2- 172.21.15.42:0/2277756396 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3bb4003050 0x7f3bb4005520 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:31:33.225 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:33.202+0000 7f3bde5f5700 1 -- 172.21.15.42:0/2277756396 shutdown_connections 2019-12-05T05:31:33.225 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:33.202+0000 7f3bde5f5700 1 -- 172.21.15.42:0/2277756396 wait complete. 2019-12-05T05:31:33.225 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:33.202+0000 7f3bde5f5700 1 -- 172.21.15.42:0/2277756396 >> 172.21.15.42:0/2277756396 conn(0x7f3bd809f930 msgr2=0x7f3bd80a69e0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:31:33.226 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T05:31:35.228 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:31:35.342 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:35.330+0000 7fd547572700 1 Processor -- start 2019-12-05T05:31:35.342 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:35.330+0000 7fd547572700 1 -- start start 2019-12-05T05:31:35.343 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:35.330+0000 7fd547572700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd5400a7740 0x7fd54008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:31:35.343 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:35.330+0000 7fd547572700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd54007e720 0x7fd54007eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:31:35.343 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:35.330+0000 7fd547572700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd54009f2e0 0x7fd54009f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:31:35.343 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:35.330+0000 7fd547572700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5400a0b00 con 0x7fd54007e720 2019-12-05T05:31:35.343 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:35.330+0000 7fd547572700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd54008e5b0 con 0x7fd5400a7740 2019-12-05T05:31:35.346 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:35.330+0000 7fd547572700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd54008e850 con 0x7fd54009f2e0 2019-12-05T05:31:35.346 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:35.330+0000 7fd545b0f700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd5400a7740 0x7fd54008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:31:35.346 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:35.330+0000 7fd545b0f700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd5400a7740 0x7fd54008df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:35638/0 (socket says 172.21.15.42:35638) 2019-12-05T05:31:35.346 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:35.330+0000 7fd545b0f700 1 -- 172.21.15.42:0/232188889 learned_addr learned my addr 172.21.15.42:0/232188889 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:31:35.346 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:35.330+0000 7fd546310700 1 --2- 172.21.15.42:0/232188889 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd54009f2e0 0x7fd54009f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:31:35.346 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:35.330+0000 7fd54530e700 1 --2- 172.21.15.42:0/232188889 >> v2:172.21.15.40:3300/0 conn(0x7fd54007e720 0x7fd54007eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:31:35.347 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:35.330+0000 7fd545b0f700 1 -- 172.21.15.42:0/232188889 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd54009f2e0 msgr2=0x7fd54009f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:31:35.347 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:35.330+0000 7fd545b0f700 1 --2- 172.21.15.42:0/232188889 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd54009f2e0 0x7fd54009f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:31:35.347 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:35.330+0000 7fd545b0f700 1 -- 172.21.15.42:0/232188889 >> v2:172.21.15.40:3300/0 conn(0x7fd54007e720 msgr2=0x7fd54007eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:31:35.347 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:35.330+0000 7fd545b0f700 1 --2- 172.21.15.42:0/232188889 >> v2:172.21.15.40:3300/0 conn(0x7fd54007e720 0x7fd54007eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:31:35.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:35.330+0000 7fd545b0f700 1 -- 172.21.15.42:0/232188889 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd54007f0a0 con 0x7fd5400a7740 2019-12-05T05:31:35.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:35.330+0000 7fd54530e700 1 --2- 172.21.15.42:0/232188889 >> v2:172.21.15.40:3300/0 conn(0x7fd54007e720 0x7fd54007eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T05:31:35.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:35.330+0000 7fd545b0f700 1 --2- 172.21.15.42:0/232188889 >> v2:172.21.15.42:3300/0 conn(0x7fd5400a7740 0x7fd54008df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fd5400a0f50 tx=0x7fd53000fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:35.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:35.330+0000 7fd545b0f700 1 --2- 172.21.15.42:0/232188889 >> v2:172.21.15.42:3300/0 conn(0x7fd5400a7740 0x7fd54008df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fd5400a0f50 tx=0x7fd53000fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:35.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:35.330+0000 7fd545b0f700 1 --2- 172.21.15.42:0/232188889 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd5400a7740 0x7fd54008df80 secure :-1 s=READY pgs=1034 cs=0 l=1 rx=0x7fd5400a0f50 tx=0x7fd53000fdb0).ready entity=mon.1 client_cookie=8f36e4d2fcf87c70 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:31:35.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:35.330+0000 7fd545b0f700 1 --2- 172.21.15.42:0/232188889 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd5400a7740 0x7fd54008df80 secure :-1 s=THROTTLE_DONE pgs=1034 cs=0 l=1 rx=0x7fd5400a0f50 tx=0x7fd53000fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:35.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:35.330+0000 7fd545b0f700 1 --2- 172.21.15.42:0/232188889 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd5400a7740 0x7fd54008df80 secure :-1 s=THROTTLE_DONE pgs=1034 cs=0 l=1 rx=0x7fd5400a0f50 tx=0x7fd53000fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:35.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:35.330+0000 7fd544b0d700 1 -- 172.21.15.42:0/232188889 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd530008c90 con 0x7fd5400a7740 2019-12-05T05:31:35.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:35.330+0000 7fd545b0f700 1 --2- 172.21.15.42:0/232188889 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd5400a7740 0x7fd54008df80 secure :-1 s=THROTTLE_DONE pgs=1034 cs=0 l=1 rx=0x7fd5400a0f50 tx=0x7fd53000fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:35.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:35.330+0000 7fd544b0d700 1 -- 172.21.15.42:0/232188889 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5300088b0 con 0x7fd5400a7740 2019-12-05T05:31:35.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:35.330+0000 7fd544b0d700 1 -- 172.21.15.42:0/232188889 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd530011b50 con 0x7fd5400a7740 2019-12-05T05:31:35.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:35.330+0000 7fd547572700 1 -- 172.21.15.42:0/232188889 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd5400a7740 msgr2=0x7fd54008df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:31:35.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:35.330+0000 7fd547572700 1 --2- 172.21.15.42:0/232188889 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd5400a7740 0x7fd54008df80 secure :-1 s=READY pgs=1034 cs=0 l=1 rx=0x7fd5400a0f50 tx=0x7fd53000fdb0).stop 2019-12-05T05:31:35.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:35.334+0000 7fd547572700 1 -- 172.21.15.42:0/232188889 shutdown_connections 2019-12-05T05:31:35.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:35.334+0000 7fd547572700 1 --2- 172.21.15.42:0/232188889 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd54009f2e0 0x7fd54009f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:31:35.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:35.334+0000 7fd547572700 1 --2- 172.21.15.42:0/232188889 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd5400a7740 0x7fd54008df80 unknown :-1 s=CLOSED pgs=1034 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:31:35.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:35.334+0000 7fd547572700 1 --2- 172.21.15.42:0/232188889 >> v2:172.21.15.40:3300/0 conn(0x7fd54007e720 0x7fd54007eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:31:35.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:35.334+0000 7fd547572700 1 -- 172.21.15.42:0/232188889 shutdown_connections 2019-12-05T05:31:35.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:35.334+0000 7fd547572700 1 -- 172.21.15.42:0/232188889 wait complete. 2019-12-05T05:31:35.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:35.334+0000 7fd547572700 1 -- 172.21.15.42:0/232188889 >> 172.21.15.42:0/232188889 conn(0x7fd5400aabe0 msgr2=0x7fd5400a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:31:35.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:35.334+0000 7fd547572700 1 Processor -- start 2019-12-05T05:31:35.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:35.334+0000 7fd547572700 1 -- start start 2019-12-05T05:31:35.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:35.338+0000 7fd547572700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd540065900 0x7fd54007da00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:31:35.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:35.338+0000 7fd547572700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd54007df50 0x7fd5400643f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:31:35.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:35.338+0000 7fd547572700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd540065060 con 0x7fd54008eeb0 2019-12-05T05:31:35.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:35.338+0000 7fd547572700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5400653c0 con 0x7fd54007df50 2019-12-05T05:31:35.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:35.338+0000 7fd547572700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd540065660 con 0x7fd540065900 2019-12-05T05:31:35.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:35.338+0000 7fd54530e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd540065900 0x7fd54007da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:31:35.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:35.338+0000 7fd54530e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd540065900 0x7fd54007da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:35644/0 (socket says 172.21.15.42:35644) 2019-12-05T05:31:35.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:35.338+0000 7fd54530e700 1 -- 172.21.15.42:0/1132467965 learned_addr learned my addr 172.21.15.42:0/1132467965 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:31:35.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:35.338+0000 7fd546310700 1 --2- 172.21.15.42:0/1132467965 >> v2:172.21.15.40:3300/0 conn(0x7fd54007df50 0x7fd5400643f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:31:35.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:35.338+0000 7fd54530e700 1 -- 172.21.15.42:0/1132467965 >> v2:172.21.15.40:3300/0 conn(0x7fd54007df50 msgr2=0x7fd5400643f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:31:35.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:35.338+0000 7fd54530e700 1 --2- 172.21.15.42:0/1132467965 >> v2:172.21.15.40:3300/0 conn(0x7fd54007df50 0x7fd5400643f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:31:35.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:35.338+0000 7fd54530e700 1 -- 172.21.15.42:0/1132467965 >> v1:172.21.15.40:6789/0 conn(0x7fd54008eeb0 legacy=0x7fd540130270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:31:35.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:35.338+0000 7fd54530e700 1 -- 172.21.15.42:0/1132467965 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd540064940 con 0x7fd540065900 2019-12-05T05:31:35.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:35.338+0000 7fd54530e700 1 --2- 172.21.15.42:0/1132467965 >> v2:172.21.15.42:3300/0 conn(0x7fd540065900 0x7fd54007da00 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fd53c0077a0 tx=0x7fd53c010a50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:35.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:35.338+0000 7fd54530e700 1 --2- 172.21.15.42:0/1132467965 >> v2:172.21.15.42:3300/0 conn(0x7fd540065900 0x7fd54007da00 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fd53c0077a0 tx=0x7fd53c010a50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:35.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:35.338+0000 7fd54530e700 1 --2- 172.21.15.42:0/1132467965 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd540065900 0x7fd54007da00 secure :-1 s=READY pgs=1035 cs=0 l=1 rx=0x7fd53c0077a0 tx=0x7fd53c010a50).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:31:35.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:35.338+0000 7fd54530e700 1 --2- 172.21.15.42:0/1132467965 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd540065900 0x7fd54007da00 secure :-1 s=THROTTLE_DONE pgs=1035 cs=0 l=1 rx=0x7fd53c0077a0 tx=0x7fd53c010a50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:35.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:35.338+0000 7fd54530e700 1 --2- 172.21.15.42:0/1132467965 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd540065900 0x7fd54007da00 secure :-1 s=THROTTLE_DONE pgs=1035 cs=0 l=1 rx=0x7fd53c0077a0 tx=0x7fd53c010a50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:35.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:35.338+0000 7fd5367fc700 1 -- 172.21.15.42:0/1132467965 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd53c00c680 con 0x7fd540065900 2019-12-05T05:31:35.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:35.338+0000 7fd54530e700 1 --2- 172.21.15.42:0/1132467965 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd540065900 0x7fd54007da00 secure :-1 s=THROTTLE_DONE pgs=1035 cs=0 l=1 rx=0x7fd53c0077a0 tx=0x7fd53c010a50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:35.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:35.338+0000 7fd5367fc700 1 -- 172.21.15.42:0/1132467965 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd53c00c2a0 con 0x7fd540065900 2019-12-05T05:31:35.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:35.338+0000 7fd547572700 1 -- 172.21.15.42:0/1132467965 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd540064c10 con 0x7fd540065900 2019-12-05T05:31:35.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:35.338+0000 7fd5367fc700 1 -- 172.21.15.42:0/1132467965 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd53c00a070 con 0x7fd540065900 2019-12-05T05:31:35.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:35.338+0000 7fd547572700 1 -- 172.21.15.42:0/1132467965 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd540137860 con 0x7fd540065900 2019-12-05T05:31:35.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:35.338+0000 7fd54530e700 1 --2- 172.21.15.42:0/1132467965 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd540065900 0x7fd54007da00 secure :-1 s=THROTTLE_DONE pgs=1035 cs=0 l=1 rx=0x7fd53c0077a0 tx=0x7fd53c010a50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:35.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:35.338+0000 7fd5367fc700 1 -- 172.21.15.42:0/1132467965 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fd53c007be0 con 0x7fd540065900 2019-12-05T05:31:35.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:35.338+0000 7fd54530e700 1 --2- 172.21.15.42:0/1132467965 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd540065900 0x7fd54007da00 secure :-1 s=THROTTLE_DONE pgs=1035 cs=0 l=1 rx=0x7fd53c0077a0 tx=0x7fd53c010a50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:35.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:35.342+0000 7fd5367fc700 1 --2- 172.21.15.42:0/1132467965 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd52803acf0 0x7fd52803d200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:31:35.358 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:35.342+0000 7fd5367fc700 1 -- 172.21.15.42:0/1132467965 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 7665+0+0 (secure 0 0 0) 0x7fd53c04aea0 con 0x7fd540065900 2019-12-05T05:31:35.358 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:35.342+0000 7fd545b0f700 1 --2- 172.21.15.42:0/1132467965 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd52803acf0 0x7fd52803d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:31:35.358 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:35.342+0000 7fd545b0f700 1 --2- 172.21.15.42:0/1132467965 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd52803acf0 0x7fd52803d200 crc :-1 s=READY pgs=441 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:31:35.358 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:35.346+0000 7fd547572700 1 -- 172.21.15.42:0/1132467965 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd54005d110 con 0x7fd540065900 2019-12-05T05:31:35.358 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:35.354+0000 7fd54530e700 1 --2- 172.21.15.42:0/1132467965 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd540065900 0x7fd54007da00 secure :-1 s=THROTTLE_DONE pgs=1035 cs=0 l=1 rx=0x7fd53c0077a0 tx=0x7fd53c010a50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:35.358 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:35.354+0000 7fd5367fc700 1 -- 172.21.15.42:0/1132467965 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fd53c031f20 con 0x7fd540065900 2019-12-05T05:31:35.530 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:35.526+0000 7fd547572700 1 -- 172.21.15.42:0/1132467965 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd54005b8c0 con 0x7fd540065900 2019-12-05T05:31:35.530 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:35.526+0000 7fd54530e700 1 --2- 172.21.15.42:0/1132467965 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd540065900 0x7fd54007da00 secure :-1 s=THROTTLE_DONE pgs=1035 cs=0 l=1 rx=0x7fd53c0077a0 tx=0x7fd53c010a50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:35.530 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:35.526+0000 7fd5367fc700 1 -- 172.21.15.42:0/1132467965 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 128 v128) v1 ==== 95+0+2112 (secure 0 0 0) 0x7fd53c032180 con 0x7fd540065900 2019-12-05T05:31:35.532 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:31:35.533 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":128,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6984,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/3543033403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3543033403},{"type":"v1","addr":"172.21.15.40:6833","nonce":3543033403}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":128}],"filesystems":[{"mdsmap":{"epoch":127,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:30:53.739046+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6778},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6778":{"gid":6778,"name":"a","rank":0,"incarnation":124,"state":"up:active","state_seq":23,"addr":"172.21.15.42:6835/3028851337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3028851337},{"type":"v1","addr":"172.21.15.42:6835","nonce":3028851337}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:31:35.533 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 128 2019-12-05T05:31:35.534 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:35.530+0000 7fd547572700 1 -- 172.21.15.42:0/1132467965 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd52803acf0 msgr2=0x7fd52803d200 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:31:35.534 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:35.530+0000 7fd547572700 1 --2- 172.21.15.42:0/1132467965 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd52803acf0 0x7fd52803d200 crc :-1 s=READY pgs=441 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:31:35.535 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:35.530+0000 7fd547572700 1 -- 172.21.15.42:0/1132467965 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd540065900 msgr2=0x7fd54007da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:31:35.535 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:35.530+0000 7fd547572700 1 --2- 172.21.15.42:0/1132467965 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd540065900 0x7fd54007da00 secure :-1 s=READY pgs=1035 cs=0 l=1 rx=0x7fd53c0077a0 tx=0x7fd53c010a50).stop 2019-12-05T05:31:35.536 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:35.530+0000 7fd547572700 1 -- 172.21.15.42:0/1132467965 shutdown_connections 2019-12-05T05:31:35.536 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:35.530+0000 7fd547572700 1 --2- 172.21.15.42:0/1132467965 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd52803acf0 0x7fd52803d200 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:31:35.536 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:35.530+0000 7fd547572700 1 --2- 172.21.15.42:0/1132467965 >> v2:172.21.15.40:3300/0 conn(0x7fd54007df50 0x7fd5400643f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:31:35.536 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:35.530+0000 7fd547572700 1 --2- 172.21.15.42:0/1132467965 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd540065900 0x7fd54007da00 unknown :-1 s=CLOSED pgs=1035 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:31:35.536 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:35.530+0000 7fd547572700 1 -- 172.21.15.42:0/1132467965 shutdown_connections 2019-12-05T05:31:35.536 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:35.530+0000 7fd547572700 1 -- 172.21.15.42:0/1132467965 wait complete. 2019-12-05T05:31:35.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:35.530+0000 7fd547572700 1 -- 172.21.15.42:0/1132467965 >> 172.21.15.42:0/1132467965 conn(0x7fd54008fda0 msgr2=0x7fd54009f690 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:31:35.545 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T05:31:37.547 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:31:37.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:37.646+0000 7ff914990700 1 Processor -- start 2019-12-05T05:31:37.654 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:37.646+0000 7ff914990700 1 -- start start 2019-12-05T05:31:37.654 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:37.646+0000 7ff914990700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff9100a7740 0x7ff91008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:31:37.654 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:37.646+0000 7ff914990700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff91009fae0 0x7ff91007eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:31:37.654 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:37.646+0000 7ff914990700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9100a0710 con 0x7ff91007e720 2019-12-05T05:31:37.654 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:37.646+0000 7ff914990700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9100a0b00 con 0x7ff91009fae0 2019-12-05T05:31:37.654 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:37.646+0000 7ff914990700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff91008e5b0 con 0x7ff9100a7740 2019-12-05T05:31:37.655 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:37.646+0000 7ff90ed9d700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff9100a7740 0x7ff91008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:31:37.655 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:37.646+0000 7ff90f59e700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff91009fae0 0x7ff91007eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:31:37.655 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:37.646+0000 7ff90ed9d700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff9100a7740 0x7ff91008df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:36120/0 (socket says 172.21.15.42:36120) 2019-12-05T05:31:37.655 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:37.646+0000 7ff90ed9d700 1 -- 172.21.15.42:0/3458896963 learned_addr learned my addr 172.21.15.42:0/3458896963 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:31:37.655 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:37.646+0000 7ff90f59e700 1 -- 172.21.15.42:0/3458896963 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff9100a7740 msgr2=0x7ff91008df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:31:37.655 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:37.646+0000 7ff90f59e700 1 --2- 172.21.15.42:0/3458896963 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff9100a7740 0x7ff91008df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:31:37.656 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:37.646+0000 7ff90f59e700 1 -- 172.21.15.42:0/3458896963 >> v1:172.21.15.40:6789/0 conn(0x7ff91007e720 legacy=0x7ff91009f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:31:37.656 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:37.646+0000 7ff90f59e700 1 -- 172.21.15.42:0/3458896963 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff91007f0a0 con 0x7ff91009fae0 2019-12-05T05:31:37.656 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:37.646+0000 7ff90ed9d700 1 --2- 172.21.15.42:0/3458896963 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff9100a7740 0x7ff91008df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:31:37.656 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:37.646+0000 7ff90f59e700 1 --2- 172.21.15.42:0/3458896963 >> v2:172.21.15.40:3300/0 conn(0x7ff91009fae0 0x7ff91007eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7ff908008b10 tx=0x7ff90800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:37.656 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:37.646+0000 7ff90f59e700 1 --2- 172.21.15.42:0/3458896963 >> v2:172.21.15.40:3300/0 conn(0x7ff91009fae0 0x7ff91007eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7ff908008b10 tx=0x7ff90800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:37.656 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:37.646+0000 7ff90f59e700 1 --2- 172.21.15.42:0/3458896963 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff91009fae0 0x7ff91007eb50 secure :-1 s=READY pgs=804 cs=0 l=1 rx=0x7ff908008b10 tx=0x7ff90800fff0).ready entity=mon.0 client_cookie=299685768bcee5b6 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:31:37.656 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:37.646+0000 7ff90f59e700 1 --2- 172.21.15.42:0/3458896963 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff91009fae0 0x7ff91007eb50 secure :-1 s=THROTTLE_DONE pgs=804 cs=0 l=1 rx=0x7ff908008b10 tx=0x7ff90800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:37.657 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:37.646+0000 7ff90f59e700 1 --2- 172.21.15.42:0/3458896963 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff91009fae0 0x7ff91007eb50 secure :-1 s=THROTTLE_DONE pgs=804 cs=0 l=1 rx=0x7ff908008b10 tx=0x7ff90800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:37.657 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:37.646+0000 7ff90f59e700 1 --2- 172.21.15.42:0/3458896963 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff91009fae0 0x7ff91007eb50 secure :-1 s=THROTTLE_DONE pgs=804 cs=0 l=1 rx=0x7ff908008b10 tx=0x7ff90800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:37.657 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:37.646+0000 7ff90dd9b700 1 -- 172.21.15.42:0/3458896963 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff908010900 con 0x7ff91009fae0 2019-12-05T05:31:37.657 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:37.646+0000 7ff90dd9b700 1 -- 172.21.15.42:0/3458896963 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff908008b50 con 0x7ff91009fae0 2019-12-05T05:31:37.657 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:37.650+0000 7ff90dd9b700 1 -- 172.21.15.42:0/3458896963 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff9080110c0 con 0x7ff91009fae0 2019-12-05T05:31:37.657 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:37.650+0000 7ff914990700 1 -- 172.21.15.42:0/3458896963 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff91009fae0 msgr2=0x7ff91007eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:31:37.658 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:37.650+0000 7ff914990700 1 --2- 172.21.15.42:0/3458896963 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff91009fae0 0x7ff91007eb50 secure :-1 s=READY pgs=804 cs=0 l=1 rx=0x7ff908008b10 tx=0x7ff90800fff0).stop 2019-12-05T05:31:37.658 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:37.650+0000 7ff914990700 1 -- 172.21.15.42:0/3458896963 shutdown_connections 2019-12-05T05:31:37.658 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:37.650+0000 7ff914990700 1 --2- 172.21.15.42:0/3458896963 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff91009fae0 0x7ff91007eb50 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:31:37.658 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:37.650+0000 7ff914990700 1 --2- 172.21.15.42:0/3458896963 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff9100a7740 0x7ff91008df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:31:37.659 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:37.650+0000 7ff914990700 1 -- 172.21.15.42:0/3458896963 shutdown_connections 2019-12-05T05:31:37.660 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:37.650+0000 7ff914990700 1 -- 172.21.15.42:0/3458896963 wait complete. 2019-12-05T05:31:37.660 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:37.650+0000 7ff914990700 1 -- 172.21.15.42:0/3458896963 >> 172.21.15.42:0/3458896963 conn(0x7ff9100aabe0 msgr2=0x7ff9100a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:31:37.665 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:37.650+0000 7ff914990700 1 Processor -- start 2019-12-05T05:31:37.665 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:37.650+0000 7ff914990700 1 -- start start 2019-12-05T05:31:37.665 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:37.654+0000 7ff914990700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff9100a6d60 0x7ff9100a38b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:31:37.665 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:37.654+0000 7ff914990700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff91007d920 0x7ff9100a2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:31:37.666 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:37.654+0000 7ff914990700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9100a3e00 con 0x7ff9100a25e0 2019-12-05T05:31:37.666 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:37.654+0000 7ff914990700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9100a4150 con 0x7ff91007d920 2019-12-05T05:31:37.666 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:37.654+0000 7ff914990700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9100a1df0 con 0x7ff9100a6d60 2019-12-05T05:31:37.666 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:37.654+0000 7ff90ed9d700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff9100a6d60 0x7ff9100a38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:31:37.667 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:37.654+0000 7ff90e59c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff91007d920 0x7ff9100a2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:31:37.667 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:37.654+0000 7ff90ed9d700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff9100a6d60 0x7ff9100a38b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:36124/0 (socket says 172.21.15.42:36124) 2019-12-05T05:31:37.667 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:37.654+0000 7ff90ed9d700 1 -- 172.21.15.42:0/166111662 learned_addr learned my addr 172.21.15.42:0/166111662 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:31:37.667 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:37.654+0000 7ff90ed9d700 1 -- 172.21.15.42:0/166111662 >> v2:172.21.15.42:3300/0 conn(0x7ff91007d920 msgr2=0x7ff9100a2090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:31:37.668 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:37.654+0000 7ff90ed9d700 1 --2- 172.21.15.42:0/166111662 >> v2:172.21.15.42:3300/0 conn(0x7ff91007d920 0x7ff9100a2090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:31:37.668 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:37.654+0000 7ff90ed9d700 1 -- 172.21.15.42:0/166111662 >> v1:172.21.15.42:6789/0 conn(0x7ff9100a25e0 legacy=0x7ff9100a4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:31:37.668 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:37.654+0000 7ff90ed9d700 1 -- 172.21.15.42:0/166111662 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9101304e0 con 0x7ff9100a6d60 2019-12-05T05:31:37.668 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:37.654+0000 7ff90ed9d700 1 --2- 172.21.15.42:0/166111662 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff9100a6d60 0x7ff9100a38b0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7ff900002ef0 tx=0x7ff900010d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:37.668 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:37.654+0000 7ff90ed9d700 1 --2- 172.21.15.42:0/166111662 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff9100a6d60 0x7ff9100a38b0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7ff900002ef0 tx=0x7ff900010d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:37.668 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:37.654+0000 7ff90ed9d700 1 --2- 172.21.15.42:0/166111662 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff9100a6d60 0x7ff9100a38b0 secure :-1 s=READY pgs=211 cs=0 l=1 rx=0x7ff900002ef0 tx=0x7ff900010d50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:31:37.668 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:37.654+0000 7ff90ed9d700 1 --2- 172.21.15.42:0/166111662 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff9100a6d60 0x7ff9100a38b0 secure :-1 s=THROTTLE_DONE pgs=211 cs=0 l=1 rx=0x7ff900002ef0 tx=0x7ff900010d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:37.669 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:37.654+0000 7ff90ed9d700 1 --2- 172.21.15.42:0/166111662 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff9100a6d60 0x7ff9100a38b0 secure :-1 s=THROTTLE_DONE pgs=211 cs=0 l=1 rx=0x7ff900002ef0 tx=0x7ff900010d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:37.669 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:37.654+0000 7ff9077fe700 1 -- 172.21.15.42:0/166111662 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff9000088b0 con 0x7ff9100a6d60 2019-12-05T05:31:37.669 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:37.654+0000 7ff90ed9d700 1 --2- 172.21.15.42:0/166111662 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff9100a6d60 0x7ff9100a38b0 secure :-1 s=THROTTLE_DONE pgs=211 cs=0 l=1 rx=0x7ff900002ef0 tx=0x7ff900010d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:37.671 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:37.654+0000 7ff9077fe700 1 -- 172.21.15.42:0/166111662 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff900007160 con 0x7ff9100a6d60 2019-12-05T05:31:37.671 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:37.654+0000 7ff914990700 1 -- 172.21.15.42:0/166111662 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff91011b770 con 0x7ff9100a6d60 2019-12-05T05:31:37.671 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:37.654+0000 7ff9077fe700 1 -- 172.21.15.42:0/166111662 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff900005750 con 0x7ff9100a6d60 2019-12-05T05:31:37.671 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:37.654+0000 7ff914990700 1 -- 172.21.15.42:0/166111662 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff91007def0 con 0x7ff9100a6d60 2019-12-05T05:31:37.671 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:37.658+0000 7ff90ed9d700 1 --2- 172.21.15.42:0/166111662 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff9100a6d60 0x7ff9100a38b0 secure :-1 s=THROTTLE_DONE pgs=211 cs=0 l=1 rx=0x7ff900002ef0 tx=0x7ff900010d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:37.672 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:37.658+0000 7ff90ed9d700 1 --2- 172.21.15.42:0/166111662 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff9100a6d60 0x7ff9100a38b0 secure :-1 s=THROTTLE_DONE pgs=211 cs=0 l=1 rx=0x7ff900002ef0 tx=0x7ff900010d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:37.672 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:37.658+0000 7ff9077fe700 1 -- 172.21.15.42:0/166111662 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7ff900005a60 con 0x7ff9100a6d60 2019-12-05T05:31:37.672 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:37.658+0000 7ff9077fe700 1 --2- 172.21.15.42:0/166111662 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff8e803ad60 0x7ff8e803d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:31:37.672 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:37.658+0000 7ff9077fe700 1 -- 172.21.15.42:0/166111662 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 7665+0+0 (secure 0 0 0) 0x7ff90004b4a0 con 0x7ff9100a6d60 2019-12-05T05:31:37.672 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:37.658+0000 7ff90e59c700 1 --2- 172.21.15.42:0/166111662 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff8e803ad60 0x7ff8e803d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:31:37.672 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:37.658+0000 7ff90e59c700 1 --2- 172.21.15.42:0/166111662 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff8e803ad60 0x7ff8e803d270 crc :-1 s=READY pgs=442 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:31:37.673 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:37.662+0000 7ff914990700 1 -- 172.21.15.42:0/166111662 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff91005d110 con 0x7ff9100a6d60 2019-12-05T05:31:37.674 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:37.670+0000 7ff90ed9d700 1 --2- 172.21.15.42:0/166111662 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff9100a6d60 0x7ff9100a38b0 secure :-1 s=THROTTLE_DONE pgs=211 cs=0 l=1 rx=0x7ff900002ef0 tx=0x7ff900010d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:37.674 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:37.670+0000 7ff9077fe700 1 -- 172.21.15.42:0/166111662 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7ff900005d70 con 0x7ff9100a6d60 2019-12-05T05:31:37.845 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:37.842+0000 7ff914990700 1 -- 172.21.15.42:0/166111662 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff91005b8c0 con 0x7ff9100a6d60 2019-12-05T05:31:37.847 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:37.842+0000 7ff90ed9d700 1 --2- 172.21.15.42:0/166111662 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff9100a6d60 0x7ff9100a38b0 secure :-1 s=THROTTLE_DONE pgs=211 cs=0 l=1 rx=0x7ff900002ef0 tx=0x7ff900010d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:37.847 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:37.842+0000 7ff9077fe700 1 -- 172.21.15.42:0/166111662 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 128 v128) v1 ==== 95+0+2112 (secure 0 0 0) 0x7ff9000324e0 con 0x7ff9100a6d60 2019-12-05T05:31:37.848 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 128 2019-12-05T05:31:37.849 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:31:37.849 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":128,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6984,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/3543033403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3543033403},{"type":"v1","addr":"172.21.15.40:6833","nonce":3543033403}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":128}],"filesystems":[{"mdsmap":{"epoch":127,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:30:53.739046+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6778},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6778":{"gid":6778,"name":"a","rank":0,"incarnation":124,"state":"up:active","state_seq":23,"addr":"172.21.15.42:6835/3028851337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3028851337},{"type":"v1","addr":"172.21.15.42:6835","nonce":3028851337}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:31:37.851 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:37.846+0000 7ff914990700 1 -- 172.21.15.42:0/166111662 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff8e803ad60 msgr2=0x7ff8e803d270 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:31:37.851 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:37.846+0000 7ff914990700 1 --2- 172.21.15.42:0/166111662 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff8e803ad60 0x7ff8e803d270 crc :-1 s=READY pgs=442 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:31:37.852 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:37.846+0000 7ff914990700 1 -- 172.21.15.42:0/166111662 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff9100a6d60 msgr2=0x7ff9100a38b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:31:37.852 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:37.846+0000 7ff914990700 1 --2- 172.21.15.42:0/166111662 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff9100a6d60 0x7ff9100a38b0 secure :-1 s=READY pgs=211 cs=0 l=1 rx=0x7ff900002ef0 tx=0x7ff900010d50).stop 2019-12-05T05:31:37.852 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:37.846+0000 7ff914990700 1 -- 172.21.15.42:0/166111662 shutdown_connections 2019-12-05T05:31:37.852 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:37.846+0000 7ff914990700 1 --2- 172.21.15.42:0/166111662 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff8e803ad60 0x7ff8e803d270 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:31:37.852 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:37.846+0000 7ff914990700 1 --2- 172.21.15.42:0/166111662 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff9100a6d60 0x7ff9100a38b0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:31:37.853 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:37.846+0000 7ff914990700 1 --2- 172.21.15.42:0/166111662 >> v2:172.21.15.42:3300/0 conn(0x7ff91007d920 0x7ff9100a2090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:31:37.853 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:37.846+0000 7ff914990700 1 -- 172.21.15.42:0/166111662 shutdown_connections 2019-12-05T05:31:37.853 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:37.846+0000 7ff914990700 1 -- 172.21.15.42:0/166111662 wait complete. 2019-12-05T05:31:37.853 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:37.846+0000 7ff914990700 1 -- 172.21.15.42:0/166111662 >> 172.21.15.42:0/166111662 conn(0x7ff91008eb70 msgr2=0x7ff9100a9790 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:31:37.861 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T05:31:39.082 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:31:39.085 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:31:39.862 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:31:39.966 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:39.958+0000 7f4a32119700 1 Processor -- start 2019-12-05T05:31:39.966 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:39.958+0000 7f4a32119700 1 -- start start 2019-12-05T05:31:39.966 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:39.958+0000 7f4a32119700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f4a2c0a7740 0x7f4a2c08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:31:39.966 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:39.958+0000 7f4a32119700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a2c0a0710 con 0x7f4a2c09fae0 2019-12-05T05:31:39.966 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:39.958+0000 7f4a32119700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a2c0a9ba0 con 0x7f4a2c07e720 2019-12-05T05:31:39.966 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:39.958+0000 7f4a32119700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a2c0a0b00 con 0x7f4a2c0a7740 2019-12-05T05:31:39.967 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:39.958+0000 7f4a2bfff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f4a2c0a7740 0x7f4a2c08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:31:39.967 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:39.958+0000 7f4a2b7fe700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f4a2c07e720 0x7f4a2c09f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:35706/0 (socket says 172.21.15.42:35706) 2019-12-05T05:31:39.967 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:39.958+0000 7f4a2b7fe700 1 -- 172.21.15.42:0/988530681 learned_addr learned my addr 172.21.15.42:0/988530681 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:31:39.967 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:39.958+0000 7f4a2bfff700 1 -- 172.21.15.42:0/988530681 >> v1:172.21.15.42:6789/0 conn(0x7f4a2c07e720 legacy=0x7f4a2c09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:31:39.967 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:39.958+0000 7f4a2bfff700 1 -- 172.21.15.42:0/988530681 >> v1:172.21.15.40:6789/0 conn(0x7f4a2c09fae0 legacy=0x7f4a2c14efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:31:39.967 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:39.958+0000 7f4a2bfff700 1 -- 172.21.15.42:0/988530681 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a2c07eb50 con 0x7f4a2c0a7740 2019-12-05T05:31:39.968 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:39.962+0000 7f4a2bfff700 1 --2- 172.21.15.42:0/988530681 >> v2:172.21.15.42:3300/0 conn(0x7f4a2c0a7740 0x7f4a2c08df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f4a1c008b10 tx=0x7f4a1c00ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:39.968 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:39.962+0000 7f4a2bfff700 1 --2- 172.21.15.42:0/988530681 >> v2:172.21.15.42:3300/0 conn(0x7f4a2c0a7740 0x7f4a2c08df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f4a1c008b10 tx=0x7f4a1c00ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:39.968 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:39.962+0000 7f4a2bfff700 1 --2- 172.21.15.42:0/988530681 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4a2c0a7740 0x7f4a2c08df80 secure :-1 s=READY pgs=1038 cs=0 l=1 rx=0x7f4a1c008b10 tx=0x7f4a1c00ff70).ready entity=mon.1 client_cookie=83828c1ad1706199 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:31:39.968 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:39.962+0000 7f4a2bfff700 1 --2- 172.21.15.42:0/988530681 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4a2c0a7740 0x7f4a2c08df80 secure :-1 s=THROTTLE_DONE pgs=1038 cs=0 l=1 rx=0x7f4a1c008b10 tx=0x7f4a1c00ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:39.968 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:39.962+0000 7f4a2affd700 1 -- 172.21.15.42:0/988530681 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4a1c0119d0 con 0x7f4a2c0a7740 2019-12-05T05:31:39.968 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:39.962+0000 7f4a2bfff700 1 --2- 172.21.15.42:0/988530681 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4a2c0a7740 0x7f4a2c08df80 secure :-1 s=THROTTLE_DONE pgs=1038 cs=0 l=1 rx=0x7f4a1c008b10 tx=0x7f4a1c00ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:39.968 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:39.962+0000 7f4a2bfff700 1 --2- 172.21.15.42:0/988530681 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4a2c0a7740 0x7f4a2c08df80 secure :-1 s=THROTTLE_DONE pgs=1038 cs=0 l=1 rx=0x7f4a1c008b10 tx=0x7f4a1c00ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:39.969 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:39.962+0000 7f4a2affd700 1 -- 172.21.15.42:0/988530681 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a1c0115f0 con 0x7f4a2c0a7740 2019-12-05T05:31:39.969 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:39.962+0000 7f4a2affd700 1 -- 172.21.15.42:0/988530681 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4a1c00a160 con 0x7f4a2c0a7740 2019-12-05T05:31:39.969 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:39.962+0000 7f4a32119700 1 -- 172.21.15.42:0/988530681 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4a2c0a7740 msgr2=0x7f4a2c08df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:31:39.969 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:39.962+0000 7f4a32119700 1 --2- 172.21.15.42:0/988530681 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4a2c0a7740 0x7f4a2c08df80 secure :-1 s=READY pgs=1038 cs=0 l=1 rx=0x7f4a1c008b10 tx=0x7f4a1c00ff70).stop 2019-12-05T05:31:39.969 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:39.962+0000 7f4a32119700 1 -- 172.21.15.42:0/988530681 shutdown_connections 2019-12-05T05:31:39.969 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:39.962+0000 7f4a32119700 1 --2- 172.21.15.42:0/988530681 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4a2c0a7740 0x7f4a2c08df80 unknown :-1 s=CLOSED pgs=1038 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:31:39.970 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:39.962+0000 7f4a32119700 1 -- 172.21.15.42:0/988530681 shutdown_connections 2019-12-05T05:31:39.970 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:39.962+0000 7f4a32119700 1 -- 172.21.15.42:0/988530681 wait complete. 2019-12-05T05:31:39.970 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:39.962+0000 7f4a32119700 1 -- 172.21.15.42:0/988530681 >> 172.21.15.42:0/988530681 conn(0x7f4a2c0aabe0 msgr2=0x7f4a2c0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:31:39.973 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:39.962+0000 7f4a32119700 1 Processor -- start 2019-12-05T05:31:39.973 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:39.962+0000 7f4a32119700 1 -- start start 2019-12-05T05:31:39.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:39.966+0000 7f4a32119700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f4a2c0a5d00 0x7f4a2c0a3dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:31:39.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:39.966+0000 7f4a32119700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f4a2c0a23a0 0x7f4a2c07d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:31:39.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:39.966+0000 7f4a32119700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a2c0a4310 con 0x7f4a2c126b40 2019-12-05T05:31:39.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:39.966+0000 7f4a32119700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a2c0a1510 con 0x7f4a2c0a5d00 2019-12-05T05:31:39.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:39.966+0000 7f4a32119700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a2c0a2900 con 0x7f4a2c0a23a0 2019-12-05T05:31:39.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:39.966+0000 7f4a30eb7700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f4a2c0a23a0 0x7f4a2c07d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:31:39.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:39.966+0000 7f4a30eb7700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f4a2c0a23a0 0x7f4a2c07d920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:35678/0 (socket says 172.21.15.42:35678) 2019-12-05T05:31:39.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:39.966+0000 7f4a30eb7700 1 -- 172.21.15.42:0/2588727042 learned_addr learned my addr 172.21.15.42:0/2588727042 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:31:39.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:39.966+0000 7f4a2b7fe700 1 --2- 172.21.15.42:0/2588727042 >> v2:172.21.15.40:3300/0 conn(0x7f4a2c0a5d00 0x7f4a2c0a3dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:31:39.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:39.966+0000 7f4a30eb7700 1 -- 172.21.15.42:0/2588727042 >> v2:172.21.15.40:3300/0 conn(0x7f4a2c0a5d00 msgr2=0x7f4a2c0a3dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:31:39.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:39.966+0000 7f4a30eb7700 1 --2- 172.21.15.42:0/2588727042 >> v2:172.21.15.40:3300/0 conn(0x7f4a2c0a5d00 0x7f4a2c0a3dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:31:39.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:39.966+0000 7f4a30eb7700 1 -- 172.21.15.42:0/2588727042 >> v1:172.21.15.40:6789/0 conn(0x7f4a2c126b40 legacy=0x7f4a2c0a1770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:31:39.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:39.966+0000 7f4a30eb7700 1 -- 172.21.15.42:0/2588727042 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a2c07de70 con 0x7f4a2c0a23a0 2019-12-05T05:31:39.976 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:39.966+0000 7f4a2b7fe700 1 --2- 172.21.15.42:0/2588727042 >> v2:172.21.15.40:3300/0 conn(0x7f4a2c0a5d00 0x7f4a2c0a3dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T05:31:39.976 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:39.966+0000 7f4a30eb7700 1 --2- 172.21.15.42:0/2588727042 >> v2:172.21.15.42:3300/0 conn(0x7f4a2c0a23a0 0x7f4a2c07d920 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f4a24000e90 tx=0x7f4a240107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:39.976 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:39.966+0000 7f4a30eb7700 1 --2- 172.21.15.42:0/2588727042 >> v2:172.21.15.42:3300/0 conn(0x7f4a2c0a23a0 0x7f4a2c07d920 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f4a24000e90 tx=0x7f4a240107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:39.976 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:39.966+0000 7f4a30eb7700 1 --2- 172.21.15.42:0/2588727042 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4a2c0a23a0 0x7f4a2c07d920 secure :-1 s=READY pgs=1039 cs=0 l=1 rx=0x7f4a24000e90 tx=0x7f4a240107c0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:31:39.976 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:39.966+0000 7f4a30eb7700 1 --2- 172.21.15.42:0/2588727042 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4a2c0a23a0 0x7f4a2c07d920 secure :-1 s=THROTTLE_DONE pgs=1039 cs=0 l=1 rx=0x7f4a24000e90 tx=0x7f4a240107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:39.976 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:39.966+0000 7f4a28ff9700 1 -- 172.21.15.42:0/2588727042 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4a24006a40 con 0x7f4a2c0a23a0 2019-12-05T05:31:39.976 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:39.966+0000 7f4a30eb7700 1 --2- 172.21.15.42:0/2588727042 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4a2c0a23a0 0x7f4a2c07d920 secure :-1 s=THROTTLE_DONE pgs=1039 cs=0 l=1 rx=0x7f4a24000e90 tx=0x7f4a240107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:39.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:39.966+0000 7f4a30eb7700 1 --2- 172.21.15.42:0/2588727042 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4a2c0a23a0 0x7f4a2c07d920 secure :-1 s=THROTTLE_DONE pgs=1039 cs=0 l=1 rx=0x7f4a24000e90 tx=0x7f4a240107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:39.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:39.966+0000 7f4a28ff9700 1 -- 172.21.15.42:0/2588727042 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a24006660 con 0x7f4a2c0a23a0 2019-12-05T05:31:39.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:39.966+0000 7f4a32119700 1 -- 172.21.15.42:0/2588727042 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a2c07e160 con 0x7f4a2c0a23a0 2019-12-05T05:31:39.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:39.966+0000 7f4a28ff9700 1 -- 172.21.15.42:0/2588727042 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4a24011000 con 0x7f4a2c0a23a0 2019-12-05T05:31:39.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:39.966+0000 7f4a32119700 1 -- 172.21.15.42:0/2588727042 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a2c12df90 con 0x7f4a2c0a23a0 2019-12-05T05:31:39.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:39.966+0000 7f4a30eb7700 1 --2- 172.21.15.42:0/2588727042 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4a2c0a23a0 0x7f4a2c07d920 secure :-1 s=THROTTLE_DONE pgs=1039 cs=0 l=1 rx=0x7f4a24000e90 tx=0x7f4a240107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:39.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:39.970+0000 7f4a28ff9700 1 -- 172.21.15.42:0/2588727042 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f4a240112e0 con 0x7f4a2c0a23a0 2019-12-05T05:31:39.978 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:39.970+0000 7f4a30eb7700 1 --2- 172.21.15.42:0/2588727042 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4a2c0a23a0 0x7f4a2c07d920 secure :-1 s=THROTTLE_DONE pgs=1039 cs=0 l=1 rx=0x7f4a24000e90 tx=0x7f4a240107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:39.978 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:39.970+0000 7f4a28ff9700 1 --2- 172.21.15.42:0/2588727042 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4a0c03ae30 0x7f4a0c03d340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:31:39.978 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:39.970+0000 7f4a28ff9700 1 -- 172.21.15.42:0/2588727042 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 7665+0+0 (secure 0 0 0) 0x7f4a2404b1d0 con 0x7f4a2c0a23a0 2019-12-05T05:31:39.978 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:39.970+0000 7f4a2bfff700 1 --2- 172.21.15.42:0/2588727042 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4a0c03ae30 0x7f4a0c03d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:31:39.978 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:39.970+0000 7f4a2bfff700 1 --2- 172.21.15.42:0/2588727042 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4a0c03ae30 0x7f4a0c03d340 crc :-1 s=READY pgs=443 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:31:39.979 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:39.974+0000 7f4a32119700 1 -- 172.21.15.42:0/2588727042 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a2c05d110 con 0x7f4a2c0a23a0 2019-12-05T05:31:39.985 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:39.982+0000 7f4a30eb7700 1 --2- 172.21.15.42:0/2588727042 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4a2c0a23a0 0x7f4a2c07d920 secure :-1 s=THROTTLE_DONE pgs=1039 cs=0 l=1 rx=0x7f4a24000e90 tx=0x7f4a240107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:39.985 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:39.982+0000 7f4a28ff9700 1 -- 172.21.15.42:0/2588727042 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f4a24032230 con 0x7f4a2c0a23a0 2019-12-05T05:31:40.157 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:40.154+0000 7f4a32119700 1 -- 172.21.15.42:0/2588727042 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4a2c05b8c0 con 0x7f4a2c0a23a0 2019-12-05T05:31:40.158 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:40.154+0000 7f4a30eb7700 1 --2- 172.21.15.42:0/2588727042 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4a2c0a23a0 0x7f4a2c07d920 secure :-1 s=THROTTLE_DONE pgs=1039 cs=0 l=1 rx=0x7f4a24000e90 tx=0x7f4a240107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:40.158 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:40.154+0000 7f4a28ff9700 1 -- 172.21.15.42:0/2588727042 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 128 v128) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f4a24032490 con 0x7f4a2c0a23a0 2019-12-05T05:31:40.158 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 128 2019-12-05T05:31:40.158 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:31:40.159 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":128,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6984,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/3543033403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3543033403},{"type":"v1","addr":"172.21.15.40:6833","nonce":3543033403}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":128}],"filesystems":[{"mdsmap":{"epoch":127,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:30:53.739046+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6778},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6778":{"gid":6778,"name":"a","rank":0,"incarnation":124,"state":"up:active","state_seq":23,"addr":"172.21.15.42:6835/3028851337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3028851337},{"type":"v1","addr":"172.21.15.42:6835","nonce":3028851337}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:31:40.161 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:40.158+0000 7f4a32119700 1 -- 172.21.15.42:0/2588727042 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4a0c03ae30 msgr2=0x7f4a0c03d340 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:31:40.161 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:40.158+0000 7f4a32119700 1 --2- 172.21.15.42:0/2588727042 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4a0c03ae30 0x7f4a0c03d340 crc :-1 s=READY pgs=443 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:31:40.161 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:40.158+0000 7f4a32119700 1 -- 172.21.15.42:0/2588727042 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4a2c0a23a0 msgr2=0x7f4a2c07d920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:31:40.161 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:40.158+0000 7f4a32119700 1 --2- 172.21.15.42:0/2588727042 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4a2c0a23a0 0x7f4a2c07d920 secure :-1 s=READY pgs=1039 cs=0 l=1 rx=0x7f4a24000e90 tx=0x7f4a240107c0).stop 2019-12-05T05:31:40.162 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:40.158+0000 7f4a32119700 1 -- 172.21.15.42:0/2588727042 shutdown_connections 2019-12-05T05:31:40.162 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:40.158+0000 7f4a32119700 1 --2- 172.21.15.42:0/2588727042 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4a0c03ae30 0x7f4a0c03d340 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:31:40.162 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:40.158+0000 7f4a32119700 1 --2- 172.21.15.42:0/2588727042 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4a2c0a23a0 0x7f4a2c07d920 unknown :-1 s=CLOSED pgs=1039 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:31:40.162 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:40.158+0000 7f4a32119700 1 --2- 172.21.15.42:0/2588727042 >> v2:172.21.15.40:3300/0 conn(0x7f4a2c0a5d00 0x7f4a2c0a3dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:31:40.162 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:40.158+0000 7f4a32119700 1 -- 172.21.15.42:0/2588727042 shutdown_connections 2019-12-05T05:31:40.162 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:40.158+0000 7f4a32119700 1 -- 172.21.15.42:0/2588727042 wait complete. 2019-12-05T05:31:40.163 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:40.158+0000 7f4a32119700 1 -- 172.21.15.42:0/2588727042 >> 172.21.15.42:0/2588727042 conn(0x7f4a2c07ee20 msgr2=0x7f4a2c059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:31:40.171 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T05:31:42.171 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:31:42.278 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:42.270+0000 7f0b59084700 1 Processor -- start 2019-12-05T05:31:42.278 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:42.270+0000 7f0b59084700 1 -- start start 2019-12-05T05:31:42.278 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:42.270+0000 7f0b59084700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0b5408df80 0x7f0b5409f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:31:42.278 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:42.270+0000 7f0b59084700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f0b5409f830 0x7f0b5414efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:31:42.278 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:42.270+0000 7f0b59084700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b5409fc60 con 0x7f0b540a7740 2019-12-05T05:31:42.279 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:42.270+0000 7f0b59084700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b540a0b00 con 0x7f0b5409f830 2019-12-05T05:31:42.279 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:42.270+0000 7f0b59084700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b5408e5b0 con 0x7f0b5408df80 2019-12-05T05:31:42.279 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:42.270+0000 7f0b5359e700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f0b540a7740 0x7f0b5407e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:35720/0 (socket says 172.21.15.42:35720) 2019-12-05T05:31:42.279 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:42.270+0000 7f0b5359e700 1 -- 172.21.15.42:0/394830365 learned_addr learned my addr 172.21.15.42:0/394830365 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:31:42.279 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:42.270+0000 7f0b58883700 1 --2- 172.21.15.42:0/394830365 >> v2:172.21.15.40:3300/0 conn(0x7f0b5409f830 0x7f0b5414efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:31:42.279 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:42.270+0000 7f0b52d9d700 1 --2- 172.21.15.42:0/394830365 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0b5408df80 0x7f0b5409f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:31:42.279 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:42.270+0000 7f0b5259c700 1 -- 172.21.15.42:0/394830365 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4282913197 0 0) 0x7f0b5409fc60 con 0x7f0b540a7740 2019-12-05T05:31:42.280 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:42.270+0000 7f0b5259c700 1 -- 172.21.15.42:0/394830365 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b400030b0 con 0x7f0b540a7740 2019-12-05T05:31:42.280 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:42.270+0000 7f0b58883700 1 -- 172.21.15.42:0/394830365 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0b5408df80 msgr2=0x7f0b5409f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:31:42.280 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:42.270+0000 7f0b58883700 1 --2- 172.21.15.42:0/394830365 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0b5408df80 0x7f0b5409f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:31:42.280 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:42.270+0000 7f0b58883700 1 -- 172.21.15.42:0/394830365 >> v1:172.21.15.42:6789/0 conn(0x7f0b540a7740 legacy=0x7f0b5407e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:31:42.280 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:42.270+0000 7f0b58883700 1 -- 172.21.15.42:0/394830365 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b540a0710 con 0x7f0b5409f830 2019-12-05T05:31:42.280 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:42.270+0000 7f0b52d9d700 1 --2- 172.21.15.42:0/394830365 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0b5408df80 0x7f0b5409f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:31:42.280 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:42.270+0000 7f0b58883700 1 --2- 172.21.15.42:0/394830365 >> v2:172.21.15.40:3300/0 conn(0x7f0b5409f830 0x7f0b5414efe0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f0b4c008b10 tx=0x7f0b4c00fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:42.281 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:42.270+0000 7f0b58883700 1 --2- 172.21.15.42:0/394830365 >> v2:172.21.15.40:3300/0 conn(0x7f0b5409f830 0x7f0b5414efe0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f0b4c008b10 tx=0x7f0b4c00fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:42.281 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:42.270+0000 7f0b58883700 1 --2- 172.21.15.42:0/394830365 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0b5409f830 0x7f0b5414efe0 secure :-1 s=READY pgs=807 cs=0 l=1 rx=0x7f0b4c008b10 tx=0x7f0b4c00fff0).ready entity=mon.0 client_cookie=894a43ee303cc52a server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:31:42.281 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:42.270+0000 7f0b58883700 1 --2- 172.21.15.42:0/394830365 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0b5409f830 0x7f0b5414efe0 secure :-1 s=THROTTLE_DONE pgs=807 cs=0 l=1 rx=0x7f0b4c008b10 tx=0x7f0b4c00fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:42.281 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:42.270+0000 7f0b5259c700 1 -- 172.21.15.42:0/394830365 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0b4c010900 con 0x7f0b5409f830 2019-12-05T05:31:42.281 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:42.270+0000 7f0b58883700 1 --2- 172.21.15.42:0/394830365 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0b5409f830 0x7f0b5414efe0 secure :-1 s=THROTTLE_DONE pgs=807 cs=0 l=1 rx=0x7f0b4c008b10 tx=0x7f0b4c00fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:42.281 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:42.270+0000 7f0b58883700 1 --2- 172.21.15.42:0/394830365 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0b5409f830 0x7f0b5414efe0 secure :-1 s=THROTTLE_DONE pgs=807 cs=0 l=1 rx=0x7f0b4c008b10 tx=0x7f0b4c00fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:42.281 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:42.270+0000 7f0b5259c700 1 -- 172.21.15.42:0/394830365 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b4c008b50 con 0x7f0b5409f830 2019-12-05T05:31:42.282 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:42.274+0000 7f0b5259c700 1 -- 172.21.15.42:0/394830365 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0b4c0110c0 con 0x7f0b5409f830 2019-12-05T05:31:42.282 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:42.274+0000 7f0b59084700 1 -- 172.21.15.42:0/394830365 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0b5409f830 msgr2=0x7f0b5414efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:31:42.282 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:42.274+0000 7f0b59084700 1 --2- 172.21.15.42:0/394830365 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0b5409f830 0x7f0b5414efe0 secure :-1 s=READY pgs=807 cs=0 l=1 rx=0x7f0b4c008b10 tx=0x7f0b4c00fff0).stop 2019-12-05T05:31:42.282 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:42.274+0000 7f0b59084700 1 -- 172.21.15.42:0/394830365 shutdown_connections 2019-12-05T05:31:42.282 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:42.274+0000 7f0b59084700 1 --2- 172.21.15.42:0/394830365 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0b5409f830 0x7f0b5414efe0 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:31:42.282 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:42.274+0000 7f0b59084700 1 --2- 172.21.15.42:0/394830365 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0b5408df80 0x7f0b5409f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:31:42.282 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:42.274+0000 7f0b59084700 1 -- 172.21.15.42:0/394830365 shutdown_connections 2019-12-05T05:31:42.282 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:42.274+0000 7f0b59084700 1 -- 172.21.15.42:0/394830365 wait complete. 2019-12-05T05:31:42.283 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:42.274+0000 7f0b59084700 1 -- 172.21.15.42:0/394830365 >> 172.21.15.42:0/394830365 conn(0x7f0b540aabe0 msgr2=0x7f0b540a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:31:42.286 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:42.278+0000 7f0b59084700 1 Processor -- start 2019-12-05T05:31:42.286 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:42.278+0000 7f0b59084700 1 -- start start 2019-12-05T05:31:42.286 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:42.278+0000 7f0b59084700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0b5411b4f0 0x7f0b541191b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:31:42.286 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:42.278+0000 7f0b59084700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f0b54122110 0x7f0b54122540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:31:42.287 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:42.278+0000 7f0b59084700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b5407dfe0 con 0x7f0b54130420 2019-12-05T05:31:42.287 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:42.278+0000 7f0b59084700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b541239c0 con 0x7f0b54122110 2019-12-05T05:31:42.287 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:42.278+0000 7f0b59084700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b54123c60 con 0x7f0b5411b4f0 2019-12-05T05:31:42.287 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:42.278+0000 7f0b5359e700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0b5411b4f0 0x7f0b541191b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:31:42.287 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:42.278+0000 7f0b52d9d700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f0b54122110 0x7f0b54122540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:31:42.287 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:42.278+0000 7f0b58883700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f0b54130420 0x7f0b54130850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:45166/0 (socket says 172.21.15.42:45166) 2019-12-05T05:31:42.287 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:42.278+0000 7f0b5359e700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0b5411b4f0 0x7f0b541191b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:36156/0 (socket says 172.21.15.42:36156) 2019-12-05T05:31:42.288 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:42.278+0000 7f0b58883700 1 -- 172.21.15.42:0/1769800850 learned_addr learned my addr 172.21.15.42:0/1769800850 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:31:42.288 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:42.278+0000 7f0b52d9d700 1 --2- 172.21.15.42:0/1769800850 >> v2:172.21.15.40:3300/0 conn(0x7f0b54122110 0x7f0b54122540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.42:51738/0 (socket says 172.21.15.42:51738) 2019-12-05T05:31:42.288 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:42.278+0000 7f0b52d9d700 1 -- 172.21.15.42:0/1769800850 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0b5411b4f0 msgr2=0x7f0b541191b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:31:42.288 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:42.278+0000 7f0b52d9d700 1 --2- 172.21.15.42:0/1769800850 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0b5411b4f0 0x7f0b541191b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:31:42.288 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:42.278+0000 7f0b52d9d700 1 -- 172.21.15.42:0/1769800850 >> v1:172.21.15.40:6789/0 conn(0x7f0b54130420 legacy=0x7f0b54130850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:31:42.288 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:42.278+0000 7f0b52d9d700 1 -- 172.21.15.42:0/1769800850 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b5411b920 con 0x7f0b54122110 2019-12-05T05:31:42.288 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:42.278+0000 7f0b52d9d700 1 --2- 172.21.15.42:0/1769800850 >> v2:172.21.15.40:3300/0 conn(0x7f0b54122110 0x7f0b54122540 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f0b4c0086f0 tx=0x7f0b4c00fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:42.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:42.278+0000 7f0b52d9d700 1 --2- 172.21.15.42:0/1769800850 >> v2:172.21.15.40:3300/0 conn(0x7f0b54122110 0x7f0b54122540 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f0b4c0086f0 tx=0x7f0b4c00fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:42.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:42.278+0000 7f0b52d9d700 1 --2- 172.21.15.42:0/1769800850 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0b54122110 0x7f0b54122540 secure :-1 s=READY pgs=809 cs=0 l=1 rx=0x7f0b4c0086f0 tx=0x7f0b4c00fff0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:31:42.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:42.278+0000 7f0b52d9d700 1 --2- 172.21.15.42:0/1769800850 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0b54122110 0x7f0b54122540 secure :-1 s=THROTTLE_DONE pgs=809 cs=0 l=1 rx=0x7f0b4c0086f0 tx=0x7f0b4c00fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:42.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:42.278+0000 7f0b52d9d700 1 --2- 172.21.15.42:0/1769800850 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0b54122110 0x7f0b54122540 secure :-1 s=THROTTLE_DONE pgs=809 cs=0 l=1 rx=0x7f0b4c0086f0 tx=0x7f0b4c00fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:42.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:42.278+0000 7f0b52d9d700 1 --2- 172.21.15.42:0/1769800850 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0b54122110 0x7f0b54122540 secure :-1 s=THROTTLE_DONE pgs=809 cs=0 l=1 rx=0x7f0b4c0086f0 tx=0x7f0b4c00fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:42.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:42.278+0000 7f0b4bfff700 1 -- 172.21.15.42:0/1769800850 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0b44005840 con 0x7f0b54122110 2019-12-05T05:31:42.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:42.278+0000 7f0b4bfff700 1 -- 172.21.15.42:0/1769800850 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b44007160 con 0x7f0b54122110 2019-12-05T05:31:42.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:42.278+0000 7f0b59084700 1 -- 172.21.15.42:0/1769800850 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b54123f00 con 0x7f0b54122110 2019-12-05T05:31:42.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:42.278+0000 7f0b4bfff700 1 -- 172.21.15.42:0/1769800850 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0b44012250 con 0x7f0b54122110 2019-12-05T05:31:42.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:42.278+0000 7f0b59084700 1 -- 172.21.15.42:0/1769800850 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b54123590 con 0x7f0b54122110 2019-12-05T05:31:42.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:42.282+0000 7f0b52d9d700 1 --2- 172.21.15.42:0/1769800850 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0b54122110 0x7f0b54122540 secure :-1 s=THROTTLE_DONE pgs=809 cs=0 l=1 rx=0x7f0b4c0086f0 tx=0x7f0b4c00fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:42.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:42.282+0000 7f0b4bfff700 1 -- 172.21.15.42:0/1769800850 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f0b44012560 con 0x7f0b54122110 2019-12-05T05:31:42.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:42.282+0000 7f0b52d9d700 1 --2- 172.21.15.42:0/1769800850 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0b54122110 0x7f0b54122540 secure :-1 s=THROTTLE_DONE pgs=809 cs=0 l=1 rx=0x7f0b4c0086f0 tx=0x7f0b4c00fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:42.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:42.282+0000 7f0b4bfff700 1 --2- 172.21.15.42:0/1769800850 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0b3803ad80 0x7f0b3803d290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:31:42.291 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:42.282+0000 7f0b4bfff700 1 -- 172.21.15.42:0/1769800850 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 7665+0+0 (secure 0 0 0) 0x7f0b4404b5a0 con 0x7f0b54122110 2019-12-05T05:31:42.291 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:42.282+0000 7f0b5359e700 1 --2- 172.21.15.42:0/1769800850 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0b3803ad80 0x7f0b3803d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:31:42.291 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:42.282+0000 7f0b5359e700 1 --2- 172.21.15.42:0/1769800850 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0b3803ad80 0x7f0b3803d290 crc :-1 s=READY pgs=444 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:31:42.291 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:42.286+0000 7f0b59084700 1 -- 172.21.15.42:0/1769800850 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b5405d110 con 0x7f0b54122110 2019-12-05T05:31:42.296 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:42.294+0000 7f0b52d9d700 1 --2- 172.21.15.42:0/1769800850 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0b54122110 0x7f0b54122540 secure :-1 s=THROTTLE_DONE pgs=809 cs=0 l=1 rx=0x7f0b4c0086f0 tx=0x7f0b4c00fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:42.296 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:42.294+0000 7f0b4bfff700 1 -- 172.21.15.42:0/1769800850 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f0b44032620 con 0x7f0b54122110 2019-12-05T05:31:42.467 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:42.462+0000 7f0b59084700 1 -- 172.21.15.42:0/1769800850 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0b5405b8c0 con 0x7f0b54122110 2019-12-05T05:31:42.469 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:42.466+0000 7f0b52d9d700 1 --2- 172.21.15.42:0/1769800850 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0b54122110 0x7f0b54122540 secure :-1 s=THROTTLE_DONE pgs=809 cs=0 l=1 rx=0x7f0b4c0086f0 tx=0x7f0b4c00fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:42.470 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:42.466+0000 7f0b4bfff700 1 -- 172.21.15.42:0/1769800850 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 128 v128) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f0b44032880 con 0x7f0b54122110 2019-12-05T05:31:42.471 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 128 2019-12-05T05:31:42.471 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:31:42.471 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":128,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6984,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/3543033403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3543033403},{"type":"v1","addr":"172.21.15.40:6833","nonce":3543033403}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":128}],"filesystems":[{"mdsmap":{"epoch":127,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:30:53.739046+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6778},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6778":{"gid":6778,"name":"a","rank":0,"incarnation":124,"state":"up:active","state_seq":23,"addr":"172.21.15.42:6835/3028851337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3028851337},{"type":"v1","addr":"172.21.15.42:6835","nonce":3028851337}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:31:42.474 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:42.470+0000 7f0b59084700 1 -- 172.21.15.42:0/1769800850 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0b3803ad80 msgr2=0x7f0b3803d290 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:31:42.474 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:42.470+0000 7f0b59084700 1 --2- 172.21.15.42:0/1769800850 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0b3803ad80 0x7f0b3803d290 crc :-1 s=READY pgs=444 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:31:42.475 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:42.470+0000 7f0b59084700 1 -- 172.21.15.42:0/1769800850 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0b54122110 msgr2=0x7f0b54122540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:31:42.475 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:42.470+0000 7f0b59084700 1 --2- 172.21.15.42:0/1769800850 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0b54122110 0x7f0b54122540 secure :-1 s=READY pgs=809 cs=0 l=1 rx=0x7f0b4c0086f0 tx=0x7f0b4c00fff0).stop 2019-12-05T05:31:42.475 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:42.470+0000 7f0b59084700 1 -- 172.21.15.42:0/1769800850 shutdown_connections 2019-12-05T05:31:42.476 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:42.470+0000 7f0b59084700 1 --2- 172.21.15.42:0/1769800850 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0b3803ad80 0x7f0b3803d290 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:31:42.476 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:42.470+0000 7f0b59084700 1 --2- 172.21.15.42:0/1769800850 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0b5411b4f0 0x7f0b541191b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:31:42.476 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:42.470+0000 7f0b59084700 1 --2- 172.21.15.42:0/1769800850 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0b54122110 0x7f0b54122540 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:31:42.477 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:42.470+0000 7f0b59084700 1 -- 172.21.15.42:0/1769800850 shutdown_connections 2019-12-05T05:31:42.477 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:42.470+0000 7f0b59084700 1 -- 172.21.15.42:0/1769800850 wait complete. 2019-12-05T05:31:42.477 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:42.470+0000 7f0b59084700 1 -- 172.21.15.42:0/1769800850 >> 172.21.15.42:0/1769800850 conn(0x7f0b540a9bc0 msgr2=0x7f0b54059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:31:42.484 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T05:31:44.486 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:31:44.591 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:44.578+0000 7f02903ba700 1 Processor -- start 2019-12-05T05:31:44.591 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:44.578+0000 7f02903ba700 1 -- start start 2019-12-05T05:31:44.592 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:44.582+0000 7f02903ba700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f02880a7740 0x7f028808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:31:44.592 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:44.582+0000 7f02903ba700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02880a0710 con 0x7f028807e720 2019-12-05T05:31:44.592 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:44.582+0000 7f02903ba700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02880a9ba0 con 0x7f028809fae0 2019-12-05T05:31:44.592 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:44.582+0000 7f02903ba700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02880a0b00 con 0x7f02880a7740 2019-12-05T05:31:44.593 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:44.582+0000 7f028f158700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f028809fae0 0x7f028814efe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:35738/0 (socket says 172.21.15.42:35738) 2019-12-05T05:31:44.593 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:44.582+0000 7f028f158700 1 -- 172.21.15.42:0/726031566 learned_addr learned my addr 172.21.15.42:0/726031566 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:31:44.593 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:44.582+0000 7f028e957700 1 --2- 172.21.15.42:0/726031566 >> v2:172.21.15.40:3300/0 conn(0x7f02880a7740 0x7f028808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:31:44.594 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:44.582+0000 7f028d955700 1 -- 172.21.15.42:0/726031566 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 372494430 0 0) 0x7f02880a9ba0 con 0x7f028809fae0 2019-12-05T05:31:44.594 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:44.582+0000 7f028d955700 1 -- 172.21.15.42:0/726031566 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02740030b0 con 0x7f028809fae0 2019-12-05T05:31:44.595 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:44.582+0000 7f028e957700 1 -- 172.21.15.42:0/726031566 >> v1:172.21.15.42:6789/0 conn(0x7f028809fae0 legacy=0x7f028814efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:31:44.595 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:44.582+0000 7f028e957700 1 -- 172.21.15.42:0/726031566 >> v1:172.21.15.40:6789/0 conn(0x7f028807e720 legacy=0x7f028809f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:31:44.595 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:44.582+0000 7f028e957700 1 -- 172.21.15.42:0/726031566 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f028807eb50 con 0x7f02880a7740 2019-12-05T05:31:44.596 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:44.582+0000 7f028e957700 1 --2- 172.21.15.42:0/726031566 >> v2:172.21.15.40:3300/0 conn(0x7f02880a7740 0x7f028808df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f0284008790 tx=0x7f028400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:44.596 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:44.582+0000 7f028e957700 1 --2- 172.21.15.42:0/726031566 >> v2:172.21.15.40:3300/0 conn(0x7f02880a7740 0x7f028808df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f0284008790 tx=0x7f028400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:44.597 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:44.582+0000 7f028e957700 1 --2- 172.21.15.42:0/726031566 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f02880a7740 0x7f028808df80 secure :-1 s=READY pgs=811 cs=0 l=1 rx=0x7f0284008790 tx=0x7f028400fbf0).ready entity=mon.0 client_cookie=22cda8e04cde17bf server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:31:44.597 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:44.582+0000 7f028e957700 1 --2- 172.21.15.42:0/726031566 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f02880a7740 0x7f028808df80 secure :-1 s=THROTTLE_DONE pgs=811 cs=0 l=1 rx=0x7f0284008790 tx=0x7f028400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:44.597 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:44.582+0000 7f028d955700 1 -- 172.21.15.42:0/726031566 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0284011270 con 0x7f02880a7740 2019-12-05T05:31:44.598 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:44.582+0000 7f028e957700 1 --2- 172.21.15.42:0/726031566 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f02880a7740 0x7f028808df80 secure :-1 s=THROTTLE_DONE pgs=811 cs=0 l=1 rx=0x7f0284008790 tx=0x7f028400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:44.599 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:44.582+0000 7f028e957700 1 --2- 172.21.15.42:0/726031566 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f02880a7740 0x7f028808df80 secure :-1 s=THROTTLE_DONE pgs=811 cs=0 l=1 rx=0x7f0284008790 tx=0x7f028400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:44.601 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:44.582+0000 7f028d955700 1 -- 172.21.15.42:0/726031566 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0284008bc0 con 0x7f02880a7740 2019-12-05T05:31:44.601 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:44.582+0000 7f028d955700 1 -- 172.21.15.42:0/726031566 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0284011850 con 0x7f02880a7740 2019-12-05T05:31:44.601 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:44.582+0000 7f02903ba700 1 -- 172.21.15.42:0/726031566 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f02880a7740 msgr2=0x7f028808df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:31:44.601 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:44.582+0000 7f02903ba700 1 --2- 172.21.15.42:0/726031566 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f02880a7740 0x7f028808df80 secure :-1 s=READY pgs=811 cs=0 l=1 rx=0x7f0284008790 tx=0x7f028400fbf0).stop 2019-12-05T05:31:44.603 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:44.586+0000 7f02903ba700 1 -- 172.21.15.42:0/726031566 shutdown_connections 2019-12-05T05:31:44.603 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:44.586+0000 7f02903ba700 1 --2- 172.21.15.42:0/726031566 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f02880a7740 0x7f028808df80 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:31:44.604 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:44.586+0000 7f02903ba700 1 -- 172.21.15.42:0/726031566 shutdown_connections 2019-12-05T05:31:44.604 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:44.586+0000 7f02903ba700 1 -- 172.21.15.42:0/726031566 wait complete. 2019-12-05T05:31:44.604 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:44.586+0000 7f02903ba700 1 -- 172.21.15.42:0/726031566 >> 172.21.15.42:0/726031566 conn(0x7f02880aabe0 msgr2=0x7f02880a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:31:44.604 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:44.586+0000 7f02903ba700 1 Processor -- start 2019-12-05T05:31:44.605 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:44.586+0000 7f02903ba700 1 -- start start 2019-12-05T05:31:44.610 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:44.586+0000 7f02903ba700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0288119cb0 0x7f028807d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:31:44.611 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:44.586+0000 7f02903ba700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0288142fc0 0x7f028807dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:31:44.611 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:44.586+0000 7f02903ba700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f028811a1a0 con 0x7f02881481e0 2019-12-05T05:31:44.611 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:44.586+0000 7f02903ba700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02881425d0 con 0x7f0288119cb0 2019-12-05T05:31:44.611 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:44.586+0000 7f02903ba700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0288142830 con 0x7f0288142fc0 2019-12-05T05:31:44.612 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:44.586+0000 7f028e957700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0288119cb0 0x7f028807d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:31:44.612 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:44.586+0000 7f028f158700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f02881481e0 0x7f02881436f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:35742/0 (socket says 172.21.15.42:35742) 2019-12-05T05:31:44.612 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:44.586+0000 7f028e957700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0288119cb0 0x7f028807d850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:35702/0 (socket says 172.21.15.42:35702) 2019-12-05T05:31:44.612 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:44.586+0000 7f028f158700 1 -- 172.21.15.42:0/3806887266 learned_addr learned my addr 172.21.15.42:0/3806887266 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:31:44.613 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:44.586+0000 7f028e156700 1 --2- 172.21.15.42:0/3806887266 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0288142fc0 0x7f028807dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:31:44.613 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:44.590+0000 7f028e957700 1 -- 172.21.15.42:0/3806887266 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0288142fc0 msgr2=0x7f028807dda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:31:44.613 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:44.590+0000 7f028e957700 1 --2- 172.21.15.42:0/3806887266 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0288142fc0 0x7f028807dda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:31:44.613 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:44.590+0000 7f027f7fe700 1 -- 172.21.15.42:0/3806887266 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 323251436 0 0) 0x7f028811a1a0 con 0x7f02881481e0 2019-12-05T05:31:44.614 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:44.590+0000 7f028e957700 1 -- 172.21.15.42:0/3806887266 >> v1:172.21.15.42:6789/0 conn(0x7f02881481e0 legacy=0x7f02881436f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:31:44.614 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:44.590+0000 7f028e957700 1 -- 172.21.15.42:0/3806887266 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0288143ef0 con 0x7f0288119cb0 2019-12-05T05:31:44.614 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:44.590+0000 7f028e957700 1 --2- 172.21.15.42:0/3806887266 >> v2:172.21.15.42:3300/0 conn(0x7f0288119cb0 0x7f028807d850 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f02881429f0 tx=0x7f0284007730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:44.614 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:44.590+0000 7f028e957700 1 --2- 172.21.15.42:0/3806887266 >> v2:172.21.15.42:3300/0 conn(0x7f0288119cb0 0x7f028807d850 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f02881429f0 tx=0x7f0284007730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:44.615 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:44.590+0000 7f028e957700 1 --2- 172.21.15.42:0/3806887266 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0288119cb0 0x7f028807d850 secure :-1 s=READY pgs=1043 cs=0 l=1 rx=0x7f02881429f0 tx=0x7f0284007730).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:31:44.624 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:44.590+0000 7f028e957700 1 --2- 172.21.15.42:0/3806887266 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0288119cb0 0x7f028807d850 secure :-1 s=THROTTLE_DONE pgs=1043 cs=0 l=1 rx=0x7f02881429f0 tx=0x7f0284007730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:44.624 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:44.590+0000 7f027f7fe700 1 -- 172.21.15.42:0/3806887266 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f028400f360 con 0x7f0288119cb0 2019-12-05T05:31:44.624 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:44.590+0000 7f028e957700 1 --2- 172.21.15.42:0/3806887266 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0288119cb0 0x7f028807d850 secure :-1 s=THROTTLE_DONE pgs=1043 cs=0 l=1 rx=0x7f02881429f0 tx=0x7f0284007730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:44.625 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:44.590+0000 7f028e957700 1 --2- 172.21.15.42:0/3806887266 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0288119cb0 0x7f028807d850 secure :-1 s=THROTTLE_DONE pgs=1043 cs=0 l=1 rx=0x7f02881429f0 tx=0x7f0284007730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:44.625 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:44.590+0000 7f027f7fe700 1 -- 172.21.15.42:0/3806887266 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0284011470 con 0x7f0288119cb0 2019-12-05T05:31:44.625 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:44.590+0000 7f027f7fe700 1 -- 172.21.15.42:0/3806887266 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0284016720 con 0x7f0288119cb0 2019-12-05T05:31:44.625 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:44.590+0000 7f02903ba700 1 -- 172.21.15.42:0/3806887266 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f028808eb70 con 0x7f0288119cb0 2019-12-05T05:31:44.626 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:44.590+0000 7f02903ba700 1 -- 172.21.15.42:0/3806887266 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f028808f0a0 con 0x7f0288119cb0 2019-12-05T05:31:44.626 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:44.590+0000 7f028e957700 1 --2- 172.21.15.42:0/3806887266 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0288119cb0 0x7f028807d850 secure :-1 s=THROTTLE_DONE pgs=1043 cs=0 l=1 rx=0x7f02881429f0 tx=0x7f0284007730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:44.626 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:44.590+0000 7f027f7fe700 1 -- 172.21.15.42:0/3806887266 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f0284007170 con 0x7f0288119cb0 2019-12-05T05:31:44.626 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:44.590+0000 7f028e957700 1 --2- 172.21.15.42:0/3806887266 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0288119cb0 0x7f028807d850 secure :-1 s=THROTTLE_DONE pgs=1043 cs=0 l=1 rx=0x7f02881429f0 tx=0x7f0284007730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:44.626 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:44.590+0000 7f027f7fe700 1 --2- 172.21.15.42:0/3806887266 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f027003ac80 0x7f027003d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:31:44.628 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:44.590+0000 7f027f7fe700 1 -- 172.21.15.42:0/3806887266 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 7665+0+0 (secure 0 0 0) 0x7f028404d600 con 0x7f0288119cb0 2019-12-05T05:31:44.628 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:44.590+0000 7f028e156700 1 --2- 172.21.15.42:0/3806887266 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f027003ac80 0x7f027003d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:31:44.628 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:44.594+0000 7f028e156700 1 --2- 172.21.15.42:0/3806887266 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f027003ac80 0x7f027003d190 crc :-1 s=READY pgs=445 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:31:44.629 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:44.594+0000 7f02903ba700 1 -- 172.21.15.42:0/3806887266 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02880a3890 con 0x7f0288119cb0 2019-12-05T05:31:44.629 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:44.602+0000 7f028e957700 1 --2- 172.21.15.42:0/3806887266 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0288119cb0 0x7f028807d850 secure :-1 s=THROTTLE_DONE pgs=1043 cs=0 l=1 rx=0x7f02881429f0 tx=0x7f0284007730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:44.629 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:44.602+0000 7f027f7fe700 1 -- 172.21.15.42:0/3806887266 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f0284034680 con 0x7f0288119cb0 2019-12-05T05:31:44.780 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:44.778+0000 7f02903ba700 1 -- 172.21.15.42:0/3806887266 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f028805d110 con 0x7f0288119cb0 2019-12-05T05:31:44.780 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:44.778+0000 7f028e957700 1 --2- 172.21.15.42:0/3806887266 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0288119cb0 0x7f028807d850 secure :-1 s=THROTTLE_DONE pgs=1043 cs=0 l=1 rx=0x7f02881429f0 tx=0x7f0284007730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:44.780 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:44.778+0000 7f027f7fe700 1 -- 172.21.15.42:0/3806887266 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 128 v128) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f02840322a0 con 0x7f0288119cb0 2019-12-05T05:31:44.784 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:31:44.784 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":128,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6984,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/3543033403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3543033403},{"type":"v1","addr":"172.21.15.40:6833","nonce":3543033403}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":128}],"filesystems":[{"mdsmap":{"epoch":127,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:30:53.739046+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6778},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6778":{"gid":6778,"name":"a","rank":0,"incarnation":124,"state":"up:active","state_seq":23,"addr":"172.21.15.42:6835/3028851337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3028851337},{"type":"v1","addr":"172.21.15.42:6835","nonce":3028851337}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:31:44.785 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 128 2019-12-05T05:31:44.785 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:44.778+0000 7f02903ba700 1 -- 172.21.15.42:0/3806887266 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f027003ac80 msgr2=0x7f027003d190 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:31:44.785 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:44.778+0000 7f02903ba700 1 --2- 172.21.15.42:0/3806887266 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f027003ac80 0x7f027003d190 crc :-1 s=READY pgs=445 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:31:44.785 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:44.778+0000 7f02903ba700 1 -- 172.21.15.42:0/3806887266 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0288119cb0 msgr2=0x7f028807d850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:31:44.786 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:44.778+0000 7f02903ba700 1 --2- 172.21.15.42:0/3806887266 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0288119cb0 0x7f028807d850 secure :-1 s=READY pgs=1043 cs=0 l=1 rx=0x7f02881429f0 tx=0x7f0284007730).stop 2019-12-05T05:31:44.786 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:44.778+0000 7f02903ba700 1 -- 172.21.15.42:0/3806887266 shutdown_connections 2019-12-05T05:31:44.786 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:44.778+0000 7f02903ba700 1 --2- 172.21.15.42:0/3806887266 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f027003ac80 0x7f027003d190 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:31:44.786 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:44.778+0000 7f02903ba700 1 --2- 172.21.15.42:0/3806887266 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0288119cb0 0x7f028807d850 unknown :-1 s=CLOSED pgs=1043 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:31:44.787 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:44.778+0000 7f02903ba700 1 --2- 172.21.15.42:0/3806887266 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0288142fc0 0x7f028807dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:31:44.787 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:44.778+0000 7f02903ba700 1 -- 172.21.15.42:0/3806887266 shutdown_connections 2019-12-05T05:31:44.787 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:44.778+0000 7f02903ba700 1 -- 172.21.15.42:0/3806887266 wait complete. 2019-12-05T05:31:44.787 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:44.778+0000 7f02903ba700 1 -- 172.21.15.42:0/3806887266 >> 172.21.15.42:0/3806887266 conn(0x7f02880a6c30 msgr2=0x7f02880a4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:31:44.793 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T05:31:46.797 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:31:46.905 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:46.894+0000 7f2bb423f700 1 Processor -- start 2019-12-05T05:31:46.905 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:46.894+0000 7f2bb423f700 1 -- start start 2019-12-05T05:31:46.906 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:46.898+0000 7f2bb423f700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2bac09fad0 0x7f2bac14ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:31:46.906 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:46.898+0000 7f2bb423f700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bac0a0700 con 0x7f2bac0a7730 2019-12-05T05:31:46.906 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:46.898+0000 7f2bb423f700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bac07ef10 con 0x7f2bac08df70 2019-12-05T05:31:46.906 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:46.898+0000 7f2bb423f700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2bac0a0af0 con 0x7f2bac09fad0 2019-12-05T05:31:46.906 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:46.898+0000 7f2bb1fdb700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f2bac08df70 0x7f2bac09f2d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:35752/0 (socket says 172.21.15.42:35752) 2019-12-05T05:31:46.906 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:46.898+0000 7f2bb1fdb700 1 -- 172.21.15.42:0/2391245229 learned_addr learned my addr 172.21.15.42:0/2391245229 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:31:46.907 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:46.898+0000 7f2bb2fdd700 1 --2- 172.21.15.42:0/2391245229 >> v2:172.21.15.40:3300/0 conn(0x7f2bac09fad0 0x7f2bac14ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:31:46.907 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:46.898+0000 7f2bb17da700 1 -- 172.21.15.42:0/2391245229 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3358730411 0 0) 0x7f2bac07ef10 con 0x7f2bac08df70 2019-12-05T05:31:46.907 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:46.898+0000 7f2bb17da700 1 -- 172.21.15.42:0/2391245229 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b980030b0 con 0x7f2bac08df70 2019-12-05T05:31:46.907 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:46.898+0000 7f2bb2fdd700 1 -- 172.21.15.42:0/2391245229 >> v1:172.21.15.42:6789/0 conn(0x7f2bac08df70 legacy=0x7f2bac09f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:31:46.907 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:46.898+0000 7f2bb2fdd700 1 -- 172.21.15.42:0/2391245229 >> v1:172.21.15.40:6789/0 conn(0x7f2bac0a7730 legacy=0x7f2bac07e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:31:46.907 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:46.898+0000 7f2bb17da700 1 -- 172.21.15.42:0/2391245229 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4271565345 0 0) 0x7f2bac0a0700 con 0x7f2bac0a7730 2019-12-05T05:31:46.907 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:46.898+0000 7f2bb2fdd700 1 -- 172.21.15.42:0/2391245229 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bac0a9b90 con 0x7f2bac09fad0 2019-12-05T05:31:46.908 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:46.898+0000 7f2bb2fdd700 1 --2- 172.21.15.42:0/2391245229 >> v2:172.21.15.40:3300/0 conn(0x7f2bac09fad0 0x7f2bac14ffe0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f2ba4008b10 tx=0x7f2ba400ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:46.908 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:46.898+0000 7f2bb2fdd700 1 --2- 172.21.15.42:0/2391245229 >> v2:172.21.15.40:3300/0 conn(0x7f2bac09fad0 0x7f2bac14ffe0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f2ba4008b10 tx=0x7f2ba400ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:46.908 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:46.898+0000 7f2bb2fdd700 1 --2- 172.21.15.42:0/2391245229 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2bac09fad0 0x7f2bac14ffe0 secure :-1 s=READY pgs=813 cs=0 l=1 rx=0x7f2ba4008b10 tx=0x7f2ba400ff70).ready entity=mon.0 client_cookie=8421078d186ba8d5 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:31:46.908 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:46.898+0000 7f2bb2fdd700 1 --2- 172.21.15.42:0/2391245229 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2bac09fad0 0x7f2bac14ffe0 secure :-1 s=THROTTLE_DONE pgs=813 cs=0 l=1 rx=0x7f2ba4008b10 tx=0x7f2ba400ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:46.908 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:46.898+0000 7f2bb2fdd700 1 --2- 172.21.15.42:0/2391245229 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2bac09fad0 0x7f2bac14ffe0 secure :-1 s=THROTTLE_DONE pgs=813 cs=0 l=1 rx=0x7f2ba4008b10 tx=0x7f2ba400ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:46.908 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:46.898+0000 7f2bb17da700 1 -- 172.21.15.42:0/2391245229 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2ba4010970 con 0x7f2bac09fad0 2019-12-05T05:31:46.908 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:46.898+0000 7f2bb2fdd700 1 --2- 172.21.15.42:0/2391245229 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2bac09fad0 0x7f2bac14ffe0 secure :-1 s=THROTTLE_DONE pgs=813 cs=0 l=1 rx=0x7f2ba4008b10 tx=0x7f2ba400ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:46.909 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:46.898+0000 7f2bb17da700 1 -- 172.21.15.42:0/2391245229 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ba4008b50 con 0x7f2bac09fad0 2019-12-05T05:31:46.909 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:46.898+0000 7f2bb17da700 1 -- 172.21.15.42:0/2391245229 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2ba40110c0 con 0x7f2bac09fad0 2019-12-05T05:31:46.909 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:46.898+0000 7f2bb423f700 1 -- 172.21.15.42:0/2391245229 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2bac09fad0 msgr2=0x7f2bac14ffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:31:46.909 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:46.898+0000 7f2bb423f700 1 --2- 172.21.15.42:0/2391245229 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2bac09fad0 0x7f2bac14ffe0 secure :-1 s=READY pgs=813 cs=0 l=1 rx=0x7f2ba4008b10 tx=0x7f2ba400ff70).stop 2019-12-05T05:31:46.909 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:46.902+0000 7f2bb423f700 1 -- 172.21.15.42:0/2391245229 shutdown_connections 2019-12-05T05:31:46.909 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:46.902+0000 7f2bb423f700 1 --2- 172.21.15.42:0/2391245229 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2bac09fad0 0x7f2bac14ffe0 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:31:46.909 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:46.902+0000 7f2bb423f700 1 -- 172.21.15.42:0/2391245229 shutdown_connections 2019-12-05T05:31:46.910 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:46.902+0000 7f2bb423f700 1 -- 172.21.15.42:0/2391245229 wait complete. 2019-12-05T05:31:46.910 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:46.902+0000 7f2bb423f700 1 -- 172.21.15.42:0/2391245229 >> 172.21.15.42:0/2391245229 conn(0x7f2bac0aabd0 msgr2=0x7f2bac0a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:31:46.913 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:46.902+0000 7f2bb423f700 1 Processor -- start 2019-12-05T05:31:46.913 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:46.902+0000 7f2bb423f700 1 -- start start 2019-12-05T05:31:46.913 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:46.902+0000 7f2bb423f700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2bac129070 0x7f2bac130d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:31:46.913 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:46.902+0000 7f2bb423f700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bac1294a0 con 0x7f2bac0a3950 2019-12-05T05:31:46.914 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:46.902+0000 7f2bb423f700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bac0658c0 con 0x7f2bac128c40 2019-12-05T05:31:46.914 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:46.902+0000 7f2bb423f700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2bac1267e0 con 0x7f2bac129070 2019-12-05T05:31:46.914 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:46.902+0000 7f2bb1fdb700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f2bac128c40 0x7f2bac14f930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:35758/0 (socket says 172.21.15.42:35758) 2019-12-05T05:31:46.914 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:46.902+0000 7f2bb1fdb700 1 -- 172.21.15.42:0/846586314 learned_addr learned my addr 172.21.15.42:0/846586314 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:31:46.914 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:46.902+0000 7f2bb2fdd700 1 --2- 172.21.15.42:0/846586314 >> v2:172.21.15.42:3300/0 conn(0x7f2bac129070 0x7f2bac130d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:31:46.914 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:46.906+0000 7f2bb2fdd700 1 -- 172.21.15.42:0/846586314 >> v1:172.21.15.42:6789/0 conn(0x7f2bac128c40 legacy=0x7f2bac14f930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:31:46.914 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:46.906+0000 7f2bb2fdd700 1 -- 172.21.15.42:0/846586314 >> v1:172.21.15.40:6789/0 conn(0x7f2bac0a3950 legacy=0x7f2bac0650c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:31:46.915 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:46.906+0000 7f2b9effd700 1 -- 172.21.15.42:0/846586314 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 980506545 0 0) 0x7f2bac0658c0 con 0x7f2bac128c40 2019-12-05T05:31:46.915 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:46.906+0000 7f2bb2fdd700 1 -- 172.21.15.42:0/846586314 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bac150130 con 0x7f2bac129070 2019-12-05T05:31:46.915 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:46.906+0000 7f2bb2fdd700 1 --2- 172.21.15.42:0/846586314 >> v2:172.21.15.42:3300/0 conn(0x7f2bac129070 0x7f2bac130d10 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f2ba80010e0 tx=0x7f2ba8001120).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:46.915 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:46.906+0000 7f2bb2fdd700 1 --2- 172.21.15.42:0/846586314 >> v2:172.21.15.42:3300/0 conn(0x7f2bac129070 0x7f2bac130d10 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f2ba80010e0 tx=0x7f2ba8001120).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:46.915 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:46.906+0000 7f2bb2fdd700 1 --2- 172.21.15.42:0/846586314 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2bac129070 0x7f2bac130d10 secure :-1 s=READY pgs=1046 cs=0 l=1 rx=0x7f2ba80010e0 tx=0x7f2ba8001120).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:31:46.915 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:46.906+0000 7f2bb2fdd700 1 --2- 172.21.15.42:0/846586314 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2bac129070 0x7f2bac130d10 secure :-1 s=THROTTLE_DONE pgs=1046 cs=0 l=1 rx=0x7f2ba80010e0 tx=0x7f2ba8001120).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:46.915 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:46.906+0000 7f2b9effd700 1 -- 172.21.15.42:0/846586314 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2ba4005ab0 con 0x7f2bac129070 2019-12-05T05:31:46.916 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:46.906+0000 7f2bb2fdd700 1 --2- 172.21.15.42:0/846586314 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2bac129070 0x7f2bac130d10 secure :-1 s=THROTTLE_DONE pgs=1046 cs=0 l=1 rx=0x7f2ba80010e0 tx=0x7f2ba8001120).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:46.916 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:46.906+0000 7f2bb2fdd700 1 --2- 172.21.15.42:0/846586314 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2bac129070 0x7f2bac130d10 secure :-1 s=THROTTLE_DONE pgs=1046 cs=0 l=1 rx=0x7f2ba80010e0 tx=0x7f2ba8001120).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:46.916 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:46.906+0000 7f2b9effd700 1 -- 172.21.15.42:0/846586314 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ba4010b70 con 0x7f2bac129070 2019-12-05T05:31:46.916 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:46.906+0000 7f2bb423f700 1 -- 172.21.15.42:0/846586314 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bac131260 con 0x7f2bac129070 2019-12-05T05:31:46.916 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:46.906+0000 7f2b9effd700 1 -- 172.21.15.42:0/846586314 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2ba400f680 con 0x7f2bac129070 2019-12-05T05:31:46.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:46.906+0000 7f2bb423f700 1 -- 172.21.15.42:0/846586314 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bac1317b0 con 0x7f2bac129070 2019-12-05T05:31:46.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:46.906+0000 7f2bb2fdd700 1 --2- 172.21.15.42:0/846586314 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2bac129070 0x7f2bac130d10 secure :-1 s=THROTTLE_DONE pgs=1046 cs=0 l=1 rx=0x7f2ba80010e0 tx=0x7f2ba8001120).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:46.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:46.906+0000 7f2b9effd700 1 -- 172.21.15.42:0/846586314 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f2ba4011390 con 0x7f2bac129070 2019-12-05T05:31:46.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:46.906+0000 7f2bb2fdd700 1 --2- 172.21.15.42:0/846586314 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2bac129070 0x7f2bac130d10 secure :-1 s=THROTTLE_DONE pgs=1046 cs=0 l=1 rx=0x7f2ba80010e0 tx=0x7f2ba8001120).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:46.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:46.906+0000 7f2b9effd700 1 --2- 172.21.15.42:0/846586314 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2b9403abf0 0x7f2b9403d100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:31:46.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:46.906+0000 7f2b9effd700 1 -- 172.21.15.42:0/846586314 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 7665+0+0 (secure 0 0 0) 0x7f2ba404afc0 con 0x7f2bac129070 2019-12-05T05:31:46.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:46.910+0000 7f2bb27dc700 1 --2- 172.21.15.42:0/846586314 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2b9403abf0 0x7f2b9403d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:31:46.918 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:46.910+0000 7f2bb27dc700 1 --2- 172.21.15.42:0/846586314 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2b9403abf0 0x7f2b9403d100 crc :-1 s=READY pgs=446 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:31:46.918 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:46.914+0000 7f2bb423f700 1 -- 172.21.15.42:0/846586314 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bac05d100 con 0x7f2bac129070 2019-12-05T05:31:46.925 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:46.922+0000 7f2bb2fdd700 1 --2- 172.21.15.42:0/846586314 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2bac129070 0x7f2bac130d10 secure :-1 s=THROTTLE_DONE pgs=1046 cs=0 l=1 rx=0x7f2ba80010e0 tx=0x7f2ba8001120).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:46.926 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:46.922+0000 7f2b9effd700 1 -- 172.21.15.42:0/846586314 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f2ba4011c90 con 0x7f2bac129070 2019-12-05T05:31:47.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:47.090+0000 7f2bb423f700 1 -- 172.21.15.42:0/846586314 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2bac05b8b0 con 0x7f2bac129070 2019-12-05T05:31:47.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:47.090+0000 7f2bb2fdd700 1 --2- 172.21.15.42:0/846586314 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2bac129070 0x7f2bac130d10 secure :-1 s=THROTTLE_DONE pgs=1046 cs=0 l=1 rx=0x7f2ba80010e0 tx=0x7f2ba8001120).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:47.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:47.090+0000 7f2b9effd700 1 -- 172.21.15.42:0/846586314 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 128 v128) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f2ba400f960 con 0x7f2bac129070 2019-12-05T05:31:47.099 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 128 2019-12-05T05:31:47.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:47.094+0000 7f2bb423f700 1 -- 172.21.15.42:0/846586314 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2b9403abf0 msgr2=0x7f2b9403d100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:31:47.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:47.094+0000 7f2bb423f700 1 --2- 172.21.15.42:0/846586314 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2b9403abf0 0x7f2b9403d100 crc :-1 s=READY pgs=446 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:31:47.100 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:47.094+0000 7f2bb423f700 1 -- 172.21.15.42:0/846586314 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2bac129070 msgr2=0x7f2bac130d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:31:47.100 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:47.094+0000 7f2bb423f700 1 --2- 172.21.15.42:0/846586314 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2bac129070 0x7f2bac130d10 secure :-1 s=READY pgs=1046 cs=0 l=1 rx=0x7f2ba80010e0 tx=0x7f2ba8001120).stop 2019-12-05T05:31:47.100 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:47.094+0000 7f2bb423f700 1 -- 172.21.15.42:0/846586314 shutdown_connections 2019-12-05T05:31:47.100 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:47.094+0000 7f2bb423f700 1 --2- 172.21.15.42:0/846586314 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2b9403abf0 0x7f2b9403d100 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:31:47.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:47.094+0000 7f2bb423f700 1 --2- 172.21.15.42:0/846586314 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2bac129070 0x7f2bac130d10 unknown :-1 s=CLOSED pgs=1046 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:31:47.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:47.094+0000 7f2bb423f700 1 -- 172.21.15.42:0/846586314 shutdown_connections 2019-12-05T05:31:47.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:47.094+0000 7f2bb423f700 1 -- 172.21.15.42:0/846586314 wait complete. 2019-12-05T05:31:47.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:47.094+0000 7f2bb423f700 1 -- 172.21.15.42:0/846586314 >> 172.21.15.42:0/846586314 conn(0x7f2bac0a7730 msgr2=0x7f2bac07e5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:31:47.102 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:31:47.102 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":128,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6984,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/3543033403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3543033403},{"type":"v1","addr":"172.21.15.40:6833","nonce":3543033403}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":128}],"filesystems":[{"mdsmap":{"epoch":127,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:30:53.739046+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6778},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6778":{"gid":6778,"name":"a","rank":0,"incarnation":124,"state":"up:active","state_seq":23,"addr":"172.21.15.42:6835/3028851337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3028851337},{"type":"v1","addr":"172.21.15.42:6835","nonce":3028851337}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:31:47.107 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T05:31:49.111 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:31:49.226 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.218+0000 7f6c629b9700 1 Processor -- start 2019-12-05T05:31:49.226 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.218+0000 7f6c629b9700 1 -- start start 2019-12-05T05:31:49.226 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.218+0000 7f6c629b9700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6c5c0a7740 0x7f6c5c08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:31:49.226 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.218+0000 7f6c629b9700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6c5c07e720 0x7f6c5c07eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:31:49.226 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.218+0000 7f6c629b9700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c5c0a0710 con 0x7f6c5c09f2e0 2019-12-05T05:31:49.227 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.218+0000 7f6c629b9700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c5c0a0b00 con 0x7f6c5c0a7740 2019-12-05T05:31:49.227 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.218+0000 7f6c629b9700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c5c08e5b0 con 0x7f6c5c07e720 2019-12-05T05:31:49.227 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.218+0000 7f6c60f56700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6c5c0a7740 0x7f6c5c08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:31:49.227 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.218+0000 7f6c61757700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f6c5c09f2e0 0x7f6c5c09f710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:35768/0 (socket says 172.21.15.42:35768) 2019-12-05T05:31:49.227 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.218+0000 7f6c60f56700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6c5c0a7740 0x7f6c5c08df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:35726/0 (socket says 172.21.15.42:35726) 2019-12-05T05:31:49.227 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.218+0000 7f6c61757700 1 -- 172.21.15.42:0/3876837954 learned_addr learned my addr 172.21.15.42:0/3876837954 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:31:49.228 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.218+0000 7f6c5bfff700 1 --2- 172.21.15.42:0/3876837954 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6c5c07e720 0x7f6c5c07eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:31:49.228 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.218+0000 7f6c5bfff700 1 -- 172.21.15.42:0/3876837954 >> v2:172.21.15.42:3300/0 conn(0x7f6c5c0a7740 msgr2=0x7f6c5c08df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:31:49.228 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.218+0000 7f6c5bfff700 1 --2- 172.21.15.42:0/3876837954 >> v2:172.21.15.42:3300/0 conn(0x7f6c5c0a7740 0x7f6c5c08df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:31:49.228 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.218+0000 7f6c5bfff700 1 -- 172.21.15.42:0/3876837954 >> v1:172.21.15.42:6789/0 conn(0x7f6c5c09f2e0 legacy=0x7f6c5c09f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:31:49.229 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.218+0000 7f6c5bfff700 1 -- 172.21.15.42:0/3876837954 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c5c07f0a0 con 0x7f6c5c07e720 2019-12-05T05:31:49.229 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.218+0000 7f6c60f56700 1 --2- 172.21.15.42:0/3876837954 >> v2:172.21.15.42:3300/0 conn(0x7f6c5c0a7740 0x7f6c5c08df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:31:49.229 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.218+0000 7f6c5bfff700 1 --2- 172.21.15.42:0/3876837954 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6c5c07e720 0x7f6c5c07eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f6c50008b10 tx=0x7f6c5000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:49.229 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.218+0000 7f6c5bfff700 1 --2- 172.21.15.42:0/3876837954 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6c5c07e720 0x7f6c5c07eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f6c50008b10 tx=0x7f6c5000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:49.229 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.218+0000 7f6c5bfff700 1 --2- 172.21.15.42:0/3876837954 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6c5c07e720 0x7f6c5c07eb50 secure :-1 s=READY pgs=212 cs=0 l=1 rx=0x7f6c50008b10 tx=0x7f6c5000fff0).ready entity=mon.2 client_cookie=dc61255aea35cfd9 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:31:49.229 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.218+0000 7f6c5bfff700 1 --2- 172.21.15.42:0/3876837954 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6c5c07e720 0x7f6c5c07eb50 secure :-1 s=THROTTLE_DONE pgs=212 cs=0 l=1 rx=0x7f6c50008b10 tx=0x7f6c5000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:49.230 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.222+0000 7f6c5bfff700 1 --2- 172.21.15.42:0/3876837954 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6c5c07e720 0x7f6c5c07eb50 secure :-1 s=THROTTLE_DONE pgs=212 cs=0 l=1 rx=0x7f6c50008b10 tx=0x7f6c5000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:49.230 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.222+0000 7f6c5bfff700 1 --2- 172.21.15.42:0/3876837954 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6c5c07e720 0x7f6c5c07eb50 secure :-1 s=THROTTLE_DONE pgs=212 cs=0 l=1 rx=0x7f6c50008b10 tx=0x7f6c5000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:49.230 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.222+0000 7f6c5b7fe700 1 -- 172.21.15.42:0/3876837954 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6c50011a50 con 0x7f6c5c07e720 2019-12-05T05:31:49.230 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.222+0000 7f6c5b7fe700 1 -- 172.21.15.42:0/3876837954 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c50011670 con 0x7f6c5c07e720 2019-12-05T05:31:49.230 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.222+0000 7f6c5b7fe700 1 -- 172.21.15.42:0/3876837954 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6c5000a350 con 0x7f6c5c07e720 2019-12-05T05:31:49.230 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.222+0000 7f6c629b9700 1 -- 172.21.15.42:0/3876837954 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6c5c07e720 msgr2=0x7f6c5c07eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:31:49.230 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.222+0000 7f6c629b9700 1 --2- 172.21.15.42:0/3876837954 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6c5c07e720 0x7f6c5c07eb50 secure :-1 s=READY pgs=212 cs=0 l=1 rx=0x7f6c50008b10 tx=0x7f6c5000fff0).stop 2019-12-05T05:31:49.231 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.222+0000 7f6c629b9700 1 -- 172.21.15.42:0/3876837954 shutdown_connections 2019-12-05T05:31:49.231 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.222+0000 7f6c629b9700 1 --2- 172.21.15.42:0/3876837954 >> v2:172.21.15.42:3300/0 conn(0x7f6c5c0a7740 0x7f6c5c08df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:31:49.231 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.222+0000 7f6c629b9700 1 --2- 172.21.15.42:0/3876837954 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6c5c07e720 0x7f6c5c07eb50 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:31:49.231 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.222+0000 7f6c629b9700 1 -- 172.21.15.42:0/3876837954 shutdown_connections 2019-12-05T05:31:49.231 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.222+0000 7f6c629b9700 1 -- 172.21.15.42:0/3876837954 wait complete. 2019-12-05T05:31:49.231 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.222+0000 7f6c629b9700 1 -- 172.21.15.42:0/3876837954 >> 172.21.15.42:0/3876837954 conn(0x7f6c5c0aabe0 msgr2=0x7f6c5c0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:31:49.238 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.222+0000 7f6c629b9700 1 Processor -- start 2019-12-05T05:31:49.238 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.222+0000 7f6c629b9700 1 -- start start 2019-12-05T05:31:49.239 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.226+0000 7f6c629b9700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6c5c116480 0x7f6c5c11bd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:31:49.239 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.226+0000 7f6c629b9700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c5c11a740 con 0x7f6c5c11a310 2019-12-05T05:31:49.239 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.226+0000 7f6c629b9700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c5c0a6f80 con 0x7f6c5c1225f0 2019-12-05T05:31:49.239 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.226+0000 7f6c629b9700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c5c0a71b0 con 0x7f6c5c116480 2019-12-05T05:31:49.240 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.226+0000 7f6c60f56700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f6c5c1225f0 0x7f6c5c11ecf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:35770/0 (socket says 172.21.15.42:35770) 2019-12-05T05:31:49.240 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.226+0000 7f6c60f56700 1 -- 172.21.15.42:0/39610039 learned_addr learned my addr 172.21.15.42:0/39610039 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:31:49.240 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.226+0000 7f6c5bfff700 1 --2- 172.21.15.42:0/39610039 >> v2:172.21.15.42:3300/0 conn(0x7f6c5c116480 0x7f6c5c11bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:31:49.240 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.226+0000 7f6c597fa700 1 -- 172.21.15.42:0/39610039 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3505098334 0 0) 0x7f6c5c0a6f80 con 0x7f6c5c1225f0 2019-12-05T05:31:49.240 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.226+0000 7f6c597fa700 1 -- 172.21.15.42:0/39610039 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c3c0030b0 con 0x7f6c5c1225f0 2019-12-05T05:31:49.240 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.226+0000 7f6c597fa700 1 -- 172.21.15.42:0/39610039 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3297412297 0 0) 0x7f6c5c11a740 con 0x7f6c5c11a310 2019-12-05T05:31:49.241 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.226+0000 7f6c597fa700 1 -- 172.21.15.42:0/39610039 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c5c0a6f80 con 0x7f6c5c11a310 2019-12-05T05:31:49.241 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.226+0000 7f6c597fa700 1 -- 172.21.15.42:0/39610039 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 4207534980 0 0) 0x7f6c3c0030b0 con 0x7f6c5c1225f0 2019-12-05T05:31:49.241 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.226+0000 7f6c5bfff700 1 -- 172.21.15.42:0/39610039 >> v1:172.21.15.42:6789/0 conn(0x7f6c5c1225f0 legacy=0x7f6c5c11ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:31:49.241 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.226+0000 7f6c5bfff700 1 -- 172.21.15.42:0/39610039 >> v1:172.21.15.40:6789/0 conn(0x7f6c5c11a310 legacy=0x7f6c5c11f4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:31:49.251 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.226+0000 7f6c5bfff700 1 -- 172.21.15.42:0/39610039 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c5c07a330 con 0x7f6c5c116480 2019-12-05T05:31:49.252 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.226+0000 7f6c5bfff700 1 --2- 172.21.15.42:0/39610039 >> v2:172.21.15.42:3300/0 conn(0x7f6c5c116480 0x7f6c5c11bd30 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f6c4c007940 tx=0x7f6c4c007720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:49.252 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.226+0000 7f6c5bfff700 1 --2- 172.21.15.42:0/39610039 >> v2:172.21.15.42:3300/0 conn(0x7f6c5c116480 0x7f6c5c11bd30 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f6c4c007940 tx=0x7f6c4c007720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:49.252 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.226+0000 7f6c5bfff700 1 --2- 172.21.15.42:0/39610039 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6c5c116480 0x7f6c5c11bd30 secure :-1 s=READY pgs=1049 cs=0 l=1 rx=0x7f6c4c007940 tx=0x7f6c4c007720).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:31:49.252 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.226+0000 7f6c5bfff700 1 --2- 172.21.15.42:0/39610039 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6c5c116480 0x7f6c5c11bd30 secure :-1 s=THROTTLE_DONE pgs=1049 cs=0 l=1 rx=0x7f6c4c007940 tx=0x7f6c4c007720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:49.252 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.226+0000 7f6c597fa700 1 -- 172.21.15.42:0/39610039 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6c4c0024c0 con 0x7f6c5c116480 2019-12-05T05:31:49.253 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.226+0000 7f6c5bfff700 1 --2- 172.21.15.42:0/39610039 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6c5c116480 0x7f6c5c11bd30 secure :-1 s=THROTTLE_DONE pgs=1049 cs=0 l=1 rx=0x7f6c4c007940 tx=0x7f6c4c007720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:49.253 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.226+0000 7f6c597fa700 1 -- 172.21.15.42:0/39610039 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c4c0022d0 con 0x7f6c5c116480 2019-12-05T05:31:49.253 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.226+0000 7f6c629b9700 1 -- 172.21.15.42:0/39610039 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c5c078310 con 0x7f6c5c116480 2019-12-05T05:31:49.253 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.226+0000 7f6c5bfff700 1 --2- 172.21.15.42:0/39610039 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6c5c116480 0x7f6c5c11bd30 secure :-1 s=THROTTLE_DONE pgs=1049 cs=0 l=1 rx=0x7f6c4c007940 tx=0x7f6c4c007720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:49.253 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.226+0000 7f6c597fa700 1 -- 172.21.15.42:0/39610039 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6c50005ca0 con 0x7f6c5c116480 2019-12-05T05:31:49.253 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.226+0000 7f6c629b9700 1 -- 172.21.15.42:0/39610039 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c5c078810 con 0x7f6c5c116480 2019-12-05T05:31:49.254 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.226+0000 7f6c5bfff700 1 --2- 172.21.15.42:0/39610039 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6c5c116480 0x7f6c5c11bd30 secure :-1 s=THROTTLE_DONE pgs=1049 cs=0 l=1 rx=0x7f6c4c007940 tx=0x7f6c4c007720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:49.254 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.230+0000 7f6c597fa700 1 -- 172.21.15.42:0/39610039 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f6c50007150 con 0x7f6c5c116480 2019-12-05T05:31:49.254 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.230+0000 7f6c5bfff700 1 --2- 172.21.15.42:0/39610039 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6c5c116480 0x7f6c5c11bd30 secure :-1 s=THROTTLE_DONE pgs=1049 cs=0 l=1 rx=0x7f6c4c007940 tx=0x7f6c4c007720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:49.254 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.230+0000 7f6c597fa700 1 --2- 172.21.15.42:0/39610039 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6c3c03b3a0 0x7f6c3c03d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:31:49.254 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.230+0000 7f6c597fa700 1 -- 172.21.15.42:0/39610039 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7751+0+0 (secure 0 0 0) 0x7f6c5004aee0 con 0x7f6c5c116480 2019-12-05T05:31:49.254 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.230+0000 7f6c60f56700 1 --2- 172.21.15.42:0/39610039 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6c3c03b3a0 0x7f6c3c03d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:31:49.255 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.230+0000 7f6c60f56700 1 --2- 172.21.15.42:0/39610039 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6c3c03b3a0 0x7f6c3c03d8b0 crc :-1 s=READY pgs=447 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:31:49.255 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.234+0000 7f6c629b9700 1 -- 172.21.15.42:0/39610039 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c5c05d110 con 0x7f6c5c116480 2019-12-05T05:31:49.256 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.242+0000 7f6c5bfff700 1 --2- 172.21.15.42:0/39610039 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6c5c116480 0x7f6c5c11bd30 secure :-1 s=THROTTLE_DONE pgs=1049 cs=0 l=1 rx=0x7f6c4c007940 tx=0x7f6c4c007720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:49.256 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.242+0000 7f6c597fa700 1 -- 172.21.15.42:0/39610039 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f6c50067400 con 0x7f6c5c116480 2019-12-05T05:31:49.411 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.406+0000 7f6c629b9700 1 -- 172.21.15.42:0/39610039 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6c5c05b8c0 con 0x7f6c5c116480 2019-12-05T05:31:49.412 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.410+0000 7f6c5bfff700 1 --2- 172.21.15.42:0/39610039 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6c5c116480 0x7f6c5c11bd30 secure :-1 s=THROTTLE_DONE pgs=1049 cs=0 l=1 rx=0x7f6c4c007940 tx=0x7f6c4c007720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:49.414 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.410+0000 7f6c597fa700 1 -- 172.21.15.42:0/39610039 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 129 v129) v1 ==== 95+0+1774 (secure 0 0 0) 0x7f6c50031f40 con 0x7f6c5c116480 2019-12-05T05:31:49.414 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 129 2019-12-05T05:31:49.414 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:31:49.415 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":129,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":129,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:31:48.712307+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":69,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6984},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6984":{"gid":6984,"name":"b","rank":0,"incarnation":129,"state":"up:replay","state_seq":1,"addr":"172.21.15.40:6833/3543033403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3543033403},{"type":"v1","addr":"172.21.15.40:6833","nonce":3543033403}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:31:49.416 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.410+0000 7f6c629b9700 1 -- 172.21.15.42:0/39610039 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6c3c03b3a0 msgr2=0x7f6c3c03d8b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:31:49.416 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.410+0000 7f6c629b9700 1 --2- 172.21.15.42:0/39610039 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6c3c03b3a0 0x7f6c3c03d8b0 crc :-1 s=READY pgs=447 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:31:49.416 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.410+0000 7f6c629b9700 1 -- 172.21.15.42:0/39610039 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6c5c116480 msgr2=0x7f6c5c11bd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:31:49.416 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.410+0000 7f6c629b9700 1 --2- 172.21.15.42:0/39610039 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6c5c116480 0x7f6c5c11bd30 secure :-1 s=READY pgs=1049 cs=0 l=1 rx=0x7f6c4c007940 tx=0x7f6c4c007720).stop 2019-12-05T05:31:49.417 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.410+0000 7f6c629b9700 1 -- 172.21.15.42:0/39610039 shutdown_connections 2019-12-05T05:31:49.417 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.410+0000 7f6c629b9700 1 --2- 172.21.15.42:0/39610039 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6c3c03b3a0 0x7f6c3c03d8b0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:31:49.417 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.410+0000 7f6c629b9700 1 --2- 172.21.15.42:0/39610039 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6c5c116480 0x7f6c5c11bd30 unknown :-1 s=CLOSED pgs=1049 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:31:49.417 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.410+0000 7f6c629b9700 1 -- 172.21.15.42:0/39610039 shutdown_connections 2019-12-05T05:31:49.417 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.410+0000 7f6c629b9700 1 -- 172.21.15.42:0/39610039 wait complete. 2019-12-05T05:31:49.417 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.410+0000 7f6c629b9700 1 -- 172.21.15.42:0/39610039 >> 172.21.15.42:0/39610039 conn(0x7f6c5c0aabe0 msgr2=0x7f6c5c0a96f0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:31:49.426 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a down, removed from mdsmap 2019-12-05T05:31:49.426 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2019-12-05T05:31:49.426 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:31:49.525 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.518+0000 7fe326328700 1 Processor -- start 2019-12-05T05:31:49.526 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.518+0000 7fe326328700 1 -- start start 2019-12-05T05:31:49.526 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.518+0000 7fe326328700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe32008df80 0x7fe32009f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:31:49.526 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.518+0000 7fe326328700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe32009f830 0x7fe32014efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:31:49.526 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.518+0000 7fe326328700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe32009fc60 con 0x7fe3200a7740 2019-12-05T05:31:49.526 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.518+0000 7fe326328700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3200a0b00 con 0x7fe32008df80 2019-12-05T05:31:49.526 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.518+0000 7fe326328700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe32008e5b0 con 0x7fe32009f830 2019-12-05T05:31:49.527 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.518+0000 7fe31ffff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe32008df80 0x7fe32009f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:31:49.527 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.518+0000 7fe31ffff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe32008df80 0x7fe32009f2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:35742/0 (socket says 172.21.15.42:35742) 2019-12-05T05:31:49.527 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.518+0000 7fe31ffff700 1 -- 172.21.15.42:0/2529119707 learned_addr learned my addr 172.21.15.42:0/2529119707 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:31:49.527 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.518+0000 7fe3250c6700 1 --2- 172.21.15.42:0/2529119707 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe32009f830 0x7fe32014efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:31:49.527 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.518+0000 7fe31ffff700 1 -- 172.21.15.42:0/2529119707 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe32009f830 msgr2=0x7fe32014efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:31:49.527 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.518+0000 7fe31ffff700 1 --2- 172.21.15.42:0/2529119707 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe32009f830 0x7fe32014efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:31:49.528 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.518+0000 7fe31ffff700 1 -- 172.21.15.42:0/2529119707 >> v1:172.21.15.42:6789/0 conn(0x7fe3200a7740 legacy=0x7fe32007e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:31:49.528 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.518+0000 7fe31ffff700 1 -- 172.21.15.42:0/2529119707 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3200a0710 con 0x7fe32008df80 2019-12-05T05:31:49.528 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.518+0000 7fe3250c6700 1 --2- 172.21.15.42:0/2529119707 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe32009f830 0x7fe32014efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T05:31:49.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.518+0000 7fe31ffff700 1 --2- 172.21.15.42:0/2529119707 >> v2:172.21.15.42:3300/0 conn(0x7fe32008df80 0x7fe32009f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fe314008790 tx=0x7fe31400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:49.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.518+0000 7fe31ffff700 1 --2- 172.21.15.42:0/2529119707 >> v2:172.21.15.42:3300/0 conn(0x7fe32008df80 0x7fe32009f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fe314008790 tx=0x7fe31400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:49.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.518+0000 7fe31ffff700 1 --2- 172.21.15.42:0/2529119707 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe32008df80 0x7fe32009f2e0 secure :-1 s=READY pgs=1051 cs=0 l=1 rx=0x7fe314008790 tx=0x7fe31400fd30).ready entity=mon.1 client_cookie=7e1cc6e3ae19f383 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:31:49.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.518+0000 7fe31ffff700 1 --2- 172.21.15.42:0/2529119707 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe32008df80 0x7fe32009f2e0 secure :-1 s=THROTTLE_DONE pgs=1051 cs=0 l=1 rx=0x7fe314008790 tx=0x7fe31400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:49.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.518+0000 7fe31ffff700 1 --2- 172.21.15.42:0/2529119707 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe32008df80 0x7fe32009f2e0 secure :-1 s=THROTTLE_DONE pgs=1051 cs=0 l=1 rx=0x7fe314008790 tx=0x7fe31400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:49.530 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.518+0000 7fe31f7fe700 1 -- 172.21.15.42:0/2529119707 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe314008c90 con 0x7fe32008df80 2019-12-05T05:31:49.530 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.518+0000 7fe31ffff700 1 --2- 172.21.15.42:0/2529119707 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe32008df80 0x7fe32009f2e0 secure :-1 s=THROTTLE_DONE pgs=1051 cs=0 l=1 rx=0x7fe314008790 tx=0x7fe31400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:49.530 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.518+0000 7fe31f7fe700 1 -- 172.21.15.42:0/2529119707 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3140088b0 con 0x7fe32008df80 2019-12-05T05:31:49.530 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.522+0000 7fe31f7fe700 1 -- 172.21.15.42:0/2529119707 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe314011b10 con 0x7fe32008df80 2019-12-05T05:31:49.530 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.522+0000 7fe326328700 1 -- 172.21.15.42:0/2529119707 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe32008df80 msgr2=0x7fe32009f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:31:49.531 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.522+0000 7fe326328700 1 --2- 172.21.15.42:0/2529119707 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe32008df80 0x7fe32009f2e0 secure :-1 s=READY pgs=1051 cs=0 l=1 rx=0x7fe314008790 tx=0x7fe31400fd30).stop 2019-12-05T05:31:49.531 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.522+0000 7fe326328700 1 -- 172.21.15.42:0/2529119707 shutdown_connections 2019-12-05T05:31:49.531 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.522+0000 7fe326328700 1 --2- 172.21.15.42:0/2529119707 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe32009f830 0x7fe32014efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:31:49.531 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.522+0000 7fe326328700 1 --2- 172.21.15.42:0/2529119707 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe32008df80 0x7fe32009f2e0 unknown :-1 s=CLOSED pgs=1051 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:31:49.532 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.522+0000 7fe326328700 1 -- 172.21.15.42:0/2529119707 shutdown_connections 2019-12-05T05:31:49.532 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.522+0000 7fe326328700 1 -- 172.21.15.42:0/2529119707 wait complete. 2019-12-05T05:31:49.532 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.522+0000 7fe326328700 1 -- 172.21.15.42:0/2529119707 >> 172.21.15.42:0/2529119707 conn(0x7fe3200aabe0 msgr2=0x7fe3200a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:31:49.536 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.522+0000 7fe326328700 1 Processor -- start 2019-12-05T05:31:49.536 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.522+0000 7fe326328700 1 -- start start 2019-12-05T05:31:49.536 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.526+0000 7fe326328700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe32008f830 0x7fe3200a6750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:31:49.536 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.526+0000 7fe326328700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3200a4c70 con 0x7fe32008eeb0 2019-12-05T05:31:49.536 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.526+0000 7fe326328700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe320129530 con 0x7fe3200a4840 2019-12-05T05:31:49.536 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.526+0000 7fe326328700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3200a6210 con 0x7fe32008f830 2019-12-05T05:31:49.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.526+0000 7fe3250c6700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fe3200a4840 0x7fe320128d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:35788/0 (socket says 172.21.15.42:35788) 2019-12-05T05:31:49.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.526+0000 7fe3250c6700 1 -- 172.21.15.42:0/1596575078 learned_addr learned my addr 172.21.15.42:0/1596575078 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:31:49.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.526+0000 7fe3248c5700 1 --2- 172.21.15.42:0/1596575078 >> v2:172.21.15.40:3300/0 conn(0x7fe32008f830 0x7fe3200a6750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:31:49.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.526+0000 7fe31d7fa700 1 -- 172.21.15.42:0/1596575078 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3317463484 0 0) 0x7fe320129530 con 0x7fe3200a4840 2019-12-05T05:31:49.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.526+0000 7fe31d7fa700 1 -- 172.21.15.42:0/1596575078 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3000030b0 con 0x7fe3200a4840 2019-12-05T05:31:49.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.526+0000 7fe3248c5700 1 -- 172.21.15.42:0/1596575078 >> v1:172.21.15.42:6789/0 conn(0x7fe3200a4840 legacy=0x7fe320128d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:31:49.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.526+0000 7fe31d7fa700 1 -- 172.21.15.42:0/1596575078 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 1907555657 0 0) 0x7fe3000030b0 con 0x7fe3200a4840 2019-12-05T05:31:49.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.526+0000 7fe3248c5700 1 -- 172.21.15.42:0/1596575078 >> v1:172.21.15.40:6789/0 conn(0x7fe32008eeb0 legacy=0x7fe320111db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:31:49.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.526+0000 7fe3248c5700 1 -- 172.21.15.42:0/1596575078 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe320129760 con 0x7fe32008f830 2019-12-05T05:31:49.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.526+0000 7fe3248c5700 1 --2- 172.21.15.42:0/1596575078 >> v2:172.21.15.40:3300/0 conn(0x7fe32008f830 0x7fe3200a6750 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fe314008370 tx=0x7fe3180032f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:49.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.526+0000 7fe3248c5700 1 --2- 172.21.15.42:0/1596575078 >> v2:172.21.15.40:3300/0 conn(0x7fe32008f830 0x7fe3200a6750 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fe314008370 tx=0x7fe3180032f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:49.539 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.526+0000 7fe3248c5700 1 --2- 172.21.15.42:0/1596575078 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe32008f830 0x7fe3200a6750 secure :-1 s=READY pgs=817 cs=0 l=1 rx=0x7fe314008370 tx=0x7fe3180032f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:31:49.539 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.526+0000 7fe3248c5700 1 --2- 172.21.15.42:0/1596575078 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe32008f830 0x7fe3200a6750 secure :-1 s=THROTTLE_DONE pgs=817 cs=0 l=1 rx=0x7fe314008370 tx=0x7fe3180032f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:49.539 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.526+0000 7fe31d7fa700 1 -- 172.21.15.42:0/1596575078 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe318001890 con 0x7fe32008f830 2019-12-05T05:31:49.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.526+0000 7fe3248c5700 1 --2- 172.21.15.42:0/1596575078 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe32008f830 0x7fe3200a6750 secure :-1 s=THROTTLE_DONE pgs=817 cs=0 l=1 rx=0x7fe314008370 tx=0x7fe3180032f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:49.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.526+0000 7fe31d7fa700 1 -- 172.21.15.42:0/1596575078 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3180016a0 con 0x7fe32008f830 2019-12-05T05:31:49.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.526+0000 7fe3248c5700 1 --2- 172.21.15.42:0/1596575078 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe32008f830 0x7fe3200a6750 secure :-1 s=THROTTLE_DONE pgs=817 cs=0 l=1 rx=0x7fe314008370 tx=0x7fe3180032f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:49.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.526+0000 7fe326328700 1 -- 172.21.15.42:0/1596575078 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe320129a30 con 0x7fe32008f830 2019-12-05T05:31:49.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.526+0000 7fe31d7fa700 1 -- 172.21.15.42:0/1596575078 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe31000cd60 con 0x7fe32008f830 2019-12-05T05:31:49.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.526+0000 7fe326328700 1 -- 172.21.15.42:0/1596575078 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3201325b0 con 0x7fe32008f830 2019-12-05T05:31:49.541 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.526+0000 7fe3248c5700 1 --2- 172.21.15.42:0/1596575078 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe32008f830 0x7fe3200a6750 secure :-1 s=THROTTLE_DONE pgs=817 cs=0 l=1 rx=0x7fe314008370 tx=0x7fe3180032f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:49.541 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.530+0000 7fe31d7fa700 1 -- 172.21.15.42:0/1596575078 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fe310006360 con 0x7fe32008f830 2019-12-05T05:31:49.541 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.530+0000 7fe3248c5700 1 --2- 172.21.15.42:0/1596575078 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe32008f830 0x7fe3200a6750 secure :-1 s=THROTTLE_DONE pgs=817 cs=0 l=1 rx=0x7fe314008370 tx=0x7fe3180032f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:49.541 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.530+0000 7fe31d7fa700 1 --2- 172.21.15.42:0/1596575078 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe30003b160 0x7fe30003d670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:31:49.542 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.530+0000 7fe31d7fa700 1 -- 172.21.15.42:0/1596575078 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7751+0+0 (secure 0 0 0) 0x7fe31004ad40 con 0x7fe32008f830 2019-12-05T05:31:49.542 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.530+0000 7fe31ffff700 1 --2- 172.21.15.42:0/1596575078 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe30003b160 0x7fe30003d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:31:49.542 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.530+0000 7fe31ffff700 1 --2- 172.21.15.42:0/1596575078 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe30003b160 0x7fe30003d670 crc :-1 s=READY pgs=448 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:31:49.543 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.534+0000 7fe326328700 1 -- 172.21.15.42:0/1596575078 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe32005d110 con 0x7fe32008f830 2019-12-05T05:31:49.548 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.546+0000 7fe3248c5700 1 --2- 172.21.15.42:0/1596575078 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe32008f830 0x7fe3200a6750 secure :-1 s=THROTTLE_DONE pgs=817 cs=0 l=1 rx=0x7fe314008370 tx=0x7fe3180032f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:49.548 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.546+0000 7fe31d7fa700 1 -- 172.21.15.42:0/1596575078 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fe310031da0 con 0x7fe32008f830 2019-12-05T05:31:49.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.710+0000 7fe326328700 1 -- 172.21.15.42:0/1596575078 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe32005b8c0 con 0x7fe32008f830 2019-12-05T05:31:49.715 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.710+0000 7fe3248c5700 1 --2- 172.21.15.42:0/1596575078 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe32008f830 0x7fe3200a6750 secure :-1 s=THROTTLE_DONE pgs=817 cs=0 l=1 rx=0x7fe314008370 tx=0x7fe3180032f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:49.716 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.710+0000 7fe31d7fa700 1 -- 172.21.15.42:0/1596575078 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 129 v129) v1 ==== 95+0+1774 (secure 0 0 0) 0x7fe310032000 con 0x7fe32008f830 2019-12-05T05:31:49.716 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 129 2019-12-05T05:31:49.716 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:31:49.716 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":129,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":129,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:31:48.712307+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":69,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6984},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6984":{"gid":6984,"name":"b","rank":0,"incarnation":129,"state":"up:replay","state_seq":1,"addr":"172.21.15.40:6833/3543033403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3543033403},{"type":"v1","addr":"172.21.15.40:6833","nonce":3543033403}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:31:49.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.714+0000 7fe326328700 1 -- 172.21.15.42:0/1596575078 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe30003b160 msgr2=0x7fe30003d670 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:31:49.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.714+0000 7fe326328700 1 --2- 172.21.15.42:0/1596575078 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe30003b160 0x7fe30003d670 crc :-1 s=READY pgs=448 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:31:49.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.714+0000 7fe326328700 1 -- 172.21.15.42:0/1596575078 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe32008f830 msgr2=0x7fe3200a6750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:31:49.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.714+0000 7fe326328700 1 --2- 172.21.15.42:0/1596575078 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe32008f830 0x7fe3200a6750 secure :-1 s=READY pgs=817 cs=0 l=1 rx=0x7fe314008370 tx=0x7fe3180032f0).stop 2019-12-05T05:31:49.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.714+0000 7fe326328700 1 -- 172.21.15.42:0/1596575078 shutdown_connections 2019-12-05T05:31:49.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.714+0000 7fe326328700 1 --2- 172.21.15.42:0/1596575078 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe30003b160 0x7fe30003d670 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:31:49.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.714+0000 7fe326328700 1 --2- 172.21.15.42:0/1596575078 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe32008f830 0x7fe3200a6750 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:31:49.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.714+0000 7fe326328700 1 -- 172.21.15.42:0/1596575078 shutdown_connections 2019-12-05T05:31:49.725 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.714+0000 7fe326328700 1 -- 172.21.15.42:0/1596575078 wait complete. 2019-12-05T05:31:49.725 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:49.714+0000 7fe326328700 1 -- 172.21.15.42:0/1596575078 >> 172.21.15.42:0/1596575078 conn(0x7fe3200aabe0 msgr2=0x7fe32007cf70 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:31:49.729 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 129, "feature_flags": { "enable_multiple": false, "ever_enabled_multiple": false }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2019-12-05T05:03:35.604738+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 129, "ever_allowed_features": 2, "explicitly_allowed_features": 2, "failed": [], "flags": 18, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_6984": { "addr": "172.21.15.40:6833/3543033403", "addrs": { "addrvec": [ { "addr": "172.21.15.40:6832", "nonce": 3543033403, "type": "v2" }, { "addr": "172.21.15.40:6833", "nonce": 3543033403, "type": "v1" } ] }, "export_targets": [], "features": 4540138292836696063, "flags": 0, "gid": 6984, "incarnation": 129, "name": "b", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 69, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 2, "min_compat_client": "0 (unknown)", "modified": "2019-12-05T05:31:48.712307+0000", "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6984 } } } ], "standbys": [] } 2019-12-05T05:31:51.730 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:31:51.837 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:51.826+0000 7fde3ab6d700 1 Processor -- start 2019-12-05T05:31:51.837 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:51.826+0000 7fde3ab6d700 1 -- start start 2019-12-05T05:31:51.838 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:51.830+0000 7fde3ab6d700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fde3408df80 0x7fde3409f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:31:51.838 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:51.830+0000 7fde3ab6d700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde3409fc60 con 0x7fde340a7740 2019-12-05T05:31:51.838 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:51.830+0000 7fde3ab6d700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde340a0710 con 0x7fde3409f830 2019-12-05T05:31:51.838 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:51.830+0000 7fde3ab6d700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde340a0b00 con 0x7fde3408df80 2019-12-05T05:31:51.838 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:51.830+0000 7fde3990b700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fde3409f830 0x7fde3414efe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:35798/0 (socket says 172.21.15.42:35798) 2019-12-05T05:31:51.838 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:51.830+0000 7fde3990b700 1 -- 172.21.15.42:0/3075378840 learned_addr learned my addr 172.21.15.42:0/3075378840 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:31:51.838 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:51.830+0000 7fde38909700 1 --2- 172.21.15.42:0/3075378840 >> v2:172.21.15.40:3300/0 conn(0x7fde3408df80 0x7fde3409f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:31:51.839 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:51.830+0000 7fde2bfff700 1 -- 172.21.15.42:0/3075378840 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2577929020 0 0) 0x7fde340a0710 con 0x7fde3409f830 2019-12-05T05:31:51.839 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:51.830+0000 7fde2bfff700 1 -- 172.21.15.42:0/3075378840 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde200030b0 con 0x7fde3409f830 2019-12-05T05:31:51.839 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:51.830+0000 7fde38909700 1 -- 172.21.15.42:0/3075378840 >> v1:172.21.15.42:6789/0 conn(0x7fde3409f830 legacy=0x7fde3414efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:31:51.839 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:51.830+0000 7fde38909700 1 -- 172.21.15.42:0/3075378840 >> v1:172.21.15.40:6789/0 conn(0x7fde340a7740 legacy=0x7fde3407e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:31:51.839 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:51.830+0000 7fde2bfff700 1 -- 172.21.15.42:0/3075378840 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2860314443 0 0) 0x7fde3409fc60 con 0x7fde340a7740 2019-12-05T05:31:51.839 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:51.830+0000 7fde38909700 1 -- 172.21.15.42:0/3075378840 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde3407ef20 con 0x7fde3408df80 2019-12-05T05:31:51.839 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:51.830+0000 7fde38909700 1 --2- 172.21.15.42:0/3075378840 >> v2:172.21.15.40:3300/0 conn(0x7fde3408df80 0x7fde3409f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fde24008790 tx=0x7fde2400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:51.840 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:51.830+0000 7fde38909700 1 --2- 172.21.15.42:0/3075378840 >> v2:172.21.15.40:3300/0 conn(0x7fde3408df80 0x7fde3409f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fde24008790 tx=0x7fde2400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:51.840 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:51.830+0000 7fde38909700 1 --2- 172.21.15.42:0/3075378840 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fde3408df80 0x7fde3409f2e0 secure :-1 s=READY pgs=821 cs=0 l=1 rx=0x7fde24008790 tx=0x7fde2400fbf0).ready entity=mon.0 client_cookie=ebc3ec83e8adae21 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:31:51.840 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:51.830+0000 7fde38909700 1 --2- 172.21.15.42:0/3075378840 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fde3408df80 0x7fde3409f2e0 secure :-1 s=THROTTLE_DONE pgs=821 cs=0 l=1 rx=0x7fde24008790 tx=0x7fde2400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:51.840 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:51.830+0000 7fde2bfff700 1 -- 172.21.15.42:0/3075378840 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fde24011270 con 0x7fde3408df80 2019-12-05T05:31:51.840 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:51.830+0000 7fde38909700 1 --2- 172.21.15.42:0/3075378840 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fde3408df80 0x7fde3409f2e0 secure :-1 s=THROTTLE_DONE pgs=821 cs=0 l=1 rx=0x7fde24008790 tx=0x7fde2400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:51.840 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:51.830+0000 7fde38909700 1 --2- 172.21.15.42:0/3075378840 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fde3408df80 0x7fde3409f2e0 secure :-1 s=THROTTLE_DONE pgs=821 cs=0 l=1 rx=0x7fde24008790 tx=0x7fde2400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:51.841 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:51.830+0000 7fde2bfff700 1 -- 172.21.15.42:0/3075378840 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde24008bc0 con 0x7fde3408df80 2019-12-05T05:31:51.841 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:51.830+0000 7fde2bfff700 1 -- 172.21.15.42:0/3075378840 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fde24011850 con 0x7fde3408df80 2019-12-05T05:31:51.841 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:51.830+0000 7fde3ab6d700 1 -- 172.21.15.42:0/3075378840 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fde3408df80 msgr2=0x7fde3409f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:31:51.841 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:51.830+0000 7fde3ab6d700 1 --2- 172.21.15.42:0/3075378840 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fde3408df80 0x7fde3409f2e0 secure :-1 s=READY pgs=821 cs=0 l=1 rx=0x7fde24008790 tx=0x7fde2400fbf0).stop 2019-12-05T05:31:51.841 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:51.834+0000 7fde3ab6d700 1 -- 172.21.15.42:0/3075378840 shutdown_connections 2019-12-05T05:31:51.841 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:51.834+0000 7fde3ab6d700 1 --2- 172.21.15.42:0/3075378840 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fde3408df80 0x7fde3409f2e0 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:31:51.841 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:51.834+0000 7fde3ab6d700 1 -- 172.21.15.42:0/3075378840 shutdown_connections 2019-12-05T05:31:51.841 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:51.834+0000 7fde3ab6d700 1 -- 172.21.15.42:0/3075378840 wait complete. 2019-12-05T05:31:51.842 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:51.834+0000 7fde3ab6d700 1 -- 172.21.15.42:0/3075378840 >> 172.21.15.42:0/3075378840 conn(0x7fde340aabe0 msgr2=0x7fde340a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:31:51.845 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:51.834+0000 7fde3ab6d700 1 Processor -- start 2019-12-05T05:31:51.845 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:51.834+0000 7fde3ab6d700 1 -- start start 2019-12-05T05:31:51.845 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:51.838+0000 7fde3ab6d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fde34120ce0 0x7fde34118660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:31:51.845 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:51.838+0000 7fde3ab6d700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fde34116500 0x7fde3408eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:31:51.846 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:51.838+0000 7fde3ab6d700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde3408f8c0 con 0x7fde34133380 2019-12-05T05:31:51.846 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:51.838+0000 7fde3ab6d700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde341178e0 con 0x7fde34116500 2019-12-05T05:31:51.846 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:51.838+0000 7fde3ab6d700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde34119f00 con 0x7fde34120ce0 2019-12-05T05:31:51.846 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:51.838+0000 7fde3910a700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fde34120ce0 0x7fde34118660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:31:51.846 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:51.838+0000 7fde3910a700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fde34120ce0 0x7fde34118660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:35762/0 (socket says 172.21.15.42:35762) 2019-12-05T05:31:51.846 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:51.838+0000 7fde3910a700 1 -- 172.21.15.42:0/4157330146 learned_addr learned my addr 172.21.15.42:0/4157330146 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:31:51.846 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:51.838+0000 7fde38909700 1 --2- 172.21.15.42:0/4157330146 >> v2:172.21.15.40:3300/0 conn(0x7fde34116500 0x7fde3408eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:31:51.847 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:51.838+0000 7fde3910a700 1 -- 172.21.15.42:0/4157330146 >> v2:172.21.15.40:3300/0 conn(0x7fde34116500 msgr2=0x7fde3408eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:31:51.847 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:51.838+0000 7fde3910a700 1 --2- 172.21.15.42:0/4157330146 >> v2:172.21.15.40:3300/0 conn(0x7fde34116500 0x7fde3408eb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:31:51.847 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:51.838+0000 7fde3910a700 1 -- 172.21.15.42:0/4157330146 >> v1:172.21.15.40:6789/0 conn(0x7fde34133380 legacy=0x7fde3408f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:31:51.847 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:51.838+0000 7fde3910a700 1 -- 172.21.15.42:0/4157330146 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde34118bb0 con 0x7fde34120ce0 2019-12-05T05:31:51.847 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:51.838+0000 7fde3910a700 1 --2- 172.21.15.42:0/4157330146 >> v2:172.21.15.42:3300/0 conn(0x7fde34120ce0 0x7fde34118660 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fde30000b80 tx=0x7fde3000c340).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:51.847 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:51.838+0000 7fde3910a700 1 --2- 172.21.15.42:0/4157330146 >> v2:172.21.15.42:3300/0 conn(0x7fde34120ce0 0x7fde34118660 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fde30000b80 tx=0x7fde3000c340).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:51.847 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:51.838+0000 7fde3910a700 1 --2- 172.21.15.42:0/4157330146 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fde34120ce0 0x7fde34118660 secure :-1 s=READY pgs=1054 cs=0 l=1 rx=0x7fde30000b80 tx=0x7fde3000c340).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:31:51.848 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:51.838+0000 7fde3910a700 1 --2- 172.21.15.42:0/4157330146 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fde34120ce0 0x7fde34118660 secure :-1 s=THROTTLE_DONE pgs=1054 cs=0 l=1 rx=0x7fde30000b80 tx=0x7fde3000c340).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:51.848 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:51.838+0000 7fde3910a700 1 --2- 172.21.15.42:0/4157330146 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fde34120ce0 0x7fde34118660 secure :-1 s=THROTTLE_DONE pgs=1054 cs=0 l=1 rx=0x7fde30000b80 tx=0x7fde3000c340).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:51.848 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:51.838+0000 7fde3910a700 1 --2- 172.21.15.42:0/4157330146 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fde34120ce0 0x7fde34118660 secure :-1 s=THROTTLE_DONE pgs=1054 cs=0 l=1 rx=0x7fde30000b80 tx=0x7fde3000c340).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:51.848 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:51.838+0000 7fde29ffb700 1 -- 172.21.15.42:0/4157330146 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fde3000caa0 con 0x7fde34120ce0 2019-12-05T05:31:51.848 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:51.838+0000 7fde29ffb700 1 -- 172.21.15.42:0/4157330146 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde3000c6c0 con 0x7fde34120ce0 2019-12-05T05:31:51.848 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:51.838+0000 7fde29ffb700 1 -- 172.21.15.42:0/4157330146 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fde30011120 con 0x7fde34120ce0 2019-12-05T05:31:51.849 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:51.838+0000 7fde3ab6d700 1 -- 172.21.15.42:0/4157330146 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde34116ed0 con 0x7fde34120ce0 2019-12-05T05:31:51.849 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:51.838+0000 7fde3ab6d700 1 -- 172.21.15.42:0/4157330146 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde341197c0 con 0x7fde34120ce0 2019-12-05T05:31:51.849 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:51.838+0000 7fde3910a700 1 --2- 172.21.15.42:0/4157330146 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fde34120ce0 0x7fde34118660 secure :-1 s=THROTTLE_DONE pgs=1054 cs=0 l=1 rx=0x7fde30000b80 tx=0x7fde3000c340).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:51.849 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:51.842+0000 7fde29ffb700 1 -- 172.21.15.42:0/4157330146 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fde30011430 con 0x7fde34120ce0 2019-12-05T05:31:51.849 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:51.842+0000 7fde3910a700 1 --2- 172.21.15.42:0/4157330146 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fde34120ce0 0x7fde34118660 secure :-1 s=THROTTLE_DONE pgs=1054 cs=0 l=1 rx=0x7fde30000b80 tx=0x7fde3000c340).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:51.849 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:51.842+0000 7fde29ffb700 1 --2- 172.21.15.42:0/4157330146 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fde1403ae70 0x7fde1403d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:31:51.849 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:51.842+0000 7fde29ffb700 1 -- 172.21.15.42:0/4157330146 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7751+0+0 (secure 0 0 0) 0x7fde3004b1e0 con 0x7fde34120ce0 2019-12-05T05:31:51.850 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:51.842+0000 7fde38909700 1 --2- 172.21.15.42:0/4157330146 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fde1403ae70 0x7fde1403d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:31:51.850 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:51.842+0000 7fde38909700 1 --2- 172.21.15.42:0/4157330146 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fde1403ae70 0x7fde1403d380 crc :-1 s=READY pgs=449 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:31:51.850 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:51.842+0000 7fde3ab6d700 1 -- 172.21.15.42:0/4157330146 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde3405d110 con 0x7fde34120ce0 2019-12-05T05:31:51.852 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:51.850+0000 7fde3910a700 1 --2- 172.21.15.42:0/4157330146 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fde34120ce0 0x7fde34118660 secure :-1 s=THROTTLE_DONE pgs=1054 cs=0 l=1 rx=0x7fde30000b80 tx=0x7fde3000c340).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:51.852 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:51.850+0000 7fde29ffb700 1 -- 172.21.15.42:0/4157330146 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fde30032260 con 0x7fde34120ce0 2019-12-05T05:31:52.026 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:52.022+0000 7fde3ab6d700 1 -- 172.21.15.42:0/4157330146 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fde3405b8c0 con 0x7fde34120ce0 2019-12-05T05:31:52.027 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:52.022+0000 7fde3910a700 1 --2- 172.21.15.42:0/4157330146 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fde34120ce0 0x7fde34118660 secure :-1 s=THROTTLE_DONE pgs=1054 cs=0 l=1 rx=0x7fde30000b80 tx=0x7fde3000c340).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:52.027 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:52.022+0000 7fde29ffb700 1 -- 172.21.15.42:0/4157330146 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 130 v130) v1 ==== 95+0+1778 (secure 0 0 0) 0x7fde300324c0 con 0x7fde34120ce0 2019-12-05T05:31:52.029 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:31:52.029 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":130,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":130,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:31:51.737479+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":69,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6984},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6984":{"gid":6984,"name":"b","rank":0,"incarnation":129,"state":"up:reconnect","state_seq":13,"addr":"172.21.15.40:6833/3543033403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3543033403},{"type":"v1","addr":"172.21.15.40:6833","nonce":3543033403}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:31:52.029 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 130 2019-12-05T05:31:52.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:52.026+0000 7fde3ab6d700 1 -- 172.21.15.42:0/4157330146 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fde1403ae70 msgr2=0x7fde1403d380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:31:52.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:52.026+0000 7fde3ab6d700 1 --2- 172.21.15.42:0/4157330146 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fde1403ae70 0x7fde1403d380 crc :-1 s=READY pgs=449 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:31:52.032 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:52.026+0000 7fde3ab6d700 1 -- 172.21.15.42:0/4157330146 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fde34120ce0 msgr2=0x7fde34118660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:31:52.032 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:52.026+0000 7fde3ab6d700 1 --2- 172.21.15.42:0/4157330146 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fde34120ce0 0x7fde34118660 secure :-1 s=READY pgs=1054 cs=0 l=1 rx=0x7fde30000b80 tx=0x7fde3000c340).stop 2019-12-05T05:31:52.032 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:52.026+0000 7fde3ab6d700 1 -- 172.21.15.42:0/4157330146 shutdown_connections 2019-12-05T05:31:52.032 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:52.026+0000 7fde3ab6d700 1 --2- 172.21.15.42:0/4157330146 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fde1403ae70 0x7fde1403d380 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:31:52.032 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:52.026+0000 7fde3ab6d700 1 --2- 172.21.15.42:0/4157330146 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fde34120ce0 0x7fde34118660 unknown :-1 s=CLOSED pgs=1054 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:31:52.032 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:52.026+0000 7fde3ab6d700 1 --2- 172.21.15.42:0/4157330146 >> v2:172.21.15.40:3300/0 conn(0x7fde34116500 0x7fde3408eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:31:52.032 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:52.030+0000 7fde3ab6d700 1 -- 172.21.15.42:0/4157330146 shutdown_connections 2019-12-05T05:31:52.033 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:52.030+0000 7fde3ab6d700 1 -- 172.21.15.42:0/4157330146 wait complete. 2019-12-05T05:31:52.033 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:52.030+0000 7fde3ab6d700 1 -- 172.21.15.42:0/4157330146 >> 172.21.15.42:0/4157330146 conn(0x7fde3408e5b0 msgr2=0x7fde340a9770 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:31:52.042 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2019-12-05T05:31:54.043 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:31:54.162 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:54.146+0000 7f1969dee700 1 Processor -- start 2019-12-05T05:31:54.163 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:54.146+0000 7f1969dee700 1 -- start start 2019-12-05T05:31:54.163 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:54.146+0000 7f1969dee700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f19640a7730 0x7f196408df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:31:54.163 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:54.146+0000 7f1969dee700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19640a0700 con 0x7f196409fad0 2019-12-05T05:31:54.163 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:54.146+0000 7f1969dee700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19640a9b90 con 0x7f196407e710 2019-12-05T05:31:54.163 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:54.146+0000 7f1969dee700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19640a0af0 con 0x7f19640a7730 2019-12-05T05:31:54.164 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:54.146+0000 7f195b7fe700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f196407e710 0x7f196409f2d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:35816/0 (socket says 172.21.15.42:35816) 2019-12-05T05:31:54.164 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:54.146+0000 7f195b7fe700 1 -- 172.21.15.42:0/240361460 learned_addr learned my addr 172.21.15.42:0/240361460 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:31:54.164 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:54.146+0000 7f1963fff700 1 --2- 172.21.15.42:0/240361460 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f19640a7730 0x7f196408df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:31:54.164 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:54.146+0000 7f19637fe700 1 -- 172.21.15.42:0/240361460 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2853970155 0 0) 0x7f19640a9b90 con 0x7f196407e710 2019-12-05T05:31:54.164 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:54.146+0000 7f19637fe700 1 -- 172.21.15.42:0/240361460 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19500030b0 con 0x7f196407e710 2019-12-05T05:31:54.164 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:54.146+0000 7f19637fe700 1 -- 172.21.15.42:0/240361460 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3099969781 0 0) 0x7f19640a0700 con 0x7f196409fad0 2019-12-05T05:31:54.164 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:54.146+0000 7f19637fe700 1 -- 172.21.15.42:0/240361460 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19640a9b90 con 0x7f196409fad0 2019-12-05T05:31:54.165 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:54.146+0000 7f1963fff700 1 -- 172.21.15.42:0/240361460 >> v1:172.21.15.42:6789/0 conn(0x7f196407e710 legacy=0x7f196409f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:31:54.165 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:54.146+0000 7f19637fe700 1 -- 172.21.15.42:0/240361460 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2074422002 0 0) 0x7f19640a9b90 con 0x7f196409fad0 2019-12-05T05:31:54.165 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:54.146+0000 7f1963fff700 1 -- 172.21.15.42:0/240361460 >> v1:172.21.15.40:6789/0 conn(0x7f196409fad0 legacy=0x7f196414efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:31:54.165 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:54.146+0000 7f1963fff700 1 -- 172.21.15.42:0/240361460 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f196407eb40 con 0x7f19640a7730 2019-12-05T05:31:54.165 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:54.146+0000 7f1963fff700 1 --2- 172.21.15.42:0/240361460 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f19640a7730 0x7f196408df70 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f194c001990 tx=0x7f194c0014a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:54.165 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:54.146+0000 7f1963fff700 1 --2- 172.21.15.42:0/240361460 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f19640a7730 0x7f196408df70 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f194c001990 tx=0x7f194c0014a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:54.166 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:54.146+0000 7f1963fff700 1 --2- 172.21.15.42:0/240361460 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f19640a7730 0x7f196408df70 secure :-1 s=READY pgs=213 cs=0 l=1 rx=0x7f194c001990 tx=0x7f194c0014a0).ready entity=mon.2 client_cookie=b61029b8660840c6 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:31:54.166 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:54.146+0000 7f1963fff700 1 --2- 172.21.15.42:0/240361460 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f19640a7730 0x7f196408df70 secure :-1 s=THROTTLE_DONE pgs=213 cs=0 l=1 rx=0x7f194c001990 tx=0x7f194c0014a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:54.166 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:54.146+0000 7f19637fe700 1 -- 172.21.15.42:0/240361460 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f195c001ab0 con 0x7f19640a7730 2019-12-05T05:31:54.166 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:54.146+0000 7f1963fff700 1 --2- 172.21.15.42:0/240361460 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f19640a7730 0x7f196408df70 secure :-1 s=THROTTLE_DONE pgs=213 cs=0 l=1 rx=0x7f194c001990 tx=0x7f194c0014a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:54.166 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:54.146+0000 7f19637fe700 1 -- 172.21.15.42:0/240361460 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f195c0018c0 con 0x7f19640a7730 2019-12-05T05:31:54.166 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:54.146+0000 7f1963fff700 1 --2- 172.21.15.42:0/240361460 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f19640a7730 0x7f196408df70 secure :-1 s=THROTTLE_DONE pgs=213 cs=0 l=1 rx=0x7f194c001990 tx=0x7f194c0014a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:54.166 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:54.146+0000 7f19637fe700 1 -- 172.21.15.42:0/240361460 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f194c001750 con 0x7f19640a7730 2019-12-05T05:31:54.167 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:54.146+0000 7f1969dee700 1 -- 172.21.15.42:0/240361460 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f19640a7730 msgr2=0x7f196408df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:31:54.167 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:54.146+0000 7f1969dee700 1 --2- 172.21.15.42:0/240361460 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f19640a7730 0x7f196408df70 secure :-1 s=READY pgs=213 cs=0 l=1 rx=0x7f194c001990 tx=0x7f194c0014a0).stop 2019-12-05T05:31:54.167 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:54.150+0000 7f1969dee700 1 -- 172.21.15.42:0/240361460 shutdown_connections 2019-12-05T05:31:54.167 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:54.150+0000 7f1969dee700 1 --2- 172.21.15.42:0/240361460 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f19640a7730 0x7f196408df70 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:31:54.167 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:54.150+0000 7f1969dee700 1 -- 172.21.15.42:0/240361460 shutdown_connections 2019-12-05T05:31:54.167 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:54.150+0000 7f1969dee700 1 -- 172.21.15.42:0/240361460 wait complete. 2019-12-05T05:31:54.168 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:54.150+0000 7f1969dee700 1 -- 172.21.15.42:0/240361460 >> 172.21.15.42:0/240361460 conn(0x7f19640aabd0 msgr2=0x7f19640a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:31:54.168 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:54.150+0000 7f1969dee700 1 Processor -- start 2019-12-05T05:31:54.168 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:54.150+0000 7f1969dee700 1 -- start start 2019-12-05T05:31:54.168 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:54.150+0000 7f1969dee700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f19640a40c0 0x7f196411a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:31:54.169 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:54.150+0000 7f1969dee700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f19640a2810 0x7f19640a1800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:31:54.169 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:54.150+0000 7f1969dee700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f19640a1d50 0x7f1964138000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:31:54.169 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:54.150+0000 7f1969dee700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19640a2f20 con 0x7f19640a40c0 2019-12-05T05:31:54.170 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:54.150+0000 7f1969dee700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19640a31c0 con 0x7f19640a2810 2019-12-05T05:31:54.170 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:54.150+0000 7f1969dee700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19640a1560 con 0x7f19640a1d50 2019-12-05T05:31:54.170 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:54.150+0000 7f195b7fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f19640a2810 0x7f19640a1800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:31:54.171 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:54.150+0000 7f195b7fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f19640a2810 0x7f19640a1800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:35784/0 (socket says 172.21.15.42:35784) 2019-12-05T05:31:54.171 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:54.150+0000 7f195b7fe700 1 -- 172.21.15.42:0/3588366297 learned_addr learned my addr 172.21.15.42:0/3588366297 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:31:54.171 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:54.150+0000 7f1963fff700 1 --2- 172.21.15.42:0/3588366297 >> v2:172.21.15.40:3300/0 conn(0x7f19640a40c0 0x7f196411a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:31:54.171 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:54.150+0000 7f1968b8c700 1 --2- 172.21.15.42:0/3588366297 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f19640a1d50 0x7f1964138000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:31:54.171 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:54.150+0000 7f195b7fe700 1 -- 172.21.15.42:0/3588366297 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f19640a1d50 msgr2=0x7f1964138000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:31:54.171 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:54.150+0000 7f195b7fe700 1 --2- 172.21.15.42:0/3588366297 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f19640a1d50 0x7f1964138000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:31:54.171 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:54.150+0000 7f195b7fe700 1 -- 172.21.15.42:0/3588366297 >> v2:172.21.15.40:3300/0 conn(0x7f19640a40c0 msgr2=0x7f196411a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:31:54.172 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:54.150+0000 7f195b7fe700 1 --2- 172.21.15.42:0/3588366297 >> v2:172.21.15.40:3300/0 conn(0x7f19640a40c0 0x7f196411a4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:31:54.172 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:54.150+0000 7f195b7fe700 1 -- 172.21.15.42:0/3588366297 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19640a2c40 con 0x7f19640a2810 2019-12-05T05:31:54.172 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:54.150+0000 7f1968b8c700 1 --2- 172.21.15.42:0/3588366297 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f19640a1d50 0x7f1964138000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T05:31:54.172 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:54.150+0000 7f195b7fe700 1 --2- 172.21.15.42:0/3588366297 >> v2:172.21.15.42:3300/0 conn(0x7f19640a2810 0x7f19640a1800 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f1964132120 tx=0x7f194c010ca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:54.172 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:54.150+0000 7f195b7fe700 1 --2- 172.21.15.42:0/3588366297 >> v2:172.21.15.42:3300/0 conn(0x7f19640a2810 0x7f19640a1800 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f1964132120 tx=0x7f194c010ca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:54.172 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:54.150+0000 7f195b7fe700 1 --2- 172.21.15.42:0/3588366297 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f19640a2810 0x7f19640a1800 secure :-1 s=READY pgs=1056 cs=0 l=1 rx=0x7f1964132120 tx=0x7f194c010ca0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:31:54.172 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:54.150+0000 7f195b7fe700 1 --2- 172.21.15.42:0/3588366297 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f19640a2810 0x7f19640a1800 secure :-1 s=THROTTLE_DONE pgs=1056 cs=0 l=1 rx=0x7f1964132120 tx=0x7f194c010ca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:54.173 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:54.150+0000 7f19617fa700 1 -- 172.21.15.42:0/3588366297 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f194c012980 con 0x7f19640a2810 2019-12-05T05:31:54.173 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:54.154+0000 7f195b7fe700 1 --2- 172.21.15.42:0/3588366297 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f19640a2810 0x7f19640a1800 secure :-1 s=THROTTLE_DONE pgs=1056 cs=0 l=1 rx=0x7f1964132120 tx=0x7f194c010ca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:54.173 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:54.154+0000 7f195b7fe700 1 --2- 172.21.15.42:0/3588366297 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f19640a2810 0x7f19640a1800 secure :-1 s=THROTTLE_DONE pgs=1056 cs=0 l=1 rx=0x7f1964132120 tx=0x7f194c010ca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:54.173 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:54.154+0000 7f19617fa700 1 -- 172.21.15.42:0/3588366297 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f194c0125a0 con 0x7f19640a2810 2019-12-05T05:31:54.173 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:54.154+0000 7f1969dee700 1 -- 172.21.15.42:0/3588366297 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1964137b70 con 0x7f19640a2810 2019-12-05T05:31:54.173 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:54.154+0000 7f19617fa700 1 -- 172.21.15.42:0/3588366297 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f194c00c4b0 con 0x7f19640a2810 2019-12-05T05:31:54.173 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:54.154+0000 7f1969dee700 1 -- 172.21.15.42:0/3588366297 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1964138770 con 0x7f19640a2810 2019-12-05T05:31:54.174 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:54.154+0000 7f195b7fe700 1 --2- 172.21.15.42:0/3588366297 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f19640a2810 0x7f19640a1800 secure :-1 s=THROTTLE_DONE pgs=1056 cs=0 l=1 rx=0x7f1964132120 tx=0x7f194c010ca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:54.174 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:54.154+0000 7f19617fa700 1 -- 172.21.15.42:0/3588366297 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f194c00c7c0 con 0x7f19640a2810 2019-12-05T05:31:54.174 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:54.154+0000 7f195b7fe700 1 --2- 172.21.15.42:0/3588366297 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f19640a2810 0x7f19640a1800 secure :-1 s=THROTTLE_DONE pgs=1056 cs=0 l=1 rx=0x7f1964132120 tx=0x7f194c010ca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:54.174 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:54.154+0000 7f19617fa700 1 --2- 172.21.15.42:0/3588366297 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f194803aec0 0x7f194803d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:31:54.174 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:54.154+0000 7f19617fa700 1 -- 172.21.15.42:0/3588366297 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7751+0+0 (secure 0 0 0) 0x7f194c04b040 con 0x7f19640a2810 2019-12-05T05:31:54.174 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:54.154+0000 7f1963fff700 1 --2- 172.21.15.42:0/3588366297 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f194803aec0 0x7f194803d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:31:54.174 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:54.154+0000 7f1963fff700 1 --2- 172.21.15.42:0/3588366297 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f194803aec0 0x7f194803d3d0 crc :-1 s=READY pgs=450 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:31:54.175 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:54.158+0000 7f1969dee700 1 -- 172.21.15.42:0/3588366297 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f196405d100 con 0x7f19640a2810 2019-12-05T05:31:54.175 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:54.166+0000 7f195b7fe700 1 --2- 172.21.15.42:0/3588366297 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f19640a2810 0x7f19640a1800 secure :-1 s=THROTTLE_DONE pgs=1056 cs=0 l=1 rx=0x7f1964132120 tx=0x7f194c010ca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:54.176 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:54.166+0000 7f19617fa700 1 -- 172.21.15.42:0/3588366297 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f194c0320c0 con 0x7f19640a2810 2019-12-05T05:31:54.335 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:54.330+0000 7f1969dee700 1 -- 172.21.15.42:0/3588366297 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f196405b8b0 con 0x7f19640a2810 2019-12-05T05:31:54.337 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:54.334+0000 7f195b7fe700 1 --2- 172.21.15.42:0/3588366297 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f19640a2810 0x7f19640a1800 secure :-1 s=THROTTLE_DONE pgs=1056 cs=0 l=1 rx=0x7f1964132120 tx=0x7f194c010ca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:54.337 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:54.334+0000 7f19617fa700 1 -- 172.21.15.42:0/3588366297 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 132 v132) v1 ==== 95+0+1775 (secure 0 0 0) 0x7f194c032320 con 0x7f19640a2810 2019-12-05T05:31:54.338 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 132 2019-12-05T05:31:54.338 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:31:54.338 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":132,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":132,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:31:53.749802+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":69,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6984},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6984":{"gid":6984,"name":"b","rank":0,"incarnation":129,"state":"up:active","state_seq":15,"addr":"172.21.15.40:6833/3543033403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3543033403},{"type":"v1","addr":"172.21.15.40:6833","nonce":3543033403}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:31:54.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:54.334+0000 7f1969dee700 1 -- 172.21.15.42:0/3588366297 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f194803aec0 msgr2=0x7f194803d3d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:31:54.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:54.334+0000 7f1969dee700 1 --2- 172.21.15.42:0/3588366297 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f194803aec0 0x7f194803d3d0 crc :-1 s=READY pgs=450 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:31:54.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:54.334+0000 7f1969dee700 1 -- 172.21.15.42:0/3588366297 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f19640a2810 msgr2=0x7f19640a1800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:31:54.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:54.334+0000 7f1969dee700 1 --2- 172.21.15.42:0/3588366297 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f19640a2810 0x7f19640a1800 secure :-1 s=READY pgs=1056 cs=0 l=1 rx=0x7f1964132120 tx=0x7f194c010ca0).stop 2019-12-05T05:31:54.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:54.334+0000 7f1969dee700 1 -- 172.21.15.42:0/3588366297 shutdown_connections 2019-12-05T05:31:54.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:54.334+0000 7f1969dee700 1 --2- 172.21.15.42:0/3588366297 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f194803aec0 0x7f194803d3d0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:31:54.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:54.334+0000 7f1969dee700 1 --2- 172.21.15.42:0/3588366297 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f19640a1d50 0x7f1964138000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:31:54.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:54.334+0000 7f1969dee700 1 --2- 172.21.15.42:0/3588366297 >> v2:172.21.15.40:3300/0 conn(0x7f19640a40c0 0x7f196411a4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:31:54.341 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:54.334+0000 7f1969dee700 1 --2- 172.21.15.42:0/3588366297 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f19640a2810 0x7f19640a1800 unknown :-1 s=CLOSED pgs=1056 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:31:54.341 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:54.334+0000 7f1969dee700 1 -- 172.21.15.42:0/3588366297 shutdown_connections 2019-12-05T05:31:54.341 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:54.334+0000 7f1969dee700 1 -- 172.21.15.42:0/3588366297 wait complete. 2019-12-05T05:31:54.341 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:54.334+0000 7f1969dee700 1 -- 172.21.15.42:0/3588366297 >> 172.21.15.42:0/3588366297 conn(0x7f19640a9860 msgr2=0x7f19640596f0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:31:54.349 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b has gained rank=0, replacing gid=6778 2019-12-05T05:31:54.349 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 4 secs before reviving mds.a 2019-12-05T05:31:58.350 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.a 2019-12-05T05:31:58.350 INFO:tasks.ceph.mds.a:Restarting daemon 2019-12-05T05:31:58.351 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2019-12-05T05:31:58.357 INFO:tasks.ceph.mds.a:Started 2019-12-05T05:31:58.357 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:31:58.417 INFO:tasks.ceph.mds.a.smithi042.stdout:starting mds.a at 2019-12-05T05:31:58.462 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:58.454+0000 7feca6eeb700 1 Processor -- start 2019-12-05T05:31:58.462 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:58.454+0000 7feca6eeb700 1 -- start start 2019-12-05T05:31:58.462 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:58.454+0000 7feca6eeb700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7feca00a7740 0x7feca008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:31:58.462 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:58.454+0000 7feca6eeb700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7feca009fae0 0x7feca007eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:31:58.462 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:58.454+0000 7feca6eeb700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feca00a0710 con 0x7feca007e720 2019-12-05T05:31:58.463 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:58.454+0000 7feca6eeb700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feca00a0b00 con 0x7feca009fae0 2019-12-05T05:31:58.463 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:58.454+0000 7feca6eeb700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feca008e5b0 con 0x7feca00a7740 2019-12-05T05:31:58.463 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:58.454+0000 7feca4c87700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7feca007e720 0x7feca009f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:35858/0 (socket says 172.21.15.42:35858) 2019-12-05T05:31:58.463 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:58.454+0000 7feca4c87700 1 -- 172.21.15.42:0/373725477 learned_addr learned my addr 172.21.15.42:0/373725477 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:31:58.463 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:58.454+0000 7feca5488700 1 --2- 172.21.15.42:0/373725477 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7feca00a7740 0x7feca008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:31:58.463 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:58.454+0000 7feca5c89700 1 --2- 172.21.15.42:0/373725477 >> v2:172.21.15.40:3300/0 conn(0x7feca009fae0 0x7feca007eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:31:58.463 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:58.454+0000 7fec97fff700 1 -- 172.21.15.42:0/373725477 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2277842148 0 0) 0x7feca00a0710 con 0x7feca007e720 2019-12-05T05:31:58.464 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:58.454+0000 7fec97fff700 1 -- 172.21.15.42:0/373725477 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec8c0030b0 con 0x7feca007e720 2019-12-05T05:31:58.464 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:58.454+0000 7feca5c89700 1 -- 172.21.15.42:0/373725477 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7feca00a7740 msgr2=0x7feca008df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:31:58.464 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:58.454+0000 7feca5c89700 1 --2- 172.21.15.42:0/373725477 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7feca00a7740 0x7feca008df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:31:58.464 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:58.454+0000 7feca5c89700 1 -- 172.21.15.42:0/373725477 >> v1:172.21.15.42:6789/0 conn(0x7feca007e720 legacy=0x7feca009f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:31:58.464 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:58.454+0000 7feca5c89700 1 -- 172.21.15.42:0/373725477 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feca007f0a0 con 0x7feca009fae0 2019-12-05T05:31:58.464 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:58.454+0000 7feca5488700 1 --2- 172.21.15.42:0/373725477 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7feca00a7740 0x7feca008df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:31:58.464 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:58.454+0000 7feca5c89700 1 --2- 172.21.15.42:0/373725477 >> v2:172.21.15.40:3300/0 conn(0x7feca009fae0 0x7feca007eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fec90008790 tx=0x7fec9000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:58.465 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:58.454+0000 7feca5c89700 1 --2- 172.21.15.42:0/373725477 >> v2:172.21.15.40:3300/0 conn(0x7feca009fae0 0x7feca007eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fec90008790 tx=0x7fec9000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:58.465 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:58.454+0000 7feca5c89700 1 --2- 172.21.15.42:0/373725477 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feca009fae0 0x7feca007eb50 secure :-1 s=READY pgs=825 cs=0 l=1 rx=0x7fec90008790 tx=0x7fec9000fc70).ready entity=mon.0 client_cookie=ea925a5db28fba46 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:31:58.465 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:58.458+0000 7feca5c89700 1 --2- 172.21.15.42:0/373725477 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feca009fae0 0x7feca007eb50 secure :-1 s=THROTTLE_DONE pgs=825 cs=0 l=1 rx=0x7fec90008790 tx=0x7fec9000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:58.465 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:58.458+0000 7fec97fff700 1 -- 172.21.15.42:0/373725477 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fec90008bb0 con 0x7feca009fae0 2019-12-05T05:31:58.465 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:58.458+0000 7feca5c89700 1 --2- 172.21.15.42:0/373725477 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feca009fae0 0x7feca007eb50 secure :-1 s=THROTTLE_DONE pgs=825 cs=0 l=1 rx=0x7fec90008790 tx=0x7fec9000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:58.465 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:58.458+0000 7feca5c89700 1 --2- 172.21.15.42:0/373725477 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feca009fae0 0x7feca007eb50 secure :-1 s=THROTTLE_DONE pgs=825 cs=0 l=1 rx=0x7fec90008790 tx=0x7fec9000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:58.465 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:58.458+0000 7fec97fff700 1 -- 172.21.15.42:0/373725477 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec900087d0 con 0x7feca009fae0 2019-12-05T05:31:58.466 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:58.458+0000 7fec97fff700 1 -- 172.21.15.42:0/373725477 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fec90011860 con 0x7feca009fae0 2019-12-05T05:31:58.466 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:58.458+0000 7feca6eeb700 1 -- 172.21.15.42:0/373725477 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feca009fae0 msgr2=0x7feca007eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:31:58.466 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:58.458+0000 7feca6eeb700 1 --2- 172.21.15.42:0/373725477 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feca009fae0 0x7feca007eb50 secure :-1 s=READY pgs=825 cs=0 l=1 rx=0x7fec90008790 tx=0x7fec9000fc70).stop 2019-12-05T05:31:58.466 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:58.458+0000 7feca6eeb700 1 -- 172.21.15.42:0/373725477 shutdown_connections 2019-12-05T05:31:58.466 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:58.458+0000 7feca6eeb700 1 --2- 172.21.15.42:0/373725477 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feca009fae0 0x7feca007eb50 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:31:58.466 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:58.458+0000 7feca6eeb700 1 --2- 172.21.15.42:0/373725477 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7feca00a7740 0x7feca008df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:31:58.466 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:58.458+0000 7feca6eeb700 1 -- 172.21.15.42:0/373725477 shutdown_connections 2019-12-05T05:31:58.467 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:58.458+0000 7feca6eeb700 1 -- 172.21.15.42:0/373725477 wait complete. 2019-12-05T05:31:58.467 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:58.458+0000 7feca6eeb700 1 -- 172.21.15.42:0/373725477 >> 172.21.15.42:0/373725477 conn(0x7feca00aabe0 msgr2=0x7feca00a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:31:58.470 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:58.458+0000 7feca6eeb700 1 Processor -- start 2019-12-05T05:31:58.470 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:58.458+0000 7feca6eeb700 1 -- start start 2019-12-05T05:31:58.470 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:58.462+0000 7feca6eeb700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7feca011a520 0x7feca007d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:31:58.470 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:58.462+0000 7feca6eeb700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7feca0121330 0x7feca007dd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:31:58.470 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:58.462+0000 7feca6eeb700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feca011b780 con 0x7feca0117620 2019-12-05T05:31:58.470 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:58.462+0000 7feca6eeb700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feca011bb00 con 0x7feca011a520 2019-12-05T05:31:58.471 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:58.462+0000 7feca6eeb700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feca011bda0 con 0x7feca0121330 2019-12-05T05:31:58.471 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:58.462+0000 7feca4c87700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7feca0121330 0x7feca007dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:31:58.471 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:58.462+0000 7feca5488700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7feca011a520 0x7feca007d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:31:58.471 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:58.462+0000 7feca4c87700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7feca0121330 0x7feca007dd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:35826/0 (socket says 172.21.15.42:35826) 2019-12-05T05:31:58.471 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:58.462+0000 7feca4c87700 1 -- 172.21.15.42:0/3726381842 learned_addr learned my addr 172.21.15.42:0/3726381842 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:31:58.471 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:58.462+0000 7feca4c87700 1 -- 172.21.15.42:0/3726381842 >> v2:172.21.15.40:3300/0 conn(0x7feca011a520 msgr2=0x7feca007d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:31:58.471 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:58.462+0000 7feca4c87700 1 --2- 172.21.15.42:0/3726381842 >> v2:172.21.15.40:3300/0 conn(0x7feca011a520 0x7feca007d7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:31:58.472 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:58.462+0000 7feca4c87700 1 -- 172.21.15.42:0/3726381842 >> v1:172.21.15.42:6789/0 conn(0x7feca0117620 legacy=0x7feca011dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:31:58.472 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:58.462+0000 7feca4c87700 1 -- 172.21.15.42:0/3726381842 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feca0118420 con 0x7feca0121330 2019-12-05T05:31:58.472 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:58.462+0000 7feca4c87700 1 --2- 172.21.15.42:0/3726381842 >> v2:172.21.15.42:3300/0 conn(0x7feca0121330 0x7feca007dd10 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fec9c004370 tx=0x7fec9c010a00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:58.472 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:58.462+0000 7feca4c87700 1 --2- 172.21.15.42:0/3726381842 >> v2:172.21.15.42:3300/0 conn(0x7feca0121330 0x7feca007dd10 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fec9c004370 tx=0x7fec9c010a00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:58.472 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:58.462+0000 7feca4c87700 1 --2- 172.21.15.42:0/3726381842 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feca0121330 0x7feca007dd10 secure :-1 s=READY pgs=1061 cs=0 l=1 rx=0x7fec9c004370 tx=0x7fec9c010a00).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:31:58.472 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:58.462+0000 7feca4c87700 1 --2- 172.21.15.42:0/3726381842 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feca0121330 0x7feca007dd10 secure :-1 s=THROTTLE_DONE pgs=1061 cs=0 l=1 rx=0x7fec9c004370 tx=0x7fec9c010a00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:58.472 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:58.462+0000 7feca4c87700 1 --2- 172.21.15.42:0/3726381842 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feca0121330 0x7feca007dd10 secure :-1 s=THROTTLE_DONE pgs=1061 cs=0 l=1 rx=0x7fec9c004370 tx=0x7fec9c010a00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:58.473 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:58.462+0000 7fec95ffb700 1 -- 172.21.15.42:0/3726381842 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fec9c00c9c0 con 0x7feca0121330 2019-12-05T05:31:58.473 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:58.462+0000 7feca4c87700 1 --2- 172.21.15.42:0/3726381842 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feca0121330 0x7feca007dd10 secure :-1 s=THROTTLE_DONE pgs=1061 cs=0 l=1 rx=0x7fec9c004370 tx=0x7fec9c010a00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:58.473 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:58.462+0000 7feca6eeb700 1 -- 172.21.15.42:0/3726381842 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feca0117a50 con 0x7feca0121330 2019-12-05T05:31:58.473 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:58.462+0000 7fec95ffb700 1 -- 172.21.15.42:0/3726381842 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec9c010250 con 0x7feca0121330 2019-12-05T05:31:58.473 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:58.462+0000 7fec95ffb700 1 -- 172.21.15.42:0/3726381842 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fec9c00d0c0 con 0x7feca0121330 2019-12-05T05:31:58.473 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:58.462+0000 7feca6eeb700 1 -- 172.21.15.42:0/3726381842 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feca0117f80 con 0x7feca0121330 2019-12-05T05:31:58.473 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:58.462+0000 7feca4c87700 1 --2- 172.21.15.42:0/3726381842 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feca0121330 0x7feca007dd10 secure :-1 s=THROTTLE_DONE pgs=1061 cs=0 l=1 rx=0x7fec9c004370 tx=0x7fec9c010a00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:58.474 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:58.466+0000 7fec95ffb700 1 -- 172.21.15.42:0/3726381842 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fec9c005460 con 0x7feca0121330 2019-12-05T05:31:58.474 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:58.466+0000 7feca4c87700 1 --2- 172.21.15.42:0/3726381842 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feca0121330 0x7feca007dd10 secure :-1 s=THROTTLE_DONE pgs=1061 cs=0 l=1 rx=0x7fec9c004370 tx=0x7fec9c010a00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:58.474 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:58.466+0000 7fec95ffb700 1 --2- 172.21.15.42:0/3726381842 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fec8003adc0 0x7fec8003d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:31:58.474 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:58.466+0000 7fec95ffb700 1 -- 172.21.15.42:0/3726381842 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7751+0+0 (secure 0 0 0) 0x7fec9c04ab30 con 0x7feca0121330 2019-12-05T05:31:58.474 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:58.466+0000 7feca5488700 1 --2- 172.21.15.42:0/3726381842 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fec8003adc0 0x7fec8003d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:31:58.474 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:58.466+0000 7feca5488700 1 --2- 172.21.15.42:0/3726381842 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fec8003adc0 0x7fec8003d2d0 crc :-1 s=READY pgs=451 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:31:58.475 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:58.470+0000 7feca6eeb700 1 -- 172.21.15.42:0/3726381842 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feca005d110 con 0x7feca0121330 2019-12-05T05:31:58.478 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:58.474+0000 7feca4c87700 1 --2- 172.21.15.42:0/3726381842 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feca0121330 0x7feca007dd10 secure :-1 s=THROTTLE_DONE pgs=1061 cs=0 l=1 rx=0x7fec9c004370 tx=0x7fec9c010a00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:58.478 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:58.474+0000 7fec95ffb700 1 -- 172.21.15.42:0/3726381842 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fec9c031bb0 con 0x7feca0121330 2019-12-05T05:31:58.688 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:58.686+0000 7feca6eeb700 1 -- 172.21.15.42:0/3726381842 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7feca005b8c0 con 0x7feca0121330 2019-12-05T05:31:58.690 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:58.686+0000 7feca4c87700 1 --2- 172.21.15.42:0/3726381842 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feca0121330 0x7feca007dd10 secure :-1 s=THROTTLE_DONE pgs=1061 cs=0 l=1 rx=0x7fec9c004370 tx=0x7fec9c010a00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:31:58.690 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:58.686+0000 7fec95ffb700 1 -- 172.21.15.42:0/3726381842 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 132 v132) v1 ==== 95+0+1775 (secure 0 0 0) 0x7fec9c031e10 con 0x7feca0121330 2019-12-05T05:31:58.690 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 132 2019-12-05T05:31:58.690 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:31:58.691 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":132,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":132,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:31:53.749802+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":69,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6984},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6984":{"gid":6984,"name":"b","rank":0,"incarnation":129,"state":"up:active","state_seq":15,"addr":"172.21.15.40:6833/3543033403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3543033403},{"type":"v1","addr":"172.21.15.40:6833","nonce":3543033403}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:31:58.692 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:58.686+0000 7feca6eeb700 1 -- 172.21.15.42:0/3726381842 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fec8003adc0 msgr2=0x7fec8003d2d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:31:58.692 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:58.686+0000 7feca6eeb700 1 --2- 172.21.15.42:0/3726381842 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fec8003adc0 0x7fec8003d2d0 crc :-1 s=READY pgs=451 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:31:58.693 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:58.686+0000 7feca6eeb700 1 -- 172.21.15.42:0/3726381842 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feca0121330 msgr2=0x7feca007dd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:31:58.693 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:58.686+0000 7feca6eeb700 1 --2- 172.21.15.42:0/3726381842 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feca0121330 0x7feca007dd10 secure :-1 s=READY pgs=1061 cs=0 l=1 rx=0x7fec9c004370 tx=0x7fec9c010a00).stop 2019-12-05T05:31:58.694 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:58.686+0000 7feca6eeb700 1 -- 172.21.15.42:0/3726381842 shutdown_connections 2019-12-05T05:31:58.694 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:58.686+0000 7feca6eeb700 1 --2- 172.21.15.42:0/3726381842 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fec8003adc0 0x7fec8003d2d0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:31:58.694 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:58.686+0000 7feca6eeb700 1 --2- 172.21.15.42:0/3726381842 >> v2:172.21.15.40:3300/0 conn(0x7feca011a520 0x7feca007d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:31:58.694 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:58.686+0000 7feca6eeb700 1 --2- 172.21.15.42:0/3726381842 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feca0121330 0x7feca007dd10 unknown :-1 s=CLOSED pgs=1061 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:31:58.694 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:58.690+0000 7feca6eeb700 1 -- 172.21.15.42:0/3726381842 shutdown_connections 2019-12-05T05:31:58.695 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:58.690+0000 7feca6eeb700 1 -- 172.21.15.42:0/3726381842 wait complete. 2019-12-05T05:31:58.695 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:31:58.690+0000 7feca6eeb700 1 -- 172.21.15.42:0/3726381842 >> 172.21.15.42:0/3726381842 conn(0x7feca007e590 msgr2=0x7feca00a9790 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:31:58.701 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2019-12-05T05:32:00.702 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:32:00.813 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:00.806+0000 7fb0d3937700 1 Processor -- start 2019-12-05T05:32:00.814 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:00.806+0000 7fb0d3937700 1 -- start start 2019-12-05T05:32:00.815 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:00.806+0000 7fb0d3937700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb0cc0a7740 0x7fb0cc08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:32:00.815 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:00.806+0000 7fb0d3937700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0cc0a0710 con 0x7fb0cc09fae0 2019-12-05T05:32:00.815 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:00.806+0000 7fb0d3937700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0cc0a9ba0 con 0x7fb0cc07e720 2019-12-05T05:32:00.815 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:00.806+0000 7fb0d3937700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0cc0a0b00 con 0x7fb0cc0a7740 2019-12-05T05:32:00.815 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:00.806+0000 7fb0d16d3700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fb0cc07e720 0x7fb0cc09f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:35880/0 (socket says 172.21.15.42:35880) 2019-12-05T05:32:00.815 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:00.806+0000 7fb0d16d3700 1 -- 172.21.15.42:0/1309521541 learned_addr learned my addr 172.21.15.42:0/1309521541 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:32:00.815 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:00.806+0000 7fb0d1ed4700 1 --2- 172.21.15.42:0/1309521541 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb0cc0a7740 0x7fb0cc08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:32:00.816 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:00.806+0000 7fb0d0ed2700 1 -- 172.21.15.42:0/1309521541 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 870136692 0 0) 0x7fb0cc0a9ba0 con 0x7fb0cc07e720 2019-12-05T05:32:00.816 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:00.806+0000 7fb0d0ed2700 1 -- 172.21.15.42:0/1309521541 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0b80030b0 con 0x7fb0cc07e720 2019-12-05T05:32:00.816 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:00.806+0000 7fb0d0ed2700 1 -- 172.21.15.42:0/1309521541 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 845950844 0 0) 0x7fb0cc0a0710 con 0x7fb0cc09fae0 2019-12-05T05:32:00.816 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:00.806+0000 7fb0d0ed2700 1 -- 172.21.15.42:0/1309521541 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0cc0a9ba0 con 0x7fb0cc09fae0 2019-12-05T05:32:00.816 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:00.806+0000 7fb0d1ed4700 1 -- 172.21.15.42:0/1309521541 >> v1:172.21.15.42:6789/0 conn(0x7fb0cc07e720 legacy=0x7fb0cc09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:32:00.816 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:00.806+0000 7fb0d1ed4700 1 -- 172.21.15.42:0/1309521541 >> v1:172.21.15.40:6789/0 conn(0x7fb0cc09fae0 legacy=0x7fb0cc14efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:32:00.816 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:00.806+0000 7fb0d1ed4700 1 -- 172.21.15.42:0/1309521541 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0cc07eb50 con 0x7fb0cc0a7740 2019-12-05T05:32:00.817 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:00.806+0000 7fb0d1ed4700 1 --2- 172.21.15.42:0/1309521541 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb0cc0a7740 0x7fb0cc08df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fb0c40032b0 tx=0x7fb0c400ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:00.817 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:00.806+0000 7fb0d1ed4700 1 --2- 172.21.15.42:0/1309521541 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb0cc0a7740 0x7fb0cc08df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fb0c40032b0 tx=0x7fb0c400ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:00.817 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:00.806+0000 7fb0d1ed4700 1 --2- 172.21.15.42:0/1309521541 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb0cc0a7740 0x7fb0cc08df80 secure :-1 s=READY pgs=216 cs=0 l=1 rx=0x7fb0c40032b0 tx=0x7fb0c400ff70).ready entity=mon.2 client_cookie=bc173fe3c46cf208 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:32:00.817 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:00.806+0000 7fb0d1ed4700 1 --2- 172.21.15.42:0/1309521541 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb0cc0a7740 0x7fb0cc08df80 secure :-1 s=THROTTLE_DONE pgs=216 cs=0 l=1 rx=0x7fb0c40032b0 tx=0x7fb0c400ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:00.817 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:00.806+0000 7fb0d0ed2700 1 -- 172.21.15.42:0/1309521541 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb0c4011a10 con 0x7fb0cc0a7740 2019-12-05T05:32:00.817 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:00.806+0000 7fb0d1ed4700 1 --2- 172.21.15.42:0/1309521541 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb0cc0a7740 0x7fb0cc08df80 secure :-1 s=THROTTLE_DONE pgs=216 cs=0 l=1 rx=0x7fb0c40032b0 tx=0x7fb0c400ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:00.818 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:00.806+0000 7fb0d1ed4700 1 --2- 172.21.15.42:0/1309521541 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb0cc0a7740 0x7fb0cc08df80 secure :-1 s=THROTTLE_DONE pgs=216 cs=0 l=1 rx=0x7fb0c40032b0 tx=0x7fb0c400ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:00.818 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:00.806+0000 7fb0d0ed2700 1 -- 172.21.15.42:0/1309521541 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0c4011630 con 0x7fb0cc0a7740 2019-12-05T05:32:00.818 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:00.806+0000 7fb0d0ed2700 1 -- 172.21.15.42:0/1309521541 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb0c400a160 con 0x7fb0cc0a7740 2019-12-05T05:32:00.818 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:00.806+0000 7fb0d3937700 1 -- 172.21.15.42:0/1309521541 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb0cc0a7740 msgr2=0x7fb0cc08df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:32:00.818 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:00.806+0000 7fb0d3937700 1 --2- 172.21.15.42:0/1309521541 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb0cc0a7740 0x7fb0cc08df80 secure :-1 s=READY pgs=216 cs=0 l=1 rx=0x7fb0c40032b0 tx=0x7fb0c400ff70).stop 2019-12-05T05:32:00.818 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:00.810+0000 7fb0d3937700 1 -- 172.21.15.42:0/1309521541 shutdown_connections 2019-12-05T05:32:00.819 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:00.810+0000 7fb0d3937700 1 --2- 172.21.15.42:0/1309521541 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb0cc0a7740 0x7fb0cc08df80 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:32:00.819 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:00.810+0000 7fb0d3937700 1 -- 172.21.15.42:0/1309521541 shutdown_connections 2019-12-05T05:32:00.819 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:00.810+0000 7fb0d3937700 1 -- 172.21.15.42:0/1309521541 wait complete. 2019-12-05T05:32:00.819 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:00.810+0000 7fb0d3937700 1 -- 172.21.15.42:0/1309521541 >> 172.21.15.42:0/1309521541 conn(0x7fb0cc0aabe0 msgr2=0x7fb0cc0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:32:00.822 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:00.810+0000 7fb0d3937700 1 Processor -- start 2019-12-05T05:32:00.822 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:00.810+0000 7fb0d3937700 1 -- start start 2019-12-05T05:32:00.822 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:00.814+0000 7fb0d3937700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb0cc064f90 0x7fb0cc0a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:32:00.823 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:00.814+0000 7fb0d3937700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0cc142b70 con 0x7fb0cc1433c0 2019-12-05T05:32:00.823 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:00.814+0000 7fb0d3937700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0cc0a3b30 con 0x7fb0cc063640 2019-12-05T05:32:00.823 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:00.814+0000 7fb0d3937700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0cc150140 con 0x7fb0cc064f90 2019-12-05T05:32:00.823 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:00.814+0000 7fb0d16d3700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb0cc064f90 0x7fb0cc0a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:32:00.823 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:00.814+0000 7fb0d26d5700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fb0cc063640 0x7fb0cc12ff60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:35888/0 (socket says 172.21.15.42:35888) 2019-12-05T05:32:00.823 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:00.814+0000 7fb0d1ed4700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fb0cc1433c0 0x7fb0cc07ac70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:45324/0 (socket says 172.21.15.42:45324) 2019-12-05T05:32:00.824 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:00.814+0000 7fb0d26d5700 1 -- 172.21.15.42:0/709028032 learned_addr learned my addr 172.21.15.42:0/709028032 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:32:00.824 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:00.814+0000 7fb0d16d3700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb0cc064f90 0x7fb0cc0a35e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:35848/0 (socket says 172.21.15.42:35848) 2019-12-05T05:32:00.824 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:00.814+0000 7fb0d16d3700 1 -- 172.21.15.42:0/709028032 >> v1:172.21.15.42:6789/0 conn(0x7fb0cc063640 legacy=0x7fb0cc12ff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:32:00.825 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:00.814+0000 7fb0d16d3700 1 -- 172.21.15.42:0/709028032 >> v1:172.21.15.40:6789/0 conn(0x7fb0cc1433c0 legacy=0x7fb0cc07ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:32:00.825 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:00.814+0000 7fb0c27fc700 1 -- 172.21.15.42:0/709028032 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1087732781 0 0) 0x7fb0cc0a3b30 con 0x7fb0cc063640 2019-12-05T05:32:00.825 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:00.814+0000 7fb0d16d3700 1 -- 172.21.15.42:0/709028032 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0cc064390 con 0x7fb0cc064f90 2019-12-05T05:32:00.825 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:00.814+0000 7fb0d16d3700 1 --2- 172.21.15.42:0/709028032 >> v2:172.21.15.42:3300/0 conn(0x7fb0cc064f90 0x7fb0cc0a35e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fb0c40086f0 tx=0x7fb0c400ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:00.825 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:00.814+0000 7fb0d16d3700 1 --2- 172.21.15.42:0/709028032 >> v2:172.21.15.42:3300/0 conn(0x7fb0cc064f90 0x7fb0cc0a35e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fb0c40086f0 tx=0x7fb0c400ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:00.826 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:00.814+0000 7fb0d16d3700 1 --2- 172.21.15.42:0/709028032 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb0cc064f90 0x7fb0cc0a35e0 secure :-1 s=READY pgs=1064 cs=0 l=1 rx=0x7fb0c40086f0 tx=0x7fb0c400ff70).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:32:00.826 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:00.814+0000 7fb0d16d3700 1 --2- 172.21.15.42:0/709028032 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb0cc064f90 0x7fb0cc0a35e0 secure :-1 s=THROTTLE_DONE pgs=1064 cs=0 l=1 rx=0x7fb0c40086f0 tx=0x7fb0c400ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:00.826 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:00.814+0000 7fb0c27fc700 1 -- 172.21.15.42:0/709028032 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb0bc005a20 con 0x7fb0cc064f90 2019-12-05T05:32:00.826 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:00.814+0000 7fb0d16d3700 1 --2- 172.21.15.42:0/709028032 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb0cc064f90 0x7fb0cc0a35e0 secure :-1 s=THROTTLE_DONE pgs=1064 cs=0 l=1 rx=0x7fb0c40086f0 tx=0x7fb0c400ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:00.826 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:00.814+0000 7fb0d16d3700 1 --2- 172.21.15.42:0/709028032 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb0cc064f90 0x7fb0cc0a35e0 secure :-1 s=THROTTLE_DONE pgs=1064 cs=0 l=1 rx=0x7fb0c40086f0 tx=0x7fb0c400ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:00.826 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:00.814+0000 7fb0c27fc700 1 -- 172.21.15.42:0/709028032 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0bc005640 con 0x7fb0cc064f90 2019-12-05T05:32:00.827 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:00.814+0000 7fb0d3937700 1 -- 172.21.15.42:0/709028032 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0cc063a70 con 0x7fb0cc064f90 2019-12-05T05:32:00.827 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:00.814+0000 7fb0c27fc700 1 -- 172.21.15.42:0/709028032 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb0bc0060c0 con 0x7fb0cc064f90 2019-12-05T05:32:00.827 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:00.814+0000 7fb0d3937700 1 -- 172.21.15.42:0/709028032 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0cc131770 con 0x7fb0cc064f90 2019-12-05T05:32:00.827 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:00.814+0000 7fb0d16d3700 1 --2- 172.21.15.42:0/709028032 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb0cc064f90 0x7fb0cc0a35e0 secure :-1 s=THROTTLE_DONE pgs=1064 cs=0 l=1 rx=0x7fb0c40086f0 tx=0x7fb0c400ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:00.827 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:00.818+0000 7fb0c27fc700 1 -- 172.21.15.42:0/709028032 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fb0bc00a320 con 0x7fb0cc064f90 2019-12-05T05:32:00.827 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:00.818+0000 7fb0d16d3700 1 --2- 172.21.15.42:0/709028032 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb0cc064f90 0x7fb0cc0a35e0 secure :-1 s=THROTTLE_DONE pgs=1064 cs=0 l=1 rx=0x7fb0c40086f0 tx=0x7fb0c400ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:00.827 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:00.818+0000 7fb0c27fc700 1 --2- 172.21.15.42:0/709028032 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb0b403ac20 0x7fb0b403d130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:32:00.828 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:00.818+0000 7fb0c27fc700 1 -- 172.21.15.42:0/709028032 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7751+0+0 (secure 0 0 0) 0x7fb0bc04ae20 con 0x7fb0cc064f90 2019-12-05T05:32:00.828 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:00.818+0000 7fb0d1ed4700 1 --2- 172.21.15.42:0/709028032 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb0b403ac20 0x7fb0b403d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:32:00.828 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:00.818+0000 7fb0d1ed4700 1 --2- 172.21.15.42:0/709028032 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb0b403ac20 0x7fb0b403d130 crc :-1 s=READY pgs=452 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:32:00.828 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:00.822+0000 7fb0d3937700 1 -- 172.21.15.42:0/709028032 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0cc05d110 con 0x7fb0cc064f90 2019-12-05T05:32:00.831 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:00.826+0000 7fb0d16d3700 1 --2- 172.21.15.42:0/709028032 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb0cc064f90 0x7fb0cc0a35e0 secure :-1 s=THROTTLE_DONE pgs=1064 cs=0 l=1 rx=0x7fb0c40086f0 tx=0x7fb0c400ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:00.831 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:00.826+0000 7fb0c27fc700 1 -- 172.21.15.42:0/709028032 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fb0bc031e80 con 0x7fb0cc064f90 2019-12-05T05:32:00.996 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:00.994+0000 7fb0d3937700 1 -- 172.21.15.42:0/709028032 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb0cc05b8c0 con 0x7fb0cc064f90 2019-12-05T05:32:00.997 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:00.994+0000 7fb0d16d3700 1 --2- 172.21.15.42:0/709028032 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb0cc064f90 0x7fb0cc0a35e0 secure :-1 s=THROTTLE_DONE pgs=1064 cs=0 l=1 rx=0x7fb0c40086f0 tx=0x7fb0c400ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:00.997 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:00.994+0000 7fb0c27fc700 1 -- 172.21.15.42:0/709028032 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 132 v132) v1 ==== 95+0+1775 (secure 0 0 0) 0x7fb0bc0320e0 con 0x7fb0cc064f90 2019-12-05T05:32:00.997 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 132 2019-12-05T05:32:00.998 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:32:00.998 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":132,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":132,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:31:53.749802+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":69,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6984},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6984":{"gid":6984,"name":"b","rank":0,"incarnation":129,"state":"up:active","state_seq":15,"addr":"172.21.15.40:6833/3543033403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3543033403},{"type":"v1","addr":"172.21.15.40:6833","nonce":3543033403}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:32:00.999 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:00.994+0000 7fb0d3937700 1 -- 172.21.15.42:0/709028032 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb0b403ac20 msgr2=0x7fb0b403d130 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:32:01.000 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:00.994+0000 7fb0d3937700 1 --2- 172.21.15.42:0/709028032 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb0b403ac20 0x7fb0b403d130 crc :-1 s=READY pgs=452 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:32:01.000 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:00.994+0000 7fb0d3937700 1 -- 172.21.15.42:0/709028032 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb0cc064f90 msgr2=0x7fb0cc0a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:32:01.000 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:00.994+0000 7fb0d3937700 1 --2- 172.21.15.42:0/709028032 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb0cc064f90 0x7fb0cc0a35e0 secure :-1 s=READY pgs=1064 cs=0 l=1 rx=0x7fb0c40086f0 tx=0x7fb0c400ff70).stop 2019-12-05T05:32:01.000 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:00.994+0000 7fb0d3937700 1 -- 172.21.15.42:0/709028032 shutdown_connections 2019-12-05T05:32:01.000 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:00.994+0000 7fb0d3937700 1 --2- 172.21.15.42:0/709028032 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb0b403ac20 0x7fb0b403d130 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:32:01.000 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:00.994+0000 7fb0d3937700 1 --2- 172.21.15.42:0/709028032 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb0cc064f90 0x7fb0cc0a35e0 unknown :-1 s=CLOSED pgs=1064 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:32:01.001 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:00.994+0000 7fb0d3937700 1 -- 172.21.15.42:0/709028032 shutdown_connections 2019-12-05T05:32:01.001 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:00.998+0000 7fb0d3937700 1 -- 172.21.15.42:0/709028032 wait complete. 2019-12-05T05:32:01.001 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:00.998+0000 7fb0d3937700 1 -- 172.21.15.42:0/709028032 >> 172.21.15.42:0/709028032 conn(0x7fb0cc08fde0 msgr2=0x7fb0cc0a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:32:01.009 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2019-12-05T05:32:03.010 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:32:03.115 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:03.106+0000 7f17a8924700 1 Processor -- start 2019-12-05T05:32:03.115 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:03.106+0000 7f17a8924700 1 -- start start 2019-12-05T05:32:03.115 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:03.106+0000 7f17a8924700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f17a40a7740 0x7f17a408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:32:03.115 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:03.106+0000 7f17a8924700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f17a409fae0 0x7f17a407eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:32:03.115 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:03.106+0000 7f17a8924700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17a40a0710 con 0x7f17a407e720 2019-12-05T05:32:03.115 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:03.106+0000 7f17a8924700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17a40a0b00 con 0x7f17a40a7740 2019-12-05T05:32:03.116 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:03.106+0000 7f17a8924700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17a408e5b0 con 0x7f17a409fae0 2019-12-05T05:32:03.116 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:03.106+0000 7f17a2d9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f17a40a7740 0x7f17a408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:32:03.116 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:03.110+0000 7f17a2d9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f17a40a7740 0x7f17a408df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:35866/0 (socket says 172.21.15.42:35866) 2019-12-05T05:32:03.116 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:03.110+0000 7f17a2d9d700 1 -- 172.21.15.42:0/1244620217 learned_addr learned my addr 172.21.15.42:0/1244620217 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:32:03.116 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:03.110+0000 7f17a359e700 1 --2- 172.21.15.42:0/1244620217 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f17a409fae0 0x7f17a407eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:32:03.116 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:03.110+0000 7f17a2d9d700 1 -- 172.21.15.42:0/1244620217 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f17a409fae0 msgr2=0x7f17a407eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:32:03.116 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:03.110+0000 7f17a2d9d700 1 --2- 172.21.15.42:0/1244620217 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f17a409fae0 0x7f17a407eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:32:03.117 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:03.110+0000 7f17a1d9b700 1 -- 172.21.15.42:0/1244620217 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1250747665 0 0) 0x7f17a40a0710 con 0x7f17a407e720 2019-12-05T05:32:03.117 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:03.110+0000 7f17a2d9d700 1 -- 172.21.15.42:0/1244620217 >> v1:172.21.15.40:6789/0 conn(0x7f17a407e720 legacy=0x7f17a409f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:32:03.117 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:03.110+0000 7f17a2d9d700 1 -- 172.21.15.42:0/1244620217 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17a407f0a0 con 0x7f17a40a7740 2019-12-05T05:32:03.117 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:03.110+0000 7f17a2d9d700 1 --2- 172.21.15.42:0/1244620217 >> v2:172.21.15.42:3300/0 conn(0x7f17a40a7740 0x7f17a408df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f1794008790 tx=0x7f179400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:03.118 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:03.110+0000 7f17a2d9d700 1 --2- 172.21.15.42:0/1244620217 >> v2:172.21.15.42:3300/0 conn(0x7f17a40a7740 0x7f17a408df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f1794008790 tx=0x7f179400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:03.119 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:03.110+0000 7f17a2d9d700 1 --2- 172.21.15.42:0/1244620217 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f17a40a7740 0x7f17a408df80 secure :-1 s=READY pgs=1065 cs=0 l=1 rx=0x7f1794008790 tx=0x7f179400fd30).ready entity=mon.1 client_cookie=97658562369c5cd0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:32:03.119 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:03.110+0000 7f17a2d9d700 1 --2- 172.21.15.42:0/1244620217 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f17a40a7740 0x7f17a408df80 secure :-1 s=THROTTLE_DONE pgs=1065 cs=0 l=1 rx=0x7f1794008790 tx=0x7f179400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:03.119 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:03.110+0000 7f17a1d9b700 1 -- 172.21.15.42:0/1244620217 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1794008c90 con 0x7f17a40a7740 2019-12-05T05:32:03.119 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:03.110+0000 7f17a2d9d700 1 --2- 172.21.15.42:0/1244620217 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f17a40a7740 0x7f17a408df80 secure :-1 s=THROTTLE_DONE pgs=1065 cs=0 l=1 rx=0x7f1794008790 tx=0x7f179400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:03.119 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:03.110+0000 7f17a1d9b700 1 -- 172.21.15.42:0/1244620217 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17940088b0 con 0x7f17a40a7740 2019-12-05T05:32:03.119 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:03.110+0000 7f17a2d9d700 1 --2- 172.21.15.42:0/1244620217 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f17a40a7740 0x7f17a408df80 secure :-1 s=THROTTLE_DONE pgs=1065 cs=0 l=1 rx=0x7f1794008790 tx=0x7f179400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:03.119 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:03.110+0000 7f17a8924700 1 -- 172.21.15.42:0/1244620217 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f17a40a7740 msgr2=0x7f17a408df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:32:03.120 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:03.110+0000 7f17a1d9b700 1 -- 172.21.15.42:0/1244620217 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1794011b10 con 0x7f17a40a7740 2019-12-05T05:32:03.120 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:03.110+0000 7f17a8924700 1 --2- 172.21.15.42:0/1244620217 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f17a40a7740 0x7f17a408df80 secure :-1 s=READY pgs=1065 cs=0 l=1 rx=0x7f1794008790 tx=0x7f179400fd30).stop 2019-12-05T05:32:03.120 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:03.110+0000 7f17a8924700 1 -- 172.21.15.42:0/1244620217 shutdown_connections 2019-12-05T05:32:03.120 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:03.110+0000 7f17a8924700 1 --2- 172.21.15.42:0/1244620217 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f17a409fae0 0x7f17a407eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:32:03.120 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:03.110+0000 7f17a8924700 1 --2- 172.21.15.42:0/1244620217 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f17a40a7740 0x7f17a408df80 unknown :-1 s=CLOSED pgs=1065 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:32:03.121 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:03.110+0000 7f17a8924700 1 -- 172.21.15.42:0/1244620217 shutdown_connections 2019-12-05T05:32:03.121 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:03.110+0000 7f17a8924700 1 -- 172.21.15.42:0/1244620217 wait complete. 2019-12-05T05:32:03.121 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:03.110+0000 7f17a8924700 1 -- 172.21.15.42:0/1244620217 >> 172.21.15.42:0/1244620217 conn(0x7f17a40aabe0 msgr2=0x7f17a40a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:32:03.125 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:03.110+0000 7f17a8924700 1 Processor -- start 2019-12-05T05:32:03.125 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:03.114+0000 7f17a8924700 1 -- start start 2019-12-05T05:32:03.125 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:03.114+0000 7f17a8924700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f17a414d3c0 0x7f17a412a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:32:03.125 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:03.114+0000 7f17a8924700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f17a40a1d90 0x7f17a412aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:32:03.125 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:03.114+0000 7f17a8924700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17a40a4840 con 0x7f17a40a4f10 2019-12-05T05:32:03.125 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:03.114+0000 7f17a8924700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17a40a2310 con 0x7f17a40a1d90 2019-12-05T05:32:03.126 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:03.114+0000 7f17a8924700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17a40a25b0 con 0x7f17a414d3c0 2019-12-05T05:32:03.126 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:03.114+0000 7f17a2d9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f17a414d3c0 0x7f17a412a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:32:03.126 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:03.114+0000 7f17a2d9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f17a414d3c0 0x7f17a412a5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:35874/0 (socket says 172.21.15.42:35874) 2019-12-05T05:32:03.126 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:03.114+0000 7f17a2d9d700 1 -- 172.21.15.42:0/641476054 learned_addr learned my addr 172.21.15.42:0/641476054 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:32:03.126 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:03.114+0000 7f17a259c700 1 --2- 172.21.15.42:0/641476054 >> v2:172.21.15.40:3300/0 conn(0x7f17a40a1d90 0x7f17a412aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:32:03.126 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:03.114+0000 7f17a359e700 1 -- 172.21.15.42:0/641476054 >> v1:172.21.15.40:6789/0 conn(0x7f17a40a4f10 legacy=0x7f17a411a9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2019-12-05T05:32:03.126 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:03.114+0000 7f17a359e700 1 -- 172.21.15.42:0/641476054 >> v1:172.21.15.40:6789/0 conn(0x7f17a40a4f10 legacy=0x7f17a411a9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2019-12-05T05:32:03.127 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:03.114+0000 7f17a359e700 1 --1- 172.21.15.42:0/641476054 >> v1:172.21.15.40:6789/0 conn(0x7f17a40a4f10 0x7f17a411a9a0 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_1 read connect reply failed 2019-12-05T05:32:03.127 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:03.114+0000 7f17a359e700 1 --1- 172.21.15.42:0/641476054 >> v1:172.21.15.40:6789/0 conn(0x7f17a40a4f10 0x7f17a411a9a0 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).fault on lossy channel, failing 2019-12-05T05:32:03.127 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:03.114+0000 7f17a2d9d700 1 -- 172.21.15.42:0/641476054 >> v2:172.21.15.40:3300/0 conn(0x7f17a40a1d90 msgr2=0x7f17a412aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:32:03.127 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:03.114+0000 7f17a2d9d700 1 --2- 172.21.15.42:0/641476054 >> v2:172.21.15.40:3300/0 conn(0x7f17a40a1d90 0x7f17a412aaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:32:03.127 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:03.114+0000 7f17a2d9d700 1 -- 172.21.15.42:0/641476054 >> v1:172.21.15.40:6789/0 conn(0x7f17a40a4f10 legacy=0x7f17a411a9a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2019-12-05T05:32:03.127 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:03.114+0000 7f17a2d9d700 1 -- 172.21.15.42:0/641476054 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17a40a60a0 con 0x7f17a414d3c0 2019-12-05T05:32:03.127 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:03.114+0000 7f17a2d9d700 1 --2- 172.21.15.42:0/641476054 >> v2:172.21.15.42:3300/0 conn(0x7f17a414d3c0 0x7f17a412a5a0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f1794008790 tx=0x7f1794008f40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:03.129 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:03.114+0000 7f17a2d9d700 1 --2- 172.21.15.42:0/641476054 >> v2:172.21.15.42:3300/0 conn(0x7f17a414d3c0 0x7f17a412a5a0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f1794008790 tx=0x7f1794008f40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:03.129 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:03.114+0000 7f17a2d9d700 1 --2- 172.21.15.42:0/641476054 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f17a414d3c0 0x7f17a412a5a0 secure :-1 s=READY pgs=1066 cs=0 l=1 rx=0x7f1794008790 tx=0x7f1794008f40).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:32:03.129 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:03.114+0000 7f17a2d9d700 1 --2- 172.21.15.42:0/641476054 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f17a414d3c0 0x7f17a412a5a0 secure :-1 s=THROTTLE_DONE pgs=1066 cs=0 l=1 rx=0x7f1794008790 tx=0x7f1794008f40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:03.129 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:03.114+0000 7f179b7fe700 1 -- 172.21.15.42:0/641476054 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f179400f470 con 0x7f17a414d3c0 2019-12-05T05:32:03.130 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:03.114+0000 7f17a2d9d700 1 --2- 172.21.15.42:0/641476054 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f17a414d3c0 0x7f17a412a5a0 secure :-1 s=THROTTLE_DONE pgs=1066 cs=0 l=1 rx=0x7f1794008790 tx=0x7f1794008f40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:03.130 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:03.114+0000 7f179b7fe700 1 -- 172.21.15.42:0/641476054 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1794011730 con 0x7f17a414d3c0 2019-12-05T05:32:03.130 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:03.114+0000 7f17a8924700 1 -- 172.21.15.42:0/641476054 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17a40a5340 con 0x7f17a414d3c0 2019-12-05T05:32:03.130 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:03.114+0000 7f17a2d9d700 1 --2- 172.21.15.42:0/641476054 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f17a414d3c0 0x7f17a412a5a0 secure :-1 s=THROTTLE_DONE pgs=1066 cs=0 l=1 rx=0x7f1794008790 tx=0x7f1794008f40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:03.130 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:03.114+0000 7f179b7fe700 1 -- 172.21.15.42:0/641476054 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1794005730 con 0x7f17a414d3c0 2019-12-05T05:32:03.130 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:03.114+0000 7f17a8924700 1 -- 172.21.15.42:0/641476054 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17a40a5870 con 0x7f17a414d3c0 2019-12-05T05:32:03.131 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:03.114+0000 7f17a2d9d700 1 --2- 172.21.15.42:0/641476054 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f17a414d3c0 0x7f17a412a5a0 secure :-1 s=THROTTLE_DONE pgs=1066 cs=0 l=1 rx=0x7f1794008790 tx=0x7f1794008f40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:03.131 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:03.118+0000 7f179b7fe700 1 -- 172.21.15.42:0/641476054 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f17940059d0 con 0x7f17a414d3c0 2019-12-05T05:32:03.131 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:03.118+0000 7f17a2d9d700 1 --2- 172.21.15.42:0/641476054 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f17a414d3c0 0x7f17a412a5a0 secure :-1 s=THROTTLE_DONE pgs=1066 cs=0 l=1 rx=0x7f1794008790 tx=0x7f1794008f40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:03.131 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:03.118+0000 7f179b7fe700 1 --2- 172.21.15.42:0/641476054 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f178803adb0 0x7f178803d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:32:03.131 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:03.118+0000 7f179b7fe700 1 -- 172.21.15.42:0/641476054 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7751+0+0 (secure 0 0 0) 0x7f179404b7e0 con 0x7f17a414d3c0 2019-12-05T05:32:03.131 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:03.118+0000 7f17a259c700 1 --2- 172.21.15.42:0/641476054 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f178803adb0 0x7f178803d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:32:03.132 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:03.118+0000 7f17a259c700 1 --2- 172.21.15.42:0/641476054 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f178803adb0 0x7f178803d2c0 crc :-1 s=READY pgs=453 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:32:03.132 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:03.122+0000 7f17a8924700 1 -- 172.21.15.42:0/641476054 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17a405d110 con 0x7f17a414d3c0 2019-12-05T05:32:03.136 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:03.130+0000 7f17a2d9d700 1 --2- 172.21.15.42:0/641476054 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f17a414d3c0 0x7f17a412a5a0 secure :-1 s=THROTTLE_DONE pgs=1066 cs=0 l=1 rx=0x7f1794008790 tx=0x7f1794008f40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:03.136 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:03.130+0000 7f179b7fe700 1 -- 172.21.15.42:0/641476054 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f17940327a0 con 0x7f17a414d3c0 2019-12-05T05:32:03.299 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:03.294+0000 7f17a8924700 1 -- 172.21.15.42:0/641476054 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f17a405b8c0 con 0x7f17a414d3c0 2019-12-05T05:32:03.300 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:03.298+0000 7f17a2d9d700 1 --2- 172.21.15.42:0/641476054 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f17a414d3c0 0x7f17a412a5a0 secure :-1 s=THROTTLE_DONE pgs=1066 cs=0 l=1 rx=0x7f1794008790 tx=0x7f1794008f40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:03.300 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:03.298+0000 7f179b7fe700 1 -- 172.21.15.42:0/641476054 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 133 v133) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f1794032a00 con 0x7f17a414d3c0 2019-12-05T05:32:03.300 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 133 2019-12-05T05:32:03.301 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:32:03.302 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":133,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6997,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/1601334502","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1601334502},{"type":"v1","addr":"172.21.15.42:6835","nonce":1601334502}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":133}],"filesystems":[{"mdsmap":{"epoch":132,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:31:53.749802+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":69,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6984},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6984":{"gid":6984,"name":"b","rank":0,"incarnation":129,"state":"up:active","state_seq":15,"addr":"172.21.15.40:6833/3543033403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3543033403},{"type":"v1","addr":"172.21.15.40:6833","nonce":3543033403}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:32:03.302 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:03.298+0000 7f17a8924700 1 -- 172.21.15.42:0/641476054 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f178803adb0 msgr2=0x7f178803d2c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:32:03.303 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:03.298+0000 7f17a8924700 1 --2- 172.21.15.42:0/641476054 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f178803adb0 0x7f178803d2c0 crc :-1 s=READY pgs=453 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:32:03.303 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:03.298+0000 7f17a8924700 1 -- 172.21.15.42:0/641476054 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f17a414d3c0 msgr2=0x7f17a412a5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:32:03.303 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:03.298+0000 7f17a8924700 1 --2- 172.21.15.42:0/641476054 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f17a414d3c0 0x7f17a412a5a0 secure :-1 s=READY pgs=1066 cs=0 l=1 rx=0x7f1794008790 tx=0x7f1794008f40).stop 2019-12-05T05:32:03.303 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:03.298+0000 7f17a8924700 1 -- 172.21.15.42:0/641476054 shutdown_connections 2019-12-05T05:32:03.303 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:03.298+0000 7f17a8924700 1 --2- 172.21.15.42:0/641476054 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f178803adb0 0x7f178803d2c0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:32:03.303 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:03.298+0000 7f17a8924700 1 --2- 172.21.15.42:0/641476054 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f17a414d3c0 0x7f17a412a5a0 unknown :-1 s=CLOSED pgs=1066 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:32:03.304 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:03.298+0000 7f17a8924700 1 --2- 172.21.15.42:0/641476054 >> v2:172.21.15.40:3300/0 conn(0x7f17a40a1d90 0x7f17a412aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:32:03.304 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:03.298+0000 7f17a8924700 1 -- 172.21.15.42:0/641476054 shutdown_connections 2019-12-05T05:32:03.304 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:03.298+0000 7f17a8924700 1 -- 172.21.15.42:0/641476054 wait complete. 2019-12-05T05:32:03.304 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:03.298+0000 7f17a8924700 1 -- 172.21.15.42:0/641476054 >> 172.21.15.42:0/641476054 conn(0x7f17a40aabe0 msgr2=0x7f17a408eb70 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:32:03.313 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a reported in up:standby state 2019-12-05T05:32:03.313 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 27 secs before thrashing 2019-12-05T05:32:09.148 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:32:09.152 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:32:30.314 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:32:30.417 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.410+0000 7eff02b3b700 1 Processor -- start 2019-12-05T05:32:30.417 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.410+0000 7eff02b3b700 1 -- start start 2019-12-05T05:32:30.418 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.410+0000 7eff02b3b700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7efefc0a7740 0x7efefc08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:32:30.418 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.410+0000 7eff02b3b700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7efefc09fae0 0x7efefc07eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:32:30.418 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.410+0000 7eff02b3b700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efefc0a0710 con 0x7efefc07e720 2019-12-05T05:32:30.418 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.410+0000 7eff02b3b700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efefc0a0b00 con 0x7efefc0a7740 2019-12-05T05:32:30.418 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.410+0000 7eff02b3b700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efefc08e5b0 con 0x7efefc09fae0 2019-12-05T05:32:30.418 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.410+0000 7eff018d9700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7efefc09fae0 0x7efefc07eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:32:30.418 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.410+0000 7eff018d9700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7efefc09fae0 0x7efefc07eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:35964/0 (socket says 172.21.15.42:35964) 2019-12-05T05:32:30.419 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.410+0000 7eff018d9700 1 -- 172.21.15.42:0/974604045 learned_addr learned my addr 172.21.15.42:0/974604045 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:32:30.419 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.410+0000 7eff010d8700 1 --2- 172.21.15.42:0/974604045 >> v2:172.21.15.40:3300/0 conn(0x7efefc0a7740 0x7efefc08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:32:30.419 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.410+0000 7eff018d9700 1 -- 172.21.15.42:0/974604045 >> v2:172.21.15.40:3300/0 conn(0x7efefc0a7740 msgr2=0x7efefc08df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:32:30.419 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.410+0000 7eff018d9700 1 --2- 172.21.15.42:0/974604045 >> v2:172.21.15.40:3300/0 conn(0x7efefc0a7740 0x7efefc08df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:32:30.419 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.410+0000 7eff018d9700 1 -- 172.21.15.42:0/974604045 >> v1:172.21.15.40:6789/0 conn(0x7efefc07e720 legacy=0x7efefc09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:32:30.419 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.410+0000 7eff018d9700 1 -- 172.21.15.42:0/974604045 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efefc07f0a0 con 0x7efefc09fae0 2019-12-05T05:32:30.419 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.410+0000 7eff018d9700 1 --2- 172.21.15.42:0/974604045 >> v2:172.21.15.42:3300/0 conn(0x7efefc09fae0 0x7efefc07eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7efef40032b0 tx=0x7efef40100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:30.420 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.410+0000 7eff018d9700 1 --2- 172.21.15.42:0/974604045 >> v2:172.21.15.42:3300/0 conn(0x7efefc09fae0 0x7efefc07eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7efef40032b0 tx=0x7efef40100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:30.420 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.410+0000 7eff018d9700 1 --2- 172.21.15.42:0/974604045 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efefc09fae0 0x7efefc07eb50 secure :-1 s=READY pgs=1068 cs=0 l=1 rx=0x7efef40032b0 tx=0x7efef40100b0).ready entity=mon.1 client_cookie=e898cc86ecd7f4bf server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:32:30.420 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.410+0000 7eff018d9700 1 --2- 172.21.15.42:0/974604045 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efefc09fae0 0x7efefc07eb50 secure :-1 s=THROTTLE_DONE pgs=1068 cs=0 l=1 rx=0x7efef40032b0 tx=0x7efef40100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:30.420 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.410+0000 7efef3fff700 1 -- 172.21.15.42:0/974604045 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efef4010d60 con 0x7efefc09fae0 2019-12-05T05:32:30.420 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.410+0000 7eff018d9700 1 --2- 172.21.15.42:0/974604045 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efefc09fae0 0x7efefc07eb50 secure :-1 s=THROTTLE_DONE pgs=1068 cs=0 l=1 rx=0x7efef40032b0 tx=0x7efef40100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:30.420 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.414+0000 7efef3fff700 1 -- 172.21.15.42:0/974604045 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efef4010980 con 0x7efefc09fae0 2019-12-05T05:32:30.420 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.414+0000 7eff018d9700 1 --2- 172.21.15.42:0/974604045 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efefc09fae0 0x7efefc07eb50 secure :-1 s=THROTTLE_DONE pgs=1068 cs=0 l=1 rx=0x7efef40032b0 tx=0x7efef40100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:30.421 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.414+0000 7efef3fff700 1 -- 172.21.15.42:0/974604045 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efef40114a0 con 0x7efefc09fae0 2019-12-05T05:32:30.421 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.414+0000 7eff02b3b700 1 -- 172.21.15.42:0/974604045 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efefc09fae0 msgr2=0x7efefc07eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:32:30.421 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.414+0000 7eff02b3b700 1 --2- 172.21.15.42:0/974604045 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efefc09fae0 0x7efefc07eb50 secure :-1 s=READY pgs=1068 cs=0 l=1 rx=0x7efef40032b0 tx=0x7efef40100b0).stop 2019-12-05T05:32:30.421 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.414+0000 7eff02b3b700 1 -- 172.21.15.42:0/974604045 shutdown_connections 2019-12-05T05:32:30.421 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.414+0000 7eff02b3b700 1 --2- 172.21.15.42:0/974604045 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efefc09fae0 0x7efefc07eb50 unknown :-1 s=CLOSED pgs=1068 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:32:30.421 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.414+0000 7eff02b3b700 1 --2- 172.21.15.42:0/974604045 >> v2:172.21.15.40:3300/0 conn(0x7efefc0a7740 0x7efefc08df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:32:30.421 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.414+0000 7eff02b3b700 1 -- 172.21.15.42:0/974604045 shutdown_connections 2019-12-05T05:32:30.422 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.414+0000 7eff02b3b700 1 -- 172.21.15.42:0/974604045 wait complete. 2019-12-05T05:32:30.422 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.414+0000 7eff02b3b700 1 -- 172.21.15.42:0/974604045 >> 172.21.15.42:0/974604045 conn(0x7efefc0aabe0 msgr2=0x7efefc0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:32:30.425 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.414+0000 7eff02b3b700 1 Processor -- start 2019-12-05T05:32:30.426 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.414+0000 7eff02b3b700 1 -- start start 2019-12-05T05:32:30.426 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.414+0000 7eff02b3b700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efefc11d130 0x7efefc11a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:32:30.426 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.414+0000 7eff02b3b700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7efefc11b8b0 0x7efefc125360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:32:30.426 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.414+0000 7eff02b3b700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7efefc1258b0 0x7efefc118890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:32:30.426 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.414+0000 7eff02b3b700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efefc11bfb0 con 0x7efefc11b8b0 2019-12-05T05:32:30.426 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.414+0000 7eff02b3b700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efefc11c250 con 0x7efefc1258b0 2019-12-05T05:32:30.427 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.414+0000 7eff02b3b700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efefc1250c0 con 0x7efefc11d130 2019-12-05T05:32:30.427 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.418+0000 7eff018d9700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7efefc1258b0 0x7efefc118890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:32:30.427 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.418+0000 7eff018d9700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7efefc1258b0 0x7efefc118890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:35970/0 (socket says 172.21.15.42:35970) 2019-12-05T05:32:30.427 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.418+0000 7eff018d9700 1 -- 172.21.15.42:0/216311372 learned_addr learned my addr 172.21.15.42:0/216311372 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:32:30.427 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.418+0000 7eff010d8700 1 --2- 172.21.15.42:0/216311372 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efefc11d130 0x7efefc11a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:32:30.427 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.418+0000 7eff008d7700 1 --2- 172.21.15.42:0/216311372 >> v2:172.21.15.40:3300/0 conn(0x7efefc11b8b0 0x7efefc125360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:32:30.427 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.418+0000 7eff018d9700 1 -- 172.21.15.42:0/216311372 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efefc11d130 msgr2=0x7efefc11a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:32:30.428 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.418+0000 7eff018d9700 1 --2- 172.21.15.42:0/216311372 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efefc11d130 0x7efefc11a290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:32:30.428 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.418+0000 7eff018d9700 1 -- 172.21.15.42:0/216311372 >> v2:172.21.15.40:3300/0 conn(0x7efefc11b8b0 msgr2=0x7efefc125360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:32:30.428 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.418+0000 7eff018d9700 1 --2- 172.21.15.42:0/216311372 >> v2:172.21.15.40:3300/0 conn(0x7efefc11b8b0 0x7efefc125360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:32:30.428 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.418+0000 7eff018d9700 1 -- 172.21.15.42:0/216311372 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efefc11bce0 con 0x7efefc1258b0 2019-12-05T05:32:30.428 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.418+0000 7eff010d8700 1 --2- 172.21.15.42:0/216311372 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efefc11d130 0x7efefc11a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:32:30.428 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.418+0000 7eff008d7700 1 --2- 172.21.15.42:0/216311372 >> v2:172.21.15.40:3300/0 conn(0x7efefc11b8b0 0x7efefc125360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:32:30.428 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.418+0000 7eff018d9700 1 --2- 172.21.15.42:0/216311372 >> v2:172.21.15.42:3300/0 conn(0x7efefc1258b0 0x7efefc118890 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7efefc11c410 tx=0x7efef4010f80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:30.429 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.418+0000 7eff018d9700 1 --2- 172.21.15.42:0/216311372 >> v2:172.21.15.42:3300/0 conn(0x7efefc1258b0 0x7efefc118890 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7efefc11c410 tx=0x7efef4010f80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:30.429 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.418+0000 7eff018d9700 1 --2- 172.21.15.42:0/216311372 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efefc1258b0 0x7efefc118890 secure :-1 s=READY pgs=1069 cs=0 l=1 rx=0x7efefc11c410 tx=0x7efef4010f80).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:32:30.429 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.418+0000 7eff018d9700 1 --2- 172.21.15.42:0/216311372 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efefc1258b0 0x7efefc118890 secure :-1 s=THROTTLE_DONE pgs=1069 cs=0 l=1 rx=0x7efefc11c410 tx=0x7efef4010f80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:30.429 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.418+0000 7efef1ffb700 1 -- 172.21.15.42:0/216311372 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efef4011c40 con 0x7efefc1258b0 2019-12-05T05:32:30.429 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.418+0000 7eff018d9700 1 --2- 172.21.15.42:0/216311372 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efefc1258b0 0x7efefc118890 secure :-1 s=THROTTLE_DONE pgs=1069 cs=0 l=1 rx=0x7efefc11c410 tx=0x7efef4010f80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:30.429 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.418+0000 7eff018d9700 1 --2- 172.21.15.42:0/216311372 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efefc1258b0 0x7efefc118890 secure :-1 s=THROTTLE_DONE pgs=1069 cs=0 l=1 rx=0x7efefc11c410 tx=0x7efef4010f80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:30.429 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.418+0000 7efef1ffb700 1 -- 172.21.15.42:0/216311372 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efef40110c0 con 0x7efefc1258b0 2019-12-05T05:32:30.430 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.418+0000 7efef1ffb700 1 -- 172.21.15.42:0/216311372 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efef4005ca0 con 0x7efefc1258b0 2019-12-05T05:32:30.430 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.418+0000 7eff02b3b700 1 -- 172.21.15.42:0/216311372 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efefc125ce0 con 0x7efefc1258b0 2019-12-05T05:32:30.430 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.418+0000 7eff02b3b700 1 -- 172.21.15.42:0/216311372 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efefc119000 con 0x7efefc1258b0 2019-12-05T05:32:30.430 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.418+0000 7eff018d9700 1 --2- 172.21.15.42:0/216311372 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efefc1258b0 0x7efefc118890 secure :-1 s=THROTTLE_DONE pgs=1069 cs=0 l=1 rx=0x7efefc11c410 tx=0x7efef4010f80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:30.430 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.418+0000 7efef1ffb700 1 -- 172.21.15.42:0/216311372 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7efef400f280 con 0x7efefc1258b0 2019-12-05T05:32:30.430 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.418+0000 7eff018d9700 1 --2- 172.21.15.42:0/216311372 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efefc1258b0 0x7efefc118890 secure :-1 s=THROTTLE_DONE pgs=1069 cs=0 l=1 rx=0x7efefc11c410 tx=0x7efef4010f80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:30.430 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.422+0000 7efef1ffb700 1 --2- 172.21.15.42:0/216311372 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7efedc03ada0 0x7efedc03d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:32:30.431 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.422+0000 7efef1ffb700 1 -- 172.21.15.42:0/216311372 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7751+0+0 (secure 0 0 0) 0x7efef408f770 con 0x7efefc1258b0 2019-12-05T05:32:30.431 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.422+0000 7eff010d8700 1 --2- 172.21.15.42:0/216311372 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7efedc03ada0 0x7efedc03d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:32:30.431 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.422+0000 7eff010d8700 1 --2- 172.21.15.42:0/216311372 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7efedc03ada0 0x7efedc03d2b0 crc :-1 s=READY pgs=457 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:32:30.431 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.426+0000 7eff02b3b700 1 -- 172.21.15.42:0/216311372 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efefc05d110 con 0x7efefc1258b0 2019-12-05T05:32:30.434 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.430+0000 7eff018d9700 1 --2- 172.21.15.42:0/216311372 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efefc1258b0 0x7efefc118890 secure :-1 s=THROTTLE_DONE pgs=1069 cs=0 l=1 rx=0x7efefc11c410 tx=0x7efef4010f80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:30.435 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.430+0000 7efef1ffb700 1 -- 172.21.15.42:0/216311372 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7efef40767f0 con 0x7efefc1258b0 2019-12-05T05:32:30.622 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.606+0000 7eff02b3b700 1 -- 172.21.15.42:0/216311372 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efefc09fd00 con 0x7efefc1258b0 2019-12-05T05:32:30.622 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.606+0000 7eff018d9700 1 --2- 172.21.15.42:0/216311372 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efefc1258b0 0x7efefc118890 secure :-1 s=THROTTLE_DONE pgs=1069 cs=0 l=1 rx=0x7efefc11c410 tx=0x7efef4010f80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:30.623 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.606+0000 7efef1ffb700 1 -- 172.21.15.42:0/216311372 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 133 v133) v1 ==== 95+0+2112 (secure 0 0 0) 0x7efef4011e40 con 0x7efefc1258b0 2019-12-05T05:32:30.623 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 133 2019-12-05T05:32:30.623 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.606+0000 7eff02b3b700 1 -- 172.21.15.42:0/216311372 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7efedc03ada0 msgr2=0x7efedc03d2b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:32:30.623 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.606+0000 7eff02b3b700 1 --2- 172.21.15.42:0/216311372 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7efedc03ada0 0x7efedc03d2b0 crc :-1 s=READY pgs=457 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:32:30.623 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.606+0000 7eff02b3b700 1 -- 172.21.15.42:0/216311372 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efefc1258b0 msgr2=0x7efefc118890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:32:30.623 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.606+0000 7eff02b3b700 1 --2- 172.21.15.42:0/216311372 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efefc1258b0 0x7efefc118890 secure :-1 s=READY pgs=1069 cs=0 l=1 rx=0x7efefc11c410 tx=0x7efef4010f80).stop 2019-12-05T05:32:30.623 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.606+0000 7eff02b3b700 1 -- 172.21.15.42:0/216311372 shutdown_connections 2019-12-05T05:32:30.624 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.606+0000 7eff02b3b700 1 --2- 172.21.15.42:0/216311372 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7efedc03ada0 0x7efedc03d2b0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:32:30.624 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.610+0000 7eff02b3b700 1 --2- 172.21.15.42:0/216311372 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efefc1258b0 0x7efefc118890 unknown :-1 s=CLOSED pgs=1069 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:32:30.624 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.610+0000 7eff02b3b700 1 --2- 172.21.15.42:0/216311372 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efefc11d130 0x7efefc11a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:32:30.624 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.610+0000 7eff02b3b700 1 --2- 172.21.15.42:0/216311372 >> v2:172.21.15.40:3300/0 conn(0x7efefc11b8b0 0x7efefc125360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:32:30.624 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.610+0000 7eff02b3b700 1 -- 172.21.15.42:0/216311372 shutdown_connections 2019-12-05T05:32:30.624 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.610+0000 7eff02b3b700 1 -- 172.21.15.42:0/216311372 wait complete. 2019-12-05T05:32:30.625 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.610+0000 7eff02b3b700 1 -- 172.21.15.42:0/216311372 >> 172.21.15.42:0/216311372 conn(0x7efefc0a7f90 msgr2=0x7efefc11d8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:32:30.625 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:32:30.625 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":133,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6997,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/1601334502","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1601334502},{"type":"v1","addr":"172.21.15.42:6835","nonce":1601334502}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":133}],"filesystems":[{"mdsmap":{"epoch":132,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:31:53.749802+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":69,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6984},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6984":{"gid":6984,"name":"b","rank":0,"incarnation":129,"state":"up:active","state_seq":15,"addr":"172.21.15.40:6833/3543033403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3543033403},{"type":"v1","addr":"172.21.15.40:6833","nonce":3543033403}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:32:30.626 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.b (rank=0) 2019-12-05T05:32:30.627 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2019-12-05T05:32:30.627 INFO:teuthology.orchestra.run:waiting for 300 2019-12-05T05:32:30.640 INFO:tasks.ceph.mds.b:Stopped 2019-12-05T05:32:30.641 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:32:30.737 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.730+0000 7f08f1ad9700 1 Processor -- start 2019-12-05T05:32:30.737 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.730+0000 7f08f1ad9700 1 -- start start 2019-12-05T05:32:30.737 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.730+0000 7f08f1ad9700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f08ec08df80 0x7f08ec09f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:32:30.737 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.730+0000 7f08f1ad9700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f08ec09f830 0x7f08ec14efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:32:30.737 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.730+0000 7f08f1ad9700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08ec09fc60 con 0x7f08ec0a7740 2019-12-05T05:32:30.738 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.730+0000 7f08f1ad9700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08ec0a0b00 con 0x7f08ec09f830 2019-12-05T05:32:30.738 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.730+0000 7f08f1ad9700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08ec08e5b0 con 0x7f08ec08df80 2019-12-05T05:32:30.738 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.730+0000 7f08f0877700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f08ec09f830 0x7f08ec14efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:32:30.738 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.730+0000 7f08f0877700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f08ec09f830 0x7f08ec14efe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:35980/0 (socket says 172.21.15.42:35980) 2019-12-05T05:32:30.738 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.730+0000 7f08f0877700 1 -- 172.21.15.42:0/1862804997 learned_addr learned my addr 172.21.15.42:0/1862804997 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:32:30.738 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.730+0000 7f08eb7fe700 1 --2- 172.21.15.42:0/1862804997 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f08ec08df80 0x7f08ec09f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:32:30.738 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.730+0000 7f08f0877700 1 -- 172.21.15.42:0/1862804997 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f08ec08df80 msgr2=0x7f08ec09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:32:30.739 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.730+0000 7f08f0877700 1 --2- 172.21.15.42:0/1862804997 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f08ec08df80 0x7f08ec09f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:32:30.739 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.730+0000 7f08f0877700 1 -- 172.21.15.42:0/1862804997 >> v1:172.21.15.40:6789/0 conn(0x7f08ec0a7740 legacy=0x7f08ec07e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:32:30.739 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.730+0000 7f08f0877700 1 -- 172.21.15.42:0/1862804997 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08ec0a0710 con 0x7f08ec09f830 2019-12-05T05:32:30.739 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.730+0000 7f08eb7fe700 1 --2- 172.21.15.42:0/1862804997 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f08ec08df80 0x7f08ec09f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T05:32:30.739 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.730+0000 7f08f0877700 1 --2- 172.21.15.42:0/1862804997 >> v2:172.21.15.42:3300/0 conn(0x7f08ec09f830 0x7f08ec14efe0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f08e4008790 tx=0x7f08e400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:30.739 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.730+0000 7f08f0877700 1 --2- 172.21.15.42:0/1862804997 >> v2:172.21.15.42:3300/0 conn(0x7f08ec09f830 0x7f08ec14efe0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f08e4008790 tx=0x7f08e400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:30.739 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.730+0000 7f08f0877700 1 --2- 172.21.15.42:0/1862804997 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f08ec09f830 0x7f08ec14efe0 secure :-1 s=READY pgs=1070 cs=0 l=1 rx=0x7f08e4008790 tx=0x7f08e400fd30).ready entity=mon.1 client_cookie=6c3d7d90c108f48a server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:32:30.740 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.730+0000 7f08f0877700 1 --2- 172.21.15.42:0/1862804997 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f08ec09f830 0x7f08ec14efe0 secure :-1 s=THROTTLE_DONE pgs=1070 cs=0 l=1 rx=0x7f08e4008790 tx=0x7f08e400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:30.740 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.730+0000 7f08eaffd700 1 -- 172.21.15.42:0/1862804997 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f08e4008c90 con 0x7f08ec09f830 2019-12-05T05:32:30.740 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.730+0000 7f08f0877700 1 --2- 172.21.15.42:0/1862804997 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f08ec09f830 0x7f08ec14efe0 secure :-1 s=THROTTLE_DONE pgs=1070 cs=0 l=1 rx=0x7f08e4008790 tx=0x7f08e400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:30.740 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.730+0000 7f08f0877700 1 --2- 172.21.15.42:0/1862804997 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f08ec09f830 0x7f08ec14efe0 secure :-1 s=THROTTLE_DONE pgs=1070 cs=0 l=1 rx=0x7f08e4008790 tx=0x7f08e400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:30.740 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.730+0000 7f08eaffd700 1 -- 172.21.15.42:0/1862804997 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08e40088b0 con 0x7f08ec09f830 2019-12-05T05:32:30.740 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.730+0000 7f08eaffd700 1 -- 172.21.15.42:0/1862804997 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f08e4011b10 con 0x7f08ec09f830 2019-12-05T05:32:30.740 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.730+0000 7f08f1ad9700 1 -- 172.21.15.42:0/1862804997 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f08ec09f830 msgr2=0x7f08ec14efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:32:30.741 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.730+0000 7f08f1ad9700 1 --2- 172.21.15.42:0/1862804997 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f08ec09f830 0x7f08ec14efe0 secure :-1 s=READY pgs=1070 cs=0 l=1 rx=0x7f08e4008790 tx=0x7f08e400fd30).stop 2019-12-05T05:32:30.741 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.734+0000 7f08f1ad9700 1 -- 172.21.15.42:0/1862804997 shutdown_connections 2019-12-05T05:32:30.741 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.734+0000 7f08f1ad9700 1 --2- 172.21.15.42:0/1862804997 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f08ec09f830 0x7f08ec14efe0 unknown :-1 s=CLOSED pgs=1070 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:32:30.741 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.734+0000 7f08f1ad9700 1 --2- 172.21.15.42:0/1862804997 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f08ec08df80 0x7f08ec09f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:32:30.741 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.734+0000 7f08f1ad9700 1 -- 172.21.15.42:0/1862804997 shutdown_connections 2019-12-05T05:32:30.741 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.734+0000 7f08f1ad9700 1 -- 172.21.15.42:0/1862804997 wait complete. 2019-12-05T05:32:30.741 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.734+0000 7f08f1ad9700 1 -- 172.21.15.42:0/1862804997 >> 172.21.15.42:0/1862804997 conn(0x7f08ec0aabe0 msgr2=0x7f08ec0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:32:30.745 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.734+0000 7f08f1ad9700 1 Processor -- start 2019-12-05T05:32:30.745 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.734+0000 7f08f1ad9700 1 -- start start 2019-12-05T05:32:30.746 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.734+0000 7f08f1ad9700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f08ec1178f0 0x7f08ec1247c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:32:30.746 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.734+0000 7f08f1ad9700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08ec113a30 con 0x7f08ec124d10 2019-12-05T05:32:30.746 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.734+0000 7f08f1ad9700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08ec0a40b0 con 0x7f08ec0a5b40 2019-12-05T05:32:30.746 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.734+0000 7f08f1ad9700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08ec125260 con 0x7f08ec1178f0 2019-12-05T05:32:30.746 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.734+0000 7f08f0877700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f08ec0a5b40 0x7f08ec0a5f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:36022/0 (socket says 172.21.15.42:36022) 2019-12-05T05:32:30.746 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.734+0000 7f08f0877700 1 -- 172.21.15.42:0/2982102498 learned_addr learned my addr 172.21.15.42:0/2982102498 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:32:30.746 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.738+0000 7f08ebfff700 1 --2- 172.21.15.42:0/2982102498 >> v2:172.21.15.40:3300/0 conn(0x7f08ec1178f0 0x7f08ec1247c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:32:30.747 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.738+0000 7f08e8ff9700 1 -- 172.21.15.42:0/2982102498 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 298667696 0 0) 0x7f08ec0a40b0 con 0x7f08ec0a5b40 2019-12-05T05:32:30.747 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.738+0000 7f08e8ff9700 1 -- 172.21.15.42:0/2982102498 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08d00030b0 con 0x7f08ec0a5b40 2019-12-05T05:32:30.747 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.738+0000 7f08e8ff9700 1 -- 172.21.15.42:0/2982102498 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2202524853 0 0) 0x7f08ec113a30 con 0x7f08ec124d10 2019-12-05T05:32:30.747 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.738+0000 7f08e8ff9700 1 -- 172.21.15.42:0/2982102498 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08ec0a40b0 con 0x7f08ec124d10 2019-12-05T05:32:30.747 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.738+0000 7f08e8ff9700 1 -- 172.21.15.42:0/2982102498 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 3748728760 0 0) 0x7f08d00030b0 con 0x7f08ec0a5b40 2019-12-05T05:32:30.747 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.738+0000 7f08e8ff9700 1 -- 172.21.15.42:0/2982102498 --> v1:172.21.15.42:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f08ec113a30 con 0x7f08ec0a5b40 2019-12-05T05:32:30.747 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.738+0000 7f08e8ff9700 1 -- 172.21.15.42:0/2982102498 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7f08e4011de0 con 0x7f08ec0a5b40 2019-12-05T05:32:30.747 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.738+0000 7f08e8ff9700 1 -- 172.21.15.42:0/2982102498 >> v2:172.21.15.40:3300/0 conn(0x7f08ec1178f0 msgr2=0x7f08ec1247c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:32:30.748 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.738+0000 7f08e8ff9700 1 --2- 172.21.15.42:0/2982102498 >> v2:172.21.15.40:3300/0 conn(0x7f08ec1178f0 0x7f08ec1247c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:32:30.748 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.738+0000 7f08e8ff9700 1 -- 172.21.15.42:0/2982102498 >> v1:172.21.15.42:6789/0 conn(0x7f08ec0a5b40 legacy=0x7f08ec0a5f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:32:30.748 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.738+0000 7f08e8ff9700 1 -- 172.21.15.42:0/2982102498 >> v1:172.21.15.40:6789/0 conn(0x7f08ec124d10 legacy=0x7f08ec0a38b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:32:30.748 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.738+0000 7f08e8ff9700 1 --2- 172.21.15.42:0/2982102498 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f08d0003ad0 0x7f08d0004420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:32:30.748 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.738+0000 7f08e8ff9700 1 --2- 172.21.15.42:0/2982102498 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f08d0004970 0x7f08d0004da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:32:30.748 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.738+0000 7f08e8ff9700 1 --2- 172.21.15.42:0/2982102498 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f08d000da70 0x7f08d000feb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:32:30.748 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.738+0000 7f08e8ff9700 1 -- 172.21.15.42:0/2982102498 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08ec125260 con 0x7f08d0003ad0 2019-12-05T05:32:30.749 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.738+0000 7f08e8ff9700 1 -- 172.21.15.42:0/2982102498 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08d0010400 con 0x7f08d0004970 2019-12-05T05:32:30.749 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.738+0000 7f08e8ff9700 1 -- 172.21.15.42:0/2982102498 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08d00105f0 con 0x7f08d000da70 2019-12-05T05:32:30.749 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.738+0000 7f08ebfff700 1 --2- 172.21.15.42:0/2982102498 >> v2:172.21.15.40:3300/0 conn(0x7f08ec1178f0 0x7f08ec1247c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:32:30.749 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.738+0000 7f08eb7fe700 1 --2- 172.21.15.42:0/2982102498 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f08d0004970 0x7f08d0004da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:32:30.749 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.738+0000 7f08f0877700 1 --2- 172.21.15.42:0/2982102498 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f08d000da70 0x7f08d000feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:32:30.749 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.738+0000 7f08ebfff700 1 --2- 172.21.15.42:0/2982102498 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f08d0003ad0 0x7f08d0004420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:32:30.749 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.738+0000 7f08eb7fe700 1 -- 172.21.15.42:0/2982102498 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f08d000da70 msgr2=0x7f08d000feb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:32:30.750 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.738+0000 7f08eb7fe700 1 --2- 172.21.15.42:0/2982102498 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f08d000da70 0x7f08d000feb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:32:30.750 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.738+0000 7f08eb7fe700 1 -- 172.21.15.42:0/2982102498 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f08d0003ad0 msgr2=0x7f08d0004420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:32:30.750 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.738+0000 7f08eb7fe700 1 --2- 172.21.15.42:0/2982102498 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f08d0003ad0 0x7f08d0004420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:32:30.750 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.738+0000 7f08eb7fe700 1 -- 172.21.15.42:0/2982102498 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08ec12ec40 con 0x7f08d0004970 2019-12-05T05:32:30.750 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.738+0000 7f08f0877700 1 -- 172.21.15.42:0/2982102498 reap_dead start 2019-12-05T05:32:30.750 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.738+0000 7f08f1ad9700 1 -- 172.21.15.42:0/2982102498 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08ec124d10 con 0x7f08d0004970 2019-12-05T05:32:30.750 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.738+0000 7f08eb7fe700 1 --2- 172.21.15.42:0/2982102498 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f08d0004970 0x7f08d0004da0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f08dc007b20 tx=0x7f08dc010f40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:30.751 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.738+0000 7f08f1ad9700 1 -- 172.21.15.42:0/2982102498 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08ec1178f0 con 0x7f08d0004970 2019-12-05T05:32:30.751 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.738+0000 7f08eb7fe700 1 --2- 172.21.15.42:0/2982102498 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f08d0004970 0x7f08d0004da0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f08dc007b20 tx=0x7f08dc010f40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:30.751 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.738+0000 7f08eb7fe700 1 --2- 172.21.15.42:0/2982102498 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f08d0004970 0x7f08d0004da0 secure :-1 s=READY pgs=1072 cs=0 l=1 rx=0x7f08dc007b20 tx=0x7f08dc010f40).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:32:30.751 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.738+0000 7f08eb7fe700 1 --2- 172.21.15.42:0/2982102498 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f08d0004970 0x7f08d0004da0 secure :-1 s=THROTTLE_DONE pgs=1072 cs=0 l=1 rx=0x7f08dc007b20 tx=0x7f08dc010f40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:30.751 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.738+0000 7f08e8ff9700 1 -- 172.21.15.42:0/2982102498 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f08dc0058c0 con 0x7f08d0004970 2019-12-05T05:32:30.751 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.738+0000 7f08eb7fe700 1 --2- 172.21.15.42:0/2982102498 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f08d0004970 0x7f08d0004da0 secure :-1 s=THROTTLE_DONE pgs=1072 cs=0 l=1 rx=0x7f08dc007b20 tx=0x7f08dc010f40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:30.751 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.738+0000 7f08e8ff9700 1 -- 172.21.15.42:0/2982102498 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08dc010160 con 0x7f08d0004970 2019-12-05T05:32:30.752 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.738+0000 7f08eb7fe700 1 --2- 172.21.15.42:0/2982102498 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f08d0004970 0x7f08d0004da0 secure :-1 s=THROTTLE_DONE pgs=1072 cs=0 l=1 rx=0x7f08dc007b20 tx=0x7f08dc010f40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:30.752 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.738+0000 7f08e8ff9700 1 -- 172.21.15.42:0/2982102498 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f08dc00e870 con 0x7f08d0004970 2019-12-05T05:32:30.752 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.742+0000 7f08eb7fe700 1 --2- 172.21.15.42:0/2982102498 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f08d0004970 0x7f08d0004da0 secure :-1 s=THROTTLE_DONE pgs=1072 cs=0 l=1 rx=0x7f08dc007b20 tx=0x7f08dc010f40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:30.752 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.742+0000 7f08e8ff9700 1 -- 172.21.15.42:0/2982102498 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f08dc00ea70 con 0x7f08d0004970 2019-12-05T05:32:30.752 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.742+0000 7f08eb7fe700 1 --2- 172.21.15.42:0/2982102498 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f08d0004970 0x7f08d0004da0 secure :-1 s=THROTTLE_DONE pgs=1072 cs=0 l=1 rx=0x7f08dc007b20 tx=0x7f08dc010f40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:30.752 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.742+0000 7f08e8ff9700 1 --2- 172.21.15.42:0/2982102498 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f08d0043170 0x7f08d0045680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:32:30.753 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.742+0000 7f08e8ff9700 1 -- 172.21.15.42:0/2982102498 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7751+0+0 (secure 0 0 0) 0x7f08dc04ae00 con 0x7f08d0004970 2019-12-05T05:32:30.753 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.742+0000 7f08ebfff700 1 --2- 172.21.15.42:0/2982102498 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f08d0043170 0x7f08d0045680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:32:30.753 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.742+0000 7f08ebfff700 1 --2- 172.21.15.42:0/2982102498 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f08d0043170 0x7f08d0045680 crc :-1 s=READY pgs=458 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:32:30.753 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.746+0000 7f08f1ad9700 1 -- 172.21.15.42:0/2982102498 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08c4003200 con 0x7f08d0004970 2019-12-05T05:32:30.756 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.754+0000 7f08eb7fe700 1 --2- 172.21.15.42:0/2982102498 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f08d0004970 0x7f08d0004da0 secure :-1 s=THROTTLE_DONE pgs=1072 cs=0 l=1 rx=0x7f08dc007b20 tx=0x7f08dc010f40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:30.756 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.754+0000 7f08e8ff9700 1 -- 172.21.15.42:0/2982102498 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f08dc031e60 con 0x7f08d0004970 2019-12-05T05:32:30.929 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.926+0000 7f08f1ad9700 1 -- 172.21.15.42:0/2982102498 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f08c4003630 con 0x7f08d0004970 2019-12-05T05:32:30.930 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.926+0000 7f08eb7fe700 1 --2- 172.21.15.42:0/2982102498 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f08d0004970 0x7f08d0004da0 secure :-1 s=THROTTLE_DONE pgs=1072 cs=0 l=1 rx=0x7f08dc007b20 tx=0x7f08dc010f40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:30.930 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.926+0000 7f08e8ff9700 1 -- 172.21.15.42:0/2982102498 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 133 v133) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f08dc0320c0 con 0x7f08d0004970 2019-12-05T05:32:30.930 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 133 2019-12-05T05:32:30.930 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:32:30.930 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":133,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6997,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/1601334502","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1601334502},{"type":"v1","addr":"172.21.15.42:6835","nonce":1601334502}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":133}],"filesystems":[{"mdsmap":{"epoch":132,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:31:53.749802+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":69,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6984},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6984":{"gid":6984,"name":"b","rank":0,"incarnation":129,"state":"up:active","state_seq":15,"addr":"172.21.15.40:6833/3543033403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3543033403},{"type":"v1","addr":"172.21.15.40:6833","nonce":3543033403}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:32:30.938 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.930+0000 7f08f1ad9700 1 -- 172.21.15.42:0/2982102498 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f08d0043170 msgr2=0x7f08d0045680 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:32:30.938 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.930+0000 7f08f1ad9700 1 --2- 172.21.15.42:0/2982102498 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f08d0043170 0x7f08d0045680 crc :-1 s=READY pgs=458 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:32:30.938 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.930+0000 7f08f1ad9700 1 -- 172.21.15.42:0/2982102498 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f08d0004970 msgr2=0x7f08d0004da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:32:30.938 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.930+0000 7f08f1ad9700 1 --2- 172.21.15.42:0/2982102498 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f08d0004970 0x7f08d0004da0 secure :-1 s=READY pgs=1072 cs=0 l=1 rx=0x7f08dc007b20 tx=0x7f08dc010f40).stop 2019-12-05T05:32:30.938 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.930+0000 7f08f1ad9700 1 -- 172.21.15.42:0/2982102498 shutdown_connections 2019-12-05T05:32:30.938 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.930+0000 7f08f1ad9700 1 --2- 172.21.15.42:0/2982102498 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f08d0043170 0x7f08d0045680 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:32:30.939 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.930+0000 7f08f1ad9700 1 --2- 172.21.15.42:0/2982102498 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f08d0004970 0x7f08d0004da0 unknown :-1 s=CLOSED pgs=1072 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:32:30.939 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.930+0000 7f08f1ad9700 1 -- 172.21.15.42:0/2982102498 shutdown_connections 2019-12-05T05:32:30.939 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.930+0000 7f08f1ad9700 1 -- 172.21.15.42:0/2982102498 wait complete. 2019-12-05T05:32:30.939 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:30.930+0000 7f08f1ad9700 1 -- 172.21.15.42:0/2982102498 >> 172.21.15.42:0/2982102498 conn(0x7f08ec09f2c0 msgr2=0x7f08ec141360 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:32:30.943 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T05:32:32.944 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:32:33.053 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:33.042+0000 7f2a787d3700 1 Processor -- start 2019-12-05T05:32:33.053 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:33.042+0000 7f2a787d3700 1 -- start start 2019-12-05T05:32:33.053 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:33.042+0000 7f2a787d3700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2a700a7740 0x7f2a7008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:32:33.053 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:33.042+0000 7f2a787d3700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2a7009fae0 0x7f2a7007eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:32:33.053 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:33.042+0000 7f2a787d3700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a700a0710 con 0x7f2a7007e720 2019-12-05T05:32:33.054 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:33.042+0000 7f2a787d3700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a700a0b00 con 0x7f2a700a7740 2019-12-05T05:32:33.054 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:33.042+0000 7f2a787d3700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a7008e5b0 con 0x7f2a7009fae0 2019-12-05T05:32:33.054 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:33.042+0000 7f2a77571700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2a7009fae0 0x7f2a7007eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:32:33.054 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:33.042+0000 7f2a76d70700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2a700a7740 0x7f2a7008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:32:33.054 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:33.042+0000 7f2a7656f700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f2a7007e720 0x7f2a7009f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:45482/0 (socket says 172.21.15.42:45482) 2019-12-05T05:32:33.054 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:33.042+0000 7f2a7656f700 1 -- 172.21.15.42:0/4258883716 learned_addr learned my addr 172.21.15.42:0/4258883716 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:32:33.055 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:33.042+0000 7f2a77571700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2a7009fae0 0x7f2a7007eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:36472/0 (socket says 172.21.15.42:36472) 2019-12-05T05:32:33.055 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:33.046+0000 7f2a76d70700 1 -- 172.21.15.42:0/4258883716 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2a7009fae0 msgr2=0x7f2a7007eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:32:33.055 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:33.046+0000 7f2a75d6e700 1 -- 172.21.15.42:0/4258883716 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3926145910 0 0) 0x7f2a700a0710 con 0x7f2a7007e720 2019-12-05T05:32:33.055 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:33.046+0000 7f2a76d70700 1 --2- 172.21.15.42:0/4258883716 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2a7009fae0 0x7f2a7007eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:32:33.055 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:33.046+0000 7f2a76d70700 1 -- 172.21.15.42:0/4258883716 >> v1:172.21.15.40:6789/0 conn(0x7f2a7007e720 legacy=0x7f2a7009f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:32:33.055 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:33.046+0000 7f2a76d70700 1 -- 172.21.15.42:0/4258883716 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a7007f0a0 con 0x7f2a700a7740 2019-12-05T05:32:33.056 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:33.046+0000 7f2a77571700 1 --2- 172.21.15.42:0/4258883716 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2a7009fae0 0x7f2a7007eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:32:33.056 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:33.046+0000 7f2a76d70700 1 --2- 172.21.15.42:0/4258883716 >> v2:172.21.15.40:3300/0 conn(0x7f2a700a7740 0x7f2a7008df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f2a6c008790 tx=0x7f2a6c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:33.056 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:33.046+0000 7f2a76d70700 1 --2- 172.21.15.42:0/4258883716 >> v2:172.21.15.40:3300/0 conn(0x7f2a700a7740 0x7f2a7008df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f2a6c008790 tx=0x7f2a6c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:33.057 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:33.046+0000 7f2a76d70700 1 --2- 172.21.15.42:0/4258883716 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a700a7740 0x7f2a7008df80 secure :-1 s=READY pgs=836 cs=0 l=1 rx=0x7f2a6c008790 tx=0x7f2a6c00fc70).ready entity=mon.0 client_cookie=147a9545a5d20d3f server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:32:33.057 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:33.046+0000 7f2a76d70700 1 --2- 172.21.15.42:0/4258883716 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a700a7740 0x7f2a7008df80 secure :-1 s=THROTTLE_DONE pgs=836 cs=0 l=1 rx=0x7f2a6c008790 tx=0x7f2a6c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:33.057 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:33.046+0000 7f2a76d70700 1 --2- 172.21.15.42:0/4258883716 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a700a7740 0x7f2a7008df80 secure :-1 s=THROTTLE_DONE pgs=836 cs=0 l=1 rx=0x7f2a6c008790 tx=0x7f2a6c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:33.057 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:33.046+0000 7f2a75d6e700 1 -- 172.21.15.42:0/4258883716 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2a6c008bb0 con 0x7f2a700a7740 2019-12-05T05:32:33.057 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:33.046+0000 7f2a76d70700 1 --2- 172.21.15.42:0/4258883716 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a700a7740 0x7f2a7008df80 secure :-1 s=THROTTLE_DONE pgs=836 cs=0 l=1 rx=0x7f2a6c008790 tx=0x7f2a6c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:33.058 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:33.046+0000 7f2a75d6e700 1 -- 172.21.15.42:0/4258883716 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a6c0087d0 con 0x7f2a700a7740 2019-12-05T05:32:33.058 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:33.046+0000 7f2a75d6e700 1 -- 172.21.15.42:0/4258883716 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2a6c011860 con 0x7f2a700a7740 2019-12-05T05:32:33.058 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:33.046+0000 7f2a787d3700 1 -- 172.21.15.42:0/4258883716 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a700a7740 msgr2=0x7f2a7008df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:32:33.058 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:33.046+0000 7f2a787d3700 1 --2- 172.21.15.42:0/4258883716 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a700a7740 0x7f2a7008df80 secure :-1 s=READY pgs=836 cs=0 l=1 rx=0x7f2a6c008790 tx=0x7f2a6c00fc70).stop 2019-12-05T05:32:33.058 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:33.046+0000 7f2a787d3700 1 -- 172.21.15.42:0/4258883716 shutdown_connections 2019-12-05T05:32:33.058 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:33.046+0000 7f2a787d3700 1 --2- 172.21.15.42:0/4258883716 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2a7009fae0 0x7f2a7007eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:32:33.059 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:33.046+0000 7f2a787d3700 1 --2- 172.21.15.42:0/4258883716 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a700a7740 0x7f2a7008df80 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:32:33.059 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:33.046+0000 7f2a787d3700 1 -- 172.21.15.42:0/4258883716 shutdown_connections 2019-12-05T05:32:33.059 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:33.046+0000 7f2a787d3700 1 -- 172.21.15.42:0/4258883716 wait complete. 2019-12-05T05:32:33.059 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:33.046+0000 7f2a787d3700 1 -- 172.21.15.42:0/4258883716 >> 172.21.15.42:0/4258883716 conn(0x7f2a700aabe0 msgr2=0x7f2a700a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:32:33.063 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:33.050+0000 7f2a787d3700 1 Processor -- start 2019-12-05T05:32:33.063 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:33.050+0000 7f2a787d3700 1 -- start start 2019-12-05T05:32:33.063 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:33.054+0000 7f2a787d3700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2a701210f0 0x7f2a7011ff70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:32:33.063 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:33.054+0000 7f2a787d3700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a7011da30 con 0x7f2a701204c0 2019-12-05T05:32:33.063 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:33.054+0000 7f2a787d3700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a701301f0 con 0x7f2a7011b950 2019-12-05T05:32:33.063 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:33.054+0000 7f2a787d3700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a70121670 con 0x7f2a701210f0 2019-12-05T05:32:33.064 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:33.054+0000 7f2a7656f700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2a701210f0 0x7f2a7011ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:32:33.064 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:33.054+0000 7f2a76d70700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f2a7011b950 0x7f2a7007da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:36046/0 (socket says 172.21.15.42:36046) 2019-12-05T05:32:33.064 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:33.054+0000 7f2a76d70700 1 -- 172.21.15.42:0/2810449813 learned_addr learned my addr 172.21.15.42:0/2810449813 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:32:33.064 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:33.054+0000 7f2a7656f700 1 -- 172.21.15.42:0/2810449813 >> v1:172.21.15.42:6789/0 conn(0x7f2a7011b950 legacy=0x7f2a7007da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:32:33.064 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:33.054+0000 7f2a677fe700 1 -- 172.21.15.42:0/2810449813 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3580116956 0 0) 0x7f2a7011da30 con 0x7f2a701204c0 2019-12-05T05:32:33.064 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:33.054+0000 7f2a7656f700 1 -- 172.21.15.42:0/2810449813 >> v1:172.21.15.40:6789/0 conn(0x7f2a701204c0 legacy=0x7f2a70137640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:32:33.065 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:33.054+0000 7f2a7656f700 1 -- 172.21.15.42:0/2810449813 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a7011e2c0 con 0x7f2a701210f0 2019-12-05T05:32:33.065 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:33.054+0000 7f2a7656f700 1 --2- 172.21.15.42:0/2810449813 >> v2:172.21.15.42:3300/0 conn(0x7f2a701210f0 0x7f2a7011ff70 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f2a6c008370 tx=0x7f2a6c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:33.065 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:33.054+0000 7f2a7656f700 1 --2- 172.21.15.42:0/2810449813 >> v2:172.21.15.42:3300/0 conn(0x7f2a701210f0 0x7f2a7011ff70 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f2a6c008370 tx=0x7f2a6c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:33.065 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:33.054+0000 7f2a7656f700 1 --2- 172.21.15.42:0/2810449813 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2a701210f0 0x7f2a7011ff70 secure :-1 s=READY pgs=1074 cs=0 l=1 rx=0x7f2a6c008370 tx=0x7f2a6c00fc70).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:32:33.065 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:33.054+0000 7f2a7656f700 1 --2- 172.21.15.42:0/2810449813 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2a701210f0 0x7f2a7011ff70 secure :-1 s=THROTTLE_DONE pgs=1074 cs=0 l=1 rx=0x7f2a6c008370 tx=0x7f2a6c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:33.065 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:33.054+0000 7f2a677fe700 1 -- 172.21.15.42:0/2810449813 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2a60011540 con 0x7f2a701210f0 2019-12-05T05:32:33.066 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:33.054+0000 7f2a7656f700 1 --2- 172.21.15.42:0/2810449813 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2a701210f0 0x7f2a7011ff70 secure :-1 s=THROTTLE_DONE pgs=1074 cs=0 l=1 rx=0x7f2a6c008370 tx=0x7f2a6c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:33.066 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:33.054+0000 7f2a7656f700 1 --2- 172.21.15.42:0/2810449813 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2a701210f0 0x7f2a7011ff70 secure :-1 s=THROTTLE_DONE pgs=1074 cs=0 l=1 rx=0x7f2a6c008370 tx=0x7f2a6c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:33.066 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:33.054+0000 7f2a677fe700 1 -- 172.21.15.42:0/2810449813 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a60011160 con 0x7f2a701210f0 2019-12-05T05:32:33.066 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:33.054+0000 7f2a787d3700 1 -- 172.21.15.42:0/2810449813 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a70138e50 con 0x7f2a701210f0 2019-12-05T05:32:33.066 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:33.054+0000 7f2a677fe700 1 -- 172.21.15.42:0/2810449813 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2a60011b20 con 0x7f2a701210f0 2019-12-05T05:32:33.066 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:33.054+0000 7f2a787d3700 1 -- 172.21.15.42:0/2810449813 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a70139260 con 0x7f2a701210f0 2019-12-05T05:32:33.067 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:33.054+0000 7f2a7656f700 1 --2- 172.21.15.42:0/2810449813 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2a701210f0 0x7f2a7011ff70 secure :-1 s=THROTTLE_DONE pgs=1074 cs=0 l=1 rx=0x7f2a6c008370 tx=0x7f2a6c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:33.067 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:33.058+0000 7f2a677fe700 1 -- 172.21.15.42:0/2810449813 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f2a6000b330 con 0x7f2a701210f0 2019-12-05T05:32:33.067 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:33.058+0000 7f2a7656f700 1 --2- 172.21.15.42:0/2810449813 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2a701210f0 0x7f2a7011ff70 secure :-1 s=THROTTLE_DONE pgs=1074 cs=0 l=1 rx=0x7f2a6c008370 tx=0x7f2a6c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:33.068 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:33.058+0000 7f2a677fe700 1 --2- 172.21.15.42:0/2810449813 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2a5803ac40 0x7f2a5803d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:32:33.068 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:33.058+0000 7f2a677fe700 1 -- 172.21.15.42:0/2810449813 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7751+0+0 (secure 0 0 0) 0x7f2a6004b130 con 0x7f2a701210f0 2019-12-05T05:32:33.068 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:33.058+0000 7f2a76d70700 1 --2- 172.21.15.42:0/2810449813 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2a5803ac40 0x7f2a5803d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:32:33.068 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:33.058+0000 7f2a76d70700 1 --2- 172.21.15.42:0/2810449813 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2a5803ac40 0x7f2a5803d150 crc :-1 s=READY pgs=459 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:32:33.068 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:33.058+0000 7f2a787d3700 1 -- 172.21.15.42:0/2810449813 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a7005d110 con 0x7f2a701210f0 2019-12-05T05:32:33.069 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:33.066+0000 7f2a7656f700 1 --2- 172.21.15.42:0/2810449813 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2a701210f0 0x7f2a7011ff70 secure :-1 s=THROTTLE_DONE pgs=1074 cs=0 l=1 rx=0x7f2a6c008370 tx=0x7f2a6c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:33.069 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:33.066+0000 7f2a677fe700 1 -- 172.21.15.42:0/2810449813 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f2a60032190 con 0x7f2a701210f0 2019-12-05T05:32:33.243 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:33.238+0000 7f2a787d3700 1 -- 172.21.15.42:0/2810449813 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2a7005b8c0 con 0x7f2a701210f0 2019-12-05T05:32:33.245 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:33.242+0000 7f2a7656f700 1 --2- 172.21.15.42:0/2810449813 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2a701210f0 0x7f2a7011ff70 secure :-1 s=THROTTLE_DONE pgs=1074 cs=0 l=1 rx=0x7f2a6c008370 tx=0x7f2a6c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:33.245 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:33.242+0000 7f2a677fe700 1 -- 172.21.15.42:0/2810449813 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 133 v133) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f2a600323f0 con 0x7f2a701210f0 2019-12-05T05:32:33.246 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 133 2019-12-05T05:32:33.246 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:32:33.247 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":133,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6997,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/1601334502","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1601334502},{"type":"v1","addr":"172.21.15.42:6835","nonce":1601334502}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":133}],"filesystems":[{"mdsmap":{"epoch":132,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:31:53.749802+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":69,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6984},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6984":{"gid":6984,"name":"b","rank":0,"incarnation":129,"state":"up:active","state_seq":15,"addr":"172.21.15.40:6833/3543033403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3543033403},{"type":"v1","addr":"172.21.15.40:6833","nonce":3543033403}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:32:33.249 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:33.246+0000 7f2a787d3700 1 -- 172.21.15.42:0/2810449813 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2a5803ac40 msgr2=0x7f2a5803d150 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:32:33.249 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:33.246+0000 7f2a787d3700 1 --2- 172.21.15.42:0/2810449813 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2a5803ac40 0x7f2a5803d150 crc :-1 s=READY pgs=459 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:32:33.249 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:33.246+0000 7f2a787d3700 1 -- 172.21.15.42:0/2810449813 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2a701210f0 msgr2=0x7f2a7011ff70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:32:33.249 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:33.246+0000 7f2a787d3700 1 --2- 172.21.15.42:0/2810449813 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2a701210f0 0x7f2a7011ff70 secure :-1 s=READY pgs=1074 cs=0 l=1 rx=0x7f2a6c008370 tx=0x7f2a6c00fc70).stop 2019-12-05T05:32:33.250 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:33.246+0000 7f2a787d3700 1 -- 172.21.15.42:0/2810449813 shutdown_connections 2019-12-05T05:32:33.250 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:33.246+0000 7f2a787d3700 1 --2- 172.21.15.42:0/2810449813 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2a5803ac40 0x7f2a5803d150 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:32:33.250 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:33.246+0000 7f2a787d3700 1 --2- 172.21.15.42:0/2810449813 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2a701210f0 0x7f2a7011ff70 unknown :-1 s=CLOSED pgs=1074 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:32:33.250 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:33.246+0000 7f2a787d3700 1 -- 172.21.15.42:0/2810449813 shutdown_connections 2019-12-05T05:32:33.250 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:33.246+0000 7f2a787d3700 1 -- 172.21.15.42:0/2810449813 wait complete. 2019-12-05T05:32:33.250 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:33.246+0000 7f2a787d3700 1 -- 172.21.15.42:0/2810449813 >> 172.21.15.42:0/2810449813 conn(0x7f2a700aabe0 msgr2=0x7f2a70059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:32:33.259 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T05:32:35.260 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:32:35.368 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:35.358+0000 7f3384565700 1 Processor -- start 2019-12-05T05:32:35.368 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:35.358+0000 7f3384565700 1 -- start start 2019-12-05T05:32:35.368 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:35.362+0000 7f3384565700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f337c0a7740 0x7f337c08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:32:35.368 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:35.362+0000 7f3384565700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f337c07e720 0x7f337c07eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:32:35.368 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:35.362+0000 7f3384565700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f337c0a0710 con 0x7f337c09f2e0 2019-12-05T05:32:35.368 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:35.362+0000 7f3384565700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f337c0a0b00 con 0x7f337c07e720 2019-12-05T05:32:35.369 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:35.362+0000 7f3384565700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f337c08e5b0 con 0x7f337c0a7740 2019-12-05T05:32:35.369 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:35.362+0000 7f3382301700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f337c07e720 0x7f337c07eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:32:35.369 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:35.362+0000 7f3382301700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f337c07e720 0x7f337c07eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:36018/0 (socket says 172.21.15.42:36018) 2019-12-05T05:32:35.369 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:35.362+0000 7f3382301700 1 -- 172.21.15.42:0/3857372515 learned_addr learned my addr 172.21.15.42:0/3857372515 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:32:35.369 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:35.362+0000 7f3382b02700 1 --2- 172.21.15.42:0/3857372515 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f337c0a7740 0x7f337c08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:32:35.369 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:35.362+0000 7f3382301700 1 -- 172.21.15.42:0/3857372515 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f337c0a7740 msgr2=0x7f337c08df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:32:35.369 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:35.362+0000 7f3382301700 1 --2- 172.21.15.42:0/3857372515 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f337c0a7740 0x7f337c08df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:32:35.370 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:35.362+0000 7f3382301700 1 -- 172.21.15.42:0/3857372515 >> v1:172.21.15.40:6789/0 conn(0x7f337c09f2e0 legacy=0x7f337c09f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:32:35.370 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:35.362+0000 7f3381b00700 1 -- 172.21.15.42:0/3857372515 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3429741044 0 0) 0x7f337c0a0710 con 0x7f337c09f2e0 2019-12-05T05:32:35.370 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:35.362+0000 7f3382301700 1 -- 172.21.15.42:0/3857372515 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f337c07f0a0 con 0x7f337c07e720 2019-12-05T05:32:35.370 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:35.362+0000 7f3382301700 1 --2- 172.21.15.42:0/3857372515 >> v2:172.21.15.42:3300/0 conn(0x7f337c07e720 0x7f337c07eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f3378008790 tx=0x7f337800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:35.370 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:35.362+0000 7f3382301700 1 --2- 172.21.15.42:0/3857372515 >> v2:172.21.15.42:3300/0 conn(0x7f337c07e720 0x7f337c07eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f3378008790 tx=0x7f337800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:35.370 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:35.362+0000 7f3382301700 1 --2- 172.21.15.42:0/3857372515 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f337c07e720 0x7f337c07eb50 secure :-1 s=READY pgs=1075 cs=0 l=1 rx=0x7f3378008790 tx=0x7f337800fd30).ready entity=mon.1 client_cookie=a9a0d458d5296644 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:32:35.370 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:35.362+0000 7f3382301700 1 --2- 172.21.15.42:0/3857372515 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f337c07e720 0x7f337c07eb50 secure :-1 s=THROTTLE_DONE pgs=1075 cs=0 l=1 rx=0x7f3378008790 tx=0x7f337800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:35.371 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:35.362+0000 7f3381b00700 1 -- 172.21.15.42:0/3857372515 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f33780113b0 con 0x7f337c07e720 2019-12-05T05:32:35.371 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:35.362+0000 7f3382301700 1 --2- 172.21.15.42:0/3857372515 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f337c07e720 0x7f337c07eb50 secure :-1 s=THROTTLE_DONE pgs=1075 cs=0 l=1 rx=0x7f3378008790 tx=0x7f337800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:35.371 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:35.362+0000 7f3382301700 1 --2- 172.21.15.42:0/3857372515 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f337c07e720 0x7f337c07eb50 secure :-1 s=THROTTLE_DONE pgs=1075 cs=0 l=1 rx=0x7f3378008790 tx=0x7f337800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:35.371 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:35.362+0000 7f3381b00700 1 -- 172.21.15.42:0/3857372515 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3378008bb0 con 0x7f337c07e720 2019-12-05T05:32:35.371 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:35.362+0000 7f3381b00700 1 -- 172.21.15.42:0/3857372515 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3378011990 con 0x7f337c07e720 2019-12-05T05:32:35.371 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:35.362+0000 7f3384565700 1 -- 172.21.15.42:0/3857372515 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f337c07e720 msgr2=0x7f337c07eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:32:35.371 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:35.362+0000 7f3384565700 1 --2- 172.21.15.42:0/3857372515 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f337c07e720 0x7f337c07eb50 secure :-1 s=READY pgs=1075 cs=0 l=1 rx=0x7f3378008790 tx=0x7f337800fd30).stop 2019-12-05T05:32:35.372 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:35.362+0000 7f3384565700 1 -- 172.21.15.42:0/3857372515 shutdown_connections 2019-12-05T05:32:35.372 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:35.362+0000 7f3384565700 1 --2- 172.21.15.42:0/3857372515 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f337c0a7740 0x7f337c08df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:32:35.372 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:35.362+0000 7f3384565700 1 --2- 172.21.15.42:0/3857372515 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f337c07e720 0x7f337c07eb50 unknown :-1 s=CLOSED pgs=1075 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:32:35.372 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:35.362+0000 7f3384565700 1 -- 172.21.15.42:0/3857372515 shutdown_connections 2019-12-05T05:32:35.372 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:35.362+0000 7f3384565700 1 -- 172.21.15.42:0/3857372515 wait complete. 2019-12-05T05:32:35.372 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:35.362+0000 7f3384565700 1 -- 172.21.15.42:0/3857372515 >> 172.21.15.42:0/3857372515 conn(0x7f337c0aabe0 msgr2=0x7f337c0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:32:35.375 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:35.366+0000 7f3384565700 1 Processor -- start 2019-12-05T05:32:35.376 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:35.366+0000 7f3384565700 1 -- start start 2019-12-05T05:32:35.376 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:35.366+0000 7f3384565700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f337c124200 0x7f337c065340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:32:35.376 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:35.366+0000 7f3384565700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f337c063220 0x7f337c0762f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:32:35.376 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:35.366+0000 7f3384565700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f337c076840 con 0x7f337c11a7f0 2019-12-05T05:32:35.376 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:35.366+0000 7f3384565700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f337c065c10 con 0x7f337c063220 2019-12-05T05:32:35.376 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:35.366+0000 7f3384565700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f337c065970 con 0x7f337c124200 2019-12-05T05:32:35.377 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:35.370+0000 7f3382b02700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f337c124200 0x7f337c065340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:32:35.377 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:35.370+0000 7f3382b02700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f337c124200 0x7f337c065340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:36490/0 (socket says 172.21.15.42:36490) 2019-12-05T05:32:35.377 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:35.370+0000 7f3382b02700 1 -- 172.21.15.42:0/1623550638 learned_addr learned my addr 172.21.15.42:0/1623550638 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:32:35.377 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:35.370+0000 7f3383303700 1 --2- 172.21.15.42:0/1623550638 >> v2:172.21.15.40:3300/0 conn(0x7f337c063220 0x7f337c0762f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:32:35.377 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:35.370+0000 7f3382b02700 1 -- 172.21.15.42:0/1623550638 >> v2:172.21.15.40:3300/0 conn(0x7f337c063220 msgr2=0x7f337c0762f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:32:35.377 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:35.370+0000 7f3382b02700 1 --2- 172.21.15.42:0/1623550638 >> v2:172.21.15.40:3300/0 conn(0x7f337c063220 0x7f337c0762f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:32:35.378 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:35.370+0000 7f3382b02700 1 -- 172.21.15.42:0/1623550638 >> v1:172.21.15.40:6789/0 conn(0x7f337c11a7f0 legacy=0x7f337c064760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:32:35.378 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:35.370+0000 7f336f7fe700 1 -- 172.21.15.42:0/1623550638 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 34008006 0 0) 0x7f337c076840 con 0x7f337c11a7f0 2019-12-05T05:32:35.378 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:35.370+0000 7f3382b02700 1 -- 172.21.15.42:0/1623550638 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f337c063650 con 0x7f337c124200 2019-12-05T05:32:35.378 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:35.370+0000 7f3383303700 1 --2- 172.21.15.42:0/1623550638 >> v2:172.21.15.40:3300/0 conn(0x7f337c063220 0x7f337c0762f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:32:35.378 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:35.370+0000 7f3382b02700 1 --2- 172.21.15.42:0/1623550638 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f337c124200 0x7f337c065340 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f3370003070 tx=0x7f3370010ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:35.378 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:35.370+0000 7f3382b02700 1 --2- 172.21.15.42:0/1623550638 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f337c124200 0x7f337c065340 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f3370003070 tx=0x7f3370010ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:35.378 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:35.370+0000 7f3382b02700 1 --2- 172.21.15.42:0/1623550638 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f337c124200 0x7f337c065340 secure :-1 s=READY pgs=218 cs=0 l=1 rx=0x7f3370003070 tx=0x7f3370010ae0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:32:35.379 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:35.370+0000 7f3382b02700 1 --2- 172.21.15.42:0/1623550638 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f337c124200 0x7f337c065340 secure :-1 s=THROTTLE_DONE pgs=218 cs=0 l=1 rx=0x7f3370003070 tx=0x7f3370010ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:35.379 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:35.370+0000 7f336f7fe700 1 -- 172.21.15.42:0/1623550638 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3370005930 con 0x7f337c124200 2019-12-05T05:32:35.379 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:35.370+0000 7f3382b02700 1 --2- 172.21.15.42:0/1623550638 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f337c124200 0x7f337c065340 secure :-1 s=THROTTLE_DONE pgs=218 cs=0 l=1 rx=0x7f3370003070 tx=0x7f3370010ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:35.379 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:35.370+0000 7f336f7fe700 1 -- 172.21.15.42:0/1623550638 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3370007160 con 0x7f337c124200 2019-12-05T05:32:35.379 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:35.370+0000 7f3382b02700 1 --2- 172.21.15.42:0/1623550638 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f337c124200 0x7f337c065340 secure :-1 s=THROTTLE_DONE pgs=218 cs=0 l=1 rx=0x7f3370003070 tx=0x7f3370010ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:35.379 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:35.370+0000 7f3384565700 1 -- 172.21.15.42:0/1623550638 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f337c076a70 con 0x7f337c124200 2019-12-05T05:32:35.379 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:35.370+0000 7f336f7fe700 1 -- 172.21.15.42:0/1623550638 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3370012160 con 0x7f337c124200 2019-12-05T05:32:35.380 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:35.370+0000 7f3384565700 1 -- 172.21.15.42:0/1623550638 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f337c076ff0 con 0x7f337c124200 2019-12-05T05:32:35.380 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:35.370+0000 7f3382b02700 1 --2- 172.21.15.42:0/1623550638 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f337c124200 0x7f337c065340 secure :-1 s=THROTTLE_DONE pgs=218 cs=0 l=1 rx=0x7f3370003070 tx=0x7f3370010ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:35.380 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:35.370+0000 7f336f7fe700 1 -- 172.21.15.42:0/1623550638 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f3370012480 con 0x7f337c124200 2019-12-05T05:32:35.380 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:35.370+0000 7f3382b02700 1 --2- 172.21.15.42:0/1623550638 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f337c124200 0x7f337c065340 secure :-1 s=THROTTLE_DONE pgs=218 cs=0 l=1 rx=0x7f3370003070 tx=0x7f3370010ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:35.380 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:35.374+0000 7f336f7fe700 1 --2- 172.21.15.42:0/1623550638 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f336403abb0 0x7f336403d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:32:35.380 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:35.374+0000 7f336f7fe700 1 -- 172.21.15.42:0/1623550638 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7751+0+0 (secure 0 0 0) 0x7f337004aec0 con 0x7f337c124200 2019-12-05T05:32:35.380 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:35.374+0000 7f3382301700 1 --2- 172.21.15.42:0/1623550638 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f336403abb0 0x7f336403d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:32:35.381 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:35.374+0000 7f3382301700 1 --2- 172.21.15.42:0/1623550638 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f336403abb0 0x7f336403d0c0 crc :-1 s=READY pgs=460 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:32:35.381 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:35.374+0000 7f3384565700 1 -- 172.21.15.42:0/1623550638 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f337c05d110 con 0x7f337c124200 2019-12-05T05:32:35.387 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:35.382+0000 7f3382b02700 1 --2- 172.21.15.42:0/1623550638 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f337c124200 0x7f337c065340 secure :-1 s=THROTTLE_DONE pgs=218 cs=0 l=1 rx=0x7f3370003070 tx=0x7f3370010ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:35.387 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:35.382+0000 7f336f7fe700 1 -- 172.21.15.42:0/1623550638 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f337c063650 con 0x7f337c124200 2019-12-05T05:32:35.554 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:35.550+0000 7f3384565700 1 -- 172.21.15.42:0/1623550638 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f337c05b8c0 con 0x7f337c124200 2019-12-05T05:32:35.555 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:35.550+0000 7f3382b02700 1 --2- 172.21.15.42:0/1623550638 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f337c124200 0x7f337c065340 secure :-1 s=THROTTLE_DONE pgs=218 cs=0 l=1 rx=0x7f3370003070 tx=0x7f3370010ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:35.555 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:35.550+0000 7f336f7fe700 1 -- 172.21.15.42:0/1623550638 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 133 v133) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f3370031f20 con 0x7f337c124200 2019-12-05T05:32:35.557 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 133 2019-12-05T05:32:35.557 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:32:35.557 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":133,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6997,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/1601334502","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1601334502},{"type":"v1","addr":"172.21.15.42:6835","nonce":1601334502}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":133}],"filesystems":[{"mdsmap":{"epoch":132,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:31:53.749802+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":69,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6984},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6984":{"gid":6984,"name":"b","rank":0,"incarnation":129,"state":"up:active","state_seq":15,"addr":"172.21.15.40:6833/3543033403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3543033403},{"type":"v1","addr":"172.21.15.40:6833","nonce":3543033403}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:32:35.560 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:35.554+0000 7f3384565700 1 -- 172.21.15.42:0/1623550638 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f336403abb0 msgr2=0x7f336403d0c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:32:35.560 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:35.554+0000 7f3384565700 1 --2- 172.21.15.42:0/1623550638 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f336403abb0 0x7f336403d0c0 crc :-1 s=READY pgs=460 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:32:35.560 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:35.554+0000 7f3384565700 1 -- 172.21.15.42:0/1623550638 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f337c124200 msgr2=0x7f337c065340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:32:35.560 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:35.554+0000 7f3384565700 1 --2- 172.21.15.42:0/1623550638 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f337c124200 0x7f337c065340 secure :-1 s=READY pgs=218 cs=0 l=1 rx=0x7f3370003070 tx=0x7f3370010ae0).stop 2019-12-05T05:32:35.560 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:35.558+0000 7f3384565700 1 -- 172.21.15.42:0/1623550638 shutdown_connections 2019-12-05T05:32:35.561 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:35.558+0000 7f3384565700 1 --2- 172.21.15.42:0/1623550638 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f336403abb0 0x7f336403d0c0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:32:35.561 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:35.558+0000 7f3384565700 1 --2- 172.21.15.42:0/1623550638 >> v2:172.21.15.40:3300/0 conn(0x7f337c063220 0x7f337c0762f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:32:35.561 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:35.558+0000 7f3384565700 1 --2- 172.21.15.42:0/1623550638 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f337c124200 0x7f337c065340 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:32:35.561 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:35.558+0000 7f3384565700 1 -- 172.21.15.42:0/1623550638 shutdown_connections 2019-12-05T05:32:35.561 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:35.558+0000 7f3384565700 1 -- 172.21.15.42:0/1623550638 wait complete. 2019-12-05T05:32:35.562 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:35.558+0000 7f3384565700 1 -- 172.21.15.42:0/1623550638 >> 172.21.15.42:0/1623550638 conn(0x7f337c0a6bb0 msgr2=0x7f337c07e5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:32:35.570 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T05:32:37.571 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:32:37.694 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:37.666+0000 7fde2ea00700 1 Processor -- start 2019-12-05T05:32:37.694 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:37.666+0000 7fde2ea00700 1 -- start start 2019-12-05T05:32:37.695 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:37.666+0000 7fde2ea00700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fde2809fae0 0x7fde2814fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:32:37.695 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:37.666+0000 7fde2ea00700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde280a0710 con 0x7fde280a7740 2019-12-05T05:32:37.695 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:37.666+0000 7fde2ea00700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde2807ef20 con 0x7fde2808df80 2019-12-05T05:32:37.695 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:37.666+0000 7fde2ea00700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde280a0b00 con 0x7fde2809fae0 2019-12-05T05:32:37.695 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:37.666+0000 7fde27fff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fde2808df80 0x7fde2809f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:36072/0 (socket says 172.21.15.42:36072) 2019-12-05T05:32:37.695 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:37.666+0000 7fde27fff700 1 -- 172.21.15.42:0/2350345192 learned_addr learned my addr 172.21.15.42:0/2350345192 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:32:37.696 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:37.666+0000 7fde2d79e700 1 --2- 172.21.15.42:0/2350345192 >> v2:172.21.15.40:3300/0 conn(0x7fde2809fae0 0x7fde2814fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:32:37.696 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:37.666+0000 7fde277fe700 1 -- 172.21.15.42:0/2350345192 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3166803297 0 0) 0x7fde2807ef20 con 0x7fde2808df80 2019-12-05T05:32:37.696 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:37.666+0000 7fde277fe700 1 -- 172.21.15.42:0/2350345192 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde080030b0 con 0x7fde2808df80 2019-12-05T05:32:37.696 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:37.670+0000 7fde277fe700 1 -- 172.21.15.42:0/2350345192 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3599617464 0 0) 0x7fde280a0710 con 0x7fde280a7740 2019-12-05T05:32:37.696 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:37.670+0000 7fde277fe700 1 -- 172.21.15.42:0/2350345192 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde2807ef20 con 0x7fde280a7740 2019-12-05T05:32:37.696 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:37.670+0000 7fde277fe700 1 -- 172.21.15.42:0/2350345192 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1959053699 0 0) 0x7fde080030b0 con 0x7fde2808df80 2019-12-05T05:32:37.696 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:37.670+0000 7fde277fe700 1 -- 172.21.15.42:0/2350345192 >> v2:172.21.15.40:3300/0 conn(0x7fde2809fae0 msgr2=0x7fde2814fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:32:37.697 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:37.670+0000 7fde277fe700 1 --2- 172.21.15.42:0/2350345192 >> v2:172.21.15.40:3300/0 conn(0x7fde2809fae0 0x7fde2814fff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:32:37.697 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:37.670+0000 7fde277fe700 1 -- 172.21.15.42:0/2350345192 >> v1:172.21.15.40:6789/0 conn(0x7fde280a7740 legacy=0x7fde2807e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:32:37.697 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:37.670+0000 7fde277fe700 1 -- 172.21.15.42:0/2350345192 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde280a9ba0 con 0x7fde2808df80 2019-12-05T05:32:37.697 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:37.670+0000 7fde277fe700 1 -- 172.21.15.42:0/2350345192 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7fde10001750 con 0x7fde2808df80 2019-12-05T05:32:37.697 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:37.670+0000 7fde277fe700 1 -- 172.21.15.42:0/2350345192 >> v1:172.21.15.42:6789/0 conn(0x7fde2808df80 legacy=0x7fde2809f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:32:37.697 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:37.670+0000 7fde277fe700 1 --2- 172.21.15.42:0/2350345192 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fde08003910 0x7fde08003d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:32:37.698 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:37.670+0000 7fde277fe700 1 --2- 172.21.15.42:0/2350345192 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fde08004420 0x7fde080048f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:32:37.698 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:37.670+0000 7fde277fe700 1 --2- 172.21.15.42:0/2350345192 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fde08004fa0 0x7fde0804f700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:32:37.698 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:37.670+0000 7fde277fe700 1 -- 172.21.15.42:0/2350345192 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde280a0b00 con 0x7fde08004420 2019-12-05T05:32:37.698 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:37.670+0000 7fde277fe700 1 -- 172.21.15.42:0/2350345192 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde0804fec0 con 0x7fde08003910 2019-12-05T05:32:37.698 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:37.670+0000 7fde277fe700 1 -- 172.21.15.42:0/2350345192 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde08050100 con 0x7fde08004fa0 2019-12-05T05:32:37.698 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:37.670+0000 7fde2cf9d700 1 --2- 172.21.15.42:0/2350345192 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fde08003910 0x7fde08003d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:32:37.699 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:37.670+0000 7fde27fff700 1 --2- 172.21.15.42:0/2350345192 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fde08004420 0x7fde080048f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:32:37.699 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:37.670+0000 7fde2d79e700 1 --2- 172.21.15.42:0/2350345192 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fde08004fa0 0x7fde0804f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:32:37.699 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:37.670+0000 7fde2cf9d700 1 -- 172.21.15.42:0/2350345192 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fde08004fa0 msgr2=0x7fde0804f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:32:37.699 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:37.670+0000 7fde2cf9d700 1 --2- 172.21.15.42:0/2350345192 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fde08004fa0 0x7fde0804f700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:32:37.699 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:37.670+0000 7fde2cf9d700 1 -- 172.21.15.42:0/2350345192 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fde08004420 msgr2=0x7fde080048f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:32:37.699 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:37.670+0000 7fde2cf9d700 1 --2- 172.21.15.42:0/2350345192 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fde08004420 0x7fde080048f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:32:37.700 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:37.670+0000 7fde2cf9d700 1 -- 172.21.15.42:0/2350345192 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde080503a0 con 0x7fde08003910 2019-12-05T05:32:37.700 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:37.670+0000 7fde27fff700 1 --2- 172.21.15.42:0/2350345192 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fde08004420 0x7fde080048f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:32:37.700 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:37.670+0000 7fde2d79e700 1 --2- 172.21.15.42:0/2350345192 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fde08004fa0 0x7fde0804f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T05:32:37.700 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:37.670+0000 7fde2d79e700 1 -- 172.21.15.42:0/2350345192 reap_dead start 2019-12-05T05:32:37.700 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:37.670+0000 7fde2cf9d700 1 --2- 172.21.15.42:0/2350345192 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fde08003910 0x7fde08003d60 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fde18000fe0 tx=0x7fde1800be70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:37.701 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:37.670+0000 7fde2cf9d700 1 --2- 172.21.15.42:0/2350345192 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fde08003910 0x7fde08003d60 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fde18000fe0 tx=0x7fde1800be70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:37.703 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:37.670+0000 7fde2cf9d700 1 --2- 172.21.15.42:0/2350345192 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fde08003910 0x7fde08003d60 secure :-1 s=READY pgs=1077 cs=0 l=1 rx=0x7fde18000fe0 tx=0x7fde1800be70).ready entity=mon.1 client_cookie=a5705a66d0bf4c04 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:32:37.703 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:37.670+0000 7fde2cf9d700 1 --2- 172.21.15.42:0/2350345192 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fde08003910 0x7fde08003d60 secure :-1 s=THROTTLE_DONE pgs=1077 cs=0 l=1 rx=0x7fde18000fe0 tx=0x7fde1800be70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:37.704 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:37.670+0000 7fde277fe700 1 -- 172.21.15.42:0/2350345192 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fde18010be0 con 0x7fde08003910 2019-12-05T05:32:37.704 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:37.670+0000 7fde2cf9d700 1 --2- 172.21.15.42:0/2350345192 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fde08003910 0x7fde08003d60 secure :-1 s=THROTTLE_DONE pgs=1077 cs=0 l=1 rx=0x7fde18000fe0 tx=0x7fde1800be70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:37.705 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:37.670+0000 7fde2cf9d700 1 --2- 172.21.15.42:0/2350345192 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fde08003910 0x7fde08003d60 secure :-1 s=THROTTLE_DONE pgs=1077 cs=0 l=1 rx=0x7fde18000fe0 tx=0x7fde1800be70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:37.705 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:37.670+0000 7fde277fe700 1 -- 172.21.15.42:0/2350345192 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde18010800 con 0x7fde08003910 2019-12-05T05:32:37.705 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:37.670+0000 7fde277fe700 1 -- 172.21.15.42:0/2350345192 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fde1800c5b0 con 0x7fde08003910 2019-12-05T05:32:37.705 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:37.670+0000 7fde2ea00700 1 -- 172.21.15.42:0/2350345192 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fde08003910 msgr2=0x7fde08003d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:32:37.705 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:37.670+0000 7fde2ea00700 1 --2- 172.21.15.42:0/2350345192 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fde08003910 0x7fde08003d60 secure :-1 s=READY pgs=1077 cs=0 l=1 rx=0x7fde18000fe0 tx=0x7fde1800be70).stop 2019-12-05T05:32:37.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:37.670+0000 7fde2ea00700 1 -- 172.21.15.42:0/2350345192 shutdown_connections 2019-12-05T05:32:37.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:37.670+0000 7fde2ea00700 1 --2- 172.21.15.42:0/2350345192 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fde08003910 0x7fde08003d60 unknown :-1 s=CLOSED pgs=1077 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:32:37.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:37.670+0000 7fde2ea00700 1 -- 172.21.15.42:0/2350345192 shutdown_connections 2019-12-05T05:32:37.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:37.670+0000 7fde2ea00700 1 -- 172.21.15.42:0/2350345192 wait complete. 2019-12-05T05:32:37.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:37.670+0000 7fde2ea00700 1 -- 172.21.15.42:0/2350345192 >> 172.21.15.42:0/2350345192 conn(0x7fde280aabe0 msgr2=0x7fde280a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:32:37.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:37.674+0000 7fde2ea00700 1 Processor -- start 2019-12-05T05:32:37.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:37.674+0000 7fde2ea00700 1 -- start start 2019-12-05T05:32:37.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:37.674+0000 7fde2ea00700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fde2808f680 0x7fde280a4050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:32:37.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:37.674+0000 7fde2ea00700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fde28064c20 0x7fde28135c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:32:37.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:37.674+0000 7fde2ea00700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde281365b0 con 0x7fde28136180 2019-12-05T05:32:37.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:37.674+0000 7fde2ea00700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde28065050 con 0x7fde28064c20 2019-12-05T05:32:37.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:37.674+0000 7fde2ea00700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde280652c0 con 0x7fde2808f680 2019-12-05T05:32:37.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:37.674+0000 7fde2cf9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fde2808f680 0x7fde280a4050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:32:37.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:37.674+0000 7fde2cf9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fde2808f680 0x7fde280a4050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:36046/0 (socket says 172.21.15.42:36046) 2019-12-05T05:32:37.708 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:37.674+0000 7fde2cf9d700 1 -- 172.21.15.42:0/3873826997 learned_addr learned my addr 172.21.15.42:0/3873826997 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:32:37.708 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:37.674+0000 7fde27fff700 1 --2- 172.21.15.42:0/3873826997 >> v2:172.21.15.40:3300/0 conn(0x7fde28064c20 0x7fde28135c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:32:37.708 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:37.674+0000 7fde2cf9d700 1 -- 172.21.15.42:0/3873826997 >> v2:172.21.15.40:3300/0 conn(0x7fde28064c20 msgr2=0x7fde28135c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:32:37.708 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:37.674+0000 7fde2cf9d700 1 --2- 172.21.15.42:0/3873826997 >> v2:172.21.15.40:3300/0 conn(0x7fde28064c20 0x7fde28135c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:32:37.708 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:37.674+0000 7fde2cf9d700 1 -- 172.21.15.42:0/3873826997 >> v1:172.21.15.40:6789/0 conn(0x7fde28136180 legacy=0x7fde280a59e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:32:37.708 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:37.674+0000 7fde2cf9d700 1 -- 172.21.15.42:0/3873826997 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde280a61e0 con 0x7fde2808f680 2019-12-05T05:32:37.708 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:37.674+0000 7fde2cf9d700 1 --2- 172.21.15.42:0/3873826997 >> v2:172.21.15.42:3300/0 conn(0x7fde2808f680 0x7fde280a4050 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fde18000fe0 tx=0x7fde18010370).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:37.709 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:37.674+0000 7fde2cf9d700 1 --2- 172.21.15.42:0/3873826997 >> v2:172.21.15.42:3300/0 conn(0x7fde2808f680 0x7fde280a4050 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fde18000fe0 tx=0x7fde18010370).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:37.709 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:37.674+0000 7fde2cf9d700 1 --2- 172.21.15.42:0/3873826997 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fde2808f680 0x7fde280a4050 secure :-1 s=READY pgs=1078 cs=0 l=1 rx=0x7fde18000fe0 tx=0x7fde18010370).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:32:37.709 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:37.678+0000 7fde2cf9d700 1 --2- 172.21.15.42:0/3873826997 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fde2808f680 0x7fde280a4050 secure :-1 s=THROTTLE_DONE pgs=1078 cs=0 l=1 rx=0x7fde18000fe0 tx=0x7fde18010370).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:37.709 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:37.678+0000 7fde2cf9d700 1 --2- 172.21.15.42:0/3873826997 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fde2808f680 0x7fde280a4050 secure :-1 s=THROTTLE_DONE pgs=1078 cs=0 l=1 rx=0x7fde18000fe0 tx=0x7fde18010370).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:37.709 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:37.678+0000 7fde257fa700 1 -- 172.21.15.42:0/3873826997 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fde1800e8f0 con 0x7fde2808f680 2019-12-05T05:32:37.709 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:37.678+0000 7fde2cf9d700 1 --2- 172.21.15.42:0/3873826997 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fde2808f680 0x7fde280a4050 secure :-1 s=THROTTLE_DONE pgs=1078 cs=0 l=1 rx=0x7fde18000fe0 tx=0x7fde18010370).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:37.709 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:37.678+0000 7fde257fa700 1 -- 172.21.15.42:0/3873826997 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde18010de0 con 0x7fde2808f680 2019-12-05T05:32:37.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:37.678+0000 7fde2ea00700 1 -- 172.21.15.42:0/3873826997 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde280a64b0 con 0x7fde2808f680 2019-12-05T05:32:37.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:37.678+0000 7fde257fa700 1 -- 172.21.15.42:0/3873826997 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fde1800c280 con 0x7fde2808f680 2019-12-05T05:32:37.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:37.678+0000 7fde2ea00700 1 -- 172.21.15.42:0/3873826997 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde28128cb0 con 0x7fde2808f680 2019-12-05T05:32:37.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:37.678+0000 7fde2cf9d700 1 --2- 172.21.15.42:0/3873826997 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fde2808f680 0x7fde280a4050 secure :-1 s=THROTTLE_DONE pgs=1078 cs=0 l=1 rx=0x7fde18000fe0 tx=0x7fde18010370).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:37.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:37.678+0000 7fde257fa700 1 -- 172.21.15.42:0/3873826997 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fde1800c590 con 0x7fde2808f680 2019-12-05T05:32:37.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:37.678+0000 7fde2cf9d700 1 --2- 172.21.15.42:0/3873826997 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fde2808f680 0x7fde280a4050 secure :-1 s=THROTTLE_DONE pgs=1078 cs=0 l=1 rx=0x7fde18000fe0 tx=0x7fde18010370).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:37.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:37.678+0000 7fde257fa700 1 --2- 172.21.15.42:0/3873826997 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fde0003ad50 0x7fde0003d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:32:37.711 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:37.678+0000 7fde257fa700 1 -- 172.21.15.42:0/3873826997 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7751+0+0 (secure 0 0 0) 0x7fde1804b4b0 con 0x7fde2808f680 2019-12-05T05:32:37.711 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:37.678+0000 7fde27fff700 1 --2- 172.21.15.42:0/3873826997 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fde0003ad50 0x7fde0003d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:32:37.711 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:37.678+0000 7fde27fff700 1 --2- 172.21.15.42:0/3873826997 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fde0003ad50 0x7fde0003d260 crc :-1 s=READY pgs=461 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:32:37.711 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:37.682+0000 7fde2ea00700 1 -- 172.21.15.42:0/3873826997 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde2805d110 con 0x7fde2808f680 2019-12-05T05:32:37.712 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:37.690+0000 7fde2cf9d700 1 --2- 172.21.15.42:0/3873826997 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fde2808f680 0x7fde280a4050 secure :-1 s=THROTTLE_DONE pgs=1078 cs=0 l=1 rx=0x7fde18000fe0 tx=0x7fde18010370).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:37.712 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:37.690+0000 7fde257fa700 1 -- 172.21.15.42:0/3873826997 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fde18032530 con 0x7fde2808f680 2019-12-05T05:32:37.863 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:37.858+0000 7fde2ea00700 1 -- 172.21.15.42:0/3873826997 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fde2805b8c0 con 0x7fde2808f680 2019-12-05T05:32:37.863 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:37.862+0000 7fde2cf9d700 1 --2- 172.21.15.42:0/3873826997 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fde2808f680 0x7fde280a4050 secure :-1 s=THROTTLE_DONE pgs=1078 cs=0 l=1 rx=0x7fde18000fe0 tx=0x7fde18010370).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:37.864 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:37.862+0000 7fde257fa700 1 -- 172.21.15.42:0/3873826997 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 133 v133) v1 ==== 95+0+2112 (secure 0 0 0) 0x7fde18032790 con 0x7fde2808f680 2019-12-05T05:32:37.866 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:32:37.866 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":133,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6997,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/1601334502","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1601334502},{"type":"v1","addr":"172.21.15.42:6835","nonce":1601334502}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":133}],"filesystems":[{"mdsmap":{"epoch":132,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:31:53.749802+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":69,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6984},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6984":{"gid":6984,"name":"b","rank":0,"incarnation":129,"state":"up:active","state_seq":15,"addr":"172.21.15.40:6833/3543033403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3543033403},{"type":"v1","addr":"172.21.15.40:6833","nonce":3543033403}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:32:37.866 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 133 2019-12-05T05:32:37.868 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:37.866+0000 7fde2ea00700 1 -- 172.21.15.42:0/3873826997 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fde0003ad50 msgr2=0x7fde0003d260 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:32:37.869 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:37.866+0000 7fde2ea00700 1 --2- 172.21.15.42:0/3873826997 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fde0003ad50 0x7fde0003d260 crc :-1 s=READY pgs=461 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:32:37.869 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:37.866+0000 7fde2ea00700 1 -- 172.21.15.42:0/3873826997 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fde2808f680 msgr2=0x7fde280a4050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:32:37.869 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:37.866+0000 7fde2ea00700 1 --2- 172.21.15.42:0/3873826997 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fde2808f680 0x7fde280a4050 secure :-1 s=READY pgs=1078 cs=0 l=1 rx=0x7fde18000fe0 tx=0x7fde18010370).stop 2019-12-05T05:32:37.870 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:37.866+0000 7fde2ea00700 1 -- 172.21.15.42:0/3873826997 shutdown_connections 2019-12-05T05:32:37.870 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:37.866+0000 7fde2ea00700 1 --2- 172.21.15.42:0/3873826997 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fde0003ad50 0x7fde0003d260 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:32:37.870 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:37.866+0000 7fde2ea00700 1 --2- 172.21.15.42:0/3873826997 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fde2808f680 0x7fde280a4050 unknown :-1 s=CLOSED pgs=1078 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:32:37.870 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:37.866+0000 7fde2ea00700 1 --2- 172.21.15.42:0/3873826997 >> v2:172.21.15.40:3300/0 conn(0x7fde28064c20 0x7fde28135c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:32:37.870 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:37.866+0000 7fde2ea00700 1 -- 172.21.15.42:0/3873826997 shutdown_connections 2019-12-05T05:32:37.870 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:37.866+0000 7fde2ea00700 1 -- 172.21.15.42:0/3873826997 wait complete. 2019-12-05T05:32:37.871 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:37.866+0000 7fde2ea00700 1 -- 172.21.15.42:0/3873826997 >> 172.21.15.42:0/3873826997 conn(0x7fde2808fde0 msgr2=0x7fde2811d3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:32:37.878 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T05:32:39.210 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:32:39.214 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:32:39.878 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:32:39.985 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:39.978+0000 7f5d58e66700 1 Processor -- start 2019-12-05T05:32:39.986 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:39.978+0000 7f5d58e66700 1 -- start start 2019-12-05T05:32:39.986 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:39.978+0000 7f5d58e66700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5d5408df80 0x7f5d5409f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:32:39.986 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:39.978+0000 7f5d58e66700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5d5409f830 0x7f5d5414efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:32:39.986 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:39.978+0000 7f5d58e66700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d5409fc60 con 0x7f5d540a7740 2019-12-05T05:32:39.986 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:39.978+0000 7f5d58e66700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d540a0b00 con 0x7f5d5409f830 2019-12-05T05:32:39.987 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:39.978+0000 7f5d58e66700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d5408e5b0 con 0x7f5d5408df80 2019-12-05T05:32:39.987 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:39.978+0000 7f5d5259c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5d5408df80 0x7f5d5409f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:32:39.987 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:39.978+0000 7f5d5259c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5d5408df80 0x7f5d5409f2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:36056/0 (socket says 172.21.15.42:36056) 2019-12-05T05:32:39.988 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:39.978+0000 7f5d5259c700 1 -- 172.21.15.42:0/433974894 learned_addr learned my addr 172.21.15.42:0/433974894 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:32:39.988 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:39.978+0000 7f5d5359e700 1 --2- 172.21.15.42:0/433974894 >> v2:172.21.15.40:3300/0 conn(0x7f5d5409f830 0x7f5d5414efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:32:39.988 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:39.978+0000 7f5d5259c700 1 -- 172.21.15.42:0/433974894 >> v2:172.21.15.40:3300/0 conn(0x7f5d5409f830 msgr2=0x7f5d5414efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:32:39.988 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:39.978+0000 7f5d5259c700 1 --2- 172.21.15.42:0/433974894 >> v2:172.21.15.40:3300/0 conn(0x7f5d5409f830 0x7f5d5414efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:32:39.988 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:39.978+0000 7f5d5259c700 1 -- 172.21.15.42:0/433974894 >> v1:172.21.15.40:6789/0 conn(0x7f5d540a7740 legacy=0x7f5d5407e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:32:39.988 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:39.978+0000 7f5d5259c700 1 -- 172.21.15.42:0/433974894 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d540a0710 con 0x7f5d5408df80 2019-12-05T05:32:39.989 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:39.978+0000 7f5d5359e700 1 --2- 172.21.15.42:0/433974894 >> v2:172.21.15.40:3300/0 conn(0x7f5d5409f830 0x7f5d5414efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T05:32:39.990 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:39.978+0000 7f5d5259c700 1 --2- 172.21.15.42:0/433974894 >> v2:172.21.15.42:3300/0 conn(0x7f5d5408df80 0x7f5d5409f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f5d44002f30 tx=0x7f5d4400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:39.990 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:39.978+0000 7f5d5259c700 1 --2- 172.21.15.42:0/433974894 >> v2:172.21.15.42:3300/0 conn(0x7f5d5408df80 0x7f5d5409f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f5d44002f30 tx=0x7f5d4400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:39.990 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:39.978+0000 7f5d5259c700 1 --2- 172.21.15.42:0/433974894 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5d5408df80 0x7f5d5409f2e0 secure :-1 s=READY pgs=1079 cs=0 l=1 rx=0x7f5d44002f30 tx=0x7f5d4400fd30).ready entity=mon.1 client_cookie=8ea924714a8cfb30 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:32:39.990 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:39.982+0000 7f5d5259c700 1 --2- 172.21.15.42:0/433974894 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5d5408df80 0x7f5d5409f2e0 secure :-1 s=THROTTLE_DONE pgs=1079 cs=0 l=1 rx=0x7f5d44002f30 tx=0x7f5d4400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:39.990 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:39.982+0000 7f5d5259c700 1 --2- 172.21.15.42:0/433974894 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5d5408df80 0x7f5d5409f2e0 secure :-1 s=THROTTLE_DONE pgs=1079 cs=0 l=1 rx=0x7f5d44002f30 tx=0x7f5d4400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:39.990 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:39.982+0000 7f5d51d9b700 1 -- 172.21.15.42:0/433974894 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5d44008c90 con 0x7f5d5408df80 2019-12-05T05:32:39.991 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:39.982+0000 7f5d5259c700 1 --2- 172.21.15.42:0/433974894 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5d5408df80 0x7f5d5409f2e0 secure :-1 s=THROTTLE_DONE pgs=1079 cs=0 l=1 rx=0x7f5d44002f30 tx=0x7f5d4400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:39.991 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:39.982+0000 7f5d51d9b700 1 -- 172.21.15.42:0/433974894 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d440088b0 con 0x7f5d5408df80 2019-12-05T05:32:39.991 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:39.982+0000 7f5d51d9b700 1 -- 172.21.15.42:0/433974894 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5d44011ad0 con 0x7f5d5408df80 2019-12-05T05:32:39.991 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:39.982+0000 7f5d58e66700 1 -- 172.21.15.42:0/433974894 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5d5408df80 msgr2=0x7f5d5409f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:32:39.991 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:39.982+0000 7f5d58e66700 1 --2- 172.21.15.42:0/433974894 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5d5408df80 0x7f5d5409f2e0 secure :-1 s=READY pgs=1079 cs=0 l=1 rx=0x7f5d44002f30 tx=0x7f5d4400fd30).stop 2019-12-05T05:32:39.991 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:39.982+0000 7f5d58e66700 1 -- 172.21.15.42:0/433974894 shutdown_connections 2019-12-05T05:32:39.991 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:39.982+0000 7f5d58e66700 1 --2- 172.21.15.42:0/433974894 >> v2:172.21.15.40:3300/0 conn(0x7f5d5409f830 0x7f5d5414efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:32:39.992 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:39.982+0000 7f5d58e66700 1 --2- 172.21.15.42:0/433974894 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5d5408df80 0x7f5d5409f2e0 unknown :-1 s=CLOSED pgs=1079 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:32:39.992 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:39.982+0000 7f5d58e66700 1 -- 172.21.15.42:0/433974894 shutdown_connections 2019-12-05T05:32:39.992 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:39.982+0000 7f5d58e66700 1 -- 172.21.15.42:0/433974894 wait complete. 2019-12-05T05:32:39.992 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:39.982+0000 7f5d58e66700 1 -- 172.21.15.42:0/433974894 >> 172.21.15.42:0/433974894 conn(0x7f5d540aabe0 msgr2=0x7f5d540a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:32:39.996 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:39.982+0000 7f5d58e66700 1 Processor -- start 2019-12-05T05:32:39.996 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:39.986+0000 7f5d58e66700 1 -- start start 2019-12-05T05:32:39.996 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:39.986+0000 7f5d58e66700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5d5407c700 0x7f5d5411b850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:32:39.996 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:39.986+0000 7f5d58e66700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5d5411bda0 0x7f5d5413ad00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:32:39.996 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:39.986+0000 7f5d58e66700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d5407cf60 con 0x7f5d54124940 2019-12-05T05:32:39.997 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:39.986+0000 7f5d58e66700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d5411aac0 con 0x7f5d5407c700 2019-12-05T05:32:39.997 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:39.986+0000 7f5d58e66700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d5411ad60 con 0x7f5d5411bda0 2019-12-05T05:32:39.997 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:39.986+0000 7f5d5259c700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5d5407c700 0x7f5d5411b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:32:39.997 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:39.986+0000 7f5d5359e700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5d5411bda0 0x7f5d5413ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:32:39.997 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:39.986+0000 7f5d5259c700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5d5407c700 0x7f5d5411b850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.42:52112/0 (socket says 172.21.15.42:52112) 2019-12-05T05:32:39.997 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:39.986+0000 7f5d5259c700 1 -- 172.21.15.42:0/3972458357 learned_addr learned my addr 172.21.15.42:0/3972458357 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:32:39.998 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:39.986+0000 7f5d5359e700 1 -- 172.21.15.42:0/3972458357 >> v2:172.21.15.40:3300/0 conn(0x7f5d5407c700 msgr2=0x7f5d5411b850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:32:39.998 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:39.986+0000 7f5d5359e700 1 --2- 172.21.15.42:0/3972458357 >> v2:172.21.15.40:3300/0 conn(0x7f5d5407c700 0x7f5d5411b850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:32:39.998 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:39.986+0000 7f5d5359e700 1 -- 172.21.15.42:0/3972458357 >> v1:172.21.15.40:6789/0 conn(0x7f5d54124940 legacy=0x7f5d5407adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:32:39.998 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:39.986+0000 7f5d5359e700 1 -- 172.21.15.42:0/3972458357 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d5407cb30 con 0x7f5d5411bda0 2019-12-05T05:32:39.998 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:39.986+0000 7f5d5359e700 1 --2- 172.21.15.42:0/3972458357 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5d5411bda0 0x7f5d5413ad00 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f5d54116aa0 tx=0x7f5d4c010b80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:39.998 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:39.986+0000 7f5d5359e700 1 --2- 172.21.15.42:0/3972458357 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5d5411bda0 0x7f5d5413ad00 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f5d54116aa0 tx=0x7f5d4c010b80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:39.998 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:39.986+0000 7f5d5359e700 1 --2- 172.21.15.42:0/3972458357 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5d5411bda0 0x7f5d5413ad00 secure :-1 s=READY pgs=219 cs=0 l=1 rx=0x7f5d54116aa0 tx=0x7f5d4c010b80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:32:39.999 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:39.986+0000 7f5d5359e700 1 --2- 172.21.15.42:0/3972458357 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5d5411bda0 0x7f5d5413ad00 secure :-1 s=THROTTLE_DONE pgs=219 cs=0 l=1 rx=0x7f5d54116aa0 tx=0x7f5d4c010b80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:39.999 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:39.986+0000 7f5d4b7fe700 1 -- 172.21.15.42:0/3972458357 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5d4c005ab0 con 0x7f5d5411bda0 2019-12-05T05:32:40.000 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:39.986+0000 7f5d5359e700 1 --2- 172.21.15.42:0/3972458357 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5d5411bda0 0x7f5d5413ad00 secure :-1 s=THROTTLE_DONE pgs=219 cs=0 l=1 rx=0x7f5d54116aa0 tx=0x7f5d4c010b80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:40.000 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:39.986+0000 7f5d5359e700 1 --2- 172.21.15.42:0/3972458357 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5d5411bda0 0x7f5d5413ad00 secure :-1 s=THROTTLE_DONE pgs=219 cs=0 l=1 rx=0x7f5d54116aa0 tx=0x7f5d4c010b80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:40.000 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:39.986+0000 7f5d4b7fe700 1 -- 172.21.15.42:0/3972458357 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d4c00f0c0 con 0x7f5d5411bda0 2019-12-05T05:32:40.000 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:39.986+0000 7f5d58e66700 1 -- 172.21.15.42:0/3972458357 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d5407b5b0 con 0x7f5d5411bda0 2019-12-05T05:32:40.000 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:39.986+0000 7f5d4b7fe700 1 -- 172.21.15.42:0/3972458357 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5d4c00e790 con 0x7f5d5411bda0 2019-12-05T05:32:40.001 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:39.986+0000 7f5d58e66700 1 -- 172.21.15.42:0/3972458357 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d5411a740 con 0x7f5d5411bda0 2019-12-05T05:32:40.001 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:39.990+0000 7f5d5359e700 1 --2- 172.21.15.42:0/3972458357 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5d5411bda0 0x7f5d5413ad00 secure :-1 s=THROTTLE_DONE pgs=219 cs=0 l=1 rx=0x7f5d54116aa0 tx=0x7f5d4c010b80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:40.001 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:39.990+0000 7f5d5359e700 1 --2- 172.21.15.42:0/3972458357 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5d5411bda0 0x7f5d5413ad00 secure :-1 s=THROTTLE_DONE pgs=219 cs=0 l=1 rx=0x7f5d54116aa0 tx=0x7f5d4c010b80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:40.001 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:39.990+0000 7f5d4b7fe700 1 -- 172.21.15.42:0/3972458357 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f5d4c00eaa0 con 0x7f5d5411bda0 2019-12-05T05:32:40.001 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:39.990+0000 7f5d4b7fe700 1 --2- 172.21.15.42:0/3972458357 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5d2c03acf0 0x7f5d2c03d200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:32:40.001 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:39.990+0000 7f5d4b7fe700 1 -- 172.21.15.42:0/3972458357 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7751+0+0 (secure 0 0 0) 0x7f5d4c04b070 con 0x7f5d5411bda0 2019-12-05T05:32:40.001 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:39.990+0000 7f5d52d9d700 1 --2- 172.21.15.42:0/3972458357 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5d2c03acf0 0x7f5d2c03d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:32:40.002 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:39.990+0000 7f5d52d9d700 1 --2- 172.21.15.42:0/3972458357 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5d2c03acf0 0x7f5d2c03d200 crc :-1 s=READY pgs=462 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:32:40.002 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:39.994+0000 7f5d58e66700 1 -- 172.21.15.42:0/3972458357 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d5405d110 con 0x7f5d5411bda0 2019-12-05T05:32:40.007 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:40.002+0000 7f5d5359e700 1 --2- 172.21.15.42:0/3972458357 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5d5411bda0 0x7f5d5413ad00 secure :-1 s=THROTTLE_DONE pgs=219 cs=0 l=1 rx=0x7f5d54116aa0 tx=0x7f5d4c010b80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:40.007 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:40.002+0000 7f5d4b7fe700 1 -- 172.21.15.42:0/3972458357 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f5d4c0320b0 con 0x7f5d5411bda0 2019-12-05T05:32:40.175 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:40.170+0000 7f5d58e66700 1 -- 172.21.15.42:0/3972458357 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5d5405b8c0 con 0x7f5d5411bda0 2019-12-05T05:32:40.176 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:40.174+0000 7f5d5359e700 1 --2- 172.21.15.42:0/3972458357 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5d5411bda0 0x7f5d5413ad00 secure :-1 s=THROTTLE_DONE pgs=219 cs=0 l=1 rx=0x7f5d54116aa0 tx=0x7f5d4c010b80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:40.176 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:40.174+0000 7f5d4b7fe700 1 -- 172.21.15.42:0/3972458357 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 133 v133) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f5d4c032310 con 0x7f5d5411bda0 2019-12-05T05:32:40.178 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:32:40.178 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":133,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6997,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/1601334502","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1601334502},{"type":"v1","addr":"172.21.15.42:6835","nonce":1601334502}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":133}],"filesystems":[{"mdsmap":{"epoch":132,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:31:53.749802+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":69,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6984},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6984":{"gid":6984,"name":"b","rank":0,"incarnation":129,"state":"up:active","state_seq":15,"addr":"172.21.15.40:6833/3543033403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3543033403},{"type":"v1","addr":"172.21.15.40:6833","nonce":3543033403}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:32:40.178 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 133 2019-12-05T05:32:40.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:40.178+0000 7f5d58e66700 1 -- 172.21.15.42:0/3972458357 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5d2c03acf0 msgr2=0x7f5d2c03d200 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:32:40.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:40.178+0000 7f5d58e66700 1 --2- 172.21.15.42:0/3972458357 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5d2c03acf0 0x7f5d2c03d200 crc :-1 s=READY pgs=462 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:32:40.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:40.178+0000 7f5d58e66700 1 -- 172.21.15.42:0/3972458357 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5d5411bda0 msgr2=0x7f5d5413ad00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:32:40.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:40.178+0000 7f5d58e66700 1 --2- 172.21.15.42:0/3972458357 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5d5411bda0 0x7f5d5413ad00 secure :-1 s=READY pgs=219 cs=0 l=1 rx=0x7f5d54116aa0 tx=0x7f5d4c010b80).stop 2019-12-05T05:32:40.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:40.178+0000 7f5d58e66700 1 -- 172.21.15.42:0/3972458357 shutdown_connections 2019-12-05T05:32:40.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:40.178+0000 7f5d58e66700 1 --2- 172.21.15.42:0/3972458357 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5d2c03acf0 0x7f5d2c03d200 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:32:40.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:40.178+0000 7f5d58e66700 1 --2- 172.21.15.42:0/3972458357 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5d5411bda0 0x7f5d5413ad00 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:32:40.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:40.178+0000 7f5d58e66700 1 --2- 172.21.15.42:0/3972458357 >> v2:172.21.15.40:3300/0 conn(0x7f5d5407c700 0x7f5d5411b850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:32:40.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:40.178+0000 7f5d58e66700 1 -- 172.21.15.42:0/3972458357 shutdown_connections 2019-12-05T05:32:40.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:40.178+0000 7f5d58e66700 1 -- 172.21.15.42:0/3972458357 wait complete. 2019-12-05T05:32:40.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:40.178+0000 7f5d58e66700 1 -- 172.21.15.42:0/3972458357 >> 172.21.15.42:0/3972458357 conn(0x7f5d540a9870 msgr2=0x7f5d5409f380 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:32:40.191 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T05:32:42.193 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:32:42.298 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:42.286+0000 7fb0dcfab700 1 Processor -- start 2019-12-05T05:32:42.298 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:42.286+0000 7fb0dcfab700 1 -- start start 2019-12-05T05:32:42.299 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:42.286+0000 7fb0dcfab700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb0d809fae0 0x7fb0d814fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:32:42.299 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:42.286+0000 7fb0dcfab700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0d80a0710 con 0x7fb0d808df80 2019-12-05T05:32:42.300 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:42.286+0000 7fb0dcfab700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0d807ef20 con 0x7fb0d80a7740 2019-12-05T05:32:42.300 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:42.286+0000 7fb0dcfab700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0d80a0b00 con 0x7fb0d809fae0 2019-12-05T05:32:42.300 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:42.286+0000 7fb0d6d9d700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fb0d80a7740 0x7fb0d807e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:36112/0 (socket says 172.21.15.42:36112) 2019-12-05T05:32:42.300 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:42.286+0000 7fb0d6d9d700 1 -- 172.21.15.42:0/780126164 learned_addr learned my addr 172.21.15.42:0/780126164 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:32:42.300 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:42.286+0000 7fb0d759e700 1 --2- 172.21.15.42:0/780126164 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb0d809fae0 0x7fb0d814fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:32:42.301 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:42.286+0000 7fb0d5d9b700 1 -- 172.21.15.42:0/780126164 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2526553968 0 0) 0x7fb0d807ef20 con 0x7fb0d80a7740 2019-12-05T05:32:42.301 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:42.286+0000 7fb0d5d9b700 1 -- 172.21.15.42:0/780126164 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0c40030b0 con 0x7fb0d80a7740 2019-12-05T05:32:42.301 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:42.286+0000 7fb0d5d9b700 1 -- 172.21.15.42:0/780126164 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3021682627 0 0) 0x7fb0d80a0710 con 0x7fb0d808df80 2019-12-05T05:32:42.301 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:42.286+0000 7fb0d5d9b700 1 -- 172.21.15.42:0/780126164 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0d807ef20 con 0x7fb0d808df80 2019-12-05T05:32:42.301 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:42.286+0000 7fb0d5d9b700 1 -- 172.21.15.42:0/780126164 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1803498160 0 0) 0x7fb0c40030b0 con 0x7fb0d80a7740 2019-12-05T05:32:42.301 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:42.286+0000 7fb0d5d9b700 1 -- 172.21.15.42:0/780126164 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb0d809fae0 msgr2=0x7fb0d814fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:32:42.302 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:42.286+0000 7fb0d5d9b700 1 --2- 172.21.15.42:0/780126164 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb0d809fae0 0x7fb0d814fff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:32:42.302 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:42.286+0000 7fb0d5d9b700 1 -- 172.21.15.42:0/780126164 >> v1:172.21.15.40:6789/0 conn(0x7fb0d808df80 legacy=0x7fb0d809f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:32:42.302 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:42.286+0000 7fb0d5d9b700 1 -- 172.21.15.42:0/780126164 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0d80a9ba0 con 0x7fb0d80a7740 2019-12-05T05:32:42.302 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:42.286+0000 7fb0d5d9b700 1 -- 172.21.15.42:0/780126164 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7fb0c0001750 con 0x7fb0d80a7740 2019-12-05T05:32:42.302 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:42.286+0000 7fb0d5d9b700 1 -- 172.21.15.42:0/780126164 >> v1:172.21.15.42:6789/0 conn(0x7fb0d80a7740 legacy=0x7fb0d807e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:32:42.302 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:42.286+0000 7fb0d5d9b700 1 --2- 172.21.15.42:0/780126164 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb0c4003940 0x7fb0c4003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:32:42.302 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:42.286+0000 7fb0d5d9b700 1 --2- 172.21.15.42:0/780126164 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb0c4004420 0x7fb0c4004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:32:42.303 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:42.290+0000 7fb0d5d9b700 1 --2- 172.21.15.42:0/780126164 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb0c4004e40 0x7fb0c404f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:32:42.303 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:42.290+0000 7fb0d5d9b700 1 -- 172.21.15.42:0/780126164 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0d80a0b00 con 0x7fb0c4004420 2019-12-05T05:32:42.303 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:42.290+0000 7fb0d5d9b700 1 -- 172.21.15.42:0/780126164 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0c404fc00 con 0x7fb0c4004e40 2019-12-05T05:32:42.303 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:42.290+0000 7fb0d659c700 1 --2- 172.21.15.42:0/780126164 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb0c4004420 0x7fb0c4004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:32:42.303 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:42.290+0000 7fb0d5d9b700 1 -- 172.21.15.42:0/780126164 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0c404fe40 con 0x7fb0c4003940 2019-12-05T05:32:42.303 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:42.290+0000 7fb0d6d9d700 1 --2- 172.21.15.42:0/780126164 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb0c4003940 0x7fb0c4003d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:32:42.304 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:42.290+0000 7fb0d759e700 1 --2- 172.21.15.42:0/780126164 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb0c4004e40 0x7fb0c404f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:32:42.304 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:42.290+0000 7fb0d6d9d700 1 -- 172.21.15.42:0/780126164 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb0c4004e40 msgr2=0x7fb0c404f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:32:42.304 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:42.290+0000 7fb0d6d9d700 1 --2- 172.21.15.42:0/780126164 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb0c4004e40 0x7fb0c404f680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:32:42.304 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:42.290+0000 7fb0d6d9d700 1 -- 172.21.15.42:0/780126164 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb0c4004420 msgr2=0x7fb0c4004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:32:42.306 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:42.290+0000 7fb0d6d9d700 1 --2- 172.21.15.42:0/780126164 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb0c4004420 0x7fb0c4004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:32:42.307 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:42.290+0000 7fb0d6d9d700 1 -- 172.21.15.42:0/780126164 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0c4050140 con 0x7fb0c4003940 2019-12-05T05:32:42.307 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:42.290+0000 7fb0d759e700 1 --2- 172.21.15.42:0/780126164 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb0c4004e40 0x7fb0c404f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:32:42.307 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:42.290+0000 7fb0d659c700 1 --2- 172.21.15.42:0/780126164 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb0c4004420 0x7fb0c4004850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:32:42.307 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:42.290+0000 7fb0d6d9d700 1 --2- 172.21.15.42:0/780126164 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb0c4003940 0x7fb0c4003d70 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fb0c0001520 tx=0x7fb0c000c160).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:42.307 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:42.290+0000 7fb0d6d9d700 1 --2- 172.21.15.42:0/780126164 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb0c4003940 0x7fb0c4003d70 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fb0c0001520 tx=0x7fb0c000c160).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:42.308 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:42.290+0000 7fb0d6d9d700 1 --2- 172.21.15.42:0/780126164 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb0c4003940 0x7fb0c4003d70 secure :-1 s=READY pgs=220 cs=0 l=1 rx=0x7fb0c0001520 tx=0x7fb0c000c160).ready entity=mon.2 client_cookie=26b764b3098d6d0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:32:42.308 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:42.290+0000 7fb0d6d9d700 1 --2- 172.21.15.42:0/780126164 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb0c4003940 0x7fb0c4003d70 secure :-1 s=THROTTLE_DONE pgs=220 cs=0 l=1 rx=0x7fb0c0001520 tx=0x7fb0c000c160).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:42.308 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:42.290+0000 7fb0d5d9b700 1 -- 172.21.15.42:0/780126164 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb0c0010350 con 0x7fb0c4003940 2019-12-05T05:32:42.308 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:42.290+0000 7fb0d6d9d700 1 --2- 172.21.15.42:0/780126164 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb0c4003940 0x7fb0c4003d70 secure :-1 s=THROTTLE_DONE pgs=220 cs=0 l=1 rx=0x7fb0c0001520 tx=0x7fb0c000c160).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:42.308 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:42.290+0000 7fb0d6d9d700 1 --2- 172.21.15.42:0/780126164 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb0c4003940 0x7fb0c4003d70 secure :-1 s=THROTTLE_DONE pgs=220 cs=0 l=1 rx=0x7fb0c0001520 tx=0x7fb0c000c160).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:42.308 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:42.290+0000 7fb0d5d9b700 1 -- 172.21.15.42:0/780126164 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0c000ec90 con 0x7fb0c4003940 2019-12-05T05:32:42.310 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:42.290+0000 7fb0d5d9b700 1 -- 172.21.15.42:0/780126164 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb0c0005460 con 0x7fb0c4003940 2019-12-05T05:32:42.311 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:42.290+0000 7fb0dcfab700 1 -- 172.21.15.42:0/780126164 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb0c4003940 msgr2=0x7fb0c4003d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:32:42.311 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:42.290+0000 7fb0dcfab700 1 --2- 172.21.15.42:0/780126164 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb0c4003940 0x7fb0c4003d70 secure :-1 s=READY pgs=220 cs=0 l=1 rx=0x7fb0c0001520 tx=0x7fb0c000c160).stop 2019-12-05T05:32:42.311 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:42.290+0000 7fb0d759e700 1 -- 172.21.15.42:0/780126164 reap_dead start 2019-12-05T05:32:42.312 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:42.290+0000 7fb0dcfab700 1 -- 172.21.15.42:0/780126164 shutdown_connections 2019-12-05T05:32:42.312 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:42.290+0000 7fb0dcfab700 1 -- 172.21.15.42:0/780126164 shutdown_connections 2019-12-05T05:32:42.312 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:42.290+0000 7fb0dcfab700 1 -- 172.21.15.42:0/780126164 wait complete. 2019-12-05T05:32:42.312 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:42.290+0000 7fb0dcfab700 1 -- 172.21.15.42:0/780126164 >> 172.21.15.42:0/780126164 conn(0x7fb0d80aabe0 msgr2=0x7fb0d80a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:32:42.312 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:42.294+0000 7fb0dcfab700 1 Processor -- start 2019-12-05T05:32:42.312 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:42.294+0000 7fb0dcfab700 1 -- start start 2019-12-05T05:32:42.313 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:42.294+0000 7fb0dcfab700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb0d80a6550 0x7fb0d80722c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:32:42.313 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:42.294+0000 7fb0dcfab700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0d811a900 con 0x7fb0d808ed10 2019-12-05T05:32:42.313 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:42.294+0000 7fb0dcfab700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0d811bb70 con 0x7fb0d8072810 2019-12-05T05:32:42.313 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:42.294+0000 7fb0dcfab700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0d808f140 con 0x7fb0d80a6550 2019-12-05T05:32:42.313 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:42.294+0000 7fb0d659c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb0d80a6550 0x7fb0d80722c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:32:42.313 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:42.294+0000 7fb0d659c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb0d80a6550 0x7fb0d80722c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:36088/0 (socket says 172.21.15.42:36088) 2019-12-05T05:32:42.314 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:42.294+0000 7fb0d659c700 1 -- 172.21.15.42:0/1458412293 learned_addr learned my addr 172.21.15.42:0/1458412293 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:32:42.319 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:42.294+0000 7fb0cf7fe700 1 -- 172.21.15.42:0/1458412293 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1647730348 0 0) 0x7fb0d811a900 con 0x7fb0d808ed10 2019-12-05T05:32:42.319 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:42.294+0000 7fb0cf7fe700 1 -- 172.21.15.42:0/1458412293 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0bc0030b0 con 0x7fb0d808ed10 2019-12-05T05:32:42.319 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:42.294+0000 7fb0cf7fe700 1 -- 172.21.15.42:0/1458412293 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1872499427 0 0) 0x7fb0d811bb70 con 0x7fb0d8072810 2019-12-05T05:32:42.319 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:42.294+0000 7fb0d659c700 1 -- 172.21.15.42:0/1458412293 >> v1:172.21.15.42:6789/0 conn(0x7fb0d8072810 legacy=0x7fb0d8074c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:32:42.320 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:42.294+0000 7fb0d659c700 1 -- 172.21.15.42:0/1458412293 >> v1:172.21.15.40:6789/0 conn(0x7fb0d808ed10 legacy=0x7fb0d80a5d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:32:42.320 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:42.294+0000 7fb0d659c700 1 -- 172.21.15.42:0/1458412293 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0d811a4b0 con 0x7fb0d80a6550 2019-12-05T05:32:42.320 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:42.294+0000 7fb0d659c700 1 --2- 172.21.15.42:0/1458412293 >> v2:172.21.15.42:3300/0 conn(0x7fb0d80a6550 0x7fb0d80722c0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fb0c8000fe0 tx=0x7fb0c8010d40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:42.321 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:42.294+0000 7fb0d659c700 1 --2- 172.21.15.42:0/1458412293 >> v2:172.21.15.42:3300/0 conn(0x7fb0d80a6550 0x7fb0d80722c0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fb0c8000fe0 tx=0x7fb0c8010d40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:42.321 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:42.294+0000 7fb0d659c700 1 --2- 172.21.15.42:0/1458412293 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb0d80a6550 0x7fb0d80722c0 secure :-1 s=READY pgs=1082 cs=0 l=1 rx=0x7fb0c8000fe0 tx=0x7fb0c8010d40).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:32:42.321 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:42.294+0000 7fb0d659c700 1 --2- 172.21.15.42:0/1458412293 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb0d80a6550 0x7fb0d80722c0 secure :-1 s=THROTTLE_DONE pgs=1082 cs=0 l=1 rx=0x7fb0c8000fe0 tx=0x7fb0c8010d40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:42.321 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:42.294+0000 7fb0cf7fe700 1 -- 172.21.15.42:0/1458412293 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb0c800cd80 con 0x7fb0d80a6550 2019-12-05T05:32:42.321 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:42.294+0000 7fb0d659c700 1 --2- 172.21.15.42:0/1458412293 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb0d80a6550 0x7fb0d80722c0 secure :-1 s=THROTTLE_DONE pgs=1082 cs=0 l=1 rx=0x7fb0c8000fe0 tx=0x7fb0c8010d40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:42.322 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:42.294+0000 7fb0d659c700 1 --2- 172.21.15.42:0/1458412293 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb0d80a6550 0x7fb0d80722c0 secure :-1 s=THROTTLE_DONE pgs=1082 cs=0 l=1 rx=0x7fb0c8000fe0 tx=0x7fb0c8010d40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:42.322 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:42.294+0000 7fb0cf7fe700 1 -- 172.21.15.42:0/1458412293 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0c800c9a0 con 0x7fb0d80a6550 2019-12-05T05:32:42.322 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:42.294+0000 7fb0dcfab700 1 -- 172.21.15.42:0/1458412293 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0d811b330 con 0x7fb0d80a6550 2019-12-05T05:32:42.322 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:42.294+0000 7fb0cf7fe700 1 -- 172.21.15.42:0/1458412293 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb0c80127e0 con 0x7fb0d80a6550 2019-12-05T05:32:42.322 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:42.294+0000 7fb0dcfab700 1 -- 172.21.15.42:0/1458412293 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0d8076460 con 0x7fb0d80a6550 2019-12-05T05:32:42.322 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:42.298+0000 7fb0d659c700 1 --2- 172.21.15.42:0/1458412293 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb0d80a6550 0x7fb0d80722c0 secure :-1 s=THROTTLE_DONE pgs=1082 cs=0 l=1 rx=0x7fb0c8000fe0 tx=0x7fb0c8010d40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:42.323 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:42.298+0000 7fb0cf7fe700 1 -- 172.21.15.42:0/1458412293 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fb0c8012ab0 con 0x7fb0d80a6550 2019-12-05T05:32:42.323 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:42.298+0000 7fb0d659c700 1 --2- 172.21.15.42:0/1458412293 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb0d80a6550 0x7fb0d80722c0 secure :-1 s=THROTTLE_DONE pgs=1082 cs=0 l=1 rx=0x7fb0c8000fe0 tx=0x7fb0c8010d40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:42.323 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:42.298+0000 7fb0cf7fe700 1 --2- 172.21.15.42:0/1458412293 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb0bc03b230 0x7fb0bc03d740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:32:42.323 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:42.298+0000 7fb0cf7fe700 1 -- 172.21.15.42:0/1458412293 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7751+0+0 (secure 0 0 0) 0x7fb0c804b3f0 con 0x7fb0d80a6550 2019-12-05T05:32:42.323 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:42.298+0000 7fb0d6d9d700 1 --2- 172.21.15.42:0/1458412293 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb0bc03b230 0x7fb0bc03d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:32:42.323 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:42.298+0000 7fb0d6d9d700 1 --2- 172.21.15.42:0/1458412293 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb0bc03b230 0x7fb0bc03d740 crc :-1 s=READY pgs=463 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:32:42.324 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:42.302+0000 7fb0dcfab700 1 -- 172.21.15.42:0/1458412293 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0d805d110 con 0x7fb0d80a6550 2019-12-05T05:32:42.324 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:42.310+0000 7fb0d659c700 1 --2- 172.21.15.42:0/1458412293 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb0d80a6550 0x7fb0d80722c0 secure :-1 s=THROTTLE_DONE pgs=1082 cs=0 l=1 rx=0x7fb0c8000fe0 tx=0x7fb0c8010d40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:42.324 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:42.310+0000 7fb0cf7fe700 1 -- 172.21.15.42:0/1458412293 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fb0c8032470 con 0x7fb0d80a6550 2019-12-05T05:32:42.483 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:42.478+0000 7fb0dcfab700 1 -- 172.21.15.42:0/1458412293 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb0d805b8c0 con 0x7fb0d80a6550 2019-12-05T05:32:42.483 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:42.478+0000 7fb0d659c700 1 --2- 172.21.15.42:0/1458412293 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb0d80a6550 0x7fb0d80722c0 secure :-1 s=THROTTLE_DONE pgs=1082 cs=0 l=1 rx=0x7fb0c8000fe0 tx=0x7fb0c8010d40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:42.483 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:42.478+0000 7fb0cf7fe700 1 -- 172.21.15.42:0/1458412293 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 133 v133) v1 ==== 95+0+2112 (secure 0 0 0) 0x7fb0c80326d0 con 0x7fb0d80a6550 2019-12-05T05:32:42.483 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 133 2019-12-05T05:32:42.484 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:32:42.484 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":133,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6997,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/1601334502","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1601334502},{"type":"v1","addr":"172.21.15.42:6835","nonce":1601334502}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":133}],"filesystems":[{"mdsmap":{"epoch":132,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:31:53.749802+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":69,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6984},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6984":{"gid":6984,"name":"b","rank":0,"incarnation":129,"state":"up:active","state_seq":15,"addr":"172.21.15.40:6833/3543033403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3543033403},{"type":"v1","addr":"172.21.15.40:6833","nonce":3543033403}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:32:42.486 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:42.482+0000 7fb0dcfab700 1 -- 172.21.15.42:0/1458412293 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb0bc03b230 msgr2=0x7fb0bc03d740 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:32:42.486 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:42.482+0000 7fb0dcfab700 1 --2- 172.21.15.42:0/1458412293 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb0bc03b230 0x7fb0bc03d740 crc :-1 s=READY pgs=463 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:32:42.486 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:42.482+0000 7fb0dcfab700 1 -- 172.21.15.42:0/1458412293 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb0d80a6550 msgr2=0x7fb0d80722c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:32:42.486 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:42.482+0000 7fb0dcfab700 1 --2- 172.21.15.42:0/1458412293 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb0d80a6550 0x7fb0d80722c0 secure :-1 s=READY pgs=1082 cs=0 l=1 rx=0x7fb0c8000fe0 tx=0x7fb0c8010d40).stop 2019-12-05T05:32:42.486 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:42.482+0000 7fb0dcfab700 1 -- 172.21.15.42:0/1458412293 shutdown_connections 2019-12-05T05:32:42.486 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:42.482+0000 7fb0dcfab700 1 --2- 172.21.15.42:0/1458412293 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb0bc03b230 0x7fb0bc03d740 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:32:42.487 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:42.482+0000 7fb0dcfab700 1 --2- 172.21.15.42:0/1458412293 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb0d80a6550 0x7fb0d80722c0 unknown :-1 s=CLOSED pgs=1082 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:32:42.487 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:42.482+0000 7fb0dcfab700 1 -- 172.21.15.42:0/1458412293 shutdown_connections 2019-12-05T05:32:42.487 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:42.482+0000 7fb0dcfab700 1 -- 172.21.15.42:0/1458412293 wait complete. 2019-12-05T05:32:42.487 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:42.482+0000 7fb0dcfab700 1 -- 172.21.15.42:0/1458412293 >> 172.21.15.42:0/1458412293 conn(0x7fb0d811c810 msgr2=0x7fb0d809f5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:32:42.495 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T05:32:44.495 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:32:44.600 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:44.590+0000 7fe74d15c700 1 Processor -- start 2019-12-05T05:32:44.600 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:44.590+0000 7fe74d15c700 1 -- start start 2019-12-05T05:32:44.601 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:44.594+0000 7fe74d15c700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe74808df80 0x7fe74809f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:32:44.601 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:44.594+0000 7fe74d15c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe74809f830 0x7fe74814efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:32:44.601 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:44.594+0000 7fe74d15c700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe74809fc60 con 0x7fe7480a7740 2019-12-05T05:32:44.601 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:44.594+0000 7fe74d15c700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7480a0b00 con 0x7fe74808df80 2019-12-05T05:32:44.601 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:44.594+0000 7fe74d15c700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe74808e5b0 con 0x7fe74809f830 2019-12-05T05:32:44.601 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:44.594+0000 7fe74c95b700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe74809f830 0x7fe74814efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:32:44.601 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:44.594+0000 7fe74c95b700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe74809f830 0x7fe74814efe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:36100/0 (socket says 172.21.15.42:36100) 2019-12-05T05:32:44.602 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:44.594+0000 7fe74c95b700 1 -- 172.21.15.42:0/1826922461 learned_addr learned my addr 172.21.15.42:0/1826922461 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:32:44.602 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:44.594+0000 7fe746d9d700 1 --2- 172.21.15.42:0/1826922461 >> v2:172.21.15.40:3300/0 conn(0x7fe74808df80 0x7fe74809f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:32:44.602 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:44.594+0000 7fe74c95b700 1 -- 172.21.15.42:0/1826922461 >> v2:172.21.15.40:3300/0 conn(0x7fe74808df80 msgr2=0x7fe74809f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:32:44.602 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:44.594+0000 7fe74c95b700 1 --2- 172.21.15.42:0/1826922461 >> v2:172.21.15.40:3300/0 conn(0x7fe74808df80 0x7fe74809f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:32:44.602 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:44.594+0000 7fe74c95b700 1 -- 172.21.15.42:0/1826922461 >> v1:172.21.15.40:6789/0 conn(0x7fe7480a7740 legacy=0x7fe74807e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:32:44.602 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:44.594+0000 7fe74c95b700 1 -- 172.21.15.42:0/1826922461 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7480a0710 con 0x7fe74809f830 2019-12-05T05:32:44.602 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:44.594+0000 7fe74c95b700 1 --2- 172.21.15.42:0/1826922461 >> v2:172.21.15.42:3300/0 conn(0x7fe74809f830 0x7fe74814efe0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fe740002f30 tx=0x7fe74000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:44.603 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:44.594+0000 7fe74c95b700 1 --2- 172.21.15.42:0/1826922461 >> v2:172.21.15.42:3300/0 conn(0x7fe74809f830 0x7fe74814efe0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fe740002f30 tx=0x7fe74000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:44.603 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:44.594+0000 7fe74c95b700 1 --2- 172.21.15.42:0/1826922461 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe74809f830 0x7fe74814efe0 secure :-1 s=READY pgs=1083 cs=0 l=1 rx=0x7fe740002f30 tx=0x7fe74000fd30).ready entity=mon.1 client_cookie=c6e1c7f406990506 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:32:44.603 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:44.594+0000 7fe74c95b700 1 --2- 172.21.15.42:0/1826922461 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe74809f830 0x7fe74814efe0 secure :-1 s=THROTTLE_DONE pgs=1083 cs=0 l=1 rx=0x7fe740002f30 tx=0x7fe74000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:44.603 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:44.594+0000 7fe74c95b700 1 --2- 172.21.15.42:0/1826922461 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe74809f830 0x7fe74814efe0 secure :-1 s=THROTTLE_DONE pgs=1083 cs=0 l=1 rx=0x7fe740002f30 tx=0x7fe74000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:44.603 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:44.594+0000 7fe74c95b700 1 --2- 172.21.15.42:0/1826922461 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe74809f830 0x7fe74814efe0 secure :-1 s=THROTTLE_DONE pgs=1083 cs=0 l=1 rx=0x7fe740002f30 tx=0x7fe74000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:44.604 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:44.594+0000 7fe74659c700 1 -- 172.21.15.42:0/1826922461 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe740008c90 con 0x7fe74809f830 2019-12-05T05:32:44.604 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:44.594+0000 7fe74659c700 1 -- 172.21.15.42:0/1826922461 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7400088b0 con 0x7fe74809f830 2019-12-05T05:32:44.604 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:44.594+0000 7fe74659c700 1 -- 172.21.15.42:0/1826922461 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe740011ad0 con 0x7fe74809f830 2019-12-05T05:32:44.604 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:44.594+0000 7fe74d15c700 1 -- 172.21.15.42:0/1826922461 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe74809f830 msgr2=0x7fe74814efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:32:44.604 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:44.594+0000 7fe74d15c700 1 --2- 172.21.15.42:0/1826922461 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe74809f830 0x7fe74814efe0 secure :-1 s=READY pgs=1083 cs=0 l=1 rx=0x7fe740002f30 tx=0x7fe74000fd30).stop 2019-12-05T05:32:44.604 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:44.598+0000 7fe74d15c700 1 -- 172.21.15.42:0/1826922461 shutdown_connections 2019-12-05T05:32:44.604 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:44.598+0000 7fe74d15c700 1 --2- 172.21.15.42:0/1826922461 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe74809f830 0x7fe74814efe0 unknown :-1 s=CLOSED pgs=1083 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:32:44.605 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:44.598+0000 7fe74d15c700 1 --2- 172.21.15.42:0/1826922461 >> v2:172.21.15.40:3300/0 conn(0x7fe74808df80 0x7fe74809f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:32:44.605 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:44.598+0000 7fe74d15c700 1 -- 172.21.15.42:0/1826922461 shutdown_connections 2019-12-05T05:32:44.606 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:44.598+0000 7fe74d15c700 1 -- 172.21.15.42:0/1826922461 wait complete. 2019-12-05T05:32:44.607 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:44.598+0000 7fe74d15c700 1 -- 172.21.15.42:0/1826922461 >> 172.21.15.42:0/1826922461 conn(0x7fe7480aabe0 msgr2=0x7fe7480a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:32:44.610 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:44.598+0000 7fe74d15c700 1 Processor -- start 2019-12-05T05:32:44.610 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:44.598+0000 7fe74d15c700 1 -- start start 2019-12-05T05:32:44.610 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:44.602+0000 7fe74d15c700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe7481236d0 0x7fe74812d700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:32:44.610 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:44.602+0000 7fe74d15c700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe74807b9e0 0x7fe74811bc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:32:44.610 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:44.602+0000 7fe74d15c700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe74812ce20 con 0x7fe7480a4d50 2019-12-05T05:32:44.611 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:44.602+0000 7fe74d15c700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe74812dc50 con 0x7fe7481236d0 2019-12-05T05:32:44.611 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:44.602+0000 7fe74d15c700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe74807c8d0 con 0x7fe74807b9e0 2019-12-05T05:32:44.611 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:44.602+0000 7fe74c95b700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fe7480a4d50 0x7fe7481303e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:36144/0 (socket says 172.21.15.42:36144) 2019-12-05T05:32:44.611 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:44.602+0000 7fe74c95b700 1 -- 172.21.15.42:0/554131061 learned_addr learned my addr 172.21.15.42:0/554131061 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:32:44.611 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:44.602+0000 7fe746d9d700 1 --2- 172.21.15.42:0/554131061 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe74807b9e0 0x7fe74811bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:32:44.611 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:44.602+0000 7fe74759e700 1 --2- 172.21.15.42:0/554131061 >> v2:172.21.15.40:3300/0 conn(0x7fe7481236d0 0x7fe74812d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:32:44.611 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:44.602+0000 7fe72ffff700 1 -- 172.21.15.42:0/554131061 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2240455732 0 0) 0x7fe74812ce20 con 0x7fe7480a4d50 2019-12-05T05:32:44.612 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:44.602+0000 7fe72ffff700 1 -- 172.21.15.42:0/554131061 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7280030b0 con 0x7fe7480a4d50 2019-12-05T05:32:44.612 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:44.602+0000 7fe746d9d700 1 -- 172.21.15.42:0/554131061 >> v2:172.21.15.40:3300/0 conn(0x7fe7481236d0 msgr2=0x7fe74812d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:32:44.612 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:44.602+0000 7fe746d9d700 1 --2- 172.21.15.42:0/554131061 >> v2:172.21.15.40:3300/0 conn(0x7fe7481236d0 0x7fe74812d700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:32:44.612 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:44.602+0000 7fe746d9d700 1 -- 172.21.15.42:0/554131061 >> v1:172.21.15.42:6789/0 conn(0x7fe7480a4d50 legacy=0x7fe7481303e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:32:44.612 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:44.602+0000 7fe746d9d700 1 -- 172.21.15.42:0/554131061 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe74811c170 con 0x7fe74807b9e0 2019-12-05T05:32:44.612 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:44.602+0000 7fe746d9d700 1 --2- 172.21.15.42:0/554131061 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe74807b9e0 0x7fe74811bc20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fe73c002f30 tx=0x7fe73c010b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:44.613 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:44.602+0000 7fe746d9d700 1 --2- 172.21.15.42:0/554131061 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe74807b9e0 0x7fe74811bc20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fe73c002f30 tx=0x7fe73c010b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:44.613 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:44.602+0000 7fe746d9d700 1 --2- 172.21.15.42:0/554131061 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe74807b9e0 0x7fe74811bc20 secure :-1 s=READY pgs=221 cs=0 l=1 rx=0x7fe73c002f30 tx=0x7fe73c010b70).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:32:44.613 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:44.602+0000 7fe746d9d700 1 --2- 172.21.15.42:0/554131061 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe74807b9e0 0x7fe74811bc20 secure :-1 s=THROTTLE_DONE pgs=221 cs=0 l=1 rx=0x7fe73c002f30 tx=0x7fe73c010b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:44.614 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:44.602+0000 7fe72ffff700 1 -- 172.21.15.42:0/554131061 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe73c012620 con 0x7fe74807b9e0 2019-12-05T05:32:44.614 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:44.602+0000 7fe746d9d700 1 --2- 172.21.15.42:0/554131061 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe74807b9e0 0x7fe74811bc20 secure :-1 s=THROTTLE_DONE pgs=221 cs=0 l=1 rx=0x7fe73c002f30 tx=0x7fe73c010b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:44.614 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:44.602+0000 7fe74d15c700 1 -- 172.21.15.42:0/554131061 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7480a5180 con 0x7fe74807b9e0 2019-12-05T05:32:44.614 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:44.602+0000 7fe746d9d700 1 --2- 172.21.15.42:0/554131061 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe74807b9e0 0x7fe74811bc20 secure :-1 s=THROTTLE_DONE pgs=221 cs=0 l=1 rx=0x7fe73c002f30 tx=0x7fe73c010b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:44.615 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:44.602+0000 7fe72ffff700 1 -- 172.21.15.42:0/554131061 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe73c007150 con 0x7fe74807b9e0 2019-12-05T05:32:44.615 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:44.602+0000 7fe72ffff700 1 -- 172.21.15.42:0/554131061 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe73c012c00 con 0x7fe74807b9e0 2019-12-05T05:32:44.615 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:44.602+0000 7fe74d15c700 1 -- 172.21.15.42:0/554131061 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe748130be0 con 0x7fe74807b9e0 2019-12-05T05:32:44.615 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:44.602+0000 7fe746d9d700 1 --2- 172.21.15.42:0/554131061 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe74807b9e0 0x7fe74811bc20 secure :-1 s=THROTTLE_DONE pgs=221 cs=0 l=1 rx=0x7fe73c002f30 tx=0x7fe73c010b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:44.616 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:44.606+0000 7fe72ffff700 1 -- 172.21.15.42:0/554131061 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fe73c005540 con 0x7fe74807b9e0 2019-12-05T05:32:44.617 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:44.606+0000 7fe746d9d700 1 --2- 172.21.15.42:0/554131061 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe74807b9e0 0x7fe74811bc20 secure :-1 s=THROTTLE_DONE pgs=221 cs=0 l=1 rx=0x7fe73c002f30 tx=0x7fe73c010b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:44.617 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:44.606+0000 7fe72ffff700 1 --2- 172.21.15.42:0/554131061 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe72803b230 0x7fe72803d740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:32:44.617 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:44.606+0000 7fe72ffff700 1 -- 172.21.15.42:0/554131061 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7751+0+0 (secure 0 0 0) 0x7fe73c04b130 con 0x7fe74807b9e0 2019-12-05T05:32:44.617 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:44.606+0000 7fe74759e700 1 --2- 172.21.15.42:0/554131061 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe72803b230 0x7fe72803d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:32:44.617 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:44.606+0000 7fe74759e700 1 --2- 172.21.15.42:0/554131061 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe72803b230 0x7fe72803d740 crc :-1 s=READY pgs=464 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:32:44.618 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:44.606+0000 7fe74d15c700 1 -- 172.21.15.42:0/554131061 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe74805d110 con 0x7fe74807b9e0 2019-12-05T05:32:44.620 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:44.618+0000 7fe746d9d700 1 --2- 172.21.15.42:0/554131061 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe74807b9e0 0x7fe74811bc20 secure :-1 s=THROTTLE_DONE pgs=221 cs=0 l=1 rx=0x7fe73c002f30 tx=0x7fe73c010b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:44.620 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:44.618+0000 7fe72ffff700 1 -- 172.21.15.42:0/554131061 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fe73c032190 con 0x7fe74807b9e0 2019-12-05T05:32:44.787 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:44.782+0000 7fe74d15c700 1 -- 172.21.15.42:0/554131061 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe74805b8c0 con 0x7fe74807b9e0 2019-12-05T05:32:44.788 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:44.786+0000 7fe746d9d700 1 --2- 172.21.15.42:0/554131061 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe74807b9e0 0x7fe74811bc20 secure :-1 s=THROTTLE_DONE pgs=221 cs=0 l=1 rx=0x7fe73c002f30 tx=0x7fe73c010b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:44.788 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:44.786+0000 7fe72ffff700 1 -- 172.21.15.42:0/554131061 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 133 v133) v1 ==== 95+0+2112 (secure 0 0 0) 0x7fe73c0323f0 con 0x7fe74807b9e0 2019-12-05T05:32:44.790 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 133 2019-12-05T05:32:44.790 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:32:44.790 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":133,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6997,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/1601334502","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1601334502},{"type":"v1","addr":"172.21.15.42:6835","nonce":1601334502}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":133}],"filesystems":[{"mdsmap":{"epoch":132,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:31:53.749802+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":69,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6984},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6984":{"gid":6984,"name":"b","rank":0,"incarnation":129,"state":"up:active","state_seq":15,"addr":"172.21.15.40:6833/3543033403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3543033403},{"type":"v1","addr":"172.21.15.40:6833","nonce":3543033403}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:32:44.793 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:44.790+0000 7fe74d15c700 1 -- 172.21.15.42:0/554131061 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe72803b230 msgr2=0x7fe72803d740 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:32:44.793 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:44.790+0000 7fe74d15c700 1 --2- 172.21.15.42:0/554131061 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe72803b230 0x7fe72803d740 crc :-1 s=READY pgs=464 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:32:44.793 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:44.790+0000 7fe74d15c700 1 -- 172.21.15.42:0/554131061 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe74807b9e0 msgr2=0x7fe74811bc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:32:44.793 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:44.790+0000 7fe74d15c700 1 --2- 172.21.15.42:0/554131061 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe74807b9e0 0x7fe74811bc20 secure :-1 s=READY pgs=221 cs=0 l=1 rx=0x7fe73c002f30 tx=0x7fe73c010b70).stop 2019-12-05T05:32:44.794 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:44.790+0000 7fe74d15c700 1 -- 172.21.15.42:0/554131061 shutdown_connections 2019-12-05T05:32:44.794 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:44.790+0000 7fe74d15c700 1 --2- 172.21.15.42:0/554131061 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe72803b230 0x7fe72803d740 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:32:44.794 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:44.790+0000 7fe74d15c700 1 --2- 172.21.15.42:0/554131061 >> v2:172.21.15.40:3300/0 conn(0x7fe7481236d0 0x7fe74812d700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:32:44.794 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:44.790+0000 7fe74d15c700 1 --2- 172.21.15.42:0/554131061 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe74807b9e0 0x7fe74811bc20 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:32:44.794 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:44.790+0000 7fe74d15c700 1 -- 172.21.15.42:0/554131061 shutdown_connections 2019-12-05T05:32:44.794 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:44.790+0000 7fe74d15c700 1 -- 172.21.15.42:0/554131061 wait complete. 2019-12-05T05:32:44.794 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:44.790+0000 7fe74d15c700 1 -- 172.21.15.42:0/554131061 >> 172.21.15.42:0/554131061 conn(0x7fe74808fda0 msgr2=0x7fe748059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:32:44.803 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T05:32:46.804 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:32:46.906 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:46.898+0000 7fd9b40b6700 1 Processor -- start 2019-12-05T05:32:46.907 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:46.898+0000 7fd9b40b6700 1 -- start start 2019-12-05T05:32:46.907 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:46.898+0000 7fd9b40b6700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd9ac0a7740 0x7fd9ac08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:32:46.907 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:46.898+0000 7fd9b40b6700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd9ac09fae0 0x7fd9ac07eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:32:46.907 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:46.898+0000 7fd9b40b6700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9ac0a0710 con 0x7fd9ac07e720 2019-12-05T05:32:46.907 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:46.898+0000 7fd9b40b6700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9ac0a0b00 con 0x7fd9ac0a7740 2019-12-05T05:32:46.907 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:46.898+0000 7fd9b40b6700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9ac08e5b0 con 0x7fd9ac09fae0 2019-12-05T05:32:46.908 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:46.898+0000 7fd9b2653700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd9ac0a7740 0x7fd9ac08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:32:46.908 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:46.898+0000 7fd9b2e54700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd9ac09fae0 0x7fd9ac07eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:32:46.908 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:46.898+0000 7fd9b2e54700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd9ac09fae0 0x7fd9ac07eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:36578/0 (socket says 172.21.15.42:36578) 2019-12-05T05:32:46.908 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:46.898+0000 7fd9b2e54700 1 -- 172.21.15.42:0/2588965340 learned_addr learned my addr 172.21.15.42:0/2588965340 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:32:46.908 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:46.898+0000 7fd9b2e54700 1 -- 172.21.15.42:0/2588965340 >> v2:172.21.15.40:3300/0 conn(0x7fd9ac0a7740 msgr2=0x7fd9ac08df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:32:46.908 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:46.898+0000 7fd9b2e54700 1 --2- 172.21.15.42:0/2588965340 >> v2:172.21.15.40:3300/0 conn(0x7fd9ac0a7740 0x7fd9ac08df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:32:46.908 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:46.898+0000 7fd9b2e54700 1 -- 172.21.15.42:0/2588965340 >> v1:172.21.15.40:6789/0 conn(0x7fd9ac07e720 legacy=0x7fd9ac09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:32:46.909 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:46.898+0000 7fd9b1651700 1 -- 172.21.15.42:0/2588965340 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1536914029 0 0) 0x7fd9ac0a0710 con 0x7fd9ac07e720 2019-12-05T05:32:46.909 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:46.898+0000 7fd9b2e54700 1 -- 172.21.15.42:0/2588965340 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9ac07f0a0 con 0x7fd9ac09fae0 2019-12-05T05:32:46.909 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:46.898+0000 7fd9b2653700 1 --2- 172.21.15.42:0/2588965340 >> v2:172.21.15.40:3300/0 conn(0x7fd9ac0a7740 0x7fd9ac08df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:32:46.909 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:46.898+0000 7fd9b2e54700 1 --2- 172.21.15.42:0/2588965340 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd9ac09fae0 0x7fd9ac07eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fd9a4008790 tx=0x7fd9a400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:46.909 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:46.898+0000 7fd9b2e54700 1 --2- 172.21.15.42:0/2588965340 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd9ac09fae0 0x7fd9ac07eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fd9a4008790 tx=0x7fd9a400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:46.909 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:46.898+0000 7fd9b2e54700 1 --2- 172.21.15.42:0/2588965340 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd9ac09fae0 0x7fd9ac07eb50 secure :-1 s=READY pgs=222 cs=0 l=1 rx=0x7fd9a4008790 tx=0x7fd9a400fd30).ready entity=mon.2 client_cookie=f0037f2f2b1d648 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:32:46.909 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:46.902+0000 7fd9b2e54700 1 --2- 172.21.15.42:0/2588965340 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd9ac09fae0 0x7fd9ac07eb50 secure :-1 s=THROTTLE_DONE pgs=222 cs=0 l=1 rx=0x7fd9a4008790 tx=0x7fd9a400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:46.910 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:46.902+0000 7fd9b1651700 1 -- 172.21.15.42:0/2588965340 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd9a4008c90 con 0x7fd9ac09fae0 2019-12-05T05:32:46.910 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:46.902+0000 7fd9b2e54700 1 --2- 172.21.15.42:0/2588965340 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd9ac09fae0 0x7fd9ac07eb50 secure :-1 s=THROTTLE_DONE pgs=222 cs=0 l=1 rx=0x7fd9a4008790 tx=0x7fd9a400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:46.910 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:46.902+0000 7fd9b2e54700 1 --2- 172.21.15.42:0/2588965340 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd9ac09fae0 0x7fd9ac07eb50 secure :-1 s=THROTTLE_DONE pgs=222 cs=0 l=1 rx=0x7fd9a4008790 tx=0x7fd9a400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:46.910 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:46.902+0000 7fd9b1651700 1 -- 172.21.15.42:0/2588965340 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9a40088b0 con 0x7fd9ac09fae0 2019-12-05T05:32:46.910 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:46.902+0000 7fd9b1651700 1 -- 172.21.15.42:0/2588965340 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd9a4011b10 con 0x7fd9ac09fae0 2019-12-05T05:32:46.910 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:46.902+0000 7fd9b40b6700 1 -- 172.21.15.42:0/2588965340 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd9ac09fae0 msgr2=0x7fd9ac07eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:32:46.910 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:46.902+0000 7fd9b40b6700 1 --2- 172.21.15.42:0/2588965340 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd9ac09fae0 0x7fd9ac07eb50 secure :-1 s=READY pgs=222 cs=0 l=1 rx=0x7fd9a4008790 tx=0x7fd9a400fd30).stop 2019-12-05T05:32:46.911 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:46.902+0000 7fd9b40b6700 1 -- 172.21.15.42:0/2588965340 shutdown_connections 2019-12-05T05:32:46.911 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:46.902+0000 7fd9b40b6700 1 --2- 172.21.15.42:0/2588965340 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd9ac09fae0 0x7fd9ac07eb50 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:32:46.911 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:46.902+0000 7fd9b40b6700 1 --2- 172.21.15.42:0/2588965340 >> v2:172.21.15.40:3300/0 conn(0x7fd9ac0a7740 0x7fd9ac08df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:32:46.911 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:46.902+0000 7fd9b40b6700 1 -- 172.21.15.42:0/2588965340 shutdown_connections 2019-12-05T05:32:46.911 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:46.902+0000 7fd9b40b6700 1 -- 172.21.15.42:0/2588965340 wait complete. 2019-12-05T05:32:46.911 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:46.902+0000 7fd9b40b6700 1 -- 172.21.15.42:0/2588965340 >> 172.21.15.42:0/2588965340 conn(0x7fd9ac0aabe0 msgr2=0x7fd9ac0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:32:46.911 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:46.902+0000 7fd9b40b6700 1 Processor -- start 2019-12-05T05:32:46.911 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:46.902+0000 7fd9b40b6700 1 -- start start 2019-12-05T05:32:46.916 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:46.906+0000 7fd9b40b6700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd9ac11ba70 0x7fd9ac11cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:32:46.916 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:46.906+0000 7fd9b40b6700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd9ac0722c0 0x7fd9ac074700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:32:46.916 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:46.906+0000 7fd9b40b6700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9ac12ef20 con 0x7fd9ac08eb70 2019-12-05T05:32:46.916 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:46.906+0000 7fd9b40b6700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9ac11eba0 con 0x7fd9ac0722c0 2019-12-05T05:32:46.916 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:46.906+0000 7fd9b40b6700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9ac11c0a0 con 0x7fd9ac11ba70 2019-12-05T05:32:46.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:46.906+0000 7fd9b1e52700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fd9ac08eb70 0x7fd9ac12e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:36156/0 (socket says 172.21.15.42:36156) 2019-12-05T05:32:46.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:46.906+0000 7fd9b1e52700 1 -- 172.21.15.42:0/3197422954 learned_addr learned my addr 172.21.15.42:0/3197422954 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:32:46.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:46.906+0000 7fd9b2653700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd9ac11ba70 0x7fd9ac11cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:32:46.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:46.906+0000 7fd9b2e54700 1 --2- 172.21.15.42:0/3197422954 >> v2:172.21.15.40:3300/0 conn(0x7fd9ac0722c0 0x7fd9ac074700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:32:46.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:46.906+0000 7fd9aaffd700 1 -- 172.21.15.42:0/3197422954 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2928319983 0 0) 0x7fd9ac12ef20 con 0x7fd9ac08eb70 2019-12-05T05:32:46.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:46.906+0000 7fd9aaffd700 1 -- 172.21.15.42:0/3197422954 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9900030b0 con 0x7fd9ac08eb70 2019-12-05T05:32:46.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:46.906+0000 7fd9b2653700 1 -- 172.21.15.42:0/3197422954 >> v2:172.21.15.40:3300/0 conn(0x7fd9ac0722c0 msgr2=0x7fd9ac074700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:32:46.918 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:46.906+0000 7fd9b2653700 1 --2- 172.21.15.42:0/3197422954 >> v2:172.21.15.40:3300/0 conn(0x7fd9ac0722c0 0x7fd9ac074700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:32:46.918 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:46.906+0000 7fd9b2653700 1 -- 172.21.15.42:0/3197422954 >> v1:172.21.15.42:6789/0 conn(0x7fd9ac08eb70 legacy=0x7fd9ac12e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:32:46.918 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:46.906+0000 7fd9b2653700 1 -- 172.21.15.42:0/3197422954 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9ac08efa0 con 0x7fd9ac11ba70 2019-12-05T05:32:46.918 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:46.906+0000 7fd9b2653700 1 --2- 172.21.15.42:0/3197422954 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd9ac11ba70 0x7fd9ac11cfe0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fd99c0032b0 tx=0x7fd99c010fa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:46.918 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:46.906+0000 7fd9b2653700 1 --2- 172.21.15.42:0/3197422954 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd9ac11ba70 0x7fd9ac11cfe0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fd99c0032b0 tx=0x7fd99c010fa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:46.918 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:46.906+0000 7fd9b2653700 1 --2- 172.21.15.42:0/3197422954 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd9ac11ba70 0x7fd9ac11cfe0 secure :-1 s=READY pgs=223 cs=0 l=1 rx=0x7fd99c0032b0 tx=0x7fd99c010fa0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:32:46.919 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:46.906+0000 7fd9b2653700 1 --2- 172.21.15.42:0/3197422954 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd9ac11ba70 0x7fd9ac11cfe0 secure :-1 s=THROTTLE_DONE pgs=223 cs=0 l=1 rx=0x7fd99c0032b0 tx=0x7fd99c010fa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:46.919 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:46.906+0000 7fd9aaffd700 1 -- 172.21.15.42:0/3197422954 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd99c012b50 con 0x7fd9ac11ba70 2019-12-05T05:32:46.919 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:46.906+0000 7fd9b2653700 1 --2- 172.21.15.42:0/3197422954 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd9ac11ba70 0x7fd9ac11cfe0 secure :-1 s=THROTTLE_DONE pgs=223 cs=0 l=1 rx=0x7fd99c0032b0 tx=0x7fd99c010fa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:46.919 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:46.906+0000 7fd9b2653700 1 --2- 172.21.15.42:0/3197422954 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd9ac11ba70 0x7fd9ac11cfe0 secure :-1 s=THROTTLE_DONE pgs=223 cs=0 l=1 rx=0x7fd99c0032b0 tx=0x7fd99c010fa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:46.919 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:46.906+0000 7fd9aaffd700 1 -- 172.21.15.42:0/3197422954 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd99c00e200 con 0x7fd9ac11ba70 2019-12-05T05:32:46.919 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:46.906+0000 7fd9b40b6700 1 -- 172.21.15.42:0/3197422954 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9ac074c50 con 0x7fd9ac11ba70 2019-12-05T05:32:46.919 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:46.906+0000 7fd9aaffd700 1 -- 172.21.15.42:0/3197422954 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd99c005ab0 con 0x7fd9ac11ba70 2019-12-05T05:32:46.920 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:46.906+0000 7fd9b40b6700 1 -- 172.21.15.42:0/3197422954 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9ac0751d0 con 0x7fd9ac11ba70 2019-12-05T05:32:46.920 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:46.910+0000 7fd9b2653700 1 --2- 172.21.15.42:0/3197422954 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd9ac11ba70 0x7fd9ac11cfe0 secure :-1 s=THROTTLE_DONE pgs=223 cs=0 l=1 rx=0x7fd99c0032b0 tx=0x7fd99c010fa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:46.920 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:46.910+0000 7fd9aaffd700 1 -- 172.21.15.42:0/3197422954 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fd99c007180 con 0x7fd9ac11ba70 2019-12-05T05:32:46.920 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:46.910+0000 7fd9b2653700 1 --2- 172.21.15.42:0/3197422954 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd9ac11ba70 0x7fd9ac11cfe0 secure :-1 s=THROTTLE_DONE pgs=223 cs=0 l=1 rx=0x7fd99c0032b0 tx=0x7fd99c010fa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:46.920 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:46.910+0000 7fd9aaffd700 1 --2- 172.21.15.42:0/3197422954 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd99003b280 0x7fd99003d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:32:46.920 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:46.910+0000 7fd9aaffd700 1 -- 172.21.15.42:0/3197422954 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7751+0+0 (secure 0 0 0) 0x7fd99c04b120 con 0x7fd9ac11ba70 2019-12-05T05:32:46.920 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:46.910+0000 7fd9b1e52700 1 --2- 172.21.15.42:0/3197422954 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd99003b280 0x7fd99003d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:32:46.921 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:46.910+0000 7fd9b1e52700 1 --2- 172.21.15.42:0/3197422954 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd99003b280 0x7fd99003d790 crc :-1 s=READY pgs=465 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:32:46.921 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:46.914+0000 7fd9b40b6700 1 -- 172.21.15.42:0/3197422954 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9ac11bea0 con 0x7fd9ac11ba70 2019-12-05T05:32:46.926 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:46.922+0000 7fd9b2653700 1 --2- 172.21.15.42:0/3197422954 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd9ac11ba70 0x7fd9ac11cfe0 secure :-1 s=THROTTLE_DONE pgs=223 cs=0 l=1 rx=0x7fd99c0032b0 tx=0x7fd99c010fa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:46.927 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:46.922+0000 7fd9aaffd700 1 -- 172.21.15.42:0/3197422954 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fd99c005dc0 con 0x7fd9ac11ba70 2019-12-05T05:32:47.092 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:47.090+0000 7fd9b40b6700 1 -- 172.21.15.42:0/3197422954 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd9ac05d110 con 0x7fd9ac11ba70 2019-12-05T05:32:47.093 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:47.090+0000 7fd9b2653700 1 --2- 172.21.15.42:0/3197422954 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd9ac11ba70 0x7fd9ac11cfe0 secure :-1 s=THROTTLE_DONE pgs=223 cs=0 l=1 rx=0x7fd99c0032b0 tx=0x7fd99c010fa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:47.093 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:47.090+0000 7fd9aaffd700 1 -- 172.21.15.42:0/3197422954 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 133 v133) v1 ==== 95+0+2112 (secure 0 0 0) 0x7fd99c0321a0 con 0x7fd9ac11ba70 2019-12-05T05:32:47.095 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:32:47.096 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":133,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6997,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/1601334502","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1601334502},{"type":"v1","addr":"172.21.15.42:6835","nonce":1601334502}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":133}],"filesystems":[{"mdsmap":{"epoch":132,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:31:53.749802+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":69,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6984},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6984":{"gid":6984,"name":"b","rank":0,"incarnation":129,"state":"up:active","state_seq":15,"addr":"172.21.15.40:6833/3543033403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3543033403},{"type":"v1","addr":"172.21.15.40:6833","nonce":3543033403}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:32:47.096 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 133 2019-12-05T05:32:47.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:47.094+0000 7fd9b40b6700 1 -- 172.21.15.42:0/3197422954 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd99003b280 msgr2=0x7fd99003d790 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:32:47.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:47.094+0000 7fd9b40b6700 1 --2- 172.21.15.42:0/3197422954 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd99003b280 0x7fd99003d790 crc :-1 s=READY pgs=465 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:32:47.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:47.094+0000 7fd9b40b6700 1 -- 172.21.15.42:0/3197422954 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd9ac11ba70 msgr2=0x7fd9ac11cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:32:47.115 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:47.094+0000 7fd9b40b6700 1 --2- 172.21.15.42:0/3197422954 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd9ac11ba70 0x7fd9ac11cfe0 secure :-1 s=READY pgs=223 cs=0 l=1 rx=0x7fd99c0032b0 tx=0x7fd99c010fa0).stop 2019-12-05T05:32:47.115 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:47.094+0000 7fd9b40b6700 1 -- 172.21.15.42:0/3197422954 shutdown_connections 2019-12-05T05:32:47.115 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:47.094+0000 7fd9b40b6700 1 --2- 172.21.15.42:0/3197422954 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd99003b280 0x7fd99003d790 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:32:47.116 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:47.094+0000 7fd9b40b6700 1 --2- 172.21.15.42:0/3197422954 >> v2:172.21.15.40:3300/0 conn(0x7fd9ac0722c0 0x7fd9ac074700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:32:47.116 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:47.094+0000 7fd9b40b6700 1 --2- 172.21.15.42:0/3197422954 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd9ac11ba70 0x7fd9ac11cfe0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:32:47.116 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:47.094+0000 7fd9b40b6700 1 -- 172.21.15.42:0/3197422954 shutdown_connections 2019-12-05T05:32:47.116 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:47.094+0000 7fd9b40b6700 1 -- 172.21.15.42:0/3197422954 wait complete. 2019-12-05T05:32:47.116 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:47.094+0000 7fd9b40b6700 1 -- 172.21.15.42:0/3197422954 >> 172.21.15.42:0/3197422954 conn(0x7fd9ac07ebe0 msgr2=0x7fd9ac0a9790 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:32:47.117 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T05:32:49.119 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:32:49.224 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.213+0000 7f021608b700 1 Processor -- start 2019-12-05T05:32:49.225 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.213+0000 7f021608b700 1 -- start start 2019-12-05T05:32:49.225 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.217+0000 7f021608b700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f02100a7730 0x7f021008df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:32:49.225 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.217+0000 7f021608b700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02100a0700 con 0x7f021009fad0 2019-12-05T05:32:49.225 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.217+0000 7f021608b700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02100a9b90 con 0x7f021007e710 2019-12-05T05:32:49.226 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.217+0000 7f021608b700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02100a0af0 con 0x7f02100a7730 2019-12-05T05:32:49.226 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.217+0000 7f020ffff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f02100a7730 0x7f021008df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:32:49.226 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.217+0000 7f020ffff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f02100a7730 0x7f021008df70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:36142/0 (socket says 172.21.15.42:36142) 2019-12-05T05:32:49.226 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.217+0000 7f020ffff700 1 -- 172.21.15.42:0/2415832383 learned_addr learned my addr 172.21.15.42:0/2415832383 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:32:49.226 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.217+0000 7f020ffff700 1 -- 172.21.15.42:0/2415832383 >> v1:172.21.15.42:6789/0 conn(0x7f021007e710 legacy=0x7f021009f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:32:49.227 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.217+0000 7f020ffff700 1 -- 172.21.15.42:0/2415832383 >> v1:172.21.15.40:6789/0 conn(0x7f021009fad0 legacy=0x7f021014efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:32:49.227 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.217+0000 7f020ffff700 1 -- 172.21.15.42:0/2415832383 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f021007eb40 con 0x7f02100a7730 2019-12-05T05:32:49.227 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.217+0000 7f020ffff700 1 --2- 172.21.15.42:0/2415832383 >> v2:172.21.15.42:3300/0 conn(0x7f02100a7730 0x7f021008df70 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f0200008790 tx=0x7f020000fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:49.227 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.217+0000 7f020ffff700 1 --2- 172.21.15.42:0/2415832383 >> v2:172.21.15.42:3300/0 conn(0x7f02100a7730 0x7f021008df70 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f0200008790 tx=0x7f020000fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:49.230 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.217+0000 7f020ffff700 1 --2- 172.21.15.42:0/2415832383 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f02100a7730 0x7f021008df70 secure :-1 s=READY pgs=1087 cs=0 l=1 rx=0x7f0200008790 tx=0x7f020000fcb0).ready entity=mon.1 client_cookie=77899b18f8266316 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:32:49.230 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.217+0000 7f020ffff700 1 --2- 172.21.15.42:0/2415832383 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f02100a7730 0x7f021008df70 secure :-1 s=THROTTLE_DONE pgs=1087 cs=0 l=1 rx=0x7f0200008790 tx=0x7f020000fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:49.230 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.217+0000 7f020f7fe700 1 -- 172.21.15.42:0/2415832383 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0200011520 con 0x7f02100a7730 2019-12-05T05:32:49.231 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.217+0000 7f020ffff700 1 --2- 172.21.15.42:0/2415832383 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f02100a7730 0x7f021008df70 secure :-1 s=THROTTLE_DONE pgs=1087 cs=0 l=1 rx=0x7f0200008790 tx=0x7f020000fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:49.231 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.217+0000 7f020ffff700 1 --2- 172.21.15.42:0/2415832383 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f02100a7730 0x7f021008df70 secure :-1 s=THROTTLE_DONE pgs=1087 cs=0 l=1 rx=0x7f0200008790 tx=0x7f020000fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:49.232 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.217+0000 7f020f7fe700 1 -- 172.21.15.42:0/2415832383 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0200008ca0 con 0x7f02100a7730 2019-12-05T05:32:49.232 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.217+0000 7f020f7fe700 1 -- 172.21.15.42:0/2415832383 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0200011b00 con 0x7f02100a7730 2019-12-05T05:32:49.232 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.217+0000 7f021608b700 1 -- 172.21.15.42:0/2415832383 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f02100a7730 msgr2=0x7f021008df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:32:49.233 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.217+0000 7f021608b700 1 --2- 172.21.15.42:0/2415832383 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f02100a7730 0x7f021008df70 secure :-1 s=READY pgs=1087 cs=0 l=1 rx=0x7f0200008790 tx=0x7f020000fcb0).stop 2019-12-05T05:32:49.233 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.217+0000 7f021608b700 1 -- 172.21.15.42:0/2415832383 shutdown_connections 2019-12-05T05:32:49.233 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.217+0000 7f021608b700 1 --2- 172.21.15.42:0/2415832383 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f02100a7730 0x7f021008df70 unknown :-1 s=CLOSED pgs=1087 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:32:49.233 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.217+0000 7f021608b700 1 -- 172.21.15.42:0/2415832383 shutdown_connections 2019-12-05T05:32:49.233 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.217+0000 7f021608b700 1 -- 172.21.15.42:0/2415832383 wait complete. 2019-12-05T05:32:49.234 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.217+0000 7f021608b700 1 -- 172.21.15.42:0/2415832383 >> 172.21.15.42:0/2415832383 conn(0x7f02100aabd0 msgr2=0x7f02100a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:32:49.234 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.221+0000 7f021608b700 1 Processor -- start 2019-12-05T05:32:49.234 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.221+0000 7f021608b700 1 -- start start 2019-12-05T05:32:49.239 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.221+0000 7f021608b700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0210142520 0x7f0210142950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:32:49.239 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.221+0000 7f021608b700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f021012ab60 0x7f021012cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:32:49.240 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.221+0000 7f021608b700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02101455a0 con 0x7f0210059a30 2019-12-05T05:32:49.240 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.221+0000 7f021608b700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0210143980 con 0x7f021012ab60 2019-12-05T05:32:49.240 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.221+0000 7f021608b700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0210143c20 con 0x7f0210142520 2019-12-05T05:32:49.241 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.221+0000 7f0214e29700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f021012ab60 0x7f021012cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:32:49.241 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.221+0000 7f0207fff700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0210142520 0x7f0210142950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:32:49.241 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.221+0000 7f0214e29700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f021012ab60 0x7f021012cfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.42:52200/0 (socket says 172.21.15.42:52200) 2019-12-05T05:32:49.241 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.221+0000 7f0214e29700 1 -- 172.21.15.42:0/3323092222 learned_addr learned my addr 172.21.15.42:0/3323092222 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:32:49.242 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.221+0000 7f0214e29700 1 -- 172.21.15.42:0/3323092222 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0210142520 msgr2=0x7f0210142950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:32:49.242 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.221+0000 7f0214e29700 1 --2- 172.21.15.42:0/3323092222 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0210142520 0x7f0210142950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:32:49.242 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.221+0000 7f0214e29700 1 -- 172.21.15.42:0/3323092222 >> v1:172.21.15.40:6789/0 conn(0x7f0210059a30 legacy=0x7f021007a320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:32:49.242 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.221+0000 7f0214e29700 1 -- 172.21.15.42:0/3323092222 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0210143ec0 con 0x7f021012ab60 2019-12-05T05:32:49.242 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.221+0000 7f0207fff700 1 --2- 172.21.15.42:0/3323092222 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0210142520 0x7f0210142950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:32:49.243 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.221+0000 7f0214e29700 1 --2- 172.21.15.42:0/3323092222 >> v2:172.21.15.40:3300/0 conn(0x7f021012ab60 0x7f021012cfa0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f02100aaf80 tx=0x7f0208010b80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:49.243 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.221+0000 7f0214e29700 1 --2- 172.21.15.42:0/3323092222 >> v2:172.21.15.40:3300/0 conn(0x7f021012ab60 0x7f021012cfa0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f02100aaf80 tx=0x7f0208010b80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:49.243 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.221+0000 7f0214e29700 1 --2- 172.21.15.42:0/3323092222 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f021012ab60 0x7f021012cfa0 secure :-1 s=READY pgs=850 cs=0 l=1 rx=0x7f02100aaf80 tx=0x7f0208010b80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:32:49.243 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.221+0000 7f0214e29700 1 --2- 172.21.15.42:0/3323092222 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f021012ab60 0x7f021012cfa0 secure :-1 s=THROTTLE_DONE pgs=850 cs=0 l=1 rx=0x7f02100aaf80 tx=0x7f0208010b80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:49.243 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.221+0000 7f020d7fa700 1 -- 172.21.15.42:0/3323092222 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0208005c20 con 0x7f021012ab60 2019-12-05T05:32:49.243 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.221+0000 7f0214e29700 1 --2- 172.21.15.42:0/3323092222 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f021012ab60 0x7f021012cfa0 secure :-1 s=THROTTLE_DONE pgs=850 cs=0 l=1 rx=0x7f02100aaf80 tx=0x7f0208010b80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:49.244 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.221+0000 7f0214e29700 1 --2- 172.21.15.42:0/3323092222 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f021012ab60 0x7f021012cfa0 secure :-1 s=THROTTLE_DONE pgs=850 cs=0 l=1 rx=0x7f02100aaf80 tx=0x7f0208010b80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:49.244 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.221+0000 7f020d7fa700 1 -- 172.21.15.42:0/3323092222 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0208005840 con 0x7f021012ab60 2019-12-05T05:32:49.244 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.221+0000 7f020d7fa700 1 -- 172.21.15.42:0/3323092222 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f02080018f0 con 0x7f021012ab60 2019-12-05T05:32:49.244 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.221+0000 7f021608b700 1 -- 172.21.15.42:0/3323092222 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0210142ea0 con 0x7f021012ab60 2019-12-05T05:32:49.244 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.221+0000 7f021608b700 1 -- 172.21.15.42:0/3323092222 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02100a7080 con 0x7f021012ab60 2019-12-05T05:32:49.244 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.225+0000 7f0214e29700 1 --2- 172.21.15.42:0/3323092222 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f021012ab60 0x7f021012cfa0 secure :-1 s=THROTTLE_DONE pgs=850 cs=0 l=1 rx=0x7f02100aaf80 tx=0x7f0208010b80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:49.245 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.225+0000 7f0214e29700 1 --2- 172.21.15.42:0/3323092222 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f021012ab60 0x7f021012cfa0 secure :-1 s=THROTTLE_DONE pgs=850 cs=0 l=1 rx=0x7f02100aaf80 tx=0x7f0208010b80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:49.245 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.225+0000 7f020d7fa700 1 -- 172.21.15.42:0/3323092222 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f0208001bd0 con 0x7f021012ab60 2019-12-05T05:32:49.245 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.225+0000 7f020d7fa700 1 --2- 172.21.15.42:0/3323092222 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f01f403aca0 0x7f01f403d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:32:49.245 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.225+0000 7f020d7fa700 1 -- 172.21.15.42:0/3323092222 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 7837+0+0 (secure 0 0 0) 0x7f02080020c0 con 0x7f021012ab60 2019-12-05T05:32:49.245 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.225+0000 7f020ffff700 1 --2- 172.21.15.42:0/3323092222 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f01f403aca0 0x7f01f403d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:32:49.245 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.225+0000 7f020ffff700 1 --2- 172.21.15.42:0/3323092222 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f01f403aca0 0x7f01f403d1b0 crc :-1 s=READY pgs=466 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:32:49.245 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.229+0000 7f021608b700 1 -- 172.21.15.42:0/3323092222 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f021005d100 con 0x7f021012ab60 2019-12-05T05:32:49.246 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.241+0000 7f0214e29700 1 --2- 172.21.15.42:0/3323092222 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f021012ab60 0x7f021012cfa0 secure :-1 s=THROTTLE_DONE pgs=850 cs=0 l=1 rx=0x7f02100aaf80 tx=0x7f0208010b80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:49.246 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.241+0000 7f020d7fa700 1 -- 172.21.15.42:0/3323092222 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f0208032270 con 0x7f021012ab60 2019-12-05T05:32:49.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.405+0000 7f021608b700 1 -- 172.21.15.42:0/3323092222 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f021005b8b0 con 0x7f021012ab60 2019-12-05T05:32:49.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.405+0000 7f0214e29700 1 --2- 172.21.15.42:0/3323092222 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f021012ab60 0x7f021012cfa0 secure :-1 s=THROTTLE_DONE pgs=850 cs=0 l=1 rx=0x7f02100aaf80 tx=0x7f0208010b80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:49.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.405+0000 7f020d7fa700 1 -- 172.21.15.42:0/3323092222 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 134 v134) v1 ==== 95+0+1774 (secure 0 0 0) 0x7f02080324d0 con 0x7f021012ab60 2019-12-05T05:32:49.411 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 134 2019-12-05T05:32:49.412 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:32:49.412 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":134,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":134,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:32:48.730996+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6997},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6997":{"gid":6997,"name":"a","rank":0,"incarnation":134,"state":"up:replay","state_seq":2,"addr":"172.21.15.42:6835/1601334502","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1601334502},{"type":"v1","addr":"172.21.15.42:6835","nonce":1601334502}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:32:49.414 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.409+0000 7f021608b700 1 -- 172.21.15.42:0/3323092222 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f01f403aca0 msgr2=0x7f01f403d1b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:32:49.414 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.409+0000 7f021608b700 1 --2- 172.21.15.42:0/3323092222 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f01f403aca0 0x7f01f403d1b0 crc :-1 s=READY pgs=466 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:32:49.414 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.409+0000 7f021608b700 1 -- 172.21.15.42:0/3323092222 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f021012ab60 msgr2=0x7f021012cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:32:49.414 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.409+0000 7f021608b700 1 --2- 172.21.15.42:0/3323092222 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f021012ab60 0x7f021012cfa0 secure :-1 s=READY pgs=850 cs=0 l=1 rx=0x7f02100aaf80 tx=0x7f0208010b80).stop 2019-12-05T05:32:49.415 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.409+0000 7f021608b700 1 -- 172.21.15.42:0/3323092222 shutdown_connections 2019-12-05T05:32:49.415 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.409+0000 7f021608b700 1 --2- 172.21.15.42:0/3323092222 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f01f403aca0 0x7f01f403d1b0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:32:49.415 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.409+0000 7f021608b700 1 --2- 172.21.15.42:0/3323092222 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f021012ab60 0x7f021012cfa0 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:32:49.415 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.409+0000 7f021608b700 1 --2- 172.21.15.42:0/3323092222 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0210142520 0x7f0210142950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:32:49.416 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.409+0000 7f021608b700 1 -- 172.21.15.42:0/3323092222 shutdown_connections 2019-12-05T05:32:49.416 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.409+0000 7f021608b700 1 -- 172.21.15.42:0/3323092222 wait complete. 2019-12-05T05:32:49.416 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.409+0000 7f021608b700 1 -- 172.21.15.42:0/3323092222 >> 172.21.15.42:0/3323092222 conn(0x7f021009ffa0 msgr2=0x7f021008f350 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:32:49.425 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b down, removed from mdsmap 2019-12-05T05:32:49.425 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2019-12-05T05:32:49.426 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:32:49.526 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.517+0000 7f30add32700 1 Processor -- start 2019-12-05T05:32:49.526 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.517+0000 7f30add32700 1 -- start start 2019-12-05T05:32:49.526 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.517+0000 7f30add32700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f30a80a7740 0x7f30a808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:32:49.526 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.517+0000 7f30add32700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f30a809fae0 0x7f30a807eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:32:49.526 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.517+0000 7f30add32700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30a80a0710 con 0x7f30a807e720 2019-12-05T05:32:49.527 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.517+0000 7f30add32700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30a80a0b00 con 0x7f30a809fae0 2019-12-05T05:32:49.527 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.517+0000 7f30add32700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30a808e5b0 con 0x7f30a80a7740 2019-12-05T05:32:49.527 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.517+0000 7f30a7fff700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f30a80a7740 0x7f30a808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:32:49.527 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.517+0000 7f30acad0700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f30a809fae0 0x7f30a807eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:32:49.527 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.517+0000 7f30a7fff700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f30a80a7740 0x7f30a808df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:36622/0 (socket says 172.21.15.42:36622) 2019-12-05T05:32:49.527 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.517+0000 7f30a7fff700 1 -- 172.21.15.42:0/197747571 learned_addr learned my addr 172.21.15.42:0/197747571 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:32:49.527 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.521+0000 7f30a7fff700 1 -- 172.21.15.42:0/197747571 >> v2:172.21.15.40:3300/0 conn(0x7f30a809fae0 msgr2=0x7f30a807eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:32:49.528 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.521+0000 7f30a7fff700 1 --2- 172.21.15.42:0/197747571 >> v2:172.21.15.40:3300/0 conn(0x7f30a809fae0 0x7f30a807eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:32:49.528 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.521+0000 7f30a7fff700 1 -- 172.21.15.42:0/197747571 >> v1:172.21.15.40:6789/0 conn(0x7f30a807e720 legacy=0x7f30a809f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:32:49.528 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.521+0000 7f30a7fff700 1 -- 172.21.15.42:0/197747571 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30a807f0a0 con 0x7f30a80a7740 2019-12-05T05:32:49.528 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.521+0000 7f30acad0700 1 --2- 172.21.15.42:0/197747571 >> v2:172.21.15.40:3300/0 conn(0x7f30a809fae0 0x7f30a807eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:32:49.528 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.521+0000 7f30a7fff700 1 --2- 172.21.15.42:0/197747571 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f30a80a7740 0x7f30a808df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f3098008790 tx=0x7f309800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:49.528 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.521+0000 7f30a7fff700 1 --2- 172.21.15.42:0/197747571 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f30a80a7740 0x7f30a808df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f3098008790 tx=0x7f309800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:49.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.521+0000 7f30a7fff700 1 --2- 172.21.15.42:0/197747571 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f30a80a7740 0x7f30a808df80 secure :-1 s=READY pgs=224 cs=0 l=1 rx=0x7f3098008790 tx=0x7f309800fd30).ready entity=mon.2 client_cookie=902f0be140b03320 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:32:49.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.521+0000 7f30a7fff700 1 --2- 172.21.15.42:0/197747571 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f30a80a7740 0x7f30a808df80 secure :-1 s=THROTTLE_DONE pgs=224 cs=0 l=1 rx=0x7f3098008790 tx=0x7f309800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:49.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.521+0000 7f30a6ffd700 1 -- 172.21.15.42:0/197747571 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3098008c90 con 0x7f30a80a7740 2019-12-05T05:32:49.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.521+0000 7f30a7fff700 1 --2- 172.21.15.42:0/197747571 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f30a80a7740 0x7f30a808df80 secure :-1 s=THROTTLE_DONE pgs=224 cs=0 l=1 rx=0x7f3098008790 tx=0x7f309800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:49.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.521+0000 7f30a6ffd700 1 -- 172.21.15.42:0/197747571 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30980088b0 con 0x7f30a80a7740 2019-12-05T05:32:49.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.521+0000 7f30a7fff700 1 --2- 172.21.15.42:0/197747571 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f30a80a7740 0x7f30a808df80 secure :-1 s=THROTTLE_DONE pgs=224 cs=0 l=1 rx=0x7f3098008790 tx=0x7f309800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:49.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.521+0000 7f30a6ffd700 1 -- 172.21.15.42:0/197747571 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3098011b10 con 0x7f30a80a7740 2019-12-05T05:32:49.530 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.521+0000 7f30add32700 1 -- 172.21.15.42:0/197747571 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f30a80a7740 msgr2=0x7f30a808df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:32:49.530 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.521+0000 7f30add32700 1 --2- 172.21.15.42:0/197747571 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f30a80a7740 0x7f30a808df80 secure :-1 s=READY pgs=224 cs=0 l=1 rx=0x7f3098008790 tx=0x7f309800fd30).stop 2019-12-05T05:32:49.530 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.521+0000 7f30add32700 1 -- 172.21.15.42:0/197747571 shutdown_connections 2019-12-05T05:32:49.530 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.521+0000 7f30add32700 1 --2- 172.21.15.42:0/197747571 >> v2:172.21.15.40:3300/0 conn(0x7f30a809fae0 0x7f30a807eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:32:49.531 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.521+0000 7f30add32700 1 --2- 172.21.15.42:0/197747571 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f30a80a7740 0x7f30a808df80 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:32:49.531 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.521+0000 7f30add32700 1 -- 172.21.15.42:0/197747571 shutdown_connections 2019-12-05T05:32:49.531 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.521+0000 7f30add32700 1 -- 172.21.15.42:0/197747571 wait complete. 2019-12-05T05:32:49.531 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.521+0000 7f30add32700 1 -- 172.21.15.42:0/197747571 >> 172.21.15.42:0/197747571 conn(0x7f30a80aabe0 msgr2=0x7f30a80a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:32:49.549 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.521+0000 7f30add32700 1 Processor -- start 2019-12-05T05:32:49.549 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.521+0000 7f30add32700 1 -- start start 2019-12-05T05:32:49.549 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.525+0000 7f30add32700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f30a80a3ef0 0x7f30a8144e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:32:49.549 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.525+0000 7f30add32700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f30a8145380 0x7f30a80a14d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:32:49.550 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.525+0000 7f30add32700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30a80a1a20 con 0x7f30a80a2970 2019-12-05T05:32:49.550 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.525+0000 7f30add32700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30a80a2190 con 0x7f30a80a3ef0 2019-12-05T05:32:49.550 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.525+0000 7f30add32700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30a80a2430 con 0x7f30a8145380 2019-12-05T05:32:49.550 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.525+0000 7f30a7fff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f30a80a2970 0x7f30a812c680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:36200/0 (socket says 172.21.15.42:36200) 2019-12-05T05:32:49.550 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.525+0000 7f30a77fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f30a80a3ef0 0x7f30a8144e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:32:49.550 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.525+0000 7f30a7fff700 1 -- 172.21.15.42:0/3312895079 learned_addr learned my addr 172.21.15.42:0/3312895079 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:32:49.551 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.525+0000 7f30acad0700 1 --2- 172.21.15.42:0/3312895079 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f30a8145380 0x7f30a80a14d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:32:49.551 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.525+0000 7f30a77fe700 1 -- 172.21.15.42:0/3312895079 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f30a8145380 msgr2=0x7f30a80a14d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:32:49.551 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.525+0000 7f30a77fe700 1 --2- 172.21.15.42:0/3312895079 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f30a8145380 0x7f30a80a14d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:32:49.551 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.525+0000 7f30a77fe700 1 -- 172.21.15.42:0/3312895079 >> v1:172.21.15.42:6789/0 conn(0x7f30a80a2970 legacy=0x7f30a812c680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:32:49.551 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.525+0000 7f30a4ff9700 1 -- 172.21.15.42:0/3312895079 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1286078669 0 0) 0x7f30a80a1a20 con 0x7f30a80a2970 2019-12-05T05:32:49.551 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.525+0000 7f30a77fe700 1 -- 172.21.15.42:0/3312895079 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30a80a1c50 con 0x7f30a80a3ef0 2019-12-05T05:32:49.551 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.525+0000 7f30a77fe700 1 --2- 172.21.15.42:0/3312895079 >> v2:172.21.15.42:3300/0 conn(0x7f30a80a3ef0 0x7f30a8144e30 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f30a8000b10 tx=0x7f309c0109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:49.552 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.525+0000 7f30a77fe700 1 --2- 172.21.15.42:0/3312895079 >> v2:172.21.15.42:3300/0 conn(0x7f30a80a3ef0 0x7f30a8144e30 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f30a8000b10 tx=0x7f309c0109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:49.552 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.525+0000 7f30a77fe700 1 --2- 172.21.15.42:0/3312895079 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f30a80a3ef0 0x7f30a8144e30 secure :-1 s=READY pgs=1089 cs=0 l=1 rx=0x7f30a8000b10 tx=0x7f309c0109a0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:32:49.553 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.525+0000 7f30a77fe700 1 --2- 172.21.15.42:0/3312895079 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f30a80a3ef0 0x7f30a8144e30 secure :-1 s=THROTTLE_DONE pgs=1089 cs=0 l=1 rx=0x7f30a8000b10 tx=0x7f309c0109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:49.553 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.525+0000 7f30a4ff9700 1 -- 172.21.15.42:0/3312895079 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f309c00caa0 con 0x7f30a80a3ef0 2019-12-05T05:32:49.553 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.525+0000 7f30a77fe700 1 --2- 172.21.15.42:0/3312895079 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f30a80a3ef0 0x7f30a8144e30 secure :-1 s=THROTTLE_DONE pgs=1089 cs=0 l=1 rx=0x7f30a8000b10 tx=0x7f309c0109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:49.553 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.525+0000 7f30a77fe700 1 --2- 172.21.15.42:0/3312895079 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f30a80a3ef0 0x7f30a8144e30 secure :-1 s=THROTTLE_DONE pgs=1089 cs=0 l=1 rx=0x7f30a8000b10 tx=0x7f309c0109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:49.553 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.525+0000 7f30a4ff9700 1 -- 172.21.15.42:0/3312895079 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f309c00c6c0 con 0x7f30a80a3ef0 2019-12-05T05:32:49.553 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.525+0000 7f30a4ff9700 1 -- 172.21.15.42:0/3312895079 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f309c011120 con 0x7f30a80a3ef0 2019-12-05T05:32:49.554 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.525+0000 7f30add32700 1 -- 172.21.15.42:0/3312895079 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30a80a9790 con 0x7f30a80a3ef0 2019-12-05T05:32:49.554 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.525+0000 7f30add32700 1 -- 172.21.15.42:0/3312895079 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30a80a9c60 con 0x7f30a80a3ef0 2019-12-05T05:32:49.554 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.525+0000 7f30a77fe700 1 --2- 172.21.15.42:0/3312895079 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f30a80a3ef0 0x7f30a8144e30 secure :-1 s=THROTTLE_DONE pgs=1089 cs=0 l=1 rx=0x7f30a8000b10 tx=0x7f309c0109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:49.555 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.525+0000 7f30a4ff9700 1 -- 172.21.15.42:0/3312895079 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f309c011430 con 0x7f30a80a3ef0 2019-12-05T05:32:49.555 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.525+0000 7f30a77fe700 1 --2- 172.21.15.42:0/3312895079 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f30a80a3ef0 0x7f30a8144e30 secure :-1 s=THROTTLE_DONE pgs=1089 cs=0 l=1 rx=0x7f30a8000b10 tx=0x7f309c0109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:49.555 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.525+0000 7f30a4ff9700 1 --2- 172.21.15.42:0/3312895079 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f308803acc0 0x7f308803d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:32:49.555 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.525+0000 7f30a4ff9700 1 -- 172.21.15.42:0/3312895079 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 7837+0+0 (secure 0 0 0) 0x7f309c04b230 con 0x7f30a80a3ef0 2019-12-05T05:32:49.555 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.529+0000 7f30a7fff700 1 --2- 172.21.15.42:0/3312895079 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f308803acc0 0x7f308803d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:32:49.555 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.529+0000 7f30a7fff700 1 --2- 172.21.15.42:0/3312895079 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f308803acc0 0x7f308803d1d0 crc :-1 s=READY pgs=467 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:32:49.556 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.533+0000 7f30add32700 1 -- 172.21.15.42:0/3312895079 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30a805d110 con 0x7f30a80a3ef0 2019-12-05T05:32:49.556 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.537+0000 7f30a77fe700 1 --2- 172.21.15.42:0/3312895079 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f30a80a3ef0 0x7f30a8144e30 secure :-1 s=THROTTLE_DONE pgs=1089 cs=0 l=1 rx=0x7f30a8000b10 tx=0x7f309c0109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:49.556 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.537+0000 7f30a4ff9700 1 -- 172.21.15.42:0/3312895079 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f309c0322b0 con 0x7f30a80a3ef0 2019-12-05T05:32:49.695 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.693+0000 7f30add32700 1 -- 172.21.15.42:0/3312895079 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f30a805b8c0 con 0x7f30a80a3ef0 2019-12-05T05:32:49.696 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.693+0000 7f30a77fe700 1 --2- 172.21.15.42:0/3312895079 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f30a80a3ef0 0x7f30a8144e30 secure :-1 s=THROTTLE_DONE pgs=1089 cs=0 l=1 rx=0x7f30a8000b10 tx=0x7f309c0109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:49.696 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.693+0000 7f30a4ff9700 1 -- 172.21.15.42:0/3312895079 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 134 v134) v1 ==== 95+0+1774 (secure 0 0 0) 0x7f309c032510 con 0x7f30a80a3ef0 2019-12-05T05:32:49.697 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 134 2019-12-05T05:32:49.697 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:32:49.697 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":134,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":134,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:32:48.730996+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6997},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6997":{"gid":6997,"name":"a","rank":0,"incarnation":134,"state":"up:replay","state_seq":2,"addr":"172.21.15.42:6835/1601334502","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1601334502},{"type":"v1","addr":"172.21.15.42:6835","nonce":1601334502}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:32:49.699 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.693+0000 7f30add32700 1 -- 172.21.15.42:0/3312895079 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f308803acc0 msgr2=0x7f308803d1d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:32:49.699 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.693+0000 7f30add32700 1 --2- 172.21.15.42:0/3312895079 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f308803acc0 0x7f308803d1d0 crc :-1 s=READY pgs=467 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:32:49.699 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.693+0000 7f30add32700 1 -- 172.21.15.42:0/3312895079 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f30a80a3ef0 msgr2=0x7f30a8144e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:32:49.700 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.693+0000 7f30add32700 1 --2- 172.21.15.42:0/3312895079 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f30a80a3ef0 0x7f30a8144e30 secure :-1 s=READY pgs=1089 cs=0 l=1 rx=0x7f30a8000b10 tx=0x7f309c0109a0).stop 2019-12-05T05:32:49.700 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.693+0000 7f30add32700 1 -- 172.21.15.42:0/3312895079 shutdown_connections 2019-12-05T05:32:49.700 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.693+0000 7f30add32700 1 --2- 172.21.15.42:0/3312895079 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f308803acc0 0x7f308803d1d0 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:32:49.700 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.693+0000 7f30add32700 1 --2- 172.21.15.42:0/3312895079 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f30a8145380 0x7f30a80a14d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:32:49.701 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.693+0000 7f30add32700 1 --2- 172.21.15.42:0/3312895079 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f30a80a3ef0 0x7f30a8144e30 unknown :-1 s=CLOSED pgs=1089 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:32:49.701 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.693+0000 7f30add32700 1 -- 172.21.15.42:0/3312895079 shutdown_connections 2019-12-05T05:32:49.701 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.693+0000 7f30add32700 1 -- 172.21.15.42:0/3312895079 wait complete. 2019-12-05T05:32:49.702 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:49.693+0000 7f30add32700 1 -- 172.21.15.42:0/3312895079 >> 172.21.15.42:0/3312895079 conn(0x7f30a809f5f0 msgr2=0x7f30a808f0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:32:49.709 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 134, "feature_flags": { "enable_multiple": false, "ever_enabled_multiple": false }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2019-12-05T05:03:35.604738+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 134, "ever_allowed_features": 2, "explicitly_allowed_features": 2, "failed": [], "flags": 18, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_6997": { "addr": "172.21.15.42:6835/1601334502", "addrs": { "addrvec": [ { "addr": "172.21.15.42:6834", "nonce": 1601334502, "type": "v2" }, { "addr": "172.21.15.42:6835", "nonce": 1601334502, "type": "v1" } ] }, "export_targets": [], "features": 4540138292836696063, "flags": 0, "gid": 6997, "incarnation": 134, "name": "a", "rank": 0, "state": "up:replay", "state_seq": 2 } }, "last_failure": 0, "last_failure_osd_epoch": 70, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 2, "min_compat_client": "0 (unknown)", "modified": "2019-12-05T05:32:48.730996+0000", "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6997 } } } ], "standbys": [] } 2019-12-05T05:32:51.709 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:32:51.816 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:51.805+0000 7f5e653c8700 1 Processor -- start 2019-12-05T05:32:51.816 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:51.805+0000 7f5e653c8700 1 -- start start 2019-12-05T05:32:51.816 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:51.809+0000 7f5e653c8700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5e600a7740 0x7f5e6008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:32:51.816 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:51.809+0000 7f5e653c8700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5e6009fae0 0x7f5e6007eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:32:51.817 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:51.809+0000 7f5e653c8700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e600a0710 con 0x7f5e6007e720 2019-12-05T05:32:51.817 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:51.809+0000 7f5e653c8700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e600a0b00 con 0x7f5e6009fae0 2019-12-05T05:32:51.817 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:51.809+0000 7f5e653c8700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e6008e5b0 con 0x7f5e600a7740 2019-12-05T05:32:51.817 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:51.809+0000 7f5e5f7fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5e600a7740 0x7f5e6008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:32:51.818 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:51.809+0000 7f5e5f7fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5e600a7740 0x7f5e6008df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:36174/0 (socket says 172.21.15.42:36174) 2019-12-05T05:32:51.818 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:51.809+0000 7f5e5f7fe700 1 -- 172.21.15.42:0/4221957267 learned_addr learned my addr 172.21.15.42:0/4221957267 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:32:51.819 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:51.809+0000 7f5e5ffff700 1 --2- 172.21.15.42:0/4221957267 >> v2:172.21.15.40:3300/0 conn(0x7f5e6009fae0 0x7f5e6007eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:32:51.819 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:51.809+0000 7f5e5f7fe700 1 -- 172.21.15.42:0/4221957267 >> v2:172.21.15.40:3300/0 conn(0x7f5e6009fae0 msgr2=0x7f5e6007eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:32:51.819 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:51.809+0000 7f5e5f7fe700 1 --2- 172.21.15.42:0/4221957267 >> v2:172.21.15.40:3300/0 conn(0x7f5e6009fae0 0x7f5e6007eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:32:51.819 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:51.809+0000 7f5e5f7fe700 1 -- 172.21.15.42:0/4221957267 >> v1:172.21.15.40:6789/0 conn(0x7f5e6007e720 legacy=0x7f5e6009f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:32:51.819 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:51.809+0000 7f5e5f7fe700 1 -- 172.21.15.42:0/4221957267 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e6007f0a0 con 0x7f5e600a7740 2019-12-05T05:32:51.819 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:51.809+0000 7f5e5f7fe700 1 --2- 172.21.15.42:0/4221957267 >> v2:172.21.15.42:3300/0 conn(0x7f5e600a7740 0x7f5e6008df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f5e500032b0 tx=0x7f5e500100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:51.819 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:51.809+0000 7f5e5f7fe700 1 --2- 172.21.15.42:0/4221957267 >> v2:172.21.15.42:3300/0 conn(0x7f5e600a7740 0x7f5e6008df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f5e500032b0 tx=0x7f5e500100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:51.820 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:51.809+0000 7f5e5f7fe700 1 --2- 172.21.15.42:0/4221957267 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5e600a7740 0x7f5e6008df80 secure :-1 s=READY pgs=1090 cs=0 l=1 rx=0x7f5e500032b0 tx=0x7f5e500100b0).ready entity=mon.1 client_cookie=96f1fa3df502764c server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:32:51.820 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:51.809+0000 7f5e5f7fe700 1 --2- 172.21.15.42:0/4221957267 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5e600a7740 0x7f5e6008df80 secure :-1 s=THROTTLE_DONE pgs=1090 cs=0 l=1 rx=0x7f5e500032b0 tx=0x7f5e500100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:51.820 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:51.809+0000 7f5e5e7fc700 1 -- 172.21.15.42:0/4221957267 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5e50010d60 con 0x7f5e600a7740 2019-12-05T05:32:51.820 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:51.809+0000 7f5e5f7fe700 1 --2- 172.21.15.42:0/4221957267 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5e600a7740 0x7f5e6008df80 secure :-1 s=THROTTLE_DONE pgs=1090 cs=0 l=1 rx=0x7f5e500032b0 tx=0x7f5e500100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:51.820 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:51.809+0000 7f5e5f7fe700 1 --2- 172.21.15.42:0/4221957267 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5e600a7740 0x7f5e6008df80 secure :-1 s=THROTTLE_DONE pgs=1090 cs=0 l=1 rx=0x7f5e500032b0 tx=0x7f5e500100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:51.820 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:51.809+0000 7f5e5e7fc700 1 -- 172.21.15.42:0/4221957267 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e50010980 con 0x7f5e600a7740 2019-12-05T05:32:51.821 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:51.809+0000 7f5e5e7fc700 1 -- 172.21.15.42:0/4221957267 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5e500114a0 con 0x7f5e600a7740 2019-12-05T05:32:51.821 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:51.809+0000 7f5e653c8700 1 -- 172.21.15.42:0/4221957267 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5e600a7740 msgr2=0x7f5e6008df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:32:51.821 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:51.809+0000 7f5e653c8700 1 --2- 172.21.15.42:0/4221957267 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5e600a7740 0x7f5e6008df80 secure :-1 s=READY pgs=1090 cs=0 l=1 rx=0x7f5e500032b0 tx=0x7f5e500100b0).stop 2019-12-05T05:32:51.821 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:51.809+0000 7f5e653c8700 1 -- 172.21.15.42:0/4221957267 shutdown_connections 2019-12-05T05:32:51.821 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:51.809+0000 7f5e653c8700 1 --2- 172.21.15.42:0/4221957267 >> v2:172.21.15.40:3300/0 conn(0x7f5e6009fae0 0x7f5e6007eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:32:51.821 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:51.809+0000 7f5e653c8700 1 --2- 172.21.15.42:0/4221957267 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5e600a7740 0x7f5e6008df80 unknown :-1 s=CLOSED pgs=1090 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:32:51.822 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:51.813+0000 7f5e653c8700 1 -- 172.21.15.42:0/4221957267 shutdown_connections 2019-12-05T05:32:51.822 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:51.813+0000 7f5e653c8700 1 -- 172.21.15.42:0/4221957267 wait complete. 2019-12-05T05:32:51.822 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:51.813+0000 7f5e653c8700 1 -- 172.21.15.42:0/4221957267 >> 172.21.15.42:0/4221957267 conn(0x7f5e600aabe0 msgr2=0x7f5e600a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:32:51.825 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:51.813+0000 7f5e653c8700 1 Processor -- start 2019-12-05T05:32:51.825 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:51.813+0000 7f5e653c8700 1 -- start start 2019-12-05T05:32:51.826 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:51.817+0000 7f5e653c8700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5e600a2be0 0x7f5e600a3010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:32:51.826 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:51.817+0000 7f5e653c8700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5e600a51e0 0x7f5e600a5610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:32:51.826 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:51.817+0000 7f5e653c8700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e6007a750 con 0x7f5e60123c30 2019-12-05T05:32:51.826 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:51.817+0000 7f5e653c8700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e60124460 con 0x7f5e600a51e0 2019-12-05T05:32:51.826 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:51.817+0000 7f5e653c8700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e600a4f40 con 0x7f5e600a2be0 2019-12-05T05:32:51.827 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:51.817+0000 7f5e5effd700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5e600a2be0 0x7f5e600a3010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:32:51.827 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:51.817+0000 7f5e5ffff700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5e600a51e0 0x7f5e600a5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:32:51.827 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:51.817+0000 7f5e5f7fe700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f5e60123c30 0x7f5e60079f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:45658/0 (socket says 172.21.15.42:45658) 2019-12-05T05:32:51.827 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:51.817+0000 7f5e5effd700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5e600a2be0 0x7f5e600a3010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:36648/0 (socket says 172.21.15.42:36648) 2019-12-05T05:32:51.827 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:51.817+0000 7f5e5f7fe700 1 -- 172.21.15.42:0/1580461358 learned_addr learned my addr 172.21.15.42:0/1580461358 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:32:51.827 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:51.817+0000 7f5e5ffff700 1 -- 172.21.15.42:0/1580461358 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5e600a2be0 msgr2=0x7f5e600a3010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:32:51.827 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:51.817+0000 7f5e5ffff700 1 --2- 172.21.15.42:0/1580461358 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5e600a2be0 0x7f5e600a3010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:32:51.828 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:51.817+0000 7f5e47fff700 1 -- 172.21.15.42:0/1580461358 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1610199576 0 0) 0x7f5e6007a750 con 0x7f5e60123c30 2019-12-05T05:32:51.828 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:51.817+0000 7f5e5ffff700 1 -- 172.21.15.42:0/1580461358 >> v1:172.21.15.40:6789/0 conn(0x7f5e60123c30 legacy=0x7f5e60079f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:32:51.829 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:51.817+0000 7f5e5ffff700 1 -- 172.21.15.42:0/1580461358 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e60124060 con 0x7f5e600a51e0 2019-12-05T05:32:51.829 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:51.817+0000 7f5e5effd700 1 --2- 172.21.15.42:0/1580461358 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5e600a2be0 0x7f5e600a3010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:32:51.829 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:51.817+0000 7f5e5ffff700 1 --2- 172.21.15.42:0/1580461358 >> v2:172.21.15.40:3300/0 conn(0x7f5e600a51e0 0x7f5e600a5610 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f5e600a6da0 tx=0x7f5e58010920).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:51.829 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:51.817+0000 7f5e5ffff700 1 --2- 172.21.15.42:0/1580461358 >> v2:172.21.15.40:3300/0 conn(0x7f5e600a51e0 0x7f5e600a5610 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f5e600a6da0 tx=0x7f5e58010920).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:51.830 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:51.817+0000 7f5e5ffff700 1 --2- 172.21.15.42:0/1580461358 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5e600a51e0 0x7f5e600a5610 secure :-1 s=READY pgs=854 cs=0 l=1 rx=0x7f5e600a6da0 tx=0x7f5e58010920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:32:51.830 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:51.817+0000 7f5e5ffff700 1 --2- 172.21.15.42:0/1580461358 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5e600a51e0 0x7f5e600a5610 secure :-1 s=THROTTLE_DONE pgs=854 cs=0 l=1 rx=0x7f5e600a6da0 tx=0x7f5e58010920).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:51.830 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:51.817+0000 7f5e5ffff700 1 --2- 172.21.15.42:0/1580461358 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5e600a51e0 0x7f5e600a5610 secure :-1 s=THROTTLE_DONE pgs=854 cs=0 l=1 rx=0x7f5e600a6da0 tx=0x7f5e58010920).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:51.830 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:51.817+0000 7f5e47fff700 1 -- 172.21.15.42:0/1580461358 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5e58001660 con 0x7f5e600a51e0 2019-12-05T05:32:51.830 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:51.817+0000 7f5e5ffff700 1 --2- 172.21.15.42:0/1580461358 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5e600a51e0 0x7f5e600a5610 secure :-1 s=THROTTLE_DONE pgs=854 cs=0 l=1 rx=0x7f5e600a6da0 tx=0x7f5e58010920).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:51.830 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:51.817+0000 7f5e47fff700 1 -- 172.21.15.42:0/1580461358 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e58001280 con 0x7f5e600a51e0 2019-12-05T05:32:51.830 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:51.817+0000 7f5e653c8700 1 -- 172.21.15.42:0/1580461358 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e600a5b60 con 0x7f5e600a51e0 2019-12-05T05:32:51.831 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:51.817+0000 7f5e47fff700 1 -- 172.21.15.42:0/1580461358 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5e58001c40 con 0x7f5e600a51e0 2019-12-05T05:32:51.831 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:51.817+0000 7f5e653c8700 1 -- 172.21.15.42:0/1580461358 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e6012e2d0 con 0x7f5e600a51e0 2019-12-05T05:32:51.831 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:51.817+0000 7f5e5ffff700 1 --2- 172.21.15.42:0/1580461358 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5e600a51e0 0x7f5e600a5610 secure :-1 s=THROTTLE_DONE pgs=854 cs=0 l=1 rx=0x7f5e600a6da0 tx=0x7f5e58010920).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:51.831 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:51.821+0000 7f5e47fff700 1 -- 172.21.15.42:0/1580461358 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f5e58005540 con 0x7f5e600a51e0 2019-12-05T05:32:51.831 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:51.821+0000 7f5e5ffff700 1 --2- 172.21.15.42:0/1580461358 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5e600a51e0 0x7f5e600a5610 secure :-1 s=THROTTLE_DONE pgs=854 cs=0 l=1 rx=0x7f5e600a6da0 tx=0x7f5e58010920).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:51.831 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:51.821+0000 7f5e47fff700 1 --2- 172.21.15.42:0/1580461358 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5e4003ac70 0x7f5e4003d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:32:51.832 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:51.821+0000 7f5e47fff700 1 -- 172.21.15.42:0/1580461358 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 7837+0+0 (secure 0 0 0) 0x7f5e5804adb0 con 0x7f5e600a51e0 2019-12-05T05:32:51.832 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:51.821+0000 7f5e5f7fe700 1 --2- 172.21.15.42:0/1580461358 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5e4003ac70 0x7f5e4003d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:32:51.832 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:51.821+0000 7f5e5f7fe700 1 --2- 172.21.15.42:0/1580461358 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5e4003ac70 0x7f5e4003d180 crc :-1 s=READY pgs=468 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:32:51.833 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:51.821+0000 7f5e653c8700 1 -- 172.21.15.42:0/1580461358 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e6005d110 con 0x7f5e600a51e0 2019-12-05T05:32:51.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:51.833+0000 7f5e5ffff700 1 --2- 172.21.15.42:0/1580461358 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5e600a51e0 0x7f5e600a5610 secure :-1 s=THROTTLE_DONE pgs=854 cs=0 l=1 rx=0x7f5e600a6da0 tx=0x7f5e58010920).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:51.837 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:51.833+0000 7f5e47fff700 1 -- 172.21.15.42:0/1580461358 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f5e58031e30 con 0x7f5e600a51e0 2019-12-05T05:32:52.006 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:52.001+0000 7f5e653c8700 1 -- 172.21.15.42:0/1580461358 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5e6005b8c0 con 0x7f5e600a51e0 2019-12-05T05:32:52.007 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:52.001+0000 7f5e5ffff700 1 --2- 172.21.15.42:0/1580461358 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5e600a51e0 0x7f5e600a5610 secure :-1 s=THROTTLE_DONE pgs=854 cs=0 l=1 rx=0x7f5e600a6da0 tx=0x7f5e58010920).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:52.007 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:52.001+0000 7f5e47fff700 1 -- 172.21.15.42:0/1580461358 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 135 v135) v1 ==== 95+0+1778 (secure 0 0 0) 0x7f5e58032090 con 0x7f5e600a51e0 2019-12-05T05:32:52.009 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 135 2019-12-05T05:32:52.009 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:32:52.009 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":135,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":135,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:32:51.755654+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6997},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6997":{"gid":6997,"name":"a","rank":0,"incarnation":134,"state":"up:reconnect","state_seq":15,"addr":"172.21.15.42:6835/1601334502","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1601334502},{"type":"v1","addr":"172.21.15.42:6835","nonce":1601334502}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:32:52.012 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:52.005+0000 7f5e653c8700 1 -- 172.21.15.42:0/1580461358 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5e4003ac70 msgr2=0x7f5e4003d180 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:32:52.012 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:52.005+0000 7f5e653c8700 1 --2- 172.21.15.42:0/1580461358 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5e4003ac70 0x7f5e4003d180 crc :-1 s=READY pgs=468 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:32:52.012 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:52.005+0000 7f5e653c8700 1 -- 172.21.15.42:0/1580461358 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5e600a51e0 msgr2=0x7f5e600a5610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:32:52.012 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:52.005+0000 7f5e653c8700 1 --2- 172.21.15.42:0/1580461358 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5e600a51e0 0x7f5e600a5610 secure :-1 s=READY pgs=854 cs=0 l=1 rx=0x7f5e600a6da0 tx=0x7f5e58010920).stop 2019-12-05T05:32:52.012 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:52.005+0000 7f5e653c8700 1 -- 172.21.15.42:0/1580461358 shutdown_connections 2019-12-05T05:32:52.012 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:52.005+0000 7f5e653c8700 1 --2- 172.21.15.42:0/1580461358 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5e4003ac70 0x7f5e4003d180 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:32:52.013 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:52.005+0000 7f5e653c8700 1 --2- 172.21.15.42:0/1580461358 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5e600a51e0 0x7f5e600a5610 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:32:52.013 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:52.005+0000 7f5e653c8700 1 --2- 172.21.15.42:0/1580461358 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5e600a2be0 0x7f5e600a3010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:32:52.013 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:52.005+0000 7f5e653c8700 1 -- 172.21.15.42:0/1580461358 shutdown_connections 2019-12-05T05:32:52.013 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:52.005+0000 7f5e653c8700 1 -- 172.21.15.42:0/1580461358 wait complete. 2019-12-05T05:32:52.013 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:52.009+0000 7f5e653c8700 1 -- 172.21.15.42:0/1580461358 >> 172.21.15.42:0/1580461358 conn(0x7f5e600aabe0 msgr2=0x7f5e60059d50 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:32:52.022 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2019-12-05T05:32:54.023 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:32:54.129 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:54.117+0000 7f7568e60700 1 Processor -- start 2019-12-05T05:32:54.129 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:54.117+0000 7f7568e60700 1 -- start start 2019-12-05T05:32:54.129 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:54.121+0000 7f7568e60700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f756408df80 0x7f756409f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:32:54.130 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:54.121+0000 7f7568e60700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f756409f830 0x7f756414efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:32:54.130 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:54.121+0000 7f7568e60700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f756409fc60 con 0x7f75640a7740 2019-12-05T05:32:54.130 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:54.121+0000 7f7568e60700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75640a0b00 con 0x7f756409f830 2019-12-05T05:32:54.130 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:54.121+0000 7f7568e60700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f756408e5b0 con 0x7f756408df80 2019-12-05T05:32:54.130 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:54.121+0000 7f755bfff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f756408df80 0x7f756409f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:32:54.130 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:54.121+0000 7f755bfff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f756408df80 0x7f756409f2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:36198/0 (socket says 172.21.15.42:36198) 2019-12-05T05:32:54.130 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:54.121+0000 7f755bfff700 1 -- 172.21.15.42:0/353880336 learned_addr learned my addr 172.21.15.42:0/353880336 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:32:54.131 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:54.121+0000 7f756359e700 1 --2- 172.21.15.42:0/353880336 >> v2:172.21.15.40:3300/0 conn(0x7f756409f830 0x7f756414efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:32:54.131 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:54.121+0000 7f755bfff700 1 -- 172.21.15.42:0/353880336 >> v2:172.21.15.40:3300/0 conn(0x7f756409f830 msgr2=0x7f756414efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:32:54.131 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:54.121+0000 7f755bfff700 1 --2- 172.21.15.42:0/353880336 >> v2:172.21.15.40:3300/0 conn(0x7f756409f830 0x7f756414efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:32:54.131 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:54.121+0000 7f755bfff700 1 -- 172.21.15.42:0/353880336 >> v1:172.21.15.40:6789/0 conn(0x7f75640a7740 legacy=0x7f756407e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:32:54.132 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:54.121+0000 7f755bfff700 1 -- 172.21.15.42:0/353880336 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75640a0710 con 0x7f756408df80 2019-12-05T05:32:54.133 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:54.121+0000 7f755bfff700 1 --2- 172.21.15.42:0/353880336 >> v2:172.21.15.42:3300/0 conn(0x7f756408df80 0x7f756409f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f754c002f30 tx=0x7f754c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:54.133 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:54.121+0000 7f755bfff700 1 --2- 172.21.15.42:0/353880336 >> v2:172.21.15.42:3300/0 conn(0x7f756408df80 0x7f756409f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f754c002f30 tx=0x7f754c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:54.133 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:54.121+0000 7f755bfff700 1 --2- 172.21.15.42:0/353880336 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f756408df80 0x7f756409f2e0 secure :-1 s=READY pgs=1091 cs=0 l=1 rx=0x7f754c002f30 tx=0x7f754c00fd30).ready entity=mon.1 client_cookie=a550535063dcbbe2 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:32:54.133 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:54.121+0000 7f755bfff700 1 --2- 172.21.15.42:0/353880336 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f756408df80 0x7f756409f2e0 secure :-1 s=THROTTLE_DONE pgs=1091 cs=0 l=1 rx=0x7f754c002f30 tx=0x7f754c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:54.133 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:54.121+0000 7f755bfff700 1 --2- 172.21.15.42:0/353880336 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f756408df80 0x7f756409f2e0 secure :-1 s=THROTTLE_DONE pgs=1091 cs=0 l=1 rx=0x7f754c002f30 tx=0x7f754c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:54.134 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:54.121+0000 7f755bfff700 1 --2- 172.21.15.42:0/353880336 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f756408df80 0x7f756409f2e0 secure :-1 s=THROTTLE_DONE pgs=1091 cs=0 l=1 rx=0x7f754c002f30 tx=0x7f754c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:54.134 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:54.121+0000 7f756259c700 1 -- 172.21.15.42:0/353880336 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f754c008c90 con 0x7f756408df80 2019-12-05T05:32:54.134 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:54.121+0000 7f756259c700 1 -- 172.21.15.42:0/353880336 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f754c0088b0 con 0x7f756408df80 2019-12-05T05:32:54.134 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:54.121+0000 7f756259c700 1 -- 172.21.15.42:0/353880336 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f754c011ad0 con 0x7f756408df80 2019-12-05T05:32:54.134 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:54.121+0000 7f7568e60700 1 -- 172.21.15.42:0/353880336 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f756408df80 msgr2=0x7f756409f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:32:54.134 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:54.121+0000 7f7568e60700 1 --2- 172.21.15.42:0/353880336 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f756408df80 0x7f756409f2e0 secure :-1 s=READY pgs=1091 cs=0 l=1 rx=0x7f754c002f30 tx=0x7f754c00fd30).stop 2019-12-05T05:32:54.134 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:54.121+0000 7f7568e60700 1 -- 172.21.15.42:0/353880336 shutdown_connections 2019-12-05T05:32:54.135 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:54.121+0000 7f7568e60700 1 --2- 172.21.15.42:0/353880336 >> v2:172.21.15.40:3300/0 conn(0x7f756409f830 0x7f756414efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:32:54.135 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:54.121+0000 7f7568e60700 1 --2- 172.21.15.42:0/353880336 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f756408df80 0x7f756409f2e0 unknown :-1 s=CLOSED pgs=1091 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:32:54.135 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:54.121+0000 7f7568e60700 1 -- 172.21.15.42:0/353880336 shutdown_connections 2019-12-05T05:32:54.135 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:54.121+0000 7f7568e60700 1 -- 172.21.15.42:0/353880336 wait complete. 2019-12-05T05:32:54.135 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:54.121+0000 7f7568e60700 1 -- 172.21.15.42:0/353880336 >> 172.21.15.42:0/353880336 conn(0x7f75640aabe0 msgr2=0x7f75640a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:32:54.139 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:54.125+0000 7f7568e60700 1 Processor -- start 2019-12-05T05:32:54.139 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:54.125+0000 7f7568e60700 1 -- start start 2019-12-05T05:32:54.139 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:54.125+0000 7f7568e60700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f756412ff40 0x7f7564132380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:32:54.139 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:54.125+0000 7f7568e60700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75641240f0 con 0x7f756414ed10 2019-12-05T05:32:54.140 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:54.125+0000 7f7568e60700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f756407a580 con 0x7f7564144a50 2019-12-05T05:32:54.140 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:54.125+0000 7f7568e60700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7564079b40 con 0x7f756412ff40 2019-12-05T05:32:54.140 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:54.125+0000 7f756359e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f756412ff40 0x7f7564132380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:32:54.140 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:54.125+0000 7f756359e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f756412ff40 0x7f7564132380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:36206/0 (socket says 172.21.15.42:36206) 2019-12-05T05:32:54.140 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:54.125+0000 7f756359e700 1 -- 172.21.15.42:0/1076573750 learned_addr learned my addr 172.21.15.42:0/1076573750 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:32:54.140 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:54.129+0000 7f756359e700 1 -- 172.21.15.42:0/1076573750 >> v1:172.21.15.42:6789/0 conn(0x7f7564144a50 legacy=0x7f75641238f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:32:54.140 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:54.129+0000 7f755b7fe700 1 -- 172.21.15.42:0/1076573750 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 298340225 0 0) 0x7f75641240f0 con 0x7f756414ed10 2019-12-05T05:32:54.141 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:54.129+0000 7f756359e700 1 -- 172.21.15.42:0/1076573750 >> v1:172.21.15.40:6789/0 conn(0x7f756414ed10 legacy=0x7f7564129af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:32:54.141 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:54.129+0000 7f756359e700 1 -- 172.21.15.42:0/1076573750 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f756412a2f0 con 0x7f756412ff40 2019-12-05T05:32:54.141 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:54.129+0000 7f756359e700 1 --2- 172.21.15.42:0/1076573750 >> v2:172.21.15.42:3300/0 conn(0x7f756412ff40 0x7f7564132380 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f755c0032b0 tx=0x7f755c010cc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:54.141 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:54.129+0000 7f756359e700 1 --2- 172.21.15.42:0/1076573750 >> v2:172.21.15.42:3300/0 conn(0x7f756412ff40 0x7f7564132380 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f755c0032b0 tx=0x7f755c010cc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:54.141 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:54.129+0000 7f756359e700 1 --2- 172.21.15.42:0/1076573750 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f756412ff40 0x7f7564132380 secure :-1 s=READY pgs=1093 cs=0 l=1 rx=0x7f755c0032b0 tx=0x7f755c010cc0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:32:54.142 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:54.129+0000 7f756359e700 1 --2- 172.21.15.42:0/1076573750 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f756412ff40 0x7f7564132380 secure :-1 s=THROTTLE_DONE pgs=1093 cs=0 l=1 rx=0x7f755c0032b0 tx=0x7f755c010cc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:54.142 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:54.129+0000 7f755b7fe700 1 -- 172.21.15.42:0/1076573750 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f755c005ca0 con 0x7f756412ff40 2019-12-05T05:32:54.142 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:54.129+0000 7f756359e700 1 --2- 172.21.15.42:0/1076573750 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f756412ff40 0x7f7564132380 secure :-1 s=THROTTLE_DONE pgs=1093 cs=0 l=1 rx=0x7f755c0032b0 tx=0x7f755c010cc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:54.142 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:54.129+0000 7f756359e700 1 --2- 172.21.15.42:0/1076573750 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f756412ff40 0x7f7564132380 secure :-1 s=THROTTLE_DONE pgs=1093 cs=0 l=1 rx=0x7f755c0032b0 tx=0x7f755c010cc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:54.143 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:54.129+0000 7f755b7fe700 1 -- 172.21.15.42:0/1076573750 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f755c0058c0 con 0x7f756412ff40 2019-12-05T05:32:54.143 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:54.129+0000 7f7568e60700 1 -- 172.21.15.42:0/1076573750 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7564064dc0 con 0x7f756412ff40 2019-12-05T05:32:54.143 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:54.129+0000 7f755b7fe700 1 -- 172.21.15.42:0/1076573750 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f755c012c20 con 0x7f756412ff40 2019-12-05T05:32:54.144 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:54.129+0000 7f7568e60700 1 -- 172.21.15.42:0/1076573750 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75641328d0 con 0x7f756412ff40 2019-12-05T05:32:54.144 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:54.129+0000 7f756359e700 1 --2- 172.21.15.42:0/1076573750 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f756412ff40 0x7f7564132380 secure :-1 s=THROTTLE_DONE pgs=1093 cs=0 l=1 rx=0x7f755c0032b0 tx=0x7f755c010cc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:54.144 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:54.129+0000 7f755b7fe700 1 -- 172.21.15.42:0/1076573750 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f755c0093b0 con 0x7f756412ff40 2019-12-05T05:32:54.144 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:54.129+0000 7f756359e700 1 --2- 172.21.15.42:0/1076573750 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f756412ff40 0x7f7564132380 secure :-1 s=THROTTLE_DONE pgs=1093 cs=0 l=1 rx=0x7f755c0032b0 tx=0x7f755c010cc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:54.144 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:54.129+0000 7f755b7fe700 1 --2- 172.21.15.42:0/1076573750 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f754803ac40 0x7f754803d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:32:54.145 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:54.129+0000 7f755b7fe700 1 -- 172.21.15.42:0/1076573750 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 7837+0+0 (secure 0 0 0) 0x7f755c04bbf0 con 0x7f756412ff40 2019-12-05T05:32:54.145 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:54.129+0000 7f7562d9d700 1 --2- 172.21.15.42:0/1076573750 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f754803ac40 0x7f754803d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:32:54.145 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:54.129+0000 7f7562d9d700 1 --2- 172.21.15.42:0/1076573750 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f754803ac40 0x7f754803d150 crc :-1 s=READY pgs=469 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:32:54.145 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:54.133+0000 7f7568e60700 1 -- 172.21.15.42:0/1076573750 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f756405d110 con 0x7f756412ff40 2019-12-05T05:32:54.146 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:54.137+0000 7f756359e700 1 --2- 172.21.15.42:0/1076573750 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f756412ff40 0x7f7564132380 secure :-1 s=THROTTLE_DONE pgs=1093 cs=0 l=1 rx=0x7f755c0032b0 tx=0x7f755c010cc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:54.146 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:54.137+0000 7f755b7fe700 1 -- 172.21.15.42:0/1076573750 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f755c032c70 con 0x7f756412ff40 2019-12-05T05:32:54.311 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:54.305+0000 7f7568e60700 1 -- 172.21.15.42:0/1076573750 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f756405b8c0 con 0x7f756412ff40 2019-12-05T05:32:54.312 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:54.309+0000 7f756359e700 1 --2- 172.21.15.42:0/1076573750 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f756412ff40 0x7f7564132380 secure :-1 s=THROTTLE_DONE pgs=1093 cs=0 l=1 rx=0x7f755c0032b0 tx=0x7f755c010cc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:32:54.312 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:54.309+0000 7f755b7fe700 1 -- 172.21.15.42:0/1076573750 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 137 v137) v1 ==== 95+0+1775 (secure 0 0 0) 0x7f755c032ed0 con 0x7f756412ff40 2019-12-05T05:32:54.313 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 137 2019-12-05T05:32:54.313 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:32:54.313 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":137,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":137,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:32:53.767447+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6997},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6997":{"gid":6997,"name":"a","rank":0,"incarnation":134,"state":"up:active","state_seq":17,"addr":"172.21.15.42:6835/1601334502","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1601334502},{"type":"v1","addr":"172.21.15.42:6835","nonce":1601334502}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:32:54.315 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:54.309+0000 7f7568e60700 1 -- 172.21.15.42:0/1076573750 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f754803ac40 msgr2=0x7f754803d150 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:32:54.315 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:54.309+0000 7f7568e60700 1 --2- 172.21.15.42:0/1076573750 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f754803ac40 0x7f754803d150 crc :-1 s=READY pgs=469 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:32:54.315 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:54.309+0000 7f7568e60700 1 -- 172.21.15.42:0/1076573750 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f756412ff40 msgr2=0x7f7564132380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:32:54.315 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:54.309+0000 7f7568e60700 1 --2- 172.21.15.42:0/1076573750 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f756412ff40 0x7f7564132380 secure :-1 s=READY pgs=1093 cs=0 l=1 rx=0x7f755c0032b0 tx=0x7f755c010cc0).stop 2019-12-05T05:32:54.316 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:54.309+0000 7f7568e60700 1 -- 172.21.15.42:0/1076573750 shutdown_connections 2019-12-05T05:32:54.316 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:54.309+0000 7f7568e60700 1 --2- 172.21.15.42:0/1076573750 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f754803ac40 0x7f754803d150 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:32:54.316 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:54.309+0000 7f7568e60700 1 --2- 172.21.15.42:0/1076573750 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f756412ff40 0x7f7564132380 unknown :-1 s=CLOSED pgs=1093 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:32:54.316 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:54.309+0000 7f7568e60700 1 -- 172.21.15.42:0/1076573750 shutdown_connections 2019-12-05T05:32:54.316 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:54.309+0000 7f7568e60700 1 -- 172.21.15.42:0/1076573750 wait complete. 2019-12-05T05:32:54.317 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:32:54.309+0000 7f7568e60700 1 -- 172.21.15.42:0/1076573750 >> 172.21.15.42:0/1076573750 conn(0x7f756408fde0 msgr2=0x7f75640a6a60 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:32:54.325 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a has gained rank=0, replacing gid=6984 2019-12-05T05:32:54.325 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 7 secs before reviving mds.b 2019-12-05T05:33:01.326 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.b 2019-12-05T05:33:01.326 INFO:tasks.ceph.mds.b:Restarting daemon 2019-12-05T05:33:01.326 INFO:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2019-12-05T05:33:01.330 INFO:tasks.ceph.mds.b:Started 2019-12-05T05:33:01.330 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:33:01.389 INFO:tasks.ceph.mds.b.smithi040.stdout:starting mds.b at 2019-12-05T05:33:01.441 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:01.433+0000 7f4e9f842700 1 Processor -- start 2019-12-05T05:33:01.442 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:01.433+0000 7f4e9f842700 1 -- start start 2019-12-05T05:33:01.442 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:01.433+0000 7f4e9f842700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4e980a7730 0x7f4e9808df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:33:01.442 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:01.433+0000 7f4e9f842700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e980a0700 con 0x7f4e9809fad0 2019-12-05T05:33:01.442 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:01.433+0000 7f4e9f842700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e980a9b90 con 0x7f4e9807e710 2019-12-05T05:33:01.442 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:01.433+0000 7f4e9f842700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e980a0af0 con 0x7f4e980a7730 2019-12-05T05:33:01.443 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:01.433+0000 7f4e9d5de700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f4e9807e710 0x7f4e9809f2d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:36280/0 (socket says 172.21.15.42:36280) 2019-12-05T05:33:01.443 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:01.433+0000 7f4e9d5de700 1 -- 172.21.15.42:0/3643790392 learned_addr learned my addr 172.21.15.42:0/3643790392 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:33:01.443 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:01.433+0000 7f4e9dddf700 1 --2- 172.21.15.42:0/3643790392 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4e980a7730 0x7f4e9808df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:33:01.443 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:01.433+0000 7f4e9cddd700 1 -- 172.21.15.42:0/3643790392 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 384991308 0 0) 0x7f4e980a9b90 con 0x7f4e9807e710 2019-12-05T05:33:01.443 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:01.433+0000 7f4e9cddd700 1 -- 172.21.15.42:0/3643790392 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e840030b0 con 0x7f4e9807e710 2019-12-05T05:33:01.443 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:01.433+0000 7f4e9dddf700 1 -- 172.21.15.42:0/3643790392 >> v1:172.21.15.42:6789/0 conn(0x7f4e9807e710 legacy=0x7f4e9809f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:33:01.443 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:01.433+0000 7f4e9dddf700 1 -- 172.21.15.42:0/3643790392 >> v1:172.21.15.40:6789/0 conn(0x7f4e9809fad0 legacy=0x7f4e9814efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:33:01.444 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:01.433+0000 7f4e9dddf700 1 -- 172.21.15.42:0/3643790392 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e9807eb40 con 0x7f4e980a7730 2019-12-05T05:33:01.444 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:01.433+0000 7f4e9dddf700 1 --2- 172.21.15.42:0/3643790392 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4e980a7730 0x7f4e9808df70 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f4e90001490 tx=0x7f4e900014d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:33:01.444 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:01.437+0000 7f4e9dddf700 1 --2- 172.21.15.42:0/3643790392 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4e980a7730 0x7f4e9808df70 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f4e90001490 tx=0x7f4e900014d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:33:01.444 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:01.437+0000 7f4e9dddf700 1 --2- 172.21.15.42:0/3643790392 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4e980a7730 0x7f4e9808df70 secure :-1 s=READY pgs=225 cs=0 l=1 rx=0x7f4e90001490 tx=0x7f4e900014d0).ready entity=mon.2 client_cookie=63b37cc62abd11bf server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:33:01.444 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:01.437+0000 7f4e9dddf700 1 --2- 172.21.15.42:0/3643790392 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4e980a7730 0x7f4e9808df70 secure :-1 s=THROTTLE_DONE pgs=225 cs=0 l=1 rx=0x7f4e90001490 tx=0x7f4e900014d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:33:01.444 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:01.437+0000 7f4e9dddf700 1 --2- 172.21.15.42:0/3643790392 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4e980a7730 0x7f4e9808df70 secure :-1 s=THROTTLE_DONE pgs=225 cs=0 l=1 rx=0x7f4e90001490 tx=0x7f4e900014d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:33:01.445 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:01.437+0000 7f4e9cddd700 1 -- 172.21.15.42:0/3643790392 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4e94008bb0 con 0x7f4e980a7730 2019-12-05T05:33:01.445 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:01.437+0000 7f4e9dddf700 1 --2- 172.21.15.42:0/3643790392 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4e980a7730 0x7f4e9808df70 secure :-1 s=THROTTLE_DONE pgs=225 cs=0 l=1 rx=0x7f4e90001490 tx=0x7f4e900014d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:33:01.445 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:01.437+0000 7f4e9cddd700 1 -- 172.21.15.42:0/3643790392 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e940087d0 con 0x7f4e980a7730 2019-12-05T05:33:01.445 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:01.437+0000 7f4e9cddd700 1 -- 172.21.15.42:0/3643790392 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4e940117d0 con 0x7f4e980a7730 2019-12-05T05:33:01.445 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:01.437+0000 7f4e9f842700 1 -- 172.21.15.42:0/3643790392 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4e980a7730 msgr2=0x7f4e9808df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:33:01.445 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:01.437+0000 7f4e9f842700 1 --2- 172.21.15.42:0/3643790392 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4e980a7730 0x7f4e9808df70 secure :-1 s=READY pgs=225 cs=0 l=1 rx=0x7f4e90001490 tx=0x7f4e900014d0).stop 2019-12-05T05:33:01.445 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:01.437+0000 7f4e9f842700 1 -- 172.21.15.42:0/3643790392 shutdown_connections 2019-12-05T05:33:01.446 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:01.437+0000 7f4e9f842700 1 --2- 172.21.15.42:0/3643790392 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4e980a7730 0x7f4e9808df70 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:33:01.446 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:01.437+0000 7f4e9f842700 1 -- 172.21.15.42:0/3643790392 shutdown_connections 2019-12-05T05:33:01.446 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:01.437+0000 7f4e9f842700 1 -- 172.21.15.42:0/3643790392 wait complete. 2019-12-05T05:33:01.446 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:01.437+0000 7f4e9f842700 1 -- 172.21.15.42:0/3643790392 >> 172.21.15.42:0/3643790392 conn(0x7f4e980aabd0 msgr2=0x7f4e980a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:33:01.449 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:01.437+0000 7f4e9f842700 1 Processor -- start 2019-12-05T05:33:01.449 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:01.437+0000 7f4e9f842700 1 -- start start 2019-12-05T05:33:01.450 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:01.441+0000 7f4e9f842700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4e98121b40 0x7f4e98124440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:33:01.450 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:01.441+0000 7f4e9f842700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f4e98135980 0x7f4e98137dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:33:01.450 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:01.441+0000 7f4e9f842700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e98125280 con 0x7f4e9811dfc0 2019-12-05T05:33:01.450 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:01.441+0000 7f4e9f842700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e98117170 con 0x7f4e98135980 2019-12-05T05:33:01.450 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:01.441+0000 7f4e9f842700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e98124aa0 con 0x7f4e98121b40 2019-12-05T05:33:01.450 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:01.441+0000 7f4e9e5e0700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f4e98135980 0x7f4e98137dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:33:01.451 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:01.441+0000 7f4e9e5e0700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f4e98135980 0x7f4e98137dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:36250/0 (socket says 172.21.15.42:36250) 2019-12-05T05:33:01.451 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:01.441+0000 7f4e9e5e0700 1 -- 172.21.15.42:0/2416938069 learned_addr learned my addr 172.21.15.42:0/2416938069 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:33:01.451 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:01.441+0000 7f4e9dddf700 1 --2- 172.21.15.42:0/2416938069 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4e98121b40 0x7f4e98124440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:33:01.451 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:01.441+0000 7f4e9e5e0700 1 -- 172.21.15.42:0/2416938069 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4e98121b40 msgr2=0x7f4e98124440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:33:01.451 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:01.441+0000 7f4e9e5e0700 1 --2- 172.21.15.42:0/2416938069 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4e98121b40 0x7f4e98124440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:33:01.451 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:01.441+0000 7f4e9e5e0700 1 -- 172.21.15.42:0/2416938069 >> v1:172.21.15.42:6789/0 conn(0x7f4e9811dfc0 legacy=0x7f4e98130540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:33:01.451 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:01.441+0000 7f4e9e5e0700 1 -- 172.21.15.42:0/2416938069 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e98130d40 con 0x7f4e98135980 2019-12-05T05:33:01.452 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:01.441+0000 7f4e8e7fc700 1 -- 172.21.15.42:0/2416938069 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1154087101 0 0) 0x7f4e98125280 con 0x7f4e9811dfc0 2019-12-05T05:33:01.452 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:01.441+0000 7f4e9e5e0700 1 --2- 172.21.15.42:0/2416938069 >> v2:172.21.15.42:3300/0 conn(0x7f4e98135980 0x7f4e98137dc0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f4e90001450 tx=0x7f4e90010d20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:33:01.452 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:01.441+0000 7f4e9e5e0700 1 --2- 172.21.15.42:0/2416938069 >> v2:172.21.15.42:3300/0 conn(0x7f4e98135980 0x7f4e98137dc0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f4e90001450 tx=0x7f4e90010d20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:33:01.452 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:01.441+0000 7f4e9e5e0700 1 --2- 172.21.15.42:0/2416938069 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4e98135980 0x7f4e98137dc0 secure :-1 s=READY pgs=1099 cs=0 l=1 rx=0x7f4e90001450 tx=0x7f4e90010d20).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:33:01.452 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:01.441+0000 7f4e9e5e0700 1 --2- 172.21.15.42:0/2416938069 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4e98135980 0x7f4e98137dc0 secure :-1 s=THROTTLE_DONE pgs=1099 cs=0 l=1 rx=0x7f4e90001450 tx=0x7f4e90010d20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:33:01.452 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:01.441+0000 7f4e8e7fc700 1 -- 172.21.15.42:0/2416938069 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4e9000ca10 con 0x7f4e98135980 2019-12-05T05:33:01.453 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:01.441+0000 7f4e9f842700 1 -- 172.21.15.42:0/2416938069 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e9811e3f0 con 0x7f4e98135980 2019-12-05T05:33:01.453 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:01.441+0000 7f4e9e5e0700 1 --2- 172.21.15.42:0/2416938069 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4e98135980 0x7f4e98137dc0 secure :-1 s=THROTTLE_DONE pgs=1099 cs=0 l=1 rx=0x7f4e90001450 tx=0x7f4e90010d20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:33:01.453 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:01.441+0000 7f4e9f842700 1 -- 172.21.15.42:0/2416938069 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e9808eb60 con 0x7f4e98135980 2019-12-05T05:33:01.453 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:01.441+0000 7f4e9e5e0700 1 --2- 172.21.15.42:0/2416938069 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4e98135980 0x7f4e98137dc0 secure :-1 s=THROTTLE_DONE pgs=1099 cs=0 l=1 rx=0x7f4e90001450 tx=0x7f4e90010d20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:33:01.453 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:01.441+0000 7f4e8e7fc700 1 -- 172.21.15.42:0/2416938069 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e900101c0 con 0x7f4e98135980 2019-12-05T05:33:01.453 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:01.441+0000 7f4e8e7fc700 1 -- 172.21.15.42:0/2416938069 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4e90012590 con 0x7f4e98135980 2019-12-05T05:33:01.453 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:01.441+0000 7f4e9e5e0700 1 --2- 172.21.15.42:0/2416938069 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4e98135980 0x7f4e98137dc0 secure :-1 s=THROTTLE_DONE pgs=1099 cs=0 l=1 rx=0x7f4e90001450 tx=0x7f4e90010d20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:33:01.454 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:01.445+0000 7f4e8e7fc700 1 -- 172.21.15.42:0/2416938069 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f4e900128a0 con 0x7f4e98135980 2019-12-05T05:33:01.454 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:01.445+0000 7f4e9e5e0700 1 --2- 172.21.15.42:0/2416938069 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4e98135980 0x7f4e98137dc0 secure :-1 s=THROTTLE_DONE pgs=1099 cs=0 l=1 rx=0x7f4e90001450 tx=0x7f4e90010d20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:33:01.454 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:01.445+0000 7f4e8e7fc700 1 --2- 172.21.15.42:0/2416938069 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4e8003ada0 0x7f4e8003d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:33:01.454 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:01.445+0000 7f4e8e7fc700 1 -- 172.21.15.42:0/2416938069 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 7837+0+0 (secure 0 0 0) 0x7f4e9004b410 con 0x7f4e98135980 2019-12-05T05:33:01.454 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:01.445+0000 7f4e9dddf700 1 --2- 172.21.15.42:0/2416938069 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4e8003ada0 0x7f4e8003d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:33:01.454 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:01.445+0000 7f4e9dddf700 1 --2- 172.21.15.42:0/2416938069 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4e8003ada0 0x7f4e8003d2b0 crc :-1 s=READY pgs=470 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:33:01.455 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:01.449+0000 7f4e9f842700 1 -- 172.21.15.42:0/2416938069 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e74003200 con 0x7f4e98135980 2019-12-05T05:33:01.457 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:01.453+0000 7f4e9e5e0700 1 --2- 172.21.15.42:0/2416938069 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4e98135980 0x7f4e98137dc0 secure :-1 s=THROTTLE_DONE pgs=1099 cs=0 l=1 rx=0x7f4e90001450 tx=0x7f4e90010d20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:33:01.457 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:01.453+0000 7f4e8e7fc700 1 -- 172.21.15.42:0/2416938069 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f4e900332e0 con 0x7f4e98135980 2019-12-05T05:33:01.618 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:01.613+0000 7f4e9f842700 1 -- 172.21.15.42:0/2416938069 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4e74003630 con 0x7f4e98135980 2019-12-05T05:33:01.618 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:01.613+0000 7f4e9e5e0700 1 --2- 172.21.15.42:0/2416938069 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4e98135980 0x7f4e98137dc0 secure :-1 s=THROTTLE_DONE pgs=1099 cs=0 l=1 rx=0x7f4e90001450 tx=0x7f4e90010d20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:33:01.619 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:01.613+0000 7f4e8e7fc700 1 -- 172.21.15.42:0/2416938069 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 137 v137) v1 ==== 95+0+1775 (secure 0 0 0) 0x7f4e90012bb0 con 0x7f4e98135980 2019-12-05T05:33:01.619 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 137 2019-12-05T05:33:01.619 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:33:01.619 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":137,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":137,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:32:53.767447+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6997},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6997":{"gid":6997,"name":"a","rank":0,"incarnation":134,"state":"up:active","state_seq":17,"addr":"172.21.15.42:6835/1601334502","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1601334502},{"type":"v1","addr":"172.21.15.42:6835","nonce":1601334502}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:33:01.622 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:01.617+0000 7f4e9f842700 1 -- 172.21.15.42:0/2416938069 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4e8003ada0 msgr2=0x7f4e8003d2b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:33:01.622 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:01.617+0000 7f4e9f842700 1 --2- 172.21.15.42:0/2416938069 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4e8003ada0 0x7f4e8003d2b0 crc :-1 s=READY pgs=470 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:33:01.622 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:01.617+0000 7f4e9f842700 1 -- 172.21.15.42:0/2416938069 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4e98135980 msgr2=0x7f4e98137dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:33:01.622 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:01.617+0000 7f4e9f842700 1 --2- 172.21.15.42:0/2416938069 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4e98135980 0x7f4e98137dc0 secure :-1 s=READY pgs=1099 cs=0 l=1 rx=0x7f4e90001450 tx=0x7f4e90010d20).stop 2019-12-05T05:33:01.623 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:01.617+0000 7f4e9f842700 1 -- 172.21.15.42:0/2416938069 shutdown_connections 2019-12-05T05:33:01.623 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:01.617+0000 7f4e9f842700 1 --2- 172.21.15.42:0/2416938069 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4e8003ada0 0x7f4e8003d2b0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:33:01.623 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:01.617+0000 7f4e9f842700 1 --2- 172.21.15.42:0/2416938069 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4e98135980 0x7f4e98137dc0 unknown :-1 s=CLOSED pgs=1099 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:33:01.623 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:01.617+0000 7f4e9f842700 1 --2- 172.21.15.42:0/2416938069 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4e98121b40 0x7f4e98124440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:33:01.623 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:01.617+0000 7f4e9f842700 1 -- 172.21.15.42:0/2416938069 shutdown_connections 2019-12-05T05:33:01.624 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:01.617+0000 7f4e9f842700 1 -- 172.21.15.42:0/2416938069 wait complete. 2019-12-05T05:33:01.624 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:01.617+0000 7f4e9f842700 1 -- 172.21.15.42:0/2416938069 >> 172.21.15.42:0/2416938069 conn(0x7f4e9809ffa0 msgr2=0x7f4e980596f0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:33:01.635 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2019-12-05T05:33:03.636 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:33:03.744 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:03.737+0000 7f54694e4700 1 Processor -- start 2019-12-05T05:33:03.745 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:03.737+0000 7f54694e4700 1 -- start start 2019-12-05T05:33:03.745 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:03.737+0000 7f54694e4700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f54640a7740 0x7f546408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:33:03.745 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:03.737+0000 7f54694e4700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f546407e720 0x7f546407eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:33:03.745 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:03.737+0000 7f54694e4700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54640a0710 con 0x7f546409f2e0 2019-12-05T05:33:03.745 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:03.737+0000 7f54694e4700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54640a0b00 con 0x7f546407e720 2019-12-05T05:33:03.745 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:03.737+0000 7f54694e4700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f546408e5b0 con 0x7f54640a7740 2019-12-05T05:33:03.746 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:03.737+0000 7f54637fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f54640a7740 0x7f546408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:33:03.746 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:03.737+0000 7f54637fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f54640a7740 0x7f546408df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:36260/0 (socket says 172.21.15.42:36260) 2019-12-05T05:33:03.746 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:03.737+0000 7f5463fff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f546409f2e0 0x7f546409f710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:36298/0 (socket says 172.21.15.42:36298) 2019-12-05T05:33:03.746 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:03.737+0000 7f54637fe700 1 -- 172.21.15.42:0/2060591454 learned_addr learned my addr 172.21.15.42:0/2060591454 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:33:03.746 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:03.737+0000 7f5462ffd700 1 --2- 172.21.15.42:0/2060591454 >> v2:172.21.15.40:3300/0 conn(0x7f546407e720 0x7f546407eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:33:03.746 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:03.737+0000 7f54637fe700 1 -- 172.21.15.42:0/2060591454 >> v2:172.21.15.42:3300/0 conn(0x7f54640a7740 msgr2=0x7f546408df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2019-12-05T05:33:03.746 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:03.737+0000 7f54637fe700 1 -- 172.21.15.42:0/2060591454 >> v2:172.21.15.42:3300/0 conn(0x7f54640a7740 msgr2=0x7f546408df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2019-12-05T05:33:03.747 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:03.737+0000 7f54637fe700 1 --2- 172.21.15.42:0/2060591454 >> v2:172.21.15.42:3300/0 conn(0x7f54640a7740 0x7f546408df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-12-05T05:33:03.747 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:03.737+0000 7f54637fe700 1 --2- 172.21.15.42:0/2060591454 >> v2:172.21.15.42:3300/0 conn(0x7f54640a7740 0x7f546408df80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rx=0 tx=0)._fault waiting 0.200000 2019-12-05T05:33:03.747 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:03.737+0000 7f5462ffd700 1 -- 172.21.15.42:0/2060591454 >> v2:172.21.15.42:3300/0 conn(0x7f54640a7740 msgr2=0x7f546408df80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2019-12-05T05:33:03.747 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:03.737+0000 7f5462ffd700 1 --2- 172.21.15.42:0/2060591454 >> v2:172.21.15.42:3300/0 conn(0x7f54640a7740 0x7f546408df80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:33:03.747 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:03.737+0000 7f5462ffd700 1 -- 172.21.15.42:0/2060591454 >> v1:172.21.15.42:6789/0 conn(0x7f546409f2e0 legacy=0x7f546409f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:33:03.747 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:03.737+0000 7f5462ffd700 1 -- 172.21.15.42:0/2060591454 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f546407f0a0 con 0x7f546407e720 2019-12-05T05:33:03.747 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:03.737+0000 7f5462ffd700 1 --2- 172.21.15.42:0/2060591454 >> v2:172.21.15.40:3300/0 conn(0x7f546407e720 0x7f546407eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f54540032b0 tx=0x7f545400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:33:03.748 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:03.737+0000 7f5462ffd700 1 --2- 172.21.15.42:0/2060591454 >> v2:172.21.15.40:3300/0 conn(0x7f546407e720 0x7f546407eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f54540032b0 tx=0x7f545400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:33:03.748 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:03.737+0000 7f5462ffd700 1 --2- 172.21.15.42:0/2060591454 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f546407e720 0x7f546407eb50 secure :-1 s=READY pgs=861 cs=0 l=1 rx=0x7f54540032b0 tx=0x7f545400fff0).ready entity=mon.0 client_cookie=b1155a199d49dab4 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:33:03.748 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:03.737+0000 7f5462ffd700 1 --2- 172.21.15.42:0/2060591454 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f546407e720 0x7f546407eb50 secure :-1 s=THROTTLE_DONE pgs=861 cs=0 l=1 rx=0x7f54540032b0 tx=0x7f545400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:33:03.748 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:03.737+0000 7f54627fc700 1 -- 172.21.15.42:0/2060591454 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5454011a10 con 0x7f546407e720 2019-12-05T05:33:03.748 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:03.737+0000 7f5462ffd700 1 --2- 172.21.15.42:0/2060591454 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f546407e720 0x7f546407eb50 secure :-1 s=THROTTLE_DONE pgs=861 cs=0 l=1 rx=0x7f54540032b0 tx=0x7f545400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:33:03.748 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:03.737+0000 7f5462ffd700 1 --2- 172.21.15.42:0/2060591454 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f546407e720 0x7f546407eb50 secure :-1 s=THROTTLE_DONE pgs=861 cs=0 l=1 rx=0x7f54540032b0 tx=0x7f545400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:33:03.749 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:03.737+0000 7f54627fc700 1 -- 172.21.15.42:0/2060591454 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5454011630 con 0x7f546407e720 2019-12-05T05:33:03.749 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:03.737+0000 7f54627fc700 1 -- 172.21.15.42:0/2060591454 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f545400a160 con 0x7f546407e720 2019-12-05T05:33:03.749 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:03.737+0000 7f54694e4700 1 -- 172.21.15.42:0/2060591454 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f546407e720 msgr2=0x7f546407eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:33:03.749 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:03.737+0000 7f54694e4700 1 --2- 172.21.15.42:0/2060591454 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f546407e720 0x7f546407eb50 secure :-1 s=READY pgs=861 cs=0 l=1 rx=0x7f54540032b0 tx=0x7f545400fff0).stop 2019-12-05T05:33:03.749 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:03.741+0000 7f54694e4700 1 -- 172.21.15.42:0/2060591454 shutdown_connections 2019-12-05T05:33:03.749 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:03.741+0000 7f54694e4700 1 --2- 172.21.15.42:0/2060591454 >> v2:172.21.15.42:3300/0 conn(0x7f54640a7740 0x7f546408df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:33:03.749 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:03.741+0000 7f54694e4700 1 --2- 172.21.15.42:0/2060591454 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f546407e720 0x7f546407eb50 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:33:03.749 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:03.741+0000 7f54694e4700 1 -- 172.21.15.42:0/2060591454 shutdown_connections 2019-12-05T05:33:03.750 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:03.741+0000 7f54694e4700 1 -- 172.21.15.42:0/2060591454 wait complete. 2019-12-05T05:33:03.750 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:03.741+0000 7f54694e4700 1 -- 172.21.15.42:0/2060591454 >> 172.21.15.42:0/2060591454 conn(0x7f54640aabe0 msgr2=0x7f54640a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:33:03.753 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:03.741+0000 7f54694e4700 1 Processor -- start 2019-12-05T05:33:03.753 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:03.741+0000 7f54694e4700 1 -- start start 2019-12-05T05:33:03.753 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:03.741+0000 7f54694e4700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f546407d0b0 0x7f546407b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:33:03.753 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:03.741+0000 7f54694e4700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5464135910 0x7f5464137d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:33:03.754 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:03.741+0000 7f54694e4700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f546407b800 con 0x7f546407bb70 2019-12-05T05:33:03.754 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:03.741+0000 7f54694e4700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f546407d730 con 0x7f5464135910 2019-12-05T05:33:03.754 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:03.741+0000 7f54694e4700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f546411c120 con 0x7f546407d0b0 2019-12-05T05:33:03.754 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:03.741+0000 7f54637fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f546407d0b0 0x7f546407b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:33:03.754 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:03.741+0000 7f54637fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f546407d0b0 0x7f546407b2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:36262/0 (socket says 172.21.15.42:36262) 2019-12-05T05:33:03.754 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:03.741+0000 7f54637fe700 1 -- 172.21.15.42:0/4152218649 learned_addr learned my addr 172.21.15.42:0/4152218649 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:33:03.754 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:03.741+0000 7f5463fff700 1 --2- 172.21.15.42:0/4152218649 >> v2:172.21.15.40:3300/0 conn(0x7f5464135910 0x7f5464137d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:33:03.755 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:03.745+0000 7f545bfff700 1 -- 172.21.15.42:0/4152218649 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2055008775 0 0) 0x7f546407b800 con 0x7f546407bb70 2019-12-05T05:33:03.755 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:03.745+0000 7f545bfff700 1 -- 172.21.15.42:0/4152218649 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54480030b0 con 0x7f546407bb70 2019-12-05T05:33:03.755 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:03.745+0000 7f54637fe700 1 -- 172.21.15.42:0/4152218649 >> v2:172.21.15.40:3300/0 conn(0x7f5464135910 msgr2=0x7f5464137d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:33:03.755 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:03.745+0000 7f54637fe700 1 --2- 172.21.15.42:0/4152218649 >> v2:172.21.15.40:3300/0 conn(0x7f5464135910 0x7f5464137d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:33:03.755 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:03.745+0000 7f54637fe700 1 -- 172.21.15.42:0/4152218649 >> v1:172.21.15.40:6789/0 conn(0x7f546407bb70 legacy=0x7f54641304d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:33:03.755 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:03.745+0000 7f54637fe700 1 -- 172.21.15.42:0/4152218649 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f546407d9d0 con 0x7f546407d0b0 2019-12-05T05:33:03.756 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:03.745+0000 7f54637fe700 1 --2- 172.21.15.42:0/4152218649 >> v2:172.21.15.42:3300/0 conn(0x7f546407d0b0 0x7f546407b2b0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f544c002f30 tx=0x7f544c0109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:33:03.756 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:03.745+0000 7f54637fe700 1 --2- 172.21.15.42:0/4152218649 >> v2:172.21.15.42:3300/0 conn(0x7f546407d0b0 0x7f546407b2b0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f544c002f30 tx=0x7f544c0109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:33:03.756 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:03.745+0000 7f54637fe700 1 --2- 172.21.15.42:0/4152218649 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f546407d0b0 0x7f546407b2b0 secure :-1 s=READY pgs=1101 cs=0 l=1 rx=0x7f544c002f30 tx=0x7f544c0109e0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:33:03.756 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:03.745+0000 7f54637fe700 1 --2- 172.21.15.42:0/4152218649 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f546407d0b0 0x7f546407b2b0 secure :-1 s=THROTTLE_DONE pgs=1101 cs=0 l=1 rx=0x7f544c002f30 tx=0x7f544c0109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:33:03.756 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:03.745+0000 7f545bfff700 1 -- 172.21.15.42:0/4152218649 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f544c0088b0 con 0x7f546407d0b0 2019-12-05T05:33:03.756 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:03.745+0000 7f54637fe700 1 --2- 172.21.15.42:0/4152218649 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f546407d0b0 0x7f546407b2b0 secure :-1 s=THROTTLE_DONE pgs=1101 cs=0 l=1 rx=0x7f544c002f30 tx=0x7f544c0109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:33:03.756 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:03.745+0000 7f54637fe700 1 --2- 172.21.15.42:0/4152218649 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f546407d0b0 0x7f546407b2b0 secure :-1 s=THROTTLE_DONE pgs=1101 cs=0 l=1 rx=0x7f544c002f30 tx=0x7f544c0109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:33:03.757 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:03.745+0000 7f545bfff700 1 -- 172.21.15.42:0/4152218649 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f544c007160 con 0x7f546407d0b0 2019-12-05T05:33:03.757 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:03.745+0000 7f545bfff700 1 -- 172.21.15.42:0/4152218649 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f544c005840 con 0x7f546407d0b0 2019-12-05T05:33:03.757 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:03.745+0000 7f54694e4700 1 -- 172.21.15.42:0/4152218649 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f546407bfa0 con 0x7f546407d0b0 2019-12-05T05:33:03.757 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:03.745+0000 7f54694e4700 1 -- 172.21.15.42:0/4152218649 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5464118990 con 0x7f546407d0b0 2019-12-05T05:33:03.757 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:03.745+0000 7f54637fe700 1 --2- 172.21.15.42:0/4152218649 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f546407d0b0 0x7f546407b2b0 secure :-1 s=THROTTLE_DONE pgs=1101 cs=0 l=1 rx=0x7f544c002f30 tx=0x7f544c0109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:33:03.757 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:03.745+0000 7f545bfff700 1 -- 172.21.15.42:0/4152218649 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f544c005b50 con 0x7f546407d0b0 2019-12-05T05:33:03.757 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:03.745+0000 7f54637fe700 1 --2- 172.21.15.42:0/4152218649 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f546407d0b0 0x7f546407b2b0 secure :-1 s=THROTTLE_DONE pgs=1101 cs=0 l=1 rx=0x7f544c002f30 tx=0x7f544c0109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:33:03.758 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:03.745+0000 7f545bfff700 1 --2- 172.21.15.42:0/4152218649 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f544803b2c0 0x7f544803d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:33:03.758 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:03.745+0000 7f545bfff700 1 -- 172.21.15.42:0/4152218649 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 7837+0+0 (secure 0 0 0) 0x7f544c04ade0 con 0x7f546407d0b0 2019-12-05T05:33:03.758 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:03.745+0000 7f5462ffd700 1 --2- 172.21.15.42:0/4152218649 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f544803b2c0 0x7f544803d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:33:03.758 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:03.749+0000 7f5462ffd700 1 --2- 172.21.15.42:0/4152218649 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f544803b2c0 0x7f544803d7d0 crc :-1 s=READY pgs=471 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:33:03.758 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:03.753+0000 7f54694e4700 1 -- 172.21.15.42:0/4152218649 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f546405d110 con 0x7f546407d0b0 2019-12-05T05:33:03.762 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:03.757+0000 7f54637fe700 1 --2- 172.21.15.42:0/4152218649 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f546407d0b0 0x7f546407b2b0 secure :-1 s=THROTTLE_DONE pgs=1101 cs=0 l=1 rx=0x7f544c002f30 tx=0x7f544c0109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:33:03.762 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:03.757+0000 7f545bfff700 1 -- 172.21.15.42:0/4152218649 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f544c0320c0 con 0x7f546407d0b0 2019-12-05T05:33:03.927 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:03.921+0000 7f54694e4700 1 -- 172.21.15.42:0/4152218649 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f546405b8c0 con 0x7f546407d0b0 2019-12-05T05:33:03.928 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:03.925+0000 7f54637fe700 1 --2- 172.21.15.42:0/4152218649 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f546407d0b0 0x7f546407b2b0 secure :-1 s=THROTTLE_DONE pgs=1101 cs=0 l=1 rx=0x7f544c002f30 tx=0x7f544c0109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:33:03.928 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:03.925+0000 7f545bfff700 1 -- 172.21.15.42:0/4152218649 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 137 v137) v1 ==== 95+0+1775 (secure 0 0 0) 0x7f544c032c70 con 0x7f546407d0b0 2019-12-05T05:33:03.930 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 137 2019-12-05T05:33:03.930 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:33:03.930 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":137,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":137,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:32:53.767447+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6997},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6997":{"gid":6997,"name":"a","rank":0,"incarnation":134,"state":"up:active","state_seq":17,"addr":"172.21.15.42:6835/1601334502","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1601334502},{"type":"v1","addr":"172.21.15.42:6835","nonce":1601334502}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:33:03.934 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:03.929+0000 7f54694e4700 1 -- 172.21.15.42:0/4152218649 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f544803b2c0 msgr2=0x7f544803d7d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:33:03.934 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:03.929+0000 7f54694e4700 1 --2- 172.21.15.42:0/4152218649 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f544803b2c0 0x7f544803d7d0 crc :-1 s=READY pgs=471 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:33:03.934 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:03.929+0000 7f54694e4700 1 -- 172.21.15.42:0/4152218649 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f546407d0b0 msgr2=0x7f546407b2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:33:03.935 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:03.929+0000 7f54694e4700 1 --2- 172.21.15.42:0/4152218649 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f546407d0b0 0x7f546407b2b0 secure :-1 s=READY pgs=1101 cs=0 l=1 rx=0x7f544c002f30 tx=0x7f544c0109e0).stop 2019-12-05T05:33:03.935 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:03.929+0000 7f54694e4700 1 -- 172.21.15.42:0/4152218649 shutdown_connections 2019-12-05T05:33:03.935 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:03.929+0000 7f54694e4700 1 --2- 172.21.15.42:0/4152218649 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f544803b2c0 0x7f544803d7d0 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:33:03.935 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:03.929+0000 7f54694e4700 1 --2- 172.21.15.42:0/4152218649 >> v2:172.21.15.40:3300/0 conn(0x7f5464135910 0x7f5464137d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:33:03.935 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:03.929+0000 7f54694e4700 1 --2- 172.21.15.42:0/4152218649 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f546407d0b0 0x7f546407b2b0 unknown :-1 s=CLOSED pgs=1101 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:33:03.936 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:03.929+0000 7f54694e4700 1 -- 172.21.15.42:0/4152218649 shutdown_connections 2019-12-05T05:33:03.936 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:03.929+0000 7f54694e4700 1 -- 172.21.15.42:0/4152218649 wait complete. 2019-12-05T05:33:03.936 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:03.929+0000 7f54694e4700 1 -- 172.21.15.42:0/4152218649 >> 172.21.15.42:0/4152218649 conn(0x7f54640a82a0 msgr2=0x7f5464059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:33:03.950 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2019-12-05T05:33:05.952 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:33:06.061 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:06.053+0000 7f26e69d8700 1 Processor -- start 2019-12-05T05:33:06.062 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:06.053+0000 7f26e69d8700 1 -- start start 2019-12-05T05:33:06.062 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:06.053+0000 7f26e69d8700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f26e00a7740 0x7f26e008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:33:06.062 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:06.053+0000 7f26e69d8700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26e00a0710 con 0x7f26e007e720 2019-12-05T05:33:06.062 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:06.053+0000 7f26e69d8700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26e00a9ba0 con 0x7f26e009fae0 2019-12-05T05:33:06.062 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:06.053+0000 7f26e69d8700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26e00a0b00 con 0x7f26e00a7740 2019-12-05T05:33:06.062 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:06.053+0000 7f26e5776700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f26e009fae0 0x7f26e014efe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:36324/0 (socket says 172.21.15.42:36324) 2019-12-05T05:33:06.063 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:06.053+0000 7f26e5776700 1 -- 172.21.15.42:0/3901199140 learned_addr learned my addr 172.21.15.42:0/3901199140 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:33:06.063 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:06.053+0000 7f26e4f75700 1 --2- 172.21.15.42:0/3901199140 >> v2:172.21.15.40:3300/0 conn(0x7f26e00a7740 0x7f26e008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:33:06.063 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:06.053+0000 7f26df7fe700 1 -- 172.21.15.42:0/3901199140 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3403281499 0 0) 0x7f26e00a9ba0 con 0x7f26e009fae0 2019-12-05T05:33:06.063 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:06.053+0000 7f26df7fe700 1 -- 172.21.15.42:0/3901199140 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26d00030b0 con 0x7f26e009fae0 2019-12-05T05:33:06.063 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:06.053+0000 7f26df7fe700 1 -- 172.21.15.42:0/3901199140 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2407016426 0 0) 0x7f26d00030b0 con 0x7f26e009fae0 2019-12-05T05:33:06.063 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:06.053+0000 7f26e4f75700 1 -- 172.21.15.42:0/3901199140 >> v1:172.21.15.42:6789/0 conn(0x7f26e009fae0 legacy=0x7f26e014efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:33:06.063 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:06.053+0000 7f26e4f75700 1 -- 172.21.15.42:0/3901199140 >> v1:172.21.15.40:6789/0 conn(0x7f26e007e720 legacy=0x7f26e009f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:33:06.064 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:06.053+0000 7f26e4f75700 1 -- 172.21.15.42:0/3901199140 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26e007eb50 con 0x7f26e00a7740 2019-12-05T05:33:06.064 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:06.053+0000 7f26e4f75700 1 --2- 172.21.15.42:0/3901199140 >> v2:172.21.15.40:3300/0 conn(0x7f26e00a7740 0x7f26e008df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f26d4008790 tx=0x7f26d8001d10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:33:06.064 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:06.053+0000 7f26e4f75700 1 --2- 172.21.15.42:0/3901199140 >> v2:172.21.15.40:3300/0 conn(0x7f26e00a7740 0x7f26e008df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f26d4008790 tx=0x7f26d8001d10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:33:06.064 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:06.053+0000 7f26e4f75700 1 --2- 172.21.15.42:0/3901199140 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f26e00a7740 0x7f26e008df80 secure :-1 s=READY pgs=864 cs=0 l=1 rx=0x7f26d4008790 tx=0x7f26d8001d10).ready entity=mon.0 client_cookie=80d3bd93973996d3 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:33:06.064 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:06.053+0000 7f26e4f75700 1 --2- 172.21.15.42:0/3901199140 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f26e00a7740 0x7f26e008df80 secure :-1 s=THROTTLE_DONE pgs=864 cs=0 l=1 rx=0x7f26d4008790 tx=0x7f26d8001d10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:33:06.064 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:06.053+0000 7f26df7fe700 1 -- 172.21.15.42:0/3901199140 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f26d8001ad0 con 0x7f26e00a7740 2019-12-05T05:33:06.064 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:06.053+0000 7f26e4f75700 1 --2- 172.21.15.42:0/3901199140 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f26e00a7740 0x7f26e008df80 secure :-1 s=THROTTLE_DONE pgs=864 cs=0 l=1 rx=0x7f26d4008790 tx=0x7f26d8001d10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:33:06.065 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:06.053+0000 7f26df7fe700 1 -- 172.21.15.42:0/3901199140 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26d80018e0 con 0x7f26e00a7740 2019-12-05T05:33:06.065 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:06.053+0000 7f26e4f75700 1 --2- 172.21.15.42:0/3901199140 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f26e00a7740 0x7f26e008df80 secure :-1 s=THROTTLE_DONE pgs=864 cs=0 l=1 rx=0x7f26d4008790 tx=0x7f26d8001d10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:33:06.065 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:06.053+0000 7f26df7fe700 1 -- 172.21.15.42:0/3901199140 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f26d4011360 con 0x7f26e00a7740 2019-12-05T05:33:06.065 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:06.053+0000 7f26e69d8700 1 -- 172.21.15.42:0/3901199140 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f26e00a7740 msgr2=0x7f26e008df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:33:06.065 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:06.053+0000 7f26e69d8700 1 --2- 172.21.15.42:0/3901199140 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f26e00a7740 0x7f26e008df80 secure :-1 s=READY pgs=864 cs=0 l=1 rx=0x7f26d4008790 tx=0x7f26d8001d10).stop 2019-12-05T05:33:06.065 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:06.057+0000 7f26e69d8700 1 -- 172.21.15.42:0/3901199140 shutdown_connections 2019-12-05T05:33:06.065 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:06.057+0000 7f26e69d8700 1 --2- 172.21.15.42:0/3901199140 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f26e00a7740 0x7f26e008df80 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:33:06.066 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:06.057+0000 7f26e69d8700 1 -- 172.21.15.42:0/3901199140 shutdown_connections 2019-12-05T05:33:06.066 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:06.057+0000 7f26e69d8700 1 -- 172.21.15.42:0/3901199140 wait complete. 2019-12-05T05:33:06.066 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:06.057+0000 7f26e69d8700 1 -- 172.21.15.42:0/3901199140 >> 172.21.15.42:0/3901199140 conn(0x7f26e00aabe0 msgr2=0x7f26e00a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:33:06.072 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:06.057+0000 7f26e69d8700 1 Processor -- start 2019-12-05T05:33:06.072 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:06.057+0000 7f26e69d8700 1 -- start start 2019-12-05T05:33:06.072 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:06.061+0000 7f26e69d8700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f26e0116500 0x7f26e0119300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:33:06.072 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:06.061+0000 7f26e69d8700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f26e0130640 0x7f26e007a330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:33:06.073 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:06.061+0000 7f26e69d8700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26e011a1b0 con 0x7f26e0130210 2019-12-05T05:33:06.073 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:06.061+0000 7f26e69d8700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26e0123a80 con 0x7f26e0116500 2019-12-05T05:33:06.073 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:06.061+0000 7f26e69d8700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26e0123d20 con 0x7f26e0130640 2019-12-05T05:33:06.073 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:06.061+0000 7f26dffff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f26e0130210 0x7f26e01258f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:36330/0 (socket says 172.21.15.42:36330) 2019-12-05T05:33:06.073 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:06.061+0000 7f26dffff700 1 -- 172.21.15.42:0/3849593588 learned_addr learned my addr 172.21.15.42:0/3849593588 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:33:06.073 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:06.061+0000 7f26e5776700 1 --2- 172.21.15.42:0/3849593588 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f26e0130640 0x7f26e007a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:33:06.073 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:06.061+0000 7f26e4f75700 1 --2- 172.21.15.42:0/3849593588 >> v2:172.21.15.40:3300/0 conn(0x7f26e0116500 0x7f26e0119300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:33:06.074 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:06.061+0000 7f26dd7fa700 1 -- 172.21.15.42:0/3849593588 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3864200453 0 0) 0x7f26e011a1b0 con 0x7f26e0130210 2019-12-05T05:33:06.074 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:06.061+0000 7f26dd7fa700 1 -- 172.21.15.42:0/3849593588 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26c00030b0 con 0x7f26e0130210 2019-12-05T05:33:06.074 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:06.061+0000 7f26e4f75700 1 -- 172.21.15.42:0/3849593588 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f26e0130640 msgr2=0x7f26e007a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:33:06.074 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:06.061+0000 7f26e4f75700 1 --2- 172.21.15.42:0/3849593588 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f26e0130640 0x7f26e007a330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:33:06.074 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:06.061+0000 7f26e4f75700 1 -- 172.21.15.42:0/3849593588 >> v1:172.21.15.42:6789/0 conn(0x7f26e0130210 legacy=0x7f26e01258f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:33:06.074 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:06.061+0000 7f26e4f75700 1 -- 172.21.15.42:0/3849593588 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26e0130a70 con 0x7f26e0116500 2019-12-05T05:33:06.075 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:06.061+0000 7f26e4f75700 1 --2- 172.21.15.42:0/3849593588 >> v2:172.21.15.40:3300/0 conn(0x7f26e0116500 0x7f26e0119300 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f26e0123c40 tx=0x7f26d40111f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:33:06.075 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:06.061+0000 7f26e4f75700 1 --2- 172.21.15.42:0/3849593588 >> v2:172.21.15.40:3300/0 conn(0x7f26e0116500 0x7f26e0119300 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f26e0123c40 tx=0x7f26d40111f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:33:06.075 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:06.061+0000 7f26e4f75700 1 --2- 172.21.15.42:0/3849593588 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f26e0116500 0x7f26e0119300 secure :-1 s=READY pgs=865 cs=0 l=1 rx=0x7f26e0123c40 tx=0x7f26d40111f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:33:06.075 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:06.061+0000 7f26e4f75700 1 --2- 172.21.15.42:0/3849593588 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f26e0116500 0x7f26e0119300 secure :-1 s=THROTTLE_DONE pgs=865 cs=0 l=1 rx=0x7f26e0123c40 tx=0x7f26d40111f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:33:06.075 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:06.061+0000 7f26dd7fa700 1 -- 172.21.15.42:0/3849593588 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f26d4007340 con 0x7f26e0116500 2019-12-05T05:33:06.075 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:06.061+0000 7f26e4f75700 1 --2- 172.21.15.42:0/3849593588 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f26e0116500 0x7f26e0119300 secure :-1 s=THROTTLE_DONE pgs=865 cs=0 l=1 rx=0x7f26e0123c40 tx=0x7f26d40111f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:33:06.075 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:06.061+0000 7f26e4f75700 1 --2- 172.21.15.42:0/3849593588 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f26e0116500 0x7f26e0119300 secure :-1 s=THROTTLE_DONE pgs=865 cs=0 l=1 rx=0x7f26e0123c40 tx=0x7f26d40111f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:33:06.076 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:06.061+0000 7f26dd7fa700 1 -- 172.21.15.42:0/3849593588 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26d4008db0 con 0x7f26e0116500 2019-12-05T05:33:06.076 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:06.061+0000 7f26e69d8700 1 -- 172.21.15.42:0/3849593588 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26e0124260 con 0x7f26e0116500 2019-12-05T05:33:06.076 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:06.061+0000 7f26dd7fa700 1 -- 172.21.15.42:0/3849593588 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f26d4011bf0 con 0x7f26e0116500 2019-12-05T05:33:06.076 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:06.061+0000 7f26e69d8700 1 -- 172.21.15.42:0/3849593588 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26e0119850 con 0x7f26e0116500 2019-12-05T05:33:06.076 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:06.061+0000 7f26e4f75700 1 --2- 172.21.15.42:0/3849593588 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f26e0116500 0x7f26e0119300 secure :-1 s=THROTTLE_DONE pgs=865 cs=0 l=1 rx=0x7f26e0123c40 tx=0x7f26d40111f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:33:06.076 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:06.065+0000 7f26dd7fa700 1 -- 172.21.15.42:0/3849593588 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f26d4005540 con 0x7f26e0116500 2019-12-05T05:33:06.076 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:06.065+0000 7f26e4f75700 1 --2- 172.21.15.42:0/3849593588 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f26e0116500 0x7f26e0119300 secure :-1 s=THROTTLE_DONE pgs=865 cs=0 l=1 rx=0x7f26e0123c40 tx=0x7f26d40111f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:33:06.077 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:06.065+0000 7f26dd7fa700 1 --2- 172.21.15.42:0/3849593588 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f26c003b250 0x7f26c003d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:33:06.077 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:06.065+0000 7f26dd7fa700 1 -- 172.21.15.42:0/3849593588 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 7837+0+0 (secure 0 0 0) 0x7f26d404b2d0 con 0x7f26e0116500 2019-12-05T05:33:06.077 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:06.065+0000 7f26dffff700 1 --2- 172.21.15.42:0/3849593588 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f26c003b250 0x7f26c003d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:33:06.077 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:06.065+0000 7f26dffff700 1 --2- 172.21.15.42:0/3849593588 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f26c003b250 0x7f26c003d760 crc :-1 s=READY pgs=473 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:33:06.077 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:06.069+0000 7f26e69d8700 1 -- 172.21.15.42:0/3849593588 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26e005d110 con 0x7f26e0116500 2019-12-05T05:33:06.080 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:06.077+0000 7f26e4f75700 1 --2- 172.21.15.42:0/3849593588 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f26e0116500 0x7f26e0119300 secure :-1 s=THROTTLE_DONE pgs=865 cs=0 l=1 rx=0x7f26e0123c40 tx=0x7f26d40111f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:33:06.080 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:06.077+0000 7f26dd7fa700 1 -- 172.21.15.42:0/3849593588 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f26e0130a70 con 0x7f26e0116500 2019-12-05T05:33:06.278 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:06.273+0000 7f26e69d8700 1 -- 172.21.15.42:0/3849593588 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f26e005b8c0 con 0x7f26e0116500 2019-12-05T05:33:06.279 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:06.273+0000 7f26e4f75700 1 --2- 172.21.15.42:0/3849593588 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f26e0116500 0x7f26e0119300 secure :-1 s=THROTTLE_DONE pgs=865 cs=0 l=1 rx=0x7f26e0123c40 tx=0x7f26d40111f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:33:06.279 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:06.273+0000 7f26dd7fa700 1 -- 172.21.15.42:0/3849593588 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 138 v138) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f26d4033140 con 0x7f26e0116500 2019-12-05T05:33:06.281 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 138 2019-12-05T05:33:06.281 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:33:06.281 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":138,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7221,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/3789641053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3789641053},{"type":"v1","addr":"172.21.15.40:6833","nonce":3789641053}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":138}],"filesystems":[{"mdsmap":{"epoch":137,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:32:53.767447+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6997},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6997":{"gid":6997,"name":"a","rank":0,"incarnation":134,"state":"up:active","state_seq":17,"addr":"172.21.15.42:6835/1601334502","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1601334502},{"type":"v1","addr":"172.21.15.42:6835","nonce":1601334502}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:33:06.285 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:06.281+0000 7f26e69d8700 1 -- 172.21.15.42:0/3849593588 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f26c003b250 msgr2=0x7f26c003d760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:33:06.285 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:06.281+0000 7f26e69d8700 1 --2- 172.21.15.42:0/3849593588 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f26c003b250 0x7f26c003d760 crc :-1 s=READY pgs=473 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:33:06.285 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:06.281+0000 7f26e69d8700 1 -- 172.21.15.42:0/3849593588 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f26e0116500 msgr2=0x7f26e0119300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:33:06.285 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:06.281+0000 7f26e69d8700 1 --2- 172.21.15.42:0/3849593588 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f26e0116500 0x7f26e0119300 secure :-1 s=READY pgs=865 cs=0 l=1 rx=0x7f26e0123c40 tx=0x7f26d40111f0).stop 2019-12-05T05:33:06.286 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:06.281+0000 7f26e69d8700 1 -- 172.21.15.42:0/3849593588 shutdown_connections 2019-12-05T05:33:06.286 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:06.281+0000 7f26e69d8700 1 --2- 172.21.15.42:0/3849593588 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f26c003b250 0x7f26c003d760 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:33:06.286 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:06.281+0000 7f26e69d8700 1 --2- 172.21.15.42:0/3849593588 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f26e0130640 0x7f26e007a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:33:06.286 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:06.281+0000 7f26e69d8700 1 --2- 172.21.15.42:0/3849593588 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f26e0116500 0x7f26e0119300 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:33:06.286 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:06.281+0000 7f26e69d8700 1 -- 172.21.15.42:0/3849593588 shutdown_connections 2019-12-05T05:33:06.286 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:06.281+0000 7f26e69d8700 1 -- 172.21.15.42:0/3849593588 wait complete. 2019-12-05T05:33:06.286 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:33:06.281+0000 7f26e69d8700 1 -- 172.21.15.42:0/3849593588 >> 172.21.15.42:0/3849593588 conn(0x7f26e007ee20 msgr2=0x7f26e00a9790 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:33:06.297 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b reported in up:standby state 2019-12-05T05:33:06.297 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 75 secs before thrashing 2019-12-05T05:33:09.385 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:33:09.389 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:33:39.424 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:33:39.429 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:34:09.490 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:34:09.494 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:34:21.298 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:34:21.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.393+0000 7f3ced588700 1 Processor -- start 2019-12-05T05:34:21.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.393+0000 7f3ced588700 1 -- start start 2019-12-05T05:34:21.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.397+0000 7f3ced588700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f3ce80a7740 0x7f3ce808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:34:21.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.397+0000 7f3ced588700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ce80a0710 con 0x7f3ce807e720 2019-12-05T05:34:21.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.397+0000 7f3ced588700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ce80a9ba0 con 0x7f3ce809fae0 2019-12-05T05:34:21.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.397+0000 7f3ced588700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ce80a0b00 con 0x7f3ce80a7740 2019-12-05T05:34:21.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.397+0000 7f3ce7fff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f3ce809fae0 0x7f3ce814efe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:36656/0 (socket says 172.21.15.42:36656) 2019-12-05T05:34:21.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.397+0000 7f3ce7fff700 1 -- 172.21.15.42:0/4210430885 learned_addr learned my addr 172.21.15.42:0/4210430885 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:34:21.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.397+0000 7f3ce77fe700 1 --2- 172.21.15.42:0/4210430885 >> v2:172.21.15.40:3300/0 conn(0x7f3ce80a7740 0x7f3ce808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:34:21.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.397+0000 7f3ce67fc700 1 -- 172.21.15.42:0/4210430885 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3034288828 0 0) 0x7f3ce80a9ba0 con 0x7f3ce809fae0 2019-12-05T05:34:21.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.397+0000 7f3ce67fc700 1 -- 172.21.15.42:0/4210430885 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3cd00030b0 con 0x7f3ce809fae0 2019-12-05T05:34:21.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.397+0000 7f3ce77fe700 1 -- 172.21.15.42:0/4210430885 >> v1:172.21.15.42:6789/0 conn(0x7f3ce809fae0 legacy=0x7f3ce814efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:34:21.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.397+0000 7f3ce77fe700 1 -- 172.21.15.42:0/4210430885 >> v1:172.21.15.40:6789/0 conn(0x7f3ce807e720 legacy=0x7f3ce809f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:34:21.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.397+0000 7f3ce67fc700 1 -- 172.21.15.42:0/4210430885 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 565544289 0 0) 0x7f3ce80a0710 con 0x7f3ce807e720 2019-12-05T05:34:21.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.397+0000 7f3ce77fe700 1 -- 172.21.15.42:0/4210430885 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ce807eb50 con 0x7f3ce80a7740 2019-12-05T05:34:21.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.397+0000 7f3ce77fe700 1 --2- 172.21.15.42:0/4210430885 >> v2:172.21.15.40:3300/0 conn(0x7f3ce80a7740 0x7f3ce808df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f3cd8008790 tx=0x7f3cd800fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:21.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.397+0000 7f3ce77fe700 1 --2- 172.21.15.42:0/4210430885 >> v2:172.21.15.40:3300/0 conn(0x7f3ce80a7740 0x7f3ce808df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f3cd8008790 tx=0x7f3cd800fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:21.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.397+0000 7f3ce77fe700 1 --2- 172.21.15.42:0/4210430885 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3ce80a7740 0x7f3ce808df80 secure :-1 s=READY pgs=872 cs=0 l=1 rx=0x7f3cd8008790 tx=0x7f3cd800fbf0).ready entity=mon.0 client_cookie=e34d3e3a3b2f660e server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:34:21.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.397+0000 7f3ce77fe700 1 --2- 172.21.15.42:0/4210430885 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3ce80a7740 0x7f3ce808df80 secure :-1 s=THROTTLE_DONE pgs=872 cs=0 l=1 rx=0x7f3cd8008790 tx=0x7f3cd800fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:21.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.397+0000 7f3ce67fc700 1 -- 172.21.15.42:0/4210430885 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3cd8011270 con 0x7f3ce80a7740 2019-12-05T05:34:21.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.397+0000 7f3ce77fe700 1 --2- 172.21.15.42:0/4210430885 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3ce80a7740 0x7f3ce808df80 secure :-1 s=THROTTLE_DONE pgs=872 cs=0 l=1 rx=0x7f3cd8008790 tx=0x7f3cd800fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:21.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.397+0000 7f3ce67fc700 1 -- 172.21.15.42:0/4210430885 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cd8008bc0 con 0x7f3ce80a7740 2019-12-05T05:34:21.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.397+0000 7f3ce77fe700 1 --2- 172.21.15.42:0/4210430885 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3ce80a7740 0x7f3ce808df80 secure :-1 s=THROTTLE_DONE pgs=872 cs=0 l=1 rx=0x7f3cd8008790 tx=0x7f3cd800fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:21.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.397+0000 7f3ce67fc700 1 -- 172.21.15.42:0/4210430885 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3cd8011850 con 0x7f3ce80a7740 2019-12-05T05:34:21.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.397+0000 7f3ced588700 1 -- 172.21.15.42:0/4210430885 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3ce80a7740 msgr2=0x7f3ce808df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:34:21.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.397+0000 7f3ced588700 1 --2- 172.21.15.42:0/4210430885 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3ce80a7740 0x7f3ce808df80 secure :-1 s=READY pgs=872 cs=0 l=1 rx=0x7f3cd8008790 tx=0x7f3cd800fbf0).stop 2019-12-05T05:34:21.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.397+0000 7f3ced588700 1 -- 172.21.15.42:0/4210430885 shutdown_connections 2019-12-05T05:34:21.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.397+0000 7f3ced588700 1 --2- 172.21.15.42:0/4210430885 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3ce80a7740 0x7f3ce808df80 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:34:21.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.397+0000 7f3ced588700 1 -- 172.21.15.42:0/4210430885 shutdown_connections 2019-12-05T05:34:21.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.401+0000 7f3ced588700 1 -- 172.21.15.42:0/4210430885 wait complete. 2019-12-05T05:34:21.411 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.401+0000 7f3ced588700 1 -- 172.21.15.42:0/4210430885 >> 172.21.15.42:0/4210430885 conn(0x7f3ce80aabe0 msgr2=0x7f3ce80a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:34:21.415 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.401+0000 7f3ced588700 1 Processor -- start 2019-12-05T05:34:21.416 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.401+0000 7f3ced588700 1 -- start start 2019-12-05T05:34:21.416 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.401+0000 7f3ced588700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f3ce81448e0 0x7f3ce8142f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:34:21.416 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.401+0000 7f3ced588700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3ce81434e0 0x7f3ce807a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:34:21.416 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.401+0000 7f3ced588700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ce8144220 con 0x7f3ce807af10 2019-12-05T05:34:21.416 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.401+0000 7f3ced588700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ce81439c0 con 0x7f3ce81448e0 2019-12-05T05:34:21.416 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.401+0000 7f3ced588700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ce8121f40 con 0x7f3ce81434e0 2019-12-05T05:34:21.417 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.401+0000 7f3ce77fe700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f3ce81448e0 0x7f3ce8142f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:34:21.417 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.401+0000 7f3ce6ffd700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3ce81434e0 0x7f3ce807a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:34:21.417 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.401+0000 7f3ce77fe700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f3ce81448e0 0x7f3ce8142f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.42:52668/0 (socket says 172.21.15.42:52668) 2019-12-05T05:34:21.418 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.401+0000 7f3ce77fe700 1 -- 172.21.15.42:0/753033911 learned_addr learned my addr 172.21.15.42:0/753033911 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:34:21.418 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.401+0000 7f3ce77fe700 1 -- 172.21.15.42:0/753033911 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3ce81434e0 msgr2=0x7f3ce807a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:34:21.418 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.401+0000 7f3ce77fe700 1 --2- 172.21.15.42:0/753033911 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3ce81434e0 0x7f3ce807a9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:34:21.418 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.405+0000 7f3ce77fe700 1 -- 172.21.15.42:0/753033911 >> v1:172.21.15.40:6789/0 conn(0x7f3ce807af10 legacy=0x7f3ce81221e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:34:21.418 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.405+0000 7f3ce77fe700 1 -- 172.21.15.42:0/753033911 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ce807b340 con 0x7f3ce81448e0 2019-12-05T05:34:21.418 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.405+0000 7f3ce77fe700 1 --2- 172.21.15.42:0/753033911 >> v2:172.21.15.40:3300/0 conn(0x7f3ce81448e0 0x7f3ce8142f90 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f3ce8143b80 tx=0x7f3cd8011910).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:21.419 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.405+0000 7f3ce77fe700 1 --2- 172.21.15.42:0/753033911 >> v2:172.21.15.40:3300/0 conn(0x7f3ce81448e0 0x7f3ce8142f90 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f3ce8143b80 tx=0x7f3cd8011910).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:21.419 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.405+0000 7f3ce77fe700 1 --2- 172.21.15.42:0/753033911 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3ce81448e0 0x7f3ce8142f90 secure :-1 s=READY pgs=874 cs=0 l=1 rx=0x7f3ce8143b80 tx=0x7f3cd8011910).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:34:21.419 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.405+0000 7f3ce77fe700 1 --2- 172.21.15.42:0/753033911 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3ce81448e0 0x7f3ce8142f90 secure :-1 s=THROTTLE_DONE pgs=874 cs=0 l=1 rx=0x7f3ce8143b80 tx=0x7f3cd8011910).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:21.419 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.405+0000 7f3ce77fe700 1 --2- 172.21.15.42:0/753033911 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3ce81448e0 0x7f3ce8142f90 secure :-1 s=THROTTLE_DONE pgs=874 cs=0 l=1 rx=0x7f3ce8143b80 tx=0x7f3cd8011910).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:21.419 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.405+0000 7f3ccffff700 1 -- 172.21.15.42:0/753033911 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3cd8007340 con 0x7f3ce81448e0 2019-12-05T05:34:21.419 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.405+0000 7f3ce77fe700 1 --2- 172.21.15.42:0/753033911 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3ce81448e0 0x7f3ce8142f90 secure :-1 s=THROTTLE_DONE pgs=874 cs=0 l=1 rx=0x7f3ce8143b80 tx=0x7f3cd8011910).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:21.419 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.405+0000 7f3ccffff700 1 -- 172.21.15.42:0/753033911 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cd8011470 con 0x7f3ce81448e0 2019-12-05T05:34:21.420 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.405+0000 7f3ced588700 1 -- 172.21.15.42:0/753033911 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ce81229e0 con 0x7f3ce81448e0 2019-12-05T05:34:21.420 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.405+0000 7f3ccffff700 1 -- 172.21.15.42:0/753033911 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3cd8005540 con 0x7f3ce81448e0 2019-12-05T05:34:21.420 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.405+0000 7f3ced588700 1 -- 172.21.15.42:0/753033911 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ce812c940 con 0x7f3ce81448e0 2019-12-05T05:34:21.420 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.405+0000 7f3ce77fe700 1 --2- 172.21.15.42:0/753033911 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3ce81448e0 0x7f3ce8142f90 secure :-1 s=THROTTLE_DONE pgs=874 cs=0 l=1 rx=0x7f3ce8143b80 tx=0x7f3cd8011910).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:21.420 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.405+0000 7f3ccffff700 1 -- 172.21.15.42:0/753033911 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f3cd8005850 con 0x7f3ce81448e0 2019-12-05T05:34:21.421 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.405+0000 7f3ce77fe700 1 --2- 172.21.15.42:0/753033911 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3ce81448e0 0x7f3ce8142f90 secure :-1 s=THROTTLE_DONE pgs=874 cs=0 l=1 rx=0x7f3ce8143b80 tx=0x7f3cd8011910).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:21.421 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.405+0000 7f3ccffff700 1 --2- 172.21.15.42:0/753033911 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3cc803ada0 0x7f3cc803d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:34:21.421 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.405+0000 7f3ccffff700 1 -- 172.21.15.42:0/753033911 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 7837+0+0 (secure 0 0 0) 0x7f3cd804bba0 con 0x7f3ce81448e0 2019-12-05T05:34:21.421 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.405+0000 7f3ce6ffd700 1 --2- 172.21.15.42:0/753033911 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3cc803ada0 0x7f3cc803d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:34:21.421 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.405+0000 7f3ce6ffd700 1 --2- 172.21.15.42:0/753033911 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3cc803ada0 0x7f3cc803d2b0 crc :-1 s=READY pgs=479 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:34:21.421 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.409+0000 7f3ced588700 1 -- 172.21.15.42:0/753033911 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ce805d110 con 0x7f3ce81448e0 2019-12-05T05:34:21.423 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.421+0000 7f3ce77fe700 1 --2- 172.21.15.42:0/753033911 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3ce81448e0 0x7f3ce8142f90 secure :-1 s=THROTTLE_DONE pgs=874 cs=0 l=1 rx=0x7f3ce8143b80 tx=0x7f3cd8011910).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:21.424 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.421+0000 7f3ccffff700 1 -- 172.21.15.42:0/753033911 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f3cd8032c20 con 0x7f3ce81448e0 2019-12-05T05:34:21.614 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.605+0000 7f3ced588700 1 -- 172.21.15.42:0/753033911 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3ce805b8c0 con 0x7f3ce81448e0 2019-12-05T05:34:21.615 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.605+0000 7f3ce77fe700 1 --2- 172.21.15.42:0/753033911 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3ce81448e0 0x7f3ce8142f90 secure :-1 s=THROTTLE_DONE pgs=874 cs=0 l=1 rx=0x7f3ce8143b80 tx=0x7f3cd8011910).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:21.615 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.605+0000 7f3ccffff700 1 -- 172.21.15.42:0/753033911 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 138 v138) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f3cd8032e80 con 0x7f3ce81448e0 2019-12-05T05:34:21.615 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 138 2019-12-05T05:34:21.615 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.609+0000 7f3ced588700 1 -- 172.21.15.42:0/753033911 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3cc803ada0 msgr2=0x7f3cc803d2b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:34:21.615 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.609+0000 7f3ced588700 1 --2- 172.21.15.42:0/753033911 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3cc803ada0 0x7f3cc803d2b0 crc :-1 s=READY pgs=479 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:34:21.616 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.609+0000 7f3ced588700 1 -- 172.21.15.42:0/753033911 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3ce81448e0 msgr2=0x7f3ce8142f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:34:21.616 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.609+0000 7f3ced588700 1 --2- 172.21.15.42:0/753033911 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3ce81448e0 0x7f3ce8142f90 secure :-1 s=READY pgs=874 cs=0 l=1 rx=0x7f3ce8143b80 tx=0x7f3cd8011910).stop 2019-12-05T05:34:21.616 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.609+0000 7f3ced588700 1 -- 172.21.15.42:0/753033911 shutdown_connections 2019-12-05T05:34:21.616 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.609+0000 7f3ced588700 1 --2- 172.21.15.42:0/753033911 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3cc803ada0 0x7f3cc803d2b0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:34:21.620 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.609+0000 7f3ced588700 1 --2- 172.21.15.42:0/753033911 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3ce81448e0 0x7f3ce8142f90 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:34:21.621 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.609+0000 7f3ced588700 1 --2- 172.21.15.42:0/753033911 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3ce81434e0 0x7f3ce807a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:34:21.621 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.609+0000 7f3ced588700 1 -- 172.21.15.42:0/753033911 shutdown_connections 2019-12-05T05:34:21.621 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.609+0000 7f3ced588700 1 -- 172.21.15.42:0/753033911 wait complete. 2019-12-05T05:34:21.621 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.609+0000 7f3ced588700 1 -- 172.21.15.42:0/753033911 >> 172.21.15.42:0/753033911 conn(0x7f3ce807b880 msgr2=0x7f3ce809f3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:34:21.621 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:34:21.622 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":138,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7221,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/3789641053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3789641053},{"type":"v1","addr":"172.21.15.40:6833","nonce":3789641053}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":138}],"filesystems":[{"mdsmap":{"epoch":137,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:32:53.767447+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6997},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6997":{"gid":6997,"name":"a","rank":0,"incarnation":134,"state":"up:active","state_seq":17,"addr":"172.21.15.42:6835/1601334502","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1601334502},{"type":"v1","addr":"172.21.15.42:6835","nonce":1601334502}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:34:21.625 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.a (rank=0) 2019-12-05T05:34:21.625 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2019-12-05T05:34:21.625 INFO:teuthology.orchestra.run:waiting for 300 2019-12-05T05:34:21.636 INFO:tasks.ceph.mds.a:Stopped 2019-12-05T05:34:21.637 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:34:21.808 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.797+0000 7f5b623c7700 1 Processor -- start 2019-12-05T05:34:21.808 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.797+0000 7f5b623c7700 1 -- start start 2019-12-05T05:34:21.809 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.801+0000 7f5b623c7700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5b5c09fad0 0x7f5b5c14ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:34:21.809 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.801+0000 7f5b623c7700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b5c0a0700 con 0x7f5b5c0a7730 2019-12-05T05:34:21.809 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.801+0000 7f5b623c7700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b5c07ef10 con 0x7f5b5c08df70 2019-12-05T05:34:21.809 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.801+0000 7f5b623c7700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b5c0a0af0 con 0x7f5b5c09fad0 2019-12-05T05:34:21.809 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.801+0000 7f5b5bfff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f5b5c08df70 0x7f5b5c09f2d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:36670/0 (socket says 172.21.15.42:36670) 2019-12-05T05:34:21.809 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.801+0000 7f5b5bfff700 1 -- 172.21.15.42:0/3524963929 learned_addr learned my addr 172.21.15.42:0/3524963929 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:34:21.809 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.801+0000 7f5b61165700 1 --2- 172.21.15.42:0/3524963929 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5b5c09fad0 0x7f5b5c14ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:34:21.810 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.801+0000 7f5b5b7fe700 1 -- 172.21.15.42:0/3524963929 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3197832027 0 0) 0x7f5b5c07ef10 con 0x7f5b5c08df70 2019-12-05T05:34:21.810 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.801+0000 7f5b5b7fe700 1 -- 172.21.15.42:0/3524963929 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b480030b0 con 0x7f5b5c08df70 2019-12-05T05:34:21.810 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.801+0000 7f5b5b7fe700 1 -- 172.21.15.42:0/3524963929 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 22448355 0 0) 0x7f5b5c0a0700 con 0x7f5b5c0a7730 2019-12-05T05:34:21.810 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.801+0000 7f5b5b7fe700 1 -- 172.21.15.42:0/3524963929 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b5c07ef10 con 0x7f5b5c0a7730 2019-12-05T05:34:21.810 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.801+0000 7f5b61165700 1 -- 172.21.15.42:0/3524963929 >> v1:172.21.15.42:6789/0 conn(0x7f5b5c08df70 legacy=0x7f5b5c09f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:34:21.810 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.801+0000 7f5b61165700 1 -- 172.21.15.42:0/3524963929 >> v1:172.21.15.40:6789/0 conn(0x7f5b5c0a7730 legacy=0x7f5b5c07e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:34:21.811 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.801+0000 7f5b61165700 1 -- 172.21.15.42:0/3524963929 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b5c0a9b90 con 0x7f5b5c09fad0 2019-12-05T05:34:21.811 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.801+0000 7f5b61165700 1 --2- 172.21.15.42:0/3524963929 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5b5c09fad0 0x7f5b5c14ffe0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f5b540032b0 tx=0x7f5b5400ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:21.811 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.801+0000 7f5b61165700 1 --2- 172.21.15.42:0/3524963929 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5b5c09fad0 0x7f5b5c14ffe0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f5b540032b0 tx=0x7f5b5400ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:21.811 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.801+0000 7f5b61165700 1 --2- 172.21.15.42:0/3524963929 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5b5c09fad0 0x7f5b5c14ffe0 secure :-1 s=READY pgs=227 cs=0 l=1 rx=0x7f5b540032b0 tx=0x7f5b5400ff70).ready entity=mon.2 client_cookie=c54fe847ecf957b2 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:34:21.811 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.801+0000 7f5b61165700 1 --2- 172.21.15.42:0/3524963929 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5b5c09fad0 0x7f5b5c14ffe0 secure :-1 s=THROTTLE_DONE pgs=227 cs=0 l=1 rx=0x7f5b540032b0 tx=0x7f5b5400ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:21.811 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.801+0000 7f5b5b7fe700 1 -- 172.21.15.42:0/3524963929 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5b540109b0 con 0x7f5b5c09fad0 2019-12-05T05:34:21.811 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.801+0000 7f5b61165700 1 --2- 172.21.15.42:0/3524963929 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5b5c09fad0 0x7f5b5c14ffe0 secure :-1 s=THROTTLE_DONE pgs=227 cs=0 l=1 rx=0x7f5b540032b0 tx=0x7f5b5400ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:21.812 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.801+0000 7f5b61165700 1 --2- 172.21.15.42:0/3524963929 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5b5c09fad0 0x7f5b5c14ffe0 secure :-1 s=THROTTLE_DONE pgs=227 cs=0 l=1 rx=0x7f5b540032b0 tx=0x7f5b5400ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:21.812 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.801+0000 7f5b5b7fe700 1 -- 172.21.15.42:0/3524963929 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b54008b50 con 0x7f5b5c09fad0 2019-12-05T05:34:21.812 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.801+0000 7f5b5b7fe700 1 -- 172.21.15.42:0/3524963929 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5b540110c0 con 0x7f5b5c09fad0 2019-12-05T05:34:21.812 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.801+0000 7f5b623c7700 1 -- 172.21.15.42:0/3524963929 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5b5c09fad0 msgr2=0x7f5b5c14ffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:34:21.812 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.801+0000 7f5b623c7700 1 --2- 172.21.15.42:0/3524963929 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5b5c09fad0 0x7f5b5c14ffe0 secure :-1 s=READY pgs=227 cs=0 l=1 rx=0x7f5b540032b0 tx=0x7f5b5400ff70).stop 2019-12-05T05:34:21.812 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.801+0000 7f5b623c7700 1 -- 172.21.15.42:0/3524963929 shutdown_connections 2019-12-05T05:34:21.812 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.801+0000 7f5b623c7700 1 --2- 172.21.15.42:0/3524963929 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5b5c09fad0 0x7f5b5c14ffe0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:34:21.813 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.805+0000 7f5b623c7700 1 -- 172.21.15.42:0/3524963929 shutdown_connections 2019-12-05T05:34:21.813 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.805+0000 7f5b623c7700 1 -- 172.21.15.42:0/3524963929 wait complete. 2019-12-05T05:34:21.813 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.805+0000 7f5b623c7700 1 -- 172.21.15.42:0/3524963929 >> 172.21.15.42:0/3524963929 conn(0x7f5b5c0aabd0 msgr2=0x7f5b5c0a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:34:21.818 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.805+0000 7f5b623c7700 1 Processor -- start 2019-12-05T05:34:21.818 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.805+0000 7f5b623c7700 1 -- start start 2019-12-05T05:34:21.818 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.805+0000 7f5b623c7700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5b5c123f40 0x7f5b5c11b690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:34:21.818 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.805+0000 7f5b623c7700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b5c11bbe0 con 0x7f5b5c07de00 2019-12-05T05:34:21.818 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.805+0000 7f5b623c7700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b5c1248c0 con 0x7f5b5c07d9d0 2019-12-05T05:34:21.819 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.805+0000 7f5b623c7700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b5c11bec0 con 0x7f5b5c123f40 2019-12-05T05:34:21.819 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.809+0000 7f5b5bfff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f5b5c07d9d0 0x7f5b5c121050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:36678/0 (socket says 172.21.15.42:36678) 2019-12-05T05:34:21.819 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.809+0000 7f5b5bfff700 1 -- 172.21.15.42:0/2166119459 learned_addr learned my addr 172.21.15.42:0/2166119459 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:34:21.819 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.809+0000 7f5b60964700 1 --2- 172.21.15.42:0/2166119459 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5b5c123f40 0x7f5b5c11b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:34:21.819 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.809+0000 7f5b597fa700 1 -- 172.21.15.42:0/2166119459 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3413785595 0 0) 0x7f5b5c1248c0 con 0x7f5b5c07d9d0 2019-12-05T05:34:21.819 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.809+0000 7f5b597fa700 1 -- 172.21.15.42:0/2166119459 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b400030b0 con 0x7f5b5c07d9d0 2019-12-05T05:34:21.819 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.809+0000 7f5b597fa700 1 -- 172.21.15.42:0/2166119459 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1251594187 0 0) 0x7f5b54010da0 con 0x7f5b5c07de00 2019-12-05T05:34:21.820 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.809+0000 7f5b597fa700 1 -- 172.21.15.42:0/2166119459 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b5c1248c0 con 0x7f5b5c07de00 2019-12-05T05:34:21.820 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.809+0000 7f5b60964700 1 -- 172.21.15.42:0/2166119459 >> v1:172.21.15.42:6789/0 conn(0x7f5b5c07d9d0 legacy=0x7f5b5c121050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:34:21.820 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.809+0000 7f5b60964700 1 -- 172.21.15.42:0/2166119459 >> v1:172.21.15.40:6789/0 conn(0x7f5b5c07de00 legacy=0x7f5b5c137930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:34:21.820 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.809+0000 7f5b60964700 1 -- 172.21.15.42:0/2166119459 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b5c121850 con 0x7f5b5c123f40 2019-12-05T05:34:21.820 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.809+0000 7f5b60964700 1 --2- 172.21.15.42:0/2166119459 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5b5c123f40 0x7f5b5c11b690 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f5b44000e90 tx=0x7f5b4400c2f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:21.820 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.809+0000 7f5b60964700 1 --2- 172.21.15.42:0/2166119459 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5b5c123f40 0x7f5b5c11b690 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f5b44000e90 tx=0x7f5b4400c2f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:21.821 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.809+0000 7f5b60964700 1 --2- 172.21.15.42:0/2166119459 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5b5c123f40 0x7f5b5c11b690 secure :-1 s=READY pgs=228 cs=0 l=1 rx=0x7f5b44000e90 tx=0x7f5b4400c2f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:34:21.821 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.809+0000 7f5b60964700 1 --2- 172.21.15.42:0/2166119459 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5b5c123f40 0x7f5b5c11b690 secure :-1 s=THROTTLE_DONE pgs=228 cs=0 l=1 rx=0x7f5b44000e90 tx=0x7f5b4400c2f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:21.821 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.809+0000 7f5b597fa700 1 -- 172.21.15.42:0/2166119459 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5b4400cb90 con 0x7f5b5c123f40 2019-12-05T05:34:21.821 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.809+0000 7f5b60964700 1 --2- 172.21.15.42:0/2166119459 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5b5c123f40 0x7f5b5c11b690 secure :-1 s=THROTTLE_DONE pgs=228 cs=0 l=1 rx=0x7f5b44000e90 tx=0x7f5b4400c2f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:21.821 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.809+0000 7f5b60964700 1 --2- 172.21.15.42:0/2166119459 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5b5c123f40 0x7f5b5c11b690 secure :-1 s=THROTTLE_DONE pgs=228 cs=0 l=1 rx=0x7f5b44000e90 tx=0x7f5b4400c2f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:21.821 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.809+0000 7f5b597fa700 1 -- 172.21.15.42:0/2166119459 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b4400c7b0 con 0x7f5b5c123f40 2019-12-05T05:34:21.821 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.809+0000 7f5b623c7700 1 -- 172.21.15.42:0/2166119459 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b5c121b20 con 0x7f5b5c123f40 2019-12-05T05:34:21.822 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.809+0000 7f5b597fa700 1 -- 172.21.15.42:0/2166119459 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5b4400d6a0 con 0x7f5b5c123f40 2019-12-05T05:34:21.822 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.809+0000 7f5b623c7700 1 -- 172.21.15.42:0/2166119459 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b5c139140 con 0x7f5b5c123f40 2019-12-05T05:34:21.822 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.809+0000 7f5b60964700 1 --2- 172.21.15.42:0/2166119459 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5b5c123f40 0x7f5b5c11b690 secure :-1 s=THROTTLE_DONE pgs=228 cs=0 l=1 rx=0x7f5b44000e90 tx=0x7f5b4400c2f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:21.822 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.809+0000 7f5b597fa700 1 -- 172.21.15.42:0/2166119459 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f5b4400d970 con 0x7f5b5c123f40 2019-12-05T05:34:21.822 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.809+0000 7f5b60964700 1 --2- 172.21.15.42:0/2166119459 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5b5c123f40 0x7f5b5c11b690 secure :-1 s=THROTTLE_DONE pgs=228 cs=0 l=1 rx=0x7f5b44000e90 tx=0x7f5b4400c2f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:21.822 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.813+0000 7f5b597fa700 1 --2- 172.21.15.42:0/2166119459 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5b4003b4f0 0x7f5b4003da00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:34:21.822 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.813+0000 7f5b597fa700 1 -- 172.21.15.42:0/2166119459 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 7837+0+0 (secure 0 0 0) 0x7f5b4404b260 con 0x7f5b5c123f40 2019-12-05T05:34:21.823 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.813+0000 7f5b5bfff700 1 --2- 172.21.15.42:0/2166119459 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5b4003b4f0 0x7f5b4003da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:34:21.823 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.813+0000 7f5b5bfff700 1 --2- 172.21.15.42:0/2166119459 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5b4003b4f0 0x7f5b4003da00 crc :-1 s=READY pgs=480 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:34:21.823 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.813+0000 7f5b623c7700 1 -- 172.21.15.42:0/2166119459 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b5c05d100 con 0x7f5b5c123f40 2019-12-05T05:34:21.828 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.825+0000 7f5b60964700 1 --2- 172.21.15.42:0/2166119459 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5b5c123f40 0x7f5b5c11b690 secure :-1 s=THROTTLE_DONE pgs=228 cs=0 l=1 rx=0x7f5b44000e90 tx=0x7f5b4400c2f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:21.828 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.825+0000 7f5b597fa700 1 -- 172.21.15.42:0/2166119459 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f5b440322e0 con 0x7f5b5c123f40 2019-12-05T05:34:21.998 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.993+0000 7f5b623c7700 1 -- 172.21.15.42:0/2166119459 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5b5c05b8b0 con 0x7f5b5c123f40 2019-12-05T05:34:21.999 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.997+0000 7f5b60964700 1 --2- 172.21.15.42:0/2166119459 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5b5c123f40 0x7f5b5c11b690 secure :-1 s=THROTTLE_DONE pgs=228 cs=0 l=1 rx=0x7f5b44000e90 tx=0x7f5b4400c2f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:21.999 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:21.997+0000 7f5b597fa700 1 -- 172.21.15.42:0/2166119459 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 138 v138) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f5b44032540 con 0x7f5b5c123f40 2019-12-05T05:34:22.001 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 138 2019-12-05T05:34:22.002 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:34:22.002 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":138,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7221,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/3789641053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3789641053},{"type":"v1","addr":"172.21.15.40:6833","nonce":3789641053}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":138}],"filesystems":[{"mdsmap":{"epoch":137,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:32:53.767447+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6997},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6997":{"gid":6997,"name":"a","rank":0,"incarnation":134,"state":"up:active","state_seq":17,"addr":"172.21.15.42:6835/1601334502","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1601334502},{"type":"v1","addr":"172.21.15.42:6835","nonce":1601334502}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:34:22.004 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:22.001+0000 7f5b623c7700 1 -- 172.21.15.42:0/2166119459 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5b4003b4f0 msgr2=0x7f5b4003da00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:34:22.005 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:22.001+0000 7f5b623c7700 1 --2- 172.21.15.42:0/2166119459 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5b4003b4f0 0x7f5b4003da00 crc :-1 s=READY pgs=480 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:34:22.005 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:22.001+0000 7f5b623c7700 1 -- 172.21.15.42:0/2166119459 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5b5c123f40 msgr2=0x7f5b5c11b690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:34:22.005 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:22.001+0000 7f5b623c7700 1 --2- 172.21.15.42:0/2166119459 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5b5c123f40 0x7f5b5c11b690 secure :-1 s=READY pgs=228 cs=0 l=1 rx=0x7f5b44000e90 tx=0x7f5b4400c2f0).stop 2019-12-05T05:34:22.005 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:22.001+0000 7f5b623c7700 1 -- 172.21.15.42:0/2166119459 shutdown_connections 2019-12-05T05:34:22.005 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:22.001+0000 7f5b623c7700 1 --2- 172.21.15.42:0/2166119459 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5b4003b4f0 0x7f5b4003da00 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:34:22.005 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:22.001+0000 7f5b623c7700 1 --2- 172.21.15.42:0/2166119459 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5b5c123f40 0x7f5b5c11b690 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:34:22.005 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:22.001+0000 7f5b623c7700 1 -- 172.21.15.42:0/2166119459 shutdown_connections 2019-12-05T05:34:22.006 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:22.001+0000 7f5b623c7700 1 -- 172.21.15.42:0/2166119459 wait complete. 2019-12-05T05:34:22.006 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:22.001+0000 7f5b623c7700 1 -- 172.21.15.42:0/2166119459 >> 172.21.15.42:0/2166119459 conn(0x7f5b5c0a7730 msgr2=0x7f5b5c11daf0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:34:22.019 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T05:34:24.020 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:34:24.126 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:24.113+0000 7f80acfc9700 1 Processor -- start 2019-12-05T05:34:24.126 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:24.113+0000 7f80acfc9700 1 -- start start 2019-12-05T05:34:24.127 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:24.117+0000 7f80acfc9700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f80a808df80 0x7f80a809f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:34:24.127 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:24.117+0000 7f80acfc9700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80a809fc60 con 0x7f80a809f830 2019-12-05T05:34:24.127 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:24.117+0000 7f80acfc9700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80a80a0710 con 0x7f80a80a7740 2019-12-05T05:34:24.127 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:24.117+0000 7f80acfc9700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80a80a0b00 con 0x7f80a808df80 2019-12-05T05:34:24.127 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:24.117+0000 7f80a6d9d700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f80a80a7740 0x7f80a807e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:36694/0 (socket says 172.21.15.42:36694) 2019-12-05T05:34:24.128 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:24.117+0000 7f80a6d9d700 1 -- 172.21.15.42:0/674051132 learned_addr learned my addr 172.21.15.42:0/674051132 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:34:24.128 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:24.117+0000 7f80a659c700 1 --2- 172.21.15.42:0/674051132 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f80a808df80 0x7f80a809f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:34:24.128 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:24.117+0000 7f80a5d9b700 1 -- 172.21.15.42:0/674051132 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4001017265 0 0) 0x7f80a80a0710 con 0x7f80a80a7740 2019-12-05T05:34:24.129 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:24.117+0000 7f80a5d9b700 1 -- 172.21.15.42:0/674051132 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80900030b0 con 0x7f80a80a7740 2019-12-05T05:34:24.145 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:24.117+0000 7f80a659c700 1 -- 172.21.15.42:0/674051132 >> v1:172.21.15.42:6789/0 conn(0x7f80a80a7740 legacy=0x7f80a807e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:34:24.145 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:24.117+0000 7f80a5d9b700 1 -- 172.21.15.42:0/674051132 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1078615689 0 0) 0x7f80a809fc60 con 0x7f80a809f830 2019-12-05T05:34:24.145 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:24.117+0000 7f80a659c700 1 -- 172.21.15.42:0/674051132 >> v1:172.21.15.40:6789/0 conn(0x7f80a809f830 legacy=0x7f80a814efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:34:24.145 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:24.117+0000 7f80a659c700 1 -- 172.21.15.42:0/674051132 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80a807ef20 con 0x7f80a808df80 2019-12-05T05:34:24.145 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:24.117+0000 7f80a659c700 1 --2- 172.21.15.42:0/674051132 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f80a808df80 0x7f80a809f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f8098001d10 tx=0x7f8098001820).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:24.145 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:24.117+0000 7f80a659c700 1 --2- 172.21.15.42:0/674051132 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f80a808df80 0x7f80a809f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f8098001d10 tx=0x7f8098001820).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:24.146 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:24.117+0000 7f80a659c700 1 --2- 172.21.15.42:0/674051132 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f80a808df80 0x7f80a809f2e0 secure :-1 s=READY pgs=229 cs=0 l=1 rx=0x7f8098001d10 tx=0x7f8098001820).ready entity=mon.2 client_cookie=528b5d92ad2c5f1d server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:34:24.146 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:24.117+0000 7f80a659c700 1 --2- 172.21.15.42:0/674051132 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f80a808df80 0x7f80a809f2e0 secure :-1 s=THROTTLE_DONE pgs=229 cs=0 l=1 rx=0x7f8098001d10 tx=0x7f8098001820).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:24.146 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:24.117+0000 7f80a5d9b700 1 -- 172.21.15.42:0/674051132 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8098001ad0 con 0x7f80a808df80 2019-12-05T05:34:24.146 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:24.117+0000 7f80a659c700 1 --2- 172.21.15.42:0/674051132 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f80a808df80 0x7f80a809f2e0 secure :-1 s=THROTTLE_DONE pgs=229 cs=0 l=1 rx=0x7f8098001d10 tx=0x7f8098001820).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:24.146 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:24.117+0000 7f80a659c700 1 --2- 172.21.15.42:0/674051132 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f80a808df80 0x7f80a809f2e0 secure :-1 s=THROTTLE_DONE pgs=229 cs=0 l=1 rx=0x7f8098001d10 tx=0x7f8098001820).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:24.146 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:24.117+0000 7f80a5d9b700 1 -- 172.21.15.42:0/674051132 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80980018e0 con 0x7f80a808df80 2019-12-05T05:34:24.146 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:24.117+0000 7f80a5d9b700 1 -- 172.21.15.42:0/674051132 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f809c011170 con 0x7f80a808df80 2019-12-05T05:34:24.147 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:24.117+0000 7f80acfc9700 1 -- 172.21.15.42:0/674051132 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f80a808df80 msgr2=0x7f80a809f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:34:24.147 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:24.121+0000 7f80acfc9700 1 --2- 172.21.15.42:0/674051132 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f80a808df80 0x7f80a809f2e0 secure :-1 s=READY pgs=229 cs=0 l=1 rx=0x7f8098001d10 tx=0x7f8098001820).stop 2019-12-05T05:34:24.147 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:24.121+0000 7f80acfc9700 1 -- 172.21.15.42:0/674051132 shutdown_connections 2019-12-05T05:34:24.147 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:24.121+0000 7f80acfc9700 1 --2- 172.21.15.42:0/674051132 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f80a808df80 0x7f80a809f2e0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:34:24.147 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:24.121+0000 7f80acfc9700 1 -- 172.21.15.42:0/674051132 shutdown_connections 2019-12-05T05:34:24.147 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:24.121+0000 7f80acfc9700 1 -- 172.21.15.42:0/674051132 wait complete. 2019-12-05T05:34:24.147 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:24.121+0000 7f80acfc9700 1 -- 172.21.15.42:0/674051132 >> 172.21.15.42:0/674051132 conn(0x7f80a80aabe0 msgr2=0x7f80a80a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:34:24.153 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:24.121+0000 7f80acfc9700 1 Processor -- start 2019-12-05T05:34:24.153 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:24.121+0000 7f80acfc9700 1 -- start start 2019-12-05T05:34:24.153 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:24.125+0000 7f80acfc9700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f80a8116fd0 0x7f80a811b950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:34:24.154 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:24.125+0000 7f80acfc9700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80a8077b00 con 0x7f80a811d9f0 2019-12-05T05:34:24.154 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:24.125+0000 7f80acfc9700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80a8078d40 con 0x7f80a811bea0 2019-12-05T05:34:24.154 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:24.125+0000 7f80acfc9700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80a8116a90 con 0x7f80a8116fd0 2019-12-05T05:34:24.155 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:24.125+0000 7f80a659c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f80a8116fd0 0x7f80a811b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:34:24.156 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:24.125+0000 7f80a659c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f80a8116fd0 0x7f80a811b950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:36664/0 (socket says 172.21.15.42:36664) 2019-12-05T05:34:24.156 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:24.125+0000 7f80a659c700 1 -- 172.21.15.42:0/2560471304 learned_addr learned my addr 172.21.15.42:0/2560471304 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:34:24.156 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:24.125+0000 7f80a659c700 1 -- 172.21.15.42:0/2560471304 >> v1:172.21.15.42:6789/0 conn(0x7f80a811bea0 legacy=0x7f80a80762f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:34:24.156 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:24.125+0000 7f80a659c700 1 -- 172.21.15.42:0/2560471304 >> v1:172.21.15.40:6789/0 conn(0x7f80a811d9f0 legacy=0x7f80a811a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:34:24.157 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:24.125+0000 7f80a659c700 1 -- 172.21.15.42:0/2560471304 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80a8079f80 con 0x7f80a8116fd0 2019-12-05T05:34:24.157 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:24.125+0000 7f80977fe700 1 -- 172.21.15.42:0/2560471304 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2465363871 0 0) 0x7f80a8077b00 con 0x7f80a811d9f0 2019-12-05T05:34:24.157 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:24.125+0000 7f80a659c700 1 --2- 172.21.15.42:0/2560471304 >> v2:172.21.15.42:3300/0 conn(0x7f80a8116fd0 0x7f80a811b950 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f8098001d10 tx=0x7f809c0086a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:24.157 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:24.125+0000 7f80a659c700 1 --2- 172.21.15.42:0/2560471304 >> v2:172.21.15.42:3300/0 conn(0x7f80a8116fd0 0x7f80a811b950 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f8098001d10 tx=0x7f809c0086a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:24.158 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:24.125+0000 7f80a659c700 1 --2- 172.21.15.42:0/2560471304 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f80a8116fd0 0x7f80a811b950 secure :-1 s=READY pgs=1109 cs=0 l=1 rx=0x7f8098001d10 tx=0x7f809c0086a0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:34:24.158 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:24.125+0000 7f80a659c700 1 --2- 172.21.15.42:0/2560471304 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f80a8116fd0 0x7f80a811b950 secure :-1 s=THROTTLE_DONE pgs=1109 cs=0 l=1 rx=0x7f8098001d10 tx=0x7f809c0086a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:24.158 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:24.125+0000 7f80977fe700 1 -- 172.21.15.42:0/2560471304 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f809c007340 con 0x7f80a8116fd0 2019-12-05T05:34:24.158 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:24.125+0000 7f80a659c700 1 --2- 172.21.15.42:0/2560471304 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f80a8116fd0 0x7f80a811b950 secure :-1 s=THROTTLE_DONE pgs=1109 cs=0 l=1 rx=0x7f8098001d10 tx=0x7f809c0086a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:24.158 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:24.125+0000 7f80977fe700 1 -- 172.21.15.42:0/2560471304 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f809c008b80 con 0x7f80a8116fd0 2019-12-05T05:34:24.158 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:24.125+0000 7f80a659c700 1 --2- 172.21.15.42:0/2560471304 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f80a8116fd0 0x7f80a811b950 secure :-1 s=THROTTLE_DONE pgs=1109 cs=0 l=1 rx=0x7f8098001d10 tx=0x7f809c0086a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:24.158 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:24.125+0000 7f80acfc9700 1 -- 172.21.15.42:0/2560471304 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80a8078f70 con 0x7f80a8116fd0 2019-12-05T05:34:24.159 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:24.125+0000 7f80977fe700 1 -- 172.21.15.42:0/2560471304 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f809c011b00 con 0x7f80a8116fd0 2019-12-05T05:34:24.159 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:24.125+0000 7f80acfc9700 1 -- 172.21.15.42:0/2560471304 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80a8079470 con 0x7f80a8116fd0 2019-12-05T05:34:24.159 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:24.125+0000 7f80a659c700 1 --2- 172.21.15.42:0/2560471304 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f80a8116fd0 0x7f80a811b950 secure :-1 s=THROTTLE_DONE pgs=1109 cs=0 l=1 rx=0x7f8098001d10 tx=0x7f809c0086a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:24.159 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:24.125+0000 7f80977fe700 1 -- 172.21.15.42:0/2560471304 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f809c005540 con 0x7f80a8116fd0 2019-12-05T05:34:24.160 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:24.125+0000 7f80a659c700 1 --2- 172.21.15.42:0/2560471304 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f80a8116fd0 0x7f80a811b950 secure :-1 s=THROTTLE_DONE pgs=1109 cs=0 l=1 rx=0x7f8098001d10 tx=0x7f809c0086a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:24.160 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:24.125+0000 7f80977fe700 1 --2- 172.21.15.42:0/2560471304 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f808c03ac70 0x7f808c03d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:34:24.160 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:24.125+0000 7f80977fe700 1 -- 172.21.15.42:0/2560471304 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 7837+0+0 (secure 0 0 0) 0x7f809c04b3d0 con 0x7f80a8116fd0 2019-12-05T05:34:24.160 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:24.125+0000 7f80a6d9d700 1 --2- 172.21.15.42:0/2560471304 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f808c03ac70 0x7f808c03d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:34:24.160 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:24.129+0000 7f80a6d9d700 1 --2- 172.21.15.42:0/2560471304 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f808c03ac70 0x7f808c03d180 crc :-1 s=READY pgs=481 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:34:24.160 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:24.129+0000 7f80acfc9700 1 -- 172.21.15.42:0/2560471304 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80a805d110 con 0x7f80a8116fd0 2019-12-05T05:34:24.160 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:24.137+0000 7f80a659c700 1 --2- 172.21.15.42:0/2560471304 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f80a8116fd0 0x7f80a811b950 secure :-1 s=THROTTLE_DONE pgs=1109 cs=0 l=1 rx=0x7f8098001d10 tx=0x7f809c0086a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:24.161 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:24.137+0000 7f80977fe700 1 -- 172.21.15.42:0/2560471304 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f809c032430 con 0x7f80a8116fd0 2019-12-05T05:34:24.303 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:24.301+0000 7f80acfc9700 1 -- 172.21.15.42:0/2560471304 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f80a805b8c0 con 0x7f80a8116fd0 2019-12-05T05:34:24.304 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:24.301+0000 7f80a659c700 1 --2- 172.21.15.42:0/2560471304 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f80a8116fd0 0x7f80a811b950 secure :-1 s=THROTTLE_DONE pgs=1109 cs=0 l=1 rx=0x7f8098001d10 tx=0x7f809c0086a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:24.304 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:24.301+0000 7f80977fe700 1 -- 172.21.15.42:0/2560471304 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 138 v138) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f809c032690 con 0x7f80a8116fd0 2019-12-05T05:34:24.306 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 138 2019-12-05T05:34:24.307 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:34:24.307 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":138,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7221,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/3789641053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3789641053},{"type":"v1","addr":"172.21.15.40:6833","nonce":3789641053}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":138}],"filesystems":[{"mdsmap":{"epoch":137,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:32:53.767447+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6997},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6997":{"gid":6997,"name":"a","rank":0,"incarnation":134,"state":"up:active","state_seq":17,"addr":"172.21.15.42:6835/1601334502","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1601334502},{"type":"v1","addr":"172.21.15.42:6835","nonce":1601334502}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:34:24.309 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:24.305+0000 7f80acfc9700 1 -- 172.21.15.42:0/2560471304 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f808c03ac70 msgr2=0x7f808c03d180 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:34:24.309 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:24.305+0000 7f80acfc9700 1 --2- 172.21.15.42:0/2560471304 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f808c03ac70 0x7f808c03d180 crc :-1 s=READY pgs=481 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:34:24.309 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:24.305+0000 7f80acfc9700 1 -- 172.21.15.42:0/2560471304 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f80a8116fd0 msgr2=0x7f80a811b950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:34:24.310 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:24.305+0000 7f80acfc9700 1 --2- 172.21.15.42:0/2560471304 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f80a8116fd0 0x7f80a811b950 secure :-1 s=READY pgs=1109 cs=0 l=1 rx=0x7f8098001d10 tx=0x7f809c0086a0).stop 2019-12-05T05:34:24.310 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:24.305+0000 7f80acfc9700 1 -- 172.21.15.42:0/2560471304 shutdown_connections 2019-12-05T05:34:24.310 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:24.305+0000 7f80acfc9700 1 --2- 172.21.15.42:0/2560471304 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f808c03ac70 0x7f808c03d180 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:34:24.310 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:24.305+0000 7f80acfc9700 1 --2- 172.21.15.42:0/2560471304 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f80a8116fd0 0x7f80a811b950 unknown :-1 s=CLOSED pgs=1109 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:34:24.310 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:24.305+0000 7f80acfc9700 1 -- 172.21.15.42:0/2560471304 shutdown_connections 2019-12-05T05:34:24.310 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:24.305+0000 7f80acfc9700 1 -- 172.21.15.42:0/2560471304 wait complete. 2019-12-05T05:34:24.310 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:24.305+0000 7f80acfc9700 1 -- 172.21.15.42:0/2560471304 >> 172.21.15.42:0/2560471304 conn(0x7f80a80aabe0 msgr2=0x7f80a808eb70 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:34:24.319 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T05:34:26.320 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:34:26.429 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:26.417+0000 7f491e63c700 1 Processor -- start 2019-12-05T05:34:26.429 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:26.417+0000 7f491e63c700 1 -- start start 2019-12-05T05:34:26.429 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:26.421+0000 7f491e63c700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f491808df70 0x7f491809f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:34:26.429 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:26.421+0000 7f491e63c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f491809f820 0x7f491814efd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:34:26.429 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:26.421+0000 7f491e63c700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f491809fc50 con 0x7f49180a7730 2019-12-05T05:34:26.430 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:26.421+0000 7f491e63c700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49180a0af0 con 0x7f491809f820 2019-12-05T05:34:26.430 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:26.421+0000 7f491e63c700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f491808e5a0 con 0x7f491808df70 2019-12-05T05:34:26.430 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:26.421+0000 7f491d3da700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f491809f820 0x7f491814efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:34:26.430 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:26.421+0000 7f4917fff700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f491808df70 0x7f491809f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:34:26.430 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:26.421+0000 7f491d3da700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f491809f820 0x7f491814efd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:36678/0 (socket says 172.21.15.42:36678) 2019-12-05T05:34:26.430 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:26.421+0000 7f491d3da700 1 -- 172.21.15.42:0/2249885884 learned_addr learned my addr 172.21.15.42:0/2249885884 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:34:26.430 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:26.421+0000 7f49177fe700 1 -- 172.21.15.42:0/2249885884 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 613984131 0 0) 0x7f491809fc50 con 0x7f49180a7730 2019-12-05T05:34:26.431 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:26.421+0000 7f4917fff700 1 -- 172.21.15.42:0/2249885884 >> v2:172.21.15.42:3300/0 conn(0x7f491809f820 msgr2=0x7f491814efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:34:26.431 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:26.421+0000 7f4917fff700 1 --2- 172.21.15.42:0/2249885884 >> v2:172.21.15.42:3300/0 conn(0x7f491809f820 0x7f491814efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:34:26.431 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:26.421+0000 7f4917fff700 1 -- 172.21.15.42:0/2249885884 >> v1:172.21.15.40:6789/0 conn(0x7f49180a7730 legacy=0x7f491807e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:34:26.431 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:26.421+0000 7f4917fff700 1 -- 172.21.15.42:0/2249885884 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49180a0700 con 0x7f491808df70 2019-12-05T05:34:26.431 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:26.421+0000 7f491d3da700 1 --2- 172.21.15.42:0/2249885884 >> v2:172.21.15.42:3300/0 conn(0x7f491809f820 0x7f491814efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:34:26.431 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:26.421+0000 7f4917fff700 1 --2- 172.21.15.42:0/2249885884 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f491808df70 0x7f491809f2d0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f490c008790 tx=0x7f490c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:26.431 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:26.421+0000 7f4917fff700 1 --2- 172.21.15.42:0/2249885884 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f491808df70 0x7f491809f2d0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f490c008790 tx=0x7f490c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:26.432 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:26.421+0000 7f4917fff700 1 --2- 172.21.15.42:0/2249885884 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f491808df70 0x7f491809f2d0 secure :-1 s=READY pgs=230 cs=0 l=1 rx=0x7f490c008790 tx=0x7f490c00fc70).ready entity=mon.2 client_cookie=690733330b4999b8 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:34:26.432 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:26.421+0000 7f4917fff700 1 --2- 172.21.15.42:0/2249885884 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f491808df70 0x7f491809f2d0 secure :-1 s=THROTTLE_DONE pgs=230 cs=0 l=1 rx=0x7f490c008790 tx=0x7f490c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:26.432 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:26.421+0000 7f49177fe700 1 -- 172.21.15.42:0/2249885884 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f490c008bb0 con 0x7f491808df70 2019-12-05T05:34:26.432 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:26.421+0000 7f4917fff700 1 --2- 172.21.15.42:0/2249885884 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f491808df70 0x7f491809f2d0 secure :-1 s=THROTTLE_DONE pgs=230 cs=0 l=1 rx=0x7f490c008790 tx=0x7f490c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:26.433 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:26.421+0000 7f4917fff700 1 --2- 172.21.15.42:0/2249885884 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f491808df70 0x7f491809f2d0 secure :-1 s=THROTTLE_DONE pgs=230 cs=0 l=1 rx=0x7f490c008790 tx=0x7f490c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:26.433 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:26.421+0000 7f49177fe700 1 -- 172.21.15.42:0/2249885884 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f490c0087d0 con 0x7f491808df70 2019-12-05T05:34:26.433 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:26.421+0000 7f49177fe700 1 -- 172.21.15.42:0/2249885884 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f490c011860 con 0x7f491808df70 2019-12-05T05:34:26.433 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:26.421+0000 7f491e63c700 1 -- 172.21.15.42:0/2249885884 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f491808df70 msgr2=0x7f491809f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:34:26.433 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:26.421+0000 7f491e63c700 1 --2- 172.21.15.42:0/2249885884 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f491808df70 0x7f491809f2d0 secure :-1 s=READY pgs=230 cs=0 l=1 rx=0x7f490c008790 tx=0x7f490c00fc70).stop 2019-12-05T05:34:26.433 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:26.425+0000 7f491e63c700 1 -- 172.21.15.42:0/2249885884 shutdown_connections 2019-12-05T05:34:26.433 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:26.425+0000 7f491e63c700 1 --2- 172.21.15.42:0/2249885884 >> v2:172.21.15.42:3300/0 conn(0x7f491809f820 0x7f491814efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:34:26.434 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:26.425+0000 7f491e63c700 1 --2- 172.21.15.42:0/2249885884 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f491808df70 0x7f491809f2d0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:34:26.434 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:26.425+0000 7f491e63c700 1 -- 172.21.15.42:0/2249885884 shutdown_connections 2019-12-05T05:34:26.434 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:26.425+0000 7f491e63c700 1 -- 172.21.15.42:0/2249885884 wait complete. 2019-12-05T05:34:26.434 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:26.425+0000 7f491e63c700 1 -- 172.21.15.42:0/2249885884 >> 172.21.15.42:0/2249885884 conn(0x7f49180aabd0 msgr2=0x7f49180a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:34:26.438 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:26.425+0000 7f491e63c700 1 Processor -- start 2019-12-05T05:34:26.439 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:26.425+0000 7f491e63c700 1 -- start start 2019-12-05T05:34:26.439 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:26.429+0000 7f491e63c700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f491808eea0 0x7f49180a6c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:34:26.439 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:26.429+0000 7f491e63c700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49181221e0 con 0x7f491807ada0 2019-12-05T05:34:26.439 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:26.429+0000 7f491e63c700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f491807b1d0 con 0x7f49181239b0 2019-12-05T05:34:26.439 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:26.429+0000 7f491e63c700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49181245c0 con 0x7f491808eea0 2019-12-05T05:34:26.439 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:26.429+0000 7f491cbd9700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f49181239b0 0x7f4918124830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:36718/0 (socket says 172.21.15.42:36718) 2019-12-05T05:34:26.440 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:26.429+0000 7f491cbd9700 1 -- 172.21.15.42:0/2035379251 learned_addr learned my addr 172.21.15.42:0/2035379251 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:34:26.440 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:26.429+0000 7f4917fff700 1 --2- 172.21.15.42:0/2035379251 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f491808eea0 0x7f49180a6c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:34:26.440 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:26.429+0000 7f49157fa700 1 -- 172.21.15.42:0/2035379251 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3577239692 0 0) 0x7f491807b1d0 con 0x7f49181239b0 2019-12-05T05:34:26.440 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:26.429+0000 7f49157fa700 1 -- 172.21.15.42:0/2035379251 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48f80030b0 con 0x7f49181239b0 2019-12-05T05:34:26.440 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:26.429+0000 7f49157fa700 1 -- 172.21.15.42:0/2035379251 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1944790810 0 0) 0x7f49181221e0 con 0x7f491807ada0 2019-12-05T05:34:26.441 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:26.429+0000 7f49157fa700 1 -- 172.21.15.42:0/2035379251 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f491807b1d0 con 0x7f491807ada0 2019-12-05T05:34:26.441 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:26.429+0000 7f4917fff700 1 -- 172.21.15.42:0/2035379251 >> v1:172.21.15.42:6789/0 conn(0x7f49181239b0 legacy=0x7f4918124830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:34:26.441 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:26.429+0000 7f4917fff700 1 -- 172.21.15.42:0/2035379251 >> v1:172.21.15.40:6789/0 conn(0x7f491807ada0 legacy=0x7f49181388c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:34:26.441 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:26.429+0000 7f4917fff700 1 -- 172.21.15.42:0/2035379251 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f491807b400 con 0x7f491808eea0 2019-12-05T05:34:26.441 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:26.429+0000 7f4917fff700 1 --2- 172.21.15.42:0/2035379251 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f491808eea0 0x7f49180a6c50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f490c008790 tx=0x7f490c011920).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:26.441 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:26.429+0000 7f4917fff700 1 --2- 172.21.15.42:0/2035379251 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f491808eea0 0x7f49180a6c50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f490c008790 tx=0x7f490c011920).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:26.442 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:26.429+0000 7f4917fff700 1 --2- 172.21.15.42:0/2035379251 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f491808eea0 0x7f49180a6c50 secure :-1 s=READY pgs=231 cs=0 l=1 rx=0x7f490c008790 tx=0x7f490c011920).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:34:26.442 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:26.429+0000 7f4917fff700 1 --2- 172.21.15.42:0/2035379251 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f491808eea0 0x7f49180a6c50 secure :-1 s=THROTTLE_DONE pgs=231 cs=0 l=1 rx=0x7f490c008790 tx=0x7f490c011920).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:26.442 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:26.429+0000 7f49157fa700 1 -- 172.21.15.42:0/2035379251 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f490c00f280 con 0x7f491808eea0 2019-12-05T05:34:26.442 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:26.429+0000 7f4917fff700 1 --2- 172.21.15.42:0/2035379251 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f491808eea0 0x7f49180a6c50 secure :-1 s=THROTTLE_DONE pgs=231 cs=0 l=1 rx=0x7f490c008790 tx=0x7f490c011920).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:26.442 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:26.429+0000 7f4917fff700 1 --2- 172.21.15.42:0/2035379251 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f491808eea0 0x7f49180a6c50 secure :-1 s=THROTTLE_DONE pgs=231 cs=0 l=1 rx=0x7f490c008790 tx=0x7f490c011920).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:26.442 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:26.429+0000 7f49157fa700 1 -- 172.21.15.42:0/2035379251 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f490c008db0 con 0x7f491808eea0 2019-12-05T05:34:26.442 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:26.429+0000 7f49157fa700 1 -- 172.21.15.42:0/2035379251 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f490c005540 con 0x7f491808eea0 2019-12-05T05:34:26.443 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:26.429+0000 7f491e63c700 1 -- 172.21.15.42:0/2035379251 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f491813a0d0 con 0x7f491808eea0 2019-12-05T05:34:26.443 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:26.429+0000 7f491e63c700 1 -- 172.21.15.42:0/2035379251 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f491813a5d0 con 0x7f491808eea0 2019-12-05T05:34:26.443 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:26.429+0000 7f4917fff700 1 --2- 172.21.15.42:0/2035379251 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f491808eea0 0x7f49180a6c50 secure :-1 s=THROTTLE_DONE pgs=231 cs=0 l=1 rx=0x7f490c008790 tx=0x7f490c011920).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:26.443 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:26.433+0000 7f49157fa700 1 -- 172.21.15.42:0/2035379251 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f490c005820 con 0x7f491808eea0 2019-12-05T05:34:26.443 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:26.433+0000 7f4917fff700 1 --2- 172.21.15.42:0/2035379251 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f491808eea0 0x7f49180a6c50 secure :-1 s=THROTTLE_DONE pgs=231 cs=0 l=1 rx=0x7f490c008790 tx=0x7f490c011920).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:26.443 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:26.433+0000 7f49157fa700 1 --2- 172.21.15.42:0/2035379251 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f48f803b460 0x7f48f803d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:34:26.443 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:26.433+0000 7f49157fa700 1 -- 172.21.15.42:0/2035379251 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 7837+0+0 (secure 0 0 0) 0x7f490c04b490 con 0x7f491808eea0 2019-12-05T05:34:26.444 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:26.433+0000 7f491cbd9700 1 --2- 172.21.15.42:0/2035379251 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f48f803b460 0x7f48f803d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:34:26.444 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:26.433+0000 7f491cbd9700 1 --2- 172.21.15.42:0/2035379251 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f48f803b460 0x7f48f803d970 crc :-1 s=READY pgs=482 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:34:26.444 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:26.433+0000 7f491e63c700 1 -- 172.21.15.42:0/2035379251 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49180a71a0 con 0x7f491808eea0 2019-12-05T05:34:26.448 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:26.441+0000 7f4917fff700 1 --2- 172.21.15.42:0/2035379251 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f491808eea0 0x7f49180a6c50 secure :-1 s=THROTTLE_DONE pgs=231 cs=0 l=1 rx=0x7f490c008790 tx=0x7f490c011920).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:26.454 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:26.441+0000 7f49157fa700 1 -- 172.21.15.42:0/2035379251 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f490c0324f0 con 0x7f491808eea0 2019-12-05T05:34:26.617 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:26.613+0000 7f491e63c700 1 -- 172.21.15.42:0/2035379251 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f491805d100 con 0x7f491808eea0 2019-12-05T05:34:26.618 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:26.613+0000 7f4917fff700 1 --2- 172.21.15.42:0/2035379251 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f491808eea0 0x7f49180a6c50 secure :-1 s=THROTTLE_DONE pgs=231 cs=0 l=1 rx=0x7f490c008790 tx=0x7f490c011920).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:26.618 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:26.613+0000 7f49157fa700 1 -- 172.21.15.42:0/2035379251 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 138 v138) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f490c007f30 con 0x7f491808eea0 2019-12-05T05:34:26.622 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 138 2019-12-05T05:34:26.622 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:34:26.622 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":138,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7221,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/3789641053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3789641053},{"type":"v1","addr":"172.21.15.40:6833","nonce":3789641053}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":138}],"filesystems":[{"mdsmap":{"epoch":137,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:32:53.767447+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6997},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6997":{"gid":6997,"name":"a","rank":0,"incarnation":134,"state":"up:active","state_seq":17,"addr":"172.21.15.42:6835/1601334502","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1601334502},{"type":"v1","addr":"172.21.15.42:6835","nonce":1601334502}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:34:26.623 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:26.617+0000 7f491e63c700 1 -- 172.21.15.42:0/2035379251 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f48f803b460 msgr2=0x7f48f803d970 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:34:26.624 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:26.617+0000 7f491e63c700 1 --2- 172.21.15.42:0/2035379251 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f48f803b460 0x7f48f803d970 crc :-1 s=READY pgs=482 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:34:26.624 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:26.617+0000 7f491e63c700 1 -- 172.21.15.42:0/2035379251 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f491808eea0 msgr2=0x7f49180a6c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:34:26.624 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:26.617+0000 7f491e63c700 1 --2- 172.21.15.42:0/2035379251 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f491808eea0 0x7f49180a6c50 secure :-1 s=READY pgs=231 cs=0 l=1 rx=0x7f490c008790 tx=0x7f490c011920).stop 2019-12-05T05:34:26.625 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:26.617+0000 7f491e63c700 1 -- 172.21.15.42:0/2035379251 shutdown_connections 2019-12-05T05:34:26.625 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:26.617+0000 7f491e63c700 1 --2- 172.21.15.42:0/2035379251 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f48f803b460 0x7f48f803d970 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:34:26.625 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:26.617+0000 7f491e63c700 1 --2- 172.21.15.42:0/2035379251 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f491808eea0 0x7f49180a6c50 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:34:26.625 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:26.617+0000 7f491e63c700 1 -- 172.21.15.42:0/2035379251 shutdown_connections 2019-12-05T05:34:26.626 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:26.617+0000 7f491e63c700 1 -- 172.21.15.42:0/2035379251 wait complete. 2019-12-05T05:34:26.626 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:26.621+0000 7f491e63c700 1 -- 172.21.15.42:0/2035379251 >> 172.21.15.42:0/2035379251 conn(0x7f4918066fd0 msgr2=0x7f491811c8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:34:26.634 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T05:34:28.634 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:34:28.741 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:28.729+0000 7f5a0cda4700 1 Processor -- start 2019-12-05T05:34:28.741 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:28.729+0000 7f5a0cda4700 1 -- start start 2019-12-05T05:34:28.741 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:28.733+0000 7f5a0cda4700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5a0808df80 0x7f5a0809f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:34:28.741 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:28.733+0000 7f5a0cda4700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5a0809f830 0x7f5a0814efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:34:28.742 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:28.733+0000 7f5a0cda4700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a0809fc60 con 0x7f5a080a7740 2019-12-05T05:34:28.742 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:28.733+0000 7f5a0cda4700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a080a0b00 con 0x7f5a0809f830 2019-12-05T05:34:28.742 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:28.733+0000 7f5a0cda4700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a0808e5b0 con 0x7f5a0808df80 2019-12-05T05:34:28.742 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:28.733+0000 7f59fe59c700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5a0808df80 0x7f5a0809f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:34:28.742 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:28.733+0000 7f59fe59c700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5a0808df80 0x7f5a0809f2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:37158/0 (socket says 172.21.15.42:37158) 2019-12-05T05:34:28.742 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:28.733+0000 7f59fe59c700 1 -- 172.21.15.42:0/2634373787 learned_addr learned my addr 172.21.15.42:0/2634373787 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:34:28.742 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:28.733+0000 7f5a0759e700 1 --2- 172.21.15.42:0/2634373787 >> v2:172.21.15.40:3300/0 conn(0x7f5a0809f830 0x7f5a0814efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:34:28.743 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:28.733+0000 7f59fe59c700 1 -- 172.21.15.42:0/2634373787 >> v2:172.21.15.40:3300/0 conn(0x7f5a0809f830 msgr2=0x7f5a0814efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:34:28.743 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:28.733+0000 7f59fe59c700 1 --2- 172.21.15.42:0/2634373787 >> v2:172.21.15.40:3300/0 conn(0x7f5a0809f830 0x7f5a0814efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:34:28.743 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:28.733+0000 7f59fe59c700 1 -- 172.21.15.42:0/2634373787 >> v1:172.21.15.40:6789/0 conn(0x7f5a080a7740 legacy=0x7f5a0807e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:34:28.743 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:28.733+0000 7f59fe59c700 1 -- 172.21.15.42:0/2634373787 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a080a0710 con 0x7f5a0808df80 2019-12-05T05:34:28.743 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:28.733+0000 7f59fe59c700 1 --2- 172.21.15.42:0/2634373787 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5a0808df80 0x7f5a0809f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f59f0008790 tx=0x7f59f000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:28.743 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:28.733+0000 7f59fe59c700 1 --2- 172.21.15.42:0/2634373787 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5a0808df80 0x7f5a0809f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f59f0008790 tx=0x7f59f000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:28.743 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:28.733+0000 7f59fe59c700 1 --2- 172.21.15.42:0/2634373787 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5a0808df80 0x7f5a0809f2e0 secure :-1 s=READY pgs=232 cs=0 l=1 rx=0x7f59f0008790 tx=0x7f59f000fd30).ready entity=mon.2 client_cookie=c5d54549521dbd84 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:34:28.744 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:28.733+0000 7f59fe59c700 1 --2- 172.21.15.42:0/2634373787 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5a0808df80 0x7f5a0809f2e0 secure :-1 s=THROTTLE_DONE pgs=232 cs=0 l=1 rx=0x7f59f0008790 tx=0x7f59f000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:28.744 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:28.733+0000 7f5a0659c700 1 -- 172.21.15.42:0/2634373787 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f59f0008c90 con 0x7f5a0808df80 2019-12-05T05:34:28.744 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:28.733+0000 7f59fe59c700 1 --2- 172.21.15.42:0/2634373787 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5a0808df80 0x7f5a0809f2e0 secure :-1 s=THROTTLE_DONE pgs=232 cs=0 l=1 rx=0x7f59f0008790 tx=0x7f59f000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:28.744 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:28.733+0000 7f5a0659c700 1 -- 172.21.15.42:0/2634373787 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59f00088b0 con 0x7f5a0808df80 2019-12-05T05:34:28.744 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:28.733+0000 7f59fe59c700 1 --2- 172.21.15.42:0/2634373787 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5a0808df80 0x7f5a0809f2e0 secure :-1 s=THROTTLE_DONE pgs=232 cs=0 l=1 rx=0x7f59f0008790 tx=0x7f59f000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:28.744 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:28.733+0000 7f5a0659c700 1 -- 172.21.15.42:0/2634373787 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f59f0011b10 con 0x7f5a0808df80 2019-12-05T05:34:28.745 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:28.733+0000 7f5a0cda4700 1 -- 172.21.15.42:0/2634373787 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5a0808df80 msgr2=0x7f5a0809f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:34:28.745 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:28.733+0000 7f5a0cda4700 1 --2- 172.21.15.42:0/2634373787 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5a0808df80 0x7f5a0809f2e0 secure :-1 s=READY pgs=232 cs=0 l=1 rx=0x7f59f0008790 tx=0x7f59f000fd30).stop 2019-12-05T05:34:28.745 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:28.737+0000 7f5a0cda4700 1 -- 172.21.15.42:0/2634373787 shutdown_connections 2019-12-05T05:34:28.745 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:28.737+0000 7f5a0cda4700 1 --2- 172.21.15.42:0/2634373787 >> v2:172.21.15.40:3300/0 conn(0x7f5a0809f830 0x7f5a0814efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:34:28.745 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:28.737+0000 7f5a0cda4700 1 --2- 172.21.15.42:0/2634373787 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5a0808df80 0x7f5a0809f2e0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:34:28.745 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:28.737+0000 7f5a0cda4700 1 -- 172.21.15.42:0/2634373787 shutdown_connections 2019-12-05T05:34:28.745 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:28.737+0000 7f5a0cda4700 1 -- 172.21.15.42:0/2634373787 wait complete. 2019-12-05T05:34:28.745 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:28.737+0000 7f5a0cda4700 1 -- 172.21.15.42:0/2634373787 >> 172.21.15.42:0/2634373787 conn(0x7f5a080aabe0 msgr2=0x7f5a080a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:34:28.749 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:28.737+0000 7f5a0cda4700 1 Processor -- start 2019-12-05T05:34:28.749 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:28.737+0000 7f5a0cda4700 1 -- start start 2019-12-05T05:34:28.749 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:28.741+0000 7f5a0cda4700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5a080a5040 0x7f5a080a1bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:34:28.749 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:28.741+0000 7f5a0cda4700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5a08067320 0x7f5a080a5db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:34:28.749 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:28.741+0000 7f5a0cda4700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a08119cc0 con 0x7f5a080a6300 2019-12-05T05:34:28.749 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:28.741+0000 7f5a0cda4700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a080a23c0 con 0x7f5a080a5040 2019-12-05T05:34:28.750 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:28.741+0000 7f5a0cda4700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a080a5b10 con 0x7f5a08067320 2019-12-05T05:34:28.750 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:28.741+0000 7f59fe59c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5a08067320 0x7f5a080a5db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:34:28.750 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:28.741+0000 7f59fe59c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5a08067320 0x7f5a080a5db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:36698/0 (socket says 172.21.15.42:36698) 2019-12-05T05:34:28.750 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:28.741+0000 7f59fe59c700 1 -- 172.21.15.42:0/1710741936 learned_addr learned my addr 172.21.15.42:0/1710741936 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:34:28.750 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:28.741+0000 7f5a06d9d700 1 --2- 172.21.15.42:0/1710741936 >> v2:172.21.15.40:3300/0 conn(0x7f5a080a5040 0x7f5a080a1bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:34:28.751 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:28.741+0000 7f59fe59c700 1 -- 172.21.15.42:0/1710741936 >> v2:172.21.15.40:3300/0 conn(0x7f5a080a5040 msgr2=0x7f5a080a1bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:34:28.751 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:28.741+0000 7f59fe59c700 1 --2- 172.21.15.42:0/1710741936 >> v2:172.21.15.40:3300/0 conn(0x7f5a080a5040 0x7f5a080a1bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:34:28.751 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:28.741+0000 7f59fe59c700 1 -- 172.21.15.42:0/1710741936 >> v1:172.21.15.42:6789/0 conn(0x7f5a080a6300 legacy=0x7f5a0811a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:34:28.751 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:28.741+0000 7f59fe59c700 1 -- 172.21.15.42:0/1710741936 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a0811aa00 con 0x7f5a08067320 2019-12-05T05:34:28.751 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:28.741+0000 7f59fe59c700 1 --2- 172.21.15.42:0/1710741936 >> v2:172.21.15.42:3300/0 conn(0x7f5a08067320 0x7f5a080a5db0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f5a08112e70 tx=0x7f59f0011920).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:28.751 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:28.741+0000 7f59fe59c700 1 --2- 172.21.15.42:0/1710741936 >> v2:172.21.15.42:3300/0 conn(0x7f5a08067320 0x7f5a080a5db0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f5a08112e70 tx=0x7f59f0011920).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:28.751 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:28.741+0000 7f59fe59c700 1 --2- 172.21.15.42:0/1710741936 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5a08067320 0x7f5a080a5db0 secure :-1 s=READY pgs=1115 cs=0 l=1 rx=0x7f5a08112e70 tx=0x7f59f0011920).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:34:28.752 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:28.741+0000 7f59fe59c700 1 --2- 172.21.15.42:0/1710741936 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5a08067320 0x7f5a080a5db0 secure :-1 s=THROTTLE_DONE pgs=1115 cs=0 l=1 rx=0x7f5a08112e70 tx=0x7f59f0011920).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:28.752 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:28.741+0000 7f59fffff700 1 -- 172.21.15.42:0/1710741936 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f59f000f490 con 0x7f5a08067320 2019-12-05T05:34:28.752 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:28.741+0000 7f59fe59c700 1 --2- 172.21.15.42:0/1710741936 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5a08067320 0x7f5a080a5db0 secure :-1 s=THROTTLE_DONE pgs=1115 cs=0 l=1 rx=0x7f5a08112e70 tx=0x7f59f0011920).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:28.752 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:28.741+0000 7f59fe59c700 1 --2- 172.21.15.42:0/1710741936 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5a08067320 0x7f5a080a5db0 secure :-1 s=THROTTLE_DONE pgs=1115 cs=0 l=1 rx=0x7f5a08112e70 tx=0x7f59f0011920).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:28.752 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:28.741+0000 7f59fffff700 1 -- 172.21.15.42:0/1710741936 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59f0011730 con 0x7f5a08067320 2019-12-05T05:34:28.752 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:28.741+0000 7f5a0cda4700 1 -- 172.21.15.42:0/1710741936 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a0811acd0 con 0x7f5a08067320 2019-12-05T05:34:28.753 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:28.741+0000 7f59fffff700 1 -- 172.21.15.42:0/1710741936 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f59f0005730 con 0x7f5a08067320 2019-12-05T05:34:28.753 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:28.741+0000 7f5a0cda4700 1 -- 172.21.15.42:0/1710741936 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a08137a70 con 0x7f5a08067320 2019-12-05T05:34:28.753 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:28.741+0000 7f59fe59c700 1 --2- 172.21.15.42:0/1710741936 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5a08067320 0x7f5a080a5db0 secure :-1 s=THROTTLE_DONE pgs=1115 cs=0 l=1 rx=0x7f5a08112e70 tx=0x7f59f0011920).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:28.753 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:28.741+0000 7f59fffff700 1 -- 172.21.15.42:0/1710741936 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f59f0005a40 con 0x7f5a08067320 2019-12-05T05:34:28.753 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:28.741+0000 7f59fe59c700 1 --2- 172.21.15.42:0/1710741936 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5a08067320 0x7f5a080a5db0 secure :-1 s=THROTTLE_DONE pgs=1115 cs=0 l=1 rx=0x7f5a08112e70 tx=0x7f59f0011920).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:28.753 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:28.741+0000 7f59fffff700 1 --2- 172.21.15.42:0/1710741936 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f59ec03ae40 0x7f59ec03d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:34:28.753 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:28.741+0000 7f59fffff700 1 -- 172.21.15.42:0/1710741936 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 7837+0+0 (secure 0 0 0) 0x7f59f008f5e0 con 0x7f5a08067320 2019-12-05T05:34:28.753 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:28.745+0000 7f5a06d9d700 1 --2- 172.21.15.42:0/1710741936 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f59ec03ae40 0x7f59ec03d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:34:28.754 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:28.745+0000 7f5a06d9d700 1 --2- 172.21.15.42:0/1710741936 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f59ec03ae40 0x7f59ec03d350 crc :-1 s=READY pgs=483 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:34:28.754 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:28.745+0000 7f5a0cda4700 1 -- 172.21.15.42:0/1710741936 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a0805d110 con 0x7f5a08067320 2019-12-05T05:34:28.758 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:28.753+0000 7f59fe59c700 1 --2- 172.21.15.42:0/1710741936 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5a08067320 0x7f5a080a5db0 secure :-1 s=THROTTLE_DONE pgs=1115 cs=0 l=1 rx=0x7f5a08112e70 tx=0x7f59f0011920).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:28.758 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:28.753+0000 7f59fffff700 1 -- 172.21.15.42:0/1710741936 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f59f0005d50 con 0x7f5a08067320 2019-12-05T05:34:28.931 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:28.925+0000 7f5a0cda4700 1 -- 172.21.15.42:0/1710741936 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5a0805b8c0 con 0x7f5a08067320 2019-12-05T05:34:28.932 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:28.929+0000 7f59fe59c700 1 --2- 172.21.15.42:0/1710741936 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5a08067320 0x7f5a080a5db0 secure :-1 s=THROTTLE_DONE pgs=1115 cs=0 l=1 rx=0x7f5a08112e70 tx=0x7f59f0011920).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:28.932 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:28.929+0000 7f59fffff700 1 -- 172.21.15.42:0/1710741936 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 138 v138) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f59f0074290 con 0x7f5a08067320 2019-12-05T05:34:28.934 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:34:28.934 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":138,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7221,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/3789641053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3789641053},{"type":"v1","addr":"172.21.15.40:6833","nonce":3789641053}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":138}],"filesystems":[{"mdsmap":{"epoch":137,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:32:53.767447+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6997},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6997":{"gid":6997,"name":"a","rank":0,"incarnation":134,"state":"up:active","state_seq":17,"addr":"172.21.15.42:6835/1601334502","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1601334502},{"type":"v1","addr":"172.21.15.42:6835","nonce":1601334502}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:34:28.934 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 138 2019-12-05T05:34:28.938 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:28.933+0000 7f5a0cda4700 1 -- 172.21.15.42:0/1710741936 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f59ec03ae40 msgr2=0x7f59ec03d350 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:34:28.938 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:28.933+0000 7f5a0cda4700 1 --2- 172.21.15.42:0/1710741936 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f59ec03ae40 0x7f59ec03d350 crc :-1 s=READY pgs=483 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:34:28.938 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:28.933+0000 7f5a0cda4700 1 -- 172.21.15.42:0/1710741936 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5a08067320 msgr2=0x7f5a080a5db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:34:28.938 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:28.933+0000 7f5a0cda4700 1 --2- 172.21.15.42:0/1710741936 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5a08067320 0x7f5a080a5db0 secure :-1 s=READY pgs=1115 cs=0 l=1 rx=0x7f5a08112e70 tx=0x7f59f0011920).stop 2019-12-05T05:34:28.938 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:28.933+0000 7f5a0cda4700 1 -- 172.21.15.42:0/1710741936 shutdown_connections 2019-12-05T05:34:28.938 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:28.933+0000 7f5a0cda4700 1 --2- 172.21.15.42:0/1710741936 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f59ec03ae40 0x7f59ec03d350 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:34:28.938 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:28.933+0000 7f5a0cda4700 1 --2- 172.21.15.42:0/1710741936 >> v2:172.21.15.40:3300/0 conn(0x7f5a080a5040 0x7f5a080a1bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:34:28.939 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:28.933+0000 7f5a0cda4700 1 --2- 172.21.15.42:0/1710741936 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5a08067320 0x7f5a080a5db0 unknown :-1 s=CLOSED pgs=1115 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:34:28.939 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:28.933+0000 7f5a0cda4700 1 -- 172.21.15.42:0/1710741936 shutdown_connections 2019-12-05T05:34:28.939 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:28.933+0000 7f5a0cda4700 1 -- 172.21.15.42:0/1710741936 wait complete. 2019-12-05T05:34:28.939 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:28.933+0000 7f5a0cda4700 1 -- 172.21.15.42:0/1710741936 >> 172.21.15.42:0/1710741936 conn(0x7f5a0808f4e0 msgr2=0x7f5a08059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:34:28.948 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T05:34:30.949 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:34:31.150 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:31.141+0000 7f554874f700 1 Processor -- start 2019-12-05T05:34:31.165 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:31.141+0000 7f554874f700 1 -- start start 2019-12-05T05:34:31.165 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:31.145+0000 7f554874f700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f55400a7740 0x7f554008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:34:31.165 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:31.145+0000 7f554874f700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f554009fae0 0x7f554007eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:34:31.165 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:31.145+0000 7f554874f700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55400a0710 con 0x7f554007e720 2019-12-05T05:34:31.165 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:31.145+0000 7f554874f700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55400a0b00 con 0x7f554009fae0 2019-12-05T05:34:31.165 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:31.145+0000 7f554874f700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f554008e5b0 con 0x7f55400a7740 2019-12-05T05:34:31.166 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:31.145+0000 7f5546cec700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f55400a7740 0x7f554008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:34:31.166 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:31.145+0000 7f5546cec700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f55400a7740 0x7f554008df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:36704/0 (socket says 172.21.15.42:36704) 2019-12-05T05:34:31.166 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:31.145+0000 7f5546cec700 1 -- 172.21.15.42:0/2463969272 learned_addr learned my addr 172.21.15.42:0/2463969272 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:34:31.172 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:31.145+0000 7f55464eb700 1 --1- 172.21.15.42:0/2463969272 >> v1:172.21.15.42:6789/0 conn(0x7f554007e720 0x7f554009f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:36744/0 (socket says 172.21.15.42:36744) 2019-12-05T05:34:31.172 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:31.145+0000 7f55474ed700 1 --2- 172.21.15.42:0/2463969272 >> v2:172.21.15.40:3300/0 conn(0x7f554009fae0 0x7f554007eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:34:31.172 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:31.145+0000 7f5546cec700 1 -- 172.21.15.42:0/2463969272 >> v2:172.21.15.40:3300/0 conn(0x7f554009fae0 msgr2=0x7f554007eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:34:31.173 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:31.145+0000 7f5546cec700 1 --2- 172.21.15.42:0/2463969272 >> v2:172.21.15.40:3300/0 conn(0x7f554009fae0 0x7f554007eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:34:31.173 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:31.145+0000 7f5546cec700 1 -- 172.21.15.42:0/2463969272 >> v1:172.21.15.42:6789/0 conn(0x7f554007e720 legacy=0x7f554009f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:34:31.173 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:31.145+0000 7f5546cec700 1 -- 172.21.15.42:0/2463969272 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f554007f0a0 con 0x7f55400a7740 2019-12-05T05:34:31.173 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:31.145+0000 7f55474ed700 1 --2- 172.21.15.42:0/2463969272 >> v2:172.21.15.40:3300/0 conn(0x7f554009fae0 0x7f554007eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).send_auth_request state changed! 2019-12-05T05:34:31.173 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:31.145+0000 7f5546cec700 1 --2- 172.21.15.42:0/2463969272 >> v2:172.21.15.42:3300/0 conn(0x7f55400a7740 0x7f554008df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f553c002f30 tx=0x7f553c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:31.173 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:31.145+0000 7f5546cec700 1 --2- 172.21.15.42:0/2463969272 >> v2:172.21.15.42:3300/0 conn(0x7f55400a7740 0x7f554008df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f553c002f30 tx=0x7f553c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:31.174 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:31.145+0000 7f5546cec700 1 --2- 172.21.15.42:0/2463969272 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f55400a7740 0x7f554008df80 secure :-1 s=READY pgs=1117 cs=0 l=1 rx=0x7f553c002f30 tx=0x7f553c00fd30).ready entity=mon.1 client_cookie=fed0b89376ccaca6 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:34:31.174 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:31.145+0000 7f5546cec700 1 --2- 172.21.15.42:0/2463969272 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f55400a7740 0x7f554008df80 secure :-1 s=THROTTLE_DONE pgs=1117 cs=0 l=1 rx=0x7f553c002f30 tx=0x7f553c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:31.174 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:31.145+0000 7f5546cec700 1 --2- 172.21.15.42:0/2463969272 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f55400a7740 0x7f554008df80 secure :-1 s=THROTTLE_DONE pgs=1117 cs=0 l=1 rx=0x7f553c002f30 tx=0x7f553c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:31.174 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:31.145+0000 7f5546cec700 1 --2- 172.21.15.42:0/2463969272 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f55400a7740 0x7f554008df80 secure :-1 s=THROTTLE_DONE pgs=1117 cs=0 l=1 rx=0x7f553c002f30 tx=0x7f553c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:31.174 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:31.145+0000 7f5545cea700 1 -- 172.21.15.42:0/2463969272 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f553c008c90 con 0x7f55400a7740 2019-12-05T05:34:31.174 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:31.145+0000 7f5545cea700 1 -- 172.21.15.42:0/2463969272 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f553c0088b0 con 0x7f55400a7740 2019-12-05T05:34:31.175 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:31.145+0000 7f5545cea700 1 -- 172.21.15.42:0/2463969272 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f553c011ad0 con 0x7f55400a7740 2019-12-05T05:34:31.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:31.145+0000 7f554874f700 1 -- 172.21.15.42:0/2463969272 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f55400a7740 msgr2=0x7f554008df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:34:31.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:31.145+0000 7f554874f700 1 --2- 172.21.15.42:0/2463969272 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f55400a7740 0x7f554008df80 secure :-1 s=READY pgs=1117 cs=0 l=1 rx=0x7f553c002f30 tx=0x7f553c00fd30).stop 2019-12-05T05:34:31.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:31.145+0000 7f554874f700 1 -- 172.21.15.42:0/2463969272 shutdown_connections 2019-12-05T05:34:31.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:31.145+0000 7f554874f700 1 --2- 172.21.15.42:0/2463969272 >> v2:172.21.15.40:3300/0 conn(0x7f554009fae0 0x7f554007eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:34:31.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:31.145+0000 7f554874f700 1 --2- 172.21.15.42:0/2463969272 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f55400a7740 0x7f554008df80 unknown :-1 s=CLOSED pgs=1117 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:34:31.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:31.145+0000 7f554874f700 1 -- 172.21.15.42:0/2463969272 shutdown_connections 2019-12-05T05:34:31.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:31.145+0000 7f554874f700 1 -- 172.21.15.42:0/2463969272 wait complete. 2019-12-05T05:34:31.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:31.145+0000 7f554874f700 1 -- 172.21.15.42:0/2463969272 >> 172.21.15.42:0/2463969272 conn(0x7f55400aabe0 msgr2=0x7f55400a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:34:31.188 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:31.149+0000 7f554874f700 1 Processor -- start 2019-12-05T05:34:31.188 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:31.149+0000 7f554874f700 1 -- start start 2019-12-05T05:34:31.188 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:31.149+0000 7f554874f700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5540119b90 0x7f554012ba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:34:31.188 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:31.149+0000 7f554874f700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f554012bfb0 0x7f554012e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:34:31.188 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:31.149+0000 7f554874f700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55400a3cc0 con 0x7f5540079f10 2019-12-05T05:34:31.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:31.149+0000 7f554874f700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55400a6f80 con 0x7f5540119b90 2019-12-05T05:34:31.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:31.149+0000 7f554874f700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55400a5ec0 con 0x7f554012bfb0 2019-12-05T05:34:31.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:31.149+0000 7f5546cec700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f5540079f10 0x7f55400a34c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:36748/0 (socket says 172.21.15.42:36748) 2019-12-05T05:34:31.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:31.149+0000 7f5546cec700 1 -- 172.21.15.42:0/2870247027 learned_addr learned my addr 172.21.15.42:0/2870247027 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:34:31.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:31.149+0000 7f55464eb700 1 --2- 172.21.15.42:0/2870247027 >> v2:172.21.15.40:3300/0 conn(0x7f5540119b90 0x7f554012ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:34:31.193 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:31.149+0000 7f55474ed700 1 --2- 172.21.15.42:0/2870247027 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f554012bfb0 0x7f554012e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:34:31.193 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:31.149+0000 7f55377fe700 1 -- 172.21.15.42:0/2870247027 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 19176721 0 0) 0x7f55400a3cc0 con 0x7f5540079f10 2019-12-05T05:34:31.193 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:31.149+0000 7f55377fe700 1 -- 172.21.15.42:0/2870247027 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55280030b0 con 0x7f5540079f10 2019-12-05T05:34:31.193 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:31.153+0000 7f55377fe700 1 -- 172.21.15.42:0/2870247027 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 3922713033 0 0) 0x7f55280030b0 con 0x7f5540079f10 2019-12-05T05:34:31.194 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:31.153+0000 7f55377fe700 1 -- 172.21.15.42:0/2870247027 --> v1:172.21.15.42:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f55400a3cc0 con 0x7f5540079f10 2019-12-05T05:34:31.194 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:31.153+0000 7f55377fe700 1 -- 172.21.15.42:0/2870247027 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7f553c011de0 con 0x7f5540079f10 2019-12-05T05:34:31.194 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:31.153+0000 7f55464eb700 1 -- 172.21.15.42:0/2870247027 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f554012bfb0 msgr2=0x7f554012e3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:34:31.194 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:31.153+0000 7f55464eb700 1 --2- 172.21.15.42:0/2870247027 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f554012bfb0 0x7f554012e3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:34:31.194 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:31.153+0000 7f55464eb700 1 -- 172.21.15.42:0/2870247027 >> v1:172.21.15.42:6789/0 conn(0x7f5540079f10 legacy=0x7f55400a34c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:34:31.194 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:31.153+0000 7f55464eb700 1 -- 172.21.15.42:0/2870247027 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55400a6400 con 0x7f5540119b90 2019-12-05T05:34:31.194 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:31.153+0000 7f55474ed700 1 --2- 172.21.15.42:0/2870247027 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f554012bfb0 0x7f554012e3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:34:31.195 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:31.153+0000 7f55464eb700 1 --2- 172.21.15.42:0/2870247027 >> v2:172.21.15.40:3300/0 conn(0x7f5540119b90 0x7f554012ba60 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f55400a71b0 tx=0x7f553c0082d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:31.195 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:31.153+0000 7f55464eb700 1 --2- 172.21.15.42:0/2870247027 >> v2:172.21.15.40:3300/0 conn(0x7f5540119b90 0x7f554012ba60 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f55400a71b0 tx=0x7f553c0082d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:31.195 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:31.153+0000 7f55464eb700 1 --2- 172.21.15.42:0/2870247027 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5540119b90 0x7f554012ba60 secure :-1 s=READY pgs=885 cs=0 l=1 rx=0x7f55400a71b0 tx=0x7f553c0082d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:34:31.195 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:31.153+0000 7f55464eb700 1 --2- 172.21.15.42:0/2870247027 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5540119b90 0x7f554012ba60 secure :-1 s=THROTTLE_DONE pgs=885 cs=0 l=1 rx=0x7f55400a71b0 tx=0x7f553c0082d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:31.195 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:31.153+0000 7f55377fe700 1 -- 172.21.15.42:0/2870247027 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f553000d820 con 0x7f5540119b90 2019-12-05T05:34:31.195 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:31.153+0000 7f55464eb700 1 --2- 172.21.15.42:0/2870247027 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5540119b90 0x7f554012ba60 secure :-1 s=THROTTLE_DONE pgs=885 cs=0 l=1 rx=0x7f55400a71b0 tx=0x7f553c0082d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:31.195 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:31.153+0000 7f55377fe700 1 -- 172.21.15.42:0/2870247027 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f553000d440 con 0x7f5540119b90 2019-12-05T05:34:31.196 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:31.153+0000 7f55464eb700 1 --2- 172.21.15.42:0/2870247027 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5540119b90 0x7f554012ba60 secure :-1 s=THROTTLE_DONE pgs=885 cs=0 l=1 rx=0x7f55400a71b0 tx=0x7f553c0082d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:31.196 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:31.153+0000 7f55377fe700 1 -- 172.21.15.42:0/2870247027 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f553000de00 con 0x7f5540119b90 2019-12-05T05:34:31.196 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:31.153+0000 7f554874f700 1 -- 172.21.15.42:0/2870247027 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55400a5900 con 0x7f5540119b90 2019-12-05T05:34:31.196 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:31.153+0000 7f554874f700 1 -- 172.21.15.42:0/2870247027 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55400a1510 con 0x7f5540119b90 2019-12-05T05:34:31.196 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:31.153+0000 7f55464eb700 1 --2- 172.21.15.42:0/2870247027 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5540119b90 0x7f554012ba60 secure :-1 s=THROTTLE_DONE pgs=885 cs=0 l=1 rx=0x7f55400a71b0 tx=0x7f553c0082d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:31.196 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:31.153+0000 7f55377fe700 1 -- 172.21.15.42:0/2870247027 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f5530006460 con 0x7f5540119b90 2019-12-05T05:34:31.196 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:31.153+0000 7f55464eb700 1 --2- 172.21.15.42:0/2870247027 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5540119b90 0x7f554012ba60 secure :-1 s=THROTTLE_DONE pgs=885 cs=0 l=1 rx=0x7f55400a71b0 tx=0x7f553c0082d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:31.197 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:31.153+0000 7f55377fe700 1 --2- 172.21.15.42:0/2870247027 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f552803b5d0 0x7f552803dae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:34:31.197 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:31.153+0000 7f55377fe700 1 -- 172.21.15.42:0/2870247027 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 7837+0+0 (secure 0 0 0) 0x7f553004b0f0 con 0x7f5540119b90 2019-12-05T05:34:31.197 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:31.153+0000 7f5546cec700 1 --2- 172.21.15.42:0/2870247027 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f552803b5d0 0x7f552803dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:34:31.197 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:31.153+0000 7f5546cec700 1 --2- 172.21.15.42:0/2870247027 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f552803b5d0 0x7f552803dae0 crc :-1 s=READY pgs=484 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:34:31.197 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:31.157+0000 7f554874f700 1 -- 172.21.15.42:0/2870247027 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f554005d110 con 0x7f5540119b90 2019-12-05T05:34:31.197 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:31.165+0000 7f55464eb700 1 --2- 172.21.15.42:0/2870247027 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5540119b90 0x7f554012ba60 secure :-1 s=THROTTLE_DONE pgs=885 cs=0 l=1 rx=0x7f55400a71b0 tx=0x7f553c0082d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:31.197 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:31.165+0000 7f55377fe700 1 -- 172.21.15.42:0/2870247027 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f5530032150 con 0x7f5540119b90 2019-12-05T05:34:31.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:31.333+0000 7f554874f700 1 -- 172.21.15.42:0/2870247027 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f554005b8c0 con 0x7f5540119b90 2019-12-05T05:34:31.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:31.333+0000 7f55464eb700 1 --2- 172.21.15.42:0/2870247027 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5540119b90 0x7f554012ba60 secure :-1 s=THROTTLE_DONE pgs=885 cs=0 l=1 rx=0x7f55400a71b0 tx=0x7f553c0082d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:31.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:31.333+0000 7f55377fe700 1 -- 172.21.15.42:0/2870247027 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 138 v138) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f55300323b0 con 0x7f5540119b90 2019-12-05T05:34:31.341 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 138 2019-12-05T05:34:31.342 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:34:31.342 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":138,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7221,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/3789641053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3789641053},{"type":"v1","addr":"172.21.15.40:6833","nonce":3789641053}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":138}],"filesystems":[{"mdsmap":{"epoch":137,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:32:53.767447+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6997},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6997":{"gid":6997,"name":"a","rank":0,"incarnation":134,"state":"up:active","state_seq":17,"addr":"172.21.15.42:6835/1601334502","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1601334502},{"type":"v1","addr":"172.21.15.42:6835","nonce":1601334502}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:34:31.344 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:31.337+0000 7f554874f700 1 -- 172.21.15.42:0/2870247027 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f552803b5d0 msgr2=0x7f552803dae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:34:31.345 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:31.337+0000 7f554874f700 1 --2- 172.21.15.42:0/2870247027 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f552803b5d0 0x7f552803dae0 crc :-1 s=READY pgs=484 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:34:31.345 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:31.337+0000 7f554874f700 1 -- 172.21.15.42:0/2870247027 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5540119b90 msgr2=0x7f554012ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:34:31.345 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:31.337+0000 7f554874f700 1 --2- 172.21.15.42:0/2870247027 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5540119b90 0x7f554012ba60 secure :-1 s=READY pgs=885 cs=0 l=1 rx=0x7f55400a71b0 tx=0x7f553c0082d0).stop 2019-12-05T05:34:31.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:31.341+0000 7f554874f700 1 -- 172.21.15.42:0/2870247027 shutdown_connections 2019-12-05T05:34:31.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:31.341+0000 7f554874f700 1 --2- 172.21.15.42:0/2870247027 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f552803b5d0 0x7f552803dae0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:34:31.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:31.341+0000 7f554874f700 1 --2- 172.21.15.42:0/2870247027 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f554012bfb0 0x7f554012e3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:34:31.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:31.341+0000 7f554874f700 1 --2- 172.21.15.42:0/2870247027 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5540119b90 0x7f554012ba60 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:34:31.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:31.341+0000 7f554874f700 1 -- 172.21.15.42:0/2870247027 shutdown_connections 2019-12-05T05:34:31.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:31.341+0000 7f554874f700 1 -- 172.21.15.42:0/2870247027 wait complete. 2019-12-05T05:34:31.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:31.341+0000 7f554874f700 1 -- 172.21.15.42:0/2870247027 >> 172.21.15.42:0/2870247027 conn(0x7f55400aabe0 msgr2=0x7f554008ef70 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:34:31.354 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T05:34:33.356 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:34:33.460 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:33.449+0000 7f4e31987700 1 Processor -- start 2019-12-05T05:34:33.460 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:33.449+0000 7f4e31987700 1 -- start start 2019-12-05T05:34:33.461 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:33.449+0000 7f4e31987700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f4e2c0a7740 0x7f4e2c08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:34:33.461 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:33.449+0000 7f4e31987700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e2c0a0710 con 0x7f4e2c09fae0 2019-12-05T05:34:33.461 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:33.449+0000 7f4e31987700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e2c0a9ba0 con 0x7f4e2c07e720 2019-12-05T05:34:33.461 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:33.449+0000 7f4e31987700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e2c0a0b00 con 0x7f4e2c0a7740 2019-12-05T05:34:33.461 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:33.449+0000 7f4e2affd700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f4e2c07e720 0x7f4e2c09f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:36758/0 (socket says 172.21.15.42:36758) 2019-12-05T05:34:33.462 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:33.449+0000 7f4e2affd700 1 -- 172.21.15.42:0/1646458742 learned_addr learned my addr 172.21.15.42:0/1646458742 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:34:33.462 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:33.453+0000 7f4e2b7fe700 1 --2- 172.21.15.42:0/1646458742 >> v2:172.21.15.40:3300/0 conn(0x7f4e2c0a7740 0x7f4e2c08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:34:33.462 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:33.453+0000 7f4e2a7fc700 1 -- 172.21.15.42:0/1646458742 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2826428285 0 0) 0x7f4e2c0a9ba0 con 0x7f4e2c07e720 2019-12-05T05:34:33.462 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:33.453+0000 7f4e2a7fc700 1 -- 172.21.15.42:0/1646458742 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e140030b0 con 0x7f4e2c07e720 2019-12-05T05:34:33.462 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:33.453+0000 7f4e2a7fc700 1 -- 172.21.15.42:0/1646458742 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 656254952 0 0) 0x7f4e2c0a0710 con 0x7f4e2c09fae0 2019-12-05T05:34:33.463 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:33.453+0000 7f4e2a7fc700 1 -- 172.21.15.42:0/1646458742 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e2c0a9ba0 con 0x7f4e2c09fae0 2019-12-05T05:34:33.463 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:33.453+0000 7f4e2b7fe700 1 -- 172.21.15.42:0/1646458742 >> v1:172.21.15.42:6789/0 conn(0x7f4e2c07e720 legacy=0x7f4e2c09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:34:33.463 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:33.453+0000 7f4e2b7fe700 1 -- 172.21.15.42:0/1646458742 >> v1:172.21.15.40:6789/0 conn(0x7f4e2c09fae0 legacy=0x7f4e2c14efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:34:33.463 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:33.453+0000 7f4e2b7fe700 1 -- 172.21.15.42:0/1646458742 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e2c07eb50 con 0x7f4e2c0a7740 2019-12-05T05:34:33.463 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:33.453+0000 7f4e2b7fe700 1 --2- 172.21.15.42:0/1646458742 >> v2:172.21.15.40:3300/0 conn(0x7f4e2c0a7740 0x7f4e2c08df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f4e1c008790 tx=0x7f4e1c00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:33.464 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:33.453+0000 7f4e2b7fe700 1 --2- 172.21.15.42:0/1646458742 >> v2:172.21.15.40:3300/0 conn(0x7f4e2c0a7740 0x7f4e2c08df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f4e1c008790 tx=0x7f4e1c00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:33.464 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:33.453+0000 7f4e2b7fe700 1 --2- 172.21.15.42:0/1646458742 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e2c0a7740 0x7f4e2c08df80 secure :-1 s=READY pgs=887 cs=0 l=1 rx=0x7f4e1c008790 tx=0x7f4e1c00fbf0).ready entity=mon.0 client_cookie=60168112ca8f71a7 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:34:33.464 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:33.453+0000 7f4e2b7fe700 1 --2- 172.21.15.42:0/1646458742 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e2c0a7740 0x7f4e2c08df80 secure :-1 s=THROTTLE_DONE pgs=887 cs=0 l=1 rx=0x7f4e1c008790 tx=0x7f4e1c00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:33.465 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:33.453+0000 7f4e2a7fc700 1 -- 172.21.15.42:0/1646458742 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4e1c011270 con 0x7f4e2c0a7740 2019-12-05T05:34:33.465 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:33.453+0000 7f4e2b7fe700 1 --2- 172.21.15.42:0/1646458742 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e2c0a7740 0x7f4e2c08df80 secure :-1 s=THROTTLE_DONE pgs=887 cs=0 l=1 rx=0x7f4e1c008790 tx=0x7f4e1c00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:33.465 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:33.453+0000 7f4e2a7fc700 1 -- 172.21.15.42:0/1646458742 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e1c008bc0 con 0x7f4e2c0a7740 2019-12-05T05:34:33.465 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:33.453+0000 7f4e2b7fe700 1 --2- 172.21.15.42:0/1646458742 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e2c0a7740 0x7f4e2c08df80 secure :-1 s=THROTTLE_DONE pgs=887 cs=0 l=1 rx=0x7f4e1c008790 tx=0x7f4e1c00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:33.465 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:33.453+0000 7f4e2a7fc700 1 -- 172.21.15.42:0/1646458742 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4e1c011850 con 0x7f4e2c0a7740 2019-12-05T05:34:33.466 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:33.453+0000 7f4e31987700 1 -- 172.21.15.42:0/1646458742 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e2c0a7740 msgr2=0x7f4e2c08df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:34:33.467 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:33.453+0000 7f4e31987700 1 --2- 172.21.15.42:0/1646458742 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e2c0a7740 0x7f4e2c08df80 secure :-1 s=READY pgs=887 cs=0 l=1 rx=0x7f4e1c008790 tx=0x7f4e1c00fbf0).stop 2019-12-05T05:34:33.467 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:33.453+0000 7f4e31987700 1 -- 172.21.15.42:0/1646458742 shutdown_connections 2019-12-05T05:34:33.467 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:33.453+0000 7f4e31987700 1 --2- 172.21.15.42:0/1646458742 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e2c0a7740 0x7f4e2c08df80 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:34:33.467 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:33.453+0000 7f4e31987700 1 -- 172.21.15.42:0/1646458742 shutdown_connections 2019-12-05T05:34:33.468 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:33.453+0000 7f4e31987700 1 -- 172.21.15.42:0/1646458742 wait complete. 2019-12-05T05:34:33.468 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:33.453+0000 7f4e31987700 1 -- 172.21.15.42:0/1646458742 >> 172.21.15.42:0/1646458742 conn(0x7f4e2c0aabe0 msgr2=0x7f4e2c0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:34:33.477 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:33.457+0000 7f4e31987700 1 Processor -- start 2019-12-05T05:34:33.477 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:33.457+0000 7f4e31987700 1 -- start start 2019-12-05T05:34:33.477 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:33.457+0000 7f4e31987700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4e2c1122c0 0x7f4e2c143330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:34:33.477 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:33.457+0000 7f4e31987700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f4e2c122130 0x7f4e2c142840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:34:33.477 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:33.457+0000 7f4e31987700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e2c1439d0 con 0x7f4e2c144170 2019-12-05T05:34:33.478 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:33.457+0000 7f4e31987700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e2c150600 con 0x7f4e2c122130 2019-12-05T05:34:33.478 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:33.457+0000 7f4e31987700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e2c143090 con 0x7f4e2c1122c0 2019-12-05T05:34:33.478 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:33.457+0000 7f4e2b7fe700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f4e2c144170 0x7f4e2c14fe00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:36762/0 (socket says 172.21.15.42:36762) 2019-12-05T05:34:33.478 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:33.457+0000 7f4e2b7fe700 1 -- 172.21.15.42:0/4167195964 learned_addr learned my addr 172.21.15.42:0/4167195964 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:34:33.478 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:33.457+0000 7f4e2bfff700 1 --2- 172.21.15.42:0/4167195964 >> v2:172.21.15.40:3300/0 conn(0x7f4e2c122130 0x7f4e2c142840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:34:33.478 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:33.457+0000 7f4e2affd700 1 --2- 172.21.15.42:0/4167195964 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4e2c1122c0 0x7f4e2c143330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:34:33.478 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:33.457+0000 7f4e2affd700 1 -- 172.21.15.42:0/4167195964 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4e2c1122c0 msgr2=0x7f4e2c143330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2019-12-05T05:34:33.479 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:33.457+0000 7f4e2affd700 1 -- 172.21.15.42:0/4167195964 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4e2c1122c0 msgr2=0x7f4e2c143330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2019-12-05T05:34:33.479 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:33.457+0000 7f4e2affd700 1 --2- 172.21.15.42:0/4167195964 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4e2c1122c0 0x7f4e2c143330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-12-05T05:34:33.479 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:33.461+0000 7f4e2affd700 1 --2- 172.21.15.42:0/4167195964 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4e2c1122c0 0x7f4e2c143330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rx=0 tx=0)._fault waiting 0.200000 2019-12-05T05:34:33.479 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:33.461+0000 7f4e13fff700 1 -- 172.21.15.42:0/4167195964 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 946481354 0 0) 0x7f4e2c1439d0 con 0x7f4e2c144170 2019-12-05T05:34:33.479 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:33.461+0000 7f4e13fff700 1 -- 172.21.15.42:0/4167195964 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e0c0030b0 con 0x7f4e2c144170 2019-12-05T05:34:33.479 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:33.461+0000 7f4e2bfff700 1 -- 172.21.15.42:0/4167195964 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4e2c1122c0 msgr2=0x7f4e2c143330 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2019-12-05T05:34:33.479 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:33.461+0000 7f4e2bfff700 1 --2- 172.21.15.42:0/4167195964 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4e2c1122c0 0x7f4e2c143330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:34:33.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:33.461+0000 7f4e2bfff700 1 -- 172.21.15.42:0/4167195964 >> v1:172.21.15.42:6789/0 conn(0x7f4e2c144170 legacy=0x7f4e2c14fe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:34:33.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:33.461+0000 7f4e2bfff700 1 -- 172.21.15.42:0/4167195964 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e2c142d90 con 0x7f4e2c122130 2019-12-05T05:34:33.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:33.461+0000 7f4e2bfff700 1 --2- 172.21.15.42:0/4167195964 >> v2:172.21.15.40:3300/0 conn(0x7f4e2c122130 0x7f4e2c142840 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f4e24001210 tx=0x7f4e24010b20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:33.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:33.461+0000 7f4e2bfff700 1 --2- 172.21.15.42:0/4167195964 >> v2:172.21.15.40:3300/0 conn(0x7f4e2c122130 0x7f4e2c142840 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f4e24001210 tx=0x7f4e24010b20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:33.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:33.461+0000 7f4e2bfff700 1 --2- 172.21.15.42:0/4167195964 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e2c122130 0x7f4e2c142840 secure :-1 s=READY pgs=888 cs=0 l=1 rx=0x7f4e24001210 tx=0x7f4e24010b20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:34:33.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:33.461+0000 7f4e2bfff700 1 --2- 172.21.15.42:0/4167195964 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e2c122130 0x7f4e2c142840 secure :-1 s=THROTTLE_DONE pgs=888 cs=0 l=1 rx=0x7f4e24001210 tx=0x7f4e24010b20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:33.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:33.461+0000 7f4e13fff700 1 -- 172.21.15.42:0/4167195964 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4e24006bb0 con 0x7f4e2c122130 2019-12-05T05:34:33.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:33.461+0000 7f4e2bfff700 1 --2- 172.21.15.42:0/4167195964 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e2c122130 0x7f4e2c142840 secure :-1 s=THROTTLE_DONE pgs=888 cs=0 l=1 rx=0x7f4e24001210 tx=0x7f4e24010b20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:33.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:33.461+0000 7f4e2bfff700 1 --2- 172.21.15.42:0/4167195964 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e2c122130 0x7f4e2c142840 secure :-1 s=THROTTLE_DONE pgs=888 cs=0 l=1 rx=0x7f4e24001210 tx=0x7f4e24010b20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:33.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:33.461+0000 7f4e13fff700 1 -- 172.21.15.42:0/4167195964 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e240067d0 con 0x7f4e2c122130 2019-12-05T05:34:33.482 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:33.461+0000 7f4e31987700 1 -- 172.21.15.42:0/4167195964 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e2c122560 con 0x7f4e2c122130 2019-12-05T05:34:33.482 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:33.461+0000 7f4e13fff700 1 -- 172.21.15.42:0/4167195964 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4e240123d0 con 0x7f4e2c122130 2019-12-05T05:34:33.482 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:33.461+0000 7f4e31987700 1 -- 172.21.15.42:0/4167195964 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e2c12cce0 con 0x7f4e2c122130 2019-12-05T05:34:33.482 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:33.461+0000 7f4e2bfff700 1 --2- 172.21.15.42:0/4167195964 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e2c122130 0x7f4e2c142840 secure :-1 s=THROTTLE_DONE pgs=888 cs=0 l=1 rx=0x7f4e24001210 tx=0x7f4e24010b20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:33.482 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:33.461+0000 7f4e13fff700 1 -- 172.21.15.42:0/4167195964 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f4e240126f0 con 0x7f4e2c122130 2019-12-05T05:34:33.482 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:33.461+0000 7f4e2bfff700 1 --2- 172.21.15.42:0/4167195964 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e2c122130 0x7f4e2c142840 secure :-1 s=THROTTLE_DONE pgs=888 cs=0 l=1 rx=0x7f4e24001210 tx=0x7f4e24010b20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:33.482 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:33.461+0000 7f4e13fff700 1 --2- 172.21.15.42:0/4167195964 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4e0c03b280 0x7f4e0c03d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:34:33.483 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:33.461+0000 7f4e13fff700 1 -- 172.21.15.42:0/4167195964 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 7837+0+0 (secure 0 0 0) 0x7f4e2404b330 con 0x7f4e2c122130 2019-12-05T05:34:33.483 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:33.461+0000 7f4e2b7fe700 1 --2- 172.21.15.42:0/4167195964 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4e0c03b280 0x7f4e0c03d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:34:33.483 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:33.465+0000 7f4e2b7fe700 1 --2- 172.21.15.42:0/4167195964 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4e0c03b280 0x7f4e0c03d790 crc :-1 s=READY pgs=485 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:34:33.483 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:33.465+0000 7f4e31987700 1 -- 172.21.15.42:0/4167195964 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e2c05d110 con 0x7f4e2c122130 2019-12-05T05:34:33.484 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:33.473+0000 7f4e2bfff700 1 --2- 172.21.15.42:0/4167195964 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e2c122130 0x7f4e2c142840 secure :-1 s=THROTTLE_DONE pgs=888 cs=0 l=1 rx=0x7f4e24001210 tx=0x7f4e24010b20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:33.484 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:33.473+0000 7f4e13fff700 1 -- 172.21.15.42:0/4167195964 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f4e24032390 con 0x7f4e2c122130 2019-12-05T05:34:33.647 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:33.641+0000 7f4e31987700 1 -- 172.21.15.42:0/4167195964 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4e2c0a8cb0 con 0x7f4e2c122130 2019-12-05T05:34:33.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:33.645+0000 7f4e2bfff700 1 --2- 172.21.15.42:0/4167195964 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e2c122130 0x7f4e2c142840 secure :-1 s=THROTTLE_DONE pgs=888 cs=0 l=1 rx=0x7f4e24001210 tx=0x7f4e24010b20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:33.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:33.645+0000 7f4e13fff700 1 -- 172.21.15.42:0/4167195964 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 138 v138) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f4e240325f0 con 0x7f4e2c122130 2019-12-05T05:34:33.650 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 138 2019-12-05T05:34:33.651 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:34:33.651 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":138,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7221,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/3789641053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3789641053},{"type":"v1","addr":"172.21.15.40:6833","nonce":3789641053}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":138}],"filesystems":[{"mdsmap":{"epoch":137,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:32:53.767447+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6997},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6997":{"gid":6997,"name":"a","rank":0,"incarnation":134,"state":"up:active","state_seq":17,"addr":"172.21.15.42:6835/1601334502","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1601334502},{"type":"v1","addr":"172.21.15.42:6835","nonce":1601334502}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:34:33.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:33.649+0000 7f4e31987700 1 -- 172.21.15.42:0/4167195964 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4e0c03b280 msgr2=0x7f4e0c03d790 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:34:33.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:33.649+0000 7f4e31987700 1 --2- 172.21.15.42:0/4167195964 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4e0c03b280 0x7f4e0c03d790 crc :-1 s=READY pgs=485 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:34:33.654 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:33.649+0000 7f4e31987700 1 -- 172.21.15.42:0/4167195964 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e2c122130 msgr2=0x7f4e2c142840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:34:33.654 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:33.649+0000 7f4e31987700 1 --2- 172.21.15.42:0/4167195964 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e2c122130 0x7f4e2c142840 secure :-1 s=READY pgs=888 cs=0 l=1 rx=0x7f4e24001210 tx=0x7f4e24010b20).stop 2019-12-05T05:34:33.654 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:33.649+0000 7f4e31987700 1 -- 172.21.15.42:0/4167195964 shutdown_connections 2019-12-05T05:34:33.654 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:33.649+0000 7f4e31987700 1 --2- 172.21.15.42:0/4167195964 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4e0c03b280 0x7f4e0c03d790 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:34:33.654 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:33.649+0000 7f4e31987700 1 --2- 172.21.15.42:0/4167195964 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e2c122130 0x7f4e2c142840 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:34:33.654 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:33.649+0000 7f4e31987700 1 --2- 172.21.15.42:0/4167195964 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4e2c1122c0 0x7f4e2c143330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:34:33.654 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:33.649+0000 7f4e31987700 1 -- 172.21.15.42:0/4167195964 shutdown_connections 2019-12-05T05:34:33.655 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:33.649+0000 7f4e31987700 1 -- 172.21.15.42:0/4167195964 wait complete. 2019-12-05T05:34:33.655 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:33.649+0000 7f4e31987700 1 -- 172.21.15.42:0/4167195964 >> 172.21.15.42:0/4167195964 conn(0x7f4e2c07ee20 msgr2=0x7f4e2c0a6b90 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:34:33.665 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T05:34:35.665 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:34:35.779 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:35.769+0000 7f30f4647700 1 Processor -- start 2019-12-05T05:34:35.779 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:35.769+0000 7f30f4647700 1 -- start start 2019-12-05T05:34:35.780 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:35.769+0000 7f30f4647700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f30ec08df80 0x7f30ec09f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:34:35.780 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:35.769+0000 7f30f4647700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f30ec09f830 0x7f30ec14efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:34:35.780 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:35.769+0000 7f30f4647700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30ec09fc60 con 0x7f30ec0a7740 2019-12-05T05:34:35.780 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:35.769+0000 7f30f4647700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30ec0a0b00 con 0x7f30ec08df80 2019-12-05T05:34:35.780 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:35.769+0000 7f30f4647700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30ec08e5b0 con 0x7f30ec09f830 2019-12-05T05:34:35.780 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:35.769+0000 7f30f23e3700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f30ec08df80 0x7f30ec09f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:34:35.781 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:35.769+0000 7f30f2be4700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f30ec0a7740 0x7f30ec07e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:36774/0 (socket says 172.21.15.42:36774) 2019-12-05T05:34:35.781 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:35.769+0000 7f30f2be4700 1 -- 172.21.15.42:0/1583126182 learned_addr learned my addr 172.21.15.42:0/1583126182 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:34:35.781 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:35.773+0000 7f30f33e5700 1 --2- 172.21.15.42:0/1583126182 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f30ec09f830 0x7f30ec14efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:34:35.781 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:35.773+0000 7f30f1be2700 1 -- 172.21.15.42:0/1583126182 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2763501324 0 0) 0x7f30ec09fc60 con 0x7f30ec0a7740 2019-12-05T05:34:35.781 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:35.773+0000 7f30f1be2700 1 -- 172.21.15.42:0/1583126182 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30d40030b0 con 0x7f30ec0a7740 2019-12-05T05:34:35.781 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:35.773+0000 7f30f33e5700 1 -- 172.21.15.42:0/1583126182 >> v2:172.21.15.42:3300/0 conn(0x7f30ec08df80 msgr2=0x7f30ec09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:34:35.781 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:35.773+0000 7f30f33e5700 1 --2- 172.21.15.42:0/1583126182 >> v2:172.21.15.42:3300/0 conn(0x7f30ec08df80 0x7f30ec09f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:34:35.782 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:35.773+0000 7f30f33e5700 1 -- 172.21.15.42:0/1583126182 >> v1:172.21.15.42:6789/0 conn(0x7f30ec0a7740 legacy=0x7f30ec07e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:34:35.782 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:35.773+0000 7f30f33e5700 1 -- 172.21.15.42:0/1583126182 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30ec0a0710 con 0x7f30ec09f830 2019-12-05T05:34:35.782 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:35.773+0000 7f30f33e5700 1 --2- 172.21.15.42:0/1583126182 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f30ec09f830 0x7f30ec14efe0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f30dc008790 tx=0x7f30dc00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:35.782 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:35.773+0000 7f30f33e5700 1 --2- 172.21.15.42:0/1583126182 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f30ec09f830 0x7f30ec14efe0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f30dc008790 tx=0x7f30dc00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:35.782 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:35.773+0000 7f30f33e5700 1 --2- 172.21.15.42:0/1583126182 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f30ec09f830 0x7f30ec14efe0 secure :-1 s=READY pgs=233 cs=0 l=1 rx=0x7f30dc008790 tx=0x7f30dc00fc70).ready entity=mon.2 client_cookie=6335a7ee227302ce server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:34:35.782 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:35.773+0000 7f30f33e5700 1 --2- 172.21.15.42:0/1583126182 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f30ec09f830 0x7f30ec14efe0 secure :-1 s=THROTTLE_DONE pgs=233 cs=0 l=1 rx=0x7f30dc008790 tx=0x7f30dc00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:35.782 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:35.773+0000 7f30f1be2700 1 -- 172.21.15.42:0/1583126182 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f30dc008bb0 con 0x7f30ec09f830 2019-12-05T05:34:35.783 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:35.773+0000 7f30f33e5700 1 --2- 172.21.15.42:0/1583126182 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f30ec09f830 0x7f30ec14efe0 secure :-1 s=THROTTLE_DONE pgs=233 cs=0 l=1 rx=0x7f30dc008790 tx=0x7f30dc00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:35.783 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:35.773+0000 7f30f33e5700 1 --2- 172.21.15.42:0/1583126182 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f30ec09f830 0x7f30ec14efe0 secure :-1 s=THROTTLE_DONE pgs=233 cs=0 l=1 rx=0x7f30dc008790 tx=0x7f30dc00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:35.783 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:35.773+0000 7f30f1be2700 1 -- 172.21.15.42:0/1583126182 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30dc0087d0 con 0x7f30ec09f830 2019-12-05T05:34:35.783 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:35.773+0000 7f30f1be2700 1 -- 172.21.15.42:0/1583126182 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f30dc011860 con 0x7f30ec09f830 2019-12-05T05:34:35.783 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:35.773+0000 7f30f4647700 1 -- 172.21.15.42:0/1583126182 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f30ec09f830 msgr2=0x7f30ec14efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:34:35.783 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:35.773+0000 7f30f4647700 1 --2- 172.21.15.42:0/1583126182 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f30ec09f830 0x7f30ec14efe0 secure :-1 s=READY pgs=233 cs=0 l=1 rx=0x7f30dc008790 tx=0x7f30dc00fc70).stop 2019-12-05T05:34:35.783 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:35.773+0000 7f30f4647700 1 -- 172.21.15.42:0/1583126182 shutdown_connections 2019-12-05T05:34:35.784 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:35.773+0000 7f30f4647700 1 --2- 172.21.15.42:0/1583126182 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f30ec09f830 0x7f30ec14efe0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:34:35.784 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:35.773+0000 7f30f4647700 1 --2- 172.21.15.42:0/1583126182 >> v2:172.21.15.42:3300/0 conn(0x7f30ec08df80 0x7f30ec09f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:34:35.784 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:35.773+0000 7f30f4647700 1 -- 172.21.15.42:0/1583126182 shutdown_connections 2019-12-05T05:34:35.784 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:35.773+0000 7f30f4647700 1 -- 172.21.15.42:0/1583126182 wait complete. 2019-12-05T05:34:35.784 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:35.773+0000 7f30f4647700 1 -- 172.21.15.42:0/1583126182 >> 172.21.15.42:0/1583126182 conn(0x7f30ec0aabe0 msgr2=0x7f30ec0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:34:35.788 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:35.777+0000 7f30f4647700 1 Processor -- start 2019-12-05T05:34:35.788 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:35.777+0000 7f30f4647700 1 -- start start 2019-12-05T05:34:35.789 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:35.777+0000 7f30f4647700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f30ec08eeb0 0x7f30ec132010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:34:35.789 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:35.777+0000 7f30f4647700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f30ec0a3c60 0x7f30ec138900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:34:35.789 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:35.777+0000 7f30f4647700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30ec1314b0 con 0x7f30ec12def0 2019-12-05T05:34:35.789 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:35.777+0000 7f30f4647700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30ec131830 con 0x7f30ec0a3c60 2019-12-05T05:34:35.789 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:35.777+0000 7f30f4647700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30ec131ad0 con 0x7f30ec08eeb0 2019-12-05T05:34:35.789 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:35.777+0000 7f30f23e3700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f30ec12def0 0x7f30ec143400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:36780/0 (socket says 172.21.15.42:36780) 2019-12-05T05:34:35.790 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:35.777+0000 7f30f23e3700 1 -- 172.21.15.42:0/2711008270 learned_addr learned my addr 172.21.15.42:0/2711008270 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:34:35.790 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:35.777+0000 7f30f33e5700 1 --2- 172.21.15.42:0/2711008270 >> v2:172.21.15.40:3300/0 conn(0x7f30ec0a3c60 0x7f30ec138900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:34:35.790 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:35.777+0000 7f30f2be4700 1 --2- 172.21.15.42:0/2711008270 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f30ec08eeb0 0x7f30ec132010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:34:35.790 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:35.781+0000 7f30eb7fe700 1 -- 172.21.15.42:0/2711008270 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 672520365 0 0) 0x7f30ec1314b0 con 0x7f30ec12def0 2019-12-05T05:34:35.790 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:35.781+0000 7f30eb7fe700 1 -- 172.21.15.42:0/2711008270 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30cc0030b0 con 0x7f30ec12def0 2019-12-05T05:34:35.791 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:35.781+0000 7f30f2be4700 1 -- 172.21.15.42:0/2711008270 >> v2:172.21.15.40:3300/0 conn(0x7f30ec0a3c60 msgr2=0x7f30ec138900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:34:35.791 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:35.781+0000 7f30f2be4700 1 --2- 172.21.15.42:0/2711008270 >> v2:172.21.15.40:3300/0 conn(0x7f30ec0a3c60 0x7f30ec138900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:34:35.791 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:35.781+0000 7f30f2be4700 1 -- 172.21.15.42:0/2711008270 >> v1:172.21.15.42:6789/0 conn(0x7f30ec12def0 legacy=0x7f30ec143400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:34:35.791 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:35.781+0000 7f30f2be4700 1 -- 172.21.15.42:0/2711008270 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30ec12e320 con 0x7f30ec08eeb0 2019-12-05T05:34:35.791 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:35.781+0000 7f30f33e5700 1 --2- 172.21.15.42:0/2711008270 >> v2:172.21.15.40:3300/0 conn(0x7f30ec0a3c60 0x7f30ec138900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:34:35.791 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:35.781+0000 7f30f2be4700 1 --2- 172.21.15.42:0/2711008270 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f30ec08eeb0 0x7f30ec132010 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f30e40046f0 tx=0x7f30e4010c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:35.791 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:35.781+0000 7f30f2be4700 1 --2- 172.21.15.42:0/2711008270 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f30ec08eeb0 0x7f30ec132010 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f30e40046f0 tx=0x7f30e4010c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:35.792 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:35.781+0000 7f30f2be4700 1 --2- 172.21.15.42:0/2711008270 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f30ec08eeb0 0x7f30ec132010 secure :-1 s=READY pgs=234 cs=0 l=1 rx=0x7f30e40046f0 tx=0x7f30e4010c30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:34:35.792 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:35.781+0000 7f30f2be4700 1 --2- 172.21.15.42:0/2711008270 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f30ec08eeb0 0x7f30ec132010 secure :-1 s=THROTTLE_DONE pgs=234 cs=0 l=1 rx=0x7f30e40046f0 tx=0x7f30e4010c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:35.792 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:35.781+0000 7f30eb7fe700 1 -- 172.21.15.42:0/2711008270 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f30e4005bc0 con 0x7f30ec08eeb0 2019-12-05T05:34:35.792 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:35.781+0000 7f30f2be4700 1 --2- 172.21.15.42:0/2711008270 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f30ec08eeb0 0x7f30ec132010 secure :-1 s=THROTTLE_DONE pgs=234 cs=0 l=1 rx=0x7f30e40046f0 tx=0x7f30e4010c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:35.792 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:35.781+0000 7f30f2be4700 1 --2- 172.21.15.42:0/2711008270 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f30ec08eeb0 0x7f30ec132010 secure :-1 s=THROTTLE_DONE pgs=234 cs=0 l=1 rx=0x7f30e40046f0 tx=0x7f30e4010c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:35.792 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:35.781+0000 7f30eb7fe700 1 -- 172.21.15.42:0/2711008270 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30e40057e0 con 0x7f30ec08eeb0 2019-12-05T05:34:35.793 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:35.781+0000 7f30f4647700 1 -- 172.21.15.42:0/2711008270 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30ec12e530 con 0x7f30ec08eeb0 2019-12-05T05:34:35.793 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:35.781+0000 7f30eb7fe700 1 -- 172.21.15.42:0/2711008270 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f30e4012860 con 0x7f30ec08eeb0 2019-12-05T05:34:35.793 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:35.781+0000 7f30f4647700 1 -- 172.21.15.42:0/2711008270 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30ec0a4090 con 0x7f30ec08eeb0 2019-12-05T05:34:35.793 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:35.781+0000 7f30f2be4700 1 --2- 172.21.15.42:0/2711008270 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f30ec08eeb0 0x7f30ec132010 secure :-1 s=THROTTLE_DONE pgs=234 cs=0 l=1 rx=0x7f30e40046f0 tx=0x7f30e4010c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:35.793 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:35.781+0000 7f30eb7fe700 1 -- 172.21.15.42:0/2711008270 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f30e4012a60 con 0x7f30ec08eeb0 2019-12-05T05:34:35.793 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:35.781+0000 7f30f2be4700 1 --2- 172.21.15.42:0/2711008270 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f30ec08eeb0 0x7f30ec132010 secure :-1 s=THROTTLE_DONE pgs=234 cs=0 l=1 rx=0x7f30e40046f0 tx=0x7f30e4010c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:35.793 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:35.781+0000 7f30eb7fe700 1 --2- 172.21.15.42:0/2711008270 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f30cc03b2d0 0x7f30cc03d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:34:35.794 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:35.781+0000 7f30eb7fe700 1 -- 172.21.15.42:0/2711008270 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 7923+0+0 (secure 0 0 0) 0x7f30e404b490 con 0x7f30ec08eeb0 2019-12-05T05:34:35.794 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:35.781+0000 7f30f23e3700 1 --2- 172.21.15.42:0/2711008270 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f30cc03b2d0 0x7f30cc03d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:34:35.794 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:35.785+0000 7f30f23e3700 1 --2- 172.21.15.42:0/2711008270 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f30cc03b2d0 0x7f30cc03d7e0 crc :-1 s=READY pgs=486 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:34:35.794 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:35.785+0000 7f30f4647700 1 -- 172.21.15.42:0/2711008270 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30ec05d110 con 0x7f30ec08eeb0 2019-12-05T05:34:35.799 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:35.793+0000 7f30f2be4700 1 --2- 172.21.15.42:0/2711008270 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f30ec08eeb0 0x7f30ec132010 secure :-1 s=THROTTLE_DONE pgs=234 cs=0 l=1 rx=0x7f30e40046f0 tx=0x7f30e4010c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:35.800 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:35.797+0000 7f30eb7fe700 1 -- 172.21.15.42:0/2711008270 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f30e4012d70 con 0x7f30ec08eeb0 2019-12-05T05:34:35.969 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:35.965+0000 7f30f4647700 1 -- 172.21.15.42:0/2711008270 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f30ec05b8c0 con 0x7f30ec08eeb0 2019-12-05T05:34:35.970 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:35.965+0000 7f30f2be4700 1 --2- 172.21.15.42:0/2711008270 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f30ec08eeb0 0x7f30ec132010 secure :-1 s=THROTTLE_DONE pgs=234 cs=0 l=1 rx=0x7f30e40046f0 tx=0x7f30e4010c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:35.970 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:35.965+0000 7f30eb7fe700 1 -- 172.21.15.42:0/2711008270 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 139 v139) v1 ==== 95+0+1774 (secure 0 0 0) 0x7f30e4032510 con 0x7f30ec08eeb0 2019-12-05T05:34:35.971 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 139 2019-12-05T05:34:35.972 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:34:35.972 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":139,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":139,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:34:33.763070+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7221},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7221":{"gid":7221,"name":"b","rank":0,"incarnation":139,"state":"up:replay","state_seq":2,"addr":"172.21.15.40:6833/3789641053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3789641053},{"type":"v1","addr":"172.21.15.40:6833","nonce":3789641053}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:34:35.973 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:35.969+0000 7f30f4647700 1 -- 172.21.15.42:0/2711008270 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f30cc03b2d0 msgr2=0x7f30cc03d7e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:34:35.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:35.969+0000 7f30f4647700 1 --2- 172.21.15.42:0/2711008270 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f30cc03b2d0 0x7f30cc03d7e0 crc :-1 s=READY pgs=486 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:34:35.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:35.969+0000 7f30f4647700 1 -- 172.21.15.42:0/2711008270 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f30ec08eeb0 msgr2=0x7f30ec132010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:34:35.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:35.969+0000 7f30f4647700 1 --2- 172.21.15.42:0/2711008270 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f30ec08eeb0 0x7f30ec132010 secure :-1 s=READY pgs=234 cs=0 l=1 rx=0x7f30e40046f0 tx=0x7f30e4010c30).stop 2019-12-05T05:34:35.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:35.969+0000 7f30f4647700 1 -- 172.21.15.42:0/2711008270 shutdown_connections 2019-12-05T05:34:35.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:35.969+0000 7f30f4647700 1 --2- 172.21.15.42:0/2711008270 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f30cc03b2d0 0x7f30cc03d7e0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:34:35.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:35.969+0000 7f30f4647700 1 --2- 172.21.15.42:0/2711008270 >> v2:172.21.15.40:3300/0 conn(0x7f30ec0a3c60 0x7f30ec138900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:34:35.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:35.969+0000 7f30f4647700 1 --2- 172.21.15.42:0/2711008270 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f30ec08eeb0 0x7f30ec132010 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:34:35.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:35.969+0000 7f30f4647700 1 -- 172.21.15.42:0/2711008270 shutdown_connections 2019-12-05T05:34:35.976 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:35.969+0000 7f30f4647700 1 -- 172.21.15.42:0/2711008270 wait complete. 2019-12-05T05:34:35.976 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:35.969+0000 7f30f4647700 1 -- 172.21.15.42:0/2711008270 >> 172.21.15.42:0/2711008270 conn(0x7f30ec0a9790 msgr2=0x7f30ec059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:34:35.987 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a down, removed from mdsmap 2019-12-05T05:34:35.987 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2019-12-05T05:34:35.998 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:34:36.110 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:36.097+0000 7f028cdb9700 1 Processor -- start 2019-12-05T05:34:36.110 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:36.097+0000 7f028cdb9700 1 -- start start 2019-12-05T05:34:36.110 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:36.101+0000 7f028cdb9700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f02880a7740 0x7f028808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:34:36.111 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:36.101+0000 7f028cdb9700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f028807e720 0x7f028807eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:34:36.111 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:36.101+0000 7f028cdb9700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02880a0710 con 0x7f028809f2e0 2019-12-05T05:34:36.111 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:36.101+0000 7f028cdb9700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02880a0b00 con 0x7f028807e720 2019-12-05T05:34:36.111 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:36.101+0000 7f028cdb9700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f028808e5b0 con 0x7f02880a7740 2019-12-05T05:34:36.111 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:36.101+0000 7f028659c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f028807e720 0x7f028807eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:34:36.111 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:36.101+0000 7f0286d9d700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f02880a7740 0x7f028808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:34:36.111 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:36.101+0000 7f0286d9d700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f02880a7740 0x7f028808df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:37214/0 (socket says 172.21.15.42:37214) 2019-12-05T05:34:36.112 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:36.101+0000 7f028659c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f028807e720 0x7f028807eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:36750/0 (socket says 172.21.15.42:36750) 2019-12-05T05:34:36.112 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:36.101+0000 7f0286d9d700 1 -- 172.21.15.42:0/2773922681 learned_addr learned my addr 172.21.15.42:0/2773922681 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:34:36.112 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:36.101+0000 7f0286d9d700 1 -- 172.21.15.42:0/2773922681 >> v2:172.21.15.42:3300/0 conn(0x7f028807e720 msgr2=0x7f028807eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:34:36.112 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:36.101+0000 7f0286d9d700 1 --2- 172.21.15.42:0/2773922681 >> v2:172.21.15.42:3300/0 conn(0x7f028807e720 0x7f028807eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:34:36.115 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:36.101+0000 7f0286d9d700 1 -- 172.21.15.42:0/2773922681 >> v1:172.21.15.40:6789/0 conn(0x7f028809f2e0 legacy=0x7f028809f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:34:36.116 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:36.101+0000 7f0286d9d700 1 -- 172.21.15.42:0/2773922681 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f028807f0a0 con 0x7f02880a7740 2019-12-05T05:34:36.116 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:36.101+0000 7f0286d9d700 1 --2- 172.21.15.42:0/2773922681 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f02880a7740 0x7f028808df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f0278008790 tx=0x7f027800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:36.116 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:36.101+0000 7f0286d9d700 1 --2- 172.21.15.42:0/2773922681 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f02880a7740 0x7f028808df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f0278008790 tx=0x7f027800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:36.116 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:36.101+0000 7f0286d9d700 1 --2- 172.21.15.42:0/2773922681 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f02880a7740 0x7f028808df80 secure :-1 s=READY pgs=235 cs=0 l=1 rx=0x7f0278008790 tx=0x7f027800fd30).ready entity=mon.2 client_cookie=81c83a3079a9f430 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:34:36.116 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:36.101+0000 7f0286d9d700 1 --2- 172.21.15.42:0/2773922681 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f02880a7740 0x7f028808df80 secure :-1 s=THROTTLE_DONE pgs=235 cs=0 l=1 rx=0x7f0278008790 tx=0x7f027800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:36.116 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:36.101+0000 7f0285d9b700 1 -- 172.21.15.42:0/2773922681 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f02780113b0 con 0x7f02880a7740 2019-12-05T05:34:36.117 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:36.101+0000 7f0286d9d700 1 --2- 172.21.15.42:0/2773922681 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f02880a7740 0x7f028808df80 secure :-1 s=THROTTLE_DONE pgs=235 cs=0 l=1 rx=0x7f0278008790 tx=0x7f027800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:36.117 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:36.101+0000 7f0286d9d700 1 --2- 172.21.15.42:0/2773922681 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f02880a7740 0x7f028808df80 secure :-1 s=THROTTLE_DONE pgs=235 cs=0 l=1 rx=0x7f0278008790 tx=0x7f027800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:36.117 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:36.101+0000 7f0285d9b700 1 -- 172.21.15.42:0/2773922681 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0278008bb0 con 0x7f02880a7740 2019-12-05T05:34:36.117 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:36.101+0000 7f0285d9b700 1 -- 172.21.15.42:0/2773922681 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0278011990 con 0x7f02880a7740 2019-12-05T05:34:36.117 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:36.101+0000 7f028cdb9700 1 -- 172.21.15.42:0/2773922681 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f02880a7740 msgr2=0x7f028808df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:34:36.117 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:36.101+0000 7f028cdb9700 1 --2- 172.21.15.42:0/2773922681 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f02880a7740 0x7f028808df80 secure :-1 s=READY pgs=235 cs=0 l=1 rx=0x7f0278008790 tx=0x7f027800fd30).stop 2019-12-05T05:34:36.118 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:36.101+0000 7f028cdb9700 1 -- 172.21.15.42:0/2773922681 shutdown_connections 2019-12-05T05:34:36.118 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:36.101+0000 7f028cdb9700 1 --2- 172.21.15.42:0/2773922681 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f02880a7740 0x7f028808df80 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:34:36.118 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:36.101+0000 7f028cdb9700 1 --2- 172.21.15.42:0/2773922681 >> v2:172.21.15.42:3300/0 conn(0x7f028807e720 0x7f028807eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:34:36.118 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:36.101+0000 7f028cdb9700 1 -- 172.21.15.42:0/2773922681 shutdown_connections 2019-12-05T05:34:36.118 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:36.101+0000 7f028cdb9700 1 -- 172.21.15.42:0/2773922681 wait complete. 2019-12-05T05:34:36.118 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:36.101+0000 7f028cdb9700 1 -- 172.21.15.42:0/2773922681 >> 172.21.15.42:0/2773922681 conn(0x7f02880aabe0 msgr2=0x7f02880a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:34:36.118 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:36.105+0000 7f028cdb9700 1 Processor -- start 2019-12-05T05:34:36.118 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:36.105+0000 7f028cdb9700 1 -- start start 2019-12-05T05:34:36.124 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:36.105+0000 7f028cdb9700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f028808eeb0 0x7f02880a2c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:34:36.124 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:36.105+0000 7f028cdb9700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f02880a19e0 0x7f02880a1e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:34:36.125 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:36.105+0000 7f028cdb9700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f02880789b0 0x7f0288078de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:34:36.125 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:36.105+0000 7f028cdb9700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02880a31b0 con 0x7f02880789b0 2019-12-05T05:34:36.125 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:36.105+0000 7f028cdb9700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02880a3440 con 0x7f028808eeb0 2019-12-05T05:34:36.125 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:36.105+0000 7f028cdb9700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02880a36e0 con 0x7f02880a19e0 2019-12-05T05:34:36.125 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:36.105+0000 7f0286d9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f028808eeb0 0x7f02880a2c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:34:36.125 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:36.105+0000 7f0286d9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f028808eeb0 0x7f02880a2c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:36754/0 (socket says 172.21.15.42:36754) 2019-12-05T05:34:36.126 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:36.105+0000 7f0286d9d700 1 -- 172.21.15.42:0/2909248771 learned_addr learned my addr 172.21.15.42:0/2909248771 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:34:36.126 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:36.105+0000 7f028659c700 1 --2- 172.21.15.42:0/2909248771 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f02880a19e0 0x7f02880a1e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:34:36.126 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:36.105+0000 7f028759e700 1 --2- 172.21.15.42:0/2909248771 >> v2:172.21.15.40:3300/0 conn(0x7f02880789b0 0x7f0288078de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:34:36.126 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:36.105+0000 7f0286d9d700 1 -- 172.21.15.42:0/2909248771 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f02880a19e0 msgr2=0x7f02880a1e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:34:36.127 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:36.105+0000 7f0286d9d700 1 --2- 172.21.15.42:0/2909248771 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f02880a19e0 0x7f02880a1e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:34:36.127 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:36.105+0000 7f0286d9d700 1 -- 172.21.15.42:0/2909248771 >> v2:172.21.15.40:3300/0 conn(0x7f02880789b0 msgr2=0x7f0288078de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:34:36.127 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:36.105+0000 7f0286d9d700 1 --2- 172.21.15.42:0/2909248771 >> v2:172.21.15.40:3300/0 conn(0x7f02880789b0 0x7f0288078de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:34:36.127 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:36.105+0000 7f0286d9d700 1 -- 172.21.15.42:0/2909248771 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0288064850 con 0x7f028808eeb0 2019-12-05T05:34:36.128 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:36.109+0000 7f0286d9d700 1 --2- 172.21.15.42:0/2909248771 >> v2:172.21.15.42:3300/0 conn(0x7f028808eeb0 0x7f02880a2c60 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f0288079f70 tx=0x7f0278008c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:36.128 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:36.109+0000 7f0286d9d700 1 --2- 172.21.15.42:0/2909248771 >> v2:172.21.15.42:3300/0 conn(0x7f028808eeb0 0x7f02880a2c60 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f0288079f70 tx=0x7f0278008c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:36.128 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:36.109+0000 7f0286d9d700 1 --2- 172.21.15.42:0/2909248771 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f028808eeb0 0x7f02880a2c60 secure :-1 s=READY pgs=1126 cs=0 l=1 rx=0x7f0288079f70 tx=0x7f0278008c30).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:34:36.128 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:36.109+0000 7f0286d9d700 1 --2- 172.21.15.42:0/2909248771 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f028808eeb0 0x7f02880a2c60 secure :-1 s=THROTTLE_DONE pgs=1126 cs=0 l=1 rx=0x7f0288079f70 tx=0x7f0278008c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:36.128 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:36.109+0000 7f02777fe700 1 -- 172.21.15.42:0/2909248771 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0278007340 con 0x7f028808eeb0 2019-12-05T05:34:36.128 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:36.109+0000 7f0286d9d700 1 --2- 172.21.15.42:0/2909248771 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f028808eeb0 0x7f02880a2c60 secure :-1 s=THROTTLE_DONE pgs=1126 cs=0 l=1 rx=0x7f0288079f70 tx=0x7f0278008c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:36.128 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:36.109+0000 7f0286d9d700 1 --2- 172.21.15.42:0/2909248771 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f028808eeb0 0x7f02880a2c60 secure :-1 s=THROTTLE_DONE pgs=1126 cs=0 l=1 rx=0x7f0288079f70 tx=0x7f0278008c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:36.129 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:36.109+0000 7f02777fe700 1 -- 172.21.15.42:0/2909248771 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02780115b0 con 0x7f028808eeb0 2019-12-05T05:34:36.129 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:36.109+0000 7f028cdb9700 1 -- 172.21.15.42:0/2909248771 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f028814eb90 con 0x7f028808eeb0 2019-12-05T05:34:36.129 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:36.109+0000 7f02777fe700 1 -- 172.21.15.42:0/2909248771 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0278005540 con 0x7f028808eeb0 2019-12-05T05:34:36.129 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:36.109+0000 7f028cdb9700 1 -- 172.21.15.42:0/2909248771 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f028812a1d0 con 0x7f028808eeb0 2019-12-05T05:34:36.129 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:36.109+0000 7f0286d9d700 1 --2- 172.21.15.42:0/2909248771 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f028808eeb0 0x7f02880a2c60 secure :-1 s=THROTTLE_DONE pgs=1126 cs=0 l=1 rx=0x7f0288079f70 tx=0x7f0278008c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:36.129 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:36.109+0000 7f02777fe700 1 -- 172.21.15.42:0/2909248771 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f0278005860 con 0x7f028808eeb0 2019-12-05T05:34:36.130 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:36.109+0000 7f0286d9d700 1 --2- 172.21.15.42:0/2909248771 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f028808eeb0 0x7f02880a2c60 secure :-1 s=THROTTLE_DONE pgs=1126 cs=0 l=1 rx=0x7f0288079f70 tx=0x7f0278008c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:36.130 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:36.109+0000 7f02777fe700 1 --2- 172.21.15.42:0/2909248771 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f026c03ae10 0x7f026c03d320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:34:36.130 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:36.109+0000 7f02777fe700 1 -- 172.21.15.42:0/2909248771 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 7923+0+0 (secure 0 0 0) 0x7f027804b3c0 con 0x7f028808eeb0 2019-12-05T05:34:36.130 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:36.109+0000 7f028659c700 1 --2- 172.21.15.42:0/2909248771 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f026c03ae10 0x7f026c03d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:34:36.130 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:36.109+0000 7f028659c700 1 --2- 172.21.15.42:0/2909248771 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f026c03ae10 0x7f026c03d320 crc :-1 s=READY pgs=487 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:34:36.131 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:36.113+0000 7f028cdb9700 1 -- 172.21.15.42:0/2909248771 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f028805d110 con 0x7f028808eeb0 2019-12-05T05:34:36.131 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:36.121+0000 7f0286d9d700 1 --2- 172.21.15.42:0/2909248771 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f028808eeb0 0x7f02880a2c60 secure :-1 s=THROTTLE_DONE pgs=1126 cs=0 l=1 rx=0x7f0288079f70 tx=0x7f0278008c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:36.132 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:36.125+0000 7f02777fe700 1 -- 172.21.15.42:0/2909248771 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f02780075c0 con 0x7f028808eeb0 2019-12-05T05:34:36.295 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:36.289+0000 7f028cdb9700 1 -- 172.21.15.42:0/2909248771 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f028805b8c0 con 0x7f028808eeb0 2019-12-05T05:34:36.295 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:36.289+0000 7f0286d9d700 1 --2- 172.21.15.42:0/2909248771 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f028808eeb0 0x7f02880a2c60 secure :-1 s=THROTTLE_DONE pgs=1126 cs=0 l=1 rx=0x7f0288079f70 tx=0x7f0278008c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:36.296 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:36.289+0000 7f02777fe700 1 -- 172.21.15.42:0/2909248771 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 139 v139) v1 ==== 95+0+1774 (secure 0 0 0) 0x7f0278032420 con 0x7f028808eeb0 2019-12-05T05:34:36.296 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 139 2019-12-05T05:34:36.296 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:34:36.296 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":139,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":139,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:34:33.763070+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7221},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7221":{"gid":7221,"name":"b","rank":0,"incarnation":139,"state":"up:replay","state_seq":2,"addr":"172.21.15.40:6833/3789641053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3789641053},{"type":"v1","addr":"172.21.15.40:6833","nonce":3789641053}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:34:36.299 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:36.293+0000 7f028cdb9700 1 -- 172.21.15.42:0/2909248771 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f026c03ae10 msgr2=0x7f026c03d320 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:34:36.299 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:36.293+0000 7f028cdb9700 1 --2- 172.21.15.42:0/2909248771 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f026c03ae10 0x7f026c03d320 crc :-1 s=READY pgs=487 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:34:36.299 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:36.293+0000 7f028cdb9700 1 -- 172.21.15.42:0/2909248771 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f028808eeb0 msgr2=0x7f02880a2c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:34:36.299 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:36.293+0000 7f028cdb9700 1 --2- 172.21.15.42:0/2909248771 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f028808eeb0 0x7f02880a2c60 secure :-1 s=READY pgs=1126 cs=0 l=1 rx=0x7f0288079f70 tx=0x7f0278008c30).stop 2019-12-05T05:34:36.299 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:36.293+0000 7f028cdb9700 1 -- 172.21.15.42:0/2909248771 shutdown_connections 2019-12-05T05:34:36.299 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:36.293+0000 7f028cdb9700 1 --2- 172.21.15.42:0/2909248771 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f026c03ae10 0x7f026c03d320 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:34:36.300 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:36.293+0000 7f028cdb9700 1 --2- 172.21.15.42:0/2909248771 >> v2:172.21.15.40:3300/0 conn(0x7f02880789b0 0x7f0288078de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:34:36.300 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:36.293+0000 7f028cdb9700 1 --2- 172.21.15.42:0/2909248771 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f028808eeb0 0x7f02880a2c60 unknown :-1 s=CLOSED pgs=1126 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:34:36.300 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:36.293+0000 7f028cdb9700 1 --2- 172.21.15.42:0/2909248771 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f02880a19e0 0x7f02880a1e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:34:36.300 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:36.293+0000 7f028cdb9700 1 -- 172.21.15.42:0/2909248771 shutdown_connections 2019-12-05T05:34:36.300 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:36.293+0000 7f028cdb9700 1 -- 172.21.15.42:0/2909248771 wait complete. 2019-12-05T05:34:36.300 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:36.293+0000 7f028cdb9700 1 -- 172.21.15.42:0/2909248771 >> 172.21.15.42:0/2909248771 conn(0x7f02880a82a0 msgr2=0x7f02880a4840 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:34:36.314 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 139, "feature_flags": { "enable_multiple": false, "ever_enabled_multiple": false }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2019-12-05T05:03:35.604738+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 139, "ever_allowed_features": 2, "explicitly_allowed_features": 2, "failed": [], "flags": 18, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_7221": { "addr": "172.21.15.40:6833/3789641053", "addrs": { "addrvec": [ { "addr": "172.21.15.40:6832", "nonce": 3789641053, "type": "v2" }, { "addr": "172.21.15.40:6833", "nonce": 3789641053, "type": "v1" } ] }, "export_targets": [], "features": 4540138292836696063, "flags": 0, "gid": 7221, "incarnation": 139, "name": "b", "rank": 0, "state": "up:replay", "state_seq": 2 } }, "last_failure": 0, "last_failure_osd_epoch": 71, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 2, "min_compat_client": "0 (unknown)", "modified": "2019-12-05T05:34:33.763070+0000", "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7221 } } } ], "standbys": [] } 2019-12-05T05:34:38.321 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:34:38.429 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:38.421+0000 7f696b4a7700 1 Processor -- start 2019-12-05T05:34:38.430 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:38.421+0000 7f696b4a7700 1 -- start start 2019-12-05T05:34:38.430 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:38.421+0000 7f696b4a7700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f69640a7740 0x7f696408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:34:38.430 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:38.421+0000 7f696b4a7700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f696409fae0 0x7f696407eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:34:38.430 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:38.421+0000 7f696b4a7700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69640a0710 con 0x7f696407e720 2019-12-05T05:34:38.430 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:38.421+0000 7f696b4a7700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69640a0b00 con 0x7f696409fae0 2019-12-05T05:34:38.430 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:38.421+0000 7f696b4a7700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f696408e5b0 con 0x7f69640a7740 2019-12-05T05:34:38.430 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:38.421+0000 7f6969243700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f696407e720 0x7f696409f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:36804/0 (socket says 172.21.15.42:36804) 2019-12-05T05:34:38.431 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:38.421+0000 7f6969243700 1 -- 172.21.15.42:0/1754910754 learned_addr learned my addr 172.21.15.42:0/1754910754 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:34:38.431 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:38.421+0000 7f6969a44700 1 --2- 172.21.15.42:0/1754910754 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f69640a7740 0x7f696408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:34:38.431 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:38.421+0000 7f696a245700 1 --2- 172.21.15.42:0/1754910754 >> v2:172.21.15.40:3300/0 conn(0x7f696409fae0 0x7f696407eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:34:38.431 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:38.421+0000 7f6968a42700 1 -- 172.21.15.42:0/1754910754 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 426269882 0 0) 0x7f69640a0710 con 0x7f696407e720 2019-12-05T05:34:38.431 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:38.421+0000 7f6968a42700 1 -- 172.21.15.42:0/1754910754 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f694c0030b0 con 0x7f696407e720 2019-12-05T05:34:38.431 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:38.421+0000 7f6968a42700 1 -- 172.21.15.42:0/1754910754 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1957242193 0 0) 0x7f694c0030b0 con 0x7f696407e720 2019-12-05T05:34:38.431 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:38.421+0000 7f696a245700 1 -- 172.21.15.42:0/1754910754 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f69640a7740 msgr2=0x7f696408df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:34:38.432 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:38.421+0000 7f696a245700 1 --2- 172.21.15.42:0/1754910754 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f69640a7740 0x7f696408df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:34:38.432 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:38.421+0000 7f696a245700 1 -- 172.21.15.42:0/1754910754 >> v1:172.21.15.42:6789/0 conn(0x7f696407e720 legacy=0x7f696409f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:34:38.432 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:38.421+0000 7f696a245700 1 -- 172.21.15.42:0/1754910754 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f696407f0a0 con 0x7f696409fae0 2019-12-05T05:34:38.432 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:38.421+0000 7f6969a44700 1 --2- 172.21.15.42:0/1754910754 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f69640a7740 0x7f696408df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:34:38.432 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:38.421+0000 7f696a245700 1 --2- 172.21.15.42:0/1754910754 >> v2:172.21.15.40:3300/0 conn(0x7f696409fae0 0x7f696407eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f6954008790 tx=0x7f69580019d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:38.432 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:38.421+0000 7f696a245700 1 --2- 172.21.15.42:0/1754910754 >> v2:172.21.15.40:3300/0 conn(0x7f696409fae0 0x7f696407eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f6954008790 tx=0x7f69580019d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:38.433 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:38.421+0000 7f696a245700 1 --2- 172.21.15.42:0/1754910754 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f696409fae0 0x7f696407eb50 secure :-1 s=READY pgs=892 cs=0 l=1 rx=0x7f6954008790 tx=0x7f69580019d0).ready entity=mon.0 client_cookie=acd8ebaa774f1d46 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:34:38.433 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:38.421+0000 7f696a245700 1 --2- 172.21.15.42:0/1754910754 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f696409fae0 0x7f696407eb50 secure :-1 s=THROTTLE_DONE pgs=892 cs=0 l=1 rx=0x7f6954008790 tx=0x7f69580019d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:38.433 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:38.421+0000 7f6968a42700 1 -- 172.21.15.42:0/1754910754 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6958001750 con 0x7f696409fae0 2019-12-05T05:34:38.433 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:38.425+0000 7f696a245700 1 --2- 172.21.15.42:0/1754910754 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f696409fae0 0x7f696407eb50 secure :-1 s=THROTTLE_DONE pgs=892 cs=0 l=1 rx=0x7f6954008790 tx=0x7f69580019d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:38.433 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:38.425+0000 7f696a245700 1 --2- 172.21.15.42:0/1754910754 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f696409fae0 0x7f696407eb50 secure :-1 s=THROTTLE_DONE pgs=892 cs=0 l=1 rx=0x7f6954008790 tx=0x7f69580019d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:38.433 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:38.425+0000 7f6968a42700 1 -- 172.21.15.42:0/1754910754 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6958001560 con 0x7f696409fae0 2019-12-05T05:34:38.433 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:38.425+0000 7f6968a42700 1 -- 172.21.15.42:0/1754910754 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6954008da0 con 0x7f696409fae0 2019-12-05T05:34:38.433 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:38.425+0000 7f696b4a7700 1 -- 172.21.15.42:0/1754910754 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f696409fae0 msgr2=0x7f696407eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:34:38.434 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:38.425+0000 7f696b4a7700 1 --2- 172.21.15.42:0/1754910754 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f696409fae0 0x7f696407eb50 secure :-1 s=READY pgs=892 cs=0 l=1 rx=0x7f6954008790 tx=0x7f69580019d0).stop 2019-12-05T05:34:38.434 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:38.425+0000 7f696b4a7700 1 -- 172.21.15.42:0/1754910754 shutdown_connections 2019-12-05T05:34:38.434 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:38.425+0000 7f696b4a7700 1 --2- 172.21.15.42:0/1754910754 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f696409fae0 0x7f696407eb50 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:34:38.434 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:38.425+0000 7f696b4a7700 1 --2- 172.21.15.42:0/1754910754 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f69640a7740 0x7f696408df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:34:38.434 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:38.425+0000 7f696b4a7700 1 -- 172.21.15.42:0/1754910754 shutdown_connections 2019-12-05T05:34:38.434 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:38.425+0000 7f696b4a7700 1 -- 172.21.15.42:0/1754910754 wait complete. 2019-12-05T05:34:38.434 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:38.425+0000 7f696b4a7700 1 -- 172.21.15.42:0/1754910754 >> 172.21.15.42:0/1754910754 conn(0x7f69640aabe0 msgr2=0x7f69640a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:34:38.438 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:38.425+0000 7f696b4a7700 1 Processor -- start 2019-12-05T05:34:38.438 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:38.425+0000 7f696b4a7700 1 -- start start 2019-12-05T05:34:38.438 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:38.429+0000 7f696b4a7700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6964116500 0x7f696411b450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:34:38.438 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:38.429+0000 7f696b4a7700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f6964145d30 0x7f6964146160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:34:38.439 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:38.429+0000 7f696b4a7700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f696407c030 con 0x7f6964130930 2019-12-05T05:34:38.439 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:38.429+0000 7f696b4a7700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69641466b0 con 0x7f6964145d30 2019-12-05T05:34:38.439 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:38.429+0000 7f696b4a7700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f696411af10 con 0x7f6964116500 2019-12-05T05:34:38.439 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:38.429+0000 7f6969243700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f6964130930 0x7f696407b830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:36810/0 (socket says 172.21.15.42:36810) 2019-12-05T05:34:38.439 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:38.429+0000 7f6969243700 1 -- 172.21.15.42:0/2455959959 learned_addr learned my addr 172.21.15.42:0/2455959959 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:34:38.439 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:38.429+0000 7f6969a44700 1 --2- 172.21.15.42:0/2455959959 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6964116500 0x7f696411b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:34:38.439 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:38.429+0000 7f696a245700 1 --2- 172.21.15.42:0/2455959959 >> v2:172.21.15.40:3300/0 conn(0x7f6964145d30 0x7f6964146160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:34:38.440 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:38.429+0000 7f69627fc700 1 -- 172.21.15.42:0/2455959959 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 651159369 0 0) 0x7f696407c030 con 0x7f6964130930 2019-12-05T05:34:38.440 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:38.429+0000 7f69627fc700 1 -- 172.21.15.42:0/2455959959 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69440030b0 con 0x7f6964130930 2019-12-05T05:34:38.440 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:38.429+0000 7f6969a44700 1 -- 172.21.15.42:0/2455959959 >> v2:172.21.15.40:3300/0 conn(0x7f6964145d30 msgr2=0x7f6964146160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:34:38.440 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:38.429+0000 7f6969a44700 1 --2- 172.21.15.42:0/2455959959 >> v2:172.21.15.40:3300/0 conn(0x7f6964145d30 0x7f6964146160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:34:38.440 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:38.429+0000 7f6969a44700 1 -- 172.21.15.42:0/2455959959 >> v1:172.21.15.42:6789/0 conn(0x7f6964130930 legacy=0x7f696407b830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:34:38.440 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:38.429+0000 7f6969a44700 1 -- 172.21.15.42:0/2455959959 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f696411b9a0 con 0x7f6964116500 2019-12-05T05:34:38.440 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:38.429+0000 7f6969a44700 1 --2- 172.21.15.42:0/2455959959 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6964116500 0x7f696411b450 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f6958001b20 tx=0x7f6958001b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:38.441 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:38.429+0000 7f6969a44700 1 --2- 172.21.15.42:0/2455959959 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6964116500 0x7f696411b450 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f6958001b20 tx=0x7f6958001b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:38.441 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:38.429+0000 7f6969a44700 1 --2- 172.21.15.42:0/2455959959 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6964116500 0x7f696411b450 secure :-1 s=READY pgs=236 cs=0 l=1 rx=0x7f6958001b20 tx=0x7f6958001b60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:34:38.441 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:38.429+0000 7f6969a44700 1 --2- 172.21.15.42:0/2455959959 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6964116500 0x7f696411b450 secure :-1 s=THROTTLE_DONE pgs=236 cs=0 l=1 rx=0x7f6958001b20 tx=0x7f6958001b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:38.441 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:38.429+0000 7f69627fc700 1 -- 172.21.15.42:0/2455959959 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f695c0058c0 con 0x7f6964116500 2019-12-05T05:34:38.441 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:38.429+0000 7f6969a44700 1 --2- 172.21.15.42:0/2455959959 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6964116500 0x7f696411b450 secure :-1 s=THROTTLE_DONE pgs=236 cs=0 l=1 rx=0x7f6958001b20 tx=0x7f6958001b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:38.441 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:38.429+0000 7f6969a44700 1 --2- 172.21.15.42:0/2455959959 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6964116500 0x7f696411b450 secure :-1 s=THROTTLE_DONE pgs=236 cs=0 l=1 rx=0x7f6958001b20 tx=0x7f6958001b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:38.441 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:38.429+0000 7f69627fc700 1 -- 172.21.15.42:0/2455959959 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f695c00e160 con 0x7f6964116500 2019-12-05T05:34:38.442 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:38.429+0000 7f69627fc700 1 -- 172.21.15.42:0/2455959959 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f695c00c850 con 0x7f6964116500 2019-12-05T05:34:38.442 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:38.429+0000 7f696b4a7700 1 -- 172.21.15.42:0/2455959959 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6964137d80 con 0x7f6964116500 2019-12-05T05:34:38.442 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:38.429+0000 7f696b4a7700 1 -- 172.21.15.42:0/2455959959 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6964138300 con 0x7f6964116500 2019-12-05T05:34:38.442 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:38.429+0000 7f6969a44700 1 --2- 172.21.15.42:0/2455959959 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6964116500 0x7f696411b450 secure :-1 s=THROTTLE_DONE pgs=236 cs=0 l=1 rx=0x7f6958001b20 tx=0x7f6958001b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:38.442 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:38.429+0000 7f6969a44700 1 --2- 172.21.15.42:0/2455959959 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6964116500 0x7f696411b450 secure :-1 s=THROTTLE_DONE pgs=236 cs=0 l=1 rx=0x7f6958001b20 tx=0x7f6958001b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:38.442 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:38.429+0000 7f69627fc700 1 -- 172.21.15.42:0/2455959959 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f695c00cb60 con 0x7f6964116500 2019-12-05T05:34:38.442 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:38.433+0000 7f69627fc700 1 --2- 172.21.15.42:0/2455959959 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f694403b2d0 0x7f694403d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:34:38.443 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:38.433+0000 7f69627fc700 1 -- 172.21.15.42:0/2455959959 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 7923+0+0 (secure 0 0 0) 0x7f695c04aef0 con 0x7f6964116500 2019-12-05T05:34:38.443 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:38.433+0000 7f6969243700 1 --2- 172.21.15.42:0/2455959959 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f694403b2d0 0x7f694403d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:34:38.443 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:38.433+0000 7f6969243700 1 --2- 172.21.15.42:0/2455959959 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f694403b2d0 0x7f694403d7e0 crc :-1 s=READY pgs=488 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:34:38.443 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:38.437+0000 7f696b4a7700 1 -- 172.21.15.42:0/2455959959 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f696405d110 con 0x7f6964116500 2019-12-05T05:34:38.453 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:38.445+0000 7f6969a44700 1 --2- 172.21.15.42:0/2455959959 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6964116500 0x7f696411b450 secure :-1 s=THROTTLE_DONE pgs=236 cs=0 l=1 rx=0x7f6958001b20 tx=0x7f6958001b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:38.454 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:38.445+0000 7f69627fc700 1 -- 172.21.15.42:0/2455959959 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f695c032190 con 0x7f6964116500 2019-12-05T05:34:38.620 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:38.617+0000 7f696b4a7700 1 -- 172.21.15.42:0/2455959959 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f69640a6990 con 0x7f6964116500 2019-12-05T05:34:38.621 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:38.617+0000 7f6969a44700 1 --2- 172.21.15.42:0/2455959959 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6964116500 0x7f696411b450 secure :-1 s=THROTTLE_DONE pgs=236 cs=0 l=1 rx=0x7f6958001b20 tx=0x7f6958001b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:38.621 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:38.617+0000 7f69627fc700 1 -- 172.21.15.42:0/2455959959 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 141 v141) v1 ==== 95+0+1775 (secure 0 0 0) 0x7f695c032d40 con 0x7f6964116500 2019-12-05T05:34:38.623 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 141 2019-12-05T05:34:38.624 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:34:38.624 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":141,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":141,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:34:38.032868+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7221},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7221":{"gid":7221,"name":"b","rank":0,"incarnation":139,"state":"up:rejoin","state_seq":26,"addr":"172.21.15.40:6833/3789641053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3789641053},{"type":"v1","addr":"172.21.15.40:6833","nonce":3789641053}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:34:38.626 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:38.621+0000 7f696b4a7700 1 -- 172.21.15.42:0/2455959959 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f694403b2d0 msgr2=0x7f694403d7e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:34:38.626 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:38.621+0000 7f696b4a7700 1 --2- 172.21.15.42:0/2455959959 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f694403b2d0 0x7f694403d7e0 crc :-1 s=READY pgs=488 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:34:38.626 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:38.621+0000 7f696b4a7700 1 -- 172.21.15.42:0/2455959959 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6964116500 msgr2=0x7f696411b450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:34:38.627 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:38.621+0000 7f696b4a7700 1 --2- 172.21.15.42:0/2455959959 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6964116500 0x7f696411b450 secure :-1 s=READY pgs=236 cs=0 l=1 rx=0x7f6958001b20 tx=0x7f6958001b60).stop 2019-12-05T05:34:38.627 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:38.621+0000 7f696b4a7700 1 -- 172.21.15.42:0/2455959959 shutdown_connections 2019-12-05T05:34:38.627 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:38.621+0000 7f696b4a7700 1 --2- 172.21.15.42:0/2455959959 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f694403b2d0 0x7f694403d7e0 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:34:38.627 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:38.621+0000 7f696b4a7700 1 --2- 172.21.15.42:0/2455959959 >> v2:172.21.15.40:3300/0 conn(0x7f6964145d30 0x7f6964146160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:34:38.628 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:38.621+0000 7f696b4a7700 1 --2- 172.21.15.42:0/2455959959 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6964116500 0x7f696411b450 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:34:38.628 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:38.621+0000 7f696b4a7700 1 -- 172.21.15.42:0/2455959959 shutdown_connections 2019-12-05T05:34:38.628 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:38.621+0000 7f696b4a7700 1 -- 172.21.15.42:0/2455959959 wait complete. 2019-12-05T05:34:38.628 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:38.621+0000 7f696b4a7700 1 -- 172.21.15.42:0/2455959959 >> 172.21.15.42:0/2455959959 conn(0x7f69640a6bb0 msgr2=0x7f696407e4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:34:38.636 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2019-12-05T05:34:39.538 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:34:39.542 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:34:40.639 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:34:40.745 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:40.737+0000 7f64f15b7700 1 Processor -- start 2019-12-05T05:34:40.745 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:40.737+0000 7f64f15b7700 1 -- start start 2019-12-05T05:34:40.745 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:40.737+0000 7f64f15b7700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f64ec0a7740 0x7f64ec08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:34:40.745 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:40.737+0000 7f64f15b7700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f64ec07e720 0x7f64ec07eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:34:40.746 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:40.737+0000 7f64f15b7700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f64ec09f2e0 0x7f64ec09f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:34:40.746 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:40.737+0000 7f64f15b7700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64ec0a0b00 con 0x7f64ec0a7740 2019-12-05T05:34:40.746 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:40.737+0000 7f64f15b7700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64ec08e5b0 con 0x7f64ec09f2e0 2019-12-05T05:34:40.746 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:40.737+0000 7f64f15b7700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64ec08e850 con 0x7f64ec07e720 2019-12-05T05:34:40.746 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:40.737+0000 7f64ebfff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f64ec09f2e0 0x7f64ec09f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:34:40.746 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:40.737+0000 7f64ebfff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f64ec09f2e0 0x7f64ec09f710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:36784/0 (socket says 172.21.15.42:36784) 2019-12-05T05:34:40.747 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:40.737+0000 7f64eaffd700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f64ec07e720 0x7f64ec07eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:34:40.747 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:40.737+0000 7f64ebfff700 1 -- 172.21.15.42:0/2516064689 learned_addr learned my addr 172.21.15.42:0/2516064689 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:34:40.747 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:40.737+0000 7f64eb7fe700 1 --2- 172.21.15.42:0/2516064689 >> v2:172.21.15.40:3300/0 conn(0x7f64ec0a7740 0x7f64ec08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:34:40.747 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:40.737+0000 7f64ebfff700 1 -- 172.21.15.42:0/2516064689 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f64ec07e720 msgr2=0x7f64ec07eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:34:40.747 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:40.737+0000 7f64ebfff700 1 --2- 172.21.15.42:0/2516064689 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f64ec07e720 0x7f64ec07eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:34:40.748 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:40.737+0000 7f64ebfff700 1 -- 172.21.15.42:0/2516064689 >> v2:172.21.15.40:3300/0 conn(0x7f64ec0a7740 msgr2=0x7f64ec08df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:34:40.748 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:40.737+0000 7f64ebfff700 1 --2- 172.21.15.42:0/2516064689 >> v2:172.21.15.40:3300/0 conn(0x7f64ec0a7740 0x7f64ec08df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:34:40.748 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:40.737+0000 7f64ebfff700 1 -- 172.21.15.42:0/2516064689 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64ec07f0a0 con 0x7f64ec09f2e0 2019-12-05T05:34:40.748 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:40.737+0000 7f64ebfff700 1 --2- 172.21.15.42:0/2516064689 >> v2:172.21.15.42:3300/0 conn(0x7f64ec09f2e0 0x7f64ec09f710 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f64ec08e770 tx=0x7f64e4010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:40.748 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:40.737+0000 7f64ebfff700 1 --2- 172.21.15.42:0/2516064689 >> v2:172.21.15.42:3300/0 conn(0x7f64ec09f2e0 0x7f64ec09f710 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f64ec08e770 tx=0x7f64e4010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:40.749 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:40.737+0000 7f64ebfff700 1 --2- 172.21.15.42:0/2516064689 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f64ec09f2e0 0x7f64ec09f710 secure :-1 s=READY pgs=1129 cs=0 l=1 rx=0x7f64ec08e770 tx=0x7f64e4010130).ready entity=mon.1 client_cookie=e514029bf5225235 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:34:40.749 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:40.737+0000 7f64ebfff700 1 --2- 172.21.15.42:0/2516064689 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f64ec09f2e0 0x7f64ec09f710 secure :-1 s=THROTTLE_DONE pgs=1129 cs=0 l=1 rx=0x7f64ec08e770 tx=0x7f64e4010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:40.749 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:40.737+0000 7f64ea7fc700 1 -- 172.21.15.42:0/2516064689 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f64e4010de0 con 0x7f64ec09f2e0 2019-12-05T05:34:40.749 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:40.737+0000 7f64ebfff700 1 --2- 172.21.15.42:0/2516064689 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f64ec09f2e0 0x7f64ec09f710 secure :-1 s=THROTTLE_DONE pgs=1129 cs=0 l=1 rx=0x7f64ec08e770 tx=0x7f64e4010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:40.749 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:40.737+0000 7f64ea7fc700 1 -- 172.21.15.42:0/2516064689 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64e4010a00 con 0x7f64ec09f2e0 2019-12-05T05:34:40.749 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:40.737+0000 7f64ebfff700 1 --2- 172.21.15.42:0/2516064689 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f64ec09f2e0 0x7f64ec09f710 secure :-1 s=THROTTLE_DONE pgs=1129 cs=0 l=1 rx=0x7f64ec08e770 tx=0x7f64e4010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:40.749 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:40.737+0000 7f64f15b7700 1 -- 172.21.15.42:0/2516064689 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f64ec09f2e0 msgr2=0x7f64ec09f710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:34:40.750 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:40.737+0000 7f64ea7fc700 1 -- 172.21.15.42:0/2516064689 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f64e40114a0 con 0x7f64ec09f2e0 2019-12-05T05:34:40.750 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:40.737+0000 7f64f15b7700 1 --2- 172.21.15.42:0/2516064689 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f64ec09f2e0 0x7f64ec09f710 secure :-1 s=READY pgs=1129 cs=0 l=1 rx=0x7f64ec08e770 tx=0x7f64e4010130).stop 2019-12-05T05:34:40.750 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:40.741+0000 7f64f15b7700 1 -- 172.21.15.42:0/2516064689 shutdown_connections 2019-12-05T05:34:40.750 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:40.741+0000 7f64f15b7700 1 --2- 172.21.15.42:0/2516064689 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f64ec09f2e0 0x7f64ec09f710 unknown :-1 s=CLOSED pgs=1129 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:34:40.750 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:40.741+0000 7f64f15b7700 1 --2- 172.21.15.42:0/2516064689 >> v2:172.21.15.40:3300/0 conn(0x7f64ec0a7740 0x7f64ec08df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:34:40.750 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:40.741+0000 7f64f15b7700 1 --2- 172.21.15.42:0/2516064689 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f64ec07e720 0x7f64ec07eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:34:40.751 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:40.741+0000 7f64f15b7700 1 -- 172.21.15.42:0/2516064689 shutdown_connections 2019-12-05T05:34:40.751 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:40.741+0000 7f64f15b7700 1 -- 172.21.15.42:0/2516064689 wait complete. 2019-12-05T05:34:40.751 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:40.741+0000 7f64f15b7700 1 -- 172.21.15.42:0/2516064689 >> 172.21.15.42:0/2516064689 conn(0x7f64ec0aabe0 msgr2=0x7f64ec0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:34:40.761 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:40.741+0000 7f64f15b7700 1 Processor -- start 2019-12-05T05:34:40.761 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:40.741+0000 7f64f15b7700 1 -- start start 2019-12-05T05:34:40.762 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:40.745+0000 7f64f15b7700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f64ec0a5270 0x7f64ec138600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:34:40.762 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:40.745+0000 7f64f15b7700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64ec0a5d70 con 0x7f64ec1236e0 2019-12-05T05:34:40.762 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:40.745+0000 7f64f15b7700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64ec07d720 con 0x7f64ec0a1510 2019-12-05T05:34:40.762 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:40.745+0000 7f64f15b7700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64ec1301a0 con 0x7f64ec0a5270 2019-12-05T05:34:40.762 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:40.745+0000 7f64eb7fe700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f64ec0a1510 0x7f64ec1309d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:36826/0 (socket says 172.21.15.42:36826) 2019-12-05T05:34:40.763 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:40.745+0000 7f64eb7fe700 1 -- 172.21.15.42:0/3187253209 learned_addr learned my addr 172.21.15.42:0/3187253209 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:34:40.763 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:40.745+0000 7f64ebfff700 1 --2- 172.21.15.42:0/3187253209 >> v2:172.21.15.42:3300/0 conn(0x7f64ec0a5270 0x7f64ec138600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:34:40.763 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:40.745+0000 7f64e3fff700 1 -- 172.21.15.42:0/3187253209 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3777349149 0 0) 0x7f64ec07d720 con 0x7f64ec0a1510 2019-12-05T05:34:40.763 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:40.745+0000 7f64e3fff700 1 -- 172.21.15.42:0/3187253209 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64c40030b0 con 0x7f64ec0a1510 2019-12-05T05:34:40.763 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:40.745+0000 7f64e3fff700 1 -- 172.21.15.42:0/3187253209 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3900295641 0 0) 0x7f64ec0a5d70 con 0x7f64ec1236e0 2019-12-05T05:34:40.764 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:40.745+0000 7f64e3fff700 1 -- 172.21.15.42:0/3187253209 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64ec07d720 con 0x7f64ec1236e0 2019-12-05T05:34:40.764 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:40.745+0000 7f64ebfff700 1 -- 172.21.15.42:0/3187253209 >> v1:172.21.15.42:6789/0 conn(0x7f64ec0a1510 legacy=0x7f64ec1309d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:34:40.764 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:40.745+0000 7f64ebfff700 1 -- 172.21.15.42:0/3187253209 >> v1:172.21.15.40:6789/0 conn(0x7f64ec1236e0 legacy=0x7f64ec07cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:34:40.764 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:40.745+0000 7f64ebfff700 1 -- 172.21.15.42:0/3187253209 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64ec13ab70 con 0x7f64ec0a5270 2019-12-05T05:34:40.764 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:40.745+0000 7f64ebfff700 1 --2- 172.21.15.42:0/3187253209 >> v2:172.21.15.42:3300/0 conn(0x7f64ec0a5270 0x7f64ec138600 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f64ec08e770 tx=0x7f64e40114e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:40.766 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:40.745+0000 7f64ebfff700 1 --2- 172.21.15.42:0/3187253209 >> v2:172.21.15.42:3300/0 conn(0x7f64ec0a5270 0x7f64ec138600 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f64ec08e770 tx=0x7f64e40114e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:40.766 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:40.745+0000 7f64ebfff700 1 --2- 172.21.15.42:0/3187253209 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f64ec0a5270 0x7f64ec138600 secure :-1 s=READY pgs=1131 cs=0 l=1 rx=0x7f64ec08e770 tx=0x7f64e40114e0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:34:40.766 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:40.745+0000 7f64ebfff700 1 --2- 172.21.15.42:0/3187253209 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f64ec0a5270 0x7f64ec138600 secure :-1 s=THROTTLE_DONE pgs=1131 cs=0 l=1 rx=0x7f64ec08e770 tx=0x7f64e40114e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:40.766 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:40.745+0000 7f64e3fff700 1 -- 172.21.15.42:0/3187253209 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f64e4011b40 con 0x7f64ec0a5270 2019-12-05T05:34:40.766 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:40.745+0000 7f64ebfff700 1 --2- 172.21.15.42:0/3187253209 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f64ec0a5270 0x7f64ec138600 secure :-1 s=THROTTLE_DONE pgs=1131 cs=0 l=1 rx=0x7f64ec08e770 tx=0x7f64e40114e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:40.766 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:40.745+0000 7f64e3fff700 1 -- 172.21.15.42:0/3187253209 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64e40110c0 con 0x7f64ec0a5270 2019-12-05T05:34:40.767 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:40.745+0000 7f64ebfff700 1 --2- 172.21.15.42:0/3187253209 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f64ec0a5270 0x7f64ec138600 secure :-1 s=THROTTLE_DONE pgs=1131 cs=0 l=1 rx=0x7f64ec08e770 tx=0x7f64e40114e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:40.767 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:40.745+0000 7f64e3fff700 1 -- 172.21.15.42:0/3187253209 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f64e4005ab0 con 0x7f64ec0a5270 2019-12-05T05:34:40.767 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:40.745+0000 7f64f15b7700 1 -- 172.21.15.42:0/3187253209 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64ec138b50 con 0x7f64ec0a5270 2019-12-05T05:34:40.767 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:40.745+0000 7f64f15b7700 1 -- 172.21.15.42:0/3187253209 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64ec08eb70 con 0x7f64ec0a5270 2019-12-05T05:34:40.767 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:40.745+0000 7f64ebfff700 1 --2- 172.21.15.42:0/3187253209 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f64ec0a5270 0x7f64ec138600 secure :-1 s=THROTTLE_DONE pgs=1131 cs=0 l=1 rx=0x7f64ec08e770 tx=0x7f64e40114e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:40.767 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:40.749+0000 7f64e3fff700 1 -- 172.21.15.42:0/3187253209 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f64e400f280 con 0x7f64ec0a5270 2019-12-05T05:34:40.767 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:40.749+0000 7f64ebfff700 1 --2- 172.21.15.42:0/3187253209 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f64ec0a5270 0x7f64ec138600 secure :-1 s=THROTTLE_DONE pgs=1131 cs=0 l=1 rx=0x7f64ec08e770 tx=0x7f64e40114e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:40.768 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:40.749+0000 7f64e3fff700 1 --2- 172.21.15.42:0/3187253209 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f64c403b450 0x7f64c403d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:34:40.768 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:40.749+0000 7f64e3fff700 1 -- 172.21.15.42:0/3187253209 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 7923+0+0 (secure 0 0 0) 0x7f64e408f460 con 0x7f64ec0a5270 2019-12-05T05:34:40.768 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:40.749+0000 7f64eb7fe700 1 --2- 172.21.15.42:0/3187253209 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f64c403b450 0x7f64c403d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:34:40.768 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:40.749+0000 7f64eb7fe700 1 --2- 172.21.15.42:0/3187253209 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f64c403b450 0x7f64c403d960 crc :-1 s=READY pgs=489 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:34:40.769 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:40.753+0000 7f64f15b7700 1 -- 172.21.15.42:0/3187253209 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64ec05d110 con 0x7f64ec0a5270 2019-12-05T05:34:40.769 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:40.757+0000 7f64ebfff700 1 --2- 172.21.15.42:0/3187253209 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f64ec0a5270 0x7f64ec138600 secure :-1 s=THROTTLE_DONE pgs=1131 cs=0 l=1 rx=0x7f64ec08e770 tx=0x7f64e40114e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:40.769 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:40.757+0000 7f64e3fff700 1 -- 172.21.15.42:0/3187253209 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f64e4005d90 con 0x7f64ec0a5270 2019-12-05T05:34:40.927 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:40.921+0000 7f64f15b7700 1 -- 172.21.15.42:0/3187253209 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f64ec05b8c0 con 0x7f64ec0a5270 2019-12-05T05:34:40.928 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:40.921+0000 7f64ebfff700 1 --2- 172.21.15.42:0/3187253209 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f64ec0a5270 0x7f64ec138600 secure :-1 s=THROTTLE_DONE pgs=1131 cs=0 l=1 rx=0x7f64ec08e770 tx=0x7f64e40114e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:40.928 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:40.921+0000 7f64e3fff700 1 -- 172.21.15.42:0/3187253209 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 142 v142) v1 ==== 95+0+1775 (secure 0 0 0) 0x7f64e4074110 con 0x7f64ec0a5270 2019-12-05T05:34:40.929 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 142 2019-12-05T05:34:40.929 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:34:40.929 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":142,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":142,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:34:39.042388+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7221},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7221":{"gid":7221,"name":"b","rank":0,"incarnation":139,"state":"up:active","state_seq":27,"addr":"172.21.15.40:6833/3789641053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3789641053},{"type":"v1","addr":"172.21.15.40:6833","nonce":3789641053}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:34:40.931 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:40.925+0000 7f64f15b7700 1 -- 172.21.15.42:0/3187253209 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f64c403b450 msgr2=0x7f64c403d960 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:34:40.931 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:40.925+0000 7f64f15b7700 1 --2- 172.21.15.42:0/3187253209 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f64c403b450 0x7f64c403d960 crc :-1 s=READY pgs=489 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:34:40.931 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:40.925+0000 7f64f15b7700 1 -- 172.21.15.42:0/3187253209 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f64ec0a5270 msgr2=0x7f64ec138600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:34:40.932 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:40.925+0000 7f64f15b7700 1 --2- 172.21.15.42:0/3187253209 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f64ec0a5270 0x7f64ec138600 secure :-1 s=READY pgs=1131 cs=0 l=1 rx=0x7f64ec08e770 tx=0x7f64e40114e0).stop 2019-12-05T05:34:40.932 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:40.925+0000 7f64f15b7700 1 -- 172.21.15.42:0/3187253209 shutdown_connections 2019-12-05T05:34:40.932 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:40.925+0000 7f64f15b7700 1 --2- 172.21.15.42:0/3187253209 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f64c403b450 0x7f64c403d960 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:34:40.932 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:40.925+0000 7f64f15b7700 1 --2- 172.21.15.42:0/3187253209 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f64ec0a5270 0x7f64ec138600 unknown :-1 s=CLOSED pgs=1131 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:34:40.932 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:40.925+0000 7f64f15b7700 1 -- 172.21.15.42:0/3187253209 shutdown_connections 2019-12-05T05:34:40.932 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:40.925+0000 7f64f15b7700 1 -- 172.21.15.42:0/3187253209 wait complete. 2019-12-05T05:34:40.932 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:40.925+0000 7f64f15b7700 1 -- 172.21.15.42:0/3187253209 >> 172.21.15.42:0/3187253209 conn(0x7f64ec09fa10 msgr2=0x7f64ec059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:34:40.943 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b has gained rank=0, replacing gid=6997 2019-12-05T05:34:40.943 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 9 secs before reviving mds.a 2019-12-05T05:34:49.944 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.a 2019-12-05T05:34:49.944 INFO:tasks.ceph.mds.a:Restarting daemon 2019-12-05T05:34:49.945 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2019-12-05T05:34:49.949 INFO:tasks.ceph.mds.a:Started 2019-12-05T05:34:49.949 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:34:50.009 INFO:tasks.ceph.mds.a.smithi042.stdout:starting mds.a at 2019-12-05T05:34:50.051 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:50.040+0000 7efe81487700 1 Processor -- start 2019-12-05T05:34:50.051 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:50.040+0000 7efe81487700 1 -- start start 2019-12-05T05:34:50.051 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:50.044+0000 7efe81487700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7efe7c0a7740 0x7efe7c08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:34:50.051 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:50.044+0000 7efe81487700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efe7c09fae0 0x7efe7c07eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:34:50.051 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:50.044+0000 7efe81487700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe7c0a0710 con 0x7efe7c07e720 2019-12-05T05:34:50.052 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:50.044+0000 7efe81487700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe7c0a0b00 con 0x7efe7c0a7740 2019-12-05T05:34:50.052 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:50.044+0000 7efe81487700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe7c08e5b0 con 0x7efe7c09fae0 2019-12-05T05:34:50.052 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:50.044+0000 7efe7affd700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7efe7c07e720 0x7efe7c09f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:36888/0 (socket says 172.21.15.42:36888) 2019-12-05T05:34:50.052 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:50.044+0000 7efe7affd700 1 -- 172.21.15.42:0/2597432910 learned_addr learned my addr 172.21.15.42:0/2597432910 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:34:50.052 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:50.044+0000 7efe7b7fe700 1 --2- 172.21.15.42:0/2597432910 >> v2:172.21.15.42:3300/0 conn(0x7efe7c0a7740 0x7efe7c08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:34:50.052 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:50.044+0000 7efe7bfff700 1 --2- 172.21.15.42:0/2597432910 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efe7c09fae0 0x7efe7c07eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:34:50.053 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:50.044+0000 7efe7a7fc700 1 -- 172.21.15.42:0/2597432910 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2313626503 0 0) 0x7efe7c0a0710 con 0x7efe7c07e720 2019-12-05T05:34:50.053 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:50.044+0000 7efe7a7fc700 1 -- 172.21.15.42:0/2597432910 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe640030b0 con 0x7efe7c07e720 2019-12-05T05:34:50.053 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:50.044+0000 7efe7bfff700 1 -- 172.21.15.42:0/2597432910 >> v2:172.21.15.42:3300/0 conn(0x7efe7c0a7740 msgr2=0x7efe7c08df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:34:50.053 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:50.044+0000 7efe7bfff700 1 --2- 172.21.15.42:0/2597432910 >> v2:172.21.15.42:3300/0 conn(0x7efe7c0a7740 0x7efe7c08df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:34:50.053 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:50.044+0000 7efe7bfff700 1 -- 172.21.15.42:0/2597432910 >> v1:172.21.15.42:6789/0 conn(0x7efe7c07e720 legacy=0x7efe7c09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:34:50.053 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:50.044+0000 7efe7a7fc700 1 -- 172.21.15.42:0/2597432910 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1389306548 0 0) 0x7efe640030b0 con 0x7efe7c07e720 2019-12-05T05:34:50.053 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:50.044+0000 7efe7bfff700 1 -- 172.21.15.42:0/2597432910 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe7c07f0a0 con 0x7efe7c09fae0 2019-12-05T05:34:50.054 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:50.044+0000 7efe7b7fe700 1 --2- 172.21.15.42:0/2597432910 >> v2:172.21.15.42:3300/0 conn(0x7efe7c0a7740 0x7efe7c08df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:34:50.054 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:50.044+0000 7efe7bfff700 1 --2- 172.21.15.42:0/2597432910 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efe7c09fae0 0x7efe7c07eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7efe74008b10 tx=0x7efe700014a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:50.054 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:50.044+0000 7efe7bfff700 1 --2- 172.21.15.42:0/2597432910 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efe7c09fae0 0x7efe7c07eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7efe74008b10 tx=0x7efe700014a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:50.054 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:50.044+0000 7efe7bfff700 1 --2- 172.21.15.42:0/2597432910 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efe7c09fae0 0x7efe7c07eb50 secure :-1 s=READY pgs=238 cs=0 l=1 rx=0x7efe74008b10 tx=0x7efe700014a0).ready entity=mon.2 client_cookie=632eb7fb93a88f2e server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:34:50.054 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:50.044+0000 7efe7bfff700 1 --2- 172.21.15.42:0/2597432910 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efe7c09fae0 0x7efe7c07eb50 secure :-1 s=THROTTLE_DONE pgs=238 cs=0 l=1 rx=0x7efe74008b10 tx=0x7efe700014a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:50.054 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:50.044+0000 7efe7a7fc700 1 -- 172.21.15.42:0/2597432910 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efe70001750 con 0x7efe7c09fae0 2019-12-05T05:34:50.054 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:50.044+0000 7efe7bfff700 1 --2- 172.21.15.42:0/2597432910 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efe7c09fae0 0x7efe7c07eb50 secure :-1 s=THROTTLE_DONE pgs=238 cs=0 l=1 rx=0x7efe74008b10 tx=0x7efe700014a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:50.055 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:50.044+0000 7efe7bfff700 1 --2- 172.21.15.42:0/2597432910 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efe7c09fae0 0x7efe7c07eb50 secure :-1 s=THROTTLE_DONE pgs=238 cs=0 l=1 rx=0x7efe74008b10 tx=0x7efe700014a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:50.055 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:50.044+0000 7efe7a7fc700 1 -- 172.21.15.42:0/2597432910 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe70001560 con 0x7efe7c09fae0 2019-12-05T05:34:50.055 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:50.044+0000 7efe7a7fc700 1 -- 172.21.15.42:0/2597432910 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efe740109f0 con 0x7efe7c09fae0 2019-12-05T05:34:50.055 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:50.044+0000 7efe81487700 1 -- 172.21.15.42:0/2597432910 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efe7c09fae0 msgr2=0x7efe7c07eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:34:50.055 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:50.044+0000 7efe81487700 1 --2- 172.21.15.42:0/2597432910 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efe7c09fae0 0x7efe7c07eb50 secure :-1 s=READY pgs=238 cs=0 l=1 rx=0x7efe74008b10 tx=0x7efe700014a0).stop 2019-12-05T05:34:50.055 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:50.044+0000 7efe81487700 1 -- 172.21.15.42:0/2597432910 shutdown_connections 2019-12-05T05:34:50.055 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:50.044+0000 7efe81487700 1 --2- 172.21.15.42:0/2597432910 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efe7c09fae0 0x7efe7c07eb50 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:34:50.056 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:50.044+0000 7efe81487700 1 --2- 172.21.15.42:0/2597432910 >> v2:172.21.15.42:3300/0 conn(0x7efe7c0a7740 0x7efe7c08df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:34:50.056 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:50.044+0000 7efe81487700 1 -- 172.21.15.42:0/2597432910 shutdown_connections 2019-12-05T05:34:50.056 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:50.044+0000 7efe81487700 1 -- 172.21.15.42:0/2597432910 wait complete. 2019-12-05T05:34:50.056 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:50.044+0000 7efe81487700 1 -- 172.21.15.42:0/2597432910 >> 172.21.15.42:0/2597432910 conn(0x7efe7c0aabe0 msgr2=0x7efe7c0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:34:50.060 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:50.048+0000 7efe81487700 1 Processor -- start 2019-12-05T05:34:50.060 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:50.048+0000 7efe81487700 1 -- start start 2019-12-05T05:34:50.060 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:50.048+0000 7efe81487700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efe7c123aa0 0x7efe7c07af50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:34:50.060 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:50.048+0000 7efe81487700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7efe7c120b30 0x7efe7c120f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:34:50.061 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:50.048+0000 7efe81487700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe7c07d9b0 con 0x7efe7c07b4a0 2019-12-05T05:34:50.061 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:50.048+0000 7efe81487700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe7c07bc00 con 0x7efe7c120b30 2019-12-05T05:34:50.061 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:50.048+0000 7efe81487700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe7c07bea0 con 0x7efe7c123aa0 2019-12-05T05:34:50.061 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:50.048+0000 7efe7b7fe700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efe7c123aa0 0x7efe7c07af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:34:50.061 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:50.048+0000 7efe7affd700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7efe7c07b4a0 0x7efe7c07d1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:46334/0 (socket says 172.21.15.42:46334) 2019-12-05T05:34:50.061 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:50.048+0000 7efe7affd700 1 -- 172.21.15.42:0/1135748703 learned_addr learned my addr 172.21.15.42:0/1135748703 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:34:50.062 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:50.048+0000 7efe7bfff700 1 --2- 172.21.15.42:0/1135748703 >> v2:172.21.15.40:3300/0 conn(0x7efe7c120b30 0x7efe7c120f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:34:50.062 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:50.048+0000 7efe63fff700 1 -- 172.21.15.42:0/1135748703 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2082261304 0 0) 0x7efe7c07d9b0 con 0x7efe7c07b4a0 2019-12-05T05:34:50.062 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:50.048+0000 7efe63fff700 1 -- 172.21.15.42:0/1135748703 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe5c0030b0 con 0x7efe7c07b4a0 2019-12-05T05:34:50.062 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:50.048+0000 7efe7b7fe700 1 -- 172.21.15.42:0/1135748703 >> v2:172.21.15.40:3300/0 conn(0x7efe7c120b30 msgr2=0x7efe7c120f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:34:50.062 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:50.048+0000 7efe7b7fe700 1 --2- 172.21.15.42:0/1135748703 >> v2:172.21.15.40:3300/0 conn(0x7efe7c120b30 0x7efe7c120f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:34:50.062 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:50.048+0000 7efe7b7fe700 1 -- 172.21.15.42:0/1135748703 >> v1:172.21.15.40:6789/0 conn(0x7efe7c07b4a0 legacy=0x7efe7c07d1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:34:50.063 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:50.048+0000 7efe7b7fe700 1 -- 172.21.15.42:0/1135748703 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe7c1214b0 con 0x7efe7c123aa0 2019-12-05T05:34:50.063 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:50.048+0000 7efe7b7fe700 1 --2- 172.21.15.42:0/1135748703 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efe7c123aa0 0x7efe7c07af50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7efe6c002f30 tx=0x7efe6c010c20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:50.063 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:50.048+0000 7efe7b7fe700 1 --2- 172.21.15.42:0/1135748703 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efe7c123aa0 0x7efe7c07af50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7efe6c002f30 tx=0x7efe6c010c20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:50.063 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:50.048+0000 7efe7b7fe700 1 --2- 172.21.15.42:0/1135748703 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efe7c123aa0 0x7efe7c07af50 secure :-1 s=READY pgs=239 cs=0 l=1 rx=0x7efe6c002f30 tx=0x7efe6c010c20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:34:50.063 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:50.048+0000 7efe7b7fe700 1 --2- 172.21.15.42:0/1135748703 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efe7c123aa0 0x7efe7c07af50 secure :-1 s=THROTTLE_DONE pgs=239 cs=0 l=1 rx=0x7efe6c002f30 tx=0x7efe6c010c20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:50.063 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:50.048+0000 7efe63fff700 1 -- 172.21.15.42:0/1135748703 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efe6c00d0c0 con 0x7efe7c123aa0 2019-12-05T05:34:50.064 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:50.048+0000 7efe7b7fe700 1 --2- 172.21.15.42:0/1135748703 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efe7c123aa0 0x7efe7c07af50 secure :-1 s=THROTTLE_DONE pgs=239 cs=0 l=1 rx=0x7efe6c002f30 tx=0x7efe6c010c20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:50.064 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:50.048+0000 7efe63fff700 1 -- 172.21.15.42:0/1135748703 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe6c007150 con 0x7efe7c123aa0 2019-12-05T05:34:50.064 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:50.048+0000 7efe7b7fe700 1 --2- 172.21.15.42:0/1135748703 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efe7c123aa0 0x7efe7c07af50 secure :-1 s=THROTTLE_DONE pgs=239 cs=0 l=1 rx=0x7efe6c002f30 tx=0x7efe6c010c20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:50.064 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:50.048+0000 7efe81487700 1 -- 172.21.15.42:0/1135748703 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe7c07c3e0 con 0x7efe7c123aa0 2019-12-05T05:34:50.064 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:50.048+0000 7efe81487700 1 -- 172.21.15.42:0/1135748703 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe7c137940 con 0x7efe7c123aa0 2019-12-05T05:34:50.064 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:50.048+0000 7efe63fff700 1 -- 172.21.15.42:0/1135748703 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efe6c0129c0 con 0x7efe7c123aa0 2019-12-05T05:34:50.064 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:50.052+0000 7efe7b7fe700 1 --2- 172.21.15.42:0/1135748703 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efe7c123aa0 0x7efe7c07af50 secure :-1 s=THROTTLE_DONE pgs=239 cs=0 l=1 rx=0x7efe6c002f30 tx=0x7efe6c010c20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:50.065 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:50.052+0000 7efe63fff700 1 -- 172.21.15.42:0/1135748703 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7efe6c013060 con 0x7efe7c123aa0 2019-12-05T05:34:50.065 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:50.052+0000 7efe7b7fe700 1 --2- 172.21.15.42:0/1135748703 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efe7c123aa0 0x7efe7c07af50 secure :-1 s=THROTTLE_DONE pgs=239 cs=0 l=1 rx=0x7efe6c002f30 tx=0x7efe6c010c20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:50.065 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:50.052+0000 7efe63fff700 1 --2- 172.21.15.42:0/1135748703 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7efe5c03b2b0 0x7efe5c03d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:34:50.065 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:50.052+0000 7efe63fff700 1 -- 172.21.15.42:0/1135748703 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 7923+0+0 (secure 0 0 0) 0x7efe6c04b060 con 0x7efe7c123aa0 2019-12-05T05:34:50.065 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:50.052+0000 7efe7affd700 1 --2- 172.21.15.42:0/1135748703 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7efe5c03b2b0 0x7efe5c03d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:34:50.066 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:50.052+0000 7efe7affd700 1 --2- 172.21.15.42:0/1135748703 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7efe5c03b2b0 0x7efe5c03d7c0 crc :-1 s=READY pgs=490 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:34:50.068 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:50.056+0000 7efe81487700 1 -- 172.21.15.42:0/1135748703 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe7c05d110 con 0x7efe7c123aa0 2019-12-05T05:34:50.070 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:50.064+0000 7efe7b7fe700 1 --2- 172.21.15.42:0/1135748703 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efe7c123aa0 0x7efe7c07af50 secure :-1 s=THROTTLE_DONE pgs=239 cs=0 l=1 rx=0x7efe6c002f30 tx=0x7efe6c010c20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:50.071 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:50.064+0000 7efe63fff700 1 -- 172.21.15.42:0/1135748703 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7efe6c0320c0 con 0x7efe7c123aa0 2019-12-05T05:34:50.247 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:50.240+0000 7efe81487700 1 -- 172.21.15.42:0/1135748703 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efe7c05b8c0 con 0x7efe7c123aa0 2019-12-05T05:34:50.248 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:50.244+0000 7efe7b7fe700 1 --2- 172.21.15.42:0/1135748703 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efe7c123aa0 0x7efe7c07af50 secure :-1 s=THROTTLE_DONE pgs=239 cs=0 l=1 rx=0x7efe6c002f30 tx=0x7efe6c010c20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:50.248 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:50.244+0000 7efe63fff700 1 -- 172.21.15.42:0/1135748703 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 142 v142) v1 ==== 95+0+1775 (secure 0 0 0) 0x7efe6c032320 con 0x7efe7c123aa0 2019-12-05T05:34:50.248 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 142 2019-12-05T05:34:50.249 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:34:50.249 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":142,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":142,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:34:39.042388+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7221},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7221":{"gid":7221,"name":"b","rank":0,"incarnation":139,"state":"up:active","state_seq":27,"addr":"172.21.15.40:6833/3789641053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3789641053},{"type":"v1","addr":"172.21.15.40:6833","nonce":3789641053}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:34:50.258 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:50.244+0000 7efe81487700 1 -- 172.21.15.42:0/1135748703 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7efe5c03b2b0 msgr2=0x7efe5c03d7c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:34:50.258 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:50.244+0000 7efe81487700 1 --2- 172.21.15.42:0/1135748703 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7efe5c03b2b0 0x7efe5c03d7c0 crc :-1 s=READY pgs=490 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:34:50.258 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:50.244+0000 7efe81487700 1 -- 172.21.15.42:0/1135748703 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efe7c123aa0 msgr2=0x7efe7c07af50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:34:50.258 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:50.244+0000 7efe81487700 1 --2- 172.21.15.42:0/1135748703 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efe7c123aa0 0x7efe7c07af50 secure :-1 s=READY pgs=239 cs=0 l=1 rx=0x7efe6c002f30 tx=0x7efe6c010c20).stop 2019-12-05T05:34:50.258 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:50.244+0000 7efe81487700 1 -- 172.21.15.42:0/1135748703 shutdown_connections 2019-12-05T05:34:50.258 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:50.244+0000 7efe81487700 1 --2- 172.21.15.42:0/1135748703 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7efe5c03b2b0 0x7efe5c03d7c0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:34:50.259 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:50.244+0000 7efe81487700 1 --2- 172.21.15.42:0/1135748703 >> v2:172.21.15.40:3300/0 conn(0x7efe7c120b30 0x7efe7c120f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:34:50.259 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:50.244+0000 7efe81487700 1 --2- 172.21.15.42:0/1135748703 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efe7c123aa0 0x7efe7c07af50 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:34:50.259 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:50.244+0000 7efe81487700 1 -- 172.21.15.42:0/1135748703 shutdown_connections 2019-12-05T05:34:50.259 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:50.244+0000 7efe81487700 1 -- 172.21.15.42:0/1135748703 wait complete. 2019-12-05T05:34:50.259 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:50.244+0000 7efe81487700 1 -- 172.21.15.42:0/1135748703 >> 172.21.15.42:0/1135748703 conn(0x7efe7c11c6b0 msgr2=0x7efe7c09f380 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:34:50.260 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2019-12-05T05:34:52.262 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:34:52.381 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:52.360+0000 7f1830607700 1 Processor -- start 2019-12-05T05:34:52.381 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:52.360+0000 7f1830607700 1 -- start start 2019-12-05T05:34:52.381 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:52.364+0000 7f1830607700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f18280a7740 0x7f182808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:34:52.381 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:52.364+0000 7f1830607700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f182807e720 0x7f182807eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:34:52.381 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:52.364+0000 7f1830607700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18280a0710 con 0x7f182809f2e0 2019-12-05T05:34:52.382 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:52.364+0000 7f1830607700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18280a0b00 con 0x7f18280a7740 2019-12-05T05:34:52.382 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:52.364+0000 7f1830607700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f182808e5b0 con 0x7f182807e720 2019-12-05T05:34:52.382 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:52.364+0000 7f182f3a5700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f182809f2e0 0x7f182809f710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:36920/0 (socket says 172.21.15.42:36920) 2019-12-05T05:34:52.382 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:52.364+0000 7f182f3a5700 1 -- 172.21.15.42:0/4150580429 learned_addr learned my addr 172.21.15.42:0/4150580429 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:34:52.384 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:52.364+0000 7f182eba4700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f18280a7740 0x7f182808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:34:52.384 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:52.364+0000 7f182e3a3700 1 --2- 172.21.15.42:0/4150580429 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f182807e720 0x7f182807eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:34:52.384 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:52.364+0000 7f182dba2700 1 -- 172.21.15.42:0/4150580429 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2552288842 0 0) 0x7f18280a0710 con 0x7f182809f2e0 2019-12-05T05:34:52.384 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:52.364+0000 7f182dba2700 1 -- 172.21.15.42:0/4150580429 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18140030b0 con 0x7f182809f2e0 2019-12-05T05:34:52.384 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:52.364+0000 7f182eba4700 1 -- 172.21.15.42:0/4150580429 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f182807e720 msgr2=0x7f182807eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:34:52.384 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:52.364+0000 7f182eba4700 1 --2- 172.21.15.42:0/4150580429 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f182807e720 0x7f182807eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:34:52.385 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:52.364+0000 7f182eba4700 1 -- 172.21.15.42:0/4150580429 >> v1:172.21.15.42:6789/0 conn(0x7f182809f2e0 legacy=0x7f182809f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:34:52.385 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:52.364+0000 7f182eba4700 1 -- 172.21.15.42:0/4150580429 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f182807f0a0 con 0x7f18280a7740 2019-12-05T05:34:52.385 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:52.364+0000 7f182eba4700 1 --2- 172.21.15.42:0/4150580429 >> v2:172.21.15.42:3300/0 conn(0x7f18280a7740 0x7f182808df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f1824008790 tx=0x7f182400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:52.385 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:52.364+0000 7f182eba4700 1 --2- 172.21.15.42:0/4150580429 >> v2:172.21.15.42:3300/0 conn(0x7f18280a7740 0x7f182808df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f1824008790 tx=0x7f182400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:52.385 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:52.364+0000 7f182eba4700 1 --2- 172.21.15.42:0/4150580429 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f18280a7740 0x7f182808df80 secure :-1 s=READY pgs=1137 cs=0 l=1 rx=0x7f1824008790 tx=0x7f182400fc70).ready entity=mon.1 client_cookie=ca867f5dde114b67 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:34:52.385 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:52.364+0000 7f182eba4700 1 --2- 172.21.15.42:0/4150580429 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f18280a7740 0x7f182808df80 secure :-1 s=THROTTLE_DONE pgs=1137 cs=0 l=1 rx=0x7f1824008790 tx=0x7f182400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:52.385 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:52.364+0000 7f182dba2700 1 -- 172.21.15.42:0/4150580429 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f18240112f0 con 0x7f18280a7740 2019-12-05T05:34:52.386 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:52.364+0000 7f182eba4700 1 --2- 172.21.15.42:0/4150580429 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f18280a7740 0x7f182808df80 secure :-1 s=THROTTLE_DONE pgs=1137 cs=0 l=1 rx=0x7f1824008790 tx=0x7f182400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:52.386 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:52.364+0000 7f182eba4700 1 --2- 172.21.15.42:0/4150580429 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f18280a7740 0x7f182808df80 secure :-1 s=THROTTLE_DONE pgs=1137 cs=0 l=1 rx=0x7f1824008790 tx=0x7f182400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:52.386 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:52.364+0000 7f182dba2700 1 -- 172.21.15.42:0/4150580429 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1824008ad0 con 0x7f18280a7740 2019-12-05T05:34:52.386 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:52.364+0000 7f182dba2700 1 -- 172.21.15.42:0/4150580429 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f18240118d0 con 0x7f18280a7740 2019-12-05T05:34:52.386 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:52.364+0000 7f1830607700 1 -- 172.21.15.42:0/4150580429 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f18280a7740 msgr2=0x7f182808df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:34:52.386 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:52.364+0000 7f1830607700 1 --2- 172.21.15.42:0/4150580429 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f18280a7740 0x7f182808df80 secure :-1 s=READY pgs=1137 cs=0 l=1 rx=0x7f1824008790 tx=0x7f182400fc70).stop 2019-12-05T05:34:52.386 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:52.368+0000 7f1830607700 1 -- 172.21.15.42:0/4150580429 shutdown_connections 2019-12-05T05:34:52.387 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:52.368+0000 7f1830607700 1 --2- 172.21.15.42:0/4150580429 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f18280a7740 0x7f182808df80 unknown :-1 s=CLOSED pgs=1137 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:34:52.387 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:52.368+0000 7f1830607700 1 --2- 172.21.15.42:0/4150580429 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f182807e720 0x7f182807eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:34:52.387 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:52.368+0000 7f1830607700 1 -- 172.21.15.42:0/4150580429 shutdown_connections 2019-12-05T05:34:52.387 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:52.368+0000 7f1830607700 1 -- 172.21.15.42:0/4150580429 wait complete. 2019-12-05T05:34:52.387 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:52.368+0000 7f1830607700 1 -- 172.21.15.42:0/4150580429 >> 172.21.15.42:0/4150580429 conn(0x7f18280aabe0 msgr2=0x7f18280a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:34:52.387 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:52.368+0000 7f1830607700 1 Processor -- start 2019-12-05T05:34:52.388 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:52.368+0000 7f1830607700 1 -- start start 2019-12-05T05:34:52.388 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:52.372+0000 7f1830607700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f1828116500 0x7f182811d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:34:52.388 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:52.372+0000 7f1830607700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1828119c00 0x7f1828137690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:34:52.388 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:52.372+0000 7f1830607700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18281173e0 con 0x7f182811df40 2019-12-05T05:34:52.388 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:52.372+0000 7f1830607700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18281187b0 con 0x7f1828116500 2019-12-05T05:34:52.388 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:52.372+0000 7f1830607700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1828118a50 con 0x7f1828119c00 2019-12-05T05:34:52.388 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:52.372+0000 7f182eba4700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f1828116500 0x7f182811d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:34:52.389 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:52.372+0000 7f182f3a5700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1828119c00 0x7f1828137690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:34:52.389 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:52.372+0000 7f182eba4700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f1828116500 0x7f182811d9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.42:52932/0 (socket says 172.21.15.42:52932) 2019-12-05T05:34:52.389 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:52.372+0000 7f182eba4700 1 -- 172.21.15.42:0/2852981305 learned_addr learned my addr 172.21.15.42:0/2852981305 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:34:52.389 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:52.372+0000 7f182eba4700 1 -- 172.21.15.42:0/2852981305 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1828119c00 msgr2=0x7f1828137690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:34:52.389 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:52.372+0000 7f182eba4700 1 --2- 172.21.15.42:0/2852981305 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1828119c00 0x7f1828137690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:34:52.389 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:52.372+0000 7f182eba4700 1 -- 172.21.15.42:0/2852981305 >> v1:172.21.15.40:6789/0 conn(0x7f182811df40 legacy=0x7f182811b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:34:52.389 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:52.372+0000 7f182eba4700 1 -- 172.21.15.42:0/2852981305 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1828118470 con 0x7f1828116500 2019-12-05T05:34:52.390 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:52.372+0000 7f182f3a5700 1 --2- 172.21.15.42:0/2852981305 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1828119c00 0x7f1828137690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:34:52.390 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:52.372+0000 7f182eba4700 1 --2- 172.21.15.42:0/2852981305 >> v2:172.21.15.40:3300/0 conn(0x7f1828116500 0x7f182811d9f0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f1828118970 tx=0x7f18240119c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:52.390 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:52.372+0000 7f182eba4700 1 --2- 172.21.15.42:0/2852981305 >> v2:172.21.15.40:3300/0 conn(0x7f1828116500 0x7f182811d9f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f1828118970 tx=0x7f18240119c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:52.390 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:52.372+0000 7f182eba4700 1 --2- 172.21.15.42:0/2852981305 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1828116500 0x7f182811d9f0 secure :-1 s=READY pgs=898 cs=0 l=1 rx=0x7f1828118970 tx=0x7f18240119c0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:34:52.390 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:52.372+0000 7f182eba4700 1 --2- 172.21.15.42:0/2852981305 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1828116500 0x7f182811d9f0 secure :-1 s=THROTTLE_DONE pgs=898 cs=0 l=1 rx=0x7f1828118970 tx=0x7f18240119c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:52.390 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:52.372+0000 7f182eba4700 1 --2- 172.21.15.42:0/2852981305 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1828116500 0x7f182811d9f0 secure :-1 s=THROTTLE_DONE pgs=898 cs=0 l=1 rx=0x7f1828118970 tx=0x7f18240119c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:52.390 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:52.372+0000 7f181f7fe700 1 -- 172.21.15.42:0/2852981305 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1824007150 con 0x7f1828116500 2019-12-05T05:34:52.391 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:52.372+0000 7f182eba4700 1 --2- 172.21.15.42:0/2852981305 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1828116500 0x7f182811d9f0 secure :-1 s=THROTTLE_DONE pgs=898 cs=0 l=1 rx=0x7f1828118970 tx=0x7f18240119c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:52.391 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:52.372+0000 7f181f7fe700 1 -- 172.21.15.42:0/2852981305 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18240114f0 con 0x7f1828116500 2019-12-05T05:34:52.391 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:52.372+0000 7f1830607700 1 -- 172.21.15.42:0/2852981305 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1828130240 con 0x7f1828116500 2019-12-05T05:34:52.391 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:52.372+0000 7f181f7fe700 1 -- 172.21.15.42:0/2852981305 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1824008cc0 con 0x7f1828116500 2019-12-05T05:34:52.391 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:52.372+0000 7f1830607700 1 -- 172.21.15.42:0/2852981305 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f182811bf20 con 0x7f1828116500 2019-12-05T05:34:52.391 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:52.372+0000 7f182eba4700 1 --2- 172.21.15.42:0/2852981305 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1828116500 0x7f182811d9f0 secure :-1 s=THROTTLE_DONE pgs=898 cs=0 l=1 rx=0x7f1828118970 tx=0x7f18240119c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:52.391 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:52.376+0000 7f181f7fe700 1 -- 172.21.15.42:0/2852981305 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f1824005540 con 0x7f1828116500 2019-12-05T05:34:52.392 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:52.376+0000 7f182eba4700 1 --2- 172.21.15.42:0/2852981305 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1828116500 0x7f182811d9f0 secure :-1 s=THROTTLE_DONE pgs=898 cs=0 l=1 rx=0x7f1828118970 tx=0x7f18240119c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:52.392 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:52.376+0000 7f181f7fe700 1 --2- 172.21.15.42:0/2852981305 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f181003ae30 0x7f181003d340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:34:52.392 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:52.376+0000 7f181f7fe700 1 -- 172.21.15.42:0/2852981305 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 7923+0+0 (secure 0 0 0) 0x7f182404b770 con 0x7f1828116500 2019-12-05T05:34:52.392 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:52.376+0000 7f182e3a3700 1 --2- 172.21.15.42:0/2852981305 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f181003ae30 0x7f181003d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:34:52.392 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:52.376+0000 7f182e3a3700 1 --2- 172.21.15.42:0/2852981305 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f181003ae30 0x7f181003d340 crc :-1 s=READY pgs=491 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:34:52.393 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:52.376+0000 7f1830607700 1 -- 172.21.15.42:0/2852981305 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f182811e850 con 0x7f1828116500 2019-12-05T05:34:52.393 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:52.384+0000 7f182eba4700 1 --2- 172.21.15.42:0/2852981305 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1828116500 0x7f182811d9f0 secure :-1 s=THROTTLE_DONE pgs=898 cs=0 l=1 rx=0x7f1828118970 tx=0x7f18240119c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:52.393 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:52.384+0000 7f181f7fe700 1 -- 172.21.15.42:0/2852981305 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f1824007690 con 0x7f1828116500 2019-12-05T05:34:52.573 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:52.568+0000 7f1830607700 1 -- 172.21.15.42:0/2852981305 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f182805d110 con 0x7f1828116500 2019-12-05T05:34:52.575 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:52.568+0000 7f182eba4700 1 --2- 172.21.15.42:0/2852981305 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1828116500 0x7f182811d9f0 secure :-1 s=THROTTLE_DONE pgs=898 cs=0 l=1 rx=0x7f1828118970 tx=0x7f18240119c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:52.575 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:52.568+0000 7f181f7fe700 1 -- 172.21.15.42:0/2852981305 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 142 v142) v1 ==== 95+0+1775 (secure 0 0 0) 0x7f1824033600 con 0x7f1828116500 2019-12-05T05:34:52.576 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 142 2019-12-05T05:34:52.576 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:34:52.576 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":142,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":142,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:34:39.042388+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7221},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7221":{"gid":7221,"name":"b","rank":0,"incarnation":139,"state":"up:active","state_seq":27,"addr":"172.21.15.40:6833/3789641053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3789641053},{"type":"v1","addr":"172.21.15.40:6833","nonce":3789641053}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:34:52.579 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:52.572+0000 7f1830607700 1 -- 172.21.15.42:0/2852981305 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f181003ae30 msgr2=0x7f181003d340 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:34:52.580 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:52.572+0000 7f1830607700 1 --2- 172.21.15.42:0/2852981305 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f181003ae30 0x7f181003d340 crc :-1 s=READY pgs=491 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:34:52.580 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:52.572+0000 7f1830607700 1 -- 172.21.15.42:0/2852981305 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1828116500 msgr2=0x7f182811d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:34:52.580 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:52.572+0000 7f1830607700 1 --2- 172.21.15.42:0/2852981305 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1828116500 0x7f182811d9f0 secure :-1 s=READY pgs=898 cs=0 l=1 rx=0x7f1828118970 tx=0x7f18240119c0).stop 2019-12-05T05:34:52.580 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:52.572+0000 7f1830607700 1 -- 172.21.15.42:0/2852981305 shutdown_connections 2019-12-05T05:34:52.580 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:52.572+0000 7f1830607700 1 --2- 172.21.15.42:0/2852981305 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f181003ae30 0x7f181003d340 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:34:52.580 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:52.572+0000 7f1830607700 1 --2- 172.21.15.42:0/2852981305 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1828119c00 0x7f1828137690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:34:52.581 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:52.572+0000 7f1830607700 1 --2- 172.21.15.42:0/2852981305 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1828116500 0x7f182811d9f0 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:34:52.581 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:52.572+0000 7f1830607700 1 -- 172.21.15.42:0/2852981305 shutdown_connections 2019-12-05T05:34:52.581 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:52.572+0000 7f1830607700 1 -- 172.21.15.42:0/2852981305 wait complete. 2019-12-05T05:34:52.581 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:52.572+0000 7f1830607700 1 -- 172.21.15.42:0/2852981305 >> 172.21.15.42:0/2852981305 conn(0x7f182807e920 msgr2=0x7f182811c4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:34:52.589 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2019-12-05T05:34:54.591 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:34:54.690 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:54.680+0000 7f4ef7736700 1 Processor -- start 2019-12-05T05:34:54.691 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:54.680+0000 7f4ef7736700 1 -- start start 2019-12-05T05:34:54.691 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:54.680+0000 7f4ef7736700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f4ef00a7740 0x7f4ef008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:34:54.691 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:54.680+0000 7f4ef7736700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f4ef007e720 0x7f4ef007eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:34:54.691 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:54.680+0000 7f4ef7736700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4ef009f2e0 0x7f4ef009f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:34:54.691 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:54.680+0000 7f4ef7736700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ef00a0b00 con 0x7f4ef007e720 2019-12-05T05:34:54.692 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:54.680+0000 7f4ef7736700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ef008e5b0 con 0x7f4ef00a7740 2019-12-05T05:34:54.692 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:54.680+0000 7f4ef7736700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ef008e850 con 0x7f4ef009f2e0 2019-12-05T05:34:54.692 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:54.680+0000 7f4ef5cd3700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f4ef00a7740 0x7f4ef008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:34:54.693 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:54.680+0000 7f4ef5cd3700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f4ef00a7740 0x7f4ef008df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:36902/0 (socket says 172.21.15.42:36902) 2019-12-05T05:34:54.693 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:54.680+0000 7f4ef5cd3700 1 -- 172.21.15.42:0/2263944828 learned_addr learned my addr 172.21.15.42:0/2263944828 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:34:54.693 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:54.680+0000 7f4ef54d2700 1 --2- 172.21.15.42:0/2263944828 >> v2:172.21.15.40:3300/0 conn(0x7f4ef007e720 0x7f4ef007eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:34:54.693 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:54.680+0000 7f4ef64d4700 1 --2- 172.21.15.42:0/2263944828 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4ef009f2e0 0x7f4ef009f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:34:54.693 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:54.684+0000 7f4ef5cd3700 1 -- 172.21.15.42:0/2263944828 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4ef009f2e0 msgr2=0x7f4ef009f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:34:54.693 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:54.684+0000 7f4ef5cd3700 1 --2- 172.21.15.42:0/2263944828 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4ef009f2e0 0x7f4ef009f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:34:54.694 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:54.684+0000 7f4ef5cd3700 1 -- 172.21.15.42:0/2263944828 >> v2:172.21.15.40:3300/0 conn(0x7f4ef007e720 msgr2=0x7f4ef007eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:34:54.694 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:54.684+0000 7f4ef5cd3700 1 --2- 172.21.15.42:0/2263944828 >> v2:172.21.15.40:3300/0 conn(0x7f4ef007e720 0x7f4ef007eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:34:54.694 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:54.684+0000 7f4ef5cd3700 1 -- 172.21.15.42:0/2263944828 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ef007f0a0 con 0x7f4ef00a7740 2019-12-05T05:34:54.694 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:54.684+0000 7f4ef5cd3700 1 --2- 172.21.15.42:0/2263944828 >> v2:172.21.15.42:3300/0 conn(0x7f4ef00a7740 0x7f4ef008df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f4ef00a0f50 tx=0x7f4eec00fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:54.694 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:54.684+0000 7f4ef5cd3700 1 --2- 172.21.15.42:0/2263944828 >> v2:172.21.15.42:3300/0 conn(0x7f4ef00a7740 0x7f4ef008df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f4ef00a0f50 tx=0x7f4eec00fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:54.694 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:54.684+0000 7f4ef5cd3700 1 --2- 172.21.15.42:0/2263944828 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4ef00a7740 0x7f4ef008df80 secure :-1 s=READY pgs=1138 cs=0 l=1 rx=0x7f4ef00a0f50 tx=0x7f4eec00fdb0).ready entity=mon.1 client_cookie=5e69cdb0eed86f9b server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:34:54.694 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:54.684+0000 7f4ef5cd3700 1 --2- 172.21.15.42:0/2263944828 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4ef00a7740 0x7f4ef008df80 secure :-1 s=THROTTLE_DONE pgs=1138 cs=0 l=1 rx=0x7f4ef00a0f50 tx=0x7f4eec00fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:54.695 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:54.684+0000 7f4ef4cd1700 1 -- 172.21.15.42:0/2263944828 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4eec008c90 con 0x7f4ef00a7740 2019-12-05T05:34:54.695 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:54.684+0000 7f4ef5cd3700 1 --2- 172.21.15.42:0/2263944828 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4ef00a7740 0x7f4ef008df80 secure :-1 s=THROTTLE_DONE pgs=1138 cs=0 l=1 rx=0x7f4ef00a0f50 tx=0x7f4eec00fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:54.695 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:54.684+0000 7f4ef5cd3700 1 --2- 172.21.15.42:0/2263944828 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4ef00a7740 0x7f4ef008df80 secure :-1 s=THROTTLE_DONE pgs=1138 cs=0 l=1 rx=0x7f4ef00a0f50 tx=0x7f4eec00fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:54.696 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:54.684+0000 7f4ef4cd1700 1 -- 172.21.15.42:0/2263944828 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4eec0088b0 con 0x7f4ef00a7740 2019-12-05T05:34:54.696 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:54.684+0000 7f4ef4cd1700 1 -- 172.21.15.42:0/2263944828 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4eec011b50 con 0x7f4ef00a7740 2019-12-05T05:34:54.696 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:54.684+0000 7f4ef7736700 1 -- 172.21.15.42:0/2263944828 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4ef00a7740 msgr2=0x7f4ef008df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:34:54.696 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:54.684+0000 7f4ef7736700 1 --2- 172.21.15.42:0/2263944828 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4ef00a7740 0x7f4ef008df80 secure :-1 s=READY pgs=1138 cs=0 l=1 rx=0x7f4ef00a0f50 tx=0x7f4eec00fdb0).stop 2019-12-05T05:34:54.696 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:54.684+0000 7f4ef7736700 1 -- 172.21.15.42:0/2263944828 shutdown_connections 2019-12-05T05:34:54.696 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:54.684+0000 7f4ef7736700 1 --2- 172.21.15.42:0/2263944828 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4ef009f2e0 0x7f4ef009f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:34:54.696 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:54.684+0000 7f4ef7736700 1 --2- 172.21.15.42:0/2263944828 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4ef00a7740 0x7f4ef008df80 unknown :-1 s=CLOSED pgs=1138 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:34:54.697 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:54.684+0000 7f4ef7736700 1 --2- 172.21.15.42:0/2263944828 >> v2:172.21.15.40:3300/0 conn(0x7f4ef007e720 0x7f4ef007eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:34:54.697 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:54.684+0000 7f4ef7736700 1 -- 172.21.15.42:0/2263944828 shutdown_connections 2019-12-05T05:34:54.697 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:54.684+0000 7f4ef7736700 1 -- 172.21.15.42:0/2263944828 wait complete. 2019-12-05T05:34:54.697 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:54.684+0000 7f4ef7736700 1 -- 172.21.15.42:0/2263944828 >> 172.21.15.42:0/2263944828 conn(0x7f4ef00aabe0 msgr2=0x7f4ef00a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:34:54.705 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:54.688+0000 7f4ef7736700 1 Processor -- start 2019-12-05T05:34:54.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:54.688+0000 7f4ef7736700 1 -- start start 2019-12-05T05:34:54.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:54.688+0000 7f4ef7736700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f4ef0120fc0 0x7f4ef011b590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:34:54.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:54.688+0000 7f4ef7736700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f4ef0130cf0 0x7f4ef011df50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:34:54.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:54.688+0000 7f4ef7736700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ef011e4a0 con 0x7f4ef01308c0 2019-12-05T05:34:54.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:54.688+0000 7f4ef7736700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ef012cdc0 con 0x7f4ef0130cf0 2019-12-05T05:34:54.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:54.688+0000 7f4ef7736700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ef011bea0 con 0x7f4ef0120fc0 2019-12-05T05:34:54.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:54.688+0000 7f4ef5cd3700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f4ef0120fc0 0x7f4ef011b590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:34:54.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:54.688+0000 7f4ef54d2700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f4ef01308c0 0x7f4ef011d750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:36948/0 (socket says 172.21.15.42:36948) 2019-12-05T05:34:54.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:54.688+0000 7f4ef54d2700 1 -- 172.21.15.42:0/3562862529 learned_addr learned my addr 172.21.15.42:0/3562862529 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:34:54.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:54.688+0000 7f4ef64d4700 1 --2- 172.21.15.42:0/3562862529 >> v2:172.21.15.40:3300/0 conn(0x7f4ef0130cf0 0x7f4ef011df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:34:54.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:54.688+0000 7f4ef5cd3700 1 -- 172.21.15.42:0/3562862529 >> v2:172.21.15.40:3300/0 conn(0x7f4ef0130cf0 msgr2=0x7f4ef011df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:34:54.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:54.688+0000 7f4ef5cd3700 1 --2- 172.21.15.42:0/3562862529 >> v2:172.21.15.40:3300/0 conn(0x7f4ef0130cf0 0x7f4ef011df50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:34:54.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:54.688+0000 7f4ef5cd3700 1 -- 172.21.15.42:0/3562862529 >> v1:172.21.15.42:6789/0 conn(0x7f4ef01308c0 legacy=0x7f4ef011d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:34:54.708 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:54.688+0000 7f4ef5cd3700 1 -- 172.21.15.42:0/3562862529 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ef0131120 con 0x7f4ef0120fc0 2019-12-05T05:34:54.708 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:54.688+0000 7f4ee67fc700 1 -- 172.21.15.42:0/3562862529 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1602972330 0 0) 0x7f4ef011e4a0 con 0x7f4ef01308c0 2019-12-05T05:34:54.708 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:54.692+0000 7f4ef5cd3700 1 --2- 172.21.15.42:0/3562862529 >> v2:172.21.15.42:3300/0 conn(0x7f4ef0120fc0 0x7f4ef011b590 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f4ef0121d40 tx=0x7f4eec0075e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:54.708 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:54.692+0000 7f4ef5cd3700 1 --2- 172.21.15.42:0/3562862529 >> v2:172.21.15.42:3300/0 conn(0x7f4ef0120fc0 0x7f4ef011b590 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f4ef0121d40 tx=0x7f4eec0075e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:54.708 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:54.692+0000 7f4ef5cd3700 1 --2- 172.21.15.42:0/3562862529 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4ef0120fc0 0x7f4ef011b590 secure :-1 s=READY pgs=1140 cs=0 l=1 rx=0x7f4ef0121d40 tx=0x7f4eec0075e0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:34:54.708 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:54.692+0000 7f4ef5cd3700 1 --2- 172.21.15.42:0/3562862529 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4ef0120fc0 0x7f4ef011b590 secure :-1 s=THROTTLE_DONE pgs=1140 cs=0 l=1 rx=0x7f4ef0121d40 tx=0x7f4eec0075e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:54.708 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:54.692+0000 7f4ee67fc700 1 -- 172.21.15.42:0/3562862529 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4eec00f280 con 0x7f4ef0120fc0 2019-12-05T05:34:54.709 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:54.692+0000 7f4ef5cd3700 1 --2- 172.21.15.42:0/3562862529 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4ef0120fc0 0x7f4ef011b590 secure :-1 s=THROTTLE_DONE pgs=1140 cs=0 l=1 rx=0x7f4ef0121d40 tx=0x7f4eec0075e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:54.709 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:54.692+0000 7f4ef5cd3700 1 --2- 172.21.15.42:0/3562862529 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4ef0120fc0 0x7f4ef011b590 secure :-1 s=THROTTLE_DONE pgs=1140 cs=0 l=1 rx=0x7f4ef0121d40 tx=0x7f4eec0075e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:54.709 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:54.692+0000 7f4ee67fc700 1 -- 172.21.15.42:0/3562862529 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4eec011770 con 0x7f4ef0120fc0 2019-12-05T05:34:54.709 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:54.692+0000 7f4ef7736700 1 -- 172.21.15.42:0/3562862529 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ef011bae0 con 0x7f4ef0120fc0 2019-12-05T05:34:54.709 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:54.692+0000 7f4ee67fc700 1 -- 172.21.15.42:0/3562862529 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4eec05a660 con 0x7f4ef0120fc0 2019-12-05T05:34:54.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:54.692+0000 7f4ef7736700 1 -- 172.21.15.42:0/3562862529 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ef0123ea0 con 0x7f4ef0120fc0 2019-12-05T05:34:54.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:54.692+0000 7f4ef5cd3700 1 --2- 172.21.15.42:0/3562862529 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4ef0120fc0 0x7f4ef011b590 secure :-1 s=THROTTLE_DONE pgs=1140 cs=0 l=1 rx=0x7f4ef0121d40 tx=0x7f4eec0075e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:54.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:54.692+0000 7f4ee67fc700 1 -- 172.21.15.42:0/3562862529 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f4eec007180 con 0x7f4ef0120fc0 2019-12-05T05:34:54.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:54.692+0000 7f4ef5cd3700 1 --2- 172.21.15.42:0/3562862529 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4ef0120fc0 0x7f4ef011b590 secure :-1 s=THROTTLE_DONE pgs=1140 cs=0 l=1 rx=0x7f4ef0121d40 tx=0x7f4eec0075e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:54.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:54.692+0000 7f4ee67fc700 1 --2- 172.21.15.42:0/3562862529 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4ed803ace0 0x7f4ed803d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:34:54.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:54.692+0000 7f4ee67fc700 1 -- 172.21.15.42:0/3562862529 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 7923+0+0 (secure 0 0 0) 0x7f4eec0912d0 con 0x7f4ef0120fc0 2019-12-05T05:34:54.711 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:54.692+0000 7f4ef54d2700 1 --2- 172.21.15.42:0/3562862529 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4ed803ace0 0x7f4ed803d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:34:54.711 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:54.696+0000 7f4ef54d2700 1 --2- 172.21.15.42:0/3562862529 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4ed803ace0 0x7f4ed803d1f0 crc :-1 s=READY pgs=492 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:34:54.711 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:54.696+0000 7f4ef7736700 1 -- 172.21.15.42:0/3562862529 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ef005d110 con 0x7f4ef0120fc0 2019-12-05T05:34:54.711 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:54.696+0000 7f4ef5cd3700 1 -- 172.21.15.42:0/3562862529 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4ef0120fc0 msgr2=0x7f4ef011b590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2019-12-05T05:34:54.711 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:54.696+0000 7f4ef5cd3700 1 -- 172.21.15.42:0/3562862529 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4ef0120fc0 msgr2=0x7f4ef011b590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2019-12-05T05:34:54.711 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:54.696+0000 7f4ef5cd3700 1 --2- 172.21.15.42:0/3562862529 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4ef0120fc0 0x7f4ef011b590 secure :-1 s=READY pgs=1140 cs=0 l=1 rx=0x7f4ef0121d40 tx=0x7f4eec0075e0).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-12-05T05:34:54.712 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:54.696+0000 7f4ef5cd3700 1 --2- 172.21.15.42:0/3562862529 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4ef0120fc0 0x7f4ef011b590 secure :-1 s=READY pgs=1140 cs=0 l=1 rx=0x7f4ef0121d40 tx=0x7f4eec0075e0).stop 2019-12-05T05:34:54.712 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:54.696+0000 7f4ee67fc700 1 -- 172.21.15.42:0/3562862529 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4ef0120fc0 msgr2=0x7f4ef011b590 unknown :-1 s=STATE_CLOSED l=1).mark_down 2019-12-05T05:34:54.712 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:54.696+0000 7f4ee67fc700 1 --2- 172.21.15.42:0/3562862529 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4ef0120fc0 0x7f4ef011b590 unknown :-1 s=CLOSED pgs=1140 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:34:54.712 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:54.696+0000 7f4ee67fc700 1 --2- 172.21.15.42:0/3562862529 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4ed80426e0 0x7f4ed8044b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:34:54.712 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:54.696+0000 7f4ee67fc700 1 --2- 172.21.15.42:0/3562862529 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4ed8045230 0x7f4ed8047740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:34:54.712 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:54.696+0000 7f4ee67fc700 1 --2- 172.21.15.42:0/3562862529 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ed8047f20 0x7f4ed804a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:34:54.713 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:54.696+0000 7f4ee67fc700 1 -- 172.21.15.42:0/3562862529 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ed804ac50 con 0x7f4ed8047f20 2019-12-05T05:34:54.713 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:54.696+0000 7f4ee67fc700 1 -- 172.21.15.42:0/3562862529 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ed804ae40 con 0x7f4ed80426e0 2019-12-05T05:34:54.713 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:54.696+0000 7f4ee67fc700 1 -- 172.21.15.42:0/3562862529 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ed804b080 con 0x7f4ed8045230 2019-12-05T05:34:54.716 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:54.696+0000 7f4ef5cd3700 1 --2- 172.21.15.42:0/3562862529 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4ed80426e0 0x7f4ed8044b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:34:54.716 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:54.696+0000 7f4ef64d4700 1 --2- 172.21.15.42:0/3562862529 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4ed8045230 0x7f4ed8047740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:34:54.716 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:54.696+0000 7f4ef5cd3700 1 --2- 172.21.15.42:0/3562862529 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ed8047f20 0x7f4ed804a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:34:54.716 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:54.696+0000 7f4ef5cd3700 1 -- 172.21.15.42:0/3562862529 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4ed8045230 msgr2=0x7f4ed8047740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:34:54.716 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:54.696+0000 7f4ef5cd3700 1 --2- 172.21.15.42:0/3562862529 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4ed8045230 0x7f4ed8047740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:34:54.716 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:54.696+0000 7f4ef5cd3700 1 -- 172.21.15.42:0/3562862529 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ed8047f20 msgr2=0x7f4ed804a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:34:54.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:54.696+0000 7f4ef5cd3700 1 --2- 172.21.15.42:0/3562862529 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ed8047f20 0x7f4ed804a430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:34:54.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:54.696+0000 7f4ef64d4700 1 -- 172.21.15.42:0/3562862529 reap_dead start 2019-12-05T05:34:54.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:54.696+0000 7f4ef5cd3700 1 -- 172.21.15.42:0/3562862529 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f4ed804b380 con 0x7f4ed80426e0 2019-12-05T05:34:54.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:54.696+0000 7f4ef5cd3700 1 -- 172.21.15.42:0/3562862529 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4eec058130 con 0x7f4ed80426e0 2019-12-05T05:34:54.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:54.696+0000 7f4ef5cd3700 1 --2- 172.21.15.42:0/3562862529 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4ed80426e0 0x7f4ed8044b20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f4eec008f60 tx=0x7f4eec058330).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:54.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:54.696+0000 7f4ef5cd3700 1 --2- 172.21.15.42:0/3562862529 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4ed80426e0 0x7f4ed8044b20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f4eec008f60 tx=0x7f4eec058330).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:54.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:54.696+0000 7f4ef5cd3700 1 --2- 172.21.15.42:0/3562862529 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4ed80426e0 0x7f4ed8044b20 secure :-1 s=READY pgs=1141 cs=0 l=1 rx=0x7f4eec008f60 tx=0x7f4eec058330).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:34:54.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:54.704+0000 7f4ef5cd3700 1 --2- 172.21.15.42:0/3562862529 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4ed80426e0 0x7f4ed8044b20 secure :-1 s=THROTTLE_DONE pgs=1141 cs=0 l=1 rx=0x7f4eec008f60 tx=0x7f4eec058330).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:54.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:54.704+0000 7f4ee67fc700 1 -- 172.21.15.42:0/3562862529 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4eec058130 con 0x7f4ed80426e0 2019-12-05T05:34:54.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:54.704+0000 7f4ef5cd3700 1 --2- 172.21.15.42:0/3562862529 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4ed80426e0 0x7f4ed8044b20 secure :-1 s=THROTTLE_DONE pgs=1141 cs=0 l=1 rx=0x7f4eec008f60 tx=0x7f4eec058330).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:54.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:54.704+0000 7f4ee67fc700 1 -- 172.21.15.42:0/3562862529 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4eec00f480 con 0x7f4ed80426e0 2019-12-05T05:34:54.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:54.704+0000 7f4ef5cd3700 1 --2- 172.21.15.42:0/3562862529 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4ed80426e0 0x7f4ed8044b20 secure :-1 s=THROTTLE_DONE pgs=1141 cs=0 l=1 rx=0x7f4eec008f60 tx=0x7f4eec058330).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:54.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:54.704+0000 7f4ee67fc700 1 -- 172.21.15.42:0/3562862529 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f4eec078160 con 0x7f4ed80426e0 2019-12-05T05:34:54.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:54.712+0000 7f4ef5cd3700 1 --2- 172.21.15.42:0/3562862529 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4ed80426e0 0x7f4ed8044b20 secure :-1 s=THROTTLE_DONE pgs=1141 cs=0 l=1 rx=0x7f4eec008f60 tx=0x7f4eec058330).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:54.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:54.712+0000 7f4ee67fc700 1 -- 172.21.15.42:0/3562862529 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f4eec07a500 con 0x7f4ed80426e0 2019-12-05T05:34:54.899 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:54.892+0000 7f4ef7736700 1 -- 172.21.15.42:0/3562862529 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4ef005b8c0 con 0x7f4ed80426e0 2019-12-05T05:34:54.900 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:54.896+0000 7f4ef5cd3700 1 --2- 172.21.15.42:0/3562862529 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4ed80426e0 0x7f4ed8044b20 secure :-1 s=THROTTLE_DONE pgs=1141 cs=0 l=1 rx=0x7f4eec008f60 tx=0x7f4eec058330).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:34:54.900 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:54.896+0000 7f4ee67fc700 1 -- 172.21.15.42:0/3562862529 <== mon.1 v2:172.21.15.42:3300/0 5 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 143 v143) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f4eec078470 con 0x7f4ed80426e0 2019-12-05T05:34:54.902 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 143 2019-12-05T05:34:54.903 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:34:54.903 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":143,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/3838877789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3838877789},{"type":"v1","addr":"172.21.15.42:6835","nonce":3838877789}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":143}],"filesystems":[{"mdsmap":{"epoch":142,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:34:39.042388+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7221},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7221":{"gid":7221,"name":"b","rank":0,"incarnation":139,"state":"up:active","state_seq":27,"addr":"172.21.15.40:6833/3789641053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3789641053},{"type":"v1","addr":"172.21.15.40:6833","nonce":3789641053}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:34:54.907 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:54.900+0000 7f4ef7736700 1 -- 172.21.15.42:0/3562862529 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4ed803ace0 msgr2=0x7f4ed803d1f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:34:54.907 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:54.900+0000 7f4ef7736700 1 --2- 172.21.15.42:0/3562862529 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4ed803ace0 0x7f4ed803d1f0 crc :-1 s=READY pgs=492 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:34:54.907 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:54.900+0000 7f4ef7736700 1 -- 172.21.15.42:0/3562862529 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4ed80426e0 msgr2=0x7f4ed8044b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:34:54.907 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:54.900+0000 7f4ef7736700 1 --2- 172.21.15.42:0/3562862529 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4ed80426e0 0x7f4ed8044b20 secure :-1 s=READY pgs=1141 cs=0 l=1 rx=0x7f4eec008f60 tx=0x7f4eec058330).stop 2019-12-05T05:34:54.907 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:54.900+0000 7f4ef7736700 1 -- 172.21.15.42:0/3562862529 shutdown_connections 2019-12-05T05:34:54.908 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:54.900+0000 7f4ef7736700 1 --2- 172.21.15.42:0/3562862529 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4ef0120fc0 0x7f4ef011b590 unknown :-1 s=CLOSED pgs=1140 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:34:54.908 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:54.900+0000 7f4ef7736700 1 --2- 172.21.15.42:0/3562862529 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4ed803ace0 0x7f4ed803d1f0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:34:54.908 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:54.900+0000 7f4ef7736700 1 --2- 172.21.15.42:0/3562862529 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4ed80426e0 0x7f4ed8044b20 unknown :-1 s=CLOSED pgs=1141 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:34:54.908 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:54.900+0000 7f4ef7736700 1 -- 172.21.15.42:0/3562862529 shutdown_connections 2019-12-05T05:34:54.908 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:54.900+0000 7f4ef7736700 1 -- 172.21.15.42:0/3562862529 wait complete. 2019-12-05T05:34:54.909 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:34:54.900+0000 7f4ef7736700 1 -- 172.21.15.42:0/3562862529 >> 172.21.15.42:0/3562862529 conn(0x7f4ef00aabe0 msgr2=0x7f4ef00a9770 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:34:54.919 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a reported in up:standby state 2019-12-05T05:34:54.920 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 116 secs before thrashing 2019-12-05T05:35:09.813 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:35:09.820 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:35:39.861 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:35:39.865 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:36:09.912 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:36:09.917 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:36:39.956 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:36:39.959 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:36:50.924 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:36:51.033 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.015+0000 7f016e19d700 1 Processor -- start 2019-12-05T05:36:51.033 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.015+0000 7f016e19d700 1 -- start start 2019-12-05T05:36:51.033 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.019+0000 7f016e19d700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f016808df80 0x7f016809f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:36:51.033 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.019+0000 7f016e19d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f016809f830 0x7f016814efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:36:51.044 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.019+0000 7f016e19d700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f016809fc60 con 0x7f01680a7740 2019-12-05T05:36:51.044 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.019+0000 7f016e19d700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01680a0b00 con 0x7f016808df80 2019-12-05T05:36:51.044 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.019+0000 7f016e19d700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f016808e5b0 con 0x7f016809f830 2019-12-05T05:36:51.044 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.019+0000 7f016cf3b700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f016809f830 0x7f016814efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:36:51.044 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.019+0000 7f01677fe700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f016808df80 0x7f016809f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:36:51.044 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.019+0000 7f01677fe700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f016808df80 0x7f016809f2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.42:53322/0 (socket says 172.21.15.42:53322) 2019-12-05T05:36:51.045 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.019+0000 7f0167fff700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f01680a7740 0x7f016807e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:46750/0 (socket says 172.21.15.42:46750) 2019-12-05T05:36:51.045 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.019+0000 7f01677fe700 1 -- 172.21.15.42:0/874052222 learned_addr learned my addr 172.21.15.42:0/874052222 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:36:51.045 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.019+0000 7f0166ffd700 1 -- 172.21.15.42:0/874052222 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2135992792 0 0) 0x7f016809fc60 con 0x7f01680a7740 2019-12-05T05:36:51.045 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.019+0000 7f016cf3b700 1 -- 172.21.15.42:0/874052222 >> v2:172.21.15.40:3300/0 conn(0x7f016808df80 msgr2=0x7f016809f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:36:51.045 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.019+0000 7f016cf3b700 1 --2- 172.21.15.42:0/874052222 >> v2:172.21.15.40:3300/0 conn(0x7f016808df80 0x7f016809f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:36:51.045 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.019+0000 7f016cf3b700 1 -- 172.21.15.42:0/874052222 >> v1:172.21.15.40:6789/0 conn(0x7f01680a7740 legacy=0x7f016807e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:36:51.046 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.019+0000 7f016cf3b700 1 -- 172.21.15.42:0/874052222 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01680a0710 con 0x7f016809f830 2019-12-05T05:36:51.046 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.019+0000 7f016cf3b700 1 --2- 172.21.15.42:0/874052222 >> v2:172.21.15.42:3300/0 conn(0x7f016809f830 0x7f016814efe0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f01580032b0 tx=0x7f015800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:36:51.046 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.019+0000 7f016cf3b700 1 --2- 172.21.15.42:0/874052222 >> v2:172.21.15.42:3300/0 conn(0x7f016809f830 0x7f016814efe0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f01580032b0 tx=0x7f015800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:36:51.047 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.019+0000 7f016cf3b700 1 --2- 172.21.15.42:0/874052222 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f016809f830 0x7f016814efe0 secure :-1 s=READY pgs=1144 cs=0 l=1 rx=0x7f01580032b0 tx=0x7f015800fff0).ready entity=mon.1 client_cookie=4443efb77f4f7e2a server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:36:51.047 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.019+0000 7f016cf3b700 1 --2- 172.21.15.42:0/874052222 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f016809f830 0x7f016814efe0 secure :-1 s=THROTTLE_DONE pgs=1144 cs=0 l=1 rx=0x7f01580032b0 tx=0x7f015800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:36:51.047 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.019+0000 7f0166ffd700 1 -- 172.21.15.42:0/874052222 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f01580108c0 con 0x7f016809f830 2019-12-05T05:36:51.047 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.019+0000 7f016cf3b700 1 --2- 172.21.15.42:0/874052222 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f016809f830 0x7f016814efe0 secure :-1 s=THROTTLE_DONE pgs=1144 cs=0 l=1 rx=0x7f01580032b0 tx=0x7f015800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:36:51.047 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.019+0000 7f016cf3b700 1 --2- 172.21.15.42:0/874052222 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f016809f830 0x7f016814efe0 secure :-1 s=THROTTLE_DONE pgs=1144 cs=0 l=1 rx=0x7f01580032b0 tx=0x7f015800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:36:51.047 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.019+0000 7f0166ffd700 1 -- 172.21.15.42:0/874052222 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0158008b50 con 0x7f016809f830 2019-12-05T05:36:51.047 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.019+0000 7f0166ffd700 1 -- 172.21.15.42:0/874052222 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f01580110c0 con 0x7f016809f830 2019-12-05T05:36:51.048 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.019+0000 7f016e19d700 1 -- 172.21.15.42:0/874052222 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f016809f830 msgr2=0x7f016814efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:36:51.048 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.019+0000 7f016e19d700 1 --2- 172.21.15.42:0/874052222 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f016809f830 0x7f016814efe0 secure :-1 s=READY pgs=1144 cs=0 l=1 rx=0x7f01580032b0 tx=0x7f015800fff0).stop 2019-12-05T05:36:51.048 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.019+0000 7f016e19d700 1 -- 172.21.15.42:0/874052222 shutdown_connections 2019-12-05T05:36:51.048 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.019+0000 7f016e19d700 1 --2- 172.21.15.42:0/874052222 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f016809f830 0x7f016814efe0 unknown :-1 s=CLOSED pgs=1144 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:36:51.048 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.019+0000 7f016e19d700 1 --2- 172.21.15.42:0/874052222 >> v2:172.21.15.40:3300/0 conn(0x7f016808df80 0x7f016809f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:36:51.050 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.019+0000 7f016e19d700 1 -- 172.21.15.42:0/874052222 shutdown_connections 2019-12-05T05:36:51.050 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.019+0000 7f016e19d700 1 -- 172.21.15.42:0/874052222 wait complete. 2019-12-05T05:36:51.050 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.019+0000 7f016e19d700 1 -- 172.21.15.42:0/874052222 >> 172.21.15.42:0/874052222 conn(0x7f01680aabe0 msgr2=0x7f01680a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:36:51.050 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.023+0000 7f016e19d700 1 Processor -- start 2019-12-05T05:36:51.050 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.023+0000 7f016e19d700 1 -- start start 2019-12-05T05:36:51.050 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.023+0000 7f016e19d700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0168116500 0x7f01680a55e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:36:51.051 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.023+0000 7f016e19d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f016807ca30 0x7f016807ce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:36:51.051 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.023+0000 7f016e19d700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0168134360 con 0x7f0168133670 2019-12-05T05:36:51.051 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.023+0000 7f016e19d700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01681308b0 con 0x7f016807ca30 2019-12-05T05:36:51.051 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.023+0000 7f016e19d700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0168133b80 con 0x7f0168116500 2019-12-05T05:36:51.051 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.023+0000 7f016cf3b700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f016807ca30 0x7f016807ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:36:51.051 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.023+0000 7f0167fff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f0168133670 0x7f01681300b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:37316/0 (socket says 172.21.15.42:37316) 2019-12-05T05:36:51.051 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.023+0000 7f0167fff700 1 -- 172.21.15.42:0/662249747 learned_addr learned my addr 172.21.15.42:0/662249747 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:36:51.052 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.023+0000 7f01677fe700 1 --2- 172.21.15.42:0/662249747 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0168116500 0x7f01680a55e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:36:51.052 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.023+0000 7f0167fff700 1 -- 172.21.15.42:0/662249747 >> v1:172.21.15.42:6789/0 conn(0x7f0168133670 legacy=0x7f01681300b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2019-12-05T05:36:51.052 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.023+0000 7f0167fff700 1 -- 172.21.15.42:0/662249747 >> v1:172.21.15.42:6789/0 conn(0x7f0168133670 legacy=0x7f01681300b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2019-12-05T05:36:51.052 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.023+0000 7f0167fff700 1 --1- 172.21.15.42:0/662249747 >> v1:172.21.15.42:6789/0 conn(0x7f0168133670 0x7f01681300b0 :-1 s=OPENED pgs=1145 cs=1 l=1).handle_message read tag failed 2019-12-05T05:36:51.053 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.023+0000 7f0167fff700 1 --1- 172.21.15.42:0/662249747 >> v1:172.21.15.42:6789/0 conn(0x7f0168133670 0x7f01681300b0 :-1 s=OPENED pgs=1145 cs=1 l=1).fault on lossy channel, failing 2019-12-05T05:36:51.053 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.023+0000 7f016cf3b700 1 -- 172.21.15.42:0/662249747 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0168116500 msgr2=0x7f01680a55e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:36:51.053 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.023+0000 7f016cf3b700 1 --2- 172.21.15.42:0/662249747 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0168116500 0x7f01680a55e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:36:51.053 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.023+0000 7f016cf3b700 1 -- 172.21.15.42:0/662249747 >> v1:172.21.15.42:6789/0 conn(0x7f0168133670 legacy=0x7f01681300b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2019-12-05T05:36:51.053 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.023+0000 7f016cf3b700 1 -- 172.21.15.42:0/662249747 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f016813bad0 con 0x7f016807ca30 2019-12-05T05:36:51.053 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.027+0000 7f016cf3b700 1 --2- 172.21.15.42:0/662249747 >> v2:172.21.15.42:3300/0 conn(0x7f016807ca30 0x7f016807ce60 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f0168133d40 tx=0x7f0158010cb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:36:51.053 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.027+0000 7f016cf3b700 1 --2- 172.21.15.42:0/662249747 >> v2:172.21.15.42:3300/0 conn(0x7f016807ca30 0x7f016807ce60 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f0168133d40 tx=0x7f0158010cb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:36:51.054 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.027+0000 7f016cf3b700 1 --2- 172.21.15.42:0/662249747 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f016807ca30 0x7f016807ce60 secure :-1 s=READY pgs=1146 cs=0 l=1 rx=0x7f0168133d40 tx=0x7f0158010cb0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:36:51.054 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.027+0000 7f016cf3b700 1 --2- 172.21.15.42:0/662249747 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f016807ca30 0x7f016807ce60 secure :-1 s=THROTTLE_DONE pgs=1146 cs=0 l=1 rx=0x7f0168133d40 tx=0x7f0158010cb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:36:51.054 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.027+0000 7f016cf3b700 1 --2- 172.21.15.42:0/662249747 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f016807ca30 0x7f016807ce60 secure :-1 s=THROTTLE_DONE pgs=1146 cs=0 l=1 rx=0x7f0168133d40 tx=0x7f0158010cb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:36:51.054 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.027+0000 7f0164ff9700 1 -- 172.21.15.42:0/662249747 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0158005ab0 con 0x7f016807ca30 2019-12-05T05:36:51.054 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.027+0000 7f016cf3b700 1 --2- 172.21.15.42:0/662249747 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f016807ca30 0x7f016807ce60 secure :-1 s=THROTTLE_DONE pgs=1146 cs=0 l=1 rx=0x7f0168133d40 tx=0x7f0158010cb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:36:51.054 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.027+0000 7f0164ff9700 1 -- 172.21.15.42:0/662249747 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0158010ac0 con 0x7f016807ca30 2019-12-05T05:36:51.054 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.027+0000 7f016e19d700 1 -- 172.21.15.42:0/662249747 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f016813aac0 con 0x7f016807ca30 2019-12-05T05:36:51.055 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.027+0000 7f0164ff9700 1 -- 172.21.15.42:0/662249747 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f015800f680 con 0x7f016807ca30 2019-12-05T05:36:51.055 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.027+0000 7f016e19d700 1 -- 172.21.15.42:0/662249747 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f016813ae70 con 0x7f016807ca30 2019-12-05T05:36:51.055 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.027+0000 7f016cf3b700 1 --2- 172.21.15.42:0/662249747 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f016807ca30 0x7f016807ce60 secure :-1 s=THROTTLE_DONE pgs=1146 cs=0 l=1 rx=0x7f0168133d40 tx=0x7f0158010cb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:36:51.057 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.027+0000 7f0164ff9700 1 -- 172.21.15.42:0/662249747 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f015800f880 con 0x7f016807ca30 2019-12-05T05:36:51.057 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.027+0000 7f016cf3b700 1 --2- 172.21.15.42:0/662249747 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f016807ca30 0x7f016807ce60 secure :-1 s=THROTTLE_DONE pgs=1146 cs=0 l=1 rx=0x7f0168133d40 tx=0x7f0158010cb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:36:51.057 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.027+0000 7f0164ff9700 1 --2- 172.21.15.42:0/662249747 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f014003adb0 0x7f014003d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:36:51.058 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.027+0000 7f0164ff9700 1 -- 172.21.15.42:0/662249747 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 7923+0+0 (secure 0 0 0) 0x7f015804b000 con 0x7f016807ca30 2019-12-05T05:36:51.058 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.027+0000 7f0167fff700 1 --2- 172.21.15.42:0/662249747 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f014003adb0 0x7f014003d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:36:51.058 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.027+0000 7f0167fff700 1 --2- 172.21.15.42:0/662249747 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f014003adb0 0x7f014003d2c0 crc :-1 s=READY pgs=497 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:36:51.058 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.031+0000 7f016e19d700 1 -- 172.21.15.42:0/662249747 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f016805d110 con 0x7f016807ca30 2019-12-05T05:36:51.058 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.039+0000 7f016cf3b700 1 --2- 172.21.15.42:0/662249747 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f016807ca30 0x7f016807ce60 secure :-1 s=THROTTLE_DONE pgs=1146 cs=0 l=1 rx=0x7f0168133d40 tx=0x7f0158010cb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:36:51.058 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.039+0000 7f0164ff9700 1 -- 172.21.15.42:0/662249747 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f0158032eb0 con 0x7f016807ca30 2019-12-05T05:36:51.213 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.207+0000 7f016e19d700 1 -- 172.21.15.42:0/662249747 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f016805b8c0 con 0x7f016807ca30 2019-12-05T05:36:51.214 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.207+0000 7f016cf3b700 1 --2- 172.21.15.42:0/662249747 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f016807ca30 0x7f016807ce60 secure :-1 s=THROTTLE_DONE pgs=1146 cs=0 l=1 rx=0x7f0168133d40 tx=0x7f0158010cb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:36:51.214 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.207+0000 7f0164ff9700 1 -- 172.21.15.42:0/662249747 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 143 v143) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f0158033970 con 0x7f016807ca30 2019-12-05T05:36:51.216 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 143 2019-12-05T05:36:51.217 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:36:51.217 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":143,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/3838877789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3838877789},{"type":"v1","addr":"172.21.15.42:6835","nonce":3838877789}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":143}],"filesystems":[{"mdsmap":{"epoch":142,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:34:39.042388+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7221},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7221":{"gid":7221,"name":"b","rank":0,"incarnation":139,"state":"up:active","state_seq":27,"addr":"172.21.15.40:6833/3789641053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3789641053},{"type":"v1","addr":"172.21.15.40:6833","nonce":3789641053}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:36:51.220 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.215+0000 7f016e19d700 1 -- 172.21.15.42:0/662249747 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f014003adb0 msgr2=0x7f014003d2c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:36:51.221 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.215+0000 7f016e19d700 1 --2- 172.21.15.42:0/662249747 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f014003adb0 0x7f014003d2c0 crc :-1 s=READY pgs=497 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:36:51.221 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.215+0000 7f016e19d700 1 -- 172.21.15.42:0/662249747 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f016807ca30 msgr2=0x7f016807ce60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:36:51.221 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.215+0000 7f016e19d700 1 --2- 172.21.15.42:0/662249747 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f016807ca30 0x7f016807ce60 secure :-1 s=READY pgs=1146 cs=0 l=1 rx=0x7f0168133d40 tx=0x7f0158010cb0).stop 2019-12-05T05:36:51.222 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.215+0000 7f016e19d700 1 -- 172.21.15.42:0/662249747 shutdown_connections 2019-12-05T05:36:51.222 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.215+0000 7f016e19d700 1 --2- 172.21.15.42:0/662249747 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f014003adb0 0x7f014003d2c0 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:36:51.223 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.215+0000 7f016e19d700 1 --2- 172.21.15.42:0/662249747 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f016807ca30 0x7f016807ce60 unknown :-1 s=CLOSED pgs=1146 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:36:51.223 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.215+0000 7f016e19d700 1 --2- 172.21.15.42:0/662249747 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0168116500 0x7f01680a55e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:36:51.223 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.215+0000 7f016e19d700 1 -- 172.21.15.42:0/662249747 shutdown_connections 2019-12-05T05:36:51.223 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.215+0000 7f016e19d700 1 -- 172.21.15.42:0/662249747 wait complete. 2019-12-05T05:36:51.223 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.215+0000 7f016e19d700 1 -- 172.21.15.42:0/662249747 >> 172.21.15.42:0/662249747 conn(0x7f01680a9790 msgr2=0x7f0168059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:36:51.234 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.b (rank=0) 2019-12-05T05:36:51.234 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2019-12-05T05:36:51.234 INFO:teuthology.orchestra.run:waiting for 300 2019-12-05T05:36:51.250 INFO:tasks.ceph.mds.b:Stopped 2019-12-05T05:36:51.251 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:36:51.361 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.343+0000 7f26d05bb700 1 Processor -- start 2019-12-05T05:36:51.362 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.343+0000 7f26d05bb700 1 -- start start 2019-12-05T05:36:51.362 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.347+0000 7f26d05bb700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f26c80a7730 0x7f26c808df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:36:51.362 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.347+0000 7f26d05bb700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f26c807e710 0x7f26c807eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:36:51.363 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.347+0000 7f26d05bb700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f26c809f2d0 0x7f26c809f700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:36:51.363 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.347+0000 7f26d05bb700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26c80a0af0 con 0x7f26c807e710 2019-12-05T05:36:51.363 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.347+0000 7f26d05bb700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26c808e5a0 con 0x7f26c80a7730 2019-12-05T05:36:51.363 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.347+0000 7f26d05bb700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26c808e840 con 0x7f26c809f2d0 2019-12-05T05:36:51.363 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.347+0000 7f26ceb58700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f26c80a7730 0x7f26c808df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:36:51.363 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.347+0000 7f26ceb58700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f26c80a7730 0x7f26c808df70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:37290/0 (socket says 172.21.15.42:37290) 2019-12-05T05:36:51.364 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.347+0000 7f26ce357700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f26c807e710 0x7f26c807eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:36:51.364 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.347+0000 7f26ceb58700 1 -- 172.21.15.42:0/3320639886 learned_addr learned my addr 172.21.15.42:0/3320639886 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:36:51.364 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.347+0000 7f26cf359700 1 --2- 172.21.15.42:0/3320639886 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f26c809f2d0 0x7f26c809f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:36:51.364 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.347+0000 7f26ceb58700 1 -- 172.21.15.42:0/3320639886 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f26c809f2d0 msgr2=0x7f26c809f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:36:51.364 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.347+0000 7f26ceb58700 1 --2- 172.21.15.42:0/3320639886 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f26c809f2d0 0x7f26c809f700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:36:51.364 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.347+0000 7f26ceb58700 1 -- 172.21.15.42:0/3320639886 >> v2:172.21.15.40:3300/0 conn(0x7f26c807e710 msgr2=0x7f26c807eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:36:51.364 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.347+0000 7f26ceb58700 1 --2- 172.21.15.42:0/3320639886 >> v2:172.21.15.40:3300/0 conn(0x7f26c807e710 0x7f26c807eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:36:51.365 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.347+0000 7f26ceb58700 1 -- 172.21.15.42:0/3320639886 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26c807f090 con 0x7f26c80a7730 2019-12-05T05:36:51.365 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.347+0000 7f26cf359700 1 --2- 172.21.15.42:0/3320639886 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f26c809f2d0 0x7f26c809f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T05:36:51.365 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.347+0000 7f26ceb58700 1 --2- 172.21.15.42:0/3320639886 >> v2:172.21.15.42:3300/0 conn(0x7f26c80a7730 0x7f26c808df70 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f26c80a0f40 tx=0x7f26c400fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:36:51.365 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.347+0000 7f26ceb58700 1 --2- 172.21.15.42:0/3320639886 >> v2:172.21.15.42:3300/0 conn(0x7f26c80a7730 0x7f26c808df70 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f26c80a0f40 tx=0x7f26c400fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:36:51.365 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.347+0000 7f26ceb58700 1 --2- 172.21.15.42:0/3320639886 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f26c80a7730 0x7f26c808df70 secure :-1 s=READY pgs=1147 cs=0 l=1 rx=0x7f26c80a0f40 tx=0x7f26c400fdb0).ready entity=mon.1 client_cookie=338fa2f167e7667f server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:36:51.365 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.347+0000 7f26ceb58700 1 --2- 172.21.15.42:0/3320639886 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f26c80a7730 0x7f26c808df70 secure :-1 s=THROTTLE_DONE pgs=1147 cs=0 l=1 rx=0x7f26c80a0f40 tx=0x7f26c400fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:36:51.365 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.347+0000 7f26cdb56700 1 -- 172.21.15.42:0/3320639886 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f26c4008c90 con 0x7f26c80a7730 2019-12-05T05:36:51.366 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.347+0000 7f26ceb58700 1 --2- 172.21.15.42:0/3320639886 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f26c80a7730 0x7f26c808df70 secure :-1 s=THROTTLE_DONE pgs=1147 cs=0 l=1 rx=0x7f26c80a0f40 tx=0x7f26c400fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:36:51.367 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.347+0000 7f26ceb58700 1 --2- 172.21.15.42:0/3320639886 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f26c80a7730 0x7f26c808df70 secure :-1 s=THROTTLE_DONE pgs=1147 cs=0 l=1 rx=0x7f26c80a0f40 tx=0x7f26c400fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:36:51.372 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.347+0000 7f26cdb56700 1 -- 172.21.15.42:0/3320639886 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26c40088b0 con 0x7f26c80a7730 2019-12-05T05:36:51.372 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.347+0000 7f26cdb56700 1 -- 172.21.15.42:0/3320639886 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f26c4011b50 con 0x7f26c80a7730 2019-12-05T05:36:51.372 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.347+0000 7f26d05bb700 1 -- 172.21.15.42:0/3320639886 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f26c80a7730 msgr2=0x7f26c808df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:36:51.372 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.347+0000 7f26d05bb700 1 --2- 172.21.15.42:0/3320639886 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f26c80a7730 0x7f26c808df70 secure :-1 s=READY pgs=1147 cs=0 l=1 rx=0x7f26c80a0f40 tx=0x7f26c400fdb0).stop 2019-12-05T05:36:51.372 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.347+0000 7f26d05bb700 1 -- 172.21.15.42:0/3320639886 shutdown_connections 2019-12-05T05:36:51.372 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.347+0000 7f26d05bb700 1 --2- 172.21.15.42:0/3320639886 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f26c809f2d0 0x7f26c809f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:36:51.374 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.347+0000 7f26d05bb700 1 --2- 172.21.15.42:0/3320639886 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f26c80a7730 0x7f26c808df70 unknown :-1 s=CLOSED pgs=1147 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:36:51.374 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.347+0000 7f26d05bb700 1 --2- 172.21.15.42:0/3320639886 >> v2:172.21.15.40:3300/0 conn(0x7f26c807e710 0x7f26c807eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:36:51.374 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.347+0000 7f26d05bb700 1 -- 172.21.15.42:0/3320639886 shutdown_connections 2019-12-05T05:36:51.374 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.347+0000 7f26d05bb700 1 -- 172.21.15.42:0/3320639886 wait complete. 2019-12-05T05:36:51.374 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.347+0000 7f26d05bb700 1 -- 172.21.15.42:0/3320639886 >> 172.21.15.42:0/3320639886 conn(0x7f26c80aabd0 msgr2=0x7f26c80a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:36:51.374 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.351+0000 7f26d05bb700 1 Processor -- start 2019-12-05T05:36:51.375 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.351+0000 7f26d05bb700 1 -- start start 2019-12-05T05:36:51.375 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.351+0000 7f26d05bb700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f26c812b600 0x7f26c80a62c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:36:51.375 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.351+0000 7f26d05bb700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26c8142ea0 con 0x7f26c80a5e90 2019-12-05T05:36:51.375 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.351+0000 7f26d05bb700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26c8137790 con 0x7f26c811cd90 2019-12-05T05:36:51.376 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.351+0000 7f26d05bb700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26c812eca0 con 0x7f26c812b600 2019-12-05T05:36:51.376 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.351+0000 7f26ceb58700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f26c811cd90 0x7f26c812e320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:37332/0 (socket says 172.21.15.42:37332) 2019-12-05T05:36:51.376 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.351+0000 7f26cf359700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f26c812b600 0x7f26c80a62c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:36:51.376 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.351+0000 7f26ceb58700 1 -- 172.21.15.42:0/2950237917 learned_addr learned my addr 172.21.15.42:0/2950237917 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:36:51.377 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.355+0000 7f26cf359700 1 -- 172.21.15.42:0/2950237917 >> v1:172.21.15.42:6789/0 conn(0x7f26c811cd90 legacy=0x7f26c812e320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:36:51.377 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.355+0000 7f26bb7fe700 1 -- 172.21.15.42:0/2950237917 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2413385664 0 0) 0x7f26c8142ea0 con 0x7f26c80a5e90 2019-12-05T05:36:51.378 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.355+0000 7f26cf359700 1 -- 172.21.15.42:0/2950237917 >> v1:172.21.15.40:6789/0 conn(0x7f26c80a5e90 legacy=0x7f26c812ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:36:51.378 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.355+0000 7f26cf359700 1 -- 172.21.15.42:0/2950237917 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26c81389d0 con 0x7f26c812b600 2019-12-05T05:36:51.378 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.355+0000 7f26cf359700 1 --2- 172.21.15.42:0/2950237917 >> v2:172.21.15.42:3300/0 conn(0x7f26c812b600 0x7f26c80a62c0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f26c4007ef0 tx=0x7f26c400fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:36:51.378 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.355+0000 7f26cf359700 1 --2- 172.21.15.42:0/2950237917 >> v2:172.21.15.42:3300/0 conn(0x7f26c812b600 0x7f26c80a62c0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f26c4007ef0 tx=0x7f26c400fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:36:51.378 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.355+0000 7f26cf359700 1 --2- 172.21.15.42:0/2950237917 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f26c812b600 0x7f26c80a62c0 secure :-1 s=READY pgs=1149 cs=0 l=1 rx=0x7f26c4007ef0 tx=0x7f26c400fdb0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:36:51.378 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.355+0000 7f26cf359700 1 --2- 172.21.15.42:0/2950237917 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f26c812b600 0x7f26c80a62c0 secure :-1 s=THROTTLE_DONE pgs=1149 cs=0 l=1 rx=0x7f26c4007ef0 tx=0x7f26c400fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:36:51.378 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.355+0000 7f26bb7fe700 1 -- 172.21.15.42:0/2950237917 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f26c0005ca0 con 0x7f26c812b600 2019-12-05T05:36:51.379 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.355+0000 7f26cf359700 1 --2- 172.21.15.42:0/2950237917 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f26c812b600 0x7f26c80a62c0 secure :-1 s=THROTTLE_DONE pgs=1149 cs=0 l=1 rx=0x7f26c4007ef0 tx=0x7f26c400fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:36:51.379 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.355+0000 7f26bb7fe700 1 -- 172.21.15.42:0/2950237917 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26c00058c0 con 0x7f26c812b600 2019-12-05T05:36:51.379 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.355+0000 7f26d05bb700 1 -- 172.21.15.42:0/2950237917 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26c8136780 con 0x7f26c812b600 2019-12-05T05:36:51.379 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.355+0000 7f26cf359700 1 --2- 172.21.15.42:0/2950237917 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f26c812b600 0x7f26c80a62c0 secure :-1 s=THROTTLE_DONE pgs=1149 cs=0 l=1 rx=0x7f26c4007ef0 tx=0x7f26c400fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:36:51.379 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.355+0000 7f26bb7fe700 1 -- 172.21.15.42:0/2950237917 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f26c0007530 con 0x7f26c812b600 2019-12-05T05:36:51.379 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.355+0000 7f26d05bb700 1 -- 172.21.15.42:0/2950237917 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26c8136d30 con 0x7f26c812b600 2019-12-05T05:36:51.379 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.355+0000 7f26cf359700 1 --2- 172.21.15.42:0/2950237917 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f26c812b600 0x7f26c80a62c0 secure :-1 s=THROTTLE_DONE pgs=1149 cs=0 l=1 rx=0x7f26c4007ef0 tx=0x7f26c400fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:36:51.389 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.355+0000 7f26bb7fe700 1 -- 172.21.15.42:0/2950237917 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f26c00125b0 con 0x7f26c812b600 2019-12-05T05:36:51.390 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.355+0000 7f26cf359700 1 --2- 172.21.15.42:0/2950237917 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f26c812b600 0x7f26c80a62c0 secure :-1 s=THROTTLE_DONE pgs=1149 cs=0 l=1 rx=0x7f26c4007ef0 tx=0x7f26c400fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:36:51.390 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.355+0000 7f26bb7fe700 1 --2- 172.21.15.42:0/2950237917 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f26b003ab90 0x7f26b003d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:36:51.390 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.355+0000 7f26bb7fe700 1 -- 172.21.15.42:0/2950237917 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 7923+0+0 (secure 0 0 0) 0x7f26c004afa0 con 0x7f26c812b600 2019-12-05T05:36:51.391 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.355+0000 7f26ceb58700 1 --2- 172.21.15.42:0/2950237917 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f26b003ab90 0x7f26b003d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:36:51.391 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.355+0000 7f26ceb58700 1 --2- 172.21.15.42:0/2950237917 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f26b003ab90 0x7f26b003d0a0 crc :-1 s=READY pgs=498 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:36:51.391 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.359+0000 7f26d05bb700 1 -- 172.21.15.42:0/2950237917 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26c805d100 con 0x7f26c812b600 2019-12-05T05:36:51.391 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.367+0000 7f26cf359700 1 --2- 172.21.15.42:0/2950237917 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f26c812b600 0x7f26c80a62c0 secure :-1 s=THROTTLE_DONE pgs=1149 cs=0 l=1 rx=0x7f26c4007ef0 tx=0x7f26c400fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:36:51.391 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.367+0000 7f26bb7fe700 1 -- 172.21.15.42:0/2950237917 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f26c0032000 con 0x7f26c812b600 2019-12-05T05:36:51.535 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.527+0000 7f26d05bb700 1 -- 172.21.15.42:0/2950237917 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f26c805b8b0 con 0x7f26c812b600 2019-12-05T05:36:51.536 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.531+0000 7f26cf359700 1 --2- 172.21.15.42:0/2950237917 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f26c812b600 0x7f26c80a62c0 secure :-1 s=THROTTLE_DONE pgs=1149 cs=0 l=1 rx=0x7f26c4007ef0 tx=0x7f26c400fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:36:51.536 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.531+0000 7f26bb7fe700 1 -- 172.21.15.42:0/2950237917 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 143 v143) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f26c0032260 con 0x7f26c812b600 2019-12-05T05:36:51.536 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:36:51.537 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":143,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/3838877789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3838877789},{"type":"v1","addr":"172.21.15.42:6835","nonce":3838877789}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":143}],"filesystems":[{"mdsmap":{"epoch":142,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:34:39.042388+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7221},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7221":{"gid":7221,"name":"b","rank":0,"incarnation":139,"state":"up:active","state_seq":27,"addr":"172.21.15.40:6833/3789641053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3789641053},{"type":"v1","addr":"172.21.15.40:6833","nonce":3789641053}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:36:51.537 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 143 2019-12-05T05:36:51.539 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.531+0000 7f26d05bb700 1 -- 172.21.15.42:0/2950237917 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f26b003ab90 msgr2=0x7f26b003d0a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:36:51.539 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.531+0000 7f26d05bb700 1 --2- 172.21.15.42:0/2950237917 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f26b003ab90 0x7f26b003d0a0 crc :-1 s=READY pgs=498 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:36:51.539 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.531+0000 7f26d05bb700 1 -- 172.21.15.42:0/2950237917 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f26c812b600 msgr2=0x7f26c80a62c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:36:51.539 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.531+0000 7f26d05bb700 1 --2- 172.21.15.42:0/2950237917 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f26c812b600 0x7f26c80a62c0 secure :-1 s=READY pgs=1149 cs=0 l=1 rx=0x7f26c4007ef0 tx=0x7f26c400fdb0).stop 2019-12-05T05:36:51.539 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.531+0000 7f26d05bb700 1 -- 172.21.15.42:0/2950237917 shutdown_connections 2019-12-05T05:36:51.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.531+0000 7f26d05bb700 1 --2- 172.21.15.42:0/2950237917 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f26b003ab90 0x7f26b003d0a0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:36:51.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.531+0000 7f26d05bb700 1 --2- 172.21.15.42:0/2950237917 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f26c812b600 0x7f26c80a62c0 unknown :-1 s=CLOSED pgs=1149 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:36:51.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.531+0000 7f26d05bb700 1 -- 172.21.15.42:0/2950237917 shutdown_connections 2019-12-05T05:36:51.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.531+0000 7f26d05bb700 1 -- 172.21.15.42:0/2950237917 wait complete. 2019-12-05T05:36:51.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:51.531+0000 7f26d05bb700 1 -- 172.21.15.42:0/2950237917 >> 172.21.15.42:0/2950237917 conn(0x7f26c807df60 msgr2=0x7f26c8141350 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:36:51.548 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T05:36:53.549 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:36:53.659 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:53.647+0000 7f93afb5e700 1 Processor -- start 2019-12-05T05:36:53.659 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:53.647+0000 7f93afb5e700 1 -- start start 2019-12-05T05:36:53.659 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:53.651+0000 7f93afb5e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f93a808df80 0x7f93a809f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:36:53.659 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:53.651+0000 7f93afb5e700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f93a809f830 0x7f93a814efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:36:53.659 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:53.651+0000 7f93afb5e700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93a809fc60 con 0x7f93a80a7740 2019-12-05T05:36:53.660 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:53.651+0000 7f93afb5e700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93a80a0b00 con 0x7f93a809f830 2019-12-05T05:36:53.660 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:53.651+0000 7f93afb5e700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93a808e5b0 con 0x7f93a808df80 2019-12-05T05:36:53.660 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:53.651+0000 7f93ad8fa700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f93a808df80 0x7f93a809f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:36:53.660 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:53.651+0000 7f93ad8fa700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f93a808df80 0x7f93a809f2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:37310/0 (socket says 172.21.15.42:37310) 2019-12-05T05:36:53.660 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:53.651+0000 7f93ad8fa700 1 -- 172.21.15.42:0/2138287027 learned_addr learned my addr 172.21.15.42:0/2138287027 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:36:53.660 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:53.651+0000 7f93ae8fc700 1 --2- 172.21.15.42:0/2138287027 >> v2:172.21.15.40:3300/0 conn(0x7f93a809f830 0x7f93a814efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:36:53.661 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:53.651+0000 7f93ad8fa700 1 -- 172.21.15.42:0/2138287027 >> v2:172.21.15.40:3300/0 conn(0x7f93a809f830 msgr2=0x7f93a814efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:36:53.661 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:53.651+0000 7f93ad8fa700 1 --2- 172.21.15.42:0/2138287027 >> v2:172.21.15.40:3300/0 conn(0x7f93a809f830 0x7f93a814efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:36:53.661 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:53.651+0000 7f93ad8fa700 1 -- 172.21.15.42:0/2138287027 >> v1:172.21.15.40:6789/0 conn(0x7f93a80a7740 legacy=0x7f93a807e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:36:53.661 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:53.651+0000 7f93ad8fa700 1 -- 172.21.15.42:0/2138287027 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93a80a0710 con 0x7f93a808df80 2019-12-05T05:36:53.661 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:53.651+0000 7f93ad8fa700 1 --2- 172.21.15.42:0/2138287027 >> v2:172.21.15.42:3300/0 conn(0x7f93a808df80 0x7f93a809f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f9394002f30 tx=0x7f939400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:36:53.661 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:53.651+0000 7f93ad8fa700 1 --2- 172.21.15.42:0/2138287027 >> v2:172.21.15.42:3300/0 conn(0x7f93a808df80 0x7f93a809f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f9394002f30 tx=0x7f939400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:36:53.661 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:53.651+0000 7f93ad8fa700 1 --2- 172.21.15.42:0/2138287027 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f93a808df80 0x7f93a809f2e0 secure :-1 s=READY pgs=1150 cs=0 l=1 rx=0x7f9394002f30 tx=0x7f939400fd30).ready entity=mon.1 client_cookie=523942089158cae5 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:36:53.662 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:53.651+0000 7f93ad8fa700 1 --2- 172.21.15.42:0/2138287027 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f93a808df80 0x7f93a809f2e0 secure :-1 s=THROTTLE_DONE pgs=1150 cs=0 l=1 rx=0x7f9394002f30 tx=0x7f939400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:36:53.662 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:53.651+0000 7f93ad0f9700 1 -- 172.21.15.42:0/2138287027 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9394008c90 con 0x7f93a808df80 2019-12-05T05:36:53.662 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:53.651+0000 7f93ad8fa700 1 --2- 172.21.15.42:0/2138287027 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f93a808df80 0x7f93a809f2e0 secure :-1 s=THROTTLE_DONE pgs=1150 cs=0 l=1 rx=0x7f9394002f30 tx=0x7f939400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:36:53.663 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:53.651+0000 7f93ad8fa700 1 --2- 172.21.15.42:0/2138287027 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f93a808df80 0x7f93a809f2e0 secure :-1 s=THROTTLE_DONE pgs=1150 cs=0 l=1 rx=0x7f9394002f30 tx=0x7f939400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:36:53.663 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:53.651+0000 7f93ad0f9700 1 -- 172.21.15.42:0/2138287027 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93940088b0 con 0x7f93a808df80 2019-12-05T05:36:53.663 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:53.651+0000 7f93ad0f9700 1 -- 172.21.15.42:0/2138287027 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9394011ad0 con 0x7f93a808df80 2019-12-05T05:36:53.663 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:53.651+0000 7f93afb5e700 1 -- 172.21.15.42:0/2138287027 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f93a808df80 msgr2=0x7f93a809f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:36:53.663 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:53.651+0000 7f93afb5e700 1 --2- 172.21.15.42:0/2138287027 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f93a808df80 0x7f93a809f2e0 secure :-1 s=READY pgs=1150 cs=0 l=1 rx=0x7f9394002f30 tx=0x7f939400fd30).stop 2019-12-05T05:36:53.663 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:53.651+0000 7f93afb5e700 1 -- 172.21.15.42:0/2138287027 shutdown_connections 2019-12-05T05:36:53.663 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:53.651+0000 7f93afb5e700 1 --2- 172.21.15.42:0/2138287027 >> v2:172.21.15.40:3300/0 conn(0x7f93a809f830 0x7f93a814efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:36:53.664 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:53.651+0000 7f93afb5e700 1 --2- 172.21.15.42:0/2138287027 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f93a808df80 0x7f93a809f2e0 unknown :-1 s=CLOSED pgs=1150 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:36:53.664 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:53.651+0000 7f93afb5e700 1 -- 172.21.15.42:0/2138287027 shutdown_connections 2019-12-05T05:36:53.664 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:53.651+0000 7f93afb5e700 1 -- 172.21.15.42:0/2138287027 wait complete. 2019-12-05T05:36:53.664 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:53.651+0000 7f93afb5e700 1 -- 172.21.15.42:0/2138287027 >> 172.21.15.42:0/2138287027 conn(0x7f93a80aabe0 msgr2=0x7f93a80a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:36:53.667 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:53.655+0000 7f93afb5e700 1 Processor -- start 2019-12-05T05:36:53.667 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:53.655+0000 7f93afb5e700 1 -- start start 2019-12-05T05:36:53.667 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:53.655+0000 7f93afb5e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f93a8143be0 0x7f93a8144010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:36:53.667 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:53.655+0000 7f93afb5e700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f93a8144560 0x7f93a81379e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:36:53.667 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:53.655+0000 7f93afb5e700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93a80a1c50 con 0x7f93a811bc20 2019-12-05T05:36:53.668 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:53.655+0000 7f93afb5e700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93a8143980 con 0x7f93a8144560 2019-12-05T05:36:53.668 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:53.655+0000 7f93afb5e700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93a8142f60 con 0x7f93a8143be0 2019-12-05T05:36:53.668 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:53.655+0000 7f93ad8fa700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f93a8143be0 0x7f93a8144010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:36:53.668 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:53.655+0000 7f93ae0fb700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f93a811bc20 0x7f93a8142530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:37352/0 (socket says 172.21.15.42:37352) 2019-12-05T05:36:53.668 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:53.655+0000 7f93ae0fb700 1 -- 172.21.15.42:0/1010571608 learned_addr learned my addr 172.21.15.42:0/1010571608 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:36:53.668 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:53.655+0000 7f93ae8fc700 1 --2- 172.21.15.42:0/1010571608 >> v2:172.21.15.40:3300/0 conn(0x7f93a8144560 0x7f93a81379e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:36:53.668 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:53.659+0000 7f93ad8fa700 1 -- 172.21.15.42:0/1010571608 >> v2:172.21.15.40:3300/0 conn(0x7f93a8144560 msgr2=0x7f93a81379e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:36:53.669 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:53.659+0000 7f93ad8fa700 1 --2- 172.21.15.42:0/1010571608 >> v2:172.21.15.40:3300/0 conn(0x7f93a8144560 0x7f93a81379e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:36:53.669 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:53.659+0000 7f93ad8fa700 1 -- 172.21.15.42:0/1010571608 >> v1:172.21.15.42:6789/0 conn(0x7f93a811bc20 legacy=0x7f93a8142530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:36:53.669 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:53.659+0000 7f93ad8fa700 1 -- 172.21.15.42:0/1010571608 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93a807d110 con 0x7f93a8143be0 2019-12-05T05:36:53.669 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:53.659+0000 7f93a6ffd700 1 -- 172.21.15.42:0/1010571608 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 470305352 0 0) 0x7f93a80a1c50 con 0x7f93a811bc20 2019-12-05T05:36:53.669 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:53.659+0000 7f93ad8fa700 1 --2- 172.21.15.42:0/1010571608 >> v2:172.21.15.42:3300/0 conn(0x7f93a8143be0 0x7f93a8144010 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f9394002f30 tx=0x7f9394008e30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:36:53.669 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:53.659+0000 7f93ad8fa700 1 --2- 172.21.15.42:0/1010571608 >> v2:172.21.15.42:3300/0 conn(0x7f93a8143be0 0x7f93a8144010 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f9394002f30 tx=0x7f9394008e30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:36:53.669 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:53.659+0000 7f93ad8fa700 1 --2- 172.21.15.42:0/1010571608 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f93a8143be0 0x7f93a8144010 secure :-1 s=READY pgs=1152 cs=0 l=1 rx=0x7f9394002f30 tx=0x7f9394008e30).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:36:53.670 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:53.659+0000 7f93ad8fa700 1 --2- 172.21.15.42:0/1010571608 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f93a8143be0 0x7f93a8144010 secure :-1 s=THROTTLE_DONE pgs=1152 cs=0 l=1 rx=0x7f9394002f30 tx=0x7f9394008e30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:36:53.670 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:53.659+0000 7f93ad8fa700 1 --2- 172.21.15.42:0/1010571608 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f93a8143be0 0x7f93a8144010 secure :-1 s=THROTTLE_DONE pgs=1152 cs=0 l=1 rx=0x7f9394002f30 tx=0x7f9394008e30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:36:53.670 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:53.659+0000 7f93ad8fa700 1 --2- 172.21.15.42:0/1010571608 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f93a8143be0 0x7f93a8144010 secure :-1 s=THROTTLE_DONE pgs=1152 cs=0 l=1 rx=0x7f9394002f30 tx=0x7f9394008e30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:36:53.670 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:53.659+0000 7f93a6ffd700 1 -- 172.21.15.42:0/1010571608 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f939400f680 con 0x7f93a8143be0 2019-12-05T05:36:53.670 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:53.659+0000 7f93a6ffd700 1 -- 172.21.15.42:0/1010571608 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93940116f0 con 0x7f93a8143be0 2019-12-05T05:36:53.670 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:53.659+0000 7f93afb5e700 1 -- 172.21.15.42:0/1010571608 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93a811c050 con 0x7f93a8143be0 2019-12-05T05:36:53.670 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:53.659+0000 7f93a6ffd700 1 -- 172.21.15.42:0/1010571608 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9394005920 con 0x7f93a8143be0 2019-12-05T05:36:53.671 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:53.659+0000 7f93afb5e700 1 -- 172.21.15.42:0/1010571608 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93a8137f30 con 0x7f93a8143be0 2019-12-05T05:36:53.671 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:53.659+0000 7f93ad8fa700 1 --2- 172.21.15.42:0/1010571608 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f93a8143be0 0x7f93a8144010 secure :-1 s=THROTTLE_DONE pgs=1152 cs=0 l=1 rx=0x7f9394002f30 tx=0x7f9394008e30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:36:53.671 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:53.659+0000 7f93a6ffd700 1 -- 172.21.15.42:0/1010571608 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f9394005bb0 con 0x7f93a8143be0 2019-12-05T05:36:53.671 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:53.659+0000 7f93ad8fa700 1 --2- 172.21.15.42:0/1010571608 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f93a8143be0 0x7f93a8144010 secure :-1 s=THROTTLE_DONE pgs=1152 cs=0 l=1 rx=0x7f9394002f30 tx=0x7f9394008e30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:36:53.671 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:53.659+0000 7f93a6ffd700 1 --2- 172.21.15.42:0/1010571608 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f939003ace0 0x7f939003d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:36:53.671 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:53.659+0000 7f93a6ffd700 1 -- 172.21.15.42:0/1010571608 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 7923+0+0 (secure 0 0 0) 0x7f939404bab0 con 0x7f93a8143be0 2019-12-05T05:36:53.672 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:53.659+0000 7f93ae0fb700 1 --2- 172.21.15.42:0/1010571608 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f939003ace0 0x7f939003d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:36:53.672 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:53.663+0000 7f93ae0fb700 1 --2- 172.21.15.42:0/1010571608 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f939003ace0 0x7f939003d1f0 crc :-1 s=READY pgs=499 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:36:53.673 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:53.663+0000 7f93afb5e700 1 -- 172.21.15.42:0/1010571608 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93a805d110 con 0x7f93a8143be0 2019-12-05T05:36:53.676 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:53.671+0000 7f93ad8fa700 1 --2- 172.21.15.42:0/1010571608 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f93a8143be0 0x7f93a8144010 secure :-1 s=THROTTLE_DONE pgs=1152 cs=0 l=1 rx=0x7f9394002f30 tx=0x7f9394008e30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:36:53.676 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:53.671+0000 7f93a6ffd700 1 -- 172.21.15.42:0/1010571608 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f9394032b30 con 0x7f93a8143be0 2019-12-05T05:36:53.848 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:53.835+0000 7f93afb5e700 1 -- 172.21.15.42:0/1010571608 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f93a805b8c0 con 0x7f93a8143be0 2019-12-05T05:36:53.849 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:53.839+0000 7f93ad8fa700 1 --2- 172.21.15.42:0/1010571608 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f93a8143be0 0x7f93a8144010 secure :-1 s=THROTTLE_DONE pgs=1152 cs=0 l=1 rx=0x7f9394002f30 tx=0x7f9394008e30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:36:53.849 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:53.839+0000 7f93a6ffd700 1 -- 172.21.15.42:0/1010571608 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 143 v143) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f9394032d90 con 0x7f93a8143be0 2019-12-05T05:36:53.849 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 143 2019-12-05T05:36:53.849 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:53.839+0000 7f93afb5e700 1 -- 172.21.15.42:0/1010571608 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f939003ace0 msgr2=0x7f939003d1f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:36:53.849 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:53.839+0000 7f93afb5e700 1 --2- 172.21.15.42:0/1010571608 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f939003ace0 0x7f939003d1f0 crc :-1 s=READY pgs=499 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:36:53.850 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:53.839+0000 7f93afb5e700 1 -- 172.21.15.42:0/1010571608 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f93a8143be0 msgr2=0x7f93a8144010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:36:53.850 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:53.839+0000 7f93afb5e700 1 --2- 172.21.15.42:0/1010571608 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f93a8143be0 0x7f93a8144010 secure :-1 s=READY pgs=1152 cs=0 l=1 rx=0x7f9394002f30 tx=0x7f9394008e30).stop 2019-12-05T05:36:53.850 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:53.839+0000 7f93afb5e700 1 -- 172.21.15.42:0/1010571608 shutdown_connections 2019-12-05T05:36:53.850 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:53.839+0000 7f93afb5e700 1 --2- 172.21.15.42:0/1010571608 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f939003ace0 0x7f939003d1f0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:36:53.850 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:53.839+0000 7f93afb5e700 1 --2- 172.21.15.42:0/1010571608 >> v2:172.21.15.40:3300/0 conn(0x7f93a8144560 0x7f93a81379e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:36:53.851 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:53.839+0000 7f93afb5e700 1 --2- 172.21.15.42:0/1010571608 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f93a8143be0 0x7f93a8144010 unknown :-1 s=CLOSED pgs=1152 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:36:53.851 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:53.839+0000 7f93afb5e700 1 -- 172.21.15.42:0/1010571608 shutdown_connections 2019-12-05T05:36:53.851 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:53.839+0000 7f93afb5e700 1 -- 172.21.15.42:0/1010571608 wait complete. 2019-12-05T05:36:53.851 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:53.839+0000 7f93afb5e700 1 -- 172.21.15.42:0/1010571608 >> 172.21.15.42:0/1010571608 conn(0x7f93a807dc80 msgr2=0x7f93a8119300 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:36:53.851 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:36:53.852 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":143,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/3838877789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3838877789},{"type":"v1","addr":"172.21.15.42:6835","nonce":3838877789}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":143}],"filesystems":[{"mdsmap":{"epoch":142,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:34:39.042388+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7221},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7221":{"gid":7221,"name":"b","rank":0,"incarnation":139,"state":"up:active","state_seq":27,"addr":"172.21.15.40:6833/3789641053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3789641053},{"type":"v1","addr":"172.21.15.40:6833","nonce":3789641053}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:36:53.856 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T05:36:55.858 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:36:55.973 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:55.955+0000 7f71a4c1f700 1 Processor -- start 2019-12-05T05:36:55.973 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:55.955+0000 7f71a4c1f700 1 -- start start 2019-12-05T05:36:55.973 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:55.955+0000 7f71a4c1f700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f71a00a7740 0x7f71a008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:36:55.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:55.955+0000 7f71a4c1f700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f71a009fae0 0x7f71a007eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:36:55.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:55.955+0000 7f71a4c1f700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71a00a0710 con 0x7f71a007e720 2019-12-05T05:36:55.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:55.955+0000 7f71a4c1f700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71a00a0b00 con 0x7f71a009fae0 2019-12-05T05:36:55.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:55.955+0000 7f71a4c1f700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71a008e5b0 con 0x7f71a00a7740 2019-12-05T05:36:55.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:55.955+0000 7f719f59e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f71a009fae0 0x7f71a007eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:36:55.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:55.955+0000 7f719e59c700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f71a007e720 0x7f71a009f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:46802/0 (socket says 172.21.15.42:46802) 2019-12-05T05:36:55.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:55.955+0000 7f719e59c700 1 -- 172.21.15.42:0/496959945 learned_addr learned my addr 172.21.15.42:0/496959945 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:36:55.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:55.955+0000 7f719f59e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f71a009fae0 0x7f71a007eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:37328/0 (socket says 172.21.15.42:37328) 2019-12-05T05:36:55.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:55.955+0000 7f719ed9d700 1 --2- 172.21.15.42:0/496959945 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f71a00a7740 0x7f71a008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:36:55.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:55.959+0000 7f719f59e700 1 -- 172.21.15.42:0/496959945 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f71a00a7740 msgr2=0x7f71a008df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:36:55.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:55.959+0000 7f719f59e700 1 --2- 172.21.15.42:0/496959945 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f71a00a7740 0x7f71a008df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:36:55.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:55.959+0000 7f719dd9b700 1 -- 172.21.15.42:0/496959945 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1736541527 0 0) 0x7f71a00a0710 con 0x7f71a007e720 2019-12-05T05:36:55.976 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:55.959+0000 7f719f59e700 1 -- 172.21.15.42:0/496959945 >> v1:172.21.15.40:6789/0 conn(0x7f71a007e720 legacy=0x7f71a009f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:36:55.976 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:55.959+0000 7f719f59e700 1 -- 172.21.15.42:0/496959945 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71a007f0a0 con 0x7f71a009fae0 2019-12-05T05:36:55.976 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:55.959+0000 7f719f59e700 1 --2- 172.21.15.42:0/496959945 >> v2:172.21.15.42:3300/0 conn(0x7f71a009fae0 0x7f71a007eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f7198008b10 tx=0x7f719800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:36:55.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:55.959+0000 7f719f59e700 1 --2- 172.21.15.42:0/496959945 >> v2:172.21.15.42:3300/0 conn(0x7f71a009fae0 0x7f71a007eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f7198008b10 tx=0x7f719800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:36:55.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:55.959+0000 7f719f59e700 1 --2- 172.21.15.42:0/496959945 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f71a009fae0 0x7f71a007eb50 secure :-1 s=READY pgs=1153 cs=0 l=1 rx=0x7f7198008b10 tx=0x7f719800fff0).ready entity=mon.1 client_cookie=d1349a386422e4ea server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:36:55.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:55.959+0000 7f719f59e700 1 --2- 172.21.15.42:0/496959945 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f71a009fae0 0x7f71a007eb50 secure :-1 s=THROTTLE_DONE pgs=1153 cs=0 l=1 rx=0x7f7198008b10 tx=0x7f719800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:36:55.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:55.959+0000 7f719dd9b700 1 -- 172.21.15.42:0/496959945 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7198010900 con 0x7f71a009fae0 2019-12-05T05:36:55.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:55.959+0000 7f719f59e700 1 --2- 172.21.15.42:0/496959945 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f71a009fae0 0x7f71a007eb50 secure :-1 s=THROTTLE_DONE pgs=1153 cs=0 l=1 rx=0x7f7198008b10 tx=0x7f719800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:36:55.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:55.959+0000 7f719f59e700 1 --2- 172.21.15.42:0/496959945 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f71a009fae0 0x7f71a007eb50 secure :-1 s=THROTTLE_DONE pgs=1153 cs=0 l=1 rx=0x7f7198008b10 tx=0x7f719800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:36:55.978 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:55.959+0000 7f719dd9b700 1 -- 172.21.15.42:0/496959945 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7198008b50 con 0x7f71a009fae0 2019-12-05T05:36:55.978 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:55.959+0000 7f719dd9b700 1 -- 172.21.15.42:0/496959945 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f71980110c0 con 0x7f71a009fae0 2019-12-05T05:36:55.978 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:55.959+0000 7f71a4c1f700 1 -- 172.21.15.42:0/496959945 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f71a009fae0 msgr2=0x7f71a007eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:36:55.978 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:55.959+0000 7f71a4c1f700 1 --2- 172.21.15.42:0/496959945 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f71a009fae0 0x7f71a007eb50 secure :-1 s=READY pgs=1153 cs=0 l=1 rx=0x7f7198008b10 tx=0x7f719800fff0).stop 2019-12-05T05:36:55.979 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:55.959+0000 7f71a4c1f700 1 -- 172.21.15.42:0/496959945 shutdown_connections 2019-12-05T05:36:55.979 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:55.959+0000 7f71a4c1f700 1 --2- 172.21.15.42:0/496959945 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f71a009fae0 0x7f71a007eb50 unknown :-1 s=CLOSED pgs=1153 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:36:55.979 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:55.959+0000 7f71a4c1f700 1 --2- 172.21.15.42:0/496959945 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f71a00a7740 0x7f71a008df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:36:55.979 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:55.959+0000 7f71a4c1f700 1 -- 172.21.15.42:0/496959945 shutdown_connections 2019-12-05T05:36:55.979 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:55.959+0000 7f71a4c1f700 1 -- 172.21.15.42:0/496959945 wait complete. 2019-12-05T05:36:55.979 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:55.959+0000 7f71a4c1f700 1 -- 172.21.15.42:0/496959945 >> 172.21.15.42:0/496959945 conn(0x7f71a00aabe0 msgr2=0x7f71a00a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:36:55.979 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:55.963+0000 7f71a4c1f700 1 Processor -- start 2019-12-05T05:36:55.980 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:55.963+0000 7f71a4c1f700 1 -- start start 2019-12-05T05:36:55.980 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:55.963+0000 7f71a4c1f700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f71a008eeb0 0x7f71a0113270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:36:55.980 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:55.963+0000 7f71a4c1f700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f71a007d940 0x7f71a007dd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:36:55.980 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:55.963+0000 7f71a4c1f700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71a0113b40 con 0x7f71a0111e90 2019-12-05T05:36:55.980 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:55.963+0000 7f71a4c1f700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71a01138a0 con 0x7f71a008eeb0 2019-12-05T05:36:55.980 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:55.963+0000 7f71a4c1f700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71a014d5c0 con 0x7f71a007d940 2019-12-05T05:36:55.980 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:55.963+0000 7f719ed9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f71a008eeb0 0x7f71a0113270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:36:55.981 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:55.963+0000 7f719ed9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f71a008eeb0 0x7f71a0113270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:37330/0 (socket says 172.21.15.42:37330) 2019-12-05T05:36:55.981 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:55.963+0000 7f719ed9d700 1 -- 172.21.15.42:0/1249243609 learned_addr learned my addr 172.21.15.42:0/1249243609 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:36:55.981 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:55.963+0000 7f719f59e700 1 --2- 172.21.15.42:0/1249243609 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f71a007d940 0x7f71a007dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:36:55.981 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:55.963+0000 7f718f7fe700 1 -- 172.21.15.42:0/1249243609 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 693643699 0 0) 0x7f71a0113b40 con 0x7f71a0111e90 2019-12-05T05:36:55.981 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:55.963+0000 7f719ed9d700 1 -- 172.21.15.42:0/1249243609 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f71a007d940 msgr2=0x7f71a007dd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:36:55.982 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:55.963+0000 7f719ed9d700 1 --2- 172.21.15.42:0/1249243609 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f71a007d940 0x7f71a007dd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:36:55.983 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:55.963+0000 7f719ed9d700 1 -- 172.21.15.42:0/1249243609 >> v1:172.21.15.40:6789/0 conn(0x7f71a0111e90 legacy=0x7f71a014d860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:36:55.983 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:55.963+0000 7f719ed9d700 1 -- 172.21.15.42:0/1249243609 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71a01122c0 con 0x7f71a008eeb0 2019-12-05T05:36:55.983 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:55.967+0000 7f719ed9d700 1 --2- 172.21.15.42:0/1249243609 >> v2:172.21.15.42:3300/0 conn(0x7f71a008eeb0 0x7f71a0113270 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f71a0124540 tx=0x7f7190010ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:36:55.983 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:55.967+0000 7f719ed9d700 1 --2- 172.21.15.42:0/1249243609 >> v2:172.21.15.42:3300/0 conn(0x7f71a008eeb0 0x7f71a0113270 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f71a0124540 tx=0x7f7190010ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:36:55.983 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:55.967+0000 7f719ed9d700 1 --2- 172.21.15.42:0/1249243609 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f71a008eeb0 0x7f71a0113270 secure :-1 s=READY pgs=1154 cs=0 l=1 rx=0x7f71a0124540 tx=0x7f7190010ae0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:36:55.984 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:55.967+0000 7f719ed9d700 1 --2- 172.21.15.42:0/1249243609 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f71a008eeb0 0x7f71a0113270 secure :-1 s=THROTTLE_DONE pgs=1154 cs=0 l=1 rx=0x7f71a0124540 tx=0x7f7190010ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:36:55.984 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:55.967+0000 7f718f7fe700 1 -- 172.21.15.42:0/1249243609 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7190005930 con 0x7f71a008eeb0 2019-12-05T05:36:55.984 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:55.967+0000 7f719ed9d700 1 --2- 172.21.15.42:0/1249243609 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f71a008eeb0 0x7f71a0113270 secure :-1 s=THROTTLE_DONE pgs=1154 cs=0 l=1 rx=0x7f71a0124540 tx=0x7f7190010ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:36:55.984 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:55.967+0000 7f719ed9d700 1 --2- 172.21.15.42:0/1249243609 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f71a008eeb0 0x7f71a0113270 secure :-1 s=THROTTLE_DONE pgs=1154 cs=0 l=1 rx=0x7f71a0124540 tx=0x7f7190010ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:36:55.984 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:55.967+0000 7f718f7fe700 1 -- 172.21.15.42:0/1249243609 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7190007160 con 0x7f71a008eeb0 2019-12-05T05:36:55.984 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:55.967+0000 7f718f7fe700 1 -- 172.21.15.42:0/1249243609 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7190012120 con 0x7f71a008eeb0 2019-12-05T05:36:55.984 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:55.967+0000 7f71a4c1f700 1 -- 172.21.15.42:0/1249243609 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71a0129f90 con 0x7f71a008eeb0 2019-12-05T05:36:55.985 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:55.967+0000 7f71a4c1f700 1 -- 172.21.15.42:0/1249243609 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71a01313e0 con 0x7f71a008eeb0 2019-12-05T05:36:55.985 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:55.967+0000 7f719ed9d700 1 --2- 172.21.15.42:0/1249243609 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f71a008eeb0 0x7f71a0113270 secure :-1 s=THROTTLE_DONE pgs=1154 cs=0 l=1 rx=0x7f71a0124540 tx=0x7f7190010ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:36:55.986 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:55.967+0000 7f718f7fe700 1 -- 172.21.15.42:0/1249243609 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f7190012430 con 0x7f71a008eeb0 2019-12-05T05:36:55.986 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:55.967+0000 7f719ed9d700 1 --2- 172.21.15.42:0/1249243609 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f71a008eeb0 0x7f71a0113270 secure :-1 s=THROTTLE_DONE pgs=1154 cs=0 l=1 rx=0x7f71a0124540 tx=0x7f7190010ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:36:55.986 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:55.967+0000 7f718f7fe700 1 --2- 172.21.15.42:0/1249243609 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f718403ac50 0x7f718403d160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:36:55.987 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:55.967+0000 7f718f7fe700 1 -- 172.21.15.42:0/1249243609 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 7923+0+0 (secure 0 0 0) 0x7f719004ae40 con 0x7f71a008eeb0 2019-12-05T05:36:55.987 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:55.967+0000 7f719e59c700 1 --2- 172.21.15.42:0/1249243609 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f718403ac50 0x7f718403d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:36:55.987 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:55.967+0000 7f719e59c700 1 --2- 172.21.15.42:0/1249243609 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f718403ac50 0x7f718403d160 crc :-1 s=READY pgs=500 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:36:55.987 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:55.971+0000 7f71a4c1f700 1 -- 172.21.15.42:0/1249243609 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71a005d110 con 0x7f71a008eeb0 2019-12-05T05:36:55.987 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:55.979+0000 7f719ed9d700 1 --2- 172.21.15.42:0/1249243609 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f71a008eeb0 0x7f71a0113270 secure :-1 s=THROTTLE_DONE pgs=1154 cs=0 l=1 rx=0x7f71a0124540 tx=0x7f7190010ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:36:55.987 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:55.979+0000 7f718f7fe700 1 -- 172.21.15.42:0/1249243609 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f7190031e60 con 0x7f71a008eeb0 2019-12-05T05:36:56.145 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:56.139+0000 7f71a4c1f700 1 -- 172.21.15.42:0/1249243609 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f71a005b8c0 con 0x7f71a008eeb0 2019-12-05T05:36:56.146 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:56.139+0000 7f719ed9d700 1 --2- 172.21.15.42:0/1249243609 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f71a008eeb0 0x7f71a0113270 secure :-1 s=THROTTLE_DONE pgs=1154 cs=0 l=1 rx=0x7f71a0124540 tx=0x7f7190010ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:36:56.146 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:56.139+0000 7f718f7fe700 1 -- 172.21.15.42:0/1249243609 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 143 v143) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f71900320c0 con 0x7f71a008eeb0 2019-12-05T05:36:56.146 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 143 2019-12-05T05:36:56.146 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:36:56.147 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":143,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/3838877789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3838877789},{"type":"v1","addr":"172.21.15.42:6835","nonce":3838877789}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":143}],"filesystems":[{"mdsmap":{"epoch":142,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:34:39.042388+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7221},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7221":{"gid":7221,"name":"b","rank":0,"incarnation":139,"state":"up:active","state_seq":27,"addr":"172.21.15.40:6833/3789641053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3789641053},{"type":"v1","addr":"172.21.15.40:6833","nonce":3789641053}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:36:56.149 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:56.143+0000 7f71a4c1f700 1 -- 172.21.15.42:0/1249243609 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f718403ac50 msgr2=0x7f718403d160 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:36:56.150 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:56.143+0000 7f71a4c1f700 1 --2- 172.21.15.42:0/1249243609 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f718403ac50 0x7f718403d160 crc :-1 s=READY pgs=500 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:36:56.150 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:56.143+0000 7f71a4c1f700 1 -- 172.21.15.42:0/1249243609 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f71a008eeb0 msgr2=0x7f71a0113270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:36:56.150 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:56.143+0000 7f71a4c1f700 1 --2- 172.21.15.42:0/1249243609 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f71a008eeb0 0x7f71a0113270 secure :-1 s=READY pgs=1154 cs=0 l=1 rx=0x7f71a0124540 tx=0x7f7190010ae0).stop 2019-12-05T05:36:56.151 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:56.143+0000 7f71a4c1f700 1 -- 172.21.15.42:0/1249243609 shutdown_connections 2019-12-05T05:36:56.151 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:56.143+0000 7f71a4c1f700 1 --2- 172.21.15.42:0/1249243609 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f718403ac50 0x7f718403d160 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:36:56.151 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:56.143+0000 7f71a4c1f700 1 --2- 172.21.15.42:0/1249243609 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f71a007d940 0x7f71a007dd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:36:56.152 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:56.143+0000 7f71a4c1f700 1 --2- 172.21.15.42:0/1249243609 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f71a008eeb0 0x7f71a0113270 unknown :-1 s=CLOSED pgs=1154 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:36:56.152 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:56.143+0000 7f71a4c1f700 1 -- 172.21.15.42:0/1249243609 shutdown_connections 2019-12-05T05:36:56.153 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:56.143+0000 7f71a4c1f700 1 -- 172.21.15.42:0/1249243609 wait complete. 2019-12-05T05:36:56.153 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:56.143+0000 7f71a4c1f700 1 -- 172.21.15.42:0/1249243609 >> 172.21.15.42:0/1249243609 conn(0x7f71a00a7740 msgr2=0x7f71a0116220 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:36:56.159 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T05:36:58.161 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:36:58.267 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:58.255+0000 7ff06dea8700 1 Processor -- start 2019-12-05T05:36:58.268 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:58.255+0000 7ff06dea8700 1 -- start start 2019-12-05T05:36:58.268 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:58.255+0000 7ff06dea8700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff0680a7740 0x7ff06808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:36:58.268 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:58.255+0000 7ff06dea8700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff06809fae0 0x7ff06807eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:36:58.269 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:58.255+0000 7ff06dea8700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0680a0710 con 0x7ff06807e720 2019-12-05T05:36:58.269 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:58.255+0000 7ff06dea8700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0680a0b00 con 0x7ff0680a7740 2019-12-05T05:36:58.270 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:58.255+0000 7ff06dea8700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff06808e5b0 con 0x7ff06809fae0 2019-12-05T05:36:58.270 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:58.255+0000 7ff067fff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff0680a7740 0x7ff06808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:36:58.270 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:58.255+0000 7ff0677fe700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7ff06807e720 0x7ff06809f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:46818/0 (socket says 172.21.15.42:46818) 2019-12-05T05:36:58.270 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:58.255+0000 7ff067fff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff0680a7740 0x7ff06808df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:37340/0 (socket says 172.21.15.42:37340) 2019-12-05T05:36:58.270 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:58.255+0000 7ff06cc46700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff06809fae0 0x7ff06807eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:36:58.270 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:58.255+0000 7ff0677fe700 1 -- 172.21.15.42:0/176070715 learned_addr learned my addr 172.21.15.42:0/176070715 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:36:58.270 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:58.255+0000 7ff066ffd700 1 -- 172.21.15.42:0/176070715 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2361492065 0 0) 0x7ff0680a0710 con 0x7ff06807e720 2019-12-05T05:36:58.271 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:58.255+0000 7ff066ffd700 1 -- 172.21.15.42:0/176070715 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0540030b0 con 0x7ff06807e720 2019-12-05T05:36:58.271 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:58.255+0000 7ff067fff700 1 -- 172.21.15.42:0/176070715 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff06809fae0 msgr2=0x7ff06807eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:36:58.271 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:58.255+0000 7ff067fff700 1 --2- 172.21.15.42:0/176070715 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff06809fae0 0x7ff06807eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:36:58.271 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:58.255+0000 7ff067fff700 1 -- 172.21.15.42:0/176070715 >> v1:172.21.15.40:6789/0 conn(0x7ff06807e720 legacy=0x7ff06809f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:36:58.271 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:58.255+0000 7ff067fff700 1 -- 172.21.15.42:0/176070715 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff06807f0a0 con 0x7ff0680a7740 2019-12-05T05:36:58.271 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:58.255+0000 7ff06cc46700 1 --2- 172.21.15.42:0/176070715 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff06809fae0 0x7ff06807eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:36:58.272 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:58.255+0000 7ff067fff700 1 --2- 172.21.15.42:0/176070715 >> v2:172.21.15.42:3300/0 conn(0x7ff0680a7740 0x7ff06808df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7ff058008790 tx=0x7ff05800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:36:58.272 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:58.255+0000 7ff067fff700 1 --2- 172.21.15.42:0/176070715 >> v2:172.21.15.42:3300/0 conn(0x7ff0680a7740 0x7ff06808df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7ff058008790 tx=0x7ff05800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:36:58.272 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:58.255+0000 7ff067fff700 1 --2- 172.21.15.42:0/176070715 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff0680a7740 0x7ff06808df80 secure :-1 s=READY pgs=1155 cs=0 l=1 rx=0x7ff058008790 tx=0x7ff05800fc70).ready entity=mon.1 client_cookie=7e8b7a5d73f10193 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:36:58.273 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:58.255+0000 7ff067fff700 1 --2- 172.21.15.42:0/176070715 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff0680a7740 0x7ff06808df80 secure :-1 s=THROTTLE_DONE pgs=1155 cs=0 l=1 rx=0x7ff058008790 tx=0x7ff05800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:36:58.273 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:58.255+0000 7ff066ffd700 1 -- 172.21.15.42:0/176070715 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff058008bb0 con 0x7ff0680a7740 2019-12-05T05:36:58.273 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:58.255+0000 7ff067fff700 1 --2- 172.21.15.42:0/176070715 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff0680a7740 0x7ff06808df80 secure :-1 s=THROTTLE_DONE pgs=1155 cs=0 l=1 rx=0x7ff058008790 tx=0x7ff05800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:36:58.273 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:58.259+0000 7ff067fff700 1 --2- 172.21.15.42:0/176070715 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff0680a7740 0x7ff06808df80 secure :-1 s=THROTTLE_DONE pgs=1155 cs=0 l=1 rx=0x7ff058008790 tx=0x7ff05800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:36:58.273 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:58.259+0000 7ff066ffd700 1 -- 172.21.15.42:0/176070715 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0580087d0 con 0x7ff0680a7740 2019-12-05T05:36:58.274 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:58.259+0000 7ff066ffd700 1 -- 172.21.15.42:0/176070715 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff058011860 con 0x7ff0680a7740 2019-12-05T05:36:58.277 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:58.259+0000 7ff06dea8700 1 -- 172.21.15.42:0/176070715 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff0680a7740 msgr2=0x7ff06808df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:36:58.277 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:58.259+0000 7ff06dea8700 1 --2- 172.21.15.42:0/176070715 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff0680a7740 0x7ff06808df80 secure :-1 s=READY pgs=1155 cs=0 l=1 rx=0x7ff058008790 tx=0x7ff05800fc70).stop 2019-12-05T05:36:58.277 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:58.259+0000 7ff06dea8700 1 -- 172.21.15.42:0/176070715 shutdown_connections 2019-12-05T05:36:58.277 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:58.259+0000 7ff06dea8700 1 --2- 172.21.15.42:0/176070715 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff06809fae0 0x7ff06807eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:36:58.277 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:58.259+0000 7ff06dea8700 1 --2- 172.21.15.42:0/176070715 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff0680a7740 0x7ff06808df80 unknown :-1 s=CLOSED pgs=1155 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:36:58.277 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:58.259+0000 7ff06dea8700 1 -- 172.21.15.42:0/176070715 shutdown_connections 2019-12-05T05:36:58.278 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:58.259+0000 7ff06dea8700 1 -- 172.21.15.42:0/176070715 wait complete. 2019-12-05T05:36:58.278 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:58.259+0000 7ff06dea8700 1 -- 172.21.15.42:0/176070715 >> 172.21.15.42:0/176070715 conn(0x7ff0680aabe0 msgr2=0x7ff0680a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:36:58.278 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:58.259+0000 7ff06dea8700 1 Processor -- start 2019-12-05T05:36:58.278 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:58.259+0000 7ff06dea8700 1 -- start start 2019-12-05T05:36:58.278 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:58.259+0000 7ff06dea8700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff068130d20 0x7ff068134020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:36:58.278 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:58.259+0000 7ff06dea8700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff068134570 con 0x7ff068116530 2019-12-05T05:36:58.278 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:58.259+0000 7ff06dea8700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff06811fdb0 con 0x7ff068133bf0 2019-12-05T05:36:58.279 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:58.259+0000 7ff06dea8700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff068123a60 con 0x7ff068130d20 2019-12-05T05:36:58.280 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:58.259+0000 7ff0677fe700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7ff068133bf0 0x7ff068130520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:37384/0 (socket says 172.21.15.42:37384) 2019-12-05T05:36:58.280 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:58.259+0000 7ff0677fe700 1 -- 172.21.15.42:0/1628244350 learned_addr learned my addr 172.21.15.42:0/1628244350 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:36:58.280 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:58.259+0000 7ff06cc46700 1 --2- 172.21.15.42:0/1628244350 >> v2:172.21.15.40:3300/0 conn(0x7ff068130d20 0x7ff068134020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:36:58.280 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:58.263+0000 7ff064ff9700 1 -- 172.21.15.42:0/1628244350 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3842104504 0 0) 0x7ff06811fdb0 con 0x7ff068133bf0 2019-12-05T05:36:58.280 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:58.263+0000 7ff064ff9700 1 -- 172.21.15.42:0/1628244350 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff04c0030b0 con 0x7ff068133bf0 2019-12-05T05:36:58.283 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:58.263+0000 7ff06cc46700 1 -- 172.21.15.42:0/1628244350 >> v1:172.21.15.42:6789/0 conn(0x7ff068133bf0 legacy=0x7ff068130520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:36:58.283 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:58.263+0000 7ff064ff9700 1 -- 172.21.15.42:0/1628244350 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 1735825798 0 0) 0x7ff04c0030b0 con 0x7ff068133bf0 2019-12-05T05:36:58.283 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:58.263+0000 7ff06cc46700 1 -- 172.21.15.42:0/1628244350 >> v1:172.21.15.40:6789/0 conn(0x7ff068116530 legacy=0x7ff068123150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:36:58.283 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:58.263+0000 7ff06cc46700 1 -- 172.21.15.42:0/1628244350 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff06807a330 con 0x7ff068130d20 2019-12-05T05:36:58.283 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:58.263+0000 7ff06cc46700 1 --2- 172.21.15.42:0/1628244350 >> v2:172.21.15.40:3300/0 conn(0x7ff068130d20 0x7ff068134020 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7ff058008370 tx=0x7ff05800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:36:58.283 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:58.263+0000 7ff06cc46700 1 --2- 172.21.15.42:0/1628244350 >> v2:172.21.15.40:3300/0 conn(0x7ff068130d20 0x7ff068134020 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7ff058008370 tx=0x7ff05800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:36:58.284 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:58.263+0000 7ff06cc46700 1 --2- 172.21.15.42:0/1628244350 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff068130d20 0x7ff068134020 secure :-1 s=READY pgs=908 cs=0 l=1 rx=0x7ff058008370 tx=0x7ff05800fc70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:36:58.284 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:58.263+0000 7ff06cc46700 1 --2- 172.21.15.42:0/1628244350 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff068130d20 0x7ff068134020 secure :-1 s=THROTTLE_DONE pgs=908 cs=0 l=1 rx=0x7ff058008370 tx=0x7ff05800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:36:58.284 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:58.263+0000 7ff064ff9700 1 -- 172.21.15.42:0/1628244350 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff050001d10 con 0x7ff068130d20 2019-12-05T05:36:58.284 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:58.263+0000 7ff06dea8700 1 -- 172.21.15.42:0/1628244350 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff068078310 con 0x7ff068130d20 2019-12-05T05:36:58.284 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:58.263+0000 7ff06cc46700 1 --2- 172.21.15.42:0/1628244350 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff068130d20 0x7ff068134020 secure :-1 s=THROTTLE_DONE pgs=908 cs=0 l=1 rx=0x7ff058008370 tx=0x7ff05800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:36:58.284 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:58.263+0000 7ff064ff9700 1 -- 172.21.15.42:0/1628244350 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff050001b20 con 0x7ff068130d20 2019-12-05T05:36:58.284 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:58.263+0000 7ff06cc46700 1 --2- 172.21.15.42:0/1628244350 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff068130d20 0x7ff068134020 secure :-1 s=THROTTLE_DONE pgs=908 cs=0 l=1 rx=0x7ff058008370 tx=0x7ff05800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:36:58.285 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:58.263+0000 7ff06dea8700 1 -- 172.21.15.42:0/1628244350 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff068078840 con 0x7ff068130d20 2019-12-05T05:36:58.285 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:58.263+0000 7ff064ff9700 1 -- 172.21.15.42:0/1628244350 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff060005b10 con 0x7ff068130d20 2019-12-05T05:36:58.285 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:58.263+0000 7ff06cc46700 1 --2- 172.21.15.42:0/1628244350 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff068130d20 0x7ff068134020 secure :-1 s=THROTTLE_DONE pgs=908 cs=0 l=1 rx=0x7ff058008370 tx=0x7ff05800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:36:58.285 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:58.263+0000 7ff064ff9700 1 -- 172.21.15.42:0/1628244350 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7ff060007140 con 0x7ff068130d20 2019-12-05T05:36:58.285 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:58.263+0000 7ff06cc46700 1 --2- 172.21.15.42:0/1628244350 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff068130d20 0x7ff068134020 secure :-1 s=THROTTLE_DONE pgs=908 cs=0 l=1 rx=0x7ff058008370 tx=0x7ff05800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:36:58.285 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:58.263+0000 7ff064ff9700 1 --2- 172.21.15.42:0/1628244350 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff04c03b1c0 0x7ff04c03d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:36:58.285 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:58.267+0000 7ff064ff9700 1 -- 172.21.15.42:0/1628244350 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 7923+0+0 (secure 0 0 0) 0x7ff06004a7e0 con 0x7ff068130d20 2019-12-05T05:36:58.286 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:58.267+0000 7ff067fff700 1 --2- 172.21.15.42:0/1628244350 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff04c03b1c0 0x7ff04c03d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:36:58.286 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:58.267+0000 7ff067fff700 1 --2- 172.21.15.42:0/1628244350 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff04c03b1c0 0x7ff04c03d6d0 crc :-1 s=READY pgs=501 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:36:58.286 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:58.271+0000 7ff06dea8700 1 -- 172.21.15.42:0/1628244350 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff06805d110 con 0x7ff068130d20 2019-12-05T05:36:58.287 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:58.279+0000 7ff06cc46700 1 --2- 172.21.15.42:0/1628244350 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff068130d20 0x7ff068134020 secure :-1 s=THROTTLE_DONE pgs=908 cs=0 l=1 rx=0x7ff058008370 tx=0x7ff05800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:36:58.287 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:58.279+0000 7ff064ff9700 1 -- 172.21.15.42:0/1628244350 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7ff060031840 con 0x7ff068130d20 2019-12-05T05:36:58.457 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:58.451+0000 7ff06dea8700 1 -- 172.21.15.42:0/1628244350 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff06805b8c0 con 0x7ff068130d20 2019-12-05T05:36:58.457 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:58.451+0000 7ff06cc46700 1 --2- 172.21.15.42:0/1628244350 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff068130d20 0x7ff068134020 secure :-1 s=THROTTLE_DONE pgs=908 cs=0 l=1 rx=0x7ff058008370 tx=0x7ff05800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:36:58.457 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:58.451+0000 7ff064ff9700 1 -- 172.21.15.42:0/1628244350 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 143 v143) v1 ==== 95+0+2112 (secure 0 0 0) 0x7ff060005d60 con 0x7ff068130d20 2019-12-05T05:36:58.460 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 143 2019-12-05T05:36:58.460 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:36:58.460 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":143,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/3838877789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3838877789},{"type":"v1","addr":"172.21.15.42:6835","nonce":3838877789}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":143}],"filesystems":[{"mdsmap":{"epoch":142,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:34:39.042388+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7221},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7221":{"gid":7221,"name":"b","rank":0,"incarnation":139,"state":"up:active","state_seq":27,"addr":"172.21.15.40:6833/3789641053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3789641053},{"type":"v1","addr":"172.21.15.40:6833","nonce":3789641053}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:36:58.464 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:58.455+0000 7ff06dea8700 1 -- 172.21.15.42:0/1628244350 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff04c03b1c0 msgr2=0x7ff04c03d6d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:36:58.464 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:58.455+0000 7ff06dea8700 1 --2- 172.21.15.42:0/1628244350 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff04c03b1c0 0x7ff04c03d6d0 crc :-1 s=READY pgs=501 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:36:58.464 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:58.455+0000 7ff06dea8700 1 -- 172.21.15.42:0/1628244350 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff068130d20 msgr2=0x7ff068134020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:36:58.464 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:58.455+0000 7ff06dea8700 1 --2- 172.21.15.42:0/1628244350 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff068130d20 0x7ff068134020 secure :-1 s=READY pgs=908 cs=0 l=1 rx=0x7ff058008370 tx=0x7ff05800fc70).stop 2019-12-05T05:36:58.464 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:58.459+0000 7ff06dea8700 1 -- 172.21.15.42:0/1628244350 shutdown_connections 2019-12-05T05:36:58.465 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:58.459+0000 7ff06dea8700 1 --2- 172.21.15.42:0/1628244350 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff04c03b1c0 0x7ff04c03d6d0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:36:58.465 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:58.459+0000 7ff06dea8700 1 --2- 172.21.15.42:0/1628244350 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff068130d20 0x7ff068134020 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:36:58.465 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:58.459+0000 7ff06dea8700 1 -- 172.21.15.42:0/1628244350 shutdown_connections 2019-12-05T05:36:58.465 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:58.459+0000 7ff06dea8700 1 -- 172.21.15.42:0/1628244350 wait complete. 2019-12-05T05:36:58.465 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:36:58.459+0000 7ff06dea8700 1 -- 172.21.15.42:0/1628244350 >> 172.21.15.42:0/1628244350 conn(0x7ff0680aabe0 msgr2=0x7ff0680a9640 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:36:58.479 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T05:37:00.486 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:37:00.591 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:00.579+0000 7f7c902e3700 1 Processor -- start 2019-12-05T05:37:00.592 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:00.579+0000 7f7c902e3700 1 -- start start 2019-12-05T05:37:00.592 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:00.583+0000 7f7c902e3700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7c8808df80 0x7f7c8809f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:37:00.592 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:00.583+0000 7f7c902e3700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f7c8809f830 0x7f7c8814efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:37:00.592 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:00.583+0000 7f7c902e3700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c8809fc60 con 0x7f7c880a7740 2019-12-05T05:37:00.592 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:00.583+0000 7f7c902e3700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c880a0b00 con 0x7f7c8809f830 2019-12-05T05:37:00.593 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:00.583+0000 7f7c902e3700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c8808e5b0 con 0x7f7c8808df80 2019-12-05T05:37:00.593 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:00.583+0000 7f7c8f081700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f7c8809f830 0x7f7c8814efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:37:00.593 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:00.583+0000 7f7c8f081700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f7c8809f830 0x7f7c8814efe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:37356/0 (socket says 172.21.15.42:37356) 2019-12-05T05:37:00.593 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:00.583+0000 7f7c8f081700 1 -- 172.21.15.42:0/3256327599 learned_addr learned my addr 172.21.15.42:0/3256327599 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:37:00.593 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:00.583+0000 7f7c8e07f700 1 --2- 172.21.15.42:0/3256327599 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7c8808df80 0x7f7c8809f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:37:00.593 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:00.583+0000 7f7c8f081700 1 -- 172.21.15.42:0/3256327599 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7c8808df80 msgr2=0x7f7c8809f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:37:00.593 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:00.583+0000 7f7c8f081700 1 --2- 172.21.15.42:0/3256327599 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7c8808df80 0x7f7c8809f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:37:00.594 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:00.583+0000 7f7c8f081700 1 -- 172.21.15.42:0/3256327599 >> v1:172.21.15.40:6789/0 conn(0x7f7c880a7740 legacy=0x7f7c8807e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:37:00.594 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:00.583+0000 7f7c8f081700 1 -- 172.21.15.42:0/3256327599 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c880a0710 con 0x7f7c8809f830 2019-12-05T05:37:00.594 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:00.583+0000 7f7c8e07f700 1 --2- 172.21.15.42:0/3256327599 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7c8808df80 0x7f7c8809f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T05:37:00.594 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:00.583+0000 7f7c8f081700 1 --2- 172.21.15.42:0/3256327599 >> v2:172.21.15.42:3300/0 conn(0x7f7c8809f830 0x7f7c8814efe0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f7c80008b10 tx=0x7f7c800100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:00.594 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:00.583+0000 7f7c8f081700 1 --2- 172.21.15.42:0/3256327599 >> v2:172.21.15.42:3300/0 conn(0x7f7c8809f830 0x7f7c8814efe0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f7c80008b10 tx=0x7f7c800100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:00.594 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:00.583+0000 7f7c8f081700 1 --2- 172.21.15.42:0/3256327599 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7c8809f830 0x7f7c8814efe0 secure :-1 s=READY pgs=1157 cs=0 l=1 rx=0x7f7c80008b10 tx=0x7f7c800100b0).ready entity=mon.1 client_cookie=d9cca4aaad6e8a6 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:37:00.594 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:00.583+0000 7f7c8f081700 1 --2- 172.21.15.42:0/3256327599 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7c8809f830 0x7f7c8814efe0 secure :-1 s=THROTTLE_DONE pgs=1157 cs=0 l=1 rx=0x7f7c80008b10 tx=0x7f7c800100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:00.594 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:00.583+0000 7f7c8d87e700 1 -- 172.21.15.42:0/3256327599 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7c80010da0 con 0x7f7c8809f830 2019-12-05T05:37:00.595 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:00.583+0000 7f7c8f081700 1 --2- 172.21.15.42:0/3256327599 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7c8809f830 0x7f7c8814efe0 secure :-1 s=THROTTLE_DONE pgs=1157 cs=0 l=1 rx=0x7f7c80008b10 tx=0x7f7c800100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:00.595 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:00.583+0000 7f7c8f081700 1 --2- 172.21.15.42:0/3256327599 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7c8809f830 0x7f7c8814efe0 secure :-1 s=THROTTLE_DONE pgs=1157 cs=0 l=1 rx=0x7f7c80008b10 tx=0x7f7c800100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:00.595 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:00.583+0000 7f7c8d87e700 1 -- 172.21.15.42:0/3256327599 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c800109c0 con 0x7f7c8809f830 2019-12-05T05:37:00.595 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:00.583+0000 7f7c8d87e700 1 -- 172.21.15.42:0/3256327599 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7c800114a0 con 0x7f7c8809f830 2019-12-05T05:37:00.595 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:00.583+0000 7f7c902e3700 1 -- 172.21.15.42:0/3256327599 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7c8809f830 msgr2=0x7f7c8814efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:37:00.595 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:00.583+0000 7f7c902e3700 1 --2- 172.21.15.42:0/3256327599 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7c8809f830 0x7f7c8814efe0 secure :-1 s=READY pgs=1157 cs=0 l=1 rx=0x7f7c80008b10 tx=0x7f7c800100b0).stop 2019-12-05T05:37:00.595 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:00.583+0000 7f7c902e3700 1 -- 172.21.15.42:0/3256327599 shutdown_connections 2019-12-05T05:37:00.596 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:00.583+0000 7f7c902e3700 1 --2- 172.21.15.42:0/3256327599 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7c8809f830 0x7f7c8814efe0 unknown :-1 s=CLOSED pgs=1157 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:37:00.596 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:00.583+0000 7f7c902e3700 1 --2- 172.21.15.42:0/3256327599 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7c8808df80 0x7f7c8809f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:37:00.596 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:00.583+0000 7f7c902e3700 1 -- 172.21.15.42:0/3256327599 shutdown_connections 2019-12-05T05:37:00.596 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:00.583+0000 7f7c902e3700 1 -- 172.21.15.42:0/3256327599 wait complete. 2019-12-05T05:37:00.596 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:00.583+0000 7f7c902e3700 1 -- 172.21.15.42:0/3256327599 >> 172.21.15.42:0/3256327599 conn(0x7f7c880aabe0 msgr2=0x7f7c880a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:37:00.605 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:00.587+0000 7f7c902e3700 1 Processor -- start 2019-12-05T05:37:00.606 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:00.587+0000 7f7c902e3700 1 -- start start 2019-12-05T05:37:00.606 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:00.587+0000 7f7c902e3700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7c8809f480 0x7f7c88124f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:37:00.606 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:00.587+0000 7f7c902e3700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7c88125460 0x7f7c880a3b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:37:00.606 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:00.587+0000 7f7c902e3700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f7c880a4050 0x7f7c88127d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:37:00.606 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:00.587+0000 7f7c902e3700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c8809f8b0 con 0x7f7c8809f480 2019-12-05T05:37:00.606 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:00.587+0000 7f7c902e3700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c8811a5b0 con 0x7f7c880a4050 2019-12-05T05:37:00.607 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:00.587+0000 7f7c902e3700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c8811a850 con 0x7f7c88125460 2019-12-05T05:37:00.607 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:00.587+0000 7f7c8f081700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f7c880a4050 0x7f7c88127d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:37:00.607 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:00.587+0000 7f7c8f081700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f7c880a4050 0x7f7c88127d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:37362/0 (socket says 172.21.15.42:37362) 2019-12-05T05:37:00.607 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:00.587+0000 7f7c8f081700 1 -- 172.21.15.42:0/174680675 learned_addr learned my addr 172.21.15.42:0/174680675 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:37:00.607 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:00.591+0000 7f7c8e07f700 1 --2- 172.21.15.42:0/174680675 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7c88125460 0x7f7c880a3b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:37:00.608 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:00.591+0000 7f7c8e880700 1 --2- 172.21.15.42:0/174680675 >> v2:172.21.15.40:3300/0 conn(0x7f7c8809f480 0x7f7c88124f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:37:00.608 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:00.591+0000 7f7c8f081700 1 -- 172.21.15.42:0/174680675 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7c88125460 msgr2=0x7f7c880a3b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:37:00.608 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:00.591+0000 7f7c8f081700 1 --2- 172.21.15.42:0/174680675 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7c88125460 0x7f7c880a3b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:37:00.608 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:00.591+0000 7f7c8f081700 1 -- 172.21.15.42:0/174680675 >> v2:172.21.15.40:3300/0 conn(0x7f7c8809f480 msgr2=0x7f7c88124f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:37:00.608 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:00.591+0000 7f7c8f081700 1 --2- 172.21.15.42:0/174680675 >> v2:172.21.15.40:3300/0 conn(0x7f7c8809f480 0x7f7c88124f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:37:00.609 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:00.591+0000 7f7c8f081700 1 -- 172.21.15.42:0/174680675 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c88124740 con 0x7f7c880a4050 2019-12-05T05:37:00.609 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:00.591+0000 7f7c8e880700 1 --2- 172.21.15.42:0/174680675 >> v2:172.21.15.40:3300/0 conn(0x7f7c8809f480 0x7f7c88124f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T05:37:00.609 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:00.591+0000 7f7c8f081700 1 --2- 172.21.15.42:0/174680675 >> v2:172.21.15.42:3300/0 conn(0x7f7c880a4050 0x7f7c88127d20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f7c8811a770 tx=0x7f7c80008970).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:00.609 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:00.591+0000 7f7c8f081700 1 --2- 172.21.15.42:0/174680675 >> v2:172.21.15.42:3300/0 conn(0x7f7c880a4050 0x7f7c88127d20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f7c8811a770 tx=0x7f7c80008970).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:00.609 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:00.591+0000 7f7c8f081700 1 --2- 172.21.15.42:0/174680675 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7c880a4050 0x7f7c88127d20 secure :-1 s=READY pgs=1158 cs=0 l=1 rx=0x7f7c8811a770 tx=0x7f7c80008970).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:37:00.610 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:00.591+0000 7f7c8f081700 1 --2- 172.21.15.42:0/174680675 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7c880a4050 0x7f7c88127d20 secure :-1 s=THROTTLE_DONE pgs=1158 cs=0 l=1 rx=0x7f7c8811a770 tx=0x7f7c80008970).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:00.611 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:00.591+0000 7f7c8f081700 1 --2- 172.21.15.42:0/174680675 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7c880a4050 0x7f7c88127d20 secure :-1 s=THROTTLE_DONE pgs=1158 cs=0 l=1 rx=0x7f7c8811a770 tx=0x7f7c80008970).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:00.611 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:00.591+0000 7f7c7f7fe700 1 -- 172.21.15.42:0/174680675 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7c80011c80 con 0x7f7c880a4050 2019-12-05T05:37:00.611 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:00.591+0000 7f7c8f081700 1 --2- 172.21.15.42:0/174680675 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7c880a4050 0x7f7c88127d20 secure :-1 s=THROTTLE_DONE pgs=1158 cs=0 l=1 rx=0x7f7c8811a770 tx=0x7f7c80008970).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:00.611 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:00.591+0000 7f7c7f7fe700 1 -- 172.21.15.42:0/174680675 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c800110c0 con 0x7f7c880a4050 2019-12-05T05:37:00.611 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:00.591+0000 7f7c902e3700 1 -- 172.21.15.42:0/174680675 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c880a3740 con 0x7f7c880a4050 2019-12-05T05:37:00.612 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:00.591+0000 7f7c7f7fe700 1 -- 172.21.15.42:0/174680675 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7c80005ca0 con 0x7f7c880a4050 2019-12-05T05:37:00.612 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:00.591+0000 7f7c902e3700 1 -- 172.21.15.42:0/174680675 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c8811a130 con 0x7f7c880a4050 2019-12-05T05:37:00.612 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:00.591+0000 7f7c8f081700 1 --2- 172.21.15.42:0/174680675 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7c880a4050 0x7f7c88127d20 secure :-1 s=THROTTLE_DONE pgs=1158 cs=0 l=1 rx=0x7f7c8811a770 tx=0x7f7c80008970).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:00.613 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:00.591+0000 7f7c7f7fe700 1 -- 172.21.15.42:0/174680675 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f7c8000f280 con 0x7f7c880a4050 2019-12-05T05:37:00.613 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:00.591+0000 7f7c8f081700 1 --2- 172.21.15.42:0/174680675 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7c880a4050 0x7f7c88127d20 secure :-1 s=THROTTLE_DONE pgs=1158 cs=0 l=1 rx=0x7f7c8811a770 tx=0x7f7c80008970).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:00.613 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:00.591+0000 7f7c7f7fe700 1 --2- 172.21.15.42:0/174680675 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f7c7003ae40 0x7f7c7003d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:37:00.613 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:00.591+0000 7f7c7f7fe700 1 -- 172.21.15.42:0/174680675 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 7923+0+0 (secure 0 0 0) 0x7f7c8004b640 con 0x7f7c880a4050 2019-12-05T05:37:00.614 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:00.591+0000 7f7c8e880700 1 --2- 172.21.15.42:0/174680675 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f7c7003ae40 0x7f7c7003d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:37:00.615 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:00.591+0000 7f7c8e880700 1 --2- 172.21.15.42:0/174680675 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f7c7003ae40 0x7f7c7003d350 crc :-1 s=READY pgs=502 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:37:00.615 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:00.595+0000 7f7c902e3700 1 -- 172.21.15.42:0/174680675 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c8805d110 con 0x7f7c880a4050 2019-12-05T05:37:00.615 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:00.603+0000 7f7c8f081700 1 --2- 172.21.15.42:0/174680675 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7c880a4050 0x7f7c88127d20 secure :-1 s=THROTTLE_DONE pgs=1158 cs=0 l=1 rx=0x7f7c8811a770 tx=0x7f7c80008970).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:00.616 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:00.603+0000 7f7c7f7fe700 1 -- 172.21.15.42:0/174680675 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f7c800334d0 con 0x7f7c880a4050 2019-12-05T05:37:00.779 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:00.771+0000 7f7c902e3700 1 -- 172.21.15.42:0/174680675 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7c8805b8c0 con 0x7f7c880a4050 2019-12-05T05:37:00.779 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:00.771+0000 7f7c8f081700 1 --2- 172.21.15.42:0/174680675 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7c880a4050 0x7f7c88127d20 secure :-1 s=THROTTLE_DONE pgs=1158 cs=0 l=1 rx=0x7f7c8811a770 tx=0x7f7c80008970).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:00.780 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:00.771+0000 7f7c7f7fe700 1 -- 172.21.15.42:0/174680675 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 143 v143) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f7c800331d0 con 0x7f7c880a4050 2019-12-05T05:37:00.780 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 143 2019-12-05T05:37:00.781 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:37:00.781 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":143,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/3838877789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3838877789},{"type":"v1","addr":"172.21.15.42:6835","nonce":3838877789}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":143}],"filesystems":[{"mdsmap":{"epoch":142,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:34:39.042388+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7221},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7221":{"gid":7221,"name":"b","rank":0,"incarnation":139,"state":"up:active","state_seq":27,"addr":"172.21.15.40:6833/3789641053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3789641053},{"type":"v1","addr":"172.21.15.40:6833","nonce":3789641053}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:37:00.783 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:00.775+0000 7f7c902e3700 1 -- 172.21.15.42:0/174680675 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f7c7003ae40 msgr2=0x7f7c7003d350 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:37:00.784 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:00.775+0000 7f7c902e3700 1 --2- 172.21.15.42:0/174680675 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f7c7003ae40 0x7f7c7003d350 crc :-1 s=READY pgs=502 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:37:00.784 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:00.775+0000 7f7c902e3700 1 -- 172.21.15.42:0/174680675 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7c880a4050 msgr2=0x7f7c88127d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:37:00.784 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:00.775+0000 7f7c902e3700 1 --2- 172.21.15.42:0/174680675 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7c880a4050 0x7f7c88127d20 secure :-1 s=READY pgs=1158 cs=0 l=1 rx=0x7f7c8811a770 tx=0x7f7c80008970).stop 2019-12-05T05:37:00.785 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:00.775+0000 7f7c902e3700 1 -- 172.21.15.42:0/174680675 shutdown_connections 2019-12-05T05:37:00.785 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:00.775+0000 7f7c902e3700 1 --2- 172.21.15.42:0/174680675 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f7c7003ae40 0x7f7c7003d350 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:37:00.785 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:00.775+0000 7f7c902e3700 1 --2- 172.21.15.42:0/174680675 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7c880a4050 0x7f7c88127d20 unknown :-1 s=CLOSED pgs=1158 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:37:00.785 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:00.775+0000 7f7c902e3700 1 --2- 172.21.15.42:0/174680675 >> v2:172.21.15.40:3300/0 conn(0x7f7c8809f480 0x7f7c88124f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:37:00.786 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:00.775+0000 7f7c902e3700 1 --2- 172.21.15.42:0/174680675 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7c88125460 0x7f7c880a3b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:37:00.786 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:00.775+0000 7f7c902e3700 1 -- 172.21.15.42:0/174680675 shutdown_connections 2019-12-05T05:37:00.787 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:00.775+0000 7f7c902e3700 1 -- 172.21.15.42:0/174680675 wait complete. 2019-12-05T05:37:00.787 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:00.775+0000 7f7c902e3700 1 -- 172.21.15.42:0/174680675 >> 172.21.15.42:0/174680675 conn(0x7f7c880a99f0 msgr2=0x7f7c8807c450 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:37:00.793 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T05:37:02.793 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:37:02.910 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:02.895+0000 7fd5dee0e700 1 Processor -- start 2019-12-05T05:37:02.910 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:02.895+0000 7fd5dee0e700 1 -- start start 2019-12-05T05:37:02.910 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:02.895+0000 7fd5dee0e700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd5d80a7740 0x7fd5d808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:37:02.911 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:02.895+0000 7fd5dee0e700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5d80a0710 con 0x7fd5d809fae0 2019-12-05T05:37:02.911 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:02.895+0000 7fd5dee0e700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5d80a9ba0 con 0x7fd5d807e720 2019-12-05T05:37:02.911 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:02.895+0000 7fd5dee0e700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5d80a0b00 con 0x7fd5d80a7740 2019-12-05T05:37:02.911 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:02.895+0000 7fd5dcbaa700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fd5d807e720 0x7fd5d809f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:37404/0 (socket says 172.21.15.42:37404) 2019-12-05T05:37:02.911 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:02.895+0000 7fd5dcbaa700 1 -- 172.21.15.42:0/2862599941 learned_addr learned my addr 172.21.15.42:0/2862599941 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:37:02.911 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:02.899+0000 7fd5dd3ab700 1 --2- 172.21.15.42:0/2862599941 >> v2:172.21.15.40:3300/0 conn(0x7fd5d80a7740 0x7fd5d808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:37:02.911 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:02.899+0000 7fd5cbfff700 1 -- 172.21.15.42:0/2862599941 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3675171769 0 0) 0x7fd5d80a9ba0 con 0x7fd5d807e720 2019-12-05T05:37:02.912 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:02.899+0000 7fd5cbfff700 1 -- 172.21.15.42:0/2862599941 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5c40030b0 con 0x7fd5d807e720 2019-12-05T05:37:02.912 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:02.899+0000 7fd5cbfff700 1 -- 172.21.15.42:0/2862599941 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 951225955 0 0) 0x7fd5d80a0710 con 0x7fd5d809fae0 2019-12-05T05:37:02.912 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:02.899+0000 7fd5cbfff700 1 -- 172.21.15.42:0/2862599941 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5d80a9ba0 con 0x7fd5d809fae0 2019-12-05T05:37:02.912 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:02.899+0000 7fd5cbfff700 1 -- 172.21.15.42:0/2862599941 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1253166374 0 0) 0x7fd5c40030b0 con 0x7fd5d807e720 2019-12-05T05:37:02.912 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:02.899+0000 7fd5cbfff700 1 -- 172.21.15.42:0/2862599941 >> v2:172.21.15.40:3300/0 conn(0x7fd5d80a7740 msgr2=0x7fd5d808df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:37:02.912 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:02.899+0000 7fd5cbfff700 1 --2- 172.21.15.42:0/2862599941 >> v2:172.21.15.40:3300/0 conn(0x7fd5d80a7740 0x7fd5d808df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:37:02.912 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:02.899+0000 7fd5cbfff700 1 -- 172.21.15.42:0/2862599941 >> v1:172.21.15.40:6789/0 conn(0x7fd5d809fae0 legacy=0x7fd5d814efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:37:02.913 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:02.899+0000 7fd5cbfff700 1 -- 172.21.15.42:0/2862599941 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5d807eb50 con 0x7fd5d807e720 2019-12-05T05:37:02.913 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:02.899+0000 7fd5cbfff700 1 -- 172.21.15.42:0/2862599941 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7fd5cc001750 con 0x7fd5d807e720 2019-12-05T05:37:02.913 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:02.899+0000 7fd5cbfff700 1 -- 172.21.15.42:0/2862599941 >> v1:172.21.15.42:6789/0 conn(0x7fd5d807e720 legacy=0x7fd5d809f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:37:02.913 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:02.899+0000 7fd5cbfff700 1 --2- 172.21.15.42:0/2862599941 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd5c4003960 0x7fd5c4003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:37:02.913 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:02.899+0000 7fd5cbfff700 1 --2- 172.21.15.42:0/2862599941 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd5c4004470 0x7fd5c4004940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:37:02.913 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:02.899+0000 7fd5cbfff700 1 --2- 172.21.15.42:0/2862599941 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd5c4004fa0 0x7fd5c404f700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:37:02.913 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:02.899+0000 7fd5cbfff700 1 -- 172.21.15.42:0/2862599941 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5d80a0b00 con 0x7fd5c4004470 2019-12-05T05:37:02.914 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:02.899+0000 7fd5cbfff700 1 -- 172.21.15.42:0/2862599941 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5c404fec0 con 0x7fd5c4004fa0 2019-12-05T05:37:02.914 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:02.899+0000 7fd5cbfff700 1 -- 172.21.15.42:0/2862599941 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5c4050100 con 0x7fd5c4003960 2019-12-05T05:37:02.914 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:02.899+0000 7fd5dcbaa700 1 --2- 172.21.15.42:0/2862599941 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd5c4004470 0x7fd5c4004940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:37:02.914 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:02.899+0000 7fd5dd3ab700 1 --2- 172.21.15.42:0/2862599941 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd5c4003960 0x7fd5c4003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:37:02.914 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:02.899+0000 7fd5ddbac700 1 --2- 172.21.15.42:0/2862599941 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd5c4004fa0 0x7fd5c404f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:37:02.914 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:02.899+0000 7fd5dcbaa700 1 -- 172.21.15.42:0/2862599941 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd5c4003960 msgr2=0x7fd5c4003db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:37:02.914 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:02.899+0000 7fd5dcbaa700 1 --2- 172.21.15.42:0/2862599941 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd5c4003960 0x7fd5c4003db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:37:02.915 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:02.899+0000 7fd5dcbaa700 1 -- 172.21.15.42:0/2862599941 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd5c4004fa0 msgr2=0x7fd5c404f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:37:02.915 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:02.899+0000 7fd5dcbaa700 1 --2- 172.21.15.42:0/2862599941 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd5c4004fa0 0x7fd5c404f700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:37:02.915 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:02.899+0000 7fd5dcbaa700 1 -- 172.21.15.42:0/2862599941 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5c4050400 con 0x7fd5c4004470 2019-12-05T05:37:02.915 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:02.899+0000 7fd5ddbac700 1 --2- 172.21.15.42:0/2862599941 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd5c4004fa0 0x7fd5c404f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:37:02.915 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:02.899+0000 7fd5ddbac700 1 -- 172.21.15.42:0/2862599941 reap_dead start 2019-12-05T05:37:02.915 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:02.899+0000 7fd5dcbaa700 1 --2- 172.21.15.42:0/2862599941 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd5c4004470 0x7fd5c4004940 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fd5cc001520 tx=0x7fd5cc00c090).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:02.915 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:02.899+0000 7fd5dcbaa700 1 --2- 172.21.15.42:0/2862599941 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd5c4004470 0x7fd5c4004940 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fd5cc001520 tx=0x7fd5cc00c090).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:02.916 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:02.899+0000 7fd5dcbaa700 1 --2- 172.21.15.42:0/2862599941 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd5c4004470 0x7fd5c4004940 secure :-1 s=READY pgs=911 cs=0 l=1 rx=0x7fd5cc001520 tx=0x7fd5cc00c090).ready entity=mon.0 client_cookie=95adff0ecdd2cc6a server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:37:02.916 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:02.899+0000 7fd5dcbaa700 1 --2- 172.21.15.42:0/2862599941 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd5c4004470 0x7fd5c4004940 secure :-1 s=THROTTLE_DONE pgs=911 cs=0 l=1 rx=0x7fd5cc001520 tx=0x7fd5cc00c090).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:02.916 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:02.899+0000 7fd5dcbaa700 1 --2- 172.21.15.42:0/2862599941 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd5c4004470 0x7fd5c4004940 secure :-1 s=THROTTLE_DONE pgs=911 cs=0 l=1 rx=0x7fd5cc001520 tx=0x7fd5cc00c090).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:02.916 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:02.899+0000 7fd5cbfff700 1 -- 172.21.15.42:0/2862599941 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd5cc00ead0 con 0x7fd5c4004470 2019-12-05T05:37:02.916 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:02.899+0000 7fd5dcbaa700 1 --2- 172.21.15.42:0/2862599941 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd5c4004470 0x7fd5c4004940 secure :-1 s=THROTTLE_DONE pgs=911 cs=0 l=1 rx=0x7fd5cc001520 tx=0x7fd5cc00c090).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:02.916 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:02.899+0000 7fd5cbfff700 1 -- 172.21.15.42:0/2862599941 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5cc010bc0 con 0x7fd5c4004470 2019-12-05T05:37:02.916 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:02.899+0000 7fd5cbfff700 1 -- 172.21.15.42:0/2862599941 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd5cc010390 con 0x7fd5c4004470 2019-12-05T05:37:02.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:02.899+0000 7fd5dee0e700 1 -- 172.21.15.42:0/2862599941 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd5c4004470 msgr2=0x7fd5c4004940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:37:02.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:02.899+0000 7fd5dee0e700 1 --2- 172.21.15.42:0/2862599941 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd5c4004470 0x7fd5c4004940 secure :-1 s=READY pgs=911 cs=0 l=1 rx=0x7fd5cc001520 tx=0x7fd5cc00c090).stop 2019-12-05T05:37:02.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:02.903+0000 7fd5dee0e700 1 -- 172.21.15.42:0/2862599941 shutdown_connections 2019-12-05T05:37:02.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:02.903+0000 7fd5dee0e700 1 --2- 172.21.15.42:0/2862599941 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd5c4004470 0x7fd5c4004940 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:37:02.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:02.903+0000 7fd5dee0e700 1 -- 172.21.15.42:0/2862599941 shutdown_connections 2019-12-05T05:37:02.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:02.903+0000 7fd5dee0e700 1 -- 172.21.15.42:0/2862599941 wait complete. 2019-12-05T05:37:02.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:02.903+0000 7fd5dee0e700 1 -- 172.21.15.42:0/2862599941 >> 172.21.15.42:0/2862599941 conn(0x7fd5d80aabe0 msgr2=0x7fd5d80a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:37:02.921 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:02.903+0000 7fd5dee0e700 1 Processor -- start 2019-12-05T05:37:02.921 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:02.903+0000 7fd5dee0e700 1 -- start start 2019-12-05T05:37:02.921 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:02.907+0000 7fd5dee0e700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd5d8075af0 0x7fd5d8077f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:37:02.921 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:02.907+0000 7fd5dee0e700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5d8124b30 con 0x7fd5d8124700 2019-12-05T05:37:02.921 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:02.907+0000 7fd5dee0e700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5d8123eb0 con 0x7fd5d81264b0 2019-12-05T05:37:02.921 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:02.907+0000 7fd5dee0e700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5d8127c80 con 0x7fd5d8075af0 2019-12-05T05:37:02.922 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:02.907+0000 7fd5dcbaa700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fd5d81264b0 0x7fd5d80742e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:37416/0 (socket says 172.21.15.42:37416) 2019-12-05T05:37:02.922 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:02.907+0000 7fd5dcbaa700 1 -- 172.21.15.42:0/1385348042 learned_addr learned my addr 172.21.15.42:0/1385348042 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:37:02.922 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:02.907+0000 7fd5ddbac700 1 --2- 172.21.15.42:0/1385348042 >> v2:172.21.15.40:3300/0 conn(0x7fd5d8075af0 0x7fd5d8077f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:37:02.922 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:02.907+0000 7fd5c9ffb700 1 -- 172.21.15.42:0/1385348042 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 86530421 0 0) 0x7fd5d8123eb0 con 0x7fd5d81264b0 2019-12-05T05:37:02.922 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:02.907+0000 7fd5c9ffb700 1 -- 172.21.15.42:0/1385348042 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5b80030b0 con 0x7fd5d81264b0 2019-12-05T05:37:02.922 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:02.907+0000 7fd5ddbac700 1 -- 172.21.15.42:0/1385348042 >> v1:172.21.15.42:6789/0 conn(0x7fd5d81264b0 legacy=0x7fd5d80742e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:37:02.923 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:02.907+0000 7fd5ddbac700 1 -- 172.21.15.42:0/1385348042 >> v1:172.21.15.40:6789/0 conn(0x7fd5d8124700 legacy=0x7fd5d811b4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:37:02.923 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:02.907+0000 7fd5c9ffb700 1 -- 172.21.15.42:0/1385348042 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 936657951 0 0) 0x7fd5d8124b30 con 0x7fd5d8124700 2019-12-05T05:37:02.923 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:02.907+0000 7fd5ddbac700 1 -- 172.21.15.42:0/1385348042 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5d811bcb0 con 0x7fd5d8075af0 2019-12-05T05:37:02.923 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:02.907+0000 7fd5ddbac700 1 --2- 172.21.15.42:0/1385348042 >> v2:172.21.15.40:3300/0 conn(0x7fd5d8075af0 0x7fd5d8077f30 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fd5d81278c0 tx=0x7fd5d000c890).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:02.923 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:02.907+0000 7fd5ddbac700 1 --2- 172.21.15.42:0/1385348042 >> v2:172.21.15.40:3300/0 conn(0x7fd5d8075af0 0x7fd5d8077f30 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fd5d81278c0 tx=0x7fd5d000c890).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:02.923 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:02.907+0000 7fd5ddbac700 1 --2- 172.21.15.42:0/1385348042 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd5d8075af0 0x7fd5d8077f30 secure :-1 s=READY pgs=913 cs=0 l=1 rx=0x7fd5d81278c0 tx=0x7fd5d000c890).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:37:02.923 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:02.907+0000 7fd5ddbac700 1 --2- 172.21.15.42:0/1385348042 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd5d8075af0 0x7fd5d8077f30 secure :-1 s=THROTTLE_DONE pgs=913 cs=0 l=1 rx=0x7fd5d81278c0 tx=0x7fd5d000c890).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:02.924 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:02.907+0000 7fd5c9ffb700 1 -- 172.21.15.42:0/1385348042 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd5d0010170 con 0x7fd5d8075af0 2019-12-05T05:37:02.924 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:02.907+0000 7fd5ddbac700 1 --2- 172.21.15.42:0/1385348042 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd5d8075af0 0x7fd5d8077f30 secure :-1 s=THROTTLE_DONE pgs=913 cs=0 l=1 rx=0x7fd5d81278c0 tx=0x7fd5d000c890).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:02.924 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:02.907+0000 7fd5ddbac700 1 --2- 172.21.15.42:0/1385348042 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd5d8075af0 0x7fd5d8077f30 secure :-1 s=THROTTLE_DONE pgs=913 cs=0 l=1 rx=0x7fd5d81278c0 tx=0x7fd5d000c890).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:02.924 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:02.907+0000 7fd5c9ffb700 1 -- 172.21.15.42:0/1385348042 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5d000caf0 con 0x7fd5d8075af0 2019-12-05T05:37:02.924 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:02.907+0000 7fd5dee0e700 1 -- 172.21.15.42:0/1385348042 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5d8141ec0 con 0x7fd5d8075af0 2019-12-05T05:37:02.925 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:02.907+0000 7fd5c9ffb700 1 -- 172.21.15.42:0/1385348042 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd5d0002450 con 0x7fd5d8075af0 2019-12-05T05:37:02.925 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:02.907+0000 7fd5dee0e700 1 -- 172.21.15.42:0/1385348042 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5d81268e0 con 0x7fd5d8075af0 2019-12-05T05:37:02.925 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:02.911+0000 7fd5ddbac700 1 --2- 172.21.15.42:0/1385348042 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd5d8075af0 0x7fd5d8077f30 secure :-1 s=THROTTLE_DONE pgs=913 cs=0 l=1 rx=0x7fd5d81278c0 tx=0x7fd5d000c890).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:02.925 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:02.911+0000 7fd5c9ffb700 1 -- 172.21.15.42:0/1385348042 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fd5d00066e0 con 0x7fd5d8075af0 2019-12-05T05:37:02.925 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:02.911+0000 7fd5ddbac700 1 --2- 172.21.15.42:0/1385348042 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd5d8075af0 0x7fd5d8077f30 secure :-1 s=THROTTLE_DONE pgs=913 cs=0 l=1 rx=0x7fd5d81278c0 tx=0x7fd5d000c890).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:02.925 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:02.911+0000 7fd5c9ffb700 1 --2- 172.21.15.42:0/1385348042 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd5b803b230 0x7fd5b803d740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:37:02.926 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:02.911+0000 7fd5c9ffb700 1 -- 172.21.15.42:0/1385348042 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 7923+0+0 (secure 0 0 0) 0x7fd5d000b730 con 0x7fd5d8075af0 2019-12-05T05:37:02.926 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:02.911+0000 7fd5dd3ab700 1 --2- 172.21.15.42:0/1385348042 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd5b803b230 0x7fd5b803d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:37:02.926 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:02.911+0000 7fd5dd3ab700 1 --2- 172.21.15.42:0/1385348042 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd5b803b230 0x7fd5b803d740 crc :-1 s=READY pgs=503 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:37:02.926 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:02.911+0000 7fd5dee0e700 1 -- 172.21.15.42:0/1385348042 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5d8124d60 con 0x7fd5d8075af0 2019-12-05T05:37:02.927 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:02.919+0000 7fd5ddbac700 1 --2- 172.21.15.42:0/1385348042 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd5d8075af0 0x7fd5d8077f30 secure :-1 s=THROTTLE_DONE pgs=913 cs=0 l=1 rx=0x7fd5d81278c0 tx=0x7fd5d000c890).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:02.927 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:02.923+0000 7fd5c9ffb700 1 -- 172.21.15.42:0/1385348042 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fd5d00323c0 con 0x7fd5d8075af0 2019-12-05T05:37:03.096 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:03.091+0000 7fd5dee0e700 1 -- 172.21.15.42:0/1385348042 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd5d805d110 con 0x7fd5d8075af0 2019-12-05T05:37:03.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:03.091+0000 7fd5ddbac700 1 --2- 172.21.15.42:0/1385348042 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd5d8075af0 0x7fd5d8077f30 secure :-1 s=THROTTLE_DONE pgs=913 cs=0 l=1 rx=0x7fd5d81278c0 tx=0x7fd5d000c890).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:03.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:03.091+0000 7fd5c9ffb700 1 -- 172.21.15.42:0/1385348042 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 143 v143) v1 ==== 95+0+2112 (secure 0 0 0) 0x7fd5d0032620 con 0x7fd5d8075af0 2019-12-05T05:37:03.099 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:37:03.099 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":143,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/3838877789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3838877789},{"type":"v1","addr":"172.21.15.42:6835","nonce":3838877789}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":143}],"filesystems":[{"mdsmap":{"epoch":142,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:34:39.042388+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7221},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7221":{"gid":7221,"name":"b","rank":0,"incarnation":139,"state":"up:active","state_seq":27,"addr":"172.21.15.40:6833/3789641053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3789641053},{"type":"v1","addr":"172.21.15.40:6833","nonce":3789641053}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:37:03.099 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 143 2019-12-05T05:37:03.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:03.095+0000 7fd5dee0e700 1 -- 172.21.15.42:0/1385348042 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd5b803b230 msgr2=0x7fd5b803d740 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:37:03.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:03.095+0000 7fd5dee0e700 1 --2- 172.21.15.42:0/1385348042 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd5b803b230 0x7fd5b803d740 crc :-1 s=READY pgs=503 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:37:03.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:03.095+0000 7fd5dee0e700 1 -- 172.21.15.42:0/1385348042 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd5d8075af0 msgr2=0x7fd5d8077f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:37:03.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:03.095+0000 7fd5dee0e700 1 --2- 172.21.15.42:0/1385348042 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd5d8075af0 0x7fd5d8077f30 secure :-1 s=READY pgs=913 cs=0 l=1 rx=0x7fd5d81278c0 tx=0x7fd5d000c890).stop 2019-12-05T05:37:03.103 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:03.095+0000 7fd5dee0e700 1 -- 172.21.15.42:0/1385348042 shutdown_connections 2019-12-05T05:37:03.103 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:03.095+0000 7fd5dee0e700 1 --2- 172.21.15.42:0/1385348042 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd5b803b230 0x7fd5b803d740 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:37:03.103 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:03.095+0000 7fd5dee0e700 1 --2- 172.21.15.42:0/1385348042 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd5d8075af0 0x7fd5d8077f30 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:37:03.103 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:03.095+0000 7fd5dee0e700 1 -- 172.21.15.42:0/1385348042 shutdown_connections 2019-12-05T05:37:03.103 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:03.095+0000 7fd5dee0e700 1 -- 172.21.15.42:0/1385348042 wait complete. 2019-12-05T05:37:03.103 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:03.095+0000 7fd5dee0e700 1 -- 172.21.15.42:0/1385348042 >> 172.21.15.42:0/1385348042 conn(0x7fd5d80aabe0 msgr2=0x7fd5d809f3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:37:03.112 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T05:37:05.114 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:37:05.216 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:05.203+0000 7f7eb1bf9700 1 Processor -- start 2019-12-05T05:37:05.217 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:05.203+0000 7f7eb1bf9700 1 -- start start 2019-12-05T05:37:05.217 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:05.207+0000 7f7eb1bf9700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f7eac0a7740 0x7f7eac08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:37:05.217 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:05.207+0000 7f7eb1bf9700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7eac09fae0 0x7f7eac07eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:37:05.217 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:05.207+0000 7f7eb1bf9700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7eac0a0710 con 0x7f7eac07e720 2019-12-05T05:37:05.217 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:05.207+0000 7f7eb1bf9700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7eac0a0b00 con 0x7f7eac0a7740 2019-12-05T05:37:05.217 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:05.207+0000 7f7eb1bf9700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7eac08e5b0 con 0x7f7eac09fae0 2019-12-05T05:37:05.218 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:05.207+0000 7f7eabfff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f7eac0a7740 0x7f7eac08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:37:05.218 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:05.207+0000 7f7eabfff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f7eac0a7740 0x7f7eac08df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:37388/0 (socket says 172.21.15.42:37388) 2019-12-05T05:37:05.218 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:05.207+0000 7f7eab7fe700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f7eac07e720 0x7f7eac09f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:37424/0 (socket says 172.21.15.42:37424) 2019-12-05T05:37:05.218 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:05.207+0000 7f7eabfff700 1 -- 172.21.15.42:0/4057937573 learned_addr learned my addr 172.21.15.42:0/4057937573 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:37:05.218 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:05.207+0000 7f7eb0997700 1 --2- 172.21.15.42:0/4057937573 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7eac09fae0 0x7f7eac07eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:37:05.218 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:05.207+0000 7f7eabfff700 1 -- 172.21.15.42:0/4057937573 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7eac09fae0 msgr2=0x7f7eac07eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:37:05.218 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:05.207+0000 7f7eabfff700 1 --2- 172.21.15.42:0/4057937573 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7eac09fae0 0x7f7eac07eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:37:05.219 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:05.207+0000 7f7eabfff700 1 -- 172.21.15.42:0/4057937573 >> v1:172.21.15.42:6789/0 conn(0x7f7eac07e720 legacy=0x7f7eac09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:37:05.219 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:05.207+0000 7f7eaaffd700 1 -- 172.21.15.42:0/4057937573 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1432659775 0 0) 0x7f7eac0a0710 con 0x7f7eac07e720 2019-12-05T05:37:05.219 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:05.207+0000 7f7eabfff700 1 -- 172.21.15.42:0/4057937573 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7eac07f0a0 con 0x7f7eac0a7740 2019-12-05T05:37:05.219 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:05.207+0000 7f7eabfff700 1 --2- 172.21.15.42:0/4057937573 >> v2:172.21.15.42:3300/0 conn(0x7f7eac0a7740 0x7f7eac08df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f7e94008790 tx=0x7f7e9400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:05.219 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:05.207+0000 7f7eabfff700 1 --2- 172.21.15.42:0/4057937573 >> v2:172.21.15.42:3300/0 conn(0x7f7eac0a7740 0x7f7eac08df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f7e94008790 tx=0x7f7e9400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:05.219 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:05.207+0000 7f7eabfff700 1 --2- 172.21.15.42:0/4057937573 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7eac0a7740 0x7f7eac08df80 secure :-1 s=READY pgs=1162 cs=0 l=1 rx=0x7f7e94008790 tx=0x7f7e9400fd30).ready entity=mon.1 client_cookie=d20dfbc9dfe68e58 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:37:05.219 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:05.207+0000 7f7eabfff700 1 --2- 172.21.15.42:0/4057937573 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7eac0a7740 0x7f7eac08df80 secure :-1 s=THROTTLE_DONE pgs=1162 cs=0 l=1 rx=0x7f7e94008790 tx=0x7f7e9400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:05.220 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:05.207+0000 7f7eaaffd700 1 -- 172.21.15.42:0/4057937573 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7e94008c90 con 0x7f7eac0a7740 2019-12-05T05:37:05.221 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:05.207+0000 7f7eabfff700 1 --2- 172.21.15.42:0/4057937573 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7eac0a7740 0x7f7eac08df80 secure :-1 s=THROTTLE_DONE pgs=1162 cs=0 l=1 rx=0x7f7e94008790 tx=0x7f7e9400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:05.221 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:05.207+0000 7f7eabfff700 1 --2- 172.21.15.42:0/4057937573 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7eac0a7740 0x7f7eac08df80 secure :-1 s=THROTTLE_DONE pgs=1162 cs=0 l=1 rx=0x7f7e94008790 tx=0x7f7e9400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:05.221 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:05.207+0000 7f7eaaffd700 1 -- 172.21.15.42:0/4057937573 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e940088b0 con 0x7f7eac0a7740 2019-12-05T05:37:05.221 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:05.207+0000 7f7eaaffd700 1 -- 172.21.15.42:0/4057937573 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7e94011b10 con 0x7f7eac0a7740 2019-12-05T05:37:05.221 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:05.207+0000 7f7eb1bf9700 1 -- 172.21.15.42:0/4057937573 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7eac0a7740 msgr2=0x7f7eac08df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:37:05.221 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:05.207+0000 7f7eb1bf9700 1 --2- 172.21.15.42:0/4057937573 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7eac0a7740 0x7f7eac08df80 secure :-1 s=READY pgs=1162 cs=0 l=1 rx=0x7f7e94008790 tx=0x7f7e9400fd30).stop 2019-12-05T05:37:05.221 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:05.207+0000 7f7eb1bf9700 1 -- 172.21.15.42:0/4057937573 shutdown_connections 2019-12-05T05:37:05.222 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:05.207+0000 7f7eb1bf9700 1 --2- 172.21.15.42:0/4057937573 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7eac09fae0 0x7f7eac07eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:37:05.222 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:05.207+0000 7f7eb1bf9700 1 --2- 172.21.15.42:0/4057937573 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7eac0a7740 0x7f7eac08df80 unknown :-1 s=CLOSED pgs=1162 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:37:05.222 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:05.207+0000 7f7eb1bf9700 1 -- 172.21.15.42:0/4057937573 shutdown_connections 2019-12-05T05:37:05.222 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:05.207+0000 7f7eb1bf9700 1 -- 172.21.15.42:0/4057937573 wait complete. 2019-12-05T05:37:05.222 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:05.207+0000 7f7eb1bf9700 1 -- 172.21.15.42:0/4057937573 >> 172.21.15.42:0/4057937573 conn(0x7f7eac0aabe0 msgr2=0x7f7eac0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:37:05.232 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:05.211+0000 7f7eb1bf9700 1 Processor -- start 2019-12-05T05:37:05.232 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:05.211+0000 7f7eb1bf9700 1 -- start start 2019-12-05T05:37:05.232 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:05.211+0000 7f7eb1bf9700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f7eac11c7b0 0x7f7eac07c930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:37:05.232 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:05.211+0000 7f7eb1bf9700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7eac133540 0x7f7eac135980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:37:05.232 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:05.211+0000 7f7eb1bf9700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7eac0a4120 con 0x7f7eac07ce80 2019-12-05T05:37:05.232 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:05.211+0000 7f7eb1bf9700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7eac145860 con 0x7f7eac11c7b0 2019-12-05T05:37:05.233 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:05.211+0000 7f7eb1bf9700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7eac1437b0 con 0x7f7eac133540 2019-12-05T05:37:05.233 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:05.211+0000 7f7eabfff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f7eac11c7b0 0x7f7eac07c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:37:05.233 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:05.211+0000 7f7eab7fe700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f7eac07ce80 0x7f7eac142530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:37432/0 (socket says 172.21.15.42:37432) 2019-12-05T05:37:05.233 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:05.211+0000 7f7eab7fe700 1 -- 172.21.15.42:0/3684054717 learned_addr learned my addr 172.21.15.42:0/3684054717 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:37:05.233 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:05.211+0000 7f7eb0997700 1 --2- 172.21.15.42:0/3684054717 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7eac133540 0x7f7eac135980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:37:05.233 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:05.211+0000 7f7eabfff700 1 -- 172.21.15.42:0/3684054717 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7eac133540 msgr2=0x7f7eac135980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:37:05.233 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:05.211+0000 7f7eabfff700 1 --2- 172.21.15.42:0/3684054717 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7eac133540 0x7f7eac135980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:37:05.234 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:05.211+0000 7f7eabfff700 1 -- 172.21.15.42:0/3684054717 >> v1:172.21.15.42:6789/0 conn(0x7f7eac07ce80 legacy=0x7f7eac142530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:37:05.234 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:05.211+0000 7f7eabfff700 1 -- 172.21.15.42:0/3684054717 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7eac142d30 con 0x7f7eac11c7b0 2019-12-05T05:37:05.234 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:05.211+0000 7f7ea8ff9700 1 -- 172.21.15.42:0/3684054717 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 615354788 0 0) 0x7f7eac0a4120 con 0x7f7eac07ce80 2019-12-05T05:37:05.234 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:05.211+0000 7f7eabfff700 1 --2- 172.21.15.42:0/3684054717 >> v2:172.21.15.42:3300/0 conn(0x7f7eac11c7b0 0x7f7eac07c930 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f7eac09fa10 tx=0x7f7e94008f40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:05.234 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:05.211+0000 7f7eabfff700 1 --2- 172.21.15.42:0/3684054717 >> v2:172.21.15.42:3300/0 conn(0x7f7eac11c7b0 0x7f7eac07c930 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f7eac09fa10 tx=0x7f7e94008f40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:05.234 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:05.211+0000 7f7eabfff700 1 --2- 172.21.15.42:0/3684054717 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7eac11c7b0 0x7f7eac07c930 secure :-1 s=READY pgs=1164 cs=0 l=1 rx=0x7f7eac09fa10 tx=0x7f7e94008f40).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:37:05.234 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:05.215+0000 7f7eabfff700 1 --2- 172.21.15.42:0/3684054717 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7eac11c7b0 0x7f7eac07c930 secure :-1 s=THROTTLE_DONE pgs=1164 cs=0 l=1 rx=0x7f7eac09fa10 tx=0x7f7e94008f40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:05.235 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:05.215+0000 7f7ea8ff9700 1 -- 172.21.15.42:0/3684054717 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7e9400f470 con 0x7f7eac11c7b0 2019-12-05T05:37:05.235 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:05.215+0000 7f7eabfff700 1 --2- 172.21.15.42:0/3684054717 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7eac11c7b0 0x7f7eac07c930 secure :-1 s=THROTTLE_DONE pgs=1164 cs=0 l=1 rx=0x7f7eac09fa10 tx=0x7f7e94008f40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:05.235 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:05.215+0000 7f7ea8ff9700 1 -- 172.21.15.42:0/3684054717 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e94011730 con 0x7f7eac11c7b0 2019-12-05T05:37:05.235 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:05.215+0000 7f7eabfff700 1 --2- 172.21.15.42:0/3684054717 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7eac11c7b0 0x7f7eac07c930 secure :-1 s=THROTTLE_DONE pgs=1164 cs=0 l=1 rx=0x7f7eac09fa10 tx=0x7f7e94008f40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:05.235 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:05.215+0000 7f7eb1bf9700 1 -- 172.21.15.42:0/3684054717 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7eac145280 con 0x7f7eac11c7b0 2019-12-05T05:37:05.235 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:05.215+0000 7f7ea8ff9700 1 -- 172.21.15.42:0/3684054717 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7e94005730 con 0x7f7eac11c7b0 2019-12-05T05:37:05.235 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:05.215+0000 7f7eb1bf9700 1 -- 172.21.15.42:0/3684054717 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7eac0a19f0 con 0x7f7eac11c7b0 2019-12-05T05:37:05.236 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:05.215+0000 7f7eabfff700 1 --2- 172.21.15.42:0/3684054717 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7eac11c7b0 0x7f7eac07c930 secure :-1 s=THROTTLE_DONE pgs=1164 cs=0 l=1 rx=0x7f7eac09fa10 tx=0x7f7e94008f40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:05.236 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:05.215+0000 7f7eabfff700 1 --2- 172.21.15.42:0/3684054717 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7eac11c7b0 0x7f7eac07c930 secure :-1 s=THROTTLE_DONE pgs=1164 cs=0 l=1 rx=0x7f7eac09fa10 tx=0x7f7e94008f40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:05.236 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:05.215+0000 7f7ea8ff9700 1 -- 172.21.15.42:0/3684054717 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f7e940059d0 con 0x7f7eac11c7b0 2019-12-05T05:37:05.236 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:05.215+0000 7f7ea8ff9700 1 --2- 172.21.15.42:0/3684054717 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f7e9003ac30 0x7f7e9003d140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:37:05.236 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:05.215+0000 7f7ea8ff9700 1 -- 172.21.15.42:0/3684054717 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 7923+0+0 (secure 0 0 0) 0x7f7e9404b710 con 0x7f7eac11c7b0 2019-12-05T05:37:05.242 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:05.215+0000 7f7eab7fe700 1 --2- 172.21.15.42:0/3684054717 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f7e9003ac30 0x7f7e9003d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:37:05.242 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:05.215+0000 7f7eab7fe700 1 --2- 172.21.15.42:0/3684054717 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f7e9003ac30 0x7f7e9003d140 crc :-1 s=READY pgs=504 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:37:05.242 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:05.219+0000 7f7eb1bf9700 1 -- 172.21.15.42:0/3684054717 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7eac05d110 con 0x7f7eac11c7b0 2019-12-05T05:37:05.243 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:05.227+0000 7f7eabfff700 1 --2- 172.21.15.42:0/3684054717 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7eac11c7b0 0x7f7eac07c930 secure :-1 s=THROTTLE_DONE pgs=1164 cs=0 l=1 rx=0x7f7eac09fa10 tx=0x7f7e94008f40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:05.243 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:05.227+0000 7f7ea8ff9700 1 -- 172.21.15.42:0/3684054717 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f7e94032930 con 0x7f7eac11c7b0 2019-12-05T05:37:05.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:05.395+0000 7f7eb1bf9700 1 -- 172.21.15.42:0/3684054717 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7eac05b8c0 con 0x7f7eac11c7b0 2019-12-05T05:37:05.404 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:05.399+0000 7f7eabfff700 1 --2- 172.21.15.42:0/3684054717 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7eac11c7b0 0x7f7eac07c930 secure :-1 s=THROTTLE_DONE pgs=1164 cs=0 l=1 rx=0x7f7eac09fa10 tx=0x7f7e94008f40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:05.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:05.399+0000 7f7ea8ff9700 1 -- 172.21.15.42:0/3684054717 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 143 v143) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f7e940334e0 con 0x7f7eac11c7b0 2019-12-05T05:37:05.405 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 143 2019-12-05T05:37:05.405 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:37:05.405 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":143,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/3838877789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3838877789},{"type":"v1","addr":"172.21.15.42:6835","nonce":3838877789}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":143}],"filesystems":[{"mdsmap":{"epoch":142,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:34:39.042388+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7221},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7221":{"gid":7221,"name":"b","rank":0,"incarnation":139,"state":"up:active","state_seq":27,"addr":"172.21.15.40:6833/3789641053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3789641053},{"type":"v1","addr":"172.21.15.40:6833","nonce":3789641053}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:37:05.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:05.403+0000 7f7eb1bf9700 1 -- 172.21.15.42:0/3684054717 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f7e9003ac30 msgr2=0x7f7e9003d140 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:37:05.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:05.403+0000 7f7eb1bf9700 1 --2- 172.21.15.42:0/3684054717 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f7e9003ac30 0x7f7e9003d140 crc :-1 s=READY pgs=504 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:37:05.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:05.403+0000 7f7eb1bf9700 1 -- 172.21.15.42:0/3684054717 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7eac11c7b0 msgr2=0x7f7eac07c930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:37:05.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:05.403+0000 7f7eb1bf9700 1 --2- 172.21.15.42:0/3684054717 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7eac11c7b0 0x7f7eac07c930 secure :-1 s=READY pgs=1164 cs=0 l=1 rx=0x7f7eac09fa10 tx=0x7f7e94008f40).stop 2019-12-05T05:37:05.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:05.403+0000 7f7eb1bf9700 1 -- 172.21.15.42:0/3684054717 shutdown_connections 2019-12-05T05:37:05.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:05.403+0000 7f7eb1bf9700 1 --2- 172.21.15.42:0/3684054717 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f7e9003ac30 0x7f7e9003d140 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:37:05.411 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:05.403+0000 7f7eb1bf9700 1 --2- 172.21.15.42:0/3684054717 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7eac133540 0x7f7eac135980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:37:05.411 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:05.403+0000 7f7eb1bf9700 1 --2- 172.21.15.42:0/3684054717 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7eac11c7b0 0x7f7eac07c930 unknown :-1 s=CLOSED pgs=1164 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:37:05.411 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:05.403+0000 7f7eb1bf9700 1 -- 172.21.15.42:0/3684054717 shutdown_connections 2019-12-05T05:37:05.411 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:05.403+0000 7f7eb1bf9700 1 -- 172.21.15.42:0/3684054717 wait complete. 2019-12-05T05:37:05.412 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:05.403+0000 7f7eb1bf9700 1 -- 172.21.15.42:0/3684054717 >> 172.21.15.42:0/3684054717 conn(0x7f7eac07de90 msgr2=0x7f7eac0a9640 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:37:05.424 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T05:37:07.425 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:37:07.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:07.515+0000 7f405e2b9700 1 Processor -- start 2019-12-05T05:37:07.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:07.515+0000 7f405e2b9700 1 -- start start 2019-12-05T05:37:07.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:07.519+0000 7f405e2b9700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f40580a7740 0x7f405808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:37:07.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:07.519+0000 7f405e2b9700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f405807e720 0x7f405807eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:37:07.530 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:07.519+0000 7f405e2b9700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40580a0710 con 0x7f405809f2e0 2019-12-05T05:37:07.530 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:07.519+0000 7f405e2b9700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40580a0b00 con 0x7f40580a7740 2019-12-05T05:37:07.530 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:07.519+0000 7f405e2b9700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f405808e5b0 con 0x7f405807e720 2019-12-05T05:37:07.530 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:07.519+0000 7f405d057700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f405809f2e0 0x7f405809f710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:37442/0 (socket says 172.21.15.42:37442) 2019-12-05T05:37:07.530 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:07.519+0000 7f405d057700 1 -- 172.21.15.42:0/1170547682 learned_addr learned my addr 172.21.15.42:0/1170547682 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:37:07.530 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:07.519+0000 7f405c856700 1 --2- 172.21.15.42:0/1170547682 >> v2:172.21.15.42:3300/0 conn(0x7f40580a7740 0x7f405808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:37:07.531 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:07.519+0000 7f4057fff700 1 --2- 172.21.15.42:0/1170547682 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f405807e720 0x7f405807eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:37:07.531 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:07.519+0000 7f40577fe700 1 -- 172.21.15.42:0/1170547682 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1142412648 0 0) 0x7f40580a0710 con 0x7f405809f2e0 2019-12-05T05:37:07.531 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:07.519+0000 7f40577fe700 1 -- 172.21.15.42:0/1170547682 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40480030b0 con 0x7f405809f2e0 2019-12-05T05:37:07.531 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:07.519+0000 7f405c856700 1 -- 172.21.15.42:0/1170547682 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f405807e720 msgr2=0x7f405807eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:37:07.531 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:07.519+0000 7f405c856700 1 --2- 172.21.15.42:0/1170547682 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f405807e720 0x7f405807eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:37:07.532 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:07.519+0000 7f40577fe700 1 -- 172.21.15.42:0/1170547682 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2341401334 0 0) 0x7f40480030b0 con 0x7f405809f2e0 2019-12-05T05:37:07.532 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:07.519+0000 7f405c856700 1 -- 172.21.15.42:0/1170547682 >> v1:172.21.15.42:6789/0 conn(0x7f405809f2e0 legacy=0x7f405809f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:37:07.532 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:07.519+0000 7f405c856700 1 -- 172.21.15.42:0/1170547682 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f405807f0a0 con 0x7f40580a7740 2019-12-05T05:37:07.532 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:07.519+0000 7f405c856700 1 --2- 172.21.15.42:0/1170547682 >> v2:172.21.15.42:3300/0 conn(0x7f40580a7740 0x7f405808df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f404c008790 tx=0x7f4050001820).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:07.532 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:07.519+0000 7f405c856700 1 --2- 172.21.15.42:0/1170547682 >> v2:172.21.15.42:3300/0 conn(0x7f40580a7740 0x7f405808df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f404c008790 tx=0x7f4050001820).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:07.532 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:07.519+0000 7f405c856700 1 --2- 172.21.15.42:0/1170547682 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f40580a7740 0x7f405808df80 secure :-1 s=READY pgs=1166 cs=0 l=1 rx=0x7f404c008790 tx=0x7f4050001820).ready entity=mon.1 client_cookie=e178c51ff9cbd304 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:37:07.533 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:07.519+0000 7f405c856700 1 --2- 172.21.15.42:0/1170547682 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f40580a7740 0x7f405808df80 secure :-1 s=THROTTLE_DONE pgs=1166 cs=0 l=1 rx=0x7f404c008790 tx=0x7f4050001820).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:07.533 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:07.519+0000 7f405c856700 1 --2- 172.21.15.42:0/1170547682 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f40580a7740 0x7f405808df80 secure :-1 s=THROTTLE_DONE pgs=1166 cs=0 l=1 rx=0x7f404c008790 tx=0x7f4050001820).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:07.533 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:07.519+0000 7f40577fe700 1 -- 172.21.15.42:0/1170547682 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4050001ad0 con 0x7f40580a7740 2019-12-05T05:37:07.533 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:07.519+0000 7f405c856700 1 --2- 172.21.15.42:0/1170547682 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f40580a7740 0x7f405808df80 secure :-1 s=THROTTLE_DONE pgs=1166 cs=0 l=1 rx=0x7f404c008790 tx=0x7f4050001820).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:07.533 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:07.519+0000 7f40577fe700 1 -- 172.21.15.42:0/1170547682 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40500018e0 con 0x7f40580a7740 2019-12-05T05:37:07.533 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:07.519+0000 7f40577fe700 1 -- 172.21.15.42:0/1170547682 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f404c0113e0 con 0x7f40580a7740 2019-12-05T05:37:07.533 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:07.519+0000 7f405e2b9700 1 -- 172.21.15.42:0/1170547682 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f40580a7740 msgr2=0x7f405808df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:37:07.534 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:07.519+0000 7f405e2b9700 1 --2- 172.21.15.42:0/1170547682 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f40580a7740 0x7f405808df80 secure :-1 s=READY pgs=1166 cs=0 l=1 rx=0x7f404c008790 tx=0x7f4050001820).stop 2019-12-05T05:37:07.534 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:07.523+0000 7f405e2b9700 1 -- 172.21.15.42:0/1170547682 shutdown_connections 2019-12-05T05:37:07.534 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:07.523+0000 7f405e2b9700 1 --2- 172.21.15.42:0/1170547682 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f40580a7740 0x7f405808df80 unknown :-1 s=CLOSED pgs=1166 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:37:07.534 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:07.523+0000 7f405e2b9700 1 --2- 172.21.15.42:0/1170547682 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f405807e720 0x7f405807eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:37:07.534 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:07.523+0000 7f405e2b9700 1 -- 172.21.15.42:0/1170547682 shutdown_connections 2019-12-05T05:37:07.534 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:07.523+0000 7f405e2b9700 1 -- 172.21.15.42:0/1170547682 wait complete. 2019-12-05T05:37:07.534 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:07.523+0000 7f405e2b9700 1 -- 172.21.15.42:0/1170547682 >> 172.21.15.42:0/1170547682 conn(0x7f40580aabe0 msgr2=0x7f40580a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:37:07.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:07.523+0000 7f405e2b9700 1 Processor -- start 2019-12-05T05:37:07.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:07.523+0000 7f405e2b9700 1 -- start start 2019-12-05T05:37:07.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:07.527+0000 7f405e2b9700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f405811a1b0 0x7f405807d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:37:07.539 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:07.527+0000 7f405e2b9700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f405807df40 0x7f4058130740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:37:07.539 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:07.527+0000 7f405e2b9700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f405811bff0 con 0x7f405805a190 2019-12-05T05:37:07.539 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:07.527+0000 7f405e2b9700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4058130200 con 0x7f405807df40 2019-12-05T05:37:07.539 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:07.527+0000 7f405e2b9700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40581304a0 con 0x7f405811a1b0 2019-12-05T05:37:07.539 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:07.527+0000 7f4057fff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f405807df40 0x7f4058130740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:37:07.539 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:07.527+0000 7f4057fff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f405807df40 0x7f4058130740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:37410/0 (socket says 172.21.15.42:37410) 2019-12-05T05:37:07.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:07.527+0000 7f4057fff700 1 -- 172.21.15.42:0/1244230783 learned_addr learned my addr 172.21.15.42:0/1244230783 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:37:07.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:07.527+0000 7f405c856700 1 --2- 172.21.15.42:0/1244230783 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f405811a1b0 0x7f405807d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:37:07.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:07.527+0000 7f40557fa700 1 -- 172.21.15.42:0/1244230783 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2706224114 0 0) 0x7f405811bff0 con 0x7f405805a190 2019-12-05T05:37:07.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:07.527+0000 7f40557fa700 1 -- 172.21.15.42:0/1244230783 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40380030b0 con 0x7f405805a190 2019-12-05T05:37:07.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:07.527+0000 7f4057fff700 1 -- 172.21.15.42:0/1244230783 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f405811a1b0 msgr2=0x7f405807d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:37:07.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:07.527+0000 7f4057fff700 1 --2- 172.21.15.42:0/1244230783 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f405811a1b0 0x7f405807d9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:37:07.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:07.527+0000 7f4057fff700 1 -- 172.21.15.42:0/1244230783 >> v1:172.21.15.40:6789/0 conn(0x7f405805a190 legacy=0x7f40580a4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:37:07.541 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:07.527+0000 7f4057fff700 1 -- 172.21.15.42:0/1244230783 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f405812cba0 con 0x7f405807df40 2019-12-05T05:37:07.541 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:07.527+0000 7f405c856700 1 --2- 172.21.15.42:0/1244230783 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f405811a1b0 0x7f405807d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:37:07.542 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:07.527+0000 7f4057fff700 1 --2- 172.21.15.42:0/1244230783 >> v2:172.21.15.42:3300/0 conn(0x7f405807df40 0x7f4058130740 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f405811c220 tx=0x7f4044010c70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:07.542 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:07.527+0000 7f4057fff700 1 --2- 172.21.15.42:0/1244230783 >> v2:172.21.15.42:3300/0 conn(0x7f405807df40 0x7f4058130740 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f405811c220 tx=0x7f4044010c70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:07.542 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:07.527+0000 7f4057fff700 1 --2- 172.21.15.42:0/1244230783 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f405807df40 0x7f4058130740 secure :-1 s=READY pgs=1167 cs=0 l=1 rx=0x7f405811c220 tx=0x7f4044010c70).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:37:07.542 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:07.527+0000 7f4057fff700 1 --2- 172.21.15.42:0/1244230783 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f405807df40 0x7f4058130740 secure :-1 s=THROTTLE_DONE pgs=1167 cs=0 l=1 rx=0x7f405811c220 tx=0x7f4044010c70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:07.542 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:07.527+0000 7f40557fa700 1 -- 172.21.15.42:0/1244230783 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4044005930 con 0x7f405807df40 2019-12-05T05:37:07.542 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:07.527+0000 7f4057fff700 1 --2- 172.21.15.42:0/1244230783 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f405807df40 0x7f4058130740 secure :-1 s=THROTTLE_DONE pgs=1167 cs=0 l=1 rx=0x7f405811c220 tx=0x7f4044010c70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:07.543 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:07.527+0000 7f4057fff700 1 --2- 172.21.15.42:0/1244230783 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f405807df40 0x7f4058130740 secure :-1 s=THROTTLE_DONE pgs=1167 cs=0 l=1 rx=0x7f405811c220 tx=0x7f4044010c70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:07.543 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:07.527+0000 7f40557fa700 1 -- 172.21.15.42:0/1244230783 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4044007160 con 0x7f405807df40 2019-12-05T05:37:07.543 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:07.527+0000 7f40557fa700 1 -- 172.21.15.42:0/1244230783 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f40440122b0 con 0x7f405807df40 2019-12-05T05:37:07.543 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:07.527+0000 7f405e2b9700 1 -- 172.21.15.42:0/1244230783 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f405811bbc0 con 0x7f405807df40 2019-12-05T05:37:07.543 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:07.527+0000 7f405e2b9700 1 -- 172.21.15.42:0/1244230783 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4058130c90 con 0x7f405807df40 2019-12-05T05:37:07.543 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:07.527+0000 7f4057fff700 1 --2- 172.21.15.42:0/1244230783 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f405807df40 0x7f4058130740 secure :-1 s=THROTTLE_DONE pgs=1167 cs=0 l=1 rx=0x7f405811c220 tx=0x7f4044010c70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:07.543 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:07.527+0000 7f40557fa700 1 -- 172.21.15.42:0/1244230783 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f40440125c0 con 0x7f405807df40 2019-12-05T05:37:07.544 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:07.527+0000 7f4057fff700 1 --2- 172.21.15.42:0/1244230783 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f405807df40 0x7f4058130740 secure :-1 s=THROTTLE_DONE pgs=1167 cs=0 l=1 rx=0x7f405811c220 tx=0x7f4044010c70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:07.544 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:07.531+0000 7f40557fa700 1 --2- 172.21.15.42:0/1244230783 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f403803b2d0 0x7f403803d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:37:07.544 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:07.531+0000 7f40557fa700 1 -- 172.21.15.42:0/1244230783 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 7923+0+0 (secure 0 0 0) 0x7f404404b370 con 0x7f405807df40 2019-12-05T05:37:07.544 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:07.531+0000 7f405c856700 1 --2- 172.21.15.42:0/1244230783 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f403803b2d0 0x7f403803d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:37:07.544 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:07.531+0000 7f405c856700 1 --2- 172.21.15.42:0/1244230783 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f403803b2d0 0x7f403803d7e0 crc :-1 s=READY pgs=505 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:37:07.545 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:07.531+0000 7f405e2b9700 1 -- 172.21.15.42:0/1244230783 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f405805d110 con 0x7f405807df40 2019-12-05T05:37:07.546 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:07.539+0000 7f4057fff700 1 --2- 172.21.15.42:0/1244230783 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f405807df40 0x7f4058130740 secure :-1 s=THROTTLE_DONE pgs=1167 cs=0 l=1 rx=0x7f405811c220 tx=0x7f4044010c70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:07.547 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:07.539+0000 7f40557fa700 1 -- 172.21.15.42:0/1244230783 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f40440323f0 con 0x7f405807df40 2019-12-05T05:37:07.716 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:07.711+0000 7f405e2b9700 1 -- 172.21.15.42:0/1244230783 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f405805b8c0 con 0x7f405807df40 2019-12-05T05:37:07.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:07.711+0000 7f4057fff700 1 --2- 172.21.15.42:0/1244230783 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f405807df40 0x7f4058130740 secure :-1 s=THROTTLE_DONE pgs=1167 cs=0 l=1 rx=0x7f405811c220 tx=0x7f4044010c70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:07.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:07.711+0000 7f40557fa700 1 -- 172.21.15.42:0/1244230783 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 143 v143) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f4044032650 con 0x7f405807df40 2019-12-05T05:37:07.718 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 143 2019-12-05T05:37:07.718 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:37:07.718 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":143,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5693,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/3838877789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3838877789},{"type":"v1","addr":"172.21.15.42:6835","nonce":3838877789}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":143}],"filesystems":[{"mdsmap":{"epoch":142,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:34:39.042388+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7221},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7221":{"gid":7221,"name":"b","rank":0,"incarnation":139,"state":"up:active","state_seq":27,"addr":"172.21.15.40:6833/3789641053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3789641053},{"type":"v1","addr":"172.21.15.40:6833","nonce":3789641053}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:37:07.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:07.711+0000 7f405e2b9700 1 -- 172.21.15.42:0/1244230783 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f403803b2d0 msgr2=0x7f403803d7e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:37:07.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:07.711+0000 7f405e2b9700 1 --2- 172.21.15.42:0/1244230783 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f403803b2d0 0x7f403803d7e0 crc :-1 s=READY pgs=505 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:37:07.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:07.715+0000 7f405e2b9700 1 -- 172.21.15.42:0/1244230783 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f405807df40 msgr2=0x7f4058130740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:37:07.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:07.715+0000 7f405e2b9700 1 --2- 172.21.15.42:0/1244230783 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f405807df40 0x7f4058130740 secure :-1 s=READY pgs=1167 cs=0 l=1 rx=0x7f405811c220 tx=0x7f4044010c70).stop 2019-12-05T05:37:07.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:07.715+0000 7f405e2b9700 1 -- 172.21.15.42:0/1244230783 shutdown_connections 2019-12-05T05:37:07.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:07.715+0000 7f405e2b9700 1 --2- 172.21.15.42:0/1244230783 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f403803b2d0 0x7f403803d7e0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:37:07.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:07.715+0000 7f405e2b9700 1 --2- 172.21.15.42:0/1244230783 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f405811a1b0 0x7f405807d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:37:07.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:07.715+0000 7f405e2b9700 1 --2- 172.21.15.42:0/1244230783 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f405807df40 0x7f4058130740 unknown :-1 s=CLOSED pgs=1167 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:37:07.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:07.715+0000 7f405e2b9700 1 -- 172.21.15.42:0/1244230783 shutdown_connections 2019-12-05T05:37:07.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:07.715+0000 7f405e2b9700 1 -- 172.21.15.42:0/1244230783 wait complete. 2019-12-05T05:37:07.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:07.715+0000 7f405e2b9700 1 -- 172.21.15.42:0/1244230783 >> 172.21.15.42:0/1244230783 conn(0x7f40580a9b50 msgr2=0x7f405809f380 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:37:07.729 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T05:37:09.731 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:37:09.834 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:09.823+0000 7f8130f9f700 1 Processor -- start 2019-12-05T05:37:09.834 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:09.823+0000 7f8130f9f700 1 -- start start 2019-12-05T05:37:09.834 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:09.823+0000 7f8130f9f700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f812c0a7730 0x7f812c08df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:37:09.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:09.823+0000 7f8130f9f700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f812c07e710 0x7f812c07eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:37:09.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:09.823+0000 7f8130f9f700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f812c0a0700 con 0x7f812c09f2d0 2019-12-05T05:37:09.840 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:09.823+0000 7f8130f9f700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f812c0a0af0 con 0x7f812c07e710 2019-12-05T05:37:09.840 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:09.823+0000 7f8130f9f700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f812c08e5a0 con 0x7f812c0a7730 2019-12-05T05:37:09.840 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:09.823+0000 7f812ad9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f812c0a7730 0x7f812c08df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:37:09.841 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:09.823+0000 7f812b59e700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f812c09f2d0 0x7f812c09f700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:37470/0 (socket says 172.21.15.42:37470) 2019-12-05T05:37:09.841 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:09.823+0000 7f812b59e700 1 -- 172.21.15.42:0/3566750840 learned_addr learned my addr 172.21.15.42:0/3566750840 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:37:09.841 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:09.827+0000 7f8123fff700 1 --2- 172.21.15.42:0/3566750840 >> v2:172.21.15.40:3300/0 conn(0x7f812c07e710 0x7f812c07eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:37:09.841 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:09.827+0000 7f812ad9d700 1 -- 172.21.15.42:0/3566750840 >> v2:172.21.15.40:3300/0 conn(0x7f812c07e710 msgr2=0x7f812c07eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:37:09.841 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:09.827+0000 7f812ad9d700 1 --2- 172.21.15.42:0/3566750840 >> v2:172.21.15.40:3300/0 conn(0x7f812c07e710 0x7f812c07eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:37:09.841 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:09.827+0000 7f812ad9d700 1 -- 172.21.15.42:0/3566750840 >> v1:172.21.15.42:6789/0 conn(0x7f812c09f2d0 legacy=0x7f812c09f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:37:09.842 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:09.827+0000 7f812a59c700 1 -- 172.21.15.42:0/3566750840 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4252274605 0 0) 0x7f812c0a0700 con 0x7f812c09f2d0 2019-12-05T05:37:09.842 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:09.827+0000 7f812ad9d700 1 -- 172.21.15.42:0/3566750840 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f812c07f090 con 0x7f812c0a7730 2019-12-05T05:37:09.842 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:09.827+0000 7f812ad9d700 1 --2- 172.21.15.42:0/3566750840 >> v2:172.21.15.42:3300/0 conn(0x7f812c0a7730 0x7f812c08df70 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f811c002f30 tx=0x7f811c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:09.842 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:09.827+0000 7f812ad9d700 1 --2- 172.21.15.42:0/3566750840 >> v2:172.21.15.42:3300/0 conn(0x7f812c0a7730 0x7f812c08df70 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f811c002f30 tx=0x7f811c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:09.843 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:09.827+0000 7f812ad9d700 1 --2- 172.21.15.42:0/3566750840 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f812c0a7730 0x7f812c08df70 secure :-1 s=READY pgs=1169 cs=0 l=1 rx=0x7f811c002f30 tx=0x7f811c00fc70).ready entity=mon.1 client_cookie=881f314bcf40397e server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:37:09.843 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:09.827+0000 7f812ad9d700 1 --2- 172.21.15.42:0/3566750840 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f812c0a7730 0x7f812c08df70 secure :-1 s=THROTTLE_DONE pgs=1169 cs=0 l=1 rx=0x7f811c002f30 tx=0x7f811c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:09.844 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:09.827+0000 7f812a59c700 1 -- 172.21.15.42:0/3566750840 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f811c0112b0 con 0x7f812c0a7730 2019-12-05T05:37:09.844 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:09.827+0000 7f812ad9d700 1 --2- 172.21.15.42:0/3566750840 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f812c0a7730 0x7f812c08df70 secure :-1 s=THROTTLE_DONE pgs=1169 cs=0 l=1 rx=0x7f811c002f30 tx=0x7f811c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:09.844 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:09.827+0000 7f812ad9d700 1 --2- 172.21.15.42:0/3566750840 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f812c0a7730 0x7f812c08df70 secure :-1 s=THROTTLE_DONE pgs=1169 cs=0 l=1 rx=0x7f811c002f30 tx=0x7f811c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:09.849 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:09.827+0000 7f812a59c700 1 -- 172.21.15.42:0/3566750840 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f811c008ad0 con 0x7f812c0a7730 2019-12-05T05:37:09.849 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:09.827+0000 7f812a59c700 1 -- 172.21.15.42:0/3566750840 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f811c011890 con 0x7f812c0a7730 2019-12-05T05:37:09.849 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:09.827+0000 7f8130f9f700 1 -- 172.21.15.42:0/3566750840 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f812c0a7730 msgr2=0x7f812c08df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:37:09.849 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:09.827+0000 7f8130f9f700 1 --2- 172.21.15.42:0/3566750840 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f812c0a7730 0x7f812c08df70 secure :-1 s=READY pgs=1169 cs=0 l=1 rx=0x7f811c002f30 tx=0x7f811c00fc70).stop 2019-12-05T05:37:09.849 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:09.827+0000 7f8130f9f700 1 -- 172.21.15.42:0/3566750840 shutdown_connections 2019-12-05T05:37:09.850 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:09.827+0000 7f8130f9f700 1 --2- 172.21.15.42:0/3566750840 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f812c0a7730 0x7f812c08df70 unknown :-1 s=CLOSED pgs=1169 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:37:09.850 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:09.827+0000 7f8130f9f700 1 --2- 172.21.15.42:0/3566750840 >> v2:172.21.15.40:3300/0 conn(0x7f812c07e710 0x7f812c07eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:37:09.850 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:09.827+0000 7f8130f9f700 1 -- 172.21.15.42:0/3566750840 shutdown_connections 2019-12-05T05:37:09.851 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:09.827+0000 7f8130f9f700 1 -- 172.21.15.42:0/3566750840 wait complete. 2019-12-05T05:37:09.851 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:09.827+0000 7f8130f9f700 1 -- 172.21.15.42:0/3566750840 >> 172.21.15.42:0/3566750840 conn(0x7f812c0aabd0 msgr2=0x7f812c0a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:37:09.859 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:09.827+0000 7f8130f9f700 1 Processor -- start 2019-12-05T05:37:09.860 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:09.827+0000 7f8130f9f700 1 -- start start 2019-12-05T05:37:09.860 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:09.831+0000 7f8130f9f700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f812c11b0c0 0x7f812c07d710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:37:09.860 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:09.831+0000 7f8130f9f700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f812c122f10 con 0x7f812c07dc60 2019-12-05T05:37:09.865 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:09.831+0000 7f8130f9f700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f812c122400 con 0x7f812c135730 2019-12-05T05:37:09.865 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:09.831+0000 7f8130f9f700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f812c123200 con 0x7f812c11b0c0 2019-12-05T05:37:09.865 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:09.831+0000 7f812b59e700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f812c135730 0x7f812c137b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:37476/0 (socket says 172.21.15.42:37476) 2019-12-05T05:37:09.865 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:09.831+0000 7f812b59e700 1 -- 172.21.15.42:0/86024322 learned_addr learned my addr 172.21.15.42:0/86024322 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:37:09.865 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:09.831+0000 7f812ad9d700 1 --2- 172.21.15.42:0/86024322 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f812c11b0c0 0x7f812c07d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:37:09.866 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:09.831+0000 7f81237fe700 1 -- 172.21.15.42:0/86024322 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3050929307 0 0) 0x7f812c122400 con 0x7f812c135730 2019-12-05T05:37:09.866 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:09.831+0000 7f81237fe700 1 -- 172.21.15.42:0/86024322 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81100030b0 con 0x7f812c135730 2019-12-05T05:37:09.866 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:09.831+0000 7f81237fe700 1 -- 172.21.15.42:0/86024322 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 857897373 0 0) 0x7f812c122f10 con 0x7f812c07dc60 2019-12-05T05:37:09.866 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:09.831+0000 7f81237fe700 1 -- 172.21.15.42:0/86024322 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f812c122400 con 0x7f812c07dc60 2019-12-05T05:37:09.866 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:09.831+0000 7f81237fe700 1 -- 172.21.15.42:0/86024322 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 4118670241 0 0) 0x7f81100030b0 con 0x7f812c135730 2019-12-05T05:37:09.866 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:09.831+0000 7f81237fe700 1 -- 172.21.15.42:0/86024322 --> v1:172.21.15.42:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f812c122f10 con 0x7f812c135730 2019-12-05T05:37:09.867 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:09.831+0000 7f81237fe700 1 -- 172.21.15.42:0/86024322 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7f8124001c00 con 0x7f812c135730 2019-12-05T05:37:09.867 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:09.831+0000 7f81237fe700 1 -- 172.21.15.42:0/86024322 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f812c11b0c0 msgr2=0x7f812c07d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:37:09.867 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:09.831+0000 7f81237fe700 1 --2- 172.21.15.42:0/86024322 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f812c11b0c0 0x7f812c07d710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:37:09.867 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:09.831+0000 7f81237fe700 1 -- 172.21.15.42:0/86024322 >> v1:172.21.15.42:6789/0 conn(0x7f812c135730 legacy=0x7f812c137b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:37:09.867 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:09.831+0000 7f81237fe700 1 -- 172.21.15.42:0/86024322 >> v1:172.21.15.40:6789/0 conn(0x7f812c07dc60 legacy=0x7f812c124780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:37:09.867 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:09.831+0000 7f81237fe700 1 --2- 172.21.15.42:0/86024322 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8110003ab0 0x7f8110004420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:37:09.867 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:09.831+0000 7f81237fe700 1 --2- 172.21.15.42:0/86024322 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8110003f90 0x7f81100049d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:37:09.867 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:09.831+0000 7f81237fe700 1 --2- 172.21.15.42:0/86024322 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8110009660 0x7f811000baa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:37:09.868 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:09.831+0000 7f81237fe700 1 -- 172.21.15.42:0/86024322 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f812c123200 con 0x7f8110009660 2019-12-05T05:37:09.868 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:09.831+0000 7f81237fe700 1 -- 172.21.15.42:0/86024322 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f811000bff0 con 0x7f8110003f90 2019-12-05T05:37:09.868 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:09.831+0000 7f81237fe700 1 -- 172.21.15.42:0/86024322 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f811000c1b0 con 0x7f8110003ab0 2019-12-05T05:37:09.868 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:09.831+0000 7f812ad9d700 1 --2- 172.21.15.42:0/86024322 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f812c11b0c0 0x7f812c07d710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:37:09.868 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:09.831+0000 7f8123fff700 1 --2- 172.21.15.42:0/86024322 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8110003f90 0x7f81100049d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:37:09.868 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:09.831+0000 7f812ad9d700 1 --2- 172.21.15.42:0/86024322 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8110003ab0 0x7f8110004420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:37:09.868 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:09.831+0000 7f812b59e700 1 --2- 172.21.15.42:0/86024322 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8110009660 0x7f811000baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:37:09.869 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:09.831+0000 7f8123fff700 1 -- 172.21.15.42:0/86024322 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8110003ab0 msgr2=0x7f8110004420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:37:09.869 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:09.831+0000 7f8123fff700 1 --2- 172.21.15.42:0/86024322 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8110003ab0 0x7f8110004420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:37:09.869 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:09.831+0000 7f8123fff700 1 -- 172.21.15.42:0/86024322 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8110009660 msgr2=0x7f811000baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:37:09.869 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:09.831+0000 7f8123fff700 1 --2- 172.21.15.42:0/86024322 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8110009660 0x7f811000baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:37:09.869 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:09.831+0000 7f8123fff700 1 -- 172.21.15.42:0/86024322 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f812c124f80 con 0x7f8110003f90 2019-12-05T05:37:09.869 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:09.831+0000 7f8123fff700 1 --2- 172.21.15.42:0/86024322 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8110003f90 0x7f81100049d0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f8114003070 tx=0x7f8114010e00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:09.869 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:09.831+0000 7f8130f9f700 1 -- 172.21.15.42:0/86024322 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f812c122bd0 con 0x7f8110003f90 2019-12-05T05:37:09.870 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:09.831+0000 7f8130f9f700 1 -- 172.21.15.42:0/86024322 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f812c123740 con 0x7f8110003f90 2019-12-05T05:37:09.870 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:09.831+0000 7f8123fff700 1 --2- 172.21.15.42:0/86024322 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8110003f90 0x7f81100049d0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f8114003070 tx=0x7f8114010e00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:09.870 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:09.831+0000 7f8123fff700 1 --2- 172.21.15.42:0/86024322 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8110003f90 0x7f81100049d0 secure :-1 s=READY pgs=1171 cs=0 l=1 rx=0x7f8114003070 tx=0x7f8114010e00).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:37:09.870 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:09.831+0000 7f8123fff700 1 --2- 172.21.15.42:0/86024322 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8110003f90 0x7f81100049d0 secure :-1 s=THROTTLE_DONE pgs=1171 cs=0 l=1 rx=0x7f8114003070 tx=0x7f8114010e00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:09.870 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:09.831+0000 7f8123fff700 1 --2- 172.21.15.42:0/86024322 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8110003f90 0x7f81100049d0 secure :-1 s=THROTTLE_DONE pgs=1171 cs=0 l=1 rx=0x7f8114003070 tx=0x7f8114010e00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:09.870 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:09.831+0000 7f81237fe700 1 -- 172.21.15.42:0/86024322 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f811400e750 con 0x7f8110003f90 2019-12-05T05:37:09.870 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:09.831+0000 7f8123fff700 1 --2- 172.21.15.42:0/86024322 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8110003f90 0x7f81100049d0 secure :-1 s=THROTTLE_DONE pgs=1171 cs=0 l=1 rx=0x7f8114003070 tx=0x7f8114010e00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:09.871 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:09.831+0000 7f81237fe700 1 -- 172.21.15.42:0/86024322 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8114007180 con 0x7f8110003f90 2019-12-05T05:37:09.871 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:09.831+0000 7f81237fe700 1 -- 172.21.15.42:0/86024322 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f811400ed30 con 0x7f8110003f90 2019-12-05T05:37:09.871 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:09.831+0000 7f8123fff700 1 --2- 172.21.15.42:0/86024322 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8110003f90 0x7f81100049d0 secure :-1 s=THROTTLE_DONE pgs=1171 cs=0 l=1 rx=0x7f8114003070 tx=0x7f8114010e00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:09.871 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:09.835+0000 7f81237fe700 1 -- 172.21.15.42:0/86024322 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f8114005540 con 0x7f8110003f90 2019-12-05T05:37:09.871 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:09.835+0000 7f8123fff700 1 --2- 172.21.15.42:0/86024322 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8110003f90 0x7f81100049d0 secure :-1 s=THROTTLE_DONE pgs=1171 cs=0 l=1 rx=0x7f8114003070 tx=0x7f8114010e00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:09.871 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:09.835+0000 7f81237fe700 1 --2- 172.21.15.42:0/86024322 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8110044040 0x7f8110046550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:37:09.871 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:09.835+0000 7f81237fe700 1 -- 172.21.15.42:0/86024322 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 8009+0+0 (secure 0 0 0) 0x7f811404b210 con 0x7f8110003f90 2019-12-05T05:37:09.872 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:09.835+0000 7f812ad9d700 1 --2- 172.21.15.42:0/86024322 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8110044040 0x7f8110046550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:37:09.872 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:09.835+0000 7f812ad9d700 1 --2- 172.21.15.42:0/86024322 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8110044040 0x7f8110046550 crc :-1 s=READY pgs=506 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:37:09.872 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:09.839+0000 7f8130f9f700 1 -- 172.21.15.42:0/86024322 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8104003200 con 0x7f8110003f90 2019-12-05T05:37:09.872 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:09.847+0000 7f8123fff700 1 --2- 172.21.15.42:0/86024322 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8110003f90 0x7f81100049d0 secure :-1 s=THROTTLE_DONE pgs=1171 cs=0 l=1 rx=0x7f8114003070 tx=0x7f8114010e00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:09.872 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:09.847+0000 7f81237fe700 1 -- 172.21.15.42:0/86024322 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f811400d0a0 con 0x7f8110003f90 2019-12-05T05:37:10.004 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:37:10.008 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:37:10.011 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:10.007+0000 7f8130f9f700 1 -- 172.21.15.42:0/86024322 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8104003630 con 0x7f8110003f90 2019-12-05T05:37:10.012 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:10.007+0000 7f8123fff700 1 --2- 172.21.15.42:0/86024322 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8110003f90 0x7f81100049d0 secure :-1 s=THROTTLE_DONE pgs=1171 cs=0 l=1 rx=0x7f8114003070 tx=0x7f8114010e00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:10.012 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:10.007+0000 7f81237fe700 1 -- 172.21.15.42:0/86024322 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 144 v144) v1 ==== 95+0+1774 (secure 0 0 0) 0x7f81140322d0 con 0x7f8110003f90 2019-12-05T05:37:10.013 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 144 2019-12-05T05:37:10.013 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:37:10.013 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":144,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":144,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:37:08.827151+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5693},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5693":{"gid":5693,"name":"a","rank":0,"incarnation":144,"state":"up:replay","state_seq":2,"addr":"172.21.15.42:6835/3838877789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3838877789},{"type":"v1","addr":"172.21.15.42:6835","nonce":3838877789}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:37:10.016 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:10.007+0000 7f8130f9f700 1 -- 172.21.15.42:0/86024322 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8110044040 msgr2=0x7f8110046550 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:37:10.016 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:10.007+0000 7f8130f9f700 1 --2- 172.21.15.42:0/86024322 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8110044040 0x7f8110046550 crc :-1 s=READY pgs=506 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:37:10.016 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:10.007+0000 7f812b59e700 1 -- 172.21.15.42:0/86024322 reap_dead start 2019-12-05T05:37:10.016 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:10.011+0000 7f8130f9f700 1 -- 172.21.15.42:0/86024322 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8110003f90 msgr2=0x7f81100049d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:37:10.016 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:10.011+0000 7f8130f9f700 1 --2- 172.21.15.42:0/86024322 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8110003f90 0x7f81100049d0 secure :-1 s=READY pgs=1171 cs=0 l=1 rx=0x7f8114003070 tx=0x7f8114010e00).stop 2019-12-05T05:37:10.016 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:10.011+0000 7f8130f9f700 1 -- 172.21.15.42:0/86024322 shutdown_connections 2019-12-05T05:37:10.017 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:10.011+0000 7f8130f9f700 1 --2- 172.21.15.42:0/86024322 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8110003f90 0x7f81100049d0 unknown :-1 s=CLOSED pgs=1171 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:37:10.017 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:10.011+0000 7f8130f9f700 1 -- 172.21.15.42:0/86024322 shutdown_connections 2019-12-05T05:37:10.017 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:10.011+0000 7f8130f9f700 1 -- 172.21.15.42:0/86024322 wait complete. 2019-12-05T05:37:10.017 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:10.011+0000 7f8130f9f700 1 -- 172.21.15.42:0/86024322 >> 172.21.15.42:0/86024322 conn(0x7f812c0aabd0 msgr2=0x7f812c0a9780 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:37:10.027 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b down, removed from mdsmap 2019-12-05T05:37:10.027 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2019-12-05T05:37:10.028 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:37:10.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:10.167+0000 7fc2542d8700 1 Processor -- start 2019-12-05T05:37:10.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:10.167+0000 7fc2542d8700 1 -- start start 2019-12-05T05:37:10.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:10.167+0000 7fc2542d8700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc24c0a7740 0x7fc24c08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:37:10.184 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:10.167+0000 7fc2542d8700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc24c07e720 0x7fc24c07eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:37:10.184 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:10.167+0000 7fc2542d8700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc24c09f2e0 0x7fc24c09f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:37:10.184 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:10.167+0000 7fc2542d8700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc24c0a0b00 con 0x7fc24c0a7740 2019-12-05T05:37:10.185 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:10.167+0000 7fc2542d8700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc24c08e5b0 con 0x7fc24c09f2e0 2019-12-05T05:37:10.185 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:10.167+0000 7fc2542d8700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc24c08e850 con 0x7fc24c07e720 2019-12-05T05:37:10.185 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:10.167+0000 7fc253076700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc24c09f2e0 0x7fc24c09f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:37:10.185 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:10.167+0000 7fc253076700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc24c09f2e0 0x7fc24c09f710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:37452/0 (socket says 172.21.15.42:37452) 2019-12-05T05:37:10.185 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:10.167+0000 7fc253076700 1 -- 172.21.15.42:0/268893626 learned_addr learned my addr 172.21.15.42:0/268893626 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:37:10.186 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:10.167+0000 7fc252875700 1 --2- 172.21.15.42:0/268893626 >> v2:172.21.15.40:3300/0 conn(0x7fc24c0a7740 0x7fc24c08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:37:10.186 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:10.167+0000 7fc252074700 1 --2- 172.21.15.42:0/268893626 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc24c07e720 0x7fc24c07eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:37:10.186 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:10.167+0000 7fc253076700 1 -- 172.21.15.42:0/268893626 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc24c07e720 msgr2=0x7fc24c07eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:37:10.186 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:10.167+0000 7fc253076700 1 --2- 172.21.15.42:0/268893626 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc24c07e720 0x7fc24c07eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:37:10.186 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:10.167+0000 7fc253076700 1 -- 172.21.15.42:0/268893626 >> v2:172.21.15.40:3300/0 conn(0x7fc24c0a7740 msgr2=0x7fc24c08df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:37:10.186 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:10.167+0000 7fc253076700 1 --2- 172.21.15.42:0/268893626 >> v2:172.21.15.40:3300/0 conn(0x7fc24c0a7740 0x7fc24c08df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:37:10.187 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:10.167+0000 7fc253076700 1 -- 172.21.15.42:0/268893626 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc24c07f0a0 con 0x7fc24c09f2e0 2019-12-05T05:37:10.194 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:10.167+0000 7fc252074700 1 --2- 172.21.15.42:0/268893626 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc24c07e720 0x7fc24c07eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T05:37:10.194 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:10.167+0000 7fc253076700 1 --2- 172.21.15.42:0/268893626 >> v2:172.21.15.42:3300/0 conn(0x7fc24c09f2e0 0x7fc24c09f710 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fc24c08e770 tx=0x7fc23c00fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:10.194 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:10.167+0000 7fc253076700 1 --2- 172.21.15.42:0/268893626 >> v2:172.21.15.42:3300/0 conn(0x7fc24c09f2e0 0x7fc24c09f710 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fc24c08e770 tx=0x7fc23c00fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:10.194 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:10.167+0000 7fc253076700 1 --2- 172.21.15.42:0/268893626 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc24c09f2e0 0x7fc24c09f710 secure :-1 s=READY pgs=1172 cs=0 l=1 rx=0x7fc24c08e770 tx=0x7fc23c00fdb0).ready entity=mon.1 client_cookie=fced9ac032e312b9 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:37:10.195 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:10.167+0000 7fc253076700 1 --2- 172.21.15.42:0/268893626 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc24c09f2e0 0x7fc24c09f710 secure :-1 s=THROTTLE_DONE pgs=1172 cs=0 l=1 rx=0x7fc24c08e770 tx=0x7fc23c00fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:10.195 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:10.167+0000 7fc253076700 1 --2- 172.21.15.42:0/268893626 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc24c09f2e0 0x7fc24c09f710 secure :-1 s=THROTTLE_DONE pgs=1172 cs=0 l=1 rx=0x7fc24c08e770 tx=0x7fc23c00fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:10.195 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:10.167+0000 7fc251873700 1 -- 172.21.15.42:0/268893626 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc23c008c90 con 0x7fc24c09f2e0 2019-12-05T05:37:10.195 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:10.167+0000 7fc253076700 1 --2- 172.21.15.42:0/268893626 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc24c09f2e0 0x7fc24c09f710 secure :-1 s=THROTTLE_DONE pgs=1172 cs=0 l=1 rx=0x7fc24c08e770 tx=0x7fc23c00fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:10.195 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:10.167+0000 7fc251873700 1 -- 172.21.15.42:0/268893626 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc23c0088b0 con 0x7fc24c09f2e0 2019-12-05T05:37:10.195 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:10.167+0000 7fc251873700 1 -- 172.21.15.42:0/268893626 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc23c011b50 con 0x7fc24c09f2e0 2019-12-05T05:37:10.195 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:10.167+0000 7fc2542d8700 1 -- 172.21.15.42:0/268893626 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc24c09f2e0 msgr2=0x7fc24c09f710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:37:10.196 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:10.167+0000 7fc2542d8700 1 --2- 172.21.15.42:0/268893626 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc24c09f2e0 0x7fc24c09f710 secure :-1 s=READY pgs=1172 cs=0 l=1 rx=0x7fc24c08e770 tx=0x7fc23c00fdb0).stop 2019-12-05T05:37:10.196 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:10.171+0000 7fc2542d8700 1 -- 172.21.15.42:0/268893626 shutdown_connections 2019-12-05T05:37:10.197 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:10.171+0000 7fc2542d8700 1 --2- 172.21.15.42:0/268893626 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc24c09f2e0 0x7fc24c09f710 unknown :-1 s=CLOSED pgs=1172 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:37:10.198 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:10.171+0000 7fc2542d8700 1 --2- 172.21.15.42:0/268893626 >> v2:172.21.15.40:3300/0 conn(0x7fc24c0a7740 0x7fc24c08df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:37:10.198 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:10.171+0000 7fc2542d8700 1 --2- 172.21.15.42:0/268893626 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc24c07e720 0x7fc24c07eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:37:10.198 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:10.171+0000 7fc2542d8700 1 -- 172.21.15.42:0/268893626 shutdown_connections 2019-12-05T05:37:10.198 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:10.171+0000 7fc2542d8700 1 -- 172.21.15.42:0/268893626 wait complete. 2019-12-05T05:37:10.198 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:10.171+0000 7fc2542d8700 1 -- 172.21.15.42:0/268893626 >> 172.21.15.42:0/268893626 conn(0x7fc24c0aabe0 msgr2=0x7fc24c0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:37:10.199 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:10.171+0000 7fc2542d8700 1 Processor -- start 2019-12-05T05:37:10.199 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:10.171+0000 7fc2542d8700 1 -- start start 2019-12-05T05:37:10.199 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:10.171+0000 7fc2542d8700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc24c05a210 0x7fc24c1200f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:37:10.199 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:10.171+0000 7fc2542d8700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc24c120a60 0x7fc24c120e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:37:10.199 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:10.171+0000 7fc2542d8700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc24c131430 con 0x7fc24c131970 2019-12-05T05:37:10.200 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:10.171+0000 7fc2542d8700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc24c11eba0 con 0x7fc24c05a210 2019-12-05T05:37:10.200 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:10.171+0000 7fc2542d8700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc24c11ee40 con 0x7fc24c120a60 2019-12-05T05:37:10.200 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:10.171+0000 7fc253076700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fc24c131970 0x7fc24c08eb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:37494/0 (socket says 172.21.15.42:37494) 2019-12-05T05:37:10.200 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:10.171+0000 7fc253076700 1 -- 172.21.15.42:0/856223463 learned_addr learned my addr 172.21.15.42:0/856223463 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:37:10.200 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:10.171+0000 7fc252875700 1 --2- 172.21.15.42:0/856223463 >> v2:172.21.15.40:3300/0 conn(0x7fc24c05a210 0x7fc24c1200f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:37:10.200 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:10.171+0000 7fc24b7fe700 1 -- 172.21.15.42:0/856223463 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1987957163 0 0) 0x7fc24c131430 con 0x7fc24c131970 2019-12-05T05:37:10.201 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:10.171+0000 7fc24b7fe700 1 -- 172.21.15.42:0/856223463 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc22c0030b0 con 0x7fc24c131970 2019-12-05T05:37:10.201 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:10.171+0000 7fc252074700 1 --2- 172.21.15.42:0/856223463 >> v2:172.21.15.42:3300/0 conn(0x7fc24c120a60 0x7fc24c120e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:37:10.201 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:10.171+0000 7fc24b7fe700 1 -- 172.21.15.42:0/856223463 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 85358259 0 0) 0x7fc22c0030b0 con 0x7fc24c131970 2019-12-05T05:37:10.201 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:10.171+0000 7fc24b7fe700 1 -- 172.21.15.42:0/856223463 --> v1:172.21.15.42:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc24c131430 con 0x7fc24c131970 2019-12-05T05:37:10.201 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:10.171+0000 7fc24b7fe700 1 -- 172.21.15.42:0/856223463 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7fc23c008470 con 0x7fc24c131970 2019-12-05T05:37:10.201 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:10.171+0000 7fc24b7fe700 1 -- 172.21.15.42:0/856223463 >> v2:172.21.15.42:3300/0 conn(0x7fc24c120a60 msgr2=0x7fc24c120e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:37:10.201 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:10.171+0000 7fc24b7fe700 1 --2- 172.21.15.42:0/856223463 >> v2:172.21.15.42:3300/0 conn(0x7fc24c120a60 0x7fc24c120e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:37:10.202 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:10.171+0000 7fc24b7fe700 1 -- 172.21.15.42:0/856223463 >> v2:172.21.15.40:3300/0 conn(0x7fc24c05a210 msgr2=0x7fc24c1200f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:37:10.202 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:10.171+0000 7fc24b7fe700 1 --2- 172.21.15.42:0/856223463 >> v2:172.21.15.40:3300/0 conn(0x7fc24c05a210 0x7fc24c1200f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:37:10.202 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:10.171+0000 7fc24b7fe700 1 -- 172.21.15.42:0/856223463 >> v1:172.21.15.42:6789/0 conn(0x7fc24c131970 legacy=0x7fc24c08eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:37:10.202 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:10.171+0000 7fc24b7fe700 1 --2- 172.21.15.42:0/856223463 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc22c0038a0 0x7fc22c003d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:37:10.202 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:10.171+0000 7fc24b7fe700 1 --2- 172.21.15.42:0/856223463 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc22c004420 0x7fc22c004870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:37:10.202 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:10.171+0000 7fc24b7fe700 1 --2- 172.21.15.42:0/856223463 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc22c004f60 0x7fc22c00b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:37:10.203 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:10.171+0000 7fc24b7fe700 1 -- 172.21.15.42:0/856223463 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc24c11eba0 con 0x7fc22c004f60 2019-12-05T05:37:10.203 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:10.171+0000 7fc24b7fe700 1 -- 172.21.15.42:0/856223463 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc24c11ee40 con 0x7fc22c004420 2019-12-05T05:37:10.203 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:10.171+0000 7fc24b7fe700 1 -- 172.21.15.42:0/856223463 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc22c00beb0 con 0x7fc22c0038a0 2019-12-05T05:37:10.203 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:10.171+0000 7fc252074700 1 --2- 172.21.15.42:0/856223463 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc22c004420 0x7fc22c004870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:37:10.203 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:10.171+0000 7fc253076700 1 --2- 172.21.15.42:0/856223463 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc22c004f60 0x7fc22c00b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:37:10.203 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:10.175+0000 7fc253076700 1 -- 172.21.15.42:0/856223463 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc22c0038a0 msgr2=0x7fc22c003d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:37:10.203 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:10.175+0000 7fc253076700 1 --2- 172.21.15.42:0/856223463 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc22c0038a0 0x7fc22c003d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:37:10.204 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:10.175+0000 7fc253076700 1 -- 172.21.15.42:0/856223463 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc22c004420 msgr2=0x7fc22c004870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:37:10.204 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:10.175+0000 7fc253076700 1 --2- 172.21.15.42:0/856223463 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc22c004420 0x7fc22c004870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:37:10.204 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:10.175+0000 7fc253076700 1 -- 172.21.15.42:0/856223463 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc24c0a4120 con 0x7fc22c004f60 2019-12-05T05:37:10.205 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:10.175+0000 7fc252074700 1 --2- 172.21.15.42:0/856223463 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc22c004420 0x7fc22c004870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:37:10.205 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:10.175+0000 7fc2542d8700 1 -- 172.21.15.42:0/856223463 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc24c120640 con 0x7fc22c004f60 2019-12-05T05:37:10.205 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:10.175+0000 7fc253076700 1 --2- 172.21.15.42:0/856223463 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc22c004f60 0x7fc22c00b670 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fc23c0078c0 tx=0x7fc23c00f530).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:10.205 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:10.175+0000 7fc2542d8700 1 -- 172.21.15.42:0/856223463 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc24c131da0 con 0x7fc22c004f60 2019-12-05T05:37:10.206 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:10.175+0000 7fc253076700 1 -- 172.21.15.42:0/856223463 reap_dead start 2019-12-05T05:37:10.206 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:10.175+0000 7fc253076700 1 --2- 172.21.15.42:0/856223463 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc22c004f60 0x7fc22c00b670 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fc23c0078c0 tx=0x7fc23c00f530).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:10.206 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:10.175+0000 7fc253076700 1 --2- 172.21.15.42:0/856223463 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc22c004f60 0x7fc22c00b670 secure :-1 s=READY pgs=916 cs=0 l=1 rx=0x7fc23c0078c0 tx=0x7fc23c00f530).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:37:10.206 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:10.175+0000 7fc253076700 1 --2- 172.21.15.42:0/856223463 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc22c004f60 0x7fc22c00b670 secure :-1 s=THROTTLE_DONE pgs=916 cs=0 l=1 rx=0x7fc23c0078c0 tx=0x7fc23c00f530).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:10.207 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:10.175+0000 7fc24b7fe700 1 -- 172.21.15.42:0/856223463 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc23c005750 con 0x7fc22c004f60 2019-12-05T05:37:10.207 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:10.175+0000 7fc253076700 1 --2- 172.21.15.42:0/856223463 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc22c004f60 0x7fc22c00b670 secure :-1 s=THROTTLE_DONE pgs=916 cs=0 l=1 rx=0x7fc23c0078c0 tx=0x7fc23c00f530).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:10.207 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:10.175+0000 7fc24b7fe700 1 -- 172.21.15.42:0/856223463 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc23c007160 con 0x7fc22c004f60 2019-12-05T05:37:10.207 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:10.175+0000 7fc253076700 1 --2- 172.21.15.42:0/856223463 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc22c004f60 0x7fc22c00b670 secure :-1 s=THROTTLE_DONE pgs=916 cs=0 l=1 rx=0x7fc23c0078c0 tx=0x7fc23c00f530).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:10.207 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:10.175+0000 7fc24b7fe700 1 -- 172.21.15.42:0/856223463 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc23c005d30 con 0x7fc22c004f60 2019-12-05T05:37:10.207 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:10.175+0000 7fc253076700 1 --2- 172.21.15.42:0/856223463 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc22c004f60 0x7fc22c00b670 secure :-1 s=THROTTLE_DONE pgs=916 cs=0 l=1 rx=0x7fc23c0078c0 tx=0x7fc23c00f530).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:10.209 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:10.175+0000 7fc24b7fe700 1 -- 172.21.15.42:0/856223463 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fc23c010380 con 0x7fc22c004f60 2019-12-05T05:37:10.210 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:10.175+0000 7fc253076700 1 --2- 172.21.15.42:0/856223463 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc22c004f60 0x7fc22c00b670 secure :-1 s=THROTTLE_DONE pgs=916 cs=0 l=1 rx=0x7fc23c0078c0 tx=0x7fc23c00f530).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:10.210 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:10.175+0000 7fc24b7fe700 1 --2- 172.21.15.42:0/856223463 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc22c03e970 0x7fc22c040e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:37:10.210 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:10.175+0000 7fc24b7fe700 1 -- 172.21.15.42:0/856223463 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 8009+0+0 (secure 0 0 0) 0x7fc23c08fc50 con 0x7fc22c004f60 2019-12-05T05:37:10.210 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:10.175+0000 7fc252875700 1 --2- 172.21.15.42:0/856223463 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc22c03e970 0x7fc22c040e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:37:10.210 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:10.175+0000 7fc252875700 1 --2- 172.21.15.42:0/856223463 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc22c03e970 0x7fc22c040e80 crc :-1 s=READY pgs=507 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:37:10.210 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:10.183+0000 7fc2542d8700 1 -- 172.21.15.42:0/856223463 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc228003200 con 0x7fc22c004f60 2019-12-05T05:37:10.211 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:10.191+0000 7fc253076700 1 --2- 172.21.15.42:0/856223463 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc22c004f60 0x7fc22c00b670 secure :-1 s=THROTTLE_DONE pgs=916 cs=0 l=1 rx=0x7fc23c0078c0 tx=0x7fc23c00f530).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:10.211 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:10.191+0000 7fc24b7fe700 1 -- 172.21.15.42:0/856223463 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fc23c076cd0 con 0x7fc22c004f60 2019-12-05T05:37:10.360 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:10.351+0000 7fc2542d8700 1 -- 172.21.15.42:0/856223463 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc228003630 con 0x7fc22c004f60 2019-12-05T05:37:10.361 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:10.355+0000 7fc253076700 1 --2- 172.21.15.42:0/856223463 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc22c004f60 0x7fc22c00b670 secure :-1 s=THROTTLE_DONE pgs=916 cs=0 l=1 rx=0x7fc23c0078c0 tx=0x7fc23c00f530).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:10.361 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:10.355+0000 7fc24b7fe700 1 -- 172.21.15.42:0/856223463 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 144 v144) v1 ==== 95+0+1774 (secure 0 0 0) 0x7fc23c076f30 con 0x7fc22c004f60 2019-12-05T05:37:10.361 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 144 2019-12-05T05:37:10.361 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:37:10.361 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":144,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":144,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:37:08.827151+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5693},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5693":{"gid":5693,"name":"a","rank":0,"incarnation":144,"state":"up:replay","state_seq":2,"addr":"172.21.15.42:6835/3838877789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3838877789},{"type":"v1","addr":"172.21.15.42:6835","nonce":3838877789}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:37:10.363 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:10.355+0000 7fc2542d8700 1 -- 172.21.15.42:0/856223463 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc22c03e970 msgr2=0x7fc22c040e80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:37:10.363 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:10.355+0000 7fc2542d8700 1 --2- 172.21.15.42:0/856223463 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc22c03e970 0x7fc22c040e80 crc :-1 s=READY pgs=507 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:37:10.363 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:10.355+0000 7fc2542d8700 1 -- 172.21.15.42:0/856223463 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc22c004f60 msgr2=0x7fc22c00b670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:37:10.363 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:10.355+0000 7fc2542d8700 1 --2- 172.21.15.42:0/856223463 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc22c004f60 0x7fc22c00b670 secure :-1 s=READY pgs=916 cs=0 l=1 rx=0x7fc23c0078c0 tx=0x7fc23c00f530).stop 2019-12-05T05:37:10.363 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:10.355+0000 7fc2542d8700 1 -- 172.21.15.42:0/856223463 shutdown_connections 2019-12-05T05:37:10.363 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:10.355+0000 7fc2542d8700 1 --2- 172.21.15.42:0/856223463 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc22c03e970 0x7fc22c040e80 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:37:10.364 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:10.355+0000 7fc2542d8700 1 --2- 172.21.15.42:0/856223463 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc22c004f60 0x7fc22c00b670 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:37:10.364 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:10.355+0000 7fc2542d8700 1 -- 172.21.15.42:0/856223463 shutdown_connections 2019-12-05T05:37:10.364 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:10.355+0000 7fc2542d8700 1 -- 172.21.15.42:0/856223463 wait complete. 2019-12-05T05:37:10.364 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:10.355+0000 7fc2542d8700 1 -- 172.21.15.42:0/856223463 >> 172.21.15.42:0/856223463 conn(0x7fc24c0a80f0 msgr2=0x7fc24c11c3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:37:10.381 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 144, "feature_flags": { "enable_multiple": false, "ever_enabled_multiple": false }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2019-12-05T05:03:35.604738+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 144, "ever_allowed_features": 2, "explicitly_allowed_features": 2, "failed": [], "flags": 18, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5693": { "addr": "172.21.15.42:6835/3838877789", "addrs": { "addrvec": [ { "addr": "172.21.15.42:6834", "nonce": 3838877789, "type": "v2" }, { "addr": "172.21.15.42:6835", "nonce": 3838877789, "type": "v1" } ] }, "export_targets": [], "features": 4540138292836696063, "flags": 0, "gid": 5693, "incarnation": 144, "name": "a", "rank": 0, "state": "up:replay", "state_seq": 2 } }, "last_failure": 0, "last_failure_osd_epoch": 72, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 2, "min_compat_client": "0 (unknown)", "modified": "2019-12-05T05:37:08.827151+0000", "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5693 } } } ], "standbys": [] } 2019-12-05T05:37:12.383 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:37:12.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:12.483+0000 7fb9d02a1700 1 Processor -- start 2019-12-05T05:37:12.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:12.483+0000 7fb9d02a1700 1 -- start start 2019-12-05T05:37:12.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:12.483+0000 7fb9d02a1700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb9c80a7740 0x7fb9c808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:37:12.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:12.483+0000 7fb9d02a1700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb9c807e720 0x7fb9c807eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:37:12.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:12.483+0000 7fb9d02a1700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb9c809f2e0 0x7fb9c809f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:37:12.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:12.483+0000 7fb9d02a1700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9c80a0b00 con 0x7fb9c809f2e0 2019-12-05T05:37:12.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:12.483+0000 7fb9d02a1700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9c808e5b0 con 0x7fb9c807e720 2019-12-05T05:37:12.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:12.483+0000 7fb9d02a1700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9c808e850 con 0x7fb9c80a7740 2019-12-05T05:37:12.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:12.483+0000 7fb9ce03d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb9c807e720 0x7fb9c807eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:37:12.496 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:12.483+0000 7fb9ce03d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb9c807e720 0x7fb9c807eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:37470/0 (socket says 172.21.15.42:37470) 2019-12-05T05:37:12.496 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:12.483+0000 7fb9ce03d700 1 -- 172.21.15.42:0/2002652252 learned_addr learned my addr 172.21.15.42:0/2002652252 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:37:12.496 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:12.483+0000 7fb9ce83e700 1 --2- 172.21.15.42:0/2002652252 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb9c80a7740 0x7fb9c808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:37:12.496 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:12.483+0000 7fb9cf03f700 1 --2- 172.21.15.42:0/2002652252 >> v2:172.21.15.40:3300/0 conn(0x7fb9c809f2e0 0x7fb9c809f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:37:12.496 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:12.483+0000 7fb9ce03d700 1 -- 172.21.15.42:0/2002652252 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb9c80a7740 msgr2=0x7fb9c808df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:37:12.496 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:12.483+0000 7fb9ce03d700 1 --2- 172.21.15.42:0/2002652252 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb9c80a7740 0x7fb9c808df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:37:12.497 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:12.483+0000 7fb9ce03d700 1 -- 172.21.15.42:0/2002652252 >> v2:172.21.15.40:3300/0 conn(0x7fb9c809f2e0 msgr2=0x7fb9c809f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:37:12.497 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:12.483+0000 7fb9ce03d700 1 --2- 172.21.15.42:0/2002652252 >> v2:172.21.15.40:3300/0 conn(0x7fb9c809f2e0 0x7fb9c809f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:37:12.497 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:12.483+0000 7fb9ce03d700 1 -- 172.21.15.42:0/2002652252 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9c807f0a0 con 0x7fb9c807e720 2019-12-05T05:37:12.497 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:12.483+0000 7fb9ce83e700 1 --2- 172.21.15.42:0/2002652252 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb9c80a7740 0x7fb9c808df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T05:37:12.497 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:12.483+0000 7fb9cf03f700 1 --2- 172.21.15.42:0/2002652252 >> v2:172.21.15.40:3300/0 conn(0x7fb9c809f2e0 0x7fb9c809f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T05:37:12.497 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:12.483+0000 7fb9ce03d700 1 --2- 172.21.15.42:0/2002652252 >> v2:172.21.15.42:3300/0 conn(0x7fb9c807e720 0x7fb9c807eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fb9c808e770 tx=0x7fb9bc00fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:12.497 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:12.483+0000 7fb9ce03d700 1 --2- 172.21.15.42:0/2002652252 >> v2:172.21.15.42:3300/0 conn(0x7fb9c807e720 0x7fb9c807eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fb9c808e770 tx=0x7fb9bc00fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:12.497 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:12.483+0000 7fb9ce03d700 1 --2- 172.21.15.42:0/2002652252 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb9c807e720 0x7fb9c807eb50 secure :-1 s=READY pgs=1174 cs=0 l=1 rx=0x7fb9c808e770 tx=0x7fb9bc00fdb0).ready entity=mon.1 client_cookie=cf968bc18b0bcfdb server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:37:12.498 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:12.483+0000 7fb9ce03d700 1 --2- 172.21.15.42:0/2002652252 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb9c807e720 0x7fb9c807eb50 secure :-1 s=THROTTLE_DONE pgs=1174 cs=0 l=1 rx=0x7fb9c808e770 tx=0x7fb9bc00fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:12.498 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:12.483+0000 7fb9cd83c700 1 -- 172.21.15.42:0/2002652252 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb9bc008c90 con 0x7fb9c807e720 2019-12-05T05:37:12.498 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:12.487+0000 7fb9ce03d700 1 --2- 172.21.15.42:0/2002652252 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb9c807e720 0x7fb9c807eb50 secure :-1 s=THROTTLE_DONE pgs=1174 cs=0 l=1 rx=0x7fb9c808e770 tx=0x7fb9bc00fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:12.498 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:12.487+0000 7fb9ce03d700 1 --2- 172.21.15.42:0/2002652252 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb9c807e720 0x7fb9c807eb50 secure :-1 s=THROTTLE_DONE pgs=1174 cs=0 l=1 rx=0x7fb9c808e770 tx=0x7fb9bc00fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:12.498 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:12.487+0000 7fb9cd83c700 1 -- 172.21.15.42:0/2002652252 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9bc0088b0 con 0x7fb9c807e720 2019-12-05T05:37:12.498 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:12.487+0000 7fb9cd83c700 1 -- 172.21.15.42:0/2002652252 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb9bc011b50 con 0x7fb9c807e720 2019-12-05T05:37:12.499 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:12.487+0000 7fb9d02a1700 1 -- 172.21.15.42:0/2002652252 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb9c807e720 msgr2=0x7fb9c807eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:37:12.499 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:12.487+0000 7fb9d02a1700 1 --2- 172.21.15.42:0/2002652252 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb9c807e720 0x7fb9c807eb50 secure :-1 s=READY pgs=1174 cs=0 l=1 rx=0x7fb9c808e770 tx=0x7fb9bc00fdb0).stop 2019-12-05T05:37:12.499 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:12.487+0000 7fb9d02a1700 1 -- 172.21.15.42:0/2002652252 shutdown_connections 2019-12-05T05:37:12.499 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:12.487+0000 7fb9d02a1700 1 --2- 172.21.15.42:0/2002652252 >> v2:172.21.15.40:3300/0 conn(0x7fb9c809f2e0 0x7fb9c809f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:37:12.499 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:12.487+0000 7fb9d02a1700 1 --2- 172.21.15.42:0/2002652252 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb9c80a7740 0x7fb9c808df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:37:12.499 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:12.487+0000 7fb9d02a1700 1 --2- 172.21.15.42:0/2002652252 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb9c807e720 0x7fb9c807eb50 unknown :-1 s=CLOSED pgs=1174 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:37:12.499 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:12.487+0000 7fb9d02a1700 1 -- 172.21.15.42:0/2002652252 shutdown_connections 2019-12-05T05:37:12.499 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:12.487+0000 7fb9d02a1700 1 -- 172.21.15.42:0/2002652252 wait complete. 2019-12-05T05:37:12.500 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:12.487+0000 7fb9d02a1700 1 -- 172.21.15.42:0/2002652252 >> 172.21.15.42:0/2002652252 conn(0x7fb9c80aabe0 msgr2=0x7fb9c80a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:37:12.503 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:12.487+0000 7fb9d02a1700 1 Processor -- start 2019-12-05T05:37:12.504 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:12.487+0000 7fb9d02a1700 1 -- start start 2019-12-05T05:37:12.504 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:12.491+0000 7fb9d02a1700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb9c8142ea0 0x7fb9c807dac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:37:12.504 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:12.491+0000 7fb9d02a1700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9c811b510 con 0x7fb9c8124430 2019-12-05T05:37:12.504 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:12.491+0000 7fb9d02a1700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9c807e010 con 0x7fb9c81357b0 2019-12-05T05:37:12.504 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:12.491+0000 7fb9d02a1700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9c8142810 con 0x7fb9c8142ea0 2019-12-05T05:37:12.504 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:12.491+0000 7fb9cf03f700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fb9c81357b0 0x7fb9c8137bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:37518/0 (socket says 172.21.15.42:37518) 2019-12-05T05:37:12.505 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:12.491+0000 7fb9cf03f700 1 -- 172.21.15.42:0/3925238529 learned_addr learned my addr 172.21.15.42:0/3925238529 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:37:12.505 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:12.491+0000 7fb9ce83e700 1 --2- 172.21.15.42:0/3925238529 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb9c8142ea0 0x7fb9c807dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:37:12.505 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:12.491+0000 7fb9bb7fe700 1 -- 172.21.15.42:0/3925238529 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 627848399 0 0) 0x7fb9c807e010 con 0x7fb9c81357b0 2019-12-05T05:37:12.505 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:12.491+0000 7fb9bb7fe700 1 -- 172.21.15.42:0/3925238529 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9b00030b0 con 0x7fb9c81357b0 2019-12-05T05:37:12.505 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:12.491+0000 7fb9bb7fe700 1 -- 172.21.15.42:0/3925238529 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 729376929 0 0) 0x7fb9c811b510 con 0x7fb9c8124430 2019-12-05T05:37:12.505 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:12.491+0000 7fb9bb7fe700 1 -- 172.21.15.42:0/3925238529 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9c807e010 con 0x7fb9c8124430 2019-12-05T05:37:12.505 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:12.491+0000 7fb9ce83e700 1 -- 172.21.15.42:0/3925238529 >> v1:172.21.15.42:6789/0 conn(0x7fb9c81357b0 legacy=0x7fb9c8137bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:37:12.506 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:12.491+0000 7fb9ce83e700 1 -- 172.21.15.42:0/3925238529 >> v1:172.21.15.40:6789/0 conn(0x7fb9c8124430 legacy=0x7fb9c8144a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:37:12.506 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:12.491+0000 7fb9ce83e700 1 -- 172.21.15.42:0/3925238529 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9c8145290 con 0x7fb9c8142ea0 2019-12-05T05:37:12.506 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:12.491+0000 7fb9ce83e700 1 --2- 172.21.15.42:0/3925238529 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb9c8142ea0 0x7fb9c807dac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fb9c4002f30 tx=0x7fb9c4012b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:12.506 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:12.491+0000 7fb9ce83e700 1 --2- 172.21.15.42:0/3925238529 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb9c8142ea0 0x7fb9c807dac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fb9c4002f30 tx=0x7fb9c4012b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:12.506 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:12.491+0000 7fb9ce83e700 1 --2- 172.21.15.42:0/3925238529 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb9c8142ea0 0x7fb9c807dac0 secure :-1 s=READY pgs=243 cs=0 l=1 rx=0x7fb9c4002f30 tx=0x7fb9c4012b30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:37:12.506 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:12.491+0000 7fb9ce83e700 1 --2- 172.21.15.42:0/3925238529 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb9c8142ea0 0x7fb9c807dac0 secure :-1 s=THROTTLE_DONE pgs=243 cs=0 l=1 rx=0x7fb9c4002f30 tx=0x7fb9c4012b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:12.506 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:12.491+0000 7fb9bb7fe700 1 -- 172.21.15.42:0/3925238529 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb9c4005930 con 0x7fb9c8142ea0 2019-12-05T05:37:12.507 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:12.491+0000 7fb9ce83e700 1 --2- 172.21.15.42:0/3925238529 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb9c8142ea0 0x7fb9c807dac0 secure :-1 s=THROTTLE_DONE pgs=243 cs=0 l=1 rx=0x7fb9c4002f30 tx=0x7fb9c4012b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:12.507 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:12.491+0000 7fb9bb7fe700 1 -- 172.21.15.42:0/3925238529 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9c4007160 con 0x7fb9c8142ea0 2019-12-05T05:37:12.507 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:12.491+0000 7fb9d02a1700 1 -- 172.21.15.42:0/3925238529 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9c808eb70 con 0x7fb9c8142ea0 2019-12-05T05:37:12.507 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:12.491+0000 7fb9ce83e700 1 --2- 172.21.15.42:0/3925238529 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb9c8142ea0 0x7fb9c807dac0 secure :-1 s=THROTTLE_DONE pgs=243 cs=0 l=1 rx=0x7fb9c4002f30 tx=0x7fb9c4012b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:12.507 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:12.491+0000 7fb9bb7fe700 1 -- 172.21.15.42:0/3925238529 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb9c40141f0 con 0x7fb9c8142ea0 2019-12-05T05:37:12.507 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:12.491+0000 7fb9d02a1700 1 -- 172.21.15.42:0/3925238529 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9c808f0a0 con 0x7fb9c8142ea0 2019-12-05T05:37:12.507 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:12.491+0000 7fb9ce83e700 1 --2- 172.21.15.42:0/3925238529 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb9c8142ea0 0x7fb9c807dac0 secure :-1 s=THROTTLE_DONE pgs=243 cs=0 l=1 rx=0x7fb9c4002f30 tx=0x7fb9c4012b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:12.508 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:12.495+0000 7fb9bb7fe700 1 -- 172.21.15.42:0/3925238529 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fb9c4007bb0 con 0x7fb9c8142ea0 2019-12-05T05:37:12.508 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:12.495+0000 7fb9ce83e700 1 --2- 172.21.15.42:0/3925238529 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb9c8142ea0 0x7fb9c807dac0 secure :-1 s=THROTTLE_DONE pgs=243 cs=0 l=1 rx=0x7fb9c4002f30 tx=0x7fb9c4012b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:12.508 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:12.495+0000 7fb9bb7fe700 1 --2- 172.21.15.42:0/3925238529 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb9b003b470 0x7fb9b003d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:37:12.508 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:12.495+0000 7fb9bb7fe700 1 -- 172.21.15.42:0/3925238529 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 8009+0+0 (secure 0 0 0) 0x7fb9c404ce60 con 0x7fb9c8142ea0 2019-12-05T05:37:12.508 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:12.495+0000 7fb9ce03d700 1 --2- 172.21.15.42:0/3925238529 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb9b003b470 0x7fb9b003d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:37:12.508 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:12.495+0000 7fb9ce03d700 1 --2- 172.21.15.42:0/3925238529 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb9b003b470 0x7fb9b003d980 crc :-1 s=READY pgs=508 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:37:12.509 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:12.499+0000 7fb9d02a1700 1 -- 172.21.15.42:0/3925238529 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9c80a4590 con 0x7fb9c8142ea0 2019-12-05T05:37:12.515 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:12.507+0000 7fb9ce83e700 1 --2- 172.21.15.42:0/3925238529 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb9c8142ea0 0x7fb9c807dac0 secure :-1 s=THROTTLE_DONE pgs=243 cs=0 l=1 rx=0x7fb9c4002f30 tx=0x7fb9c4012b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:12.515 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:12.507+0000 7fb9bb7fe700 1 -- 172.21.15.42:0/3925238529 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fb9c4015060 con 0x7fb9c8142ea0 2019-12-05T05:37:12.680 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:12.675+0000 7fb9d02a1700 1 -- 172.21.15.42:0/3925238529 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb9c8143800 con 0x7fb9c8142ea0 2019-12-05T05:37:12.680 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:12.675+0000 7fb9ce83e700 1 --2- 172.21.15.42:0/3925238529 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb9c8142ea0 0x7fb9c807dac0 secure :-1 s=THROTTLE_DONE pgs=243 cs=0 l=1 rx=0x7fb9c4002f30 tx=0x7fb9c4012b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:12.680 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:12.675+0000 7fb9bb7fe700 1 -- 172.21.15.42:0/3925238529 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 145 v145) v1 ==== 95+0+1778 (secure 0 0 0) 0x7fb9c4033ec0 con 0x7fb9c8142ea0 2019-12-05T05:37:12.682 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:37:12.683 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":145,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":145,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:37:11.866434+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5693},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5693":{"gid":5693,"name":"a","rank":0,"incarnation":144,"state":"up:reconnect","state_seq":37,"addr":"172.21.15.42:6835/3838877789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3838877789},{"type":"v1","addr":"172.21.15.42:6835","nonce":3838877789}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:37:12.683 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 145 2019-12-05T05:37:12.685 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:12.679+0000 7fb9d02a1700 1 -- 172.21.15.42:0/3925238529 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb9b003b470 msgr2=0x7fb9b003d980 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:37:12.685 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:12.679+0000 7fb9d02a1700 1 --2- 172.21.15.42:0/3925238529 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb9b003b470 0x7fb9b003d980 crc :-1 s=READY pgs=508 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:37:12.685 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:12.679+0000 7fb9d02a1700 1 -- 172.21.15.42:0/3925238529 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb9c8142ea0 msgr2=0x7fb9c807dac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:37:12.685 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:12.679+0000 7fb9d02a1700 1 --2- 172.21.15.42:0/3925238529 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb9c8142ea0 0x7fb9c807dac0 secure :-1 s=READY pgs=243 cs=0 l=1 rx=0x7fb9c4002f30 tx=0x7fb9c4012b30).stop 2019-12-05T05:37:12.686 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:12.679+0000 7fb9d02a1700 1 -- 172.21.15.42:0/3925238529 shutdown_connections 2019-12-05T05:37:12.686 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:12.679+0000 7fb9d02a1700 1 --2- 172.21.15.42:0/3925238529 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb9b003b470 0x7fb9b003d980 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:37:12.686 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:12.679+0000 7fb9d02a1700 1 --2- 172.21.15.42:0/3925238529 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb9c8142ea0 0x7fb9c807dac0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:37:12.686 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:12.679+0000 7fb9d02a1700 1 -- 172.21.15.42:0/3925238529 shutdown_connections 2019-12-05T05:37:12.686 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:12.679+0000 7fb9d02a1700 1 -- 172.21.15.42:0/3925238529 wait complete. 2019-12-05T05:37:12.686 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:12.679+0000 7fb9d02a1700 1 -- 172.21.15.42:0/3925238529 >> 172.21.15.42:0/3925238529 conn(0x7fb9c80a8240 msgr2=0x7fb9c809f460 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:37:12.694 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2019-12-05T05:37:14.696 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:37:14.856 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:14.827+0000 7f41bc9c8700 1 Processor -- start 2019-12-05T05:37:14.856 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:14.827+0000 7f41bc9c8700 1 -- start start 2019-12-05T05:37:14.856 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:14.831+0000 7f41bc9c8700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f41b80a7740 0x7f41b808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:37:14.856 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:14.831+0000 7f41bc9c8700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f41b809fae0 0x7f41b807eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:37:14.857 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:14.831+0000 7f41bc9c8700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41b80a0710 con 0x7f41b807e720 2019-12-05T05:37:14.857 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:14.831+0000 7f41bc9c8700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41b80a0b00 con 0x7f41b809fae0 2019-12-05T05:37:14.857 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:14.831+0000 7f41bc9c8700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41b808e5b0 con 0x7f41b80a7740 2019-12-05T05:37:14.857 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:14.831+0000 7f41b759e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f41b809fae0 0x7f41b807eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:37:14.857 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:14.831+0000 7f41b759e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f41b809fae0 0x7f41b807eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:37494/0 (socket says 172.21.15.42:37494) 2019-12-05T05:37:14.857 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:14.831+0000 7f41b759e700 1 -- 172.21.15.42:0/1051235269 learned_addr learned my addr 172.21.15.42:0/1051235269 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:37:14.857 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:14.831+0000 7f41b6d9d700 1 --2- 172.21.15.42:0/1051235269 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f41b80a7740 0x7f41b808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:37:14.858 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:14.831+0000 7f41b759e700 1 -- 172.21.15.42:0/1051235269 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f41b80a7740 msgr2=0x7f41b808df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:37:14.858 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:14.831+0000 7f41b759e700 1 --2- 172.21.15.42:0/1051235269 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f41b80a7740 0x7f41b808df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:37:14.858 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:14.831+0000 7f41b759e700 1 -- 172.21.15.42:0/1051235269 >> v1:172.21.15.42:6789/0 conn(0x7f41b807e720 legacy=0x7f41b809f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:37:14.858 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:14.831+0000 7f41b759e700 1 -- 172.21.15.42:0/1051235269 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41b807f0a0 con 0x7f41b809fae0 2019-12-05T05:37:14.858 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:14.831+0000 7f41b759e700 1 --2- 172.21.15.42:0/1051235269 >> v2:172.21.15.42:3300/0 conn(0x7f41b809fae0 0x7f41b807eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f41b0008b10 tx=0x7f41b00100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:14.858 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:14.831+0000 7f41b759e700 1 --2- 172.21.15.42:0/1051235269 >> v2:172.21.15.42:3300/0 conn(0x7f41b809fae0 0x7f41b807eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f41b0008b10 tx=0x7f41b00100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:14.858 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:14.831+0000 7f41b759e700 1 --2- 172.21.15.42:0/1051235269 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f41b809fae0 0x7f41b807eb50 secure :-1 s=READY pgs=1177 cs=0 l=1 rx=0x7f41b0008b10 tx=0x7f41b00100b0).ready entity=mon.1 client_cookie=25749d8b8054968b server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:37:14.859 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:14.831+0000 7f41b759e700 1 --2- 172.21.15.42:0/1051235269 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f41b809fae0 0x7f41b807eb50 secure :-1 s=THROTTLE_DONE pgs=1177 cs=0 l=1 rx=0x7f41b0008b10 tx=0x7f41b00100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:14.859 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:14.831+0000 7f41b659c700 1 -- 172.21.15.42:0/1051235269 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f41b0010da0 con 0x7f41b809fae0 2019-12-05T05:37:14.859 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:14.831+0000 7f41b759e700 1 --2- 172.21.15.42:0/1051235269 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f41b809fae0 0x7f41b807eb50 secure :-1 s=THROTTLE_DONE pgs=1177 cs=0 l=1 rx=0x7f41b0008b10 tx=0x7f41b00100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:14.859 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:14.831+0000 7f41b759e700 1 --2- 172.21.15.42:0/1051235269 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f41b809fae0 0x7f41b807eb50 secure :-1 s=THROTTLE_DONE pgs=1177 cs=0 l=1 rx=0x7f41b0008b10 tx=0x7f41b00100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:14.859 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:14.831+0000 7f41b659c700 1 -- 172.21.15.42:0/1051235269 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41b00109c0 con 0x7f41b809fae0 2019-12-05T05:37:14.859 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:14.831+0000 7f41b659c700 1 -- 172.21.15.42:0/1051235269 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f41b00114a0 con 0x7f41b809fae0 2019-12-05T05:37:14.859 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:14.831+0000 7f41bc9c8700 1 -- 172.21.15.42:0/1051235269 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f41b809fae0 msgr2=0x7f41b807eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:37:14.860 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:14.831+0000 7f41bc9c8700 1 --2- 172.21.15.42:0/1051235269 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f41b809fae0 0x7f41b807eb50 secure :-1 s=READY pgs=1177 cs=0 l=1 rx=0x7f41b0008b10 tx=0x7f41b00100b0).stop 2019-12-05T05:37:14.860 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:14.831+0000 7f41bc9c8700 1 -- 172.21.15.42:0/1051235269 shutdown_connections 2019-12-05T05:37:14.860 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:14.831+0000 7f41bc9c8700 1 --2- 172.21.15.42:0/1051235269 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f41b809fae0 0x7f41b807eb50 unknown :-1 s=CLOSED pgs=1177 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:37:14.860 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:14.831+0000 7f41bc9c8700 1 --2- 172.21.15.42:0/1051235269 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f41b80a7740 0x7f41b808df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:37:14.860 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:14.831+0000 7f41bc9c8700 1 -- 172.21.15.42:0/1051235269 shutdown_connections 2019-12-05T05:37:14.860 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:14.831+0000 7f41bc9c8700 1 -- 172.21.15.42:0/1051235269 wait complete. 2019-12-05T05:37:14.860 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:14.831+0000 7f41bc9c8700 1 -- 172.21.15.42:0/1051235269 >> 172.21.15.42:0/1051235269 conn(0x7f41b80aabe0 msgr2=0x7f41b80a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:37:14.861 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:14.835+0000 7f41bc9c8700 1 Processor -- start 2019-12-05T05:37:14.861 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:14.835+0000 7f41bc9c8700 1 -- start start 2019-12-05T05:37:14.861 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:14.839+0000 7f41bc9c8700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f41b807b940 0x7f41b8118830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:37:14.861 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:14.839+0000 7f41bc9c8700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f41b811b5e0 0x7f41b812b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:37:14.861 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:14.839+0000 7f41bc9c8700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41b812ba60 con 0x7f41b808eeb0 2019-12-05T05:37:14.861 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:14.839+0000 7f41bc9c8700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41b807be50 con 0x7f41b811b5e0 2019-12-05T05:37:14.861 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:14.839+0000 7f41bc9c8700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41b807c0f0 con 0x7f41b807b940 2019-12-05T05:37:14.862 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:14.839+0000 7f41b6d9d700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f41b807b940 0x7f41b8118830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:37:14.862 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:14.839+0000 7f41b759e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f41b811b5e0 0x7f41b812b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:37:14.862 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:14.839+0000 7f41ae59c700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f41b808eeb0 0x7f41b811ade0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:37536/0 (socket says 172.21.15.42:37536) 2019-12-05T05:37:14.862 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:14.839+0000 7f41b6d9d700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f41b807b940 0x7f41b8118830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:37962/0 (socket says 172.21.15.42:37962) 2019-12-05T05:37:14.862 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:14.839+0000 7f41ae59c700 1 -- 172.21.15.42:0/2878623326 learned_addr learned my addr 172.21.15.42:0/2878623326 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:37:14.862 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:14.839+0000 7f41b6d9d700 1 -- 172.21.15.42:0/2878623326 >> v2:172.21.15.42:3300/0 conn(0x7f41b811b5e0 msgr2=0x7f41b812b510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:37:14.862 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:14.839+0000 7f41b6d9d700 1 --2- 172.21.15.42:0/2878623326 >> v2:172.21.15.42:3300/0 conn(0x7f41b811b5e0 0x7f41b812b510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:37:14.863 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:14.839+0000 7f41b6d9d700 1 -- 172.21.15.42:0/2878623326 >> v1:172.21.15.42:6789/0 conn(0x7f41b808eeb0 legacy=0x7f41b811ade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:37:14.863 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:14.839+0000 7f41affff700 1 -- 172.21.15.42:0/2878623326 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2363623617 0 0) 0x7f41b812ba60 con 0x7f41b808eeb0 2019-12-05T05:37:14.863 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:14.839+0000 7f41b6d9d700 1 -- 172.21.15.42:0/2878623326 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41b812bc90 con 0x7f41b807b940 2019-12-05T05:37:14.863 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:14.839+0000 7f41b6d9d700 1 --2- 172.21.15.42:0/2878623326 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f41b807b940 0x7f41b8118830 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f41a8002f30 tx=0x7f41a8010900).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:14.863 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:14.839+0000 7f41b6d9d700 1 --2- 172.21.15.42:0/2878623326 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f41b807b940 0x7f41b8118830 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f41a8002f30 tx=0x7f41a8010900).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:14.863 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:14.839+0000 7f41b6d9d700 1 --2- 172.21.15.42:0/2878623326 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f41b807b940 0x7f41b8118830 secure :-1 s=READY pgs=244 cs=0 l=1 rx=0x7f41a8002f30 tx=0x7f41a8010900).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:37:14.863 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:14.839+0000 7f41b6d9d700 1 --2- 172.21.15.42:0/2878623326 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f41b807b940 0x7f41b8118830 secure :-1 s=THROTTLE_DONE pgs=244 cs=0 l=1 rx=0x7f41a8002f30 tx=0x7f41a8010900).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:14.863 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:14.839+0000 7f41affff700 1 -- 172.21.15.42:0/2878623326 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f41a8005930 con 0x7f41b807b940 2019-12-05T05:37:14.864 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:14.839+0000 7f41b6d9d700 1 --2- 172.21.15.42:0/2878623326 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f41b807b940 0x7f41b8118830 secure :-1 s=THROTTLE_DONE pgs=244 cs=0 l=1 rx=0x7f41a8002f30 tx=0x7f41a8010900).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:14.864 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:14.839+0000 7f41affff700 1 -- 172.21.15.42:0/2878623326 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41a8007160 con 0x7f41b807b940 2019-12-05T05:37:14.864 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:14.839+0000 7f41b6d9d700 1 --2- 172.21.15.42:0/2878623326 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f41b807b940 0x7f41b8118830 secure :-1 s=THROTTLE_DONE pgs=244 cs=0 l=1 rx=0x7f41a8002f30 tx=0x7f41a8010900).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:14.864 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:14.839+0000 7f41bc9c8700 1 -- 172.21.15.42:0/2878623326 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41b812bea0 con 0x7f41b807b940 2019-12-05T05:37:14.864 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:14.839+0000 7f41affff700 1 -- 172.21.15.42:0/2878623326 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f41a80060c0 con 0x7f41b807b940 2019-12-05T05:37:14.864 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:14.839+0000 7f41bc9c8700 1 -- 172.21.15.42:0/2878623326 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41b812c280 con 0x7f41b807b940 2019-12-05T05:37:14.864 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:14.839+0000 7f41b6d9d700 1 --2- 172.21.15.42:0/2878623326 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f41b807b940 0x7f41b8118830 secure :-1 s=THROTTLE_DONE pgs=244 cs=0 l=1 rx=0x7f41a8002f30 tx=0x7f41a8010900).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:14.865 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:14.843+0000 7f41affff700 1 -- 172.21.15.42:0/2878623326 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f41a8007920 con 0x7f41b807b940 2019-12-05T05:37:14.865 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:14.843+0000 7f41b6d9d700 1 --2- 172.21.15.42:0/2878623326 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f41b807b940 0x7f41b8118830 secure :-1 s=THROTTLE_DONE pgs=244 cs=0 l=1 rx=0x7f41a8002f30 tx=0x7f41a8010900).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:14.865 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:14.843+0000 7f41affff700 1 --2- 172.21.15.42:0/2878623326 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f419c03ac10 0x7f419c03d120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:37:14.865 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:14.843+0000 7f41affff700 1 -- 172.21.15.42:0/2878623326 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 8009+0+0 (secure 0 0 0) 0x7f41a804ac80 con 0x7f41b807b940 2019-12-05T05:37:14.865 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:14.843+0000 7f41ae59c700 1 --2- 172.21.15.42:0/2878623326 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f419c03ac10 0x7f419c03d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:37:14.865 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:14.843+0000 7f41ae59c700 1 --2- 172.21.15.42:0/2878623326 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f419c03ac10 0x7f419c03d120 crc :-1 s=READY pgs=509 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:37:14.865 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:14.843+0000 7f41bc9c8700 1 -- 172.21.15.42:0/2878623326 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41b805d110 con 0x7f41b807b940 2019-12-05T05:37:14.866 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:14.855+0000 7f41b6d9d700 1 --2- 172.21.15.42:0/2878623326 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f41b807b940 0x7f41b8118830 secure :-1 s=THROTTLE_DONE pgs=244 cs=0 l=1 rx=0x7f41a8002f30 tx=0x7f41a8010900).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:14.866 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:14.855+0000 7f41affff700 1 -- 172.21.15.42:0/2878623326 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f41a8031ce0 con 0x7f41b807b940 2019-12-05T05:37:15.048 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:15.043+0000 7f41bc9c8700 1 -- 172.21.15.42:0/2878623326 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f41b805b8c0 con 0x7f41b807b940 2019-12-05T05:37:15.049 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:15.043+0000 7f41b6d9d700 1 --2- 172.21.15.42:0/2878623326 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f41b807b940 0x7f41b8118830 secure :-1 s=THROTTLE_DONE pgs=244 cs=0 l=1 rx=0x7f41a8002f30 tx=0x7f41a8010900).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:15.049 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:15.043+0000 7f41affff700 1 -- 172.21.15.42:0/2878623326 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 147 v147) v1 ==== 95+0+1775 (secure 0 0 0) 0x7f41a8031f40 con 0x7f41b807b940 2019-12-05T05:37:15.049 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 147 2019-12-05T05:37:15.049 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:37:15.049 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":147,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":147,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:37:13.880687+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5693},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5693":{"gid":5693,"name":"a","rank":0,"incarnation":144,"state":"up:active","state_seq":39,"addr":"172.21.15.42:6835/3838877789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3838877789},{"type":"v1","addr":"172.21.15.42:6835","nonce":3838877789}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:37:15.052 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:15.047+0000 7f41bc9c8700 1 -- 172.21.15.42:0/2878623326 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f419c03ac10 msgr2=0x7f419c03d120 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:37:15.052 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:15.047+0000 7f41bc9c8700 1 --2- 172.21.15.42:0/2878623326 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f419c03ac10 0x7f419c03d120 crc :-1 s=READY pgs=509 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:37:15.052 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:15.047+0000 7f41bc9c8700 1 -- 172.21.15.42:0/2878623326 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f41b807b940 msgr2=0x7f41b8118830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:37:15.053 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:15.047+0000 7f41bc9c8700 1 --2- 172.21.15.42:0/2878623326 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f41b807b940 0x7f41b8118830 secure :-1 s=READY pgs=244 cs=0 l=1 rx=0x7f41a8002f30 tx=0x7f41a8010900).stop 2019-12-05T05:37:15.053 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:15.047+0000 7f41bc9c8700 1 -- 172.21.15.42:0/2878623326 shutdown_connections 2019-12-05T05:37:15.053 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:15.047+0000 7f41bc9c8700 1 --2- 172.21.15.42:0/2878623326 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f419c03ac10 0x7f419c03d120 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:37:15.053 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:15.047+0000 7f41bc9c8700 1 --2- 172.21.15.42:0/2878623326 >> v2:172.21.15.42:3300/0 conn(0x7f41b811b5e0 0x7f41b812b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:37:15.053 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:15.047+0000 7f41bc9c8700 1 --2- 172.21.15.42:0/2878623326 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f41b807b940 0x7f41b8118830 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:37:15.053 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:15.047+0000 7f41bc9c8700 1 -- 172.21.15.42:0/2878623326 shutdown_connections 2019-12-05T05:37:15.053 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:15.047+0000 7f41bc9c8700 1 -- 172.21.15.42:0/2878623326 wait complete. 2019-12-05T05:37:15.054 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:15.047+0000 7f41bc9c8700 1 -- 172.21.15.42:0/2878623326 >> 172.21.15.42:0/2878623326 conn(0x7f41b80a7740 msgr2=0x7f41b8059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:37:15.062 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a has gained rank=0, replacing gid=7221 2019-12-05T05:37:15.062 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 9 secs before reviving mds.b 2019-12-05T05:37:24.063 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.b 2019-12-05T05:37:24.063 INFO:tasks.ceph.mds.b:Restarting daemon 2019-12-05T05:37:24.064 INFO:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2019-12-05T05:37:24.068 INFO:tasks.ceph.mds.b:Started 2019-12-05T05:37:24.070 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:37:24.129 INFO:tasks.ceph.mds.b.smithi040.stdout:starting mds.b at 2019-12-05T05:37:24.180 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:24.167+0000 7f8ac4fc1700 1 Processor -- start 2019-12-05T05:37:24.180 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:24.167+0000 7f8ac4fc1700 1 -- start start 2019-12-05T05:37:24.180 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:24.171+0000 7f8ac4fc1700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f8ac00a7740 0x7f8ac008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:37:24.180 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:24.171+0000 7f8ac4fc1700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8ac007e720 0x7f8ac007eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:37:24.180 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:24.171+0000 7f8ac4fc1700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ac00a0710 con 0x7f8ac009f2e0 2019-12-05T05:37:24.180 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:24.171+0000 7f8ac4fc1700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ac00a0b00 con 0x7f8ac007e720 2019-12-05T05:37:24.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:24.171+0000 7f8ac4fc1700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ac008e5b0 con 0x7f8ac00a7740 2019-12-05T05:37:24.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:24.171+0000 7f8abed9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f8ac00a7740 0x7f8ac008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:37:24.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:24.171+0000 7f8abf59e700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f8ac009f2e0 0x7f8ac009f710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:37580/0 (socket says 172.21.15.42:37580) 2019-12-05T05:37:24.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:24.171+0000 7f8abed9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f8ac00a7740 0x7f8ac008df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:37540/0 (socket says 172.21.15.42:37540) 2019-12-05T05:37:24.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:24.171+0000 7f8abf59e700 1 -- 172.21.15.42:0/1794232738 learned_addr learned my addr 172.21.15.42:0/1794232738 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:37:24.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:24.171+0000 7f8ab659c700 1 --2- 172.21.15.42:0/1794232738 >> v2:172.21.15.40:3300/0 conn(0x7f8ac007e720 0x7f8ac007eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:37:24.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:24.171+0000 7f8abed9d700 1 -- 172.21.15.42:0/1794232738 >> v2:172.21.15.40:3300/0 conn(0x7f8ac007e720 msgr2=0x7f8ac007eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:37:24.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:24.171+0000 7f8abed9d700 1 --2- 172.21.15.42:0/1794232738 >> v2:172.21.15.40:3300/0 conn(0x7f8ac007e720 0x7f8ac007eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:37:24.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:24.171+0000 7f8abed9d700 1 -- 172.21.15.42:0/1794232738 >> v1:172.21.15.42:6789/0 conn(0x7f8ac009f2e0 legacy=0x7f8ac009f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:37:24.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:24.171+0000 7f8abed9d700 1 -- 172.21.15.42:0/1794232738 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ac007f0a0 con 0x7f8ac00a7740 2019-12-05T05:37:24.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:24.171+0000 7f8abed9d700 1 --2- 172.21.15.42:0/1794232738 >> v2:172.21.15.42:3300/0 conn(0x7f8ac00a7740 0x7f8ac008df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f8ab0002f30 tx=0x7f8ab000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:24.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:24.171+0000 7f8abed9d700 1 --2- 172.21.15.42:0/1794232738 >> v2:172.21.15.42:3300/0 conn(0x7f8ac00a7740 0x7f8ac008df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f8ab0002f30 tx=0x7f8ab000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:24.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:24.171+0000 7f8abed9d700 1 --2- 172.21.15.42:0/1794232738 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8ac00a7740 0x7f8ac008df80 secure :-1 s=READY pgs=1182 cs=0 l=1 rx=0x7f8ab0002f30 tx=0x7f8ab000fc70).ready entity=mon.1 client_cookie=2370a6bd63e39148 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:37:24.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:24.171+0000 7f8abed9d700 1 --2- 172.21.15.42:0/1794232738 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8ac00a7740 0x7f8ac008df80 secure :-1 s=THROTTLE_DONE pgs=1182 cs=0 l=1 rx=0x7f8ab0002f30 tx=0x7f8ab000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:24.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:24.171+0000 7f8abed9d700 1 --2- 172.21.15.42:0/1794232738 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8ac00a7740 0x7f8ac008df80 secure :-1 s=THROTTLE_DONE pgs=1182 cs=0 l=1 rx=0x7f8ab0002f30 tx=0x7f8ab000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:24.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:24.171+0000 7f8abed9d700 1 --2- 172.21.15.42:0/1794232738 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8ac00a7740 0x7f8ac008df80 secure :-1 s=THROTTLE_DONE pgs=1182 cs=0 l=1 rx=0x7f8ab0002f30 tx=0x7f8ab000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:24.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:24.171+0000 7f8abe59c700 1 -- 172.21.15.42:0/1794232738 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8ab00112b0 con 0x7f8ac00a7740 2019-12-05T05:37:24.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:24.171+0000 7f8abe59c700 1 -- 172.21.15.42:0/1794232738 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ab0008ad0 con 0x7f8ac00a7740 2019-12-05T05:37:24.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:24.171+0000 7f8abe59c700 1 -- 172.21.15.42:0/1794232738 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8ab0011890 con 0x7f8ac00a7740 2019-12-05T05:37:24.184 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:24.171+0000 7f8ac4fc1700 1 -- 172.21.15.42:0/1794232738 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8ac00a7740 msgr2=0x7f8ac008df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:37:24.184 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:24.171+0000 7f8ac4fc1700 1 --2- 172.21.15.42:0/1794232738 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8ac00a7740 0x7f8ac008df80 secure :-1 s=READY pgs=1182 cs=0 l=1 rx=0x7f8ab0002f30 tx=0x7f8ab000fc70).stop 2019-12-05T05:37:24.184 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:24.171+0000 7f8ac4fc1700 1 -- 172.21.15.42:0/1794232738 shutdown_connections 2019-12-05T05:37:24.184 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:24.171+0000 7f8ac4fc1700 1 --2- 172.21.15.42:0/1794232738 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8ac00a7740 0x7f8ac008df80 unknown :-1 s=CLOSED pgs=1182 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:37:24.184 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:24.171+0000 7f8ac4fc1700 1 --2- 172.21.15.42:0/1794232738 >> v2:172.21.15.40:3300/0 conn(0x7f8ac007e720 0x7f8ac007eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:37:24.184 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:24.171+0000 7f8ac4fc1700 1 -- 172.21.15.42:0/1794232738 shutdown_connections 2019-12-05T05:37:24.185 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:24.171+0000 7f8ac4fc1700 1 -- 172.21.15.42:0/1794232738 wait complete. 2019-12-05T05:37:24.185 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:24.171+0000 7f8ac4fc1700 1 -- 172.21.15.42:0/1794232738 >> 172.21.15.42:0/1794232738 conn(0x7f8ac00aabe0 msgr2=0x7f8ac00a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:37:24.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:24.175+0000 7f8ac4fc1700 1 Processor -- start 2019-12-05T05:37:24.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:24.175+0000 7f8ac4fc1700 1 -- start start 2019-12-05T05:37:24.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:24.175+0000 7f8ac4fc1700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8ac00a1a30 0x7f8ac00a3520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:37:24.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:24.175+0000 7f8ac4fc1700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8ac0126e10 0x7f8ac0127240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:37:24.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:24.175+0000 7f8ac4fc1700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ac0127790 con 0x7f8ac0115b50 2019-12-05T05:37:24.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:24.175+0000 7f8ac4fc1700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ac00a1e60 con 0x7f8ac00a1a30 2019-12-05T05:37:24.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:24.175+0000 7f8ac4fc1700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ac00a20d0 con 0x7f8ac0126e10 2019-12-05T05:37:24.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:24.175+0000 7f8ab659c700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8ac00a1a30 0x7f8ac00a3520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:37:24.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:24.175+0000 7f8abed9d700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f8ac0115b50 0x7f8ac0144d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:47022/0 (socket says 172.21.15.42:47022) 2019-12-05T05:37:24.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:24.175+0000 7f8abf59e700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8ac0126e10 0x7f8ac0127240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:37:24.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:24.175+0000 7f8abed9d700 1 -- 172.21.15.42:0/1241580550 learned_addr learned my addr 172.21.15.42:0/1241580550 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:37:24.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:24.175+0000 7f8ab659c700 1 -- 172.21.15.42:0/1241580550 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8ac0126e10 msgr2=0x7f8ac0127240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:37:24.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:24.175+0000 7f8ab659c700 1 --2- 172.21.15.42:0/1241580550 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8ac0126e10 0x7f8ac0127240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:37:24.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:24.175+0000 7f8ab659c700 1 -- 172.21.15.42:0/1241580550 >> v1:172.21.15.40:6789/0 conn(0x7f8ac0115b50 legacy=0x7f8ac0144d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:37:24.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:24.175+0000 7f8ab7fff700 1 -- 172.21.15.42:0/1241580550 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1462158135 0 0) 0x7f8ac0127790 con 0x7f8ac0115b50 2019-12-05T05:37:24.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:24.175+0000 7f8ab659c700 1 -- 172.21.15.42:0/1241580550 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ac012f270 con 0x7f8ac00a1a30 2019-12-05T05:37:24.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:24.175+0000 7f8ab659c700 1 --2- 172.21.15.42:0/1241580550 >> v2:172.21.15.40:3300/0 conn(0x7f8ac00a1a30 0x7f8ac00a3520 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f8ac00a2290 tx=0x7f8aa8010ba0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:24.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:24.175+0000 7f8ab659c700 1 --2- 172.21.15.42:0/1241580550 >> v2:172.21.15.40:3300/0 conn(0x7f8ac00a1a30 0x7f8ac00a3520 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f8ac00a2290 tx=0x7f8aa8010ba0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:24.193 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:24.175+0000 7f8ab659c700 1 --2- 172.21.15.42:0/1241580550 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8ac00a1a30 0x7f8ac00a3520 secure :-1 s=READY pgs=922 cs=0 l=1 rx=0x7f8ac00a2290 tx=0x7f8aa8010ba0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:37:24.193 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:24.175+0000 7f8ab659c700 1 --2- 172.21.15.42:0/1241580550 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8ac00a1a30 0x7f8ac00a3520 secure :-1 s=THROTTLE_DONE pgs=922 cs=0 l=1 rx=0x7f8ac00a2290 tx=0x7f8aa8010ba0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:24.193 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:24.175+0000 7f8ab7fff700 1 -- 172.21.15.42:0/1241580550 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8aa8012610 con 0x7f8ac00a1a30 2019-12-05T05:37:24.193 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:24.175+0000 7f8ac4fc1700 1 -- 172.21.15.42:0/1241580550 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ac012e260 con 0x7f8ac00a1a30 2019-12-05T05:37:24.194 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:24.175+0000 7f8ab659c700 1 --2- 172.21.15.42:0/1241580550 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8ac00a1a30 0x7f8ac00a3520 secure :-1 s=THROTTLE_DONE pgs=922 cs=0 l=1 rx=0x7f8ac00a2290 tx=0x7f8aa8010ba0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:24.194 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:24.179+0000 7f8ab7fff700 1 -- 172.21.15.42:0/1241580550 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8aa8007150 con 0x7f8ac00a1a30 2019-12-05T05:37:24.194 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:24.179+0000 7f8ab659c700 1 --2- 172.21.15.42:0/1241580550 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8ac00a1a30 0x7f8ac00a3520 secure :-1 s=THROTTLE_DONE pgs=922 cs=0 l=1 rx=0x7f8ac00a2290 tx=0x7f8aa8010ba0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:24.194 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:24.179+0000 7f8ab7fff700 1 -- 172.21.15.42:0/1241580550 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8aa8012bf0 con 0x7f8ac00a1a30 2019-12-05T05:37:24.194 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:24.179+0000 7f8ac4fc1700 1 -- 172.21.15.42:0/1241580550 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ac012e810 con 0x7f8ac00a1a30 2019-12-05T05:37:24.194 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:24.179+0000 7f8ab659c700 1 --2- 172.21.15.42:0/1241580550 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8ac00a1a30 0x7f8ac00a3520 secure :-1 s=THROTTLE_DONE pgs=922 cs=0 l=1 rx=0x7f8ac00a2290 tx=0x7f8aa8010ba0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:24.194 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:24.179+0000 7f8ab7fff700 1 -- 172.21.15.42:0/1241580550 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f8aa8005540 con 0x7f8ac00a1a30 2019-12-05T05:37:24.195 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:24.179+0000 7f8ab659c700 1 --2- 172.21.15.42:0/1241580550 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8ac00a1a30 0x7f8ac00a3520 secure :-1 s=THROTTLE_DONE pgs=922 cs=0 l=1 rx=0x7f8ac00a2290 tx=0x7f8aa8010ba0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:24.195 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:24.179+0000 7f8ab7fff700 1 --2- 172.21.15.42:0/1241580550 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8aa403ce00 0x7f8aa403f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:37:24.195 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:24.179+0000 7f8ab7fff700 1 -- 172.21.15.42:0/1241580550 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 8009+0+0 (secure 0 0 0) 0x7f8aa804b1d0 con 0x7f8ac00a1a30 2019-12-05T05:37:24.195 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:24.179+0000 7f8abed9d700 1 --2- 172.21.15.42:0/1241580550 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8aa403ce00 0x7f8aa403f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:37:24.195 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:24.179+0000 7f8abed9d700 1 --2- 172.21.15.42:0/1241580550 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8aa403ce00 0x7f8aa403f310 crc :-1 s=READY pgs=511 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:37:24.195 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:24.183+0000 7f8ac4fc1700 1 -- 172.21.15.42:0/1241580550 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ac0145530 con 0x7f8ac00a1a30 2019-12-05T05:37:24.200 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:24.195+0000 7f8ab659c700 1 --2- 172.21.15.42:0/1241580550 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8ac00a1a30 0x7f8ac00a3520 secure :-1 s=THROTTLE_DONE pgs=922 cs=0 l=1 rx=0x7f8ac00a2290 tx=0x7f8aa8010ba0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:24.200 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:24.195+0000 7f8ab7fff700 1 -- 172.21.15.42:0/1241580550 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f8aa8030de0 con 0x7f8ac00a1a30 2019-12-05T05:37:24.366 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:24.359+0000 7f8ac4fc1700 1 -- 172.21.15.42:0/1241580550 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8ac005d110 con 0x7f8ac00a1a30 2019-12-05T05:37:24.367 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:24.359+0000 7f8ab659c700 1 --2- 172.21.15.42:0/1241580550 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8ac00a1a30 0x7f8ac00a3520 secure :-1 s=THROTTLE_DONE pgs=922 cs=0 l=1 rx=0x7f8ac00a2290 tx=0x7f8aa8010ba0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:24.367 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:24.359+0000 7f8ab7fff700 1 -- 172.21.15.42:0/1241580550 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 147 v147) v1 ==== 95+0+1775 (secure 0 0 0) 0x7f8aa8032230 con 0x7f8ac00a1a30 2019-12-05T05:37:24.372 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:37:24.372 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":147,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":147,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:37:13.880687+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5693},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5693":{"gid":5693,"name":"a","rank":0,"incarnation":144,"state":"up:active","state_seq":39,"addr":"172.21.15.42:6835/3838877789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3838877789},{"type":"v1","addr":"172.21.15.42:6835","nonce":3838877789}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:37:24.372 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 147 2019-12-05T05:37:24.372 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:24.367+0000 7f8ac4fc1700 1 -- 172.21.15.42:0/1241580550 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8aa403ce00 msgr2=0x7f8aa403f310 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:37:24.373 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:24.367+0000 7f8ac4fc1700 1 --2- 172.21.15.42:0/1241580550 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8aa403ce00 0x7f8aa403f310 crc :-1 s=READY pgs=511 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:37:24.373 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:24.367+0000 7f8ac4fc1700 1 -- 172.21.15.42:0/1241580550 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8ac00a1a30 msgr2=0x7f8ac00a3520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:37:24.373 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:24.367+0000 7f8ac4fc1700 1 --2- 172.21.15.42:0/1241580550 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8ac00a1a30 0x7f8ac00a3520 secure :-1 s=READY pgs=922 cs=0 l=1 rx=0x7f8ac00a2290 tx=0x7f8aa8010ba0).stop 2019-12-05T05:37:24.373 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:24.367+0000 7f8ac4fc1700 1 -- 172.21.15.42:0/1241580550 shutdown_connections 2019-12-05T05:37:24.373 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:24.367+0000 7f8ac4fc1700 1 --2- 172.21.15.42:0/1241580550 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8aa403ce00 0x7f8aa403f310 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:37:24.374 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:24.367+0000 7f8ac4fc1700 1 --2- 172.21.15.42:0/1241580550 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8ac0126e10 0x7f8ac0127240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:37:24.374 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:24.367+0000 7f8ac4fc1700 1 --2- 172.21.15.42:0/1241580550 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8ac00a1a30 0x7f8ac00a3520 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:37:24.374 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:24.367+0000 7f8ac4fc1700 1 -- 172.21.15.42:0/1241580550 shutdown_connections 2019-12-05T05:37:24.374 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:24.367+0000 7f8ac4fc1700 1 -- 172.21.15.42:0/1241580550 wait complete. 2019-12-05T05:37:24.374 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:24.367+0000 7f8ac4fc1700 1 -- 172.21.15.42:0/1241580550 >> 172.21.15.42:0/1241580550 conn(0x7f8ac00aabe0 msgr2=0x7f8ac009f240 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:37:24.382 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2019-12-05T05:37:26.388 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:37:26.508 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:26.491+0000 7f900c0e9700 1 Processor -- start 2019-12-05T05:37:26.508 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:26.491+0000 7f900c0e9700 1 -- start start 2019-12-05T05:37:26.508 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:26.491+0000 7f900c0e9700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f900408df80 0x7f900409f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:37:26.508 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:26.491+0000 7f900c0e9700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f900409f830 0x7f900414efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:37:26.509 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:26.491+0000 7f900c0e9700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f900409fc60 con 0x7f90040a7740 2019-12-05T05:37:26.509 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:26.491+0000 7f900c0e9700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90040a0b00 con 0x7f900408df80 2019-12-05T05:37:26.509 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:26.491+0000 7f900c0e9700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f900408e5b0 con 0x7f900409f830 2019-12-05T05:37:26.509 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:26.491+0000 7f900ae87700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f900409f830 0x7f900414efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:37:26.509 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:26.491+0000 7f900ae87700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f900409f830 0x7f900414efe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:37560/0 (socket says 172.21.15.42:37560) 2019-12-05T05:37:26.509 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:26.491+0000 7f900ae87700 1 -- 172.21.15.42:0/1264124306 learned_addr learned my addr 172.21.15.42:0/1264124306 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:37:26.509 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:26.491+0000 7f9009e85700 1 --2- 172.21.15.42:0/1264124306 >> v2:172.21.15.40:3300/0 conn(0x7f900408df80 0x7f900409f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:37:26.510 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:26.495+0000 7f900ae87700 1 -- 172.21.15.42:0/1264124306 >> v2:172.21.15.40:3300/0 conn(0x7f900408df80 msgr2=0x7f900409f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:37:26.510 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:26.495+0000 7f900ae87700 1 --2- 172.21.15.42:0/1264124306 >> v2:172.21.15.40:3300/0 conn(0x7f900408df80 0x7f900409f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:37:26.510 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:26.495+0000 7f900ae87700 1 -- 172.21.15.42:0/1264124306 >> v1:172.21.15.40:6789/0 conn(0x7f90040a7740 legacy=0x7f900407e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:37:26.510 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:26.495+0000 7f900ae87700 1 -- 172.21.15.42:0/1264124306 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90040a0710 con 0x7f900409f830 2019-12-05T05:37:26.510 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:26.495+0000 7f900ae87700 1 --2- 172.21.15.42:0/1264124306 >> v2:172.21.15.42:3300/0 conn(0x7f900409f830 0x7f900414efe0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f8ffc0032b0 tx=0x7f8ffc0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:26.510 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:26.495+0000 7f900ae87700 1 --2- 172.21.15.42:0/1264124306 >> v2:172.21.15.42:3300/0 conn(0x7f900409f830 0x7f900414efe0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f8ffc0032b0 tx=0x7f8ffc0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:26.511 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:26.495+0000 7f900ae87700 1 --2- 172.21.15.42:0/1264124306 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f900409f830 0x7f900414efe0 secure :-1 s=READY pgs=1183 cs=0 l=1 rx=0x7f8ffc0032b0 tx=0x7f8ffc0100b0).ready entity=mon.1 client_cookie=576c5f7bba316e1e server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:37:26.511 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:26.495+0000 7f900ae87700 1 --2- 172.21.15.42:0/1264124306 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f900409f830 0x7f900414efe0 secure :-1 s=THROTTLE_DONE pgs=1183 cs=0 l=1 rx=0x7f8ffc0032b0 tx=0x7f8ffc0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:26.511 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:26.495+0000 7f900ae87700 1 --2- 172.21.15.42:0/1264124306 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f900409f830 0x7f900414efe0 secure :-1 s=THROTTLE_DONE pgs=1183 cs=0 l=1 rx=0x7f8ffc0032b0 tx=0x7f8ffc0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:26.511 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:26.495+0000 7f9009684700 1 -- 172.21.15.42:0/1264124306 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8ffc010d60 con 0x7f900409f830 2019-12-05T05:37:26.511 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:26.495+0000 7f900ae87700 1 --2- 172.21.15.42:0/1264124306 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f900409f830 0x7f900414efe0 secure :-1 s=THROTTLE_DONE pgs=1183 cs=0 l=1 rx=0x7f8ffc0032b0 tx=0x7f8ffc0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:26.511 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:26.495+0000 7f9009684700 1 -- 172.21.15.42:0/1264124306 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ffc010980 con 0x7f900409f830 2019-12-05T05:37:26.511 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:26.495+0000 7f9009684700 1 -- 172.21.15.42:0/1264124306 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8ffc0114a0 con 0x7f900409f830 2019-12-05T05:37:26.512 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:26.495+0000 7f900c0e9700 1 -- 172.21.15.42:0/1264124306 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f900409f830 msgr2=0x7f900414efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:37:26.512 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:26.495+0000 7f900c0e9700 1 --2- 172.21.15.42:0/1264124306 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f900409f830 0x7f900414efe0 secure :-1 s=READY pgs=1183 cs=0 l=1 rx=0x7f8ffc0032b0 tx=0x7f8ffc0100b0).stop 2019-12-05T05:37:26.512 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:26.495+0000 7f900c0e9700 1 -- 172.21.15.42:0/1264124306 shutdown_connections 2019-12-05T05:37:26.512 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:26.495+0000 7f900c0e9700 1 --2- 172.21.15.42:0/1264124306 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f900409f830 0x7f900414efe0 unknown :-1 s=CLOSED pgs=1183 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:37:26.512 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:26.495+0000 7f900c0e9700 1 --2- 172.21.15.42:0/1264124306 >> v2:172.21.15.40:3300/0 conn(0x7f900408df80 0x7f900409f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:37:26.528 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:26.495+0000 7f900c0e9700 1 -- 172.21.15.42:0/1264124306 shutdown_connections 2019-12-05T05:37:26.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:26.495+0000 7f900c0e9700 1 -- 172.21.15.42:0/1264124306 wait complete. 2019-12-05T05:37:26.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:26.495+0000 7f900c0e9700 1 -- 172.21.15.42:0/1264124306 >> 172.21.15.42:0/1264124306 conn(0x7f90040aabe0 msgr2=0x7f90040a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:37:26.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:26.499+0000 7f900c0e9700 1 Processor -- start 2019-12-05T05:37:26.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:26.499+0000 7f900c0e9700 1 -- start start 2019-12-05T05:37:26.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:26.499+0000 7f900c0e9700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f900411b6a0 0x7f900407bf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:37:26.530 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:26.499+0000 7f900c0e9700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f900407c480 con 0x7f90041236d0 2019-12-05T05:37:26.530 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:26.499+0000 7f900c0e9700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9004116670 con 0x7f900411a8f0 2019-12-05T05:37:26.530 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:26.499+0000 7f900c0e9700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90041169f0 con 0x7f900411b6a0 2019-12-05T05:37:26.530 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:26.499+0000 7f9009e85700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f900411a8f0 0x7f9004138b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:37602/0 (socket says 172.21.15.42:37602) 2019-12-05T05:37:26.530 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:26.499+0000 7f9009e85700 1 -- 172.21.15.42:0/1662361623 learned_addr learned my addr 172.21.15.42:0/1662361623 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:37:26.530 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:26.499+0000 7f900ae87700 1 --2- 172.21.15.42:0/1662361623 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f900411b6a0 0x7f900407bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:37:26.530 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:26.499+0000 7f8ffaffd700 1 -- 172.21.15.42:0/1662361623 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 677945542 0 0) 0x7f9004116670 con 0x7f900411a8f0 2019-12-05T05:37:26.531 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:26.499+0000 7f8ffaffd700 1 -- 172.21.15.42:0/1662361623 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8fec0030b0 con 0x7f900411a8f0 2019-12-05T05:37:26.531 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:26.499+0000 7f8ffaffd700 1 -- 172.21.15.42:0/1662361623 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 382226247 0 0) 0x7f900407c480 con 0x7f90041236d0 2019-12-05T05:37:26.531 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:26.499+0000 7f8ffaffd700 1 -- 172.21.15.42:0/1662361623 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9004116670 con 0x7f90041236d0 2019-12-05T05:37:26.531 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:26.499+0000 7f8ffaffd700 1 -- 172.21.15.42:0/1662361623 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 895914494 0 0) 0x7f8fec0030b0 con 0x7f900411a8f0 2019-12-05T05:37:26.531 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:26.499+0000 7f900ae87700 1 -- 172.21.15.42:0/1662361623 >> v1:172.21.15.42:6789/0 conn(0x7f900411a8f0 legacy=0x7f9004138b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:37:26.531 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:26.499+0000 7f900ae87700 1 -- 172.21.15.42:0/1662361623 >> v1:172.21.15.40:6789/0 conn(0x7f90041236d0 legacy=0x7f900407ae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:37:26.531 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:26.499+0000 7f900ae87700 1 -- 172.21.15.42:0/1662361623 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f900411ae90 con 0x7f900411b6a0 2019-12-05T05:37:26.532 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:26.499+0000 7f900ae87700 1 --2- 172.21.15.42:0/1662361623 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f900411b6a0 0x7f900407bf30 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f8ff40016d0 tx=0x7f8ff4002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:26.532 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:26.499+0000 7f900ae87700 1 --2- 172.21.15.42:0/1662361623 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f900411b6a0 0x7f900407bf30 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f8ff40016d0 tx=0x7f8ff4002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:26.532 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:26.499+0000 7f900ae87700 1 --2- 172.21.15.42:0/1662361623 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f900411b6a0 0x7f900407bf30 secure :-1 s=READY pgs=245 cs=0 l=1 rx=0x7f8ff40016d0 tx=0x7f8ff4002ff0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:37:26.532 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:26.499+0000 7f900ae87700 1 --2- 172.21.15.42:0/1662361623 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f900411b6a0 0x7f900407bf30 secure :-1 s=THROTTLE_DONE pgs=245 cs=0 l=1 rx=0x7f8ff40016d0 tx=0x7f8ff4002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:26.532 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:26.499+0000 7f8ffaffd700 1 -- 172.21.15.42:0/1662361623 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8ff4001490 con 0x7f900411b6a0 2019-12-05T05:37:26.532 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:26.499+0000 7f900ae87700 1 --2- 172.21.15.42:0/1662361623 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f900411b6a0 0x7f900407bf30 secure :-1 s=THROTTLE_DONE pgs=245 cs=0 l=1 rx=0x7f8ff40016d0 tx=0x7f8ff4002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:26.532 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:26.499+0000 7f900ae87700 1 --2- 172.21.15.42:0/1662361623 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f900411b6a0 0x7f900407bf30 secure :-1 s=THROTTLE_DONE pgs=245 cs=0 l=1 rx=0x7f8ff40016d0 tx=0x7f8ff4002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:26.533 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:26.499+0000 7f8ffaffd700 1 -- 172.21.15.42:0/1662361623 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ff40012a0 con 0x7f900411b6a0 2019-12-05T05:37:26.533 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:26.499+0000 7f8ffaffd700 1 -- 172.21.15.42:0/1662361623 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8ffc005ca0 con 0x7f900411b6a0 2019-12-05T05:37:26.533 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:26.499+0000 7f900c0e9700 1 -- 172.21.15.42:0/1662361623 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f900413c330 con 0x7f900411b6a0 2019-12-05T05:37:26.533 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:26.499+0000 7f900c0e9700 1 -- 172.21.15.42:0/1662361623 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f900413c880 con 0x7f900411b6a0 2019-12-05T05:37:26.533 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:26.503+0000 7f900ae87700 1 --2- 172.21.15.42:0/1662361623 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f900411b6a0 0x7f900407bf30 secure :-1 s=THROTTLE_DONE pgs=245 cs=0 l=1 rx=0x7f8ff40016d0 tx=0x7f8ff4002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:26.534 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:26.503+0000 7f8ffaffd700 1 -- 172.21.15.42:0/1662361623 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f8ffc0117b0 con 0x7f900411b6a0 2019-12-05T05:37:26.535 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:26.503+0000 7f900ae87700 1 --2- 172.21.15.42:0/1662361623 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f900411b6a0 0x7f900407bf30 secure :-1 s=THROTTLE_DONE pgs=245 cs=0 l=1 rx=0x7f8ff40016d0 tx=0x7f8ff4002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:26.535 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:26.503+0000 7f8ffaffd700 1 --2- 172.21.15.42:0/1662361623 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8fec03b440 0x7f8fec03d950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:37:26.535 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:26.503+0000 7f8ffaffd700 1 -- 172.21.15.42:0/1662361623 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 8009+0+0 (secure 0 0 0) 0x7f8ffc04ae60 con 0x7f900411b6a0 2019-12-05T05:37:26.535 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:26.503+0000 7f900a686700 1 --2- 172.21.15.42:0/1662361623 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8fec03b440 0x7f8fec03d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:37:26.535 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:26.503+0000 7f900a686700 1 --2- 172.21.15.42:0/1662361623 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8fec03b440 0x7f8fec03d950 crc :-1 s=READY pgs=512 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:37:26.535 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:26.507+0000 7f900c0e9700 1 -- 172.21.15.42:0/1662361623 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f900407b680 con 0x7f900411b6a0 2019-12-05T05:37:26.536 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:26.515+0000 7f900ae87700 1 --2- 172.21.15.42:0/1662361623 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f900411b6a0 0x7f900407bf30 secure :-1 s=THROTTLE_DONE pgs=245 cs=0 l=1 rx=0x7f8ff40016d0 tx=0x7f8ff4002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:26.536 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:26.515+0000 7f8ffaffd700 1 -- 172.21.15.42:0/1662361623 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f8ffc031ee0 con 0x7f900411b6a0 2019-12-05T05:37:26.725 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:26.707+0000 7f900c0e9700 1 -- 172.21.15.42:0/1662361623 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f900405d110 con 0x7f900411b6a0 2019-12-05T05:37:26.726 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:26.711+0000 7f900ae87700 1 --2- 172.21.15.42:0/1662361623 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f900411b6a0 0x7f900407bf30 secure :-1 s=THROTTLE_DONE pgs=245 cs=0 l=1 rx=0x7f8ff40016d0 tx=0x7f8ff4002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:26.726 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:26.711+0000 7f8ffaffd700 1 -- 172.21.15.42:0/1662361623 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 147 v147) v1 ==== 95+0+1775 (secure 0 0 0) 0x7f8ffc032140 con 0x7f900411b6a0 2019-12-05T05:37:26.726 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 147 2019-12-05T05:37:26.726 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:26.711+0000 7f900c0e9700 1 -- 172.21.15.42:0/1662361623 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8fec03b440 msgr2=0x7f8fec03d950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:37:26.726 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:26.711+0000 7f900c0e9700 1 --2- 172.21.15.42:0/1662361623 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8fec03b440 0x7f8fec03d950 crc :-1 s=READY pgs=512 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:37:26.726 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:26.711+0000 7f900c0e9700 1 -- 172.21.15.42:0/1662361623 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f900411b6a0 msgr2=0x7f900407bf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:37:26.727 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:26.711+0000 7f900c0e9700 1 --2- 172.21.15.42:0/1662361623 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f900411b6a0 0x7f900407bf30 secure :-1 s=READY pgs=245 cs=0 l=1 rx=0x7f8ff40016d0 tx=0x7f8ff4002ff0).stop 2019-12-05T05:37:26.727 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:26.711+0000 7f900c0e9700 1 -- 172.21.15.42:0/1662361623 shutdown_connections 2019-12-05T05:37:26.727 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:26.711+0000 7f900c0e9700 1 --2- 172.21.15.42:0/1662361623 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8fec03b440 0x7f8fec03d950 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:37:26.727 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:26.711+0000 7f900c0e9700 1 --2- 172.21.15.42:0/1662361623 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f900411b6a0 0x7f900407bf30 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:37:26.727 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:26.711+0000 7f900c0e9700 1 -- 172.21.15.42:0/1662361623 shutdown_connections 2019-12-05T05:37:26.727 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:26.711+0000 7f900c0e9700 1 -- 172.21.15.42:0/1662361623 wait complete. 2019-12-05T05:37:26.727 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:26.711+0000 7f900c0e9700 1 -- 172.21.15.42:0/1662361623 >> 172.21.15.42:0/1662361623 conn(0x7f900408fda0 msgr2=0x7f9004059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:37:26.728 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:37:26.728 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":147,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":147,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:37:13.880687+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5693},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5693":{"gid":5693,"name":"a","rank":0,"incarnation":144,"state":"up:active","state_seq":39,"addr":"172.21.15.42:6835/3838877789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3838877789},{"type":"v1","addr":"172.21.15.42:6835","nonce":3838877789}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:37:26.734 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2019-12-05T05:37:28.736 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:37:28.838 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:28.827+0000 7f16035d3700 1 Processor -- start 2019-12-05T05:37:28.838 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:28.827+0000 7f16035d3700 1 -- start start 2019-12-05T05:37:28.838 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:28.827+0000 7f16035d3700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f15fc08df80 0x7f15fc09f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:37:28.839 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:28.827+0000 7f16035d3700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15fc09fc60 con 0x7f15fc0a7740 2019-12-05T05:37:28.839 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:28.827+0000 7f16035d3700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15fc0a0710 con 0x7f15fc09f830 2019-12-05T05:37:28.839 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:28.827+0000 7f16035d3700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15fc0a0b00 con 0x7f15fc08df80 2019-12-05T05:37:28.839 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:28.827+0000 7f1602371700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f15fc09f830 0x7f15fc14efe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:37624/0 (socket says 172.21.15.42:37624) 2019-12-05T05:37:28.839 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:28.827+0000 7f1602371700 1 -- 172.21.15.42:0/756161036 learned_addr learned my addr 172.21.15.42:0/756161036 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:37:28.839 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:28.827+0000 7f160136f700 1 --2- 172.21.15.42:0/756161036 >> v2:172.21.15.40:3300/0 conn(0x7f15fc08df80 0x7f15fc09f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:37:28.839 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:28.827+0000 7f1600b6e700 1 -- 172.21.15.42:0/756161036 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 661108208 0 0) 0x7f15fc0a0710 con 0x7f15fc09f830 2019-12-05T05:37:28.840 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:28.827+0000 7f1600b6e700 1 -- 172.21.15.42:0/756161036 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15ec0030b0 con 0x7f15fc09f830 2019-12-05T05:37:28.840 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:28.827+0000 7f1600b6e700 1 -- 172.21.15.42:0/756161036 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1159302756 0 0) 0x7f15ec0030b0 con 0x7f15fc09f830 2019-12-05T05:37:28.840 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:28.827+0000 7f160136f700 1 -- 172.21.15.42:0/756161036 >> v1:172.21.15.42:6789/0 conn(0x7f15fc09f830 legacy=0x7f15fc14efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:37:28.840 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:28.827+0000 7f160136f700 1 -- 172.21.15.42:0/756161036 >> v1:172.21.15.40:6789/0 conn(0x7f15fc0a7740 legacy=0x7f15fc07e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:37:28.840 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:28.827+0000 7f160136f700 1 -- 172.21.15.42:0/756161036 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15fc07ef20 con 0x7f15fc08df80 2019-12-05T05:37:28.841 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:28.827+0000 7f160136f700 1 --2- 172.21.15.42:0/756161036 >> v2:172.21.15.40:3300/0 conn(0x7f15fc08df80 0x7f15fc09f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f15e8008790 tx=0x7f15f4001820).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:28.841 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:28.827+0000 7f160136f700 1 --2- 172.21.15.42:0/756161036 >> v2:172.21.15.40:3300/0 conn(0x7f15fc08df80 0x7f15fc09f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f15e8008790 tx=0x7f15f4001820).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:28.842 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:28.827+0000 7f160136f700 1 --2- 172.21.15.42:0/756161036 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f15fc08df80 0x7f15fc09f2e0 secure :-1 s=READY pgs=926 cs=0 l=1 rx=0x7f15e8008790 tx=0x7f15f4001820).ready entity=mon.0 client_cookie=f136a76db17d37b6 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:37:28.842 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:28.831+0000 7f160136f700 1 --2- 172.21.15.42:0/756161036 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f15fc08df80 0x7f15fc09f2e0 secure :-1 s=THROTTLE_DONE pgs=926 cs=0 l=1 rx=0x7f15e8008790 tx=0x7f15f4001820).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:28.843 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:28.831+0000 7f1600b6e700 1 -- 172.21.15.42:0/756161036 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f15f4001ad0 con 0x7f15fc08df80 2019-12-05T05:37:28.843 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:28.831+0000 7f160136f700 1 --2- 172.21.15.42:0/756161036 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f15fc08df80 0x7f15fc09f2e0 secure :-1 s=THROTTLE_DONE pgs=926 cs=0 l=1 rx=0x7f15e8008790 tx=0x7f15f4001820).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:28.843 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:28.831+0000 7f160136f700 1 --2- 172.21.15.42:0/756161036 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f15fc08df80 0x7f15fc09f2e0 secure :-1 s=THROTTLE_DONE pgs=926 cs=0 l=1 rx=0x7f15e8008790 tx=0x7f15f4001820).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:28.843 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:28.831+0000 7f1600b6e700 1 -- 172.21.15.42:0/756161036 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15f40018e0 con 0x7f15fc08df80 2019-12-05T05:37:28.843 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:28.831+0000 7f16035d3700 1 -- 172.21.15.42:0/756161036 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f15fc08df80 msgr2=0x7f15fc09f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:37:28.843 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:28.831+0000 7f1600b6e700 1 -- 172.21.15.42:0/756161036 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f15e8011360 con 0x7f15fc08df80 2019-12-05T05:37:28.844 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:28.831+0000 7f16035d3700 1 --2- 172.21.15.42:0/756161036 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f15fc08df80 0x7f15fc09f2e0 secure :-1 s=READY pgs=926 cs=0 l=1 rx=0x7f15e8008790 tx=0x7f15f4001820).stop 2019-12-05T05:37:28.844 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:28.831+0000 7f16035d3700 1 -- 172.21.15.42:0/756161036 shutdown_connections 2019-12-05T05:37:28.844 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:28.831+0000 7f16035d3700 1 --2- 172.21.15.42:0/756161036 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f15fc08df80 0x7f15fc09f2e0 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:37:28.844 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:28.831+0000 7f16035d3700 1 -- 172.21.15.42:0/756161036 shutdown_connections 2019-12-05T05:37:28.845 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:28.831+0000 7f16035d3700 1 -- 172.21.15.42:0/756161036 wait complete. 2019-12-05T05:37:28.845 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:28.831+0000 7f16035d3700 1 -- 172.21.15.42:0/756161036 >> 172.21.15.42:0/756161036 conn(0x7f15fc0aabe0 msgr2=0x7f15fc0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:37:28.848 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:28.831+0000 7f16035d3700 1 Processor -- start 2019-12-05T05:37:28.849 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:28.831+0000 7f16035d3700 1 -- start start 2019-12-05T05:37:28.849 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:28.835+0000 7f16035d3700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f15fc079cd0 0x7f15fc120280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:37:28.849 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:28.835+0000 7f16035d3700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15fc14fa40 con 0x7f15fc0798a0 2019-12-05T05:37:28.849 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:28.835+0000 7f16035d3700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15fc07a290 con 0x7f15fc1207d0 2019-12-05T05:37:28.849 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:28.835+0000 7f16035d3700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15fc0a3fe0 con 0x7f15fc079cd0 2019-12-05T05:37:28.849 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:28.835+0000 7f1602371700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f15fc1207d0 0x7f15fc12a950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:37630/0 (socket says 172.21.15.42:37630) 2019-12-05T05:37:28.850 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:28.835+0000 7f1602371700 1 -- 172.21.15.42:0/3564040337 learned_addr learned my addr 172.21.15.42:0/3564040337 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:37:28.850 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:28.835+0000 7f160136f700 1 --2- 172.21.15.42:0/3564040337 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f15fc079cd0 0x7f15fc120280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:37:28.850 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:28.835+0000 7f15fa7fc700 1 -- 172.21.15.42:0/3564040337 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3681555141 0 0) 0x7f15fc14fa40 con 0x7f15fc0798a0 2019-12-05T05:37:28.850 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:28.835+0000 7f15fa7fc700 1 -- 172.21.15.42:0/3564040337 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15e40030b0 con 0x7f15fc0798a0 2019-12-05T05:37:28.850 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:28.835+0000 7f160136f700 1 -- 172.21.15.42:0/3564040337 >> v1:172.21.15.42:6789/0 conn(0x7f15fc1207d0 legacy=0x7f15fc12a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:37:28.850 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:28.835+0000 7f15fa7fc700 1 -- 172.21.15.42:0/3564040337 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 1790672109 0 0) 0x7f15e40030b0 con 0x7f15fc0798a0 2019-12-05T05:37:28.850 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:28.835+0000 7f160136f700 1 -- 172.21.15.42:0/3564040337 >> v1:172.21.15.40:6789/0 conn(0x7f15fc0798a0 legacy=0x7f15fc14f240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:37:28.851 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:28.835+0000 7f160136f700 1 -- 172.21.15.42:0/3564040337 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15fc120c00 con 0x7f15fc079cd0 2019-12-05T05:37:28.851 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:28.835+0000 7f160136f700 1 --2- 172.21.15.42:0/3564040337 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f15fc079cd0 0x7f15fc120280 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f15f0001b50 tx=0x7f15f0001b90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:28.851 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:28.835+0000 7f160136f700 1 --2- 172.21.15.42:0/3564040337 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f15fc079cd0 0x7f15fc120280 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f15f0001b50 tx=0x7f15f0001b90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:28.851 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:28.835+0000 7f160136f700 1 --2- 172.21.15.42:0/3564040337 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f15fc079cd0 0x7f15fc120280 secure :-1 s=READY pgs=246 cs=0 l=1 rx=0x7f15f0001b50 tx=0x7f15f0001b90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:37:28.851 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:28.835+0000 7f160136f700 1 --2- 172.21.15.42:0/3564040337 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f15fc079cd0 0x7f15fc120280 secure :-1 s=THROTTLE_DONE pgs=246 cs=0 l=1 rx=0x7f15f0001b50 tx=0x7f15f0001b90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:28.851 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:28.835+0000 7f15fa7fc700 1 -- 172.21.15.42:0/3564040337 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f15f0001950 con 0x7f15fc079cd0 2019-12-05T05:37:28.851 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:28.835+0000 7f160136f700 1 --2- 172.21.15.42:0/3564040337 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f15fc079cd0 0x7f15fc120280 secure :-1 s=THROTTLE_DONE pgs=246 cs=0 l=1 rx=0x7f15f0001b50 tx=0x7f15f0001b90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:28.852 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:28.835+0000 7f160136f700 1 --2- 172.21.15.42:0/3564040337 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f15fc079cd0 0x7f15fc120280 secure :-1 s=THROTTLE_DONE pgs=246 cs=0 l=1 rx=0x7f15f0001b50 tx=0x7f15f0001b90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:28.852 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:28.835+0000 7f15fa7fc700 1 -- 172.21.15.42:0/3564040337 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15f0001760 con 0x7f15fc079cd0 2019-12-05T05:37:28.852 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:28.835+0000 7f15fa7fc700 1 -- 172.21.15.42:0/3564040337 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f15e8011600 con 0x7f15fc079cd0 2019-12-05T05:37:28.852 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:28.835+0000 7f16035d3700 1 -- 172.21.15.42:0/3564040337 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15fc12c160 con 0x7f15fc079cd0 2019-12-05T05:37:28.852 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:28.835+0000 7f16035d3700 1 -- 172.21.15.42:0/3564040337 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15fc12c4f0 con 0x7f15fc079cd0 2019-12-05T05:37:28.852 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:28.835+0000 7f160136f700 1 --2- 172.21.15.42:0/3564040337 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f15fc079cd0 0x7f15fc120280 secure :-1 s=THROTTLE_DONE pgs=246 cs=0 l=1 rx=0x7f15f0001b50 tx=0x7f15f0001b90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:28.852 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:28.839+0000 7f15fa7fc700 1 -- 172.21.15.42:0/3564040337 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f15e80118d0 con 0x7f15fc079cd0 2019-12-05T05:37:28.852 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:28.839+0000 7f160136f700 1 --2- 172.21.15.42:0/3564040337 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f15fc079cd0 0x7f15fc120280 secure :-1 s=THROTTLE_DONE pgs=246 cs=0 l=1 rx=0x7f15f0001b50 tx=0x7f15f0001b90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:28.853 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:28.839+0000 7f15fa7fc700 1 --2- 172.21.15.42:0/3564040337 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f15e403b230 0x7f15e403d740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:37:28.853 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:28.839+0000 7f15fa7fc700 1 -- 172.21.15.42:0/3564040337 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 8009+0+0 (secure 0 0 0) 0x7f15e804acf0 con 0x7f15fc079cd0 2019-12-05T05:37:28.853 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:28.839+0000 7f1601b70700 1 --2- 172.21.15.42:0/3564040337 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f15e403b230 0x7f15e403d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:37:28.853 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:28.839+0000 7f1601b70700 1 --2- 172.21.15.42:0/3564040337 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f15e403b230 0x7f15e403d740 crc :-1 s=READY pgs=514 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:37:28.854 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:28.843+0000 7f16035d3700 1 -- 172.21.15.42:0/3564040337 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15fc05d110 con 0x7f15fc079cd0 2019-12-05T05:37:28.854 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:28.847+0000 7f160136f700 1 --2- 172.21.15.42:0/3564040337 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f15fc079cd0 0x7f15fc120280 secure :-1 s=THROTTLE_DONE pgs=246 cs=0 l=1 rx=0x7f15f0001b50 tx=0x7f15f0001b90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:28.854 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:28.847+0000 7f15fa7fc700 1 -- 172.21.15.42:0/3564040337 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f15e8011be0 con 0x7f15fc079cd0 2019-12-05T05:37:29.019 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:29.011+0000 7f16035d3700 1 -- 172.21.15.42:0/3564040337 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f15fc05b8c0 con 0x7f15fc079cd0 2019-12-05T05:37:29.020 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:29.015+0000 7f160136f700 1 --2- 172.21.15.42:0/3564040337 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f15fc079cd0 0x7f15fc120280 secure :-1 s=THROTTLE_DONE pgs=246 cs=0 l=1 rx=0x7f15f0001b50 tx=0x7f15f0001b90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:37:29.020 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:29.015+0000 7f15fa7fc700 1 -- 172.21.15.42:0/3564040337 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 148 v148) v1 ==== 95+0+2109 (secure 0 0 0) 0x7f15e8032be0 con 0x7f15fc079cd0 2019-12-05T05:37:29.022 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 148 2019-12-05T05:37:29.022 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:37:29.022 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":148,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7416,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/123054133","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":123054133},{"type":"v1","addr":"172.21.15.40:6833","nonce":123054133}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":148}],"filesystems":[{"mdsmap":{"epoch":147,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:37:13.880687+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5693},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5693":{"gid":5693,"name":"a","rank":0,"incarnation":144,"state":"up:active","state_seq":39,"addr":"172.21.15.42:6835/3838877789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3838877789},{"type":"v1","addr":"172.21.15.42:6835","nonce":3838877789}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:37:29.026 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:29.019+0000 7f16035d3700 1 -- 172.21.15.42:0/3564040337 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f15e403b230 msgr2=0x7f15e403d740 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:37:29.026 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:29.019+0000 7f16035d3700 1 --2- 172.21.15.42:0/3564040337 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f15e403b230 0x7f15e403d740 crc :-1 s=READY pgs=514 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:37:29.026 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:29.019+0000 7f16035d3700 1 -- 172.21.15.42:0/3564040337 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f15fc079cd0 msgr2=0x7f15fc120280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:37:29.026 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:29.019+0000 7f16035d3700 1 --2- 172.21.15.42:0/3564040337 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f15fc079cd0 0x7f15fc120280 secure :-1 s=READY pgs=246 cs=0 l=1 rx=0x7f15f0001b50 tx=0x7f15f0001b90).stop 2019-12-05T05:37:29.027 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:29.019+0000 7f16035d3700 1 -- 172.21.15.42:0/3564040337 shutdown_connections 2019-12-05T05:37:29.027 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:29.019+0000 7f16035d3700 1 --2- 172.21.15.42:0/3564040337 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f15e403b230 0x7f15e403d740 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:37:29.027 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:29.019+0000 7f16035d3700 1 --2- 172.21.15.42:0/3564040337 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f15fc079cd0 0x7f15fc120280 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:37:29.027 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:29.019+0000 7f16035d3700 1 -- 172.21.15.42:0/3564040337 shutdown_connections 2019-12-05T05:37:29.027 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:29.019+0000 7f16035d3700 1 -- 172.21.15.42:0/3564040337 wait complete. 2019-12-05T05:37:29.027 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:37:29.019+0000 7f16035d3700 1 -- 172.21.15.42:0/3564040337 >> 172.21.15.42:0/3564040337 conn(0x7f15fc08fde0 msgr2=0x7f15fc0a9790 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:37:29.034 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b reported in up:standby state 2019-12-05T05:37:29.035 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 80 secs before thrashing 2019-12-05T05:37:40.060 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:37:40.065 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:38:10.135 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:38:10.139 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:38:40.527 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:38:40.533 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:38:49.035 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:38:49.149 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.138+0000 7fd9e90b9700 1 Processor -- start 2019-12-05T05:38:49.149 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.138+0000 7fd9e90b9700 1 -- start start 2019-12-05T05:38:49.150 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.138+0000 7fd9e90b9700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd9e408df70 0x7fd9e409f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:38:49.150 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.138+0000 7fd9e90b9700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd9e409f820 0x7fd9e414efd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:38:49.150 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.138+0000 7fd9e90b9700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9e409fc50 con 0x7fd9e40a7730 2019-12-05T05:38:49.150 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.138+0000 7fd9e90b9700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9e40a0af0 con 0x7fd9e409f820 2019-12-05T05:38:49.150 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.138+0000 7fd9e90b9700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9e408e5a0 con 0x7fd9e408df70 2019-12-05T05:38:49.150 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.138+0000 7fd9e88b8700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd9e409f820 0x7fd9e414efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:38:49.151 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.138+0000 7fd9e359e700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fd9e40a7730 0x7fd9e407e710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:37908/0 (socket says 172.21.15.42:37908) 2019-12-05T05:38:49.151 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.138+0000 7fd9e359e700 1 -- 172.21.15.42:0/3544551308 learned_addr learned my addr 172.21.15.42:0/3544551308 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:38:49.151 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.138+0000 7fd9dbfff700 1 --2- 172.21.15.42:0/3544551308 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd9e408df70 0x7fd9e409f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:38:49.151 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.138+0000 7fd9e88b8700 1 -- 172.21.15.42:0/3544551308 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd9e408df70 msgr2=0x7fd9e409f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:38:49.151 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.138+0000 7fd9e88b8700 1 --2- 172.21.15.42:0/3544551308 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd9e408df70 0x7fd9e409f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:38:49.151 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.138+0000 7fd9e88b8700 1 -- 172.21.15.42:0/3544551308 >> v1:172.21.15.42:6789/0 conn(0x7fd9e40a7730 legacy=0x7fd9e407e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:38:49.151 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.138+0000 7fd9e2d9d700 1 -- 172.21.15.42:0/3544551308 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1715603249 0 0) 0x7fd9e409fc50 con 0x7fd9e40a7730 2019-12-05T05:38:49.152 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.138+0000 7fd9e88b8700 1 -- 172.21.15.42:0/3544551308 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9e40a0700 con 0x7fd9e409f820 2019-12-05T05:38:49.152 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.138+0000 7fd9e88b8700 1 --2- 172.21.15.42:0/3544551308 >> v2:172.21.15.42:3300/0 conn(0x7fd9e409f820 0x7fd9e414efd0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fd9dc008b10 tx=0x7fd9dc00fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:38:49.152 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.138+0000 7fd9e88b8700 1 --2- 172.21.15.42:0/3544551308 >> v2:172.21.15.42:3300/0 conn(0x7fd9e409f820 0x7fd9e414efd0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fd9dc008b10 tx=0x7fd9dc00fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:38:49.153 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.138+0000 7fd9e88b8700 1 --2- 172.21.15.42:0/3544551308 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd9e409f820 0x7fd9e414efd0 secure :-1 s=READY pgs=1189 cs=0 l=1 rx=0x7fd9dc008b10 tx=0x7fd9dc00fff0).ready entity=mon.1 client_cookie=521622a80e23575c server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:38:49.153 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.138+0000 7fd9e88b8700 1 --2- 172.21.15.42:0/3544551308 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd9e409f820 0x7fd9e414efd0 secure :-1 s=THROTTLE_DONE pgs=1189 cs=0 l=1 rx=0x7fd9dc008b10 tx=0x7fd9dc00fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:38:49.154 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.138+0000 7fd9e88b8700 1 --2- 172.21.15.42:0/3544551308 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd9e409f820 0x7fd9e414efd0 secure :-1 s=THROTTLE_DONE pgs=1189 cs=0 l=1 rx=0x7fd9dc008b10 tx=0x7fd9dc00fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:38:49.154 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.138+0000 7fd9e2d9d700 1 -- 172.21.15.42:0/3544551308 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd9dc010900 con 0x7fd9e409f820 2019-12-05T05:38:49.154 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.138+0000 7fd9e88b8700 1 --2- 172.21.15.42:0/3544551308 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd9e409f820 0x7fd9e414efd0 secure :-1 s=THROTTLE_DONE pgs=1189 cs=0 l=1 rx=0x7fd9dc008b10 tx=0x7fd9dc00fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:38:49.157 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.138+0000 7fd9e2d9d700 1 -- 172.21.15.42:0/3544551308 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9dc008b50 con 0x7fd9e409f820 2019-12-05T05:38:49.157 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.138+0000 7fd9e2d9d700 1 -- 172.21.15.42:0/3544551308 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd9dc0110c0 con 0x7fd9e409f820 2019-12-05T05:38:49.158 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.138+0000 7fd9e90b9700 1 -- 172.21.15.42:0/3544551308 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd9e409f820 msgr2=0x7fd9e414efd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:38:49.158 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.138+0000 7fd9e90b9700 1 --2- 172.21.15.42:0/3544551308 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd9e409f820 0x7fd9e414efd0 secure :-1 s=READY pgs=1189 cs=0 l=1 rx=0x7fd9dc008b10 tx=0x7fd9dc00fff0).stop 2019-12-05T05:38:49.158 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.142+0000 7fd9e90b9700 1 -- 172.21.15.42:0/3544551308 shutdown_connections 2019-12-05T05:38:49.159 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.142+0000 7fd9e90b9700 1 --2- 172.21.15.42:0/3544551308 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd9e409f820 0x7fd9e414efd0 unknown :-1 s=CLOSED pgs=1189 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:38:49.159 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.142+0000 7fd9e90b9700 1 --2- 172.21.15.42:0/3544551308 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd9e408df70 0x7fd9e409f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:38:49.159 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.142+0000 7fd9e90b9700 1 -- 172.21.15.42:0/3544551308 shutdown_connections 2019-12-05T05:38:49.159 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.142+0000 7fd9e90b9700 1 -- 172.21.15.42:0/3544551308 wait complete. 2019-12-05T05:38:49.159 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.142+0000 7fd9e90b9700 1 -- 172.21.15.42:0/3544551308 >> 172.21.15.42:0/3544551308 conn(0x7fd9e40aabd0 msgr2=0x7fd9e40a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:38:49.163 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.142+0000 7fd9e90b9700 1 Processor -- start 2019-12-05T05:38:49.164 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.142+0000 7fd9e90b9700 1 -- start start 2019-12-05T05:38:49.164 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.146+0000 7fd9e90b9700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd9e41166f0 0x7fd9e4116b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:38:49.164 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.146+0000 7fd9e90b9700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9e4127540 con 0x7fd9e4117dc0 2019-12-05T05:38:49.164 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.146+0000 7fd9e90b9700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9e41304b0 con 0x7fd9e408eea0 2019-12-05T05:38:49.164 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.146+0000 7fd9e90b9700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9e4146440 con 0x7fd9e41166f0 2019-12-05T05:38:49.164 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.146+0000 7fd9dbfff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fd9e408eea0 0x7fd9e4126d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:37916/0 (socket says 172.21.15.42:37916) 2019-12-05T05:38:49.165 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.146+0000 7fd9dbfff700 1 -- 172.21.15.42:0/1207128992 learned_addr learned my addr 172.21.15.42:0/1207128992 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:38:49.165 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.146+0000 7fd9e88b8700 1 --2- 172.21.15.42:0/1207128992 >> v2:172.21.15.40:3300/0 conn(0x7fd9e41166f0 0x7fd9e4116b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:38:49.166 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.146+0000 7fd9e359e700 1 -- 172.21.15.42:0/1207128992 >> v1:172.21.15.40:6789/0 conn(0x7fd9e4117dc0 legacy=0x7fd9e4145b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2019-12-05T05:38:49.166 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.146+0000 7fd9e359e700 1 -- 172.21.15.42:0/1207128992 >> v1:172.21.15.40:6789/0 conn(0x7fd9e4117dc0 legacy=0x7fd9e4145b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2019-12-05T05:38:49.166 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.146+0000 7fd9e359e700 1 --1- 172.21.15.42:0/1207128992 >> v1:172.21.15.40:6789/0 conn(0x7fd9e4117dc0 0x7fd9e4145b70 :-1 s=OPENED pgs=928 cs=1 l=1).handle_message read tag failed 2019-12-05T05:38:49.166 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.146+0000 7fd9e359e700 1 --1- 172.21.15.42:0/1207128992 >> v1:172.21.15.40:6789/0 conn(0x7fd9e4117dc0 0x7fd9e4145b70 :-1 s=OPENED pgs=928 cs=1 l=1).fault on lossy channel, failing 2019-12-05T05:38:49.167 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.146+0000 7fd9e0d99700 1 -- 172.21.15.42:0/1207128992 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 446955326 0 0) 0x7fd9e41304b0 con 0x7fd9e408eea0 2019-12-05T05:38:49.167 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.146+0000 7fd9e0d99700 1 -- 172.21.15.42:0/1207128992 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9c80030b0 con 0x7fd9e408eea0 2019-12-05T05:38:49.167 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.146+0000 7fd9e88b8700 1 -- 172.21.15.42:0/1207128992 >> v1:172.21.15.42:6789/0 conn(0x7fd9e408eea0 legacy=0x7fd9e4126d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:38:49.167 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.146+0000 7fd9e88b8700 1 -- 172.21.15.42:0/1207128992 >> v1:172.21.15.40:6789/0 conn(0x7fd9e4117dc0 legacy=0x7fd9e4145b70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2019-12-05T05:38:49.167 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.146+0000 7fd9e88b8700 1 -- 172.21.15.42:0/1207128992 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9e413a930 con 0x7fd9e41166f0 2019-12-05T05:38:49.167 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.146+0000 7fd9e88b8700 1 --2- 172.21.15.42:0/1207128992 >> v2:172.21.15.40:3300/0 conn(0x7fd9e41166f0 0x7fd9e4116b20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fd9dc008b10 tx=0x7fd9dc0112f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:38:49.167 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.146+0000 7fd9e88b8700 1 --2- 172.21.15.42:0/1207128992 >> v2:172.21.15.40:3300/0 conn(0x7fd9e41166f0 0x7fd9e4116b20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fd9dc008b10 tx=0x7fd9dc0112f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:38:49.168 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.146+0000 7fd9e88b8700 1 --2- 172.21.15.42:0/1207128992 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd9e41166f0 0x7fd9e4116b20 secure :-1 s=READY pgs=929 cs=0 l=1 rx=0x7fd9dc008b10 tx=0x7fd9dc0112f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:38:49.168 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.146+0000 7fd9e88b8700 1 --2- 172.21.15.42:0/1207128992 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd9e41166f0 0x7fd9e4116b20 secure :-1 s=THROTTLE_DONE pgs=929 cs=0 l=1 rx=0x7fd9dc008b10 tx=0x7fd9dc0112f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:38:49.168 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.146+0000 7fd9e0d99700 1 -- 172.21.15.42:0/1207128992 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd9dc005ab0 con 0x7fd9e41166f0 2019-12-05T05:38:49.168 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.146+0000 7fd9e88b8700 1 --2- 172.21.15.42:0/1207128992 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd9e41166f0 0x7fd9e4116b20 secure :-1 s=THROTTLE_DONE pgs=929 cs=0 l=1 rx=0x7fd9dc008b10 tx=0x7fd9dc0112f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:38:49.168 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.146+0000 7fd9e88b8700 1 --2- 172.21.15.42:0/1207128992 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd9e41166f0 0x7fd9e4116b20 secure :-1 s=THROTTLE_DONE pgs=929 cs=0 l=1 rx=0x7fd9dc008b10 tx=0x7fd9dc0112f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:38:49.168 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.146+0000 7fd9e0d99700 1 -- 172.21.15.42:0/1207128992 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9dc010b00 con 0x7fd9e41166f0 2019-12-05T05:38:49.172 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.146+0000 7fd9e90b9700 1 -- 172.21.15.42:0/1207128992 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9e4138910 con 0x7fd9e41166f0 2019-12-05T05:38:49.172 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.146+0000 7fd9e0d99700 1 -- 172.21.15.42:0/1207128992 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd9dc00f470 con 0x7fd9e41166f0 2019-12-05T05:38:49.172 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.146+0000 7fd9e90b9700 1 -- 172.21.15.42:0/1207128992 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9e4138de0 con 0x7fd9e41166f0 2019-12-05T05:38:49.172 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.146+0000 7fd9e88b8700 1 --2- 172.21.15.42:0/1207128992 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd9e41166f0 0x7fd9e4116b20 secure :-1 s=THROTTLE_DONE pgs=929 cs=0 l=1 rx=0x7fd9dc008b10 tx=0x7fd9dc0112f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:38:49.172 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.150+0000 7fd9e0d99700 1 -- 172.21.15.42:0/1207128992 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fd9dc00f740 con 0x7fd9e41166f0 2019-12-05T05:38:49.173 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.150+0000 7fd9e88b8700 1 --2- 172.21.15.42:0/1207128992 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd9e41166f0 0x7fd9e4116b20 secure :-1 s=THROTTLE_DONE pgs=929 cs=0 l=1 rx=0x7fd9dc008b10 tx=0x7fd9dc0112f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:38:49.173 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.150+0000 7fd9e0d99700 1 --2- 172.21.15.42:0/1207128992 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd9c803b250 0x7fd9c803d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:38:49.173 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.150+0000 7fd9e0d99700 1 -- 172.21.15.42:0/1207128992 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 8009+0+0 (secure 0 0 0) 0x7fd9dc04aff0 con 0x7fd9e41166f0 2019-12-05T05:38:49.173 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.150+0000 7fd9e359e700 1 --2- 172.21.15.42:0/1207128992 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd9c803b250 0x7fd9c803d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:38:49.173 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.150+0000 7fd9e359e700 1 --2- 172.21.15.42:0/1207128992 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd9c803b250 0x7fd9c803d760 crc :-1 s=READY pgs=516 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:38:49.173 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.150+0000 7fd9e90b9700 1 -- 172.21.15.42:0/1207128992 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9e405d100 con 0x7fd9e41166f0 2019-12-05T05:38:49.174 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.162+0000 7fd9e88b8700 1 --2- 172.21.15.42:0/1207128992 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd9e41166f0 0x7fd9e4116b20 secure :-1 s=THROTTLE_DONE pgs=929 cs=0 l=1 rx=0x7fd9dc008b10 tx=0x7fd9dc0112f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:38:49.174 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.162+0000 7fd9e0d99700 1 -- 172.21.15.42:0/1207128992 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fd9dc0322d0 con 0x7fd9e41166f0 2019-12-05T05:38:49.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.342+0000 7fd9e90b9700 1 -- 172.21.15.42:0/1207128992 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd9e405b8b0 con 0x7fd9e41166f0 2019-12-05T05:38:49.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.342+0000 7fd9e88b8700 1 --2- 172.21.15.42:0/1207128992 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd9e41166f0 0x7fd9e4116b20 secure :-1 s=THROTTLE_DONE pgs=929 cs=0 l=1 rx=0x7fd9dc008b10 tx=0x7fd9dc0112f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:38:49.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.342+0000 7fd9e0d99700 1 -- 172.21.15.42:0/1207128992 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 148 v148) v1 ==== 95+0+2109 (secure 0 0 0) 0x7fd9dc032e80 con 0x7fd9e41166f0 2019-12-05T05:38:49.351 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:38:49.351 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":148,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7416,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/123054133","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":123054133},{"type":"v1","addr":"172.21.15.40:6833","nonce":123054133}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":148}],"filesystems":[{"mdsmap":{"epoch":147,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:37:13.880687+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5693},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5693":{"gid":5693,"name":"a","rank":0,"incarnation":144,"state":"up:active","state_seq":39,"addr":"172.21.15.42:6835/3838877789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3838877789},{"type":"v1","addr":"172.21.15.42:6835","nonce":3838877789}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:38:49.351 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 148 2019-12-05T05:38:49.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.346+0000 7fd9e90b9700 1 -- 172.21.15.42:0/1207128992 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd9c803b250 msgr2=0x7fd9c803d760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:38:49.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.346+0000 7fd9e90b9700 1 --2- 172.21.15.42:0/1207128992 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd9c803b250 0x7fd9c803d760 crc :-1 s=READY pgs=516 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:38:49.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.346+0000 7fd9e90b9700 1 -- 172.21.15.42:0/1207128992 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd9e41166f0 msgr2=0x7fd9e4116b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:38:49.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.346+0000 7fd9e90b9700 1 --2- 172.21.15.42:0/1207128992 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd9e41166f0 0x7fd9e4116b20 secure :-1 s=READY pgs=929 cs=0 l=1 rx=0x7fd9dc008b10 tx=0x7fd9dc0112f0).stop 2019-12-05T05:38:49.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.346+0000 7fd9e90b9700 1 -- 172.21.15.42:0/1207128992 shutdown_connections 2019-12-05T05:38:49.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.346+0000 7fd9e90b9700 1 --2- 172.21.15.42:0/1207128992 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd9c803b250 0x7fd9c803d760 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:38:49.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.346+0000 7fd9e90b9700 1 --2- 172.21.15.42:0/1207128992 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd9e41166f0 0x7fd9e4116b20 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:38:49.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.346+0000 7fd9e90b9700 1 -- 172.21.15.42:0/1207128992 shutdown_connections 2019-12-05T05:38:49.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.346+0000 7fd9e90b9700 1 -- 172.21.15.42:0/1207128992 wait complete. 2019-12-05T05:38:49.358 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.346+0000 7fd9e90b9700 1 -- 172.21.15.42:0/1207128992 >> 172.21.15.42:0/1207128992 conn(0x7fd9e40aabd0 msgr2=0x7fd9e411c460 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:38:49.369 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.a (rank=0) 2019-12-05T05:38:49.369 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2019-12-05T05:38:49.369 INFO:teuthology.orchestra.run:waiting for 300 2019-12-05T05:38:49.386 INFO:tasks.ceph.mds.a:Stopped 2019-12-05T05:38:49.388 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:38:49.562 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.550+0000 7fd9c17c4700 1 Processor -- start 2019-12-05T05:38:49.562 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.550+0000 7fd9c17c4700 1 -- start start 2019-12-05T05:38:49.562 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.550+0000 7fd9c17c4700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd9bc08df80 0x7fd9bc09f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:38:49.563 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.550+0000 7fd9c17c4700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd9bc09f830 0x7fd9bc14efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:38:49.563 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.550+0000 7fd9c17c4700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9bc09fc60 con 0x7fd9bc0a7740 2019-12-05T05:38:49.563 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.550+0000 7fd9c17c4700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9bc0a0b00 con 0x7fd9bc08df80 2019-12-05T05:38:49.563 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.550+0000 7fd9c17c4700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9bc08e5b0 con 0x7fd9bc09f830 2019-12-05T05:38:49.564 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.550+0000 7fd9bb7fe700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fd9bc0a7740 0x7fd9bc07e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:37924/0 (socket says 172.21.15.42:37924) 2019-12-05T05:38:49.564 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.550+0000 7fd9bb7fe700 1 -- 172.21.15.42:0/243578944 learned_addr learned my addr 172.21.15.42:0/243578944 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:38:49.564 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.550+0000 7fd9bbfff700 1 --2- 172.21.15.42:0/243578944 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd9bc09f830 0x7fd9bc14efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:38:49.564 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.550+0000 7fd9baffd700 1 --2- 172.21.15.42:0/243578944 >> v2:172.21.15.40:3300/0 conn(0x7fd9bc08df80 0x7fd9bc09f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:38:49.564 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.550+0000 7fd9ba7fc700 1 -- 172.21.15.42:0/243578944 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3439475557 0 0) 0x7fd9bc09fc60 con 0x7fd9bc0a7740 2019-12-05T05:38:49.564 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.550+0000 7fd9ba7fc700 1 -- 172.21.15.42:0/243578944 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9a40030b0 con 0x7fd9bc0a7740 2019-12-05T05:38:49.565 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.550+0000 7fd9bbfff700 1 -- 172.21.15.42:0/243578944 >> v2:172.21.15.40:3300/0 conn(0x7fd9bc08df80 msgr2=0x7fd9bc09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:38:49.565 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.550+0000 7fd9bbfff700 1 --2- 172.21.15.42:0/243578944 >> v2:172.21.15.40:3300/0 conn(0x7fd9bc08df80 0x7fd9bc09f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:38:49.565 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.550+0000 7fd9bbfff700 1 -- 172.21.15.42:0/243578944 >> v1:172.21.15.42:6789/0 conn(0x7fd9bc0a7740 legacy=0x7fd9bc07e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:38:49.567 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.550+0000 7fd9bbfff700 1 -- 172.21.15.42:0/243578944 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9bc0a0710 con 0x7fd9bc09f830 2019-12-05T05:38:49.567 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.550+0000 7fd9baffd700 1 --2- 172.21.15.42:0/243578944 >> v2:172.21.15.40:3300/0 conn(0x7fd9bc08df80 0x7fd9bc09f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:38:49.567 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.550+0000 7fd9bbfff700 1 --2- 172.21.15.42:0/243578944 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd9bc09f830 0x7fd9bc14efe0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fd9b4008b10 tx=0x7fd9b400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:38:49.567 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.550+0000 7fd9bbfff700 1 --2- 172.21.15.42:0/243578944 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd9bc09f830 0x7fd9bc14efe0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fd9b4008b10 tx=0x7fd9b400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:38:49.568 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.550+0000 7fd9bbfff700 1 --2- 172.21.15.42:0/243578944 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd9bc09f830 0x7fd9bc14efe0 secure :-1 s=READY pgs=249 cs=0 l=1 rx=0x7fd9b4008b10 tx=0x7fd9b400fff0).ready entity=mon.2 client_cookie=1a773242a52f7c1 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:38:49.568 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.550+0000 7fd9bbfff700 1 --2- 172.21.15.42:0/243578944 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd9bc09f830 0x7fd9bc14efe0 secure :-1 s=THROTTLE_DONE pgs=249 cs=0 l=1 rx=0x7fd9b4008b10 tx=0x7fd9b400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:38:49.568 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.550+0000 7fd9ba7fc700 1 -- 172.21.15.42:0/243578944 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd9b4010900 con 0x7fd9bc09f830 2019-12-05T05:38:49.568 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.550+0000 7fd9bbfff700 1 --2- 172.21.15.42:0/243578944 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd9bc09f830 0x7fd9bc14efe0 secure :-1 s=THROTTLE_DONE pgs=249 cs=0 l=1 rx=0x7fd9b4008b10 tx=0x7fd9b400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:38:49.568 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.550+0000 7fd9bbfff700 1 --2- 172.21.15.42:0/243578944 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd9bc09f830 0x7fd9bc14efe0 secure :-1 s=THROTTLE_DONE pgs=249 cs=0 l=1 rx=0x7fd9b4008b10 tx=0x7fd9b400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:38:49.568 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.554+0000 7fd9ba7fc700 1 -- 172.21.15.42:0/243578944 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9b4008b50 con 0x7fd9bc09f830 2019-12-05T05:38:49.569 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.554+0000 7fd9ba7fc700 1 -- 172.21.15.42:0/243578944 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd9b40110c0 con 0x7fd9bc09f830 2019-12-05T05:38:49.569 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.554+0000 7fd9c17c4700 1 -- 172.21.15.42:0/243578944 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd9bc09f830 msgr2=0x7fd9bc14efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:38:49.569 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.554+0000 7fd9c17c4700 1 --2- 172.21.15.42:0/243578944 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd9bc09f830 0x7fd9bc14efe0 secure :-1 s=READY pgs=249 cs=0 l=1 rx=0x7fd9b4008b10 tx=0x7fd9b400fff0).stop 2019-12-05T05:38:49.569 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.554+0000 7fd9c17c4700 1 -- 172.21.15.42:0/243578944 shutdown_connections 2019-12-05T05:38:49.569 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.554+0000 7fd9c17c4700 1 --2- 172.21.15.42:0/243578944 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd9bc09f830 0x7fd9bc14efe0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:38:49.570 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.554+0000 7fd9c17c4700 1 --2- 172.21.15.42:0/243578944 >> v2:172.21.15.40:3300/0 conn(0x7fd9bc08df80 0x7fd9bc09f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:38:49.570 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.554+0000 7fd9c17c4700 1 -- 172.21.15.42:0/243578944 shutdown_connections 2019-12-05T05:38:49.570 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.554+0000 7fd9c17c4700 1 -- 172.21.15.42:0/243578944 wait complete. 2019-12-05T05:38:49.570 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.554+0000 7fd9c17c4700 1 -- 172.21.15.42:0/243578944 >> 172.21.15.42:0/243578944 conn(0x7fd9bc0aabe0 msgr2=0x7fd9bc0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:38:49.574 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.558+0000 7fd9c17c4700 1 Processor -- start 2019-12-05T05:38:49.574 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.558+0000 7fd9c17c4700 1 -- start start 2019-12-05T05:38:49.575 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.558+0000 7fd9c17c4700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd9bc123990 0x7fd9bc133e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:38:49.575 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.558+0000 7fd9c17c4700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd9bc134380 0x7fd9bc08eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:38:49.575 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.558+0000 7fd9c17c4700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd9bc08f0c0 0x7fd9bc08f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:38:49.575 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.558+0000 7fd9c17c4700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9bc1337b0 con 0x7fd9bc08f0c0 2019-12-05T05:38:49.576 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.558+0000 7fd9c17c4700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9bc11ae90 con 0x7fd9bc134380 2019-12-05T05:38:49.577 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.558+0000 7fd9c17c4700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9bc11c1c0 con 0x7fd9bc123990 2019-12-05T05:38:49.578 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.558+0000 7fd9baffd700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd9bc134380 0x7fd9bc08eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:38:49.578 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.558+0000 7fd9baffd700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd9bc134380 0x7fd9bc08eb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:37894/0 (socket says 172.21.15.42:37894) 2019-12-05T05:38:49.578 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.558+0000 7fd9bbfff700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd9bc08f0c0 0x7fd9bc08f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:38:49.578 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.558+0000 7fd9bb7fe700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd9bc123990 0x7fd9bc133e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:38:49.578 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.558+0000 7fd9baffd700 1 -- 172.21.15.42:0/1629899497 learned_addr learned my addr 172.21.15.42:0/1629899497 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:38:49.578 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.558+0000 7fd9baffd700 1 -- 172.21.15.42:0/1629899497 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd9bc123990 msgr2=0x7fd9bc133e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:38:49.578 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.558+0000 7fd9baffd700 1 --2- 172.21.15.42:0/1629899497 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd9bc123990 0x7fd9bc133e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:38:49.579 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.558+0000 7fd9baffd700 1 -- 172.21.15.42:0/1629899497 >> v2:172.21.15.40:3300/0 conn(0x7fd9bc08f0c0 msgr2=0x7fd9bc08f4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:38:49.579 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.558+0000 7fd9baffd700 1 --2- 172.21.15.42:0/1629899497 >> v2:172.21.15.40:3300/0 conn(0x7fd9bc08f0c0 0x7fd9bc08f4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:38:49.579 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.558+0000 7fd9baffd700 1 -- 172.21.15.42:0/1629899497 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9bc11a520 con 0x7fd9bc134380 2019-12-05T05:38:49.579 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.558+0000 7fd9bb7fe700 1 --2- 172.21.15.42:0/1629899497 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd9bc123990 0x7fd9bc133e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:38:49.579 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.558+0000 7fd9bbfff700 1 --2- 172.21.15.42:0/1629899497 >> v2:172.21.15.40:3300/0 conn(0x7fd9bc08f0c0 0x7fd9bc08f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:38:49.579 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.558+0000 7fd9baffd700 1 --2- 172.21.15.42:0/1629899497 >> v2:172.21.15.42:3300/0 conn(0x7fd9bc134380 0x7fd9bc08eb70 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fd9bc11c380 tx=0x7fd9b0010e30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:38:49.579 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.558+0000 7fd9baffd700 1 --2- 172.21.15.42:0/1629899497 >> v2:172.21.15.42:3300/0 conn(0x7fd9bc134380 0x7fd9bc08eb70 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fd9bc11c380 tx=0x7fd9b0010e30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:38:49.580 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.558+0000 7fd9baffd700 1 --2- 172.21.15.42:0/1629899497 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd9bc134380 0x7fd9bc08eb70 secure :-1 s=READY pgs=1192 cs=0 l=1 rx=0x7fd9bc11c380 tx=0x7fd9b0010e30).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:38:49.580 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.558+0000 7fd9baffd700 1 --2- 172.21.15.42:0/1629899497 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd9bc134380 0x7fd9bc08eb70 secure :-1 s=THROTTLE_DONE pgs=1192 cs=0 l=1 rx=0x7fd9bc11c380 tx=0x7fd9b0010e30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:38:49.580 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.558+0000 7fd9a3fff700 1 -- 172.21.15.42:0/1629899497 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd9b0005840 con 0x7fd9bc134380 2019-12-05T05:38:49.580 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.558+0000 7fd9baffd700 1 --2- 172.21.15.42:0/1629899497 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd9bc134380 0x7fd9bc08eb70 secure :-1 s=THROTTLE_DONE pgs=1192 cs=0 l=1 rx=0x7fd9bc11c380 tx=0x7fd9b0010e30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:38:49.580 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.558+0000 7fd9baffd700 1 --2- 172.21.15.42:0/1629899497 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd9bc134380 0x7fd9bc08eb70 secure :-1 s=THROTTLE_DONE pgs=1192 cs=0 l=1 rx=0x7fd9bc11c380 tx=0x7fd9b0010e30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:38:49.580 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.558+0000 7fd9a3fff700 1 -- 172.21.15.42:0/1629899497 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9b0007160 con 0x7fd9bc134380 2019-12-05T05:38:49.580 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.558+0000 7fd9c17c4700 1 -- 172.21.15.42:0/1629899497 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9bc121c70 con 0x7fd9bc134380 2019-12-05T05:38:49.581 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.558+0000 7fd9a3fff700 1 -- 172.21.15.42:0/1629899497 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd9b0012470 con 0x7fd9bc134380 2019-12-05T05:38:49.581 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.558+0000 7fd9c17c4700 1 -- 172.21.15.42:0/1629899497 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9bc1347b0 con 0x7fd9bc134380 2019-12-05T05:38:49.581 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.562+0000 7fd9baffd700 1 --2- 172.21.15.42:0/1629899497 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd9bc134380 0x7fd9bc08eb70 secure :-1 s=THROTTLE_DONE pgs=1192 cs=0 l=1 rx=0x7fd9bc11c380 tx=0x7fd9b0010e30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:38:49.581 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.562+0000 7fd9a3fff700 1 -- 172.21.15.42:0/1629899497 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fd9b0012780 con 0x7fd9bc134380 2019-12-05T05:38:49.581 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.562+0000 7fd9baffd700 1 --2- 172.21.15.42:0/1629899497 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd9bc134380 0x7fd9bc08eb70 secure :-1 s=THROTTLE_DONE pgs=1192 cs=0 l=1 rx=0x7fd9bc11c380 tx=0x7fd9b0010e30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:38:49.581 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.562+0000 7fd9a3fff700 1 --2- 172.21.15.42:0/1629899497 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd99c03add0 0x7fd99c03d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:38:49.581 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.562+0000 7fd9a3fff700 1 -- 172.21.15.42:0/1629899497 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 8009+0+0 (secure 0 0 0) 0x7fd9b004b620 con 0x7fd9bc134380 2019-12-05T05:38:49.582 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.562+0000 7fd9bb7fe700 1 --2- 172.21.15.42:0/1629899497 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd99c03add0 0x7fd99c03d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:38:49.582 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.562+0000 7fd9bb7fe700 1 --2- 172.21.15.42:0/1629899497 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd99c03add0 0x7fd99c03d2e0 crc :-1 s=READY pgs=517 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:38:49.582 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.566+0000 7fd9c17c4700 1 -- 172.21.15.42:0/1629899497 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9bc05d110 con 0x7fd9bc134380 2019-12-05T05:38:49.582 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.570+0000 7fd9baffd700 1 --2- 172.21.15.42:0/1629899497 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd9bc134380 0x7fd9bc08eb70 secure :-1 s=THROTTLE_DONE pgs=1192 cs=0 l=1 rx=0x7fd9bc11c380 tx=0x7fd9b0010e30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:38:49.583 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.570+0000 7fd9a3fff700 1 -- 172.21.15.42:0/1629899497 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fd9b00326a0 con 0x7fd9bc134380 2019-12-05T05:38:49.741 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.734+0000 7fd9c17c4700 1 -- 172.21.15.42:0/1629899497 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd9bc05b8c0 con 0x7fd9bc134380 2019-12-05T05:38:49.742 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.734+0000 7fd9baffd700 1 --2- 172.21.15.42:0/1629899497 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd9bc134380 0x7fd9bc08eb70 secure :-1 s=THROTTLE_DONE pgs=1192 cs=0 l=1 rx=0x7fd9bc11c380 tx=0x7fd9b0010e30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:38:49.742 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.734+0000 7fd9a3fff700 1 -- 172.21.15.42:0/1629899497 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 148 v148) v1 ==== 95+0+2109 (secure 0 0 0) 0x7fd9b0032900 con 0x7fd9bc134380 2019-12-05T05:38:49.742 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 148 2019-12-05T05:38:49.743 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:38:49.743 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":148,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7416,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/123054133","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":123054133},{"type":"v1","addr":"172.21.15.40:6833","nonce":123054133}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":148}],"filesystems":[{"mdsmap":{"epoch":147,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:37:13.880687+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5693},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5693":{"gid":5693,"name":"a","rank":0,"incarnation":144,"state":"up:active","state_seq":39,"addr":"172.21.15.42:6835/3838877789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3838877789},{"type":"v1","addr":"172.21.15.42:6835","nonce":3838877789}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:38:49.745 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.738+0000 7fd9c17c4700 1 -- 172.21.15.42:0/1629899497 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd99c03add0 msgr2=0x7fd99c03d2e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:38:49.745 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.738+0000 7fd9c17c4700 1 --2- 172.21.15.42:0/1629899497 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd99c03add0 0x7fd99c03d2e0 crc :-1 s=READY pgs=517 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:38:49.745 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.738+0000 7fd9c17c4700 1 -- 172.21.15.42:0/1629899497 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd9bc134380 msgr2=0x7fd9bc08eb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:38:49.745 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.738+0000 7fd9c17c4700 1 --2- 172.21.15.42:0/1629899497 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd9bc134380 0x7fd9bc08eb70 secure :-1 s=READY pgs=1192 cs=0 l=1 rx=0x7fd9bc11c380 tx=0x7fd9b0010e30).stop 2019-12-05T05:38:49.747 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.738+0000 7fd9c17c4700 1 -- 172.21.15.42:0/1629899497 shutdown_connections 2019-12-05T05:38:49.747 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.738+0000 7fd9c17c4700 1 --2- 172.21.15.42:0/1629899497 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd99c03add0 0x7fd99c03d2e0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:38:49.747 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.738+0000 7fd9c17c4700 1 --2- 172.21.15.42:0/1629899497 >> v2:172.21.15.40:3300/0 conn(0x7fd9bc08f0c0 0x7fd9bc08f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:38:49.747 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.738+0000 7fd9c17c4700 1 --2- 172.21.15.42:0/1629899497 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd9bc123990 0x7fd9bc133e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:38:49.747 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.738+0000 7fd9c17c4700 1 --2- 172.21.15.42:0/1629899497 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd9bc134380 0x7fd9bc08eb70 unknown :-1 s=CLOSED pgs=1192 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:38:49.748 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.738+0000 7fd9c17c4700 1 -- 172.21.15.42:0/1629899497 shutdown_connections 2019-12-05T05:38:49.748 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.738+0000 7fd9c17c4700 1 -- 172.21.15.42:0/1629899497 wait complete. 2019-12-05T05:38:49.748 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:49.738+0000 7fd9c17c4700 1 -- 172.21.15.42:0/1629899497 >> 172.21.15.42:0/1629899497 conn(0x7fd9bc08fde0 msgr2=0x7fd9bc059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:38:49.755 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T05:38:51.756 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:38:51.868 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:51.854+0000 7fbacd509700 1 Processor -- start 2019-12-05T05:38:51.868 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:51.854+0000 7fbacd509700 1 -- start start 2019-12-05T05:38:51.868 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:51.854+0000 7fbacd509700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fbac809fae0 0x7fbac814fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:38:51.869 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:51.854+0000 7fbacd509700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbac80a0710 con 0x7fbac80a7740 2019-12-05T05:38:51.869 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:51.854+0000 7fbacd509700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbac807ef20 con 0x7fbac808df80 2019-12-05T05:38:51.869 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:51.854+0000 7fbacd509700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbac80a0b00 con 0x7fbac809fae0 2019-12-05T05:38:51.869 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:51.854+0000 7fbac7fff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fbac809fae0 0x7fbac814fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:38:51.870 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:51.854+0000 7fbac7fff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fbac809fae0 0x7fbac814fff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:37914/0 (socket says 172.21.15.42:37914) 2019-12-05T05:38:51.870 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:51.854+0000 7fbac7fff700 1 -- 172.21.15.42:0/852521045 learned_addr learned my addr 172.21.15.42:0/852521045 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:38:51.870 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:51.854+0000 7fbac67fc700 1 -- 172.21.15.42:0/852521045 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4056497368 0 0) 0x7fbac80a0710 con 0x7fbac80a7740 2019-12-05T05:38:51.870 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:51.854+0000 7fbac7fff700 1 -- 172.21.15.42:0/852521045 >> v1:172.21.15.42:6789/0 conn(0x7fbac808df80 legacy=0x7fbac809f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:38:51.870 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:51.854+0000 7fbac7fff700 1 -- 172.21.15.42:0/852521045 >> v1:172.21.15.40:6789/0 conn(0x7fbac80a7740 legacy=0x7fbac807e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:38:51.870 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:51.854+0000 7fbac7fff700 1 -- 172.21.15.42:0/852521045 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbac80a9ba0 con 0x7fbac809fae0 2019-12-05T05:38:51.870 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:51.854+0000 7fbac7fff700 1 --2- 172.21.15.42:0/852521045 >> v2:172.21.15.42:3300/0 conn(0x7fbac809fae0 0x7fbac814fff0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fbac0008790 tx=0x7fbac000fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:38:51.871 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:51.854+0000 7fbac7fff700 1 --2- 172.21.15.42:0/852521045 >> v2:172.21.15.42:3300/0 conn(0x7fbac809fae0 0x7fbac814fff0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fbac0008790 tx=0x7fbac000fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:38:51.871 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:51.854+0000 7fbac7fff700 1 --2- 172.21.15.42:0/852521045 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbac809fae0 0x7fbac814fff0 secure :-1 s=READY pgs=1196 cs=0 l=1 rx=0x7fbac0008790 tx=0x7fbac000fcb0).ready entity=mon.1 client_cookie=9ba61fc9004b764d server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:38:51.871 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:51.854+0000 7fbac7fff700 1 -- 172.21.15.42:0/852521045 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbac809fae0 msgr2=0x7fbac814fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2019-12-05T05:38:51.871 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:51.854+0000 7fbac7fff700 1 -- 172.21.15.42:0/852521045 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbac809fae0 msgr2=0x7fbac814fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2019-12-05T05:38:51.871 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:51.854+0000 7fbac7fff700 1 --2- 172.21.15.42:0/852521045 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbac809fae0 0x7fbac814fff0 secure :-1 s=READY pgs=1196 cs=0 l=1 rx=0x7fbac0008790 tx=0x7fbac000fcb0).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-12-05T05:38:51.871 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:51.854+0000 7fbac7fff700 1 --2- 172.21.15.42:0/852521045 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbac809fae0 0x7fbac814fff0 secure :-1 s=READY pgs=1196 cs=0 l=1 rx=0x7fbac0008790 tx=0x7fbac000fcb0).stop 2019-12-05T05:38:51.871 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:51.854+0000 7fbac67fc700 1 -- 172.21.15.42:0/852521045 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbac809fae0 msgr2=0x7fbac814fff0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2019-12-05T05:38:51.872 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:51.854+0000 7fbac67fc700 1 --2- 172.21.15.42:0/852521045 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbac809fae0 0x7fbac814fff0 unknown :-1 s=CLOSED pgs=1196 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:38:51.872 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:51.858+0000 7fbac67fc700 1 --2- 172.21.15.42:0/852521045 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbab4003050 0x7fbab40054f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:38:51.872 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:51.858+0000 7fbac67fc700 1 --2- 172.21.15.42:0/852521045 >> v2:172.21.15.40:3300/0 conn(0x7fbab4005b80 0x7fbab4008090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:38:51.872 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:51.858+0000 7fbac67fc700 1 -- 172.21.15.42:0/852521045 --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbab40087b0 con 0x7fbab4005b80 2019-12-05T05:38:51.872 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:51.858+0000 7fbac67fc700 1 -- 172.21.15.42:0/852521045 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbab40089a0 con 0x7fbab4003050 2019-12-05T05:38:51.872 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:51.858+0000 7fbac77fe700 1 --2- 172.21.15.42:0/852521045 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbab4003050 0x7fbab40054f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:38:51.872 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:51.858+0000 7fbac6ffd700 1 --2- 172.21.15.42:0/852521045 >> v2:172.21.15.40:3300/0 conn(0x7fbab4005b80 0x7fbab4008090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:38:51.873 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:51.858+0000 7fbac77fe700 1 -- 172.21.15.42:0/852521045 >> v2:172.21.15.40:3300/0 conn(0x7fbab4005b80 msgr2=0x7fbab4008090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:38:51.873 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:51.858+0000 7fbac77fe700 1 --2- 172.21.15.42:0/852521045 >> v2:172.21.15.40:3300/0 conn(0x7fbab4005b80 0x7fbab4008090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:38:51.873 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:51.858+0000 7fbac77fe700 1 -- 172.21.15.42:0/852521045 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbab404cc60 con 0x7fbab4003050 2019-12-05T05:38:51.873 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:51.858+0000 7fbac6ffd700 1 --2- 172.21.15.42:0/852521045 >> v2:172.21.15.40:3300/0 conn(0x7fbab4005b80 0x7fbab4008090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:38:51.873 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:51.858+0000 7fbac77fe700 1 --2- 172.21.15.42:0/852521045 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbab4003050 0x7fbab40054f0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fbab8001000 tx=0x7fbab800be90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:38:51.873 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:51.858+0000 7fbac77fe700 1 --2- 172.21.15.42:0/852521045 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbab4003050 0x7fbab40054f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fbab8001000 tx=0x7fbab800be90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:38:51.873 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:51.858+0000 7fbac77fe700 1 --2- 172.21.15.42:0/852521045 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbab4003050 0x7fbab40054f0 secure :-1 s=READY pgs=253 cs=0 l=1 rx=0x7fbab8001000 tx=0x7fbab800be90).ready entity=mon.2 client_cookie=a9c1ed3ace763dba server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:38:51.874 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:51.858+0000 7fbac77fe700 1 --2- 172.21.15.42:0/852521045 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbab4003050 0x7fbab40054f0 secure :-1 s=THROTTLE_DONE pgs=253 cs=0 l=1 rx=0x7fbab8001000 tx=0x7fbab800be90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:38:51.874 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:51.858+0000 7fbac67fc700 1 -- 172.21.15.42:0/852521045 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbab8003540 con 0x7fbab4003050 2019-12-05T05:38:51.874 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:51.858+0000 7fbac77fe700 1 --2- 172.21.15.42:0/852521045 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbab4003050 0x7fbab40054f0 secure :-1 s=THROTTLE_DONE pgs=253 cs=0 l=1 rx=0x7fbab8001000 tx=0x7fbab800be90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:38:51.874 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:51.858+0000 7fbac77fe700 1 --2- 172.21.15.42:0/852521045 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbab4003050 0x7fbab40054f0 secure :-1 s=THROTTLE_DONE pgs=253 cs=0 l=1 rx=0x7fbab8001000 tx=0x7fbab800be90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:38:51.874 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:51.858+0000 7fbac67fc700 1 -- 172.21.15.42:0/852521045 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbab8003160 con 0x7fbab4003050 2019-12-05T05:38:51.874 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:51.858+0000 7fbac67fc700 1 -- 172.21.15.42:0/852521045 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbab8003b60 con 0x7fbab4003050 2019-12-05T05:38:51.874 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:51.858+0000 7fbacd509700 1 -- 172.21.15.42:0/852521045 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbab4003050 msgr2=0x7fbab40054f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:38:51.875 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:51.858+0000 7fbacd509700 1 --2- 172.21.15.42:0/852521045 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbab4003050 0x7fbab40054f0 secure :-1 s=READY pgs=253 cs=0 l=1 rx=0x7fbab8001000 tx=0x7fbab800be90).stop 2019-12-05T05:38:51.875 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:51.858+0000 7fbac7fff700 1 -- 172.21.15.42:0/852521045 reap_dead start 2019-12-05T05:38:51.875 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:51.858+0000 7fbacd509700 1 -- 172.21.15.42:0/852521045 shutdown_connections 2019-12-05T05:38:51.875 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:51.858+0000 7fbacd509700 1 --2- 172.21.15.42:0/852521045 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbac809fae0 0x7fbac814fff0 unknown :-1 s=CLOSED pgs=1196 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:38:51.875 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:51.858+0000 7fbacd509700 1 -- 172.21.15.42:0/852521045 shutdown_connections 2019-12-05T05:38:51.875 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:51.858+0000 7fbacd509700 1 -- 172.21.15.42:0/852521045 wait complete. 2019-12-05T05:38:51.876 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:51.858+0000 7fbacd509700 1 -- 172.21.15.42:0/852521045 >> 172.21.15.42:0/852521045 conn(0x7fbac80aabe0 msgr2=0x7fbac80a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:38:51.880 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:51.862+0000 7fbacd509700 1 Processor -- start 2019-12-05T05:38:51.880 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:51.862+0000 7fbacd509700 1 -- start start 2019-12-05T05:38:51.880 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:51.862+0000 7fbacd509700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fbac811a890 0x7fbac811c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:38:51.880 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:51.862+0000 7fbacd509700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fbac807c500 0x7fbac811bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:38:51.880 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:51.862+0000 7fbacd509700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbac807c930 con 0x7fbac81417c0 2019-12-05T05:38:51.881 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:51.862+0000 7fbacd509700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbac80a4840 con 0x7fbac811a890 2019-12-05T05:38:51.881 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:51.862+0000 7fbacd509700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbac807d1f0 con 0x7fbac807c500 2019-12-05T05:38:51.881 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:51.862+0000 7fbac6ffd700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fbac807c500 0x7fbac811bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:38:51.881 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:51.862+0000 7fbac6ffd700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fbac807c500 0x7fbac811bbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:37924/0 (socket says 172.21.15.42:37924) 2019-12-05T05:38:51.881 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:51.862+0000 7fbac6ffd700 1 -- 172.21.15.42:0/2281491434 learned_addr learned my addr 172.21.15.42:0/2281491434 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:38:51.881 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:51.862+0000 7fbac77fe700 1 --2- 172.21.15.42:0/2281491434 >> v2:172.21.15.40:3300/0 conn(0x7fbac811a890 0x7fbac811c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:38:51.881 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:51.862+0000 7fbac6ffd700 1 -- 172.21.15.42:0/2281491434 >> v2:172.21.15.40:3300/0 conn(0x7fbac811a890 msgr2=0x7fbac811c7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:38:51.882 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:51.862+0000 7fbac6ffd700 1 --2- 172.21.15.42:0/2281491434 >> v2:172.21.15.40:3300/0 conn(0x7fbac811a890 0x7fbac811c7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:38:51.882 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:51.862+0000 7fbac6ffd700 1 -- 172.21.15.42:0/2281491434 >> v1:172.21.15.40:6789/0 conn(0x7fbac81417c0 legacy=0x7fbac80a4b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:38:51.882 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:51.862+0000 7fbabffff700 1 -- 172.21.15.42:0/2281491434 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3783817547 0 0) 0x7fbac807c930 con 0x7fbac81417c0 2019-12-05T05:38:51.882 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:51.862+0000 7fbac6ffd700 1 -- 172.21.15.42:0/2281491434 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbac811b900 con 0x7fbac807c500 2019-12-05T05:38:51.882 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:51.862+0000 7fbac6ffd700 1 --2- 172.21.15.42:0/2281491434 >> v2:172.21.15.42:3300/0 conn(0x7fbac807c500 0x7fbac811bbc0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fbab0001730 tx=0x7fbab0011050).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:38:51.882 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:51.862+0000 7fbac6ffd700 1 --2- 172.21.15.42:0/2281491434 >> v2:172.21.15.42:3300/0 conn(0x7fbac807c500 0x7fbac811bbc0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fbab0001730 tx=0x7fbab0011050).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:38:51.882 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:51.862+0000 7fbac6ffd700 1 --2- 172.21.15.42:0/2281491434 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbac807c500 0x7fbac811bbc0 secure :-1 s=READY pgs=1197 cs=0 l=1 rx=0x7fbab0001730 tx=0x7fbab0011050).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:38:51.883 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:51.866+0000 7fbac6ffd700 1 --2- 172.21.15.42:0/2281491434 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbac807c500 0x7fbac811bbc0 secure :-1 s=THROTTLE_DONE pgs=1197 cs=0 l=1 rx=0x7fbab0001730 tx=0x7fbab0011050).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:38:51.883 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:51.866+0000 7fbabffff700 1 -- 172.21.15.42:0/2281491434 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbab00101c0 con 0x7fbac807c500 2019-12-05T05:38:51.883 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:51.866+0000 7fbac6ffd700 1 --2- 172.21.15.42:0/2281491434 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbac807c500 0x7fbac811bbc0 secure :-1 s=THROTTLE_DONE pgs=1197 cs=0 l=1 rx=0x7fbab0001730 tx=0x7fbab0011050).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:38:51.883 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:51.866+0000 7fbabffff700 1 -- 172.21.15.42:0/2281491434 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbab000ca40 con 0x7fbac807c500 2019-12-05T05:38:51.883 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:51.866+0000 7fbac6ffd700 1 --2- 172.21.15.42:0/2281491434 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbac807c500 0x7fbac811bbc0 secure :-1 s=THROTTLE_DONE pgs=1197 cs=0 l=1 rx=0x7fbab0001730 tx=0x7fbab0011050).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:38:51.883 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:51.866+0000 7fbacd509700 1 -- 172.21.15.42:0/2281491434 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbac811c110 con 0x7fbac807c500 2019-12-05T05:38:51.883 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:51.866+0000 7fbabffff700 1 -- 172.21.15.42:0/2281491434 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbab0012690 con 0x7fbac807c500 2019-12-05T05:38:51.884 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:51.866+0000 7fbacd509700 1 -- 172.21.15.42:0/2281491434 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbac807d680 con 0x7fbac807c500 2019-12-05T05:38:51.884 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:51.866+0000 7fbac6ffd700 1 --2- 172.21.15.42:0/2281491434 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbac807c500 0x7fbac811bbc0 secure :-1 s=THROTTLE_DONE pgs=1197 cs=0 l=1 rx=0x7fbab0001730 tx=0x7fbab0011050).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:38:51.884 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:51.866+0000 7fbabffff700 1 -- 172.21.15.42:0/2281491434 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fbab0012970 con 0x7fbac807c500 2019-12-05T05:38:51.884 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:51.866+0000 7fbac6ffd700 1 --2- 172.21.15.42:0/2281491434 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbac807c500 0x7fbac811bbc0 secure :-1 s=THROTTLE_DONE pgs=1197 cs=0 l=1 rx=0x7fbab0001730 tx=0x7fbab0011050).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:38:51.884 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:51.866+0000 7fbabffff700 1 --2- 172.21.15.42:0/2281491434 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbaac03ac10 0x7fbaac03d120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:38:51.884 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:51.866+0000 7fbabffff700 1 -- 172.21.15.42:0/2281491434 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 8009+0+0 (secure 0 0 0) 0x7fbab004b440 con 0x7fbac807c500 2019-12-05T05:38:51.884 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:51.866+0000 7fbac77fe700 1 --2- 172.21.15.42:0/2281491434 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbaac03ac10 0x7fbaac03d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:38:51.885 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:51.866+0000 7fbac77fe700 1 --2- 172.21.15.42:0/2281491434 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbaac03ac10 0x7fbaac03d120 crc :-1 s=READY pgs=518 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:38:51.885 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:51.870+0000 7fbacd509700 1 -- 172.21.15.42:0/2281491434 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbac805d110 con 0x7fbac807c500 2019-12-05T05:38:51.885 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:51.878+0000 7fbac6ffd700 1 --2- 172.21.15.42:0/2281491434 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbac807c500 0x7fbac811bbc0 secure :-1 s=THROTTLE_DONE pgs=1197 cs=0 l=1 rx=0x7fbab0001730 tx=0x7fbab0011050).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:38:51.885 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:51.878+0000 7fbabffff700 1 -- 172.21.15.42:0/2281491434 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fbac811b900 con 0x7fbac807c500 2019-12-05T05:38:52.048 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:52.042+0000 7fbacd509700 1 -- 172.21.15.42:0/2281491434 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbac805b8c0 con 0x7fbac807c500 2019-12-05T05:38:52.049 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:52.042+0000 7fbac6ffd700 1 --2- 172.21.15.42:0/2281491434 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbac807c500 0x7fbac811bbc0 secure :-1 s=THROTTLE_DONE pgs=1197 cs=0 l=1 rx=0x7fbab0001730 tx=0x7fbab0011050).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:38:52.049 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:52.042+0000 7fbabffff700 1 -- 172.21.15.42:0/2281491434 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 148 v148) v1 ==== 95+0+2109 (secure 0 0 0) 0x7fbab00324a0 con 0x7fbac807c500 2019-12-05T05:38:52.049 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 148 2019-12-05T05:38:52.050 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:38:52.050 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":148,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7416,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/123054133","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":123054133},{"type":"v1","addr":"172.21.15.40:6833","nonce":123054133}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":148}],"filesystems":[{"mdsmap":{"epoch":147,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:37:13.880687+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5693},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5693":{"gid":5693,"name":"a","rank":0,"incarnation":144,"state":"up:active","state_seq":39,"addr":"172.21.15.42:6835/3838877789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3838877789},{"type":"v1","addr":"172.21.15.42:6835","nonce":3838877789}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:38:52.051 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:52.042+0000 7fbacd509700 1 -- 172.21.15.42:0/2281491434 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbaac03ac10 msgr2=0x7fbaac03d120 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:38:52.051 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:52.042+0000 7fbacd509700 1 --2- 172.21.15.42:0/2281491434 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbaac03ac10 0x7fbaac03d120 crc :-1 s=READY pgs=518 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:38:52.051 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:52.042+0000 7fbacd509700 1 -- 172.21.15.42:0/2281491434 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbac807c500 msgr2=0x7fbac811bbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:38:52.051 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:52.042+0000 7fbacd509700 1 --2- 172.21.15.42:0/2281491434 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbac807c500 0x7fbac811bbc0 secure :-1 s=READY pgs=1197 cs=0 l=1 rx=0x7fbab0001730 tx=0x7fbab0011050).stop 2019-12-05T05:38:52.051 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:52.042+0000 7fbacd509700 1 -- 172.21.15.42:0/2281491434 shutdown_connections 2019-12-05T05:38:52.052 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:52.042+0000 7fbacd509700 1 --2- 172.21.15.42:0/2281491434 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbaac03ac10 0x7fbaac03d120 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:38:52.052 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:52.042+0000 7fbacd509700 1 --2- 172.21.15.42:0/2281491434 >> v2:172.21.15.40:3300/0 conn(0x7fbac811a890 0x7fbac811c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:38:52.052 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:52.042+0000 7fbacd509700 1 --2- 172.21.15.42:0/2281491434 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbac807c500 0x7fbac811bbc0 unknown :-1 s=CLOSED pgs=1197 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:38:52.052 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:52.042+0000 7fbacd509700 1 -- 172.21.15.42:0/2281491434 shutdown_connections 2019-12-05T05:38:52.052 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:52.042+0000 7fbacd509700 1 -- 172.21.15.42:0/2281491434 wait complete. 2019-12-05T05:38:52.052 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:52.042+0000 7fbacd509700 1 -- 172.21.15.42:0/2281491434 >> 172.21.15.42:0/2281491434 conn(0x7fbac807e5b0 msgr2=0x7fbac808eb70 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:38:52.061 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T05:38:54.062 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:38:54.170 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:54.158+0000 7f29e3c00700 1 Processor -- start 2019-12-05T05:38:54.170 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:54.158+0000 7f29e3c00700 1 -- start start 2019-12-05T05:38:54.170 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:54.158+0000 7f29e3c00700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f29dc0a7740 0x7f29dc08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:38:54.170 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:54.158+0000 7f29e3c00700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f29dc07e720 0x7f29dc07eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:38:54.171 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:54.158+0000 7f29e3c00700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29dc0a0710 con 0x7f29dc09f2e0 2019-12-05T05:38:54.171 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:54.158+0000 7f29e3c00700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29dc0a0b00 con 0x7f29dc0a7740 2019-12-05T05:38:54.171 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:54.158+0000 7f29e3c00700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29dc08e5b0 con 0x7f29dc07e720 2019-12-05T05:38:54.171 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:54.158+0000 7f29e219d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f29dc0a7740 0x7f29dc08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:38:54.171 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:54.158+0000 7f29e299e700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f29dc09f2e0 0x7f29dc09f710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:37972/0 (socket says 172.21.15.42:37972) 2019-12-05T05:38:54.171 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:54.158+0000 7f29e299e700 1 -- 172.21.15.42:0/2728830982 learned_addr learned my addr 172.21.15.42:0/2728830982 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:38:54.172 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:54.158+0000 7f29e219d700 1 --2- 172.21.15.42:0/2728830982 >> v2:172.21.15.42:3300/0 conn(0x7f29dc0a7740 0x7f29dc08df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:37930/0 (socket says 172.21.15.42:37930) 2019-12-05T05:38:54.172 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:54.158+0000 7f29e199c700 1 --2- 172.21.15.42:0/2728830982 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f29dc07e720 0x7f29dc07eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:38:54.172 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:54.158+0000 7f29e219d700 1 -- 172.21.15.42:0/2728830982 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f29dc07e720 msgr2=0x7f29dc07eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:38:54.172 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:54.158+0000 7f29e219d700 1 --2- 172.21.15.42:0/2728830982 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f29dc07e720 0x7f29dc07eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:38:54.173 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:54.158+0000 7f29e219d700 1 -- 172.21.15.42:0/2728830982 >> v1:172.21.15.42:6789/0 conn(0x7f29dc09f2e0 legacy=0x7f29dc09f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:38:54.173 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:54.158+0000 7f29e119b700 1 -- 172.21.15.42:0/2728830982 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3329790833 0 0) 0x7f29dc0a0710 con 0x7f29dc09f2e0 2019-12-05T05:38:54.173 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:54.158+0000 7f29e219d700 1 -- 172.21.15.42:0/2728830982 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29dc07f0a0 con 0x7f29dc0a7740 2019-12-05T05:38:54.173 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:54.158+0000 7f29e219d700 1 --2- 172.21.15.42:0/2728830982 >> v2:172.21.15.42:3300/0 conn(0x7f29dc0a7740 0x7f29dc08df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f29d8008790 tx=0x7f29d800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:38:54.174 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:54.158+0000 7f29e219d700 1 --2- 172.21.15.42:0/2728830982 >> v2:172.21.15.42:3300/0 conn(0x7f29dc0a7740 0x7f29dc08df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f29d8008790 tx=0x7f29d800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:38:54.174 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:54.158+0000 7f29e219d700 1 --2- 172.21.15.42:0/2728830982 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f29dc0a7740 0x7f29dc08df80 secure :-1 s=READY pgs=1199 cs=0 l=1 rx=0x7f29d8008790 tx=0x7f29d800fc70).ready entity=mon.1 client_cookie=446e321e2b812376 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:38:54.174 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:54.158+0000 7f29e219d700 1 --2- 172.21.15.42:0/2728830982 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f29dc0a7740 0x7f29dc08df80 secure :-1 s=THROTTLE_DONE pgs=1199 cs=0 l=1 rx=0x7f29d8008790 tx=0x7f29d800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:38:54.174 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:54.158+0000 7f29e119b700 1 -- 172.21.15.42:0/2728830982 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f29d80112f0 con 0x7f29dc0a7740 2019-12-05T05:38:54.174 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:54.158+0000 7f29e219d700 1 --2- 172.21.15.42:0/2728830982 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f29dc0a7740 0x7f29dc08df80 secure :-1 s=THROTTLE_DONE pgs=1199 cs=0 l=1 rx=0x7f29d8008790 tx=0x7f29d800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:38:54.175 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:54.158+0000 7f29e219d700 1 --2- 172.21.15.42:0/2728830982 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f29dc0a7740 0x7f29dc08df80 secure :-1 s=THROTTLE_DONE pgs=1199 cs=0 l=1 rx=0x7f29d8008790 tx=0x7f29d800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:38:54.175 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:54.158+0000 7f29e119b700 1 -- 172.21.15.42:0/2728830982 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29d8008ad0 con 0x7f29dc0a7740 2019-12-05T05:38:54.175 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:54.158+0000 7f29e119b700 1 -- 172.21.15.42:0/2728830982 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f29d80118d0 con 0x7f29dc0a7740 2019-12-05T05:38:54.175 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:54.158+0000 7f29e3c00700 1 -- 172.21.15.42:0/2728830982 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f29dc0a7740 msgr2=0x7f29dc08df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:38:54.175 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:54.158+0000 7f29e3c00700 1 --2- 172.21.15.42:0/2728830982 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f29dc0a7740 0x7f29dc08df80 secure :-1 s=READY pgs=1199 cs=0 l=1 rx=0x7f29d8008790 tx=0x7f29d800fc70).stop 2019-12-05T05:38:54.175 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:54.162+0000 7f29e3c00700 1 -- 172.21.15.42:0/2728830982 shutdown_connections 2019-12-05T05:38:54.175 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:54.162+0000 7f29e3c00700 1 --2- 172.21.15.42:0/2728830982 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f29dc0a7740 0x7f29dc08df80 unknown :-1 s=CLOSED pgs=1199 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:38:54.176 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:54.162+0000 7f29e3c00700 1 --2- 172.21.15.42:0/2728830982 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f29dc07e720 0x7f29dc07eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:38:54.176 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:54.162+0000 7f29e3c00700 1 -- 172.21.15.42:0/2728830982 shutdown_connections 2019-12-05T05:38:54.176 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:54.162+0000 7f29e3c00700 1 -- 172.21.15.42:0/2728830982 wait complete. 2019-12-05T05:38:54.176 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:54.162+0000 7f29e3c00700 1 -- 172.21.15.42:0/2728830982 >> 172.21.15.42:0/2728830982 conn(0x7f29dc0aabe0 msgr2=0x7f29dc0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:38:54.180 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:54.166+0000 7f29e3c00700 1 Processor -- start 2019-12-05T05:38:54.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:54.166+0000 7f29e3c00700 1 -- start start 2019-12-05T05:38:54.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:54.166+0000 7f29e3c00700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f29dc119c30 0x7f29dc12c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:38:54.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:54.166+0000 7f29e3c00700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f29dc0a4e90 0x7f29dc12c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:38:54.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:54.166+0000 7f29e3c00700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29dc11b980 con 0x7f29dc11a730 2019-12-05T05:38:54.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:54.166+0000 7f29e3c00700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29dc0a6a20 con 0x7f29dc0a4e90 2019-12-05T05:38:54.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:54.166+0000 7f29e3c00700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29dc0a71c0 con 0x7f29dc119c30 2019-12-05T05:38:54.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:54.166+0000 7f29e199c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f29dc119c30 0x7f29dc12c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:38:54.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:54.166+0000 7f29e199c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f29dc119c30 0x7f29dc12c3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:37938/0 (socket says 172.21.15.42:37938) 2019-12-05T05:38:54.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:54.166+0000 7f29e199c700 1 -- 172.21.15.42:0/2272553909 learned_addr learned my addr 172.21.15.42:0/2272553909 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:38:54.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:54.166+0000 7f29e299e700 1 --2- 172.21.15.42:0/2272553909 >> v2:172.21.15.40:3300/0 conn(0x7f29dc0a4e90 0x7f29dc12c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:38:54.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:54.166+0000 7f29d2ffd700 1 -- 172.21.15.42:0/2272553909 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2779183777 0 0) 0x7f29dc11b980 con 0x7f29dc11a730 2019-12-05T05:38:54.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:54.166+0000 7f29d2ffd700 1 -- 172.21.15.42:0/2272553909 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29c40030b0 con 0x7f29dc11a730 2019-12-05T05:38:54.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:54.166+0000 7f29e199c700 1 -- 172.21.15.42:0/2272553909 >> v2:172.21.15.40:3300/0 conn(0x7f29dc0a4e90 msgr2=0x7f29dc12c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:38:54.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:54.166+0000 7f29e199c700 1 --2- 172.21.15.42:0/2272553909 >> v2:172.21.15.40:3300/0 conn(0x7f29dc0a4e90 0x7f29dc12c940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:38:54.184 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:54.166+0000 7f29e199c700 1 -- 172.21.15.42:0/2272553909 >> v1:172.21.15.40:6789/0 conn(0x7f29dc11a730 legacy=0x7f29dc117c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:38:54.184 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:54.166+0000 7f29e199c700 1 -- 172.21.15.42:0/2272553909 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29dc0a9640 con 0x7f29dc119c30 2019-12-05T05:38:54.184 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:54.166+0000 7f29e199c700 1 --2- 172.21.15.42:0/2272553909 >> v2:172.21.15.42:3300/0 conn(0x7f29dc119c30 0x7f29dc12c3f0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f29dc11a0c0 tx=0x7f29cc010cc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:38:54.184 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:54.166+0000 7f29e199c700 1 --2- 172.21.15.42:0/2272553909 >> v2:172.21.15.42:3300/0 conn(0x7f29dc119c30 0x7f29dc12c3f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f29dc11a0c0 tx=0x7f29cc010cc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:38:54.185 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:54.166+0000 7f29e199c700 1 --2- 172.21.15.42:0/2272553909 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f29dc119c30 0x7f29dc12c3f0 secure :-1 s=READY pgs=1200 cs=0 l=1 rx=0x7f29dc11a0c0 tx=0x7f29cc010cc0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:38:54.185 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:54.166+0000 7f29e199c700 1 --2- 172.21.15.42:0/2272553909 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f29dc119c30 0x7f29dc12c3f0 secure :-1 s=THROTTLE_DONE pgs=1200 cs=0 l=1 rx=0x7f29dc11a0c0 tx=0x7f29cc010cc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:38:54.185 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:54.166+0000 7f29d2ffd700 1 -- 172.21.15.42:0/2272553909 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f29cc00eb10 con 0x7f29dc119c30 2019-12-05T05:38:54.185 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:54.166+0000 7f29e199c700 1 --2- 172.21.15.42:0/2272553909 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f29dc119c30 0x7f29dc12c3f0 secure :-1 s=THROTTLE_DONE pgs=1200 cs=0 l=1 rx=0x7f29dc11a0c0 tx=0x7f29cc010cc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:38:54.185 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:54.166+0000 7f29e199c700 1 --2- 172.21.15.42:0/2272553909 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f29dc119c30 0x7f29dc12c3f0 secure :-1 s=THROTTLE_DONE pgs=1200 cs=0 l=1 rx=0x7f29dc11a0c0 tx=0x7f29cc010cc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:38:54.185 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:54.166+0000 7f29d2ffd700 1 -- 172.21.15.42:0/2272553909 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29cc00e730 con 0x7f29dc119c30 2019-12-05T05:38:54.185 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:54.166+0000 7f29d2ffd700 1 -- 172.21.15.42:0/2272553909 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f29cc005730 con 0x7f29dc119c30 2019-12-05T05:38:54.186 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:54.166+0000 7f29e3c00700 1 -- 172.21.15.42:0/2272553909 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29dc0a9910 con 0x7f29dc119c30 2019-12-05T05:38:54.186 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:54.166+0000 7f29e3c00700 1 -- 172.21.15.42:0/2272553909 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29dc137f10 con 0x7f29dc119c30 2019-12-05T05:38:54.186 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:54.170+0000 7f29e199c700 1 --2- 172.21.15.42:0/2272553909 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f29dc119c30 0x7f29dc12c3f0 secure :-1 s=THROTTLE_DONE pgs=1200 cs=0 l=1 rx=0x7f29dc11a0c0 tx=0x7f29cc010cc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:38:54.186 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:54.170+0000 7f29d2ffd700 1 -- 172.21.15.42:0/2272553909 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f29cc007180 con 0x7f29dc119c30 2019-12-05T05:38:54.186 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:54.170+0000 7f29e199c700 1 --2- 172.21.15.42:0/2272553909 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f29dc119c30 0x7f29dc12c3f0 secure :-1 s=THROTTLE_DONE pgs=1200 cs=0 l=1 rx=0x7f29dc11a0c0 tx=0x7f29cc010cc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:38:54.186 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:54.170+0000 7f29d2ffd700 1 --2- 172.21.15.42:0/2272553909 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f29c403b280 0x7f29c403d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:38:54.186 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:54.170+0000 7f29d2ffd700 1 -- 172.21.15.42:0/2272553909 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 8009+0+0 (secure 0 0 0) 0x7f29cc04b540 con 0x7f29dc119c30 2019-12-05T05:38:54.187 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:54.170+0000 7f29e219d700 1 --2- 172.21.15.42:0/2272553909 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f29c403b280 0x7f29c403d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:38:54.188 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:54.170+0000 7f29e219d700 1 --2- 172.21.15.42:0/2272553909 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f29c403b280 0x7f29c403d790 crc :-1 s=READY pgs=519 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:38:54.188 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:54.174+0000 7f29e3c00700 1 -- 172.21.15.42:0/2272553909 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29dc05d110 con 0x7f29dc119c30 2019-12-05T05:38:54.188 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:54.182+0000 7f29e199c700 1 --2- 172.21.15.42:0/2272553909 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f29dc119c30 0x7f29dc12c3f0 secure :-1 s=THROTTLE_DONE pgs=1200 cs=0 l=1 rx=0x7f29dc11a0c0 tx=0x7f29cc010cc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:38:54.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:54.182+0000 7f29d2ffd700 1 -- 172.21.15.42:0/2272553909 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f29cc0325c0 con 0x7f29dc119c30 2019-12-05T05:38:54.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:54.350+0000 7f29e3c00700 1 -- 172.21.15.42:0/2272553909 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f29dc05b8c0 con 0x7f29dc119c30 2019-12-05T05:38:54.358 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:54.350+0000 7f29e199c700 1 --2- 172.21.15.42:0/2272553909 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f29dc119c30 0x7f29dc12c3f0 secure :-1 s=THROTTLE_DONE pgs=1200 cs=0 l=1 rx=0x7f29dc11a0c0 tx=0x7f29cc010cc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:38:54.358 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:54.350+0000 7f29d2ffd700 1 -- 172.21.15.42:0/2272553909 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 148 v148) v1 ==== 95+0+2109 (secure 0 0 0) 0x7f29cc032820 con 0x7f29dc119c30 2019-12-05T05:38:54.358 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 148 2019-12-05T05:38:54.358 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:38:54.359 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":148,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7416,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/123054133","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":123054133},{"type":"v1","addr":"172.21.15.40:6833","nonce":123054133}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":148}],"filesystems":[{"mdsmap":{"epoch":147,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:37:13.880687+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5693},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5693":{"gid":5693,"name":"a","rank":0,"incarnation":144,"state":"up:active","state_seq":39,"addr":"172.21.15.42:6835/3838877789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3838877789},{"type":"v1","addr":"172.21.15.42:6835","nonce":3838877789}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:38:54.367 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:54.354+0000 7f29e3c00700 1 -- 172.21.15.42:0/2272553909 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f29c403b280 msgr2=0x7f29c403d790 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:38:54.368 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:54.354+0000 7f29e3c00700 1 --2- 172.21.15.42:0/2272553909 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f29c403b280 0x7f29c403d790 crc :-1 s=READY pgs=519 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:38:54.368 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:54.354+0000 7f29e3c00700 1 -- 172.21.15.42:0/2272553909 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f29dc119c30 msgr2=0x7f29dc12c3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:38:54.368 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:54.354+0000 7f29e3c00700 1 --2- 172.21.15.42:0/2272553909 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f29dc119c30 0x7f29dc12c3f0 secure :-1 s=READY pgs=1200 cs=0 l=1 rx=0x7f29dc11a0c0 tx=0x7f29cc010cc0).stop 2019-12-05T05:38:54.368 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:54.354+0000 7f29e3c00700 1 -- 172.21.15.42:0/2272553909 shutdown_connections 2019-12-05T05:38:54.368 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:54.354+0000 7f29e3c00700 1 --2- 172.21.15.42:0/2272553909 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f29c403b280 0x7f29c403d790 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:38:54.368 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:54.354+0000 7f29e3c00700 1 --2- 172.21.15.42:0/2272553909 >> v2:172.21.15.40:3300/0 conn(0x7f29dc0a4e90 0x7f29dc12c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:38:54.369 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:54.354+0000 7f29e3c00700 1 --2- 172.21.15.42:0/2272553909 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f29dc119c30 0x7f29dc12c3f0 unknown :-1 s=CLOSED pgs=1200 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:38:54.369 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:54.354+0000 7f29e3c00700 1 -- 172.21.15.42:0/2272553909 shutdown_connections 2019-12-05T05:38:54.369 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:54.354+0000 7f29e3c00700 1 -- 172.21.15.42:0/2272553909 wait complete. 2019-12-05T05:38:54.369 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:54.354+0000 7f29e3c00700 1 -- 172.21.15.42:0/2272553909 >> 172.21.15.42:0/2272553909 conn(0x7f29dc059ef0 msgr2=0x7f29dc0a7f90 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:38:54.370 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T05:38:56.371 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:38:56.479 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:56.470+0000 7efc9b9f7700 1 Processor -- start 2019-12-05T05:38:56.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:56.470+0000 7efc9b9f7700 1 -- start start 2019-12-05T05:38:56.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:56.470+0000 7efc9b9f7700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7efc9409fae0 0x7efc9414fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:38:56.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:56.470+0000 7efc9b9f7700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc940a0710 con 0x7efc9408df80 2019-12-05T05:38:56.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:56.470+0000 7efc9b9f7700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc9407ef20 con 0x7efc940a7740 2019-12-05T05:38:56.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:56.470+0000 7efc9b9f7700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc940a0b00 con 0x7efc9409fae0 2019-12-05T05:38:56.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:56.470+0000 7efc99f94700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7efc940a7740 0x7efc9407e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:37986/0 (socket says 172.21.15.42:37986) 2019-12-05T05:38:56.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:56.470+0000 7efc99f94700 1 -- 172.21.15.42:0/1569605440 learned_addr learned my addr 172.21.15.42:0/1569605440 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:38:56.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:56.470+0000 7efc9a795700 1 --2- 172.21.15.42:0/1569605440 >> v2:172.21.15.42:3300/0 conn(0x7efc9409fae0 0x7efc9414fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:38:56.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:56.470+0000 7efc98f92700 1 -- 172.21.15.42:0/1569605440 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 760766244 0 0) 0x7efc940a0710 con 0x7efc9408df80 2019-12-05T05:38:56.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:56.470+0000 7efc98f92700 1 -- 172.21.15.42:0/1569605440 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc800030b0 con 0x7efc9408df80 2019-12-05T05:38:56.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:56.470+0000 7efc98f92700 1 -- 172.21.15.42:0/1569605440 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2088830018 0 0) 0x7efc9407ef20 con 0x7efc940a7740 2019-12-05T05:38:56.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:56.470+0000 7efc98f92700 1 -- 172.21.15.42:0/1569605440 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc940a0710 con 0x7efc940a7740 2019-12-05T05:38:56.482 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:56.470+0000 7efc9a795700 1 -- 172.21.15.42:0/1569605440 >> v1:172.21.15.42:6789/0 conn(0x7efc940a7740 legacy=0x7efc9407e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:38:56.482 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:56.470+0000 7efc9a795700 1 -- 172.21.15.42:0/1569605440 >> v1:172.21.15.40:6789/0 conn(0x7efc9408df80 legacy=0x7efc9409f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:38:56.482 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:56.470+0000 7efc9a795700 1 -- 172.21.15.42:0/1569605440 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc940a9ba0 con 0x7efc9409fae0 2019-12-05T05:38:56.482 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:56.470+0000 7efc9a795700 1 --2- 172.21.15.42:0/1569605440 >> v2:172.21.15.42:3300/0 conn(0x7efc9409fae0 0x7efc9414fff0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7efc8c008b10 tx=0x7efc8c00ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:38:56.482 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:56.470+0000 7efc9a795700 1 --2- 172.21.15.42:0/1569605440 >> v2:172.21.15.42:3300/0 conn(0x7efc9409fae0 0x7efc9414fff0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7efc8c008b10 tx=0x7efc8c00ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:38:56.482 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:56.470+0000 7efc9a795700 1 --2- 172.21.15.42:0/1569605440 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efc9409fae0 0x7efc9414fff0 secure :-1 s=READY pgs=1202 cs=0 l=1 rx=0x7efc8c008b10 tx=0x7efc8c00ff70).ready entity=mon.1 client_cookie=beda554464c18442 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:38:56.482 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:56.470+0000 7efc9a795700 1 --2- 172.21.15.42:0/1569605440 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efc9409fae0 0x7efc9414fff0 secure :-1 s=THROTTLE_DONE pgs=1202 cs=0 l=1 rx=0x7efc8c008b10 tx=0x7efc8c00ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:38:56.482 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:56.470+0000 7efc98f92700 1 -- 172.21.15.42:0/1569605440 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efc8c010970 con 0x7efc9409fae0 2019-12-05T05:38:56.483 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:56.470+0000 7efc9a795700 1 --2- 172.21.15.42:0/1569605440 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efc9409fae0 0x7efc9414fff0 secure :-1 s=THROTTLE_DONE pgs=1202 cs=0 l=1 rx=0x7efc8c008b10 tx=0x7efc8c00ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:38:56.483 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:56.470+0000 7efc98f92700 1 -- 172.21.15.42:0/1569605440 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc8c008b50 con 0x7efc9409fae0 2019-12-05T05:38:56.483 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:56.470+0000 7efc9a795700 1 --2- 172.21.15.42:0/1569605440 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efc9409fae0 0x7efc9414fff0 secure :-1 s=THROTTLE_DONE pgs=1202 cs=0 l=1 rx=0x7efc8c008b10 tx=0x7efc8c00ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:38:56.483 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:56.470+0000 7efc98f92700 1 -- 172.21.15.42:0/1569605440 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efc8c0110c0 con 0x7efc9409fae0 2019-12-05T05:38:56.483 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:56.470+0000 7efc9b9f7700 1 -- 172.21.15.42:0/1569605440 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efc9409fae0 msgr2=0x7efc9414fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:38:56.483 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:56.470+0000 7efc9b9f7700 1 --2- 172.21.15.42:0/1569605440 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efc9409fae0 0x7efc9414fff0 secure :-1 s=READY pgs=1202 cs=0 l=1 rx=0x7efc8c008b10 tx=0x7efc8c00ff70).stop 2019-12-05T05:38:56.483 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:56.470+0000 7efc9b9f7700 1 -- 172.21.15.42:0/1569605440 shutdown_connections 2019-12-05T05:38:56.484 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:56.470+0000 7efc9b9f7700 1 --2- 172.21.15.42:0/1569605440 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efc9409fae0 0x7efc9414fff0 unknown :-1 s=CLOSED pgs=1202 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:38:56.484 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:56.470+0000 7efc9b9f7700 1 -- 172.21.15.42:0/1569605440 shutdown_connections 2019-12-05T05:38:56.484 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:56.470+0000 7efc9b9f7700 1 -- 172.21.15.42:0/1569605440 wait complete. 2019-12-05T05:38:56.484 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:56.470+0000 7efc9b9f7700 1 -- 172.21.15.42:0/1569605440 >> 172.21.15.42:0/1569605440 conn(0x7efc940aabe0 msgr2=0x7efc940a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:38:56.492 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:56.474+0000 7efc9b9f7700 1 Processor -- start 2019-12-05T05:38:56.492 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:56.474+0000 7efc9b9f7700 1 -- start start 2019-12-05T05:38:56.492 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:56.474+0000 7efc9b9f7700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7efc941259d0 0x7efc940a57e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:38:56.492 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:56.474+0000 7efc9b9f7700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc9407d9e0 con 0x7efc9414a1c0 2019-12-05T05:38:56.492 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:56.474+0000 7efc9b9f7700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc9414a760 con 0x7efc94128f90 2019-12-05T05:38:56.492 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:56.474+0000 7efc9b9f7700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc94120000 con 0x7efc941259d0 2019-12-05T05:38:56.493 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:56.474+0000 7efc99793700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7efc94128f90 0x7efc9412e3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:37992/0 (socket says 172.21.15.42:37992) 2019-12-05T05:38:56.493 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:56.474+0000 7efc99f94700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7efc9414a1c0 0x7efc9407d1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:47430/0 (socket says 172.21.15.42:47430) 2019-12-05T05:38:56.493 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:56.474+0000 7efc99793700 1 -- 172.21.15.42:0/3027584717 learned_addr learned my addr 172.21.15.42:0/3027584717 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:38:56.493 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:56.474+0000 7efc9a795700 1 --2- 172.21.15.42:0/3027584717 >> v2:172.21.15.42:3300/0 conn(0x7efc941259d0 0x7efc940a57e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:38:56.493 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:56.474+0000 7efc8a7fc700 1 -- 172.21.15.42:0/3027584717 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2555578152 0 0) 0x7efc9407d9e0 con 0x7efc9414a1c0 2019-12-05T05:38:56.493 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:56.474+0000 7efc8a7fc700 1 -- 172.21.15.42:0/3027584717 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc7c0030b0 con 0x7efc9414a1c0 2019-12-05T05:38:56.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:56.478+0000 7efc9a795700 1 -- 172.21.15.42:0/3027584717 >> v1:172.21.15.42:6789/0 conn(0x7efc94128f90 legacy=0x7efc9412e3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:38:56.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:56.478+0000 7efc8a7fc700 1 -- 172.21.15.42:0/3027584717 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1886761761 0 0) 0x7efc9414a760 con 0x7efc94128f90 2019-12-05T05:38:56.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:56.478+0000 7efc9a795700 1 -- 172.21.15.42:0/3027584717 >> v1:172.21.15.40:6789/0 conn(0x7efc9414a1c0 legacy=0x7efc9407d1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:38:56.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:56.478+0000 7efc9a795700 1 -- 172.21.15.42:0/3027584717 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc94129880 con 0x7efc941259d0 2019-12-05T05:38:56.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:56.478+0000 7efc9a795700 1 --2- 172.21.15.42:0/3027584717 >> v2:172.21.15.42:3300/0 conn(0x7efc941259d0 0x7efc940a57e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7efc8c008b10 tx=0x7efc8c008810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:38:56.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:56.478+0000 7efc9a795700 1 --2- 172.21.15.42:0/3027584717 >> v2:172.21.15.42:3300/0 conn(0x7efc941259d0 0x7efc940a57e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7efc8c008b10 tx=0x7efc8c008810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:38:56.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:56.478+0000 7efc9a795700 1 --2- 172.21.15.42:0/3027584717 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efc941259d0 0x7efc940a57e0 secure :-1 s=READY pgs=1204 cs=0 l=1 rx=0x7efc8c008b10 tx=0x7efc8c008810).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:38:56.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:56.478+0000 7efc9a795700 1 --2- 172.21.15.42:0/3027584717 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efc941259d0 0x7efc940a57e0 secure :-1 s=THROTTLE_DONE pgs=1204 cs=0 l=1 rx=0x7efc8c008b10 tx=0x7efc8c008810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:38:56.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:56.478+0000 7efc8a7fc700 1 -- 172.21.15.42:0/3027584717 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efc8c005ab0 con 0x7efc941259d0 2019-12-05T05:38:56.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:56.478+0000 7efc9a795700 1 --2- 172.21.15.42:0/3027584717 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efc941259d0 0x7efc940a57e0 secure :-1 s=THROTTLE_DONE pgs=1204 cs=0 l=1 rx=0x7efc8c008b10 tx=0x7efc8c008810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:38:56.496 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:56.478+0000 7efc9a795700 1 --2- 172.21.15.42:0/3027584717 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efc941259d0 0x7efc940a57e0 secure :-1 s=THROTTLE_DONE pgs=1204 cs=0 l=1 rx=0x7efc8c008b10 tx=0x7efc8c008810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:38:56.496 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:56.478+0000 7efc8a7fc700 1 -- 172.21.15.42:0/3027584717 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc8c010b70 con 0x7efc941259d0 2019-12-05T05:38:56.496 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:56.478+0000 7efc9b9f7700 1 -- 172.21.15.42:0/3027584717 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc94129b50 con 0x7efc941259d0 2019-12-05T05:38:56.497 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:56.478+0000 7efc8a7fc700 1 -- 172.21.15.42:0/3027584717 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efc8c00f470 con 0x7efc941259d0 2019-12-05T05:38:56.497 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:56.478+0000 7efc9b9f7700 1 -- 172.21.15.42:0/3027584717 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc94131bf0 con 0x7efc941259d0 2019-12-05T05:38:56.497 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:56.478+0000 7efc9a795700 1 --2- 172.21.15.42:0/3027584717 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efc941259d0 0x7efc940a57e0 secure :-1 s=THROTTLE_DONE pgs=1204 cs=0 l=1 rx=0x7efc8c008b10 tx=0x7efc8c008810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:38:56.498 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:56.478+0000 7efc8a7fc700 1 -- 172.21.15.42:0/3027584717 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7efc8c00f670 con 0x7efc941259d0 2019-12-05T05:38:56.498 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:56.478+0000 7efc9a795700 1 --2- 172.21.15.42:0/3027584717 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efc941259d0 0x7efc940a57e0 secure :-1 s=THROTTLE_DONE pgs=1204 cs=0 l=1 rx=0x7efc8c008b10 tx=0x7efc8c008810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:38:56.499 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:56.478+0000 7efc8a7fc700 1 --2- 172.21.15.42:0/3027584717 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7efc7c03b130 0x7efc7c03d640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:38:56.499 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:56.478+0000 7efc8a7fc700 1 -- 172.21.15.42:0/3027584717 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 8009+0+0 (secure 0 0 0) 0x7efc8c04adf0 con 0x7efc941259d0 2019-12-05T05:38:56.499 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:56.478+0000 7efc99f94700 1 --2- 172.21.15.42:0/3027584717 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7efc7c03b130 0x7efc7c03d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:38:56.499 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:56.482+0000 7efc99f94700 1 --2- 172.21.15.42:0/3027584717 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7efc7c03b130 0x7efc7c03d640 crc :-1 s=READY pgs=520 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:38:56.499 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:56.482+0000 7efc9b9f7700 1 -- 172.21.15.42:0/3027584717 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc9405d110 con 0x7efc941259d0 2019-12-05T05:38:56.502 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:56.486+0000 7efc9a795700 1 --2- 172.21.15.42:0/3027584717 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efc941259d0 0x7efc940a57e0 secure :-1 s=THROTTLE_DONE pgs=1204 cs=0 l=1 rx=0x7efc8c008b10 tx=0x7efc8c008810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:38:56.503 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:56.486+0000 7efc8a7fc700 1 -- 172.21.15.42:0/3027584717 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7efc8c00f980 con 0x7efc941259d0 2019-12-05T05:38:56.661 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:56.654+0000 7efc9b9f7700 1 -- 172.21.15.42:0/3027584717 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efc9407e180 con 0x7efc941259d0 2019-12-05T05:38:56.662 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:56.654+0000 7efc9a795700 1 --2- 172.21.15.42:0/3027584717 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efc941259d0 0x7efc940a57e0 secure :-1 s=THROTTLE_DONE pgs=1204 cs=0 l=1 rx=0x7efc8c008b10 tx=0x7efc8c008810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:38:56.662 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:56.654+0000 7efc8a7fc700 1 -- 172.21.15.42:0/3027584717 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 148 v148) v1 ==== 95+0+2109 (secure 0 0 0) 0x7efc8c032ca0 con 0x7efc941259d0 2019-12-05T05:38:56.662 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 148 2019-12-05T05:38:56.662 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:38:56.662 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":148,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7416,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/123054133","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":123054133},{"type":"v1","addr":"172.21.15.40:6833","nonce":123054133}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":148}],"filesystems":[{"mdsmap":{"epoch":147,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:37:13.880687+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5693},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5693":{"gid":5693,"name":"a","rank":0,"incarnation":144,"state":"up:active","state_seq":39,"addr":"172.21.15.42:6835/3838877789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3838877789},{"type":"v1","addr":"172.21.15.42:6835","nonce":3838877789}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:38:56.666 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:56.658+0000 7efc9b9f7700 1 -- 172.21.15.42:0/3027584717 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7efc7c03b130 msgr2=0x7efc7c03d640 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:38:56.667 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:56.658+0000 7efc9b9f7700 1 --2- 172.21.15.42:0/3027584717 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7efc7c03b130 0x7efc7c03d640 crc :-1 s=READY pgs=520 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:38:56.667 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:56.658+0000 7efc9b9f7700 1 -- 172.21.15.42:0/3027584717 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efc941259d0 msgr2=0x7efc940a57e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:38:56.668 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:56.658+0000 7efc9b9f7700 1 --2- 172.21.15.42:0/3027584717 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efc941259d0 0x7efc940a57e0 secure :-1 s=READY pgs=1204 cs=0 l=1 rx=0x7efc8c008b10 tx=0x7efc8c008810).stop 2019-12-05T05:38:56.668 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:56.658+0000 7efc9b9f7700 1 -- 172.21.15.42:0/3027584717 shutdown_connections 2019-12-05T05:38:56.668 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:56.658+0000 7efc9b9f7700 1 --2- 172.21.15.42:0/3027584717 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7efc7c03b130 0x7efc7c03d640 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:38:56.668 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:56.658+0000 7efc9b9f7700 1 --2- 172.21.15.42:0/3027584717 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efc941259d0 0x7efc940a57e0 unknown :-1 s=CLOSED pgs=1204 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:38:56.668 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:56.658+0000 7efc9b9f7700 1 -- 172.21.15.42:0/3027584717 shutdown_connections 2019-12-05T05:38:56.668 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:56.658+0000 7efc9b9f7700 1 -- 172.21.15.42:0/3027584717 wait complete. 2019-12-05T05:38:56.669 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:56.658+0000 7efc9b9f7700 1 -- 172.21.15.42:0/3027584717 >> 172.21.15.42:0/3027584717 conn(0x7efc940a8110 msgr2=0x7efc94119870 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:38:56.679 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T05:38:58.681 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:38:58.786 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:58.774+0000 7f5f5b711700 1 Processor -- start 2019-12-05T05:38:58.786 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:58.774+0000 7f5f5b711700 1 -- start start 2019-12-05T05:38:58.786 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:58.774+0000 7f5f5b711700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5f5409fae0 0x7f5f5414fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:38:58.786 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:58.774+0000 7f5f5b711700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f540a0710 con 0x7f5f540a7740 2019-12-05T05:38:58.786 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:58.774+0000 7f5f5b711700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f5407ef20 con 0x7f5f5408df80 2019-12-05T05:38:58.787 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:58.774+0000 7f5f5b711700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f540a0b00 con 0x7f5f5409fae0 2019-12-05T05:38:58.787 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:58.774+0000 7f5f594ad700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f5f5408df80 0x7f5f5409f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:38000/0 (socket says 172.21.15.42:38000) 2019-12-05T05:38:58.788 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:58.774+0000 7f5f594ad700 1 -- 172.21.15.42:0/1350009190 learned_addr learned my addr 172.21.15.42:0/1350009190 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:38:58.788 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:58.774+0000 7f5f5a4af700 1 --2- 172.21.15.42:0/1350009190 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5f5409fae0 0x7f5f5414fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:38:58.788 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:58.774+0000 7f5f58cac700 1 -- 172.21.15.42:0/1350009190 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 498947057 0 0) 0x7f5f5407ef20 con 0x7f5f5408df80 2019-12-05T05:38:58.788 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:58.774+0000 7f5f58cac700 1 -- 172.21.15.42:0/1350009190 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f400030b0 con 0x7f5f5408df80 2019-12-05T05:38:58.789 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:58.774+0000 7f5f58cac700 1 -- 172.21.15.42:0/1350009190 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2080614611 0 0) 0x7f5f540a0710 con 0x7f5f540a7740 2019-12-05T05:38:58.789 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:58.774+0000 7f5f58cac700 1 -- 172.21.15.42:0/1350009190 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f5407ef20 con 0x7f5f540a7740 2019-12-05T05:38:58.789 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:58.774+0000 7f5f58cac700 1 -- 172.21.15.42:0/1350009190 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3442332471 0 0) 0x7f5f400030b0 con 0x7f5f5408df80 2019-12-05T05:38:58.789 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:58.774+0000 7f5f5a4af700 1 -- 172.21.15.42:0/1350009190 >> v1:172.21.15.42:6789/0 conn(0x7f5f5408df80 legacy=0x7f5f5409f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:38:58.789 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:58.774+0000 7f5f5a4af700 1 -- 172.21.15.42:0/1350009190 >> v1:172.21.15.40:6789/0 conn(0x7f5f540a7740 legacy=0x7f5f5407e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:38:58.790 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:58.774+0000 7f5f5a4af700 1 -- 172.21.15.42:0/1350009190 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f540a9ba0 con 0x7f5f5409fae0 2019-12-05T05:38:58.790 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:58.774+0000 7f5f5a4af700 1 --2- 172.21.15.42:0/1350009190 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5f5409fae0 0x7f5f5414fff0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f5f4c008b10 tx=0x7f5f44001950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:38:58.790 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:58.778+0000 7f5f5a4af700 1 --2- 172.21.15.42:0/1350009190 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5f5409fae0 0x7f5f5414fff0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f5f4c008b10 tx=0x7f5f44001950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:38:58.790 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:58.778+0000 7f5f5a4af700 1 --2- 172.21.15.42:0/1350009190 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5f5409fae0 0x7f5f5414fff0 secure :-1 s=READY pgs=254 cs=0 l=1 rx=0x7f5f4c008b10 tx=0x7f5f44001950).ready entity=mon.2 client_cookie=8d902d8a32fd24a4 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:38:58.790 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:58.778+0000 7f5f5a4af700 1 --2- 172.21.15.42:0/1350009190 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5f5409fae0 0x7f5f5414fff0 secure :-1 s=THROTTLE_DONE pgs=254 cs=0 l=1 rx=0x7f5f4c008b10 tx=0x7f5f44001950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:38:58.791 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:58.778+0000 7f5f58cac700 1 -- 172.21.15.42:0/1350009190 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5f44001750 con 0x7f5f5409fae0 2019-12-05T05:38:58.791 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:58.778+0000 7f5f5a4af700 1 --2- 172.21.15.42:0/1350009190 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5f5409fae0 0x7f5f5414fff0 secure :-1 s=THROTTLE_DONE pgs=254 cs=0 l=1 rx=0x7f5f4c008b10 tx=0x7f5f44001950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:38:58.791 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:58.778+0000 7f5f58cac700 1 -- 172.21.15.42:0/1350009190 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f44001560 con 0x7f5f5409fae0 2019-12-05T05:38:58.791 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:58.778+0000 7f5f5a4af700 1 --2- 172.21.15.42:0/1350009190 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5f5409fae0 0x7f5f5414fff0 secure :-1 s=THROTTLE_DONE pgs=254 cs=0 l=1 rx=0x7f5f4c008b10 tx=0x7f5f44001950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:38:58.791 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:58.778+0000 7f5f58cac700 1 -- 172.21.15.42:0/1350009190 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5f4c010a60 con 0x7f5f5409fae0 2019-12-05T05:38:58.791 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:58.778+0000 7f5f5b711700 1 -- 172.21.15.42:0/1350009190 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5f5409fae0 msgr2=0x7f5f5414fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:38:58.791 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:58.778+0000 7f5f5b711700 1 --2- 172.21.15.42:0/1350009190 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5f5409fae0 0x7f5f5414fff0 secure :-1 s=READY pgs=254 cs=0 l=1 rx=0x7f5f4c008b10 tx=0x7f5f44001950).stop 2019-12-05T05:38:58.792 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:58.778+0000 7f5f5b711700 1 -- 172.21.15.42:0/1350009190 shutdown_connections 2019-12-05T05:38:58.792 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:58.778+0000 7f5f5b711700 1 --2- 172.21.15.42:0/1350009190 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5f5409fae0 0x7f5f5414fff0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:38:58.792 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:58.778+0000 7f5f5b711700 1 -- 172.21.15.42:0/1350009190 shutdown_connections 2019-12-05T05:38:58.792 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:58.778+0000 7f5f5b711700 1 -- 172.21.15.42:0/1350009190 wait complete. 2019-12-05T05:38:58.792 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:58.778+0000 7f5f5b711700 1 -- 172.21.15.42:0/1350009190 >> 172.21.15.42:0/1350009190 conn(0x7f5f540aabe0 msgr2=0x7f5f540a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:38:58.795 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:58.782+0000 7f5f5b711700 1 Processor -- start 2019-12-05T05:38:58.795 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:58.782+0000 7f5f5b711700 1 -- start start 2019-12-05T05:38:58.795 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:58.782+0000 7f5f5b711700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5f54124880 0x7f5f5411bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:38:58.795 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:58.782+0000 7f5f5b711700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5f54135840 0x7f5f54137c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:38:58.796 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:58.782+0000 7f5f5b711700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f5411c320 con 0x7f5f54119a10 2019-12-05T05:38:58.796 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:58.782+0000 7f5f5b711700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f541214a0 con 0x7f5f54135840 2019-12-05T05:38:58.796 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:58.782+0000 7f5f5b711700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f54121740 con 0x7f5f54124880 2019-12-05T05:38:58.796 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:58.782+0000 7f5f5a4af700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5f54135840 0x7f5f54137c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:38:58.796 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:58.782+0000 7f5f5a4af700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5f54135840 0x7f5f54137c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:37970/0 (socket says 172.21.15.42:37970) 2019-12-05T05:38:58.796 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:58.782+0000 7f5f5a4af700 1 -- 172.21.15.42:0/3097903853 learned_addr learned my addr 172.21.15.42:0/3097903853 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:38:58.797 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:58.782+0000 7f5f59cae700 1 --2- 172.21.15.42:0/3097903853 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5f54124880 0x7f5f5411bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:38:58.797 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:58.782+0000 7f5f5a4af700 1 -- 172.21.15.42:0/3097903853 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5f54124880 msgr2=0x7f5f5411bdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:38:58.797 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:58.782+0000 7f5f5a4af700 1 --2- 172.21.15.42:0/3097903853 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5f54124880 0x7f5f5411bdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:38:58.797 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:58.782+0000 7f5f5a4af700 1 -- 172.21.15.42:0/3097903853 >> v1:172.21.15.40:6789/0 conn(0x7f5f54119a10 legacy=0x7f5f5411f820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:38:58.797 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:58.782+0000 7f5f4a7fc700 1 -- 172.21.15.42:0/3097903853 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 401545618 0 0) 0x7f5f44001a20 con 0x7f5f54119a10 2019-12-05T05:38:58.797 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:58.782+0000 7f5f5a4af700 1 -- 172.21.15.42:0/3097903853 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f54120020 con 0x7f5f54135840 2019-12-05T05:38:58.798 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:58.782+0000 7f5f5a4af700 1 --2- 172.21.15.42:0/3097903853 >> v2:172.21.15.42:3300/0 conn(0x7f5f54135840 0x7f5f54137c80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f5f54121660 tx=0x7f5f4c010d80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:38:58.798 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:58.782+0000 7f5f5a4af700 1 --2- 172.21.15.42:0/3097903853 >> v2:172.21.15.42:3300/0 conn(0x7f5f54135840 0x7f5f54137c80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f5f54121660 tx=0x7f5f4c010d80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:38:58.798 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:58.782+0000 7f5f5a4af700 1 --2- 172.21.15.42:0/3097903853 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5f54135840 0x7f5f54137c80 secure :-1 s=READY pgs=1206 cs=0 l=1 rx=0x7f5f54121660 tx=0x7f5f4c010d80).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:38:58.799 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:58.782+0000 7f5f5a4af700 1 --2- 172.21.15.42:0/3097903853 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5f54135840 0x7f5f54137c80 secure :-1 s=THROTTLE_DONE pgs=1206 cs=0 l=1 rx=0x7f5f54121660 tx=0x7f5f4c010d80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:38:58.799 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:58.782+0000 7f5f4a7fc700 1 -- 172.21.15.42:0/3097903853 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5f4c00f600 con 0x7f5f54135840 2019-12-05T05:38:58.799 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:58.782+0000 7f5f5a4af700 1 --2- 172.21.15.42:0/3097903853 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5f54135840 0x7f5f54137c80 secure :-1 s=THROTTLE_DONE pgs=1206 cs=0 l=1 rx=0x7f5f54121660 tx=0x7f5f4c010d80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:38:58.799 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:58.782+0000 7f5f5a4af700 1 --2- 172.21.15.42:0/3097903853 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5f54135840 0x7f5f54137c80 secure :-1 s=THROTTLE_DONE pgs=1206 cs=0 l=1 rx=0x7f5f54121660 tx=0x7f5f4c010d80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:38:58.799 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:58.782+0000 7f5f4a7fc700 1 -- 172.21.15.42:0/3097903853 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f4c008d40 con 0x7f5f54135840 2019-12-05T05:38:58.799 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:58.782+0000 7f5f5b711700 1 -- 172.21.15.42:0/3097903853 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f54130400 con 0x7f5f54135840 2019-12-05T05:38:58.799 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:58.782+0000 7f5f4a7fc700 1 -- 172.21.15.42:0/3097903853 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5f4c05b390 con 0x7f5f54135840 2019-12-05T05:38:58.800 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:58.786+0000 7f5f5b711700 1 -- 172.21.15.42:0/3097903853 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f5411ed10 con 0x7f5f54135840 2019-12-05T05:38:58.800 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:58.786+0000 7f5f5a4af700 1 --2- 172.21.15.42:0/3097903853 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5f54135840 0x7f5f54137c80 secure :-1 s=THROTTLE_DONE pgs=1206 cs=0 l=1 rx=0x7f5f54121660 tx=0x7f5f4c010d80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:38:58.800 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:58.786+0000 7f5f4a7fc700 1 -- 172.21.15.42:0/3097903853 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f5f4c05b6a0 con 0x7f5f54135840 2019-12-05T05:38:58.800 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:58.786+0000 7f5f5a4af700 1 --2- 172.21.15.42:0/3097903853 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5f54135840 0x7f5f54137c80 secure :-1 s=THROTTLE_DONE pgs=1206 cs=0 l=1 rx=0x7f5f54121660 tx=0x7f5f4c010d80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:38:58.800 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:58.786+0000 7f5f4a7fc700 1 --2- 172.21.15.42:0/3097903853 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5f3c03ac30 0x7f5f3c03d140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:38:58.800 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:58.786+0000 7f5f4a7fc700 1 -- 172.21.15.42:0/3097903853 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 8009+0+0 (secure 0 0 0) 0x7f5f4c091890 con 0x7f5f54135840 2019-12-05T05:38:58.801 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:58.786+0000 7f5f59cae700 1 --2- 172.21.15.42:0/3097903853 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5f3c03ac30 0x7f5f3c03d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:38:58.801 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:58.786+0000 7f5f59cae700 1 --2- 172.21.15.42:0/3097903853 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5f3c03ac30 0x7f5f3c03d140 crc :-1 s=READY pgs=521 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:38:58.801 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:58.790+0000 7f5f5b711700 1 -- 172.21.15.42:0/3097903853 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f5405d110 con 0x7f5f54135840 2019-12-05T05:38:58.803 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:58.794+0000 7f5f5a4af700 1 --2- 172.21.15.42:0/3097903853 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5f54135840 0x7f5f54137c80 secure :-1 s=THROTTLE_DONE pgs=1206 cs=0 l=1 rx=0x7f5f54121660 tx=0x7f5f4c010d80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:38:58.803 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:58.794+0000 7f5f4a7fc700 1 -- 172.21.15.42:0/3097903853 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f5f4c078b70 con 0x7f5f54135840 2019-12-05T05:38:58.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:58.966+0000 7f5f5b711700 1 -- 172.21.15.42:0/3097903853 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5f5405b8c0 con 0x7f5f54135840 2019-12-05T05:38:58.976 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:58.970+0000 7f5f5a4af700 1 --2- 172.21.15.42:0/3097903853 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5f54135840 0x7f5f54137c80 secure :-1 s=THROTTLE_DONE pgs=1206 cs=0 l=1 rx=0x7f5f54121660 tx=0x7f5f4c010d80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:38:58.976 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:58.970+0000 7f5f4a7fc700 1 -- 172.21.15.42:0/3097903853 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 148 v148) v1 ==== 95+0+2109 (secure 0 0 0) 0x7f5f4c079720 con 0x7f5f54135840 2019-12-05T05:38:58.976 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 148 2019-12-05T05:38:58.977 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:38:58.977 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":148,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7416,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/123054133","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":123054133},{"type":"v1","addr":"172.21.15.40:6833","nonce":123054133}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":148}],"filesystems":[{"mdsmap":{"epoch":147,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:37:13.880687+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5693},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5693":{"gid":5693,"name":"a","rank":0,"incarnation":144,"state":"up:active","state_seq":39,"addr":"172.21.15.42:6835/3838877789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3838877789},{"type":"v1","addr":"172.21.15.42:6835","nonce":3838877789}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:38:58.980 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:58.970+0000 7f5f5b711700 1 -- 172.21.15.42:0/3097903853 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5f3c03ac30 msgr2=0x7f5f3c03d140 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:38:58.980 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:58.970+0000 7f5f5b711700 1 --2- 172.21.15.42:0/3097903853 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5f3c03ac30 0x7f5f3c03d140 crc :-1 s=READY pgs=521 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:38:58.980 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:58.970+0000 7f5f5b711700 1 -- 172.21.15.42:0/3097903853 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5f54135840 msgr2=0x7f5f54137c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:38:58.980 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:58.970+0000 7f5f5b711700 1 --2- 172.21.15.42:0/3097903853 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5f54135840 0x7f5f54137c80 secure :-1 s=READY pgs=1206 cs=0 l=1 rx=0x7f5f54121660 tx=0x7f5f4c010d80).stop 2019-12-05T05:38:58.980 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:58.970+0000 7f5f5b711700 1 -- 172.21.15.42:0/3097903853 shutdown_connections 2019-12-05T05:38:58.981 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:58.970+0000 7f5f5b711700 1 --2- 172.21.15.42:0/3097903853 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5f3c03ac30 0x7f5f3c03d140 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:38:58.981 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:58.970+0000 7f5f5b711700 1 --2- 172.21.15.42:0/3097903853 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5f54135840 0x7f5f54137c80 unknown :-1 s=CLOSED pgs=1206 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:38:58.981 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:58.970+0000 7f5f5b711700 1 --2- 172.21.15.42:0/3097903853 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5f54124880 0x7f5f5411bdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:38:58.981 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:58.970+0000 7f5f5b711700 1 -- 172.21.15.42:0/3097903853 shutdown_connections 2019-12-05T05:38:58.981 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:58.970+0000 7f5f5b711700 1 -- 172.21.15.42:0/3097903853 wait complete. 2019-12-05T05:38:58.981 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:38:58.970+0000 7f5f5b711700 1 -- 172.21.15.42:0/3097903853 >> 172.21.15.42:0/3097903853 conn(0x7f5f5409f220 msgr2=0x7f5f540a6b90 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:38:58.989 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T05:39:00.990 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:39:01.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:01.082+0000 7f43f25b3700 1 Processor -- start 2019-12-05T05:39:01.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:01.082+0000 7f43f25b3700 1 -- start start 2019-12-05T05:39:01.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:01.090+0000 7f43f25b3700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f43ec0a7740 0x7f43ec08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:39:01.103 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:01.090+0000 7f43f25b3700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f43ec07e720 0x7f43ec07eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:39:01.103 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:01.090+0000 7f43f25b3700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43ec0a0710 con 0x7f43ec09f2e0 2019-12-05T05:39:01.103 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:01.090+0000 7f43f25b3700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43ec0a0b00 con 0x7f43ec07e720 2019-12-05T05:39:01.103 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:01.090+0000 7f43f25b3700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43ec08e5b0 con 0x7f43ec0a7740 2019-12-05T05:39:01.103 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:01.090+0000 7f43ebfff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f43ec07e720 0x7f43ec07eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:39:01.104 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:01.090+0000 7f43ebfff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f43ec07e720 0x7f43ec07eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:37974/0 (socket says 172.21.15.42:37974) 2019-12-05T05:39:01.104 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:01.090+0000 7f43ebfff700 1 -- 172.21.15.42:0/147061430 learned_addr learned my addr 172.21.15.42:0/147061430 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:39:01.116 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:01.090+0000 7f43f0b50700 1 --2- 172.21.15.42:0/147061430 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f43ec0a7740 0x7f43ec08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:39:01.116 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:01.090+0000 7f43ebfff700 1 -- 172.21.15.42:0/147061430 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f43ec0a7740 msgr2=0x7f43ec08df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:39:01.116 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:01.090+0000 7f43ebfff700 1 --2- 172.21.15.42:0/147061430 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f43ec0a7740 0x7f43ec08df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:39:01.116 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:01.090+0000 7f43ebfff700 1 -- 172.21.15.42:0/147061430 >> v1:172.21.15.40:6789/0 conn(0x7f43ec09f2e0 legacy=0x7f43ec09f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:39:01.119 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:01.090+0000 7f43ebfff700 1 -- 172.21.15.42:0/147061430 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43ec07f0a0 con 0x7f43ec07e720 2019-12-05T05:39:01.119 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:01.090+0000 7f43ebfff700 1 --2- 172.21.15.42:0/147061430 >> v2:172.21.15.42:3300/0 conn(0x7f43ec07e720 0x7f43ec07eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f43d4008790 tx=0x7f43e4001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:01.119 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:01.090+0000 7f43ebfff700 1 --2- 172.21.15.42:0/147061430 >> v2:172.21.15.42:3300/0 conn(0x7f43ec07e720 0x7f43ec07eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f43d4008790 tx=0x7f43e4001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:01.119 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:01.090+0000 7f43ebfff700 1 --2- 172.21.15.42:0/147061430 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f43ec07e720 0x7f43ec07eb50 secure :-1 s=READY pgs=1207 cs=0 l=1 rx=0x7f43d4008790 tx=0x7f43e4001150).ready entity=mon.1 client_cookie=ab29fcb5f676eb5f server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:39:01.121 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:01.090+0000 7f43ebfff700 1 --2- 172.21.15.42:0/147061430 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f43ec07e720 0x7f43ec07eb50 secure :-1 s=THROTTLE_DONE pgs=1207 cs=0 l=1 rx=0x7f43d4008790 tx=0x7f43e4001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:01.122 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:01.090+0000 7f43eb7fe700 1 -- 172.21.15.42:0/147061430 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f43d4008c90 con 0x7f43ec07e720 2019-12-05T05:39:01.122 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:01.090+0000 7f43ebfff700 1 --2- 172.21.15.42:0/147061430 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f43ec07e720 0x7f43ec07eb50 secure :-1 s=THROTTLE_DONE pgs=1207 cs=0 l=1 rx=0x7f43d4008790 tx=0x7f43e4001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:01.122 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:01.090+0000 7f43ebfff700 1 --2- 172.21.15.42:0/147061430 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f43ec07e720 0x7f43ec07eb50 secure :-1 s=THROTTLE_DONE pgs=1207 cs=0 l=1 rx=0x7f43d4008790 tx=0x7f43e4001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:01.122 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:01.090+0000 7f43eb7fe700 1 -- 172.21.15.42:0/147061430 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43d40088b0 con 0x7f43ec07e720 2019-12-05T05:39:01.122 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:01.090+0000 7f43eb7fe700 1 -- 172.21.15.42:0/147061430 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f43d4011a10 con 0x7f43ec07e720 2019-12-05T05:39:01.122 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:01.090+0000 7f43f25b3700 1 -- 172.21.15.42:0/147061430 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f43ec07e720 msgr2=0x7f43ec07eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:39:01.125 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:01.090+0000 7f43f25b3700 1 --2- 172.21.15.42:0/147061430 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f43ec07e720 0x7f43ec07eb50 secure :-1 s=READY pgs=1207 cs=0 l=1 rx=0x7f43d4008790 tx=0x7f43e4001150).stop 2019-12-05T05:39:01.125 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:01.090+0000 7f43f25b3700 1 -- 172.21.15.42:0/147061430 shutdown_connections 2019-12-05T05:39:01.126 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:01.090+0000 7f43f25b3700 1 --2- 172.21.15.42:0/147061430 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f43ec0a7740 0x7f43ec08df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:39:01.126 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:01.090+0000 7f43f25b3700 1 --2- 172.21.15.42:0/147061430 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f43ec07e720 0x7f43ec07eb50 unknown :-1 s=CLOSED pgs=1207 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:39:01.126 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:01.090+0000 7f43f25b3700 1 -- 172.21.15.42:0/147061430 shutdown_connections 2019-12-05T05:39:01.126 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:01.090+0000 7f43f25b3700 1 -- 172.21.15.42:0/147061430 wait complete. 2019-12-05T05:39:01.126 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:01.090+0000 7f43f25b3700 1 -- 172.21.15.42:0/147061430 >> 172.21.15.42:0/147061430 conn(0x7f43ec0aabe0 msgr2=0x7f43ec0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:39:01.126 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:01.090+0000 7f43f25b3700 1 Processor -- start 2019-12-05T05:39:01.126 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:01.090+0000 7f43f25b3700 1 -- start start 2019-12-05T05:39:01.127 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:01.090+0000 7f43f25b3700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f43ec11e700 0x7f43ec11aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:39:01.127 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:01.090+0000 7f43f25b3700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f43ec12b960 0x7f43ec12dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:39:01.127 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:01.090+0000 7f43f25b3700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43ec11a2c0 con 0x7f43ec0a6b80 2019-12-05T05:39:01.127 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:01.090+0000 7f43f25b3700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43ec11b650 con 0x7f43ec11e700 2019-12-05T05:39:01.127 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:01.090+0000 7f43f25b3700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43ec07dd30 con 0x7f43ec12b960 2019-12-05T05:39:01.127 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:01.094+0000 7f43f0b50700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f43ec11e700 0x7f43ec11aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:39:01.127 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:01.094+0000 7f43f0b50700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f43ec11e700 0x7f43ec11aad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:37978/0 (socket says 172.21.15.42:37978) 2019-12-05T05:39:01.128 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:01.094+0000 7f43f0b50700 1 -- 172.21.15.42:0/1425887036 learned_addr learned my addr 172.21.15.42:0/1425887036 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:39:01.128 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:01.094+0000 7f43f1351700 1 --2- 172.21.15.42:0/1425887036 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f43ec12b960 0x7f43ec12dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:39:01.128 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:01.094+0000 7f43f0b50700 1 -- 172.21.15.42:0/1425887036 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f43ec12b960 msgr2=0x7f43ec12dda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:39:01.129 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:01.094+0000 7f43f0b50700 1 --2- 172.21.15.42:0/1425887036 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f43ec12b960 0x7f43ec12dda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:39:01.129 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:01.094+0000 7f43f0b50700 1 -- 172.21.15.42:0/1425887036 >> v1:172.21.15.42:6789/0 conn(0x7f43ec0a6b80 legacy=0x7f43ec07afe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:39:01.129 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:01.094+0000 7f43e97fa700 1 -- 172.21.15.42:0/1425887036 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3112911830 0 0) 0x7f43ec11a2c0 con 0x7f43ec0a6b80 2019-12-05T05:39:01.129 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:01.094+0000 7f43f0b50700 1 -- 172.21.15.42:0/1425887036 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43ec11c560 con 0x7f43ec11e700 2019-12-05T05:39:01.129 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:01.094+0000 7f43f0b50700 1 --2- 172.21.15.42:0/1425887036 >> v2:172.21.15.42:3300/0 conn(0x7f43ec11e700 0x7f43ec11aad0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f43dc0033f0 tx=0x7f43dc010ee0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:01.129 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:01.094+0000 7f43f0b50700 1 --2- 172.21.15.42:0/1425887036 >> v2:172.21.15.42:3300/0 conn(0x7f43ec11e700 0x7f43ec11aad0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f43dc0033f0 tx=0x7f43dc010ee0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:01.131 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:01.094+0000 7f43f0b50700 1 --2- 172.21.15.42:0/1425887036 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f43ec11e700 0x7f43ec11aad0 secure :-1 s=READY pgs=1209 cs=0 l=1 rx=0x7f43dc0033f0 tx=0x7f43dc010ee0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:39:01.132 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:01.094+0000 7f43f0b50700 1 --2- 172.21.15.42:0/1425887036 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f43ec11e700 0x7f43ec11aad0 secure :-1 s=THROTTLE_DONE pgs=1209 cs=0 l=1 rx=0x7f43dc0033f0 tx=0x7f43dc010ee0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:01.132 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:01.094+0000 7f43e97fa700 1 -- 172.21.15.42:0/1425887036 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f43dc012b40 con 0x7f43ec11e700 2019-12-05T05:39:01.132 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:01.094+0000 7f43f0b50700 1 --2- 172.21.15.42:0/1425887036 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f43ec11e700 0x7f43ec11aad0 secure :-1 s=THROTTLE_DONE pgs=1209 cs=0 l=1 rx=0x7f43dc0033f0 tx=0x7f43dc010ee0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:01.132 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:01.094+0000 7f43f0b50700 1 --2- 172.21.15.42:0/1425887036 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f43ec11e700 0x7f43ec11aad0 secure :-1 s=THROTTLE_DONE pgs=1209 cs=0 l=1 rx=0x7f43dc0033f0 tx=0x7f43dc010ee0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:01.133 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:01.094+0000 7f43e97fa700 1 -- 172.21.15.42:0/1425887036 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43dc00e160 con 0x7f43ec11e700 2019-12-05T05:39:01.133 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:01.094+0000 7f43e97fa700 1 -- 172.21.15.42:0/1425887036 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f43dc005ab0 con 0x7f43ec11e700 2019-12-05T05:39:01.133 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:01.094+0000 7f43f25b3700 1 -- 172.21.15.42:0/1425887036 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43ec11c1e0 con 0x7f43ec11e700 2019-12-05T05:39:01.133 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:01.094+0000 7f43f25b3700 1 -- 172.21.15.42:0/1425887036 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43ec07b7e0 con 0x7f43ec11e700 2019-12-05T05:39:01.133 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:01.094+0000 7f43f0b50700 1 --2- 172.21.15.42:0/1425887036 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f43ec11e700 0x7f43ec11aad0 secure :-1 s=THROTTLE_DONE pgs=1209 cs=0 l=1 rx=0x7f43dc0033f0 tx=0x7f43dc010ee0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:01.135 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:01.094+0000 7f43e97fa700 1 -- 172.21.15.42:0/1425887036 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f43dc007180 con 0x7f43ec11e700 2019-12-05T05:39:01.135 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:01.094+0000 7f43f0b50700 1 --2- 172.21.15.42:0/1425887036 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f43ec11e700 0x7f43ec11aad0 secure :-1 s=THROTTLE_DONE pgs=1209 cs=0 l=1 rx=0x7f43dc0033f0 tx=0x7f43dc010ee0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:01.135 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:01.094+0000 7f43e97fa700 1 --2- 172.21.15.42:0/1425887036 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f43d003ad20 0x7f43d003d230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:39:01.135 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:01.094+0000 7f43e97fa700 1 -- 172.21.15.42:0/1425887036 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 8009+0+0 (secure 0 0 0) 0x7f43dc04af40 con 0x7f43ec11e700 2019-12-05T05:39:01.135 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:01.094+0000 7f43ebfff700 1 --2- 172.21.15.42:0/1425887036 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f43d003ad20 0x7f43d003d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:39:01.136 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:01.098+0000 7f43ebfff700 1 --2- 172.21.15.42:0/1425887036 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f43d003ad20 0x7f43d003d230 crc :-1 s=READY pgs=522 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:39:01.136 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:01.098+0000 7f43f25b3700 1 -- 172.21.15.42:0/1425887036 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43ec05d110 con 0x7f43ec11e700 2019-12-05T05:39:01.136 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:01.106+0000 7f43f0b50700 1 --2- 172.21.15.42:0/1425887036 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f43ec11e700 0x7f43ec11aad0 secure :-1 s=THROTTLE_DONE pgs=1209 cs=0 l=1 rx=0x7f43dc0033f0 tx=0x7f43dc010ee0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:01.136 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:01.106+0000 7f43e97fa700 1 -- 172.21.15.42:0/1425887036 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f43dc031fc0 con 0x7f43ec11e700 2019-12-05T05:39:01.272 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:01.266+0000 7f43f25b3700 1 -- 172.21.15.42:0/1425887036 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f43ec05b8c0 con 0x7f43ec11e700 2019-12-05T05:39:01.274 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:01.266+0000 7f43f0b50700 1 --2- 172.21.15.42:0/1425887036 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f43ec11e700 0x7f43ec11aad0 secure :-1 s=THROTTLE_DONE pgs=1209 cs=0 l=1 rx=0x7f43dc0033f0 tx=0x7f43dc010ee0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:01.274 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:01.266+0000 7f43e97fa700 1 -- 172.21.15.42:0/1425887036 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 148 v148) v1 ==== 95+0+2109 (secure 0 0 0) 0x7f43dc032220 con 0x7f43ec11e700 2019-12-05T05:39:01.274 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 148 2019-12-05T05:39:01.274 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:39:01.275 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":148,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7416,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/123054133","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":123054133},{"type":"v1","addr":"172.21.15.40:6833","nonce":123054133}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":148}],"filesystems":[{"mdsmap":{"epoch":147,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:37:13.880687+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5693},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5693":{"gid":5693,"name":"a","rank":0,"incarnation":144,"state":"up:active","state_seq":39,"addr":"172.21.15.42:6835/3838877789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3838877789},{"type":"v1","addr":"172.21.15.42:6835","nonce":3838877789}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:39:01.276 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:01.266+0000 7f43f25b3700 1 -- 172.21.15.42:0/1425887036 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f43d003ad20 msgr2=0x7f43d003d230 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:39:01.276 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:01.270+0000 7f43f25b3700 1 --2- 172.21.15.42:0/1425887036 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f43d003ad20 0x7f43d003d230 crc :-1 s=READY pgs=522 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:39:01.276 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:01.270+0000 7f43f25b3700 1 -- 172.21.15.42:0/1425887036 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f43ec11e700 msgr2=0x7f43ec11aad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:39:01.276 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:01.270+0000 7f43f25b3700 1 --2- 172.21.15.42:0/1425887036 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f43ec11e700 0x7f43ec11aad0 secure :-1 s=READY pgs=1209 cs=0 l=1 rx=0x7f43dc0033f0 tx=0x7f43dc010ee0).stop 2019-12-05T05:39:01.276 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:01.270+0000 7f43f25b3700 1 -- 172.21.15.42:0/1425887036 shutdown_connections 2019-12-05T05:39:01.276 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:01.270+0000 7f43f25b3700 1 --2- 172.21.15.42:0/1425887036 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f43d003ad20 0x7f43d003d230 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:39:01.277 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:01.270+0000 7f43f25b3700 1 --2- 172.21.15.42:0/1425887036 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f43ec12b960 0x7f43ec12dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:39:01.277 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:01.270+0000 7f43f25b3700 1 --2- 172.21.15.42:0/1425887036 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f43ec11e700 0x7f43ec11aad0 unknown :-1 s=CLOSED pgs=1209 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:39:01.277 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:01.270+0000 7f43f25b3700 1 -- 172.21.15.42:0/1425887036 shutdown_connections 2019-12-05T05:39:01.278 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:01.270+0000 7f43f25b3700 1 -- 172.21.15.42:0/1425887036 wait complete. 2019-12-05T05:39:01.278 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:01.270+0000 7f43f25b3700 1 -- 172.21.15.42:0/1425887036 >> 172.21.15.42:0/1425887036 conn(0x7f43ec0a7f90 msgr2=0x7f43ec09f580 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:39:01.286 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T05:39:03.287 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:39:03.396 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:03.390+0000 7fbae6422700 1 Processor -- start 2019-12-05T05:39:03.397 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:03.390+0000 7fbae6422700 1 -- start start 2019-12-05T05:39:03.397 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:03.390+0000 7fbae6422700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fbae00a7740 0x7fbae008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:39:03.397 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:03.390+0000 7fbae6422700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fbae009fae0 0x7fbae007eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:39:03.397 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:03.390+0000 7fbae6422700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbae00a0710 con 0x7fbae007e720 2019-12-05T05:39:03.398 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:03.390+0000 7fbae6422700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbae00a0b00 con 0x7fbae009fae0 2019-12-05T05:39:03.398 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:03.390+0000 7fbae6422700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbae008e5b0 con 0x7fbae00a7740 2019-12-05T05:39:03.398 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:03.390+0000 7fbae49bf700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fbae00a7740 0x7fbae008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:39:03.398 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:03.390+0000 7fbadffff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fbae007e720 0x7fbae009f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:38030/0 (socket says 172.21.15.42:38030) 2019-12-05T05:39:03.398 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:03.390+0000 7fbadffff700 1 -- 172.21.15.42:0/2297510105 learned_addr learned my addr 172.21.15.42:0/2297510105 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:39:03.399 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:03.390+0000 7fbae51c0700 1 --2- 172.21.15.42:0/2297510105 >> v2:172.21.15.40:3300/0 conn(0x7fbae009fae0 0x7fbae007eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:39:03.399 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:03.390+0000 7fbae49bf700 1 -- 172.21.15.42:0/2297510105 >> v2:172.21.15.40:3300/0 conn(0x7fbae009fae0 msgr2=0x7fbae007eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:39:03.399 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:03.390+0000 7fbae49bf700 1 --2- 172.21.15.42:0/2297510105 >> v2:172.21.15.40:3300/0 conn(0x7fbae009fae0 0x7fbae007eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:39:03.399 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:03.390+0000 7fbae49bf700 1 -- 172.21.15.42:0/2297510105 >> v1:172.21.15.42:6789/0 conn(0x7fbae007e720 legacy=0x7fbae009f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:39:03.399 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:03.390+0000 7fbae49bf700 1 -- 172.21.15.42:0/2297510105 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbae007f0a0 con 0x7fbae00a7740 2019-12-05T05:39:03.399 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:03.390+0000 7fbae49bf700 1 --2- 172.21.15.42:0/2297510105 >> v2:172.21.15.42:3300/0 conn(0x7fbae00a7740 0x7fbae008df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fbad0002f30 tx=0x7fbad000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:03.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:03.390+0000 7fbae49bf700 1 --2- 172.21.15.42:0/2297510105 >> v2:172.21.15.42:3300/0 conn(0x7fbae00a7740 0x7fbae008df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fbad0002f30 tx=0x7fbad000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:03.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:03.390+0000 7fbae49bf700 1 --2- 172.21.15.42:0/2297510105 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbae00a7740 0x7fbae008df80 secure :-1 s=READY pgs=1211 cs=0 l=1 rx=0x7fbad0002f30 tx=0x7fbad000fc70).ready entity=mon.1 client_cookie=70c26ee496f4b775 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:39:03.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:03.390+0000 7fbae49bf700 1 --2- 172.21.15.42:0/2297510105 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbae00a7740 0x7fbae008df80 secure :-1 s=THROTTLE_DONE pgs=1211 cs=0 l=1 rx=0x7fbad0002f30 tx=0x7fbad000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:03.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:03.390+0000 7fbadf7fe700 1 -- 172.21.15.42:0/2297510105 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbad0008bb0 con 0x7fbae00a7740 2019-12-05T05:39:03.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:03.390+0000 7fbae49bf700 1 --2- 172.21.15.42:0/2297510105 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbae00a7740 0x7fbae008df80 secure :-1 s=THROTTLE_DONE pgs=1211 cs=0 l=1 rx=0x7fbad0002f30 tx=0x7fbad000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:03.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:03.390+0000 7fbae49bf700 1 --2- 172.21.15.42:0/2297510105 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbae00a7740 0x7fbae008df80 secure :-1 s=THROTTLE_DONE pgs=1211 cs=0 l=1 rx=0x7fbad0002f30 tx=0x7fbad000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:03.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:03.390+0000 7fbadf7fe700 1 -- 172.21.15.42:0/2297510105 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbad00087d0 con 0x7fbae00a7740 2019-12-05T05:39:03.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:03.390+0000 7fbadf7fe700 1 -- 172.21.15.42:0/2297510105 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbad0011820 con 0x7fbae00a7740 2019-12-05T05:39:03.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:03.390+0000 7fbae6422700 1 -- 172.21.15.42:0/2297510105 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbae00a7740 msgr2=0x7fbae008df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:39:03.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:03.390+0000 7fbae6422700 1 --2- 172.21.15.42:0/2297510105 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbae00a7740 0x7fbae008df80 secure :-1 s=READY pgs=1211 cs=0 l=1 rx=0x7fbad0002f30 tx=0x7fbad000fc70).stop 2019-12-05T05:39:03.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:03.394+0000 7fbae6422700 1 -- 172.21.15.42:0/2297510105 shutdown_connections 2019-12-05T05:39:03.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:03.394+0000 7fbae6422700 1 --2- 172.21.15.42:0/2297510105 >> v2:172.21.15.40:3300/0 conn(0x7fbae009fae0 0x7fbae007eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:39:03.411 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:03.394+0000 7fbae6422700 1 --2- 172.21.15.42:0/2297510105 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbae00a7740 0x7fbae008df80 unknown :-1 s=CLOSED pgs=1211 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:39:03.411 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:03.394+0000 7fbae6422700 1 -- 172.21.15.42:0/2297510105 shutdown_connections 2019-12-05T05:39:03.419 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:03.394+0000 7fbae6422700 1 -- 172.21.15.42:0/2297510105 wait complete. 2019-12-05T05:39:03.419 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:03.394+0000 7fbae6422700 1 -- 172.21.15.42:0/2297510105 >> 172.21.15.42:0/2297510105 conn(0x7fbae00aabe0 msgr2=0x7fbae00a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:39:03.423 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:03.394+0000 7fbae6422700 1 Processor -- start 2019-12-05T05:39:03.423 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:03.394+0000 7fbae6422700 1 -- start start 2019-12-05T05:39:03.423 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:03.398+0000 7fbae6422700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fbae0117610 0x7fbae00a3060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:39:03.423 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:03.398+0000 7fbae6422700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbae01336e0 con 0x7fbae0129df0 2019-12-05T05:39:03.423 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:03.398+0000 7fbae6422700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbae00a35b0 con 0x7fbae0116500 2019-12-05T05:39:03.424 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:03.398+0000 7fbae6422700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbae0133910 con 0x7fbae0117610 2019-12-05T05:39:03.424 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:03.398+0000 7fbadffff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fbae0117610 0x7fbae00a3060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:39:03.424 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:03.398+0000 7fbae49bf700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fbae0116500 0x7fbae01182c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:38036/0 (socket says 172.21.15.42:38036) 2019-12-05T05:39:03.424 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:03.398+0000 7fbae49bf700 1 -- 172.21.15.42:0/1291148104 learned_addr learned my addr 172.21.15.42:0/1291148104 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:39:03.426 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:03.398+0000 7fbadffff700 1 -- 172.21.15.42:0/1291148104 >> v1:172.21.15.42:6789/0 conn(0x7fbae0116500 legacy=0x7fbae01182c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:39:03.426 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:03.398+0000 7fbadd7fa700 1 -- 172.21.15.42:0/1291148104 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3901947812 0 0) 0x7fbae00a35b0 con 0x7fbae0116500 2019-12-05T05:39:03.426 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:03.398+0000 7fbadffff700 1 -- 172.21.15.42:0/1291148104 >> v1:172.21.15.40:6789/0 conn(0x7fbae0129df0 legacy=0x7fbae00762f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:39:03.426 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:03.398+0000 7fbadffff700 1 -- 172.21.15.42:0/1291148104 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbae012a220 con 0x7fbae0117610 2019-12-05T05:39:03.427 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:03.398+0000 7fbadffff700 1 --2- 172.21.15.42:0/1291148104 >> v2:172.21.15.42:3300/0 conn(0x7fbae0117610 0x7fbae00a3060 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fbad8007e10 tx=0x7fbad8007720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:03.427 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:03.398+0000 7fbadffff700 1 --2- 172.21.15.42:0/1291148104 >> v2:172.21.15.42:3300/0 conn(0x7fbae0117610 0x7fbae00a3060 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fbad8007e10 tx=0x7fbad8007720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:03.427 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:03.398+0000 7fbadffff700 1 --2- 172.21.15.42:0/1291148104 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbae0117610 0x7fbae00a3060 secure :-1 s=READY pgs=1213 cs=0 l=1 rx=0x7fbad8007e10 tx=0x7fbad8007720).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:39:03.427 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:03.398+0000 7fbadffff700 1 --2- 172.21.15.42:0/1291148104 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbae0117610 0x7fbae00a3060 secure :-1 s=THROTTLE_DONE pgs=1213 cs=0 l=1 rx=0x7fbad8007e10 tx=0x7fbad8007720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:03.427 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:03.398+0000 7fbadd7fa700 1 -- 172.21.15.42:0/1291148104 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbad400d0c0 con 0x7fbae0117610 2019-12-05T05:39:03.427 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:03.398+0000 7fbadffff700 1 --2- 172.21.15.42:0/1291148104 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbae0117610 0x7fbae00a3060 secure :-1 s=THROTTLE_DONE pgs=1213 cs=0 l=1 rx=0x7fbad8007e10 tx=0x7fbad8007720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:03.428 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:03.398+0000 7fbadffff700 1 --2- 172.21.15.42:0/1291148104 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbae0117610 0x7fbae00a3060 secure :-1 s=THROTTLE_DONE pgs=1213 cs=0 l=1 rx=0x7fbad8007e10 tx=0x7fbad8007720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:03.428 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:03.398+0000 7fbadd7fa700 1 -- 172.21.15.42:0/1291148104 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbad40114e0 con 0x7fbae0117610 2019-12-05T05:39:03.428 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:03.398+0000 7fbae6422700 1 -- 172.21.15.42:0/1291148104 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbae012a4f0 con 0x7fbae0117610 2019-12-05T05:39:03.428 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:03.398+0000 7fbadd7fa700 1 -- 172.21.15.42:0/1291148104 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbad400d6a0 con 0x7fbae0117610 2019-12-05T05:39:03.428 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:03.398+0000 7fbae6422700 1 -- 172.21.15.42:0/1291148104 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbae0077b00 con 0x7fbae0117610 2019-12-05T05:39:03.429 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:03.398+0000 7fbadffff700 1 --2- 172.21.15.42:0/1291148104 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbae0117610 0x7fbae00a3060 secure :-1 s=THROTTLE_DONE pgs=1213 cs=0 l=1 rx=0x7fbad8007e10 tx=0x7fbad8007720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:03.429 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:03.402+0000 7fbadd7fa700 1 -- 172.21.15.42:0/1291148104 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fbad400d940 con 0x7fbae0117610 2019-12-05T05:39:03.429 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:03.402+0000 7fbadffff700 1 --2- 172.21.15.42:0/1291148104 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbae0117610 0x7fbae00a3060 secure :-1 s=THROTTLE_DONE pgs=1213 cs=0 l=1 rx=0x7fbad8007e10 tx=0x7fbad8007720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:03.430 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:03.402+0000 7fbadd7fa700 1 --2- 172.21.15.42:0/1291148104 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbac003ac50 0x7fbac003d160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:39:03.430 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:03.402+0000 7fbadd7fa700 1 -- 172.21.15.42:0/1291148104 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 8009+0+0 (secure 0 0 0) 0x7fbad404af00 con 0x7fbae0117610 2019-12-05T05:39:03.430 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:03.402+0000 7fbae49bf700 1 --2- 172.21.15.42:0/1291148104 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbac003ac50 0x7fbac003d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:39:03.430 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:03.402+0000 7fbae49bf700 1 --2- 172.21.15.42:0/1291148104 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbac003ac50 0x7fbac003d160 crc :-1 s=READY pgs=523 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:39:03.430 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:03.406+0000 7fbae6422700 1 -- 172.21.15.42:0/1291148104 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbae005d110 con 0x7fbae0117610 2019-12-05T05:39:03.430 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:03.410+0000 7fbadffff700 1 --2- 172.21.15.42:0/1291148104 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbae0117610 0x7fbae00a3060 secure :-1 s=THROTTLE_DONE pgs=1213 cs=0 l=1 rx=0x7fbad8007e10 tx=0x7fbad8007720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:03.430 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:03.410+0000 7fbadd7fa700 1 -- 172.21.15.42:0/1291148104 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fbad4031f60 con 0x7fbae0117610 2019-12-05T05:39:03.578 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:03.574+0000 7fbae6422700 1 -- 172.21.15.42:0/1291148104 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbae005b8c0 con 0x7fbae0117610 2019-12-05T05:39:03.579 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:03.574+0000 7fbadffff700 1 --2- 172.21.15.42:0/1291148104 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbae0117610 0x7fbae00a3060 secure :-1 s=THROTTLE_DONE pgs=1213 cs=0 l=1 rx=0x7fbad8007e10 tx=0x7fbad8007720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:03.579 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:03.574+0000 7fbadd7fa700 1 -- 172.21.15.42:0/1291148104 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 148 v148) v1 ==== 95+0+2109 (secure 0 0 0) 0x7fbad40321c0 con 0x7fbae0117610 2019-12-05T05:39:03.580 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 148 2019-12-05T05:39:03.580 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:39:03.580 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":148,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7416,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/123054133","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":123054133},{"type":"v1","addr":"172.21.15.40:6833","nonce":123054133}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":148}],"filesystems":[{"mdsmap":{"epoch":147,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:37:13.880687+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5693},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5693":{"gid":5693,"name":"a","rank":0,"incarnation":144,"state":"up:active","state_seq":39,"addr":"172.21.15.42:6835/3838877789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3838877789},{"type":"v1","addr":"172.21.15.42:6835","nonce":3838877789}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:39:03.581 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:03.578+0000 7fbae6422700 1 -- 172.21.15.42:0/1291148104 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbac003ac50 msgr2=0x7fbac003d160 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:39:03.581 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:03.578+0000 7fbae6422700 1 --2- 172.21.15.42:0/1291148104 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbac003ac50 0x7fbac003d160 crc :-1 s=READY pgs=523 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:39:03.581 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:03.578+0000 7fbae6422700 1 -- 172.21.15.42:0/1291148104 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbae0117610 msgr2=0x7fbae00a3060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:39:03.581 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:03.578+0000 7fbae6422700 1 --2- 172.21.15.42:0/1291148104 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbae0117610 0x7fbae00a3060 secure :-1 s=READY pgs=1213 cs=0 l=1 rx=0x7fbad8007e10 tx=0x7fbad8007720).stop 2019-12-05T05:39:03.583 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:03.578+0000 7fbae6422700 1 -- 172.21.15.42:0/1291148104 shutdown_connections 2019-12-05T05:39:03.583 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:03.578+0000 7fbae6422700 1 --2- 172.21.15.42:0/1291148104 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbac003ac50 0x7fbac003d160 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:39:03.583 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:03.578+0000 7fbae6422700 1 --2- 172.21.15.42:0/1291148104 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbae0117610 0x7fbae00a3060 unknown :-1 s=CLOSED pgs=1213 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:39:03.583 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:03.578+0000 7fbae6422700 1 -- 172.21.15.42:0/1291148104 shutdown_connections 2019-12-05T05:39:03.583 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:03.578+0000 7fbae6422700 1 -- 172.21.15.42:0/1291148104 wait complete. 2019-12-05T05:39:03.584 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:03.578+0000 7fbae6422700 1 -- 172.21.15.42:0/1291148104 >> 172.21.15.42:0/1291148104 conn(0x7fbae00a7f90 msgr2=0x7fbae00a6a60 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:39:03.591 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T05:39:05.596 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:39:05.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:05.690+0000 7f9906cad700 1 Processor -- start 2019-12-05T05:39:05.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:05.690+0000 7f9906cad700 1 -- start start 2019-12-05T05:39:05.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:05.690+0000 7f9906cad700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f99000a7740 0x7f990008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:39:05.711 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:05.690+0000 7f9906cad700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99000a0710 con 0x7f990009fae0 2019-12-05T05:39:05.711 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:05.690+0000 7f9906cad700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99000a9ba0 con 0x7f990007e720 2019-12-05T05:39:05.711 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:05.690+0000 7f9906cad700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99000a0b00 con 0x7f99000a7740 2019-12-05T05:39:05.711 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:05.690+0000 7f9904a49700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f990007e720 0x7f990009f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:38044/0 (socket says 172.21.15.42:38044) 2019-12-05T05:39:05.712 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:05.690+0000 7f9904a49700 1 -- 172.21.15.42:0/457019892 learned_addr learned my addr 172.21.15.42:0/457019892 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:39:05.712 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:05.690+0000 7f990524a700 1 --2- 172.21.15.42:0/457019892 >> v2:172.21.15.40:3300/0 conn(0x7f99000a7740 0x7f990008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:39:05.712 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:05.690+0000 7f98f3fff700 1 -- 172.21.15.42:0/457019892 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2904251335 0 0) 0x7f99000a9ba0 con 0x7f990007e720 2019-12-05T05:39:05.712 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:05.690+0000 7f98f3fff700 1 -- 172.21.15.42:0/457019892 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98ec0030b0 con 0x7f990007e720 2019-12-05T05:39:05.713 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:05.690+0000 7f990524a700 1 -- 172.21.15.42:0/457019892 >> v1:172.21.15.42:6789/0 conn(0x7f990007e720 legacy=0x7f990009f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:39:05.713 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:05.690+0000 7f990524a700 1 -- 172.21.15.42:0/457019892 >> v1:172.21.15.40:6789/0 conn(0x7f990009fae0 legacy=0x7f990014efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:39:05.713 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:05.690+0000 7f990524a700 1 -- 172.21.15.42:0/457019892 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f990007eb50 con 0x7f99000a7740 2019-12-05T05:39:05.713 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:05.690+0000 7f990524a700 1 --2- 172.21.15.42:0/457019892 >> v2:172.21.15.40:3300/0 conn(0x7f99000a7740 0x7f990008df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f98f4001990 tx=0x7f98f40014a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:05.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:05.690+0000 7f990524a700 1 --2- 172.21.15.42:0/457019892 >> v2:172.21.15.40:3300/0 conn(0x7f99000a7740 0x7f990008df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f98f4001990 tx=0x7f98f40014a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:05.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:05.690+0000 7f990524a700 1 --2- 172.21.15.42:0/457019892 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f99000a7740 0x7f990008df80 secure :-1 s=READY pgs=943 cs=0 l=1 rx=0x7f98f4001990 tx=0x7f98f40014a0).ready entity=mon.0 client_cookie=878078fc690e8506 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:39:05.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:05.690+0000 7f990524a700 1 --2- 172.21.15.42:0/457019892 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f99000a7740 0x7f990008df80 secure :-1 s=THROTTLE_DONE pgs=943 cs=0 l=1 rx=0x7f98f4001990 tx=0x7f98f40014a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:05.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:05.690+0000 7f990524a700 1 --2- 172.21.15.42:0/457019892 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f99000a7740 0x7f990008df80 secure :-1 s=THROTTLE_DONE pgs=943 cs=0 l=1 rx=0x7f98f4001990 tx=0x7f98f40014a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:05.715 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:05.690+0000 7f98f3fff700 1 -- 172.21.15.42:0/457019892 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f98f4001750 con 0x7f99000a7740 2019-12-05T05:39:05.715 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:05.690+0000 7f990524a700 1 --2- 172.21.15.42:0/457019892 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f99000a7740 0x7f990008df80 secure :-1 s=THROTTLE_DONE pgs=943 cs=0 l=1 rx=0x7f98f4001990 tx=0x7f98f40014a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:05.715 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:05.690+0000 7f98f3fff700 1 -- 172.21.15.42:0/457019892 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98f4001560 con 0x7f99000a7740 2019-12-05T05:39:05.715 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:05.690+0000 7f98f3fff700 1 -- 172.21.15.42:0/457019892 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f98fc0118d0 con 0x7f99000a7740 2019-12-05T05:39:05.716 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:05.690+0000 7f9906cad700 1 -- 172.21.15.42:0/457019892 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f99000a7740 msgr2=0x7f990008df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:39:05.716 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:05.690+0000 7f9906cad700 1 --2- 172.21.15.42:0/457019892 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f99000a7740 0x7f990008df80 secure :-1 s=READY pgs=943 cs=0 l=1 rx=0x7f98f4001990 tx=0x7f98f40014a0).stop 2019-12-05T05:39:05.716 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:05.694+0000 7f9906cad700 1 -- 172.21.15.42:0/457019892 shutdown_connections 2019-12-05T05:39:05.716 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:05.694+0000 7f9906cad700 1 --2- 172.21.15.42:0/457019892 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f99000a7740 0x7f990008df80 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:39:05.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:05.694+0000 7f9906cad700 1 -- 172.21.15.42:0/457019892 shutdown_connections 2019-12-05T05:39:05.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:05.694+0000 7f9906cad700 1 -- 172.21.15.42:0/457019892 wait complete. 2019-12-05T05:39:05.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:05.694+0000 7f9906cad700 1 -- 172.21.15.42:0/457019892 >> 172.21.15.42:0/457019892 conn(0x7f99000aabe0 msgr2=0x7f99000a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:39:05.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:05.694+0000 7f9906cad700 1 Processor -- start 2019-12-05T05:39:05.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:05.694+0000 7f9906cad700 1 -- start start 2019-12-05T05:39:05.718 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:05.694+0000 7f9906cad700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f9900121f10 0x7f99000a1e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:39:05.718 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:05.694+0000 7f9906cad700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99000a2360 con 0x7f99000a40c0 2019-12-05T05:39:05.718 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:05.694+0000 7f9906cad700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9900122340 con 0x7f99000a3c90 2019-12-05T05:39:05.718 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:05.694+0000 7f9906cad700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99000a2f10 con 0x7f9900121f10 2019-12-05T05:39:05.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:05.698+0000 7f990524a700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f9900121f10 0x7f99000a1e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:39:05.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:05.698+0000 7f9904a49700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f99000a3c90 0x7f990007dac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:38052/0 (socket says 172.21.15.42:38052) 2019-12-05T05:39:05.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:05.698+0000 7f9904a49700 1 -- 172.21.15.42:0/4174174337 learned_addr learned my addr 172.21.15.42:0/4174174337 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:39:05.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:05.698+0000 7f98f1ffb700 1 -- 172.21.15.42:0/4174174337 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3092837090 0 0) 0x7f99000a2360 con 0x7f99000a40c0 2019-12-05T05:39:05.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:05.698+0000 7f98f1ffb700 1 -- 172.21.15.42:0/4174174337 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98e00030b0 con 0x7f99000a40c0 2019-12-05T05:39:05.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:05.698+0000 7f990524a700 1 -- 172.21.15.42:0/4174174337 >> v1:172.21.15.42:6789/0 conn(0x7f99000a3c90 legacy=0x7f990007dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:39:05.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:05.698+0000 7f98f1ffb700 1 -- 172.21.15.42:0/4174174337 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4279266612 0 0) 0x7f9900122340 con 0x7f99000a3c90 2019-12-05T05:39:05.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:05.698+0000 7f990524a700 1 -- 172.21.15.42:0/4174174337 >> v1:172.21.15.40:6789/0 conn(0x7f99000a40c0 legacy=0x7f990012c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:39:05.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:05.698+0000 7f990524a700 1 -- 172.21.15.42:0/4174174337 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99001147b0 con 0x7f9900121f10 2019-12-05T05:39:05.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:05.698+0000 7f990524a700 1 --2- 172.21.15.42:0/4174174337 >> v2:172.21.15.42:3300/0 conn(0x7f9900121f10 0x7f99000a1e10 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f98f4001990 tx=0x7f98fc011990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:05.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:05.698+0000 7f990524a700 1 --2- 172.21.15.42:0/4174174337 >> v2:172.21.15.42:3300/0 conn(0x7f9900121f10 0x7f99000a1e10 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f98f4001990 tx=0x7f98fc011990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:05.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:05.698+0000 7f990524a700 1 --2- 172.21.15.42:0/4174174337 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9900121f10 0x7f99000a1e10 secure :-1 s=READY pgs=1216 cs=0 l=1 rx=0x7f98f4001990 tx=0x7f98fc011990).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:39:05.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:05.698+0000 7f990524a700 1 --2- 172.21.15.42:0/4174174337 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9900121f10 0x7f99000a1e10 secure :-1 s=THROTTLE_DONE pgs=1216 cs=0 l=1 rx=0x7f98f4001990 tx=0x7f98fc011990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:05.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:05.698+0000 7f98f1ffb700 1 -- 172.21.15.42:0/4174174337 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f98fc00f780 con 0x7f9900121f10 2019-12-05T05:39:05.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:05.698+0000 7f990524a700 1 --2- 172.21.15.42:0/4174174337 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9900121f10 0x7f99000a1e10 secure :-1 s=THROTTLE_DONE pgs=1216 cs=0 l=1 rx=0x7f98f4001990 tx=0x7f98fc011990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:05.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:05.698+0000 7f990524a700 1 --2- 172.21.15.42:0/4174174337 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9900121f10 0x7f99000a1e10 secure :-1 s=THROTTLE_DONE pgs=1216 cs=0 l=1 rx=0x7f98f4001990 tx=0x7f98fc011990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:05.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:05.698+0000 7f98f1ffb700 1 -- 172.21.15.42:0/4174174337 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98fc0114f0 con 0x7f9900121f10 2019-12-05T05:39:05.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:05.698+0000 7f98f1ffb700 1 -- 172.21.15.42:0/4174174337 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f98fc005ab0 con 0x7f9900121f10 2019-12-05T05:39:05.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:05.698+0000 7f9906cad700 1 -- 172.21.15.42:0/4174174337 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f990008eb70 con 0x7f9900121f10 2019-12-05T05:39:05.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:05.698+0000 7f9906cad700 1 -- 172.21.15.42:0/4174174337 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f990008f070 con 0x7f9900121f10 2019-12-05T05:39:05.724 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:05.698+0000 7f990524a700 1 --2- 172.21.15.42:0/4174174337 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9900121f10 0x7f99000a1e10 secure :-1 s=THROTTLE_DONE pgs=1216 cs=0 l=1 rx=0x7f98f4001990 tx=0x7f98fc011990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:05.724 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:05.698+0000 7f98f1ffb700 1 -- 172.21.15.42:0/4174174337 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f98fc007150 con 0x7f9900121f10 2019-12-05T05:39:05.724 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:05.698+0000 7f990524a700 1 --2- 172.21.15.42:0/4174174337 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9900121f10 0x7f99000a1e10 secure :-1 s=THROTTLE_DONE pgs=1216 cs=0 l=1 rx=0x7f98f4001990 tx=0x7f98fc011990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:05.724 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:05.702+0000 7f98f1ffb700 1 --2- 172.21.15.42:0/4174174337 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f98e003b1b0 0x7f98e003d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:39:05.724 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:05.702+0000 7f98f1ffb700 1 -- 172.21.15.42:0/4174174337 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 8009+0+0 (secure 0 0 0) 0x7f98fc04b320 con 0x7f9900121f10 2019-12-05T05:39:05.725 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:05.702+0000 7f9904a49700 1 --2- 172.21.15.42:0/4174174337 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f98e003b1b0 0x7f98e003d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:39:05.725 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:05.702+0000 7f9904a49700 1 --2- 172.21.15.42:0/4174174337 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f98e003b1b0 0x7f98e003d6c0 crc :-1 s=READY pgs=524 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:39:05.725 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:05.706+0000 7f9906cad700 1 -- 172.21.15.42:0/4174174337 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f990005d110 con 0x7f9900121f10 2019-12-05T05:39:05.726 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:05.714+0000 7f990524a700 1 --2- 172.21.15.42:0/4174174337 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9900121f10 0x7f99000a1e10 secure :-1 s=THROTTLE_DONE pgs=1216 cs=0 l=1 rx=0x7f98f4001990 tx=0x7f98fc011990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:05.726 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:05.714+0000 7f98f1ffb700 1 -- 172.21.15.42:0/4174174337 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f98fc005d50 con 0x7f9900121f10 2019-12-05T05:39:05.885 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:05.882+0000 7f9906cad700 1 -- 172.21.15.42:0/4174174337 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f990005b8c0 con 0x7f9900121f10 2019-12-05T05:39:05.886 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:05.882+0000 7f990524a700 1 --2- 172.21.15.42:0/4174174337 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9900121f10 0x7f99000a1e10 secure :-1 s=THROTTLE_DONE pgs=1216 cs=0 l=1 rx=0x7f98f4001990 tx=0x7f98fc011990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:05.886 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:05.882+0000 7f98f1ffb700 1 -- 172.21.15.42:0/4174174337 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 148 v148) v1 ==== 95+0+2109 (secure 0 0 0) 0x7f98fc02ffc0 con 0x7f9900121f10 2019-12-05T05:39:05.888 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 148 2019-12-05T05:39:05.888 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:39:05.888 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":148,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7416,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/123054133","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":123054133},{"type":"v1","addr":"172.21.15.40:6833","nonce":123054133}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":148}],"filesystems":[{"mdsmap":{"epoch":147,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:37:13.880687+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5693},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5693":{"gid":5693,"name":"a","rank":0,"incarnation":144,"state":"up:active","state_seq":39,"addr":"172.21.15.42:6835/3838877789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3838877789},{"type":"v1","addr":"172.21.15.42:6835","nonce":3838877789}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:39:05.891 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:05.886+0000 7f9906cad700 1 -- 172.21.15.42:0/4174174337 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f98e003b1b0 msgr2=0x7f98e003d6c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:39:05.891 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:05.886+0000 7f9906cad700 1 --2- 172.21.15.42:0/4174174337 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f98e003b1b0 0x7f98e003d6c0 crc :-1 s=READY pgs=524 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:39:05.891 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:05.886+0000 7f9906cad700 1 -- 172.21.15.42:0/4174174337 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9900121f10 msgr2=0x7f99000a1e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:39:05.891 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:05.886+0000 7f9906cad700 1 --2- 172.21.15.42:0/4174174337 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9900121f10 0x7f99000a1e10 secure :-1 s=READY pgs=1216 cs=0 l=1 rx=0x7f98f4001990 tx=0x7f98fc011990).stop 2019-12-05T05:39:05.891 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:05.886+0000 7f9906cad700 1 -- 172.21.15.42:0/4174174337 shutdown_connections 2019-12-05T05:39:05.892 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:05.886+0000 7f9906cad700 1 --2- 172.21.15.42:0/4174174337 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f98e003b1b0 0x7f98e003d6c0 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:39:05.892 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:05.886+0000 7f9906cad700 1 --2- 172.21.15.42:0/4174174337 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9900121f10 0x7f99000a1e10 unknown :-1 s=CLOSED pgs=1216 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:39:05.892 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:05.886+0000 7f9906cad700 1 -- 172.21.15.42:0/4174174337 shutdown_connections 2019-12-05T05:39:05.892 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:05.886+0000 7f9906cad700 1 -- 172.21.15.42:0/4174174337 wait complete. 2019-12-05T05:39:05.892 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:05.886+0000 7f9906cad700 1 -- 172.21.15.42:0/4174174337 >> 172.21.15.42:0/4174174337 conn(0x7f99000a82a0 msgr2=0x7f9900059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:39:05.901 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T05:39:07.902 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:39:08.012 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:08.002+0000 7f524b22e700 1 Processor -- start 2019-12-05T05:39:08.012 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:08.002+0000 7f524b22e700 1 -- start start 2019-12-05T05:39:08.012 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:08.006+0000 7f524b22e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f524408df80 0x7f524409f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:39:08.012 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:08.006+0000 7f524b22e700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f524409f830 0x7f524414efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:39:08.012 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:08.006+0000 7f524b22e700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f524409fc60 con 0x7f52440a7740 2019-12-05T05:39:08.013 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:08.006+0000 7f524b22e700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52440a0b00 con 0x7f524408df80 2019-12-05T05:39:08.013 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:08.006+0000 7f524b22e700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f524408e5b0 con 0x7f524409f830 2019-12-05T05:39:08.013 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:08.006+0000 7f5248fca700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f524408df80 0x7f524409f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:39:08.013 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:08.006+0000 7f5248fca700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f524408df80 0x7f524409f2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:38022/0 (socket says 172.21.15.42:38022) 2019-12-05T05:39:08.013 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:08.006+0000 7f5248fca700 1 -- 172.21.15.42:0/1290222156 learned_addr learned my addr 172.21.15.42:0/1290222156 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:39:08.013 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:08.006+0000 7f5249fcc700 1 --2- 172.21.15.42:0/1290222156 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f524409f830 0x7f524414efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:39:08.014 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:08.006+0000 7f5248fca700 1 -- 172.21.15.42:0/1290222156 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f524409f830 msgr2=0x7f524414efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:39:08.014 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:08.006+0000 7f5248fca700 1 --2- 172.21.15.42:0/1290222156 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f524409f830 0x7f524414efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:39:08.014 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:08.006+0000 7f5248fca700 1 -- 172.21.15.42:0/1290222156 >> v1:172.21.15.40:6789/0 conn(0x7f52440a7740 legacy=0x7f524407e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:39:08.014 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:08.006+0000 7f5248fca700 1 -- 172.21.15.42:0/1290222156 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52440a0710 con 0x7f524408df80 2019-12-05T05:39:08.014 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:08.006+0000 7f5248fca700 1 --2- 172.21.15.42:0/1290222156 >> v2:172.21.15.42:3300/0 conn(0x7f524408df80 0x7f524409f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f5238008790 tx=0x7f523800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:08.014 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:08.006+0000 7f5248fca700 1 --2- 172.21.15.42:0/1290222156 >> v2:172.21.15.42:3300/0 conn(0x7f524408df80 0x7f524409f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f5238008790 tx=0x7f523800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:08.015 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:08.006+0000 7f5248fca700 1 --2- 172.21.15.42:0/1290222156 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f524408df80 0x7f524409f2e0 secure :-1 s=READY pgs=1217 cs=0 l=1 rx=0x7f5238008790 tx=0x7f523800fd30).ready entity=mon.1 client_cookie=50529ec0ff1fffc7 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:39:08.015 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:08.006+0000 7f5248fca700 1 --2- 172.21.15.42:0/1290222156 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f524408df80 0x7f524409f2e0 secure :-1 s=THROTTLE_DONE pgs=1217 cs=0 l=1 rx=0x7f5238008790 tx=0x7f523800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:08.015 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:08.006+0000 7f5237fff700 1 -- 172.21.15.42:0/1290222156 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5238008c90 con 0x7f524408df80 2019-12-05T05:39:08.015 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:08.006+0000 7f5248fca700 1 --2- 172.21.15.42:0/1290222156 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f524408df80 0x7f524409f2e0 secure :-1 s=THROTTLE_DONE pgs=1217 cs=0 l=1 rx=0x7f5238008790 tx=0x7f523800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:08.015 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:08.006+0000 7f5248fca700 1 --2- 172.21.15.42:0/1290222156 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f524408df80 0x7f524409f2e0 secure :-1 s=THROTTLE_DONE pgs=1217 cs=0 l=1 rx=0x7f5238008790 tx=0x7f523800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:08.015 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:08.006+0000 7f5237fff700 1 -- 172.21.15.42:0/1290222156 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52380088b0 con 0x7f524408df80 2019-12-05T05:39:08.015 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:08.006+0000 7f5237fff700 1 -- 172.21.15.42:0/1290222156 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5238011b10 con 0x7f524408df80 2019-12-05T05:39:08.016 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:08.006+0000 7f524b22e700 1 -- 172.21.15.42:0/1290222156 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f524408df80 msgr2=0x7f524409f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:39:08.016 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:08.006+0000 7f524b22e700 1 --2- 172.21.15.42:0/1290222156 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f524408df80 0x7f524409f2e0 secure :-1 s=READY pgs=1217 cs=0 l=1 rx=0x7f5238008790 tx=0x7f523800fd30).stop 2019-12-05T05:39:08.016 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:08.006+0000 7f524b22e700 1 -- 172.21.15.42:0/1290222156 shutdown_connections 2019-12-05T05:39:08.016 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:08.006+0000 7f524b22e700 1 --2- 172.21.15.42:0/1290222156 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f524409f830 0x7f524414efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:39:08.016 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:08.006+0000 7f524b22e700 1 --2- 172.21.15.42:0/1290222156 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f524408df80 0x7f524409f2e0 unknown :-1 s=CLOSED pgs=1217 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:39:08.016 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:08.006+0000 7f524b22e700 1 -- 172.21.15.42:0/1290222156 shutdown_connections 2019-12-05T05:39:08.016 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:08.006+0000 7f524b22e700 1 -- 172.21.15.42:0/1290222156 wait complete. 2019-12-05T05:39:08.017 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:08.006+0000 7f524b22e700 1 -- 172.21.15.42:0/1290222156 >> 172.21.15.42:0/1290222156 conn(0x7f52440aabe0 msgr2=0x7f52440a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:39:08.020 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:08.010+0000 7f524b22e700 1 Processor -- start 2019-12-05T05:39:08.020 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:08.010+0000 7f524b22e700 1 -- start start 2019-12-05T05:39:08.020 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:08.010+0000 7f524b22e700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f52440a6ea0 0x7f5244118bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:39:08.020 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:08.010+0000 7f524b22e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f52441282b0 0x7f524414e810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:39:08.021 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:08.010+0000 7f524b22e700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52440a3720 con 0x7f524414e3e0 2019-12-05T05:39:08.021 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:08.010+0000 7f524b22e700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5244119100 con 0x7f52440a6ea0 2019-12-05T05:39:08.021 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:08.010+0000 7f524b22e700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52440a3950 con 0x7f52441282b0 2019-12-05T05:39:08.021 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:08.010+0000 7f5249fcc700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f52441282b0 0x7f524414e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:39:08.021 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:08.010+0000 7f5249fcc700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f52441282b0 0x7f524414e810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:38030/0 (socket says 172.21.15.42:38030) 2019-12-05T05:39:08.021 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:08.010+0000 7f5249fcc700 1 -- 172.21.15.42:0/1236718398 learned_addr learned my addr 172.21.15.42:0/1236718398 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:39:08.022 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:08.010+0000 7f52497cb700 1 --2- 172.21.15.42:0/1236718398 >> v2:172.21.15.40:3300/0 conn(0x7f52440a6ea0 0x7f5244118bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:39:08.022 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:08.010+0000 7f5249fcc700 1 -- 172.21.15.42:0/1236718398 >> v2:172.21.15.40:3300/0 conn(0x7f52440a6ea0 msgr2=0x7f5244118bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:39:08.022 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:08.010+0000 7f5249fcc700 1 --2- 172.21.15.42:0/1236718398 >> v2:172.21.15.40:3300/0 conn(0x7f52440a6ea0 0x7f5244118bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:39:08.022 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:08.010+0000 7f5235ffb700 1 -- 172.21.15.42:0/1236718398 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2836613385 0 0) 0x7f52440a3720 con 0x7f524414e3e0 2019-12-05T05:39:08.022 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:08.010+0000 7f5249fcc700 1 -- 172.21.15.42:0/1236718398 >> v1:172.21.15.40:6789/0 conn(0x7f524414e3e0 legacy=0x7f5244127ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:39:08.022 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:08.010+0000 7f5249fcc700 1 -- 172.21.15.42:0/1236718398 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f524412fd90 con 0x7f52441282b0 2019-12-05T05:39:08.022 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:08.010+0000 7f5249fcc700 1 --2- 172.21.15.42:0/1236718398 >> v2:172.21.15.42:3300/0 conn(0x7f52441282b0 0x7f524414e810 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f523c002f30 tx=0x7f523c0108c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:08.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:08.010+0000 7f5249fcc700 1 --2- 172.21.15.42:0/1236718398 >> v2:172.21.15.42:3300/0 conn(0x7f52441282b0 0x7f524414e810 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f523c002f30 tx=0x7f523c0108c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:08.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:08.010+0000 7f5249fcc700 1 --2- 172.21.15.42:0/1236718398 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f52441282b0 0x7f524414e810 secure :-1 s=READY pgs=1218 cs=0 l=1 rx=0x7f523c002f30 tx=0x7f523c0108c0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:39:08.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:08.010+0000 7f5249fcc700 1 --2- 172.21.15.42:0/1236718398 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f52441282b0 0x7f524414e810 secure :-1 s=THROTTLE_DONE pgs=1218 cs=0 l=1 rx=0x7f523c002f30 tx=0x7f523c0108c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:08.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:08.010+0000 7f5235ffb700 1 -- 172.21.15.42:0/1236718398 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f523c005930 con 0x7f52441282b0 2019-12-05T05:39:08.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:08.010+0000 7f5249fcc700 1 --2- 172.21.15.42:0/1236718398 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f52441282b0 0x7f524414e810 secure :-1 s=THROTTLE_DONE pgs=1218 cs=0 l=1 rx=0x7f523c002f30 tx=0x7f523c0108c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:08.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:08.010+0000 7f5249fcc700 1 --2- 172.21.15.42:0/1236718398 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f52441282b0 0x7f524414e810 secure :-1 s=THROTTLE_DONE pgs=1218 cs=0 l=1 rx=0x7f523c002f30 tx=0x7f523c0108c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:08.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:08.010+0000 7f5235ffb700 1 -- 172.21.15.42:0/1236718398 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f523c007160 con 0x7f52441282b0 2019-12-05T05:39:08.024 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:08.010+0000 7f524b22e700 1 -- 172.21.15.42:0/1236718398 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f524412ed80 con 0x7f52441282b0 2019-12-05T05:39:08.024 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:08.010+0000 7f5235ffb700 1 -- 172.21.15.42:0/1236718398 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f523c0060c0 con 0x7f52441282b0 2019-12-05T05:39:08.024 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:08.014+0000 7f524b22e700 1 -- 172.21.15.42:0/1236718398 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f524412f2b0 con 0x7f52441282b0 2019-12-05T05:39:08.024 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:08.014+0000 7f5249fcc700 1 --2- 172.21.15.42:0/1236718398 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f52441282b0 0x7f524414e810 secure :-1 s=THROTTLE_DONE pgs=1218 cs=0 l=1 rx=0x7f523c002f30 tx=0x7f523c0108c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:08.024 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:08.014+0000 7f5235ffb700 1 -- 172.21.15.42:0/1236718398 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f523c007ae0 con 0x7f52441282b0 2019-12-05T05:39:08.024 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:08.014+0000 7f5249fcc700 1 --2- 172.21.15.42:0/1236718398 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f52441282b0 0x7f524414e810 secure :-1 s=THROTTLE_DONE pgs=1218 cs=0 l=1 rx=0x7f523c002f30 tx=0x7f523c0108c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:08.024 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:08.014+0000 7f5235ffb700 1 --2- 172.21.15.42:0/1236718398 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f522403ad50 0x7f522403d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:39:08.025 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:08.014+0000 7f5235ffb700 1 -- 172.21.15.42:0/1236718398 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 8009+0+0 (secure 0 0 0) 0x7f523c04ab20 con 0x7f52441282b0 2019-12-05T05:39:08.025 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:08.014+0000 7f52497cb700 1 --2- 172.21.15.42:0/1236718398 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f522403ad50 0x7f522403d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:39:08.025 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:08.014+0000 7f52497cb700 1 --2- 172.21.15.42:0/1236718398 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f522403ad50 0x7f522403d260 crc :-1 s=READY pgs=525 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:39:08.025 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:08.018+0000 7f524b22e700 1 -- 172.21.15.42:0/1236718398 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f524405d110 con 0x7f52441282b0 2019-12-05T05:39:08.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:08.026+0000 7f5249fcc700 1 --2- 172.21.15.42:0/1236718398 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f52441282b0 0x7f524414e810 secure :-1 s=THROTTLE_DONE pgs=1218 cs=0 l=1 rx=0x7f523c002f30 tx=0x7f523c0108c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:08.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:08.026+0000 7f5235ffb700 1 -- 172.21.15.42:0/1236718398 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f523c031ba0 con 0x7f52441282b0 2019-12-05T05:39:08.214 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:08.198+0000 7f524b22e700 1 -- 172.21.15.42:0/1236718398 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f524405b8c0 con 0x7f52441282b0 2019-12-05T05:39:08.214 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:08.198+0000 7f5249fcc700 1 --2- 172.21.15.42:0/1236718398 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f52441282b0 0x7f524414e810 secure :-1 s=THROTTLE_DONE pgs=1218 cs=0 l=1 rx=0x7f523c002f30 tx=0x7f523c0108c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:08.214 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:08.198+0000 7f5235ffb700 1 -- 172.21.15.42:0/1236718398 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 148 v148) v1 ==== 95+0+2109 (secure 0 0 0) 0x7f523c031e00 con 0x7f52441282b0 2019-12-05T05:39:08.214 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 148 2019-12-05T05:39:08.215 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:08.202+0000 7f524b22e700 1 -- 172.21.15.42:0/1236718398 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f522403ad50 msgr2=0x7f522403d260 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:39:08.215 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:08.202+0000 7f524b22e700 1 --2- 172.21.15.42:0/1236718398 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f522403ad50 0x7f522403d260 crc :-1 s=READY pgs=525 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:39:08.215 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:08.202+0000 7f524b22e700 1 -- 172.21.15.42:0/1236718398 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f52441282b0 msgr2=0x7f524414e810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:39:08.215 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:08.202+0000 7f524b22e700 1 --2- 172.21.15.42:0/1236718398 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f52441282b0 0x7f524414e810 secure :-1 s=READY pgs=1218 cs=0 l=1 rx=0x7f523c002f30 tx=0x7f523c0108c0).stop 2019-12-05T05:39:08.215 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:08.202+0000 7f524b22e700 1 -- 172.21.15.42:0/1236718398 shutdown_connections 2019-12-05T05:39:08.215 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:08.202+0000 7f524b22e700 1 --2- 172.21.15.42:0/1236718398 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f522403ad50 0x7f522403d260 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:39:08.215 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:08.202+0000 7f524b22e700 1 --2- 172.21.15.42:0/1236718398 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f52441282b0 0x7f524414e810 unknown :-1 s=CLOSED pgs=1218 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:39:08.216 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:08.202+0000 7f524b22e700 1 --2- 172.21.15.42:0/1236718398 >> v2:172.21.15.40:3300/0 conn(0x7f52440a6ea0 0x7f5244118bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:39:08.216 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:08.202+0000 7f524b22e700 1 -- 172.21.15.42:0/1236718398 shutdown_connections 2019-12-05T05:39:08.216 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:08.202+0000 7f524b22e700 1 -- 172.21.15.42:0/1236718398 wait complete. 2019-12-05T05:39:08.216 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:08.202+0000 7f524b22e700 1 -- 172.21.15.42:0/1236718398 >> 172.21.15.42:0/1236718398 conn(0x7f5244059700 msgr2=0x7f5244117370 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:39:08.216 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:39:08.216 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":148,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7416,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/123054133","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":123054133},{"type":"v1","addr":"172.21.15.40:6833","nonce":123054133}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":148}],"filesystems":[{"mdsmap":{"epoch":147,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:37:13.880687+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5693},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5693":{"gid":5693,"name":"a","rank":0,"incarnation":144,"state":"up:active","state_seq":39,"addr":"172.21.15.42:6835/3838877789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3838877789},{"type":"v1","addr":"172.21.15.42:6835","nonce":3838877789}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:39:08.218 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T05:39:10.219 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:39:10.327 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.318+0000 7fa3d23ef700 1 Processor -- start 2019-12-05T05:39:10.327 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.318+0000 7fa3d23ef700 1 -- start start 2019-12-05T05:39:10.327 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.318+0000 7fa3d23ef700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa3cc0a7730 0x7fa3cc08df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:39:10.327 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.318+0000 7fa3d23ef700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fa3cc07e710 0x7fa3cc07eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:39:10.327 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.318+0000 7fa3d23ef700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3cc0a0700 con 0x7fa3cc09f2d0 2019-12-05T05:39:10.328 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.318+0000 7fa3d23ef700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3cc0a0af0 con 0x7fa3cc0a7730 2019-12-05T05:39:10.328 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.318+0000 7fa3d23ef700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3cc08e5a0 con 0x7fa3cc07e710 2019-12-05T05:39:10.328 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.318+0000 7fa3c3fff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fa3cc07e710 0x7fa3cc07eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:39:10.328 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.318+0000 7fa3c3fff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fa3cc07e710 0x7fa3cc07eb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:38036/0 (socket says 172.21.15.42:38036) 2019-12-05T05:39:10.328 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.318+0000 7fa3c3fff700 1 -- 172.21.15.42:0/332730366 learned_addr learned my addr 172.21.15.42:0/332730366 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:39:10.328 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.318+0000 7fa3d098c700 1 --2- 172.21.15.42:0/332730366 >> v2:172.21.15.40:3300/0 conn(0x7fa3cc0a7730 0x7fa3cc08df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:39:10.328 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.318+0000 7fa3c3fff700 1 -- 172.21.15.42:0/332730366 >> v2:172.21.15.40:3300/0 conn(0x7fa3cc0a7730 msgr2=0x7fa3cc08df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:39:10.329 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.318+0000 7fa3c3fff700 1 --2- 172.21.15.42:0/332730366 >> v2:172.21.15.40:3300/0 conn(0x7fa3cc0a7730 0x7fa3cc08df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:39:10.329 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.318+0000 7fa3c3fff700 1 -- 172.21.15.42:0/332730366 >> v1:172.21.15.40:6789/0 conn(0x7fa3cc09f2d0 legacy=0x7fa3cc09f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:39:10.329 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.318+0000 7fa3c3fff700 1 -- 172.21.15.42:0/332730366 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3cc07f090 con 0x7fa3cc07e710 2019-12-05T05:39:10.329 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.318+0000 7fa3c3fff700 1 --2- 172.21.15.42:0/332730366 >> v2:172.21.15.42:3300/0 conn(0x7fa3cc07e710 0x7fa3cc07eb40 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fa3bc002f30 tx=0x7fa3bc00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:10.329 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.318+0000 7fa3c3fff700 1 --2- 172.21.15.42:0/332730366 >> v2:172.21.15.42:3300/0 conn(0x7fa3cc07e710 0x7fa3cc07eb40 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fa3bc002f30 tx=0x7fa3bc00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:10.329 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.318+0000 7fa3c3fff700 1 --2- 172.21.15.42:0/332730366 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa3cc07e710 0x7fa3cc07eb40 secure :-1 s=READY pgs=1219 cs=0 l=1 rx=0x7fa3bc002f30 tx=0x7fa3bc00fd30).ready entity=mon.1 client_cookie=51e23acbef7cbd8f server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:39:10.330 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.322+0000 7fa3c3fff700 1 --2- 172.21.15.42:0/332730366 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa3cc07e710 0x7fa3cc07eb40 secure :-1 s=THROTTLE_DONE pgs=1219 cs=0 l=1 rx=0x7fa3bc002f30 tx=0x7fa3bc00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:10.330 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.322+0000 7fa3c3fff700 1 --2- 172.21.15.42:0/332730366 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa3cc07e710 0x7fa3cc07eb40 secure :-1 s=THROTTLE_DONE pgs=1219 cs=0 l=1 rx=0x7fa3bc002f30 tx=0x7fa3bc00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:10.330 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.322+0000 7fa3c3fff700 1 --2- 172.21.15.42:0/332730366 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa3cc07e710 0x7fa3cc07eb40 secure :-1 s=THROTTLE_DONE pgs=1219 cs=0 l=1 rx=0x7fa3bc002f30 tx=0x7fa3bc00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:10.330 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.322+0000 7fa3c37fe700 1 -- 172.21.15.42:0/332730366 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa3bc011370 con 0x7fa3cc07e710 2019-12-05T05:39:10.330 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.322+0000 7fa3c37fe700 1 -- 172.21.15.42:0/332730366 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3bc008bb0 con 0x7fa3cc07e710 2019-12-05T05:39:10.330 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.322+0000 7fa3c37fe700 1 -- 172.21.15.42:0/332730366 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa3bc011950 con 0x7fa3cc07e710 2019-12-05T05:39:10.330 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.322+0000 7fa3d23ef700 1 -- 172.21.15.42:0/332730366 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa3cc07e710 msgr2=0x7fa3cc07eb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:39:10.331 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.322+0000 7fa3d23ef700 1 --2- 172.21.15.42:0/332730366 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa3cc07e710 0x7fa3cc07eb40 secure :-1 s=READY pgs=1219 cs=0 l=1 rx=0x7fa3bc002f30 tx=0x7fa3bc00fd30).stop 2019-12-05T05:39:10.331 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.322+0000 7fa3d23ef700 1 -- 172.21.15.42:0/332730366 shutdown_connections 2019-12-05T05:39:10.331 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.322+0000 7fa3d23ef700 1 --2- 172.21.15.42:0/332730366 >> v2:172.21.15.40:3300/0 conn(0x7fa3cc0a7730 0x7fa3cc08df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:39:10.331 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.322+0000 7fa3d23ef700 1 --2- 172.21.15.42:0/332730366 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa3cc07e710 0x7fa3cc07eb40 unknown :-1 s=CLOSED pgs=1219 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:39:10.331 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.322+0000 7fa3d23ef700 1 -- 172.21.15.42:0/332730366 shutdown_connections 2019-12-05T05:39:10.331 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.322+0000 7fa3d23ef700 1 -- 172.21.15.42:0/332730366 wait complete. 2019-12-05T05:39:10.331 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.322+0000 7fa3d23ef700 1 -- 172.21.15.42:0/332730366 >> 172.21.15.42:0/332730366 conn(0x7fa3cc0aabd0 msgr2=0x7fa3cc0a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:39:10.335 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.326+0000 7fa3d23ef700 1 Processor -- start 2019-12-05T05:39:10.335 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.326+0000 7fa3d23ef700 1 -- start start 2019-12-05T05:39:10.336 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.326+0000 7fa3d23ef700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa3cc0a9aa0 0x7fa3cc116bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:39:10.336 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.326+0000 7fa3d23ef700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3cc119c00 con 0x7fa3cc117110 2019-12-05T05:39:10.336 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.326+0000 7fa3d23ef700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3cc11b860 con 0x7fa3cc118fc0 2019-12-05T05:39:10.336 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.326+0000 7fa3d23ef700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3cc11b080 con 0x7fa3cc0a9aa0 2019-12-05T05:39:10.336 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.326+0000 7fa3d118d700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fa3cc118fc0 0x7fa3cc077520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:38082/0 (socket says 172.21.15.42:38082) 2019-12-05T05:39:10.336 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.326+0000 7fa3d118d700 1 -- 172.21.15.42:0/1251193007 learned_addr learned my addr 172.21.15.42:0/1251193007 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:39:10.337 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.326+0000 7fa3d098c700 1 --2- 172.21.15.42:0/1251193007 >> v2:172.21.15.40:3300/0 conn(0x7fa3cc0a9aa0 0x7fa3cc116bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:39:10.337 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.326+0000 7fa3c17fa700 1 -- 172.21.15.42:0/1251193007 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1573452332 0 0) 0x7fa3cc11b860 con 0x7fa3cc118fc0 2019-12-05T05:39:10.337 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.326+0000 7fa3c17fa700 1 -- 172.21.15.42:0/1251193007 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3ac0030b0 con 0x7fa3cc118fc0 2019-12-05T05:39:10.337 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.326+0000 7fa3c17fa700 1 -- 172.21.15.42:0/1251193007 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2349276983 0 0) 0x7fa3cc119c00 con 0x7fa3cc117110 2019-12-05T05:39:10.337 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.326+0000 7fa3c17fa700 1 -- 172.21.15.42:0/1251193007 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3cc11b860 con 0x7fa3cc117110 2019-12-05T05:39:10.337 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.330+0000 7fa3d098c700 1 -- 172.21.15.42:0/1251193007 >> v1:172.21.15.42:6789/0 conn(0x7fa3cc118fc0 legacy=0x7fa3cc077520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:39:10.337 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.330+0000 7fa3d098c700 1 -- 172.21.15.42:0/1251193007 >> v1:172.21.15.40:6789/0 conn(0x7fa3cc117110 legacy=0x7fa3cc1187c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:39:10.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.330+0000 7fa3d098c700 1 -- 172.21.15.42:0/1251193007 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3cc1183b0 con 0x7fa3cc0a9aa0 2019-12-05T05:39:10.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.330+0000 7fa3d098c700 1 --2- 172.21.15.42:0/1251193007 >> v2:172.21.15.40:3300/0 conn(0x7fa3cc0a9aa0 0x7fa3cc116bc0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fa3c8002f30 tx=0x7fa3c8010960).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:10.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.330+0000 7fa3d098c700 1 --2- 172.21.15.42:0/1251193007 >> v2:172.21.15.40:3300/0 conn(0x7fa3cc0a9aa0 0x7fa3cc116bc0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fa3c8002f30 tx=0x7fa3c8010960).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:10.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.330+0000 7fa3d098c700 1 --2- 172.21.15.42:0/1251193007 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa3cc0a9aa0 0x7fa3cc116bc0 secure :-1 s=READY pgs=949 cs=0 l=1 rx=0x7fa3c8002f30 tx=0x7fa3c8010960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:39:10.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.330+0000 7fa3d098c700 1 --2- 172.21.15.42:0/1251193007 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa3cc0a9aa0 0x7fa3cc116bc0 secure :-1 s=THROTTLE_DONE pgs=949 cs=0 l=1 rx=0x7fa3c8002f30 tx=0x7fa3c8010960).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:10.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.330+0000 7fa3c17fa700 1 -- 172.21.15.42:0/1251193007 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa3c8005920 con 0x7fa3cc0a9aa0 2019-12-05T05:39:10.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.330+0000 7fa3d098c700 1 --2- 172.21.15.42:0/1251193007 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa3cc0a9aa0 0x7fa3cc116bc0 secure :-1 s=THROTTLE_DONE pgs=949 cs=0 l=1 rx=0x7fa3c8002f30 tx=0x7fa3c8010960).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:10.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.330+0000 7fa3d098c700 1 --2- 172.21.15.42:0/1251193007 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa3cc0a9aa0 0x7fa3cc116bc0 secure :-1 s=THROTTLE_DONE pgs=949 cs=0 l=1 rx=0x7fa3c8002f30 tx=0x7fa3c8010960).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:10.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.330+0000 7fa3c17fa700 1 -- 172.21.15.42:0/1251193007 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3c8005540 con 0x7fa3cc0a9aa0 2019-12-05T05:39:10.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.330+0000 7fa3c17fa700 1 -- 172.21.15.42:0/1251193007 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa3c8012460 con 0x7fa3cc0a9aa0 2019-12-05T05:39:10.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.330+0000 7fa3d23ef700 1 -- 172.21.15.42:0/1251193007 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3cc1193f0 con 0x7fa3cc0a9aa0 2019-12-05T05:39:10.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.330+0000 7fa3d23ef700 1 -- 172.21.15.42:0/1251193007 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3cc117880 con 0x7fa3cc0a9aa0 2019-12-05T05:39:10.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.330+0000 7fa3d098c700 1 --2- 172.21.15.42:0/1251193007 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa3cc0a9aa0 0x7fa3cc116bc0 secure :-1 s=THROTTLE_DONE pgs=949 cs=0 l=1 rx=0x7fa3c8002f30 tx=0x7fa3c8010960).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:10.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.330+0000 7fa3d098c700 1 --2- 172.21.15.42:0/1251193007 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa3cc0a9aa0 0x7fa3cc116bc0 secure :-1 s=THROTTLE_DONE pgs=949 cs=0 l=1 rx=0x7fa3c8002f30 tx=0x7fa3c8010960).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:10.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.330+0000 7fa3c17fa700 1 -- 172.21.15.42:0/1251193007 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fa3c8007180 con 0x7fa3cc0a9aa0 2019-12-05T05:39:10.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.330+0000 7fa3c17fa700 1 --2- 172.21.15.42:0/1251193007 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa3ac03b410 0x7fa3ac03d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:39:10.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.330+0000 7fa3c17fa700 1 -- 172.21.15.42:0/1251193007 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 8095+0+0 (secure 0 0 0) 0x7fa3c804ac00 con 0x7fa3cc0a9aa0 2019-12-05T05:39:10.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.330+0000 7fa3c3fff700 1 --2- 172.21.15.42:0/1251193007 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa3ac03b410 0x7fa3ac03d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:39:10.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.330+0000 7fa3c3fff700 1 --2- 172.21.15.42:0/1251193007 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa3ac03b410 0x7fa3ac03d920 crc :-1 s=READY pgs=526 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:39:10.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.334+0000 7fa3d23ef700 1 -- 172.21.15.42:0/1251193007 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3cc05d100 con 0x7fa3cc0a9aa0 2019-12-05T05:39:10.341 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.338+0000 7fa3d098c700 1 --2- 172.21.15.42:0/1251193007 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa3cc0a9aa0 0x7fa3cc116bc0 secure :-1 s=THROTTLE_DONE pgs=949 cs=0 l=1 rx=0x7fa3c8002f30 tx=0x7fa3c8010960).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:10.341 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.338+0000 7fa3c17fa700 1 -- 172.21.15.42:0/1251193007 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fa3c8031c40 con 0x7fa3cc0a9aa0 2019-12-05T05:39:10.505 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.502+0000 7fa3d23ef700 1 -- 172.21.15.42:0/1251193007 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa3cc11ae60 con 0x7fa3cc0a9aa0 2019-12-05T05:39:10.505 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.502+0000 7fa3d098c700 1 --2- 172.21.15.42:0/1251193007 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa3cc0a9aa0 0x7fa3cc116bc0 secure :-1 s=THROTTLE_DONE pgs=949 cs=0 l=1 rx=0x7fa3c8002f30 tx=0x7fa3c8010960).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:10.506 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.502+0000 7fa3c17fa700 1 -- 172.21.15.42:0/1251193007 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 149 v149) v1 ==== 95+0+1771 (secure 0 0 0) 0x7fa3c8031ea0 con 0x7fa3cc0a9aa0 2019-12-05T05:39:10.506 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 149 2019-12-05T05:39:10.506 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:39:10.506 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":149,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":149,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:39:08.863175+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7416},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7416":{"gid":7416,"name":"b","rank":0,"incarnation":149,"state":"up:replay","state_seq":2,"addr":"172.21.15.40:6833/123054133","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":123054133},{"type":"v1","addr":"172.21.15.40:6833","nonce":123054133}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:39:10.509 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.506+0000 7fa3d23ef700 1 -- 172.21.15.42:0/1251193007 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa3ac03b410 msgr2=0x7fa3ac03d920 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:39:10.509 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.506+0000 7fa3d23ef700 1 --2- 172.21.15.42:0/1251193007 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa3ac03b410 0x7fa3ac03d920 crc :-1 s=READY pgs=526 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:39:10.509 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.506+0000 7fa3d23ef700 1 -- 172.21.15.42:0/1251193007 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa3cc0a9aa0 msgr2=0x7fa3cc116bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:39:10.509 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.506+0000 7fa3d23ef700 1 --2- 172.21.15.42:0/1251193007 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa3cc0a9aa0 0x7fa3cc116bc0 secure :-1 s=READY pgs=949 cs=0 l=1 rx=0x7fa3c8002f30 tx=0x7fa3c8010960).stop 2019-12-05T05:39:10.509 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.506+0000 7fa3d23ef700 1 -- 172.21.15.42:0/1251193007 shutdown_connections 2019-12-05T05:39:10.509 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.506+0000 7fa3d23ef700 1 --2- 172.21.15.42:0/1251193007 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa3ac03b410 0x7fa3ac03d920 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:39:10.509 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.506+0000 7fa3d23ef700 1 --2- 172.21.15.42:0/1251193007 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa3cc0a9aa0 0x7fa3cc116bc0 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:39:10.510 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.506+0000 7fa3d23ef700 1 -- 172.21.15.42:0/1251193007 shutdown_connections 2019-12-05T05:39:10.510 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.506+0000 7fa3d23ef700 1 -- 172.21.15.42:0/1251193007 wait complete. 2019-12-05T05:39:10.510 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.506+0000 7fa3d23ef700 1 -- 172.21.15.42:0/1251193007 >> 172.21.15.42:0/1251193007 conn(0x7fa3cc0aabd0 msgr2=0x7fa3cc11d1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:39:10.518 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a down, removed from mdsmap 2019-12-05T05:39:10.518 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2019-12-05T05:39:10.519 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:39:10.578 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:39:10.603 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:39:10.617 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.602+0000 7f5881e69700 1 Processor -- start 2019-12-05T05:39:10.617 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.602+0000 7f5881e69700 1 -- start start 2019-12-05T05:39:10.618 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.606+0000 7f5881e69700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f587c0a7740 0x7f587c08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:39:10.618 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.606+0000 7f5881e69700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f587c0a0710 con 0x7f587c07e720 2019-12-05T05:39:10.618 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.606+0000 7f5881e69700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f587c0a9ba0 con 0x7f587c09fae0 2019-12-05T05:39:10.618 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.606+0000 7f5881e69700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f587c0a0b00 con 0x7f587c0a7740 2019-12-05T05:39:10.618 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.606+0000 7f587bfff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f587c0a7740 0x7f587c08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:39:10.618 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.606+0000 7f5880c07700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f587c09fae0 0x7f587c14efe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:38090/0 (socket says 172.21.15.42:38090) 2019-12-05T05:39:10.619 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.606+0000 7f5880c07700 1 -- 172.21.15.42:0/3523598131 learned_addr learned my addr 172.21.15.42:0/3523598131 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:39:10.619 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.606+0000 7f587b7fe700 1 --1- 172.21.15.42:0/3523598131 >> v1:172.21.15.40:6789/0 conn(0x7f587c07e720 0x7f587c09f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:47528/0 (socket says 172.21.15.42:47528) 2019-12-05T05:39:10.619 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.606+0000 7f587bfff700 1 -- 172.21.15.42:0/3523598131 >> v1:172.21.15.42:6789/0 conn(0x7f587c09fae0 legacy=0x7f587c14efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:39:10.619 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.606+0000 7f587affd700 1 -- 172.21.15.42:0/3523598131 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 65609869 0 0) 0x7f587c0a0710 con 0x7f587c07e720 2019-12-05T05:39:10.619 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.606+0000 7f587bfff700 1 -- 172.21.15.42:0/3523598131 >> v1:172.21.15.40:6789/0 conn(0x7f587c07e720 legacy=0x7f587c09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:39:10.619 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.606+0000 7f587bfff700 1 -- 172.21.15.42:0/3523598131 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f587c07eb50 con 0x7f587c0a7740 2019-12-05T05:39:10.619 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.606+0000 7f587bfff700 1 --2- 172.21.15.42:0/3523598131 >> v2:172.21.15.42:3300/0 conn(0x7f587c0a7740 0x7f587c08df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f5874008790 tx=0x7f586c001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:10.620 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.606+0000 7f587bfff700 1 --2- 172.21.15.42:0/3523598131 >> v2:172.21.15.42:3300/0 conn(0x7f587c0a7740 0x7f587c08df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f5874008790 tx=0x7f586c001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:10.620 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.606+0000 7f587bfff700 1 --2- 172.21.15.42:0/3523598131 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f587c0a7740 0x7f587c08df80 secure :-1 s=READY pgs=1222 cs=0 l=1 rx=0x7f5874008790 tx=0x7f586c001150).ready entity=mon.1 client_cookie=58f5e4b18b1c51a8 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:39:10.620 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.606+0000 7f587bfff700 1 --2- 172.21.15.42:0/3523598131 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f587c0a7740 0x7f587c08df80 secure :-1 s=THROTTLE_DONE pgs=1222 cs=0 l=1 rx=0x7f5874008790 tx=0x7f586c001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:10.620 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.606+0000 7f587affd700 1 -- 172.21.15.42:0/3523598131 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5874008bb0 con 0x7f587c0a7740 2019-12-05T05:39:10.620 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.606+0000 7f587bfff700 1 --2- 172.21.15.42:0/3523598131 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f587c0a7740 0x7f587c08df80 secure :-1 s=THROTTLE_DONE pgs=1222 cs=0 l=1 rx=0x7f5874008790 tx=0x7f586c001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:10.620 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.606+0000 7f587bfff700 1 --2- 172.21.15.42:0/3523598131 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f587c0a7740 0x7f587c08df80 secure :-1 s=THROTTLE_DONE pgs=1222 cs=0 l=1 rx=0x7f5874008790 tx=0x7f586c001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:10.620 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.606+0000 7f587affd700 1 -- 172.21.15.42:0/3523598131 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58740087d0 con 0x7f587c0a7740 2019-12-05T05:39:10.621 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.606+0000 7f587affd700 1 -- 172.21.15.42:0/3523598131 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f58740117d0 con 0x7f587c0a7740 2019-12-05T05:39:10.621 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.606+0000 7f5881e69700 1 -- 172.21.15.42:0/3523598131 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f587c0a7740 msgr2=0x7f587c08df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:39:10.621 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.606+0000 7f5881e69700 1 --2- 172.21.15.42:0/3523598131 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f587c0a7740 0x7f587c08df80 secure :-1 s=READY pgs=1222 cs=0 l=1 rx=0x7f5874008790 tx=0x7f586c001150).stop 2019-12-05T05:39:10.621 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.606+0000 7f5881e69700 1 -- 172.21.15.42:0/3523598131 shutdown_connections 2019-12-05T05:39:10.621 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.606+0000 7f5881e69700 1 --2- 172.21.15.42:0/3523598131 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f587c0a7740 0x7f587c08df80 unknown :-1 s=CLOSED pgs=1222 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:39:10.621 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.606+0000 7f5881e69700 1 -- 172.21.15.42:0/3523598131 shutdown_connections 2019-12-05T05:39:10.621 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.606+0000 7f5881e69700 1 -- 172.21.15.42:0/3523598131 wait complete. 2019-12-05T05:39:10.622 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.606+0000 7f5881e69700 1 -- 172.21.15.42:0/3523598131 >> 172.21.15.42:0/3523598131 conn(0x7f587c0aabe0 msgr2=0x7f587c0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:39:10.622 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.606+0000 7f5881e69700 1 Processor -- start 2019-12-05T05:39:10.622 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.610+0000 7f5881e69700 1 -- start start 2019-12-05T05:39:10.622 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.610+0000 7f5881e69700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f587c12d020 0x7f587c0a2c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:39:10.622 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.610+0000 7f5881e69700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f587c126f30 0x7f587c127360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:39:10.622 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.610+0000 7f5881e69700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f587c142ca0 con 0x7f587c142870 2019-12-05T05:39:10.622 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.610+0000 7f5881e69700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f587c130810 con 0x7f587c12d020 2019-12-05T05:39:10.623 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.610+0000 7f5881e69700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f587c130ab0 con 0x7f587c126f30 2019-12-05T05:39:10.623 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.610+0000 7f587bfff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f587c12d020 0x7f587c0a2c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:39:10.623 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.610+0000 7f587b7fe700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f587c142870 0x7f587c130d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:38094/0 (socket says 172.21.15.42:38094) 2019-12-05T05:39:10.623 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.610+0000 7f587b7fe700 1 -- 172.21.15.42:0/1451442045 learned_addr learned my addr 172.21.15.42:0/1451442045 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:39:10.623 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.610+0000 7f5880c07700 1 --2- 172.21.15.42:0/1451442045 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f587c126f30 0x7f587c127360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:39:10.623 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.610+0000 7f5878ff9700 1 -- 172.21.15.42:0/1451442045 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 77715021 0 0) 0x7f587c142ca0 con 0x7f587c142870 2019-12-05T05:39:10.623 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.610+0000 7f5878ff9700 1 -- 172.21.15.42:0/1451442045 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58600030b0 con 0x7f587c142870 2019-12-05T05:39:10.624 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.610+0000 7f587bfff700 1 -- 172.21.15.42:0/1451442045 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f587c126f30 msgr2=0x7f587c127360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:39:10.624 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.610+0000 7f587bfff700 1 --2- 172.21.15.42:0/1451442045 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f587c126f30 0x7f587c127360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:39:10.624 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.610+0000 7f587bfff700 1 -- 172.21.15.42:0/1451442045 >> v1:172.21.15.42:6789/0 conn(0x7f587c142870 legacy=0x7f587c130d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:39:10.624 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.610+0000 7f587bfff700 1 -- 172.21.15.42:0/1451442045 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f587c138a40 con 0x7f587c12d020 2019-12-05T05:39:10.624 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.610+0000 7f587bfff700 1 --2- 172.21.15.42:0/1451442045 >> v2:172.21.15.42:3300/0 conn(0x7f587c12d020 0x7f587c0a2c00 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f587c1309d0 tx=0x7f5874008a80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:10.624 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.610+0000 7f587bfff700 1 --2- 172.21.15.42:0/1451442045 >> v2:172.21.15.42:3300/0 conn(0x7f587c12d020 0x7f587c0a2c00 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f587c1309d0 tx=0x7f5874008a80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:10.624 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.610+0000 7f587bfff700 1 --2- 172.21.15.42:0/1451442045 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f587c12d020 0x7f587c0a2c00 secure :-1 s=READY pgs=1224 cs=0 l=1 rx=0x7f587c1309d0 tx=0x7f5874008a80).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:39:10.625 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.610+0000 7f587bfff700 1 --2- 172.21.15.42:0/1451442045 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f587c12d020 0x7f587c0a2c00 secure :-1 s=THROTTLE_DONE pgs=1224 cs=0 l=1 rx=0x7f587c1309d0 tx=0x7f5874008a80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:10.625 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.610+0000 7f5878ff9700 1 -- 172.21.15.42:0/1451442045 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5874005540 con 0x7f587c12d020 2019-12-05T05:39:10.625 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.610+0000 7f587bfff700 1 --2- 172.21.15.42:0/1451442045 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f587c12d020 0x7f587c0a2c00 secure :-1 s=THROTTLE_DONE pgs=1224 cs=0 l=1 rx=0x7f587c1309d0 tx=0x7f5874008a80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:10.625 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.610+0000 7f5881e69700 1 -- 172.21.15.42:0/1451442045 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f587c137a30 con 0x7f587c12d020 2019-12-05T05:39:10.625 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.610+0000 7f5878ff9700 1 -- 172.21.15.42:0/1451442045 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5874008db0 con 0x7f587c12d020 2019-12-05T05:39:10.625 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.610+0000 7f587bfff700 1 --2- 172.21.15.42:0/1451442045 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f587c12d020 0x7f587c0a2c00 secure :-1 s=THROTTLE_DONE pgs=1224 cs=0 l=1 rx=0x7f587c1309d0 tx=0x7f5874008a80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:10.625 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.610+0000 7f5881e69700 1 -- 172.21.15.42:0/1451442045 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f587c137e10 con 0x7f587c12d020 2019-12-05T05:39:10.626 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.610+0000 7f5878ff9700 1 -- 172.21.15.42:0/1451442045 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5874005b20 con 0x7f587c12d020 2019-12-05T05:39:10.626 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.610+0000 7f587bfff700 1 --2- 172.21.15.42:0/1451442045 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f587c12d020 0x7f587c0a2c00 secure :-1 s=THROTTLE_DONE pgs=1224 cs=0 l=1 rx=0x7f587c1309d0 tx=0x7f5874008a80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:10.626 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.610+0000 7f5878ff9700 1 -- 172.21.15.42:0/1451442045 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f5874015490 con 0x7f587c12d020 2019-12-05T05:39:10.626 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.610+0000 7f587bfff700 1 --2- 172.21.15.42:0/1451442045 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f587c12d020 0x7f587c0a2c00 secure :-1 s=THROTTLE_DONE pgs=1224 cs=0 l=1 rx=0x7f587c1309d0 tx=0x7f5874008a80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:10.626 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.614+0000 7f5878ff9700 1 --2- 172.21.15.42:0/1451442045 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f586003b270 0x7f586003d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:39:10.626 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.614+0000 7f5878ff9700 1 -- 172.21.15.42:0/1451442045 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 8095+0+0 (secure 0 0 0) 0x7f587404d980 con 0x7f587c12d020 2019-12-05T05:39:10.626 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.614+0000 7f587b7fe700 1 --2- 172.21.15.42:0/1451442045 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f586003b270 0x7f586003d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:39:10.627 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.614+0000 7f587b7fe700 1 --2- 172.21.15.42:0/1451442045 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f586003b270 0x7f586003d780 crc :-1 s=READY pgs=527 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:39:10.627 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.618+0000 7f5881e69700 1 -- 172.21.15.42:0/1451442045 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f587c05d110 con 0x7f587c12d020 2019-12-05T05:39:10.628 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.622+0000 7f587bfff700 1 --2- 172.21.15.42:0/1451442045 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f587c12d020 0x7f587c0a2c00 secure :-1 s=THROTTLE_DONE pgs=1224 cs=0 l=1 rx=0x7f587c1309d0 tx=0x7f5874008a80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:10.628 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.622+0000 7f5878ff9700 1 -- 172.21.15.42:0/1451442045 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f5874015d90 con 0x7f587c12d020 2019-12-05T05:39:10.791 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.786+0000 7f5881e69700 1 -- 172.21.15.42:0/1451442045 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f587c05b8c0 con 0x7f587c12d020 2019-12-05T05:39:10.792 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.790+0000 7f587bfff700 1 --2- 172.21.15.42:0/1451442045 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f587c12d020 0x7f587c0a2c00 secure :-1 s=THROTTLE_DONE pgs=1224 cs=0 l=1 rx=0x7f587c1309d0 tx=0x7f5874008a80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:10.792 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.790+0000 7f5878ff9700 1 -- 172.21.15.42:0/1451442045 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 149 v149) v1 ==== 95+0+1771 (secure 0 0 0) 0x7f5874011e30 con 0x7f587c12d020 2019-12-05T05:39:10.792 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 149 2019-12-05T05:39:10.792 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:39:10.793 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":149,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":149,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:39:08.863175+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7416},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7416":{"gid":7416,"name":"b","rank":0,"incarnation":149,"state":"up:replay","state_seq":2,"addr":"172.21.15.40:6833/123054133","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":123054133},{"type":"v1","addr":"172.21.15.40:6833","nonce":123054133}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:39:10.795 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.790+0000 7f5881e69700 1 -- 172.21.15.42:0/1451442045 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f586003b270 msgr2=0x7f586003d780 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:39:10.795 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.790+0000 7f5881e69700 1 --2- 172.21.15.42:0/1451442045 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f586003b270 0x7f586003d780 crc :-1 s=READY pgs=527 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:39:10.795 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.790+0000 7f5881e69700 1 -- 172.21.15.42:0/1451442045 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f587c12d020 msgr2=0x7f587c0a2c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:39:10.795 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.790+0000 7f5881e69700 1 --2- 172.21.15.42:0/1451442045 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f587c12d020 0x7f587c0a2c00 secure :-1 s=READY pgs=1224 cs=0 l=1 rx=0x7f587c1309d0 tx=0x7f5874008a80).stop 2019-12-05T05:39:10.795 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.790+0000 7f5881e69700 1 -- 172.21.15.42:0/1451442045 shutdown_connections 2019-12-05T05:39:10.795 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.790+0000 7f5881e69700 1 --2- 172.21.15.42:0/1451442045 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f586003b270 0x7f586003d780 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:39:10.796 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.790+0000 7f5881e69700 1 --2- 172.21.15.42:0/1451442045 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f587c126f30 0x7f587c127360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:39:10.796 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.790+0000 7f5881e69700 1 --2- 172.21.15.42:0/1451442045 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f587c12d020 0x7f587c0a2c00 unknown :-1 s=CLOSED pgs=1224 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:39:10.796 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.790+0000 7f5881e69700 1 -- 172.21.15.42:0/1451442045 shutdown_connections 2019-12-05T05:39:10.796 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.790+0000 7f5881e69700 1 -- 172.21.15.42:0/1451442045 wait complete. 2019-12-05T05:39:10.796 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:10.790+0000 7f5881e69700 1 -- 172.21.15.42:0/1451442045 >> 172.21.15.42:0/1451442045 conn(0x7f587c066fe0 msgr2=0x7f587c07d810 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:39:10.804 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 149, "feature_flags": { "enable_multiple": false, "ever_enabled_multiple": false }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2019-12-05T05:03:35.604738+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 149, "ever_allowed_features": 2, "explicitly_allowed_features": 2, "failed": [], "flags": 18, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_7416": { "addr": "172.21.15.40:6833/123054133", "addrs": { "addrvec": [ { "addr": "172.21.15.40:6832", "nonce": 123054133, "type": "v2" }, { "addr": "172.21.15.40:6833", "nonce": 123054133, "type": "v1" } ] }, "export_targets": [], "features": 4540138292836696063, "flags": 0, "gid": 7416, "incarnation": 149, "name": "b", "rank": 0, "state": "up:replay", "state_seq": 2 } }, "last_failure": 0, "last_failure_osd_epoch": 73, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 2, "min_compat_client": "0 (unknown)", "modified": "2019-12-05T05:39:08.863175+0000", "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7416 } } } ], "standbys": [] } 2019-12-05T05:39:12.806 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:39:12.915 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:12.906+0000 7f5718c7e700 1 Processor -- start 2019-12-05T05:39:12.915 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:12.906+0000 7f5718c7e700 1 -- start start 2019-12-05T05:39:12.915 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:12.906+0000 7f5718c7e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f57140a7740 0x7f571408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:39:12.915 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:12.906+0000 7f5718c7e700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f571407e720 0x7f571407eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:39:12.916 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:12.906+0000 7f5718c7e700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f571409f2e0 0x7f571409f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:39:12.916 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:12.906+0000 7f5718c7e700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57140a0b00 con 0x7f571407e720 2019-12-05T05:39:12.916 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:12.906+0000 7f5718c7e700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f571408e5b0 con 0x7f57140a7740 2019-12-05T05:39:12.916 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:12.906+0000 7f5718c7e700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f571408e850 con 0x7f571409f2e0 2019-12-05T05:39:12.916 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:12.906+0000 7f5712d9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f57140a7740 0x7f571408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:39:12.916 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:12.906+0000 7f571259c700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f571407e720 0x7f571407eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:39:12.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:12.906+0000 7f5712d9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f57140a7740 0x7f571408df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:38068/0 (socket says 172.21.15.42:38068) 2019-12-05T05:39:12.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:12.906+0000 7f571259c700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f571407e720 0x7f571407eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.42:54118/0 (socket says 172.21.15.42:54118) 2019-12-05T05:39:12.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:12.906+0000 7f5712d9d700 1 -- 172.21.15.42:0/3612433781 learned_addr learned my addr 172.21.15.42:0/3612433781 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:39:12.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:12.906+0000 7f571359e700 1 --2- 172.21.15.42:0/3612433781 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f571409f2e0 0x7f571409f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:39:12.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:12.910+0000 7f571259c700 1 -- 172.21.15.42:0/3612433781 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f571409f2e0 msgr2=0x7f571409f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:39:12.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:12.910+0000 7f571259c700 1 --2- 172.21.15.42:0/3612433781 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f571409f2e0 0x7f571409f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:39:12.918 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:12.910+0000 7f571259c700 1 -- 172.21.15.42:0/3612433781 >> v2:172.21.15.42:3300/0 conn(0x7f57140a7740 msgr2=0x7f571408df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:39:12.918 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:12.910+0000 7f571259c700 1 --2- 172.21.15.42:0/3612433781 >> v2:172.21.15.42:3300/0 conn(0x7f57140a7740 0x7f571408df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:39:12.918 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:12.910+0000 7f571259c700 1 -- 172.21.15.42:0/3612433781 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f571407f0a0 con 0x7f571407e720 2019-12-05T05:39:12.918 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:12.910+0000 7f571259c700 1 --2- 172.21.15.42:0/3612433781 >> v2:172.21.15.40:3300/0 conn(0x7f571407e720 0x7f571407eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f57140a0f50 tx=0x7f570800fcf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:12.918 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:12.910+0000 7f571259c700 1 --2- 172.21.15.42:0/3612433781 >> v2:172.21.15.40:3300/0 conn(0x7f571407e720 0x7f571407eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f57140a0f50 tx=0x7f570800fcf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:12.918 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:12.910+0000 7f571259c700 1 --2- 172.21.15.42:0/3612433781 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f571407e720 0x7f571407eb50 secure :-1 s=READY pgs=951 cs=0 l=1 rx=0x7f57140a0f50 tx=0x7f570800fcf0).ready entity=mon.0 client_cookie=4b0f9a04ae542664 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:39:12.918 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:12.910+0000 7f571259c700 1 --2- 172.21.15.42:0/3612433781 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f571407e720 0x7f571407eb50 secure :-1 s=THROTTLE_DONE pgs=951 cs=0 l=1 rx=0x7f57140a0f50 tx=0x7f570800fcf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:12.919 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:12.910+0000 7f5711d9b700 1 -- 172.21.15.42:0/3612433781 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5708008bb0 con 0x7f571407e720 2019-12-05T05:39:12.919 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:12.910+0000 7f571259c700 1 --2- 172.21.15.42:0/3612433781 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f571407e720 0x7f571407eb50 secure :-1 s=THROTTLE_DONE pgs=951 cs=0 l=1 rx=0x7f57140a0f50 tx=0x7f570800fcf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:12.919 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:12.910+0000 7f571259c700 1 --2- 172.21.15.42:0/3612433781 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f571407e720 0x7f571407eb50 secure :-1 s=THROTTLE_DONE pgs=951 cs=0 l=1 rx=0x7f57140a0f50 tx=0x7f570800fcf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:12.919 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:12.910+0000 7f5711d9b700 1 -- 172.21.15.42:0/3612433781 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57080087d0 con 0x7f571407e720 2019-12-05T05:39:12.919 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:12.910+0000 7f5711d9b700 1 -- 172.21.15.42:0/3612433781 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f57080118a0 con 0x7f571407e720 2019-12-05T05:39:12.919 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:12.910+0000 7f5718c7e700 1 -- 172.21.15.42:0/3612433781 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f571407e720 msgr2=0x7f571407eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:39:12.920 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:12.910+0000 7f5718c7e700 1 --2- 172.21.15.42:0/3612433781 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f571407e720 0x7f571407eb50 secure :-1 s=READY pgs=951 cs=0 l=1 rx=0x7f57140a0f50 tx=0x7f570800fcf0).stop 2019-12-05T05:39:12.920 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:12.910+0000 7f5718c7e700 1 -- 172.21.15.42:0/3612433781 shutdown_connections 2019-12-05T05:39:12.920 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:12.910+0000 7f5718c7e700 1 --2- 172.21.15.42:0/3612433781 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f571409f2e0 0x7f571409f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:39:12.920 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:12.910+0000 7f5718c7e700 1 --2- 172.21.15.42:0/3612433781 >> v2:172.21.15.42:3300/0 conn(0x7f57140a7740 0x7f571408df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:39:12.920 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:12.910+0000 7f5718c7e700 1 --2- 172.21.15.42:0/3612433781 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f571407e720 0x7f571407eb50 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:39:12.920 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:12.910+0000 7f5718c7e700 1 -- 172.21.15.42:0/3612433781 shutdown_connections 2019-12-05T05:39:12.921 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:12.910+0000 7f5718c7e700 1 -- 172.21.15.42:0/3612433781 wait complete. 2019-12-05T05:39:12.921 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:12.910+0000 7f5718c7e700 1 -- 172.21.15.42:0/3612433781 >> 172.21.15.42:0/3612433781 conn(0x7f57140aabe0 msgr2=0x7f57140a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:39:12.925 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:12.914+0000 7f5718c7e700 1 Processor -- start 2019-12-05T05:39:12.925 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:12.914+0000 7f5718c7e700 1 -- start start 2019-12-05T05:39:12.925 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:12.914+0000 7f5718c7e700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5714121180 0x7f571407c480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:39:12.925 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:12.914+0000 7f5718c7e700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f571407c9d0 0x7f5714123110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:39:12.926 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:12.914+0000 7f5718c7e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5714123660 0x7f5714123a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:39:12.926 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:12.914+0000 7f5718c7e700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f571407d420 con 0x7f5714121180 2019-12-05T05:39:12.926 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:12.914+0000 7f5718c7e700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f571411ba00 con 0x7f5714123660 2019-12-05T05:39:12.926 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:12.914+0000 7f5718c7e700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f571411bca0 con 0x7f571407c9d0 2019-12-05T05:39:12.926 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:12.914+0000 7f571359e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5714123660 0x7f5714123a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:39:12.926 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:12.914+0000 7f571359e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5714123660 0x7f5714123a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:38078/0 (socket says 172.21.15.42:38078) 2019-12-05T05:39:12.927 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:12.914+0000 7f571359e700 1 -- 172.21.15.42:0/1576062172 learned_addr learned my addr 172.21.15.42:0/1576062172 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:39:12.927 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:12.914+0000 7f571259c700 1 --2- 172.21.15.42:0/1576062172 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f571407c9d0 0x7f5714123110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:39:12.927 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:12.914+0000 7f5712d9d700 1 --2- 172.21.15.42:0/1576062172 >> v2:172.21.15.40:3300/0 conn(0x7f5714121180 0x7f571407c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:39:12.927 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:12.914+0000 7f5712d9d700 1 -- 172.21.15.42:0/1576062172 >> v2:172.21.15.40:3300/0 conn(0x7f5714121180 msgr2=0x7f571407c480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2019-12-05T05:39:12.927 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:12.914+0000 7f5712d9d700 1 -- 172.21.15.42:0/1576062172 >> v2:172.21.15.40:3300/0 conn(0x7f5714121180 msgr2=0x7f571407c480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2019-12-05T05:39:12.927 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:12.914+0000 7f5712d9d700 1 --2- 172.21.15.42:0/1576062172 >> v2:172.21.15.40:3300/0 conn(0x7f5714121180 0x7f571407c480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-12-05T05:39:12.927 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:12.914+0000 7f5712d9d700 1 --2- 172.21.15.42:0/1576062172 >> v2:172.21.15.40:3300/0 conn(0x7f5714121180 0x7f571407c480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rx=0 tx=0)._fault waiting 0.200000 2019-12-05T05:39:12.928 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:12.914+0000 7f571359e700 1 -- 172.21.15.42:0/1576062172 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f571407c9d0 msgr2=0x7f5714123110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:39:12.928 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:12.914+0000 7f571359e700 1 --2- 172.21.15.42:0/1576062172 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f571407c9d0 0x7f5714123110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:39:12.928 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:12.914+0000 7f571359e700 1 -- 172.21.15.42:0/1576062172 >> v2:172.21.15.40:3300/0 conn(0x7f5714121180 msgr2=0x7f571407c480 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2019-12-05T05:39:12.928 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:12.914+0000 7f571359e700 1 --2- 172.21.15.42:0/1576062172 >> v2:172.21.15.40:3300/0 conn(0x7f5714121180 0x7f571407c480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:39:12.928 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:12.914+0000 7f571359e700 1 -- 172.21.15.42:0/1576062172 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f571407ce00 con 0x7f5714123660 2019-12-05T05:39:12.928 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:12.914+0000 7f571259c700 1 --2- 172.21.15.42:0/1576062172 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f571407c9d0 0x7f5714123110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:39:12.929 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:12.914+0000 7f571359e700 1 --2- 172.21.15.42:0/1576062172 >> v2:172.21.15.42:3300/0 conn(0x7f5714123660 0x7f5714123a90 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f571411bbc0 tx=0x7f570c010d80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:12.929 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:12.914+0000 7f571359e700 1 --2- 172.21.15.42:0/1576062172 >> v2:172.21.15.42:3300/0 conn(0x7f5714123660 0x7f5714123a90 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f571411bbc0 tx=0x7f570c010d80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:12.929 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:12.914+0000 7f571359e700 1 --2- 172.21.15.42:0/1576062172 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5714123660 0x7f5714123a90 secure :-1 s=READY pgs=1225 cs=0 l=1 rx=0x7f571411bbc0 tx=0x7f570c010d80).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:39:12.929 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:12.914+0000 7f571359e700 1 --2- 172.21.15.42:0/1576062172 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5714123660 0x7f5714123a90 secure :-1 s=THROTTLE_DONE pgs=1225 cs=0 l=1 rx=0x7f571411bbc0 tx=0x7f570c010d80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:12.929 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:12.918+0000 7f571359e700 1 --2- 172.21.15.42:0/1576062172 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5714123660 0x7f5714123a90 secure :-1 s=THROTTLE_DONE pgs=1225 cs=0 l=1 rx=0x7f571411bbc0 tx=0x7f570c010d80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:12.929 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:12.918+0000 7f57037fe700 1 -- 172.21.15.42:0/1576062172 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f570c012a70 con 0x7f5714123660 2019-12-05T05:39:12.930 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:12.918+0000 7f571359e700 1 --2- 172.21.15.42:0/1576062172 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5714123660 0x7f5714123a90 secure :-1 s=THROTTLE_DONE pgs=1225 cs=0 l=1 rx=0x7f571411bbc0 tx=0x7f570c010d80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:12.930 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:12.918+0000 7f57037fe700 1 -- 172.21.15.42:0/1576062172 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f570c012690 con 0x7f5714123660 2019-12-05T05:39:12.930 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:12.918+0000 7f5718c7e700 1 -- 172.21.15.42:0/1576062172 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f571407d0e0 con 0x7f5714123660 2019-12-05T05:39:12.930 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:12.918+0000 7f57037fe700 1 -- 172.21.15.42:0/1576062172 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f570c00e760 con 0x7f5714123660 2019-12-05T05:39:12.930 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:12.918+0000 7f5718c7e700 1 -- 172.21.15.42:0/1576062172 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57141215b0 con 0x7f5714123660 2019-12-05T05:39:12.930 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:12.918+0000 7f571359e700 1 --2- 172.21.15.42:0/1576062172 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5714123660 0x7f5714123a90 secure :-1 s=THROTTLE_DONE pgs=1225 cs=0 l=1 rx=0x7f571411bbc0 tx=0x7f570c010d80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:12.930 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:12.918+0000 7f571359e700 1 --2- 172.21.15.42:0/1576062172 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5714123660 0x7f5714123a90 secure :-1 s=THROTTLE_DONE pgs=1225 cs=0 l=1 rx=0x7f571411bbc0 tx=0x7f570c010d80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:12.931 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:12.918+0000 7f57037fe700 1 -- 172.21.15.42:0/1576062172 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f570c00ea70 con 0x7f5714123660 2019-12-05T05:39:12.931 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:12.918+0000 7f57037fe700 1 --2- 172.21.15.42:0/1576062172 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f56f803ae90 0x7f56f803d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:39:12.931 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:12.918+0000 7f57037fe700 1 -- 172.21.15.42:0/1576062172 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 8095+0+0 (secure 0 0 0) 0x7f570c04b880 con 0x7f5714123660 2019-12-05T05:39:12.931 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:12.918+0000 7f5712d9d700 1 --2- 172.21.15.42:0/1576062172 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f56f803ae90 0x7f56f803d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:39:12.931 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:12.918+0000 7f5712d9d700 1 --2- 172.21.15.42:0/1576062172 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f56f803ae90 0x7f56f803d3a0 crc :-1 s=READY pgs=528 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:39:12.931 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:12.922+0000 7f5718c7e700 1 -- 172.21.15.42:0/1576062172 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f571411b2e0 con 0x7f5714123660 2019-12-05T05:39:12.932 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:12.926+0000 7f571359e700 1 --2- 172.21.15.42:0/1576062172 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5714123660 0x7f5714123a90 secure :-1 s=THROTTLE_DONE pgs=1225 cs=0 l=1 rx=0x7f571411bbc0 tx=0x7f570c010d80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:12.932 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:12.926+0000 7f57037fe700 1 -- 172.21.15.42:0/1576062172 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f570c0328c0 con 0x7f5714123660 2019-12-05T05:39:13.095 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:13.090+0000 7f5718c7e700 1 -- 172.21.15.42:0/1576062172 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f571405d110 con 0x7f5714123660 2019-12-05T05:39:13.096 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:13.094+0000 7f571359e700 1 --2- 172.21.15.42:0/1576062172 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5714123660 0x7f5714123a90 secure :-1 s=THROTTLE_DONE pgs=1225 cs=0 l=1 rx=0x7f571411bbc0 tx=0x7f570c010d80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:13.096 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:13.094+0000 7f57037fe700 1 -- 172.21.15.42:0/1576062172 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 150 v150) v1 ==== 95+0+1775 (secure 0 0 0) 0x7f570c032b20 con 0x7f5714123660 2019-12-05T05:39:13.098 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:39:13.098 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":150,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":150,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:39:12.322193+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7416},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7416":{"gid":7416,"name":"b","rank":0,"incarnation":149,"state":"up:reconnect","state_seq":28,"addr":"172.21.15.40:6833/123054133","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":123054133},{"type":"v1","addr":"172.21.15.40:6833","nonce":123054133}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:39:13.099 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 150 2019-12-05T05:39:13.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:13.098+0000 7f5718c7e700 1 -- 172.21.15.42:0/1576062172 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f56f803ae90 msgr2=0x7f56f803d3a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:39:13.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:13.098+0000 7f5718c7e700 1 --2- 172.21.15.42:0/1576062172 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f56f803ae90 0x7f56f803d3a0 crc :-1 s=READY pgs=528 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:39:13.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:13.098+0000 7f5718c7e700 1 -- 172.21.15.42:0/1576062172 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5714123660 msgr2=0x7f5714123a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:39:13.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:13.098+0000 7f5718c7e700 1 --2- 172.21.15.42:0/1576062172 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5714123660 0x7f5714123a90 secure :-1 s=READY pgs=1225 cs=0 l=1 rx=0x7f571411bbc0 tx=0x7f570c010d80).stop 2019-12-05T05:39:13.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:13.098+0000 7f5718c7e700 1 -- 172.21.15.42:0/1576062172 shutdown_connections 2019-12-05T05:39:13.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:13.098+0000 7f5718c7e700 1 --2- 172.21.15.42:0/1576062172 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f56f803ae90 0x7f56f803d3a0 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:39:13.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:13.098+0000 7f5718c7e700 1 --2- 172.21.15.42:0/1576062172 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5714123660 0x7f5714123a90 unknown :-1 s=CLOSED pgs=1225 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:39:13.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:13.098+0000 7f5718c7e700 1 --2- 172.21.15.42:0/1576062172 >> v2:172.21.15.40:3300/0 conn(0x7f5714121180 0x7f571407c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:39:13.103 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:13.098+0000 7f5718c7e700 1 --2- 172.21.15.42:0/1576062172 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f571407c9d0 0x7f5714123110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:39:13.103 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:13.098+0000 7f5718c7e700 1 -- 172.21.15.42:0/1576062172 shutdown_connections 2019-12-05T05:39:13.103 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:13.098+0000 7f5718c7e700 1 -- 172.21.15.42:0/1576062172 wait complete. 2019-12-05T05:39:13.103 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:13.098+0000 7f5718c7e700 1 -- 172.21.15.42:0/1576062172 >> 172.21.15.42:0/1576062172 conn(0x7f571411c380 msgr2=0x7f57140a9770 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:39:13.112 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2019-12-05T05:39:15.113 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:39:15.219 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:15.210+0000 7fae38f39700 1 Processor -- start 2019-12-05T05:39:15.220 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:15.210+0000 7fae38f39700 1 -- start start 2019-12-05T05:39:15.220 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:15.214+0000 7fae38f39700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fae3408df80 0x7fae3409f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:39:15.220 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:15.214+0000 7fae38f39700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fae3409f830 0x7fae3414efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:39:15.220 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:15.214+0000 7fae38f39700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae3409fc60 con 0x7fae340a7740 2019-12-05T05:39:15.220 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:15.214+0000 7fae38f39700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae340a0b00 con 0x7fae3409f830 2019-12-05T05:39:15.221 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:15.214+0000 7fae38f39700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae3408e5b0 con 0x7fae3408df80 2019-12-05T05:39:15.221 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:15.214+0000 7fae32d9d700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fae340a7740 0x7fae3407e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:38122/0 (socket says 172.21.15.42:38122) 2019-12-05T05:39:15.221 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:15.214+0000 7fae32d9d700 1 -- 172.21.15.42:0/3555077298 learned_addr learned my addr 172.21.15.42:0/3555077298 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:39:15.221 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:15.214+0000 7fae3359e700 1 --2- 172.21.15.42:0/3555077298 >> v2:172.21.15.40:3300/0 conn(0x7fae3409f830 0x7fae3414efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:39:15.221 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:15.214+0000 7fae2bfff700 1 --2- 172.21.15.42:0/3555077298 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fae3408df80 0x7fae3409f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:39:15.222 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:15.214+0000 7fae3259c700 1 -- 172.21.15.42:0/3555077298 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 35575645 0 0) 0x7fae3409fc60 con 0x7fae340a7740 2019-12-05T05:39:15.222 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:15.214+0000 7fae3259c700 1 -- 172.21.15.42:0/3555077298 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae200030b0 con 0x7fae340a7740 2019-12-05T05:39:15.222 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:15.214+0000 7fae3359e700 1 -- 172.21.15.42:0/3555077298 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fae3408df80 msgr2=0x7fae3409f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:39:15.222 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:15.214+0000 7fae3359e700 1 --2- 172.21.15.42:0/3555077298 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fae3408df80 0x7fae3409f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:39:15.222 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:15.214+0000 7fae3359e700 1 -- 172.21.15.42:0/3555077298 >> v1:172.21.15.42:6789/0 conn(0x7fae340a7740 legacy=0x7fae3407e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:39:15.222 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:15.214+0000 7fae3359e700 1 -- 172.21.15.42:0/3555077298 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae340a0710 con 0x7fae3409f830 2019-12-05T05:39:15.222 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:15.214+0000 7fae3359e700 1 --2- 172.21.15.42:0/3555077298 >> v2:172.21.15.40:3300/0 conn(0x7fae3409f830 0x7fae3414efe0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fae2c008b10 tx=0x7fae2c00fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:15.223 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:15.214+0000 7fae3359e700 1 --2- 172.21.15.42:0/3555077298 >> v2:172.21.15.40:3300/0 conn(0x7fae3409f830 0x7fae3414efe0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fae2c008b10 tx=0x7fae2c00fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:15.223 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:15.214+0000 7fae3359e700 1 --2- 172.21.15.42:0/3555077298 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fae3409f830 0x7fae3414efe0 secure :-1 s=READY pgs=952 cs=0 l=1 rx=0x7fae2c008b10 tx=0x7fae2c00fff0).ready entity=mon.0 client_cookie=5c95b51b5ef61a5 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:39:15.223 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:15.214+0000 7fae3359e700 1 --2- 172.21.15.42:0/3555077298 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fae3409f830 0x7fae3414efe0 secure :-1 s=THROTTLE_DONE pgs=952 cs=0 l=1 rx=0x7fae2c008b10 tx=0x7fae2c00fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:15.223 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:15.214+0000 7fae3259c700 1 -- 172.21.15.42:0/3555077298 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fae2c010900 con 0x7fae3409f830 2019-12-05T05:39:15.223 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:15.214+0000 7fae3359e700 1 --2- 172.21.15.42:0/3555077298 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fae3409f830 0x7fae3414efe0 secure :-1 s=THROTTLE_DONE pgs=952 cs=0 l=1 rx=0x7fae2c008b10 tx=0x7fae2c00fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:15.223 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:15.214+0000 7fae3359e700 1 --2- 172.21.15.42:0/3555077298 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fae3409f830 0x7fae3414efe0 secure :-1 s=THROTTLE_DONE pgs=952 cs=0 l=1 rx=0x7fae2c008b10 tx=0x7fae2c00fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:15.223 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:15.214+0000 7fae3259c700 1 -- 172.21.15.42:0/3555077298 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae2c008b50 con 0x7fae3409f830 2019-12-05T05:39:15.224 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:15.214+0000 7fae3259c700 1 -- 172.21.15.42:0/3555077298 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fae2c0110c0 con 0x7fae3409f830 2019-12-05T05:39:15.224 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:15.214+0000 7fae38f39700 1 -- 172.21.15.42:0/3555077298 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fae3409f830 msgr2=0x7fae3414efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:39:15.224 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:15.214+0000 7fae38f39700 1 --2- 172.21.15.42:0/3555077298 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fae3409f830 0x7fae3414efe0 secure :-1 s=READY pgs=952 cs=0 l=1 rx=0x7fae2c008b10 tx=0x7fae2c00fff0).stop 2019-12-05T05:39:15.224 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:15.214+0000 7fae38f39700 1 -- 172.21.15.42:0/3555077298 shutdown_connections 2019-12-05T05:39:15.224 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:15.214+0000 7fae38f39700 1 --2- 172.21.15.42:0/3555077298 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fae3409f830 0x7fae3414efe0 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:39:15.224 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:15.214+0000 7fae38f39700 1 --2- 172.21.15.42:0/3555077298 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fae3408df80 0x7fae3409f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:39:15.225 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:15.214+0000 7fae38f39700 1 -- 172.21.15.42:0/3555077298 shutdown_connections 2019-12-05T05:39:15.225 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:15.214+0000 7fae38f39700 1 -- 172.21.15.42:0/3555077298 wait complete. 2019-12-05T05:39:15.225 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:15.214+0000 7fae38f39700 1 -- 172.21.15.42:0/3555077298 >> 172.21.15.42:0/3555077298 conn(0x7fae340aabe0 msgr2=0x7fae340a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:39:15.228 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:15.218+0000 7fae38f39700 1 Processor -- start 2019-12-05T05:39:15.229 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:15.218+0000 7fae38f39700 1 -- start start 2019-12-05T05:39:15.229 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:15.218+0000 7fae38f39700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fae341434a0 0x7fae341191f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:39:15.229 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:15.218+0000 7fae38f39700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae3407bfa0 con 0x7fae341482e0 2019-12-05T05:39:15.229 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:15.218+0000 7fae38f39700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae34136580 con 0x7fae34129cd0 2019-12-05T05:39:15.229 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:15.218+0000 7fae38f39700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae341454c0 con 0x7fae341434a0 2019-12-05T05:39:15.229 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:15.218+0000 7fae3359e700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fae34129cd0 0x7fae3412a100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:38132/0 (socket says 172.21.15.42:38132) 2019-12-05T05:39:15.230 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:15.218+0000 7fae3359e700 1 -- 172.21.15.42:0/1920496571 learned_addr learned my addr 172.21.15.42:0/1920496571 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:39:15.230 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:15.218+0000 7fae32d9d700 1 --2- 172.21.15.42:0/1920496571 >> v2:172.21.15.40:3300/0 conn(0x7fae341434a0 0x7fae341191f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:39:15.230 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:15.218+0000 7fae2b7fe700 1 -- 172.21.15.42:0/1920496571 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2044923769 0 0) 0x7fae34136580 con 0x7fae34129cd0 2019-12-05T05:39:15.230 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:15.218+0000 7fae2b7fe700 1 -- 172.21.15.42:0/1920496571 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae180030b0 con 0x7fae34129cd0 2019-12-05T05:39:15.230 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:15.218+0000 7fae2b7fe700 1 -- 172.21.15.42:0/1920496571 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1658458306 0 0) 0x7fae3407bfa0 con 0x7fae341482e0 2019-12-05T05:39:15.230 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:15.218+0000 7fae32d9d700 1 -- 172.21.15.42:0/1920496571 >> v1:172.21.15.42:6789/0 conn(0x7fae34129cd0 legacy=0x7fae3412a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:39:15.230 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:15.218+0000 7fae32d9d700 1 -- 172.21.15.42:0/1920496571 >> v1:172.21.15.40:6789/0 conn(0x7fae341482e0 legacy=0x7fae3407b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:39:15.231 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:15.218+0000 7fae32d9d700 1 -- 172.21.15.42:0/1920496571 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae34148710 con 0x7fae341434a0 2019-12-05T05:39:15.231 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:15.218+0000 7fae32d9d700 1 --2- 172.21.15.42:0/1920496571 >> v2:172.21.15.40:3300/0 conn(0x7fae341434a0 0x7fae341191f0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fae240043a0 tx=0x7fae24010860).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:15.231 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:15.222+0000 7fae32d9d700 1 --2- 172.21.15.42:0/1920496571 >> v2:172.21.15.40:3300/0 conn(0x7fae341434a0 0x7fae341191f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fae240043a0 tx=0x7fae24010860).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:15.231 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:15.222+0000 7fae32d9d700 1 --2- 172.21.15.42:0/1920496571 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fae341434a0 0x7fae341191f0 secure :-1 s=READY pgs=954 cs=0 l=1 rx=0x7fae240043a0 tx=0x7fae24010860).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:39:15.231 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:15.222+0000 7fae32d9d700 1 --2- 172.21.15.42:0/1920496571 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fae341434a0 0x7fae341191f0 secure :-1 s=THROTTLE_DONE pgs=954 cs=0 l=1 rx=0x7fae240043a0 tx=0x7fae24010860).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:15.232 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:15.222+0000 7fae2b7fe700 1 -- 172.21.15.42:0/1920496571 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fae24005c30 con 0x7fae341434a0 2019-12-05T05:39:15.232 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:15.222+0000 7fae32d9d700 1 --2- 172.21.15.42:0/1920496571 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fae341434a0 0x7fae341191f0 secure :-1 s=THROTTLE_DONE pgs=954 cs=0 l=1 rx=0x7fae240043a0 tx=0x7fae24010860).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:15.232 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:15.222+0000 7fae38f39700 1 -- 172.21.15.42:0/1920496571 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae34135570 con 0x7fae341434a0 2019-12-05T05:39:15.232 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:15.222+0000 7fae2b7fe700 1 -- 172.21.15.42:0/1920496571 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae24005850 con 0x7fae341434a0 2019-12-05T05:39:15.232 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:15.222+0000 7fae32d9d700 1 --2- 172.21.15.42:0/1920496571 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fae341434a0 0x7fae341191f0 secure :-1 s=THROTTLE_DONE pgs=954 cs=0 l=1 rx=0x7fae240043a0 tx=0x7fae24010860).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:15.232 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:15.222+0000 7fae2b7fe700 1 -- 172.21.15.42:0/1920496571 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fae240064a0 con 0x7fae341434a0 2019-12-05T05:39:15.232 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:15.222+0000 7fae38f39700 1 -- 172.21.15.42:0/1920496571 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae34135980 con 0x7fae341434a0 2019-12-05T05:39:15.233 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:15.222+0000 7fae32d9d700 1 --2- 172.21.15.42:0/1920496571 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fae341434a0 0x7fae341191f0 secure :-1 s=THROTTLE_DONE pgs=954 cs=0 l=1 rx=0x7fae240043a0 tx=0x7fae24010860).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:15.233 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:15.222+0000 7fae2b7fe700 1 -- 172.21.15.42:0/1920496571 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fae24006730 con 0x7fae341434a0 2019-12-05T05:39:15.233 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:15.222+0000 7fae32d9d700 1 --2- 172.21.15.42:0/1920496571 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fae341434a0 0x7fae341191f0 secure :-1 s=THROTTLE_DONE pgs=954 cs=0 l=1 rx=0x7fae240043a0 tx=0x7fae24010860).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:15.233 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:15.222+0000 7fae2b7fe700 1 --2- 172.21.15.42:0/1920496571 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fae1803b210 0x7fae1803d720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:39:15.233 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:15.222+0000 7fae2b7fe700 1 -- 172.21.15.42:0/1920496571 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 8095+0+0 (secure 0 0 0) 0x7fae2404af00 con 0x7fae341434a0 2019-12-05T05:39:15.233 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:15.222+0000 7fae2bfff700 1 --2- 172.21.15.42:0/1920496571 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fae1803b210 0x7fae1803d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:39:15.233 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:15.222+0000 7fae2bfff700 1 --2- 172.21.15.42:0/1920496571 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fae1803b210 0x7fae1803d720 crc :-1 s=READY pgs=530 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:39:15.234 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:15.226+0000 7fae38f39700 1 -- 172.21.15.42:0/1920496571 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae3405d110 con 0x7fae341434a0 2019-12-05T05:39:15.238 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:15.234+0000 7fae32d9d700 1 --2- 172.21.15.42:0/1920496571 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fae341434a0 0x7fae341191f0 secure :-1 s=THROTTLE_DONE pgs=954 cs=0 l=1 rx=0x7fae240043a0 tx=0x7fae24010860).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:15.238 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:15.234+0000 7fae2b7fe700 1 -- 172.21.15.42:0/1920496571 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fae24031f60 con 0x7fae341434a0 2019-12-05T05:39:15.413 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:15.410+0000 7fae38f39700 1 -- 172.21.15.42:0/1920496571 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fae3405b8c0 con 0x7fae341434a0 2019-12-05T05:39:15.414 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:15.410+0000 7fae32d9d700 1 --2- 172.21.15.42:0/1920496571 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fae341434a0 0x7fae341191f0 secure :-1 s=THROTTLE_DONE pgs=954 cs=0 l=1 rx=0x7fae240043a0 tx=0x7fae24010860).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:15.414 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:15.410+0000 7fae2b7fe700 1 -- 172.21.15.42:0/1920496571 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 152 v152) v1 ==== 95+0+1772 (secure 0 0 0) 0x7fae240321c0 con 0x7fae341434a0 2019-12-05T05:39:15.416 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 152 2019-12-05T05:39:15.416 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:39:15.417 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":152,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":152,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:39:14.336435+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7416},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7416":{"gid":7416,"name":"b","rank":0,"incarnation":149,"state":"up:active","state_seq":30,"addr":"172.21.15.40:6833/123054133","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":123054133},{"type":"v1","addr":"172.21.15.40:6833","nonce":123054133}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:39:15.419 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:15.414+0000 7fae38f39700 1 -- 172.21.15.42:0/1920496571 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fae1803b210 msgr2=0x7fae1803d720 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:39:15.420 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:15.414+0000 7fae38f39700 1 --2- 172.21.15.42:0/1920496571 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fae1803b210 0x7fae1803d720 crc :-1 s=READY pgs=530 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:39:15.420 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:15.414+0000 7fae38f39700 1 -- 172.21.15.42:0/1920496571 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fae341434a0 msgr2=0x7fae341191f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:39:15.420 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:15.414+0000 7fae38f39700 1 --2- 172.21.15.42:0/1920496571 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fae341434a0 0x7fae341191f0 secure :-1 s=READY pgs=954 cs=0 l=1 rx=0x7fae240043a0 tx=0x7fae24010860).stop 2019-12-05T05:39:15.420 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:15.414+0000 7fae38f39700 1 -- 172.21.15.42:0/1920496571 shutdown_connections 2019-12-05T05:39:15.420 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:15.414+0000 7fae38f39700 1 --2- 172.21.15.42:0/1920496571 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fae1803b210 0x7fae1803d720 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:39:15.420 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:15.414+0000 7fae38f39700 1 --2- 172.21.15.42:0/1920496571 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fae341434a0 0x7fae341191f0 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:39:15.421 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:15.414+0000 7fae38f39700 1 -- 172.21.15.42:0/1920496571 shutdown_connections 2019-12-05T05:39:15.421 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:15.414+0000 7fae38f39700 1 -- 172.21.15.42:0/1920496571 wait complete. 2019-12-05T05:39:15.421 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:15.414+0000 7fae38f39700 1 -- 172.21.15.42:0/1920496571 >> 172.21.15.42:0/1920496571 conn(0x7fae340aabe0 msgr2=0x7fae3411c470 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:39:15.429 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b has gained rank=0, replacing gid=5693 2019-12-05T05:39:15.430 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 1 secs before reviving mds.a 2019-12-05T05:39:16.430 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.a 2019-12-05T05:39:16.430 INFO:tasks.ceph.mds.a:Restarting daemon 2019-12-05T05:39:16.430 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2019-12-05T05:39:16.441 INFO:tasks.ceph.mds.a:Started 2019-12-05T05:39:16.441 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:39:16.489 INFO:tasks.ceph.mds.a.smithi042.stdout:starting mds.a at 2019-12-05T05:39:16.554 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:16.546+0000 7f69f60fe700 1 Processor -- start 2019-12-05T05:39:16.555 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:16.546+0000 7f69f60fe700 1 -- start start 2019-12-05T05:39:16.555 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:16.546+0000 7f69f60fe700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f69f008df70 0x7f69f009f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:39:16.555 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:16.546+0000 7f69f60fe700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69f009fc50 con 0x7f69f009f820 2019-12-05T05:39:16.555 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:16.546+0000 7f69f60fe700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69f00a0700 con 0x7f69f00a7730 2019-12-05T05:39:16.555 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:16.546+0000 7f69f60fe700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69f00a0af0 con 0x7f69f008df70 2019-12-05T05:39:16.556 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:16.546+0000 7f69effff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f69f00a7730 0x7f69f007e710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:38152/0 (socket says 172.21.15.42:38152) 2019-12-05T05:39:16.556 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:16.546+0000 7f69effff700 1 -- 172.21.15.42:0/4175045915 learned_addr learned my addr 172.21.15.42:0/4175045915 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:39:16.556 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:16.546+0000 7f69ef7fe700 1 --2- 172.21.15.42:0/4175045915 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f69f008df70 0x7f69f009f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:39:16.556 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:16.546+0000 7f69eeffd700 1 -- 172.21.15.42:0/4175045915 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1194937522 0 0) 0x7f69f00a0700 con 0x7f69f00a7730 2019-12-05T05:39:16.556 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:16.546+0000 7f69eeffd700 1 -- 172.21.15.42:0/4175045915 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69d80030b0 con 0x7f69f00a7730 2019-12-05T05:39:16.556 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:16.546+0000 7f69eeffd700 1 -- 172.21.15.42:0/4175045915 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1118074983 0 0) 0x7f69f009fc50 con 0x7f69f009f820 2019-12-05T05:39:16.556 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:16.550+0000 7f69eeffd700 1 -- 172.21.15.42:0/4175045915 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69f00a0700 con 0x7f69f009f820 2019-12-05T05:39:16.557 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:16.550+0000 7f69ef7fe700 1 -- 172.21.15.42:0/4175045915 >> v1:172.21.15.42:6789/0 conn(0x7f69f00a7730 legacy=0x7f69f007e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:39:16.557 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:16.550+0000 7f69ef7fe700 1 -- 172.21.15.42:0/4175045915 >> v1:172.21.15.40:6789/0 conn(0x7f69f009f820 legacy=0x7f69f014efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:39:16.557 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:16.550+0000 7f69ef7fe700 1 -- 172.21.15.42:0/4175045915 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69f007ef10 con 0x7f69f008df70 2019-12-05T05:39:16.557 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:16.550+0000 7f69ef7fe700 1 --2- 172.21.15.42:0/4175045915 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f69f008df70 0x7f69f009f2d0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f69e4002f30 tx=0x7f69e400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:16.557 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:16.550+0000 7f69ef7fe700 1 --2- 172.21.15.42:0/4175045915 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f69f008df70 0x7f69f009f2d0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f69e4002f30 tx=0x7f69e400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:16.557 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:16.550+0000 7f69ef7fe700 1 --2- 172.21.15.42:0/4175045915 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f69f008df70 0x7f69f009f2d0 secure :-1 s=READY pgs=255 cs=0 l=1 rx=0x7f69e4002f30 tx=0x7f69e400fbf0).ready entity=mon.2 client_cookie=4bb0e83de7fc3029 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:39:16.565 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:16.550+0000 7f69ef7fe700 1 --2- 172.21.15.42:0/4175045915 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f69f008df70 0x7f69f009f2d0 secure :-1 s=THROTTLE_DONE pgs=255 cs=0 l=1 rx=0x7f69e4002f30 tx=0x7f69e400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:16.565 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:16.550+0000 7f69eeffd700 1 -- 172.21.15.42:0/4175045915 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f69e40112b0 con 0x7f69f008df70 2019-12-05T05:39:16.565 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:16.550+0000 7f69ef7fe700 1 --2- 172.21.15.42:0/4175045915 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f69f008df70 0x7f69f009f2d0 secure :-1 s=THROTTLE_DONE pgs=255 cs=0 l=1 rx=0x7f69e4002f30 tx=0x7f69e400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:16.565 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:16.550+0000 7f69eeffd700 1 -- 172.21.15.42:0/4175045915 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69e4008bc0 con 0x7f69f008df70 2019-12-05T05:39:16.566 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:16.550+0000 7f69ef7fe700 1 --2- 172.21.15.42:0/4175045915 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f69f008df70 0x7f69f009f2d0 secure :-1 s=THROTTLE_DONE pgs=255 cs=0 l=1 rx=0x7f69e4002f30 tx=0x7f69e400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:16.566 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:16.550+0000 7f69eeffd700 1 -- 172.21.15.42:0/4175045915 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f69e4011890 con 0x7f69f008df70 2019-12-05T05:39:16.566 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:16.550+0000 7f69f60fe700 1 -- 172.21.15.42:0/4175045915 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f69f008df70 msgr2=0x7f69f009f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:39:16.566 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:16.550+0000 7f69f60fe700 1 --2- 172.21.15.42:0/4175045915 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f69f008df70 0x7f69f009f2d0 secure :-1 s=READY pgs=255 cs=0 l=1 rx=0x7f69e4002f30 tx=0x7f69e400fbf0).stop 2019-12-05T05:39:16.566 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:16.550+0000 7f69f60fe700 1 -- 172.21.15.42:0/4175045915 shutdown_connections 2019-12-05T05:39:16.566 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:16.550+0000 7f69f60fe700 1 --2- 172.21.15.42:0/4175045915 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f69f008df70 0x7f69f009f2d0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:39:16.567 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:16.550+0000 7f69f60fe700 1 -- 172.21.15.42:0/4175045915 shutdown_connections 2019-12-05T05:39:16.567 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:16.550+0000 7f69f60fe700 1 -- 172.21.15.42:0/4175045915 wait complete. 2019-12-05T05:39:16.567 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:16.550+0000 7f69f60fe700 1 -- 172.21.15.42:0/4175045915 >> 172.21.15.42:0/4175045915 conn(0x7f69f00aabd0 msgr2=0x7f69f00a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:39:16.593 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:16.554+0000 7f69f60fe700 1 Processor -- start 2019-12-05T05:39:16.593 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:16.554+0000 7f69f60fe700 1 -- start start 2019-12-05T05:39:16.593 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:16.554+0000 7f69f60fe700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f69f00a6f10 0x7f69f00a59d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:39:16.593 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:16.554+0000 7f69f60fe700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f69f0067310 0x7f69f0133a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:39:16.594 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:16.554+0000 7f69f60fe700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69f0142520 con 0x7f69f0133fb0 2019-12-05T05:39:16.594 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:16.554+0000 7f69f60fe700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69f00a6000 con 0x7f69f0067310 2019-12-05T05:39:16.594 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:16.554+0000 7f69f60fe700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69f00a62a0 con 0x7f69f00a6f10 2019-12-05T05:39:16.594 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:16.554+0000 7f69f4e9c700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f69f0133fb0 0x7f69f0129d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:38162/0 (socket says 172.21.15.42:38162) 2019-12-05T05:39:16.594 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:16.554+0000 7f69f4e9c700 1 -- 172.21.15.42:0/3683725682 learned_addr learned my addr 172.21.15.42:0/3683725682 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:39:16.594 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:16.554+0000 7f69ef7fe700 1 --2- 172.21.15.42:0/3683725682 >> v2:172.21.15.40:3300/0 conn(0x7f69f0067310 0x7f69f0133a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:39:16.594 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:16.554+0000 7f69effff700 1 --2- 172.21.15.42:0/3683725682 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f69f00a6f10 0x7f69f00a59d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:39:16.595 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:16.554+0000 7f69ecff9700 1 -- 172.21.15.42:0/3683725682 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1848211121 0 0) 0x7f69f0142520 con 0x7f69f0133fb0 2019-12-05T05:39:16.595 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:16.554+0000 7f69ecff9700 1 -- 172.21.15.42:0/3683725682 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69d00030b0 con 0x7f69f0133fb0 2019-12-05T05:39:16.595 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:16.554+0000 7f69ef7fe700 1 -- 172.21.15.42:0/3683725682 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f69f00a6f10 msgr2=0x7f69f00a59d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:39:16.595 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:16.554+0000 7f69ef7fe700 1 --2- 172.21.15.42:0/3683725682 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f69f00a6f10 0x7f69f00a59d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:39:16.595 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:16.554+0000 7f69ecff9700 1 -- 172.21.15.42:0/3683725682 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 4208639485 0 0) 0x7f69d00030b0 con 0x7f69f0133fb0 2019-12-05T05:39:16.595 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:16.554+0000 7f69ef7fe700 1 -- 172.21.15.42:0/3683725682 >> v1:172.21.15.42:6789/0 conn(0x7f69f0133fb0 legacy=0x7f69f0129d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:39:16.596 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:16.554+0000 7f69ef7fe700 1 -- 172.21.15.42:0/3683725682 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69f01343e0 con 0x7f69f0067310 2019-12-05T05:39:16.596 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:16.554+0000 7f69effff700 1 --2- 172.21.15.42:0/3683725682 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f69f00a6f10 0x7f69f00a59d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:39:16.596 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:16.554+0000 7f69ef7fe700 1 --2- 172.21.15.42:0/3683725682 >> v2:172.21.15.40:3300/0 conn(0x7f69f0067310 0x7f69f0133a60 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f69e8001ef0 tx=0x7f69e8001f30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:16.596 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:16.554+0000 7f69ef7fe700 1 --2- 172.21.15.42:0/3683725682 >> v2:172.21.15.40:3300/0 conn(0x7f69f0067310 0x7f69f0133a60 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f69e8001ef0 tx=0x7f69e8001f30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:16.596 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:16.554+0000 7f69ef7fe700 1 --2- 172.21.15.42:0/3683725682 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f69f0067310 0x7f69f0133a60 secure :-1 s=READY pgs=958 cs=0 l=1 rx=0x7f69e8001ef0 tx=0x7f69e8001f30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:39:16.596 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:16.554+0000 7f69ef7fe700 1 --2- 172.21.15.42:0/3683725682 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f69f0067310 0x7f69f0133a60 secure :-1 s=THROTTLE_DONE pgs=958 cs=0 l=1 rx=0x7f69e8001ef0 tx=0x7f69e8001f30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:16.596 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:16.554+0000 7f69ecff9700 1 -- 172.21.15.42:0/3683725682 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f69e8001cb0 con 0x7f69f0067310 2019-12-05T05:39:16.597 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:16.554+0000 7f69ef7fe700 1 --2- 172.21.15.42:0/3683725682 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f69f0067310 0x7f69f0133a60 secure :-1 s=THROTTLE_DONE pgs=958 cs=0 l=1 rx=0x7f69e8001ef0 tx=0x7f69e8001f30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:16.597 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:16.554+0000 7f69ef7fe700 1 --2- 172.21.15.42:0/3683725682 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f69f0067310 0x7f69f0133a60 secure :-1 s=THROTTLE_DONE pgs=958 cs=0 l=1 rx=0x7f69e8001ef0 tx=0x7f69e8001f30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:16.597 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:16.554+0000 7f69ecff9700 1 -- 172.21.15.42:0/3683725682 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69e8001ac0 con 0x7f69f0067310 2019-12-05T05:39:16.597 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:16.554+0000 7f69f60fe700 1 -- 172.21.15.42:0/3683725682 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69f012a570 con 0x7f69f0067310 2019-12-05T05:39:16.597 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:16.554+0000 7f69ecff9700 1 -- 172.21.15.42:0/3683725682 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f69e4011b70 con 0x7f69f0067310 2019-12-05T05:39:16.597 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:16.554+0000 7f69f60fe700 1 -- 172.21.15.42:0/3683725682 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69f00762e0 con 0x7f69f0067310 2019-12-05T05:39:16.597 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:16.558+0000 7f69ef7fe700 1 --2- 172.21.15.42:0/3683725682 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f69f0067310 0x7f69f0133a60 secure :-1 s=THROTTLE_DONE pgs=958 cs=0 l=1 rx=0x7f69e8001ef0 tx=0x7f69e8001f30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:16.598 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:16.558+0000 7f69ecff9700 1 -- 172.21.15.42:0/3683725682 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f69e400f360 con 0x7f69f0067310 2019-12-05T05:39:16.598 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:16.558+0000 7f69ef7fe700 1 --2- 172.21.15.42:0/3683725682 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f69f0067310 0x7f69f0133a60 secure :-1 s=THROTTLE_DONE pgs=958 cs=0 l=1 rx=0x7f69e8001ef0 tx=0x7f69e8001f30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:16.598 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:16.558+0000 7f69ecff9700 1 --2- 172.21.15.42:0/3683725682 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f69d003b1a0 0x7f69d003d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:39:16.598 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:16.558+0000 7f69ecff9700 1 -- 172.21.15.42:0/3683725682 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 8095+0+0 (secure 0 0 0) 0x7f69e404b080 con 0x7f69f0067310 2019-12-05T05:39:16.598 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:16.558+0000 7f69effff700 1 --2- 172.21.15.42:0/3683725682 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f69d003b1a0 0x7f69d003d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:39:16.598 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:16.558+0000 7f69effff700 1 --2- 172.21.15.42:0/3683725682 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f69d003b1a0 0x7f69d003d6b0 crc :-1 s=READY pgs=531 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:39:16.598 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:16.562+0000 7f69f60fe700 1 -- 172.21.15.42:0/3683725682 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69f005d100 con 0x7f69f0067310 2019-12-05T05:39:16.599 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:16.570+0000 7f69ef7fe700 1 --2- 172.21.15.42:0/3683725682 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f69f0067310 0x7f69f0133a60 secure :-1 s=THROTTLE_DONE pgs=958 cs=0 l=1 rx=0x7f69e8001ef0 tx=0x7f69e8001f30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:16.599 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:16.570+0000 7f69ecff9700 1 -- 172.21.15.42:0/3683725682 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f69e40320e0 con 0x7f69f0067310 2019-12-05T05:39:16.754 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:16.750+0000 7f69f60fe700 1 -- 172.21.15.42:0/3683725682 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f69f005b8b0 con 0x7f69f0067310 2019-12-05T05:39:16.755 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:16.750+0000 7f69ef7fe700 1 --2- 172.21.15.42:0/3683725682 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f69f0067310 0x7f69f0133a60 secure :-1 s=THROTTLE_DONE pgs=958 cs=0 l=1 rx=0x7f69e8001ef0 tx=0x7f69e8001f30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:16.755 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:16.750+0000 7f69ecff9700 1 -- 172.21.15.42:0/3683725682 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 152 v152) v1 ==== 95+0+1772 (secure 0 0 0) 0x7f69e4032340 con 0x7f69f0067310 2019-12-05T05:39:16.755 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 152 2019-12-05T05:39:16.755 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:39:16.755 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":152,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":152,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:39:14.336435+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7416},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7416":{"gid":7416,"name":"b","rank":0,"incarnation":149,"state":"up:active","state_seq":30,"addr":"172.21.15.40:6833/123054133","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":123054133},{"type":"v1","addr":"172.21.15.40:6833","nonce":123054133}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:39:16.757 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:16.754+0000 7f69f60fe700 1 -- 172.21.15.42:0/3683725682 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f69d003b1a0 msgr2=0x7f69d003d6b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:39:16.757 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:16.754+0000 7f69f60fe700 1 --2- 172.21.15.42:0/3683725682 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f69d003b1a0 0x7f69d003d6b0 crc :-1 s=READY pgs=531 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:39:16.758 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:16.754+0000 7f69f60fe700 1 -- 172.21.15.42:0/3683725682 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f69f0067310 msgr2=0x7f69f0133a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:39:16.758 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:16.754+0000 7f69f60fe700 1 --2- 172.21.15.42:0/3683725682 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f69f0067310 0x7f69f0133a60 secure :-1 s=READY pgs=958 cs=0 l=1 rx=0x7f69e8001ef0 tx=0x7f69e8001f30).stop 2019-12-05T05:39:16.758 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:16.754+0000 7f69f60fe700 1 -- 172.21.15.42:0/3683725682 shutdown_connections 2019-12-05T05:39:16.758 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:16.754+0000 7f69f60fe700 1 --2- 172.21.15.42:0/3683725682 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f69d003b1a0 0x7f69d003d6b0 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:39:16.758 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:16.754+0000 7f69f60fe700 1 --2- 172.21.15.42:0/3683725682 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f69f00a6f10 0x7f69f00a59d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:39:16.758 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:16.754+0000 7f69f60fe700 1 --2- 172.21.15.42:0/3683725682 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f69f0067310 0x7f69f0133a60 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:39:16.758 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:16.754+0000 7f69f60fe700 1 -- 172.21.15.42:0/3683725682 shutdown_connections 2019-12-05T05:39:16.759 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:16.754+0000 7f69f60fe700 1 -- 172.21.15.42:0/3683725682 wait complete. 2019-12-05T05:39:16.759 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:16.754+0000 7f69f60fe700 1 -- 172.21.15.42:0/3683725682 >> 172.21.15.42:0/3683725682 conn(0x7f69f007dd00 msgr2=0x7f69f011ce40 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:39:16.772 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2019-12-05T05:39:18.774 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:39:18.885 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:18.870+0000 7eff2cb20700 1 Processor -- start 2019-12-05T05:39:18.885 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:18.870+0000 7eff2cb20700 1 -- start start 2019-12-05T05:39:18.886 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:18.874+0000 7eff2cb20700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7eff2809fae0 0x7eff2814fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:39:18.886 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:18.874+0000 7eff2cb20700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff280a0710 con 0x7eff280a7740 2019-12-05T05:39:18.886 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:18.874+0000 7eff2cb20700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff2807ef20 con 0x7eff2808df80 2019-12-05T05:39:18.886 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:18.874+0000 7eff2cb20700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff280a0b00 con 0x7eff2809fae0 2019-12-05T05:39:18.887 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:18.874+0000 7eff26d9d700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7eff280a7740 0x7eff2807e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:47626/0 (socket says 172.21.15.42:47626) 2019-12-05T05:39:18.887 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:18.874+0000 7eff2659c700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7eff2808df80 0x7eff2809f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:38188/0 (socket says 172.21.15.42:38188) 2019-12-05T05:39:18.887 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:18.874+0000 7eff26d9d700 1 -- 172.21.15.42:0/2015592733 learned_addr learned my addr 172.21.15.42:0/2015592733 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:39:18.887 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:18.874+0000 7eff2759e700 1 --2- 172.21.15.42:0/2015592733 >> v2:172.21.15.40:3300/0 conn(0x7eff2809fae0 0x7eff2814fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:39:18.887 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:18.874+0000 7eff25d9b700 1 -- 172.21.15.42:0/2015592733 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2584753287 0 0) 0x7eff2807ef20 con 0x7eff2808df80 2019-12-05T05:39:18.888 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:18.874+0000 7eff25d9b700 1 -- 172.21.15.42:0/2015592733 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff100030b0 con 0x7eff2808df80 2019-12-05T05:39:18.888 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:18.874+0000 7eff25d9b700 1 -- 172.21.15.42:0/2015592733 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1238278177 0 0) 0x7eff280a0710 con 0x7eff280a7740 2019-12-05T05:39:18.888 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:18.874+0000 7eff25d9b700 1 -- 172.21.15.42:0/2015592733 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff2807ef20 con 0x7eff280a7740 2019-12-05T05:39:18.888 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:18.874+0000 7eff25d9b700 1 -- 172.21.15.42:0/2015592733 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2459105238 0 0) 0x7eff100030b0 con 0x7eff2808df80 2019-12-05T05:39:18.888 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:18.874+0000 7eff25d9b700 1 -- 172.21.15.42:0/2015592733 >> v2:172.21.15.40:3300/0 conn(0x7eff2809fae0 msgr2=0x7eff2814fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:39:18.888 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:18.874+0000 7eff25d9b700 1 --2- 172.21.15.42:0/2015592733 >> v2:172.21.15.40:3300/0 conn(0x7eff2809fae0 0x7eff2814fff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:39:18.889 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:18.874+0000 7eff25d9b700 1 -- 172.21.15.42:0/2015592733 >> v1:172.21.15.40:6789/0 conn(0x7eff280a7740 legacy=0x7eff2807e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:39:18.889 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:18.874+0000 7eff25d9b700 1 -- 172.21.15.42:0/2015592733 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff280a9ba0 con 0x7eff2808df80 2019-12-05T05:39:18.889 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:18.874+0000 7eff25d9b700 1 -- 172.21.15.42:0/2015592733 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7eff1c001680 con 0x7eff2808df80 2019-12-05T05:39:18.891 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:18.874+0000 7eff2759e700 1 --2- 172.21.15.42:0/2015592733 >> v2:172.21.15.40:3300/0 conn(0x7eff2809fae0 0x7eff2814fff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:39:18.892 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:18.874+0000 7eff25d9b700 1 -- 172.21.15.42:0/2015592733 >> v1:172.21.15.42:6789/0 conn(0x7eff2808df80 legacy=0x7eff2809f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:39:18.892 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:18.874+0000 7eff25d9b700 1 --2- 172.21.15.42:0/2015592733 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7eff10003960 0x7eff10003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:39:18.892 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:18.874+0000 7eff25d9b700 1 --2- 172.21.15.42:0/2015592733 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7eff10004470 0x7eff10004940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:39:18.892 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:18.874+0000 7eff25d9b700 1 --2- 172.21.15.42:0/2015592733 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7eff10004fa0 0x7eff1004f700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:39:18.892 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:18.874+0000 7eff25d9b700 1 -- 172.21.15.42:0/2015592733 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff280a0b00 con 0x7eff10004fa0 2019-12-05T05:39:18.892 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:18.874+0000 7eff25d9b700 1 -- 172.21.15.42:0/2015592733 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff1004fec0 con 0x7eff10004470 2019-12-05T05:39:18.893 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:18.874+0000 7eff25d9b700 1 -- 172.21.15.42:0/2015592733 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff10050100 con 0x7eff10003960 2019-12-05T05:39:18.893 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:18.874+0000 7eff26d9d700 1 --2- 172.21.15.42:0/2015592733 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7eff10003960 0x7eff10003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:39:18.893 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:18.874+0000 7eff2659c700 1 --2- 172.21.15.42:0/2015592733 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7eff10004470 0x7eff10004940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:39:18.893 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:18.874+0000 7eff2759e700 1 --2- 172.21.15.42:0/2015592733 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7eff10004fa0 0x7eff1004f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:39:18.893 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:18.874+0000 7eff2659c700 1 -- 172.21.15.42:0/2015592733 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7eff10003960 msgr2=0x7eff10003db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:39:18.893 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:18.874+0000 7eff2659c700 1 --2- 172.21.15.42:0/2015592733 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7eff10003960 0x7eff10003db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:39:18.893 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:18.874+0000 7eff2659c700 1 -- 172.21.15.42:0/2015592733 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7eff10004fa0 msgr2=0x7eff1004f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:39:18.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:18.874+0000 7eff2659c700 1 --2- 172.21.15.42:0/2015592733 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7eff10004fa0 0x7eff1004f700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:39:18.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:18.874+0000 7eff2759e700 1 -- 172.21.15.42:0/2015592733 reap_dead start 2019-12-05T05:39:18.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:18.874+0000 7eff2659c700 1 -- 172.21.15.42:0/2015592733 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff10050400 con 0x7eff10004470 2019-12-05T05:39:18.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:18.874+0000 7eff2659c700 1 --2- 172.21.15.42:0/2015592733 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7eff10004470 0x7eff10004940 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7eff1c001450 tx=0x7eff1c00bfd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:18.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:18.878+0000 7eff2659c700 1 --2- 172.21.15.42:0/2015592733 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7eff10004470 0x7eff10004940 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7eff1c001450 tx=0x7eff1c00bfd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:18.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:18.878+0000 7eff2659c700 1 --2- 172.21.15.42:0/2015592733 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7eff10004470 0x7eff10004940 secure :-1 s=READY pgs=1234 cs=0 l=1 rx=0x7eff1c001450 tx=0x7eff1c00bfd0).ready entity=mon.1 client_cookie=e8eb43d9b7bdaa6c server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:39:18.895 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:18.878+0000 7eff2659c700 1 --2- 172.21.15.42:0/2015592733 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7eff10004470 0x7eff10004940 secure :-1 s=THROTTLE_DONE pgs=1234 cs=0 l=1 rx=0x7eff1c001450 tx=0x7eff1c00bfd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:18.895 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:18.878+0000 7eff25d9b700 1 -- 172.21.15.42:0/2015592733 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7eff1c00ea10 con 0x7eff10004470 2019-12-05T05:39:18.895 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:18.878+0000 7eff2659c700 1 --2- 172.21.15.42:0/2015592733 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7eff10004470 0x7eff10004940 secure :-1 s=THROTTLE_DONE pgs=1234 cs=0 l=1 rx=0x7eff1c001450 tx=0x7eff1c00bfd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:18.895 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:18.878+0000 7eff2659c700 1 --2- 172.21.15.42:0/2015592733 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7eff10004470 0x7eff10004940 secure :-1 s=THROTTLE_DONE pgs=1234 cs=0 l=1 rx=0x7eff1c001450 tx=0x7eff1c00bfd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:18.895 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:18.878+0000 7eff25d9b700 1 -- 172.21.15.42:0/2015592733 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff1c010b00 con 0x7eff10004470 2019-12-05T05:39:18.895 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:18.878+0000 7eff25d9b700 1 -- 172.21.15.42:0/2015592733 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7eff1c0101a0 con 0x7eff10004470 2019-12-05T05:39:18.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:18.878+0000 7eff2cb20700 1 -- 172.21.15.42:0/2015592733 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7eff10004470 msgr2=0x7eff10004940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:39:18.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:18.878+0000 7eff2cb20700 1 --2- 172.21.15.42:0/2015592733 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7eff10004470 0x7eff10004940 secure :-1 s=READY pgs=1234 cs=0 l=1 rx=0x7eff1c001450 tx=0x7eff1c00bfd0).stop 2019-12-05T05:39:18.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:18.878+0000 7eff2cb20700 1 -- 172.21.15.42:0/2015592733 shutdown_connections 2019-12-05T05:39:18.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:18.878+0000 7eff2cb20700 1 --2- 172.21.15.42:0/2015592733 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7eff10004470 0x7eff10004940 unknown :-1 s=CLOSED pgs=1234 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:39:18.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:18.878+0000 7eff2cb20700 1 -- 172.21.15.42:0/2015592733 shutdown_connections 2019-12-05T05:39:18.897 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:18.878+0000 7eff2cb20700 1 -- 172.21.15.42:0/2015592733 wait complete. 2019-12-05T05:39:18.897 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:18.878+0000 7eff2cb20700 1 -- 172.21.15.42:0/2015592733 >> 172.21.15.42:0/2015592733 conn(0x7eff280aabe0 msgr2=0x7eff280a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:39:18.897 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:18.878+0000 7eff2cb20700 1 Processor -- start 2019-12-05T05:39:18.897 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:18.878+0000 7eff2cb20700 1 -- start start 2019-12-05T05:39:18.897 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:18.882+0000 7eff2cb20700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7eff2809f620 0x7eff28123310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:39:18.897 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:18.882+0000 7eff2cb20700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff281210c0 con 0x7eff280742e0 2019-12-05T05:39:18.898 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:18.882+0000 7eff2cb20700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff28118eb0 con 0x7eff281221e0 2019-12-05T05:39:18.898 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:18.882+0000 7eff2cb20700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff281198d0 con 0x7eff2809f620 2019-12-05T05:39:18.898 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:18.882+0000 7eff26d9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7eff2809f620 0x7eff28123310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:39:18.898 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:18.882+0000 7eff2659c700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7eff281221e0 0x7eff28123860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:38198/0 (socket says 172.21.15.42:38198) 2019-12-05T05:39:18.898 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:18.882+0000 7eff2659c700 1 -- 172.21.15.42:0/814075294 learned_addr learned my addr 172.21.15.42:0/814075294 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:39:18.899 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:18.882+0000 7eff26d9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7eff2809f620 0x7eff28123310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:38160/0 (socket says 172.21.15.42:38160) 2019-12-05T05:39:18.899 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:18.882+0000 7eff177fe700 1 -- 172.21.15.42:0/814075294 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3587872666 0 0) 0x7eff281210c0 con 0x7eff280742e0 2019-12-05T05:39:18.899 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:18.882+0000 7eff26d9d700 1 -- 172.21.15.42:0/814075294 >> v1:172.21.15.42:6789/0 conn(0x7eff281221e0 legacy=0x7eff28123860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:39:18.899 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:18.882+0000 7eff26d9d700 1 -- 172.21.15.42:0/814075294 >> v1:172.21.15.40:6789/0 conn(0x7eff280742e0 legacy=0x7eff28076720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:39:18.899 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:18.882+0000 7eff26d9d700 1 -- 172.21.15.42:0/814075294 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff28119e10 con 0x7eff2809f620 2019-12-05T05:39:18.899 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:18.882+0000 7eff26d9d700 1 --2- 172.21.15.42:0/814075294 >> v2:172.21.15.42:3300/0 conn(0x7eff2809f620 0x7eff28123310 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7eff1c00b920 tx=0x7eff1c00bfd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:18.899 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:18.882+0000 7eff26d9d700 1 --2- 172.21.15.42:0/814075294 >> v2:172.21.15.42:3300/0 conn(0x7eff2809f620 0x7eff28123310 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7eff1c00b920 tx=0x7eff1c00bfd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:18.900 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:18.882+0000 7eff26d9d700 1 --2- 172.21.15.42:0/814075294 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7eff2809f620 0x7eff28123310 secure :-1 s=READY pgs=1236 cs=0 l=1 rx=0x7eff1c00b920 tx=0x7eff1c00bfd0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:39:18.900 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:18.882+0000 7eff26d9d700 1 --2- 172.21.15.42:0/814075294 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7eff2809f620 0x7eff28123310 secure :-1 s=THROTTLE_DONE pgs=1236 cs=0 l=1 rx=0x7eff1c00b920 tx=0x7eff1c00bfd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:18.900 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:18.882+0000 7eff177fe700 1 -- 172.21.15.42:0/814075294 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7eff1800cbc0 con 0x7eff2809f620 2019-12-05T05:39:18.900 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:18.882+0000 7eff26d9d700 1 --2- 172.21.15.42:0/814075294 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7eff2809f620 0x7eff28123310 secure :-1 s=THROTTLE_DONE pgs=1236 cs=0 l=1 rx=0x7eff1c00b920 tx=0x7eff1c00bfd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:18.900 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:18.882+0000 7eff26d9d700 1 --2- 172.21.15.42:0/814075294 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7eff2809f620 0x7eff28123310 secure :-1 s=THROTTLE_DONE pgs=1236 cs=0 l=1 rx=0x7eff1c00b920 tx=0x7eff1c00bfd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:18.901 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:18.882+0000 7eff177fe700 1 -- 172.21.15.42:0/814075294 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff1800c7e0 con 0x7eff2809f620 2019-12-05T05:39:18.901 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:18.882+0000 7eff177fe700 1 -- 172.21.15.42:0/814075294 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7eff18004560 con 0x7eff2809f620 2019-12-05T05:39:18.901 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:18.882+0000 7eff2cb20700 1 -- 172.21.15.42:0/814075294 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff28077f30 con 0x7eff2809f620 2019-12-05T05:39:18.902 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:18.882+0000 7eff2cb20700 1 -- 172.21.15.42:0/814075294 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff280784b0 con 0x7eff2809f620 2019-12-05T05:39:18.902 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:18.882+0000 7eff26d9d700 1 --2- 172.21.15.42:0/814075294 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7eff2809f620 0x7eff28123310 secure :-1 s=THROTTLE_DONE pgs=1236 cs=0 l=1 rx=0x7eff1c00b920 tx=0x7eff1c00bfd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:18.903 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:18.882+0000 7eff177fe700 1 -- 172.21.15.42:0/814075294 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7eff18004760 con 0x7eff2809f620 2019-12-05T05:39:18.903 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:18.882+0000 7eff26d9d700 1 --2- 172.21.15.42:0/814075294 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7eff2809f620 0x7eff28123310 secure :-1 s=THROTTLE_DONE pgs=1236 cs=0 l=1 rx=0x7eff1c00b920 tx=0x7eff1c00bfd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:18.903 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:18.882+0000 7eff177fe700 1 --2- 172.21.15.42:0/814075294 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7eff0c03ac10 0x7eff0c03d120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:39:18.904 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:18.882+0000 7eff177fe700 1 -- 172.21.15.42:0/814075294 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 8095+0+0 (secure 0 0 0) 0x7eff1804ad90 con 0x7eff2809f620 2019-12-05T05:39:18.904 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:18.882+0000 7eff2659c700 1 --2- 172.21.15.42:0/814075294 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7eff0c03ac10 0x7eff0c03d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:39:18.904 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:18.882+0000 7eff2659c700 1 --2- 172.21.15.42:0/814075294 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7eff0c03ac10 0x7eff0c03d120 crc :-1 s=READY pgs=533 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:39:18.904 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:18.886+0000 7eff2cb20700 1 -- 172.21.15.42:0/814075294 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff2805d110 con 0x7eff2809f620 2019-12-05T05:39:18.904 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:18.894+0000 7eff26d9d700 1 --2- 172.21.15.42:0/814075294 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7eff2809f620 0x7eff28123310 secure :-1 s=THROTTLE_DONE pgs=1236 cs=0 l=1 rx=0x7eff1c00b920 tx=0x7eff1c00bfd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:18.904 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:18.894+0000 7eff177fe700 1 -- 172.21.15.42:0/814075294 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7eff18031e10 con 0x7eff2809f620 2019-12-05T05:39:19.057 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:19.054+0000 7eff2cb20700 1 -- 172.21.15.42:0/814075294 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7eff28065bd0 con 0x7eff2809f620 2019-12-05T05:39:19.058 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:19.054+0000 7eff26d9d700 1 --2- 172.21.15.42:0/814075294 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7eff2809f620 0x7eff28123310 secure :-1 s=THROTTLE_DONE pgs=1236 cs=0 l=1 rx=0x7eff1c00b920 tx=0x7eff1c00bfd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:19.058 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:19.054+0000 7eff177fe700 1 -- 172.21.15.42:0/814075294 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 152 v152) v1 ==== 95+0+1772 (secure 0 0 0) 0x7eff18032070 con 0x7eff2809f620 2019-12-05T05:39:19.058 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 152 2019-12-05T05:39:19.059 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:39:19.059 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":152,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":152,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:39:14.336435+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7416},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7416":{"gid":7416,"name":"b","rank":0,"incarnation":149,"state":"up:active","state_seq":30,"addr":"172.21.15.40:6833/123054133","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":123054133},{"type":"v1","addr":"172.21.15.40:6833","nonce":123054133}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:39:19.061 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:19.058+0000 7eff2cb20700 1 -- 172.21.15.42:0/814075294 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7eff0c03ac10 msgr2=0x7eff0c03d120 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:39:19.061 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:19.058+0000 7eff2cb20700 1 --2- 172.21.15.42:0/814075294 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7eff0c03ac10 0x7eff0c03d120 crc :-1 s=READY pgs=533 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:39:19.061 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:19.058+0000 7eff2cb20700 1 -- 172.21.15.42:0/814075294 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7eff2809f620 msgr2=0x7eff28123310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:39:19.061 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:19.058+0000 7eff2cb20700 1 --2- 172.21.15.42:0/814075294 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7eff2809f620 0x7eff28123310 secure :-1 s=READY pgs=1236 cs=0 l=1 rx=0x7eff1c00b920 tx=0x7eff1c00bfd0).stop 2019-12-05T05:39:19.062 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:19.058+0000 7eff2cb20700 1 -- 172.21.15.42:0/814075294 shutdown_connections 2019-12-05T05:39:19.062 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:19.058+0000 7eff2cb20700 1 --2- 172.21.15.42:0/814075294 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7eff0c03ac10 0x7eff0c03d120 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:39:19.062 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:19.058+0000 7eff2cb20700 1 --2- 172.21.15.42:0/814075294 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7eff2809f620 0x7eff28123310 unknown :-1 s=CLOSED pgs=1236 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:39:19.062 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:19.058+0000 7eff2cb20700 1 -- 172.21.15.42:0/814075294 shutdown_connections 2019-12-05T05:39:19.063 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:19.058+0000 7eff2cb20700 1 -- 172.21.15.42:0/814075294 wait complete. 2019-12-05T05:39:19.063 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:19.058+0000 7eff2cb20700 1 -- 172.21.15.42:0/814075294 >> 172.21.15.42:0/814075294 conn(0x7eff280a80b0 msgr2=0x7eff2807e450 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:39:19.070 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2019-12-05T05:39:21.071 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:39:21.184 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:21.174+0000 7f8b4e1ce700 1 Processor -- start 2019-12-05T05:39:21.184 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:21.174+0000 7f8b4e1ce700 1 -- start start 2019-12-05T05:39:21.184 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:21.174+0000 7f8b4e1ce700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8b4808df80 0x7f8b4809f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:39:21.185 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:21.174+0000 7f8b4e1ce700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8b4809f830 0x7f8b4814efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:39:21.185 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:21.174+0000 7f8b4e1ce700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b4809fc60 con 0x7f8b480a7740 2019-12-05T05:39:21.185 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:21.174+0000 7f8b4e1ce700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b480a0b00 con 0x7f8b4809f830 2019-12-05T05:39:21.185 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:21.174+0000 7f8b4e1ce700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b4808e5b0 con 0x7f8b4808df80 2019-12-05T05:39:21.186 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:21.174+0000 7f8b47fff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f8b480a7740 0x7f8b4807e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:38216/0 (socket says 172.21.15.42:38216) 2019-12-05T05:39:21.186 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:21.174+0000 7f8b47fff700 1 -- 172.21.15.42:0/3084558495 learned_addr learned my addr 172.21.15.42:0/3084558495 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:39:21.186 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:21.174+0000 7f8b477fe700 1 --2- 172.21.15.42:0/3084558495 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8b4808df80 0x7f8b4809f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:39:21.186 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:21.174+0000 7f8b4cf6c700 1 --2- 172.21.15.42:0/3084558495 >> v2:172.21.15.40:3300/0 conn(0x7f8b4809f830 0x7f8b4814efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:39:21.186 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:21.174+0000 7f8b46ffd700 1 -- 172.21.15.42:0/3084558495 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3804332115 0 0) 0x7f8b4809fc60 con 0x7f8b480a7740 2019-12-05T05:39:21.186 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:21.174+0000 7f8b46ffd700 1 -- 172.21.15.42:0/3084558495 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b300030b0 con 0x7f8b480a7740 2019-12-05T05:39:21.186 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:21.178+0000 7f8b477fe700 1 -- 172.21.15.42:0/3084558495 >> v2:172.21.15.40:3300/0 conn(0x7f8b4809f830 msgr2=0x7f8b4814efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:39:21.187 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:21.178+0000 7f8b477fe700 1 --2- 172.21.15.42:0/3084558495 >> v2:172.21.15.40:3300/0 conn(0x7f8b4809f830 0x7f8b4814efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:39:21.187 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:21.178+0000 7f8b477fe700 1 -- 172.21.15.42:0/3084558495 >> v1:172.21.15.42:6789/0 conn(0x7f8b480a7740 legacy=0x7f8b4807e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:39:21.187 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:21.178+0000 7f8b477fe700 1 -- 172.21.15.42:0/3084558495 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b480a0710 con 0x7f8b4808df80 2019-12-05T05:39:21.187 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:21.178+0000 7f8b477fe700 1 --2- 172.21.15.42:0/3084558495 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8b4808df80 0x7f8b4809f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f8b3c008790 tx=0x7f8b3c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:21.187 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:21.178+0000 7f8b477fe700 1 --2- 172.21.15.42:0/3084558495 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8b4808df80 0x7f8b4809f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f8b3c008790 tx=0x7f8b3c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:21.187 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:21.178+0000 7f8b477fe700 1 --2- 172.21.15.42:0/3084558495 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8b4808df80 0x7f8b4809f2e0 secure :-1 s=READY pgs=256 cs=0 l=1 rx=0x7f8b3c008790 tx=0x7f8b3c00fc70).ready entity=mon.2 client_cookie=524ebcbf1559a73c server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:39:21.187 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:21.178+0000 7f8b477fe700 1 --2- 172.21.15.42:0/3084558495 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8b4808df80 0x7f8b4809f2e0 secure :-1 s=THROTTLE_DONE pgs=256 cs=0 l=1 rx=0x7f8b3c008790 tx=0x7f8b3c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:21.188 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:21.178+0000 7f8b46ffd700 1 -- 172.21.15.42:0/3084558495 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8b3c008bb0 con 0x7f8b4808df80 2019-12-05T05:39:21.188 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:21.178+0000 7f8b477fe700 1 --2- 172.21.15.42:0/3084558495 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8b4808df80 0x7f8b4809f2e0 secure :-1 s=THROTTLE_DONE pgs=256 cs=0 l=1 rx=0x7f8b3c008790 tx=0x7f8b3c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:21.188 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:21.178+0000 7f8b46ffd700 1 -- 172.21.15.42:0/3084558495 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b3c0087d0 con 0x7f8b4808df80 2019-12-05T05:39:21.188 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:21.178+0000 7f8b477fe700 1 --2- 172.21.15.42:0/3084558495 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8b4808df80 0x7f8b4809f2e0 secure :-1 s=THROTTLE_DONE pgs=256 cs=0 l=1 rx=0x7f8b3c008790 tx=0x7f8b3c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:21.188 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:21.178+0000 7f8b4e1ce700 1 -- 172.21.15.42:0/3084558495 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8b4808df80 msgr2=0x7f8b4809f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:39:21.188 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:21.178+0000 7f8b4e1ce700 1 --2- 172.21.15.42:0/3084558495 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8b4808df80 0x7f8b4809f2e0 secure :-1 s=READY pgs=256 cs=0 l=1 rx=0x7f8b3c008790 tx=0x7f8b3c00fc70).stop 2019-12-05T05:39:21.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:21.178+0000 7f8b46ffd700 1 -- 172.21.15.42:0/3084558495 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8b3c011860 con 0x7f8b4808df80 2019-12-05T05:39:21.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:21.178+0000 7f8b4e1ce700 1 -- 172.21.15.42:0/3084558495 shutdown_connections 2019-12-05T05:39:21.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:21.178+0000 7f8b4e1ce700 1 --2- 172.21.15.42:0/3084558495 >> v2:172.21.15.40:3300/0 conn(0x7f8b4809f830 0x7f8b4814efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:39:21.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:21.178+0000 7f8b4e1ce700 1 --2- 172.21.15.42:0/3084558495 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8b4808df80 0x7f8b4809f2e0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:39:21.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:21.178+0000 7f8b4e1ce700 1 -- 172.21.15.42:0/3084558495 shutdown_connections 2019-12-05T05:39:21.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:21.178+0000 7f8b4e1ce700 1 -- 172.21.15.42:0/3084558495 wait complete. 2019-12-05T05:39:21.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:21.178+0000 7f8b4e1ce700 1 -- 172.21.15.42:0/3084558495 >> 172.21.15.42:0/3084558495 conn(0x7f8b480aabe0 msgr2=0x7f8b480a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:39:21.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:21.178+0000 7f8b4e1ce700 1 Processor -- start 2019-12-05T05:39:21.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:21.178+0000 7f8b4e1ce700 1 -- start start 2019-12-05T05:39:21.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:21.182+0000 7f8b4e1ce700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8b480a5f10 0x7f8b48142bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:39:21.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:21.182+0000 7f8b4e1ce700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f8b480a1bc0 0x7f8b480a1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:39:21.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:21.182+0000 7f8b4e1ce700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b480a2540 con 0x7f8b4811b140 2019-12-05T05:39:21.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:21.182+0000 7f8b4e1ce700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b480a3260 con 0x7f8b480a1bc0 2019-12-05T05:39:21.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:21.182+0000 7f8b4e1ce700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b480a3500 con 0x7f8b480a5f10 2019-12-05T05:39:21.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:21.182+0000 7f8b47fff700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8b480a5f10 0x7f8b48142bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:39:21.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:21.182+0000 7f8b477fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f8b480a1bc0 0x7f8b480a1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:39:21.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:21.182+0000 7f8b4cf6c700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f8b4811b140 0x7f8b4811b570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:47666/0 (socket says 172.21.15.42:47666) 2019-12-05T05:39:21.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:21.182+0000 7f8b4cf6c700 1 -- 172.21.15.42:0/2908158084 learned_addr learned my addr 172.21.15.42:0/2908158084 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:39:21.194 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:21.182+0000 7f8b477fe700 1 -- 172.21.15.42:0/2908158084 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8b480a5f10 msgr2=0x7f8b48142bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:39:21.194 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:21.182+0000 7f8b477fe700 1 --2- 172.21.15.42:0/2908158084 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8b480a5f10 0x7f8b48142bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:39:21.194 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:21.182+0000 7f8b477fe700 1 -- 172.21.15.42:0/2908158084 >> v1:172.21.15.40:6789/0 conn(0x7f8b4811b140 legacy=0x7f8b4811b570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:39:21.194 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:21.182+0000 7f8b477fe700 1 -- 172.21.15.42:0/2908158084 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b4811bd70 con 0x7f8b480a1bc0 2019-12-05T05:39:21.195 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:21.182+0000 7f8b477fe700 1 --2- 172.21.15.42:0/2908158084 >> v2:172.21.15.42:3300/0 conn(0x7f8b480a1bc0 0x7f8b480a1ff0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f8b480a3420 tx=0x7f8b3c011920).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:21.195 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:21.182+0000 7f8b477fe700 1 --2- 172.21.15.42:0/2908158084 >> v2:172.21.15.42:3300/0 conn(0x7f8b480a1bc0 0x7f8b480a1ff0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f8b480a3420 tx=0x7f8b3c011920).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:21.195 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:21.182+0000 7f8b477fe700 1 --2- 172.21.15.42:0/2908158084 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8b480a1bc0 0x7f8b480a1ff0 secure :-1 s=READY pgs=1238 cs=0 l=1 rx=0x7f8b480a3420 tx=0x7f8b3c011920).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:39:21.195 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:21.182+0000 7f8b477fe700 1 --2- 172.21.15.42:0/2908158084 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8b480a1bc0 0x7f8b480a1ff0 secure :-1 s=THROTTLE_DONE pgs=1238 cs=0 l=1 rx=0x7f8b480a3420 tx=0x7f8b3c011920).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:21.196 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:21.182+0000 7f8b44ff9700 1 -- 172.21.15.42:0/2908158084 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8b3c00f280 con 0x7f8b480a1bc0 2019-12-05T05:39:21.196 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:21.182+0000 7f8b477fe700 1 --2- 172.21.15.42:0/2908158084 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8b480a1bc0 0x7f8b480a1ff0 secure :-1 s=THROTTLE_DONE pgs=1238 cs=0 l=1 rx=0x7f8b480a3420 tx=0x7f8b3c011920).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:21.196 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:21.182+0000 7f8b477fe700 1 --2- 172.21.15.42:0/2908158084 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8b480a1bc0 0x7f8b480a1ff0 secure :-1 s=THROTTLE_DONE pgs=1238 cs=0 l=1 rx=0x7f8b480a3420 tx=0x7f8b3c011920).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:21.196 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:21.182+0000 7f8b44ff9700 1 -- 172.21.15.42:0/2908158084 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b3c008db0 con 0x7f8b480a1bc0 2019-12-05T05:39:21.196 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:21.182+0000 7f8b4e1ce700 1 -- 172.21.15.42:0/2908158084 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b480a2da0 con 0x7f8b480a1bc0 2019-12-05T05:39:21.196 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:21.182+0000 7f8b44ff9700 1 -- 172.21.15.42:0/2908158084 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8b3c005540 con 0x7f8b480a1bc0 2019-12-05T05:39:21.197 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:21.182+0000 7f8b4e1ce700 1 -- 172.21.15.42:0/2908158084 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b48077300 con 0x7f8b480a1bc0 2019-12-05T05:39:21.197 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:21.182+0000 7f8b477fe700 1 --2- 172.21.15.42:0/2908158084 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8b480a1bc0 0x7f8b480a1ff0 secure :-1 s=THROTTLE_DONE pgs=1238 cs=0 l=1 rx=0x7f8b480a3420 tx=0x7f8b3c011920).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:21.197 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:21.182+0000 7f8b44ff9700 1 -- 172.21.15.42:0/2908158084 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f8b3c005850 con 0x7f8b480a1bc0 2019-12-05T05:39:21.197 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:21.182+0000 7f8b477fe700 1 --2- 172.21.15.42:0/2908158084 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8b480a1bc0 0x7f8b480a1ff0 secure :-1 s=THROTTLE_DONE pgs=1238 cs=0 l=1 rx=0x7f8b480a3420 tx=0x7f8b3c011920).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:21.197 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:21.186+0000 7f8b44ff9700 1 --2- 172.21.15.42:0/2908158084 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8b2803ae00 0x7f8b2803d310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:39:21.197 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:21.186+0000 7f8b44ff9700 1 -- 172.21.15.42:0/2908158084 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 8095+0+0 (secure 0 0 0) 0x7f8b3c04b700 con 0x7f8b480a1bc0 2019-12-05T05:39:21.197 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:21.186+0000 7f8b47fff700 1 --2- 172.21.15.42:0/2908158084 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8b2803ae00 0x7f8b2803d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:39:21.198 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:21.186+0000 7f8b47fff700 1 --2- 172.21.15.42:0/2908158084 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8b2803ae00 0x7f8b2803d310 crc :-1 s=READY pgs=535 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:39:21.198 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:21.190+0000 7f8b4e1ce700 1 -- 172.21.15.42:0/2908158084 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b4805d110 con 0x7f8b480a1bc0 2019-12-05T05:39:21.199 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:21.194+0000 7f8b477fe700 1 --2- 172.21.15.42:0/2908158084 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8b480a1bc0 0x7f8b480a1ff0 secure :-1 s=THROTTLE_DONE pgs=1238 cs=0 l=1 rx=0x7f8b480a3420 tx=0x7f8b3c011920).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:21.199 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:21.194+0000 7f8b44ff9700 1 -- 172.21.15.42:0/2908158084 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f8b3c0329e0 con 0x7f8b480a1bc0 2019-12-05T05:39:21.369 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:21.366+0000 7f8b4e1ce700 1 -- 172.21.15.42:0/2908158084 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8b4805b8c0 con 0x7f8b480a1bc0 2019-12-05T05:39:21.371 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:21.366+0000 7f8b477fe700 1 --2- 172.21.15.42:0/2908158084 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8b480a1bc0 0x7f8b480a1ff0 secure :-1 s=THROTTLE_DONE pgs=1238 cs=0 l=1 rx=0x7f8b480a3420 tx=0x7f8b3c011920).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:39:21.372 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:21.366+0000 7f8b44ff9700 1 -- 172.21.15.42:0/2908158084 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 153 v153) v1 ==== 95+0+2109 (secure 0 0 0) 0x7f8b3c033590 con 0x7f8b480a1bc0 2019-12-05T05:39:21.372 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:39:21.373 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":153,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7539,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/1233824070","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1233824070},{"type":"v1","addr":"172.21.15.42:6835","nonce":1233824070}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":153}],"filesystems":[{"mdsmap":{"epoch":152,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:39:14.336435+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7416},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7416":{"gid":7416,"name":"b","rank":0,"incarnation":149,"state":"up:active","state_seq":30,"addr":"172.21.15.40:6833/123054133","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":123054133},{"type":"v1","addr":"172.21.15.40:6833","nonce":123054133}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:39:21.373 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 153 2019-12-05T05:39:21.375 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:21.370+0000 7f8b4e1ce700 1 -- 172.21.15.42:0/2908158084 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8b2803ae00 msgr2=0x7f8b2803d310 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:39:21.375 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:21.370+0000 7f8b4e1ce700 1 --2- 172.21.15.42:0/2908158084 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8b2803ae00 0x7f8b2803d310 crc :-1 s=READY pgs=535 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:39:21.376 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:21.370+0000 7f8b4e1ce700 1 -- 172.21.15.42:0/2908158084 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8b480a1bc0 msgr2=0x7f8b480a1ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:39:21.376 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:21.370+0000 7f8b4e1ce700 1 --2- 172.21.15.42:0/2908158084 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8b480a1bc0 0x7f8b480a1ff0 secure :-1 s=READY pgs=1238 cs=0 l=1 rx=0x7f8b480a3420 tx=0x7f8b3c011920).stop 2019-12-05T05:39:21.376 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:21.370+0000 7f8b4e1ce700 1 -- 172.21.15.42:0/2908158084 shutdown_connections 2019-12-05T05:39:21.376 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:21.370+0000 7f8b4e1ce700 1 --2- 172.21.15.42:0/2908158084 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8b2803ae00 0x7f8b2803d310 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:39:21.376 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:21.370+0000 7f8b4e1ce700 1 --2- 172.21.15.42:0/2908158084 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8b480a5f10 0x7f8b48142bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:39:21.376 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:21.370+0000 7f8b4e1ce700 1 --2- 172.21.15.42:0/2908158084 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8b480a1bc0 0x7f8b480a1ff0 unknown :-1 s=CLOSED pgs=1238 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:39:21.376 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:21.370+0000 7f8b4e1ce700 1 -- 172.21.15.42:0/2908158084 shutdown_connections 2019-12-05T05:39:21.377 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:21.370+0000 7f8b4e1ce700 1 -- 172.21.15.42:0/2908158084 wait complete. 2019-12-05T05:39:21.377 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:39:21.370+0000 7f8b4e1ce700 1 -- 172.21.15.42:0/2908158084 >> 172.21.15.42:0/2908158084 conn(0x7f8b4807ebb0 msgr2=0x7f8b4811c330 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:39:21.385 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a reported in up:standby state 2019-12-05T05:39:21.385 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 83 secs before thrashing 2019-12-05T05:39:40.647 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:39:40.651 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:40:10.706 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:40:10.710 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:40:40.745 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:40:40.749 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:40:44.387 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:40:44.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.485+0000 7fe79d7f4700 1 Processor -- start 2019-12-05T05:40:44.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.485+0000 7fe79d7f4700 1 -- start start 2019-12-05T05:40:44.496 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.485+0000 7fe79d7f4700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe79809fad0 0x7fe79814ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:40:44.496 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.485+0000 7fe79d7f4700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7980a0700 con 0x7fe7980a7730 2019-12-05T05:40:44.496 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.485+0000 7fe79d7f4700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe79807ef10 con 0x7fe79808df70 2019-12-05T05:40:44.496 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.485+0000 7fe79d7f4700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7980a0af0 con 0x7fe79809fad0 2019-12-05T05:40:44.496 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.485+0000 7fe797fff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe79809fad0 0x7fe79814ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:40:44.496 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.485+0000 7fe796ffd700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fe79808df70 0x7fe79809f2d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:38564/0 (socket says 172.21.15.42:38564) 2019-12-05T05:40:44.496 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.485+0000 7fe7977fe700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fe7980a7730 0x7fe79807e710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:48002/0 (socket says 172.21.15.42:48002) 2019-12-05T05:40:44.497 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.485+0000 7fe797fff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe79809fad0 0x7fe79814ffe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:38528/0 (socket says 172.21.15.42:38528) 2019-12-05T05:40:44.497 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.485+0000 7fe7977fe700 1 -- 172.21.15.42:0/3631610106 learned_addr learned my addr 172.21.15.42:0/3631610106 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:40:44.497 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.485+0000 7fe7967fc700 1 -- 172.21.15.42:0/3631610106 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4048671100 0 0) 0x7fe7980a0700 con 0x7fe7980a7730 2019-12-05T05:40:44.497 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.485+0000 7fe797fff700 1 -- 172.21.15.42:0/3631610106 >> v1:172.21.15.42:6789/0 conn(0x7fe79808df70 legacy=0x7fe79809f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:40:44.497 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.485+0000 7fe797fff700 1 -- 172.21.15.42:0/3631610106 >> v1:172.21.15.40:6789/0 conn(0x7fe7980a7730 legacy=0x7fe79807e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:40:44.497 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.489+0000 7fe797fff700 1 -- 172.21.15.42:0/3631610106 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7980a9b90 con 0x7fe79809fad0 2019-12-05T05:40:44.497 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.489+0000 7fe797fff700 1 --2- 172.21.15.42:0/3631610106 >> v2:172.21.15.42:3300/0 conn(0x7fe79809fad0 0x7fe79814ffe0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fe788008790 tx=0x7fe780001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:40:44.498 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.489+0000 7fe797fff700 1 --2- 172.21.15.42:0/3631610106 >> v2:172.21.15.42:3300/0 conn(0x7fe79809fad0 0x7fe79814ffe0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fe788008790 tx=0x7fe780001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:40:44.498 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.489+0000 7fe797fff700 1 --2- 172.21.15.42:0/3631610106 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe79809fad0 0x7fe79814ffe0 secure :-1 s=READY pgs=1244 cs=0 l=1 rx=0x7fe788008790 tx=0x7fe780001150).ready entity=mon.1 client_cookie=c7b592cba49dad1 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:40:44.498 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.489+0000 7fe797fff700 1 --2- 172.21.15.42:0/3631610106 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe79809fad0 0x7fe79814ffe0 secure :-1 s=THROTTLE_DONE pgs=1244 cs=0 l=1 rx=0x7fe788008790 tx=0x7fe780001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:40:44.498 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.489+0000 7fe7967fc700 1 -- 172.21.15.42:0/3631610106 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe788008bb0 con 0x7fe79809fad0 2019-12-05T05:40:44.498 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.489+0000 7fe797fff700 1 --2- 172.21.15.42:0/3631610106 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe79809fad0 0x7fe79814ffe0 secure :-1 s=THROTTLE_DONE pgs=1244 cs=0 l=1 rx=0x7fe788008790 tx=0x7fe780001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:40:44.498 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.489+0000 7fe7967fc700 1 -- 172.21.15.42:0/3631610106 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7880087d0 con 0x7fe79809fad0 2019-12-05T05:40:44.498 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.489+0000 7fe797fff700 1 --2- 172.21.15.42:0/3631610106 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe79809fad0 0x7fe79814ffe0 secure :-1 s=THROTTLE_DONE pgs=1244 cs=0 l=1 rx=0x7fe788008790 tx=0x7fe780001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:40:44.499 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.489+0000 7fe7967fc700 1 -- 172.21.15.42:0/3631610106 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe7880117d0 con 0x7fe79809fad0 2019-12-05T05:40:44.499 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.489+0000 7fe79d7f4700 1 -- 172.21.15.42:0/3631610106 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe79809fad0 msgr2=0x7fe79814ffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:40:44.499 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.489+0000 7fe79d7f4700 1 --2- 172.21.15.42:0/3631610106 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe79809fad0 0x7fe79814ffe0 secure :-1 s=READY pgs=1244 cs=0 l=1 rx=0x7fe788008790 tx=0x7fe780001150).stop 2019-12-05T05:40:44.499 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.489+0000 7fe79d7f4700 1 -- 172.21.15.42:0/3631610106 shutdown_connections 2019-12-05T05:40:44.499 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.489+0000 7fe79d7f4700 1 --2- 172.21.15.42:0/3631610106 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe79809fad0 0x7fe79814ffe0 unknown :-1 s=CLOSED pgs=1244 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:40:44.500 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.489+0000 7fe79d7f4700 1 -- 172.21.15.42:0/3631610106 shutdown_connections 2019-12-05T05:40:44.500 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.489+0000 7fe79d7f4700 1 -- 172.21.15.42:0/3631610106 wait complete. 2019-12-05T05:40:44.500 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.489+0000 7fe79d7f4700 1 -- 172.21.15.42:0/3631610106 >> 172.21.15.42:0/3631610106 conn(0x7fe7980aabd0 msgr2=0x7fe7980a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:40:44.500 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.489+0000 7fe79d7f4700 1 Processor -- start 2019-12-05T05:40:44.500 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.489+0000 7fe79d7f4700 1 -- start start 2019-12-05T05:40:44.500 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.489+0000 7fe79d7f4700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe79811cd60 0x7fe798123450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:40:44.501 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.489+0000 7fe79d7f4700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7981248a0 con 0x7fe798124470 2019-12-05T05:40:44.501 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.489+0000 7fe79d7f4700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe79811bb60 con 0x7fe7980752e0 2019-12-05T05:40:44.501 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.489+0000 7fe79d7f4700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe798124b70 con 0x7fe79811cd60 2019-12-05T05:40:44.501 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.493+0000 7fe797fff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fe7980752e0 0x7fe798077720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:38572/0 (socket says 172.21.15.42:38572) 2019-12-05T05:40:44.501 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.493+0000 7fe797fff700 1 -- 172.21.15.42:0/2022138295 learned_addr learned my addr 172.21.15.42:0/2022138295 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:40:44.505 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.493+0000 7fe7977fe700 1 --2- 172.21.15.42:0/2022138295 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe79811cd60 0x7fe798123450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:40:44.505 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.493+0000 7fe78ffff700 1 -- 172.21.15.42:0/2022138295 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 349409006 0 0) 0x7fe79811bb60 con 0x7fe7980752e0 2019-12-05T05:40:44.505 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.493+0000 7fe78ffff700 1 -- 172.21.15.42:0/2022138295 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe77c0030b0 con 0x7fe7980752e0 2019-12-05T05:40:44.506 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.493+0000 7fe78ffff700 1 -- 172.21.15.42:0/2022138295 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1774554598 0 0) 0x7fe7981248a0 con 0x7fe798124470 2019-12-05T05:40:44.506 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.493+0000 7fe78ffff700 1 -- 172.21.15.42:0/2022138295 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe79811bb60 con 0x7fe798124470 2019-12-05T05:40:44.506 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.493+0000 7fe78ffff700 1 -- 172.21.15.42:0/2022138295 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 3606221240 0 0) 0x7fe77c0030b0 con 0x7fe7980752e0 2019-12-05T05:40:44.506 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.493+0000 7fe7977fe700 1 -- 172.21.15.42:0/2022138295 >> v1:172.21.15.42:6789/0 conn(0x7fe7980752e0 legacy=0x7fe798077720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:40:44.506 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.493+0000 7fe7977fe700 1 -- 172.21.15.42:0/2022138295 >> v1:172.21.15.40:6789/0 conn(0x7fe798124470 legacy=0x7fe79811b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:40:44.506 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.493+0000 7fe7977fe700 1 -- 172.21.15.42:0/2022138295 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe79807af50 con 0x7fe79811cd60 2019-12-05T05:40:44.506 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.493+0000 7fe7977fe700 1 --2- 172.21.15.42:0/2022138295 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe79811cd60 0x7fe798123450 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fe788007780 tx=0x7fe788010000).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:40:44.507 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.493+0000 7fe7977fe700 1 --2- 172.21.15.42:0/2022138295 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe79811cd60 0x7fe798123450 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fe788007780 tx=0x7fe788010000).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:40:44.507 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.493+0000 7fe7977fe700 1 --2- 172.21.15.42:0/2022138295 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe79811cd60 0x7fe798123450 secure :-1 s=READY pgs=257 cs=0 l=1 rx=0x7fe788007780 tx=0x7fe788010000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:40:44.507 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.493+0000 7fe7977fe700 1 --2- 172.21.15.42:0/2022138295 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe79811cd60 0x7fe798123450 secure :-1 s=THROTTLE_DONE pgs=257 cs=0 l=1 rx=0x7fe788007780 tx=0x7fe788010000).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:40:44.507 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.493+0000 7fe78ffff700 1 -- 172.21.15.42:0/2022138295 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe788008470 con 0x7fe79811cd60 2019-12-05T05:40:44.507 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.493+0000 7fe7977fe700 1 --2- 172.21.15.42:0/2022138295 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe79811cd60 0x7fe798123450 secure :-1 s=THROTTLE_DONE pgs=257 cs=0 l=1 rx=0x7fe788007780 tx=0x7fe788010000).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:40:44.507 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.493+0000 7fe79d7f4700 1 -- 172.21.15.42:0/2022138295 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe798078f30 con 0x7fe79811cd60 2019-12-05T05:40:44.507 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.493+0000 7fe78ffff700 1 -- 172.21.15.42:0/2022138295 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe788008db0 con 0x7fe79811cd60 2019-12-05T05:40:44.508 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.493+0000 7fe79d7f4700 1 -- 172.21.15.42:0/2022138295 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe798079290 con 0x7fe79811cd60 2019-12-05T05:40:44.508 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.493+0000 7fe7977fe700 1 --2- 172.21.15.42:0/2022138295 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe79811cd60 0x7fe798123450 secure :-1 s=THROTTLE_DONE pgs=257 cs=0 l=1 rx=0x7fe788007780 tx=0x7fe788010000).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:40:44.508 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.493+0000 7fe78ffff700 1 -- 172.21.15.42:0/2022138295 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe79000d4a0 con 0x7fe79811cd60 2019-12-05T05:40:44.508 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.493+0000 7fe7977fe700 1 --2- 172.21.15.42:0/2022138295 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe79811cd60 0x7fe798123450 secure :-1 s=THROTTLE_DONE pgs=257 cs=0 l=1 rx=0x7fe788007780 tx=0x7fe788010000).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:40:44.508 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.493+0000 7fe78ffff700 1 -- 172.21.15.42:0/2022138295 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fe79000d6a0 con 0x7fe79811cd60 2019-12-05T05:40:44.508 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.493+0000 7fe7977fe700 1 --2- 172.21.15.42:0/2022138295 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe79811cd60 0x7fe798123450 secure :-1 s=THROTTLE_DONE pgs=257 cs=0 l=1 rx=0x7fe788007780 tx=0x7fe788010000).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:40:44.508 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.497+0000 7fe78ffff700 1 --2- 172.21.15.42:0/2022138295 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe77c03b3f0 0x7fe77c03d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:40:44.509 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.497+0000 7fe78ffff700 1 -- 172.21.15.42:0/2022138295 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 8095+0+0 (secure 0 0 0) 0x7fe79004a9a0 con 0x7fe79811cd60 2019-12-05T05:40:44.509 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.497+0000 7fe796ffd700 1 --2- 172.21.15.42:0/2022138295 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe77c03b3f0 0x7fe77c03d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:40:44.509 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.497+0000 7fe796ffd700 1 --2- 172.21.15.42:0/2022138295 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe77c03b3f0 0x7fe77c03d900 crc :-1 s=READY pgs=538 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:40:44.509 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.501+0000 7fe79d7f4700 1 -- 172.21.15.42:0/2022138295 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe79811d7f0 con 0x7fe79811cd60 2019-12-05T05:40:44.515 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.509+0000 7fe7977fe700 1 --2- 172.21.15.42:0/2022138295 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe79811cd60 0x7fe798123450 secure :-1 s=THROTTLE_DONE pgs=257 cs=0 l=1 rx=0x7fe788007780 tx=0x7fe788010000).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:40:44.515 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.509+0000 7fe78ffff700 1 -- 172.21.15.42:0/2022138295 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fe790031a00 con 0x7fe79811cd60 2019-12-05T05:40:44.686 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.681+0000 7fe79d7f4700 1 -- 172.21.15.42:0/2022138295 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe79805d100 con 0x7fe79811cd60 2019-12-05T05:40:44.687 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.681+0000 7fe7977fe700 1 --2- 172.21.15.42:0/2022138295 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe79811cd60 0x7fe798123450 secure :-1 s=THROTTLE_DONE pgs=257 cs=0 l=1 rx=0x7fe788007780 tx=0x7fe788010000).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:40:44.687 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.681+0000 7fe78ffff700 1 -- 172.21.15.42:0/2022138295 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 153 v153) v1 ==== 95+0+2109 (secure 0 0 0) 0x7fe790031c60 con 0x7fe79811cd60 2019-12-05T05:40:44.689 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 153 2019-12-05T05:40:44.689 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:40:44.690 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":153,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7539,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/1233824070","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1233824070},{"type":"v1","addr":"172.21.15.42:6835","nonce":1233824070}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":153}],"filesystems":[{"mdsmap":{"epoch":152,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:39:14.336435+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7416},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7416":{"gid":7416,"name":"b","rank":0,"incarnation":149,"state":"up:active","state_seq":30,"addr":"172.21.15.40:6833/123054133","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":123054133},{"type":"v1","addr":"172.21.15.40:6833","nonce":123054133}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:40:44.693 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.689+0000 7fe79d7f4700 1 -- 172.21.15.42:0/2022138295 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe77c03b3f0 msgr2=0x7fe77c03d900 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:40:44.693 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.689+0000 7fe79d7f4700 1 --2- 172.21.15.42:0/2022138295 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe77c03b3f0 0x7fe77c03d900 crc :-1 s=READY pgs=538 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:40:44.693 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.689+0000 7fe79d7f4700 1 -- 172.21.15.42:0/2022138295 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe79811cd60 msgr2=0x7fe798123450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:40:44.693 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.689+0000 7fe79d7f4700 1 --2- 172.21.15.42:0/2022138295 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe79811cd60 0x7fe798123450 secure :-1 s=READY pgs=257 cs=0 l=1 rx=0x7fe788007780 tx=0x7fe788010000).stop 2019-12-05T05:40:44.694 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.689+0000 7fe79d7f4700 1 -- 172.21.15.42:0/2022138295 shutdown_connections 2019-12-05T05:40:44.694 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.689+0000 7fe79d7f4700 1 --2- 172.21.15.42:0/2022138295 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe77c03b3f0 0x7fe77c03d900 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:40:44.694 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.689+0000 7fe79d7f4700 1 --2- 172.21.15.42:0/2022138295 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe79811cd60 0x7fe798123450 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:40:44.694 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.689+0000 7fe79d7f4700 1 -- 172.21.15.42:0/2022138295 shutdown_connections 2019-12-05T05:40:44.694 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.689+0000 7fe79d7f4700 1 -- 172.21.15.42:0/2022138295 wait complete. 2019-12-05T05:40:44.694 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.689+0000 7fe79d7f4700 1 -- 172.21.15.42:0/2022138295 >> 172.21.15.42:0/2022138295 conn(0x7fe79807e820 msgr2=0x7fe7980a7f80 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:40:44.706 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.b (rank=0) 2019-12-05T05:40:44.707 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2019-12-05T05:40:44.708 INFO:teuthology.orchestra.run:waiting for 300 2019-12-05T05:40:44.720 INFO:tasks.ceph.mds.b:Stopped 2019-12-05T05:40:44.721 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:40:44.825 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.813+0000 7f6cab323700 1 Processor -- start 2019-12-05T05:40:44.825 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.813+0000 7f6cab323700 1 -- start start 2019-12-05T05:40:44.825 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.817+0000 7f6cab323700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6ca40a7740 0x7f6ca408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:40:44.826 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.817+0000 7f6cab323700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f6ca407e720 0x7f6ca407eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:40:44.826 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.817+0000 7f6cab323700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ca40a0710 con 0x7f6ca409f2e0 2019-12-05T05:40:44.826 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.817+0000 7f6cab323700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ca40a0b00 con 0x7f6ca407e720 2019-12-05T05:40:44.826 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.817+0000 7f6cab323700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ca408e5b0 con 0x7f6ca40a7740 2019-12-05T05:40:44.826 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.817+0000 7f6ca98c0700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6ca40a7740 0x7f6ca408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:40:44.826 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.817+0000 7f6ca98c0700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6ca40a7740 0x7f6ca408df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:38540/0 (socket says 172.21.15.42:38540) 2019-12-05T05:40:44.826 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.817+0000 7f6ca90bf700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f6ca407e720 0x7f6ca407eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:40:44.827 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.817+0000 7f6ca98c0700 1 -- 172.21.15.42:0/1771259286 learned_addr learned my addr 172.21.15.42:0/1771259286 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:40:44.827 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.817+0000 7f6ca88be700 1 -- 172.21.15.42:0/1771259286 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3389865157 0 0) 0x7f6ca40a0710 con 0x7f6ca409f2e0 2019-12-05T05:40:44.827 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.817+0000 7f6ca88be700 1 -- 172.21.15.42:0/1771259286 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c900030b0 con 0x7f6ca409f2e0 2019-12-05T05:40:44.827 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.817+0000 7f6ca90bf700 1 -- 172.21.15.42:0/1771259286 >> v2:172.21.15.42:3300/0 conn(0x7f6ca40a7740 msgr2=0x7f6ca408df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:40:44.827 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.817+0000 7f6ca90bf700 1 --2- 172.21.15.42:0/1771259286 >> v2:172.21.15.42:3300/0 conn(0x7f6ca40a7740 0x7f6ca408df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:40:44.827 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.817+0000 7f6ca90bf700 1 -- 172.21.15.42:0/1771259286 >> v1:172.21.15.40:6789/0 conn(0x7f6ca409f2e0 legacy=0x7f6ca409f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:40:44.827 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.817+0000 7f6ca90bf700 1 -- 172.21.15.42:0/1771259286 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ca407f0a0 con 0x7f6ca407e720 2019-12-05T05:40:44.828 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.817+0000 7f6ca98c0700 1 --2- 172.21.15.42:0/1771259286 >> v2:172.21.15.42:3300/0 conn(0x7f6ca40a7740 0x7f6ca408df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:40:44.828 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.817+0000 7f6ca90bf700 1 --2- 172.21.15.42:0/1771259286 >> v2:172.21.15.40:3300/0 conn(0x7f6ca407e720 0x7f6ca407eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f6c94002f30 tx=0x7f6c9400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:40:44.828 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.817+0000 7f6ca90bf700 1 --2- 172.21.15.42:0/1771259286 >> v2:172.21.15.40:3300/0 conn(0x7f6ca407e720 0x7f6ca407eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f6c94002f30 tx=0x7f6c9400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:40:44.828 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.817+0000 7f6ca90bf700 1 --2- 172.21.15.42:0/1771259286 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ca407e720 0x7f6ca407eb50 secure :-1 s=READY pgs=970 cs=0 l=1 rx=0x7f6c94002f30 tx=0x7f6c9400fc70).ready entity=mon.0 client_cookie=c6aba70a441bc32a server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:40:44.828 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.817+0000 7f6ca90bf700 1 --2- 172.21.15.42:0/1771259286 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ca407e720 0x7f6ca407eb50 secure :-1 s=THROTTLE_DONE pgs=970 cs=0 l=1 rx=0x7f6c94002f30 tx=0x7f6c9400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:40:44.828 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.817+0000 7f6ca88be700 1 -- 172.21.15.42:0/1771259286 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6c940112b0 con 0x7f6ca407e720 2019-12-05T05:40:44.828 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.817+0000 7f6ca90bf700 1 --2- 172.21.15.42:0/1771259286 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ca407e720 0x7f6ca407eb50 secure :-1 s=THROTTLE_DONE pgs=970 cs=0 l=1 rx=0x7f6c94002f30 tx=0x7f6c9400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:40:44.829 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.817+0000 7f6ca90bf700 1 --2- 172.21.15.42:0/1771259286 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ca407e720 0x7f6ca407eb50 secure :-1 s=THROTTLE_DONE pgs=970 cs=0 l=1 rx=0x7f6c94002f30 tx=0x7f6c9400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:40:44.829 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.817+0000 7f6ca88be700 1 -- 172.21.15.42:0/1771259286 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c94008ad0 con 0x7f6ca407e720 2019-12-05T05:40:44.829 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.817+0000 7f6ca88be700 1 -- 172.21.15.42:0/1771259286 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6c94011890 con 0x7f6ca407e720 2019-12-05T05:40:44.829 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.817+0000 7f6cab323700 1 -- 172.21.15.42:0/1771259286 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ca407e720 msgr2=0x7f6ca407eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:40:44.829 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.817+0000 7f6cab323700 1 --2- 172.21.15.42:0/1771259286 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ca407e720 0x7f6ca407eb50 secure :-1 s=READY pgs=970 cs=0 l=1 rx=0x7f6c94002f30 tx=0x7f6c9400fc70).stop 2019-12-05T05:40:44.829 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.821+0000 7f6cab323700 1 -- 172.21.15.42:0/1771259286 shutdown_connections 2019-12-05T05:40:44.829 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.821+0000 7f6cab323700 1 --2- 172.21.15.42:0/1771259286 >> v2:172.21.15.42:3300/0 conn(0x7f6ca40a7740 0x7f6ca408df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:40:44.830 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.821+0000 7f6cab323700 1 --2- 172.21.15.42:0/1771259286 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ca407e720 0x7f6ca407eb50 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:40:44.830 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.821+0000 7f6cab323700 1 -- 172.21.15.42:0/1771259286 shutdown_connections 2019-12-05T05:40:44.830 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.821+0000 7f6cab323700 1 -- 172.21.15.42:0/1771259286 wait complete. 2019-12-05T05:40:44.830 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.821+0000 7f6cab323700 1 -- 172.21.15.42:0/1771259286 >> 172.21.15.42:0/1771259286 conn(0x7f6ca40aabe0 msgr2=0x7f6ca40a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:40:44.834 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.821+0000 7f6cab323700 1 Processor -- start 2019-12-05T05:40:44.834 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.821+0000 7f6cab323700 1 -- start start 2019-12-05T05:40:44.834 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.821+0000 7f6cab323700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f6ca4124000 0x7f6ca40a5560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:40:44.834 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.821+0000 7f6cab323700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6ca40a5ab0 0x7f6ca411f530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:40:44.834 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.821+0000 7f6cab323700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ca4132150 con 0x7f6ca411fa80 2019-12-05T05:40:44.834 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.821+0000 7f6cab323700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ca40a52c0 con 0x7f6ca4124000 2019-12-05T05:40:44.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.821+0000 7f6cab323700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ca4131340 con 0x7f6ca40a5ab0 2019-12-05T05:40:44.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.821+0000 7f6ca90bf700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6ca40a5ab0 0x7f6ca411f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:40:44.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.825+0000 7f6ca98c0700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f6ca4124000 0x7f6ca40a5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:40:44.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.825+0000 7f6ca90bf700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6ca40a5ab0 0x7f6ca411f530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:38548/0 (socket says 172.21.15.42:38548) 2019-12-05T05:40:44.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.825+0000 7f6caa0c1700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f6ca411fa80 0x7f6ca408eb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:48028/0 (socket says 172.21.15.42:48028) 2019-12-05T05:40:44.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.825+0000 7f6ca98c0700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f6ca4124000 0x7f6ca40a5560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.42:54594/0 (socket says 172.21.15.42:54594) 2019-12-05T05:40:44.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.825+0000 7f6ca90bf700 1 -- 172.21.15.42:0/380783139 learned_addr learned my addr 172.21.15.42:0/380783139 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:40:44.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.825+0000 7f6ca90bf700 1 -- 172.21.15.42:0/380783139 >> v2:172.21.15.40:3300/0 conn(0x7f6ca4124000 msgr2=0x7f6ca40a5560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:40:44.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.825+0000 7f6ca90bf700 1 --2- 172.21.15.42:0/380783139 >> v2:172.21.15.40:3300/0 conn(0x7f6ca4124000 0x7f6ca40a5560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:40:44.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.825+0000 7f6ca90bf700 1 -- 172.21.15.42:0/380783139 >> v1:172.21.15.40:6789/0 conn(0x7f6ca411fa80 legacy=0x7f6ca408eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:40:44.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.825+0000 7f6ca90bf700 1 -- 172.21.15.42:0/380783139 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ca40a60a0 con 0x7f6ca40a5ab0 2019-12-05T05:40:44.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.825+0000 7f6c9a7fc700 1 -- 172.21.15.42:0/380783139 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 548356011 0 0) 0x7f6ca4132150 con 0x7f6ca411fa80 2019-12-05T05:40:44.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.825+0000 7f6ca90bf700 1 --2- 172.21.15.42:0/380783139 >> v2:172.21.15.42:3300/0 conn(0x7f6ca40a5ab0 0x7f6ca411f530 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f6c94002f30 tx=0x7f6c94011e20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:40:44.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.825+0000 7f6ca90bf700 1 --2- 172.21.15.42:0/380783139 >> v2:172.21.15.42:3300/0 conn(0x7f6ca40a5ab0 0x7f6ca411f530 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f6c94002f30 tx=0x7f6c94011e20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:40:44.837 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.825+0000 7f6ca90bf700 1 --2- 172.21.15.42:0/380783139 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6ca40a5ab0 0x7f6ca411f530 secure :-1 s=READY pgs=1246 cs=0 l=1 rx=0x7f6c94002f30 tx=0x7f6c94011e20).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:40:44.837 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.825+0000 7f6ca90bf700 1 --2- 172.21.15.42:0/380783139 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6ca40a5ab0 0x7f6ca411f530 secure :-1 s=THROTTLE_DONE pgs=1246 cs=0 l=1 rx=0x7f6c94002f30 tx=0x7f6c94011e20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:40:44.837 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.825+0000 7f6c9a7fc700 1 -- 172.21.15.42:0/380783139 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6c94007150 con 0x7f6ca40a5ab0 2019-12-05T05:40:44.837 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.825+0000 7f6ca90bf700 1 --2- 172.21.15.42:0/380783139 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6ca40a5ab0 0x7f6ca411f530 secure :-1 s=THROTTLE_DONE pgs=1246 cs=0 l=1 rx=0x7f6c94002f30 tx=0x7f6c94011e20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:40:44.837 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.825+0000 7f6ca90bf700 1 --2- 172.21.15.42:0/380783139 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6ca40a5ab0 0x7f6ca411f530 secure :-1 s=THROTTLE_DONE pgs=1246 cs=0 l=1 rx=0x7f6c94002f30 tx=0x7f6c94011e20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:40:44.837 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.825+0000 7f6c9a7fc700 1 -- 172.21.15.42:0/380783139 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c940114b0 con 0x7f6ca40a5ab0 2019-12-05T05:40:44.837 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.825+0000 7f6cab323700 1 -- 172.21.15.42:0/380783139 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ca4124430 con 0x7f6ca40a5ab0 2019-12-05T05:40:44.838 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.825+0000 7f6c9a7fc700 1 -- 172.21.15.42:0/380783139 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6c94008cc0 con 0x7f6ca40a5ab0 2019-12-05T05:40:44.838 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.825+0000 7f6cab323700 1 -- 172.21.15.42:0/380783139 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ca408f570 con 0x7f6ca40a5ab0 2019-12-05T05:40:44.838 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.825+0000 7f6ca90bf700 1 --2- 172.21.15.42:0/380783139 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6ca40a5ab0 0x7f6ca411f530 secure :-1 s=THROTTLE_DONE pgs=1246 cs=0 l=1 rx=0x7f6c94002f30 tx=0x7f6c94011e20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:40:44.838 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.825+0000 7f6c9a7fc700 1 -- 172.21.15.42:0/380783139 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f6c94005540 con 0x7f6ca40a5ab0 2019-12-05T05:40:44.838 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.825+0000 7f6ca90bf700 1 --2- 172.21.15.42:0/380783139 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6ca40a5ab0 0x7f6ca411f530 secure :-1 s=THROTTLE_DONE pgs=1246 cs=0 l=1 rx=0x7f6c94002f30 tx=0x7f6c94011e20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:40:44.838 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.829+0000 7f6c9a7fc700 1 --2- 172.21.15.42:0/380783139 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6c8c03ac90 0x7f6c8c03d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:40:44.838 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.829+0000 7f6c9a7fc700 1 -- 172.21.15.42:0/380783139 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 8095+0+0 (secure 0 0 0) 0x7f6c9408f540 con 0x7f6ca40a5ab0 2019-12-05T05:40:44.839 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.829+0000 7f6ca98c0700 1 --2- 172.21.15.42:0/380783139 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6c8c03ac90 0x7f6c8c03d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:40:44.839 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.829+0000 7f6ca98c0700 1 --2- 172.21.15.42:0/380783139 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6c8c03ac90 0x7f6c8c03d1a0 crc :-1 s=READY pgs=539 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:40:44.839 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.829+0000 7f6cab323700 1 -- 172.21.15.42:0/380783139 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ca405d110 con 0x7f6ca40a5ab0 2019-12-05T05:40:44.841 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.837+0000 7f6ca90bf700 1 --2- 172.21.15.42:0/380783139 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6ca40a5ab0 0x7f6ca411f530 secure :-1 s=THROTTLE_DONE pgs=1246 cs=0 l=1 rx=0x7f6c94002f30 tx=0x7f6c94011e20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:40:44.841 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:44.837+0000 7f6c9a7fc700 1 -- 172.21.15.42:0/380783139 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f6c94007f50 con 0x7f6ca40a5ab0 2019-12-05T05:40:45.009 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:45.005+0000 7f6cab323700 1 -- 172.21.15.42:0/380783139 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6ca405b8c0 con 0x7f6ca40a5ab0 2019-12-05T05:40:45.010 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:45.005+0000 7f6ca90bf700 1 --2- 172.21.15.42:0/380783139 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6ca40a5ab0 0x7f6ca411f530 secure :-1 s=THROTTLE_DONE pgs=1246 cs=0 l=1 rx=0x7f6c94002f30 tx=0x7f6c94011e20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:40:45.010 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:45.005+0000 7f6c9a7fc700 1 -- 172.21.15.42:0/380783139 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 153 v153) v1 ==== 95+0+2109 (secure 0 0 0) 0x7f6c9400ebd0 con 0x7f6ca40a5ab0 2019-12-05T05:40:45.012 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 153 2019-12-05T05:40:45.013 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:40:45.013 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":153,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7539,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/1233824070","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1233824070},{"type":"v1","addr":"172.21.15.42:6835","nonce":1233824070}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":153}],"filesystems":[{"mdsmap":{"epoch":152,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:39:14.336435+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7416},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7416":{"gid":7416,"name":"b","rank":0,"incarnation":149,"state":"up:active","state_seq":30,"addr":"172.21.15.40:6833/123054133","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":123054133},{"type":"v1","addr":"172.21.15.40:6833","nonce":123054133}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:40:45.016 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:45.013+0000 7f6cab323700 1 -- 172.21.15.42:0/380783139 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6c8c03ac90 msgr2=0x7f6c8c03d1a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:40:45.016 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:45.013+0000 7f6cab323700 1 --2- 172.21.15.42:0/380783139 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6c8c03ac90 0x7f6c8c03d1a0 crc :-1 s=READY pgs=539 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:40:45.017 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:45.013+0000 7f6cab323700 1 -- 172.21.15.42:0/380783139 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6ca40a5ab0 msgr2=0x7f6ca411f530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:40:45.017 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:45.013+0000 7f6cab323700 1 --2- 172.21.15.42:0/380783139 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6ca40a5ab0 0x7f6ca411f530 secure :-1 s=READY pgs=1246 cs=0 l=1 rx=0x7f6c94002f30 tx=0x7f6c94011e20).stop 2019-12-05T05:40:45.017 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:45.013+0000 7f6cab323700 1 -- 172.21.15.42:0/380783139 shutdown_connections 2019-12-05T05:40:45.017 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:45.013+0000 7f6cab323700 1 --2- 172.21.15.42:0/380783139 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6c8c03ac90 0x7f6c8c03d1a0 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:40:45.017 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:45.013+0000 7f6cab323700 1 --2- 172.21.15.42:0/380783139 >> v2:172.21.15.40:3300/0 conn(0x7f6ca4124000 0x7f6ca40a5560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:40:45.017 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:45.013+0000 7f6cab323700 1 --2- 172.21.15.42:0/380783139 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6ca40a5ab0 0x7f6ca411f530 unknown :-1 s=CLOSED pgs=1246 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:40:45.017 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:45.013+0000 7f6cab323700 1 -- 172.21.15.42:0/380783139 shutdown_connections 2019-12-05T05:40:45.018 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:45.013+0000 7f6cab323700 1 -- 172.21.15.42:0/380783139 wait complete. 2019-12-05T05:40:45.018 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:45.013+0000 7f6cab323700 1 -- 172.21.15.42:0/380783139 >> 172.21.15.42:0/380783139 conn(0x7f6ca409f2c0 msgr2=0x7f6ca40a7f90 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:40:45.029 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T05:40:47.030 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:40:47.140 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:47.129+0000 7f9c76767700 1 Processor -- start 2019-12-05T05:40:47.140 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:47.129+0000 7f9c76767700 1 -- start start 2019-12-05T05:40:47.140 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:47.133+0000 7f9c76767700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9c700a7740 0x7f9c7008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:40:47.140 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:47.133+0000 7f9c76767700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f9c7009fae0 0x7f9c7007eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:40:47.140 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:47.133+0000 7f9c76767700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c700a0710 con 0x7f9c7007e720 2019-12-05T05:40:47.140 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:47.133+0000 7f9c76767700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c700a0b00 con 0x7f9c7009fae0 2019-12-05T05:40:47.141 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:47.133+0000 7f9c76767700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c7008e5b0 con 0x7f9c700a7740 2019-12-05T05:40:47.141 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:47.133+0000 7f9c75505700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f9c7009fae0 0x7f9c7007eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:40:47.141 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:47.133+0000 7f9c6ffff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f9c7007e720 0x7f9c7009f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:38600/0 (socket says 172.21.15.42:38600) 2019-12-05T05:40:47.141 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:47.133+0000 7f9c6ffff700 1 -- 172.21.15.42:0/2883710518 learned_addr learned my addr 172.21.15.42:0/2883710518 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:40:47.141 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:47.133+0000 7f9c74d04700 1 --2- 172.21.15.42:0/2883710518 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9c700a7740 0x7f9c7008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:40:47.141 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:47.133+0000 7f9c75505700 1 -- 172.21.15.42:0/2883710518 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9c700a7740 msgr2=0x7f9c7008df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:40:47.142 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:47.133+0000 7f9c75505700 1 --2- 172.21.15.42:0/2883710518 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9c700a7740 0x7f9c7008df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:40:47.142 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:47.133+0000 7f9c75505700 1 -- 172.21.15.42:0/2883710518 >> v1:172.21.15.42:6789/0 conn(0x7f9c7007e720 legacy=0x7f9c7009f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:40:47.142 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:47.133+0000 7f9c6f7fe700 1 -- 172.21.15.42:0/2883710518 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 940533785 0 0) 0x7f9c700a0710 con 0x7f9c7007e720 2019-12-05T05:40:47.142 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:47.133+0000 7f9c75505700 1 -- 172.21.15.42:0/2883710518 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c7007f0a0 con 0x7f9c7009fae0 2019-12-05T05:40:47.142 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:47.133+0000 7f9c75505700 1 --2- 172.21.15.42:0/2883710518 >> v2:172.21.15.42:3300/0 conn(0x7f9c7009fae0 0x7f9c7007eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f9c68008b10 tx=0x7f9c6800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:40:47.142 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:47.133+0000 7f9c75505700 1 --2- 172.21.15.42:0/2883710518 >> v2:172.21.15.42:3300/0 conn(0x7f9c7009fae0 0x7f9c7007eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f9c68008b10 tx=0x7f9c6800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:40:47.143 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:47.133+0000 7f9c75505700 1 --2- 172.21.15.42:0/2883710518 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9c7009fae0 0x7f9c7007eb50 secure :-1 s=READY pgs=1248 cs=0 l=1 rx=0x7f9c68008b10 tx=0x7f9c6800fff0).ready entity=mon.1 client_cookie=a57b1d085574333e server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:40:47.143 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:47.133+0000 7f9c75505700 1 --2- 172.21.15.42:0/2883710518 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9c7009fae0 0x7f9c7007eb50 secure :-1 s=THROTTLE_DONE pgs=1248 cs=0 l=1 rx=0x7f9c68008b10 tx=0x7f9c6800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:40:47.143 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:47.133+0000 7f9c6f7fe700 1 -- 172.21.15.42:0/2883710518 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9c68010900 con 0x7f9c7009fae0 2019-12-05T05:40:47.143 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:47.133+0000 7f9c75505700 1 --2- 172.21.15.42:0/2883710518 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9c7009fae0 0x7f9c7007eb50 secure :-1 s=THROTTLE_DONE pgs=1248 cs=0 l=1 rx=0x7f9c68008b10 tx=0x7f9c6800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:40:47.143 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:47.133+0000 7f9c75505700 1 --2- 172.21.15.42:0/2883710518 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9c7009fae0 0x7f9c7007eb50 secure :-1 s=THROTTLE_DONE pgs=1248 cs=0 l=1 rx=0x7f9c68008b10 tx=0x7f9c6800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:40:47.143 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:47.133+0000 7f9c6f7fe700 1 -- 172.21.15.42:0/2883710518 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c68008b50 con 0x7f9c7009fae0 2019-12-05T05:40:47.143 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:47.133+0000 7f9c6f7fe700 1 -- 172.21.15.42:0/2883710518 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9c680110c0 con 0x7f9c7009fae0 2019-12-05T05:40:47.143 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:47.133+0000 7f9c76767700 1 -- 172.21.15.42:0/2883710518 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9c7009fae0 msgr2=0x7f9c7007eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:40:47.144 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:47.133+0000 7f9c76767700 1 --2- 172.21.15.42:0/2883710518 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9c7009fae0 0x7f9c7007eb50 secure :-1 s=READY pgs=1248 cs=0 l=1 rx=0x7f9c68008b10 tx=0x7f9c6800fff0).stop 2019-12-05T05:40:47.144 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:47.133+0000 7f9c76767700 1 -- 172.21.15.42:0/2883710518 shutdown_connections 2019-12-05T05:40:47.144 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:47.133+0000 7f9c76767700 1 --2- 172.21.15.42:0/2883710518 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9c7009fae0 0x7f9c7007eb50 unknown :-1 s=CLOSED pgs=1248 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:40:47.144 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:47.133+0000 7f9c76767700 1 --2- 172.21.15.42:0/2883710518 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9c700a7740 0x7f9c7008df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:40:47.145 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:47.133+0000 7f9c76767700 1 -- 172.21.15.42:0/2883710518 shutdown_connections 2019-12-05T05:40:47.145 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:47.133+0000 7f9c76767700 1 -- 172.21.15.42:0/2883710518 wait complete. 2019-12-05T05:40:47.145 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:47.133+0000 7f9c76767700 1 -- 172.21.15.42:0/2883710518 >> 172.21.15.42:0/2883710518 conn(0x7f9c700aabe0 msgr2=0x7f9c700a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:40:47.148 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:47.137+0000 7f9c76767700 1 Processor -- start 2019-12-05T05:40:47.148 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:47.137+0000 7f9c76767700 1 -- start start 2019-12-05T05:40:47.148 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:47.137+0000 7f9c76767700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f9c7012f630 0x7f9c7012fa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:40:47.149 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:47.137+0000 7f9c76767700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f9c7007b420 0x7f9c7007b850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:40:47.149 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:47.137+0000 7f9c76767700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c700a5e10 con 0x7f9c7011ca60 2019-12-05T05:40:47.149 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:47.137+0000 7f9c76767700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c700a6ae0 con 0x7f9c7007b420 2019-12-05T05:40:47.149 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:47.137+0000 7f9c76767700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c700a6d80 con 0x7f9c7012f630 2019-12-05T05:40:47.149 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:47.137+0000 7f9c6ffff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f9c7012f630 0x7f9c7012fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:40:47.149 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:47.137+0000 7f9c6ffff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f9c7012f630 0x7f9c7012fa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:38568/0 (socket says 172.21.15.42:38568) 2019-12-05T05:40:47.149 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:47.137+0000 7f9c6ffff700 1 -- 172.21.15.42:0/3300605564 learned_addr learned my addr 172.21.15.42:0/3300605564 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:40:47.150 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:47.137+0000 7f9c75505700 1 --2- 172.21.15.42:0/3300605564 >> v2:172.21.15.40:3300/0 conn(0x7f9c7007b420 0x7f9c7007b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:40:47.150 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:47.137+0000 7f9c6ffff700 1 -- 172.21.15.42:0/3300605564 >> v2:172.21.15.40:3300/0 conn(0x7f9c7007b420 msgr2=0x7f9c7007b850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:40:47.150 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:47.137+0000 7f9c6ffff700 1 --2- 172.21.15.42:0/3300605564 >> v2:172.21.15.40:3300/0 conn(0x7f9c7007b420 0x7f9c7007b850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:40:47.150 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:47.137+0000 7f9c6ffff700 1 -- 172.21.15.42:0/3300605564 >> v1:172.21.15.42:6789/0 conn(0x7f9c7011ca60 legacy=0x7f9c700a6240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:40:47.150 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:47.137+0000 7f9c6ffff700 1 -- 172.21.15.42:0/3300605564 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c7012ffb0 con 0x7f9c7012f630 2019-12-05T05:40:47.150 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:47.137+0000 7f9c6ffff700 1 --2- 172.21.15.42:0/3300605564 >> v2:172.21.15.42:3300/0 conn(0x7f9c7012f630 0x7f9c7012fa60 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f9c5c004370 tx=0x7f9c5c010a00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:40:47.150 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:47.137+0000 7f9c6ffff700 1 --2- 172.21.15.42:0/3300605564 >> v2:172.21.15.42:3300/0 conn(0x7f9c7012f630 0x7f9c7012fa60 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f9c5c004370 tx=0x7f9c5c010a00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:40:47.151 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:47.137+0000 7f9c6ffff700 1 --2- 172.21.15.42:0/3300605564 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9c7012f630 0x7f9c7012fa60 secure :-1 s=READY pgs=1250 cs=0 l=1 rx=0x7f9c5c004370 tx=0x7f9c5c010a00).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:40:47.151 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:47.137+0000 7f9c6ffff700 1 --2- 172.21.15.42:0/3300605564 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9c7012f630 0x7f9c7012fa60 secure :-1 s=THROTTLE_DONE pgs=1250 cs=0 l=1 rx=0x7f9c5c004370 tx=0x7f9c5c010a00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:40:47.151 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:47.137+0000 7f9c6ffff700 1 --2- 172.21.15.42:0/3300605564 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9c7012f630 0x7f9c7012fa60 secure :-1 s=THROTTLE_DONE pgs=1250 cs=0 l=1 rx=0x7f9c5c004370 tx=0x7f9c5c010a00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:40:47.151 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:47.141+0000 7f9c6ffff700 1 --2- 172.21.15.42:0/3300605564 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9c7012f630 0x7f9c7012fa60 secure :-1 s=THROTTLE_DONE pgs=1250 cs=0 l=1 rx=0x7f9c5c004370 tx=0x7f9c5c010a00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:40:47.151 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:47.137+0000 7f9c6d7fa700 1 -- 172.21.15.42:0/3300605564 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9c5c00cbb0 con 0x7f9c7012f630 2019-12-05T05:40:47.151 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:47.141+0000 7f9c6d7fa700 1 -- 172.21.15.42:0/3300605564 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c5c00c7d0 con 0x7f9c7012f630 2019-12-05T05:40:47.151 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:47.141+0000 7f9c76767700 1 -- 172.21.15.42:0/3300605564 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c7007bda0 con 0x7f9c7012f630 2019-12-05T05:40:47.152 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:47.141+0000 7f9c6d7fa700 1 -- 172.21.15.42:0/3300605564 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9c5c004620 con 0x7f9c7012f630 2019-12-05T05:40:47.152 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:47.141+0000 7f9c76767700 1 -- 172.21.15.42:0/3300605564 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c70136ab0 con 0x7f9c7012f630 2019-12-05T05:40:47.152 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:47.141+0000 7f9c6ffff700 1 --2- 172.21.15.42:0/3300605564 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9c7012f630 0x7f9c7012fa60 secure :-1 s=THROTTLE_DONE pgs=1250 cs=0 l=1 rx=0x7f9c5c004370 tx=0x7f9c5c010a00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:40:47.152 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:47.141+0000 7f9c6d7fa700 1 -- 172.21.15.42:0/3300605564 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f9c5c004930 con 0x7f9c7012f630 2019-12-05T05:40:47.152 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:47.141+0000 7f9c6ffff700 1 --2- 172.21.15.42:0/3300605564 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9c7012f630 0x7f9c7012fa60 secure :-1 s=THROTTLE_DONE pgs=1250 cs=0 l=1 rx=0x7f9c5c004370 tx=0x7f9c5c010a00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:40:47.152 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:47.141+0000 7f9c6d7fa700 1 --2- 172.21.15.42:0/3300605564 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9c5003acc0 0x7f9c5003d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:40:47.152 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:47.141+0000 7f9c6d7fa700 1 -- 172.21.15.42:0/3300605564 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 8095+0+0 (secure 0 0 0) 0x7f9c5c04ad30 con 0x7f9c7012f630 2019-12-05T05:40:47.153 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:47.141+0000 7f9c74d04700 1 --2- 172.21.15.42:0/3300605564 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9c5003acc0 0x7f9c5003d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:40:47.153 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:47.141+0000 7f9c74d04700 1 --2- 172.21.15.42:0/3300605564 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9c5003acc0 0x7f9c5003d1d0 crc :-1 s=READY pgs=540 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:40:47.153 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:47.145+0000 7f9c76767700 1 -- 172.21.15.42:0/3300605564 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c7005d110 con 0x7f9c7012f630 2019-12-05T05:40:47.159 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:47.153+0000 7f9c6ffff700 1 --2- 172.21.15.42:0/3300605564 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9c7012f630 0x7f9c7012fa60 secure :-1 s=THROTTLE_DONE pgs=1250 cs=0 l=1 rx=0x7f9c5c004370 tx=0x7f9c5c010a00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:40:47.159 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:47.153+0000 7f9c6d7fa700 1 -- 172.21.15.42:0/3300605564 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f9c5c031d90 con 0x7f9c7012f630 2019-12-05T05:40:47.330 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:47.325+0000 7f9c76767700 1 -- 172.21.15.42:0/3300605564 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9c7005b8c0 con 0x7f9c7012f630 2019-12-05T05:40:47.332 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:47.329+0000 7f9c6ffff700 1 --2- 172.21.15.42:0/3300605564 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9c7012f630 0x7f9c7012fa60 secure :-1 s=THROTTLE_DONE pgs=1250 cs=0 l=1 rx=0x7f9c5c004370 tx=0x7f9c5c010a00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:40:47.332 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:47.329+0000 7f9c6d7fa700 1 -- 172.21.15.42:0/3300605564 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 153 v153) v1 ==== 95+0+2109 (secure 0 0 0) 0x7f9c5c031ff0 con 0x7f9c7012f630 2019-12-05T05:40:47.334 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:40:47.334 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":153,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7539,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/1233824070","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1233824070},{"type":"v1","addr":"172.21.15.42:6835","nonce":1233824070}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":153}],"filesystems":[{"mdsmap":{"epoch":152,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:39:14.336435+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7416},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7416":{"gid":7416,"name":"b","rank":0,"incarnation":149,"state":"up:active","state_seq":30,"addr":"172.21.15.40:6833/123054133","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":123054133},{"type":"v1","addr":"172.21.15.40:6833","nonce":123054133}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:40:47.334 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 153 2019-12-05T05:40:47.336 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:47.333+0000 7f9c76767700 1 -- 172.21.15.42:0/3300605564 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9c5003acc0 msgr2=0x7f9c5003d1d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:40:47.336 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:47.333+0000 7f9c76767700 1 --2- 172.21.15.42:0/3300605564 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9c5003acc0 0x7f9c5003d1d0 crc :-1 s=READY pgs=540 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:40:47.337 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:47.333+0000 7f9c76767700 1 -- 172.21.15.42:0/3300605564 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9c7012f630 msgr2=0x7f9c7012fa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:40:47.337 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:47.333+0000 7f9c76767700 1 --2- 172.21.15.42:0/3300605564 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9c7012f630 0x7f9c7012fa60 secure :-1 s=READY pgs=1250 cs=0 l=1 rx=0x7f9c5c004370 tx=0x7f9c5c010a00).stop 2019-12-05T05:40:47.337 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:47.333+0000 7f9c76767700 1 -- 172.21.15.42:0/3300605564 shutdown_connections 2019-12-05T05:40:47.337 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:47.333+0000 7f9c76767700 1 --2- 172.21.15.42:0/3300605564 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9c5003acc0 0x7f9c5003d1d0 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:40:47.337 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:47.333+0000 7f9c76767700 1 --2- 172.21.15.42:0/3300605564 >> v2:172.21.15.40:3300/0 conn(0x7f9c7007b420 0x7f9c7007b850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:40:47.337 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:47.333+0000 7f9c76767700 1 --2- 172.21.15.42:0/3300605564 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9c7012f630 0x7f9c7012fa60 unknown :-1 s=CLOSED pgs=1250 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:40:47.337 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:47.333+0000 7f9c76767700 1 -- 172.21.15.42:0/3300605564 shutdown_connections 2019-12-05T05:40:47.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:47.333+0000 7f9c76767700 1 -- 172.21.15.42:0/3300605564 wait complete. 2019-12-05T05:40:47.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:47.333+0000 7f9c76767700 1 -- 172.21.15.42:0/3300605564 >> 172.21.15.42:0/3300605564 conn(0x7f9c7007dd30 msgr2=0x7f9c700a9790 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:40:47.347 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T05:40:49.348 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:40:49.454 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:49.445+0000 7f3c56665700 1 Processor -- start 2019-12-05T05:40:49.454 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:49.445+0000 7f3c56665700 1 -- start start 2019-12-05T05:40:49.455 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:49.445+0000 7f3c56665700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3c500a7740 0x7f3c5008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:40:49.455 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:49.445+0000 7f3c56665700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3c5009fae0 0x7f3c5007eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:40:49.455 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:49.445+0000 7f3c56665700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c500a0710 con 0x7f3c5007e720 2019-12-05T05:40:49.455 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:49.445+0000 7f3c56665700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c500a0b00 con 0x7f3c500a7740 2019-12-05T05:40:49.455 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:49.445+0000 7f3c56665700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c5008e5b0 con 0x7f3c5009fae0 2019-12-05T05:40:49.455 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:49.445+0000 7f3c54c02700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3c500a7740 0x7f3c5008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:40:49.455 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:49.445+0000 7f3c4ffff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f3c5007e720 0x7f3c5009f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:38622/0 (socket says 172.21.15.42:38622) 2019-12-05T05:40:49.456 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:49.445+0000 7f3c4ffff700 1 -- 172.21.15.42:0/3672680065 learned_addr learned my addr 172.21.15.42:0/3672680065 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:40:49.456 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:49.445+0000 7f3c55403700 1 --2- 172.21.15.42:0/3672680065 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3c5009fae0 0x7f3c5007eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:40:49.456 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:49.445+0000 7f3c54c02700 1 -- 172.21.15.42:0/3672680065 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3c5009fae0 msgr2=0x7f3c5007eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:40:49.456 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:49.445+0000 7f3c54c02700 1 --2- 172.21.15.42:0/3672680065 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3c5009fae0 0x7f3c5007eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:40:49.456 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:49.445+0000 7f3c54c02700 1 -- 172.21.15.42:0/3672680065 >> v1:172.21.15.42:6789/0 conn(0x7f3c5007e720 legacy=0x7f3c5009f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:40:49.456 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:49.445+0000 7f3c4f7fe700 1 -- 172.21.15.42:0/3672680065 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1771552012 0 0) 0x7f3c500a0710 con 0x7f3c5007e720 2019-12-05T05:40:49.456 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:49.445+0000 7f3c54c02700 1 -- 172.21.15.42:0/3672680065 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c5007f0a0 con 0x7f3c500a7740 2019-12-05T05:40:49.457 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:49.445+0000 7f3c54c02700 1 --2- 172.21.15.42:0/3672680065 >> v2:172.21.15.42:3300/0 conn(0x7f3c500a7740 0x7f3c5008df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f3c44008790 tx=0x7f3c4400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:40:49.457 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:49.445+0000 7f3c54c02700 1 --2- 172.21.15.42:0/3672680065 >> v2:172.21.15.42:3300/0 conn(0x7f3c500a7740 0x7f3c5008df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f3c44008790 tx=0x7f3c4400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:40:49.457 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:49.445+0000 7f3c54c02700 1 --2- 172.21.15.42:0/3672680065 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3c500a7740 0x7f3c5008df80 secure :-1 s=READY pgs=1252 cs=0 l=1 rx=0x7f3c44008790 tx=0x7f3c4400fc70).ready entity=mon.1 client_cookie=c190ad0f7c644c05 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:40:49.457 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:49.445+0000 7f3c54c02700 1 --2- 172.21.15.42:0/3672680065 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3c500a7740 0x7f3c5008df80 secure :-1 s=THROTTLE_DONE pgs=1252 cs=0 l=1 rx=0x7f3c44008790 tx=0x7f3c4400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:40:49.457 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:49.449+0000 7f3c4f7fe700 1 -- 172.21.15.42:0/3672680065 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3c44008bb0 con 0x7f3c500a7740 2019-12-05T05:40:49.457 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:49.449+0000 7f3c54c02700 1 --2- 172.21.15.42:0/3672680065 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3c500a7740 0x7f3c5008df80 secure :-1 s=THROTTLE_DONE pgs=1252 cs=0 l=1 rx=0x7f3c44008790 tx=0x7f3c4400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:40:49.457 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:49.449+0000 7f3c54c02700 1 --2- 172.21.15.42:0/3672680065 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3c500a7740 0x7f3c5008df80 secure :-1 s=THROTTLE_DONE pgs=1252 cs=0 l=1 rx=0x7f3c44008790 tx=0x7f3c4400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:40:49.458 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:49.449+0000 7f3c4f7fe700 1 -- 172.21.15.42:0/3672680065 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c440087d0 con 0x7f3c500a7740 2019-12-05T05:40:49.458 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:49.449+0000 7f3c4f7fe700 1 -- 172.21.15.42:0/3672680065 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3c44011860 con 0x7f3c500a7740 2019-12-05T05:40:49.458 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:49.449+0000 7f3c56665700 1 -- 172.21.15.42:0/3672680065 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3c500a7740 msgr2=0x7f3c5008df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:40:49.458 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:49.449+0000 7f3c56665700 1 --2- 172.21.15.42:0/3672680065 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3c500a7740 0x7f3c5008df80 secure :-1 s=READY pgs=1252 cs=0 l=1 rx=0x7f3c44008790 tx=0x7f3c4400fc70).stop 2019-12-05T05:40:49.458 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:49.449+0000 7f3c56665700 1 -- 172.21.15.42:0/3672680065 shutdown_connections 2019-12-05T05:40:49.458 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:49.449+0000 7f3c56665700 1 --2- 172.21.15.42:0/3672680065 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3c5009fae0 0x7f3c5007eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:40:49.459 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:49.449+0000 7f3c56665700 1 --2- 172.21.15.42:0/3672680065 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3c500a7740 0x7f3c5008df80 unknown :-1 s=CLOSED pgs=1252 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:40:49.459 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:49.449+0000 7f3c56665700 1 -- 172.21.15.42:0/3672680065 shutdown_connections 2019-12-05T05:40:49.459 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:49.449+0000 7f3c56665700 1 -- 172.21.15.42:0/3672680065 wait complete. 2019-12-05T05:40:49.459 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:49.449+0000 7f3c56665700 1 -- 172.21.15.42:0/3672680065 >> 172.21.15.42:0/3672680065 conn(0x7f3c500aabe0 msgr2=0x7f3c500a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:40:49.462 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:49.449+0000 7f3c56665700 1 Processor -- start 2019-12-05T05:40:49.462 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:49.453+0000 7f3c56665700 1 -- start start 2019-12-05T05:40:49.463 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:49.453+0000 7f3c56665700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3c500a38c0 0x7f3c5011d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:40:49.463 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:49.453+0000 7f3c56665700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f3c50122a80 0x7f3c5008f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:40:49.463 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:49.453+0000 7f3c56665700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c50121e50 con 0x7f3c50122650 2019-12-05T05:40:49.463 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:49.453+0000 7f3c56665700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c500a3cf0 con 0x7f3c50122a80 2019-12-05T05:40:49.463 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:49.453+0000 7f3c56665700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c50131700 con 0x7f3c500a38c0 2019-12-05T05:40:49.463 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:49.453+0000 7f3c4ffff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f3c50122650 0x7f3c5008eb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:38630/0 (socket says 172.21.15.42:38630) 2019-12-05T05:40:49.463 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:49.453+0000 7f3c4ffff700 1 -- 172.21.15.42:0/1329352910 learned_addr learned my addr 172.21.15.42:0/1329352910 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:40:49.464 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:49.453+0000 7f3c54c02700 1 --2- 172.21.15.42:0/1329352910 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3c500a38c0 0x7f3c5011d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:40:49.464 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:49.453+0000 7f3c55403700 1 --2- 172.21.15.42:0/1329352910 >> v2:172.21.15.40:3300/0 conn(0x7f3c50122a80 0x7f3c5008f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:40:49.464 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:49.453+0000 7f3c4d7fa700 1 -- 172.21.15.42:0/1329352910 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2588436882 0 0) 0x7f3c3c0010c0 con 0x7f3c50122650 2019-12-05T05:40:49.464 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:49.453+0000 7f3c4d7fa700 1 -- 172.21.15.42:0/1329352910 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c300030b0 con 0x7f3c50122650 2019-12-05T05:40:49.464 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:49.453+0000 7f3c55403700 1 -- 172.21.15.42:0/1329352910 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3c500a38c0 msgr2=0x7f3c5011d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:40:49.464 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:49.453+0000 7f3c4d7fa700 1 -- 172.21.15.42:0/1329352910 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 3325909289 0 0) 0x7f3c300030b0 con 0x7f3c50122650 2019-12-05T05:40:49.464 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:49.453+0000 7f3c55403700 1 --2- 172.21.15.42:0/1329352910 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3c500a38c0 0x7f3c5011d960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:40:49.465 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:49.453+0000 7f3c55403700 1 -- 172.21.15.42:0/1329352910 >> v1:172.21.15.42:6789/0 conn(0x7f3c50122650 legacy=0x7f3c5008eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:40:49.465 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:49.453+0000 7f3c55403700 1 -- 172.21.15.42:0/1329352910 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c50119180 con 0x7f3c50122a80 2019-12-05T05:40:49.465 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:49.453+0000 7f3c55403700 1 --2- 172.21.15.42:0/1329352910 >> v2:172.21.15.40:3300/0 conn(0x7f3c50122a80 0x7f3c5008f370 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f3c3c001df0 tx=0x7f3c48010e40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:40:49.465 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:49.453+0000 7f3c55403700 1 --2- 172.21.15.42:0/1329352910 >> v2:172.21.15.40:3300/0 conn(0x7f3c50122a80 0x7f3c5008f370 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f3c3c001df0 tx=0x7f3c48010e40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:40:49.465 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:49.453+0000 7f3c55403700 1 --2- 172.21.15.42:0/1329352910 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3c50122a80 0x7f3c5008f370 secure :-1 s=READY pgs=974 cs=0 l=1 rx=0x7f3c3c001df0 tx=0x7f3c48010e40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:40:49.465 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:49.453+0000 7f3c55403700 1 --2- 172.21.15.42:0/1329352910 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3c50122a80 0x7f3c5008f370 secure :-1 s=THROTTLE_DONE pgs=974 cs=0 l=1 rx=0x7f3c3c001df0 tx=0x7f3c48010e40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:40:49.465 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:49.453+0000 7f3c4d7fa700 1 -- 172.21.15.42:0/1329352910 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3c3c001bb0 con 0x7f3c50122a80 2019-12-05T05:40:49.466 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:49.453+0000 7f3c55403700 1 --2- 172.21.15.42:0/1329352910 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3c50122a80 0x7f3c5008f370 secure :-1 s=THROTTLE_DONE pgs=974 cs=0 l=1 rx=0x7f3c3c001df0 tx=0x7f3c48010e40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:40:49.466 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:49.453+0000 7f3c55403700 1 --2- 172.21.15.42:0/1329352910 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3c50122a80 0x7f3c5008f370 secure :-1 s=THROTTLE_DONE pgs=974 cs=0 l=1 rx=0x7f3c3c001df0 tx=0x7f3c48010e40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:40:49.466 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:49.453+0000 7f3c4d7fa700 1 -- 172.21.15.42:0/1329352910 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c3c0019c0 con 0x7f3c50122a80 2019-12-05T05:40:49.466 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:49.453+0000 7f3c56665700 1 -- 172.21.15.42:0/1329352910 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c50122eb0 con 0x7f3c50122a80 2019-12-05T05:40:49.466 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:49.453+0000 7f3c4d7fa700 1 -- 172.21.15.42:0/1329352910 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3c480128b0 con 0x7f3c50122a80 2019-12-05T05:40:49.466 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:49.453+0000 7f3c56665700 1 -- 172.21.15.42:0/1329352910 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c5008f8c0 con 0x7f3c50122a80 2019-12-05T05:40:49.466 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:49.457+0000 7f3c55403700 1 --2- 172.21.15.42:0/1329352910 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3c50122a80 0x7f3c5008f370 secure :-1 s=THROTTLE_DONE pgs=974 cs=0 l=1 rx=0x7f3c3c001df0 tx=0x7f3c48010e40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:40:49.467 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:49.457+0000 7f3c4d7fa700 1 -- 172.21.15.42:0/1329352910 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f3c48012b00 con 0x7f3c50122a80 2019-12-05T05:40:49.467 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:49.457+0000 7f3c55403700 1 --2- 172.21.15.42:0/1329352910 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3c50122a80 0x7f3c5008f370 secure :-1 s=THROTTLE_DONE pgs=974 cs=0 l=1 rx=0x7f3c3c001df0 tx=0x7f3c48010e40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:40:49.467 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:49.457+0000 7f3c4d7fa700 1 --2- 172.21.15.42:0/1329352910 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3c3003b210 0x7f3c3003d720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:40:49.467 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:49.457+0000 7f3c4d7fa700 1 -- 172.21.15.42:0/1329352910 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 8095+0+0 (secure 0 0 0) 0x7f3c4804b5e0 con 0x7f3c50122a80 2019-12-05T05:40:49.467 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:49.457+0000 7f3c54c02700 1 --2- 172.21.15.42:0/1329352910 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3c3003b210 0x7f3c3003d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:40:49.467 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:49.457+0000 7f3c54c02700 1 --2- 172.21.15.42:0/1329352910 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3c3003b210 0x7f3c3003d720 crc :-1 s=READY pgs=541 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:40:49.468 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:49.461+0000 7f3c56665700 1 -- 172.21.15.42:0/1329352910 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c5005d110 con 0x7f3c50122a80 2019-12-05T05:40:49.474 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:49.469+0000 7f3c55403700 1 --2- 172.21.15.42:0/1329352910 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3c50122a80 0x7f3c5008f370 secure :-1 s=THROTTLE_DONE pgs=974 cs=0 l=1 rx=0x7f3c3c001df0 tx=0x7f3c48010e40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:40:49.474 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:49.469+0000 7f3c4d7fa700 1 -- 172.21.15.42:0/1329352910 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f3c50122eb0 con 0x7f3c50122a80 2019-12-05T05:40:49.646 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:49.641+0000 7f3c56665700 1 -- 172.21.15.42:0/1329352910 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3c5005b8c0 con 0x7f3c50122a80 2019-12-05T05:40:49.647 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:49.641+0000 7f3c55403700 1 --2- 172.21.15.42:0/1329352910 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3c50122a80 0x7f3c5008f370 secure :-1 s=THROTTLE_DONE pgs=974 cs=0 l=1 rx=0x7f3c3c001df0 tx=0x7f3c48010e40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:40:49.647 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:49.641+0000 7f3c4d7fa700 1 -- 172.21.15.42:0/1329352910 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 153 v153) v1 ==== 95+0+2109 (secure 0 0 0) 0x7f3c50119180 con 0x7f3c50122a80 2019-12-05T05:40:49.649 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:40:49.649 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":153,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7539,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/1233824070","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1233824070},{"type":"v1","addr":"172.21.15.42:6835","nonce":1233824070}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":153}],"filesystems":[{"mdsmap":{"epoch":152,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:39:14.336435+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7416},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7416":{"gid":7416,"name":"b","rank":0,"incarnation":149,"state":"up:active","state_seq":30,"addr":"172.21.15.40:6833/123054133","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":123054133},{"type":"v1","addr":"172.21.15.40:6833","nonce":123054133}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:40:49.649 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 153 2019-12-05T05:40:49.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:49.645+0000 7f3c56665700 1 -- 172.21.15.42:0/1329352910 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3c3003b210 msgr2=0x7f3c3003d720 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:40:49.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:49.645+0000 7f3c56665700 1 --2- 172.21.15.42:0/1329352910 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3c3003b210 0x7f3c3003d720 crc :-1 s=READY pgs=541 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:40:49.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:49.645+0000 7f3c56665700 1 -- 172.21.15.42:0/1329352910 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3c50122a80 msgr2=0x7f3c5008f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:40:49.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:49.649+0000 7f3c56665700 1 --2- 172.21.15.42:0/1329352910 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3c50122a80 0x7f3c5008f370 secure :-1 s=READY pgs=974 cs=0 l=1 rx=0x7f3c3c001df0 tx=0x7f3c48010e40).stop 2019-12-05T05:40:49.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:49.649+0000 7f3c56665700 1 -- 172.21.15.42:0/1329352910 shutdown_connections 2019-12-05T05:40:49.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:49.649+0000 7f3c56665700 1 --2- 172.21.15.42:0/1329352910 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3c3003b210 0x7f3c3003d720 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:40:49.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:49.649+0000 7f3c56665700 1 --2- 172.21.15.42:0/1329352910 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3c50122a80 0x7f3c5008f370 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:40:49.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:49.649+0000 7f3c56665700 1 --2- 172.21.15.42:0/1329352910 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3c500a38c0 0x7f3c5011d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:40:49.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:49.649+0000 7f3c56665700 1 -- 172.21.15.42:0/1329352910 shutdown_connections 2019-12-05T05:40:49.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:49.649+0000 7f3c56665700 1 -- 172.21.15.42:0/1329352910 wait complete. 2019-12-05T05:40:49.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:49.649+0000 7f3c56665700 1 -- 172.21.15.42:0/1329352910 >> 172.21.15.42:0/1329352910 conn(0x7f3c500a7740 msgr2=0x7f3c50059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:40:49.662 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T05:40:51.663 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:40:51.774 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:51.765+0000 7fa3799b2700 1 Processor -- start 2019-12-05T05:40:51.775 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:51.765+0000 7fa3799b2700 1 -- start start 2019-12-05T05:40:51.775 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:51.765+0000 7fa3799b2700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa3740a7730 0x7fa37408df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:40:51.775 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:51.765+0000 7fa3799b2700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fa37409fad0 0x7fa37407eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:40:51.775 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:51.765+0000 7fa3799b2700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3740a0700 con 0x7fa37407e710 2019-12-05T05:40:51.775 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:51.765+0000 7fa3799b2700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3740a0af0 con 0x7fa37409fad0 2019-12-05T05:40:51.775 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:51.765+0000 7fa3799b2700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa37408e5a0 con 0x7fa3740a7730 2019-12-05T05:40:51.776 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:51.765+0000 7fa373fff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fa37409fad0 0x7fa37407eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:40:51.776 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:51.765+0000 7fa373fff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fa37409fad0 0x7fa37407eb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:38602/0 (socket says 172.21.15.42:38602) 2019-12-05T05:40:51.776 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:51.765+0000 7fa373fff700 1 -- 172.21.15.42:0/3348385579 learned_addr learned my addr 172.21.15.42:0/3348385579 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:40:51.776 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:51.765+0000 7fa3737fe700 1 --2- 172.21.15.42:0/3348385579 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa3740a7730 0x7fa37408df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:40:51.776 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:51.769+0000 7fa373fff700 1 -- 172.21.15.42:0/3348385579 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa3740a7730 msgr2=0x7fa37408df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:40:51.776 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:51.769+0000 7fa373fff700 1 --2- 172.21.15.42:0/3348385579 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa3740a7730 0x7fa37408df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:40:51.776 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:51.769+0000 7fa373fff700 1 -- 172.21.15.42:0/3348385579 >> v1:172.21.15.40:6789/0 conn(0x7fa37407e710 legacy=0x7fa37409f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:40:51.777 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:51.769+0000 7fa373fff700 1 -- 172.21.15.42:0/3348385579 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa37407f090 con 0x7fa37409fad0 2019-12-05T05:40:51.777 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:51.769+0000 7fa3737fe700 1 --2- 172.21.15.42:0/3348385579 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa3740a7730 0x7fa37408df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T05:40:51.777 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:51.769+0000 7fa373fff700 1 --2- 172.21.15.42:0/3348385579 >> v2:172.21.15.42:3300/0 conn(0x7fa37409fad0 0x7fa37407eb40 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fa36c008b10 tx=0x7fa36c0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:40:51.777 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:51.769+0000 7fa373fff700 1 --2- 172.21.15.42:0/3348385579 >> v2:172.21.15.42:3300/0 conn(0x7fa37409fad0 0x7fa37407eb40 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fa36c008b10 tx=0x7fa36c0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:40:51.777 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:51.769+0000 7fa373fff700 1 --2- 172.21.15.42:0/3348385579 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa37409fad0 0x7fa37407eb40 secure :-1 s=READY pgs=1254 cs=0 l=1 rx=0x7fa36c008b10 tx=0x7fa36c0100b0).ready entity=mon.1 client_cookie=dd227ee892ee645 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:40:51.777 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:51.769+0000 7fa373fff700 1 --2- 172.21.15.42:0/3348385579 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa37409fad0 0x7fa37407eb40 secure :-1 s=THROTTLE_DONE pgs=1254 cs=0 l=1 rx=0x7fa36c008b10 tx=0x7fa36c0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:40:51.778 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:51.769+0000 7fa372ffd700 1 -- 172.21.15.42:0/3348385579 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa36c010da0 con 0x7fa37409fad0 2019-12-05T05:40:51.778 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:51.769+0000 7fa373fff700 1 --2- 172.21.15.42:0/3348385579 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa37409fad0 0x7fa37407eb40 secure :-1 s=THROTTLE_DONE pgs=1254 cs=0 l=1 rx=0x7fa36c008b10 tx=0x7fa36c0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:40:51.778 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:51.769+0000 7fa373fff700 1 --2- 172.21.15.42:0/3348385579 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa37409fad0 0x7fa37407eb40 secure :-1 s=THROTTLE_DONE pgs=1254 cs=0 l=1 rx=0x7fa36c008b10 tx=0x7fa36c0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:40:51.778 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:51.769+0000 7fa372ffd700 1 -- 172.21.15.42:0/3348385579 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa36c0109c0 con 0x7fa37409fad0 2019-12-05T05:40:51.778 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:51.769+0000 7fa372ffd700 1 -- 172.21.15.42:0/3348385579 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa36c0114a0 con 0x7fa37409fad0 2019-12-05T05:40:51.779 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:51.769+0000 7fa3799b2700 1 -- 172.21.15.42:0/3348385579 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa37409fad0 msgr2=0x7fa37407eb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:40:51.779 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:51.769+0000 7fa3799b2700 1 --2- 172.21.15.42:0/3348385579 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa37409fad0 0x7fa37407eb40 secure :-1 s=READY pgs=1254 cs=0 l=1 rx=0x7fa36c008b10 tx=0x7fa36c0100b0).stop 2019-12-05T05:40:51.779 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:51.769+0000 7fa3799b2700 1 -- 172.21.15.42:0/3348385579 shutdown_connections 2019-12-05T05:40:51.779 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:51.769+0000 7fa3799b2700 1 --2- 172.21.15.42:0/3348385579 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa37409fad0 0x7fa37407eb40 unknown :-1 s=CLOSED pgs=1254 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:40:51.779 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:51.769+0000 7fa3799b2700 1 --2- 172.21.15.42:0/3348385579 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa3740a7730 0x7fa37408df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:40:51.779 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:51.769+0000 7fa3799b2700 1 -- 172.21.15.42:0/3348385579 shutdown_connections 2019-12-05T05:40:51.779 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:51.769+0000 7fa3799b2700 1 -- 172.21.15.42:0/3348385579 wait complete. 2019-12-05T05:40:51.780 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:51.769+0000 7fa3799b2700 1 -- 172.21.15.42:0/3348385579 >> 172.21.15.42:0/3348385579 conn(0x7fa3740aabd0 msgr2=0x7fa3740a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:40:51.783 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:51.773+0000 7fa3799b2700 1 Processor -- start 2019-12-05T05:40:51.783 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:51.773+0000 7fa3799b2700 1 -- start start 2019-12-05T05:40:51.783 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:51.773+0000 7fa3799b2700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa3741445e0 0x7fa3741436f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:40:51.783 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:51.773+0000 7fa3799b2700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fa37412d490 0x7fa37412f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:40:51.783 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:51.773+0000 7fa3799b2700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa374124ec0 con 0x7fa374124a90 2019-12-05T05:40:51.784 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:51.773+0000 7fa3799b2700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa37407ace0 con 0x7fa3741445e0 2019-12-05T05:40:51.784 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:51.773+0000 7fa3799b2700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3741431b0 con 0x7fa37412d490 2019-12-05T05:40:51.784 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:51.773+0000 7fa373fff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fa37412d490 0x7fa37412f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:40:51.784 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:51.773+0000 7fa373fff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fa37412d490 0x7fa37412f8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:38606/0 (socket says 172.21.15.42:38606) 2019-12-05T05:40:51.784 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:51.773+0000 7fa373fff700 1 -- 172.21.15.42:0/351785168 learned_addr learned my addr 172.21.15.42:0/351785168 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:40:51.784 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:51.773+0000 7fa3737fe700 1 --2- 172.21.15.42:0/351785168 >> v2:172.21.15.40:3300/0 conn(0x7fa3741445e0 0x7fa3741436f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:40:51.785 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:51.773+0000 7fa373fff700 1 -- 172.21.15.42:0/351785168 >> v2:172.21.15.40:3300/0 conn(0x7fa3741445e0 msgr2=0x7fa3741436f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:40:51.785 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:51.773+0000 7fa373fff700 1 --2- 172.21.15.42:0/351785168 >> v2:172.21.15.40:3300/0 conn(0x7fa3741445e0 0x7fa3741436f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:40:51.785 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:51.773+0000 7fa373fff700 1 -- 172.21.15.42:0/351785168 >> v1:172.21.15.40:6789/0 conn(0x7fa374124a90 legacy=0x7fa37407c160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:40:51.785 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:51.773+0000 7fa373fff700 1 -- 172.21.15.42:0/351785168 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa37407cd90 con 0x7fa37412d490 2019-12-05T05:40:51.785 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:51.773+0000 7fa373fff700 1 --2- 172.21.15.42:0/351785168 >> v2:172.21.15.42:3300/0 conn(0x7fa37412d490 0x7fa37412f8d0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fa36c008b10 tx=0x7fa36c010f40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:40:51.785 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:51.773+0000 7fa373fff700 1 --2- 172.21.15.42:0/351785168 >> v2:172.21.15.42:3300/0 conn(0x7fa37412d490 0x7fa37412f8d0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fa36c008b10 tx=0x7fa36c010f40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:40:51.785 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:51.773+0000 7fa373fff700 1 --2- 172.21.15.42:0/351785168 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa37412d490 0x7fa37412f8d0 secure :-1 s=READY pgs=1255 cs=0 l=1 rx=0x7fa36c008b10 tx=0x7fa36c010f40).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:40:51.786 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:51.773+0000 7fa373fff700 1 --2- 172.21.15.42:0/351785168 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa37412d490 0x7fa37412f8d0 secure :-1 s=THROTTLE_DONE pgs=1255 cs=0 l=1 rx=0x7fa36c008b10 tx=0x7fa36c010f40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:40:51.786 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:51.773+0000 7fa370ff9700 1 -- 172.21.15.42:0/351785168 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa36c011c30 con 0x7fa37412d490 2019-12-05T05:40:51.786 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:51.773+0000 7fa373fff700 1 --2- 172.21.15.42:0/351785168 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa37412d490 0x7fa37412f8d0 secure :-1 s=THROTTLE_DONE pgs=1255 cs=0 l=1 rx=0x7fa36c008b10 tx=0x7fa36c010f40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:40:51.786 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:51.773+0000 7fa373fff700 1 --2- 172.21.15.42:0/351785168 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa37412d490 0x7fa37412f8d0 secure :-1 s=THROTTLE_DONE pgs=1255 cs=0 l=1 rx=0x7fa36c008b10 tx=0x7fa36c010f40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:40:51.786 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:51.773+0000 7fa370ff9700 1 -- 172.21.15.42:0/351785168 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa36c0110c0 con 0x7fa37412d490 2019-12-05T05:40:51.786 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:51.773+0000 7fa3799b2700 1 -- 172.21.15.42:0/351785168 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa37407c960 con 0x7fa37412d490 2019-12-05T05:40:51.786 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:51.773+0000 7fa370ff9700 1 -- 172.21.15.42:0/351785168 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa36c005ca0 con 0x7fa37412d490 2019-12-05T05:40:51.787 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:51.773+0000 7fa3799b2700 1 -- 172.21.15.42:0/351785168 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa374143e40 con 0x7fa37412d490 2019-12-05T05:40:51.787 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:51.777+0000 7fa373fff700 1 --2- 172.21.15.42:0/351785168 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa37412d490 0x7fa37412f8d0 secure :-1 s=THROTTLE_DONE pgs=1255 cs=0 l=1 rx=0x7fa36c008b10 tx=0x7fa36c010f40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:40:51.787 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:51.777+0000 7fa370ff9700 1 -- 172.21.15.42:0/351785168 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fa36c00f280 con 0x7fa37412d490 2019-12-05T05:40:51.787 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:51.777+0000 7fa373fff700 1 --2- 172.21.15.42:0/351785168 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa37412d490 0x7fa37412f8d0 secure :-1 s=THROTTLE_DONE pgs=1255 cs=0 l=1 rx=0x7fa36c008b10 tx=0x7fa36c010f40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:40:51.787 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:51.777+0000 7fa370ff9700 1 --2- 172.21.15.42:0/351785168 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa35803ae70 0x7fa35803d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:40:51.787 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:51.777+0000 7fa370ff9700 1 -- 172.21.15.42:0/351785168 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 8095+0+0 (secure 0 0 0) 0x7fa36c04b5c0 con 0x7fa37412d490 2019-12-05T05:40:51.788 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:51.777+0000 7fa3737fe700 1 --2- 172.21.15.42:0/351785168 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa35803ae70 0x7fa35803d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:40:51.788 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:51.777+0000 7fa3737fe700 1 --2- 172.21.15.42:0/351785168 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa35803ae70 0x7fa35803d380 crc :-1 s=READY pgs=542 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:40:51.788 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:51.781+0000 7fa3799b2700 1 -- 172.21.15.42:0/351785168 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa37405d100 con 0x7fa37412d490 2019-12-05T05:40:51.792 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:51.789+0000 7fa373fff700 1 --2- 172.21.15.42:0/351785168 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa37412d490 0x7fa37412f8d0 secure :-1 s=THROTTLE_DONE pgs=1255 cs=0 l=1 rx=0x7fa36c008b10 tx=0x7fa36c010f40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:40:51.792 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:51.789+0000 7fa370ff9700 1 -- 172.21.15.42:0/351785168 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fa36c032640 con 0x7fa37412d490 2019-12-05T05:40:51.965 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:51.961+0000 7fa3799b2700 1 -- 172.21.15.42:0/351785168 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa37405b8b0 con 0x7fa37412d490 2019-12-05T05:40:51.966 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:51.961+0000 7fa373fff700 1 --2- 172.21.15.42:0/351785168 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa37412d490 0x7fa37412f8d0 secure :-1 s=THROTTLE_DONE pgs=1255 cs=0 l=1 rx=0x7fa36c008b10 tx=0x7fa36c010f40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:40:51.967 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:51.961+0000 7fa370ff9700 1 -- 172.21.15.42:0/351785168 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 153 v153) v1 ==== 95+0+2109 (secure 0 0 0) 0x7fa36c032360 con 0x7fa37412d490 2019-12-05T05:40:51.967 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 153 2019-12-05T05:40:51.967 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:40:51.967 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":153,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7539,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/1233824070","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1233824070},{"type":"v1","addr":"172.21.15.42:6835","nonce":1233824070}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":153}],"filesystems":[{"mdsmap":{"epoch":152,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:39:14.336435+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7416},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7416":{"gid":7416,"name":"b","rank":0,"incarnation":149,"state":"up:active","state_seq":30,"addr":"172.21.15.40:6833/123054133","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":123054133},{"type":"v1","addr":"172.21.15.40:6833","nonce":123054133}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:40:51.969 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:51.965+0000 7fa3799b2700 1 -- 172.21.15.42:0/351785168 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa35803ae70 msgr2=0x7fa35803d380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:40:51.969 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:51.965+0000 7fa3799b2700 1 --2- 172.21.15.42:0/351785168 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa35803ae70 0x7fa35803d380 crc :-1 s=READY pgs=542 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:40:51.970 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:51.965+0000 7fa3799b2700 1 -- 172.21.15.42:0/351785168 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa37412d490 msgr2=0x7fa37412f8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:40:51.970 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:51.965+0000 7fa3799b2700 1 --2- 172.21.15.42:0/351785168 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa37412d490 0x7fa37412f8d0 secure :-1 s=READY pgs=1255 cs=0 l=1 rx=0x7fa36c008b10 tx=0x7fa36c010f40).stop 2019-12-05T05:40:51.970 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:51.965+0000 7fa3799b2700 1 -- 172.21.15.42:0/351785168 shutdown_connections 2019-12-05T05:40:51.970 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:51.965+0000 7fa3799b2700 1 --2- 172.21.15.42:0/351785168 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa35803ae70 0x7fa35803d380 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:40:51.970 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:51.965+0000 7fa3799b2700 1 --2- 172.21.15.42:0/351785168 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa37412d490 0x7fa37412f8d0 unknown :-1 s=CLOSED pgs=1255 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:40:51.970 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:51.965+0000 7fa3799b2700 1 --2- 172.21.15.42:0/351785168 >> v2:172.21.15.40:3300/0 conn(0x7fa3741445e0 0x7fa3741436f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:40:51.979 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:51.965+0000 7fa3799b2700 1 -- 172.21.15.42:0/351785168 shutdown_connections 2019-12-05T05:40:51.979 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:51.965+0000 7fa3799b2700 1 -- 172.21.15.42:0/351785168 wait complete. 2019-12-05T05:40:51.979 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:51.965+0000 7fa3799b2700 1 -- 172.21.15.42:0/351785168 >> 172.21.15.42:0/351785168 conn(0x7fa3740a6d20 msgr2=0x7fa37408f030 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:40:51.980 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T05:40:53.982 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:40:54.086 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:54.077+0000 7fc66b417700 1 Processor -- start 2019-12-05T05:40:54.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:54.077+0000 7fc66b417700 1 -- start start 2019-12-05T05:40:54.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:54.077+0000 7fc66b417700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc66408df80 0x7fc66409f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:40:54.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:54.077+0000 7fc66b417700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc66409fc60 con 0x7fc6640a7740 2019-12-05T05:40:54.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:54.077+0000 7fc66b417700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6640a0710 con 0x7fc66409f830 2019-12-05T05:40:54.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:54.077+0000 7fc66b417700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6640a0b00 con 0x7fc66408df80 2019-12-05T05:40:54.088 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:54.077+0000 7fc6691b3700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc66408df80 0x7fc66409f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:40:54.088 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:54.077+0000 7fc66a1b5700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fc66409f830 0x7fc66414efe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:38664/0 (socket says 172.21.15.42:38664) 2019-12-05T05:40:54.088 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:54.077+0000 7fc66a1b5700 1 -- 172.21.15.42:0/1506376590 learned_addr learned my addr 172.21.15.42:0/1506376590 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:40:54.088 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:54.077+0000 7fc6689b2700 1 -- 172.21.15.42:0/1506376590 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2463515317 0 0) 0x7fc66409fc60 con 0x7fc6640a7740 2019-12-05T05:40:54.088 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:54.077+0000 7fc6689b2700 1 -- 172.21.15.42:0/1506376590 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6500030b0 con 0x7fc6640a7740 2019-12-05T05:40:54.088 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:54.077+0000 7fc6691b3700 1 -- 172.21.15.42:0/1506376590 >> v1:172.21.15.42:6789/0 conn(0x7fc66409f830 legacy=0x7fc66414efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:40:54.088 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:54.081+0000 7fc6691b3700 1 -- 172.21.15.42:0/1506376590 >> v1:172.21.15.40:6789/0 conn(0x7fc6640a7740 legacy=0x7fc66407e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:40:54.089 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:54.077+0000 7fc6689b2700 1 -- 172.21.15.42:0/1506376590 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3392521734 0 0) 0x7fc6640a0710 con 0x7fc66409f830 2019-12-05T05:40:54.089 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:54.081+0000 7fc6691b3700 1 -- 172.21.15.42:0/1506376590 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc66407ef20 con 0x7fc66408df80 2019-12-05T05:40:54.089 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:54.081+0000 7fc6691b3700 1 --2- 172.21.15.42:0/1506376590 >> v2:172.21.15.42:3300/0 conn(0x7fc66408df80 0x7fc66409f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fc658008b10 tx=0x7fc65800ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:40:54.089 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:54.081+0000 7fc6691b3700 1 --2- 172.21.15.42:0/1506376590 >> v2:172.21.15.42:3300/0 conn(0x7fc66408df80 0x7fc66409f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fc658008b10 tx=0x7fc65800ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:40:54.089 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:54.081+0000 7fc6691b3700 1 --2- 172.21.15.42:0/1506376590 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc66408df80 0x7fc66409f2e0 secure :-1 s=READY pgs=1257 cs=0 l=1 rx=0x7fc658008b10 tx=0x7fc65800ff70).ready entity=mon.1 client_cookie=f5043740fb820940 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:40:54.089 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:54.081+0000 7fc6691b3700 1 --2- 172.21.15.42:0/1506376590 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc66408df80 0x7fc66409f2e0 secure :-1 s=THROTTLE_DONE pgs=1257 cs=0 l=1 rx=0x7fc658008b10 tx=0x7fc65800ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:40:54.089 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:54.081+0000 7fc6691b3700 1 --2- 172.21.15.42:0/1506376590 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc66408df80 0x7fc66409f2e0 secure :-1 s=THROTTLE_DONE pgs=1257 cs=0 l=1 rx=0x7fc658008b10 tx=0x7fc65800ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:40:54.090 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:54.081+0000 7fc6689b2700 1 -- 172.21.15.42:0/1506376590 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc6580119d0 con 0x7fc66408df80 2019-12-05T05:40:54.090 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:54.081+0000 7fc6691b3700 1 --2- 172.21.15.42:0/1506376590 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc66408df80 0x7fc66409f2e0 secure :-1 s=THROTTLE_DONE pgs=1257 cs=0 l=1 rx=0x7fc658008b10 tx=0x7fc65800ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:40:54.090 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:54.081+0000 7fc6689b2700 1 -- 172.21.15.42:0/1506376590 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6580115f0 con 0x7fc66408df80 2019-12-05T05:40:54.090 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:54.081+0000 7fc6689b2700 1 -- 172.21.15.42:0/1506376590 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc65800a160 con 0x7fc66408df80 2019-12-05T05:40:54.090 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:54.081+0000 7fc66b417700 1 -- 172.21.15.42:0/1506376590 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc66408df80 msgr2=0x7fc66409f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:40:54.090 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:54.081+0000 7fc66b417700 1 --2- 172.21.15.42:0/1506376590 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc66408df80 0x7fc66409f2e0 secure :-1 s=READY pgs=1257 cs=0 l=1 rx=0x7fc658008b10 tx=0x7fc65800ff70).stop 2019-12-05T05:40:54.090 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:54.081+0000 7fc66b417700 1 -- 172.21.15.42:0/1506376590 shutdown_connections 2019-12-05T05:40:54.091 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:54.081+0000 7fc66b417700 1 --2- 172.21.15.42:0/1506376590 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc66408df80 0x7fc66409f2e0 unknown :-1 s=CLOSED pgs=1257 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:40:54.091 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:54.081+0000 7fc66b417700 1 -- 172.21.15.42:0/1506376590 shutdown_connections 2019-12-05T05:40:54.091 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:54.081+0000 7fc66b417700 1 -- 172.21.15.42:0/1506376590 wait complete. 2019-12-05T05:40:54.091 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:54.081+0000 7fc66b417700 1 -- 172.21.15.42:0/1506376590 >> 172.21.15.42:0/1506376590 conn(0x7fc6640aabe0 msgr2=0x7fc6640a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:40:54.095 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:54.085+0000 7fc66b417700 1 Processor -- start 2019-12-05T05:40:54.096 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:54.085+0000 7fc66b417700 1 -- start start 2019-12-05T05:40:54.096 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:54.085+0000 7fc66b417700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc66411a890 0x7fc664119ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:40:54.096 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:54.085+0000 7fc66b417700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc6640722c0 0x7fc664074700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:40:54.096 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:54.085+0000 7fc66b417700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc66411a1f0 con 0x7fc664117a70 2019-12-05T05:40:54.096 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:54.085+0000 7fc66b417700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6640a1930 con 0x7fc66411a890 2019-12-05T05:40:54.096 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:54.085+0000 7fc66b417700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6640a1bd0 con 0x7fc6640722c0 2019-12-05T05:40:54.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:54.085+0000 7fc66a1b5700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc6640722c0 0x7fc664074700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:40:54.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:54.085+0000 7fc6699b4700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc66411a890 0x7fc664119ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:40:54.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:54.085+0000 7fc66a1b5700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc6640722c0 0x7fc664074700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:39098/0 (socket says 172.21.15.42:39098) 2019-12-05T05:40:54.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:54.085+0000 7fc66a1b5700 1 -- 172.21.15.42:0/981286402 learned_addr learned my addr 172.21.15.42:0/981286402 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:40:54.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:54.085+0000 7fc6699b4700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc66411a890 0x7fc664119ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.42:54678/0 (socket says 172.21.15.42:54678) 2019-12-05T05:40:54.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:54.085+0000 7fc66a1b5700 1 -- 172.21.15.42:0/981286402 >> v2:172.21.15.40:3300/0 conn(0x7fc66411a890 msgr2=0x7fc664119ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:40:54.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:54.085+0000 7fc66a1b5700 1 --2- 172.21.15.42:0/981286402 >> v2:172.21.15.40:3300/0 conn(0x7fc66411a890 0x7fc664119ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:40:54.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:54.085+0000 7fc66a1b5700 1 -- 172.21.15.42:0/981286402 >> v1:172.21.15.40:6789/0 conn(0x7fc664117a70 legacy=0x7fc664117ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:40:54.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:54.085+0000 7fc66a1b5700 1 -- 172.21.15.42:0/981286402 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6640a1510 con 0x7fc6640722c0 2019-12-05T05:40:54.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:54.085+0000 7fc66a1b5700 1 --2- 172.21.15.42:0/981286402 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc6640722c0 0x7fc664074700 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fc65c0043a0 tx=0x7fc65c010860).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:40:54.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:54.085+0000 7fc66a1b5700 1 --2- 172.21.15.42:0/981286402 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc6640722c0 0x7fc664074700 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fc65c0043a0 tx=0x7fc65c010860).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:40:54.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:54.085+0000 7fc66a1b5700 1 --2- 172.21.15.42:0/981286402 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc6640722c0 0x7fc664074700 secure :-1 s=READY pgs=260 cs=0 l=1 rx=0x7fc65c0043a0 tx=0x7fc65c010860).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:40:54.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:54.089+0000 7fc66a1b5700 1 --2- 172.21.15.42:0/981286402 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc6640722c0 0x7fc664074700 secure :-1 s=THROTTLE_DONE pgs=260 cs=0 l=1 rx=0x7fc65c0043a0 tx=0x7fc65c010860).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:40:54.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:54.089+0000 7fc66a1b5700 1 --2- 172.21.15.42:0/981286402 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc6640722c0 0x7fc664074700 secure :-1 s=THROTTLE_DONE pgs=260 cs=0 l=1 rx=0x7fc65c0043a0 tx=0x7fc65c010860).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:40:54.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:54.089+0000 7fc6567fc700 1 -- 172.21.15.42:0/981286402 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc65c011590 con 0x7fc6640722c0 2019-12-05T05:40:54.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:54.089+0000 7fc66a1b5700 1 --2- 172.21.15.42:0/981286402 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc6640722c0 0x7fc664074700 secure :-1 s=THROTTLE_DONE pgs=260 cs=0 l=1 rx=0x7fc65c0043a0 tx=0x7fc65c010860).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:40:54.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:54.089+0000 7fc6567fc700 1 -- 172.21.15.42:0/981286402 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc65c0111b0 con 0x7fc6640722c0 2019-12-05T05:40:54.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:54.089+0000 7fc6567fc700 1 -- 172.21.15.42:0/981286402 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc65c011b70 con 0x7fc6640722c0 2019-12-05T05:40:54.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:54.089+0000 7fc66b417700 1 -- 172.21.15.42:0/981286402 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc664074c50 con 0x7fc6640722c0 2019-12-05T05:40:54.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:54.089+0000 7fc66b417700 1 -- 172.21.15.42:0/981286402 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc664075200 con 0x7fc6640722c0 2019-12-05T05:40:54.100 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:54.089+0000 7fc66a1b5700 1 --2- 172.21.15.42:0/981286402 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc6640722c0 0x7fc664074700 secure :-1 s=THROTTLE_DONE pgs=260 cs=0 l=1 rx=0x7fc65c0043a0 tx=0x7fc65c010860).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:40:54.100 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:54.089+0000 7fc6567fc700 1 -- 172.21.15.42:0/981286402 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fc65c00b330 con 0x7fc6640722c0 2019-12-05T05:40:54.100 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:54.089+0000 7fc66a1b5700 1 --2- 172.21.15.42:0/981286402 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc6640722c0 0x7fc664074700 secure :-1 s=THROTTLE_DONE pgs=260 cs=0 l=1 rx=0x7fc65c0043a0 tx=0x7fc65c010860).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:40:54.100 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:54.089+0000 7fc6567fc700 1 --2- 172.21.15.42:0/981286402 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc64c03ae60 0x7fc64c03d370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:40:54.100 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:54.089+0000 7fc6567fc700 1 -- 172.21.15.42:0/981286402 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 8095+0+0 (secure 0 0 0) 0x7fc65c04b100 con 0x7fc6640722c0 2019-12-05T05:40:54.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:54.089+0000 7fc6699b4700 1 --2- 172.21.15.42:0/981286402 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc64c03ae60 0x7fc64c03d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:40:54.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:54.089+0000 7fc6699b4700 1 --2- 172.21.15.42:0/981286402 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc64c03ae60 0x7fc64c03d370 crc :-1 s=READY pgs=543 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:40:54.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:54.093+0000 7fc66b417700 1 -- 172.21.15.42:0/981286402 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc66405d110 con 0x7fc6640722c0 2019-12-05T05:40:54.107 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:54.105+0000 7fc66a1b5700 1 --2- 172.21.15.42:0/981286402 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc6640722c0 0x7fc664074700 secure :-1 s=THROTTLE_DONE pgs=260 cs=0 l=1 rx=0x7fc65c0043a0 tx=0x7fc65c010860).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:40:54.107 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:54.105+0000 7fc6567fc700 1 -- 172.21.15.42:0/981286402 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fc65c032180 con 0x7fc6640722c0 2019-12-05T05:40:54.278 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:54.273+0000 7fc66b417700 1 -- 172.21.15.42:0/981286402 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc66405b8c0 con 0x7fc6640722c0 2019-12-05T05:40:54.278 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:54.273+0000 7fc66a1b5700 1 --2- 172.21.15.42:0/981286402 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc6640722c0 0x7fc664074700 secure :-1 s=THROTTLE_DONE pgs=260 cs=0 l=1 rx=0x7fc65c0043a0 tx=0x7fc65c010860).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:40:54.279 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:54.273+0000 7fc6567fc700 1 -- 172.21.15.42:0/981286402 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 153 v153) v1 ==== 95+0+2109 (secure 0 0 0) 0x7fc65c0323e0 con 0x7fc6640722c0 2019-12-05T05:40:54.281 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:40:54.281 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":153,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7539,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/1233824070","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1233824070},{"type":"v1","addr":"172.21.15.42:6835","nonce":1233824070}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":153}],"filesystems":[{"mdsmap":{"epoch":152,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:39:14.336435+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7416},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7416":{"gid":7416,"name":"b","rank":0,"incarnation":149,"state":"up:active","state_seq":30,"addr":"172.21.15.40:6833/123054133","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":123054133},{"type":"v1","addr":"172.21.15.40:6833","nonce":123054133}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:40:54.281 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 153 2019-12-05T05:40:54.283 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:54.277+0000 7fc66b417700 1 -- 172.21.15.42:0/981286402 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc64c03ae60 msgr2=0x7fc64c03d370 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:40:54.284 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:54.277+0000 7fc66b417700 1 --2- 172.21.15.42:0/981286402 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc64c03ae60 0x7fc64c03d370 crc :-1 s=READY pgs=543 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:40:54.284 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:54.277+0000 7fc66b417700 1 -- 172.21.15.42:0/981286402 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc6640722c0 msgr2=0x7fc664074700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:40:54.284 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:54.277+0000 7fc66b417700 1 --2- 172.21.15.42:0/981286402 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc6640722c0 0x7fc664074700 secure :-1 s=READY pgs=260 cs=0 l=1 rx=0x7fc65c0043a0 tx=0x7fc65c010860).stop 2019-12-05T05:40:54.284 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:54.277+0000 7fc66b417700 1 -- 172.21.15.42:0/981286402 shutdown_connections 2019-12-05T05:40:54.284 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:54.277+0000 7fc66b417700 1 --2- 172.21.15.42:0/981286402 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc64c03ae60 0x7fc64c03d370 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:40:54.284 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:54.277+0000 7fc66b417700 1 --2- 172.21.15.42:0/981286402 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc6640722c0 0x7fc664074700 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:40:54.284 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:54.277+0000 7fc66b417700 1 --2- 172.21.15.42:0/981286402 >> v2:172.21.15.40:3300/0 conn(0x7fc66411a890 0x7fc664119ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:40:54.285 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:54.281+0000 7fc66b417700 1 -- 172.21.15.42:0/981286402 shutdown_connections 2019-12-05T05:40:54.285 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:54.281+0000 7fc66b417700 1 -- 172.21.15.42:0/981286402 wait complete. 2019-12-05T05:40:54.285 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:54.281+0000 7fc66b417700 1 -- 172.21.15.42:0/981286402 >> 172.21.15.42:0/981286402 conn(0x7fc66409fa90 msgr2=0x7fc664059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:40:54.294 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T05:40:56.295 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:40:56.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:56.397+0000 7f1a88dcc700 1 Processor -- start 2019-12-05T05:40:56.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:56.397+0000 7f1a88dcc700 1 -- start start 2019-12-05T05:40:56.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:56.397+0000 7f1a88dcc700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f1a8409fae0 0x7f1a8414fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:40:56.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:56.397+0000 7f1a88dcc700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a840a0710 con 0x7f1a840a7740 2019-12-05T05:40:56.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:56.397+0000 7f1a88dcc700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a8407ef20 con 0x7f1a8408df80 2019-12-05T05:40:56.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:56.397+0000 7f1a88dcc700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a840a0b00 con 0x7f1a8409fae0 2019-12-05T05:40:56.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:56.397+0000 7f1a8259c700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f1a8408df80 0x7f1a8409f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:38676/0 (socket says 172.21.15.42:38676) 2019-12-05T05:40:56.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:56.397+0000 7f1a8259c700 1 -- 172.21.15.42:0/3724948853 learned_addr learned my addr 172.21.15.42:0/3724948853 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:40:56.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:56.397+0000 7f1a8359e700 1 --2- 172.21.15.42:0/3724948853 >> v2:172.21.15.40:3300/0 conn(0x7f1a8409fae0 0x7f1a8414fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:40:56.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:56.397+0000 7f1a81d9b700 1 -- 172.21.15.42:0/3724948853 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2296438009 0 0) 0x7f1a8407ef20 con 0x7f1a8408df80 2019-12-05T05:40:56.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:56.397+0000 7f1a81d9b700 1 -- 172.21.15.42:0/3724948853 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a700030b0 con 0x7f1a8408df80 2019-12-05T05:40:56.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:56.397+0000 7f1a8359e700 1 -- 172.21.15.42:0/3724948853 >> v1:172.21.15.42:6789/0 conn(0x7f1a8408df80 legacy=0x7f1a8409f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:40:56.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:56.397+0000 7f1a8359e700 1 -- 172.21.15.42:0/3724948853 >> v1:172.21.15.40:6789/0 conn(0x7f1a840a7740 legacy=0x7f1a8407e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:40:56.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:56.397+0000 7f1a8359e700 1 -- 172.21.15.42:0/3724948853 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a840a9ba0 con 0x7f1a8409fae0 2019-12-05T05:40:56.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:56.397+0000 7f1a8359e700 1 --2- 172.21.15.42:0/3724948853 >> v2:172.21.15.40:3300/0 conn(0x7f1a8409fae0 0x7f1a8414fff0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f1a74008b10 tx=0x7f1a7400ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:40:56.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:56.397+0000 7f1a8359e700 1 --2- 172.21.15.42:0/3724948853 >> v2:172.21.15.40:3300/0 conn(0x7f1a8409fae0 0x7f1a8414fff0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f1a74008b10 tx=0x7f1a7400ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:40:56.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:56.397+0000 7f1a8359e700 1 --2- 172.21.15.42:0/3724948853 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1a8409fae0 0x7f1a8414fff0 secure :-1 s=READY pgs=980 cs=0 l=1 rx=0x7f1a74008b10 tx=0x7f1a7400ff70).ready entity=mon.0 client_cookie=cd1c8821e3ec22b6 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:40:56.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:56.397+0000 7f1a8359e700 1 --2- 172.21.15.42:0/3724948853 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1a8409fae0 0x7f1a8414fff0 secure :-1 s=THROTTLE_DONE pgs=980 cs=0 l=1 rx=0x7f1a74008b10 tx=0x7f1a7400ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:40:56.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:56.397+0000 7f1a81d9b700 1 -- 172.21.15.42:0/3724948853 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1a74010970 con 0x7f1a8409fae0 2019-12-05T05:40:56.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:56.397+0000 7f1a8359e700 1 --2- 172.21.15.42:0/3724948853 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1a8409fae0 0x7f1a8414fff0 secure :-1 s=THROTTLE_DONE pgs=980 cs=0 l=1 rx=0x7f1a74008b10 tx=0x7f1a7400ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:40:56.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:56.397+0000 7f1a8359e700 1 --2- 172.21.15.42:0/3724948853 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1a8409fae0 0x7f1a8414fff0 secure :-1 s=THROTTLE_DONE pgs=980 cs=0 l=1 rx=0x7f1a74008b10 tx=0x7f1a7400ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:40:56.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:56.397+0000 7f1a81d9b700 1 -- 172.21.15.42:0/3724948853 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a74008b50 con 0x7f1a8409fae0 2019-12-05T05:40:56.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:56.397+0000 7f1a81d9b700 1 -- 172.21.15.42:0/3724948853 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1a740110c0 con 0x7f1a8409fae0 2019-12-05T05:40:56.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:56.397+0000 7f1a88dcc700 1 -- 172.21.15.42:0/3724948853 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1a8409fae0 msgr2=0x7f1a8414fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:40:56.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:56.397+0000 7f1a88dcc700 1 --2- 172.21.15.42:0/3724948853 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1a8409fae0 0x7f1a8414fff0 secure :-1 s=READY pgs=980 cs=0 l=1 rx=0x7f1a74008b10 tx=0x7f1a7400ff70).stop 2019-12-05T05:40:56.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:56.401+0000 7f1a88dcc700 1 -- 172.21.15.42:0/3724948853 shutdown_connections 2019-12-05T05:40:56.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:56.401+0000 7f1a88dcc700 1 --2- 172.21.15.42:0/3724948853 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1a8409fae0 0x7f1a8414fff0 unknown :-1 s=CLOSED pgs=980 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:40:56.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:56.401+0000 7f1a88dcc700 1 -- 172.21.15.42:0/3724948853 shutdown_connections 2019-12-05T05:40:56.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:56.401+0000 7f1a88dcc700 1 -- 172.21.15.42:0/3724948853 wait complete. 2019-12-05T05:40:56.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:56.401+0000 7f1a88dcc700 1 -- 172.21.15.42:0/3724948853 >> 172.21.15.42:0/3724948853 conn(0x7f1a840aabe0 msgr2=0x7f1a840a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:40:56.413 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:56.401+0000 7f1a88dcc700 1 Processor -- start 2019-12-05T05:40:56.413 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:56.401+0000 7f1a88dcc700 1 -- start start 2019-12-05T05:40:56.413 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:56.401+0000 7f1a88dcc700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1a84143780 0x7f1a84122db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:40:56.413 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:56.401+0000 7f1a88dcc700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f1a840a3960 0x7f1a84079fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:40:56.413 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:56.401+0000 7f1a88dcc700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a8407b740 con 0x7f1a84123300 2019-12-05T05:40:56.413 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:56.401+0000 7f1a88dcc700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a84122640 con 0x7f1a840a3960 2019-12-05T05:40:56.414 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:56.401+0000 7f1a88dcc700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a84143c50 con 0x7f1a84143780 2019-12-05T05:40:56.414 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:56.401+0000 7f1a8259c700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f1a840a3960 0x7f1a84079fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:40:56.414 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:56.401+0000 7f1a82d9d700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1a84143780 0x7f1a84122db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:40:56.414 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:56.401+0000 7f1a8359e700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f1a84123300 0x7f1a84125c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:48124/0 (socket says 172.21.15.42:48124) 2019-12-05T05:40:56.414 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:56.401+0000 7f1a8359e700 1 -- 172.21.15.42:0/1926302783 learned_addr learned my addr 172.21.15.42:0/1926302783 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:40:56.414 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:56.405+0000 7f1a7b7fe700 1 -- 172.21.15.42:0/1926302783 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3823291224 0 0) 0x7f1a8407b740 con 0x7f1a84123300 2019-12-05T05:40:56.414 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:56.405+0000 7f1a7b7fe700 1 -- 172.21.15.42:0/1926302783 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a680030b0 con 0x7f1a84123300 2019-12-05T05:40:56.414 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:56.405+0000 7f1a82d9d700 1 -- 172.21.15.42:0/1926302783 >> v2:172.21.15.40:3300/0 conn(0x7f1a840a3960 msgr2=0x7f1a84079fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:40:56.415 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:56.405+0000 7f1a82d9d700 1 --2- 172.21.15.42:0/1926302783 >> v2:172.21.15.40:3300/0 conn(0x7f1a840a3960 0x7f1a84079fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:40:56.415 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:56.405+0000 7f1a82d9d700 1 -- 172.21.15.42:0/1926302783 >> v1:172.21.15.40:6789/0 conn(0x7f1a84123300 legacy=0x7f1a84125c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:40:56.415 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:56.405+0000 7f1a82d9d700 1 -- 172.21.15.42:0/1926302783 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a84142530 con 0x7f1a84143780 2019-12-05T05:40:56.415 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:56.405+0000 7f1a8259c700 1 --2- 172.21.15.42:0/1926302783 >> v2:172.21.15.40:3300/0 conn(0x7f1a840a3960 0x7f1a84079fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:40:56.415 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:56.405+0000 7f1a82d9d700 1 --2- 172.21.15.42:0/1926302783 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1a84143780 0x7f1a84122db0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f1a7c000e90 tx=0x7f1a7c0108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:40:56.415 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:56.405+0000 7f1a82d9d700 1 --2- 172.21.15.42:0/1926302783 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1a84143780 0x7f1a84122db0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f1a7c000e90 tx=0x7f1a7c0108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:40:56.415 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:56.405+0000 7f1a82d9d700 1 --2- 172.21.15.42:0/1926302783 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1a84143780 0x7f1a84122db0 secure :-1 s=READY pgs=261 cs=0 l=1 rx=0x7f1a7c000e90 tx=0x7f1a7c0108f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:40:56.416 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:56.405+0000 7f1a82d9d700 1 --2- 172.21.15.42:0/1926302783 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1a84143780 0x7f1a84122db0 secure :-1 s=THROTTLE_DONE pgs=261 cs=0 l=1 rx=0x7f1a7c000e90 tx=0x7f1a7c0108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:40:56.416 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:56.405+0000 7f1a7b7fe700 1 -- 172.21.15.42:0/1926302783 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1a7c00ca80 con 0x7f1a84143780 2019-12-05T05:40:56.416 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:56.405+0000 7f1a82d9d700 1 --2- 172.21.15.42:0/1926302783 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1a84143780 0x7f1a84122db0 secure :-1 s=THROTTLE_DONE pgs=261 cs=0 l=1 rx=0x7f1a7c000e90 tx=0x7f1a7c0108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:40:56.416 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:56.405+0000 7f1a82d9d700 1 --2- 172.21.15.42:0/1926302783 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1a84143780 0x7f1a84122db0 secure :-1 s=THROTTLE_DONE pgs=261 cs=0 l=1 rx=0x7f1a7c000e90 tx=0x7f1a7c0108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:40:56.416 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:56.405+0000 7f1a7b7fe700 1 -- 172.21.15.42:0/1926302783 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a7c00c6a0 con 0x7f1a84143780 2019-12-05T05:40:56.416 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:56.405+0000 7f1a88dcc700 1 -- 172.21.15.42:0/1926302783 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a84144190 con 0x7f1a84143780 2019-12-05T05:40:56.416 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:56.405+0000 7f1a7b7fe700 1 -- 172.21.15.42:0/1926302783 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1a7c0110f0 con 0x7f1a84143780 2019-12-05T05:40:56.417 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:56.405+0000 7f1a88dcc700 1 -- 172.21.15.42:0/1926302783 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a8407ac00 con 0x7f1a84143780 2019-12-05T05:40:56.417 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:56.405+0000 7f1a82d9d700 1 --2- 172.21.15.42:0/1926302783 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1a84143780 0x7f1a84122db0 secure :-1 s=THROTTLE_DONE pgs=261 cs=0 l=1 rx=0x7f1a7c000e90 tx=0x7f1a7c0108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:40:56.417 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:56.405+0000 7f1a7b7fe700 1 -- 172.21.15.42:0/1926302783 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f1a7c0113c0 con 0x7f1a84143780 2019-12-05T05:40:56.417 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:56.405+0000 7f1a82d9d700 1 --2- 172.21.15.42:0/1926302783 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1a84143780 0x7f1a84122db0 secure :-1 s=THROTTLE_DONE pgs=261 cs=0 l=1 rx=0x7f1a7c000e90 tx=0x7f1a7c0108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:40:56.417 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:56.409+0000 7f1a7b7fe700 1 --2- 172.21.15.42:0/1926302783 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f1a6803b2c0 0x7f1a6803d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:40:56.417 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:56.409+0000 7f1a7b7fe700 1 -- 172.21.15.42:0/1926302783 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 8095+0+0 (secure 0 0 0) 0x7f1a7c04b120 con 0x7f1a84143780 2019-12-05T05:40:56.417 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:56.409+0000 7f1a8259c700 1 --2- 172.21.15.42:0/1926302783 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f1a6803b2c0 0x7f1a6803d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:40:56.418 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:56.409+0000 7f1a8259c700 1 --2- 172.21.15.42:0/1926302783 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f1a6803b2c0 0x7f1a6803d7d0 crc :-1 s=READY pgs=544 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:40:56.418 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:56.413+0000 7f1a88dcc700 1 -- 172.21.15.42:0/1926302783 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a8405d110 con 0x7f1a84143780 2019-12-05T05:40:56.426 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:56.421+0000 7f1a82d9d700 1 --2- 172.21.15.42:0/1926302783 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1a84143780 0x7f1a84122db0 secure :-1 s=THROTTLE_DONE pgs=261 cs=0 l=1 rx=0x7f1a7c000e90 tx=0x7f1a7c0108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:40:56.427 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:56.421+0000 7f1a7b7fe700 1 -- 172.21.15.42:0/1926302783 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f1a7c0321a0 con 0x7f1a84143780 2019-12-05T05:40:56.596 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:56.593+0000 7f1a88dcc700 1 -- 172.21.15.42:0/1926302783 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1a8405b8c0 con 0x7f1a84143780 2019-12-05T05:40:56.597 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:56.593+0000 7f1a82d9d700 1 --2- 172.21.15.42:0/1926302783 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1a84143780 0x7f1a84122db0 secure :-1 s=THROTTLE_DONE pgs=261 cs=0 l=1 rx=0x7f1a7c000e90 tx=0x7f1a7c0108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:40:56.597 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:56.593+0000 7f1a7b7fe700 1 -- 172.21.15.42:0/1926302783 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 153 v153) v1 ==== 95+0+2109 (secure 0 0 0) 0x7f1a7c032400 con 0x7f1a84143780 2019-12-05T05:40:56.599 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 153 2019-12-05T05:40:56.599 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:40:56.600 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":153,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7539,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/1233824070","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1233824070},{"type":"v1","addr":"172.21.15.42:6835","nonce":1233824070}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":153}],"filesystems":[{"mdsmap":{"epoch":152,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:39:14.336435+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7416},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7416":{"gid":7416,"name":"b","rank":0,"incarnation":149,"state":"up:active","state_seq":30,"addr":"172.21.15.40:6833/123054133","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":123054133},{"type":"v1","addr":"172.21.15.40:6833","nonce":123054133}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:40:56.602 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:56.597+0000 7f1a88dcc700 1 -- 172.21.15.42:0/1926302783 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f1a6803b2c0 msgr2=0x7f1a6803d7d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:40:56.602 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:56.597+0000 7f1a88dcc700 1 --2- 172.21.15.42:0/1926302783 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f1a6803b2c0 0x7f1a6803d7d0 crc :-1 s=READY pgs=544 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:40:56.602 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:56.597+0000 7f1a88dcc700 1 -- 172.21.15.42:0/1926302783 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1a84143780 msgr2=0x7f1a84122db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:40:56.602 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:56.597+0000 7f1a88dcc700 1 --2- 172.21.15.42:0/1926302783 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1a84143780 0x7f1a84122db0 secure :-1 s=READY pgs=261 cs=0 l=1 rx=0x7f1a7c000e90 tx=0x7f1a7c0108f0).stop 2019-12-05T05:40:56.603 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:56.597+0000 7f1a88dcc700 1 -- 172.21.15.42:0/1926302783 shutdown_connections 2019-12-05T05:40:56.603 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:56.597+0000 7f1a88dcc700 1 --2- 172.21.15.42:0/1926302783 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f1a6803b2c0 0x7f1a6803d7d0 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:40:56.603 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:56.597+0000 7f1a88dcc700 1 --2- 172.21.15.42:0/1926302783 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1a84143780 0x7f1a84122db0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:40:56.603 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:56.597+0000 7f1a88dcc700 1 --2- 172.21.15.42:0/1926302783 >> v2:172.21.15.40:3300/0 conn(0x7f1a840a3960 0x7f1a84079fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:40:56.603 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:56.597+0000 7f1a88dcc700 1 -- 172.21.15.42:0/1926302783 shutdown_connections 2019-12-05T05:40:56.603 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:56.597+0000 7f1a88dcc700 1 -- 172.21.15.42:0/1926302783 wait complete. 2019-12-05T05:40:56.603 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:56.597+0000 7f1a88dcc700 1 -- 172.21.15.42:0/1926302783 >> 172.21.15.42:0/1926302783 conn(0x7f1a8407bcd0 msgr2=0x7f1a840a6a60 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:40:56.612 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T05:40:58.613 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:40:58.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:58.713+0000 7f02776c3700 1 Processor -- start 2019-12-05T05:40:58.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:58.713+0000 7f02776c3700 1 -- start start 2019-12-05T05:40:58.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:58.713+0000 7f02776c3700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f02700a7740 0x7f027008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:40:58.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:58.713+0000 7f02776c3700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f027007e720 0x7f027007eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:40:58.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:58.713+0000 7f02776c3700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02700a0710 con 0x7f027009f2e0 2019-12-05T05:40:58.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:58.713+0000 7f02776c3700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02700a0b00 con 0x7f02700a7740 2019-12-05T05:40:58.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:58.713+0000 7f02776c3700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f027008e5b0 con 0x7f027007e720 2019-12-05T05:40:58.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:58.713+0000 7f0276461700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f027009f2e0 0x7f027009f710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:38696/0 (socket says 172.21.15.42:38696) 2019-12-05T05:40:58.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:58.713+0000 7f0276461700 1 -- 172.21.15.42:0/762238740 learned_addr learned my addr 172.21.15.42:0/762238740 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:40:58.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:58.713+0000 7f0275c60700 1 --2- 172.21.15.42:0/762238740 >> v2:172.21.15.40:3300/0 conn(0x7f02700a7740 0x7f027008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:40:58.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:58.713+0000 7f027545f700 1 --2- 172.21.15.42:0/762238740 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f027007e720 0x7f027007eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:40:58.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:58.713+0000 7f0274c5e700 1 -- 172.21.15.42:0/762238740 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1788987708 0 0) 0x7f02700a0710 con 0x7f027009f2e0 2019-12-05T05:40:58.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:58.713+0000 7f0274c5e700 1 -- 172.21.15.42:0/762238740 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f025c0030b0 con 0x7f027009f2e0 2019-12-05T05:40:58.724 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:58.713+0000 7f0275c60700 1 -- 172.21.15.42:0/762238740 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f027007e720 msgr2=0x7f027007eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:40:58.724 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:58.713+0000 7f0275c60700 1 --2- 172.21.15.42:0/762238740 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f027007e720 0x7f027007eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:40:58.724 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:58.713+0000 7f0274c5e700 1 -- 172.21.15.42:0/762238740 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1159361292 0 0) 0x7f025c0030b0 con 0x7f027009f2e0 2019-12-05T05:40:58.724 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:58.713+0000 7f0275c60700 1 -- 172.21.15.42:0/762238740 >> v1:172.21.15.42:6789/0 conn(0x7f027009f2e0 legacy=0x7f027009f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:40:58.724 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:58.713+0000 7f0275c60700 1 -- 172.21.15.42:0/762238740 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f027007f0a0 con 0x7f02700a7740 2019-12-05T05:40:58.724 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:58.713+0000 7f027545f700 1 --2- 172.21.15.42:0/762238740 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f027007e720 0x7f027007eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:40:58.725 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:58.713+0000 7f0275c60700 1 --2- 172.21.15.42:0/762238740 >> v2:172.21.15.40:3300/0 conn(0x7f02700a7740 0x7f027008df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f0268008790 tx=0x7f02600019d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:40:58.725 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:58.717+0000 7f0275c60700 1 --2- 172.21.15.42:0/762238740 >> v2:172.21.15.40:3300/0 conn(0x7f02700a7740 0x7f027008df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f0268008790 tx=0x7f02600019d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:40:58.725 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:58.717+0000 7f0275c60700 1 --2- 172.21.15.42:0/762238740 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f02700a7740 0x7f027008df80 secure :-1 s=READY pgs=982 cs=0 l=1 rx=0x7f0268008790 tx=0x7f02600019d0).ready entity=mon.0 client_cookie=6df4a81bfac16a2 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:40:58.725 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:58.717+0000 7f0275c60700 1 --2- 172.21.15.42:0/762238740 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f02700a7740 0x7f027008df80 secure :-1 s=THROTTLE_DONE pgs=982 cs=0 l=1 rx=0x7f0268008790 tx=0x7f02600019d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:40:58.725 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:58.717+0000 7f0274c5e700 1 -- 172.21.15.42:0/762238740 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0260001750 con 0x7f02700a7740 2019-12-05T05:40:58.725 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:58.717+0000 7f0275c60700 1 --2- 172.21.15.42:0/762238740 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f02700a7740 0x7f027008df80 secure :-1 s=THROTTLE_DONE pgs=982 cs=0 l=1 rx=0x7f0268008790 tx=0x7f02600019d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:40:58.726 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:58.717+0000 7f0274c5e700 1 -- 172.21.15.42:0/762238740 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0260001560 con 0x7f02700a7740 2019-12-05T05:40:58.726 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:58.717+0000 7f0275c60700 1 --2- 172.21.15.42:0/762238740 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f02700a7740 0x7f027008df80 secure :-1 s=THROTTLE_DONE pgs=982 cs=0 l=1 rx=0x7f0268008790 tx=0x7f02600019d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:40:58.726 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:58.717+0000 7f0274c5e700 1 -- 172.21.15.42:0/762238740 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f02680113e0 con 0x7f02700a7740 2019-12-05T05:40:58.726 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:58.717+0000 7f02776c3700 1 -- 172.21.15.42:0/762238740 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f02700a7740 msgr2=0x7f027008df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:40:58.726 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:58.717+0000 7f02776c3700 1 --2- 172.21.15.42:0/762238740 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f02700a7740 0x7f027008df80 secure :-1 s=READY pgs=982 cs=0 l=1 rx=0x7f0268008790 tx=0x7f02600019d0).stop 2019-12-05T05:40:58.726 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:58.717+0000 7f02776c3700 1 -- 172.21.15.42:0/762238740 shutdown_connections 2019-12-05T05:40:58.727 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:58.717+0000 7f02776c3700 1 --2- 172.21.15.42:0/762238740 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f02700a7740 0x7f027008df80 unknown :-1 s=CLOSED pgs=982 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:40:58.727 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:58.717+0000 7f02776c3700 1 --2- 172.21.15.42:0/762238740 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f027007e720 0x7f027007eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:40:58.727 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:58.717+0000 7f02776c3700 1 -- 172.21.15.42:0/762238740 shutdown_connections 2019-12-05T05:40:58.727 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:58.717+0000 7f02776c3700 1 -- 172.21.15.42:0/762238740 wait complete. 2019-12-05T05:40:58.727 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:58.717+0000 7f02776c3700 1 -- 172.21.15.42:0/762238740 >> 172.21.15.42:0/762238740 conn(0x7f02700aabe0 msgr2=0x7f02700a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:40:58.730 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:58.717+0000 7f02776c3700 1 Processor -- start 2019-12-05T05:40:58.730 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:58.717+0000 7f02776c3700 1 -- start start 2019-12-05T05:40:58.730 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:58.721+0000 7f02776c3700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0270121220 0x7f027007d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:40:58.731 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:58.721+0000 7f02776c3700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f027007de90 0x7f027011b820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:40:58.731 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:58.721+0000 7f02776c3700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f027011ab70 con 0x7f027012d110 2019-12-05T05:40:58.731 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:58.721+0000 7f02776c3700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0270117980 con 0x7f0270121220 2019-12-05T05:40:58.731 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:58.721+0000 7f02776c3700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0270117c20 con 0x7f027007de90 2019-12-05T05:40:58.731 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:58.721+0000 7f027545f700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0270121220 0x7f027007d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:40:58.731 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:58.721+0000 7f027545f700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0270121220 0x7f027007d940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:38660/0 (socket says 172.21.15.42:38660) 2019-12-05T05:40:58.732 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:58.721+0000 7f027545f700 1 -- 172.21.15.42:0/988744841 learned_addr learned my addr 172.21.15.42:0/988744841 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:40:58.732 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:58.721+0000 7f0276461700 1 --2- 172.21.15.42:0/988744841 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f027007de90 0x7f027011b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:40:58.732 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:58.721+0000 7f027545f700 1 -- 172.21.15.42:0/988744841 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f027007de90 msgr2=0x7f027011b820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:40:58.732 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:58.721+0000 7f027545f700 1 --2- 172.21.15.42:0/988744841 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f027007de90 0x7f027011b820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:40:58.732 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:58.721+0000 7f027545f700 1 -- 172.21.15.42:0/988744841 >> v1:172.21.15.42:6789/0 conn(0x7f027012d110 legacy=0x7f0270117ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:40:58.732 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:58.721+0000 7f027545f700 1 -- 172.21.15.42:0/988744841 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f027011bd70 con 0x7f0270121220 2019-12-05T05:40:58.732 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:58.721+0000 7f0276461700 1 --2- 172.21.15.42:0/988744841 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f027007de90 0x7f027011b820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T05:40:58.733 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:58.721+0000 7f027545f700 1 --2- 172.21.15.42:0/988744841 >> v2:172.21.15.42:3300/0 conn(0x7f0270121220 0x7f027007d940 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f0270117b40 tx=0x7f026c013370).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:40:58.733 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:58.721+0000 7f027545f700 1 --2- 172.21.15.42:0/988744841 >> v2:172.21.15.42:3300/0 conn(0x7f0270121220 0x7f027007d940 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f0270117b40 tx=0x7f026c013370).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:40:58.733 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:58.721+0000 7f027545f700 1 --2- 172.21.15.42:0/988744841 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0270121220 0x7f027007d940 secure :-1 s=READY pgs=1261 cs=0 l=1 rx=0x7f0270117b40 tx=0x7f026c013370).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:40:58.733 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:58.721+0000 7f027545f700 1 --2- 172.21.15.42:0/988744841 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0270121220 0x7f027007d940 secure :-1 s=THROTTLE_DONE pgs=1261 cs=0 l=1 rx=0x7f0270117b40 tx=0x7f026c013370).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:40:58.733 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:58.721+0000 7f02667fc700 1 -- 172.21.15.42:0/988744841 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f026c005ab0 con 0x7f0270121220 2019-12-05T05:40:58.733 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:58.721+0000 7f027545f700 1 --2- 172.21.15.42:0/988744841 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0270121220 0x7f027007d940 secure :-1 s=THROTTLE_DONE pgs=1261 cs=0 l=1 rx=0x7f0270117b40 tx=0x7f026c013370).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:40:58.734 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:58.721+0000 7f027545f700 1 --2- 172.21.15.42:0/988744841 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0270121220 0x7f027007d940 secure :-1 s=THROTTLE_DONE pgs=1261 cs=0 l=1 rx=0x7f0270117b40 tx=0x7f026c013370).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:40:58.734 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:58.721+0000 7f02667fc700 1 -- 172.21.15.42:0/988744841 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f026c014d00 con 0x7f0270121220 2019-12-05T05:40:58.734 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:58.721+0000 7f02776c3700 1 -- 172.21.15.42:0/988744841 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f027011bf80 con 0x7f0270121220 2019-12-05T05:40:58.734 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:58.721+0000 7f02667fc700 1 -- 172.21.15.42:0/988744841 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f026c007350 con 0x7f0270121220 2019-12-05T05:40:58.734 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:58.721+0000 7f02776c3700 1 -- 172.21.15.42:0/988744841 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f027008ed90 con 0x7f0270121220 2019-12-05T05:40:58.734 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:58.721+0000 7f027545f700 1 --2- 172.21.15.42:0/988744841 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0270121220 0x7f027007d940 secure :-1 s=THROTTLE_DONE pgs=1261 cs=0 l=1 rx=0x7f0270117b40 tx=0x7f026c013370).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:40:58.734 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:58.721+0000 7f02667fc700 1 -- 172.21.15.42:0/988744841 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f026c007550 con 0x7f0270121220 2019-12-05T05:40:58.734 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:58.721+0000 7f027545f700 1 --2- 172.21.15.42:0/988744841 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0270121220 0x7f027007d940 secure :-1 s=THROTTLE_DONE pgs=1261 cs=0 l=1 rx=0x7f0270117b40 tx=0x7f026c013370).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:40:58.735 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:58.721+0000 7f02667fc700 1 --2- 172.21.15.42:0/988744841 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f025803acf0 0x7f025803d200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:40:58.735 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:58.721+0000 7f02667fc700 1 -- 172.21.15.42:0/988744841 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 8095+0+0 (secure 0 0 0) 0x7f026c04d4a0 con 0x7f0270121220 2019-12-05T05:40:58.735 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:58.721+0000 7f0275c60700 1 --2- 172.21.15.42:0/988744841 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f025803acf0 0x7f025803d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:40:58.735 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:58.725+0000 7f0275c60700 1 --2- 172.21.15.42:0/988744841 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f025803acf0 0x7f025803d200 crc :-1 s=READY pgs=546 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:40:58.735 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:58.729+0000 7f02776c3700 1 -- 172.21.15.42:0/988744841 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f027005d110 con 0x7f0270121220 2019-12-05T05:40:58.738 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:58.733+0000 7f027545f700 1 --2- 172.21.15.42:0/988744841 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0270121220 0x7f027007d940 secure :-1 s=THROTTLE_DONE pgs=1261 cs=0 l=1 rx=0x7f0270117b40 tx=0x7f026c013370).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:40:58.738 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:58.733+0000 7f02667fc700 1 -- 172.21.15.42:0/988744841 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f026c034520 con 0x7f0270121220 2019-12-05T05:40:58.903 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:58.901+0000 7f02776c3700 1 -- 172.21.15.42:0/988744841 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f027005b8c0 con 0x7f0270121220 2019-12-05T05:40:58.911 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:58.905+0000 7f027545f700 1 --2- 172.21.15.42:0/988744841 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0270121220 0x7f027007d940 secure :-1 s=THROTTLE_DONE pgs=1261 cs=0 l=1 rx=0x7f0270117b40 tx=0x7f026c013370).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:40:58.911 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:58.905+0000 7f02667fc700 1 -- 172.21.15.42:0/988744841 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 154 v154) v1 ==== 95+0+1774 (secure 0 0 0) 0x7f026c034780 con 0x7f0270121220 2019-12-05T05:40:58.919 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 154 2019-12-05T05:40:58.919 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:40:58.919 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":154,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":154,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:40:58.897711+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7539},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7539":{"gid":7539,"name":"a","rank":0,"incarnation":154,"state":"up:replay","state_seq":2,"addr":"172.21.15.42:6835/1233824070","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1233824070},{"type":"v1","addr":"172.21.15.42:6835","nonce":1233824070}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:40:58.922 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:58.917+0000 7f02776c3700 1 -- 172.21.15.42:0/988744841 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f025803acf0 msgr2=0x7f025803d200 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:40:58.922 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:58.917+0000 7f02776c3700 1 --2- 172.21.15.42:0/988744841 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f025803acf0 0x7f025803d200 crc :-1 s=READY pgs=546 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:40:58.922 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:58.917+0000 7f02776c3700 1 -- 172.21.15.42:0/988744841 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0270121220 msgr2=0x7f027007d940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:40:58.922 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:58.917+0000 7f02776c3700 1 --2- 172.21.15.42:0/988744841 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0270121220 0x7f027007d940 secure :-1 s=READY pgs=1261 cs=0 l=1 rx=0x7f0270117b40 tx=0x7f026c013370).stop 2019-12-05T05:40:58.922 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:58.917+0000 7f02776c3700 1 -- 172.21.15.42:0/988744841 shutdown_connections 2019-12-05T05:40:58.922 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:58.917+0000 7f02776c3700 1 --2- 172.21.15.42:0/988744841 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f025803acf0 0x7f025803d200 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:40:58.923 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:58.917+0000 7f02776c3700 1 --2- 172.21.15.42:0/988744841 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f027007de90 0x7f027011b820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:40:58.923 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:58.917+0000 7f02776c3700 1 --2- 172.21.15.42:0/988744841 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0270121220 0x7f027007d940 unknown :-1 s=CLOSED pgs=1261 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:40:58.923 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:58.917+0000 7f02776c3700 1 -- 172.21.15.42:0/988744841 shutdown_connections 2019-12-05T05:40:58.923 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:58.917+0000 7f02776c3700 1 -- 172.21.15.42:0/988744841 wait complete. 2019-12-05T05:40:58.923 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:58.917+0000 7f02776c3700 1 -- 172.21.15.42:0/988744841 >> 172.21.15.42:0/988744841 conn(0x7f027007e4f0 msgr2=0x7f027011c540 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:40:58.937 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b down, removed from mdsmap 2019-12-05T05:40:58.937 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2019-12-05T05:40:58.938 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:40:59.042 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:59.033+0000 7fe4f4bb7700 1 Processor -- start 2019-12-05T05:40:59.042 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:59.033+0000 7fe4f4bb7700 1 -- start start 2019-12-05T05:40:59.049 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:59.033+0000 7fe4f4bb7700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe4f00a7730 0x7fe4f008df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:40:59.049 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:59.033+0000 7fe4f4bb7700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4f00a0700 con 0x7fe4f009fad0 2019-12-05T05:40:59.049 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:59.033+0000 7fe4f4bb7700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4f00a9b90 con 0x7fe4f007e710 2019-12-05T05:40:59.049 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:59.033+0000 7fe4f4bb7700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4f00a0af0 con 0x7fe4f00a7730 2019-12-05T05:40:59.049 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:59.033+0000 7fe4ee59c700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fe4f007e710 0x7fe4f009f2d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:38722/0 (socket says 172.21.15.42:38722) 2019-12-05T05:40:59.050 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:59.033+0000 7fe4ee59c700 1 -- 172.21.15.42:0/3766461563 learned_addr learned my addr 172.21.15.42:0/3766461563 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:40:59.050 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:59.033+0000 7fe4eed9d700 1 --2- 172.21.15.42:0/3766461563 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe4f00a7730 0x7fe4f008df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:40:59.050 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:59.033+0000 7fe4edd9b700 1 -- 172.21.15.42:0/3766461563 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1912518078 0 0) 0x7fe4f00a9b90 con 0x7fe4f007e710 2019-12-05T05:40:59.050 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:59.033+0000 7fe4edd9b700 1 -- 172.21.15.42:0/3766461563 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4dc0030b0 con 0x7fe4f007e710 2019-12-05T05:40:59.050 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:59.033+0000 7fe4edd9b700 1 -- 172.21.15.42:0/3766461563 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2889259732 0 0) 0x7fe4f00a0700 con 0x7fe4f009fad0 2019-12-05T05:40:59.050 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:59.033+0000 7fe4edd9b700 1 -- 172.21.15.42:0/3766461563 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4f00a9b90 con 0x7fe4f009fad0 2019-12-05T05:40:59.051 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:59.033+0000 7fe4edd9b700 1 -- 172.21.15.42:0/3766461563 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4047185887 0 0) 0x7fe4dc0030b0 con 0x7fe4f007e710 2019-12-05T05:40:59.051 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:59.033+0000 7fe4edd9b700 1 -- 172.21.15.42:0/3766461563 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe4f00a7730 msgr2=0x7fe4f008df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:40:59.051 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:59.033+0000 7fe4edd9b700 1 --2- 172.21.15.42:0/3766461563 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe4f00a7730 0x7fe4f008df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:40:59.051 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:59.033+0000 7fe4edd9b700 1 -- 172.21.15.42:0/3766461563 >> v1:172.21.15.40:6789/0 conn(0x7fe4f009fad0 legacy=0x7fe4f014efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:40:59.051 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:59.033+0000 7fe4edd9b700 1 -- 172.21.15.42:0/3766461563 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4f007eb40 con 0x7fe4f007e710 2019-12-05T05:40:59.052 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:59.033+0000 7fe4edd9b700 1 -- 172.21.15.42:0/3766461563 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7fe4d8001750 con 0x7fe4f007e710 2019-12-05T05:40:59.052 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:59.033+0000 7fe4edd9b700 1 -- 172.21.15.42:0/3766461563 >> v1:172.21.15.42:6789/0 conn(0x7fe4f007e710 legacy=0x7fe4f009f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:40:59.052 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:59.033+0000 7fe4edd9b700 1 --2- 172.21.15.42:0/3766461563 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe4dc003940 0x7fe4dc003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:40:59.052 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:59.033+0000 7fe4edd9b700 1 --2- 172.21.15.42:0/3766461563 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe4dc004420 0x7fe4dc004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:40:59.052 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:59.033+0000 7fe4edd9b700 1 --2- 172.21.15.42:0/3766461563 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe4dc004e40 0x7fe4dc04f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:40:59.052 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:59.033+0000 7fe4edd9b700 1 -- 172.21.15.42:0/3766461563 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4f00a0af0 con 0x7fe4dc004420 2019-12-05T05:40:59.053 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:59.033+0000 7fe4edd9b700 1 -- 172.21.15.42:0/3766461563 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4dc04fc00 con 0x7fe4dc004e40 2019-12-05T05:40:59.053 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:59.033+0000 7fe4edd9b700 1 -- 172.21.15.42:0/3766461563 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4dc04fe40 con 0x7fe4dc003940 2019-12-05T05:40:59.053 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:59.033+0000 7fe4ee59c700 1 --2- 172.21.15.42:0/3766461563 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe4dc004420 0x7fe4dc004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:40:59.053 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:59.033+0000 7fe4ef59e700 1 --2- 172.21.15.42:0/3766461563 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe4dc004e40 0x7fe4dc04f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:40:59.053 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:59.033+0000 7fe4eed9d700 1 --2- 172.21.15.42:0/3766461563 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe4dc003940 0x7fe4dc003d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:40:59.053 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:59.033+0000 7fe4ef59e700 1 -- 172.21.15.42:0/3766461563 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe4dc003940 msgr2=0x7fe4dc003d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:40:59.053 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:59.033+0000 7fe4ef59e700 1 --2- 172.21.15.42:0/3766461563 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe4dc003940 0x7fe4dc003d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:40:59.054 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:59.033+0000 7fe4ef59e700 1 -- 172.21.15.42:0/3766461563 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe4dc004420 msgr2=0x7fe4dc004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:40:59.054 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:59.033+0000 7fe4ef59e700 1 --2- 172.21.15.42:0/3766461563 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe4dc004420 0x7fe4dc004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:40:59.054 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:59.033+0000 7fe4ef59e700 1 -- 172.21.15.42:0/3766461563 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4dc050140 con 0x7fe4dc004e40 2019-12-05T05:40:59.054 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:59.033+0000 7fe4ef59e700 1 --2- 172.21.15.42:0/3766461563 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe4dc004e40 0x7fe4dc04f680 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fe4e8000fe0 tx=0x7fe4e800be40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:40:59.054 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:59.033+0000 7fe4ef59e700 1 --2- 172.21.15.42:0/3766461563 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe4dc004e40 0x7fe4dc04f680 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fe4e8000fe0 tx=0x7fe4e800be40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:40:59.054 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:59.033+0000 7fe4ef59e700 1 --2- 172.21.15.42:0/3766461563 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe4dc004e40 0x7fe4dc04f680 secure :-1 s=READY pgs=1263 cs=0 l=1 rx=0x7fe4e8000fe0 tx=0x7fe4e800be40).ready entity=mon.1 client_cookie=d807525cb146d3f7 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:40:59.054 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:59.037+0000 7fe4ef59e700 1 --2- 172.21.15.42:0/3766461563 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe4dc004e40 0x7fe4dc04f680 secure :-1 s=THROTTLE_DONE pgs=1263 cs=0 l=1 rx=0x7fe4e8000fe0 tx=0x7fe4e800be40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:40:59.055 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:59.037+0000 7fe4edd9b700 1 -- 172.21.15.42:0/3766461563 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe4e8010af0 con 0x7fe4dc004e40 2019-12-05T05:40:59.055 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:59.037+0000 7fe4ef59e700 1 --2- 172.21.15.42:0/3766461563 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe4dc004e40 0x7fe4dc04f680 secure :-1 s=THROTTLE_DONE pgs=1263 cs=0 l=1 rx=0x7fe4e8000fe0 tx=0x7fe4e800be40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:40:59.055 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:59.037+0000 7fe4edd9b700 1 -- 172.21.15.42:0/3766461563 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4e8010710 con 0x7fe4dc004e40 2019-12-05T05:40:59.055 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:59.037+0000 7fe4ef59e700 1 --2- 172.21.15.42:0/3766461563 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe4dc004e40 0x7fe4dc04f680 secure :-1 s=THROTTLE_DONE pgs=1263 cs=0 l=1 rx=0x7fe4e8000fe0 tx=0x7fe4e800be40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:40:59.055 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:59.037+0000 7fe4edd9b700 1 -- 172.21.15.42:0/3766461563 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe4e800c760 con 0x7fe4dc004e40 2019-12-05T05:40:59.055 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:59.037+0000 7fe4f4bb7700 1 -- 172.21.15.42:0/3766461563 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe4dc004e40 msgr2=0x7fe4dc04f680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:40:59.055 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:59.037+0000 7fe4f4bb7700 1 --2- 172.21.15.42:0/3766461563 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe4dc004e40 0x7fe4dc04f680 secure :-1 s=READY pgs=1263 cs=0 l=1 rx=0x7fe4e8000fe0 tx=0x7fe4e800be40).stop 2019-12-05T05:40:59.056 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:59.037+0000 7fe4ef59e700 1 -- 172.21.15.42:0/3766461563 reap_dead start 2019-12-05T05:40:59.056 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:59.037+0000 7fe4f4bb7700 1 -- 172.21.15.42:0/3766461563 shutdown_connections 2019-12-05T05:40:59.056 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:59.037+0000 7fe4f4bb7700 1 -- 172.21.15.42:0/3766461563 shutdown_connections 2019-12-05T05:40:59.056 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:59.037+0000 7fe4f4bb7700 1 -- 172.21.15.42:0/3766461563 wait complete. 2019-12-05T05:40:59.056 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:59.037+0000 7fe4f4bb7700 1 -- 172.21.15.42:0/3766461563 >> 172.21.15.42:0/3766461563 conn(0x7fe4f00aabd0 msgr2=0x7fe4f00a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:40:59.056 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:59.037+0000 7fe4f4bb7700 1 Processor -- start 2019-12-05T05:40:59.056 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:59.037+0000 7fe4f4bb7700 1 -- start start 2019-12-05T05:40:59.063 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:59.041+0000 7fe4f4bb7700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe4f008f670 0x7fe4f01304c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:40:59.063 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:59.041+0000 7fe4f4bb7700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe4f011a840 0x7fe4f0130a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:40:59.063 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:59.041+0000 7fe4f4bb7700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4f0130f60 con 0x7fe4f007d380 2019-12-05T05:40:59.063 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:59.041+0000 7fe4f4bb7700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4f0118ce0 con 0x7fe4f011a840 2019-12-05T05:40:59.063 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:59.041+0000 7fe4f4bb7700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4f01175e0 con 0x7fe4f008f670 2019-12-05T05:40:59.063 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:59.041+0000 7fe4ef59e700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fe4f007d380 0x7fe4f007d7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:38734/0 (socket says 172.21.15.42:38734) 2019-12-05T05:40:59.064 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:59.041+0000 7fe4ef59e700 1 -- 172.21.15.42:0/3771278256 learned_addr learned my addr 172.21.15.42:0/3771278256 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:40:59.064 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:59.041+0000 7fe4eed9d700 1 --2- 172.21.15.42:0/3771278256 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe4f008f670 0x7fe4f01304c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:40:59.064 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:59.041+0000 7fe4ee59c700 1 --2- 172.21.15.42:0/3771278256 >> v2:172.21.15.40:3300/0 conn(0x7fe4f011a840 0x7fe4f0130a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:40:59.064 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:59.041+0000 7fe4e77fe700 1 -- 172.21.15.42:0/3771278256 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2566029155 0 0) 0x7fe4f0130f60 con 0x7fe4f007d380 2019-12-05T05:40:59.064 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:59.041+0000 7fe4e77fe700 1 -- 172.21.15.42:0/3771278256 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4d40030b0 con 0x7fe4f007d380 2019-12-05T05:40:59.064 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:59.041+0000 7fe4e77fe700 1 -- 172.21.15.42:0/3771278256 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 2935857318 0 0) 0x7fe4d40030b0 con 0x7fe4f007d380 2019-12-05T05:40:59.064 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:59.041+0000 7fe4e77fe700 1 -- 172.21.15.42:0/3771278256 --> v1:172.21.15.42:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe4f0130f60 con 0x7fe4f007d380 2019-12-05T05:40:59.065 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:59.041+0000 7fe4e77fe700 1 -- 172.21.15.42:0/3771278256 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7fe4e8001340 con 0x7fe4f007d380 2019-12-05T05:40:59.065 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:59.041+0000 7fe4e77fe700 1 -- 172.21.15.42:0/3771278256 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe4f008f670 msgr2=0x7fe4f01304c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:40:59.065 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:59.041+0000 7fe4e77fe700 1 --2- 172.21.15.42:0/3771278256 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe4f008f670 0x7fe4f01304c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:40:59.065 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:59.041+0000 7fe4e77fe700 1 -- 172.21.15.42:0/3771278256 >> v2:172.21.15.40:3300/0 conn(0x7fe4f011a840 msgr2=0x7fe4f0130a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:40:59.066 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:59.041+0000 7fe4e77fe700 1 --2- 172.21.15.42:0/3771278256 >> v2:172.21.15.40:3300/0 conn(0x7fe4f011a840 0x7fe4f0130a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:40:59.066 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:59.041+0000 7fe4e77fe700 1 -- 172.21.15.42:0/3771278256 >> v1:172.21.15.42:6789/0 conn(0x7fe4f007d380 legacy=0x7fe4f007d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:40:59.066 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:59.041+0000 7fe4e77fe700 1 --2- 172.21.15.42:0/3771278256 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe4d40038a0 0x7fe4d4003d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:40:59.066 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:59.041+0000 7fe4e77fe700 1 --2- 172.21.15.42:0/3771278256 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe4d4004420 0x7fe4d4004850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:40:59.066 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:59.041+0000 7fe4e77fe700 1 --2- 172.21.15.42:0/3771278256 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe4d4004e90 0x7fe4d400b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:40:59.067 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:59.041+0000 7fe4e77fe700 1 -- 172.21.15.42:0/3771278256 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4f0118ce0 con 0x7fe4d40038a0 2019-12-05T05:40:59.067 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:59.041+0000 7fe4e77fe700 1 -- 172.21.15.42:0/3771278256 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4f01175e0 con 0x7fe4d4004420 2019-12-05T05:40:59.067 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:59.041+0000 7fe4e77fe700 1 -- 172.21.15.42:0/3771278256 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4d400bbc0 con 0x7fe4d4004e90 2019-12-05T05:40:59.067 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:59.041+0000 7fe4eed9d700 1 --2- 172.21.15.42:0/3771278256 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe4d40038a0 0x7fe4d4003d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:40:59.067 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:59.041+0000 7fe4ef59e700 1 --2- 172.21.15.42:0/3771278256 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe4d4004e90 0x7fe4d400b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:40:59.067 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:59.041+0000 7fe4ee59c700 1 --2- 172.21.15.42:0/3771278256 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe4d4004420 0x7fe4d4004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:40:59.068 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:59.041+0000 7fe4ee59c700 1 -- 172.21.15.42:0/3771278256 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe4d4004e90 msgr2=0x7fe4d400b670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:40:59.068 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:59.041+0000 7fe4ee59c700 1 --2- 172.21.15.42:0/3771278256 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe4d4004e90 0x7fe4d400b670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:40:59.068 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:59.041+0000 7fe4ee59c700 1 -- 172.21.15.42:0/3771278256 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe4d40038a0 msgr2=0x7fe4d4003d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:40:59.068 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:59.041+0000 7fe4ee59c700 1 --2- 172.21.15.42:0/3771278256 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe4d40038a0 0x7fe4d4003d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:40:59.068 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:59.041+0000 7fe4ee59c700 1 -- 172.21.15.42:0/3771278256 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4f0116f50 con 0x7fe4d4004420 2019-12-05T05:40:59.068 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:59.041+0000 7fe4f4bb7700 1 -- 172.21.15.42:0/3771278256 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4f007dfb0 con 0x7fe4d4004420 2019-12-05T05:40:59.069 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:59.041+0000 7fe4ee59c700 1 --2- 172.21.15.42:0/3771278256 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe4d4004420 0x7fe4d4004850 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fe4d8001520 tx=0x7fe4d8010f00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:40:59.069 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:59.041+0000 7fe4ee59c700 1 --2- 172.21.15.42:0/3771278256 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe4d4004420 0x7fe4d4004850 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fe4d8001520 tx=0x7fe4d8010f00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:40:59.069 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:59.041+0000 7fe4f4bb7700 1 -- 172.21.15.42:0/3771278256 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4f008f670 con 0x7fe4d4004420 2019-12-05T05:40:59.070 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:59.041+0000 7fe4ee59c700 1 --2- 172.21.15.42:0/3771278256 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe4d4004420 0x7fe4d4004850 secure :-1 s=READY pgs=1265 cs=0 l=1 rx=0x7fe4d8001520 tx=0x7fe4d8010f00).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:40:59.070 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:59.041+0000 7fe4ee59c700 1 --2- 172.21.15.42:0/3771278256 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe4d4004420 0x7fe4d4004850 secure :-1 s=THROTTLE_DONE pgs=1265 cs=0 l=1 rx=0x7fe4d8001520 tx=0x7fe4d8010f00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:40:59.070 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:59.041+0000 7fe4ee59c700 1 --2- 172.21.15.42:0/3771278256 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe4d4004420 0x7fe4d4004850 secure :-1 s=THROTTLE_DONE pgs=1265 cs=0 l=1 rx=0x7fe4d8001520 tx=0x7fe4d8010f00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:40:59.070 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:59.041+0000 7fe4ee59c700 1 --2- 172.21.15.42:0/3771278256 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe4d4004420 0x7fe4d4004850 secure :-1 s=THROTTLE_DONE pgs=1265 cs=0 l=1 rx=0x7fe4d8001520 tx=0x7fe4d8010f00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:40:59.070 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:59.041+0000 7fe4e77fe700 1 -- 172.21.15.42:0/3771278256 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe4d8011b30 con 0x7fe4d4004420 2019-12-05T05:40:59.070 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:59.041+0000 7fe4e77fe700 1 -- 172.21.15.42:0/3771278256 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4d8007220 con 0x7fe4d4004420 2019-12-05T05:40:59.071 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:59.041+0000 7fe4e77fe700 1 -- 172.21.15.42:0/3771278256 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe4d8008440 con 0x7fe4d4004420 2019-12-05T05:40:59.071 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:59.041+0000 7fe4ee59c700 1 --2- 172.21.15.42:0/3771278256 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe4d4004420 0x7fe4d4004850 secure :-1 s=THROTTLE_DONE pgs=1265 cs=0 l=1 rx=0x7fe4d8001520 tx=0x7fe4d8010f00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:40:59.071 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:59.041+0000 7fe4e77fe700 1 -- 172.21.15.42:0/3771278256 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fe4d8008640 con 0x7fe4d4004420 2019-12-05T05:40:59.071 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:59.041+0000 7fe4ee59c700 1 --2- 172.21.15.42:0/3771278256 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe4d4004420 0x7fe4d4004850 secure :-1 s=THROTTLE_DONE pgs=1265 cs=0 l=1 rx=0x7fe4d8001520 tx=0x7fe4d8010f00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:40:59.072 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:59.041+0000 7fe4e77fe700 1 --2- 172.21.15.42:0/3771278256 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe4d4043a70 0x7fe4d4045f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:40:59.072 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:59.041+0000 7fe4e77fe700 1 -- 172.21.15.42:0/3771278256 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 8181+0+0 (secure 0 0 0) 0x7fe4d804ac30 con 0x7fe4d4004420 2019-12-05T05:40:59.072 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:59.041+0000 7fe4eed9d700 1 --2- 172.21.15.42:0/3771278256 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe4d4043a70 0x7fe4d4045f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:40:59.072 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:59.041+0000 7fe4eed9d700 1 --2- 172.21.15.42:0/3771278256 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe4d4043a70 0x7fe4d4045f80 crc :-1 s=READY pgs=547 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:40:59.072 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:59.049+0000 7fe4f4bb7700 1 -- 172.21.15.42:0/3771278256 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4c8003200 con 0x7fe4d4004420 2019-12-05T05:40:59.072 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:59.053+0000 7fe4ee59c700 1 --2- 172.21.15.42:0/3771278256 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe4d4004420 0x7fe4d4004850 secure :-1 s=THROTTLE_DONE pgs=1265 cs=0 l=1 rx=0x7fe4d8001520 tx=0x7fe4d8010f00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:40:59.072 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:59.053+0000 7fe4e77fe700 1 -- 172.21.15.42:0/3771278256 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fe4d8032b00 con 0x7fe4d4004420 2019-12-05T05:40:59.240 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:59.237+0000 7fe4f4bb7700 1 -- 172.21.15.42:0/3771278256 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe4c8003630 con 0x7fe4d4004420 2019-12-05T05:40:59.241 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:59.237+0000 7fe4ee59c700 1 --2- 172.21.15.42:0/3771278256 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe4d4004420 0x7fe4d4004850 secure :-1 s=THROTTLE_DONE pgs=1265 cs=0 l=1 rx=0x7fe4d8001520 tx=0x7fe4d8010f00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:40:59.241 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:59.237+0000 7fe4e77fe700 1 -- 172.21.15.42:0/3771278256 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 154 v154) v1 ==== 95+0+1774 (secure 0 0 0) 0x7fe4d802f1c0 con 0x7fe4d4004420 2019-12-05T05:40:59.242 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 154 2019-12-05T05:40:59.242 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:40:59.242 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":154,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":154,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:40:58.897711+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7539},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7539":{"gid":7539,"name":"a","rank":0,"incarnation":154,"state":"up:replay","state_seq":2,"addr":"172.21.15.42:6835/1233824070","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1233824070},{"type":"v1","addr":"172.21.15.42:6835","nonce":1233824070}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:40:59.244 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:59.241+0000 7fe4f4bb7700 1 -- 172.21.15.42:0/3771278256 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe4d4043a70 msgr2=0x7fe4d4045f80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:40:59.244 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:59.241+0000 7fe4f4bb7700 1 --2- 172.21.15.42:0/3771278256 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe4d4043a70 0x7fe4d4045f80 crc :-1 s=READY pgs=547 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:40:59.244 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:59.241+0000 7fe4f4bb7700 1 -- 172.21.15.42:0/3771278256 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe4d4004420 msgr2=0x7fe4d4004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:40:59.245 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:59.241+0000 7fe4f4bb7700 1 --2- 172.21.15.42:0/3771278256 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe4d4004420 0x7fe4d4004850 secure :-1 s=READY pgs=1265 cs=0 l=1 rx=0x7fe4d8001520 tx=0x7fe4d8010f00).stop 2019-12-05T05:40:59.245 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:59.241+0000 7fe4ef59e700 1 -- 172.21.15.42:0/3771278256 reap_dead start 2019-12-05T05:40:59.245 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:59.241+0000 7fe4f4bb7700 1 -- 172.21.15.42:0/3771278256 shutdown_connections 2019-12-05T05:40:59.246 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:59.241+0000 7fe4f4bb7700 1 -- 172.21.15.42:0/3771278256 shutdown_connections 2019-12-05T05:40:59.246 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:59.241+0000 7fe4f4bb7700 1 -- 172.21.15.42:0/3771278256 wait complete. 2019-12-05T05:40:59.246 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:40:59.241+0000 7fe4f4bb7700 1 -- 172.21.15.42:0/3771278256 >> 172.21.15.42:0/3771278256 conn(0x7fe4f00a8930 msgr2=0x7fe4f011c4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:40:59.257 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 154, "feature_flags": { "enable_multiple": false, "ever_enabled_multiple": false }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2019-12-05T05:03:35.604738+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 154, "ever_allowed_features": 2, "explicitly_allowed_features": 2, "failed": [], "flags": 18, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_7539": { "addr": "172.21.15.42:6835/1233824070", "addrs": { "addrvec": [ { "addr": "172.21.15.42:6834", "nonce": 1233824070, "type": "v2" }, { "addr": "172.21.15.42:6835", "nonce": 1233824070, "type": "v1" } ] }, "export_targets": [], "features": 4540138292836696063, "flags": 0, "gid": 7539, "incarnation": 154, "name": "a", "rank": 0, "state": "up:replay", "state_seq": 2 } }, "last_failure": 0, "last_failure_osd_epoch": 74, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 2, "min_compat_client": "0 (unknown)", "modified": "2019-12-05T05:40:58.897711+0000", "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7539 } } } ], "standbys": [] } 2019-12-05T05:41:01.257 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:41:01.335 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:01.325+0000 7f5c34c02700 1 Processor -- start 2019-12-05T05:41:01.336 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:01.325+0000 7f5c34c02700 1 -- start start 2019-12-05T05:41:01.336 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:01.329+0000 7f5c34c02700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5c300a7740 0x7f5c3008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:41:01.336 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:01.329+0000 7f5c34c02700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5c3007e720 0x7f5c3007eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:41:01.336 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:01.329+0000 7f5c34c02700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c300a0710 con 0x7f5c3009f2e0 2019-12-05T05:41:01.336 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:01.329+0000 7f5c34c02700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c300a0b00 con 0x7f5c300a7740 2019-12-05T05:41:01.336 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:01.329+0000 7f5c34c02700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c3008e5b0 con 0x7f5c3007e720 2019-12-05T05:41:01.337 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:01.329+0000 7f5c2ed9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5c300a7740 0x7f5c3008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:41:01.337 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:01.329+0000 7f5c2ed9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5c300a7740 0x7f5c3008df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:38712/0 (socket says 172.21.15.42:38712) 2019-12-05T05:41:01.337 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:01.329+0000 7f5c2ed9d700 1 -- 172.21.15.42:0/688956331 learned_addr learned my addr 172.21.15.42:0/688956331 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:41:01.337 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:01.329+0000 7f5c2e59c700 1 --2- 172.21.15.42:0/688956331 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5c3007e720 0x7f5c3007eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:41:01.337 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:01.329+0000 7f5c2ed9d700 1 -- 172.21.15.42:0/688956331 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5c3007e720 msgr2=0x7f5c3007eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:41:01.337 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:01.329+0000 7f5c2ed9d700 1 --2- 172.21.15.42:0/688956331 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5c3007e720 0x7f5c3007eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:41:01.337 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:01.329+0000 7f5c2ed9d700 1 -- 172.21.15.42:0/688956331 >> v1:172.21.15.42:6789/0 conn(0x7f5c3009f2e0 legacy=0x7f5c3009f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:41:01.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:01.329+0000 7f5c2dd9b700 1 -- 172.21.15.42:0/688956331 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1678233562 0 0) 0x7f5c300a0710 con 0x7f5c3009f2e0 2019-12-05T05:41:01.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:01.329+0000 7f5c2ed9d700 1 -- 172.21.15.42:0/688956331 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c3007f0a0 con 0x7f5c300a7740 2019-12-05T05:41:01.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:01.329+0000 7f5c2e59c700 1 --2- 172.21.15.42:0/688956331 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5c3007e720 0x7f5c3007eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T05:41:01.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:01.329+0000 7f5c2ed9d700 1 --2- 172.21.15.42:0/688956331 >> v2:172.21.15.42:3300/0 conn(0x7f5c300a7740 0x7f5c3008df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f5c20008790 tx=0x7f5c2000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:41:01.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:01.329+0000 7f5c2ed9d700 1 --2- 172.21.15.42:0/688956331 >> v2:172.21.15.42:3300/0 conn(0x7f5c300a7740 0x7f5c3008df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f5c20008790 tx=0x7f5c2000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:41:01.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:01.329+0000 7f5c2ed9d700 1 --2- 172.21.15.42:0/688956331 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5c300a7740 0x7f5c3008df80 secure :-1 s=READY pgs=1267 cs=0 l=1 rx=0x7f5c20008790 tx=0x7f5c2000fd30).ready entity=mon.1 client_cookie=319cb4e487ddedbd server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:41:01.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:01.329+0000 7f5c2ed9d700 1 --2- 172.21.15.42:0/688956331 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5c300a7740 0x7f5c3008df80 secure :-1 s=THROTTLE_DONE pgs=1267 cs=0 l=1 rx=0x7f5c20008790 tx=0x7f5c2000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:41:01.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:01.329+0000 7f5c2dd9b700 1 -- 172.21.15.42:0/688956331 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5c200113b0 con 0x7f5c300a7740 2019-12-05T05:41:01.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:01.329+0000 7f5c2ed9d700 1 --2- 172.21.15.42:0/688956331 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5c300a7740 0x7f5c3008df80 secure :-1 s=THROTTLE_DONE pgs=1267 cs=0 l=1 rx=0x7f5c20008790 tx=0x7f5c2000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:41:01.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:01.329+0000 7f5c2dd9b700 1 -- 172.21.15.42:0/688956331 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c20008bb0 con 0x7f5c300a7740 2019-12-05T05:41:01.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:01.329+0000 7f5c2ed9d700 1 --2- 172.21.15.42:0/688956331 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5c300a7740 0x7f5c3008df80 secure :-1 s=THROTTLE_DONE pgs=1267 cs=0 l=1 rx=0x7f5c20008790 tx=0x7f5c2000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:41:01.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:01.329+0000 7f5c2dd9b700 1 -- 172.21.15.42:0/688956331 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5c20011990 con 0x7f5c300a7740 2019-12-05T05:41:01.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:01.329+0000 7f5c34c02700 1 -- 172.21.15.42:0/688956331 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5c300a7740 msgr2=0x7f5c3008df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:41:01.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:01.329+0000 7f5c34c02700 1 --2- 172.21.15.42:0/688956331 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5c300a7740 0x7f5c3008df80 secure :-1 s=READY pgs=1267 cs=0 l=1 rx=0x7f5c20008790 tx=0x7f5c2000fd30).stop 2019-12-05T05:41:01.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:01.329+0000 7f5c34c02700 1 -- 172.21.15.42:0/688956331 shutdown_connections 2019-12-05T05:41:01.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:01.329+0000 7f5c34c02700 1 --2- 172.21.15.42:0/688956331 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5c300a7740 0x7f5c3008df80 unknown :-1 s=CLOSED pgs=1267 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:41:01.346 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:01.329+0000 7f5c34c02700 1 --2- 172.21.15.42:0/688956331 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5c3007e720 0x7f5c3007eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:41:01.347 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:01.329+0000 7f5c34c02700 1 -- 172.21.15.42:0/688956331 shutdown_connections 2019-12-05T05:41:01.347 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:01.329+0000 7f5c34c02700 1 -- 172.21.15.42:0/688956331 wait complete. 2019-12-05T05:41:01.347 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:01.329+0000 7f5c34c02700 1 -- 172.21.15.42:0/688956331 >> 172.21.15.42:0/688956331 conn(0x7f5c300aabe0 msgr2=0x7f5c300a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:41:01.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:01.333+0000 7f5c34c02700 1 Processor -- start 2019-12-05T05:41:01.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:01.333+0000 7f5c34c02700 1 -- start start 2019-12-05T05:41:01.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:01.333+0000 7f5c34c02700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5c301462c0 0x7f5c301466f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:41:01.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:01.333+0000 7f5c34c02700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5c301210e0 0x7f5c30121510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:41:01.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:01.333+0000 7f5c34c02700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c30121a60 con 0x7f5c3008ed10 2019-12-05T05:41:01.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:01.333+0000 7f5c34c02700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c3008f140 con 0x7f5c301462c0 2019-12-05T05:41:01.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:01.333+0000 7f5c34c02700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c3012b050 con 0x7f5c301210e0 2019-12-05T05:41:01.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:01.333+0000 7f5c2e59c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5c301462c0 0x7f5c301466f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:41:01.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:01.333+0000 7f5c2e59c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5c301462c0 0x7f5c301466f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:38722/0 (socket says 172.21.15.42:38722) 2019-12-05T05:41:01.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:01.333+0000 7f5c2e59c700 1 -- 172.21.15.42:0/3523363049 learned_addr learned my addr 172.21.15.42:0/3523363049 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:41:01.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:01.333+0000 7f5c2f59e700 1 --2- 172.21.15.42:0/3523363049 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5c301210e0 0x7f5c30121510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:41:01.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:01.333+0000 7f5c2e59c700 1 -- 172.21.15.42:0/3523363049 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5c301210e0 msgr2=0x7f5c30121510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:41:01.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:01.333+0000 7f5c2e59c700 1 --2- 172.21.15.42:0/3523363049 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5c301210e0 0x7f5c30121510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:41:01.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:01.333+0000 7f5c2e59c700 1 -- 172.21.15.42:0/3523363049 >> v1:172.21.15.40:6789/0 conn(0x7f5c3008ed10 legacy=0x7f5c3012b590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:41:01.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:01.333+0000 7f5c2e59c700 1 -- 172.21.15.42:0/3523363049 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c30120b30 con 0x7f5c301462c0 2019-12-05T05:41:01.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:01.333+0000 7f5c2e59c700 1 --2- 172.21.15.42:0/3523363049 >> v2:172.21.15.42:3300/0 conn(0x7f5c301462c0 0x7f5c301466f0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f5c240077a0 tx=0x7f5c24010a80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:41:01.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:01.333+0000 7f5c2e59c700 1 --2- 172.21.15.42:0/3523363049 >> v2:172.21.15.42:3300/0 conn(0x7f5c301462c0 0x7f5c301466f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f5c240077a0 tx=0x7f5c24010a80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:41:01.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:01.333+0000 7f5c2e59c700 1 --2- 172.21.15.42:0/3523363049 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5c301462c0 0x7f5c301466f0 secure :-1 s=READY pgs=1268 cs=0 l=1 rx=0x7f5c240077a0 tx=0x7f5c24010a80).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:41:01.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:01.333+0000 7f5c2e59c700 1 --2- 172.21.15.42:0/3523363049 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5c301462c0 0x7f5c301466f0 secure :-1 s=THROTTLE_DONE pgs=1268 cs=0 l=1 rx=0x7f5c240077a0 tx=0x7f5c24010a80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:41:01.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:01.333+0000 7f5c2e59c700 1 --2- 172.21.15.42:0/3523363049 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5c301462c0 0x7f5c301466f0 secure :-1 s=THROTTLE_DONE pgs=1268 cs=0 l=1 rx=0x7f5c240077a0 tx=0x7f5c24010a80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:41:01.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:01.333+0000 7f5c1f7fe700 1 -- 172.21.15.42:0/3523363049 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5c2400a4b0 con 0x7f5c301462c0 2019-12-05T05:41:01.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:01.333+0000 7f5c2e59c700 1 --2- 172.21.15.42:0/3523363049 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5c301462c0 0x7f5c301466f0 secure :-1 s=THROTTLE_DONE pgs=1268 cs=0 l=1 rx=0x7f5c240077a0 tx=0x7f5c24010a80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:41:01.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:01.333+0000 7f5c1f7fe700 1 -- 172.21.15.42:0/3523363049 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c24007be0 con 0x7f5c301462c0 2019-12-05T05:41:01.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:01.333+0000 7f5c34c02700 1 -- 172.21.15.42:0/3523363049 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c30135830 con 0x7f5c301462c0 2019-12-05T05:41:01.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:01.333+0000 7f5c1f7fe700 1 -- 172.21.15.42:0/3523363049 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5c2400aad0 con 0x7f5c301462c0 2019-12-05T05:41:01.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:01.333+0000 7f5c34c02700 1 -- 172.21.15.42:0/3523363049 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c30135d60 con 0x7f5c301462c0 2019-12-05T05:41:01.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:01.333+0000 7f5c2e59c700 1 --2- 172.21.15.42:0/3523363049 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5c301462c0 0x7f5c301466f0 secure :-1 s=THROTTLE_DONE pgs=1268 cs=0 l=1 rx=0x7f5c240077a0 tx=0x7f5c24010a80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:41:01.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:01.337+0000 7f5c1f7fe700 1 -- 172.21.15.42:0/3523363049 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f5c24007170 con 0x7f5c301462c0 2019-12-05T05:41:01.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:01.337+0000 7f5c2e59c700 1 --2- 172.21.15.42:0/3523363049 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5c301462c0 0x7f5c301466f0 secure :-1 s=THROTTLE_DONE pgs=1268 cs=0 l=1 rx=0x7f5c240077a0 tx=0x7f5c24010a80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:41:01.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:01.337+0000 7f5c1f7fe700 1 --2- 172.21.15.42:0/3523363049 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5c1403acc0 0x7f5c1403d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:41:01.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:01.337+0000 7f5c1f7fe700 1 -- 172.21.15.42:0/3523363049 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 8181+0+0 (secure 0 0 0) 0x7f5c2404b0c0 con 0x7f5c301462c0 2019-12-05T05:41:01.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:01.337+0000 7f5c2ed9d700 1 --2- 172.21.15.42:0/3523363049 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5c1403acc0 0x7f5c1403d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:41:01.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:01.337+0000 7f5c2ed9d700 1 --2- 172.21.15.42:0/3523363049 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5c1403acc0 0x7f5c1403d1d0 crc :-1 s=READY pgs=548 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:41:01.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:01.341+0000 7f5c34c02700 1 -- 172.21.15.42:0/3523363049 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c3012bd90 con 0x7f5c301462c0 2019-12-05T05:41:01.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:01.349+0000 7f5c2e59c700 1 --2- 172.21.15.42:0/3523363049 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5c301462c0 0x7f5c301466f0 secure :-1 s=THROTTLE_DONE pgs=1268 cs=0 l=1 rx=0x7f5c240077a0 tx=0x7f5c24010a80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:41:01.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:01.349+0000 7f5c1f7fe700 1 -- 172.21.15.42:0/3523363049 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f5c24032140 con 0x7f5c301462c0 2019-12-05T05:41:01.523 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:01.513+0000 7f5c34c02700 1 -- 172.21.15.42:0/3523363049 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5c3005d110 con 0x7f5c301462c0 2019-12-05T05:41:01.523 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:01.513+0000 7f5c2e59c700 1 --2- 172.21.15.42:0/3523363049 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5c301462c0 0x7f5c301466f0 secure :-1 s=THROTTLE_DONE pgs=1268 cs=0 l=1 rx=0x7f5c240077a0 tx=0x7f5c24010a80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:41:01.523 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:01.513+0000 7f5c1f7fe700 1 -- 172.21.15.42:0/3523363049 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 155 v155) v1 ==== 95+0+1778 (secure 0 0 0) 0x7f5c240323a0 con 0x7f5c301462c0 2019-12-05T05:41:01.523 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 155 2019-12-05T05:41:01.524 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:01.513+0000 7f5c34c02700 1 -- 172.21.15.42:0/3523363049 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5c1403acc0 msgr2=0x7f5c1403d1d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:41:01.524 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:01.513+0000 7f5c34c02700 1 --2- 172.21.15.42:0/3523363049 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5c1403acc0 0x7f5c1403d1d0 crc :-1 s=READY pgs=548 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:41:01.524 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:01.513+0000 7f5c34c02700 1 -- 172.21.15.42:0/3523363049 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5c301462c0 msgr2=0x7f5c301466f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:41:01.524 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:01.513+0000 7f5c34c02700 1 --2- 172.21.15.42:0/3523363049 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5c301462c0 0x7f5c301466f0 secure :-1 s=READY pgs=1268 cs=0 l=1 rx=0x7f5c240077a0 tx=0x7f5c24010a80).stop 2019-12-05T05:41:01.524 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:01.513+0000 7f5c34c02700 1 -- 172.21.15.42:0/3523363049 shutdown_connections 2019-12-05T05:41:01.525 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:01.513+0000 7f5c34c02700 1 --2- 172.21.15.42:0/3523363049 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5c1403acc0 0x7f5c1403d1d0 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:41:01.525 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:01.513+0000 7f5c34c02700 1 --2- 172.21.15.42:0/3523363049 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5c301210e0 0x7f5c30121510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:41:01.525 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:01.513+0000 7f5c34c02700 1 --2- 172.21.15.42:0/3523363049 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5c301462c0 0x7f5c301466f0 unknown :-1 s=CLOSED pgs=1268 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:41:01.525 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:01.517+0000 7f5c34c02700 1 -- 172.21.15.42:0/3523363049 shutdown_connections 2019-12-05T05:41:01.525 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:01.517+0000 7f5c34c02700 1 -- 172.21.15.42:0/3523363049 wait complete. 2019-12-05T05:41:01.525 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:01.517+0000 7f5c34c02700 1 -- 172.21.15.42:0/3523363049 >> 172.21.15.42:0/3523363049 conn(0x7f5c300aabe0 msgr2=0x7f5c3009f1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:41:01.526 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:41:01.526 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":155,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":155,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:41:01.419567+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7539},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7539":{"gid":7539,"name":"a","rank":0,"incarnation":154,"state":"up:reconnect","state_seq":28,"addr":"172.21.15.42:6835/1233824070","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1233824070},{"type":"v1","addr":"172.21.15.42:6835","nonce":1233824070}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:41:01.528 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2019-12-05T05:41:03.530 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:41:03.636 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.629+0000 7f9561d68700 1 Processor -- start 2019-12-05T05:41:03.637 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.629+0000 7f9561d68700 1 -- start start 2019-12-05T05:41:03.637 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.629+0000 7f9561d68700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f955c0a7730 0x7f955c08df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:41:03.637 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.629+0000 7f9561d68700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f955c09fad0 0x7f955c07eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:41:03.637 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.629+0000 7f9561d68700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f955c0a0700 con 0x7f955c07e710 2019-12-05T05:41:03.637 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.629+0000 7f9561d68700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f955c0a0af0 con 0x7f955c09fad0 2019-12-05T05:41:03.637 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.629+0000 7f9561d68700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f955c08e5a0 con 0x7f955c0a7730 2019-12-05T05:41:03.637 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.629+0000 7f955bfff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f955c0a7730 0x7f955c08df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:41:03.638 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.629+0000 7f955bfff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f955c0a7730 0x7f955c08df70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:38734/0 (socket says 172.21.15.42:38734) 2019-12-05T05:41:03.638 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.629+0000 7f955bfff700 1 -- 172.21.15.42:0/866162522 learned_addr learned my addr 172.21.15.42:0/866162522 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:41:03.638 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.629+0000 7f9560b06700 1 --2- 172.21.15.42:0/866162522 >> v2:172.21.15.40:3300/0 conn(0x7f955c09fad0 0x7f955c07eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:41:03.638 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.629+0000 7f955bfff700 1 -- 172.21.15.42:0/866162522 >> v2:172.21.15.40:3300/0 conn(0x7f955c09fad0 msgr2=0x7f955c07eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:41:03.638 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.629+0000 7f955bfff700 1 --2- 172.21.15.42:0/866162522 >> v2:172.21.15.40:3300/0 conn(0x7f955c09fad0 0x7f955c07eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:41:03.638 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.629+0000 7f955bfff700 1 -- 172.21.15.42:0/866162522 >> v1:172.21.15.40:6789/0 conn(0x7f955c07e710 legacy=0x7f955c09f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:41:03.639 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.629+0000 7f955bfff700 1 -- 172.21.15.42:0/866162522 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f955c07f090 con 0x7f955c0a7730 2019-12-05T05:41:03.639 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.629+0000 7f955bfff700 1 --2- 172.21.15.42:0/866162522 >> v2:172.21.15.42:3300/0 conn(0x7f955c0a7730 0x7f955c08df70 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f9544002f30 tx=0x7f954400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:41:03.639 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.629+0000 7f955bfff700 1 --2- 172.21.15.42:0/866162522 >> v2:172.21.15.42:3300/0 conn(0x7f955c0a7730 0x7f955c08df70 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f9544002f30 tx=0x7f954400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:41:03.639 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.629+0000 7f955bfff700 1 --2- 172.21.15.42:0/866162522 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f955c0a7730 0x7f955c08df70 secure :-1 s=READY pgs=1269 cs=0 l=1 rx=0x7f9544002f30 tx=0x7f954400fd30).ready entity=mon.1 client_cookie=a5ddfb6d89d66d7b server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:41:03.639 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.629+0000 7f955bfff700 1 --2- 172.21.15.42:0/866162522 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f955c0a7730 0x7f955c08df70 secure :-1 s=THROTTLE_DONE pgs=1269 cs=0 l=1 rx=0x7f9544002f30 tx=0x7f954400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:41:03.639 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.629+0000 7f955b7fe700 1 -- 172.21.15.42:0/866162522 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9544008c90 con 0x7f955c0a7730 2019-12-05T05:41:03.639 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.629+0000 7f955bfff700 1 --2- 172.21.15.42:0/866162522 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f955c0a7730 0x7f955c08df70 secure :-1 s=THROTTLE_DONE pgs=1269 cs=0 l=1 rx=0x7f9544002f30 tx=0x7f954400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:41:03.640 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.629+0000 7f955bfff700 1 --2- 172.21.15.42:0/866162522 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f955c0a7730 0x7f955c08df70 secure :-1 s=THROTTLE_DONE pgs=1269 cs=0 l=1 rx=0x7f9544002f30 tx=0x7f954400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:41:03.640 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.629+0000 7f955b7fe700 1 -- 172.21.15.42:0/866162522 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95440088b0 con 0x7f955c0a7730 2019-12-05T05:41:03.641 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.629+0000 7f955b7fe700 1 -- 172.21.15.42:0/866162522 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9544011ad0 con 0x7f955c0a7730 2019-12-05T05:41:03.641 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.629+0000 7f9561d68700 1 -- 172.21.15.42:0/866162522 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f955c0a7730 msgr2=0x7f955c08df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:41:03.641 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.629+0000 7f9561d68700 1 --2- 172.21.15.42:0/866162522 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f955c0a7730 0x7f955c08df70 secure :-1 s=READY pgs=1269 cs=0 l=1 rx=0x7f9544002f30 tx=0x7f954400fd30).stop 2019-12-05T05:41:03.641 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.629+0000 7f9561d68700 1 -- 172.21.15.42:0/866162522 shutdown_connections 2019-12-05T05:41:03.641 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.629+0000 7f9561d68700 1 --2- 172.21.15.42:0/866162522 >> v2:172.21.15.40:3300/0 conn(0x7f955c09fad0 0x7f955c07eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:41:03.641 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.629+0000 7f9561d68700 1 --2- 172.21.15.42:0/866162522 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f955c0a7730 0x7f955c08df70 unknown :-1 s=CLOSED pgs=1269 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:41:03.642 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.633+0000 7f9561d68700 1 -- 172.21.15.42:0/866162522 shutdown_connections 2019-12-05T05:41:03.644 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.633+0000 7f9561d68700 1 -- 172.21.15.42:0/866162522 wait complete. 2019-12-05T05:41:03.644 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.633+0000 7f9561d68700 1 -- 172.21.15.42:0/866162522 >> 172.21.15.42:0/866162522 conn(0x7f955c0aabd0 msgr2=0x7f955c0a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:41:03.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.633+0000 7f9561d68700 1 Processor -- start 2019-12-05T05:41:03.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.633+0000 7f9561d68700 1 -- start start 2019-12-05T05:41:03.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.633+0000 7f9561d68700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f955c1164f0 0x7f955c118120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:41:03.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.633+0000 7f9561d68700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f955c134460 con 0x7f955c134030 2019-12-05T05:41:03.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.633+0000 7f9561d68700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f955c145360 con 0x7f955c133c00 2019-12-05T05:41:03.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.633+0000 7f9561d68700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f955c118ad0 con 0x7f955c1164f0 2019-12-05T05:41:03.650 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.633+0000 7f95537fe700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f955c133c00 0x7f955c08eb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:38782/0 (socket says 172.21.15.42:38782) 2019-12-05T05:41:03.650 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.633+0000 7f95537fe700 1 -- 172.21.15.42:0/3717392866 learned_addr learned my addr 172.21.15.42:0/3717392866 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:41:03.650 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.633+0000 7f955bfff700 1 --2- 172.21.15.42:0/3717392866 >> v2:172.21.15.40:3300/0 conn(0x7f955c1164f0 0x7f955c118120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:41:03.650 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.633+0000 7f95597fa700 1 -- 172.21.15.42:0/3717392866 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3909136031 0 0) 0x7f955c145360 con 0x7f955c133c00 2019-12-05T05:41:03.650 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.633+0000 7f95597fa700 1 -- 172.21.15.42:0/3717392866 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95400030b0 con 0x7f955c133c00 2019-12-05T05:41:03.650 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.637+0000 7f955bfff700 1 -- 172.21.15.42:0/3717392866 >> v1:172.21.15.42:6789/0 conn(0x7f955c133c00 legacy=0x7f955c08eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:41:03.650 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.637+0000 7f955bfff700 1 -- 172.21.15.42:0/3717392866 >> v1:172.21.15.40:6789/0 conn(0x7f955c134030 legacy=0x7f955c08f340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:41:03.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.637+0000 7f955bfff700 1 -- 172.21.15.42:0/3717392866 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f955c118670 con 0x7f955c1164f0 2019-12-05T05:41:03.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.637+0000 7f955bfff700 1 --2- 172.21.15.42:0/3717392866 >> v2:172.21.15.40:3300/0 conn(0x7f955c1164f0 0x7f955c118120 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f955c11a240 tx=0x7f9544008ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:41:03.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.637+0000 7f955bfff700 1 --2- 172.21.15.42:0/3717392866 >> v2:172.21.15.40:3300/0 conn(0x7f955c1164f0 0x7f955c118120 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f955c11a240 tx=0x7f9544008ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:41:03.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.637+0000 7f955bfff700 1 --2- 172.21.15.42:0/3717392866 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f955c1164f0 0x7f955c118120 secure :-1 s=READY pgs=988 cs=0 l=1 rx=0x7f955c11a240 tx=0x7f9544008ae0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:41:03.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.637+0000 7f955bfff700 1 --2- 172.21.15.42:0/3717392866 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f955c1164f0 0x7f955c118120 secure :-1 s=THROTTLE_DONE pgs=988 cs=0 l=1 rx=0x7f955c11a240 tx=0x7f9544008ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:41:03.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.637+0000 7f95597fa700 1 -- 172.21.15.42:0/3717392866 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f954400f470 con 0x7f955c1164f0 2019-12-05T05:41:03.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.637+0000 7f955bfff700 1 --2- 172.21.15.42:0/3717392866 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f955c1164f0 0x7f955c118120 secure :-1 s=THROTTLE_DONE pgs=988 cs=0 l=1 rx=0x7f955c11a240 tx=0x7f9544008ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:41:03.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.637+0000 7f9561d68700 1 -- 172.21.15.42:0/3717392866 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f955c0772f0 con 0x7f955c1164f0 2019-12-05T05:41:03.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.637+0000 7f955bfff700 1 --2- 172.21.15.42:0/3717392866 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f955c1164f0 0x7f955c118120 secure :-1 s=THROTTLE_DONE pgs=988 cs=0 l=1 rx=0x7f955c11a240 tx=0x7f9544008ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:41:03.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.637+0000 7f95597fa700 1 -- 172.21.15.42:0/3717392866 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95440116f0 con 0x7f955c1164f0 2019-12-05T05:41:03.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.637+0000 7f95597fa700 1 -- 172.21.15.42:0/3717392866 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9544005730 con 0x7f955c1164f0 2019-12-05T05:41:03.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.637+0000 7f9561d68700 1 -- 172.21.15.42:0/3717392866 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f955c0778a0 con 0x7f955c1164f0 2019-12-05T05:41:03.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.637+0000 7f955bfff700 1 --2- 172.21.15.42:0/3717392866 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f955c1164f0 0x7f955c118120 secure :-1 s=THROTTLE_DONE pgs=988 cs=0 l=1 rx=0x7f955c11a240 tx=0x7f9544008ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:41:03.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.637+0000 7f95597fa700 1 -- 172.21.15.42:0/3717392866 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f9544005a40 con 0x7f955c1164f0 2019-12-05T05:41:03.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.637+0000 7f955bfff700 1 --2- 172.21.15.42:0/3717392866 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f955c1164f0 0x7f955c118120 secure :-1 s=THROTTLE_DONE pgs=988 cs=0 l=1 rx=0x7f955c11a240 tx=0x7f9544008ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:41:03.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.637+0000 7f95597fa700 1 --2- 172.21.15.42:0/3717392866 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f954003b1f0 0x7f954003d700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:41:03.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.637+0000 7f95597fa700 1 -- 172.21.15.42:0/3717392866 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 8181+0+0 (secure 0 0 0) 0x7f954404b950 con 0x7f955c1164f0 2019-12-05T05:41:03.654 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.637+0000 7f95537fe700 1 --2- 172.21.15.42:0/3717392866 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f954003b1f0 0x7f954003d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:41:03.654 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.637+0000 7f95537fe700 1 --2- 172.21.15.42:0/3717392866 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f954003b1f0 0x7f954003d700 crc :-1 s=READY pgs=549 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:41:03.654 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.645+0000 7f9561d68700 1 -- 172.21.15.42:0/3717392866 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f955c05d100 con 0x7f955c1164f0 2019-12-05T05:41:03.658 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.653+0000 7f955bfff700 1 --2- 172.21.15.42:0/3717392866 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f955c1164f0 0x7f955c118120 secure :-1 s=THROTTLE_DONE pgs=988 cs=0 l=1 rx=0x7f955c11a240 tx=0x7f9544008ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:41:03.658 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.653+0000 7f95597fa700 1 -- 172.21.15.42:0/3717392866 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f9544005d50 con 0x7f955c1164f0 2019-12-05T05:41:03.843 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.837+0000 7f9561d68700 1 -- 172.21.15.42:0/3717392866 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f955c05b8b0 con 0x7f955c1164f0 2019-12-05T05:41:03.844 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.841+0000 7f955bfff700 1 --2- 172.21.15.42:0/3717392866 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f955c1164f0 0x7f955c118120 secure :-1 s=THROTTLE_DONE pgs=988 cs=0 l=1 rx=0x7f955c11a240 tx=0x7f9544008ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:41:03.844 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.841+0000 7f95597fa700 1 -- 172.21.15.42:0/3717392866 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 157 v157) v1 ==== 95+0+1775 (secure 0 0 0) 0x7f95440337e0 con 0x7f955c1164f0 2019-12-05T05:41:03.846 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 157 2019-12-05T05:41:03.846 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:41:03.846 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":157,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":157,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:41:03.431537+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7539},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7539":{"gid":7539,"name":"a","rank":0,"incarnation":154,"state":"up:active","state_seq":30,"addr":"172.21.15.42:6835/1233824070","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1233824070},{"type":"v1","addr":"172.21.15.42:6835","nonce":1233824070}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:41:03.849 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.845+0000 7f9561d68700 1 -- 172.21.15.42:0/3717392866 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f954003b1f0 msgr2=0x7f954003d700 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:41:03.849 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.845+0000 7f9561d68700 1 --2- 172.21.15.42:0/3717392866 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f954003b1f0 0x7f954003d700 crc :-1 s=READY pgs=549 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:41:03.849 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.845+0000 7f9561d68700 1 -- 172.21.15.42:0/3717392866 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f955c1164f0 msgr2=0x7f955c118120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:41:03.849 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.845+0000 7f9561d68700 1 --2- 172.21.15.42:0/3717392866 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f955c1164f0 0x7f955c118120 secure :-1 s=READY pgs=988 cs=0 l=1 rx=0x7f955c11a240 tx=0x7f9544008ae0).stop 2019-12-05T05:41:03.849 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.845+0000 7f9561d68700 1 -- 172.21.15.42:0/3717392866 shutdown_connections 2019-12-05T05:41:03.849 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.845+0000 7f9561d68700 1 --2- 172.21.15.42:0/3717392866 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f954003b1f0 0x7f954003d700 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:41:03.850 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.845+0000 7f9561d68700 1 --2- 172.21.15.42:0/3717392866 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f955c1164f0 0x7f955c118120 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:41:03.850 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.845+0000 7f9561d68700 1 -- 172.21.15.42:0/3717392866 shutdown_connections 2019-12-05T05:41:03.850 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.845+0000 7f9561d68700 1 -- 172.21.15.42:0/3717392866 wait complete. 2019-12-05T05:41:03.850 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.845+0000 7f9561d68700 1 -- 172.21.15.42:0/3717392866 >> 172.21.15.42:0/3717392866 conn(0x7f955c0a7730 msgr2=0x7f955c09f2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:41:03.864 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a has gained rank=0, replacing gid=7416 2019-12-05T05:41:03.864 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 0 secs before reviving mds.b 2019-12-05T05:41:03.865 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.b 2019-12-05T05:41:03.865 INFO:tasks.ceph.mds.b:Restarting daemon 2019-12-05T05:41:03.865 INFO:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2019-12-05T05:41:03.869 INFO:tasks.ceph.mds.b:Started 2019-12-05T05:41:03.869 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:41:03.929 INFO:tasks.ceph.mds.b.smithi040.stdout:starting mds.b at 2019-12-05T05:41:03.966 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.957+0000 7f3d9394a700 1 Processor -- start 2019-12-05T05:41:03.966 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.957+0000 7f3d9394a700 1 -- start start 2019-12-05T05:41:03.967 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.957+0000 7f3d9394a700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3d8c0a7740 0x7f3d8c08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:41:03.967 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.957+0000 7f3d9394a700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f3d8c09fae0 0x7f3d8c07eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:41:03.967 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.957+0000 7f3d9394a700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d8c0a0710 con 0x7f3d8c07e720 2019-12-05T05:41:03.967 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.957+0000 7f3d9394a700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d8c0a0b00 con 0x7f3d8c09fae0 2019-12-05T05:41:03.967 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.957+0000 7f3d9394a700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d8c08e5b0 con 0x7f3d8c0a7740 2019-12-05T05:41:03.967 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.957+0000 7f3d916e6700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f3d8c07e720 0x7f3d8c09f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:38794/0 (socket says 172.21.15.42:38794) 2019-12-05T05:41:03.967 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.957+0000 7f3d916e6700 1 -- 172.21.15.42:0/2186941683 learned_addr learned my addr 172.21.15.42:0/2186941683 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:41:03.968 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.957+0000 7f3d91ee7700 1 --2- 172.21.15.42:0/2186941683 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3d8c0a7740 0x7f3d8c08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:41:03.968 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.957+0000 7f3d926e8700 1 --2- 172.21.15.42:0/2186941683 >> v2:172.21.15.40:3300/0 conn(0x7f3d8c09fae0 0x7f3d8c07eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:41:03.968 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.957+0000 7f3d90ee5700 1 -- 172.21.15.42:0/2186941683 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1629420581 0 0) 0x7f3d8c0a0710 con 0x7f3d8c07e720 2019-12-05T05:41:03.968 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.957+0000 7f3d90ee5700 1 -- 172.21.15.42:0/2186941683 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d780030b0 con 0x7f3d8c07e720 2019-12-05T05:41:03.968 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.957+0000 7f3d91ee7700 1 -- 172.21.15.42:0/2186941683 >> v2:172.21.15.40:3300/0 conn(0x7f3d8c09fae0 msgr2=0x7f3d8c07eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:41:03.969 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.957+0000 7f3d91ee7700 1 --2- 172.21.15.42:0/2186941683 >> v2:172.21.15.40:3300/0 conn(0x7f3d8c09fae0 0x7f3d8c07eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:41:03.969 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.957+0000 7f3d91ee7700 1 -- 172.21.15.42:0/2186941683 >> v1:172.21.15.42:6789/0 conn(0x7f3d8c07e720 legacy=0x7f3d8c09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:41:03.969 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.957+0000 7f3d91ee7700 1 -- 172.21.15.42:0/2186941683 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d8c07f0a0 con 0x7f3d8c0a7740 2019-12-05T05:41:03.969 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.957+0000 7f3d91ee7700 1 --2- 172.21.15.42:0/2186941683 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3d8c0a7740 0x7f3d8c08df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f3d88008790 tx=0x7f3d8800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:41:03.969 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.961+0000 7f3d91ee7700 1 --2- 172.21.15.42:0/2186941683 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3d8c0a7740 0x7f3d8c08df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f3d88008790 tx=0x7f3d8800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:41:03.969 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.961+0000 7f3d91ee7700 1 --2- 172.21.15.42:0/2186941683 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3d8c0a7740 0x7f3d8c08df80 secure :-1 s=READY pgs=263 cs=0 l=1 rx=0x7f3d88008790 tx=0x7f3d8800fc70).ready entity=mon.2 client_cookie=7d1b0ada00607996 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:41:03.969 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.961+0000 7f3d91ee7700 1 --2- 172.21.15.42:0/2186941683 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3d8c0a7740 0x7f3d8c08df80 secure :-1 s=THROTTLE_DONE pgs=263 cs=0 l=1 rx=0x7f3d88008790 tx=0x7f3d8800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:41:03.972 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.961+0000 7f3d90ee5700 1 -- 172.21.15.42:0/2186941683 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3d88008bb0 con 0x7f3d8c0a7740 2019-12-05T05:41:03.972 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.961+0000 7f3d91ee7700 1 --2- 172.21.15.42:0/2186941683 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3d8c0a7740 0x7f3d8c08df80 secure :-1 s=THROTTLE_DONE pgs=263 cs=0 l=1 rx=0x7f3d88008790 tx=0x7f3d8800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:41:03.973 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.961+0000 7f3d90ee5700 1 -- 172.21.15.42:0/2186941683 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d880087d0 con 0x7f3d8c0a7740 2019-12-05T05:41:03.973 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.961+0000 7f3d91ee7700 1 --2- 172.21.15.42:0/2186941683 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3d8c0a7740 0x7f3d8c08df80 secure :-1 s=THROTTLE_DONE pgs=263 cs=0 l=1 rx=0x7f3d88008790 tx=0x7f3d8800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:41:03.973 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.961+0000 7f3d90ee5700 1 -- 172.21.15.42:0/2186941683 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3d88011860 con 0x7f3d8c0a7740 2019-12-05T05:41:03.973 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.961+0000 7f3d9394a700 1 -- 172.21.15.42:0/2186941683 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3d8c0a7740 msgr2=0x7f3d8c08df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:41:03.973 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.961+0000 7f3d9394a700 1 --2- 172.21.15.42:0/2186941683 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3d8c0a7740 0x7f3d8c08df80 secure :-1 s=READY pgs=263 cs=0 l=1 rx=0x7f3d88008790 tx=0x7f3d8800fc70).stop 2019-12-05T05:41:03.973 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.961+0000 7f3d9394a700 1 -- 172.21.15.42:0/2186941683 shutdown_connections 2019-12-05T05:41:03.973 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.961+0000 7f3d9394a700 1 --2- 172.21.15.42:0/2186941683 >> v2:172.21.15.40:3300/0 conn(0x7f3d8c09fae0 0x7f3d8c07eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:41:03.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.961+0000 7f3d9394a700 1 --2- 172.21.15.42:0/2186941683 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3d8c0a7740 0x7f3d8c08df80 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:41:03.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.961+0000 7f3d9394a700 1 -- 172.21.15.42:0/2186941683 shutdown_connections 2019-12-05T05:41:03.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.961+0000 7f3d9394a700 1 -- 172.21.15.42:0/2186941683 wait complete. 2019-12-05T05:41:03.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.961+0000 7f3d9394a700 1 -- 172.21.15.42:0/2186941683 >> 172.21.15.42:0/2186941683 conn(0x7f3d8c0aabe0 msgr2=0x7f3d8c0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:41:03.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.961+0000 7f3d9394a700 1 Processor -- start 2019-12-05T05:41:03.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.961+0000 7f3d9394a700 1 -- start start 2019-12-05T05:41:03.978 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.961+0000 7f3d9394a700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3d8c1215b0 0x7f3d8c124030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:41:03.979 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.961+0000 7f3d9394a700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f3d8c11b250 0x7f3d8c11a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:41:03.979 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.961+0000 7f3d9394a700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d8c1233f0 con 0x7f3d8c11ab30 2019-12-05T05:41:03.979 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.961+0000 7f3d9394a700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d8c123620 con 0x7f3d8c11b250 2019-12-05T05:41:03.979 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.961+0000 7f3d9394a700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d8c11af60 con 0x7f3d8c1215b0 2019-12-05T05:41:03.979 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.961+0000 7f3d926e8700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f3d8c11ab30 0x7f3d8c08eb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:38800/0 (socket says 172.21.15.42:38800) 2019-12-05T05:41:03.979 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.961+0000 7f3d926e8700 1 -- 172.21.15.42:0/1537009819 learned_addr learned my addr 172.21.15.42:0/1537009819 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:41:03.980 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.961+0000 7f3d91ee7700 1 --2- 172.21.15.42:0/1537009819 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3d8c1215b0 0x7f3d8c124030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:41:03.980 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.965+0000 7f3d916e6700 1 --2- 172.21.15.42:0/1537009819 >> v2:172.21.15.40:3300/0 conn(0x7f3d8c11b250 0x7f3d8c11a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:41:03.980 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.965+0000 7f3d827fc700 1 -- 172.21.15.42:0/1537009819 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3977648475 0 0) 0x7f3d8c1233f0 con 0x7f3d8c11ab30 2019-12-05T05:41:03.980 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.965+0000 7f3d827fc700 1 -- 172.21.15.42:0/1537009819 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d740030b0 con 0x7f3d8c11ab30 2019-12-05T05:41:03.980 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.965+0000 7f3d91ee7700 1 -- 172.21.15.42:0/1537009819 >> v2:172.21.15.40:3300/0 conn(0x7f3d8c11b250 msgr2=0x7f3d8c11a5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:41:03.980 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.965+0000 7f3d91ee7700 1 --2- 172.21.15.42:0/1537009819 >> v2:172.21.15.40:3300/0 conn(0x7f3d8c11b250 0x7f3d8c11a5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:41:03.980 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.965+0000 7f3d91ee7700 1 -- 172.21.15.42:0/1537009819 >> v1:172.21.15.42:6789/0 conn(0x7f3d8c11ab30 legacy=0x7f3d8c08eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:41:03.981 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.965+0000 7f3d91ee7700 1 -- 172.21.15.42:0/1537009819 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d8c08f370 con 0x7f3d8c1215b0 2019-12-05T05:41:03.981 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.965+0000 7f3d916e6700 1 --2- 172.21.15.42:0/1537009819 >> v2:172.21.15.40:3300/0 conn(0x7f3d8c11b250 0x7f3d8c11a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:41:03.981 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.965+0000 7f3d91ee7700 1 --2- 172.21.15.42:0/1537009819 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3d8c1215b0 0x7f3d8c124030 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f3d88008790 tx=0x7f3d880118a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:41:03.981 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.965+0000 7f3d91ee7700 1 --2- 172.21.15.42:0/1537009819 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3d8c1215b0 0x7f3d8c124030 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f3d88008790 tx=0x7f3d880118a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:41:03.981 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.965+0000 7f3d91ee7700 1 --2- 172.21.15.42:0/1537009819 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3d8c1215b0 0x7f3d8c124030 secure :-1 s=READY pgs=264 cs=0 l=1 rx=0x7f3d88008790 tx=0x7f3d880118a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:41:03.981 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.965+0000 7f3d91ee7700 1 --2- 172.21.15.42:0/1537009819 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3d8c1215b0 0x7f3d8c124030 secure :-1 s=THROTTLE_DONE pgs=264 cs=0 l=1 rx=0x7f3d88008790 tx=0x7f3d880118a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:41:03.981 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.965+0000 7f3d827fc700 1 -- 172.21.15.42:0/1537009819 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3d8800f280 con 0x7f3d8c1215b0 2019-12-05T05:41:03.982 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.965+0000 7f3d91ee7700 1 --2- 172.21.15.42:0/1537009819 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3d8c1215b0 0x7f3d8c124030 secure :-1 s=THROTTLE_DONE pgs=264 cs=0 l=1 rx=0x7f3d88008790 tx=0x7f3d880118a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:41:03.982 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.965+0000 7f3d9394a700 1 -- 172.21.15.42:0/1537009819 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d8c08f5e0 con 0x7f3d8c1215b0 2019-12-05T05:41:03.982 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.965+0000 7f3d827fc700 1 -- 172.21.15.42:0/1537009819 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d88008db0 con 0x7f3d8c1215b0 2019-12-05T05:41:03.982 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.965+0000 7f3d91ee7700 1 --2- 172.21.15.42:0/1537009819 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3d8c1215b0 0x7f3d8c124030 secure :-1 s=THROTTLE_DONE pgs=264 cs=0 l=1 rx=0x7f3d88008790 tx=0x7f3d880118a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:41:03.982 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.965+0000 7f3d827fc700 1 -- 172.21.15.42:0/1537009819 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3d88005540 con 0x7f3d8c1215b0 2019-12-05T05:41:03.982 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.965+0000 7f3d9394a700 1 -- 172.21.15.42:0/1537009819 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d8c077300 con 0x7f3d8c1215b0 2019-12-05T05:41:03.982 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.965+0000 7f3d91ee7700 1 --2- 172.21.15.42:0/1537009819 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3d8c1215b0 0x7f3d8c124030 secure :-1 s=THROTTLE_DONE pgs=264 cs=0 l=1 rx=0x7f3d88008790 tx=0x7f3d880118a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:41:03.982 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.965+0000 7f3d827fc700 1 -- 172.21.15.42:0/1537009819 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f3d880057e0 con 0x7f3d8c1215b0 2019-12-05T05:41:03.983 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.965+0000 7f3d91ee7700 1 --2- 172.21.15.42:0/1537009819 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3d8c1215b0 0x7f3d8c124030 secure :-1 s=THROTTLE_DONE pgs=264 cs=0 l=1 rx=0x7f3d88008790 tx=0x7f3d880118a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:41:03.983 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.965+0000 7f3d827fc700 1 --2- 172.21.15.42:0/1537009819 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3d7403b270 0x7f3d7403d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:41:03.983 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.965+0000 7f3d827fc700 1 -- 172.21.15.42:0/1537009819 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 8181+0+0 (secure 0 0 0) 0x7f3d8804b610 con 0x7f3d8c1215b0 2019-12-05T05:41:03.983 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.965+0000 7f3d916e6700 1 --2- 172.21.15.42:0/1537009819 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3d7403b270 0x7f3d7403d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:41:03.983 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.965+0000 7f3d916e6700 1 --2- 172.21.15.42:0/1537009819 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3d7403b270 0x7f3d7403d780 crc :-1 s=READY pgs=550 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:41:03.983 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.973+0000 7f3d9394a700 1 -- 172.21.15.42:0/1537009819 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d8c121f80 con 0x7f3d8c1215b0 2019-12-05T05:41:03.986 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.981+0000 7f3d91ee7700 1 --2- 172.21.15.42:0/1537009819 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3d8c1215b0 0x7f3d8c124030 secure :-1 s=THROTTLE_DONE pgs=264 cs=0 l=1 rx=0x7f3d88008790 tx=0x7f3d880118a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:41:03.986 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:03.981+0000 7f3d827fc700 1 -- 172.21.15.42:0/1537009819 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f3d88033480 con 0x7f3d8c1215b0 2019-12-05T05:41:04.157 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:04.153+0000 7f3d9394a700 1 -- 172.21.15.42:0/1537009819 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3d8c05b8c0 con 0x7f3d8c1215b0 2019-12-05T05:41:04.158 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:04.153+0000 7f3d91ee7700 1 --2- 172.21.15.42:0/1537009819 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3d8c1215b0 0x7f3d8c124030 secure :-1 s=THROTTLE_DONE pgs=264 cs=0 l=1 rx=0x7f3d88008790 tx=0x7f3d880118a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:41:04.158 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:04.153+0000 7f3d827fc700 1 -- 172.21.15.42:0/1537009819 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 157 v157) v1 ==== 95+0+1775 (secure 0 0 0) 0x7f3d88005af0 con 0x7f3d8c1215b0 2019-12-05T05:41:04.161 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:41:04.161 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":157,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":157,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:41:03.431537+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7539},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7539":{"gid":7539,"name":"a","rank":0,"incarnation":154,"state":"up:active","state_seq":30,"addr":"172.21.15.42:6835/1233824070","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1233824070},{"type":"v1","addr":"172.21.15.42:6835","nonce":1233824070}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:41:04.161 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 157 2019-12-05T05:41:04.163 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:04.157+0000 7f3d9394a700 1 -- 172.21.15.42:0/1537009819 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3d7403b270 msgr2=0x7f3d7403d780 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:41:04.163 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:04.157+0000 7f3d9394a700 1 --2- 172.21.15.42:0/1537009819 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3d7403b270 0x7f3d7403d780 crc :-1 s=READY pgs=550 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:41:04.163 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:04.157+0000 7f3d9394a700 1 -- 172.21.15.42:0/1537009819 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3d8c1215b0 msgr2=0x7f3d8c124030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:41:04.163 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:04.157+0000 7f3d9394a700 1 --2- 172.21.15.42:0/1537009819 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3d8c1215b0 0x7f3d8c124030 secure :-1 s=READY pgs=264 cs=0 l=1 rx=0x7f3d88008790 tx=0x7f3d880118a0).stop 2019-12-05T05:41:04.163 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:04.157+0000 7f3d9394a700 1 -- 172.21.15.42:0/1537009819 shutdown_connections 2019-12-05T05:41:04.164 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:04.157+0000 7f3d9394a700 1 --2- 172.21.15.42:0/1537009819 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3d7403b270 0x7f3d7403d780 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:41:04.164 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:04.157+0000 7f3d9394a700 1 --2- 172.21.15.42:0/1537009819 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3d8c1215b0 0x7f3d8c124030 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:41:04.164 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:04.157+0000 7f3d9394a700 1 --2- 172.21.15.42:0/1537009819 >> v2:172.21.15.40:3300/0 conn(0x7f3d8c11b250 0x7f3d8c11a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:41:04.164 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:04.157+0000 7f3d9394a700 1 -- 172.21.15.42:0/1537009819 shutdown_connections 2019-12-05T05:41:04.164 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:04.157+0000 7f3d9394a700 1 -- 172.21.15.42:0/1537009819 wait complete. 2019-12-05T05:41:04.164 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:04.157+0000 7f3d9394a700 1 -- 172.21.15.42:0/1537009819 >> 172.21.15.42:0/1537009819 conn(0x7f3d8c0a7740 msgr2=0x7f3d8c0a9790 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:41:04.172 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2019-12-05T05:41:06.173 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:41:06.286 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:06.273+0000 7f0c71e09700 1 Processor -- start 2019-12-05T05:41:06.287 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:06.273+0000 7f0c71e09700 1 -- start start 2019-12-05T05:41:06.287 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:06.273+0000 7f0c71e09700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0c6c0a7740 0x7f0c6c08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:41:06.287 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:06.273+0000 7f0c71e09700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0c6c07e720 0x7f0c6c07eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:41:06.287 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:06.273+0000 7f0c71e09700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c6c0a0710 con 0x7f0c6c09f2e0 2019-12-05T05:41:06.287 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:06.273+0000 7f0c71e09700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c6c0a0b00 con 0x7f0c6c07e720 2019-12-05T05:41:06.288 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:06.273+0000 7f0c71e09700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c6c08e5b0 con 0x7f0c6c0a7740 2019-12-05T05:41:06.288 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:06.273+0000 7f0c6b7fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0c6c07e720 0x7f0c6c07eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:41:06.288 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:06.273+0000 7f0c6b7fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0c6c07e720 0x7f0c6c07eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:38784/0 (socket says 172.21.15.42:38784) 2019-12-05T05:41:06.288 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:06.273+0000 7f0c6b7fe700 1 -- 172.21.15.42:0/1722201866 learned_addr learned my addr 172.21.15.42:0/1722201866 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:41:06.288 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:06.273+0000 7f0c6bfff700 1 --2- 172.21.15.42:0/1722201866 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0c6c0a7740 0x7f0c6c08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:41:06.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:06.273+0000 7f0c6b7fe700 1 -- 172.21.15.42:0/1722201866 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0c6c0a7740 msgr2=0x7f0c6c08df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:41:06.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:06.273+0000 7f0c6b7fe700 1 --2- 172.21.15.42:0/1722201866 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0c6c0a7740 0x7f0c6c08df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:41:06.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:06.273+0000 7f0c6b7fe700 1 -- 172.21.15.42:0/1722201866 >> v1:172.21.15.42:6789/0 conn(0x7f0c6c09f2e0 legacy=0x7f0c6c09f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:41:06.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:06.273+0000 7f0c6b7fe700 1 -- 172.21.15.42:0/1722201866 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c6c07f0a0 con 0x7f0c6c07e720 2019-12-05T05:41:06.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:06.273+0000 7f0c6b7fe700 1 --2- 172.21.15.42:0/1722201866 >> v2:172.21.15.42:3300/0 conn(0x7f0c6c07e720 0x7f0c6c07eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f0c54008790 tx=0x7f0c5400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:41:06.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:06.273+0000 7f0c6b7fe700 1 --2- 172.21.15.42:0/1722201866 >> v2:172.21.15.42:3300/0 conn(0x7f0c6c07e720 0x7f0c6c07eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f0c54008790 tx=0x7f0c5400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:41:06.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:06.273+0000 7f0c6b7fe700 1 --2- 172.21.15.42:0/1722201866 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0c6c07e720 0x7f0c6c07eb50 secure :-1 s=READY pgs=1274 cs=0 l=1 rx=0x7f0c54008790 tx=0x7f0c5400fd30).ready entity=mon.1 client_cookie=c26b9fb8c5daecbf server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:41:06.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:06.273+0000 7f0c6b7fe700 1 --2- 172.21.15.42:0/1722201866 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0c6c07e720 0x7f0c6c07eb50 secure :-1 s=THROTTLE_DONE pgs=1274 cs=0 l=1 rx=0x7f0c54008790 tx=0x7f0c5400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:41:06.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:06.273+0000 7f0c6affd700 1 -- 172.21.15.42:0/1722201866 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0c540113b0 con 0x7f0c6c07e720 2019-12-05T05:41:06.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:06.273+0000 7f0c6b7fe700 1 --2- 172.21.15.42:0/1722201866 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0c6c07e720 0x7f0c6c07eb50 secure :-1 s=THROTTLE_DONE pgs=1274 cs=0 l=1 rx=0x7f0c54008790 tx=0x7f0c5400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:41:06.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:06.273+0000 7f0c6b7fe700 1 --2- 172.21.15.42:0/1722201866 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0c6c07e720 0x7f0c6c07eb50 secure :-1 s=THROTTLE_DONE pgs=1274 cs=0 l=1 rx=0x7f0c54008790 tx=0x7f0c5400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:41:06.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:06.273+0000 7f0c6affd700 1 -- 172.21.15.42:0/1722201866 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c54008bb0 con 0x7f0c6c07e720 2019-12-05T05:41:06.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:06.273+0000 7f0c6affd700 1 -- 172.21.15.42:0/1722201866 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0c54011990 con 0x7f0c6c07e720 2019-12-05T05:41:06.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:06.273+0000 7f0c71e09700 1 -- 172.21.15.42:0/1722201866 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0c6c07e720 msgr2=0x7f0c6c07eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:41:06.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:06.273+0000 7f0c71e09700 1 --2- 172.21.15.42:0/1722201866 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0c6c07e720 0x7f0c6c07eb50 secure :-1 s=READY pgs=1274 cs=0 l=1 rx=0x7f0c54008790 tx=0x7f0c5400fd30).stop 2019-12-05T05:41:06.291 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:06.277+0000 7f0c71e09700 1 -- 172.21.15.42:0/1722201866 shutdown_connections 2019-12-05T05:41:06.291 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:06.277+0000 7f0c71e09700 1 --2- 172.21.15.42:0/1722201866 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0c6c0a7740 0x7f0c6c08df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:41:06.291 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:06.277+0000 7f0c71e09700 1 --2- 172.21.15.42:0/1722201866 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0c6c07e720 0x7f0c6c07eb50 unknown :-1 s=CLOSED pgs=1274 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:41:06.291 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:06.277+0000 7f0c71e09700 1 -- 172.21.15.42:0/1722201866 shutdown_connections 2019-12-05T05:41:06.291 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:06.277+0000 7f0c71e09700 1 -- 172.21.15.42:0/1722201866 wait complete. 2019-12-05T05:41:06.291 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:06.277+0000 7f0c71e09700 1 -- 172.21.15.42:0/1722201866 >> 172.21.15.42:0/1722201866 conn(0x7f0c6c0aabe0 msgr2=0x7f0c6c0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:41:06.291 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:06.277+0000 7f0c71e09700 1 Processor -- start 2019-12-05T05:41:06.292 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:06.277+0000 7f0c71e09700 1 -- start start 2019-12-05T05:41:06.292 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:06.277+0000 7f0c71e09700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f0c6c12cf30 0x7f0c6c124510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:41:06.292 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:06.277+0000 7f0c71e09700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c6c124a60 con 0x7f0c6c07d5b0 2019-12-05T05:41:06.292 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:06.277+0000 7f0c71e09700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c6c11f0d0 con 0x7f0c6c07d180 2019-12-05T05:41:06.292 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:06.277+0000 7f0c71e09700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c6c1304f0 con 0x7f0c6c12cf30 2019-12-05T05:41:06.292 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:06.277+0000 7f0c6b7fe700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f0c6c07d180 0x7f0c6c130c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:38826/0 (socket says 172.21.15.42:38826) 2019-12-05T05:41:06.292 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:06.277+0000 7f0c6b7fe700 1 -- 172.21.15.42:0/2813601966 learned_addr learned my addr 172.21.15.42:0/2813601966 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:41:06.293 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:06.277+0000 7f0c6bfff700 1 --2- 172.21.15.42:0/2813601966 >> v2:172.21.15.40:3300/0 conn(0x7f0c6c12cf30 0x7f0c6c124510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:41:06.293 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:06.277+0000 7f0c68ff9700 1 -- 172.21.15.42:0/2813601966 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3375648361 0 0) 0x7f0c6c11f0d0 con 0x7f0c6c07d180 2019-12-05T05:41:06.293 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:06.281+0000 7f0c68ff9700 1 -- 172.21.15.42:0/2813601966 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c500030b0 con 0x7f0c6c07d180 2019-12-05T05:41:06.293 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:06.281+0000 7f0c6bfff700 1 -- 172.21.15.42:0/2813601966 >> v1:172.21.15.42:6789/0 conn(0x7f0c6c07d180 legacy=0x7f0c6c130c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:41:06.293 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:06.281+0000 7f0c6bfff700 1 -- 172.21.15.42:0/2813601966 >> v1:172.21.15.40:6789/0 conn(0x7f0c6c07d5b0 legacy=0x7f0c6c137940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:41:06.293 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:06.281+0000 7f0c68ff9700 1 -- 172.21.15.42:0/2813601966 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2797202262 0 0) 0x7f0c6c124a60 con 0x7f0c6c07d5b0 2019-12-05T05:41:06.293 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:06.281+0000 7f0c6bfff700 1 -- 172.21.15.42:0/2813601966 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c6c12dca0 con 0x7f0c6c12cf30 2019-12-05T05:41:06.294 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:06.281+0000 7f0c6bfff700 1 --2- 172.21.15.42:0/2813601966 >> v2:172.21.15.40:3300/0 conn(0x7f0c6c12cf30 0x7f0c6c124510 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f0c5c0032b0 tx=0x7f0c5c010ce0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:41:06.294 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:06.281+0000 7f0c6bfff700 1 --2- 172.21.15.42:0/2813601966 >> v2:172.21.15.40:3300/0 conn(0x7f0c6c12cf30 0x7f0c6c124510 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f0c5c0032b0 tx=0x7f0c5c010ce0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:41:06.294 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:06.281+0000 7f0c6bfff700 1 --2- 172.21.15.42:0/2813601966 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0c6c12cf30 0x7f0c6c124510 secure :-1 s=READY pgs=996 cs=0 l=1 rx=0x7f0c5c0032b0 tx=0x7f0c5c010ce0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:41:06.294 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:06.281+0000 7f0c6bfff700 1 --2- 172.21.15.42:0/2813601966 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0c6c12cf30 0x7f0c6c124510 secure :-1 s=THROTTLE_DONE pgs=996 cs=0 l=1 rx=0x7f0c5c0032b0 tx=0x7f0c5c010ce0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:41:06.294 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:06.281+0000 7f0c68ff9700 1 -- 172.21.15.42:0/2813601966 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0c5c012c90 con 0x7f0c6c12cf30 2019-12-05T05:41:06.294 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:06.281+0000 7f0c6bfff700 1 --2- 172.21.15.42:0/2813601966 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0c6c12cf30 0x7f0c6c124510 secure :-1 s=THROTTLE_DONE pgs=996 cs=0 l=1 rx=0x7f0c5c0032b0 tx=0x7f0c5c010ce0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:41:06.294 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:06.281+0000 7f0c71e09700 1 -- 172.21.15.42:0/2813601966 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c6c139150 con 0x7f0c6c12cf30 2019-12-05T05:41:06.295 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:06.281+0000 7f0c68ff9700 1 -- 172.21.15.42:0/2813601966 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c5c0128b0 con 0x7f0c6c12cf30 2019-12-05T05:41:06.295 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:06.281+0000 7f0c6bfff700 1 --2- 172.21.15.42:0/2813601966 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0c6c12cf30 0x7f0c6c124510 secure :-1 s=THROTTLE_DONE pgs=996 cs=0 l=1 rx=0x7f0c5c0032b0 tx=0x7f0c5c010ce0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:41:06.295 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:06.281+0000 7f0c68ff9700 1 -- 172.21.15.42:0/2813601966 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0c5c00e770 con 0x7f0c6c12cf30 2019-12-05T05:41:06.295 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:06.281+0000 7f0c71e09700 1 -- 172.21.15.42:0/2813601966 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c6c139650 con 0x7f0c6c12cf30 2019-12-05T05:41:06.295 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:06.281+0000 7f0c6bfff700 1 --2- 172.21.15.42:0/2813601966 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0c6c12cf30 0x7f0c6c124510 secure :-1 s=THROTTLE_DONE pgs=996 cs=0 l=1 rx=0x7f0c5c0032b0 tx=0x7f0c5c010ce0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:41:06.295 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:06.281+0000 7f0c68ff9700 1 -- 172.21.15.42:0/2813601966 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f0c5c00ea50 con 0x7f0c6c12cf30 2019-12-05T05:41:06.296 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:06.281+0000 7f0c6bfff700 1 --2- 172.21.15.42:0/2813601966 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0c6c12cf30 0x7f0c6c124510 secure :-1 s=THROTTLE_DONE pgs=996 cs=0 l=1 rx=0x7f0c5c0032b0 tx=0x7f0c5c010ce0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:41:06.297 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:06.281+0000 7f0c68ff9700 1 --2- 172.21.15.42:0/2813601966 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0c5003b1e0 0x7f0c5003d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:41:06.297 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:06.281+0000 7f0c68ff9700 1 -- 172.21.15.42:0/2813601966 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 8181+0+0 (secure 0 0 0) 0x7f0c5c04b9d0 con 0x7f0c6c12cf30 2019-12-05T05:41:06.297 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:06.281+0000 7f0c6b7fe700 1 --2- 172.21.15.42:0/2813601966 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0c5003b1e0 0x7f0c5003d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:41:06.297 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:06.285+0000 7f0c6b7fe700 1 --2- 172.21.15.42:0/2813601966 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0c5003b1e0 0x7f0c5003d6f0 crc :-1 s=READY pgs=551 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:41:06.297 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:06.289+0000 7f0c71e09700 1 -- 172.21.15.42:0/2813601966 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c6c05d110 con 0x7f0c6c12cf30 2019-12-05T05:41:06.302 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:06.297+0000 7f0c6bfff700 1 --2- 172.21.15.42:0/2813601966 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0c6c12cf30 0x7f0c6c124510 secure :-1 s=THROTTLE_DONE pgs=996 cs=0 l=1 rx=0x7f0c5c0032b0 tx=0x7f0c5c010ce0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:41:06.302 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:06.297+0000 7f0c68ff9700 1 -- 172.21.15.42:0/2813601966 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f0c5c032a50 con 0x7f0c6c12cf30 2019-12-05T05:41:06.472 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:06.469+0000 7f0c71e09700 1 -- 172.21.15.42:0/2813601966 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0c6c0a71b0 con 0x7f0c6c12cf30 2019-12-05T05:41:06.473 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:06.469+0000 7f0c6bfff700 1 --2- 172.21.15.42:0/2813601966 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0c6c12cf30 0x7f0c6c124510 secure :-1 s=THROTTLE_DONE pgs=996 cs=0 l=1 rx=0x7f0c5c0032b0 tx=0x7f0c5c010ce0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:41:06.473 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:06.469+0000 7f0c68ff9700 1 -- 172.21.15.42:0/2813601966 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 157 v157) v1 ==== 95+0+1775 (secure 0 0 0) 0x7f0c5c032cb0 con 0x7f0c6c12cf30 2019-12-05T05:41:06.475 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:41:06.475 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":157,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":157,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:41:03.431537+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7539},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7539":{"gid":7539,"name":"a","rank":0,"incarnation":154,"state":"up:active","state_seq":30,"addr":"172.21.15.42:6835/1233824070","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1233824070},{"type":"v1","addr":"172.21.15.42:6835","nonce":1233824070}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:41:06.476 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 157 2019-12-05T05:41:06.478 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:06.473+0000 7f0c71e09700 1 -- 172.21.15.42:0/2813601966 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0c5003b1e0 msgr2=0x7f0c5003d6f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:41:06.479 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:06.473+0000 7f0c71e09700 1 --2- 172.21.15.42:0/2813601966 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0c5003b1e0 0x7f0c5003d6f0 crc :-1 s=READY pgs=551 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:41:06.479 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:06.473+0000 7f0c71e09700 1 -- 172.21.15.42:0/2813601966 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0c6c12cf30 msgr2=0x7f0c6c124510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:41:06.479 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:06.473+0000 7f0c71e09700 1 --2- 172.21.15.42:0/2813601966 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0c6c12cf30 0x7f0c6c124510 secure :-1 s=READY pgs=996 cs=0 l=1 rx=0x7f0c5c0032b0 tx=0x7f0c5c010ce0).stop 2019-12-05T05:41:06.479 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:06.473+0000 7f0c71e09700 1 -- 172.21.15.42:0/2813601966 shutdown_connections 2019-12-05T05:41:06.479 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:06.473+0000 7f0c71e09700 1 --2- 172.21.15.42:0/2813601966 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0c5003b1e0 0x7f0c5003d6f0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:41:06.479 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:06.473+0000 7f0c71e09700 1 --2- 172.21.15.42:0/2813601966 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0c6c12cf30 0x7f0c6c124510 unknown :-1 s=CLOSED pgs=996 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:41:06.479 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:06.473+0000 7f0c71e09700 1 -- 172.21.15.42:0/2813601966 shutdown_connections 2019-12-05T05:41:06.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:06.473+0000 7f0c71e09700 1 -- 172.21.15.42:0/2813601966 wait complete. 2019-12-05T05:41:06.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:06.473+0000 7f0c71e09700 1 -- 172.21.15.42:0/2813601966 >> 172.21.15.42:0/2813601966 conn(0x7f0c6c0a9870 msgr2=0x7f0c6c11c610 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:41:06.488 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2019-12-05T05:41:08.489 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:41:08.595 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:08.585+0000 7f061f629700 1 Processor -- start 2019-12-05T05:41:08.595 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:08.585+0000 7f061f629700 1 -- start start 2019-12-05T05:41:08.596 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:08.585+0000 7f061f629700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f06180a7740 0x7f061808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:41:08.596 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:08.585+0000 7f061f629700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f061807e720 0x7f061807eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:41:08.596 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:08.585+0000 7f061f629700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06180a0710 con 0x7f061809f2e0 2019-12-05T05:41:08.596 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:08.585+0000 7f061f629700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06180a0b00 con 0x7f061807e720 2019-12-05T05:41:08.597 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:08.585+0000 7f061f629700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f061808e5b0 con 0x7f06180a7740 2019-12-05T05:41:08.597 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:08.585+0000 7f061dbc6700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f06180a7740 0x7f061808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:41:08.597 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:08.585+0000 7f061e3c7700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f061809f2e0 0x7f061809f710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:38844/0 (socket says 172.21.15.42:38844) 2019-12-05T05:41:08.598 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:08.585+0000 7f061e3c7700 1 -- 172.21.15.42:0/2649485402 learned_addr learned my addr 172.21.15.42:0/2649485402 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:41:08.598 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:08.585+0000 7f061d3c5700 1 --2- 172.21.15.42:0/2649485402 >> v2:172.21.15.40:3300/0 conn(0x7f061807e720 0x7f061807eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:41:08.598 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:08.589+0000 7f061dbc6700 1 -- 172.21.15.42:0/2649485402 >> v2:172.21.15.40:3300/0 conn(0x7f061807e720 msgr2=0x7f061807eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:41:08.598 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:08.589+0000 7f061dbc6700 1 --2- 172.21.15.42:0/2649485402 >> v2:172.21.15.40:3300/0 conn(0x7f061807e720 0x7f061807eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:41:08.599 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:08.589+0000 7f061dbc6700 1 -- 172.21.15.42:0/2649485402 >> v1:172.21.15.42:6789/0 conn(0x7f061809f2e0 legacy=0x7f061809f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:41:08.599 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:08.589+0000 7f061cbc4700 1 -- 172.21.15.42:0/2649485402 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4271614934 0 0) 0x7f06180a0710 con 0x7f061809f2e0 2019-12-05T05:41:08.599 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:08.589+0000 7f061dbc6700 1 -- 172.21.15.42:0/2649485402 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f061807f0a0 con 0x7f06180a7740 2019-12-05T05:41:08.600 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:08.589+0000 7f061dbc6700 1 --2- 172.21.15.42:0/2649485402 >> v2:172.21.15.42:3300/0 conn(0x7f06180a7740 0x7f061808df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f0614002f30 tx=0x7f061400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:41:08.600 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:08.589+0000 7f061dbc6700 1 --2- 172.21.15.42:0/2649485402 >> v2:172.21.15.42:3300/0 conn(0x7f06180a7740 0x7f061808df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f0614002f30 tx=0x7f061400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:41:08.600 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:08.589+0000 7f061dbc6700 1 --2- 172.21.15.42:0/2649485402 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f06180a7740 0x7f061808df80 secure :-1 s=READY pgs=1277 cs=0 l=1 rx=0x7f0614002f30 tx=0x7f061400fc70).ready entity=mon.1 client_cookie=c823a50797c3e873 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:41:08.600 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:08.589+0000 7f061dbc6700 1 --2- 172.21.15.42:0/2649485402 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f06180a7740 0x7f061808df80 secure :-1 s=THROTTLE_DONE pgs=1277 cs=0 l=1 rx=0x7f0614002f30 tx=0x7f061400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:41:08.600 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:08.589+0000 7f061cbc4700 1 -- 172.21.15.42:0/2649485402 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f06140112b0 con 0x7f06180a7740 2019-12-05T05:41:08.600 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:08.589+0000 7f061dbc6700 1 --2- 172.21.15.42:0/2649485402 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f06180a7740 0x7f061808df80 secure :-1 s=THROTTLE_DONE pgs=1277 cs=0 l=1 rx=0x7f0614002f30 tx=0x7f061400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:41:08.600 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:08.589+0000 7f061dbc6700 1 --2- 172.21.15.42:0/2649485402 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f06180a7740 0x7f061808df80 secure :-1 s=THROTTLE_DONE pgs=1277 cs=0 l=1 rx=0x7f0614002f30 tx=0x7f061400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:41:08.601 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:08.589+0000 7f061cbc4700 1 -- 172.21.15.42:0/2649485402 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0614008ad0 con 0x7f06180a7740 2019-12-05T05:41:08.601 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:08.589+0000 7f061cbc4700 1 -- 172.21.15.42:0/2649485402 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0614011890 con 0x7f06180a7740 2019-12-05T05:41:08.601 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:08.589+0000 7f061f629700 1 -- 172.21.15.42:0/2649485402 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f06180a7740 msgr2=0x7f061808df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:41:08.601 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:08.589+0000 7f061f629700 1 --2- 172.21.15.42:0/2649485402 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f06180a7740 0x7f061808df80 secure :-1 s=READY pgs=1277 cs=0 l=1 rx=0x7f0614002f30 tx=0x7f061400fc70).stop 2019-12-05T05:41:08.601 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:08.589+0000 7f061f629700 1 -- 172.21.15.42:0/2649485402 shutdown_connections 2019-12-05T05:41:08.601 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:08.589+0000 7f061f629700 1 --2- 172.21.15.42:0/2649485402 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f06180a7740 0x7f061808df80 unknown :-1 s=CLOSED pgs=1277 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:41:08.601 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:08.589+0000 7f061f629700 1 --2- 172.21.15.42:0/2649485402 >> v2:172.21.15.40:3300/0 conn(0x7f061807e720 0x7f061807eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:41:08.602 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:08.589+0000 7f061f629700 1 -- 172.21.15.42:0/2649485402 shutdown_connections 2019-12-05T05:41:08.602 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:08.589+0000 7f061f629700 1 -- 172.21.15.42:0/2649485402 wait complete. 2019-12-05T05:41:08.602 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:08.589+0000 7f061f629700 1 -- 172.21.15.42:0/2649485402 >> 172.21.15.42:0/2649485402 conn(0x7f06180aabe0 msgr2=0x7f06180a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:41:08.606 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:08.593+0000 7f061f629700 1 Processor -- start 2019-12-05T05:41:08.606 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:08.593+0000 7f061f629700 1 -- start start 2019-12-05T05:41:08.606 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:08.593+0000 7f061f629700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f061811ccd0 0x7f06181174d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:41:08.606 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:08.593+0000 7f061f629700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f06181161c0 0x7f06181165f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:41:08.606 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:08.593+0000 7f061f629700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0618116b40 con 0x7f061808eb50 2019-12-05T05:41:08.606 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:08.593+0000 7f061f629700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06180a5ab0 con 0x7f06181161c0 2019-12-05T05:41:08.607 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:08.593+0000 7f061f629700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06180a5d50 con 0x7f061811ccd0 2019-12-05T05:41:08.607 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:08.593+0000 7f061e3c7700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f061808eb50 0x7f061808ef80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:38850/0 (socket says 172.21.15.42:38850) 2019-12-05T05:41:08.607 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:08.593+0000 7f061e3c7700 1 -- 172.21.15.42:0/586679464 learned_addr learned my addr 172.21.15.42:0/586679464 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:41:08.607 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:08.593+0000 7f061d3c5700 1 --2- 172.21.15.42:0/586679464 >> v2:172.21.15.40:3300/0 conn(0x7f06181161c0 0x7f06181165f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:41:08.608 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:08.593+0000 7f061dbc6700 1 --2- 172.21.15.42:0/586679464 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f061811ccd0 0x7f06181174d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:41:08.608 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:08.593+0000 7f060e7fc700 1 -- 172.21.15.42:0/586679464 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1423558412 0 0) 0x7f0618116b40 con 0x7f061808eb50 2019-12-05T05:41:08.608 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:08.593+0000 7f060e7fc700 1 -- 172.21.15.42:0/586679464 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06000030b0 con 0x7f061808eb50 2019-12-05T05:41:08.608 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:08.593+0000 7f061dbc6700 1 -- 172.21.15.42:0/586679464 >> v2:172.21.15.40:3300/0 conn(0x7f06181161c0 msgr2=0x7f06181165f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:41:08.608 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:08.593+0000 7f061dbc6700 1 --2- 172.21.15.42:0/586679464 >> v2:172.21.15.40:3300/0 conn(0x7f06181161c0 0x7f06181165f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:41:08.609 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:08.593+0000 7f061dbc6700 1 -- 172.21.15.42:0/586679464 >> v1:172.21.15.42:6789/0 conn(0x7f061808eb50 legacy=0x7f061808ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:41:08.609 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:08.593+0000 7f061dbc6700 1 -- 172.21.15.42:0/586679464 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06180a5730 con 0x7f061811ccd0 2019-12-05T05:41:08.609 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:08.593+0000 7f061dbc6700 1 --2- 172.21.15.42:0/586679464 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f061811ccd0 0x7f06181174d0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f0614002f30 tx=0x7f06140119d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:41:08.609 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:08.593+0000 7f061d3c5700 1 --2- 172.21.15.42:0/586679464 >> v2:172.21.15.40:3300/0 conn(0x7f06181161c0 0x7f06181165f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:41:08.609 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:08.593+0000 7f061dbc6700 1 --2- 172.21.15.42:0/586679464 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f061811ccd0 0x7f06181174d0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f0614002f30 tx=0x7f06140119d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:41:08.609 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:08.593+0000 7f061dbc6700 1 --2- 172.21.15.42:0/586679464 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f061811ccd0 0x7f06181174d0 secure :-1 s=READY pgs=265 cs=0 l=1 rx=0x7f0614002f30 tx=0x7f06140119d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:41:08.609 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:08.597+0000 7f061dbc6700 1 --2- 172.21.15.42:0/586679464 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f061811ccd0 0x7f06181174d0 secure :-1 s=THROTTLE_DONE pgs=265 cs=0 l=1 rx=0x7f0614002f30 tx=0x7f06140119d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:41:08.610 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:08.597+0000 7f060e7fc700 1 -- 172.21.15.42:0/586679464 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0614007340 con 0x7f061811ccd0 2019-12-05T05:41:08.610 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:08.597+0000 7f061dbc6700 1 --2- 172.21.15.42:0/586679464 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f061811ccd0 0x7f06181174d0 secure :-1 s=THROTTLE_DONE pgs=265 cs=0 l=1 rx=0x7f0614002f30 tx=0x7f06140119d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:41:08.610 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:08.597+0000 7f061dbc6700 1 --2- 172.21.15.42:0/586679464 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f061811ccd0 0x7f06181174d0 secure :-1 s=THROTTLE_DONE pgs=265 cs=0 l=1 rx=0x7f0614002f30 tx=0x7f06140119d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:41:08.610 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:08.597+0000 7f060e7fc700 1 -- 172.21.15.42:0/586679464 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06140114b0 con 0x7f061811ccd0 2019-12-05T05:41:08.610 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:08.597+0000 7f060e7fc700 1 -- 172.21.15.42:0/586679464 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0614005540 con 0x7f061811ccd0 2019-12-05T05:41:08.610 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:08.597+0000 7f061f629700 1 -- 172.21.15.42:0/586679464 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f061808f780 con 0x7f061811ccd0 2019-12-05T05:41:08.610 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:08.597+0000 7f061f629700 1 -- 172.21.15.42:0/586679464 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0618117a20 con 0x7f061811ccd0 2019-12-05T05:41:08.611 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:08.597+0000 7f061dbc6700 1 --2- 172.21.15.42:0/586679464 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f061811ccd0 0x7f06181174d0 secure :-1 s=THROTTLE_DONE pgs=265 cs=0 l=1 rx=0x7f0614002f30 tx=0x7f06140119d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:41:08.611 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:08.597+0000 7f060e7fc700 1 -- 172.21.15.42:0/586679464 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f06140057e0 con 0x7f061811ccd0 2019-12-05T05:41:08.611 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:08.597+0000 7f061dbc6700 1 --2- 172.21.15.42:0/586679464 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f061811ccd0 0x7f06181174d0 secure :-1 s=THROTTLE_DONE pgs=265 cs=0 l=1 rx=0x7f0614002f30 tx=0x7f06140119d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:41:08.611 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:08.597+0000 7f060e7fc700 1 --2- 172.21.15.42:0/586679464 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f060003b230 0x7f060003d740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:41:08.611 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:08.597+0000 7f060e7fc700 1 -- 172.21.15.42:0/586679464 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 8181+0+0 (secure 0 0 0) 0x7f061404b7b0 con 0x7f061811ccd0 2019-12-05T05:41:08.611 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:08.597+0000 7f061d3c5700 1 --2- 172.21.15.42:0/586679464 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f060003b230 0x7f060003d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:41:08.611 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:08.597+0000 7f061d3c5700 1 --2- 172.21.15.42:0/586679464 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f060003b230 0x7f060003d740 crc :-1 s=READY pgs=553 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:41:08.612 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:08.601+0000 7f061f629700 1 -- 172.21.15.42:0/586679464 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f061805d110 con 0x7f061811ccd0 2019-12-05T05:41:08.613 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:08.605+0000 7f061dbc6700 1 --2- 172.21.15.42:0/586679464 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f061811ccd0 0x7f06181174d0 secure :-1 s=THROTTLE_DONE pgs=265 cs=0 l=1 rx=0x7f0614002f30 tx=0x7f06140119d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:41:08.613 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:08.605+0000 7f060e7fc700 1 -- 172.21.15.42:0/586679464 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f0614032810 con 0x7f061811ccd0 2019-12-05T05:41:08.771 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:08.765+0000 7f061f629700 1 -- 172.21.15.42:0/586679464 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0618141b20 con 0x7f061811ccd0 2019-12-05T05:41:08.772 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:08.769+0000 7f061dbc6700 1 --2- 172.21.15.42:0/586679464 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f061811ccd0 0x7f06181174d0 secure :-1 s=THROTTLE_DONE pgs=265 cs=0 l=1 rx=0x7f0614002f30 tx=0x7f06140119d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:41:08.772 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:08.769+0000 7f060e7fc700 1 -- 172.21.15.42:0/586679464 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 158 v158) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f0614032a70 con 0x7f061811ccd0 2019-12-05T05:41:08.774 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 158 2019-12-05T05:41:08.774 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:41:08.775 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":158,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7650,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/1029013337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1029013337},{"type":"v1","addr":"172.21.15.40:6833","nonce":1029013337}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":158}],"filesystems":[{"mdsmap":{"epoch":157,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:41:03.431537+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7539},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7539":{"gid":7539,"name":"a","rank":0,"incarnation":154,"state":"up:active","state_seq":30,"addr":"172.21.15.42:6835/1233824070","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1233824070},{"type":"v1","addr":"172.21.15.42:6835","nonce":1233824070}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:41:08.778 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:08.773+0000 7f061f629700 1 -- 172.21.15.42:0/586679464 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f060003b230 msgr2=0x7f060003d740 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:41:08.778 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:08.773+0000 7f061f629700 1 --2- 172.21.15.42:0/586679464 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f060003b230 0x7f060003d740 crc :-1 s=READY pgs=553 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:41:08.778 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:08.773+0000 7f061f629700 1 -- 172.21.15.42:0/586679464 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f061811ccd0 msgr2=0x7f06181174d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:41:08.778 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:08.773+0000 7f061f629700 1 --2- 172.21.15.42:0/586679464 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f061811ccd0 0x7f06181174d0 secure :-1 s=READY pgs=265 cs=0 l=1 rx=0x7f0614002f30 tx=0x7f06140119d0).stop 2019-12-05T05:41:08.778 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:08.773+0000 7f061f629700 1 -- 172.21.15.42:0/586679464 shutdown_connections 2019-12-05T05:41:08.778 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:08.773+0000 7f061f629700 1 --2- 172.21.15.42:0/586679464 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f060003b230 0x7f060003d740 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:41:08.779 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:08.773+0000 7f061f629700 1 --2- 172.21.15.42:0/586679464 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f061811ccd0 0x7f06181174d0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:41:08.779 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:08.773+0000 7f061f629700 1 --2- 172.21.15.42:0/586679464 >> v2:172.21.15.40:3300/0 conn(0x7f06181161c0 0x7f06181165f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:41:08.779 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:08.773+0000 7f061f629700 1 -- 172.21.15.42:0/586679464 shutdown_connections 2019-12-05T05:41:08.779 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:08.773+0000 7f061f629700 1 -- 172.21.15.42:0/586679464 wait complete. 2019-12-05T05:41:08.779 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:41:08.773+0000 7f061f629700 1 -- 172.21.15.42:0/586679464 >> 172.21.15.42:0/586679464 conn(0x7f061808fde0 msgr2=0x7f061809f690 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:41:08.793 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b reported in up:standby state 2019-12-05T05:41:08.793 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 80 secs before thrashing 2019-12-05T05:41:10.794 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:41:10.798 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:41:40.850 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:41:40.855 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:42:10.921 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:42:10.925 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:42:28.793 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:42:28.902 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:28.889+0000 7feaea855700 1 Processor -- start 2019-12-05T05:42:28.902 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:28.889+0000 7feaea855700 1 -- start start 2019-12-05T05:42:28.903 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:28.893+0000 7feaea855700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7feae40a7740 0x7feae408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:42:28.903 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:28.893+0000 7feaea855700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feae40a0710 con 0x7feae407e720 2019-12-05T05:42:28.903 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:28.893+0000 7feaea855700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feae40a9ba0 con 0x7feae409fae0 2019-12-05T05:42:28.903 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:28.893+0000 7feaea855700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feae40a0b00 con 0x7feae40a7740 2019-12-05T05:42:28.903 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:28.893+0000 7feae95f3700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7feae409fae0 0x7feae414efe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:39206/0 (socket says 172.21.15.42:39206) 2019-12-05T05:42:28.903 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:28.893+0000 7feae95f3700 1 -- 172.21.15.42:0/1462411416 learned_addr learned my addr 172.21.15.42:0/1462411416 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:42:28.904 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:28.893+0000 7feae8df2700 1 --2- 172.21.15.42:0/1462411416 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7feae40a7740 0x7feae408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:42:28.904 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:28.893+0000 7feae37fe700 1 -- 172.21.15.42:0/1462411416 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 547079983 0 0) 0x7feae40a9ba0 con 0x7feae409fae0 2019-12-05T05:42:28.904 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:28.893+0000 7feae37fe700 1 -- 172.21.15.42:0/1462411416 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feacc0030b0 con 0x7feae409fae0 2019-12-05T05:42:28.904 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:28.893+0000 7feae37fe700 1 -- 172.21.15.42:0/1462411416 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1449350432 0 0) 0x7feae40a0710 con 0x7feae407e720 2019-12-05T05:42:28.904 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:28.893+0000 7feae37fe700 1 -- 172.21.15.42:0/1462411416 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feae40a9ba0 con 0x7feae407e720 2019-12-05T05:42:28.904 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:28.893+0000 7feae37fe700 1 -- 172.21.15.42:0/1462411416 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 781141260 0 0) 0x7feacc0030b0 con 0x7feae409fae0 2019-12-05T05:42:28.904 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:28.893+0000 7feae37fe700 1 -- 172.21.15.42:0/1462411416 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7feae40a7740 msgr2=0x7feae408df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:42:28.905 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:28.893+0000 7feae37fe700 1 --2- 172.21.15.42:0/1462411416 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7feae40a7740 0x7feae408df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:42:28.905 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:28.893+0000 7feae37fe700 1 -- 172.21.15.42:0/1462411416 >> v1:172.21.15.40:6789/0 conn(0x7feae407e720 legacy=0x7feae409f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:42:28.905 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:28.893+0000 7feae37fe700 1 -- 172.21.15.42:0/1462411416 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feae407eb50 con 0x7feae409fae0 2019-12-05T05:42:28.905 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:28.893+0000 7feae37fe700 1 -- 172.21.15.42:0/1462411416 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7feadc001ad0 con 0x7feae409fae0 2019-12-05T05:42:28.905 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:28.893+0000 7feae8df2700 1 --2- 172.21.15.42:0/1462411416 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7feae40a7740 0x7feae408df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:42:28.905 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:28.893+0000 7feae37fe700 1 -- 172.21.15.42:0/1462411416 >> v1:172.21.15.42:6789/0 conn(0x7feae409fae0 legacy=0x7feae414efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:42:28.905 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:28.893+0000 7feae37fe700 1 --2- 172.21.15.42:0/1462411416 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feacc003940 0x7feacc003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:42:28.906 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:28.893+0000 7feae37fe700 1 --2- 172.21.15.42:0/1462411416 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feacc004470 0x7feacc004920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:42:28.906 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:28.893+0000 7feae37fe700 1 --2- 172.21.15.42:0/1462411416 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7feacc04d670 0x7feacc04fab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:42:28.906 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:28.893+0000 7feae8df2700 1 --2- 172.21.15.42:0/1462411416 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feacc003940 0x7feacc003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:42:28.906 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:28.893+0000 7feae37fe700 1 -- 172.21.15.42:0/1462411416 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feae40a0b00 con 0x7feacc004470 2019-12-05T05:42:28.906 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:28.893+0000 7feae37fe700 1 -- 172.21.15.42:0/1462411416 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feacc005130 con 0x7feacc003940 2019-12-05T05:42:28.906 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:28.893+0000 7feae37fe700 1 -- 172.21.15.42:0/1462411416 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feacc050000 con 0x7feacc04d670 2019-12-05T05:42:28.906 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:28.893+0000 7feae95f3700 1 --2- 172.21.15.42:0/1462411416 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7feacc04d670 0x7feacc04fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:42:28.907 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:28.893+0000 7feae3fff700 1 --2- 172.21.15.42:0/1462411416 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feacc004470 0x7feacc004920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:42:28.907 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:28.893+0000 7feae8df2700 1 -- 172.21.15.42:0/1462411416 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7feacc04d670 msgr2=0x7feacc04fab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:42:28.907 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:28.893+0000 7feae8df2700 1 --2- 172.21.15.42:0/1462411416 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7feacc04d670 0x7feacc04fab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:42:28.907 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:28.893+0000 7feae8df2700 1 -- 172.21.15.42:0/1462411416 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feacc004470 msgr2=0x7feacc004920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:42:28.907 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:28.893+0000 7feae8df2700 1 --2- 172.21.15.42:0/1462411416 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feacc004470 0x7feacc004920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:42:28.907 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:28.893+0000 7feae8df2700 1 -- 172.21.15.42:0/1462411416 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feacc050280 con 0x7feacc003940 2019-12-05T05:42:28.907 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:28.897+0000 7feae8df2700 1 --2- 172.21.15.42:0/1462411416 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feacc003940 0x7feacc003d90 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fead4002f30 tx=0x7fead400bed0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:28.907 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:28.897+0000 7feae8df2700 1 --2- 172.21.15.42:0/1462411416 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feacc003940 0x7feacc003d90 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fead4002f30 tx=0x7fead400bed0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:28.908 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:28.897+0000 7feae8df2700 1 --2- 172.21.15.42:0/1462411416 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feacc003940 0x7feacc003d90 secure :-1 s=READY pgs=1281 cs=0 l=1 rx=0x7fead4002f30 tx=0x7fead400bed0).ready entity=mon.1 client_cookie=d1b6a7ee16bebc66 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:42:28.908 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:28.897+0000 7feae8df2700 1 --2- 172.21.15.42:0/1462411416 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feacc003940 0x7feacc003d90 secure :-1 s=THROTTLE_DONE pgs=1281 cs=0 l=1 rx=0x7fead4002f30 tx=0x7fead400bed0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:28.908 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:28.897+0000 7feae8df2700 1 --2- 172.21.15.42:0/1462411416 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feacc003940 0x7feacc003d90 secure :-1 s=THROTTLE_DONE pgs=1281 cs=0 l=1 rx=0x7fead4002f30 tx=0x7fead400bed0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:28.908 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:28.897+0000 7feae37fe700 1 -- 172.21.15.42:0/1462411416 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fead4011a10 con 0x7feacc003940 2019-12-05T05:42:28.908 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:28.897+0000 7feae8df2700 1 --2- 172.21.15.42:0/1462411416 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feacc003940 0x7feacc003d90 secure :-1 s=THROTTLE_DONE pgs=1281 cs=0 l=1 rx=0x7fead4002f30 tx=0x7fead400bed0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:28.908 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:28.897+0000 7feae37fe700 1 -- 172.21.15.42:0/1462411416 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fead4007150 con 0x7feacc003940 2019-12-05T05:42:28.909 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:28.897+0000 7feae37fe700 1 -- 172.21.15.42:0/1462411416 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fead4005540 con 0x7feacc003940 2019-12-05T05:42:28.909 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:28.897+0000 7feaea855700 1 -- 172.21.15.42:0/1462411416 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feacc003940 msgr2=0x7feacc003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:42:28.909 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:28.897+0000 7feaea855700 1 --2- 172.21.15.42:0/1462411416 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feacc003940 0x7feacc003d90 secure :-1 s=READY pgs=1281 cs=0 l=1 rx=0x7fead4002f30 tx=0x7fead400bed0).stop 2019-12-05T05:42:28.909 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:28.897+0000 7feae95f3700 1 -- 172.21.15.42:0/1462411416 reap_dead start 2019-12-05T05:42:28.909 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:28.897+0000 7feaea855700 1 -- 172.21.15.42:0/1462411416 shutdown_connections 2019-12-05T05:42:28.909 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:28.897+0000 7feaea855700 1 -- 172.21.15.42:0/1462411416 shutdown_connections 2019-12-05T05:42:28.909 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:28.897+0000 7feaea855700 1 -- 172.21.15.42:0/1462411416 wait complete. 2019-12-05T05:42:28.910 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:28.897+0000 7feaea855700 1 -- 172.21.15.42:0/1462411416 >> 172.21.15.42:0/1462411416 conn(0x7feae40aabe0 msgr2=0x7feae40a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:42:28.914 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:28.897+0000 7feaea855700 1 Processor -- start 2019-12-05T05:42:28.914 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:28.901+0000 7feaea855700 1 -- start start 2019-12-05T05:42:28.914 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:28.901+0000 7feaea855700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7feae41481e0 0x7feae407af60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:42:28.914 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:28.901+0000 7feaea855700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7feae407b4b0 0x7feae407b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:42:28.914 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:28.901+0000 7feaea855700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feae40a1a10 con 0x7feae409f620 2019-12-05T05:42:28.915 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:28.901+0000 7feaea855700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feae40a3850 con 0x7feae41481e0 2019-12-05T05:42:28.915 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:28.901+0000 7feaea855700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feae40a3af0 con 0x7feae407b4b0 2019-12-05T05:42:28.915 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:28.901+0000 7feae3fff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7feae41481e0 0x7feae407af60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:42:28.915 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:28.901+0000 7feae3fff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7feae41481e0 0x7feae407af60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:39180/0 (socket says 172.21.15.42:39180) 2019-12-05T05:42:28.915 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:28.901+0000 7feae3fff700 1 -- 172.21.15.42:0/855777346 learned_addr learned my addr 172.21.15.42:0/855777346 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:42:28.915 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:28.901+0000 7feae95f3700 1 --2- 172.21.15.42:0/855777346 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7feae407b4b0 0x7feae407b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:42:28.915 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:28.901+0000 7feae3fff700 1 -- 172.21.15.42:0/855777346 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7feae407b4b0 msgr2=0x7feae407b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:42:28.916 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:28.901+0000 7feae3fff700 1 --2- 172.21.15.42:0/855777346 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7feae407b4b0 0x7feae407b8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:42:28.916 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:28.901+0000 7feae3fff700 1 -- 172.21.15.42:0/855777346 >> v1:172.21.15.40:6789/0 conn(0x7feae409f620 legacy=0x7feae40a1f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:42:28.916 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:28.901+0000 7feae17fa700 1 -- 172.21.15.42:0/855777346 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 624537444 0 0) 0x7feae40a1a10 con 0x7feae409f620 2019-12-05T05:42:28.916 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:28.901+0000 7feae3fff700 1 -- 172.21.15.42:0/855777346 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feae4148610 con 0x7feae41481e0 2019-12-05T05:42:28.916 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:28.901+0000 7feae3fff700 1 --2- 172.21.15.42:0/855777346 >> v2:172.21.15.42:3300/0 conn(0x7feae41481e0 0x7feae407af60 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7feae409fdb0 tx=0x7fead8010b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:28.916 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:28.901+0000 7feae3fff700 1 --2- 172.21.15.42:0/855777346 >> v2:172.21.15.42:3300/0 conn(0x7feae41481e0 0x7feae407af60 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7feae409fdb0 tx=0x7fead8010b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:28.916 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:28.901+0000 7feae3fff700 1 --2- 172.21.15.42:0/855777346 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feae41481e0 0x7feae407af60 secure :-1 s=READY pgs=1282 cs=0 l=1 rx=0x7feae409fdb0 tx=0x7fead8010b00).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:42:28.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:28.901+0000 7feae3fff700 1 -- 172.21.15.42:0/855777346 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feae41481e0 msgr2=0x7feae407af60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2019-12-05T05:42:28.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:28.901+0000 7feae3fff700 1 -- 172.21.15.42:0/855777346 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feae41481e0 msgr2=0x7feae407af60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2019-12-05T05:42:28.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:28.901+0000 7feae3fff700 1 --2- 172.21.15.42:0/855777346 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feae41481e0 0x7feae407af60 secure :-1 s=READY pgs=1282 cs=0 l=1 rx=0x7feae409fdb0 tx=0x7fead8010b00).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-12-05T05:42:28.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:28.901+0000 7feae3fff700 1 --2- 172.21.15.42:0/855777346 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feae41481e0 0x7feae407af60 secure :-1 s=READY pgs=1282 cs=0 l=1 rx=0x7feae409fdb0 tx=0x7fead8010b00).stop 2019-12-05T05:42:28.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:28.901+0000 7feae17fa700 1 -- 172.21.15.42:0/855777346 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feae41481e0 msgr2=0x7feae407af60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2019-12-05T05:42:28.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:28.901+0000 7feae17fa700 1 --2- 172.21.15.42:0/855777346 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feae41481e0 0x7feae407af60 unknown :-1 s=CLOSED pgs=1282 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:42:28.918 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:28.901+0000 7feae17fa700 1 --2- 172.21.15.42:0/855777346 >> v2:172.21.15.40:3300/0 conn(0x7feac4003050 0x7feac4005520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:42:28.918 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:28.901+0000 7feae17fa700 1 -- 172.21.15.42:0/855777346 --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feae40a1a10 con 0x7feac4005c10 2019-12-05T05:42:28.918 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:28.901+0000 7feae17fa700 1 -- 172.21.15.42:0/855777346 --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feac400aba0 con 0x7feac4003050 2019-12-05T05:42:28.918 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:28.901+0000 7feae8df2700 1 --2- 172.21.15.42:0/855777346 >> v2:172.21.15.40:3300/0 conn(0x7feac4003050 0x7feac4005520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:42:28.918 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:28.901+0000 7feae17fa700 1 -- 172.21.15.42:0/855777346 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3554105861 0 0) 0x7feae40a1a10 con 0x7feac4005c10 2019-12-05T05:42:28.918 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:28.901+0000 7feae17fa700 1 -- 172.21.15.42:0/855777346 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feac4009be0 con 0x7feac4005c10 2019-12-05T05:42:28.918 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:28.901+0000 7feae17fa700 1 -- 172.21.15.42:0/855777346 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 1373221957 0 0) 0x7feac4009be0 con 0x7feac4005c10 2019-12-05T05:42:28.919 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:28.901+0000 7feae8df2700 1 -- 172.21.15.42:0/855777346 >> v1:172.21.15.42:6789/0 conn(0x7feac4005c10 legacy=0x7feac4008120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:42:28.919 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:28.901+0000 7feae8df2700 1 -- 172.21.15.42:0/855777346 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feac400adf0 con 0x7feac4003050 2019-12-05T05:42:28.919 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:28.905+0000 7feae8df2700 1 --2- 172.21.15.42:0/855777346 >> v2:172.21.15.40:3300/0 conn(0x7feac4003050 0x7feac4005520 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fead8001660 tx=0x7fead400ba60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:28.919 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:28.905+0000 7feae8df2700 1 --2- 172.21.15.42:0/855777346 >> v2:172.21.15.40:3300/0 conn(0x7feac4003050 0x7feac4005520 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fead8001660 tx=0x7fead400ba60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:28.919 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:28.905+0000 7feae8df2700 1 --2- 172.21.15.42:0/855777346 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feac4003050 0x7feac4005520 secure :-1 s=READY pgs=1003 cs=0 l=1 rx=0x7fead8001660 tx=0x7fead400ba60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:42:28.919 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:28.905+0000 7feae8df2700 1 --2- 172.21.15.42:0/855777346 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feac4003050 0x7feac4005520 secure :-1 s=THROTTLE_DONE pgs=1003 cs=0 l=1 rx=0x7fead8001660 tx=0x7fead400ba60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:28.919 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:28.905+0000 7feae17fa700 1 -- 172.21.15.42:0/855777346 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fead800ccc0 con 0x7feac4003050 2019-12-05T05:42:28.920 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:28.905+0000 7feae8df2700 1 --2- 172.21.15.42:0/855777346 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feac4003050 0x7feac4005520 secure :-1 s=THROTTLE_DONE pgs=1003 cs=0 l=1 rx=0x7fead8001660 tx=0x7fead400ba60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:28.920 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:28.905+0000 7feae17fa700 1 -- 172.21.15.42:0/855777346 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fead800cad0 con 0x7feac4003050 2019-12-05T05:42:28.920 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:28.905+0000 7feae8df2700 1 --2- 172.21.15.42:0/855777346 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feac4003050 0x7feac4005520 secure :-1 s=THROTTLE_DONE pgs=1003 cs=0 l=1 rx=0x7fead8001660 tx=0x7fead400ba60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:28.920 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:28.905+0000 7feaea855700 1 -- 172.21.15.42:0/855777346 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feae40a16c0 con 0x7feac4003050 2019-12-05T05:42:28.920 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:28.905+0000 7feae17fa700 1 -- 172.21.15.42:0/855777346 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fead4005b30 con 0x7feac4003050 2019-12-05T05:42:28.920 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:28.905+0000 7feaea855700 1 -- 172.21.15.42:0/855777346 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feae40a4030 con 0x7feac4003050 2019-12-05T05:42:28.920 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:28.905+0000 7feae8df2700 1 --2- 172.21.15.42:0/855777346 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feac4003050 0x7feac4005520 secure :-1 s=THROTTLE_DONE pgs=1003 cs=0 l=1 rx=0x7fead8001660 tx=0x7fead400ba60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:28.921 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:28.905+0000 7feae17fa700 1 -- 172.21.15.42:0/855777346 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fead400d420 con 0x7feac4003050 2019-12-05T05:42:28.921 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:28.905+0000 7feae8df2700 1 --2- 172.21.15.42:0/855777346 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feac4003050 0x7feac4005520 secure :-1 s=THROTTLE_DONE pgs=1003 cs=0 l=1 rx=0x7fead8001660 tx=0x7fead400ba60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:28.921 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:28.905+0000 7feae17fa700 1 --2- 172.21.15.42:0/855777346 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7feac4042140 0x7feac4044650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:42:28.921 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:28.905+0000 7feae17fa700 1 -- 172.21.15.42:0/855777346 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 8181+0+0 (secure 0 0 0) 0x7fead404ac40 con 0x7feac4003050 2019-12-05T05:42:28.921 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:28.905+0000 7feae3fff700 1 --2- 172.21.15.42:0/855777346 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7feac4042140 0x7feac4044650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:42:28.921 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:28.905+0000 7feaea855700 1 -- 172.21.15.42:0/855777346 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feae405d110 con 0x7feac4003050 2019-12-05T05:42:28.921 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:28.905+0000 7feae3fff700 1 --2- 172.21.15.42:0/855777346 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7feac4042140 0x7feac4044650 crc :-1 s=READY pgs=555 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:42:28.922 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:28.917+0000 7feae8df2700 1 --2- 172.21.15.42:0/855777346 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feac4003050 0x7feac4005520 secure :-1 s=THROTTLE_DONE pgs=1003 cs=0 l=1 rx=0x7fead8001660 tx=0x7fead400ba60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:28.922 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:28.917+0000 7feae17fa700 1 -- 172.21.15.42:0/855777346 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fead4005dd0 con 0x7feac4003050 2019-12-05T05:42:29.103 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:29.085+0000 7feaea855700 1 -- 172.21.15.42:0/855777346 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7feae405b8c0 con 0x7feac4003050 2019-12-05T05:42:29.103 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:29.085+0000 7feae8df2700 1 --2- 172.21.15.42:0/855777346 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feac4003050 0x7feac4005520 secure :-1 s=THROTTLE_DONE pgs=1003 cs=0 l=1 rx=0x7fead8001660 tx=0x7fead400ba60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:29.103 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:29.085+0000 7feae17fa700 1 -- 172.21.15.42:0/855777346 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 158 v158) v1 ==== 95+0+2112 (secure 0 0 0) 0x7fead400dd20 con 0x7feac4003050 2019-12-05T05:42:29.103 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 158 2019-12-05T05:42:29.104 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:29.089+0000 7feaea855700 1 -- 172.21.15.42:0/855777346 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7feac4042140 msgr2=0x7feac4044650 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:42:29.104 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:29.089+0000 7feaea855700 1 --2- 172.21.15.42:0/855777346 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7feac4042140 0x7feac4044650 crc :-1 s=READY pgs=555 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:42:29.104 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:29.089+0000 7feaea855700 1 -- 172.21.15.42:0/855777346 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feac4003050 msgr2=0x7feac4005520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:42:29.104 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:29.089+0000 7feaea855700 1 --2- 172.21.15.42:0/855777346 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feac4003050 0x7feac4005520 secure :-1 s=READY pgs=1003 cs=0 l=1 rx=0x7fead8001660 tx=0x7fead400ba60).stop 2019-12-05T05:42:29.104 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:29.089+0000 7feae95f3700 1 -- 172.21.15.42:0/855777346 reap_dead start 2019-12-05T05:42:29.104 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:29.089+0000 7feae95f3700 1 -- 172.21.15.42:0/855777346 reap_dead start 2019-12-05T05:42:29.104 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:29.089+0000 7feaea855700 1 -- 172.21.15.42:0/855777346 shutdown_connections 2019-12-05T05:42:29.105 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:29.089+0000 7feaea855700 1 --2- 172.21.15.42:0/855777346 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feac4003050 0x7feac4005520 unknown :-1 s=CLOSED pgs=1003 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:42:29.105 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:29.089+0000 7feaea855700 1 --2- 172.21.15.42:0/855777346 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feae41481e0 0x7feae407af60 unknown :-1 s=CLOSED pgs=1282 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:42:29.105 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:29.089+0000 7feaea855700 1 -- 172.21.15.42:0/855777346 shutdown_connections 2019-12-05T05:42:29.105 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:29.089+0000 7feaea855700 1 -- 172.21.15.42:0/855777346 wait complete. 2019-12-05T05:42:29.105 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:29.089+0000 7feaea855700 1 -- 172.21.15.42:0/855777346 >> 172.21.15.42:0/855777346 conn(0x7feae40a82a0 msgr2=0x7feae411c3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:42:29.105 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:42:29.105 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":158,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7650,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/1029013337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1029013337},{"type":"v1","addr":"172.21.15.40:6833","nonce":1029013337}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":158}],"filesystems":[{"mdsmap":{"epoch":157,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:41:03.431537+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7539},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7539":{"gid":7539,"name":"a","rank":0,"incarnation":154,"state":"up:active","state_seq":30,"addr":"172.21.15.42:6835/1233824070","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1233824070},{"type":"v1","addr":"172.21.15.42:6835","nonce":1233824070}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:42:29.107 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.a (rank=0) 2019-12-05T05:42:29.107 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2019-12-05T05:42:29.107 INFO:teuthology.orchestra.run:waiting for 300 2019-12-05T05:42:29.122 INFO:tasks.ceph.mds.a:Stopped 2019-12-05T05:42:29.122 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:42:29.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:29.277+0000 7f0eac1cc700 1 Processor -- start 2019-12-05T05:42:29.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:29.281+0000 7f0eac1cc700 1 -- start start 2019-12-05T05:42:29.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:29.281+0000 7f0eac1cc700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0ea40a7740 0x7f0ea408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:42:29.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:29.281+0000 7f0eac1cc700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0ea409fae0 0x7f0ea407eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:42:29.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:29.281+0000 7f0eac1cc700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ea40a0710 con 0x7f0ea407e720 2019-12-05T05:42:29.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:29.281+0000 7f0eac1cc700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ea40a0b00 con 0x7f0ea40a7740 2019-12-05T05:42:29.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:29.281+0000 7f0eac1cc700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ea408e5b0 con 0x7f0ea409fae0 2019-12-05T05:42:29.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:29.281+0000 7f0eaa769700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0ea40a7740 0x7f0ea408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:42:29.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:29.281+0000 7f0eaa769700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0ea40a7740 0x7f0ea408df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:39190/0 (socket says 172.21.15.42:39190) 2019-12-05T05:42:29.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:29.281+0000 7f0eaa769700 1 -- 172.21.15.42:0/1891821753 learned_addr learned my addr 172.21.15.42:0/1891821753 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:42:29.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:29.281+0000 7f0eaaf6a700 1 --2- 172.21.15.42:0/1891821753 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0ea409fae0 0x7f0ea407eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:42:29.291 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:29.281+0000 7f0eaa769700 1 -- 172.21.15.42:0/1891821753 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0ea409fae0 msgr2=0x7f0ea407eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:42:29.291 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:29.281+0000 7f0eaa769700 1 --2- 172.21.15.42:0/1891821753 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0ea409fae0 0x7f0ea407eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:42:29.291 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:29.281+0000 7f0eaa769700 1 -- 172.21.15.42:0/1891821753 >> v1:172.21.15.40:6789/0 conn(0x7f0ea407e720 legacy=0x7f0ea409f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:42:29.291 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:29.281+0000 7f0eaa769700 1 -- 172.21.15.42:0/1891821753 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ea407f0a0 con 0x7f0ea40a7740 2019-12-05T05:42:29.291 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:29.281+0000 7f0eaa769700 1 --2- 172.21.15.42:0/1891821753 >> v2:172.21.15.42:3300/0 conn(0x7f0ea40a7740 0x7f0ea408df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f0ea0008790 tx=0x7f0ea000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:29.291 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:29.281+0000 7f0eaa769700 1 --2- 172.21.15.42:0/1891821753 >> v2:172.21.15.42:3300/0 conn(0x7f0ea40a7740 0x7f0ea408df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f0ea0008790 tx=0x7f0ea000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:29.291 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:29.281+0000 7f0eaa769700 1 --2- 172.21.15.42:0/1891821753 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0ea40a7740 0x7f0ea408df80 secure :-1 s=READY pgs=1284 cs=0 l=1 rx=0x7f0ea0008790 tx=0x7f0ea000fd30).ready entity=mon.1 client_cookie=25f9c0e747e8a206 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:42:29.292 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:29.281+0000 7f0eaa769700 1 --2- 172.21.15.42:0/1891821753 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0ea40a7740 0x7f0ea408df80 secure :-1 s=THROTTLE_DONE pgs=1284 cs=0 l=1 rx=0x7f0ea0008790 tx=0x7f0ea000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:29.292 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:29.281+0000 7f0ea9767700 1 -- 172.21.15.42:0/1891821753 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0ea0008c90 con 0x7f0ea40a7740 2019-12-05T05:42:29.292 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:29.281+0000 7f0eaa769700 1 --2- 172.21.15.42:0/1891821753 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0ea40a7740 0x7f0ea408df80 secure :-1 s=THROTTLE_DONE pgs=1284 cs=0 l=1 rx=0x7f0ea0008790 tx=0x7f0ea000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:29.292 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:29.281+0000 7f0ea9767700 1 -- 172.21.15.42:0/1891821753 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ea00088b0 con 0x7f0ea40a7740 2019-12-05T05:42:29.292 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:29.281+0000 7f0eaa769700 1 --2- 172.21.15.42:0/1891821753 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0ea40a7740 0x7f0ea408df80 secure :-1 s=THROTTLE_DONE pgs=1284 cs=0 l=1 rx=0x7f0ea0008790 tx=0x7f0ea000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:29.292 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:29.281+0000 7f0eac1cc700 1 -- 172.21.15.42:0/1891821753 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0ea40a7740 msgr2=0x7f0ea408df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:42:29.292 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:29.281+0000 7f0ea9767700 1 -- 172.21.15.42:0/1891821753 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0ea0011b10 con 0x7f0ea40a7740 2019-12-05T05:42:29.293 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:29.281+0000 7f0eac1cc700 1 --2- 172.21.15.42:0/1891821753 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0ea40a7740 0x7f0ea408df80 secure :-1 s=READY pgs=1284 cs=0 l=1 rx=0x7f0ea0008790 tx=0x7f0ea000fd30).stop 2019-12-05T05:42:29.293 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:29.285+0000 7f0eac1cc700 1 -- 172.21.15.42:0/1891821753 shutdown_connections 2019-12-05T05:42:29.293 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:29.285+0000 7f0eac1cc700 1 --2- 172.21.15.42:0/1891821753 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0ea409fae0 0x7f0ea407eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:42:29.293 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:29.285+0000 7f0eac1cc700 1 --2- 172.21.15.42:0/1891821753 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0ea40a7740 0x7f0ea408df80 unknown :-1 s=CLOSED pgs=1284 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:42:29.293 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:29.285+0000 7f0eac1cc700 1 -- 172.21.15.42:0/1891821753 shutdown_connections 2019-12-05T05:42:29.293 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:29.285+0000 7f0eac1cc700 1 -- 172.21.15.42:0/1891821753 wait complete. 2019-12-05T05:42:29.293 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:29.285+0000 7f0eac1cc700 1 -- 172.21.15.42:0/1891821753 >> 172.21.15.42:0/1891821753 conn(0x7f0ea40aabe0 msgr2=0x7f0ea40a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:42:29.296 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:29.285+0000 7f0eac1cc700 1 Processor -- start 2019-12-05T05:42:29.297 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:29.285+0000 7f0eac1cc700 1 -- start start 2019-12-05T05:42:29.297 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:29.289+0000 7f0eac1cc700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0ea4142a40 0x7f0ea4118500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:42:29.297 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:29.289+0000 7f0eac1cc700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f0ea412ce80 0x7f0ea412d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:42:29.302 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:29.289+0000 7f0eac1cc700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ea4118a50 con 0x7f0ea4130440 2019-12-05T05:42:29.303 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:29.289+0000 7f0eac1cc700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ea4117210 con 0x7f0ea412ce80 2019-12-05T05:42:29.303 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:29.289+0000 7f0eac1cc700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ea41174b0 con 0x7f0ea4142a40 2019-12-05T05:42:29.303 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:29.289+0000 7f0eaa769700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0ea4142a40 0x7f0ea4118500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:42:29.303 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:29.289+0000 7f0eaa769700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0ea4142a40 0x7f0ea4118500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:39198/0 (socket says 172.21.15.42:39198) 2019-12-05T05:42:29.303 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:29.289+0000 7f0eaa769700 1 -- 172.21.15.42:0/3125731699 learned_addr learned my addr 172.21.15.42:0/3125731699 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:42:29.303 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:29.289+0000 7f0ea9f68700 1 --2- 172.21.15.42:0/3125731699 >> v2:172.21.15.40:3300/0 conn(0x7f0ea412ce80 0x7f0ea412d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:42:29.304 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:29.289+0000 7f0eaa769700 1 -- 172.21.15.42:0/3125731699 >> v2:172.21.15.40:3300/0 conn(0x7f0ea412ce80 msgr2=0x7f0ea412d2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:42:29.304 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:29.289+0000 7f0eaa769700 1 --2- 172.21.15.42:0/3125731699 >> v2:172.21.15.40:3300/0 conn(0x7f0ea412ce80 0x7f0ea412d2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:42:29.304 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:29.289+0000 7f0eaa769700 1 -- 172.21.15.42:0/3125731699 >> v1:172.21.15.40:6789/0 conn(0x7f0ea4130440 legacy=0x7f0ea4130870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:42:29.304 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:29.289+0000 7f0eaa769700 1 -- 172.21.15.42:0/3125731699 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ea4118e10 con 0x7f0ea4142a40 2019-12-05T05:42:29.304 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:29.289+0000 7f0eaa769700 1 --2- 172.21.15.42:0/3125731699 >> v2:172.21.15.42:3300/0 conn(0x7f0ea4142a40 0x7f0ea4118500 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f0ea409f2e0 tx=0x7f0ea0008be0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:29.304 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:29.289+0000 7f0eaa769700 1 --2- 172.21.15.42:0/3125731699 >> v2:172.21.15.42:3300/0 conn(0x7f0ea4142a40 0x7f0ea4118500 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f0ea409f2e0 tx=0x7f0ea0008be0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:29.304 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:29.289+0000 7f0eaa769700 1 --2- 172.21.15.42:0/3125731699 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0ea4142a40 0x7f0ea4118500 secure :-1 s=READY pgs=1285 cs=0 l=1 rx=0x7f0ea409f2e0 tx=0x7f0ea0008be0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:42:29.305 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:29.289+0000 7f0eaa769700 1 --2- 172.21.15.42:0/3125731699 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0ea4142a40 0x7f0ea4118500 secure :-1 s=THROTTLE_DONE pgs=1285 cs=0 l=1 rx=0x7f0ea409f2e0 tx=0x7f0ea0008be0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:29.305 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:29.289+0000 7f0eaa769700 1 --2- 172.21.15.42:0/3125731699 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0ea4142a40 0x7f0ea4118500 secure :-1 s=THROTTLE_DONE pgs=1285 cs=0 l=1 rx=0x7f0ea409f2e0 tx=0x7f0ea0008be0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:29.305 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:29.289+0000 7f0eaa769700 1 --2- 172.21.15.42:0/3125731699 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0ea4142a40 0x7f0ea4118500 secure :-1 s=THROTTLE_DONE pgs=1285 cs=0 l=1 rx=0x7f0ea409f2e0 tx=0x7f0ea0008be0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:29.305 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:29.289+0000 7f0e96ffd700 1 -- 172.21.15.42:0/3125731699 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0ea000f680 con 0x7f0ea4142a40 2019-12-05T05:42:29.305 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:29.289+0000 7f0e96ffd700 1 -- 172.21.15.42:0/3125731699 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ea0011730 con 0x7f0ea4142a40 2019-12-05T05:42:29.305 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:29.289+0000 7f0eac1cc700 1 -- 172.21.15.42:0/3125731699 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ea4117c90 con 0x7f0ea4142a40 2019-12-05T05:42:29.305 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:29.289+0000 7f0e96ffd700 1 -- 172.21.15.42:0/3125731699 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0ea0005920 con 0x7f0ea4142a40 2019-12-05T05:42:29.306 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:29.289+0000 7f0eac1cc700 1 -- 172.21.15.42:0/3125731699 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ea4142e70 con 0x7f0ea4142a40 2019-12-05T05:42:29.306 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:29.289+0000 7f0eaa769700 1 --2- 172.21.15.42:0/3125731699 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0ea4142a40 0x7f0ea4118500 secure :-1 s=THROTTLE_DONE pgs=1285 cs=0 l=1 rx=0x7f0ea409f2e0 tx=0x7f0ea0008be0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:29.306 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:29.293+0000 7f0e96ffd700 1 -- 172.21.15.42:0/3125731699 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f0ea0005c30 con 0x7f0ea4142a40 2019-12-05T05:42:29.306 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:29.293+0000 7f0eaa769700 1 --2- 172.21.15.42:0/3125731699 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0ea4142a40 0x7f0ea4118500 secure :-1 s=THROTTLE_DONE pgs=1285 cs=0 l=1 rx=0x7f0ea409f2e0 tx=0x7f0ea0008be0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:29.306 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:29.293+0000 7f0e96ffd700 1 --2- 172.21.15.42:0/3125731699 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0e8c03ae60 0x7f0e8c03d370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:42:29.306 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:29.293+0000 7f0e96ffd700 1 -- 172.21.15.42:0/3125731699 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 8181+0+0 (secure 0 0 0) 0x7f0ea004b750 con 0x7f0ea4142a40 2019-12-05T05:42:29.306 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:29.293+0000 7f0ea9f68700 1 --2- 172.21.15.42:0/3125731699 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0e8c03ae60 0x7f0e8c03d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:42:29.307 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:29.293+0000 7f0ea9f68700 1 --2- 172.21.15.42:0/3125731699 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0e8c03ae60 0x7f0e8c03d370 crc :-1 s=READY pgs=556 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:42:29.307 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:29.293+0000 7f0eac1cc700 1 -- 172.21.15.42:0/3125731699 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ea405d110 con 0x7f0ea4142a40 2019-12-05T05:42:29.307 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:29.301+0000 7f0eaa769700 1 --2- 172.21.15.42:0/3125731699 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0ea4142a40 0x7f0ea4118500 secure :-1 s=THROTTLE_DONE pgs=1285 cs=0 l=1 rx=0x7f0ea409f2e0 tx=0x7f0ea0008be0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:29.307 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:29.301+0000 7f0e96ffd700 1 -- 172.21.15.42:0/3125731699 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f0ea0030420 con 0x7f0ea4142a40 2019-12-05T05:42:29.474 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:29.469+0000 7f0eac1cc700 1 -- 172.21.15.42:0/3125731699 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0ea40aadf0 con 0x7f0ea4142a40 2019-12-05T05:42:29.475 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:29.469+0000 7f0eaa769700 1 --2- 172.21.15.42:0/3125731699 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0ea4142a40 0x7f0ea4118500 secure :-1 s=THROTTLE_DONE pgs=1285 cs=0 l=1 rx=0x7f0ea409f2e0 tx=0x7f0ea0008be0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:29.475 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:29.469+0000 7f0e96ffd700 1 -- 172.21.15.42:0/3125731699 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 158 v158) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f0ea0013d70 con 0x7f0ea4142a40 2019-12-05T05:42:29.475 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 158 2019-12-05T05:42:29.476 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:42:29.476 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":158,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7650,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/1029013337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1029013337},{"type":"v1","addr":"172.21.15.40:6833","nonce":1029013337}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":158}],"filesystems":[{"mdsmap":{"epoch":157,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:41:03.431537+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7539},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7539":{"gid":7539,"name":"a","rank":0,"incarnation":154,"state":"up:active","state_seq":30,"addr":"172.21.15.42:6835/1233824070","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1233824070},{"type":"v1","addr":"172.21.15.42:6835","nonce":1233824070}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:42:29.479 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:29.473+0000 7f0eac1cc700 1 -- 172.21.15.42:0/3125731699 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0e8c03ae60 msgr2=0x7f0e8c03d370 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:42:29.479 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:29.473+0000 7f0eac1cc700 1 --2- 172.21.15.42:0/3125731699 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0e8c03ae60 0x7f0e8c03d370 crc :-1 s=READY pgs=556 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:42:29.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:29.473+0000 7f0eac1cc700 1 -- 172.21.15.42:0/3125731699 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0ea4142a40 msgr2=0x7f0ea4118500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:42:29.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:29.473+0000 7f0eac1cc700 1 --2- 172.21.15.42:0/3125731699 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0ea4142a40 0x7f0ea4118500 secure :-1 s=READY pgs=1285 cs=0 l=1 rx=0x7f0ea409f2e0 tx=0x7f0ea0008be0).stop 2019-12-05T05:42:29.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:29.473+0000 7f0eac1cc700 1 -- 172.21.15.42:0/3125731699 shutdown_connections 2019-12-05T05:42:29.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:29.473+0000 7f0eac1cc700 1 --2- 172.21.15.42:0/3125731699 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0e8c03ae60 0x7f0e8c03d370 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:42:29.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:29.473+0000 7f0eac1cc700 1 --2- 172.21.15.42:0/3125731699 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0ea4142a40 0x7f0ea4118500 unknown :-1 s=CLOSED pgs=1285 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:42:29.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:29.473+0000 7f0eac1cc700 1 --2- 172.21.15.42:0/3125731699 >> v2:172.21.15.40:3300/0 conn(0x7f0ea412ce80 0x7f0ea412d2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:42:29.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:29.477+0000 7f0eac1cc700 1 -- 172.21.15.42:0/3125731699 shutdown_connections 2019-12-05T05:42:29.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:29.477+0000 7f0eac1cc700 1 -- 172.21.15.42:0/3125731699 wait complete. 2019-12-05T05:42:29.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:29.477+0000 7f0eac1cc700 1 -- 172.21.15.42:0/3125731699 >> 172.21.15.42:0/3125731699 conn(0x7f0ea40a82a0 msgr2=0x7f0ea409f4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:42:29.495 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T05:42:31.496 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:42:31.607 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:31.597+0000 7f437f783700 1 Processor -- start 2019-12-05T05:42:31.607 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:31.597+0000 7f437f783700 1 -- start start 2019-12-05T05:42:31.607 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:31.597+0000 7f437f783700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f437808df70 0x7f437809f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:42:31.607 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:31.597+0000 7f437f783700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f437809f820 0x7f437814efd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:42:31.607 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:31.597+0000 7f437f783700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f437809fc50 con 0x7f43780a7730 2019-12-05T05:42:31.608 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:31.597+0000 7f437f783700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43780a0af0 con 0x7f437809f820 2019-12-05T05:42:31.608 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:31.597+0000 7f437f783700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f437808e5a0 con 0x7f437808df70 2019-12-05T05:42:31.608 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:31.597+0000 7f437e521700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f437809f820 0x7f437814efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:42:31.608 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:31.597+0000 7f437e521700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f437809f820 0x7f437814efd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:39228/0 (socket says 172.21.15.42:39228) 2019-12-05T05:42:31.608 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:31.597+0000 7f437e521700 1 -- 172.21.15.42:0/2789751967 learned_addr learned my addr 172.21.15.42:0/2789751967 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:42:31.608 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:31.597+0000 7f437d51f700 1 --2- 172.21.15.42:0/2789751967 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f437808df70 0x7f437809f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:42:31.608 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:31.601+0000 7f437e521700 1 -- 172.21.15.42:0/2789751967 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f437808df70 msgr2=0x7f437809f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:42:31.609 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:31.601+0000 7f437e521700 1 --2- 172.21.15.42:0/2789751967 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f437808df70 0x7f437809f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:42:31.609 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:31.601+0000 7f437e521700 1 -- 172.21.15.42:0/2789751967 >> v1:172.21.15.42:6789/0 conn(0x7f43780a7730 legacy=0x7f437807e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:42:31.609 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:31.601+0000 7f437e521700 1 -- 172.21.15.42:0/2789751967 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43780a0700 con 0x7f437809f820 2019-12-05T05:42:31.609 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:31.601+0000 7f437e521700 1 --2- 172.21.15.42:0/2789751967 >> v2:172.21.15.42:3300/0 conn(0x7f437809f820 0x7f437814efd0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f4370008b10 tx=0x7f43700100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:31.609 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:31.601+0000 7f437e521700 1 --2- 172.21.15.42:0/2789751967 >> v2:172.21.15.42:3300/0 conn(0x7f437809f820 0x7f437814efd0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f4370008b10 tx=0x7f43700100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:31.610 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:31.601+0000 7f437e521700 1 --2- 172.21.15.42:0/2789751967 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f437809f820 0x7f437814efd0 secure :-1 s=READY pgs=1287 cs=0 l=1 rx=0x7f4370008b10 tx=0x7f43700100b0).ready entity=mon.1 client_cookie=94b873f760ab9023 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:42:31.610 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:31.601+0000 7f437e521700 1 --2- 172.21.15.42:0/2789751967 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f437809f820 0x7f437814efd0 secure :-1 s=THROTTLE_DONE pgs=1287 cs=0 l=1 rx=0x7f4370008b10 tx=0x7f43700100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:31.610 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:31.601+0000 7f437e521700 1 --2- 172.21.15.42:0/2789751967 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f437809f820 0x7f437814efd0 secure :-1 s=THROTTLE_DONE pgs=1287 cs=0 l=1 rx=0x7f4370008b10 tx=0x7f43700100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:31.610 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:31.601+0000 7f437cd1e700 1 -- 172.21.15.42:0/2789751967 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4370010da0 con 0x7f437809f820 2019-12-05T05:42:31.610 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:31.601+0000 7f437e521700 1 --2- 172.21.15.42:0/2789751967 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f437809f820 0x7f437814efd0 secure :-1 s=THROTTLE_DONE pgs=1287 cs=0 l=1 rx=0x7f4370008b10 tx=0x7f43700100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:31.611 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:31.601+0000 7f437cd1e700 1 -- 172.21.15.42:0/2789751967 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43700109c0 con 0x7f437809f820 2019-12-05T05:42:31.611 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:31.601+0000 7f437cd1e700 1 -- 172.21.15.42:0/2789751967 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f43700114a0 con 0x7f437809f820 2019-12-05T05:42:31.611 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:31.601+0000 7f437f783700 1 -- 172.21.15.42:0/2789751967 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f437809f820 msgr2=0x7f437814efd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:42:31.611 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:31.601+0000 7f437f783700 1 --2- 172.21.15.42:0/2789751967 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f437809f820 0x7f437814efd0 secure :-1 s=READY pgs=1287 cs=0 l=1 rx=0x7f4370008b10 tx=0x7f43700100b0).stop 2019-12-05T05:42:31.611 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:31.601+0000 7f437f783700 1 -- 172.21.15.42:0/2789751967 shutdown_connections 2019-12-05T05:42:31.611 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:31.601+0000 7f437f783700 1 --2- 172.21.15.42:0/2789751967 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f437809f820 0x7f437814efd0 unknown :-1 s=CLOSED pgs=1287 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:42:31.612 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:31.601+0000 7f437f783700 1 --2- 172.21.15.42:0/2789751967 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f437808df70 0x7f437809f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:42:31.612 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:31.601+0000 7f437f783700 1 -- 172.21.15.42:0/2789751967 shutdown_connections 2019-12-05T05:42:31.612 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:31.601+0000 7f437f783700 1 -- 172.21.15.42:0/2789751967 wait complete. 2019-12-05T05:42:31.612 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:31.601+0000 7f437f783700 1 -- 172.21.15.42:0/2789751967 >> 172.21.15.42:0/2789751967 conn(0x7f43780aabd0 msgr2=0x7f43780a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:42:31.615 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:31.605+0000 7f437f783700 1 Processor -- start 2019-12-05T05:42:31.615 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:31.605+0000 7f437f783700 1 -- start start 2019-12-05T05:42:31.615 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:31.605+0000 7f437f783700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f43780a4030 0x7f43780a2970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:42:31.615 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:31.605+0000 7f437f783700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43780a2ec0 con 0x7f43780a1840 2019-12-05T05:42:31.616 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:31.605+0000 7f437f783700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4378127a90 con 0x7f437805a200 2019-12-05T05:42:31.616 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:31.605+0000 7f437f783700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43780a15a0 con 0x7f43780a4030 2019-12-05T05:42:31.617 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:31.605+0000 7f437d51f700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f43780a4030 0x7f43780a2970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:42:31.617 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:31.605+0000 7f437d51f700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f43780a4030 0x7f43780a2970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:39232/0 (socket says 172.21.15.42:39232) 2019-12-05T05:42:31.617 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:31.605+0000 7f437d51f700 1 -- 172.21.15.42:0/1987395973 learned_addr learned my addr 172.21.15.42:0/1987395973 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:42:31.617 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:31.605+0000 7f437dd20700 1 --1- 172.21.15.42:0/1987395973 >> v1:172.21.15.42:6789/0 conn(0x7f437805a200 0x7f43781282d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:39268/0 (socket says 172.21.15.42:39268) 2019-12-05T05:42:31.617 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:31.605+0000 7f437d51f700 1 -- 172.21.15.42:0/1987395973 >> v1:172.21.15.42:6789/0 conn(0x7f437805a200 legacy=0x7f43781282d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:42:31.617 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:31.605+0000 7f437d51f700 1 -- 172.21.15.42:0/1987395973 >> v1:172.21.15.40:6789/0 conn(0x7f43780a1840 legacy=0x7f437812eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:42:31.618 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:31.605+0000 7f436e7fc700 1 -- 172.21.15.42:0/1987395973 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 383998041 0 0) 0x7f4378127a90 con 0x7f437805a200 2019-12-05T05:42:31.618 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:31.605+0000 7f437d51f700 1 -- 172.21.15.42:0/1987395973 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4378115720 con 0x7f43780a4030 2019-12-05T05:42:31.618 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:31.605+0000 7f437d51f700 1 --2- 172.21.15.42:0/1987395973 >> v2:172.21.15.42:3300/0 conn(0x7f43780a4030 0x7f43780a2970 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f4368003070 tx=0x7f4368010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:31.618 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:31.605+0000 7f437d51f700 1 --2- 172.21.15.42:0/1987395973 >> v2:172.21.15.42:3300/0 conn(0x7f43780a4030 0x7f43780a2970 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f4368003070 tx=0x7f4368010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:31.619 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:31.605+0000 7f437d51f700 1 --2- 172.21.15.42:0/1987395973 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f43780a4030 0x7f43780a2970 secure :-1 s=READY pgs=1289 cs=0 l=1 rx=0x7f4368003070 tx=0x7f4368010b40).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:42:31.619 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:31.605+0000 7f437d51f700 1 --2- 172.21.15.42:0/1987395973 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f43780a4030 0x7f43780a2970 secure :-1 s=THROTTLE_DONE pgs=1289 cs=0 l=1 rx=0x7f4368003070 tx=0x7f4368010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:31.619 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:31.609+0000 7f436e7fc700 1 -- 172.21.15.42:0/1987395973 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4368007350 con 0x7f43780a4030 2019-12-05T05:42:31.619 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:31.609+0000 7f437d51f700 1 --2- 172.21.15.42:0/1987395973 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f43780a4030 0x7f43780a2970 secure :-1 s=THROTTLE_DONE pgs=1289 cs=0 l=1 rx=0x7f4368003070 tx=0x7f4368010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:31.619 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:31.609+0000 7f436e7fc700 1 -- 172.21.15.42:0/1987395973 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4368013060 con 0x7f43780a4030 2019-12-05T05:42:31.619 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:31.609+0000 7f437d51f700 1 --2- 172.21.15.42:0/1987395973 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f43780a4030 0x7f43780a2970 secure :-1 s=THROTTLE_DONE pgs=1289 cs=0 l=1 rx=0x7f4368003070 tx=0x7f4368010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:31.619 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:31.609+0000 7f437f783700 1 -- 172.21.15.42:0/1987395973 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43780a1c70 con 0x7f43780a4030 2019-12-05T05:42:31.620 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:31.609+0000 7f436e7fc700 1 -- 172.21.15.42:0/1987395973 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4368005d00 con 0x7f43780a4030 2019-12-05T05:42:31.620 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:31.609+0000 7f437f783700 1 -- 172.21.15.42:0/1987395973 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43781306f0 con 0x7f43780a4030 2019-12-05T05:42:31.620 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:31.609+0000 7f437d51f700 1 --2- 172.21.15.42:0/1987395973 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f43780a4030 0x7f43780a2970 secure :-1 s=THROTTLE_DONE pgs=1289 cs=0 l=1 rx=0x7f4368003070 tx=0x7f4368010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:31.620 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:31.609+0000 7f436e7fc700 1 -- 172.21.15.42:0/1987395973 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f43680121c0 con 0x7f43780a4030 2019-12-05T05:42:31.620 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:31.609+0000 7f437d51f700 1 --2- 172.21.15.42:0/1987395973 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f43780a4030 0x7f43780a2970 secure :-1 s=THROTTLE_DONE pgs=1289 cs=0 l=1 rx=0x7f4368003070 tx=0x7f4368010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:31.620 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:31.609+0000 7f436e7fc700 1 --2- 172.21.15.42:0/1987395973 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f436003abf0 0x7f436003d100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:42:31.620 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:31.609+0000 7f436e7fc700 1 -- 172.21.15.42:0/1987395973 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 8181+0+0 (secure 0 0 0) 0x7f436804af90 con 0x7f43780a4030 2019-12-05T05:42:31.621 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:31.609+0000 7f437dd20700 1 --2- 172.21.15.42:0/1987395973 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f436003abf0 0x7f436003d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:42:31.621 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:31.609+0000 7f437dd20700 1 --2- 172.21.15.42:0/1987395973 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f436003abf0 0x7f436003d100 crc :-1 s=READY pgs=557 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:42:31.621 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:31.613+0000 7f437f783700 1 -- 172.21.15.42:0/1987395973 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f437805d100 con 0x7f43780a4030 2019-12-05T05:42:31.622 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:31.617+0000 7f437d51f700 1 --2- 172.21.15.42:0/1987395973 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f43780a4030 0x7f43780a2970 secure :-1 s=THROTTLE_DONE pgs=1289 cs=0 l=1 rx=0x7f4368003070 tx=0x7f4368010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:31.622 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:31.617+0000 7f436e7fc700 1 -- 172.21.15.42:0/1987395973 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f4368031f90 con 0x7f43780a4030 2019-12-05T05:42:31.789 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:31.785+0000 7f437f783700 1 -- 172.21.15.42:0/1987395973 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f437805b8b0 con 0x7f43780a4030 2019-12-05T05:42:31.789 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:31.785+0000 7f437d51f700 1 --2- 172.21.15.42:0/1987395973 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f43780a4030 0x7f43780a2970 secure :-1 s=THROTTLE_DONE pgs=1289 cs=0 l=1 rx=0x7f4368003070 tx=0x7f4368010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:31.790 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:31.785+0000 7f436e7fc700 1 -- 172.21.15.42:0/1987395973 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 158 v158) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f43680321f0 con 0x7f43780a4030 2019-12-05T05:42:31.790 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 158 2019-12-05T05:42:31.790 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:42:31.790 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":158,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7650,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/1029013337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1029013337},{"type":"v1","addr":"172.21.15.40:6833","nonce":1029013337}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":158}],"filesystems":[{"mdsmap":{"epoch":157,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:41:03.431537+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7539},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7539":{"gid":7539,"name":"a","rank":0,"incarnation":154,"state":"up:active","state_seq":30,"addr":"172.21.15.42:6835/1233824070","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1233824070},{"type":"v1","addr":"172.21.15.42:6835","nonce":1233824070}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:42:31.791 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:31.785+0000 7f437f783700 1 -- 172.21.15.42:0/1987395973 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f436003abf0 msgr2=0x7f436003d100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:42:31.791 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:31.785+0000 7f437f783700 1 --2- 172.21.15.42:0/1987395973 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f436003abf0 0x7f436003d100 crc :-1 s=READY pgs=557 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:42:31.792 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:31.785+0000 7f437f783700 1 -- 172.21.15.42:0/1987395973 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f43780a4030 msgr2=0x7f43780a2970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:42:31.792 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:31.785+0000 7f437f783700 1 --2- 172.21.15.42:0/1987395973 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f43780a4030 0x7f43780a2970 secure :-1 s=READY pgs=1289 cs=0 l=1 rx=0x7f4368003070 tx=0x7f4368010b40).stop 2019-12-05T05:42:31.792 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:31.785+0000 7f437f783700 1 -- 172.21.15.42:0/1987395973 shutdown_connections 2019-12-05T05:42:31.793 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:31.785+0000 7f437f783700 1 --2- 172.21.15.42:0/1987395973 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f436003abf0 0x7f436003d100 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:42:31.793 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:31.785+0000 7f437f783700 1 --2- 172.21.15.42:0/1987395973 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f43780a4030 0x7f43780a2970 unknown :-1 s=CLOSED pgs=1289 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:42:31.793 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:31.789+0000 7f437f783700 1 -- 172.21.15.42:0/1987395973 shutdown_connections 2019-12-05T05:42:31.793 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:31.789+0000 7f437f783700 1 -- 172.21.15.42:0/1987395973 wait complete. 2019-12-05T05:42:31.794 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:31.789+0000 7f437f783700 1 -- 172.21.15.42:0/1987395973 >> 172.21.15.42:0/1987395973 conn(0x7f43780a6de0 msgr2=0x7f4378115e70 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:42:31.806 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T05:42:33.808 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:42:33.915 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:33.901+0000 7feea6c4d700 1 Processor -- start 2019-12-05T05:42:33.915 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:33.901+0000 7feea6c4d700 1 -- start start 2019-12-05T05:42:33.915 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:33.905+0000 7feea6c4d700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7feea008df80 0x7feea009f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:42:33.915 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:33.905+0000 7feea6c4d700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feea009fc60 con 0x7feea009f830 2019-12-05T05:42:33.915 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:33.905+0000 7feea6c4d700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feea00a0710 con 0x7feea00a7740 2019-12-05T05:42:33.916 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:33.905+0000 7feea6c4d700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feea00a0b00 con 0x7feea008df80 2019-12-05T05:42:33.916 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:33.905+0000 7feea51ea700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7feea00a7740 0x7feea007e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:39282/0 (socket says 172.21.15.42:39282) 2019-12-05T05:42:33.916 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:33.905+0000 7feea51ea700 1 -- 172.21.15.42:0/1842201948 learned_addr learned my addr 172.21.15.42:0/1842201948 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:42:33.916 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:33.905+0000 7feea49e9700 1 --2- 172.21.15.42:0/1842201948 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7feea008df80 0x7feea009f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:42:33.916 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:33.905+0000 7fee97fff700 1 -- 172.21.15.42:0/1842201948 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3196209021 0 0) 0x7feea00a0710 con 0x7feea00a7740 2019-12-05T05:42:33.916 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:33.905+0000 7fee97fff700 1 -- 172.21.15.42:0/1842201948 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee8c0030b0 con 0x7feea00a7740 2019-12-05T05:42:33.916 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:33.905+0000 7fee97fff700 1 -- 172.21.15.42:0/1842201948 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3575856989 0 0) 0x7feea009fc60 con 0x7feea009f830 2019-12-05T05:42:33.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:33.905+0000 7fee97fff700 1 -- 172.21.15.42:0/1842201948 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feea00a0710 con 0x7feea009f830 2019-12-05T05:42:33.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:33.905+0000 7fee97fff700 1 -- 172.21.15.42:0/1842201948 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1208361181 0 0) 0x7fee8c0030b0 con 0x7feea00a7740 2019-12-05T05:42:33.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:33.905+0000 7fee97fff700 1 -- 172.21.15.42:0/1842201948 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7feea008df80 msgr2=0x7feea009f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:42:33.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:33.905+0000 7fee97fff700 1 --2- 172.21.15.42:0/1842201948 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7feea008df80 0x7feea009f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:42:33.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:33.905+0000 7fee97fff700 1 -- 172.21.15.42:0/1842201948 >> v1:172.21.15.40:6789/0 conn(0x7feea009f830 legacy=0x7feea014efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:42:33.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:33.905+0000 7fee97fff700 1 -- 172.21.15.42:0/1842201948 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feea007ef20 con 0x7feea00a7740 2019-12-05T05:42:33.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:33.905+0000 7fee97fff700 1 -- 172.21.15.42:0/1842201948 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7fee9c001750 con 0x7feea00a7740 2019-12-05T05:42:33.918 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:33.905+0000 7feea49e9700 1 --2- 172.21.15.42:0/1842201948 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7feea008df80 0x7feea009f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:42:33.918 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:33.905+0000 7fee97fff700 1 -- 172.21.15.42:0/1842201948 >> v1:172.21.15.42:6789/0 conn(0x7feea00a7740 legacy=0x7feea007e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:42:33.918 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:33.905+0000 7fee97fff700 1 --2- 172.21.15.42:0/1842201948 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fee8c003940 0x7fee8c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:42:33.918 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:33.905+0000 7fee97fff700 1 --2- 172.21.15.42:0/1842201948 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fee8c004470 0x7fee8c0048c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:42:33.918 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:33.905+0000 7fee97fff700 1 --2- 172.21.15.42:0/1842201948 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fee8c004e80 0x7fee8c04f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:42:33.918 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:33.905+0000 7fee97fff700 1 -- 172.21.15.42:0/1842201948 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feea00a0b00 con 0x7fee8c004e80 2019-12-05T05:42:33.918 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:33.905+0000 7fee97fff700 1 -- 172.21.15.42:0/1842201948 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee8c04fc90 con 0x7fee8c003940 2019-12-05T05:42:33.919 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:33.905+0000 7fee97fff700 1 -- 172.21.15.42:0/1842201948 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee8c04fed0 con 0x7fee8c004470 2019-12-05T05:42:33.919 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:33.905+0000 7feea51ea700 1 --2- 172.21.15.42:0/1842201948 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fee8c003940 0x7fee8c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:42:33.919 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:33.905+0000 7feea49e9700 1 --2- 172.21.15.42:0/1842201948 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fee8c004470 0x7fee8c0048c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:42:33.919 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:33.905+0000 7feea59eb700 1 --2- 172.21.15.42:0/1842201948 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fee8c004e80 0x7fee8c04f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:42:33.919 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:33.905+0000 7feea51ea700 1 -- 172.21.15.42:0/1842201948 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fee8c004470 msgr2=0x7fee8c0048c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:42:33.919 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:33.905+0000 7feea51ea700 1 --2- 172.21.15.42:0/1842201948 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fee8c004470 0x7fee8c0048c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:42:33.919 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:33.905+0000 7feea51ea700 1 -- 172.21.15.42:0/1842201948 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fee8c004e80 msgr2=0x7fee8c04f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:42:33.919 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:33.905+0000 7feea51ea700 1 --2- 172.21.15.42:0/1842201948 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fee8c004e80 0x7fee8c04f680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:42:33.920 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:33.905+0000 7feea51ea700 1 -- 172.21.15.42:0/1842201948 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee8c0501d0 con 0x7fee8c003940 2019-12-05T05:42:33.920 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:33.905+0000 7feea59eb700 1 --2- 172.21.15.42:0/1842201948 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fee8c004e80 0x7fee8c04f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T05:42:33.920 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:33.905+0000 7feea51ea700 1 --2- 172.21.15.42:0/1842201948 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fee8c003940 0x7fee8c003d90 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fee9c001520 tx=0x7fee9c00c090).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:33.920 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:33.905+0000 7feea51ea700 1 --2- 172.21.15.42:0/1842201948 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fee8c003940 0x7fee8c003d90 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fee9c001520 tx=0x7fee9c00c090).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:33.920 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:33.905+0000 7feea51ea700 1 --2- 172.21.15.42:0/1842201948 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fee8c003940 0x7fee8c003d90 secure :-1 s=READY pgs=1291 cs=0 l=1 rx=0x7fee9c001520 tx=0x7fee9c00c090).ready entity=mon.1 client_cookie=5bdbda2e8e3441c3 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:42:33.920 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:33.909+0000 7feea51ea700 1 --2- 172.21.15.42:0/1842201948 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fee8c003940 0x7fee8c003d90 secure :-1 s=THROTTLE_DONE pgs=1291 cs=0 l=1 rx=0x7fee9c001520 tx=0x7fee9c00c090).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:33.920 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:33.909+0000 7fee97fff700 1 -- 172.21.15.42:0/1842201948 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fee9c00ead0 con 0x7fee8c003940 2019-12-05T05:42:33.921 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:33.909+0000 7feea51ea700 1 --2- 172.21.15.42:0/1842201948 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fee8c003940 0x7fee8c003d90 secure :-1 s=THROTTLE_DONE pgs=1291 cs=0 l=1 rx=0x7fee9c001520 tx=0x7fee9c00c090).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:33.921 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:33.909+0000 7feea51ea700 1 --2- 172.21.15.42:0/1842201948 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fee8c003940 0x7fee8c003d90 secure :-1 s=THROTTLE_DONE pgs=1291 cs=0 l=1 rx=0x7fee9c001520 tx=0x7fee9c00c090).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:33.921 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:33.909+0000 7fee97fff700 1 -- 172.21.15.42:0/1842201948 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee9c010bc0 con 0x7fee8c003940 2019-12-05T05:42:33.921 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:33.909+0000 7fee97fff700 1 -- 172.21.15.42:0/1842201948 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fee9c010390 con 0x7fee8c003940 2019-12-05T05:42:33.921 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:33.909+0000 7feea6c4d700 1 -- 172.21.15.42:0/1842201948 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fee8c003940 msgr2=0x7fee8c003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:42:33.922 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:33.909+0000 7feea6c4d700 1 --2- 172.21.15.42:0/1842201948 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fee8c003940 0x7fee8c003d90 secure :-1 s=READY pgs=1291 cs=0 l=1 rx=0x7fee9c001520 tx=0x7fee9c00c090).stop 2019-12-05T05:42:33.923 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:33.909+0000 7feea59eb700 1 -- 172.21.15.42:0/1842201948 reap_dead start 2019-12-05T05:42:33.923 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:33.909+0000 7feea6c4d700 1 -- 172.21.15.42:0/1842201948 shutdown_connections 2019-12-05T05:42:33.924 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:33.909+0000 7feea6c4d700 1 -- 172.21.15.42:0/1842201948 shutdown_connections 2019-12-05T05:42:33.924 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:33.909+0000 7feea6c4d700 1 -- 172.21.15.42:0/1842201948 wait complete. 2019-12-05T05:42:33.924 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:33.909+0000 7feea6c4d700 1 -- 172.21.15.42:0/1842201948 >> 172.21.15.42:0/1842201948 conn(0x7feea00aabe0 msgr2=0x7feea00a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:42:33.930 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:33.909+0000 7feea6c4d700 1 Processor -- start 2019-12-05T05:42:33.930 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:33.913+0000 7feea6c4d700 1 -- start start 2019-12-05T05:42:33.930 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:33.913+0000 7feea6c4d700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7feea008eeb0 0x7feea00a35f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:42:33.930 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:33.913+0000 7feea6c4d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7feea00a3b40 0x7feea0130300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:42:33.931 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:33.913+0000 7feea6c4d700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7feea00a24b0 0x7feea0130850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:42:33.931 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:33.913+0000 7feea6c4d700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feea00a1f70 con 0x7feea00a24b0 2019-12-05T05:42:33.931 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:33.913+0000 7feea6c4d700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feea00a2210 con 0x7feea00a3b40 2019-12-05T05:42:33.931 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:33.913+0000 7feea6c4d700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feea00a1990 con 0x7feea008eeb0 2019-12-05T05:42:33.931 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:33.913+0000 7feea49e9700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7feea00a3b40 0x7feea0130300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:42:33.931 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:33.913+0000 7feea49e9700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7feea00a3b40 0x7feea0130300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:39256/0 (socket says 172.21.15.42:39256) 2019-12-05T05:42:33.932 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:33.913+0000 7feea49e9700 1 -- 172.21.15.42:0/2052236750 learned_addr learned my addr 172.21.15.42:0/2052236750 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:42:33.932 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:33.913+0000 7feea51ea700 1 --2- 172.21.15.42:0/2052236750 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7feea008eeb0 0x7feea00a35f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:42:33.932 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:33.913+0000 7feea59eb700 1 --2- 172.21.15.42:0/2052236750 >> v2:172.21.15.40:3300/0 conn(0x7feea00a24b0 0x7feea0130850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:42:33.932 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:33.913+0000 7feea49e9700 1 -- 172.21.15.42:0/2052236750 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7feea008eeb0 msgr2=0x7feea00a35f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:42:33.932 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:33.913+0000 7feea49e9700 1 --2- 172.21.15.42:0/2052236750 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7feea008eeb0 0x7feea00a35f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:42:33.932 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:33.913+0000 7feea49e9700 1 -- 172.21.15.42:0/2052236750 >> v2:172.21.15.40:3300/0 conn(0x7feea00a24b0 msgr2=0x7feea0130850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:42:33.933 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:33.913+0000 7feea49e9700 1 --2- 172.21.15.42:0/2052236750 >> v2:172.21.15.40:3300/0 conn(0x7feea00a24b0 0x7feea0130850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:42:33.933 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:33.913+0000 7feea49e9700 1 -- 172.21.15.42:0/2052236750 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feea0148370 con 0x7feea00a3b40 2019-12-05T05:42:33.933 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:33.913+0000 7feea59eb700 1 --2- 172.21.15.42:0/2052236750 >> v2:172.21.15.40:3300/0 conn(0x7feea00a24b0 0x7feea0130850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T05:42:33.933 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:33.913+0000 7feea51ea700 1 --2- 172.21.15.42:0/2052236750 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7feea008eeb0 0x7feea00a35f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T05:42:33.933 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:33.913+0000 7feea49e9700 1 --2- 172.21.15.42:0/2052236750 >> v2:172.21.15.42:3300/0 conn(0x7feea00a3b40 0x7feea0130300 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fee90002f30 tx=0x7fee90010bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:33.933 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:33.913+0000 7feea49e9700 1 --2- 172.21.15.42:0/2052236750 >> v2:172.21.15.42:3300/0 conn(0x7feea00a3b40 0x7feea0130300 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fee90002f30 tx=0x7fee90010bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:33.933 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:33.913+0000 7feea49e9700 1 --2- 172.21.15.42:0/2052236750 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feea00a3b40 0x7feea0130300 secure :-1 s=READY pgs=1292 cs=0 l=1 rx=0x7fee90002f30 tx=0x7fee90010bc0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:42:33.934 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:33.913+0000 7feea49e9700 1 --2- 172.21.15.42:0/2052236750 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feea00a3b40 0x7feea0130300 secure :-1 s=THROTTLE_DONE pgs=1292 cs=0 l=1 rx=0x7fee90002f30 tx=0x7fee90010bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:33.934 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:33.913+0000 7fee95ffb700 1 -- 172.21.15.42:0/2052236750 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fee90012b70 con 0x7feea00a3b40 2019-12-05T05:42:33.934 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:33.913+0000 7feea49e9700 1 --2- 172.21.15.42:0/2052236750 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feea00a3b40 0x7feea0130300 secure :-1 s=THROTTLE_DONE pgs=1292 cs=0 l=1 rx=0x7fee90002f30 tx=0x7fee90010bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:33.934 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:33.913+0000 7fee95ffb700 1 -- 172.21.15.42:0/2052236750 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee90012790 con 0x7feea00a3b40 2019-12-05T05:42:33.934 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:33.913+0000 7feea49e9700 1 --2- 172.21.15.42:0/2052236750 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feea00a3b40 0x7feea0130300 secure :-1 s=THROTTLE_DONE pgs=1292 cs=0 l=1 rx=0x7fee90002f30 tx=0x7fee90010bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:33.934 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:33.913+0000 7feea6c4d700 1 -- 172.21.15.42:0/2052236750 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feea00a4300 con 0x7feea00a3b40 2019-12-05T05:42:33.934 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:33.913+0000 7fee95ffb700 1 -- 172.21.15.42:0/2052236750 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fee90005730 con 0x7feea00a3b40 2019-12-05T05:42:33.935 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:33.913+0000 7feea6c4d700 1 -- 172.21.15.42:0/2052236750 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feea00a16b0 con 0x7feea00a3b40 2019-12-05T05:42:33.935 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:33.917+0000 7feea49e9700 1 --2- 172.21.15.42:0/2052236750 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feea00a3b40 0x7feea0130300 secure :-1 s=THROTTLE_DONE pgs=1292 cs=0 l=1 rx=0x7fee90002f30 tx=0x7fee90010bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:33.935 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:33.917+0000 7fee95ffb700 1 -- 172.21.15.42:0/2052236750 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fee90007180 con 0x7feea00a3b40 2019-12-05T05:42:33.935 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:33.917+0000 7feea49e9700 1 --2- 172.21.15.42:0/2052236750 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feea00a3b40 0x7feea0130300 secure :-1 s=THROTTLE_DONE pgs=1292 cs=0 l=1 rx=0x7fee90002f30 tx=0x7fee90010bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:33.935 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:33.917+0000 7fee95ffb700 1 --2- 172.21.15.42:0/2052236750 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fee8003aca0 0x7fee8003d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:42:33.936 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:33.917+0000 7fee95ffb700 1 -- 172.21.15.42:0/2052236750 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 8181+0+0 (secure 0 0 0) 0x7fee900120c0 con 0x7feea00a3b40 2019-12-05T05:42:33.936 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:33.917+0000 7feea51ea700 1 --2- 172.21.15.42:0/2052236750 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fee8003aca0 0x7fee8003d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:42:33.936 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:33.917+0000 7feea51ea700 1 --2- 172.21.15.42:0/2052236750 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fee8003aca0 0x7fee8003d1b0 crc :-1 s=READY pgs=558 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:42:33.936 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:33.917+0000 7feea6c4d700 1 -- 172.21.15.42:0/2052236750 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feea005d110 con 0x7feea00a3b40 2019-12-05T05:42:33.936 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:33.925+0000 7feea49e9700 1 --2- 172.21.15.42:0/2052236750 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feea00a3b40 0x7feea0130300 secure :-1 s=THROTTLE_DONE pgs=1292 cs=0 l=1 rx=0x7fee90002f30 tx=0x7fee90010bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:33.936 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:33.925+0000 7fee95ffb700 1 -- 172.21.15.42:0/2052236750 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fee90032880 con 0x7feea00a3b40 2019-12-05T05:42:34.091 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:34.085+0000 7feea6c4d700 1 -- 172.21.15.42:0/2052236750 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7feea005b8c0 con 0x7feea00a3b40 2019-12-05T05:42:34.092 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:34.089+0000 7feea49e9700 1 --2- 172.21.15.42:0/2052236750 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feea00a3b40 0x7feea0130300 secure :-1 s=THROTTLE_DONE pgs=1292 cs=0 l=1 rx=0x7fee90002f30 tx=0x7fee90010bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:34.092 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:34.089+0000 7fee95ffb700 1 -- 172.21.15.42:0/2052236750 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 158 v158) v1 ==== 95+0+2112 (secure 0 0 0) 0x7fee90032ae0 con 0x7feea00a3b40 2019-12-05T05:42:34.092 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 158 2019-12-05T05:42:34.093 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:42:34.093 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":158,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7650,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/1029013337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1029013337},{"type":"v1","addr":"172.21.15.40:6833","nonce":1029013337}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":158}],"filesystems":[{"mdsmap":{"epoch":157,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:41:03.431537+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7539},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7539":{"gid":7539,"name":"a","rank":0,"incarnation":154,"state":"up:active","state_seq":30,"addr":"172.21.15.42:6835/1233824070","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1233824070},{"type":"v1","addr":"172.21.15.42:6835","nonce":1233824070}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:42:34.095 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:34.089+0000 7feea6c4d700 1 -- 172.21.15.42:0/2052236750 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fee8003aca0 msgr2=0x7fee8003d1b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:42:34.095 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:34.089+0000 7feea6c4d700 1 --2- 172.21.15.42:0/2052236750 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fee8003aca0 0x7fee8003d1b0 crc :-1 s=READY pgs=558 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:42:34.095 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:34.089+0000 7feea6c4d700 1 -- 172.21.15.42:0/2052236750 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feea00a3b40 msgr2=0x7feea0130300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:42:34.095 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:34.089+0000 7feea6c4d700 1 --2- 172.21.15.42:0/2052236750 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feea00a3b40 0x7feea0130300 secure :-1 s=READY pgs=1292 cs=0 l=1 rx=0x7fee90002f30 tx=0x7fee90010bc0).stop 2019-12-05T05:42:34.095 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:34.089+0000 7feea6c4d700 1 -- 172.21.15.42:0/2052236750 shutdown_connections 2019-12-05T05:42:34.096 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:34.089+0000 7feea6c4d700 1 --2- 172.21.15.42:0/2052236750 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fee8003aca0 0x7fee8003d1b0 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:42:34.096 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:34.089+0000 7feea6c4d700 1 --2- 172.21.15.42:0/2052236750 >> v2:172.21.15.40:3300/0 conn(0x7feea00a24b0 0x7feea0130850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:42:34.096 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:34.089+0000 7feea6c4d700 1 --2- 172.21.15.42:0/2052236750 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7feea008eeb0 0x7feea00a35f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:42:34.096 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:34.089+0000 7feea6c4d700 1 --2- 172.21.15.42:0/2052236750 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feea00a3b40 0x7feea0130300 unknown :-1 s=CLOSED pgs=1292 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:42:34.096 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:34.089+0000 7feea6c4d700 1 -- 172.21.15.42:0/2052236750 shutdown_connections 2019-12-05T05:42:34.096 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:34.089+0000 7feea6c4d700 1 -- 172.21.15.42:0/2052236750 wait complete. 2019-12-05T05:42:34.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:34.089+0000 7feea6c4d700 1 -- 172.21.15.42:0/2052236750 >> 172.21.15.42:0/2052236750 conn(0x7feea00aabe0 msgr2=0x7feea007e5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:42:34.105 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T05:42:36.105 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:42:36.217 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:36.201+0000 7ff89abe7700 1 Processor -- start 2019-12-05T05:42:36.217 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:36.201+0000 7ff89abe7700 1 -- start start 2019-12-05T05:42:36.218 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:36.201+0000 7ff89abe7700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff8940a7730 0x7ff89408df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:42:36.218 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:36.201+0000 7ff89abe7700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8940a0700 con 0x7ff89407e710 2019-12-05T05:42:36.218 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:36.201+0000 7ff89abe7700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8940a9b90 con 0x7ff89409fad0 2019-12-05T05:42:36.218 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:36.201+0000 7ff89abe7700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8940a0af0 con 0x7ff8940a7730 2019-12-05T05:42:36.218 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:36.201+0000 7ff899184700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff8940a7730 0x7ff89408df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:42:36.218 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:36.201+0000 7ff899985700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7ff89409fad0 0x7ff89414efd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:39310/0 (socket says 172.21.15.42:39310) 2019-12-05T05:42:36.218 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:36.201+0000 7ff899985700 1 -- 172.21.15.42:0/4030158053 learned_addr learned my addr 172.21.15.42:0/4030158053 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:42:36.219 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:36.201+0000 7ff899184700 1 -- 172.21.15.42:0/4030158053 >> v1:172.21.15.42:6789/0 conn(0x7ff89409fad0 legacy=0x7ff89414efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:42:36.219 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:36.201+0000 7ff899184700 1 -- 172.21.15.42:0/4030158053 >> v1:172.21.15.40:6789/0 conn(0x7ff89407e710 legacy=0x7ff89409f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:42:36.219 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:36.201+0000 7ff88bfff700 1 -- 172.21.15.42:0/4030158053 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2720705841 0 0) 0x7ff8940a9b90 con 0x7ff89409fad0 2019-12-05T05:42:36.219 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:36.201+0000 7ff899184700 1 -- 172.21.15.42:0/4030158053 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff89407eb40 con 0x7ff8940a7730 2019-12-05T05:42:36.219 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:36.201+0000 7ff899184700 1 --2- 172.21.15.42:0/4030158053 >> v2:172.21.15.42:3300/0 conn(0x7ff8940a7730 0x7ff89408df70 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7ff890008790 tx=0x7ff89000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:36.219 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:36.201+0000 7ff899184700 1 --2- 172.21.15.42:0/4030158053 >> v2:172.21.15.42:3300/0 conn(0x7ff8940a7730 0x7ff89408df70 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7ff890008790 tx=0x7ff89000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:36.219 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:36.201+0000 7ff899184700 1 --2- 172.21.15.42:0/4030158053 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff8940a7730 0x7ff89408df70 secure :-1 s=READY pgs=1297 cs=0 l=1 rx=0x7ff890008790 tx=0x7ff89000fbf0).ready entity=mon.1 client_cookie=15485960aca6d086 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:42:36.220 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:36.201+0000 7ff899184700 1 --2- 172.21.15.42:0/4030158053 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff8940a7730 0x7ff89408df70 secure :-1 s=THROTTLE_DONE pgs=1297 cs=0 l=1 rx=0x7ff890008790 tx=0x7ff89000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:36.220 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:36.201+0000 7ff899184700 1 --2- 172.21.15.42:0/4030158053 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff8940a7730 0x7ff89408df70 secure :-1 s=THROTTLE_DONE pgs=1297 cs=0 l=1 rx=0x7ff890008790 tx=0x7ff89000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:36.220 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:36.201+0000 7ff88bfff700 1 -- 172.21.15.42:0/4030158053 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff890011270 con 0x7ff8940a7730 2019-12-05T05:42:36.220 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:36.201+0000 7ff899184700 1 --2- 172.21.15.42:0/4030158053 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff8940a7730 0x7ff89408df70 secure :-1 s=THROTTLE_DONE pgs=1297 cs=0 l=1 rx=0x7ff890008790 tx=0x7ff89000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:36.220 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:36.201+0000 7ff88bfff700 1 -- 172.21.15.42:0/4030158053 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff890008bc0 con 0x7ff8940a7730 2019-12-05T05:42:36.220 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:36.201+0000 7ff88bfff700 1 -- 172.21.15.42:0/4030158053 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff890011850 con 0x7ff8940a7730 2019-12-05T05:42:36.221 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:36.201+0000 7ff89abe7700 1 -- 172.21.15.42:0/4030158053 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff8940a7730 msgr2=0x7ff89408df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:42:36.221 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:36.205+0000 7ff89abe7700 1 --2- 172.21.15.42:0/4030158053 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff8940a7730 0x7ff89408df70 secure :-1 s=READY pgs=1297 cs=0 l=1 rx=0x7ff890008790 tx=0x7ff89000fbf0).stop 2019-12-05T05:42:36.221 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:36.205+0000 7ff89abe7700 1 -- 172.21.15.42:0/4030158053 shutdown_connections 2019-12-05T05:42:36.221 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:36.205+0000 7ff89abe7700 1 --2- 172.21.15.42:0/4030158053 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff8940a7730 0x7ff89408df70 unknown :-1 s=CLOSED pgs=1297 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:42:36.221 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:36.205+0000 7ff89abe7700 1 -- 172.21.15.42:0/4030158053 shutdown_connections 2019-12-05T05:42:36.221 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:36.205+0000 7ff89abe7700 1 -- 172.21.15.42:0/4030158053 wait complete. 2019-12-05T05:42:36.221 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:36.205+0000 7ff89abe7700 1 -- 172.21.15.42:0/4030158053 >> 172.21.15.42:0/4030158053 conn(0x7ff8940aabd0 msgr2=0x7ff8940a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:42:36.222 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:36.205+0000 7ff89abe7700 1 Processor -- start 2019-12-05T05:42:36.222 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:36.205+0000 7ff89abe7700 1 -- start start 2019-12-05T05:42:36.222 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:36.209+0000 7ff89abe7700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff894119d70 0x7ff89407d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:42:36.222 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:36.209+0000 7ff89abe7700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff8941129d0 0x7ff894126b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:42:36.222 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:36.209+0000 7ff89abe7700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff89407e130 con 0x7ff89407dd00 2019-12-05T05:42:36.222 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:36.209+0000 7ff89abe7700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8941268e0 con 0x7ff8941129d0 2019-12-05T05:42:36.222 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:36.209+0000 7ff89abe7700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8940a2800 con 0x7ff894119d70 2019-12-05T05:42:36.223 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:36.209+0000 7ff899985700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff8941129d0 0x7ff894126b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:42:36.223 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:36.209+0000 7ff899184700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff894119d70 0x7ff89407d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:42:36.223 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:36.209+0000 7ff899184700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff894119d70 0x7ff89407d7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:39740/0 (socket says 172.21.15.42:39740) 2019-12-05T05:42:36.223 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:36.209+0000 7ff899184700 1 -- 172.21.15.42:0/1765906422 learned_addr learned my addr 172.21.15.42:0/1765906422 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:42:36.223 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:36.209+0000 7ff899184700 1 -- 172.21.15.42:0/1765906422 >> v2:172.21.15.40:3300/0 conn(0x7ff8941129d0 msgr2=0x7ff894126b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:42:36.223 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:36.209+0000 7ff899184700 1 --2- 172.21.15.42:0/1765906422 >> v2:172.21.15.40:3300/0 conn(0x7ff8941129d0 0x7ff894126b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:42:36.223 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:36.209+0000 7ff899184700 1 -- 172.21.15.42:0/1765906422 >> v1:172.21.15.40:6789/0 conn(0x7ff89407dd00 legacy=0x7ff8941121d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:42:36.225 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:36.209+0000 7ff899184700 1 -- 172.21.15.42:0/1765906422 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8940a29c0 con 0x7ff894119d70 2019-12-05T05:42:36.225 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:36.209+0000 7ff899184700 1 --2- 172.21.15.42:0/1765906422 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff894119d70 0x7ff89407d7b0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7ff890008790 tx=0x7ff890008f00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:36.225 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:36.209+0000 7ff899184700 1 --2- 172.21.15.42:0/1765906422 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff894119d70 0x7ff89407d7b0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7ff890008790 tx=0x7ff890008f00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:36.225 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:36.209+0000 7ff899184700 1 --2- 172.21.15.42:0/1765906422 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff894119d70 0x7ff89407d7b0 secure :-1 s=READY pgs=267 cs=0 l=1 rx=0x7ff890008790 tx=0x7ff890008f00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:42:36.225 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:36.209+0000 7ff899184700 1 --2- 172.21.15.42:0/1765906422 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff894119d70 0x7ff89407d7b0 secure :-1 s=THROTTLE_DONE pgs=267 cs=0 l=1 rx=0x7ff890008790 tx=0x7ff890008f00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:36.226 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:36.209+0000 7ff899184700 1 --2- 172.21.15.42:0/1765906422 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff894119d70 0x7ff89407d7b0 secure :-1 s=THROTTLE_DONE pgs=267 cs=0 l=1 rx=0x7ff890008790 tx=0x7ff890008f00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:36.226 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:36.209+0000 7ff889ffb700 1 -- 172.21.15.42:0/1765906422 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff89000f550 con 0x7ff894119d70 2019-12-05T05:42:36.226 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:36.209+0000 7ff899184700 1 --2- 172.21.15.42:0/1765906422 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff894119d70 0x7ff89407d7b0 secure :-1 s=THROTTLE_DONE pgs=267 cs=0 l=1 rx=0x7ff890008790 tx=0x7ff890008f00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:36.226 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:36.209+0000 7ff889ffb700 1 -- 172.21.15.42:0/1765906422 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff890011470 con 0x7ff894119d70 2019-12-05T05:42:36.226 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:36.209+0000 7ff89abe7700 1 -- 172.21.15.42:0/1765906422 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8941270d0 con 0x7ff894119d70 2019-12-05T05:42:36.227 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:36.209+0000 7ff889ffb700 1 -- 172.21.15.42:0/1765906422 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff8900126e0 con 0x7ff894119d70 2019-12-05T05:42:36.227 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:36.209+0000 7ff89abe7700 1 -- 172.21.15.42:0/1765906422 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff89412dc20 con 0x7ff894119d70 2019-12-05T05:42:36.227 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:36.213+0000 7ff899184700 1 --2- 172.21.15.42:0/1765906422 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff894119d70 0x7ff89407d7b0 secure :-1 s=THROTTLE_DONE pgs=267 cs=0 l=1 rx=0x7ff890008790 tx=0x7ff890008f00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:36.227 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:36.213+0000 7ff889ffb700 1 -- 172.21.15.42:0/1765906422 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7ff8900129b0 con 0x7ff894119d70 2019-12-05T05:42:36.227 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:36.213+0000 7ff899184700 1 --2- 172.21.15.42:0/1765906422 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff894119d70 0x7ff89407d7b0 secure :-1 s=THROTTLE_DONE pgs=267 cs=0 l=1 rx=0x7ff890008790 tx=0x7ff890008f00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:36.229 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:36.213+0000 7ff889ffb700 1 --2- 172.21.15.42:0/1765906422 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff87403ad70 0x7ff87403d280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:42:36.229 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:36.213+0000 7ff889ffb700 1 -- 172.21.15.42:0/1765906422 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 8181+0+0 (secure 0 0 0) 0x7ff89004bf00 con 0x7ff894119d70 2019-12-05T05:42:36.229 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:36.213+0000 7ff898983700 1 --2- 172.21.15.42:0/1765906422 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff87403ad70 0x7ff87403d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:42:36.229 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:36.213+0000 7ff898983700 1 --2- 172.21.15.42:0/1765906422 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff87403ad70 0x7ff87403d280 crc :-1 s=READY pgs=559 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:42:36.229 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:36.217+0000 7ff89abe7700 1 -- 172.21.15.42:0/1765906422 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff89405d100 con 0x7ff894119d70 2019-12-05T05:42:36.230 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:36.225+0000 7ff899184700 1 --2- 172.21.15.42:0/1765906422 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff894119d70 0x7ff89407d7b0 secure :-1 s=THROTTLE_DONE pgs=267 cs=0 l=1 rx=0x7ff890008790 tx=0x7ff890008f00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:36.230 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:36.225+0000 7ff889ffb700 1 -- 172.21.15.42:0/1765906422 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7ff890032f80 con 0x7ff894119d70 2019-12-05T05:42:36.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:36.397+0000 7ff89abe7700 1 -- 172.21.15.42:0/1765906422 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff89405b8b0 con 0x7ff894119d70 2019-12-05T05:42:36.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:36.397+0000 7ff899184700 1 --2- 172.21.15.42:0/1765906422 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff894119d70 0x7ff89407d7b0 secure :-1 s=THROTTLE_DONE pgs=267 cs=0 l=1 rx=0x7ff890008790 tx=0x7ff890008f00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:36.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:36.397+0000 7ff889ffb700 1 -- 172.21.15.42:0/1765906422 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 158 v158) v1 ==== 95+0+2112 (secure 0 0 0) 0x7ff8900331e0 con 0x7ff894119d70 2019-12-05T05:42:36.404 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:42:36.404 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":158,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7650,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/1029013337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1029013337},{"type":"v1","addr":"172.21.15.40:6833","nonce":1029013337}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":158}],"filesystems":[{"mdsmap":{"epoch":157,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:41:03.431537+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7539},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7539":{"gid":7539,"name":"a","rank":0,"incarnation":154,"state":"up:active","state_seq":30,"addr":"172.21.15.42:6835/1233824070","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1233824070},{"type":"v1","addr":"172.21.15.42:6835","nonce":1233824070}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:42:36.404 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 158 2019-12-05T05:42:36.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:36.401+0000 7ff89abe7700 1 -- 172.21.15.42:0/1765906422 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff87403ad70 msgr2=0x7ff87403d280 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:42:36.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:36.401+0000 7ff89abe7700 1 --2- 172.21.15.42:0/1765906422 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff87403ad70 0x7ff87403d280 crc :-1 s=READY pgs=559 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:42:36.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:36.401+0000 7ff89abe7700 1 -- 172.21.15.42:0/1765906422 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff894119d70 msgr2=0x7ff89407d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:42:36.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:36.401+0000 7ff89abe7700 1 --2- 172.21.15.42:0/1765906422 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff894119d70 0x7ff89407d7b0 secure :-1 s=READY pgs=267 cs=0 l=1 rx=0x7ff890008790 tx=0x7ff890008f00).stop 2019-12-05T05:42:36.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:36.401+0000 7ff89abe7700 1 -- 172.21.15.42:0/1765906422 shutdown_connections 2019-12-05T05:42:36.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:36.401+0000 7ff89abe7700 1 --2- 172.21.15.42:0/1765906422 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff87403ad70 0x7ff87403d280 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:42:36.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:36.401+0000 7ff89abe7700 1 --2- 172.21.15.42:0/1765906422 >> v2:172.21.15.40:3300/0 conn(0x7ff8941129d0 0x7ff894126b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:42:36.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:36.401+0000 7ff89abe7700 1 --2- 172.21.15.42:0/1765906422 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff894119d70 0x7ff89407d7b0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:42:36.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:36.401+0000 7ff89abe7700 1 -- 172.21.15.42:0/1765906422 shutdown_connections 2019-12-05T05:42:36.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:36.401+0000 7ff89abe7700 1 -- 172.21.15.42:0/1765906422 wait complete. 2019-12-05T05:42:36.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:36.401+0000 7ff89abe7700 1 -- 172.21.15.42:0/1765906422 >> 172.21.15.42:0/1765906422 conn(0x7ff8940a83d0 msgr2=0x7ff8940596f0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:42:36.417 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T05:42:38.419 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:42:38.523 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:38.513+0000 7f5a60844700 1 Processor -- start 2019-12-05T05:42:38.523 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:38.513+0000 7f5a60844700 1 -- start start 2019-12-05T05:42:38.523 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:38.517+0000 7f5a60844700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5a5c0a7730 0x7f5a5c08df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:42:38.524 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:38.517+0000 7f5a60844700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5a5c09fad0 0x7f5a5c07eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:42:38.524 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:38.517+0000 7f5a60844700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a5c0a0700 con 0x7f5a5c07e710 2019-12-05T05:42:38.524 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:38.517+0000 7f5a60844700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a5c0a0af0 con 0x7f5a5c0a7730 2019-12-05T05:42:38.524 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:38.517+0000 7f5a60844700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a5c08e5a0 con 0x7f5a5c09fad0 2019-12-05T05:42:38.524 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:38.517+0000 7f5a5ad9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5a5c0a7730 0x7f5a5c08df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:42:38.525 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:38.517+0000 7f5a5ad9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5a5c0a7730 0x7f5a5c08df70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:39282/0 (socket says 172.21.15.42:39282) 2019-12-05T05:42:38.525 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:38.517+0000 7f5a5ad9d700 1 -- 172.21.15.42:0/3087557371 learned_addr learned my addr 172.21.15.42:0/3087557371 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:42:38.525 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:38.517+0000 7f5a5a59c700 1 --1- 172.21.15.42:0/3087557371 >> v1:172.21.15.42:6789/0 conn(0x7f5a5c07e710 0x7f5a5c09f2d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:39320/0 (socket says 172.21.15.42:39320) 2019-12-05T05:42:38.525 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:38.517+0000 7f5a5b59e700 1 --2- 172.21.15.42:0/3087557371 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5a5c09fad0 0x7f5a5c07eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:42:38.525 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:38.517+0000 7f5a5ad9d700 1 -- 172.21.15.42:0/3087557371 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5a5c09fad0 msgr2=0x7f5a5c07eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:42:38.525 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:38.517+0000 7f5a5ad9d700 1 --2- 172.21.15.42:0/3087557371 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5a5c09fad0 0x7f5a5c07eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:42:38.526 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:38.517+0000 7f5a5ad9d700 1 -- 172.21.15.42:0/3087557371 >> v1:172.21.15.42:6789/0 conn(0x7f5a5c07e710 legacy=0x7f5a5c09f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:42:38.526 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:38.517+0000 7f5a5ad9d700 1 -- 172.21.15.42:0/3087557371 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a5c07f090 con 0x7f5a5c0a7730 2019-12-05T05:42:38.526 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:38.517+0000 7f5a59d9b700 1 -- 172.21.15.42:0/3087557371 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 580411771 0 0) 0x7f5a5c0a0700 con 0x7f5a5c07e710 2019-12-05T05:42:38.527 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:38.517+0000 7f5a5b59e700 1 --2- 172.21.15.42:0/3087557371 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5a5c09fad0 0x7f5a5c07eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T05:42:38.527 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:38.517+0000 7f5a5ad9d700 1 --2- 172.21.15.42:0/3087557371 >> v2:172.21.15.42:3300/0 conn(0x7f5a5c0a7730 0x7f5a5c08df70 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f5a54008b10 tx=0x7f5a540100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:38.527 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:38.517+0000 7f5a5ad9d700 1 --2- 172.21.15.42:0/3087557371 >> v2:172.21.15.42:3300/0 conn(0x7f5a5c0a7730 0x7f5a5c08df70 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f5a54008b10 tx=0x7f5a540100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:38.527 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:38.517+0000 7f5a5ad9d700 1 --2- 172.21.15.42:0/3087557371 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5a5c0a7730 0x7f5a5c08df70 secure :-1 s=READY pgs=1299 cs=0 l=1 rx=0x7f5a54008b10 tx=0x7f5a540100b0).ready entity=mon.1 client_cookie=657ba2ab99fd7f1 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:42:38.527 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:38.517+0000 7f5a5ad9d700 1 --2- 172.21.15.42:0/3087557371 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5a5c0a7730 0x7f5a5c08df70 secure :-1 s=THROTTLE_DONE pgs=1299 cs=0 l=1 rx=0x7f5a54008b10 tx=0x7f5a540100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:38.528 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:38.517+0000 7f5a59d9b700 1 -- 172.21.15.42:0/3087557371 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5a540110c0 con 0x7f5a5c0a7730 2019-12-05T05:42:38.528 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:38.517+0000 7f5a5ad9d700 1 --2- 172.21.15.42:0/3087557371 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5a5c0a7730 0x7f5a5c08df70 secure :-1 s=THROTTLE_DONE pgs=1299 cs=0 l=1 rx=0x7f5a54008b10 tx=0x7f5a540100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:38.528 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:38.517+0000 7f5a5ad9d700 1 --2- 172.21.15.42:0/3087557371 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5a5c0a7730 0x7f5a5c08df70 secure :-1 s=THROTTLE_DONE pgs=1299 cs=0 l=1 rx=0x7f5a54008b10 tx=0x7f5a540100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:38.528 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:38.517+0000 7f5a59d9b700 1 -- 172.21.15.42:0/3087557371 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a54010ab0 con 0x7f5a5c0a7730 2019-12-05T05:42:38.528 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:38.517+0000 7f5a59d9b700 1 -- 172.21.15.42:0/3087557371 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5a540116a0 con 0x7f5a5c0a7730 2019-12-05T05:42:38.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:38.517+0000 7f5a60844700 1 -- 172.21.15.42:0/3087557371 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5a5c0a7730 msgr2=0x7f5a5c08df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:42:38.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:38.517+0000 7f5a60844700 1 --2- 172.21.15.42:0/3087557371 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5a5c0a7730 0x7f5a5c08df70 secure :-1 s=READY pgs=1299 cs=0 l=1 rx=0x7f5a54008b10 tx=0x7f5a540100b0).stop 2019-12-05T05:42:38.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:38.517+0000 7f5a60844700 1 -- 172.21.15.42:0/3087557371 shutdown_connections 2019-12-05T05:42:38.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:38.517+0000 7f5a60844700 1 --2- 172.21.15.42:0/3087557371 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5a5c09fad0 0x7f5a5c07eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:42:38.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:38.517+0000 7f5a60844700 1 --2- 172.21.15.42:0/3087557371 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5a5c0a7730 0x7f5a5c08df70 unknown :-1 s=CLOSED pgs=1299 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:42:38.530 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:38.517+0000 7f5a60844700 1 -- 172.21.15.42:0/3087557371 shutdown_connections 2019-12-05T05:42:38.531 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:38.517+0000 7f5a60844700 1 -- 172.21.15.42:0/3087557371 wait complete. 2019-12-05T05:42:38.531 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:38.517+0000 7f5a60844700 1 -- 172.21.15.42:0/3087557371 >> 172.21.15.42:0/3087557371 conn(0x7f5a5c0aabd0 msgr2=0x7f5a5c0a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:42:38.535 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:38.521+0000 7f5a60844700 1 Processor -- start 2019-12-05T05:42:38.536 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:38.521+0000 7f5a60844700 1 -- start start 2019-12-05T05:42:38.536 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:38.521+0000 7f5a60844700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5a5c0a9970 0x7f5a5c078270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:42:38.536 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:38.521+0000 7f5a60844700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5a5c0a2210 0x7f5a5c144700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:42:38.536 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:38.521+0000 7f5a60844700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a5c078c60 con 0x7f5a5c144c50 2019-12-05T05:42:38.536 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:38.521+0000 7f5a60844700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a5c11a4a0 con 0x7f5a5c0a2210 2019-12-05T05:42:38.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:38.521+0000 7f5a60844700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a5c0a2750 con 0x7f5a5c0a9970 2019-12-05T05:42:38.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:38.521+0000 7f5a5b59e700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f5a5c144c50 0x7f5a5c119330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:39330/0 (socket says 172.21.15.42:39330) 2019-12-05T05:42:38.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:38.521+0000 7f5a5b59e700 1 -- 172.21.15.42:0/1237965932 learned_addr learned my addr 172.21.15.42:0/1237965932 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:42:38.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:38.521+0000 7f5a5ad9d700 1 --2- 172.21.15.42:0/1237965932 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5a5c0a9970 0x7f5a5c078270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:42:38.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:38.521+0000 7f5a5a59c700 1 --2- 172.21.15.42:0/1237965932 >> v2:172.21.15.40:3300/0 conn(0x7f5a5c0a2210 0x7f5a5c144700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:42:38.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:38.521+0000 7f5a537fe700 1 -- 172.21.15.42:0/1237965932 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2187170992 0 0) 0x7f5a5c078c60 con 0x7f5a5c144c50 2019-12-05T05:42:38.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:38.521+0000 7f5a537fe700 1 -- 172.21.15.42:0/1237965932 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a400030b0 con 0x7f5a5c144c50 2019-12-05T05:42:38.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:38.525+0000 7f5a5ad9d700 1 -- 172.21.15.42:0/1237965932 >> v2:172.21.15.40:3300/0 conn(0x7f5a5c0a2210 msgr2=0x7f5a5c144700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:42:38.539 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:38.525+0000 7f5a5ad9d700 1 --2- 172.21.15.42:0/1237965932 >> v2:172.21.15.40:3300/0 conn(0x7f5a5c0a2210 0x7f5a5c144700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:42:38.539 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:38.525+0000 7f5a5ad9d700 1 -- 172.21.15.42:0/1237965932 >> v1:172.21.15.42:6789/0 conn(0x7f5a5c144c50 legacy=0x7f5a5c119330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:42:38.539 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:38.525+0000 7f5a5ad9d700 1 -- 172.21.15.42:0/1237965932 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a5c0a2c90 con 0x7f5a5c0a9970 2019-12-05T05:42:38.539 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:38.525+0000 7f5a5ad9d700 1 --2- 172.21.15.42:0/1237965932 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5a5c0a9970 0x7f5a5c078270 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f5a54008b10 tx=0x7f5a54010d60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:38.539 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:38.525+0000 7f5a5ad9d700 1 --2- 172.21.15.42:0/1237965932 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5a5c0a9970 0x7f5a5c078270 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f5a54008b10 tx=0x7f5a54010d60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:38.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:38.525+0000 7f5a5ad9d700 1 --2- 172.21.15.42:0/1237965932 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5a5c0a9970 0x7f5a5c078270 secure :-1 s=READY pgs=268 cs=0 l=1 rx=0x7f5a54008b10 tx=0x7f5a54010d60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:42:38.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:38.525+0000 7f5a5ad9d700 1 --2- 172.21.15.42:0/1237965932 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5a5c0a9970 0x7f5a5c078270 secure :-1 s=THROTTLE_DONE pgs=268 cs=0 l=1 rx=0x7f5a54008b10 tx=0x7f5a54010d60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:38.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:38.525+0000 7f5a537fe700 1 -- 172.21.15.42:0/1237965932 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5a54011d80 con 0x7f5a5c0a9970 2019-12-05T05:42:38.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:38.525+0000 7f5a5ad9d700 1 --2- 172.21.15.42:0/1237965932 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5a5c0a9970 0x7f5a5c078270 secure :-1 s=THROTTLE_DONE pgs=268 cs=0 l=1 rx=0x7f5a54008b10 tx=0x7f5a54010d60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:38.541 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:38.525+0000 7f5a5ad9d700 1 --2- 172.21.15.42:0/1237965932 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5a5c0a9970 0x7f5a5c078270 secure :-1 s=THROTTLE_DONE pgs=268 cs=0 l=1 rx=0x7f5a54008b10 tx=0x7f5a54010d60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:38.541 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:38.525+0000 7f5a537fe700 1 -- 172.21.15.42:0/1237965932 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a540112c0 con 0x7f5a5c0a9970 2019-12-05T05:42:38.541 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:38.525+0000 7f5a60844700 1 -- 172.21.15.42:0/1237965932 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a5c0a42d0 con 0x7f5a5c0a9970 2019-12-05T05:42:38.541 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:38.525+0000 7f5a537fe700 1 -- 172.21.15.42:0/1237965932 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5a54005ca0 con 0x7f5a5c0a9970 2019-12-05T05:42:38.541 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:38.525+0000 7f5a60844700 1 -- 172.21.15.42:0/1237965932 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a5c0787c0 con 0x7f5a5c0a9970 2019-12-05T05:42:38.542 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:38.525+0000 7f5a5ad9d700 1 --2- 172.21.15.42:0/1237965932 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5a5c0a9970 0x7f5a5c078270 secure :-1 s=THROTTLE_DONE pgs=268 cs=0 l=1 rx=0x7f5a54008b10 tx=0x7f5a54010d60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:38.542 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:38.525+0000 7f5a537fe700 1 -- 172.21.15.42:0/1237965932 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f5a5400f280 con 0x7f5a5c0a9970 2019-12-05T05:42:38.542 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:38.525+0000 7f5a5ad9d700 1 --2- 172.21.15.42:0/1237965932 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5a5c0a9970 0x7f5a5c078270 secure :-1 s=THROTTLE_DONE pgs=268 cs=0 l=1 rx=0x7f5a54008b10 tx=0x7f5a54010d60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:38.542 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:38.525+0000 7f5a537fe700 1 --2- 172.21.15.42:0/1237965932 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5a4003b2a0 0x7f5a4003d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:42:38.542 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:38.525+0000 7f5a537fe700 1 -- 172.21.15.42:0/1237965932 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 8181+0+0 (secure 0 0 0) 0x7f5a5404b5e0 con 0x7f5a5c0a9970 2019-12-05T05:42:38.543 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:38.525+0000 7f5a5a59c700 1 --2- 172.21.15.42:0/1237965932 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5a4003b2a0 0x7f5a4003d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:42:38.543 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:38.525+0000 7f5a5a59c700 1 --2- 172.21.15.42:0/1237965932 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5a4003b2a0 0x7f5a4003d7b0 crc :-1 s=READY pgs=560 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:42:38.543 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:38.529+0000 7f5a60844700 1 -- 172.21.15.42:0/1237965932 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a5c05d100 con 0x7f5a5c0a9970 2019-12-05T05:42:38.544 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:38.537+0000 7f5a5ad9d700 1 --2- 172.21.15.42:0/1237965932 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5a5c0a9970 0x7f5a5c078270 secure :-1 s=THROTTLE_DONE pgs=268 cs=0 l=1 rx=0x7f5a54008b10 tx=0x7f5a54010d60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:38.544 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:38.537+0000 7f5a537fe700 1 -- 172.21.15.42:0/1237965932 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f5a540328c0 con 0x7f5a5c0a9970 2019-12-05T05:42:38.711 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:38.705+0000 7f5a60844700 1 -- 172.21.15.42:0/1237965932 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5a5c05b8b0 con 0x7f5a5c0a9970 2019-12-05T05:42:38.711 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:38.709+0000 7f5a5ad9d700 1 --2- 172.21.15.42:0/1237965932 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5a5c0a9970 0x7f5a5c078270 secure :-1 s=THROTTLE_DONE pgs=268 cs=0 l=1 rx=0x7f5a54008b10 tx=0x7f5a54010d60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:38.712 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:38.709+0000 7f5a537fe700 1 -- 172.21.15.42:0/1237965932 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 158 v158) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f5a54033470 con 0x7f5a5c0a9970 2019-12-05T05:42:38.715 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 158 2019-12-05T05:42:38.715 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:42:38.716 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":158,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7650,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/1029013337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1029013337},{"type":"v1","addr":"172.21.15.40:6833","nonce":1029013337}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":158}],"filesystems":[{"mdsmap":{"epoch":157,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:41:03.431537+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7539},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7539":{"gid":7539,"name":"a","rank":0,"incarnation":154,"state":"up:active","state_seq":30,"addr":"172.21.15.42:6835/1233824070","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1233824070},{"type":"v1","addr":"172.21.15.42:6835","nonce":1233824070}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:42:38.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:38.713+0000 7f5a60844700 1 -- 172.21.15.42:0/1237965932 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5a4003b2a0 msgr2=0x7f5a4003d7b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:42:38.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:38.713+0000 7f5a60844700 1 --2- 172.21.15.42:0/1237965932 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5a4003b2a0 0x7f5a4003d7b0 crc :-1 s=READY pgs=560 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:42:38.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:38.713+0000 7f5a60844700 1 -- 172.21.15.42:0/1237965932 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5a5c0a9970 msgr2=0x7f5a5c078270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:42:38.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:38.713+0000 7f5a60844700 1 --2- 172.21.15.42:0/1237965932 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5a5c0a9970 0x7f5a5c078270 secure :-1 s=READY pgs=268 cs=0 l=1 rx=0x7f5a54008b10 tx=0x7f5a54010d60).stop 2019-12-05T05:42:38.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:38.713+0000 7f5a60844700 1 -- 172.21.15.42:0/1237965932 shutdown_connections 2019-12-05T05:42:38.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:38.713+0000 7f5a60844700 1 --2- 172.21.15.42:0/1237965932 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5a4003b2a0 0x7f5a4003d7b0 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:42:38.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:38.713+0000 7f5a60844700 1 --2- 172.21.15.42:0/1237965932 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5a5c0a9970 0x7f5a5c078270 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:42:38.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:38.713+0000 7f5a60844700 1 --2- 172.21.15.42:0/1237965932 >> v2:172.21.15.40:3300/0 conn(0x7f5a5c0a2210 0x7f5a5c144700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:42:38.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:38.713+0000 7f5a60844700 1 -- 172.21.15.42:0/1237965932 shutdown_connections 2019-12-05T05:42:38.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:38.713+0000 7f5a60844700 1 -- 172.21.15.42:0/1237965932 wait complete. 2019-12-05T05:42:38.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:38.713+0000 7f5a60844700 1 -- 172.21.15.42:0/1237965932 >> 172.21.15.42:0/1237965932 conn(0x7f5a5c08ed90 msgr2=0x7f5a5c0a6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:42:38.743 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T05:42:40.753 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:42:40.857 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:40.844+0000 7fb2d28f5700 1 Processor -- start 2019-12-05T05:42:40.858 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:40.844+0000 7fb2d28f5700 1 -- start start 2019-12-05T05:42:40.858 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:40.848+0000 7fb2d28f5700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb2cc08df80 0x7fb2cc09f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:42:40.858 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:40.848+0000 7fb2d28f5700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb2cc09f830 0x7fb2cc14efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:42:40.858 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:40.848+0000 7fb2d28f5700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2cc09fc60 con 0x7fb2cc0a7740 2019-12-05T05:42:40.858 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:40.848+0000 7fb2d28f5700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2cc0a0b00 con 0x7fb2cc08df80 2019-12-05T05:42:40.858 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:40.848+0000 7fb2d28f5700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2cc08e5b0 con 0x7fb2cc09f830 2019-12-05T05:42:40.859 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:40.848+0000 7fb2d0e92700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fb2cc0a7740 0x7fb2cc07e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:39334/0 (socket says 172.21.15.42:39334) 2019-12-05T05:42:40.859 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:40.848+0000 7fb2d0e92700 1 -- 172.21.15.42:0/2326533590 learned_addr learned my addr 172.21.15.42:0/2326533590 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:42:40.859 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:40.848+0000 7fb2d1693700 1 --2- 172.21.15.42:0/2326533590 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb2cc09f830 0x7fb2cc14efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:42:40.859 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:40.848+0000 7fb2cbfff700 1 --2- 172.21.15.42:0/2326533590 >> v2:172.21.15.40:3300/0 conn(0x7fb2cc08df80 0x7fb2cc09f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:42:40.859 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:40.848+0000 7fb2cb7fe700 1 -- 172.21.15.42:0/2326533590 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 946670394 0 0) 0x7fb2cc09fc60 con 0x7fb2cc0a7740 2019-12-05T05:42:40.859 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:40.848+0000 7fb2cb7fe700 1 -- 172.21.15.42:0/2326533590 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2b40030b0 con 0x7fb2cc0a7740 2019-12-05T05:42:40.859 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:40.848+0000 7fb2d1693700 1 -- 172.21.15.42:0/2326533590 >> v2:172.21.15.40:3300/0 conn(0x7fb2cc08df80 msgr2=0x7fb2cc09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:42:40.860 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:40.848+0000 7fb2d1693700 1 --2- 172.21.15.42:0/2326533590 >> v2:172.21.15.40:3300/0 conn(0x7fb2cc08df80 0x7fb2cc09f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:42:40.860 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:40.848+0000 7fb2d1693700 1 -- 172.21.15.42:0/2326533590 >> v1:172.21.15.42:6789/0 conn(0x7fb2cc0a7740 legacy=0x7fb2cc07e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:42:40.860 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:40.848+0000 7fb2d1693700 1 -- 172.21.15.42:0/2326533590 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2cc0a0710 con 0x7fb2cc09f830 2019-12-05T05:42:40.860 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:40.848+0000 7fb2cbfff700 1 --2- 172.21.15.42:0/2326533590 >> v2:172.21.15.40:3300/0 conn(0x7fb2cc08df80 0x7fb2cc09f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:42:40.860 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:40.848+0000 7fb2d1693700 1 --2- 172.21.15.42:0/2326533590 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb2cc09f830 0x7fb2cc14efe0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fb2c4008790 tx=0x7fb2c400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:40.860 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:40.848+0000 7fb2d1693700 1 --2- 172.21.15.42:0/2326533590 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb2cc09f830 0x7fb2cc14efe0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fb2c4008790 tx=0x7fb2c400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:40.860 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:40.848+0000 7fb2d1693700 1 --2- 172.21.15.42:0/2326533590 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb2cc09f830 0x7fb2cc14efe0 secure :-1 s=READY pgs=269 cs=0 l=1 rx=0x7fb2c4008790 tx=0x7fb2c400fc70).ready entity=mon.2 client_cookie=ef6ff0957e7bd374 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:42:40.861 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:40.848+0000 7fb2d1693700 1 --2- 172.21.15.42:0/2326533590 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb2cc09f830 0x7fb2cc14efe0 secure :-1 s=THROTTLE_DONE pgs=269 cs=0 l=1 rx=0x7fb2c4008790 tx=0x7fb2c400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:40.861 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:40.848+0000 7fb2cb7fe700 1 -- 172.21.15.42:0/2326533590 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb2c4008bb0 con 0x7fb2cc09f830 2019-12-05T05:42:40.861 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:40.848+0000 7fb2d1693700 1 --2- 172.21.15.42:0/2326533590 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb2cc09f830 0x7fb2cc14efe0 secure :-1 s=THROTTLE_DONE pgs=269 cs=0 l=1 rx=0x7fb2c4008790 tx=0x7fb2c400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:40.861 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:40.848+0000 7fb2d1693700 1 --2- 172.21.15.42:0/2326533590 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb2cc09f830 0x7fb2cc14efe0 secure :-1 s=THROTTLE_DONE pgs=269 cs=0 l=1 rx=0x7fb2c4008790 tx=0x7fb2c400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:40.861 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:40.848+0000 7fb2cb7fe700 1 -- 172.21.15.42:0/2326533590 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2c40087d0 con 0x7fb2cc09f830 2019-12-05T05:42:40.861 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:40.848+0000 7fb2cb7fe700 1 -- 172.21.15.42:0/2326533590 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb2c4011860 con 0x7fb2cc09f830 2019-12-05T05:42:40.861 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:40.848+0000 7fb2d28f5700 1 -- 172.21.15.42:0/2326533590 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb2cc09f830 msgr2=0x7fb2cc14efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:42:40.862 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:40.848+0000 7fb2d28f5700 1 --2- 172.21.15.42:0/2326533590 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb2cc09f830 0x7fb2cc14efe0 secure :-1 s=READY pgs=269 cs=0 l=1 rx=0x7fb2c4008790 tx=0x7fb2c400fc70).stop 2019-12-05T05:42:40.862 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:40.852+0000 7fb2d28f5700 1 -- 172.21.15.42:0/2326533590 shutdown_connections 2019-12-05T05:42:40.862 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:40.852+0000 7fb2d28f5700 1 --2- 172.21.15.42:0/2326533590 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb2cc09f830 0x7fb2cc14efe0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:42:40.862 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:40.852+0000 7fb2d28f5700 1 --2- 172.21.15.42:0/2326533590 >> v2:172.21.15.40:3300/0 conn(0x7fb2cc08df80 0x7fb2cc09f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:42:40.862 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:40.852+0000 7fb2d28f5700 1 -- 172.21.15.42:0/2326533590 shutdown_connections 2019-12-05T05:42:40.862 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:40.852+0000 7fb2d28f5700 1 -- 172.21.15.42:0/2326533590 wait complete. 2019-12-05T05:42:40.863 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:40.852+0000 7fb2d28f5700 1 -- 172.21.15.42:0/2326533590 >> 172.21.15.42:0/2326533590 conn(0x7fb2cc0aabe0 msgr2=0x7fb2cc0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:42:40.872 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:40.852+0000 7fb2d28f5700 1 Processor -- start 2019-12-05T05:42:40.872 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:40.852+0000 7fb2d28f5700 1 -- start start 2019-12-05T05:42:40.872 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:40.856+0000 7fb2d28f5700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb2cc07c030 0x7fb2cc123990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:42:40.873 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:40.856+0000 7fb2d28f5700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb2cc131190 0x7fb2cc07d240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:42:40.873 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:40.856+0000 7fb2d28f5700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2cc123ee0 con 0x7fb2cc07ce10 2019-12-05T05:42:40.873 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:40.856+0000 7fb2d28f5700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2cc07d940 con 0x7fb2cc131190 2019-12-05T05:42:40.873 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:40.856+0000 7fb2d28f5700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2cc07dbe0 con 0x7fb2cc07c030 2019-12-05T05:42:40.873 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:40.856+0000 7fb2cbfff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fb2cc07ce10 0x7fb2cc130990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:39344/0 (socket says 172.21.15.42:39344) 2019-12-05T05:42:40.873 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:40.856+0000 7fb2cbfff700 1 -- 172.21.15.42:0/2865107916 learned_addr learned my addr 172.21.15.42:0/2865107916 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:42:40.873 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:40.856+0000 7fb2d1693700 1 --2- 172.21.15.42:0/2865107916 >> v2:172.21.15.40:3300/0 conn(0x7fb2cc131190 0x7fb2cc07d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:42:40.874 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:40.856+0000 7fb2d0e92700 1 --2- 172.21.15.42:0/2865107916 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb2cc07c030 0x7fb2cc123990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:42:40.874 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:40.856+0000 7fb2c97fa700 1 -- 172.21.15.42:0/2865107916 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1135101724 0 0) 0x7fb2cc123ee0 con 0x7fb2cc07ce10 2019-12-05T05:42:40.874 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:40.856+0000 7fb2c97fa700 1 -- 172.21.15.42:0/2865107916 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2ac0030b0 con 0x7fb2cc07ce10 2019-12-05T05:42:40.874 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:40.856+0000 7fb2c97fa700 1 -- 172.21.15.42:0/2865107916 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 640795697 0 0) 0x7fb2ac0030b0 con 0x7fb2cc07ce10 2019-12-05T05:42:40.874 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:40.856+0000 7fb2c97fa700 1 -- 172.21.15.42:0/2865107916 --> v1:172.21.15.42:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb2cc123ee0 con 0x7fb2cc07ce10 2019-12-05T05:42:40.874 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:40.856+0000 7fb2c97fa700 1 -- 172.21.15.42:0/2865107916 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7fb2c0002500 con 0x7fb2cc07ce10 2019-12-05T05:42:40.874 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:40.856+0000 7fb2c97fa700 1 -- 172.21.15.42:0/2865107916 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb2cc07c030 msgr2=0x7fb2cc123990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:42:40.875 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:40.856+0000 7fb2c97fa700 1 --2- 172.21.15.42:0/2865107916 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb2cc07c030 0x7fb2cc123990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:42:40.875 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:40.856+0000 7fb2c97fa700 1 -- 172.21.15.42:0/2865107916 >> v2:172.21.15.40:3300/0 conn(0x7fb2cc131190 msgr2=0x7fb2cc07d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:42:40.875 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:40.856+0000 7fb2c97fa700 1 --2- 172.21.15.42:0/2865107916 >> v2:172.21.15.40:3300/0 conn(0x7fb2cc131190 0x7fb2cc07d240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:42:40.875 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:40.856+0000 7fb2c97fa700 1 -- 172.21.15.42:0/2865107916 >> v1:172.21.15.42:6789/0 conn(0x7fb2cc07ce10 legacy=0x7fb2cc130990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:42:40.875 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:40.856+0000 7fb2c97fa700 1 --2- 172.21.15.42:0/2865107916 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb2ac0038a0 0x7fb2ac003d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:42:40.875 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:40.856+0000 7fb2c97fa700 1 --2- 172.21.15.42:0/2865107916 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb2ac004420 0x7fb2ac004850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:42:40.875 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:40.856+0000 7fb2c97fa700 1 --2- 172.21.15.42:0/2865107916 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb2ac004eb0 0x7fb2ac00b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:42:40.875 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:40.856+0000 7fb2c97fa700 1 -- 172.21.15.42:0/2865107916 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2cc07d940 con 0x7fb2ac0038a0 2019-12-05T05:42:40.876 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:40.856+0000 7fb2c97fa700 1 -- 172.21.15.42:0/2865107916 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2cc07dbe0 con 0x7fb2ac004420 2019-12-05T05:42:40.876 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:40.856+0000 7fb2c97fa700 1 -- 172.21.15.42:0/2865107916 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2ac00bbc0 con 0x7fb2ac004eb0 2019-12-05T05:42:40.876 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:40.856+0000 7fb2cbfff700 1 --2- 172.21.15.42:0/2865107916 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb2ac004420 0x7fb2ac004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:42:40.876 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:40.856+0000 7fb2d1693700 1 --2- 172.21.15.42:0/2865107916 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb2ac004eb0 0x7fb2ac00b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:42:40.876 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:40.856+0000 7fb2d0e92700 1 --2- 172.21.15.42:0/2865107916 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb2ac0038a0 0x7fb2ac003d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:42:40.877 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:40.856+0000 7fb2d1693700 1 -- 172.21.15.42:0/2865107916 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb2ac004420 msgr2=0x7fb2ac004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:42:40.877 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:40.856+0000 7fb2d1693700 1 --2- 172.21.15.42:0/2865107916 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb2ac004420 0x7fb2ac004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:42:40.877 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:40.856+0000 7fb2d1693700 1 -- 172.21.15.42:0/2865107916 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb2ac0038a0 msgr2=0x7fb2ac003d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:42:40.877 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:40.856+0000 7fb2d1693700 1 --2- 172.21.15.42:0/2865107916 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb2ac0038a0 0x7fb2ac003d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:42:40.877 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:40.856+0000 7fb2d1693700 1 -- 172.21.15.42:0/2865107916 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2cc07b660 con 0x7fb2ac004eb0 2019-12-05T05:42:40.877 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:40.856+0000 7fb2d1693700 1 --2- 172.21.15.42:0/2865107916 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb2ac004eb0 0x7fb2ac00b670 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fb2c4008790 tx=0x7fb2c4008b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:40.877 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:40.856+0000 7fb2d28f5700 1 -- 172.21.15.42:0/2865107916 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2cc07c030 con 0x7fb2ac004eb0 2019-12-05T05:42:40.877 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:40.856+0000 7fb2cbfff700 1 --2- 172.21.15.42:0/2865107916 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb2ac004420 0x7fb2ac004850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:42:40.878 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:40.856+0000 7fb2d28f5700 1 -- 172.21.15.42:0/2865107916 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2cc123990 con 0x7fb2ac004eb0 2019-12-05T05:42:40.878 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:40.856+0000 7fb2d1693700 1 --2- 172.21.15.42:0/2865107916 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb2ac004eb0 0x7fb2ac00b670 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fb2c4008790 tx=0x7fb2c4008b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:40.878 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:40.856+0000 7fb2d1693700 1 --2- 172.21.15.42:0/2865107916 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb2ac004eb0 0x7fb2ac00b670 secure :-1 s=READY pgs=270 cs=0 l=1 rx=0x7fb2c4008790 tx=0x7fb2c4008b00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:42:40.878 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:40.856+0000 7fb2d1693700 1 --2- 172.21.15.42:0/2865107916 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb2ac004eb0 0x7fb2ac00b670 secure :-1 s=THROTTLE_DONE pgs=270 cs=0 l=1 rx=0x7fb2c4008790 tx=0x7fb2c4008b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:40.878 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:40.856+0000 7fb2d1693700 1 --2- 172.21.15.42:0/2865107916 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb2ac004eb0 0x7fb2ac00b670 secure :-1 s=THROTTLE_DONE pgs=270 cs=0 l=1 rx=0x7fb2c4008790 tx=0x7fb2c4008b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:40.878 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:40.856+0000 7fb2c97fa700 1 -- 172.21.15.42:0/2865107916 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb2c4005540 con 0x7fb2ac004eb0 2019-12-05T05:42:40.878 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:40.856+0000 7fb2d1693700 1 --2- 172.21.15.42:0/2865107916 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb2ac004eb0 0x7fb2ac00b670 secure :-1 s=THROTTLE_DONE pgs=270 cs=0 l=1 rx=0x7fb2c4008790 tx=0x7fb2c4008b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:40.879 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:40.860+0000 7fb2c97fa700 1 -- 172.21.15.42:0/2865107916 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2c4008db0 con 0x7fb2ac004eb0 2019-12-05T05:42:40.879 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:40.860+0000 7fb2c97fa700 1 -- 172.21.15.42:0/2865107916 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb2c4005b60 con 0x7fb2ac004eb0 2019-12-05T05:42:40.879 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:40.860+0000 7fb2d1693700 1 --2- 172.21.15.42:0/2865107916 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb2ac004eb0 0x7fb2ac00b670 secure :-1 s=THROTTLE_DONE pgs=270 cs=0 l=1 rx=0x7fb2c4008790 tx=0x7fb2c4008b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:40.879 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:40.860+0000 7fb2c97fa700 1 -- 172.21.15.42:0/2865107916 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fb2c4010490 con 0x7fb2ac004eb0 2019-12-05T05:42:40.879 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:40.860+0000 7fb2d1693700 1 --2- 172.21.15.42:0/2865107916 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb2ac004eb0 0x7fb2ac00b670 secure :-1 s=THROTTLE_DONE pgs=270 cs=0 l=1 rx=0x7fb2c4008790 tx=0x7fb2c4008b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:40.879 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:40.860+0000 7fb2c97fa700 1 --2- 172.21.15.42:0/2865107916 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb2ac0439d0 0x7fb2ac045ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:42:40.879 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:40.860+0000 7fb2c97fa700 1 -- 172.21.15.42:0/2865107916 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 8181+0+0 (secure 0 0 0) 0x7fb2c404b820 con 0x7fb2ac004eb0 2019-12-05T05:42:40.879 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:40.860+0000 7fb2d0e92700 1 --2- 172.21.15.42:0/2865107916 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb2ac0439d0 0x7fb2ac045ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:42:40.880 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:40.860+0000 7fb2d0e92700 1 --2- 172.21.15.42:0/2865107916 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb2ac0439d0 0x7fb2ac045ee0 crc :-1 s=READY pgs=561 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:42:40.880 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:40.860+0000 7fb2d28f5700 1 -- 172.21.15.42:0/2865107916 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2a4003200 con 0x7fb2ac004eb0 2019-12-05T05:42:40.880 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:40.868+0000 7fb2d1693700 1 --2- 172.21.15.42:0/2865107916 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb2ac004eb0 0x7fb2ac00b670 secure :-1 s=THROTTLE_DONE pgs=270 cs=0 l=1 rx=0x7fb2c4008790 tx=0x7fb2c4008b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:40.881 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:40.868+0000 7fb2c97fa700 1 -- 172.21.15.42:0/2865107916 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fb2c4010de0 con 0x7fb2ac004eb0 2019-12-05T05:42:40.989 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:42:40.992 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:42:41.048 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:41.040+0000 7fb2d28f5700 1 -- 172.21.15.42:0/2865107916 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb2a4003630 con 0x7fb2ac004eb0 2019-12-05T05:42:41.049 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:41.040+0000 7fb2d1693700 1 --2- 172.21.15.42:0/2865107916 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb2ac004eb0 0x7fb2ac00b670 secure :-1 s=THROTTLE_DONE pgs=270 cs=0 l=1 rx=0x7fb2c4008790 tx=0x7fb2c4008b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:41.049 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:41.040+0000 7fb2c97fa700 1 -- 172.21.15.42:0/2865107916 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 158 v158) v1 ==== 95+0+2112 (secure 0 0 0) 0x7fb2c40328c0 con 0x7fb2ac004eb0 2019-12-05T05:42:41.049 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 158 2019-12-05T05:42:41.049 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:42:41.049 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":158,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7650,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/1029013337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1029013337},{"type":"v1","addr":"172.21.15.40:6833","nonce":1029013337}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":158}],"filesystems":[{"mdsmap":{"epoch":157,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:41:03.431537+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7539},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7539":{"gid":7539,"name":"a","rank":0,"incarnation":154,"state":"up:active","state_seq":30,"addr":"172.21.15.42:6835/1233824070","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1233824070},{"type":"v1","addr":"172.21.15.42:6835","nonce":1233824070}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:42:41.052 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:41.044+0000 7fb2d28f5700 1 -- 172.21.15.42:0/2865107916 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb2ac0439d0 msgr2=0x7fb2ac045ee0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:42:41.052 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:41.044+0000 7fb2d28f5700 1 --2- 172.21.15.42:0/2865107916 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb2ac0439d0 0x7fb2ac045ee0 crc :-1 s=READY pgs=561 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:42:41.053 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:41.044+0000 7fb2d28f5700 1 -- 172.21.15.42:0/2865107916 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb2ac004eb0 msgr2=0x7fb2ac00b670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:42:41.053 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:41.044+0000 7fb2d28f5700 1 --2- 172.21.15.42:0/2865107916 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb2ac004eb0 0x7fb2ac00b670 secure :-1 s=READY pgs=270 cs=0 l=1 rx=0x7fb2c4008790 tx=0x7fb2c4008b00).stop 2019-12-05T05:42:41.053 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:41.044+0000 7fb2d1693700 1 -- 172.21.15.42:0/2865107916 reap_dead start 2019-12-05T05:42:41.053 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:41.044+0000 7fb2d1693700 1 -- 172.21.15.42:0/2865107916 reap_dead start 2019-12-05T05:42:41.053 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:41.044+0000 7fb2d28f5700 1 -- 172.21.15.42:0/2865107916 shutdown_connections 2019-12-05T05:42:41.053 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:41.044+0000 7fb2d28f5700 1 -- 172.21.15.42:0/2865107916 shutdown_connections 2019-12-05T05:42:41.053 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:41.044+0000 7fb2d28f5700 1 -- 172.21.15.42:0/2865107916 wait complete. 2019-12-05T05:42:41.054 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:41.044+0000 7fb2d28f5700 1 -- 172.21.15.42:0/2865107916 >> 172.21.15.42:0/2865107916 conn(0x7fb2cc08fde0 msgr2=0x7fb2cc059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:42:41.060 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T05:42:43.061 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:42:43.169 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:43.156+0000 7f4a8a066700 1 Processor -- start 2019-12-05T05:42:43.169 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:43.156+0000 7f4a8a066700 1 -- start start 2019-12-05T05:42:43.169 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:43.160+0000 7f4a8a066700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f4a840a7740 0x7f4a8408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:42:43.170 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:43.160+0000 7f4a8a066700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f4a8409fae0 0x7f4a8407eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:42:43.170 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:43.160+0000 7f4a8a066700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a840a0710 con 0x7f4a8407e720 2019-12-05T05:42:43.170 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:43.160+0000 7f4a8a066700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a840a0b00 con 0x7f4a840a7740 2019-12-05T05:42:43.170 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:43.160+0000 7f4a8a066700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a8408e5b0 con 0x7f4a8409fae0 2019-12-05T05:42:43.170 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:43.160+0000 7f4a88e04700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f4a8409fae0 0x7f4a8407eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:42:43.170 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:43.160+0000 7f4a88e04700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f4a8409fae0 0x7f4a8407eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:39322/0 (socket says 172.21.15.42:39322) 2019-12-05T05:42:43.170 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:43.160+0000 7f4a88e04700 1 -- 172.21.15.42:0/3806370665 learned_addr learned my addr 172.21.15.42:0/3806370665 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:42:43.171 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:43.160+0000 7f4a83fff700 1 --2- 172.21.15.42:0/3806370665 >> v2:172.21.15.40:3300/0 conn(0x7f4a840a7740 0x7f4a8408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:42:43.171 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:43.160+0000 7f4a88e04700 1 -- 172.21.15.42:0/3806370665 >> v2:172.21.15.40:3300/0 conn(0x7f4a840a7740 msgr2=0x7f4a8408df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:42:43.171 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:43.160+0000 7f4a88e04700 1 --2- 172.21.15.42:0/3806370665 >> v2:172.21.15.40:3300/0 conn(0x7f4a840a7740 0x7f4a8408df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:42:43.171 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:43.160+0000 7f4a88e04700 1 -- 172.21.15.42:0/3806370665 >> v1:172.21.15.40:6789/0 conn(0x7f4a8407e720 legacy=0x7f4a8409f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:42:43.171 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:43.160+0000 7f4a88e04700 1 -- 172.21.15.42:0/3806370665 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a8407f0a0 con 0x7f4a8409fae0 2019-12-05T05:42:43.171 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:43.160+0000 7f4a88e04700 1 --2- 172.21.15.42:0/3806370665 >> v2:172.21.15.42:3300/0 conn(0x7f4a8409fae0 0x7f4a8407eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f4a7c002f30 tx=0x7f4a7c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:43.174 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:43.160+0000 7f4a88e04700 1 --2- 172.21.15.42:0/3806370665 >> v2:172.21.15.42:3300/0 conn(0x7f4a8409fae0 0x7f4a8407eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f4a7c002f30 tx=0x7f4a7c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:43.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:43.160+0000 7f4a88e04700 1 --2- 172.21.15.42:0/3806370665 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4a8409fae0 0x7f4a8407eb50 secure :-1 s=READY pgs=1303 cs=0 l=1 rx=0x7f4a7c002f30 tx=0x7f4a7c00fd30).ready entity=mon.1 client_cookie=64d59692f1cc6f2b server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:42:43.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:43.160+0000 7f4a88e04700 1 --2- 172.21.15.42:0/3806370665 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4a8409fae0 0x7f4a8407eb50 secure :-1 s=THROTTLE_DONE pgs=1303 cs=0 l=1 rx=0x7f4a7c002f30 tx=0x7f4a7c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:43.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:43.160+0000 7f4a88e04700 1 --2- 172.21.15.42:0/3806370665 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4a8409fae0 0x7f4a8407eb50 secure :-1 s=THROTTLE_DONE pgs=1303 cs=0 l=1 rx=0x7f4a7c002f30 tx=0x7f4a7c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:43.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:43.160+0000 7f4a82ffd700 1 -- 172.21.15.42:0/3806370665 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4a7c008c90 con 0x7f4a8409fae0 2019-12-05T05:42:43.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:43.160+0000 7f4a88e04700 1 --2- 172.21.15.42:0/3806370665 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4a8409fae0 0x7f4a8407eb50 secure :-1 s=THROTTLE_DONE pgs=1303 cs=0 l=1 rx=0x7f4a7c002f30 tx=0x7f4a7c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:43.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:43.160+0000 7f4a82ffd700 1 -- 172.21.15.42:0/3806370665 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a7c0088b0 con 0x7f4a8409fae0 2019-12-05T05:42:43.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:43.160+0000 7f4a82ffd700 1 -- 172.21.15.42:0/3806370665 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4a7c011ad0 con 0x7f4a8409fae0 2019-12-05T05:42:43.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:43.160+0000 7f4a8a066700 1 -- 172.21.15.42:0/3806370665 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4a8409fae0 msgr2=0x7f4a8407eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:42:43.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:43.160+0000 7f4a8a066700 1 --2- 172.21.15.42:0/3806370665 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4a8409fae0 0x7f4a8407eb50 secure :-1 s=READY pgs=1303 cs=0 l=1 rx=0x7f4a7c002f30 tx=0x7f4a7c00fd30).stop 2019-12-05T05:42:43.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:43.160+0000 7f4a8a066700 1 -- 172.21.15.42:0/3806370665 shutdown_connections 2019-12-05T05:42:43.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:43.160+0000 7f4a8a066700 1 --2- 172.21.15.42:0/3806370665 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4a8409fae0 0x7f4a8407eb50 unknown :-1 s=CLOSED pgs=1303 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:42:43.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:43.160+0000 7f4a8a066700 1 --2- 172.21.15.42:0/3806370665 >> v2:172.21.15.40:3300/0 conn(0x7f4a840a7740 0x7f4a8408df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:42:43.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:43.160+0000 7f4a8a066700 1 -- 172.21.15.42:0/3806370665 shutdown_connections 2019-12-05T05:42:43.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:43.160+0000 7f4a8a066700 1 -- 172.21.15.42:0/3806370665 wait complete. 2019-12-05T05:42:43.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:43.160+0000 7f4a8a066700 1 -- 172.21.15.42:0/3806370665 >> 172.21.15.42:0/3806370665 conn(0x7f4a840aabe0 msgr2=0x7f4a840a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:42:43.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:43.164+0000 7f4a8a066700 1 Processor -- start 2019-12-05T05:42:43.184 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:43.164+0000 7f4a8a066700 1 -- start start 2019-12-05T05:42:43.187 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:43.164+0000 7f4a8a066700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f4a840a9ad0 0x7f4a84142620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:42:43.187 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:43.164+0000 7f4a8a066700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4a84144df0 0x7f4a8412e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:42:43.187 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:43.164+0000 7f4a8a066700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a84142fa0 con 0x7f4a84142b70 2019-12-05T05:42:43.187 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:43.164+0000 7f4a8a066700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a841433a0 con 0x7f4a840a9ad0 2019-12-05T05:42:43.187 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:43.164+0000 7f4a8a066700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a84143e10 con 0x7f4a84144df0 2019-12-05T05:42:43.188 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:43.164+0000 7f4a837fe700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f4a84142b70 0x7f4a841445f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:39364/0 (socket says 172.21.15.42:39364) 2019-12-05T05:42:43.188 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:43.164+0000 7f4a837fe700 1 -- 172.21.15.42:0/1488715047 learned_addr learned my addr 172.21.15.42:0/1488715047 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:42:43.188 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:43.164+0000 7f4a83fff700 1 --2- 172.21.15.42:0/1488715047 >> v2:172.21.15.40:3300/0 conn(0x7f4a840a9ad0 0x7f4a84142620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:42:43.188 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:43.164+0000 7f4a88e04700 1 --2- 172.21.15.42:0/1488715047 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4a84144df0 0x7f4a8412e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:42:43.188 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:43.164+0000 7f4a80ff9700 1 -- 172.21.15.42:0/1488715047 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3675615858 0 0) 0x7f4a84142fa0 con 0x7f4a84142b70 2019-12-05T05:42:43.188 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:43.164+0000 7f4a80ff9700 1 -- 172.21.15.42:0/1488715047 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a640030b0 con 0x7f4a84142b70 2019-12-05T05:42:43.188 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:43.168+0000 7f4a83fff700 1 -- 172.21.15.42:0/1488715047 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4a84144df0 msgr2=0x7f4a8412e2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:42:43.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:43.168+0000 7f4a80ff9700 1 -- 172.21.15.42:0/1488715047 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 3454834844 0 0) 0x7f4a640030b0 con 0x7f4a84142b70 2019-12-05T05:42:43.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:43.168+0000 7f4a83fff700 1 --2- 172.21.15.42:0/1488715047 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4a84144df0 0x7f4a8412e2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:42:43.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:43.168+0000 7f4a83fff700 1 -- 172.21.15.42:0/1488715047 >> v1:172.21.15.42:6789/0 conn(0x7f4a84142b70 legacy=0x7f4a841445f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:42:43.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:43.168+0000 7f4a83fff700 1 -- 172.21.15.42:0/1488715047 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a8412e7f0 con 0x7f4a840a9ad0 2019-12-05T05:42:43.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:43.168+0000 7f4a83fff700 1 --2- 172.21.15.42:0/1488715047 >> v2:172.21.15.40:3300/0 conn(0x7f4a840a9ad0 0x7f4a84142620 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f4a78001ca0 tx=0x7f4a78001ce0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:43.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:43.168+0000 7f4a83fff700 1 --2- 172.21.15.42:0/1488715047 >> v2:172.21.15.40:3300/0 conn(0x7f4a840a9ad0 0x7f4a84142620 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f4a78001ca0 tx=0x7f4a78001ce0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:43.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:43.168+0000 7f4a83fff700 1 --2- 172.21.15.42:0/1488715047 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4a840a9ad0 0x7f4a84142620 secure :-1 s=READY pgs=1014 cs=0 l=1 rx=0x7f4a78001ca0 tx=0x7f4a78001ce0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:42:43.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:43.168+0000 7f4a83fff700 1 --2- 172.21.15.42:0/1488715047 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4a840a9ad0 0x7f4a84142620 secure :-1 s=THROTTLE_DONE pgs=1014 cs=0 l=1 rx=0x7f4a78001ca0 tx=0x7f4a78001ce0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:43.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:43.168+0000 7f4a80ff9700 1 -- 172.21.15.42:0/1488715047 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4a78001a60 con 0x7f4a840a9ad0 2019-12-05T05:42:43.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:43.168+0000 7f4a83fff700 1 --2- 172.21.15.42:0/1488715047 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4a840a9ad0 0x7f4a84142620 secure :-1 s=THROTTLE_DONE pgs=1014 cs=0 l=1 rx=0x7f4a78001ca0 tx=0x7f4a78001ce0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:43.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:43.168+0000 7f4a83fff700 1 --2- 172.21.15.42:0/1488715047 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4a840a9ad0 0x7f4a84142620 secure :-1 s=THROTTLE_DONE pgs=1014 cs=0 l=1 rx=0x7f4a78001ca0 tx=0x7f4a78001ce0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:43.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:43.168+0000 7f4a80ff9700 1 -- 172.21.15.42:0/1488715047 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a78001870 con 0x7f4a840a9ad0 2019-12-05T05:42:43.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:43.168+0000 7f4a80ff9700 1 -- 172.21.15.42:0/1488715047 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4a740127d0 con 0x7f4a840a9ad0 2019-12-05T05:42:43.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:43.168+0000 7f4a8a066700 1 -- 172.21.15.42:0/1488715047 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a8412eac0 con 0x7f4a840a9ad0 2019-12-05T05:42:43.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:43.168+0000 7f4a8a066700 1 -- 172.21.15.42:0/1488715047 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a8412eff0 con 0x7f4a840a9ad0 2019-12-05T05:42:43.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:43.168+0000 7f4a83fff700 1 --2- 172.21.15.42:0/1488715047 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4a840a9ad0 0x7f4a84142620 secure :-1 s=THROTTLE_DONE pgs=1014 cs=0 l=1 rx=0x7f4a78001ca0 tx=0x7f4a78001ce0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:43.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:43.168+0000 7f4a83fff700 1 --2- 172.21.15.42:0/1488715047 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4a840a9ad0 0x7f4a84142620 secure :-1 s=THROTTLE_DONE pgs=1014 cs=0 l=1 rx=0x7f4a78001ca0 tx=0x7f4a78001ce0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:43.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:43.168+0000 7f4a80ff9700 1 -- 172.21.15.42:0/1488715047 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f4a74012aa0 con 0x7f4a840a9ad0 2019-12-05T05:42:43.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:43.168+0000 7f4a80ff9700 1 --2- 172.21.15.42:0/1488715047 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4a6403b210 0x7f4a6403d720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:42:43.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:43.168+0000 7f4a80ff9700 1 -- 172.21.15.42:0/1488715047 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 8181+0+0 (secure 0 0 0) 0x7f4a740090b0 con 0x7f4a840a9ad0 2019-12-05T05:42:43.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:43.168+0000 7f4a837fe700 1 --2- 172.21.15.42:0/1488715047 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4a6403b210 0x7f4a6403d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:42:43.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:43.168+0000 7f4a837fe700 1 --2- 172.21.15.42:0/1488715047 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4a6403b210 0x7f4a6403d720 crc :-1 s=READY pgs=562 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:42:43.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:43.172+0000 7f4a8a066700 1 -- 172.21.15.42:0/1488715047 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a8405d110 con 0x7f4a840a9ad0 2019-12-05T05:42:43.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:43.180+0000 7f4a83fff700 1 --2- 172.21.15.42:0/1488715047 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4a840a9ad0 0x7f4a84142620 secure :-1 s=THROTTLE_DONE pgs=1014 cs=0 l=1 rx=0x7f4a78001ca0 tx=0x7f4a78001ce0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:43.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:43.180+0000 7f4a80ff9700 1 -- 172.21.15.42:0/1488715047 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f4a74032370 con 0x7f4a840a9ad0 2019-12-05T05:42:43.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:43.352+0000 7f4a8a066700 1 -- 172.21.15.42:0/1488715047 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4a8405b8c0 con 0x7f4a840a9ad0 2019-12-05T05:42:43.358 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:43.352+0000 7f4a83fff700 1 --2- 172.21.15.42:0/1488715047 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4a840a9ad0 0x7f4a84142620 secure :-1 s=THROTTLE_DONE pgs=1014 cs=0 l=1 rx=0x7f4a78001ca0 tx=0x7f4a78001ce0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:43.358 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:43.352+0000 7f4a80ff9700 1 -- 172.21.15.42:0/1488715047 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 158 v158) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f4a74012db0 con 0x7f4a840a9ad0 2019-12-05T05:42:43.360 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 158 2019-12-05T05:42:43.361 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:42:43.361 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":158,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7650,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/1029013337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1029013337},{"type":"v1","addr":"172.21.15.40:6833","nonce":1029013337}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":158}],"filesystems":[{"mdsmap":{"epoch":157,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:41:03.431537+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7539},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7539":{"gid":7539,"name":"a","rank":0,"incarnation":154,"state":"up:active","state_seq":30,"addr":"172.21.15.42:6835/1233824070","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1233824070},{"type":"v1","addr":"172.21.15.42:6835","nonce":1233824070}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:42:43.363 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:43.356+0000 7f4a8a066700 1 -- 172.21.15.42:0/1488715047 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4a6403b210 msgr2=0x7f4a6403d720 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:42:43.363 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:43.356+0000 7f4a8a066700 1 --2- 172.21.15.42:0/1488715047 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4a6403b210 0x7f4a6403d720 crc :-1 s=READY pgs=562 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:42:43.364 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:43.356+0000 7f4a8a066700 1 -- 172.21.15.42:0/1488715047 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4a840a9ad0 msgr2=0x7f4a84142620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:42:43.364 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:43.356+0000 7f4a8a066700 1 --2- 172.21.15.42:0/1488715047 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4a840a9ad0 0x7f4a84142620 secure :-1 s=READY pgs=1014 cs=0 l=1 rx=0x7f4a78001ca0 tx=0x7f4a78001ce0).stop 2019-12-05T05:42:43.364 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:43.356+0000 7f4a8a066700 1 -- 172.21.15.42:0/1488715047 shutdown_connections 2019-12-05T05:42:43.364 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:43.356+0000 7f4a8a066700 1 --2- 172.21.15.42:0/1488715047 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4a6403b210 0x7f4a6403d720 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:42:43.364 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:43.356+0000 7f4a8a066700 1 --2- 172.21.15.42:0/1488715047 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4a84144df0 0x7f4a8412e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:42:43.364 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:43.356+0000 7f4a8a066700 1 --2- 172.21.15.42:0/1488715047 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4a840a9ad0 0x7f4a84142620 unknown :-1 s=CLOSED pgs=1014 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:42:43.364 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:43.356+0000 7f4a8a066700 1 -- 172.21.15.42:0/1488715047 shutdown_connections 2019-12-05T05:42:43.365 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:43.356+0000 7f4a8a066700 1 -- 172.21.15.42:0/1488715047 wait complete. 2019-12-05T05:42:43.365 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:43.356+0000 7f4a8a066700 1 -- 172.21.15.42:0/1488715047 >> 172.21.15.42:0/1488715047 conn(0x7f4a840a7f90 msgr2=0x7f4a84118730 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:42:43.374 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T05:42:45.374 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:42:45.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.460+0000 7fa0ad44f700 1 Processor -- start 2019-12-05T05:42:45.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.460+0000 7fa0ad44f700 1 -- start start 2019-12-05T05:42:45.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.460+0000 7fa0ad44f700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa0a80a7740 0x7fa0a808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:42:45.483 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.460+0000 7fa0ad44f700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa0a807e720 0x7fa0a807eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:42:45.483 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.460+0000 7fa0ad44f700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0a80a0710 con 0x7fa0a809f2e0 2019-12-05T05:42:45.483 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.460+0000 7fa0ad44f700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0a80a0b00 con 0x7fa0a80a7740 2019-12-05T05:42:45.483 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.460+0000 7fa0ad44f700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0a808e5b0 con 0x7fa0a807e720 2019-12-05T05:42:45.484 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.460+0000 7fa0a7fff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fa0a809f2e0 0x7fa0a809f710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:39374/0 (socket says 172.21.15.42:39374) 2019-12-05T05:42:45.484 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.460+0000 7fa0a7fff700 1 -- 172.21.15.42:0/1057473590 learned_addr learned my addr 172.21.15.42:0/1057473590 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:42:45.484 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.460+0000 7fa0a6ffd700 1 --2- 172.21.15.42:0/1057473590 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa0a807e720 0x7fa0a807eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:42:45.484 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.460+0000 7fa0a77fe700 1 --2- 172.21.15.42:0/1057473590 >> v2:172.21.15.40:3300/0 conn(0x7fa0a80a7740 0x7fa0a808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:42:45.489 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.464+0000 7fa0a6ffd700 1 -- 172.21.15.42:0/1057473590 >> v2:172.21.15.40:3300/0 conn(0x7fa0a80a7740 msgr2=0x7fa0a808df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:42:45.489 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.464+0000 7fa0a6ffd700 1 --2- 172.21.15.42:0/1057473590 >> v2:172.21.15.40:3300/0 conn(0x7fa0a80a7740 0x7fa0a808df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:42:45.489 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.464+0000 7fa0a6ffd700 1 -- 172.21.15.42:0/1057473590 >> v1:172.21.15.42:6789/0 conn(0x7fa0a809f2e0 legacy=0x7fa0a809f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:42:45.489 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.464+0000 7fa0a67fc700 1 -- 172.21.15.42:0/1057473590 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 41770599 0 0) 0x7fa0a80a0710 con 0x7fa0a809f2e0 2019-12-05T05:42:45.489 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.464+0000 7fa0a6ffd700 1 -- 172.21.15.42:0/1057473590 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0a807f0a0 con 0x7fa0a807e720 2019-12-05T05:42:45.490 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.464+0000 7fa0a6ffd700 1 --2- 172.21.15.42:0/1057473590 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa0a807e720 0x7fa0a807eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fa098008790 tx=0x7fa09800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:45.490 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.464+0000 7fa0a6ffd700 1 --2- 172.21.15.42:0/1057473590 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa0a807e720 0x7fa0a807eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fa098008790 tx=0x7fa09800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:45.490 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.464+0000 7fa0a6ffd700 1 --2- 172.21.15.42:0/1057473590 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa0a807e720 0x7fa0a807eb50 secure :-1 s=READY pgs=271 cs=0 l=1 rx=0x7fa098008790 tx=0x7fa09800fc70).ready entity=mon.2 client_cookie=38d15ad78e01465f server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:42:45.490 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.464+0000 7fa0a6ffd700 1 --2- 172.21.15.42:0/1057473590 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa0a807e720 0x7fa0a807eb50 secure :-1 s=THROTTLE_DONE pgs=271 cs=0 l=1 rx=0x7fa098008790 tx=0x7fa09800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:45.490 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.464+0000 7fa0a67fc700 1 -- 172.21.15.42:0/1057473590 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa0980112f0 con 0x7fa0a807e720 2019-12-05T05:42:45.490 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.464+0000 7fa0a6ffd700 1 --2- 172.21.15.42:0/1057473590 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa0a807e720 0x7fa0a807eb50 secure :-1 s=THROTTLE_DONE pgs=271 cs=0 l=1 rx=0x7fa098008790 tx=0x7fa09800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:45.490 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.464+0000 7fa0a6ffd700 1 --2- 172.21.15.42:0/1057473590 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa0a807e720 0x7fa0a807eb50 secure :-1 s=THROTTLE_DONE pgs=271 cs=0 l=1 rx=0x7fa098008790 tx=0x7fa09800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:45.491 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.464+0000 7fa0a67fc700 1 -- 172.21.15.42:0/1057473590 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa098008ad0 con 0x7fa0a807e720 2019-12-05T05:42:45.491 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.464+0000 7fa0a67fc700 1 -- 172.21.15.42:0/1057473590 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa0980118d0 con 0x7fa0a807e720 2019-12-05T05:42:45.491 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.464+0000 7fa0ad44f700 1 -- 172.21.15.42:0/1057473590 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa0a807e720 msgr2=0x7fa0a807eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:42:45.491 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.464+0000 7fa0ad44f700 1 --2- 172.21.15.42:0/1057473590 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa0a807e720 0x7fa0a807eb50 secure :-1 s=READY pgs=271 cs=0 l=1 rx=0x7fa098008790 tx=0x7fa09800fc70).stop 2019-12-05T05:42:45.491 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.464+0000 7fa0ad44f700 1 -- 172.21.15.42:0/1057473590 shutdown_connections 2019-12-05T05:42:45.491 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.464+0000 7fa0ad44f700 1 --2- 172.21.15.42:0/1057473590 >> v2:172.21.15.40:3300/0 conn(0x7fa0a80a7740 0x7fa0a808df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:42:45.491 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.464+0000 7fa0ad44f700 1 --2- 172.21.15.42:0/1057473590 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa0a807e720 0x7fa0a807eb50 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:42:45.492 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.464+0000 7fa0ad44f700 1 -- 172.21.15.42:0/1057473590 shutdown_connections 2019-12-05T05:42:45.492 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.464+0000 7fa0ad44f700 1 -- 172.21.15.42:0/1057473590 wait complete. 2019-12-05T05:42:45.492 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.464+0000 7fa0ad44f700 1 -- 172.21.15.42:0/1057473590 >> 172.21.15.42:0/1057473590 conn(0x7fa0a80aabe0 msgr2=0x7fa0a80a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:42:45.492 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.468+0000 7fa0ad44f700 1 Processor -- start 2019-12-05T05:42:45.493 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.468+0000 7fa0ad44f700 1 -- start start 2019-12-05T05:42:45.493 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.468+0000 7fa0ad44f700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fa0a80a3ad0 0x7fa0a812aee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:42:45.493 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.468+0000 7fa0ad44f700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa0a811bee0 0x7fa0a811a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:42:45.493 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.468+0000 7fa0ad44f700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0a807e050 con 0x7fa0a81192d0 2019-12-05T05:42:45.493 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.468+0000 7fa0ad44f700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0a811b750 con 0x7fa0a80a3ad0 2019-12-05T05:42:45.493 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.468+0000 7fa0ad44f700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0a811aa70 con 0x7fa0a811bee0 2019-12-05T05:42:45.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.468+0000 7fa0a77fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fa0a80a3ad0 0x7fa0a812aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:42:45.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.468+0000 7fa0a6ffd700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa0a811bee0 0x7fa0a811a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:42:45.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.468+0000 7fa0a6ffd700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa0a811bee0 0x7fa0a811a2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:39804/0 (socket says 172.21.15.42:39804) 2019-12-05T05:42:45.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.468+0000 7fa0a6ffd700 1 -- 172.21.15.42:0/2996708660 learned_addr learned my addr 172.21.15.42:0/2996708660 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:42:45.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.468+0000 7fa0a6ffd700 1 -- 172.21.15.42:0/2996708660 >> v2:172.21.15.42:3300/0 conn(0x7fa0a80a3ad0 msgr2=0x7fa0a812aee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:42:45.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.468+0000 7fa0a6ffd700 1 --2- 172.21.15.42:0/2996708660 >> v2:172.21.15.42:3300/0 conn(0x7fa0a80a3ad0 0x7fa0a812aee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:42:45.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.468+0000 7fa0a6ffd700 1 -- 172.21.15.42:0/2996708660 >> v1:172.21.15.40:6789/0 conn(0x7fa0a81192d0 legacy=0x7fa0a807d850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:42:45.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.468+0000 7fa0a6ffd700 1 -- 172.21.15.42:0/2996708660 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0a8119700 con 0x7fa0a811bee0 2019-12-05T05:42:45.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.468+0000 7fa09ffff700 1 -- 172.21.15.42:0/2996708660 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1241404415 0 0) 0x7fa0a807e050 con 0x7fa0a81192d0 2019-12-05T05:42:45.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.468+0000 7fa0a6ffd700 1 --2- 172.21.15.42:0/2996708660 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa0a811bee0 0x7fa0a811a2f0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fa098008790 tx=0x7fa098011f10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:45.496 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.468+0000 7fa0a6ffd700 1 --2- 172.21.15.42:0/2996708660 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa0a811bee0 0x7fa0a811a2f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fa098008790 tx=0x7fa098011f10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:45.496 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.468+0000 7fa0a6ffd700 1 --2- 172.21.15.42:0/2996708660 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa0a811bee0 0x7fa0a811a2f0 secure :-1 s=READY pgs=272 cs=0 l=1 rx=0x7fa098008790 tx=0x7fa098011f10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:42:45.496 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.472+0000 7fa0a6ffd700 1 --2- 172.21.15.42:0/2996708660 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa0a811bee0 0x7fa0a811a2f0 secure :-1 s=THROTTLE_DONE pgs=272 cs=0 l=1 rx=0x7fa098008790 tx=0x7fa098011f10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:45.496 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.472+0000 7fa09ffff700 1 -- 172.21.15.42:0/2996708660 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa098007340 con 0x7fa0a811bee0 2019-12-05T05:42:45.496 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.472+0000 7fa0a6ffd700 1 --2- 172.21.15.42:0/2996708660 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa0a811bee0 0x7fa0a811a2f0 secure :-1 s=THROTTLE_DONE pgs=272 cs=0 l=1 rx=0x7fa098008790 tx=0x7fa098011f10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:45.496 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.472+0000 7fa0a6ffd700 1 --2- 172.21.15.42:0/2996708660 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa0a811bee0 0x7fa0a811a2f0 secure :-1 s=THROTTLE_DONE pgs=272 cs=0 l=1 rx=0x7fa098008790 tx=0x7fa098011f10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:45.496 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.472+0000 7fa09ffff700 1 -- 172.21.15.42:0/2996708660 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0980114f0 con 0x7fa0a811bee0 2019-12-05T05:42:45.497 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.472+0000 7fa0ad44f700 1 -- 172.21.15.42:0/2996708660 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0a81358f0 con 0x7fa0a811bee0 2019-12-05T05:42:45.497 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.472+0000 7fa09ffff700 1 -- 172.21.15.42:0/2996708660 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa098005540 con 0x7fa0a811bee0 2019-12-05T05:42:45.497 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.472+0000 7fa0ad44f700 1 -- 172.21.15.42:0/2996708660 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0a8135ea0 con 0x7fa0a811bee0 2019-12-05T05:42:45.497 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.472+0000 7fa0a6ffd700 1 --2- 172.21.15.42:0/2996708660 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa0a811bee0 0x7fa0a811a2f0 secure :-1 s=THROTTLE_DONE pgs=272 cs=0 l=1 rx=0x7fa098008790 tx=0x7fa098011f10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:45.497 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.472+0000 7fa09ffff700 1 -- 172.21.15.42:0/2996708660 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fa098005820 con 0x7fa0a811bee0 2019-12-05T05:42:45.497 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.472+0000 7fa0a6ffd700 1 --2- 172.21.15.42:0/2996708660 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa0a811bee0 0x7fa0a811a2f0 secure :-1 s=THROTTLE_DONE pgs=272 cs=0 l=1 rx=0x7fa098008790 tx=0x7fa098011f10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:45.497 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.472+0000 7fa09ffff700 1 --2- 172.21.15.42:0/2996708660 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa08c03acc0 0x7fa08c03d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:42:45.503 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.472+0000 7fa09ffff700 1 -- 172.21.15.42:0/2996708660 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 8267+0+0 (secure 0 0 0) 0x7fa09808fab0 con 0x7fa0a811bee0 2019-12-05T05:42:45.503 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.472+0000 7fa0a77fe700 1 --2- 172.21.15.42:0/2996708660 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa08c03acc0 0x7fa08c03d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:42:45.503 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.472+0000 7fa0a77fe700 1 --2- 172.21.15.42:0/2996708660 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa08c03acc0 0x7fa08c03d1d0 crc :-1 s=READY pgs=563 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:42:45.518 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.476+0000 7fa0ad44f700 1 -- 172.21.15.42:0/2996708660 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0a805d110 con 0x7fa0a811bee0 2019-12-05T05:42:45.519 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.480+0000 7fa0a6ffd700 1 --2- 172.21.15.42:0/2996708660 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa0a811bee0 0x7fa0a811a2f0 secure :-1 s=THROTTLE_DONE pgs=272 cs=0 l=1 rx=0x7fa098008790 tx=0x7fa098011f10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:45.519 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.480+0000 7fa09ffff700 1 -- 172.21.15.42:0/2996708660 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fa098005b30 con 0x7fa0a811bee0 2019-12-05T05:42:45.650 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.644+0000 7fa0ad44f700 1 -- 172.21.15.42:0/2996708660 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa0a80a9ed0 con 0x7fa0a811bee0 2019-12-05T05:42:45.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.644+0000 7fa0a6ffd700 1 --2- 172.21.15.42:0/2996708660 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa0a811bee0 0x7fa0a811a2f0 secure :-1 s=THROTTLE_DONE pgs=272 cs=0 l=1 rx=0x7fa098008790 tx=0x7fa098011f10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:45.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.644+0000 7fa09ffff700 1 -- 172.21.15.42:0/2996708660 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 159 v159) v1 ==== 95+0+1774 (secure 0 0 0) 0x7fa098077920 con 0x7fa0a811bee0 2019-12-05T05:42:45.651 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 159 2019-12-05T05:42:45.651 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:42:45.651 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":159,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":159,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:42:43.929040+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7650},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7650":{"gid":7650,"name":"b","rank":0,"incarnation":159,"state":"up:replay","state_seq":2,"addr":"172.21.15.40:6833/1029013337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1029013337},{"type":"v1","addr":"172.21.15.40:6833","nonce":1029013337}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:42:45.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.648+0000 7fa0ad44f700 1 -- 172.21.15.42:0/2996708660 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa08c03acc0 msgr2=0x7fa08c03d1d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:42:45.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.648+0000 7fa0ad44f700 1 --2- 172.21.15.42:0/2996708660 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa08c03acc0 0x7fa08c03d1d0 crc :-1 s=READY pgs=563 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:42:45.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.648+0000 7fa0ad44f700 1 -- 172.21.15.42:0/2996708660 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa0a811bee0 msgr2=0x7fa0a811a2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:42:45.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.648+0000 7fa0ad44f700 1 --2- 172.21.15.42:0/2996708660 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa0a811bee0 0x7fa0a811a2f0 secure :-1 s=READY pgs=272 cs=0 l=1 rx=0x7fa098008790 tx=0x7fa098011f10).stop 2019-12-05T05:42:45.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.648+0000 7fa0ad44f700 1 -- 172.21.15.42:0/2996708660 shutdown_connections 2019-12-05T05:42:45.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.648+0000 7fa0ad44f700 1 --2- 172.21.15.42:0/2996708660 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa08c03acc0 0x7fa08c03d1d0 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:42:45.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.648+0000 7fa0ad44f700 1 --2- 172.21.15.42:0/2996708660 >> v2:172.21.15.42:3300/0 conn(0x7fa0a80a3ad0 0x7fa0a812aee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:42:45.654 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.648+0000 7fa0ad44f700 1 --2- 172.21.15.42:0/2996708660 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa0a811bee0 0x7fa0a811a2f0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:42:45.654 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.648+0000 7fa0ad44f700 1 -- 172.21.15.42:0/2996708660 shutdown_connections 2019-12-05T05:42:45.654 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.648+0000 7fa0ad44f700 1 -- 172.21.15.42:0/2996708660 wait complete. 2019-12-05T05:42:45.654 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.648+0000 7fa0ad44f700 1 -- 172.21.15.42:0/2996708660 >> 172.21.15.42:0/2996708660 conn(0x7fa0a80a96e0 msgr2=0x7fa0a811c470 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:42:45.663 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a down, removed from mdsmap 2019-12-05T05:42:45.663 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2019-12-05T05:42:45.663 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:42:45.764 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.752+0000 7f2e8157c700 1 Processor -- start 2019-12-05T05:42:45.764 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.752+0000 7f2e8157c700 1 -- start start 2019-12-05T05:42:45.764 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.752+0000 7f2e8157c700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2e7c0a7740 0x7f2e7c08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:42:45.765 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.752+0000 7f2e8157c700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2e7c07e720 0x7f2e7c07eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:42:45.765 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.752+0000 7f2e8157c700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e7c0a0710 con 0x7f2e7c09f2e0 2019-12-05T05:42:45.765 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.752+0000 7f2e8157c700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e7c0a0b00 con 0x7f2e7c07e720 2019-12-05T05:42:45.765 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.752+0000 7f2e8157c700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e7c08e5b0 con 0x7f2e7c0a7740 2019-12-05T05:42:45.765 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.752+0000 7f2e7b7fe700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2e7c0a7740 0x7f2e7c08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:42:45.765 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.752+0000 7f2e7bfff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f2e7c09f2e0 0x7f2e7c09f710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:39388/0 (socket says 172.21.15.42:39388) 2019-12-05T05:42:45.766 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.752+0000 7f2e7bfff700 1 -- 172.21.15.42:0/1650605722 learned_addr learned my addr 172.21.15.42:0/1650605722 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:42:45.766 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.752+0000 7f2e7affd700 1 --2- 172.21.15.42:0/1650605722 >> v2:172.21.15.40:3300/0 conn(0x7f2e7c07e720 0x7f2e7c07eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:42:45.766 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.752+0000 7f2e7b7fe700 1 -- 172.21.15.42:0/1650605722 >> v2:172.21.15.40:3300/0 conn(0x7f2e7c07e720 msgr2=0x7f2e7c07eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:42:45.766 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.752+0000 7f2e7b7fe700 1 --2- 172.21.15.42:0/1650605722 >> v2:172.21.15.40:3300/0 conn(0x7f2e7c07e720 0x7f2e7c07eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:42:45.766 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.752+0000 7f2e7a7fc700 1 -- 172.21.15.42:0/1650605722 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3593988576 0 0) 0x7f2e7c0a0710 con 0x7f2e7c09f2e0 2019-12-05T05:42:45.766 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.756+0000 7f2e7b7fe700 1 -- 172.21.15.42:0/1650605722 >> v1:172.21.15.42:6789/0 conn(0x7f2e7c09f2e0 legacy=0x7f2e7c09f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:42:45.766 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.756+0000 7f2e7b7fe700 1 -- 172.21.15.42:0/1650605722 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e7c07f0a0 con 0x7f2e7c0a7740 2019-12-05T05:42:45.767 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.756+0000 7f2e7b7fe700 1 --2- 172.21.15.42:0/1650605722 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2e7c0a7740 0x7f2e7c08df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f2e6c008790 tx=0x7f2e6c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:45.767 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.756+0000 7f2e7b7fe700 1 --2- 172.21.15.42:0/1650605722 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2e7c0a7740 0x7f2e7c08df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f2e6c008790 tx=0x7f2e6c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:45.767 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.756+0000 7f2e7b7fe700 1 --2- 172.21.15.42:0/1650605722 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2e7c0a7740 0x7f2e7c08df80 secure :-1 s=READY pgs=273 cs=0 l=1 rx=0x7f2e6c008790 tx=0x7f2e6c00fc70).ready entity=mon.2 client_cookie=9ed7d489dc287c5e server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:42:45.767 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.756+0000 7f2e7b7fe700 1 --2- 172.21.15.42:0/1650605722 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2e7c0a7740 0x7f2e7c08df80 secure :-1 s=THROTTLE_DONE pgs=273 cs=0 l=1 rx=0x7f2e6c008790 tx=0x7f2e6c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:45.767 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.756+0000 7f2e7a7fc700 1 -- 172.21.15.42:0/1650605722 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2e6c0112f0 con 0x7f2e7c0a7740 2019-12-05T05:42:45.767 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.756+0000 7f2e7b7fe700 1 --2- 172.21.15.42:0/1650605722 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2e7c0a7740 0x7f2e7c08df80 secure :-1 s=THROTTLE_DONE pgs=273 cs=0 l=1 rx=0x7f2e6c008790 tx=0x7f2e6c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:45.767 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.756+0000 7f2e7b7fe700 1 --2- 172.21.15.42:0/1650605722 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2e7c0a7740 0x7f2e7c08df80 secure :-1 s=THROTTLE_DONE pgs=273 cs=0 l=1 rx=0x7f2e6c008790 tx=0x7f2e6c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:45.768 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.756+0000 7f2e7a7fc700 1 -- 172.21.15.42:0/1650605722 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e6c008ad0 con 0x7f2e7c0a7740 2019-12-05T05:42:45.768 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.756+0000 7f2e7a7fc700 1 -- 172.21.15.42:0/1650605722 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2e6c0118d0 con 0x7f2e7c0a7740 2019-12-05T05:42:45.768 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.756+0000 7f2e8157c700 1 -- 172.21.15.42:0/1650605722 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2e7c0a7740 msgr2=0x7f2e7c08df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:42:45.768 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.756+0000 7f2e8157c700 1 --2- 172.21.15.42:0/1650605722 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2e7c0a7740 0x7f2e7c08df80 secure :-1 s=READY pgs=273 cs=0 l=1 rx=0x7f2e6c008790 tx=0x7f2e6c00fc70).stop 2019-12-05T05:42:45.768 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.756+0000 7f2e8157c700 1 -- 172.21.15.42:0/1650605722 shutdown_connections 2019-12-05T05:42:45.768 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.756+0000 7f2e8157c700 1 --2- 172.21.15.42:0/1650605722 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2e7c0a7740 0x7f2e7c08df80 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:42:45.768 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.756+0000 7f2e8157c700 1 --2- 172.21.15.42:0/1650605722 >> v2:172.21.15.40:3300/0 conn(0x7f2e7c07e720 0x7f2e7c07eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:42:45.769 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.756+0000 7f2e8157c700 1 -- 172.21.15.42:0/1650605722 shutdown_connections 2019-12-05T05:42:45.769 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.756+0000 7f2e8157c700 1 -- 172.21.15.42:0/1650605722 wait complete. 2019-12-05T05:42:45.769 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.756+0000 7f2e8157c700 1 -- 172.21.15.42:0/1650605722 >> 172.21.15.42:0/1650605722 conn(0x7f2e7c0aabe0 msgr2=0x7f2e7c0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:42:45.772 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.760+0000 7f2e8157c700 1 Processor -- start 2019-12-05T05:42:45.772 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.760+0000 7f2e8157c700 1 -- start start 2019-12-05T05:42:45.772 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.760+0000 7f2e8157c700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2e7c143aa0 0x7f2e7c144160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:42:45.772 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.760+0000 7f2e8157c700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e7c144b70 con 0x7f2e7c0a1d10 2019-12-05T05:42:45.773 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.760+0000 7f2e8157c700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e7c12ffe0 con 0x7f2e7c116500 2019-12-05T05:42:45.773 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.760+0000 7f2e8157c700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e7c05a2c0 con 0x7f2e7c143aa0 2019-12-05T05:42:45.773 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.760+0000 7f2e7affd700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2e7c143aa0 0x7f2e7c144160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:42:45.773 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.760+0000 7f2e7bfff700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f2e7c0a1d10 0x7f2e7c137430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:48834/0 (socket says 172.21.15.42:48834) 2019-12-05T05:42:45.773 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.760+0000 7f2e7affd700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2e7c143aa0 0x7f2e7c144160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:39820/0 (socket says 172.21.15.42:39820) 2019-12-05T05:42:45.773 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.760+0000 7f2e7b7fe700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f2e7c116500 0x7f2e7c130700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:39390/0 (socket says 172.21.15.42:39390) 2019-12-05T05:42:45.774 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.760+0000 7f2e7affd700 1 -- 172.21.15.42:0/3887591664 learned_addr learned my addr 172.21.15.42:0/3887591664 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:42:45.774 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.760+0000 7f2e63fff700 1 -- 172.21.15.42:0/3887591664 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1676381513 0 0) 0x7f2e7c12ffe0 con 0x7f2e7c116500 2019-12-05T05:42:45.774 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.760+0000 7f2e63fff700 1 -- 172.21.15.42:0/3887591664 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e5c0030b0 con 0x7f2e7c116500 2019-12-05T05:42:45.774 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.760+0000 7f2e63fff700 1 -- 172.21.15.42:0/3887591664 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3537665083 0 0) 0x7f2e7c144b70 con 0x7f2e7c0a1d10 2019-12-05T05:42:45.774 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.760+0000 7f2e7affd700 1 -- 172.21.15.42:0/3887591664 >> v1:172.21.15.42:6789/0 conn(0x7f2e7c116500 legacy=0x7f2e7c130700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:42:45.774 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.760+0000 7f2e7affd700 1 -- 172.21.15.42:0/3887591664 >> v1:172.21.15.40:6789/0 conn(0x7f2e7c0a1d10 legacy=0x7f2e7c137430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:42:45.775 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.760+0000 7f2e7affd700 1 -- 172.21.15.42:0/3887591664 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e7c0a2140 con 0x7f2e7c143aa0 2019-12-05T05:42:45.775 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.760+0000 7f2e7affd700 1 --2- 172.21.15.42:0/3887591664 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2e7c143aa0 0x7f2e7c144160 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f2e70002f30 tx=0x7f2e70010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:45.775 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.764+0000 7f2e7affd700 1 --2- 172.21.15.42:0/3887591664 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2e7c143aa0 0x7f2e7c144160 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f2e70002f30 tx=0x7f2e70010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:45.775 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.764+0000 7f2e7affd700 1 --2- 172.21.15.42:0/3887591664 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2e7c143aa0 0x7f2e7c144160 secure :-1 s=READY pgs=274 cs=0 l=1 rx=0x7f2e70002f30 tx=0x7f2e70010b60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:42:45.775 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.764+0000 7f2e7affd700 1 --2- 172.21.15.42:0/3887591664 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2e7c143aa0 0x7f2e7c144160 secure :-1 s=THROTTLE_DONE pgs=274 cs=0 l=1 rx=0x7f2e70002f30 tx=0x7f2e70010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:45.775 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.764+0000 7f2e63fff700 1 -- 172.21.15.42:0/3887591664 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2e70012610 con 0x7f2e7c143aa0 2019-12-05T05:42:45.776 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.764+0000 7f2e7affd700 1 --2- 172.21.15.42:0/3887591664 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2e7c143aa0 0x7f2e7c144160 secure :-1 s=THROTTLE_DONE pgs=274 cs=0 l=1 rx=0x7f2e70002f30 tx=0x7f2e70010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:45.776 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.764+0000 7f2e63fff700 1 -- 172.21.15.42:0/3887591664 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e70007150 con 0x7f2e7c143aa0 2019-12-05T05:42:45.776 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.764+0000 7f2e7affd700 1 --2- 172.21.15.42:0/3887591664 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2e7c143aa0 0x7f2e7c144160 secure :-1 s=THROTTLE_DONE pgs=274 cs=0 l=1 rx=0x7f2e70002f30 tx=0x7f2e70010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:45.776 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.764+0000 7f2e8157c700 1 -- 172.21.15.42:0/3887591664 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e7c0a2410 con 0x7f2e7c143aa0 2019-12-05T05:42:45.776 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.764+0000 7f2e63fff700 1 -- 172.21.15.42:0/3887591664 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2e70012bf0 con 0x7f2e7c143aa0 2019-12-05T05:42:45.776 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.764+0000 7f2e8157c700 1 -- 172.21.15.42:0/3887591664 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e7c138c40 con 0x7f2e7c143aa0 2019-12-05T05:42:45.776 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.764+0000 7f2e7affd700 1 --2- 172.21.15.42:0/3887591664 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2e7c143aa0 0x7f2e7c144160 secure :-1 s=THROTTLE_DONE pgs=274 cs=0 l=1 rx=0x7f2e70002f30 tx=0x7f2e70010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:45.777 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.764+0000 7f2e63fff700 1 -- 172.21.15.42:0/3887591664 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f2e70005540 con 0x7f2e7c143aa0 2019-12-05T05:42:45.777 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.764+0000 7f2e7affd700 1 --2- 172.21.15.42:0/3887591664 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2e7c143aa0 0x7f2e7c144160 secure :-1 s=THROTTLE_DONE pgs=274 cs=0 l=1 rx=0x7f2e70002f30 tx=0x7f2e70010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:45.777 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.764+0000 7f2e63fff700 1 --2- 172.21.15.42:0/3887591664 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2e5c03b180 0x7f2e5c03d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:42:45.777 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.764+0000 7f2e63fff700 1 -- 172.21.15.42:0/3887591664 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 8267+0+0 (secure 0 0 0) 0x7f2e7004b0c0 con 0x7f2e7c143aa0 2019-12-05T05:42:45.777 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.764+0000 7f2e7b7fe700 1 --2- 172.21.15.42:0/3887591664 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2e5c03b180 0x7f2e5c03d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:42:45.777 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.764+0000 7f2e7b7fe700 1 --2- 172.21.15.42:0/3887591664 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2e5c03b180 0x7f2e5c03d690 crc :-1 s=READY pgs=564 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:42:45.777 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.768+0000 7f2e8157c700 1 -- 172.21.15.42:0/3887591664 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e7c05d110 con 0x7f2e7c143aa0 2019-12-05T05:42:45.778 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.772+0000 7f2e7affd700 1 --2- 172.21.15.42:0/3887591664 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2e7c143aa0 0x7f2e7c144160 secure :-1 s=THROTTLE_DONE pgs=274 cs=0 l=1 rx=0x7f2e70002f30 tx=0x7f2e70010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:45.779 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.772+0000 7f2e63fff700 1 -- 172.21.15.42:0/3887591664 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f2e70032140 con 0x7f2e7c143aa0 2019-12-05T05:42:45.940 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.936+0000 7f2e8157c700 1 -- 172.21.15.42:0/3887591664 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2e7c05b8c0 con 0x7f2e7c143aa0 2019-12-05T05:42:45.941 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.936+0000 7f2e7affd700 1 --2- 172.21.15.42:0/3887591664 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2e7c143aa0 0x7f2e7c144160 secure :-1 s=THROTTLE_DONE pgs=274 cs=0 l=1 rx=0x7f2e70002f30 tx=0x7f2e70010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:45.941 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.936+0000 7f2e63fff700 1 -- 172.21.15.42:0/3887591664 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 159 v159) v1 ==== 95+0+1774 (secure 0 0 0) 0x7f2e700323a0 con 0x7f2e7c143aa0 2019-12-05T05:42:45.941 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 159 2019-12-05T05:42:45.941 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:42:45.942 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":159,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":159,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:42:43.929040+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7650},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7650":{"gid":7650,"name":"b","rank":0,"incarnation":159,"state":"up:replay","state_seq":2,"addr":"172.21.15.40:6833/1029013337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1029013337},{"type":"v1","addr":"172.21.15.40:6833","nonce":1029013337}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:42:45.943 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.936+0000 7f2e8157c700 1 -- 172.21.15.42:0/3887591664 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2e5c03b180 msgr2=0x7f2e5c03d690 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:42:45.944 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.936+0000 7f2e8157c700 1 --2- 172.21.15.42:0/3887591664 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2e5c03b180 0x7f2e5c03d690 crc :-1 s=READY pgs=564 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:42:45.944 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.936+0000 7f2e8157c700 1 -- 172.21.15.42:0/3887591664 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2e7c143aa0 msgr2=0x7f2e7c144160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:42:45.944 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.936+0000 7f2e8157c700 1 --2- 172.21.15.42:0/3887591664 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2e7c143aa0 0x7f2e7c144160 secure :-1 s=READY pgs=274 cs=0 l=1 rx=0x7f2e70002f30 tx=0x7f2e70010b60).stop 2019-12-05T05:42:45.944 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.940+0000 7f2e8157c700 1 -- 172.21.15.42:0/3887591664 shutdown_connections 2019-12-05T05:42:45.944 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.940+0000 7f2e8157c700 1 --2- 172.21.15.42:0/3887591664 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2e5c03b180 0x7f2e5c03d690 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:42:45.944 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.940+0000 7f2e8157c700 1 --2- 172.21.15.42:0/3887591664 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2e7c143aa0 0x7f2e7c144160 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:42:45.945 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.940+0000 7f2e8157c700 1 -- 172.21.15.42:0/3887591664 shutdown_connections 2019-12-05T05:42:45.945 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.940+0000 7f2e8157c700 1 -- 172.21.15.42:0/3887591664 wait complete. 2019-12-05T05:42:45.945 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:45.940+0000 7f2e8157c700 1 -- 172.21.15.42:0/3887591664 >> 172.21.15.42:0/3887591664 conn(0x7f2e7c0aabe0 msgr2=0x7f2e7c0a9790 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:42:45.954 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 159, "feature_flags": { "enable_multiple": false, "ever_enabled_multiple": false }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2019-12-05T05:03:35.604738+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 159, "ever_allowed_features": 2, "explicitly_allowed_features": 2, "failed": [], "flags": 18, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_7650": { "addr": "172.21.15.40:6833/1029013337", "addrs": { "addrvec": [ { "addr": "172.21.15.40:6832", "nonce": 1029013337, "type": "v2" }, { "addr": "172.21.15.40:6833", "nonce": 1029013337, "type": "v1" } ] }, "export_targets": [], "features": 4540138292836696063, "flags": 0, "gid": 7650, "incarnation": 159, "name": "b", "rank": 0, "state": "up:replay", "state_seq": 2 } }, "last_failure": 0, "last_failure_osd_epoch": 75, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 2, "min_compat_client": "0 (unknown)", "modified": "2019-12-05T05:42:43.929040+0000", "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7650 } } } ], "standbys": [] } 2019-12-05T05:42:47.955 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:42:48.061 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:48.048+0000 7f4712346700 1 Processor -- start 2019-12-05T05:42:48.061 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:48.048+0000 7f4712346700 1 -- start start 2019-12-05T05:42:48.061 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:48.052+0000 7f4712346700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f470c0a7740 0x7f470c08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:42:48.062 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:48.052+0000 7f4712346700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f470c07e720 0x7f470c07eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:42:48.062 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:48.052+0000 7f4712346700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f470c0a0710 con 0x7f470c09f2e0 2019-12-05T05:42:48.062 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:48.052+0000 7f4712346700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f470c0a0b00 con 0x7f470c0a7740 2019-12-05T05:42:48.063 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:48.052+0000 7f4712346700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f470c08e5b0 con 0x7f470c07e720 2019-12-05T05:42:48.063 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:48.052+0000 7f470bfff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f470c07e720 0x7f470c07eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:42:48.063 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:48.052+0000 7f47108e3700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f470c0a7740 0x7f470c08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:42:48.063 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:48.052+0000 7f470bfff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f470c07e720 0x7f470c07eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:39368/0 (socket says 172.21.15.42:39368) 2019-12-05T05:42:48.063 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:48.052+0000 7f470bfff700 1 -- 172.21.15.42:0/2024919631 learned_addr learned my addr 172.21.15.42:0/2024919631 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:42:48.064 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:48.052+0000 7f47110e4700 1 --1- 172.21.15.42:0/2024919631 >> v1:172.21.15.40:6789/0 conn(0x7f470c09f2e0 0x7f470c09f710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:48848/0 (socket says 172.21.15.42:48848) 2019-12-05T05:42:48.064 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:48.052+0000 7f470bfff700 1 -- 172.21.15.42:0/2024919631 >> v2:172.21.15.40:3300/0 conn(0x7f470c0a7740 msgr2=0x7f470c08df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:42:48.064 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:48.052+0000 7f470bfff700 1 --2- 172.21.15.42:0/2024919631 >> v2:172.21.15.40:3300/0 conn(0x7f470c0a7740 0x7f470c08df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:42:48.064 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:48.052+0000 7f470bfff700 1 -- 172.21.15.42:0/2024919631 >> v1:172.21.15.40:6789/0 conn(0x7f470c09f2e0 legacy=0x7f470c09f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:42:48.065 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:48.052+0000 7f470bfff700 1 -- 172.21.15.42:0/2024919631 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f470c07f0a0 con 0x7f470c07e720 2019-12-05T05:42:48.065 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:48.052+0000 7f470bfff700 1 --2- 172.21.15.42:0/2024919631 >> v2:172.21.15.42:3300/0 conn(0x7f470c07e720 0x7f470c07eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f4700002f30 tx=0x7f470000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:48.065 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:48.052+0000 7f470bfff700 1 --2- 172.21.15.42:0/2024919631 >> v2:172.21.15.42:3300/0 conn(0x7f470c07e720 0x7f470c07eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f4700002f30 tx=0x7f470000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:48.065 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:48.052+0000 7f470bfff700 1 --2- 172.21.15.42:0/2024919631 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f470c07e720 0x7f470c07eb50 secure :-1 s=READY pgs=1308 cs=0 l=1 rx=0x7f4700002f30 tx=0x7f470000fd30).ready entity=mon.1 client_cookie=11adaa1c7a401586 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:42:48.066 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:48.052+0000 7f470bfff700 1 --2- 172.21.15.42:0/2024919631 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f470c07e720 0x7f470c07eb50 secure :-1 s=THROTTLE_DONE pgs=1308 cs=0 l=1 rx=0x7f4700002f30 tx=0x7f470000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:48.066 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:48.052+0000 7f470b7fe700 1 -- 172.21.15.42:0/2024919631 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4700011370 con 0x7f470c07e720 2019-12-05T05:42:48.066 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:48.052+0000 7f470bfff700 1 --2- 172.21.15.42:0/2024919631 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f470c07e720 0x7f470c07eb50 secure :-1 s=THROTTLE_DONE pgs=1308 cs=0 l=1 rx=0x7f4700002f30 tx=0x7f470000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:48.066 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:48.052+0000 7f470bfff700 1 --2- 172.21.15.42:0/2024919631 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f470c07e720 0x7f470c07eb50 secure :-1 s=THROTTLE_DONE pgs=1308 cs=0 l=1 rx=0x7f4700002f30 tx=0x7f470000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:48.066 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:48.052+0000 7f470b7fe700 1 -- 172.21.15.42:0/2024919631 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4700008bb0 con 0x7f470c07e720 2019-12-05T05:42:48.067 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:48.052+0000 7f470b7fe700 1 -- 172.21.15.42:0/2024919631 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4700011950 con 0x7f470c07e720 2019-12-05T05:42:48.067 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:48.052+0000 7f4712346700 1 -- 172.21.15.42:0/2024919631 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f470c07e720 msgr2=0x7f470c07eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:42:48.067 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:48.052+0000 7f4712346700 1 --2- 172.21.15.42:0/2024919631 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f470c07e720 0x7f470c07eb50 secure :-1 s=READY pgs=1308 cs=0 l=1 rx=0x7f4700002f30 tx=0x7f470000fd30).stop 2019-12-05T05:42:48.067 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:48.052+0000 7f4712346700 1 -- 172.21.15.42:0/2024919631 shutdown_connections 2019-12-05T05:42:48.068 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:48.052+0000 7f4712346700 1 --2- 172.21.15.42:0/2024919631 >> v2:172.21.15.40:3300/0 conn(0x7f470c0a7740 0x7f470c08df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:42:48.068 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:48.052+0000 7f4712346700 1 --2- 172.21.15.42:0/2024919631 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f470c07e720 0x7f470c07eb50 unknown :-1 s=CLOSED pgs=1308 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:42:48.068 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:48.052+0000 7f4712346700 1 -- 172.21.15.42:0/2024919631 shutdown_connections 2019-12-05T05:42:48.069 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:48.052+0000 7f4712346700 1 -- 172.21.15.42:0/2024919631 wait complete. 2019-12-05T05:42:48.069 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:48.052+0000 7f4712346700 1 -- 172.21.15.42:0/2024919631 >> 172.21.15.42:0/2024919631 conn(0x7f470c0aabe0 msgr2=0x7f470c0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:42:48.074 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:48.056+0000 7f4712346700 1 Processor -- start 2019-12-05T05:42:48.075 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:48.056+0000 7f4712346700 1 -- start start 2019-12-05T05:42:48.075 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:48.060+0000 7f4712346700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f470c1156c0 0x7f470c077d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:42:48.075 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:48.060+0000 7f4712346700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f470c112290 con 0x7f470c14ea60 2019-12-05T05:42:48.075 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:48.060+0000 7f4712346700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f470c078520 con 0x7f470c14e630 2019-12-05T05:42:48.076 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:48.060+0000 7f4712346700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f470c078290 con 0x7f470c1156c0 2019-12-05T05:42:48.076 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:48.060+0000 7f470bfff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f470c14e630 0x7f470c079330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:39412/0 (socket says 172.21.15.42:39412) 2019-12-05T05:42:48.076 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:48.060+0000 7f470bfff700 1 -- 172.21.15.42:0/129210727 learned_addr learned my addr 172.21.15.42:0/129210727 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:42:48.076 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:48.060+0000 7f47108e3700 1 --2- 172.21.15.42:0/129210727 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f470c1156c0 0x7f470c077d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:42:48.077 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:48.060+0000 7f47097fa700 1 -- 172.21.15.42:0/129210727 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1853085431 0 0) 0x7f470c112290 con 0x7f470c14ea60 2019-12-05T05:42:48.077 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:48.060+0000 7f47097fa700 1 -- 172.21.15.42:0/129210727 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46ec0030b0 con 0x7f470c14ea60 2019-12-05T05:42:48.077 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:48.060+0000 7f47097fa700 1 -- 172.21.15.42:0/129210727 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2108211362 0 0) 0x7f470c078520 con 0x7f470c14e630 2019-12-05T05:42:48.077 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:48.060+0000 7f47097fa700 1 -- 172.21.15.42:0/129210727 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f470c112290 con 0x7f470c14e630 2019-12-05T05:42:48.077 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:48.060+0000 7f47108e3700 1 -- 172.21.15.42:0/129210727 >> v1:172.21.15.42:6789/0 conn(0x7f470c14e630 legacy=0x7f470c079330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:42:48.078 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:48.060+0000 7f47108e3700 1 -- 172.21.15.42:0/129210727 >> v1:172.21.15.40:6789/0 conn(0x7f470c14ea60 legacy=0x7f470c117840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:42:48.078 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:48.060+0000 7f47108e3700 1 -- 172.21.15.42:0/129210727 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f470c118040 con 0x7f470c1156c0 2019-12-05T05:42:48.078 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:48.060+0000 7f47108e3700 1 --2- 172.21.15.42:0/129210727 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f470c1156c0 0x7f470c077d40 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f46fc002f30 tx=0x7f46fc010960).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:48.079 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:48.060+0000 7f47108e3700 1 --2- 172.21.15.42:0/129210727 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f470c1156c0 0x7f470c077d40 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f46fc002f30 tx=0x7f46fc010960).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:48.079 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:48.060+0000 7f47108e3700 1 --2- 172.21.15.42:0/129210727 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f470c1156c0 0x7f470c077d40 secure :-1 s=READY pgs=275 cs=0 l=1 rx=0x7f46fc002f30 tx=0x7f46fc010960).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:42:48.079 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:48.060+0000 7f47108e3700 1 --2- 172.21.15.42:0/129210727 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f470c1156c0 0x7f470c077d40 secure :-1 s=THROTTLE_DONE pgs=275 cs=0 l=1 rx=0x7f46fc002f30 tx=0x7f46fc010960).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:48.079 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:48.060+0000 7f47097fa700 1 -- 172.21.15.42:0/129210727 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f46fc005920 con 0x7f470c1156c0 2019-12-05T05:42:48.080 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:48.060+0000 7f47108e3700 1 --2- 172.21.15.42:0/129210727 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f470c1156c0 0x7f470c077d40 secure :-1 s=THROTTLE_DONE pgs=275 cs=0 l=1 rx=0x7f46fc002f30 tx=0x7f46fc010960).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:48.080 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:48.060+0000 7f4712346700 1 -- 172.21.15.42:0/129210727 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f470c118310 con 0x7f470c1156c0 2019-12-05T05:42:48.081 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:48.060+0000 7f47097fa700 1 -- 172.21.15.42:0/129210727 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46fc005540 con 0x7f470c1156c0 2019-12-05T05:42:48.081 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:48.060+0000 7f47108e3700 1 --2- 172.21.15.42:0/129210727 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f470c1156c0 0x7f470c077d40 secure :-1 s=THROTTLE_DONE pgs=275 cs=0 l=1 rx=0x7f46fc002f30 tx=0x7f46fc010960).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:48.081 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:48.060+0000 7f47097fa700 1 -- 172.21.15.42:0/129210727 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f46fc012410 con 0x7f470c1156c0 2019-12-05T05:42:48.081 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:48.060+0000 7f4712346700 1 -- 172.21.15.42:0/129210727 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f470c130470 con 0x7f470c1156c0 2019-12-05T05:42:48.081 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:48.060+0000 7f47108e3700 1 --2- 172.21.15.42:0/129210727 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f470c1156c0 0x7f470c077d40 secure :-1 s=THROTTLE_DONE pgs=275 cs=0 l=1 rx=0x7f46fc002f30 tx=0x7f46fc010960).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:48.081 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:48.064+0000 7f47097fa700 1 -- 172.21.15.42:0/129210727 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f46fc007180 con 0x7f470c1156c0 2019-12-05T05:42:48.082 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:48.064+0000 7f47108e3700 1 --2- 172.21.15.42:0/129210727 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f470c1156c0 0x7f470c077d40 secure :-1 s=THROTTLE_DONE pgs=275 cs=0 l=1 rx=0x7f46fc002f30 tx=0x7f46fc010960).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:48.082 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:48.064+0000 7f47097fa700 1 --2- 172.21.15.42:0/129210727 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f46ec03b400 0x7f46ec03d910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:42:48.082 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:48.064+0000 7f47097fa700 1 -- 172.21.15.42:0/129210727 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 8267+0+0 (secure 0 0 0) 0x7f46fc04aba0 con 0x7f470c1156c0 2019-12-05T05:42:48.082 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:48.064+0000 7f470bfff700 1 --2- 172.21.15.42:0/129210727 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f46ec03b400 0x7f46ec03d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:42:48.082 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:48.064+0000 7f470bfff700 1 --2- 172.21.15.42:0/129210727 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f46ec03b400 0x7f46ec03d910 crc :-1 s=READY pgs=565 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:42:48.082 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:48.064+0000 7f4712346700 1 -- 172.21.15.42:0/129210727 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f470c05d110 con 0x7f470c1156c0 2019-12-05T05:42:48.083 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:48.072+0000 7f47108e3700 1 --2- 172.21.15.42:0/129210727 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f470c1156c0 0x7f470c077d40 secure :-1 s=THROTTLE_DONE pgs=275 cs=0 l=1 rx=0x7f46fc002f30 tx=0x7f46fc010960).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:48.083 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:48.072+0000 7f47097fa700 1 -- 172.21.15.42:0/129210727 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f470c118310 con 0x7f470c1156c0 2019-12-05T05:42:48.256 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:48.244+0000 7f4712346700 1 -- 172.21.15.42:0/129210727 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f470c05b8c0 con 0x7f470c1156c0 2019-12-05T05:42:48.256 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:48.244+0000 7f47108e3700 1 --2- 172.21.15.42:0/129210727 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f470c1156c0 0x7f470c077d40 secure :-1 s=THROTTLE_DONE pgs=275 cs=0 l=1 rx=0x7f46fc002f30 tx=0x7f46fc010960).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:48.256 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:48.244+0000 7f47097fa700 1 -- 172.21.15.42:0/129210727 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 161 v161) v1 ==== 95+0+1775 (secure 0 0 0) 0x7f46fc031c00 con 0x7f470c1156c0 2019-12-05T05:42:48.256 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 161 2019-12-05T05:42:48.257 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:48.248+0000 7f4712346700 1 -- 172.21.15.42:0/129210727 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f46ec03b400 msgr2=0x7f46ec03d910 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:42:48.257 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:48.248+0000 7f4712346700 1 --2- 172.21.15.42:0/129210727 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f46ec03b400 0x7f46ec03d910 crc :-1 s=READY pgs=565 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:42:48.257 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:48.248+0000 7f4712346700 1 -- 172.21.15.42:0/129210727 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f470c1156c0 msgr2=0x7f470c077d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:42:48.257 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:48.248+0000 7f4712346700 1 --2- 172.21.15.42:0/129210727 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f470c1156c0 0x7f470c077d40 secure :-1 s=READY pgs=275 cs=0 l=1 rx=0x7f46fc002f30 tx=0x7f46fc010960).stop 2019-12-05T05:42:48.258 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:48.248+0000 7f4712346700 1 -- 172.21.15.42:0/129210727 shutdown_connections 2019-12-05T05:42:48.258 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:48.248+0000 7f4712346700 1 --2- 172.21.15.42:0/129210727 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f46ec03b400 0x7f46ec03d910 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:42:48.258 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:48.248+0000 7f4712346700 1 --2- 172.21.15.42:0/129210727 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f470c1156c0 0x7f470c077d40 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:42:48.258 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:48.248+0000 7f4712346700 1 -- 172.21.15.42:0/129210727 shutdown_connections 2019-12-05T05:42:48.258 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:48.248+0000 7f4712346700 1 -- 172.21.15.42:0/129210727 wait complete. 2019-12-05T05:42:48.258 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:48.248+0000 7f4712346700 1 -- 172.21.15.42:0/129210727 >> 172.21.15.42:0/129210727 conn(0x7f470c08fdc0 msgr2=0x7f470c0a97e0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:42:48.259 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:42:48.259 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":161,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":161,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:42:47.956725+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7650},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7650":{"gid":7650,"name":"b","rank":0,"incarnation":159,"state":"up:rejoin","state_seq":28,"addr":"172.21.15.40:6833/1029013337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1029013337},{"type":"v1","addr":"172.21.15.40:6833","nonce":1029013337}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:42:48.276 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2019-12-05T05:42:50.277 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:42:50.386 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:50.376+0000 7f983520a700 1 Processor -- start 2019-12-05T05:42:50.386 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:50.376+0000 7f983520a700 1 -- start start 2019-12-05T05:42:50.386 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:50.376+0000 7f983520a700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f983009fae0 0x7f983014fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:42:50.386 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:50.376+0000 7f983520a700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98300a0710 con 0x7f98300a7740 2019-12-05T05:42:50.387 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:50.376+0000 7f983520a700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f983007ef20 con 0x7f983008df80 2019-12-05T05:42:50.387 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:50.376+0000 7f983520a700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98300a0b00 con 0x7f983009fae0 2019-12-05T05:42:50.387 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:50.376+0000 7f9834a09700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f983009fae0 0x7f983014fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:42:50.387 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:50.376+0000 7f9834a09700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f983009fae0 0x7f983014fff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:39396/0 (socket says 172.21.15.42:39396) 2019-12-05T05:42:50.388 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:50.376+0000 7f9834a09700 1 -- 172.21.15.42:0/183121844 learned_addr learned my addr 172.21.15.42:0/183121844 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:42:50.388 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:50.376+0000 7f982f59e700 1 --1- 172.21.15.42:0/183121844 >> v1:172.21.15.40:6789/0 conn(0x7f98300a7740 0x7f983007e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:48870/0 (socket says 172.21.15.42:48870) 2019-12-05T05:42:50.388 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:50.376+0000 7f9826d9d700 1 --1- 172.21.15.42:0/183121844 >> v1:172.21.15.42:6789/0 conn(0x7f983008df80 0x7f983009f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:39432/0 (socket says 172.21.15.42:39432) 2019-12-05T05:42:50.388 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:50.376+0000 7f9834a09700 1 -- 172.21.15.42:0/183121844 >> v1:172.21.15.42:6789/0 conn(0x7f983008df80 legacy=0x7f983009f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:42:50.388 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:50.376+0000 7f982ed9d700 1 -- 172.21.15.42:0/183121844 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2139725754 0 0) 0x7f98300a0710 con 0x7f98300a7740 2019-12-05T05:42:50.388 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:50.376+0000 7f9834a09700 1 -- 172.21.15.42:0/183121844 >> v1:172.21.15.40:6789/0 conn(0x7f98300a7740 legacy=0x7f983007e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:42:50.389 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:50.376+0000 7f9834a09700 1 -- 172.21.15.42:0/183121844 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98300a9ba0 con 0x7f983009fae0 2019-12-05T05:42:50.389 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:50.376+0000 7f9834a09700 1 --2- 172.21.15.42:0/183121844 >> v2:172.21.15.42:3300/0 conn(0x7f983009fae0 0x7f983014fff0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f9828008b10 tx=0x7f9818001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:50.389 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:50.380+0000 7f9834a09700 1 --2- 172.21.15.42:0/183121844 >> v2:172.21.15.42:3300/0 conn(0x7f983009fae0 0x7f983014fff0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f9828008b10 tx=0x7f9818001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:50.389 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:50.380+0000 7f9834a09700 1 --2- 172.21.15.42:0/183121844 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f983009fae0 0x7f983014fff0 secure :-1 s=READY pgs=1312 cs=0 l=1 rx=0x7f9828008b10 tx=0x7f9818001150).ready entity=mon.1 client_cookie=2a7193be636fdf67 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:42:50.389 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:50.380+0000 7f9834a09700 1 --2- 172.21.15.42:0/183121844 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f983009fae0 0x7f983014fff0 secure :-1 s=THROTTLE_DONE pgs=1312 cs=0 l=1 rx=0x7f9828008b10 tx=0x7f9818001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:50.389 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:50.380+0000 7f9834a09700 1 --2- 172.21.15.42:0/183121844 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f983009fae0 0x7f983014fff0 secure :-1 s=THROTTLE_DONE pgs=1312 cs=0 l=1 rx=0x7f9828008b10 tx=0x7f9818001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:50.391 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:50.380+0000 7f982ed9d700 1 -- 172.21.15.42:0/183121844 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9828010d10 con 0x7f983009fae0 2019-12-05T05:42:50.392 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:50.380+0000 7f9834a09700 1 --2- 172.21.15.42:0/183121844 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f983009fae0 0x7f983014fff0 secure :-1 s=THROTTLE_DONE pgs=1312 cs=0 l=1 rx=0x7f9828008b10 tx=0x7f9818001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:50.392 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:50.380+0000 7f982ed9d700 1 -- 172.21.15.42:0/183121844 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9828010930 con 0x7f983009fae0 2019-12-05T05:42:50.392 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:50.380+0000 7f982ed9d700 1 -- 172.21.15.42:0/183121844 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f98280114a0 con 0x7f983009fae0 2019-12-05T05:42:50.392 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:50.380+0000 7f983520a700 1 -- 172.21.15.42:0/183121844 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f983009fae0 msgr2=0x7f983014fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:42:50.392 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:50.380+0000 7f983520a700 1 --2- 172.21.15.42:0/183121844 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f983009fae0 0x7f983014fff0 secure :-1 s=READY pgs=1312 cs=0 l=1 rx=0x7f9828008b10 tx=0x7f9818001150).stop 2019-12-05T05:42:50.392 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:50.380+0000 7f983520a700 1 -- 172.21.15.42:0/183121844 shutdown_connections 2019-12-05T05:42:50.393 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:50.380+0000 7f983520a700 1 --2- 172.21.15.42:0/183121844 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f983009fae0 0x7f983014fff0 unknown :-1 s=CLOSED pgs=1312 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:42:50.393 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:50.380+0000 7f983520a700 1 -- 172.21.15.42:0/183121844 shutdown_connections 2019-12-05T05:42:50.393 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:50.380+0000 7f983520a700 1 -- 172.21.15.42:0/183121844 wait complete. 2019-12-05T05:42:50.393 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:50.380+0000 7f983520a700 1 -- 172.21.15.42:0/183121844 >> 172.21.15.42:0/183121844 conn(0x7f98300aabe0 msgr2=0x7f98300a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:42:50.396 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:50.384+0000 7f983520a700 1 Processor -- start 2019-12-05T05:42:50.397 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:50.384+0000 7f983520a700 1 -- start start 2019-12-05T05:42:50.397 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:50.384+0000 7f983520a700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f98301485e0 0x7f983011b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:42:50.397 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:50.384+0000 7f983520a700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f98301471f0 0x7f983011be20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:42:50.397 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:50.384+0000 7f983520a700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9830146700 con 0x7f98301194f0 2019-12-05T05:42:50.397 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:50.384+0000 7f983520a700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9830146460 con 0x7f98301485e0 2019-12-05T05:42:50.397 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:50.384+0000 7f983520a700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9830147760 con 0x7f98301471f0 2019-12-05T05:42:50.398 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:50.384+0000 7f982f59e700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f98301194f0 0x7f9830147a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:39436/0 (socket says 172.21.15.42:39436) 2019-12-05T05:42:50.398 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:50.384+0000 7f982f59e700 1 -- 172.21.15.42:0/3060932935 learned_addr learned my addr 172.21.15.42:0/3060932935 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:42:50.398 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:50.384+0000 7f9826d9d700 1 --2- 172.21.15.42:0/3060932935 >> v2:172.21.15.40:3300/0 conn(0x7f98301485e0 0x7f983011b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:42:50.398 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:50.384+0000 7f9834a09700 1 --2- 172.21.15.42:0/3060932935 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f98301471f0 0x7f983011be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:42:50.398 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:50.384+0000 7f982cd99700 1 -- 172.21.15.42:0/3060932935 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 563436305 0 0) 0x7f9830146700 con 0x7f98301194f0 2019-12-05T05:42:50.398 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:50.384+0000 7f982cd99700 1 -- 172.21.15.42:0/3060932935 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98140030b0 con 0x7f98301194f0 2019-12-05T05:42:50.398 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:50.384+0000 7f982cd99700 1 -- 172.21.15.42:0/3060932935 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 3143724977 0 0) 0x7f98140030b0 con 0x7f98301194f0 2019-12-05T05:42:50.399 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:50.384+0000 7f982cd99700 1 -- 172.21.15.42:0/3060932935 --> v1:172.21.15.42:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9830146700 con 0x7f98301194f0 2019-12-05T05:42:50.399 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:50.384+0000 7f982cd99700 1 -- 172.21.15.42:0/3060932935 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7f9820001a20 con 0x7f98301194f0 2019-12-05T05:42:50.399 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:50.384+0000 7f9826d9d700 1 -- 172.21.15.42:0/3060932935 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f98301471f0 msgr2=0x7f983011be20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:42:50.399 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:50.384+0000 7f9826d9d700 1 --2- 172.21.15.42:0/3060932935 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f98301471f0 0x7f983011be20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:42:50.399 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:50.384+0000 7f9826d9d700 1 -- 172.21.15.42:0/3060932935 >> v1:172.21.15.42:6789/0 conn(0x7f98301194f0 legacy=0x7f9830147a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:42:50.399 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:50.384+0000 7f9826d9d700 1 -- 172.21.15.42:0/3060932935 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9830145fc0 con 0x7f98301485e0 2019-12-05T05:42:50.400 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:50.384+0000 7f9834a09700 1 --2- 172.21.15.42:0/3060932935 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f98301471f0 0x7f983011be20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:42:50.400 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:50.384+0000 7f9826d9d700 1 --2- 172.21.15.42:0/3060932935 >> v2:172.21.15.40:3300/0 conn(0x7f98301485e0 0x7f983011b8d0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f9830146620 tx=0x7f98200011e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:50.400 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:50.384+0000 7f9826d9d700 1 --2- 172.21.15.42:0/3060932935 >> v2:172.21.15.40:3300/0 conn(0x7f98301485e0 0x7f983011b8d0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f9830146620 tx=0x7f98200011e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:50.400 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:50.384+0000 7f9826d9d700 1 --2- 172.21.15.42:0/3060932935 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f98301485e0 0x7f983011b8d0 secure :-1 s=READY pgs=1020 cs=0 l=1 rx=0x7f9830146620 tx=0x7f98200011e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:42:50.400 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:50.384+0000 7f9826d9d700 1 --2- 172.21.15.42:0/3060932935 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f98301485e0 0x7f983011b8d0 secure :-1 s=THROTTLE_DONE pgs=1020 cs=0 l=1 rx=0x7f9830146620 tx=0x7f98200011e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:50.400 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:50.384+0000 7f982cd99700 1 -- 172.21.15.42:0/3060932935 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9818011410 con 0x7f98301485e0 2019-12-05T05:42:50.400 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:50.384+0000 7f9826d9d700 1 --2- 172.21.15.42:0/3060932935 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f98301485e0 0x7f983011b8d0 secure :-1 s=THROTTLE_DONE pgs=1020 cs=0 l=1 rx=0x7f9830146620 tx=0x7f98200011e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:50.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:50.384+0000 7f982cd99700 1 -- 172.21.15.42:0/3060932935 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9818011030 con 0x7f98301485e0 2019-12-05T05:42:50.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:50.384+0000 7f9826d9d700 1 --2- 172.21.15.42:0/3060932935 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f98301485e0 0x7f983011b8d0 secure :-1 s=THROTTLE_DONE pgs=1020 cs=0 l=1 rx=0x7f9830146620 tx=0x7f98200011e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:50.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:50.384+0000 7f983520a700 1 -- 172.21.15.42:0/3060932935 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9830129f20 con 0x7f98301485e0 2019-12-05T05:42:50.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:50.384+0000 7f982cd99700 1 -- 172.21.15.42:0/3060932935 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f98180119f0 con 0x7f98301485e0 2019-12-05T05:42:50.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:50.384+0000 7f983520a700 1 -- 172.21.15.42:0/3060932935 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9830134930 con 0x7f98301485e0 2019-12-05T05:42:50.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:50.388+0000 7f9826d9d700 1 --2- 172.21.15.42:0/3060932935 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f98301485e0 0x7f983011b8d0 secure :-1 s=THROTTLE_DONE pgs=1020 cs=0 l=1 rx=0x7f9830146620 tx=0x7f98200011e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:50.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:50.388+0000 7f982cd99700 1 -- 172.21.15.42:0/3060932935 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f9818006460 con 0x7f98301485e0 2019-12-05T05:42:50.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:50.388+0000 7f9826d9d700 1 --2- 172.21.15.42:0/3060932935 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f98301485e0 0x7f983011b8d0 secure :-1 s=THROTTLE_DONE pgs=1020 cs=0 l=1 rx=0x7f9830146620 tx=0x7f98200011e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:50.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:50.388+0000 7f982cd99700 1 --2- 172.21.15.42:0/3060932935 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f981403b6a0 0x7f981403dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:42:50.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:50.388+0000 7f982cd99700 1 -- 172.21.15.42:0/3060932935 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 8267+0+0 (secure 0 0 0) 0x7f981804ac30 con 0x7f98301485e0 2019-12-05T05:42:50.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:50.388+0000 7f982f59e700 1 --2- 172.21.15.42:0/3060932935 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f981403b6a0 0x7f981403dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:42:50.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:50.388+0000 7f982f59e700 1 --2- 172.21.15.42:0/3060932935 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f981403b6a0 0x7f981403dbb0 crc :-1 s=READY pgs=566 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:42:50.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:50.392+0000 7f983520a700 1 -- 172.21.15.42:0/3060932935 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f983005d110 con 0x7f98301485e0 2019-12-05T05:42:50.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:50.400+0000 7f9826d9d700 1 --2- 172.21.15.42:0/3060932935 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f98301485e0 0x7f983011b8d0 secure :-1 s=THROTTLE_DONE pgs=1020 cs=0 l=1 rx=0x7f9830146620 tx=0x7f98200011e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:50.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:50.400+0000 7f982cd99700 1 -- 172.21.15.42:0/3060932935 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f9818031c30 con 0x7f98301485e0 2019-12-05T05:42:50.583 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:50.580+0000 7f983520a700 1 -- 172.21.15.42:0/3060932935 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f983005b8c0 con 0x7f98301485e0 2019-12-05T05:42:50.584 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:50.580+0000 7f9826d9d700 1 --2- 172.21.15.42:0/3060932935 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f98301485e0 0x7f983011b8d0 secure :-1 s=THROTTLE_DONE pgs=1020 cs=0 l=1 rx=0x7f9830146620 tx=0x7f98200011e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:50.584 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:50.580+0000 7f982cd99700 1 -- 172.21.15.42:0/3060932935 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 162 v162) v1 ==== 95+0+1775 (secure 0 0 0) 0x7f9818031e90 con 0x7f98301485e0 2019-12-05T05:42:50.586 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 162 2019-12-05T05:42:50.586 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:42:50.587 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":162,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":162,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:42:48.961753+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7650},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7650":{"gid":7650,"name":"b","rank":0,"incarnation":159,"state":"up:active","state_seq":29,"addr":"172.21.15.40:6833/1029013337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1029013337},{"type":"v1","addr":"172.21.15.40:6833","nonce":1029013337}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:42:50.590 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:50.584+0000 7f983520a700 1 -- 172.21.15.42:0/3060932935 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f981403b6a0 msgr2=0x7f981403dbb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:42:50.590 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:50.584+0000 7f983520a700 1 --2- 172.21.15.42:0/3060932935 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f981403b6a0 0x7f981403dbb0 crc :-1 s=READY pgs=566 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:42:50.590 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:50.584+0000 7f983520a700 1 -- 172.21.15.42:0/3060932935 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f98301485e0 msgr2=0x7f983011b8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:42:50.590 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:50.584+0000 7f983520a700 1 --2- 172.21.15.42:0/3060932935 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f98301485e0 0x7f983011b8d0 secure :-1 s=READY pgs=1020 cs=0 l=1 rx=0x7f9830146620 tx=0x7f98200011e0).stop 2019-12-05T05:42:50.591 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:50.584+0000 7f983520a700 1 -- 172.21.15.42:0/3060932935 shutdown_connections 2019-12-05T05:42:50.591 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:50.584+0000 7f983520a700 1 --2- 172.21.15.42:0/3060932935 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f981403b6a0 0x7f981403dbb0 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:42:50.591 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:50.584+0000 7f983520a700 1 --2- 172.21.15.42:0/3060932935 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f98301471f0 0x7f983011be20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:42:50.591 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:50.584+0000 7f983520a700 1 --2- 172.21.15.42:0/3060932935 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f98301485e0 0x7f983011b8d0 unknown :-1 s=CLOSED pgs=1020 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:42:50.591 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:50.584+0000 7f983520a700 1 -- 172.21.15.42:0/3060932935 shutdown_connections 2019-12-05T05:42:50.591 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:50.584+0000 7f983520a700 1 -- 172.21.15.42:0/3060932935 wait complete. 2019-12-05T05:42:50.592 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:50.584+0000 7f983520a700 1 -- 172.21.15.42:0/3060932935 >> 172.21.15.42:0/3060932935 conn(0x7f983007eae0 msgr2=0x7f983009f460 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:42:50.599 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b has gained rank=0, replacing gid=7539 2019-12-05T05:42:50.599 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 2 secs before reviving mds.a 2019-12-05T05:42:52.600 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.a 2019-12-05T05:42:52.600 INFO:tasks.ceph.mds.a:Restarting daemon 2019-12-05T05:42:52.601 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2019-12-05T05:42:52.604 INFO:tasks.ceph.mds.a:Started 2019-12-05T05:42:52.605 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:42:52.663 INFO:tasks.ceph.mds.a.smithi042.stdout:starting mds.a at 2019-12-05T05:42:52.705 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:52.696+0000 7fb72ab67700 1 Processor -- start 2019-12-05T05:42:52.705 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:52.696+0000 7fb72ab67700 1 -- start start 2019-12-05T05:42:52.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:52.696+0000 7fb72ab67700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb7240a7740 0x7fb72408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:42:52.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:52.696+0000 7fb72ab67700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7240a0710 con 0x7fb72409fae0 2019-12-05T05:42:52.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:52.696+0000 7fb72ab67700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7240a9ba0 con 0x7fb72407e720 2019-12-05T05:42:52.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:52.696+0000 7fb72ab67700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7240a0b00 con 0x7fb7240a7740 2019-12-05T05:42:52.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:52.696+0000 7fb729104700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb7240a7740 0x7fb72408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:42:52.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:52.696+0000 7fb729104700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb7240a7740 0x7fb72408df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:39432/0 (socket says 172.21.15.42:39432) 2019-12-05T05:42:52.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:52.696+0000 7fb729104700 1 -- 172.21.15.42:0/718692724 learned_addr learned my addr 172.21.15.42:0/718692724 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:42:52.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:52.696+0000 7fb729104700 1 -- 172.21.15.42:0/718692724 >> v1:172.21.15.42:6789/0 conn(0x7fb72407e720 legacy=0x7fb72409f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:42:52.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:52.696+0000 7fb729104700 1 -- 172.21.15.42:0/718692724 >> v1:172.21.15.40:6789/0 conn(0x7fb72409fae0 legacy=0x7fb72414efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:42:52.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:52.696+0000 7fb729104700 1 -- 172.21.15.42:0/718692724 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb72407eb50 con 0x7fb7240a7740 2019-12-05T05:42:52.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:52.696+0000 7fb729104700 1 --2- 172.21.15.42:0/718692724 >> v2:172.21.15.42:3300/0 conn(0x7fb7240a7740 0x7fb72408df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fb720008790 tx=0x7fb72000fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:52.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:52.696+0000 7fb729104700 1 --2- 172.21.15.42:0/718692724 >> v2:172.21.15.42:3300/0 conn(0x7fb7240a7740 0x7fb72408df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fb720008790 tx=0x7fb72000fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:52.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:52.696+0000 7fb729104700 1 --2- 172.21.15.42:0/718692724 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb7240a7740 0x7fb72408df80 secure :-1 s=READY pgs=1317 cs=0 l=1 rx=0x7fb720008790 tx=0x7fb72000fcb0).ready entity=mon.1 client_cookie=a06a49af42708b52 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:42:52.708 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:52.696+0000 7fb729104700 1 --2- 172.21.15.42:0/718692724 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb7240a7740 0x7fb72408df80 secure :-1 s=THROTTLE_DONE pgs=1317 cs=0 l=1 rx=0x7fb720008790 tx=0x7fb72000fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:52.708 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:52.696+0000 7fb71bfff700 1 -- 172.21.15.42:0/718692724 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb720011520 con 0x7fb7240a7740 2019-12-05T05:42:52.708 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:52.696+0000 7fb729104700 1 --2- 172.21.15.42:0/718692724 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb7240a7740 0x7fb72408df80 secure :-1 s=THROTTLE_DONE pgs=1317 cs=0 l=1 rx=0x7fb720008790 tx=0x7fb72000fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:52.708 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:52.696+0000 7fb729104700 1 --2- 172.21.15.42:0/718692724 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb7240a7740 0x7fb72408df80 secure :-1 s=THROTTLE_DONE pgs=1317 cs=0 l=1 rx=0x7fb720008790 tx=0x7fb72000fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:52.708 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:52.696+0000 7fb71bfff700 1 -- 172.21.15.42:0/718692724 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb720008ca0 con 0x7fb7240a7740 2019-12-05T05:42:52.708 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:52.696+0000 7fb72ab67700 1 -- 172.21.15.42:0/718692724 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb7240a7740 msgr2=0x7fb72408df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:42:52.709 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:52.696+0000 7fb71bfff700 1 -- 172.21.15.42:0/718692724 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb720011b00 con 0x7fb7240a7740 2019-12-05T05:42:52.709 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:52.696+0000 7fb72ab67700 1 --2- 172.21.15.42:0/718692724 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb7240a7740 0x7fb72408df80 secure :-1 s=READY pgs=1317 cs=0 l=1 rx=0x7fb720008790 tx=0x7fb72000fcb0).stop 2019-12-05T05:42:52.709 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:52.700+0000 7fb72ab67700 1 -- 172.21.15.42:0/718692724 shutdown_connections 2019-12-05T05:42:52.709 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:52.700+0000 7fb72ab67700 1 --2- 172.21.15.42:0/718692724 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb7240a7740 0x7fb72408df80 unknown :-1 s=CLOSED pgs=1317 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:42:52.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:52.700+0000 7fb72ab67700 1 -- 172.21.15.42:0/718692724 shutdown_connections 2019-12-05T05:42:52.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:52.700+0000 7fb72ab67700 1 -- 172.21.15.42:0/718692724 wait complete. 2019-12-05T05:42:52.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:52.700+0000 7fb72ab67700 1 -- 172.21.15.42:0/718692724 >> 172.21.15.42:0/718692724 conn(0x7fb7240aabe0 msgr2=0x7fb7240a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:42:52.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:52.700+0000 7fb72ab67700 1 Processor -- start 2019-12-05T05:42:52.711 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:52.700+0000 7fb72ab67700 1 -- start start 2019-12-05T05:42:52.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:52.700+0000 7fb72ab67700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb7240a6c90 0x7fb724113fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:42:52.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:52.700+0000 7fb72ab67700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb724115910 con 0x7fb724114530 2019-12-05T05:42:52.715 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:52.700+0000 7fb72ab67700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7241487e0 con 0x7fb724116bc0 2019-12-05T05:42:52.715 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:52.700+0000 7fb72ab67700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb724146a90 con 0x7fb7240a6c90 2019-12-05T05:42:52.715 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:52.700+0000 7fb728903700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb7240a6c90 0x7fb724113fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:42:52.715 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:52.700+0000 7fb728903700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb7240a6c90 0x7fb724113fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:39438/0 (socket says 172.21.15.42:39438) 2019-12-05T05:42:52.715 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:52.700+0000 7fb728903700 1 -- 172.21.15.42:0/3123250625 learned_addr learned my addr 172.21.15.42:0/3123250625 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:42:52.715 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:52.700+0000 7fb729104700 1 --1- 172.21.15.42:0/3123250625 >> v1:172.21.15.42:6789/0 conn(0x7fb724116bc0 0x7fb724115110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:39478/0 (socket says 172.21.15.42:39478) 2019-12-05T05:42:52.716 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:52.700+0000 7fb728903700 1 -- 172.21.15.42:0/3123250625 >> v1:172.21.15.42:6789/0 conn(0x7fb724116bc0 legacy=0x7fb724115110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:42:52.716 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:52.700+0000 7fb728903700 1 -- 172.21.15.42:0/3123250625 >> v1:172.21.15.40:6789/0 conn(0x7fb724114530 legacy=0x7fb72412af80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:42:52.716 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:52.700+0000 7fb728903700 1 -- 172.21.15.42:0/3123250625 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb724116320 con 0x7fb7240a6c90 2019-12-05T05:42:52.716 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:52.700+0000 7fb728903700 1 --2- 172.21.15.42:0/3123250625 >> v2:172.21.15.42:3300/0 conn(0x7fb7240a6c90 0x7fb724113fe0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fb71c000f00 tx=0x7fb71c010c70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:52.716 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:52.700+0000 7fb728903700 1 --2- 172.21.15.42:0/3123250625 >> v2:172.21.15.42:3300/0 conn(0x7fb7240a6c90 0x7fb724113fe0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fb71c000f00 tx=0x7fb71c010c70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:52.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:52.700+0000 7fb728903700 1 --2- 172.21.15.42:0/3123250625 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb7240a6c90 0x7fb724113fe0 secure :-1 s=READY pgs=1319 cs=0 l=1 rx=0x7fb71c000f00 tx=0x7fb71c010c70).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:42:52.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:52.700+0000 7fb728903700 1 --2- 172.21.15.42:0/3123250625 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb7240a6c90 0x7fb724113fe0 secure :-1 s=THROTTLE_DONE pgs=1319 cs=0 l=1 rx=0x7fb71c000f00 tx=0x7fb71c010c70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:52.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:52.700+0000 7fb719ffb700 1 -- 172.21.15.42:0/3123250625 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb71c00ca10 con 0x7fb7240a6c90 2019-12-05T05:42:52.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:52.700+0000 7fb728903700 1 --2- 172.21.15.42:0/3123250625 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb7240a6c90 0x7fb724113fe0 secure :-1 s=THROTTLE_DONE pgs=1319 cs=0 l=1 rx=0x7fb71c000f00 tx=0x7fb71c010c70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:52.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:52.700+0000 7fb728903700 1 --2- 172.21.15.42:0/3123250625 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb7240a6c90 0x7fb724113fe0 secure :-1 s=THROTTLE_DONE pgs=1319 cs=0 l=1 rx=0x7fb71c000f00 tx=0x7fb71c010c70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:52.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:52.700+0000 7fb719ffb700 1 -- 172.21.15.42:0/3123250625 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb71c0101c0 con 0x7fb7240a6c90 2019-12-05T05:42:52.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:52.700+0000 7fb72ab67700 1 -- 172.21.15.42:0/3123250625 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb724114960 con 0x7fb7240a6c90 2019-12-05T05:42:52.718 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:52.700+0000 7fb719ffb700 1 -- 172.21.15.42:0/3123250625 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb71c0122b0 con 0x7fb7240a6c90 2019-12-05T05:42:52.718 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:52.704+0000 7fb72ab67700 1 -- 172.21.15.42:0/3123250625 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb72412c790 con 0x7fb7240a6c90 2019-12-05T05:42:52.718 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:52.704+0000 7fb728903700 1 --2- 172.21.15.42:0/3123250625 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb7240a6c90 0x7fb724113fe0 secure :-1 s=THROTTLE_DONE pgs=1319 cs=0 l=1 rx=0x7fb71c000f00 tx=0x7fb71c010c70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:52.718 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:52.704+0000 7fb719ffb700 1 -- 172.21.15.42:0/3123250625 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fb71c0066e0 con 0x7fb7240a6c90 2019-12-05T05:42:52.718 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:52.704+0000 7fb728903700 1 --2- 172.21.15.42:0/3123250625 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb7240a6c90 0x7fb724113fe0 secure :-1 s=THROTTLE_DONE pgs=1319 cs=0 l=1 rx=0x7fb71c000f00 tx=0x7fb71c010c70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:52.718 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:52.704+0000 7fb719ffb700 1 --2- 172.21.15.42:0/3123250625 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb70403acb0 0x7fb70403d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:42:52.718 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:52.704+0000 7fb719ffb700 1 -- 172.21.15.42:0/3123250625 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 8267+0+0 (secure 0 0 0) 0x7fb71c04b3c0 con 0x7fb7240a6c90 2019-12-05T05:42:52.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:52.704+0000 7fb729104700 1 --2- 172.21.15.42:0/3123250625 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb70403acb0 0x7fb70403d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:42:52.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:52.704+0000 7fb729104700 1 --2- 172.21.15.42:0/3123250625 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb70403acb0 0x7fb70403d1c0 crc :-1 s=READY pgs=567 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:42:52.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:52.712+0000 7fb72ab67700 1 -- 172.21.15.42:0/3123250625 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb72405d110 con 0x7fb7240a6c90 2019-12-05T05:42:52.725 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:52.720+0000 7fb728903700 1 --2- 172.21.15.42:0/3123250625 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb7240a6c90 0x7fb724113fe0 secure :-1 s=THROTTLE_DONE pgs=1319 cs=0 l=1 rx=0x7fb71c000f00 tx=0x7fb71c010c70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:52.725 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:52.720+0000 7fb719ffb700 1 -- 172.21.15.42:0/3123250625 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fb71c032440 con 0x7fb7240a6c90 2019-12-05T05:42:52.904 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:52.900+0000 7fb72ab67700 1 -- 172.21.15.42:0/3123250625 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb72405b8c0 con 0x7fb7240a6c90 2019-12-05T05:42:52.905 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:52.900+0000 7fb728903700 1 --2- 172.21.15.42:0/3123250625 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb7240a6c90 0x7fb724113fe0 secure :-1 s=THROTTLE_DONE pgs=1319 cs=0 l=1 rx=0x7fb71c000f00 tx=0x7fb71c010c70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:52.905 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:52.900+0000 7fb719ffb700 1 -- 172.21.15.42:0/3123250625 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 162 v162) v1 ==== 95+0+1775 (secure 0 0 0) 0x7fb71c0326a0 con 0x7fb7240a6c90 2019-12-05T05:42:52.905 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 162 2019-12-05T05:42:52.905 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:42:52.905 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":162,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":162,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:42:48.961753+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7650},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7650":{"gid":7650,"name":"b","rank":0,"incarnation":159,"state":"up:active","state_seq":29,"addr":"172.21.15.40:6833/1029013337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1029013337},{"type":"v1","addr":"172.21.15.40:6833","nonce":1029013337}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:42:52.908 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:52.904+0000 7fb72ab67700 1 -- 172.21.15.42:0/3123250625 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb70403acb0 msgr2=0x7fb70403d1c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:42:52.908 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:52.904+0000 7fb72ab67700 1 --2- 172.21.15.42:0/3123250625 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb70403acb0 0x7fb70403d1c0 crc :-1 s=READY pgs=567 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:42:52.908 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:52.904+0000 7fb72ab67700 1 -- 172.21.15.42:0/3123250625 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb7240a6c90 msgr2=0x7fb724113fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:42:52.908 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:52.904+0000 7fb72ab67700 1 --2- 172.21.15.42:0/3123250625 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb7240a6c90 0x7fb724113fe0 secure :-1 s=READY pgs=1319 cs=0 l=1 rx=0x7fb71c000f00 tx=0x7fb71c010c70).stop 2019-12-05T05:42:52.908 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:52.904+0000 7fb72ab67700 1 -- 172.21.15.42:0/3123250625 shutdown_connections 2019-12-05T05:42:52.909 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:52.904+0000 7fb72ab67700 1 --2- 172.21.15.42:0/3123250625 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb70403acb0 0x7fb70403d1c0 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:42:52.909 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:52.904+0000 7fb72ab67700 1 --2- 172.21.15.42:0/3123250625 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb7240a6c90 0x7fb724113fe0 unknown :-1 s=CLOSED pgs=1319 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:42:52.909 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:52.904+0000 7fb72ab67700 1 -- 172.21.15.42:0/3123250625 shutdown_connections 2019-12-05T05:42:52.909 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:52.904+0000 7fb72ab67700 1 -- 172.21.15.42:0/3123250625 wait complete. 2019-12-05T05:42:52.909 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:52.904+0000 7fb72ab67700 1 -- 172.21.15.42:0/3123250625 >> 172.21.15.42:0/3123250625 conn(0x7fb72408fda0 msgr2=0x7fb72408eb70 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:42:52.925 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2019-12-05T05:42:54.926 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:42:55.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:55.024+0000 7fe7a6379700 1 Processor -- start 2019-12-05T05:42:55.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:55.024+0000 7fe7a6379700 1 -- start start 2019-12-05T05:42:55.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:55.024+0000 7fe7a6379700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe7a008df80 0x7fe7a009f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:42:55.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:55.024+0000 7fe7a6379700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7a009fc60 con 0x7fe7a009f830 2019-12-05T05:42:55.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:55.024+0000 7fe7a6379700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7a00a0710 con 0x7fe7a00a7740 2019-12-05T05:42:55.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:55.024+0000 7fe7a6379700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7a00a0b00 con 0x7fe7a008df80 2019-12-05T05:42:55.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:55.024+0000 7fe7a4916700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fe7a00a7740 0x7fe7a007e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:39486/0 (socket says 172.21.15.42:39486) 2019-12-05T05:42:55.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:55.024+0000 7fe7a4916700 1 -- 172.21.15.42:0/1574483531 learned_addr learned my addr 172.21.15.42:0/1574483531 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:42:55.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:55.024+0000 7fe79ffff700 1 --2- 172.21.15.42:0/1574483531 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe7a008df80 0x7fe7a009f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:42:55.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:55.028+0000 7fe79f7fe700 1 -- 172.21.15.42:0/1574483531 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3122864655 0 0) 0x7fe7a00a0710 con 0x7fe7a00a7740 2019-12-05T05:42:55.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:55.028+0000 7fe79f7fe700 1 -- 172.21.15.42:0/1574483531 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe78c0030b0 con 0x7fe7a00a7740 2019-12-05T05:42:55.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:55.028+0000 7fe79f7fe700 1 -- 172.21.15.42:0/1574483531 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 586260450 0 0) 0x7fe7a009fc60 con 0x7fe7a009f830 2019-12-05T05:42:55.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:55.028+0000 7fe79f7fe700 1 -- 172.21.15.42:0/1574483531 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7a00a0710 con 0x7fe7a009f830 2019-12-05T05:42:55.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:55.028+0000 7fe79ffff700 1 -- 172.21.15.42:0/1574483531 >> v1:172.21.15.42:6789/0 conn(0x7fe7a00a7740 legacy=0x7fe7a007e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:42:55.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:55.028+0000 7fe79ffff700 1 -- 172.21.15.42:0/1574483531 >> v1:172.21.15.40:6789/0 conn(0x7fe7a009f830 legacy=0x7fe7a014efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:42:55.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:55.028+0000 7fe79ffff700 1 -- 172.21.15.42:0/1574483531 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7a007ef20 con 0x7fe7a008df80 2019-12-05T05:42:55.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:55.028+0000 7fe79ffff700 1 --2- 172.21.15.42:0/1574483531 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe7a008df80 0x7fe7a009f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fe788002f30 tx=0x7fe78800fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:55.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:55.028+0000 7fe79ffff700 1 --2- 172.21.15.42:0/1574483531 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe7a008df80 0x7fe7a009f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fe788002f30 tx=0x7fe78800fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:55.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:55.028+0000 7fe79ffff700 1 --2- 172.21.15.42:0/1574483531 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe7a008df80 0x7fe7a009f2e0 secure :-1 s=READY pgs=276 cs=0 l=1 rx=0x7fe788002f30 tx=0x7fe78800fbf0).ready entity=mon.2 client_cookie=f5c4eaac4f05e999 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:42:55.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:55.028+0000 7fe79ffff700 1 --2- 172.21.15.42:0/1574483531 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe7a008df80 0x7fe7a009f2e0 secure :-1 s=THROTTLE_DONE pgs=276 cs=0 l=1 rx=0x7fe788002f30 tx=0x7fe78800fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:55.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:55.028+0000 7fe79f7fe700 1 -- 172.21.15.42:0/1574483531 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe7880112b0 con 0x7fe7a008df80 2019-12-05T05:42:55.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:55.028+0000 7fe79ffff700 1 --2- 172.21.15.42:0/1574483531 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe7a008df80 0x7fe7a009f2e0 secure :-1 s=THROTTLE_DONE pgs=276 cs=0 l=1 rx=0x7fe788002f30 tx=0x7fe78800fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:55.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:55.028+0000 7fe79ffff700 1 --2- 172.21.15.42:0/1574483531 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe7a008df80 0x7fe7a009f2e0 secure :-1 s=THROTTLE_DONE pgs=276 cs=0 l=1 rx=0x7fe788002f30 tx=0x7fe78800fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:55.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:55.028+0000 7fe79f7fe700 1 -- 172.21.15.42:0/1574483531 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe788008bc0 con 0x7fe7a008df80 2019-12-05T05:42:55.039 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:55.028+0000 7fe79f7fe700 1 -- 172.21.15.42:0/1574483531 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe788011890 con 0x7fe7a008df80 2019-12-05T05:42:55.039 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:55.028+0000 7fe7a6379700 1 -- 172.21.15.42:0/1574483531 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe7a008df80 msgr2=0x7fe7a009f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:42:55.039 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:55.028+0000 7fe7a6379700 1 --2- 172.21.15.42:0/1574483531 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe7a008df80 0x7fe7a009f2e0 secure :-1 s=READY pgs=276 cs=0 l=1 rx=0x7fe788002f30 tx=0x7fe78800fbf0).stop 2019-12-05T05:42:55.040 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:55.028+0000 7fe7a6379700 1 -- 172.21.15.42:0/1574483531 shutdown_connections 2019-12-05T05:42:55.040 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:55.028+0000 7fe7a6379700 1 --2- 172.21.15.42:0/1574483531 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe7a008df80 0x7fe7a009f2e0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:42:55.040 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:55.028+0000 7fe7a6379700 1 -- 172.21.15.42:0/1574483531 shutdown_connections 2019-12-05T05:42:55.040 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:55.028+0000 7fe7a6379700 1 -- 172.21.15.42:0/1574483531 wait complete. 2019-12-05T05:42:55.040 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:55.028+0000 7fe7a6379700 1 -- 172.21.15.42:0/1574483531 >> 172.21.15.42:0/1574483531 conn(0x7fe7a00aabe0 msgr2=0x7fe7a00a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:42:55.044 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:55.032+0000 7fe7a6379700 1 Processor -- start 2019-12-05T05:42:55.044 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:55.032+0000 7fe7a6379700 1 -- start start 2019-12-05T05:42:55.044 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:55.032+0000 7fe7a6379700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe7a0144b50 0x7fe7a007a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:42:55.045 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:55.032+0000 7fe7a6379700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe7a0079600 0x7fe7a0077e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:42:55.045 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:55.032+0000 7fe7a6379700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe7a00783a0 0x7fe7a00a3790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:42:55.045 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:55.032+0000 7fe7a6379700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7a007ad60 con 0x7fe7a0079600 2019-12-05T05:42:55.045 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:55.032+0000 7fe7a6379700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7a007b000 con 0x7fe7a0144b50 2019-12-05T05:42:55.045 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:55.032+0000 7fe7a6379700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7a007b2a0 con 0x7fe7a00783a0 2019-12-05T05:42:55.045 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:55.032+0000 7fe7a4916700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe7a0144b50 0x7fe7a007a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:42:55.045 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:55.032+0000 7fe7a4916700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe7a0144b50 0x7fe7a007a620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:39458/0 (socket says 172.21.15.42:39458) 2019-12-05T05:42:55.046 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:55.036+0000 7fe7a4916700 1 -- 172.21.15.42:0/3146520154 learned_addr learned my addr 172.21.15.42:0/3146520154 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:42:55.046 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:55.036+0000 7fe79ffff700 1 --2- 172.21.15.42:0/3146520154 >> v2:172.21.15.40:3300/0 conn(0x7fe7a0079600 0x7fe7a0077e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:42:55.046 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:55.036+0000 7fe7a5117700 1 --2- 172.21.15.42:0/3146520154 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe7a00783a0 0x7fe7a00a3790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:42:55.046 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:55.036+0000 7fe7a4916700 1 -- 172.21.15.42:0/3146520154 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe7a00783a0 msgr2=0x7fe7a00a3790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:42:55.046 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:55.036+0000 7fe7a4916700 1 --2- 172.21.15.42:0/3146520154 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe7a00783a0 0x7fe7a00a3790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:42:55.046 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:55.036+0000 7fe7a4916700 1 -- 172.21.15.42:0/3146520154 >> v2:172.21.15.40:3300/0 conn(0x7fe7a0079600 msgr2=0x7fe7a0077e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:42:55.046 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:55.036+0000 7fe7a4916700 1 --2- 172.21.15.42:0/3146520154 >> v2:172.21.15.40:3300/0 conn(0x7fe7a0079600 0x7fe7a0077e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:42:55.047 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:55.036+0000 7fe7a4916700 1 -- 172.21.15.42:0/3146520154 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7a00a3ce0 con 0x7fe7a0144b50 2019-12-05T05:42:55.047 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:55.036+0000 7fe7a4916700 1 --2- 172.21.15.42:0/3146520154 >> v2:172.21.15.42:3300/0 conn(0x7fe7a0144b50 0x7fe7a007a620 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fe7a007b1c0 tx=0x7fe790010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:55.047 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:55.036+0000 7fe7a4916700 1 --2- 172.21.15.42:0/3146520154 >> v2:172.21.15.42:3300/0 conn(0x7fe7a0144b50 0x7fe7a007a620 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fe7a007b1c0 tx=0x7fe790010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:55.047 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:55.036+0000 7fe7a4916700 1 --2- 172.21.15.42:0/3146520154 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe7a0144b50 0x7fe7a007a620 secure :-1 s=READY pgs=1321 cs=0 l=1 rx=0x7fe7a007b1c0 tx=0x7fe790010ab0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:42:55.047 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:55.036+0000 7fe7a4916700 1 --2- 172.21.15.42:0/3146520154 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe7a0144b50 0x7fe7a007a620 secure :-1 s=THROTTLE_DONE pgs=1321 cs=0 l=1 rx=0x7fe7a007b1c0 tx=0x7fe790010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:55.047 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:55.036+0000 7fe79d7fa700 1 -- 172.21.15.42:0/3146520154 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe79000cab0 con 0x7fe7a0144b50 2019-12-05T05:42:55.047 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:55.036+0000 7fe7a4916700 1 --2- 172.21.15.42:0/3146520154 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe7a0144b50 0x7fe7a007a620 secure :-1 s=THROTTLE_DONE pgs=1321 cs=0 l=1 rx=0x7fe7a007b1c0 tx=0x7fe790010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:55.047 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:55.036+0000 7fe7a6379700 1 -- 172.21.15.42:0/3146520154 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7a00a4010 con 0x7fe7a0144b50 2019-12-05T05:42:55.048 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:55.036+0000 7fe79d7fa700 1 -- 172.21.15.42:0/3146520154 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe790010250 con 0x7fe7a0144b50 2019-12-05T05:42:55.048 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:55.036+0000 7fe7a4916700 1 --2- 172.21.15.42:0/3146520154 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe7a0144b50 0x7fe7a007a620 secure :-1 s=THROTTLE_DONE pgs=1321 cs=0 l=1 rx=0x7fe7a007b1c0 tx=0x7fe790010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:55.048 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:55.036+0000 7fe79d7fa700 1 -- 172.21.15.42:0/3146520154 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe79000d2b0 con 0x7fe7a0144b50 2019-12-05T05:42:55.048 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:55.036+0000 7fe7a6379700 1 -- 172.21.15.42:0/3146520154 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7a008ed10 con 0x7fe7a0144b50 2019-12-05T05:42:55.048 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:55.036+0000 7fe7a4916700 1 --2- 172.21.15.42:0/3146520154 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe7a0144b50 0x7fe7a007a620 secure :-1 s=THROTTLE_DONE pgs=1321 cs=0 l=1 rx=0x7fe7a007b1c0 tx=0x7fe790010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:55.048 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:55.036+0000 7fe79d7fa700 1 -- 172.21.15.42:0/3146520154 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fe79000d580 con 0x7fe7a0144b50 2019-12-05T05:42:55.049 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:55.036+0000 7fe7a4916700 1 --2- 172.21.15.42:0/3146520154 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe7a0144b50 0x7fe7a007a620 secure :-1 s=THROTTLE_DONE pgs=1321 cs=0 l=1 rx=0x7fe7a007b1c0 tx=0x7fe790010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:55.049 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:55.036+0000 7fe79d7fa700 1 --2- 172.21.15.42:0/3146520154 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe78403ad20 0x7fe78403d230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:42:55.049 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:55.036+0000 7fe79d7fa700 1 -- 172.21.15.42:0/3146520154 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 8267+0+0 (secure 0 0 0) 0x7fe79004abf0 con 0x7fe7a0144b50 2019-12-05T05:42:55.049 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:55.036+0000 7fe79ffff700 1 --2- 172.21.15.42:0/3146520154 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe78403ad20 0x7fe78403d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:42:55.050 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:55.040+0000 7fe79ffff700 1 --2- 172.21.15.42:0/3146520154 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe78403ad20 0x7fe78403d230 crc :-1 s=READY pgs=568 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:42:55.050 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:55.040+0000 7fe7a6379700 1 -- 172.21.15.42:0/3146520154 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7a005d110 con 0x7fe7a0144b50 2019-12-05T05:42:55.052 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:55.048+0000 7fe7a4916700 1 --2- 172.21.15.42:0/3146520154 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe7a0144b50 0x7fe7a007a620 secure :-1 s=THROTTLE_DONE pgs=1321 cs=0 l=1 rx=0x7fe7a007b1c0 tx=0x7fe790010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:55.052 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:55.048+0000 7fe79d7fa700 1 -- 172.21.15.42:0/3146520154 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fe790031c50 con 0x7fe7a0144b50 2019-12-05T05:42:55.220 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:55.216+0000 7fe7a6379700 1 -- 172.21.15.42:0/3146520154 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe7a005b8c0 con 0x7fe7a0144b50 2019-12-05T05:42:55.221 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:55.216+0000 7fe7a4916700 1 --2- 172.21.15.42:0/3146520154 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe7a0144b50 0x7fe7a007a620 secure :-1 s=THROTTLE_DONE pgs=1321 cs=0 l=1 rx=0x7fe7a007b1c0 tx=0x7fe790010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:55.222 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:55.216+0000 7fe79d7fa700 1 -- 172.21.15.42:0/3146520154 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 162 v162) v1 ==== 95+0+1775 (secure 0 0 0) 0x7fe790031eb0 con 0x7fe7a0144b50 2019-12-05T05:42:55.222 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 162 2019-12-05T05:42:55.222 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:42:55.222 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":162,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":162,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:42:48.961753+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7650},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7650":{"gid":7650,"name":"b","rank":0,"incarnation":159,"state":"up:active","state_seq":29,"addr":"172.21.15.40:6833/1029013337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1029013337},{"type":"v1","addr":"172.21.15.40:6833","nonce":1029013337}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:42:55.224 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:55.220+0000 7fe7a6379700 1 -- 172.21.15.42:0/3146520154 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe78403ad20 msgr2=0x7fe78403d230 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:42:55.224 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:55.220+0000 7fe7a6379700 1 --2- 172.21.15.42:0/3146520154 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe78403ad20 0x7fe78403d230 crc :-1 s=READY pgs=568 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:42:55.225 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:55.220+0000 7fe7a6379700 1 -- 172.21.15.42:0/3146520154 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe7a0144b50 msgr2=0x7fe7a007a620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:42:55.225 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:55.220+0000 7fe7a6379700 1 --2- 172.21.15.42:0/3146520154 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe7a0144b50 0x7fe7a007a620 secure :-1 s=READY pgs=1321 cs=0 l=1 rx=0x7fe7a007b1c0 tx=0x7fe790010ab0).stop 2019-12-05T05:42:55.225 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:55.220+0000 7fe7a6379700 1 -- 172.21.15.42:0/3146520154 shutdown_connections 2019-12-05T05:42:55.225 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:55.220+0000 7fe7a6379700 1 --2- 172.21.15.42:0/3146520154 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe78403ad20 0x7fe78403d230 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:42:55.225 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:55.220+0000 7fe7a6379700 1 --2- 172.21.15.42:0/3146520154 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe7a00783a0 0x7fe7a00a3790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:42:55.225 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:55.220+0000 7fe7a6379700 1 --2- 172.21.15.42:0/3146520154 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe7a0144b50 0x7fe7a007a620 unknown :-1 s=CLOSED pgs=1321 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:42:55.226 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:55.220+0000 7fe7a6379700 1 --2- 172.21.15.42:0/3146520154 >> v2:172.21.15.40:3300/0 conn(0x7fe7a0079600 0x7fe7a0077e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:42:55.226 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:55.220+0000 7fe7a6379700 1 -- 172.21.15.42:0/3146520154 shutdown_connections 2019-12-05T05:42:55.226 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:55.220+0000 7fe7a6379700 1 -- 172.21.15.42:0/3146520154 wait complete. 2019-12-05T05:42:55.226 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:55.220+0000 7fe7a6379700 1 -- 172.21.15.42:0/3146520154 >> 172.21.15.42:0/3146520154 conn(0x7fe7a01501e0 msgr2=0x7fe7a007e570 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:42:55.234 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2019-12-05T05:42:57.235 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:42:57.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:57.336+0000 7faeec1da700 1 Processor -- start 2019-12-05T05:42:57.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:57.336+0000 7faeec1da700 1 -- start start 2019-12-05T05:42:57.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:57.340+0000 7faeec1da700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7faee408df80 0x7faee409f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:42:57.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:57.340+0000 7faeec1da700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faee409fc60 con 0x7faee409f830 2019-12-05T05:42:57.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:57.340+0000 7faeec1da700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faee40a0710 con 0x7faee40a7740 2019-12-05T05:42:57.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:57.340+0000 7faeec1da700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faee40a0b00 con 0x7faee408df80 2019-12-05T05:42:57.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:57.340+0000 7faeea777700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7faee40a7740 0x7faee407e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:39516/0 (socket says 172.21.15.42:39516) 2019-12-05T05:42:57.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:57.340+0000 7faeea777700 1 -- 172.21.15.42:0/2780803544 learned_addr learned my addr 172.21.15.42:0/2780803544 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:42:57.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:57.340+0000 7faee9f76700 1 --2- 172.21.15.42:0/2780803544 >> v2:172.21.15.40:3300/0 conn(0x7faee408df80 0x7faee409f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:42:57.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:57.340+0000 7faee9775700 1 -- 172.21.15.42:0/2780803544 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2557422976 0 0) 0x7faee40a0710 con 0x7faee40a7740 2019-12-05T05:42:57.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:57.340+0000 7faee9775700 1 -- 172.21.15.42:0/2780803544 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faed00030b0 con 0x7faee40a7740 2019-12-05T05:42:57.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:57.340+0000 7faee9f76700 1 -- 172.21.15.42:0/2780803544 >> v1:172.21.15.42:6789/0 conn(0x7faee40a7740 legacy=0x7faee407e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:42:57.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:57.340+0000 7faee9f76700 1 -- 172.21.15.42:0/2780803544 >> v1:172.21.15.40:6789/0 conn(0x7faee409f830 legacy=0x7faee414efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:42:57.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:57.340+0000 7faee9775700 1 -- 172.21.15.42:0/2780803544 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1346539113 0 0) 0x7faee409fc60 con 0x7faee409f830 2019-12-05T05:42:57.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:57.340+0000 7faee9f76700 1 -- 172.21.15.42:0/2780803544 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faee407ef20 con 0x7faee408df80 2019-12-05T05:42:57.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:57.340+0000 7faee9f76700 1 --2- 172.21.15.42:0/2780803544 >> v2:172.21.15.40:3300/0 conn(0x7faee408df80 0x7faee409f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7faed4008790 tx=0x7faed400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:57.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:57.340+0000 7faee9f76700 1 --2- 172.21.15.42:0/2780803544 >> v2:172.21.15.40:3300/0 conn(0x7faee408df80 0x7faee409f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7faed4008790 tx=0x7faed400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:57.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:57.340+0000 7faee9f76700 1 --2- 172.21.15.42:0/2780803544 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faee408df80 0x7faee409f2e0 secure :-1 s=READY pgs=1029 cs=0 l=1 rx=0x7faed4008790 tx=0x7faed400fbf0).ready entity=mon.0 client_cookie=3b53dbf00fca2387 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:42:57.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:57.340+0000 7faee9f76700 1 --2- 172.21.15.42:0/2780803544 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faee408df80 0x7faee409f2e0 secure :-1 s=THROTTLE_DONE pgs=1029 cs=0 l=1 rx=0x7faed4008790 tx=0x7faed400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:57.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:57.340+0000 7faee9775700 1 -- 172.21.15.42:0/2780803544 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faed4011270 con 0x7faee408df80 2019-12-05T05:42:57.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:57.340+0000 7faee9f76700 1 --2- 172.21.15.42:0/2780803544 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faee408df80 0x7faee409f2e0 secure :-1 s=THROTTLE_DONE pgs=1029 cs=0 l=1 rx=0x7faed4008790 tx=0x7faed400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:57.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:57.340+0000 7faee9775700 1 -- 172.21.15.42:0/2780803544 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faed4008bc0 con 0x7faee408df80 2019-12-05T05:42:57.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:57.340+0000 7faee9f76700 1 --2- 172.21.15.42:0/2780803544 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faee408df80 0x7faee409f2e0 secure :-1 s=THROTTLE_DONE pgs=1029 cs=0 l=1 rx=0x7faed4008790 tx=0x7faed400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:57.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:57.340+0000 7faee9775700 1 -- 172.21.15.42:0/2780803544 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faed4011850 con 0x7faee408df80 2019-12-05T05:42:57.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:57.340+0000 7faeec1da700 1 -- 172.21.15.42:0/2780803544 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faee408df80 msgr2=0x7faee409f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:42:57.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:57.340+0000 7faeec1da700 1 --2- 172.21.15.42:0/2780803544 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faee408df80 0x7faee409f2e0 secure :-1 s=READY pgs=1029 cs=0 l=1 rx=0x7faed4008790 tx=0x7faed400fbf0).stop 2019-12-05T05:42:57.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:57.344+0000 7faeec1da700 1 -- 172.21.15.42:0/2780803544 shutdown_connections 2019-12-05T05:42:57.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:57.344+0000 7faeec1da700 1 --2- 172.21.15.42:0/2780803544 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faee408df80 0x7faee409f2e0 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:42:57.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:57.344+0000 7faeec1da700 1 -- 172.21.15.42:0/2780803544 shutdown_connections 2019-12-05T05:42:57.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:57.344+0000 7faeec1da700 1 -- 172.21.15.42:0/2780803544 wait complete. 2019-12-05T05:42:57.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:57.344+0000 7faeec1da700 1 -- 172.21.15.42:0/2780803544 >> 172.21.15.42:0/2780803544 conn(0x7faee40aabe0 msgr2=0x7faee40a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:42:57.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:57.344+0000 7faeec1da700 1 Processor -- start 2019-12-05T05:42:57.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:57.344+0000 7faeec1da700 1 -- start start 2019-12-05T05:42:57.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:57.344+0000 7faeec1da700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7faee4064af0 0x7faee4129080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:42:57.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:57.344+0000 7faeec1da700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7faee407bd00 0x7faee407c130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:42:57.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:57.344+0000 7faeec1da700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faee4143010 con 0x7faee408eeb0 2019-12-05T05:42:57.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:57.344+0000 7faeec1da700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faee414fe70 con 0x7faee4064af0 2019-12-05T05:42:57.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:57.344+0000 7faeec1da700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faee4142ad0 con 0x7faee407bd00 2019-12-05T05:42:57.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:57.344+0000 7faee9f76700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7faee4064af0 0x7faee4129080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:42:57.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:57.344+0000 7faee9f76700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7faee4064af0 0x7faee4129080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:39486/0 (socket says 172.21.15.42:39486) 2019-12-05T05:42:57.358 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:57.344+0000 7faee9f76700 1 -- 172.21.15.42:0/529860991 learned_addr learned my addr 172.21.15.42:0/529860991 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:42:57.358 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:57.344+0000 7faeeaf78700 1 --2- 172.21.15.42:0/529860991 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7faee407bd00 0x7faee407c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:42:57.358 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:57.348+0000 7faedaffd700 1 -- 172.21.15.42:0/529860991 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3541413501 0 0) 0x7faee4143010 con 0x7faee408eeb0 2019-12-05T05:42:57.358 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:57.348+0000 7faedaffd700 1 -- 172.21.15.42:0/529860991 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faecc0030b0 con 0x7faee408eeb0 2019-12-05T05:42:57.358 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:57.348+0000 7faee9f76700 1 -- 172.21.15.42:0/529860991 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7faee407bd00 msgr2=0x7faee407c130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:42:57.358 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:57.348+0000 7faee9f76700 1 --2- 172.21.15.42:0/529860991 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7faee407bd00 0x7faee407c130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:42:57.358 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:57.348+0000 7faee9f76700 1 -- 172.21.15.42:0/529860991 >> v1:172.21.15.40:6789/0 conn(0x7faee408eeb0 legacy=0x7faee4128880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:42:57.359 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:57.348+0000 7faee9f76700 1 -- 172.21.15.42:0/529860991 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faee41426a0 con 0x7faee4064af0 2019-12-05T05:42:57.359 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:57.348+0000 7faee9f76700 1 --2- 172.21.15.42:0/529860991 >> v2:172.21.15.42:3300/0 conn(0x7faee4064af0 0x7faee4129080 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7faee4150030 tx=0x7faed4011990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:57.359 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:57.348+0000 7faee9f76700 1 --2- 172.21.15.42:0/529860991 >> v2:172.21.15.42:3300/0 conn(0x7faee4064af0 0x7faee4129080 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7faee4150030 tx=0x7faed4011990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:57.359 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:57.348+0000 7faee9f76700 1 --2- 172.21.15.42:0/529860991 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faee4064af0 0x7faee4129080 secure :-1 s=READY pgs=1323 cs=0 l=1 rx=0x7faee4150030 tx=0x7faed4011990).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:42:57.359 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:57.348+0000 7faee9f76700 1 --2- 172.21.15.42:0/529860991 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faee4064af0 0x7faee4129080 secure :-1 s=THROTTLE_DONE pgs=1323 cs=0 l=1 rx=0x7faee4150030 tx=0x7faed4011990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:57.359 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:57.348+0000 7faedaffd700 1 -- 172.21.15.42:0/529860991 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faed4007340 con 0x7faee4064af0 2019-12-05T05:42:57.359 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:57.348+0000 7faee9f76700 1 --2- 172.21.15.42:0/529860991 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faee4064af0 0x7faee4129080 secure :-1 s=THROTTLE_DONE pgs=1323 cs=0 l=1 rx=0x7faee4150030 tx=0x7faed4011990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:57.360 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:57.348+0000 7faee9f76700 1 --2- 172.21.15.42:0/529860991 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faee4064af0 0x7faee4129080 secure :-1 s=THROTTLE_DONE pgs=1323 cs=0 l=1 rx=0x7faee4150030 tx=0x7faed4011990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:57.360 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:57.348+0000 7faedaffd700 1 -- 172.21.15.42:0/529860991 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faed4011470 con 0x7faee4064af0 2019-12-05T05:42:57.360 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:57.348+0000 7faeec1da700 1 -- 172.21.15.42:0/529860991 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faee407c680 con 0x7faee4064af0 2019-12-05T05:42:57.360 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:57.348+0000 7faedaffd700 1 -- 172.21.15.42:0/529860991 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faed4005540 con 0x7faee4064af0 2019-12-05T05:42:57.360 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:57.348+0000 7faeec1da700 1 -- 172.21.15.42:0/529860991 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faee412ffb0 con 0x7faee4064af0 2019-12-05T05:42:57.360 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:57.348+0000 7faee9f76700 1 --2- 172.21.15.42:0/529860991 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faee4064af0 0x7faee4129080 secure :-1 s=THROTTLE_DONE pgs=1323 cs=0 l=1 rx=0x7faee4150030 tx=0x7faed4011990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:57.360 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:57.348+0000 7faedaffd700 1 -- 172.21.15.42:0/529860991 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7faed4005850 con 0x7faee4064af0 2019-12-05T05:42:57.361 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:57.348+0000 7faee9f76700 1 --2- 172.21.15.42:0/529860991 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faee4064af0 0x7faee4129080 secure :-1 s=THROTTLE_DONE pgs=1323 cs=0 l=1 rx=0x7faee4150030 tx=0x7faed4011990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:57.361 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:57.348+0000 7faedaffd700 1 --2- 172.21.15.42:0/529860991 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7faecc03b2c0 0x7faecc03d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:42:57.361 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:57.348+0000 7faedaffd700 1 -- 172.21.15.42:0/529860991 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 8267+0+0 (secure 0 0 0) 0x7faed404bb50 con 0x7faee4064af0 2019-12-05T05:42:57.361 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:57.348+0000 7faeea777700 1 --2- 172.21.15.42:0/529860991 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7faecc03b2c0 0x7faecc03d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:42:57.361 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:57.352+0000 7faeea777700 1 --2- 172.21.15.42:0/529860991 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7faecc03b2c0 0x7faecc03d7d0 crc :-1 s=READY pgs=570 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:42:57.362 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:57.352+0000 7faeec1da700 1 -- 172.21.15.42:0/529860991 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faee405d110 con 0x7faee4064af0 2019-12-05T05:42:57.365 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:57.360+0000 7faee9f76700 1 --2- 172.21.15.42:0/529860991 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faee4064af0 0x7faee4129080 secure :-1 s=THROTTLE_DONE pgs=1323 cs=0 l=1 rx=0x7faee4150030 tx=0x7faed4011990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:57.365 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:57.360+0000 7faedaffd700 1 -- 172.21.15.42:0/529860991 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7faed4032e30 con 0x7faee4064af0 2019-12-05T05:42:57.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:57.532+0000 7faeec1da700 1 -- 172.21.15.42:0/529860991 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faee405b8c0 con 0x7faee4064af0 2019-12-05T05:42:57.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:57.532+0000 7faee9f76700 1 --2- 172.21.15.42:0/529860991 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faee4064af0 0x7faee4129080 secure :-1 s=THROTTLE_DONE pgs=1323 cs=0 l=1 rx=0x7faee4150030 tx=0x7faed4011990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:42:57.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:57.532+0000 7faedaffd700 1 -- 172.21.15.42:0/529860991 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 163 v163) v1 ==== 95+0+2109 (secure 0 0 0) 0x7faed40307f0 con 0x7faee4064af0 2019-12-05T05:42:57.540 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 163 2019-12-05T05:42:57.541 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:42:57.546 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":163,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7755,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/358497296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":358497296},{"type":"v1","addr":"172.21.15.42:6835","nonce":358497296}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":163}],"filesystems":[{"mdsmap":{"epoch":162,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:42:48.961753+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7650},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7650":{"gid":7650,"name":"b","rank":0,"incarnation":159,"state":"up:active","state_seq":29,"addr":"172.21.15.40:6833/1029013337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1029013337},{"type":"v1","addr":"172.21.15.40:6833","nonce":1029013337}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:42:57.548 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:57.536+0000 7faeec1da700 1 -- 172.21.15.42:0/529860991 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7faecc03b2c0 msgr2=0x7faecc03d7d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:42:57.548 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:57.536+0000 7faeec1da700 1 --2- 172.21.15.42:0/529860991 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7faecc03b2c0 0x7faecc03d7d0 crc :-1 s=READY pgs=570 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:42:57.548 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:57.536+0000 7faeec1da700 1 -- 172.21.15.42:0/529860991 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faee4064af0 msgr2=0x7faee4129080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:42:57.548 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:57.536+0000 7faeec1da700 1 --2- 172.21.15.42:0/529860991 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faee4064af0 0x7faee4129080 secure :-1 s=READY pgs=1323 cs=0 l=1 rx=0x7faee4150030 tx=0x7faed4011990).stop 2019-12-05T05:42:57.549 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:57.536+0000 7faeec1da700 1 -- 172.21.15.42:0/529860991 shutdown_connections 2019-12-05T05:42:57.549 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:57.536+0000 7faeec1da700 1 --2- 172.21.15.42:0/529860991 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7faecc03b2c0 0x7faecc03d7d0 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:42:57.549 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:57.536+0000 7faeec1da700 1 --2- 172.21.15.42:0/529860991 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7faee407bd00 0x7faee407c130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:42:57.549 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:57.536+0000 7faeec1da700 1 --2- 172.21.15.42:0/529860991 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faee4064af0 0x7faee4129080 unknown :-1 s=CLOSED pgs=1323 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:42:57.549 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:57.536+0000 7faeec1da700 1 -- 172.21.15.42:0/529860991 shutdown_connections 2019-12-05T05:42:57.549 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:57.536+0000 7faeec1da700 1 -- 172.21.15.42:0/529860991 wait complete. 2019-12-05T05:42:57.549 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:42:57.536+0000 7faeec1da700 1 -- 172.21.15.42:0/529860991 >> 172.21.15.42:0/529860991 conn(0x7faee40a82a0 msgr2=0x7faee40a9790 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:42:57.554 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a reported in up:standby state 2019-12-05T05:42:57.560 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 47 secs before thrashing 2019-12-05T05:43:11.055 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:43:11.088 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:43:41.151 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:43:41.160 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:43:44.561 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:43:44.669 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:44.660+0000 7ff8f2f43700 1 Processor -- start 2019-12-05T05:43:44.670 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:44.660+0000 7ff8f2f43700 1 -- start start 2019-12-05T05:43:44.670 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:44.660+0000 7ff8f2f43700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff8ec08df80 0x7ff8ec09f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:43:44.670 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:44.660+0000 7ff8f2f43700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff8ec09f830 0x7ff8ec14efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:43:44.670 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:44.660+0000 7ff8f2f43700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8ec09fc60 con 0x7ff8ec0a7740 2019-12-05T05:43:44.670 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:44.660+0000 7ff8f2f43700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8ec0a0b00 con 0x7ff8ec08df80 2019-12-05T05:43:44.670 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:44.660+0000 7ff8f2f43700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8ec08e5b0 con 0x7ff8ec09f830 2019-12-05T05:43:44.671 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:44.660+0000 7ff8f1ce1700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff8ec09f830 0x7ff8ec14efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:43:44.671 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:44.660+0000 7ff8f14e0700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7ff8ec0a7740 0x7ff8ec07e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:39716/0 (socket says 172.21.15.42:39716) 2019-12-05T05:43:44.671 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:44.660+0000 7ff8f14e0700 1 -- 172.21.15.42:0/1871289067 learned_addr learned my addr 172.21.15.42:0/1871289067 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:43:44.671 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:44.660+0000 7ff8f0cdf700 1 --2- 172.21.15.42:0/1871289067 >> v2:172.21.15.40:3300/0 conn(0x7ff8ec08df80 0x7ff8ec09f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:43:44.671 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:44.660+0000 7ff8f1ce1700 1 -- 172.21.15.42:0/1871289067 >> v2:172.21.15.40:3300/0 conn(0x7ff8ec08df80 msgr2=0x7ff8ec09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:43:44.671 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:44.660+0000 7ff8f1ce1700 1 --2- 172.21.15.42:0/1871289067 >> v2:172.21.15.40:3300/0 conn(0x7ff8ec08df80 0x7ff8ec09f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:43:44.671 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:44.660+0000 7ff8f1ce1700 1 -- 172.21.15.42:0/1871289067 >> v1:172.21.15.42:6789/0 conn(0x7ff8ec0a7740 legacy=0x7ff8ec07e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:43:44.672 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:44.660+0000 7ff8e3fff700 1 -- 172.21.15.42:0/1871289067 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2751696893 0 0) 0x7ff8ec09fc60 con 0x7ff8ec0a7740 2019-12-05T05:43:44.672 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:44.660+0000 7ff8f1ce1700 1 -- 172.21.15.42:0/1871289067 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8ec0a0710 con 0x7ff8ec09f830 2019-12-05T05:43:44.672 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:44.660+0000 7ff8f1ce1700 1 --2- 172.21.15.42:0/1871289067 >> v2:172.21.15.42:3300/0 conn(0x7ff8ec09f830 0x7ff8ec14efe0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7ff8e40032b0 tx=0x7ff8e400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:43:44.672 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:44.660+0000 7ff8f1ce1700 1 --2- 172.21.15.42:0/1871289067 >> v2:172.21.15.42:3300/0 conn(0x7ff8ec09f830 0x7ff8ec14efe0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7ff8e40032b0 tx=0x7ff8e400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:43:44.672 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:44.660+0000 7ff8f1ce1700 1 --2- 172.21.15.42:0/1871289067 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff8ec09f830 0x7ff8ec14efe0 secure :-1 s=READY pgs=1327 cs=0 l=1 rx=0x7ff8e40032b0 tx=0x7ff8e400fff0).ready entity=mon.1 client_cookie=11fce00a19be34fe server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:43:44.672 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:44.660+0000 7ff8f1ce1700 1 --2- 172.21.15.42:0/1871289067 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff8ec09f830 0x7ff8ec14efe0 secure :-1 s=THROTTLE_DONE pgs=1327 cs=0 l=1 rx=0x7ff8e40032b0 tx=0x7ff8e400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:43:44.673 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:44.660+0000 7ff8e3fff700 1 -- 172.21.15.42:0/1871289067 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff8e40108c0 con 0x7ff8ec09f830 2019-12-05T05:43:44.673 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:44.660+0000 7ff8f1ce1700 1 --2- 172.21.15.42:0/1871289067 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff8ec09f830 0x7ff8ec14efe0 secure :-1 s=THROTTLE_DONE pgs=1327 cs=0 l=1 rx=0x7ff8e40032b0 tx=0x7ff8e400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:43:44.673 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:44.660+0000 7ff8f1ce1700 1 --2- 172.21.15.42:0/1871289067 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff8ec09f830 0x7ff8ec14efe0 secure :-1 s=THROTTLE_DONE pgs=1327 cs=0 l=1 rx=0x7ff8e40032b0 tx=0x7ff8e400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:43:44.673 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:44.660+0000 7ff8e3fff700 1 -- 172.21.15.42:0/1871289067 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8e4008b50 con 0x7ff8ec09f830 2019-12-05T05:43:44.673 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:44.664+0000 7ff8e3fff700 1 -- 172.21.15.42:0/1871289067 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff8e40110c0 con 0x7ff8ec09f830 2019-12-05T05:43:44.673 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:44.664+0000 7ff8f2f43700 1 -- 172.21.15.42:0/1871289067 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff8ec09f830 msgr2=0x7ff8ec14efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:43:44.674 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:44.664+0000 7ff8f2f43700 1 --2- 172.21.15.42:0/1871289067 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff8ec09f830 0x7ff8ec14efe0 secure :-1 s=READY pgs=1327 cs=0 l=1 rx=0x7ff8e40032b0 tx=0x7ff8e400fff0).stop 2019-12-05T05:43:44.674 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:44.664+0000 7ff8f2f43700 1 -- 172.21.15.42:0/1871289067 shutdown_connections 2019-12-05T05:43:44.674 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:44.664+0000 7ff8f2f43700 1 --2- 172.21.15.42:0/1871289067 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff8ec09f830 0x7ff8ec14efe0 unknown :-1 s=CLOSED pgs=1327 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:43:44.674 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:44.664+0000 7ff8f2f43700 1 --2- 172.21.15.42:0/1871289067 >> v2:172.21.15.40:3300/0 conn(0x7ff8ec08df80 0x7ff8ec09f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:43:44.674 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:44.664+0000 7ff8f2f43700 1 -- 172.21.15.42:0/1871289067 shutdown_connections 2019-12-05T05:43:44.674 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:44.664+0000 7ff8f2f43700 1 -- 172.21.15.42:0/1871289067 wait complete. 2019-12-05T05:43:44.674 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:44.664+0000 7ff8f2f43700 1 -- 172.21.15.42:0/1871289067 >> 172.21.15.42:0/1871289067 conn(0x7ff8ec0aabe0 msgr2=0x7ff8ec0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:43:44.678 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:44.664+0000 7ff8f2f43700 1 Processor -- start 2019-12-05T05:43:44.678 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:44.664+0000 7ff8f2f43700 1 -- start start 2019-12-05T05:43:44.678 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:44.668+0000 7ff8f2f43700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff8ec116500 0x7ff8ec11be80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:43:44.678 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:44.668+0000 7ff8f2f43700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff8ec07ad00 0x7ff8ec07b130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:43:44.678 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:44.668+0000 7ff8f2f43700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff8ec0a6110 0x7ff8ec0a6540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:43:44.679 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:44.668+0000 7ff8f2f43700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8ec0a6e50 con 0x7ff8ec0a6110 2019-12-05T05:43:44.679 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:44.668+0000 7ff8f2f43700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8ec0a70f0 con 0x7ff8ec07ad00 2019-12-05T05:43:44.679 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:44.668+0000 7ff8f2f43700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8ec0a5e00 con 0x7ff8ec116500 2019-12-05T05:43:44.679 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:44.668+0000 7ff8f14e0700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff8ec116500 0x7ff8ec11be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:43:44.679 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:44.668+0000 7ff8f1ce1700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff8ec0a6110 0x7ff8ec0a6540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:43:44.679 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:44.668+0000 7ff8f1ce1700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff8ec0a6110 0x7ff8ec0a6540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.42:55734/0 (socket says 172.21.15.42:55734) 2019-12-05T05:43:44.680 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:44.668+0000 7ff8f14e0700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff8ec116500 0x7ff8ec11be80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:40150/0 (socket says 172.21.15.42:40150) 2019-12-05T05:43:44.680 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:44.668+0000 7ff8f1ce1700 1 -- 172.21.15.42:0/1743979229 learned_addr learned my addr 172.21.15.42:0/1743979229 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:43:44.680 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:44.668+0000 7ff8f0cdf700 1 --2- 172.21.15.42:0/1743979229 >> v2:172.21.15.42:3300/0 conn(0x7ff8ec07ad00 0x7ff8ec07b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:43:44.680 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:44.668+0000 7ff8f1ce1700 1 -- 172.21.15.42:0/1743979229 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff8ec116500 msgr2=0x7ff8ec11be80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:43:44.680 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:44.668+0000 7ff8f1ce1700 1 --2- 172.21.15.42:0/1743979229 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff8ec116500 0x7ff8ec11be80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:43:44.680 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:44.668+0000 7ff8f1ce1700 1 -- 172.21.15.42:0/1743979229 >> v2:172.21.15.42:3300/0 conn(0x7ff8ec07ad00 msgr2=0x7ff8ec07b130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:43:44.680 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:44.668+0000 7ff8f1ce1700 1 --2- 172.21.15.42:0/1743979229 >> v2:172.21.15.42:3300/0 conn(0x7ff8ec07ad00 0x7ff8ec07b130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:43:44.681 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:44.668+0000 7ff8f1ce1700 1 -- 172.21.15.42:0/1743979229 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8ec0a6a90 con 0x7ff8ec0a6110 2019-12-05T05:43:44.681 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:44.668+0000 7ff8f0cdf700 1 --2- 172.21.15.42:0/1743979229 >> v2:172.21.15.42:3300/0 conn(0x7ff8ec07ad00 0x7ff8ec07b130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).send_auth_request state changed! 2019-12-05T05:43:44.681 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:44.668+0000 7ff8f1ce1700 1 --2- 172.21.15.42:0/1743979229 >> v2:172.21.15.40:3300/0 conn(0x7ff8ec0a6110 0x7ff8ec0a6540 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7ff8ec0a7010 tx=0x7ff8e4010d30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:43:44.681 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:44.668+0000 7ff8f1ce1700 1 --2- 172.21.15.42:0/1743979229 >> v2:172.21.15.40:3300/0 conn(0x7ff8ec0a6110 0x7ff8ec0a6540 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7ff8ec0a7010 tx=0x7ff8e4010d30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:43:44.681 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:44.668+0000 7ff8f1ce1700 1 --2- 172.21.15.42:0/1743979229 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff8ec0a6110 0x7ff8ec0a6540 secure :-1 s=READY pgs=1032 cs=0 l=1 rx=0x7ff8ec0a7010 tx=0x7ff8e4010d30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:43:44.681 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:44.668+0000 7ff8f1ce1700 1 --2- 172.21.15.42:0/1743979229 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff8ec0a6110 0x7ff8ec0a6540 secure :-1 s=THROTTLE_DONE pgs=1032 cs=0 l=1 rx=0x7ff8ec0a7010 tx=0x7ff8e4010d30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:43:44.681 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:44.668+0000 7ff8e1ffb700 1 -- 172.21.15.42:0/1743979229 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff8e4005ab0 con 0x7ff8ec0a6110 2019-12-05T05:43:44.682 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:44.668+0000 7ff8f1ce1700 1 --2- 172.21.15.42:0/1743979229 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff8ec0a6110 0x7ff8ec0a6540 secure :-1 s=THROTTLE_DONE pgs=1032 cs=0 l=1 rx=0x7ff8ec0a7010 tx=0x7ff8e4010d30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:43:44.682 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:44.668+0000 7ff8f2f43700 1 -- 172.21.15.42:0/1743979229 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8ec07c800 con 0x7ff8ec0a6110 2019-12-05T05:43:44.682 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:44.668+0000 7ff8e1ffb700 1 -- 172.21.15.42:0/1743979229 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8e4010ac0 con 0x7ff8ec0a6110 2019-12-05T05:43:44.682 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:44.668+0000 7ff8f1ce1700 1 --2- 172.21.15.42:0/1743979229 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff8ec0a6110 0x7ff8ec0a6540 secure :-1 s=THROTTLE_DONE pgs=1032 cs=0 l=1 rx=0x7ff8ec0a7010 tx=0x7ff8e4010d30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:43:44.682 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:44.668+0000 7ff8e1ffb700 1 -- 172.21.15.42:0/1743979229 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff8e400f680 con 0x7ff8ec0a6110 2019-12-05T05:43:44.682 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:44.668+0000 7ff8f2f43700 1 -- 172.21.15.42:0/1743979229 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8ec08ed10 con 0x7ff8ec0a6110 2019-12-05T05:43:44.682 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:44.672+0000 7ff8f1ce1700 1 --2- 172.21.15.42:0/1743979229 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff8ec0a6110 0x7ff8ec0a6540 secure :-1 s=THROTTLE_DONE pgs=1032 cs=0 l=1 rx=0x7ff8ec0a7010 tx=0x7ff8e4010d30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:43:44.683 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:44.672+0000 7ff8e1ffb700 1 -- 172.21.15.42:0/1743979229 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7ff8e400f880 con 0x7ff8ec0a6110 2019-12-05T05:43:44.683 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:44.672+0000 7ff8f1ce1700 1 --2- 172.21.15.42:0/1743979229 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff8ec0a6110 0x7ff8ec0a6540 secure :-1 s=THROTTLE_DONE pgs=1032 cs=0 l=1 rx=0x7ff8ec0a7010 tx=0x7ff8e4010d30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:43:44.683 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:44.672+0000 7ff8e1ffb700 1 --2- 172.21.15.42:0/1743979229 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff8cc03ae10 0x7ff8cc03d320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:43:44.683 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:44.672+0000 7ff8e1ffb700 1 -- 172.21.15.42:0/1743979229 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 8267+0+0 (secure 0 0 0) 0x7ff8e408f350 con 0x7ff8ec0a6110 2019-12-05T05:43:44.683 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:44.672+0000 7ff8f14e0700 1 --2- 172.21.15.42:0/1743979229 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff8cc03ae10 0x7ff8cc03d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:43:44.683 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:44.672+0000 7ff8f14e0700 1 --2- 172.21.15.42:0/1743979229 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff8cc03ae10 0x7ff8cc03d320 crc :-1 s=READY pgs=572 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:43:44.684 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:44.676+0000 7ff8f2f43700 1 -- 172.21.15.42:0/1743979229 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8ec05d110 con 0x7ff8ec0a6110 2019-12-05T05:43:44.696 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:44.684+0000 7ff8f1ce1700 1 --2- 172.21.15.42:0/1743979229 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff8ec0a6110 0x7ff8ec0a6540 secure :-1 s=THROTTLE_DONE pgs=1032 cs=0 l=1 rx=0x7ff8ec0a7010 tx=0x7ff8e4010d30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:43:44.696 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:44.684+0000 7ff8e1ffb700 1 -- 172.21.15.42:0/1743979229 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7ff8e4077200 con 0x7ff8ec0a6110 2019-12-05T05:43:44.871 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:44.868+0000 7ff8f2f43700 1 -- 172.21.15.42:0/1743979229 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff8ec05b8c0 con 0x7ff8ec0a6110 2019-12-05T05:43:44.873 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:44.868+0000 7ff8f1ce1700 1 --2- 172.21.15.42:0/1743979229 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff8ec0a6110 0x7ff8ec0a6540 secure :-1 s=THROTTLE_DONE pgs=1032 cs=0 l=1 rx=0x7ff8ec0a7010 tx=0x7ff8e4010d30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:43:44.873 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:44.868+0000 7ff8e1ffb700 1 -- 172.21.15.42:0/1743979229 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 163 v163) v1 ==== 95+0+2109 (secure 0 0 0) 0x7ff8e4007160 con 0x7ff8ec0a6110 2019-12-05T05:43:44.873 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 163 2019-12-05T05:43:44.873 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:43:44.873 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":163,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7755,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/358497296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":358497296},{"type":"v1","addr":"172.21.15.42:6835","nonce":358497296}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":163}],"filesystems":[{"mdsmap":{"epoch":162,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:42:48.961753+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7650},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7650":{"gid":7650,"name":"b","rank":0,"incarnation":159,"state":"up:active","state_seq":29,"addr":"172.21.15.40:6833/1029013337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1029013337},{"type":"v1","addr":"172.21.15.40:6833","nonce":1029013337}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:43:44.876 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:44.868+0000 7ff8f2f43700 1 -- 172.21.15.42:0/1743979229 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff8cc03ae10 msgr2=0x7ff8cc03d320 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:43:44.876 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:44.868+0000 7ff8f2f43700 1 --2- 172.21.15.42:0/1743979229 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff8cc03ae10 0x7ff8cc03d320 crc :-1 s=READY pgs=572 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:43:44.876 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:44.868+0000 7ff8f2f43700 1 -- 172.21.15.42:0/1743979229 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff8ec0a6110 msgr2=0x7ff8ec0a6540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:43:44.876 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:44.868+0000 7ff8f2f43700 1 --2- 172.21.15.42:0/1743979229 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff8ec0a6110 0x7ff8ec0a6540 secure :-1 s=READY pgs=1032 cs=0 l=1 rx=0x7ff8ec0a7010 tx=0x7ff8e4010d30).stop 2019-12-05T05:43:44.876 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:44.872+0000 7ff8f2f43700 1 -- 172.21.15.42:0/1743979229 shutdown_connections 2019-12-05T05:43:44.876 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:44.872+0000 7ff8f2f43700 1 --2- 172.21.15.42:0/1743979229 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff8cc03ae10 0x7ff8cc03d320 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:43:44.876 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:44.872+0000 7ff8f2f43700 1 --2- 172.21.15.42:0/1743979229 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff8ec0a6110 0x7ff8ec0a6540 unknown :-1 s=CLOSED pgs=1032 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:43:44.877 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:44.872+0000 7ff8f2f43700 1 --2- 172.21.15.42:0/1743979229 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff8ec116500 0x7ff8ec11be80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:43:44.877 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:44.872+0000 7ff8f2f43700 1 --2- 172.21.15.42:0/1743979229 >> v2:172.21.15.42:3300/0 conn(0x7ff8ec07ad00 0x7ff8ec07b130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:43:44.877 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:44.872+0000 7ff8f2f43700 1 -- 172.21.15.42:0/1743979229 shutdown_connections 2019-12-05T05:43:44.877 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:44.872+0000 7ff8f2f43700 1 -- 172.21.15.42:0/1743979229 wait complete. 2019-12-05T05:43:44.877 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:44.872+0000 7ff8f2f43700 1 -- 172.21.15.42:0/1743979229 >> 172.21.15.42:0/1743979229 conn(0x7ff8ec11c610 msgr2=0x7ff8ec059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:43:44.888 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.b (rank=0) 2019-12-05T05:43:44.889 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2019-12-05T05:43:44.889 INFO:teuthology.orchestra.run:waiting for 300 2019-12-05T05:43:44.905 INFO:tasks.ceph.mds.b:Stopped 2019-12-05T05:43:44.905 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:43:45.004 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:44.992+0000 7fee85f49700 1 Processor -- start 2019-12-05T05:43:45.004 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:44.992+0000 7fee85f49700 1 -- start start 2019-12-05T05:43:45.005 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:44.996+0000 7fee85f49700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fee800a7740 0x7fee8008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:43:45.005 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:44.996+0000 7fee85f49700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fee8009fae0 0x7fee8007eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:43:45.005 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:44.996+0000 7fee85f49700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee800a0710 con 0x7fee8007e720 2019-12-05T05:43:45.005 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:44.996+0000 7fee85f49700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee800a0b00 con 0x7fee800a7740 2019-12-05T05:43:45.005 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:44.996+0000 7fee85f49700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee8008e5b0 con 0x7fee8009fae0 2019-12-05T05:43:45.005 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:44.996+0000 7fee7ffff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fee800a7740 0x7fee8008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:43:45.005 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:44.996+0000 7fee7ffff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fee800a7740 0x7fee8008df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:39694/0 (socket says 172.21.15.42:39694) 2019-12-05T05:43:45.006 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:44.996+0000 7fee7ffff700 1 -- 172.21.15.42:0/2205991137 learned_addr learned my addr 172.21.15.42:0/2205991137 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:43:45.006 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:44.996+0000 7fee84ce7700 1 --2- 172.21.15.42:0/2205991137 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fee8009fae0 0x7fee8007eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:43:45.006 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:44.996+0000 7fee7ffff700 1 -- 172.21.15.42:0/2205991137 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fee8009fae0 msgr2=0x7fee8007eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:43:45.006 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:44.996+0000 7fee7ffff700 1 --2- 172.21.15.42:0/2205991137 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fee8009fae0 0x7fee8007eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:43:45.006 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:44.996+0000 7fee7ffff700 1 -- 172.21.15.42:0/2205991137 >> v1:172.21.15.40:6789/0 conn(0x7fee8007e720 legacy=0x7fee8009f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:43:45.006 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:44.996+0000 7fee7ffff700 1 -- 172.21.15.42:0/2205991137 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee8007f0a0 con 0x7fee800a7740 2019-12-05T05:43:45.006 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:44.996+0000 7fee7ffff700 1 --2- 172.21.15.42:0/2205991137 >> v2:172.21.15.42:3300/0 conn(0x7fee800a7740 0x7fee8008df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fee70008b10 tx=0x7fee700100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:43:45.007 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:44.996+0000 7fee7ffff700 1 --2- 172.21.15.42:0/2205991137 >> v2:172.21.15.42:3300/0 conn(0x7fee800a7740 0x7fee8008df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fee70008b10 tx=0x7fee700100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:43:45.007 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:44.996+0000 7fee7ffff700 1 --2- 172.21.15.42:0/2205991137 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fee800a7740 0x7fee8008df80 secure :-1 s=READY pgs=1328 cs=0 l=1 rx=0x7fee70008b10 tx=0x7fee700100b0).ready entity=mon.1 client_cookie=8082d55e19031689 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:43:45.007 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:44.996+0000 7fee7ffff700 1 --2- 172.21.15.42:0/2205991137 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fee800a7740 0x7fee8008df80 secure :-1 s=THROTTLE_DONE pgs=1328 cs=0 l=1 rx=0x7fee70008b10 tx=0x7fee700100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:43:45.007 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:44.996+0000 7fee7effd700 1 -- 172.21.15.42:0/2205991137 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fee70010da0 con 0x7fee800a7740 2019-12-05T05:43:45.007 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:44.996+0000 7fee7ffff700 1 --2- 172.21.15.42:0/2205991137 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fee800a7740 0x7fee8008df80 secure :-1 s=THROTTLE_DONE pgs=1328 cs=0 l=1 rx=0x7fee70008b10 tx=0x7fee700100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:43:45.007 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:44.996+0000 7fee7ffff700 1 --2- 172.21.15.42:0/2205991137 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fee800a7740 0x7fee8008df80 secure :-1 s=THROTTLE_DONE pgs=1328 cs=0 l=1 rx=0x7fee70008b10 tx=0x7fee700100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:43:45.007 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:44.996+0000 7fee7effd700 1 -- 172.21.15.42:0/2205991137 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee700109c0 con 0x7fee800a7740 2019-12-05T05:43:45.008 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:44.996+0000 7fee7effd700 1 -- 172.21.15.42:0/2205991137 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fee700114a0 con 0x7fee800a7740 2019-12-05T05:43:45.008 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:44.996+0000 7fee85f49700 1 -- 172.21.15.42:0/2205991137 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fee800a7740 msgr2=0x7fee8008df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:43:45.008 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:44.996+0000 7fee85f49700 1 --2- 172.21.15.42:0/2205991137 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fee800a7740 0x7fee8008df80 secure :-1 s=READY pgs=1328 cs=0 l=1 rx=0x7fee70008b10 tx=0x7fee700100b0).stop 2019-12-05T05:43:45.008 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:45.000+0000 7fee85f49700 1 -- 172.21.15.42:0/2205991137 shutdown_connections 2019-12-05T05:43:45.008 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:45.000+0000 7fee85f49700 1 --2- 172.21.15.42:0/2205991137 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fee8009fae0 0x7fee8007eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:43:45.008 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:45.000+0000 7fee85f49700 1 --2- 172.21.15.42:0/2205991137 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fee800a7740 0x7fee8008df80 unknown :-1 s=CLOSED pgs=1328 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:43:45.008 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:45.000+0000 7fee85f49700 1 -- 172.21.15.42:0/2205991137 shutdown_connections 2019-12-05T05:43:45.009 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:45.000+0000 7fee85f49700 1 -- 172.21.15.42:0/2205991137 wait complete. 2019-12-05T05:43:45.009 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:45.000+0000 7fee85f49700 1 -- 172.21.15.42:0/2205991137 >> 172.21.15.42:0/2205991137 conn(0x7fee800aabe0 msgr2=0x7fee800a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:43:45.011 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:45.000+0000 7fee85f49700 1 Processor -- start 2019-12-05T05:43:45.011 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:45.000+0000 7fee85f49700 1 -- start start 2019-12-05T05:43:45.011 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:45.004+0000 7fee85f49700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fee80144890 0x7fee80126e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:43:45.011 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:45.004+0000 7fee85f49700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee80127c60 con 0x7fee80115a70 2019-12-05T05:43:45.012 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:45.004+0000 7fee85f49700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee80127360 con 0x7fee801238f0 2019-12-05T05:43:45.012 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:45.004+0000 7fee85f49700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee80127720 con 0x7fee80144890 2019-12-05T05:43:45.012 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:45.004+0000 7fee7ffff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fee80144890 0x7fee80126e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:43:45.012 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:45.004+0000 7fee7ffff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fee80144890 0x7fee80126e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:39700/0 (socket says 172.21.15.42:39700) 2019-12-05T05:43:45.012 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:45.004+0000 7fee7ffff700 1 -- 172.21.15.42:0/2836959144 learned_addr learned my addr 172.21.15.42:0/2836959144 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:43:45.012 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:45.004+0000 7fee7ffff700 1 -- 172.21.15.42:0/2836959144 >> v1:172.21.15.42:6789/0 conn(0x7fee801238f0 legacy=0x7fee80123d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:43:45.013 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:45.004+0000 7fee7cff9700 1 -- 172.21.15.42:0/2836959144 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2108116538 0 0) 0x7fee80127360 con 0x7fee801238f0 2019-12-05T05:43:45.013 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:45.004+0000 7fee7ffff700 1 -- 172.21.15.42:0/2836959144 >> v1:172.21.15.40:6789/0 conn(0x7fee80115a70 legacy=0x7fee800a3800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:43:45.013 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:45.004+0000 7fee7ffff700 1 -- 172.21.15.42:0/2836959144 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee800a4000 con 0x7fee80144890 2019-12-05T05:43:45.013 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:45.004+0000 7fee7ffff700 1 --2- 172.21.15.42:0/2836959144 >> v2:172.21.15.42:3300/0 conn(0x7fee80144890 0x7fee80126e10 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fee70008b10 tx=0x7fee70010c90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:43:45.015 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:45.004+0000 7fee7ffff700 1 --2- 172.21.15.42:0/2836959144 >> v2:172.21.15.42:3300/0 conn(0x7fee80144890 0x7fee80126e10 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fee70008b10 tx=0x7fee70010c90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:43:45.016 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:45.004+0000 7fee7ffff700 1 --2- 172.21.15.42:0/2836959144 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fee80144890 0x7fee80126e10 secure :-1 s=READY pgs=1330 cs=0 l=1 rx=0x7fee70008b10 tx=0x7fee70010c90).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:43:45.016 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:45.004+0000 7fee7ffff700 1 --2- 172.21.15.42:0/2836959144 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fee80144890 0x7fee80126e10 secure :-1 s=THROTTLE_DONE pgs=1330 cs=0 l=1 rx=0x7fee70008b10 tx=0x7fee70010c90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:43:45.016 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:45.004+0000 7fee7cff9700 1 -- 172.21.15.42:0/2836959144 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fee70011bb0 con 0x7fee80144890 2019-12-05T05:43:45.016 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:45.004+0000 7fee7ffff700 1 --2- 172.21.15.42:0/2836959144 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fee80144890 0x7fee80126e10 secure :-1 s=THROTTLE_DONE pgs=1330 cs=0 l=1 rx=0x7fee70008b10 tx=0x7fee70010c90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:43:45.017 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:45.004+0000 7fee7cff9700 1 -- 172.21.15.42:0/2836959144 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee700110c0 con 0x7fee80144890 2019-12-05T05:43:45.017 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:45.004+0000 7fee85f49700 1 -- 172.21.15.42:0/2836959144 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee8012f270 con 0x7fee80144890 2019-12-05T05:43:45.017 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:45.004+0000 7fee7ffff700 1 --2- 172.21.15.42:0/2836959144 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fee80144890 0x7fee80126e10 secure :-1 s=THROTTLE_DONE pgs=1330 cs=0 l=1 rx=0x7fee70008b10 tx=0x7fee70010c90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:43:45.017 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:45.004+0000 7fee7cff9700 1 -- 172.21.15.42:0/2836959144 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fee70005ab0 con 0x7fee80144890 2019-12-05T05:43:45.017 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:45.004+0000 7fee85f49700 1 -- 172.21.15.42:0/2836959144 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee8012f820 con 0x7fee80144890 2019-12-05T05:43:45.017 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:45.004+0000 7fee7ffff700 1 --2- 172.21.15.42:0/2836959144 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fee80144890 0x7fee80126e10 secure :-1 s=THROTTLE_DONE pgs=1330 cs=0 l=1 rx=0x7fee70008b10 tx=0x7fee70010c90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:43:45.018 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:45.004+0000 7fee7cff9700 1 -- 172.21.15.42:0/2836959144 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fee7000f280 con 0x7fee80144890 2019-12-05T05:43:45.018 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:45.004+0000 7fee7ffff700 1 --2- 172.21.15.42:0/2836959144 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fee80144890 0x7fee80126e10 secure :-1 s=THROTTLE_DONE pgs=1330 cs=0 l=1 rx=0x7fee70008b10 tx=0x7fee70010c90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:43:45.019 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:45.008+0000 7fee7cff9700 1 --2- 172.21.15.42:0/2836959144 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fee6403ac50 0x7fee6403d160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:43:45.019 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:45.008+0000 7fee7cff9700 1 -- 172.21.15.42:0/2836959144 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 8267+0+0 (secure 0 0 0) 0x7fee7004b640 con 0x7fee80144890 2019-12-05T05:43:45.019 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:45.008+0000 7fee7f7fe700 1 --2- 172.21.15.42:0/2836959144 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fee6403ac50 0x7fee6403d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:43:45.019 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:45.008+0000 7fee7f7fe700 1 --2- 172.21.15.42:0/2836959144 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fee6403ac50 0x7fee6403d160 crc :-1 s=READY pgs=573 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:43:45.019 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:45.008+0000 7fee85f49700 1 -- 172.21.15.42:0/2836959144 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee8005d110 con 0x7fee80144890 2019-12-05T05:43:45.022 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:45.016+0000 7fee7ffff700 1 --2- 172.21.15.42:0/2836959144 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fee80144890 0x7fee80126e10 secure :-1 s=THROTTLE_DONE pgs=1330 cs=0 l=1 rx=0x7fee70008b10 tx=0x7fee70010c90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:43:45.022 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:45.016+0000 7fee7cff9700 1 -- 172.21.15.42:0/2836959144 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fee70005d50 con 0x7fee80144890 2019-12-05T05:43:45.187 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:45.180+0000 7fee85f49700 1 -- 172.21.15.42:0/2836959144 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fee8005b8c0 con 0x7fee80144890 2019-12-05T05:43:45.187 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:45.184+0000 7fee7ffff700 1 --2- 172.21.15.42:0/2836959144 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fee80144890 0x7fee80126e10 secure :-1 s=THROTTLE_DONE pgs=1330 cs=0 l=1 rx=0x7fee70008b10 tx=0x7fee70010c90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:43:45.187 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:45.184+0000 7fee7cff9700 1 -- 172.21.15.42:0/2836959144 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 163 v163) v1 ==== 95+0+2109 (secure 0 0 0) 0x7fee700331b0 con 0x7fee80144890 2019-12-05T05:43:45.188 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 163 2019-12-05T05:43:45.188 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:43:45.188 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":163,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7755,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/358497296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":358497296},{"type":"v1","addr":"172.21.15.42:6835","nonce":358497296}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":163}],"filesystems":[{"mdsmap":{"epoch":162,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:42:48.961753+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7650},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7650":{"gid":7650,"name":"b","rank":0,"incarnation":159,"state":"up:active","state_seq":29,"addr":"172.21.15.40:6833/1029013337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1029013337},{"type":"v1","addr":"172.21.15.40:6833","nonce":1029013337}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:43:45.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:45.188+0000 7fee85f49700 1 -- 172.21.15.42:0/2836959144 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fee6403ac50 msgr2=0x7fee6403d160 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:43:45.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:45.188+0000 7fee85f49700 1 --2- 172.21.15.42:0/2836959144 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fee6403ac50 0x7fee6403d160 crc :-1 s=READY pgs=573 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:43:45.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:45.188+0000 7fee85f49700 1 -- 172.21.15.42:0/2836959144 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fee80144890 msgr2=0x7fee80126e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:43:45.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:45.188+0000 7fee85f49700 1 --2- 172.21.15.42:0/2836959144 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fee80144890 0x7fee80126e10 secure :-1 s=READY pgs=1330 cs=0 l=1 rx=0x7fee70008b10 tx=0x7fee70010c90).stop 2019-12-05T05:43:45.193 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:45.188+0000 7fee85f49700 1 -- 172.21.15.42:0/2836959144 shutdown_connections 2019-12-05T05:43:45.193 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:45.188+0000 7fee85f49700 1 --2- 172.21.15.42:0/2836959144 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fee6403ac50 0x7fee6403d160 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:43:45.193 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:45.188+0000 7fee85f49700 1 --2- 172.21.15.42:0/2836959144 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fee80144890 0x7fee80126e10 unknown :-1 s=CLOSED pgs=1330 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:43:45.193 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:45.188+0000 7fee85f49700 1 -- 172.21.15.42:0/2836959144 shutdown_connections 2019-12-05T05:43:45.193 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:45.188+0000 7fee85f49700 1 -- 172.21.15.42:0/2836959144 wait complete. 2019-12-05T05:43:45.193 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:45.188+0000 7fee85f49700 1 -- 172.21.15.42:0/2836959144 >> 172.21.15.42:0/2836959144 conn(0x7fee800aabe0 msgr2=0x7fee8009f430 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:43:45.205 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T05:43:47.206 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:43:47.315 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:47.304+0000 7fe90ef47700 1 Processor -- start 2019-12-05T05:43:47.315 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:47.304+0000 7fe90ef47700 1 -- start start 2019-12-05T05:43:47.315 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:47.304+0000 7fe90ef47700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe9080a7740 0x7fe90808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:43:47.315 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:47.304+0000 7fe90ef47700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9080a0710 con 0x7fe90807e720 2019-12-05T05:43:47.315 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:47.304+0000 7fe90ef47700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9080a9ba0 con 0x7fe90809fae0 2019-12-05T05:43:47.315 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:47.304+0000 7fe90ef47700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9080a0b00 con 0x7fe9080a7740 2019-12-05T05:43:47.316 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:47.304+0000 7fe90d4e4700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe9080a7740 0x7fe90808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:43:47.316 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:47.304+0000 7fe90dce5700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fe90809fae0 0x7fe90814efe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:39756/0 (socket says 172.21.15.42:39756) 2019-12-05T05:43:47.316 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:47.304+0000 7fe90dce5700 1 -- 172.21.15.42:0/1802848884 learned_addr learned my addr 172.21.15.42:0/1802848884 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:43:47.316 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:47.308+0000 7fe90d4e4700 1 -- 172.21.15.42:0/1802848884 >> v1:172.21.15.42:6789/0 conn(0x7fe90809fae0 legacy=0x7fe90814efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:43:47.316 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:47.308+0000 7fe90d4e4700 1 -- 172.21.15.42:0/1802848884 >> v1:172.21.15.40:6789/0 conn(0x7fe90807e720 legacy=0x7fe90809f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:43:47.316 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:47.308+0000 7fe8fffff700 1 -- 172.21.15.42:0/1802848884 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1656109317 0 0) 0x7fe9080a9ba0 con 0x7fe90809fae0 2019-12-05T05:43:47.316 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:47.308+0000 7fe90d4e4700 1 -- 172.21.15.42:0/1802848884 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe90807eb50 con 0x7fe9080a7740 2019-12-05T05:43:47.317 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:47.308+0000 7fe90d4e4700 1 --2- 172.21.15.42:0/1802848884 >> v2:172.21.15.42:3300/0 conn(0x7fe9080a7740 0x7fe90808df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fe904008790 tx=0x7fe90400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:43:47.317 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:47.308+0000 7fe90d4e4700 1 --2- 172.21.15.42:0/1802848884 >> v2:172.21.15.42:3300/0 conn(0x7fe9080a7740 0x7fe90808df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fe904008790 tx=0x7fe90400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:43:47.317 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:47.308+0000 7fe90d4e4700 1 --2- 172.21.15.42:0/1802848884 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe9080a7740 0x7fe90808df80 secure :-1 s=READY pgs=1332 cs=0 l=1 rx=0x7fe904008790 tx=0x7fe90400fbf0).ready entity=mon.1 client_cookie=fafd9624d1262949 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:43:47.317 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:47.308+0000 7fe90d4e4700 1 --2- 172.21.15.42:0/1802848884 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe9080a7740 0x7fe90808df80 secure :-1 s=THROTTLE_DONE pgs=1332 cs=0 l=1 rx=0x7fe904008790 tx=0x7fe90400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:43:47.317 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:47.308+0000 7fe8fffff700 1 -- 172.21.15.42:0/1802848884 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe904011270 con 0x7fe9080a7740 2019-12-05T05:43:47.317 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:47.308+0000 7fe90d4e4700 1 --2- 172.21.15.42:0/1802848884 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe9080a7740 0x7fe90808df80 secure :-1 s=THROTTLE_DONE pgs=1332 cs=0 l=1 rx=0x7fe904008790 tx=0x7fe90400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:43:47.317 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:47.308+0000 7fe90d4e4700 1 --2- 172.21.15.42:0/1802848884 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe9080a7740 0x7fe90808df80 secure :-1 s=THROTTLE_DONE pgs=1332 cs=0 l=1 rx=0x7fe904008790 tx=0x7fe90400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:43:47.318 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:47.308+0000 7fe8fffff700 1 -- 172.21.15.42:0/1802848884 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe904008bc0 con 0x7fe9080a7740 2019-12-05T05:43:47.318 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:47.308+0000 7fe90ef47700 1 -- 172.21.15.42:0/1802848884 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe9080a7740 msgr2=0x7fe90808df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:43:47.318 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:47.308+0000 7fe8fffff700 1 -- 172.21.15.42:0/1802848884 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe904011850 con 0x7fe9080a7740 2019-12-05T05:43:47.318 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:47.308+0000 7fe90ef47700 1 --2- 172.21.15.42:0/1802848884 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe9080a7740 0x7fe90808df80 secure :-1 s=READY pgs=1332 cs=0 l=1 rx=0x7fe904008790 tx=0x7fe90400fbf0).stop 2019-12-05T05:43:47.318 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:47.308+0000 7fe90ef47700 1 -- 172.21.15.42:0/1802848884 shutdown_connections 2019-12-05T05:43:47.318 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:47.308+0000 7fe90ef47700 1 --2- 172.21.15.42:0/1802848884 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe9080a7740 0x7fe90808df80 unknown :-1 s=CLOSED pgs=1332 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:43:47.318 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:47.308+0000 7fe90ef47700 1 -- 172.21.15.42:0/1802848884 shutdown_connections 2019-12-05T05:43:47.319 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:47.308+0000 7fe90ef47700 1 -- 172.21.15.42:0/1802848884 wait complete. 2019-12-05T05:43:47.319 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:47.308+0000 7fe90ef47700 1 -- 172.21.15.42:0/1802848884 >> 172.21.15.42:0/1802848884 conn(0x7fe9080aabe0 msgr2=0x7fe9080a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:43:47.322 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:47.312+0000 7fe90ef47700 1 Processor -- start 2019-12-05T05:43:47.322 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:47.312+0000 7fe90ef47700 1 -- start start 2019-12-05T05:43:47.323 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:47.312+0000 7fe90ef47700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe90811ba70 0x7fe90811bea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:43:47.323 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:47.312+0000 7fe90ef47700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe908138a80 con 0x7fe908118390 2019-12-05T05:43:47.323 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:47.312+0000 7fe90ef47700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe908139cc0 con 0x7fe90811a9d0 2019-12-05T05:43:47.323 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:47.312+0000 7fe90ef47700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe908117bf0 con 0x7fe90811ba70 2019-12-05T05:43:47.323 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:47.312+0000 7fe90d4e4700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fe90811a9d0 0x7fe90812d060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:39760/0 (socket says 172.21.15.42:39760) 2019-12-05T05:43:47.323 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:47.312+0000 7fe90d4e4700 1 -- 172.21.15.42:0/2866370667 learned_addr learned my addr 172.21.15.42:0/2866370667 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:43:47.323 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:47.312+0000 7fe90dce5700 1 --2- 172.21.15.42:0/2866370667 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe90811ba70 0x7fe90811bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:43:47.324 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:47.312+0000 7fe8fdffb700 1 -- 172.21.15.42:0/2866370667 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1252985756 0 0) 0x7fe908139cc0 con 0x7fe90811a9d0 2019-12-05T05:43:47.324 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:47.312+0000 7fe8fdffb700 1 -- 172.21.15.42:0/2866370667 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8e80030b0 con 0x7fe90811a9d0 2019-12-05T05:43:47.324 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:47.312+0000 7fe8fdffb700 1 -- 172.21.15.42:0/2866370667 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1942939158 0 0) 0x7fe908138a80 con 0x7fe908118390 2019-12-05T05:43:47.324 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:47.312+0000 7fe8fdffb700 1 -- 172.21.15.42:0/2866370667 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe908139cc0 con 0x7fe908118390 2019-12-05T05:43:47.324 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:47.312+0000 7fe8fdffb700 1 -- 172.21.15.42:0/2866370667 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 2506550482 0 0) 0x7fe8e80030b0 con 0x7fe90811a9d0 2019-12-05T05:43:47.324 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:47.312+0000 7fe90dce5700 1 -- 172.21.15.42:0/2866370667 >> v1:172.21.15.42:6789/0 conn(0x7fe90811a9d0 legacy=0x7fe90812d060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:43:47.324 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:47.312+0000 7fe90dce5700 1 -- 172.21.15.42:0/2866370667 >> v1:172.21.15.40:6789/0 conn(0x7fe908118390 legacy=0x7fe90807db50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:43:47.325 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:47.312+0000 7fe90dce5700 1 -- 172.21.15.42:0/2866370667 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe90813af00 con 0x7fe90811ba70 2019-12-05T05:43:47.325 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:47.312+0000 7fe90dce5700 1 --2- 172.21.15.42:0/2866370667 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe90811ba70 0x7fe90811bea0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fe904007e50 tx=0x7fe904010100).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:43:47.325 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:47.312+0000 7fe90dce5700 1 --2- 172.21.15.42:0/2866370667 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe90811ba70 0x7fe90811bea0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fe904007e50 tx=0x7fe904010100).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:43:47.325 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:47.312+0000 7fe90dce5700 1 --2- 172.21.15.42:0/2866370667 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe90811ba70 0x7fe90811bea0 secure :-1 s=READY pgs=279 cs=0 l=1 rx=0x7fe904007e50 tx=0x7fe904010100).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:43:47.325 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:47.312+0000 7fe90dce5700 1 --2- 172.21.15.42:0/2866370667 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe90811ba70 0x7fe90811bea0 secure :-1 s=THROTTLE_DONE pgs=279 cs=0 l=1 rx=0x7fe904007e50 tx=0x7fe904010100).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:43:47.325 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:47.312+0000 7fe8fdffb700 1 -- 172.21.15.42:0/2866370667 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe904011270 con 0x7fe90811ba70 2019-12-05T05:43:47.325 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:47.312+0000 7fe90dce5700 1 --2- 172.21.15.42:0/2866370667 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe90811ba70 0x7fe90811bea0 secure :-1 s=THROTTLE_DONE pgs=279 cs=0 l=1 rx=0x7fe904007e50 tx=0x7fe904010100).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:43:47.326 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:47.312+0000 7fe90dce5700 1 --2- 172.21.15.42:0/2866370667 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe90811ba70 0x7fe90811bea0 secure :-1 s=THROTTLE_DONE pgs=279 cs=0 l=1 rx=0x7fe904007e50 tx=0x7fe904010100).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:43:47.326 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:47.312+0000 7fe8fdffb700 1 -- 172.21.15.42:0/2866370667 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe904011470 con 0x7fe90811ba70 2019-12-05T05:43:47.326 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:47.312+0000 7fe90ef47700 1 -- 172.21.15.42:0/2866370667 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe908138cb0 con 0x7fe90811ba70 2019-12-05T05:43:47.326 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:47.316+0000 7fe8fdffb700 1 -- 172.21.15.42:0/2866370667 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe900004d80 con 0x7fe90811ba70 2019-12-05T05:43:47.326 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:47.316+0000 7fe90ef47700 1 -- 172.21.15.42:0/2866370667 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe908139230 con 0x7fe90811ba70 2019-12-05T05:43:47.326 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:47.316+0000 7fe90dce5700 1 --2- 172.21.15.42:0/2866370667 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe90811ba70 0x7fe90811bea0 secure :-1 s=THROTTLE_DONE pgs=279 cs=0 l=1 rx=0x7fe904007e50 tx=0x7fe904010100).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:43:47.326 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:47.316+0000 7fe8fdffb700 1 -- 172.21.15.42:0/2866370667 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fe900012490 con 0x7fe90811ba70 2019-12-05T05:43:47.327 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:47.316+0000 7fe90dce5700 1 --2- 172.21.15.42:0/2866370667 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe90811ba70 0x7fe90811bea0 secure :-1 s=THROTTLE_DONE pgs=279 cs=0 l=1 rx=0x7fe904007e50 tx=0x7fe904010100).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:43:47.327 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:47.316+0000 7fe8fdffb700 1 --2- 172.21.15.42:0/2866370667 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe8e803b400 0x7fe8e803d910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:43:47.327 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:47.316+0000 7fe8fdffb700 1 -- 172.21.15.42:0/2866370667 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 8267+0+0 (secure 0 0 0) 0x7fe90004aa60 con 0x7fe90811ba70 2019-12-05T05:43:47.327 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:47.316+0000 7fe90d4e4700 1 --2- 172.21.15.42:0/2866370667 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe8e803b400 0x7fe8e803d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:43:47.327 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:47.316+0000 7fe90d4e4700 1 --2- 172.21.15.42:0/2866370667 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe8e803b400 0x7fe8e803d910 crc :-1 s=READY pgs=574 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:43:47.328 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:47.320+0000 7fe90ef47700 1 -- 172.21.15.42:0/2866370667 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe90805d110 con 0x7fe90811ba70 2019-12-05T05:43:47.335 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:47.328+0000 7fe90dce5700 1 --2- 172.21.15.42:0/2866370667 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe90811ba70 0x7fe90811bea0 secure :-1 s=THROTTLE_DONE pgs=279 cs=0 l=1 rx=0x7fe904007e50 tx=0x7fe904010100).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:43:47.335 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:47.332+0000 7fe8fdffb700 1 -- 172.21.15.42:0/2866370667 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fe900031ae0 con 0x7fe90811ba70 2019-12-05T05:43:47.505 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:47.500+0000 7fe90ef47700 1 -- 172.21.15.42:0/2866370667 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe90805b8c0 con 0x7fe90811ba70 2019-12-05T05:43:47.506 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:47.500+0000 7fe90dce5700 1 --2- 172.21.15.42:0/2866370667 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe90811ba70 0x7fe90811bea0 secure :-1 s=THROTTLE_DONE pgs=279 cs=0 l=1 rx=0x7fe904007e50 tx=0x7fe904010100).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:43:47.506 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:47.500+0000 7fe8fdffb700 1 -- 172.21.15.42:0/2866370667 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 163 v163) v1 ==== 95+0+2109 (secure 0 0 0) 0x7fe900031d40 con 0x7fe90811ba70 2019-12-05T05:43:47.509 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:43:47.509 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":163,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7755,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/358497296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":358497296},{"type":"v1","addr":"172.21.15.42:6835","nonce":358497296}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":163}],"filesystems":[{"mdsmap":{"epoch":162,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:42:48.961753+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7650},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7650":{"gid":7650,"name":"b","rank":0,"incarnation":159,"state":"up:active","state_seq":29,"addr":"172.21.15.40:6833/1029013337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1029013337},{"type":"v1","addr":"172.21.15.40:6833","nonce":1029013337}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:43:47.509 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 163 2019-12-05T05:43:47.511 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:47.504+0000 7fe90ef47700 1 -- 172.21.15.42:0/2866370667 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe8e803b400 msgr2=0x7fe8e803d910 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:43:47.511 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:47.504+0000 7fe90ef47700 1 --2- 172.21.15.42:0/2866370667 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe8e803b400 0x7fe8e803d910 crc :-1 s=READY pgs=574 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:43:47.512 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:47.504+0000 7fe90ef47700 1 -- 172.21.15.42:0/2866370667 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe90811ba70 msgr2=0x7fe90811bea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:43:47.512 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:47.504+0000 7fe90ef47700 1 --2- 172.21.15.42:0/2866370667 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe90811ba70 0x7fe90811bea0 secure :-1 s=READY pgs=279 cs=0 l=1 rx=0x7fe904007e50 tx=0x7fe904010100).stop 2019-12-05T05:43:47.512 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:47.504+0000 7fe90ef47700 1 -- 172.21.15.42:0/2866370667 shutdown_connections 2019-12-05T05:43:47.512 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:47.504+0000 7fe90ef47700 1 --2- 172.21.15.42:0/2866370667 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe8e803b400 0x7fe8e803d910 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:43:47.512 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:47.504+0000 7fe90ef47700 1 --2- 172.21.15.42:0/2866370667 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe90811ba70 0x7fe90811bea0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:43:47.512 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:47.508+0000 7fe90ef47700 1 -- 172.21.15.42:0/2866370667 shutdown_connections 2019-12-05T05:43:47.512 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:47.508+0000 7fe90ef47700 1 -- 172.21.15.42:0/2866370667 wait complete. 2019-12-05T05:43:47.513 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:47.508+0000 7fe90ef47700 1 -- 172.21.15.42:0/2866370667 >> 172.21.15.42:0/2866370667 conn(0x7fe9080a6a00 msgr2=0x7fe90811c810 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:43:47.521 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T05:43:49.529 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:43:49.638 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:49.628+0000 7f302db64700 1 Processor -- start 2019-12-05T05:43:49.639 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:49.628+0000 7f302db64700 1 -- start start 2019-12-05T05:43:49.639 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:49.628+0000 7f302db64700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f30280a7740 0x7f302808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:43:49.639 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:49.628+0000 7f302db64700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f302807e720 0x7f302807eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:43:49.639 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:49.628+0000 7f302db64700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30280a0710 con 0x7f302809f2e0 2019-12-05T05:43:49.639 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:49.628+0000 7f302db64700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30280a0b00 con 0x7f30280a7740 2019-12-05T05:43:49.640 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:49.628+0000 7f302db64700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f302808e5b0 con 0x7f302807e720 2019-12-05T05:43:49.640 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:49.628+0000 7f3027fff700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f30280a7740 0x7f302808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:43:49.640 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:49.628+0000 7f30277fe700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f302807e720 0x7f302807eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:43:49.640 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:49.628+0000 7f30277fe700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f302807e720 0x7f302807eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:40196/0 (socket says 172.21.15.42:40196) 2019-12-05T05:43:49.640 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:49.628+0000 7f3027fff700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f30280a7740 0x7f302808df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.42:55778/0 (socket says 172.21.15.42:55778) 2019-12-05T05:43:49.641 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:49.628+0000 7f30277fe700 1 -- 172.21.15.42:0/2180130654 learned_addr learned my addr 172.21.15.42:0/2180130654 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:43:49.641 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:49.628+0000 7f302c902700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f302809f2e0 0x7f302809f710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:49212/0 (socket says 172.21.15.42:49212) 2019-12-05T05:43:49.642 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:49.632+0000 7f30277fe700 1 -- 172.21.15.42:0/2180130654 >> v2:172.21.15.40:3300/0 conn(0x7f30280a7740 msgr2=0x7f302808df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:43:49.642 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:49.632+0000 7f30277fe700 1 --2- 172.21.15.42:0/2180130654 >> v2:172.21.15.40:3300/0 conn(0x7f30280a7740 0x7f302808df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:43:49.642 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:49.632+0000 7f30277fe700 1 -- 172.21.15.42:0/2180130654 >> v1:172.21.15.40:6789/0 conn(0x7f302809f2e0 legacy=0x7f302809f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:43:49.642 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:49.632+0000 7f30277fe700 1 -- 172.21.15.42:0/2180130654 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f302807f0a0 con 0x7f302807e720 2019-12-05T05:43:49.642 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:49.632+0000 7f30277fe700 1 --2- 172.21.15.42:0/2180130654 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f302807e720 0x7f302807eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f3010008790 tx=0x7f301000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:43:49.643 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:49.632+0000 7f30277fe700 1 --2- 172.21.15.42:0/2180130654 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f302807e720 0x7f302807eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f3010008790 tx=0x7f301000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:43:49.643 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:49.632+0000 7f30277fe700 1 --2- 172.21.15.42:0/2180130654 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f302807e720 0x7f302807eb50 secure :-1 s=READY pgs=282 cs=0 l=1 rx=0x7f3010008790 tx=0x7f301000fd30).ready entity=mon.2 client_cookie=d660e8fb8f3f7796 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:43:49.643 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:49.632+0000 7f30277fe700 1 --2- 172.21.15.42:0/2180130654 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f302807e720 0x7f302807eb50 secure :-1 s=THROTTLE_DONE pgs=282 cs=0 l=1 rx=0x7f3010008790 tx=0x7f301000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:43:49.643 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:49.632+0000 7f3026ffd700 1 -- 172.21.15.42:0/2180130654 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f30100113b0 con 0x7f302807e720 2019-12-05T05:43:49.643 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:49.632+0000 7f30277fe700 1 --2- 172.21.15.42:0/2180130654 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f302807e720 0x7f302807eb50 secure :-1 s=THROTTLE_DONE pgs=282 cs=0 l=1 rx=0x7f3010008790 tx=0x7f301000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:43:49.643 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:49.632+0000 7f3026ffd700 1 -- 172.21.15.42:0/2180130654 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3010008bb0 con 0x7f302807e720 2019-12-05T05:43:49.644 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:49.632+0000 7f30277fe700 1 --2- 172.21.15.42:0/2180130654 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f302807e720 0x7f302807eb50 secure :-1 s=THROTTLE_DONE pgs=282 cs=0 l=1 rx=0x7f3010008790 tx=0x7f301000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:43:49.644 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:49.632+0000 7f3026ffd700 1 -- 172.21.15.42:0/2180130654 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3010011990 con 0x7f302807e720 2019-12-05T05:43:49.644 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:49.632+0000 7f302db64700 1 -- 172.21.15.42:0/2180130654 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f302807e720 msgr2=0x7f302807eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:43:49.644 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:49.632+0000 7f302db64700 1 --2- 172.21.15.42:0/2180130654 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f302807e720 0x7f302807eb50 secure :-1 s=READY pgs=282 cs=0 l=1 rx=0x7f3010008790 tx=0x7f301000fd30).stop 2019-12-05T05:43:49.644 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:49.632+0000 7f302db64700 1 -- 172.21.15.42:0/2180130654 shutdown_connections 2019-12-05T05:43:49.644 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:49.632+0000 7f302db64700 1 --2- 172.21.15.42:0/2180130654 >> v2:172.21.15.40:3300/0 conn(0x7f30280a7740 0x7f302808df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:43:49.645 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:49.632+0000 7f302db64700 1 --2- 172.21.15.42:0/2180130654 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f302807e720 0x7f302807eb50 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:43:49.645 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:49.632+0000 7f302db64700 1 -- 172.21.15.42:0/2180130654 shutdown_connections 2019-12-05T05:43:49.645 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:49.632+0000 7f302db64700 1 -- 172.21.15.42:0/2180130654 wait complete. 2019-12-05T05:43:49.645 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:49.632+0000 7f302db64700 1 -- 172.21.15.42:0/2180130654 >> 172.21.15.42:0/2180130654 conn(0x7f30280aabe0 msgr2=0x7f30280a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:43:49.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:49.636+0000 7f302db64700 1 Processor -- start 2019-12-05T05:43:49.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:49.636+0000 7f302db64700 1 -- start start 2019-12-05T05:43:49.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:49.636+0000 7f302db64700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f302811afc0 0x7f3028123950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:43:49.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:49.636+0000 7f302db64700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3028130880 0x7f3028130cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:43:49.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:49.636+0000 7f302db64700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3028123ea0 con 0x7f30281192c0 2019-12-05T05:43:49.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:49.636+0000 7f302db64700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f302811a660 con 0x7f3028130880 2019-12-05T05:43:49.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:49.636+0000 7f302db64700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f302811aa80 con 0x7f302811afc0 2019-12-05T05:43:49.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:49.636+0000 7f3027fff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f30281192c0 0x7f3028117910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:39776/0 (socket says 172.21.15.42:39776) 2019-12-05T05:43:49.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:49.636+0000 7f3027fff700 1 -- 172.21.15.42:0/1523224274 learned_addr learned my addr 172.21.15.42:0/1523224274 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:43:49.650 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:49.636+0000 7f302c902700 1 --2- 172.21.15.42:0/1523224274 >> v2:172.21.15.42:3300/0 conn(0x7f3028130880 0x7f3028130cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:43:49.650 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:49.636+0000 7f30277fe700 1 --2- 172.21.15.42:0/1523224274 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f302811afc0 0x7f3028123950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:43:49.650 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:49.636+0000 7f3024ff9700 1 -- 172.21.15.42:0/1523224274 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4000093378 0 0) 0x7f3028123ea0 con 0x7f30281192c0 2019-12-05T05:43:49.650 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:49.636+0000 7f3024ff9700 1 -- 172.21.15.42:0/1523224274 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f300c0030b0 con 0x7f30281192c0 2019-12-05T05:43:49.650 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:49.636+0000 7f30277fe700 1 -- 172.21.15.42:0/1523224274 >> v2:172.21.15.42:3300/0 conn(0x7f3028130880 msgr2=0x7f3028130cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:43:49.650 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:49.636+0000 7f30277fe700 1 --2- 172.21.15.42:0/1523224274 >> v2:172.21.15.42:3300/0 conn(0x7f3028130880 0x7f3028130cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:43:49.650 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:49.636+0000 7f30277fe700 1 -- 172.21.15.42:0/1523224274 >> v1:172.21.15.42:6789/0 conn(0x7f30281192c0 legacy=0x7f3028117910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:43:49.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:49.636+0000 7f30277fe700 1 -- 172.21.15.42:0/1523224274 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3028131200 con 0x7f302811afc0 2019-12-05T05:43:49.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:49.636+0000 7f30277fe700 1 --2- 172.21.15.42:0/1523224274 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f302811afc0 0x7f3028123950 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f3010008790 tx=0x7f3010008bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:43:49.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:49.636+0000 7f30277fe700 1 --2- 172.21.15.42:0/1523224274 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f302811afc0 0x7f3028123950 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f3010008790 tx=0x7f3010008bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:43:49.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:49.636+0000 7f30277fe700 1 --2- 172.21.15.42:0/1523224274 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f302811afc0 0x7f3028123950 secure :-1 s=READY pgs=283 cs=0 l=1 rx=0x7f3010008790 tx=0x7f3010008bb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:43:49.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:49.636+0000 7f30277fe700 1 --2- 172.21.15.42:0/1523224274 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f302811afc0 0x7f3028123950 secure :-1 s=THROTTLE_DONE pgs=283 cs=0 l=1 rx=0x7f3010008790 tx=0x7f3010008bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:43:49.654 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:49.636+0000 7f3024ff9700 1 -- 172.21.15.42:0/1523224274 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3010007340 con 0x7f302811afc0 2019-12-05T05:43:49.654 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:49.636+0000 7f30277fe700 1 --2- 172.21.15.42:0/1523224274 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f302811afc0 0x7f3028123950 secure :-1 s=THROTTLE_DONE pgs=283 cs=0 l=1 rx=0x7f3010008790 tx=0x7f3010008bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:43:49.654 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:49.636+0000 7f30277fe700 1 --2- 172.21.15.42:0/1523224274 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f302811afc0 0x7f3028123950 secure :-1 s=THROTTLE_DONE pgs=283 cs=0 l=1 rx=0x7f3010008790 tx=0x7f3010008bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:43:49.654 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:49.636+0000 7f3024ff9700 1 -- 172.21.15.42:0/1523224274 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30100115b0 con 0x7f302811afc0 2019-12-05T05:43:49.655 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:49.636+0000 7f302db64700 1 -- 172.21.15.42:0/1523224274 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3028131470 con 0x7f302811afc0 2019-12-05T05:43:49.655 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:49.636+0000 7f3024ff9700 1 -- 172.21.15.42:0/1523224274 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3010005540 con 0x7f302811afc0 2019-12-05T05:43:49.655 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:49.636+0000 7f302db64700 1 -- 172.21.15.42:0/1523224274 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3028137b90 con 0x7f302811afc0 2019-12-05T05:43:49.655 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:49.640+0000 7f30277fe700 1 --2- 172.21.15.42:0/1523224274 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f302811afc0 0x7f3028123950 secure :-1 s=THROTTLE_DONE pgs=283 cs=0 l=1 rx=0x7f3010008790 tx=0x7f3010008bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:43:49.655 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:49.640+0000 7f3024ff9700 1 -- 172.21.15.42:0/1523224274 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f3010005850 con 0x7f302811afc0 2019-12-05T05:43:49.655 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:49.640+0000 7f30277fe700 1 --2- 172.21.15.42:0/1523224274 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f302811afc0 0x7f3028123950 secure :-1 s=THROTTLE_DONE pgs=283 cs=0 l=1 rx=0x7f3010008790 tx=0x7f3010008bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:43:49.655 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:49.640+0000 7f3024ff9700 1 --2- 172.21.15.42:0/1523224274 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f300c03b2c0 0x7f300c03d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:43:49.656 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:49.640+0000 7f3024ff9700 1 -- 172.21.15.42:0/1523224274 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 8267+0+0 (secure 0 0 0) 0x7f301004b370 con 0x7f302811afc0 2019-12-05T05:43:49.656 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:49.640+0000 7f3027fff700 1 --2- 172.21.15.42:0/1523224274 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f300c03b2c0 0x7f300c03d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:43:49.656 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:49.640+0000 7f3027fff700 1 --2- 172.21.15.42:0/1523224274 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f300c03b2c0 0x7f300c03d7d0 crc :-1 s=READY pgs=575 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:43:49.657 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:49.644+0000 7f302db64700 1 -- 172.21.15.42:0/1523224274 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f302805d110 con 0x7f302811afc0 2019-12-05T05:43:49.659 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:49.652+0000 7f30277fe700 1 --2- 172.21.15.42:0/1523224274 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f302811afc0 0x7f3028123950 secure :-1 s=THROTTLE_DONE pgs=283 cs=0 l=1 rx=0x7f3010008790 tx=0x7f3010008bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:43:49.659 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:49.652+0000 7f3024ff9700 1 -- 172.21.15.42:0/1523224274 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f30100323f0 con 0x7f302811afc0 2019-12-05T05:43:49.833 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:49.828+0000 7f302db64700 1 -- 172.21.15.42:0/1523224274 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f302805b8c0 con 0x7f302811afc0 2019-12-05T05:43:49.833 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:49.828+0000 7f30277fe700 1 --2- 172.21.15.42:0/1523224274 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f302811afc0 0x7f3028123950 secure :-1 s=THROTTLE_DONE pgs=283 cs=0 l=1 rx=0x7f3010008790 tx=0x7f3010008bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:43:49.834 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:49.828+0000 7f3024ff9700 1 -- 172.21.15.42:0/1523224274 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 163 v163) v1 ==== 95+0+2109 (secure 0 0 0) 0x7f3010032650 con 0x7f302811afc0 2019-12-05T05:43:49.836 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:43:49.836 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":163,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7755,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/358497296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":358497296},{"type":"v1","addr":"172.21.15.42:6835","nonce":358497296}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":163}],"filesystems":[{"mdsmap":{"epoch":162,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:42:48.961753+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7650},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7650":{"gid":7650,"name":"b","rank":0,"incarnation":159,"state":"up:active","state_seq":29,"addr":"172.21.15.40:6833/1029013337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1029013337},{"type":"v1","addr":"172.21.15.40:6833","nonce":1029013337}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:43:49.836 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 163 2019-12-05T05:43:49.840 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:49.832+0000 7f302db64700 1 -- 172.21.15.42:0/1523224274 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f300c03b2c0 msgr2=0x7f300c03d7d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:43:49.840 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:49.832+0000 7f302db64700 1 --2- 172.21.15.42:0/1523224274 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f300c03b2c0 0x7f300c03d7d0 crc :-1 s=READY pgs=575 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:43:49.841 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:49.832+0000 7f302db64700 1 -- 172.21.15.42:0/1523224274 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f302811afc0 msgr2=0x7f3028123950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:43:49.841 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:49.832+0000 7f302db64700 1 --2- 172.21.15.42:0/1523224274 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f302811afc0 0x7f3028123950 secure :-1 s=READY pgs=283 cs=0 l=1 rx=0x7f3010008790 tx=0x7f3010008bb0).stop 2019-12-05T05:43:49.841 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:49.832+0000 7f302db64700 1 -- 172.21.15.42:0/1523224274 shutdown_connections 2019-12-05T05:43:49.841 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:49.832+0000 7f302db64700 1 --2- 172.21.15.42:0/1523224274 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f300c03b2c0 0x7f300c03d7d0 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:43:49.841 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:49.832+0000 7f302db64700 1 --2- 172.21.15.42:0/1523224274 >> v2:172.21.15.42:3300/0 conn(0x7f3028130880 0x7f3028130cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:43:49.842 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:49.832+0000 7f302db64700 1 --2- 172.21.15.42:0/1523224274 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f302811afc0 0x7f3028123950 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:43:49.842 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:49.832+0000 7f302db64700 1 -- 172.21.15.42:0/1523224274 shutdown_connections 2019-12-05T05:43:49.842 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:49.832+0000 7f302db64700 1 -- 172.21.15.42:0/1523224274 wait complete. 2019-12-05T05:43:49.842 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:49.832+0000 7f302db64700 1 -- 172.21.15.42:0/1523224274 >> 172.21.15.42:0/1523224274 conn(0x7f30280a9640 msgr2=0x7f302809f620 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:43:49.848 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T05:43:51.849 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:43:51.953 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:51.944+0000 7f8508253700 1 Processor -- start 2019-12-05T05:43:51.953 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:51.944+0000 7f8508253700 1 -- start start 2019-12-05T05:43:51.953 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:51.944+0000 7f8508253700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f850008df80 0x7f850009f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:43:51.954 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:51.944+0000 7f8508253700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f850009f830 0x7f850014efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:43:51.954 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:51.944+0000 7f8508253700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f850009fc60 con 0x7f85000a7740 2019-12-05T05:43:51.954 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:51.944+0000 7f8508253700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85000a0b00 con 0x7f850009f830 2019-12-05T05:43:51.954 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:51.944+0000 7f8508253700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f850008e5b0 con 0x7f850008df80 2019-12-05T05:43:51.954 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:51.944+0000 7f8505fef700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f850008df80 0x7f850009f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:43:51.955 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:51.944+0000 7f8505fef700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f850008df80 0x7f850009f2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:39748/0 (socket says 172.21.15.42:39748) 2019-12-05T05:43:51.955 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:51.944+0000 7f8505fef700 1 -- 172.21.15.42:0/251049779 learned_addr learned my addr 172.21.15.42:0/251049779 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:43:51.955 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:51.944+0000 7f8506ff1700 1 --2- 172.21.15.42:0/251049779 >> v2:172.21.15.40:3300/0 conn(0x7f850009f830 0x7f850014efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:43:51.955 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:51.944+0000 7f8505fef700 1 -- 172.21.15.42:0/251049779 >> v2:172.21.15.40:3300/0 conn(0x7f850009f830 msgr2=0x7f850014efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:43:51.955 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:51.944+0000 7f8505fef700 1 --2- 172.21.15.42:0/251049779 >> v2:172.21.15.40:3300/0 conn(0x7f850009f830 0x7f850014efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:43:51.955 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:51.944+0000 7f8505fef700 1 -- 172.21.15.42:0/251049779 >> v1:172.21.15.42:6789/0 conn(0x7f85000a7740 legacy=0x7f850007e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:43:51.955 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:51.944+0000 7f8505fef700 1 -- 172.21.15.42:0/251049779 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85000a0710 con 0x7f850008df80 2019-12-05T05:43:51.956 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:51.944+0000 7f8505fef700 1 --2- 172.21.15.42:0/251049779 >> v2:172.21.15.42:3300/0 conn(0x7f850008df80 0x7f850009f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f84f80032b0 tx=0x7f84f80100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:43:51.956 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:51.944+0000 7f8505fef700 1 --2- 172.21.15.42:0/251049779 >> v2:172.21.15.42:3300/0 conn(0x7f850008df80 0x7f850009f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f84f80032b0 tx=0x7f84f80100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:43:51.956 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:51.944+0000 7f8505fef700 1 --2- 172.21.15.42:0/251049779 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f850008df80 0x7f850009f2e0 secure :-1 s=READY pgs=1336 cs=0 l=1 rx=0x7f84f80032b0 tx=0x7f84f80100b0).ready entity=mon.1 client_cookie=da0a71dc5f4c2f62 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:43:51.956 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:51.944+0000 7f8505fef700 1 --2- 172.21.15.42:0/251049779 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f850008df80 0x7f850009f2e0 secure :-1 s=THROTTLE_DONE pgs=1336 cs=0 l=1 rx=0x7f84f80032b0 tx=0x7f84f80100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:43:51.956 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:51.944+0000 7f85057ee700 1 -- 172.21.15.42:0/251049779 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f84f8010d60 con 0x7f850008df80 2019-12-05T05:43:51.956 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:51.948+0000 7f8505fef700 1 --2- 172.21.15.42:0/251049779 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f850008df80 0x7f850009f2e0 secure :-1 s=THROTTLE_DONE pgs=1336 cs=0 l=1 rx=0x7f84f80032b0 tx=0x7f84f80100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:43:51.956 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:51.948+0000 7f85057ee700 1 -- 172.21.15.42:0/251049779 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84f8010980 con 0x7f850008df80 2019-12-05T05:43:51.957 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:51.948+0000 7f8505fef700 1 --2- 172.21.15.42:0/251049779 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f850008df80 0x7f850009f2e0 secure :-1 s=THROTTLE_DONE pgs=1336 cs=0 l=1 rx=0x7f84f80032b0 tx=0x7f84f80100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:43:51.957 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:51.948+0000 7f85057ee700 1 -- 172.21.15.42:0/251049779 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f84f80114a0 con 0x7f850008df80 2019-12-05T05:43:51.957 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:51.948+0000 7f8508253700 1 -- 172.21.15.42:0/251049779 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f850008df80 msgr2=0x7f850009f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:43:51.957 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:51.948+0000 7f8508253700 1 --2- 172.21.15.42:0/251049779 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f850008df80 0x7f850009f2e0 secure :-1 s=READY pgs=1336 cs=0 l=1 rx=0x7f84f80032b0 tx=0x7f84f80100b0).stop 2019-12-05T05:43:51.958 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:51.948+0000 7f8508253700 1 -- 172.21.15.42:0/251049779 shutdown_connections 2019-12-05T05:43:51.958 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:51.948+0000 7f8508253700 1 --2- 172.21.15.42:0/251049779 >> v2:172.21.15.40:3300/0 conn(0x7f850009f830 0x7f850014efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:43:51.958 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:51.948+0000 7f8508253700 1 --2- 172.21.15.42:0/251049779 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f850008df80 0x7f850009f2e0 unknown :-1 s=CLOSED pgs=1336 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:43:51.958 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:51.948+0000 7f8508253700 1 -- 172.21.15.42:0/251049779 shutdown_connections 2019-12-05T05:43:51.958 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:51.948+0000 7f8508253700 1 -- 172.21.15.42:0/251049779 wait complete. 2019-12-05T05:43:51.958 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:51.948+0000 7f8508253700 1 -- 172.21.15.42:0/251049779 >> 172.21.15.42:0/251049779 conn(0x7f85000aabe0 msgr2=0x7f85000a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:43:51.963 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:51.948+0000 7f8508253700 1 Processor -- start 2019-12-05T05:43:51.963 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:51.948+0000 7f8508253700 1 -- start start 2019-12-05T05:43:51.964 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:51.952+0000 7f8508253700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f85001369e0 0x7f8500138e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:43:51.964 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:51.952+0000 7f8508253700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8500144810 con 0x7f850011bc20 2019-12-05T05:43:51.965 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:51.952+0000 7f8508253700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85000a1c50 con 0x7f8500143be0 2019-12-05T05:43:51.965 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:51.952+0000 7f8508253700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8500143980 con 0x7f85001369e0 2019-12-05T05:43:51.966 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:51.952+0000 7f85067f0700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f850011bc20 0x7f8500142530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:49230/0 (socket says 172.21.15.42:49230) 2019-12-05T05:43:51.966 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:51.952+0000 7f85067f0700 1 -- 172.21.15.42:0/1368110819 learned_addr learned my addr 172.21.15.42:0/1368110819 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:43:51.966 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:51.952+0000 7f8506ff1700 1 --2- 172.21.15.42:0/1368110819 >> v2:172.21.15.40:3300/0 conn(0x7f85001369e0 0x7f8500138e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:43:51.966 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:51.952+0000 7f84f6ffd700 1 -- 172.21.15.42:0/1368110819 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2765075850 0 0) 0x7f850009fc60 con 0x7f850011bc20 2019-12-05T05:43:51.967 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:51.952+0000 7f84f6ffd700 1 -- 172.21.15.42:0/1368110819 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84e00030b0 con 0x7f850011bc20 2019-12-05T05:43:51.967 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:51.952+0000 7f84f6ffd700 1 -- 172.21.15.42:0/1368110819 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1228088972 0 0) 0x7f85000a1c50 con 0x7f8500143be0 2019-12-05T05:43:51.967 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:51.952+0000 7f84f6ffd700 1 -- 172.21.15.42:0/1368110819 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f850009fc60 con 0x7f8500143be0 2019-12-05T05:43:51.967 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:51.952+0000 7f8506ff1700 1 -- 172.21.15.42:0/1368110819 >> v1:172.21.15.42:6789/0 conn(0x7f8500143be0 legacy=0x7f8500144010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:43:51.968 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:51.952+0000 7f8506ff1700 1 -- 172.21.15.42:0/1368110819 >> v1:172.21.15.40:6789/0 conn(0x7f850011bc20 legacy=0x7f8500142530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:43:51.968 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:51.952+0000 7f8506ff1700 1 -- 172.21.15.42:0/1368110819 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f850007d110 con 0x7f85001369e0 2019-12-05T05:43:51.968 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:51.952+0000 7f8506ff1700 1 --2- 172.21.15.42:0/1368110819 >> v2:172.21.15.40:3300/0 conn(0x7f85001369e0 0x7f8500138e20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f84f0002f30 tx=0x7f84f0010940).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:43:51.969 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:51.952+0000 7f8506ff1700 1 --2- 172.21.15.42:0/1368110819 >> v2:172.21.15.40:3300/0 conn(0x7f85001369e0 0x7f8500138e20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f84f0002f30 tx=0x7f84f0010940).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:43:51.969 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:51.952+0000 7f8506ff1700 1 --2- 172.21.15.42:0/1368110819 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f85001369e0 0x7f8500138e20 secure :-1 s=READY pgs=1043 cs=0 l=1 rx=0x7f84f0002f30 tx=0x7f84f0010940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:43:51.969 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:51.952+0000 7f8506ff1700 1 --2- 172.21.15.42:0/1368110819 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f85001369e0 0x7f8500138e20 secure :-1 s=THROTTLE_DONE pgs=1043 cs=0 l=1 rx=0x7f84f0002f30 tx=0x7f84f0010940).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:43:51.969 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:51.952+0000 7f84f6ffd700 1 -- 172.21.15.42:0/1368110819 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f84f0002730 con 0x7f85001369e0 2019-12-05T05:43:51.970 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:51.952+0000 7f8506ff1700 1 --2- 172.21.15.42:0/1368110819 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f85001369e0 0x7f8500138e20 secure :-1 s=THROTTLE_DONE pgs=1043 cs=0 l=1 rx=0x7f84f0002f30 tx=0x7f84f0010940).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:43:51.970 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:51.952+0000 7f8506ff1700 1 --2- 172.21.15.42:0/1368110819 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f85001369e0 0x7f8500138e20 secure :-1 s=THROTTLE_DONE pgs=1043 cs=0 l=1 rx=0x7f84f0002f30 tx=0x7f84f0010940).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:43:51.970 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:51.952+0000 7f84f6ffd700 1 -- 172.21.15.42:0/1368110819 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84f0002350 con 0x7f85001369e0 2019-12-05T05:43:51.970 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:51.952+0000 7f8508253700 1 -- 172.21.15.42:0/1368110819 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f850011c050 con 0x7f85001369e0 2019-12-05T05:43:51.971 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:51.952+0000 7f84f6ffd700 1 -- 172.21.15.42:0/1368110819 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f84f000d160 con 0x7f85001369e0 2019-12-05T05:43:51.971 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:51.952+0000 7f8508253700 1 -- 172.21.15.42:0/1368110819 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8500139370 con 0x7f85001369e0 2019-12-05T05:43:51.971 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:51.956+0000 7f8506ff1700 1 --2- 172.21.15.42:0/1368110819 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f85001369e0 0x7f8500138e20 secure :-1 s=THROTTLE_DONE pgs=1043 cs=0 l=1 rx=0x7f84f0002f30 tx=0x7f84f0010940).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:43:51.971 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:51.956+0000 7f84f6ffd700 1 -- 172.21.15.42:0/1368110819 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f84f0007170 con 0x7f85001369e0 2019-12-05T05:43:51.971 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:51.956+0000 7f8506ff1700 1 --2- 172.21.15.42:0/1368110819 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f85001369e0 0x7f8500138e20 secure :-1 s=THROTTLE_DONE pgs=1043 cs=0 l=1 rx=0x7f84f0002f30 tx=0x7f84f0010940).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:43:51.972 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:51.956+0000 7f84f6ffd700 1 --2- 172.21.15.42:0/1368110819 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f84e003b4a0 0x7f84e003d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:43:51.972 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:51.956+0000 7f84f6ffd700 1 -- 172.21.15.42:0/1368110819 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 8267+0+0 (secure 0 0 0) 0x7f84f004aa70 con 0x7f85001369e0 2019-12-05T05:43:51.973 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:51.956+0000 7f85067f0700 1 --2- 172.21.15.42:0/1368110819 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f84e003b4a0 0x7f84e003d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:43:51.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:51.956+0000 7f85067f0700 1 --2- 172.21.15.42:0/1368110819 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f84e003b4a0 0x7f84e003d9b0 crc :-1 s=READY pgs=576 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:43:51.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:51.960+0000 7f8508253700 1 -- 172.21.15.42:0/1368110819 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f850005d110 con 0x7f85001369e0 2019-12-05T05:43:51.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:51.968+0000 7f8506ff1700 1 --2- 172.21.15.42:0/1368110819 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f85001369e0 0x7f8500138e20 secure :-1 s=THROTTLE_DONE pgs=1043 cs=0 l=1 rx=0x7f84f0002f30 tx=0x7f84f0010940).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:43:51.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:51.968+0000 7f84f6ffd700 1 -- 172.21.15.42:0/1368110819 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f84f0031af0 con 0x7f85001369e0 2019-12-05T05:43:52.144 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:52.140+0000 7f8508253700 1 -- 172.21.15.42:0/1368110819 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f850005b8c0 con 0x7f85001369e0 2019-12-05T05:43:52.156 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:52.140+0000 7f8506ff1700 1 --2- 172.21.15.42:0/1368110819 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f85001369e0 0x7f8500138e20 secure :-1 s=THROTTLE_DONE pgs=1043 cs=0 l=1 rx=0x7f84f0002f30 tx=0x7f84f0010940).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:43:52.156 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:52.140+0000 7f84f6ffd700 1 -- 172.21.15.42:0/1368110819 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 163 v163) v1 ==== 95+0+2109 (secure 0 0 0) 0x7f84f0031d50 con 0x7f85001369e0 2019-12-05T05:43:52.157 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 163 2019-12-05T05:43:52.157 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:52.144+0000 7f8508253700 1 -- 172.21.15.42:0/1368110819 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f84e003b4a0 msgr2=0x7f84e003d9b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:43:52.157 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:52.144+0000 7f8508253700 1 --2- 172.21.15.42:0/1368110819 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f84e003b4a0 0x7f84e003d9b0 crc :-1 s=READY pgs=576 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:43:52.157 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:52.144+0000 7f8508253700 1 -- 172.21.15.42:0/1368110819 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f85001369e0 msgr2=0x7f8500138e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:43:52.158 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:52.144+0000 7f8508253700 1 --2- 172.21.15.42:0/1368110819 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f85001369e0 0x7f8500138e20 secure :-1 s=READY pgs=1043 cs=0 l=1 rx=0x7f84f0002f30 tx=0x7f84f0010940).stop 2019-12-05T05:43:52.158 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:52.144+0000 7f8508253700 1 -- 172.21.15.42:0/1368110819 shutdown_connections 2019-12-05T05:43:52.158 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:52.144+0000 7f8508253700 1 --2- 172.21.15.42:0/1368110819 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f84e003b4a0 0x7f84e003d9b0 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:43:52.158 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:52.144+0000 7f8508253700 1 --2- 172.21.15.42:0/1368110819 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f85001369e0 0x7f8500138e20 unknown :-1 s=CLOSED pgs=1043 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:43:52.159 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:52.144+0000 7f8508253700 1 -- 172.21.15.42:0/1368110819 shutdown_connections 2019-12-05T05:43:52.159 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:52.144+0000 7f8508253700 1 -- 172.21.15.42:0/1368110819 wait complete. 2019-12-05T05:43:52.159 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:52.144+0000 7f8508253700 1 -- 172.21.15.42:0/1368110819 >> 172.21.15.42:0/1368110819 conn(0x7f850007dc80 msgr2=0x7f8500119300 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:43:52.159 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:43:52.160 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":163,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7755,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/358497296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":358497296},{"type":"v1","addr":"172.21.15.42:6835","nonce":358497296}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":163}],"filesystems":[{"mdsmap":{"epoch":162,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:42:48.961753+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7650},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7650":{"gid":7650,"name":"b","rank":0,"incarnation":159,"state":"up:active","state_seq":29,"addr":"172.21.15.40:6833/1029013337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1029013337},{"type":"v1","addr":"172.21.15.40:6833","nonce":1029013337}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:43:52.162 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T05:43:54.163 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:43:54.269 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:54.256+0000 7f4beaec0700 1 Processor -- start 2019-12-05T05:43:54.269 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:54.256+0000 7f4beaec0700 1 -- start start 2019-12-05T05:43:54.270 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:54.260+0000 7f4beaec0700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4be40a7740 0x7f4be408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:43:54.270 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:54.260+0000 7f4beaec0700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f4be407e720 0x7f4be407eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:43:54.270 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:54.260+0000 7f4beaec0700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4be40a0710 con 0x7f4be409f2e0 2019-12-05T05:43:54.270 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:54.260+0000 7f4beaec0700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4be40a0b00 con 0x7f4be407e720 2019-12-05T05:43:54.270 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:54.260+0000 7f4beaec0700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4be408e5b0 con 0x7f4be40a7740 2019-12-05T05:43:54.271 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:54.260+0000 7f4be9c5e700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f4be409f2e0 0x7f4be409f710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:39802/0 (socket says 172.21.15.42:39802) 2019-12-05T05:43:54.271 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:54.260+0000 7f4be9c5e700 1 -- 172.21.15.42:0/991508255 learned_addr learned my addr 172.21.15.42:0/991508255 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:43:54.271 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:54.260+0000 7f4be945d700 1 --2- 172.21.15.42:0/991508255 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4be40a7740 0x7f4be408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:43:54.271 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:54.260+0000 7f4be8c5c700 1 --2- 172.21.15.42:0/991508255 >> v2:172.21.15.40:3300/0 conn(0x7f4be407e720 0x7f4be407eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:43:54.271 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:54.260+0000 7f4be3fff700 1 -- 172.21.15.42:0/991508255 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2198021796 0 0) 0x7f4be40a0710 con 0x7f4be409f2e0 2019-12-05T05:43:54.271 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:54.260+0000 7f4be3fff700 1 -- 172.21.15.42:0/991508255 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4bd40030b0 con 0x7f4be409f2e0 2019-12-05T05:43:54.271 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:54.260+0000 7f4be945d700 1 -- 172.21.15.42:0/991508255 >> v2:172.21.15.40:3300/0 conn(0x7f4be407e720 msgr2=0x7f4be407eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:43:54.272 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:54.260+0000 7f4be945d700 1 --2- 172.21.15.42:0/991508255 >> v2:172.21.15.40:3300/0 conn(0x7f4be407e720 0x7f4be407eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:43:54.272 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:54.260+0000 7f4be945d700 1 -- 172.21.15.42:0/991508255 >> v1:172.21.15.42:6789/0 conn(0x7f4be409f2e0 legacy=0x7f4be409f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:43:54.272 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:54.260+0000 7f4be945d700 1 -- 172.21.15.42:0/991508255 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4be407f0a0 con 0x7f4be40a7740 2019-12-05T05:43:54.272 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:54.260+0000 7f4be945d700 1 --2- 172.21.15.42:0/991508255 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4be40a7740 0x7f4be408df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f4bd8008b10 tx=0x7f4bd800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:43:54.272 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:54.260+0000 7f4be945d700 1 --2- 172.21.15.42:0/991508255 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4be40a7740 0x7f4be408df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f4bd8008b10 tx=0x7f4bd800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:43:54.272 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:54.260+0000 7f4be945d700 1 --2- 172.21.15.42:0/991508255 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4be40a7740 0x7f4be408df80 secure :-1 s=READY pgs=284 cs=0 l=1 rx=0x7f4bd8008b10 tx=0x7f4bd800fff0).ready entity=mon.2 client_cookie=ebc8319f194678f3 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:43:54.273 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:54.260+0000 7f4be945d700 1 --2- 172.21.15.42:0/991508255 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4be40a7740 0x7f4be408df80 secure :-1 s=THROTTLE_DONE pgs=284 cs=0 l=1 rx=0x7f4bd8008b10 tx=0x7f4bd800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:43:54.273 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:54.260+0000 7f4be3fff700 1 -- 172.21.15.42:0/991508255 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4bd8011a50 con 0x7f4be40a7740 2019-12-05T05:43:54.273 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:54.260+0000 7f4be945d700 1 --2- 172.21.15.42:0/991508255 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4be40a7740 0x7f4be408df80 secure :-1 s=THROTTLE_DONE pgs=284 cs=0 l=1 rx=0x7f4bd8008b10 tx=0x7f4bd800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:43:54.273 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:54.260+0000 7f4be945d700 1 --2- 172.21.15.42:0/991508255 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4be40a7740 0x7f4be408df80 secure :-1 s=THROTTLE_DONE pgs=284 cs=0 l=1 rx=0x7f4bd8008b10 tx=0x7f4bd800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:43:54.273 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:54.260+0000 7f4be3fff700 1 -- 172.21.15.42:0/991508255 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bd8011670 con 0x7f4be40a7740 2019-12-05T05:43:54.273 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:54.260+0000 7f4be3fff700 1 -- 172.21.15.42:0/991508255 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4bd800a350 con 0x7f4be40a7740 2019-12-05T05:43:54.274 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:54.260+0000 7f4beaec0700 1 -- 172.21.15.42:0/991508255 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4be40a7740 msgr2=0x7f4be408df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:43:54.274 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:54.260+0000 7f4beaec0700 1 --2- 172.21.15.42:0/991508255 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4be40a7740 0x7f4be408df80 secure :-1 s=READY pgs=284 cs=0 l=1 rx=0x7f4bd8008b10 tx=0x7f4bd800fff0).stop 2019-12-05T05:43:54.274 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:54.264+0000 7f4beaec0700 1 -- 172.21.15.42:0/991508255 shutdown_connections 2019-12-05T05:43:54.274 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:54.264+0000 7f4beaec0700 1 --2- 172.21.15.42:0/991508255 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4be40a7740 0x7f4be408df80 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:43:54.274 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:54.264+0000 7f4beaec0700 1 --2- 172.21.15.42:0/991508255 >> v2:172.21.15.40:3300/0 conn(0x7f4be407e720 0x7f4be407eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:43:54.274 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:54.264+0000 7f4beaec0700 1 -- 172.21.15.42:0/991508255 shutdown_connections 2019-12-05T05:43:54.274 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:54.264+0000 7f4beaec0700 1 -- 172.21.15.42:0/991508255 wait complete. 2019-12-05T05:43:54.275 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:54.264+0000 7f4beaec0700 1 -- 172.21.15.42:0/991508255 >> 172.21.15.42:0/991508255 conn(0x7f4be40aabe0 msgr2=0x7f4be40a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:43:54.279 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:54.264+0000 7f4beaec0700 1 Processor -- start 2019-12-05T05:43:54.279 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:54.264+0000 7f4beaec0700 1 -- start start 2019-12-05T05:43:54.279 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:54.268+0000 7f4beaec0700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f4be4124610 0x7f4be4122930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:43:54.279 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:54.268+0000 7f4beaec0700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f4be4130420 0x7f4be4130850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:43:54.279 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:54.268+0000 7f4beaec0700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4be4123580 con 0x7f4be4121790 2019-12-05T05:43:54.280 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:54.268+0000 7f4beaec0700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4be407e260 con 0x7f4be4130420 2019-12-05T05:43:54.280 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:54.268+0000 7f4beaec0700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4be4123040 con 0x7f4be4124610 2019-12-05T05:43:54.280 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:54.268+0000 7f4be945d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f4be4124610 0x7f4be4122930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:43:54.280 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:54.268+0000 7f4be945d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f4be4124610 0x7f4be4122930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:39766/0 (socket says 172.21.15.42:39766) 2019-12-05T05:43:54.283 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:54.268+0000 7f4be945d700 1 -- 172.21.15.42:0/454180341 learned_addr learned my addr 172.21.15.42:0/454180341 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:43:54.283 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:54.268+0000 7f4be9c5e700 1 --2- 172.21.15.42:0/454180341 >> v2:172.21.15.40:3300/0 conn(0x7f4be4130420 0x7f4be4130850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:43:54.283 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:54.268+0000 7f4be945d700 1 -- 172.21.15.42:0/454180341 >> v2:172.21.15.40:3300/0 conn(0x7f4be4130420 msgr2=0x7f4be4130850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:43:54.283 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:54.268+0000 7f4be945d700 1 --2- 172.21.15.42:0/454180341 >> v2:172.21.15.40:3300/0 conn(0x7f4be4130420 0x7f4be4130850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:43:54.283 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:54.268+0000 7f4be945d700 1 -- 172.21.15.42:0/454180341 >> v1:172.21.15.42:6789/0 conn(0x7f4be4121790 legacy=0x7f4be407da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:43:54.284 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:54.268+0000 7f4be945d700 1 -- 172.21.15.42:0/454180341 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4be4130da0 con 0x7f4be4124610 2019-12-05T05:43:54.284 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:54.268+0000 7f4be945d700 1 --2- 172.21.15.42:0/454180341 >> v2:172.21.15.42:3300/0 conn(0x7f4be4124610 0x7f4be4122930 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f4bd8008b10 tx=0x7f4bd8011ec0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:43:54.284 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:54.268+0000 7f4be945d700 1 --2- 172.21.15.42:0/454180341 >> v2:172.21.15.42:3300/0 conn(0x7f4be4124610 0x7f4be4122930 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f4bd8008b10 tx=0x7f4bd8011ec0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:43:54.284 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:54.268+0000 7f4be945d700 1 --2- 172.21.15.42:0/454180341 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4be4124610 0x7f4be4122930 secure :-1 s=READY pgs=1340 cs=0 l=1 rx=0x7f4bd8008b10 tx=0x7f4bd8011ec0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:43:54.284 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:54.268+0000 7f4be945d700 1 --2- 172.21.15.42:0/454180341 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4be4124610 0x7f4be4122930 secure :-1 s=THROTTLE_DONE pgs=1340 cs=0 l=1 rx=0x7f4bd8008b10 tx=0x7f4bd8011ec0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:43:54.284 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:54.268+0000 7f4be1ffb700 1 -- 172.21.15.42:0/454180341 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4bd80058c0 con 0x7f4be4124610 2019-12-05T05:43:54.284 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:54.268+0000 7f4be945d700 1 --2- 172.21.15.42:0/454180341 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4be4124610 0x7f4be4122930 secure :-1 s=THROTTLE_DONE pgs=1340 cs=0 l=1 rx=0x7f4bd8008b10 tx=0x7f4bd8011ec0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:43:54.285 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:54.268+0000 7f4be1ffb700 1 -- 172.21.15.42:0/454180341 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bd8011c50 con 0x7f4be4124610 2019-12-05T05:43:54.285 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:54.268+0000 7f4be945d700 1 --2- 172.21.15.42:0/454180341 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4be4124610 0x7f4be4122930 secure :-1 s=THROTTLE_DONE pgs=1340 cs=0 l=1 rx=0x7f4bd8008b10 tx=0x7f4bd8011ec0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:43:54.285 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:54.268+0000 7f4beaec0700 1 -- 172.21.15.42:0/454180341 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4be4137760 con 0x7f4be4124610 2019-12-05T05:43:54.285 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:54.268+0000 7f4be1ffb700 1 -- 172.21.15.42:0/454180341 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4bd800f360 con 0x7f4be4124610 2019-12-05T05:43:54.285 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:54.268+0000 7f4beaec0700 1 -- 172.21.15.42:0/454180341 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4be4137c90 con 0x7f4be4124610 2019-12-05T05:43:54.285 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:54.268+0000 7f4be945d700 1 --2- 172.21.15.42:0/454180341 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4be4124610 0x7f4be4122930 secure :-1 s=THROTTLE_DONE pgs=1340 cs=0 l=1 rx=0x7f4bd8008b10 tx=0x7f4bd8011ec0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:43:54.286 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:54.272+0000 7f4be1ffb700 1 -- 172.21.15.42:0/454180341 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f4bd800f640 con 0x7f4be4124610 2019-12-05T05:43:54.286 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:54.272+0000 7f4be945d700 1 --2- 172.21.15.42:0/454180341 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4be4124610 0x7f4be4122930 secure :-1 s=THROTTLE_DONE pgs=1340 cs=0 l=1 rx=0x7f4bd8008b10 tx=0x7f4bd8011ec0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:43:54.286 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:54.272+0000 7f4be1ffb700 1 --2- 172.21.15.42:0/454180341 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4bc403ad60 0x7f4bc403d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:43:54.286 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:54.272+0000 7f4be1ffb700 1 -- 172.21.15.42:0/454180341 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 8267+0+0 (secure 0 0 0) 0x7f4bd808f750 con 0x7f4be4124610 2019-12-05T05:43:54.286 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:54.272+0000 7f4be8c5c700 1 --2- 172.21.15.42:0/454180341 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4bc403ad60 0x7f4bc403d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:43:54.286 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:54.272+0000 7f4be8c5c700 1 --2- 172.21.15.42:0/454180341 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4bc403ad60 0x7f4bc403d270 crc :-1 s=READY pgs=577 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:43:54.286 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:54.272+0000 7f4beaec0700 1 -- 172.21.15.42:0/454180341 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4be405d110 con 0x7f4be4124610 2019-12-05T05:43:54.288 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:54.280+0000 7f4be945d700 1 --2- 172.21.15.42:0/454180341 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4be4124610 0x7f4be4122930 secure :-1 s=THROTTLE_DONE pgs=1340 cs=0 l=1 rx=0x7f4bd8008b10 tx=0x7f4bd8011ec0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:43:54.288 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:54.280+0000 7f4be1ffb700 1 -- 172.21.15.42:0/454180341 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f4bd800f950 con 0x7f4be4124610 2019-12-05T05:43:54.458 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:54.452+0000 7f4beaec0700 1 -- 172.21.15.42:0/454180341 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4be405b8c0 con 0x7f4be4124610 2019-12-05T05:43:54.459 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:54.452+0000 7f4be945d700 1 --2- 172.21.15.42:0/454180341 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4be4124610 0x7f4be4122930 secure :-1 s=THROTTLE_DONE pgs=1340 cs=0 l=1 rx=0x7f4bd8008b10 tx=0x7f4bd8011ec0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:43:54.459 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:54.452+0000 7f4be1ffb700 1 -- 172.21.15.42:0/454180341 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 163 v163) v1 ==== 95+0+2109 (secure 0 0 0) 0x7f4bd800ec60 con 0x7f4be4124610 2019-12-05T05:43:54.459 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 163 2019-12-05T05:43:54.460 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:43:54.460 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":163,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7755,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/358497296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":358497296},{"type":"v1","addr":"172.21.15.42:6835","nonce":358497296}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":163}],"filesystems":[{"mdsmap":{"epoch":162,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:42:48.961753+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7650},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7650":{"gid":7650,"name":"b","rank":0,"incarnation":159,"state":"up:active","state_seq":29,"addr":"172.21.15.40:6833/1029013337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1029013337},{"type":"v1","addr":"172.21.15.40:6833","nonce":1029013337}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:43:54.461 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:54.456+0000 7f4beaec0700 1 -- 172.21.15.42:0/454180341 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4bc403ad60 msgr2=0x7f4bc403d270 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:43:54.461 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:54.456+0000 7f4beaec0700 1 --2- 172.21.15.42:0/454180341 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4bc403ad60 0x7f4bc403d270 crc :-1 s=READY pgs=577 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:43:54.461 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:54.456+0000 7f4beaec0700 1 -- 172.21.15.42:0/454180341 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4be4124610 msgr2=0x7f4be4122930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:43:54.462 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:54.456+0000 7f4beaec0700 1 --2- 172.21.15.42:0/454180341 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4be4124610 0x7f4be4122930 secure :-1 s=READY pgs=1340 cs=0 l=1 rx=0x7f4bd8008b10 tx=0x7f4bd8011ec0).stop 2019-12-05T05:43:54.462 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:54.456+0000 7f4beaec0700 1 -- 172.21.15.42:0/454180341 shutdown_connections 2019-12-05T05:43:54.462 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:54.456+0000 7f4beaec0700 1 --2- 172.21.15.42:0/454180341 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4bc403ad60 0x7f4bc403d270 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:43:54.462 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:54.456+0000 7f4beaec0700 1 --2- 172.21.15.42:0/454180341 >> v2:172.21.15.40:3300/0 conn(0x7f4be4130420 0x7f4be4130850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:43:54.462 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:54.456+0000 7f4beaec0700 1 --2- 172.21.15.42:0/454180341 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4be4124610 0x7f4be4122930 unknown :-1 s=CLOSED pgs=1340 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:43:54.462 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:54.456+0000 7f4beaec0700 1 -- 172.21.15.42:0/454180341 shutdown_connections 2019-12-05T05:43:54.462 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:54.456+0000 7f4beaec0700 1 -- 172.21.15.42:0/454180341 wait complete. 2019-12-05T05:43:54.463 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:54.456+0000 7f4beaec0700 1 -- 172.21.15.42:0/454180341 >> 172.21.15.42:0/454180341 conn(0x7f4be40a82a0 msgr2=0x7f4be40a9790 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:43:54.471 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T05:43:56.481 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:43:56.608 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:56.580+0000 7f1fd3819700 1 Processor -- start 2019-12-05T05:43:56.615 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:56.580+0000 7f1fd3819700 1 -- start start 2019-12-05T05:43:56.615 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:56.580+0000 7f1fd3819700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f1fcc0a7740 0x7f1fcc08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:43:56.616 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:56.580+0000 7f1fd3819700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fcc0a0710 con 0x7f1fcc09fae0 2019-12-05T05:43:56.616 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:56.580+0000 7f1fd3819700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fcc0a9ba0 con 0x7f1fcc07e720 2019-12-05T05:43:56.616 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:56.580+0000 7f1fd3819700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1fcc0a0b00 con 0x7f1fcc0a7740 2019-12-05T05:43:56.616 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:56.580+0000 7f1fd15b5700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f1fcc07e720 0x7f1fcc09f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:39812/0 (socket says 172.21.15.42:39812) 2019-12-05T05:43:56.616 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:56.580+0000 7f1fd15b5700 1 -- 172.21.15.42:0/3607927150 learned_addr learned my addr 172.21.15.42:0/3607927150 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:43:56.616 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:56.580+0000 7f1fd1db6700 1 --2- 172.21.15.42:0/3607927150 >> v2:172.21.15.40:3300/0 conn(0x7f1fcc0a7740 0x7f1fcc08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:43:56.616 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:56.580+0000 7f1fd0db4700 1 -- 172.21.15.42:0/3607927150 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3505176170 0 0) 0x7f1fcc0a9ba0 con 0x7f1fcc07e720 2019-12-05T05:43:56.617 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:56.580+0000 7f1fd0db4700 1 -- 172.21.15.42:0/3607927150 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1fb80030b0 con 0x7f1fcc07e720 2019-12-05T05:43:56.617 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:56.584+0000 7f1fd0db4700 1 -- 172.21.15.42:0/3607927150 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2202933405 0 0) 0x7f1fcc0a0710 con 0x7f1fcc09fae0 2019-12-05T05:43:56.617 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:56.584+0000 7f1fd0db4700 1 -- 172.21.15.42:0/3607927150 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1fcc0a9ba0 con 0x7f1fcc09fae0 2019-12-05T05:43:56.617 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:56.584+0000 7f1fd1db6700 1 -- 172.21.15.42:0/3607927150 >> v1:172.21.15.42:6789/0 conn(0x7f1fcc07e720 legacy=0x7f1fcc09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:43:56.617 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:56.584+0000 7f1fd1db6700 1 -- 172.21.15.42:0/3607927150 >> v1:172.21.15.40:6789/0 conn(0x7f1fcc09fae0 legacy=0x7f1fcc14efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:43:56.617 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:56.584+0000 7f1fd1db6700 1 -- 172.21.15.42:0/3607927150 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fcc07eb50 con 0x7f1fcc0a7740 2019-12-05T05:43:56.617 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:56.584+0000 7f1fd1db6700 1 --2- 172.21.15.42:0/3607927150 >> v2:172.21.15.40:3300/0 conn(0x7f1fcc0a7740 0x7f1fcc08df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f1fc0008790 tx=0x7f1fc000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:43:56.618 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:56.584+0000 7f1fd1db6700 1 --2- 172.21.15.42:0/3607927150 >> v2:172.21.15.40:3300/0 conn(0x7f1fcc0a7740 0x7f1fcc08df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f1fc0008790 tx=0x7f1fc000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:43:56.626 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:56.584+0000 7f1fd1db6700 1 --2- 172.21.15.42:0/3607927150 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1fcc0a7740 0x7f1fcc08df80 secure :-1 s=READY pgs=1047 cs=0 l=1 rx=0x7f1fc0008790 tx=0x7f1fc000fbf0).ready entity=mon.0 client_cookie=657ffad4d45aef48 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:43:56.626 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:56.584+0000 7f1fd1db6700 1 --2- 172.21.15.42:0/3607927150 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1fcc0a7740 0x7f1fcc08df80 secure :-1 s=THROTTLE_DONE pgs=1047 cs=0 l=1 rx=0x7f1fc0008790 tx=0x7f1fc000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:43:56.626 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:56.584+0000 7f1fd0db4700 1 -- 172.21.15.42:0/3607927150 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1fc0011270 con 0x7f1fcc0a7740 2019-12-05T05:43:56.627 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:56.584+0000 7f1fd1db6700 1 --2- 172.21.15.42:0/3607927150 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1fcc0a7740 0x7f1fcc08df80 secure :-1 s=THROTTLE_DONE pgs=1047 cs=0 l=1 rx=0x7f1fc0008790 tx=0x7f1fc000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:43:56.627 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:56.584+0000 7f1fd1db6700 1 --2- 172.21.15.42:0/3607927150 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1fcc0a7740 0x7f1fcc08df80 secure :-1 s=THROTTLE_DONE pgs=1047 cs=0 l=1 rx=0x7f1fc0008790 tx=0x7f1fc000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:43:56.627 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:56.584+0000 7f1fd0db4700 1 -- 172.21.15.42:0/3607927150 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fc0008bc0 con 0x7f1fcc0a7740 2019-12-05T05:43:56.627 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:56.584+0000 7f1fd0db4700 1 -- 172.21.15.42:0/3607927150 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1fc0011850 con 0x7f1fcc0a7740 2019-12-05T05:43:56.627 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:56.584+0000 7f1fd3819700 1 -- 172.21.15.42:0/3607927150 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1fcc0a7740 msgr2=0x7f1fcc08df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:43:56.627 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:56.584+0000 7f1fd3819700 1 --2- 172.21.15.42:0/3607927150 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1fcc0a7740 0x7f1fcc08df80 secure :-1 s=READY pgs=1047 cs=0 l=1 rx=0x7f1fc0008790 tx=0x7f1fc000fbf0).stop 2019-12-05T05:43:56.627 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:56.584+0000 7f1fd3819700 1 -- 172.21.15.42:0/3607927150 shutdown_connections 2019-12-05T05:43:56.628 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:56.584+0000 7f1fd3819700 1 --2- 172.21.15.42:0/3607927150 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1fcc0a7740 0x7f1fcc08df80 unknown :-1 s=CLOSED pgs=1047 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:43:56.628 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:56.584+0000 7f1fd3819700 1 -- 172.21.15.42:0/3607927150 shutdown_connections 2019-12-05T05:43:56.628 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:56.584+0000 7f1fd3819700 1 -- 172.21.15.42:0/3607927150 wait complete. 2019-12-05T05:43:56.628 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:56.584+0000 7f1fd3819700 1 -- 172.21.15.42:0/3607927150 >> 172.21.15.42:0/3607927150 conn(0x7f1fcc0aabe0 msgr2=0x7f1fcc0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:43:56.647 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:56.588+0000 7f1fd3819700 1 Processor -- start 2019-12-05T05:43:56.647 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:56.588+0000 7f1fd3819700 1 -- start start 2019-12-05T05:43:56.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:56.588+0000 7f1fd3819700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1fcc117ae0 0x7f1fcc117f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:43:56.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:56.588+0000 7f1fd3819700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f1fcc077f20 0x7f1fcc078350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:43:56.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:56.588+0000 7f1fd3819700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fcc0a30d0 con 0x7f1fcc144b50 2019-12-05T05:43:56.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:56.588+0000 7f1fd3819700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1fcc0a28f0 con 0x7f1fcc077f20 2019-12-05T05:43:56.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:56.588+0000 7f1fd3819700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fcc0a2b90 con 0x7f1fcc117ae0 2019-12-05T05:43:56.650 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:56.588+0000 7f1fd15b5700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1fcc117ae0 0x7f1fcc117f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:43:56.650 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:56.588+0000 7f1fd25b7700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f1fcc077f20 0x7f1fcc078350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:43:56.650 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:56.588+0000 7f1fd1db6700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f1fcc144b50 0x7f1fcc118b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:49258/0 (socket says 172.21.15.42:49258) 2019-12-05T05:43:56.650 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:56.588+0000 7f1fd15b5700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1fcc117ae0 0x7f1fcc117f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:40248/0 (socket says 172.21.15.42:40248) 2019-12-05T05:43:56.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:56.588+0000 7f1fd1db6700 1 -- 172.21.15.42:0/1260820722 learned_addr learned my addr 172.21.15.42:0/1260820722 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:43:56.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:56.588+0000 7f1fbe7fc700 1 -- 172.21.15.42:0/1260820722 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 27025003 0 0) 0x7f1fcc0a30d0 con 0x7f1fcc144b50 2019-12-05T05:43:56.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:56.588+0000 7f1fbe7fc700 1 -- 172.21.15.42:0/1260820722 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1fb40030b0 con 0x7f1fcc144b50 2019-12-05T05:43:56.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:56.588+0000 7f1fd15b5700 1 -- 172.21.15.42:0/1260820722 >> v2:172.21.15.42:3300/0 conn(0x7f1fcc077f20 msgr2=0x7f1fcc078350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:43:56.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:56.588+0000 7f1fd15b5700 1 --2- 172.21.15.42:0/1260820722 >> v2:172.21.15.42:3300/0 conn(0x7f1fcc077f20 0x7f1fcc078350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:43:56.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:56.588+0000 7f1fd15b5700 1 -- 172.21.15.42:0/1260820722 >> v1:172.21.15.40:6789/0 conn(0x7f1fcc144b50 legacy=0x7f1fcc118b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:43:56.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:56.588+0000 7f1fd15b5700 1 -- 172.21.15.42:0/1260820722 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fcc0788a0 con 0x7f1fcc117ae0 2019-12-05T05:43:56.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:56.588+0000 7f1fd25b7700 1 --2- 172.21.15.42:0/1260820722 >> v2:172.21.15.42:3300/0 conn(0x7f1fcc077f20 0x7f1fcc078350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:43:56.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:56.588+0000 7f1fd15b5700 1 --2- 172.21.15.42:0/1260820722 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1fcc117ae0 0x7f1fcc117f10 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f1fc80043a0 tx=0x7f1fc80108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:43:56.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:56.588+0000 7f1fd15b5700 1 --2- 172.21.15.42:0/1260820722 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1fcc117ae0 0x7f1fcc117f10 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f1fc80043a0 tx=0x7f1fc80108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:43:56.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:56.588+0000 7f1fd15b5700 1 --2- 172.21.15.42:0/1260820722 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1fcc117ae0 0x7f1fcc117f10 secure :-1 s=READY pgs=285 cs=0 l=1 rx=0x7f1fc80043a0 tx=0x7f1fc80108f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:43:56.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:56.588+0000 7f1fd15b5700 1 --2- 172.21.15.42:0/1260820722 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1fcc117ae0 0x7f1fcc117f10 secure :-1 s=THROTTLE_DONE pgs=285 cs=0 l=1 rx=0x7f1fc80043a0 tx=0x7f1fc80108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:43:56.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:56.588+0000 7f1fbe7fc700 1 -- 172.21.15.42:0/1260820722 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1fc8005b40 con 0x7f1fcc117ae0 2019-12-05T05:43:56.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:56.592+0000 7f1fd15b5700 1 --2- 172.21.15.42:0/1260820722 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1fcc117ae0 0x7f1fcc117f10 secure :-1 s=THROTTLE_DONE pgs=285 cs=0 l=1 rx=0x7f1fc80043a0 tx=0x7f1fc80108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:43:56.654 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:56.592+0000 7f1fd15b5700 1 --2- 172.21.15.42:0/1260820722 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1fcc117ae0 0x7f1fcc117f10 secure :-1 s=THROTTLE_DONE pgs=285 cs=0 l=1 rx=0x7f1fc80043a0 tx=0x7f1fc80108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:43:56.654 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:56.592+0000 7f1fbe7fc700 1 -- 172.21.15.42:0/1260820722 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fc8005760 con 0x7f1fcc117ae0 2019-12-05T05:43:56.654 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:56.592+0000 7f1fbe7fc700 1 -- 172.21.15.42:0/1260820722 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1fc80062b0 con 0x7f1fcc117ae0 2019-12-05T05:43:56.654 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:56.592+0000 7f1fd3819700 1 -- 172.21.15.42:0/1260820722 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fcc0a1a20 con 0x7f1fcc117ae0 2019-12-05T05:43:56.655 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:56.592+0000 7f1fd3819700 1 -- 172.21.15.42:0/1260820722 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fcc12dbf0 con 0x7f1fcc117ae0 2019-12-05T05:43:56.655 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:56.592+0000 7f1fd15b5700 1 --2- 172.21.15.42:0/1260820722 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1fcc117ae0 0x7f1fcc117f10 secure :-1 s=THROTTLE_DONE pgs=285 cs=0 l=1 rx=0x7f1fc80043a0 tx=0x7f1fc80108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:43:56.655 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:56.592+0000 7f1fbe7fc700 1 -- 172.21.15.42:0/1260820722 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f1fc80065c0 con 0x7f1fcc117ae0 2019-12-05T05:43:56.655 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:56.592+0000 7f1fd15b5700 1 --2- 172.21.15.42:0/1260820722 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1fcc117ae0 0x7f1fcc117f10 secure :-1 s=THROTTLE_DONE pgs=285 cs=0 l=1 rx=0x7f1fc80043a0 tx=0x7f1fc80108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:43:56.655 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:56.592+0000 7f1fbe7fc700 1 --2- 172.21.15.42:0/1260820722 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f1fb403b280 0x7f1fb403d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:43:56.656 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:56.592+0000 7f1fbe7fc700 1 -- 172.21.15.42:0/1260820722 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 8267+0+0 (secure 0 0 0) 0x7f1fc804aeb0 con 0x7f1fcc117ae0 2019-12-05T05:43:56.656 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:56.592+0000 7f1fd1db6700 1 --2- 172.21.15.42:0/1260820722 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f1fb403b280 0x7f1fb403d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:43:56.656 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:56.592+0000 7f1fd1db6700 1 --2- 172.21.15.42:0/1260820722 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f1fb403b280 0x7f1fb403d790 crc :-1 s=READY pgs=578 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:43:56.656 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:56.596+0000 7f1fd3819700 1 -- 172.21.15.42:0/1260820722 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fcc05d110 con 0x7f1fcc117ae0 2019-12-05T05:43:56.657 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:56.604+0000 7f1fd15b5700 1 --2- 172.21.15.42:0/1260820722 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1fcc117ae0 0x7f1fcc117f10 secure :-1 s=THROTTLE_DONE pgs=285 cs=0 l=1 rx=0x7f1fc80043a0 tx=0x7f1fc80108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:43:56.658 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:56.604+0000 7f1fbe7fc700 1 -- 172.21.15.42:0/1260820722 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f1fc8031f30 con 0x7f1fcc117ae0 2019-12-05T05:43:56.788 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:56.772+0000 7f1fd3819700 1 -- 172.21.15.42:0/1260820722 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1fcc0a26d0 con 0x7f1fcc117ae0 2019-12-05T05:43:56.788 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:56.772+0000 7f1fd15b5700 1 --2- 172.21.15.42:0/1260820722 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1fcc117ae0 0x7f1fcc117f10 secure :-1 s=THROTTLE_DONE pgs=285 cs=0 l=1 rx=0x7f1fc80043a0 tx=0x7f1fc80108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:43:56.788 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:56.772+0000 7f1fbe7fc700 1 -- 172.21.15.42:0/1260820722 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 163 v163) v1 ==== 95+0+2109 (secure 0 0 0) 0x7f1fc8032190 con 0x7f1fcc117ae0 2019-12-05T05:43:56.788 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 163 2019-12-05T05:43:56.789 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:56.776+0000 7f1fd3819700 1 -- 172.21.15.42:0/1260820722 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f1fb403b280 msgr2=0x7f1fb403d790 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:43:56.789 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:56.776+0000 7f1fd3819700 1 --2- 172.21.15.42:0/1260820722 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f1fb403b280 0x7f1fb403d790 crc :-1 s=READY pgs=578 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:43:56.789 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:56.776+0000 7f1fd3819700 1 -- 172.21.15.42:0/1260820722 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1fcc117ae0 msgr2=0x7f1fcc117f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:43:56.789 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:56.776+0000 7f1fd3819700 1 --2- 172.21.15.42:0/1260820722 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1fcc117ae0 0x7f1fcc117f10 secure :-1 s=READY pgs=285 cs=0 l=1 rx=0x7f1fc80043a0 tx=0x7f1fc80108f0).stop 2019-12-05T05:43:56.789 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:56.776+0000 7f1fd3819700 1 -- 172.21.15.42:0/1260820722 shutdown_connections 2019-12-05T05:43:56.789 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:56.776+0000 7f1fd3819700 1 --2- 172.21.15.42:0/1260820722 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f1fb403b280 0x7f1fb403d790 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:43:56.789 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:56.776+0000 7f1fd3819700 1 --2- 172.21.15.42:0/1260820722 >> v2:172.21.15.42:3300/0 conn(0x7f1fcc077f20 0x7f1fcc078350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:43:56.790 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:56.776+0000 7f1fd3819700 1 --2- 172.21.15.42:0/1260820722 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1fcc117ae0 0x7f1fcc117f10 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:43:56.790 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:56.776+0000 7f1fd3819700 1 -- 172.21.15.42:0/1260820722 shutdown_connections 2019-12-05T05:43:56.790 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:56.776+0000 7f1fd3819700 1 -- 172.21.15.42:0/1260820722 wait complete. 2019-12-05T05:43:56.790 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:56.776+0000 7f1fd3819700 1 -- 172.21.15.42:0/1260820722 >> 172.21.15.42:0/1260820722 conn(0x7f1fcc0aabe0 msgr2=0x7f1fcc09f5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:43:56.790 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:43:56.790 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":163,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7755,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/358497296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":358497296},{"type":"v1","addr":"172.21.15.42:6835","nonce":358497296}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":163}],"filesystems":[{"mdsmap":{"epoch":162,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:42:48.961753+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7650},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7650":{"gid":7650,"name":"b","rank":0,"incarnation":159,"state":"up:active","state_seq":29,"addr":"172.21.15.40:6833/1029013337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1029013337},{"type":"v1","addr":"172.21.15.40:6833","nonce":1029013337}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:43:56.792 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T05:43:58.793 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:43:58.897 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:58.884+0000 7fef09fda700 1 Processor -- start 2019-12-05T05:43:58.897 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:58.888+0000 7fef09fda700 1 -- start start 2019-12-05T05:43:58.897 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:58.888+0000 7fef09fda700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fef0408df80 0x7fef0409f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:43:58.897 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:58.888+0000 7fef09fda700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fef0409f830 0x7fef0414efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:43:58.897 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:58.888+0000 7fef09fda700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef0409fc60 con 0x7fef040a7740 2019-12-05T05:43:58.898 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:58.888+0000 7fef09fda700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef040a0b00 con 0x7fef0409f830 2019-12-05T05:43:58.898 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:58.888+0000 7fef09fda700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef0408e5b0 con 0x7fef0408df80 2019-12-05T05:43:58.898 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:58.888+0000 7fef037fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fef0408df80 0x7fef0409f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:43:58.898 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:58.888+0000 7fef037fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fef0408df80 0x7fef0409f2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:39796/0 (socket says 172.21.15.42:39796) 2019-12-05T05:43:58.898 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:58.888+0000 7fef037fe700 1 -- 172.21.15.42:0/672839584 learned_addr learned my addr 172.21.15.42:0/672839584 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:43:58.898 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:58.888+0000 7fef08d78700 1 --2- 172.21.15.42:0/672839584 >> v2:172.21.15.40:3300/0 conn(0x7fef0409f830 0x7fef0414efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:43:58.899 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:58.888+0000 7fef037fe700 1 -- 172.21.15.42:0/672839584 >> v2:172.21.15.40:3300/0 conn(0x7fef0409f830 msgr2=0x7fef0414efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:43:58.899 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:58.888+0000 7fef037fe700 1 --2- 172.21.15.42:0/672839584 >> v2:172.21.15.40:3300/0 conn(0x7fef0409f830 0x7fef0414efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:43:58.899 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:58.888+0000 7fef037fe700 1 -- 172.21.15.42:0/672839584 >> v1:172.21.15.42:6789/0 conn(0x7fef040a7740 legacy=0x7fef0407e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:43:58.899 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:58.888+0000 7fef037fe700 1 -- 172.21.15.42:0/672839584 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef040a0710 con 0x7fef0408df80 2019-12-05T05:43:58.899 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:58.888+0000 7fef037fe700 1 --2- 172.21.15.42:0/672839584 >> v2:172.21.15.42:3300/0 conn(0x7fef0408df80 0x7fef0409f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7feef8002f30 tx=0x7feef800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:43:58.899 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:58.888+0000 7fef037fe700 1 --2- 172.21.15.42:0/672839584 >> v2:172.21.15.42:3300/0 conn(0x7fef0408df80 0x7fef0409f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7feef8002f30 tx=0x7feef800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:43:58.899 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:58.888+0000 7fef037fe700 1 --2- 172.21.15.42:0/672839584 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fef0408df80 0x7fef0409f2e0 secure :-1 s=READY pgs=1343 cs=0 l=1 rx=0x7feef8002f30 tx=0x7feef800fd30).ready entity=mon.1 client_cookie=55797856ddb2ac6c server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:43:58.900 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:58.888+0000 7fef037fe700 1 --2- 172.21.15.42:0/672839584 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fef0408df80 0x7fef0409f2e0 secure :-1 s=THROTTLE_DONE pgs=1343 cs=0 l=1 rx=0x7feef8002f30 tx=0x7feef800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:43:58.900 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:58.888+0000 7fef02ffd700 1 -- 172.21.15.42:0/672839584 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7feef8008c90 con 0x7fef0408df80 2019-12-05T05:43:58.900 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:58.888+0000 7fef037fe700 1 --2- 172.21.15.42:0/672839584 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fef0408df80 0x7fef0409f2e0 secure :-1 s=THROTTLE_DONE pgs=1343 cs=0 l=1 rx=0x7feef8002f30 tx=0x7feef800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:43:58.900 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:58.888+0000 7fef037fe700 1 --2- 172.21.15.42:0/672839584 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fef0408df80 0x7fef0409f2e0 secure :-1 s=THROTTLE_DONE pgs=1343 cs=0 l=1 rx=0x7feef8002f30 tx=0x7feef800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:43:58.901 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:58.888+0000 7fef02ffd700 1 -- 172.21.15.42:0/672839584 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feef80088b0 con 0x7fef0408df80 2019-12-05T05:43:58.901 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:58.888+0000 7fef02ffd700 1 -- 172.21.15.42:0/672839584 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7feef8011ad0 con 0x7fef0408df80 2019-12-05T05:43:58.901 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:58.888+0000 7fef09fda700 1 -- 172.21.15.42:0/672839584 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fef0408df80 msgr2=0x7fef0409f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:43:58.901 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:58.888+0000 7fef09fda700 1 --2- 172.21.15.42:0/672839584 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fef0408df80 0x7fef0409f2e0 secure :-1 s=READY pgs=1343 cs=0 l=1 rx=0x7feef8002f30 tx=0x7feef800fd30).stop 2019-12-05T05:43:58.901 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:58.892+0000 7fef09fda700 1 -- 172.21.15.42:0/672839584 shutdown_connections 2019-12-05T05:43:58.902 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:58.892+0000 7fef09fda700 1 --2- 172.21.15.42:0/672839584 >> v2:172.21.15.40:3300/0 conn(0x7fef0409f830 0x7fef0414efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:43:58.902 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:58.892+0000 7fef09fda700 1 --2- 172.21.15.42:0/672839584 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fef0408df80 0x7fef0409f2e0 unknown :-1 s=CLOSED pgs=1343 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:43:58.902 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:58.892+0000 7fef09fda700 1 -- 172.21.15.42:0/672839584 shutdown_connections 2019-12-05T05:43:58.902 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:58.892+0000 7fef09fda700 1 -- 172.21.15.42:0/672839584 wait complete. 2019-12-05T05:43:58.902 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:58.892+0000 7fef09fda700 1 -- 172.21.15.42:0/672839584 >> 172.21.15.42:0/672839584 conn(0x7fef040aabe0 msgr2=0x7fef040a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:43:58.906 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:58.892+0000 7fef09fda700 1 Processor -- start 2019-12-05T05:43:58.906 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:58.892+0000 7fef09fda700 1 -- start start 2019-12-05T05:43:58.906 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:58.896+0000 7fef09fda700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fef041231d0 0x7fef041386a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:43:58.906 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:58.896+0000 7fef09fda700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef0407b7f0 con 0x7fef04142530 2019-12-05T05:43:58.906 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:58.896+0000 7fef09fda700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef04145310 con 0x7fef0407b1c0 2019-12-05T05:43:58.906 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:58.896+0000 7fef09fda700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef0407bb30 con 0x7fef041231d0 2019-12-05T05:43:58.906 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:58.896+0000 7fef03fff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fef0407b1c0 0x7fef0407d070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:39840/0 (socket says 172.21.15.42:39840) 2019-12-05T05:43:58.907 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:58.896+0000 7fef03fff700 1 -- 172.21.15.42:0/306406152 learned_addr learned my addr 172.21.15.42:0/306406152 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:43:58.907 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:58.896+0000 7fef08d78700 1 --2- 172.21.15.42:0/306406152 >> v2:172.21.15.40:3300/0 conn(0x7fef041231d0 0x7fef041386a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:43:58.907 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:58.896+0000 7fef00ff9700 1 -- 172.21.15.42:0/306406152 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 953606651 0 0) 0x7fef04145310 con 0x7fef0407b1c0 2019-12-05T05:43:58.907 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:58.896+0000 7fef00ff9700 1 -- 172.21.15.42:0/306406152 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feee40030b0 con 0x7fef0407b1c0 2019-12-05T05:43:58.907 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:58.896+0000 7fef00ff9700 1 -- 172.21.15.42:0/306406152 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 3084528443 0 0) 0x7feee40030b0 con 0x7fef0407b1c0 2019-12-05T05:43:58.907 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:58.896+0000 7fef08d78700 1 -- 172.21.15.42:0/306406152 >> v1:172.21.15.42:6789/0 conn(0x7fef0407b1c0 legacy=0x7fef0407d070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:43:58.908 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:58.896+0000 7fef08d78700 1 -- 172.21.15.42:0/306406152 >> v1:172.21.15.40:6789/0 conn(0x7fef04142530 legacy=0x7fef041229d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:43:58.908 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:58.896+0000 7fef08d78700 1 -- 172.21.15.42:0/306406152 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef040a4b20 con 0x7fef041231d0 2019-12-05T05:43:58.908 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:58.896+0000 7fef08d78700 1 --2- 172.21.15.42:0/306406152 >> v2:172.21.15.40:3300/0 conn(0x7fef041231d0 0x7fef041386a0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7feef4001fe0 tx=0x7feef4002020).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:43:58.908 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:58.896+0000 7fef08d78700 1 --2- 172.21.15.42:0/306406152 >> v2:172.21.15.40:3300/0 conn(0x7fef041231d0 0x7fef041386a0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7feef4001fe0 tx=0x7feef4002020).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:43:58.908 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:58.896+0000 7fef08d78700 1 --2- 172.21.15.42:0/306406152 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fef041231d0 0x7fef041386a0 secure :-1 s=READY pgs=1050 cs=0 l=1 rx=0x7feef4001fe0 tx=0x7feef4002020).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:43:58.909 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:58.896+0000 7fef08d78700 1 --2- 172.21.15.42:0/306406152 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fef041231d0 0x7fef041386a0 secure :-1 s=THROTTLE_DONE pgs=1050 cs=0 l=1 rx=0x7feef4001fe0 tx=0x7feef4002020).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:43:58.909 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:58.896+0000 7fef00ff9700 1 -- 172.21.15.42:0/306406152 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7feef4001de0 con 0x7fef041231d0 2019-12-05T05:43:58.909 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:58.896+0000 7fef08d78700 1 --2- 172.21.15.42:0/306406152 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fef041231d0 0x7fef041386a0 secure :-1 s=THROTTLE_DONE pgs=1050 cs=0 l=1 rx=0x7feef4001fe0 tx=0x7feef4002020).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:43:58.909 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:58.896+0000 7fef09fda700 1 -- 172.21.15.42:0/306406152 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef040a4df0 con 0x7fef041231d0 2019-12-05T05:43:58.909 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:58.896+0000 7fef00ff9700 1 -- 172.21.15.42:0/306406152 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feef4001bf0 con 0x7fef041231d0 2019-12-05T05:43:58.909 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:58.896+0000 7fef08d78700 1 --2- 172.21.15.42:0/306406152 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fef041231d0 0x7fef041386a0 secure :-1 s=THROTTLE_DONE pgs=1050 cs=0 l=1 rx=0x7feef4001fe0 tx=0x7feef4002020).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:43:58.909 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:58.896+0000 7fef00ff9700 1 -- 172.21.15.42:0/306406152 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7feefc005b20 con 0x7fef041231d0 2019-12-05T05:43:58.910 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:58.896+0000 7fef09fda700 1 -- 172.21.15.42:0/306406152 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef040a52c0 con 0x7fef041231d0 2019-12-05T05:43:58.910 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:58.896+0000 7fef08d78700 1 --2- 172.21.15.42:0/306406152 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fef041231d0 0x7fef041386a0 secure :-1 s=THROTTLE_DONE pgs=1050 cs=0 l=1 rx=0x7feef4001fe0 tx=0x7feef4002020).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:43:58.910 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:58.900+0000 7fef00ff9700 1 -- 172.21.15.42:0/306406152 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7feefc00a250 con 0x7fef041231d0 2019-12-05T05:43:58.910 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:58.900+0000 7fef08d78700 1 --2- 172.21.15.42:0/306406152 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fef041231d0 0x7fef041386a0 secure :-1 s=THROTTLE_DONE pgs=1050 cs=0 l=1 rx=0x7feef4001fe0 tx=0x7feef4002020).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:43:58.910 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:58.900+0000 7fef00ff9700 1 --2- 172.21.15.42:0/306406152 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7feee403b1c0 0x7feee403d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:43:58.910 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:58.900+0000 7fef00ff9700 1 -- 172.21.15.42:0/306406152 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 8267+0+0 (secure 0 0 0) 0x7feefc04a8c0 con 0x7fef041231d0 2019-12-05T05:43:58.910 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:58.900+0000 7fef03fff700 1 --2- 172.21.15.42:0/306406152 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7feee403b1c0 0x7feee403d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:43:58.911 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:58.900+0000 7fef03fff700 1 --2- 172.21.15.42:0/306406152 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7feee403b1c0 0x7feee403d6d0 crc :-1 s=READY pgs=579 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:43:58.912 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:58.900+0000 7fef09fda700 1 -- 172.21.15.42:0/306406152 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef0405d110 con 0x7fef041231d0 2019-12-05T05:43:58.916 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:58.912+0000 7fef08d78700 1 --2- 172.21.15.42:0/306406152 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fef041231d0 0x7fef041386a0 secure :-1 s=THROTTLE_DONE pgs=1050 cs=0 l=1 rx=0x7feef4001fe0 tx=0x7feef4002020).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:43:58.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:58.912+0000 7fef00ff9700 1 -- 172.21.15.42:0/306406152 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7feefc031940 con 0x7fef041231d0 2019-12-05T05:43:59.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:59.080+0000 7fef09fda700 1 -- 172.21.15.42:0/306406152 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fef0405b8c0 con 0x7fef041231d0 2019-12-05T05:43:59.088 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:59.084+0000 7fef08d78700 1 --2- 172.21.15.42:0/306406152 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fef041231d0 0x7fef041386a0 secure :-1 s=THROTTLE_DONE pgs=1050 cs=0 l=1 rx=0x7feef4001fe0 tx=0x7feef4002020).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:43:59.088 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:59.084+0000 7fef00ff9700 1 -- 172.21.15.42:0/306406152 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 163 v163) v1 ==== 95+0+2109 (secure 0 0 0) 0x7feefc005dc0 con 0x7fef041231d0 2019-12-05T05:43:59.090 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:43:59.090 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":163,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7755,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/358497296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":358497296},{"type":"v1","addr":"172.21.15.42:6835","nonce":358497296}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":163}],"filesystems":[{"mdsmap":{"epoch":162,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:42:48.961753+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7650},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7650":{"gid":7650,"name":"b","rank":0,"incarnation":159,"state":"up:active","state_seq":29,"addr":"172.21.15.40:6833/1029013337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1029013337},{"type":"v1","addr":"172.21.15.40:6833","nonce":1029013337}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:43:59.090 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 163 2019-12-05T05:43:59.092 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:59.088+0000 7fef09fda700 1 -- 172.21.15.42:0/306406152 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7feee403b1c0 msgr2=0x7feee403d6d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:43:59.093 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:59.088+0000 7fef09fda700 1 --2- 172.21.15.42:0/306406152 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7feee403b1c0 0x7feee403d6d0 crc :-1 s=READY pgs=579 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:43:59.093 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:59.088+0000 7fef09fda700 1 -- 172.21.15.42:0/306406152 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fef041231d0 msgr2=0x7fef041386a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:43:59.093 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:59.088+0000 7fef09fda700 1 --2- 172.21.15.42:0/306406152 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fef041231d0 0x7fef041386a0 secure :-1 s=READY pgs=1050 cs=0 l=1 rx=0x7feef4001fe0 tx=0x7feef4002020).stop 2019-12-05T05:43:59.093 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:59.088+0000 7fef09fda700 1 -- 172.21.15.42:0/306406152 shutdown_connections 2019-12-05T05:43:59.093 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:59.088+0000 7fef09fda700 1 --2- 172.21.15.42:0/306406152 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7feee403b1c0 0x7feee403d6d0 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:43:59.094 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:59.088+0000 7fef09fda700 1 --2- 172.21.15.42:0/306406152 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fef041231d0 0x7fef041386a0 unknown :-1 s=CLOSED pgs=1050 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:43:59.094 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:59.088+0000 7fef09fda700 1 -- 172.21.15.42:0/306406152 shutdown_connections 2019-12-05T05:43:59.094 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:59.088+0000 7fef09fda700 1 -- 172.21.15.42:0/306406152 wait complete. 2019-12-05T05:43:59.094 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:43:59.088+0000 7fef09fda700 1 -- 172.21.15.42:0/306406152 >> 172.21.15.42:0/306406152 conn(0x7fef0409f240 msgr2=0x7fef04059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:43:59.102 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T05:44:01.103 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:44:01.210 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:01.196+0000 7fe82ebed700 1 Processor -- start 2019-12-05T05:44:01.211 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:01.196+0000 7fe82ebed700 1 -- start start 2019-12-05T05:44:01.211 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:01.200+0000 7fe82ebed700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe8280a7740 0x7fe82808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:44:01.211 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:01.200+0000 7fe82ebed700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8280a0710 con 0x7fe82807e720 2019-12-05T05:44:01.211 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:01.200+0000 7fe82ebed700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8280a9ba0 con 0x7fe82809fae0 2019-12-05T05:44:01.211 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:01.200+0000 7fe82ebed700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8280a0b00 con 0x7fe8280a7740 2019-12-05T05:44:01.211 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:01.200+0000 7fe82d98b700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fe82809fae0 0x7fe82814efe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:39850/0 (socket says 172.21.15.42:39850) 2019-12-05T05:44:01.212 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:01.200+0000 7fe82d98b700 1 -- 172.21.15.42:0/2338156125 learned_addr learned my addr 172.21.15.42:0/2338156125 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:44:01.212 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:01.200+0000 7fe82d18a700 1 --2- 172.21.15.42:0/2338156125 >> v2:172.21.15.40:3300/0 conn(0x7fe8280a7740 0x7fe82808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:44:01.212 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:01.200+0000 7fe81ffff700 1 -- 172.21.15.42:0/2338156125 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1541686737 0 0) 0x7fe8280a9ba0 con 0x7fe82809fae0 2019-12-05T05:44:01.212 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:01.200+0000 7fe81ffff700 1 -- 172.21.15.42:0/2338156125 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8140030b0 con 0x7fe82809fae0 2019-12-05T05:44:01.213 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:01.200+0000 7fe81ffff700 1 -- 172.21.15.42:0/2338156125 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3114878824 0 0) 0x7fe8140030b0 con 0x7fe82809fae0 2019-12-05T05:44:01.213 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:01.200+0000 7fe81ffff700 1 -- 172.21.15.42:0/2338156125 >> v2:172.21.15.40:3300/0 conn(0x7fe8280a7740 msgr2=0x7fe82808df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:44:01.213 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:01.200+0000 7fe81ffff700 1 --2- 172.21.15.42:0/2338156125 >> v2:172.21.15.40:3300/0 conn(0x7fe8280a7740 0x7fe82808df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:44:01.213 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:01.200+0000 7fe81ffff700 1 -- 172.21.15.42:0/2338156125 >> v1:172.21.15.40:6789/0 conn(0x7fe82807e720 legacy=0x7fe82809f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:01.213 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:01.200+0000 7fe81ffff700 1 -- 172.21.15.42:0/2338156125 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe82807eb50 con 0x7fe82809fae0 2019-12-05T05:44:01.213 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:01.200+0000 7fe81ffff700 1 -- 172.21.15.42:0/2338156125 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7fe820001ad0 con 0x7fe82809fae0 2019-12-05T05:44:01.213 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:01.200+0000 7fe81ffff700 1 -- 172.21.15.42:0/2338156125 >> v1:172.21.15.42:6789/0 conn(0x7fe82809fae0 legacy=0x7fe82814efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:01.214 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:01.200+0000 7fe81ffff700 1 --2- 172.21.15.42:0/2338156125 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe8140036e0 0x7fe814003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:44:01.214 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:01.200+0000 7fe81ffff700 1 --2- 172.21.15.42:0/2338156125 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe814004420 0x7fe814004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:44:01.214 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:01.200+0000 7fe81ffff700 1 --2- 172.21.15.42:0/2338156125 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe814004df0 0x7fe81400b670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:44:01.214 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:01.200+0000 7fe81ffff700 1 -- 172.21.15.42:0/2338156125 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8280a0b00 con 0x7fe8140036e0 2019-12-05T05:44:01.214 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:01.200+0000 7fe81ffff700 1 -- 172.21.15.42:0/2338156125 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe81404fce0 con 0x7fe814004df0 2019-12-05T05:44:01.214 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:01.200+0000 7fe81ffff700 1 -- 172.21.15.42:0/2338156125 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe81404ff20 con 0x7fe814004420 2019-12-05T05:44:01.214 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:01.200+0000 7fe82d98b700 1 --2- 172.21.15.42:0/2338156125 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe814004df0 0x7fe81400b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:44:01.215 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:01.200+0000 7fe82d18a700 1 --2- 172.21.15.42:0/2338156125 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe8140036e0 0x7fe814003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:44:01.215 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:01.200+0000 7fe82c989700 1 --2- 172.21.15.42:0/2338156125 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe814004420 0x7fe814004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:44:01.215 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:01.200+0000 7fe82d98b700 1 -- 172.21.15.42:0/2338156125 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe814004420 msgr2=0x7fe814004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:44:01.215 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:01.200+0000 7fe82d98b700 1 --2- 172.21.15.42:0/2338156125 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe814004420 0x7fe814004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:44:01.216 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:01.200+0000 7fe82d98b700 1 -- 172.21.15.42:0/2338156125 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe8140036e0 msgr2=0x7fe814003b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:44:01.216 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:01.200+0000 7fe82d98b700 1 --2- 172.21.15.42:0/2338156125 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe8140036e0 0x7fe814003b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:44:01.216 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:01.200+0000 7fe82d98b700 1 -- 172.21.15.42:0/2338156125 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe814050220 con 0x7fe814004df0 2019-12-05T05:44:01.216 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:01.200+0000 7fe82d98b700 1 --2- 172.21.15.42:0/2338156125 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe814004df0 0x7fe81400b670 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fe820001cd0 tx=0x7fe82000c5c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:01.216 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:01.200+0000 7fe82d98b700 1 -- 172.21.15.42:0/2338156125 reap_dead start 2019-12-05T05:44:01.216 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:01.200+0000 7fe82d98b700 1 --2- 172.21.15.42:0/2338156125 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe814004df0 0x7fe81400b670 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fe820001cd0 tx=0x7fe82000c5c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:01.217 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:01.200+0000 7fe82d98b700 1 --2- 172.21.15.42:0/2338156125 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe814004df0 0x7fe81400b670 secure :-1 s=READY pgs=1346 cs=0 l=1 rx=0x7fe820001cd0 tx=0x7fe82000c5c0).ready entity=mon.1 client_cookie=d8740a2d72c3a82b server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:44:01.217 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:01.200+0000 7fe82d98b700 1 --2- 172.21.15.42:0/2338156125 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe814004df0 0x7fe81400b670 secure :-1 s=THROTTLE_DONE pgs=1346 cs=0 l=1 rx=0x7fe820001cd0 tx=0x7fe82000c5c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:01.217 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:01.200+0000 7fe81ffff700 1 -- 172.21.15.42:0/2338156125 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe820005bc0 con 0x7fe814004df0 2019-12-05T05:44:01.217 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:01.200+0000 7fe82d98b700 1 --2- 172.21.15.42:0/2338156125 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe814004df0 0x7fe81400b670 secure :-1 s=THROTTLE_DONE pgs=1346 cs=0 l=1 rx=0x7fe820001cd0 tx=0x7fe82000c5c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:01.217 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:01.200+0000 7fe82d98b700 1 --2- 172.21.15.42:0/2338156125 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe814004df0 0x7fe81400b670 secure :-1 s=THROTTLE_DONE pgs=1346 cs=0 l=1 rx=0x7fe820001cd0 tx=0x7fe82000c5c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:01.217 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:01.200+0000 7fe81ffff700 1 -- 172.21.15.42:0/2338156125 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8200057e0 con 0x7fe814004df0 2019-12-05T05:44:01.217 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:01.200+0000 7fe81ffff700 1 -- 172.21.15.42:0/2338156125 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe820004e90 con 0x7fe814004df0 2019-12-05T05:44:01.218 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:01.200+0000 7fe82ebed700 1 -- 172.21.15.42:0/2338156125 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe814004df0 msgr2=0x7fe81400b670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:01.218 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:01.200+0000 7fe82ebed700 1 --2- 172.21.15.42:0/2338156125 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe814004df0 0x7fe81400b670 secure :-1 s=READY pgs=1346 cs=0 l=1 rx=0x7fe820001cd0 tx=0x7fe82000c5c0).stop 2019-12-05T05:44:01.218 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:01.204+0000 7fe82ebed700 1 -- 172.21.15.42:0/2338156125 shutdown_connections 2019-12-05T05:44:01.218 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:01.204+0000 7fe82ebed700 1 --2- 172.21.15.42:0/2338156125 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe814004df0 0x7fe81400b670 unknown :-1 s=CLOSED pgs=1346 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:44:01.218 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:01.204+0000 7fe82ebed700 1 -- 172.21.15.42:0/2338156125 shutdown_connections 2019-12-05T05:44:01.218 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:01.204+0000 7fe82ebed700 1 -- 172.21.15.42:0/2338156125 wait complete. 2019-12-05T05:44:01.218 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:01.204+0000 7fe82ebed700 1 -- 172.21.15.42:0/2338156125 >> 172.21.15.42:0/2338156125 conn(0x7fe8280aabe0 msgr2=0x7fe8280a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:44:01.219 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:01.204+0000 7fe82ebed700 1 Processor -- start 2019-12-05T05:44:01.219 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:01.204+0000 7fe82ebed700 1 -- start start 2019-12-05T05:44:01.222 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:01.204+0000 7fe82ebed700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe828065860 0x7fe8280639d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:44:01.222 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:01.204+0000 7fe82ebed700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe82809fcd0 0x7fe828125e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:44:01.222 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:01.204+0000 7fe82ebed700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe828063190 con 0x7fe8281263a0 2019-12-05T05:44:01.222 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:01.204+0000 7fe82ebed700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe828063500 con 0x7fe82809fcd0 2019-12-05T05:44:01.222 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:01.204+0000 7fe82ebed700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8281244f0 con 0x7fe828065860 2019-12-05T05:44:01.222 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:01.208+0000 7fe82d98b700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fe8281263a0 0x7fe8280722e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:39862/0 (socket says 172.21.15.42:39862) 2019-12-05T05:44:01.223 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:01.208+0000 7fe82d98b700 1 -- 172.21.15.42:0/4045167994 learned_addr learned my addr 172.21.15.42:0/4045167994 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:44:01.223 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:01.208+0000 7fe82d18a700 1 --2- 172.21.15.42:0/4045167994 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe828065860 0x7fe8280639d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:44:01.223 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:01.208+0000 7fe82c989700 1 --2- 172.21.15.42:0/4045167994 >> v2:172.21.15.40:3300/0 conn(0x7fe82809fcd0 0x7fe828125e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:44:01.223 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:01.208+0000 7fe81dffb700 1 -- 172.21.15.42:0/4045167994 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2248819670 0 0) 0x7fe828063190 con 0x7fe8281263a0 2019-12-05T05:44:01.223 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:01.208+0000 7fe81dffb700 1 -- 172.21.15.42:0/4045167994 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8080030b0 con 0x7fe8281263a0 2019-12-05T05:44:01.223 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:01.208+0000 7fe82c989700 1 -- 172.21.15.42:0/4045167994 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe828065860 msgr2=0x7fe8280639d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:01.223 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:01.208+0000 7fe82c989700 1 --2- 172.21.15.42:0/4045167994 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe828065860 0x7fe8280639d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:44:01.224 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:01.208+0000 7fe82c989700 1 -- 172.21.15.42:0/4045167994 >> v1:172.21.15.42:6789/0 conn(0x7fe8281263a0 legacy=0x7fe8280722e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:01.224 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:01.208+0000 7fe82c989700 1 -- 172.21.15.42:0/4045167994 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8280a4840 con 0x7fe82809fcd0 2019-12-05T05:44:01.224 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:01.208+0000 7fe82c989700 1 --2- 172.21.15.42:0/4045167994 >> v2:172.21.15.40:3300/0 conn(0x7fe82809fcd0 0x7fe828125e50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fe8180013b0 tx=0x7fe818010a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:01.224 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:01.208+0000 7fe82c989700 1 --2- 172.21.15.42:0/4045167994 >> v2:172.21.15.40:3300/0 conn(0x7fe82809fcd0 0x7fe828125e50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fe8180013b0 tx=0x7fe818010a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:01.224 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:01.208+0000 7fe82c989700 1 --2- 172.21.15.42:0/4045167994 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe82809fcd0 0x7fe828125e50 secure :-1 s=READY pgs=1052 cs=0 l=1 rx=0x7fe8180013b0 tx=0x7fe818010a20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:44:01.224 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:01.208+0000 7fe82c989700 1 --2- 172.21.15.42:0/4045167994 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe82809fcd0 0x7fe828125e50 secure :-1 s=THROTTLE_DONE pgs=1052 cs=0 l=1 rx=0x7fe8180013b0 tx=0x7fe818010a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:01.224 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:01.208+0000 7fe81dffb700 1 -- 172.21.15.42:0/4045167994 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe8180034b0 con 0x7fe82809fcd0 2019-12-05T05:44:01.225 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:01.208+0000 7fe82c989700 1 --2- 172.21.15.42:0/4045167994 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe82809fcd0 0x7fe828125e50 secure :-1 s=THROTTLE_DONE pgs=1052 cs=0 l=1 rx=0x7fe8180013b0 tx=0x7fe818010a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:01.225 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:01.208+0000 7fe81dffb700 1 -- 172.21.15.42:0/4045167994 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe818001af0 con 0x7fe82809fcd0 2019-12-05T05:44:01.225 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:01.208+0000 7fe82ebed700 1 -- 172.21.15.42:0/4045167994 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe82811a170 con 0x7fe82809fcd0 2019-12-05T05:44:01.225 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:01.208+0000 7fe82c989700 1 --2- 172.21.15.42:0/4045167994 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe82809fcd0 0x7fe828125e50 secure :-1 s=THROTTLE_DONE pgs=1052 cs=0 l=1 rx=0x7fe8180013b0 tx=0x7fe818010a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:01.225 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:01.208+0000 7fe81dffb700 1 -- 172.21.15.42:0/4045167994 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe81800f4c0 con 0x7fe82809fcd0 2019-12-05T05:44:01.225 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:01.208+0000 7fe82ebed700 1 -- 172.21.15.42:0/4045167994 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8280a0100 con 0x7fe82809fcd0 2019-12-05T05:44:01.226 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:01.208+0000 7fe82c989700 1 --2- 172.21.15.42:0/4045167994 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe82809fcd0 0x7fe828125e50 secure :-1 s=THROTTLE_DONE pgs=1052 cs=0 l=1 rx=0x7fe8180013b0 tx=0x7fe818010a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:01.226 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:01.208+0000 7fe81dffb700 1 -- 172.21.15.42:0/4045167994 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fe818007430 con 0x7fe82809fcd0 2019-12-05T05:44:01.226 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:01.208+0000 7fe82c989700 1 --2- 172.21.15.42:0/4045167994 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe82809fcd0 0x7fe828125e50 secure :-1 s=THROTTLE_DONE pgs=1052 cs=0 l=1 rx=0x7fe8180013b0 tx=0x7fe818010a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:01.226 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:01.212+0000 7fe81dffb700 1 --2- 172.21.15.42:0/4045167994 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe80803b280 0x7fe80803d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:44:01.227 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:01.212+0000 7fe81dffb700 1 -- 172.21.15.42:0/4045167994 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 8267+0+0 (secure 0 0 0) 0x7fe81804b080 con 0x7fe82809fcd0 2019-12-05T05:44:01.227 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:01.212+0000 7fe82d18a700 1 --2- 172.21.15.42:0/4045167994 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe80803b280 0x7fe80803d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:44:01.227 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:01.212+0000 7fe82d18a700 1 --2- 172.21.15.42:0/4045167994 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe80803b280 0x7fe80803d790 crc :-1 s=READY pgs=580 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:44:01.227 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:01.212+0000 7fe82ebed700 1 -- 172.21.15.42:0/4045167994 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe82805d110 con 0x7fe82809fcd0 2019-12-05T05:44:01.228 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:01.220+0000 7fe82c989700 1 --2- 172.21.15.42:0/4045167994 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe82809fcd0 0x7fe828125e50 secure :-1 s=THROTTLE_DONE pgs=1052 cs=0 l=1 rx=0x7fe8180013b0 tx=0x7fe818010a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:01.228 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:01.220+0000 7fe81dffb700 1 -- 172.21.15.42:0/4045167994 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fe82811a170 con 0x7fe82809fcd0 2019-12-05T05:44:01.394 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:01.388+0000 7fe82ebed700 1 -- 172.21.15.42:0/4045167994 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe82805ae30 con 0x7fe82809fcd0 2019-12-05T05:44:01.395 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:01.388+0000 7fe82c989700 1 --2- 172.21.15.42:0/4045167994 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe82809fcd0 0x7fe828125e50 secure :-1 s=THROTTLE_DONE pgs=1052 cs=0 l=1 rx=0x7fe8180013b0 tx=0x7fe818010a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:01.395 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:01.388+0000 7fe81dffb700 1 -- 172.21.15.42:0/4045167994 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 163 v163) v1 ==== 95+0+2109 (secure 0 0 0) 0x7fe818032080 con 0x7fe82809fcd0 2019-12-05T05:44:01.397 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:44:01.397 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":163,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7755,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/358497296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":358497296},{"type":"v1","addr":"172.21.15.42:6835","nonce":358497296}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":163}],"filesystems":[{"mdsmap":{"epoch":162,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:42:48.961753+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7650},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7650":{"gid":7650,"name":"b","rank":0,"incarnation":159,"state":"up:active","state_seq":29,"addr":"172.21.15.40:6833/1029013337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1029013337},{"type":"v1","addr":"172.21.15.40:6833","nonce":1029013337}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:44:01.397 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 163 2019-12-05T05:44:01.400 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:01.392+0000 7fe82ebed700 1 -- 172.21.15.42:0/4045167994 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe80803b280 msgr2=0x7fe80803d790 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:01.400 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:01.392+0000 7fe82ebed700 1 --2- 172.21.15.42:0/4045167994 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe80803b280 0x7fe80803d790 crc :-1 s=READY pgs=580 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:44:01.400 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:01.396+0000 7fe82ebed700 1 -- 172.21.15.42:0/4045167994 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe82809fcd0 msgr2=0x7fe828125e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:01.400 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:01.396+0000 7fe82ebed700 1 --2- 172.21.15.42:0/4045167994 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe82809fcd0 0x7fe828125e50 secure :-1 s=READY pgs=1052 cs=0 l=1 rx=0x7fe8180013b0 tx=0x7fe818010a20).stop 2019-12-05T05:44:01.400 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:01.396+0000 7fe82ebed700 1 -- 172.21.15.42:0/4045167994 shutdown_connections 2019-12-05T05:44:01.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:01.396+0000 7fe82ebed700 1 --2- 172.21.15.42:0/4045167994 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe80803b280 0x7fe80803d790 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:44:01.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:01.396+0000 7fe82ebed700 1 --2- 172.21.15.42:0/4045167994 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe828065860 0x7fe8280639d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:44:01.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:01.396+0000 7fe82ebed700 1 --2- 172.21.15.42:0/4045167994 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe82809fcd0 0x7fe828125e50 unknown :-1 s=CLOSED pgs=1052 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:44:01.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:01.396+0000 7fe82ebed700 1 -- 172.21.15.42:0/4045167994 shutdown_connections 2019-12-05T05:44:01.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:01.396+0000 7fe82ebed700 1 -- 172.21.15.42:0/4045167994 wait complete. 2019-12-05T05:44:01.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:01.396+0000 7fe82ebed700 1 -- 172.21.15.42:0/4045167994 >> 172.21.15.42:0/4045167994 conn(0x7fe8280a6f80 msgr2=0x7fe8280a9640 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:44:01.410 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T05:44:03.411 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:44:03.514 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:03.504+0000 7f0225830700 1 Processor -- start 2019-12-05T05:44:03.514 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:03.504+0000 7f0225830700 1 -- start start 2019-12-05T05:44:03.515 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:03.504+0000 7f0225830700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f022008df80 0x7f022009f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:44:03.515 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:03.504+0000 7f0225830700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f022009f830 0x7f022014efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:44:03.515 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:03.504+0000 7f0225830700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f022009fc60 con 0x7f02200a7740 2019-12-05T05:44:03.515 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:03.504+0000 7f0225830700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02200a0b00 con 0x7f022008df80 2019-12-05T05:44:03.515 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:03.504+0000 7f0225830700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f022008e5b0 con 0x7f022009f830 2019-12-05T05:44:03.516 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:03.504+0000 7f021ffff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f022009f830 0x7f022014efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:44:03.516 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:03.504+0000 7f021ffff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f022009f830 0x7f022014efe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:39832/0 (socket says 172.21.15.42:39832) 2019-12-05T05:44:03.516 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:03.504+0000 7f021effd700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f022008df80 0x7f022009f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:44:03.516 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:03.504+0000 7f021ffff700 1 -- 172.21.15.42:0/106483401 learned_addr learned my addr 172.21.15.42:0/106483401 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:44:03.517 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:03.504+0000 7f021ffff700 1 -- 172.21.15.42:0/106483401 >> v2:172.21.15.40:3300/0 conn(0x7f022008df80 msgr2=0x7f022009f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:44:03.517 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:03.504+0000 7f021ffff700 1 --2- 172.21.15.42:0/106483401 >> v2:172.21.15.40:3300/0 conn(0x7f022008df80 0x7f022009f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:44:03.517 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:03.504+0000 7f021ffff700 1 -- 172.21.15.42:0/106483401 >> v1:172.21.15.40:6789/0 conn(0x7f02200a7740 legacy=0x7f022007e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:03.517 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:03.504+0000 7f021ffff700 1 -- 172.21.15.42:0/106483401 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02200a0710 con 0x7f022009f830 2019-12-05T05:44:03.517 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:03.504+0000 7f021ffff700 1 --2- 172.21.15.42:0/106483401 >> v2:172.21.15.42:3300/0 conn(0x7f022009f830 0x7f022014efe0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f02180032b0 tx=0x7f02180100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:03.518 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:03.504+0000 7f021ffff700 1 --2- 172.21.15.42:0/106483401 >> v2:172.21.15.42:3300/0 conn(0x7f022009f830 0x7f022014efe0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f02180032b0 tx=0x7f02180100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:03.518 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:03.504+0000 7f021ffff700 1 --2- 172.21.15.42:0/106483401 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f022009f830 0x7f022014efe0 secure :-1 s=READY pgs=1348 cs=0 l=1 rx=0x7f02180032b0 tx=0x7f02180100b0).ready entity=mon.1 client_cookie=2d177f668efcf8ae server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:44:03.518 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:03.504+0000 7f021ffff700 1 --2- 172.21.15.42:0/106483401 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f022009f830 0x7f022014efe0 secure :-1 s=THROTTLE_DONE pgs=1348 cs=0 l=1 rx=0x7f02180032b0 tx=0x7f02180100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:03.518 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:03.504+0000 7f021ffff700 1 --2- 172.21.15.42:0/106483401 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f022009f830 0x7f022014efe0 secure :-1 s=THROTTLE_DONE pgs=1348 cs=0 l=1 rx=0x7f02180032b0 tx=0x7f02180100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:03.519 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:03.504+0000 7f021e7fc700 1 -- 172.21.15.42:0/106483401 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0218010d60 con 0x7f022009f830 2019-12-05T05:44:03.519 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:03.504+0000 7f021ffff700 1 --2- 172.21.15.42:0/106483401 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f022009f830 0x7f022014efe0 secure :-1 s=THROTTLE_DONE pgs=1348 cs=0 l=1 rx=0x7f02180032b0 tx=0x7f02180100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:03.519 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:03.508+0000 7f021e7fc700 1 -- 172.21.15.42:0/106483401 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0218010980 con 0x7f022009f830 2019-12-05T05:44:03.519 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:03.508+0000 7f021e7fc700 1 -- 172.21.15.42:0/106483401 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f02180114a0 con 0x7f022009f830 2019-12-05T05:44:03.520 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:03.508+0000 7f0225830700 1 -- 172.21.15.42:0/106483401 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f022009f830 msgr2=0x7f022014efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:03.520 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:03.508+0000 7f0225830700 1 --2- 172.21.15.42:0/106483401 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f022009f830 0x7f022014efe0 secure :-1 s=READY pgs=1348 cs=0 l=1 rx=0x7f02180032b0 tx=0x7f02180100b0).stop 2019-12-05T05:44:03.520 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:03.508+0000 7f0225830700 1 -- 172.21.15.42:0/106483401 shutdown_connections 2019-12-05T05:44:03.521 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:03.508+0000 7f0225830700 1 --2- 172.21.15.42:0/106483401 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f022009f830 0x7f022014efe0 unknown :-1 s=CLOSED pgs=1348 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:44:03.521 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:03.508+0000 7f0225830700 1 --2- 172.21.15.42:0/106483401 >> v2:172.21.15.40:3300/0 conn(0x7f022008df80 0x7f022009f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:44:03.521 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:03.508+0000 7f0225830700 1 -- 172.21.15.42:0/106483401 shutdown_connections 2019-12-05T05:44:03.521 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:03.508+0000 7f0225830700 1 -- 172.21.15.42:0/106483401 wait complete. 2019-12-05T05:44:03.522 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:03.508+0000 7f0225830700 1 -- 172.21.15.42:0/106483401 >> 172.21.15.42:0/106483401 conn(0x7f02200aabe0 msgr2=0x7f02200a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:44:03.522 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:03.508+0000 7f0225830700 1 Processor -- start 2019-12-05T05:44:03.522 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:03.508+0000 7f0225830700 1 -- start start 2019-12-05T05:44:03.526 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:03.508+0000 7f0225830700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f02201236d0 0x7f022012d700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:44:03.526 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:03.508+0000 7f0225830700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f022012ce20 con 0x7f02200a4d50 2019-12-05T05:44:03.526 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:03.508+0000 7f0225830700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f022012dc50 con 0x7f022007b9e0 2019-12-05T05:44:03.527 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:03.508+0000 7f0225830700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f022007c8d0 con 0x7f02201236d0 2019-12-05T05:44:03.527 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:03.508+0000 7f021f7fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f02201236d0 0x7f022012d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:44:03.527 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:03.508+0000 7f021effd700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f022007b9e0 0x7f022011bc20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:39874/0 (socket says 172.21.15.42:39874) 2019-12-05T05:44:03.527 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:03.508+0000 7f021effd700 1 -- 172.21.15.42:0/1547433820 learned_addr learned my addr 172.21.15.42:0/1547433820 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:44:03.528 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:03.512+0000 7f0217fff700 1 -- 172.21.15.42:0/1547433820 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1866067570 0 0) 0x7f022012ce20 con 0x7f02200a4d50 2019-12-05T05:44:03.528 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:03.512+0000 7f0217fff700 1 -- 172.21.15.42:0/1547433820 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01f80030b0 con 0x7f02200a4d50 2019-12-05T05:44:03.528 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:03.512+0000 7f021f7fe700 1 -- 172.21.15.42:0/1547433820 >> v1:172.21.15.42:6789/0 conn(0x7f022007b9e0 legacy=0x7f022011bc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:03.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:03.512+0000 7f021f7fe700 1 -- 172.21.15.42:0/1547433820 >> v1:172.21.15.40:6789/0 conn(0x7f02200a4d50 legacy=0x7f0220137830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:03.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:03.512+0000 7f0217fff700 1 -- 172.21.15.42:0/1547433820 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1989963608 0 0) 0x7f022012dc50 con 0x7f022007b9e0 2019-12-05T05:44:03.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:03.512+0000 7f021f7fe700 1 -- 172.21.15.42:0/1547433820 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02200a5180 con 0x7f02201236d0 2019-12-05T05:44:03.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:03.512+0000 7f021f7fe700 1 --2- 172.21.15.42:0/1547433820 >> v2:172.21.15.42:3300/0 conn(0x7f02201236d0 0x7f022012d700 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f0208000e90 tx=0x7f020800c2f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:03.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:03.512+0000 7f021f7fe700 1 --2- 172.21.15.42:0/1547433820 >> v2:172.21.15.42:3300/0 conn(0x7f02201236d0 0x7f022012d700 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f0208000e90 tx=0x7f020800c2f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:03.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:03.512+0000 7f021f7fe700 1 --2- 172.21.15.42:0/1547433820 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f02201236d0 0x7f022012d700 secure :-1 s=READY pgs=1350 cs=0 l=1 rx=0x7f0208000e90 tx=0x7f020800c2f0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:44:03.530 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:03.512+0000 7f021f7fe700 1 --2- 172.21.15.42:0/1547433820 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f02201236d0 0x7f022012d700 secure :-1 s=THROTTLE_DONE pgs=1350 cs=0 l=1 rx=0x7f0208000e90 tx=0x7f020800c2f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:03.530 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:03.512+0000 7f0217fff700 1 -- 172.21.15.42:0/1547433820 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f020800cb90 con 0x7f02201236d0 2019-12-05T05:44:03.530 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:03.512+0000 7f021f7fe700 1 --2- 172.21.15.42:0/1547433820 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f02201236d0 0x7f022012d700 secure :-1 s=THROTTLE_DONE pgs=1350 cs=0 l=1 rx=0x7f0208000e90 tx=0x7f020800c2f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:03.531 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:03.512+0000 7f021f7fe700 1 --2- 172.21.15.42:0/1547433820 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f02201236d0 0x7f022012d700 secure :-1 s=THROTTLE_DONE pgs=1350 cs=0 l=1 rx=0x7f0208000e90 tx=0x7f020800c2f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:03.531 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:03.512+0000 7f0217fff700 1 -- 172.21.15.42:0/1547433820 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f020800c7b0 con 0x7f02201236d0 2019-12-05T05:44:03.531 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:03.512+0000 7f0217fff700 1 -- 172.21.15.42:0/1547433820 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0208011030 con 0x7f02201236d0 2019-12-05T05:44:03.531 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:03.512+0000 7f0225830700 1 -- 172.21.15.42:0/1547433820 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02200a5470 con 0x7f02201236d0 2019-12-05T05:44:03.531 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:03.512+0000 7f0225830700 1 -- 172.21.15.42:0/1547433820 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f022008eb70 con 0x7f02201236d0 2019-12-05T05:44:03.532 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:03.512+0000 7f021f7fe700 1 --2- 172.21.15.42:0/1547433820 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f02201236d0 0x7f022012d700 secure :-1 s=THROTTLE_DONE pgs=1350 cs=0 l=1 rx=0x7f0208000e90 tx=0x7f020800c2f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:03.532 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:03.512+0000 7f0217fff700 1 -- 172.21.15.42:0/1547433820 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f0208011340 con 0x7f02201236d0 2019-12-05T05:44:03.532 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:03.512+0000 7f021f7fe700 1 --2- 172.21.15.42:0/1547433820 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f02201236d0 0x7f022012d700 secure :-1 s=THROTTLE_DONE pgs=1350 cs=0 l=1 rx=0x7f0208000e90 tx=0x7f020800c2f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:03.532 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:03.516+0000 7f0217fff700 1 --2- 172.21.15.42:0/1547433820 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f01f803d3a0 0x7f01f803f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:44:03.533 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:03.516+0000 7f0217fff700 1 -- 172.21.15.42:0/1547433820 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 8267+0+0 (secure 0 0 0) 0x7f020804b0b0 con 0x7f02201236d0 2019-12-05T05:44:03.533 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:03.516+0000 7f021effd700 1 --2- 172.21.15.42:0/1547433820 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f01f803d3a0 0x7f01f803f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:44:03.533 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:03.516+0000 7f021effd700 1 --2- 172.21.15.42:0/1547433820 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f01f803d3a0 0x7f01f803f8b0 crc :-1 s=READY pgs=581 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:44:03.533 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:03.520+0000 7f0225830700 1 -- 172.21.15.42:0/1547433820 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f022005d110 con 0x7f02201236d0 2019-12-05T05:44:03.534 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:03.528+0000 7f021f7fe700 1 --2- 172.21.15.42:0/1547433820 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f02201236d0 0x7f022012d700 secure :-1 s=THROTTLE_DONE pgs=1350 cs=0 l=1 rx=0x7f0208000e90 tx=0x7f020800c2f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:03.534 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:03.528+0000 7f0217fff700 1 -- 172.21.15.42:0/1547433820 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f0208032130 con 0x7f02201236d0 2019-12-05T05:44:03.701 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:03.696+0000 7f0225830700 1 -- 172.21.15.42:0/1547433820 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f022005b8c0 con 0x7f02201236d0 2019-12-05T05:44:03.702 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:03.696+0000 7f021f7fe700 1 --2- 172.21.15.42:0/1547433820 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f02201236d0 0x7f022012d700 secure :-1 s=THROTTLE_DONE pgs=1350 cs=0 l=1 rx=0x7f0208000e90 tx=0x7f020800c2f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:03.702 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:03.696+0000 7f0217fff700 1 -- 172.21.15.42:0/1547433820 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 163 v163) v1 ==== 95+0+2109 (secure 0 0 0) 0x7f0208032390 con 0x7f02201236d0 2019-12-05T05:44:03.703 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 163 2019-12-05T05:44:03.703 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:44:03.703 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":163,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7755,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/358497296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":358497296},{"type":"v1","addr":"172.21.15.42:6835","nonce":358497296}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":163}],"filesystems":[{"mdsmap":{"epoch":162,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:42:48.961753+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7650},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7650":{"gid":7650,"name":"b","rank":0,"incarnation":159,"state":"up:active","state_seq":29,"addr":"172.21.15.40:6833/1029013337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1029013337},{"type":"v1","addr":"172.21.15.40:6833","nonce":1029013337}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:44:03.705 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:03.700+0000 7f0225830700 1 -- 172.21.15.42:0/1547433820 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f01f803d3a0 msgr2=0x7f01f803f8b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:03.705 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:03.700+0000 7f0225830700 1 --2- 172.21.15.42:0/1547433820 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f01f803d3a0 0x7f01f803f8b0 crc :-1 s=READY pgs=581 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:44:03.705 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:03.700+0000 7f0225830700 1 -- 172.21.15.42:0/1547433820 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f02201236d0 msgr2=0x7f022012d700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:03.705 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:03.700+0000 7f0225830700 1 --2- 172.21.15.42:0/1547433820 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f02201236d0 0x7f022012d700 secure :-1 s=READY pgs=1350 cs=0 l=1 rx=0x7f0208000e90 tx=0x7f020800c2f0).stop 2019-12-05T05:44:03.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:03.700+0000 7f0225830700 1 -- 172.21.15.42:0/1547433820 shutdown_connections 2019-12-05T05:44:03.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:03.700+0000 7f0225830700 1 --2- 172.21.15.42:0/1547433820 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f01f803d3a0 0x7f01f803f8b0 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:44:03.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:03.700+0000 7f0225830700 1 --2- 172.21.15.42:0/1547433820 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f02201236d0 0x7f022012d700 unknown :-1 s=CLOSED pgs=1350 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:44:03.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:03.700+0000 7f0225830700 1 -- 172.21.15.42:0/1547433820 shutdown_connections 2019-12-05T05:44:03.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:03.700+0000 7f0225830700 1 -- 172.21.15.42:0/1547433820 wait complete. 2019-12-05T05:44:03.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:03.700+0000 7f0225830700 1 -- 172.21.15.42:0/1547433820 >> 172.21.15.42:0/1547433820 conn(0x7f022008fda0 msgr2=0x7f0220059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:44:03.715 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T05:44:05.717 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:44:05.825 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:05.816+0000 7fc3d4c36700 1 Processor -- start 2019-12-05T05:44:05.826 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:05.816+0000 7fc3d4c36700 1 -- start start 2019-12-05T05:44:05.826 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:05.816+0000 7fc3d4c36700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc3d008df80 0x7fc3d009f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:44:05.826 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:05.816+0000 7fc3d4c36700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc3d009f830 0x7fc3d014efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:44:05.826 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:05.816+0000 7fc3d4c36700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3d009fc60 con 0x7fc3d00a7740 2019-12-05T05:44:05.826 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:05.816+0000 7fc3d4c36700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3d00a0b00 con 0x7fc3d009f830 2019-12-05T05:44:05.827 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:05.816+0000 7fc3d4c36700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3d008e5b0 con 0x7fc3d008df80 2019-12-05T05:44:05.827 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:05.816+0000 7fc3ce59c700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc3d008df80 0x7fc3d009f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:44:05.827 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:05.816+0000 7fc3cf59e700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc3d009f830 0x7fc3d014efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:44:05.827 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:05.816+0000 7fc3cf59e700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc3d009f830 0x7fc3d014efe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.42:55908/0 (socket says 172.21.15.42:55908) 2019-12-05T05:44:05.828 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:05.816+0000 7fc3ce59c700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc3d008df80 0x7fc3d009f2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:40324/0 (socket says 172.21.15.42:40324) 2019-12-05T05:44:05.828 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:05.816+0000 7fc3cf59e700 1 -- 172.21.15.42:0/1949419477 learned_addr learned my addr 172.21.15.42:0/1949419477 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:44:05.828 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:05.816+0000 7fc3ce59c700 1 -- 172.21.15.42:0/1949419477 >> v2:172.21.15.40:3300/0 conn(0x7fc3d009f830 msgr2=0x7fc3d014efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:44:05.828 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:05.816+0000 7fc3ce59c700 1 --2- 172.21.15.42:0/1949419477 >> v2:172.21.15.40:3300/0 conn(0x7fc3d009f830 0x7fc3d014efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:44:05.828 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:05.816+0000 7fc3ce59c700 1 -- 172.21.15.42:0/1949419477 >> v1:172.21.15.40:6789/0 conn(0x7fc3d00a7740 legacy=0x7fc3d007e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:05.828 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:05.816+0000 7fc3ce59c700 1 -- 172.21.15.42:0/1949419477 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3d00a0710 con 0x7fc3d008df80 2019-12-05T05:44:05.828 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:05.816+0000 7fc3cf59e700 1 --2- 172.21.15.42:0/1949419477 >> v2:172.21.15.40:3300/0 conn(0x7fc3d009f830 0x7fc3d014efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:44:05.829 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:05.816+0000 7fc3ce59c700 1 --2- 172.21.15.42:0/1949419477 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc3d008df80 0x7fc3d009f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fc3b8008790 tx=0x7fc3b800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:05.829 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:05.816+0000 7fc3ce59c700 1 --2- 172.21.15.42:0/1949419477 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc3d008df80 0x7fc3d009f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fc3b8008790 tx=0x7fc3b800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:05.829 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:05.816+0000 7fc3ce59c700 1 --2- 172.21.15.42:0/1949419477 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc3d008df80 0x7fc3d009f2e0 secure :-1 s=READY pgs=286 cs=0 l=1 rx=0x7fc3b8008790 tx=0x7fc3b800fc70).ready entity=mon.2 client_cookie=7f5eb0504599f3aa server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:44:05.829 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:05.816+0000 7fc3ce59c700 1 --2- 172.21.15.42:0/1949419477 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc3d008df80 0x7fc3d009f2e0 secure :-1 s=THROTTLE_DONE pgs=286 cs=0 l=1 rx=0x7fc3b8008790 tx=0x7fc3b800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:05.829 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:05.816+0000 7fc3cdd9b700 1 -- 172.21.15.42:0/1949419477 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc3b8008bb0 con 0x7fc3d008df80 2019-12-05T05:44:05.829 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:05.816+0000 7fc3ce59c700 1 --2- 172.21.15.42:0/1949419477 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc3d008df80 0x7fc3d009f2e0 secure :-1 s=THROTTLE_DONE pgs=286 cs=0 l=1 rx=0x7fc3b8008790 tx=0x7fc3b800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:05.830 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:05.816+0000 7fc3ce59c700 1 --2- 172.21.15.42:0/1949419477 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc3d008df80 0x7fc3d009f2e0 secure :-1 s=THROTTLE_DONE pgs=286 cs=0 l=1 rx=0x7fc3b8008790 tx=0x7fc3b800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:05.830 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:05.816+0000 7fc3cdd9b700 1 -- 172.21.15.42:0/1949419477 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3b80087d0 con 0x7fc3d008df80 2019-12-05T05:44:05.830 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:05.816+0000 7fc3cdd9b700 1 -- 172.21.15.42:0/1949419477 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc3b8011860 con 0x7fc3d008df80 2019-12-05T05:44:05.830 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:05.816+0000 7fc3d4c36700 1 -- 172.21.15.42:0/1949419477 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc3d008df80 msgr2=0x7fc3d009f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:05.830 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:05.816+0000 7fc3d4c36700 1 --2- 172.21.15.42:0/1949419477 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc3d008df80 0x7fc3d009f2e0 secure :-1 s=READY pgs=286 cs=0 l=1 rx=0x7fc3b8008790 tx=0x7fc3b800fc70).stop 2019-12-05T05:44:05.830 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:05.820+0000 7fc3d4c36700 1 -- 172.21.15.42:0/1949419477 shutdown_connections 2019-12-05T05:44:05.830 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:05.820+0000 7fc3d4c36700 1 --2- 172.21.15.42:0/1949419477 >> v2:172.21.15.40:3300/0 conn(0x7fc3d009f830 0x7fc3d014efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:44:05.831 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:05.820+0000 7fc3d4c36700 1 --2- 172.21.15.42:0/1949419477 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc3d008df80 0x7fc3d009f2e0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:44:05.831 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:05.820+0000 7fc3d4c36700 1 -- 172.21.15.42:0/1949419477 shutdown_connections 2019-12-05T05:44:05.831 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:05.820+0000 7fc3d4c36700 1 -- 172.21.15.42:0/1949419477 wait complete. 2019-12-05T05:44:05.831 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:05.820+0000 7fc3d4c36700 1 -- 172.21.15.42:0/1949419477 >> 172.21.15.42:0/1949419477 conn(0x7fc3d00aabe0 msgr2=0x7fc3d00a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:44:05.831 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:05.820+0000 7fc3d4c36700 1 Processor -- start 2019-12-05T05:44:05.832 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:05.820+0000 7fc3d4c36700 1 -- start start 2019-12-05T05:44:05.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:05.820+0000 7fc3d4c36700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc3d011a950 0x7fc3d0124810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:44:05.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:05.820+0000 7fc3d4c36700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc3d011c390 0x7fc3d008f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:44:05.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:05.820+0000 7fc3d4c36700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3d01128f0 con 0x7fc3d011bf60 2019-12-05T05:44:05.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:05.820+0000 7fc3d4c36700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3d011bd00 con 0x7fc3d011a950 2019-12-05T05:44:05.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:05.820+0000 7fc3d4c36700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3d00a6d50 con 0x7fc3d011c390 2019-12-05T05:44:05.837 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:05.820+0000 7fc3ced9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc3d011a950 0x7fc3d0124810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:44:05.837 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:05.820+0000 7fc3ce59c700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fc3d011bf60 0x7fc3d008eb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:39902/0 (socket says 172.21.15.42:39902) 2019-12-05T05:44:05.837 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:05.820+0000 7fc3ce59c700 1 -- 172.21.15.42:0/2238579985 learned_addr learned my addr 172.21.15.42:0/2238579985 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:44:05.837 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:05.820+0000 7fc3cf59e700 1 --2- 172.21.15.42:0/2238579985 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc3d011c390 0x7fc3d008f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:44:05.837 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:05.820+0000 7fc3ced9d700 1 -- 172.21.15.42:0/2238579985 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc3d011c390 msgr2=0x7fc3d008f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:05.838 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:05.820+0000 7fc3ced9d700 1 --2- 172.21.15.42:0/2238579985 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc3d011c390 0x7fc3d008f370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:44:05.838 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:05.820+0000 7fc3ced9d700 1 -- 172.21.15.42:0/2238579985 >> v1:172.21.15.42:6789/0 conn(0x7fc3d011bf60 legacy=0x7fc3d008eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:05.846 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:05.820+0000 7fc3c77fe700 1 -- 172.21.15.42:0/2238579985 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3405151088 0 0) 0x7fc3d01128f0 con 0x7fc3d011bf60 2019-12-05T05:44:05.846 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:05.820+0000 7fc3ced9d700 1 -- 172.21.15.42:0/2238579985 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3d011c7c0 con 0x7fc3d011a950 2019-12-05T05:44:05.846 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:05.820+0000 7fc3cf59e700 1 --2- 172.21.15.42:0/2238579985 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc3d011c390 0x7fc3d008f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T05:44:05.846 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:05.820+0000 7fc3ced9d700 1 --2- 172.21.15.42:0/2238579985 >> v2:172.21.15.42:3300/0 conn(0x7fc3d011a950 0x7fc3d0124810 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fc3d00a6f10 tx=0x7fc3c800c6a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:05.846 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:05.820+0000 7fc3ced9d700 1 --2- 172.21.15.42:0/2238579985 >> v2:172.21.15.42:3300/0 conn(0x7fc3d011a950 0x7fc3d0124810 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fc3d00a6f10 tx=0x7fc3c800c6a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:05.847 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:05.820+0000 7fc3ced9d700 1 --2- 172.21.15.42:0/2238579985 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc3d011a950 0x7fc3d0124810 secure :-1 s=READY pgs=1352 cs=0 l=1 rx=0x7fc3d00a6f10 tx=0x7fc3c800c6a0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:44:05.847 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:05.820+0000 7fc3ced9d700 1 --2- 172.21.15.42:0/2238579985 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc3d011a950 0x7fc3d0124810 secure :-1 s=THROTTLE_DONE pgs=1352 cs=0 l=1 rx=0x7fc3d00a6f10 tx=0x7fc3c800c6a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:05.847 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:05.820+0000 7fc3c77fe700 1 -- 172.21.15.42:0/2238579985 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc3c800ca20 con 0x7fc3d011a950 2019-12-05T05:44:05.847 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:05.820+0000 7fc3ced9d700 1 --2- 172.21.15.42:0/2238579985 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc3d011a950 0x7fc3d0124810 secure :-1 s=THROTTLE_DONE pgs=1352 cs=0 l=1 rx=0x7fc3d00a6f10 tx=0x7fc3c800c6a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:05.858 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:05.820+0000 7fc3ced9d700 1 --2- 172.21.15.42:0/2238579985 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc3d011a950 0x7fc3d0124810 secure :-1 s=THROTTLE_DONE pgs=1352 cs=0 l=1 rx=0x7fc3d00a6f10 tx=0x7fc3c800c6a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:05.858 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:05.820+0000 7fc3c77fe700 1 -- 172.21.15.42:0/2238579985 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3c80101c0 con 0x7fc3d011a950 2019-12-05T05:44:05.858 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:05.820+0000 7fc3c77fe700 1 -- 172.21.15.42:0/2238579985 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc3c80124d0 con 0x7fc3d011a950 2019-12-05T05:44:05.858 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:05.820+0000 7fc3d4c36700 1 -- 172.21.15.42:0/2238579985 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3d0124d60 con 0x7fc3d011a950 2019-12-05T05:44:05.858 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:05.820+0000 7fc3d4c36700 1 -- 172.21.15.42:0/2238579985 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3d008f8c0 con 0x7fc3d011a950 2019-12-05T05:44:05.858 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:05.820+0000 7fc3ced9d700 1 --2- 172.21.15.42:0/2238579985 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc3d011a950 0x7fc3d0124810 secure :-1 s=THROTTLE_DONE pgs=1352 cs=0 l=1 rx=0x7fc3d00a6f10 tx=0x7fc3c800c6a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:05.859 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:05.824+0000 7fc3c77fe700 1 -- 172.21.15.42:0/2238579985 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fc3c8013060 con 0x7fc3d011a950 2019-12-05T05:44:05.859 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:05.824+0000 7fc3ced9d700 1 --2- 172.21.15.42:0/2238579985 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc3d011a950 0x7fc3d0124810 secure :-1 s=THROTTLE_DONE pgs=1352 cs=0 l=1 rx=0x7fc3d00a6f10 tx=0x7fc3c800c6a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:05.859 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:05.824+0000 7fc3c77fe700 1 --2- 172.21.15.42:0/2238579985 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc3b403ac00 0x7fc3b403d110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:44:05.859 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:05.824+0000 7fc3c77fe700 1 -- 172.21.15.42:0/2238579985 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 8353+0+0 (secure 0 0 0) 0x7fc3c804afc0 con 0x7fc3d011a950 2019-12-05T05:44:05.859 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:05.824+0000 7fc3ce59c700 1 --2- 172.21.15.42:0/2238579985 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc3b403ac00 0x7fc3b403d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:44:05.859 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:05.824+0000 7fc3ce59c700 1 --2- 172.21.15.42:0/2238579985 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc3b403ac00 0x7fc3b403d110 crc :-1 s=READY pgs=582 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:44:05.859 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:05.828+0000 7fc3d4c36700 1 -- 172.21.15.42:0/2238579985 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3d005d110 con 0x7fc3d011a950 2019-12-05T05:44:05.860 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:05.836+0000 7fc3ced9d700 1 --2- 172.21.15.42:0/2238579985 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc3d011a950 0x7fc3d0124810 secure :-1 s=THROTTLE_DONE pgs=1352 cs=0 l=1 rx=0x7fc3d00a6f10 tx=0x7fc3c800c6a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:05.860 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:05.836+0000 7fc3c77fe700 1 -- 172.21.15.42:0/2238579985 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fc3c8032040 con 0x7fc3d011a950 2019-12-05T05:44:06.000 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:05.992+0000 7fc3d4c36700 1 -- 172.21.15.42:0/2238579985 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc3d005b8c0 con 0x7fc3d011a950 2019-12-05T05:44:06.000 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:05.992+0000 7fc3ced9d700 1 --2- 172.21.15.42:0/2238579985 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc3d011a950 0x7fc3d0124810 secure :-1 s=THROTTLE_DONE pgs=1352 cs=0 l=1 rx=0x7fc3d00a6f10 tx=0x7fc3c800c6a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:06.000 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:05.992+0000 7fc3c77fe700 1 -- 172.21.15.42:0/2238579985 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 164 v164) v1 ==== 95+0+1771 (secure 0 0 0) 0x7fc3c80322a0 con 0x7fc3d011a950 2019-12-05T05:44:06.000 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 164 2019-12-05T05:44:06.000 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:05.992+0000 7fc3d4c36700 1 -- 172.21.15.42:0/2238579985 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc3b403ac00 msgr2=0x7fc3b403d110 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:06.000 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:05.992+0000 7fc3d4c36700 1 --2- 172.21.15.42:0/2238579985 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc3b403ac00 0x7fc3b403d110 crc :-1 s=READY pgs=582 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:44:06.001 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:05.992+0000 7fc3d4c36700 1 -- 172.21.15.42:0/2238579985 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc3d011a950 msgr2=0x7fc3d0124810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:06.001 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:05.992+0000 7fc3d4c36700 1 --2- 172.21.15.42:0/2238579985 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc3d011a950 0x7fc3d0124810 secure :-1 s=READY pgs=1352 cs=0 l=1 rx=0x7fc3d00a6f10 tx=0x7fc3c800c6a0).stop 2019-12-05T05:44:06.001 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:05.996+0000 7fc3d4c36700 1 -- 172.21.15.42:0/2238579985 shutdown_connections 2019-12-05T05:44:06.001 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:05.996+0000 7fc3d4c36700 1 --2- 172.21.15.42:0/2238579985 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc3b403ac00 0x7fc3b403d110 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:44:06.001 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:05.996+0000 7fc3d4c36700 1 --2- 172.21.15.42:0/2238579985 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc3d011c390 0x7fc3d008f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:44:06.001 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:05.996+0000 7fc3d4c36700 1 --2- 172.21.15.42:0/2238579985 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc3d011a950 0x7fc3d0124810 unknown :-1 s=CLOSED pgs=1352 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:44:06.001 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:05.996+0000 7fc3d4c36700 1 -- 172.21.15.42:0/2238579985 shutdown_connections 2019-12-05T05:44:06.002 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:05.996+0000 7fc3d4c36700 1 -- 172.21.15.42:0/2238579985 wait complete. 2019-12-05T05:44:06.002 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:05.996+0000 7fc3d4c36700 1 -- 172.21.15.42:0/2238579985 >> 172.21.15.42:0/2238579985 conn(0x7fc3d0066fe0 msgr2=0x7fc3d009f350 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:44:06.002 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:44:06.002 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":164,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":164,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:44:03.954170+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":76,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7755},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7755":{"gid":7755,"name":"a","rank":0,"incarnation":164,"state":"up:replay","state_seq":2,"addr":"172.21.15.42:6835/358497296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":358497296},{"type":"v1","addr":"172.21.15.42:6835","nonce":358497296}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:44:06.009 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b down, removed from mdsmap 2019-12-05T05:44:06.009 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2019-12-05T05:44:06.009 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:44:06.090 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:06.076+0000 7f8fef8a4700 1 Processor -- start 2019-12-05T05:44:06.090 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:06.076+0000 7f8fef8a4700 1 -- start start 2019-12-05T05:44:06.090 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:06.076+0000 7f8fef8a4700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8fe808df80 0x7f8fe809f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:44:06.090 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:06.076+0000 7f8fef8a4700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8fe809f830 0x7f8fe814efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:44:06.091 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:06.076+0000 7f8fef8a4700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fe809fc60 con 0x7f8fe80a7740 2019-12-05T05:44:06.091 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:06.076+0000 7f8fef8a4700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fe80a0b00 con 0x7f8fe808df80 2019-12-05T05:44:06.091 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:06.076+0000 7f8fef8a4700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fe808e5b0 con 0x7f8fe809f830 2019-12-05T05:44:06.091 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:06.076+0000 7f8fed640700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8fe808df80 0x7f8fe809f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:44:06.091 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:06.076+0000 7f8fee642700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8fe809f830 0x7f8fe814efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:44:06.091 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:06.076+0000 7f8fed640700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8fe808df80 0x7f8fe809f2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.42:55920/0 (socket says 172.21.15.42:55920) 2019-12-05T05:44:06.092 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:06.076+0000 7f8fed640700 1 -- 172.21.15.42:0/2560559627 learned_addr learned my addr 172.21.15.42:0/2560559627 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:44:06.092 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:06.076+0000 7f8fede41700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f8fe80a7740 0x7f8fe807e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:49348/0 (socket says 172.21.15.42:49348) 2019-12-05T05:44:06.092 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:06.076+0000 7f8fed640700 1 -- 172.21.15.42:0/2560559627 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8fe809f830 msgr2=0x7f8fe814efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:44:06.092 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:06.076+0000 7f8fed640700 1 --2- 172.21.15.42:0/2560559627 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8fe809f830 0x7f8fe814efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:44:06.092 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:06.076+0000 7f8fed640700 1 -- 172.21.15.42:0/2560559627 >> v1:172.21.15.40:6789/0 conn(0x7f8fe80a7740 legacy=0x7f8fe807e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:06.092 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:06.076+0000 7f8fed640700 1 -- 172.21.15.42:0/2560559627 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fe80a0710 con 0x7f8fe808df80 2019-12-05T05:44:06.092 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:06.076+0000 7f8fed640700 1 --2- 172.21.15.42:0/2560559627 >> v2:172.21.15.40:3300/0 conn(0x7f8fe808df80 0x7f8fe809f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f8fd8008790 tx=0x7f8fd800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:06.093 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:06.076+0000 7f8fed640700 1 --2- 172.21.15.42:0/2560559627 >> v2:172.21.15.40:3300/0 conn(0x7f8fe808df80 0x7f8fe809f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f8fd8008790 tx=0x7f8fd800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:06.093 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:06.076+0000 7f8fed640700 1 --2- 172.21.15.42:0/2560559627 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8fe808df80 0x7f8fe809f2e0 secure :-1 s=READY pgs=1057 cs=0 l=1 rx=0x7f8fd8008790 tx=0x7f8fd800fd30).ready entity=mon.0 client_cookie=f2c58d8f448f2524 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:44:06.093 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:06.076+0000 7f8fed640700 1 --2- 172.21.15.42:0/2560559627 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8fe808df80 0x7f8fe809f2e0 secure :-1 s=THROTTLE_DONE pgs=1057 cs=0 l=1 rx=0x7f8fd8008790 tx=0x7f8fd800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:06.093 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:06.076+0000 7f8fece3f700 1 -- 172.21.15.42:0/2560559627 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8fd8008c90 con 0x7f8fe808df80 2019-12-05T05:44:06.093 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:06.076+0000 7f8fed640700 1 --2- 172.21.15.42:0/2560559627 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8fe808df80 0x7f8fe809f2e0 secure :-1 s=THROTTLE_DONE pgs=1057 cs=0 l=1 rx=0x7f8fd8008790 tx=0x7f8fd800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:06.093 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:06.076+0000 7f8fed640700 1 --2- 172.21.15.42:0/2560559627 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8fe808df80 0x7f8fe809f2e0 secure :-1 s=THROTTLE_DONE pgs=1057 cs=0 l=1 rx=0x7f8fd8008790 tx=0x7f8fd800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:06.093 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:06.076+0000 7f8fece3f700 1 -- 172.21.15.42:0/2560559627 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fd80088b0 con 0x7f8fe808df80 2019-12-05T05:44:06.093 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:06.076+0000 7f8fece3f700 1 -- 172.21.15.42:0/2560559627 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8fd8011b10 con 0x7f8fe808df80 2019-12-05T05:44:06.094 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:06.076+0000 7f8fef8a4700 1 -- 172.21.15.42:0/2560559627 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8fe808df80 msgr2=0x7f8fe809f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:06.094 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:06.076+0000 7f8fef8a4700 1 --2- 172.21.15.42:0/2560559627 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8fe808df80 0x7f8fe809f2e0 secure :-1 s=READY pgs=1057 cs=0 l=1 rx=0x7f8fd8008790 tx=0x7f8fd800fd30).stop 2019-12-05T05:44:06.094 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:06.076+0000 7f8fef8a4700 1 -- 172.21.15.42:0/2560559627 shutdown_connections 2019-12-05T05:44:06.094 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:06.076+0000 7f8fef8a4700 1 --2- 172.21.15.42:0/2560559627 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8fe809f830 0x7f8fe814efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:44:06.094 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:06.076+0000 7f8fef8a4700 1 --2- 172.21.15.42:0/2560559627 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8fe808df80 0x7f8fe809f2e0 unknown :-1 s=CLOSED pgs=1057 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:44:06.094 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:06.080+0000 7f8fef8a4700 1 -- 172.21.15.42:0/2560559627 shutdown_connections 2019-12-05T05:44:06.094 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:06.080+0000 7f8fef8a4700 1 -- 172.21.15.42:0/2560559627 wait complete. 2019-12-05T05:44:06.095 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:06.080+0000 7f8fef8a4700 1 -- 172.21.15.42:0/2560559627 >> 172.21.15.42:0/2560559627 conn(0x7f8fe80aabe0 msgr2=0x7f8fe80a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:44:06.095 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:06.080+0000 7f8fef8a4700 1 Processor -- start 2019-12-05T05:44:06.095 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:06.080+0000 7f8fef8a4700 1 -- start start 2019-12-05T05:44:06.095 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:06.080+0000 7f8fef8a4700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8fe8145d70 0x7f8fe81436a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:44:06.095 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:06.080+0000 7f8fef8a4700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fe8143bf0 con 0x7f8fe811a920 2019-12-05T05:44:06.095 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:06.080+0000 7f8fef8a4700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fe8130fe0 con 0x7f8fe8142760 2019-12-05T05:44:06.095 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:06.080+0000 7f8fef8a4700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fe8143ed0 con 0x7f8fe8145d70 2019-12-05T05:44:06.096 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:06.080+0000 7f8fee642700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f8fe8142760 0x7f8fe8142b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:39918/0 (socket says 172.21.15.42:39918) 2019-12-05T05:44:06.096 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:06.080+0000 7f8fee642700 1 -- 172.21.15.42:0/913318717 learned_addr learned my addr 172.21.15.42:0/913318717 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:44:06.096 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:06.080+0000 7f8fede41700 1 --2- 172.21.15.42:0/913318717 >> v2:172.21.15.40:3300/0 conn(0x7f8fe8145d70 0x7f8fe81436a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:44:06.096 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:06.080+0000 7f8fde7fc700 1 -- 172.21.15.42:0/913318717 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1006613344 0 0) 0x7f8fe8130fe0 con 0x7f8fe8142760 2019-12-05T05:44:06.096 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:06.080+0000 7f8fde7fc700 1 -- 172.21.15.42:0/913318717 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8fd00030b0 con 0x7f8fe8142760 2019-12-05T05:44:06.096 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:06.080+0000 7f8fde7fc700 1 -- 172.21.15.42:0/913318717 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2541742825 0 0) 0x7f8fd8008470 con 0x7f8fe811a920 2019-12-05T05:44:06.096 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:06.080+0000 7f8fde7fc700 1 -- 172.21.15.42:0/913318717 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8fe8130fe0 con 0x7f8fe811a920 2019-12-05T05:44:06.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:06.080+0000 7f8fde7fc700 1 -- 172.21.15.42:0/913318717 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 2755424676 0 0) 0x7f8fd00030b0 con 0x7f8fe8142760 2019-12-05T05:44:06.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:06.080+0000 7f8fde7fc700 1 -- 172.21.15.42:0/913318717 --> v1:172.21.15.42:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8fd8008470 con 0x7f8fe8142760 2019-12-05T05:44:06.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:06.080+0000 7f8fde7fc700 1 -- 172.21.15.42:0/913318717 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7f8fe0001810 con 0x7f8fe8142760 2019-12-05T05:44:06.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:06.080+0000 7f8fde7fc700 1 -- 172.21.15.42:0/913318717 >> v2:172.21.15.40:3300/0 conn(0x7f8fe8145d70 msgr2=0x7f8fe81436a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:06.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:06.080+0000 7f8fde7fc700 1 --2- 172.21.15.42:0/913318717 >> v2:172.21.15.40:3300/0 conn(0x7f8fe8145d70 0x7f8fe81436a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:44:06.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:06.080+0000 7f8fde7fc700 1 -- 172.21.15.42:0/913318717 >> v1:172.21.15.42:6789/0 conn(0x7f8fe8142760 legacy=0x7f8fe8142b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:06.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:06.080+0000 7f8fde7fc700 1 -- 172.21.15.42:0/913318717 >> v1:172.21.15.40:6789/0 conn(0x7f8fe811a920 legacy=0x7f8fe8076a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:06.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:06.080+0000 7f8fde7fc700 1 --2- 172.21.15.42:0/913318717 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8fd0003ab0 0x7f8fd0004420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:44:06.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:06.080+0000 7f8fde7fc700 1 --2- 172.21.15.42:0/913318717 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8fd0004970 0x7f8fd0004da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:44:06.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:06.080+0000 7f8fde7fc700 1 --2- 172.21.15.42:0/913318717 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8fd0009660 0x7f8fd000baa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:44:06.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:06.080+0000 7f8fde7fc700 1 -- 172.21.15.42:0/913318717 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fe8143ed0 con 0x7f8fd0004970 2019-12-05T05:44:06.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:06.080+0000 7f8fde7fc700 1 -- 172.21.15.42:0/913318717 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fd000bff0 con 0x7f8fd0009660 2019-12-05T05:44:06.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:06.080+0000 7f8fde7fc700 1 -- 172.21.15.42:0/913318717 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fd000c1b0 con 0x7f8fd0003ab0 2019-12-05T05:44:06.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:06.084+0000 7f8fee642700 1 --2- 172.21.15.42:0/913318717 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8fd0009660 0x7f8fd000baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:44:06.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:06.084+0000 7f8fede41700 1 --2- 172.21.15.42:0/913318717 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8fd0003ab0 0x7f8fd0004420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:44:06.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:06.084+0000 7f8fed640700 1 --2- 172.21.15.42:0/913318717 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8fd0004970 0x7f8fd0004da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:44:06.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:06.084+0000 7f8fee642700 1 -- 172.21.15.42:0/913318717 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8fd0003ab0 msgr2=0x7f8fd0004420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:06.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:06.084+0000 7f8fee642700 1 --2- 172.21.15.42:0/913318717 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8fd0003ab0 0x7f8fd0004420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:44:06.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:06.084+0000 7f8fee642700 1 -- 172.21.15.42:0/913318717 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8fd0004970 msgr2=0x7f8fd0004da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:06.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:06.084+0000 7f8fee642700 1 --2- 172.21.15.42:0/913318717 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8fd0004970 0x7f8fd0004da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:44:06.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:06.084+0000 7f8fee642700 1 -- 172.21.15.42:0/913318717 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fe8132220 con 0x7f8fd0009660 2019-12-05T05:44:06.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:06.084+0000 7f8fef8a4700 1 -- 172.21.15.42:0/913318717 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fe812ffd0 con 0x7f8fd0009660 2019-12-05T05:44:06.100 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:06.084+0000 7f8fee642700 1 --2- 172.21.15.42:0/913318717 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8fd0009660 0x7f8fd000baa0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f8fe0008100 tx=0x7f8fe000d2a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:06.100 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:06.084+0000 7f8fee642700 1 -- 172.21.15.42:0/913318717 reap_dead start 2019-12-05T05:44:06.100 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:06.084+0000 7f8fef8a4700 1 -- 172.21.15.42:0/913318717 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fe8145d70 con 0x7f8fd0009660 2019-12-05T05:44:06.100 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:06.084+0000 7f8fee642700 1 --2- 172.21.15.42:0/913318717 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8fd0009660 0x7f8fd000baa0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f8fe0008100 tx=0x7f8fe000d2a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:06.100 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:06.084+0000 7f8fee642700 1 --2- 172.21.15.42:0/913318717 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8fd0009660 0x7f8fd000baa0 secure :-1 s=READY pgs=1354 cs=0 l=1 rx=0x7f8fe0008100 tx=0x7f8fe000d2a0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:44:06.100 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:06.084+0000 7f8fee642700 1 --2- 172.21.15.42:0/913318717 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8fd0009660 0x7f8fd000baa0 secure :-1 s=THROTTLE_DONE pgs=1354 cs=0 l=1 rx=0x7f8fe0008100 tx=0x7f8fe000d2a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:06.100 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:06.084+0000 7f8fee642700 1 --2- 172.21.15.42:0/913318717 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8fd0009660 0x7f8fd000baa0 secure :-1 s=THROTTLE_DONE pgs=1354 cs=0 l=1 rx=0x7f8fe0008100 tx=0x7f8fe000d2a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:06.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:06.084+0000 7f8fde7fc700 1 -- 172.21.15.42:0/913318717 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8fe0007160 con 0x7f8fd0009660 2019-12-05T05:44:06.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:06.084+0000 7f8fee642700 1 --2- 172.21.15.42:0/913318717 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8fd0009660 0x7f8fd000baa0 secure :-1 s=THROTTLE_DONE pgs=1354 cs=0 l=1 rx=0x7f8fe0008100 tx=0x7f8fe000d2a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:06.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:06.084+0000 7f8fde7fc700 1 -- 172.21.15.42:0/913318717 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fe0005ad0 con 0x7f8fd0009660 2019-12-05T05:44:06.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:06.084+0000 7f8fde7fc700 1 -- 172.21.15.42:0/913318717 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8fe000f410 con 0x7f8fd0009660 2019-12-05T05:44:06.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:06.084+0000 7f8fee642700 1 --2- 172.21.15.42:0/913318717 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8fd0009660 0x7f8fd000baa0 secure :-1 s=THROTTLE_DONE pgs=1354 cs=0 l=1 rx=0x7f8fe0008100 tx=0x7f8fe000d2a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:06.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:06.084+0000 7f8fde7fc700 1 -- 172.21.15.42:0/913318717 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f8fe000f610 con 0x7f8fd0009660 2019-12-05T05:44:06.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:06.084+0000 7f8fee642700 1 --2- 172.21.15.42:0/913318717 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8fd0009660 0x7f8fd000baa0 secure :-1 s=THROTTLE_DONE pgs=1354 cs=0 l=1 rx=0x7f8fe0008100 tx=0x7f8fe000d2a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:06.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:06.084+0000 7f8fde7fc700 1 --2- 172.21.15.42:0/913318717 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8fd0042fb0 0x7f8fd00454c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:44:06.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:06.084+0000 7f8fde7fc700 1 -- 172.21.15.42:0/913318717 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 8353+0+0 (secure 0 0 0) 0x7f8fe004bbd0 con 0x7f8fd0009660 2019-12-05T05:44:06.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:06.084+0000 7f8fede41700 1 --2- 172.21.15.42:0/913318717 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8fd0042fb0 0x7f8fd00454c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:44:06.103 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:06.084+0000 7f8fede41700 1 --2- 172.21.15.42:0/913318717 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8fd0042fb0 0x7f8fd00454c0 crc :-1 s=READY pgs=583 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:44:06.103 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:06.092+0000 7f8fef8a4700 1 -- 172.21.15.42:0/913318717 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fc4003200 con 0x7f8fd0009660 2019-12-05T05:44:06.103 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:06.096+0000 7f8fee642700 1 --2- 172.21.15.42:0/913318717 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8fd0009660 0x7f8fd000baa0 secure :-1 s=THROTTLE_DONE pgs=1354 cs=0 l=1 rx=0x7f8fe0008100 tx=0x7f8fe000d2a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:06.103 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:06.096+0000 7f8fde7fc700 1 -- 172.21.15.42:0/913318717 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f8fe0032c50 con 0x7f8fd0009660 2019-12-05T05:44:06.267 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:06.264+0000 7f8fef8a4700 1 -- 172.21.15.42:0/913318717 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8fc4003630 con 0x7f8fd0009660 2019-12-05T05:44:06.276 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:06.264+0000 7f8fee642700 1 --2- 172.21.15.42:0/913318717 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8fd0009660 0x7f8fd000baa0 secure :-1 s=THROTTLE_DONE pgs=1354 cs=0 l=1 rx=0x7f8fe0008100 tx=0x7f8fe000d2a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:06.276 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:06.264+0000 7f8fde7fc700 1 -- 172.21.15.42:0/913318717 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 164 v164) v1 ==== 95+0+1771 (secure 0 0 0) 0x7f8fe0032eb0 con 0x7f8fd0009660 2019-12-05T05:44:06.276 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 164 2019-12-05T05:44:06.277 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:06.264+0000 7f8fef8a4700 1 -- 172.21.15.42:0/913318717 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8fd0042fb0 msgr2=0x7f8fd00454c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:06.277 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:06.264+0000 7f8fef8a4700 1 --2- 172.21.15.42:0/913318717 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8fd0042fb0 0x7f8fd00454c0 crc :-1 s=READY pgs=583 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:44:06.277 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:06.264+0000 7f8fef8a4700 1 -- 172.21.15.42:0/913318717 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8fd0009660 msgr2=0x7f8fd000baa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:06.277 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:06.264+0000 7f8fef8a4700 1 --2- 172.21.15.42:0/913318717 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8fd0009660 0x7f8fd000baa0 secure :-1 s=READY pgs=1354 cs=0 l=1 rx=0x7f8fe0008100 tx=0x7f8fe000d2a0).stop 2019-12-05T05:44:06.277 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:06.264+0000 7f8fef8a4700 1 -- 172.21.15.42:0/913318717 shutdown_connections 2019-12-05T05:44:06.277 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:06.264+0000 7f8fef8a4700 1 --2- 172.21.15.42:0/913318717 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8fd0042fb0 0x7f8fd00454c0 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:44:06.278 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:06.264+0000 7f8fef8a4700 1 --2- 172.21.15.42:0/913318717 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8fd0009660 0x7f8fd000baa0 unknown :-1 s=CLOSED pgs=1354 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:44:06.278 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:06.264+0000 7f8fef8a4700 1 -- 172.21.15.42:0/913318717 shutdown_connections 2019-12-05T05:44:06.278 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:06.264+0000 7f8fef8a4700 1 -- 172.21.15.42:0/913318717 wait complete. 2019-12-05T05:44:06.278 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:06.264+0000 7f8fef8a4700 1 -- 172.21.15.42:0/913318717 >> 172.21.15.42:0/913318717 conn(0x7f8fe808fde0 msgr2=0x7f8fe80a97e0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:44:06.278 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:44:06.278 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":164,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":164,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:44:03.954170+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":76,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7755},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7755":{"gid":7755,"name":"a","rank":0,"incarnation":164,"state":"up:replay","state_seq":2,"addr":"172.21.15.42:6835/358497296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":358497296},{"type":"v1","addr":"172.21.15.42:6835","nonce":358497296}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:44:06.280 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 164, "feature_flags": { "enable_multiple": false, "ever_enabled_multiple": false }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2019-12-05T05:03:35.604738+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 164, "ever_allowed_features": 2, "explicitly_allowed_features": 2, "failed": [], "flags": 18, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_7755": { "addr": "172.21.15.42:6835/358497296", "addrs": { "addrvec": [ { "addr": "172.21.15.42:6834", "nonce": 358497296, "type": "v2" }, { "addr": "172.21.15.42:6835", "nonce": 358497296, "type": "v1" } ] }, "export_targets": [], "features": 4540138292836696063, "flags": 0, "gid": 7755, "incarnation": 164, "name": "a", "rank": 0, "state": "up:replay", "state_seq": 2 } }, "last_failure": 0, "last_failure_osd_epoch": 76, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 2, "min_compat_client": "0 (unknown)", "modified": "2019-12-05T05:44:03.954170+0000", "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7755 } } } ], "standbys": [] } 2019-12-05T05:44:08.282 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:44:08.387 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:08.376+0000 7f5974c53700 1 Processor -- start 2019-12-05T05:44:08.387 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:08.376+0000 7f5974c53700 1 -- start start 2019-12-05T05:44:08.387 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:08.376+0000 7f5974c53700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f59700a7740 0x7f597008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:44:08.387 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:08.376+0000 7f5974c53700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f597009fae0 0x7f597007eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:44:08.387 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:08.376+0000 7f5974c53700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59700a0710 con 0x7f597007e720 2019-12-05T05:44:08.387 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:08.376+0000 7f5974c53700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59700a0b00 con 0x7f59700a7740 2019-12-05T05:44:08.388 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:08.376+0000 7f5974c53700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f597008e5b0 con 0x7f597009fae0 2019-12-05T05:44:08.388 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:08.376+0000 7f596e59c700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f597007e720 0x7f597009f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:39936/0 (socket says 172.21.15.42:39936) 2019-12-05T05:44:08.388 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:08.376+0000 7f596e59c700 1 -- 172.21.15.42:0/904966464 learned_addr learned my addr 172.21.15.42:0/904966464 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:44:08.388 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:08.376+0000 7f596f59e700 1 --2- 172.21.15.42:0/904966464 >> v2:172.21.15.42:3300/0 conn(0x7f597009fae0 0x7f597007eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:44:08.388 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:08.376+0000 7f596ed9d700 1 --2- 172.21.15.42:0/904966464 >> v2:172.21.15.40:3300/0 conn(0x7f59700a7740 0x7f597008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:44:08.388 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:08.376+0000 7f596dd9b700 1 -- 172.21.15.42:0/904966464 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1281144610 0 0) 0x7f59700a0710 con 0x7f597007e720 2019-12-05T05:44:08.388 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:08.376+0000 7f596dd9b700 1 -- 172.21.15.42:0/904966464 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f595c0030b0 con 0x7f597007e720 2019-12-05T05:44:08.389 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:08.376+0000 7f596f59e700 1 -- 172.21.15.42:0/904966464 >> v2:172.21.15.40:3300/0 conn(0x7f59700a7740 msgr2=0x7f597008df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:44:08.389 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:08.376+0000 7f596f59e700 1 --2- 172.21.15.42:0/904966464 >> v2:172.21.15.40:3300/0 conn(0x7f59700a7740 0x7f597008df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:44:08.389 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:08.376+0000 7f596f59e700 1 -- 172.21.15.42:0/904966464 >> v1:172.21.15.42:6789/0 conn(0x7f597007e720 legacy=0x7f597009f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:08.389 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:08.376+0000 7f596f59e700 1 -- 172.21.15.42:0/904966464 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f597007f0a0 con 0x7f597009fae0 2019-12-05T05:44:08.389 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:08.376+0000 7f596ed9d700 1 --2- 172.21.15.42:0/904966464 >> v2:172.21.15.40:3300/0 conn(0x7f59700a7740 0x7f597008df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:44:08.389 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:08.376+0000 7f596f59e700 1 --2- 172.21.15.42:0/904966464 >> v2:172.21.15.42:3300/0 conn(0x7f597009fae0 0x7f597007eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f5960002f30 tx=0x7f596000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:08.389 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:08.376+0000 7f596f59e700 1 --2- 172.21.15.42:0/904966464 >> v2:172.21.15.42:3300/0 conn(0x7f597009fae0 0x7f597007eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f5960002f30 tx=0x7f596000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:08.390 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:08.376+0000 7f596f59e700 1 --2- 172.21.15.42:0/904966464 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f597009fae0 0x7f597007eb50 secure :-1 s=READY pgs=1356 cs=0 l=1 rx=0x7f5960002f30 tx=0x7f596000fc70).ready entity=mon.1 client_cookie=b00af7367faf1b05 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:44:08.390 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:08.376+0000 7f596f59e700 1 --2- 172.21.15.42:0/904966464 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f597009fae0 0x7f597007eb50 secure :-1 s=THROTTLE_DONE pgs=1356 cs=0 l=1 rx=0x7f5960002f30 tx=0x7f596000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:08.390 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:08.376+0000 7f596dd9b700 1 -- 172.21.15.42:0/904966464 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5960008bb0 con 0x7f597009fae0 2019-12-05T05:44:08.390 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:08.376+0000 7f596f59e700 1 --2- 172.21.15.42:0/904966464 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f597009fae0 0x7f597007eb50 secure :-1 s=THROTTLE_DONE pgs=1356 cs=0 l=1 rx=0x7f5960002f30 tx=0x7f596000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:08.390 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:08.376+0000 7f596f59e700 1 --2- 172.21.15.42:0/904966464 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f597009fae0 0x7f597007eb50 secure :-1 s=THROTTLE_DONE pgs=1356 cs=0 l=1 rx=0x7f5960002f30 tx=0x7f596000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:08.390 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:08.380+0000 7f596dd9b700 1 -- 172.21.15.42:0/904966464 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59600087d0 con 0x7f597009fae0 2019-12-05T05:44:08.390 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:08.380+0000 7f596dd9b700 1 -- 172.21.15.42:0/904966464 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5960011820 con 0x7f597009fae0 2019-12-05T05:44:08.391 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:08.380+0000 7f5974c53700 1 -- 172.21.15.42:0/904966464 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f597009fae0 msgr2=0x7f597007eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:08.391 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:08.380+0000 7f5974c53700 1 --2- 172.21.15.42:0/904966464 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f597009fae0 0x7f597007eb50 secure :-1 s=READY pgs=1356 cs=0 l=1 rx=0x7f5960002f30 tx=0x7f596000fc70).stop 2019-12-05T05:44:08.391 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:08.380+0000 7f5974c53700 1 -- 172.21.15.42:0/904966464 shutdown_connections 2019-12-05T05:44:08.391 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:08.380+0000 7f5974c53700 1 --2- 172.21.15.42:0/904966464 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f597009fae0 0x7f597007eb50 unknown :-1 s=CLOSED pgs=1356 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:44:08.391 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:08.380+0000 7f5974c53700 1 --2- 172.21.15.42:0/904966464 >> v2:172.21.15.40:3300/0 conn(0x7f59700a7740 0x7f597008df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:44:08.391 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:08.380+0000 7f5974c53700 1 -- 172.21.15.42:0/904966464 shutdown_connections 2019-12-05T05:44:08.391 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:08.380+0000 7f5974c53700 1 -- 172.21.15.42:0/904966464 wait complete. 2019-12-05T05:44:08.392 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:08.380+0000 7f5974c53700 1 -- 172.21.15.42:0/904966464 >> 172.21.15.42:0/904966464 conn(0x7f59700aabe0 msgr2=0x7f59700a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:44:08.392 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:08.380+0000 7f5974c53700 1 Processor -- start 2019-12-05T05:44:08.392 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:08.380+0000 7f5974c53700 1 -- start start 2019-12-05T05:44:08.397 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:08.384+0000 7f5974c53700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f59701226c0 0x7f59700a49d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:44:08.397 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:08.384+0000 7f5974c53700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59700a4f20 con 0x7f5970116600 2019-12-05T05:44:08.397 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:08.384+0000 7f5974c53700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5970123d40 con 0x7f59700732f0 2019-12-05T05:44:08.397 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:08.384+0000 7f5974c53700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5970122180 con 0x7f59701226c0 2019-12-05T05:44:08.397 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:08.384+0000 7f596f59e700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f59700732f0 0x7f5970073720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:39944/0 (socket says 172.21.15.42:39944) 2019-12-05T05:44:08.397 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:08.384+0000 7f596f59e700 1 -- 172.21.15.42:0/60012362 learned_addr learned my addr 172.21.15.42:0/60012362 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:44:08.398 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:08.384+0000 7f596e59c700 1 --2- 172.21.15.42:0/60012362 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f59701226c0 0x7f59700a49d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:44:08.398 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:08.384+0000 7f59677fe700 1 -- 172.21.15.42:0/60012362 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2628316354 0 0) 0x7f5970123d40 con 0x7f59700732f0 2019-12-05T05:44:08.398 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:08.384+0000 7f59677fe700 1 -- 172.21.15.42:0/60012362 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59540030b0 con 0x7f59700732f0 2019-12-05T05:44:08.398 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:08.384+0000 7f59677fe700 1 -- 172.21.15.42:0/60012362 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 292998932 0 0) 0x7f59700a4f20 con 0x7f5970116600 2019-12-05T05:44:08.398 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:08.384+0000 7f59677fe700 1 -- 172.21.15.42:0/60012362 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5970123d40 con 0x7f5970116600 2019-12-05T05:44:08.398 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:08.384+0000 7f596e59c700 1 -- 172.21.15.42:0/60012362 >> v1:172.21.15.42:6789/0 conn(0x7f59700732f0 legacy=0x7f5970073720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:08.398 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:08.384+0000 7f596e59c700 1 -- 172.21.15.42:0/60012362 >> v1:172.21.15.40:6789/0 conn(0x7f5970116600 legacy=0x7f597011bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:08.399 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:08.384+0000 7f596e59c700 1 -- 172.21.15.42:0/60012362 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5970073f20 con 0x7f59701226c0 2019-12-05T05:44:08.399 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:08.384+0000 7f596e59c700 1 --2- 172.21.15.42:0/60012362 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f59701226c0 0x7f59700a49d0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f5958004720 tx=0x7f5958010be0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:08.399 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:08.384+0000 7f596e59c700 1 --2- 172.21.15.42:0/60012362 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f59701226c0 0x7f59700a49d0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f5958004720 tx=0x7f5958010be0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:08.399 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:08.384+0000 7f596e59c700 1 --2- 172.21.15.42:0/60012362 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f59701226c0 0x7f59700a49d0 secure :-1 s=READY pgs=287 cs=0 l=1 rx=0x7f5958004720 tx=0x7f5958010be0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:44:08.399 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:08.384+0000 7f596e59c700 1 --2- 172.21.15.42:0/60012362 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f59701226c0 0x7f59700a49d0 secure :-1 s=THROTTLE_DONE pgs=287 cs=0 l=1 rx=0x7f5958004720 tx=0x7f5958010be0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:08.399 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:08.384+0000 7f59677fe700 1 -- 172.21.15.42:0/60012362 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f59580122a0 con 0x7f59701226c0 2019-12-05T05:44:08.399 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:08.384+0000 7f596e59c700 1 --2- 172.21.15.42:0/60012362 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f59701226c0 0x7f59700a49d0 secure :-1 s=THROTTLE_DONE pgs=287 cs=0 l=1 rx=0x7f5958004720 tx=0x7f5958010be0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:08.400 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:08.384+0000 7f5974c53700 1 -- 172.21.15.42:0/60012362 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f597007b570 con 0x7f59701226c0 2019-12-05T05:44:08.400 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:08.384+0000 7f59677fe700 1 -- 172.21.15.42:0/60012362 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5958005bd0 con 0x7f59701226c0 2019-12-05T05:44:08.400 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:08.384+0000 7f596e59c700 1 --2- 172.21.15.42:0/60012362 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f59701226c0 0x7f59700a49d0 secure :-1 s=THROTTLE_DONE pgs=287 cs=0 l=1 rx=0x7f5958004720 tx=0x7f5958010be0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:08.400 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:08.384+0000 7f59677fe700 1 -- 172.21.15.42:0/60012362 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5958012880 con 0x7f59701226c0 2019-12-05T05:44:08.400 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:08.384+0000 7f5974c53700 1 -- 172.21.15.42:0/60012362 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f597007baf0 con 0x7f59701226c0 2019-12-05T05:44:08.400 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:08.384+0000 7f596e59c700 1 --2- 172.21.15.42:0/60012362 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f59701226c0 0x7f59700a49d0 secure :-1 s=THROTTLE_DONE pgs=287 cs=0 l=1 rx=0x7f5958004720 tx=0x7f5958010be0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:08.400 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:08.384+0000 7f59677fe700 1 -- 172.21.15.42:0/60012362 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f5958012a80 con 0x7f59701226c0 2019-12-05T05:44:08.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:08.384+0000 7f596e59c700 1 --2- 172.21.15.42:0/60012362 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f59701226c0 0x7f59700a49d0 secure :-1 s=THROTTLE_DONE pgs=287 cs=0 l=1 rx=0x7f5958004720 tx=0x7f5958010be0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:08.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:08.388+0000 7f59677fe700 1 --2- 172.21.15.42:0/60012362 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f595403b400 0x7f595403d910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:44:08.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:08.388+0000 7f59677fe700 1 -- 172.21.15.42:0/60012362 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 8353+0+0 (secure 0 0 0) 0x7f595804b480 con 0x7f59701226c0 2019-12-05T05:44:08.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:08.388+0000 7f596ed9d700 1 --2- 172.21.15.42:0/60012362 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f595403b400 0x7f595403d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:44:08.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:08.388+0000 7f596ed9d700 1 --2- 172.21.15.42:0/60012362 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f595403b400 0x7f595403d910 crc :-1 s=READY pgs=584 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:44:08.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:08.388+0000 7f5974c53700 1 -- 172.21.15.42:0/60012362 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f597011c360 con 0x7f59701226c0 2019-12-05T05:44:08.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:08.400+0000 7f596e59c700 1 --2- 172.21.15.42:0/60012362 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f59701226c0 0x7f59700a49d0 secure :-1 s=THROTTLE_DONE pgs=287 cs=0 l=1 rx=0x7f5958004720 tx=0x7f5958010be0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:08.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:08.400+0000 7f59677fe700 1 -- 172.21.15.42:0/60012362 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f59580324e0 con 0x7f59701226c0 2019-12-05T05:44:08.575 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:08.568+0000 7f5974c53700 1 -- 172.21.15.42:0/60012362 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f597005d110 con 0x7f59701226c0 2019-12-05T05:44:08.576 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:08.572+0000 7f596e59c700 1 --2- 172.21.15.42:0/60012362 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f59701226c0 0x7f59700a49d0 secure :-1 s=THROTTLE_DONE pgs=287 cs=0 l=1 rx=0x7f5958004720 tx=0x7f5958010be0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:08.576 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:08.572+0000 7f59677fe700 1 -- 172.21.15.42:0/60012362 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 166 v166) v1 ==== 95+0+1772 (secure 0 0 0) 0x7f5958032740 con 0x7f59701226c0 2019-12-05T05:44:08.578 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:44:08.578 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":166,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":166,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:44:08.060201+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":76,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7755},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7755":{"gid":7755,"name":"a","rank":0,"incarnation":164,"state":"up:rejoin","state_seq":21,"addr":"172.21.15.42:6835/358497296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":358497296},{"type":"v1","addr":"172.21.15.42:6835","nonce":358497296}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:44:08.579 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 166 2019-12-05T05:44:08.581 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:08.576+0000 7f5974c53700 1 -- 172.21.15.42:0/60012362 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f595403b400 msgr2=0x7f595403d910 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:08.581 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:08.576+0000 7f5974c53700 1 --2- 172.21.15.42:0/60012362 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f595403b400 0x7f595403d910 crc :-1 s=READY pgs=584 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:44:08.581 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:08.576+0000 7f5974c53700 1 -- 172.21.15.42:0/60012362 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f59701226c0 msgr2=0x7f59700a49d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:08.581 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:08.576+0000 7f5974c53700 1 --2- 172.21.15.42:0/60012362 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f59701226c0 0x7f59700a49d0 secure :-1 s=READY pgs=287 cs=0 l=1 rx=0x7f5958004720 tx=0x7f5958010be0).stop 2019-12-05T05:44:08.582 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:08.576+0000 7f5974c53700 1 -- 172.21.15.42:0/60012362 shutdown_connections 2019-12-05T05:44:08.582 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:08.576+0000 7f5974c53700 1 --2- 172.21.15.42:0/60012362 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f595403b400 0x7f595403d910 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:44:08.582 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:08.576+0000 7f5974c53700 1 --2- 172.21.15.42:0/60012362 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f59701226c0 0x7f59700a49d0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:44:08.582 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:08.576+0000 7f5974c53700 1 -- 172.21.15.42:0/60012362 shutdown_connections 2019-12-05T05:44:08.582 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:08.576+0000 7f5974c53700 1 -- 172.21.15.42:0/60012362 wait complete. 2019-12-05T05:44:08.583 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:08.576+0000 7f5974c53700 1 -- 172.21.15.42:0/60012362 >> 172.21.15.42:0/60012362 conn(0x7f59700a82a0 msgr2=0x7f59700a9790 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:44:08.591 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2019-12-05T05:44:10.592 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:44:10.702 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:10.692+0000 7fab2a95e700 1 Processor -- start 2019-12-05T05:44:10.703 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:10.692+0000 7fab2a95e700 1 -- start start 2019-12-05T05:44:10.703 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:10.692+0000 7fab2a95e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fab240a7740 0x7fab2408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:44:10.703 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:10.692+0000 7fab2a95e700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fab2407e720 0x7fab2407eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:44:10.703 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:10.692+0000 7fab2a95e700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab240a0710 con 0x7fab2409f2e0 2019-12-05T05:44:10.703 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:10.692+0000 7fab2a95e700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab240a0b00 con 0x7fab2407e720 2019-12-05T05:44:10.704 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:10.692+0000 7fab2a95e700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab2408e5b0 con 0x7fab240a7740 2019-12-05T05:44:10.704 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:10.692+0000 7fab28efb700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fab240a7740 0x7fab2408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:44:10.704 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:10.692+0000 7fab28efb700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fab240a7740 0x7fab2408df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:39916/0 (socket says 172.21.15.42:39916) 2019-12-05T05:44:10.704 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:10.692+0000 7fab28efb700 1 -- 172.21.15.42:0/119221862 learned_addr learned my addr 172.21.15.42:0/119221862 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:44:10.704 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:10.692+0000 7fab23fff700 1 --2- 172.21.15.42:0/119221862 >> v2:172.21.15.40:3300/0 conn(0x7fab2407e720 0x7fab2407eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:44:10.704 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:10.696+0000 7fab28efb700 1 -- 172.21.15.42:0/119221862 >> v2:172.21.15.40:3300/0 conn(0x7fab2407e720 msgr2=0x7fab2407eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:44:10.705 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:10.696+0000 7fab28efb700 1 --2- 172.21.15.42:0/119221862 >> v2:172.21.15.40:3300/0 conn(0x7fab2407e720 0x7fab2407eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:44:10.705 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:10.696+0000 7fab28efb700 1 -- 172.21.15.42:0/119221862 >> v1:172.21.15.40:6789/0 conn(0x7fab2409f2e0 legacy=0x7fab2409f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:44:10.705 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:10.696+0000 7fab28efb700 1 -- 172.21.15.42:0/119221862 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab2407f0a0 con 0x7fab240a7740 2019-12-05T05:44:10.705 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:10.696+0000 7fab28efb700 1 --2- 172.21.15.42:0/119221862 >> v2:172.21.15.42:3300/0 conn(0x7fab240a7740 0x7fab2408df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fab14002f30 tx=0x7fab1400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:10.705 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:10.696+0000 7fab28efb700 1 --2- 172.21.15.42:0/119221862 >> v2:172.21.15.42:3300/0 conn(0x7fab240a7740 0x7fab2408df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fab14002f30 tx=0x7fab1400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:10.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:10.696+0000 7fab28efb700 1 --2- 172.21.15.42:0/119221862 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fab240a7740 0x7fab2408df80 secure :-1 s=READY pgs=1358 cs=0 l=1 rx=0x7fab14002f30 tx=0x7fab1400fd30).ready entity=mon.1 client_cookie=7bfde0a5c159a96a server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:44:10.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:10.696+0000 7fab28efb700 1 --2- 172.21.15.42:0/119221862 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fab240a7740 0x7fab2408df80 secure :-1 s=THROTTLE_DONE pgs=1358 cs=0 l=1 rx=0x7fab14002f30 tx=0x7fab1400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:10.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:10.696+0000 7fab28efb700 1 --2- 172.21.15.42:0/119221862 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fab240a7740 0x7fab2408df80 secure :-1 s=THROTTLE_DONE pgs=1358 cs=0 l=1 rx=0x7fab14002f30 tx=0x7fab1400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:10.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:10.696+0000 7fab237fe700 1 -- 172.21.15.42:0/119221862 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fab14011370 con 0x7fab240a7740 2019-12-05T05:44:10.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:10.696+0000 7fab28efb700 1 --2- 172.21.15.42:0/119221862 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fab240a7740 0x7fab2408df80 secure :-1 s=THROTTLE_DONE pgs=1358 cs=0 l=1 rx=0x7fab14002f30 tx=0x7fab1400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:10.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:10.696+0000 7fab237fe700 1 -- 172.21.15.42:0/119221862 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab14008bb0 con 0x7fab240a7740 2019-12-05T05:44:10.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:10.696+0000 7fab237fe700 1 -- 172.21.15.42:0/119221862 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fab14011950 con 0x7fab240a7740 2019-12-05T05:44:10.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:10.696+0000 7fab2a95e700 1 -- 172.21.15.42:0/119221862 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fab240a7740 msgr2=0x7fab2408df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:10.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:10.696+0000 7fab2a95e700 1 --2- 172.21.15.42:0/119221862 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fab240a7740 0x7fab2408df80 secure :-1 s=READY pgs=1358 cs=0 l=1 rx=0x7fab14002f30 tx=0x7fab1400fd30).stop 2019-12-05T05:44:10.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:10.696+0000 7fab2a95e700 1 -- 172.21.15.42:0/119221862 shutdown_connections 2019-12-05T05:44:10.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:10.696+0000 7fab2a95e700 1 --2- 172.21.15.42:0/119221862 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fab240a7740 0x7fab2408df80 unknown :-1 s=CLOSED pgs=1358 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:44:10.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:10.696+0000 7fab2a95e700 1 --2- 172.21.15.42:0/119221862 >> v2:172.21.15.40:3300/0 conn(0x7fab2407e720 0x7fab2407eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:44:10.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:10.696+0000 7fab2a95e700 1 -- 172.21.15.42:0/119221862 shutdown_connections 2019-12-05T05:44:10.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:10.696+0000 7fab2a95e700 1 -- 172.21.15.42:0/119221862 wait complete. 2019-12-05T05:44:10.708 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:10.696+0000 7fab2a95e700 1 -- 172.21.15.42:0/119221862 >> 172.21.15.42:0/119221862 conn(0x7fab240aabe0 msgr2=0x7fab240a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:44:10.711 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:10.700+0000 7fab2a95e700 1 Processor -- start 2019-12-05T05:44:10.712 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:10.700+0000 7fab2a95e700 1 -- start start 2019-12-05T05:44:10.712 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:10.700+0000 7fab2a95e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fab2411af70 0x7fab2412e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:44:10.712 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:10.700+0000 7fab2a95e700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab2412f3b0 con 0x7fab241253d0 2019-12-05T05:44:10.712 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:10.700+0000 7fab2a95e700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab2412b770 con 0x7fab2411cb20 2019-12-05T05:44:10.712 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:10.700+0000 7fab2a95e700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab2412ebd0 con 0x7fab2411af70 2019-12-05T05:44:10.713 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:10.700+0000 7fab23fff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fab2411af70 0x7fab2412e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:44:10.713 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:10.700+0000 7fab23fff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fab2411af70 0x7fab2412e530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:39924/0 (socket says 172.21.15.42:39924) 2019-12-05T05:44:10.713 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:10.700+0000 7fab23fff700 1 -- 172.21.15.42:0/2428590963 learned_addr learned my addr 172.21.15.42:0/2428590963 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:44:10.713 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:10.700+0000 7fab23fff700 1 -- 172.21.15.42:0/2428590963 >> v1:172.21.15.42:6789/0 conn(0x7fab2411cb20 legacy=0x7fab24135980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:10.713 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:10.700+0000 7fab23fff700 1 -- 172.21.15.42:0/2428590963 >> v1:172.21.15.40:6789/0 conn(0x7fab241253d0 legacy=0x7fab2412af70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:10.713 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:10.700+0000 7fab23fff700 1 -- 172.21.15.42:0/2428590963 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab2411cf50 con 0x7fab2411af70 2019-12-05T05:44:10.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:10.700+0000 7fab23fff700 1 --2- 172.21.15.42:0/2428590963 >> v2:172.21.15.42:3300/0 conn(0x7fab2411af70 0x7fab2412e530 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fab18003070 tx=0x7fab18010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:10.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:10.700+0000 7fab23fff700 1 --2- 172.21.15.42:0/2428590963 >> v2:172.21.15.42:3300/0 conn(0x7fab2411af70 0x7fab2412e530 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fab18003070 tx=0x7fab18010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:10.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:10.700+0000 7fab23fff700 1 --2- 172.21.15.42:0/2428590963 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fab2411af70 0x7fab2412e530 secure :-1 s=READY pgs=1360 cs=0 l=1 rx=0x7fab18003070 tx=0x7fab18010b40).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:44:10.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:10.700+0000 7fab23fff700 1 --2- 172.21.15.42:0/2428590963 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fab2411af70 0x7fab2412e530 secure :-1 s=THROTTLE_DONE pgs=1360 cs=0 l=1 rx=0x7fab18003070 tx=0x7fab18010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:10.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:10.700+0000 7fab23fff700 1 --2- 172.21.15.42:0/2428590963 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fab2411af70 0x7fab2412e530 secure :-1 s=THROTTLE_DONE pgs=1360 cs=0 l=1 rx=0x7fab18003070 tx=0x7fab18010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:10.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:10.700+0000 7fab217fa700 1 -- 172.21.15.42:0/2428590963 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fab18007350 con 0x7fab2411af70 2019-12-05T05:44:10.715 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:10.700+0000 7fab23fff700 1 --2- 172.21.15.42:0/2428590963 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fab2411af70 0x7fab2412e530 secure :-1 s=THROTTLE_DONE pgs=1360 cs=0 l=1 rx=0x7fab18003070 tx=0x7fab18010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:10.715 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:10.700+0000 7fab217fa700 1 -- 172.21.15.42:0/2428590963 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab18013060 con 0x7fab2411af70 2019-12-05T05:44:10.715 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:10.700+0000 7fab2a95e700 1 -- 172.21.15.42:0/2428590963 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab24137190 con 0x7fab2411af70 2019-12-05T05:44:10.715 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:10.700+0000 7fab217fa700 1 -- 172.21.15.42:0/2428590963 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fab18005d80 con 0x7fab2411af70 2019-12-05T05:44:10.715 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:10.700+0000 7fab2a95e700 1 -- 172.21.15.42:0/2428590963 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab24137740 con 0x7fab2411af70 2019-12-05T05:44:10.716 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:10.704+0000 7fab23fff700 1 --2- 172.21.15.42:0/2428590963 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fab2411af70 0x7fab2412e530 secure :-1 s=THROTTLE_DONE pgs=1360 cs=0 l=1 rx=0x7fab18003070 tx=0x7fab18010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:10.716 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:10.704+0000 7fab217fa700 1 -- 172.21.15.42:0/2428590963 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fab18012250 con 0x7fab2411af70 2019-12-05T05:44:10.716 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:10.704+0000 7fab23fff700 1 --2- 172.21.15.42:0/2428590963 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fab2411af70 0x7fab2412e530 secure :-1 s=THROTTLE_DONE pgs=1360 cs=0 l=1 rx=0x7fab18003070 tx=0x7fab18010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:10.716 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:10.704+0000 7fab217fa700 1 --2- 172.21.15.42:0/2428590963 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fab0403ad30 0x7fab0403d240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:44:10.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:10.704+0000 7fab217fa700 1 -- 172.21.15.42:0/2428590963 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 8353+0+0 (secure 0 0 0) 0x7fab1804af70 con 0x7fab2411af70 2019-12-05T05:44:10.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:10.704+0000 7fab28efb700 1 --2- 172.21.15.42:0/2428590963 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fab0403ad30 0x7fab0403d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:44:10.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:10.704+0000 7fab28efb700 1 --2- 172.21.15.42:0/2428590963 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fab0403ad30 0x7fab0403d240 crc :-1 s=READY pgs=585 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:44:10.718 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:10.708+0000 7fab2a95e700 1 -- 172.21.15.42:0/2428590963 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab2405d110 con 0x7fab2411af70 2019-12-05T05:44:10.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:10.716+0000 7fab23fff700 1 --2- 172.21.15.42:0/2428590963 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fab2411af70 0x7fab2412e530 secure :-1 s=THROTTLE_DONE pgs=1360 cs=0 l=1 rx=0x7fab18003070 tx=0x7fab18010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:10.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:10.716+0000 7fab217fa700 1 -- 172.21.15.42:0/2428590963 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fab18031ff0 con 0x7fab2411af70 2019-12-05T05:44:10.925 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:10.916+0000 7fab2a95e700 1 -- 172.21.15.42:0/2428590963 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fab2405b8c0 con 0x7fab2411af70 2019-12-05T05:44:10.925 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:10.920+0000 7fab23fff700 1 --2- 172.21.15.42:0/2428590963 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fab2411af70 0x7fab2412e530 secure :-1 s=THROTTLE_DONE pgs=1360 cs=0 l=1 rx=0x7fab18003070 tx=0x7fab18010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:10.926 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:10.920+0000 7fab217fa700 1 -- 172.21.15.42:0/2428590963 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 167 v167) v1 ==== 95+0+1772 (secure 0 0 0) 0x7fab18032250 con 0x7fab2411af70 2019-12-05T05:44:10.926 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 167 2019-12-05T05:44:10.926 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:44:10.926 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":167,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":167,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:44:09.069915+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":76,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7755},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7755":{"gid":7755,"name":"a","rank":0,"incarnation":164,"state":"up:active","state_seq":22,"addr":"172.21.15.42:6835/358497296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":358497296},{"type":"v1","addr":"172.21.15.42:6835","nonce":358497296}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:44:10.928 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:10.924+0000 7fab2a95e700 1 -- 172.21.15.42:0/2428590963 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fab0403ad30 msgr2=0x7fab0403d240 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:10.928 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:10.924+0000 7fab2a95e700 1 --2- 172.21.15.42:0/2428590963 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fab0403ad30 0x7fab0403d240 crc :-1 s=READY pgs=585 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:44:10.928 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:10.924+0000 7fab2a95e700 1 -- 172.21.15.42:0/2428590963 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fab2411af70 msgr2=0x7fab2412e530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:10.928 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:10.924+0000 7fab2a95e700 1 --2- 172.21.15.42:0/2428590963 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fab2411af70 0x7fab2412e530 secure :-1 s=READY pgs=1360 cs=0 l=1 rx=0x7fab18003070 tx=0x7fab18010b40).stop 2019-12-05T05:44:10.928 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:10.924+0000 7fab2a95e700 1 -- 172.21.15.42:0/2428590963 shutdown_connections 2019-12-05T05:44:10.929 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:10.924+0000 7fab2a95e700 1 --2- 172.21.15.42:0/2428590963 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fab0403ad30 0x7fab0403d240 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:44:10.929 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:10.924+0000 7fab2a95e700 1 --2- 172.21.15.42:0/2428590963 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fab2411af70 0x7fab2412e530 unknown :-1 s=CLOSED pgs=1360 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:44:10.929 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:10.924+0000 7fab2a95e700 1 -- 172.21.15.42:0/2428590963 shutdown_connections 2019-12-05T05:44:10.929 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:10.924+0000 7fab2a95e700 1 -- 172.21.15.42:0/2428590963 wait complete. 2019-12-05T05:44:10.929 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:10.924+0000 7fab2a95e700 1 -- 172.21.15.42:0/2428590963 >> 172.21.15.42:0/2428590963 conn(0x7fab240aabe0 msgr2=0x7fab240a9790 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:44:10.940 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a has gained rank=0, replacing gid=7650 2019-12-05T05:44:10.940 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 6 secs before reviving mds.b 2019-12-05T05:44:11.195 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:44:11.198 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:44:16.941 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.b 2019-12-05T05:44:16.941 INFO:tasks.ceph.mds.b:Restarting daemon 2019-12-05T05:44:16.941 INFO:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2019-12-05T05:44:16.945 INFO:tasks.ceph.mds.b:Started 2019-12-05T05:44:16.946 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:44:17.006 INFO:tasks.ceph.mds.b.smithi040.stdout:starting mds.b at 2019-12-05T05:44:17.053 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:17.044+0000 7f3cb37ae700 1 Processor -- start 2019-12-05T05:44:17.054 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:17.044+0000 7f3cb37ae700 1 -- start start 2019-12-05T05:44:17.054 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:17.044+0000 7f3cb37ae700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3cac0a7740 0x7f3cac08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:44:17.054 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:17.044+0000 7f3cb37ae700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f3cac09fae0 0x7f3cac07eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:44:17.054 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:17.044+0000 7f3cb37ae700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cac0a0710 con 0x7f3cac07e720 2019-12-05T05:44:17.054 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:17.044+0000 7f3cb37ae700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3cac0a0b00 con 0x7f3cac09fae0 2019-12-05T05:44:17.054 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:17.044+0000 7f3cb37ae700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3cac08e5b0 con 0x7f3cac0a7740 2019-12-05T05:44:17.055 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:17.044+0000 7f3cb1d4b700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3cac0a7740 0x7f3cac08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:44:17.055 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:17.044+0000 7f3cb1d4b700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3cac0a7740 0x7f3cac08df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:39966/0 (socket says 172.21.15.42:39966) 2019-12-05T05:44:17.055 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:17.044+0000 7f3cb1d4b700 1 -- 172.21.15.42:0/799948990 learned_addr learned my addr 172.21.15.42:0/799948990 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:44:17.055 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:17.044+0000 7f3cb254c700 1 --2- 172.21.15.42:0/799948990 >> v2:172.21.15.40:3300/0 conn(0x7f3cac09fae0 0x7f3cac07eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:44:17.055 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:17.044+0000 7f3cb1d4b700 1 -- 172.21.15.42:0/799948990 >> v2:172.21.15.40:3300/0 conn(0x7f3cac09fae0 msgr2=0x7f3cac07eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:44:17.055 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:17.044+0000 7f3cb1d4b700 1 --2- 172.21.15.42:0/799948990 >> v2:172.21.15.40:3300/0 conn(0x7f3cac09fae0 0x7f3cac07eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:44:17.055 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:17.044+0000 7f3cb1d4b700 1 -- 172.21.15.42:0/799948990 >> v1:172.21.15.42:6789/0 conn(0x7f3cac07e720 legacy=0x7f3cac09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:17.056 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:17.044+0000 7f3cb1d4b700 1 -- 172.21.15.42:0/799948990 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cac07f0a0 con 0x7f3cac0a7740 2019-12-05T05:44:17.056 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:17.044+0000 7f3cb1d4b700 1 --2- 172.21.15.42:0/799948990 >> v2:172.21.15.42:3300/0 conn(0x7f3cac0a7740 0x7f3cac08df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f3ca0002f30 tx=0x7f3ca000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:17.056 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:17.044+0000 7f3cb1d4b700 1 --2- 172.21.15.42:0/799948990 >> v2:172.21.15.42:3300/0 conn(0x7f3cac0a7740 0x7f3cac08df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f3ca0002f30 tx=0x7f3ca000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:17.056 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:17.044+0000 7f3cb1d4b700 1 --2- 172.21.15.42:0/799948990 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3cac0a7740 0x7f3cac08df80 secure :-1 s=READY pgs=1364 cs=0 l=1 rx=0x7f3ca0002f30 tx=0x7f3ca000fd30).ready entity=mon.1 client_cookie=48db5897cdbe739f server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:44:17.056 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:17.044+0000 7f3cb1d4b700 1 --2- 172.21.15.42:0/799948990 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3cac0a7740 0x7f3cac08df80 secure :-1 s=THROTTLE_DONE pgs=1364 cs=0 l=1 rx=0x7f3ca0002f30 tx=0x7f3ca000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:17.056 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:17.044+0000 7f3cb0d49700 1 -- 172.21.15.42:0/799948990 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3ca0008c90 con 0x7f3cac0a7740 2019-12-05T05:44:17.056 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:17.044+0000 7f3cb1d4b700 1 --2- 172.21.15.42:0/799948990 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3cac0a7740 0x7f3cac08df80 secure :-1 s=THROTTLE_DONE pgs=1364 cs=0 l=1 rx=0x7f3ca0002f30 tx=0x7f3ca000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:17.057 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:17.044+0000 7f3cb0d49700 1 -- 172.21.15.42:0/799948990 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ca00088b0 con 0x7f3cac0a7740 2019-12-05T05:44:17.057 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:17.044+0000 7f3cb1d4b700 1 --2- 172.21.15.42:0/799948990 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3cac0a7740 0x7f3cac08df80 secure :-1 s=THROTTLE_DONE pgs=1364 cs=0 l=1 rx=0x7f3ca0002f30 tx=0x7f3ca000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:17.057 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:17.044+0000 7f3cb37ae700 1 -- 172.21.15.42:0/799948990 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3cac0a7740 msgr2=0x7f3cac08df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:17.057 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:17.044+0000 7f3cb0d49700 1 -- 172.21.15.42:0/799948990 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3ca0011ad0 con 0x7f3cac0a7740 2019-12-05T05:44:17.057 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:17.044+0000 7f3cb37ae700 1 --2- 172.21.15.42:0/799948990 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3cac0a7740 0x7f3cac08df80 secure :-1 s=READY pgs=1364 cs=0 l=1 rx=0x7f3ca0002f30 tx=0x7f3ca000fd30).stop 2019-12-05T05:44:17.057 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:17.048+0000 7f3cb37ae700 1 -- 172.21.15.42:0/799948990 shutdown_connections 2019-12-05T05:44:17.057 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:17.048+0000 7f3cb37ae700 1 --2- 172.21.15.42:0/799948990 >> v2:172.21.15.40:3300/0 conn(0x7f3cac09fae0 0x7f3cac07eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:44:17.058 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:17.048+0000 7f3cb37ae700 1 --2- 172.21.15.42:0/799948990 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3cac0a7740 0x7f3cac08df80 unknown :-1 s=CLOSED pgs=1364 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:44:17.058 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:17.048+0000 7f3cb37ae700 1 -- 172.21.15.42:0/799948990 shutdown_connections 2019-12-05T05:44:17.058 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:17.048+0000 7f3cb37ae700 1 -- 172.21.15.42:0/799948990 wait complete. 2019-12-05T05:44:17.058 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:17.048+0000 7f3cb37ae700 1 -- 172.21.15.42:0/799948990 >> 172.21.15.42:0/799948990 conn(0x7f3cac0aabe0 msgr2=0x7f3cac0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:44:17.058 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:17.048+0000 7f3cb37ae700 1 Processor -- start 2019-12-05T05:44:17.058 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:17.048+0000 7f3cb37ae700 1 -- start start 2019-12-05T05:44:17.061 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:17.048+0000 7f3cb37ae700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f3cac11d9c0 0x7f3cac12e290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:44:17.061 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:17.048+0000 7f3cb37ae700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3cac1228c0 0x7f3cac0752e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:44:17.062 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:17.048+0000 7f3cb37ae700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cac11b6a0 con 0x7f3cac12e7e0 2019-12-05T05:44:17.062 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:17.048+0000 7f3cb37ae700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3cac11ddf0 con 0x7f3cac11d9c0 2019-12-05T05:44:17.062 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:17.048+0000 7f3cb37ae700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cac11aaa0 con 0x7f3cac1228c0 2019-12-05T05:44:17.062 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:17.048+0000 7f3cb1d4b700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f3cac11d9c0 0x7f3cac12e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:44:17.062 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:17.048+0000 7f3cb1d4b700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f3cac11d9c0 0x7f3cac12e290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.42:56020/0 (socket says 172.21.15.42:56020) 2019-12-05T05:44:17.062 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:17.048+0000 7f3cb1d4b700 1 -- 172.21.15.42:0/3758240057 learned_addr learned my addr 172.21.15.42:0/3758240057 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:44:17.062 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:17.048+0000 7f3cb254c700 1 --2- 172.21.15.42:0/3758240057 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3cac1228c0 0x7f3cac0752e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:44:17.063 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:17.052+0000 7f3cb254c700 1 -- 172.21.15.42:0/3758240057 >> v2:172.21.15.40:3300/0 conn(0x7f3cac11d9c0 msgr2=0x7f3cac12e290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:17.063 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:17.052+0000 7f3cb254c700 1 --2- 172.21.15.42:0/3758240057 >> v2:172.21.15.40:3300/0 conn(0x7f3cac11d9c0 0x7f3cac12e290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:44:17.063 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:17.052+0000 7f3cb254c700 1 -- 172.21.15.42:0/3758240057 >> v1:172.21.15.40:6789/0 conn(0x7f3cac12e7e0 legacy=0x7f3cac1220c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:17.063 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:17.052+0000 7f3cb254c700 1 -- 172.21.15.42:0/3758240057 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cac11a7e0 con 0x7f3cac1228c0 2019-12-05T05:44:17.063 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:17.052+0000 7f3cb1d4b700 1 --2- 172.21.15.42:0/3758240057 >> v2:172.21.15.40:3300/0 conn(0x7f3cac11d9c0 0x7f3cac12e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:44:17.063 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:17.052+0000 7f3cb254c700 1 --2- 172.21.15.42:0/3758240057 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3cac1228c0 0x7f3cac0752e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f3ca80010d0 tx=0x7f3ca8001110).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:17.063 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:17.052+0000 7f3cb254c700 1 --2- 172.21.15.42:0/3758240057 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3cac1228c0 0x7f3cac0752e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f3ca80010d0 tx=0x7f3ca8001110).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:17.064 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:17.052+0000 7f3cb254c700 1 --2- 172.21.15.42:0/3758240057 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3cac1228c0 0x7f3cac0752e0 secure :-1 s=READY pgs=289 cs=0 l=1 rx=0x7f3ca80010d0 tx=0x7f3ca8001110).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:44:17.064 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:17.052+0000 7f3cb254c700 1 --2- 172.21.15.42:0/3758240057 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3cac1228c0 0x7f3cac0752e0 secure :-1 s=THROTTLE_DONE pgs=289 cs=0 l=1 rx=0x7f3ca80010d0 tx=0x7f3ca8001110).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:17.064 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:17.052+0000 7f3cb254c700 1 --2- 172.21.15.42:0/3758240057 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3cac1228c0 0x7f3cac0752e0 secure :-1 s=THROTTLE_DONE pgs=289 cs=0 l=1 rx=0x7f3ca80010d0 tx=0x7f3ca8001110).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:17.064 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:17.052+0000 7f3c9e7fc700 1 -- 172.21.15.42:0/3758240057 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3ca400e950 con 0x7f3cac1228c0 2019-12-05T05:44:17.064 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:17.052+0000 7f3cb254c700 1 --2- 172.21.15.42:0/3758240057 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3cac1228c0 0x7f3cac0752e0 secure :-1 s=THROTTLE_DONE pgs=289 cs=0 l=1 rx=0x7f3ca80010d0 tx=0x7f3ca8001110).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:17.064 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:17.052+0000 7f3c9e7fc700 1 -- 172.21.15.42:0/3758240057 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ca400e570 con 0x7f3cac1228c0 2019-12-05T05:44:17.064 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:17.052+0000 7f3cb37ae700 1 -- 172.21.15.42:0/3758240057 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cac12ec10 con 0x7f3cac1228c0 2019-12-05T05:44:17.065 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:17.052+0000 7f3c9e7fc700 1 -- 172.21.15.42:0/3758240057 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3ca4012510 con 0x7f3cac1228c0 2019-12-05T05:44:17.065 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:17.052+0000 7f3cb37ae700 1 -- 172.21.15.42:0/3758240057 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cac11c030 con 0x7f3cac1228c0 2019-12-05T05:44:17.065 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:17.052+0000 7f3cb254c700 1 --2- 172.21.15.42:0/3758240057 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3cac1228c0 0x7f3cac0752e0 secure :-1 s=THROTTLE_DONE pgs=289 cs=0 l=1 rx=0x7f3ca80010d0 tx=0x7f3ca8001110).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:17.065 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:17.052+0000 7f3c9e7fc700 1 -- 172.21.15.42:0/3758240057 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f3ca40058c0 con 0x7f3cac1228c0 2019-12-05T05:44:17.065 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:17.052+0000 7f3cb254c700 1 --2- 172.21.15.42:0/3758240057 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3cac1228c0 0x7f3cac0752e0 secure :-1 s=THROTTLE_DONE pgs=289 cs=0 l=1 rx=0x7f3ca80010d0 tx=0x7f3ca8001110).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:17.065 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:17.052+0000 7f3c9e7fc700 1 --2- 172.21.15.42:0/3758240057 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3c9403ad80 0x7f3c9403d290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:44:17.065 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:17.056+0000 7f3c9e7fc700 1 -- 172.21.15.42:0/3758240057 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 8353+0+0 (secure 0 0 0) 0x7f3ca40120c0 con 0x7f3cac1228c0 2019-12-05T05:44:17.066 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:17.056+0000 7f3cb1d4b700 1 --2- 172.21.15.42:0/3758240057 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3c9403ad80 0x7f3c9403d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:44:17.066 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:17.056+0000 7f3cb1d4b700 1 --2- 172.21.15.42:0/3758240057 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3c9403ad80 0x7f3c9403d290 crc :-1 s=READY pgs=587 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:44:17.066 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:17.060+0000 7f3cb37ae700 1 -- 172.21.15.42:0/3758240057 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cac05d110 con 0x7f3cac1228c0 2019-12-05T05:44:17.073 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:17.068+0000 7f3cb254c700 1 --2- 172.21.15.42:0/3758240057 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3cac1228c0 0x7f3cac0752e0 secure :-1 s=THROTTLE_DONE pgs=289 cs=0 l=1 rx=0x7f3ca80010d0 tx=0x7f3ca8001110).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:17.073 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:17.068+0000 7f3c9e7fc700 1 -- 172.21.15.42:0/3758240057 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f3cac11a7e0 con 0x7f3cac1228c0 2019-12-05T05:44:17.283 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:17.276+0000 7f3cb37ae700 1 -- 172.21.15.42:0/3758240057 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3cac05b8c0 con 0x7f3cac1228c0 2019-12-05T05:44:17.283 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:17.276+0000 7f3cb254c700 1 --2- 172.21.15.42:0/3758240057 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3cac1228c0 0x7f3cac0752e0 secure :-1 s=THROTTLE_DONE pgs=289 cs=0 l=1 rx=0x7f3ca80010d0 tx=0x7f3ca8001110).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:17.283 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:17.276+0000 7f3c9e7fc700 1 -- 172.21.15.42:0/3758240057 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 167 v167) v1 ==== 95+0+1772 (secure 0 0 0) 0x7f3ca4032320 con 0x7f3cac1228c0 2019-12-05T05:44:17.283 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 167 2019-12-05T05:44:17.283 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:44:17.284 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":167,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":167,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:44:09.069915+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":76,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7755},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7755":{"gid":7755,"name":"a","rank":0,"incarnation":164,"state":"up:active","state_seq":22,"addr":"172.21.15.42:6835/358497296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":358497296},{"type":"v1","addr":"172.21.15.42:6835","nonce":358497296}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:44:17.286 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:17.280+0000 7f3cb37ae700 1 -- 172.21.15.42:0/3758240057 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3c9403ad80 msgr2=0x7f3c9403d290 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:17.286 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:17.280+0000 7f3cb37ae700 1 --2- 172.21.15.42:0/3758240057 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3c9403ad80 0x7f3c9403d290 crc :-1 s=READY pgs=587 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:44:17.286 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:17.280+0000 7f3cb37ae700 1 -- 172.21.15.42:0/3758240057 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3cac1228c0 msgr2=0x7f3cac0752e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:17.287 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:17.280+0000 7f3cb37ae700 1 --2- 172.21.15.42:0/3758240057 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3cac1228c0 0x7f3cac0752e0 secure :-1 s=READY pgs=289 cs=0 l=1 rx=0x7f3ca80010d0 tx=0x7f3ca8001110).stop 2019-12-05T05:44:17.287 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:17.280+0000 7f3cb37ae700 1 -- 172.21.15.42:0/3758240057 shutdown_connections 2019-12-05T05:44:17.287 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:17.280+0000 7f3cb37ae700 1 --2- 172.21.15.42:0/3758240057 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3c9403ad80 0x7f3c9403d290 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:44:17.287 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:17.280+0000 7f3cb37ae700 1 --2- 172.21.15.42:0/3758240057 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3cac1228c0 0x7f3cac0752e0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:44:17.287 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:17.280+0000 7f3cb37ae700 1 --2- 172.21.15.42:0/3758240057 >> v2:172.21.15.40:3300/0 conn(0x7f3cac11d9c0 0x7f3cac12e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:44:17.287 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:17.280+0000 7f3cb37ae700 1 -- 172.21.15.42:0/3758240057 shutdown_connections 2019-12-05T05:44:17.287 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:17.280+0000 7f3cb37ae700 1 -- 172.21.15.42:0/3758240057 wait complete. 2019-12-05T05:44:17.288 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:17.280+0000 7f3cb37ae700 1 -- 172.21.15.42:0/3758240057 >> 172.21.15.42:0/3758240057 conn(0x7f3cac0aabe0 msgr2=0x7f3cac08eb50 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:44:17.298 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2019-12-05T05:44:19.299 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:44:19.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:19.392+0000 7fa1b9750700 1 Processor -- start 2019-12-05T05:44:19.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:19.392+0000 7fa1b9750700 1 -- start start 2019-12-05T05:44:19.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:19.392+0000 7fa1b9750700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa1b40a7740 0x7fa1b408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:44:19.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:19.392+0000 7fa1b9750700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fa1b407e720 0x7fa1b407eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:44:19.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:19.392+0000 7fa1b9750700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1b40a0710 con 0x7fa1b409f2e0 2019-12-05T05:44:19.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:19.392+0000 7fa1b9750700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1b40a0b00 con 0x7fa1b40a7740 2019-12-05T05:44:19.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:19.392+0000 7fa1b9750700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1b408e5b0 con 0x7fa1b407e720 2019-12-05T05:44:19.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:19.392+0000 7fa1b3fff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fa1b409f2e0 0x7fa1b409f710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:40030/0 (socket says 172.21.15.42:40030) 2019-12-05T05:44:19.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:19.392+0000 7fa1abfff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fa1b407e720 0x7fa1b407eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:44:19.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:19.392+0000 7fa1b3fff700 1 -- 172.21.15.42:0/1046091042 learned_addr learned my addr 172.21.15.42:0/1046091042 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:44:19.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:19.392+0000 7fa1b37fe700 1 --2- 172.21.15.42:0/1046091042 >> v2:172.21.15.40:3300/0 conn(0x7fa1b40a7740 0x7fa1b408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:44:19.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:19.392+0000 7fa1b2ffd700 1 -- 172.21.15.42:0/1046091042 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3893010525 0 0) 0x7fa1b40a0710 con 0x7fa1b409f2e0 2019-12-05T05:44:19.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:19.392+0000 7fa1b2ffd700 1 -- 172.21.15.42:0/1046091042 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1a00030b0 con 0x7fa1b409f2e0 2019-12-05T05:44:19.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:19.396+0000 7fa1abfff700 1 -- 172.21.15.42:0/1046091042 >> v2:172.21.15.40:3300/0 conn(0x7fa1b40a7740 msgr2=0x7fa1b408df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:44:19.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:19.396+0000 7fa1abfff700 1 --2- 172.21.15.42:0/1046091042 >> v2:172.21.15.40:3300/0 conn(0x7fa1b40a7740 0x7fa1b408df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:44:19.404 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:19.396+0000 7fa1abfff700 1 -- 172.21.15.42:0/1046091042 >> v1:172.21.15.42:6789/0 conn(0x7fa1b409f2e0 legacy=0x7fa1b409f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:19.404 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:19.396+0000 7fa1abfff700 1 -- 172.21.15.42:0/1046091042 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1b407f0a0 con 0x7fa1b407e720 2019-12-05T05:44:19.404 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:19.396+0000 7fa1abfff700 1 --2- 172.21.15.42:0/1046091042 >> v2:172.21.15.42:3300/0 conn(0x7fa1b407e720 0x7fa1b407eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fa19c002f30 tx=0x7fa19c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:19.404 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:19.396+0000 7fa1abfff700 1 --2- 172.21.15.42:0/1046091042 >> v2:172.21.15.42:3300/0 conn(0x7fa1b407e720 0x7fa1b407eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fa19c002f30 tx=0x7fa19c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:19.404 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:19.396+0000 7fa1abfff700 1 --2- 172.21.15.42:0/1046091042 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa1b407e720 0x7fa1b407eb50 secure :-1 s=READY pgs=1366 cs=0 l=1 rx=0x7fa19c002f30 tx=0x7fa19c00fc70).ready entity=mon.1 client_cookie=a9b833bc51bae39f server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:44:19.404 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:19.396+0000 7fa1abfff700 1 --2- 172.21.15.42:0/1046091042 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa1b407e720 0x7fa1b407eb50 secure :-1 s=THROTTLE_DONE pgs=1366 cs=0 l=1 rx=0x7fa19c002f30 tx=0x7fa19c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:19.404 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:19.396+0000 7fa1abfff700 1 --2- 172.21.15.42:0/1046091042 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa1b407e720 0x7fa1b407eb50 secure :-1 s=THROTTLE_DONE pgs=1366 cs=0 l=1 rx=0x7fa19c002f30 tx=0x7fa19c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:19.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:19.396+0000 7fa1b2ffd700 1 -- 172.21.15.42:0/1046091042 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa19c0112b0 con 0x7fa1b407e720 2019-12-05T05:44:19.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:19.396+0000 7fa1abfff700 1 --2- 172.21.15.42:0/1046091042 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa1b407e720 0x7fa1b407eb50 secure :-1 s=THROTTLE_DONE pgs=1366 cs=0 l=1 rx=0x7fa19c002f30 tx=0x7fa19c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:19.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:19.396+0000 7fa1b2ffd700 1 -- 172.21.15.42:0/1046091042 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa19c008ad0 con 0x7fa1b407e720 2019-12-05T05:44:19.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:19.396+0000 7fa1b2ffd700 1 -- 172.21.15.42:0/1046091042 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa19c011890 con 0x7fa1b407e720 2019-12-05T05:44:19.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:19.396+0000 7fa1b9750700 1 -- 172.21.15.42:0/1046091042 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa1b407e720 msgr2=0x7fa1b407eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:19.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:19.396+0000 7fa1b9750700 1 --2- 172.21.15.42:0/1046091042 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa1b407e720 0x7fa1b407eb50 secure :-1 s=READY pgs=1366 cs=0 l=1 rx=0x7fa19c002f30 tx=0x7fa19c00fc70).stop 2019-12-05T05:44:19.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:19.396+0000 7fa1b9750700 1 -- 172.21.15.42:0/1046091042 shutdown_connections 2019-12-05T05:44:19.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:19.396+0000 7fa1b9750700 1 --2- 172.21.15.42:0/1046091042 >> v2:172.21.15.40:3300/0 conn(0x7fa1b40a7740 0x7fa1b408df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:44:19.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:19.396+0000 7fa1b9750700 1 --2- 172.21.15.42:0/1046091042 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa1b407e720 0x7fa1b407eb50 unknown :-1 s=CLOSED pgs=1366 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:44:19.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:19.396+0000 7fa1b9750700 1 -- 172.21.15.42:0/1046091042 shutdown_connections 2019-12-05T05:44:19.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:19.396+0000 7fa1b9750700 1 -- 172.21.15.42:0/1046091042 wait complete. 2019-12-05T05:44:19.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:19.396+0000 7fa1b9750700 1 -- 172.21.15.42:0/1046091042 >> 172.21.15.42:0/1046091042 conn(0x7fa1b40aabe0 msgr2=0x7fa1b40a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:44:19.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:19.400+0000 7fa1b9750700 1 Processor -- start 2019-12-05T05:44:19.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:19.400+0000 7fa1b9750700 1 -- start start 2019-12-05T05:44:19.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:19.400+0000 7fa1b9750700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa1b4123990 0x7fa1b40a67f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:44:19.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:19.400+0000 7fa1b9750700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fa1b41302c0 0x7fa1b4064290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:44:19.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:19.400+0000 7fa1b9750700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1b40a42c0 con 0x7fa1b40647e0 2019-12-05T05:44:19.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:19.400+0000 7fa1b9750700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1b4063d50 con 0x7fa1b4123990 2019-12-05T05:44:19.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:19.400+0000 7fa1b9750700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1b4063ff0 con 0x7fa1b41302c0 2019-12-05T05:44:19.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:19.400+0000 7fa1abfff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fa1b41302c0 0x7fa1b4064290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:44:19.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:19.400+0000 7fa1abfff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fa1b41302c0 0x7fa1b4064290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:39998/0 (socket says 172.21.15.42:39998) 2019-12-05T05:44:19.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:19.400+0000 7fa1abfff700 1 -- 172.21.15.42:0/648789404 learned_addr learned my addr 172.21.15.42:0/648789404 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:44:19.411 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:19.400+0000 7fa1b37fe700 1 --2- 172.21.15.42:0/648789404 >> v2:172.21.15.40:3300/0 conn(0x7fa1b4123990 0x7fa1b40a67f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:44:19.411 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:19.400+0000 7fa1abfff700 1 -- 172.21.15.42:0/648789404 >> v2:172.21.15.40:3300/0 conn(0x7fa1b4123990 msgr2=0x7fa1b40a67f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:19.411 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:19.400+0000 7fa1abfff700 1 --2- 172.21.15.42:0/648789404 >> v2:172.21.15.40:3300/0 conn(0x7fa1b4123990 0x7fa1b40a67f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:44:19.411 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:19.400+0000 7fa1abfff700 1 -- 172.21.15.42:0/648789404 >> v1:172.21.15.40:6789/0 conn(0x7fa1b40647e0 legacy=0x7fa1b40a3ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:19.411 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:19.400+0000 7fa1abfff700 1 -- 172.21.15.42:0/648789404 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1b4065a50 con 0x7fa1b41302c0 2019-12-05T05:44:19.411 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:19.400+0000 7fa1abfff700 1 --2- 172.21.15.42:0/648789404 >> v2:172.21.15.42:3300/0 conn(0x7fa1b41302c0 0x7fa1b4064290 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fa19c002f30 tx=0x7fa19c011d90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:19.411 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:19.400+0000 7fa1abfff700 1 --2- 172.21.15.42:0/648789404 >> v2:172.21.15.42:3300/0 conn(0x7fa1b41302c0 0x7fa1b4064290 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fa19c002f30 tx=0x7fa19c011d90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:19.412 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:19.400+0000 7fa1abfff700 1 --2- 172.21.15.42:0/648789404 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa1b41302c0 0x7fa1b4064290 secure :-1 s=READY pgs=1367 cs=0 l=1 rx=0x7fa19c002f30 tx=0x7fa19c011d90).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:44:19.412 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:19.400+0000 7fa1abfff700 1 --2- 172.21.15.42:0/648789404 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa1b41302c0 0x7fa1b4064290 secure :-1 s=THROTTLE_DONE pgs=1367 cs=0 l=1 rx=0x7fa19c002f30 tx=0x7fa19c011d90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:19.412 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:19.400+0000 7fa1abfff700 1 --2- 172.21.15.42:0/648789404 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa1b41302c0 0x7fa1b4064290 secure :-1 s=THROTTLE_DONE pgs=1367 cs=0 l=1 rx=0x7fa19c002f30 tx=0x7fa19c011d90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:19.412 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:19.400+0000 7fa1abfff700 1 --2- 172.21.15.42:0/648789404 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa1b41302c0 0x7fa1b4064290 secure :-1 s=THROTTLE_DONE pgs=1367 cs=0 l=1 rx=0x7fa19c002f30 tx=0x7fa19c011d90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:19.412 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:19.400+0000 7fa1b0ff9700 1 -- 172.21.15.42:0/648789404 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa19c007150 con 0x7fa1b41302c0 2019-12-05T05:44:19.412 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:19.400+0000 7fa1b0ff9700 1 -- 172.21.15.42:0/648789404 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa19c0114b0 con 0x7fa1b41302c0 2019-12-05T05:44:19.412 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:19.400+0000 7fa1b9750700 1 -- 172.21.15.42:0/648789404 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1b40a6d40 con 0x7fa1b41302c0 2019-12-05T05:44:19.413 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:19.400+0000 7fa1b0ff9700 1 -- 172.21.15.42:0/648789404 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa19c00f550 con 0x7fa1b41302c0 2019-12-05T05:44:19.413 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:19.400+0000 7fa1b9750700 1 -- 172.21.15.42:0/648789404 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1b41378b0 con 0x7fa1b41302c0 2019-12-05T05:44:19.413 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:19.404+0000 7fa1abfff700 1 --2- 172.21.15.42:0/648789404 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa1b41302c0 0x7fa1b4064290 secure :-1 s=THROTTLE_DONE pgs=1367 cs=0 l=1 rx=0x7fa19c002f30 tx=0x7fa19c011d90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:19.413 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:19.404+0000 7fa1b0ff9700 1 -- 172.21.15.42:0/648789404 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fa19c005540 con 0x7fa1b41302c0 2019-12-05T05:44:19.413 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:19.404+0000 7fa1abfff700 1 --2- 172.21.15.42:0/648789404 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa1b41302c0 0x7fa1b4064290 secure :-1 s=THROTTLE_DONE pgs=1367 cs=0 l=1 rx=0x7fa19c002f30 tx=0x7fa19c011d90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:19.413 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:19.404+0000 7fa1b0ff9700 1 --2- 172.21.15.42:0/648789404 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa19803ae60 0x7fa19803d370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:44:19.413 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:19.404+0000 7fa1b0ff9700 1 -- 172.21.15.42:0/648789404 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 8353+0+0 (secure 0 0 0) 0x7fa19c08f840 con 0x7fa1b41302c0 2019-12-05T05:44:19.414 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:19.404+0000 7fa1b37fe700 1 --2- 172.21.15.42:0/648789404 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa19803ae60 0x7fa19803d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:44:19.414 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:19.404+0000 7fa1b37fe700 1 --2- 172.21.15.42:0/648789404 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa19803ae60 0x7fa19803d370 crc :-1 s=READY pgs=588 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:44:19.414 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:19.408+0000 7fa1b9750700 1 -- 172.21.15.42:0/648789404 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1b405d110 con 0x7fa1b41302c0 2019-12-05T05:44:19.418 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:19.412+0000 7fa1abfff700 1 --2- 172.21.15.42:0/648789404 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa1b41302c0 0x7fa1b4064290 secure :-1 s=THROTTLE_DONE pgs=1367 cs=0 l=1 rx=0x7fa19c002f30 tx=0x7fa19c011d90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:19.418 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:19.412+0000 7fa1b0ff9700 1 -- 172.21.15.42:0/648789404 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fa19c007740 con 0x7fa1b41302c0 2019-12-05T05:44:19.592 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:19.588+0000 7fa1b9750700 1 -- 172.21.15.42:0/648789404 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa1b405b8c0 con 0x7fa1b41302c0 2019-12-05T05:44:19.594 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:19.588+0000 7fa1abfff700 1 --2- 172.21.15.42:0/648789404 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa1b41302c0 0x7fa1b4064290 secure :-1 s=THROTTLE_DONE pgs=1367 cs=0 l=1 rx=0x7fa19c002f30 tx=0x7fa19c011d90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:19.594 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:19.588+0000 7fa1b0ff9700 1 -- 172.21.15.42:0/648789404 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 167 v167) v1 ==== 95+0+1772 (secure 0 0 0) 0x7fa19c074550 con 0x7fa1b41302c0 2019-12-05T05:44:19.594 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 167 2019-12-05T05:44:19.594 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:44:19.594 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":167,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":167,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:44:09.069915+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":76,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7755},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7755":{"gid":7755,"name":"a","rank":0,"incarnation":164,"state":"up:active","state_seq":22,"addr":"172.21.15.42:6835/358497296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":358497296},{"type":"v1","addr":"172.21.15.42:6835","nonce":358497296}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:44:19.596 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:19.592+0000 7fa1b9750700 1 -- 172.21.15.42:0/648789404 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa19803ae60 msgr2=0x7fa19803d370 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:19.596 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:19.592+0000 7fa1b9750700 1 --2- 172.21.15.42:0/648789404 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa19803ae60 0x7fa19803d370 crc :-1 s=READY pgs=588 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:44:19.596 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:19.592+0000 7fa1b9750700 1 -- 172.21.15.42:0/648789404 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa1b41302c0 msgr2=0x7fa1b4064290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:19.596 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:19.592+0000 7fa1b9750700 1 --2- 172.21.15.42:0/648789404 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa1b41302c0 0x7fa1b4064290 secure :-1 s=READY pgs=1367 cs=0 l=1 rx=0x7fa19c002f30 tx=0x7fa19c011d90).stop 2019-12-05T05:44:19.597 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:19.592+0000 7fa1b9750700 1 -- 172.21.15.42:0/648789404 shutdown_connections 2019-12-05T05:44:19.597 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:19.592+0000 7fa1b9750700 1 --2- 172.21.15.42:0/648789404 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa19803ae60 0x7fa19803d370 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:44:19.597 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:19.592+0000 7fa1b9750700 1 --2- 172.21.15.42:0/648789404 >> v2:172.21.15.40:3300/0 conn(0x7fa1b4123990 0x7fa1b40a67f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:44:19.597 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:19.592+0000 7fa1b9750700 1 --2- 172.21.15.42:0/648789404 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa1b41302c0 0x7fa1b4064290 unknown :-1 s=CLOSED pgs=1367 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:44:19.597 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:19.592+0000 7fa1b9750700 1 -- 172.21.15.42:0/648789404 shutdown_connections 2019-12-05T05:44:19.597 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:19.592+0000 7fa1b9750700 1 -- 172.21.15.42:0/648789404 wait complete. 2019-12-05T05:44:19.598 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:19.592+0000 7fa1b9750700 1 -- 172.21.15.42:0/648789404 >> 172.21.15.42:0/648789404 conn(0x7fa1b40a7740 msgr2=0x7fa1b409f3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:44:19.606 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2019-12-05T05:44:21.612 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:44:21.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:21.708+0000 7f5538215700 1 Processor -- start 2019-12-05T05:44:21.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:21.708+0000 7f5538215700 1 -- start start 2019-12-05T05:44:21.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:21.712+0000 7f5538215700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f55300a7740 0x7f553008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:44:21.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:21.712+0000 7f5538215700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f553009fae0 0x7f553007eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:44:21.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:21.712+0000 7f5538215700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55300a0710 con 0x7f553007e720 2019-12-05T05:44:21.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:21.712+0000 7f5538215700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55300a0b00 con 0x7f553009fae0 2019-12-05T05:44:21.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:21.712+0000 7f5538215700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f553008e5b0 con 0x7f55300a7740 2019-12-05T05:44:21.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:21.712+0000 7f5536fb3700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f553009fae0 0x7f553007eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:44:21.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:21.712+0000 7f5535fb1700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f553007e720 0x7f553009f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:40056/0 (socket says 172.21.15.42:40056) 2019-12-05T05:44:21.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:21.712+0000 7f5535fb1700 1 -- 172.21.15.42:0/1503185112 learned_addr learned my addr 172.21.15.42:0/1503185112 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:44:21.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:21.712+0000 7f55367b2700 1 --2- 172.21.15.42:0/1503185112 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f55300a7740 0x7f553008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:44:21.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:21.712+0000 7f55357b0700 1 -- 172.21.15.42:0/1503185112 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 960415684 0 0) 0x7f55300a0710 con 0x7f553007e720 2019-12-05T05:44:21.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:21.712+0000 7f55357b0700 1 -- 172.21.15.42:0/1503185112 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f551c0030b0 con 0x7f553007e720 2019-12-05T05:44:21.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:21.712+0000 7f55357b0700 1 -- 172.21.15.42:0/1503185112 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 17181202 0 0) 0x7f551c0030b0 con 0x7f553007e720 2019-12-05T05:44:21.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:21.712+0000 7f5536fb3700 1 -- 172.21.15.42:0/1503185112 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f55300a7740 msgr2=0x7f553008df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:44:21.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:21.712+0000 7f5536fb3700 1 --2- 172.21.15.42:0/1503185112 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f55300a7740 0x7f553008df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:44:21.724 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:21.712+0000 7f5536fb3700 1 -- 172.21.15.42:0/1503185112 >> v1:172.21.15.42:6789/0 conn(0x7f553007e720 legacy=0x7f553009f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:21.724 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:21.712+0000 7f5536fb3700 1 -- 172.21.15.42:0/1503185112 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f553007f0a0 con 0x7f553009fae0 2019-12-05T05:44:21.724 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:21.712+0000 7f55367b2700 1 --2- 172.21.15.42:0/1503185112 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f55300a7740 0x7f553008df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:44:21.724 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:21.712+0000 7f5536fb3700 1 --2- 172.21.15.42:0/1503185112 >> v2:172.21.15.42:3300/0 conn(0x7f553009fae0 0x7f553007eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f5528008b10 tx=0x7f55200019d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:21.724 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:21.712+0000 7f5536fb3700 1 --2- 172.21.15.42:0/1503185112 >> v2:172.21.15.42:3300/0 conn(0x7f553009fae0 0x7f553007eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f5528008b10 tx=0x7f55200019d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:21.724 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:21.712+0000 7f5536fb3700 1 --2- 172.21.15.42:0/1503185112 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f553009fae0 0x7f553007eb50 secure :-1 s=READY pgs=1369 cs=0 l=1 rx=0x7f5528008b10 tx=0x7f55200019d0).ready entity=mon.1 client_cookie=f367b2b3ec6f8fd7 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:44:21.725 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:21.712+0000 7f5536fb3700 1 --2- 172.21.15.42:0/1503185112 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f553009fae0 0x7f553007eb50 secure :-1 s=THROTTLE_DONE pgs=1369 cs=0 l=1 rx=0x7f5528008b10 tx=0x7f55200019d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:21.725 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:21.712+0000 7f55357b0700 1 -- 172.21.15.42:0/1503185112 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5520001750 con 0x7f553009fae0 2019-12-05T05:44:21.725 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:21.712+0000 7f5536fb3700 1 --2- 172.21.15.42:0/1503185112 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f553009fae0 0x7f553007eb50 secure :-1 s=THROTTLE_DONE pgs=1369 cs=0 l=1 rx=0x7f5528008b10 tx=0x7f55200019d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:21.725 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:21.712+0000 7f5536fb3700 1 --2- 172.21.15.42:0/1503185112 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f553009fae0 0x7f553007eb50 secure :-1 s=THROTTLE_DONE pgs=1369 cs=0 l=1 rx=0x7f5528008b10 tx=0x7f55200019d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:21.725 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:21.712+0000 7f55357b0700 1 -- 172.21.15.42:0/1503185112 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5520001560 con 0x7f553009fae0 2019-12-05T05:44:21.725 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:21.712+0000 7f55357b0700 1 -- 172.21.15.42:0/1503185112 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f55280109f0 con 0x7f553009fae0 2019-12-05T05:44:21.725 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:21.712+0000 7f5538215700 1 -- 172.21.15.42:0/1503185112 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f553009fae0 msgr2=0x7f553007eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:21.725 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:21.712+0000 7f5538215700 1 --2- 172.21.15.42:0/1503185112 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f553009fae0 0x7f553007eb50 secure :-1 s=READY pgs=1369 cs=0 l=1 rx=0x7f5528008b10 tx=0x7f55200019d0).stop 2019-12-05T05:44:21.726 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:21.716+0000 7f5538215700 1 -- 172.21.15.42:0/1503185112 shutdown_connections 2019-12-05T05:44:21.726 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:21.716+0000 7f5538215700 1 --2- 172.21.15.42:0/1503185112 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f553009fae0 0x7f553007eb50 unknown :-1 s=CLOSED pgs=1369 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:44:21.726 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:21.716+0000 7f5538215700 1 --2- 172.21.15.42:0/1503185112 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f55300a7740 0x7f553008df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:44:21.726 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:21.716+0000 7f5538215700 1 -- 172.21.15.42:0/1503185112 shutdown_connections 2019-12-05T05:44:21.726 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:21.716+0000 7f5538215700 1 -- 172.21.15.42:0/1503185112 wait complete. 2019-12-05T05:44:21.726 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:21.716+0000 7f5538215700 1 -- 172.21.15.42:0/1503185112 >> 172.21.15.42:0/1503185112 conn(0x7f55300aabe0 msgr2=0x7f55300a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:44:21.729 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:21.716+0000 7f5538215700 1 Processor -- start 2019-12-05T05:44:21.730 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:21.716+0000 7f5538215700 1 -- start start 2019-12-05T05:44:21.730 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:21.720+0000 7f5538215700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f55300a9ad0 0x7f55301170a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:44:21.730 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:21.720+0000 7f5538215700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f55301183e0 0x7f5530079320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:44:21.730 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:21.720+0000 7f5538215700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5530079870 con 0x7f553011a450 2019-12-05T05:44:21.730 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:21.720+0000 7f5538215700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5530119f10 con 0x7f55301183e0 2019-12-05T05:44:21.730 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:21.720+0000 7f5538215700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f553011a1b0 con 0x7f55300a9ad0 2019-12-05T05:44:21.730 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:21.720+0000 7f55367b2700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f55300a9ad0 0x7f55301170a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:44:21.731 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:21.720+0000 7f55367b2700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f55300a9ad0 0x7f55301170a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:40024/0 (socket says 172.21.15.42:40024) 2019-12-05T05:44:21.731 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:21.720+0000 7f55367b2700 1 -- 172.21.15.42:0/3446112424 learned_addr learned my addr 172.21.15.42:0/3446112424 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:44:21.731 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:21.720+0000 7f5536fb3700 1 --2- 172.21.15.42:0/3446112424 >> v2:172.21.15.40:3300/0 conn(0x7f55301183e0 0x7f5530079320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:44:21.731 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:21.720+0000 7f55367b2700 1 -- 172.21.15.42:0/3446112424 >> v2:172.21.15.40:3300/0 conn(0x7f55301183e0 msgr2=0x7f5530079320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:21.731 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:21.720+0000 7f55367b2700 1 --2- 172.21.15.42:0/3446112424 >> v2:172.21.15.40:3300/0 conn(0x7f55301183e0 0x7f5530079320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:44:21.731 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:21.720+0000 7f55367b2700 1 -- 172.21.15.42:0/3446112424 >> v1:172.21.15.40:6789/0 conn(0x7f553011a450 legacy=0x7f553011b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:21.732 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:21.720+0000 7f55367b2700 1 -- 172.21.15.42:0/3446112424 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f553007aab0 con 0x7f55300a9ad0 2019-12-05T05:44:21.732 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:21.720+0000 7f55367b2700 1 --2- 172.21.15.42:0/3446112424 >> v2:172.21.15.42:3300/0 conn(0x7f55300a9ad0 0x7f55301170a0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f5530117c10 tx=0x7f552c010d80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:21.732 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:21.720+0000 7f55367b2700 1 --2- 172.21.15.42:0/3446112424 >> v2:172.21.15.42:3300/0 conn(0x7f55300a9ad0 0x7f55301170a0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f5530117c10 tx=0x7f552c010d80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:21.732 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:21.720+0000 7f55367b2700 1 --2- 172.21.15.42:0/3446112424 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f55300a9ad0 0x7f55301170a0 secure :-1 s=READY pgs=1370 cs=0 l=1 rx=0x7f5530117c10 tx=0x7f552c010d80).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:44:21.732 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:21.720+0000 7f55367b2700 1 --2- 172.21.15.42:0/3446112424 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f55300a9ad0 0x7f55301170a0 secure :-1 s=THROTTLE_DONE pgs=1370 cs=0 l=1 rx=0x7f5530117c10 tx=0x7f552c010d80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:21.732 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:21.720+0000 7f55367b2700 1 --2- 172.21.15.42:0/3446112424 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f55300a9ad0 0x7f55301170a0 secure :-1 s=THROTTLE_DONE pgs=1370 cs=0 l=1 rx=0x7f5530117c10 tx=0x7f552c010d80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:21.732 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:21.720+0000 7f5526ffd700 1 -- 172.21.15.42:0/3446112424 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f552c005a30 con 0x7f55300a9ad0 2019-12-05T05:44:21.733 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:21.720+0000 7f55367b2700 1 --2- 172.21.15.42:0/3446112424 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f55300a9ad0 0x7f55301170a0 secure :-1 s=THROTTLE_DONE pgs=1370 cs=0 l=1 rx=0x7f5530117c10 tx=0x7f552c010d80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:21.733 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:21.720+0000 7f5526ffd700 1 -- 172.21.15.42:0/3446112424 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f552c007200 con 0x7f55300a9ad0 2019-12-05T05:44:21.733 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:21.720+0000 7f5538215700 1 -- 172.21.15.42:0/3446112424 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5530079aa0 con 0x7f55300a9ad0 2019-12-05T05:44:21.733 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:21.720+0000 7f5526ffd700 1 -- 172.21.15.42:0/3446112424 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f552c0125b0 con 0x7f55300a9ad0 2019-12-05T05:44:21.733 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:21.720+0000 7f5538215700 1 -- 172.21.15.42:0/3446112424 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5530079ff0 con 0x7f55300a9ad0 2019-12-05T05:44:21.733 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:21.724+0000 7f55367b2700 1 --2- 172.21.15.42:0/3446112424 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f55300a9ad0 0x7f55301170a0 secure :-1 s=THROTTLE_DONE pgs=1370 cs=0 l=1 rx=0x7f5530117c10 tx=0x7f552c010d80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:21.733 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:21.724+0000 7f5526ffd700 1 -- 172.21.15.42:0/3446112424 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f552c0128c0 con 0x7f55300a9ad0 2019-12-05T05:44:21.734 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:21.724+0000 7f55367b2700 1 --2- 172.21.15.42:0/3446112424 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f55300a9ad0 0x7f55301170a0 secure :-1 s=THROTTLE_DONE pgs=1370 cs=0 l=1 rx=0x7f5530117c10 tx=0x7f552c010d80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:21.734 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:21.724+0000 7f5526ffd700 1 --2- 172.21.15.42:0/3446112424 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f551803ad80 0x7f551803d290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:44:21.734 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:21.724+0000 7f5526ffd700 1 -- 172.21.15.42:0/3446112424 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 8353+0+0 (secure 0 0 0) 0x7f552c0090a0 con 0x7f55300a9ad0 2019-12-05T05:44:21.734 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:21.724+0000 7f5535fb1700 1 --2- 172.21.15.42:0/3446112424 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f551803ad80 0x7f551803d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:44:21.734 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:21.724+0000 7f5535fb1700 1 --2- 172.21.15.42:0/3446112424 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f551803ad80 0x7f551803d290 crc :-1 s=READY pgs=589 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:44:21.735 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:21.724+0000 7f5538215700 1 -- 172.21.15.42:0/3446112424 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f553005d110 con 0x7f55300a9ad0 2019-12-05T05:44:21.737 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:21.732+0000 7f55367b2700 1 --2- 172.21.15.42:0/3446112424 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f55300a9ad0 0x7f55301170a0 secure :-1 s=THROTTLE_DONE pgs=1370 cs=0 l=1 rx=0x7f5530117c10 tx=0x7f552c010d80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:21.737 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:21.732+0000 7f5526ffd700 1 -- 172.21.15.42:0/3446112424 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f552c032c70 con 0x7f55300a9ad0 2019-12-05T05:44:21.913 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:21.908+0000 7f5538215700 1 -- 172.21.15.42:0/3446112424 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f553005b8c0 con 0x7f55300a9ad0 2019-12-05T05:44:21.914 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:21.908+0000 7f55367b2700 1 --2- 172.21.15.42:0/3446112424 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f55300a9ad0 0x7f55301170a0 secure :-1 s=THROTTLE_DONE pgs=1370 cs=0 l=1 rx=0x7f5530117c10 tx=0x7f552c010d80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:21.914 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:21.908+0000 7f5526ffd700 1 -- 172.21.15.42:0/3446112424 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 167 v167) v1 ==== 95+0+1772 (secure 0 0 0) 0x7f552c032ed0 con 0x7f55300a9ad0 2019-12-05T05:44:21.914 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 167 2019-12-05T05:44:21.915 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:44:21.915 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":167,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":167,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:44:09.069915+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":76,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7755},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7755":{"gid":7755,"name":"a","rank":0,"incarnation":164,"state":"up:active","state_seq":22,"addr":"172.21.15.42:6835/358497296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":358497296},{"type":"v1","addr":"172.21.15.42:6835","nonce":358497296}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:44:21.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:21.912+0000 7f5538215700 1 -- 172.21.15.42:0/3446112424 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f551803ad80 msgr2=0x7f551803d290 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:21.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:21.912+0000 7f5538215700 1 --2- 172.21.15.42:0/3446112424 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f551803ad80 0x7f551803d290 crc :-1 s=READY pgs=589 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:44:21.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:21.912+0000 7f5538215700 1 -- 172.21.15.42:0/3446112424 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f55300a9ad0 msgr2=0x7f55301170a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:21.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:21.912+0000 7f5538215700 1 --2- 172.21.15.42:0/3446112424 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f55300a9ad0 0x7f55301170a0 secure :-1 s=READY pgs=1370 cs=0 l=1 rx=0x7f5530117c10 tx=0x7f552c010d80).stop 2019-12-05T05:44:21.918 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:21.912+0000 7f5538215700 1 -- 172.21.15.42:0/3446112424 shutdown_connections 2019-12-05T05:44:21.918 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:21.912+0000 7f5538215700 1 --2- 172.21.15.42:0/3446112424 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f551803ad80 0x7f551803d290 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:44:21.918 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:21.912+0000 7f5538215700 1 --2- 172.21.15.42:0/3446112424 >> v2:172.21.15.40:3300/0 conn(0x7f55301183e0 0x7f5530079320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:44:21.918 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:21.912+0000 7f5538215700 1 --2- 172.21.15.42:0/3446112424 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f55300a9ad0 0x7f55301170a0 unknown :-1 s=CLOSED pgs=1370 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:44:21.919 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:21.912+0000 7f5538215700 1 -- 172.21.15.42:0/3446112424 shutdown_connections 2019-12-05T05:44:21.919 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:21.912+0000 7f5538215700 1 -- 172.21.15.42:0/3446112424 wait complete. 2019-12-05T05:44:21.919 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:21.912+0000 7f5538215700 1 -- 172.21.15.42:0/3446112424 >> 172.21.15.42:0/3446112424 conn(0x7f553009f120 msgr2=0x7f553008eb70 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:44:21.927 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2019-12-05T05:44:23.928 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:44:24.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.024+0000 7f99b4163700 1 Processor -- start 2019-12-05T05:44:24.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.024+0000 7f99b4163700 1 -- start start 2019-12-05T05:44:24.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.024+0000 7f99b4163700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f99ac09fae0 0x7f99ac14fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:44:24.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.024+0000 7f99b4163700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99ac0a0710 con 0x7f99ac08df80 2019-12-05T05:44:24.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.024+0000 7f99b4163700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99ac07ef20 con 0x7f99ac0a7740 2019-12-05T05:44:24.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.024+0000 7f99b4163700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99ac0a0b00 con 0x7f99ac09fae0 2019-12-05T05:44:24.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.024+0000 7f99b2700700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f99ac0a7740 0x7f99ac07e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:40076/0 (socket says 172.21.15.42:40076) 2019-12-05T05:44:24.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.024+0000 7f99b2700700 1 -- 172.21.15.42:0/1527954140 learned_addr learned my addr 172.21.15.42:0/1527954140 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:44:24.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.024+0000 7f99b2f01700 1 --2- 172.21.15.42:0/1527954140 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f99ac09fae0 0x7f99ac14fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:44:24.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.028+0000 7f99b16fe700 1 -- 172.21.15.42:0/1527954140 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 205026696 0 0) 0x7f99ac07ef20 con 0x7f99ac0a7740 2019-12-05T05:44:24.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.028+0000 7f99b16fe700 1 -- 172.21.15.42:0/1527954140 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99940030b0 con 0x7f99ac0a7740 2019-12-05T05:44:24.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.028+0000 7f99b16fe700 1 -- 172.21.15.42:0/1527954140 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2965285682 0 0) 0x7f99ac0a0710 con 0x7f99ac08df80 2019-12-05T05:44:24.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.028+0000 7f99b16fe700 1 -- 172.21.15.42:0/1527954140 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99ac07ef20 con 0x7f99ac08df80 2019-12-05T05:44:24.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.028+0000 7f99b2f01700 1 -- 172.21.15.42:0/1527954140 >> v1:172.21.15.42:6789/0 conn(0x7f99ac0a7740 legacy=0x7f99ac07e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:24.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.028+0000 7f99b2f01700 1 -- 172.21.15.42:0/1527954140 >> v1:172.21.15.40:6789/0 conn(0x7f99ac08df80 legacy=0x7f99ac09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:24.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.028+0000 7f99b2f01700 1 -- 172.21.15.42:0/1527954140 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99ac0a9ba0 con 0x7f99ac09fae0 2019-12-05T05:44:24.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.028+0000 7f99b2f01700 1 --2- 172.21.15.42:0/1527954140 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f99ac09fae0 0x7f99ac14fff0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f999c002f30 tx=0x7f999c00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:24.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.028+0000 7f99b2f01700 1 --2- 172.21.15.42:0/1527954140 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f99ac09fae0 0x7f99ac14fff0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f999c002f30 tx=0x7f999c00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:24.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.028+0000 7f99b2f01700 1 --2- 172.21.15.42:0/1527954140 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f99ac09fae0 0x7f99ac14fff0 secure :-1 s=READY pgs=292 cs=0 l=1 rx=0x7f999c002f30 tx=0x7f999c00fbf0).ready entity=mon.2 client_cookie=edac9ffd89c8bdb5 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:44:24.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.028+0000 7f99b2f01700 1 --2- 172.21.15.42:0/1527954140 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f99ac09fae0 0x7f99ac14fff0 secure :-1 s=THROTTLE_DONE pgs=292 cs=0 l=1 rx=0x7f999c002f30 tx=0x7f999c00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:24.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.028+0000 7f99b16fe700 1 -- 172.21.15.42:0/1527954140 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f999c008bb0 con 0x7f99ac09fae0 2019-12-05T05:44:24.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.028+0000 7f99b2f01700 1 --2- 172.21.15.42:0/1527954140 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f99ac09fae0 0x7f99ac14fff0 secure :-1 s=THROTTLE_DONE pgs=292 cs=0 l=1 rx=0x7f999c002f30 tx=0x7f999c00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:24.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.028+0000 7f99b16fe700 1 -- 172.21.15.42:0/1527954140 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f999c0087d0 con 0x7f99ac09fae0 2019-12-05T05:44:24.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.028+0000 7f99b2f01700 1 --2- 172.21.15.42:0/1527954140 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f99ac09fae0 0x7f99ac14fff0 secure :-1 s=THROTTLE_DONE pgs=292 cs=0 l=1 rx=0x7f999c002f30 tx=0x7f999c00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:24.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.028+0000 7f99b16fe700 1 -- 172.21.15.42:0/1527954140 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f999c011910 con 0x7f99ac09fae0 2019-12-05T05:44:24.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.028+0000 7f99b4163700 1 -- 172.21.15.42:0/1527954140 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f99ac09fae0 msgr2=0x7f99ac14fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:24.039 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.028+0000 7f99b4163700 1 --2- 172.21.15.42:0/1527954140 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f99ac09fae0 0x7f99ac14fff0 secure :-1 s=READY pgs=292 cs=0 l=1 rx=0x7f999c002f30 tx=0x7f999c00fbf0).stop 2019-12-05T05:44:24.039 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.028+0000 7f99b4163700 1 -- 172.21.15.42:0/1527954140 shutdown_connections 2019-12-05T05:44:24.039 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.028+0000 7f99b4163700 1 --2- 172.21.15.42:0/1527954140 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f99ac09fae0 0x7f99ac14fff0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:44:24.039 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.028+0000 7f99b4163700 1 -- 172.21.15.42:0/1527954140 shutdown_connections 2019-12-05T05:44:24.039 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.028+0000 7f99b4163700 1 -- 172.21.15.42:0/1527954140 wait complete. 2019-12-05T05:44:24.039 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.028+0000 7f99b4163700 1 -- 172.21.15.42:0/1527954140 >> 172.21.15.42:0/1527954140 conn(0x7f99ac0aabe0 msgr2=0x7f99ac0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:44:24.042 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.028+0000 7f99b4163700 1 Processor -- start 2019-12-05T05:44:24.043 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.028+0000 7f99b4163700 1 -- start start 2019-12-05T05:44:24.043 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.032+0000 7f99b4163700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f99ac116c00 0x7f99ac1256c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:44:24.043 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.032+0000 7f99b4163700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f99ac1295a0 0x7f99ac115640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:44:24.043 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.032+0000 7f99b4163700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f99ac115b90 0x7f99ac1138f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:44:24.044 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.032+0000 7f99b4163700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99ac128e00 con 0x7f99ac115b90 2019-12-05T05:44:24.044 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.032+0000 7f99b4163700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99ac129060 con 0x7f99ac116c00 2019-12-05T05:44:24.044 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.032+0000 7f99b4163700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99ac129300 con 0x7f99ac1295a0 2019-12-05T05:44:24.044 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.032+0000 7f99b2700700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f99ac116c00 0x7f99ac1256c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:44:24.044 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.032+0000 7f99b2700700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f99ac116c00 0x7f99ac1256c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:40044/0 (socket says 172.21.15.42:40044) 2019-12-05T05:44:24.044 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.032+0000 7f99b2700700 1 -- 172.21.15.42:0/1291809181 learned_addr learned my addr 172.21.15.42:0/1291809181 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:44:24.044 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.032+0000 7f99b2f01700 1 --2- 172.21.15.42:0/1291809181 >> v2:172.21.15.40:3300/0 conn(0x7f99ac115b90 0x7f99ac1138f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:44:24.045 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.032+0000 7f99b1eff700 1 --2- 172.21.15.42:0/1291809181 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f99ac1295a0 0x7f99ac115640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:44:24.045 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.032+0000 7f99b2700700 1 -- 172.21.15.42:0/1291809181 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f99ac1295a0 msgr2=0x7f99ac115640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:24.045 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.032+0000 7f99b2700700 1 --2- 172.21.15.42:0/1291809181 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f99ac1295a0 0x7f99ac115640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:44:24.047 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.032+0000 7f99b2700700 1 -- 172.21.15.42:0/1291809181 >> v2:172.21.15.40:3300/0 conn(0x7f99ac115b90 msgr2=0x7f99ac1138f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:24.048 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.032+0000 7f99b2700700 1 --2- 172.21.15.42:0/1291809181 >> v2:172.21.15.40:3300/0 conn(0x7f99ac115b90 0x7f99ac1138f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:44:24.048 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.032+0000 7f99b2700700 1 -- 172.21.15.42:0/1291809181 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99ac1299d0 con 0x7f99ac116c00 2019-12-05T05:44:24.048 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.032+0000 7f99b2700700 1 --2- 172.21.15.42:0/1291809181 >> v2:172.21.15.42:3300/0 conn(0x7f99ac116c00 0x7f99ac1256c0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f99ac0658d0 tx=0x7f99a4010e30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:24.048 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.032+0000 7f99b2700700 1 --2- 172.21.15.42:0/1291809181 >> v2:172.21.15.42:3300/0 conn(0x7f99ac116c00 0x7f99ac1256c0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f99ac0658d0 tx=0x7f99a4010e30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:24.048 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.032+0000 7f99b2700700 1 --2- 172.21.15.42:0/1291809181 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f99ac116c00 0x7f99ac1256c0 secure :-1 s=READY pgs=1372 cs=0 l=1 rx=0x7f99ac0658d0 tx=0x7f99a4010e30).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:44:24.048 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.032+0000 7f99b2700700 1 --2- 172.21.15.42:0/1291809181 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f99ac116c00 0x7f99ac1256c0 secure :-1 s=THROTTLE_DONE pgs=1372 cs=0 l=1 rx=0x7f99ac0658d0 tx=0x7f99a4010e30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:24.049 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.032+0000 7f99aaffd700 1 -- 172.21.15.42:0/1291809181 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f99a40105f0 con 0x7f99ac116c00 2019-12-05T05:44:24.049 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.032+0000 7f99b2700700 1 --2- 172.21.15.42:0/1291809181 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f99ac116c00 0x7f99ac1256c0 secure :-1 s=THROTTLE_DONE pgs=1372 cs=0 l=1 rx=0x7f99ac0658d0 tx=0x7f99a4010e30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:24.049 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.032+0000 7f99b4163700 1 -- 172.21.15.42:0/1291809181 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99ac113e40 con 0x7f99ac116c00 2019-12-05T05:44:24.049 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.032+0000 7f99b2700700 1 --2- 172.21.15.42:0/1291809181 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f99ac116c00 0x7f99ac1256c0 secure :-1 s=THROTTLE_DONE pgs=1372 cs=0 l=1 rx=0x7f99ac0658d0 tx=0x7f99a4010e30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:24.050 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.032+0000 7f99aaffd700 1 -- 172.21.15.42:0/1291809181 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99a4010210 con 0x7f99ac116c00 2019-12-05T05:44:24.050 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.032+0000 7f99aaffd700 1 -- 172.21.15.42:0/1291809181 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f99a400cc30 con 0x7f99ac116c00 2019-12-05T05:44:24.050 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.032+0000 7f99b4163700 1 -- 172.21.15.42:0/1291809181 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99ac1141f0 con 0x7f99ac116c00 2019-12-05T05:44:24.050 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.032+0000 7f99b2700700 1 --2- 172.21.15.42:0/1291809181 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f99ac116c00 0x7f99ac1256c0 secure :-1 s=THROTTLE_DONE pgs=1372 cs=0 l=1 rx=0x7f99ac0658d0 tx=0x7f99a4010e30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:24.050 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.032+0000 7f99aaffd700 1 -- 172.21.15.42:0/1291809181 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f99a40057e0 con 0x7f99ac116c00 2019-12-05T05:44:24.050 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.032+0000 7f99b2700700 1 --2- 172.21.15.42:0/1291809181 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f99ac116c00 0x7f99ac1256c0 secure :-1 s=THROTTLE_DONE pgs=1372 cs=0 l=1 rx=0x7f99ac0658d0 tx=0x7f99a4010e30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:24.050 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.036+0000 7f99aaffd700 1 --2- 172.21.15.42:0/1291809181 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f998c03ae70 0x7f998c03d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:44:24.051 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.036+0000 7f99aaffd700 1 -- 172.21.15.42:0/1291809181 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 8353+0+0 (secure 0 0 0) 0x7f99a404b620 con 0x7f99ac116c00 2019-12-05T05:44:24.051 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.036+0000 7f99b1eff700 1 --2- 172.21.15.42:0/1291809181 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f998c03ae70 0x7f998c03d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:44:24.051 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.036+0000 7f99b1eff700 1 --2- 172.21.15.42:0/1291809181 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f998c03ae70 0x7f998c03d380 crc :-1 s=READY pgs=591 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:44:24.051 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.040+0000 7f99b4163700 1 -- 172.21.15.42:0/1291809181 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99ac05d110 con 0x7f99ac116c00 2019-12-05T05:44:24.052 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.048+0000 7f99b2700700 1 --2- 172.21.15.42:0/1291809181 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f99ac116c00 0x7f99ac1256c0 secure :-1 s=THROTTLE_DONE pgs=1372 cs=0 l=1 rx=0x7f99ac0658d0 tx=0x7f99a4010e30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:24.052 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.048+0000 7f99aaffd700 1 -- 172.21.15.42:0/1291809181 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f99a4032680 con 0x7f99ac116c00 2019-12-05T05:44:24.240 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.236+0000 7f99b4163700 1 -- 172.21.15.42:0/1291809181 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f99ac05b8c0 con 0x7f99ac116c00 2019-12-05T05:44:24.241 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.236+0000 7f99b2700700 1 --2- 172.21.15.42:0/1291809181 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f99ac116c00 0x7f99ac1256c0 secure :-1 s=THROTTLE_DONE pgs=1372 cs=0 l=1 rx=0x7f99ac0658d0 tx=0x7f99a4010e30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:24.241 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.236+0000 7f99aaffd700 1 -- 172.21.15.42:0/1291809181 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 168 v168) v1 ==== 95+0+2109 (secure 0 0 0) 0x7f99a40328e0 con 0x7f99ac116c00 2019-12-05T05:44:24.241 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 168 2019-12-05T05:44:24.242 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:44:24.242 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":168,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7884,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/1127133319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1127133319},{"type":"v1","addr":"172.21.15.40:6833","nonce":1127133319}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":168}],"filesystems":[{"mdsmap":{"epoch":167,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:44:09.069915+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":76,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7755},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7755":{"gid":7755,"name":"a","rank":0,"incarnation":164,"state":"up:active","state_seq":22,"addr":"172.21.15.42:6835/358497296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":358497296},{"type":"v1","addr":"172.21.15.42:6835","nonce":358497296}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:44:24.249 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.236+0000 7f99b4163700 1 -- 172.21.15.42:0/1291809181 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f998c03ae70 msgr2=0x7f998c03d380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:24.249 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.240+0000 7f99b4163700 1 --2- 172.21.15.42:0/1291809181 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f998c03ae70 0x7f998c03d380 crc :-1 s=READY pgs=591 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:44:24.249 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.240+0000 7f99b4163700 1 -- 172.21.15.42:0/1291809181 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f99ac116c00 msgr2=0x7f99ac1256c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:24.249 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.240+0000 7f99b4163700 1 --2- 172.21.15.42:0/1291809181 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f99ac116c00 0x7f99ac1256c0 secure :-1 s=READY pgs=1372 cs=0 l=1 rx=0x7f99ac0658d0 tx=0x7f99a4010e30).stop 2019-12-05T05:44:24.249 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.240+0000 7f99b4163700 1 -- 172.21.15.42:0/1291809181 shutdown_connections 2019-12-05T05:44:24.250 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.240+0000 7f99b4163700 1 --2- 172.21.15.42:0/1291809181 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f998c03ae70 0x7f998c03d380 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:44:24.250 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.240+0000 7f99b4163700 1 --2- 172.21.15.42:0/1291809181 >> v2:172.21.15.40:3300/0 conn(0x7f99ac115b90 0x7f99ac1138f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:44:24.250 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.240+0000 7f99b4163700 1 --2- 172.21.15.42:0/1291809181 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f99ac116c00 0x7f99ac1256c0 secure :-1 s=CLOSED pgs=1372 cs=0 l=1 rx=0x7f99ac0658d0 tx=0x7f99a4010e30).stop 2019-12-05T05:44:24.250 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.240+0000 7f99b4163700 1 --2- 172.21.15.42:0/1291809181 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f99ac1295a0 0x7f99ac115640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:44:24.250 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.240+0000 7f99b4163700 1 -- 172.21.15.42:0/1291809181 shutdown_connections 2019-12-05T05:44:24.250 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.240+0000 7f99b4163700 1 -- 172.21.15.42:0/1291809181 wait complete. 2019-12-05T05:44:24.251 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.240+0000 7f99b4163700 1 -- 172.21.15.42:0/1291809181 >> 172.21.15.42:0/1291809181 conn(0x7f99ac08fde0 msgr2=0x7f99ac0a6b30 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:44:24.256 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b reported in up:standby state 2019-12-05T05:44:24.256 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:44:24.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.340+0000 7f977624f700 1 Processor -- start 2019-12-05T05:44:24.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.340+0000 7f977624f700 1 -- start start 2019-12-05T05:44:24.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.344+0000 7f977624f700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f97700a7740 0x7f977008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:44:24.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.344+0000 7f977624f700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f977007e720 0x7f977007eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:44:24.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.344+0000 7f977624f700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97700a0710 con 0x7f977009f2e0 2019-12-05T05:44:24.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.344+0000 7f977624f700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97700a0b00 con 0x7f977007e720 2019-12-05T05:44:24.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.344+0000 7f977624f700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f977008e5b0 con 0x7f97700a7740 2019-12-05T05:44:24.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.344+0000 7f976ffff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f97700a7740 0x7f977008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:44:24.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.344+0000 7f9774fed700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f977009f2e0 0x7f977009f710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:40098/0 (socket says 172.21.15.42:40098) 2019-12-05T05:44:24.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.344+0000 7f9774fed700 1 -- 172.21.15.42:0/146440842 learned_addr learned my addr 172.21.15.42:0/146440842 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:44:24.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.344+0000 7f9767fff700 1 --2- 172.21.15.42:0/146440842 >> v2:172.21.15.40:3300/0 conn(0x7f977007e720 0x7f977007eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:44:24.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.344+0000 7f976ffff700 1 -- 172.21.15.42:0/146440842 >> v2:172.21.15.40:3300/0 conn(0x7f977007e720 msgr2=0x7f977007eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:44:24.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.344+0000 7f976ffff700 1 --2- 172.21.15.42:0/146440842 >> v2:172.21.15.40:3300/0 conn(0x7f977007e720 0x7f977007eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:44:24.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.344+0000 7f976ffff700 1 -- 172.21.15.42:0/146440842 >> v1:172.21.15.42:6789/0 conn(0x7f977009f2e0 legacy=0x7f977009f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:24.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.344+0000 7f976f7fe700 1 -- 172.21.15.42:0/146440842 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2609785161 0 0) 0x7f97700a0710 con 0x7f977009f2e0 2019-12-05T05:44:24.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.344+0000 7f976ffff700 1 -- 172.21.15.42:0/146440842 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f977007f0a0 con 0x7f97700a7740 2019-12-05T05:44:24.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.344+0000 7f9767fff700 1 --2- 172.21.15.42:0/146440842 >> v2:172.21.15.40:3300/0 conn(0x7f977007e720 0x7f977007eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T05:44:24.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.344+0000 7f976ffff700 1 --2- 172.21.15.42:0/146440842 >> v2:172.21.15.42:3300/0 conn(0x7f97700a7740 0x7f977008df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f9760002f30 tx=0x7f976000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:24.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.344+0000 7f976ffff700 1 --2- 172.21.15.42:0/146440842 >> v2:172.21.15.42:3300/0 conn(0x7f97700a7740 0x7f977008df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f9760002f30 tx=0x7f976000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:24.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.344+0000 7f976ffff700 1 --2- 172.21.15.42:0/146440842 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f97700a7740 0x7f977008df80 secure :-1 s=READY pgs=1374 cs=0 l=1 rx=0x7f9760002f30 tx=0x7f976000fc70).ready entity=mon.1 client_cookie=5ec19a4375e889ce server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:44:24.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.344+0000 7f976ffff700 1 --2- 172.21.15.42:0/146440842 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f97700a7740 0x7f977008df80 secure :-1 s=THROTTLE_DONE pgs=1374 cs=0 l=1 rx=0x7f9760002f30 tx=0x7f976000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:24.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.344+0000 7f976f7fe700 1 -- 172.21.15.42:0/146440842 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f97600112b0 con 0x7f97700a7740 2019-12-05T05:44:24.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.344+0000 7f976ffff700 1 --2- 172.21.15.42:0/146440842 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f97700a7740 0x7f977008df80 secure :-1 s=THROTTLE_DONE pgs=1374 cs=0 l=1 rx=0x7f9760002f30 tx=0x7f976000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:24.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.344+0000 7f976ffff700 1 --2- 172.21.15.42:0/146440842 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f97700a7740 0x7f977008df80 secure :-1 s=THROTTLE_DONE pgs=1374 cs=0 l=1 rx=0x7f9760002f30 tx=0x7f976000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:24.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.344+0000 7f976f7fe700 1 -- 172.21.15.42:0/146440842 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9760008ad0 con 0x7f97700a7740 2019-12-05T05:44:24.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.344+0000 7f977624f700 1 -- 172.21.15.42:0/146440842 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f97700a7740 msgr2=0x7f977008df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:24.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.344+0000 7f976f7fe700 1 -- 172.21.15.42:0/146440842 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9760011890 con 0x7f97700a7740 2019-12-05T05:44:24.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.344+0000 7f977624f700 1 --2- 172.21.15.42:0/146440842 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f97700a7740 0x7f977008df80 secure :-1 s=READY pgs=1374 cs=0 l=1 rx=0x7f9760002f30 tx=0x7f976000fc70).stop 2019-12-05T05:44:24.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.344+0000 7f977624f700 1 -- 172.21.15.42:0/146440842 shutdown_connections 2019-12-05T05:44:24.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.344+0000 7f977624f700 1 --2- 172.21.15.42:0/146440842 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f97700a7740 0x7f977008df80 unknown :-1 s=CLOSED pgs=1374 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:44:24.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.344+0000 7f977624f700 1 --2- 172.21.15.42:0/146440842 >> v2:172.21.15.40:3300/0 conn(0x7f977007e720 0x7f977007eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:44:24.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.344+0000 7f977624f700 1 -- 172.21.15.42:0/146440842 shutdown_connections 2019-12-05T05:44:24.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.344+0000 7f977624f700 1 -- 172.21.15.42:0/146440842 wait complete. 2019-12-05T05:44:24.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.344+0000 7f977624f700 1 -- 172.21.15.42:0/146440842 >> 172.21.15.42:0/146440842 conn(0x7f97700aabe0 msgr2=0x7f97700a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:44:24.361 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.348+0000 7f977624f700 1 Processor -- start 2019-12-05T05:44:24.361 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.348+0000 7f977624f700 1 -- start start 2019-12-05T05:44:24.361 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.348+0000 7f977624f700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f9770143ae0 0x7f97700a1a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:44:24.361 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.348+0000 7f977624f700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9770126f40 0x7f9770127370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:44:24.361 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.348+0000 7f977624f700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f977007c100 con 0x7f977007d0a0 2019-12-05T05:44:24.361 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.348+0000 7f977624f700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9770144060 con 0x7f9770143ae0 2019-12-05T05:44:24.362 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.348+0000 7f977624f700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9770144300 con 0x7f9770126f40 2019-12-05T05:44:24.362 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.348+0000 7f9767fff700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f9770143ae0 0x7f97700a1a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:44:24.362 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.348+0000 7f9767fff700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f9770143ae0 0x7f97700a1a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.42:56112/0 (socket says 172.21.15.42:56112) 2019-12-05T05:44:24.362 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.348+0000 7f9767fff700 1 -- 172.21.15.42:0/1673003080 learned_addr learned my addr 172.21.15.42:0/1673003080 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:44:24.362 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.348+0000 7f9774fed700 1 --2- 172.21.15.42:0/1673003080 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9770126f40 0x7f9770127370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:44:24.362 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.348+0000 7f9774fed700 1 -- 172.21.15.42:0/1673003080 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9770126f40 msgr2=0x7f9770127370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2019-12-05T05:44:24.362 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.348+0000 7f9774fed700 1 -- 172.21.15.42:0/1673003080 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9770126f40 msgr2=0x7f9770127370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2019-12-05T05:44:24.363 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.348+0000 7f9774fed700 1 --2- 172.21.15.42:0/1673003080 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9770126f40 0x7f9770127370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-12-05T05:44:24.363 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.348+0000 7f9774fed700 1 --2- 172.21.15.42:0/1673003080 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9770126f40 0x7f9770127370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rx=0 tx=0)._fault waiting 0.200000 2019-12-05T05:44:24.363 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.348+0000 7f9767fff700 1 -- 172.21.15.42:0/1673003080 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9770126f40 msgr2=0x7f9770127370 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2019-12-05T05:44:24.363 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.348+0000 7f9767fff700 1 --2- 172.21.15.42:0/1673003080 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9770126f40 0x7f9770127370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:44:24.363 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.348+0000 7f9767fff700 1 -- 172.21.15.42:0/1673003080 >> v1:172.21.15.40:6789/0 conn(0x7f977007d0a0 legacy=0x7f977007b900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:24.363 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.348+0000 7f976d7fa700 1 -- 172.21.15.42:0/1673003080 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1764602154 0 0) 0x7f977007c100 con 0x7f977007d0a0 2019-12-05T05:44:24.363 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.348+0000 7f9767fff700 1 -- 172.21.15.42:0/1673003080 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97701445a0 con 0x7f9770143ae0 2019-12-05T05:44:24.364 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.348+0000 7f9767fff700 1 --2- 172.21.15.42:0/1673003080 >> v2:172.21.15.40:3300/0 conn(0x7f9770143ae0 0x7f97700a1a20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f9770144220 tx=0x7f9758010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:24.364 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.348+0000 7f9767fff700 1 --2- 172.21.15.42:0/1673003080 >> v2:172.21.15.40:3300/0 conn(0x7f9770143ae0 0x7f97700a1a20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f9770144220 tx=0x7f9758010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:24.364 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.348+0000 7f9767fff700 1 --2- 172.21.15.42:0/1673003080 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9770143ae0 0x7f97700a1a20 secure :-1 s=READY pgs=1067 cs=0 l=1 rx=0x7f9770144220 tx=0x7f9758010ab0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:44:24.364 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.352+0000 7f9767fff700 1 --2- 172.21.15.42:0/1673003080 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9770143ae0 0x7f97700a1a20 secure :-1 s=THROTTLE_DONE pgs=1067 cs=0 l=1 rx=0x7f9770144220 tx=0x7f9758010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:24.364 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.352+0000 7f976d7fa700 1 -- 172.21.15.42:0/1673003080 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f975800a4b0 con 0x7f9770143ae0 2019-12-05T05:44:24.364 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.352+0000 7f9767fff700 1 --2- 172.21.15.42:0/1673003080 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9770143ae0 0x7f97700a1a20 secure :-1 s=THROTTLE_DONE pgs=1067 cs=0 l=1 rx=0x7f9770144220 tx=0x7f9758010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:24.364 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.352+0000 7f9767fff700 1 --2- 172.21.15.42:0/1673003080 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9770143ae0 0x7f97700a1a20 secure :-1 s=THROTTLE_DONE pgs=1067 cs=0 l=1 rx=0x7f9770144220 tx=0x7f9758010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:24.365 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.352+0000 7f976d7fa700 1 -- 172.21.15.42:0/1673003080 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9758007be0 con 0x7f9770143ae0 2019-12-05T05:44:24.365 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.352+0000 7f976d7fa700 1 -- 172.21.15.42:0/1673003080 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f975800aa90 con 0x7f9770143ae0 2019-12-05T05:44:24.365 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.352+0000 7f977624f700 1 -- 172.21.15.42:0/1673003080 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97701278c0 con 0x7f9770143ae0 2019-12-05T05:44:24.365 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.352+0000 7f977624f700 1 -- 172.21.15.42:0/1673003080 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f977012e2f0 con 0x7f9770143ae0 2019-12-05T05:44:24.365 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.352+0000 7f9767fff700 1 --2- 172.21.15.42:0/1673003080 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9770143ae0 0x7f97700a1a20 secure :-1 s=THROTTLE_DONE pgs=1067 cs=0 l=1 rx=0x7f9770144220 tx=0x7f9758010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:24.365 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.352+0000 7f976d7fa700 1 -- 172.21.15.42:0/1673003080 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f9758007170 con 0x7f9770143ae0 2019-12-05T05:44:24.365 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.352+0000 7f9767fff700 1 --2- 172.21.15.42:0/1673003080 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9770143ae0 0x7f97700a1a20 secure :-1 s=THROTTLE_DONE pgs=1067 cs=0 l=1 rx=0x7f9770144220 tx=0x7f9758010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:24.366 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.352+0000 7f976d7fa700 1 --2- 172.21.15.42:0/1673003080 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f975403ad30 0x7f975403d240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:44:24.366 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.352+0000 7f976d7fa700 1 -- 172.21.15.42:0/1673003080 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 8353+0+0 (secure 0 0 0) 0x7f975804b050 con 0x7f9770143ae0 2019-12-05T05:44:24.366 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.352+0000 7f976ffff700 1 --2- 172.21.15.42:0/1673003080 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f975403ad30 0x7f975403d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:44:24.366 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.352+0000 7f976ffff700 1 --2- 172.21.15.42:0/1673003080 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f975403ad30 0x7f975403d240 crc :-1 s=READY pgs=592 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:44:24.366 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.356+0000 7f977624f700 1 -- 172.21.15.42:0/1673003080 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f977005d110 con 0x7f9770143ae0 2019-12-05T05:44:24.373 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.368+0000 7f9767fff700 1 --2- 172.21.15.42:0/1673003080 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9770143ae0 0x7f97700a1a20 secure :-1 s=THROTTLE_DONE pgs=1067 cs=0 l=1 rx=0x7f9770144220 tx=0x7f9758010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:24.373 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.368+0000 7f976d7fa700 1 -- 172.21.15.42:0/1673003080 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f97580320b0 con 0x7f9770143ae0 2019-12-05T05:44:24.561 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.556+0000 7f977624f700 1 -- 172.21.15.42:0/1673003080 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f977005b8c0 con 0x7f9770143ae0 2019-12-05T05:44:24.561 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.556+0000 7f9767fff700 1 --2- 172.21.15.42:0/1673003080 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9770143ae0 0x7f97700a1a20 secure :-1 s=THROTTLE_DONE pgs=1067 cs=0 l=1 rx=0x7f9770144220 tx=0x7f9758010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:24.562 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.556+0000 7f976d7fa700 1 -- 172.21.15.42:0/1673003080 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 168 v168) v1 ==== 95+0+2109 (secure 0 0 0) 0x7f9758032310 con 0x7f9770143ae0 2019-12-05T05:44:24.563 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 168 2019-12-05T05:44:24.564 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:44:24.564 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":168,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7884,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/1127133319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1127133319},{"type":"v1","addr":"172.21.15.40:6833","nonce":1127133319}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":168}],"filesystems":[{"mdsmap":{"epoch":167,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:44:09.069915+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":76,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7755},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7755":{"gid":7755,"name":"a","rank":0,"incarnation":164,"state":"up:active","state_seq":22,"addr":"172.21.15.42:6835/358497296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":358497296},{"type":"v1","addr":"172.21.15.42:6835","nonce":358497296}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:44:24.567 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.560+0000 7f977624f700 1 -- 172.21.15.42:0/1673003080 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f975403ad30 msgr2=0x7f975403d240 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:24.567 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.560+0000 7f977624f700 1 --2- 172.21.15.42:0/1673003080 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f975403ad30 0x7f975403d240 crc :-1 s=READY pgs=592 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:44:24.567 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.560+0000 7f977624f700 1 -- 172.21.15.42:0/1673003080 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9770143ae0 msgr2=0x7f97700a1a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:24.567 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.560+0000 7f977624f700 1 --2- 172.21.15.42:0/1673003080 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9770143ae0 0x7f97700a1a20 secure :-1 s=READY pgs=1067 cs=0 l=1 rx=0x7f9770144220 tx=0x7f9758010ab0).stop 2019-12-05T05:44:24.567 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.560+0000 7f977624f700 1 -- 172.21.15.42:0/1673003080 shutdown_connections 2019-12-05T05:44:24.568 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.560+0000 7f977624f700 1 --2- 172.21.15.42:0/1673003080 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f975403ad30 0x7f975403d240 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:44:24.568 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.560+0000 7f977624f700 1 --2- 172.21.15.42:0/1673003080 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9770126f40 0x7f9770127370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:44:24.568 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.560+0000 7f977624f700 1 --2- 172.21.15.42:0/1673003080 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9770143ae0 0x7f97700a1a20 unknown :-1 s=CLOSED pgs=1067 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:44:24.568 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.560+0000 7f977624f700 1 -- 172.21.15.42:0/1673003080 shutdown_connections 2019-12-05T05:44:24.568 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.560+0000 7f977624f700 1 -- 172.21.15.42:0/1673003080 wait complete. 2019-12-05T05:44:24.568 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.560+0000 7f977624f700 1 -- 172.21.15.42:0/1673003080 >> 172.21.15.42:0/1673003080 conn(0x7f97700aabe0 msgr2=0x7f97701162d0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:44:24.578 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.a (rank=0) 2019-12-05T05:44:24.579 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2019-12-05T05:44:24.579 INFO:teuthology.orchestra.run:waiting for 300 2019-12-05T05:44:24.590 INFO:tasks.ceph.mds.a:Stopped 2019-12-05T05:44:24.590 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:44:24.774 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.760+0000 7f1dd13a8700 1 Processor -- start 2019-12-05T05:44:24.774 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.760+0000 7f1dd13a8700 1 -- start start 2019-12-05T05:44:24.774 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.760+0000 7f1dd13a8700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f1dcc0a7730 0x7f1dcc08df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:44:24.775 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.760+0000 7f1dd13a8700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1dcc0a0700 con 0x7f1dcc09fad0 2019-12-05T05:44:24.775 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.760+0000 7f1dd13a8700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1dcc0a9b90 con 0x7f1dcc07e710 2019-12-05T05:44:24.775 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.760+0000 7f1dd13a8700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1dcc0a0af0 con 0x7f1dcc0a7730 2019-12-05T05:44:24.775 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.760+0000 7f1dcaffd700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f1dcc07e710 0x7f1dcc09f2d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:40114/0 (socket says 172.21.15.42:40114) 2019-12-05T05:44:24.775 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.760+0000 7f1dcaffd700 1 -- 172.21.15.42:0/485146057 learned_addr learned my addr 172.21.15.42:0/485146057 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:44:24.776 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.760+0000 7f1dcb7fe700 1 --2- 172.21.15.42:0/485146057 >> v2:172.21.15.40:3300/0 conn(0x7f1dcc0a7730 0x7f1dcc08df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:44:24.776 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.760+0000 7f1dca7fc700 1 -- 172.21.15.42:0/485146057 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 179578272 0 0) 0x7f1dcc0a9b90 con 0x7f1dcc07e710 2019-12-05T05:44:24.781 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.760+0000 7f1dca7fc700 1 -- 172.21.15.42:0/485146057 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1db40030b0 con 0x7f1dcc07e710 2019-12-05T05:44:24.781 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.764+0000 7f1dca7fc700 1 -- 172.21.15.42:0/485146057 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3451716920 0 0) 0x7f1dcc0a0700 con 0x7f1dcc09fad0 2019-12-05T05:44:24.781 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.764+0000 7f1dca7fc700 1 -- 172.21.15.42:0/485146057 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1dcc0a9b90 con 0x7f1dcc09fad0 2019-12-05T05:44:24.781 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.764+0000 7f1dcb7fe700 1 -- 172.21.15.42:0/485146057 >> v1:172.21.15.42:6789/0 conn(0x7f1dcc07e710 legacy=0x7f1dcc09f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:24.781 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.764+0000 7f1dcb7fe700 1 -- 172.21.15.42:0/485146057 >> v1:172.21.15.40:6789/0 conn(0x7f1dcc09fad0 legacy=0x7f1dcc14efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:24.781 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.764+0000 7f1dcb7fe700 1 -- 172.21.15.42:0/485146057 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dcc07eb40 con 0x7f1dcc0a7730 2019-12-05T05:44:24.782 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.764+0000 7f1dcb7fe700 1 --2- 172.21.15.42:0/485146057 >> v2:172.21.15.40:3300/0 conn(0x7f1dcc0a7730 0x7f1dcc08df70 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f1dc0008790 tx=0x7f1dc000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:24.782 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.764+0000 7f1dcb7fe700 1 --2- 172.21.15.42:0/485146057 >> v2:172.21.15.40:3300/0 conn(0x7f1dcc0a7730 0x7f1dcc08df70 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f1dc0008790 tx=0x7f1dc000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:24.782 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.764+0000 7f1dcb7fe700 1 --2- 172.21.15.42:0/485146057 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1dcc0a7730 0x7f1dcc08df70 secure :-1 s=READY pgs=1069 cs=0 l=1 rx=0x7f1dc0008790 tx=0x7f1dc000fbf0).ready entity=mon.0 client_cookie=8c4cbba3d23584dd server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:44:24.782 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.764+0000 7f1dcb7fe700 1 --2- 172.21.15.42:0/485146057 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1dcc0a7730 0x7f1dcc08df70 secure :-1 s=THROTTLE_DONE pgs=1069 cs=0 l=1 rx=0x7f1dc0008790 tx=0x7f1dc000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:24.782 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.764+0000 7f1dca7fc700 1 -- 172.21.15.42:0/485146057 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1dc0011270 con 0x7f1dcc0a7730 2019-12-05T05:44:24.782 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.764+0000 7f1dcb7fe700 1 --2- 172.21.15.42:0/485146057 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1dcc0a7730 0x7f1dcc08df70 secure :-1 s=THROTTLE_DONE pgs=1069 cs=0 l=1 rx=0x7f1dc0008790 tx=0x7f1dc000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:24.782 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.764+0000 7f1dcb7fe700 1 --2- 172.21.15.42:0/485146057 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1dcc0a7730 0x7f1dcc08df70 secure :-1 s=THROTTLE_DONE pgs=1069 cs=0 l=1 rx=0x7f1dc0008790 tx=0x7f1dc000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:24.783 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.764+0000 7f1dca7fc700 1 -- 172.21.15.42:0/485146057 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1dc0008bc0 con 0x7f1dcc0a7730 2019-12-05T05:44:24.783 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.764+0000 7f1dca7fc700 1 -- 172.21.15.42:0/485146057 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1dc0011850 con 0x7f1dcc0a7730 2019-12-05T05:44:24.783 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.764+0000 7f1dd13a8700 1 -- 172.21.15.42:0/485146057 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1dcc0a7730 msgr2=0x7f1dcc08df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:24.783 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.764+0000 7f1dd13a8700 1 --2- 172.21.15.42:0/485146057 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1dcc0a7730 0x7f1dcc08df70 secure :-1 s=READY pgs=1069 cs=0 l=1 rx=0x7f1dc0008790 tx=0x7f1dc000fbf0).stop 2019-12-05T05:44:24.783 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.764+0000 7f1dd13a8700 1 -- 172.21.15.42:0/485146057 shutdown_connections 2019-12-05T05:44:24.783 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.764+0000 7f1dd13a8700 1 --2- 172.21.15.42:0/485146057 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1dcc0a7730 0x7f1dcc08df70 unknown :-1 s=CLOSED pgs=1069 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:44:24.783 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.764+0000 7f1dd13a8700 1 -- 172.21.15.42:0/485146057 shutdown_connections 2019-12-05T05:44:24.784 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.764+0000 7f1dd13a8700 1 -- 172.21.15.42:0/485146057 wait complete. 2019-12-05T05:44:24.784 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.764+0000 7f1dd13a8700 1 -- 172.21.15.42:0/485146057 >> 172.21.15.42:0/485146057 conn(0x7f1dcc0aabd0 msgr2=0x7f1dcc0a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:44:24.784 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.764+0000 7f1dd13a8700 1 Processor -- start 2019-12-05T05:44:24.784 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.764+0000 7f1dd13a8700 1 -- start start 2019-12-05T05:44:24.784 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.764+0000 7f1dd13a8700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f1dcc114cb0 0x7f1dcc113ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:44:24.784 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.764+0000 7f1dd13a8700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1dcc1131f0 0x7f1dcc113620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:44:24.784 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.764+0000 7f1dd13a8700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1dcc065b90 con 0x7f1dcc0a3850 2019-12-05T05:44:24.785 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.764+0000 7f1dd13a8700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1dcc1238b0 con 0x7f1dcc114cb0 2019-12-05T05:44:24.785 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.764+0000 7f1dd13a8700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1dcc1144d0 con 0x7f1dcc1131f0 2019-12-05T05:44:24.785 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.768+0000 7f1dcb7fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f1dcc114cb0 0x7f1dcc113ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:44:24.785 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.768+0000 7f1dcb7fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f1dcc114cb0 0x7f1dcc113ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:40082/0 (socket says 172.21.15.42:40082) 2019-12-05T05:44:24.785 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.768+0000 7f1dcb7fe700 1 -- 172.21.15.42:0/857869066 learned_addr learned my addr 172.21.15.42:0/857869066 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:44:24.785 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.768+0000 7f1dcbfff700 1 --2- 172.21.15.42:0/857869066 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1dcc1131f0 0x7f1dcc113620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:44:24.785 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.768+0000 7f1dcb7fe700 1 -- 172.21.15.42:0/857869066 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1dcc1131f0 msgr2=0x7f1dcc113620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:24.786 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.768+0000 7f1dcb7fe700 1 --2- 172.21.15.42:0/857869066 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1dcc1131f0 0x7f1dcc113620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:44:24.786 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.768+0000 7f1dcb7fe700 1 -- 172.21.15.42:0/857869066 >> v1:172.21.15.42:6789/0 conn(0x7f1dcc0a3850 legacy=0x7f1dcc1230b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:24.786 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.768+0000 7f1dcb7fe700 1 -- 172.21.15.42:0/857869066 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dcc05f280 con 0x7f1dcc114cb0 2019-12-05T05:44:24.786 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.768+0000 7f1dcb7fe700 1 --2- 172.21.15.42:0/857869066 >> v2:172.21.15.42:3300/0 conn(0x7f1dcc114cb0 0x7f1dcc113ed0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f1dcc114690 tx=0x7f1dc0008ca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:24.786 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.768+0000 7f1dcb7fe700 1 --2- 172.21.15.42:0/857869066 >> v2:172.21.15.42:3300/0 conn(0x7f1dcc114cb0 0x7f1dcc113ed0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f1dcc114690 tx=0x7f1dc0008ca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:24.787 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.768+0000 7f1dcb7fe700 1 --2- 172.21.15.42:0/857869066 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1dcc114cb0 0x7f1dcc113ed0 secure :-1 s=READY pgs=1377 cs=0 l=1 rx=0x7f1dcc114690 tx=0x7f1dc0008ca0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:44:24.787 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.768+0000 7f1dcb7fe700 1 --2- 172.21.15.42:0/857869066 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1dcc114cb0 0x7f1dcc113ed0 secure :-1 s=THROTTLE_DONE pgs=1377 cs=0 l=1 rx=0x7f1dcc114690 tx=0x7f1dc0008ca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:24.787 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.768+0000 7f1db3fff700 1 -- 172.21.15.42:0/857869066 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1dc0007150 con 0x7f1dcc114cb0 2019-12-05T05:44:24.787 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.768+0000 7f1dcb7fe700 1 --2- 172.21.15.42:0/857869066 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1dcc114cb0 0x7f1dcc113ed0 secure :-1 s=THROTTLE_DONE pgs=1377 cs=0 l=1 rx=0x7f1dcc114690 tx=0x7f1dc0008ca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:24.787 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.768+0000 7f1dcb7fe700 1 --2- 172.21.15.42:0/857869066 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1dcc114cb0 0x7f1dcc113ed0 secure :-1 s=THROTTLE_DONE pgs=1377 cs=0 l=1 rx=0x7f1dcc114690 tx=0x7f1dc0008ca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:24.787 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.768+0000 7f1db3fff700 1 -- 172.21.15.42:0/857869066 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1dc0011470 con 0x7f1dcc114cb0 2019-12-05T05:44:24.788 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.768+0000 7f1dd13a8700 1 -- 172.21.15.42:0/857869066 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1dcc12dac0 con 0x7f1dcc114cb0 2019-12-05T05:44:24.788 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.768+0000 7f1db3fff700 1 -- 172.21.15.42:0/857869066 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1dc000f550 con 0x7f1dcc114cb0 2019-12-05T05:44:24.788 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.768+0000 7f1dd13a8700 1 -- 172.21.15.42:0/857869066 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1dcc12dff0 con 0x7f1dcc114cb0 2019-12-05T05:44:24.788 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.768+0000 7f1dcb7fe700 1 --2- 172.21.15.42:0/857869066 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1dcc114cb0 0x7f1dcc113ed0 secure :-1 s=THROTTLE_DONE pgs=1377 cs=0 l=1 rx=0x7f1dcc114690 tx=0x7f1dc0008ca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:24.790 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.768+0000 7f1db3fff700 1 -- 172.21.15.42:0/857869066 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f1dc0005540 con 0x7f1dcc114cb0 2019-12-05T05:44:24.790 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.768+0000 7f1dcb7fe700 1 --2- 172.21.15.42:0/857869066 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1dcc114cb0 0x7f1dcc113ed0 secure :-1 s=THROTTLE_DONE pgs=1377 cs=0 l=1 rx=0x7f1dcc114690 tx=0x7f1dc0008ca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:24.790 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.772+0000 7f1db3fff700 1 --2- 172.21.15.42:0/857869066 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f1dac03af00 0x7f1dac03d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:44:24.790 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.772+0000 7f1db3fff700 1 -- 172.21.15.42:0/857869066 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 8353+0+0 (secure 0 0 0) 0x7f1dc004b950 con 0x7f1dcc114cb0 2019-12-05T05:44:24.790 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.772+0000 7f1dcaffd700 1 --2- 172.21.15.42:0/857869066 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f1dac03af00 0x7f1dac03d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:44:24.791 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.772+0000 7f1dcaffd700 1 --2- 172.21.15.42:0/857869066 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f1dac03af00 0x7f1dac03d410 crc :-1 s=READY pgs=593 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:44:24.791 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.776+0000 7f1dd13a8700 1 -- 172.21.15.42:0/857869066 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1dcc05d100 con 0x7f1dcc114cb0 2019-12-05T05:44:24.791 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.784+0000 7f1dcb7fe700 1 --2- 172.21.15.42:0/857869066 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1dcc114cb0 0x7f1dcc113ed0 secure :-1 s=THROTTLE_DONE pgs=1377 cs=0 l=1 rx=0x7f1dcc114690 tx=0x7f1dc0008ca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:24.791 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.784+0000 7f1db3fff700 1 -- 172.21.15.42:0/857869066 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f1dc00337e0 con 0x7f1dcc114cb0 2019-12-05T05:44:24.962 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.956+0000 7f1dd13a8700 1 -- 172.21.15.42:0/857869066 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1dcc05b8b0 con 0x7f1dcc114cb0 2019-12-05T05:44:24.963 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.960+0000 7f1dcb7fe700 1 --2- 172.21.15.42:0/857869066 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1dcc114cb0 0x7f1dcc113ed0 secure :-1 s=THROTTLE_DONE pgs=1377 cs=0 l=1 rx=0x7f1dcc114690 tx=0x7f1dc0008ca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:24.964 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.960+0000 7f1db3fff700 1 -- 172.21.15.42:0/857869066 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 168 v168) v1 ==== 95+0+2109 (secure 0 0 0) 0x7f1dc0012bb0 con 0x7f1dcc114cb0 2019-12-05T05:44:24.964 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 168 2019-12-05T05:44:24.964 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:44:24.964 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":168,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7884,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/1127133319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1127133319},{"type":"v1","addr":"172.21.15.40:6833","nonce":1127133319}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":168}],"filesystems":[{"mdsmap":{"epoch":167,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:44:09.069915+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":76,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7755},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7755":{"gid":7755,"name":"a","rank":0,"incarnation":164,"state":"up:active","state_seq":22,"addr":"172.21.15.42:6835/358497296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":358497296},{"type":"v1","addr":"172.21.15.42:6835","nonce":358497296}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:44:24.967 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.960+0000 7f1dd13a8700 1 -- 172.21.15.42:0/857869066 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f1dac03af00 msgr2=0x7f1dac03d410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:24.967 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.960+0000 7f1dd13a8700 1 --2- 172.21.15.42:0/857869066 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f1dac03af00 0x7f1dac03d410 crc :-1 s=READY pgs=593 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:44:24.967 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.960+0000 7f1dd13a8700 1 -- 172.21.15.42:0/857869066 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1dcc114cb0 msgr2=0x7f1dcc113ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:24.967 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.960+0000 7f1dd13a8700 1 --2- 172.21.15.42:0/857869066 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1dcc114cb0 0x7f1dcc113ed0 secure :-1 s=READY pgs=1377 cs=0 l=1 rx=0x7f1dcc114690 tx=0x7f1dc0008ca0).stop 2019-12-05T05:44:24.968 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.964+0000 7f1dd13a8700 1 -- 172.21.15.42:0/857869066 shutdown_connections 2019-12-05T05:44:24.968 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.964+0000 7f1dd13a8700 1 --2- 172.21.15.42:0/857869066 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f1dac03af00 0x7f1dac03d410 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:44:24.968 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.964+0000 7f1dd13a8700 1 --2- 172.21.15.42:0/857869066 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1dcc1131f0 0x7f1dcc113620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:44:24.969 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.964+0000 7f1dd13a8700 1 --2- 172.21.15.42:0/857869066 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1dcc114cb0 0x7f1dcc113ed0 unknown :-1 s=CLOSED pgs=1377 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:44:24.969 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.964+0000 7f1dd13a8700 1 -- 172.21.15.42:0/857869066 shutdown_connections 2019-12-05T05:44:24.969 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.964+0000 7f1dd13a8700 1 -- 172.21.15.42:0/857869066 wait complete. 2019-12-05T05:44:24.969 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:24.964+0000 7f1dd13a8700 1 -- 172.21.15.42:0/857869066 >> 172.21.15.42:0/857869066 conn(0x7f1dcc0aabd0 msgr2=0x7f1dcc09f5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:44:24.982 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T05:44:26.985 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:44:27.092 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:27.080+0000 7fa75bb68700 1 Processor -- start 2019-12-05T05:44:27.092 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:27.080+0000 7fa75bb68700 1 -- start start 2019-12-05T05:44:27.093 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:27.084+0000 7fa75bb68700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa75409fae0 0x7fa75414fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:44:27.093 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:27.084+0000 7fa75bb68700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7540a0710 con 0x7fa75408df80 2019-12-05T05:44:27.093 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:27.084+0000 7fa75bb68700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa75407ef20 con 0x7fa7540a7740 2019-12-05T05:44:27.093 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:27.084+0000 7fa75bb68700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7540a0b00 con 0x7fa75409fae0 2019-12-05T05:44:27.093 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:27.084+0000 7fa75a105700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fa7540a7740 0x7fa75407e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:40134/0 (socket says 172.21.15.42:40134) 2019-12-05T05:44:27.093 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:27.084+0000 7fa75a105700 1 -- 172.21.15.42:0/3461008079 learned_addr learned my addr 172.21.15.42:0/3461008079 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:44:27.094 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:27.084+0000 7fa75a906700 1 --2- 172.21.15.42:0/3461008079 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa75409fae0 0x7fa75414fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:44:27.094 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:27.084+0000 7fa759103700 1 -- 172.21.15.42:0/3461008079 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 185402327 0 0) 0x7fa75407ef20 con 0x7fa7540a7740 2019-12-05T05:44:27.094 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:27.084+0000 7fa759103700 1 -- 172.21.15.42:0/3461008079 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7400030b0 con 0x7fa7540a7740 2019-12-05T05:44:27.094 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:27.084+0000 7fa759103700 1 -- 172.21.15.42:0/3461008079 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 269383715 0 0) 0x7fa7540a0710 con 0x7fa75408df80 2019-12-05T05:44:27.094 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:27.084+0000 7fa75a906700 1 -- 172.21.15.42:0/3461008079 >> v1:172.21.15.42:6789/0 conn(0x7fa7540a7740 legacy=0x7fa75407e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:27.094 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:27.084+0000 7fa75a906700 1 -- 172.21.15.42:0/3461008079 >> v1:172.21.15.40:6789/0 conn(0x7fa75408df80 legacy=0x7fa75409f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:27.094 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:27.084+0000 7fa75a906700 1 -- 172.21.15.42:0/3461008079 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7540a9ba0 con 0x7fa75409fae0 2019-12-05T05:44:27.095 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:27.084+0000 7fa75a906700 1 --2- 172.21.15.42:0/3461008079 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa75409fae0 0x7fa75414fff0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fa74c0032b0 tx=0x7fa74c00ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:27.095 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:27.084+0000 7fa75a906700 1 --2- 172.21.15.42:0/3461008079 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa75409fae0 0x7fa75414fff0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fa74c0032b0 tx=0x7fa74c00ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:27.095 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:27.084+0000 7fa75a906700 1 --2- 172.21.15.42:0/3461008079 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa75409fae0 0x7fa75414fff0 secure :-1 s=READY pgs=293 cs=0 l=1 rx=0x7fa74c0032b0 tx=0x7fa74c00ff70).ready entity=mon.2 client_cookie=3f286a9a703ca43e server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:44:27.095 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:27.084+0000 7fa75a906700 1 --2- 172.21.15.42:0/3461008079 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa75409fae0 0x7fa75414fff0 secure :-1 s=THROTTLE_DONE pgs=293 cs=0 l=1 rx=0x7fa74c0032b0 tx=0x7fa74c00ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:27.095 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:27.084+0000 7fa759103700 1 -- 172.21.15.42:0/3461008079 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa74c0109b0 con 0x7fa75409fae0 2019-12-05T05:44:27.095 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:27.084+0000 7fa75a906700 1 --2- 172.21.15.42:0/3461008079 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa75409fae0 0x7fa75414fff0 secure :-1 s=THROTTLE_DONE pgs=293 cs=0 l=1 rx=0x7fa74c0032b0 tx=0x7fa74c00ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:27.095 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:27.084+0000 7fa75a906700 1 --2- 172.21.15.42:0/3461008079 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa75409fae0 0x7fa75414fff0 secure :-1 s=THROTTLE_DONE pgs=293 cs=0 l=1 rx=0x7fa74c0032b0 tx=0x7fa74c00ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:27.096 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:27.084+0000 7fa759103700 1 -- 172.21.15.42:0/3461008079 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa74c008b50 con 0x7fa75409fae0 2019-12-05T05:44:27.096 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:27.084+0000 7fa759103700 1 -- 172.21.15.42:0/3461008079 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa74c0110c0 con 0x7fa75409fae0 2019-12-05T05:44:27.096 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:27.084+0000 7fa75bb68700 1 -- 172.21.15.42:0/3461008079 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa75409fae0 msgr2=0x7fa75414fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:27.096 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:27.084+0000 7fa75bb68700 1 --2- 172.21.15.42:0/3461008079 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa75409fae0 0x7fa75414fff0 secure :-1 s=READY pgs=293 cs=0 l=1 rx=0x7fa74c0032b0 tx=0x7fa74c00ff70).stop 2019-12-05T05:44:27.096 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:27.084+0000 7fa75bb68700 1 -- 172.21.15.42:0/3461008079 shutdown_connections 2019-12-05T05:44:27.096 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:27.084+0000 7fa75bb68700 1 --2- 172.21.15.42:0/3461008079 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa75409fae0 0x7fa75414fff0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:44:27.096 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:27.084+0000 7fa75bb68700 1 -- 172.21.15.42:0/3461008079 shutdown_connections 2019-12-05T05:44:27.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:27.084+0000 7fa75bb68700 1 -- 172.21.15.42:0/3461008079 wait complete. 2019-12-05T05:44:27.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:27.084+0000 7fa75bb68700 1 -- 172.21.15.42:0/3461008079 >> 172.21.15.42:0/3461008079 conn(0x7fa7540aabe0 msgr2=0x7fa7540a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:44:27.103 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:27.088+0000 7fa75bb68700 1 Processor -- start 2019-12-05T05:44:27.103 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:27.088+0000 7fa75bb68700 1 -- start start 2019-12-05T05:44:27.103 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:27.092+0000 7fa75bb68700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa754144b50 0x7fa754115780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:44:27.103 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:27.092+0000 7fa75bb68700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fa7541123f0 0x7fa754112820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:44:27.104 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:27.092+0000 7fa75bb68700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa754114430 con 0x7fa75408eb70 2019-12-05T05:44:27.104 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:27.092+0000 7fa75bb68700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7541139b0 con 0x7fa7541123f0 2019-12-05T05:44:27.104 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:27.092+0000 7fa75bb68700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa754113c50 con 0x7fa754144b50 2019-12-05T05:44:27.104 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:27.092+0000 7fa759904700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fa7541123f0 0x7fa754112820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:44:27.104 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:27.092+0000 7fa759904700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fa7541123f0 0x7fa754112820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:40104/0 (socket says 172.21.15.42:40104) 2019-12-05T05:44:27.104 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:27.092+0000 7fa759904700 1 -- 172.21.15.42:0/1208438137 learned_addr learned my addr 172.21.15.42:0/1208438137 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:44:27.105 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:27.092+0000 7fa75a105700 1 --2- 172.21.15.42:0/1208438137 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa754144b50 0x7fa754115780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:44:27.105 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:27.092+0000 7fa759904700 1 -- 172.21.15.42:0/1208438137 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa754144b50 msgr2=0x7fa754115780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:27.105 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:27.092+0000 7fa759904700 1 --2- 172.21.15.42:0/1208438137 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa754144b50 0x7fa754115780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:44:27.105 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:27.092+0000 7fa759904700 1 -- 172.21.15.42:0/1208438137 >> v1:172.21.15.42:6789/0 conn(0x7fa75408eb70 legacy=0x7fa75408efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:27.105 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:27.092+0000 7fa759904700 1 -- 172.21.15.42:0/1208438137 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa754150140 con 0x7fa7541123f0 2019-12-05T05:44:27.105 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:27.092+0000 7fa759904700 1 --2- 172.21.15.42:0/1208438137 >> v2:172.21.15.42:3300/0 conn(0x7fa7541123f0 0x7fa754112820 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fa748000b80 tx=0x7fa7480109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:27.105 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:27.092+0000 7fa759904700 1 --2- 172.21.15.42:0/1208438137 >> v2:172.21.15.42:3300/0 conn(0x7fa7541123f0 0x7fa754112820 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fa748000b80 tx=0x7fa7480109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:27.106 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:27.092+0000 7fa759904700 1 --2- 172.21.15.42:0/1208438137 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa7541123f0 0x7fa754112820 secure :-1 s=READY pgs=1380 cs=0 l=1 rx=0x7fa748000b80 tx=0x7fa7480109a0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:44:27.106 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:27.092+0000 7fa759904700 1 --2- 172.21.15.42:0/1208438137 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa7541123f0 0x7fa754112820 secure :-1 s=THROTTLE_DONE pgs=1380 cs=0 l=1 rx=0x7fa748000b80 tx=0x7fa7480109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:27.106 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:27.092+0000 7fa759904700 1 --2- 172.21.15.42:0/1208438137 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa7541123f0 0x7fa754112820 secure :-1 s=THROTTLE_DONE pgs=1380 cs=0 l=1 rx=0x7fa748000b80 tx=0x7fa7480109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:27.106 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:27.092+0000 7fa746ffd700 1 -- 172.21.15.42:0/1208438137 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa74800caa0 con 0x7fa7541123f0 2019-12-05T05:44:27.106 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:27.092+0000 7fa759904700 1 --2- 172.21.15.42:0/1208438137 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa7541123f0 0x7fa754112820 secure :-1 s=THROTTLE_DONE pgs=1380 cs=0 l=1 rx=0x7fa748000b80 tx=0x7fa7480109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:27.106 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:27.092+0000 7fa746ffd700 1 -- 172.21.15.42:0/1208438137 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa74800c6c0 con 0x7fa7541123f0 2019-12-05T05:44:27.106 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:27.092+0000 7fa75bb68700 1 -- 172.21.15.42:0/1208438137 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa75408f7a0 con 0x7fa7541123f0 2019-12-05T05:44:27.107 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:27.092+0000 7fa746ffd700 1 -- 172.21.15.42:0/1208438137 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa748011160 con 0x7fa7541123f0 2019-12-05T05:44:27.107 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:27.092+0000 7fa75bb68700 1 -- 172.21.15.42:0/1208438137 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa75412e2a0 con 0x7fa7541123f0 2019-12-05T05:44:27.107 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:27.092+0000 7fa759904700 1 --2- 172.21.15.42:0/1208438137 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa7541123f0 0x7fa754112820 secure :-1 s=THROTTLE_DONE pgs=1380 cs=0 l=1 rx=0x7fa748000b80 tx=0x7fa7480109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:27.107 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:27.092+0000 7fa746ffd700 1 -- 172.21.15.42:0/1208438137 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fa748007430 con 0x7fa7541123f0 2019-12-05T05:44:27.107 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:27.092+0000 7fa759904700 1 --2- 172.21.15.42:0/1208438137 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa7541123f0 0x7fa754112820 secure :-1 s=THROTTLE_DONE pgs=1380 cs=0 l=1 rx=0x7fa748000b80 tx=0x7fa7480109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:27.107 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:27.092+0000 7fa746ffd700 1 --2- 172.21.15.42:0/1208438137 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa73c03ae40 0x7fa73c03d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:44:27.107 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:27.096+0000 7fa746ffd700 1 -- 172.21.15.42:0/1208438137 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 8353+0+0 (secure 0 0 0) 0x7fa74804b170 con 0x7fa7541123f0 2019-12-05T05:44:27.108 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:27.096+0000 7fa75a105700 1 --2- 172.21.15.42:0/1208438137 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa73c03ae40 0x7fa73c03d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:44:27.108 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:27.096+0000 7fa75a105700 1 --2- 172.21.15.42:0/1208438137 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa73c03ae40 0x7fa73c03d350 crc :-1 s=READY pgs=594 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:44:27.108 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:27.096+0000 7fa75bb68700 1 -- 172.21.15.42:0/1208438137 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa75405d110 con 0x7fa7541123f0 2019-12-05T05:44:27.109 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:27.100+0000 7fa759904700 1 --2- 172.21.15.42:0/1208438137 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa7541123f0 0x7fa754112820 secure :-1 s=THROTTLE_DONE pgs=1380 cs=0 l=1 rx=0x7fa748000b80 tx=0x7fa7480109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:27.109 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:27.100+0000 7fa746ffd700 1 -- 172.21.15.42:0/1208438137 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fa7480321f0 con 0x7fa7541123f0 2019-12-05T05:44:27.272 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:27.268+0000 7fa75bb68700 1 -- 172.21.15.42:0/1208438137 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa75405b8c0 con 0x7fa7541123f0 2019-12-05T05:44:27.272 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:27.268+0000 7fa759904700 1 --2- 172.21.15.42:0/1208438137 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa7541123f0 0x7fa754112820 secure :-1 s=THROTTLE_DONE pgs=1380 cs=0 l=1 rx=0x7fa748000b80 tx=0x7fa7480109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:27.273 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:27.268+0000 7fa746ffd700 1 -- 172.21.15.42:0/1208438137 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 168 v168) v1 ==== 95+0+2109 (secure 0 0 0) 0x7fa748032450 con 0x7fa7541123f0 2019-12-05T05:44:27.275 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 168 2019-12-05T05:44:27.275 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:44:27.275 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":168,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7884,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/1127133319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1127133319},{"type":"v1","addr":"172.21.15.40:6833","nonce":1127133319}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":168}],"filesystems":[{"mdsmap":{"epoch":167,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:44:09.069915+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":76,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7755},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7755":{"gid":7755,"name":"a","rank":0,"incarnation":164,"state":"up:active","state_seq":22,"addr":"172.21.15.42:6835/358497296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":358497296},{"type":"v1","addr":"172.21.15.42:6835","nonce":358497296}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:44:27.280 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:27.272+0000 7fa75bb68700 1 -- 172.21.15.42:0/1208438137 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa73c03ae40 msgr2=0x7fa73c03d350 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:27.280 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:27.272+0000 7fa75bb68700 1 --2- 172.21.15.42:0/1208438137 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa73c03ae40 0x7fa73c03d350 crc :-1 s=READY pgs=594 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:44:27.281 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:27.272+0000 7fa75bb68700 1 -- 172.21.15.42:0/1208438137 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa7541123f0 msgr2=0x7fa754112820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:27.281 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:27.272+0000 7fa75bb68700 1 --2- 172.21.15.42:0/1208438137 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa7541123f0 0x7fa754112820 secure :-1 s=READY pgs=1380 cs=0 l=1 rx=0x7fa748000b80 tx=0x7fa7480109a0).stop 2019-12-05T05:44:27.281 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:27.272+0000 7fa75bb68700 1 -- 172.21.15.42:0/1208438137 shutdown_connections 2019-12-05T05:44:27.281 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:27.272+0000 7fa75bb68700 1 --2- 172.21.15.42:0/1208438137 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa73c03ae40 0x7fa73c03d350 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:44:27.281 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:27.272+0000 7fa75bb68700 1 --2- 172.21.15.42:0/1208438137 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa754144b50 0x7fa754115780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:44:27.282 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:27.272+0000 7fa75bb68700 1 --2- 172.21.15.42:0/1208438137 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa7541123f0 0x7fa754112820 unknown :-1 s=CLOSED pgs=1380 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:44:27.282 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:27.272+0000 7fa75bb68700 1 -- 172.21.15.42:0/1208438137 shutdown_connections 2019-12-05T05:44:27.282 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:27.272+0000 7fa75bb68700 1 -- 172.21.15.42:0/1208438137 wait complete. 2019-12-05T05:44:27.282 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:27.272+0000 7fa75bb68700 1 -- 172.21.15.42:0/1208438137 >> 172.21.15.42:0/1208438137 conn(0x7fa75409f9b0 msgr2=0x7fa75407e670 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:44:27.292 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T05:44:29.293 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:44:29.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:29.396+0000 7f8ff1517700 1 Processor -- start 2019-12-05T05:44:29.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:29.396+0000 7f8ff1517700 1 -- start start 2019-12-05T05:44:29.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:29.396+0000 7f8ff1517700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8fec0a7740 0x7f8fec08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:44:29.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:29.396+0000 7f8ff1517700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f8fec09fae0 0x7f8fec07eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:44:29.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:29.396+0000 7f8ff1517700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fec0a0710 con 0x7f8fec07e720 2019-12-05T05:44:29.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:29.396+0000 7f8ff1517700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fec0a0b00 con 0x7f8fec0a7740 2019-12-05T05:44:29.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:29.396+0000 7f8ff1517700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fec08e5b0 con 0x7f8fec09fae0 2019-12-05T05:44:29.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:29.396+0000 7f8febfff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f8fec09fae0 0x7f8fec07eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:44:29.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:29.396+0000 7f8febfff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f8fec09fae0 0x7f8fec07eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:40116/0 (socket says 172.21.15.42:40116) 2019-12-05T05:44:29.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:29.396+0000 7f8febfff700 1 -- 172.21.15.42:0/1519396532 learned_addr learned my addr 172.21.15.42:0/1519396532 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:44:29.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:29.396+0000 7f8feb7fe700 1 --2- 172.21.15.42:0/1519396532 >> v2:172.21.15.40:3300/0 conn(0x7f8fec0a7740 0x7f8fec08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:44:29.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:29.396+0000 7f8febfff700 1 -- 172.21.15.42:0/1519396532 >> v2:172.21.15.40:3300/0 conn(0x7f8fec0a7740 msgr2=0x7f8fec08df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:44:29.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:29.396+0000 7f8febfff700 1 --2- 172.21.15.42:0/1519396532 >> v2:172.21.15.40:3300/0 conn(0x7f8fec0a7740 0x7f8fec08df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:44:29.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:29.396+0000 7f8febfff700 1 -- 172.21.15.42:0/1519396532 >> v1:172.21.15.40:6789/0 conn(0x7f8fec07e720 legacy=0x7f8fec09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:44:29.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:29.396+0000 7f8febfff700 1 -- 172.21.15.42:0/1519396532 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fec07f0a0 con 0x7f8fec09fae0 2019-12-05T05:44:29.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:29.396+0000 7f8feb7fe700 1 --2- 172.21.15.42:0/1519396532 >> v2:172.21.15.40:3300/0 conn(0x7f8fec0a7740 0x7f8fec08df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T05:44:29.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:29.396+0000 7f8febfff700 1 --2- 172.21.15.42:0/1519396532 >> v2:172.21.15.42:3300/0 conn(0x7f8fec09fae0 0x7f8fec07eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f8fe4002f30 tx=0x7f8fe400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:29.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:29.400+0000 7f8febfff700 1 --2- 172.21.15.42:0/1519396532 >> v2:172.21.15.42:3300/0 conn(0x7f8fec09fae0 0x7f8fec07eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f8fe4002f30 tx=0x7f8fe400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:29.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:29.400+0000 7f8febfff700 1 --2- 172.21.15.42:0/1519396532 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8fec09fae0 0x7f8fec07eb50 secure :-1 s=READY pgs=1381 cs=0 l=1 rx=0x7f8fe4002f30 tx=0x7f8fe400fd30).ready entity=mon.1 client_cookie=2f31d9412695c37f server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:44:29.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:29.400+0000 7f8febfff700 1 --2- 172.21.15.42:0/1519396532 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8fec09fae0 0x7f8fec07eb50 secure :-1 s=THROTTLE_DONE pgs=1381 cs=0 l=1 rx=0x7f8fe4002f30 tx=0x7f8fe400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:29.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:29.400+0000 7f8fea7fc700 1 -- 172.21.15.42:0/1519396532 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8fe4008c90 con 0x7f8fec09fae0 2019-12-05T05:44:29.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:29.400+0000 7f8febfff700 1 --2- 172.21.15.42:0/1519396532 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8fec09fae0 0x7f8fec07eb50 secure :-1 s=THROTTLE_DONE pgs=1381 cs=0 l=1 rx=0x7f8fe4002f30 tx=0x7f8fe400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:29.411 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:29.400+0000 7f8febfff700 1 --2- 172.21.15.42:0/1519396532 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8fec09fae0 0x7f8fec07eb50 secure :-1 s=THROTTLE_DONE pgs=1381 cs=0 l=1 rx=0x7f8fe4002f30 tx=0x7f8fe400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:29.411 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:29.400+0000 7f8fea7fc700 1 -- 172.21.15.42:0/1519396532 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fe40088b0 con 0x7f8fec09fae0 2019-12-05T05:44:29.411 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:29.400+0000 7f8fea7fc700 1 -- 172.21.15.42:0/1519396532 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8fe4011ad0 con 0x7f8fec09fae0 2019-12-05T05:44:29.411 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:29.400+0000 7f8ff1517700 1 -- 172.21.15.42:0/1519396532 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8fec09fae0 msgr2=0x7f8fec07eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:29.411 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:29.400+0000 7f8ff1517700 1 --2- 172.21.15.42:0/1519396532 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8fec09fae0 0x7f8fec07eb50 secure :-1 s=READY pgs=1381 cs=0 l=1 rx=0x7f8fe4002f30 tx=0x7f8fe400fd30).stop 2019-12-05T05:44:29.412 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:29.400+0000 7f8ff1517700 1 -- 172.21.15.42:0/1519396532 shutdown_connections 2019-12-05T05:44:29.412 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:29.400+0000 7f8ff1517700 1 --2- 172.21.15.42:0/1519396532 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8fec09fae0 0x7f8fec07eb50 unknown :-1 s=CLOSED pgs=1381 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:44:29.412 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:29.400+0000 7f8ff1517700 1 --2- 172.21.15.42:0/1519396532 >> v2:172.21.15.40:3300/0 conn(0x7f8fec0a7740 0x7f8fec08df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:44:29.412 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:29.400+0000 7f8ff1517700 1 -- 172.21.15.42:0/1519396532 shutdown_connections 2019-12-05T05:44:29.412 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:29.400+0000 7f8ff1517700 1 -- 172.21.15.42:0/1519396532 wait complete. 2019-12-05T05:44:29.412 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:29.400+0000 7f8ff1517700 1 -- 172.21.15.42:0/1519396532 >> 172.21.15.42:0/1519396532 conn(0x7f8fec0aabe0 msgr2=0x7f8fec0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:44:29.413 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:29.400+0000 7f8ff1517700 1 Processor -- start 2019-12-05T05:44:29.416 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:29.400+0000 7f8ff1517700 1 -- start start 2019-12-05T05:44:29.416 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:29.404+0000 7f8ff1517700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8fec116780 0x7f8fec1144e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:44:29.416 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:29.404+0000 7f8ff1517700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8fec0a6360 0x7f8fec07d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:44:29.416 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:29.404+0000 7f8ff1517700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fec111e80 con 0x7f8fec114a30 2019-12-05T05:44:29.417 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:29.404+0000 7f8ff1517700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fec112190 con 0x7f8fec0a6360 2019-12-05T05:44:29.417 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:29.404+0000 7f8ff1517700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fec112430 con 0x7f8fec116780 2019-12-05T05:44:29.417 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:29.404+0000 7f8feaffd700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f8fec114a30 0x7f8fec0a5b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:40160/0 (socket says 172.21.15.42:40160) 2019-12-05T05:44:29.417 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:29.404+0000 7f8feaffd700 1 -- 172.21.15.42:0/603457777 learned_addr learned my addr 172.21.15.42:0/603457777 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:44:29.417 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:29.404+0000 7f8feb7fe700 1 --2- 172.21.15.42:0/603457777 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8fec116780 0x7f8fec1144e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:44:29.417 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:29.404+0000 7f8febfff700 1 --2- 172.21.15.42:0/603457777 >> v2:172.21.15.40:3300/0 conn(0x7f8fec0a6360 0x7f8fec07d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:44:29.417 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:29.404+0000 7f8fe3fff700 1 -- 172.21.15.42:0/603457777 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4146544669 0 0) 0x7f8fec111e80 con 0x7f8fec114a30 2019-12-05T05:44:29.418 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:29.404+0000 7f8fe3fff700 1 -- 172.21.15.42:0/603457777 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8fd00030b0 con 0x7f8fec114a30 2019-12-05T05:44:29.418 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:29.404+0000 7f8feb7fe700 1 -- 172.21.15.42:0/603457777 >> v2:172.21.15.40:3300/0 conn(0x7f8fec0a6360 msgr2=0x7f8fec07d390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:29.418 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:29.404+0000 7f8feb7fe700 1 --2- 172.21.15.42:0/603457777 >> v2:172.21.15.40:3300/0 conn(0x7f8fec0a6360 0x7f8fec07d390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:44:29.418 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:29.404+0000 7f8feb7fe700 1 -- 172.21.15.42:0/603457777 >> v1:172.21.15.42:6789/0 conn(0x7f8fec114a30 legacy=0x7f8fec0a5b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:29.418 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:29.404+0000 7f8feb7fe700 1 -- 172.21.15.42:0/603457777 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fec07d8e0 con 0x7f8fec116780 2019-12-05T05:44:29.418 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:29.404+0000 7f8feb7fe700 1 --2- 172.21.15.42:0/603457777 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8fec116780 0x7f8fec1144e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f8fdc002f30 tx=0x7f8fdc0108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:29.418 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:29.404+0000 7f8feb7fe700 1 --2- 172.21.15.42:0/603457777 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8fec116780 0x7f8fec1144e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f8fdc002f30 tx=0x7f8fdc0108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:29.418 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:29.404+0000 7f8feb7fe700 1 --2- 172.21.15.42:0/603457777 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8fec116780 0x7f8fec1144e0 secure :-1 s=READY pgs=294 cs=0 l=1 rx=0x7f8fdc002f30 tx=0x7f8fdc0108f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:44:29.419 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:29.404+0000 7f8feb7fe700 1 --2- 172.21.15.42:0/603457777 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8fec116780 0x7f8fec1144e0 secure :-1 s=THROTTLE_DONE pgs=294 cs=0 l=1 rx=0x7f8fdc002f30 tx=0x7f8fdc0108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:29.419 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:29.404+0000 7f8fe3fff700 1 -- 172.21.15.42:0/603457777 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8fdc005930 con 0x7f8fec116780 2019-12-05T05:44:29.419 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:29.404+0000 7f8feb7fe700 1 --2- 172.21.15.42:0/603457777 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8fec116780 0x7f8fec1144e0 secure :-1 s=THROTTLE_DONE pgs=294 cs=0 l=1 rx=0x7f8fdc002f30 tx=0x7f8fdc0108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:29.419 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:29.404+0000 7f8feb7fe700 1 --2- 172.21.15.42:0/603457777 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8fec116780 0x7f8fec1144e0 secure :-1 s=THROTTLE_DONE pgs=294 cs=0 l=1 rx=0x7f8fdc002f30 tx=0x7f8fdc0108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:29.419 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:29.404+0000 7f8fe3fff700 1 -- 172.21.15.42:0/603457777 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fdc007160 con 0x7f8fec116780 2019-12-05T05:44:29.419 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:29.404+0000 7f8ff1517700 1 -- 172.21.15.42:0/603457777 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fec07db50 con 0x7f8fec116780 2019-12-05T05:44:29.419 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:29.404+0000 7f8fe3fff700 1 -- 172.21.15.42:0/603457777 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8fdc0060c0 con 0x7f8fec116780 2019-12-05T05:44:29.420 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:29.404+0000 7f8ff1517700 1 -- 172.21.15.42:0/603457777 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fec07e100 con 0x7f8fec116780 2019-12-05T05:44:29.420 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:29.404+0000 7f8feb7fe700 1 --2- 172.21.15.42:0/603457777 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8fec116780 0x7f8fec1144e0 secure :-1 s=THROTTLE_DONE pgs=294 cs=0 l=1 rx=0x7f8fdc002f30 tx=0x7f8fdc0108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:29.420 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:29.408+0000 7f8fe3fff700 1 -- 172.21.15.42:0/603457777 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f8fdc007b30 con 0x7f8fec116780 2019-12-05T05:44:29.420 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:29.408+0000 7f8feb7fe700 1 --2- 172.21.15.42:0/603457777 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8fec116780 0x7f8fec1144e0 secure :-1 s=THROTTLE_DONE pgs=294 cs=0 l=1 rx=0x7f8fdc002f30 tx=0x7f8fdc0108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:29.420 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:29.408+0000 7f8fe3fff700 1 --2- 172.21.15.42:0/603457777 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8fd003b260 0x7f8fd003d770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:44:29.420 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:29.408+0000 7f8fe3fff700 1 -- 172.21.15.42:0/603457777 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 8353+0+0 (secure 0 0 0) 0x7f8fdc04ac10 con 0x7f8fec116780 2019-12-05T05:44:29.420 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:29.408+0000 7f8feaffd700 1 --2- 172.21.15.42:0/603457777 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8fd003b260 0x7f8fd003d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:44:29.421 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:29.408+0000 7f8feaffd700 1 --2- 172.21.15.42:0/603457777 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8fd003b260 0x7f8fd003d770 crc :-1 s=READY pgs=595 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:44:29.421 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:29.412+0000 7f8ff1517700 1 -- 172.21.15.42:0/603457777 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fec05d110 con 0x7f8fec116780 2019-12-05T05:44:29.429 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:29.424+0000 7f8feb7fe700 1 --2- 172.21.15.42:0/603457777 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8fec116780 0x7f8fec1144e0 secure :-1 s=THROTTLE_DONE pgs=294 cs=0 l=1 rx=0x7f8fdc002f30 tx=0x7f8fdc0108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:29.429 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:29.424+0000 7f8fe3fff700 1 -- 172.21.15.42:0/603457777 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f8fdc031c10 con 0x7f8fec116780 2019-12-05T05:44:29.596 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:29.592+0000 7f8ff1517700 1 -- 172.21.15.42:0/603457777 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8fec1125f0 con 0x7f8fec116780 2019-12-05T05:44:29.598 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:29.592+0000 7f8feb7fe700 1 --2- 172.21.15.42:0/603457777 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8fec116780 0x7f8fec1144e0 secure :-1 s=THROTTLE_DONE pgs=294 cs=0 l=1 rx=0x7f8fdc002f30 tx=0x7f8fdc0108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:29.598 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:29.592+0000 7f8fe3fff700 1 -- 172.21.15.42:0/603457777 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 168 v168) v1 ==== 95+0+2109 (secure 0 0 0) 0x7f8fdc031e70 con 0x7f8fec116780 2019-12-05T05:44:29.603 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:44:29.603 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":168,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7884,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/1127133319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1127133319},{"type":"v1","addr":"172.21.15.40:6833","nonce":1127133319}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":168}],"filesystems":[{"mdsmap":{"epoch":167,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:44:09.069915+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":76,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7755},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7755":{"gid":7755,"name":"a","rank":0,"incarnation":164,"state":"up:active","state_seq":22,"addr":"172.21.15.42:6835/358497296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":358497296},{"type":"v1","addr":"172.21.15.42:6835","nonce":358497296}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:44:29.603 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 168 2019-12-05T05:44:29.604 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:29.596+0000 7f8ff1517700 1 -- 172.21.15.42:0/603457777 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8fd003b260 msgr2=0x7f8fd003d770 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:29.604 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:29.596+0000 7f8ff1517700 1 --2- 172.21.15.42:0/603457777 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8fd003b260 0x7f8fd003d770 crc :-1 s=READY pgs=595 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:44:29.604 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:29.596+0000 7f8ff1517700 1 -- 172.21.15.42:0/603457777 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8fec116780 msgr2=0x7f8fec1144e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:29.604 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:29.596+0000 7f8ff1517700 1 --2- 172.21.15.42:0/603457777 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8fec116780 0x7f8fec1144e0 secure :-1 s=READY pgs=294 cs=0 l=1 rx=0x7f8fdc002f30 tx=0x7f8fdc0108f0).stop 2019-12-05T05:44:29.604 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:29.596+0000 7f8ff1517700 1 -- 172.21.15.42:0/603457777 shutdown_connections 2019-12-05T05:44:29.604 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:29.596+0000 7f8ff1517700 1 --2- 172.21.15.42:0/603457777 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8fd003b260 0x7f8fd003d770 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:44:29.605 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:29.596+0000 7f8ff1517700 1 --2- 172.21.15.42:0/603457777 >> v2:172.21.15.40:3300/0 conn(0x7f8fec0a6360 0x7f8fec07d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:44:29.605 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:29.596+0000 7f8ff1517700 1 --2- 172.21.15.42:0/603457777 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8fec116780 0x7f8fec1144e0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:44:29.605 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:29.596+0000 7f8ff1517700 1 -- 172.21.15.42:0/603457777 shutdown_connections 2019-12-05T05:44:29.605 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:29.596+0000 7f8ff1517700 1 -- 172.21.15.42:0/603457777 wait complete. 2019-12-05T05:44:29.605 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:29.596+0000 7f8ff1517700 1 -- 172.21.15.42:0/603457777 >> 172.21.15.42:0/603457777 conn(0x7f8fec0aabe0 msgr2=0x7f8fec117440 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:44:29.612 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T05:44:31.617 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:44:31.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:31.712+0000 7f86e36b5700 1 Processor -- start 2019-12-05T05:44:31.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:31.712+0000 7f86e36b5700 1 -- start start 2019-12-05T05:44:31.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:31.712+0000 7f86e36b5700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f86dc0a7740 0x7f86dc08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:44:31.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:31.712+0000 7f86e36b5700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f86dc07e720 0x7f86dc07eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:44:31.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:31.712+0000 7f86e36b5700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f86dc09f2e0 0x7f86dc09f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:44:31.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:31.712+0000 7f86e36b5700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86dc0a0b00 con 0x7f86dc09f2e0 2019-12-05T05:44:31.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:31.712+0000 7f86e36b5700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86dc08e5b0 con 0x7f86dc07e720 2019-12-05T05:44:31.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:31.712+0000 7f86e36b5700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86dc08e850 con 0x7f86dc0a7740 2019-12-05T05:44:31.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:31.712+0000 7f86e1451700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f86dc07e720 0x7f86dc07eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:44:31.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:31.712+0000 7f86e1451700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f86dc07e720 0x7f86dc07eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:40132/0 (socket says 172.21.15.42:40132) 2019-12-05T05:44:31.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:31.712+0000 7f86e1451700 1 -- 172.21.15.42:0/2764913186 learned_addr learned my addr 172.21.15.42:0/2764913186 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:44:31.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:31.712+0000 7f86e2453700 1 --2- 172.21.15.42:0/2764913186 >> v2:172.21.15.40:3300/0 conn(0x7f86dc09f2e0 0x7f86dc09f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:44:31.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:31.712+0000 7f86e1c52700 1 --2- 172.21.15.42:0/2764913186 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f86dc0a7740 0x7f86dc08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:44:31.724 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:31.712+0000 7f86e1451700 1 -- 172.21.15.42:0/2764913186 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f86dc0a7740 msgr2=0x7f86dc08df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:44:31.724 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:31.712+0000 7f86e1451700 1 --2- 172.21.15.42:0/2764913186 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f86dc0a7740 0x7f86dc08df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:44:31.724 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:31.716+0000 7f86e1451700 1 -- 172.21.15.42:0/2764913186 >> v2:172.21.15.40:3300/0 conn(0x7f86dc09f2e0 msgr2=0x7f86dc09f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:44:31.730 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:31.716+0000 7f86e1451700 1 --2- 172.21.15.42:0/2764913186 >> v2:172.21.15.40:3300/0 conn(0x7f86dc09f2e0 0x7f86dc09f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:44:31.730 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:31.716+0000 7f86e1451700 1 -- 172.21.15.42:0/2764913186 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86dc07f0a0 con 0x7f86dc07e720 2019-12-05T05:44:31.738 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:31.716+0000 7f86e1451700 1 --2- 172.21.15.42:0/2764913186 >> v2:172.21.15.42:3300/0 conn(0x7f86dc07e720 0x7f86dc07eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f86dc08e770 tx=0x7f86cc00fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:31.738 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:31.716+0000 7f86e1451700 1 --2- 172.21.15.42:0/2764913186 >> v2:172.21.15.42:3300/0 conn(0x7f86dc07e720 0x7f86dc07eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f86dc08e770 tx=0x7f86cc00fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:31.739 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:31.716+0000 7f86e1451700 1 --2- 172.21.15.42:0/2764913186 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f86dc07e720 0x7f86dc07eb50 secure :-1 s=READY pgs=1383 cs=0 l=1 rx=0x7f86dc08e770 tx=0x7f86cc00fdb0).ready entity=mon.1 client_cookie=b3ddb57e71f8f7c1 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:44:31.739 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:31.716+0000 7f86e1451700 1 --2- 172.21.15.42:0/2764913186 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f86dc07e720 0x7f86dc07eb50 secure :-1 s=THROTTLE_DONE pgs=1383 cs=0 l=1 rx=0x7f86dc08e770 tx=0x7f86cc00fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:31.739 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:31.716+0000 7f86e0c50700 1 -- 172.21.15.42:0/2764913186 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f86cc008c90 con 0x7f86dc07e720 2019-12-05T05:44:31.739 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:31.716+0000 7f86e1451700 1 --2- 172.21.15.42:0/2764913186 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f86dc07e720 0x7f86dc07eb50 secure :-1 s=THROTTLE_DONE pgs=1383 cs=0 l=1 rx=0x7f86dc08e770 tx=0x7f86cc00fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:31.739 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:31.716+0000 7f86e1451700 1 --2- 172.21.15.42:0/2764913186 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f86dc07e720 0x7f86dc07eb50 secure :-1 s=THROTTLE_DONE pgs=1383 cs=0 l=1 rx=0x7f86dc08e770 tx=0x7f86cc00fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:31.739 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:31.716+0000 7f86e0c50700 1 -- 172.21.15.42:0/2764913186 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86cc0088b0 con 0x7f86dc07e720 2019-12-05T05:44:31.739 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:31.716+0000 7f86e0c50700 1 -- 172.21.15.42:0/2764913186 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f86cc011b50 con 0x7f86dc07e720 2019-12-05T05:44:31.740 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:31.716+0000 7f86e36b5700 1 -- 172.21.15.42:0/2764913186 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f86dc07e720 msgr2=0x7f86dc07eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:31.740 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:31.716+0000 7f86e36b5700 1 --2- 172.21.15.42:0/2764913186 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f86dc07e720 0x7f86dc07eb50 secure :-1 s=READY pgs=1383 cs=0 l=1 rx=0x7f86dc08e770 tx=0x7f86cc00fdb0).stop 2019-12-05T05:44:31.740 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:31.716+0000 7f86e36b5700 1 -- 172.21.15.42:0/2764913186 shutdown_connections 2019-12-05T05:44:31.740 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:31.716+0000 7f86e36b5700 1 --2- 172.21.15.42:0/2764913186 >> v2:172.21.15.40:3300/0 conn(0x7f86dc09f2e0 0x7f86dc09f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:44:31.740 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:31.716+0000 7f86e36b5700 1 --2- 172.21.15.42:0/2764913186 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f86dc0a7740 0x7f86dc08df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:44:31.740 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:31.716+0000 7f86e36b5700 1 --2- 172.21.15.42:0/2764913186 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f86dc07e720 0x7f86dc07eb50 unknown :-1 s=CLOSED pgs=1383 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:44:31.741 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:31.716+0000 7f86e36b5700 1 -- 172.21.15.42:0/2764913186 shutdown_connections 2019-12-05T05:44:31.741 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:31.716+0000 7f86e36b5700 1 -- 172.21.15.42:0/2764913186 wait complete. 2019-12-05T05:44:31.741 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:31.716+0000 7f86e36b5700 1 -- 172.21.15.42:0/2764913186 >> 172.21.15.42:0/2764913186 conn(0x7f86dc0aabe0 msgr2=0x7f86dc0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:44:31.748 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:31.716+0000 7f86e36b5700 1 Processor -- start 2019-12-05T05:44:31.748 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:31.716+0000 7f86e36b5700 1 -- start start 2019-12-05T05:44:31.748 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:31.720+0000 7f86e36b5700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f86dc119380 0x7f86dc0a55d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:44:31.748 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:31.720+0000 7f86e36b5700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f86dc144810 0x7f86dc1313a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:44:31.748 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:31.720+0000 7f86e36b5700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86dc141f70 con 0x7f86dc1318f0 2019-12-05T05:44:31.749 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:31.720+0000 7f86e36b5700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86dc119900 con 0x7f86dc119380 2019-12-05T05:44:31.749 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:31.720+0000 7f86e36b5700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86dc119ba0 con 0x7f86dc144810 2019-12-05T05:44:31.749 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:31.720+0000 7f86e1451700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f86dc144810 0x7f86dc1313a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:44:31.749 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:31.720+0000 7f86e1451700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f86dc144810 0x7f86dc1313a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:40602/0 (socket says 172.21.15.42:40602) 2019-12-05T05:44:31.749 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:31.720+0000 7f86e1451700 1 -- 172.21.15.42:0/3707425539 learned_addr learned my addr 172.21.15.42:0/3707425539 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:44:31.749 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:31.720+0000 7f86e1c52700 1 --2- 172.21.15.42:0/3707425539 >> v2:172.21.15.40:3300/0 conn(0x7f86dc119380 0x7f86dc0a55d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:44:31.749 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:31.720+0000 7f86e1451700 1 -- 172.21.15.42:0/3707425539 >> v2:172.21.15.40:3300/0 conn(0x7f86dc119380 msgr2=0x7f86dc0a55d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:31.750 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:31.720+0000 7f86e1451700 1 --2- 172.21.15.42:0/3707425539 >> v2:172.21.15.40:3300/0 conn(0x7f86dc119380 0x7f86dc0a55d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:44:31.750 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:31.720+0000 7f86e1451700 1 -- 172.21.15.42:0/3707425539 >> v1:172.21.15.40:6789/0 conn(0x7f86dc1318f0 legacy=0x7f86dc08eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:31.750 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:31.720+0000 7f86e1451700 1 -- 172.21.15.42:0/3707425539 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86dc131d20 con 0x7f86dc144810 2019-12-05T05:44:31.750 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:31.720+0000 7f86e1451700 1 --2- 172.21.15.42:0/3707425539 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f86dc144810 0x7f86dc1313a0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f86dc08e770 tx=0x7f86cc008b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:31.750 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:31.720+0000 7f86e1451700 1 --2- 172.21.15.42:0/3707425539 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f86dc144810 0x7f86dc1313a0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f86dc08e770 tx=0x7f86cc008b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:31.750 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:31.720+0000 7f86e1451700 1 --2- 172.21.15.42:0/3707425539 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f86dc144810 0x7f86dc1313a0 secure :-1 s=READY pgs=295 cs=0 l=1 rx=0x7f86dc08e770 tx=0x7f86cc008b70).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:44:31.750 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:31.720+0000 7f86e1451700 1 --2- 172.21.15.42:0/3707425539 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f86dc144810 0x7f86dc1313a0 secure :-1 s=THROTTLE_DONE pgs=295 cs=0 l=1 rx=0x7f86dc08e770 tx=0x7f86cc008b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:31.751 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:31.720+0000 7f86d27fc700 1 -- 172.21.15.42:0/3707425539 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f86cc00f680 con 0x7f86dc144810 2019-12-05T05:44:31.751 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:31.720+0000 7f86e1451700 1 --2- 172.21.15.42:0/3707425539 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f86dc144810 0x7f86dc1313a0 secure :-1 s=THROTTLE_DONE pgs=295 cs=0 l=1 rx=0x7f86dc08e770 tx=0x7f86cc008b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:31.751 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:31.720+0000 7f86e1451700 1 --2- 172.21.15.42:0/3707425539 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f86dc144810 0x7f86dc1313a0 secure :-1 s=THROTTLE_DONE pgs=295 cs=0 l=1 rx=0x7f86dc08e770 tx=0x7f86cc008b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:31.751 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:31.720+0000 7f86d27fc700 1 -- 172.21.15.42:0/3707425539 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86cc011770 con 0x7f86dc144810 2019-12-05T05:44:31.751 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:31.720+0000 7f86e36b5700 1 -- 172.21.15.42:0/3707425539 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86dc144c40 con 0x7f86dc144810 2019-12-05T05:44:31.751 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:31.720+0000 7f86d27fc700 1 -- 172.21.15.42:0/3707425539 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f86cc005920 con 0x7f86dc144810 2019-12-05T05:44:31.751 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:31.720+0000 7f86e36b5700 1 -- 172.21.15.42:0/3707425539 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86dc08f370 con 0x7f86dc144810 2019-12-05T05:44:31.751 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:31.720+0000 7f86e1451700 1 --2- 172.21.15.42:0/3707425539 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f86dc144810 0x7f86dc1313a0 secure :-1 s=THROTTLE_DONE pgs=295 cs=0 l=1 rx=0x7f86dc08e770 tx=0x7f86cc008b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:31.752 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:31.720+0000 7f86d27fc700 1 -- 172.21.15.42:0/3707425539 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f86cc005c30 con 0x7f86dc144810 2019-12-05T05:44:31.752 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:31.720+0000 7f86e1451700 1 --2- 172.21.15.42:0/3707425539 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f86dc144810 0x7f86dc1313a0 secure :-1 s=THROTTLE_DONE pgs=295 cs=0 l=1 rx=0x7f86dc08e770 tx=0x7f86cc008b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:31.752 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:31.724+0000 7f86d27fc700 1 --2- 172.21.15.42:0/3707425539 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f86c403ada0 0x7f86c403d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:44:31.752 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:31.724+0000 7f86d27fc700 1 -- 172.21.15.42:0/3707425539 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 8353+0+0 (secure 0 0 0) 0x7f86cc08f4b0 con 0x7f86dc144810 2019-12-05T05:44:31.752 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:31.724+0000 7f86e1c52700 1 --2- 172.21.15.42:0/3707425539 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f86c403ada0 0x7f86c403d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:44:31.752 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:31.724+0000 7f86e1c52700 1 --2- 172.21.15.42:0/3707425539 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f86c403ada0 0x7f86c403d2b0 crc :-1 s=READY pgs=596 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:44:31.752 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:31.728+0000 7f86e36b5700 1 -- 172.21.15.42:0/3707425539 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86dc05d110 con 0x7f86dc144810 2019-12-05T05:44:31.753 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:31.736+0000 7f86e1451700 1 --2- 172.21.15.42:0/3707425539 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f86dc144810 0x7f86dc1313a0 secure :-1 s=THROTTLE_DONE pgs=295 cs=0 l=1 rx=0x7f86dc08e770 tx=0x7f86cc008b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:31.753 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:31.736+0000 7f86d27fc700 1 -- 172.21.15.42:0/3707425539 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f86cc077360 con 0x7f86dc144810 2019-12-05T05:44:31.915 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:31.908+0000 7f86e36b5700 1 -- 172.21.15.42:0/3707425539 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f86dc05b8c0 con 0x7f86dc144810 2019-12-05T05:44:31.916 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:31.912+0000 7f86e1451700 1 --2- 172.21.15.42:0/3707425539 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f86dc144810 0x7f86dc1313a0 secure :-1 s=THROTTLE_DONE pgs=295 cs=0 l=1 rx=0x7f86dc08e770 tx=0x7f86cc008b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:31.916 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:31.912+0000 7f86d27fc700 1 -- 172.21.15.42:0/3707425539 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 168 v168) v1 ==== 95+0+2109 (secure 0 0 0) 0x7f86cc077e20 con 0x7f86dc144810 2019-12-05T05:44:31.918 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 168 2019-12-05T05:44:31.918 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:44:31.918 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":168,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7884,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/1127133319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1127133319},{"type":"v1","addr":"172.21.15.40:6833","nonce":1127133319}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":168}],"filesystems":[{"mdsmap":{"epoch":167,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:44:09.069915+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":76,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7755},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7755":{"gid":7755,"name":"a","rank":0,"incarnation":164,"state":"up:active","state_seq":22,"addr":"172.21.15.42:6835/358497296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":358497296},{"type":"v1","addr":"172.21.15.42:6835","nonce":358497296}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:44:31.920 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:31.916+0000 7f86e36b5700 1 -- 172.21.15.42:0/3707425539 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f86c403ada0 msgr2=0x7f86c403d2b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:31.920 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:31.916+0000 7f86e36b5700 1 --2- 172.21.15.42:0/3707425539 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f86c403ada0 0x7f86c403d2b0 crc :-1 s=READY pgs=596 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:44:31.921 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:31.916+0000 7f86e36b5700 1 -- 172.21.15.42:0/3707425539 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f86dc144810 msgr2=0x7f86dc1313a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:31.921 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:31.916+0000 7f86e36b5700 1 --2- 172.21.15.42:0/3707425539 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f86dc144810 0x7f86dc1313a0 secure :-1 s=READY pgs=295 cs=0 l=1 rx=0x7f86dc08e770 tx=0x7f86cc008b70).stop 2019-12-05T05:44:31.921 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:31.916+0000 7f86e36b5700 1 -- 172.21.15.42:0/3707425539 shutdown_connections 2019-12-05T05:44:31.921 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:31.916+0000 7f86e36b5700 1 --2- 172.21.15.42:0/3707425539 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f86c403ada0 0x7f86c403d2b0 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:44:31.921 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:31.916+0000 7f86e36b5700 1 --2- 172.21.15.42:0/3707425539 >> v2:172.21.15.40:3300/0 conn(0x7f86dc119380 0x7f86dc0a55d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:44:31.921 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:31.916+0000 7f86e36b5700 1 --2- 172.21.15.42:0/3707425539 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f86dc144810 0x7f86dc1313a0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:44:31.922 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:31.916+0000 7f86e36b5700 1 -- 172.21.15.42:0/3707425539 shutdown_connections 2019-12-05T05:44:31.922 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:31.916+0000 7f86e36b5700 1 -- 172.21.15.42:0/3707425539 wait complete. 2019-12-05T05:44:31.922 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:31.916+0000 7f86e36b5700 1 -- 172.21.15.42:0/3707425539 >> 172.21.15.42:0/3707425539 conn(0x7f86dc0a8fc0 msgr2=0x7f86dc0a9790 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:44:31.931 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T05:44:33.932 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:44:34.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:34.024+0000 7f2855dff700 1 Processor -- start 2019-12-05T05:44:34.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:34.024+0000 7f2855dff700 1 -- start start 2019-12-05T05:44:34.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:34.028+0000 7f2855dff700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f28500a7740 0x7f285008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:44:34.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:34.028+0000 7f2855dff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f285007e720 0x7f285007eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:44:34.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:34.028+0000 7f2855dff700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f285009f2e0 0x7f285009f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:44:34.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:34.028+0000 7f2855dff700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28500a0b00 con 0x7f285009f2e0 2019-12-05T05:44:34.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:34.028+0000 7f2855dff700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f285008e5b0 con 0x7f285007e720 2019-12-05T05:44:34.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:34.028+0000 7f2855dff700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f285008e850 con 0x7f28500a7740 2019-12-05T05:44:34.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:34.028+0000 7f284f7fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f285007e720 0x7f285007eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:44:34.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:34.028+0000 7f284ffff700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f28500a7740 0x7f285008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:44:34.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:34.028+0000 7f284f7fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f285007e720 0x7f285007eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:40146/0 (socket says 172.21.15.42:40146) 2019-12-05T05:44:34.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:34.028+0000 7f284f7fe700 1 -- 172.21.15.42:0/1208568858 learned_addr learned my addr 172.21.15.42:0/1208568858 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:44:34.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:34.028+0000 7f2854b9d700 1 --2- 172.21.15.42:0/1208568858 >> v2:172.21.15.40:3300/0 conn(0x7f285009f2e0 0x7f285009f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:44:34.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:34.028+0000 7f284f7fe700 1 -- 172.21.15.42:0/1208568858 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f28500a7740 msgr2=0x7f285008df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:44:34.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:34.028+0000 7f284f7fe700 1 --2- 172.21.15.42:0/1208568858 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f28500a7740 0x7f285008df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:44:34.039 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:34.028+0000 7f284f7fe700 1 -- 172.21.15.42:0/1208568858 >> v2:172.21.15.40:3300/0 conn(0x7f285009f2e0 msgr2=0x7f285009f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:44:34.039 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:34.028+0000 7f284f7fe700 1 --2- 172.21.15.42:0/1208568858 >> v2:172.21.15.40:3300/0 conn(0x7f285009f2e0 0x7f285009f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:44:34.039 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:34.028+0000 7f284f7fe700 1 -- 172.21.15.42:0/1208568858 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f285007f0a0 con 0x7f285007e720 2019-12-05T05:44:34.039 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:34.028+0000 7f284f7fe700 1 --2- 172.21.15.42:0/1208568858 >> v2:172.21.15.42:3300/0 conn(0x7f285007e720 0x7f285007eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f285008e770 tx=0x7f284400fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:34.039 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:34.028+0000 7f284f7fe700 1 --2- 172.21.15.42:0/1208568858 >> v2:172.21.15.42:3300/0 conn(0x7f285007e720 0x7f285007eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f285008e770 tx=0x7f284400fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:34.040 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:34.028+0000 7f284f7fe700 1 --2- 172.21.15.42:0/1208568858 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f285007e720 0x7f285007eb50 secure :-1 s=READY pgs=1384 cs=0 l=1 rx=0x7f285008e770 tx=0x7f284400fdb0).ready entity=mon.1 client_cookie=8525eb58e466a533 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:44:34.040 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:34.028+0000 7f284f7fe700 1 --2- 172.21.15.42:0/1208568858 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f285007e720 0x7f285007eb50 secure :-1 s=THROTTLE_DONE pgs=1384 cs=0 l=1 rx=0x7f285008e770 tx=0x7f284400fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:34.040 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:34.028+0000 7f284f7fe700 1 --2- 172.21.15.42:0/1208568858 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f285007e720 0x7f285007eb50 secure :-1 s=THROTTLE_DONE pgs=1384 cs=0 l=1 rx=0x7f285008e770 tx=0x7f284400fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:34.040 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:34.028+0000 7f284f7fe700 1 --2- 172.21.15.42:0/1208568858 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f285007e720 0x7f285007eb50 secure :-1 s=THROTTLE_DONE pgs=1384 cs=0 l=1 rx=0x7f285008e770 tx=0x7f284400fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:34.041 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:34.028+0000 7f284effd700 1 -- 172.21.15.42:0/1208568858 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2844008c90 con 0x7f285007e720 2019-12-05T05:44:34.041 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:34.028+0000 7f284effd700 1 -- 172.21.15.42:0/1208568858 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28440088b0 con 0x7f285007e720 2019-12-05T05:44:34.041 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:34.028+0000 7f284effd700 1 -- 172.21.15.42:0/1208568858 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2844011b50 con 0x7f285007e720 2019-12-05T05:44:34.042 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:34.028+0000 7f2855dff700 1 -- 172.21.15.42:0/1208568858 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f285007e720 msgr2=0x7f285007eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:34.042 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:34.028+0000 7f2855dff700 1 --2- 172.21.15.42:0/1208568858 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f285007e720 0x7f285007eb50 secure :-1 s=READY pgs=1384 cs=0 l=1 rx=0x7f285008e770 tx=0x7f284400fdb0).stop 2019-12-05T05:44:34.042 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:34.028+0000 7f2855dff700 1 -- 172.21.15.42:0/1208568858 shutdown_connections 2019-12-05T05:44:34.042 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:34.028+0000 7f2855dff700 1 --2- 172.21.15.42:0/1208568858 >> v2:172.21.15.40:3300/0 conn(0x7f285009f2e0 0x7f285009f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:44:34.042 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:34.028+0000 7f2855dff700 1 --2- 172.21.15.42:0/1208568858 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f28500a7740 0x7f285008df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:44:34.042 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:34.028+0000 7f2855dff700 1 --2- 172.21.15.42:0/1208568858 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f285007e720 0x7f285007eb50 unknown :-1 s=CLOSED pgs=1384 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:44:34.042 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:34.032+0000 7f2855dff700 1 -- 172.21.15.42:0/1208568858 shutdown_connections 2019-12-05T05:44:34.043 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:34.032+0000 7f2855dff700 1 -- 172.21.15.42:0/1208568858 wait complete. 2019-12-05T05:44:34.043 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:34.032+0000 7f2855dff700 1 -- 172.21.15.42:0/1208568858 >> 172.21.15.42:0/1208568858 conn(0x7f28500aabe0 msgr2=0x7f28500a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:44:34.045 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:34.032+0000 7f2855dff700 1 Processor -- start 2019-12-05T05:44:34.045 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:34.032+0000 7f2855dff700 1 -- start start 2019-12-05T05:44:34.046 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:34.032+0000 7f2855dff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2850123270 0x7f28501176b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:44:34.046 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:34.032+0000 7f2855dff700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f285007dbb0 con 0x7f2850124250 2019-12-05T05:44:34.046 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:34.032+0000 7f2855dff700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28501398b0 con 0x7f2850130780 2019-12-05T05:44:34.046 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:34.032+0000 7f2855dff700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2850117c00 con 0x7f2850123270 2019-12-05T05:44:34.046 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:34.036+0000 7f284f7fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2850123270 0x7f28501176b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:44:34.046 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:34.036+0000 7f284f7fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2850123270 0x7f28501176b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:40150/0 (socket says 172.21.15.42:40150) 2019-12-05T05:44:34.046 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:34.036+0000 7f284f7fe700 1 -- 172.21.15.42:0/165694256 learned_addr learned my addr 172.21.15.42:0/165694256 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:44:34.047 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:34.036+0000 7f2854b9d700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f2850130780 0x7f2850130bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:40190/0 (socket says 172.21.15.42:40190) 2019-12-05T05:44:34.047 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:34.036+0000 7f284f7fe700 1 -- 172.21.15.42:0/165694256 >> v1:172.21.15.42:6789/0 conn(0x7f2850130780 legacy=0x7f2850130bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:34.047 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:34.036+0000 7f284cff9700 1 -- 172.21.15.42:0/165694256 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1753615626 0 0) 0x7f28501398b0 con 0x7f2850130780 2019-12-05T05:44:34.047 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:34.036+0000 7f284f7fe700 1 -- 172.21.15.42:0/165694256 >> v1:172.21.15.40:6789/0 conn(0x7f2850124250 legacy=0x7f285007d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:34.047 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:34.036+0000 7f284f7fe700 1 -- 172.21.15.42:0/165694256 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f285013aaf0 con 0x7f2850123270 2019-12-05T05:44:34.047 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:34.036+0000 7f284f7fe700 1 --2- 172.21.15.42:0/165694256 >> v2:172.21.15.42:3300/0 conn(0x7f2850123270 0x7f28501176b0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f2840008050 tx=0x7f2840007aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:34.047 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:34.036+0000 7f284f7fe700 1 --2- 172.21.15.42:0/165694256 >> v2:172.21.15.42:3300/0 conn(0x7f2850123270 0x7f28501176b0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f2840008050 tx=0x7f2840007aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:34.048 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:34.036+0000 7f284f7fe700 1 --2- 172.21.15.42:0/165694256 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2850123270 0x7f28501176b0 secure :-1 s=READY pgs=1386 cs=0 l=1 rx=0x7f2840008050 tx=0x7f2840007aa0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:44:34.048 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:34.036+0000 7f284f7fe700 1 --2- 172.21.15.42:0/165694256 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2850123270 0x7f28501176b0 secure :-1 s=THROTTLE_DONE pgs=1386 cs=0 l=1 rx=0x7f2840008050 tx=0x7f2840007aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:34.048 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:34.036+0000 7f284cff9700 1 -- 172.21.15.42:0/165694256 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f284400f280 con 0x7f2850123270 2019-12-05T05:44:34.048 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:34.036+0000 7f284f7fe700 1 --2- 172.21.15.42:0/165694256 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2850123270 0x7f28501176b0 secure :-1 s=THROTTLE_DONE pgs=1386 cs=0 l=1 rx=0x7f2840008050 tx=0x7f2840007aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:34.048 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:34.036+0000 7f284f7fe700 1 --2- 172.21.15.42:0/165694256 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2850123270 0x7f28501176b0 secure :-1 s=THROTTLE_DONE pgs=1386 cs=0 l=1 rx=0x7f2840008050 tx=0x7f2840007aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:34.049 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:34.036+0000 7f284cff9700 1 -- 172.21.15.42:0/165694256 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2844011770 con 0x7f2850123270 2019-12-05T05:44:34.049 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:34.036+0000 7f284cff9700 1 -- 172.21.15.42:0/165694256 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f284405a660 con 0x7f2850123270 2019-12-05T05:44:34.050 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:34.036+0000 7f2855dff700 1 -- 172.21.15.42:0/165694256 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28501388a0 con 0x7f2850123270 2019-12-05T05:44:34.050 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:34.036+0000 7f2855dff700 1 -- 172.21.15.42:0/165694256 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2850138df0 con 0x7f2850123270 2019-12-05T05:44:34.051 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:34.036+0000 7f284f7fe700 1 --2- 172.21.15.42:0/165694256 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2850123270 0x7f28501176b0 secure :-1 s=THROTTLE_DONE pgs=1386 cs=0 l=1 rx=0x7f2840008050 tx=0x7f2840007aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:34.051 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:34.036+0000 7f284cff9700 1 -- 172.21.15.42:0/165694256 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f2844007180 con 0x7f2850123270 2019-12-05T05:44:34.051 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:34.036+0000 7f284f7fe700 1 --2- 172.21.15.42:0/165694256 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2850123270 0x7f28501176b0 secure :-1 s=THROTTLE_DONE pgs=1386 cs=0 l=1 rx=0x7f2840008050 tx=0x7f2840007aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:34.051 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:34.036+0000 7f284cff9700 1 --2- 172.21.15.42:0/165694256 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f283003ac70 0x7f283003d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:44:34.051 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:34.036+0000 7f284cff9700 1 -- 172.21.15.42:0/165694256 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 8353+0+0 (secure 0 0 0) 0x7f2844091420 con 0x7f2850123270 2019-12-05T05:44:34.051 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:34.036+0000 7f284ffff700 1 --2- 172.21.15.42:0/165694256 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f283003ac70 0x7f283003d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:44:34.051 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:34.036+0000 7f284ffff700 1 --2- 172.21.15.42:0/165694256 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f283003ac70 0x7f283003d180 crc :-1 s=READY pgs=597 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:44:34.052 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:34.040+0000 7f2855dff700 1 -- 172.21.15.42:0/165694256 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f285005d110 con 0x7f2850123270 2019-12-05T05:44:34.053 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:34.048+0000 7f284f7fe700 1 --2- 172.21.15.42:0/165694256 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2850123270 0x7f28501176b0 secure :-1 s=THROTTLE_DONE pgs=1386 cs=0 l=1 rx=0x7f2840008050 tx=0x7f2840007aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:34.053 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:34.048+0000 7f284cff9700 1 -- 172.21.15.42:0/165694256 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f28440786e0 con 0x7f2850123270 2019-12-05T05:44:34.222 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:34.216+0000 7f2855dff700 1 -- 172.21.15.42:0/165694256 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f285005b8c0 con 0x7f2850123270 2019-12-05T05:44:34.223 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:34.216+0000 7f284f7fe700 1 --2- 172.21.15.42:0/165694256 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2850123270 0x7f28501176b0 secure :-1 s=THROTTLE_DONE pgs=1386 cs=0 l=1 rx=0x7f2840008050 tx=0x7f2840007aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:34.223 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:34.216+0000 7f284cff9700 1 -- 172.21.15.42:0/165694256 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 168 v168) v1 ==== 95+0+2109 (secure 0 0 0) 0x7f2844076150 con 0x7f2850123270 2019-12-05T05:44:34.223 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 168 2019-12-05T05:44:34.223 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:44:34.223 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":168,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7884,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/1127133319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1127133319},{"type":"v1","addr":"172.21.15.40:6833","nonce":1127133319}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":168}],"filesystems":[{"mdsmap":{"epoch":167,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:44:09.069915+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":76,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7755},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7755":{"gid":7755,"name":"a","rank":0,"incarnation":164,"state":"up:active","state_seq":22,"addr":"172.21.15.42:6835/358497296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":358497296},{"type":"v1","addr":"172.21.15.42:6835","nonce":358497296}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:44:34.225 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:34.220+0000 7f2855dff700 1 -- 172.21.15.42:0/165694256 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f283003ac70 msgr2=0x7f283003d180 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:34.225 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:34.220+0000 7f2855dff700 1 --2- 172.21.15.42:0/165694256 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f283003ac70 0x7f283003d180 crc :-1 s=READY pgs=597 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:44:34.225 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:34.220+0000 7f2855dff700 1 -- 172.21.15.42:0/165694256 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2850123270 msgr2=0x7f28501176b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:34.225 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:34.220+0000 7f2855dff700 1 --2- 172.21.15.42:0/165694256 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2850123270 0x7f28501176b0 secure :-1 s=READY pgs=1386 cs=0 l=1 rx=0x7f2840008050 tx=0x7f2840007aa0).stop 2019-12-05T05:44:34.226 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:34.220+0000 7f2855dff700 1 -- 172.21.15.42:0/165694256 shutdown_connections 2019-12-05T05:44:34.226 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:34.220+0000 7f2855dff700 1 --2- 172.21.15.42:0/165694256 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f283003ac70 0x7f283003d180 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:44:34.226 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:34.220+0000 7f2855dff700 1 --2- 172.21.15.42:0/165694256 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2850123270 0x7f28501176b0 unknown :-1 s=CLOSED pgs=1386 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:44:34.226 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:34.220+0000 7f2855dff700 1 -- 172.21.15.42:0/165694256 shutdown_connections 2019-12-05T05:44:34.226 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:34.220+0000 7f2855dff700 1 -- 172.21.15.42:0/165694256 wait complete. 2019-12-05T05:44:34.226 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:34.220+0000 7f2855dff700 1 -- 172.21.15.42:0/165694256 >> 172.21.15.42:0/165694256 conn(0x7f28500aabe0 msgr2=0x7f285011bac0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:44:34.235 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T05:44:36.236 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:44:36.345 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:36.332+0000 7f9afcb46700 1 Processor -- start 2019-12-05T05:44:36.345 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:36.332+0000 7f9afcb46700 1 -- start start 2019-12-05T05:44:36.345 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:36.336+0000 7f9afcb46700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9af80a7740 0x7f9af808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:44:36.345 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:36.336+0000 7f9afcb46700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f9af807e720 0x7f9af807eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:44:36.345 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:36.336+0000 7f9afcb46700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9af80a0710 con 0x7f9af809f2e0 2019-12-05T05:44:36.346 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:36.336+0000 7f9afcb46700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9af80a0b00 con 0x7f9af807e720 2019-12-05T05:44:36.346 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:36.336+0000 7f9afcb46700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9af808e5b0 con 0x7f9af80a7740 2019-12-05T05:44:36.346 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:36.336+0000 7f9af659c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f9af807e720 0x7f9af807eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:44:36.346 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:36.336+0000 7f9af6d9d700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9af80a7740 0x7f9af808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:44:36.346 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:36.336+0000 7f9af659c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f9af807e720 0x7f9af807eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:40162/0 (socket says 172.21.15.42:40162) 2019-12-05T05:44:36.346 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:36.336+0000 7f9af6d9d700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9af80a7740 0x7f9af808df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:40624/0 (socket says 172.21.15.42:40624) 2019-12-05T05:44:36.346 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:36.336+0000 7f9af659c700 1 -- 172.21.15.42:0/146334197 learned_addr learned my addr 172.21.15.42:0/146334197 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:44:36.347 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:36.336+0000 7f9af6d9d700 1 -- 172.21.15.42:0/146334197 >> v2:172.21.15.42:3300/0 conn(0x7f9af807e720 msgr2=0x7f9af807eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:44:36.347 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:36.336+0000 7f9af6d9d700 1 --2- 172.21.15.42:0/146334197 >> v2:172.21.15.42:3300/0 conn(0x7f9af807e720 0x7f9af807eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:44:36.347 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:36.336+0000 7f9af6d9d700 1 -- 172.21.15.42:0/146334197 >> v1:172.21.15.42:6789/0 conn(0x7f9af809f2e0 legacy=0x7f9af809f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:36.347 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:36.336+0000 7f9af6d9d700 1 -- 172.21.15.42:0/146334197 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9af807f0a0 con 0x7f9af80a7740 2019-12-05T05:44:36.347 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:36.336+0000 7f9af6d9d700 1 --2- 172.21.15.42:0/146334197 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9af80a7740 0x7f9af808df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f9ae8008b10 tx=0x7f9ae800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:36.347 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:36.336+0000 7f9af6d9d700 1 --2- 172.21.15.42:0/146334197 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9af80a7740 0x7f9af808df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f9ae8008b10 tx=0x7f9ae800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:36.347 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:36.336+0000 7f9af6d9d700 1 --2- 172.21.15.42:0/146334197 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9af80a7740 0x7f9af808df80 secure :-1 s=READY pgs=296 cs=0 l=1 rx=0x7f9ae8008b10 tx=0x7f9ae800fff0).ready entity=mon.2 client_cookie=cecb7693e4816764 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:44:36.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:36.336+0000 7f9af6d9d700 1 --2- 172.21.15.42:0/146334197 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9af80a7740 0x7f9af808df80 secure :-1 s=THROTTLE_DONE pgs=296 cs=0 l=1 rx=0x7f9ae8008b10 tx=0x7f9ae800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:36.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:36.336+0000 7f9af5d9b700 1 -- 172.21.15.42:0/146334197 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9ae8011a50 con 0x7f9af80a7740 2019-12-05T05:44:36.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:36.336+0000 7f9af6d9d700 1 --2- 172.21.15.42:0/146334197 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9af80a7740 0x7f9af808df80 secure :-1 s=THROTTLE_DONE pgs=296 cs=0 l=1 rx=0x7f9ae8008b10 tx=0x7f9ae800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:36.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:36.336+0000 7f9af6d9d700 1 --2- 172.21.15.42:0/146334197 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9af80a7740 0x7f9af808df80 secure :-1 s=THROTTLE_DONE pgs=296 cs=0 l=1 rx=0x7f9ae8008b10 tx=0x7f9ae800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:36.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:36.336+0000 7f9af5d9b700 1 -- 172.21.15.42:0/146334197 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ae8011670 con 0x7f9af80a7740 2019-12-05T05:44:36.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:36.336+0000 7f9af5d9b700 1 -- 172.21.15.42:0/146334197 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9ae800a350 con 0x7f9af80a7740 2019-12-05T05:44:36.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:36.336+0000 7f9afcb46700 1 -- 172.21.15.42:0/146334197 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9af80a7740 msgr2=0x7f9af808df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:36.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:36.336+0000 7f9afcb46700 1 --2- 172.21.15.42:0/146334197 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9af80a7740 0x7f9af808df80 secure :-1 s=READY pgs=296 cs=0 l=1 rx=0x7f9ae8008b10 tx=0x7f9ae800fff0).stop 2019-12-05T05:44:36.358 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:36.336+0000 7f9afcb46700 1 -- 172.21.15.42:0/146334197 shutdown_connections 2019-12-05T05:44:36.358 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:36.336+0000 7f9afcb46700 1 --2- 172.21.15.42:0/146334197 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9af80a7740 0x7f9af808df80 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:44:36.358 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:36.336+0000 7f9afcb46700 1 --2- 172.21.15.42:0/146334197 >> v2:172.21.15.42:3300/0 conn(0x7f9af807e720 0x7f9af807eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:44:36.358 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:36.336+0000 7f9afcb46700 1 -- 172.21.15.42:0/146334197 shutdown_connections 2019-12-05T05:44:36.358 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:36.336+0000 7f9afcb46700 1 -- 172.21.15.42:0/146334197 wait complete. 2019-12-05T05:44:36.358 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:36.336+0000 7f9afcb46700 1 -- 172.21.15.42:0/146334197 >> 172.21.15.42:0/146334197 conn(0x7f9af80aabe0 msgr2=0x7f9af80a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:44:36.358 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:36.340+0000 7f9afcb46700 1 Processor -- start 2019-12-05T05:44:36.359 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:36.340+0000 7f9afcb46700 1 -- start start 2019-12-05T05:44:36.362 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:36.340+0000 7f9afcb46700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9af8117860 0x7f9af811ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:44:36.362 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:36.340+0000 7f9afcb46700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9af811c3f0 con 0x7f9af8116500 2019-12-05T05:44:36.362 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:36.340+0000 7f9afcb46700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9af8120e30 con 0x7f9af8135a80 2019-12-05T05:44:36.362 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:36.340+0000 7f9afcb46700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9af8117320 con 0x7f9af8117860 2019-12-05T05:44:36.362 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:36.340+0000 7f9af659c700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9af8117860 0x7f9af811ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:44:36.363 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:36.340+0000 7f9af759e700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f9af8135a80 0x7f9af8137ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:40206/0 (socket says 172.21.15.42:40206) 2019-12-05T05:44:36.363 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:36.340+0000 7f9af6d9d700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f9af8116500 0x7f9af811ba60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:49644/0 (socket says 172.21.15.42:49644) 2019-12-05T05:44:36.363 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:36.340+0000 7f9af759e700 1 -- 172.21.15.42:0/2683210554 learned_addr learned my addr 172.21.15.42:0/2683210554 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:44:36.363 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:36.340+0000 7f9aef7fe700 1 -- 172.21.15.42:0/2683210554 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3887491535 0 0) 0x7f9af811c3f0 con 0x7f9af8116500 2019-12-05T05:44:36.363 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:36.340+0000 7f9aef7fe700 1 -- 172.21.15.42:0/2683210554 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9adc0030b0 con 0x7f9af8116500 2019-12-05T05:44:36.363 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:36.340+0000 7f9aef7fe700 1 -- 172.21.15.42:0/2683210554 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4110640605 0 0) 0x7f9af8120e30 con 0x7f9af8135a80 2019-12-05T05:44:36.363 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:36.340+0000 7f9aef7fe700 1 -- 172.21.15.42:0/2683210554 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9af811c3f0 con 0x7f9af8135a80 2019-12-05T05:44:36.364 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:36.340+0000 7f9aef7fe700 1 -- 172.21.15.42:0/2683210554 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 3948813513 0 0) 0x7f9adc0030b0 con 0x7f9af8116500 2019-12-05T05:44:36.364 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:36.340+0000 7f9af659c700 1 -- 172.21.15.42:0/2683210554 >> v1:172.21.15.42:6789/0 conn(0x7f9af8135a80 legacy=0x7f9af8137ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:36.364 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:36.340+0000 7f9af659c700 1 -- 172.21.15.42:0/2683210554 >> v1:172.21.15.40:6789/0 conn(0x7f9af8116500 legacy=0x7f9af811ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:36.364 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:36.340+0000 7f9af659c700 1 -- 172.21.15.42:0/2683210554 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9af811a690 con 0x7f9af8117860 2019-12-05T05:44:36.364 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:36.344+0000 7f9af659c700 1 --2- 172.21.15.42:0/2683210554 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9af8117860 0x7f9af811ab20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f9ae8010400 tx=0x7f9ae8008650).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:36.364 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:36.344+0000 7f9af659c700 1 --2- 172.21.15.42:0/2683210554 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9af8117860 0x7f9af811ab20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f9ae8010400 tx=0x7f9ae8008650).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:36.365 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:36.344+0000 7f9af659c700 1 --2- 172.21.15.42:0/2683210554 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9af8117860 0x7f9af811ab20 secure :-1 s=READY pgs=297 cs=0 l=1 rx=0x7f9ae8010400 tx=0x7f9ae8008650).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:44:36.365 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:36.344+0000 7f9af659c700 1 --2- 172.21.15.42:0/2683210554 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9af8117860 0x7f9af811ab20 secure :-1 s=THROTTLE_DONE pgs=297 cs=0 l=1 rx=0x7f9ae8010400 tx=0x7f9ae8008650).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:36.365 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:36.344+0000 7f9af659c700 1 --2- 172.21.15.42:0/2683210554 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9af8117860 0x7f9af811ab20 secure :-1 s=THROTTLE_DONE pgs=297 cs=0 l=1 rx=0x7f9ae8010400 tx=0x7f9ae8008650).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:36.365 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:36.344+0000 7f9af659c700 1 --2- 172.21.15.42:0/2683210554 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9af8117860 0x7f9af811ab20 secure :-1 s=THROTTLE_DONE pgs=297 cs=0 l=1 rx=0x7f9ae8010400 tx=0x7f9ae8008650).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:36.365 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:36.344+0000 7f9aef7fe700 1 -- 172.21.15.42:0/2683210554 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9ae8007c70 con 0x7f9af8117860 2019-12-05T05:44:36.365 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:36.344+0000 7f9aef7fe700 1 -- 172.21.15.42:0/2683210554 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ae8011c50 con 0x7f9af8117860 2019-12-05T05:44:36.365 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:36.344+0000 7f9afcb46700 1 -- 172.21.15.42:0/2683210554 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9af811b070 con 0x7f9af8117860 2019-12-05T05:44:36.366 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:36.344+0000 7f9aef7fe700 1 -- 172.21.15.42:0/2683210554 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9ae0005b20 con 0x7f9af8117860 2019-12-05T05:44:36.366 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:36.344+0000 7f9afcb46700 1 -- 172.21.15.42:0/2683210554 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9af811fc20 con 0x7f9af8117860 2019-12-05T05:44:36.366 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:36.344+0000 7f9af659c700 1 --2- 172.21.15.42:0/2683210554 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9af8117860 0x7f9af811ab20 secure :-1 s=THROTTLE_DONE pgs=297 cs=0 l=1 rx=0x7f9ae8010400 tx=0x7f9ae8008650).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:36.366 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:36.344+0000 7f9aef7fe700 1 -- 172.21.15.42:0/2683210554 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f9ae00088b0 con 0x7f9af8117860 2019-12-05T05:44:36.366 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:36.344+0000 7f9af659c700 1 --2- 172.21.15.42:0/2683210554 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9af8117860 0x7f9af811ab20 secure :-1 s=THROTTLE_DONE pgs=297 cs=0 l=1 rx=0x7f9ae8010400 tx=0x7f9ae8008650).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:36.366 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:36.344+0000 7f9aef7fe700 1 --2- 172.21.15.42:0/2683210554 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9adc03b3d0 0x7f9adc03d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:44:36.366 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:36.344+0000 7f9aef7fe700 1 -- 172.21.15.42:0/2683210554 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 8353+0+0 (secure 0 0 0) 0x7f9ae004a9b0 con 0x7f9af8117860 2019-12-05T05:44:36.369 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:36.344+0000 7f9af6d9d700 1 --2- 172.21.15.42:0/2683210554 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9adc03b3d0 0x7f9adc03d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:44:36.370 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:36.344+0000 7f9af6d9d700 1 --2- 172.21.15.42:0/2683210554 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9adc03b3d0 0x7f9adc03d8e0 crc :-1 s=READY pgs=598 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:44:36.370 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:36.348+0000 7f9afcb46700 1 -- 172.21.15.42:0/2683210554 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9af805d110 con 0x7f9af8117860 2019-12-05T05:44:36.370 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:36.360+0000 7f9af659c700 1 --2- 172.21.15.42:0/2683210554 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9af8117860 0x7f9af811ab20 secure :-1 s=THROTTLE_DONE pgs=297 cs=0 l=1 rx=0x7f9ae8010400 tx=0x7f9ae8008650).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:36.370 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:36.360+0000 7f9aef7fe700 1 -- 172.21.15.42:0/2683210554 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f9ae00319d0 con 0x7f9af8117860 2019-12-05T05:44:36.535 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:36.528+0000 7f9afcb46700 1 -- 172.21.15.42:0/2683210554 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9af805b8c0 con 0x7f9af8117860 2019-12-05T05:44:36.536 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:36.532+0000 7f9af659c700 1 --2- 172.21.15.42:0/2683210554 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9af8117860 0x7f9af811ab20 secure :-1 s=THROTTLE_DONE pgs=297 cs=0 l=1 rx=0x7f9ae8010400 tx=0x7f9ae8008650).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:36.536 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:36.532+0000 7f9aef7fe700 1 -- 172.21.15.42:0/2683210554 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 168 v168) v1 ==== 95+0+2109 (secure 0 0 0) 0x7f9ae0031c30 con 0x7f9af8117860 2019-12-05T05:44:36.538 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 168 2019-12-05T05:44:36.538 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:44:36.538 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":168,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7884,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/1127133319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1127133319},{"type":"v1","addr":"172.21.15.40:6833","nonce":1127133319}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":168}],"filesystems":[{"mdsmap":{"epoch":167,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:44:09.069915+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":76,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7755},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7755":{"gid":7755,"name":"a","rank":0,"incarnation":164,"state":"up:active","state_seq":22,"addr":"172.21.15.42:6835/358497296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":358497296},{"type":"v1","addr":"172.21.15.42:6835","nonce":358497296}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:44:36.543 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:36.536+0000 7f9afcb46700 1 -- 172.21.15.42:0/2683210554 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9adc03b3d0 msgr2=0x7f9adc03d8e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:36.543 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:36.536+0000 7f9afcb46700 1 --2- 172.21.15.42:0/2683210554 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9adc03b3d0 0x7f9adc03d8e0 crc :-1 s=READY pgs=598 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:44:36.543 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:36.536+0000 7f9afcb46700 1 -- 172.21.15.42:0/2683210554 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9af8117860 msgr2=0x7f9af811ab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:36.543 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:36.536+0000 7f9afcb46700 1 --2- 172.21.15.42:0/2683210554 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9af8117860 0x7f9af811ab20 secure :-1 s=READY pgs=297 cs=0 l=1 rx=0x7f9ae8010400 tx=0x7f9ae8008650).stop 2019-12-05T05:44:36.543 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:36.536+0000 7f9afcb46700 1 -- 172.21.15.42:0/2683210554 shutdown_connections 2019-12-05T05:44:36.544 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:36.536+0000 7f9afcb46700 1 --2- 172.21.15.42:0/2683210554 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9adc03b3d0 0x7f9adc03d8e0 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:44:36.544 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:36.536+0000 7f9afcb46700 1 --2- 172.21.15.42:0/2683210554 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9af8117860 0x7f9af811ab20 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:44:36.544 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:36.536+0000 7f9afcb46700 1 -- 172.21.15.42:0/2683210554 shutdown_connections 2019-12-05T05:44:36.544 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:36.536+0000 7f9afcb46700 1 -- 172.21.15.42:0/2683210554 wait complete. 2019-12-05T05:44:36.544 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:36.536+0000 7f9afcb46700 1 -- 172.21.15.42:0/2683210554 >> 172.21.15.42:0/2683210554 conn(0x7f9af8066fe0 msgr2=0x7f9af80a9800 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:44:36.555 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T05:44:38.564 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:44:38.673 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:38.660+0000 7fed2cc76700 1 Processor -- start 2019-12-05T05:44:38.673 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:38.660+0000 7fed2cc76700 1 -- start start 2019-12-05T05:44:38.673 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:38.664+0000 7fed2cc76700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fed280a7740 0x7fed2808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:44:38.673 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:38.664+0000 7fed2cc76700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fed2809fae0 0x7fed2807eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:44:38.673 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:38.664+0000 7fed2cc76700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed280a0710 con 0x7fed2807e720 2019-12-05T05:44:38.674 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:38.664+0000 7fed2cc76700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed280a0b00 con 0x7fed2809fae0 2019-12-05T05:44:38.674 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:38.664+0000 7fed2cc76700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed2808e5b0 con 0x7fed280a7740 2019-12-05T05:44:38.674 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:38.664+0000 7fed2659c700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fed2807e720 0x7fed2809f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:40216/0 (socket says 172.21.15.42:40216) 2019-12-05T05:44:38.674 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:38.664+0000 7fed2659c700 1 -- 172.21.15.42:0/2840056626 learned_addr learned my addr 172.21.15.42:0/2840056626 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:44:38.674 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:38.664+0000 7fed26d9d700 1 --2- 172.21.15.42:0/2840056626 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fed280a7740 0x7fed2808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:44:38.674 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:38.664+0000 7fed2759e700 1 --2- 172.21.15.42:0/2840056626 >> v2:172.21.15.40:3300/0 conn(0x7fed2809fae0 0x7fed2807eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:44:38.675 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:38.664+0000 7fed25d9b700 1 -- 172.21.15.42:0/2840056626 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 272098538 0 0) 0x7fed280a0710 con 0x7fed2807e720 2019-12-05T05:44:38.675 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:38.664+0000 7fed25d9b700 1 -- 172.21.15.42:0/2840056626 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed140030b0 con 0x7fed2807e720 2019-12-05T05:44:38.675 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:38.664+0000 7fed26d9d700 1 -- 172.21.15.42:0/2840056626 >> v2:172.21.15.40:3300/0 conn(0x7fed2809fae0 msgr2=0x7fed2807eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:44:38.675 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:38.664+0000 7fed26d9d700 1 --2- 172.21.15.42:0/2840056626 >> v2:172.21.15.40:3300/0 conn(0x7fed2809fae0 0x7fed2807eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:44:38.675 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:38.664+0000 7fed26d9d700 1 -- 172.21.15.42:0/2840056626 >> v1:172.21.15.42:6789/0 conn(0x7fed2807e720 legacy=0x7fed2809f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:38.675 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:38.664+0000 7fed26d9d700 1 -- 172.21.15.42:0/2840056626 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed2807f0a0 con 0x7fed280a7740 2019-12-05T05:44:38.676 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:38.664+0000 7fed26d9d700 1 --2- 172.21.15.42:0/2840056626 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fed280a7740 0x7fed2808df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fed18008b10 tx=0x7fed1800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:38.676 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:38.664+0000 7fed26d9d700 1 --2- 172.21.15.42:0/2840056626 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fed280a7740 0x7fed2808df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fed18008b10 tx=0x7fed1800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:38.676 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:38.664+0000 7fed26d9d700 1 --2- 172.21.15.42:0/2840056626 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fed280a7740 0x7fed2808df80 secure :-1 s=READY pgs=298 cs=0 l=1 rx=0x7fed18008b10 tx=0x7fed1800fff0).ready entity=mon.2 client_cookie=c2df1005f9b10c1d server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:44:38.676 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:38.664+0000 7fed26d9d700 1 --2- 172.21.15.42:0/2840056626 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fed280a7740 0x7fed2808df80 secure :-1 s=THROTTLE_DONE pgs=298 cs=0 l=1 rx=0x7fed18008b10 tx=0x7fed1800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:38.676 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:38.664+0000 7fed25d9b700 1 -- 172.21.15.42:0/2840056626 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fed18010900 con 0x7fed280a7740 2019-12-05T05:44:38.676 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:38.664+0000 7fed26d9d700 1 --2- 172.21.15.42:0/2840056626 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fed280a7740 0x7fed2808df80 secure :-1 s=THROTTLE_DONE pgs=298 cs=0 l=1 rx=0x7fed18008b10 tx=0x7fed1800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:38.676 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:38.664+0000 7fed26d9d700 1 --2- 172.21.15.42:0/2840056626 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fed280a7740 0x7fed2808df80 secure :-1 s=THROTTLE_DONE pgs=298 cs=0 l=1 rx=0x7fed18008b10 tx=0x7fed1800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:38.677 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:38.664+0000 7fed25d9b700 1 -- 172.21.15.42:0/2840056626 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed18008b50 con 0x7fed280a7740 2019-12-05T05:44:38.677 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:38.664+0000 7fed25d9b700 1 -- 172.21.15.42:0/2840056626 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fed180110c0 con 0x7fed280a7740 2019-12-05T05:44:38.677 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:38.664+0000 7fed2cc76700 1 -- 172.21.15.42:0/2840056626 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fed280a7740 msgr2=0x7fed2808df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:38.677 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:38.664+0000 7fed2cc76700 1 --2- 172.21.15.42:0/2840056626 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fed280a7740 0x7fed2808df80 secure :-1 s=READY pgs=298 cs=0 l=1 rx=0x7fed18008b10 tx=0x7fed1800fff0).stop 2019-12-05T05:44:38.677 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:38.668+0000 7fed2cc76700 1 -- 172.21.15.42:0/2840056626 shutdown_connections 2019-12-05T05:44:38.677 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:38.668+0000 7fed2cc76700 1 --2- 172.21.15.42:0/2840056626 >> v2:172.21.15.40:3300/0 conn(0x7fed2809fae0 0x7fed2807eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:44:38.678 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:38.668+0000 7fed2cc76700 1 --2- 172.21.15.42:0/2840056626 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fed280a7740 0x7fed2808df80 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:44:38.678 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:38.668+0000 7fed2cc76700 1 -- 172.21.15.42:0/2840056626 shutdown_connections 2019-12-05T05:44:38.678 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:38.668+0000 7fed2cc76700 1 -- 172.21.15.42:0/2840056626 wait complete. 2019-12-05T05:44:38.678 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:38.668+0000 7fed2cc76700 1 -- 172.21.15.42:0/2840056626 >> 172.21.15.42:0/2840056626 conn(0x7fed280aabe0 msgr2=0x7fed280a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:44:38.682 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:38.668+0000 7fed2cc76700 1 Processor -- start 2019-12-05T05:44:38.682 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:38.668+0000 7fed2cc76700 1 -- start start 2019-12-05T05:44:38.682 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:38.672+0000 7fed2cc76700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fed28116ed0 0x7fed280a36e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:44:38.682 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:38.672+0000 7fed2cc76700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed280a3c30 con 0x7fed28118090 2019-12-05T05:44:38.682 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:38.672+0000 7fed2cc76700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed28119d20 con 0x7fed280722c0 2019-12-05T05:44:38.682 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:38.672+0000 7fed2cc76700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed28119070 con 0x7fed28116ed0 2019-12-05T05:44:38.683 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:38.672+0000 7fed2759e700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fed280722c0 0x7fed28074700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:40224/0 (socket says 172.21.15.42:40224) 2019-12-05T05:44:38.683 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:38.672+0000 7fed2759e700 1 -- 172.21.15.42:0/826385564 learned_addr learned my addr 172.21.15.42:0/826385564 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:44:38.683 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:38.672+0000 7fed26d9d700 1 --2- 172.21.15.42:0/826385564 >> v2:172.21.15.40:3300/0 conn(0x7fed28116ed0 0x7fed280a36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:44:38.683 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:38.672+0000 7fed1f7fe700 1 -- 172.21.15.42:0/826385564 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2694604740 0 0) 0x7fed28119d20 con 0x7fed280722c0 2019-12-05T05:44:38.683 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:38.672+0000 7fed1f7fe700 1 -- 172.21.15.42:0/826385564 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed0c0030b0 con 0x7fed280722c0 2019-12-05T05:44:38.683 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:38.672+0000 7fed1f7fe700 1 -- 172.21.15.42:0/826385564 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 116785634 0 0) 0x7fed0c0030b0 con 0x7fed280722c0 2019-12-05T05:44:38.684 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:38.672+0000 7fed26d9d700 1 -- 172.21.15.42:0/826385564 >> v1:172.21.15.42:6789/0 conn(0x7fed280722c0 legacy=0x7fed28074700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:38.684 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:38.672+0000 7fed26d9d700 1 -- 172.21.15.42:0/826385564 >> v1:172.21.15.40:6789/0 conn(0x7fed28118090 legacy=0x7fed281184c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:38.684 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:38.672+0000 7fed26d9d700 1 -- 172.21.15.42:0/826385564 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed28077f30 con 0x7fed28116ed0 2019-12-05T05:44:38.684 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:38.672+0000 7fed26d9d700 1 --2- 172.21.15.42:0/826385564 >> v2:172.21.15.40:3300/0 conn(0x7fed28116ed0 0x7fed280a36e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fed10001090 tx=0x7fed20002f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:38.684 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:38.672+0000 7fed26d9d700 1 --2- 172.21.15.42:0/826385564 >> v2:172.21.15.40:3300/0 conn(0x7fed28116ed0 0x7fed280a36e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fed10001090 tx=0x7fed20002f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:38.684 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:38.672+0000 7fed26d9d700 1 --2- 172.21.15.42:0/826385564 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fed28116ed0 0x7fed280a36e0 secure :-1 s=READY pgs=1076 cs=0 l=1 rx=0x7fed10001090 tx=0x7fed20002f70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:44:38.684 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:38.672+0000 7fed26d9d700 1 --2- 172.21.15.42:0/826385564 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fed28116ed0 0x7fed280a36e0 secure :-1 s=THROTTLE_DONE pgs=1076 cs=0 l=1 rx=0x7fed10001090 tx=0x7fed20002f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:38.685 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:38.672+0000 7fed1f7fe700 1 -- 172.21.15.42:0/826385564 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fed20001490 con 0x7fed28116ed0 2019-12-05T05:44:38.685 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:38.672+0000 7fed26d9d700 1 --2- 172.21.15.42:0/826385564 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fed28116ed0 0x7fed280a36e0 secure :-1 s=THROTTLE_DONE pgs=1076 cs=0 l=1 rx=0x7fed10001090 tx=0x7fed20002f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:38.685 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:38.672+0000 7fed26d9d700 1 --2- 172.21.15.42:0/826385564 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fed28116ed0 0x7fed280a36e0 secure :-1 s=THROTTLE_DONE pgs=1076 cs=0 l=1 rx=0x7fed10001090 tx=0x7fed20002f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:38.685 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:38.672+0000 7fed1f7fe700 1 -- 172.21.15.42:0/826385564 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed200012a0 con 0x7fed28116ed0 2019-12-05T05:44:38.685 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:38.672+0000 7fed2cc76700 1 -- 172.21.15.42:0/826385564 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed28075f10 con 0x7fed28116ed0 2019-12-05T05:44:38.685 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:38.672+0000 7fed1f7fe700 1 -- 172.21.15.42:0/826385564 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fed18005ca0 con 0x7fed28116ed0 2019-12-05T05:44:38.686 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:38.672+0000 7fed2cc76700 1 -- 172.21.15.42:0/826385564 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed280762f0 con 0x7fed28116ed0 2019-12-05T05:44:38.686 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:38.672+0000 7fed26d9d700 1 --2- 172.21.15.42:0/826385564 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fed28116ed0 0x7fed280a36e0 secure :-1 s=THROTTLE_DONE pgs=1076 cs=0 l=1 rx=0x7fed10001090 tx=0x7fed20002f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:38.686 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:38.676+0000 7fed1f7fe700 1 -- 172.21.15.42:0/826385564 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fed1800f280 con 0x7fed28116ed0 2019-12-05T05:44:38.686 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:38.676+0000 7fed26d9d700 1 --2- 172.21.15.42:0/826385564 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fed28116ed0 0x7fed280a36e0 secure :-1 s=THROTTLE_DONE pgs=1076 cs=0 l=1 rx=0x7fed10001090 tx=0x7fed20002f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:38.686 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:38.676+0000 7fed1f7fe700 1 --2- 172.21.15.42:0/826385564 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fed0c03b210 0x7fed0c03d720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:44:38.687 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:38.676+0000 7fed1f7fe700 1 -- 172.21.15.42:0/826385564 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 8353+0+0 (secure 0 0 0) 0x7fed1804a8f0 con 0x7fed28116ed0 2019-12-05T05:44:38.687 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:38.676+0000 7fed2659c700 1 --2- 172.21.15.42:0/826385564 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fed0c03b210 0x7fed0c03d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:44:38.687 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:38.676+0000 7fed2659c700 1 --2- 172.21.15.42:0/826385564 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fed0c03b210 0x7fed0c03d720 crc :-1 s=READY pgs=599 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:44:38.688 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:38.676+0000 7fed2cc76700 1 -- 172.21.15.42:0/826385564 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed2805d110 con 0x7fed28116ed0 2019-12-05T05:44:38.692 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:38.688+0000 7fed26d9d700 1 --2- 172.21.15.42:0/826385564 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fed28116ed0 0x7fed280a36e0 secure :-1 s=THROTTLE_DONE pgs=1076 cs=0 l=1 rx=0x7fed10001090 tx=0x7fed20002f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:38.692 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:38.688+0000 7fed1f7fe700 1 -- 172.21.15.42:0/826385564 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fed180327a0 con 0x7fed28116ed0 2019-12-05T05:44:38.860 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:38.856+0000 7fed2cc76700 1 -- 172.21.15.42:0/826385564 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fed2805b8c0 con 0x7fed28116ed0 2019-12-05T05:44:38.860 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:38.856+0000 7fed26d9d700 1 --2- 172.21.15.42:0/826385564 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fed28116ed0 0x7fed280a36e0 secure :-1 s=THROTTLE_DONE pgs=1076 cs=0 l=1 rx=0x7fed10001090 tx=0x7fed20002f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:38.861 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:38.856+0000 7fed1f7fe700 1 -- 172.21.15.42:0/826385564 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 168 v168) v1 ==== 95+0+2109 (secure 0 0 0) 0x7fed18033b20 con 0x7fed28116ed0 2019-12-05T05:44:38.863 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 168 2019-12-05T05:44:38.863 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:44:38.863 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":168,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7884,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/1127133319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1127133319},{"type":"v1","addr":"172.21.15.40:6833","nonce":1127133319}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":168}],"filesystems":[{"mdsmap":{"epoch":167,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:44:09.069915+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":76,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7755},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7755":{"gid":7755,"name":"a","rank":0,"incarnation":164,"state":"up:active","state_seq":22,"addr":"172.21.15.42:6835/358497296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":358497296},{"type":"v1","addr":"172.21.15.42:6835","nonce":358497296}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:44:38.866 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:38.860+0000 7fed2cc76700 1 -- 172.21.15.42:0/826385564 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fed0c03b210 msgr2=0x7fed0c03d720 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:38.866 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:38.860+0000 7fed2cc76700 1 --2- 172.21.15.42:0/826385564 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fed0c03b210 0x7fed0c03d720 crc :-1 s=READY pgs=599 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:44:38.866 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:38.860+0000 7fed2cc76700 1 -- 172.21.15.42:0/826385564 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fed28116ed0 msgr2=0x7fed280a36e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:38.866 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:38.860+0000 7fed2cc76700 1 --2- 172.21.15.42:0/826385564 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fed28116ed0 0x7fed280a36e0 secure :-1 s=READY pgs=1076 cs=0 l=1 rx=0x7fed10001090 tx=0x7fed20002f70).stop 2019-12-05T05:44:38.866 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:38.860+0000 7fed2cc76700 1 -- 172.21.15.42:0/826385564 shutdown_connections 2019-12-05T05:44:38.867 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:38.860+0000 7fed2cc76700 1 --2- 172.21.15.42:0/826385564 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fed0c03b210 0x7fed0c03d720 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:44:38.867 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:38.860+0000 7fed2cc76700 1 --2- 172.21.15.42:0/826385564 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fed28116ed0 0x7fed280a36e0 unknown :-1 s=CLOSED pgs=1076 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:44:38.867 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:38.860+0000 7fed2cc76700 1 -- 172.21.15.42:0/826385564 shutdown_connections 2019-12-05T05:44:38.867 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:38.860+0000 7fed2cc76700 1 -- 172.21.15.42:0/826385564 wait complete. 2019-12-05T05:44:38.867 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:38.860+0000 7fed2cc76700 1 -- 172.21.15.42:0/826385564 >> 172.21.15.42:0/826385564 conn(0x7fed280a9790 msgr2=0x7fed2809f380 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:44:38.876 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T05:44:40.877 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:44:40.981 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:40.971+0000 7ff73e231700 1 Processor -- start 2019-12-05T05:44:40.981 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:40.971+0000 7ff73e231700 1 -- start start 2019-12-05T05:44:40.982 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:40.971+0000 7ff73e231700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff7380a7740 0x7ff73808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:44:40.982 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:40.971+0000 7ff73e231700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff73809fae0 0x7ff73807eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:44:40.982 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:40.971+0000 7ff73e231700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7380a0710 con 0x7ff73807e720 2019-12-05T05:44:40.982 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:40.971+0000 7ff73e231700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7380a0b00 con 0x7ff73809fae0 2019-12-05T05:44:40.982 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:40.971+0000 7ff73e231700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff73808e5b0 con 0x7ff7380a7740 2019-12-05T05:44:40.982 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:40.971+0000 7ff73cfcf700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff73809fae0 0x7ff73807eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:44:40.983 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:40.971+0000 7ff73cfcf700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff73809fae0 0x7ff73807eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:40194/0 (socket says 172.21.15.42:40194) 2019-12-05T05:44:40.983 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:40.971+0000 7ff73cfcf700 1 -- 172.21.15.42:0/503007606 learned_addr learned my addr 172.21.15.42:0/503007606 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:44:40.983 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:40.971+0000 7ff737fff700 1 --2- 172.21.15.42:0/503007606 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff7380a7740 0x7ff73808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:44:40.983 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:40.971+0000 7ff73cfcf700 1 -- 172.21.15.42:0/503007606 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff7380a7740 msgr2=0x7ff73808df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:44:40.983 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:40.971+0000 7ff73cfcf700 1 --2- 172.21.15.42:0/503007606 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff7380a7740 0x7ff73808df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:44:40.983 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:40.971+0000 7ff73cfcf700 1 -- 172.21.15.42:0/503007606 >> v1:172.21.15.40:6789/0 conn(0x7ff73807e720 legacy=0x7ff73809f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:40.983 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:40.971+0000 7ff73cfcf700 1 -- 172.21.15.42:0/503007606 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff73807f0a0 con 0x7ff73809fae0 2019-12-05T05:44:40.984 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:40.971+0000 7ff73cfcf700 1 --2- 172.21.15.42:0/503007606 >> v2:172.21.15.42:3300/0 conn(0x7ff73809fae0 0x7ff73807eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7ff730008b10 tx=0x7ff7300100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:40.984 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:40.971+0000 7ff73cfcf700 1 --2- 172.21.15.42:0/503007606 >> v2:172.21.15.42:3300/0 conn(0x7ff73809fae0 0x7ff73807eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7ff730008b10 tx=0x7ff7300100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:40.984 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:40.971+0000 7ff73cfcf700 1 --2- 172.21.15.42:0/503007606 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff73809fae0 0x7ff73807eb50 secure :-1 s=READY pgs=1392 cs=0 l=1 rx=0x7ff730008b10 tx=0x7ff7300100b0).ready entity=mon.1 client_cookie=3b4c482d2cfce5c7 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:44:40.984 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:40.971+0000 7ff73cfcf700 1 --2- 172.21.15.42:0/503007606 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff73809fae0 0x7ff73807eb50 secure :-1 s=THROTTLE_DONE pgs=1392 cs=0 l=1 rx=0x7ff730008b10 tx=0x7ff7300100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:40.984 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:40.971+0000 7ff736ffd700 1 -- 172.21.15.42:0/503007606 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff730010da0 con 0x7ff73809fae0 2019-12-05T05:44:40.984 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:40.971+0000 7ff73cfcf700 1 --2- 172.21.15.42:0/503007606 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff73809fae0 0x7ff73807eb50 secure :-1 s=THROTTLE_DONE pgs=1392 cs=0 l=1 rx=0x7ff730008b10 tx=0x7ff7300100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:40.984 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:40.971+0000 7ff736ffd700 1 -- 172.21.15.42:0/503007606 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7300109c0 con 0x7ff73809fae0 2019-12-05T05:44:40.985 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:40.971+0000 7ff73cfcf700 1 --2- 172.21.15.42:0/503007606 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff73809fae0 0x7ff73807eb50 secure :-1 s=THROTTLE_DONE pgs=1392 cs=0 l=1 rx=0x7ff730008b10 tx=0x7ff7300100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:40.985 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:40.971+0000 7ff73e231700 1 -- 172.21.15.42:0/503007606 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff73809fae0 msgr2=0x7ff73807eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:40.985 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:40.971+0000 7ff736ffd700 1 -- 172.21.15.42:0/503007606 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff7300114a0 con 0x7ff73809fae0 2019-12-05T05:44:40.985 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:40.971+0000 7ff73e231700 1 --2- 172.21.15.42:0/503007606 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff73809fae0 0x7ff73807eb50 secure :-1 s=READY pgs=1392 cs=0 l=1 rx=0x7ff730008b10 tx=0x7ff7300100b0).stop 2019-12-05T05:44:40.985 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:40.975+0000 7ff73e231700 1 -- 172.21.15.42:0/503007606 shutdown_connections 2019-12-05T05:44:40.985 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:40.975+0000 7ff73e231700 1 --2- 172.21.15.42:0/503007606 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff73809fae0 0x7ff73807eb50 unknown :-1 s=CLOSED pgs=1392 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:44:40.985 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:40.975+0000 7ff73e231700 1 --2- 172.21.15.42:0/503007606 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff7380a7740 0x7ff73808df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:44:40.986 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:40.975+0000 7ff73e231700 1 -- 172.21.15.42:0/503007606 shutdown_connections 2019-12-05T05:44:40.986 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:40.975+0000 7ff73e231700 1 -- 172.21.15.42:0/503007606 wait complete. 2019-12-05T05:44:40.986 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:40.975+0000 7ff73e231700 1 -- 172.21.15.42:0/503007606 >> 172.21.15.42:0/503007606 conn(0x7ff7380aabe0 msgr2=0x7ff7380a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:44:40.989 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:40.979+0000 7ff73e231700 1 Processor -- start 2019-12-05T05:44:40.989 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:40.979+0000 7ff73e231700 1 -- start start 2019-12-05T05:44:40.989 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:40.979+0000 7ff73e231700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff738116500 0x7ff738130a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:44:40.989 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:40.979+0000 7ff73e231700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff7380a5100 0x7ff738133d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:44:40.990 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:40.979+0000 7ff73e231700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff73807d630 con 0x7ff738130fd0 2019-12-05T05:44:40.990 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:40.979+0000 7ff73e231700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7381302a0 con 0x7ff7380a5100 2019-12-05T05:44:40.990 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:40.979+0000 7ff73e231700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff738130540 con 0x7ff738116500 2019-12-05T05:44:40.990 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:40.979+0000 7ff7377fe700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7ff738130fd0 0x7ff73807ce30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:40236/0 (socket says 172.21.15.42:40236) 2019-12-05T05:44:40.990 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:40.979+0000 7ff7377fe700 1 -- 172.21.15.42:0/854128188 learned_addr learned my addr 172.21.15.42:0/854128188 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:44:40.990 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:40.979+0000 7ff737fff700 1 --2- 172.21.15.42:0/854128188 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff738116500 0x7ff738130a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:44:40.991 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:40.979+0000 7ff73cfcf700 1 --2- 172.21.15.42:0/854128188 >> v2:172.21.15.40:3300/0 conn(0x7ff7380a5100 0x7ff738133d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:44:40.991 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:40.979+0000 7ff734ff9700 1 -- 172.21.15.42:0/854128188 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 816760183 0 0) 0x7ff73807d630 con 0x7ff738130fd0 2019-12-05T05:44:40.991 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:40.979+0000 7ff734ff9700 1 -- 172.21.15.42:0/854128188 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff71c0030b0 con 0x7ff738130fd0 2019-12-05T05:44:40.991 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:40.979+0000 7ff737fff700 1 -- 172.21.15.42:0/854128188 >> v2:172.21.15.40:3300/0 conn(0x7ff7380a5100 msgr2=0x7ff738133d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:40.991 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:40.979+0000 7ff737fff700 1 --2- 172.21.15.42:0/854128188 >> v2:172.21.15.40:3300/0 conn(0x7ff7380a5100 0x7ff738133d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:44:40.991 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:40.979+0000 7ff737fff700 1 -- 172.21.15.42:0/854128188 >> v1:172.21.15.42:6789/0 conn(0x7ff738130fd0 legacy=0x7ff73807ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:40.991 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:40.979+0000 7ff737fff700 1 -- 172.21.15.42:0/854128188 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff738134290 con 0x7ff738116500 2019-12-05T05:44:40.992 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:40.979+0000 7ff737fff700 1 --2- 172.21.15.42:0/854128188 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff738116500 0x7ff738130a80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7ff728002f30 tx=0x7ff7280109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:40.992 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:40.979+0000 7ff737fff700 1 --2- 172.21.15.42:0/854128188 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff738116500 0x7ff738130a80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7ff728002f30 tx=0x7ff7280109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:40.992 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:40.979+0000 7ff737fff700 1 --2- 172.21.15.42:0/854128188 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff738116500 0x7ff738130a80 secure :-1 s=READY pgs=299 cs=0 l=1 rx=0x7ff728002f30 tx=0x7ff7280109e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:44:40.992 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:40.979+0000 7ff737fff700 1 --2- 172.21.15.42:0/854128188 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff738116500 0x7ff738130a80 secure :-1 s=THROTTLE_DONE pgs=299 cs=0 l=1 rx=0x7ff728002f30 tx=0x7ff7280109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:40.992 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:40.979+0000 7ff734ff9700 1 -- 172.21.15.42:0/854128188 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff728005920 con 0x7ff738116500 2019-12-05T05:44:40.992 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:40.979+0000 7ff737fff700 1 --2- 172.21.15.42:0/854128188 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff738116500 0x7ff738130a80 secure :-1 s=THROTTLE_DONE pgs=299 cs=0 l=1 rx=0x7ff728002f30 tx=0x7ff7280109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:40.992 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:40.979+0000 7ff73e231700 1 -- 172.21.15.42:0/854128188 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff738134500 con 0x7ff738116500 2019-12-05T05:44:40.992 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:40.979+0000 7ff734ff9700 1 -- 172.21.15.42:0/854128188 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff728005540 con 0x7ff738116500 2019-12-05T05:44:40.993 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:40.979+0000 7ff737fff700 1 --2- 172.21.15.42:0/854128188 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff738116500 0x7ff738130a80 secure :-1 s=THROTTLE_DONE pgs=299 cs=0 l=1 rx=0x7ff728002f30 tx=0x7ff7280109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:40.993 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:40.979+0000 7ff734ff9700 1 -- 172.21.15.42:0/854128188 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff728012510 con 0x7ff738116500 2019-12-05T05:44:40.993 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:40.983+0000 7ff73e231700 1 -- 172.21.15.42:0/854128188 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7380742d0 con 0x7ff738116500 2019-12-05T05:44:40.993 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:40.983+0000 7ff737fff700 1 --2- 172.21.15.42:0/854128188 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff738116500 0x7ff738130a80 secure :-1 s=THROTTLE_DONE pgs=299 cs=0 l=1 rx=0x7ff728002f30 tx=0x7ff7280109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:40.993 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:40.983+0000 7ff734ff9700 1 -- 172.21.15.42:0/854128188 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7ff7280127f0 con 0x7ff738116500 2019-12-05T05:44:40.993 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:40.983+0000 7ff737fff700 1 --2- 172.21.15.42:0/854128188 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff738116500 0x7ff738130a80 secure :-1 s=THROTTLE_DONE pgs=299 cs=0 l=1 rx=0x7ff728002f30 tx=0x7ff7280109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:40.993 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:40.983+0000 7ff734ff9700 1 --2- 172.21.15.42:0/854128188 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff71c03b270 0x7ff71c03d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:44:40.994 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:40.983+0000 7ff734ff9700 1 -- 172.21.15.42:0/854128188 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 8353+0+0 (secure 0 0 0) 0x7ff72804ab40 con 0x7ff738116500 2019-12-05T05:44:40.994 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:40.983+0000 7ff7377fe700 1 --2- 172.21.15.42:0/854128188 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff71c03b270 0x7ff71c03d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:44:40.994 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:40.983+0000 7ff7377fe700 1 --2- 172.21.15.42:0/854128188 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff71c03b270 0x7ff71c03d780 crc :-1 s=READY pgs=600 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:44:40.995 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:40.983+0000 7ff73e231700 1 -- 172.21.15.42:0/854128188 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff73805d110 con 0x7ff738116500 2019-12-05T05:44:41.000 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:40.995+0000 7ff737fff700 1 --2- 172.21.15.42:0/854128188 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff738116500 0x7ff738130a80 secure :-1 s=THROTTLE_DONE pgs=299 cs=0 l=1 rx=0x7ff728002f30 tx=0x7ff7280109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:41.000 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:40.995+0000 7ff734ff9700 1 -- 172.21.15.42:0/854128188 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7ff728031ba0 con 0x7ff738116500 2019-12-05T05:44:41.166 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:41.159+0000 7ff73e231700 1 -- 172.21.15.42:0/854128188 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff738134710 con 0x7ff738116500 2019-12-05T05:44:41.170 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:41.159+0000 7ff737fff700 1 --2- 172.21.15.42:0/854128188 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff738116500 0x7ff738130a80 secure :-1 s=THROTTLE_DONE pgs=299 cs=0 l=1 rx=0x7ff728002f30 tx=0x7ff7280109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:41.171 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:41.163+0000 7ff734ff9700 1 -- 172.21.15.42:0/854128188 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 168 v168) v1 ==== 95+0+2109 (secure 0 0 0) 0x7ff728031e00 con 0x7ff738116500 2019-12-05T05:44:41.171 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 168 2019-12-05T05:44:41.171 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:44:41.171 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":168,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7884,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/1127133319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1127133319},{"type":"v1","addr":"172.21.15.40:6833","nonce":1127133319}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":168}],"filesystems":[{"mdsmap":{"epoch":167,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:44:09.069915+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":76,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7755},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7755":{"gid":7755,"name":"a","rank":0,"incarnation":164,"state":"up:active","state_seq":22,"addr":"172.21.15.42:6835/358497296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":358497296},{"type":"v1","addr":"172.21.15.42:6835","nonce":358497296}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:44:41.172 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:41.167+0000 7ff73e231700 1 -- 172.21.15.42:0/854128188 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff71c03b270 msgr2=0x7ff71c03d780 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:41.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:41.167+0000 7ff73e231700 1 --2- 172.21.15.42:0/854128188 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff71c03b270 0x7ff71c03d780 crc :-1 s=READY pgs=600 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:44:41.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:41.167+0000 7ff73e231700 1 -- 172.21.15.42:0/854128188 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff738116500 msgr2=0x7ff738130a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:41.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:41.167+0000 7ff73e231700 1 --2- 172.21.15.42:0/854128188 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff738116500 0x7ff738130a80 secure :-1 s=READY pgs=299 cs=0 l=1 rx=0x7ff728002f30 tx=0x7ff7280109e0).stop 2019-12-05T05:44:41.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:41.167+0000 7ff73e231700 1 -- 172.21.15.42:0/854128188 shutdown_connections 2019-12-05T05:44:41.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:41.167+0000 7ff73e231700 1 --2- 172.21.15.42:0/854128188 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff71c03b270 0x7ff71c03d780 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:44:41.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:41.167+0000 7ff73e231700 1 --2- 172.21.15.42:0/854128188 >> v2:172.21.15.40:3300/0 conn(0x7ff7380a5100 0x7ff738133d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:44:41.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:41.167+0000 7ff73e231700 1 --2- 172.21.15.42:0/854128188 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff738116500 0x7ff738130a80 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:44:41.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:41.167+0000 7ff73e231700 1 -- 172.21.15.42:0/854128188 shutdown_connections 2019-12-05T05:44:41.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:41.167+0000 7ff73e231700 1 -- 172.21.15.42:0/854128188 wait complete. 2019-12-05T05:44:41.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:41.167+0000 7ff73e231700 1 -- 172.21.15.42:0/854128188 >> 172.21.15.42:0/854128188 conn(0x7ff738066fe0 msgr2=0x7ff73811c610 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:44:41.191 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T05:44:41.273 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:44:41.277 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:44:43.193 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:44:43.294 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:43.283+0000 7fd8d28b0700 1 Processor -- start 2019-12-05T05:44:43.294 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:43.283+0000 7fd8d28b0700 1 -- start start 2019-12-05T05:44:43.294 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:43.283+0000 7fd8d28b0700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd8cc08df80 0x7fd8cc09f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:44:43.295 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:43.283+0000 7fd8d28b0700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd8cc09f830 0x7fd8cc14efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:44:43.295 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:43.283+0000 7fd8d28b0700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8cc09fc60 con 0x7fd8cc0a7740 2019-12-05T05:44:43.295 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:43.283+0000 7fd8d28b0700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8cc0a0b00 con 0x7fd8cc09f830 2019-12-05T05:44:43.295 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:43.283+0000 7fd8d28b0700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8cc08e5b0 con 0x7fd8cc08df80 2019-12-05T05:44:43.295 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:43.287+0000 7fd8d164e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd8cc09f830 0x7fd8cc14efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:44:43.295 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:43.287+0000 7fd8d164e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd8cc09f830 0x7fd8cc14efe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:40216/0 (socket says 172.21.15.42:40216) 2019-12-05T05:44:43.295 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:43.287+0000 7fd8d164e700 1 -- 172.21.15.42:0/1225283850 learned_addr learned my addr 172.21.15.42:0/1225283850 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:44:43.296 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:43.287+0000 7fd8c3fff700 1 --2- 172.21.15.42:0/1225283850 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd8cc08df80 0x7fd8cc09f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:44:43.296 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:43.287+0000 7fd8d164e700 1 -- 172.21.15.42:0/1225283850 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd8cc08df80 msgr2=0x7fd8cc09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:44:43.296 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:43.287+0000 7fd8d164e700 1 --2- 172.21.15.42:0/1225283850 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd8cc08df80 0x7fd8cc09f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:44:43.296 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:43.287+0000 7fd8d164e700 1 -- 172.21.15.42:0/1225283850 >> v1:172.21.15.40:6789/0 conn(0x7fd8cc0a7740 legacy=0x7fd8cc07e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:43.296 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:43.287+0000 7fd8d164e700 1 -- 172.21.15.42:0/1225283850 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8cc0a0710 con 0x7fd8cc09f830 2019-12-05T05:44:43.296 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:43.287+0000 7fd8d164e700 1 --2- 172.21.15.42:0/1225283850 >> v2:172.21.15.42:3300/0 conn(0x7fd8cc09f830 0x7fd8cc14efe0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fd8c4008b10 tx=0x7fd8c40100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:43.297 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:43.287+0000 7fd8d164e700 1 --2- 172.21.15.42:0/1225283850 >> v2:172.21.15.42:3300/0 conn(0x7fd8cc09f830 0x7fd8cc14efe0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fd8c4008b10 tx=0x7fd8c40100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:43.297 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:43.287+0000 7fd8d164e700 1 --2- 172.21.15.42:0/1225283850 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd8cc09f830 0x7fd8cc14efe0 secure :-1 s=READY pgs=1394 cs=0 l=1 rx=0x7fd8c4008b10 tx=0x7fd8c40100b0).ready entity=mon.1 client_cookie=6ac28d3aea83b9cf server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:44:43.297 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:43.287+0000 7fd8d164e700 1 --2- 172.21.15.42:0/1225283850 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd8cc09f830 0x7fd8cc14efe0 secure :-1 s=THROTTLE_DONE pgs=1394 cs=0 l=1 rx=0x7fd8c4008b10 tx=0x7fd8c40100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:43.297 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:43.287+0000 7fd8d164e700 1 --2- 172.21.15.42:0/1225283850 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd8cc09f830 0x7fd8cc14efe0 secure :-1 s=THROTTLE_DONE pgs=1394 cs=0 l=1 rx=0x7fd8c4008b10 tx=0x7fd8c40100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:43.297 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:43.287+0000 7fd8d164e700 1 --2- 172.21.15.42:0/1225283850 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd8cc09f830 0x7fd8cc14efe0 secure :-1 s=THROTTLE_DONE pgs=1394 cs=0 l=1 rx=0x7fd8c4008b10 tx=0x7fd8c40100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:43.297 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:43.287+0000 7fd8c37fe700 1 -- 172.21.15.42:0/1225283850 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd8c4010da0 con 0x7fd8cc09f830 2019-12-05T05:44:43.297 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:43.287+0000 7fd8c37fe700 1 -- 172.21.15.42:0/1225283850 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8c40109c0 con 0x7fd8cc09f830 2019-12-05T05:44:43.298 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:43.287+0000 7fd8c37fe700 1 -- 172.21.15.42:0/1225283850 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd8c40114a0 con 0x7fd8cc09f830 2019-12-05T05:44:43.298 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:43.287+0000 7fd8d28b0700 1 -- 172.21.15.42:0/1225283850 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd8cc09f830 msgr2=0x7fd8cc14efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:43.298 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:43.287+0000 7fd8d28b0700 1 --2- 172.21.15.42:0/1225283850 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd8cc09f830 0x7fd8cc14efe0 secure :-1 s=READY pgs=1394 cs=0 l=1 rx=0x7fd8c4008b10 tx=0x7fd8c40100b0).stop 2019-12-05T05:44:43.298 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:43.287+0000 7fd8d28b0700 1 -- 172.21.15.42:0/1225283850 shutdown_connections 2019-12-05T05:44:43.298 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:43.287+0000 7fd8d28b0700 1 --2- 172.21.15.42:0/1225283850 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd8cc09f830 0x7fd8cc14efe0 unknown :-1 s=CLOSED pgs=1394 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:44:43.298 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:43.287+0000 7fd8d28b0700 1 --2- 172.21.15.42:0/1225283850 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd8cc08df80 0x7fd8cc09f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:44:43.298 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:43.287+0000 7fd8d28b0700 1 -- 172.21.15.42:0/1225283850 shutdown_connections 2019-12-05T05:44:43.299 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:43.287+0000 7fd8d28b0700 1 -- 172.21.15.42:0/1225283850 wait complete. 2019-12-05T05:44:43.299 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:43.287+0000 7fd8d28b0700 1 -- 172.21.15.42:0/1225283850 >> 172.21.15.42:0/1225283850 conn(0x7fd8cc0aabe0 msgr2=0x7fd8cc0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:44:43.300 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:43.291+0000 7fd8d28b0700 1 Processor -- start 2019-12-05T05:44:43.300 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:43.291+0000 7fd8d28b0700 1 -- start start 2019-12-05T05:44:43.300 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:43.291+0000 7fd8d28b0700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd8cc0a6050 0x7fd8cc113d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:44:43.300 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:43.291+0000 7fd8d28b0700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8cc0a6bc0 con 0x7fd8cc0a4c00 2019-12-05T05:44:43.300 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:43.291+0000 7fd8d28b0700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8cc139880 con 0x7fd8cc115b40 2019-12-05T05:44:43.300 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:43.291+0000 7fd8d28b0700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8cc0a6680 con 0x7fd8cc0a6050 2019-12-05T05:44:43.300 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:43.291+0000 7fd8d0e4d700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fd8cc115b40 0x7fd8cc07bbd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:40256/0 (socket says 172.21.15.42:40256) 2019-12-05T05:44:43.301 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:43.291+0000 7fd8d0e4d700 1 -- 172.21.15.42:0/1987584028 learned_addr learned my addr 172.21.15.42:0/1987584028 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:44:43.303 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:43.291+0000 7fd8c3fff700 1 --2- 172.21.15.42:0/1987584028 >> v2:172.21.15.40:3300/0 conn(0x7fd8cc0a6050 0x7fd8cc113d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:44:43.303 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:43.291+0000 7fd8c17fa700 1 -- 172.21.15.42:0/1987584028 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3140485236 0 0) 0x7fd8cc139880 con 0x7fd8cc115b40 2019-12-05T05:44:43.303 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:43.291+0000 7fd8c17fa700 1 -- 172.21.15.42:0/1987584028 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8ac0030b0 con 0x7fd8cc115b40 2019-12-05T05:44:43.304 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:43.291+0000 7fd8c3fff700 1 -- 172.21.15.42:0/1987584028 >> v1:172.21.15.42:6789/0 conn(0x7fd8cc115b40 legacy=0x7fd8cc07bbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:43.304 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:43.291+0000 7fd8c3fff700 1 -- 172.21.15.42:0/1987584028 >> v1:172.21.15.40:6789/0 conn(0x7fd8cc0a4c00 legacy=0x7fd8cc0a5030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:43.304 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:43.295+0000 7fd8c3fff700 1 -- 172.21.15.42:0/1987584028 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8cc13aac0 con 0x7fd8cc0a6050 2019-12-05T05:44:43.304 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:43.295+0000 7fd8c3fff700 1 --2- 172.21.15.42:0/1987584028 >> v2:172.21.15.40:3300/0 conn(0x7fd8cc0a6050 0x7fd8cc113d40 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fd8c8001ca0 tx=0x7fd8c80017b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:43.304 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:43.295+0000 7fd8c3fff700 1 --2- 172.21.15.42:0/1987584028 >> v2:172.21.15.40:3300/0 conn(0x7fd8cc0a6050 0x7fd8cc113d40 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fd8c8001ca0 tx=0x7fd8c80017b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:43.304 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:43.295+0000 7fd8c3fff700 1 --2- 172.21.15.42:0/1987584028 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd8cc0a6050 0x7fd8cc113d40 secure :-1 s=READY pgs=1080 cs=0 l=1 rx=0x7fd8c8001ca0 tx=0x7fd8c80017b0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:44:43.305 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:43.295+0000 7fd8c3fff700 1 --2- 172.21.15.42:0/1987584028 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd8cc0a6050 0x7fd8cc113d40 secure :-1 s=THROTTLE_DONE pgs=1080 cs=0 l=1 rx=0x7fd8c8001ca0 tx=0x7fd8c80017b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:43.305 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:43.295+0000 7fd8c17fa700 1 -- 172.21.15.42:0/1987584028 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd8c8001a60 con 0x7fd8cc0a6050 2019-12-05T05:44:43.305 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:43.295+0000 7fd8c3fff700 1 --2- 172.21.15.42:0/1987584028 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd8cc0a6050 0x7fd8cc113d40 secure :-1 s=THROTTLE_DONE pgs=1080 cs=0 l=1 rx=0x7fd8c8001ca0 tx=0x7fd8c80017b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:43.305 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:43.295+0000 7fd8c17fa700 1 -- 172.21.15.42:0/1987584028 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8c8001870 con 0x7fd8cc0a6050 2019-12-05T05:44:43.305 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:43.295+0000 7fd8c3fff700 1 --2- 172.21.15.42:0/1987584028 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd8cc0a6050 0x7fd8cc113d40 secure :-1 s=THROTTLE_DONE pgs=1080 cs=0 l=1 rx=0x7fd8c8001ca0 tx=0x7fd8c80017b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:43.305 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:43.295+0000 7fd8d28b0700 1 -- 172.21.15.42:0/1987584028 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8cc138870 con 0x7fd8cc0a6050 2019-12-05T05:44:43.305 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:43.295+0000 7fd8c17fa700 1 -- 172.21.15.42:0/1987584028 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd8bc001850 con 0x7fd8cc0a6050 2019-12-05T05:44:43.305 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:43.295+0000 7fd8d28b0700 1 -- 172.21.15.42:0/1987584028 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8cc138ae0 con 0x7fd8cc0a6050 2019-12-05T05:44:43.306 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:43.295+0000 7fd8c3fff700 1 --2- 172.21.15.42:0/1987584028 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd8cc0a6050 0x7fd8cc113d40 secure :-1 s=THROTTLE_DONE pgs=1080 cs=0 l=1 rx=0x7fd8c8001ca0 tx=0x7fd8c80017b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:43.306 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:43.295+0000 7fd8c17fa700 1 -- 172.21.15.42:0/1987584028 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fd8bc001b30 con 0x7fd8cc0a6050 2019-12-05T05:44:43.306 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:43.295+0000 7fd8c3fff700 1 --2- 172.21.15.42:0/1987584028 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd8cc0a6050 0x7fd8cc113d40 secure :-1 s=THROTTLE_DONE pgs=1080 cs=0 l=1 rx=0x7fd8c8001ca0 tx=0x7fd8c80017b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:43.306 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:43.295+0000 7fd8c17fa700 1 --2- 172.21.15.42:0/1987584028 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd8ac03b210 0x7fd8ac03d720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:44:43.306 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:43.295+0000 7fd8c17fa700 1 -- 172.21.15.42:0/1987584028 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 8353+0+0 (secure 0 0 0) 0x7fd8bc04a7d0 con 0x7fd8cc0a6050 2019-12-05T05:44:43.306 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:43.295+0000 7fd8d0e4d700 1 --2- 172.21.15.42:0/1987584028 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd8ac03b210 0x7fd8ac03d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:44:43.307 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:43.295+0000 7fd8d0e4d700 1 --2- 172.21.15.42:0/1987584028 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd8ac03b210 0x7fd8ac03d720 crc :-1 s=READY pgs=601 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:44:43.307 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:43.299+0000 7fd8d28b0700 1 -- 172.21.15.42:0/1987584028 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8cc0a6480 con 0x7fd8cc0a6050 2019-12-05T05:44:43.314 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:43.307+0000 7fd8c3fff700 1 --2- 172.21.15.42:0/1987584028 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd8cc0a6050 0x7fd8cc113d40 secure :-1 s=THROTTLE_DONE pgs=1080 cs=0 l=1 rx=0x7fd8c8001ca0 tx=0x7fd8c80017b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:43.314 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:43.307+0000 7fd8c17fa700 1 -- 172.21.15.42:0/1987584028 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fd8bc031830 con 0x7fd8cc0a6050 2019-12-05T05:44:43.485 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:43.475+0000 7fd8d28b0700 1 -- 172.21.15.42:0/1987584028 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd8cc05d110 con 0x7fd8cc0a6050 2019-12-05T05:44:43.485 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:43.479+0000 7fd8c3fff700 1 --2- 172.21.15.42:0/1987584028 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd8cc0a6050 0x7fd8cc113d40 secure :-1 s=THROTTLE_DONE pgs=1080 cs=0 l=1 rx=0x7fd8c8001ca0 tx=0x7fd8c80017b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:43.485 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:43.479+0000 7fd8c17fa700 1 -- 172.21.15.42:0/1987584028 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 168 v168) v1 ==== 95+0+2109 (secure 0 0 0) 0x7fd8bc031a90 con 0x7fd8cc0a6050 2019-12-05T05:44:43.486 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 168 2019-12-05T05:44:43.486 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:44:43.486 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":168,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7884,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/1127133319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1127133319},{"type":"v1","addr":"172.21.15.40:6833","nonce":1127133319}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":168}],"filesystems":[{"mdsmap":{"epoch":167,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:44:09.069915+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":76,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7755},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7755":{"gid":7755,"name":"a","rank":0,"incarnation":164,"state":"up:active","state_seq":22,"addr":"172.21.15.42:6835/358497296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":358497296},{"type":"v1","addr":"172.21.15.42:6835","nonce":358497296}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:44:43.489 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:43.483+0000 7fd8d28b0700 1 -- 172.21.15.42:0/1987584028 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd8ac03b210 msgr2=0x7fd8ac03d720 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:43.489 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:43.483+0000 7fd8d28b0700 1 --2- 172.21.15.42:0/1987584028 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd8ac03b210 0x7fd8ac03d720 crc :-1 s=READY pgs=601 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:44:43.489 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:43.483+0000 7fd8d28b0700 1 -- 172.21.15.42:0/1987584028 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd8cc0a6050 msgr2=0x7fd8cc113d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:43.489 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:43.483+0000 7fd8d28b0700 1 --2- 172.21.15.42:0/1987584028 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd8cc0a6050 0x7fd8cc113d40 secure :-1 s=READY pgs=1080 cs=0 l=1 rx=0x7fd8c8001ca0 tx=0x7fd8c80017b0).stop 2019-12-05T05:44:43.489 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:43.483+0000 7fd8d28b0700 1 -- 172.21.15.42:0/1987584028 shutdown_connections 2019-12-05T05:44:43.489 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:43.483+0000 7fd8d28b0700 1 --2- 172.21.15.42:0/1987584028 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd8ac03b210 0x7fd8ac03d720 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:44:43.490 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:43.483+0000 7fd8d28b0700 1 --2- 172.21.15.42:0/1987584028 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd8cc0a6050 0x7fd8cc113d40 unknown :-1 s=CLOSED pgs=1080 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:44:43.490 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:43.483+0000 7fd8d28b0700 1 -- 172.21.15.42:0/1987584028 shutdown_connections 2019-12-05T05:44:43.490 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:43.483+0000 7fd8d28b0700 1 -- 172.21.15.42:0/1987584028 wait complete. 2019-12-05T05:44:43.490 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:43.483+0000 7fd8d28b0700 1 -- 172.21.15.42:0/1987584028 >> 172.21.15.42:0/1987584028 conn(0x7fd8cc0a97d0 msgr2=0x7fd8cc09f220 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:44:43.500 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T05:44:45.502 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:44:45.604 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.591+0000 7ff461f95700 1 Processor -- start 2019-12-05T05:44:45.605 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.591+0000 7ff461f95700 1 -- start start 2019-12-05T05:44:45.605 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.595+0000 7ff461f95700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff45c0a7730 0x7ff45c08df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:44:45.605 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.595+0000 7ff461f95700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff45c09fad0 0x7ff45c07eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:44:45.605 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.595+0000 7ff461f95700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff45c0a0700 con 0x7ff45c07e710 2019-12-05T05:44:45.605 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.595+0000 7ff461f95700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff45c0a0af0 con 0x7ff45c09fad0 2019-12-05T05:44:45.605 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.595+0000 7ff461f95700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff45c08e5a0 con 0x7ff45c0a7730 2019-12-05T05:44:45.606 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.595+0000 7ff45bfff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff45c0a7730 0x7ff45c08df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:44:45.606 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.595+0000 7ff45bfff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff45c0a7730 0x7ff45c08df70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:40226/0 (socket says 172.21.15.42:40226) 2019-12-05T05:44:45.606 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.595+0000 7ff45bfff700 1 -- 172.21.15.42:0/3073468428 learned_addr learned my addr 172.21.15.42:0/3073468428 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:44:45.606 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.595+0000 7ff45b7fe700 1 --1- 172.21.15.42:0/3073468428 >> v1:172.21.15.40:6789/0 conn(0x7ff45c07e710 0x7ff45c09f2d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:49704/0 (socket says 172.21.15.42:49704) 2019-12-05T05:44:45.606 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.595+0000 7ff460d33700 1 --2- 172.21.15.42:0/3073468428 >> v2:172.21.15.40:3300/0 conn(0x7ff45c09fad0 0x7ff45c07eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:44:45.606 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.595+0000 7ff45bfff700 1 -- 172.21.15.42:0/3073468428 >> v2:172.21.15.40:3300/0 conn(0x7ff45c09fad0 msgr2=0x7ff45c07eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:44:45.606 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.595+0000 7ff45bfff700 1 --2- 172.21.15.42:0/3073468428 >> v2:172.21.15.40:3300/0 conn(0x7ff45c09fad0 0x7ff45c07eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:44:45.607 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.595+0000 7ff45bfff700 1 -- 172.21.15.42:0/3073468428 >> v1:172.21.15.40:6789/0 conn(0x7ff45c07e710 legacy=0x7ff45c09f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:45.607 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.595+0000 7ff45bfff700 1 -- 172.21.15.42:0/3073468428 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff45c07f090 con 0x7ff45c0a7730 2019-12-05T05:44:45.607 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.595+0000 7ff45bfff700 1 --2- 172.21.15.42:0/3073468428 >> v2:172.21.15.42:3300/0 conn(0x7ff45c0a7730 0x7ff45c08df70 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7ff44c002f30 tx=0x7ff44c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:45.607 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.595+0000 7ff45bfff700 1 --2- 172.21.15.42:0/3073468428 >> v2:172.21.15.42:3300/0 conn(0x7ff45c0a7730 0x7ff45c08df70 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7ff44c002f30 tx=0x7ff44c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:45.607 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.595+0000 7ff45bfff700 1 --2- 172.21.15.42:0/3073468428 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff45c0a7730 0x7ff45c08df70 secure :-1 s=READY pgs=1396 cs=0 l=1 rx=0x7ff44c002f30 tx=0x7ff44c00fd30).ready entity=mon.1 client_cookie=62325e3da8ca633f server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:44:45.607 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.595+0000 7ff45bfff700 1 --2- 172.21.15.42:0/3073468428 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff45c0a7730 0x7ff45c08df70 secure :-1 s=THROTTLE_DONE pgs=1396 cs=0 l=1 rx=0x7ff44c002f30 tx=0x7ff44c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:45.607 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.595+0000 7ff45affd700 1 -- 172.21.15.42:0/3073468428 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff44c008c90 con 0x7ff45c0a7730 2019-12-05T05:44:45.608 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.595+0000 7ff45bfff700 1 --2- 172.21.15.42:0/3073468428 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff45c0a7730 0x7ff45c08df70 secure :-1 s=THROTTLE_DONE pgs=1396 cs=0 l=1 rx=0x7ff44c002f30 tx=0x7ff44c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:45.608 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.595+0000 7ff45bfff700 1 --2- 172.21.15.42:0/3073468428 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff45c0a7730 0x7ff45c08df70 secure :-1 s=THROTTLE_DONE pgs=1396 cs=0 l=1 rx=0x7ff44c002f30 tx=0x7ff44c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:45.608 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.595+0000 7ff45affd700 1 -- 172.21.15.42:0/3073468428 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff44c0088b0 con 0x7ff45c0a7730 2019-12-05T05:44:45.608 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.595+0000 7ff45affd700 1 -- 172.21.15.42:0/3073468428 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff44c011ad0 con 0x7ff45c0a7730 2019-12-05T05:44:45.608 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.595+0000 7ff461f95700 1 -- 172.21.15.42:0/3073468428 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff45c0a7730 msgr2=0x7ff45c08df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:45.608 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.595+0000 7ff461f95700 1 --2- 172.21.15.42:0/3073468428 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff45c0a7730 0x7ff45c08df70 secure :-1 s=READY pgs=1396 cs=0 l=1 rx=0x7ff44c002f30 tx=0x7ff44c00fd30).stop 2019-12-05T05:44:45.608 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.599+0000 7ff461f95700 1 -- 172.21.15.42:0/3073468428 shutdown_connections 2019-12-05T05:44:45.609 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.599+0000 7ff461f95700 1 --2- 172.21.15.42:0/3073468428 >> v2:172.21.15.40:3300/0 conn(0x7ff45c09fad0 0x7ff45c07eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:44:45.609 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.599+0000 7ff461f95700 1 --2- 172.21.15.42:0/3073468428 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff45c0a7730 0x7ff45c08df70 unknown :-1 s=CLOSED pgs=1396 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:44:45.609 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.599+0000 7ff461f95700 1 -- 172.21.15.42:0/3073468428 shutdown_connections 2019-12-05T05:44:45.609 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.599+0000 7ff461f95700 1 -- 172.21.15.42:0/3073468428 wait complete. 2019-12-05T05:44:45.609 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.599+0000 7ff461f95700 1 -- 172.21.15.42:0/3073468428 >> 172.21.15.42:0/3073468428 conn(0x7ff45c0aabd0 msgr2=0x7ff45c0a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:44:45.614 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.599+0000 7ff461f95700 1 Processor -- start 2019-12-05T05:44:45.615 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.599+0000 7ff461f95700 1 -- start start 2019-12-05T05:44:45.615 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.599+0000 7ff461f95700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff45c144e70 0x7ff45c143960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:44:45.615 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.599+0000 7ff461f95700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff45c143eb0 0x7ff45c123f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:44:45.615 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.599+0000 7ff461f95700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff45c078a20 con 0x7ff45c1244e0 2019-12-05T05:44:45.615 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.599+0000 7ff461f95700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff45c144900 con 0x7ff45c144e70 2019-12-05T05:44:45.615 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.599+0000 7ff461f95700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff45c077530 con 0x7ff45c143eb0 2019-12-05T05:44:45.615 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.599+0000 7ff45bfff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff45c144e70 0x7ff45c143960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:44:45.616 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.599+0000 7ff45bfff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff45c144e70 0x7ff45c143960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:40232/0 (socket says 172.21.15.42:40232) 2019-12-05T05:44:45.617 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.599+0000 7ff45bfff700 1 -- 172.21.15.42:0/3234035282 learned_addr learned my addr 172.21.15.42:0/3234035282 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:44:45.617 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.599+0000 7ff45b7fe700 1 --2- 172.21.15.42:0/3234035282 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff45c143eb0 0x7ff45c123f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:44:45.617 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.603+0000 7ff45bfff700 1 -- 172.21.15.42:0/3234035282 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff45c143eb0 msgr2=0x7ff45c123f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:45.617 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.603+0000 7ff45bfff700 1 --2- 172.21.15.42:0/3234035282 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff45c143eb0 0x7ff45c123f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:44:45.617 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.603+0000 7ff45bfff700 1 -- 172.21.15.42:0/3234035282 >> v1:172.21.15.42:6789/0 conn(0x7ff45c1244e0 legacy=0x7ff45c126ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:45.617 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.603+0000 7ff458ff9700 1 -- 172.21.15.42:0/3234035282 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3358531731 0 0) 0x7ff454000e90 con 0x7ff45c1244e0 2019-12-05T05:44:45.618 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.603+0000 7ff45bfff700 1 -- 172.21.15.42:0/3234035282 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff45c078170 con 0x7ff45c144e70 2019-12-05T05:44:45.618 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.603+0000 7ff45bfff700 1 --2- 172.21.15.42:0/3234035282 >> v2:172.21.15.42:3300/0 conn(0x7ff45c144e70 0x7ff45c143960 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7ff45c0776f0 tx=0x7ff44c008b90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:45.618 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.603+0000 7ff45bfff700 1 --2- 172.21.15.42:0/3234035282 >> v2:172.21.15.42:3300/0 conn(0x7ff45c144e70 0x7ff45c143960 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7ff45c0776f0 tx=0x7ff44c008b90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:45.619 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.603+0000 7ff45bfff700 1 --2- 172.21.15.42:0/3234035282 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff45c144e70 0x7ff45c143960 secure :-1 s=READY pgs=1398 cs=0 l=1 rx=0x7ff45c0776f0 tx=0x7ff44c008b90).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:44:45.619 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.603+0000 7ff45bfff700 1 --2- 172.21.15.42:0/3234035282 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff45c144e70 0x7ff45c143960 secure :-1 s=THROTTLE_DONE pgs=1398 cs=0 l=1 rx=0x7ff45c0776f0 tx=0x7ff44c008b90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:45.619 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.603+0000 7ff458ff9700 1 -- 172.21.15.42:0/3234035282 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff44c00f470 con 0x7ff45c144e70 2019-12-05T05:44:45.619 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.603+0000 7ff45bfff700 1 --2- 172.21.15.42:0/3234035282 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff45c144e70 0x7ff45c143960 secure :-1 s=THROTTLE_DONE pgs=1398 cs=0 l=1 rx=0x7ff45c0776f0 tx=0x7ff44c008b90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:45.619 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.603+0000 7ff45bfff700 1 --2- 172.21.15.42:0/3234035282 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff45c144e70 0x7ff45c143960 secure :-1 s=THROTTLE_DONE pgs=1398 cs=0 l=1 rx=0x7ff45c0776f0 tx=0x7ff44c008b90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:45.620 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.603+0000 7ff458ff9700 1 -- 172.21.15.42:0/3234035282 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff44c0116f0 con 0x7ff45c144e70 2019-12-05T05:44:45.620 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.603+0000 7ff461f95700 1 -- 172.21.15.42:0/3234035282 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff45c123910 con 0x7ff45c144e70 2019-12-05T05:44:45.620 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.603+0000 7ff458ff9700 1 -- 172.21.15.42:0/3234035282 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff44c005730 con 0x7ff45c144e70 2019-12-05T05:44:45.620 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.603+0000 7ff461f95700 1 -- 172.21.15.42:0/3234035282 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff45c144420 con 0x7ff45c144e70 2019-12-05T05:44:45.620 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.603+0000 7ff45bfff700 1 --2- 172.21.15.42:0/3234035282 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff45c144e70 0x7ff45c143960 secure :-1 s=THROTTLE_DONE pgs=1398 cs=0 l=1 rx=0x7ff45c0776f0 tx=0x7ff44c008b90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:45.620 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.603+0000 7ff458ff9700 1 -- 172.21.15.42:0/3234035282 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7ff44c0059d0 con 0x7ff45c144e70 2019-12-05T05:44:45.620 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.603+0000 7ff45bfff700 1 --2- 172.21.15.42:0/3234035282 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff45c144e70 0x7ff45c143960 secure :-1 s=THROTTLE_DONE pgs=1398 cs=0 l=1 rx=0x7ff45c0776f0 tx=0x7ff44c008b90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:45.621 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.603+0000 7ff458ff9700 1 --2- 172.21.15.42:0/3234035282 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff43c03ac60 0x7ff43c03d170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:44:45.621 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.603+0000 7ff458ff9700 1 -- 172.21.15.42:0/3234035282 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 8439+0+0 (secure 0 0 0) 0x7ff44c04b9f0 con 0x7ff45c144e70 2019-12-05T05:44:45.621 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.603+0000 7ff45b7fe700 1 --2- 172.21.15.42:0/3234035282 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff43c03ac60 0x7ff43c03d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:44:45.621 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.603+0000 7ff45b7fe700 1 --2- 172.21.15.42:0/3234035282 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff43c03ac60 0x7ff43c03d170 crc :-1 s=READY pgs=602 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:44:45.622 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.607+0000 7ff461f95700 1 -- 172.21.15.42:0/3234035282 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff45c05d100 con 0x7ff45c144e70 2019-12-05T05:44:45.622 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.615+0000 7ff45bfff700 1 --2- 172.21.15.42:0/3234035282 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff45c144e70 0x7ff45c143960 secure :-1 s=THROTTLE_DONE pgs=1398 cs=0 l=1 rx=0x7ff45c0776f0 tx=0x7ff44c008b90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:45.622 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.615+0000 7ff458ff9700 1 -- 172.21.15.42:0/3234035282 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7ff44c032cb0 con 0x7ff45c144e70 2019-12-05T05:44:45.795 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.787+0000 7ff461f95700 1 -- 172.21.15.42:0/3234035282 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff45c05b8b0 con 0x7ff45c144e70 2019-12-05T05:44:45.796 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.791+0000 7ff45bfff700 1 --2- 172.21.15.42:0/3234035282 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff45c144e70 0x7ff45c143960 secure :-1 s=THROTTLE_DONE pgs=1398 cs=0 l=1 rx=0x7ff45c0776f0 tx=0x7ff44c008b90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:45.796 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.791+0000 7ff458ff9700 1 -- 172.21.15.42:0/3234035282 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 169 v169) v1 ==== 95+0+1774 (secure 0 0 0) 0x7ff44c0306e0 con 0x7ff45c144e70 2019-12-05T05:44:45.796 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 169 2019-12-05T05:44:45.796 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:44:45.797 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":169,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":169,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:44:43.966427+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7884},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7884":{"gid":7884,"name":"b","rank":0,"incarnation":169,"state":"up:replay","state_seq":2,"addr":"172.21.15.40:6833/1127133319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1127133319},{"type":"v1","addr":"172.21.15.40:6833","nonce":1127133319}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:44:45.799 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.791+0000 7ff461f95700 1 -- 172.21.15.42:0/3234035282 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff43c03ac60 msgr2=0x7ff43c03d170 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:45.799 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.791+0000 7ff461f95700 1 --2- 172.21.15.42:0/3234035282 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff43c03ac60 0x7ff43c03d170 crc :-1 s=READY pgs=602 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:44:45.799 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.791+0000 7ff461f95700 1 -- 172.21.15.42:0/3234035282 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff45c144e70 msgr2=0x7ff45c143960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:45.799 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.791+0000 7ff461f95700 1 --2- 172.21.15.42:0/3234035282 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff45c144e70 0x7ff45c143960 secure :-1 s=READY pgs=1398 cs=0 l=1 rx=0x7ff45c0776f0 tx=0x7ff44c008b90).stop 2019-12-05T05:44:45.800 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.791+0000 7ff461f95700 1 -- 172.21.15.42:0/3234035282 shutdown_connections 2019-12-05T05:44:45.800 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.791+0000 7ff461f95700 1 --2- 172.21.15.42:0/3234035282 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff43c03ac60 0x7ff43c03d170 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:44:45.800 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.791+0000 7ff461f95700 1 --2- 172.21.15.42:0/3234035282 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff45c144e70 0x7ff45c143960 unknown :-1 s=CLOSED pgs=1398 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:44:45.800 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.791+0000 7ff461f95700 1 --2- 172.21.15.42:0/3234035282 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff45c143eb0 0x7ff45c123f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:44:45.800 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.791+0000 7ff461f95700 1 -- 172.21.15.42:0/3234035282 shutdown_connections 2019-12-05T05:44:45.800 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.791+0000 7ff461f95700 1 -- 172.21.15.42:0/3234035282 wait complete. 2019-12-05T05:44:45.800 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.791+0000 7ff461f95700 1 -- 172.21.15.42:0/3234035282 >> 172.21.15.42:0/3234035282 conn(0x7ff45c0a7730 msgr2=0x7ff45c0a9630 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:44:45.809 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a down, removed from mdsmap 2019-12-05T05:44:45.809 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2019-12-05T05:44:45.809 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:44:45.909 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.895+0000 7f388fed1700 1 Processor -- start 2019-12-05T05:44:45.909 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.895+0000 7f388fed1700 1 -- start start 2019-12-05T05:44:45.909 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.899+0000 7f388fed1700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f38880a7740 0x7f388808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:44:45.909 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.899+0000 7f388fed1700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f388809fae0 0x7f388807eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:44:45.910 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.899+0000 7f388fed1700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38880a0710 con 0x7f388807e720 2019-12-05T05:44:45.910 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.899+0000 7f388fed1700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38880a0b00 con 0x7f388809fae0 2019-12-05T05:44:45.910 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.899+0000 7f388fed1700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f388808e5b0 con 0x7f38880a7740 2019-12-05T05:44:45.910 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.899+0000 7f388ec6f700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f388809fae0 0x7f388807eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:44:45.910 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.899+0000 7f388ec6f700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f388809fae0 0x7f388807eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:40244/0 (socket says 172.21.15.42:40244) 2019-12-05T05:44:45.910 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.899+0000 7f388ec6f700 1 -- 172.21.15.42:0/359822560 learned_addr learned my addr 172.21.15.42:0/359822560 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:44:45.910 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.899+0000 7f388e46e700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f38880a7740 0x7f388808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:44:45.911 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.899+0000 7f388e46e700 1 -- 172.21.15.42:0/359822560 >> v2:172.21.15.42:3300/0 conn(0x7f388809fae0 msgr2=0x7f388807eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:44:45.911 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.899+0000 7f388e46e700 1 --2- 172.21.15.42:0/359822560 >> v2:172.21.15.42:3300/0 conn(0x7f388809fae0 0x7f388807eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:44:45.911 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.899+0000 7f388e46e700 1 -- 172.21.15.42:0/359822560 >> v1:172.21.15.40:6789/0 conn(0x7f388807e720 legacy=0x7f388809f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:45.911 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.899+0000 7f388e46e700 1 -- 172.21.15.42:0/359822560 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f388807f0a0 con 0x7f38880a7740 2019-12-05T05:44:45.911 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.899+0000 7f388ec6f700 1 --2- 172.21.15.42:0/359822560 >> v2:172.21.15.42:3300/0 conn(0x7f388809fae0 0x7f388807eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:44:45.911 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.899+0000 7f388e46e700 1 --2- 172.21.15.42:0/359822560 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f38880a7740 0x7f388808df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f3884008790 tx=0x7f387c001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:45.911 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.899+0000 7f388e46e700 1 --2- 172.21.15.42:0/359822560 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f38880a7740 0x7f388808df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f3884008790 tx=0x7f387c001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:45.912 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.899+0000 7f388e46e700 1 --2- 172.21.15.42:0/359822560 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f38880a7740 0x7f388808df80 secure :-1 s=READY pgs=300 cs=0 l=1 rx=0x7f3884008790 tx=0x7f387c001150).ready entity=mon.2 client_cookie=1a0b7e41e8eea172 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:44:45.912 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.899+0000 7f388e46e700 1 --2- 172.21.15.42:0/359822560 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f38880a7740 0x7f388808df80 secure :-1 s=THROTTLE_DONE pgs=300 cs=0 l=1 rx=0x7f3884008790 tx=0x7f387c001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:45.912 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.899+0000 7f388e46e700 1 --2- 172.21.15.42:0/359822560 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f38880a7740 0x7f388808df80 secure :-1 s=THROTTLE_DONE pgs=300 cs=0 l=1 rx=0x7f3884008790 tx=0x7f387c001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:45.912 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.899+0000 7f388e46e700 1 --2- 172.21.15.42:0/359822560 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f38880a7740 0x7f388808df80 secure :-1 s=THROTTLE_DONE pgs=300 cs=0 l=1 rx=0x7f3884008790 tx=0x7f387c001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:45.912 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.899+0000 7f388d46c700 1 -- 172.21.15.42:0/359822560 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3884008bb0 con 0x7f38880a7740 2019-12-05T05:44:45.912 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.899+0000 7f388d46c700 1 -- 172.21.15.42:0/359822560 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38840087d0 con 0x7f38880a7740 2019-12-05T05:44:45.913 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.899+0000 7f388d46c700 1 -- 172.21.15.42:0/359822560 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3884011760 con 0x7f38880a7740 2019-12-05T05:44:45.913 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.899+0000 7f388fed1700 1 -- 172.21.15.42:0/359822560 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f38880a7740 msgr2=0x7f388808df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:45.913 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.899+0000 7f388fed1700 1 --2- 172.21.15.42:0/359822560 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f38880a7740 0x7f388808df80 secure :-1 s=READY pgs=300 cs=0 l=1 rx=0x7f3884008790 tx=0x7f387c001150).stop 2019-12-05T05:44:45.913 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.899+0000 7f388fed1700 1 -- 172.21.15.42:0/359822560 shutdown_connections 2019-12-05T05:44:45.913 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.899+0000 7f388fed1700 1 --2- 172.21.15.42:0/359822560 >> v2:172.21.15.42:3300/0 conn(0x7f388809fae0 0x7f388807eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:44:45.913 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.899+0000 7f388fed1700 1 --2- 172.21.15.42:0/359822560 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f38880a7740 0x7f388808df80 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:44:45.913 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.899+0000 7f388fed1700 1 -- 172.21.15.42:0/359822560 shutdown_connections 2019-12-05T05:44:45.913 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.899+0000 7f388fed1700 1 -- 172.21.15.42:0/359822560 wait complete. 2019-12-05T05:44:45.914 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.899+0000 7f388fed1700 1 -- 172.21.15.42:0/359822560 >> 172.21.15.42:0/359822560 conn(0x7f38880aabe0 msgr2=0x7f38880a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:44:45.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.903+0000 7f388fed1700 1 Processor -- start 2019-12-05T05:44:45.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.903+0000 7f388fed1700 1 -- start start 2019-12-05T05:44:45.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.907+0000 7f388fed1700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3888116500 0x7f38881181a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:44:45.918 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.907+0000 7f388fed1700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f3888133a10 0x7f3888133e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:44:45.918 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.907+0000 7f388fed1700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38881186f0 con 0x7f388808eb70 2019-12-05T05:44:45.918 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.907+0000 7f388fed1700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38881189c0 con 0x7f3888133a10 2019-12-05T05:44:45.918 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.907+0000 7f388fed1700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3888133770 con 0x7f3888116500 2019-12-05T05:44:45.918 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.907+0000 7f388e46e700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3888116500 0x7f38881181a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:44:45.918 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.907+0000 7f388e46e700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3888116500 0x7f38881181a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:40712/0 (socket says 172.21.15.42:40712) 2019-12-05T05:44:45.918 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.907+0000 7f388e46e700 1 -- 172.21.15.42:0/1821516561 learned_addr learned my addr 172.21.15.42:0/1821516561 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:44:45.919 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.907+0000 7f388dc6d700 1 --2- 172.21.15.42:0/1821516561 >> v2:172.21.15.40:3300/0 conn(0x7f3888133a10 0x7f3888133e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:44:45.919 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.907+0000 7f387affd700 1 -- 172.21.15.42:0/1821516561 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3020784557 0 0) 0x7f38881186f0 con 0x7f388808eb70 2019-12-05T05:44:45.919 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.907+0000 7f388e46e700 1 -- 172.21.15.42:0/1821516561 >> v2:172.21.15.40:3300/0 conn(0x7f3888133a10 msgr2=0x7f3888133e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:45.919 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.907+0000 7f388e46e700 1 --2- 172.21.15.42:0/1821516561 >> v2:172.21.15.40:3300/0 conn(0x7f3888133a10 0x7f3888133e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:44:45.919 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.907+0000 7f388e46e700 1 -- 172.21.15.42:0/1821516561 >> v1:172.21.15.42:6789/0 conn(0x7f388808eb70 legacy=0x7f388808efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:45.919 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.907+0000 7f388e46e700 1 -- 172.21.15.42:0/1821516561 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3888134390 con 0x7f3888116500 2019-12-05T05:44:45.920 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.907+0000 7f388e46e700 1 --2- 172.21.15.42:0/1821516561 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3888116500 0x7f38881181a0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f3884008790 tx=0x7f38840085f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:45.920 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.907+0000 7f388e46e700 1 --2- 172.21.15.42:0/1821516561 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3888116500 0x7f38881181a0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f3884008790 tx=0x7f38840085f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:45.920 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.907+0000 7f388e46e700 1 --2- 172.21.15.42:0/1821516561 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3888116500 0x7f38881181a0 secure :-1 s=READY pgs=301 cs=0 l=1 rx=0x7f3884008790 tx=0x7f38840085f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:44:45.920 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.907+0000 7f388e46e700 1 --2- 172.21.15.42:0/1821516561 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3888116500 0x7f38881181a0 secure :-1 s=THROTTLE_DONE pgs=301 cs=0 l=1 rx=0x7f3884008790 tx=0x7f38840085f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:45.920 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.907+0000 7f388e46e700 1 --2- 172.21.15.42:0/1821516561 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3888116500 0x7f38881181a0 secure :-1 s=THROTTLE_DONE pgs=301 cs=0 l=1 rx=0x7f3884008790 tx=0x7f38840085f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:45.921 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.907+0000 7f387affd700 1 -- 172.21.15.42:0/1821516561 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f388400f280 con 0x7f3888116500 2019-12-05T05:44:45.921 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.907+0000 7f388e46e700 1 --2- 172.21.15.42:0/1821516561 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3888116500 0x7f38881181a0 secure :-1 s=THROTTLE_DONE pgs=301 cs=0 l=1 rx=0x7f3884008790 tx=0x7f38840085f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:45.922 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.907+0000 7f387affd700 1 -- 172.21.15.42:0/1821516561 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3884008db0 con 0x7f3888116500 2019-12-05T05:44:45.922 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.907+0000 7f388fed1700 1 -- 172.21.15.42:0/1821516561 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38880a4280 con 0x7f3888116500 2019-12-05T05:44:45.922 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.907+0000 7f387affd700 1 -- 172.21.15.42:0/1821516561 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3884005540 con 0x7f3888116500 2019-12-05T05:44:45.922 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.907+0000 7f388fed1700 1 -- 172.21.15.42:0/1821516561 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f388808f940 con 0x7f3888116500 2019-12-05T05:44:45.922 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.907+0000 7f388e46e700 1 --2- 172.21.15.42:0/1821516561 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3888116500 0x7f38881181a0 secure :-1 s=THROTTLE_DONE pgs=301 cs=0 l=1 rx=0x7f3884008790 tx=0x7f38840085f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:45.923 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.907+0000 7f387affd700 1 -- 172.21.15.42:0/1821516561 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f3884005850 con 0x7f3888116500 2019-12-05T05:44:45.923 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.907+0000 7f388e46e700 1 --2- 172.21.15.42:0/1821516561 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3888116500 0x7f38881181a0 secure :-1 s=THROTTLE_DONE pgs=301 cs=0 l=1 rx=0x7f3884008790 tx=0x7f38840085f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:45.923 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.911+0000 7f387affd700 1 --2- 172.21.15.42:0/1821516561 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f387003ad70 0x7f387003d280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:44:45.923 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.911+0000 7f387affd700 1 -- 172.21.15.42:0/1821516561 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 8439+0+0 (secure 0 0 0) 0x7f388404b780 con 0x7f3888116500 2019-12-05T05:44:45.923 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.911+0000 7f388dc6d700 1 --2- 172.21.15.42:0/1821516561 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f387003ad70 0x7f387003d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:44:45.924 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.911+0000 7f388dc6d700 1 --2- 172.21.15.42:0/1821516561 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f387003ad70 0x7f387003d280 crc :-1 s=READY pgs=603 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:44:45.924 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.911+0000 7f388fed1700 1 -- 172.21.15.42:0/1821516561 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f388805d110 con 0x7f3888116500 2019-12-05T05:44:45.926 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.919+0000 7f388e46e700 1 --2- 172.21.15.42:0/1821516561 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3888116500 0x7f38881181a0 secure :-1 s=THROTTLE_DONE pgs=301 cs=0 l=1 rx=0x7f3884008790 tx=0x7f38840085f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:45.927 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:45.919+0000 7f387affd700 1 -- 172.21.15.42:0/1821516561 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f3884007f50 con 0x7f3888116500 2019-12-05T05:44:46.095 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:46.087+0000 7f388fed1700 1 -- 172.21.15.42:0/1821516561 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f388805b8c0 con 0x7f3888116500 2019-12-05T05:44:46.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:46.091+0000 7f388e46e700 1 --2- 172.21.15.42:0/1821516561 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3888116500 0x7f38881181a0 secure :-1 s=THROTTLE_DONE pgs=301 cs=0 l=1 rx=0x7f3884008790 tx=0x7f38840085f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:46.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:46.091+0000 7f387affd700 1 -- 172.21.15.42:0/1821516561 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 169 v169) v1 ==== 95+0+1774 (secure 0 0 0) 0x7f3884032a60 con 0x7f3888116500 2019-12-05T05:44:46.097 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 169 2019-12-05T05:44:46.097 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:44:46.097 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":169,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":169,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:44:43.966427+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7884},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7884":{"gid":7884,"name":"b","rank":0,"incarnation":169,"state":"up:replay","state_seq":2,"addr":"172.21.15.40:6833/1127133319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1127133319},{"type":"v1","addr":"172.21.15.40:6833","nonce":1127133319}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:44:46.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:46.091+0000 7f388fed1700 1 -- 172.21.15.42:0/1821516561 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f387003ad70 msgr2=0x7f387003d280 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:46.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:46.091+0000 7f388fed1700 1 --2- 172.21.15.42:0/1821516561 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f387003ad70 0x7f387003d280 crc :-1 s=READY pgs=603 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:44:46.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:46.091+0000 7f388fed1700 1 -- 172.21.15.42:0/1821516561 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3888116500 msgr2=0x7f38881181a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:46.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:46.091+0000 7f388fed1700 1 --2- 172.21.15.42:0/1821516561 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3888116500 0x7f38881181a0 secure :-1 s=READY pgs=301 cs=0 l=1 rx=0x7f3884008790 tx=0x7f38840085f0).stop 2019-12-05T05:44:46.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:46.091+0000 7f388fed1700 1 -- 172.21.15.42:0/1821516561 shutdown_connections 2019-12-05T05:44:46.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:46.091+0000 7f388fed1700 1 --2- 172.21.15.42:0/1821516561 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f387003ad70 0x7f387003d280 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:44:46.100 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:46.091+0000 7f388fed1700 1 --2- 172.21.15.42:0/1821516561 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3888116500 0x7f38881181a0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:44:46.100 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:46.091+0000 7f388fed1700 1 --2- 172.21.15.42:0/1821516561 >> v2:172.21.15.40:3300/0 conn(0x7f3888133a10 0x7f3888133e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:44:46.100 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:46.091+0000 7f388fed1700 1 -- 172.21.15.42:0/1821516561 shutdown_connections 2019-12-05T05:44:46.100 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:46.091+0000 7f388fed1700 1 -- 172.21.15.42:0/1821516561 wait complete. 2019-12-05T05:44:46.100 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:46.091+0000 7f388fed1700 1 -- 172.21.15.42:0/1821516561 >> 172.21.15.42:0/1821516561 conn(0x7f388807da80 msgr2=0x7f38880a9790 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:44:46.109 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 169, "feature_flags": { "enable_multiple": false, "ever_enabled_multiple": false }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2019-12-05T05:03:35.604738+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 169, "ever_allowed_features": 2, "explicitly_allowed_features": 2, "failed": [], "flags": 18, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_7884": { "addr": "172.21.15.40:6833/1127133319", "addrs": { "addrvec": [ { "addr": "172.21.15.40:6832", "nonce": 1127133319, "type": "v2" }, { "addr": "172.21.15.40:6833", "nonce": 1127133319, "type": "v1" } ] }, "export_targets": [], "features": 4540138292836696063, "flags": 0, "gid": 7884, "incarnation": 169, "name": "b", "rank": 0, "state": "up:replay", "state_seq": 2 } }, "last_failure": 0, "last_failure_osd_epoch": 77, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 2, "min_compat_client": "0 (unknown)", "modified": "2019-12-05T05:44:43.966427+0000", "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7884 } } } ], "standbys": [] } 2019-12-05T05:44:48.110 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:44:48.217 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:48.203+0000 7f5cce2f4700 1 Processor -- start 2019-12-05T05:44:48.217 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:48.203+0000 7f5cce2f4700 1 -- start start 2019-12-05T05:44:48.218 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:48.207+0000 7f5cce2f4700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5cc808df80 0x7f5cc809f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:44:48.218 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:48.207+0000 7f5cce2f4700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5cc809f830 0x7f5cc814efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:44:48.218 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:48.207+0000 7f5cce2f4700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5cc809fc60 con 0x7f5cc80a7740 2019-12-05T05:44:48.218 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:48.207+0000 7f5cce2f4700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5cc80a0b00 con 0x7f5cc808df80 2019-12-05T05:44:48.218 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:48.207+0000 7f5cce2f4700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5cc808e5b0 con 0x7f5cc809f830 2019-12-05T05:44:48.219 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:48.207+0000 7f5ccc891700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f5cc80a7740 0x7f5cc807e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:40294/0 (socket says 172.21.15.42:40294) 2019-12-05T05:44:48.220 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:48.207+0000 7f5ccc891700 1 -- 172.21.15.42:0/2155006186 learned_addr learned my addr 172.21.15.42:0/2155006186 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:44:48.220 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:48.207+0000 7f5cc7fff700 1 --2- 172.21.15.42:0/2155006186 >> v2:172.21.15.40:3300/0 conn(0x7f5cc808df80 0x7f5cc809f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:44:48.220 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:48.207+0000 7f5ccd092700 1 --2- 172.21.15.42:0/2155006186 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5cc809f830 0x7f5cc814efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:44:48.221 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:48.207+0000 7f5cc77fe700 1 -- 172.21.15.42:0/2155006186 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4202549917 0 0) 0x7f5cc809fc60 con 0x7f5cc80a7740 2019-12-05T05:44:48.221 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:48.207+0000 7f5cc77fe700 1 -- 172.21.15.42:0/2155006186 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5cb00030b0 con 0x7f5cc80a7740 2019-12-05T05:44:48.221 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:48.207+0000 7f5cc7fff700 1 -- 172.21.15.42:0/2155006186 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5cc809f830 msgr2=0x7f5cc814efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:44:48.221 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:48.207+0000 7f5cc7fff700 1 --2- 172.21.15.42:0/2155006186 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5cc809f830 0x7f5cc814efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:44:48.221 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:48.207+0000 7f5cc7fff700 1 -- 172.21.15.42:0/2155006186 >> v1:172.21.15.42:6789/0 conn(0x7f5cc80a7740 legacy=0x7f5cc807e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:48.221 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:48.207+0000 7f5cc7fff700 1 -- 172.21.15.42:0/2155006186 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cc80a0710 con 0x7f5cc808df80 2019-12-05T05:44:48.221 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:48.207+0000 7f5ccd092700 1 --2- 172.21.15.42:0/2155006186 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5cc809f830 0x7f5cc814efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:44:48.222 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:48.207+0000 7f5cc7fff700 1 --2- 172.21.15.42:0/2155006186 >> v2:172.21.15.40:3300/0 conn(0x7f5cc808df80 0x7f5cc809f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f5cbc008790 tx=0x7f5cbc00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:48.222 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:48.207+0000 7f5cc7fff700 1 --2- 172.21.15.42:0/2155006186 >> v2:172.21.15.40:3300/0 conn(0x7f5cc808df80 0x7f5cc809f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f5cbc008790 tx=0x7f5cbc00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:48.222 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:48.207+0000 7f5cc7fff700 1 --2- 172.21.15.42:0/2155006186 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5cc808df80 0x7f5cc809f2e0 secure :-1 s=READY pgs=1084 cs=0 l=1 rx=0x7f5cbc008790 tx=0x7f5cbc00fc70).ready entity=mon.0 client_cookie=2dc350a82cbbfacd server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:44:48.222 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:48.207+0000 7f5cc7fff700 1 --2- 172.21.15.42:0/2155006186 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5cc808df80 0x7f5cc809f2e0 secure :-1 s=THROTTLE_DONE pgs=1084 cs=0 l=1 rx=0x7f5cbc008790 tx=0x7f5cbc00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:48.222 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:48.207+0000 7f5cc77fe700 1 -- 172.21.15.42:0/2155006186 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5cbc008bb0 con 0x7f5cc808df80 2019-12-05T05:44:48.222 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:48.207+0000 7f5cc7fff700 1 --2- 172.21.15.42:0/2155006186 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5cc808df80 0x7f5cc809f2e0 secure :-1 s=THROTTLE_DONE pgs=1084 cs=0 l=1 rx=0x7f5cbc008790 tx=0x7f5cbc00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:48.222 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:48.207+0000 7f5cc77fe700 1 -- 172.21.15.42:0/2155006186 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cbc0087d0 con 0x7f5cc808df80 2019-12-05T05:44:48.223 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:48.207+0000 7f5cc7fff700 1 --2- 172.21.15.42:0/2155006186 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5cc808df80 0x7f5cc809f2e0 secure :-1 s=THROTTLE_DONE pgs=1084 cs=0 l=1 rx=0x7f5cbc008790 tx=0x7f5cbc00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:48.223 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:48.207+0000 7f5cc77fe700 1 -- 172.21.15.42:0/2155006186 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5cbc011860 con 0x7f5cc808df80 2019-12-05T05:44:48.223 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:48.207+0000 7f5cce2f4700 1 -- 172.21.15.42:0/2155006186 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5cc808df80 msgr2=0x7f5cc809f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:48.223 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:48.207+0000 7f5cce2f4700 1 --2- 172.21.15.42:0/2155006186 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5cc808df80 0x7f5cc809f2e0 secure :-1 s=READY pgs=1084 cs=0 l=1 rx=0x7f5cbc008790 tx=0x7f5cbc00fc70).stop 2019-12-05T05:44:48.223 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:48.207+0000 7f5cce2f4700 1 -- 172.21.15.42:0/2155006186 shutdown_connections 2019-12-05T05:44:48.223 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:48.207+0000 7f5cce2f4700 1 --2- 172.21.15.42:0/2155006186 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5cc809f830 0x7f5cc814efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:44:48.223 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:48.207+0000 7f5cce2f4700 1 --2- 172.21.15.42:0/2155006186 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5cc808df80 0x7f5cc809f2e0 unknown :-1 s=CLOSED pgs=1084 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:44:48.224 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:48.207+0000 7f5cce2f4700 1 -- 172.21.15.42:0/2155006186 shutdown_connections 2019-12-05T05:44:48.224 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:48.207+0000 7f5cce2f4700 1 -- 172.21.15.42:0/2155006186 wait complete. 2019-12-05T05:44:48.224 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:48.207+0000 7f5cce2f4700 1 -- 172.21.15.42:0/2155006186 >> 172.21.15.42:0/2155006186 conn(0x7f5cc80aabe0 msgr2=0x7f5cc80a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:44:48.227 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:48.211+0000 7f5cce2f4700 1 Processor -- start 2019-12-05T05:44:48.228 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:48.211+0000 7f5cce2f4700 1 -- start start 2019-12-05T05:44:48.228 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:48.215+0000 7f5cce2f4700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5cc81176a0 0x7f5cc811ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:44:48.229 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:48.215+0000 7f5cce2f4700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5cc811b820 con 0x7f5cc80a6d90 2019-12-05T05:44:48.229 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:48.215+0000 7f5cce2f4700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5cc811bde0 con 0x7f5cc8117e20 2019-12-05T05:44:48.229 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:48.215+0000 7f5cce2f4700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5cc811b580 con 0x7f5cc81176a0 2019-12-05T05:44:48.229 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:48.215+0000 7f5cc7fff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f5cc8117e20 0x7f5cc8136850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:40302/0 (socket says 172.21.15.42:40302) 2019-12-05T05:44:48.229 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:48.215+0000 7f5cc7fff700 1 -- 172.21.15.42:0/2610443438 learned_addr learned my addr 172.21.15.42:0/2610443438 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:44:48.229 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:48.215+0000 7f5ccd092700 1 --2- 172.21.15.42:0/2610443438 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5cc81176a0 0x7f5cc811ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:44:48.230 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:48.215+0000 7f5cc57fa700 1 -- 172.21.15.42:0/2610443438 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1794328829 0 0) 0x7f5cc811bde0 con 0x7f5cc8117e20 2019-12-05T05:44:48.230 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:48.215+0000 7f5cc57fa700 1 -- 172.21.15.42:0/2610443438 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ca80030b0 con 0x7f5cc8117e20 2019-12-05T05:44:48.230 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:48.215+0000 7f5cc57fa700 1 -- 172.21.15.42:0/2610443438 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 548780730 0 0) 0x7f5cc811b820 con 0x7f5cc80a6d90 2019-12-05T05:44:48.230 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:48.215+0000 7f5cc57fa700 1 -- 172.21.15.42:0/2610443438 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5cc811bde0 con 0x7f5cc80a6d90 2019-12-05T05:44:48.230 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:48.215+0000 7f5ccd092700 1 -- 172.21.15.42:0/2610443438 >> v1:172.21.15.42:6789/0 conn(0x7f5cc8117e20 legacy=0x7f5cc8136850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:48.231 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:48.215+0000 7f5ccd092700 1 -- 172.21.15.42:0/2610443438 >> v1:172.21.15.40:6789/0 conn(0x7f5cc80a6d90 legacy=0x7f5cc807c010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:48.231 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:48.215+0000 7f5ccd092700 1 -- 172.21.15.42:0/2610443438 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cc811b2c0 con 0x7f5cc81176a0 2019-12-05T05:44:48.231 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:48.215+0000 7f5ccd092700 1 --2- 172.21.15.42:0/2610443438 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5cc81176a0 0x7f5cc811ad70 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f5cc00032b0 tx=0x7f5cc0010e80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:48.231 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:48.215+0000 7f5ccd092700 1 --2- 172.21.15.42:0/2610443438 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5cc81176a0 0x7f5cc811ad70 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f5cc00032b0 tx=0x7f5cc0010e80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:48.231 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:48.215+0000 7f5ccd092700 1 --2- 172.21.15.42:0/2610443438 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5cc81176a0 0x7f5cc811ad70 secure :-1 s=READY pgs=302 cs=0 l=1 rx=0x7f5cc00032b0 tx=0x7f5cc0010e80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:44:48.231 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:48.215+0000 7f5ccd092700 1 --2- 172.21.15.42:0/2610443438 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5cc81176a0 0x7f5cc811ad70 secure :-1 s=THROTTLE_DONE pgs=302 cs=0 l=1 rx=0x7f5cc00032b0 tx=0x7f5cc0010e80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:48.231 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:48.215+0000 7f5cc57fa700 1 -- 172.21.15.42:0/2610443438 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5cc00058c0 con 0x7f5cc81176a0 2019-12-05T05:44:48.232 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:48.215+0000 7f5cce2f4700 1 -- 172.21.15.42:0/2610443438 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5cc811c010 con 0x7f5cc81176a0 2019-12-05T05:44:48.232 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:48.215+0000 7f5ccd092700 1 --2- 172.21.15.42:0/2610443438 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5cc81176a0 0x7f5cc811ad70 secure :-1 s=THROTTLE_DONE pgs=302 cs=0 l=1 rx=0x7f5cc00032b0 tx=0x7f5cc0010e80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:48.232 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:48.215+0000 7f5cc57fa700 1 -- 172.21.15.42:0/2610443438 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cc000e160 con 0x7f5cc81176a0 2019-12-05T05:44:48.232 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:48.215+0000 7f5ccd092700 1 --2- 172.21.15.42:0/2610443438 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5cc81176a0 0x7f5cc811ad70 secure :-1 s=THROTTLE_DONE pgs=302 cs=0 l=1 rx=0x7f5cc00032b0 tx=0x7f5cc0010e80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:48.232 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:48.215+0000 7f5cce2f4700 1 -- 172.21.15.42:0/2610443438 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5cc813a070 con 0x7f5cc81176a0 2019-12-05T05:44:48.232 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:48.215+0000 7f5cc57fa700 1 -- 172.21.15.42:0/2610443438 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5cc000c8a0 con 0x7f5cc81176a0 2019-12-05T05:44:48.233 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:48.215+0000 7f5ccd092700 1 --2- 172.21.15.42:0/2610443438 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5cc81176a0 0x7f5cc811ad70 secure :-1 s=THROTTLE_DONE pgs=302 cs=0 l=1 rx=0x7f5cc00032b0 tx=0x7f5cc0010e80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:48.233 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:48.215+0000 7f5cc57fa700 1 -- 172.21.15.42:0/2610443438 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f5cc000cb40 con 0x7f5cc81176a0 2019-12-05T05:44:48.233 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:48.215+0000 7f5ccd092700 1 --2- 172.21.15.42:0/2610443438 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5cc81176a0 0x7f5cc811ad70 secure :-1 s=THROTTLE_DONE pgs=302 cs=0 l=1 rx=0x7f5cc00032b0 tx=0x7f5cc0010e80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:48.233 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:48.219+0000 7f5cc57fa700 1 --2- 172.21.15.42:0/2610443438 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5ca803b480 0x7f5ca803d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:44:48.233 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:48.219+0000 7f5cc57fa700 1 -- 172.21.15.42:0/2610443438 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 8439+0+0 (secure 0 0 0) 0x7f5cc004ae00 con 0x7f5cc81176a0 2019-12-05T05:44:48.233 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:48.219+0000 7f5ccc891700 1 --2- 172.21.15.42:0/2610443438 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5ca803b480 0x7f5ca803d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:44:48.234 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:48.219+0000 7f5ccc891700 1 --2- 172.21.15.42:0/2610443438 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5ca803b480 0x7f5ca803d990 crc :-1 s=READY pgs=604 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:44:48.234 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:48.219+0000 7f5cce2f4700 1 -- 172.21.15.42:0/2610443438 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5cc805d110 con 0x7f5cc81176a0 2019-12-05T05:44:48.237 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:48.231+0000 7f5ccd092700 1 --2- 172.21.15.42:0/2610443438 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5cc81176a0 0x7f5cc811ad70 secure :-1 s=THROTTLE_DONE pgs=302 cs=0 l=1 rx=0x7f5cc00032b0 tx=0x7f5cc0010e80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:48.237 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:48.231+0000 7f5cc57fa700 1 -- 172.21.15.42:0/2610443438 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f5cc811b2c0 con 0x7f5cc81176a0 2019-12-05T05:44:48.413 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:48.407+0000 7f5cce2f4700 1 -- 172.21.15.42:0/2610443438 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5cc805b8c0 con 0x7f5cc81176a0 2019-12-05T05:44:48.414 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:48.407+0000 7f5ccd092700 1 --2- 172.21.15.42:0/2610443438 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5cc81176a0 0x7f5cc811ad70 secure :-1 s=THROTTLE_DONE pgs=302 cs=0 l=1 rx=0x7f5cc00032b0 tx=0x7f5cc0010e80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:48.414 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:48.407+0000 7f5cc57fa700 1 -- 172.21.15.42:0/2610443438 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 171 v171) v1 ==== 95+0+1775 (secure 0 0 0) 0x7f5cc0031e80 con 0x7f5cc81176a0 2019-12-05T05:44:48.416 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 171 2019-12-05T05:44:48.416 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:44:48.416 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":171,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":171,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:44:47.993178+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7884},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7884":{"gid":7884,"name":"b","rank":0,"incarnation":169,"state":"up:rejoin","state_seq":10,"addr":"172.21.15.40:6833/1127133319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1127133319},{"type":"v1","addr":"172.21.15.40:6833","nonce":1127133319}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:44:48.419 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:48.411+0000 7f5cce2f4700 1 -- 172.21.15.42:0/2610443438 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5ca803b480 msgr2=0x7f5ca803d990 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:48.419 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:48.411+0000 7f5cce2f4700 1 --2- 172.21.15.42:0/2610443438 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5ca803b480 0x7f5ca803d990 crc :-1 s=READY pgs=604 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:44:48.419 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:48.411+0000 7f5cce2f4700 1 -- 172.21.15.42:0/2610443438 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5cc81176a0 msgr2=0x7f5cc811ad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:48.419 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:48.411+0000 7f5cce2f4700 1 --2- 172.21.15.42:0/2610443438 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5cc81176a0 0x7f5cc811ad70 secure :-1 s=READY pgs=302 cs=0 l=1 rx=0x7f5cc00032b0 tx=0x7f5cc0010e80).stop 2019-12-05T05:44:48.420 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:48.411+0000 7f5cce2f4700 1 -- 172.21.15.42:0/2610443438 shutdown_connections 2019-12-05T05:44:48.420 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:48.411+0000 7f5cce2f4700 1 --2- 172.21.15.42:0/2610443438 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5ca803b480 0x7f5ca803d990 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:44:48.420 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:48.411+0000 7f5cce2f4700 1 --2- 172.21.15.42:0/2610443438 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5cc81176a0 0x7f5cc811ad70 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:44:48.420 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:48.411+0000 7f5cce2f4700 1 -- 172.21.15.42:0/2610443438 shutdown_connections 2019-12-05T05:44:48.420 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:48.411+0000 7f5cce2f4700 1 -- 172.21.15.42:0/2610443438 wait complete. 2019-12-05T05:44:48.420 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:48.411+0000 7f5cce2f4700 1 -- 172.21.15.42:0/2610443438 >> 172.21.15.42:0/2610443438 conn(0x7f5cc80a8410 msgr2=0x7f5cc8059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:44:48.436 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2019-12-05T05:44:50.436 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:44:50.549 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:50.539+0000 7f6bed8e5700 1 Processor -- start 2019-12-05T05:44:50.549 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:50.539+0000 7f6bed8e5700 1 -- start start 2019-12-05T05:44:50.549 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:50.539+0000 7f6bed8e5700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6be808df80 0x7f6be809f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:44:50.549 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:50.539+0000 7f6bed8e5700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6be809fc60 con 0x7f6be809f830 2019-12-05T05:44:50.549 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:50.539+0000 7f6bed8e5700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6be80a0710 con 0x7f6be80a7740 2019-12-05T05:44:50.550 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:50.539+0000 7f6bed8e5700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6be80a0b00 con 0x7f6be808df80 2019-12-05T05:44:50.550 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:50.539+0000 7f6bdffff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6be808df80 0x7f6be809f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:44:50.550 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:50.539+0000 7f6bdffff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6be808df80 0x7f6be809f2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:40284/0 (socket says 172.21.15.42:40284) 2019-12-05T05:44:50.550 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:50.539+0000 7f6bdffff700 1 -- 172.21.15.42:0/785616308 learned_addr learned my addr 172.21.15.42:0/785616308 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:44:50.550 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:50.539+0000 7f6bdffff700 1 -- 172.21.15.42:0/785616308 >> v1:172.21.15.42:6789/0 conn(0x7f6be80a7740 legacy=0x7f6be807e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:50.550 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:50.539+0000 7f6bdffff700 1 -- 172.21.15.42:0/785616308 >> v1:172.21.15.40:6789/0 conn(0x7f6be809f830 legacy=0x7f6be814efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:50.550 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:50.539+0000 7f6bdffff700 1 -- 172.21.15.42:0/785616308 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6be807ef20 con 0x7f6be808df80 2019-12-05T05:44:50.551 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:50.539+0000 7f6bdffff700 1 --2- 172.21.15.42:0/785616308 >> v2:172.21.15.42:3300/0 conn(0x7f6be808df80 0x7f6be809f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f6bd0008790 tx=0x7f6be00014d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:50.551 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:50.539+0000 7f6bdffff700 1 --2- 172.21.15.42:0/785616308 >> v2:172.21.15.42:3300/0 conn(0x7f6be808df80 0x7f6be809f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f6bd0008790 tx=0x7f6be00014d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:50.551 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:50.539+0000 7f6bdffff700 1 --2- 172.21.15.42:0/785616308 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6be808df80 0x7f6be809f2e0 secure :-1 s=READY pgs=1403 cs=0 l=1 rx=0x7f6bd0008790 tx=0x7f6be00014d0).ready entity=mon.1 client_cookie=42acf20845d9594d server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:44:50.551 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:50.539+0000 7f6bdffff700 1 --2- 172.21.15.42:0/785616308 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6be808df80 0x7f6be809f2e0 secure :-1 s=THROTTLE_DONE pgs=1403 cs=0 l=1 rx=0x7f6bd0008790 tx=0x7f6be00014d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:50.551 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:50.539+0000 7f6bdffff700 1 --2- 172.21.15.42:0/785616308 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6be808df80 0x7f6be809f2e0 secure :-1 s=THROTTLE_DONE pgs=1403 cs=0 l=1 rx=0x7f6bd0008790 tx=0x7f6be00014d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:50.551 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:50.539+0000 7f6be6ffd700 1 -- 172.21.15.42:0/785616308 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6bd0012060 con 0x7f6be808df80 2019-12-05T05:44:50.551 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:50.539+0000 7f6bdffff700 1 --2- 172.21.15.42:0/785616308 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6be808df80 0x7f6be809f2e0 secure :-1 s=THROTTLE_DONE pgs=1403 cs=0 l=1 rx=0x7f6bd0008790 tx=0x7f6be00014d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:50.552 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:50.539+0000 7f6be6ffd700 1 -- 172.21.15.42:0/785616308 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bd00110c0 con 0x7f6be808df80 2019-12-05T05:44:50.552 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:50.539+0000 7f6be6ffd700 1 -- 172.21.15.42:0/785616308 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6bd00116a0 con 0x7f6be808df80 2019-12-05T05:44:50.552 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:50.539+0000 7f6bed8e5700 1 -- 172.21.15.42:0/785616308 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6be808df80 msgr2=0x7f6be809f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:50.552 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:50.539+0000 7f6bed8e5700 1 --2- 172.21.15.42:0/785616308 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6be808df80 0x7f6be809f2e0 secure :-1 s=READY pgs=1403 cs=0 l=1 rx=0x7f6bd0008790 tx=0x7f6be00014d0).stop 2019-12-05T05:44:50.566 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:50.543+0000 7f6bed8e5700 1 -- 172.21.15.42:0/785616308 shutdown_connections 2019-12-05T05:44:50.566 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:50.543+0000 7f6bed8e5700 1 --2- 172.21.15.42:0/785616308 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6be808df80 0x7f6be809f2e0 unknown :-1 s=CLOSED pgs=1403 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:44:50.566 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:50.543+0000 7f6bed8e5700 1 -- 172.21.15.42:0/785616308 shutdown_connections 2019-12-05T05:44:50.566 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:50.543+0000 7f6bed8e5700 1 -- 172.21.15.42:0/785616308 wait complete. 2019-12-05T05:44:50.566 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:50.543+0000 7f6bed8e5700 1 -- 172.21.15.42:0/785616308 >> 172.21.15.42:0/785616308 conn(0x7f6be80aabe0 msgr2=0x7f6be80a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:44:50.570 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:50.543+0000 7f6bed8e5700 1 Processor -- start 2019-12-05T05:44:50.570 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:50.543+0000 7f6bed8e5700 1 -- start start 2019-12-05T05:44:50.570 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:50.543+0000 7f6bed8e5700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6be80a67f0 0x7f6be8143c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:44:50.570 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:50.543+0000 7f6bed8e5700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6be8144ae0 0x7f6be8144f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:44:50.570 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:50.543+0000 7f6bed8e5700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6be80a4250 con 0x7f6be80a3cd0 2019-12-05T05:44:50.571 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:50.543+0000 7f6bed8e5700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6be8145460 con 0x7f6be80a67f0 2019-12-05T05:44:50.571 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:50.543+0000 7f6bed8e5700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6be81442b0 con 0x7f6be8144ae0 2019-12-05T05:44:50.571 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:50.547+0000 7f6bdffff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6be80a67f0 0x7f6be8143c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:44:50.571 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:50.547+0000 7f6be77fe700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f6be80a3cd0 0x7f6be807d920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:40326/0 (socket says 172.21.15.42:40326) 2019-12-05T05:44:50.571 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:50.547+0000 7f6be77fe700 1 -- 172.21.15.42:0/1658731712 learned_addr learned my addr 172.21.15.42:0/1658731712 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:44:50.571 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:50.547+0000 7f6be7fff700 1 --2- 172.21.15.42:0/1658731712 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6be8144ae0 0x7f6be8144f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:44:50.571 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:50.547+0000 7f6bdffff700 1 -- 172.21.15.42:0/1658731712 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6be8144ae0 msgr2=0x7f6be8144f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:50.572 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:50.547+0000 7f6bdffff700 1 --2- 172.21.15.42:0/1658731712 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6be8144ae0 0x7f6be8144f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:44:50.572 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:50.547+0000 7f6bdffff700 1 -- 172.21.15.42:0/1658731712 >> v1:172.21.15.42:6789/0 conn(0x7f6be80a3cd0 legacy=0x7f6be807d920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:50.572 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:50.547+0000 7f6be4ff9700 1 -- 172.21.15.42:0/1658731712 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3151206419 0 0) 0x7f6be80a4250 con 0x7f6be80a3cd0 2019-12-05T05:44:50.572 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:50.547+0000 7f6bdffff700 1 -- 172.21.15.42:0/1658731712 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6be807e120 con 0x7f6be80a67f0 2019-12-05T05:44:50.572 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:50.547+0000 7f6bdffff700 1 --2- 172.21.15.42:0/1658731712 >> v2:172.21.15.42:3300/0 conn(0x7f6be80a67f0 0x7f6be8143c30 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f6be8144470 tx=0x7f6bd0011e10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:50.576 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:50.547+0000 7f6bdffff700 1 --2- 172.21.15.42:0/1658731712 >> v2:172.21.15.42:3300/0 conn(0x7f6be80a67f0 0x7f6be8143c30 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f6be8144470 tx=0x7f6bd0011e10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:50.576 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:50.547+0000 7f6bdffff700 1 --2- 172.21.15.42:0/1658731712 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6be80a67f0 0x7f6be8143c30 secure :-1 s=READY pgs=1405 cs=0 l=1 rx=0x7f6be8144470 tx=0x7f6bd0011e10).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:44:50.576 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:50.547+0000 7f6bdffff700 1 --2- 172.21.15.42:0/1658731712 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6be80a67f0 0x7f6be8143c30 secure :-1 s=THROTTLE_DONE pgs=1405 cs=0 l=1 rx=0x7f6be8144470 tx=0x7f6bd0011e10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:50.576 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:50.547+0000 7f6bdffff700 1 --2- 172.21.15.42:0/1658731712 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6be80a67f0 0x7f6be8143c30 secure :-1 s=THROTTLE_DONE pgs=1405 cs=0 l=1 rx=0x7f6be8144470 tx=0x7f6bd0011e10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:50.577 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:50.547+0000 7f6be4ff9700 1 -- 172.21.15.42:0/1658731712 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6bd000f470 con 0x7f6be80a67f0 2019-12-05T05:44:50.577 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:50.547+0000 7f6bdffff700 1 --2- 172.21.15.42:0/1658731712 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6be80a67f0 0x7f6be8143c30 secure :-1 s=THROTTLE_DONE pgs=1405 cs=0 l=1 rx=0x7f6be8144470 tx=0x7f6bd0011e10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:50.577 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:50.547+0000 7f6be4ff9700 1 -- 172.21.15.42:0/1658731712 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bd0008aa0 con 0x7f6be80a67f0 2019-12-05T05:44:50.577 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:50.547+0000 7f6be4ff9700 1 -- 172.21.15.42:0/1658731712 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6bd0005730 con 0x7f6be80a67f0 2019-12-05T05:44:50.577 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:50.547+0000 7f6bed8e5700 1 -- 172.21.15.42:0/1658731712 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6be808eb70 con 0x7f6be80a67f0 2019-12-05T05:44:50.577 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:50.547+0000 7f6bed8e5700 1 -- 172.21.15.42:0/1658731712 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6be808f0a0 con 0x7f6be80a67f0 2019-12-05T05:44:50.577 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:50.547+0000 7f6bdffff700 1 --2- 172.21.15.42:0/1658731712 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6be80a67f0 0x7f6be8143c30 secure :-1 s=THROTTLE_DONE pgs=1405 cs=0 l=1 rx=0x7f6be8144470 tx=0x7f6bd0011e10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:50.578 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:50.547+0000 7f6be4ff9700 1 -- 172.21.15.42:0/1658731712 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f6bd0005a40 con 0x7f6be80a67f0 2019-12-05T05:44:50.578 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:50.547+0000 7f6bdffff700 1 --2- 172.21.15.42:0/1658731712 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6be80a67f0 0x7f6be8143c30 secure :-1 s=THROTTLE_DONE pgs=1405 cs=0 l=1 rx=0x7f6be8144470 tx=0x7f6bd0011e10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:50.578 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:50.547+0000 7f6be4ff9700 1 --2- 172.21.15.42:0/1658731712 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6bcc03ac90 0x7f6bcc03d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:44:50.578 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:50.547+0000 7f6be4ff9700 1 -- 172.21.15.42:0/1658731712 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 8439+0+0 (secure 0 0 0) 0x7f6bd004d430 con 0x7f6be80a67f0 2019-12-05T05:44:50.578 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:50.547+0000 7f6be77fe700 1 --2- 172.21.15.42:0/1658731712 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6bcc03ac90 0x7f6bcc03d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:44:50.579 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:50.551+0000 7f6be77fe700 1 --2- 172.21.15.42:0/1658731712 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6bcc03ac90 0x7f6bcc03d1a0 crc :-1 s=READY pgs=605 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:44:50.579 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:50.555+0000 7f6bed8e5700 1 -- 172.21.15.42:0/1658731712 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6be805d110 con 0x7f6be80a67f0 2019-12-05T05:44:50.580 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:50.559+0000 7f6bdffff700 1 --2- 172.21.15.42:0/1658731712 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6be80a67f0 0x7f6be8143c30 secure :-1 s=THROTTLE_DONE pgs=1405 cs=0 l=1 rx=0x7f6be8144470 tx=0x7f6bd0011e10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:50.580 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:50.559+0000 7f6be4ff9700 1 -- 172.21.15.42:0/1658731712 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f6bd0005d50 con 0x7f6be80a67f0 2019-12-05T05:44:50.764 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:50.759+0000 7f6bed8e5700 1 -- 172.21.15.42:0/1658731712 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6be805b8c0 con 0x7f6be80a67f0 2019-12-05T05:44:50.765 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:50.759+0000 7f6bdffff700 1 --2- 172.21.15.42:0/1658731712 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6be80a67f0 0x7f6be8143c30 secure :-1 s=THROTTLE_DONE pgs=1405 cs=0 l=1 rx=0x7f6be8144470 tx=0x7f6bd0011e10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:50.765 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:50.759+0000 7f6be4ff9700 1 -- 172.21.15.42:0/1658731712 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 172 v172) v1 ==== 95+0+1775 (secure 0 0 0) 0x7f6bd0035320 con 0x7f6be80a67f0 2019-12-05T05:44:50.765 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 172 2019-12-05T05:44:50.766 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:44:50.766 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":172,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":172,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:44:49.006174+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7884},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7884":{"gid":7884,"name":"b","rank":0,"incarnation":169,"state":"up:active","state_seq":11,"addr":"172.21.15.40:6833/1127133319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1127133319},{"type":"v1","addr":"172.21.15.40:6833","nonce":1127133319}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:44:50.777 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:50.763+0000 7f6bed8e5700 1 -- 172.21.15.42:0/1658731712 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6bcc03ac90 msgr2=0x7f6bcc03d1a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:50.778 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:50.763+0000 7f6bed8e5700 1 --2- 172.21.15.42:0/1658731712 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6bcc03ac90 0x7f6bcc03d1a0 crc :-1 s=READY pgs=605 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:44:50.778 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:50.763+0000 7f6bed8e5700 1 -- 172.21.15.42:0/1658731712 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6be80a67f0 msgr2=0x7f6be8143c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:50.778 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:50.763+0000 7f6bed8e5700 1 --2- 172.21.15.42:0/1658731712 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6be80a67f0 0x7f6be8143c30 secure :-1 s=READY pgs=1405 cs=0 l=1 rx=0x7f6be8144470 tx=0x7f6bd0011e10).stop 2019-12-05T05:44:50.778 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:50.763+0000 7f6bed8e5700 1 -- 172.21.15.42:0/1658731712 shutdown_connections 2019-12-05T05:44:50.778 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:50.763+0000 7f6bed8e5700 1 --2- 172.21.15.42:0/1658731712 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6bcc03ac90 0x7f6bcc03d1a0 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:44:50.778 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:50.763+0000 7f6bed8e5700 1 --2- 172.21.15.42:0/1658731712 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6be8144ae0 0x7f6be8144f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:44:50.779 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:50.763+0000 7f6bed8e5700 1 --2- 172.21.15.42:0/1658731712 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6be80a67f0 0x7f6be8143c30 unknown :-1 s=CLOSED pgs=1405 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:44:50.780 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:50.763+0000 7f6bed8e5700 1 -- 172.21.15.42:0/1658731712 shutdown_connections 2019-12-05T05:44:50.780 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:50.763+0000 7f6bed8e5700 1 -- 172.21.15.42:0/1658731712 wait complete. 2019-12-05T05:44:50.780 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:50.763+0000 7f6bed8e5700 1 -- 172.21.15.42:0/1658731712 >> 172.21.15.42:0/1658731712 conn(0x7f6be80aabe0 msgr2=0x7f6be80a98e0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:44:50.781 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b has gained rank=0, replacing gid=7755 2019-12-05T05:44:50.782 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 1 secs before reviving mds.a 2019-12-05T05:44:51.782 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.a 2019-12-05T05:44:51.782 INFO:tasks.ceph.mds.a:Restarting daemon 2019-12-05T05:44:51.782 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2019-12-05T05:44:51.786 INFO:tasks.ceph.mds.a:Started 2019-12-05T05:44:51.786 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:44:51.858 INFO:tasks.ceph.mds.a.smithi042.stdout:starting mds.a at 2019-12-05T05:44:51.895 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:51.879+0000 7f2f681f0700 1 Processor -- start 2019-12-05T05:44:51.895 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:51.879+0000 7f2f681f0700 1 -- start start 2019-12-05T05:44:51.895 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:51.879+0000 7f2f681f0700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2f600a7730 0x7f2f6008df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:44:51.895 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:51.879+0000 7f2f681f0700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2f6007e710 0x7f2f6007eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:44:51.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:51.879+0000 7f2f681f0700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f600a0700 con 0x7f2f6009f2d0 2019-12-05T05:44:51.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:51.879+0000 7f2f681f0700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f600a0af0 con 0x7f2f6007e710 2019-12-05T05:44:51.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:51.879+0000 7f2f681f0700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f6008e5a0 con 0x7f2f600a7730 2019-12-05T05:44:51.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:51.879+0000 7f2f65f8c700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2f6007e710 0x7f2f6007eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:44:51.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:51.879+0000 7f2f6678d700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2f600a7730 0x7f2f6008df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:44:51.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:51.879+0000 7f2f65f8c700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2f6007e710 0x7f2f6007eb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.42:56362/0 (socket says 172.21.15.42:56362) 2019-12-05T05:44:51.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:51.879+0000 7f2f65f8c700 1 -- 172.21.15.42:0/1731923891 learned_addr learned my addr 172.21.15.42:0/1731923891 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:44:51.897 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:51.883+0000 7f2f6678d700 1 -- 172.21.15.42:0/1731923891 >> v2:172.21.15.40:3300/0 conn(0x7f2f6007e710 msgr2=0x7f2f6007eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:44:51.897 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:51.883+0000 7f2f6678d700 1 --2- 172.21.15.42:0/1731923891 >> v2:172.21.15.40:3300/0 conn(0x7f2f6007e710 0x7f2f6007eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:44:51.897 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:51.883+0000 7f2f6678d700 1 -- 172.21.15.42:0/1731923891 >> v1:172.21.15.40:6789/0 conn(0x7f2f6009f2d0 legacy=0x7f2f6009f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:51.897 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:51.883+0000 7f2f6678d700 1 -- 172.21.15.42:0/1731923891 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f6007f090 con 0x7f2f600a7730 2019-12-05T05:44:51.897 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:51.883+0000 7f2f6678d700 1 --2- 172.21.15.42:0/1731923891 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2f600a7730 0x7f2f6008df70 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f2f5c008790 tx=0x7f2f5c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:51.898 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:51.883+0000 7f2f6678d700 1 --2- 172.21.15.42:0/1731923891 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2f600a7730 0x7f2f6008df70 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f2f5c008790 tx=0x7f2f5c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:51.898 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:51.883+0000 7f2f6678d700 1 --2- 172.21.15.42:0/1731923891 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2f600a7730 0x7f2f6008df70 secure :-1 s=READY pgs=305 cs=0 l=1 rx=0x7f2f5c008790 tx=0x7f2f5c00fc70).ready entity=mon.2 client_cookie=4b860273e7ff6599 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:44:51.898 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:51.887+0000 7f2f6678d700 1 --2- 172.21.15.42:0/1731923891 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2f600a7730 0x7f2f6008df70 secure :-1 s=THROTTLE_DONE pgs=305 cs=0 l=1 rx=0x7f2f5c008790 tx=0x7f2f5c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:51.898 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:51.887+0000 7f2f6578b700 1 -- 172.21.15.42:0/1731923891 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2f5c0112f0 con 0x7f2f600a7730 2019-12-05T05:44:51.899 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:51.887+0000 7f2f6678d700 1 --2- 172.21.15.42:0/1731923891 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2f600a7730 0x7f2f6008df70 secure :-1 s=THROTTLE_DONE pgs=305 cs=0 l=1 rx=0x7f2f5c008790 tx=0x7f2f5c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:51.899 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:51.887+0000 7f2f6578b700 1 -- 172.21.15.42:0/1731923891 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f5c008ad0 con 0x7f2f600a7730 2019-12-05T05:44:51.899 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:51.887+0000 7f2f6678d700 1 --2- 172.21.15.42:0/1731923891 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2f600a7730 0x7f2f6008df70 secure :-1 s=THROTTLE_DONE pgs=305 cs=0 l=1 rx=0x7f2f5c008790 tx=0x7f2f5c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:51.899 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:51.887+0000 7f2f681f0700 1 -- 172.21.15.42:0/1731923891 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2f600a7730 msgr2=0x7f2f6008df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:51.899 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:51.887+0000 7f2f6578b700 1 -- 172.21.15.42:0/1731923891 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2f5c0118d0 con 0x7f2f600a7730 2019-12-05T05:44:51.899 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:51.887+0000 7f2f681f0700 1 --2- 172.21.15.42:0/1731923891 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2f600a7730 0x7f2f6008df70 secure :-1 s=READY pgs=305 cs=0 l=1 rx=0x7f2f5c008790 tx=0x7f2f5c00fc70).stop 2019-12-05T05:44:51.899 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:51.887+0000 7f2f681f0700 1 -- 172.21.15.42:0/1731923891 shutdown_connections 2019-12-05T05:44:51.900 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:51.887+0000 7f2f681f0700 1 --2- 172.21.15.42:0/1731923891 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2f600a7730 0x7f2f6008df70 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:44:51.900 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:51.887+0000 7f2f681f0700 1 --2- 172.21.15.42:0/1731923891 >> v2:172.21.15.40:3300/0 conn(0x7f2f6007e710 0x7f2f6007eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:44:51.900 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:51.887+0000 7f2f681f0700 1 -- 172.21.15.42:0/1731923891 shutdown_connections 2019-12-05T05:44:51.900 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:51.887+0000 7f2f681f0700 1 -- 172.21.15.42:0/1731923891 wait complete. 2019-12-05T05:44:51.900 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:51.887+0000 7f2f681f0700 1 -- 172.21.15.42:0/1731923891 >> 172.21.15.42:0/1731923891 conn(0x7f2f600aabd0 msgr2=0x7f2f600a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:44:51.903 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:51.891+0000 7f2f681f0700 1 Processor -- start 2019-12-05T05:44:51.903 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:51.891+0000 7f2f681f0700 1 -- start start 2019-12-05T05:44:51.904 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:51.891+0000 7f2f681f0700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2f6008f3b0 0x7f2f6007ba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:44:51.904 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:51.891+0000 7f2f681f0700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f600a5030 con 0x7f2f601339a0 2019-12-05T05:44:51.904 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:51.891+0000 7f2f681f0700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f600a41f0 con 0x7f2f6011d370 2019-12-05T05:44:51.904 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:51.891+0000 7f2f681f0700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f600a57b0 con 0x7f2f6008f3b0 2019-12-05T05:44:51.904 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:51.891+0000 7f2f65f8c700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f2f6011d370 0x7f2f600a4830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:40358/0 (socket says 172.21.15.42:40358) 2019-12-05T05:44:51.904 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:51.891+0000 7f2f65f8c700 1 -- 172.21.15.42:0/1612052437 learned_addr learned my addr 172.21.15.42:0/1612052437 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:44:51.904 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:51.891+0000 7f2f6678d700 1 --2- 172.21.15.42:0/1612052437 >> v2:172.21.15.40:3300/0 conn(0x7f2f6008f3b0 0x7f2f6007ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:44:51.905 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:51.891+0000 7f2f56ffd700 1 -- 172.21.15.42:0/1612052437 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2358492173 0 0) 0x7f2f600a41f0 con 0x7f2f6011d370 2019-12-05T05:44:51.905 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:51.891+0000 7f2f56ffd700 1 -- 172.21.15.42:0/1612052437 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f480030b0 con 0x7f2f6011d370 2019-12-05T05:44:51.905 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:51.891+0000 7f2f6678d700 1 -- 172.21.15.42:0/1612052437 >> v1:172.21.15.42:6789/0 conn(0x7f2f6011d370 legacy=0x7f2f600a4830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:51.905 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:51.891+0000 7f2f6678d700 1 -- 172.21.15.42:0/1612052437 >> v1:172.21.15.40:6789/0 conn(0x7f2f601339a0 legacy=0x7f2f60135de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:51.905 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:51.891+0000 7f2f6678d700 1 -- 172.21.15.42:0/1612052437 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f600a5f90 con 0x7f2f6008f3b0 2019-12-05T05:44:51.905 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:51.891+0000 7f2f6678d700 1 --2- 172.21.15.42:0/1612052437 >> v2:172.21.15.40:3300/0 conn(0x7f2f6008f3b0 0x7f2f6007ba50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f2f5c008790 tx=0x7f2f5c011de0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:51.906 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:51.891+0000 7f2f6678d700 1 --2- 172.21.15.42:0/1612052437 >> v2:172.21.15.40:3300/0 conn(0x7f2f6008f3b0 0x7f2f6007ba50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f2f5c008790 tx=0x7f2f5c011de0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:51.906 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:51.891+0000 7f2f6678d700 1 --2- 172.21.15.42:0/1612052437 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2f6008f3b0 0x7f2f6007ba50 secure :-1 s=READY pgs=1089 cs=0 l=1 rx=0x7f2f5c008790 tx=0x7f2f5c011de0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:44:51.906 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:51.891+0000 7f2f6678d700 1 --2- 172.21.15.42:0/1612052437 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2f6008f3b0 0x7f2f6007ba50 secure :-1 s=THROTTLE_DONE pgs=1089 cs=0 l=1 rx=0x7f2f5c008790 tx=0x7f2f5c011de0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:51.906 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:51.891+0000 7f2f56ffd700 1 -- 172.21.15.42:0/1612052437 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2f5c007150 con 0x7f2f6008f3b0 2019-12-05T05:44:51.906 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:51.891+0000 7f2f6678d700 1 --2- 172.21.15.42:0/1612052437 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2f6008f3b0 0x7f2f6007ba50 secure :-1 s=THROTTLE_DONE pgs=1089 cs=0 l=1 rx=0x7f2f5c008790 tx=0x7f2f5c011de0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:51.906 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:51.891+0000 7f2f56ffd700 1 -- 172.21.15.42:0/1612052437 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f5c0114f0 con 0x7f2f6008f3b0 2019-12-05T05:44:51.907 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:51.891+0000 7f2f6678d700 1 --2- 172.21.15.42:0/1612052437 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2f6008f3b0 0x7f2f6007ba50 secure :-1 s=THROTTLE_DONE pgs=1089 cs=0 l=1 rx=0x7f2f5c008790 tx=0x7f2f5c011de0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:51.907 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:51.891+0000 7f2f681f0700 1 -- 172.21.15.42:0/1612052437 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f600a19d0 con 0x7f2f6008f3b0 2019-12-05T05:44:51.907 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:51.891+0000 7f2f56ffd700 1 -- 172.21.15.42:0/1612052437 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2f5c00f550 con 0x7f2f6008f3b0 2019-12-05T05:44:51.907 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:51.891+0000 7f2f681f0700 1 -- 172.21.15.42:0/1612052437 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f6007bfa0 con 0x7f2f6008f3b0 2019-12-05T05:44:51.907 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:51.895+0000 7f2f6678d700 1 --2- 172.21.15.42:0/1612052437 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2f6008f3b0 0x7f2f6007ba50 secure :-1 s=THROTTLE_DONE pgs=1089 cs=0 l=1 rx=0x7f2f5c008790 tx=0x7f2f5c011de0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:51.907 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:51.895+0000 7f2f56ffd700 1 -- 172.21.15.42:0/1612052437 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f2f5c005540 con 0x7f2f6008f3b0 2019-12-05T05:44:51.908 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:51.895+0000 7f2f6678d700 1 --2- 172.21.15.42:0/1612052437 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2f6008f3b0 0x7f2f6007ba50 secure :-1 s=THROTTLE_DONE pgs=1089 cs=0 l=1 rx=0x7f2f5c008790 tx=0x7f2f5c011de0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:51.908 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:51.895+0000 7f2f56ffd700 1 --2- 172.21.15.42:0/1612052437 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2f4803b1d0 0x7f2f4803d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:44:51.908 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:51.895+0000 7f2f56ffd700 1 -- 172.21.15.42:0/1612052437 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 8439+0+0 (secure 0 0 0) 0x7f2f5c04b8f0 con 0x7f2f6008f3b0 2019-12-05T05:44:51.908 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:51.895+0000 7f2f65f8c700 1 --2- 172.21.15.42:0/1612052437 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2f4803b1d0 0x7f2f4803d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:44:51.922 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:51.895+0000 7f2f681f0700 1 -- 172.21.15.42:0/1612052437 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f6005d100 con 0x7f2f6008f3b0 2019-12-05T05:44:51.922 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:51.895+0000 7f2f65f8c700 1 --2- 172.21.15.42:0/1612052437 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2f4803b1d0 0x7f2f4803d6e0 crc :-1 s=READY pgs=606 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:44:51.922 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:51.907+0000 7f2f6678d700 1 --2- 172.21.15.42:0/1612052437 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2f6008f3b0 0x7f2f6007ba50 secure :-1 s=THROTTLE_DONE pgs=1089 cs=0 l=1 rx=0x7f2f5c008790 tx=0x7f2f5c011de0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:51.923 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:51.907+0000 7f2f56ffd700 1 -- 172.21.15.42:0/1612052437 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f2f5c0305d0 con 0x7f2f6008f3b0 2019-12-05T05:44:52.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:52.087+0000 7f2f681f0700 1 -- 172.21.15.42:0/1612052437 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2f6005b8b0 con 0x7f2f6008f3b0 2019-12-05T05:44:52.100 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:52.091+0000 7f2f6678d700 1 --2- 172.21.15.42:0/1612052437 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2f6008f3b0 0x7f2f6007ba50 secure :-1 s=THROTTLE_DONE pgs=1089 cs=0 l=1 rx=0x7f2f5c008790 tx=0x7f2f5c011de0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:44:52.100 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:52.091+0000 7f2f56ffd700 1 -- 172.21.15.42:0/1612052437 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 173 v173) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f2f5c033760 con 0x7f2f6008f3b0 2019-12-05T05:44:52.100 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 173 2019-12-05T05:44:52.100 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:52.091+0000 7f2f681f0700 1 -- 172.21.15.42:0/1612052437 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2f4803b1d0 msgr2=0x7f2f4803d6e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:52.100 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:52.091+0000 7f2f681f0700 1 --2- 172.21.15.42:0/1612052437 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2f4803b1d0 0x7f2f4803d6e0 crc :-1 s=READY pgs=606 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:44:52.100 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:52.091+0000 7f2f681f0700 1 -- 172.21.15.42:0/1612052437 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2f6008f3b0 msgr2=0x7f2f6007ba50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:44:52.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:52.091+0000 7f2f681f0700 1 --2- 172.21.15.42:0/1612052437 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2f6008f3b0 0x7f2f6007ba50 secure :-1 s=READY pgs=1089 cs=0 l=1 rx=0x7f2f5c008790 tx=0x7f2f5c011de0).stop 2019-12-05T05:44:52.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:52.091+0000 7f2f681f0700 1 -- 172.21.15.42:0/1612052437 shutdown_connections 2019-12-05T05:44:52.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:52.091+0000 7f2f681f0700 1 --2- 172.21.15.42:0/1612052437 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2f4803b1d0 0x7f2f4803d6e0 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:44:52.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:52.091+0000 7f2f681f0700 1 --2- 172.21.15.42:0/1612052437 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2f6008f3b0 0x7f2f6007ba50 unknown :-1 s=CLOSED pgs=1089 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:44:52.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:52.091+0000 7f2f681f0700 1 -- 172.21.15.42:0/1612052437 shutdown_connections 2019-12-05T05:44:52.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:52.091+0000 7f2f681f0700 1 -- 172.21.15.42:0/1612052437 wait complete. 2019-12-05T05:44:52.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:44:52.091+0000 7f2f681f0700 1 -- 172.21.15.42:0/1612052437 >> 172.21.15.42:0/1612052437 conn(0x7f2f6007df50 msgr2=0x7f2f6011d8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:44:52.102 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:44:52.102 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":173,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6077,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6835/3757738536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3757738536},{"type":"v1","addr":"172.21.15.42:6835","nonce":3757738536}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":173}],"filesystems":[{"mdsmap":{"epoch":172,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:44:49.006174+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7884},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7884":{"gid":7884,"name":"b","rank":0,"incarnation":169,"state":"up:active","state_seq":11,"addr":"172.21.15.40:6833/1127133319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1127133319},{"type":"v1","addr":"172.21.15.40:6833","nonce":1127133319}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:44:52.107 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a reported in up:standby state 2019-12-05T05:44:52.108 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 28 secs before thrashing 2019-12-05T05:45:11.316 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:45:11.320 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:45:20.109 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:45:20.211 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.199+0000 7f3f23e27700 1 Processor -- start 2019-12-05T05:45:20.211 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.199+0000 7f3f23e27700 1 -- start start 2019-12-05T05:45:20.211 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.199+0000 7f3f23e27700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3f1c0a7730 0x7f3f1c08df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:45:20.211 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.199+0000 7f3f23e27700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f3f1c07e710 0x7f3f1c07eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:45:20.212 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.199+0000 7f3f23e27700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f1c0a0700 con 0x7f3f1c09f2d0 2019-12-05T05:45:20.212 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.199+0000 7f3f23e27700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f1c0a0af0 con 0x7f3f1c07e710 2019-12-05T05:45:20.212 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.199+0000 7f3f23e27700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f1c08e5a0 con 0x7f3f1c0a7730 2019-12-05T05:45:20.212 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.203+0000 7f3f223c4700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3f1c0a7730 0x7f3f1c08df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:45:20.212 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.203+0000 7f3f21bc3700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f3f1c07e710 0x7f3f1c07eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:45:20.212 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.203+0000 7f3f223c4700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3f1c0a7730 0x7f3f1c08df70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:40430/0 (socket says 172.21.15.42:40430) 2019-12-05T05:45:20.212 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.203+0000 7f3f22bc5700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f3f1c09f2d0 0x7f3f1c09f700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:40472/0 (socket says 172.21.15.42:40472) 2019-12-05T05:45:20.213 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.203+0000 7f3f223c4700 1 -- 172.21.15.42:0/1550795223 learned_addr learned my addr 172.21.15.42:0/1550795223 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:45:20.213 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.203+0000 7f3f223c4700 1 -- 172.21.15.42:0/1550795223 >> v2:172.21.15.40:3300/0 conn(0x7f3f1c07e710 msgr2=0x7f3f1c07eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:45:20.213 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.203+0000 7f3f223c4700 1 --2- 172.21.15.42:0/1550795223 >> v2:172.21.15.40:3300/0 conn(0x7f3f1c07e710 0x7f3f1c07eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:45:20.213 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.203+0000 7f3f223c4700 1 -- 172.21.15.42:0/1550795223 >> v1:172.21.15.42:6789/0 conn(0x7f3f1c09f2d0 legacy=0x7f3f1c09f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:45:20.213 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.203+0000 7f3f223c4700 1 -- 172.21.15.42:0/1550795223 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f1c07f090 con 0x7f3f1c0a7730 2019-12-05T05:45:20.213 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.203+0000 7f3f21bc3700 1 --2- 172.21.15.42:0/1550795223 >> v2:172.21.15.40:3300/0 conn(0x7f3f1c07e710 0x7f3f1c07eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:45:20.213 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.203+0000 7f3f223c4700 1 --2- 172.21.15.42:0/1550795223 >> v2:172.21.15.42:3300/0 conn(0x7f3f1c0a7730 0x7f3f1c08df70 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f3f0c002f30 tx=0x7f3f0c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:20.214 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.203+0000 7f3f223c4700 1 --2- 172.21.15.42:0/1550795223 >> v2:172.21.15.42:3300/0 conn(0x7f3f1c0a7730 0x7f3f1c08df70 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f3f0c002f30 tx=0x7f3f0c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:20.214 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.203+0000 7f3f223c4700 1 --2- 172.21.15.42:0/1550795223 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3f1c0a7730 0x7f3f1c08df70 secure :-1 s=READY pgs=1414 cs=0 l=1 rx=0x7f3f0c002f30 tx=0x7f3f0c00fd30).ready entity=mon.1 client_cookie=d9c8f5efc9f45742 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:45:20.214 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.203+0000 7f3f223c4700 1 --2- 172.21.15.42:0/1550795223 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3f1c0a7730 0x7f3f1c08df70 secure :-1 s=THROTTLE_DONE pgs=1414 cs=0 l=1 rx=0x7f3f0c002f30 tx=0x7f3f0c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:20.214 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.203+0000 7f3f223c4700 1 --2- 172.21.15.42:0/1550795223 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3f1c0a7730 0x7f3f1c08df70 secure :-1 s=THROTTLE_DONE pgs=1414 cs=0 l=1 rx=0x7f3f0c002f30 tx=0x7f3f0c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:20.214 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.203+0000 7f3f213c2700 1 -- 172.21.15.42:0/1550795223 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3f0c011370 con 0x7f3f1c0a7730 2019-12-05T05:45:20.214 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.203+0000 7f3f223c4700 1 --2- 172.21.15.42:0/1550795223 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3f1c0a7730 0x7f3f1c08df70 secure :-1 s=THROTTLE_DONE pgs=1414 cs=0 l=1 rx=0x7f3f0c002f30 tx=0x7f3f0c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:20.214 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.203+0000 7f3f213c2700 1 -- 172.21.15.42:0/1550795223 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f0c008bb0 con 0x7f3f1c0a7730 2019-12-05T05:45:20.215 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.203+0000 7f3f213c2700 1 -- 172.21.15.42:0/1550795223 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3f0c011950 con 0x7f3f1c0a7730 2019-12-05T05:45:20.215 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.203+0000 7f3f23e27700 1 -- 172.21.15.42:0/1550795223 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3f1c0a7730 msgr2=0x7f3f1c08df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:45:20.215 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.203+0000 7f3f23e27700 1 --2- 172.21.15.42:0/1550795223 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3f1c0a7730 0x7f3f1c08df70 secure :-1 s=READY pgs=1414 cs=0 l=1 rx=0x7f3f0c002f30 tx=0x7f3f0c00fd30).stop 2019-12-05T05:45:20.215 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.203+0000 7f3f23e27700 1 -- 172.21.15.42:0/1550795223 shutdown_connections 2019-12-05T05:45:20.215 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.203+0000 7f3f23e27700 1 --2- 172.21.15.42:0/1550795223 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3f1c0a7730 0x7f3f1c08df70 unknown :-1 s=CLOSED pgs=1414 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:45:20.215 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.203+0000 7f3f23e27700 1 --2- 172.21.15.42:0/1550795223 >> v2:172.21.15.40:3300/0 conn(0x7f3f1c07e710 0x7f3f1c07eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:45:20.216 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.203+0000 7f3f23e27700 1 -- 172.21.15.42:0/1550795223 shutdown_connections 2019-12-05T05:45:20.216 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.203+0000 7f3f23e27700 1 -- 172.21.15.42:0/1550795223 wait complete. 2019-12-05T05:45:20.216 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.203+0000 7f3f23e27700 1 -- 172.21.15.42:0/1550795223 >> 172.21.15.42:0/1550795223 conn(0x7f3f1c0aabd0 msgr2=0x7f3f1c0a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:45:20.219 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.207+0000 7f3f23e27700 1 Processor -- start 2019-12-05T05:45:20.219 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.207+0000 7f3f23e27700 1 -- start start 2019-12-05T05:45:20.220 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.207+0000 7f3f23e27700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3f1c1249c0 0x7f3f1c07d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:45:20.225 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.207+0000 7f3f23e27700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3f1c11b3e0 0x7f3f1c11b810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:45:20.226 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.207+0000 7f3f23e27700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f3f1c08eb40 0x7f3f1c08ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:45:20.226 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.207+0000 7f3f23e27700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f1c07c6e0 con 0x7f3f1c08eb40 2019-12-05T05:45:20.228 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.207+0000 7f3f23e27700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f1c07c980 con 0x7f3f1c11b3e0 2019-12-05T05:45:20.228 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.207+0000 7f3f23e27700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f1c07cc20 con 0x7f3f1c1249c0 2019-12-05T05:45:20.228 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.207+0000 7f3f21bc3700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3f1c11b3e0 0x7f3f1c11b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:45:20.228 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.207+0000 7f3f21bc3700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3f1c11b3e0 0x7f3f1c11b810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:40438/0 (socket says 172.21.15.42:40438) 2019-12-05T05:45:20.228 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.207+0000 7f3f21bc3700 1 -- 172.21.15.42:0/1861046780 learned_addr learned my addr 172.21.15.42:0/1861046780 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:45:20.228 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.207+0000 7f3f223c4700 1 --2- 172.21.15.42:0/1861046780 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3f1c1249c0 0x7f3f1c07d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:45:20.229 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.207+0000 7f3f22bc5700 1 --2- 172.21.15.42:0/1861046780 >> v2:172.21.15.40:3300/0 conn(0x7f3f1c08eb40 0x7f3f1c08ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:45:20.229 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.207+0000 7f3f21bc3700 1 -- 172.21.15.42:0/1861046780 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3f1c1249c0 msgr2=0x7f3f1c07d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:45:20.229 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.207+0000 7f3f21bc3700 1 --2- 172.21.15.42:0/1861046780 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3f1c1249c0 0x7f3f1c07d4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:45:20.229 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.207+0000 7f3f21bc3700 1 -- 172.21.15.42:0/1861046780 >> v2:172.21.15.40:3300/0 conn(0x7f3f1c08eb40 msgr2=0x7f3f1c08ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:45:20.229 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.207+0000 7f3f21bc3700 1 --2- 172.21.15.42:0/1861046780 >> v2:172.21.15.40:3300/0 conn(0x7f3f1c08eb40 0x7f3f1c08ef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:45:20.229 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.207+0000 7f3f21bc3700 1 -- 172.21.15.42:0/1861046780 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f1c07c3d0 con 0x7f3f1c11b3e0 2019-12-05T05:45:20.229 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.207+0000 7f3f21bc3700 1 --2- 172.21.15.42:0/1861046780 >> v2:172.21.15.42:3300/0 conn(0x7f3f1c11b3e0 0x7f3f1c11b810 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f3f1c07c8a0 tx=0x7f3f10010e30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:20.230 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.207+0000 7f3f21bc3700 1 --2- 172.21.15.42:0/1861046780 >> v2:172.21.15.42:3300/0 conn(0x7f3f1c11b3e0 0x7f3f1c11b810 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f3f1c07c8a0 tx=0x7f3f10010e30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:20.230 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.207+0000 7f3f21bc3700 1 --2- 172.21.15.42:0/1861046780 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3f1c11b3e0 0x7f3f1c11b810 secure :-1 s=READY pgs=1415 cs=0 l=1 rx=0x7f3f1c07c8a0 tx=0x7f3f10010e30).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:45:20.230 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.207+0000 7f3f21bc3700 1 --2- 172.21.15.42:0/1861046780 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3f1c11b3e0 0x7f3f1c11b810 secure :-1 s=THROTTLE_DONE pgs=1415 cs=0 l=1 rx=0x7f3f1c07c8a0 tx=0x7f3f10010e30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:20.230 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.207+0000 7f3f21bc3700 1 --2- 172.21.15.42:0/1861046780 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3f1c11b3e0 0x7f3f1c11b810 secure :-1 s=THROTTLE_DONE pgs=1415 cs=0 l=1 rx=0x7f3f1c07c8a0 tx=0x7f3f10010e30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:20.230 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.207+0000 7f3f1affd700 1 -- 172.21.15.42:0/1861046780 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3f10005840 con 0x7f3f1c11b3e0 2019-12-05T05:45:20.230 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.207+0000 7f3f21bc3700 1 --2- 172.21.15.42:0/1861046780 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3f1c11b3e0 0x7f3f1c11b810 secure :-1 s=THROTTLE_DONE pgs=1415 cs=0 l=1 rx=0x7f3f1c07c8a0 tx=0x7f3f10010e30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:20.231 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.207+0000 7f3f1affd700 1 -- 172.21.15.42:0/1861046780 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f10007160 con 0x7f3f1c11b3e0 2019-12-05T05:45:20.231 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.207+0000 7f3f23e27700 1 -- 172.21.15.42:0/1861046780 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f1c11bd60 con 0x7f3f1c11b3e0 2019-12-05T05:45:20.231 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.207+0000 7f3f1affd700 1 -- 172.21.15.42:0/1861046780 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3f10012470 con 0x7f3f1c11b3e0 2019-12-05T05:45:20.231 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.207+0000 7f3f23e27700 1 -- 172.21.15.42:0/1861046780 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f1c08f660 con 0x7f3f1c11b3e0 2019-12-05T05:45:20.231 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.211+0000 7f3f21bc3700 1 --2- 172.21.15.42:0/1861046780 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3f1c11b3e0 0x7f3f1c11b810 secure :-1 s=THROTTLE_DONE pgs=1415 cs=0 l=1 rx=0x7f3f1c07c8a0 tx=0x7f3f10010e30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:20.231 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.211+0000 7f3f1affd700 1 -- 172.21.15.42:0/1861046780 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f3f10012780 con 0x7f3f1c11b3e0 2019-12-05T05:45:20.231 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.211+0000 7f3f21bc3700 1 --2- 172.21.15.42:0/1861046780 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3f1c11b3e0 0x7f3f1c11b810 secure :-1 s=THROTTLE_DONE pgs=1415 cs=0 l=1 rx=0x7f3f1c07c8a0 tx=0x7f3f10010e30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:20.232 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.211+0000 7f3f1affd700 1 --2- 172.21.15.42:0/1861046780 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3efc03add0 0x7f3efc03d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:45:20.232 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.211+0000 7f3f1affd700 1 -- 172.21.15.42:0/1861046780 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 8439+0+0 (secure 0 0 0) 0x7f3f1004b670 con 0x7f3f1c11b3e0 2019-12-05T05:45:20.232 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.211+0000 7f3f223c4700 1 --2- 172.21.15.42:0/1861046780 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3efc03add0 0x7f3efc03d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:45:20.232 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.211+0000 7f3f223c4700 1 --2- 172.21.15.42:0/1861046780 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3efc03add0 0x7f3efc03d2e0 crc :-1 s=READY pgs=608 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:45:20.237 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.215+0000 7f3f23e27700 1 -- 172.21.15.42:0/1861046780 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f1c07c1c0 con 0x7f3f1c11b3e0 2019-12-05T05:45:20.237 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.223+0000 7f3f21bc3700 1 --2- 172.21.15.42:0/1861046780 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3f1c11b3e0 0x7f3f1c11b810 secure :-1 s=THROTTLE_DONE pgs=1415 cs=0 l=1 rx=0x7f3f1c07c8a0 tx=0x7f3f10010e30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:20.237 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.223+0000 7f3f1affd700 1 -- 172.21.15.42:0/1861046780 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f3f100326f0 con 0x7f3f1c11b3e0 2019-12-05T05:45:20.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.399+0000 7f3f23e27700 1 -- 172.21.15.42:0/1861046780 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3f1c05d100 con 0x7f3f1c11b3e0 2019-12-05T05:45:20.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.399+0000 7f3f21bc3700 1 --2- 172.21.15.42:0/1861046780 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3f1c11b3e0 0x7f3f1c11b810 secure :-1 s=THROTTLE_DONE pgs=1415 cs=0 l=1 rx=0x7f3f1c07c8a0 tx=0x7f3f10010e30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:20.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.399+0000 7f3f1affd700 1 -- 172.21.15.42:0/1861046780 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 173 v173) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f3f10032950 con 0x7f3f1c11b3e0 2019-12-05T05:45:20.406 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 173 2019-12-05T05:45:20.407 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:45:20.407 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":173,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6077,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6835/3757738536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3757738536},{"type":"v1","addr":"172.21.15.42:6835","nonce":3757738536}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":173}],"filesystems":[{"mdsmap":{"epoch":172,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:44:49.006174+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7884},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7884":{"gid":7884,"name":"b","rank":0,"incarnation":169,"state":"up:active","state_seq":11,"addr":"172.21.15.40:6833/1127133319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1127133319},{"type":"v1","addr":"172.21.15.40:6833","nonce":1127133319}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:45:20.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.403+0000 7f3f23e27700 1 -- 172.21.15.42:0/1861046780 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3efc03add0 msgr2=0x7f3efc03d2e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:45:20.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.403+0000 7f3f23e27700 1 --2- 172.21.15.42:0/1861046780 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3efc03add0 0x7f3efc03d2e0 crc :-1 s=READY pgs=608 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:45:20.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.403+0000 7f3f23e27700 1 -- 172.21.15.42:0/1861046780 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3f1c11b3e0 msgr2=0x7f3f1c11b810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:45:20.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.403+0000 7f3f23e27700 1 --2- 172.21.15.42:0/1861046780 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3f1c11b3e0 0x7f3f1c11b810 secure :-1 s=READY pgs=1415 cs=0 l=1 rx=0x7f3f1c07c8a0 tx=0x7f3f10010e30).stop 2019-12-05T05:45:20.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.403+0000 7f3f23e27700 1 -- 172.21.15.42:0/1861046780 shutdown_connections 2019-12-05T05:45:20.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.403+0000 7f3f23e27700 1 --2- 172.21.15.42:0/1861046780 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3efc03add0 0x7f3efc03d2e0 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:45:20.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.403+0000 7f3f23e27700 1 --2- 172.21.15.42:0/1861046780 >> v2:172.21.15.40:3300/0 conn(0x7f3f1c08eb40 0x7f3f1c08ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:45:20.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.403+0000 7f3f23e27700 1 --2- 172.21.15.42:0/1861046780 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3f1c1249c0 0x7f3f1c07d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:45:20.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.403+0000 7f3f23e27700 1 --2- 172.21.15.42:0/1861046780 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3f1c11b3e0 0x7f3f1c11b810 unknown :-1 s=CLOSED pgs=1415 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:45:20.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.403+0000 7f3f23e27700 1 -- 172.21.15.42:0/1861046780 shutdown_connections 2019-12-05T05:45:20.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.403+0000 7f3f23e27700 1 -- 172.21.15.42:0/1861046780 wait complete. 2019-12-05T05:45:20.411 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.403+0000 7f3f23e27700 1 -- 172.21.15.42:0/1861046780 >> 172.21.15.42:0/1861046780 conn(0x7f3f1c0a9bc0 msgr2=0x7f3f1c09f570 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:45:20.421 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.b (rank=0) 2019-12-05T05:45:20.422 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2019-12-05T05:45:20.422 INFO:teuthology.orchestra.run:waiting for 300 2019-12-05T05:45:20.439 INFO:tasks.ceph.mds.b:Stopped 2019-12-05T05:45:20.439 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:45:20.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.527+0000 7f00424b8700 1 Processor -- start 2019-12-05T05:45:20.539 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.527+0000 7f00424b8700 1 -- start start 2019-12-05T05:45:20.539 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.527+0000 7f00424b8700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f003c0a7740 0x7f003c08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:45:20.539 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.527+0000 7f00424b8700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f003c07e720 0x7f003c07eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:45:20.539 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.527+0000 7f00424b8700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f003c0a0710 con 0x7f003c09f2e0 2019-12-05T05:45:20.539 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.527+0000 7f00424b8700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f003c0a0b00 con 0x7f003c0a7740 2019-12-05T05:45:20.539 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.527+0000 7f00424b8700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f003c08e5b0 con 0x7f003c07e720 2019-12-05T05:45:20.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.531+0000 7f003bfff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f003c07e720 0x7f003c07eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:45:20.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.531+0000 7f003bfff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f003c07e720 0x7f003c07eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:40448/0 (socket says 172.21.15.42:40448) 2019-12-05T05:45:20.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.531+0000 7f003bfff700 1 -- 172.21.15.42:0/3757331607 learned_addr learned my addr 172.21.15.42:0/3757331607 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:45:20.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.531+0000 7f0040a55700 1 --2- 172.21.15.42:0/3757331607 >> v2:172.21.15.40:3300/0 conn(0x7f003c0a7740 0x7f003c08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:45:20.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.531+0000 7f003bfff700 1 -- 172.21.15.42:0/3757331607 >> v2:172.21.15.40:3300/0 conn(0x7f003c0a7740 msgr2=0x7f003c08df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:45:20.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.531+0000 7f003bfff700 1 --2- 172.21.15.42:0/3757331607 >> v2:172.21.15.40:3300/0 conn(0x7f003c0a7740 0x7f003c08df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:45:20.541 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.531+0000 7f003bfff700 1 -- 172.21.15.42:0/3757331607 >> v1:172.21.15.40:6789/0 conn(0x7f003c09f2e0 legacy=0x7f003c09f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:45:20.541 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.531+0000 7f003bfff700 1 -- 172.21.15.42:0/3757331607 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f003c07f0a0 con 0x7f003c07e720 2019-12-05T05:45:20.541 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.531+0000 7f003bfff700 1 --2- 172.21.15.42:0/3757331607 >> v2:172.21.15.42:3300/0 conn(0x7f003c07e720 0x7f003c07eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f0030002f30 tx=0x7f003000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:20.541 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.531+0000 7f003bfff700 1 --2- 172.21.15.42:0/3757331607 >> v2:172.21.15.42:3300/0 conn(0x7f003c07e720 0x7f003c07eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f0030002f30 tx=0x7f003000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:20.541 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.531+0000 7f003bfff700 1 --2- 172.21.15.42:0/3757331607 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f003c07e720 0x7f003c07eb50 secure :-1 s=READY pgs=1416 cs=0 l=1 rx=0x7f0030002f30 tx=0x7f003000fd30).ready entity=mon.1 client_cookie=52375ddf990bdc09 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:45:20.541 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.531+0000 7f003bfff700 1 --2- 172.21.15.42:0/3757331607 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f003c07e720 0x7f003c07eb50 secure :-1 s=THROTTLE_DONE pgs=1416 cs=0 l=1 rx=0x7f0030002f30 tx=0x7f003000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:20.541 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.531+0000 7f003bfff700 1 --2- 172.21.15.42:0/3757331607 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f003c07e720 0x7f003c07eb50 secure :-1 s=THROTTLE_DONE pgs=1416 cs=0 l=1 rx=0x7f0030002f30 tx=0x7f003000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:20.542 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.531+0000 7f003bfff700 1 --2- 172.21.15.42:0/3757331607 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f003c07e720 0x7f003c07eb50 secure :-1 s=THROTTLE_DONE pgs=1416 cs=0 l=1 rx=0x7f0030002f30 tx=0x7f003000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:20.542 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.531+0000 7f003b7fe700 1 -- 172.21.15.42:0/3757331607 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0030011370 con 0x7f003c07e720 2019-12-05T05:45:20.542 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.531+0000 7f003b7fe700 1 -- 172.21.15.42:0/3757331607 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0030008bb0 con 0x7f003c07e720 2019-12-05T05:45:20.542 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.531+0000 7f003b7fe700 1 -- 172.21.15.42:0/3757331607 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0030011950 con 0x7f003c07e720 2019-12-05T05:45:20.542 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.531+0000 7f00424b8700 1 -- 172.21.15.42:0/3757331607 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f003c07e720 msgr2=0x7f003c07eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:45:20.542 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.531+0000 7f00424b8700 1 --2- 172.21.15.42:0/3757331607 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f003c07e720 0x7f003c07eb50 secure :-1 s=READY pgs=1416 cs=0 l=1 rx=0x7f0030002f30 tx=0x7f003000fd30).stop 2019-12-05T05:45:20.542 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.531+0000 7f00424b8700 1 -- 172.21.15.42:0/3757331607 shutdown_connections 2019-12-05T05:45:20.543 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.531+0000 7f00424b8700 1 --2- 172.21.15.42:0/3757331607 >> v2:172.21.15.40:3300/0 conn(0x7f003c0a7740 0x7f003c08df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:45:20.543 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.531+0000 7f00424b8700 1 --2- 172.21.15.42:0/3757331607 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f003c07e720 0x7f003c07eb50 unknown :-1 s=CLOSED pgs=1416 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:45:20.543 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.531+0000 7f00424b8700 1 -- 172.21.15.42:0/3757331607 shutdown_connections 2019-12-05T05:45:20.543 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.531+0000 7f00424b8700 1 -- 172.21.15.42:0/3757331607 wait complete. 2019-12-05T05:45:20.543 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.531+0000 7f00424b8700 1 -- 172.21.15.42:0/3757331607 >> 172.21.15.42:0/3757331607 conn(0x7f003c0aabe0 msgr2=0x7f003c0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:45:20.548 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.535+0000 7f00424b8700 1 Processor -- start 2019-12-05T05:45:20.548 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.535+0000 7f00424b8700 1 -- start start 2019-12-05T05:45:20.548 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.535+0000 7f00424b8700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f003c14e0c0 0x7f003c14e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:45:20.548 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.535+0000 7f00424b8700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f003c14ea40 con 0x7f003c1156c0 2019-12-05T05:45:20.549 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.535+0000 7f00424b8700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f003c0a7100 con 0x7f003c113a10 2019-12-05T05:45:20.549 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.535+0000 7f00424b8700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f003c113230 con 0x7f003c14e0c0 2019-12-05T05:45:20.549 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.535+0000 7f003bfff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f003c113a10 0x7f003c079390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:40492/0 (socket says 172.21.15.42:40492) 2019-12-05T05:45:20.549 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.535+0000 7f003bfff700 1 -- 172.21.15.42:0/2649996971 learned_addr learned my addr 172.21.15.42:0/2649996971 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:45:20.549 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.535+0000 7f0041256700 1 --2- 172.21.15.42:0/2649996971 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f003c14e0c0 0x7f003c14e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:45:20.549 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.535+0000 7f00397fa700 1 -- 172.21.15.42:0/2649996971 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 594498266 0 0) 0x7f003c0a7100 con 0x7f003c113a10 2019-12-05T05:45:20.549 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.535+0000 7f00397fa700 1 -- 172.21.15.42:0/2649996971 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f001c0030b0 con 0x7f003c113a10 2019-12-05T05:45:20.550 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.535+0000 7f00397fa700 1 -- 172.21.15.42:0/2649996971 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3008105222 0 0) 0x7f003c14ea40 con 0x7f003c1156c0 2019-12-05T05:45:20.550 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.535+0000 7f00397fa700 1 -- 172.21.15.42:0/2649996971 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f003c0a7100 con 0x7f003c1156c0 2019-12-05T05:45:20.550 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.535+0000 7f00397fa700 1 -- 172.21.15.42:0/2649996971 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 1565742525 0 0) 0x7f001c0030b0 con 0x7f003c113a10 2019-12-05T05:45:20.550 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.535+0000 7f00397fa700 1 -- 172.21.15.42:0/2649996971 --> v1:172.21.15.42:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f003c14ea40 con 0x7f003c113a10 2019-12-05T05:45:20.550 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.535+0000 7f00397fa700 1 -- 172.21.15.42:0/2649996971 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7f0030011370 con 0x7f003c113a10 2019-12-05T05:45:20.550 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.535+0000 7f00397fa700 1 -- 172.21.15.42:0/2649996971 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f003c14e0c0 msgr2=0x7f003c14e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:45:20.550 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.535+0000 7f00397fa700 1 --2- 172.21.15.42:0/2649996971 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f003c14e0c0 0x7f003c14e4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:45:20.551 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.539+0000 7f00397fa700 1 -- 172.21.15.42:0/2649996971 >> v1:172.21.15.42:6789/0 conn(0x7f003c113a10 legacy=0x7f003c079390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:45:20.551 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.539+0000 7f00397fa700 1 -- 172.21.15.42:0/2649996971 >> v1:172.21.15.40:6789/0 conn(0x7f003c1156c0 legacy=0x7f003c126d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:45:20.551 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.539+0000 7f00397fa700 1 --2- 172.21.15.42:0/2649996971 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f001c003ab0 0x7f001c004420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:45:20.551 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.539+0000 7f00397fa700 1 --2- 172.21.15.42:0/2649996971 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f001c003ee0 0x7f001c004970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:45:20.551 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.539+0000 7f00397fa700 1 --2- 172.21.15.42:0/2649996971 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f001c004f10 0x7f001c00b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:45:20.551 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.539+0000 7f00397fa700 1 -- 172.21.15.42:0/2649996971 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f003c113230 con 0x7f001c003ee0 2019-12-05T05:45:20.551 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.539+0000 7f00397fa700 1 -- 172.21.15.42:0/2649996971 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f001c00bcc0 con 0x7f001c004f10 2019-12-05T05:45:20.552 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.539+0000 7f00397fa700 1 -- 172.21.15.42:0/2649996971 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f001c00bf00 con 0x7f001c003ab0 2019-12-05T05:45:20.552 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.539+0000 7f0041256700 1 --2- 172.21.15.42:0/2649996971 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f001c004f10 0x7f001c00b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:45:20.552 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.539+0000 7f003bfff700 1 --2- 172.21.15.42:0/2649996971 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f001c003ee0 0x7f001c004970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:45:20.552 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.539+0000 7f0040a55700 1 --2- 172.21.15.42:0/2649996971 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f001c003ab0 0x7f001c004420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:45:20.552 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.539+0000 7f0041256700 1 -- 172.21.15.42:0/2649996971 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f001c003ab0 msgr2=0x7f001c004420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:45:20.552 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.539+0000 7f0041256700 1 --2- 172.21.15.42:0/2649996971 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f001c003ab0 0x7f001c004420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:45:20.552 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.539+0000 7f0041256700 1 -- 172.21.15.42:0/2649996971 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f001c003ee0 msgr2=0x7f001c004970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:45:20.553 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.539+0000 7f0041256700 1 --2- 172.21.15.42:0/2649996971 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f001c003ee0 0x7f001c004970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:45:20.553 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.539+0000 7f0041256700 1 -- 172.21.15.42:0/2649996971 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f003c08eb50 con 0x7f001c004f10 2019-12-05T05:45:20.553 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.539+0000 7f00424b8700 1 -- 172.21.15.42:0/2649996971 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f003c14e0c0 con 0x7f001c004f10 2019-12-05T05:45:20.553 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.539+0000 7f00424b8700 1 -- 172.21.15.42:0/2649996971 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f003c14e5c0 con 0x7f001c004f10 2019-12-05T05:45:20.553 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.539+0000 7f0041256700 1 --2- 172.21.15.42:0/2649996971 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f001c004f10 0x7f001c00b670 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f0034001210 tx=0x7f0034010ec0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:20.553 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.539+0000 7f0041256700 1 --2- 172.21.15.42:0/2649996971 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f001c004f10 0x7f001c00b670 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f0034001210 tx=0x7f0034010ec0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:20.553 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.539+0000 7f0041256700 1 --2- 172.21.15.42:0/2649996971 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f001c004f10 0x7f001c00b670 secure :-1 s=READY pgs=1418 cs=0 l=1 rx=0x7f0034001210 tx=0x7f0034010ec0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:45:20.554 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.539+0000 7f0041256700 1 --2- 172.21.15.42:0/2649996971 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f001c004f10 0x7f001c00b670 secure :-1 s=THROTTLE_DONE pgs=1418 cs=0 l=1 rx=0x7f0034001210 tx=0x7f0034010ec0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:20.554 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.539+0000 7f00397fa700 1 -- 172.21.15.42:0/2649996971 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0034011bf0 con 0x7f001c004f10 2019-12-05T05:45:20.554 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.539+0000 7f0041256700 1 --2- 172.21.15.42:0/2649996971 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f001c004f10 0x7f001c00b670 secure :-1 s=THROTTLE_DONE pgs=1418 cs=0 l=1 rx=0x7f0034001210 tx=0x7f0034010ec0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:20.554 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.539+0000 7f0041256700 1 --2- 172.21.15.42:0/2649996971 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f001c004f10 0x7f001c00b670 secure :-1 s=THROTTLE_DONE pgs=1418 cs=0 l=1 rx=0x7f0034001210 tx=0x7f0034010ec0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:20.554 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.539+0000 7f00397fa700 1 -- 172.21.15.42:0/2649996971 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0034011810 con 0x7f001c004f10 2019-12-05T05:45:20.554 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.539+0000 7f00397fa700 1 -- 172.21.15.42:0/2649996971 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f00340066e0 con 0x7f001c004f10 2019-12-05T05:45:20.554 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.539+0000 7f0041256700 1 --2- 172.21.15.42:0/2649996971 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f001c004f10 0x7f001c00b670 secure :-1 s=THROTTLE_DONE pgs=1418 cs=0 l=1 rx=0x7f0034001210 tx=0x7f0034010ec0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:20.554 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.539+0000 7f00397fa700 1 -- 172.21.15.42:0/2649996971 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f00340068e0 con 0x7f001c004f10 2019-12-05T05:45:20.555 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.539+0000 7f0041256700 1 --2- 172.21.15.42:0/2649996971 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f001c004f10 0x7f001c00b670 secure :-1 s=THROTTLE_DONE pgs=1418 cs=0 l=1 rx=0x7f0034001210 tx=0x7f0034010ec0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:20.555 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.539+0000 7f00397fa700 1 --2- 172.21.15.42:0/2649996971 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f001c043e00 0x7f001c046310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:45:20.555 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.539+0000 7f00397fa700 1 -- 172.21.15.42:0/2649996971 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 8439+0+0 (secure 0 0 0) 0x7f003404ae40 con 0x7f001c004f10 2019-12-05T05:45:20.555 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.539+0000 7f0040a55700 1 --2- 172.21.15.42:0/2649996971 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f001c043e00 0x7f001c046310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:45:20.555 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.543+0000 7f0040a55700 1 --2- 172.21.15.42:0/2649996971 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f001c043e00 0x7f001c046310 crc :-1 s=READY pgs=609 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:45:20.556 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.543+0000 7f00424b8700 1 -- 172.21.15.42:0/2649996971 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0014003200 con 0x7f001c004f10 2019-12-05T05:45:20.557 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.551+0000 7f0041256700 1 --2- 172.21.15.42:0/2649996971 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f001c004f10 0x7f001c00b670 secure :-1 s=THROTTLE_DONE pgs=1418 cs=0 l=1 rx=0x7f0034001210 tx=0x7f0034010ec0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:20.557 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.551+0000 7f00397fa700 1 -- 172.21.15.42:0/2649996971 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f0034031ee0 con 0x7f001c004f10 2019-12-05T05:45:20.726 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.719+0000 7f00424b8700 1 -- 172.21.15.42:0/2649996971 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0014003630 con 0x7f001c004f10 2019-12-05T05:45:20.726 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.719+0000 7f0041256700 1 --2- 172.21.15.42:0/2649996971 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f001c004f10 0x7f001c00b670 secure :-1 s=THROTTLE_DONE pgs=1418 cs=0 l=1 rx=0x7f0034001210 tx=0x7f0034010ec0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:20.726 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.719+0000 7f00397fa700 1 -- 172.21.15.42:0/2649996971 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 173 v173) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f0034032140 con 0x7f001c004f10 2019-12-05T05:45:20.727 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:45:20.727 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":173,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6077,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6835/3757738536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3757738536},{"type":"v1","addr":"172.21.15.42:6835","nonce":3757738536}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":173}],"filesystems":[{"mdsmap":{"epoch":172,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:44:49.006174+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7884},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7884":{"gid":7884,"name":"b","rank":0,"incarnation":169,"state":"up:active","state_seq":11,"addr":"172.21.15.40:6833/1127133319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1127133319},{"type":"v1","addr":"172.21.15.40:6833","nonce":1127133319}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:45:20.727 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 173 2019-12-05T05:45:20.729 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.723+0000 7f00424b8700 1 -- 172.21.15.42:0/2649996971 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f001c043e00 msgr2=0x7f001c046310 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:45:20.729 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.723+0000 7f00424b8700 1 --2- 172.21.15.42:0/2649996971 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f001c043e00 0x7f001c046310 crc :-1 s=READY pgs=609 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:45:20.730 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.723+0000 7f00424b8700 1 -- 172.21.15.42:0/2649996971 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f001c004f10 msgr2=0x7f001c00b670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:45:20.730 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.723+0000 7f0041256700 1 -- 172.21.15.42:0/2649996971 reap_dead start 2019-12-05T05:45:20.730 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.723+0000 7f00424b8700 1 --2- 172.21.15.42:0/2649996971 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f001c004f10 0x7f001c00b670 secure :-1 s=READY pgs=1418 cs=0 l=1 rx=0x7f0034001210 tx=0x7f0034010ec0).stop 2019-12-05T05:45:20.730 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.723+0000 7f00424b8700 1 -- 172.21.15.42:0/2649996971 shutdown_connections 2019-12-05T05:45:20.730 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.723+0000 7f00424b8700 1 --2- 172.21.15.42:0/2649996971 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f001c004f10 0x7f001c00b670 unknown :-1 s=CLOSED pgs=1418 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:45:20.730 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.723+0000 7f00424b8700 1 -- 172.21.15.42:0/2649996971 shutdown_connections 2019-12-05T05:45:20.731 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.723+0000 7f00424b8700 1 -- 172.21.15.42:0/2649996971 wait complete. 2019-12-05T05:45:20.731 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:20.723+0000 7f00424b8700 1 -- 172.21.15.42:0/2649996971 >> 172.21.15.42:0/2649996971 conn(0x7f003c08fdc0 msgr2=0x7f003c0a97e0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:45:20.738 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T05:45:22.739 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:45:22.844 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:22.831+0000 7f6938fa9700 1 Processor -- start 2019-12-05T05:45:22.844 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:22.831+0000 7f6938fa9700 1 -- start start 2019-12-05T05:45:22.844 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:22.835+0000 7f6938fa9700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f69340a7740 0x7f693408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:45:22.844 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:22.835+0000 7f6938fa9700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f693407e720 0x7f693407eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:45:22.844 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:22.835+0000 7f6938fa9700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69340a0710 con 0x7f693409f2e0 2019-12-05T05:45:22.845 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:22.835+0000 7f6938fa9700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69340a0b00 con 0x7f69340a7740 2019-12-05T05:45:22.845 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:22.835+0000 7f6938fa9700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f693408e5b0 con 0x7f693407e720 2019-12-05T05:45:22.845 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:22.835+0000 7f693359e700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f693409f2e0 0x7f693409f710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:40514/0 (socket says 172.21.15.42:40514) 2019-12-05T05:45:22.845 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:22.835+0000 7f693359e700 1 -- 172.21.15.42:0/1871334763 learned_addr learned my addr 172.21.15.42:0/1871334763 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:45:22.845 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:22.835+0000 7f692a59c700 1 --2- 172.21.15.42:0/1871334763 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f693407e720 0x7f693407eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:45:22.845 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:22.835+0000 7f6932d9d700 1 --2- 172.21.15.42:0/1871334763 >> v2:172.21.15.40:3300/0 conn(0x7f69340a7740 0x7f693408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:45:22.846 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:22.835+0000 7f693259c700 1 -- 172.21.15.42:0/1871334763 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1209821419 0 0) 0x7f69340a0710 con 0x7f693409f2e0 2019-12-05T05:45:22.846 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:22.835+0000 7f693259c700 1 -- 172.21.15.42:0/1871334763 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69200030b0 con 0x7f693409f2e0 2019-12-05T05:45:22.846 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:22.835+0000 7f692a59c700 1 -- 172.21.15.42:0/1871334763 >> v2:172.21.15.40:3300/0 conn(0x7f69340a7740 msgr2=0x7f693408df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:45:22.846 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:22.835+0000 7f692a59c700 1 --2- 172.21.15.42:0/1871334763 >> v2:172.21.15.40:3300/0 conn(0x7f69340a7740 0x7f693408df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:45:22.846 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:22.835+0000 7f692a59c700 1 -- 172.21.15.42:0/1871334763 >> v1:172.21.15.42:6789/0 conn(0x7f693409f2e0 legacy=0x7f693409f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:45:22.846 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:22.835+0000 7f692a59c700 1 -- 172.21.15.42:0/1871334763 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f693407f0a0 con 0x7f693407e720 2019-12-05T05:45:22.846 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:22.835+0000 7f692a59c700 1 --2- 172.21.15.42:0/1871334763 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f693407e720 0x7f693407eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f691c008790 tx=0x7f691c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:22.847 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:22.835+0000 7f692a59c700 1 --2- 172.21.15.42:0/1871334763 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f693407e720 0x7f693407eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f691c008790 tx=0x7f691c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:22.847 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:22.835+0000 7f692a59c700 1 --2- 172.21.15.42:0/1871334763 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f693407e720 0x7f693407eb50 secure :-1 s=READY pgs=306 cs=0 l=1 rx=0x7f691c008790 tx=0x7f691c00fc70).ready entity=mon.2 client_cookie=5fdc393fe6f5b56d server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:45:22.847 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:22.835+0000 7f692a59c700 1 --2- 172.21.15.42:0/1871334763 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f693407e720 0x7f693407eb50 secure :-1 s=THROTTLE_DONE pgs=306 cs=0 l=1 rx=0x7f691c008790 tx=0x7f691c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:22.847 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:22.835+0000 7f693259c700 1 -- 172.21.15.42:0/1871334763 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f691c0112f0 con 0x7f693407e720 2019-12-05T05:45:22.847 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:22.835+0000 7f692a59c700 1 --2- 172.21.15.42:0/1871334763 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f693407e720 0x7f693407eb50 secure :-1 s=THROTTLE_DONE pgs=306 cs=0 l=1 rx=0x7f691c008790 tx=0x7f691c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:22.847 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:22.835+0000 7f693259c700 1 -- 172.21.15.42:0/1871334763 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f691c008ad0 con 0x7f693407e720 2019-12-05T05:45:22.848 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:22.835+0000 7f692a59c700 1 --2- 172.21.15.42:0/1871334763 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f693407e720 0x7f693407eb50 secure :-1 s=THROTTLE_DONE pgs=306 cs=0 l=1 rx=0x7f691c008790 tx=0x7f691c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:22.848 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:22.835+0000 7f693259c700 1 -- 172.21.15.42:0/1871334763 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f691c0118d0 con 0x7f693407e720 2019-12-05T05:45:22.848 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:22.835+0000 7f6938fa9700 1 -- 172.21.15.42:0/1871334763 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f693407e720 msgr2=0x7f693407eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:45:22.849 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:22.835+0000 7f6938fa9700 1 --2- 172.21.15.42:0/1871334763 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f693407e720 0x7f693407eb50 secure :-1 s=READY pgs=306 cs=0 l=1 rx=0x7f691c008790 tx=0x7f691c00fc70).stop 2019-12-05T05:45:22.849 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:22.835+0000 7f6938fa9700 1 -- 172.21.15.42:0/1871334763 shutdown_connections 2019-12-05T05:45:22.849 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:22.835+0000 7f6938fa9700 1 --2- 172.21.15.42:0/1871334763 >> v2:172.21.15.40:3300/0 conn(0x7f69340a7740 0x7f693408df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:45:22.849 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:22.835+0000 7f6938fa9700 1 --2- 172.21.15.42:0/1871334763 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f693407e720 0x7f693407eb50 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:45:22.849 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:22.835+0000 7f6938fa9700 1 -- 172.21.15.42:0/1871334763 shutdown_connections 2019-12-05T05:45:22.849 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:22.835+0000 7f6938fa9700 1 -- 172.21.15.42:0/1871334763 wait complete. 2019-12-05T05:45:22.849 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:22.835+0000 7f6938fa9700 1 -- 172.21.15.42:0/1871334763 >> 172.21.15.42:0/1871334763 conn(0x7f69340aabe0 msgr2=0x7f69340a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:45:22.853 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:22.839+0000 7f6938fa9700 1 Processor -- start 2019-12-05T05:45:22.853 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:22.839+0000 7f6938fa9700 1 -- start start 2019-12-05T05:45:22.853 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:22.839+0000 7f6938fa9700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6934144810 0x7f6934120ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:45:22.853 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:22.839+0000 7f6938fa9700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6934118120 0x7f6934118550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:45:22.853 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:22.839+0000 7f6938fa9700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f693408eb70 0x7f693408efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:45:22.853 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:22.839+0000 7f6938fa9700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6934118c90 con 0x7f693408eb70 2019-12-05T05:45:22.854 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:22.839+0000 7f6938fa9700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6934117b80 con 0x7f6934144810 2019-12-05T05:45:22.854 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:22.839+0000 7f6938fa9700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6934131130 con 0x7f6934118120 2019-12-05T05:45:22.854 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:22.839+0000 7f6932d9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6934144810 0x7f6934120ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:45:22.854 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:22.843+0000 7f693359e700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f693408eb70 0x7f693408efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:45:22.854 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:22.843+0000 7f6932d9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6934144810 0x7f6934120ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:40478/0 (socket says 172.21.15.42:40478) 2019-12-05T05:45:22.854 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:22.843+0000 7f6932d9d700 1 -- 172.21.15.42:0/2094500766 learned_addr learned my addr 172.21.15.42:0/2094500766 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:45:22.854 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:22.843+0000 7f692a59c700 1 --2- 172.21.15.42:0/2094500766 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6934118120 0x7f6934118550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:45:22.855 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:22.843+0000 7f6932d9d700 1 -- 172.21.15.42:0/2094500766 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6934118120 msgr2=0x7f6934118550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:45:22.855 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:22.843+0000 7f6932d9d700 1 --2- 172.21.15.42:0/2094500766 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6934118120 0x7f6934118550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:45:22.855 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:22.843+0000 7f6932d9d700 1 -- 172.21.15.42:0/2094500766 >> v2:172.21.15.40:3300/0 conn(0x7f693408eb70 msgr2=0x7f693408efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:45:22.855 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:22.843+0000 7f6932d9d700 1 --2- 172.21.15.42:0/2094500766 >> v2:172.21.15.40:3300/0 conn(0x7f693408eb70 0x7f693408efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:45:22.855 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:22.843+0000 7f6932d9d700 1 -- 172.21.15.42:0/2094500766 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6934121540 con 0x7f6934144810 2019-12-05T05:45:22.855 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:22.843+0000 7f6932d9d700 1 --2- 172.21.15.42:0/2094500766 >> v2:172.21.15.42:3300/0 conn(0x7f6934144810 0x7f6934120ff0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f6924003070 tx=0x7f6924010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:22.855 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:22.843+0000 7f6932d9d700 1 --2- 172.21.15.42:0/2094500766 >> v2:172.21.15.42:3300/0 conn(0x7f6934144810 0x7f6934120ff0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f6924003070 tx=0x7f6924010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:22.855 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:22.843+0000 7f6932d9d700 1 --2- 172.21.15.42:0/2094500766 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6934144810 0x7f6934120ff0 secure :-1 s=READY pgs=1420 cs=0 l=1 rx=0x7f6924003070 tx=0x7f6924010b60).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:45:22.856 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:22.843+0000 7f6932d9d700 1 --2- 172.21.15.42:0/2094500766 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6934144810 0x7f6934120ff0 secure :-1 s=THROTTLE_DONE pgs=1420 cs=0 l=1 rx=0x7f6924003070 tx=0x7f6924010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:22.856 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:22.843+0000 7f6932d9d700 1 --2- 172.21.15.42:0/2094500766 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6934144810 0x7f6934120ff0 secure :-1 s=THROTTLE_DONE pgs=1420 cs=0 l=1 rx=0x7f6924003070 tx=0x7f6924010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:22.856 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:22.843+0000 7f692bfff700 1 -- 172.21.15.42:0/2094500766 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6924005920 con 0x7f6934144810 2019-12-05T05:45:22.856 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:22.843+0000 7f6932d9d700 1 --2- 172.21.15.42:0/2094500766 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6934144810 0x7f6934120ff0 secure :-1 s=THROTTLE_DONE pgs=1420 cs=0 l=1 rx=0x7f6924003070 tx=0x7f6924010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:22.856 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:22.843+0000 7f692bfff700 1 -- 172.21.15.42:0/2094500766 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6924005540 con 0x7f6934144810 2019-12-05T05:45:22.856 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:22.843+0000 7f6938fa9700 1 -- 172.21.15.42:0/2094500766 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6934119cb0 con 0x7f6934144810 2019-12-05T05:45:22.856 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:22.843+0000 7f692bfff700 1 -- 172.21.15.42:0/2094500766 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f692400e440 con 0x7f6934144810 2019-12-05T05:45:22.857 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:22.843+0000 7f6938fa9700 1 -- 172.21.15.42:0/2094500766 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6934116ed0 con 0x7f6934144810 2019-12-05T05:45:22.857 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:22.843+0000 7f6932d9d700 1 --2- 172.21.15.42:0/2094500766 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6934144810 0x7f6934120ff0 secure :-1 s=THROTTLE_DONE pgs=1420 cs=0 l=1 rx=0x7f6924003070 tx=0x7f6924010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:22.857 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:22.843+0000 7f692bfff700 1 -- 172.21.15.42:0/2094500766 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f692400e750 con 0x7f6934144810 2019-12-05T05:45:22.857 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:22.843+0000 7f6932d9d700 1 --2- 172.21.15.42:0/2094500766 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6934144810 0x7f6934120ff0 secure :-1 s=THROTTLE_DONE pgs=1420 cs=0 l=1 rx=0x7f6924003070 tx=0x7f6924010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:22.857 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:22.843+0000 7f692bfff700 1 --2- 172.21.15.42:0/2094500766 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f691803ad40 0x7f691803d250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:45:22.857 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:22.843+0000 7f692bfff700 1 -- 172.21.15.42:0/2094500766 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 8439+0+0 (secure 0 0 0) 0x7f692404ad90 con 0x7f6934144810 2019-12-05T05:45:22.857 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:22.847+0000 7f692a59c700 1 --2- 172.21.15.42:0/2094500766 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f691803ad40 0x7f691803d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:45:22.857 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:22.847+0000 7f692a59c700 1 --2- 172.21.15.42:0/2094500766 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f691803ad40 0x7f691803d250 crc :-1 s=READY pgs=610 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:45:22.858 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:22.847+0000 7f6938fa9700 1 -- 172.21.15.42:0/2094500766 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f693405d110 con 0x7f6934144810 2019-12-05T05:45:22.863 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:22.855+0000 7f6932d9d700 1 --2- 172.21.15.42:0/2094500766 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6934144810 0x7f6934120ff0 secure :-1 s=THROTTLE_DONE pgs=1420 cs=0 l=1 rx=0x7f6924003070 tx=0x7f6924010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:22.863 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:22.855+0000 7f692bfff700 1 -- 172.21.15.42:0/2094500766 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f6924031e10 con 0x7f6934144810 2019-12-05T05:45:23.032 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:23.027+0000 7f6938fa9700 1 -- 172.21.15.42:0/2094500766 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f693405b8c0 con 0x7f6934144810 2019-12-05T05:45:23.033 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:23.027+0000 7f6932d9d700 1 --2- 172.21.15.42:0/2094500766 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6934144810 0x7f6934120ff0 secure :-1 s=THROTTLE_DONE pgs=1420 cs=0 l=1 rx=0x7f6924003070 tx=0x7f6924010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:23.033 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:23.027+0000 7f692bfff700 1 -- 172.21.15.42:0/2094500766 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 173 v173) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f6924032070 con 0x7f6934144810 2019-12-05T05:45:23.035 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:45:23.035 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":173,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6077,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6835/3757738536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3757738536},{"type":"v1","addr":"172.21.15.42:6835","nonce":3757738536}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":173}],"filesystems":[{"mdsmap":{"epoch":172,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:44:49.006174+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7884},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7884":{"gid":7884,"name":"b","rank":0,"incarnation":169,"state":"up:active","state_seq":11,"addr":"172.21.15.40:6833/1127133319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1127133319},{"type":"v1","addr":"172.21.15.40:6833","nonce":1127133319}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:45:23.036 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 173 2019-12-05T05:45:23.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:23.031+0000 7f6938fa9700 1 -- 172.21.15.42:0/2094500766 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f691803ad40 msgr2=0x7f691803d250 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:45:23.039 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:23.031+0000 7f6938fa9700 1 --2- 172.21.15.42:0/2094500766 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f691803ad40 0x7f691803d250 crc :-1 s=READY pgs=610 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:45:23.039 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:23.031+0000 7f6938fa9700 1 -- 172.21.15.42:0/2094500766 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6934144810 msgr2=0x7f6934120ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:45:23.039 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:23.031+0000 7f6938fa9700 1 --2- 172.21.15.42:0/2094500766 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6934144810 0x7f6934120ff0 secure :-1 s=READY pgs=1420 cs=0 l=1 rx=0x7f6924003070 tx=0x7f6924010b60).stop 2019-12-05T05:45:23.039 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:23.031+0000 7f6938fa9700 1 -- 172.21.15.42:0/2094500766 shutdown_connections 2019-12-05T05:45:23.039 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:23.031+0000 7f6938fa9700 1 --2- 172.21.15.42:0/2094500766 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f691803ad40 0x7f691803d250 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:45:23.039 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:23.031+0000 7f6938fa9700 1 --2- 172.21.15.42:0/2094500766 >> v2:172.21.15.40:3300/0 conn(0x7f693408eb70 0x7f693408efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:45:23.040 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:23.031+0000 7f6938fa9700 1 --2- 172.21.15.42:0/2094500766 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6934144810 0x7f6934120ff0 unknown :-1 s=CLOSED pgs=1420 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:45:23.040 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:23.031+0000 7f6938fa9700 1 --2- 172.21.15.42:0/2094500766 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6934118120 0x7f6934118550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:45:23.040 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:23.031+0000 7f6938fa9700 1 -- 172.21.15.42:0/2094500766 shutdown_connections 2019-12-05T05:45:23.040 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:23.031+0000 7f6938fa9700 1 -- 172.21.15.42:0/2094500766 wait complete. 2019-12-05T05:45:23.040 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:23.031+0000 7f6938fa9700 1 -- 172.21.15.42:0/2094500766 >> 172.21.15.42:0/2094500766 conn(0x7f693407e4f0 msgr2=0x7f693411c540 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:45:23.048 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T05:45:25.049 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:45:25.154 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:25.143+0000 7f7d9a930700 1 Processor -- start 2019-12-05T05:45:25.154 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:25.143+0000 7f7d9a930700 1 -- start start 2019-12-05T05:45:25.154 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:25.143+0000 7f7d9a930700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7d940a7740 0x7f7d9408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:45:25.155 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:25.143+0000 7f7d9a930700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7d9409fae0 0x7f7d9407eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:45:25.155 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:25.143+0000 7f7d9a930700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d940a0710 con 0x7f7d9407e720 2019-12-05T05:45:25.155 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:25.143+0000 7f7d9a930700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d940a0b00 con 0x7f7d9409fae0 2019-12-05T05:45:25.156 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:25.143+0000 7f7d9a930700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d9408e5b0 con 0x7f7d940a7740 2019-12-05T05:45:25.156 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:25.143+0000 7f7d98ecd700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7d940a7740 0x7f7d9408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:45:25.157 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:25.143+0000 7f7d98ecd700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7d940a7740 0x7f7d9408df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:40954/0 (socket says 172.21.15.42:40954) 2019-12-05T05:45:25.157 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:25.143+0000 7f7d98ecd700 1 -- 172.21.15.42:0/214627749 learned_addr learned my addr 172.21.15.42:0/214627749 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:45:25.157 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:25.143+0000 7f7d996ce700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7d9409fae0 0x7f7d9407eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:45:25.157 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:25.143+0000 7f7d98ecd700 1 -- 172.21.15.42:0/214627749 >> v2:172.21.15.40:3300/0 conn(0x7f7d9409fae0 msgr2=0x7f7d9407eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:45:25.158 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:25.143+0000 7f7d98ecd700 1 --2- 172.21.15.42:0/214627749 >> v2:172.21.15.40:3300/0 conn(0x7f7d9409fae0 0x7f7d9407eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:45:25.158 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:25.143+0000 7f7d98ecd700 1 -- 172.21.15.42:0/214627749 >> v1:172.21.15.40:6789/0 conn(0x7f7d9407e720 legacy=0x7f7d9409f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:45:25.158 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:25.143+0000 7f7d98ecd700 1 -- 172.21.15.42:0/214627749 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d9407f0a0 con 0x7f7d940a7740 2019-12-05T05:45:25.158 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:25.143+0000 7f7d98ecd700 1 --2- 172.21.15.42:0/214627749 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7d940a7740 0x7f7d9408df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f7d84008b10 tx=0x7f7d840100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:25.158 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:25.143+0000 7f7d98ecd700 1 --2- 172.21.15.42:0/214627749 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7d940a7740 0x7f7d9408df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f7d84008b10 tx=0x7f7d840100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:25.159 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:25.143+0000 7f7d98ecd700 1 --2- 172.21.15.42:0/214627749 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7d940a7740 0x7f7d9408df80 secure :-1 s=READY pgs=307 cs=0 l=1 rx=0x7f7d84008b10 tx=0x7f7d840100b0).ready entity=mon.2 client_cookie=cc7edcdf87b5995b server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:45:25.159 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:25.147+0000 7f7d98ecd700 1 --2- 172.21.15.42:0/214627749 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7d940a7740 0x7f7d9408df80 secure :-1 s=THROTTLE_DONE pgs=307 cs=0 l=1 rx=0x7f7d84008b10 tx=0x7f7d840100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:25.159 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:25.147+0000 7f7d937fe700 1 -- 172.21.15.42:0/214627749 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7d84010da0 con 0x7f7d940a7740 2019-12-05T05:45:25.159 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:25.147+0000 7f7d98ecd700 1 --2- 172.21.15.42:0/214627749 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7d940a7740 0x7f7d9408df80 secure :-1 s=THROTTLE_DONE pgs=307 cs=0 l=1 rx=0x7f7d84008b10 tx=0x7f7d840100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:25.160 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:25.147+0000 7f7d98ecd700 1 --2- 172.21.15.42:0/214627749 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7d940a7740 0x7f7d9408df80 secure :-1 s=THROTTLE_DONE pgs=307 cs=0 l=1 rx=0x7f7d84008b10 tx=0x7f7d840100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:25.160 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:25.147+0000 7f7d937fe700 1 -- 172.21.15.42:0/214627749 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d840109c0 con 0x7f7d940a7740 2019-12-05T05:45:25.160 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:25.147+0000 7f7d937fe700 1 -- 172.21.15.42:0/214627749 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7d840114a0 con 0x7f7d940a7740 2019-12-05T05:45:25.160 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:25.147+0000 7f7d9a930700 1 -- 172.21.15.42:0/214627749 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7d940a7740 msgr2=0x7f7d9408df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:45:25.160 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:25.147+0000 7f7d9a930700 1 --2- 172.21.15.42:0/214627749 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7d940a7740 0x7f7d9408df80 secure :-1 s=READY pgs=307 cs=0 l=1 rx=0x7f7d84008b10 tx=0x7f7d840100b0).stop 2019-12-05T05:45:25.160 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:25.147+0000 7f7d9a930700 1 -- 172.21.15.42:0/214627749 shutdown_connections 2019-12-05T05:45:25.161 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:25.147+0000 7f7d9a930700 1 --2- 172.21.15.42:0/214627749 >> v2:172.21.15.40:3300/0 conn(0x7f7d9409fae0 0x7f7d9407eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:45:25.161 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:25.147+0000 7f7d9a930700 1 --2- 172.21.15.42:0/214627749 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7d940a7740 0x7f7d9408df80 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:45:25.161 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:25.147+0000 7f7d9a930700 1 -- 172.21.15.42:0/214627749 shutdown_connections 2019-12-05T05:45:25.161 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:25.147+0000 7f7d9a930700 1 -- 172.21.15.42:0/214627749 wait complete. 2019-12-05T05:45:25.162 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:25.147+0000 7f7d9a930700 1 -- 172.21.15.42:0/214627749 >> 172.21.15.42:0/214627749 conn(0x7f7d940aabe0 msgr2=0x7f7d940a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:45:25.169 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:25.151+0000 7f7d9a930700 1 Processor -- start 2019-12-05T05:45:25.169 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:25.151+0000 7f7d9a930700 1 -- start start 2019-12-05T05:45:25.169 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:25.151+0000 7f7d9a930700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f7d94124260 0x7f7d94138dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:45:25.169 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:25.151+0000 7f7d9a930700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d94120150 con 0x7f7d9407df50 2019-12-05T05:45:25.170 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:25.151+0000 7f7d9a930700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d9411f840 con 0x7f7d94116eb0 2019-12-05T05:45:25.170 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:25.151+0000 7f7d9a930700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d9411fc10 con 0x7f7d94124260 2019-12-05T05:45:25.170 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:25.151+0000 7f7d98ecd700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f7d94116eb0 0x7f7d9411e470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:40532/0 (socket says 172.21.15.42:40532) 2019-12-05T05:45:25.170 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:25.151+0000 7f7d996ce700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f7d94124260 0x7f7d94138dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:45:25.170 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:25.151+0000 7f7d98ecd700 1 -- 172.21.15.42:0/1314840958 learned_addr learned my addr 172.21.15.42:0/1314840958 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:45:25.170 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:25.151+0000 7f7d917fa700 1 -- 172.21.15.42:0/1314840958 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 140695383 0 0) 0x7f7d940a0710 con 0x7f7d9407df50 2019-12-05T05:45:25.170 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:25.151+0000 7f7d917fa700 1 -- 172.21.15.42:0/1314840958 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d740030b0 con 0x7f7d9407df50 2019-12-05T05:45:25.171 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:25.151+0000 7f7d917fa700 1 -- 172.21.15.42:0/1314840958 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2300882840 0 0) 0x7f7d9411f840 con 0x7f7d94116eb0 2019-12-05T05:45:25.171 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:25.151+0000 7f7d996ce700 1 -- 172.21.15.42:0/1314840958 >> v1:172.21.15.42:6789/0 conn(0x7f7d94116eb0 legacy=0x7f7d9411e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:45:25.171 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:25.151+0000 7f7d996ce700 1 -- 172.21.15.42:0/1314840958 >> v1:172.21.15.40:6789/0 conn(0x7f7d9407df50 legacy=0x7f7d941355a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:45:25.171 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:25.151+0000 7f7d996ce700 1 -- 172.21.15.42:0/1314840958 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d94124690 con 0x7f7d94124260 2019-12-05T05:45:25.171 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:25.151+0000 7f7d996ce700 1 --2- 172.21.15.42:0/1314840958 >> v2:172.21.15.42:3300/0 conn(0x7f7d94124260 0x7f7d94138dc0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f7d8c002f30 tx=0x7f7d8c010900).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:25.174 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:25.151+0000 7f7d996ce700 1 --2- 172.21.15.42:0/1314840958 >> v2:172.21.15.42:3300/0 conn(0x7f7d94124260 0x7f7d94138dc0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f7d8c002f30 tx=0x7f7d8c010900).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:25.175 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:25.151+0000 7f7d996ce700 1 --2- 172.21.15.42:0/1314840958 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7d94124260 0x7f7d94138dc0 secure :-1 s=READY pgs=1422 cs=0 l=1 rx=0x7f7d8c002f30 tx=0x7f7d8c010900).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:45:25.175 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:25.151+0000 7f7d996ce700 1 --2- 172.21.15.42:0/1314840958 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7d94124260 0x7f7d94138dc0 secure :-1 s=THROTTLE_DONE pgs=1422 cs=0 l=1 rx=0x7f7d8c002f30 tx=0x7f7d8c010900).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:25.175 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:25.151+0000 7f7d996ce700 1 --2- 172.21.15.42:0/1314840958 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7d94124260 0x7f7d94138dc0 secure :-1 s=THROTTLE_DONE pgs=1422 cs=0 l=1 rx=0x7f7d8c002f30 tx=0x7f7d8c010900).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:25.175 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:25.151+0000 7f7d917fa700 1 -- 172.21.15.42:0/1314840958 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7d8c002730 con 0x7f7d94124260 2019-12-05T05:45:25.175 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:25.151+0000 7f7d996ce700 1 --2- 172.21.15.42:0/1314840958 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7d94124260 0x7f7d94138dc0 secure :-1 s=THROTTLE_DONE pgs=1422 cs=0 l=1 rx=0x7f7d8c002f30 tx=0x7f7d8c010900).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:25.175 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:25.151+0000 7f7d917fa700 1 -- 172.21.15.42:0/1314840958 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d8c002350 con 0x7f7d94124260 2019-12-05T05:45:25.176 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:25.151+0000 7f7d9a930700 1 -- 172.21.15.42:0/1314840958 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d94118290 con 0x7f7d94124260 2019-12-05T05:45:25.176 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:25.151+0000 7f7d917fa700 1 -- 172.21.15.42:0/1314840958 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7d8c00d160 con 0x7f7d94124260 2019-12-05T05:45:25.176 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:25.151+0000 7f7d9a930700 1 -- 172.21.15.42:0/1314840958 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d9411ec70 con 0x7f7d94124260 2019-12-05T05:45:25.176 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:25.151+0000 7f7d996ce700 1 -- 172.21.15.42:0/1314840958 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7d94124260 msgr2=0x7f7d94138dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2019-12-05T05:45:25.176 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:25.151+0000 7f7d996ce700 1 -- 172.21.15.42:0/1314840958 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7d94124260 msgr2=0x7f7d94138dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2019-12-05T05:45:25.176 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:25.151+0000 7f7d996ce700 1 --2- 172.21.15.42:0/1314840958 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7d94124260 0x7f7d94138dc0 secure :-1 s=READY pgs=1422 cs=0 l=1 rx=0x7f7d8c002f30 tx=0x7f7d8c010900).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-12-05T05:45:25.176 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:25.151+0000 7f7d996ce700 1 --2- 172.21.15.42:0/1314840958 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7d94124260 0x7f7d94138dc0 secure :-1 s=READY pgs=1422 cs=0 l=1 rx=0x7f7d8c002f30 tx=0x7f7d8c010900).stop 2019-12-05T05:45:25.177 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:25.151+0000 7f7d917fa700 1 -- 172.21.15.42:0/1314840958 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7d94124260 msgr2=0x7f7d94138dc0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2019-12-05T05:45:25.177 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:25.151+0000 7f7d917fa700 1 --2- 172.21.15.42:0/1314840958 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7d94124260 0x7f7d94138dc0 unknown :-1 s=CLOSED pgs=1422 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:45:25.177 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:25.155+0000 7f7d917fa700 1 --2- 172.21.15.42:0/1314840958 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7d740033c0 0x7f7d740037f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:45:25.177 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:25.155+0000 7f7d917fa700 1 --2- 172.21.15.42:0/1314840958 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7d74003d40 0x7f7d74004420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:45:25.177 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:25.155+0000 7f7d917fa700 1 --2- 172.21.15.42:0/1314840958 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7d74004970 0x7f7d74004da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:45:25.177 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:25.155+0000 7f7d917fa700 1 -- 172.21.15.42:0/1314840958 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d74005640 con 0x7f7d740033c0 2019-12-05T05:45:25.177 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:25.155+0000 7f7d917fa700 1 -- 172.21.15.42:0/1314840958 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d74005830 con 0x7f7d74003d40 2019-12-05T05:45:25.178 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:25.155+0000 7f7d917fa700 1 -- 172.21.15.42:0/1314840958 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d74005a70 con 0x7f7d74004970 2019-12-05T05:45:25.178 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:25.155+0000 7f7d93fff700 1 --2- 172.21.15.42:0/1314840958 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7d74003d40 0x7f7d74004420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:45:25.178 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:25.155+0000 7f7d98ecd700 1 --2- 172.21.15.42:0/1314840958 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7d740033c0 0x7f7d740037f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:45:25.178 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:25.155+0000 7f7d996ce700 1 --2- 172.21.15.42:0/1314840958 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7d74004970 0x7f7d74004da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:45:25.178 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:25.155+0000 7f7d996ce700 1 -- 172.21.15.42:0/1314840958 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7d74003d40 msgr2=0x7f7d74004420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:45:25.178 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:25.155+0000 7f7d996ce700 1 --2- 172.21.15.42:0/1314840958 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7d74003d40 0x7f7d74004420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:45:25.179 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:25.155+0000 7f7d996ce700 1 -- 172.21.15.42:0/1314840958 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7d740033c0 msgr2=0x7f7d740037f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:45:25.179 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:25.155+0000 7f7d996ce700 1 --2- 172.21.15.42:0/1314840958 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7d740033c0 0x7f7d740037f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:45:25.179 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:25.155+0000 7f7d996ce700 1 -- 172.21.15.42:0/1314840958 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+,osdmap=0}) v3 -- 0x7f7d74005e30 con 0x7f7d74004970 2019-12-05T05:45:25.179 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:25.155+0000 7f7d93fff700 1 --2- 172.21.15.42:0/1314840958 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7d74003d40 0x7f7d74004420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:45:25.179 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:25.155+0000 7f7d996ce700 1 --2- 172.21.15.42:0/1314840958 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7d74004970 0x7f7d74004da0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f7d8c002f30 tx=0x7f7d8c005ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:25.180 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:25.155+0000 7f7d996ce700 1 -- 172.21.15.42:0/1314840958 reap_dead start 2019-12-05T05:45:25.180 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:25.155+0000 7f7d996ce700 1 --2- 172.21.15.42:0/1314840958 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7d74004970 0x7f7d74004da0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f7d8c002f30 tx=0x7f7d8c005ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:25.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:25.155+0000 7f7d996ce700 1 --2- 172.21.15.42:0/1314840958 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7d74004970 0x7f7d74004da0 secure :-1 s=READY pgs=308 cs=0 l=1 rx=0x7f7d8c002f30 tx=0x7f7d8c005ae0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:45:25.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:25.155+0000 7f7d996ce700 1 --2- 172.21.15.42:0/1314840958 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7d74004970 0x7f7d74004da0 secure :-1 s=THROTTLE_DONE pgs=308 cs=0 l=1 rx=0x7f7d8c002f30 tx=0x7f7d8c005ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:25.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:25.155+0000 7f7d917fa700 1 -- 172.21.15.42:0/1314840958 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7d8c00d610 con 0x7f7d74004970 2019-12-05T05:45:25.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:25.155+0000 7f7d996ce700 1 --2- 172.21.15.42:0/1314840958 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7d74004970 0x7f7d74004da0 secure :-1 s=THROTTLE_DONE pgs=308 cs=0 l=1 rx=0x7f7d8c002f30 tx=0x7f7d8c005ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:25.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:25.155+0000 7f7d917fa700 1 -- 172.21.15.42:0/1314840958 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d8c002930 con 0x7f7d74004970 2019-12-05T05:45:25.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:25.155+0000 7f7d996ce700 1 --2- 172.21.15.42:0/1314840958 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7d74004970 0x7f7d74004da0 secure :-1 s=THROTTLE_DONE pgs=308 cs=0 l=1 rx=0x7f7d8c002f30 tx=0x7f7d8c005ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:25.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:25.155+0000 7f7d9a930700 1 -- 172.21.15.42:0/1314840958 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d9405d110 con 0x7f7d74004970 2019-12-05T05:45:25.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:25.155+0000 7f7d917fa700 1 -- 172.21.15.42:0/1314840958 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f7d8c00d810 con 0x7f7d74004970 2019-12-05T05:45:25.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:25.155+0000 7f7d996ce700 1 --2- 172.21.15.42:0/1314840958 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7d74004970 0x7f7d74004da0 secure :-1 s=THROTTLE_DONE pgs=308 cs=0 l=1 rx=0x7f7d8c002f30 tx=0x7f7d8c005ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:25.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:25.155+0000 7f7d917fa700 1 --2- 172.21.15.42:0/1314840958 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f7d740471c0 0x7f7d740496d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:45:25.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:25.155+0000 7f7d917fa700 1 -- 172.21.15.42:0/1314840958 <== mon.2 v2:172.21.15.40:3301/0 4 ==== osd_map(77..77 src has 1..77) v4 ==== 8439+0+0 (secure 0 0 0) 0x7f7d8c04c0e0 con 0x7f7d74004970 2019-12-05T05:45:25.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:25.159+0000 7f7d98ecd700 1 --2- 172.21.15.42:0/1314840958 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f7d740471c0 0x7f7d740496d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:45:25.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:25.159+0000 7f7d98ecd700 1 --2- 172.21.15.42:0/1314840958 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f7d740471c0 0x7f7d740496d0 crc :-1 s=READY pgs=611 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:45:25.184 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:25.163+0000 7f7d996ce700 1 --2- 172.21.15.42:0/1314840958 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7d74004970 0x7f7d74004da0 secure :-1 s=THROTTLE_DONE pgs=308 cs=0 l=1 rx=0x7f7d8c002f30 tx=0x7f7d8c005ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:25.187 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:25.163+0000 7f7d917fa700 1 -- 172.21.15.42:0/1314840958 <== mon.2 v2:172.21.15.40:3301/0 5 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f7d8c033160 con 0x7f7d74004970 2019-12-05T05:45:25.342 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:25.335+0000 7f7d9a930700 1 -- 172.21.15.42:0/1314840958 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7d9405b8c0 con 0x7f7d74004970 2019-12-05T05:45:25.343 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:25.335+0000 7f7d996ce700 1 --2- 172.21.15.42:0/1314840958 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7d74004970 0x7f7d74004da0 secure :-1 s=THROTTLE_DONE pgs=308 cs=0 l=1 rx=0x7f7d8c002f30 tx=0x7f7d8c005ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:25.343 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:25.335+0000 7f7d917fa700 1 -- 172.21.15.42:0/1314840958 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 173 v173) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f7d8c0333c0 con 0x7f7d74004970 2019-12-05T05:45:25.345 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 173 2019-12-05T05:45:25.345 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:45:25.345 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":173,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6077,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6835/3757738536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3757738536},{"type":"v1","addr":"172.21.15.42:6835","nonce":3757738536}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":173}],"filesystems":[{"mdsmap":{"epoch":172,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:44:49.006174+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7884},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7884":{"gid":7884,"name":"b","rank":0,"incarnation":169,"state":"up:active","state_seq":11,"addr":"172.21.15.40:6833/1127133319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1127133319},{"type":"v1","addr":"172.21.15.40:6833","nonce":1127133319}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:45:25.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:25.343+0000 7f7d9a930700 1 -- 172.21.15.42:0/1314840958 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f7d740471c0 msgr2=0x7f7d740496d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:45:25.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:25.343+0000 7f7d9a930700 1 --2- 172.21.15.42:0/1314840958 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f7d740471c0 0x7f7d740496d0 crc :-1 s=READY pgs=611 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:45:25.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:25.343+0000 7f7d9a930700 1 -- 172.21.15.42:0/1314840958 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7d74004970 msgr2=0x7f7d74004da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:45:25.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:25.343+0000 7f7d9a930700 1 --2- 172.21.15.42:0/1314840958 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7d74004970 0x7f7d74004da0 secure :-1 s=READY pgs=308 cs=0 l=1 rx=0x7f7d8c002f30 tx=0x7f7d8c005ae0).stop 2019-12-05T05:45:25.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:25.343+0000 7f7d9a930700 1 -- 172.21.15.42:0/1314840958 shutdown_connections 2019-12-05T05:45:25.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:25.343+0000 7f7d9a930700 1 --2- 172.21.15.42:0/1314840958 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f7d740471c0 0x7f7d740496d0 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:45:25.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:25.343+0000 7f7d9a930700 1 --2- 172.21.15.42:0/1314840958 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7d74004970 0x7f7d74004da0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:45:25.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:25.343+0000 7f7d9a930700 1 --2- 172.21.15.42:0/1314840958 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7d94124260 0x7f7d94138dc0 unknown :-1 s=CLOSED pgs=1422 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:45:25.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:25.343+0000 7f7d9a930700 1 -- 172.21.15.42:0/1314840958 shutdown_connections 2019-12-05T05:45:25.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:25.343+0000 7f7d9a930700 1 -- 172.21.15.42:0/1314840958 wait complete. 2019-12-05T05:45:25.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:25.343+0000 7f7d9a930700 1 -- 172.21.15.42:0/1314840958 >> 172.21.15.42:0/1314840958 conn(0x7f7d9405a190 msgr2=0x7f7d940a7f90 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:45:25.357 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T05:45:27.359 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:45:27.473 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:27.459+0000 7f9e7287c700 1 Processor -- start 2019-12-05T05:45:27.473 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:27.459+0000 7f9e7287c700 1 -- start start 2019-12-05T05:45:27.473 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:27.463+0000 7f9e7287c700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9e6c08df80 0x7f9e6c09f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:45:27.473 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:27.463+0000 7f9e7287c700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f9e6c09f830 0x7f9e6c14efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:45:27.473 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:27.463+0000 7f9e7287c700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e6c09fc60 con 0x7f9e6c0a7740 2019-12-05T05:45:27.474 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:27.463+0000 7f9e7287c700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e6c0a0b00 con 0x7f9e6c09f830 2019-12-05T05:45:27.474 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:27.463+0000 7f9e7287c700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e6c08e5b0 con 0x7f9e6c08df80 2019-12-05T05:45:27.474 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:27.463+0000 7f9e70e19700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f9e6c0a7740 0x7f9e6c07e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:40548/0 (socket says 172.21.15.42:40548) 2019-12-05T05:45:27.474 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:27.463+0000 7f9e70e19700 1 -- 172.21.15.42:0/1262749242 learned_addr learned my addr 172.21.15.42:0/1262749242 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:45:27.475 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:27.463+0000 7f9e7161a700 1 --2- 172.21.15.42:0/1262749242 >> v2:172.21.15.40:3300/0 conn(0x7f9e6c09f830 0x7f9e6c14efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:45:27.475 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:27.463+0000 7f9e6bfff700 1 --2- 172.21.15.42:0/1262749242 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9e6c08df80 0x7f9e6c09f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:45:27.475 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:27.463+0000 7f9e6b7fe700 1 -- 172.21.15.42:0/1262749242 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 495976088 0 0) 0x7f9e6c09fc60 con 0x7f9e6c0a7740 2019-12-05T05:45:27.475 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:27.463+0000 7f9e6b7fe700 1 -- 172.21.15.42:0/1262749242 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e540030b0 con 0x7f9e6c0a7740 2019-12-05T05:45:27.476 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:27.463+0000 7f9e6bfff700 1 -- 172.21.15.42:0/1262749242 >> v2:172.21.15.40:3300/0 conn(0x7f9e6c09f830 msgr2=0x7f9e6c14efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:45:27.476 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:27.463+0000 7f9e6bfff700 1 --2- 172.21.15.42:0/1262749242 >> v2:172.21.15.40:3300/0 conn(0x7f9e6c09f830 0x7f9e6c14efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:45:27.476 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:27.463+0000 7f9e6bfff700 1 -- 172.21.15.42:0/1262749242 >> v1:172.21.15.42:6789/0 conn(0x7f9e6c0a7740 legacy=0x7f9e6c07e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:45:27.476 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:27.463+0000 7f9e6bfff700 1 -- 172.21.15.42:0/1262749242 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e6c0a0710 con 0x7f9e6c08df80 2019-12-05T05:45:27.477 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:27.463+0000 7f9e7161a700 1 --2- 172.21.15.42:0/1262749242 >> v2:172.21.15.40:3300/0 conn(0x7f9e6c09f830 0x7f9e6c14efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:45:27.477 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:27.463+0000 7f9e6bfff700 1 --2- 172.21.15.42:0/1262749242 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9e6c08df80 0x7f9e6c09f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f9e60008790 tx=0x7f9e6000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:27.477 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:27.463+0000 7f9e6bfff700 1 --2- 172.21.15.42:0/1262749242 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9e6c08df80 0x7f9e6c09f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f9e60008790 tx=0x7f9e6000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:27.477 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:27.463+0000 7f9e6bfff700 1 --2- 172.21.15.42:0/1262749242 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9e6c08df80 0x7f9e6c09f2e0 secure :-1 s=READY pgs=309 cs=0 l=1 rx=0x7f9e60008790 tx=0x7f9e6000fc70).ready entity=mon.2 client_cookie=7ec39093b785510e server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:45:27.477 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:27.463+0000 7f9e6bfff700 1 --2- 172.21.15.42:0/1262749242 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9e6c08df80 0x7f9e6c09f2e0 secure :-1 s=THROTTLE_DONE pgs=309 cs=0 l=1 rx=0x7f9e60008790 tx=0x7f9e6000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:27.477 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:27.463+0000 7f9e6b7fe700 1 -- 172.21.15.42:0/1262749242 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9e60008bb0 con 0x7f9e6c08df80 2019-12-05T05:45:27.477 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:27.463+0000 7f9e6bfff700 1 --2- 172.21.15.42:0/1262749242 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9e6c08df80 0x7f9e6c09f2e0 secure :-1 s=THROTTLE_DONE pgs=309 cs=0 l=1 rx=0x7f9e60008790 tx=0x7f9e6000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:27.478 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:27.463+0000 7f9e6bfff700 1 --2- 172.21.15.42:0/1262749242 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9e6c08df80 0x7f9e6c09f2e0 secure :-1 s=THROTTLE_DONE pgs=309 cs=0 l=1 rx=0x7f9e60008790 tx=0x7f9e6000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:27.478 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:27.463+0000 7f9e6b7fe700 1 -- 172.21.15.42:0/1262749242 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e600087d0 con 0x7f9e6c08df80 2019-12-05T05:45:27.479 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:27.463+0000 7f9e6b7fe700 1 -- 172.21.15.42:0/1262749242 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9e60011860 con 0x7f9e6c08df80 2019-12-05T05:45:27.479 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:27.463+0000 7f9e7287c700 1 -- 172.21.15.42:0/1262749242 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9e6c08df80 msgr2=0x7f9e6c09f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:45:27.479 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:27.463+0000 7f9e7287c700 1 --2- 172.21.15.42:0/1262749242 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9e6c08df80 0x7f9e6c09f2e0 secure :-1 s=READY pgs=309 cs=0 l=1 rx=0x7f9e60008790 tx=0x7f9e6000fc70).stop 2019-12-05T05:45:27.479 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:27.467+0000 7f9e7287c700 1 -- 172.21.15.42:0/1262749242 shutdown_connections 2019-12-05T05:45:27.479 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:27.467+0000 7f9e7287c700 1 --2- 172.21.15.42:0/1262749242 >> v2:172.21.15.40:3300/0 conn(0x7f9e6c09f830 0x7f9e6c14efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:45:27.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:27.467+0000 7f9e7287c700 1 --2- 172.21.15.42:0/1262749242 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9e6c08df80 0x7f9e6c09f2e0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:45:27.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:27.467+0000 7f9e7287c700 1 -- 172.21.15.42:0/1262749242 shutdown_connections 2019-12-05T05:45:27.486 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:27.467+0000 7f9e7287c700 1 -- 172.21.15.42:0/1262749242 wait complete. 2019-12-05T05:45:27.486 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:27.467+0000 7f9e7287c700 1 -- 172.21.15.42:0/1262749242 >> 172.21.15.42:0/1262749242 conn(0x7f9e6c0aabe0 msgr2=0x7f9e6c0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:45:27.492 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:27.467+0000 7f9e7287c700 1 Processor -- start 2019-12-05T05:45:27.492 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:27.467+0000 7f9e7287c700 1 -- start start 2019-12-05T05:45:27.492 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:27.471+0000 7f9e7287c700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f9e6c073ad0 0x7f9e6c075f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:45:27.492 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:27.471+0000 7f9e7287c700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e6c1165f0 con 0x7f9e6c118840 2019-12-05T05:45:27.493 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:27.471+0000 7f9e7287c700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e6c119f10 con 0x7f9e6c1161c0 2019-12-05T05:45:27.493 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:27.471+0000 7f9e7287c700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e6c1190e0 con 0x7f9e6c073ad0 2019-12-05T05:45:27.493 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:27.471+0000 7f9e70e19700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f9e6c1161c0 0x7f9e6c11ba50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:40554/0 (socket says 172.21.15.42:40554) 2019-12-05T05:45:27.493 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:27.471+0000 7f9e70e19700 1 -- 172.21.15.42:0/2785051717 learned_addr learned my addr 172.21.15.42:0/2785051717 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:45:27.493 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:27.471+0000 7f9e7161a700 1 --2- 172.21.15.42:0/2785051717 >> v2:172.21.15.40:3300/0 conn(0x7f9e6c073ad0 0x7f9e6c075f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:45:27.493 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:27.471+0000 7f9e697fa700 1 -- 172.21.15.42:0/2785051717 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 28147841 0 0) 0x7f9e6c119f10 con 0x7f9e6c1161c0 2019-12-05T05:45:27.493 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:27.471+0000 7f9e697fa700 1 -- 172.21.15.42:0/2785051717 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e4c0030b0 con 0x7f9e6c1161c0 2019-12-05T05:45:27.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:27.471+0000 7f9e7161a700 1 -- 172.21.15.42:0/2785051717 >> v1:172.21.15.42:6789/0 conn(0x7f9e6c1161c0 legacy=0x7f9e6c11ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:45:27.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:27.471+0000 7f9e7161a700 1 -- 172.21.15.42:0/2785051717 >> v1:172.21.15.40:6789/0 conn(0x7f9e6c118840 legacy=0x7f9e6c0722c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:45:27.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:27.471+0000 7f9e7161a700 1 -- 172.21.15.42:0/2785051717 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e6c134670 con 0x7f9e6c073ad0 2019-12-05T05:45:27.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:27.471+0000 7f9e7161a700 1 --2- 172.21.15.42:0/2785051717 >> v2:172.21.15.40:3300/0 conn(0x7f9e6c073ad0 0x7f9e6c075f10 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f9e640032b0 tx=0x7f9e64010e80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:27.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:27.471+0000 7f9e7161a700 1 --2- 172.21.15.42:0/2785051717 >> v2:172.21.15.40:3300/0 conn(0x7f9e6c073ad0 0x7f9e6c075f10 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f9e640032b0 tx=0x7f9e64010e80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:27.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:27.471+0000 7f9e7161a700 1 --2- 172.21.15.42:0/2785051717 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9e6c073ad0 0x7f9e6c075f10 secure :-1 s=READY pgs=1098 cs=0 l=1 rx=0x7f9e640032b0 tx=0x7f9e64010e80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:45:27.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:27.471+0000 7f9e7161a700 1 --2- 172.21.15.42:0/2785051717 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9e6c073ad0 0x7f9e6c075f10 secure :-1 s=THROTTLE_DONE pgs=1098 cs=0 l=1 rx=0x7f9e640032b0 tx=0x7f9e64010e80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:27.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:27.471+0000 7f9e7161a700 1 --2- 172.21.15.42:0/2785051717 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9e6c073ad0 0x7f9e6c075f10 secure :-1 s=THROTTLE_DONE pgs=1098 cs=0 l=1 rx=0x7f9e640032b0 tx=0x7f9e64010e80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:27.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:27.471+0000 7f9e697fa700 1 -- 172.21.15.42:0/2785051717 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9e64005d40 con 0x7f9e6c073ad0 2019-12-05T05:45:27.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:27.471+0000 7f9e7161a700 1 --2- 172.21.15.42:0/2785051717 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9e6c073ad0 0x7f9e6c075f10 secure :-1 s=THROTTLE_DONE pgs=1098 cs=0 l=1 rx=0x7f9e640032b0 tx=0x7f9e64010e80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:27.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:27.471+0000 7f9e697fa700 1 -- 172.21.15.42:0/2785051717 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e64005960 con 0x7f9e6c073ad0 2019-12-05T05:45:27.496 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:27.471+0000 7f9e697fa700 1 -- 172.21.15.42:0/2785051717 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9e64011c10 con 0x7f9e6c073ad0 2019-12-05T05:45:27.496 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:27.471+0000 7f9e7287c700 1 -- 172.21.15.42:0/2785051717 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e6c118c70 con 0x7f9e6c073ad0 2019-12-05T05:45:27.496 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:27.471+0000 7f9e7287c700 1 -- 172.21.15.42:0/2785051717 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e6c11c250 con 0x7f9e6c073ad0 2019-12-05T05:45:27.496 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:27.475+0000 7f9e7161a700 1 --2- 172.21.15.42:0/2785051717 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9e6c073ad0 0x7f9e6c075f10 secure :-1 s=THROTTLE_DONE pgs=1098 cs=0 l=1 rx=0x7f9e640032b0 tx=0x7f9e64010e80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:27.496 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:27.475+0000 7f9e697fa700 1 -- 172.21.15.42:0/2785051717 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f9e64007180 con 0x7f9e6c073ad0 2019-12-05T05:45:27.496 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:27.475+0000 7f9e7161a700 1 --2- 172.21.15.42:0/2785051717 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9e6c073ad0 0x7f9e6c075f10 secure :-1 s=THROTTLE_DONE pgs=1098 cs=0 l=1 rx=0x7f9e640032b0 tx=0x7f9e64010e80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:27.497 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:27.475+0000 7f9e697fa700 1 --2- 172.21.15.42:0/2785051717 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9e4c03b260 0x7f9e4c03d770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:45:27.497 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:27.475+0000 7f9e697fa700 1 -- 172.21.15.42:0/2785051717 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 8439+0+0 (secure 0 0 0) 0x7f9e6404ae40 con 0x7f9e6c073ad0 2019-12-05T05:45:27.497 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:27.475+0000 7f9e70e19700 1 --2- 172.21.15.42:0/2785051717 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9e4c03b260 0x7f9e4c03d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:45:27.497 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:27.475+0000 7f9e70e19700 1 --2- 172.21.15.42:0/2785051717 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9e4c03b260 0x7f9e4c03d770 crc :-1 s=READY pgs=612 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:45:27.497 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:27.475+0000 7f9e7287c700 1 -- 172.21.15.42:0/2785051717 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e6c05d110 con 0x7f9e6c073ad0 2019-12-05T05:45:27.498 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:27.487+0000 7f9e7161a700 1 --2- 172.21.15.42:0/2785051717 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9e6c073ad0 0x7f9e6c075f10 secure :-1 s=THROTTLE_DONE pgs=1098 cs=0 l=1 rx=0x7f9e640032b0 tx=0x7f9e64010e80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:27.498 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:27.487+0000 7f9e697fa700 1 -- 172.21.15.42:0/2785051717 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f9e6400e160 con 0x7f9e6c073ad0 2019-12-05T05:45:27.667 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:27.659+0000 7f9e7287c700 1 -- 172.21.15.42:0/2785051717 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9e6c05b8c0 con 0x7f9e6c073ad0 2019-12-05T05:45:27.668 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:27.659+0000 7f9e7161a700 1 --2- 172.21.15.42:0/2785051717 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9e6c073ad0 0x7f9e6c075f10 secure :-1 s=THROTTLE_DONE pgs=1098 cs=0 l=1 rx=0x7f9e640032b0 tx=0x7f9e64010e80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:27.668 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:27.659+0000 7f9e697fa700 1 -- 172.21.15.42:0/2785051717 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 173 v173) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f9e64032cd0 con 0x7f9e6c073ad0 2019-12-05T05:45:27.668 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 173 2019-12-05T05:45:27.669 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:45:27.669 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":173,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6077,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6835/3757738536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3757738536},{"type":"v1","addr":"172.21.15.42:6835","nonce":3757738536}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":173}],"filesystems":[{"mdsmap":{"epoch":172,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:44:49.006174+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7884},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7884":{"gid":7884,"name":"b","rank":0,"incarnation":169,"state":"up:active","state_seq":11,"addr":"172.21.15.40:6833/1127133319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1127133319},{"type":"v1","addr":"172.21.15.40:6833","nonce":1127133319}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:45:27.670 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:27.663+0000 7f9e7287c700 1 -- 172.21.15.42:0/2785051717 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9e4c03b260 msgr2=0x7f9e4c03d770 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:45:27.670 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:27.663+0000 7f9e7287c700 1 --2- 172.21.15.42:0/2785051717 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9e4c03b260 0x7f9e4c03d770 crc :-1 s=READY pgs=612 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:45:27.671 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:27.663+0000 7f9e7287c700 1 -- 172.21.15.42:0/2785051717 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9e6c073ad0 msgr2=0x7f9e6c075f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:45:27.671 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:27.663+0000 7f9e7287c700 1 --2- 172.21.15.42:0/2785051717 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9e6c073ad0 0x7f9e6c075f10 secure :-1 s=READY pgs=1098 cs=0 l=1 rx=0x7f9e640032b0 tx=0x7f9e64010e80).stop 2019-12-05T05:45:27.671 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:27.663+0000 7f9e7287c700 1 -- 172.21.15.42:0/2785051717 shutdown_connections 2019-12-05T05:45:27.672 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:27.663+0000 7f9e7287c700 1 --2- 172.21.15.42:0/2785051717 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9e4c03b260 0x7f9e4c03d770 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:45:27.672 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:27.663+0000 7f9e7287c700 1 --2- 172.21.15.42:0/2785051717 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9e6c073ad0 0x7f9e6c075f10 unknown :-1 s=CLOSED pgs=1098 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:45:27.672 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:27.663+0000 7f9e7287c700 1 -- 172.21.15.42:0/2785051717 shutdown_connections 2019-12-05T05:45:27.672 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:27.663+0000 7f9e7287c700 1 -- 172.21.15.42:0/2785051717 wait complete. 2019-12-05T05:45:27.672 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:27.663+0000 7f9e7287c700 1 -- 172.21.15.42:0/2785051717 >> 172.21.15.42:0/2785051717 conn(0x7f9e6c0a9aa0 msgr2=0x7f9e6c059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:45:27.681 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T05:45:29.682 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:45:29.791 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:29.779+0000 7f25f5f3f700 1 Processor -- start 2019-12-05T05:45:29.791 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:29.779+0000 7f25f5f3f700 1 -- start start 2019-12-05T05:45:29.791 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:29.779+0000 7f25f5f3f700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f25f008df70 0x7f25f009f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:45:29.791 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:29.779+0000 7f25f5f3f700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25f009fc50 con 0x7f25f009f820 2019-12-05T05:45:29.792 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:29.779+0000 7f25f5f3f700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25f00a0700 con 0x7f25f00a7730 2019-12-05T05:45:29.792 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:29.779+0000 7f25f5f3f700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25f00a0af0 con 0x7f25f008df70 2019-12-05T05:45:29.792 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:29.779+0000 7f25effff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f25f00a7730 0x7f25f007e710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:40562/0 (socket says 172.21.15.42:40562) 2019-12-05T05:45:29.792 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:29.779+0000 7f25effff700 1 -- 172.21.15.42:0/1115076392 learned_addr learned my addr 172.21.15.42:0/1115076392 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:45:29.792 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:29.779+0000 7f25ef7fe700 1 --2- 172.21.15.42:0/1115076392 >> v2:172.21.15.40:3300/0 conn(0x7f25f008df70 0x7f25f009f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:45:29.792 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:29.783+0000 7f25eeffd700 1 -- 172.21.15.42:0/1115076392 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1673569415 0 0) 0x7f25f00a0700 con 0x7f25f00a7730 2019-12-05T05:45:29.792 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:29.783+0000 7f25eeffd700 1 -- 172.21.15.42:0/1115076392 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25d00030b0 con 0x7f25f00a7730 2019-12-05T05:45:29.793 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:29.783+0000 7f25ef7fe700 1 -- 172.21.15.42:0/1115076392 >> v1:172.21.15.42:6789/0 conn(0x7f25f00a7730 legacy=0x7f25f007e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:45:29.793 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:29.783+0000 7f25ef7fe700 1 -- 172.21.15.42:0/1115076392 >> v1:172.21.15.40:6789/0 conn(0x7f25f009f820 legacy=0x7f25f014efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:45:29.793 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:29.783+0000 7f25ef7fe700 1 -- 172.21.15.42:0/1115076392 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25f007ef10 con 0x7f25f008df70 2019-12-05T05:45:29.793 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:29.783+0000 7f25ef7fe700 1 --2- 172.21.15.42:0/1115076392 >> v2:172.21.15.40:3300/0 conn(0x7f25f008df70 0x7f25f009f2d0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f25e0008790 tx=0x7f25e000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:29.793 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:29.783+0000 7f25ef7fe700 1 --2- 172.21.15.42:0/1115076392 >> v2:172.21.15.40:3300/0 conn(0x7f25f008df70 0x7f25f009f2d0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f25e0008790 tx=0x7f25e000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:29.793 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:29.783+0000 7f25ef7fe700 1 --2- 172.21.15.42:0/1115076392 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f25f008df70 0x7f25f009f2d0 secure :-1 s=READY pgs=1100 cs=0 l=1 rx=0x7f25e0008790 tx=0x7f25e000fbf0).ready entity=mon.0 client_cookie=4d9f31e6668eb659 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:45:29.793 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:29.783+0000 7f25ef7fe700 1 --2- 172.21.15.42:0/1115076392 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f25f008df70 0x7f25f009f2d0 secure :-1 s=THROTTLE_DONE pgs=1100 cs=0 l=1 rx=0x7f25e0008790 tx=0x7f25e000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:29.794 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:29.783+0000 7f25eeffd700 1 -- 172.21.15.42:0/1115076392 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f25e0011270 con 0x7f25f008df70 2019-12-05T05:45:29.794 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:29.783+0000 7f25ef7fe700 1 --2- 172.21.15.42:0/1115076392 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f25f008df70 0x7f25f009f2d0 secure :-1 s=THROTTLE_DONE pgs=1100 cs=0 l=1 rx=0x7f25e0008790 tx=0x7f25e000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:29.794 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:29.783+0000 7f25eeffd700 1 -- 172.21.15.42:0/1115076392 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25e0008bc0 con 0x7f25f008df70 2019-12-05T05:45:29.794 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:29.783+0000 7f25ef7fe700 1 --2- 172.21.15.42:0/1115076392 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f25f008df70 0x7f25f009f2d0 secure :-1 s=THROTTLE_DONE pgs=1100 cs=0 l=1 rx=0x7f25e0008790 tx=0x7f25e000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:29.794 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:29.783+0000 7f25f5f3f700 1 -- 172.21.15.42:0/1115076392 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f25f008df70 msgr2=0x7f25f009f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:45:29.794 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:29.783+0000 7f25f5f3f700 1 --2- 172.21.15.42:0/1115076392 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f25f008df70 0x7f25f009f2d0 secure :-1 s=READY pgs=1100 cs=0 l=1 rx=0x7f25e0008790 tx=0x7f25e000fbf0).stop 2019-12-05T05:45:29.794 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:29.783+0000 7f25eeffd700 1 -- 172.21.15.42:0/1115076392 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f25e0011850 con 0x7f25f008df70 2019-12-05T05:45:29.795 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:29.783+0000 7f25f5f3f700 1 -- 172.21.15.42:0/1115076392 shutdown_connections 2019-12-05T05:45:29.795 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:29.783+0000 7f25f5f3f700 1 --2- 172.21.15.42:0/1115076392 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f25f008df70 0x7f25f009f2d0 unknown :-1 s=CLOSED pgs=1100 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:45:29.795 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:29.783+0000 7f25f5f3f700 1 -- 172.21.15.42:0/1115076392 shutdown_connections 2019-12-05T05:45:29.795 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:29.783+0000 7f25f5f3f700 1 -- 172.21.15.42:0/1115076392 wait complete. 2019-12-05T05:45:29.795 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:29.783+0000 7f25f5f3f700 1 -- 172.21.15.42:0/1115076392 >> 172.21.15.42:0/1115076392 conn(0x7f25f00aabd0 msgr2=0x7f25f00a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:45:29.795 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:29.783+0000 7f25f5f3f700 1 Processor -- start 2019-12-05T05:45:29.795 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:29.783+0000 7f25f5f3f700 1 -- start start 2019-12-05T05:45:29.798 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:29.787+0000 7f25f5f3f700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f25f0124af0 0x7f25f0117f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:45:29.798 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:29.787+0000 7f25f5f3f700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f25f0118460 0x7f25f011af30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:45:29.798 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:29.787+0000 7f25f5f3f700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25f01175d0 con 0x7f25f011b480 2019-12-05T05:45:29.798 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:29.787+0000 7f25f5f3f700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25f011acd0 con 0x7f25f0118460 2019-12-05T05:45:29.799 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:29.787+0000 7f25f5f3f700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25f0116c60 con 0x7f25f0124af0 2019-12-05T05:45:29.799 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:29.787+0000 7f25ef7fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f25f0118460 0x7f25f011af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:45:29.799 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:29.787+0000 7f25ef7fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f25f0118460 0x7f25f011af30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:40534/0 (socket says 172.21.15.42:40534) 2019-12-05T05:45:29.799 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:29.787+0000 7f25ef7fe700 1 -- 172.21.15.42:0/2491076570 learned_addr learned my addr 172.21.15.42:0/2491076570 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:45:29.799 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:29.787+0000 7f25effff700 1 --2- 172.21.15.42:0/2491076570 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f25f0124af0 0x7f25f0117f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:45:29.799 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:29.787+0000 7f25ef7fe700 1 -- 172.21.15.42:0/2491076570 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f25f0124af0 msgr2=0x7f25f0117f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:45:29.800 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:29.787+0000 7f25ef7fe700 1 --2- 172.21.15.42:0/2491076570 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f25f0124af0 0x7f25f0117f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:45:29.800 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:29.787+0000 7f25ef7fe700 1 -- 172.21.15.42:0/2491076570 >> v1:172.21.15.40:6789/0 conn(0x7f25f011b480 legacy=0x7f25f008eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:45:29.800 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:29.787+0000 7f25ecff9700 1 -- 172.21.15.42:0/2491076570 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3143490698 0 0) 0x7f25f01175d0 con 0x7f25f011b480 2019-12-05T05:45:29.800 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:29.787+0000 7f25ef7fe700 1 -- 172.21.15.42:0/2491076570 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25f011b8b0 con 0x7f25f0118460 2019-12-05T05:45:29.800 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:29.787+0000 7f25ef7fe700 1 --2- 172.21.15.42:0/2491076570 >> v2:172.21.15.42:3300/0 conn(0x7f25f0118460 0x7f25f011af30 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f25f011aad0 tx=0x7f25e0008ea0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:29.800 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:29.787+0000 7f25ef7fe700 1 --2- 172.21.15.42:0/2491076570 >> v2:172.21.15.42:3300/0 conn(0x7f25f0118460 0x7f25f011af30 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f25f011aad0 tx=0x7f25e0008ea0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:29.800 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:29.787+0000 7f25ef7fe700 1 --2- 172.21.15.42:0/2491076570 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f25f0118460 0x7f25f011af30 secure :-1 s=READY pgs=1426 cs=0 l=1 rx=0x7f25f011aad0 tx=0x7f25e0008ea0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:45:29.801 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:29.787+0000 7f25ef7fe700 1 --2- 172.21.15.42:0/2491076570 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f25f0118460 0x7f25f011af30 secure :-1 s=THROTTLE_DONE pgs=1426 cs=0 l=1 rx=0x7f25f011aad0 tx=0x7f25e0008ea0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:29.801 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:29.787+0000 7f25ecff9700 1 -- 172.21.15.42:0/2491076570 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f25e000f360 con 0x7f25f0118460 2019-12-05T05:45:29.801 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:29.787+0000 7f25ef7fe700 1 --2- 172.21.15.42:0/2491076570 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f25f0118460 0x7f25f011af30 secure :-1 s=THROTTLE_DONE pgs=1426 cs=0 l=1 rx=0x7f25f011aad0 tx=0x7f25e0008ea0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:29.801 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:29.787+0000 7f25ef7fe700 1 --2- 172.21.15.42:0/2491076570 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f25f0118460 0x7f25f011af30 secure :-1 s=THROTTLE_DONE pgs=1426 cs=0 l=1 rx=0x7f25f011aad0 tx=0x7f25e0008ea0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:29.801 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:29.787+0000 7f25ecff9700 1 -- 172.21.15.42:0/2491076570 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25e0011470 con 0x7f25f0118460 2019-12-05T05:45:29.803 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:29.787+0000 7f25ecff9700 1 -- 172.21.15.42:0/2491076570 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f25e0007340 con 0x7f25f0118460 2019-12-05T05:45:29.803 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:29.787+0000 7f25f5f3f700 1 -- 172.21.15.42:0/2491076570 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25f0117a90 con 0x7f25f0118460 2019-12-05T05:45:29.803 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:29.787+0000 7f25f5f3f700 1 -- 172.21.15.42:0/2491076570 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25f008f580 con 0x7f25f0118460 2019-12-05T05:45:29.804 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:29.787+0000 7f25ef7fe700 1 --2- 172.21.15.42:0/2491076570 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f25f0118460 0x7f25f011af30 secure :-1 s=THROTTLE_DONE pgs=1426 cs=0 l=1 rx=0x7f25f011aad0 tx=0x7f25e0008ea0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:29.804 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:29.791+0000 7f25ecff9700 1 -- 172.21.15.42:0/2491076570 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f25e0005540 con 0x7f25f0118460 2019-12-05T05:45:29.804 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:29.791+0000 7f25ef7fe700 1 --2- 172.21.15.42:0/2491076570 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f25f0118460 0x7f25f011af30 secure :-1 s=THROTTLE_DONE pgs=1426 cs=0 l=1 rx=0x7f25f011aad0 tx=0x7f25e0008ea0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:29.804 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:29.791+0000 7f25ecff9700 1 --2- 172.21.15.42:0/2491076570 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f25c803ac30 0x7f25c803d140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:45:29.804 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:29.791+0000 7f25ecff9700 1 -- 172.21.15.42:0/2491076570 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 8439+0+0 (secure 0 0 0) 0x7f25e004bc50 con 0x7f25f0118460 2019-12-05T05:45:29.805 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:29.791+0000 7f25effff700 1 --2- 172.21.15.42:0/2491076570 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f25c803ac30 0x7f25c803d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:45:29.805 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:29.791+0000 7f25effff700 1 --2- 172.21.15.42:0/2491076570 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f25c803ac30 0x7f25c803d140 crc :-1 s=READY pgs=613 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:45:29.805 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:29.795+0000 7f25f5f3f700 1 -- 172.21.15.42:0/2491076570 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25f005d100 con 0x7f25f0118460 2019-12-05T05:45:29.808 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:29.803+0000 7f25ef7fe700 1 --2- 172.21.15.42:0/2491076570 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f25f0118460 0x7f25f011af30 secure :-1 s=THROTTLE_DONE pgs=1426 cs=0 l=1 rx=0x7f25f011aad0 tx=0x7f25e0008ea0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:29.808 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:29.803+0000 7f25ecff9700 1 -- 172.21.15.42:0/2491076570 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f25e0030930 con 0x7f25f0118460 2019-12-05T05:45:29.969 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:29.963+0000 7f25f5f3f700 1 -- 172.21.15.42:0/2491076570 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f25f005b8b0 con 0x7f25f0118460 2019-12-05T05:45:29.970 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:29.963+0000 7f25ef7fe700 1 --2- 172.21.15.42:0/2491076570 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f25f0118460 0x7f25f011af30 secure :-1 s=THROTTLE_DONE pgs=1426 cs=0 l=1 rx=0x7f25f011aad0 tx=0x7f25e0008ea0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:29.971 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:29.963+0000 7f25ecff9700 1 -- 172.21.15.42:0/2491076570 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 173 v173) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f25e0012bb0 con 0x7f25f0118460 2019-12-05T05:45:29.971 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 173 2019-12-05T05:45:29.971 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:45:29.971 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":173,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6077,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6835/3757738536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3757738536},{"type":"v1","addr":"172.21.15.42:6835","nonce":3757738536}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":173}],"filesystems":[{"mdsmap":{"epoch":172,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:44:49.006174+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7884},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7884":{"gid":7884,"name":"b","rank":0,"incarnation":169,"state":"up:active","state_seq":11,"addr":"172.21.15.40:6833/1127133319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1127133319},{"type":"v1","addr":"172.21.15.40:6833","nonce":1127133319}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:45:29.973 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:29.967+0000 7f25f5f3f700 1 -- 172.21.15.42:0/2491076570 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f25c803ac30 msgr2=0x7f25c803d140 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:45:29.973 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:29.967+0000 7f25f5f3f700 1 --2- 172.21.15.42:0/2491076570 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f25c803ac30 0x7f25c803d140 crc :-1 s=READY pgs=613 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:45:29.973 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:29.967+0000 7f25f5f3f700 1 -- 172.21.15.42:0/2491076570 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f25f0118460 msgr2=0x7f25f011af30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:45:29.973 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:29.967+0000 7f25f5f3f700 1 --2- 172.21.15.42:0/2491076570 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f25f0118460 0x7f25f011af30 secure :-1 s=READY pgs=1426 cs=0 l=1 rx=0x7f25f011aad0 tx=0x7f25e0008ea0).stop 2019-12-05T05:45:29.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:29.967+0000 7f25f5f3f700 1 -- 172.21.15.42:0/2491076570 shutdown_connections 2019-12-05T05:45:29.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:29.967+0000 7f25f5f3f700 1 --2- 172.21.15.42:0/2491076570 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f25c803ac30 0x7f25c803d140 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:45:29.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:29.967+0000 7f25f5f3f700 1 --2- 172.21.15.42:0/2491076570 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f25f0124af0 0x7f25f0117f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:45:29.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:29.967+0000 7f25f5f3f700 1 --2- 172.21.15.42:0/2491076570 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f25f0118460 0x7f25f011af30 unknown :-1 s=CLOSED pgs=1426 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:45:29.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:29.967+0000 7f25f5f3f700 1 -- 172.21.15.42:0/2491076570 shutdown_connections 2019-12-05T05:45:29.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:29.967+0000 7f25f5f3f700 1 -- 172.21.15.42:0/2491076570 wait complete. 2019-12-05T05:45:29.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:29.967+0000 7f25f5f3f700 1 -- 172.21.15.42:0/2491076570 >> 172.21.15.42:0/2491076570 conn(0x7f25f00a6ba0 msgr2=0x7f25f00a9760 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:45:29.984 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T05:45:31.985 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:45:32.096 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:32.083+0000 7fcccb0e2700 1 Processor -- start 2019-12-05T05:45:32.096 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:32.083+0000 7fcccb0e2700 1 -- start start 2019-12-05T05:45:32.096 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:32.083+0000 7fcccb0e2700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fccc40a7740 0x7fccc408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:45:32.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:32.083+0000 7fcccb0e2700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fccc409fae0 0x7fccc407eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:45:32.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:32.083+0000 7fcccb0e2700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccc40a0710 con 0x7fccc407e720 2019-12-05T05:45:32.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:32.083+0000 7fcccb0e2700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fccc40a0b00 con 0x7fccc409fae0 2019-12-05T05:45:32.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:32.083+0000 7fcccb0e2700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccc408e5b0 con 0x7fccc40a7740 2019-12-05T05:45:32.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:32.087+0000 7fccc9e80700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fccc409fae0 0x7fccc407eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:45:32.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:32.087+0000 7fccc9e80700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fccc409fae0 0x7fccc407eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.42:56590/0 (socket says 172.21.15.42:56590) 2019-12-05T05:45:32.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:32.087+0000 7fccc967f700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fccc40a7740 0x7fccc408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:45:32.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:32.087+0000 7fccc9e80700 1 -- 172.21.15.42:0/4126943081 learned_addr learned my addr 172.21.15.42:0/4126943081 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:45:32.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:32.087+0000 7fccc8e7e700 1 --1- 172.21.15.42:0/4126943081 >> v1:172.21.15.40:6789/0 conn(0x7fccc407e720 0x7fccc409f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:50016/0 (socket says 172.21.15.42:50016) 2019-12-05T05:45:32.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:32.087+0000 7fccc967f700 1 -- 172.21.15.42:0/4126943081 >> v2:172.21.15.40:3300/0 conn(0x7fccc409fae0 msgr2=0x7fccc407eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:45:32.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:32.087+0000 7fccc967f700 1 --2- 172.21.15.42:0/4126943081 >> v2:172.21.15.40:3300/0 conn(0x7fccc409fae0 0x7fccc407eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:45:32.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:32.087+0000 7fccc967f700 1 -- 172.21.15.42:0/4126943081 >> v1:172.21.15.40:6789/0 conn(0x7fccc407e720 legacy=0x7fccc409f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:45:32.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:32.087+0000 7fccc967f700 1 -- 172.21.15.42:0/4126943081 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccc407f0a0 con 0x7fccc40a7740 2019-12-05T05:45:32.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:32.087+0000 7fccc9e80700 1 --2- 172.21.15.42:0/4126943081 >> v2:172.21.15.40:3300/0 conn(0x7fccc409fae0 0x7fccc407eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:45:32.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:32.087+0000 7fccc967f700 1 --2- 172.21.15.42:0/4126943081 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fccc40a7740 0x7fccc408df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fccc0008790 tx=0x7fccc000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:32.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:32.087+0000 7fccc967f700 1 --2- 172.21.15.42:0/4126943081 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fccc40a7740 0x7fccc408df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fccc0008790 tx=0x7fccc000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:32.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:32.087+0000 7fccc967f700 1 --2- 172.21.15.42:0/4126943081 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fccc40a7740 0x7fccc408df80 secure :-1 s=READY pgs=310 cs=0 l=1 rx=0x7fccc0008790 tx=0x7fccc000fc70).ready entity=mon.2 client_cookie=3e0f0973a22fda7d server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:45:32.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:32.087+0000 7fccc967f700 1 --2- 172.21.15.42:0/4126943081 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fccc40a7740 0x7fccc408df80 secure :-1 s=THROTTLE_DONE pgs=310 cs=0 l=1 rx=0x7fccc0008790 tx=0x7fccc000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:32.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:32.087+0000 7fccc967f700 1 --2- 172.21.15.42:0/4126943081 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fccc40a7740 0x7fccc408df80 secure :-1 s=THROTTLE_DONE pgs=310 cs=0 l=1 rx=0x7fccc0008790 tx=0x7fccc000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:32.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:32.087+0000 7fccbbfff700 1 -- 172.21.15.42:0/4126943081 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fccc0008bb0 con 0x7fccc40a7740 2019-12-05T05:45:32.100 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:32.087+0000 7fccc967f700 1 --2- 172.21.15.42:0/4126943081 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fccc40a7740 0x7fccc408df80 secure :-1 s=THROTTLE_DONE pgs=310 cs=0 l=1 rx=0x7fccc0008790 tx=0x7fccc000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:32.100 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:32.087+0000 7fccbbfff700 1 -- 172.21.15.42:0/4126943081 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccc00087d0 con 0x7fccc40a7740 2019-12-05T05:45:32.100 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:32.087+0000 7fccbbfff700 1 -- 172.21.15.42:0/4126943081 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fccc0011860 con 0x7fccc40a7740 2019-12-05T05:45:32.100 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:32.087+0000 7fcccb0e2700 1 -- 172.21.15.42:0/4126943081 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fccc40a7740 msgr2=0x7fccc408df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:45:32.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:32.087+0000 7fcccb0e2700 1 --2- 172.21.15.42:0/4126943081 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fccc40a7740 0x7fccc408df80 secure :-1 s=READY pgs=310 cs=0 l=1 rx=0x7fccc0008790 tx=0x7fccc000fc70).stop 2019-12-05T05:45:32.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:32.087+0000 7fcccb0e2700 1 -- 172.21.15.42:0/4126943081 shutdown_connections 2019-12-05T05:45:32.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:32.087+0000 7fcccb0e2700 1 --2- 172.21.15.42:0/4126943081 >> v2:172.21.15.40:3300/0 conn(0x7fccc409fae0 0x7fccc407eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:45:32.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:32.087+0000 7fcccb0e2700 1 --2- 172.21.15.42:0/4126943081 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fccc40a7740 0x7fccc408df80 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:45:32.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:32.087+0000 7fcccb0e2700 1 -- 172.21.15.42:0/4126943081 shutdown_connections 2019-12-05T05:45:32.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:32.087+0000 7fcccb0e2700 1 -- 172.21.15.42:0/4126943081 wait complete. 2019-12-05T05:45:32.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:32.087+0000 7fcccb0e2700 1 -- 172.21.15.42:0/4126943081 >> 172.21.15.42:0/4126943081 conn(0x7fccc40aabe0 msgr2=0x7fccc40a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:45:32.106 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:32.091+0000 7fcccb0e2700 1 Processor -- start 2019-12-05T05:45:32.106 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:32.091+0000 7fcccb0e2700 1 -- start start 2019-12-05T05:45:32.106 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:32.091+0000 7fcccb0e2700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fccc4121aa0 0x7fccc407afe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:45:32.106 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:32.091+0000 7fcccb0e2700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fccc407b530 0x7fccc411a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:45:32.106 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:32.091+0000 7fcccb0e2700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccc4117a20 con 0x7fccc4123800 2019-12-05T05:45:32.107 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:32.091+0000 7fcccb0e2700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fccc4117da0 con 0x7fccc407b530 2019-12-05T05:45:32.107 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:32.091+0000 7fcccb0e2700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccc4118040 con 0x7fccc4121aa0 2019-12-05T05:45:32.107 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:32.095+0000 7fccc9e80700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fccc4123800 0x7fccc411af10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:40586/0 (socket says 172.21.15.42:40586) 2019-12-05T05:45:32.107 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:32.095+0000 7fccc9e80700 1 -- 172.21.15.42:0/2288118661 learned_addr learned my addr 172.21.15.42:0/2288118661 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:45:32.107 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:32.095+0000 7fccc967f700 1 --2- 172.21.15.42:0/2288118661 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fccc4121aa0 0x7fccc407afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:45:32.107 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:32.095+0000 7fccc8e7e700 1 --2- 172.21.15.42:0/2288118661 >> v2:172.21.15.40:3300/0 conn(0x7fccc407b530 0x7fccc411a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:45:32.107 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:32.095+0000 7fccb9ffb700 1 -- 172.21.15.42:0/2288118661 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 711701578 0 0) 0x7fccc4117a20 con 0x7fccc4123800 2019-12-05T05:45:32.108 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:32.095+0000 7fccb9ffb700 1 -- 172.21.15.42:0/2288118661 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcca40030b0 con 0x7fccc4123800 2019-12-05T05:45:32.108 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:32.095+0000 7fccc967f700 1 -- 172.21.15.42:0/2288118661 >> v2:172.21.15.40:3300/0 conn(0x7fccc407b530 msgr2=0x7fccc411a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:45:32.108 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:32.095+0000 7fccc967f700 1 --2- 172.21.15.42:0/2288118661 >> v2:172.21.15.40:3300/0 conn(0x7fccc407b530 0x7fccc411a9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:45:32.108 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:32.095+0000 7fccc967f700 1 -- 172.21.15.42:0/2288118661 >> v1:172.21.15.42:6789/0 conn(0x7fccc4123800 legacy=0x7fccc411af10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:45:32.108 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:32.095+0000 7fccc967f700 1 -- 172.21.15.42:0/2288118661 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccc41171a0 con 0x7fccc4121aa0 2019-12-05T05:45:32.108 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:32.095+0000 7fccc8e7e700 1 --2- 172.21.15.42:0/2288118661 >> v2:172.21.15.40:3300/0 conn(0x7fccc407b530 0x7fccc411a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:45:32.108 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:32.095+0000 7fccc967f700 1 --2- 172.21.15.42:0/2288118661 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fccc4121aa0 0x7fccc407afe0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fccc0008790 tx=0x7fccc0011e80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:32.109 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:32.095+0000 7fccc967f700 1 --2- 172.21.15.42:0/2288118661 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fccc4121aa0 0x7fccc407afe0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fccc0008790 tx=0x7fccc0011e80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:32.109 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:32.095+0000 7fccc967f700 1 --2- 172.21.15.42:0/2288118661 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fccc4121aa0 0x7fccc407afe0 secure :-1 s=READY pgs=311 cs=0 l=1 rx=0x7fccc0008790 tx=0x7fccc0011e80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:45:32.109 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:32.095+0000 7fccc967f700 1 --2- 172.21.15.42:0/2288118661 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fccc4121aa0 0x7fccc407afe0 secure :-1 s=THROTTLE_DONE pgs=311 cs=0 l=1 rx=0x7fccc0008790 tx=0x7fccc0011e80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:32.109 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:32.095+0000 7fccc967f700 1 --2- 172.21.15.42:0/2288118661 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fccc4121aa0 0x7fccc407afe0 secure :-1 s=THROTTLE_DONE pgs=311 cs=0 l=1 rx=0x7fccc0008790 tx=0x7fccc0011e80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:32.109 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:32.095+0000 7fccb9ffb700 1 -- 172.21.15.42:0/2288118661 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fccc000f280 con 0x7fccc4121aa0 2019-12-05T05:45:32.109 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:32.095+0000 7fccc967f700 1 --2- 172.21.15.42:0/2288118661 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fccc4121aa0 0x7fccc407afe0 secure :-1 s=THROTTLE_DONE pgs=311 cs=0 l=1 rx=0x7fccc0008790 tx=0x7fccc0011e80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:32.109 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:32.095+0000 7fccb9ffb700 1 -- 172.21.15.42:0/2288118661 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccc0008db0 con 0x7fccc4121aa0 2019-12-05T05:45:32.110 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:32.095+0000 7fccb9ffb700 1 -- 172.21.15.42:0/2288118661 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fccc0005540 con 0x7fccc4121aa0 2019-12-05T05:45:32.110 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:32.095+0000 7fcccb0e2700 1 -- 172.21.15.42:0/2288118661 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccc4137aa0 con 0x7fccc4121aa0 2019-12-05T05:45:32.110 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:32.095+0000 7fcccb0e2700 1 -- 172.21.15.42:0/2288118661 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccc4137fd0 con 0x7fccc4121aa0 2019-12-05T05:45:32.110 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:32.095+0000 7fccc967f700 1 --2- 172.21.15.42:0/2288118661 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fccc4121aa0 0x7fccc407afe0 secure :-1 s=THROTTLE_DONE pgs=311 cs=0 l=1 rx=0x7fccc0008790 tx=0x7fccc0011e80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:32.110 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:32.099+0000 7fccb9ffb700 1 -- 172.21.15.42:0/2288118661 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fccc0005810 con 0x7fccc4121aa0 2019-12-05T05:45:32.110 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:32.099+0000 7fccc967f700 1 --2- 172.21.15.42:0/2288118661 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fccc4121aa0 0x7fccc407afe0 secure :-1 s=THROTTLE_DONE pgs=311 cs=0 l=1 rx=0x7fccc0008790 tx=0x7fccc0011e80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:32.110 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:32.099+0000 7fccb9ffb700 1 --2- 172.21.15.42:0/2288118661 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fcca403b2c0 0x7fcca403d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:45:32.111 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:32.099+0000 7fccb9ffb700 1 -- 172.21.15.42:0/2288118661 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 8439+0+0 (secure 0 0 0) 0x7fccc004b4b0 con 0x7fccc4121aa0 2019-12-05T05:45:32.111 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:32.099+0000 7fccc8e7e700 1 --2- 172.21.15.42:0/2288118661 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fcca403b2c0 0x7fcca403d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:45:32.111 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:32.099+0000 7fccc8e7e700 1 --2- 172.21.15.42:0/2288118661 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fcca403b2c0 0x7fcca403d7d0 crc :-1 s=READY pgs=614 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:45:32.111 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:32.099+0000 7fcccb0e2700 1 -- 172.21.15.42:0/2288118661 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccc405d110 con 0x7fccc4121aa0 2019-12-05T05:45:32.115 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:32.107+0000 7fccc967f700 1 --2- 172.21.15.42:0/2288118661 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fccc4121aa0 0x7fccc407afe0 secure :-1 s=THROTTLE_DONE pgs=311 cs=0 l=1 rx=0x7fccc0008790 tx=0x7fccc0011e80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:32.115 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:32.107+0000 7fccb9ffb700 1 -- 172.21.15.42:0/2288118661 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fccc0032530 con 0x7fccc4121aa0 2019-12-05T05:45:32.283 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:32.275+0000 7fcccb0e2700 1 -- 172.21.15.42:0/2288118661 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fccc405b8c0 con 0x7fccc4121aa0 2019-12-05T05:45:32.284 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:32.279+0000 7fccc967f700 1 --2- 172.21.15.42:0/2288118661 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fccc4121aa0 0x7fccc407afe0 secure :-1 s=THROTTLE_DONE pgs=311 cs=0 l=1 rx=0x7fccc0008790 tx=0x7fccc0011e80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:32.284 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:32.279+0000 7fccb9ffb700 1 -- 172.21.15.42:0/2288118661 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 173 v173) v1 ==== 95+0+2112 (secure 0 0 0) 0x7fccc0032790 con 0x7fccc4121aa0 2019-12-05T05:45:32.286 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 173 2019-12-05T05:45:32.286 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:45:32.287 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":173,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6077,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6835/3757738536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3757738536},{"type":"v1","addr":"172.21.15.42:6835","nonce":3757738536}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":173}],"filesystems":[{"mdsmap":{"epoch":172,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:44:49.006174+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7884},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7884":{"gid":7884,"name":"b","rank":0,"incarnation":169,"state":"up:active","state_seq":11,"addr":"172.21.15.40:6833/1127133319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1127133319},{"type":"v1","addr":"172.21.15.40:6833","nonce":1127133319}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:45:32.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:32.283+0000 7fcccb0e2700 1 -- 172.21.15.42:0/2288118661 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fcca403b2c0 msgr2=0x7fcca403d7d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:45:32.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:32.283+0000 7fcccb0e2700 1 --2- 172.21.15.42:0/2288118661 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fcca403b2c0 0x7fcca403d7d0 crc :-1 s=READY pgs=614 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:45:32.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:32.283+0000 7fcccb0e2700 1 -- 172.21.15.42:0/2288118661 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fccc4121aa0 msgr2=0x7fccc407afe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:45:32.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:32.283+0000 7fcccb0e2700 1 --2- 172.21.15.42:0/2288118661 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fccc4121aa0 0x7fccc407afe0 secure :-1 s=READY pgs=311 cs=0 l=1 rx=0x7fccc0008790 tx=0x7fccc0011e80).stop 2019-12-05T05:45:32.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:32.283+0000 7fcccb0e2700 1 -- 172.21.15.42:0/2288118661 shutdown_connections 2019-12-05T05:45:32.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:32.283+0000 7fcccb0e2700 1 --2- 172.21.15.42:0/2288118661 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fcca403b2c0 0x7fcca403d7d0 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:45:32.291 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:32.283+0000 7fcccb0e2700 1 --2- 172.21.15.42:0/2288118661 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fccc4121aa0 0x7fccc407afe0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:45:32.291 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:32.283+0000 7fcccb0e2700 1 --2- 172.21.15.42:0/2288118661 >> v2:172.21.15.40:3300/0 conn(0x7fccc407b530 0x7fccc411a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:45:32.291 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:32.283+0000 7fcccb0e2700 1 -- 172.21.15.42:0/2288118661 shutdown_connections 2019-12-05T05:45:32.291 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:32.283+0000 7fcccb0e2700 1 -- 172.21.15.42:0/2288118661 wait complete. 2019-12-05T05:45:32.291 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:32.283+0000 7fcccb0e2700 1 -- 172.21.15.42:0/2288118661 >> 172.21.15.42:0/2288118661 conn(0x7fccc40a82a0 msgr2=0x7fccc409f120 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:45:32.299 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T05:45:34.300 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:45:34.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:34.395+0000 7f31dd846700 1 Processor -- start 2019-12-05T05:45:34.412 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:34.395+0000 7f31dd846700 1 -- start start 2019-12-05T05:45:34.412 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:34.395+0000 7f31dd846700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f31d80a7730 0x7f31d808df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:45:34.412 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:34.395+0000 7f31dd846700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f31d807e710 0x7f31d807eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:45:34.413 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:34.395+0000 7f31dd846700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31d80a0700 con 0x7f31d809f2d0 2019-12-05T05:45:34.413 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:34.395+0000 7f31dd846700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31d80a0af0 con 0x7f31d807e710 2019-12-05T05:45:34.413 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:34.395+0000 7f31dd846700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31d808e5a0 con 0x7f31d80a7730 2019-12-05T05:45:34.413 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:34.395+0000 7f31d7fff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f31d809f2d0 0x7f31d809f700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:40596/0 (socket says 172.21.15.42:40596) 2019-12-05T05:45:34.413 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:34.395+0000 7f31d7fff700 1 -- 172.21.15.42:0/3800500899 learned_addr learned my addr 172.21.15.42:0/3800500899 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:45:34.413 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:34.395+0000 7f31d6ffd700 1 --2- 172.21.15.42:0/3800500899 >> v2:172.21.15.40:3300/0 conn(0x7f31d807e710 0x7f31d807eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:45:34.413 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:34.395+0000 7f31d77fe700 1 --2- 172.21.15.42:0/3800500899 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f31d80a7730 0x7f31d808df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:45:34.414 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:34.395+0000 7f31d67fc700 1 -- 172.21.15.42:0/3800500899 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1085126549 0 0) 0x7f31d80a0700 con 0x7f31d809f2d0 2019-12-05T05:45:34.414 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:34.395+0000 7f31d67fc700 1 -- 172.21.15.42:0/3800500899 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31c00030b0 con 0x7f31d809f2d0 2019-12-05T05:45:34.414 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:34.395+0000 7f31d77fe700 1 -- 172.21.15.42:0/3800500899 >> v2:172.21.15.40:3300/0 conn(0x7f31d807e710 msgr2=0x7f31d807eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:45:34.414 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:34.395+0000 7f31d77fe700 1 --2- 172.21.15.42:0/3800500899 >> v2:172.21.15.40:3300/0 conn(0x7f31d807e710 0x7f31d807eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:45:34.414 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:34.395+0000 7f31d77fe700 1 -- 172.21.15.42:0/3800500899 >> v1:172.21.15.42:6789/0 conn(0x7f31d809f2d0 legacy=0x7f31d809f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:45:34.414 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:34.395+0000 7f31d77fe700 1 -- 172.21.15.42:0/3800500899 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31d807f090 con 0x7f31d80a7730 2019-12-05T05:45:34.414 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:34.395+0000 7f31d77fe700 1 --2- 172.21.15.42:0/3800500899 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f31d80a7730 0x7f31d808df70 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f31c8008790 tx=0x7f31c800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:34.415 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:34.395+0000 7f31d77fe700 1 --2- 172.21.15.42:0/3800500899 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f31d80a7730 0x7f31d808df70 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f31c8008790 tx=0x7f31c800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:34.415 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:34.395+0000 7f31d77fe700 1 --2- 172.21.15.42:0/3800500899 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f31d80a7730 0x7f31d808df70 secure :-1 s=READY pgs=312 cs=0 l=1 rx=0x7f31c8008790 tx=0x7f31c800fc70).ready entity=mon.2 client_cookie=337b5c2681d10ebc server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:45:34.415 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:34.395+0000 7f31d77fe700 1 --2- 172.21.15.42:0/3800500899 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f31d80a7730 0x7f31d808df70 secure :-1 s=THROTTLE_DONE pgs=312 cs=0 l=1 rx=0x7f31c8008790 tx=0x7f31c800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:34.415 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:34.395+0000 7f31d67fc700 1 -- 172.21.15.42:0/3800500899 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f31c80112f0 con 0x7f31d80a7730 2019-12-05T05:45:34.415 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:34.395+0000 7f31d77fe700 1 --2- 172.21.15.42:0/3800500899 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f31d80a7730 0x7f31d808df70 secure :-1 s=THROTTLE_DONE pgs=312 cs=0 l=1 rx=0x7f31c8008790 tx=0x7f31c800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:34.415 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:34.399+0000 7f31d67fc700 1 -- 172.21.15.42:0/3800500899 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31c8008ad0 con 0x7f31d80a7730 2019-12-05T05:45:34.415 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:34.399+0000 7f31d77fe700 1 --2- 172.21.15.42:0/3800500899 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f31d80a7730 0x7f31d808df70 secure :-1 s=THROTTLE_DONE pgs=312 cs=0 l=1 rx=0x7f31c8008790 tx=0x7f31c800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:34.416 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:34.399+0000 7f31dd846700 1 -- 172.21.15.42:0/3800500899 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f31d80a7730 msgr2=0x7f31d808df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:45:34.416 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:34.399+0000 7f31d67fc700 1 -- 172.21.15.42:0/3800500899 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f31c80118d0 con 0x7f31d80a7730 2019-12-05T05:45:34.416 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:34.399+0000 7f31dd846700 1 --2- 172.21.15.42:0/3800500899 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f31d80a7730 0x7f31d808df70 secure :-1 s=READY pgs=312 cs=0 l=1 rx=0x7f31c8008790 tx=0x7f31c800fc70).stop 2019-12-05T05:45:34.416 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:34.399+0000 7f31dd846700 1 -- 172.21.15.42:0/3800500899 shutdown_connections 2019-12-05T05:45:34.417 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:34.399+0000 7f31dd846700 1 --2- 172.21.15.42:0/3800500899 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f31d80a7730 0x7f31d808df70 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:45:34.417 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:34.399+0000 7f31dd846700 1 --2- 172.21.15.42:0/3800500899 >> v2:172.21.15.40:3300/0 conn(0x7f31d807e710 0x7f31d807eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:45:34.417 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:34.399+0000 7f31dd846700 1 -- 172.21.15.42:0/3800500899 shutdown_connections 2019-12-05T05:45:34.417 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:34.399+0000 7f31dd846700 1 -- 172.21.15.42:0/3800500899 wait complete. 2019-12-05T05:45:34.417 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:34.399+0000 7f31dd846700 1 -- 172.21.15.42:0/3800500899 >> 172.21.15.42:0/3800500899 conn(0x7f31d80aabd0 msgr2=0x7f31d80a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:45:34.421 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:34.403+0000 7f31dd846700 1 Processor -- start 2019-12-05T05:45:34.421 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:34.403+0000 7f31dd846700 1 -- start start 2019-12-05T05:45:34.421 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:34.403+0000 7f31dd846700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f31d811b690 0x7f31d811ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:45:34.422 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:34.403+0000 7f31dd846700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f31d81161b0 0x7f31d8117f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:45:34.422 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:34.403+0000 7f31dd846700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f31d808eb60 0x7f31d808ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:45:34.422 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:34.403+0000 7f31dd846700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31d811b150 con 0x7f31d81161b0 2019-12-05T05:45:34.422 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:34.403+0000 7f31dd846700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31d811b3f0 con 0x7f31d811b690 2019-12-05T05:45:34.422 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:34.403+0000 7f31dd846700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31d8117c70 con 0x7f31d808eb60 2019-12-05T05:45:34.422 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:34.403+0000 7f31d77fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f31d811b690 0x7f31d811ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:45:34.422 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:34.403+0000 7f31d77fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f31d811b690 0x7f31d811ab20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:40560/0 (socket says 172.21.15.42:40560) 2019-12-05T05:45:34.423 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:34.403+0000 7f31d77fe700 1 -- 172.21.15.42:0/4251480516 learned_addr learned my addr 172.21.15.42:0/4251480516 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:45:34.423 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:34.403+0000 7f31d6ffd700 1 --2- 172.21.15.42:0/4251480516 >> v2:172.21.15.40:3300/0 conn(0x7f31d81161b0 0x7f31d8117f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:45:34.423 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:34.403+0000 7f31d7fff700 1 --2- 172.21.15.42:0/4251480516 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f31d808eb60 0x7f31d808ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:45:34.423 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:34.403+0000 7f31d77fe700 1 -- 172.21.15.42:0/4251480516 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f31d808eb60 msgr2=0x7f31d808ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:45:34.423 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:34.403+0000 7f31d77fe700 1 --2- 172.21.15.42:0/4251480516 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f31d808eb60 0x7f31d808ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:45:34.423 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:34.403+0000 7f31d77fe700 1 -- 172.21.15.42:0/4251480516 >> v2:172.21.15.40:3300/0 conn(0x7f31d81161b0 msgr2=0x7f31d8117f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:45:34.423 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:34.403+0000 7f31d77fe700 1 --2- 172.21.15.42:0/4251480516 >> v2:172.21.15.40:3300/0 conn(0x7f31d81161b0 0x7f31d8117f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:45:34.424 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:34.403+0000 7f31d77fe700 1 -- 172.21.15.42:0/4251480516 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31d811a0e0 con 0x7f31d811b690 2019-12-05T05:45:34.424 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:34.403+0000 7f31d77fe700 1 --2- 172.21.15.42:0/4251480516 >> v2:172.21.15.42:3300/0 conn(0x7f31d811b690 0x7f31d811ab20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f31d811b5b0 tx=0x7f31c8011c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:34.424 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:34.403+0000 7f31d77fe700 1 --2- 172.21.15.42:0/4251480516 >> v2:172.21.15.42:3300/0 conn(0x7f31d811b690 0x7f31d811ab20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f31d811b5b0 tx=0x7f31c8011c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:34.424 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:34.403+0000 7f31d77fe700 1 --2- 172.21.15.42:0/4251480516 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f31d811b690 0x7f31d811ab20 secure :-1 s=READY pgs=1429 cs=0 l=1 rx=0x7f31d811b5b0 tx=0x7f31c8011c30).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:45:34.424 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:34.403+0000 7f31d77fe700 1 --2- 172.21.15.42:0/4251480516 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f31d811b690 0x7f31d811ab20 secure :-1 s=THROTTLE_DONE pgs=1429 cs=0 l=1 rx=0x7f31d811b5b0 tx=0x7f31c8011c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:34.424 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:34.403+0000 7f31d77fe700 1 --2- 172.21.15.42:0/4251480516 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f31d811b690 0x7f31d811ab20 secure :-1 s=THROTTLE_DONE pgs=1429 cs=0 l=1 rx=0x7f31d811b5b0 tx=0x7f31c8011c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:34.424 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:34.403+0000 7f31d77fe700 1 --2- 172.21.15.42:0/4251480516 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f31d811b690 0x7f31d811ab20 secure :-1 s=THROTTLE_DONE pgs=1429 cs=0 l=1 rx=0x7f31d811b5b0 tx=0x7f31c8011c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:34.425 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:34.403+0000 7f31bffff700 1 -- 172.21.15.42:0/4251480516 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f31c8008ad0 con 0x7f31d811b690 2019-12-05T05:45:34.425 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:34.403+0000 7f31bffff700 1 -- 172.21.15.42:0/4251480516 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31c80114f0 con 0x7f31d811b690 2019-12-05T05:45:34.425 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:34.403+0000 7f31dd846700 1 -- 172.21.15.42:0/4251480516 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31d81165e0 con 0x7f31d811b690 2019-12-05T05:45:34.425 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:34.403+0000 7f31bffff700 1 -- 172.21.15.42:0/4251480516 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f31c8007340 con 0x7f31d811b690 2019-12-05T05:45:34.425 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:34.403+0000 7f31dd846700 1 -- 172.21.15.42:0/4251480516 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31d8118600 con 0x7f31d811b690 2019-12-05T05:45:34.425 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:34.407+0000 7f31d77fe700 1 --2- 172.21.15.42:0/4251480516 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f31d811b690 0x7f31d811ab20 secure :-1 s=THROTTLE_DONE pgs=1429 cs=0 l=1 rx=0x7f31d811b5b0 tx=0x7f31c8011c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:34.425 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:34.407+0000 7f31bffff700 1 -- 172.21.15.42:0/4251480516 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f31c8005540 con 0x7f31d811b690 2019-12-05T05:45:34.426 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:34.407+0000 7f31d77fe700 1 --2- 172.21.15.42:0/4251480516 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f31d811b690 0x7f31d811ab20 secure :-1 s=THROTTLE_DONE pgs=1429 cs=0 l=1 rx=0x7f31d811b5b0 tx=0x7f31c8011c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:34.426 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:34.407+0000 7f31bffff700 1 --2- 172.21.15.42:0/4251480516 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f31b803ad70 0x7f31b803d280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:45:34.426 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:34.407+0000 7f31bffff700 1 -- 172.21.15.42:0/4251480516 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 8439+0+0 (secure 0 0 0) 0x7f31c804b750 con 0x7f31d811b690 2019-12-05T05:45:34.426 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:34.407+0000 7f31d6ffd700 1 --2- 172.21.15.42:0/4251480516 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f31b803ad70 0x7f31b803d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:45:34.426 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:34.407+0000 7f31d6ffd700 1 --2- 172.21.15.42:0/4251480516 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f31b803ad70 0x7f31b803d280 crc :-1 s=READY pgs=615 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:45:34.427 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:34.407+0000 7f31dd846700 1 -- 172.21.15.42:0/4251480516 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31d805d100 con 0x7f31d811b690 2019-12-05T05:45:34.427 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:34.415+0000 7f31d77fe700 1 --2- 172.21.15.42:0/4251480516 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f31d811b690 0x7f31d811ab20 secure :-1 s=THROTTLE_DONE pgs=1429 cs=0 l=1 rx=0x7f31d811b5b0 tx=0x7f31c8011c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:34.427 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:34.415+0000 7f31bffff700 1 -- 172.21.15.42:0/4251480516 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f31c80335e0 con 0x7f31d811b690 2019-12-05T05:45:34.583 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:34.575+0000 7f31dd846700 1 -- 172.21.15.42:0/4251480516 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f31d805b8b0 con 0x7f31d811b690 2019-12-05T05:45:34.584 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:34.579+0000 7f31d77fe700 1 --2- 172.21.15.42:0/4251480516 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f31d811b690 0x7f31d811ab20 secure :-1 s=THROTTLE_DONE pgs=1429 cs=0 l=1 rx=0x7f31d811b5b0 tx=0x7f31c8011c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:34.584 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:34.579+0000 7f31bffff700 1 -- 172.21.15.42:0/4251480516 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 173 v173) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f31c8013d00 con 0x7f31d811b690 2019-12-05T05:45:34.584 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 173 2019-12-05T05:45:34.585 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:45:34.585 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":173,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6077,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6835/3757738536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3757738536},{"type":"v1","addr":"172.21.15.42:6835","nonce":3757738536}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":173}],"filesystems":[{"mdsmap":{"epoch":172,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:44:49.006174+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7884},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7884":{"gid":7884,"name":"b","rank":0,"incarnation":169,"state":"up:active","state_seq":11,"addr":"172.21.15.40:6833/1127133319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1127133319},{"type":"v1","addr":"172.21.15.40:6833","nonce":1127133319}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:45:34.587 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:34.579+0000 7f31dd846700 1 -- 172.21.15.42:0/4251480516 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f31b803ad70 msgr2=0x7f31b803d280 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:45:34.588 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:34.579+0000 7f31dd846700 1 --2- 172.21.15.42:0/4251480516 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f31b803ad70 0x7f31b803d280 crc :-1 s=READY pgs=615 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:45:34.588 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:34.579+0000 7f31dd846700 1 -- 172.21.15.42:0/4251480516 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f31d811b690 msgr2=0x7f31d811ab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:45:34.588 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:34.579+0000 7f31dd846700 1 --2- 172.21.15.42:0/4251480516 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f31d811b690 0x7f31d811ab20 secure :-1 s=READY pgs=1429 cs=0 l=1 rx=0x7f31d811b5b0 tx=0x7f31c8011c30).stop 2019-12-05T05:45:34.588 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:34.579+0000 7f31dd846700 1 -- 172.21.15.42:0/4251480516 shutdown_connections 2019-12-05T05:45:34.588 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:34.579+0000 7f31dd846700 1 --2- 172.21.15.42:0/4251480516 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f31b803ad70 0x7f31b803d280 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:45:34.588 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:34.579+0000 7f31dd846700 1 --2- 172.21.15.42:0/4251480516 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f31d808eb60 0x7f31d808ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:45:34.589 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:34.579+0000 7f31dd846700 1 --2- 172.21.15.42:0/4251480516 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f31d811b690 0x7f31d811ab20 unknown :-1 s=CLOSED pgs=1429 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:45:34.589 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:34.579+0000 7f31dd846700 1 --2- 172.21.15.42:0/4251480516 >> v2:172.21.15.40:3300/0 conn(0x7f31d81161b0 0x7f31d8117f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:45:34.589 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:34.579+0000 7f31dd846700 1 -- 172.21.15.42:0/4251480516 shutdown_connections 2019-12-05T05:45:34.589 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:34.579+0000 7f31dd846700 1 -- 172.21.15.42:0/4251480516 wait complete. 2019-12-05T05:45:34.589 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:34.579+0000 7f31dd846700 1 -- 172.21.15.42:0/4251480516 >> 172.21.15.42:0/4251480516 conn(0x7f31d80a8290 msgr2=0x7f31d80a9780 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:45:34.600 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T05:45:36.601 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:45:36.704 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:36.695+0000 7f19cf080700 1 Processor -- start 2019-12-05T05:45:36.705 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:36.695+0000 7f19cf080700 1 -- start start 2019-12-05T05:45:36.705 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:36.695+0000 7f19cf080700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f19c80a7740 0x7f19c808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:45:36.705 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:36.695+0000 7f19cf080700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19c80a0710 con 0x7f19c807e720 2019-12-05T05:45:36.705 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:36.695+0000 7f19cf080700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19c80a9ba0 con 0x7f19c809fae0 2019-12-05T05:45:36.705 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:36.695+0000 7f19cf080700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19c80a0b00 con 0x7f19c80a7740 2019-12-05T05:45:36.705 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:36.695+0000 7f19cd61d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f19c80a7740 0x7f19c808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:45:36.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:36.695+0000 7f19cd61d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f19c80a7740 0x7f19c808df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:40574/0 (socket says 172.21.15.42:40574) 2019-12-05T05:45:36.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:36.695+0000 7f19cde1e700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f19c809fae0 0x7f19c814efe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:40612/0 (socket says 172.21.15.42:40612) 2019-12-05T05:45:36.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:36.695+0000 7f19cd61d700 1 -- 172.21.15.42:0/647437805 learned_addr learned my addr 172.21.15.42:0/647437805 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:45:36.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:36.695+0000 7f19cd61d700 1 -- 172.21.15.42:0/647437805 >> v1:172.21.15.42:6789/0 conn(0x7f19c809fae0 legacy=0x7f19c814efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:45:36.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:36.695+0000 7f19cd61d700 1 -- 172.21.15.42:0/647437805 >> v1:172.21.15.40:6789/0 conn(0x7f19c807e720 legacy=0x7f19c809f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:45:36.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:36.695+0000 7f19cd61d700 1 -- 172.21.15.42:0/647437805 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19c807eb50 con 0x7f19c80a7740 2019-12-05T05:45:36.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:36.695+0000 7f19cd61d700 1 --2- 172.21.15.42:0/647437805 >> v2:172.21.15.42:3300/0 conn(0x7f19c80a7740 0x7f19c808df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f19bc008790 tx=0x7f19bc00fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:36.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:36.695+0000 7f19cd61d700 1 --2- 172.21.15.42:0/647437805 >> v2:172.21.15.42:3300/0 conn(0x7f19c80a7740 0x7f19c808df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f19bc008790 tx=0x7f19bc00fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:36.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:36.695+0000 7f19cd61d700 1 --2- 172.21.15.42:0/647437805 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f19c80a7740 0x7f19c808df80 secure :-1 s=READY pgs=1431 cs=0 l=1 rx=0x7f19bc008790 tx=0x7f19bc00fcb0).ready entity=mon.1 client_cookie=368e491e948c7993 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:45:36.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:36.695+0000 7f19cd61d700 1 --2- 172.21.15.42:0/647437805 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f19c80a7740 0x7f19c808df80 secure :-1 s=THROTTLE_DONE pgs=1431 cs=0 l=1 rx=0x7f19bc008790 tx=0x7f19bc00fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:36.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:36.695+0000 7f19cd61d700 1 --2- 172.21.15.42:0/647437805 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f19c80a7740 0x7f19c808df80 secure :-1 s=THROTTLE_DONE pgs=1431 cs=0 l=1 rx=0x7f19bc008790 tx=0x7f19bc00fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:36.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:36.695+0000 7f19c7fff700 1 -- 172.21.15.42:0/647437805 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f19bc011520 con 0x7f19c80a7740 2019-12-05T05:45:36.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:36.695+0000 7f19cd61d700 1 --2- 172.21.15.42:0/647437805 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f19c80a7740 0x7f19c808df80 secure :-1 s=THROTTLE_DONE pgs=1431 cs=0 l=1 rx=0x7f19bc008790 tx=0x7f19bc00fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:36.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:36.695+0000 7f19c7fff700 1 -- 172.21.15.42:0/647437805 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19bc008ca0 con 0x7f19c80a7740 2019-12-05T05:45:36.708 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:36.695+0000 7f19c7fff700 1 -- 172.21.15.42:0/647437805 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f19bc011b00 con 0x7f19c80a7740 2019-12-05T05:45:36.708 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:36.695+0000 7f19cf080700 1 -- 172.21.15.42:0/647437805 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f19c80a7740 msgr2=0x7f19c808df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:45:36.708 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:36.695+0000 7f19cf080700 1 --2- 172.21.15.42:0/647437805 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f19c80a7740 0x7f19c808df80 secure :-1 s=READY pgs=1431 cs=0 l=1 rx=0x7f19bc008790 tx=0x7f19bc00fcb0).stop 2019-12-05T05:45:36.708 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:36.699+0000 7f19cf080700 1 -- 172.21.15.42:0/647437805 shutdown_connections 2019-12-05T05:45:36.708 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:36.699+0000 7f19cf080700 1 --2- 172.21.15.42:0/647437805 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f19c80a7740 0x7f19c808df80 unknown :-1 s=CLOSED pgs=1431 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:45:36.708 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:36.699+0000 7f19cf080700 1 -- 172.21.15.42:0/647437805 shutdown_connections 2019-12-05T05:45:36.709 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:36.699+0000 7f19cf080700 1 -- 172.21.15.42:0/647437805 wait complete. 2019-12-05T05:45:36.709 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:36.699+0000 7f19cf080700 1 -- 172.21.15.42:0/647437805 >> 172.21.15.42:0/647437805 conn(0x7f19c80aabe0 msgr2=0x7f19c80a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:45:36.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:36.699+0000 7f19cf080700 1 Processor -- start 2019-12-05T05:45:36.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:36.699+0000 7f19cf080700 1 -- start start 2019-12-05T05:45:36.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:36.699+0000 7f19cf080700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f19c812b050 0x7f19c812d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:45:36.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:36.699+0000 7f19cf080700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19c8144460 con 0x7f19c81120f0 2019-12-05T05:45:36.715 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:36.699+0000 7f19cf080700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19c8150390 con 0x7f19c81259e0 2019-12-05T05:45:36.715 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:36.699+0000 7f19cf080700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19c81441c0 con 0x7f19c812b050 2019-12-05T05:45:36.715 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:36.699+0000 7f19cce1c700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f19c81259e0 0x7f19c8125e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:40614/0 (socket says 172.21.15.42:40614) 2019-12-05T05:45:36.715 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:36.699+0000 7f19cce1c700 1 -- 172.21.15.42:0/3634008072 learned_addr learned my addr 172.21.15.42:0/3634008072 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:45:36.715 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:36.699+0000 7f19cde1e700 1 --2- 172.21.15.42:0/3634008072 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f19c812b050 0x7f19c812d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:45:36.716 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:36.699+0000 7f19c5ffb700 1 -- 172.21.15.42:0/3634008072 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 74592953 0 0) 0x7f19c8150390 con 0x7f19c81259e0 2019-12-05T05:45:36.716 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:36.699+0000 7f19c5ffb700 1 -- 172.21.15.42:0/3634008072 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19a80030b0 con 0x7f19c81259e0 2019-12-05T05:45:36.716 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:36.699+0000 7f19c5ffb700 1 -- 172.21.15.42:0/3634008072 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 143490075 0 0) 0x7f19c8144460 con 0x7f19c81120f0 2019-12-05T05:45:36.716 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:36.699+0000 7f19c5ffb700 1 -- 172.21.15.42:0/3634008072 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19c8150390 con 0x7f19c81120f0 2019-12-05T05:45:36.716 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:36.699+0000 7f19c5ffb700 1 -- 172.21.15.42:0/3634008072 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 143611807 0 0) 0x7f19a80030b0 con 0x7f19c81259e0 2019-12-05T05:45:36.716 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:36.699+0000 7f19c5ffb700 1 -- 172.21.15.42:0/3634008072 --> v1:172.21.15.42:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f19c8144460 con 0x7f19c81259e0 2019-12-05T05:45:36.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:36.699+0000 7f19c5ffb700 1 -- 172.21.15.42:0/3634008072 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7f19c0001d90 con 0x7f19c81259e0 2019-12-05T05:45:36.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:36.699+0000 7f19c5ffb700 1 -- 172.21.15.42:0/3634008072 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f19c812b050 msgr2=0x7f19c812d490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:45:36.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:36.699+0000 7f19c5ffb700 1 --2- 172.21.15.42:0/3634008072 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f19c812b050 0x7f19c812d490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:45:36.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:36.699+0000 7f19c5ffb700 1 -- 172.21.15.42:0/3634008072 >> v1:172.21.15.42:6789/0 conn(0x7f19c81259e0 legacy=0x7f19c8125e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:45:36.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:36.703+0000 7f19c5ffb700 1 -- 172.21.15.42:0/3634008072 >> v1:172.21.15.40:6789/0 conn(0x7f19c81120f0 legacy=0x7f19c8143610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:45:36.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:36.703+0000 7f19c5ffb700 1 --2- 172.21.15.42:0/3634008072 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f19a8003ab0 0x7f19a8004420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:45:36.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:36.703+0000 7f19c5ffb700 1 --2- 172.21.15.42:0/3634008072 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f19a8003fc0 0x7f19a8004a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:45:36.718 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:36.703+0000 7f19c5ffb700 1 --2- 172.21.15.42:0/3634008072 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f19a8009660 0x7f19a800baa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:45:36.718 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:36.703+0000 7f19c5ffb700 1 -- 172.21.15.42:0/3634008072 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19c81441c0 con 0x7f19a8003ab0 2019-12-05T05:45:36.718 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:36.703+0000 7f19c5ffb700 1 -- 172.21.15.42:0/3634008072 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19a8005110 con 0x7f19a8003fc0 2019-12-05T05:45:36.718 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:36.703+0000 7f19c5ffb700 1 -- 172.21.15.42:0/3634008072 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19a800bff0 con 0x7f19a8009660 2019-12-05T05:45:36.718 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:36.703+0000 7f19cce1c700 1 --2- 172.21.15.42:0/3634008072 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f19a8003fc0 0x7f19a8004a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:45:36.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:36.703+0000 7f19cd61d700 1 --2- 172.21.15.42:0/3634008072 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f19a8003ab0 0x7f19a8004420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:45:36.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:36.703+0000 7f19cde1e700 1 --2- 172.21.15.42:0/3634008072 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f19a8009660 0x7f19a800baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:45:36.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:36.703+0000 7f19cd61d700 1 -- 172.21.15.42:0/3634008072 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f19a8009660 msgr2=0x7f19a800baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:45:36.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:36.703+0000 7f19cd61d700 1 --2- 172.21.15.42:0/3634008072 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f19a8009660 0x7f19a800baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:45:36.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:36.703+0000 7f19cd61d700 1 -- 172.21.15.42:0/3634008072 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f19a8003fc0 msgr2=0x7f19a8004a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:45:36.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:36.703+0000 7f19cd61d700 1 --2- 172.21.15.42:0/3634008072 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f19a8003fc0 0x7f19a8004a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:45:36.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:36.703+0000 7f19cd61d700 1 -- 172.21.15.42:0/3634008072 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19c8143e10 con 0x7f19a8003ab0 2019-12-05T05:45:36.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:36.703+0000 7f19cce1c700 1 --2- 172.21.15.42:0/3634008072 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f19a8003fc0 0x7f19a8004a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:45:36.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:36.703+0000 7f19cf080700 1 -- 172.21.15.42:0/3634008072 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19c808eb70 con 0x7f19a8003ab0 2019-12-05T05:45:36.724 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:36.703+0000 7f19cd61d700 1 --2- 172.21.15.42:0/3634008072 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f19a8003ab0 0x7f19a8004420 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f19bc008820 tx=0x7f19bc011660).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:36.724 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:36.703+0000 7f19cf080700 1 -- 172.21.15.42:0/3634008072 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19c808ef80 con 0x7f19a8003ab0 2019-12-05T05:45:36.724 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:36.703+0000 7f19cd61d700 1 --2- 172.21.15.42:0/3634008072 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f19a8003ab0 0x7f19a8004420 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f19bc008820 tx=0x7f19bc011660).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:36.724 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:36.703+0000 7f19cd61d700 1 --2- 172.21.15.42:0/3634008072 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f19a8003ab0 0x7f19a8004420 secure :-1 s=READY pgs=1105 cs=0 l=1 rx=0x7f19bc008820 tx=0x7f19bc011660).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:45:36.724 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:36.703+0000 7f19cd61d700 1 --2- 172.21.15.42:0/3634008072 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f19a8003ab0 0x7f19a8004420 secure :-1 s=THROTTLE_DONE pgs=1105 cs=0 l=1 rx=0x7f19bc008820 tx=0x7f19bc011660).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:36.724 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:36.703+0000 7f19c5ffb700 1 -- 172.21.15.42:0/3634008072 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f19bc00f700 con 0x7f19a8003ab0 2019-12-05T05:45:36.725 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:36.703+0000 7f19cd61d700 1 --2- 172.21.15.42:0/3634008072 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f19a8003ab0 0x7f19a8004420 secure :-1 s=THROTTLE_DONE pgs=1105 cs=0 l=1 rx=0x7f19bc008820 tx=0x7f19bc011660).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:36.725 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:36.703+0000 7f19c5ffb700 1 -- 172.21.15.42:0/3634008072 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19bc011720 con 0x7f19a8003ab0 2019-12-05T05:45:36.725 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:36.703+0000 7f19cd61d700 1 --2- 172.21.15.42:0/3634008072 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f19a8003ab0 0x7f19a8004420 secure :-1 s=THROTTLE_DONE pgs=1105 cs=0 l=1 rx=0x7f19bc008820 tx=0x7f19bc011660).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:36.725 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:36.703+0000 7f19c5ffb700 1 -- 172.21.15.42:0/3634008072 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f19bc005540 con 0x7f19a8003ab0 2019-12-05T05:45:36.725 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:36.703+0000 7f19cd61d700 1 --2- 172.21.15.42:0/3634008072 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f19a8003ab0 0x7f19a8004420 secure :-1 s=THROTTLE_DONE pgs=1105 cs=0 l=1 rx=0x7f19bc008820 tx=0x7f19bc011660).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:36.725 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:36.703+0000 7f19c5ffb700 1 -- 172.21.15.42:0/3634008072 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f19bc005740 con 0x7f19a8003ab0 2019-12-05T05:45:36.725 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:36.703+0000 7f19cd61d700 1 --2- 172.21.15.42:0/3634008072 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f19a8003ab0 0x7f19a8004420 secure :-1 s=THROTTLE_DONE pgs=1105 cs=0 l=1 rx=0x7f19bc008820 tx=0x7f19bc011660).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:36.726 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:36.703+0000 7f19c5ffb700 1 --2- 172.21.15.42:0/3634008072 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f19a8043d50 0x7f19a8046260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:45:36.726 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:36.703+0000 7f19c5ffb700 1 -- 172.21.15.42:0/3634008072 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 8439+0+0 (secure 0 0 0) 0x7f19bc04b6c0 con 0x7f19a8003ab0 2019-12-05T05:45:36.726 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:36.703+0000 7f19cce1c700 1 --2- 172.21.15.42:0/3634008072 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f19a8043d50 0x7f19a8046260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:45:36.726 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:36.707+0000 7f19cce1c700 1 --2- 172.21.15.42:0/3634008072 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f19a8043d50 0x7f19a8046260 crc :-1 s=READY pgs=616 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:45:36.727 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:36.711+0000 7f19cf080700 1 -- 172.21.15.42:0/3634008072 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19a0003200 con 0x7f19a8003ab0 2019-12-05T05:45:36.727 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:36.719+0000 7f19cd61d700 1 --2- 172.21.15.42:0/3634008072 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f19a8003ab0 0x7f19a8004420 secure :-1 s=THROTTLE_DONE pgs=1105 cs=0 l=1 rx=0x7f19bc008820 tx=0x7f19bc011660).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:36.727 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:36.719+0000 7f19c5ffb700 1 -- 172.21.15.42:0/3634008072 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f19bc032720 con 0x7f19a8003ab0 2019-12-05T05:45:36.897 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:36.891+0000 7f19cf080700 1 -- 172.21.15.42:0/3634008072 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f19a0003630 con 0x7f19a8003ab0 2019-12-05T05:45:36.897 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:36.891+0000 7f19cd61d700 1 --2- 172.21.15.42:0/3634008072 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f19a8003ab0 0x7f19a8004420 secure :-1 s=THROTTLE_DONE pgs=1105 cs=0 l=1 rx=0x7f19bc008820 tx=0x7f19bc011660).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:36.898 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:36.891+0000 7f19c5ffb700 1 -- 172.21.15.42:0/3634008072 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 173 v173) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f19bc032440 con 0x7f19a8003ab0 2019-12-05T05:45:36.902 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 173 2019-12-05T05:45:36.902 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:45:36.902 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":173,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6077,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6835/3757738536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3757738536},{"type":"v1","addr":"172.21.15.42:6835","nonce":3757738536}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":173}],"filesystems":[{"mdsmap":{"epoch":172,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:44:49.006174+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7884},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7884":{"gid":7884,"name":"b","rank":0,"incarnation":169,"state":"up:active","state_seq":11,"addr":"172.21.15.40:6833/1127133319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1127133319},{"type":"v1","addr":"172.21.15.40:6833","nonce":1127133319}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:45:36.911 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:36.895+0000 7f19cf080700 1 -- 172.21.15.42:0/3634008072 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f19a8043d50 msgr2=0x7f19a8046260 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:45:36.911 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:36.895+0000 7f19cf080700 1 --2- 172.21.15.42:0/3634008072 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f19a8043d50 0x7f19a8046260 crc :-1 s=READY pgs=616 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:45:36.911 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:36.895+0000 7f19cf080700 1 -- 172.21.15.42:0/3634008072 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f19a8003ab0 msgr2=0x7f19a8004420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:45:36.911 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:36.895+0000 7f19cf080700 1 --2- 172.21.15.42:0/3634008072 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f19a8003ab0 0x7f19a8004420 secure :-1 s=READY pgs=1105 cs=0 l=1 rx=0x7f19bc008820 tx=0x7f19bc011660).stop 2019-12-05T05:45:36.911 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:36.895+0000 7f19cde1e700 1 -- 172.21.15.42:0/3634008072 reap_dead start 2019-12-05T05:45:36.911 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:36.899+0000 7f19cf080700 1 -- 172.21.15.42:0/3634008072 shutdown_connections 2019-12-05T05:45:36.912 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:36.899+0000 7f19cf080700 1 -- 172.21.15.42:0/3634008072 shutdown_connections 2019-12-05T05:45:36.912 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:36.899+0000 7f19cf080700 1 -- 172.21.15.42:0/3634008072 wait complete. 2019-12-05T05:45:36.912 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:36.899+0000 7f19cf080700 1 -- 172.21.15.42:0/3634008072 >> 172.21.15.42:0/3634008072 conn(0x7f19c80a80d0 msgr2=0x7f19c80a4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:45:36.918 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T05:45:38.919 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:45:39.022 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.011+0000 7f568293b700 1 Processor -- start 2019-12-05T05:45:39.022 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.011+0000 7f568293b700 1 -- start start 2019-12-05T05:45:39.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.011+0000 7f568293b700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f567c0a7740 0x7f567c08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:45:39.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.011+0000 7f568293b700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f567c09fae0 0x7f567c07eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:45:39.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.011+0000 7f568293b700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f567c0a0710 con 0x7f567c07e720 2019-12-05T05:45:39.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.011+0000 7f568293b700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f567c0a0b00 con 0x7f567c0a7740 2019-12-05T05:45:39.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.011+0000 7f568293b700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f567c08e5b0 con 0x7f567c09fae0 2019-12-05T05:45:39.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.011+0000 7f56816d9700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f567c09fae0 0x7f567c07eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:45:39.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.011+0000 7f56816d9700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f567c09fae0 0x7f567c07eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:40594/0 (socket says 172.21.15.42:40594) 2019-12-05T05:45:39.024 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.011+0000 7f56816d9700 1 -- 172.21.15.42:0/3186002925 learned_addr learned my addr 172.21.15.42:0/3186002925 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:45:39.024 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.011+0000 7f5680ed8700 1 --2- 172.21.15.42:0/3186002925 >> v2:172.21.15.40:3300/0 conn(0x7f567c0a7740 0x7f567c08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:45:39.024 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.011+0000 7f56816d9700 1 -- 172.21.15.42:0/3186002925 >> v2:172.21.15.40:3300/0 conn(0x7f567c0a7740 msgr2=0x7f567c08df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:45:39.024 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.011+0000 7f56816d9700 1 --2- 172.21.15.42:0/3186002925 >> v2:172.21.15.40:3300/0 conn(0x7f567c0a7740 0x7f567c08df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:45:39.024 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.011+0000 7f56816d9700 1 -- 172.21.15.42:0/3186002925 >> v1:172.21.15.40:6789/0 conn(0x7f567c07e720 legacy=0x7f567c09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:45:39.024 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.011+0000 7f56816d9700 1 -- 172.21.15.42:0/3186002925 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f567c07f0a0 con 0x7f567c09fae0 2019-12-05T05:45:39.025 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.011+0000 7f56816d9700 1 --2- 172.21.15.42:0/3186002925 >> v2:172.21.15.42:3300/0 conn(0x7f567c09fae0 0x7f567c07eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f56740032b0 tx=0x7f56740100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:39.025 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.011+0000 7f56816d9700 1 --2- 172.21.15.42:0/3186002925 >> v2:172.21.15.42:3300/0 conn(0x7f567c09fae0 0x7f567c07eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f56740032b0 tx=0x7f56740100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:39.025 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.011+0000 7f56816d9700 1 --2- 172.21.15.42:0/3186002925 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f567c09fae0 0x7f567c07eb50 secure :-1 s=READY pgs=1433 cs=0 l=1 rx=0x7f56740032b0 tx=0x7f56740100b0).ready entity=mon.1 client_cookie=7a6a8c1ce8811d32 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:45:39.025 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.011+0000 7f56816d9700 1 --2- 172.21.15.42:0/3186002925 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f567c09fae0 0x7f567c07eb50 secure :-1 s=THROTTLE_DONE pgs=1433 cs=0 l=1 rx=0x7f56740032b0 tx=0x7f56740100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:39.026 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.011+0000 7f567b7fe700 1 -- 172.21.15.42:0/3186002925 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5674010d60 con 0x7f567c09fae0 2019-12-05T05:45:39.026 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.011+0000 7f56816d9700 1 --2- 172.21.15.42:0/3186002925 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f567c09fae0 0x7f567c07eb50 secure :-1 s=THROTTLE_DONE pgs=1433 cs=0 l=1 rx=0x7f56740032b0 tx=0x7f56740100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:39.026 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.011+0000 7f567b7fe700 1 -- 172.21.15.42:0/3186002925 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5674010980 con 0x7f567c09fae0 2019-12-05T05:45:39.026 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.011+0000 7f56816d9700 1 --2- 172.21.15.42:0/3186002925 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f567c09fae0 0x7f567c07eb50 secure :-1 s=THROTTLE_DONE pgs=1433 cs=0 l=1 rx=0x7f56740032b0 tx=0x7f56740100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:39.026 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.011+0000 7f567b7fe700 1 -- 172.21.15.42:0/3186002925 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f56740114a0 con 0x7f567c09fae0 2019-12-05T05:45:39.026 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.011+0000 7f568293b700 1 -- 172.21.15.42:0/3186002925 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f567c09fae0 msgr2=0x7f567c07eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:45:39.027 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.011+0000 7f568293b700 1 --2- 172.21.15.42:0/3186002925 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f567c09fae0 0x7f567c07eb50 secure :-1 s=READY pgs=1433 cs=0 l=1 rx=0x7f56740032b0 tx=0x7f56740100b0).stop 2019-12-05T05:45:39.027 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.015+0000 7f568293b700 1 -- 172.21.15.42:0/3186002925 shutdown_connections 2019-12-05T05:45:39.027 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.015+0000 7f568293b700 1 --2- 172.21.15.42:0/3186002925 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f567c09fae0 0x7f567c07eb50 unknown :-1 s=CLOSED pgs=1433 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:45:39.027 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.015+0000 7f568293b700 1 --2- 172.21.15.42:0/3186002925 >> v2:172.21.15.40:3300/0 conn(0x7f567c0a7740 0x7f567c08df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:45:39.027 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.015+0000 7f568293b700 1 -- 172.21.15.42:0/3186002925 shutdown_connections 2019-12-05T05:45:39.027 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.015+0000 7f568293b700 1 -- 172.21.15.42:0/3186002925 wait complete. 2019-12-05T05:45:39.027 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.015+0000 7f568293b700 1 -- 172.21.15.42:0/3186002925 >> 172.21.15.42:0/3186002925 conn(0x7f567c0aabe0 msgr2=0x7f567c0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:45:39.029 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.015+0000 7f568293b700 1 Processor -- start 2019-12-05T05:45:39.029 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.019+0000 7f568293b700 1 -- start start 2019-12-05T05:45:39.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.019+0000 7f568293b700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f567c123a60 0x7f567c121100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:45:39.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.019+0000 7f568293b700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f567c1199e0 0x7f567c0a5710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:45:39.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.019+0000 7f568293b700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f567c122e60 con 0x7f567c122a30 2019-12-05T05:45:39.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.019+0000 7f568293b700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f567c123330 con 0x7f567c1199e0 2019-12-05T05:45:39.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.019+0000 7f568293b700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f567c0a5470 con 0x7f567c123a60 2019-12-05T05:45:39.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.019+0000 7f5680ed8700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f567c123a60 0x7f567c121100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:45:39.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.019+0000 7f5680ed8700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f567c123a60 0x7f567c121100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:41062/0 (socket says 172.21.15.42:41062) 2019-12-05T05:45:39.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.019+0000 7f567bfff700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f567c1199e0 0x7f567c0a5710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:45:39.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.019+0000 7f5680ed8700 1 -- 172.21.15.42:0/3600635930 learned_addr learned my addr 172.21.15.42:0/3600635930 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:45:39.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.019+0000 7f5680ed8700 1 -- 172.21.15.42:0/3600635930 >> v2:172.21.15.40:3300/0 conn(0x7f567c1199e0 msgr2=0x7f567c0a5710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:45:39.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.019+0000 7f5680ed8700 1 --2- 172.21.15.42:0/3600635930 >> v2:172.21.15.40:3300/0 conn(0x7f567c1199e0 0x7f567c0a5710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:45:39.032 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.019+0000 7f5680ed8700 1 -- 172.21.15.42:0/3600635930 >> v1:172.21.15.40:6789/0 conn(0x7f567c122a30 legacy=0x7f567c08eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:45:39.032 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.019+0000 7f5680ed8700 1 -- 172.21.15.42:0/3600635930 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f567c08f350 con 0x7f567c123a60 2019-12-05T05:45:39.032 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.019+0000 7f5680ed8700 1 --2- 172.21.15.42:0/3600635930 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f567c123a60 0x7f567c121100 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f5670003070 tx=0x7f5670010ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:39.032 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.019+0000 7f5680ed8700 1 --2- 172.21.15.42:0/3600635930 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f567c123a60 0x7f567c121100 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f5670003070 tx=0x7f5670010ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:39.032 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.019+0000 7f5680ed8700 1 --2- 172.21.15.42:0/3600635930 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f567c123a60 0x7f567c121100 secure :-1 s=READY pgs=313 cs=0 l=1 rx=0x7f5670003070 tx=0x7f5670010ae0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:45:39.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.027+0000 7f5680ed8700 1 --2- 172.21.15.42:0/3600635930 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f567c123a60 0x7f567c121100 secure :-1 s=THROTTLE_DONE pgs=313 cs=0 l=1 rx=0x7f5670003070 tx=0x7f5670010ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:39.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.027+0000 7f5680ed8700 1 --2- 172.21.15.42:0/3600635930 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f567c123a60 0x7f567c121100 secure :-1 s=THROTTLE_DONE pgs=313 cs=0 l=1 rx=0x7f5670003070 tx=0x7f5670010ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:39.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.027+0000 7f56797fa700 1 -- 172.21.15.42:0/3600635930 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5670005930 con 0x7f567c123a60 2019-12-05T05:45:39.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.027+0000 7f5680ed8700 1 --2- 172.21.15.42:0/3600635930 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f567c123a60 0x7f567c121100 secure :-1 s=THROTTLE_DONE pgs=313 cs=0 l=1 rx=0x7f5670003070 tx=0x7f5670010ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:39.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.027+0000 7f56797fa700 1 -- 172.21.15.42:0/3600635930 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5670007160 con 0x7f567c123a60 2019-12-05T05:45:39.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.027+0000 7f56797fa700 1 -- 172.21.15.42:0/3600635930 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5670012160 con 0x7f567c123a60 2019-12-05T05:45:39.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.027+0000 7f568293b700 1 -- 172.21.15.42:0/3600635930 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f567c08f5c0 con 0x7f567c123a60 2019-12-05T05:45:39.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.027+0000 7f568293b700 1 -- 172.21.15.42:0/3600635930 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f567c077680 con 0x7f567c123a60 2019-12-05T05:45:39.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.027+0000 7f5680ed8700 1 --2- 172.21.15.42:0/3600635930 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f567c123a60 0x7f567c121100 secure :-1 s=THROTTLE_DONE pgs=313 cs=0 l=1 rx=0x7f5670003070 tx=0x7f5670010ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:39.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.027+0000 7f56797fa700 1 -- 172.21.15.42:0/3600635930 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f5670012470 con 0x7f567c123a60 2019-12-05T05:45:39.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.027+0000 7f5680ed8700 1 --2- 172.21.15.42:0/3600635930 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f567c123a60 0x7f567c121100 secure :-1 s=THROTTLE_DONE pgs=313 cs=0 l=1 rx=0x7f5670003070 tx=0x7f5670010ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:39.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.027+0000 7f56797fa700 1 --2- 172.21.15.42:0/3600635930 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f565c03ad70 0x7f565c03d280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:45:39.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.027+0000 7f56797fa700 1 -- 172.21.15.42:0/3600635930 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 8525+0+0 (secure 0 0 0) 0x7f567004ae90 con 0x7f567c123a60 2019-12-05T05:45:39.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.027+0000 7f567bfff700 1 --2- 172.21.15.42:0/3600635930 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f565c03ad70 0x7f565c03d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:45:39.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.027+0000 7f567bfff700 1 --2- 172.21.15.42:0/3600635930 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f565c03ad70 0x7f565c03d280 crc :-1 s=READY pgs=617 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:45:39.047 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.043+0000 7f568293b700 1 -- 172.21.15.42:0/3600635930 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f567c05d110 con 0x7f567c123a60 2019-12-05T05:45:39.052 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.047+0000 7f5680ed8700 1 --2- 172.21.15.42:0/3600635930 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f567c123a60 0x7f567c121100 secure :-1 s=THROTTLE_DONE pgs=313 cs=0 l=1 rx=0x7f5670003070 tx=0x7f5670010ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:39.053 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.047+0000 7f56797fa700 1 -- 172.21.15.42:0/3600635930 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f5670031f10 con 0x7f567c123a60 2019-12-05T05:45:39.220 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.215+0000 7f568293b700 1 -- 172.21.15.42:0/3600635930 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f567c0a6990 con 0x7f567c123a60 2019-12-05T05:45:39.221 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.215+0000 7f5680ed8700 1 --2- 172.21.15.42:0/3600635930 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f567c123a60 0x7f567c121100 secure :-1 s=THROTTLE_DONE pgs=313 cs=0 l=1 rx=0x7f5670003070 tx=0x7f5670010ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:39.221 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.215+0000 7f56797fa700 1 -- 172.21.15.42:0/3600635930 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 174 v174) v1 ==== 95+0+1774 (secure 0 0 0) 0x7f5670032170 con 0x7f567c123a60 2019-12-05T05:45:39.221 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 174 2019-12-05T05:45:39.221 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:45:39.222 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":174,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":174,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:45:39.023639+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":78,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6077},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6077":{"gid":6077,"name":"a","rank":0,"incarnation":174,"state":"up:replay","state_seq":1,"addr":"172.21.15.42:6835/3757738536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3757738536},{"type":"v1","addr":"172.21.15.42:6835","nonce":3757738536}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:45:39.223 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.215+0000 7f568293b700 1 -- 172.21.15.42:0/3600635930 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f565c03ad70 msgr2=0x7f565c03d280 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:45:39.223 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.215+0000 7f568293b700 1 --2- 172.21.15.42:0/3600635930 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f565c03ad70 0x7f565c03d280 crc :-1 s=READY pgs=617 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:45:39.223 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.215+0000 7f568293b700 1 -- 172.21.15.42:0/3600635930 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f567c123a60 msgr2=0x7f567c121100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:45:39.223 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.219+0000 7f568293b700 1 --2- 172.21.15.42:0/3600635930 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f567c123a60 0x7f567c121100 secure :-1 s=READY pgs=313 cs=0 l=1 rx=0x7f5670003070 tx=0x7f5670010ae0).stop 2019-12-05T05:45:39.224 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.219+0000 7f568293b700 1 -- 172.21.15.42:0/3600635930 shutdown_connections 2019-12-05T05:45:39.224 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.219+0000 7f568293b700 1 --2- 172.21.15.42:0/3600635930 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f565c03ad70 0x7f565c03d280 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:45:39.225 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.219+0000 7f568293b700 1 --2- 172.21.15.42:0/3600635930 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f567c123a60 0x7f567c121100 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:45:39.225 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.219+0000 7f568293b700 1 --2- 172.21.15.42:0/3600635930 >> v2:172.21.15.40:3300/0 conn(0x7f567c1199e0 0x7f567c0a5710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:45:39.225 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.219+0000 7f568293b700 1 -- 172.21.15.42:0/3600635930 shutdown_connections 2019-12-05T05:45:39.225 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.219+0000 7f568293b700 1 -- 172.21.15.42:0/3600635930 wait complete. 2019-12-05T05:45:39.225 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.219+0000 7f568293b700 1 -- 172.21.15.42:0/3600635930 >> 172.21.15.42:0/3600635930 conn(0x7f567c0aabe0 msgr2=0x7f567c059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:45:39.233 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b down, removed from mdsmap 2019-12-05T05:45:39.234 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2019-12-05T05:45:39.234 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:45:39.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.327+0000 7f325a0a8700 1 Processor -- start 2019-12-05T05:45:39.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.327+0000 7f325a0a8700 1 -- start start 2019-12-05T05:45:39.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.327+0000 7f325a0a8700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f325408df80 0x7f325409f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:45:39.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.327+0000 7f325a0a8700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f325409f830 0x7f325414efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:45:39.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.327+0000 7f325a0a8700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f325409fc60 con 0x7f32540a7740 2019-12-05T05:45:39.341 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.327+0000 7f325a0a8700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32540a0b00 con 0x7f325408df80 2019-12-05T05:45:39.341 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.327+0000 7f325a0a8700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f325408e5b0 con 0x7f325409f830 2019-12-05T05:45:39.341 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.327+0000 7f3258e46700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f325409f830 0x7f325414efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:45:39.341 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.327+0000 7f3258e46700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f325409f830 0x7f325414efe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:40622/0 (socket says 172.21.15.42:40622) 2019-12-05T05:45:39.341 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.327+0000 7f3258e46700 1 -- 172.21.15.42:0/1990995170 learned_addr learned my addr 172.21.15.42:0/1990995170 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:45:39.341 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.327+0000 7f32537fe700 1 --2- 172.21.15.42:0/1990995170 >> v2:172.21.15.40:3300/0 conn(0x7f325408df80 0x7f325409f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:45:39.342 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.327+0000 7f3258e46700 1 -- 172.21.15.42:0/1990995170 >> v2:172.21.15.40:3300/0 conn(0x7f325408df80 msgr2=0x7f325409f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:45:39.342 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.327+0000 7f3258e46700 1 --2- 172.21.15.42:0/1990995170 >> v2:172.21.15.40:3300/0 conn(0x7f325408df80 0x7f325409f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:45:39.342 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.327+0000 7f3258e46700 1 -- 172.21.15.42:0/1990995170 >> v1:172.21.15.40:6789/0 conn(0x7f32540a7740 legacy=0x7f325407e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:45:39.342 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.327+0000 7f3258e46700 1 -- 172.21.15.42:0/1990995170 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32540a0710 con 0x7f325409f830 2019-12-05T05:45:39.343 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.327+0000 7f3258e46700 1 --2- 172.21.15.42:0/1990995170 >> v2:172.21.15.42:3300/0 conn(0x7f325409f830 0x7f325414efe0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f324c0032b0 tx=0x7f324c0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:39.343 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.327+0000 7f3258e46700 1 --2- 172.21.15.42:0/1990995170 >> v2:172.21.15.42:3300/0 conn(0x7f325409f830 0x7f325414efe0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f324c0032b0 tx=0x7f324c0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:39.343 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.327+0000 7f3258e46700 1 --2- 172.21.15.42:0/1990995170 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f325409f830 0x7f325414efe0 secure :-1 s=READY pgs=1434 cs=0 l=1 rx=0x7f324c0032b0 tx=0x7f324c0100b0).ready entity=mon.1 client_cookie=b3d4a6c94cd41866 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:45:39.343 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.327+0000 7f3258e46700 1 --2- 172.21.15.42:0/1990995170 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f325409f830 0x7f325414efe0 secure :-1 s=THROTTLE_DONE pgs=1434 cs=0 l=1 rx=0x7f324c0032b0 tx=0x7f324c0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:39.343 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.327+0000 7f3252ffd700 1 -- 172.21.15.42:0/1990995170 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f324c010d60 con 0x7f325409f830 2019-12-05T05:45:39.343 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.327+0000 7f3258e46700 1 --2- 172.21.15.42:0/1990995170 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f325409f830 0x7f325414efe0 secure :-1 s=THROTTLE_DONE pgs=1434 cs=0 l=1 rx=0x7f324c0032b0 tx=0x7f324c0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:39.344 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.327+0000 7f3258e46700 1 --2- 172.21.15.42:0/1990995170 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f325409f830 0x7f325414efe0 secure :-1 s=THROTTLE_DONE pgs=1434 cs=0 l=1 rx=0x7f324c0032b0 tx=0x7f324c0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:39.344 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.327+0000 7f3252ffd700 1 -- 172.21.15.42:0/1990995170 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f324c010980 con 0x7f325409f830 2019-12-05T05:45:39.344 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.327+0000 7f3252ffd700 1 -- 172.21.15.42:0/1990995170 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f324c0114a0 con 0x7f325409f830 2019-12-05T05:45:39.344 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.327+0000 7f325a0a8700 1 -- 172.21.15.42:0/1990995170 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f325409f830 msgr2=0x7f325414efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:45:39.344 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.327+0000 7f325a0a8700 1 --2- 172.21.15.42:0/1990995170 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f325409f830 0x7f325414efe0 secure :-1 s=READY pgs=1434 cs=0 l=1 rx=0x7f324c0032b0 tx=0x7f324c0100b0).stop 2019-12-05T05:45:39.344 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.327+0000 7f325a0a8700 1 -- 172.21.15.42:0/1990995170 shutdown_connections 2019-12-05T05:45:39.344 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.327+0000 7f325a0a8700 1 --2- 172.21.15.42:0/1990995170 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f325409f830 0x7f325414efe0 unknown :-1 s=CLOSED pgs=1434 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:45:39.345 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.327+0000 7f325a0a8700 1 --2- 172.21.15.42:0/1990995170 >> v2:172.21.15.40:3300/0 conn(0x7f325408df80 0x7f325409f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:45:39.345 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.327+0000 7f325a0a8700 1 -- 172.21.15.42:0/1990995170 shutdown_connections 2019-12-05T05:45:39.345 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.327+0000 7f325a0a8700 1 -- 172.21.15.42:0/1990995170 wait complete. 2019-12-05T05:45:39.345 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.327+0000 7f325a0a8700 1 -- 172.21.15.42:0/1990995170 >> 172.21.15.42:0/1990995170 conn(0x7f32540aabe0 msgr2=0x7f32540a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:45:39.345 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.331+0000 7f325a0a8700 1 Processor -- start 2019-12-05T05:45:39.345 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.331+0000 7f325a0a8700 1 -- start start 2019-12-05T05:45:39.345 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.331+0000 7f325a0a8700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f32540a2f90 0x7f32540a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:45:39.346 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.331+0000 7f325a0a8700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f32540a6890 0x7f32540a5a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:45:39.346 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.331+0000 7f325a0a8700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32540a5fd0 con 0x7f32540a1850 2019-12-05T05:45:39.346 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.331+0000 7f325a0a8700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32540a26e0 con 0x7f32540a2f90 2019-12-05T05:45:39.346 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.331+0000 7f325a0a8700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32540a3b20 con 0x7f32540a6890 2019-12-05T05:45:39.346 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.331+0000 7f3258e46700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f32540a1850 0x7f325407d9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:40666/0 (socket says 172.21.15.42:40666) 2019-12-05T05:45:39.346 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.331+0000 7f3258e46700 1 -- 172.21.15.42:0/2873613254 learned_addr learned my addr 172.21.15.42:0/2873613254 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:45:39.346 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.331+0000 7f3253fff700 1 --2- 172.21.15.42:0/2873613254 >> v2:172.21.15.40:3300/0 conn(0x7f32540a2f90 0x7f32540a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:45:39.347 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.331+0000 7f32537fe700 1 --2- 172.21.15.42:0/2873613254 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f32540a6890 0x7f32540a5a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:45:39.347 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.331+0000 7f3250ff9700 1 -- 172.21.15.42:0/2873613254 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2087085338 0 0) 0x7f32540a5fd0 con 0x7f32540a1850 2019-12-05T05:45:39.347 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.331+0000 7f3250ff9700 1 -- 172.21.15.42:0/2873613254 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32380030b0 con 0x7f32540a1850 2019-12-05T05:45:39.347 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.331+0000 7f3253fff700 1 -- 172.21.15.42:0/2873613254 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f32540a6890 msgr2=0x7f32540a5a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:45:39.347 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.331+0000 7f3253fff700 1 --2- 172.21.15.42:0/2873613254 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f32540a6890 0x7f32540a5a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:45:39.347 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.331+0000 7f3253fff700 1 -- 172.21.15.42:0/2873613254 >> v1:172.21.15.42:6789/0 conn(0x7f32540a1850 legacy=0x7f325407d9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:45:39.347 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.331+0000 7f3253fff700 1 -- 172.21.15.42:0/2873613254 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f325407e1d0 con 0x7f32540a2f90 2019-12-05T05:45:39.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.331+0000 7f3253fff700 1 --2- 172.21.15.42:0/2873613254 >> v2:172.21.15.40:3300/0 conn(0x7f32540a2f90 0x7f32540a4060 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f3244002ef0 tx=0x7f3244010890).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:39.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.331+0000 7f3253fff700 1 --2- 172.21.15.42:0/2873613254 >> v2:172.21.15.40:3300/0 conn(0x7f32540a2f90 0x7f32540a4060 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f3244002ef0 tx=0x7f3244010890).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:39.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.331+0000 7f3253fff700 1 --2- 172.21.15.42:0/2873613254 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f32540a2f90 0x7f32540a4060 secure :-1 s=READY pgs=1109 cs=0 l=1 rx=0x7f3244002ef0 tx=0x7f3244010890).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:45:39.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.331+0000 7f3253fff700 1 --2- 172.21.15.42:0/2873613254 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f32540a2f90 0x7f32540a4060 secure :-1 s=THROTTLE_DONE pgs=1109 cs=0 l=1 rx=0x7f3244002ef0 tx=0x7f3244010890).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:39.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.331+0000 7f3250ff9700 1 -- 172.21.15.42:0/2873613254 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f32440057c0 con 0x7f32540a2f90 2019-12-05T05:45:39.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.331+0000 7f325a0a8700 1 -- 172.21.15.42:0/2873613254 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32540a6fe0 con 0x7f32540a2f90 2019-12-05T05:45:39.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.331+0000 7f3253fff700 1 --2- 172.21.15.42:0/2873613254 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f32540a2f90 0x7f32540a4060 secure :-1 s=THROTTLE_DONE pgs=1109 cs=0 l=1 rx=0x7f3244002ef0 tx=0x7f3244010890).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:39.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.331+0000 7f3250ff9700 1 -- 172.21.15.42:0/2873613254 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3244007160 con 0x7f32540a2f90 2019-12-05T05:45:39.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.331+0000 7f3253fff700 1 --2- 172.21.15.42:0/2873613254 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f32540a2f90 0x7f32540a4060 secure :-1 s=THROTTLE_DONE pgs=1109 cs=0 l=1 rx=0x7f3244002ef0 tx=0x7f3244010890).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:39.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.331+0000 7f3250ff9700 1 -- 172.21.15.42:0/2873613254 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3244005da0 con 0x7f32540a2f90 2019-12-05T05:45:39.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.331+0000 7f325a0a8700 1 -- 172.21.15.42:0/2873613254 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3254135360 con 0x7f32540a2f90 2019-12-05T05:45:39.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.335+0000 7f3253fff700 1 --2- 172.21.15.42:0/2873613254 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f32540a2f90 0x7f32540a4060 secure :-1 s=THROTTLE_DONE pgs=1109 cs=0 l=1 rx=0x7f3244002ef0 tx=0x7f3244010890).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:39.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.335+0000 7f3250ff9700 1 -- 172.21.15.42:0/2873613254 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f3244008730 con 0x7f32540a2f90 2019-12-05T05:45:39.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.335+0000 7f3253fff700 1 --2- 172.21.15.42:0/2873613254 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f32540a2f90 0x7f32540a4060 secure :-1 s=THROTTLE_DONE pgs=1109 cs=0 l=1 rx=0x7f3244002ef0 tx=0x7f3244010890).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:39.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.335+0000 7f3250ff9700 1 --2- 172.21.15.42:0/2873613254 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f323803b2b0 0x7f323803d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:45:39.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.335+0000 7f3250ff9700 1 -- 172.21.15.42:0/2873613254 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 8525+0+0 (secure 0 0 0) 0x7f324404ad80 con 0x7f32540a2f90 2019-12-05T05:45:39.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.335+0000 7f32537fe700 1 --2- 172.21.15.42:0/2873613254 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f323803b2b0 0x7f323803d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:45:39.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.335+0000 7f32537fe700 1 --2- 172.21.15.42:0/2873613254 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f323803b2b0 0x7f323803d7c0 crc :-1 s=READY pgs=618 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:45:39.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.343+0000 7f325a0a8700 1 -- 172.21.15.42:0/2873613254 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f325405d110 con 0x7f32540a2f90 2019-12-05T05:45:39.360 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.355+0000 7f3253fff700 1 --2- 172.21.15.42:0/2873613254 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f32540a2f90 0x7f32540a4060 secure :-1 s=THROTTLE_DONE pgs=1109 cs=0 l=1 rx=0x7f3244002ef0 tx=0x7f3244010890).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:39.360 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.355+0000 7f3250ff9700 1 -- 172.21.15.42:0/2873613254 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f3244031d80 con 0x7f32540a2f90 2019-12-05T05:45:39.528 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.519+0000 7f325a0a8700 1 -- 172.21.15.42:0/2873613254 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f325405b8c0 con 0x7f32540a2f90 2019-12-05T05:45:39.528 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.523+0000 7f3253fff700 1 --2- 172.21.15.42:0/2873613254 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f32540a2f90 0x7f32540a4060 secure :-1 s=THROTTLE_DONE pgs=1109 cs=0 l=1 rx=0x7f3244002ef0 tx=0x7f3244010890).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:39.528 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.523+0000 7f3250ff9700 1 -- 172.21.15.42:0/2873613254 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 174 v174) v1 ==== 95+0+1774 (secure 0 0 0) 0x7f3244031fe0 con 0x7f32540a2f90 2019-12-05T05:45:39.555 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 174 2019-12-05T05:45:39.555 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.527+0000 7f325a0a8700 1 -- 172.21.15.42:0/2873613254 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f323803b2b0 msgr2=0x7f323803d7c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:45:39.556 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.527+0000 7f325a0a8700 1 --2- 172.21.15.42:0/2873613254 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f323803b2b0 0x7f323803d7c0 crc :-1 s=READY pgs=618 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:45:39.556 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.527+0000 7f325a0a8700 1 -- 172.21.15.42:0/2873613254 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f32540a2f90 msgr2=0x7f32540a4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:45:39.556 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.527+0000 7f325a0a8700 1 --2- 172.21.15.42:0/2873613254 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f32540a2f90 0x7f32540a4060 secure :-1 s=READY pgs=1109 cs=0 l=1 rx=0x7f3244002ef0 tx=0x7f3244010890).stop 2019-12-05T05:45:39.558 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.527+0000 7f325a0a8700 1 -- 172.21.15.42:0/2873613254 shutdown_connections 2019-12-05T05:45:39.558 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.527+0000 7f325a0a8700 1 --2- 172.21.15.42:0/2873613254 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f323803b2b0 0x7f323803d7c0 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:45:39.558 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.527+0000 7f325a0a8700 1 --2- 172.21.15.42:0/2873613254 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f32540a2f90 0x7f32540a4060 unknown :-1 s=CLOSED pgs=1109 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:45:39.558 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.527+0000 7f325a0a8700 1 --2- 172.21.15.42:0/2873613254 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f32540a6890 0x7f32540a5a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:45:39.558 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.527+0000 7f325a0a8700 1 -- 172.21.15.42:0/2873613254 shutdown_connections 2019-12-05T05:45:39.559 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.527+0000 7f325a0a8700 1 -- 172.21.15.42:0/2873613254 wait complete. 2019-12-05T05:45:39.559 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:39.527+0000 7f325a0a8700 1 -- 172.21.15.42:0/2873613254 >> 172.21.15.42:0/2873613254 conn(0x7f325408ee10 msgr2=0x7f3254059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:45:39.559 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:45:39.560 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":174,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":174,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:45:39.023639+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":78,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6077},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6077":{"gid":6077,"name":"a","rank":0,"incarnation":174,"state":"up:replay","state_seq":1,"addr":"172.21.15.42:6835/3757738536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3757738536},{"type":"v1","addr":"172.21.15.42:6835","nonce":3757738536}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:45:39.561 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 174, "feature_flags": { "enable_multiple": false, "ever_enabled_multiple": false }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2019-12-05T05:03:35.604738+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 174, "ever_allowed_features": 2, "explicitly_allowed_features": 2, "failed": [], "flags": 18, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_6077": { "addr": "172.21.15.42:6835/3757738536", "addrs": { "addrvec": [ { "addr": "172.21.15.42:6834", "nonce": 3757738536, "type": "v2" }, { "addr": "172.21.15.42:6835", "nonce": 3757738536, "type": "v1" } ] }, "export_targets": [], "features": 4540138292836696063, "flags": 0, "gid": 6077, "incarnation": 174, "name": "a", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 78, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 2, "min_compat_client": "0 (unknown)", "modified": "2019-12-05T05:45:39.023639+0000", "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6077 } } } ], "standbys": [] } 2019-12-05T05:45:41.366 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:45:41.369 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:45:41.563 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:45:41.869 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:41.651+0000 7f83ba5ca700 1 Processor -- start 2019-12-05T05:45:41.869 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:41.651+0000 7f83ba5ca700 1 -- start start 2019-12-05T05:45:41.869 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:41.655+0000 7f83ba5ca700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f83b40a7740 0x7f83b408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:45:41.869 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:41.655+0000 7f83ba5ca700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83b40a0710 con 0x7f83b409fae0 2019-12-05T05:45:41.869 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:41.655+0000 7f83ba5ca700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83b40a9ba0 con 0x7f83b407e720 2019-12-05T05:45:41.870 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:41.655+0000 7f83ba5ca700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83b40a0b00 con 0x7f83b40a7740 2019-12-05T05:45:41.870 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:41.655+0000 7f83b3fff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f83b407e720 0x7f83b409f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:40674/0 (socket says 172.21.15.42:40674) 2019-12-05T05:45:41.870 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:41.655+0000 7f83b3fff700 1 -- 172.21.15.42:0/1603430405 learned_addr learned my addr 172.21.15.42:0/1603430405 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:45:41.870 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:41.655+0000 7f83b8b67700 1 --2- 172.21.15.42:0/1603430405 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f83b40a7740 0x7f83b408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:45:41.870 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:41.655+0000 7f83b37fe700 1 -- 172.21.15.42:0/1603430405 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 102557201 0 0) 0x7f83b40a9ba0 con 0x7f83b407e720 2019-12-05T05:45:41.870 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:41.655+0000 7f83b37fe700 1 -- 172.21.15.42:0/1603430405 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83a40030b0 con 0x7f83b407e720 2019-12-05T05:45:41.870 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:41.655+0000 7f83b8b67700 1 -- 172.21.15.42:0/1603430405 >> v1:172.21.15.42:6789/0 conn(0x7f83b407e720 legacy=0x7f83b409f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:45:41.871 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:41.655+0000 7f83b8b67700 1 -- 172.21.15.42:0/1603430405 >> v1:172.21.15.40:6789/0 conn(0x7f83b409fae0 legacy=0x7f83b414efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:45:41.871 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:41.655+0000 7f83b8b67700 1 -- 172.21.15.42:0/1603430405 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83b407eb50 con 0x7f83b40a7740 2019-12-05T05:45:41.871 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:41.655+0000 7f83b8b67700 1 --2- 172.21.15.42:0/1603430405 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f83b40a7740 0x7f83b408df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f83a8002f30 tx=0x7f83a800fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:41.871 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:41.655+0000 7f83b8b67700 1 -- 172.21.15.42:0/1603430405 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f83b40a7740 msgr2=0x7f83b408df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2019-12-05T05:45:41.871 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:41.655+0000 7f83b8b67700 1 -- 172.21.15.42:0/1603430405 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f83b40a7740 msgr2=0x7f83b408df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2019-12-05T05:45:41.871 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:41.655+0000 7f83b8b67700 1 --2- 172.21.15.42:0/1603430405 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f83b40a7740 0x7f83b408df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f83a8002f30 tx=0x7f83a800fbf0).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-12-05T05:45:41.872 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:41.655+0000 7f83b8b67700 1 --2- 172.21.15.42:0/1603430405 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f83b40a7740 0x7f83b408df80 secure :-1 s=START_CONNECT pgs=0 cs=0 l=0 rx=0x7f83a8002f30 tx=0x7f83a800fbf0)._fault waiting 0.200000 2019-12-05T05:45:41.872 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:41.859+0000 7f83b8b67700 1 --2- 172.21.15.42:0/1603430405 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f83b40a7740 0x7f83b408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:45:41.872 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:41.859+0000 7f83b8b67700 0 --2- 172.21.15.42:0/1603430405 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f83b40a7740 0x7f83b408df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2019-12-05T05:45:41.872 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:41.859+0000 7f83b8b67700 1 --2- 172.21.15.42:0/1603430405 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f83b40a7740 0x7f83b408df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:45:41.872 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:41.859+0000 7f83b37fe700 1 -- 172.21.15.42:0/1603430405 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f83b40a7740 msgr2=0x7f83b408df80 unknown :-1 s=STATE_CLOSED l=0).mark_down 2019-12-05T05:45:41.872 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:41.859+0000 7f83b37fe700 1 --2- 172.21.15.42:0/1603430405 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f83b40a7740 0x7f83b408df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:45:41.872 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:41.859+0000 7f83b37fe700 1 --2- 172.21.15.42:0/1603430405 >> v2:172.21.15.42:3300/0 conn(0x7f83a4003420 0x7f83a4003880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:45:41.873 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:41.859+0000 7f83b37fe700 1 --2- 172.21.15.42:0/1603430405 >> v2:172.21.15.40:3300/0 conn(0x7f83a4003ee0 0x7f83a4004440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:45:41.873 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:41.859+0000 7f83b37fe700 1 -- 172.21.15.42:0/1603430405 --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83a4004ad0 con 0x7f83a4003ee0 2019-12-05T05:45:41.873 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:41.859+0000 7f83b37fe700 1 -- 172.21.15.42:0/1603430405 --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83a4004cc0 con 0x7f83a4003420 2019-12-05T05:45:41.873 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:41.859+0000 7f83b8b67700 1 --2- 172.21.15.42:0/1603430405 >> v2:172.21.15.42:3300/0 conn(0x7f83a4003420 0x7f83a4003880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:45:41.873 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:41.859+0000 7f83b3fff700 1 --2- 172.21.15.42:0/1603430405 >> v2:172.21.15.40:3300/0 conn(0x7f83a4003ee0 0x7f83a4004440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:45:41.873 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:41.859+0000 7f83b8b67700 1 -- 172.21.15.42:0/1603430405 >> v2:172.21.15.40:3300/0 conn(0x7f83a4003ee0 msgr2=0x7f83a4004440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:45:41.873 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:41.859+0000 7f83b8b67700 1 --2- 172.21.15.42:0/1603430405 >> v2:172.21.15.40:3300/0 conn(0x7f83a4003ee0 0x7f83a4004440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:45:41.874 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:41.859+0000 7f83b8b67700 1 -- 172.21.15.42:0/1603430405 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83a4004f70 con 0x7f83a4003420 2019-12-05T05:45:41.874 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:41.859+0000 7f83b8b67700 1 --2- 172.21.15.42:0/1603430405 >> v2:172.21.15.42:3300/0 conn(0x7f83a4003420 0x7f83a4003880 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f83a8010000 tx=0x7f83a8008d20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:41.874 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:41.859+0000 7f83b8b67700 1 --2- 172.21.15.42:0/1603430405 >> v2:172.21.15.42:3300/0 conn(0x7f83a4003420 0x7f83a4003880 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f83a8010000 tx=0x7f83a8008d20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:41.874 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:41.859+0000 7f83b8b67700 1 --2- 172.21.15.42:0/1603430405 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f83a4003420 0x7f83a4003880 secure :-1 s=READY pgs=1437 cs=0 l=1 rx=0x7f83a8010000 tx=0x7f83a8008d20).ready entity=mon.1 client_cookie=943a070d0a641458 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:45:41.874 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:41.859+0000 7f83b8b67700 1 --2- 172.21.15.42:0/1603430405 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f83a4003420 0x7f83a4003880 secure :-1 s=THROTTLE_DONE pgs=1437 cs=0 l=1 rx=0x7f83a8010000 tx=0x7f83a8008d20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:41.874 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:41.859+0000 7f83b37fe700 1 -- 172.21.15.42:0/1603430405 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f83a80139f0 con 0x7f83a4003420 2019-12-05T05:45:41.874 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:41.859+0000 7f83b8b67700 1 --2- 172.21.15.42:0/1603430405 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f83a4003420 0x7f83a4003880 secure :-1 s=THROTTLE_DONE pgs=1437 cs=0 l=1 rx=0x7f83a8010000 tx=0x7f83a8008d20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:41.875 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:41.859+0000 7f83b37fe700 1 -- 172.21.15.42:0/1603430405 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83a8013610 con 0x7f83a4003420 2019-12-05T05:45:41.875 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:41.859+0000 7f83b8b67700 1 --2- 172.21.15.42:0/1603430405 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f83a4003420 0x7f83a4003880 secure :-1 s=THROTTLE_DONE pgs=1437 cs=0 l=1 rx=0x7f83a8010000 tx=0x7f83a8008d20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:41.875 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:41.859+0000 7f83b37fe700 1 -- 172.21.15.42:0/1603430405 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f83a8012530 con 0x7f83a4003420 2019-12-05T05:45:41.875 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:41.859+0000 7f83ba5ca700 1 -- 172.21.15.42:0/1603430405 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f83a4003420 msgr2=0x7f83a4003880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:45:41.875 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:41.859+0000 7f83ba5ca700 1 --2- 172.21.15.42:0/1603430405 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f83a4003420 0x7f83a4003880 secure :-1 s=READY pgs=1437 cs=0 l=1 rx=0x7f83a8010000 tx=0x7f83a8008d20).stop 2019-12-05T05:45:41.875 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:41.859+0000 7f83b9368700 1 -- 172.21.15.42:0/1603430405 reap_dead start 2019-12-05T05:45:41.876 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:41.859+0000 7f83ba5ca700 1 -- 172.21.15.42:0/1603430405 shutdown_connections 2019-12-05T05:45:41.876 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:41.859+0000 7f83ba5ca700 1 --2- 172.21.15.42:0/1603430405 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f83a4003420 0x7f83a4003880 unknown :-1 s=CLOSED pgs=1437 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:45:41.876 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:41.859+0000 7f83ba5ca700 1 -- 172.21.15.42:0/1603430405 shutdown_connections 2019-12-05T05:45:41.876 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:41.859+0000 7f83ba5ca700 1 -- 172.21.15.42:0/1603430405 wait complete. 2019-12-05T05:45:41.876 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:41.863+0000 7f83ba5ca700 1 -- 172.21.15.42:0/1603430405 >> 172.21.15.42:0/1603430405 conn(0x7f83b40aabe0 msgr2=0x7f83b40a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:45:41.880 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:41.863+0000 7f83ba5ca700 1 Processor -- start 2019-12-05T05:45:41.880 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:41.863+0000 7f83ba5ca700 1 -- start start 2019-12-05T05:45:41.880 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:41.867+0000 7f83ba5ca700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f83b4147b90 0x7f83b40a51e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:45:41.880 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:41.867+0000 7f83ba5ca700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f83b412e6b0 0x7f83b412eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:45:41.880 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:41.867+0000 7f83ba5ca700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83b4119460 con 0x7f83b4144960 2019-12-05T05:45:41.881 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:41.867+0000 7f83ba5ca700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83b4147fc0 con 0x7f83b4147b90 2019-12-05T05:45:41.881 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:41.867+0000 7f83ba5ca700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83b411ccc0 con 0x7f83b412e6b0 2019-12-05T05:45:41.881 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:41.867+0000 7f83b8b67700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f83b4147b90 0x7f83b40a51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:45:41.881 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:41.867+0000 7f83b3fff700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f83b412e6b0 0x7f83b412eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:45:41.881 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:41.867+0000 7f83b9368700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f83b4144960 0x7f83b408eb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:50128/0 (socket says 172.21.15.42:50128) 2019-12-05T05:45:41.881 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:41.867+0000 7f83b3fff700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f83b412e6b0 0x7f83b412eae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:41112/0 (socket says 172.21.15.42:41112) 2019-12-05T05:45:41.886 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:41.867+0000 7f83b9368700 1 -- 172.21.15.42:0/496679460 learned_addr learned my addr 172.21.15.42:0/496679460 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:45:41.886 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:41.867+0000 7f83b3fff700 1 -- 172.21.15.42:0/496679460 >> v2:172.21.15.40:3300/0 conn(0x7f83b4147b90 msgr2=0x7f83b40a51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:45:41.886 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:41.867+0000 7f83b3fff700 1 --2- 172.21.15.42:0/496679460 >> v2:172.21.15.40:3300/0 conn(0x7f83b4147b90 0x7f83b40a51e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:45:41.886 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:41.867+0000 7f83b3fff700 1 -- 172.21.15.42:0/496679460 >> v1:172.21.15.40:6789/0 conn(0x7f83b4144960 legacy=0x7f83b408eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:45:41.886 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:41.867+0000 7f83b3fff700 1 -- 172.21.15.42:0/496679460 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83b411d200 con 0x7f83b412e6b0 2019-12-05T05:45:41.887 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:41.867+0000 7f83b8b67700 1 --2- 172.21.15.42:0/496679460 >> v2:172.21.15.40:3300/0 conn(0x7f83b4147b90 0x7f83b40a51e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:45:41.887 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:41.867+0000 7f83b3fff700 1 --2- 172.21.15.42:0/496679460 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f83b412e6b0 0x7f83b412eae0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f83a0001000 tx=0x7f83a0010be0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:41.887 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:41.867+0000 7f83b3fff700 1 --2- 172.21.15.42:0/496679460 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f83b412e6b0 0x7f83b412eae0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f83a0001000 tx=0x7f83a0010be0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:41.887 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:41.867+0000 7f83b3fff700 1 --2- 172.21.15.42:0/496679460 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f83b412e6b0 0x7f83b412eae0 secure :-1 s=READY pgs=317 cs=0 l=1 rx=0x7f83a0001000 tx=0x7f83a0010be0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:45:41.887 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:41.867+0000 7f83b3fff700 1 --2- 172.21.15.42:0/496679460 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f83b412e6b0 0x7f83b412eae0 secure :-1 s=THROTTLE_DONE pgs=317 cs=0 l=1 rx=0x7f83a0001000 tx=0x7f83a0010be0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:41.887 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:41.867+0000 7f83b3fff700 1 --2- 172.21.15.42:0/496679460 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f83b412e6b0 0x7f83b412eae0 secure :-1 s=THROTTLE_DONE pgs=317 cs=0 l=1 rx=0x7f83a0001000 tx=0x7f83a0010be0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:41.887 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:41.867+0000 7f83b17fa700 1 -- 172.21.15.42:0/496679460 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f83a000caa0 con 0x7f83b412e6b0 2019-12-05T05:45:41.888 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:41.867+0000 7f83b3fff700 1 --2- 172.21.15.42:0/496679460 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f83b412e6b0 0x7f83b412eae0 secure :-1 s=THROTTLE_DONE pgs=317 cs=0 l=1 rx=0x7f83a0001000 tx=0x7f83a0010be0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:41.888 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:41.867+0000 7f83b17fa700 1 -- 172.21.15.42:0/496679460 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83a000c6c0 con 0x7f83b412e6b0 2019-12-05T05:45:41.888 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:41.867+0000 7f83ba5ca700 1 -- 172.21.15.42:0/496679460 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83b40a5730 con 0x7f83b412e6b0 2019-12-05T05:45:41.888 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:41.867+0000 7f83b17fa700 1 -- 172.21.15.42:0/496679460 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f83a000d6a0 con 0x7f83b412e6b0 2019-12-05T05:45:41.888 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:41.867+0000 7f83ba5ca700 1 -- 172.21.15.42:0/496679460 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83b4144d90 con 0x7f83b412e6b0 2019-12-05T05:45:41.888 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:41.871+0000 7f83b3fff700 1 --2- 172.21.15.42:0/496679460 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f83b412e6b0 0x7f83b412eae0 secure :-1 s=THROTTLE_DONE pgs=317 cs=0 l=1 rx=0x7f83a0001000 tx=0x7f83a0010be0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:41.889 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:41.871+0000 7f83b17fa700 1 -- 172.21.15.42:0/496679460 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f83a000d9b0 con 0x7f83b412e6b0 2019-12-05T05:45:41.889 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:41.871+0000 7f83b3fff700 1 --2- 172.21.15.42:0/496679460 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f83b412e6b0 0x7f83b412eae0 secure :-1 s=THROTTLE_DONE pgs=317 cs=0 l=1 rx=0x7f83a0001000 tx=0x7f83a0010be0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:41.889 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:41.871+0000 7f83b17fa700 1 --2- 172.21.15.42:0/496679460 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f839403ad80 0x7f839403d290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:45:41.889 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:41.871+0000 7f83b17fa700 1 -- 172.21.15.42:0/496679460 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 8525+0+0 (secure 0 0 0) 0x7f83a004b630 con 0x7f83b412e6b0 2019-12-05T05:45:41.889 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:41.871+0000 7f83b8b67700 1 --2- 172.21.15.42:0/496679460 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f839403ad80 0x7f839403d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:45:41.889 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:41.871+0000 7f83b8b67700 1 --2- 172.21.15.42:0/496679460 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f839403ad80 0x7f839403d290 crc :-1 s=READY pgs=619 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:45:41.923 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:41.915+0000 7f83ba5ca700 1 -- 172.21.15.42:0/496679460 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83b405d110 con 0x7f83b412e6b0 2019-12-05T05:45:41.932 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:41.927+0000 7f83b3fff700 1 --2- 172.21.15.42:0/496679460 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f83b412e6b0 0x7f83b412eae0 secure :-1 s=THROTTLE_DONE pgs=317 cs=0 l=1 rx=0x7f83a0001000 tx=0x7f83a0010be0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:41.933 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:41.927+0000 7f83b17fa700 1 -- 172.21.15.42:0/496679460 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f83a0032670 con 0x7f83b412e6b0 2019-12-05T05:45:42.100 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:42.095+0000 7f83ba5ca700 1 -- 172.21.15.42:0/496679460 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f83b405b8c0 con 0x7f83b412e6b0 2019-12-05T05:45:42.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:42.095+0000 7f83b3fff700 1 --2- 172.21.15.42:0/496679460 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f83b412e6b0 0x7f83b412eae0 secure :-1 s=THROTTLE_DONE pgs=317 cs=0 l=1 rx=0x7f83a0001000 tx=0x7f83a0010be0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:42.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:42.095+0000 7f83b17fa700 1 -- 172.21.15.42:0/496679460 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 175 v175) v1 ==== 95+0+1778 (secure 0 0 0) 0x7f83a00328d0 con 0x7f83b412e6b0 2019-12-05T05:45:42.103 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 175 2019-12-05T05:45:42.103 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:45:42.104 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":175,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":175,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:45:41.652957+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":78,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6077},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6077":{"gid":6077,"name":"a","rank":0,"incarnation":174,"state":"up:reconnect","state_seq":14,"addr":"172.21.15.42:6835/3757738536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3757738536},{"type":"v1","addr":"172.21.15.42:6835","nonce":3757738536}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:45:42.106 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:42.099+0000 7f83ba5ca700 1 -- 172.21.15.42:0/496679460 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f839403ad80 msgr2=0x7f839403d290 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:45:42.106 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:42.099+0000 7f83ba5ca700 1 --2- 172.21.15.42:0/496679460 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f839403ad80 0x7f839403d290 crc :-1 s=READY pgs=619 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:45:42.106 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:42.099+0000 7f83ba5ca700 1 -- 172.21.15.42:0/496679460 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f83b412e6b0 msgr2=0x7f83b412eae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:45:42.106 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:42.099+0000 7f83ba5ca700 1 --2- 172.21.15.42:0/496679460 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f83b412e6b0 0x7f83b412eae0 secure :-1 s=READY pgs=317 cs=0 l=1 rx=0x7f83a0001000 tx=0x7f83a0010be0).stop 2019-12-05T05:45:42.107 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:42.099+0000 7f83ba5ca700 1 -- 172.21.15.42:0/496679460 shutdown_connections 2019-12-05T05:45:42.107 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:42.099+0000 7f83ba5ca700 1 --2- 172.21.15.42:0/496679460 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f839403ad80 0x7f839403d290 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:45:42.107 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:42.099+0000 7f83ba5ca700 1 --2- 172.21.15.42:0/496679460 >> v2:172.21.15.40:3300/0 conn(0x7f83b4147b90 0x7f83b40a51e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:45:42.107 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:42.099+0000 7f83ba5ca700 1 --2- 172.21.15.42:0/496679460 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f83b412e6b0 0x7f83b412eae0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:45:42.107 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:42.099+0000 7f83ba5ca700 1 -- 172.21.15.42:0/496679460 shutdown_connections 2019-12-05T05:45:42.107 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:42.099+0000 7f83ba5ca700 1 -- 172.21.15.42:0/496679460 wait complete. 2019-12-05T05:45:42.107 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:42.099+0000 7f83ba5ca700 1 -- 172.21.15.42:0/496679460 >> 172.21.15.42:0/496679460 conn(0x7f83b408fde0 msgr2=0x7f83b40a9790 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:45:42.119 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2019-12-05T05:45:44.120 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:45:44.227 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:44.215+0000 7f3e0f16a700 1 Processor -- start 2019-12-05T05:45:44.227 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:44.215+0000 7f3e0f16a700 1 -- start start 2019-12-05T05:45:44.227 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:44.219+0000 7f3e0f16a700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3e0808df80 0x7f3e0809f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:45:44.228 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:44.219+0000 7f3e0f16a700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f3e0809f830 0x7f3e0814efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:45:44.228 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:44.219+0000 7f3e0f16a700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e0809fc60 con 0x7f3e080a7740 2019-12-05T05:45:44.228 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:44.219+0000 7f3e0f16a700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e080a0b00 con 0x7f3e0809f830 2019-12-05T05:45:44.228 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:44.219+0000 7f3e0f16a700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e0808e5b0 con 0x7f3e0808df80 2019-12-05T05:45:44.228 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:44.219+0000 7f3e0d707700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f3e080a7740 0x7f3e0807e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:40698/0 (socket says 172.21.15.42:40698) 2019-12-05T05:45:44.228 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:44.219+0000 7f3e0d707700 1 -- 172.21.15.42:0/3899818430 learned_addr learned my addr 172.21.15.42:0/3899818430 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:45:44.229 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:44.219+0000 7f3e0cf06700 1 --2- 172.21.15.42:0/3899818430 >> v2:172.21.15.42:3300/0 conn(0x7f3e0808df80 0x7f3e0809f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:45:44.229 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:44.219+0000 7f3e0df08700 1 --2- 172.21.15.42:0/3899818430 >> v2:172.21.15.40:3300/0 conn(0x7f3e0809f830 0x7f3e0814efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:45:44.229 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:44.219+0000 7f3dfffff700 1 -- 172.21.15.42:0/3899818430 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1797361338 0 0) 0x7f3e0809fc60 con 0x7f3e080a7740 2019-12-05T05:45:44.229 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:44.219+0000 7f3dfffff700 1 -- 172.21.15.42:0/3899818430 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3df40030b0 con 0x7f3e080a7740 2019-12-05T05:45:44.229 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:44.219+0000 7f3e0cf06700 1 -- 172.21.15.42:0/3899818430 >> v2:172.21.15.40:3300/0 conn(0x7f3e0809f830 msgr2=0x7f3e0814efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:45:44.229 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:44.219+0000 7f3e0cf06700 1 --2- 172.21.15.42:0/3899818430 >> v2:172.21.15.40:3300/0 conn(0x7f3e0809f830 0x7f3e0814efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:45:44.229 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:44.219+0000 7f3e0cf06700 1 -- 172.21.15.42:0/3899818430 >> v1:172.21.15.42:6789/0 conn(0x7f3e080a7740 legacy=0x7f3e0807e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:45:44.229 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:44.219+0000 7f3e0cf06700 1 -- 172.21.15.42:0/3899818430 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e080a0710 con 0x7f3e0808df80 2019-12-05T05:45:44.230 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:44.219+0000 7f3e0cf06700 1 --2- 172.21.15.42:0/3899818430 >> v2:172.21.15.42:3300/0 conn(0x7f3e0808df80 0x7f3e0809f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f3df8002f30 tx=0x7f3df800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:44.230 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:44.219+0000 7f3e0cf06700 1 --2- 172.21.15.42:0/3899818430 >> v2:172.21.15.42:3300/0 conn(0x7f3e0808df80 0x7f3e0809f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f3df8002f30 tx=0x7f3df800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:44.230 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:44.219+0000 7f3e0cf06700 1 --2- 172.21.15.42:0/3899818430 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3e0808df80 0x7f3e0809f2e0 secure :-1 s=READY pgs=1439 cs=0 l=1 rx=0x7f3df8002f30 tx=0x7f3df800fc70).ready entity=mon.1 client_cookie=260a764bd98f7ea4 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:45:44.230 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:44.219+0000 7f3e0cf06700 1 --2- 172.21.15.42:0/3899818430 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3e0808df80 0x7f3e0809f2e0 secure :-1 s=THROTTLE_DONE pgs=1439 cs=0 l=1 rx=0x7f3df8002f30 tx=0x7f3df800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:44.230 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:44.219+0000 7f3dfffff700 1 -- 172.21.15.42:0/3899818430 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3df8008bb0 con 0x7f3e0808df80 2019-12-05T05:45:44.230 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:44.219+0000 7f3e0cf06700 1 --2- 172.21.15.42:0/3899818430 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3e0808df80 0x7f3e0809f2e0 secure :-1 s=THROTTLE_DONE pgs=1439 cs=0 l=1 rx=0x7f3df8002f30 tx=0x7f3df800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:44.231 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:44.219+0000 7f3e0cf06700 1 --2- 172.21.15.42:0/3899818430 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3e0808df80 0x7f3e0809f2e0 secure :-1 s=THROTTLE_DONE pgs=1439 cs=0 l=1 rx=0x7f3df8002f30 tx=0x7f3df800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:44.231 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:44.219+0000 7f3dfffff700 1 -- 172.21.15.42:0/3899818430 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3df80087d0 con 0x7f3e0808df80 2019-12-05T05:45:44.231 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:44.219+0000 7f3dfffff700 1 -- 172.21.15.42:0/3899818430 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3df8011820 con 0x7f3e0808df80 2019-12-05T05:45:44.231 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:44.219+0000 7f3e0f16a700 1 -- 172.21.15.42:0/3899818430 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3e0808df80 msgr2=0x7f3e0809f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:45:44.231 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:44.219+0000 7f3e0f16a700 1 --2- 172.21.15.42:0/3899818430 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3e0808df80 0x7f3e0809f2e0 secure :-1 s=READY pgs=1439 cs=0 l=1 rx=0x7f3df8002f30 tx=0x7f3df800fc70).stop 2019-12-05T05:45:44.231 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:44.219+0000 7f3e0f16a700 1 -- 172.21.15.42:0/3899818430 shutdown_connections 2019-12-05T05:45:44.231 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:44.219+0000 7f3e0f16a700 1 --2- 172.21.15.42:0/3899818430 >> v2:172.21.15.40:3300/0 conn(0x7f3e0809f830 0x7f3e0814efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:45:44.231 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:44.219+0000 7f3e0f16a700 1 --2- 172.21.15.42:0/3899818430 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3e0808df80 0x7f3e0809f2e0 unknown :-1 s=CLOSED pgs=1439 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:45:44.232 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:44.219+0000 7f3e0f16a700 1 -- 172.21.15.42:0/3899818430 shutdown_connections 2019-12-05T05:45:44.232 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:44.219+0000 7f3e0f16a700 1 -- 172.21.15.42:0/3899818430 wait complete. 2019-12-05T05:45:44.232 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:44.219+0000 7f3e0f16a700 1 -- 172.21.15.42:0/3899818430 >> 172.21.15.42:0/3899818430 conn(0x7f3e080aabe0 msgr2=0x7f3e080a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:45:44.236 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:44.223+0000 7f3e0f16a700 1 Processor -- start 2019-12-05T05:45:44.236 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:44.223+0000 7f3e0f16a700 1 -- start start 2019-12-05T05:45:44.236 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:44.223+0000 7f3e0f16a700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f3e08117950 0x7f3e0811b960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:45:44.236 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:44.223+0000 7f3e0f16a700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3e08121f90 0x7f3e08118ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:45:44.237 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:44.223+0000 7f3e0f16a700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3e08132660 0x7f3e08132a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:45:44.237 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:44.223+0000 7f3e0f16a700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e0811c0e0 con 0x7f3e08117950 2019-12-05T05:45:44.237 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:44.223+0000 7f3e0f16a700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e08118a40 con 0x7f3e08132660 2019-12-05T05:45:44.237 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:44.223+0000 7f3e0f16a700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e081223c0 con 0x7f3e08121f90 2019-12-05T05:45:44.237 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:44.223+0000 7f3e0df08700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3e08132660 0x7f3e08132a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:45:44.237 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:44.223+0000 7f3e0d707700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f3e08117950 0x7f3e0811b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:45:44.237 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:44.223+0000 7f3e0df08700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3e08132660 0x7f3e08132a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:40670/0 (socket says 172.21.15.42:40670) 2019-12-05T05:45:44.238 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:44.223+0000 7f3e0df08700 1 -- 172.21.15.42:0/600989081 learned_addr learned my addr 172.21.15.42:0/600989081 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:45:44.238 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:44.223+0000 7f3e0cf06700 1 --2- 172.21.15.42:0/600989081 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3e08121f90 0x7f3e08118ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:45:44.238 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:44.223+0000 7f3e0df08700 1 -- 172.21.15.42:0/600989081 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3e08121f90 msgr2=0x7f3e08118ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:45:44.238 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:44.223+0000 7f3e0df08700 1 --2- 172.21.15.42:0/600989081 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3e08121f90 0x7f3e08118ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:45:44.238 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:44.223+0000 7f3e0df08700 1 -- 172.21.15.42:0/600989081 >> v2:172.21.15.40:3300/0 conn(0x7f3e08117950 msgr2=0x7f3e0811b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:45:44.238 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:44.223+0000 7f3e0df08700 1 --2- 172.21.15.42:0/600989081 >> v2:172.21.15.40:3300/0 conn(0x7f3e08117950 0x7f3e0811b960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:45:44.238 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:44.223+0000 7f3e0df08700 1 -- 172.21.15.42:0/600989081 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e08119230 con 0x7f3e08132660 2019-12-05T05:45:44.239 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:44.223+0000 7f3e0df08700 1 --2- 172.21.15.42:0/600989081 >> v2:172.21.15.42:3300/0 conn(0x7f3e08132660 0x7f3e08132a90 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f3e000032b0 tx=0x7f3e00010d90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:44.239 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:44.223+0000 7f3e0df08700 1 --2- 172.21.15.42:0/600989081 >> v2:172.21.15.42:3300/0 conn(0x7f3e08132660 0x7f3e08132a90 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f3e000032b0 tx=0x7f3e00010d90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:44.239 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:44.223+0000 7f3e0df08700 1 --2- 172.21.15.42:0/600989081 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3e08132660 0x7f3e08132a90 secure :-1 s=READY pgs=1440 cs=0 l=1 rx=0x7f3e000032b0 tx=0x7f3e00010d90).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:45:44.239 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:44.223+0000 7f3e0df08700 1 --2- 172.21.15.42:0/600989081 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3e08132660 0x7f3e08132a90 secure :-1 s=THROTTLE_DONE pgs=1440 cs=0 l=1 rx=0x7f3e000032b0 tx=0x7f3e00010d90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:44.239 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:44.223+0000 7f3e0df08700 1 --2- 172.21.15.42:0/600989081 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3e08132660 0x7f3e08132a90 secure :-1 s=THROTTLE_DONE pgs=1440 cs=0 l=1 rx=0x7f3e000032b0 tx=0x7f3e00010d90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:44.239 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:44.223+0000 7f3dfdffb700 1 -- 172.21.15.42:0/600989081 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3e00005d40 con 0x7f3e08132660 2019-12-05T05:45:44.240 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:44.223+0000 7f3e0df08700 1 --2- 172.21.15.42:0/600989081 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3e08132660 0x7f3e08132a90 secure :-1 s=THROTTLE_DONE pgs=1440 cs=0 l=1 rx=0x7f3e000032b0 tx=0x7f3e00010d90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:44.240 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:44.223+0000 7f3dfdffb700 1 -- 172.21.15.42:0/600989081 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e00005960 con 0x7f3e08132660 2019-12-05T05:45:44.240 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:44.223+0000 7f3dfdffb700 1 -- 172.21.15.42:0/600989081 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3e00012ca0 con 0x7f3e08132660 2019-12-05T05:45:44.240 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:44.223+0000 7f3e0f16a700 1 -- 172.21.15.42:0/600989081 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e08132fe0 con 0x7f3e08132660 2019-12-05T05:45:44.240 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:44.223+0000 7f3e0f16a700 1 -- 172.21.15.42:0/600989081 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e081333f0 con 0x7f3e08132660 2019-12-05T05:45:44.240 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:44.227+0000 7f3e0df08700 1 --2- 172.21.15.42:0/600989081 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3e08132660 0x7f3e08132a90 secure :-1 s=THROTTLE_DONE pgs=1440 cs=0 l=1 rx=0x7f3e000032b0 tx=0x7f3e00010d90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:44.240 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:44.227+0000 7f3dfdffb700 1 -- 172.21.15.42:0/600989081 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f3e0000c3f0 con 0x7f3e08132660 2019-12-05T05:45:44.241 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:44.227+0000 7f3e0df08700 1 --2- 172.21.15.42:0/600989081 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3e08132660 0x7f3e08132a90 secure :-1 s=THROTTLE_DONE pgs=1440 cs=0 l=1 rx=0x7f3e000032b0 tx=0x7f3e00010d90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:44.241 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:44.227+0000 7f3dfdffb700 1 --2- 172.21.15.42:0/600989081 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3de803ae00 0x7f3de803d310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:45:44.241 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:44.227+0000 7f3dfdffb700 1 -- 172.21.15.42:0/600989081 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 8525+0+0 (secure 0 0 0) 0x7f3e0004b750 con 0x7f3e08132660 2019-12-05T05:45:44.241 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:44.227+0000 7f3e0d707700 1 --2- 172.21.15.42:0/600989081 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3de803ae00 0x7f3de803d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:45:44.241 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:44.227+0000 7f3e0d707700 1 --2- 172.21.15.42:0/600989081 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3de803ae00 0x7f3de803d310 crc :-1 s=READY pgs=620 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:45:44.242 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:44.231+0000 7f3e0f16a700 1 -- 172.21.15.42:0/600989081 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e0805d110 con 0x7f3e08132660 2019-12-05T05:45:44.243 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:44.235+0000 7f3e0df08700 1 --2- 172.21.15.42:0/600989081 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3e08132660 0x7f3e08132a90 secure :-1 s=THROTTLE_DONE pgs=1440 cs=0 l=1 rx=0x7f3e000032b0 tx=0x7f3e00010d90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:44.243 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:44.235+0000 7f3dfdffb700 1 -- 172.21.15.42:0/600989081 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f3e000327d0 con 0x7f3e08132660 2019-12-05T05:45:44.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:44.399+0000 7f3e0f16a700 1 -- 172.21.15.42:0/600989081 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3e0805b8c0 con 0x7f3e08132660 2019-12-05T05:45:44.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:44.399+0000 7f3e0df08700 1 --2- 172.21.15.42:0/600989081 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3e08132660 0x7f3e08132a90 secure :-1 s=THROTTLE_DONE pgs=1440 cs=0 l=1 rx=0x7f3e000032b0 tx=0x7f3e00010d90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:44.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:44.399+0000 7f3dfdffb700 1 -- 172.21.15.42:0/600989081 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 177 v177) v1 ==== 95+0+1775 (secure 0 0 0) 0x7f3e00032a30 con 0x7f3e08132660 2019-12-05T05:45:44.405 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 177 2019-12-05T05:45:44.405 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:45:44.405 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":177,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":177,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:45:43.664230+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":78,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6077},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6077":{"gid":6077,"name":"a","rank":0,"incarnation":174,"state":"up:active","state_seq":16,"addr":"172.21.15.42:6835/3757738536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3757738536},{"type":"v1","addr":"172.21.15.42:6835","nonce":3757738536}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:45:44.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:44.399+0000 7f3e0f16a700 1 -- 172.21.15.42:0/600989081 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3de803ae00 msgr2=0x7f3de803d310 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:45:44.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:44.399+0000 7f3e0f16a700 1 --2- 172.21.15.42:0/600989081 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3de803ae00 0x7f3de803d310 crc :-1 s=READY pgs=620 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:45:44.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:44.399+0000 7f3e0f16a700 1 -- 172.21.15.42:0/600989081 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3e08132660 msgr2=0x7f3e08132a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:45:44.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:44.399+0000 7f3e0f16a700 1 --2- 172.21.15.42:0/600989081 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3e08132660 0x7f3e08132a90 secure :-1 s=READY pgs=1440 cs=0 l=1 rx=0x7f3e000032b0 tx=0x7f3e00010d90).stop 2019-12-05T05:45:44.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:44.399+0000 7f3e0f16a700 1 -- 172.21.15.42:0/600989081 shutdown_connections 2019-12-05T05:45:44.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:44.399+0000 7f3e0f16a700 1 --2- 172.21.15.42:0/600989081 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3de803ae00 0x7f3de803d310 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:45:44.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:44.399+0000 7f3e0f16a700 1 --2- 172.21.15.42:0/600989081 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3e08132660 0x7f3e08132a90 unknown :-1 s=CLOSED pgs=1440 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:45:44.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:44.399+0000 7f3e0f16a700 1 --2- 172.21.15.42:0/600989081 >> v2:172.21.15.40:3300/0 conn(0x7f3e08117950 0x7f3e0811b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:45:44.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:44.399+0000 7f3e0f16a700 1 --2- 172.21.15.42:0/600989081 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3e08121f90 0x7f3e08118ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:45:44.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:44.399+0000 7f3e0f16a700 1 -- 172.21.15.42:0/600989081 shutdown_connections 2019-12-05T05:45:44.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:44.399+0000 7f3e0f16a700 1 -- 172.21.15.42:0/600989081 wait complete. 2019-12-05T05:45:44.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:44.399+0000 7f3e0f16a700 1 -- 172.21.15.42:0/600989081 >> 172.21.15.42:0/600989081 conn(0x7f3e080a82a0 msgr2=0x7f3e0807ea30 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:45:44.417 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a has gained rank=0, replacing gid=7884 2019-12-05T05:45:44.417 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 3 secs before reviving mds.b 2019-12-05T05:45:47.418 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.b 2019-12-05T05:45:47.418 INFO:tasks.ceph.mds.b:Restarting daemon 2019-12-05T05:45:47.418 INFO:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2019-12-05T05:45:47.422 INFO:tasks.ceph.mds.b:Started 2019-12-05T05:45:47.422 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:45:47.482 INFO:tasks.ceph.mds.b.smithi040.stdout:starting mds.b at 2019-12-05T05:45:47.524 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:47.515+0000 7fe94be53700 1 Processor -- start 2019-12-05T05:45:47.524 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:47.515+0000 7fe94be53700 1 -- start start 2019-12-05T05:45:47.524 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:47.515+0000 7fe94be53700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe9440a7730 0x7fe94408df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:45:47.525 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:47.515+0000 7fe94be53700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe94407e710 0x7fe94407eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:45:47.525 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:47.515+0000 7fe94be53700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe94409f2d0 0x7fe94409f700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:45:47.525 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:47.515+0000 7fe94be53700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9440a0af0 con 0x7fe94409f2d0 2019-12-05T05:45:47.525 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:47.515+0000 7fe94be53700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe94408e5a0 con 0x7fe9440a7730 2019-12-05T05:45:47.525 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:47.515+0000 7fe94be53700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe94408e840 con 0x7fe94407e710 2019-12-05T05:45:47.525 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:47.515+0000 7fe94a3f0700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe9440a7730 0x7fe94408df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:45:47.525 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:47.515+0000 7fe94a3f0700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe9440a7730 0x7fe94408df70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:40686/0 (socket says 172.21.15.42:40686) 2019-12-05T05:45:47.526 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:47.515+0000 7fe94a3f0700 1 -- 172.21.15.42:0/2551300959 learned_addr learned my addr 172.21.15.42:0/2551300959 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:45:47.526 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:47.515+0000 7fe949bef700 1 --2- 172.21.15.42:0/2551300959 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe94407e710 0x7fe94407eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:45:47.526 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:47.515+0000 7fe94abf1700 1 --2- 172.21.15.42:0/2551300959 >> v2:172.21.15.40:3300/0 conn(0x7fe94409f2d0 0x7fe94409f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:45:47.526 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:47.515+0000 7fe94a3f0700 1 -- 172.21.15.42:0/2551300959 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe94407e710 msgr2=0x7fe94407eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:45:47.526 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:47.515+0000 7fe94a3f0700 1 --2- 172.21.15.42:0/2551300959 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe94407e710 0x7fe94407eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:45:47.527 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:47.515+0000 7fe94a3f0700 1 -- 172.21.15.42:0/2551300959 >> v2:172.21.15.40:3300/0 conn(0x7fe94409f2d0 msgr2=0x7fe94409f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:45:47.527 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:47.515+0000 7fe94a3f0700 1 --2- 172.21.15.42:0/2551300959 >> v2:172.21.15.40:3300/0 conn(0x7fe94409f2d0 0x7fe94409f700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:45:47.527 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:47.515+0000 7fe94a3f0700 1 -- 172.21.15.42:0/2551300959 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe94407f090 con 0x7fe9440a7730 2019-12-05T05:45:47.527 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:47.515+0000 7fe94a3f0700 1 --2- 172.21.15.42:0/2551300959 >> v2:172.21.15.42:3300/0 conn(0x7fe9440a7730 0x7fe94408df70 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fe94408e760 tx=0x7fe94000fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:47.527 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:47.515+0000 7fe94a3f0700 1 --2- 172.21.15.42:0/2551300959 >> v2:172.21.15.42:3300/0 conn(0x7fe9440a7730 0x7fe94408df70 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fe94408e760 tx=0x7fe94000fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:47.528 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:47.515+0000 7fe94a3f0700 1 --2- 172.21.15.42:0/2551300959 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe9440a7730 0x7fe94408df70 secure :-1 s=READY pgs=1442 cs=0 l=1 rx=0x7fe94408e760 tx=0x7fe94000fdb0).ready entity=mon.1 client_cookie=f964b95da1fc8120 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:45:47.528 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:47.515+0000 7fe94a3f0700 1 --2- 172.21.15.42:0/2551300959 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe9440a7730 0x7fe94408df70 secure :-1 s=THROTTLE_DONE pgs=1442 cs=0 l=1 rx=0x7fe94408e760 tx=0x7fe94000fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:47.528 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:47.515+0000 7fe9493ee700 1 -- 172.21.15.42:0/2551300959 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe940008c90 con 0x7fe9440a7730 2019-12-05T05:45:47.528 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:47.515+0000 7fe94a3f0700 1 --2- 172.21.15.42:0/2551300959 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe9440a7730 0x7fe94408df70 secure :-1 s=THROTTLE_DONE pgs=1442 cs=0 l=1 rx=0x7fe94408e760 tx=0x7fe94000fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:47.528 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:47.515+0000 7fe94a3f0700 1 --2- 172.21.15.42:0/2551300959 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe9440a7730 0x7fe94408df70 secure :-1 s=THROTTLE_DONE pgs=1442 cs=0 l=1 rx=0x7fe94408e760 tx=0x7fe94000fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:47.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:47.515+0000 7fe9493ee700 1 -- 172.21.15.42:0/2551300959 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9400088b0 con 0x7fe9440a7730 2019-12-05T05:45:47.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:47.515+0000 7fe9493ee700 1 -- 172.21.15.42:0/2551300959 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe940011b50 con 0x7fe9440a7730 2019-12-05T05:45:47.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:47.515+0000 7fe94be53700 1 -- 172.21.15.42:0/2551300959 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe9440a7730 msgr2=0x7fe94408df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:45:47.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:47.515+0000 7fe94be53700 1 --2- 172.21.15.42:0/2551300959 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe9440a7730 0x7fe94408df70 secure :-1 s=READY pgs=1442 cs=0 l=1 rx=0x7fe94408e760 tx=0x7fe94000fdb0).stop 2019-12-05T05:45:47.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:47.515+0000 7fe94be53700 1 -- 172.21.15.42:0/2551300959 shutdown_connections 2019-12-05T05:45:47.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:47.515+0000 7fe94be53700 1 --2- 172.21.15.42:0/2551300959 >> v2:172.21.15.40:3300/0 conn(0x7fe94409f2d0 0x7fe94409f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:45:47.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:47.515+0000 7fe94be53700 1 --2- 172.21.15.42:0/2551300959 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe9440a7730 0x7fe94408df70 unknown :-1 s=CLOSED pgs=1442 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:45:47.530 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:47.515+0000 7fe94be53700 1 --2- 172.21.15.42:0/2551300959 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe94407e710 0x7fe94407eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:45:47.530 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:47.515+0000 7fe94be53700 1 -- 172.21.15.42:0/2551300959 shutdown_connections 2019-12-05T05:45:47.530 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:47.515+0000 7fe94be53700 1 -- 172.21.15.42:0/2551300959 wait complete. 2019-12-05T05:45:47.530 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:47.515+0000 7fe94be53700 1 -- 172.21.15.42:0/2551300959 >> 172.21.15.42:0/2551300959 conn(0x7fe9440aabd0 msgr2=0x7fe9440a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:45:47.534 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:47.519+0000 7fe94be53700 1 Processor -- start 2019-12-05T05:45:47.534 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:47.519+0000 7fe94be53700 1 -- start start 2019-12-05T05:45:47.534 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:47.519+0000 7fe94be53700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe94411a0f0 0x7fe94407da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:45:47.534 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:47.519+0000 7fe94be53700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe944135640 0x7fe944137a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:45:47.534 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:47.519+0000 7fe94be53700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe94411be90 con 0x7fe94411b930 2019-12-05T05:45:47.535 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:47.519+0000 7fe94be53700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe944123a30 con 0x7fe944135640 2019-12-05T05:45:47.535 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:47.519+0000 7fe94be53700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe944130200 con 0x7fe94411a0f0 2019-12-05T05:45:47.535 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:47.519+0000 7fe94a3f0700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe94411a0f0 0x7fe94407da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:45:47.535 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:47.519+0000 7fe94a3f0700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe94411a0f0 0x7fe94407da50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:40692/0 (socket says 172.21.15.42:40692) 2019-12-05T05:45:47.536 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:47.519+0000 7fe94a3f0700 1 -- 172.21.15.42:0/1734334489 learned_addr learned my addr 172.21.15.42:0/1734334489 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:45:47.536 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:47.519+0000 7fe94abf1700 1 --2- 172.21.15.42:0/1734334489 >> v2:172.21.15.40:3300/0 conn(0x7fe944135640 0x7fe944137a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:45:47.536 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:47.519+0000 7fe94a3f0700 1 -- 172.21.15.42:0/1734334489 >> v2:172.21.15.40:3300/0 conn(0x7fe944135640 msgr2=0x7fe944137a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:45:47.536 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:47.519+0000 7fe94a3f0700 1 --2- 172.21.15.42:0/1734334489 >> v2:172.21.15.40:3300/0 conn(0x7fe944135640 0x7fe944137a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:45:47.536 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:47.519+0000 7fe94a3f0700 1 -- 172.21.15.42:0/1734334489 >> v1:172.21.15.40:6789/0 conn(0x7fe94411b930 legacy=0x7fe944123c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:45:47.536 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:47.519+0000 7fe94a3f0700 1 -- 172.21.15.42:0/1734334489 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe944124490 con 0x7fe94411a0f0 2019-12-05T05:45:47.536 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:47.519+0000 7fe94a3f0700 1 --2- 172.21.15.42:0/1734334489 >> v2:172.21.15.42:3300/0 conn(0x7fe94411a0f0 0x7fe94407da50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fe94408e760 tx=0x7fe9400075e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:47.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:47.519+0000 7fe94a3f0700 1 --2- 172.21.15.42:0/1734334489 >> v2:172.21.15.42:3300/0 conn(0x7fe94411a0f0 0x7fe94407da50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fe94408e760 tx=0x7fe9400075e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:47.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:47.523+0000 7fe94a3f0700 1 --2- 172.21.15.42:0/1734334489 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe94411a0f0 0x7fe94407da50 secure :-1 s=READY pgs=1443 cs=0 l=1 rx=0x7fe94408e760 tx=0x7fe9400075e0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:45:47.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:47.523+0000 7fe94a3f0700 1 --2- 172.21.15.42:0/1734334489 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe94411a0f0 0x7fe94407da50 secure :-1 s=THROTTLE_DONE pgs=1443 cs=0 l=1 rx=0x7fe94408e760 tx=0x7fe9400075e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:47.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:47.523+0000 7fe94a3f0700 1 --2- 172.21.15.42:0/1734334489 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe94411a0f0 0x7fe94407da50 secure :-1 s=THROTTLE_DONE pgs=1443 cs=0 l=1 rx=0x7fe94408e760 tx=0x7fe9400075e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:47.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:47.523+0000 7fe94a3f0700 1 --2- 172.21.15.42:0/1734334489 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe94411a0f0 0x7fe94407da50 secure :-1 s=THROTTLE_DONE pgs=1443 cs=0 l=1 rx=0x7fe94408e760 tx=0x7fe9400075e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:47.539 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:47.523+0000 7fe93affd700 1 -- 172.21.15.42:0/1734334489 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe94000f470 con 0x7fe94411a0f0 2019-12-05T05:45:47.539 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:47.523+0000 7fe93affd700 1 -- 172.21.15.42:0/1734334489 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe940011770 con 0x7fe94411a0f0 2019-12-05T05:45:47.539 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:47.523+0000 7fe94be53700 1 -- 172.21.15.42:0/1734334489 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe944122d30 con 0x7fe94411a0f0 2019-12-05T05:45:47.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:47.523+0000 7fe93affd700 1 -- 172.21.15.42:0/1734334489 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe940005730 con 0x7fe94411a0f0 2019-12-05T05:45:47.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:47.523+0000 7fe94be53700 1 -- 172.21.15.42:0/1734334489 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe94407dfa0 con 0x7fe94411a0f0 2019-12-05T05:45:47.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:47.523+0000 7fe94a3f0700 1 --2- 172.21.15.42:0/1734334489 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe94411a0f0 0x7fe94407da50 secure :-1 s=THROTTLE_DONE pgs=1443 cs=0 l=1 rx=0x7fe94408e760 tx=0x7fe9400075e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:47.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:47.523+0000 7fe93affd700 1 -- 172.21.15.42:0/1734334489 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fe940005a00 con 0x7fe94411a0f0 2019-12-05T05:45:47.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:47.523+0000 7fe94a3f0700 1 --2- 172.21.15.42:0/1734334489 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe94411a0f0 0x7fe94407da50 secure :-1 s=THROTTLE_DONE pgs=1443 cs=0 l=1 rx=0x7fe94408e760 tx=0x7fe9400075e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:47.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:47.523+0000 7fe93affd700 1 --2- 172.21.15.42:0/1734334489 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe92c03ad50 0x7fe92c03d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:45:47.541 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:47.523+0000 7fe93affd700 1 -- 172.21.15.42:0/1734334489 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 8525+0+0 (secure 0 0 0) 0x7fe94008f370 con 0x7fe94411a0f0 2019-12-05T05:45:47.541 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:47.523+0000 7fe949bef700 1 --2- 172.21.15.42:0/1734334489 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe92c03ad50 0x7fe92c03d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:45:47.541 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:47.523+0000 7fe949bef700 1 --2- 172.21.15.42:0/1734334489 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe92c03ad50 0x7fe92c03d260 crc :-1 s=READY pgs=621 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:45:47.542 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:47.527+0000 7fe94be53700 1 -- 172.21.15.42:0/1734334489 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9441233d0 con 0x7fe94411a0f0 2019-12-05T05:45:47.543 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:47.535+0000 7fe94a3f0700 1 --2- 172.21.15.42:0/1734334489 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe94411a0f0 0x7fe94407da50 secure :-1 s=THROTTLE_DONE pgs=1443 cs=0 l=1 rx=0x7fe94408e760 tx=0x7fe9400075e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:47.543 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:47.535+0000 7fe93affd700 1 -- 172.21.15.42:0/1734334489 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fe940077240 con 0x7fe94411a0f0 2019-12-05T05:45:47.743 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:47.739+0000 7fe94be53700 1 -- 172.21.15.42:0/1734334489 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe94405d100 con 0x7fe94411a0f0 2019-12-05T05:45:47.744 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:47.739+0000 7fe94a3f0700 1 --2- 172.21.15.42:0/1734334489 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe94411a0f0 0x7fe94407da50 secure :-1 s=THROTTLE_DONE pgs=1443 cs=0 l=1 rx=0x7fe94408e760 tx=0x7fe9400075e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:47.744 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:47.739+0000 7fe93affd700 1 -- 172.21.15.42:0/1734334489 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 177 v177) v1 ==== 95+0+1775 (secure 0 0 0) 0x7fe940076950 con 0x7fe94411a0f0 2019-12-05T05:45:47.745 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 177 2019-12-05T05:45:47.745 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:45:47.745 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":177,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":177,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:45:43.664230+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":78,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6077},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6077":{"gid":6077,"name":"a","rank":0,"incarnation":174,"state":"up:active","state_seq":16,"addr":"172.21.15.42:6835/3757738536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3757738536},{"type":"v1","addr":"172.21.15.42:6835","nonce":3757738536}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:45:47.747 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:47.739+0000 7fe94be53700 1 -- 172.21.15.42:0/1734334489 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe92c03ad50 msgr2=0x7fe92c03d260 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:45:47.747 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:47.739+0000 7fe94be53700 1 --2- 172.21.15.42:0/1734334489 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe92c03ad50 0x7fe92c03d260 crc :-1 s=READY pgs=621 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:45:47.747 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:47.739+0000 7fe94be53700 1 -- 172.21.15.42:0/1734334489 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe94411a0f0 msgr2=0x7fe94407da50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:45:47.748 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:47.739+0000 7fe94be53700 1 --2- 172.21.15.42:0/1734334489 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe94411a0f0 0x7fe94407da50 secure :-1 s=READY pgs=1443 cs=0 l=1 rx=0x7fe94408e760 tx=0x7fe9400075e0).stop 2019-12-05T05:45:47.748 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:47.739+0000 7fe94be53700 1 -- 172.21.15.42:0/1734334489 shutdown_connections 2019-12-05T05:45:47.748 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:47.739+0000 7fe94be53700 1 --2- 172.21.15.42:0/1734334489 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe92c03ad50 0x7fe92c03d260 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:45:47.749 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:47.739+0000 7fe94be53700 1 --2- 172.21.15.42:0/1734334489 >> v2:172.21.15.40:3300/0 conn(0x7fe944135640 0x7fe944137a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:45:47.749 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:47.739+0000 7fe94be53700 1 --2- 172.21.15.42:0/1734334489 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe94411a0f0 0x7fe94407da50 unknown :-1 s=CLOSED pgs=1443 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:45:47.749 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:47.739+0000 7fe94be53700 1 -- 172.21.15.42:0/1734334489 shutdown_connections 2019-12-05T05:45:47.749 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:47.739+0000 7fe94be53700 1 -- 172.21.15.42:0/1734334489 wait complete. 2019-12-05T05:45:47.749 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:47.739+0000 7fe94be53700 1 -- 172.21.15.42:0/1734334489 >> 172.21.15.42:0/1734334489 conn(0x7fe9440a69d0 msgr2=0x7fe94409f3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:45:47.757 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2019-12-05T05:45:49.758 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:45:49.864 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:49.851+0000 7f6d4e2b4700 1 Processor -- start 2019-12-05T05:45:49.864 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:49.855+0000 7f6d4e2b4700 1 -- start start 2019-12-05T05:45:49.865 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:49.855+0000 7f6d4e2b4700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6d480a7740 0x7f6d4808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:45:49.865 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:49.855+0000 7f6d4e2b4700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f6d4807e720 0x7f6d4807eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:45:49.865 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:49.855+0000 7f6d4e2b4700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6d4809f2e0 0x7f6d4809f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:45:49.865 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:49.855+0000 7f6d4e2b4700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d480a0b00 con 0x7f6d4807e720 2019-12-05T05:45:49.865 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:49.855+0000 7f6d4e2b4700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d4808e5b0 con 0x7f6d4809f2e0 2019-12-05T05:45:49.865 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:49.855+0000 7f6d4e2b4700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d4808e850 con 0x7f6d480a7740 2019-12-05T05:45:49.866 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:49.855+0000 7f6d4d052700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6d4809f2e0 0x7f6d4809f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:45:49.866 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:49.855+0000 7f6d4d052700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6d4809f2e0 0x7f6d4809f710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:40734/0 (socket says 172.21.15.42:40734) 2019-12-05T05:45:49.866 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:49.855+0000 7f6d4d052700 1 -- 172.21.15.42:0/499320086 learned_addr learned my addr 172.21.15.42:0/499320086 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:45:49.866 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:49.855+0000 7f6d4c851700 1 --2- 172.21.15.42:0/499320086 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6d480a7740 0x7f6d4808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:45:49.866 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:49.855+0000 7f6d3ffff700 1 --2- 172.21.15.42:0/499320086 >> v2:172.21.15.40:3300/0 conn(0x7f6d4807e720 0x7f6d4807eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:45:49.866 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:49.855+0000 7f6d4d052700 1 -- 172.21.15.42:0/499320086 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6d480a7740 msgr2=0x7f6d4808df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:45:49.867 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:49.855+0000 7f6d4d052700 1 --2- 172.21.15.42:0/499320086 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6d480a7740 0x7f6d4808df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:45:49.867 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:49.855+0000 7f6d4d052700 1 -- 172.21.15.42:0/499320086 >> v2:172.21.15.40:3300/0 conn(0x7f6d4807e720 msgr2=0x7f6d4807eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:45:49.867 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:49.855+0000 7f6d4d052700 1 --2- 172.21.15.42:0/499320086 >> v2:172.21.15.40:3300/0 conn(0x7f6d4807e720 0x7f6d4807eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:45:49.867 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:49.855+0000 7f6d4d052700 1 -- 172.21.15.42:0/499320086 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d4807f0a0 con 0x7f6d4809f2e0 2019-12-05T05:45:49.867 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:49.855+0000 7f6d4d052700 1 --2- 172.21.15.42:0/499320086 >> v2:172.21.15.42:3300/0 conn(0x7f6d4809f2e0 0x7f6d4809f710 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f6d4808e770 tx=0x7f6d40010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:49.868 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:49.855+0000 7f6d4d052700 1 --2- 172.21.15.42:0/499320086 >> v2:172.21.15.42:3300/0 conn(0x7f6d4809f2e0 0x7f6d4809f710 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f6d4808e770 tx=0x7f6d40010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:49.868 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:49.855+0000 7f6d4d052700 1 --2- 172.21.15.42:0/499320086 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6d4809f2e0 0x7f6d4809f710 secure :-1 s=READY pgs=1444 cs=0 l=1 rx=0x7f6d4808e770 tx=0x7f6d40010130).ready entity=mon.1 client_cookie=93be73c8aa4f748c server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:45:49.868 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:49.855+0000 7f6d4d052700 1 --2- 172.21.15.42:0/499320086 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6d4809f2e0 0x7f6d4809f710 secure :-1 s=THROTTLE_DONE pgs=1444 cs=0 l=1 rx=0x7f6d4808e770 tx=0x7f6d40010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:49.868 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:49.855+0000 7f6d4d052700 1 --2- 172.21.15.42:0/499320086 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6d4809f2e0 0x7f6d4809f710 secure :-1 s=THROTTLE_DONE pgs=1444 cs=0 l=1 rx=0x7f6d4808e770 tx=0x7f6d40010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:49.868 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:49.855+0000 7f6d3f7fe700 1 -- 172.21.15.42:0/499320086 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6d40010de0 con 0x7f6d4809f2e0 2019-12-05T05:45:49.869 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:49.855+0000 7f6d4d052700 1 --2- 172.21.15.42:0/499320086 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6d4809f2e0 0x7f6d4809f710 secure :-1 s=THROTTLE_DONE pgs=1444 cs=0 l=1 rx=0x7f6d4808e770 tx=0x7f6d40010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:49.869 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:49.855+0000 7f6d3f7fe700 1 -- 172.21.15.42:0/499320086 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d40010a00 con 0x7f6d4809f2e0 2019-12-05T05:45:49.869 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:49.855+0000 7f6d3f7fe700 1 -- 172.21.15.42:0/499320086 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6d400114a0 con 0x7f6d4809f2e0 2019-12-05T05:45:49.869 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:49.855+0000 7f6d4e2b4700 1 -- 172.21.15.42:0/499320086 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6d4809f2e0 msgr2=0x7f6d4809f710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:45:49.869 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:49.855+0000 7f6d4e2b4700 1 --2- 172.21.15.42:0/499320086 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6d4809f2e0 0x7f6d4809f710 secure :-1 s=READY pgs=1444 cs=0 l=1 rx=0x7f6d4808e770 tx=0x7f6d40010130).stop 2019-12-05T05:45:49.869 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:49.855+0000 7f6d4e2b4700 1 -- 172.21.15.42:0/499320086 shutdown_connections 2019-12-05T05:45:49.870 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:49.855+0000 7f6d4e2b4700 1 --2- 172.21.15.42:0/499320086 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6d4809f2e0 0x7f6d4809f710 unknown :-1 s=CLOSED pgs=1444 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:45:49.870 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:49.855+0000 7f6d4e2b4700 1 --2- 172.21.15.42:0/499320086 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6d480a7740 0x7f6d4808df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:45:49.870 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:49.855+0000 7f6d4e2b4700 1 --2- 172.21.15.42:0/499320086 >> v2:172.21.15.40:3300/0 conn(0x7f6d4807e720 0x7f6d4807eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:45:49.870 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:49.855+0000 7f6d4e2b4700 1 -- 172.21.15.42:0/499320086 shutdown_connections 2019-12-05T05:45:49.870 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:49.855+0000 7f6d4e2b4700 1 -- 172.21.15.42:0/499320086 wait complete. 2019-12-05T05:45:49.871 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:49.855+0000 7f6d4e2b4700 1 -- 172.21.15.42:0/499320086 >> 172.21.15.42:0/499320086 conn(0x7f6d480aabe0 msgr2=0x7f6d480a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:45:49.874 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:49.859+0000 7f6d4e2b4700 1 Processor -- start 2019-12-05T05:45:49.875 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:49.859+0000 7f6d4e2b4700 1 -- start start 2019-12-05T05:45:49.875 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:49.859+0000 7f6d4e2b4700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6d48121270 0x7f6d48059c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:45:49.875 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:49.859+0000 7f6d4e2b4700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d48120bc0 con 0x7f6d480722c0 2019-12-05T05:45:49.875 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:49.859+0000 7f6d4e2b4700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d4805a160 con 0x7f6d4811fe10 2019-12-05T05:45:49.875 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:49.859+0000 7f6d4e2b4700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d481450e0 con 0x7f6d48121270 2019-12-05T05:45:49.875 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:49.859+0000 7f6d4c851700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6d48121270 0x7f6d48059c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:45:49.876 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:49.859+0000 7f6d3ffff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f6d4811fe10 0x7f6d48120240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:40776/0 (socket says 172.21.15.42:40776) 2019-12-05T05:45:49.876 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:49.859+0000 7f6d3ffff700 1 -- 172.21.15.42:0/3564895455 learned_addr learned my addr 172.21.15.42:0/3564895455 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:45:49.876 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:49.859+0000 7f6d4c851700 1 -- 172.21.15.42:0/3564895455 >> v1:172.21.15.42:6789/0 conn(0x7f6d4811fe10 legacy=0x7f6d48120240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:45:49.876 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:49.859+0000 7f6d4c851700 1 -- 172.21.15.42:0/3564895455 >> v1:172.21.15.40:6789/0 conn(0x7f6d480722c0 legacy=0x7f6d48074700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:45:49.876 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:49.859+0000 7f6d4c851700 1 -- 172.21.15.42:0/3564895455 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d4811eb80 con 0x7f6d48121270 2019-12-05T05:45:49.876 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:49.859+0000 7f6d4c851700 1 --2- 172.21.15.42:0/3564895455 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6d48121270 0x7f6d48059c10 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f6d48145650 tx=0x7f6d44010960).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:49.877 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:49.859+0000 7f6d4c851700 1 --2- 172.21.15.42:0/3564895455 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6d48121270 0x7f6d48059c10 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f6d48145650 tx=0x7f6d44010960).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:49.877 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:49.859+0000 7f6d4c851700 1 --2- 172.21.15.42:0/3564895455 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6d48121270 0x7f6d48059c10 secure :-1 s=READY pgs=319 cs=0 l=1 rx=0x7f6d48145650 tx=0x7f6d44010960).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:45:49.877 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:49.863+0000 7f6d4c851700 1 --2- 172.21.15.42:0/3564895455 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6d48121270 0x7f6d48059c10 secure :-1 s=THROTTLE_DONE pgs=319 cs=0 l=1 rx=0x7f6d48145650 tx=0x7f6d44010960).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:49.877 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:49.863+0000 7f6d4c851700 1 --2- 172.21.15.42:0/3564895455 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6d48121270 0x7f6d48059c10 secure :-1 s=THROTTLE_DONE pgs=319 cs=0 l=1 rx=0x7f6d48145650 tx=0x7f6d44010960).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:49.877 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:49.863+0000 7f6d4c851700 1 --2- 172.21.15.42:0/3564895455 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6d48121270 0x7f6d48059c10 secure :-1 s=THROTTLE_DONE pgs=319 cs=0 l=1 rx=0x7f6d48145650 tx=0x7f6d44010960).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:49.877 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:49.863+0000 7f6d3d7fa700 1 -- 172.21.15.42:0/3564895455 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6d44005930 con 0x7f6d48121270 2019-12-05T05:45:49.877 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:49.863+0000 7f6d4e2b4700 1 -- 172.21.15.42:0/3564895455 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d48144cc0 con 0x7f6d48121270 2019-12-05T05:45:49.878 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:49.863+0000 7f6d4e2b4700 1 -- 172.21.15.42:0/3564895455 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d48075f10 con 0x7f6d48121270 2019-12-05T05:45:49.878 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:49.863+0000 7f6d3d7fa700 1 -- 172.21.15.42:0/3564895455 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d44007200 con 0x7f6d48121270 2019-12-05T05:45:49.878 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:49.863+0000 7f6d3d7fa700 1 -- 172.21.15.42:0/3564895455 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6d440060c0 con 0x7f6d48121270 2019-12-05T05:45:49.879 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:49.863+0000 7f6d4c851700 1 --2- 172.21.15.42:0/3564895455 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6d48121270 0x7f6d48059c10 secure :-1 s=THROTTLE_DONE pgs=319 cs=0 l=1 rx=0x7f6d48145650 tx=0x7f6d44010960).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:49.879 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:49.863+0000 7f6d3d7fa700 1 -- 172.21.15.42:0/3564895455 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f6d440063d0 con 0x7f6d48121270 2019-12-05T05:45:49.879 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:49.863+0000 7f6d4c851700 1 --2- 172.21.15.42:0/3564895455 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6d48121270 0x7f6d48059c10 secure :-1 s=THROTTLE_DONE pgs=319 cs=0 l=1 rx=0x7f6d48145650 tx=0x7f6d44010960).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:49.879 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:49.863+0000 7f6d3d7fa700 1 --2- 172.21.15.42:0/3564895455 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6d2803adf0 0x7f6d2803d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:45:49.879 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:49.863+0000 7f6d3d7fa700 1 -- 172.21.15.42:0/3564895455 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 8525+0+0 (secure 0 0 0) 0x7f6d4404aae0 con 0x7f6d48121270 2019-12-05T05:45:49.879 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:49.863+0000 7f6d3ffff700 1 --2- 172.21.15.42:0/3564895455 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6d2803adf0 0x7f6d2803d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:45:49.879 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:49.863+0000 7f6d3ffff700 1 -- 172.21.15.42:0/3564895455 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6d2803adf0 msgr2=0x7f6d2803d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2019-12-05T05:45:49.880 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:49.863+0000 7f6d3ffff700 1 -- 172.21.15.42:0/3564895455 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6d2803adf0 msgr2=0x7f6d2803d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2019-12-05T05:45:49.880 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:49.863+0000 7f6d3ffff700 1 --2- 172.21.15.42:0/3564895455 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6d2803adf0 0x7f6d2803d300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-12-05T05:45:49.880 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:49.863+0000 7f6d3ffff700 1 --2- 172.21.15.42:0/3564895455 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6d2803adf0 0x7f6d2803d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rx=0 tx=0)._fault waiting 0.200000 2019-12-05T05:45:49.880 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:49.871+0000 7f6d4e2b4700 1 -- 172.21.15.42:0/3564895455 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d20003200 con 0x7f6d48121270 2019-12-05T05:45:49.881 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:49.875+0000 7f6d4c851700 1 --2- 172.21.15.42:0/3564895455 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6d48121270 0x7f6d48059c10 secure :-1 s=THROTTLE_DONE pgs=319 cs=0 l=1 rx=0x7f6d48145650 tx=0x7f6d44010960).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:49.881 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:49.875+0000 7f6d3d7fa700 1 -- 172.21.15.42:0/3564895455 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f6d44031b80 con 0x7f6d48121270 2019-12-05T05:45:50.052 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:50.047+0000 7f6d4e2b4700 1 -- 172.21.15.42:0/3564895455 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6d20003630 con 0x7f6d48121270 2019-12-05T05:45:50.053 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:50.047+0000 7f6d4c851700 1 --2- 172.21.15.42:0/3564895455 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6d48121270 0x7f6d48059c10 secure :-1 s=THROTTLE_DONE pgs=319 cs=0 l=1 rx=0x7f6d48145650 tx=0x7f6d44010960).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:50.053 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:50.047+0000 7f6d3d7fa700 1 -- 172.21.15.42:0/3564895455 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 177 v177) v1 ==== 95+0+1775 (secure 0 0 0) 0x7f6d44031de0 con 0x7f6d48121270 2019-12-05T05:45:50.055 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:45:50.055 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":177,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":177,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:45:43.664230+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":78,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6077},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6077":{"gid":6077,"name":"a","rank":0,"incarnation":174,"state":"up:active","state_seq":16,"addr":"172.21.15.42:6835/3757738536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3757738536},{"type":"v1","addr":"172.21.15.42:6835","nonce":3757738536}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:45:50.055 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 177 2019-12-05T05:45:50.058 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:50.051+0000 7f6d4e2b4700 1 -- 172.21.15.42:0/3564895455 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6d2803adf0 msgr2=0x7f6d2803d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2019-12-05T05:45:50.058 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:50.051+0000 7f6d4e2b4700 1 --2- 172.21.15.42:0/3564895455 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6d2803adf0 0x7f6d2803d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:45:50.058 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:50.051+0000 7f6d4e2b4700 1 -- 172.21.15.42:0/3564895455 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6d48121270 msgr2=0x7f6d48059c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:45:50.058 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:50.051+0000 7f6d4e2b4700 1 --2- 172.21.15.42:0/3564895455 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6d48121270 0x7f6d48059c10 secure :-1 s=READY pgs=319 cs=0 l=1 rx=0x7f6d48145650 tx=0x7f6d44010960).stop 2019-12-05T05:45:50.058 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:50.051+0000 7f6d4e2b4700 1 -- 172.21.15.42:0/3564895455 shutdown_connections 2019-12-05T05:45:50.059 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:50.051+0000 7f6d4e2b4700 1 --2- 172.21.15.42:0/3564895455 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6d2803adf0 0x7f6d2803d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:45:50.059 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:50.051+0000 7f6d4e2b4700 1 --2- 172.21.15.42:0/3564895455 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6d48121270 0x7f6d48059c10 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:45:50.059 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:50.051+0000 7f6d4e2b4700 1 -- 172.21.15.42:0/3564895455 shutdown_connections 2019-12-05T05:45:50.059 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:50.051+0000 7f6d4e2b4700 1 -- 172.21.15.42:0/3564895455 wait complete. 2019-12-05T05:45:50.059 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:50.051+0000 7f6d4e2b4700 1 -- 172.21.15.42:0/3564895455 >> 172.21.15.42:0/3564895455 conn(0x7f6d480a7f90 msgr2=0x7f6d4811c540 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:45:50.068 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2019-12-05T05:45:52.069 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:45:52.177 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:52.167+0000 7f0ada2bd700 1 Processor -- start 2019-12-05T05:45:52.178 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:52.167+0000 7f0ada2bd700 1 -- start start 2019-12-05T05:45:52.178 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:52.167+0000 7f0ada2bd700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0ad408df80 0x7f0ad409f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:45:52.178 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:52.167+0000 7f0ada2bd700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ad409fc60 con 0x7f0ad40a7740 2019-12-05T05:45:52.178 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:52.167+0000 7f0ada2bd700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ad40a0710 con 0x7f0ad409f830 2019-12-05T05:45:52.178 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:52.167+0000 7f0ada2bd700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ad40a0b00 con 0x7f0ad408df80 2019-12-05T05:45:52.178 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:52.167+0000 7f0ad3fff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0ad408df80 0x7f0ad409f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:45:52.179 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:52.167+0000 7f0ad3fff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0ad408df80 0x7f0ad409f2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:40752/0 (socket says 172.21.15.42:40752) 2019-12-05T05:45:52.179 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:52.167+0000 7f0ad3fff700 1 -- 172.21.15.42:0/3783804419 learned_addr learned my addr 172.21.15.42:0/3783804419 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:45:52.179 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:52.167+0000 7f0ad3fff700 1 -- 172.21.15.42:0/3783804419 >> v1:172.21.15.42:6789/0 conn(0x7f0ad409f830 legacy=0x7f0ad414efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:45:52.179 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:52.167+0000 7f0ad3fff700 1 -- 172.21.15.42:0/3783804419 >> v1:172.21.15.40:6789/0 conn(0x7f0ad40a7740 legacy=0x7f0ad407e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:45:52.179 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:52.167+0000 7f0ad3fff700 1 -- 172.21.15.42:0/3783804419 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ad407ef20 con 0x7f0ad408df80 2019-12-05T05:45:52.179 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:52.167+0000 7f0ad3fff700 1 --2- 172.21.15.42:0/3783804419 >> v2:172.21.15.42:3300/0 conn(0x7f0ad408df80 0x7f0ad409f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f0abc008790 tx=0x7f0abc00fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:52.180 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:52.167+0000 7f0ad3fff700 1 --2- 172.21.15.42:0/3783804419 >> v2:172.21.15.42:3300/0 conn(0x7f0ad408df80 0x7f0ad409f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f0abc008790 tx=0x7f0abc00fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:52.180 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:52.167+0000 7f0ad3fff700 1 --2- 172.21.15.42:0/3783804419 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0ad408df80 0x7f0ad409f2e0 secure :-1 s=READY pgs=1447 cs=0 l=1 rx=0x7f0abc008790 tx=0x7f0abc00fcb0).ready entity=mon.1 client_cookie=d82f104712f277c3 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:45:52.180 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:52.171+0000 7f0ad3fff700 1 --2- 172.21.15.42:0/3783804419 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0ad408df80 0x7f0ad409f2e0 secure :-1 s=THROTTLE_DONE pgs=1447 cs=0 l=1 rx=0x7f0abc008790 tx=0x7f0abc00fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:52.180 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:52.171+0000 7f0ad37fe700 1 -- 172.21.15.42:0/3783804419 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0abc011520 con 0x7f0ad408df80 2019-12-05T05:45:52.180 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:52.171+0000 7f0ad3fff700 1 --2- 172.21.15.42:0/3783804419 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0ad408df80 0x7f0ad409f2e0 secure :-1 s=THROTTLE_DONE pgs=1447 cs=0 l=1 rx=0x7f0abc008790 tx=0x7f0abc00fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:52.180 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:52.171+0000 7f0ad37fe700 1 -- 172.21.15.42:0/3783804419 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0abc008ca0 con 0x7f0ad408df80 2019-12-05T05:45:52.180 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:52.171+0000 7f0ad3fff700 1 --2- 172.21.15.42:0/3783804419 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0ad408df80 0x7f0ad409f2e0 secure :-1 s=THROTTLE_DONE pgs=1447 cs=0 l=1 rx=0x7f0abc008790 tx=0x7f0abc00fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:52.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:52.171+0000 7f0ada2bd700 1 -- 172.21.15.42:0/3783804419 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0ad408df80 msgr2=0x7f0ad409f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:45:52.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:52.171+0000 7f0ad37fe700 1 -- 172.21.15.42:0/3783804419 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0abc011b00 con 0x7f0ad408df80 2019-12-05T05:45:52.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:52.171+0000 7f0ada2bd700 1 --2- 172.21.15.42:0/3783804419 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0ad408df80 0x7f0ad409f2e0 secure :-1 s=READY pgs=1447 cs=0 l=1 rx=0x7f0abc008790 tx=0x7f0abc00fcb0).stop 2019-12-05T05:45:52.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:52.171+0000 7f0ada2bd700 1 -- 172.21.15.42:0/3783804419 shutdown_connections 2019-12-05T05:45:52.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:52.171+0000 7f0ada2bd700 1 --2- 172.21.15.42:0/3783804419 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0ad408df80 0x7f0ad409f2e0 unknown :-1 s=CLOSED pgs=1447 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:45:52.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:52.171+0000 7f0ada2bd700 1 -- 172.21.15.42:0/3783804419 shutdown_connections 2019-12-05T05:45:52.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:52.171+0000 7f0ada2bd700 1 -- 172.21.15.42:0/3783804419 wait complete. 2019-12-05T05:45:52.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:52.171+0000 7f0ada2bd700 1 -- 172.21.15.42:0/3783804419 >> 172.21.15.42:0/3783804419 conn(0x7f0ad40aabe0 msgr2=0x7f0ad40a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:45:52.185 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:52.171+0000 7f0ada2bd700 1 Processor -- start 2019-12-05T05:45:52.185 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:52.171+0000 7f0ada2bd700 1 -- start start 2019-12-05T05:45:52.186 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:52.175+0000 7f0ada2bd700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0ad408eeb0 0x7f0ad4128380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:45:52.186 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:52.175+0000 7f0ada2bd700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0ad411a150 0x7f0ad4130210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:45:52.186 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:52.175+0000 7f0ada2bd700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ad4064530 con 0x7f0ad41288d0 2019-12-05T05:45:52.186 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:52.175+0000 7f0ada2bd700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ad4119170 con 0x7f0ad408eeb0 2019-12-05T05:45:52.186 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:52.175+0000 7f0ada2bd700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ad4119410 con 0x7f0ad411a150 2019-12-05T05:45:52.186 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:52.175+0000 7f0ad885a700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0ad408eeb0 0x7f0ad4128380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:45:52.187 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:52.175+0000 7f0ad3fff700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f0ad41288d0 0x7f0ad4119950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:50234/0 (socket says 172.21.15.42:50234) 2019-12-05T05:45:52.187 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:52.175+0000 7f0ad885a700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0ad408eeb0 0x7f0ad4128380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:40758/0 (socket says 172.21.15.42:40758) 2019-12-05T05:45:52.188 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:52.175+0000 7f0ad3fff700 1 -- 172.21.15.42:0/2306873412 learned_addr learned my addr 172.21.15.42:0/2306873412 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:45:52.188 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:52.175+0000 7f0ad905b700 1 --2- 172.21.15.42:0/2306873412 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0ad411a150 0x7f0ad4130210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:45:52.188 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:52.175+0000 7f0ad17fa700 1 -- 172.21.15.42:0/2306873412 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1387346071 0 0) 0x7f0ad4064530 con 0x7f0ad41288d0 2019-12-05T05:45:52.188 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:52.175+0000 7f0ad17fa700 1 -- 172.21.15.42:0/2306873412 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ab80030b0 con 0x7f0ad41288d0 2019-12-05T05:45:52.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:52.175+0000 7f0ad885a700 1 -- 172.21.15.42:0/2306873412 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0ad411a150 msgr2=0x7f0ad4130210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:45:52.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:52.175+0000 7f0ad885a700 1 --2- 172.21.15.42:0/2306873412 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0ad411a150 0x7f0ad4130210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:45:52.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:52.175+0000 7f0ad885a700 1 -- 172.21.15.42:0/2306873412 >> v1:172.21.15.40:6789/0 conn(0x7f0ad41288d0 legacy=0x7f0ad4119950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:45:52.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:52.175+0000 7f0ad885a700 1 -- 172.21.15.42:0/2306873412 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ad40648d0 con 0x7f0ad408eeb0 2019-12-05T05:45:52.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:52.175+0000 7f0ad885a700 1 --2- 172.21.15.42:0/2306873412 >> v2:172.21.15.42:3300/0 conn(0x7f0ad408eeb0 0x7f0ad4128380 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f0ad4129e00 tx=0x7f0ac400c320).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:52.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:52.175+0000 7f0ad885a700 1 --2- 172.21.15.42:0/2306873412 >> v2:172.21.15.42:3300/0 conn(0x7f0ad408eeb0 0x7f0ad4128380 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f0ad4129e00 tx=0x7f0ac400c320).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:52.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:52.175+0000 7f0ad885a700 1 --2- 172.21.15.42:0/2306873412 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0ad408eeb0 0x7f0ad4128380 secure :-1 s=READY pgs=1448 cs=0 l=1 rx=0x7f0ad4129e00 tx=0x7f0ac400c320).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:45:52.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:52.175+0000 7f0ad885a700 1 --2- 172.21.15.42:0/2306873412 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0ad408eeb0 0x7f0ad4128380 secure :-1 s=THROTTLE_DONE pgs=1448 cs=0 l=1 rx=0x7f0ad4129e00 tx=0x7f0ac400c320).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:52.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:52.175+0000 7f0ad17fa700 1 -- 172.21.15.42:0/2306873412 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0ac400ca80 con 0x7f0ad408eeb0 2019-12-05T05:45:52.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:52.175+0000 7f0ad885a700 1 --2- 172.21.15.42:0/2306873412 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0ad408eeb0 0x7f0ad4128380 secure :-1 s=THROTTLE_DONE pgs=1448 cs=0 l=1 rx=0x7f0ad4129e00 tx=0x7f0ac400c320).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:52.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:52.175+0000 7f0ada2bd700 1 -- 172.21.15.42:0/2306873412 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ad4118d10 con 0x7f0ad408eeb0 2019-12-05T05:45:52.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:52.175+0000 7f0ad17fa700 1 -- 172.21.15.42:0/2306873412 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ac400c6a0 con 0x7f0ad408eeb0 2019-12-05T05:45:52.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:52.175+0000 7f0ad885a700 1 --2- 172.21.15.42:0/2306873412 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0ad408eeb0 0x7f0ad4128380 secure :-1 s=THROTTLE_DONE pgs=1448 cs=0 l=1 rx=0x7f0ad4129e00 tx=0x7f0ac400c320).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:52.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:52.175+0000 7f0ad17fa700 1 -- 172.21.15.42:0/2306873412 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0ac40110e0 con 0x7f0ad408eeb0 2019-12-05T05:45:52.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:52.175+0000 7f0ada2bd700 1 -- 172.21.15.42:0/2306873412 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ad4065820 con 0x7f0ad408eeb0 2019-12-05T05:45:52.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:52.179+0000 7f0ad885a700 1 --2- 172.21.15.42:0/2306873412 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0ad408eeb0 0x7f0ad4128380 secure :-1 s=THROTTLE_DONE pgs=1448 cs=0 l=1 rx=0x7f0ad4129e00 tx=0x7f0ac400c320).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:52.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:52.179+0000 7f0ad17fa700 1 -- 172.21.15.42:0/2306873412 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f0ac4011400 con 0x7f0ad408eeb0 2019-12-05T05:45:52.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:52.179+0000 7f0ad885a700 1 --2- 172.21.15.42:0/2306873412 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0ad408eeb0 0x7f0ad4128380 secure :-1 s=THROTTLE_DONE pgs=1448 cs=0 l=1 rx=0x7f0ad4129e00 tx=0x7f0ac400c320).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:52.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:52.179+0000 7f0ad17fa700 1 --2- 172.21.15.42:0/2306873412 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0ab803b220 0x7f0ab803d730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:45:52.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:52.179+0000 7f0ad17fa700 1 -- 172.21.15.42:0/2306873412 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 8525+0+0 (secure 0 0 0) 0x7f0ac404b210 con 0x7f0ad408eeb0 2019-12-05T05:45:52.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:52.179+0000 7f0ad3fff700 1 --2- 172.21.15.42:0/2306873412 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0ab803b220 0x7f0ab803d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:45:52.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:52.179+0000 7f0ad3fff700 1 --2- 172.21.15.42:0/2306873412 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0ab803b220 0x7f0ab803d730 crc :-1 s=READY pgs=623 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:45:52.193 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:52.183+0000 7f0ada2bd700 1 -- 172.21.15.42:0/2306873412 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ad405d110 con 0x7f0ad408eeb0 2019-12-05T05:45:52.201 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:52.191+0000 7f0ad885a700 1 --2- 172.21.15.42:0/2306873412 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0ad408eeb0 0x7f0ad4128380 secure :-1 s=THROTTLE_DONE pgs=1448 cs=0 l=1 rx=0x7f0ad4129e00 tx=0x7f0ac400c320).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:52.201 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:52.191+0000 7f0ad17fa700 1 -- 172.21.15.42:0/2306873412 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f0ac4032210 con 0x7f0ad408eeb0 2019-12-05T05:45:52.367 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:52.359+0000 7f0ada2bd700 1 -- 172.21.15.42:0/2306873412 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0ad405a4f0 con 0x7f0ad408eeb0 2019-12-05T05:45:52.367 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:52.359+0000 7f0ad885a700 1 --2- 172.21.15.42:0/2306873412 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0ad408eeb0 0x7f0ad4128380 secure :-1 s=THROTTLE_DONE pgs=1448 cs=0 l=1 rx=0x7f0ad4129e00 tx=0x7f0ac400c320).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:45:52.368 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:52.359+0000 7f0ad17fa700 1 -- 172.21.15.42:0/2306873412 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 178 v178) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f0ac4032470 con 0x7f0ad408eeb0 2019-12-05T05:45:52.368 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 178 2019-12-05T05:45:52.368 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:45:52.368 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":178,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6140,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/2146240383","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2146240383},{"type":"v1","addr":"172.21.15.40:6833","nonce":2146240383}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":178}],"filesystems":[{"mdsmap":{"epoch":177,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:45:43.664230+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":78,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6077},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6077":{"gid":6077,"name":"a","rank":0,"incarnation":174,"state":"up:active","state_seq":16,"addr":"172.21.15.42:6835/3757738536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3757738536},{"type":"v1","addr":"172.21.15.42:6835","nonce":3757738536}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:45:52.370 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:52.363+0000 7f0ada2bd700 1 -- 172.21.15.42:0/2306873412 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0ab803b220 msgr2=0x7f0ab803d730 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:45:52.370 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:52.363+0000 7f0ada2bd700 1 --2- 172.21.15.42:0/2306873412 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0ab803b220 0x7f0ab803d730 crc :-1 s=READY pgs=623 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:45:52.370 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:52.363+0000 7f0ada2bd700 1 -- 172.21.15.42:0/2306873412 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0ad408eeb0 msgr2=0x7f0ad4128380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:45:52.371 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:52.363+0000 7f0ada2bd700 1 --2- 172.21.15.42:0/2306873412 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0ad408eeb0 0x7f0ad4128380 secure :-1 s=READY pgs=1448 cs=0 l=1 rx=0x7f0ad4129e00 tx=0x7f0ac400c320).stop 2019-12-05T05:45:52.371 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:52.363+0000 7f0ada2bd700 1 -- 172.21.15.42:0/2306873412 shutdown_connections 2019-12-05T05:45:52.371 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:52.363+0000 7f0ada2bd700 1 --2- 172.21.15.42:0/2306873412 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0ab803b220 0x7f0ab803d730 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:45:52.371 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:52.363+0000 7f0ada2bd700 1 --2- 172.21.15.42:0/2306873412 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0ad411a150 0x7f0ad4130210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:45:52.371 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:52.363+0000 7f0ada2bd700 1 --2- 172.21.15.42:0/2306873412 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0ad408eeb0 0x7f0ad4128380 unknown :-1 s=CLOSED pgs=1448 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:45:52.371 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:52.363+0000 7f0ada2bd700 1 -- 172.21.15.42:0/2306873412 shutdown_connections 2019-12-05T05:45:52.371 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:52.363+0000 7f0ada2bd700 1 -- 172.21.15.42:0/2306873412 wait complete. 2019-12-05T05:45:52.372 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:45:52.363+0000 7f0ada2bd700 1 -- 172.21.15.42:0/2306873412 >> 172.21.15.42:0/2306873412 conn(0x7f0ad409f220 msgr2=0x7f0ad4111db0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:45:52.384 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b reported in up:standby state 2019-12-05T05:45:52.384 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 68 secs before thrashing 2019-12-05T05:46:11.432 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:46:11.436 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:46:41.497 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:46:41.501 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:47:00.385 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:47:00.493 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.486+0000 7fb546ccd700 1 Processor -- start 2019-12-05T05:47:00.493 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.486+0000 7fb546ccd700 1 -- start start 2019-12-05T05:47:00.493 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.486+0000 7fb546ccd700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb5400a7740 0x7fb54008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:47:00.493 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.486+0000 7fb546ccd700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb54007e720 0x7fb54007eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:47:00.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.486+0000 7fb546ccd700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5400a0710 con 0x7fb54009f2e0 2019-12-05T05:47:00.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.486+0000 7fb546ccd700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5400a0b00 con 0x7fb5400a7740 2019-12-05T05:47:00.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.486+0000 7fb546ccd700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb54008e5b0 con 0x7fb54007e720 2019-12-05T05:47:00.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.486+0000 7fb544a69700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb54007e720 0x7fb54007eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:47:00.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.486+0000 7fb544a69700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb54007e720 0x7fb54007eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:41096/0 (socket says 172.21.15.42:41096) 2019-12-05T05:47:00.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.486+0000 7fb544a69700 1 -- 172.21.15.42:0/2979273356 learned_addr learned my addr 172.21.15.42:0/2979273356 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:47:00.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.486+0000 7fb54526a700 1 --2- 172.21.15.42:0/2979273356 >> v2:172.21.15.40:3300/0 conn(0x7fb5400a7740 0x7fb54008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:47:00.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.486+0000 7fb544a69700 1 -- 172.21.15.42:0/2979273356 >> v2:172.21.15.40:3300/0 conn(0x7fb5400a7740 msgr2=0x7fb54008df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:47:00.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.486+0000 7fb544a69700 1 --2- 172.21.15.42:0/2979273356 >> v2:172.21.15.40:3300/0 conn(0x7fb5400a7740 0x7fb54008df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:47:00.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.486+0000 7fb544a69700 1 -- 172.21.15.42:0/2979273356 >> v1:172.21.15.40:6789/0 conn(0x7fb54009f2e0 legacy=0x7fb54009f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:47:00.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.486+0000 7fb544a69700 1 -- 172.21.15.42:0/2979273356 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb54007f0a0 con 0x7fb54007e720 2019-12-05T05:47:00.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.486+0000 7fb54526a700 1 --2- 172.21.15.42:0/2979273356 >> v2:172.21.15.40:3300/0 conn(0x7fb5400a7740 0x7fb54008df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T05:47:00.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.486+0000 7fb544a69700 1 --2- 172.21.15.42:0/2979273356 >> v2:172.21.15.42:3300/0 conn(0x7fb54007e720 0x7fb54007eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fb530002f30 tx=0x7fb53000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:00.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.486+0000 7fb544a69700 1 --2- 172.21.15.42:0/2979273356 >> v2:172.21.15.42:3300/0 conn(0x7fb54007e720 0x7fb54007eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fb530002f30 tx=0x7fb53000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:00.496 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.486+0000 7fb544a69700 1 --2- 172.21.15.42:0/2979273356 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb54007e720 0x7fb54007eb50 secure :-1 s=READY pgs=1452 cs=0 l=1 rx=0x7fb530002f30 tx=0x7fb53000fd30).ready entity=mon.1 client_cookie=e197588b281272b7 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:47:00.496 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.486+0000 7fb544a69700 1 --2- 172.21.15.42:0/2979273356 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb54007e720 0x7fb54007eb50 secure :-1 s=THROTTLE_DONE pgs=1452 cs=0 l=1 rx=0x7fb530002f30 tx=0x7fb53000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:00.496 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.486+0000 7fb544a69700 1 --2- 172.21.15.42:0/2979273356 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb54007e720 0x7fb54007eb50 secure :-1 s=THROTTLE_DONE pgs=1452 cs=0 l=1 rx=0x7fb530002f30 tx=0x7fb53000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:00.496 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.486+0000 7fb544a69700 1 --2- 172.21.15.42:0/2979273356 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb54007e720 0x7fb54007eb50 secure :-1 s=THROTTLE_DONE pgs=1452 cs=0 l=1 rx=0x7fb530002f30 tx=0x7fb53000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:00.496 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.486+0000 7fb537fff700 1 -- 172.21.15.42:0/2979273356 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb530011370 con 0x7fb54007e720 2019-12-05T05:47:00.496 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.486+0000 7fb537fff700 1 -- 172.21.15.42:0/2979273356 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb530008bb0 con 0x7fb54007e720 2019-12-05T05:47:00.496 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.486+0000 7fb537fff700 1 -- 172.21.15.42:0/2979273356 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb530011950 con 0x7fb54007e720 2019-12-05T05:47:00.497 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.486+0000 7fb546ccd700 1 -- 172.21.15.42:0/2979273356 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb54007e720 msgr2=0x7fb54007eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:47:00.497 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.486+0000 7fb546ccd700 1 --2- 172.21.15.42:0/2979273356 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb54007e720 0x7fb54007eb50 secure :-1 s=READY pgs=1452 cs=0 l=1 rx=0x7fb530002f30 tx=0x7fb53000fd30).stop 2019-12-05T05:47:00.497 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.490+0000 7fb546ccd700 1 -- 172.21.15.42:0/2979273356 shutdown_connections 2019-12-05T05:47:00.497 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.490+0000 7fb546ccd700 1 --2- 172.21.15.42:0/2979273356 >> v2:172.21.15.40:3300/0 conn(0x7fb5400a7740 0x7fb54008df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:47:00.497 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.490+0000 7fb546ccd700 1 --2- 172.21.15.42:0/2979273356 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb54007e720 0x7fb54007eb50 unknown :-1 s=CLOSED pgs=1452 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:47:00.497 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.490+0000 7fb546ccd700 1 -- 172.21.15.42:0/2979273356 shutdown_connections 2019-12-05T05:47:00.497 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.490+0000 7fb546ccd700 1 -- 172.21.15.42:0/2979273356 wait complete. 2019-12-05T05:47:00.498 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.490+0000 7fb546ccd700 1 -- 172.21.15.42:0/2979273356 >> 172.21.15.42:0/2979273356 conn(0x7fb5400aabe0 msgr2=0x7fb5400a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:47:00.501 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.490+0000 7fb546ccd700 1 Processor -- start 2019-12-05T05:47:00.501 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.490+0000 7fb546ccd700 1 -- start start 2019-12-05T05:47:00.501 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.490+0000 7fb546ccd700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb5400a1a20 0x7fb5400a39e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:47:00.501 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.490+0000 7fb546ccd700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb5400a3f30 0x7fb54012f540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:47:00.502 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.490+0000 7fb546ccd700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5400a4360 con 0x7fb540116af0 2019-12-05T05:47:00.502 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.490+0000 7fb546ccd700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5400a1e50 con 0x7fb5400a1a20 2019-12-05T05:47:00.502 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.490+0000 7fb546ccd700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5400a26e0 con 0x7fb5400a3f30 2019-12-05T05:47:00.502 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.490+0000 7fb544a69700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb5400a1a20 0x7fb5400a39e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:47:00.502 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.490+0000 7fb545a6b700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb5400a3f30 0x7fb54012f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:47:00.503 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.490+0000 7fb544a69700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb5400a1a20 0x7fb5400a39e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.42:57148/0 (socket says 172.21.15.42:57148) 2019-12-05T05:47:00.503 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.490+0000 7fb544a69700 1 -- 172.21.15.42:0/48566100 learned_addr learned my addr 172.21.15.42:0/48566100 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:47:00.503 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.494+0000 7fb545a6b700 1 -- 172.21.15.42:0/48566100 >> v2:172.21.15.40:3300/0 conn(0x7fb5400a1a20 msgr2=0x7fb5400a39e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:47:00.503 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.494+0000 7fb545a6b700 1 --2- 172.21.15.42:0/48566100 >> v2:172.21.15.40:3300/0 conn(0x7fb5400a1a20 0x7fb5400a39e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:47:00.503 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.494+0000 7fb545a6b700 1 -- 172.21.15.42:0/48566100 >> v1:172.21.15.40:6789/0 conn(0x7fb540116af0 legacy=0x7fb54007d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:47:00.503 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.494+0000 7fb545a6b700 1 -- 172.21.15.42:0/48566100 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb540113be0 con 0x7fb5400a3f30 2019-12-05T05:47:00.503 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.494+0000 7fb544a69700 1 --2- 172.21.15.42:0/48566100 >> v2:172.21.15.40:3300/0 conn(0x7fb5400a1a20 0x7fb5400a39e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:47:00.503 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.494+0000 7fb545a6b700 1 --2- 172.21.15.42:0/48566100 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb5400a3f30 0x7fb54012f540 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fb538001210 tx=0x7fb538010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:00.504 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.494+0000 7fb545a6b700 1 --2- 172.21.15.42:0/48566100 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb5400a3f30 0x7fb54012f540 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fb538001210 tx=0x7fb538010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:00.504 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.494+0000 7fb545a6b700 1 --2- 172.21.15.42:0/48566100 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb5400a3f30 0x7fb54012f540 secure :-1 s=READY pgs=322 cs=0 l=1 rx=0x7fb538001210 tx=0x7fb538010b40).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:47:00.504 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.494+0000 7fb545a6b700 1 --2- 172.21.15.42:0/48566100 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb5400a3f30 0x7fb54012f540 secure :-1 s=THROTTLE_DONE pgs=322 cs=0 l=1 rx=0x7fb538001210 tx=0x7fb538010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:00.504 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.494+0000 7fb545a6b700 1 --2- 172.21.15.42:0/48566100 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb5400a3f30 0x7fb54012f540 secure :-1 s=THROTTLE_DONE pgs=322 cs=0 l=1 rx=0x7fb538001210 tx=0x7fb538010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:00.504 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.494+0000 7fb535ffb700 1 -- 172.21.15.42:0/48566100 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb538006ac0 con 0x7fb5400a3f30 2019-12-05T05:47:00.504 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.494+0000 7fb545a6b700 1 --2- 172.21.15.42:0/48566100 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb5400a3f30 0x7fb54012f540 secure :-1 s=THROTTLE_DONE pgs=322 cs=0 l=1 rx=0x7fb538001210 tx=0x7fb538010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:00.505 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.494+0000 7fb535ffb700 1 -- 172.21.15.42:0/48566100 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5380066e0 con 0x7fb5400a3f30 2019-12-05T05:47:00.505 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.494+0000 7fb546ccd700 1 -- 172.21.15.42:0/48566100 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb54007dfc0 con 0x7fb5400a3f30 2019-12-05T05:47:00.505 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.494+0000 7fb535ffb700 1 -- 172.21.15.42:0/48566100 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb538002730 con 0x7fb5400a3f30 2019-12-05T05:47:00.505 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.494+0000 7fb546ccd700 1 -- 172.21.15.42:0/48566100 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb54012fa90 con 0x7fb5400a3f30 2019-12-05T05:47:00.505 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.494+0000 7fb545a6b700 1 --2- 172.21.15.42:0/48566100 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb5400a3f30 0x7fb54012f540 secure :-1 s=THROTTLE_DONE pgs=322 cs=0 l=1 rx=0x7fb538001210 tx=0x7fb538010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:00.505 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.494+0000 7fb535ffb700 1 -- 172.21.15.42:0/48566100 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fb538002a40 con 0x7fb5400a3f30 2019-12-05T05:47:00.506 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.494+0000 7fb545a6b700 1 --2- 172.21.15.42:0/48566100 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb5400a3f30 0x7fb54012f540 secure :-1 s=THROTTLE_DONE pgs=322 cs=0 l=1 rx=0x7fb538001210 tx=0x7fb538010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:00.506 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.494+0000 7fb535ffb700 1 --2- 172.21.15.42:0/48566100 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb52003ae00 0x7fb52003d310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:47:00.506 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.494+0000 7fb535ffb700 1 -- 172.21.15.42:0/48566100 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 8525+0+0 (secure 0 0 0) 0x7fb53804b290 con 0x7fb5400a3f30 2019-12-05T05:47:00.506 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.494+0000 7fb54526a700 1 --2- 172.21.15.42:0/48566100 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb52003ae00 0x7fb52003d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:47:00.506 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.494+0000 7fb54526a700 1 --2- 172.21.15.42:0/48566100 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb52003ae00 0x7fb52003d310 crc :-1 s=READY pgs=625 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:47:00.507 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.502+0000 7fb546ccd700 1 -- 172.21.15.42:0/48566100 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb54005d110 con 0x7fb5400a3f30 2019-12-05T05:47:00.514 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.510+0000 7fb545a6b700 1 --2- 172.21.15.42:0/48566100 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb5400a3f30 0x7fb54012f540 secure :-1 s=THROTTLE_DONE pgs=322 cs=0 l=1 rx=0x7fb538001210 tx=0x7fb538010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:00.514 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.510+0000 7fb535ffb700 1 -- 172.21.15.42:0/48566100 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fb5380322f0 con 0x7fb5400a3f30 2019-12-05T05:47:00.684 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.682+0000 7fb546ccd700 1 -- 172.21.15.42:0/48566100 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb54005b8c0 con 0x7fb5400a3f30 2019-12-05T05:47:00.685 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.682+0000 7fb545a6b700 1 --2- 172.21.15.42:0/48566100 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb5400a3f30 0x7fb54012f540 secure :-1 s=THROTTLE_DONE pgs=322 cs=0 l=1 rx=0x7fb538001210 tx=0x7fb538010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:00.685 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.682+0000 7fb535ffb700 1 -- 172.21.15.42:0/48566100 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 178 v178) v1 ==== 95+0+2112 (secure 0 0 0) 0x7fb538032550 con 0x7fb5400a3f30 2019-12-05T05:47:00.687 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 178 2019-12-05T05:47:00.687 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:47:00.687 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":178,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6140,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/2146240383","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2146240383},{"type":"v1","addr":"172.21.15.40:6833","nonce":2146240383}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":178}],"filesystems":[{"mdsmap":{"epoch":177,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:45:43.664230+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":78,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6077},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6077":{"gid":6077,"name":"a","rank":0,"incarnation":174,"state":"up:active","state_seq":16,"addr":"172.21.15.42:6835/3757738536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3757738536},{"type":"v1","addr":"172.21.15.42:6835","nonce":3757738536}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:47:00.690 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.686+0000 7fb546ccd700 1 -- 172.21.15.42:0/48566100 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb52003ae00 msgr2=0x7fb52003d310 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:47:00.690 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.686+0000 7fb546ccd700 1 --2- 172.21.15.42:0/48566100 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb52003ae00 0x7fb52003d310 crc :-1 s=READY pgs=625 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:47:00.690 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.686+0000 7fb546ccd700 1 -- 172.21.15.42:0/48566100 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb5400a3f30 msgr2=0x7fb54012f540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:47:00.690 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.686+0000 7fb546ccd700 1 --2- 172.21.15.42:0/48566100 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb5400a3f30 0x7fb54012f540 secure :-1 s=READY pgs=322 cs=0 l=1 rx=0x7fb538001210 tx=0x7fb538010b40).stop 2019-12-05T05:47:00.690 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.686+0000 7fb546ccd700 1 -- 172.21.15.42:0/48566100 shutdown_connections 2019-12-05T05:47:00.690 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.686+0000 7fb546ccd700 1 --2- 172.21.15.42:0/48566100 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb52003ae00 0x7fb52003d310 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:47:00.691 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.686+0000 7fb546ccd700 1 --2- 172.21.15.42:0/48566100 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb5400a3f30 0x7fb54012f540 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:47:00.691 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.686+0000 7fb546ccd700 1 --2- 172.21.15.42:0/48566100 >> v2:172.21.15.40:3300/0 conn(0x7fb5400a1a20 0x7fb5400a39e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:47:00.691 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.686+0000 7fb546ccd700 1 -- 172.21.15.42:0/48566100 shutdown_connections 2019-12-05T05:47:00.691 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.686+0000 7fb546ccd700 1 -- 172.21.15.42:0/48566100 wait complete. 2019-12-05T05:47:00.691 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.686+0000 7fb546ccd700 1 -- 172.21.15.42:0/48566100 >> 172.21.15.42:0/48566100 conn(0x7fb5400a6b70 msgr2=0x7fb5400a9770 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:47:00.700 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.a (rank=0) 2019-12-05T05:47:00.700 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2019-12-05T05:47:00.700 INFO:teuthology.orchestra.run:waiting for 300 2019-12-05T05:47:00.713 INFO:tasks.ceph.mds.a:Stopped 2019-12-05T05:47:00.714 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:47:00.889 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.870+0000 7f3eaf296700 1 Processor -- start 2019-12-05T05:47:00.890 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.874+0000 7f3eaf296700 1 -- start start 2019-12-05T05:47:00.890 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.874+0000 7f3eaf296700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f3ea809fae0 0x7f3ea814fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:47:00.890 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.874+0000 7f3eaf296700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ea80a0710 con 0x7f3ea808df80 2019-12-05T05:47:00.890 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.874+0000 7f3eaf296700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ea807ef20 con 0x7f3ea80a7740 2019-12-05T05:47:00.890 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.874+0000 7f3eaf296700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ea80a0b00 con 0x7f3ea809fae0 2019-12-05T05:47:00.890 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.874+0000 7f3ead833700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f3ea80a7740 0x7f3ea807e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:41148/0 (socket says 172.21.15.42:41148) 2019-12-05T05:47:00.891 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.874+0000 7f3ead833700 1 -- 172.21.15.42:0/1082910378 learned_addr learned my addr 172.21.15.42:0/1082910378 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:47:00.891 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.874+0000 7f3eae034700 1 --2- 172.21.15.42:0/1082910378 >> v2:172.21.15.40:3300/0 conn(0x7f3ea809fae0 0x7f3ea814fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:47:00.891 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.874+0000 7f3eac831700 1 -- 172.21.15.42:0/1082910378 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3003568056 0 0) 0x7f3ea807ef20 con 0x7f3ea80a7740 2019-12-05T05:47:00.891 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.874+0000 7f3eac831700 1 -- 172.21.15.42:0/1082910378 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e940030b0 con 0x7f3ea80a7740 2019-12-05T05:47:00.891 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.874+0000 7f3eac831700 1 -- 172.21.15.42:0/1082910378 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1984866632 0 0) 0x7f3e940030b0 con 0x7f3ea80a7740 2019-12-05T05:47:00.891 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.874+0000 7f3eac831700 1 -- 172.21.15.42:0/1082910378 >> v2:172.21.15.40:3300/0 conn(0x7f3ea809fae0 msgr2=0x7f3ea814fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:47:00.891 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.874+0000 7f3eac831700 1 --2- 172.21.15.42:0/1082910378 >> v2:172.21.15.40:3300/0 conn(0x7f3ea809fae0 0x7f3ea814fff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:47:00.892 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.874+0000 7f3eac831700 1 -- 172.21.15.42:0/1082910378 >> v1:172.21.15.40:6789/0 conn(0x7f3ea808df80 legacy=0x7f3ea809f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:47:00.892 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.874+0000 7f3eac831700 1 -- 172.21.15.42:0/1082910378 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ea80a9ba0 con 0x7f3ea80a7740 2019-12-05T05:47:00.892 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.874+0000 7f3eac831700 1 -- 172.21.15.42:0/1082910378 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7f3ea4001ad0 con 0x7f3ea80a7740 2019-12-05T05:47:00.892 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.874+0000 7f3eac831700 1 -- 172.21.15.42:0/1082910378 >> v1:172.21.15.42:6789/0 conn(0x7f3ea80a7740 legacy=0x7f3ea807e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:47:00.892 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.874+0000 7f3eac831700 1 --2- 172.21.15.42:0/1082910378 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3e940036e0 0x7f3e94003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:47:00.893 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.874+0000 7f3eac831700 1 --2- 172.21.15.42:0/1082910378 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e94004420 0x7f3e94004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:47:00.893 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.874+0000 7f3eac831700 1 --2- 172.21.15.42:0/1082910378 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3e94004da0 0x7f3e9400b670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:47:00.893 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.874+0000 7f3eac831700 1 -- 172.21.15.42:0/1082910378 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ea80a0b00 con 0x7f3e94004420 2019-12-05T05:47:00.893 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.874+0000 7f3eac831700 1 -- 172.21.15.42:0/1082910378 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e9404fc60 con 0x7f3e940036e0 2019-12-05T05:47:00.893 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.874+0000 7f3eac831700 1 -- 172.21.15.42:0/1082910378 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e9404fea0 con 0x7f3e94004da0 2019-12-05T05:47:00.893 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.874+0000 7f3ead833700 1 --2- 172.21.15.42:0/1082910378 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3e940036e0 0x7f3e94003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:47:00.893 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.874+0000 7f3eae034700 1 --2- 172.21.15.42:0/1082910378 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3e94004da0 0x7f3e9400b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:47:00.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.874+0000 7f3ead032700 1 --2- 172.21.15.42:0/1082910378 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e94004420 0x7f3e94004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:47:00.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.878+0000 7f3ead833700 1 -- 172.21.15.42:0/1082910378 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3e94004da0 msgr2=0x7f3e9400b670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:47:00.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.878+0000 7f3ead833700 1 --2- 172.21.15.42:0/1082910378 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3e94004da0 0x7f3e9400b670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:47:00.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.878+0000 7f3ead833700 1 -- 172.21.15.42:0/1082910378 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e94004420 msgr2=0x7f3e94004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:47:00.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.878+0000 7f3ead833700 1 --2- 172.21.15.42:0/1082910378 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e94004420 0x7f3e94004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:47:00.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.878+0000 7f3ead833700 1 -- 172.21.15.42:0/1082910378 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e940501a0 con 0x7f3e940036e0 2019-12-05T05:47:00.895 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.878+0000 7f3eae034700 1 -- 172.21.15.42:0/1082910378 reap_dead start 2019-12-05T05:47:00.895 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.878+0000 7f3ead032700 1 --2- 172.21.15.42:0/1082910378 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e94004420 0x7f3e94004850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T05:47:00.895 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.878+0000 7f3ead833700 1 --2- 172.21.15.42:0/1082910378 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3e940036e0 0x7f3e94003b50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f3ea4001cd0 tx=0x7f3ea400c5c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:00.895 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.878+0000 7f3ead833700 1 --2- 172.21.15.42:0/1082910378 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3e940036e0 0x7f3e94003b50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f3ea4001cd0 tx=0x7f3ea400c5c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:00.895 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.878+0000 7f3ead833700 1 --2- 172.21.15.42:0/1082910378 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3e940036e0 0x7f3e94003b50 secure :-1 s=READY pgs=1454 cs=0 l=1 rx=0x7f3ea4001cd0 tx=0x7f3ea400c5c0).ready entity=mon.1 client_cookie=92c07a756a4fe88a server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:47:00.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.878+0000 7f3ead833700 1 --2- 172.21.15.42:0/1082910378 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3e940036e0 0x7f3e94003b50 secure :-1 s=THROTTLE_DONE pgs=1454 cs=0 l=1 rx=0x7f3ea4001cd0 tx=0x7f3ea400c5c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:00.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.878+0000 7f3eac831700 1 -- 172.21.15.42:0/1082910378 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3ea4005c60 con 0x7f3e940036e0 2019-12-05T05:47:00.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.878+0000 7f3ead833700 1 --2- 172.21.15.42:0/1082910378 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3e940036e0 0x7f3e94003b50 secure :-1 s=THROTTLE_DONE pgs=1454 cs=0 l=1 rx=0x7f3ea4001cd0 tx=0x7f3ea400c5c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:00.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.878+0000 7f3eac831700 1 -- 172.21.15.42:0/1082910378 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ea4005880 con 0x7f3e940036e0 2019-12-05T05:47:00.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.878+0000 7f3ead833700 1 --2- 172.21.15.42:0/1082910378 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3e940036e0 0x7f3e94003b50 secure :-1 s=THROTTLE_DONE pgs=1454 cs=0 l=1 rx=0x7f3ea4001cd0 tx=0x7f3ea400c5c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:00.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.878+0000 7f3eac831700 1 -- 172.21.15.42:0/1082910378 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3ea4004e90 con 0x7f3e940036e0 2019-12-05T05:47:00.897 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.878+0000 7f3eaf296700 1 -- 172.21.15.42:0/1082910378 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3e940036e0 msgr2=0x7f3e94003b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:47:00.897 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.878+0000 7f3eaf296700 1 --2- 172.21.15.42:0/1082910378 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3e940036e0 0x7f3e94003b50 secure :-1 s=READY pgs=1454 cs=0 l=1 rx=0x7f3ea4001cd0 tx=0x7f3ea400c5c0).stop 2019-12-05T05:47:00.897 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.878+0000 7f3eaf296700 1 -- 172.21.15.42:0/1082910378 shutdown_connections 2019-12-05T05:47:00.897 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.878+0000 7f3eaf296700 1 --2- 172.21.15.42:0/1082910378 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3e940036e0 0x7f3e94003b50 unknown :-1 s=CLOSED pgs=1454 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:47:00.897 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.878+0000 7f3eaf296700 1 -- 172.21.15.42:0/1082910378 shutdown_connections 2019-12-05T05:47:00.897 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.878+0000 7f3eaf296700 1 -- 172.21.15.42:0/1082910378 wait complete. 2019-12-05T05:47:00.897 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.878+0000 7f3eaf296700 1 -- 172.21.15.42:0/1082910378 >> 172.21.15.42:0/1082910378 conn(0x7f3ea80aabe0 msgr2=0x7f3ea80a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:47:00.898 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.882+0000 7f3eaf296700 1 Processor -- start 2019-12-05T05:47:00.898 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.882+0000 7f3eaf296700 1 -- start start 2019-12-05T05:47:00.898 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.882+0000 7f3eaf296700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3ea811b590 0x7f3ea811a440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:47:00.898 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.882+0000 7f3eaf296700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f3ea8072dc0 0x7f3ea8079980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:47:00.898 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.882+0000 7f3eaf296700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ea811b9c0 con 0x7f3ea8124160 2019-12-05T05:47:00.898 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.882+0000 7f3eaf296700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ea8126620 con 0x7f3ea8072dc0 2019-12-05T05:47:00.898 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.882+0000 7f3eaf296700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ea81268c0 con 0x7f3ea811b590 2019-12-05T05:47:00.899 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.882+0000 7f3ead833700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f3ea8124160 0x7f3ea80725c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:41160/0 (socket says 172.21.15.42:41160) 2019-12-05T05:47:00.899 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.882+0000 7f3ead833700 1 -- 172.21.15.42:0/2789599751 learned_addr learned my addr 172.21.15.42:0/2789599751 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:47:00.899 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.882+0000 7f3ead032700 1 --2- 172.21.15.42:0/2789599751 >> v2:172.21.15.42:3300/0 conn(0x7f3ea811b590 0x7f3ea811a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:47:00.899 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.882+0000 7f3eae034700 1 --2- 172.21.15.42:0/2789599751 >> v2:172.21.15.40:3300/0 conn(0x7f3ea8072dc0 0x7f3ea8079980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:47:00.899 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.882+0000 7f3e9e7fc700 1 -- 172.21.15.42:0/2789599751 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2745780290 0 0) 0x7f3ea811b9c0 con 0x7f3ea8124160 2019-12-05T05:47:00.899 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.882+0000 7f3e9e7fc700 1 -- 172.21.15.42:0/2789599751 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e900030b0 con 0x7f3ea8124160 2019-12-05T05:47:00.899 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.886+0000 7f3ead032700 1 -- 172.21.15.42:0/2789599751 >> v2:172.21.15.40:3300/0 conn(0x7f3ea8072dc0 msgr2=0x7f3ea8079980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:47:00.900 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.886+0000 7f3ead032700 1 --2- 172.21.15.42:0/2789599751 >> v2:172.21.15.40:3300/0 conn(0x7f3ea8072dc0 0x7f3ea8079980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:47:00.900 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.886+0000 7f3ead032700 1 -- 172.21.15.42:0/2789599751 >> v1:172.21.15.42:6789/0 conn(0x7f3ea8124160 legacy=0x7f3ea80725c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:47:00.900 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.886+0000 7f3ead032700 1 -- 172.21.15.42:0/2789599751 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ea811a990 con 0x7f3ea811b590 2019-12-05T05:47:00.900 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.886+0000 7f3ead032700 1 --2- 172.21.15.42:0/2789599751 >> v2:172.21.15.42:3300/0 conn(0x7f3ea811b590 0x7f3ea811a440 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f3ea811b350 tx=0x7f3e9800c340).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:00.900 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.886+0000 7f3ead032700 1 --2- 172.21.15.42:0/2789599751 >> v2:172.21.15.42:3300/0 conn(0x7f3ea811b590 0x7f3ea811a440 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f3ea811b350 tx=0x7f3e9800c340).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:00.900 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.886+0000 7f3ead032700 1 --2- 172.21.15.42:0/2789599751 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3ea811b590 0x7f3ea811a440 secure :-1 s=READY pgs=1456 cs=0 l=1 rx=0x7f3ea811b350 tx=0x7f3e9800c340).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:47:00.900 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.886+0000 7f3ead032700 1 --2- 172.21.15.42:0/2789599751 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3ea811b590 0x7f3ea811a440 secure :-1 s=THROTTLE_DONE pgs=1456 cs=0 l=1 rx=0x7f3ea811b350 tx=0x7f3e9800c340).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:00.901 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.886+0000 7f3e9e7fc700 1 -- 172.21.15.42:0/2789599751 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3e9800c9b0 con 0x7f3ea811b590 2019-12-05T05:47:00.901 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.886+0000 7f3ead032700 1 --2- 172.21.15.42:0/2789599751 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3ea811b590 0x7f3ea811a440 secure :-1 s=THROTTLE_DONE pgs=1456 cs=0 l=1 rx=0x7f3ea811b350 tx=0x7f3e9800c340).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:00.901 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.886+0000 7f3e9e7fc700 1 -- 172.21.15.42:0/2789599751 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e9800c5d0 con 0x7f3ea811b590 2019-12-05T05:47:00.901 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.886+0000 7f3eaf296700 1 -- 172.21.15.42:0/2789599751 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ea80a5980 con 0x7f3ea811b590 2019-12-05T05:47:00.901 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.886+0000 7f3ead032700 1 --2- 172.21.15.42:0/2789599751 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3ea811b590 0x7f3ea811a440 secure :-1 s=THROTTLE_DONE pgs=1456 cs=0 l=1 rx=0x7f3ea811b350 tx=0x7f3e9800c340).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:00.901 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.886+0000 7f3e9e7fc700 1 -- 172.21.15.42:0/2789599751 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3e98010ef0 con 0x7f3ea811b590 2019-12-05T05:47:00.901 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.886+0000 7f3eaf296700 1 -- 172.21.15.42:0/2789599751 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ea81262a0 con 0x7f3ea811b590 2019-12-05T05:47:00.902 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.886+0000 7f3ead032700 1 --2- 172.21.15.42:0/2789599751 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3ea811b590 0x7f3ea811a440 secure :-1 s=THROTTLE_DONE pgs=1456 cs=0 l=1 rx=0x7f3ea811b350 tx=0x7f3e9800c340).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:00.903 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.886+0000 7f3e9e7fc700 1 -- 172.21.15.42:0/2789599751 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f3e98007430 con 0x7f3ea811b590 2019-12-05T05:47:00.903 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.886+0000 7f3ead032700 1 --2- 172.21.15.42:0/2789599751 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3ea811b590 0x7f3ea811a440 secure :-1 s=THROTTLE_DONE pgs=1456 cs=0 l=1 rx=0x7f3ea811b350 tx=0x7f3e9800c340).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:00.903 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.886+0000 7f3e9e7fc700 1 --2- 172.21.15.42:0/2789599751 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3e9003b230 0x7f3e9003d740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:47:00.904 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.886+0000 7f3e9e7fc700 1 -- 172.21.15.42:0/2789599751 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 8525+0+0 (secure 0 0 0) 0x7f3e9804b1d0 con 0x7f3ea811b590 2019-12-05T05:47:00.904 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.886+0000 7f3ead833700 1 --2- 172.21.15.42:0/2789599751 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3e9003b230 0x7f3e9003d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:47:00.904 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.886+0000 7f3eaf296700 1 -- 172.21.15.42:0/2789599751 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ea805d110 con 0x7f3ea811b590 2019-12-05T05:47:00.904 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.890+0000 7f3ead833700 1 --2- 172.21.15.42:0/2789599751 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3e9003b230 0x7f3e9003d740 crc :-1 s=READY pgs=626 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:47:00.904 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.894+0000 7f3ead032700 1 --2- 172.21.15.42:0/2789599751 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3ea811b590 0x7f3ea811a440 secure :-1 s=THROTTLE_DONE pgs=1456 cs=0 l=1 rx=0x7f3ea811b350 tx=0x7f3e9800c340).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:00.904 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:00.894+0000 7f3e9e7fc700 1 -- 172.21.15.42:0/2789599751 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f3ea811a990 con 0x7f3ea811b590 2019-12-05T05:47:01.068 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:01.066+0000 7f3eaf296700 1 -- 172.21.15.42:0/2789599751 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3ea805b8c0 con 0x7f3ea811b590 2019-12-05T05:47:01.069 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:01.066+0000 7f3ead032700 1 --2- 172.21.15.42:0/2789599751 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3ea811b590 0x7f3ea811a440 secure :-1 s=THROTTLE_DONE pgs=1456 cs=0 l=1 rx=0x7f3ea811b350 tx=0x7f3e9800c340).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:01.069 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:01.066+0000 7f3e9e7fc700 1 -- 172.21.15.42:0/2789599751 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 178 v178) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f3e980321d0 con 0x7f3ea811b590 2019-12-05T05:47:01.072 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 178 2019-12-05T05:47:01.072 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:47:01.072 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":178,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6140,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/2146240383","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2146240383},{"type":"v1","addr":"172.21.15.40:6833","nonce":2146240383}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":178}],"filesystems":[{"mdsmap":{"epoch":177,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:45:43.664230+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":78,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6077},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6077":{"gid":6077,"name":"a","rank":0,"incarnation":174,"state":"up:active","state_seq":16,"addr":"172.21.15.42:6835/3757738536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3757738536},{"type":"v1","addr":"172.21.15.42:6835","nonce":3757738536}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:47:01.075 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:01.070+0000 7f3eaf296700 1 -- 172.21.15.42:0/2789599751 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3e9003b230 msgr2=0x7f3e9003d740 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:47:01.075 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:01.070+0000 7f3eaf296700 1 --2- 172.21.15.42:0/2789599751 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3e9003b230 0x7f3e9003d740 crc :-1 s=READY pgs=626 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:47:01.075 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:01.070+0000 7f3eaf296700 1 -- 172.21.15.42:0/2789599751 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3ea811b590 msgr2=0x7f3ea811a440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:47:01.076 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:01.070+0000 7f3eaf296700 1 --2- 172.21.15.42:0/2789599751 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3ea811b590 0x7f3ea811a440 secure :-1 s=READY pgs=1456 cs=0 l=1 rx=0x7f3ea811b350 tx=0x7f3e9800c340).stop 2019-12-05T05:47:01.076 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:01.070+0000 7f3eaf296700 1 -- 172.21.15.42:0/2789599751 shutdown_connections 2019-12-05T05:47:01.076 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:01.070+0000 7f3eaf296700 1 --2- 172.21.15.42:0/2789599751 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3e9003b230 0x7f3e9003d740 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:47:01.076 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:01.070+0000 7f3eaf296700 1 --2- 172.21.15.42:0/2789599751 >> v2:172.21.15.40:3300/0 conn(0x7f3ea8072dc0 0x7f3ea8079980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:47:01.076 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:01.070+0000 7f3eaf296700 1 --2- 172.21.15.42:0/2789599751 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3ea811b590 0x7f3ea811a440 unknown :-1 s=CLOSED pgs=1456 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:47:01.076 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:01.070+0000 7f3eaf296700 1 -- 172.21.15.42:0/2789599751 shutdown_connections 2019-12-05T05:47:01.077 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:01.070+0000 7f3eaf296700 1 -- 172.21.15.42:0/2789599751 wait complete. 2019-12-05T05:47:01.077 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:01.070+0000 7f3eaf296700 1 -- 172.21.15.42:0/2789599751 >> 172.21.15.42:0/2789599751 conn(0x7f3ea80a5c30 msgr2=0x7f3ea8059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:47:01.084 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T05:47:03.085 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:47:03.193 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:03.182+0000 7fa8bea29700 1 Processor -- start 2019-12-05T05:47:03.193 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:03.182+0000 7fa8bea29700 1 -- start start 2019-12-05T05:47:03.193 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:03.186+0000 7fa8bea29700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa8b80a7730 0x7fa8b808df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:47:03.193 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:03.186+0000 7fa8bea29700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa8b807e710 0x7fa8b807eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:47:03.193 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:03.186+0000 7fa8bea29700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8b80a0700 con 0x7fa8b809f2d0 2019-12-05T05:47:03.194 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:03.186+0000 7fa8bea29700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8b80a0af0 con 0x7fa8b80a7730 2019-12-05T05:47:03.194 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:03.186+0000 7fa8bea29700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8b808e5a0 con 0x7fa8b807e710 2019-12-05T05:47:03.194 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:03.186+0000 7fa8b7fff700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa8b807e710 0x7fa8b807eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:47:03.194 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:03.186+0000 7fa8bcfc6700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa8b80a7730 0x7fa8b808df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:47:03.194 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:03.186+0000 7fa8b7fff700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa8b807e710 0x7fa8b807eb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:41602/0 (socket says 172.21.15.42:41602) 2019-12-05T05:47:03.194 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:03.186+0000 7fa8b7fff700 1 -- 172.21.15.42:0/1263060445 learned_addr learned my addr 172.21.15.42:0/1263060445 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:47:03.194 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:03.186+0000 7fa8b7fff700 1 -- 172.21.15.42:0/1263060445 >> v2:172.21.15.40:3300/0 conn(0x7fa8b80a7730 msgr2=0x7fa8b808df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:47:03.195 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:03.186+0000 7fa8b7fff700 1 --2- 172.21.15.42:0/1263060445 >> v2:172.21.15.40:3300/0 conn(0x7fa8b80a7730 0x7fa8b808df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:47:03.195 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:03.186+0000 7fa8b77fe700 1 -- 172.21.15.42:0/1263060445 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3155989249 0 0) 0x7fa8b80a0700 con 0x7fa8b809f2d0 2019-12-05T05:47:03.195 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:03.186+0000 7fa8b7fff700 1 -- 172.21.15.42:0/1263060445 >> v1:172.21.15.40:6789/0 conn(0x7fa8b809f2d0 legacy=0x7fa8b809f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:47:03.195 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:03.186+0000 7fa8b7fff700 1 -- 172.21.15.42:0/1263060445 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8b807f090 con 0x7fa8b807e710 2019-12-05T05:47:03.195 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:03.186+0000 7fa8bcfc6700 1 --2- 172.21.15.42:0/1263060445 >> v2:172.21.15.40:3300/0 conn(0x7fa8b80a7730 0x7fa8b808df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:47:03.195 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:03.186+0000 7fa8b7fff700 1 --2- 172.21.15.42:0/1263060445 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa8b807e710 0x7fa8b807eb40 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fa8a0008790 tx=0x7fa8a000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:03.195 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:03.186+0000 7fa8b7fff700 1 --2- 172.21.15.42:0/1263060445 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa8b807e710 0x7fa8b807eb40 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fa8a0008790 tx=0x7fa8a000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:03.196 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:03.186+0000 7fa8b7fff700 1 --2- 172.21.15.42:0/1263060445 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa8b807e710 0x7fa8b807eb40 secure :-1 s=READY pgs=323 cs=0 l=1 rx=0x7fa8a0008790 tx=0x7fa8a000fd30).ready entity=mon.2 client_cookie=c8e05903de6a25b2 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:47:03.196 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:03.186+0000 7fa8b7fff700 1 --2- 172.21.15.42:0/1263060445 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa8b807e710 0x7fa8b807eb40 secure :-1 s=THROTTLE_DONE pgs=323 cs=0 l=1 rx=0x7fa8a0008790 tx=0x7fa8a000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:03.196 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:03.186+0000 7fa8b77fe700 1 -- 172.21.15.42:0/1263060445 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa8a00113b0 con 0x7fa8b807e710 2019-12-05T05:47:03.196 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:03.186+0000 7fa8b7fff700 1 --2- 172.21.15.42:0/1263060445 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa8b807e710 0x7fa8b807eb40 secure :-1 s=THROTTLE_DONE pgs=323 cs=0 l=1 rx=0x7fa8a0008790 tx=0x7fa8a000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:03.196 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:03.186+0000 7fa8b7fff700 1 --2- 172.21.15.42:0/1263060445 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa8b807e710 0x7fa8b807eb40 secure :-1 s=THROTTLE_DONE pgs=323 cs=0 l=1 rx=0x7fa8a0008790 tx=0x7fa8a000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:03.196 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:03.186+0000 7fa8b77fe700 1 -- 172.21.15.42:0/1263060445 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8a0008bb0 con 0x7fa8b807e710 2019-12-05T05:47:03.196 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:03.186+0000 7fa8b77fe700 1 -- 172.21.15.42:0/1263060445 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa8a0011990 con 0x7fa8b807e710 2019-12-05T05:47:03.197 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:03.186+0000 7fa8bea29700 1 -- 172.21.15.42:0/1263060445 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa8b807e710 msgr2=0x7fa8b807eb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:47:03.197 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:03.186+0000 7fa8bea29700 1 --2- 172.21.15.42:0/1263060445 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa8b807e710 0x7fa8b807eb40 secure :-1 s=READY pgs=323 cs=0 l=1 rx=0x7fa8a0008790 tx=0x7fa8a000fd30).stop 2019-12-05T05:47:03.197 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:03.190+0000 7fa8bea29700 1 -- 172.21.15.42:0/1263060445 shutdown_connections 2019-12-05T05:47:03.197 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:03.190+0000 7fa8bea29700 1 --2- 172.21.15.42:0/1263060445 >> v2:172.21.15.40:3300/0 conn(0x7fa8b80a7730 0x7fa8b808df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:47:03.197 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:03.190+0000 7fa8bea29700 1 --2- 172.21.15.42:0/1263060445 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa8b807e710 0x7fa8b807eb40 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:47:03.197 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:03.190+0000 7fa8bea29700 1 -- 172.21.15.42:0/1263060445 shutdown_connections 2019-12-05T05:47:03.197 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:03.190+0000 7fa8bea29700 1 -- 172.21.15.42:0/1263060445 wait complete. 2019-12-05T05:47:03.198 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:03.190+0000 7fa8bea29700 1 -- 172.21.15.42:0/1263060445 >> 172.21.15.42:0/1263060445 conn(0x7fa8b80aabd0 msgr2=0x7fa8b80a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:47:03.201 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:03.190+0000 7fa8bea29700 1 Processor -- start 2019-12-05T05:47:03.201 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:03.190+0000 7fa8bea29700 1 -- start start 2019-12-05T05:47:03.201 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:03.190+0000 7fa8bea29700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa8b811f350 0x7fa8b812f990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:47:03.201 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:03.190+0000 7fa8bea29700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa8b811d1a0 0x7fa8b812fee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:47:03.201 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:03.190+0000 7fa8bea29700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8b81161b0 con 0x7fa8b8132cb0 2019-12-05T05:47:03.201 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:03.190+0000 7fa8bea29700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8b811c050 con 0x7fa8b811f350 2019-12-05T05:47:03.202 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:03.190+0000 7fa8bea29700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8b811caa0 con 0x7fa8b811d1a0 2019-12-05T05:47:03.202 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:03.190+0000 7fa8b7fff700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa8b811d1a0 0x7fa8b812fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:47:03.202 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:03.190+0000 7fa8bcfc6700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa8b811f350 0x7fa8b812f990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:47:03.202 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:03.194+0000 7fa8b7fff700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa8b811d1a0 0x7fa8b812fee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:41608/0 (socket says 172.21.15.42:41608) 2019-12-05T05:47:03.202 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:03.194+0000 7fa8b7fff700 1 -- 172.21.15.42:0/4214211427 learned_addr learned my addr 172.21.15.42:0/4214211427 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:47:03.202 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:03.194+0000 7fa8bcfc6700 1 --2- 172.21.15.42:0/4214211427 >> v2:172.21.15.40:3300/0 conn(0x7fa8b811f350 0x7fa8b812f990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.42:57190/0 (socket says 172.21.15.42:57190) 2019-12-05T05:47:03.202 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:03.194+0000 7fa8b7fff700 1 -- 172.21.15.42:0/4214211427 >> v2:172.21.15.40:3300/0 conn(0x7fa8b811f350 msgr2=0x7fa8b812f990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:47:03.203 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:03.194+0000 7fa8b7fff700 1 --2- 172.21.15.42:0/4214211427 >> v2:172.21.15.40:3300/0 conn(0x7fa8b811f350 0x7fa8b812f990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:47:03.203 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:03.194+0000 7fa8b7fff700 1 -- 172.21.15.42:0/4214211427 >> v1:172.21.15.40:6789/0 conn(0x7fa8b8132cb0 legacy=0x7fa8b811d6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:47:03.203 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:03.194+0000 7fa8b7fff700 1 -- 172.21.15.42:0/4214211427 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8b811c720 con 0x7fa8b811d1a0 2019-12-05T05:47:03.203 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:03.194+0000 7fa8bcfc6700 1 --2- 172.21.15.42:0/4214211427 >> v2:172.21.15.40:3300/0 conn(0x7fa8b811f350 0x7fa8b812f990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:47:03.203 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:03.194+0000 7fa8b7fff700 1 --2- 172.21.15.42:0/4214211427 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa8b811d1a0 0x7fa8b812fee0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fa8a0008790 tx=0x7fa8a0011f20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:03.203 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:03.194+0000 7fa8b7fff700 1 --2- 172.21.15.42:0/4214211427 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa8b811d1a0 0x7fa8b812fee0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fa8a0008790 tx=0x7fa8a0011f20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:03.203 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:03.194+0000 7fa8b7fff700 1 --2- 172.21.15.42:0/4214211427 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa8b811d1a0 0x7fa8b812fee0 secure :-1 s=READY pgs=324 cs=0 l=1 rx=0x7fa8a0008790 tx=0x7fa8a0011f20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:47:03.204 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:03.194+0000 7fa8b7fff700 1 --2- 172.21.15.42:0/4214211427 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa8b811d1a0 0x7fa8b812fee0 secure :-1 s=THROTTLE_DONE pgs=324 cs=0 l=1 rx=0x7fa8a0008790 tx=0x7fa8a0011f20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:03.207 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:03.194+0000 7fa8b7fff700 1 --2- 172.21.15.42:0/4214211427 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa8b811d1a0 0x7fa8b812fee0 secure :-1 s=THROTTLE_DONE pgs=324 cs=0 l=1 rx=0x7fa8a0008790 tx=0x7fa8a0011f20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:03.207 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:03.194+0000 7fa8b57fa700 1 -- 172.21.15.42:0/4214211427 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa8a0007340 con 0x7fa8b811d1a0 2019-12-05T05:47:03.207 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:03.194+0000 7fa8b7fff700 1 --2- 172.21.15.42:0/4214211427 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa8b811d1a0 0x7fa8b812fee0 secure :-1 s=THROTTLE_DONE pgs=324 cs=0 l=1 rx=0x7fa8a0008790 tx=0x7fa8a0011f20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:03.207 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:03.194+0000 7fa8b57fa700 1 -- 172.21.15.42:0/4214211427 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8a00115b0 con 0x7fa8b811d1a0 2019-12-05T05:47:03.207 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:03.194+0000 7fa8bea29700 1 -- 172.21.15.42:0/4214211427 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8b81330e0 con 0x7fa8b811d1a0 2019-12-05T05:47:03.208 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:03.194+0000 7fa8b57fa700 1 -- 172.21.15.42:0/4214211427 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa8a0005540 con 0x7fa8b811d1a0 2019-12-05T05:47:03.208 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:03.194+0000 7fa8bea29700 1 -- 172.21.15.42:0/4214211427 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8b8133610 con 0x7fa8b811d1a0 2019-12-05T05:47:03.208 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:03.194+0000 7fa8b7fff700 1 --2- 172.21.15.42:0/4214211427 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa8b811d1a0 0x7fa8b812fee0 secure :-1 s=THROTTLE_DONE pgs=324 cs=0 l=1 rx=0x7fa8a0008790 tx=0x7fa8a0011f20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:03.208 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:03.194+0000 7fa8b57fa700 1 -- 172.21.15.42:0/4214211427 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fa8a0005850 con 0x7fa8b811d1a0 2019-12-05T05:47:03.208 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:03.194+0000 7fa8b7fff700 1 --2- 172.21.15.42:0/4214211427 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa8b811d1a0 0x7fa8b812fee0 secure :-1 s=THROTTLE_DONE pgs=324 cs=0 l=1 rx=0x7fa8a0008790 tx=0x7fa8a0011f20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:03.208 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:03.194+0000 7fa8b57fa700 1 --2- 172.21.15.42:0/4214211427 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa89c03ad10 0x7fa89c03d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:47:03.209 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:03.194+0000 7fa8b57fa700 1 -- 172.21.15.42:0/4214211427 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 8525+0+0 (secure 0 0 0) 0x7fa8a004b1e0 con 0x7fa8b811d1a0 2019-12-05T05:47:03.209 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:03.198+0000 7fa8bcfc6700 1 --2- 172.21.15.42:0/4214211427 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa89c03ad10 0x7fa89c03d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:47:03.209 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:03.198+0000 7fa8bcfc6700 1 --2- 172.21.15.42:0/4214211427 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa89c03ad10 0x7fa89c03d220 crc :-1 s=READY pgs=627 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:47:03.209 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:03.202+0000 7fa8bea29700 1 -- 172.21.15.42:0/4214211427 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8b805d100 con 0x7fa8b811d1a0 2019-12-05T05:47:03.215 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:03.210+0000 7fa8b7fff700 1 --2- 172.21.15.42:0/4214211427 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa8b811d1a0 0x7fa8b812fee0 secure :-1 s=THROTTLE_DONE pgs=324 cs=0 l=1 rx=0x7fa8a0008790 tx=0x7fa8a0011f20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:03.215 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:03.210+0000 7fa8b57fa700 1 -- 172.21.15.42:0/4214211427 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fa8a0032260 con 0x7fa8b811d1a0 2019-12-05T05:47:03.382 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:03.378+0000 7fa8bea29700 1 -- 172.21.15.42:0/4214211427 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa8b805b8b0 con 0x7fa8b811d1a0 2019-12-05T05:47:03.383 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:03.382+0000 7fa8b7fff700 1 --2- 172.21.15.42:0/4214211427 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa8b811d1a0 0x7fa8b812fee0 secure :-1 s=THROTTLE_DONE pgs=324 cs=0 l=1 rx=0x7fa8a0008790 tx=0x7fa8a0011f20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:03.384 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:03.382+0000 7fa8b57fa700 1 -- 172.21.15.42:0/4214211427 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 178 v178) v1 ==== 95+0+2112 (secure 0 0 0) 0x7fa8a0007f50 con 0x7fa8b811d1a0 2019-12-05T05:47:03.385 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 178 2019-12-05T05:47:03.386 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:47:03.386 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":178,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6140,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/2146240383","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2146240383},{"type":"v1","addr":"172.21.15.40:6833","nonce":2146240383}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":178}],"filesystems":[{"mdsmap":{"epoch":177,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:45:43.664230+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":78,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6077},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6077":{"gid":6077,"name":"a","rank":0,"incarnation":174,"state":"up:active","state_seq":16,"addr":"172.21.15.42:6835/3757738536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3757738536},{"type":"v1","addr":"172.21.15.42:6835","nonce":3757738536}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:47:03.388 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:03.386+0000 7fa8bea29700 1 -- 172.21.15.42:0/4214211427 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa89c03ad10 msgr2=0x7fa89c03d220 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:47:03.389 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:03.386+0000 7fa8bea29700 1 --2- 172.21.15.42:0/4214211427 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa89c03ad10 0x7fa89c03d220 crc :-1 s=READY pgs=627 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:47:03.389 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:03.386+0000 7fa8bea29700 1 -- 172.21.15.42:0/4214211427 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa8b811d1a0 msgr2=0x7fa8b812fee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:47:03.389 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:03.386+0000 7fa8bea29700 1 --2- 172.21.15.42:0/4214211427 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa8b811d1a0 0x7fa8b812fee0 secure :-1 s=READY pgs=324 cs=0 l=1 rx=0x7fa8a0008790 tx=0x7fa8a0011f20).stop 2019-12-05T05:47:03.389 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:03.386+0000 7fa8bea29700 1 -- 172.21.15.42:0/4214211427 shutdown_connections 2019-12-05T05:47:03.389 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:03.386+0000 7fa8bea29700 1 --2- 172.21.15.42:0/4214211427 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa89c03ad10 0x7fa89c03d220 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:47:03.389 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:03.386+0000 7fa8bea29700 1 --2- 172.21.15.42:0/4214211427 >> v2:172.21.15.40:3300/0 conn(0x7fa8b811f350 0x7fa8b812f990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:47:03.390 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:03.386+0000 7fa8bea29700 1 --2- 172.21.15.42:0/4214211427 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa8b811d1a0 0x7fa8b812fee0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:47:03.390 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:03.386+0000 7fa8bea29700 1 -- 172.21.15.42:0/4214211427 shutdown_connections 2019-12-05T05:47:03.390 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:03.386+0000 7fa8bea29700 1 -- 172.21.15.42:0/4214211427 wait complete. 2019-12-05T05:47:03.390 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:03.386+0000 7fa8bea29700 1 -- 172.21.15.42:0/4214211427 >> 172.21.15.42:0/4214211427 conn(0x7fa8b809f3e0 msgr2=0x7fa8b8141350 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:47:03.399 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T05:47:05.400 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:47:05.505 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:05.498+0000 7f66e187c700 1 Processor -- start 2019-12-05T05:47:05.505 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:05.498+0000 7f66e187c700 1 -- start start 2019-12-05T05:47:05.505 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:05.498+0000 7f66e187c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f66dc0a7740 0x7f66dc08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:47:05.506 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:05.498+0000 7f66e187c700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f66dc09fae0 0x7f66dc07eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:47:05.506 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:05.498+0000 7f66e187c700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66dc0a0710 con 0x7f66dc07e720 2019-12-05T05:47:05.506 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:05.498+0000 7f66e187c700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66dc0a0b00 con 0x7f66dc0a7740 2019-12-05T05:47:05.506 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:05.498+0000 7f66e187c700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66dc08e5b0 con 0x7f66dc09fae0 2019-12-05T05:47:05.506 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:05.498+0000 7f66db7fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f66dc0a7740 0x7f66dc08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:47:05.507 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:05.498+0000 7f66daffd700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f66dc07e720 0x7f66dc09f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:50630/0 (socket says 172.21.15.42:50630) 2019-12-05T05:47:05.507 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:05.498+0000 7f66daffd700 1 -- 172.21.15.42:0/1371209321 learned_addr learned my addr 172.21.15.42:0/1371209321 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:47:05.507 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:05.498+0000 7f66dbfff700 1 --2- 172.21.15.42:0/1371209321 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f66dc09fae0 0x7f66dc07eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:47:05.507 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:05.498+0000 7f66da7fc700 1 -- 172.21.15.42:0/1371209321 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4211288795 0 0) 0x7f66dc0a0710 con 0x7f66dc07e720 2019-12-05T05:47:05.507 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:05.498+0000 7f66db7fe700 1 -- 172.21.15.42:0/1371209321 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f66dc09fae0 msgr2=0x7f66dc07eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:47:05.507 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:05.498+0000 7f66db7fe700 1 --2- 172.21.15.42:0/1371209321 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f66dc09fae0 0x7f66dc07eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:47:05.507 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:05.498+0000 7f66db7fe700 1 -- 172.21.15.42:0/1371209321 >> v1:172.21.15.40:6789/0 conn(0x7f66dc07e720 legacy=0x7f66dc09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:47:05.508 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:05.498+0000 7f66db7fe700 1 -- 172.21.15.42:0/1371209321 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66dc07f0a0 con 0x7f66dc0a7740 2019-12-05T05:47:05.508 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:05.498+0000 7f66db7fe700 1 --2- 172.21.15.42:0/1371209321 >> v2:172.21.15.42:3300/0 conn(0x7f66dc0a7740 0x7f66dc08df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f66c4008790 tx=0x7f66c400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:05.508 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:05.498+0000 7f66db7fe700 1 --2- 172.21.15.42:0/1371209321 >> v2:172.21.15.42:3300/0 conn(0x7f66dc0a7740 0x7f66dc08df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f66c4008790 tx=0x7f66c400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:05.508 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:05.498+0000 7f66db7fe700 1 --2- 172.21.15.42:0/1371209321 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f66dc0a7740 0x7f66dc08df80 secure :-1 s=READY pgs=1457 cs=0 l=1 rx=0x7f66c4008790 tx=0x7f66c400fc70).ready entity=mon.1 client_cookie=1fd8d8092b3bb191 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:47:05.508 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:05.498+0000 7f66db7fe700 1 --2- 172.21.15.42:0/1371209321 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f66dc0a7740 0x7f66dc08df80 secure :-1 s=THROTTLE_DONE pgs=1457 cs=0 l=1 rx=0x7f66c4008790 tx=0x7f66c400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:05.508 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:05.498+0000 7f66da7fc700 1 -- 172.21.15.42:0/1371209321 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f66c4008bb0 con 0x7f66dc0a7740 2019-12-05T05:47:05.509 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:05.498+0000 7f66db7fe700 1 --2- 172.21.15.42:0/1371209321 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f66dc0a7740 0x7f66dc08df80 secure :-1 s=THROTTLE_DONE pgs=1457 cs=0 l=1 rx=0x7f66c4008790 tx=0x7f66c400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:05.509 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:05.498+0000 7f66db7fe700 1 --2- 172.21.15.42:0/1371209321 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f66dc0a7740 0x7f66dc08df80 secure :-1 s=THROTTLE_DONE pgs=1457 cs=0 l=1 rx=0x7f66c4008790 tx=0x7f66c400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:05.509 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:05.502+0000 7f66da7fc700 1 -- 172.21.15.42:0/1371209321 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66c40087d0 con 0x7f66dc0a7740 2019-12-05T05:47:05.509 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:05.502+0000 7f66da7fc700 1 -- 172.21.15.42:0/1371209321 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f66c4011860 con 0x7f66dc0a7740 2019-12-05T05:47:05.509 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:05.502+0000 7f66e187c700 1 -- 172.21.15.42:0/1371209321 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f66dc0a7740 msgr2=0x7f66dc08df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:47:05.509 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:05.502+0000 7f66e187c700 1 --2- 172.21.15.42:0/1371209321 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f66dc0a7740 0x7f66dc08df80 secure :-1 s=READY pgs=1457 cs=0 l=1 rx=0x7f66c4008790 tx=0x7f66c400fc70).stop 2019-12-05T05:47:05.509 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:05.502+0000 7f66e187c700 1 -- 172.21.15.42:0/1371209321 shutdown_connections 2019-12-05T05:47:05.509 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:05.502+0000 7f66e187c700 1 --2- 172.21.15.42:0/1371209321 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f66dc09fae0 0x7f66dc07eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:47:05.510 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:05.502+0000 7f66e187c700 1 --2- 172.21.15.42:0/1371209321 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f66dc0a7740 0x7f66dc08df80 unknown :-1 s=CLOSED pgs=1457 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:47:05.510 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:05.502+0000 7f66e187c700 1 -- 172.21.15.42:0/1371209321 shutdown_connections 2019-12-05T05:47:05.510 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:05.502+0000 7f66e187c700 1 -- 172.21.15.42:0/1371209321 wait complete. 2019-12-05T05:47:05.510 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:05.502+0000 7f66e187c700 1 -- 172.21.15.42:0/1371209321 >> 172.21.15.42:0/1371209321 conn(0x7f66dc0aabe0 msgr2=0x7f66dc0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:47:05.513 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:05.502+0000 7f66e187c700 1 Processor -- start 2019-12-05T05:47:05.513 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:05.502+0000 7f66e187c700 1 -- start start 2019-12-05T05:47:05.514 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:05.502+0000 7f66e187c700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f66dc11a3c0 0x7f66dc116a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:47:05.514 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:05.502+0000 7f66e187c700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f66dc11c810 0x7f66dc11cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:47:05.514 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:05.502+0000 7f66e187c700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66dc11b120 con 0x7f66dc121fd0 2019-12-05T05:47:05.514 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:05.502+0000 7f66e187c700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66dc11a940 con 0x7f66dc11c810 2019-12-05T05:47:05.514 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:05.502+0000 7f66e187c700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66dc11abe0 con 0x7f66dc11a3c0 2019-12-05T05:47:05.514 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:05.502+0000 7f66daffd700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f66dc121fd0 0x7f66dc132590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:41196/0 (socket says 172.21.15.42:41196) 2019-12-05T05:47:05.515 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:05.502+0000 7f66daffd700 1 -- 172.21.15.42:0/286937519 learned_addr learned my addr 172.21.15.42:0/286937519 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:47:05.515 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:05.502+0000 7f66dbfff700 1 --2- 172.21.15.42:0/286937519 >> v2:172.21.15.40:3300/0 conn(0x7f66dc11c810 0x7f66dc11cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:47:05.515 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:05.502+0000 7f66db7fe700 1 --2- 172.21.15.42:0/286937519 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f66dc11a3c0 0x7f66dc116a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:47:05.515 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:05.502+0000 7f66d3fff700 1 -- 172.21.15.42:0/286937519 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1449211070 0 0) 0x7f66dc11b120 con 0x7f66dc121fd0 2019-12-05T05:47:05.515 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:05.506+0000 7f66d3fff700 1 -- 172.21.15.42:0/286937519 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66b40030b0 con 0x7f66dc121fd0 2019-12-05T05:47:05.515 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:05.506+0000 7f66db7fe700 1 -- 172.21.15.42:0/286937519 >> v2:172.21.15.40:3300/0 conn(0x7f66dc11c810 msgr2=0x7f66dc11cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:47:05.515 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:05.506+0000 7f66db7fe700 1 --2- 172.21.15.42:0/286937519 >> v2:172.21.15.40:3300/0 conn(0x7f66dc11c810 0x7f66dc11cc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:47:05.516 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:05.506+0000 7f66db7fe700 1 -- 172.21.15.42:0/286937519 >> v1:172.21.15.42:6789/0 conn(0x7f66dc121fd0 legacy=0x7f66dc132590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:47:05.516 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:05.506+0000 7f66d3fff700 1 -- 172.21.15.42:0/286937519 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 2605044222 0 0) 0x7f66b40030b0 con 0x7f66dc121fd0 2019-12-05T05:47:05.516 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:05.506+0000 7f66db7fe700 1 -- 172.21.15.42:0/286937519 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66dc132d90 con 0x7f66dc11a3c0 2019-12-05T05:47:05.516 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:05.506+0000 7f66dbfff700 1 --2- 172.21.15.42:0/286937519 >> v2:172.21.15.40:3300/0 conn(0x7f66dc11c810 0x7f66dc11cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:47:05.516 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:05.506+0000 7f66db7fe700 1 --2- 172.21.15.42:0/286937519 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f66dc11a3c0 0x7f66dc116a20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f66cc001f40 tx=0x7f66cc001f80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:05.517 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:05.506+0000 7f66db7fe700 1 --2- 172.21.15.42:0/286937519 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f66dc11a3c0 0x7f66dc116a20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f66cc001f40 tx=0x7f66cc001f80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:05.517 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:05.506+0000 7f66db7fe700 1 --2- 172.21.15.42:0/286937519 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f66dc11a3c0 0x7f66dc116a20 secure :-1 s=READY pgs=325 cs=0 l=1 rx=0x7f66cc001f40 tx=0x7f66cc001f80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:47:05.517 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:05.506+0000 7f66db7fe700 1 --2- 172.21.15.42:0/286937519 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f66dc11a3c0 0x7f66dc116a20 secure :-1 s=THROTTLE_DONE pgs=325 cs=0 l=1 rx=0x7f66cc001f40 tx=0x7f66cc001f80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:05.517 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:05.506+0000 7f66d3fff700 1 -- 172.21.15.42:0/286937519 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f66cc001d00 con 0x7f66dc11a3c0 2019-12-05T05:47:05.517 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:05.506+0000 7f66db7fe700 1 --2- 172.21.15.42:0/286937519 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f66dc11a3c0 0x7f66dc116a20 secure :-1 s=THROTTLE_DONE pgs=325 cs=0 l=1 rx=0x7f66cc001f40 tx=0x7f66cc001f80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:05.517 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:05.506+0000 7f66db7fe700 1 --2- 172.21.15.42:0/286937519 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f66dc11a3c0 0x7f66dc116a20 secure :-1 s=THROTTLE_DONE pgs=325 cs=0 l=1 rx=0x7f66cc001f40 tx=0x7f66cc001f80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:05.517 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:05.506+0000 7f66d3fff700 1 -- 172.21.15.42:0/286937519 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66cc001b10 con 0x7f66dc11a3c0 2019-12-05T05:47:05.518 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:05.506+0000 7f66e187c700 1 -- 172.21.15.42:0/286937519 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66dc122400 con 0x7f66dc11a3c0 2019-12-05T05:47:05.518 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:05.506+0000 7f66d3fff700 1 -- 172.21.15.42:0/286937519 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f66c4005730 con 0x7f66dc11a3c0 2019-12-05T05:47:05.518 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:05.506+0000 7f66e187c700 1 -- 172.21.15.42:0/286937519 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66dc11d190 con 0x7f66dc11a3c0 2019-12-05T05:47:05.518 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:05.506+0000 7f66db7fe700 1 --2- 172.21.15.42:0/286937519 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f66dc11a3c0 0x7f66dc116a20 secure :-1 s=THROTTLE_DONE pgs=325 cs=0 l=1 rx=0x7f66cc001f40 tx=0x7f66cc001f80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:05.518 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:05.506+0000 7f66d3fff700 1 -- 172.21.15.42:0/286937519 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f66c4005980 con 0x7f66dc11a3c0 2019-12-05T05:47:05.518 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:05.506+0000 7f66db7fe700 1 --2- 172.21.15.42:0/286937519 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f66dc11a3c0 0x7f66dc116a20 secure :-1 s=THROTTLE_DONE pgs=325 cs=0 l=1 rx=0x7f66cc001f40 tx=0x7f66cc001f80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:05.518 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:05.506+0000 7f66d3fff700 1 --2- 172.21.15.42:0/286937519 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f66b403b2b0 0x7f66b403d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:47:05.519 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:05.506+0000 7f66d3fff700 1 -- 172.21.15.42:0/286937519 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 8525+0+0 (secure 0 0 0) 0x7f66c404b140 con 0x7f66dc11a3c0 2019-12-05T05:47:05.519 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:05.506+0000 7f66daffd700 1 --2- 172.21.15.42:0/286937519 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f66b403b2b0 0x7f66b403d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:47:05.519 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:05.510+0000 7f66daffd700 1 --2- 172.21.15.42:0/286937519 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f66b403b2b0 0x7f66b403d7c0 crc :-1 s=READY pgs=629 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:47:05.519 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:05.514+0000 7f66e187c700 1 -- 172.21.15.42:0/286937519 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66dc05d110 con 0x7f66dc11a3c0 2019-12-05T05:47:05.525 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:05.522+0000 7f66db7fe700 1 --2- 172.21.15.42:0/286937519 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f66dc11a3c0 0x7f66dc116a20 secure :-1 s=THROTTLE_DONE pgs=325 cs=0 l=1 rx=0x7f66cc001f40 tx=0x7f66cc001f80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:05.525 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:05.522+0000 7f66d3fff700 1 -- 172.21.15.42:0/286937519 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f66c40321c0 con 0x7f66dc11a3c0 2019-12-05T05:47:05.695 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:05.690+0000 7f66e187c700 1 -- 172.21.15.42:0/286937519 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f66dc05b8c0 con 0x7f66dc11a3c0 2019-12-05T05:47:05.696 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:05.694+0000 7f66db7fe700 1 --2- 172.21.15.42:0/286937519 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f66dc11a3c0 0x7f66dc116a20 secure :-1 s=THROTTLE_DONE pgs=325 cs=0 l=1 rx=0x7f66cc001f40 tx=0x7f66cc001f80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:05.696 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:05.694+0000 7f66d3fff700 1 -- 172.21.15.42:0/286937519 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 178 v178) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f66c4032420 con 0x7f66dc11a3c0 2019-12-05T05:47:05.698 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 178 2019-12-05T05:47:05.698 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:47:05.698 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":178,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6140,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/2146240383","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2146240383},{"type":"v1","addr":"172.21.15.40:6833","nonce":2146240383}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":178}],"filesystems":[{"mdsmap":{"epoch":177,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:45:43.664230+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":78,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6077},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6077":{"gid":6077,"name":"a","rank":0,"incarnation":174,"state":"up:active","state_seq":16,"addr":"172.21.15.42:6835/3757738536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3757738536},{"type":"v1","addr":"172.21.15.42:6835","nonce":3757738536}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:47:05.702 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:05.698+0000 7f66e187c700 1 -- 172.21.15.42:0/286937519 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f66b403b2b0 msgr2=0x7f66b403d7c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:47:05.702 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:05.698+0000 7f66e187c700 1 --2- 172.21.15.42:0/286937519 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f66b403b2b0 0x7f66b403d7c0 crc :-1 s=READY pgs=629 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:47:05.702 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:05.698+0000 7f66e187c700 1 -- 172.21.15.42:0/286937519 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f66dc11a3c0 msgr2=0x7f66dc116a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:47:05.702 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:05.698+0000 7f66e187c700 1 --2- 172.21.15.42:0/286937519 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f66dc11a3c0 0x7f66dc116a20 secure :-1 s=READY pgs=325 cs=0 l=1 rx=0x7f66cc001f40 tx=0x7f66cc001f80).stop 2019-12-05T05:47:05.702 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:05.698+0000 7f66e187c700 1 -- 172.21.15.42:0/286937519 shutdown_connections 2019-12-05T05:47:05.703 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:05.698+0000 7f66e187c700 1 --2- 172.21.15.42:0/286937519 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f66b403b2b0 0x7f66b403d7c0 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:47:05.703 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:05.698+0000 7f66e187c700 1 --2- 172.21.15.42:0/286937519 >> v2:172.21.15.40:3300/0 conn(0x7f66dc11c810 0x7f66dc11cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:47:05.703 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:05.698+0000 7f66e187c700 1 --2- 172.21.15.42:0/286937519 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f66dc11a3c0 0x7f66dc116a20 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:47:05.703 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:05.698+0000 7f66e187c700 1 -- 172.21.15.42:0/286937519 shutdown_connections 2019-12-05T05:47:05.703 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:05.698+0000 7f66e187c700 1 -- 172.21.15.42:0/286937519 wait complete. 2019-12-05T05:47:05.703 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:05.698+0000 7f66e187c700 1 -- 172.21.15.42:0/286937519 >> 172.21.15.42:0/286937519 conn(0x7f66dc0a8f60 msgr2=0x7f66dc07e4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:47:05.716 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T05:47:07.717 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:47:07.825 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:07.818+0000 7f4bafb60700 1 Processor -- start 2019-12-05T05:47:07.825 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:07.818+0000 7f4bafb60700 1 -- start start 2019-12-05T05:47:07.826 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:07.818+0000 7f4bafb60700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4ba80a7740 0x7f4ba808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:47:07.827 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:07.818+0000 7f4bafb60700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f4ba807e720 0x7f4ba807eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:47:07.827 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:07.818+0000 7f4bafb60700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ba80a0710 con 0x7f4ba809f2e0 2019-12-05T05:47:07.827 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:07.818+0000 7f4bafb60700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ba80a0b00 con 0x7f4ba807e720 2019-12-05T05:47:07.827 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:07.818+0000 7f4bafb60700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ba808e5b0 con 0x7f4ba80a7740 2019-12-05T05:47:07.827 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:07.818+0000 7f4bad8fc700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f4ba807e720 0x7f4ba807eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:47:07.827 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:07.818+0000 7f4bad8fc700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f4ba807e720 0x7f4ba807eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:41170/0 (socket says 172.21.15.42:41170) 2019-12-05T05:47:07.827 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:07.818+0000 7f4bad8fc700 1 -- 172.21.15.42:0/3228765856 learned_addr learned my addr 172.21.15.42:0/3228765856 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:47:07.828 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:07.818+0000 7f4bae0fd700 1 --2- 172.21.15.42:0/3228765856 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4ba80a7740 0x7f4ba808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:47:07.828 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:07.818+0000 7f4bad8fc700 1 -- 172.21.15.42:0/3228765856 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4ba80a7740 msgr2=0x7f4ba808df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:47:07.828 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:07.818+0000 7f4bad8fc700 1 --2- 172.21.15.42:0/3228765856 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4ba80a7740 0x7f4ba808df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:47:07.828 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:07.818+0000 7f4bad8fc700 1 -- 172.21.15.42:0/3228765856 >> v1:172.21.15.40:6789/0 conn(0x7f4ba809f2e0 legacy=0x7f4ba809f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:47:07.829 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:07.818+0000 7f4bad8fc700 1 -- 172.21.15.42:0/3228765856 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ba807f0a0 con 0x7f4ba807e720 2019-12-05T05:47:07.829 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:07.818+0000 7f4bad8fc700 1 --2- 172.21.15.42:0/3228765856 >> v2:172.21.15.42:3300/0 conn(0x7f4ba807e720 0x7f4ba807eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f4b9c008790 tx=0x7f4b9c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:07.829 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:07.818+0000 7f4bad8fc700 1 --2- 172.21.15.42:0/3228765856 >> v2:172.21.15.42:3300/0 conn(0x7f4ba807e720 0x7f4ba807eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f4b9c008790 tx=0x7f4b9c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:07.829 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:07.818+0000 7f4bad8fc700 1 --2- 172.21.15.42:0/3228765856 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4ba807e720 0x7f4ba807eb50 secure :-1 s=READY pgs=1460 cs=0 l=1 rx=0x7f4b9c008790 tx=0x7f4b9c00fd30).ready entity=mon.1 client_cookie=13d6cdaf27585796 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:47:07.829 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:07.818+0000 7f4bad8fc700 1 --2- 172.21.15.42:0/3228765856 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4ba807e720 0x7f4ba807eb50 secure :-1 s=THROTTLE_DONE pgs=1460 cs=0 l=1 rx=0x7f4b9c008790 tx=0x7f4b9c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:07.829 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:07.818+0000 7f4bad8fc700 1 --2- 172.21.15.42:0/3228765856 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4ba807e720 0x7f4ba807eb50 secure :-1 s=THROTTLE_DONE pgs=1460 cs=0 l=1 rx=0x7f4b9c008790 tx=0x7f4b9c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:07.830 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:07.818+0000 7f4bad0fb700 1 -- 172.21.15.42:0/3228765856 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4b9c0113b0 con 0x7f4ba807e720 2019-12-05T05:47:07.830 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:07.818+0000 7f4bad8fc700 1 --2- 172.21.15.42:0/3228765856 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4ba807e720 0x7f4ba807eb50 secure :-1 s=THROTTLE_DONE pgs=1460 cs=0 l=1 rx=0x7f4b9c008790 tx=0x7f4b9c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:07.830 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:07.818+0000 7f4bad0fb700 1 -- 172.21.15.42:0/3228765856 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b9c008bb0 con 0x7f4ba807e720 2019-12-05T05:47:07.830 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:07.818+0000 7f4bad0fb700 1 -- 172.21.15.42:0/3228765856 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4b9c011990 con 0x7f4ba807e720 2019-12-05T05:47:07.830 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:07.818+0000 7f4bafb60700 1 -- 172.21.15.42:0/3228765856 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4ba807e720 msgr2=0x7f4ba807eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:47:07.830 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:07.818+0000 7f4bafb60700 1 --2- 172.21.15.42:0/3228765856 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4ba807e720 0x7f4ba807eb50 secure :-1 s=READY pgs=1460 cs=0 l=1 rx=0x7f4b9c008790 tx=0x7f4b9c00fd30).stop 2019-12-05T05:47:07.830 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:07.822+0000 7f4bafb60700 1 -- 172.21.15.42:0/3228765856 shutdown_connections 2019-12-05T05:47:07.831 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:07.822+0000 7f4bafb60700 1 --2- 172.21.15.42:0/3228765856 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4ba80a7740 0x7f4ba808df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:47:07.831 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:07.822+0000 7f4bafb60700 1 --2- 172.21.15.42:0/3228765856 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4ba807e720 0x7f4ba807eb50 unknown :-1 s=CLOSED pgs=1460 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:47:07.831 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:07.822+0000 7f4bafb60700 1 -- 172.21.15.42:0/3228765856 shutdown_connections 2019-12-05T05:47:07.831 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:07.822+0000 7f4bafb60700 1 -- 172.21.15.42:0/3228765856 wait complete. 2019-12-05T05:47:07.831 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:07.822+0000 7f4bafb60700 1 -- 172.21.15.42:0/3228765856 >> 172.21.15.42:0/3228765856 conn(0x7f4ba80aabe0 msgr2=0x7f4ba80a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:47:07.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:07.822+0000 7f4bafb60700 1 Processor -- start 2019-12-05T05:47:07.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:07.822+0000 7f4bafb60700 1 -- start start 2019-12-05T05:47:07.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:07.826+0000 7f4bafb60700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f4ba80a6d90 0x7f4ba8114000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:47:07.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:07.826+0000 7f4bafb60700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f4ba81180f0 0x7f4ba812f3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:47:07.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:07.826+0000 7f4bafb60700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ba8113af0 con 0x7f4ba8117cc0 2019-12-05T05:47:07.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:07.826+0000 7f4bafb60700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ba8118520 con 0x7f4ba81180f0 2019-12-05T05:47:07.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:07.826+0000 7f4bafb60700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ba81175c0 con 0x7f4ba80a6d90 2019-12-05T05:47:07.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:07.826+0000 7f4bae0fd700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f4ba80a6d90 0x7f4ba8114000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:47:07.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:07.826+0000 7f4bae0fd700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f4ba80a6d90 0x7f4ba8114000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:41176/0 (socket says 172.21.15.42:41176) 2019-12-05T05:47:07.837 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:07.826+0000 7f4bae0fd700 1 -- 172.21.15.42:0/771266685 learned_addr learned my addr 172.21.15.42:0/771266685 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:47:07.837 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:07.826+0000 7f4bae8fe700 1 --2- 172.21.15.42:0/771266685 >> v2:172.21.15.40:3300/0 conn(0x7f4ba81180f0 0x7f4ba812f3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:47:07.837 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:07.826+0000 7f4bae0fd700 1 -- 172.21.15.42:0/771266685 >> v2:172.21.15.40:3300/0 conn(0x7f4ba81180f0 msgr2=0x7f4ba812f3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:47:07.837 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:07.826+0000 7f4bae0fd700 1 --2- 172.21.15.42:0/771266685 >> v2:172.21.15.40:3300/0 conn(0x7f4ba81180f0 0x7f4ba812f3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:47:07.837 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:07.826+0000 7f4bae0fd700 1 -- 172.21.15.42:0/771266685 >> v1:172.21.15.42:6789/0 conn(0x7f4ba8117cc0 legacy=0x7f4ba81284c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:47:07.837 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:07.826+0000 7f4bae0fd700 1 -- 172.21.15.42:0/771266685 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ba8128cc0 con 0x7f4ba80a6d90 2019-12-05T05:47:07.838 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:07.826+0000 7f4bae0fd700 1 --2- 172.21.15.42:0/771266685 >> v2:172.21.15.42:3300/0 conn(0x7f4ba80a6d90 0x7f4ba8114000 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f4ba4003070 tx=0x7f4ba4010ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:07.838 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:07.826+0000 7f4bae0fd700 1 --2- 172.21.15.42:0/771266685 >> v2:172.21.15.42:3300/0 conn(0x7f4ba80a6d90 0x7f4ba8114000 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f4ba4003070 tx=0x7f4ba4010ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:07.838 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:07.826+0000 7f4bae0fd700 1 --2- 172.21.15.42:0/771266685 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4ba80a6d90 0x7f4ba8114000 secure :-1 s=READY pgs=1462 cs=0 l=1 rx=0x7f4ba4003070 tx=0x7f4ba4010ae0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:47:07.838 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:07.826+0000 7f4bae0fd700 1 --2- 172.21.15.42:0/771266685 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4ba80a6d90 0x7f4ba8114000 secure :-1 s=THROTTLE_DONE pgs=1462 cs=0 l=1 rx=0x7f4ba4003070 tx=0x7f4ba4010ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:07.838 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:07.826+0000 7f4b9affd700 1 -- 172.21.15.42:0/771266685 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4ba4005920 con 0x7f4ba80a6d90 2019-12-05T05:47:07.838 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:07.826+0000 7f4bae0fd700 1 --2- 172.21.15.42:0/771266685 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4ba80a6d90 0x7f4ba8114000 secure :-1 s=THROTTLE_DONE pgs=1462 cs=0 l=1 rx=0x7f4ba4003070 tx=0x7f4ba4010ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:07.838 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:07.826+0000 7f4bafb60700 1 -- 172.21.15.42:0/771266685 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ba8112780 con 0x7f4ba80a6d90 2019-12-05T05:47:07.839 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:07.826+0000 7f4b9affd700 1 -- 172.21.15.42:0/771266685 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ba4005540 con 0x7f4ba80a6d90 2019-12-05T05:47:07.839 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:07.826+0000 7f4bae0fd700 1 --2- 172.21.15.42:0/771266685 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4ba80a6d90 0x7f4ba8114000 secure :-1 s=THROTTLE_DONE pgs=1462 cs=0 l=1 rx=0x7f4ba4003070 tx=0x7f4ba4010ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:07.839 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:07.826+0000 7f4b9affd700 1 -- 172.21.15.42:0/771266685 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4ba400e460 con 0x7f4ba80a6d90 2019-12-05T05:47:07.839 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:07.826+0000 7f4bafb60700 1 -- 172.21.15.42:0/771266685 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ba812f920 con 0x7f4ba80a6d90 2019-12-05T05:47:07.839 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:07.826+0000 7f4bae0fd700 1 --2- 172.21.15.42:0/771266685 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4ba80a6d90 0x7f4ba8114000 secure :-1 s=THROTTLE_DONE pgs=1462 cs=0 l=1 rx=0x7f4ba4003070 tx=0x7f4ba4010ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:07.839 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:07.830+0000 7f4b9affd700 1 -- 172.21.15.42:0/771266685 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f4ba4007180 con 0x7f4ba80a6d90 2019-12-05T05:47:07.839 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:07.830+0000 7f4bae0fd700 1 --2- 172.21.15.42:0/771266685 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4ba80a6d90 0x7f4ba8114000 secure :-1 s=THROTTLE_DONE pgs=1462 cs=0 l=1 rx=0x7f4ba4003070 tx=0x7f4ba4010ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:07.839 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:07.830+0000 7f4b9affd700 1 --2- 172.21.15.42:0/771266685 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4b9003ad90 0x7f4b9003d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:47:07.840 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:07.830+0000 7f4b9affd700 1 -- 172.21.15.42:0/771266685 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 8525+0+0 (secure 0 0 0) 0x7f4ba404af50 con 0x7f4ba80a6d90 2019-12-05T05:47:07.840 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:07.830+0000 7f4bad8fc700 1 --2- 172.21.15.42:0/771266685 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4b9003ad90 0x7f4b9003d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:47:07.840 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:07.830+0000 7f4bad8fc700 1 --2- 172.21.15.42:0/771266685 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4b9003ad90 0x7f4b9003d2a0 crc :-1 s=READY pgs=630 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:47:07.841 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:07.834+0000 7f4bafb60700 1 -- 172.21.15.42:0/771266685 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ba805d110 con 0x7f4ba80a6d90 2019-12-05T05:47:07.842 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:07.838+0000 7f4bae0fd700 1 --2- 172.21.15.42:0/771266685 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4ba80a6d90 0x7f4ba8114000 secure :-1 s=THROTTLE_DONE pgs=1462 cs=0 l=1 rx=0x7f4ba4003070 tx=0x7f4ba4010ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:07.842 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:07.838+0000 7f4b9affd700 1 -- 172.21.15.42:0/771266685 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f4ba4031f50 con 0x7f4ba80a6d90 2019-12-05T05:47:08.008 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:08.006+0000 7f4bafb60700 1 -- 172.21.15.42:0/771266685 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4ba805b8c0 con 0x7f4ba80a6d90 2019-12-05T05:47:08.009 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:08.006+0000 7f4bae0fd700 1 --2- 172.21.15.42:0/771266685 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4ba80a6d90 0x7f4ba8114000 secure :-1 s=THROTTLE_DONE pgs=1462 cs=0 l=1 rx=0x7f4ba4003070 tx=0x7f4ba4010ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:08.009 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:08.006+0000 7f4b9affd700 1 -- 172.21.15.42:0/771266685 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 178 v178) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f4ba40321b0 con 0x7f4ba80a6d90 2019-12-05T05:47:08.009 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 178 2019-12-05T05:47:08.009 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:47:08.009 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":178,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6140,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/2146240383","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2146240383},{"type":"v1","addr":"172.21.15.40:6833","nonce":2146240383}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":178}],"filesystems":[{"mdsmap":{"epoch":177,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:45:43.664230+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":78,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6077},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6077":{"gid":6077,"name":"a","rank":0,"incarnation":174,"state":"up:active","state_seq":16,"addr":"172.21.15.42:6835/3757738536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3757738536},{"type":"v1","addr":"172.21.15.42:6835","nonce":3757738536}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:47:08.013 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:08.006+0000 7f4bafb60700 1 -- 172.21.15.42:0/771266685 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4b9003ad90 msgr2=0x7f4b9003d2a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:47:08.013 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:08.006+0000 7f4bafb60700 1 --2- 172.21.15.42:0/771266685 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4b9003ad90 0x7f4b9003d2a0 crc :-1 s=READY pgs=630 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:47:08.013 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:08.006+0000 7f4bafb60700 1 -- 172.21.15.42:0/771266685 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4ba80a6d90 msgr2=0x7f4ba8114000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:47:08.014 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:08.006+0000 7f4bafb60700 1 --2- 172.21.15.42:0/771266685 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4ba80a6d90 0x7f4ba8114000 secure :-1 s=READY pgs=1462 cs=0 l=1 rx=0x7f4ba4003070 tx=0x7f4ba4010ae0).stop 2019-12-05T05:47:08.014 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:08.010+0000 7f4bafb60700 1 -- 172.21.15.42:0/771266685 shutdown_connections 2019-12-05T05:47:08.014 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:08.010+0000 7f4bafb60700 1 --2- 172.21.15.42:0/771266685 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4b9003ad90 0x7f4b9003d2a0 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:47:08.014 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:08.010+0000 7f4bafb60700 1 --2- 172.21.15.42:0/771266685 >> v2:172.21.15.40:3300/0 conn(0x7f4ba81180f0 0x7f4ba812f3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:47:08.014 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:08.010+0000 7f4bafb60700 1 --2- 172.21.15.42:0/771266685 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4ba80a6d90 0x7f4ba8114000 unknown :-1 s=CLOSED pgs=1462 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:47:08.014 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:08.010+0000 7f4bafb60700 1 -- 172.21.15.42:0/771266685 shutdown_connections 2019-12-05T05:47:08.014 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:08.010+0000 7f4bafb60700 1 -- 172.21.15.42:0/771266685 wait complete. 2019-12-05T05:47:08.015 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:08.010+0000 7f4bafb60700 1 -- 172.21.15.42:0/771266685 >> 172.21.15.42:0/771266685 conn(0x7f4ba80a8230 msgr2=0x7f4ba80a9870 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:47:08.021 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T05:47:10.022 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:47:10.128 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:10.118+0000 7f3df6148700 1 Processor -- start 2019-12-05T05:47:10.128 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:10.118+0000 7f3df6148700 1 -- start start 2019-12-05T05:47:10.128 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:10.122+0000 7f3df6148700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3df00a7740 0x7f3df008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:47:10.128 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:10.122+0000 7f3df6148700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3df007e720 0x7f3df007eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:47:10.128 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:10.122+0000 7f3df6148700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3df00a0710 con 0x7f3df009f2e0 2019-12-05T05:47:10.128 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:10.122+0000 7f3df6148700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3df00a0b00 con 0x7f3df007e720 2019-12-05T05:47:10.129 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:10.122+0000 7f3df6148700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3df008e5b0 con 0x7f3df00a7740 2019-12-05T05:47:10.129 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:10.122+0000 7f3def7fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3df007e720 0x7f3df007eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:47:10.129 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:10.122+0000 7f3df4ee6700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f3df009f2e0 0x7f3df009f710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:41226/0 (socket says 172.21.15.42:41226) 2019-12-05T05:47:10.129 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:10.122+0000 7f3def7fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3df007e720 0x7f3df007eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:41186/0 (socket says 172.21.15.42:41186) 2019-12-05T05:47:10.129 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:10.122+0000 7f3df4ee6700 1 -- 172.21.15.42:0/793496194 learned_addr learned my addr 172.21.15.42:0/793496194 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:47:10.129 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:10.122+0000 7f3deffff700 1 --2- 172.21.15.42:0/793496194 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3df00a7740 0x7f3df008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:47:10.129 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:10.122+0000 7f3def7fe700 1 -- 172.21.15.42:0/793496194 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3df00a7740 msgr2=0x7f3df008df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:47:10.130 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:10.122+0000 7f3def7fe700 1 --2- 172.21.15.42:0/793496194 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3df00a7740 0x7f3df008df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:47:10.130 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:10.122+0000 7f3def7fe700 1 -- 172.21.15.42:0/793496194 >> v1:172.21.15.42:6789/0 conn(0x7f3df009f2e0 legacy=0x7f3df009f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:47:10.130 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:10.122+0000 7f3deeffd700 1 -- 172.21.15.42:0/793496194 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 942182724 0 0) 0x7f3df00a0710 con 0x7f3df009f2e0 2019-12-05T05:47:10.130 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:10.122+0000 7f3def7fe700 1 -- 172.21.15.42:0/793496194 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3df007f0a0 con 0x7f3df007e720 2019-12-05T05:47:10.130 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:10.122+0000 7f3def7fe700 1 --2- 172.21.15.42:0/793496194 >> v2:172.21.15.42:3300/0 conn(0x7f3df007e720 0x7f3df007eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f3dd8008790 tx=0x7f3dd800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:10.130 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:10.122+0000 7f3def7fe700 1 --2- 172.21.15.42:0/793496194 >> v2:172.21.15.42:3300/0 conn(0x7f3df007e720 0x7f3df007eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f3dd8008790 tx=0x7f3dd800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:10.130 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:10.122+0000 7f3def7fe700 1 --2- 172.21.15.42:0/793496194 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3df007e720 0x7f3df007eb50 secure :-1 s=READY pgs=1464 cs=0 l=1 rx=0x7f3dd8008790 tx=0x7f3dd800fc70).ready entity=mon.1 client_cookie=aefdca6373f02bf3 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:47:10.131 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:10.122+0000 7f3def7fe700 1 --2- 172.21.15.42:0/793496194 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3df007e720 0x7f3df007eb50 secure :-1 s=THROTTLE_DONE pgs=1464 cs=0 l=1 rx=0x7f3dd8008790 tx=0x7f3dd800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:10.131 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:10.122+0000 7f3deeffd700 1 -- 172.21.15.42:0/793496194 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3dd80112f0 con 0x7f3df007e720 2019-12-05T05:47:10.131 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:10.122+0000 7f3def7fe700 1 --2- 172.21.15.42:0/793496194 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3df007e720 0x7f3df007eb50 secure :-1 s=THROTTLE_DONE pgs=1464 cs=0 l=1 rx=0x7f3dd8008790 tx=0x7f3dd800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:10.131 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:10.122+0000 7f3def7fe700 1 --2- 172.21.15.42:0/793496194 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3df007e720 0x7f3df007eb50 secure :-1 s=THROTTLE_DONE pgs=1464 cs=0 l=1 rx=0x7f3dd8008790 tx=0x7f3dd800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:10.131 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:10.122+0000 7f3deeffd700 1 -- 172.21.15.42:0/793496194 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3dd8008ad0 con 0x7f3df007e720 2019-12-05T05:47:10.131 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:10.122+0000 7f3deeffd700 1 -- 172.21.15.42:0/793496194 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3dd80118d0 con 0x7f3df007e720 2019-12-05T05:47:10.131 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:10.122+0000 7f3df6148700 1 -- 172.21.15.42:0/793496194 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3df007e720 msgr2=0x7f3df007eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:47:10.132 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:10.122+0000 7f3df6148700 1 --2- 172.21.15.42:0/793496194 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3df007e720 0x7f3df007eb50 secure :-1 s=READY pgs=1464 cs=0 l=1 rx=0x7f3dd8008790 tx=0x7f3dd800fc70).stop 2019-12-05T05:47:10.132 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:10.122+0000 7f3df6148700 1 -- 172.21.15.42:0/793496194 shutdown_connections 2019-12-05T05:47:10.132 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:10.122+0000 7f3df6148700 1 --2- 172.21.15.42:0/793496194 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3df00a7740 0x7f3df008df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:47:10.132 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:10.122+0000 7f3df6148700 1 --2- 172.21.15.42:0/793496194 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3df007e720 0x7f3df007eb50 unknown :-1 s=CLOSED pgs=1464 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:47:10.132 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:10.122+0000 7f3df6148700 1 -- 172.21.15.42:0/793496194 shutdown_connections 2019-12-05T05:47:10.132 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:10.122+0000 7f3df6148700 1 -- 172.21.15.42:0/793496194 wait complete. 2019-12-05T05:47:10.132 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:10.122+0000 7f3df6148700 1 -- 172.21.15.42:0/793496194 >> 172.21.15.42:0/793496194 conn(0x7f3df00aabe0 msgr2=0x7f3df00a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:47:10.136 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:10.126+0000 7f3df6148700 1 Processor -- start 2019-12-05T05:47:10.136 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:10.126+0000 7f3df6148700 1 -- start start 2019-12-05T05:47:10.136 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:10.126+0000 7f3df6148700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f3df0122a80 0x7f3df01333a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:47:10.136 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:10.126+0000 7f3df6148700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3df0133fb0 0x7f3df008eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:47:10.136 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:10.126+0000 7f3df6148700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3df0122eb0 con 0x7f3df0133b80 2019-12-05T05:47:10.136 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:10.126+0000 7f3df6148700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3df01338f0 con 0x7f3df0122a80 2019-12-05T05:47:10.137 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:10.126+0000 7f3df6148700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3df011ad00 con 0x7f3df0133fb0 2019-12-05T05:47:10.137 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:10.126+0000 7f3df4ee6700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3df0133fb0 0x7f3df008eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:47:10.137 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:10.126+0000 7f3df4ee6700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3df0133fb0 0x7f3df008eb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:41192/0 (socket says 172.21.15.42:41192) 2019-12-05T05:47:10.137 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:10.126+0000 7f3df4ee6700 1 -- 172.21.15.42:0/1657533913 learned_addr learned my addr 172.21.15.42:0/1657533913 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:47:10.137 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:10.126+0000 7f3deffff700 1 --2- 172.21.15.42:0/1657533913 >> v2:172.21.15.40:3300/0 conn(0x7f3df0122a80 0x7f3df01333a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:47:10.137 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:10.126+0000 7f3df4ee6700 1 -- 172.21.15.42:0/1657533913 >> v2:172.21.15.40:3300/0 conn(0x7f3df0122a80 msgr2=0x7f3df01333a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:47:10.137 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:10.126+0000 7f3df4ee6700 1 --2- 172.21.15.42:0/1657533913 >> v2:172.21.15.40:3300/0 conn(0x7f3df0122a80 0x7f3df01333a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:47:10.138 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:10.126+0000 7f3df4ee6700 1 -- 172.21.15.42:0/1657533913 >> v1:172.21.15.40:6789/0 conn(0x7f3df0133b80 legacy=0x7f3df011a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:47:10.138 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:10.126+0000 7f3df4ee6700 1 -- 172.21.15.42:0/1657533913 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3df011aec0 con 0x7f3df0133fb0 2019-12-05T05:47:10.138 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:10.126+0000 7f3df4ee6700 1 --2- 172.21.15.42:0/1657533913 >> v2:172.21.15.42:3300/0 conn(0x7f3df0133fb0 0x7f3df008eb70 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f3df011a2c0 tx=0x7f3de0010820).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:10.138 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:10.126+0000 7f3df4ee6700 1 --2- 172.21.15.42:0/1657533913 >> v2:172.21.15.42:3300/0 conn(0x7f3df0133fb0 0x7f3df008eb70 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f3df011a2c0 tx=0x7f3de0010820).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:10.138 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:10.126+0000 7f3df4ee6700 1 --2- 172.21.15.42:0/1657533913 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3df0133fb0 0x7f3df008eb70 secure :-1 s=READY pgs=1465 cs=0 l=1 rx=0x7f3df011a2c0 tx=0x7f3de0010820).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:47:10.138 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:10.130+0000 7f3df4ee6700 1 --2- 172.21.15.42:0/1657533913 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3df0133fb0 0x7f3df008eb70 secure :-1 s=THROTTLE_DONE pgs=1465 cs=0 l=1 rx=0x7f3df011a2c0 tx=0x7f3de0010820).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:10.139 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:10.130+0000 7f3df4ee6700 1 --2- 172.21.15.42:0/1657533913 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3df0133fb0 0x7f3df008eb70 secure :-1 s=THROTTLE_DONE pgs=1465 cs=0 l=1 rx=0x7f3df011a2c0 tx=0x7f3de0010820).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:10.139 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:10.130+0000 7f3df4ee6700 1 --2- 172.21.15.42:0/1657533913 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3df0133fb0 0x7f3df008eb70 secure :-1 s=THROTTLE_DONE pgs=1465 cs=0 l=1 rx=0x7f3df011a2c0 tx=0x7f3de0010820).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:10.139 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:10.130+0000 7f3decff9700 1 -- 172.21.15.42:0/1657533913 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3de0011510 con 0x7f3df0133fb0 2019-12-05T05:47:10.139 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:10.130+0000 7f3decff9700 1 -- 172.21.15.42:0/1657533913 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3de0011130 con 0x7f3df0133fb0 2019-12-05T05:47:10.139 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:10.130+0000 7f3df6148700 1 -- 172.21.15.42:0/1657533913 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3df008f0c0 con 0x7f3df0133fb0 2019-12-05T05:47:10.139 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:10.130+0000 7f3decff9700 1 -- 172.21.15.42:0/1657533913 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3de0011b30 con 0x7f3df0133fb0 2019-12-05T05:47:10.139 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:10.130+0000 7f3df6148700 1 -- 172.21.15.42:0/1657533913 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3df008f5f0 con 0x7f3df0133fb0 2019-12-05T05:47:10.140 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:10.130+0000 7f3df4ee6700 1 --2- 172.21.15.42:0/1657533913 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3df0133fb0 0x7f3df008eb70 secure :-1 s=THROTTLE_DONE pgs=1465 cs=0 l=1 rx=0x7f3df011a2c0 tx=0x7f3de0010820).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:10.140 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:10.130+0000 7f3decff9700 1 -- 172.21.15.42:0/1657533913 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f3de0005460 con 0x7f3df0133fb0 2019-12-05T05:47:10.140 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:10.130+0000 7f3df4ee6700 1 --2- 172.21.15.42:0/1657533913 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3df0133fb0 0x7f3df008eb70 secure :-1 s=THROTTLE_DONE pgs=1465 cs=0 l=1 rx=0x7f3df011a2c0 tx=0x7f3de0010820).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:10.140 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:10.130+0000 7f3decff9700 1 --2- 172.21.15.42:0/1657533913 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3dd403ad50 0x7f3dd403d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:47:10.140 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:10.130+0000 7f3decff9700 1 -- 172.21.15.42:0/1657533913 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 8525+0+0 (secure 0 0 0) 0x7f3de004b2b0 con 0x7f3df0133fb0 2019-12-05T05:47:10.140 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:10.130+0000 7f3deffff700 1 --2- 172.21.15.42:0/1657533913 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3dd403ad50 0x7f3dd403d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:47:10.140 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:10.130+0000 7f3deffff700 1 --2- 172.21.15.42:0/1657533913 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3dd403ad50 0x7f3dd403d260 crc :-1 s=READY pgs=631 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:47:10.141 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:10.134+0000 7f3df6148700 1 -- 172.21.15.42:0/1657533913 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3df005d110 con 0x7f3df0133fb0 2019-12-05T05:47:10.147 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:10.142+0000 7f3df4ee6700 1 --2- 172.21.15.42:0/1657533913 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3df0133fb0 0x7f3df008eb70 secure :-1 s=THROTTLE_DONE pgs=1465 cs=0 l=1 rx=0x7f3df011a2c0 tx=0x7f3de0010820).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:10.148 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:10.142+0000 7f3decff9700 1 -- 172.21.15.42:0/1657533913 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f3de00322d0 con 0x7f3df0133fb0 2019-12-05T05:47:10.316 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:10.314+0000 7f3df6148700 1 -- 172.21.15.42:0/1657533913 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3df005b8c0 con 0x7f3df0133fb0 2019-12-05T05:47:10.316 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:10.314+0000 7f3df4ee6700 1 --2- 172.21.15.42:0/1657533913 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3df0133fb0 0x7f3df008eb70 secure :-1 s=THROTTLE_DONE pgs=1465 cs=0 l=1 rx=0x7f3df011a2c0 tx=0x7f3de0010820).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:10.317 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:10.314+0000 7f3decff9700 1 -- 172.21.15.42:0/1657533913 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 178 v178) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f3de0032530 con 0x7f3df0133fb0 2019-12-05T05:47:10.319 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 178 2019-12-05T05:47:10.319 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:47:10.319 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":178,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6140,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/2146240383","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2146240383},{"type":"v1","addr":"172.21.15.40:6833","nonce":2146240383}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":178}],"filesystems":[{"mdsmap":{"epoch":177,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:45:43.664230+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":78,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6077},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6077":{"gid":6077,"name":"a","rank":0,"incarnation":174,"state":"up:active","state_seq":16,"addr":"172.21.15.42:6835/3757738536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3757738536},{"type":"v1","addr":"172.21.15.42:6835","nonce":3757738536}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:47:10.323 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:10.318+0000 7f3df6148700 1 -- 172.21.15.42:0/1657533913 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3dd403ad50 msgr2=0x7f3dd403d260 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:47:10.323 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:10.318+0000 7f3df6148700 1 --2- 172.21.15.42:0/1657533913 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3dd403ad50 0x7f3dd403d260 crc :-1 s=READY pgs=631 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:47:10.323 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:10.318+0000 7f3df6148700 1 -- 172.21.15.42:0/1657533913 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3df0133fb0 msgr2=0x7f3df008eb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:47:10.323 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:10.318+0000 7f3df6148700 1 --2- 172.21.15.42:0/1657533913 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3df0133fb0 0x7f3df008eb70 secure :-1 s=READY pgs=1465 cs=0 l=1 rx=0x7f3df011a2c0 tx=0x7f3de0010820).stop 2019-12-05T05:47:10.324 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:10.318+0000 7f3df6148700 1 -- 172.21.15.42:0/1657533913 shutdown_connections 2019-12-05T05:47:10.324 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:10.318+0000 7f3df6148700 1 --2- 172.21.15.42:0/1657533913 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3dd403ad50 0x7f3dd403d260 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:47:10.324 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:10.318+0000 7f3df6148700 1 --2- 172.21.15.42:0/1657533913 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3df0133fb0 0x7f3df008eb70 unknown :-1 s=CLOSED pgs=1465 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:47:10.324 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:10.318+0000 7f3df6148700 1 --2- 172.21.15.42:0/1657533913 >> v2:172.21.15.40:3300/0 conn(0x7f3df0122a80 0x7f3df01333a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:47:10.324 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:10.318+0000 7f3df6148700 1 -- 172.21.15.42:0/1657533913 shutdown_connections 2019-12-05T05:47:10.324 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:10.318+0000 7f3df6148700 1 -- 172.21.15.42:0/1657533913 wait complete. 2019-12-05T05:47:10.324 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:10.318+0000 7f3df6148700 1 -- 172.21.15.42:0/1657533913 >> 172.21.15.42:0/1657533913 conn(0x7f3df00a6bb0 msgr2=0x7f3df011c470 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:47:10.336 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T05:47:11.554 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:47:11.558 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:47:12.336 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:47:12.458 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:12.446+0000 7fb7dce3d700 1 Processor -- start 2019-12-05T05:47:12.458 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:12.446+0000 7fb7dce3d700 1 -- start start 2019-12-05T05:47:12.458 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:12.450+0000 7fb7dce3d700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb7d80a7740 0x7fb7d808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:47:12.458 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:12.450+0000 7fb7dce3d700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb7d809fae0 0x7fb7d807eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:47:12.459 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:12.450+0000 7fb7dce3d700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7d80a0710 con 0x7fb7d807e720 2019-12-05T05:47:12.459 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:12.450+0000 7fb7dce3d700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7d80a0b00 con 0x7fb7d80a7740 2019-12-05T05:47:12.459 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:12.450+0000 7fb7dce3d700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7d808e5b0 con 0x7fb7d809fae0 2019-12-05T05:47:12.459 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:12.450+0000 7fb7d659c700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fb7d807e720 0x7fb7d809f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:41236/0 (socket says 172.21.15.42:41236) 2019-12-05T05:47:12.459 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:12.450+0000 7fb7d659c700 1 -- 172.21.15.42:0/3808753443 learned_addr learned my addr 172.21.15.42:0/3808753443 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:47:12.459 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:12.450+0000 7fb7d759e700 1 --2- 172.21.15.42:0/3808753443 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb7d809fae0 0x7fb7d807eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:47:12.459 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:12.450+0000 7fb7d6d9d700 1 --2- 172.21.15.42:0/3808753443 >> v2:172.21.15.40:3300/0 conn(0x7fb7d80a7740 0x7fb7d808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:47:12.460 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:12.450+0000 7fb7d5d9b700 1 -- 172.21.15.42:0/3808753443 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 333292821 0 0) 0x7fb7d80a0710 con 0x7fb7d807e720 2019-12-05T05:47:12.460 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:12.450+0000 7fb7d5d9b700 1 -- 172.21.15.42:0/3808753443 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7c00030b0 con 0x7fb7d807e720 2019-12-05T05:47:12.460 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:12.450+0000 7fb7d759e700 1 -- 172.21.15.42:0/3808753443 >> v2:172.21.15.40:3300/0 conn(0x7fb7d80a7740 msgr2=0x7fb7d808df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:47:12.460 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:12.450+0000 7fb7d759e700 1 --2- 172.21.15.42:0/3808753443 >> v2:172.21.15.40:3300/0 conn(0x7fb7d80a7740 0x7fb7d808df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:47:12.460 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:12.450+0000 7fb7d759e700 1 -- 172.21.15.42:0/3808753443 >> v1:172.21.15.42:6789/0 conn(0x7fb7d807e720 legacy=0x7fb7d809f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:47:12.460 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:12.450+0000 7fb7d759e700 1 -- 172.21.15.42:0/3808753443 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7d807f0a0 con 0x7fb7d809fae0 2019-12-05T05:47:12.460 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:12.450+0000 7fb7d6d9d700 1 --2- 172.21.15.42:0/3808753443 >> v2:172.21.15.40:3300/0 conn(0x7fb7d80a7740 0x7fb7d808df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:47:12.461 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:12.450+0000 7fb7d759e700 1 --2- 172.21.15.42:0/3808753443 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb7d809fae0 0x7fb7d807eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fb7d0008b10 tx=0x7fb7d000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:12.461 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:12.450+0000 7fb7d759e700 1 --2- 172.21.15.42:0/3808753443 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb7d809fae0 0x7fb7d807eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fb7d0008b10 tx=0x7fb7d000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:12.461 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:12.450+0000 7fb7d759e700 1 --2- 172.21.15.42:0/3808753443 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb7d809fae0 0x7fb7d807eb50 secure :-1 s=READY pgs=327 cs=0 l=1 rx=0x7fb7d0008b10 tx=0x7fb7d000fff0).ready entity=mon.2 client_cookie=69487655038167cf server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:47:12.461 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:12.450+0000 7fb7d759e700 1 --2- 172.21.15.42:0/3808753443 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb7d809fae0 0x7fb7d807eb50 secure :-1 s=THROTTLE_DONE pgs=327 cs=0 l=1 rx=0x7fb7d0008b10 tx=0x7fb7d000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:12.461 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:12.450+0000 7fb7d5d9b700 1 -- 172.21.15.42:0/3808753443 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb7d0010900 con 0x7fb7d809fae0 2019-12-05T05:47:12.461 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:12.450+0000 7fb7d759e700 1 --2- 172.21.15.42:0/3808753443 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb7d809fae0 0x7fb7d807eb50 secure :-1 s=THROTTLE_DONE pgs=327 cs=0 l=1 rx=0x7fb7d0008b10 tx=0x7fb7d000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:12.461 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:12.450+0000 7fb7d5d9b700 1 -- 172.21.15.42:0/3808753443 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7d0008b50 con 0x7fb7d809fae0 2019-12-05T05:47:12.462 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:12.450+0000 7fb7d759e700 1 --2- 172.21.15.42:0/3808753443 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb7d809fae0 0x7fb7d807eb50 secure :-1 s=THROTTLE_DONE pgs=327 cs=0 l=1 rx=0x7fb7d0008b10 tx=0x7fb7d000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:12.462 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:12.450+0000 7fb7d5d9b700 1 -- 172.21.15.42:0/3808753443 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb7d00110c0 con 0x7fb7d809fae0 2019-12-05T05:47:12.462 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:12.450+0000 7fb7dce3d700 1 -- 172.21.15.42:0/3808753443 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb7d809fae0 msgr2=0x7fb7d807eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:47:12.462 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:12.450+0000 7fb7dce3d700 1 --2- 172.21.15.42:0/3808753443 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb7d809fae0 0x7fb7d807eb50 secure :-1 s=READY pgs=327 cs=0 l=1 rx=0x7fb7d0008b10 tx=0x7fb7d000fff0).stop 2019-12-05T05:47:12.462 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:12.454+0000 7fb7dce3d700 1 -- 172.21.15.42:0/3808753443 shutdown_connections 2019-12-05T05:47:12.462 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:12.454+0000 7fb7dce3d700 1 --2- 172.21.15.42:0/3808753443 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb7d809fae0 0x7fb7d807eb50 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:47:12.462 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:12.454+0000 7fb7dce3d700 1 --2- 172.21.15.42:0/3808753443 >> v2:172.21.15.40:3300/0 conn(0x7fb7d80a7740 0x7fb7d808df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:47:12.463 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:12.454+0000 7fb7dce3d700 1 -- 172.21.15.42:0/3808753443 shutdown_connections 2019-12-05T05:47:12.463 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:12.454+0000 7fb7dce3d700 1 -- 172.21.15.42:0/3808753443 wait complete. 2019-12-05T05:47:12.463 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:12.454+0000 7fb7dce3d700 1 -- 172.21.15.42:0/3808753443 >> 172.21.15.42:0/3808753443 conn(0x7fb7d80aabe0 msgr2=0x7fb7d80a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:47:12.463 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:12.454+0000 7fb7dce3d700 1 Processor -- start 2019-12-05T05:47:12.463 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:12.454+0000 7fb7dce3d700 1 -- start start 2019-12-05T05:47:12.466 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:12.454+0000 7fb7dce3d700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb7d8116550 0x7fb7d811a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:47:12.466 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:12.454+0000 7fb7dce3d700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7d811a490 con 0x7fb7d807d840 2019-12-05T05:47:12.467 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:12.454+0000 7fb7dce3d700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7d8120b50 con 0x7fb7d807cc60 2019-12-05T05:47:12.467 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:12.454+0000 7fb7dce3d700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7d807bcb0 con 0x7fb7d8116550 2019-12-05T05:47:12.467 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:12.458+0000 7fb7d759e700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fb7d807cc60 0x7fb7d8130780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:41246/0 (socket says 172.21.15.42:41246) 2019-12-05T05:47:12.467 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:12.458+0000 7fb7d759e700 1 -- 172.21.15.42:0/2416729327 learned_addr learned my addr 172.21.15.42:0/2416729327 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:47:12.467 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:12.458+0000 7fb7d6d9d700 1 --2- 172.21.15.42:0/2416729327 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb7d8116550 0x7fb7d811a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:47:12.467 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:12.458+0000 7fb7c77fe700 1 -- 172.21.15.42:0/2416729327 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2041260709 0 0) 0x7fb7d8120b50 con 0x7fb7d807cc60 2019-12-05T05:47:12.467 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:12.458+0000 7fb7c77fe700 1 -- 172.21.15.42:0/2416729327 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7bc0030b0 con 0x7fb7d807cc60 2019-12-05T05:47:12.468 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:12.458+0000 7fb7c77fe700 1 -- 172.21.15.42:0/2416729327 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 174114972 0 0) 0x7fb7d811a490 con 0x7fb7d807d840 2019-12-05T05:47:12.468 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:12.458+0000 7fb7c77fe700 1 -- 172.21.15.42:0/2416729327 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7d8120b50 con 0x7fb7d807d840 2019-12-05T05:47:12.468 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:12.458+0000 7fb7d6d9d700 1 -- 172.21.15.42:0/2416729327 >> v1:172.21.15.42:6789/0 conn(0x7fb7d807cc60 legacy=0x7fb7d8130780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:47:12.468 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:12.458+0000 7fb7d6d9d700 1 -- 172.21.15.42:0/2416729327 >> v1:172.21.15.40:6789/0 conn(0x7fb7d807d840 legacy=0x7fb7d807bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:47:12.468 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:12.458+0000 7fb7d6d9d700 1 -- 172.21.15.42:0/2416729327 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7d8121c80 con 0x7fb7d8116550 2019-12-05T05:47:12.468 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:12.458+0000 7fb7d6d9d700 1 --2- 172.21.15.42:0/2416729327 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb7d8116550 0x7fb7d811a940 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fb7c8002f30 tx=0x7fb7c8010ba0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:12.468 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:12.458+0000 7fb7d6d9d700 1 --2- 172.21.15.42:0/2416729327 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb7d8116550 0x7fb7d811a940 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fb7c8002f30 tx=0x7fb7c8010ba0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:12.469 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:12.458+0000 7fb7d6d9d700 1 --2- 172.21.15.42:0/2416729327 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb7d8116550 0x7fb7d811a940 secure :-1 s=READY pgs=328 cs=0 l=1 rx=0x7fb7c8002f30 tx=0x7fb7c8010ba0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:47:12.469 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:12.458+0000 7fb7d6d9d700 1 --2- 172.21.15.42:0/2416729327 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb7d8116550 0x7fb7d811a940 secure :-1 s=THROTTLE_DONE pgs=328 cs=0 l=1 rx=0x7fb7c8002f30 tx=0x7fb7c8010ba0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:12.469 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:12.458+0000 7fb7c77fe700 1 -- 172.21.15.42:0/2416729327 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb7c800d0c0 con 0x7fb7d8116550 2019-12-05T05:47:12.469 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:12.458+0000 7fb7d6d9d700 1 --2- 172.21.15.42:0/2416729327 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb7d8116550 0x7fb7d811a940 secure :-1 s=THROTTLE_DONE pgs=328 cs=0 l=1 rx=0x7fb7c8002f30 tx=0x7fb7c8010ba0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:12.469 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:12.458+0000 7fb7c77fe700 1 -- 172.21.15.42:0/2416729327 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7c8007150 con 0x7fb7d8116550 2019-12-05T05:47:12.469 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:12.458+0000 7fb7d6d9d700 1 --2- 172.21.15.42:0/2416729327 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb7d8116550 0x7fb7d811a940 secure :-1 s=THROTTLE_DONE pgs=328 cs=0 l=1 rx=0x7fb7c8002f30 tx=0x7fb7c8010ba0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:12.470 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:12.458+0000 7fb7dce3d700 1 -- 172.21.15.42:0/2416729327 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7d807d090 con 0x7fb7d8116550 2019-12-05T05:47:12.470 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:12.458+0000 7fb7c77fe700 1 -- 172.21.15.42:0/2416729327 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb7c8012990 con 0x7fb7d8116550 2019-12-05T05:47:12.470 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:12.458+0000 7fb7dce3d700 1 -- 172.21.15.42:0/2416729327 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7d8131f90 con 0x7fb7d8116550 2019-12-05T05:47:12.470 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:12.458+0000 7fb7d6d9d700 1 --2- 172.21.15.42:0/2416729327 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb7d8116550 0x7fb7d811a940 secure :-1 s=THROTTLE_DONE pgs=328 cs=0 l=1 rx=0x7fb7c8002f30 tx=0x7fb7c8010ba0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:12.470 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:12.462+0000 7fb7c77fe700 1 -- 172.21.15.42:0/2416729327 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fb7c8012cf0 con 0x7fb7d8116550 2019-12-05T05:47:12.470 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:12.462+0000 7fb7d6d9d700 1 --2- 172.21.15.42:0/2416729327 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb7d8116550 0x7fb7d811a940 secure :-1 s=THROTTLE_DONE pgs=328 cs=0 l=1 rx=0x7fb7c8002f30 tx=0x7fb7c8010ba0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:12.470 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:12.462+0000 7fb7c77fe700 1 --2- 172.21.15.42:0/2416729327 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb7bc03b450 0x7fb7bc03d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:47:12.470 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:12.462+0000 7fb7c77fe700 1 -- 172.21.15.42:0/2416729327 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 8525+0+0 (secure 0 0 0) 0x7fb7c804b200 con 0x7fb7d8116550 2019-12-05T05:47:12.471 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:12.462+0000 7fb7d659c700 1 --2- 172.21.15.42:0/2416729327 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb7bc03b450 0x7fb7bc03d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:47:12.471 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:12.462+0000 7fb7d659c700 1 --2- 172.21.15.42:0/2416729327 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb7bc03b450 0x7fb7bc03d960 crc :-1 s=READY pgs=632 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:47:12.471 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:12.462+0000 7fb7dce3d700 1 -- 172.21.15.42:0/2416729327 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7d805d110 con 0x7fb7d8116550 2019-12-05T05:47:12.475 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:12.474+0000 7fb7d6d9d700 1 --2- 172.21.15.42:0/2416729327 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb7d8116550 0x7fb7d811a940 secure :-1 s=THROTTLE_DONE pgs=328 cs=0 l=1 rx=0x7fb7c8002f30 tx=0x7fb7c8010ba0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:12.476 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:12.474+0000 7fb7c77fe700 1 -- 172.21.15.42:0/2416729327 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fb7c8032260 con 0x7fb7d8116550 2019-12-05T05:47:12.641 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:12.638+0000 7fb7dce3d700 1 -- 172.21.15.42:0/2416729327 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb7d80a8cb0 con 0x7fb7d8116550 2019-12-05T05:47:12.642 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:12.638+0000 7fb7d6d9d700 1 --2- 172.21.15.42:0/2416729327 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb7d8116550 0x7fb7d811a940 secure :-1 s=THROTTLE_DONE pgs=328 cs=0 l=1 rx=0x7fb7c8002f30 tx=0x7fb7c8010ba0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:12.643 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:12.638+0000 7fb7c77fe700 1 -- 172.21.15.42:0/2416729327 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 178 v178) v1 ==== 95+0+2112 (secure 0 0 0) 0x7fb7c80324c0 con 0x7fb7d8116550 2019-12-05T05:47:12.645 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:47:12.645 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":178,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6140,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/2146240383","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2146240383},{"type":"v1","addr":"172.21.15.40:6833","nonce":2146240383}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":178}],"filesystems":[{"mdsmap":{"epoch":177,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:45:43.664230+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":78,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6077},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6077":{"gid":6077,"name":"a","rank":0,"incarnation":174,"state":"up:active","state_seq":16,"addr":"172.21.15.42:6835/3757738536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3757738536},{"type":"v1","addr":"172.21.15.42:6835","nonce":3757738536}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:47:12.645 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 178 2019-12-05T05:47:12.647 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:12.642+0000 7fb7dce3d700 1 -- 172.21.15.42:0/2416729327 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb7bc03b450 msgr2=0x7fb7bc03d960 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:47:12.647 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:12.642+0000 7fb7dce3d700 1 --2- 172.21.15.42:0/2416729327 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb7bc03b450 0x7fb7bc03d960 crc :-1 s=READY pgs=632 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:47:12.647 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:12.642+0000 7fb7dce3d700 1 -- 172.21.15.42:0/2416729327 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb7d8116550 msgr2=0x7fb7d811a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:47:12.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:12.642+0000 7fb7dce3d700 1 --2- 172.21.15.42:0/2416729327 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb7d8116550 0x7fb7d811a940 secure :-1 s=READY pgs=328 cs=0 l=1 rx=0x7fb7c8002f30 tx=0x7fb7c8010ba0).stop 2019-12-05T05:47:12.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:12.642+0000 7fb7dce3d700 1 -- 172.21.15.42:0/2416729327 shutdown_connections 2019-12-05T05:47:12.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:12.642+0000 7fb7dce3d700 1 --2- 172.21.15.42:0/2416729327 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb7bc03b450 0x7fb7bc03d960 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:47:12.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:12.642+0000 7fb7dce3d700 1 --2- 172.21.15.42:0/2416729327 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb7d8116550 0x7fb7d811a940 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:47:12.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:12.646+0000 7fb7dce3d700 1 -- 172.21.15.42:0/2416729327 shutdown_connections 2019-12-05T05:47:12.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:12.646+0000 7fb7dce3d700 1 -- 172.21.15.42:0/2416729327 wait complete. 2019-12-05T05:47:12.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:12.646+0000 7fb7dce3d700 1 -- 172.21.15.42:0/2416729327 >> 172.21.15.42:0/2416729327 conn(0x7fb7d80a80b0 msgr2=0x7fb7d811da30 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:47:12.657 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T05:47:14.658 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:47:14.756 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:14.750+0000 7ffa76dfb700 1 Processor -- start 2019-12-05T05:47:14.756 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:14.750+0000 7ffa76dfb700 1 -- start start 2019-12-05T05:47:14.757 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:14.750+0000 7ffa76dfb700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ffa700a7730 0x7ffa7008df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:47:14.757 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:14.750+0000 7ffa76dfb700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ffa7007e710 0x7ffa7007eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:47:14.757 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:14.750+0000 7ffa76dfb700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa700a0700 con 0x7ffa7009f2d0 2019-12-05T05:47:14.757 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:14.750+0000 7ffa76dfb700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa700a0af0 con 0x7ffa700a7730 2019-12-05T05:47:14.757 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:14.750+0000 7ffa76dfb700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa7008e5a0 con 0x7ffa7007e710 2019-12-05T05:47:14.757 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:14.750+0000 7ffa75398700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ffa700a7730 0x7ffa7008df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:47:14.758 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:14.750+0000 7ffa75398700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ffa700a7730 0x7ffa7008df70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:41214/0 (socket says 172.21.15.42:41214) 2019-12-05T05:47:14.758 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:14.750+0000 7ffa75398700 1 -- 172.21.15.42:0/990652437 learned_addr learned my addr 172.21.15.42:0/990652437 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:47:14.758 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:14.750+0000 7ffa74b97700 1 --2- 172.21.15.42:0/990652437 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ffa7007e710 0x7ffa7007eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:47:14.758 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:14.750+0000 7ffa6ffff700 1 -- 172.21.15.42:0/990652437 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1591449890 0 0) 0x7ffa700a0700 con 0x7ffa7009f2d0 2019-12-05T05:47:14.758 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:14.750+0000 7ffa6ffff700 1 -- 172.21.15.42:0/990652437 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa600030b0 con 0x7ffa7009f2d0 2019-12-05T05:47:14.758 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:14.750+0000 7ffa75398700 1 -- 172.21.15.42:0/990652437 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ffa7007e710 msgr2=0x7ffa7007eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:47:14.758 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:14.750+0000 7ffa75398700 1 --2- 172.21.15.42:0/990652437 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ffa7007e710 0x7ffa7007eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:47:14.759 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:14.750+0000 7ffa75398700 1 -- 172.21.15.42:0/990652437 >> v1:172.21.15.40:6789/0 conn(0x7ffa7009f2d0 legacy=0x7ffa7009f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:47:14.759 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:14.750+0000 7ffa75398700 1 -- 172.21.15.42:0/990652437 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa7007f090 con 0x7ffa700a7730 2019-12-05T05:47:14.759 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:14.750+0000 7ffa75398700 1 --2- 172.21.15.42:0/990652437 >> v2:172.21.15.42:3300/0 conn(0x7ffa700a7730 0x7ffa7008df70 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7ffa64008b10 tx=0x7ffa640100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:14.759 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:14.750+0000 7ffa75398700 1 --2- 172.21.15.42:0/990652437 >> v2:172.21.15.42:3300/0 conn(0x7ffa700a7730 0x7ffa7008df70 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7ffa64008b10 tx=0x7ffa640100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:14.759 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:14.750+0000 7ffa75398700 1 --2- 172.21.15.42:0/990652437 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ffa700a7730 0x7ffa7008df70 secure :-1 s=READY pgs=1468 cs=0 l=1 rx=0x7ffa64008b10 tx=0x7ffa640100b0).ready entity=mon.1 client_cookie=5c713f3eac1aeaa2 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:47:14.759 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:14.750+0000 7ffa75398700 1 --2- 172.21.15.42:0/990652437 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ffa700a7730 0x7ffa7008df70 secure :-1 s=THROTTLE_DONE pgs=1468 cs=0 l=1 rx=0x7ffa64008b10 tx=0x7ffa640100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:14.759 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:14.750+0000 7ffa75398700 1 --2- 172.21.15.42:0/990652437 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ffa700a7730 0x7ffa7008df70 secure :-1 s=THROTTLE_DONE pgs=1468 cs=0 l=1 rx=0x7ffa64008b10 tx=0x7ffa640100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:14.760 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:14.750+0000 7ffa6ffff700 1 -- 172.21.15.42:0/990652437 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ffa6400a160 con 0x7ffa700a7730 2019-12-05T05:47:14.760 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:14.750+0000 7ffa75398700 1 --2- 172.21.15.42:0/990652437 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ffa700a7730 0x7ffa7008df70 secure :-1 s=THROTTLE_DONE pgs=1468 cs=0 l=1 rx=0x7ffa64008b10 tx=0x7ffa640100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:14.760 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:14.750+0000 7ffa6ffff700 1 -- 172.21.15.42:0/990652437 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa64011a30 con 0x7ffa700a7730 2019-12-05T05:47:14.760 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:14.750+0000 7ffa6ffff700 1 -- 172.21.15.42:0/990652437 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ffa6400a550 con 0x7ffa700a7730 2019-12-05T05:47:14.760 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:14.750+0000 7ffa76dfb700 1 -- 172.21.15.42:0/990652437 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ffa700a7730 msgr2=0x7ffa7008df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:47:14.760 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:14.750+0000 7ffa76dfb700 1 --2- 172.21.15.42:0/990652437 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ffa700a7730 0x7ffa7008df70 secure :-1 s=READY pgs=1468 cs=0 l=1 rx=0x7ffa64008b10 tx=0x7ffa640100b0).stop 2019-12-05T05:47:14.760 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:14.750+0000 7ffa76dfb700 1 -- 172.21.15.42:0/990652437 shutdown_connections 2019-12-05T05:47:14.761 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:14.750+0000 7ffa76dfb700 1 --2- 172.21.15.42:0/990652437 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ffa700a7730 0x7ffa7008df70 unknown :-1 s=CLOSED pgs=1468 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:47:14.761 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:14.750+0000 7ffa76dfb700 1 --2- 172.21.15.42:0/990652437 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ffa7007e710 0x7ffa7007eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:47:14.761 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:14.750+0000 7ffa76dfb700 1 -- 172.21.15.42:0/990652437 shutdown_connections 2019-12-05T05:47:14.761 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:14.750+0000 7ffa76dfb700 1 -- 172.21.15.42:0/990652437 wait complete. 2019-12-05T05:47:14.761 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:14.750+0000 7ffa76dfb700 1 -- 172.21.15.42:0/990652437 >> 172.21.15.42:0/990652437 conn(0x7ffa700aabd0 msgr2=0x7ffa700a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:47:14.764 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:14.754+0000 7ffa76dfb700 1 Processor -- start 2019-12-05T05:47:14.764 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:14.754+0000 7ffa76dfb700 1 -- start start 2019-12-05T05:47:14.764 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:14.754+0000 7ffa76dfb700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ffa7007d840 0x7ffa7007dc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:47:14.765 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:14.754+0000 7ffa76dfb700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ffa7011b0f0 0x7ffa7011b520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:47:14.765 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:14.754+0000 7ffa76dfb700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa700a2510 con 0x7ffa7011dd00 2019-12-05T05:47:14.765 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:14.754+0000 7ffa76dfb700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa7007e1c0 con 0x7ffa7007d840 2019-12-05T05:47:14.765 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:14.754+0000 7ffa76dfb700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa7011e130 con 0x7ffa7011b0f0 2019-12-05T05:47:14.765 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:14.754+0000 7ffa75398700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7ffa7011dd00 0x7ffa700a1d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:41258/0 (socket says 172.21.15.42:41258) 2019-12-05T05:47:14.765 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:14.754+0000 7ffa75398700 1 -- 172.21.15.42:0/2714860321 learned_addr learned my addr 172.21.15.42:0/2714860321 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:47:14.765 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:14.754+0000 7ffa74b97700 1 --2- 172.21.15.42:0/2714860321 >> v2:172.21.15.42:3300/0 conn(0x7ffa7007d840 0x7ffa7007dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:47:14.766 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:14.754+0000 7ffa75b99700 1 --2- 172.21.15.42:0/2714860321 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ffa7011b0f0 0x7ffa7011b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:47:14.766 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:14.754+0000 7ffa6dffb700 1 -- 172.21.15.42:0/2714860321 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 902234713 0 0) 0x7ffa700a2510 con 0x7ffa7011dd00 2019-12-05T05:47:14.766 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:14.754+0000 7ffa6dffb700 1 -- 172.21.15.42:0/2714860321 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa500030b0 con 0x7ffa7011dd00 2019-12-05T05:47:14.766 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:14.758+0000 7ffa74b97700 1 -- 172.21.15.42:0/2714860321 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ffa7011b0f0 msgr2=0x7ffa7011b520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:47:14.766 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:14.758+0000 7ffa74b97700 1 --2- 172.21.15.42:0/2714860321 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ffa7011b0f0 0x7ffa7011b520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:47:14.766 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:14.758+0000 7ffa74b97700 1 -- 172.21.15.42:0/2714860321 >> v1:172.21.15.42:6789/0 conn(0x7ffa7011dd00 legacy=0x7ffa700a1d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:47:14.766 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:14.758+0000 7ffa74b97700 1 -- 172.21.15.42:0/2714860321 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa7011ba70 con 0x7ffa7007d840 2019-12-05T05:47:14.767 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:14.758+0000 7ffa74b97700 1 --2- 172.21.15.42:0/2714860321 >> v2:172.21.15.42:3300/0 conn(0x7ffa7007d840 0x7ffa7007dc70 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7ffa700a6470 tx=0x7ffa5c010c20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:14.767 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:14.758+0000 7ffa74b97700 1 --2- 172.21.15.42:0/2714860321 >> v2:172.21.15.42:3300/0 conn(0x7ffa7007d840 0x7ffa7007dc70 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7ffa700a6470 tx=0x7ffa5c010c20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:14.767 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:14.758+0000 7ffa74b97700 1 --2- 172.21.15.42:0/2714860321 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ffa7007d840 0x7ffa7007dc70 secure :-1 s=READY pgs=1470 cs=0 l=1 rx=0x7ffa700a6470 tx=0x7ffa5c010c20).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:47:14.767 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:14.758+0000 7ffa74b97700 1 --2- 172.21.15.42:0/2714860321 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ffa7007d840 0x7ffa7007dc70 secure :-1 s=THROTTLE_DONE pgs=1470 cs=0 l=1 rx=0x7ffa700a6470 tx=0x7ffa5c010c20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:14.767 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:14.758+0000 7ffa6dffb700 1 -- 172.21.15.42:0/2714860321 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ffa5c012690 con 0x7ffa7007d840 2019-12-05T05:47:14.767 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:14.758+0000 7ffa74b97700 1 --2- 172.21.15.42:0/2714860321 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ffa7007d840 0x7ffa7007dc70 secure :-1 s=THROTTLE_DONE pgs=1470 cs=0 l=1 rx=0x7ffa700a6470 tx=0x7ffa5c010c20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:14.767 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:14.758+0000 7ffa74b97700 1 --2- 172.21.15.42:0/2714860321 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ffa7007d840 0x7ffa7007dc70 secure :-1 s=THROTTLE_DONE pgs=1470 cs=0 l=1 rx=0x7ffa700a6470 tx=0x7ffa5c010c20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:14.768 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:14.758+0000 7ffa6dffb700 1 -- 172.21.15.42:0/2714860321 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa5c007150 con 0x7ffa7007d840 2019-12-05T05:47:14.768 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:14.758+0000 7ffa6dffb700 1 -- 172.21.15.42:0/2714860321 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ffa5c012cb0 con 0x7ffa7007d840 2019-12-05T05:47:14.768 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:14.758+0000 7ffa76dfb700 1 -- 172.21.15.42:0/2714860321 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa701355a0 con 0x7ffa7007d840 2019-12-05T05:47:14.768 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:14.758+0000 7ffa76dfb700 1 -- 172.21.15.42:0/2714860321 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa70135aa0 con 0x7ffa7007d840 2019-12-05T05:47:14.768 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:14.758+0000 7ffa74b97700 1 --2- 172.21.15.42:0/2714860321 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ffa7007d840 0x7ffa7007dc70 secure :-1 s=THROTTLE_DONE pgs=1470 cs=0 l=1 rx=0x7ffa700a6470 tx=0x7ffa5c010c20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:14.768 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:14.758+0000 7ffa6dffb700 1 -- 172.21.15.42:0/2714860321 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7ffa5c005540 con 0x7ffa7007d840 2019-12-05T05:47:14.768 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:14.758+0000 7ffa74b97700 1 --2- 172.21.15.42:0/2714860321 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ffa7007d840 0x7ffa7007dc70 secure :-1 s=THROTTLE_DONE pgs=1470 cs=0 l=1 rx=0x7ffa700a6470 tx=0x7ffa5c010c20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:14.769 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:14.758+0000 7ffa6dffb700 1 --2- 172.21.15.42:0/2714860321 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ffa5003b280 0x7ffa5003d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:47:14.769 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:14.758+0000 7ffa6dffb700 1 -- 172.21.15.42:0/2714860321 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 8611+0+0 (secure 0 0 0) 0x7ffa5c04b280 con 0x7ffa7007d840 2019-12-05T05:47:14.769 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:14.758+0000 7ffa75398700 1 --2- 172.21.15.42:0/2714860321 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ffa5003b280 0x7ffa5003d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:47:14.769 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:14.758+0000 7ffa75398700 1 --2- 172.21.15.42:0/2714860321 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ffa5003b280 0x7ffa5003d790 crc :-1 s=READY pgs=633 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:47:14.769 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:14.766+0000 7ffa76dfb700 1 -- 172.21.15.42:0/2714860321 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa7005d100 con 0x7ffa7007d840 2019-12-05T05:47:14.772 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:14.770+0000 7ffa74b97700 1 --2- 172.21.15.42:0/2714860321 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ffa7007d840 0x7ffa7007dc70 secure :-1 s=THROTTLE_DONE pgs=1470 cs=0 l=1 rx=0x7ffa700a6470 tx=0x7ffa5c010c20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:14.772 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:14.770+0000 7ffa6dffb700 1 -- 172.21.15.42:0/2714860321 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7ffa5c032300 con 0x7ffa7007d840 2019-12-05T05:47:14.934 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:14.930+0000 7ffa76dfb700 1 -- 172.21.15.42:0/2714860321 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffa7005b8b0 con 0x7ffa7007d840 2019-12-05T05:47:14.935 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:14.930+0000 7ffa74b97700 1 --2- 172.21.15.42:0/2714860321 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ffa7007d840 0x7ffa7007dc70 secure :-1 s=THROTTLE_DONE pgs=1470 cs=0 l=1 rx=0x7ffa700a6470 tx=0x7ffa5c010c20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:14.935 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:14.930+0000 7ffa6dffb700 1 -- 172.21.15.42:0/2714860321 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 179 v179) v1 ==== 95+0+1774 (secure 0 0 0) 0x7ffa5c032560 con 0x7ffa7007d840 2019-12-05T05:47:14.935 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 179 2019-12-05T05:47:14.936 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:47:14.936 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":179,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":179,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:47:14.051659+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6140},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6140":{"gid":6140,"name":"b","rank":0,"incarnation":179,"state":"up:replay","state_seq":2,"addr":"172.21.15.40:6833/2146240383","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2146240383},{"type":"v1","addr":"172.21.15.40:6833","nonce":2146240383}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:47:14.938 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:14.934+0000 7ffa76dfb700 1 -- 172.21.15.42:0/2714860321 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ffa5003b280 msgr2=0x7ffa5003d790 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:47:14.938 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:14.934+0000 7ffa76dfb700 1 --2- 172.21.15.42:0/2714860321 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ffa5003b280 0x7ffa5003d790 crc :-1 s=READY pgs=633 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:47:14.938 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:14.934+0000 7ffa76dfb700 1 -- 172.21.15.42:0/2714860321 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ffa7007d840 msgr2=0x7ffa7007dc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:47:14.938 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:14.934+0000 7ffa76dfb700 1 --2- 172.21.15.42:0/2714860321 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ffa7007d840 0x7ffa7007dc70 secure :-1 s=READY pgs=1470 cs=0 l=1 rx=0x7ffa700a6470 tx=0x7ffa5c010c20).stop 2019-12-05T05:47:14.939 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:14.934+0000 7ffa76dfb700 1 -- 172.21.15.42:0/2714860321 shutdown_connections 2019-12-05T05:47:14.939 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:14.934+0000 7ffa76dfb700 1 --2- 172.21.15.42:0/2714860321 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ffa5003b280 0x7ffa5003d790 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:47:14.939 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:14.934+0000 7ffa76dfb700 1 --2- 172.21.15.42:0/2714860321 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ffa7011b0f0 0x7ffa7011b520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:47:14.939 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:14.934+0000 7ffa76dfb700 1 --2- 172.21.15.42:0/2714860321 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ffa7007d840 0x7ffa7007dc70 unknown :-1 s=CLOSED pgs=1470 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:47:14.939 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:14.934+0000 7ffa76dfb700 1 -- 172.21.15.42:0/2714860321 shutdown_connections 2019-12-05T05:47:14.939 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:14.934+0000 7ffa76dfb700 1 -- 172.21.15.42:0/2714860321 wait complete. 2019-12-05T05:47:14.939 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:14.934+0000 7ffa76dfb700 1 -- 172.21.15.42:0/2714860321 >> 172.21.15.42:0/2714860321 conn(0x7ffa700aabd0 msgr2=0x7ffa7011c3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:47:14.948 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a down, removed from mdsmap 2019-12-05T05:47:14.948 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2019-12-05T05:47:14.949 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:47:15.048 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:15.038+0000 7f1840839700 1 Processor -- start 2019-12-05T05:47:15.048 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:15.038+0000 7f1840839700 1 -- start start 2019-12-05T05:47:15.049 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:15.042+0000 7f1840839700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f183c0a7740 0x7f183c08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:47:15.049 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:15.042+0000 7f1840839700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f183c09fae0 0x7f183c07eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:47:15.049 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:15.042+0000 7f1840839700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f183c0a0710 con 0x7f183c07e720 2019-12-05T05:47:15.049 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:15.042+0000 7f1840839700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f183c0a0b00 con 0x7f183c0a7740 2019-12-05T05:47:15.049 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:15.042+0000 7f1840839700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f183c08e5b0 con 0x7f183c09fae0 2019-12-05T05:47:15.049 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:15.042+0000 7f183ad9d700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f183c0a7740 0x7f183c08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:47:15.049 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:15.042+0000 7f183b59e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f183c09fae0 0x7f183c07eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:47:15.050 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:15.042+0000 7f183ad9d700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f183c0a7740 0x7f183c08df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.42:57276/0 (socket says 172.21.15.42:57276) 2019-12-05T05:47:15.050 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:15.042+0000 7f183b59e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f183c09fae0 0x7f183c07eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:41230/0 (socket says 172.21.15.42:41230) 2019-12-05T05:47:15.050 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:15.042+0000 7f183ad9d700 1 -- 172.21.15.42:0/727291631 learned_addr learned my addr 172.21.15.42:0/727291631 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:47:15.050 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:15.042+0000 7f183ad9d700 1 -- 172.21.15.42:0/727291631 >> v2:172.21.15.42:3300/0 conn(0x7f183c09fae0 msgr2=0x7f183c07eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:47:15.050 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:15.042+0000 7f183ad9d700 1 --2- 172.21.15.42:0/727291631 >> v2:172.21.15.42:3300/0 conn(0x7f183c09fae0 0x7f183c07eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:47:15.050 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:15.042+0000 7f183ad9d700 1 -- 172.21.15.42:0/727291631 >> v1:172.21.15.42:6789/0 conn(0x7f183c07e720 legacy=0x7f183c09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:47:15.050 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:15.042+0000 7f183ad9d700 1 -- 172.21.15.42:0/727291631 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f183c07f0a0 con 0x7f183c0a7740 2019-12-05T05:47:15.051 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:15.042+0000 7f183ad9d700 1 --2- 172.21.15.42:0/727291631 >> v2:172.21.15.40:3300/0 conn(0x7f183c0a7740 0x7f183c08df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f182c002f30 tx=0x7f182c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:15.051 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:15.042+0000 7f183ad9d700 1 --2- 172.21.15.42:0/727291631 >> v2:172.21.15.40:3300/0 conn(0x7f183c0a7740 0x7f183c08df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f182c002f30 tx=0x7f182c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:15.051 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:15.042+0000 7f183ad9d700 1 --2- 172.21.15.42:0/727291631 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f183c0a7740 0x7f183c08df80 secure :-1 s=READY pgs=1141 cs=0 l=1 rx=0x7f182c002f30 tx=0x7f182c00fd30).ready entity=mon.0 client_cookie=11fb5aaed21017af server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:47:15.051 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:15.042+0000 7f183ad9d700 1 --2- 172.21.15.42:0/727291631 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f183c0a7740 0x7f183c08df80 secure :-1 s=THROTTLE_DONE pgs=1141 cs=0 l=1 rx=0x7f182c002f30 tx=0x7f182c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:15.051 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:15.042+0000 7f183ad9d700 1 --2- 172.21.15.42:0/727291631 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f183c0a7740 0x7f183c08df80 secure :-1 s=THROTTLE_DONE pgs=1141 cs=0 l=1 rx=0x7f182c002f30 tx=0x7f182c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:15.051 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:15.042+0000 7f1839d9b700 1 -- 172.21.15.42:0/727291631 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f182c008c90 con 0x7f183c0a7740 2019-12-05T05:47:15.051 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:15.042+0000 7f183ad9d700 1 --2- 172.21.15.42:0/727291631 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f183c0a7740 0x7f183c08df80 secure :-1 s=THROTTLE_DONE pgs=1141 cs=0 l=1 rx=0x7f182c002f30 tx=0x7f182c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:15.052 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:15.042+0000 7f1839d9b700 1 -- 172.21.15.42:0/727291631 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f182c0088b0 con 0x7f183c0a7740 2019-12-05T05:47:15.052 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:15.042+0000 7f1839d9b700 1 -- 172.21.15.42:0/727291631 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f182c011ad0 con 0x7f183c0a7740 2019-12-05T05:47:15.052 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:15.042+0000 7f1840839700 1 -- 172.21.15.42:0/727291631 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f183c0a7740 msgr2=0x7f183c08df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:47:15.052 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:15.042+0000 7f1840839700 1 --2- 172.21.15.42:0/727291631 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f183c0a7740 0x7f183c08df80 secure :-1 s=READY pgs=1141 cs=0 l=1 rx=0x7f182c002f30 tx=0x7f182c00fd30).stop 2019-12-05T05:47:15.052 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:15.046+0000 7f1840839700 1 -- 172.21.15.42:0/727291631 shutdown_connections 2019-12-05T05:47:15.052 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:15.046+0000 7f1840839700 1 --2- 172.21.15.42:0/727291631 >> v2:172.21.15.42:3300/0 conn(0x7f183c09fae0 0x7f183c07eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:47:15.052 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:15.046+0000 7f1840839700 1 --2- 172.21.15.42:0/727291631 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f183c0a7740 0x7f183c08df80 unknown :-1 s=CLOSED pgs=1141 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:47:15.053 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:15.046+0000 7f1840839700 1 -- 172.21.15.42:0/727291631 shutdown_connections 2019-12-05T05:47:15.053 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:15.046+0000 7f1840839700 1 -- 172.21.15.42:0/727291631 wait complete. 2019-12-05T05:47:15.053 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:15.046+0000 7f1840839700 1 -- 172.21.15.42:0/727291631 >> 172.21.15.42:0/727291631 conn(0x7f183c0aabe0 msgr2=0x7f183c0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:47:15.056 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:15.046+0000 7f1840839700 1 Processor -- start 2019-12-05T05:47:15.056 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:15.046+0000 7f1840839700 1 -- start start 2019-12-05T05:47:15.056 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:15.050+0000 7f1840839700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f183c123a70 0x7f183c148340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:47:15.056 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:15.050+0000 7f1840839700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f183c146ae0 0x7f183c12e480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:47:15.056 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:15.050+0000 7f1840839700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f183c145860 con 0x7f183c1466b0 2019-12-05T05:47:15.057 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:15.050+0000 7f1840839700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f183c14ecd0 con 0x7f183c146ae0 2019-12-05T05:47:15.057 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:15.050+0000 7f1840839700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f183c146170 con 0x7f183c123a70 2019-12-05T05:47:15.057 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:15.050+0000 7f183ad9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f183c123a70 0x7f183c148340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:47:15.057 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:15.050+0000 7f183b59e700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f183c146ae0 0x7f183c12e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:47:15.057 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:15.050+0000 7f183b59e700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f183c146ae0 0x7f183c12e480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.42:57284/0 (socket says 172.21.15.42:57284) 2019-12-05T05:47:15.057 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:15.050+0000 7f183b59e700 1 -- 172.21.15.42:0/4090291174 learned_addr learned my addr 172.21.15.42:0/4090291174 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:47:15.057 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:15.050+0000 7f183a59c700 1 --1- 172.21.15.42:0/4090291174 >> v1:172.21.15.40:6789/0 conn(0x7f183c1466b0 0x7f183c144ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:50714/0 (socket says 172.21.15.42:50714) 2019-12-05T05:47:15.058 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:15.050+0000 7f18337fe700 1 -- 172.21.15.42:0/4090291174 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3252822607 0 0) 0x7f183c145860 con 0x7f183c1466b0 2019-12-05T05:47:15.058 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:15.050+0000 7f18337fe700 1 -- 172.21.15.42:0/4090291174 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18200030b0 con 0x7f183c1466b0 2019-12-05T05:47:15.058 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:15.050+0000 7f183b59e700 1 -- 172.21.15.42:0/4090291174 >> v2:172.21.15.42:3300/0 conn(0x7f183c123a70 msgr2=0x7f183c148340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:47:15.058 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:15.050+0000 7f183b59e700 1 --2- 172.21.15.42:0/4090291174 >> v2:172.21.15.42:3300/0 conn(0x7f183c123a70 0x7f183c148340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:47:15.058 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:15.050+0000 7f183b59e700 1 -- 172.21.15.42:0/4090291174 >> v1:172.21.15.40:6789/0 conn(0x7f183c1466b0 legacy=0x7f183c144ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:47:15.058 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:15.050+0000 7f183b59e700 1 -- 172.21.15.42:0/4090291174 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f183c148890 con 0x7f183c146ae0 2019-12-05T05:47:15.058 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:15.050+0000 7f183b59e700 1 --2- 172.21.15.42:0/4090291174 >> v2:172.21.15.40:3300/0 conn(0x7f183c146ae0 0x7f183c12e480 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f183c14ee90 tx=0x7f1834010d00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:15.059 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:15.050+0000 7f183b59e700 1 --2- 172.21.15.42:0/4090291174 >> v2:172.21.15.40:3300/0 conn(0x7f183c146ae0 0x7f183c12e480 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f183c14ee90 tx=0x7f1834010d00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:15.059 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:15.050+0000 7f183b59e700 1 --2- 172.21.15.42:0/4090291174 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f183c146ae0 0x7f183c12e480 secure :-1 s=READY pgs=1143 cs=0 l=1 rx=0x7f183c14ee90 tx=0x7f1834010d00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:47:15.059 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:15.050+0000 7f183b59e700 1 --2- 172.21.15.42:0/4090291174 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f183c146ae0 0x7f183c12e480 secure :-1 s=THROTTLE_DONE pgs=1143 cs=0 l=1 rx=0x7f183c14ee90 tx=0x7f1834010d00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:15.059 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:15.050+0000 7f18337fe700 1 -- 172.21.15.42:0/4090291174 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f18340129f0 con 0x7f183c146ae0 2019-12-05T05:47:15.059 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:15.050+0000 7f183b59e700 1 --2- 172.21.15.42:0/4090291174 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f183c146ae0 0x7f183c12e480 secure :-1 s=THROTTLE_DONE pgs=1143 cs=0 l=1 rx=0x7f183c14ee90 tx=0x7f1834010d00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:15.059 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:15.050+0000 7f183b59e700 1 --2- 172.21.15.42:0/4090291174 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f183c146ae0 0x7f183c12e480 secure :-1 s=THROTTLE_DONE pgs=1143 cs=0 l=1 rx=0x7f183c14ee90 tx=0x7f1834010d00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:15.059 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:15.050+0000 7f18337fe700 1 -- 172.21.15.42:0/4090291174 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1834012610 con 0x7f183c146ae0 2019-12-05T05:47:15.060 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:15.050+0000 7f18337fe700 1 -- 172.21.15.42:0/4090291174 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f183400e570 con 0x7f183c146ae0 2019-12-05T05:47:15.060 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:15.050+0000 7f1840839700 1 -- 172.21.15.42:0/4090291174 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f183c145eb0 con 0x7f183c146ae0 2019-12-05T05:47:15.060 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:15.050+0000 7f1840839700 1 -- 172.21.15.42:0/4090291174 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f183c1452d0 con 0x7f183c146ae0 2019-12-05T05:47:15.060 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:15.050+0000 7f183b59e700 1 --2- 172.21.15.42:0/4090291174 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f183c146ae0 0x7f183c12e480 secure :-1 s=THROTTLE_DONE pgs=1143 cs=0 l=1 rx=0x7f183c14ee90 tx=0x7f1834010d00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:15.060 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:15.054+0000 7f18337fe700 1 -- 172.21.15.42:0/4090291174 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f18340058c0 con 0x7f183c146ae0 2019-12-05T05:47:15.060 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:15.054+0000 7f183b59e700 1 --2- 172.21.15.42:0/4090291174 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f183c146ae0 0x7f183c12e480 secure :-1 s=THROTTLE_DONE pgs=1143 cs=0 l=1 rx=0x7f183c14ee90 tx=0x7f1834010d00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:15.060 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:15.054+0000 7f18337fe700 1 --2- 172.21.15.42:0/4090291174 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f182003b240 0x7f182003d750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:47:15.061 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:15.054+0000 7f18337fe700 1 -- 172.21.15.42:0/4090291174 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 8611+0+0 (secure 0 0 0) 0x7f183404b750 con 0x7f183c146ae0 2019-12-05T05:47:15.061 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:15.054+0000 7f183ad9d700 1 --2- 172.21.15.42:0/4090291174 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f182003b240 0x7f182003d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:47:15.061 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:15.054+0000 7f183ad9d700 1 --2- 172.21.15.42:0/4090291174 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f182003b240 0x7f182003d750 crc :-1 s=READY pgs=634 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:47:15.062 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:15.054+0000 7f1840839700 1 -- 172.21.15.42:0/4090291174 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f183c05d110 con 0x7f183c146ae0 2019-12-05T05:47:15.067 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:15.066+0000 7f183b59e700 1 --2- 172.21.15.42:0/4090291174 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f183c146ae0 0x7f183c12e480 secure :-1 s=THROTTLE_DONE pgs=1143 cs=0 l=1 rx=0x7f183c14ee90 tx=0x7f1834010d00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:15.067 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:15.066+0000 7f18337fe700 1 -- 172.21.15.42:0/4090291174 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f183c145eb0 con 0x7f183c146ae0 2019-12-05T05:47:15.234 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:15.230+0000 7f1840839700 1 -- 172.21.15.42:0/4090291174 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f183c05b8c0 con 0x7f183c146ae0 2019-12-05T05:47:15.235 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:15.234+0000 7f183b59e700 1 --2- 172.21.15.42:0/4090291174 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f183c146ae0 0x7f183c12e480 secure :-1 s=THROTTLE_DONE pgs=1143 cs=0 l=1 rx=0x7f183c14ee90 tx=0x7f1834010d00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:15.236 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:15.234+0000 7f18337fe700 1 -- 172.21.15.42:0/4090291174 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 179 v179) v1 ==== 95+0+1774 (secure 0 0 0) 0x7f1834032770 con 0x7f183c146ae0 2019-12-05T05:47:15.236 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 179 2019-12-05T05:47:15.236 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:47:15.236 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":179,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":179,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:47:14.051659+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6140},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6140":{"gid":6140,"name":"b","rank":0,"incarnation":179,"state":"up:replay","state_seq":2,"addr":"172.21.15.40:6833/2146240383","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2146240383},{"type":"v1","addr":"172.21.15.40:6833","nonce":2146240383}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:47:15.238 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:15.234+0000 7f1840839700 1 -- 172.21.15.42:0/4090291174 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f182003b240 msgr2=0x7f182003d750 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:47:15.238 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:15.234+0000 7f1840839700 1 --2- 172.21.15.42:0/4090291174 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f182003b240 0x7f182003d750 crc :-1 s=READY pgs=634 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:47:15.239 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:15.234+0000 7f1840839700 1 -- 172.21.15.42:0/4090291174 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f183c146ae0 msgr2=0x7f183c12e480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:47:15.239 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:15.234+0000 7f1840839700 1 --2- 172.21.15.42:0/4090291174 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f183c146ae0 0x7f183c12e480 secure :-1 s=READY pgs=1143 cs=0 l=1 rx=0x7f183c14ee90 tx=0x7f1834010d00).stop 2019-12-05T05:47:15.239 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:15.234+0000 7f1840839700 1 -- 172.21.15.42:0/4090291174 shutdown_connections 2019-12-05T05:47:15.239 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:15.234+0000 7f1840839700 1 --2- 172.21.15.42:0/4090291174 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f182003b240 0x7f182003d750 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:47:15.239 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:15.234+0000 7f1840839700 1 --2- 172.21.15.42:0/4090291174 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f183c146ae0 0x7f183c12e480 unknown :-1 s=CLOSED pgs=1143 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:47:15.239 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:15.234+0000 7f1840839700 1 --2- 172.21.15.42:0/4090291174 >> v2:172.21.15.42:3300/0 conn(0x7f183c123a70 0x7f183c148340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:47:15.239 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:15.234+0000 7f1840839700 1 -- 172.21.15.42:0/4090291174 shutdown_connections 2019-12-05T05:47:15.240 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:15.234+0000 7f1840839700 1 -- 172.21.15.42:0/4090291174 wait complete. 2019-12-05T05:47:15.240 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:15.234+0000 7f1840839700 1 -- 172.21.15.42:0/4090291174 >> 172.21.15.42:0/4090291174 conn(0x7f183c07df10 msgr2=0x7f183c059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:47:15.248 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 179, "feature_flags": { "enable_multiple": false, "ever_enabled_multiple": false }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2019-12-05T05:03:35.604738+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 179, "ever_allowed_features": 2, "explicitly_allowed_features": 2, "failed": [], "flags": 18, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_6140": { "addr": "172.21.15.40:6833/2146240383", "addrs": { "addrvec": [ { "addr": "172.21.15.40:6832", "nonce": 2146240383, "type": "v2" }, { "addr": "172.21.15.40:6833", "nonce": 2146240383, "type": "v1" } ] }, "export_targets": [], "features": 4540138292836696063, "flags": 0, "gid": 6140, "incarnation": 179, "name": "b", "rank": 0, "state": "up:replay", "state_seq": 2 } }, "last_failure": 0, "last_failure_osd_epoch": 79, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 2, "min_compat_client": "0 (unknown)", "modified": "2019-12-05T05:47:14.051659+0000", "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6140 } } } ], "standbys": [] } 2019-12-05T05:47:17.249 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:47:17.347 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:17.338+0000 7f6271b77700 1 Processor -- start 2019-12-05T05:47:17.347 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:17.338+0000 7f6271b77700 1 -- start start 2019-12-05T05:47:17.347 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:17.338+0000 7f6271b77700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f626c08df80 0x7f626c09f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:47:17.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:17.338+0000 7f6271b77700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f626c09fc60 con 0x7f626c09f830 2019-12-05T05:47:17.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:17.338+0000 7f6271b77700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f626c0a0710 con 0x7f626c0a7740 2019-12-05T05:47:17.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:17.338+0000 7f6271b77700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f626c0a0b00 con 0x7f626c08df80 2019-12-05T05:47:17.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:17.342+0000 7f626bfff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f626c0a7740 0x7f626c07e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:41280/0 (socket says 172.21.15.42:41280) 2019-12-05T05:47:17.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:17.342+0000 7f626bfff700 1 -- 172.21.15.42:0/2902941317 learned_addr learned my addr 172.21.15.42:0/2902941317 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:47:17.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:17.342+0000 7f626b7fe700 1 --2- 172.21.15.42:0/2902941317 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f626c08df80 0x7f626c09f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:47:17.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:17.342+0000 7f626affd700 1 -- 172.21.15.42:0/2902941317 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2102945179 0 0) 0x7f626c0a0710 con 0x7f626c0a7740 2019-12-05T05:47:17.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:17.342+0000 7f626affd700 1 -- 172.21.15.42:0/2902941317 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62540030b0 con 0x7f626c0a7740 2019-12-05T05:47:17.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:17.342+0000 7f626affd700 1 -- 172.21.15.42:0/2902941317 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1807698935 0 0) 0x7f626c09fc60 con 0x7f626c09f830 2019-12-05T05:47:17.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:17.342+0000 7f626affd700 1 -- 172.21.15.42:0/2902941317 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f626c0a0710 con 0x7f626c09f830 2019-12-05T05:47:17.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:17.342+0000 7f626b7fe700 1 -- 172.21.15.42:0/2902941317 >> v1:172.21.15.42:6789/0 conn(0x7f626c0a7740 legacy=0x7f626c07e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:47:17.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:17.342+0000 7f626affd700 1 -- 172.21.15.42:0/2902941317 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1416868508 0 0) 0x7f62540030b0 con 0x7f626c0a7740 2019-12-05T05:47:17.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:17.342+0000 7f626b7fe700 1 -- 172.21.15.42:0/2902941317 >> v1:172.21.15.40:6789/0 conn(0x7f626c09f830 legacy=0x7f626c14efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:47:17.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:17.342+0000 7f626b7fe700 1 -- 172.21.15.42:0/2902941317 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f626c07ef20 con 0x7f626c08df80 2019-12-05T05:47:17.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:17.342+0000 7f626b7fe700 1 --2- 172.21.15.42:0/2902941317 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f626c08df80 0x7f626c09f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f6260008b10 tx=0x7f625c001950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:17.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:17.342+0000 7f626b7fe700 1 --2- 172.21.15.42:0/2902941317 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f626c08df80 0x7f626c09f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f6260008b10 tx=0x7f625c001950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:17.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:17.342+0000 7f626b7fe700 1 --2- 172.21.15.42:0/2902941317 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f626c08df80 0x7f626c09f2e0 secure :-1 s=READY pgs=330 cs=0 l=1 rx=0x7f6260008b10 tx=0x7f625c001950).ready entity=mon.2 client_cookie=f1e3b44d71199128 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:47:17.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:17.342+0000 7f626b7fe700 1 --2- 172.21.15.42:0/2902941317 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f626c08df80 0x7f626c09f2e0 secure :-1 s=THROTTLE_DONE pgs=330 cs=0 l=1 rx=0x7f6260008b10 tx=0x7f625c001950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:17.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:17.342+0000 7f626affd700 1 -- 172.21.15.42:0/2902941317 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f625c001750 con 0x7f626c08df80 2019-12-05T05:47:17.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:17.342+0000 7f626b7fe700 1 --2- 172.21.15.42:0/2902941317 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f626c08df80 0x7f626c09f2e0 secure :-1 s=THROTTLE_DONE pgs=330 cs=0 l=1 rx=0x7f6260008b10 tx=0x7f625c001950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:17.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:17.342+0000 7f626affd700 1 -- 172.21.15.42:0/2902941317 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f625c001560 con 0x7f626c08df80 2019-12-05T05:47:17.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:17.342+0000 7f626b7fe700 1 --2- 172.21.15.42:0/2902941317 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f626c08df80 0x7f626c09f2e0 secure :-1 s=THROTTLE_DONE pgs=330 cs=0 l=1 rx=0x7f6260008b10 tx=0x7f625c001950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:17.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:17.342+0000 7f626affd700 1 -- 172.21.15.42:0/2902941317 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6260011ac0 con 0x7f626c08df80 2019-12-05T05:47:17.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:17.342+0000 7f6271b77700 1 -- 172.21.15.42:0/2902941317 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f626c08df80 msgr2=0x7f626c09f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:47:17.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:17.342+0000 7f6271b77700 1 --2- 172.21.15.42:0/2902941317 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f626c08df80 0x7f626c09f2e0 secure :-1 s=READY pgs=330 cs=0 l=1 rx=0x7f6260008b10 tx=0x7f625c001950).stop 2019-12-05T05:47:17.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:17.342+0000 7f6271b77700 1 -- 172.21.15.42:0/2902941317 shutdown_connections 2019-12-05T05:47:17.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:17.342+0000 7f6271b77700 1 --2- 172.21.15.42:0/2902941317 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f626c08df80 0x7f626c09f2e0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:47:17.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:17.342+0000 7f6271b77700 1 -- 172.21.15.42:0/2902941317 shutdown_connections 2019-12-05T05:47:17.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:17.342+0000 7f6271b77700 1 -- 172.21.15.42:0/2902941317 wait complete. 2019-12-05T05:47:17.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:17.342+0000 7f6271b77700 1 -- 172.21.15.42:0/2902941317 >> 172.21.15.42:0/2902941317 conn(0x7f626c0aabe0 msgr2=0x7f626c0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:47:17.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:17.346+0000 7f6271b77700 1 Processor -- start 2019-12-05T05:47:17.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:17.346+0000 7f6271b77700 1 -- start start 2019-12-05T05:47:17.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:17.346+0000 7f6271b77700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f626c07d470 0x7f626c08f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:47:17.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:17.346+0000 7f6271b77700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f626c11a690 con 0x7f626c07ca50 2019-12-05T05:47:17.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:17.346+0000 7f6271b77700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f626c08f8c0 con 0x7f626c118b70 2019-12-05T05:47:17.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:17.346+0000 7f6271b77700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f626c11a9a0 con 0x7f626c07d470 2019-12-05T05:47:17.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:17.346+0000 7f6270915700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f626c07d470 0x7f626c08f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:47:17.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:17.346+0000 7f6270915700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f626c07d470 0x7f626c08f370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:41250/0 (socket says 172.21.15.42:41250) 2019-12-05T05:47:17.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:17.346+0000 7f6270915700 1 -- 172.21.15.42:0/1886012750 learned_addr learned my addr 172.21.15.42:0/1886012750 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:47:17.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:17.346+0000 7f6270915700 1 -- 172.21.15.42:0/1886012750 >> v1:172.21.15.42:6789/0 conn(0x7f626c118b70 legacy=0x7f626c07c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:47:17.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:17.346+0000 7f6270915700 1 -- 172.21.15.42:0/1886012750 >> v1:172.21.15.40:6789/0 conn(0x7f626c07ca50 legacy=0x7f626c08eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:47:17.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:17.346+0000 7f6268ff9700 1 -- 172.21.15.42:0/1886012750 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 556009019 0 0) 0x7f626c11a690 con 0x7f626c07ca50 2019-12-05T05:47:17.358 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:17.346+0000 7f6270915700 1 -- 172.21.15.42:0/1886012750 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f626c11b420 con 0x7f626c07d470 2019-12-05T05:47:17.358 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:17.346+0000 7f6270915700 1 --2- 172.21.15.42:0/1886012750 >> v2:172.21.15.42:3300/0 conn(0x7f626c07d470 0x7f626c08f370 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f626c0a7300 tx=0x7f62640106e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:17.358 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:17.346+0000 7f6270915700 1 --2- 172.21.15.42:0/1886012750 >> v2:172.21.15.42:3300/0 conn(0x7f626c07d470 0x7f626c08f370 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f626c0a7300 tx=0x7f62640106e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:17.358 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:17.346+0000 7f6270915700 1 --2- 172.21.15.42:0/1886012750 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f626c07d470 0x7f626c08f370 secure :-1 s=READY pgs=1474 cs=0 l=1 rx=0x7f626c0a7300 tx=0x7f62640106e0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:47:17.358 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:17.350+0000 7f6270915700 1 --2- 172.21.15.42:0/1886012750 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f626c07d470 0x7f626c08f370 secure :-1 s=THROTTLE_DONE pgs=1474 cs=0 l=1 rx=0x7f626c0a7300 tx=0x7f62640106e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:17.358 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:17.350+0000 7f6268ff9700 1 -- 172.21.15.42:0/1886012750 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6264006b30 con 0x7f626c07d470 2019-12-05T05:47:17.358 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:17.350+0000 7f6270915700 1 --2- 172.21.15.42:0/1886012750 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f626c07d470 0x7f626c08f370 secure :-1 s=THROTTLE_DONE pgs=1474 cs=0 l=1 rx=0x7f626c0a7300 tx=0x7f62640106e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:17.359 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:17.350+0000 7f6270915700 1 --2- 172.21.15.42:0/1886012750 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f626c07d470 0x7f626c08f370 secure :-1 s=THROTTLE_DONE pgs=1474 cs=0 l=1 rx=0x7f626c0a7300 tx=0x7f62640106e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:17.359 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:17.350+0000 7f6268ff9700 1 -- 172.21.15.42:0/1886012750 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6264006750 con 0x7f626c07d470 2019-12-05T05:47:17.359 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:17.350+0000 7f6271b77700 1 -- 172.21.15.42:0/1886012750 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f626c131850 con 0x7f626c07d470 2019-12-05T05:47:17.359 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:17.350+0000 7f6268ff9700 1 -- 172.21.15.42:0/1886012750 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6264007640 con 0x7f626c07d470 2019-12-05T05:47:17.359 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:17.350+0000 7f6271b77700 1 -- 172.21.15.42:0/1886012750 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f626c131e00 con 0x7f626c07d470 2019-12-05T05:47:17.359 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:17.350+0000 7f6270915700 1 --2- 172.21.15.42:0/1886012750 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f626c07d470 0x7f626c08f370 secure :-1 s=THROTTLE_DONE pgs=1474 cs=0 l=1 rx=0x7f626c0a7300 tx=0x7f62640106e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:17.359 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:17.350+0000 7f6268ff9700 1 -- 172.21.15.42:0/1886012750 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f6264007950 con 0x7f626c07d470 2019-12-05T05:47:17.360 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:17.350+0000 7f6270915700 1 --2- 172.21.15.42:0/1886012750 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f626c07d470 0x7f626c08f370 secure :-1 s=THROTTLE_DONE pgs=1474 cs=0 l=1 rx=0x7f626c0a7300 tx=0x7f62640106e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:17.360 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:17.350+0000 7f6268ff9700 1 --2- 172.21.15.42:0/1886012750 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f624c03ac60 0x7f624c03d170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:47:17.360 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:17.350+0000 7f6268ff9700 1 -- 172.21.15.42:0/1886012750 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 8611+0+0 (secure 0 0 0) 0x7f626404b1d0 con 0x7f626c07d470 2019-12-05T05:47:17.360 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:17.350+0000 7f626bfff700 1 --2- 172.21.15.42:0/1886012750 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f624c03ac60 0x7f624c03d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:47:17.360 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:17.350+0000 7f626bfff700 1 --2- 172.21.15.42:0/1886012750 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f624c03ac60 0x7f624c03d170 crc :-1 s=READY pgs=635 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:47:17.361 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:17.354+0000 7f6271b77700 1 -- 172.21.15.42:0/1886012750 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f626c05d110 con 0x7f626c07d470 2019-12-05T05:47:17.367 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:17.366+0000 7f6270915700 1 --2- 172.21.15.42:0/1886012750 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f626c07d470 0x7f626c08f370 secure :-1 s=THROTTLE_DONE pgs=1474 cs=0 l=1 rx=0x7f626c0a7300 tx=0x7f62640106e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:17.368 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:17.366+0000 7f6268ff9700 1 -- 172.21.15.42:0/1886012750 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f6264032250 con 0x7f626c07d470 2019-12-05T05:47:17.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:17.534+0000 7f6271b77700 1 -- 172.21.15.42:0/1886012750 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f626c05b8c0 con 0x7f626c07d470 2019-12-05T05:47:17.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:17.534+0000 7f6270915700 1 --2- 172.21.15.42:0/1886012750 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f626c07d470 0x7f626c08f370 secure :-1 s=THROTTLE_DONE pgs=1474 cs=0 l=1 rx=0x7f626c0a7300 tx=0x7f62640106e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:17.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:17.534+0000 7f6268ff9700 1 -- 172.21.15.42:0/1886012750 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 180 v180) v1 ==== 95+0+1778 (secure 0 0 0) 0x7f62640324b0 con 0x7f626c07d470 2019-12-05T05:47:17.538 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 180 2019-12-05T05:47:17.539 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:47:17.539 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":180,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":180,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:47:17.487639+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6140},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6140":{"gid":6140,"name":"b","rank":0,"incarnation":179,"state":"up:reconnect","state_seq":24,"addr":"172.21.15.40:6833/2146240383","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2146240383},{"type":"v1","addr":"172.21.15.40:6833","nonce":2146240383}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:47:17.541 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:17.538+0000 7f6271b77700 1 -- 172.21.15.42:0/1886012750 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f624c03ac60 msgr2=0x7f624c03d170 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:47:17.541 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:17.538+0000 7f6271b77700 1 --2- 172.21.15.42:0/1886012750 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f624c03ac60 0x7f624c03d170 crc :-1 s=READY pgs=635 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:47:17.541 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:17.538+0000 7f6271b77700 1 -- 172.21.15.42:0/1886012750 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f626c07d470 msgr2=0x7f626c08f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:47:17.542 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:17.538+0000 7f6271b77700 1 --2- 172.21.15.42:0/1886012750 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f626c07d470 0x7f626c08f370 secure :-1 s=READY pgs=1474 cs=0 l=1 rx=0x7f626c0a7300 tx=0x7f62640106e0).stop 2019-12-05T05:47:17.542 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:17.538+0000 7f6271b77700 1 -- 172.21.15.42:0/1886012750 shutdown_connections 2019-12-05T05:47:17.542 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:17.538+0000 7f6271b77700 1 --2- 172.21.15.42:0/1886012750 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f624c03ac60 0x7f624c03d170 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:47:17.542 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:17.538+0000 7f6271b77700 1 --2- 172.21.15.42:0/1886012750 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f626c07d470 0x7f626c08f370 unknown :-1 s=CLOSED pgs=1474 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:47:17.542 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:17.538+0000 7f6271b77700 1 -- 172.21.15.42:0/1886012750 shutdown_connections 2019-12-05T05:47:17.542 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:17.538+0000 7f6271b77700 1 -- 172.21.15.42:0/1886012750 wait complete. 2019-12-05T05:47:17.542 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:17.538+0000 7f6271b77700 1 -- 172.21.15.42:0/1886012750 >> 172.21.15.42:0/1886012750 conn(0x7f626c07e5a0 msgr2=0x7f626c059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:47:17.551 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2019-12-05T05:47:19.552 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:47:19.660 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:19.650+0000 7fd7f9979700 1 Processor -- start 2019-12-05T05:47:19.660 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:19.654+0000 7fd7f9979700 1 -- start start 2019-12-05T05:47:19.661 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:19.654+0000 7fd7f9979700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd7f40a7740 0x7fd7f408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:47:19.661 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:19.654+0000 7fd7f9979700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd7f409fae0 0x7fd7f407eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:47:19.661 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:19.654+0000 7fd7f9979700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7f40a0710 con 0x7fd7f407e720 2019-12-05T05:47:19.661 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:19.654+0000 7fd7f9979700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7f40a0b00 con 0x7fd7f409fae0 2019-12-05T05:47:19.661 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:19.654+0000 7fd7f9979700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7f408e5b0 con 0x7fd7f40a7740 2019-12-05T05:47:19.662 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:19.654+0000 7fd7f37fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd7f40a7740 0x7fd7f408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:47:19.662 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:19.654+0000 7fd7f37fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd7f40a7740 0x7fd7f408df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:41258/0 (socket says 172.21.15.42:41258) 2019-12-05T05:47:19.662 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:19.654+0000 7fd7f37fe700 1 -- 172.21.15.42:0/690693012 learned_addr learned my addr 172.21.15.42:0/690693012 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:47:19.662 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:19.654+0000 7fd7f3fff700 1 --2- 172.21.15.42:0/690693012 >> v2:172.21.15.40:3300/0 conn(0x7fd7f409fae0 0x7fd7f407eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:47:19.662 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:19.654+0000 7fd7f37fe700 1 -- 172.21.15.42:0/690693012 >> v2:172.21.15.40:3300/0 conn(0x7fd7f409fae0 msgr2=0x7fd7f407eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:47:19.663 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:19.654+0000 7fd7f37fe700 1 --2- 172.21.15.42:0/690693012 >> v2:172.21.15.40:3300/0 conn(0x7fd7f409fae0 0x7fd7f407eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:47:19.663 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:19.654+0000 7fd7f37fe700 1 -- 172.21.15.42:0/690693012 >> v1:172.21.15.42:6789/0 conn(0x7fd7f407e720 legacy=0x7fd7f409f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:47:19.663 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:19.654+0000 7fd7f27fc700 1 -- 172.21.15.42:0/690693012 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3616234308 0 0) 0x7fd7f40a0710 con 0x7fd7f407e720 2019-12-05T05:47:19.664 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:19.654+0000 7fd7f37fe700 1 -- 172.21.15.42:0/690693012 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7f407f0a0 con 0x7fd7f40a7740 2019-12-05T05:47:19.664 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:19.654+0000 7fd7f37fe700 1 --2- 172.21.15.42:0/690693012 >> v2:172.21.15.42:3300/0 conn(0x7fd7f40a7740 0x7fd7f408df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fd7e4002f30 tx=0x7fd7e400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:19.664 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:19.654+0000 7fd7f37fe700 1 --2- 172.21.15.42:0/690693012 >> v2:172.21.15.42:3300/0 conn(0x7fd7f40a7740 0x7fd7f408df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fd7e4002f30 tx=0x7fd7e400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:19.664 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:19.654+0000 7fd7f37fe700 1 --2- 172.21.15.42:0/690693012 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd7f40a7740 0x7fd7f408df80 secure :-1 s=READY pgs=1476 cs=0 l=1 rx=0x7fd7e4002f30 tx=0x7fd7e400fd30).ready entity=mon.1 client_cookie=e55dcd514d324ed server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:47:19.664 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:19.654+0000 7fd7f37fe700 1 --2- 172.21.15.42:0/690693012 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd7f40a7740 0x7fd7f408df80 secure :-1 s=THROTTLE_DONE pgs=1476 cs=0 l=1 rx=0x7fd7e4002f30 tx=0x7fd7e400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:19.664 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:19.654+0000 7fd7f37fe700 1 --2- 172.21.15.42:0/690693012 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd7f40a7740 0x7fd7f408df80 secure :-1 s=THROTTLE_DONE pgs=1476 cs=0 l=1 rx=0x7fd7e4002f30 tx=0x7fd7e400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:19.665 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:19.654+0000 7fd7f27fc700 1 -- 172.21.15.42:0/690693012 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd7e4008c90 con 0x7fd7f40a7740 2019-12-05T05:47:19.665 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:19.654+0000 7fd7f37fe700 1 --2- 172.21.15.42:0/690693012 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd7f40a7740 0x7fd7f408df80 secure :-1 s=THROTTLE_DONE pgs=1476 cs=0 l=1 rx=0x7fd7e4002f30 tx=0x7fd7e400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:19.665 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:19.654+0000 7fd7f27fc700 1 -- 172.21.15.42:0/690693012 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7e40088b0 con 0x7fd7f40a7740 2019-12-05T05:47:19.665 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:19.654+0000 7fd7f27fc700 1 -- 172.21.15.42:0/690693012 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd7e4011ad0 con 0x7fd7f40a7740 2019-12-05T05:47:19.665 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:19.654+0000 7fd7f9979700 1 -- 172.21.15.42:0/690693012 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd7f40a7740 msgr2=0x7fd7f408df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:47:19.666 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:19.654+0000 7fd7f9979700 1 --2- 172.21.15.42:0/690693012 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd7f40a7740 0x7fd7f408df80 secure :-1 s=READY pgs=1476 cs=0 l=1 rx=0x7fd7e4002f30 tx=0x7fd7e400fd30).stop 2019-12-05T05:47:19.666 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:19.658+0000 7fd7f9979700 1 -- 172.21.15.42:0/690693012 shutdown_connections 2019-12-05T05:47:19.666 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:19.658+0000 7fd7f9979700 1 --2- 172.21.15.42:0/690693012 >> v2:172.21.15.40:3300/0 conn(0x7fd7f409fae0 0x7fd7f407eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:47:19.666 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:19.658+0000 7fd7f9979700 1 --2- 172.21.15.42:0/690693012 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd7f40a7740 0x7fd7f408df80 unknown :-1 s=CLOSED pgs=1476 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:47:19.666 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:19.658+0000 7fd7f9979700 1 -- 172.21.15.42:0/690693012 shutdown_connections 2019-12-05T05:47:19.666 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:19.658+0000 7fd7f9979700 1 -- 172.21.15.42:0/690693012 wait complete. 2019-12-05T05:47:19.667 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:19.658+0000 7fd7f9979700 1 -- 172.21.15.42:0/690693012 >> 172.21.15.42:0/690693012 conn(0x7fd7f40aabe0 msgr2=0x7fd7f40a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:47:19.671 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:19.658+0000 7fd7f9979700 1 Processor -- start 2019-12-05T05:47:19.671 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:19.658+0000 7fd7f9979700 1 -- start start 2019-12-05T05:47:19.671 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:19.658+0000 7fd7f9979700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd7f4133420 0x7fd7f4116360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:47:19.672 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:19.658+0000 7fd7f9979700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7f4119f00 con 0x7fd7f40742e0 2019-12-05T05:47:19.672 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:19.658+0000 7fd7f9979700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7f4119830 con 0x7fd7f41213a0 2019-12-05T05:47:19.672 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:19.658+0000 7fd7f9979700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7f4119c60 con 0x7fd7f4133420 2019-12-05T05:47:19.672 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:19.658+0000 7fd7f2ffd700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fd7f41213a0 0x7fd7f4118630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:41306/0 (socket says 172.21.15.42:41306) 2019-12-05T05:47:19.672 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:19.658+0000 7fd7f2ffd700 1 -- 172.21.15.42:0/2661363733 learned_addr learned my addr 172.21.15.42:0/2661363733 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:47:19.673 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:19.662+0000 7fd7f37fe700 1 --2- 172.21.15.42:0/2661363733 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd7f4133420 0x7fd7f4116360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:47:19.673 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:19.662+0000 7fd7ebfff700 1 -- 172.21.15.42:0/2661363733 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 619612754 0 0) 0x7fd7f4119830 con 0x7fd7f41213a0 2019-12-05T05:47:19.673 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:19.662+0000 7fd7ebfff700 1 -- 172.21.15.42:0/2661363733 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7d80030b0 con 0x7fd7f41213a0 2019-12-05T05:47:19.673 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:19.662+0000 7fd7ebfff700 1 -- 172.21.15.42:0/2661363733 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 112953509 0 0) 0x7fd7f4119f00 con 0x7fd7f40742e0 2019-12-05T05:47:19.673 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:19.662+0000 7fd7ebfff700 1 -- 172.21.15.42:0/2661363733 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7f4119830 con 0x7fd7f40742e0 2019-12-05T05:47:19.673 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:19.662+0000 7fd7f37fe700 1 -- 172.21.15.42:0/2661363733 >> v1:172.21.15.42:6789/0 conn(0x7fd7f41213a0 legacy=0x7fd7f4118630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:47:19.674 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:19.662+0000 7fd7f37fe700 1 -- 172.21.15.42:0/2661363733 >> v1:172.21.15.40:6789/0 conn(0x7fd7f40742e0 legacy=0x7fd7f4076720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:47:19.674 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:19.662+0000 7fd7f37fe700 1 -- 172.21.15.42:0/2661363733 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7f41217d0 con 0x7fd7f4133420 2019-12-05T05:47:19.674 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:19.662+0000 7fd7f37fe700 1 --2- 172.21.15.42:0/2661363733 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd7f4133420 0x7fd7f4116360 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fd7e4002f30 tx=0x7fd7e4008ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:19.674 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:19.662+0000 7fd7f37fe700 1 --2- 172.21.15.42:0/2661363733 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd7f4133420 0x7fd7f4116360 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fd7e4002f30 tx=0x7fd7e4008ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:19.674 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:19.662+0000 7fd7f37fe700 1 --2- 172.21.15.42:0/2661363733 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd7f4133420 0x7fd7f4116360 secure :-1 s=READY pgs=332 cs=0 l=1 rx=0x7fd7e4002f30 tx=0x7fd7e4008ae0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:47:19.674 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:19.662+0000 7fd7f37fe700 1 --2- 172.21.15.42:0/2661363733 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd7f4133420 0x7fd7f4116360 secure :-1 s=THROTTLE_DONE pgs=332 cs=0 l=1 rx=0x7fd7e4002f30 tx=0x7fd7e4008ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:19.675 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:19.662+0000 7fd7ebfff700 1 -- 172.21.15.42:0/2661363733 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd7e400f280 con 0x7fd7f4133420 2019-12-05T05:47:19.675 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:19.662+0000 7fd7f37fe700 1 --2- 172.21.15.42:0/2661363733 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd7f4133420 0x7fd7f4116360 secure :-1 s=THROTTLE_DONE pgs=332 cs=0 l=1 rx=0x7fd7e4002f30 tx=0x7fd7e4008ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:19.675 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:19.662+0000 7fd7f37fe700 1 --2- 172.21.15.42:0/2661363733 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd7f4133420 0x7fd7f4116360 secure :-1 s=THROTTLE_DONE pgs=332 cs=0 l=1 rx=0x7fd7e4002f30 tx=0x7fd7e4008ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:19.675 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:19.662+0000 7fd7ebfff700 1 -- 172.21.15.42:0/2661363733 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7e40116f0 con 0x7fd7f4133420 2019-12-05T05:47:19.675 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:19.662+0000 7fd7f9979700 1 -- 172.21.15.42:0/2661363733 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7f4077f30 con 0x7fd7f4133420 2019-12-05T05:47:19.676 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:19.662+0000 7fd7ebfff700 1 -- 172.21.15.42:0/2661363733 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd7e4005540 con 0x7fd7f4133420 2019-12-05T05:47:19.676 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:19.662+0000 7fd7f9979700 1 -- 172.21.15.42:0/2661363733 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7f40784b0 con 0x7fd7f4133420 2019-12-05T05:47:19.676 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:19.662+0000 7fd7f37fe700 1 --2- 172.21.15.42:0/2661363733 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd7f4133420 0x7fd7f4116360 secure :-1 s=THROTTLE_DONE pgs=332 cs=0 l=1 rx=0x7fd7e4002f30 tx=0x7fd7e4008ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:19.676 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:19.666+0000 7fd7ebfff700 1 -- 172.21.15.42:0/2661363733 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fd7e4005740 con 0x7fd7f4133420 2019-12-05T05:47:19.676 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:19.666+0000 7fd7f37fe700 1 --2- 172.21.15.42:0/2661363733 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd7f4133420 0x7fd7f4116360 secure :-1 s=THROTTLE_DONE pgs=332 cs=0 l=1 rx=0x7fd7e4002f30 tx=0x7fd7e4008ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:19.677 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:19.666+0000 7fd7ebfff700 1 --2- 172.21.15.42:0/2661363733 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd7d803b4b0 0x7fd7d803d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:47:19.677 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:19.666+0000 7fd7ebfff700 1 -- 172.21.15.42:0/2661363733 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 8611+0+0 (secure 0 0 0) 0x7fd7e404b850 con 0x7fd7f4133420 2019-12-05T05:47:19.677 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:19.666+0000 7fd7f2ffd700 1 --2- 172.21.15.42:0/2661363733 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd7d803b4b0 0x7fd7d803d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:47:19.677 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:19.666+0000 7fd7f2ffd700 1 --2- 172.21.15.42:0/2661363733 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd7d803b4b0 0x7fd7d803d9c0 crc :-1 s=READY pgs=636 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:47:19.677 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:19.666+0000 7fd7f9979700 1 -- 172.21.15.42:0/2661363733 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7f4119a60 con 0x7fd7f4133420 2019-12-05T05:47:19.680 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:19.678+0000 7fd7f37fe700 1 --2- 172.21.15.42:0/2661363733 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd7f4133420 0x7fd7f4116360 secure :-1 s=THROTTLE_DONE pgs=332 cs=0 l=1 rx=0x7fd7e4002f30 tx=0x7fd7e4008ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:19.680 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:19.678+0000 7fd7ebfff700 1 -- 172.21.15.42:0/2661363733 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fd7e40328d0 con 0x7fd7f4133420 2019-12-05T05:47:19.849 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:19.846+0000 7fd7f9979700 1 -- 172.21.15.42:0/2661363733 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd7f405d110 con 0x7fd7f4133420 2019-12-05T05:47:19.850 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:19.846+0000 7fd7f37fe700 1 --2- 172.21.15.42:0/2661363733 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd7f4133420 0x7fd7f4116360 secure :-1 s=THROTTLE_DONE pgs=332 cs=0 l=1 rx=0x7fd7e4002f30 tx=0x7fd7e4008ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:19.850 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:19.846+0000 7fd7ebfff700 1 -- 172.21.15.42:0/2661363733 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 182 v182) v1 ==== 95+0+1775 (secure 0 0 0) 0x7fd7e4032b30 con 0x7fd7f4133420 2019-12-05T05:47:19.852 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 182 2019-12-05T05:47:19.852 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:47:19.852 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":182,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":182,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:47:19.499092+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6140},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6140":{"gid":6140,"name":"b","rank":0,"incarnation":179,"state":"up:active","state_seq":26,"addr":"172.21.15.40:6833/2146240383","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2146240383},{"type":"v1","addr":"172.21.15.40:6833","nonce":2146240383}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:47:19.855 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:19.850+0000 7fd7f9979700 1 -- 172.21.15.42:0/2661363733 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd7d803b4b0 msgr2=0x7fd7d803d9c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:47:19.855 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:19.850+0000 7fd7f9979700 1 --2- 172.21.15.42:0/2661363733 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd7d803b4b0 0x7fd7d803d9c0 crc :-1 s=READY pgs=636 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:47:19.855 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:19.850+0000 7fd7f9979700 1 -- 172.21.15.42:0/2661363733 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd7f4133420 msgr2=0x7fd7f4116360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:47:19.855 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:19.850+0000 7fd7f9979700 1 --2- 172.21.15.42:0/2661363733 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd7f4133420 0x7fd7f4116360 secure :-1 s=READY pgs=332 cs=0 l=1 rx=0x7fd7e4002f30 tx=0x7fd7e4008ae0).stop 2019-12-05T05:47:19.856 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:19.850+0000 7fd7f9979700 1 -- 172.21.15.42:0/2661363733 shutdown_connections 2019-12-05T05:47:19.856 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:19.850+0000 7fd7f9979700 1 --2- 172.21.15.42:0/2661363733 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd7d803b4b0 0x7fd7d803d9c0 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:47:19.856 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:19.850+0000 7fd7f9979700 1 --2- 172.21.15.42:0/2661363733 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd7f4133420 0x7fd7f4116360 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:47:19.856 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:19.850+0000 7fd7f9979700 1 -- 172.21.15.42:0/2661363733 shutdown_connections 2019-12-05T05:47:19.856 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:19.850+0000 7fd7f9979700 1 -- 172.21.15.42:0/2661363733 wait complete. 2019-12-05T05:47:19.856 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:19.850+0000 7fd7f9979700 1 -- 172.21.15.42:0/2661363733 >> 172.21.15.42:0/2661363733 conn(0x7fd7f40a7740 msgr2=0x7fd7f40a9640 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:47:19.866 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b has gained rank=0, replacing gid=6077 2019-12-05T05:47:19.866 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 4 secs before reviving mds.a 2019-12-05T05:47:23.868 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.a 2019-12-05T05:47:23.868 INFO:tasks.ceph.mds.a:Restarting daemon 2019-12-05T05:47:23.868 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2019-12-05T05:47:23.872 INFO:tasks.ceph.mds.a:Started 2019-12-05T05:47:23.872 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:47:23.934 INFO:tasks.ceph.mds.a.smithi042.stdout:starting mds.a at 2019-12-05T05:47:23.967 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:23.958+0000 7f418ae0b700 1 Processor -- start 2019-12-05T05:47:23.967 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:23.958+0000 7f418ae0b700 1 -- start start 2019-12-05T05:47:23.967 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:23.958+0000 7f418ae0b700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f41840a7740 0x7f418408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:47:23.967 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:23.958+0000 7f418ae0b700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f418407e720 0x7f418407eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:47:23.968 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:23.958+0000 7f418ae0b700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41840a0710 con 0x7f418409f2e0 2019-12-05T05:47:23.968 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:23.958+0000 7f418ae0b700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41840a0b00 con 0x7f418407e720 2019-12-05T05:47:23.968 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:23.958+0000 7f418ae0b700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f418408e5b0 con 0x7f41840a7740 2019-12-05T05:47:23.968 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:23.958+0000 7f41893a8700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f41840a7740 0x7f418408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:47:23.968 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:23.958+0000 7f4189ba9700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f418409f2e0 0x7f418409f710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:41348/0 (socket says 172.21.15.42:41348) 2019-12-05T05:47:23.969 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:23.958+0000 7f4189ba9700 1 -- 172.21.15.42:0/2290942942 learned_addr learned my addr 172.21.15.42:0/2290942942 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:47:23.969 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:23.958+0000 7f4188ba7700 1 --2- 172.21.15.42:0/2290942942 >> v2:172.21.15.40:3300/0 conn(0x7f418407e720 0x7f418407eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:47:23.969 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:23.958+0000 7f417bfff700 1 -- 172.21.15.42:0/2290942942 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 233369424 0 0) 0x7f41840a0710 con 0x7f418409f2e0 2019-12-05T05:47:23.969 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:23.958+0000 7f417bfff700 1 -- 172.21.15.42:0/2290942942 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41700030b0 con 0x7f418409f2e0 2019-12-05T05:47:23.969 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:23.958+0000 7f4188ba7700 1 -- 172.21.15.42:0/2290942942 >> v2:172.21.15.42:3300/0 conn(0x7f41840a7740 msgr2=0x7f418408df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:47:23.969 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:23.958+0000 7f4188ba7700 1 --2- 172.21.15.42:0/2290942942 >> v2:172.21.15.42:3300/0 conn(0x7f41840a7740 0x7f418408df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:47:23.970 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:23.958+0000 7f4188ba7700 1 -- 172.21.15.42:0/2290942942 >> v1:172.21.15.42:6789/0 conn(0x7f418409f2e0 legacy=0x7f418409f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:47:23.970 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:23.958+0000 7f4188ba7700 1 -- 172.21.15.42:0/2290942942 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f418407f0a0 con 0x7f418407e720 2019-12-05T05:47:23.970 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:23.958+0000 7f4188ba7700 1 --2- 172.21.15.42:0/2290942942 >> v2:172.21.15.40:3300/0 conn(0x7f418407e720 0x7f418407eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f4174002f30 tx=0x7f417400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:23.970 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:23.958+0000 7f4188ba7700 1 --2- 172.21.15.42:0/2290942942 >> v2:172.21.15.40:3300/0 conn(0x7f418407e720 0x7f418407eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f4174002f30 tx=0x7f417400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:23.970 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:23.958+0000 7f4188ba7700 1 --2- 172.21.15.42:0/2290942942 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f418407e720 0x7f418407eb50 secure :-1 s=READY pgs=1151 cs=0 l=1 rx=0x7f4174002f30 tx=0x7f417400fc70).ready entity=mon.0 client_cookie=f06ce312b7afe399 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:47:23.970 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:23.958+0000 7f4188ba7700 1 --2- 172.21.15.42:0/2290942942 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f418407e720 0x7f418407eb50 secure :-1 s=THROTTLE_DONE pgs=1151 cs=0 l=1 rx=0x7f4174002f30 tx=0x7f417400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:23.971 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:23.958+0000 7f417bfff700 1 -- 172.21.15.42:0/2290942942 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f41740112b0 con 0x7f418407e720 2019-12-05T05:47:23.971 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:23.958+0000 7f4188ba7700 1 --2- 172.21.15.42:0/2290942942 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f418407e720 0x7f418407eb50 secure :-1 s=THROTTLE_DONE pgs=1151 cs=0 l=1 rx=0x7f4174002f30 tx=0x7f417400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:23.971 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:23.958+0000 7f417bfff700 1 -- 172.21.15.42:0/2290942942 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4174008ad0 con 0x7f418407e720 2019-12-05T05:47:23.971 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:23.958+0000 7f4188ba7700 1 --2- 172.21.15.42:0/2290942942 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f418407e720 0x7f418407eb50 secure :-1 s=THROTTLE_DONE pgs=1151 cs=0 l=1 rx=0x7f4174002f30 tx=0x7f417400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:23.971 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:23.958+0000 7f417bfff700 1 -- 172.21.15.42:0/2290942942 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4174011890 con 0x7f418407e720 2019-12-05T05:47:23.972 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:23.958+0000 7f418ae0b700 1 -- 172.21.15.42:0/2290942942 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f418407e720 msgr2=0x7f418407eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:47:23.972 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:23.958+0000 7f418ae0b700 1 --2- 172.21.15.42:0/2290942942 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f418407e720 0x7f418407eb50 secure :-1 s=READY pgs=1151 cs=0 l=1 rx=0x7f4174002f30 tx=0x7f417400fc70).stop 2019-12-05T05:47:23.972 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:23.962+0000 7f418ae0b700 1 -- 172.21.15.42:0/2290942942 shutdown_connections 2019-12-05T05:47:23.972 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:23.962+0000 7f418ae0b700 1 --2- 172.21.15.42:0/2290942942 >> v2:172.21.15.42:3300/0 conn(0x7f41840a7740 0x7f418408df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:47:23.972 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:23.962+0000 7f418ae0b700 1 --2- 172.21.15.42:0/2290942942 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f418407e720 0x7f418407eb50 unknown :-1 s=CLOSED pgs=1151 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:47:23.972 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:23.962+0000 7f418ae0b700 1 -- 172.21.15.42:0/2290942942 shutdown_connections 2019-12-05T05:47:23.973 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:23.962+0000 7f418ae0b700 1 -- 172.21.15.42:0/2290942942 wait complete. 2019-12-05T05:47:23.973 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:23.962+0000 7f418ae0b700 1 -- 172.21.15.42:0/2290942942 >> 172.21.15.42:0/2290942942 conn(0x7f41840aabe0 msgr2=0x7f41840a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:47:23.973 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:23.962+0000 7f418ae0b700 1 Processor -- start 2019-12-05T05:47:23.973 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:23.962+0000 7f418ae0b700 1 -- start start 2019-12-05T05:47:23.976 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:23.966+0000 7f418ae0b700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f4184137b00 0x7f418411a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:47:23.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:23.966+0000 7f418ae0b700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f41841182b0 0x7f41840a53a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:47:23.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:23.966+0000 7f418ae0b700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4184142680 con 0x7f4184144cd0 2019-12-05T05:47:23.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:23.966+0000 7f418ae0b700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f418411aca0 con 0x7f4184137b00 2019-12-05T05:47:23.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:23.966+0000 7f418ae0b700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41840a6d50 con 0x7f41841182b0 2019-12-05T05:47:23.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:23.966+0000 7f4188ba7700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f41841182b0 0x7f41840a53a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:47:23.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:23.966+0000 7f4189ba9700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f4184144cd0 0x7f41841193b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:41354/0 (socket says 172.21.15.42:41354) 2019-12-05T05:47:23.978 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:23.966+0000 7f4189ba9700 1 -- 172.21.15.42:0/3288033830 learned_addr learned my addr 172.21.15.42:0/3288033830 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:47:23.978 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:23.966+0000 7f41893a8700 1 --2- 172.21.15.42:0/3288033830 >> v2:172.21.15.40:3300/0 conn(0x7f4184137b00 0x7f418411a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:47:23.978 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:23.966+0000 7f4188ba7700 1 -- 172.21.15.42:0/3288033830 >> v2:172.21.15.40:3300/0 conn(0x7f4184137b00 msgr2=0x7f418411a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:47:23.978 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:23.966+0000 7f4188ba7700 1 --2- 172.21.15.42:0/3288033830 >> v2:172.21.15.40:3300/0 conn(0x7f4184137b00 0x7f418411a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:47:23.978 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:23.966+0000 7f4188ba7700 1 -- 172.21.15.42:0/3288033830 >> v1:172.21.15.42:6789/0 conn(0x7f4184144cd0 legacy=0x7f41841193b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:47:23.978 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:23.966+0000 7f4188ba7700 1 -- 172.21.15.42:0/3288033830 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41840a6a10 con 0x7f41841182b0 2019-12-05T05:47:23.979 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:23.966+0000 7f4188ba7700 1 --2- 172.21.15.42:0/3288033830 >> v2:172.21.15.42:3300/0 conn(0x7f41841182b0 0x7f41840a53a0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f4174002f30 tx=0x7f4174011da0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:23.979 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:23.966+0000 7f4188ba7700 1 --2- 172.21.15.42:0/3288033830 >> v2:172.21.15.42:3300/0 conn(0x7f41841182b0 0x7f41840a53a0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f4174002f30 tx=0x7f4174011da0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:23.979 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:23.966+0000 7f4188ba7700 1 --2- 172.21.15.42:0/3288033830 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f41841182b0 0x7f41840a53a0 secure :-1 s=READY pgs=1482 cs=0 l=1 rx=0x7f4174002f30 tx=0x7f4174011da0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:47:23.979 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:23.966+0000 7f4188ba7700 1 --2- 172.21.15.42:0/3288033830 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f41841182b0 0x7f41840a53a0 secure :-1 s=THROTTLE_DONE pgs=1482 cs=0 l=1 rx=0x7f4174002f30 tx=0x7f4174011da0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:23.979 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:23.966+0000 7f4179ffb700 1 -- 172.21.15.42:0/3288033830 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4174007150 con 0x7f41841182b0 2019-12-05T05:47:23.979 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:23.966+0000 7f4188ba7700 1 --2- 172.21.15.42:0/3288033830 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f41841182b0 0x7f41840a53a0 secure :-1 s=THROTTLE_DONE pgs=1482 cs=0 l=1 rx=0x7f4174002f30 tx=0x7f4174011da0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:23.980 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:23.966+0000 7f4188ba7700 1 --2- 172.21.15.42:0/3288033830 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f41841182b0 0x7f41840a53a0 secure :-1 s=THROTTLE_DONE pgs=1482 cs=0 l=1 rx=0x7f4174002f30 tx=0x7f4174011da0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:23.980 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:23.966+0000 7f4179ffb700 1 -- 172.21.15.42:0/3288033830 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41740114b0 con 0x7f41841182b0 2019-12-05T05:47:23.980 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:23.966+0000 7f418ae0b700 1 -- 172.21.15.42:0/3288033830 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41840a58f0 con 0x7f41841182b0 2019-12-05T05:47:23.980 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:23.966+0000 7f4179ffb700 1 -- 172.21.15.42:0/3288033830 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4174008cc0 con 0x7f41841182b0 2019-12-05T05:47:23.980 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:23.966+0000 7f418ae0b700 1 -- 172.21.15.42:0/3288033830 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4184145100 con 0x7f41841182b0 2019-12-05T05:47:23.980 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:23.966+0000 7f4188ba7700 1 --2- 172.21.15.42:0/3288033830 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f41841182b0 0x7f41840a53a0 secure :-1 s=THROTTLE_DONE pgs=1482 cs=0 l=1 rx=0x7f4174002f30 tx=0x7f4174011da0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:23.981 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:23.966+0000 7f4179ffb700 1 -- 172.21.15.42:0/3288033830 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f4174005540 con 0x7f41841182b0 2019-12-05T05:47:23.981 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:23.966+0000 7f4188ba7700 1 --2- 172.21.15.42:0/3288033830 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f41841182b0 0x7f41840a53a0 secure :-1 s=THROTTLE_DONE pgs=1482 cs=0 l=1 rx=0x7f4174002f30 tx=0x7f4174011da0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:23.981 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:23.966+0000 7f4179ffb700 1 --2- 172.21.15.42:0/3288033830 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f416403ada0 0x7f416403d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:47:23.981 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:23.966+0000 7f4179ffb700 1 -- 172.21.15.42:0/3288033830 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 8611+0+0 (secure 0 0 0) 0x7f417404b7a0 con 0x7f41841182b0 2019-12-05T05:47:23.981 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:23.966+0000 7f41893a8700 1 --2- 172.21.15.42:0/3288033830 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f416403ada0 0x7f416403d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:47:23.981 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:23.966+0000 7f41893a8700 1 --2- 172.21.15.42:0/3288033830 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f416403ada0 0x7f416403d2b0 crc :-1 s=READY pgs=637 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:47:23.982 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:23.974+0000 7f418ae0b700 1 -- 172.21.15.42:0/3288033830 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f418405d110 con 0x7f41841182b0 2019-12-05T05:47:23.983 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:23.982+0000 7f4188ba7700 1 --2- 172.21.15.42:0/3288033830 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f41841182b0 0x7f41840a53a0 secure :-1 s=THROTTLE_DONE pgs=1482 cs=0 l=1 rx=0x7f4174002f30 tx=0x7f4174011da0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:23.983 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:23.982+0000 7f4179ffb700 1 -- 172.21.15.42:0/3288033830 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f4174032820 con 0x7f41841182b0 2019-12-05T05:47:24.158 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:24.154+0000 7f418ae0b700 1 -- 172.21.15.42:0/3288033830 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f41840a8cb0 con 0x7f41841182b0 2019-12-05T05:47:24.160 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:24.158+0000 7f4188ba7700 1 --2- 172.21.15.42:0/3288033830 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f41841182b0 0x7f41840a53a0 secure :-1 s=THROTTLE_DONE pgs=1482 cs=0 l=1 rx=0x7f4174002f30 tx=0x7f4174011da0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:24.160 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:24.158+0000 7f4179ffb700 1 -- 172.21.15.42:0/3288033830 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 182 v182) v1 ==== 95+0+1775 (secure 0 0 0) 0x7f4174032a80 con 0x7f41841182b0 2019-12-05T05:47:24.160 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 182 2019-12-05T05:47:24.160 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:47:24.160 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":182,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":182,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:47:19.499092+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6140},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6140":{"gid":6140,"name":"b","rank":0,"incarnation":179,"state":"up:active","state_seq":26,"addr":"172.21.15.40:6833/2146240383","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2146240383},{"type":"v1","addr":"172.21.15.40:6833","nonce":2146240383}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:47:24.162 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:24.158+0000 7f418ae0b700 1 -- 172.21.15.42:0/3288033830 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f416403ada0 msgr2=0x7f416403d2b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:47:24.162 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:24.158+0000 7f418ae0b700 1 --2- 172.21.15.42:0/3288033830 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f416403ada0 0x7f416403d2b0 crc :-1 s=READY pgs=637 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:47:24.163 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:24.158+0000 7f418ae0b700 1 -- 172.21.15.42:0/3288033830 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f41841182b0 msgr2=0x7f41840a53a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:47:24.163 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:24.158+0000 7f418ae0b700 1 --2- 172.21.15.42:0/3288033830 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f41841182b0 0x7f41840a53a0 secure :-1 s=READY pgs=1482 cs=0 l=1 rx=0x7f4174002f30 tx=0x7f4174011da0).stop 2019-12-05T05:47:24.163 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:24.158+0000 7f418ae0b700 1 -- 172.21.15.42:0/3288033830 shutdown_connections 2019-12-05T05:47:24.163 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:24.158+0000 7f418ae0b700 1 --2- 172.21.15.42:0/3288033830 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f416403ada0 0x7f416403d2b0 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:47:24.163 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:24.158+0000 7f418ae0b700 1 --2- 172.21.15.42:0/3288033830 >> v2:172.21.15.40:3300/0 conn(0x7f4184137b00 0x7f418411a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:47:24.163 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:24.158+0000 7f418ae0b700 1 --2- 172.21.15.42:0/3288033830 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f41841182b0 0x7f41840a53a0 unknown :-1 s=CLOSED pgs=1482 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:47:24.164 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:24.158+0000 7f418ae0b700 1 -- 172.21.15.42:0/3288033830 shutdown_connections 2019-12-05T05:47:24.164 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:24.158+0000 7f418ae0b700 1 -- 172.21.15.42:0/3288033830 wait complete. 2019-12-05T05:47:24.164 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:24.158+0000 7f418ae0b700 1 -- 172.21.15.42:0/3288033830 >> 172.21.15.42:0/3288033830 conn(0x7f41840aabe0 msgr2=0x7f418407e640 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:47:24.172 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2019-12-05T05:47:26.173 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:47:26.281 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:26.274+0000 7f2112360700 1 Processor -- start 2019-12-05T05:47:26.282 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:26.274+0000 7f2112360700 1 -- start start 2019-12-05T05:47:26.282 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:26.274+0000 7f2112360700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f210c08df80 0x7f210c09f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:47:26.282 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:26.274+0000 7f2112360700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f210c09f830 0x7f210c14efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:47:26.282 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:26.274+0000 7f2112360700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f210c09fc60 con 0x7f210c0a7740 2019-12-05T05:47:26.282 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:26.274+0000 7f2112360700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f210c0a0b00 con 0x7f210c08df80 2019-12-05T05:47:26.282 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:26.274+0000 7f2112360700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f210c08e5b0 con 0x7f210c09f830 2019-12-05T05:47:26.283 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:26.274+0000 7f21108fd700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f210c0a7740 0x7f210c07e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:41366/0 (socket says 172.21.15.42:41366) 2019-12-05T05:47:26.283 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:26.274+0000 7f21108fd700 1 -- 172.21.15.42:0/1060572217 learned_addr learned my addr 172.21.15.42:0/1060572217 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:47:26.283 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:26.274+0000 7f210bfff700 1 --2- 172.21.15.42:0/1060572217 >> v2:172.21.15.40:3300/0 conn(0x7f210c08df80 0x7f210c09f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:47:26.283 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:26.274+0000 7f21110fe700 1 --2- 172.21.15.42:0/1060572217 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f210c09f830 0x7f210c14efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:47:26.283 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:26.274+0000 7f210b7fe700 1 -- 172.21.15.42:0/1060572217 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 204351227 0 0) 0x7f210c09fc60 con 0x7f210c0a7740 2019-12-05T05:47:26.283 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:26.274+0000 7f210b7fe700 1 -- 172.21.15.42:0/1060572217 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20f80030b0 con 0x7f210c0a7740 2019-12-05T05:47:26.284 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:26.274+0000 7f21110fe700 1 -- 172.21.15.42:0/1060572217 >> v2:172.21.15.40:3300/0 conn(0x7f210c08df80 msgr2=0x7f210c09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:47:26.284 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:26.274+0000 7f21110fe700 1 --2- 172.21.15.42:0/1060572217 >> v2:172.21.15.40:3300/0 conn(0x7f210c08df80 0x7f210c09f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:47:26.284 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:26.274+0000 7f21110fe700 1 -- 172.21.15.42:0/1060572217 >> v1:172.21.15.42:6789/0 conn(0x7f210c0a7740 legacy=0x7f210c07e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:47:26.284 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:26.274+0000 7f21110fe700 1 -- 172.21.15.42:0/1060572217 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f210c0a0710 con 0x7f210c09f830 2019-12-05T05:47:26.288 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:26.274+0000 7f210bfff700 1 --2- 172.21.15.42:0/1060572217 >> v2:172.21.15.40:3300/0 conn(0x7f210c08df80 0x7f210c09f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:47:26.288 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:26.274+0000 7f21110fe700 1 --2- 172.21.15.42:0/1060572217 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f210c09f830 0x7f210c14efe0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f2104008b10 tx=0x7f210400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:26.288 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:26.274+0000 7f21110fe700 1 --2- 172.21.15.42:0/1060572217 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f210c09f830 0x7f210c14efe0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f2104008b10 tx=0x7f210400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:26.288 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:26.278+0000 7f21110fe700 1 --2- 172.21.15.42:0/1060572217 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f210c09f830 0x7f210c14efe0 secure :-1 s=READY pgs=333 cs=0 l=1 rx=0x7f2104008b10 tx=0x7f210400fff0).ready entity=mon.2 client_cookie=5725d2c20bc99211 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:47:26.288 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:26.278+0000 7f21110fe700 1 --2- 172.21.15.42:0/1060572217 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f210c09f830 0x7f210c14efe0 secure :-1 s=THROTTLE_DONE pgs=333 cs=0 l=1 rx=0x7f2104008b10 tx=0x7f210400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:26.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:26.278+0000 7f210b7fe700 1 -- 172.21.15.42:0/1060572217 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2104010900 con 0x7f210c09f830 2019-12-05T05:47:26.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:26.278+0000 7f21110fe700 1 --2- 172.21.15.42:0/1060572217 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f210c09f830 0x7f210c14efe0 secure :-1 s=THROTTLE_DONE pgs=333 cs=0 l=1 rx=0x7f2104008b10 tx=0x7f210400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:26.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:26.278+0000 7f210b7fe700 1 -- 172.21.15.42:0/1060572217 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2104008b50 con 0x7f210c09f830 2019-12-05T05:47:26.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:26.278+0000 7f21110fe700 1 --2- 172.21.15.42:0/1060572217 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f210c09f830 0x7f210c14efe0 secure :-1 s=THROTTLE_DONE pgs=333 cs=0 l=1 rx=0x7f2104008b10 tx=0x7f210400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:26.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:26.278+0000 7f210b7fe700 1 -- 172.21.15.42:0/1060572217 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f21040110c0 con 0x7f210c09f830 2019-12-05T05:47:26.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:26.278+0000 7f2112360700 1 -- 172.21.15.42:0/1060572217 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f210c09f830 msgr2=0x7f210c14efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:47:26.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:26.278+0000 7f2112360700 1 --2- 172.21.15.42:0/1060572217 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f210c09f830 0x7f210c14efe0 secure :-1 s=READY pgs=333 cs=0 l=1 rx=0x7f2104008b10 tx=0x7f210400fff0).stop 2019-12-05T05:47:26.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:26.278+0000 7f2112360700 1 -- 172.21.15.42:0/1060572217 shutdown_connections 2019-12-05T05:47:26.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:26.278+0000 7f2112360700 1 --2- 172.21.15.42:0/1060572217 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f210c09f830 0x7f210c14efe0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:47:26.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:26.278+0000 7f2112360700 1 --2- 172.21.15.42:0/1060572217 >> v2:172.21.15.40:3300/0 conn(0x7f210c08df80 0x7f210c09f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:47:26.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:26.278+0000 7f2112360700 1 -- 172.21.15.42:0/1060572217 shutdown_connections 2019-12-05T05:47:26.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:26.278+0000 7f2112360700 1 -- 172.21.15.42:0/1060572217 wait complete. 2019-12-05T05:47:26.291 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:26.278+0000 7f2112360700 1 -- 172.21.15.42:0/1060572217 >> 172.21.15.42:0/1060572217 conn(0x7f210c0aabe0 msgr2=0x7f210c0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:47:26.294 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:26.278+0000 7f2112360700 1 Processor -- start 2019-12-05T05:47:26.294 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:26.278+0000 7f2112360700 1 -- start start 2019-12-05T05:47:26.294 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:26.282+0000 7f2112360700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f210c130d30 0x7f210c131160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:47:26.294 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:26.282+0000 7f2112360700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f210c120c40 con 0x7f210c1203d0 2019-12-05T05:47:26.295 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:26.282+0000 7f2112360700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f210c07bfd0 con 0x7f210c07bba0 2019-12-05T05:47:26.295 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:26.282+0000 7f2112360700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f210c07dbe0 con 0x7f210c130d30 2019-12-05T05:47:26.295 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:26.282+0000 7f210bfff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f210c130d30 0x7f210c131160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:47:26.295 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:26.282+0000 7f210bfff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f210c130d30 0x7f210c131160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:41334/0 (socket says 172.21.15.42:41334) 2019-12-05T05:47:26.295 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:26.282+0000 7f210bfff700 1 -- 172.21.15.42:0/962391453 learned_addr learned my addr 172.21.15.42:0/962391453 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:47:26.295 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:26.282+0000 7f210bfff700 1 -- 172.21.15.42:0/962391453 >> v1:172.21.15.42:6789/0 conn(0x7f210c07bba0 legacy=0x7f210c137ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:47:26.295 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:26.282+0000 7f210bfff700 1 -- 172.21.15.42:0/962391453 >> v1:172.21.15.40:6789/0 conn(0x7f210c1203d0 legacy=0x7f210c121420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:47:26.296 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:26.282+0000 7f21097fa700 1 -- 172.21.15.42:0/962391453 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3760157996 0 0) 0x7f210c07bfd0 con 0x7f210c07bba0 2019-12-05T05:47:26.296 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:26.282+0000 7f210bfff700 1 -- 172.21.15.42:0/962391453 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f210c07c200 con 0x7f210c130d30 2019-12-05T05:47:26.296 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:26.282+0000 7f210bfff700 1 --2- 172.21.15.42:0/962391453 >> v2:172.21.15.42:3300/0 conn(0x7f210c130d30 0x7f210c131160 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f20f4001090 tx=0x7f20f4000ed0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:26.296 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:26.282+0000 7f210bfff700 1 --2- 172.21.15.42:0/962391453 >> v2:172.21.15.42:3300/0 conn(0x7f210c130d30 0x7f210c131160 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f20f4001090 tx=0x7f20f4000ed0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:26.296 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:26.282+0000 7f210bfff700 1 --2- 172.21.15.42:0/962391453 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f210c130d30 0x7f210c131160 secure :-1 s=READY pgs=1485 cs=0 l=1 rx=0x7f20f4001090 tx=0x7f20f4000ed0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:47:26.297 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:26.282+0000 7f210bfff700 1 --2- 172.21.15.42:0/962391453 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f210c130d30 0x7f210c131160 secure :-1 s=THROTTLE_DONE pgs=1485 cs=0 l=1 rx=0x7f20f4001090 tx=0x7f20f4000ed0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:26.297 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:26.282+0000 7f21097fa700 1 -- 172.21.15.42:0/962391453 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f20fc0058e0 con 0x7f210c130d30 2019-12-05T05:47:26.297 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:26.282+0000 7f210bfff700 1 --2- 172.21.15.42:0/962391453 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f210c130d30 0x7f210c131160 secure :-1 s=THROTTLE_DONE pgs=1485 cs=0 l=1 rx=0x7f20f4001090 tx=0x7f20f4000ed0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:26.297 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:26.282+0000 7f210bfff700 1 --2- 172.21.15.42:0/962391453 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f210c130d30 0x7f210c131160 secure :-1 s=THROTTLE_DONE pgs=1485 cs=0 l=1 rx=0x7f20f4001090 tx=0x7f20f4000ed0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:26.297 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:26.282+0000 7f21097fa700 1 -- 172.21.15.42:0/962391453 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20fc007160 con 0x7f210c130d30 2019-12-05T05:47:26.297 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:26.282+0000 7f2112360700 1 -- 172.21.15.42:0/962391453 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f210c1392d0 con 0x7f210c130d30 2019-12-05T05:47:26.298 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:26.282+0000 7f21097fa700 1 -- 172.21.15.42:0/962391453 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f20fc00e410 con 0x7f210c130d30 2019-12-05T05:47:26.298 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:26.282+0000 7f2112360700 1 -- 172.21.15.42:0/962391453 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f210c139820 con 0x7f210c130d30 2019-12-05T05:47:26.298 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:26.282+0000 7f210bfff700 1 --2- 172.21.15.42:0/962391453 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f210c130d30 0x7f210c131160 secure :-1 s=THROTTLE_DONE pgs=1485 cs=0 l=1 rx=0x7f20f4001090 tx=0x7f20f4000ed0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:26.298 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:26.286+0000 7f21097fa700 1 -- 172.21.15.42:0/962391453 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f20fc010180 con 0x7f210c130d30 2019-12-05T05:47:26.298 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:26.286+0000 7f210bfff700 1 --2- 172.21.15.42:0/962391453 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f210c130d30 0x7f210c131160 secure :-1 s=THROTTLE_DONE pgs=1485 cs=0 l=1 rx=0x7f20f4001090 tx=0x7f20f4000ed0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:26.298 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:26.286+0000 7f21097fa700 1 --2- 172.21.15.42:0/962391453 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f20f003ac10 0x7f20f003d120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:47:26.298 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:26.286+0000 7f21097fa700 1 -- 172.21.15.42:0/962391453 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 8611+0+0 (secure 0 0 0) 0x7f20fc04b170 con 0x7f210c130d30 2019-12-05T05:47:26.299 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:26.286+0000 7f21108fd700 1 --2- 172.21.15.42:0/962391453 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f20f003ac10 0x7f20f003d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:47:26.299 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:26.286+0000 7f21108fd700 1 --2- 172.21.15.42:0/962391453 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f20f003ac10 0x7f20f003d120 crc :-1 s=READY pgs=639 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:47:26.299 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:26.290+0000 7f2112360700 1 -- 172.21.15.42:0/962391453 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f210c05d110 con 0x7f210c130d30 2019-12-05T05:47:26.302 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:26.298+0000 7f210bfff700 1 --2- 172.21.15.42:0/962391453 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f210c130d30 0x7f210c131160 secure :-1 s=THROTTLE_DONE pgs=1485 cs=0 l=1 rx=0x7f20f4001090 tx=0x7f20f4000ed0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:26.302 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:26.298+0000 7f21097fa700 1 -- 172.21.15.42:0/962391453 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f20fc032450 con 0x7f210c130d30 2019-12-05T05:47:26.477 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:26.474+0000 7f2112360700 1 -- 172.21.15.42:0/962391453 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f210c05b8c0 con 0x7f210c130d30 2019-12-05T05:47:26.478 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:26.474+0000 7f210bfff700 1 --2- 172.21.15.42:0/962391453 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f210c130d30 0x7f210c131160 secure :-1 s=THROTTLE_DONE pgs=1485 cs=0 l=1 rx=0x7f20f4001090 tx=0x7f20f4000ed0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:47:26.479 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:26.474+0000 7f21097fa700 1 -- 172.21.15.42:0/962391453 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 183 v183) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f20fc033000 con 0x7f210c130d30 2019-12-05T05:47:26.479 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 183 2019-12-05T05:47:26.479 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:47:26.479 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":183,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":8188,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6835/2456922143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2456922143},{"type":"v1","addr":"172.21.15.42:6835","nonce":2456922143}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":183}],"filesystems":[{"mdsmap":{"epoch":182,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:47:19.499092+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6140},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6140":{"gid":6140,"name":"b","rank":0,"incarnation":179,"state":"up:active","state_seq":26,"addr":"172.21.15.40:6833/2146240383","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2146240383},{"type":"v1","addr":"172.21.15.40:6833","nonce":2146240383}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:47:26.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:26.478+0000 7f2112360700 1 -- 172.21.15.42:0/962391453 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f20f003ac10 msgr2=0x7f20f003d120 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:47:26.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:26.478+0000 7f2112360700 1 --2- 172.21.15.42:0/962391453 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f20f003ac10 0x7f20f003d120 crc :-1 s=READY pgs=639 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:47:26.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:26.478+0000 7f2112360700 1 -- 172.21.15.42:0/962391453 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f210c130d30 msgr2=0x7f210c131160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:47:26.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:26.478+0000 7f2112360700 1 --2- 172.21.15.42:0/962391453 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f210c130d30 0x7f210c131160 secure :-1 s=READY pgs=1485 cs=0 l=1 rx=0x7f20f4001090 tx=0x7f20f4000ed0).stop 2019-12-05T05:47:26.482 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:26.478+0000 7f2112360700 1 -- 172.21.15.42:0/962391453 shutdown_connections 2019-12-05T05:47:26.482 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:26.478+0000 7f2112360700 1 --2- 172.21.15.42:0/962391453 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f20f003ac10 0x7f20f003d120 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:47:26.482 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:26.478+0000 7f2112360700 1 --2- 172.21.15.42:0/962391453 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f210c130d30 0x7f210c131160 unknown :-1 s=CLOSED pgs=1485 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:47:26.482 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:26.478+0000 7f2112360700 1 -- 172.21.15.42:0/962391453 shutdown_connections 2019-12-05T05:47:26.482 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:26.478+0000 7f2112360700 1 -- 172.21.15.42:0/962391453 wait complete. 2019-12-05T05:47:26.483 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:47:26.478+0000 7f2112360700 1 -- 172.21.15.42:0/962391453 >> 172.21.15.42:0/962391453 conn(0x7f210c08fde0 msgr2=0x7f210c059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:47:26.491 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a reported in up:standby state 2019-12-05T05:47:26.491 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 96 secs before thrashing 2019-12-05T05:47:41.619 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:47:41.623 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:48:11.683 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:48:11.687 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:48:41.724 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:48:41.729 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:49:02.492 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:49:02.598 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.589+0000 7fe013579700 1 Processor -- start 2019-12-05T05:49:02.599 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.589+0000 7fe013579700 1 -- start start 2019-12-05T05:49:02.599 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.589+0000 7fe013579700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe00c08df80 0x7fe00c09f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:49:02.599 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.589+0000 7fe013579700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe00c09f830 0x7fe00c14efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:49:02.599 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.589+0000 7fe013579700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe00c09fc60 con 0x7fe00c0a7740 2019-12-05T05:49:02.599 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.589+0000 7fe013579700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe00c0a0b00 con 0x7fe00c09f830 2019-12-05T05:49:02.599 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.589+0000 7fe013579700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe00c08e5b0 con 0x7fe00c08df80 2019-12-05T05:49:02.600 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.589+0000 7fe012317700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe00c09f830 0x7fe00c14efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:49:02.600 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.589+0000 7fe011b16700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fe00c0a7740 0x7fe00c07e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:41750/0 (socket says 172.21.15.42:41750) 2019-12-05T05:49:02.600 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.589+0000 7fe011b16700 1 -- 172.21.15.42:0/1208053314 learned_addr learned my addr 172.21.15.42:0/1208053314 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:49:02.600 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.589+0000 7fe011315700 1 --2- 172.21.15.42:0/1208053314 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe00c08df80 0x7fe00c09f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:49:02.600 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.589+0000 7fe011315700 1 -- 172.21.15.42:0/1208053314 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe00c08df80 msgr2=0x7fe00c09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2019-12-05T05:49:02.600 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.589+0000 7fe011315700 1 -- 172.21.15.42:0/1208053314 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe00c08df80 msgr2=0x7fe00c09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2019-12-05T05:49:02.601 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.589+0000 7fe011315700 1 --2- 172.21.15.42:0/1208053314 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe00c08df80 0x7fe00c09f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-12-05T05:49:02.601 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.589+0000 7fe011315700 1 --2- 172.21.15.42:0/1208053314 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe00c08df80 0x7fe00c09f2e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rx=0 tx=0)._fault waiting 0.200000 2019-12-05T05:49:02.601 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.589+0000 7fe012317700 1 -- 172.21.15.42:0/1208053314 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe00c08df80 msgr2=0x7fe00c09f2e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2019-12-05T05:49:02.601 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.589+0000 7fe012317700 1 --2- 172.21.15.42:0/1208053314 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe00c08df80 0x7fe00c09f2e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:49:02.601 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.589+0000 7fe012317700 1 -- 172.21.15.42:0/1208053314 >> v1:172.21.15.42:6789/0 conn(0x7fe00c0a7740 legacy=0x7fe00c07e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:49:02.601 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.589+0000 7fe010b14700 1 -- 172.21.15.42:0/1208053314 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2766219052 0 0) 0x7fe00c09fc60 con 0x7fe00c0a7740 2019-12-05T05:49:02.601 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.589+0000 7fe012317700 1 -- 172.21.15.42:0/1208053314 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe00c0a0710 con 0x7fe00c09f830 2019-12-05T05:49:02.602 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.589+0000 7fe012317700 1 --2- 172.21.15.42:0/1208053314 >> v2:172.21.15.42:3300/0 conn(0x7fe00c09f830 0x7fe00c14efe0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fe004008790 tx=0x7fe00400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:02.602 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.589+0000 7fe012317700 1 --2- 172.21.15.42:0/1208053314 >> v2:172.21.15.42:3300/0 conn(0x7fe00c09f830 0x7fe00c14efe0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fe004008790 tx=0x7fe00400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:02.602 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.589+0000 7fe012317700 1 --2- 172.21.15.42:0/1208053314 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe00c09f830 0x7fe00c14efe0 secure :-1 s=READY pgs=1491 cs=0 l=1 rx=0x7fe004008790 tx=0x7fe00400fc70).ready entity=mon.1 client_cookie=ba8eedfc111b8b65 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:49:02.602 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.593+0000 7fe012317700 1 --2- 172.21.15.42:0/1208053314 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe00c09f830 0x7fe00c14efe0 secure :-1 s=THROTTLE_DONE pgs=1491 cs=0 l=1 rx=0x7fe004008790 tx=0x7fe00400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:02.602 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.593+0000 7fe010b14700 1 -- 172.21.15.42:0/1208053314 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe004008bb0 con 0x7fe00c09f830 2019-12-05T05:49:02.602 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.593+0000 7fe012317700 1 --2- 172.21.15.42:0/1208053314 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe00c09f830 0x7fe00c14efe0 secure :-1 s=THROTTLE_DONE pgs=1491 cs=0 l=1 rx=0x7fe004008790 tx=0x7fe00400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:02.602 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.593+0000 7fe012317700 1 --2- 172.21.15.42:0/1208053314 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe00c09f830 0x7fe00c14efe0 secure :-1 s=THROTTLE_DONE pgs=1491 cs=0 l=1 rx=0x7fe004008790 tx=0x7fe00400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:02.603 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.593+0000 7fe010b14700 1 -- 172.21.15.42:0/1208053314 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0040087d0 con 0x7fe00c09f830 2019-12-05T05:49:02.603 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.593+0000 7fe010b14700 1 -- 172.21.15.42:0/1208053314 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe004011860 con 0x7fe00c09f830 2019-12-05T05:49:02.603 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.593+0000 7fe013579700 1 -- 172.21.15.42:0/1208053314 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe00c09f830 msgr2=0x7fe00c14efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:49:02.603 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.593+0000 7fe013579700 1 --2- 172.21.15.42:0/1208053314 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe00c09f830 0x7fe00c14efe0 secure :-1 s=READY pgs=1491 cs=0 l=1 rx=0x7fe004008790 tx=0x7fe00400fc70).stop 2019-12-05T05:49:02.603 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.593+0000 7fe013579700 1 -- 172.21.15.42:0/1208053314 shutdown_connections 2019-12-05T05:49:02.603 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.593+0000 7fe013579700 1 --2- 172.21.15.42:0/1208053314 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe00c09f830 0x7fe00c14efe0 unknown :-1 s=CLOSED pgs=1491 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:49:02.603 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.593+0000 7fe013579700 1 --2- 172.21.15.42:0/1208053314 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe00c08df80 0x7fe00c09f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:49:02.604 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.593+0000 7fe013579700 1 -- 172.21.15.42:0/1208053314 shutdown_connections 2019-12-05T05:49:02.604 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.593+0000 7fe013579700 1 -- 172.21.15.42:0/1208053314 wait complete. 2019-12-05T05:49:02.604 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.593+0000 7fe013579700 1 -- 172.21.15.42:0/1208053314 >> 172.21.15.42:0/1208053314 conn(0x7fe00c0aabe0 msgr2=0x7fe00c0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:49:02.604 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.593+0000 7fe013579700 1 Processor -- start 2019-12-05T05:49:02.607 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.593+0000 7fe013579700 1 -- start start 2019-12-05T05:49:02.607 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.597+0000 7fe013579700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe00c142530 0x7fe00c1436f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:49:02.607 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.597+0000 7fe013579700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe00c0722c0 0x7fe00c074700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:49:02.607 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.597+0000 7fe013579700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe00c143c40 con 0x7fe00c08eeb0 2019-12-05T05:49:02.608 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.597+0000 7fe013579700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe00c132890 con 0x7fe00c142530 2019-12-05T05:49:02.608 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.597+0000 7fe013579700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe00c143f70 con 0x7fe00c0722c0 2019-12-05T05:49:02.608 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.597+0000 7fe011b16700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fe00c08eeb0 0x7fe00c116780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:41754/0 (socket says 172.21.15.42:41754) 2019-12-05T05:49:02.608 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.597+0000 7fe011b16700 1 -- 172.21.15.42:0/43198170 learned_addr learned my addr 172.21.15.42:0/43198170 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:49:02.608 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.597+0000 7fe011315700 1 --2- 172.21.15.42:0/43198170 >> v2:172.21.15.40:3300/0 conn(0x7fe00c142530 0x7fe00c1436f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:49:02.609 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.597+0000 7fe012317700 1 --2- 172.21.15.42:0/43198170 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe00c0722c0 0x7fe00c074700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:49:02.609 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.597+0000 7fdffe7fc700 1 -- 172.21.15.42:0/43198170 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1215374221 0 0) 0x7fe00c143c40 con 0x7fe00c08eeb0 2019-12-05T05:49:02.609 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.597+0000 7fdffe7fc700 1 -- 172.21.15.42:0/43198170 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdff40030b0 con 0x7fe00c08eeb0 2019-12-05T05:49:02.609 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.597+0000 7fe011315700 1 -- 172.21.15.42:0/43198170 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe00c0722c0 msgr2=0x7fe00c074700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:49:02.609 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.597+0000 7fe011315700 1 --2- 172.21.15.42:0/43198170 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe00c0722c0 0x7fe00c074700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:49:02.609 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.597+0000 7fe011315700 1 -- 172.21.15.42:0/43198170 >> v1:172.21.15.42:6789/0 conn(0x7fe00c08eeb0 legacy=0x7fe00c116780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:49:02.609 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.597+0000 7fe011315700 1 -- 172.21.15.42:0/43198170 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe00c116f80 con 0x7fe00c142530 2019-12-05T05:49:02.610 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.597+0000 7fe012317700 1 --2- 172.21.15.42:0/43198170 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe00c0722c0 0x7fe00c074700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:49:02.610 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.597+0000 7fe011315700 1 --2- 172.21.15.42:0/43198170 >> v2:172.21.15.40:3300/0 conn(0x7fe00c142530 0x7fe00c1436f0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fe00c08f540 tx=0x7fe008010ea0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:02.610 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.597+0000 7fe011315700 1 --2- 172.21.15.42:0/43198170 >> v2:172.21.15.40:3300/0 conn(0x7fe00c142530 0x7fe00c1436f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fe00c08f540 tx=0x7fe008010ea0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:02.610 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.597+0000 7fe011315700 1 --2- 172.21.15.42:0/43198170 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe00c142530 0x7fe00c1436f0 secure :-1 s=READY pgs=1154 cs=0 l=1 rx=0x7fe00c08f540 tx=0x7fe008010ea0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:49:02.610 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.597+0000 7fe011315700 1 --2- 172.21.15.42:0/43198170 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe00c142530 0x7fe00c1436f0 secure :-1 s=THROTTLE_DONE pgs=1154 cs=0 l=1 rx=0x7fe00c08f540 tx=0x7fe008010ea0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:02.610 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.597+0000 7fdffe7fc700 1 -- 172.21.15.42:0/43198170 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe008012cf0 con 0x7fe00c142530 2019-12-05T05:49:02.610 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.597+0000 7fe011315700 1 --2- 172.21.15.42:0/43198170 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe00c142530 0x7fe00c1436f0 secure :-1 s=THROTTLE_DONE pgs=1154 cs=0 l=1 rx=0x7fe00c08f540 tx=0x7fe008010ea0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:02.611 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.597+0000 7fe013579700 1 -- 172.21.15.42:0/43198170 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe00c074c50 con 0x7fe00c142530 2019-12-05T05:49:02.611 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.597+0000 7fe011315700 1 --2- 172.21.15.42:0/43198170 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe00c142530 0x7fe00c1436f0 secure :-1 s=THROTTLE_DONE pgs=1154 cs=0 l=1 rx=0x7fe00c08f540 tx=0x7fe008010ea0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:02.611 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.597+0000 7fdffe7fc700 1 -- 172.21.15.42:0/43198170 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe008012910 con 0x7fe00c142530 2019-12-05T05:49:02.611 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.597+0000 7fdffe7fc700 1 -- 172.21.15.42:0/43198170 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe008005ca0 con 0x7fe00c142530 2019-12-05T05:49:02.611 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.597+0000 7fe013579700 1 -- 172.21.15.42:0/43198170 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe00c0751d0 con 0x7fe00c142530 2019-12-05T05:49:02.611 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.601+0000 7fe011315700 1 --2- 172.21.15.42:0/43198170 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe00c142530 0x7fe00c1436f0 secure :-1 s=THROTTLE_DONE pgs=1154 cs=0 l=1 rx=0x7fe00c08f540 tx=0x7fe008010ea0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:02.611 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.601+0000 7fdffe7fc700 1 -- 172.21.15.42:0/43198170 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fe008010170 con 0x7fe00c142530 2019-12-05T05:49:02.612 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.601+0000 7fe011315700 1 --2- 172.21.15.42:0/43198170 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe00c142530 0x7fe00c1436f0 secure :-1 s=THROTTLE_DONE pgs=1154 cs=0 l=1 rx=0x7fe00c08f540 tx=0x7fe008010ea0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:02.612 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.601+0000 7fdffe7fc700 1 --2- 172.21.15.42:0/43198170 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fdff403b2b0 0x7fdff403d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:49:02.612 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.601+0000 7fdffe7fc700 1 -- 172.21.15.42:0/43198170 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 8611+0+0 (secure 0 0 0) 0x7fe00804b380 con 0x7fe00c142530 2019-12-05T05:49:02.612 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.601+0000 7fe011b16700 1 --2- 172.21.15.42:0/43198170 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fdff403b2b0 0x7fdff403d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:49:02.612 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.601+0000 7fe011b16700 1 --2- 172.21.15.42:0/43198170 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fdff403b2b0 0x7fdff403d7c0 crc :-1 s=READY pgs=643 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:49:02.613 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.605+0000 7fe013579700 1 -- 172.21.15.42:0/43198170 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe00c05d110 con 0x7fe00c142530 2019-12-05T05:49:02.618 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.613+0000 7fe011315700 1 --2- 172.21.15.42:0/43198170 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe00c142530 0x7fe00c1436f0 secure :-1 s=THROTTLE_DONE pgs=1154 cs=0 l=1 rx=0x7fe00c08f540 tx=0x7fe008010ea0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:02.618 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.613+0000 7fdffe7fc700 1 -- 172.21.15.42:0/43198170 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fe008007150 con 0x7fe00c142530 2019-12-05T05:49:02.791 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.785+0000 7fe013579700 1 -- 172.21.15.42:0/43198170 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe00c05b8c0 con 0x7fe00c142530 2019-12-05T05:49:02.792 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.785+0000 7fe011315700 1 --2- 172.21.15.42:0/43198170 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe00c142530 0x7fe00c1436f0 secure :-1 s=THROTTLE_DONE pgs=1154 cs=0 l=1 rx=0x7fe00c08f540 tx=0x7fe008010ea0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:02.792 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.785+0000 7fdffe7fc700 1 -- 172.21.15.42:0/43198170 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 183 v183) v1 ==== 95+0+2112 (secure 0 0 0) 0x7fe008033250 con 0x7fe00c142530 2019-12-05T05:49:02.793 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 183 2019-12-05T05:49:02.793 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:49:02.793 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":183,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":8188,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6835/2456922143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2456922143},{"type":"v1","addr":"172.21.15.42:6835","nonce":2456922143}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":183}],"filesystems":[{"mdsmap":{"epoch":182,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:47:19.499092+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6140},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6140":{"gid":6140,"name":"b","rank":0,"incarnation":179,"state":"up:active","state_seq":26,"addr":"172.21.15.40:6833/2146240383","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2146240383},{"type":"v1","addr":"172.21.15.40:6833","nonce":2146240383}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:49:02.796 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.789+0000 7fe013579700 1 -- 172.21.15.42:0/43198170 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fdff403b2b0 msgr2=0x7fdff403d7c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:49:02.796 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.789+0000 7fe013579700 1 --2- 172.21.15.42:0/43198170 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fdff403b2b0 0x7fdff403d7c0 crc :-1 s=READY pgs=643 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:49:02.796 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.789+0000 7fe013579700 1 -- 172.21.15.42:0/43198170 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe00c142530 msgr2=0x7fe00c1436f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:49:02.797 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.789+0000 7fe013579700 1 --2- 172.21.15.42:0/43198170 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe00c142530 0x7fe00c1436f0 secure :-1 s=READY pgs=1154 cs=0 l=1 rx=0x7fe00c08f540 tx=0x7fe008010ea0).stop 2019-12-05T05:49:02.797 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.793+0000 7fe013579700 1 -- 172.21.15.42:0/43198170 shutdown_connections 2019-12-05T05:49:02.797 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.793+0000 7fe013579700 1 --2- 172.21.15.42:0/43198170 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fdff403b2b0 0x7fdff403d7c0 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:49:02.797 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.793+0000 7fe013579700 1 --2- 172.21.15.42:0/43198170 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe00c0722c0 0x7fe00c074700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:49:02.797 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.793+0000 7fe013579700 1 --2- 172.21.15.42:0/43198170 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe00c142530 0x7fe00c1436f0 unknown :-1 s=CLOSED pgs=1154 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:49:02.798 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.793+0000 7fe013579700 1 -- 172.21.15.42:0/43198170 shutdown_connections 2019-12-05T05:49:02.798 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.793+0000 7fe013579700 1 -- 172.21.15.42:0/43198170 wait complete. 2019-12-05T05:49:02.798 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.793+0000 7fe013579700 1 -- 172.21.15.42:0/43198170 >> 172.21.15.42:0/43198170 conn(0x7fe00c09ffb0 msgr2=0x7fe00c07e830 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:49:02.806 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.b (rank=0) 2019-12-05T05:49:02.806 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2019-12-05T05:49:02.806 INFO:teuthology.orchestra.run:waiting for 300 2019-12-05T05:49:02.820 INFO:tasks.ceph.mds.b:Stopped 2019-12-05T05:49:02.821 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:49:02.919 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.909+0000 7fc985698700 1 Processor -- start 2019-12-05T05:49:02.919 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.909+0000 7fc985698700 1 -- start start 2019-12-05T05:49:02.919 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.909+0000 7fc985698700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc98008df80 0x7fc98009f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:49:02.919 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.909+0000 7fc985698700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc98009f830 0x7fc98014efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:49:02.919 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.909+0000 7fc985698700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc98009fc60 con 0x7fc9800a7740 2019-12-05T05:49:02.920 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.909+0000 7fc985698700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9800a0b00 con 0x7fc98009f830 2019-12-05T05:49:02.920 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.909+0000 7fc985698700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc98008e5b0 con 0x7fc98008df80 2019-12-05T05:49:02.920 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.909+0000 7fc97ffff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc98009f830 0x7fc98014efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:49:02.920 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.909+0000 7fc97ffff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc98009f830 0x7fc98014efe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:41730/0 (socket says 172.21.15.42:41730) 2019-12-05T05:49:02.920 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.909+0000 7fc97ffff700 1 -- 172.21.15.42:0/3243189926 learned_addr learned my addr 172.21.15.42:0/3243189926 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:49:02.920 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.909+0000 7fc97effd700 1 --2- 172.21.15.42:0/3243189926 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc98008df80 0x7fc98009f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:49:02.921 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.913+0000 7fc97ffff700 1 -- 172.21.15.42:0/3243189926 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc98008df80 msgr2=0x7fc98009f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:49:02.921 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.913+0000 7fc97ffff700 1 --2- 172.21.15.42:0/3243189926 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc98008df80 0x7fc98009f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:49:02.921 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.913+0000 7fc97ffff700 1 -- 172.21.15.42:0/3243189926 >> v1:172.21.15.42:6789/0 conn(0x7fc9800a7740 legacy=0x7fc98007e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:49:02.921 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.913+0000 7fc97ffff700 1 -- 172.21.15.42:0/3243189926 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9800a0710 con 0x7fc98009f830 2019-12-05T05:49:02.921 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.913+0000 7fc97ffff700 1 --2- 172.21.15.42:0/3243189926 >> v2:172.21.15.42:3300/0 conn(0x7fc98009f830 0x7fc98014efe0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fc978008b10 tx=0x7fc9780100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:02.922 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.913+0000 7fc97ffff700 1 --2- 172.21.15.42:0/3243189926 >> v2:172.21.15.42:3300/0 conn(0x7fc98009f830 0x7fc98014efe0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fc978008b10 tx=0x7fc9780100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:02.922 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.913+0000 7fc97ffff700 1 --2- 172.21.15.42:0/3243189926 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc98009f830 0x7fc98014efe0 secure :-1 s=READY pgs=1494 cs=0 l=1 rx=0x7fc978008b10 tx=0x7fc9780100b0).ready entity=mon.1 client_cookie=640b9052d6339e36 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:49:02.922 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.913+0000 7fc97ffff700 1 --2- 172.21.15.42:0/3243189926 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc98009f830 0x7fc98014efe0 secure :-1 s=THROTTLE_DONE pgs=1494 cs=0 l=1 rx=0x7fc978008b10 tx=0x7fc9780100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:02.922 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.913+0000 7fc97e7fc700 1 -- 172.21.15.42:0/3243189926 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc978010da0 con 0x7fc98009f830 2019-12-05T05:49:02.922 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.913+0000 7fc97ffff700 1 --2- 172.21.15.42:0/3243189926 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc98009f830 0x7fc98014efe0 secure :-1 s=THROTTLE_DONE pgs=1494 cs=0 l=1 rx=0x7fc978008b10 tx=0x7fc9780100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:02.923 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.913+0000 7fc97ffff700 1 --2- 172.21.15.42:0/3243189926 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc98009f830 0x7fc98014efe0 secure :-1 s=THROTTLE_DONE pgs=1494 cs=0 l=1 rx=0x7fc978008b10 tx=0x7fc9780100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:02.923 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.913+0000 7fc97e7fc700 1 -- 172.21.15.42:0/3243189926 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9780109c0 con 0x7fc98009f830 2019-12-05T05:49:02.923 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.913+0000 7fc97e7fc700 1 -- 172.21.15.42:0/3243189926 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc9780114a0 con 0x7fc98009f830 2019-12-05T05:49:02.923 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.913+0000 7fc985698700 1 -- 172.21.15.42:0/3243189926 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc98009f830 msgr2=0x7fc98014efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:49:02.924 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.913+0000 7fc985698700 1 --2- 172.21.15.42:0/3243189926 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc98009f830 0x7fc98014efe0 secure :-1 s=READY pgs=1494 cs=0 l=1 rx=0x7fc978008b10 tx=0x7fc9780100b0).stop 2019-12-05T05:49:02.924 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.913+0000 7fc985698700 1 -- 172.21.15.42:0/3243189926 shutdown_connections 2019-12-05T05:49:02.924 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.913+0000 7fc985698700 1 --2- 172.21.15.42:0/3243189926 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc98009f830 0x7fc98014efe0 unknown :-1 s=CLOSED pgs=1494 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:49:02.924 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.913+0000 7fc985698700 1 --2- 172.21.15.42:0/3243189926 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc98008df80 0x7fc98009f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:49:02.924 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.913+0000 7fc985698700 1 -- 172.21.15.42:0/3243189926 shutdown_connections 2019-12-05T05:49:02.924 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.913+0000 7fc985698700 1 -- 172.21.15.42:0/3243189926 wait complete. 2019-12-05T05:49:02.925 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.913+0000 7fc985698700 1 -- 172.21.15.42:0/3243189926 >> 172.21.15.42:0/3243189926 conn(0x7fc9800aabe0 msgr2=0x7fc9800a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:49:02.928 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.917+0000 7fc985698700 1 Processor -- start 2019-12-05T05:49:02.928 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.917+0000 7fc985698700 1 -- start start 2019-12-05T05:49:02.928 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.917+0000 7fc985698700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc98005a210 0x7fc980063e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:49:02.929 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.917+0000 7fc985698700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc980065830 0x7fc980129cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:49:02.929 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.917+0000 7fc985698700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc98012a200 0x7fc9801445e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:49:02.929 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.917+0000 7fc985698700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc980064510 con 0x7fc980065830 2019-12-05T05:49:02.929 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.917+0000 7fc985698700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9800647b0 con 0x7fc98005a210 2019-12-05T05:49:02.929 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.917+0000 7fc985698700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc980064a50 con 0x7fc98012a200 2019-12-05T05:49:02.930 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.917+0000 7fc97f7fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc98005a210 0x7fc980063e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:49:02.930 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.917+0000 7fc97f7fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc98005a210 0x7fc980063e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:41732/0 (socket says 172.21.15.42:41732) 2019-12-05T05:49:02.930 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.917+0000 7fc97f7fe700 1 -- 172.21.15.42:0/899500903 learned_addr learned my addr 172.21.15.42:0/899500903 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:49:02.930 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.917+0000 7fc97effd700 1 --2- 172.21.15.42:0/899500903 >> v2:172.21.15.40:3300/0 conn(0x7fc980065830 0x7fc980129cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:49:02.930 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.917+0000 7fc97ffff700 1 --2- 172.21.15.42:0/899500903 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc98012a200 0x7fc9801445e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:49:02.930 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.917+0000 7fc97f7fe700 1 -- 172.21.15.42:0/899500903 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc98012a200 msgr2=0x7fc9801445e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:49:02.931 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.917+0000 7fc97f7fe700 1 --2- 172.21.15.42:0/899500903 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc98012a200 0x7fc9801445e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:49:02.931 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.917+0000 7fc97f7fe700 1 -- 172.21.15.42:0/899500903 >> v2:172.21.15.40:3300/0 conn(0x7fc980065830 msgr2=0x7fc980129cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:49:02.931 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.917+0000 7fc97f7fe700 1 --2- 172.21.15.42:0/899500903 >> v2:172.21.15.40:3300/0 conn(0x7fc980065830 0x7fc980129cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:49:02.931 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.917+0000 7fc97f7fe700 1 -- 172.21.15.42:0/899500903 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc980065c60 con 0x7fc98005a210 2019-12-05T05:49:02.931 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.917+0000 7fc97effd700 1 --2- 172.21.15.42:0/899500903 >> v2:172.21.15.40:3300/0 conn(0x7fc980065830 0x7fc980129cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T05:49:02.931 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.917+0000 7fc97f7fe700 1 --2- 172.21.15.42:0/899500903 >> v2:172.21.15.42:3300/0 conn(0x7fc98005a210 0x7fc980063e70 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fc980064970 tx=0x7fc970010a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:02.931 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.917+0000 7fc97f7fe700 1 --2- 172.21.15.42:0/899500903 >> v2:172.21.15.42:3300/0 conn(0x7fc98005a210 0x7fc980063e70 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fc980064970 tx=0x7fc970010a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:02.932 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.917+0000 7fc97f7fe700 1 --2- 172.21.15.42:0/899500903 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc98005a210 0x7fc980063e70 secure :-1 s=READY pgs=1495 cs=0 l=1 rx=0x7fc980064970 tx=0x7fc970010a20).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:49:02.932 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.917+0000 7fc97f7fe700 1 --2- 172.21.15.42:0/899500903 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc98005a210 0x7fc980063e70 secure :-1 s=THROTTLE_DONE pgs=1495 cs=0 l=1 rx=0x7fc980064970 tx=0x7fc970010a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:02.932 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.917+0000 7fc967fff700 1 -- 172.21.15.42:0/899500903 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc97000caa0 con 0x7fc98005a210 2019-12-05T05:49:02.932 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.917+0000 7fc97f7fe700 1 --2- 172.21.15.42:0/899500903 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc98005a210 0x7fc980063e70 secure :-1 s=THROTTLE_DONE pgs=1495 cs=0 l=1 rx=0x7fc980064970 tx=0x7fc970010a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:02.932 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.917+0000 7fc97f7fe700 1 --2- 172.21.15.42:0/899500903 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc98005a210 0x7fc980063e70 secure :-1 s=THROTTLE_DONE pgs=1495 cs=0 l=1 rx=0x7fc980064970 tx=0x7fc970010a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:02.932 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.917+0000 7fc967fff700 1 -- 172.21.15.42:0/899500903 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc97000c6c0 con 0x7fc98005a210 2019-12-05T05:49:02.933 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.917+0000 7fc985698700 1 -- 172.21.15.42:0/899500903 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc98012a630 con 0x7fc98005a210 2019-12-05T05:49:02.933 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.917+0000 7fc967fff700 1 -- 172.21.15.42:0/899500903 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc970012250 con 0x7fc98005a210 2019-12-05T05:49:02.933 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.917+0000 7fc985698700 1 -- 172.21.15.42:0/899500903 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc980144d50 con 0x7fc98005a210 2019-12-05T05:49:02.933 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.921+0000 7fc97f7fe700 1 --2- 172.21.15.42:0/899500903 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc98005a210 0x7fc980063e70 secure :-1 s=THROTTLE_DONE pgs=1495 cs=0 l=1 rx=0x7fc980064970 tx=0x7fc970010a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:02.933 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.921+0000 7fc967fff700 1 -- 172.21.15.42:0/899500903 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fc970012530 con 0x7fc98005a210 2019-12-05T05:49:02.933 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.921+0000 7fc97f7fe700 1 --2- 172.21.15.42:0/899500903 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc98005a210 0x7fc980063e70 secure :-1 s=THROTTLE_DONE pgs=1495 cs=0 l=1 rx=0x7fc980064970 tx=0x7fc970010a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:02.933 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.921+0000 7fc967fff700 1 --2- 172.21.15.42:0/899500903 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc96003ae70 0x7fc96003d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:49:02.934 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.921+0000 7fc967fff700 1 -- 172.21.15.42:0/899500903 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 8611+0+0 (secure 0 0 0) 0x7fc97004b2c0 con 0x7fc98005a210 2019-12-05T05:49:02.934 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.921+0000 7fc97effd700 1 --2- 172.21.15.42:0/899500903 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc96003ae70 0x7fc96003d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:49:02.934 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.921+0000 7fc97effd700 1 --2- 172.21.15.42:0/899500903 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc96003ae70 0x7fc96003d380 crc :-1 s=READY pgs=644 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:49:02.934 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.925+0000 7fc985698700 1 -- 172.21.15.42:0/899500903 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc980064c10 con 0x7fc98005a210 2019-12-05T05:49:02.935 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.929+0000 7fc97f7fe700 1 --2- 172.21.15.42:0/899500903 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc98005a210 0x7fc980063e70 secure :-1 s=THROTTLE_DONE pgs=1495 cs=0 l=1 rx=0x7fc980064970 tx=0x7fc970010a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:02.935 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:02.933+0000 7fc967fff700 1 -- 172.21.15.42:0/899500903 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fc980065c60 con 0x7fc98005a210 2019-12-05T05:49:03.104 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:03.101+0000 7fc985698700 1 -- 172.21.15.42:0/899500903 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc98005d110 con 0x7fc98005a210 2019-12-05T05:49:03.105 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:03.101+0000 7fc97f7fe700 1 --2- 172.21.15.42:0/899500903 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc98005a210 0x7fc980063e70 secure :-1 s=THROTTLE_DONE pgs=1495 cs=0 l=1 rx=0x7fc980064970 tx=0x7fc970010a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:03.106 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:03.101+0000 7fc967fff700 1 -- 172.21.15.42:0/899500903 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 183 v183) v1 ==== 95+0+2112 (secure 0 0 0) 0x7fc970032320 con 0x7fc98005a210 2019-12-05T05:49:03.106 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 183 2019-12-05T05:49:03.106 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:49:03.106 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":183,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":8188,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6835/2456922143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2456922143},{"type":"v1","addr":"172.21.15.42:6835","nonce":2456922143}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":183}],"filesystems":[{"mdsmap":{"epoch":182,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:47:19.499092+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6140},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6140":{"gid":6140,"name":"b","rank":0,"incarnation":179,"state":"up:active","state_seq":26,"addr":"172.21.15.40:6833/2146240383","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2146240383},{"type":"v1","addr":"172.21.15.40:6833","nonce":2146240383}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:49:03.108 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:03.105+0000 7fc985698700 1 -- 172.21.15.42:0/899500903 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc96003ae70 msgr2=0x7fc96003d380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:49:03.108 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:03.105+0000 7fc985698700 1 --2- 172.21.15.42:0/899500903 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc96003ae70 0x7fc96003d380 crc :-1 s=READY pgs=644 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:49:03.108 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:03.105+0000 7fc985698700 1 -- 172.21.15.42:0/899500903 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc98005a210 msgr2=0x7fc980063e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:49:03.108 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:03.105+0000 7fc985698700 1 --2- 172.21.15.42:0/899500903 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc98005a210 0x7fc980063e70 secure :-1 s=READY pgs=1495 cs=0 l=1 rx=0x7fc980064970 tx=0x7fc970010a20).stop 2019-12-05T05:49:03.109 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:03.105+0000 7fc985698700 1 -- 172.21.15.42:0/899500903 shutdown_connections 2019-12-05T05:49:03.109 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:03.105+0000 7fc985698700 1 --2- 172.21.15.42:0/899500903 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc96003ae70 0x7fc96003d380 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:49:03.109 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:03.105+0000 7fc985698700 1 --2- 172.21.15.42:0/899500903 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc98012a200 0x7fc9801445e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:49:03.109 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:03.105+0000 7fc985698700 1 --2- 172.21.15.42:0/899500903 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc98005a210 0x7fc980063e70 unknown :-1 s=CLOSED pgs=1495 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:49:03.109 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:03.105+0000 7fc985698700 1 --2- 172.21.15.42:0/899500903 >> v2:172.21.15.40:3300/0 conn(0x7fc980065830 0x7fc980129cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:49:03.109 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:03.105+0000 7fc985698700 1 -- 172.21.15.42:0/899500903 shutdown_connections 2019-12-05T05:49:03.109 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:03.105+0000 7fc985698700 1 -- 172.21.15.42:0/899500903 wait complete. 2019-12-05T05:49:03.110 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:03.105+0000 7fc985698700 1 -- 172.21.15.42:0/899500903 >> 172.21.15.42:0/899500903 conn(0x7fc9800a6df0 msgr2=0x7fc980115e80 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:49:03.117 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T05:49:05.118 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:49:05.224 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:05.213+0000 7f490ea7c700 1 Processor -- start 2019-12-05T05:49:05.224 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:05.213+0000 7f490ea7c700 1 -- start start 2019-12-05T05:49:05.225 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:05.217+0000 7f490ea7c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f49080a7740 0x7f490808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:49:05.225 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:05.217+0000 7f490ea7c700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f490807e720 0x7f490807eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:49:05.225 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:05.217+0000 7f490ea7c700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49080a0710 con 0x7f490809f2e0 2019-12-05T05:49:05.225 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:05.217+0000 7f490ea7c700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49080a0b00 con 0x7f49080a7740 2019-12-05T05:49:05.225 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:05.217+0000 7f490ea7c700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f490808e5b0 con 0x7f490807e720 2019-12-05T05:49:05.225 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:05.217+0000 7f490d019700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f49080a7740 0x7f490808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:49:05.225 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:05.217+0000 7f490d019700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f49080a7740 0x7f490808df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:41746/0 (socket says 172.21.15.42:41746) 2019-12-05T05:49:05.226 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:05.217+0000 7f490d019700 1 -- 172.21.15.42:0/2180570443 learned_addr learned my addr 172.21.15.42:0/2180570443 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:49:05.226 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:05.217+0000 7f490c818700 1 --2- 172.21.15.42:0/2180570443 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f490807e720 0x7f490807eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:49:05.226 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:05.217+0000 7f490d019700 1 -- 172.21.15.42:0/2180570443 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f490807e720 msgr2=0x7f490807eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:49:05.226 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:05.217+0000 7f490d019700 1 --2- 172.21.15.42:0/2180570443 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f490807e720 0x7f490807eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:49:05.226 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:05.217+0000 7f490d019700 1 -- 172.21.15.42:0/2180570443 >> v1:172.21.15.42:6789/0 conn(0x7f490809f2e0 legacy=0x7f490809f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:49:05.226 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:05.217+0000 7f490d019700 1 -- 172.21.15.42:0/2180570443 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f490807f0a0 con 0x7f49080a7740 2019-12-05T05:49:05.226 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:05.217+0000 7f490d019700 1 --2- 172.21.15.42:0/2180570443 >> v2:172.21.15.42:3300/0 conn(0x7f49080a7740 0x7f490808df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f48fc008b10 tx=0x7f48fc0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:05.227 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:05.217+0000 7f490d019700 1 --2- 172.21.15.42:0/2180570443 >> v2:172.21.15.42:3300/0 conn(0x7f49080a7740 0x7f490808df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f48fc008b10 tx=0x7f48fc0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:05.227 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:05.217+0000 7f490d019700 1 --2- 172.21.15.42:0/2180570443 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f49080a7740 0x7f490808df80 secure :-1 s=READY pgs=1497 cs=0 l=1 rx=0x7f48fc008b10 tx=0x7f48fc0100b0).ready entity=mon.1 client_cookie=d13fc8189fd7c7b5 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:49:05.227 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:05.217+0000 7f490d019700 1 --2- 172.21.15.42:0/2180570443 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f49080a7740 0x7f490808df80 secure :-1 s=THROTTLE_DONE pgs=1497 cs=0 l=1 rx=0x7f48fc008b10 tx=0x7f48fc0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:05.227 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:05.217+0000 7f490d019700 1 --2- 172.21.15.42:0/2180570443 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f49080a7740 0x7f490808df80 secure :-1 s=THROTTLE_DONE pgs=1497 cs=0 l=1 rx=0x7f48fc008b10 tx=0x7f48fc0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:05.227 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:05.217+0000 7f4907fff700 1 -- 172.21.15.42:0/2180570443 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f48fc00a160 con 0x7f49080a7740 2019-12-05T05:49:05.227 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:05.217+0000 7f490d019700 1 --2- 172.21.15.42:0/2180570443 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f49080a7740 0x7f490808df80 secure :-1 s=THROTTLE_DONE pgs=1497 cs=0 l=1 rx=0x7f48fc008b10 tx=0x7f48fc0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:05.228 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:05.217+0000 7f4907fff700 1 -- 172.21.15.42:0/2180570443 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48fc011a30 con 0x7f49080a7740 2019-12-05T05:49:05.228 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:05.217+0000 7f4907fff700 1 -- 172.21.15.42:0/2180570443 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f48fc00a550 con 0x7f49080a7740 2019-12-05T05:49:05.228 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:05.217+0000 7f490ea7c700 1 -- 172.21.15.42:0/2180570443 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f49080a7740 msgr2=0x7f490808df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:49:05.228 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:05.217+0000 7f490ea7c700 1 --2- 172.21.15.42:0/2180570443 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f49080a7740 0x7f490808df80 secure :-1 s=READY pgs=1497 cs=0 l=1 rx=0x7f48fc008b10 tx=0x7f48fc0100b0).stop 2019-12-05T05:49:05.228 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:05.217+0000 7f490ea7c700 1 -- 172.21.15.42:0/2180570443 shutdown_connections 2019-12-05T05:49:05.228 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:05.217+0000 7f490ea7c700 1 --2- 172.21.15.42:0/2180570443 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f49080a7740 0x7f490808df80 unknown :-1 s=CLOSED pgs=1497 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:49:05.228 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:05.217+0000 7f490ea7c700 1 --2- 172.21.15.42:0/2180570443 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f490807e720 0x7f490807eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:49:05.229 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:05.217+0000 7f490ea7c700 1 -- 172.21.15.42:0/2180570443 shutdown_connections 2019-12-05T05:49:05.229 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:05.217+0000 7f490ea7c700 1 -- 172.21.15.42:0/2180570443 wait complete. 2019-12-05T05:49:05.229 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:05.217+0000 7f490ea7c700 1 -- 172.21.15.42:0/2180570443 >> 172.21.15.42:0/2180570443 conn(0x7f49080aabe0 msgr2=0x7f49080a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:49:05.232 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:05.221+0000 7f490ea7c700 1 Processor -- start 2019-12-05T05:49:05.232 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:05.221+0000 7f490ea7c700 1 -- start start 2019-12-05T05:49:05.232 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:05.221+0000 7f490ea7c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f49080a6bf0 0x7f49081270d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:49:05.233 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:05.221+0000 7f490ea7c700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4908127620 con 0x7f49080a51f0 2019-12-05T05:49:05.233 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:05.221+0000 7f490ea7c700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f490807e120 con 0x7f4908126ca0 2019-12-05T05:49:05.233 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:05.221+0000 7f490ea7c700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4908127910 con 0x7f49080a6bf0 2019-12-05T05:49:05.233 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:05.221+0000 7f490c818700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f4908126ca0 0x7f49080a63f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:41792/0 (socket says 172.21.15.42:41792) 2019-12-05T05:49:05.233 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:05.221+0000 7f490c818700 1 -- 172.21.15.42:0/647333823 learned_addr learned my addr 172.21.15.42:0/647333823 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:49:05.233 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:05.221+0000 7f490d81a700 1 --2- 172.21.15.42:0/647333823 >> v2:172.21.15.42:3300/0 conn(0x7f49080a6bf0 0x7f49081270d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:49:05.233 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:05.221+0000 7f490d81a700 1 -- 172.21.15.42:0/647333823 >> v1:172.21.15.42:6789/0 conn(0x7f4908126ca0 legacy=0x7f49080a63f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:49:05.234 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:05.221+0000 7f4905ffb700 1 -- 172.21.15.42:0/647333823 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1782171252 0 0) 0x7f4908127620 con 0x7f49080a51f0 2019-12-05T05:49:05.234 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:05.221+0000 7f490d81a700 1 -- 172.21.15.42:0/647333823 >> v1:172.21.15.40:6789/0 conn(0x7f49080a51f0 legacy=0x7f490807d920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:49:05.234 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:05.221+0000 7f490d81a700 1 -- 172.21.15.42:0/647333823 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4908134560 con 0x7f49080a6bf0 2019-12-05T05:49:05.234 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:05.221+0000 7f490d81a700 1 --2- 172.21.15.42:0/647333823 >> v2:172.21.15.42:3300/0 conn(0x7f49080a6bf0 0x7f49081270d0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f48f4001ec0 tx=0x7f48f4002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:05.234 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:05.221+0000 7f490d81a700 1 --2- 172.21.15.42:0/647333823 >> v2:172.21.15.42:3300/0 conn(0x7f49080a6bf0 0x7f49081270d0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f48f4001ec0 tx=0x7f48f4002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:05.234 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:05.225+0000 7f490d81a700 1 --2- 172.21.15.42:0/647333823 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f49080a6bf0 0x7f49081270d0 secure :-1 s=READY pgs=1499 cs=0 l=1 rx=0x7f48f4001ec0 tx=0x7f48f4002ff0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:49:05.234 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:05.225+0000 7f490d81a700 1 --2- 172.21.15.42:0/647333823 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f49080a6bf0 0x7f49081270d0 secure :-1 s=THROTTLE_DONE pgs=1499 cs=0 l=1 rx=0x7f48f4001ec0 tx=0x7f48f4002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:05.235 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:05.225+0000 7f4905ffb700 1 -- 172.21.15.42:0/647333823 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4900006b30 con 0x7f49080a6bf0 2019-12-05T05:49:05.235 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:05.225+0000 7f490d81a700 1 --2- 172.21.15.42:0/647333823 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f49080a6bf0 0x7f49081270d0 secure :-1 s=THROTTLE_DONE pgs=1499 cs=0 l=1 rx=0x7f48f4001ec0 tx=0x7f48f4002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:05.235 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:05.225+0000 7f490d81a700 1 --2- 172.21.15.42:0/647333823 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f49080a6bf0 0x7f49081270d0 secure :-1 s=THROTTLE_DONE pgs=1499 cs=0 l=1 rx=0x7f48f4001ec0 tx=0x7f48f4002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:05.235 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:05.225+0000 7f4905ffb700 1 -- 172.21.15.42:0/647333823 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4900006750 con 0x7f49080a6bf0 2019-12-05T05:49:05.235 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:05.225+0000 7f490ea7c700 1 -- 172.21.15.42:0/647333823 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4908132540 con 0x7f49080a6bf0 2019-12-05T05:49:05.235 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:05.225+0000 7f4905ffb700 1 -- 172.21.15.42:0/647333823 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4900010e40 con 0x7f49080a6bf0 2019-12-05T05:49:05.236 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:05.225+0000 7f490ea7c700 1 -- 172.21.15.42:0/647333823 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4908132af0 con 0x7f49080a6bf0 2019-12-05T05:49:05.236 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:05.225+0000 7f490d81a700 1 --2- 172.21.15.42:0/647333823 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f49080a6bf0 0x7f49081270d0 secure :-1 s=THROTTLE_DONE pgs=1499 cs=0 l=1 rx=0x7f48f4001ec0 tx=0x7f48f4002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:05.236 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:05.225+0000 7f4905ffb700 1 -- 172.21.15.42:0/647333823 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f4900011150 con 0x7f49080a6bf0 2019-12-05T05:49:05.236 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:05.225+0000 7f490d81a700 1 --2- 172.21.15.42:0/647333823 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f49080a6bf0 0x7f49081270d0 secure :-1 s=THROTTLE_DONE pgs=1499 cs=0 l=1 rx=0x7f48f4001ec0 tx=0x7f48f4002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:05.236 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:05.225+0000 7f4905ffb700 1 --2- 172.21.15.42:0/647333823 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f48e803abe0 0x7f48e803d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:49:05.236 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:05.225+0000 7f4905ffb700 1 -- 172.21.15.42:0/647333823 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 8611+0+0 (secure 0 0 0) 0x7f490004afe0 con 0x7f49080a6bf0 2019-12-05T05:49:05.236 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:05.225+0000 7f490d019700 1 --2- 172.21.15.42:0/647333823 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f48e803abe0 0x7f48e803d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:49:05.237 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:05.225+0000 7f490d019700 1 --2- 172.21.15.42:0/647333823 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f48e803abe0 0x7f48e803d0f0 crc :-1 s=READY pgs=645 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:49:05.237 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:05.229+0000 7f490ea7c700 1 -- 172.21.15.42:0/647333823 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f490805d110 con 0x7f49080a6bf0 2019-12-05T05:49:05.240 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:05.237+0000 7f490d81a700 1 --2- 172.21.15.42:0/647333823 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f49080a6bf0 0x7f49081270d0 secure :-1 s=THROTTLE_DONE pgs=1499 cs=0 l=1 rx=0x7f48f4001ec0 tx=0x7f48f4002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:05.240 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:05.237+0000 7f4905ffb700 1 -- 172.21.15.42:0/647333823 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f4900032060 con 0x7f49080a6bf0 2019-12-05T05:49:05.404 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:05.401+0000 7f490ea7c700 1 -- 172.21.15.42:0/647333823 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f490805b8c0 con 0x7f49080a6bf0 2019-12-05T05:49:05.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:05.401+0000 7f490d81a700 1 --2- 172.21.15.42:0/647333823 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f49080a6bf0 0x7f49081270d0 secure :-1 s=THROTTLE_DONE pgs=1499 cs=0 l=1 rx=0x7f48f4001ec0 tx=0x7f48f4002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:05.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:05.401+0000 7f4905ffb700 1 -- 172.21.15.42:0/647333823 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 183 v183) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f49000322c0 con 0x7f49080a6bf0 2019-12-05T05:49:05.407 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 183 2019-12-05T05:49:05.407 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:49:05.408 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":183,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":8188,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6835/2456922143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2456922143},{"type":"v1","addr":"172.21.15.42:6835","nonce":2456922143}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":183}],"filesystems":[{"mdsmap":{"epoch":182,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:47:19.499092+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6140},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6140":{"gid":6140,"name":"b","rank":0,"incarnation":179,"state":"up:active","state_seq":26,"addr":"172.21.15.40:6833/2146240383","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2146240383},{"type":"v1","addr":"172.21.15.40:6833","nonce":2146240383}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:49:05.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:05.405+0000 7f490ea7c700 1 -- 172.21.15.42:0/647333823 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f48e803abe0 msgr2=0x7f48e803d0f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:49:05.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:05.405+0000 7f490ea7c700 1 --2- 172.21.15.42:0/647333823 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f48e803abe0 0x7f48e803d0f0 crc :-1 s=READY pgs=645 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:49:05.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:05.405+0000 7f490ea7c700 1 -- 172.21.15.42:0/647333823 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f49080a6bf0 msgr2=0x7f49081270d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:49:05.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:05.405+0000 7f490ea7c700 1 --2- 172.21.15.42:0/647333823 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f49080a6bf0 0x7f49081270d0 secure :-1 s=READY pgs=1499 cs=0 l=1 rx=0x7f48f4001ec0 tx=0x7f48f4002ff0).stop 2019-12-05T05:49:05.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:05.405+0000 7f490ea7c700 1 -- 172.21.15.42:0/647333823 shutdown_connections 2019-12-05T05:49:05.411 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:05.405+0000 7f490ea7c700 1 --2- 172.21.15.42:0/647333823 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f48e803abe0 0x7f48e803d0f0 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:49:05.411 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:05.405+0000 7f490ea7c700 1 --2- 172.21.15.42:0/647333823 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f49080a6bf0 0x7f49081270d0 unknown :-1 s=CLOSED pgs=1499 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:49:05.411 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:05.405+0000 7f490ea7c700 1 -- 172.21.15.42:0/647333823 shutdown_connections 2019-12-05T05:49:05.411 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:05.405+0000 7f490ea7c700 1 -- 172.21.15.42:0/647333823 wait complete. 2019-12-05T05:49:05.411 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:05.405+0000 7f490ea7c700 1 -- 172.21.15.42:0/647333823 >> 172.21.15.42:0/647333823 conn(0x7f490814eb90 msgr2=0x7f49080a9720 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:49:05.420 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T05:49:07.421 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:49:07.530 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:07.521+0000 7f3d755b7700 1 Processor -- start 2019-12-05T05:49:07.530 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:07.521+0000 7f3d755b7700 1 -- start start 2019-12-05T05:49:07.530 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:07.521+0000 7f3d755b7700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f3d7008df80 0x7f3d7009f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:49:07.530 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:07.521+0000 7f3d755b7700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d7009fc60 con 0x7f3d7009f830 2019-12-05T05:49:07.531 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:07.521+0000 7f3d755b7700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d700a0710 con 0x7f3d700a7740 2019-12-05T05:49:07.531 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:07.521+0000 7f3d755b7700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d700a0b00 con 0x7f3d7008df80 2019-12-05T05:49:07.531 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:07.521+0000 7f3d6f7fe700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f3d700a7740 0x7f3d7007e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:41800/0 (socket says 172.21.15.42:41800) 2019-12-05T05:49:07.531 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:07.521+0000 7f3d6f7fe700 1 -- 172.21.15.42:0/467377670 learned_addr learned my addr 172.21.15.42:0/467377670 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:49:07.531 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:07.521+0000 7f3d6effd700 1 --2- 172.21.15.42:0/467377670 >> v2:172.21.15.40:3300/0 conn(0x7f3d7008df80 0x7f3d7009f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:49:07.531 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:07.521+0000 7f3d6e7fc700 1 -- 172.21.15.42:0/467377670 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3425095052 0 0) 0x7f3d700a0710 con 0x7f3d700a7740 2019-12-05T05:49:07.531 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:07.521+0000 7f3d6e7fc700 1 -- 172.21.15.42:0/467377670 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d5c0030b0 con 0x7f3d700a7740 2019-12-05T05:49:07.532 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:07.521+0000 7f3d6effd700 1 -- 172.21.15.42:0/467377670 >> v1:172.21.15.42:6789/0 conn(0x7f3d700a7740 legacy=0x7f3d7007e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:49:07.532 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:07.521+0000 7f3d6effd700 1 -- 172.21.15.42:0/467377670 >> v1:172.21.15.40:6789/0 conn(0x7f3d7009f830 legacy=0x7f3d7014efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:49:07.532 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:07.521+0000 7f3d6effd700 1 -- 172.21.15.42:0/467377670 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d7007ef20 con 0x7f3d7008df80 2019-12-05T05:49:07.532 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:07.521+0000 7f3d6effd700 1 --2- 172.21.15.42:0/467377670 >> v2:172.21.15.40:3300/0 conn(0x7f3d7008df80 0x7f3d7009f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f3d60008b10 tx=0x7f3d6000ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:07.532 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:07.525+0000 7f3d6effd700 1 --2- 172.21.15.42:0/467377670 >> v2:172.21.15.40:3300/0 conn(0x7f3d7008df80 0x7f3d7009f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f3d60008b10 tx=0x7f3d6000ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:07.532 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:07.525+0000 7f3d6effd700 1 --2- 172.21.15.42:0/467377670 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3d7008df80 0x7f3d7009f2e0 secure :-1 s=READY pgs=1157 cs=0 l=1 rx=0x7f3d60008b10 tx=0x7f3d6000ff70).ready entity=mon.0 client_cookie=757b17a4f2b00572 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:49:07.533 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:07.525+0000 7f3d6effd700 1 --2- 172.21.15.42:0/467377670 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3d7008df80 0x7f3d7009f2e0 secure :-1 s=THROTTLE_DONE pgs=1157 cs=0 l=1 rx=0x7f3d60008b10 tx=0x7f3d6000ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:07.533 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:07.525+0000 7f3d6e7fc700 1 -- 172.21.15.42:0/467377670 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3d600119d0 con 0x7f3d7008df80 2019-12-05T05:49:07.533 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:07.525+0000 7f3d6effd700 1 --2- 172.21.15.42:0/467377670 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3d7008df80 0x7f3d7009f2e0 secure :-1 s=THROTTLE_DONE pgs=1157 cs=0 l=1 rx=0x7f3d60008b10 tx=0x7f3d6000ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:07.533 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:07.525+0000 7f3d6effd700 1 --2- 172.21.15.42:0/467377670 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3d7008df80 0x7f3d7009f2e0 secure :-1 s=THROTTLE_DONE pgs=1157 cs=0 l=1 rx=0x7f3d60008b10 tx=0x7f3d6000ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:07.533 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:07.525+0000 7f3d6e7fc700 1 -- 172.21.15.42:0/467377670 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d600115f0 con 0x7f3d7008df80 2019-12-05T05:49:07.533 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:07.525+0000 7f3d6e7fc700 1 -- 172.21.15.42:0/467377670 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3d6000a160 con 0x7f3d7008df80 2019-12-05T05:49:07.534 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:07.525+0000 7f3d755b7700 1 -- 172.21.15.42:0/467377670 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3d7008df80 msgr2=0x7f3d7009f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:49:07.534 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:07.525+0000 7f3d755b7700 1 --2- 172.21.15.42:0/467377670 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3d7008df80 0x7f3d7009f2e0 secure :-1 s=READY pgs=1157 cs=0 l=1 rx=0x7f3d60008b10 tx=0x7f3d6000ff70).stop 2019-12-05T05:49:07.534 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:07.525+0000 7f3d755b7700 1 -- 172.21.15.42:0/467377670 shutdown_connections 2019-12-05T05:49:07.534 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:07.525+0000 7f3d755b7700 1 --2- 172.21.15.42:0/467377670 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3d7008df80 0x7f3d7009f2e0 unknown :-1 s=CLOSED pgs=1157 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:49:07.534 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:07.525+0000 7f3d755b7700 1 -- 172.21.15.42:0/467377670 shutdown_connections 2019-12-05T05:49:07.534 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:07.525+0000 7f3d755b7700 1 -- 172.21.15.42:0/467377670 wait complete. 2019-12-05T05:49:07.534 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:07.525+0000 7f3d755b7700 1 -- 172.21.15.42:0/467377670 >> 172.21.15.42:0/467377670 conn(0x7f3d700aabe0 msgr2=0x7f3d700a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:49:07.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:07.529+0000 7f3d755b7700 1 Processor -- start 2019-12-05T05:49:07.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:07.529+0000 7f3d755b7700 1 -- start start 2019-12-05T05:49:07.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:07.529+0000 7f3d755b7700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f3d70116530 0x7f3d70119370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:49:07.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:07.529+0000 7f3d755b7700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3d70122150 0x7f3d70122580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:49:07.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:07.529+0000 7f3d755b7700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d7011c500 con 0x7f3d7011b8d0 2019-12-05T05:49:07.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:07.529+0000 7f3d755b7700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d70123a30 con 0x7f3d70116530 2019-12-05T05:49:07.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:07.529+0000 7f3d755b7700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d70123c90 con 0x7f3d70122150 2019-12-05T05:49:07.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:07.529+0000 7f3d6effd700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3d70122150 0x7f3d70122580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:49:07.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:07.529+0000 7f3d6effd700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3d70122150 0x7f3d70122580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:41770/0 (socket says 172.21.15.42:41770) 2019-12-05T05:49:07.539 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:07.529+0000 7f3d6effd700 1 -- 172.21.15.42:0/748697434 learned_addr learned my addr 172.21.15.42:0/748697434 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:49:07.539 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:07.529+0000 7f3d6f7fe700 1 --2- 172.21.15.42:0/748697434 >> v2:172.21.15.40:3300/0 conn(0x7f3d70116530 0x7f3d70119370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:49:07.539 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:07.529+0000 7f3d6effd700 1 -- 172.21.15.42:0/748697434 >> v2:172.21.15.40:3300/0 conn(0x7f3d70116530 msgr2=0x7f3d70119370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:49:07.539 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:07.529+0000 7f3d6effd700 1 --2- 172.21.15.42:0/748697434 >> v2:172.21.15.40:3300/0 conn(0x7f3d70116530 0x7f3d70119370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:49:07.539 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:07.529+0000 7f3d67fff700 1 -- 172.21.15.42:0/748697434 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2646645082 0 0) 0x7f3d7011c500 con 0x7f3d7011b8d0 2019-12-05T05:49:07.539 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:07.529+0000 7f3d6effd700 1 -- 172.21.15.42:0/748697434 >> v1:172.21.15.40:6789/0 conn(0x7f3d7011b8d0 legacy=0x7f3d7011bd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:49:07.539 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:07.529+0000 7f3d6effd700 1 -- 172.21.15.42:0/748697434 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d7008f730 con 0x7f3d70122150 2019-12-05T05:49:07.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:07.529+0000 7f3d6effd700 1 --2- 172.21.15.42:0/748697434 >> v2:172.21.15.42:3300/0 conn(0x7f3d70122150 0x7f3d70122580 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f3d60008b10 tx=0x7f3d60011f00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:07.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:07.529+0000 7f3d6effd700 1 --2- 172.21.15.42:0/748697434 >> v2:172.21.15.42:3300/0 conn(0x7f3d70122150 0x7f3d70122580 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f3d60008b10 tx=0x7f3d60011f00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:07.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:07.529+0000 7f3d6effd700 1 --2- 172.21.15.42:0/748697434 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3d70122150 0x7f3d70122580 secure :-1 s=READY pgs=1501 cs=0 l=1 rx=0x7f3d60008b10 tx=0x7f3d60011f00).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:49:07.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:07.529+0000 7f3d6effd700 1 --2- 172.21.15.42:0/748697434 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3d70122150 0x7f3d70122580 secure :-1 s=THROTTLE_DONE pgs=1501 cs=0 l=1 rx=0x7f3d60008b10 tx=0x7f3d60011f00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:07.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:07.529+0000 7f3d67fff700 1 -- 172.21.15.42:0/748697434 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3d6000f8c0 con 0x7f3d70122150 2019-12-05T05:49:07.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:07.529+0000 7f3d6effd700 1 --2- 172.21.15.42:0/748697434 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3d70122150 0x7f3d70122580 secure :-1 s=THROTTLE_DONE pgs=1501 cs=0 l=1 rx=0x7f3d60008b10 tx=0x7f3d60011f00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:07.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:07.529+0000 7f3d6effd700 1 --2- 172.21.15.42:0/748697434 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3d70122150 0x7f3d70122580 secure :-1 s=THROTTLE_DONE pgs=1501 cs=0 l=1 rx=0x7f3d60008b10 tx=0x7f3d60011f00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:07.541 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:07.529+0000 7f3d67fff700 1 -- 172.21.15.42:0/748697434 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d60011bd0 con 0x7f3d70122150 2019-12-05T05:49:07.541 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:07.529+0000 7f3d755b7700 1 -- 172.21.15.42:0/748697434 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d701337a0 con 0x7f3d70122150 2019-12-05T05:49:07.541 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:07.529+0000 7f3d67fff700 1 -- 172.21.15.42:0/748697434 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3d60005ca0 con 0x7f3d70122150 2019-12-05T05:49:07.541 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:07.529+0000 7f3d755b7700 1 -- 172.21.15.42:0/748697434 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d701198c0 con 0x7f3d70122150 2019-12-05T05:49:07.541 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:07.533+0000 7f3d6effd700 1 --2- 172.21.15.42:0/748697434 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3d70122150 0x7f3d70122580 secure :-1 s=THROTTLE_DONE pgs=1501 cs=0 l=1 rx=0x7f3d60008b10 tx=0x7f3d60011f00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:07.541 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:07.533+0000 7f3d67fff700 1 -- 172.21.15.42:0/748697434 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f3d60007150 con 0x7f3d70122150 2019-12-05T05:49:07.541 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:07.533+0000 7f3d6effd700 1 --2- 172.21.15.42:0/748697434 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3d70122150 0x7f3d70122580 secure :-1 s=THROTTLE_DONE pgs=1501 cs=0 l=1 rx=0x7f3d60008b10 tx=0x7f3d60011f00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:07.542 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:07.533+0000 7f3d67fff700 1 --2- 172.21.15.42:0/748697434 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3d5403ad20 0x7f3d5403d230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:49:07.542 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:07.533+0000 7f3d67fff700 1 -- 172.21.15.42:0/748697434 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 8611+0+0 (secure 0 0 0) 0x7f3d6004b5a0 con 0x7f3d70122150 2019-12-05T05:49:07.542 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:07.533+0000 7f3d6f7fe700 1 --2- 172.21.15.42:0/748697434 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3d5403ad20 0x7f3d5403d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:49:07.542 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:07.533+0000 7f3d6f7fe700 1 --2- 172.21.15.42:0/748697434 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3d5403ad20 0x7f3d5403d230 crc :-1 s=READY pgs=646 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:49:07.543 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:07.537+0000 7f3d755b7700 1 -- 172.21.15.42:0/748697434 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d7005d110 con 0x7f3d70122150 2019-12-05T05:49:07.549 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:07.545+0000 7f3d6effd700 1 --2- 172.21.15.42:0/748697434 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3d70122150 0x7f3d70122580 secure :-1 s=THROTTLE_DONE pgs=1501 cs=0 l=1 rx=0x7f3d60008b10 tx=0x7f3d60011f00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:07.550 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:07.545+0000 7f3d67fff700 1 -- 172.21.15.42:0/748697434 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f3d60030260 con 0x7f3d70122150 2019-12-05T05:49:07.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:07.713+0000 7f3d755b7700 1 -- 172.21.15.42:0/748697434 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3d7005b8c0 con 0x7f3d70122150 2019-12-05T05:49:07.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:07.717+0000 7f3d6effd700 1 --2- 172.21.15.42:0/748697434 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3d70122150 0x7f3d70122580 secure :-1 s=THROTTLE_DONE pgs=1501 cs=0 l=1 rx=0x7f3d60008b10 tx=0x7f3d60011f00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:07.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:07.717+0000 7f3d67fff700 1 -- 172.21.15.42:0/748697434 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 183 v183) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f3d60033430 con 0x7f3d70122150 2019-12-05T05:49:07.720 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 183 2019-12-05T05:49:07.720 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:49:07.721 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":183,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":8188,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6835/2456922143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2456922143},{"type":"v1","addr":"172.21.15.42:6835","nonce":2456922143}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":183}],"filesystems":[{"mdsmap":{"epoch":182,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:47:19.499092+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6140},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6140":{"gid":6140,"name":"b","rank":0,"incarnation":179,"state":"up:active","state_seq":26,"addr":"172.21.15.40:6833/2146240383","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2146240383},{"type":"v1","addr":"172.21.15.40:6833","nonce":2146240383}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:49:07.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:07.717+0000 7f3d755b7700 1 -- 172.21.15.42:0/748697434 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3d5403ad20 msgr2=0x7f3d5403d230 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:49:07.724 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:07.717+0000 7f3d755b7700 1 --2- 172.21.15.42:0/748697434 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3d5403ad20 0x7f3d5403d230 crc :-1 s=READY pgs=646 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:49:07.724 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:07.717+0000 7f3d755b7700 1 -- 172.21.15.42:0/748697434 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3d70122150 msgr2=0x7f3d70122580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:49:07.724 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:07.717+0000 7f3d755b7700 1 --2- 172.21.15.42:0/748697434 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3d70122150 0x7f3d70122580 secure :-1 s=READY pgs=1501 cs=0 l=1 rx=0x7f3d60008b10 tx=0x7f3d60011f00).stop 2019-12-05T05:49:07.724 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:07.717+0000 7f3d755b7700 1 -- 172.21.15.42:0/748697434 shutdown_connections 2019-12-05T05:49:07.724 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:07.717+0000 7f3d755b7700 1 --2- 172.21.15.42:0/748697434 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3d5403ad20 0x7f3d5403d230 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:49:07.725 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:07.717+0000 7f3d755b7700 1 --2- 172.21.15.42:0/748697434 >> v2:172.21.15.40:3300/0 conn(0x7f3d70116530 0x7f3d70119370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:49:07.725 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:07.717+0000 7f3d755b7700 1 --2- 172.21.15.42:0/748697434 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3d70122150 0x7f3d70122580 unknown :-1 s=CLOSED pgs=1501 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:49:07.725 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:07.717+0000 7f3d755b7700 1 -- 172.21.15.42:0/748697434 shutdown_connections 2019-12-05T05:49:07.725 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:07.717+0000 7f3d755b7700 1 -- 172.21.15.42:0/748697434 wait complete. 2019-12-05T05:49:07.725 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:07.717+0000 7f3d755b7700 1 -- 172.21.15.42:0/748697434 >> 172.21.15.42:0/748697434 conn(0x7f3d7008fda0 msgr2=0x7f3d700a9770 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:49:07.732 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T05:49:09.733 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:49:10.014 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:10.005+0000 7f463860d700 1 Processor -- start 2019-12-05T05:49:10.014 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:10.005+0000 7f463860d700 1 -- start start 2019-12-05T05:49:10.014 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:10.005+0000 7f463860d700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f463008df80 0x7f463009f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:49:10.015 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:10.005+0000 7f463860d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f463009f830 0x7f463014efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:49:10.015 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:10.005+0000 7f463860d700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f463009fc60 con 0x7f46300a7740 2019-12-05T05:49:10.015 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:10.005+0000 7f463860d700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46300a0b00 con 0x7f463008df80 2019-12-05T05:49:10.015 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:10.005+0000 7f463860d700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f463008e5b0 con 0x7f463009f830 2019-12-05T05:49:10.015 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:10.005+0000 7f46373ab700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f463009f830 0x7f463014efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:49:10.016 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:10.005+0000 7f46373ab700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f463009f830 0x7f463014efe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:41782/0 (socket says 172.21.15.42:41782) 2019-12-05T05:49:10.016 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:10.005+0000 7f46373ab700 1 -- 172.21.15.42:0/4183535000 learned_addr learned my addr 172.21.15.42:0/4183535000 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:49:10.016 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:10.005+0000 7f4636baa700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f46300a7740 0x7f463007e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:41816/0 (socket says 172.21.15.42:41816) 2019-12-05T05:49:10.016 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:10.005+0000 7f46363a9700 1 --2- 172.21.15.42:0/4183535000 >> v2:172.21.15.40:3300/0 conn(0x7f463008df80 0x7f463009f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:49:10.016 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:10.009+0000 7f46373ab700 1 -- 172.21.15.42:0/4183535000 >> v2:172.21.15.40:3300/0 conn(0x7f463008df80 msgr2=0x7f463009f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:49:10.017 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:10.009+0000 7f46373ab700 1 --2- 172.21.15.42:0/4183535000 >> v2:172.21.15.40:3300/0 conn(0x7f463008df80 0x7f463009f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:49:10.017 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:10.009+0000 7f4635ba8700 1 -- 172.21.15.42:0/4183535000 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2429740630 0 0) 0x7f463009fc60 con 0x7f46300a7740 2019-12-05T05:49:10.017 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:10.009+0000 7f46373ab700 1 -- 172.21.15.42:0/4183535000 >> v1:172.21.15.42:6789/0 conn(0x7f46300a7740 legacy=0x7f463007e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:49:10.017 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:10.009+0000 7f46373ab700 1 -- 172.21.15.42:0/4183535000 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46300a0710 con 0x7f463009f830 2019-12-05T05:49:10.017 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:10.009+0000 7f46373ab700 1 --2- 172.21.15.42:0/4183535000 >> v2:172.21.15.42:3300/0 conn(0x7f463009f830 0x7f463014efe0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f46280032b0 tx=0x7f46280100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:10.017 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:10.009+0000 7f46373ab700 1 --2- 172.21.15.42:0/4183535000 >> v2:172.21.15.42:3300/0 conn(0x7f463009f830 0x7f463014efe0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f46280032b0 tx=0x7f46280100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:10.017 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:10.009+0000 7f46373ab700 1 --2- 172.21.15.42:0/4183535000 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f463009f830 0x7f463014efe0 secure :-1 s=READY pgs=1503 cs=0 l=1 rx=0x7f46280032b0 tx=0x7f46280100b0).ready entity=mon.1 client_cookie=40a6156aacf8f932 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:49:10.018 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:10.009+0000 7f46373ab700 1 --2- 172.21.15.42:0/4183535000 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f463009f830 0x7f463014efe0 secure :-1 s=THROTTLE_DONE pgs=1503 cs=0 l=1 rx=0x7f46280032b0 tx=0x7f46280100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:10.018 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:10.009+0000 7f4635ba8700 1 -- 172.21.15.42:0/4183535000 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4628010d60 con 0x7f463009f830 2019-12-05T05:49:10.018 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:10.009+0000 7f46373ab700 1 --2- 172.21.15.42:0/4183535000 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f463009f830 0x7f463014efe0 secure :-1 s=THROTTLE_DONE pgs=1503 cs=0 l=1 rx=0x7f46280032b0 tx=0x7f46280100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:10.018 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:10.009+0000 7f46373ab700 1 --2- 172.21.15.42:0/4183535000 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f463009f830 0x7f463014efe0 secure :-1 s=THROTTLE_DONE pgs=1503 cs=0 l=1 rx=0x7f46280032b0 tx=0x7f46280100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:10.018 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:10.009+0000 7f4635ba8700 1 -- 172.21.15.42:0/4183535000 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4628010980 con 0x7f463009f830 2019-12-05T05:49:10.018 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:10.009+0000 7f4635ba8700 1 -- 172.21.15.42:0/4183535000 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f46280114a0 con 0x7f463009f830 2019-12-05T05:49:10.018 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:10.009+0000 7f463860d700 1 -- 172.21.15.42:0/4183535000 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f463009f830 msgr2=0x7f463014efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:49:10.019 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:10.009+0000 7f463860d700 1 --2- 172.21.15.42:0/4183535000 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f463009f830 0x7f463014efe0 secure :-1 s=READY pgs=1503 cs=0 l=1 rx=0x7f46280032b0 tx=0x7f46280100b0).stop 2019-12-05T05:49:10.019 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:10.009+0000 7f463860d700 1 -- 172.21.15.42:0/4183535000 shutdown_connections 2019-12-05T05:49:10.019 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:10.009+0000 7f463860d700 1 --2- 172.21.15.42:0/4183535000 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f463009f830 0x7f463014efe0 unknown :-1 s=CLOSED pgs=1503 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:49:10.019 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:10.009+0000 7f463860d700 1 --2- 172.21.15.42:0/4183535000 >> v2:172.21.15.40:3300/0 conn(0x7f463008df80 0x7f463009f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:49:10.019 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:10.009+0000 7f463860d700 1 -- 172.21.15.42:0/4183535000 shutdown_connections 2019-12-05T05:49:10.019 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:10.009+0000 7f463860d700 1 -- 172.21.15.42:0/4183535000 wait complete. 2019-12-05T05:49:10.019 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:10.009+0000 7f463860d700 1 -- 172.21.15.42:0/4183535000 >> 172.21.15.42:0/4183535000 conn(0x7f46300aabe0 msgr2=0x7f46300a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:49:10.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:10.009+0000 7f463860d700 1 Processor -- start 2019-12-05T05:49:10.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:10.009+0000 7f463860d700 1 -- start start 2019-12-05T05:49:10.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:10.013+0000 7f463860d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f4630141c00 0x7f4630076af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:49:10.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:10.013+0000 7f463860d700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4630142030 con 0x7f4630116500 2019-12-05T05:49:10.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:10.013+0000 7f463860d700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46301191c0 con 0x7f46301417d0 2019-12-05T05:49:10.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:10.013+0000 7f463860d700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4630119540 con 0x7f4630141c00 2019-12-05T05:49:10.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:10.013+0000 7f46373ab700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f4630141c00 0x7f4630076af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:49:10.024 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:10.013+0000 7f46363a9700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f46301417d0 0x7f46300732d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:41826/0 (socket says 172.21.15.42:41826) 2019-12-05T05:49:10.024 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:10.013+0000 7f46363a9700 1 -- 172.21.15.42:0/692310859 learned_addr learned my addr 172.21.15.42:0/692310859 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:49:10.024 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:10.013+0000 7f46277fe700 1 -- 172.21.15.42:0/692310859 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 844737801 0 0) 0x7f4630142030 con 0x7f4630116500 2019-12-05T05:49:10.024 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:10.013+0000 7f46373ab700 1 -- 172.21.15.42:0/692310859 >> v1:172.21.15.42:6789/0 conn(0x7f46301417d0 legacy=0x7f46300732d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:49:10.024 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:10.013+0000 7f46373ab700 1 -- 172.21.15.42:0/692310859 >> v1:172.21.15.40:6789/0 conn(0x7f4630116500 legacy=0x7f463012fe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:49:10.026 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:10.013+0000 7f46373ab700 1 -- 172.21.15.42:0/692310859 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4630124fa0 con 0x7f4630141c00 2019-12-05T05:49:10.026 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:10.013+0000 7f46373ab700 1 --2- 172.21.15.42:0/692310859 >> v2:172.21.15.42:3300/0 conn(0x7f4630141c00 0x7f4630076af0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f46280032b0 tx=0x7f4628010c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:10.026 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:10.013+0000 7f46373ab700 1 --2- 172.21.15.42:0/692310859 >> v2:172.21.15.42:3300/0 conn(0x7f4630141c00 0x7f4630076af0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f46280032b0 tx=0x7f4628010c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:10.026 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:10.013+0000 7f46373ab700 1 --2- 172.21.15.42:0/692310859 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4630141c00 0x7f4630076af0 secure :-1 s=READY pgs=1505 cs=0 l=1 rx=0x7f46280032b0 tx=0x7f4628010c30).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:49:10.026 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:10.013+0000 7f46373ab700 1 --2- 172.21.15.42:0/692310859 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4630141c00 0x7f4630076af0 secure :-1 s=THROTTLE_DONE pgs=1505 cs=0 l=1 rx=0x7f46280032b0 tx=0x7f4628010c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:10.026 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:10.013+0000 7f46277fe700 1 -- 172.21.15.42:0/692310859 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4628011bb0 con 0x7f4630141c00 2019-12-05T05:49:10.026 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:10.013+0000 7f46373ab700 1 --2- 172.21.15.42:0/692310859 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4630141c00 0x7f4630076af0 secure :-1 s=THROTTLE_DONE pgs=1505 cs=0 l=1 rx=0x7f46280032b0 tx=0x7f4628010c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:10.027 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:10.013+0000 7f46373ab700 1 --2- 172.21.15.42:0/692310859 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4630141c00 0x7f4630076af0 secure :-1 s=THROTTLE_DONE pgs=1505 cs=0 l=1 rx=0x7f46280032b0 tx=0x7f4628010c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:10.027 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:10.013+0000 7f46277fe700 1 -- 172.21.15.42:0/692310859 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46280110c0 con 0x7f4630141c00 2019-12-05T05:49:10.027 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:10.013+0000 7f463860d700 1 -- 172.21.15.42:0/692310859 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46301234c0 con 0x7f4630141c00 2019-12-05T05:49:10.027 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:10.013+0000 7f46277fe700 1 -- 172.21.15.42:0/692310859 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4628005ab0 con 0x7f4630141c00 2019-12-05T05:49:10.027 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:10.013+0000 7f463860d700 1 -- 172.21.15.42:0/692310859 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4630119d20 con 0x7f4630141c00 2019-12-05T05:49:10.027 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:10.013+0000 7f46373ab700 1 --2- 172.21.15.42:0/692310859 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4630141c00 0x7f4630076af0 secure :-1 s=THROTTLE_DONE pgs=1505 cs=0 l=1 rx=0x7f46280032b0 tx=0x7f4628010c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:10.027 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:10.013+0000 7f46277fe700 1 -- 172.21.15.42:0/692310859 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f462800f280 con 0x7f4630141c00 2019-12-05T05:49:10.028 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:10.013+0000 7f46373ab700 1 --2- 172.21.15.42:0/692310859 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4630141c00 0x7f4630076af0 secure :-1 s=THROTTLE_DONE pgs=1505 cs=0 l=1 rx=0x7f46280032b0 tx=0x7f4628010c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:10.028 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:10.013+0000 7f46277fe700 1 --2- 172.21.15.42:0/692310859 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f461803aca0 0x7f461803d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:49:10.028 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:10.013+0000 7f46277fe700 1 -- 172.21.15.42:0/692310859 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 8611+0+0 (secure 0 0 0) 0x7f462804b4b0 con 0x7f4630141c00 2019-12-05T05:49:10.028 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:10.013+0000 7f4636baa700 1 --2- 172.21.15.42:0/692310859 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f461803aca0 0x7f461803d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:49:10.028 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:10.013+0000 7f4636baa700 1 --2- 172.21.15.42:0/692310859 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f461803aca0 0x7f461803d1b0 crc :-1 s=READY pgs=647 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:49:10.028 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:10.017+0000 7f463860d700 1 -- 172.21.15.42:0/692310859 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f463005d110 con 0x7f4630141c00 2019-12-05T05:49:10.029 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:10.025+0000 7f46373ab700 1 --2- 172.21.15.42:0/692310859 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4630141c00 0x7f4630076af0 secure :-1 s=THROTTLE_DONE pgs=1505 cs=0 l=1 rx=0x7f46280032b0 tx=0x7f4628010c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:10.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:10.025+0000 7f46277fe700 1 -- 172.21.15.42:0/692310859 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f4628005dc0 con 0x7f4630141c00 2019-12-05T05:49:10.194 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:10.189+0000 7f463860d700 1 -- 172.21.15.42:0/692310859 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f463005b8c0 con 0x7f4630141c00 2019-12-05T05:49:10.195 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:10.193+0000 7f46373ab700 1 --2- 172.21.15.42:0/692310859 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4630141c00 0x7f4630076af0 secure :-1 s=THROTTLE_DONE pgs=1505 cs=0 l=1 rx=0x7f46280032b0 tx=0x7f4628010c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:10.196 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:10.193+0000 7f46277fe700 1 -- 172.21.15.42:0/692310859 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 183 v183) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f46280301c0 con 0x7f4630141c00 2019-12-05T05:49:10.196 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 183 2019-12-05T05:49:10.196 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:49:10.196 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":183,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":8188,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6835/2456922143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2456922143},{"type":"v1","addr":"172.21.15.42:6835","nonce":2456922143}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":183}],"filesystems":[{"mdsmap":{"epoch":182,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:47:19.499092+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6140},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6140":{"gid":6140,"name":"b","rank":0,"incarnation":179,"state":"up:active","state_seq":26,"addr":"172.21.15.40:6833/2146240383","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2146240383},{"type":"v1","addr":"172.21.15.40:6833","nonce":2146240383}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:49:10.199 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:10.193+0000 7f463860d700 1 -- 172.21.15.42:0/692310859 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f461803aca0 msgr2=0x7f461803d1b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:49:10.200 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:10.193+0000 7f463860d700 1 --2- 172.21.15.42:0/692310859 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f461803aca0 0x7f461803d1b0 crc :-1 s=READY pgs=647 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:49:10.200 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:10.193+0000 7f463860d700 1 -- 172.21.15.42:0/692310859 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4630141c00 msgr2=0x7f4630076af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:49:10.200 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:10.193+0000 7f463860d700 1 --2- 172.21.15.42:0/692310859 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4630141c00 0x7f4630076af0 secure :-1 s=READY pgs=1505 cs=0 l=1 rx=0x7f46280032b0 tx=0x7f4628010c30).stop 2019-12-05T05:49:10.200 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:10.193+0000 7f463860d700 1 -- 172.21.15.42:0/692310859 shutdown_connections 2019-12-05T05:49:10.200 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:10.193+0000 7f463860d700 1 --2- 172.21.15.42:0/692310859 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f461803aca0 0x7f461803d1b0 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:49:10.200 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:10.197+0000 7f463860d700 1 --2- 172.21.15.42:0/692310859 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4630141c00 0x7f4630076af0 unknown :-1 s=CLOSED pgs=1505 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:49:10.201 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:10.197+0000 7f463860d700 1 -- 172.21.15.42:0/692310859 shutdown_connections 2019-12-05T05:49:10.201 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:10.197+0000 7f463860d700 1 -- 172.21.15.42:0/692310859 wait complete. 2019-12-05T05:49:10.201 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:10.197+0000 7f463860d700 1 -- 172.21.15.42:0/692310859 >> 172.21.15.42:0/692310859 conn(0x7f46300a82a0 msgr2=0x7f463011ccb0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:49:10.214 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T05:49:11.795 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:49:11.799 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:49:12.215 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:49:12.323 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:12.313+0000 7f3d0e9ee700 1 Processor -- start 2019-12-05T05:49:12.323 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:12.313+0000 7f3d0e9ee700 1 -- start start 2019-12-05T05:49:12.323 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:12.313+0000 7f3d0e9ee700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3d080a7740 0x7f3d0808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:49:12.323 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:12.313+0000 7f3d0e9ee700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3d0807e720 0x7f3d0807eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:49:12.323 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:12.313+0000 7f3d0e9ee700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f3d0809f2e0 0x7f3d0809f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:49:12.324 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:12.313+0000 7f3d0e9ee700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d080a0b00 con 0x7f3d0809f2e0 2019-12-05T05:49:12.324 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:12.313+0000 7f3d0e9ee700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d0808e5b0 con 0x7f3d080a7740 2019-12-05T05:49:12.324 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:12.313+0000 7f3d0e9ee700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d0808e850 con 0x7f3d0807e720 2019-12-05T05:49:12.324 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:12.313+0000 7f3d0cf8b700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3d080a7740 0x7f3d0808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:49:12.324 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:12.313+0000 7f3d0cf8b700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3d080a7740 0x7f3d0808df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:41792/0 (socket says 172.21.15.42:41792) 2019-12-05T05:49:12.324 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:12.313+0000 7f3d0cf8b700 1 -- 172.21.15.42:0/200296167 learned_addr learned my addr 172.21.15.42:0/200296167 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:49:12.325 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:12.313+0000 7f3d07fff700 1 --2- 172.21.15.42:0/200296167 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3d0807e720 0x7f3d0807eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:49:12.325 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:12.313+0000 7f3d0d78c700 1 --2- 172.21.15.42:0/200296167 >> v2:172.21.15.40:3300/0 conn(0x7f3d0809f2e0 0x7f3d0809f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:49:12.325 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:12.313+0000 7f3d0cf8b700 1 -- 172.21.15.42:0/200296167 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3d0807e720 msgr2=0x7f3d0807eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:49:12.325 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:12.313+0000 7f3d0cf8b700 1 --2- 172.21.15.42:0/200296167 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3d0807e720 0x7f3d0807eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:49:12.325 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:12.313+0000 7f3d0cf8b700 1 -- 172.21.15.42:0/200296167 >> v2:172.21.15.40:3300/0 conn(0x7f3d0809f2e0 msgr2=0x7f3d0809f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:49:12.325 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:12.313+0000 7f3d0cf8b700 1 --2- 172.21.15.42:0/200296167 >> v2:172.21.15.40:3300/0 conn(0x7f3d0809f2e0 0x7f3d0809f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:49:12.325 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:12.313+0000 7f3d0cf8b700 1 -- 172.21.15.42:0/200296167 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d0807f0a0 con 0x7f3d080a7740 2019-12-05T05:49:12.326 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:12.313+0000 7f3d0d78c700 1 --2- 172.21.15.42:0/200296167 >> v2:172.21.15.40:3300/0 conn(0x7f3d0809f2e0 0x7f3d0809f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:49:12.326 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:12.317+0000 7f3d0cf8b700 1 --2- 172.21.15.42:0/200296167 >> v2:172.21.15.42:3300/0 conn(0x7f3d080a7740 0x7f3d0808df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f3d0808e770 tx=0x7f3cf800fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:12.326 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:12.317+0000 7f3d0cf8b700 1 --2- 172.21.15.42:0/200296167 >> v2:172.21.15.42:3300/0 conn(0x7f3d080a7740 0x7f3d0808df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f3d0808e770 tx=0x7f3cf800fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:12.326 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:12.317+0000 7f3d0cf8b700 1 --2- 172.21.15.42:0/200296167 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3d080a7740 0x7f3d0808df80 secure :-1 s=READY pgs=1506 cs=0 l=1 rx=0x7f3d0808e770 tx=0x7f3cf800fdb0).ready entity=mon.1 client_cookie=d0bd3b82ac2a61d3 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:49:12.326 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:12.317+0000 7f3d0cf8b700 1 --2- 172.21.15.42:0/200296167 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3d080a7740 0x7f3d0808df80 secure :-1 s=THROTTLE_DONE pgs=1506 cs=0 l=1 rx=0x7f3d0808e770 tx=0x7f3cf800fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:12.326 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:12.317+0000 7f3d0cf8b700 1 --2- 172.21.15.42:0/200296167 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3d080a7740 0x7f3d0808df80 secure :-1 s=THROTTLE_DONE pgs=1506 cs=0 l=1 rx=0x7f3d0808e770 tx=0x7f3cf800fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:12.327 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:12.317+0000 7f3d077fe700 1 -- 172.21.15.42:0/200296167 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3cf8008c90 con 0x7f3d080a7740 2019-12-05T05:49:12.327 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:12.317+0000 7f3d0cf8b700 1 --2- 172.21.15.42:0/200296167 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3d080a7740 0x7f3d0808df80 secure :-1 s=THROTTLE_DONE pgs=1506 cs=0 l=1 rx=0x7f3d0808e770 tx=0x7f3cf800fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:12.327 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:12.317+0000 7f3d077fe700 1 -- 172.21.15.42:0/200296167 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cf80088b0 con 0x7f3d080a7740 2019-12-05T05:49:12.327 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:12.317+0000 7f3d077fe700 1 -- 172.21.15.42:0/200296167 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3cf8011b50 con 0x7f3d080a7740 2019-12-05T05:49:12.327 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:12.317+0000 7f3d0e9ee700 1 -- 172.21.15.42:0/200296167 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3d080a7740 msgr2=0x7f3d0808df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:49:12.327 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:12.317+0000 7f3d0e9ee700 1 --2- 172.21.15.42:0/200296167 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3d080a7740 0x7f3d0808df80 secure :-1 s=READY pgs=1506 cs=0 l=1 rx=0x7f3d0808e770 tx=0x7f3cf800fdb0).stop 2019-12-05T05:49:12.328 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:12.317+0000 7f3d0e9ee700 1 -- 172.21.15.42:0/200296167 shutdown_connections 2019-12-05T05:49:12.328 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:12.317+0000 7f3d0e9ee700 1 --2- 172.21.15.42:0/200296167 >> v2:172.21.15.40:3300/0 conn(0x7f3d0809f2e0 0x7f3d0809f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:49:12.328 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:12.317+0000 7f3d0e9ee700 1 --2- 172.21.15.42:0/200296167 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3d080a7740 0x7f3d0808df80 unknown :-1 s=CLOSED pgs=1506 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:49:12.328 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:12.317+0000 7f3d0e9ee700 1 --2- 172.21.15.42:0/200296167 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3d0807e720 0x7f3d0807eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:49:12.328 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:12.317+0000 7f3d0e9ee700 1 -- 172.21.15.42:0/200296167 shutdown_connections 2019-12-05T05:49:12.328 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:12.317+0000 7f3d0e9ee700 1 -- 172.21.15.42:0/200296167 wait complete. 2019-12-05T05:49:12.328 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:12.317+0000 7f3d0e9ee700 1 -- 172.21.15.42:0/200296167 >> 172.21.15.42:0/200296167 conn(0x7f3d080aabe0 msgr2=0x7f3d080a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:49:12.332 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:12.321+0000 7f3d0e9ee700 1 Processor -- start 2019-12-05T05:49:12.333 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:12.321+0000 7f3d0e9ee700 1 -- start start 2019-12-05T05:49:12.333 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:12.321+0000 7f3d0e9ee700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3d0811a200 0x7f3d080a5ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:49:12.333 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:12.321+0000 7f3d0e9ee700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d0811ae70 con 0x7f3d0806ebe0 2019-12-05T05:49:12.333 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:12.321+0000 7f3d0e9ee700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d080632d0 con 0x7f3d080a6420 2019-12-05T05:49:12.333 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:12.321+0000 7f3d0e9ee700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d080a5c70 con 0x7f3d0811a200 2019-12-05T05:49:12.333 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:12.321+0000 7f3d07fff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f3d080a6420 0x7f3d0806e3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:41836/0 (socket says 172.21.15.42:41836) 2019-12-05T05:49:12.334 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:12.321+0000 7f3d07fff700 1 -- 172.21.15.42:0/2624204995 learned_addr learned my addr 172.21.15.42:0/2624204995 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:49:12.334 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:12.321+0000 7f3d0cf8b700 1 --2- 172.21.15.42:0/2624204995 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3d0811a200 0x7f3d080a5ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:49:12.334 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:12.321+0000 7f3d057fa700 1 -- 172.21.15.42:0/2624204995 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1592632477 0 0) 0x7f3d080632d0 con 0x7f3d080a6420 2019-12-05T05:49:12.334 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:12.321+0000 7f3d057fa700 1 -- 172.21.15.42:0/2624204995 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ce80030b0 con 0x7f3d080a6420 2019-12-05T05:49:12.335 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:12.321+0000 7f3d057fa700 1 -- 172.21.15.42:0/2624204995 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 139735010 0 0) 0x7f3d00001ec0 con 0x7f3d0806ebe0 2019-12-05T05:49:12.335 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:12.321+0000 7f3d057fa700 1 -- 172.21.15.42:0/2624204995 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d080632d0 con 0x7f3d0806ebe0 2019-12-05T05:49:12.335 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:12.321+0000 7f3d0cf8b700 1 -- 172.21.15.42:0/2624204995 >> v1:172.21.15.42:6789/0 conn(0x7f3d080a6420 legacy=0x7f3d0806e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:49:12.335 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:12.321+0000 7f3d0cf8b700 1 -- 172.21.15.42:0/2624204995 >> v1:172.21.15.40:6789/0 conn(0x7f3d0806ebe0 legacy=0x7f3d080757a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:49:12.335 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:12.321+0000 7f3d0cf8b700 1 -- 172.21.15.42:0/2624204995 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d080a6850 con 0x7f3d0811a200 2019-12-05T05:49:12.335 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:12.321+0000 7f3d0cf8b700 1 --2- 172.21.15.42:0/2624204995 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3d0811a200 0x7f3d080a5ed0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f3d0808e770 tx=0x7f3cf8008ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:12.336 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:12.321+0000 7f3d0cf8b700 1 --2- 172.21.15.42:0/2624204995 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3d0811a200 0x7f3d080a5ed0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f3d0808e770 tx=0x7f3cf8008ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:12.336 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:12.321+0000 7f3d0cf8b700 1 --2- 172.21.15.42:0/2624204995 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3d0811a200 0x7f3d080a5ed0 secure :-1 s=READY pgs=335 cs=0 l=1 rx=0x7f3d0808e770 tx=0x7f3cf8008ae0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:49:12.336 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:12.321+0000 7f3d0cf8b700 1 --2- 172.21.15.42:0/2624204995 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3d0811a200 0x7f3d080a5ed0 secure :-1 s=THROTTLE_DONE pgs=335 cs=0 l=1 rx=0x7f3d0808e770 tx=0x7f3cf8008ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:12.336 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:12.321+0000 7f3d057fa700 1 -- 172.21.15.42:0/2624204995 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3cf800f280 con 0x7f3d0811a200 2019-12-05T05:49:12.336 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:12.321+0000 7f3d0cf8b700 1 --2- 172.21.15.42:0/2624204995 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3d0811a200 0x7f3d080a5ed0 secure :-1 s=THROTTLE_DONE pgs=335 cs=0 l=1 rx=0x7f3d0808e770 tx=0x7f3cf8008ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:12.337 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:12.321+0000 7f3d0cf8b700 1 --2- 172.21.15.42:0/2624204995 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3d0811a200 0x7f3d080a5ed0 secure :-1 s=THROTTLE_DONE pgs=335 cs=0 l=1 rx=0x7f3d0808e770 tx=0x7f3cf8008ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:12.337 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:12.321+0000 7f3d057fa700 1 -- 172.21.15.42:0/2624204995 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cf8011770 con 0x7f3d0811a200 2019-12-05T05:49:12.337 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:12.321+0000 7f3d0e9ee700 1 -- 172.21.15.42:0/2624204995 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d0811a7a0 con 0x7f3d0811a200 2019-12-05T05:49:12.337 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:12.321+0000 7f3d057fa700 1 -- 172.21.15.42:0/2624204995 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3cf8005540 con 0x7f3d0811a200 2019-12-05T05:49:12.337 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:12.321+0000 7f3d0e9ee700 1 -- 172.21.15.42:0/2624204995 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d0806f010 con 0x7f3d0811a200 2019-12-05T05:49:12.337 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:12.325+0000 7f3d0cf8b700 1 --2- 172.21.15.42:0/2624204995 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3d0811a200 0x7f3d080a5ed0 secure :-1 s=THROTTLE_DONE pgs=335 cs=0 l=1 rx=0x7f3d0808e770 tx=0x7f3cf8008ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:12.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:12.325+0000 7f3d057fa700 1 -- 172.21.15.42:0/2624204995 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f3cf80057d0 con 0x7f3d0811a200 2019-12-05T05:49:12.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:12.325+0000 7f3d0cf8b700 1 --2- 172.21.15.42:0/2624204995 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3d0811a200 0x7f3d080a5ed0 secure :-1 s=THROTTLE_DONE pgs=335 cs=0 l=1 rx=0x7f3d0808e770 tx=0x7f3cf8008ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:12.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:12.325+0000 7f3d057fa700 1 --2- 172.21.15.42:0/2624204995 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3ce803b450 0x7f3ce803d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:49:12.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:12.325+0000 7f3d057fa700 1 -- 172.21.15.42:0/2624204995 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 8611+0+0 (secure 0 0 0) 0x7f3cf808eef0 con 0x7f3d0811a200 2019-12-05T05:49:12.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:12.325+0000 7f3d07fff700 1 --2- 172.21.15.42:0/2624204995 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3ce803b450 0x7f3ce803d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:49:12.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:12.325+0000 7f3d07fff700 1 --2- 172.21.15.42:0/2624204995 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3ce803b450 0x7f3ce803d960 crc :-1 s=READY pgs=648 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:49:12.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:12.329+0000 7f3d0e9ee700 1 -- 172.21.15.42:0/2624204995 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d0805d110 con 0x7f3d0811a200 2019-12-05T05:49:12.343 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:12.337+0000 7f3d0cf8b700 1 --2- 172.21.15.42:0/2624204995 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3d0811a200 0x7f3d080a5ed0 secure :-1 s=THROTTLE_DONE pgs=335 cs=0 l=1 rx=0x7f3d0808e770 tx=0x7f3cf8008ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:12.343 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:12.341+0000 7f3d057fa700 1 -- 172.21.15.42:0/2624204995 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f3cf8075f70 con 0x7f3d0811a200 2019-12-05T05:49:12.513 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:12.509+0000 7f3d0e9ee700 1 -- 172.21.15.42:0/2624204995 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3d0805b8c0 con 0x7f3d0811a200 2019-12-05T05:49:12.519 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:12.517+0000 7f3d0cf8b700 1 --2- 172.21.15.42:0/2624204995 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3d0811a200 0x7f3d080a5ed0 secure :-1 s=THROTTLE_DONE pgs=335 cs=0 l=1 rx=0x7f3d0808e770 tx=0x7f3cf8008ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:12.519 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:12.517+0000 7f3d057fa700 1 -- 172.21.15.42:0/2624204995 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 183 v183) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f3cf80761d0 con 0x7f3d0811a200 2019-12-05T05:49:12.520 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:49:12.521 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":183,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":8188,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6835/2456922143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2456922143},{"type":"v1","addr":"172.21.15.42:6835","nonce":2456922143}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":183}],"filesystems":[{"mdsmap":{"epoch":182,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:47:19.499092+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6140},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6140":{"gid":6140,"name":"b","rank":0,"incarnation":179,"state":"up:active","state_seq":26,"addr":"172.21.15.40:6833/2146240383","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2146240383},{"type":"v1","addr":"172.21.15.40:6833","nonce":2146240383}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:49:12.521 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 183 2019-12-05T05:49:12.523 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:12.521+0000 7f3d0e9ee700 1 -- 172.21.15.42:0/2624204995 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3ce803b450 msgr2=0x7f3ce803d960 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:49:12.523 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:12.521+0000 7f3d0e9ee700 1 --2- 172.21.15.42:0/2624204995 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3ce803b450 0x7f3ce803d960 crc :-1 s=READY pgs=648 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:49:12.523 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:12.521+0000 7f3d0e9ee700 1 -- 172.21.15.42:0/2624204995 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3d0811a200 msgr2=0x7f3d080a5ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:49:12.524 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:12.521+0000 7f3d0e9ee700 1 --2- 172.21.15.42:0/2624204995 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3d0811a200 0x7f3d080a5ed0 secure :-1 s=READY pgs=335 cs=0 l=1 rx=0x7f3d0808e770 tx=0x7f3cf8008ae0).stop 2019-12-05T05:49:12.524 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:12.521+0000 7f3d0e9ee700 1 -- 172.21.15.42:0/2624204995 shutdown_connections 2019-12-05T05:49:12.524 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:12.521+0000 7f3d0e9ee700 1 --2- 172.21.15.42:0/2624204995 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3ce803b450 0x7f3ce803d960 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:49:12.524 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:12.521+0000 7f3d0e9ee700 1 --2- 172.21.15.42:0/2624204995 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3d0811a200 0x7f3d080a5ed0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:49:12.525 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:12.521+0000 7f3d0e9ee700 1 -- 172.21.15.42:0/2624204995 shutdown_connections 2019-12-05T05:49:12.525 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:12.521+0000 7f3d0e9ee700 1 -- 172.21.15.42:0/2624204995 wait complete. 2019-12-05T05:49:12.525 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:12.521+0000 7f3d0e9ee700 1 -- 172.21.15.42:0/2624204995 >> 172.21.15.42:0/2624204995 conn(0x7f3d080a9b50 msgr2=0x7f3d0809f510 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:49:12.537 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T05:49:14.539 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:49:14.647 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:14.637+0000 7efd6a0df700 1 Processor -- start 2019-12-05T05:49:14.647 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:14.637+0000 7efd6a0df700 1 -- start start 2019-12-05T05:49:14.647 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:14.637+0000 7efd6a0df700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efd6408df80 0x7efd6409f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:49:14.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:14.637+0000 7efd6a0df700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7efd6409f830 0x7efd6414efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:49:14.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:14.637+0000 7efd6a0df700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd6409fc60 con 0x7efd640a7740 2019-12-05T05:49:14.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:14.637+0000 7efd6a0df700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd640a0b00 con 0x7efd6409f830 2019-12-05T05:49:14.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:14.637+0000 7efd6a0df700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd6408e5b0 con 0x7efd6408df80 2019-12-05T05:49:14.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:14.641+0000 7efd637fe700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efd6408df80 0x7efd6409f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:49:14.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:14.641+0000 7efd637fe700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efd6408df80 0x7efd6409f2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:42274/0 (socket says 172.21.15.42:42274) 2019-12-05T05:49:14.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:14.641+0000 7efd637fe700 1 -- 172.21.15.42:0/337480249 learned_addr learned my addr 172.21.15.42:0/337480249 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:49:14.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:14.641+0000 7efd68e7d700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7efd6409f830 0x7efd6414efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:49:14.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:14.641+0000 7efd68e7d700 1 -- 172.21.15.42:0/337480249 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efd6408df80 msgr2=0x7efd6409f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:49:14.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:14.641+0000 7efd68e7d700 1 --2- 172.21.15.42:0/337480249 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efd6408df80 0x7efd6409f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:49:14.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:14.641+0000 7efd68e7d700 1 -- 172.21.15.42:0/337480249 >> v1:172.21.15.40:6789/0 conn(0x7efd640a7740 legacy=0x7efd6407e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:49:14.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:14.641+0000 7efd68e7d700 1 -- 172.21.15.42:0/337480249 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd640a0710 con 0x7efd6409f830 2019-12-05T05:49:14.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:14.641+0000 7efd68e7d700 1 --2- 172.21.15.42:0/337480249 >> v2:172.21.15.40:3300/0 conn(0x7efd6409f830 0x7efd6414efe0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7efd5c008b10 tx=0x7efd5c00fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:14.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:14.641+0000 7efd68e7d700 1 --2- 172.21.15.42:0/337480249 >> v2:172.21.15.40:3300/0 conn(0x7efd6409f830 0x7efd6414efe0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7efd5c008b10 tx=0x7efd5c00fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:14.650 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:14.641+0000 7efd68e7d700 1 --2- 172.21.15.42:0/337480249 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efd6409f830 0x7efd6414efe0 secure :-1 s=READY pgs=1162 cs=0 l=1 rx=0x7efd5c008b10 tx=0x7efd5c00fff0).ready entity=mon.0 client_cookie=e6912f09710d33e3 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:49:14.650 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:14.641+0000 7efd68e7d700 1 --2- 172.21.15.42:0/337480249 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efd6409f830 0x7efd6414efe0 secure :-1 s=THROTTLE_DONE pgs=1162 cs=0 l=1 rx=0x7efd5c008b10 tx=0x7efd5c00fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:14.650 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:14.641+0000 7efd68e7d700 1 --2- 172.21.15.42:0/337480249 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efd6409f830 0x7efd6414efe0 secure :-1 s=THROTTLE_DONE pgs=1162 cs=0 l=1 rx=0x7efd5c008b10 tx=0x7efd5c00fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:14.650 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:14.641+0000 7efd68e7d700 1 --2- 172.21.15.42:0/337480249 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efd6409f830 0x7efd6414efe0 secure :-1 s=THROTTLE_DONE pgs=1162 cs=0 l=1 rx=0x7efd5c008b10 tx=0x7efd5c00fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:14.650 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:14.641+0000 7efd62ffd700 1 -- 172.21.15.42:0/337480249 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efd5c010900 con 0x7efd6409f830 2019-12-05T05:49:14.650 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:14.641+0000 7efd62ffd700 1 -- 172.21.15.42:0/337480249 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd5c008b50 con 0x7efd6409f830 2019-12-05T05:49:14.650 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:14.641+0000 7efd62ffd700 1 -- 172.21.15.42:0/337480249 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efd5c0110c0 con 0x7efd6409f830 2019-12-05T05:49:14.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:14.641+0000 7efd6a0df700 1 -- 172.21.15.42:0/337480249 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efd6409f830 msgr2=0x7efd6414efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:49:14.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:14.641+0000 7efd6a0df700 1 --2- 172.21.15.42:0/337480249 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efd6409f830 0x7efd6414efe0 secure :-1 s=READY pgs=1162 cs=0 l=1 rx=0x7efd5c008b10 tx=0x7efd5c00fff0).stop 2019-12-05T05:49:14.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:14.641+0000 7efd6a0df700 1 -- 172.21.15.42:0/337480249 shutdown_connections 2019-12-05T05:49:14.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:14.641+0000 7efd6a0df700 1 --2- 172.21.15.42:0/337480249 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efd6409f830 0x7efd6414efe0 unknown :-1 s=CLOSED pgs=1162 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:49:14.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:14.641+0000 7efd6a0df700 1 --2- 172.21.15.42:0/337480249 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efd6408df80 0x7efd6409f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:49:14.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:14.641+0000 7efd6a0df700 1 -- 172.21.15.42:0/337480249 shutdown_connections 2019-12-05T05:49:14.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:14.641+0000 7efd6a0df700 1 -- 172.21.15.42:0/337480249 wait complete. 2019-12-05T05:49:14.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:14.641+0000 7efd6a0df700 1 -- 172.21.15.42:0/337480249 >> 172.21.15.42:0/337480249 conn(0x7efd640aabe0 msgr2=0x7efd640a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:49:14.654 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:14.645+0000 7efd6a0df700 1 Processor -- start 2019-12-05T05:49:14.655 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:14.645+0000 7efd6a0df700 1 -- start start 2019-12-05T05:49:14.655 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:14.645+0000 7efd6a0df700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7efd640a4030 0x7efd64114990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:49:14.655 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:14.645+0000 7efd6a0df700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efd641150a0 0x7efd64128100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:49:14.655 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:14.645+0000 7efd6a0df700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd641450f0 con 0x7efd64128650 2019-12-05T05:49:14.655 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:14.645+0000 7efd6a0df700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd64115670 con 0x7efd640a4030 2019-12-05T05:49:14.655 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:14.645+0000 7efd6a0df700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd64115910 con 0x7efd641150a0 2019-12-05T05:49:14.656 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:14.649+0000 7efd63fff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7efd640a4030 0x7efd64114990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:49:14.656 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:14.649+0000 7efd68e7d700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7efd64128650 0x7efd6408eb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:51294/0 (socket says 172.21.15.42:51294) 2019-12-05T05:49:14.656 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:14.649+0000 7efd68e7d700 1 -- 172.21.15.42:0/1603038793 learned_addr learned my addr 172.21.15.42:0/1603038793 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:49:14.656 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:14.649+0000 7efd637fe700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efd641150a0 0x7efd64128100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:49:14.656 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:14.649+0000 7efd60ff9700 1 -- 172.21.15.42:0/1603038793 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3468439340 0 0) 0x7efd641450f0 con 0x7efd64128650 2019-12-05T05:49:14.656 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:14.649+0000 7efd60ff9700 1 -- 172.21.15.42:0/1603038793 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd440030b0 con 0x7efd64128650 2019-12-05T05:49:14.656 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:14.649+0000 7efd637fe700 1 -- 172.21.15.42:0/1603038793 >> v2:172.21.15.42:3300/0 conn(0x7efd640a4030 msgr2=0x7efd64114990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:49:14.657 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:14.649+0000 7efd637fe700 1 --2- 172.21.15.42:0/1603038793 >> v2:172.21.15.42:3300/0 conn(0x7efd640a4030 0x7efd64114990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:49:14.657 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:14.649+0000 7efd637fe700 1 -- 172.21.15.42:0/1603038793 >> v1:172.21.15.40:6789/0 conn(0x7efd64128650 legacy=0x7efd6408eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:49:14.657 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:14.649+0000 7efd637fe700 1 -- 172.21.15.42:0/1603038793 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd64128a80 con 0x7efd641150a0 2019-12-05T05:49:14.657 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:14.649+0000 7efd63fff700 1 --2- 172.21.15.42:0/1603038793 >> v2:172.21.15.42:3300/0 conn(0x7efd640a4030 0x7efd64114990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:49:14.657 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:14.649+0000 7efd637fe700 1 --2- 172.21.15.42:0/1603038793 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efd641150a0 0x7efd64128100 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7efd58002f30 tx=0x7efd58010b20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:14.657 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:14.649+0000 7efd637fe700 1 --2- 172.21.15.42:0/1603038793 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efd641150a0 0x7efd64128100 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7efd58002f30 tx=0x7efd58010b20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:14.657 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:14.649+0000 7efd637fe700 1 --2- 172.21.15.42:0/1603038793 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efd641150a0 0x7efd64128100 secure :-1 s=READY pgs=336 cs=0 l=1 rx=0x7efd58002f30 tx=0x7efd58010b20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:49:14.658 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:14.649+0000 7efd637fe700 1 --2- 172.21.15.42:0/1603038793 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efd641150a0 0x7efd64128100 secure :-1 s=THROTTLE_DONE pgs=336 cs=0 l=1 rx=0x7efd58002f30 tx=0x7efd58010b20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:14.658 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:14.649+0000 7efd60ff9700 1 -- 172.21.15.42:0/1603038793 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efd58005930 con 0x7efd641150a0 2019-12-05T05:49:14.658 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:14.649+0000 7efd637fe700 1 --2- 172.21.15.42:0/1603038793 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efd641150a0 0x7efd64128100 secure :-1 s=THROTTLE_DONE pgs=336 cs=0 l=1 rx=0x7efd58002f30 tx=0x7efd58010b20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:14.658 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:14.649+0000 7efd637fe700 1 --2- 172.21.15.42:0/1603038793 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efd641150a0 0x7efd64128100 secure :-1 s=THROTTLE_DONE pgs=336 cs=0 l=1 rx=0x7efd58002f30 tx=0x7efd58010b20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:14.658 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:14.649+0000 7efd60ff9700 1 -- 172.21.15.42:0/1603038793 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd58007160 con 0x7efd641150a0 2019-12-05T05:49:14.658 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:14.649+0000 7efd60ff9700 1 -- 172.21.15.42:0/1603038793 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efd580088b0 con 0x7efd641150a0 2019-12-05T05:49:14.658 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:14.649+0000 7efd6a0df700 1 -- 172.21.15.42:0/1603038793 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd6408f370 con 0x7efd641150a0 2019-12-05T05:49:14.659 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:14.649+0000 7efd6a0df700 1 -- 172.21.15.42:0/1603038793 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd6408f8a0 con 0x7efd641150a0 2019-12-05T05:49:14.659 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:14.649+0000 7efd637fe700 1 --2- 172.21.15.42:0/1603038793 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efd641150a0 0x7efd64128100 secure :-1 s=THROTTLE_DONE pgs=336 cs=0 l=1 rx=0x7efd58002f30 tx=0x7efd58010b20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:14.660 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:14.653+0000 7efd60ff9700 1 -- 172.21.15.42:0/1603038793 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7efd58008b90 con 0x7efd641150a0 2019-12-05T05:49:14.660 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:14.653+0000 7efd637fe700 1 --2- 172.21.15.42:0/1603038793 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efd641150a0 0x7efd64128100 secure :-1 s=THROTTLE_DONE pgs=336 cs=0 l=1 rx=0x7efd58002f30 tx=0x7efd58010b20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:14.660 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:14.653+0000 7efd60ff9700 1 --2- 172.21.15.42:0/1603038793 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7efd4403b2a0 0x7efd4403d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:49:14.661 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:14.653+0000 7efd60ff9700 1 -- 172.21.15.42:0/1603038793 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 8611+0+0 (secure 0 0 0) 0x7efd5804b1e0 con 0x7efd641150a0 2019-12-05T05:49:14.661 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:14.653+0000 7efd63fff700 1 --2- 172.21.15.42:0/1603038793 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7efd4403b2a0 0x7efd4403d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:49:14.661 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:14.653+0000 7efd63fff700 1 --2- 172.21.15.42:0/1603038793 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7efd4403b2a0 0x7efd4403d7b0 crc :-1 s=READY pgs=650 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:49:14.661 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:14.653+0000 7efd6a0df700 1 -- 172.21.15.42:0/1603038793 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd6405d110 con 0x7efd641150a0 2019-12-05T05:49:14.665 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:14.661+0000 7efd637fe700 1 --2- 172.21.15.42:0/1603038793 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efd641150a0 0x7efd64128100 secure :-1 s=THROTTLE_DONE pgs=336 cs=0 l=1 rx=0x7efd58002f30 tx=0x7efd58010b20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:14.665 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:14.661+0000 7efd60ff9700 1 -- 172.21.15.42:0/1603038793 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7efd58032240 con 0x7efd641150a0 2019-12-05T05:49:14.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:14.833+0000 7efd6a0df700 1 -- 172.21.15.42:0/1603038793 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efd6405b8c0 con 0x7efd641150a0 2019-12-05T05:49:14.837 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:14.833+0000 7efd637fe700 1 --2- 172.21.15.42:0/1603038793 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efd641150a0 0x7efd64128100 secure :-1 s=THROTTLE_DONE pgs=336 cs=0 l=1 rx=0x7efd58002f30 tx=0x7efd58010b20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:14.837 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:14.833+0000 7efd60ff9700 1 -- 172.21.15.42:0/1603038793 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 183 v183) v1 ==== 95+0+2112 (secure 0 0 0) 0x7efd580324a0 con 0x7efd641150a0 2019-12-05T05:49:14.839 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:49:14.840 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":183,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":8188,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6835/2456922143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2456922143},{"type":"v1","addr":"172.21.15.42:6835","nonce":2456922143}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":183}],"filesystems":[{"mdsmap":{"epoch":182,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:47:19.499092+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6140},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6140":{"gid":6140,"name":"b","rank":0,"incarnation":179,"state":"up:active","state_seq":26,"addr":"172.21.15.40:6833/2146240383","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2146240383},{"type":"v1","addr":"172.21.15.40:6833","nonce":2146240383}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:49:14.840 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 183 2019-12-05T05:49:14.842 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:14.837+0000 7efd6a0df700 1 -- 172.21.15.42:0/1603038793 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7efd4403b2a0 msgr2=0x7efd4403d7b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:49:14.842 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:14.837+0000 7efd6a0df700 1 --2- 172.21.15.42:0/1603038793 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7efd4403b2a0 0x7efd4403d7b0 crc :-1 s=READY pgs=650 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:49:14.842 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:14.837+0000 7efd6a0df700 1 -- 172.21.15.42:0/1603038793 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efd641150a0 msgr2=0x7efd64128100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:49:14.842 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:14.837+0000 7efd6a0df700 1 --2- 172.21.15.42:0/1603038793 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efd641150a0 0x7efd64128100 secure :-1 s=READY pgs=336 cs=0 l=1 rx=0x7efd58002f30 tx=0x7efd58010b20).stop 2019-12-05T05:49:14.843 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:14.837+0000 7efd6a0df700 1 -- 172.21.15.42:0/1603038793 shutdown_connections 2019-12-05T05:49:14.843 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:14.837+0000 7efd6a0df700 1 --2- 172.21.15.42:0/1603038793 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7efd4403b2a0 0x7efd4403d7b0 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:49:14.843 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:14.837+0000 7efd6a0df700 1 --2- 172.21.15.42:0/1603038793 >> v2:172.21.15.42:3300/0 conn(0x7efd640a4030 0x7efd64114990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:49:14.843 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:14.837+0000 7efd6a0df700 1 --2- 172.21.15.42:0/1603038793 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efd641150a0 0x7efd64128100 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:49:14.843 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:14.837+0000 7efd6a0df700 1 -- 172.21.15.42:0/1603038793 shutdown_connections 2019-12-05T05:49:14.843 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:14.837+0000 7efd6a0df700 1 -- 172.21.15.42:0/1603038793 wait complete. 2019-12-05T05:49:14.843 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:14.837+0000 7efd6a0df700 1 -- 172.21.15.42:0/1603038793 >> 172.21.15.42:0/1603038793 conn(0x7efd6407de00 msgr2=0x7efd64115e80 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:49:14.858 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T05:49:16.859 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:49:16.959 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:16.949+0000 7f15c328a700 1 Processor -- start 2019-12-05T05:49:16.959 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:16.949+0000 7f15c328a700 1 -- start start 2019-12-05T05:49:16.960 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:16.949+0000 7f15c328a700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f15bc0a7740 0x7f15bc08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:49:16.960 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:16.949+0000 7f15c328a700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f15bc09fae0 0x7f15bc07eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:49:16.960 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:16.949+0000 7f15c328a700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15bc0a0710 con 0x7f15bc07e720 2019-12-05T05:49:16.960 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:16.949+0000 7f15c328a700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15bc0a0b00 con 0x7f15bc09fae0 2019-12-05T05:49:16.960 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:16.949+0000 7f15c328a700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15bc08e5b0 con 0x7f15bc0a7740 2019-12-05T05:49:16.960 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:16.949+0000 7f15c2028700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f15bc09fae0 0x7f15bc07eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:49:16.961 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:16.953+0000 7f15c2028700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f15bc09fae0 0x7f15bc07eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:41826/0 (socket says 172.21.15.42:41826) 2019-12-05T05:49:16.961 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:16.953+0000 7f15c2028700 1 -- 172.21.15.42:0/2599722943 learned_addr learned my addr 172.21.15.42:0/2599722943 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:49:16.961 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:16.953+0000 7f15c1827700 1 --2- 172.21.15.42:0/2599722943 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f15bc0a7740 0x7f15bc08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:49:16.961 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:16.953+0000 7f15c2028700 1 -- 172.21.15.42:0/2599722943 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f15bc0a7740 msgr2=0x7f15bc08df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:49:16.961 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:16.953+0000 7f15c2028700 1 --2- 172.21.15.42:0/2599722943 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f15bc0a7740 0x7f15bc08df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:49:16.961 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:16.953+0000 7f15c2028700 1 -- 172.21.15.42:0/2599722943 >> v1:172.21.15.40:6789/0 conn(0x7f15bc07e720 legacy=0x7f15bc09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:49:16.961 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:16.953+0000 7f15c2028700 1 -- 172.21.15.42:0/2599722943 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15bc07f0a0 con 0x7f15bc09fae0 2019-12-05T05:49:16.962 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:16.953+0000 7f15c1827700 1 --2- 172.21.15.42:0/2599722943 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f15bc0a7740 0x7f15bc08df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T05:49:16.962 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:16.953+0000 7f15c2028700 1 --2- 172.21.15.42:0/2599722943 >> v2:172.21.15.42:3300/0 conn(0x7f15bc09fae0 0x7f15bc07eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f15b4008b10 tx=0x7f15b40100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:16.962 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:16.953+0000 7f15c2028700 1 --2- 172.21.15.42:0/2599722943 >> v2:172.21.15.42:3300/0 conn(0x7f15bc09fae0 0x7f15bc07eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f15b4008b10 tx=0x7f15b40100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:16.962 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:16.953+0000 7f15c2028700 1 --2- 172.21.15.42:0/2599722943 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f15bc09fae0 0x7f15bc07eb50 secure :-1 s=READY pgs=1508 cs=0 l=1 rx=0x7f15b4008b10 tx=0x7f15b40100b0).ready entity=mon.1 client_cookie=86dc714149af55aa server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:49:16.962 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:16.953+0000 7f15c2028700 1 --2- 172.21.15.42:0/2599722943 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f15bc09fae0 0x7f15bc07eb50 secure :-1 s=THROTTLE_DONE pgs=1508 cs=0 l=1 rx=0x7f15b4008b10 tx=0x7f15b40100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:16.962 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:16.953+0000 7f15c0825700 1 -- 172.21.15.42:0/2599722943 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f15b4010da0 con 0x7f15bc09fae0 2019-12-05T05:49:16.962 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:16.953+0000 7f15c2028700 1 --2- 172.21.15.42:0/2599722943 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f15bc09fae0 0x7f15bc07eb50 secure :-1 s=THROTTLE_DONE pgs=1508 cs=0 l=1 rx=0x7f15b4008b10 tx=0x7f15b40100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:16.963 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:16.953+0000 7f15c2028700 1 --2- 172.21.15.42:0/2599722943 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f15bc09fae0 0x7f15bc07eb50 secure :-1 s=THROTTLE_DONE pgs=1508 cs=0 l=1 rx=0x7f15b4008b10 tx=0x7f15b40100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:16.963 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:16.953+0000 7f15c0825700 1 -- 172.21.15.42:0/2599722943 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15b40109c0 con 0x7f15bc09fae0 2019-12-05T05:49:16.963 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:16.953+0000 7f15c0825700 1 -- 172.21.15.42:0/2599722943 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f15b40114a0 con 0x7f15bc09fae0 2019-12-05T05:49:16.963 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:16.953+0000 7f15c328a700 1 -- 172.21.15.42:0/2599722943 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f15bc09fae0 msgr2=0x7f15bc07eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:49:16.963 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:16.953+0000 7f15c328a700 1 --2- 172.21.15.42:0/2599722943 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f15bc09fae0 0x7f15bc07eb50 secure :-1 s=READY pgs=1508 cs=0 l=1 rx=0x7f15b4008b10 tx=0x7f15b40100b0).stop 2019-12-05T05:49:16.964 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:16.953+0000 7f15c328a700 1 -- 172.21.15.42:0/2599722943 shutdown_connections 2019-12-05T05:49:16.964 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:16.953+0000 7f15c328a700 1 --2- 172.21.15.42:0/2599722943 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f15bc09fae0 0x7f15bc07eb50 unknown :-1 s=CLOSED pgs=1508 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:49:16.964 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:16.953+0000 7f15c328a700 1 --2- 172.21.15.42:0/2599722943 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f15bc0a7740 0x7f15bc08df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:49:16.964 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:16.953+0000 7f15c328a700 1 -- 172.21.15.42:0/2599722943 shutdown_connections 2019-12-05T05:49:16.964 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:16.953+0000 7f15c328a700 1 -- 172.21.15.42:0/2599722943 wait complete. 2019-12-05T05:49:16.964 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:16.953+0000 7f15c328a700 1 -- 172.21.15.42:0/2599722943 >> 172.21.15.42:0/2599722943 conn(0x7f15bc0aabe0 msgr2=0x7f15bc0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:49:16.967 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:16.957+0000 7f15c328a700 1 Processor -- start 2019-12-05T05:49:16.967 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:16.957+0000 7f15c328a700 1 -- start start 2019-12-05T05:49:16.968 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:16.957+0000 7f15c328a700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f15bc0a5c10 0x7f15bc0a4d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:49:16.968 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:16.957+0000 7f15c328a700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15bc116f20 con 0x7f15bc122080 2019-12-05T05:49:16.968 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:16.957+0000 7f15c328a700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15bc12e6b0 con 0x7f15bc07b370 2019-12-05T05:49:16.968 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:16.957+0000 7f15c328a700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15bc0a6240 con 0x7f15bc0a5c10 2019-12-05T05:49:16.968 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:16.957+0000 7f15c1827700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f15bc0a5c10 0x7f15bc0a4d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:49:16.968 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:16.957+0000 7f15c1827700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f15bc0a5c10 0x7f15bc0a4d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:41828/0 (socket says 172.21.15.42:41828) 2019-12-05T05:49:16.968 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:16.957+0000 7f15c1827700 1 -- 172.21.15.42:0/2973965261 learned_addr learned my addr 172.21.15.42:0/2973965261 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:49:16.969 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:16.957+0000 7f15c2028700 1 --1- 172.21.15.42:0/2973965261 >> v1:172.21.15.42:6789/0 conn(0x7f15bc07b370 0x7f15bc07b7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:41868/0 (socket says 172.21.15.42:41868) 2019-12-05T05:49:16.969 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:16.957+0000 7f15c1026700 1 -- 172.21.15.42:0/2973965261 >> v1:172.21.15.40:6789/0 conn(0x7f15bc122080 legacy=0x7f15bc1224b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2019-12-05T05:49:16.969 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:16.957+0000 7f15c1827700 1 -- 172.21.15.42:0/2973965261 >> v1:172.21.15.42:6789/0 conn(0x7f15bc07b370 legacy=0x7f15bc07b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:49:16.969 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:16.957+0000 7f15c1026700 1 -- 172.21.15.42:0/2973965261 >> v1:172.21.15.40:6789/0 conn(0x7f15bc122080 legacy=0x7f15bc1224b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2019-12-05T05:49:16.969 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:16.957+0000 7f15c1827700 1 -- 172.21.15.42:0/2973965261 >> v1:172.21.15.40:6789/0 conn(0x7f15bc122080 legacy=0x7f15bc1224b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:49:16.969 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:16.957+0000 7f15c1026700 1 --1- 172.21.15.42:0/2973965261 >> v1:172.21.15.40:6789/0 conn(0x7f15bc122080 0x7f15bc1224b0 :-1 s=OPENED pgs=1165 cs=1 l=1).handle_message read tag failed 2019-12-05T05:49:16.969 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:16.957+0000 7f15c1026700 1 --1- 172.21.15.42:0/2973965261 >> v1:172.21.15.40:6789/0 conn(0x7f15bc122080 0x7f15bc1224b0 :-1 s=OPENED pgs=1165 cs=1 l=1).fault on lossy channel, failing 2019-12-05T05:49:16.970 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:16.957+0000 7f15c1827700 1 -- 172.21.15.42:0/2973965261 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15bc0a5260 con 0x7f15bc0a5c10 2019-12-05T05:49:16.970 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:16.957+0000 7f15c1827700 1 --2- 172.21.15.42:0/2973965261 >> v2:172.21.15.42:3300/0 conn(0x7f15bc0a5c10 0x7f15bc0a4d10 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f15b80077a0 tx=0x7f15b80109d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:16.973 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:16.957+0000 7f15c1827700 1 --2- 172.21.15.42:0/2973965261 >> v2:172.21.15.42:3300/0 conn(0x7f15bc0a5c10 0x7f15bc0a4d10 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f15b80077a0 tx=0x7f15b80109d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:16.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:16.957+0000 7f15c1827700 1 --2- 172.21.15.42:0/2973965261 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f15bc0a5c10 0x7f15bc0a4d10 secure :-1 s=READY pgs=1510 cs=0 l=1 rx=0x7f15b80077a0 tx=0x7f15b80109d0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:49:16.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:16.961+0000 7f15c1827700 1 --2- 172.21.15.42:0/2973965261 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f15bc0a5c10 0x7f15bc0a4d10 secure :-1 s=THROTTLE_DONE pgs=1510 cs=0 l=1 rx=0x7f15b80077a0 tx=0x7f15b80109d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:16.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:16.961+0000 7f15b27fc700 1 -- 172.21.15.42:0/2973965261 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f15b800a550 con 0x7f15bc0a5c10 2019-12-05T05:49:16.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:16.961+0000 7f15c1827700 1 --2- 172.21.15.42:0/2973965261 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f15bc0a5c10 0x7f15bc0a4d10 secure :-1 s=THROTTLE_DONE pgs=1510 cs=0 l=1 rx=0x7f15b80077a0 tx=0x7f15b80109d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:16.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:16.961+0000 7f15b27fc700 1 -- 172.21.15.42:0/2973965261 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15b8007bb0 con 0x7f15bc0a5c10 2019-12-05T05:49:16.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:16.961+0000 7f15c328a700 1 -- 172.21.15.42:0/2973965261 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15bc12d6a0 con 0x7f15bc0a5c10 2019-12-05T05:49:16.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:16.961+0000 7f15c1827700 1 --2- 172.21.15.42:0/2973965261 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f15bc0a5c10 0x7f15bc0a4d10 secure :-1 s=THROTTLE_DONE pgs=1510 cs=0 l=1 rx=0x7f15b80077a0 tx=0x7f15b80109d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:16.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:16.961+0000 7f15b27fc700 1 -- 172.21.15.42:0/2973965261 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f15b800ab30 con 0x7f15bc0a5c10 2019-12-05T05:49:16.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:16.961+0000 7f15c328a700 1 -- 172.21.15.42:0/2973965261 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15bc12dba0 con 0x7f15bc0a5c10 2019-12-05T05:49:16.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:16.961+0000 7f15c1827700 1 --2- 172.21.15.42:0/2973965261 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f15bc0a5c10 0x7f15bc0a4d10 secure :-1 s=THROTTLE_DONE pgs=1510 cs=0 l=1 rx=0x7f15b80077a0 tx=0x7f15b80109d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:16.976 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:16.961+0000 7f15b27fc700 1 -- 172.21.15.42:0/2973965261 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f15b8007170 con 0x7f15bc0a5c10 2019-12-05T05:49:16.976 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:16.961+0000 7f15c1827700 1 --2- 172.21.15.42:0/2973965261 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f15bc0a5c10 0x7f15bc0a4d10 secure :-1 s=THROTTLE_DONE pgs=1510 cs=0 l=1 rx=0x7f15b80077a0 tx=0x7f15b80109d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:16.976 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:16.961+0000 7f15b27fc700 1 --2- 172.21.15.42:0/2973965261 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f15a403ad20 0x7f15a403d230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:49:16.976 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:16.961+0000 7f15b27fc700 1 -- 172.21.15.42:0/2973965261 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 8611+0+0 (secure 0 0 0) 0x7f15b804af80 con 0x7f15bc0a5c10 2019-12-05T05:49:16.976 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:16.961+0000 7f15c1026700 1 --2- 172.21.15.42:0/2973965261 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f15a403ad20 0x7f15a403d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:49:16.976 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:16.961+0000 7f15c1026700 1 --2- 172.21.15.42:0/2973965261 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f15a403ad20 0x7f15a403d230 crc :-1 s=READY pgs=651 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:49:16.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:16.965+0000 7f15c328a700 1 -- 172.21.15.42:0/2973965261 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15bc0a6040 con 0x7f15bc0a5c10 2019-12-05T05:49:16.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:16.969+0000 7f15c1827700 1 --2- 172.21.15.42:0/2973965261 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f15bc0a5c10 0x7f15bc0a4d10 secure :-1 s=THROTTLE_DONE pgs=1510 cs=0 l=1 rx=0x7f15b80077a0 tx=0x7f15b80109d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:16.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:16.969+0000 7f15b27fc700 1 -- 172.21.15.42:0/2973965261 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f15b802f0c0 con 0x7f15bc0a5c10 2019-12-05T05:49:17.137 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:17.133+0000 7f15c328a700 1 -- 172.21.15.42:0/2973965261 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f15bc05d110 con 0x7f15bc0a5c10 2019-12-05T05:49:17.138 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:17.133+0000 7f15c1827700 1 --2- 172.21.15.42:0/2973965261 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f15bc0a5c10 0x7f15bc0a4d10 secure :-1 s=THROTTLE_DONE pgs=1510 cs=0 l=1 rx=0x7f15b80077a0 tx=0x7f15b80109d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:17.138 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:17.133+0000 7f15b27fc700 1 -- 172.21.15.42:0/2973965261 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 183 v183) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f15b8032220 con 0x7f15bc0a5c10 2019-12-05T05:49:17.141 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 183 2019-12-05T05:49:17.141 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:49:17.141 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":183,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":8188,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6835/2456922143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2456922143},{"type":"v1","addr":"172.21.15.42:6835","nonce":2456922143}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":183}],"filesystems":[{"mdsmap":{"epoch":182,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:47:19.499092+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6140},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6140":{"gid":6140,"name":"b","rank":0,"incarnation":179,"state":"up:active","state_seq":26,"addr":"172.21.15.40:6833/2146240383","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2146240383},{"type":"v1","addr":"172.21.15.40:6833","nonce":2146240383}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:49:17.144 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:17.137+0000 7f15c328a700 1 -- 172.21.15.42:0/2973965261 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f15a403ad20 msgr2=0x7f15a403d230 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:49:17.144 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:17.137+0000 7f15c328a700 1 --2- 172.21.15.42:0/2973965261 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f15a403ad20 0x7f15a403d230 crc :-1 s=READY pgs=651 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:49:17.144 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:17.137+0000 7f15c328a700 1 -- 172.21.15.42:0/2973965261 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f15bc0a5c10 msgr2=0x7f15bc0a4d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:49:17.145 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:17.137+0000 7f15c328a700 1 --2- 172.21.15.42:0/2973965261 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f15bc0a5c10 0x7f15bc0a4d10 secure :-1 s=READY pgs=1510 cs=0 l=1 rx=0x7f15b80077a0 tx=0x7f15b80109d0).stop 2019-12-05T05:49:17.145 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:17.141+0000 7f15c328a700 1 -- 172.21.15.42:0/2973965261 shutdown_connections 2019-12-05T05:49:17.145 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:17.141+0000 7f15c328a700 1 --2- 172.21.15.42:0/2973965261 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f15a403ad20 0x7f15a403d230 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:49:17.146 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:17.141+0000 7f15c328a700 1 --2- 172.21.15.42:0/2973965261 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f15bc0a5c10 0x7f15bc0a4d10 unknown :-1 s=CLOSED pgs=1510 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:49:17.146 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:17.141+0000 7f15c328a700 1 -- 172.21.15.42:0/2973965261 shutdown_connections 2019-12-05T05:49:17.146 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:17.141+0000 7f15c328a700 1 -- 172.21.15.42:0/2973965261 wait complete. 2019-12-05T05:49:17.146 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:17.141+0000 7f15c328a700 1 -- 172.21.15.42:0/2973965261 >> 172.21.15.42:0/2973965261 conn(0x7f15bc0a82a0 msgr2=0x7f15bc117420 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:49:17.153 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T05:49:19.155 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:49:19.243 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.233+0000 7f97166d1700 1 Processor -- start 2019-12-05T05:49:19.243 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.233+0000 7f97166d1700 1 -- start start 2019-12-05T05:49:19.243 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.237+0000 7f97166d1700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f97100a7740 0x7f971008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:49:19.243 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.237+0000 7f97166d1700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f971007e720 0x7f971007eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:49:19.243 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.237+0000 7f97166d1700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97100a0710 con 0x7f971009f2e0 2019-12-05T05:49:19.244 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.237+0000 7f97166d1700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97100a0b00 con 0x7f971007e720 2019-12-05T05:49:19.244 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.237+0000 7f97166d1700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f971008e5b0 con 0x7f97100a7740 2019-12-05T05:49:19.244 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.237+0000 7f971546f700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f971009f2e0 0x7f971009f710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:41890/0 (socket says 172.21.15.42:41890) 2019-12-05T05:49:19.244 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.237+0000 7f971546f700 1 -- 172.21.15.42:0/3222131458 learned_addr learned my addr 172.21.15.42:0/3222131458 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:49:19.244 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.237+0000 7f9714c6e700 1 --2- 172.21.15.42:0/3222131458 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f97100a7740 0x7f971008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:49:19.244 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.237+0000 7f970ffff700 1 --2- 172.21.15.42:0/3222131458 >> v2:172.21.15.40:3300/0 conn(0x7f971007e720 0x7f971007eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:49:19.245 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.237+0000 7f970f7fe700 1 -- 172.21.15.42:0/3222131458 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1890881881 0 0) 0x7f97100a0710 con 0x7f971009f2e0 2019-12-05T05:49:19.245 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.237+0000 7f970f7fe700 1 -- 172.21.15.42:0/3222131458 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97000030b0 con 0x7f971009f2e0 2019-12-05T05:49:19.245 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.237+0000 7f970f7fe700 1 -- 172.21.15.42:0/3222131458 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2241710727 0 0) 0x7f97000030b0 con 0x7f971009f2e0 2019-12-05T05:49:19.245 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.237+0000 7f9714c6e700 1 -- 172.21.15.42:0/3222131458 >> v2:172.21.15.40:3300/0 conn(0x7f971007e720 msgr2=0x7f971007eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:49:19.245 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.237+0000 7f9714c6e700 1 --2- 172.21.15.42:0/3222131458 >> v2:172.21.15.40:3300/0 conn(0x7f971007e720 0x7f971007eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:49:19.245 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.237+0000 7f9714c6e700 1 -- 172.21.15.42:0/3222131458 >> v1:172.21.15.42:6789/0 conn(0x7f971009f2e0 legacy=0x7f971009f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:49:19.245 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.237+0000 7f9714c6e700 1 -- 172.21.15.42:0/3222131458 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f971007f0a0 con 0x7f97100a7740 2019-12-05T05:49:19.246 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.237+0000 7f9714c6e700 1 --2- 172.21.15.42:0/3222131458 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f97100a7740 0x7f971008df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f9704008790 tx=0x7f9708001d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:19.246 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.237+0000 7f9714c6e700 1 --2- 172.21.15.42:0/3222131458 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f97100a7740 0x7f971008df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f9704008790 tx=0x7f9708001d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:19.246 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.237+0000 7f9714c6e700 1 --2- 172.21.15.42:0/3222131458 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f97100a7740 0x7f971008df80 secure :-1 s=READY pgs=337 cs=0 l=1 rx=0x7f9704008790 tx=0x7f9708001d50).ready entity=mon.2 client_cookie=7afb78593b485cc4 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:49:19.246 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.237+0000 7f9714c6e700 1 --2- 172.21.15.42:0/3222131458 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f97100a7740 0x7f971008df80 secure :-1 s=THROTTLE_DONE pgs=337 cs=0 l=1 rx=0x7f9704008790 tx=0x7f9708001d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:19.246 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.237+0000 7f970f7fe700 1 -- 172.21.15.42:0/3222131458 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9708001ad0 con 0x7f97100a7740 2019-12-05T05:49:19.246 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.237+0000 7f9714c6e700 1 --2- 172.21.15.42:0/3222131458 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f97100a7740 0x7f971008df80 secure :-1 s=THROTTLE_DONE pgs=337 cs=0 l=1 rx=0x7f9704008790 tx=0x7f9708001d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:19.247 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.237+0000 7f9714c6e700 1 --2- 172.21.15.42:0/3222131458 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f97100a7740 0x7f971008df80 secure :-1 s=THROTTLE_DONE pgs=337 cs=0 l=1 rx=0x7f9704008790 tx=0x7f9708001d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:19.247 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.237+0000 7f970f7fe700 1 -- 172.21.15.42:0/3222131458 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97080018e0 con 0x7f97100a7740 2019-12-05T05:49:19.247 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.237+0000 7f970f7fe700 1 -- 172.21.15.42:0/3222131458 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f97040113e0 con 0x7f97100a7740 2019-12-05T05:49:19.247 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.237+0000 7f97166d1700 1 -- 172.21.15.42:0/3222131458 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f97100a7740 msgr2=0x7f971008df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:49:19.247 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.237+0000 7f97166d1700 1 --2- 172.21.15.42:0/3222131458 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f97100a7740 0x7f971008df80 secure :-1 s=READY pgs=337 cs=0 l=1 rx=0x7f9704008790 tx=0x7f9708001d50).stop 2019-12-05T05:49:19.247 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.237+0000 7f97166d1700 1 -- 172.21.15.42:0/3222131458 shutdown_connections 2019-12-05T05:49:19.247 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.237+0000 7f97166d1700 1 --2- 172.21.15.42:0/3222131458 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f97100a7740 0x7f971008df80 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:49:19.248 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.237+0000 7f97166d1700 1 --2- 172.21.15.42:0/3222131458 >> v2:172.21.15.40:3300/0 conn(0x7f971007e720 0x7f971007eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:49:19.248 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.237+0000 7f97166d1700 1 -- 172.21.15.42:0/3222131458 shutdown_connections 2019-12-05T05:49:19.248 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.237+0000 7f97166d1700 1 -- 172.21.15.42:0/3222131458 wait complete. 2019-12-05T05:49:19.248 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.237+0000 7f97166d1700 1 -- 172.21.15.42:0/3222131458 >> 172.21.15.42:0/3222131458 conn(0x7f97100aabe0 msgr2=0x7f97100a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:49:19.251 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.241+0000 7f97166d1700 1 Processor -- start 2019-12-05T05:49:19.251 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.241+0000 7f97166d1700 1 -- start start 2019-12-05T05:49:19.252 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.241+0000 7f97166d1700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9710116500 0x7f9710117cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:49:19.252 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.241+0000 7f97166d1700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f9710116ed0 0x7f971011acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:49:19.252 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.241+0000 7f97166d1700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f971011b200 0x7f971008eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:49:19.252 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.241+0000 7f97166d1700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9710117410 con 0x7f9710116ed0 2019-12-05T05:49:19.252 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.241+0000 7f97166d1700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9710118220 con 0x7f971011b200 2019-12-05T05:49:19.252 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.241+0000 7f97166d1700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97101183e0 con 0x7f9710116500 2019-12-05T05:49:19.253 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.241+0000 7f971546f700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f971011b200 0x7f971008eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:49:19.253 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.241+0000 7f971546f700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f971011b200 0x7f971008eb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:41860/0 (socket says 172.21.15.42:41860) 2019-12-05T05:49:19.253 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.241+0000 7f971546f700 1 -- 172.21.15.42:0/4221161573 learned_addr learned my addr 172.21.15.42:0/4221161573 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:49:19.253 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.241+0000 7f9714c6e700 1 --2- 172.21.15.42:0/4221161573 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9710116500 0x7f9710117cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:49:19.253 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.241+0000 7f970ffff700 1 --2- 172.21.15.42:0/4221161573 >> v2:172.21.15.40:3300/0 conn(0x7f9710116ed0 0x7f971011acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:49:19.253 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.241+0000 7f971546f700 1 -- 172.21.15.42:0/4221161573 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9710116500 msgr2=0x7f9710117cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:49:19.253 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.241+0000 7f971546f700 1 --2- 172.21.15.42:0/4221161573 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9710116500 0x7f9710117cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:49:19.254 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.241+0000 7f971546f700 1 -- 172.21.15.42:0/4221161573 >> v2:172.21.15.40:3300/0 conn(0x7f9710116ed0 msgr2=0x7f971011acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:49:19.254 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.241+0000 7f971546f700 1 --2- 172.21.15.42:0/4221161573 >> v2:172.21.15.40:3300/0 conn(0x7f9710116ed0 0x7f971011acb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:49:19.254 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.241+0000 7f971546f700 1 -- 172.21.15.42:0/4221161573 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f971011b630 con 0x7f971011b200 2019-12-05T05:49:19.254 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.241+0000 7f971546f700 1 --2- 172.21.15.42:0/4221161573 >> v2:172.21.15.42:3300/0 conn(0x7f971011b200 0x7f971008eb70 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f9708001dd0 tx=0x7f9708010cc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:19.254 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.241+0000 7f971546f700 1 --2- 172.21.15.42:0/4221161573 >> v2:172.21.15.42:3300/0 conn(0x7f971011b200 0x7f971008eb70 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f9708001dd0 tx=0x7f9708010cc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:19.254 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.241+0000 7f971546f700 1 --2- 172.21.15.42:0/4221161573 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f971011b200 0x7f971008eb70 secure :-1 s=READY pgs=1512 cs=0 l=1 rx=0x7f9708001dd0 tx=0x7f9708010cc0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:49:19.255 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.241+0000 7f971546f700 1 --2- 172.21.15.42:0/4221161573 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f971011b200 0x7f971008eb70 secure :-1 s=THROTTLE_DONE pgs=1512 cs=0 l=1 rx=0x7f9708001dd0 tx=0x7f9708010cc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:19.255 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.241+0000 7f971546f700 1 --2- 172.21.15.42:0/4221161573 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f971011b200 0x7f971008eb70 secure :-1 s=THROTTLE_DONE pgs=1512 cs=0 l=1 rx=0x7f9708001dd0 tx=0x7f9708010cc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:19.255 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.241+0000 7f970d7fa700 1 -- 172.21.15.42:0/4221161573 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9708004b90 con 0x7f971011b200 2019-12-05T05:49:19.255 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.241+0000 7f971546f700 1 --2- 172.21.15.42:0/4221161573 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f971011b200 0x7f971008eb70 secure :-1 s=THROTTLE_DONE pgs=1512 cs=0 l=1 rx=0x7f9708001dd0 tx=0x7f9708010cc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:19.255 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.241+0000 7f970d7fa700 1 -- 172.21.15.42:0/4221161573 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97080047b0 con 0x7f971011b200 2019-12-05T05:49:19.255 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.241+0000 7f97166d1700 1 -- 172.21.15.42:0/4221161573 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f971008f0c0 con 0x7f971011b200 2019-12-05T05:49:19.255 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.241+0000 7f970d7fa700 1 -- 172.21.15.42:0/4221161573 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9708012870 con 0x7f971011b200 2019-12-05T05:49:19.256 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.241+0000 7f97166d1700 1 -- 172.21.15.42:0/4221161573 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f971008f670 con 0x7f971011b200 2019-12-05T05:49:19.256 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.241+0000 7f971546f700 1 --2- 172.21.15.42:0/4221161573 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f971011b200 0x7f971008eb70 secure :-1 s=THROTTLE_DONE pgs=1512 cs=0 l=1 rx=0x7f9708001dd0 tx=0x7f9708010cc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:19.256 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.245+0000 7f970d7fa700 1 -- 172.21.15.42:0/4221161573 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f9708012b40 con 0x7f971011b200 2019-12-05T05:49:19.256 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.245+0000 7f971546f700 1 --2- 172.21.15.42:0/4221161573 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f971011b200 0x7f971008eb70 secure :-1 s=THROTTLE_DONE pgs=1512 cs=0 l=1 rx=0x7f9708001dd0 tx=0x7f9708010cc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:19.256 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.245+0000 7f970d7fa700 1 --2- 172.21.15.42:0/4221161573 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f96f003ae00 0x7f96f003d310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:49:19.256 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.245+0000 7f970d7fa700 1 -- 172.21.15.42:0/4221161573 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 8697+0+0 (secure 0 0 0) 0x7f970804b580 con 0x7f971011b200 2019-12-05T05:49:19.256 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.245+0000 7f9714c6e700 1 --2- 172.21.15.42:0/4221161573 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f96f003ae00 0x7f96f003d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:49:19.257 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.245+0000 7f9714c6e700 1 --2- 172.21.15.42:0/4221161573 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f96f003ae00 0x7f96f003d310 crc :-1 s=READY pgs=652 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:49:19.257 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.249+0000 7f97166d1700 1 -- 172.21.15.42:0/4221161573 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97101175d0 con 0x7f971011b200 2019-12-05T05:49:19.262 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.257+0000 7f971546f700 1 --2- 172.21.15.42:0/4221161573 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f971011b200 0x7f971008eb70 secure :-1 s=THROTTLE_DONE pgs=1512 cs=0 l=1 rx=0x7f9708001dd0 tx=0x7f9708010cc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:19.263 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.257+0000 7f970d7fa700 1 -- 172.21.15.42:0/4221161573 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f9708032600 con 0x7f971011b200 2019-12-05T05:49:19.431 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.429+0000 7f97166d1700 1 -- 172.21.15.42:0/4221161573 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f971005d110 con 0x7f971011b200 2019-12-05T05:49:19.432 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.429+0000 7f971546f700 1 --2- 172.21.15.42:0/4221161573 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f971011b200 0x7f971008eb70 secure :-1 s=THROTTLE_DONE pgs=1512 cs=0 l=1 rx=0x7f9708001dd0 tx=0x7f9708010cc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:19.432 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.429+0000 7f970d7fa700 1 -- 172.21.15.42:0/4221161573 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 184 v184) v1 ==== 95+0+1774 (secure 0 0 0) 0x7f9708032860 con 0x7f971011b200 2019-12-05T05:49:19.432 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 184 2019-12-05T05:49:19.433 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:49:19.433 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":184,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":184,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:49:19.089508+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8188},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8188":{"gid":8188,"name":"a","rank":0,"incarnation":184,"state":"up:replay","state_seq":1,"addr":"172.21.15.42:6835/2456922143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2456922143},{"type":"v1","addr":"172.21.15.42:6835","nonce":2456922143}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:49:19.436 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.433+0000 7f97166d1700 1 -- 172.21.15.42:0/4221161573 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f96f003ae00 msgr2=0x7f96f003d310 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:49:19.436 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.433+0000 7f97166d1700 1 --2- 172.21.15.42:0/4221161573 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f96f003ae00 0x7f96f003d310 crc :-1 s=READY pgs=652 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:49:19.437 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.433+0000 7f97166d1700 1 -- 172.21.15.42:0/4221161573 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f971011b200 msgr2=0x7f971008eb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:49:19.437 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.433+0000 7f97166d1700 1 --2- 172.21.15.42:0/4221161573 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f971011b200 0x7f971008eb70 secure :-1 s=READY pgs=1512 cs=0 l=1 rx=0x7f9708001dd0 tx=0x7f9708010cc0).stop 2019-12-05T05:49:19.437 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.433+0000 7f97166d1700 1 -- 172.21.15.42:0/4221161573 shutdown_connections 2019-12-05T05:49:19.437 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.433+0000 7f97166d1700 1 --2- 172.21.15.42:0/4221161573 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f96f003ae00 0x7f96f003d310 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:49:19.437 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.433+0000 7f97166d1700 1 --2- 172.21.15.42:0/4221161573 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f971011b200 0x7f971008eb70 unknown :-1 s=CLOSED pgs=1512 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:49:19.437 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.433+0000 7f97166d1700 1 --2- 172.21.15.42:0/4221161573 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9710116500 0x7f9710117cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:49:19.437 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.433+0000 7f97166d1700 1 --2- 172.21.15.42:0/4221161573 >> v2:172.21.15.40:3300/0 conn(0x7f9710116ed0 0x7f971011acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:49:19.438 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.433+0000 7f97166d1700 1 -- 172.21.15.42:0/4221161573 shutdown_connections 2019-12-05T05:49:19.438 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.433+0000 7f97166d1700 1 -- 172.21.15.42:0/4221161573 wait complete. 2019-12-05T05:49:19.438 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.433+0000 7f97166d1700 1 -- 172.21.15.42:0/4221161573 >> 172.21.15.42:0/4221161573 conn(0x7f97100a7f90 msgr2=0x7f971011c540 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:49:19.449 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b down, removed from mdsmap 2019-12-05T05:49:19.450 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2019-12-05T05:49:19.450 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:49:19.548 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.541+0000 7fb2d2dfb700 1 Processor -- start 2019-12-05T05:49:19.549 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.541+0000 7fb2d2dfb700 1 -- start start 2019-12-05T05:49:19.549 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.541+0000 7fb2d2dfb700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb2cc0a7740 0x7fb2cc08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:49:19.549 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.541+0000 7fb2d2dfb700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb2cc07e720 0x7fb2cc07eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:49:19.549 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.541+0000 7fb2d2dfb700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2cc0a0710 con 0x7fb2cc09f2e0 2019-12-05T05:49:19.549 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.541+0000 7fb2d2dfb700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2cc0a0b00 con 0x7fb2cc0a7740 2019-12-05T05:49:19.549 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.541+0000 7fb2d2dfb700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2cc08e5b0 con 0x7fb2cc07e720 2019-12-05T05:49:19.550 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.541+0000 7fb2d0b97700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb2cc07e720 0x7fb2cc07eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:49:19.550 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.541+0000 7fb2d0b97700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb2cc07e720 0x7fb2cc07eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:41870/0 (socket says 172.21.15.42:41870) 2019-12-05T05:49:19.550 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.541+0000 7fb2d0b97700 1 -- 172.21.15.42:0/3852564963 learned_addr learned my addr 172.21.15.42:0/3852564963 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:49:19.550 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.541+0000 7fb2d1398700 1 --2- 172.21.15.42:0/3852564963 >> v2:172.21.15.40:3300/0 conn(0x7fb2cc0a7740 0x7fb2cc08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:49:19.550 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.541+0000 7fb2d0b97700 1 -- 172.21.15.42:0/3852564963 >> v2:172.21.15.40:3300/0 conn(0x7fb2cc0a7740 msgr2=0x7fb2cc08df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:49:19.551 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.541+0000 7fb2d0b97700 1 --2- 172.21.15.42:0/3852564963 >> v2:172.21.15.40:3300/0 conn(0x7fb2cc0a7740 0x7fb2cc08df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:49:19.551 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.541+0000 7fb2d0b97700 1 -- 172.21.15.42:0/3852564963 >> v1:172.21.15.40:6789/0 conn(0x7fb2cc09f2e0 legacy=0x7fb2cc09f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:49:19.551 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.541+0000 7fb2d0b97700 1 -- 172.21.15.42:0/3852564963 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2cc07f0a0 con 0x7fb2cc07e720 2019-12-05T05:49:19.551 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.541+0000 7fb2d1398700 1 --2- 172.21.15.42:0/3852564963 >> v2:172.21.15.40:3300/0 conn(0x7fb2cc0a7740 0x7fb2cc08df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T05:49:19.551 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.541+0000 7fb2d0b97700 1 --2- 172.21.15.42:0/3852564963 >> v2:172.21.15.42:3300/0 conn(0x7fb2cc07e720 0x7fb2cc07eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fb2c0002f30 tx=0x7fb2c000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:19.551 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.541+0000 7fb2d0b97700 1 --2- 172.21.15.42:0/3852564963 >> v2:172.21.15.42:3300/0 conn(0x7fb2cc07e720 0x7fb2cc07eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fb2c0002f30 tx=0x7fb2c000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:19.552 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.541+0000 7fb2d0b97700 1 --2- 172.21.15.42:0/3852564963 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb2cc07e720 0x7fb2cc07eb50 secure :-1 s=READY pgs=1513 cs=0 l=1 rx=0x7fb2c0002f30 tx=0x7fb2c000fd30).ready entity=mon.1 client_cookie=3c1e5518e6512531 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:49:19.552 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.541+0000 7fb2d0b97700 1 --2- 172.21.15.42:0/3852564963 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb2cc07e720 0x7fb2cc07eb50 secure :-1 s=THROTTLE_DONE pgs=1513 cs=0 l=1 rx=0x7fb2c0002f30 tx=0x7fb2c000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:19.552 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.541+0000 7fb2cbfff700 1 -- 172.21.15.42:0/3852564963 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb2c0011370 con 0x7fb2cc07e720 2019-12-05T05:49:19.552 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.541+0000 7fb2d0b97700 1 --2- 172.21.15.42:0/3852564963 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb2cc07e720 0x7fb2cc07eb50 secure :-1 s=THROTTLE_DONE pgs=1513 cs=0 l=1 rx=0x7fb2c0002f30 tx=0x7fb2c000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:19.552 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.541+0000 7fb2cbfff700 1 -- 172.21.15.42:0/3852564963 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2c0008bb0 con 0x7fb2cc07e720 2019-12-05T05:49:19.552 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.541+0000 7fb2d0b97700 1 --2- 172.21.15.42:0/3852564963 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb2cc07e720 0x7fb2cc07eb50 secure :-1 s=THROTTLE_DONE pgs=1513 cs=0 l=1 rx=0x7fb2c0002f30 tx=0x7fb2c000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:19.553 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.541+0000 7fb2cbfff700 1 -- 172.21.15.42:0/3852564963 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb2c0011950 con 0x7fb2cc07e720 2019-12-05T05:49:19.553 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.541+0000 7fb2d2dfb700 1 -- 172.21.15.42:0/3852564963 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb2cc07e720 msgr2=0x7fb2cc07eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:49:19.553 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.541+0000 7fb2d2dfb700 1 --2- 172.21.15.42:0/3852564963 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb2cc07e720 0x7fb2cc07eb50 secure :-1 s=READY pgs=1513 cs=0 l=1 rx=0x7fb2c0002f30 tx=0x7fb2c000fd30).stop 2019-12-05T05:49:19.553 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.545+0000 7fb2d2dfb700 1 -- 172.21.15.42:0/3852564963 shutdown_connections 2019-12-05T05:49:19.553 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.545+0000 7fb2d2dfb700 1 --2- 172.21.15.42:0/3852564963 >> v2:172.21.15.40:3300/0 conn(0x7fb2cc0a7740 0x7fb2cc08df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:49:19.553 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.545+0000 7fb2d2dfb700 1 --2- 172.21.15.42:0/3852564963 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb2cc07e720 0x7fb2cc07eb50 unknown :-1 s=CLOSED pgs=1513 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:49:19.553 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.545+0000 7fb2d2dfb700 1 -- 172.21.15.42:0/3852564963 shutdown_connections 2019-12-05T05:49:19.554 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.545+0000 7fb2d2dfb700 1 -- 172.21.15.42:0/3852564963 wait complete. 2019-12-05T05:49:19.554 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.545+0000 7fb2d2dfb700 1 -- 172.21.15.42:0/3852564963 >> 172.21.15.42:0/3852564963 conn(0x7fb2cc0aabe0 msgr2=0x7fb2cc0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:49:19.558 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.545+0000 7fb2d2dfb700 1 Processor -- start 2019-12-05T05:49:19.558 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.545+0000 7fb2d2dfb700 1 -- start start 2019-12-05T05:49:19.559 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.545+0000 7fb2d2dfb700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb2cc0a5ff0 0x7fb2cc131660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:49:19.559 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.545+0000 7fb2d2dfb700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2cc0a6420 con 0x7fb2cc08ee90 2019-12-05T05:49:19.559 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.545+0000 7fb2d2dfb700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2cc0a6760 con 0x7fb2cc0a5bc0 2019-12-05T05:49:19.559 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.545+0000 7fb2d2dfb700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2cc0a4a00 con 0x7fb2cc0a5ff0 2019-12-05T05:49:19.559 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.545+0000 7fb2d0b97700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fb2cc0a5bc0 0x7fb2cc0647f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:41912/0 (socket says 172.21.15.42:41912) 2019-12-05T05:49:19.559 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.545+0000 7fb2d0b97700 1 -- 172.21.15.42:0/1837545954 learned_addr learned my addr 172.21.15.42:0/1837545954 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:49:19.560 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.545+0000 7fb2c9ffb700 1 -- 172.21.15.42:0/1837545954 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3289078900 0 0) 0x7fb2cc0a6760 con 0x7fb2cc0a5bc0 2019-12-05T05:49:19.560 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.545+0000 7fb2c9ffb700 1 -- 172.21.15.42:0/1837545954 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2ac0030b0 con 0x7fb2cc0a5bc0 2019-12-05T05:49:19.560 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.545+0000 7fb2d1b99700 1 --2- 172.21.15.42:0/1837545954 >> v2:172.21.15.40:3300/0 conn(0x7fb2cc0a5ff0 0x7fb2cc131660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:49:19.560 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.545+0000 7fb2c9ffb700 1 -- 172.21.15.42:0/1837545954 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 1542919608 0 0) 0x7fb2ac0030b0 con 0x7fb2cc0a5bc0 2019-12-05T05:49:19.560 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.545+0000 7fb2c9ffb700 1 -- 172.21.15.42:0/1837545954 --> v1:172.21.15.42:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb2cc0a6760 con 0x7fb2cc0a5bc0 2019-12-05T05:49:19.563 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.545+0000 7fb2c9ffb700 1 -- 172.21.15.42:0/1837545954 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7fb2c0011370 con 0x7fb2cc0a5bc0 2019-12-05T05:49:19.563 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.545+0000 7fb2c9ffb700 1 -- 172.21.15.42:0/1837545954 >> v2:172.21.15.40:3300/0 conn(0x7fb2cc0a5ff0 msgr2=0x7fb2cc131660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:49:19.563 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.545+0000 7fb2c9ffb700 1 --2- 172.21.15.42:0/1837545954 >> v2:172.21.15.40:3300/0 conn(0x7fb2cc0a5ff0 0x7fb2cc131660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:49:19.563 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.545+0000 7fb2c9ffb700 1 -- 172.21.15.42:0/1837545954 >> v1:172.21.15.42:6789/0 conn(0x7fb2cc0a5bc0 legacy=0x7fb2cc0647f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:49:19.563 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.545+0000 7fb2c9ffb700 1 -- 172.21.15.42:0/1837545954 >> v1:172.21.15.40:6789/0 conn(0x7fb2cc08ee90 legacy=0x7fb2cc0a4f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:49:19.563 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.545+0000 7fb2c9ffb700 1 --2- 172.21.15.42:0/1837545954 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb2ac0038a0 0x7fb2ac003d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:49:19.564 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.545+0000 7fb2c9ffb700 1 --2- 172.21.15.42:0/1837545954 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb2ac004420 0x7fb2ac004850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:49:19.564 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.545+0000 7fb2c9ffb700 1 --2- 172.21.15.42:0/1837545954 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb2ac004eb0 0x7fb2ac00b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:49:19.564 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.545+0000 7fb2c9ffb700 1 -- 172.21.15.42:0/1837545954 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2cc0a4a00 con 0x7fb2ac004420 2019-12-05T05:49:19.564 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.545+0000 7fb2c9ffb700 1 -- 172.21.15.42:0/1837545954 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2ac00bc70 con 0x7fb2ac004eb0 2019-12-05T05:49:19.564 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.545+0000 7fb2c9ffb700 1 -- 172.21.15.42:0/1837545954 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2ac00beb0 con 0x7fb2ac0038a0 2019-12-05T05:49:19.565 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.545+0000 7fb2d1b99700 1 --2- 172.21.15.42:0/1837545954 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb2ac004eb0 0x7fb2ac00b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:49:19.565 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.549+0000 7fb2d0b97700 1 --2- 172.21.15.42:0/1837545954 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb2ac004420 0x7fb2ac004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:49:19.565 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.549+0000 7fb2d1b99700 1 -- 172.21.15.42:0/1837545954 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb2ac0038a0 msgr2=0x7fb2ac003d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:49:19.565 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.549+0000 7fb2d1398700 1 --2- 172.21.15.42:0/1837545954 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb2ac0038a0 0x7fb2ac003d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:49:19.565 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.549+0000 7fb2d1b99700 1 --2- 172.21.15.42:0/1837545954 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb2ac0038a0 0x7fb2ac003d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:49:19.566 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.549+0000 7fb2d1b99700 1 -- 172.21.15.42:0/1837545954 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb2ac004420 msgr2=0x7fb2ac004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:49:19.566 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.549+0000 7fb2d1b99700 1 --2- 172.21.15.42:0/1837545954 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb2ac004420 0x7fb2ac004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:49:19.566 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.549+0000 7fb2d1b99700 1 -- 172.21.15.42:0/1837545954 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2cc064ff0 con 0x7fb2ac004eb0 2019-12-05T05:49:19.566 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.549+0000 7fb2d2dfb700 1 -- 172.21.15.42:0/1837545954 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2cc129200 con 0x7fb2ac004eb0 2019-12-05T05:49:19.567 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.549+0000 7fb2d1b99700 1 --2- 172.21.15.42:0/1837545954 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb2ac004eb0 0x7fb2ac00b670 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fb2c4001210 tx=0x7fb2c4010fb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:19.567 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.549+0000 7fb2d1b99700 1 -- 172.21.15.42:0/1837545954 reap_dead start 2019-12-05T05:49:19.567 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.549+0000 7fb2d2dfb700 1 -- 172.21.15.42:0/1837545954 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2cc08ee90 con 0x7fb2ac004eb0 2019-12-05T05:49:19.567 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.549+0000 7fb2d1b99700 1 --2- 172.21.15.42:0/1837545954 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb2ac004eb0 0x7fb2ac00b670 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fb2c4001210 tx=0x7fb2c4010fb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:19.567 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.549+0000 7fb2d1b99700 1 --2- 172.21.15.42:0/1837545954 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb2ac004eb0 0x7fb2ac00b670 secure :-1 s=READY pgs=1515 cs=0 l=1 rx=0x7fb2c4001210 tx=0x7fb2c4010fb0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:49:19.567 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.549+0000 7fb2d1b99700 1 --2- 172.21.15.42:0/1837545954 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb2ac004eb0 0x7fb2ac00b670 secure :-1 s=THROTTLE_DONE pgs=1515 cs=0 l=1 rx=0x7fb2c4001210 tx=0x7fb2c4010fb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:19.568 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.549+0000 7fb2d1b99700 1 --2- 172.21.15.42:0/1837545954 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb2ac004eb0 0x7fb2ac00b670 secure :-1 s=THROTTLE_DONE pgs=1515 cs=0 l=1 rx=0x7fb2c4001210 tx=0x7fb2c4010fb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:19.568 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.549+0000 7fb2d1b99700 1 --2- 172.21.15.42:0/1837545954 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb2ac004eb0 0x7fb2ac00b670 secure :-1 s=THROTTLE_DONE pgs=1515 cs=0 l=1 rx=0x7fb2c4001210 tx=0x7fb2c4010fb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:19.568 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.549+0000 7fb2c9ffb700 1 -- 172.21.15.42:0/1837545954 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb2c4011ca0 con 0x7fb2ac004eb0 2019-12-05T05:49:19.568 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.549+0000 7fb2c9ffb700 1 -- 172.21.15.42:0/1837545954 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2c40118c0 con 0x7fb2ac004eb0 2019-12-05T05:49:19.568 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.549+0000 7fb2c9ffb700 1 -- 172.21.15.42:0/1837545954 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb2c40066e0 con 0x7fb2ac004eb0 2019-12-05T05:49:19.568 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.549+0000 7fb2d1b99700 1 --2- 172.21.15.42:0/1837545954 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb2ac004eb0 0x7fb2ac00b670 secure :-1 s=THROTTLE_DONE pgs=1515 cs=0 l=1 rx=0x7fb2c4001210 tx=0x7fb2c4010fb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:19.568 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.549+0000 7fb2c9ffb700 1 -- 172.21.15.42:0/1837545954 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fb2c40068e0 con 0x7fb2ac004eb0 2019-12-05T05:49:19.569 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.549+0000 7fb2d1b99700 1 --2- 172.21.15.42:0/1837545954 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb2ac004eb0 0x7fb2ac00b670 secure :-1 s=THROTTLE_DONE pgs=1515 cs=0 l=1 rx=0x7fb2c4001210 tx=0x7fb2c4010fb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:19.569 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.549+0000 7fb2c9ffb700 1 --2- 172.21.15.42:0/1837545954 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb2ac03e990 0x7fb2ac040ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:49:19.569 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.549+0000 7fb2c9ffb700 1 -- 172.21.15.42:0/1837545954 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 8697+0+0 (secure 0 0 0) 0x7fb2c404ad20 con 0x7fb2ac004eb0 2019-12-05T05:49:19.569 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.549+0000 7fb2d1398700 1 --2- 172.21.15.42:0/1837545954 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb2ac03e990 0x7fb2ac040ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:49:19.569 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.549+0000 7fb2d1398700 1 --2- 172.21.15.42:0/1837545954 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb2ac03e990 0x7fb2ac040ea0 crc :-1 s=READY pgs=653 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:49:19.570 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.557+0000 7fb2d2dfb700 1 -- 172.21.15.42:0/1837545954 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2a4003200 con 0x7fb2ac004eb0 2019-12-05T05:49:19.570 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.561+0000 7fb2d1b99700 1 --2- 172.21.15.42:0/1837545954 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb2ac004eb0 0x7fb2ac00b670 secure :-1 s=THROTTLE_DONE pgs=1515 cs=0 l=1 rx=0x7fb2c4001210 tx=0x7fb2c4010fb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:19.570 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.561+0000 7fb2c9ffb700 1 -- 172.21.15.42:0/1837545954 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fb2c4031da0 con 0x7fb2ac004eb0 2019-12-05T05:49:19.739 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.733+0000 7fb2d2dfb700 1 -- 172.21.15.42:0/1837545954 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb2a4003630 con 0x7fb2ac004eb0 2019-12-05T05:49:19.740 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.737+0000 7fb2d1b99700 1 --2- 172.21.15.42:0/1837545954 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb2ac004eb0 0x7fb2ac00b670 secure :-1 s=THROTTLE_DONE pgs=1515 cs=0 l=1 rx=0x7fb2c4001210 tx=0x7fb2c4010fb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:19.740 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.737+0000 7fb2c9ffb700 1 -- 172.21.15.42:0/1837545954 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 184 v184) v1 ==== 95+0+1774 (secure 0 0 0) 0x7fb2c4032000 con 0x7fb2ac004eb0 2019-12-05T05:49:19.741 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 184 2019-12-05T05:49:19.741 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:49:19.742 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":184,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":184,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:49:19.089508+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8188},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8188":{"gid":8188,"name":"a","rank":0,"incarnation":184,"state":"up:replay","state_seq":1,"addr":"172.21.15.42:6835/2456922143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2456922143},{"type":"v1","addr":"172.21.15.42:6835","nonce":2456922143}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:49:19.743 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.737+0000 7fb2d2dfb700 1 -- 172.21.15.42:0/1837545954 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb2ac03e990 msgr2=0x7fb2ac040ea0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:49:19.744 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.737+0000 7fb2d2dfb700 1 --2- 172.21.15.42:0/1837545954 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb2ac03e990 0x7fb2ac040ea0 crc :-1 s=READY pgs=653 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:49:19.744 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.737+0000 7fb2d2dfb700 1 -- 172.21.15.42:0/1837545954 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb2ac004eb0 msgr2=0x7fb2ac00b670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:49:19.744 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.737+0000 7fb2d2dfb700 1 --2- 172.21.15.42:0/1837545954 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb2ac004eb0 0x7fb2ac00b670 secure :-1 s=READY pgs=1515 cs=0 l=1 rx=0x7fb2c4001210 tx=0x7fb2c4010fb0).stop 2019-12-05T05:49:19.744 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.737+0000 7fb2d2dfb700 1 -- 172.21.15.42:0/1837545954 shutdown_connections 2019-12-05T05:49:19.745 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.737+0000 7fb2d2dfb700 1 --2- 172.21.15.42:0/1837545954 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb2ac03e990 0x7fb2ac040ea0 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:49:19.745 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.737+0000 7fb2d2dfb700 1 --2- 172.21.15.42:0/1837545954 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb2ac004eb0 0x7fb2ac00b670 unknown :-1 s=CLOSED pgs=1515 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:49:19.745 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.737+0000 7fb2d2dfb700 1 -- 172.21.15.42:0/1837545954 shutdown_connections 2019-12-05T05:49:19.745 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.737+0000 7fb2d2dfb700 1 -- 172.21.15.42:0/1837545954 wait complete. 2019-12-05T05:49:19.745 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:19.737+0000 7fb2d2dfb700 1 -- 172.21.15.42:0/1837545954 >> 172.21.15.42:0/1837545954 conn(0x7fb2cc0aabe0 msgr2=0x7fb2cc0a9800 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:49:19.753 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 184, "feature_flags": { "enable_multiple": false, "ever_enabled_multiple": false }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2019-12-05T05:03:35.604738+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 184, "ever_allowed_features": 2, "explicitly_allowed_features": 2, "failed": [], "flags": 18, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_8188": { "addr": "172.21.15.42:6835/2456922143", "addrs": { "addrvec": [ { "addr": "172.21.15.42:6834", "nonce": 2456922143, "type": "v2" }, { "addr": "172.21.15.42:6835", "nonce": 2456922143, "type": "v1" } ] }, "export_targets": [], "features": 4540138292836696063, "flags": 0, "gid": 8188, "incarnation": 184, "name": "a", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 80, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 2, "min_compat_client": "0 (unknown)", "modified": "2019-12-05T05:49:19.089508+0000", "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 8188 } } } ], "standbys": [] } 2019-12-05T05:49:21.755 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:49:21.855 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:21.845+0000 7f1ecf13e700 1 Processor -- start 2019-12-05T05:49:21.856 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:21.845+0000 7f1ecf13e700 1 -- start start 2019-12-05T05:49:21.856 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:21.845+0000 7f1ecf13e700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1ec80a7740 0x7f1ec808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:49:21.856 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:21.845+0000 7f1ecf13e700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f1ec807e720 0x7f1ec807eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:49:21.856 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:21.845+0000 7f1ecf13e700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ec80a0710 con 0x7f1ec809f2e0 2019-12-05T05:49:21.856 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:21.845+0000 7f1ecf13e700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ec80a0b00 con 0x7f1ec807e720 2019-12-05T05:49:21.856 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:21.845+0000 7f1ecf13e700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ec808e5b0 con 0x7f1ec80a7740 2019-12-05T05:49:21.857 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:21.845+0000 7f1ecdedc700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f1ec809f2e0 0x7f1ec809f710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:41932/0 (socket says 172.21.15.42:41932) 2019-12-05T05:49:21.857 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:21.845+0000 7f1ecdedc700 1 -- 172.21.15.42:0/3067818646 learned_addr learned my addr 172.21.15.42:0/3067818646 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:49:21.857 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:21.849+0000 7f1ecd6db700 1 --2- 172.21.15.42:0/3067818646 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1ec80a7740 0x7f1ec808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:49:21.857 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:21.849+0000 7f1ecceda700 1 --2- 172.21.15.42:0/3067818646 >> v2:172.21.15.40:3300/0 conn(0x7f1ec807e720 0x7f1ec807eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:49:21.857 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:21.849+0000 7f1ebffff700 1 -- 172.21.15.42:0/3067818646 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4119580941 0 0) 0x7f1ec80a0710 con 0x7f1ec809f2e0 2019-12-05T05:49:21.857 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:21.849+0000 7f1ebffff700 1 -- 172.21.15.42:0/3067818646 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1eb40030b0 con 0x7f1ec809f2e0 2019-12-05T05:49:21.858 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:21.849+0000 7f1ecceda700 1 -- 172.21.15.42:0/3067818646 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1ec80a7740 msgr2=0x7f1ec808df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:49:21.858 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:21.849+0000 7f1ecceda700 1 --2- 172.21.15.42:0/3067818646 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1ec80a7740 0x7f1ec808df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:49:21.858 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:21.849+0000 7f1ecceda700 1 -- 172.21.15.42:0/3067818646 >> v1:172.21.15.42:6789/0 conn(0x7f1ec809f2e0 legacy=0x7f1ec809f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:49:21.858 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:21.849+0000 7f1ecceda700 1 -- 172.21.15.42:0/3067818646 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ec807f0a0 con 0x7f1ec807e720 2019-12-05T05:49:21.858 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:21.849+0000 7f1ecceda700 1 --2- 172.21.15.42:0/3067818646 >> v2:172.21.15.40:3300/0 conn(0x7f1ec807e720 0x7f1ec807eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f1eb8008790 tx=0x7f1eb800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:21.858 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:21.849+0000 7f1ecceda700 1 --2- 172.21.15.42:0/3067818646 >> v2:172.21.15.40:3300/0 conn(0x7f1ec807e720 0x7f1ec807eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f1eb8008790 tx=0x7f1eb800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:21.859 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:21.849+0000 7f1ecceda700 1 --2- 172.21.15.42:0/3067818646 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ec807e720 0x7f1ec807eb50 secure :-1 s=READY pgs=1168 cs=0 l=1 rx=0x7f1eb8008790 tx=0x7f1eb800fc70).ready entity=mon.0 client_cookie=b3c1eff033dbfe3 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:49:21.859 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:21.849+0000 7f1ecceda700 1 --2- 172.21.15.42:0/3067818646 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ec807e720 0x7f1ec807eb50 secure :-1 s=THROTTLE_DONE pgs=1168 cs=0 l=1 rx=0x7f1eb8008790 tx=0x7f1eb800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:21.859 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:21.849+0000 7f1ebffff700 1 -- 172.21.15.42:0/3067818646 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1eb80112f0 con 0x7f1ec807e720 2019-12-05T05:49:21.859 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:21.849+0000 7f1ecceda700 1 --2- 172.21.15.42:0/3067818646 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ec807e720 0x7f1ec807eb50 secure :-1 s=THROTTLE_DONE pgs=1168 cs=0 l=1 rx=0x7f1eb8008790 tx=0x7f1eb800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:21.859 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:21.849+0000 7f1ebffff700 1 -- 172.21.15.42:0/3067818646 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1eb8008ad0 con 0x7f1ec807e720 2019-12-05T05:49:21.860 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:21.849+0000 7f1ecceda700 1 --2- 172.21.15.42:0/3067818646 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ec807e720 0x7f1ec807eb50 secure :-1 s=THROTTLE_DONE pgs=1168 cs=0 l=1 rx=0x7f1eb8008790 tx=0x7f1eb800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:21.860 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:21.849+0000 7f1ebffff700 1 -- 172.21.15.42:0/3067818646 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1eb80118d0 con 0x7f1ec807e720 2019-12-05T05:49:21.860 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:21.849+0000 7f1ecf13e700 1 -- 172.21.15.42:0/3067818646 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ec807e720 msgr2=0x7f1ec807eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:49:21.860 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:21.849+0000 7f1ecf13e700 1 --2- 172.21.15.42:0/3067818646 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ec807e720 0x7f1ec807eb50 secure :-1 s=READY pgs=1168 cs=0 l=1 rx=0x7f1eb8008790 tx=0x7f1eb800fc70).stop 2019-12-05T05:49:21.860 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:21.849+0000 7f1ecf13e700 1 -- 172.21.15.42:0/3067818646 shutdown_connections 2019-12-05T05:49:21.861 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:21.849+0000 7f1ecf13e700 1 --2- 172.21.15.42:0/3067818646 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1ec80a7740 0x7f1ec808df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:49:21.861 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:21.849+0000 7f1ecf13e700 1 --2- 172.21.15.42:0/3067818646 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ec807e720 0x7f1ec807eb50 unknown :-1 s=CLOSED pgs=1168 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:49:21.861 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:21.849+0000 7f1ecf13e700 1 -- 172.21.15.42:0/3067818646 shutdown_connections 2019-12-05T05:49:21.861 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:21.849+0000 7f1ecf13e700 1 -- 172.21.15.42:0/3067818646 wait complete. 2019-12-05T05:49:21.861 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:21.849+0000 7f1ecf13e700 1 -- 172.21.15.42:0/3067818646 >> 172.21.15.42:0/3067818646 conn(0x7f1ec80aabe0 msgr2=0x7f1ec80a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:49:21.864 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:21.853+0000 7f1ecf13e700 1 Processor -- start 2019-12-05T05:49:21.864 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:21.853+0000 7f1ecf13e700 1 -- start start 2019-12-05T05:49:21.865 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:21.853+0000 7f1ecf13e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f1ec8116500 0x7f1ec811b050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:49:21.865 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:21.853+0000 7f1ecf13e700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ec8070ac0 con 0x7f1ec8118990 2019-12-05T05:49:21.865 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:21.853+0000 7f1ecf13e700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ec8070cf0 con 0x7f1ec811b5a0 2019-12-05T05:49:21.865 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:21.853+0000 7f1ecf13e700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ec81181f0 con 0x7f1ec8116500 2019-12-05T05:49:21.865 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:21.853+0000 7f1ecceda700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f1ec8116500 0x7f1ec811b050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:49:21.865 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:21.853+0000 7f1ecceda700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f1ec8116500 0x7f1ec811b050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:41898/0 (socket says 172.21.15.42:41898) 2019-12-05T05:49:21.866 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:21.853+0000 7f1ecceda700 1 -- 172.21.15.42:0/3540821774 learned_addr learned my addr 172.21.15.42:0/3540821774 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:49:21.866 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:21.853+0000 7f1ecceda700 1 -- 172.21.15.42:0/3540821774 >> v1:172.21.15.42:6789/0 conn(0x7f1ec811b5a0 legacy=0x7f1ec80702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:49:21.866 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:21.853+0000 7f1ecceda700 1 -- 172.21.15.42:0/3540821774 >> v1:172.21.15.40:6789/0 conn(0x7f1ec8118990 legacy=0x7f1ec8116ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:49:21.866 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:21.853+0000 7f1ecceda700 1 -- 172.21.15.42:0/3540821774 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ec8070f20 con 0x7f1ec8116500 2019-12-05T05:49:21.866 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:21.853+0000 7f1ecceda700 1 --2- 172.21.15.42:0/3540821774 >> v2:172.21.15.42:3300/0 conn(0x7f1ec8116500 0x7f1ec811b050 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f1eb8008790 tx=0x7f1eb8011de0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:21.866 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:21.853+0000 7f1ecceda700 1 --2- 172.21.15.42:0/3540821774 >> v2:172.21.15.42:3300/0 conn(0x7f1ec8116500 0x7f1ec811b050 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f1eb8008790 tx=0x7f1eb8011de0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:21.867 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:21.853+0000 7f1ecceda700 1 --2- 172.21.15.42:0/3540821774 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1ec8116500 0x7f1ec811b050 secure :-1 s=READY pgs=1518 cs=0 l=1 rx=0x7f1eb8008790 tx=0x7f1eb8011de0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:49:21.867 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:21.853+0000 7f1ecceda700 1 --2- 172.21.15.42:0/3540821774 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1ec8116500 0x7f1ec811b050 secure :-1 s=THROTTLE_DONE pgs=1518 cs=0 l=1 rx=0x7f1eb8008790 tx=0x7f1eb8011de0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:21.867 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:21.853+0000 7f1ebdffb700 1 -- 172.21.15.42:0/3540821774 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1eb8007150 con 0x7f1ec8116500 2019-12-05T05:49:21.867 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:21.853+0000 7f1ecceda700 1 --2- 172.21.15.42:0/3540821774 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1ec8116500 0x7f1ec811b050 secure :-1 s=THROTTLE_DONE pgs=1518 cs=0 l=1 rx=0x7f1eb8008790 tx=0x7f1eb8011de0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:21.867 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:21.853+0000 7f1ecf13e700 1 -- 172.21.15.42:0/3540821774 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ec8078540 con 0x7f1ec8116500 2019-12-05T05:49:21.868 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:21.853+0000 7f1ebdffb700 1 -- 172.21.15.42:0/3540821774 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1eb80114f0 con 0x7f1ec8116500 2019-12-05T05:49:21.868 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:21.853+0000 7f1ecceda700 1 --2- 172.21.15.42:0/3540821774 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1ec8116500 0x7f1ec811b050 secure :-1 s=THROTTLE_DONE pgs=1518 cs=0 l=1 rx=0x7f1eb8008790 tx=0x7f1eb8011de0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:21.868 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:21.853+0000 7f1ebdffb700 1 -- 172.21.15.42:0/3540821774 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1eb8008cc0 con 0x7f1ec8116500 2019-12-05T05:49:21.868 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:21.853+0000 7f1ecf13e700 1 -- 172.21.15.42:0/3540821774 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ec8078a90 con 0x7f1ec8116500 2019-12-05T05:49:21.868 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:21.857+0000 7f1ecceda700 1 --2- 172.21.15.42:0/3540821774 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1ec8116500 0x7f1ec811b050 secure :-1 s=THROTTLE_DONE pgs=1518 cs=0 l=1 rx=0x7f1eb8008790 tx=0x7f1eb8011de0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:21.869 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:21.857+0000 7f1ebdffb700 1 -- 172.21.15.42:0/3540821774 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f1eb8005540 con 0x7f1ec8116500 2019-12-05T05:49:21.869 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:21.857+0000 7f1ecceda700 1 --2- 172.21.15.42:0/3540821774 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1ec8116500 0x7f1ec811b050 secure :-1 s=THROTTLE_DONE pgs=1518 cs=0 l=1 rx=0x7f1eb8008790 tx=0x7f1eb8011de0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:21.869 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:21.857+0000 7f1ebdffb700 1 --2- 172.21.15.42:0/3540821774 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f1ea803acb0 0x7f1ea803d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:49:21.869 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:21.857+0000 7f1ebdffb700 1 -- 172.21.15.42:0/3540821774 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 8697+0+0 (secure 0 0 0) 0x7f1eb804b960 con 0x7f1ec8116500 2019-12-05T05:49:21.869 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:21.857+0000 7f1ecd6db700 1 --2- 172.21.15.42:0/3540821774 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f1ea803acb0 0x7f1ea803d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:49:21.869 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:21.857+0000 7f1ecd6db700 1 --2- 172.21.15.42:0/3540821774 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f1ea803acb0 0x7f1ea803d1c0 crc :-1 s=READY pgs=654 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:49:21.870 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:21.861+0000 7f1ecf13e700 1 -- 172.21.15.42:0/3540821774 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ec805d110 con 0x7f1ec8116500 2019-12-05T05:49:21.872 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:21.869+0000 7f1ecceda700 1 --2- 172.21.15.42:0/3540821774 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1ec8116500 0x7f1ec811b050 secure :-1 s=THROTTLE_DONE pgs=1518 cs=0 l=1 rx=0x7f1eb8008790 tx=0x7f1eb8011de0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:21.872 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:21.869+0000 7f1ebdffb700 1 -- 172.21.15.42:0/3540821774 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f1eb8032c20 con 0x7f1ec8116500 2019-12-05T05:49:22.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:22.033+0000 7f1ecf13e700 1 -- 172.21.15.42:0/3540821774 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1ec80a6990 con 0x7f1ec8116500 2019-12-05T05:49:22.039 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:22.033+0000 7f1ecceda700 1 --2- 172.21.15.42:0/3540821774 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1ec8116500 0x7f1ec811b050 secure :-1 s=THROTTLE_DONE pgs=1518 cs=0 l=1 rx=0x7f1eb8008790 tx=0x7f1eb8011de0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:22.039 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:22.033+0000 7f1ebdffb700 1 -- 172.21.15.42:0/3540821774 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 185 v185) v1 ==== 95+0+1778 (secure 0 0 0) 0x7f1eb80305d0 con 0x7f1ec8116500 2019-12-05T05:49:22.041 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 185 2019-12-05T05:49:22.041 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:49:22.042 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":185,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":185,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:49:21.668163+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8188},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8188":{"gid":8188,"name":"a","rank":0,"incarnation":184,"state":"up:reconnect","state_seq":31,"addr":"172.21.15.42:6835/2456922143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2456922143},{"type":"v1","addr":"172.21.15.42:6835","nonce":2456922143}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:49:22.044 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:22.041+0000 7f1ecf13e700 1 -- 172.21.15.42:0/3540821774 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f1ea803acb0 msgr2=0x7f1ea803d1c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:49:22.044 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:22.041+0000 7f1ecf13e700 1 --2- 172.21.15.42:0/3540821774 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f1ea803acb0 0x7f1ea803d1c0 crc :-1 s=READY pgs=654 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:49:22.044 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:22.041+0000 7f1ecf13e700 1 -- 172.21.15.42:0/3540821774 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1ec8116500 msgr2=0x7f1ec811b050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:49:22.044 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:22.041+0000 7f1ecf13e700 1 --2- 172.21.15.42:0/3540821774 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1ec8116500 0x7f1ec811b050 secure :-1 s=READY pgs=1518 cs=0 l=1 rx=0x7f1eb8008790 tx=0x7f1eb8011de0).stop 2019-12-05T05:49:22.044 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:22.041+0000 7f1ecf13e700 1 -- 172.21.15.42:0/3540821774 shutdown_connections 2019-12-05T05:49:22.045 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:22.041+0000 7f1ecf13e700 1 --2- 172.21.15.42:0/3540821774 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f1ea803acb0 0x7f1ea803d1c0 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:49:22.045 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:22.041+0000 7f1ecf13e700 1 --2- 172.21.15.42:0/3540821774 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1ec8116500 0x7f1ec811b050 unknown :-1 s=CLOSED pgs=1518 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:49:22.045 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:22.041+0000 7f1ecf13e700 1 -- 172.21.15.42:0/3540821774 shutdown_connections 2019-12-05T05:49:22.045 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:22.041+0000 7f1ecf13e700 1 -- 172.21.15.42:0/3540821774 wait complete. 2019-12-05T05:49:22.045 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:22.041+0000 7f1ecf13e700 1 -- 172.21.15.42:0/3540821774 >> 172.21.15.42:0/3540821774 conn(0x7f1ec807e870 msgr2=0x7f1ec811c540 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:49:22.058 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2019-12-05T05:49:24.059 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:49:24.171 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:24.161+0000 7efeb5f7d700 1 Processor -- start 2019-12-05T05:49:24.172 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:24.161+0000 7efeb5f7d700 1 -- start start 2019-12-05T05:49:24.172 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:24.165+0000 7efeb5f7d700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7efeb00a7730 0x7efeb008df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:49:24.172 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:24.165+0000 7efeb5f7d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7efeb007e710 0x7efeb007eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:49:24.172 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:24.165+0000 7efeb5f7d700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efeb009f2d0 0x7efeb009f700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:49:24.172 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:24.165+0000 7efeb5f7d700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efeb00a0af0 con 0x7efeb00a7730 2019-12-05T05:49:24.173 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:24.165+0000 7efeb5f7d700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efeb008e5a0 con 0x7efeb007e710 2019-12-05T05:49:24.173 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:24.165+0000 7efeb5f7d700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efeb008e840 con 0x7efeb009f2d0 2019-12-05T05:49:24.173 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:24.165+0000 7efeaf7fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7efeb007e710 0x7efeb007eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:49:24.173 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:24.165+0000 7efeaf7fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7efeb007e710 0x7efeb007eb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:41910/0 (socket says 172.21.15.42:41910) 2019-12-05T05:49:24.173 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:24.165+0000 7efeaf7fe700 1 -- 172.21.15.42:0/709953706 learned_addr learned my addr 172.21.15.42:0/709953706 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:49:24.173 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:24.165+0000 7efeaffff700 1 --2- 172.21.15.42:0/709953706 >> v2:172.21.15.40:3300/0 conn(0x7efeb00a7730 0x7efeb008df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:49:24.173 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:24.165+0000 7efeb4d1b700 1 --2- 172.21.15.42:0/709953706 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efeb009f2d0 0x7efeb009f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:49:24.174 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:24.165+0000 7efeaf7fe700 1 -- 172.21.15.42:0/709953706 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efeb009f2d0 msgr2=0x7efeb009f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:49:24.174 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:24.165+0000 7efeaf7fe700 1 --2- 172.21.15.42:0/709953706 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efeb009f2d0 0x7efeb009f700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:49:24.174 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:24.165+0000 7efeaf7fe700 1 -- 172.21.15.42:0/709953706 >> v2:172.21.15.40:3300/0 conn(0x7efeb00a7730 msgr2=0x7efeb008df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:49:24.174 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:24.165+0000 7efeaf7fe700 1 --2- 172.21.15.42:0/709953706 >> v2:172.21.15.40:3300/0 conn(0x7efeb00a7730 0x7efeb008df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:49:24.174 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:24.165+0000 7efeaf7fe700 1 -- 172.21.15.42:0/709953706 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efeb007f090 con 0x7efeb007e710 2019-12-05T05:49:24.174 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:24.165+0000 7efeaf7fe700 1 --2- 172.21.15.42:0/709953706 >> v2:172.21.15.42:3300/0 conn(0x7efeb007e710 0x7efeb007eb40 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7efeb00a0f40 tx=0x7efea400fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:24.175 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:24.165+0000 7efeaf7fe700 1 --2- 172.21.15.42:0/709953706 >> v2:172.21.15.42:3300/0 conn(0x7efeb007e710 0x7efeb007eb40 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7efeb00a0f40 tx=0x7efea400fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:24.175 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:24.165+0000 7efeaf7fe700 1 --2- 172.21.15.42:0/709953706 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efeb007e710 0x7efeb007eb40 secure :-1 s=READY pgs=1519 cs=0 l=1 rx=0x7efeb00a0f40 tx=0x7efea400fdb0).ready entity=mon.1 client_cookie=d430f79d64e0962c server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:49:24.175 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:24.165+0000 7efeaf7fe700 1 --2- 172.21.15.42:0/709953706 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efeb007e710 0x7efeb007eb40 secure :-1 s=THROTTLE_DONE pgs=1519 cs=0 l=1 rx=0x7efeb00a0f40 tx=0x7efea400fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:24.175 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:24.165+0000 7efeaeffd700 1 -- 172.21.15.42:0/709953706 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efea4008c90 con 0x7efeb007e710 2019-12-05T05:49:24.175 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:24.165+0000 7efeaf7fe700 1 --2- 172.21.15.42:0/709953706 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efeb007e710 0x7efeb007eb40 secure :-1 s=THROTTLE_DONE pgs=1519 cs=0 l=1 rx=0x7efeb00a0f40 tx=0x7efea400fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:24.175 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:24.165+0000 7efeaf7fe700 1 --2- 172.21.15.42:0/709953706 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efeb007e710 0x7efeb007eb40 secure :-1 s=THROTTLE_DONE pgs=1519 cs=0 l=1 rx=0x7efeb00a0f40 tx=0x7efea400fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:24.176 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:24.165+0000 7efeaeffd700 1 -- 172.21.15.42:0/709953706 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efea40088b0 con 0x7efeb007e710 2019-12-05T05:49:24.176 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:24.165+0000 7efeaeffd700 1 -- 172.21.15.42:0/709953706 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efea4011b50 con 0x7efeb007e710 2019-12-05T05:49:24.176 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:24.165+0000 7efeb5f7d700 1 -- 172.21.15.42:0/709953706 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efeb007e710 msgr2=0x7efeb007eb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:49:24.176 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:24.165+0000 7efeb5f7d700 1 --2- 172.21.15.42:0/709953706 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efeb007e710 0x7efeb007eb40 secure :-1 s=READY pgs=1519 cs=0 l=1 rx=0x7efeb00a0f40 tx=0x7efea400fdb0).stop 2019-12-05T05:49:24.176 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:24.165+0000 7efeb5f7d700 1 -- 172.21.15.42:0/709953706 shutdown_connections 2019-12-05T05:49:24.176 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:24.165+0000 7efeb5f7d700 1 --2- 172.21.15.42:0/709953706 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efeb009f2d0 0x7efeb009f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:49:24.176 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:24.165+0000 7efeb5f7d700 1 --2- 172.21.15.42:0/709953706 >> v2:172.21.15.40:3300/0 conn(0x7efeb00a7730 0x7efeb008df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:49:24.177 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:24.165+0000 7efeb5f7d700 1 --2- 172.21.15.42:0/709953706 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efeb007e710 0x7efeb007eb40 unknown :-1 s=CLOSED pgs=1519 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:49:24.177 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:24.165+0000 7efeb5f7d700 1 -- 172.21.15.42:0/709953706 shutdown_connections 2019-12-05T05:49:24.177 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:24.165+0000 7efeb5f7d700 1 -- 172.21.15.42:0/709953706 wait complete. 2019-12-05T05:49:24.177 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:24.165+0000 7efeb5f7d700 1 -- 172.21.15.42:0/709953706 >> 172.21.15.42:0/709953706 conn(0x7efeb00aabd0 msgr2=0x7efeb00a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:49:24.180 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:24.169+0000 7efeb5f7d700 1 Processor -- start 2019-12-05T05:49:24.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:24.169+0000 7efeb5f7d700 1 -- start start 2019-12-05T05:49:24.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:24.169+0000 7efeb5f7d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7efeb007b480 0x7efeb00a6a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:49:24.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:24.169+0000 7efeb5f7d700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7efeb011ab50 0x7efeb01386c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:49:24.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:24.169+0000 7efeb5f7d700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efeb0131a70 con 0x7efeb011a720 2019-12-05T05:49:24.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:24.169+0000 7efeb5f7d700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efeb011af80 con 0x7efeb011ab50 2019-12-05T05:49:24.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:24.169+0000 7efeb5f7d700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efeb0120e00 con 0x7efeb007b480 2019-12-05T05:49:24.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:24.169+0000 7efeaffff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7efeb007b480 0x7efeb00a6a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:49:24.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:24.169+0000 7efeaffff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7efeb007b480 0x7efeb00a6a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:41916/0 (socket says 172.21.15.42:41916) 2019-12-05T05:49:24.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:24.169+0000 7efeaffff700 1 -- 172.21.15.42:0/2693969382 learned_addr learned my addr 172.21.15.42:0/2693969382 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:49:24.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:24.169+0000 7efeaf7fe700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7efeb011a720 0x7efeb0131270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:41954/0 (socket says 172.21.15.42:41954) 2019-12-05T05:49:24.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:24.169+0000 7efeb4d1b700 1 --2- 172.21.15.42:0/2693969382 >> v2:172.21.15.40:3300/0 conn(0x7efeb011ab50 0x7efeb01386c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:49:24.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:24.169+0000 7efeacff9700 1 -- 172.21.15.42:0/2693969382 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2343540361 0 0) 0x7efeb0131a70 con 0x7efeb011a720 2019-12-05T05:49:24.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:24.173+0000 7efeacff9700 1 -- 172.21.15.42:0/2693969382 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe900030b0 con 0x7efeb011a720 2019-12-05T05:49:24.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:24.173+0000 7efeb4d1b700 1 -- 172.21.15.42:0/2693969382 >> v2:172.21.15.42:3300/0 conn(0x7efeb007b480 msgr2=0x7efeb00a6a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:49:24.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:24.173+0000 7efeb4d1b700 1 --2- 172.21.15.42:0/2693969382 >> v2:172.21.15.42:3300/0 conn(0x7efeb007b480 0x7efeb00a6a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:49:24.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:24.173+0000 7efeb4d1b700 1 -- 172.21.15.42:0/2693969382 >> v1:172.21.15.42:6789/0 conn(0x7efeb011a720 legacy=0x7efeb0131270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:49:24.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:24.173+0000 7efeb4d1b700 1 -- 172.21.15.42:0/2693969382 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efeb0120af0 con 0x7efeb011ab50 2019-12-05T05:49:24.184 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:24.173+0000 7efeaffff700 1 --2- 172.21.15.42:0/2693969382 >> v2:172.21.15.42:3300/0 conn(0x7efeb007b480 0x7efeb00a6a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:49:24.184 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:24.173+0000 7efeb4d1b700 1 --2- 172.21.15.42:0/2693969382 >> v2:172.21.15.40:3300/0 conn(0x7efeb011ab50 0x7efeb01386c0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7efea80032b0 tx=0x7efea8010d00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:24.184 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:24.173+0000 7efeb4d1b700 1 --2- 172.21.15.42:0/2693969382 >> v2:172.21.15.40:3300/0 conn(0x7efeb011ab50 0x7efeb01386c0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7efea80032b0 tx=0x7efea8010d00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:24.184 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:24.173+0000 7efeb4d1b700 1 --2- 172.21.15.42:0/2693969382 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efeb011ab50 0x7efeb01386c0 secure :-1 s=READY pgs=1170 cs=0 l=1 rx=0x7efea80032b0 tx=0x7efea8010d00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:49:24.184 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:24.173+0000 7efeb4d1b700 1 --2- 172.21.15.42:0/2693969382 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efeb011ab50 0x7efeb01386c0 secure :-1 s=THROTTLE_DONE pgs=1170 cs=0 l=1 rx=0x7efea80032b0 tx=0x7efea8010d00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:24.184 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:24.173+0000 7efeacff9700 1 -- 172.21.15.42:0/2693969382 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efea8012a30 con 0x7efeb011ab50 2019-12-05T05:49:24.184 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:24.173+0000 7efeb4d1b700 1 --2- 172.21.15.42:0/2693969382 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efeb011ab50 0x7efeb01386c0 secure :-1 s=THROTTLE_DONE pgs=1170 cs=0 l=1 rx=0x7efea80032b0 tx=0x7efea8010d00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:24.185 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:24.173+0000 7efeb4d1b700 1 --2- 172.21.15.42:0/2693969382 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efeb011ab50 0x7efeb01386c0 secure :-1 s=THROTTLE_DONE pgs=1170 cs=0 l=1 rx=0x7efea80032b0 tx=0x7efea8010d00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:24.185 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:24.173+0000 7efeacff9700 1 -- 172.21.15.42:0/2693969382 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efea8012650 con 0x7efeb011ab50 2019-12-05T05:49:24.185 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:24.173+0000 7efeacff9700 1 -- 172.21.15.42:0/2693969382 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efea800e760 con 0x7efeb011ab50 2019-12-05T05:49:24.185 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:24.173+0000 7efeb5f7d700 1 -- 172.21.15.42:0/2693969382 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efeb00a6f90 con 0x7efeb011ab50 2019-12-05T05:49:24.185 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:24.173+0000 7efeb5f7d700 1 -- 172.21.15.42:0/2693969382 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efeb01215e0 con 0x7efeb011ab50 2019-12-05T05:49:24.185 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:24.173+0000 7efeb4d1b700 1 --2- 172.21.15.42:0/2693969382 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efeb011ab50 0x7efeb01386c0 secure :-1 s=THROTTLE_DONE pgs=1170 cs=0 l=1 rx=0x7efea80032b0 tx=0x7efea8010d00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:24.185 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:24.173+0000 7efeacff9700 1 -- 172.21.15.42:0/2693969382 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7efea800ea00 con 0x7efeb011ab50 2019-12-05T05:49:24.186 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:24.173+0000 7efeb4d1b700 1 --2- 172.21.15.42:0/2693969382 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efeb011ab50 0x7efeb01386c0 secure :-1 s=THROTTLE_DONE pgs=1170 cs=0 l=1 rx=0x7efea80032b0 tx=0x7efea8010d00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:24.186 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:24.177+0000 7efeacff9700 1 --2- 172.21.15.42:0/2693969382 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7efe9003b280 0x7efe9003d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:49:24.186 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:24.177+0000 7efeacff9700 1 -- 172.21.15.42:0/2693969382 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 8697+0+0 (secure 0 0 0) 0x7efea804b7a0 con 0x7efeb011ab50 2019-12-05T05:49:24.186 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:24.177+0000 7efeaffff700 1 --2- 172.21.15.42:0/2693969382 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7efe9003b280 0x7efe9003d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:49:24.186 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:24.177+0000 7efeaffff700 1 --2- 172.21.15.42:0/2693969382 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7efe9003b280 0x7efe9003d790 crc :-1 s=READY pgs=655 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:49:24.187 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:24.177+0000 7efeb5f7d700 1 -- 172.21.15.42:0/2693969382 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efeb005d100 con 0x7efeb011ab50 2019-12-05T05:49:24.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:24.185+0000 7efeb4d1b700 1 --2- 172.21.15.42:0/2693969382 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efeb011ab50 0x7efeb01386c0 secure :-1 s=THROTTLE_DONE pgs=1170 cs=0 l=1 rx=0x7efea80032b0 tx=0x7efea8010d00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:24.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:24.189+0000 7efeacff9700 1 -- 172.21.15.42:0/2693969382 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7efea8032800 con 0x7efeb011ab50 2019-12-05T05:49:24.360 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:24.357+0000 7efeb5f7d700 1 -- 172.21.15.42:0/2693969382 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efeb005b8b0 con 0x7efeb011ab50 2019-12-05T05:49:24.362 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:24.357+0000 7efeb4d1b700 1 --2- 172.21.15.42:0/2693969382 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efeb011ab50 0x7efeb01386c0 secure :-1 s=THROTTLE_DONE pgs=1170 cs=0 l=1 rx=0x7efea80032b0 tx=0x7efea8010d00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:24.362 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:24.357+0000 7efeacff9700 1 -- 172.21.15.42:0/2693969382 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 187 v187) v1 ==== 95+0+1775 (secure 0 0 0) 0x7efea8032a60 con 0x7efeb011ab50 2019-12-05T05:49:24.363 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 187 2019-12-05T05:49:24.364 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:49:24.364 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":187,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":187,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:49:23.679613+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8188},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8188":{"gid":8188,"name":"a","rank":0,"incarnation":184,"state":"up:active","state_seq":33,"addr":"172.21.15.42:6835/2456922143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2456922143},{"type":"v1","addr":"172.21.15.42:6835","nonce":2456922143}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:49:24.368 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:24.365+0000 7efeb5f7d700 1 -- 172.21.15.42:0/2693969382 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7efe9003b280 msgr2=0x7efe9003d790 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:49:24.368 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:24.365+0000 7efeb5f7d700 1 --2- 172.21.15.42:0/2693969382 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7efe9003b280 0x7efe9003d790 crc :-1 s=READY pgs=655 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:49:24.368 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:24.365+0000 7efeb5f7d700 1 -- 172.21.15.42:0/2693969382 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efeb011ab50 msgr2=0x7efeb01386c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:49:24.369 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:24.365+0000 7efeb5f7d700 1 --2- 172.21.15.42:0/2693969382 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efeb011ab50 0x7efeb01386c0 secure :-1 s=READY pgs=1170 cs=0 l=1 rx=0x7efea80032b0 tx=0x7efea8010d00).stop 2019-12-05T05:49:24.369 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:24.365+0000 7efeb5f7d700 1 -- 172.21.15.42:0/2693969382 shutdown_connections 2019-12-05T05:49:24.369 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:24.365+0000 7efeb5f7d700 1 --2- 172.21.15.42:0/2693969382 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7efe9003b280 0x7efe9003d790 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:49:24.369 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:24.365+0000 7efeb5f7d700 1 --2- 172.21.15.42:0/2693969382 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efeb011ab50 0x7efeb01386c0 unknown :-1 s=CLOSED pgs=1170 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:49:24.370 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:24.365+0000 7efeb5f7d700 1 --2- 172.21.15.42:0/2693969382 >> v2:172.21.15.42:3300/0 conn(0x7efeb007b480 0x7efeb00a6a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:49:24.370 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:24.365+0000 7efeb5f7d700 1 -- 172.21.15.42:0/2693969382 shutdown_connections 2019-12-05T05:49:24.371 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:24.365+0000 7efeb5f7d700 1 -- 172.21.15.42:0/2693969382 wait complete. 2019-12-05T05:49:24.372 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:24.365+0000 7efeb5f7d700 1 -- 172.21.15.42:0/2693969382 >> 172.21.15.42:0/2693969382 conn(0x7efeb008fd90 msgr2=0x7efeb00596f0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:49:24.381 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a has gained rank=0, replacing gid=6140 2019-12-05T05:49:24.381 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 3 secs before reviving mds.b 2019-12-05T05:49:27.381 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.b 2019-12-05T05:49:27.382 INFO:tasks.ceph.mds.b:Restarting daemon 2019-12-05T05:49:27.382 INFO:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2019-12-05T05:49:27.386 INFO:tasks.ceph.mds.b:Started 2019-12-05T05:49:27.386 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:49:27.446 INFO:tasks.ceph.mds.b.smithi040.stdout:starting mds.b at 2019-12-05T05:49:27.493 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:27.485+0000 7f38a0507700 1 Processor -- start 2019-12-05T05:49:27.493 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:27.485+0000 7f38a0507700 1 -- start start 2019-12-05T05:49:27.493 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:27.485+0000 7f38a0507700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f389808df80 0x7f389809f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:49:27.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:27.485+0000 7f38a0507700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f389809f830 0x7f389814efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:49:27.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:27.485+0000 7f38a0507700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f389809fc60 con 0x7f38980a7740 2019-12-05T05:49:27.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:27.485+0000 7f38a0507700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38980a0b00 con 0x7f389808df80 2019-12-05T05:49:27.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:27.485+0000 7f38a0507700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f389808e5b0 con 0x7f389809f830 2019-12-05T05:49:27.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:27.485+0000 7f389e2a3700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f389808df80 0x7f389809f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:49:27.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:27.485+0000 7f389e2a3700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f389808df80 0x7f389809f2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:41936/0 (socket says 172.21.15.42:41936) 2019-12-05T05:49:27.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:27.485+0000 7f389e2a3700 1 -- 172.21.15.42:0/2487132915 learned_addr learned my addr 172.21.15.42:0/2487132915 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:49:27.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:27.485+0000 7f389f2a5700 1 --2- 172.21.15.42:0/2487132915 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f389809f830 0x7f389814efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:49:27.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:27.485+0000 7f389e2a3700 1 -- 172.21.15.42:0/2487132915 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f389809f830 msgr2=0x7f389814efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:49:27.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:27.485+0000 7f389e2a3700 1 --2- 172.21.15.42:0/2487132915 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f389809f830 0x7f389814efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:49:27.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:27.485+0000 7f389e2a3700 1 -- 172.21.15.42:0/2487132915 >> v1:172.21.15.42:6789/0 conn(0x7f38980a7740 legacy=0x7f389807e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:49:27.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:27.485+0000 7f389e2a3700 1 -- 172.21.15.42:0/2487132915 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38980a0710 con 0x7f389808df80 2019-12-05T05:49:27.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:27.485+0000 7f389e2a3700 1 --2- 172.21.15.42:0/2487132915 >> v2:172.21.15.42:3300/0 conn(0x7f389808df80 0x7f389809f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f3888008790 tx=0x7f388800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:27.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:27.485+0000 7f389e2a3700 1 --2- 172.21.15.42:0/2487132915 >> v2:172.21.15.42:3300/0 conn(0x7f389808df80 0x7f389809f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f3888008790 tx=0x7f388800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:27.496 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:27.485+0000 7f389e2a3700 1 --2- 172.21.15.42:0/2487132915 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f389808df80 0x7f389809f2e0 secure :-1 s=READY pgs=1524 cs=0 l=1 rx=0x7f3888008790 tx=0x7f388800fd30).ready entity=mon.1 client_cookie=ec311c9ce7da60cb server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:49:27.496 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:27.485+0000 7f389e2a3700 1 --2- 172.21.15.42:0/2487132915 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f389808df80 0x7f389809f2e0 secure :-1 s=THROTTLE_DONE pgs=1524 cs=0 l=1 rx=0x7f3888008790 tx=0x7f388800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:27.496 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:27.485+0000 7f389e2a3700 1 --2- 172.21.15.42:0/2487132915 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f389808df80 0x7f389809f2e0 secure :-1 s=THROTTLE_DONE pgs=1524 cs=0 l=1 rx=0x7f3888008790 tx=0x7f388800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:27.496 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:27.485+0000 7f389daa2700 1 -- 172.21.15.42:0/2487132915 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3888008c90 con 0x7f389808df80 2019-12-05T05:49:27.496 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:27.485+0000 7f389e2a3700 1 --2- 172.21.15.42:0/2487132915 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f389808df80 0x7f389809f2e0 secure :-1 s=THROTTLE_DONE pgs=1524 cs=0 l=1 rx=0x7f3888008790 tx=0x7f388800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:27.496 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:27.485+0000 7f389daa2700 1 -- 172.21.15.42:0/2487132915 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38880088b0 con 0x7f389808df80 2019-12-05T05:49:27.497 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:27.485+0000 7f389daa2700 1 -- 172.21.15.42:0/2487132915 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3888011b10 con 0x7f389808df80 2019-12-05T05:49:27.497 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:27.485+0000 7f38a0507700 1 -- 172.21.15.42:0/2487132915 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f389808df80 msgr2=0x7f389809f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:49:27.497 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:27.485+0000 7f38a0507700 1 --2- 172.21.15.42:0/2487132915 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f389808df80 0x7f389809f2e0 secure :-1 s=READY pgs=1524 cs=0 l=1 rx=0x7f3888008790 tx=0x7f388800fd30).stop 2019-12-05T05:49:27.497 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:27.489+0000 7f38a0507700 1 -- 172.21.15.42:0/2487132915 shutdown_connections 2019-12-05T05:49:27.497 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:27.489+0000 7f38a0507700 1 --2- 172.21.15.42:0/2487132915 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f389809f830 0x7f389814efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:49:27.497 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:27.489+0000 7f38a0507700 1 --2- 172.21.15.42:0/2487132915 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f389808df80 0x7f389809f2e0 unknown :-1 s=CLOSED pgs=1524 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:49:27.497 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:27.489+0000 7f38a0507700 1 -- 172.21.15.42:0/2487132915 shutdown_connections 2019-12-05T05:49:27.498 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:27.489+0000 7f38a0507700 1 -- 172.21.15.42:0/2487132915 wait complete. 2019-12-05T05:49:27.498 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:27.489+0000 7f38a0507700 1 -- 172.21.15.42:0/2487132915 >> 172.21.15.42:0/2487132915 conn(0x7f38980aabe0 msgr2=0x7f38980a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:49:27.501 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:27.489+0000 7f38a0507700 1 Processor -- start 2019-12-05T05:49:27.501 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:27.489+0000 7f38a0507700 1 -- start start 2019-12-05T05:49:27.501 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:27.489+0000 7f38a0507700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f38980a6f20 0x7f389811c020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:49:27.502 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:27.489+0000 7f38a0507700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f38981194f0 0x7f38981203c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:49:27.502 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:27.493+0000 7f38a0507700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f389811ba90 con 0x7f389811a5f0 2019-12-05T05:49:27.502 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:27.493+0000 7f38a0507700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3898120b30 con 0x7f38981194f0 2019-12-05T05:49:27.502 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:27.493+0000 7f38a0507700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f389811bd80 con 0x7f38980a6f20 2019-12-05T05:49:27.504 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:27.493+0000 7f389eaa4700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f38980a6f20 0x7f389811c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:49:27.504 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:27.493+0000 7f389f2a5700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f389811a5f0 0x7f389811edb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:41982/0 (socket says 172.21.15.42:41982) 2019-12-05T05:49:27.504 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:27.493+0000 7f389f2a5700 1 -- 172.21.15.42:0/2821001570 learned_addr learned my addr 172.21.15.42:0/2821001570 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:49:27.504 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:27.493+0000 7f389e2a3700 1 --2- 172.21.15.42:0/2821001570 >> v2:172.21.15.40:3300/0 conn(0x7f38981194f0 0x7f38981203c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:49:27.504 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:27.493+0000 7f389eaa4700 1 -- 172.21.15.42:0/2821001570 >> v2:172.21.15.40:3300/0 conn(0x7f38981194f0 msgr2=0x7f38981203c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:49:27.505 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:27.493+0000 7f389eaa4700 1 --2- 172.21.15.42:0/2821001570 >> v2:172.21.15.40:3300/0 conn(0x7f38981194f0 0x7f38981203c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:49:27.505 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:27.493+0000 7f389eaa4700 1 -- 172.21.15.42:0/2821001570 >> v1:172.21.15.42:6789/0 conn(0x7f389811a5f0 legacy=0x7f389811edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:49:27.505 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:27.493+0000 7f388f7fe700 1 -- 172.21.15.42:0/2821001570 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1389200448 0 0) 0x7f389811ba90 con 0x7f389811a5f0 2019-12-05T05:49:27.505 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:27.493+0000 7f389eaa4700 1 -- 172.21.15.42:0/2821001570 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f389811aa20 con 0x7f38980a6f20 2019-12-05T05:49:27.505 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:27.493+0000 7f389eaa4700 1 --2- 172.21.15.42:0/2821001570 >> v2:172.21.15.42:3300/0 conn(0x7f38980a6f20 0x7f389811c020 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f3894000e90 tx=0x7f38940108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:27.506 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:27.493+0000 7f389eaa4700 1 --2- 172.21.15.42:0/2821001570 >> v2:172.21.15.42:3300/0 conn(0x7f38980a6f20 0x7f389811c020 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f3894000e90 tx=0x7f38940108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:27.506 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:27.493+0000 7f389eaa4700 1 --2- 172.21.15.42:0/2821001570 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f38980a6f20 0x7f389811c020 secure :-1 s=READY pgs=1526 cs=0 l=1 rx=0x7f3894000e90 tx=0x7f38940108f0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:49:27.506 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:27.493+0000 7f389eaa4700 1 --2- 172.21.15.42:0/2821001570 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f38980a6f20 0x7f389811c020 secure :-1 s=THROTTLE_DONE pgs=1526 cs=0 l=1 rx=0x7f3894000e90 tx=0x7f38940108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:27.506 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:27.493+0000 7f388f7fe700 1 -- 172.21.15.42:0/2821001570 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f389400cb90 con 0x7f38980a6f20 2019-12-05T05:49:27.506 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:27.493+0000 7f389eaa4700 1 --2- 172.21.15.42:0/2821001570 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f38980a6f20 0x7f389811c020 secure :-1 s=THROTTLE_DONE pgs=1526 cs=0 l=1 rx=0x7f3894000e90 tx=0x7f38940108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:27.506 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:27.493+0000 7f389eaa4700 1 --2- 172.21.15.42:0/2821001570 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f38980a6f20 0x7f389811c020 secure :-1 s=THROTTLE_DONE pgs=1526 cs=0 l=1 rx=0x7f3894000e90 tx=0x7f38940108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:27.506 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:27.493+0000 7f388f7fe700 1 -- 172.21.15.42:0/2821001570 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f389400c7b0 con 0x7f38980a6f20 2019-12-05T05:49:27.506 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:27.493+0000 7f38a0507700 1 -- 172.21.15.42:0/2821001570 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f389811f5b0 con 0x7f38980a6f20 2019-12-05T05:49:27.507 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:27.493+0000 7f388f7fe700 1 -- 172.21.15.42:0/2821001570 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f38940110b0 con 0x7f38980a6f20 2019-12-05T05:49:27.507 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:27.493+0000 7f38a0507700 1 -- 172.21.15.42:0/2821001570 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f389811fb30 con 0x7f38980a6f20 2019-12-05T05:49:27.507 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:27.493+0000 7f389eaa4700 1 --2- 172.21.15.42:0/2821001570 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f38980a6f20 0x7f389811c020 secure :-1 s=THROTTLE_DONE pgs=1526 cs=0 l=1 rx=0x7f3894000e90 tx=0x7f38940108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:27.507 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:27.493+0000 7f388f7fe700 1 -- 172.21.15.42:0/2821001570 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f38940113c0 con 0x7f38980a6f20 2019-12-05T05:49:27.507 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:27.493+0000 7f389eaa4700 1 --2- 172.21.15.42:0/2821001570 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f38980a6f20 0x7f389811c020 secure :-1 s=THROTTLE_DONE pgs=1526 cs=0 l=1 rx=0x7f3894000e90 tx=0x7f38940108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:27.507 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:27.493+0000 7f388f7fe700 1 --2- 172.21.15.42:0/2821001570 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f388003acc0 0x7f388003d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:49:27.508 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:27.493+0000 7f388f7fe700 1 -- 172.21.15.42:0/2821001570 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 8697+0+0 (secure 0 0 0) 0x7f389404b1a0 con 0x7f38980a6f20 2019-12-05T05:49:27.508 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:27.497+0000 7f389e2a3700 1 --2- 172.21.15.42:0/2821001570 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f388003acc0 0x7f388003d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:49:27.508 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:27.497+0000 7f389e2a3700 1 --2- 172.21.15.42:0/2821001570 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f388003acc0 0x7f388003d1d0 crc :-1 s=READY pgs=656 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:49:27.509 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:27.501+0000 7f38a0507700 1 -- 172.21.15.42:0/2821001570 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f389805d110 con 0x7f38980a6f20 2019-12-05T05:49:27.511 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:27.505+0000 7f389eaa4700 1 --2- 172.21.15.42:0/2821001570 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f38980a6f20 0x7f389811c020 secure :-1 s=THROTTLE_DONE pgs=1526 cs=0 l=1 rx=0x7f3894000e90 tx=0x7f38940108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:27.511 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:27.505+0000 7f388f7fe700 1 -- 172.21.15.42:0/2821001570 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f38940321c0 con 0x7f38980a6f20 2019-12-05T05:49:27.674 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:27.669+0000 7f38a0507700 1 -- 172.21.15.42:0/2821001570 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f389805b8c0 con 0x7f38980a6f20 2019-12-05T05:49:27.675 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:27.669+0000 7f389eaa4700 1 --2- 172.21.15.42:0/2821001570 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f38980a6f20 0x7f389811c020 secure :-1 s=THROTTLE_DONE pgs=1526 cs=0 l=1 rx=0x7f3894000e90 tx=0x7f38940108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:49:27.675 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:27.669+0000 7f388f7fe700 1 -- 172.21.15.42:0/2821001570 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 188 v188) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f3894032420 con 0x7f38980a6f20 2019-12-05T05:49:27.678 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 188 2019-12-05T05:49:27.678 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:49:27.679 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":188,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6245,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/2081111170","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2081111170},{"type":"v1","addr":"172.21.15.40:6833","nonce":2081111170}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":188}],"filesystems":[{"mdsmap":{"epoch":187,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:49:23.679613+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8188},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8188":{"gid":8188,"name":"a","rank":0,"incarnation":184,"state":"up:active","state_seq":33,"addr":"172.21.15.42:6835/2456922143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2456922143},{"type":"v1","addr":"172.21.15.42:6835","nonce":2456922143}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:49:27.680 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:27.677+0000 7f38a0507700 1 -- 172.21.15.42:0/2821001570 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f388003acc0 msgr2=0x7f388003d1d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:49:27.680 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:27.677+0000 7f38a0507700 1 --2- 172.21.15.42:0/2821001570 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f388003acc0 0x7f388003d1d0 crc :-1 s=READY pgs=656 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:49:27.680 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:27.677+0000 7f38a0507700 1 -- 172.21.15.42:0/2821001570 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f38980a6f20 msgr2=0x7f389811c020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:49:27.680 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:27.677+0000 7f38a0507700 1 --2- 172.21.15.42:0/2821001570 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f38980a6f20 0x7f389811c020 secure :-1 s=READY pgs=1526 cs=0 l=1 rx=0x7f3894000e90 tx=0x7f38940108f0).stop 2019-12-05T05:49:27.681 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:27.677+0000 7f38a0507700 1 -- 172.21.15.42:0/2821001570 shutdown_connections 2019-12-05T05:49:27.681 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:27.677+0000 7f38a0507700 1 --2- 172.21.15.42:0/2821001570 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f388003acc0 0x7f388003d1d0 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:49:27.681 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:27.677+0000 7f38a0507700 1 --2- 172.21.15.42:0/2821001570 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f38980a6f20 0x7f389811c020 unknown :-1 s=CLOSED pgs=1526 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:49:27.681 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:27.677+0000 7f38a0507700 1 --2- 172.21.15.42:0/2821001570 >> v2:172.21.15.40:3300/0 conn(0x7f38981194f0 0x7f38981203c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:49:27.681 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:27.677+0000 7f38a0507700 1 -- 172.21.15.42:0/2821001570 shutdown_connections 2019-12-05T05:49:27.681 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:27.677+0000 7f38a0507700 1 -- 172.21.15.42:0/2821001570 wait complete. 2019-12-05T05:49:27.681 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:49:27.677+0000 7f38a0507700 1 -- 172.21.15.42:0/2821001570 >> 172.21.15.42:0/2821001570 conn(0x7f38980a97e0 msgr2=0x7f3898059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:49:27.690 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b reported in up:standby state 2019-12-05T05:49:27.691 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 64 secs before thrashing 2019-12-05T05:49:41.845 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:49:41.849 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:50:11.893 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:50:11.897 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:50:31.692 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:50:31.793 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:31.785+0000 7f01edd06700 1 Processor -- start 2019-12-05T05:50:31.793 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:31.785+0000 7f01edd06700 1 -- start start 2019-12-05T05:50:31.794 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:31.785+0000 7f01edd06700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f01e80a7740 0x7f01e808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:50:31.794 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:31.785+0000 7f01edd06700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f01e807e720 0x7f01e807eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:50:31.794 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:31.785+0000 7f01edd06700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f01e809f2e0 0x7f01e809f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:50:31.794 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:31.785+0000 7f01edd06700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01e80a0b00 con 0x7f01e80a7740 2019-12-05T05:50:31.794 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:31.785+0000 7f01edd06700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01e808e5b0 con 0x7f01e809f2e0 2019-12-05T05:50:31.794 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:31.785+0000 7f01edd06700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01e808e850 con 0x7f01e807e720 2019-12-05T05:50:31.794 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:31.785+0000 7f01ecaa4700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f01e809f2e0 0x7f01e809f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:50:31.795 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:31.785+0000 7f01ecaa4700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f01e809f2e0 0x7f01e809f710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:42190/0 (socket says 172.21.15.42:42190) 2019-12-05T05:50:31.795 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:31.785+0000 7f01ecaa4700 1 -- 172.21.15.42:0/1739068628 learned_addr learned my addr 172.21.15.42:0/1739068628 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:50:31.795 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:31.785+0000 7f01e7fff700 1 --2- 172.21.15.42:0/1739068628 >> v2:172.21.15.40:3300/0 conn(0x7f01e80a7740 0x7f01e808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:50:31.795 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:31.785+0000 7f01e77fe700 1 --2- 172.21.15.42:0/1739068628 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f01e807e720 0x7f01e807eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:50:31.795 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:31.785+0000 7f01ecaa4700 1 -- 172.21.15.42:0/1739068628 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f01e807e720 msgr2=0x7f01e807eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:50:31.795 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:31.785+0000 7f01ecaa4700 1 --2- 172.21.15.42:0/1739068628 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f01e807e720 0x7f01e807eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:50:31.795 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:31.785+0000 7f01ecaa4700 1 -- 172.21.15.42:0/1739068628 >> v2:172.21.15.40:3300/0 conn(0x7f01e80a7740 msgr2=0x7f01e808df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:50:31.796 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:31.785+0000 7f01ecaa4700 1 --2- 172.21.15.42:0/1739068628 >> v2:172.21.15.40:3300/0 conn(0x7f01e80a7740 0x7f01e808df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:50:31.796 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:31.785+0000 7f01ecaa4700 1 -- 172.21.15.42:0/1739068628 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01e807f0a0 con 0x7f01e809f2e0 2019-12-05T05:50:31.796 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:31.785+0000 7f01ecaa4700 1 --2- 172.21.15.42:0/1739068628 >> v2:172.21.15.42:3300/0 conn(0x7f01e809f2e0 0x7f01e809f710 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f01e808e770 tx=0x7f01e0010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:31.796 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:31.785+0000 7f01ecaa4700 1 --2- 172.21.15.42:0/1739068628 >> v2:172.21.15.42:3300/0 conn(0x7f01e809f2e0 0x7f01e809f710 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f01e808e770 tx=0x7f01e0010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:31.796 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:31.785+0000 7f01ecaa4700 1 --2- 172.21.15.42:0/1739068628 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f01e809f2e0 0x7f01e809f710 secure :-1 s=READY pgs=1530 cs=0 l=1 rx=0x7f01e808e770 tx=0x7f01e0010130).ready entity=mon.1 client_cookie=1516a588af414be3 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:50:31.796 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:31.785+0000 7f01ecaa4700 1 --2- 172.21.15.42:0/1739068628 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f01e809f2e0 0x7f01e809f710 secure :-1 s=THROTTLE_DONE pgs=1530 cs=0 l=1 rx=0x7f01e808e770 tx=0x7f01e0010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:31.797 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:31.785+0000 7f01ecaa4700 1 --2- 172.21.15.42:0/1739068628 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f01e809f2e0 0x7f01e809f710 secure :-1 s=THROTTLE_DONE pgs=1530 cs=0 l=1 rx=0x7f01e808e770 tx=0x7f01e0010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:31.797 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:31.785+0000 7f01e6ffd700 1 -- 172.21.15.42:0/1739068628 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f01e0010de0 con 0x7f01e809f2e0 2019-12-05T05:50:31.797 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:31.785+0000 7f01ecaa4700 1 --2- 172.21.15.42:0/1739068628 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f01e809f2e0 0x7f01e809f710 secure :-1 s=THROTTLE_DONE pgs=1530 cs=0 l=1 rx=0x7f01e808e770 tx=0x7f01e0010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:31.797 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:31.785+0000 7f01e6ffd700 1 -- 172.21.15.42:0/1739068628 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01e0010a00 con 0x7f01e809f2e0 2019-12-05T05:50:31.797 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:31.789+0000 7f01e6ffd700 1 -- 172.21.15.42:0/1739068628 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f01e00114a0 con 0x7f01e809f2e0 2019-12-05T05:50:31.797 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:31.789+0000 7f01edd06700 1 -- 172.21.15.42:0/1739068628 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f01e809f2e0 msgr2=0x7f01e809f710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:50:31.797 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:31.789+0000 7f01edd06700 1 --2- 172.21.15.42:0/1739068628 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f01e809f2e0 0x7f01e809f710 secure :-1 s=READY pgs=1530 cs=0 l=1 rx=0x7f01e808e770 tx=0x7f01e0010130).stop 2019-12-05T05:50:31.798 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:31.789+0000 7f01edd06700 1 -- 172.21.15.42:0/1739068628 shutdown_connections 2019-12-05T05:50:31.798 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:31.789+0000 7f01edd06700 1 --2- 172.21.15.42:0/1739068628 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f01e809f2e0 0x7f01e809f710 unknown :-1 s=CLOSED pgs=1530 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:50:31.798 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:31.789+0000 7f01edd06700 1 --2- 172.21.15.42:0/1739068628 >> v2:172.21.15.40:3300/0 conn(0x7f01e80a7740 0x7f01e808df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:50:31.798 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:31.789+0000 7f01edd06700 1 --2- 172.21.15.42:0/1739068628 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f01e807e720 0x7f01e807eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:50:31.798 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:31.789+0000 7f01edd06700 1 -- 172.21.15.42:0/1739068628 shutdown_connections 2019-12-05T05:50:31.798 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:31.789+0000 7f01edd06700 1 -- 172.21.15.42:0/1739068628 wait complete. 2019-12-05T05:50:31.798 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:31.789+0000 7f01edd06700 1 -- 172.21.15.42:0/1739068628 >> 172.21.15.42:0/1739068628 conn(0x7f01e80aabe0 msgr2=0x7f01e80a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:50:31.802 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:31.789+0000 7f01edd06700 1 Processor -- start 2019-12-05T05:50:31.802 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:31.789+0000 7f01edd06700 1 -- start start 2019-12-05T05:50:31.802 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:31.793+0000 7f01edd06700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f01e807cdc0 0x7f01e81233a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:50:31.802 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:31.793+0000 7f01edd06700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f01e813b9f0 0x7f01e813de30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:50:31.802 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:31.793+0000 7f01edd06700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01e807bd50 con 0x7f01e80a5ab0 2019-12-05T05:50:31.803 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:31.793+0000 7f01edd06700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01e80a4a50 con 0x7f01e807cdc0 2019-12-05T05:50:31.803 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:31.793+0000 7f01edd06700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01e80a6ca0 con 0x7f01e813b9f0 2019-12-05T05:50:31.803 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:31.793+0000 7f01e77fe700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f01e80a5ab0 0x7f01e80a5ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:42230/0 (socket says 172.21.15.42:42230) 2019-12-05T05:50:31.803 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:31.793+0000 7f01e77fe700 1 -- 172.21.15.42:0/3034669226 learned_addr learned my addr 172.21.15.42:0/3034669226 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:50:31.803 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:31.793+0000 7f01e7fff700 1 --2- 172.21.15.42:0/3034669226 >> v2:172.21.15.40:3300/0 conn(0x7f01e807cdc0 0x7f01e81233a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:50:31.803 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:31.793+0000 7f01ecaa4700 1 --2- 172.21.15.42:0/3034669226 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f01e813b9f0 0x7f01e813de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:50:31.803 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:31.793+0000 7f01e4ff9700 1 -- 172.21.15.42:0/3034669226 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 375039751 0 0) 0x7f01e807bd50 con 0x7f01e80a5ab0 2019-12-05T05:50:31.804 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:31.793+0000 7f01e4ff9700 1 -- 172.21.15.42:0/3034669226 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01c80030b0 con 0x7f01e80a5ab0 2019-12-05T05:50:31.804 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:31.793+0000 7f01ecaa4700 1 -- 172.21.15.42:0/3034669226 >> v2:172.21.15.40:3300/0 conn(0x7f01e807cdc0 msgr2=0x7f01e81233a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:50:31.804 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:31.793+0000 7f01ecaa4700 1 --2- 172.21.15.42:0/3034669226 >> v2:172.21.15.40:3300/0 conn(0x7f01e807cdc0 0x7f01e81233a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:50:31.804 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:31.793+0000 7f01ecaa4700 1 -- 172.21.15.42:0/3034669226 >> v1:172.21.15.42:6789/0 conn(0x7f01e80a5ab0 legacy=0x7f01e80a5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:50:31.804 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:31.793+0000 7f01ecaa4700 1 -- 172.21.15.42:0/3034669226 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01e81238f0 con 0x7f01e813b9f0 2019-12-05T05:50:31.804 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:31.793+0000 7f01e7fff700 1 --2- 172.21.15.42:0/3034669226 >> v2:172.21.15.40:3300/0 conn(0x7f01e807cdc0 0x7f01e81233a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:50:31.804 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:31.793+0000 7f01ecaa4700 1 --2- 172.21.15.42:0/3034669226 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f01e813b9f0 0x7f01e813de30 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f01e808e770 tx=0x7f01e0010cb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:31.805 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:31.793+0000 7f01ecaa4700 1 --2- 172.21.15.42:0/3034669226 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f01e813b9f0 0x7f01e813de30 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f01e808e770 tx=0x7f01e0010cb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:31.805 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:31.793+0000 7f01ecaa4700 1 --2- 172.21.15.42:0/3034669226 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f01e813b9f0 0x7f01e813de30 secure :-1 s=READY pgs=342 cs=0 l=1 rx=0x7f01e808e770 tx=0x7f01e0010cb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:50:31.805 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:31.793+0000 7f01ecaa4700 1 --2- 172.21.15.42:0/3034669226 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f01e813b9f0 0x7f01e813de30 secure :-1 s=THROTTLE_DONE pgs=342 cs=0 l=1 rx=0x7f01e808e770 tx=0x7f01e0010cb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:31.805 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:31.793+0000 7f01e4ff9700 1 -- 172.21.15.42:0/3034669226 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f01e0011b70 con 0x7f01e813b9f0 2019-12-05T05:50:31.805 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:31.793+0000 7f01ecaa4700 1 --2- 172.21.15.42:0/3034669226 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f01e813b9f0 0x7f01e813de30 secure :-1 s=THROTTLE_DONE pgs=342 cs=0 l=1 rx=0x7f01e808e770 tx=0x7f01e0010cb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:31.805 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:31.793+0000 7f01ecaa4700 1 --2- 172.21.15.42:0/3034669226 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f01e813b9f0 0x7f01e813de30 secure :-1 s=THROTTLE_DONE pgs=342 cs=0 l=1 rx=0x7f01e808e770 tx=0x7f01e0010cb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:31.805 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:31.793+0000 7f01e4ff9700 1 -- 172.21.15.42:0/3034669226 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01e00110c0 con 0x7f01e813b9f0 2019-12-05T05:50:31.806 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:31.793+0000 7f01edd06700 1 -- 172.21.15.42:0/3034669226 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01e807c0f0 con 0x7f01e813b9f0 2019-12-05T05:50:31.806 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:31.793+0000 7f01e4ff9700 1 -- 172.21.15.42:0/3034669226 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f01e0005ab0 con 0x7f01e813b9f0 2019-12-05T05:50:31.806 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:31.793+0000 7f01edd06700 1 -- 172.21.15.42:0/3034669226 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01e807b340 con 0x7f01e813b9f0 2019-12-05T05:50:31.806 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:31.793+0000 7f01ecaa4700 1 --2- 172.21.15.42:0/3034669226 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f01e813b9f0 0x7f01e813de30 secure :-1 s=THROTTLE_DONE pgs=342 cs=0 l=1 rx=0x7f01e808e770 tx=0x7f01e0010cb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:31.806 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:31.797+0000 7f01e4ff9700 1 -- 172.21.15.42:0/3034669226 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f01e000f280 con 0x7f01e813b9f0 2019-12-05T05:50:31.806 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:31.797+0000 7f01ecaa4700 1 --2- 172.21.15.42:0/3034669226 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f01e813b9f0 0x7f01e813de30 secure :-1 s=THROTTLE_DONE pgs=342 cs=0 l=1 rx=0x7f01e808e770 tx=0x7f01e0010cb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:31.806 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:31.797+0000 7f01e4ff9700 1 --2- 172.21.15.42:0/3034669226 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f01c803b2c0 0x7f01c803d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:50:31.807 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:31.797+0000 7f01e4ff9700 1 -- 172.21.15.42:0/3034669226 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 8697+0+0 (secure 0 0 0) 0x7f01e008f5a0 con 0x7f01e813b9f0 2019-12-05T05:50:31.807 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:31.797+0000 7f01e7fff700 1 --2- 172.21.15.42:0/3034669226 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f01c803b2c0 0x7f01c803d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:50:31.807 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:31.797+0000 7f01e7fff700 1 --2- 172.21.15.42:0/3034669226 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f01c803b2c0 0x7f01c803d7d0 crc :-1 s=READY pgs=659 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:50:31.807 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:31.801+0000 7f01edd06700 1 -- 172.21.15.42:0/3034669226 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01e805d110 con 0x7f01e813b9f0 2019-12-05T05:50:31.813 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:31.809+0000 7f01ecaa4700 1 --2- 172.21.15.42:0/3034669226 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f01e813b9f0 0x7f01e813de30 secure :-1 s=THROTTLE_DONE pgs=342 cs=0 l=1 rx=0x7f01e808e770 tx=0x7f01e0010cb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:31.813 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:31.809+0000 7f01e4ff9700 1 -- 172.21.15.42:0/3034669226 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f01e0005dc0 con 0x7f01e813b9f0 2019-12-05T05:50:32.003 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:32.001+0000 7f01edd06700 1 -- 172.21.15.42:0/3034669226 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f01e80a4c50 con 0x7f01e813b9f0 2019-12-05T05:50:32.004 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:32.001+0000 7f01ecaa4700 1 --2- 172.21.15.42:0/3034669226 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f01e813b9f0 0x7f01e813de30 secure :-1 s=THROTTLE_DONE pgs=342 cs=0 l=1 rx=0x7f01e808e770 tx=0x7f01e0010cb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:32.004 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:32.001+0000 7f01e4ff9700 1 -- 172.21.15.42:0/3034669226 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 188 v188) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f01e0076880 con 0x7f01e813b9f0 2019-12-05T05:50:32.006 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 188 2019-12-05T05:50:32.006 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:50:32.007 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":188,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6245,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/2081111170","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2081111170},{"type":"v1","addr":"172.21.15.40:6833","nonce":2081111170}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":188}],"filesystems":[{"mdsmap":{"epoch":187,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:49:23.679613+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8188},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8188":{"gid":8188,"name":"a","rank":0,"incarnation":184,"state":"up:active","state_seq":33,"addr":"172.21.15.42:6835/2456922143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2456922143},{"type":"v1","addr":"172.21.15.42:6835","nonce":2456922143}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:50:32.010 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:32.005+0000 7f01edd06700 1 -- 172.21.15.42:0/3034669226 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f01c803b2c0 msgr2=0x7f01c803d7d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:50:32.010 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:32.005+0000 7f01edd06700 1 --2- 172.21.15.42:0/3034669226 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f01c803b2c0 0x7f01c803d7d0 crc :-1 s=READY pgs=659 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:50:32.010 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:32.005+0000 7f01edd06700 1 -- 172.21.15.42:0/3034669226 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f01e813b9f0 msgr2=0x7f01e813de30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:50:32.010 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:32.005+0000 7f01edd06700 1 --2- 172.21.15.42:0/3034669226 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f01e813b9f0 0x7f01e813de30 secure :-1 s=READY pgs=342 cs=0 l=1 rx=0x7f01e808e770 tx=0x7f01e0010cb0).stop 2019-12-05T05:50:32.010 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:32.005+0000 7f01edd06700 1 -- 172.21.15.42:0/3034669226 shutdown_connections 2019-12-05T05:50:32.011 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:32.005+0000 7f01edd06700 1 --2- 172.21.15.42:0/3034669226 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f01c803b2c0 0x7f01c803d7d0 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:50:32.011 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:32.005+0000 7f01edd06700 1 --2- 172.21.15.42:0/3034669226 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f01e813b9f0 0x7f01e813de30 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:50:32.011 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:32.005+0000 7f01edd06700 1 --2- 172.21.15.42:0/3034669226 >> v2:172.21.15.40:3300/0 conn(0x7f01e807cdc0 0x7f01e81233a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:50:32.011 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:32.005+0000 7f01edd06700 1 -- 172.21.15.42:0/3034669226 shutdown_connections 2019-12-05T05:50:32.011 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:32.005+0000 7f01edd06700 1 -- 172.21.15.42:0/3034669226 wait complete. 2019-12-05T05:50:32.011 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:32.005+0000 7f01edd06700 1 -- 172.21.15.42:0/3034669226 >> 172.21.15.42:0/3034669226 conn(0x7f01e809fa10 msgr2=0x7f01e8059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:50:32.019 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.a (rank=0) 2019-12-05T05:50:32.019 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2019-12-05T05:50:32.019 INFO:teuthology.orchestra.run:waiting for 300 2019-12-05T05:50:32.028 INFO:tasks.ceph.mds.a:Stopped 2019-12-05T05:50:32.028 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:50:32.197 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:32.185+0000 7fbb354c6700 1 Processor -- start 2019-12-05T05:50:32.197 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:32.185+0000 7fbb354c6700 1 -- start start 2019-12-05T05:50:32.197 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:32.189+0000 7fbb354c6700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fbb300a7740 0x7fbb3008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:50:32.197 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:32.189+0000 7fbb354c6700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbb3007e720 0x7fbb3007eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:50:32.197 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:32.189+0000 7fbb354c6700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb300a0710 con 0x7fbb3009f2e0 2019-12-05T05:50:32.198 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:32.189+0000 7fbb354c6700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb300a0b00 con 0x7fbb300a7740 2019-12-05T05:50:32.198 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:32.189+0000 7fbb354c6700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb3008e5b0 con 0x7fbb3007e720 2019-12-05T05:50:32.198 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:32.189+0000 7fbb2f7fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fbb300a7740 0x7fbb3008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:50:32.198 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:32.189+0000 7fbb2f7fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fbb300a7740 0x7fbb3008df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:42202/0 (socket says 172.21.15.42:42202) 2019-12-05T05:50:32.198 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:32.189+0000 7fbb2f7fe700 1 -- 172.21.15.42:0/595324278 learned_addr learned my addr 172.21.15.42:0/595324278 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:50:32.198 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:32.189+0000 7fbb2effd700 1 --2- 172.21.15.42:0/595324278 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbb3007e720 0x7fbb3007eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:50:32.198 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:32.189+0000 7fbb2e7fc700 1 -- 172.21.15.42:0/595324278 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3365685245 0 0) 0x7fbb300a0710 con 0x7fbb3009f2e0 2019-12-05T05:50:32.199 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:32.189+0000 7fbb2f7fe700 1 -- 172.21.15.42:0/595324278 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbb3007e720 msgr2=0x7fbb3007eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:50:32.199 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:32.189+0000 7fbb2f7fe700 1 --2- 172.21.15.42:0/595324278 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbb3007e720 0x7fbb3007eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:50:32.199 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:32.189+0000 7fbb2f7fe700 1 -- 172.21.15.42:0/595324278 >> v1:172.21.15.40:6789/0 conn(0x7fbb3009f2e0 legacy=0x7fbb3009f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:50:32.199 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:32.189+0000 7fbb2f7fe700 1 -- 172.21.15.42:0/595324278 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb3007f0a0 con 0x7fbb300a7740 2019-12-05T05:50:32.199 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:32.189+0000 7fbb2f7fe700 1 --2- 172.21.15.42:0/595324278 >> v2:172.21.15.42:3300/0 conn(0x7fbb300a7740 0x7fbb3008df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fbb20008790 tx=0x7fbb2000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:32.199 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:32.189+0000 7fbb2f7fe700 1 --2- 172.21.15.42:0/595324278 >> v2:172.21.15.42:3300/0 conn(0x7fbb300a7740 0x7fbb3008df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fbb20008790 tx=0x7fbb2000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:32.200 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:32.189+0000 7fbb2f7fe700 1 --2- 172.21.15.42:0/595324278 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbb300a7740 0x7fbb3008df80 secure :-1 s=READY pgs=1532 cs=0 l=1 rx=0x7fbb20008790 tx=0x7fbb2000fd30).ready entity=mon.1 client_cookie=e02818f2d05f05a9 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:50:32.200 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:32.189+0000 7fbb2f7fe700 1 --2- 172.21.15.42:0/595324278 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbb300a7740 0x7fbb3008df80 secure :-1 s=THROTTLE_DONE pgs=1532 cs=0 l=1 rx=0x7fbb20008790 tx=0x7fbb2000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:32.200 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:32.189+0000 7fbb2e7fc700 1 -- 172.21.15.42:0/595324278 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbb200113b0 con 0x7fbb300a7740 2019-12-05T05:50:32.200 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:32.189+0000 7fbb2f7fe700 1 --2- 172.21.15.42:0/595324278 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbb300a7740 0x7fbb3008df80 secure :-1 s=THROTTLE_DONE pgs=1532 cs=0 l=1 rx=0x7fbb20008790 tx=0x7fbb2000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:32.200 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:32.189+0000 7fbb2f7fe700 1 --2- 172.21.15.42:0/595324278 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbb300a7740 0x7fbb3008df80 secure :-1 s=THROTTLE_DONE pgs=1532 cs=0 l=1 rx=0x7fbb20008790 tx=0x7fbb2000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:32.200 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:32.189+0000 7fbb2e7fc700 1 -- 172.21.15.42:0/595324278 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb20008bb0 con 0x7fbb300a7740 2019-12-05T05:50:32.201 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:32.189+0000 7fbb2e7fc700 1 -- 172.21.15.42:0/595324278 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbb20011990 con 0x7fbb300a7740 2019-12-05T05:50:32.201 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:32.189+0000 7fbb354c6700 1 -- 172.21.15.42:0/595324278 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbb300a7740 msgr2=0x7fbb3008df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:50:32.201 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:32.189+0000 7fbb354c6700 1 --2- 172.21.15.42:0/595324278 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbb300a7740 0x7fbb3008df80 secure :-1 s=READY pgs=1532 cs=0 l=1 rx=0x7fbb20008790 tx=0x7fbb2000fd30).stop 2019-12-05T05:50:32.201 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:32.193+0000 7fbb354c6700 1 -- 172.21.15.42:0/595324278 shutdown_connections 2019-12-05T05:50:32.201 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:32.193+0000 7fbb354c6700 1 --2- 172.21.15.42:0/595324278 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbb300a7740 0x7fbb3008df80 unknown :-1 s=CLOSED pgs=1532 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:50:32.201 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:32.193+0000 7fbb354c6700 1 --2- 172.21.15.42:0/595324278 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbb3007e720 0x7fbb3007eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:50:32.201 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:32.193+0000 7fbb354c6700 1 -- 172.21.15.42:0/595324278 shutdown_connections 2019-12-05T05:50:32.202 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:32.193+0000 7fbb354c6700 1 -- 172.21.15.42:0/595324278 wait complete. 2019-12-05T05:50:32.202 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:32.193+0000 7fbb354c6700 1 -- 172.21.15.42:0/595324278 >> 172.21.15.42:0/595324278 conn(0x7fbb300aabe0 msgr2=0x7fbb300a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:50:32.205 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:32.193+0000 7fbb354c6700 1 Processor -- start 2019-12-05T05:50:32.205 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:32.193+0000 7fbb354c6700 1 -- start start 2019-12-05T05:50:32.205 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:32.197+0000 7fbb354c6700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fbb300a9980 0x7fbb300a4840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:50:32.205 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:32.197+0000 7fbb354c6700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbb300a6220 0x7fbb3007b850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:50:32.205 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:32.197+0000 7fbb354c6700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb300a4d90 con 0x7fbb3007bda0 2019-12-05T05:50:32.205 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:32.197+0000 7fbb354c6700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb300a5ce0 con 0x7fbb300a9980 2019-12-05T05:50:32.206 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:32.197+0000 7fbb354c6700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb300a5f80 con 0x7fbb300a6220 2019-12-05T05:50:32.206 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:32.197+0000 7fbb2f7fe700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fbb300a9980 0x7fbb300a4840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:50:32.206 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:32.197+0000 7fbb2effd700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbb300a6220 0x7fbb3007b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:50:32.206 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:32.197+0000 7fbb2f7fe700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fbb300a9980 0x7fbb300a4840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.42:58254/0 (socket says 172.21.15.42:58254) 2019-12-05T05:50:32.206 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:32.197+0000 7fbb2effd700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbb300a6220 0x7fbb3007b850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:42674/0 (socket says 172.21.15.42:42674) 2019-12-05T05:50:32.206 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:32.197+0000 7fbb2ffff700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fbb3007bda0 0x7fbb300a2e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:51688/0 (socket says 172.21.15.42:51688) 2019-12-05T05:50:32.207 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:32.197+0000 7fbb2f7fe700 1 -- 172.21.15.42:0/2958169229 learned_addr learned my addr 172.21.15.42:0/2958169229 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:50:32.207 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:32.197+0000 7fbb2effd700 1 -- 172.21.15.42:0/2958169229 >> v2:172.21.15.40:3300/0 conn(0x7fbb300a9980 msgr2=0x7fbb300a4840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:50:32.207 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:32.197+0000 7fbb2effd700 1 --2- 172.21.15.42:0/2958169229 >> v2:172.21.15.40:3300/0 conn(0x7fbb300a9980 0x7fbb300a4840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:50:32.207 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:32.197+0000 7fbb2effd700 1 -- 172.21.15.42:0/2958169229 >> v1:172.21.15.40:6789/0 conn(0x7fbb3007bda0 legacy=0x7fbb300a2e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:50:32.207 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:32.197+0000 7fbb2effd700 1 -- 172.21.15.42:0/2958169229 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb300a5970 con 0x7fbb300a6220 2019-12-05T05:50:32.207 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:32.197+0000 7fbb2effd700 1 --2- 172.21.15.42:0/2958169229 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbb300a6220 0x7fbb3007b850 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fbb24002f30 tx=0x7fbb24010b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:32.207 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:32.197+0000 7fbb2effd700 1 --2- 172.21.15.42:0/2958169229 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbb300a6220 0x7fbb3007b850 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fbb24002f30 tx=0x7fbb24010b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:32.208 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:32.197+0000 7fbb2effd700 1 --2- 172.21.15.42:0/2958169229 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbb300a6220 0x7fbb3007b850 secure :-1 s=READY pgs=343 cs=0 l=1 rx=0x7fbb24002f30 tx=0x7fbb24010b70).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:50:32.208 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:32.197+0000 7fbb2effd700 1 --2- 172.21.15.42:0/2958169229 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbb300a6220 0x7fbb3007b850 secure :-1 s=THROTTLE_DONE pgs=343 cs=0 l=1 rx=0x7fbb24002f30 tx=0x7fbb24010b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:32.208 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:32.197+0000 7fbb17fff700 1 -- 172.21.15.42:0/2958169229 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbb24012620 con 0x7fbb300a6220 2019-12-05T05:50:32.208 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:32.197+0000 7fbb2effd700 1 --2- 172.21.15.42:0/2958169229 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbb300a6220 0x7fbb3007b850 secure :-1 s=THROTTLE_DONE pgs=343 cs=0 l=1 rx=0x7fbb24002f30 tx=0x7fbb24010b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:32.208 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:32.197+0000 7fbb17fff700 1 -- 172.21.15.42:0/2958169229 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb24007150 con 0x7fbb300a6220 2019-12-05T05:50:32.208 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:32.197+0000 7fbb354c6700 1 -- 172.21.15.42:0/2958169229 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb300a3670 con 0x7fbb300a6220 2019-12-05T05:50:32.208 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:32.197+0000 7fbb2effd700 1 --2- 172.21.15.42:0/2958169229 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbb300a6220 0x7fbb3007b850 secure :-1 s=THROTTLE_DONE pgs=343 cs=0 l=1 rx=0x7fbb24002f30 tx=0x7fbb24010b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:32.209 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:32.197+0000 7fbb17fff700 1 -- 172.21.15.42:0/2958169229 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbb24012c00 con 0x7fbb300a6220 2019-12-05T05:50:32.209 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:32.197+0000 7fbb354c6700 1 -- 172.21.15.42:0/2958169229 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb30134580 con 0x7fbb300a6220 2019-12-05T05:50:32.209 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:32.201+0000 7fbb2effd700 1 --2- 172.21.15.42:0/2958169229 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbb300a6220 0x7fbb3007b850 secure :-1 s=THROTTLE_DONE pgs=343 cs=0 l=1 rx=0x7fbb24002f30 tx=0x7fbb24010b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:32.209 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:32.201+0000 7fbb17fff700 1 -- 172.21.15.42:0/2958169229 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fbb24005540 con 0x7fbb300a6220 2019-12-05T05:50:32.209 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:32.201+0000 7fbb2effd700 1 --2- 172.21.15.42:0/2958169229 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbb300a6220 0x7fbb3007b850 secure :-1 s=THROTTLE_DONE pgs=343 cs=0 l=1 rx=0x7fbb24002f30 tx=0x7fbb24010b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:32.209 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:32.201+0000 7fbb17fff700 1 --2- 172.21.15.42:0/2958169229 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbb1003ad40 0x7fbb1003d250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:50:32.209 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:32.201+0000 7fbb17fff700 1 -- 172.21.15.42:0/2958169229 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 8697+0+0 (secure 0 0 0) 0x7fbb2404b2d0 con 0x7fbb300a6220 2019-12-05T05:50:32.210 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:32.201+0000 7fbb2f7fe700 1 --2- 172.21.15.42:0/2958169229 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbb1003ad40 0x7fbb1003d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:50:32.210 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:32.201+0000 7fbb2f7fe700 1 --2- 172.21.15.42:0/2958169229 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbb1003ad40 0x7fbb1003d250 crc :-1 s=READY pgs=660 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:50:32.210 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:32.201+0000 7fbb354c6700 1 -- 172.21.15.42:0/2958169229 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb3005d110 con 0x7fbb300a6220 2019-12-05T05:50:32.215 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:32.209+0000 7fbb2effd700 1 --2- 172.21.15.42:0/2958169229 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbb300a6220 0x7fbb3007b850 secure :-1 s=THROTTLE_DONE pgs=343 cs=0 l=1 rx=0x7fbb24002f30 tx=0x7fbb24010b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:32.215 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:32.209+0000 7fbb17fff700 1 -- 172.21.15.42:0/2958169229 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fbb240322d0 con 0x7fbb300a6220 2019-12-05T05:50:32.386 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:32.381+0000 7fbb354c6700 1 -- 172.21.15.42:0/2958169229 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbb3005b8c0 con 0x7fbb300a6220 2019-12-05T05:50:32.386 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:32.381+0000 7fbb2effd700 1 --2- 172.21.15.42:0/2958169229 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbb300a6220 0x7fbb3007b850 secure :-1 s=THROTTLE_DONE pgs=343 cs=0 l=1 rx=0x7fbb24002f30 tx=0x7fbb24010b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:32.387 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:32.381+0000 7fbb17fff700 1 -- 172.21.15.42:0/2958169229 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 188 v188) v1 ==== 95+0+2112 (secure 0 0 0) 0x7fbb24032530 con 0x7fbb300a6220 2019-12-05T05:50:32.389 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:50:32.389 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":188,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6245,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/2081111170","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2081111170},{"type":"v1","addr":"172.21.15.40:6833","nonce":2081111170}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":188}],"filesystems":[{"mdsmap":{"epoch":187,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:49:23.679613+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8188},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8188":{"gid":8188,"name":"a","rank":0,"incarnation":184,"state":"up:active","state_seq":33,"addr":"172.21.15.42:6835/2456922143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2456922143},{"type":"v1","addr":"172.21.15.42:6835","nonce":2456922143}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:50:32.389 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 188 2019-12-05T05:50:32.391 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:32.385+0000 7fbb354c6700 1 -- 172.21.15.42:0/2958169229 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbb1003ad40 msgr2=0x7fbb1003d250 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:50:32.391 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:32.385+0000 7fbb354c6700 1 --2- 172.21.15.42:0/2958169229 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbb1003ad40 0x7fbb1003d250 crc :-1 s=READY pgs=660 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:50:32.392 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:32.385+0000 7fbb354c6700 1 -- 172.21.15.42:0/2958169229 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbb300a6220 msgr2=0x7fbb3007b850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:50:32.392 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:32.385+0000 7fbb354c6700 1 --2- 172.21.15.42:0/2958169229 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbb300a6220 0x7fbb3007b850 secure :-1 s=READY pgs=343 cs=0 l=1 rx=0x7fbb24002f30 tx=0x7fbb24010b70).stop 2019-12-05T05:50:32.392 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:32.385+0000 7fbb354c6700 1 -- 172.21.15.42:0/2958169229 shutdown_connections 2019-12-05T05:50:32.392 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:32.385+0000 7fbb354c6700 1 --2- 172.21.15.42:0/2958169229 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbb1003ad40 0x7fbb1003d250 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:50:32.392 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:32.385+0000 7fbb354c6700 1 --2- 172.21.15.42:0/2958169229 >> v2:172.21.15.40:3300/0 conn(0x7fbb300a9980 0x7fbb300a4840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:50:32.392 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:32.385+0000 7fbb354c6700 1 --2- 172.21.15.42:0/2958169229 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbb300a6220 0x7fbb3007b850 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:50:32.392 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:32.385+0000 7fbb354c6700 1 -- 172.21.15.42:0/2958169229 shutdown_connections 2019-12-05T05:50:32.393 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:32.389+0000 7fbb354c6700 1 -- 172.21.15.42:0/2958169229 wait complete. 2019-12-05T05:50:32.393 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:32.389+0000 7fbb354c6700 1 -- 172.21.15.42:0/2958169229 >> 172.21.15.42:0/2958169229 conn(0x7fbb300a8940 msgr2=0x7fbb3007c450 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:50:32.402 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T05:50:34.403 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:50:34.511 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:34.501+0000 7fb28779e700 1 Processor -- start 2019-12-05T05:50:34.511 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:34.501+0000 7fb28779e700 1 -- start start 2019-12-05T05:50:34.511 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:34.501+0000 7fb28779e700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb2800a7730 0x7fb28008df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:50:34.511 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:34.501+0000 7fb28779e700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb28007e710 0x7fb28007eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:50:34.511 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:34.501+0000 7fb28779e700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2800a0700 con 0x7fb28009f2d0 2019-12-05T05:50:34.512 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:34.501+0000 7fb28779e700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2800a0af0 con 0x7fb2800a7730 2019-12-05T05:50:34.512 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:34.501+0000 7fb28779e700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb28008e5a0 con 0x7fb28007e710 2019-12-05T05:50:34.512 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:34.501+0000 7fb28553a700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb28007e710 0x7fb28007eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:50:34.512 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:34.501+0000 7fb285d3b700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb2800a7730 0x7fb28008df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:50:34.512 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:34.501+0000 7fb28653c700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fb28009f2d0 0x7fb28009f700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:51704/0 (socket says 172.21.15.42:51704) 2019-12-05T05:50:34.512 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:34.501+0000 7fb28553a700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb28007e710 0x7fb28007eb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:42694/0 (socket says 172.21.15.42:42694) 2019-12-05T05:50:34.513 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:34.501+0000 7fb28653c700 1 -- 172.21.15.42:0/52552994 learned_addr learned my addr 172.21.15.42:0/52552994 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:50:34.513 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:34.501+0000 7fb285d3b700 1 -- 172.21.15.42:0/52552994 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb28007e710 msgr2=0x7fb28007eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:50:34.513 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:34.505+0000 7fb285d3b700 1 --2- 172.21.15.42:0/52552994 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb28007e710 0x7fb28007eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:50:34.513 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:34.505+0000 7fb285d3b700 1 -- 172.21.15.42:0/52552994 >> v1:172.21.15.40:6789/0 conn(0x7fb28009f2d0 legacy=0x7fb28009f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:50:34.513 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:34.505+0000 7fb284d39700 1 -- 172.21.15.42:0/52552994 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2386231978 0 0) 0x7fb2800a0700 con 0x7fb28009f2d0 2019-12-05T05:50:34.513 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:34.505+0000 7fb285d3b700 1 -- 172.21.15.42:0/52552994 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb28007f090 con 0x7fb2800a7730 2019-12-05T05:50:34.513 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:34.505+0000 7fb28553a700 1 --2- 172.21.15.42:0/52552994 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb28007e710 0x7fb28007eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:50:34.514 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:34.505+0000 7fb285d3b700 1 --2- 172.21.15.42:0/52552994 >> v2:172.21.15.40:3300/0 conn(0x7fb2800a7730 0x7fb28008df70 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fb27c008790 tx=0x7fb27c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:34.514 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:34.505+0000 7fb285d3b700 1 --2- 172.21.15.42:0/52552994 >> v2:172.21.15.40:3300/0 conn(0x7fb2800a7730 0x7fb28008df70 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fb27c008790 tx=0x7fb27c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:34.514 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:34.505+0000 7fb285d3b700 1 --2- 172.21.15.42:0/52552994 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb2800a7730 0x7fb28008df70 secure :-1 s=READY pgs=1180 cs=0 l=1 rx=0x7fb27c008790 tx=0x7fb27c00fc70).ready entity=mon.0 client_cookie=3fba3df7398c54b6 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:50:34.514 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:34.505+0000 7fb285d3b700 1 --2- 172.21.15.42:0/52552994 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb2800a7730 0x7fb28008df70 secure :-1 s=THROTTLE_DONE pgs=1180 cs=0 l=1 rx=0x7fb27c008790 tx=0x7fb27c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:34.514 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:34.505+0000 7fb284d39700 1 -- 172.21.15.42:0/52552994 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb27c0112f0 con 0x7fb2800a7730 2019-12-05T05:50:34.514 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:34.505+0000 7fb285d3b700 1 --2- 172.21.15.42:0/52552994 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb2800a7730 0x7fb28008df70 secure :-1 s=THROTTLE_DONE pgs=1180 cs=0 l=1 rx=0x7fb27c008790 tx=0x7fb27c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:34.514 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:34.505+0000 7fb285d3b700 1 --2- 172.21.15.42:0/52552994 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb2800a7730 0x7fb28008df70 secure :-1 s=THROTTLE_DONE pgs=1180 cs=0 l=1 rx=0x7fb27c008790 tx=0x7fb27c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:34.515 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:34.505+0000 7fb284d39700 1 -- 172.21.15.42:0/52552994 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb27c008ad0 con 0x7fb2800a7730 2019-12-05T05:50:34.515 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:34.505+0000 7fb284d39700 1 -- 172.21.15.42:0/52552994 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb27c0118d0 con 0x7fb2800a7730 2019-12-05T05:50:34.515 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:34.505+0000 7fb28779e700 1 -- 172.21.15.42:0/52552994 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb2800a7730 msgr2=0x7fb28008df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:50:34.515 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:34.505+0000 7fb28779e700 1 --2- 172.21.15.42:0/52552994 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb2800a7730 0x7fb28008df70 secure :-1 s=READY pgs=1180 cs=0 l=1 rx=0x7fb27c008790 tx=0x7fb27c00fc70).stop 2019-12-05T05:50:34.515 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:34.505+0000 7fb28779e700 1 -- 172.21.15.42:0/52552994 shutdown_connections 2019-12-05T05:50:34.515 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:34.505+0000 7fb28779e700 1 --2- 172.21.15.42:0/52552994 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb2800a7730 0x7fb28008df70 unknown :-1 s=CLOSED pgs=1180 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:50:34.516 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:34.505+0000 7fb28779e700 1 --2- 172.21.15.42:0/52552994 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb28007e710 0x7fb28007eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:50:34.516 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:34.505+0000 7fb28779e700 1 -- 172.21.15.42:0/52552994 shutdown_connections 2019-12-05T05:50:34.516 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:34.505+0000 7fb28779e700 1 -- 172.21.15.42:0/52552994 wait complete. 2019-12-05T05:50:34.516 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:34.505+0000 7fb28779e700 1 -- 172.21.15.42:0/52552994 >> 172.21.15.42:0/52552994 conn(0x7fb2800aabd0 msgr2=0x7fb2800a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:50:34.519 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:34.509+0000 7fb28779e700 1 Processor -- start 2019-12-05T05:50:34.520 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:34.509+0000 7fb28779e700 1 -- start start 2019-12-05T05:50:34.520 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:34.509+0000 7fb28779e700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb28011a370 0x7fb280121fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:50:34.520 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:34.509+0000 7fb28779e700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb28011c110 con 0x7fb2801168d0 2019-12-05T05:50:34.520 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:34.509+0000 7fb28779e700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2800a6f70 con 0x7fb28011bce0 2019-12-05T05:50:34.520 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:34.509+0000 7fb28779e700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2800a71a0 con 0x7fb28011a370 2019-12-05T05:50:34.520 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:34.509+0000 7fb28653c700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fb28011bce0 0x7fb28008f360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:42272/0 (socket says 172.21.15.42:42272) 2019-12-05T05:50:34.520 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:34.509+0000 7fb28653c700 1 -- 172.21.15.42:0/3119162034 learned_addr learned my addr 172.21.15.42:0/3119162034 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:50:34.521 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:34.509+0000 7fb28553a700 1 --2- 172.21.15.42:0/3119162034 >> v2:172.21.15.40:3300/0 conn(0x7fb28011a370 0x7fb280121fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:50:34.521 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:34.509+0000 7fb2767fc700 1 -- 172.21.15.42:0/3119162034 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3135733178 0 0) 0x7fb2800a6f70 con 0x7fb28011bce0 2019-12-05T05:50:34.521 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:34.509+0000 7fb2767fc700 1 -- 172.21.15.42:0/3119162034 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2680030b0 con 0x7fb28011bce0 2019-12-05T05:50:34.521 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:34.509+0000 7fb28553a700 1 -- 172.21.15.42:0/3119162034 >> v1:172.21.15.42:6789/0 conn(0x7fb28011bce0 legacy=0x7fb28008f360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:50:34.521 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:34.509+0000 7fb28553a700 1 -- 172.21.15.42:0/3119162034 >> v1:172.21.15.40:6789/0 conn(0x7fb2801168d0 legacy=0x7fb28008eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:50:34.522 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:34.509+0000 7fb28553a700 1 -- 172.21.15.42:0/3119162034 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2801165f0 con 0x7fb28011a370 2019-12-05T05:50:34.522 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:34.509+0000 7fb28553a700 1 --2- 172.21.15.42:0/3119162034 >> v2:172.21.15.40:3300/0 conn(0x7fb28011a370 0x7fb280121fb0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fb270002f30 tx=0x7fb270010c70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:34.522 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:34.509+0000 7fb28553a700 1 --2- 172.21.15.42:0/3119162034 >> v2:172.21.15.40:3300/0 conn(0x7fb28011a370 0x7fb280121fb0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fb270002f30 tx=0x7fb270010c70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:34.522 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:34.509+0000 7fb28553a700 1 --2- 172.21.15.42:0/3119162034 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb28011a370 0x7fb280121fb0 secure :-1 s=READY pgs=1182 cs=0 l=1 rx=0x7fb270002f30 tx=0x7fb270010c70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:50:34.522 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:34.513+0000 7fb28553a700 1 --2- 172.21.15.42:0/3119162034 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb28011a370 0x7fb280121fb0 secure :-1 s=THROTTLE_DONE pgs=1182 cs=0 l=1 rx=0x7fb270002f30 tx=0x7fb270010c70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:34.523 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:34.513+0000 7fb28553a700 1 --2- 172.21.15.42:0/3119162034 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb28011a370 0x7fb280121fb0 secure :-1 s=THROTTLE_DONE pgs=1182 cs=0 l=1 rx=0x7fb270002f30 tx=0x7fb270010c70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:34.523 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:34.513+0000 7fb2767fc700 1 -- 172.21.15.42:0/3119162034 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb270005890 con 0x7fb28011a370 2019-12-05T05:50:34.523 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:34.513+0000 7fb28553a700 1 --2- 172.21.15.42:0/3119162034 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb28011a370 0x7fb280121fb0 secure :-1 s=THROTTLE_DONE pgs=1182 cs=0 l=1 rx=0x7fb270002f30 tx=0x7fb270010c70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:34.523 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:34.513+0000 7fb2767fc700 1 -- 172.21.15.42:0/3119162034 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb270007160 con 0x7fb28011a370 2019-12-05T05:50:34.523 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:34.513+0000 7fb28779e700 1 -- 172.21.15.42:0/3119162034 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb280078300 con 0x7fb28011a370 2019-12-05T05:50:34.523 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:34.513+0000 7fb2767fc700 1 -- 172.21.15.42:0/3119162034 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb270012330 con 0x7fb28011a370 2019-12-05T05:50:34.523 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:34.513+0000 7fb28779e700 1 -- 172.21.15.42:0/3119162034 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb280078690 con 0x7fb28011a370 2019-12-05T05:50:34.524 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:34.513+0000 7fb28553a700 1 --2- 172.21.15.42:0/3119162034 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb28011a370 0x7fb280121fb0 secure :-1 s=THROTTLE_DONE pgs=1182 cs=0 l=1 rx=0x7fb270002f30 tx=0x7fb270010c70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:34.524 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:34.513+0000 7fb28553a700 1 --2- 172.21.15.42:0/3119162034 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb28011a370 0x7fb280121fb0 secure :-1 s=THROTTLE_DONE pgs=1182 cs=0 l=1 rx=0x7fb270002f30 tx=0x7fb270010c70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:34.524 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:34.513+0000 7fb2767fc700 1 -- 172.21.15.42:0/3119162034 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fb270012640 con 0x7fb28011a370 2019-12-05T05:50:34.524 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:34.513+0000 7fb2767fc700 1 --2- 172.21.15.42:0/3119162034 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb26803b260 0x7fb26803d770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:50:34.524 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:34.513+0000 7fb2767fc700 1 -- 172.21.15.42:0/3119162034 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 8697+0+0 (secure 0 0 0) 0x7fb2700090b0 con 0x7fb28011a370 2019-12-05T05:50:34.524 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:34.513+0000 7fb285d3b700 1 --2- 172.21.15.42:0/3119162034 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb26803b260 0x7fb26803d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:50:34.524 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:34.513+0000 7fb285d3b700 1 --2- 172.21.15.42:0/3119162034 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb26803b260 0x7fb26803d770 crc :-1 s=READY pgs=661 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:50:34.525 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:34.517+0000 7fb28779e700 1 -- 172.21.15.42:0/3119162034 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb28005d100 con 0x7fb28011a370 2019-12-05T05:50:34.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:34.525+0000 7fb28553a700 1 --2- 172.21.15.42:0/3119162034 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb28011a370 0x7fb280121fb0 secure :-1 s=THROTTLE_DONE pgs=1182 cs=0 l=1 rx=0x7fb270002f30 tx=0x7fb270010c70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:34.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:34.525+0000 7fb2767fc700 1 -- 172.21.15.42:0/3119162034 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fb2801165f0 con 0x7fb28011a370 2019-12-05T05:50:34.698 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:34.693+0000 7fb28779e700 1 -- 172.21.15.42:0/3119162034 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb28005b8b0 con 0x7fb28011a370 2019-12-05T05:50:34.698 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:34.693+0000 7fb28553a700 1 --2- 172.21.15.42:0/3119162034 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb28011a370 0x7fb280121fb0 secure :-1 s=THROTTLE_DONE pgs=1182 cs=0 l=1 rx=0x7fb270002f30 tx=0x7fb270010c70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:34.699 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:34.693+0000 7fb2767fc700 1 -- 172.21.15.42:0/3119162034 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 188 v188) v1 ==== 95+0+2112 (secure 0 0 0) 0x7fb2700329b0 con 0x7fb28011a370 2019-12-05T05:50:34.701 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 188 2019-12-05T05:50:34.701 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:50:34.701 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":188,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6245,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/2081111170","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2081111170},{"type":"v1","addr":"172.21.15.40:6833","nonce":2081111170}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":188}],"filesystems":[{"mdsmap":{"epoch":187,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:49:23.679613+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8188},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8188":{"gid":8188,"name":"a","rank":0,"incarnation":184,"state":"up:active","state_seq":33,"addr":"172.21.15.42:6835/2456922143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2456922143},{"type":"v1","addr":"172.21.15.42:6835","nonce":2456922143}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:50:34.703 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:34.697+0000 7fb28779e700 1 -- 172.21.15.42:0/3119162034 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb26803b260 msgr2=0x7fb26803d770 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:50:34.704 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:34.697+0000 7fb28779e700 1 --2- 172.21.15.42:0/3119162034 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb26803b260 0x7fb26803d770 crc :-1 s=READY pgs=661 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:50:34.704 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:34.701+0000 7fb28779e700 1 -- 172.21.15.42:0/3119162034 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb28011a370 msgr2=0x7fb280121fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:50:34.704 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:34.701+0000 7fb28779e700 1 --2- 172.21.15.42:0/3119162034 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb28011a370 0x7fb280121fb0 secure :-1 s=READY pgs=1182 cs=0 l=1 rx=0x7fb270002f30 tx=0x7fb270010c70).stop 2019-12-05T05:50:34.704 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:34.701+0000 7fb28779e700 1 -- 172.21.15.42:0/3119162034 shutdown_connections 2019-12-05T05:50:34.704 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:34.701+0000 7fb28779e700 1 --2- 172.21.15.42:0/3119162034 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb26803b260 0x7fb26803d770 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:50:34.704 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:34.701+0000 7fb28779e700 1 --2- 172.21.15.42:0/3119162034 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb28011a370 0x7fb280121fb0 unknown :-1 s=CLOSED pgs=1182 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:50:34.704 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:34.701+0000 7fb28779e700 1 -- 172.21.15.42:0/3119162034 shutdown_connections 2019-12-05T05:50:34.705 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:34.701+0000 7fb28779e700 1 -- 172.21.15.42:0/3119162034 wait complete. 2019-12-05T05:50:34.705 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:34.701+0000 7fb28779e700 1 -- 172.21.15.42:0/3119162034 >> 172.21.15.42:0/3119162034 conn(0x7fb2800a8290 msgr2=0x7fb28009f2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:50:34.713 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T05:50:36.714 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:50:36.827 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:36.813+0000 7fb45b51a700 1 Processor -- start 2019-12-05T05:50:36.827 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:36.813+0000 7fb45b51a700 1 -- start start 2019-12-05T05:50:36.827 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:36.813+0000 7fb45b51a700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb45409fae0 0x7fb45414fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:50:36.828 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:36.813+0000 7fb45b51a700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4540a0710 con 0x7fb45408df80 2019-12-05T05:50:36.828 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:36.813+0000 7fb45b51a700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb45407ef20 con 0x7fb4540a7740 2019-12-05T05:50:36.828 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:36.813+0000 7fb45b51a700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4540a0b00 con 0x7fb45409fae0 2019-12-05T05:50:36.828 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:36.813+0000 7fb459ab7700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fb4540a7740 0x7fb45407e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:42278/0 (socket says 172.21.15.42:42278) 2019-12-05T05:50:36.828 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:36.813+0000 7fb459ab7700 1 -- 172.21.15.42:0/2581261177 learned_addr learned my addr 172.21.15.42:0/2581261177 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:50:36.828 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:36.813+0000 7fb45a2b8700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb45409fae0 0x7fb45414fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:50:36.829 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:36.817+0000 7fb458ab5700 1 -- 172.21.15.42:0/2581261177 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1433890397 0 0) 0x7fb45407ef20 con 0x7fb4540a7740 2019-12-05T05:50:36.829 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:36.817+0000 7fb458ab5700 1 -- 172.21.15.42:0/2581261177 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4400030b0 con 0x7fb4540a7740 2019-12-05T05:50:36.829 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:36.817+0000 7fb45a2b8700 1 -- 172.21.15.42:0/2581261177 >> v1:172.21.15.42:6789/0 conn(0x7fb4540a7740 legacy=0x7fb45407e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:50:36.829 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:36.817+0000 7fb45a2b8700 1 -- 172.21.15.42:0/2581261177 >> v1:172.21.15.40:6789/0 conn(0x7fb45408df80 legacy=0x7fb45409f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:50:36.829 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:36.817+0000 7fb458ab5700 1 -- 172.21.15.42:0/2581261177 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 33310672 0 0) 0x7fb4540a0710 con 0x7fb45408df80 2019-12-05T05:50:36.829 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:36.817+0000 7fb45a2b8700 1 -- 172.21.15.42:0/2581261177 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4540a9ba0 con 0x7fb45409fae0 2019-12-05T05:50:36.829 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:36.817+0000 7fb45a2b8700 1 --2- 172.21.15.42:0/2581261177 >> v2:172.21.15.42:3300/0 conn(0x7fb45409fae0 0x7fb45414fff0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fb44c008b10 tx=0x7fb44c00ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:36.830 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:36.817+0000 7fb45a2b8700 1 --2- 172.21.15.42:0/2581261177 >> v2:172.21.15.42:3300/0 conn(0x7fb45409fae0 0x7fb45414fff0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fb44c008b10 tx=0x7fb44c00ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:36.830 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:36.817+0000 7fb45a2b8700 1 --2- 172.21.15.42:0/2581261177 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb45409fae0 0x7fb45414fff0 secure :-1 s=READY pgs=1535 cs=0 l=1 rx=0x7fb44c008b10 tx=0x7fb44c00ff70).ready entity=mon.1 client_cookie=d29ad6cd1cb066c6 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:50:36.830 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:36.817+0000 7fb45a2b8700 1 --2- 172.21.15.42:0/2581261177 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb45409fae0 0x7fb45414fff0 secure :-1 s=THROTTLE_DONE pgs=1535 cs=0 l=1 rx=0x7fb44c008b10 tx=0x7fb44c00ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:36.830 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:36.817+0000 7fb458ab5700 1 -- 172.21.15.42:0/2581261177 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb44c010970 con 0x7fb45409fae0 2019-12-05T05:50:36.830 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:36.817+0000 7fb45a2b8700 1 --2- 172.21.15.42:0/2581261177 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb45409fae0 0x7fb45414fff0 secure :-1 s=THROTTLE_DONE pgs=1535 cs=0 l=1 rx=0x7fb44c008b10 tx=0x7fb44c00ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:36.830 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:36.817+0000 7fb45a2b8700 1 --2- 172.21.15.42:0/2581261177 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb45409fae0 0x7fb45414fff0 secure :-1 s=THROTTLE_DONE pgs=1535 cs=0 l=1 rx=0x7fb44c008b10 tx=0x7fb44c00ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:36.831 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:36.817+0000 7fb458ab5700 1 -- 172.21.15.42:0/2581261177 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb44c008b50 con 0x7fb45409fae0 2019-12-05T05:50:36.831 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:36.817+0000 7fb458ab5700 1 -- 172.21.15.42:0/2581261177 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb44c0110c0 con 0x7fb45409fae0 2019-12-05T05:50:36.831 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:36.817+0000 7fb45b51a700 1 -- 172.21.15.42:0/2581261177 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb45409fae0 msgr2=0x7fb45414fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:50:36.831 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:36.817+0000 7fb45b51a700 1 --2- 172.21.15.42:0/2581261177 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb45409fae0 0x7fb45414fff0 secure :-1 s=READY pgs=1535 cs=0 l=1 rx=0x7fb44c008b10 tx=0x7fb44c00ff70).stop 2019-12-05T05:50:36.831 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:36.817+0000 7fb45b51a700 1 -- 172.21.15.42:0/2581261177 shutdown_connections 2019-12-05T05:50:36.831 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:36.817+0000 7fb45b51a700 1 --2- 172.21.15.42:0/2581261177 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb45409fae0 0x7fb45414fff0 unknown :-1 s=CLOSED pgs=1535 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:50:36.831 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:36.817+0000 7fb45b51a700 1 -- 172.21.15.42:0/2581261177 shutdown_connections 2019-12-05T05:50:36.832 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:36.817+0000 7fb45b51a700 1 -- 172.21.15.42:0/2581261177 wait complete. 2019-12-05T05:50:36.832 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:36.817+0000 7fb45b51a700 1 -- 172.21.15.42:0/2581261177 >> 172.21.15.42:0/2581261177 conn(0x7fb4540aabe0 msgr2=0x7fb4540a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:50:36.832 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:36.821+0000 7fb45b51a700 1 Processor -- start 2019-12-05T05:50:36.832 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:36.821+0000 7fb45b51a700 1 -- start start 2019-12-05T05:50:36.832 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:36.821+0000 7fb45b51a700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb45411d030 0x7fb454146300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:50:36.832 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:36.821+0000 7fb45b51a700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb454130440 0x7fb454137890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:50:36.833 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:36.821+0000 7fb45b51a700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4541274f0 con 0x7fb4541477a0 2019-12-05T05:50:36.834 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:36.821+0000 7fb45b51a700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb454145b20 con 0x7fb45411d030 2019-12-05T05:50:36.834 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:36.821+0000 7fb45b51a700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb454145dc0 con 0x7fb454130440 2019-12-05T05:50:36.834 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:36.821+0000 7fb459ab7700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb45411d030 0x7fb454146300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:50:36.834 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:36.821+0000 7fb459ab7700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb45411d030 0x7fb454146300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:42246/0 (socket says 172.21.15.42:42246) 2019-12-05T05:50:36.834 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:36.821+0000 7fb4592b6700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fb4541477a0 0x7fb454126cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:42286/0 (socket says 172.21.15.42:42286) 2019-12-05T05:50:36.834 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:36.821+0000 7fb459ab7700 1 -- 172.21.15.42:0/1633295469 learned_addr learned my addr 172.21.15.42:0/1633295469 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:50:36.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:36.821+0000 7fb45a2b8700 1 --2- 172.21.15.42:0/1633295469 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb454130440 0x7fb454137890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:50:36.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:36.821+0000 7fb459ab7700 1 -- 172.21.15.42:0/1633295469 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb454130440 msgr2=0x7fb454137890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:50:36.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:36.821+0000 7fb459ab7700 1 --2- 172.21.15.42:0/1633295469 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb454130440 0x7fb454137890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:50:36.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:36.821+0000 7fb459ab7700 1 -- 172.21.15.42:0/1633295469 >> v1:172.21.15.42:6789/0 conn(0x7fb4541477a0 legacy=0x7fb454126cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:50:36.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:36.821+0000 7fb44a7fc700 1 -- 172.21.15.42:0/1633295469 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 916319871 0 0) 0x7fb4541274f0 con 0x7fb4541477a0 2019-12-05T05:50:36.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:36.821+0000 7fb459ab7700 1 -- 172.21.15.42:0/1633295469 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb454146a30 con 0x7fb45411d030 2019-12-05T05:50:36.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:36.821+0000 7fb459ab7700 1 --2- 172.21.15.42:0/1633295469 >> v2:172.21.15.42:3300/0 conn(0x7fb45411d030 0x7fb454146300 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fb45405d370 tx=0x7fb450010980).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:36.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:36.821+0000 7fb459ab7700 1 --2- 172.21.15.42:0/1633295469 >> v2:172.21.15.42:3300/0 conn(0x7fb45411d030 0x7fb454146300 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fb45405d370 tx=0x7fb450010980).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:36.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:36.821+0000 7fb459ab7700 1 --2- 172.21.15.42:0/1633295469 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb45411d030 0x7fb454146300 secure :-1 s=READY pgs=1537 cs=0 l=1 rx=0x7fb45405d370 tx=0x7fb450010980).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:50:36.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:36.821+0000 7fb459ab7700 1 --2- 172.21.15.42:0/1633295469 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb45411d030 0x7fb454146300 secure :-1 s=THROTTLE_DONE pgs=1537 cs=0 l=1 rx=0x7fb45405d370 tx=0x7fb450010980).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:36.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:36.821+0000 7fb44a7fc700 1 -- 172.21.15.42:0/1633295469 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb450005b40 con 0x7fb45411d030 2019-12-05T05:50:36.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:36.821+0000 7fb459ab7700 1 --2- 172.21.15.42:0/1633295469 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb45411d030 0x7fb454146300 secure :-1 s=THROTTLE_DONE pgs=1537 cs=0 l=1 rx=0x7fb45405d370 tx=0x7fb450010980).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:36.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:36.821+0000 7fb45b51a700 1 -- 172.21.15.42:0/1633295469 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb454130870 con 0x7fb45411d030 2019-12-05T05:50:36.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:36.821+0000 7fb44a7fc700 1 -- 172.21.15.42:0/1633295469 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb450005760 con 0x7fb45411d030 2019-12-05T05:50:36.837 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:36.821+0000 7fb459ab7700 1 --2- 172.21.15.42:0/1633295469 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb45411d030 0x7fb454146300 secure :-1 s=THROTTLE_DONE pgs=1537 cs=0 l=1 rx=0x7fb45405d370 tx=0x7fb450010980).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:36.837 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:36.821+0000 7fb44a7fc700 1 -- 172.21.15.42:0/1633295469 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb4500062b0 con 0x7fb45411d030 2019-12-05T05:50:36.837 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:36.821+0000 7fb45b51a700 1 -- 172.21.15.42:0/1633295469 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb454146d80 con 0x7fb45411d030 2019-12-05T05:50:36.838 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:36.825+0000 7fb459ab7700 1 --2- 172.21.15.42:0/1633295469 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb45411d030 0x7fb454146300 secure :-1 s=THROTTLE_DONE pgs=1537 cs=0 l=1 rx=0x7fb45405d370 tx=0x7fb450010980).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:36.838 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:36.825+0000 7fb44a7fc700 1 -- 172.21.15.42:0/1633295469 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fb450006590 con 0x7fb45411d030 2019-12-05T05:50:36.838 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:36.825+0000 7fb459ab7700 1 --2- 172.21.15.42:0/1633295469 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb45411d030 0x7fb454146300 secure :-1 s=THROTTLE_DONE pgs=1537 cs=0 l=1 rx=0x7fb45405d370 tx=0x7fb450010980).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:36.839 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:36.825+0000 7fb44a7fc700 1 --2- 172.21.15.42:0/1633295469 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb43c03ac80 0x7fb43c03d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:50:36.839 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:36.825+0000 7fb44a7fc700 1 -- 172.21.15.42:0/1633295469 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 8697+0+0 (secure 0 0 0) 0x7fb45004b050 con 0x7fb45411d030 2019-12-05T05:50:36.839 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:36.825+0000 7fb4592b6700 1 --2- 172.21.15.42:0/1633295469 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb43c03ac80 0x7fb43c03d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:50:36.839 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:36.825+0000 7fb4592b6700 1 --2- 172.21.15.42:0/1633295469 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb43c03ac80 0x7fb43c03d190 crc :-1 s=READY pgs=662 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:50:36.839 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:36.829+0000 7fb45b51a700 1 -- 172.21.15.42:0/1633295469 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb45405d110 con 0x7fb45411d030 2019-12-05T05:50:36.841 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:36.837+0000 7fb459ab7700 1 --2- 172.21.15.42:0/1633295469 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb45411d030 0x7fb454146300 secure :-1 s=THROTTLE_DONE pgs=1537 cs=0 l=1 rx=0x7fb45405d370 tx=0x7fb450010980).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:36.842 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:36.837+0000 7fb44a7fc700 1 -- 172.21.15.42:0/1633295469 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fb450032030 con 0x7fb45411d030 2019-12-05T05:50:37.011 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:37.005+0000 7fb45b51a700 1 -- 172.21.15.42:0/1633295469 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb45405b8c0 con 0x7fb45411d030 2019-12-05T05:50:37.012 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:37.009+0000 7fb459ab7700 1 --2- 172.21.15.42:0/1633295469 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb45411d030 0x7fb454146300 secure :-1 s=THROTTLE_DONE pgs=1537 cs=0 l=1 rx=0x7fb45405d370 tx=0x7fb450010980).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:37.012 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:37.009+0000 7fb44a7fc700 1 -- 172.21.15.42:0/1633295469 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 188 v188) v1 ==== 95+0+2112 (secure 0 0 0) 0x7fb450032290 con 0x7fb45411d030 2019-12-05T05:50:37.012 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 188 2019-12-05T05:50:37.012 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:50:37.012 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":188,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6245,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/2081111170","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2081111170},{"type":"v1","addr":"172.21.15.40:6833","nonce":2081111170}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":188}],"filesystems":[{"mdsmap":{"epoch":187,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:49:23.679613+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8188},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8188":{"gid":8188,"name":"a","rank":0,"incarnation":184,"state":"up:active","state_seq":33,"addr":"172.21.15.42:6835/2456922143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2456922143},{"type":"v1","addr":"172.21.15.42:6835","nonce":2456922143}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:50:37.014 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:37.009+0000 7fb45b51a700 1 -- 172.21.15.42:0/1633295469 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb43c03ac80 msgr2=0x7fb43c03d190 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:50:37.014 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:37.009+0000 7fb45b51a700 1 --2- 172.21.15.42:0/1633295469 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb43c03ac80 0x7fb43c03d190 crc :-1 s=READY pgs=662 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:50:37.015 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:37.009+0000 7fb45b51a700 1 -- 172.21.15.42:0/1633295469 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb45411d030 msgr2=0x7fb454146300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:50:37.015 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:37.009+0000 7fb45b51a700 1 --2- 172.21.15.42:0/1633295469 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb45411d030 0x7fb454146300 secure :-1 s=READY pgs=1537 cs=0 l=1 rx=0x7fb45405d370 tx=0x7fb450010980).stop 2019-12-05T05:50:37.015 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:37.009+0000 7fb45b51a700 1 -- 172.21.15.42:0/1633295469 shutdown_connections 2019-12-05T05:50:37.015 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:37.009+0000 7fb45b51a700 1 --2- 172.21.15.42:0/1633295469 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb43c03ac80 0x7fb43c03d190 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:50:37.015 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:37.009+0000 7fb45b51a700 1 --2- 172.21.15.42:0/1633295469 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb454130440 0x7fb454137890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:50:37.015 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:37.009+0000 7fb45b51a700 1 --2- 172.21.15.42:0/1633295469 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb45411d030 0x7fb454146300 unknown :-1 s=CLOSED pgs=1537 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:50:37.016 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:37.009+0000 7fb45b51a700 1 -- 172.21.15.42:0/1633295469 shutdown_connections 2019-12-05T05:50:37.016 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:37.009+0000 7fb45b51a700 1 -- 172.21.15.42:0/1633295469 wait complete. 2019-12-05T05:50:37.016 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:37.009+0000 7fb45b51a700 1 -- 172.21.15.42:0/1633295469 >> 172.21.15.42:0/1633295469 conn(0x7fb45407de90 msgr2=0x7fb45407e720 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:50:37.024 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T05:50:39.025 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:50:39.137 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:39.124+0000 7f766e942700 1 Processor -- start 2019-12-05T05:50:39.137 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:39.124+0000 7f766e942700 1 -- start start 2019-12-05T05:50:39.137 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:39.128+0000 7f766e942700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f766808df80 0x7f766809f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:50:39.137 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:39.128+0000 7f766e942700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f766809f830 0x7f766814efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:50:39.138 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:39.128+0000 7f766e942700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f766809fc60 con 0x7f76680a7740 2019-12-05T05:50:39.138 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:39.128+0000 7f766e942700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76680a0b00 con 0x7f766809f830 2019-12-05T05:50:39.138 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:39.128+0000 7f766e942700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f766808e5b0 con 0x7f766808df80 2019-12-05T05:50:39.138 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:39.128+0000 7f766cedf700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f76680a7740 0x7f766807e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:42296/0 (socket says 172.21.15.42:42296) 2019-12-05T05:50:39.138 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:39.128+0000 7f766cedf700 1 -- 172.21.15.42:0/2421650906 learned_addr learned my addr 172.21.15.42:0/2421650906 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:50:39.138 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:39.128+0000 7f766d6e0700 1 --2- 172.21.15.42:0/2421650906 >> v2:172.21.15.42:3300/0 conn(0x7f766809f830 0x7f766814efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:50:39.139 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:39.128+0000 7f7667fff700 1 --2- 172.21.15.42:0/2421650906 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f766808df80 0x7f766809f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:50:39.139 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:39.128+0000 7f766d6e0700 1 -- 172.21.15.42:0/2421650906 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f766808df80 msgr2=0x7f766809f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:50:39.139 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:39.128+0000 7f766d6e0700 1 --2- 172.21.15.42:0/2421650906 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f766808df80 0x7f766809f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:50:39.139 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:39.128+0000 7f766d6e0700 1 -- 172.21.15.42:0/2421650906 >> v1:172.21.15.42:6789/0 conn(0x7f76680a7740 legacy=0x7f766807e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:50:39.139 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:39.128+0000 7f766d6e0700 1 -- 172.21.15.42:0/2421650906 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76680a0710 con 0x7f766809f830 2019-12-05T05:50:39.139 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:39.128+0000 7f76677fe700 1 -- 172.21.15.42:0/2421650906 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 155913863 0 0) 0x7f766809fc60 con 0x7f76680a7740 2019-12-05T05:50:39.139 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:39.128+0000 7f766d6e0700 1 --2- 172.21.15.42:0/2421650906 >> v2:172.21.15.42:3300/0 conn(0x7f766809f830 0x7f766814efe0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f7660008b10 tx=0x7f766000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:39.140 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:39.128+0000 7f766d6e0700 1 --2- 172.21.15.42:0/2421650906 >> v2:172.21.15.42:3300/0 conn(0x7f766809f830 0x7f766814efe0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f7660008b10 tx=0x7f766000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:39.140 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:39.128+0000 7f766d6e0700 1 --2- 172.21.15.42:0/2421650906 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f766809f830 0x7f766814efe0 secure :-1 s=READY pgs=1539 cs=0 l=1 rx=0x7f7660008b10 tx=0x7f766000fff0).ready entity=mon.1 client_cookie=c90b0417c04a9400 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:50:39.140 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:39.128+0000 7f766d6e0700 1 --2- 172.21.15.42:0/2421650906 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f766809f830 0x7f766814efe0 secure :-1 s=THROTTLE_DONE pgs=1539 cs=0 l=1 rx=0x7f7660008b10 tx=0x7f766000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:39.140 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:39.128+0000 7f76677fe700 1 -- 172.21.15.42:0/2421650906 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7660010900 con 0x7f766809f830 2019-12-05T05:50:39.140 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:39.128+0000 7f766d6e0700 1 --2- 172.21.15.42:0/2421650906 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f766809f830 0x7f766814efe0 secure :-1 s=THROTTLE_DONE pgs=1539 cs=0 l=1 rx=0x7f7660008b10 tx=0x7f766000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:39.140 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:39.128+0000 7f766d6e0700 1 --2- 172.21.15.42:0/2421650906 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f766809f830 0x7f766814efe0 secure :-1 s=THROTTLE_DONE pgs=1539 cs=0 l=1 rx=0x7f7660008b10 tx=0x7f766000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:39.140 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:39.128+0000 7f76677fe700 1 -- 172.21.15.42:0/2421650906 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7660008b50 con 0x7f766809f830 2019-12-05T05:50:39.141 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:39.128+0000 7f76677fe700 1 -- 172.21.15.42:0/2421650906 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f76600110c0 con 0x7f766809f830 2019-12-05T05:50:39.141 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:39.128+0000 7f766e942700 1 -- 172.21.15.42:0/2421650906 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f766809f830 msgr2=0x7f766814efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:50:39.141 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:39.128+0000 7f766e942700 1 --2- 172.21.15.42:0/2421650906 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f766809f830 0x7f766814efe0 secure :-1 s=READY pgs=1539 cs=0 l=1 rx=0x7f7660008b10 tx=0x7f766000fff0).stop 2019-12-05T05:50:39.141 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:39.128+0000 7f766e942700 1 -- 172.21.15.42:0/2421650906 shutdown_connections 2019-12-05T05:50:39.141 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:39.128+0000 7f766e942700 1 --2- 172.21.15.42:0/2421650906 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f766809f830 0x7f766814efe0 unknown :-1 s=CLOSED pgs=1539 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:50:39.141 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:39.128+0000 7f766e942700 1 --2- 172.21.15.42:0/2421650906 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f766808df80 0x7f766809f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:50:39.141 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:39.128+0000 7f766e942700 1 -- 172.21.15.42:0/2421650906 shutdown_connections 2019-12-05T05:50:39.142 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:39.128+0000 7f766e942700 1 -- 172.21.15.42:0/2421650906 wait complete. 2019-12-05T05:50:39.142 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:39.128+0000 7f766e942700 1 -- 172.21.15.42:0/2421650906 >> 172.21.15.42:0/2421650906 conn(0x7f76680aabe0 msgr2=0x7f76680a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:50:39.142 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:39.132+0000 7f766e942700 1 Processor -- start 2019-12-05T05:50:39.142 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:39.132+0000 7f766e942700 1 -- start start 2019-12-05T05:50:39.146 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:39.132+0000 7f766e942700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7668118110 0x7f76681161c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:50:39.146 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:39.132+0000 7f766e942700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f766811d780 0x7f766808f160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:50:39.146 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:39.132+0000 7f766e942700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7668121f70 0x7f76681223a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:50:39.146 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:39.132+0000 7f766e942700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76681178f0 con 0x7f7668121f70 2019-12-05T05:50:39.147 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:39.132+0000 7f766e942700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7668117b90 con 0x7f766811d780 2019-12-05T05:50:39.147 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:39.132+0000 7f766e942700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76681410a0 con 0x7f7668118110 2019-12-05T05:50:39.147 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:39.132+0000 7f7667fff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f766811d780 0x7f766808f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:50:39.147 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:39.132+0000 7f7667fff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f766811d780 0x7f766808f160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:42266/0 (socket says 172.21.15.42:42266) 2019-12-05T05:50:39.147 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:39.132+0000 7f7667fff700 1 -- 172.21.15.42:0/3780356297 learned_addr learned my addr 172.21.15.42:0/3780356297 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:50:39.147 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:39.132+0000 7f766d6e0700 1 --2- 172.21.15.42:0/3780356297 >> v2:172.21.15.40:3300/0 conn(0x7f7668121f70 0x7f76681223a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:50:39.147 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:39.132+0000 7f766cedf700 1 --2- 172.21.15.42:0/3780356297 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7668118110 0x7f76681161c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:50:39.148 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:39.132+0000 7f7667fff700 1 -- 172.21.15.42:0/3780356297 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7668118110 msgr2=0x7f76681161c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:50:39.148 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:39.132+0000 7f7667fff700 1 --2- 172.21.15.42:0/3780356297 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7668118110 0x7f76681161c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:50:39.148 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:39.132+0000 7f7667fff700 1 -- 172.21.15.42:0/3780356297 >> v2:172.21.15.40:3300/0 conn(0x7f7668121f70 msgr2=0x7f76681223a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:50:39.148 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:39.132+0000 7f7667fff700 1 --2- 172.21.15.42:0/3780356297 >> v2:172.21.15.40:3300/0 conn(0x7f7668121f70 0x7f76681223a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:50:39.148 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:39.132+0000 7f7667fff700 1 -- 172.21.15.42:0/3780356297 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f766811dbb0 con 0x7f766811d780 2019-12-05T05:50:39.149 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:39.132+0000 7f7667fff700 1 --2- 172.21.15.42:0/3780356297 >> v2:172.21.15.42:3300/0 conn(0x7f766811d780 0x7f766808f160 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f7650003070 tx=0x7f7650010cc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:39.149 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:39.136+0000 7f7667fff700 1 --2- 172.21.15.42:0/3780356297 >> v2:172.21.15.42:3300/0 conn(0x7f766811d780 0x7f766808f160 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f7650003070 tx=0x7f7650010cc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:39.149 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:39.136+0000 7f7667fff700 1 --2- 172.21.15.42:0/3780356297 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f766811d780 0x7f766808f160 secure :-1 s=READY pgs=1540 cs=0 l=1 rx=0x7f7650003070 tx=0x7f7650010cc0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:50:39.149 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:39.136+0000 7f7667fff700 1 --2- 172.21.15.42:0/3780356297 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f766811d780 0x7f766808f160 secure :-1 s=THROTTLE_DONE pgs=1540 cs=0 l=1 rx=0x7f7650003070 tx=0x7f7650010cc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:39.149 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:39.136+0000 7f7667fff700 1 --2- 172.21.15.42:0/3780356297 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f766811d780 0x7f766808f160 secure :-1 s=THROTTLE_DONE pgs=1540 cs=0 l=1 rx=0x7f7650003070 tx=0x7f7650010cc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:39.149 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:39.136+0000 7f76657fa700 1 -- 172.21.15.42:0/3780356297 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7650005930 con 0x7f766811d780 2019-12-05T05:50:39.150 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:39.136+0000 7f7667fff700 1 --2- 172.21.15.42:0/3780356297 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f766811d780 0x7f766808f160 secure :-1 s=THROTTLE_DONE pgs=1540 cs=0 l=1 rx=0x7f7650003070 tx=0x7f7650010cc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:39.150 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:39.136+0000 7f76657fa700 1 -- 172.21.15.42:0/3780356297 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7650007160 con 0x7f766811d780 2019-12-05T05:50:39.150 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:39.136+0000 7f766e942700 1 -- 172.21.15.42:0/3780356297 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f766808f6b0 con 0x7f766811d780 2019-12-05T05:50:39.150 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:39.136+0000 7f76657fa700 1 -- 172.21.15.42:0/3780356297 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7650012340 con 0x7f766811d780 2019-12-05T05:50:39.150 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:39.136+0000 7f766e942700 1 -- 172.21.15.42:0/3780356297 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7668122a90 con 0x7f766811d780 2019-12-05T05:50:39.150 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:39.136+0000 7f7667fff700 1 --2- 172.21.15.42:0/3780356297 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f766811d780 0x7f766808f160 secure :-1 s=THROTTLE_DONE pgs=1540 cs=0 l=1 rx=0x7f7650003070 tx=0x7f7650010cc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:39.151 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:39.136+0000 7f76657fa700 1 -- 172.21.15.42:0/3780356297 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f7650012650 con 0x7f766811d780 2019-12-05T05:50:39.151 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:39.136+0000 7f7667fff700 1 --2- 172.21.15.42:0/3780356297 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f766811d780 0x7f766808f160 secure :-1 s=THROTTLE_DONE pgs=1540 cs=0 l=1 rx=0x7f7650003070 tx=0x7f7650010cc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:39.151 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:39.136+0000 7f76657fa700 1 --2- 172.21.15.42:0/3780356297 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f764c03ae00 0x7f764c03d310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:50:39.151 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:39.136+0000 7f76657fa700 1 -- 172.21.15.42:0/3780356297 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 8697+0+0 (secure 0 0 0) 0x7f765004b4e0 con 0x7f766811d780 2019-12-05T05:50:39.151 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:39.136+0000 7f766cedf700 1 --2- 172.21.15.42:0/3780356297 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f764c03ae00 0x7f764c03d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:50:39.151 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:39.136+0000 7f766cedf700 1 --2- 172.21.15.42:0/3780356297 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f764c03ae00 0x7f764c03d310 crc :-1 s=READY pgs=664 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:50:39.151 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:39.140+0000 7f766e942700 1 -- 172.21.15.42:0/3780356297 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f766805d110 con 0x7f766811d780 2019-12-05T05:50:39.153 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:39.148+0000 7f7667fff700 1 --2- 172.21.15.42:0/3780356297 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f766811d780 0x7f766808f160 secure :-1 s=THROTTLE_DONE pgs=1540 cs=0 l=1 rx=0x7f7650003070 tx=0x7f7650010cc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:39.154 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:39.148+0000 7f76657fa700 1 -- 172.21.15.42:0/3780356297 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f7650032560 con 0x7f766811d780 2019-12-05T05:50:39.327 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:39.324+0000 7f766e942700 1 -- 172.21.15.42:0/3780356297 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f766805b8c0 con 0x7f766811d780 2019-12-05T05:50:39.328 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:39.324+0000 7f7667fff700 1 --2- 172.21.15.42:0/3780356297 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f766811d780 0x7f766808f160 secure :-1 s=THROTTLE_DONE pgs=1540 cs=0 l=1 rx=0x7f7650003070 tx=0x7f7650010cc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:39.328 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:39.324+0000 7f76657fa700 1 -- 172.21.15.42:0/3780356297 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 188 v188) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f76500327c0 con 0x7f766811d780 2019-12-05T05:50:39.329 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 188 2019-12-05T05:50:39.329 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:50:39.329 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":188,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6245,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/2081111170","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2081111170},{"type":"v1","addr":"172.21.15.40:6833","nonce":2081111170}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":188}],"filesystems":[{"mdsmap":{"epoch":187,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:49:23.679613+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8188},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8188":{"gid":8188,"name":"a","rank":0,"incarnation":184,"state":"up:active","state_seq":33,"addr":"172.21.15.42:6835/2456922143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2456922143},{"type":"v1","addr":"172.21.15.42:6835","nonce":2456922143}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:50:39.334 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:39.328+0000 7f766e942700 1 -- 172.21.15.42:0/3780356297 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f764c03ae00 msgr2=0x7f764c03d310 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:50:39.334 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:39.328+0000 7f766e942700 1 --2- 172.21.15.42:0/3780356297 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f764c03ae00 0x7f764c03d310 crc :-1 s=READY pgs=664 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:50:39.334 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:39.328+0000 7f766e942700 1 -- 172.21.15.42:0/3780356297 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f766811d780 msgr2=0x7f766808f160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:50:39.334 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:39.328+0000 7f766e942700 1 --2- 172.21.15.42:0/3780356297 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f766811d780 0x7f766808f160 secure :-1 s=READY pgs=1540 cs=0 l=1 rx=0x7f7650003070 tx=0x7f7650010cc0).stop 2019-12-05T05:50:39.334 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:39.328+0000 7f766e942700 1 -- 172.21.15.42:0/3780356297 shutdown_connections 2019-12-05T05:50:39.334 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:39.328+0000 7f766e942700 1 --2- 172.21.15.42:0/3780356297 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f764c03ae00 0x7f764c03d310 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:50:39.335 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:39.328+0000 7f766e942700 1 --2- 172.21.15.42:0/3780356297 >> v2:172.21.15.40:3300/0 conn(0x7f7668121f70 0x7f76681223a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:50:39.336 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:39.328+0000 7f766e942700 1 --2- 172.21.15.42:0/3780356297 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7668118110 0x7f76681161c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:50:39.336 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:39.328+0000 7f766e942700 1 --2- 172.21.15.42:0/3780356297 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f766811d780 0x7f766808f160 unknown :-1 s=CLOSED pgs=1540 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:50:39.336 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:39.328+0000 7f766e942700 1 -- 172.21.15.42:0/3780356297 shutdown_connections 2019-12-05T05:50:39.336 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:39.328+0000 7f766e942700 1 -- 172.21.15.42:0/3780356297 wait complete. 2019-12-05T05:50:39.336 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:39.328+0000 7f766e942700 1 -- 172.21.15.42:0/3780356297 >> 172.21.15.42:0/3780356297 conn(0x7f7668141360 msgr2=0x7f76680a9640 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:50:39.348 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T05:50:41.349 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:50:41.453 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:41.444+0000 7f23dddbe700 1 Processor -- start 2019-12-05T05:50:41.453 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:41.444+0000 7f23dddbe700 1 -- start start 2019-12-05T05:50:41.453 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:41.444+0000 7f23dddbe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f23d808df80 0x7f23d809f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:50:41.454 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:41.444+0000 7f23dddbe700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f23d809f830 0x7f23d814efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:50:41.454 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:41.444+0000 7f23dddbe700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23d809fc60 con 0x7f23d80a7740 2019-12-05T05:50:41.454 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:41.444+0000 7f23dddbe700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23d80a0b00 con 0x7f23d809f830 2019-12-05T05:50:41.454 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:41.444+0000 7f23dddbe700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23d808e5b0 con 0x7f23d808df80 2019-12-05T05:50:41.454 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:41.444+0000 7f23d77fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f23d808df80 0x7f23d809f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:50:41.454 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:41.444+0000 7f23d7fff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f23d80a7740 0x7f23d807e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:42312/0 (socket says 172.21.15.42:42312) 2019-12-05T05:50:41.454 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:41.444+0000 7f23d7fff700 1 -- 172.21.15.42:0/255727092 learned_addr learned my addr 172.21.15.42:0/255727092 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:50:41.455 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:41.444+0000 7f23dcb5c700 1 --2- 172.21.15.42:0/255727092 >> v2:172.21.15.40:3300/0 conn(0x7f23d809f830 0x7f23d814efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:50:41.455 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:41.444+0000 7f23d77fe700 1 -- 172.21.15.42:0/255727092 >> v2:172.21.15.40:3300/0 conn(0x7f23d809f830 msgr2=0x7f23d814efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:50:41.455 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:41.444+0000 7f23d6ffd700 1 -- 172.21.15.42:0/255727092 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4290614816 0 0) 0x7f23d809fc60 con 0x7f23d80a7740 2019-12-05T05:50:41.455 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:41.444+0000 7f23d77fe700 1 --2- 172.21.15.42:0/255727092 >> v2:172.21.15.40:3300/0 conn(0x7f23d809f830 0x7f23d814efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:50:41.455 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:41.444+0000 7f23d77fe700 1 -- 172.21.15.42:0/255727092 >> v1:172.21.15.42:6789/0 conn(0x7f23d80a7740 legacy=0x7f23d807e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:50:41.455 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:41.444+0000 7f23d77fe700 1 -- 172.21.15.42:0/255727092 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23d80a0710 con 0x7f23d808df80 2019-12-05T05:50:41.455 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:41.444+0000 7f23d77fe700 1 --2- 172.21.15.42:0/255727092 >> v2:172.21.15.42:3300/0 conn(0x7f23d808df80 0x7f23d809f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f23c80032b0 tx=0x7f23c800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:41.456 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:41.444+0000 7f23d77fe700 1 --2- 172.21.15.42:0/255727092 >> v2:172.21.15.42:3300/0 conn(0x7f23d808df80 0x7f23d809f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f23c80032b0 tx=0x7f23c800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:41.456 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:41.444+0000 7f23d77fe700 1 --2- 172.21.15.42:0/255727092 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f23d808df80 0x7f23d809f2e0 secure :-1 s=READY pgs=1542 cs=0 l=1 rx=0x7f23c80032b0 tx=0x7f23c800fff0).ready entity=mon.1 client_cookie=a3b7107c005cecf7 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:50:41.456 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:41.444+0000 7f23d77fe700 1 --2- 172.21.15.42:0/255727092 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f23d808df80 0x7f23d809f2e0 secure :-1 s=THROTTLE_DONE pgs=1542 cs=0 l=1 rx=0x7f23c80032b0 tx=0x7f23c800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:41.456 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:41.444+0000 7f23d77fe700 1 --2- 172.21.15.42:0/255727092 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f23d808df80 0x7f23d809f2e0 secure :-1 s=THROTTLE_DONE pgs=1542 cs=0 l=1 rx=0x7f23c80032b0 tx=0x7f23c800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:41.456 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:41.444+0000 7f23d6ffd700 1 -- 172.21.15.42:0/255727092 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f23c80108c0 con 0x7f23d808df80 2019-12-05T05:50:41.456 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:41.444+0000 7f23d77fe700 1 --2- 172.21.15.42:0/255727092 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f23d808df80 0x7f23d809f2e0 secure :-1 s=THROTTLE_DONE pgs=1542 cs=0 l=1 rx=0x7f23c80032b0 tx=0x7f23c800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:41.457 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:41.444+0000 7f23d6ffd700 1 -- 172.21.15.42:0/255727092 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23c8008b50 con 0x7f23d808df80 2019-12-05T05:50:41.457 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:41.444+0000 7f23d6ffd700 1 -- 172.21.15.42:0/255727092 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f23c80110c0 con 0x7f23d808df80 2019-12-05T05:50:41.457 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:41.444+0000 7f23dddbe700 1 -- 172.21.15.42:0/255727092 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f23d808df80 msgr2=0x7f23d809f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:50:41.457 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:41.444+0000 7f23dddbe700 1 --2- 172.21.15.42:0/255727092 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f23d808df80 0x7f23d809f2e0 secure :-1 s=READY pgs=1542 cs=0 l=1 rx=0x7f23c80032b0 tx=0x7f23c800fff0).stop 2019-12-05T05:50:41.457 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:41.448+0000 7f23dddbe700 1 -- 172.21.15.42:0/255727092 shutdown_connections 2019-12-05T05:50:41.457 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:41.448+0000 7f23dddbe700 1 --2- 172.21.15.42:0/255727092 >> v2:172.21.15.40:3300/0 conn(0x7f23d809f830 0x7f23d814efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:50:41.457 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:41.448+0000 7f23dddbe700 1 --2- 172.21.15.42:0/255727092 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f23d808df80 0x7f23d809f2e0 unknown :-1 s=CLOSED pgs=1542 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:50:41.458 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:41.448+0000 7f23dddbe700 1 -- 172.21.15.42:0/255727092 shutdown_connections 2019-12-05T05:50:41.458 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:41.448+0000 7f23dddbe700 1 -- 172.21.15.42:0/255727092 wait complete. 2019-12-05T05:50:41.458 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:41.448+0000 7f23dddbe700 1 -- 172.21.15.42:0/255727092 >> 172.21.15.42:0/255727092 conn(0x7f23d80aabe0 msgr2=0x7f23d80a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:50:41.463 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:41.448+0000 7f23dddbe700 1 Processor -- start 2019-12-05T05:50:41.463 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:41.452+0000 7f23dddbe700 1 -- start start 2019-12-05T05:50:41.463 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:41.452+0000 7f23dddbe700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f23d8132160 0x7f23d811ab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:50:41.463 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:41.452+0000 7f23dddbe700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23d80a9ba0 con 0x7f23d81180c0 2019-12-05T05:50:41.464 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:41.452+0000 7f23dddbe700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23d8141e60 con 0x7f23d80a9770 2019-12-05T05:50:41.464 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:41.452+0000 7f23dddbe700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23d811b060 con 0x7f23d8132160 2019-12-05T05:50:41.464 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:41.452+0000 7f23dcb5c700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f23d80a9770 0x7f23d80742d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:42322/0 (socket says 172.21.15.42:42322) 2019-12-05T05:50:41.465 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:41.452+0000 7f23dcb5c700 1 -- 172.21.15.42:0/3310093735 learned_addr learned my addr 172.21.15.42:0/3310093735 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:50:41.465 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:41.452+0000 7f23d77fe700 1 --2- 172.21.15.42:0/3310093735 >> v2:172.21.15.40:3300/0 conn(0x7f23d8132160 0x7f23d811ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:50:41.465 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:41.452+0000 7f23d4ff9700 1 -- 172.21.15.42:0/3310093735 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3863881950 0 0) 0x7f23d8141e60 con 0x7f23d80a9770 2019-12-05T05:50:41.465 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:41.452+0000 7f23d4ff9700 1 -- 172.21.15.42:0/3310093735 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23bc0030b0 con 0x7f23d80a9770 2019-12-05T05:50:41.465 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:41.452+0000 7f23d77fe700 1 -- 172.21.15.42:0/3310093735 >> v1:172.21.15.42:6789/0 conn(0x7f23d80a9770 legacy=0x7f23d80742d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:50:41.465 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:41.452+0000 7f23d4ff9700 1 -- 172.21.15.42:0/3310093735 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2134601227 0 0) 0x7f23d80a9ba0 con 0x7f23d81180c0 2019-12-05T05:50:41.466 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:41.452+0000 7f23d77fe700 1 -- 172.21.15.42:0/3310093735 >> v1:172.21.15.40:6789/0 conn(0x7f23d81180c0 legacy=0x7f23d8131960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:50:41.466 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:41.452+0000 7f23d77fe700 1 -- 172.21.15.42:0/3310093735 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23d8077b00 con 0x7f23d8132160 2019-12-05T05:50:41.466 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:41.452+0000 7f23d77fe700 1 --2- 172.21.15.42:0/3310093735 >> v2:172.21.15.40:3300/0 conn(0x7f23d8132160 0x7f23d811ab10 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f23c80032b0 tx=0x7f23c8008a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:41.466 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:41.452+0000 7f23d77fe700 1 --2- 172.21.15.42:0/3310093735 >> v2:172.21.15.40:3300/0 conn(0x7f23d8132160 0x7f23d811ab10 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f23c80032b0 tx=0x7f23c8008a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:41.466 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:41.452+0000 7f23d77fe700 1 --2- 172.21.15.42:0/3310093735 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f23d8132160 0x7f23d811ab10 secure :-1 s=READY pgs=1185 cs=0 l=1 rx=0x7f23c80032b0 tx=0x7f23c8008a20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:50:41.466 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:41.452+0000 7f23d77fe700 1 --2- 172.21.15.42:0/3310093735 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f23d8132160 0x7f23d811ab10 secure :-1 s=THROTTLE_DONE pgs=1185 cs=0 l=1 rx=0x7f23c80032b0 tx=0x7f23c8008a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:41.466 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:41.452+0000 7f23d4ff9700 1 -- 172.21.15.42:0/3310093735 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f23c8005ab0 con 0x7f23d8132160 2019-12-05T05:50:41.467 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:41.452+0000 7f23d77fe700 1 --2- 172.21.15.42:0/3310093735 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f23d8132160 0x7f23d811ab10 secure :-1 s=THROTTLE_DONE pgs=1185 cs=0 l=1 rx=0x7f23c80032b0 tx=0x7f23c8008a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:41.467 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:41.452+0000 7f23d77fe700 1 --2- 172.21.15.42:0/3310093735 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f23d8132160 0x7f23d811ab10 secure :-1 s=THROTTLE_DONE pgs=1185 cs=0 l=1 rx=0x7f23c80032b0 tx=0x7f23c8008a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:41.467 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:41.452+0000 7f23d4ff9700 1 -- 172.21.15.42:0/3310093735 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23c8010ac0 con 0x7f23d8132160 2019-12-05T05:50:41.467 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:41.452+0000 7f23dddbe700 1 -- 172.21.15.42:0/3310093735 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23d8075ae0 con 0x7f23d8132160 2019-12-05T05:50:41.467 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:41.452+0000 7f23d4ff9700 1 -- 172.21.15.42:0/3310093735 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f23c800f680 con 0x7f23d8132160 2019-12-05T05:50:41.467 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:41.456+0000 7f23dddbe700 1 -- 172.21.15.42:0/3310093735 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23d8076060 con 0x7f23d8132160 2019-12-05T05:50:41.467 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:41.456+0000 7f23d77fe700 1 --2- 172.21.15.42:0/3310093735 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f23d8132160 0x7f23d811ab10 secure :-1 s=THROTTLE_DONE pgs=1185 cs=0 l=1 rx=0x7f23c80032b0 tx=0x7f23c8008a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:41.468 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:41.456+0000 7f23d77fe700 1 --2- 172.21.15.42:0/3310093735 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f23d8132160 0x7f23d811ab10 secure :-1 s=THROTTLE_DONE pgs=1185 cs=0 l=1 rx=0x7f23c80032b0 tx=0x7f23c8008a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:41.468 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:41.456+0000 7f23d4ff9700 1 -- 172.21.15.42:0/3310093735 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f23c80113d0 con 0x7f23d8132160 2019-12-05T05:50:41.468 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:41.456+0000 7f23d4ff9700 1 --2- 172.21.15.42:0/3310093735 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f23bc03b1e0 0x7f23bc03d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:50:41.468 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:41.456+0000 7f23d4ff9700 1 -- 172.21.15.42:0/3310093735 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 8697+0+0 (secure 0 0 0) 0x7f23c804b0e0 con 0x7f23d8132160 2019-12-05T05:50:41.468 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:41.456+0000 7f23d7fff700 1 --2- 172.21.15.42:0/3310093735 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f23bc03b1e0 0x7f23bc03d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:50:41.468 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:41.456+0000 7f23d7fff700 1 --2- 172.21.15.42:0/3310093735 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f23bc03b1e0 0x7f23bc03d6f0 crc :-1 s=READY pgs=665 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:50:41.469 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:41.460+0000 7f23dddbe700 1 -- 172.21.15.42:0/3310093735 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23d805d110 con 0x7f23d8132160 2019-12-05T05:50:41.474 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:41.468+0000 7f23d77fe700 1 --2- 172.21.15.42:0/3310093735 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f23d8132160 0x7f23d811ab10 secure :-1 s=THROTTLE_DONE pgs=1185 cs=0 l=1 rx=0x7f23c80032b0 tx=0x7f23c8008a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:41.474 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:41.468+0000 7f23d4ff9700 1 -- 172.21.15.42:0/3310093735 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f23c8011cd0 con 0x7f23d8132160 2019-12-05T05:50:41.642 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:41.636+0000 7f23dddbe700 1 -- 172.21.15.42:0/3310093735 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f23d805b8c0 con 0x7f23d8132160 2019-12-05T05:50:41.643 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:41.640+0000 7f23d77fe700 1 --2- 172.21.15.42:0/3310093735 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f23d8132160 0x7f23d811ab10 secure :-1 s=THROTTLE_DONE pgs=1185 cs=0 l=1 rx=0x7f23c80032b0 tx=0x7f23c8008a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:41.643 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:41.640+0000 7f23d4ff9700 1 -- 172.21.15.42:0/3310093735 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 188 v188) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f23c802fd80 con 0x7f23d8132160 2019-12-05T05:50:41.645 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 188 2019-12-05T05:50:41.646 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:50:41.646 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":188,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6245,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/2081111170","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2081111170},{"type":"v1","addr":"172.21.15.40:6833","nonce":2081111170}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":188}],"filesystems":[{"mdsmap":{"epoch":187,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:49:23.679613+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8188},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8188":{"gid":8188,"name":"a","rank":0,"incarnation":184,"state":"up:active","state_seq":33,"addr":"172.21.15.42:6835/2456922143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2456922143},{"type":"v1","addr":"172.21.15.42:6835","nonce":2456922143}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:50:41.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:41.644+0000 7f23dddbe700 1 -- 172.21.15.42:0/3310093735 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f23bc03b1e0 msgr2=0x7f23bc03d6f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:50:41.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:41.644+0000 7f23dddbe700 1 --2- 172.21.15.42:0/3310093735 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f23bc03b1e0 0x7f23bc03d6f0 crc :-1 s=READY pgs=665 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:50:41.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:41.644+0000 7f23dddbe700 1 -- 172.21.15.42:0/3310093735 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f23d8132160 msgr2=0x7f23d811ab10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:50:41.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:41.644+0000 7f23dddbe700 1 --2- 172.21.15.42:0/3310093735 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f23d8132160 0x7f23d811ab10 secure :-1 s=READY pgs=1185 cs=0 l=1 rx=0x7f23c80032b0 tx=0x7f23c8008a20).stop 2019-12-05T05:50:41.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:41.644+0000 7f23dddbe700 1 -- 172.21.15.42:0/3310093735 shutdown_connections 2019-12-05T05:50:41.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:41.644+0000 7f23dddbe700 1 --2- 172.21.15.42:0/3310093735 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f23bc03b1e0 0x7f23bc03d6f0 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:50:41.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:41.644+0000 7f23dddbe700 1 --2- 172.21.15.42:0/3310093735 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f23d8132160 0x7f23d811ab10 unknown :-1 s=CLOSED pgs=1185 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:50:41.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:41.644+0000 7f23dddbe700 1 -- 172.21.15.42:0/3310093735 shutdown_connections 2019-12-05T05:50:41.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:41.644+0000 7f23dddbe700 1 -- 172.21.15.42:0/3310093735 wait complete. 2019-12-05T05:50:41.650 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:41.644+0000 7f23dddbe700 1 -- 172.21.15.42:0/3310093735 >> 172.21.15.42:0/3310093735 conn(0x7f23d80aabe0 msgr2=0x7f23d811d080 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:50:41.659 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T05:50:41.940 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:50:41.945 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:50:43.659 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:50:43.766 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:43.756+0000 7f7eceaa2700 1 Processor -- start 2019-12-05T05:50:43.766 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:43.756+0000 7f7eceaa2700 1 -- start start 2019-12-05T05:50:43.767 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:43.756+0000 7f7eceaa2700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7ec80a7740 0x7f7ec808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:50:43.767 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:43.756+0000 7f7eceaa2700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7ec809fae0 0x7f7ec807eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:50:43.767 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:43.756+0000 7f7eceaa2700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ec80a0710 con 0x7f7ec807e720 2019-12-05T05:50:43.767 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:43.756+0000 7f7eceaa2700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ec80a0b00 con 0x7f7ec80a7740 2019-12-05T05:50:43.767 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:43.756+0000 7f7eceaa2700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ec808e5b0 con 0x7f7ec809fae0 2019-12-05T05:50:43.767 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:43.756+0000 7f7ecc83e700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f7ec807e720 0x7f7ec809f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:42326/0 (socket says 172.21.15.42:42326) 2019-12-05T05:50:43.767 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:43.756+0000 7f7ecc83e700 1 -- 172.21.15.42:0/3113922415 learned_addr learned my addr 172.21.15.42:0/3113922415 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:50:43.768 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:43.756+0000 7f7ecd840700 1 --2- 172.21.15.42:0/3113922415 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7ec809fae0 0x7f7ec807eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:50:43.768 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:43.756+0000 7f7ecd03f700 1 --2- 172.21.15.42:0/3113922415 >> v2:172.21.15.40:3300/0 conn(0x7f7ec80a7740 0x7f7ec808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:50:43.768 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:43.756+0000 7f7ebbfff700 1 -- 172.21.15.42:0/3113922415 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1589012007 0 0) 0x7f7ec80a0710 con 0x7f7ec807e720 2019-12-05T05:50:43.768 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:43.756+0000 7f7ebbfff700 1 -- 172.21.15.42:0/3113922415 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7eb40030b0 con 0x7f7ec807e720 2019-12-05T05:50:43.768 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:43.756+0000 7f7ecd03f700 1 -- 172.21.15.42:0/3113922415 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7ec809fae0 msgr2=0x7f7ec807eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:50:43.768 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:43.756+0000 7f7ecd03f700 1 --2- 172.21.15.42:0/3113922415 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7ec809fae0 0x7f7ec807eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:50:43.768 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:43.756+0000 7f7ecd03f700 1 -- 172.21.15.42:0/3113922415 >> v1:172.21.15.42:6789/0 conn(0x7f7ec807e720 legacy=0x7f7ec809f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:50:43.769 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:43.756+0000 7f7ecd03f700 1 -- 172.21.15.42:0/3113922415 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ec807f0a0 con 0x7f7ec80a7740 2019-12-05T05:50:43.769 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:43.756+0000 7f7ecd840700 1 --2- 172.21.15.42:0/3113922415 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7ec809fae0 0x7f7ec807eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:50:43.769 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:43.756+0000 7f7ecd03f700 1 --2- 172.21.15.42:0/3113922415 >> v2:172.21.15.40:3300/0 conn(0x7f7ec80a7740 0x7f7ec808df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f7ec4008790 tx=0x7f7ec400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:43.769 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:43.760+0000 7f7ecd03f700 1 --2- 172.21.15.42:0/3113922415 >> v2:172.21.15.40:3300/0 conn(0x7f7ec80a7740 0x7f7ec808df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f7ec4008790 tx=0x7f7ec400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:43.769 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:43.760+0000 7f7ecd03f700 1 --2- 172.21.15.42:0/3113922415 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7ec80a7740 0x7f7ec808df80 secure :-1 s=READY pgs=1186 cs=0 l=1 rx=0x7f7ec4008790 tx=0x7f7ec400fc70).ready entity=mon.0 client_cookie=85a04ac32699b9fa server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:50:43.769 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:43.760+0000 7f7ecd03f700 1 --2- 172.21.15.42:0/3113922415 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7ec80a7740 0x7f7ec808df80 secure :-1 s=THROTTLE_DONE pgs=1186 cs=0 l=1 rx=0x7f7ec4008790 tx=0x7f7ec400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:43.771 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:43.760+0000 7f7ebbfff700 1 -- 172.21.15.42:0/3113922415 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7ec4008bb0 con 0x7f7ec80a7740 2019-12-05T05:50:43.771 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:43.760+0000 7f7ecd03f700 1 --2- 172.21.15.42:0/3113922415 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7ec80a7740 0x7f7ec808df80 secure :-1 s=THROTTLE_DONE pgs=1186 cs=0 l=1 rx=0x7f7ec4008790 tx=0x7f7ec400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:43.771 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:43.760+0000 7f7ecd03f700 1 --2- 172.21.15.42:0/3113922415 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7ec80a7740 0x7f7ec808df80 secure :-1 s=THROTTLE_DONE pgs=1186 cs=0 l=1 rx=0x7f7ec4008790 tx=0x7f7ec400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:43.771 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:43.760+0000 7f7ebbfff700 1 -- 172.21.15.42:0/3113922415 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ec40087d0 con 0x7f7ec80a7740 2019-12-05T05:50:43.772 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:43.760+0000 7f7ebbfff700 1 -- 172.21.15.42:0/3113922415 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7ec4011860 con 0x7f7ec80a7740 2019-12-05T05:50:43.772 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:43.760+0000 7f7eceaa2700 1 -- 172.21.15.42:0/3113922415 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7ec80a7740 msgr2=0x7f7ec808df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:50:43.772 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:43.760+0000 7f7eceaa2700 1 --2- 172.21.15.42:0/3113922415 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7ec80a7740 0x7f7ec808df80 secure :-1 s=READY pgs=1186 cs=0 l=1 rx=0x7f7ec4008790 tx=0x7f7ec400fc70).stop 2019-12-05T05:50:43.772 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:43.760+0000 7f7eceaa2700 1 -- 172.21.15.42:0/3113922415 shutdown_connections 2019-12-05T05:50:43.772 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:43.760+0000 7f7eceaa2700 1 --2- 172.21.15.42:0/3113922415 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7ec809fae0 0x7f7ec807eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:50:43.772 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:43.760+0000 7f7eceaa2700 1 --2- 172.21.15.42:0/3113922415 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7ec80a7740 0x7f7ec808df80 unknown :-1 s=CLOSED pgs=1186 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:50:43.772 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:43.760+0000 7f7eceaa2700 1 -- 172.21.15.42:0/3113922415 shutdown_connections 2019-12-05T05:50:43.773 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:43.760+0000 7f7eceaa2700 1 -- 172.21.15.42:0/3113922415 wait complete. 2019-12-05T05:50:43.773 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:43.760+0000 7f7eceaa2700 1 -- 172.21.15.42:0/3113922415 >> 172.21.15.42:0/3113922415 conn(0x7f7ec80aabe0 msgr2=0x7f7ec80a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:50:43.778 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:43.764+0000 7f7eceaa2700 1 Processor -- start 2019-12-05T05:50:43.778 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:43.764+0000 7f7eceaa2700 1 -- start start 2019-12-05T05:50:43.778 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:43.764+0000 7f7eceaa2700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7ec8116910 0x7f7ec811bba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:50:43.778 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:43.764+0000 7f7eceaa2700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f7ec8122410 0x7f7ec811a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:50:43.778 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:43.764+0000 7f7eceaa2700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7ec80a6e90 0x7f7ec811a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:50:43.778 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:43.764+0000 7f7eceaa2700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ec81161c0 con 0x7f7ec8116910 2019-12-05T05:50:43.779 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:43.764+0000 7f7eceaa2700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ec8116e20 con 0x7f7ec8122410 2019-12-05T05:50:43.779 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:43.764+0000 7f7eceaa2700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ec81170c0 con 0x7f7ec80a6e90 2019-12-05T05:50:43.779 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:43.764+0000 7f7ecc83e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f7ec8122410 0x7f7ec811a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:50:43.779 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:43.764+0000 7f7ecc83e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f7ec8122410 0x7f7ec811a1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:42294/0 (socket says 172.21.15.42:42294) 2019-12-05T05:50:43.779 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:43.764+0000 7f7ecc83e700 1 -- 172.21.15.42:0/207530695 learned_addr learned my addr 172.21.15.42:0/207530695 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:50:43.779 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:43.764+0000 7f7ecd03f700 1 --2- 172.21.15.42:0/207530695 >> v2:172.21.15.40:3300/0 conn(0x7f7ec8116910 0x7f7ec811bba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:50:43.780 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:43.764+0000 7f7ecd840700 1 --2- 172.21.15.42:0/207530695 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7ec80a6e90 0x7f7ec811a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:50:43.780 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:43.764+0000 7f7ecc83e700 1 -- 172.21.15.42:0/207530695 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7ec80a6e90 msgr2=0x7f7ec811a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:50:43.780 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:43.764+0000 7f7ecc83e700 1 --2- 172.21.15.42:0/207530695 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7ec80a6e90 0x7f7ec811a710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:50:43.780 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:43.764+0000 7f7ecc83e700 1 -- 172.21.15.42:0/207530695 >> v2:172.21.15.40:3300/0 conn(0x7f7ec8116910 msgr2=0x7f7ec811bba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:50:43.780 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:43.764+0000 7f7ecc83e700 1 --2- 172.21.15.42:0/207530695 >> v2:172.21.15.40:3300/0 conn(0x7f7ec8116910 0x7f7ec811bba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:50:43.780 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:43.764+0000 7f7ecc83e700 1 -- 172.21.15.42:0/207530695 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ec811ef30 con 0x7f7ec8122410 2019-12-05T05:50:43.781 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:43.764+0000 7f7ecd840700 1 --2- 172.21.15.42:0/207530695 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7ec80a6e90 0x7f7ec811a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T05:50:43.781 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:43.764+0000 7f7ecc83e700 1 --2- 172.21.15.42:0/207530695 >> v2:172.21.15.42:3300/0 conn(0x7f7ec8122410 0x7f7ec811a1c0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f7ec8116fe0 tx=0x7f7ebc010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:43.781 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:43.764+0000 7f7ecc83e700 1 --2- 172.21.15.42:0/207530695 >> v2:172.21.15.42:3300/0 conn(0x7f7ec8122410 0x7f7ec811a1c0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f7ec8116fe0 tx=0x7f7ebc010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:43.781 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:43.764+0000 7f7ecc83e700 1 --2- 172.21.15.42:0/207530695 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7ec8122410 0x7f7ec811a1c0 secure :-1 s=READY pgs=1545 cs=0 l=1 rx=0x7f7ec8116fe0 tx=0x7f7ebc010ab0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:50:43.781 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:43.764+0000 7f7ecc83e700 1 --2- 172.21.15.42:0/207530695 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7ec8122410 0x7f7ec811a1c0 secure :-1 s=THROTTLE_DONE pgs=1545 cs=0 l=1 rx=0x7f7ec8116fe0 tx=0x7f7ebc010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:43.782 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:43.764+0000 7f7eb9ffb700 1 -- 172.21.15.42:0/207530695 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7ebc00c6c0 con 0x7f7ec8122410 2019-12-05T05:50:43.782 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:43.764+0000 7f7ecc83e700 1 --2- 172.21.15.42:0/207530695 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7ec8122410 0x7f7ec811a1c0 secure :-1 s=THROTTLE_DONE pgs=1545 cs=0 l=1 rx=0x7f7ec8116fe0 tx=0x7f7ebc010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:43.782 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:43.764+0000 7f7eb9ffb700 1 -- 172.21.15.42:0/207530695 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ebc010170 con 0x7f7ec8122410 2019-12-05T05:50:43.782 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:43.764+0000 7f7ecc83e700 1 --2- 172.21.15.42:0/207530695 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7ec8122410 0x7f7ec811a1c0 secure :-1 s=THROTTLE_DONE pgs=1545 cs=0 l=1 rx=0x7f7ec8116fe0 tx=0x7f7ebc010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:43.782 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:43.764+0000 7f7eceaa2700 1 -- 172.21.15.42:0/207530695 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ec811c0f0 con 0x7f7ec8122410 2019-12-05T05:50:43.782 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:43.764+0000 7f7eb9ffb700 1 -- 172.21.15.42:0/207530695 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7ebc00cca0 con 0x7f7ec8122410 2019-12-05T05:50:43.783 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:43.764+0000 7f7eceaa2700 1 -- 172.21.15.42:0/207530695 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ec811ac60 con 0x7f7ec8122410 2019-12-05T05:50:43.783 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:43.764+0000 7f7ecc83e700 1 --2- 172.21.15.42:0/207530695 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7ec8122410 0x7f7ec811a1c0 secure :-1 s=THROTTLE_DONE pgs=1545 cs=0 l=1 rx=0x7f7ec8116fe0 tx=0x7f7ebc010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:43.783 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:43.768+0000 7f7eb9ffb700 1 -- 172.21.15.42:0/207530695 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f7ebc012430 con 0x7f7ec8122410 2019-12-05T05:50:43.783 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:43.768+0000 7f7ecc83e700 1 --2- 172.21.15.42:0/207530695 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7ec8122410 0x7f7ec811a1c0 secure :-1 s=THROTTLE_DONE pgs=1545 cs=0 l=1 rx=0x7f7ec8116fe0 tx=0x7f7ebc010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:43.783 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:43.768+0000 7f7eb9ffb700 1 --2- 172.21.15.42:0/207530695 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f7ea803aea0 0x7f7ea803d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:50:43.783 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:43.768+0000 7f7eb9ffb700 1 -- 172.21.15.42:0/207530695 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 8697+0+0 (secure 0 0 0) 0x7f7ebc04ac60 con 0x7f7ec8122410 2019-12-05T05:50:43.783 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:43.768+0000 7f7ecd03f700 1 --2- 172.21.15.42:0/207530695 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f7ea803aea0 0x7f7ea803d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:50:43.784 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:43.768+0000 7f7ecd03f700 1 --2- 172.21.15.42:0/207530695 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f7ea803aea0 0x7f7ea803d3b0 crc :-1 s=READY pgs=666 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:50:43.784 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:43.772+0000 7f7eceaa2700 1 -- 172.21.15.42:0/207530695 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ec805d110 con 0x7f7ec8122410 2019-12-05T05:50:43.785 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:43.780+0000 7f7ecc83e700 1 --2- 172.21.15.42:0/207530695 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7ec8122410 0x7f7ec811a1c0 secure :-1 s=THROTTLE_DONE pgs=1545 cs=0 l=1 rx=0x7f7ec8116fe0 tx=0x7f7ebc010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:43.785 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:43.780+0000 7f7eb9ffb700 1 -- 172.21.15.42:0/207530695 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f7ebc031c60 con 0x7f7ec8122410 2019-12-05T05:50:43.955 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:43.948+0000 7f7eceaa2700 1 -- 172.21.15.42:0/207530695 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7ec811b980 con 0x7f7ec8122410 2019-12-05T05:50:43.955 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:43.952+0000 7f7ecc83e700 1 --2- 172.21.15.42:0/207530695 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7ec8122410 0x7f7ec811a1c0 secure :-1 s=THROTTLE_DONE pgs=1545 cs=0 l=1 rx=0x7f7ec8116fe0 tx=0x7f7ebc010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:43.956 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:43.952+0000 7f7eb9ffb700 1 -- 172.21.15.42:0/207530695 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 188 v188) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f7ebc031ec0 con 0x7f7ec8122410 2019-12-05T05:50:43.958 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 188 2019-12-05T05:50:43.958 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:50:43.958 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":188,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6245,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/2081111170","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2081111170},{"type":"v1","addr":"172.21.15.40:6833","nonce":2081111170}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":188}],"filesystems":[{"mdsmap":{"epoch":187,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:49:23.679613+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8188},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8188":{"gid":8188,"name":"a","rank":0,"incarnation":184,"state":"up:active","state_seq":33,"addr":"172.21.15.42:6835/2456922143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2456922143},{"type":"v1","addr":"172.21.15.42:6835","nonce":2456922143}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:50:43.963 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:43.956+0000 7f7eceaa2700 1 -- 172.21.15.42:0/207530695 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f7ea803aea0 msgr2=0x7f7ea803d3b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:50:43.964 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:43.956+0000 7f7eceaa2700 1 --2- 172.21.15.42:0/207530695 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f7ea803aea0 0x7f7ea803d3b0 crc :-1 s=READY pgs=666 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:50:43.964 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:43.956+0000 7f7eceaa2700 1 -- 172.21.15.42:0/207530695 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7ec8122410 msgr2=0x7f7ec811a1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:50:43.964 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:43.956+0000 7f7eceaa2700 1 --2- 172.21.15.42:0/207530695 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7ec8122410 0x7f7ec811a1c0 secure :-1 s=READY pgs=1545 cs=0 l=1 rx=0x7f7ec8116fe0 tx=0x7f7ebc010ab0).stop 2019-12-05T05:50:43.964 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:43.956+0000 7f7eceaa2700 1 -- 172.21.15.42:0/207530695 shutdown_connections 2019-12-05T05:50:43.964 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:43.956+0000 7f7eceaa2700 1 --2- 172.21.15.42:0/207530695 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f7ea803aea0 0x7f7ea803d3b0 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:50:43.964 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:43.956+0000 7f7eceaa2700 1 --2- 172.21.15.42:0/207530695 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7ec80a6e90 0x7f7ec811a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:50:43.965 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:43.956+0000 7f7eceaa2700 1 --2- 172.21.15.42:0/207530695 >> v2:172.21.15.40:3300/0 conn(0x7f7ec8116910 0x7f7ec811bba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:50:43.965 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:43.956+0000 7f7eceaa2700 1 --2- 172.21.15.42:0/207530695 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7ec8122410 0x7f7ec811a1c0 unknown :-1 s=CLOSED pgs=1545 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:50:43.965 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:43.956+0000 7f7eceaa2700 1 -- 172.21.15.42:0/207530695 shutdown_connections 2019-12-05T05:50:43.965 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:43.956+0000 7f7eceaa2700 1 -- 172.21.15.42:0/207530695 wait complete. 2019-12-05T05:50:43.965 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:43.956+0000 7f7eceaa2700 1 -- 172.21.15.42:0/207530695 >> 172.21.15.42:0/207530695 conn(0x7f7ec808fda0 msgr2=0x7f7ec808eb70 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:50:43.975 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T05:50:45.976 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:50:46.085 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:46.072+0000 7fa4db200700 1 Processor -- start 2019-12-05T05:50:46.085 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:46.072+0000 7fa4db200700 1 -- start start 2019-12-05T05:50:46.085 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:46.076+0000 7fa4db200700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa4d40a7740 0x7fa4d408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:50:46.085 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:46.076+0000 7fa4db200700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4d40a0710 con 0x7fa4d409fae0 2019-12-05T05:50:46.086 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:46.076+0000 7fa4db200700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4d40a9ba0 con 0x7fa4d407e720 2019-12-05T05:50:46.086 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:46.076+0000 7fa4db200700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4d40a0b00 con 0x7fa4d40a7740 2019-12-05T05:50:46.086 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:46.076+0000 7fa4d8f9c700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fa4d407e720 0x7fa4d409f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:42340/0 (socket says 172.21.15.42:42340) 2019-12-05T05:50:46.086 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:46.076+0000 7fa4d8f9c700 1 -- 172.21.15.42:0/2641528641 learned_addr learned my addr 172.21.15.42:0/2641528641 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:50:46.086 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:46.076+0000 7fa4d979d700 1 --2- 172.21.15.42:0/2641528641 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa4d40a7740 0x7fa4d408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:50:46.086 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:46.076+0000 7fa4cbfff700 1 -- 172.21.15.42:0/2641528641 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1509537079 0 0) 0x7fa4d40a9ba0 con 0x7fa4d407e720 2019-12-05T05:50:46.086 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:46.076+0000 7fa4cbfff700 1 -- 172.21.15.42:0/2641528641 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4c00030b0 con 0x7fa4d407e720 2019-12-05T05:50:46.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:46.076+0000 7fa4cbfff700 1 -- 172.21.15.42:0/2641528641 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 518315107 0 0) 0x7fa4d40a0710 con 0x7fa4d409fae0 2019-12-05T05:50:46.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:46.076+0000 7fa4cbfff700 1 -- 172.21.15.42:0/2641528641 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4d40a9ba0 con 0x7fa4d409fae0 2019-12-05T05:50:46.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:46.076+0000 7fa4cbfff700 1 -- 172.21.15.42:0/2641528641 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2093178576 0 0) 0x7fa4c00030b0 con 0x7fa4d407e720 2019-12-05T05:50:46.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:46.076+0000 7fa4cbfff700 1 -- 172.21.15.42:0/2641528641 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa4d40a7740 msgr2=0x7fa4d408df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:50:46.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:46.076+0000 7fa4cbfff700 1 --2- 172.21.15.42:0/2641528641 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa4d40a7740 0x7fa4d408df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:50:46.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:46.076+0000 7fa4cbfff700 1 -- 172.21.15.42:0/2641528641 >> v1:172.21.15.40:6789/0 conn(0x7fa4d409fae0 legacy=0x7fa4d414efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:50:46.088 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:46.076+0000 7fa4cbfff700 1 -- 172.21.15.42:0/2641528641 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4d407eb50 con 0x7fa4d407e720 2019-12-05T05:50:46.088 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:46.076+0000 7fa4cbfff700 1 -- 172.21.15.42:0/2641528641 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7fa4c4001750 con 0x7fa4d407e720 2019-12-05T05:50:46.092 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:46.076+0000 7fa4d979d700 1 --2- 172.21.15.42:0/2641528641 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa4d40a7740 0x7fa4d408df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:50:46.093 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:46.076+0000 7fa4cbfff700 1 -- 172.21.15.42:0/2641528641 >> v1:172.21.15.42:6789/0 conn(0x7fa4d407e720 legacy=0x7fa4d409f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:50:46.093 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:46.076+0000 7fa4cbfff700 1 --2- 172.21.15.42:0/2641528641 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa4c0003940 0x7fa4c0003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:50:46.093 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:46.076+0000 7fa4cbfff700 1 --2- 172.21.15.42:0/2641528641 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa4c0004420 0x7fa4c0004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:50:46.093 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:46.076+0000 7fa4cbfff700 1 --2- 172.21.15.42:0/2641528641 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa4c0004e90 0x7fa4c004f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:50:46.093 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:46.076+0000 7fa4cbfff700 1 -- 172.21.15.42:0/2641528641 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4d40a0b00 con 0x7fa4c0004e90 2019-12-05T05:50:46.093 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:46.076+0000 7fa4cbfff700 1 -- 172.21.15.42:0/2641528641 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4c004fcc0 con 0x7fa4c0004420 2019-12-05T05:50:46.094 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:46.076+0000 7fa4cbfff700 1 -- 172.21.15.42:0/2641528641 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4c004ff00 con 0x7fa4c0003940 2019-12-05T05:50:46.094 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:46.076+0000 7fa4d8f9c700 1 --2- 172.21.15.42:0/2641528641 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa4c0004420 0x7fa4c0004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:50:46.094 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:46.076+0000 7fa4d979d700 1 --2- 172.21.15.42:0/2641528641 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa4c0003940 0x7fa4c0003d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:50:46.094 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:46.076+0000 7fa4d9f9e700 1 --2- 172.21.15.42:0/2641528641 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa4c0004e90 0x7fa4c004f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:50:46.094 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:46.076+0000 7fa4d8f9c700 1 -- 172.21.15.42:0/2641528641 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa4c0003940 msgr2=0x7fa4c0003d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:50:46.094 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:46.076+0000 7fa4d8f9c700 1 --2- 172.21.15.42:0/2641528641 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa4c0003940 0x7fa4c0003d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:50:46.095 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:46.076+0000 7fa4d8f9c700 1 -- 172.21.15.42:0/2641528641 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa4c0004e90 msgr2=0x7fa4c004f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:50:46.095 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:46.076+0000 7fa4d8f9c700 1 --2- 172.21.15.42:0/2641528641 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa4c0004e90 0x7fa4c004f680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:50:46.095 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:46.076+0000 7fa4d8f9c700 1 -- 172.21.15.42:0/2641528641 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4c0050200 con 0x7fa4c0004420 2019-12-05T05:50:46.095 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:46.076+0000 7fa4d8f9c700 1 --2- 172.21.15.42:0/2641528641 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa4c0004420 0x7fa4c0004850 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fa4c4001520 tx=0x7fa4c400c090).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:46.095 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:46.076+0000 7fa4d8f9c700 1 --2- 172.21.15.42:0/2641528641 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa4c0004420 0x7fa4c0004850 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fa4c4001520 tx=0x7fa4c400c090).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:46.095 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:46.076+0000 7fa4d8f9c700 1 --2- 172.21.15.42:0/2641528641 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa4c0004420 0x7fa4c0004850 secure :-1 s=READY pgs=1547 cs=0 l=1 rx=0x7fa4c4001520 tx=0x7fa4c400c090).ready entity=mon.1 client_cookie=5e48a5fcedf1d971 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:50:46.095 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:46.076+0000 7fa4d8f9c700 1 --2- 172.21.15.42:0/2641528641 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa4c0004420 0x7fa4c0004850 secure :-1 s=THROTTLE_DONE pgs=1547 cs=0 l=1 rx=0x7fa4c4001520 tx=0x7fa4c400c090).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:46.096 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:46.076+0000 7fa4d8f9c700 1 --2- 172.21.15.42:0/2641528641 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa4c0004420 0x7fa4c0004850 secure :-1 s=THROTTLE_DONE pgs=1547 cs=0 l=1 rx=0x7fa4c4001520 tx=0x7fa4c400c090).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:46.096 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:46.076+0000 7fa4cbfff700 1 -- 172.21.15.42:0/2641528641 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa4c400ead0 con 0x7fa4c0004420 2019-12-05T05:50:46.096 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:46.076+0000 7fa4d8f9c700 1 --2- 172.21.15.42:0/2641528641 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa4c0004420 0x7fa4c0004850 secure :-1 s=THROTTLE_DONE pgs=1547 cs=0 l=1 rx=0x7fa4c4001520 tx=0x7fa4c400c090).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:46.096 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:46.076+0000 7fa4cbfff700 1 -- 172.21.15.42:0/2641528641 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4c4010bc0 con 0x7fa4c0004420 2019-12-05T05:50:46.096 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:46.076+0000 7fa4cbfff700 1 -- 172.21.15.42:0/2641528641 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa4c4010390 con 0x7fa4c0004420 2019-12-05T05:50:46.096 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:46.076+0000 7fa4db200700 1 -- 172.21.15.42:0/2641528641 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa4c0004420 msgr2=0x7fa4c0004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:50:46.096 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:46.076+0000 7fa4db200700 1 --2- 172.21.15.42:0/2641528641 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa4c0004420 0x7fa4c0004850 secure :-1 s=READY pgs=1547 cs=0 l=1 rx=0x7fa4c4001520 tx=0x7fa4c400c090).stop 2019-12-05T05:50:46.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:46.076+0000 7fa4d9f9e700 1 -- 172.21.15.42:0/2641528641 reap_dead start 2019-12-05T05:50:46.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:46.076+0000 7fa4db200700 1 -- 172.21.15.42:0/2641528641 shutdown_connections 2019-12-05T05:50:46.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:46.076+0000 7fa4db200700 1 -- 172.21.15.42:0/2641528641 shutdown_connections 2019-12-05T05:50:46.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:46.080+0000 7fa4db200700 1 -- 172.21.15.42:0/2641528641 wait complete. 2019-12-05T05:50:46.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:46.080+0000 7fa4db200700 1 -- 172.21.15.42:0/2641528641 >> 172.21.15.42:0/2641528641 conn(0x7fa4d40aabe0 msgr2=0x7fa4d40a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:50:46.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:46.080+0000 7fa4db200700 1 Processor -- start 2019-12-05T05:50:46.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:46.080+0000 7fa4db200700 1 -- start start 2019-12-05T05:50:46.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:46.084+0000 7fa4db200700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa4d4119490 0x7fa4d40a5d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:50:46.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:46.084+0000 7fa4db200700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa4d41335a0 0x7fa4d41339d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:50:46.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:46.084+0000 7fa4db200700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4d4120a60 con 0x7fa4d409f2e0 2019-12-05T05:50:46.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:46.084+0000 7fa4db200700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4d41198c0 con 0x7fa4d4119490 2019-12-05T05:50:46.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:46.084+0000 7fa4db200700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4d4120d70 con 0x7fa4d41335a0 2019-12-05T05:50:46.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:46.084+0000 7fa4d979d700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa4d4119490 0x7fa4d40a5d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:50:46.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:46.084+0000 7fa4d8f9c700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa4d41335a0 0x7fa4d41339d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:50:46.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:46.084+0000 7fa4d979d700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa4d4119490 0x7fa4d40a5d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.42:58362/0 (socket says 172.21.15.42:58362) 2019-12-05T05:50:46.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:46.084+0000 7fa4d9f9e700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fa4d409f2e0 0x7fa4d409f710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:51794/0 (socket says 172.21.15.42:51794) 2019-12-05T05:50:46.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:46.084+0000 7fa4d979d700 1 -- 172.21.15.42:0/1772470052 learned_addr learned my addr 172.21.15.42:0/1772470052 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:50:46.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:46.084+0000 7fa4d979d700 1 -- 172.21.15.42:0/1772470052 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa4d41335a0 msgr2=0x7fa4d41339d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:50:46.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:46.084+0000 7fa4d979d700 1 --2- 172.21.15.42:0/1772470052 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa4d41335a0 0x7fa4d41339d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:50:46.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:46.084+0000 7fa4d979d700 1 -- 172.21.15.42:0/1772470052 >> v1:172.21.15.40:6789/0 conn(0x7fa4d409f2e0 legacy=0x7fa4d409f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:50:46.100 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:46.084+0000 7fa4c9ffb700 1 -- 172.21.15.42:0/1772470052 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2671207797 0 0) 0x7fa4d4120a60 con 0x7fa4d409f2e0 2019-12-05T05:50:46.100 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:46.084+0000 7fa4d979d700 1 -- 172.21.15.42:0/1772470052 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4d41218a0 con 0x7fa4d4119490 2019-12-05T05:50:46.100 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:46.084+0000 7fa4d979d700 1 --2- 172.21.15.42:0/1772470052 >> v2:172.21.15.40:3300/0 conn(0x7fa4d4119490 0x7fa4d40a5d70 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fa4d0003270 tx=0x7fa4d0010f20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:46.100 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:46.084+0000 7fa4d979d700 1 --2- 172.21.15.42:0/1772470052 >> v2:172.21.15.40:3300/0 conn(0x7fa4d4119490 0x7fa4d40a5d70 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fa4d0003270 tx=0x7fa4d0010f20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:46.100 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:46.084+0000 7fa4d979d700 1 --2- 172.21.15.42:0/1772470052 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa4d4119490 0x7fa4d40a5d70 secure :-1 s=READY pgs=1189 cs=0 l=1 rx=0x7fa4d0003270 tx=0x7fa4d0010f20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:50:46.100 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:46.084+0000 7fa4d979d700 1 --2- 172.21.15.42:0/1772470052 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa4d4119490 0x7fa4d40a5d70 secure :-1 s=THROTTLE_DONE pgs=1189 cs=0 l=1 rx=0x7fa4d0003270 tx=0x7fa4d0010f20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:46.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:46.084+0000 7fa4c9ffb700 1 -- 172.21.15.42:0/1772470052 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa4d00058c0 con 0x7fa4d4119490 2019-12-05T05:50:46.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:46.084+0000 7fa4d979d700 1 --2- 172.21.15.42:0/1772470052 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa4d4119490 0x7fa4d40a5d70 secure :-1 s=THROTTLE_DONE pgs=1189 cs=0 l=1 rx=0x7fa4d0003270 tx=0x7fa4d0010f20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:46.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:46.084+0000 7fa4d979d700 1 --2- 172.21.15.42:0/1772470052 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa4d4119490 0x7fa4d40a5d70 secure :-1 s=THROTTLE_DONE pgs=1189 cs=0 l=1 rx=0x7fa4d0003270 tx=0x7fa4d0010f20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:46.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:46.084+0000 7fa4c9ffb700 1 -- 172.21.15.42:0/1772470052 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4d000caa0 con 0x7fa4d4119490 2019-12-05T05:50:46.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:46.084+0000 7fa4db200700 1 -- 172.21.15.42:0/1772470052 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4d4133f20 con 0x7fa4d4119490 2019-12-05T05:50:46.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:46.084+0000 7fa4c9ffb700 1 -- 172.21.15.42:0/1772470052 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa4d00125a0 con 0x7fa4d4119490 2019-12-05T05:50:46.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:46.084+0000 7fa4db200700 1 -- 172.21.15.42:0/1772470052 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4d409ff10 con 0x7fa4d4119490 2019-12-05T05:50:46.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:46.088+0000 7fa4d979d700 1 --2- 172.21.15.42:0/1772470052 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa4d4119490 0x7fa4d40a5d70 secure :-1 s=THROTTLE_DONE pgs=1189 cs=0 l=1 rx=0x7fa4d0003270 tx=0x7fa4d0010f20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:46.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:46.088+0000 7fa4c9ffb700 1 -- 172.21.15.42:0/1772470052 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fa4d000f230 con 0x7fa4d4119490 2019-12-05T05:50:46.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:46.088+0000 7fa4d979d700 1 --2- 172.21.15.42:0/1772470052 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa4d4119490 0x7fa4d40a5d70 secure :-1 s=THROTTLE_DONE pgs=1189 cs=0 l=1 rx=0x7fa4d0003270 tx=0x7fa4d0010f20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:46.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:46.088+0000 7fa4c9ffb700 1 --2- 172.21.15.42:0/1772470052 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa4b403ac30 0x7fa4b403d140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:50:46.103 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:46.088+0000 7fa4c9ffb700 1 -- 172.21.15.42:0/1772470052 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 8697+0+0 (secure 0 0 0) 0x7fa4d004b1a0 con 0x7fa4d4119490 2019-12-05T05:50:46.103 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:46.088+0000 7fa4d8f9c700 1 --2- 172.21.15.42:0/1772470052 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa4b403ac30 0x7fa4b403d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:50:46.104 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:46.088+0000 7fa4d8f9c700 1 --2- 172.21.15.42:0/1772470052 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa4b403ac30 0x7fa4b403d140 crc :-1 s=READY pgs=667 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:50:46.104 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:46.088+0000 7fa4db200700 1 -- 172.21.15.42:0/1772470052 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4d405d110 con 0x7fa4d4119490 2019-12-05T05:50:46.104 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:46.096+0000 7fa4d979d700 1 --2- 172.21.15.42:0/1772470052 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa4d4119490 0x7fa4d40a5d70 secure :-1 s=THROTTLE_DONE pgs=1189 cs=0 l=1 rx=0x7fa4d0003270 tx=0x7fa4d0010f20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:46.104 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:46.096+0000 7fa4c9ffb700 1 -- 172.21.15.42:0/1772470052 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fa4d000f540 con 0x7fa4d4119490 2019-12-05T05:50:46.271 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:46.264+0000 7fa4db200700 1 -- 172.21.15.42:0/1772470052 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa4d405b8c0 con 0x7fa4d4119490 2019-12-05T05:50:46.272 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:46.268+0000 7fa4d979d700 1 --2- 172.21.15.42:0/1772470052 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa4d4119490 0x7fa4d40a5d70 secure :-1 s=THROTTLE_DONE pgs=1189 cs=0 l=1 rx=0x7fa4d0003270 tx=0x7fa4d0010f20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:46.272 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:46.268+0000 7fa4c9ffb700 1 -- 172.21.15.42:0/1772470052 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 188 v188) v1 ==== 95+0+2112 (secure 0 0 0) 0x7fa4d0032200 con 0x7fa4d4119490 2019-12-05T05:50:46.274 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 188 2019-12-05T05:50:46.274 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:50:46.274 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":188,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6245,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/2081111170","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2081111170},{"type":"v1","addr":"172.21.15.40:6833","nonce":2081111170}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":188}],"filesystems":[{"mdsmap":{"epoch":187,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:49:23.679613+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8188},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8188":{"gid":8188,"name":"a","rank":0,"incarnation":184,"state":"up:active","state_seq":33,"addr":"172.21.15.42:6835/2456922143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2456922143},{"type":"v1","addr":"172.21.15.42:6835","nonce":2456922143}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:50:46.277 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:46.272+0000 7fa4db200700 1 -- 172.21.15.42:0/1772470052 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa4b403ac30 msgr2=0x7fa4b403d140 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:50:46.277 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:46.272+0000 7fa4db200700 1 --2- 172.21.15.42:0/1772470052 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa4b403ac30 0x7fa4b403d140 crc :-1 s=READY pgs=667 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:50:46.277 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:46.272+0000 7fa4db200700 1 -- 172.21.15.42:0/1772470052 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa4d4119490 msgr2=0x7fa4d40a5d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:50:46.278 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:46.272+0000 7fa4db200700 1 --2- 172.21.15.42:0/1772470052 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa4d4119490 0x7fa4d40a5d70 secure :-1 s=READY pgs=1189 cs=0 l=1 rx=0x7fa4d0003270 tx=0x7fa4d0010f20).stop 2019-12-05T05:50:46.278 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:46.272+0000 7fa4db200700 1 -- 172.21.15.42:0/1772470052 shutdown_connections 2019-12-05T05:50:46.278 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:46.272+0000 7fa4db200700 1 --2- 172.21.15.42:0/1772470052 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa4b403ac30 0x7fa4b403d140 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:50:46.278 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:46.272+0000 7fa4db200700 1 --2- 172.21.15.42:0/1772470052 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa4d4119490 0x7fa4d40a5d70 unknown :-1 s=CLOSED pgs=1189 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:50:46.278 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:46.272+0000 7fa4db200700 1 --2- 172.21.15.42:0/1772470052 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa4d41335a0 0x7fa4d41339d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:50:46.278 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:46.272+0000 7fa4db200700 1 -- 172.21.15.42:0/1772470052 shutdown_connections 2019-12-05T05:50:46.278 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:46.272+0000 7fa4db200700 1 -- 172.21.15.42:0/1772470052 wait complete. 2019-12-05T05:50:46.279 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:46.272+0000 7fa4db200700 1 -- 172.21.15.42:0/1772470052 >> 172.21.15.42:0/1772470052 conn(0x7fa4d40a03f0 msgr2=0x7fa4d411c540 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:50:46.286 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T05:50:48.287 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:50:48.397 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:48.384+0000 7fcc77953700 1 Processor -- start 2019-12-05T05:50:48.397 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:48.384+0000 7fcc77953700 1 -- start start 2019-12-05T05:50:48.398 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:48.384+0000 7fcc77953700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcc700a7740 0x7fcc7008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:50:48.398 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:48.384+0000 7fcc77953700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fcc7007e720 0x7fcc7007eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:50:48.398 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:48.384+0000 7fcc77953700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc700a0710 con 0x7fcc7009f2e0 2019-12-05T05:50:48.398 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:48.384+0000 7fcc77953700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc700a0b00 con 0x7fcc7007e720 2019-12-05T05:50:48.398 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:48.384+0000 7fcc77953700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc7008e5b0 con 0x7fcc700a7740 2019-12-05T05:50:48.398 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:48.384+0000 7fcc766f1700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fcc7009f2e0 0x7fcc7009f710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:42368/0 (socket says 172.21.15.42:42368) 2019-12-05T05:50:48.399 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:48.384+0000 7fcc766f1700 1 -- 172.21.15.42:0/2748168172 learned_addr learned my addr 172.21.15.42:0/2748168172 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:50:48.399 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:48.384+0000 7fcc75ef0700 1 --2- 172.21.15.42:0/2748168172 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcc700a7740 0x7fcc7008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:50:48.399 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:48.384+0000 7fcc756ef700 1 --2- 172.21.15.42:0/2748168172 >> v2:172.21.15.40:3300/0 conn(0x7fcc7007e720 0x7fcc7007eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:50:48.399 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:48.384+0000 7fcc74eee700 1 -- 172.21.15.42:0/2748168172 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1577826208 0 0) 0x7fcc700a0710 con 0x7fcc7009f2e0 2019-12-05T05:50:48.399 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:48.384+0000 7fcc74eee700 1 -- 172.21.15.42:0/2748168172 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc5c0030b0 con 0x7fcc7009f2e0 2019-12-05T05:50:48.399 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:48.384+0000 7fcc75ef0700 1 -- 172.21.15.42:0/2748168172 >> v2:172.21.15.40:3300/0 conn(0x7fcc7007e720 msgr2=0x7fcc7007eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:50:48.399 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:48.384+0000 7fcc75ef0700 1 --2- 172.21.15.42:0/2748168172 >> v2:172.21.15.40:3300/0 conn(0x7fcc7007e720 0x7fcc7007eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:50:48.400 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:48.384+0000 7fcc75ef0700 1 -- 172.21.15.42:0/2748168172 >> v1:172.21.15.42:6789/0 conn(0x7fcc7009f2e0 legacy=0x7fcc7009f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:50:48.400 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:48.384+0000 7fcc74eee700 1 -- 172.21.15.42:0/2748168172 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2912244077 0 0) 0x7fcc5c0030b0 con 0x7fcc7009f2e0 2019-12-05T05:50:48.400 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:48.384+0000 7fcc75ef0700 1 -- 172.21.15.42:0/2748168172 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc7007f0a0 con 0x7fcc700a7740 2019-12-05T05:50:48.400 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:48.384+0000 7fcc756ef700 1 --2- 172.21.15.42:0/2748168172 >> v2:172.21.15.40:3300/0 conn(0x7fcc7007e720 0x7fcc7007eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:50:48.400 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:48.384+0000 7fcc75ef0700 1 --2- 172.21.15.42:0/2748168172 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcc700a7740 0x7fcc7008df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fcc6c008790 tx=0x7fcc680019d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:48.400 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:48.384+0000 7fcc75ef0700 1 --2- 172.21.15.42:0/2748168172 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcc700a7740 0x7fcc7008df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fcc6c008790 tx=0x7fcc680019d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:48.400 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:48.384+0000 7fcc75ef0700 1 --2- 172.21.15.42:0/2748168172 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcc700a7740 0x7fcc7008df80 secure :-1 s=READY pgs=346 cs=0 l=1 rx=0x7fcc6c008790 tx=0x7fcc680019d0).ready entity=mon.2 client_cookie=74c463933fbb0328 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:50:48.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:48.384+0000 7fcc75ef0700 1 --2- 172.21.15.42:0/2748168172 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcc700a7740 0x7fcc7008df80 secure :-1 s=THROTTLE_DONE pgs=346 cs=0 l=1 rx=0x7fcc6c008790 tx=0x7fcc680019d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:48.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:48.384+0000 7fcc74eee700 1 -- 172.21.15.42:0/2748168172 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcc68001750 con 0x7fcc700a7740 2019-12-05T05:50:48.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:48.384+0000 7fcc75ef0700 1 --2- 172.21.15.42:0/2748168172 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcc700a7740 0x7fcc7008df80 secure :-1 s=THROTTLE_DONE pgs=346 cs=0 l=1 rx=0x7fcc6c008790 tx=0x7fcc680019d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:48.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:48.388+0000 7fcc75ef0700 1 --2- 172.21.15.42:0/2748168172 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcc700a7740 0x7fcc7008df80 secure :-1 s=THROTTLE_DONE pgs=346 cs=0 l=1 rx=0x7fcc6c008790 tx=0x7fcc680019d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:48.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:48.388+0000 7fcc74eee700 1 -- 172.21.15.42:0/2748168172 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc68001560 con 0x7fcc700a7740 2019-12-05T05:50:48.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:48.388+0000 7fcc74eee700 1 -- 172.21.15.42:0/2748168172 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcc6c0113e0 con 0x7fcc700a7740 2019-12-05T05:50:48.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:48.388+0000 7fcc77953700 1 -- 172.21.15.42:0/2748168172 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcc700a7740 msgr2=0x7fcc7008df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:50:48.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:48.388+0000 7fcc77953700 1 --2- 172.21.15.42:0/2748168172 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcc700a7740 0x7fcc7008df80 secure :-1 s=READY pgs=346 cs=0 l=1 rx=0x7fcc6c008790 tx=0x7fcc680019d0).stop 2019-12-05T05:50:48.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:48.388+0000 7fcc77953700 1 -- 172.21.15.42:0/2748168172 shutdown_connections 2019-12-05T05:50:48.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:48.388+0000 7fcc77953700 1 --2- 172.21.15.42:0/2748168172 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcc700a7740 0x7fcc7008df80 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:50:48.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:48.388+0000 7fcc77953700 1 --2- 172.21.15.42:0/2748168172 >> v2:172.21.15.40:3300/0 conn(0x7fcc7007e720 0x7fcc7007eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:50:48.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:48.388+0000 7fcc77953700 1 -- 172.21.15.42:0/2748168172 shutdown_connections 2019-12-05T05:50:48.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:48.388+0000 7fcc77953700 1 -- 172.21.15.42:0/2748168172 wait complete. 2019-12-05T05:50:48.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:48.388+0000 7fcc77953700 1 -- 172.21.15.42:0/2748168172 >> 172.21.15.42:0/2748168172 conn(0x7fcc700aabe0 msgr2=0x7fcc700a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:50:48.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:48.388+0000 7fcc77953700 1 Processor -- start 2019-12-05T05:50:48.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:48.388+0000 7fcc77953700 1 -- start start 2019-12-05T05:50:48.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:48.388+0000 7fcc77953700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fcc7008f680 0x7fcc70119740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:50:48.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:48.388+0000 7fcc77953700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcc7008eb70 0x7fcc70135720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:50:48.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:48.388+0000 7fcc77953700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc7011a7e0 con 0x7fcc701181b0 2019-12-05T05:50:48.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:48.388+0000 7fcc77953700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc70119c90 con 0x7fcc7008f680 2019-12-05T05:50:48.404 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:48.388+0000 7fcc77953700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc7011a490 con 0x7fcc7008eb70 2019-12-05T05:50:48.404 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:48.392+0000 7fcc756ef700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fcc7008f680 0x7fcc70119740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:50:48.404 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:48.392+0000 7fcc766f1700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcc7008eb70 0x7fcc70135720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:50:48.404 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:48.392+0000 7fcc756ef700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fcc7008f680 0x7fcc70119740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.42:58382/0 (socket says 172.21.15.42:58382) 2019-12-05T05:50:48.404 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:48.392+0000 7fcc756ef700 1 -- 172.21.15.42:0/1878955045 learned_addr learned my addr 172.21.15.42:0/1878955045 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:50:48.404 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:48.392+0000 7fcc75ef0700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fcc701181b0 0x7fcc70118f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:51810/0 (socket says 172.21.15.42:51810) 2019-12-05T05:50:48.404 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:48.392+0000 7fcc756ef700 1 -- 172.21.15.42:0/1878955045 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcc7008eb70 msgr2=0x7fcc70135720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:50:48.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:48.392+0000 7fcc627fc700 1 -- 172.21.15.42:0/1878955045 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 689536503 0 0) 0x7fcc7011a7e0 con 0x7fcc701181b0 2019-12-05T05:50:48.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:48.392+0000 7fcc756ef700 1 --2- 172.21.15.42:0/1878955045 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcc7008eb70 0x7fcc70135720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:50:48.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:48.392+0000 7fcc756ef700 1 -- 172.21.15.42:0/1878955045 >> v1:172.21.15.40:6789/0 conn(0x7fcc701181b0 legacy=0x7fcc70118f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:50:48.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:48.392+0000 7fcc756ef700 1 -- 172.21.15.42:0/1878955045 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc7008efa0 con 0x7fcc7008f680 2019-12-05T05:50:48.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:48.392+0000 7fcc766f1700 1 --2- 172.21.15.42:0/1878955045 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcc7008eb70 0x7fcc70135720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:50:48.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:48.392+0000 7fcc756ef700 1 --2- 172.21.15.42:0/1878955045 >> v2:172.21.15.40:3300/0 conn(0x7fcc7008f680 0x7fcc70119740 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fcc7011aa10 tx=0x7fcc64011160).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:48.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:48.392+0000 7fcc756ef700 1 --2- 172.21.15.42:0/1878955045 >> v2:172.21.15.40:3300/0 conn(0x7fcc7008f680 0x7fcc70119740 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fcc7011aa10 tx=0x7fcc64011160).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:48.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:48.392+0000 7fcc756ef700 1 --2- 172.21.15.42:0/1878955045 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc7008f680 0x7fcc70119740 secure :-1 s=READY pgs=1191 cs=0 l=1 rx=0x7fcc7011aa10 tx=0x7fcc64011160).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:50:48.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:48.392+0000 7fcc756ef700 1 --2- 172.21.15.42:0/1878955045 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc7008f680 0x7fcc70119740 secure :-1 s=THROTTLE_DONE pgs=1191 cs=0 l=1 rx=0x7fcc7011aa10 tx=0x7fcc64011160).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:48.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:48.392+0000 7fcc627fc700 1 -- 172.21.15.42:0/1878955045 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcc64007160 con 0x7fcc7008f680 2019-12-05T05:50:48.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:48.392+0000 7fcc756ef700 1 --2- 172.21.15.42:0/1878955045 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc7008f680 0x7fcc70119740 secure :-1 s=THROTTLE_DONE pgs=1191 cs=0 l=1 rx=0x7fcc7011aa10 tx=0x7fcc64011160).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:48.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:48.392+0000 7fcc756ef700 1 --2- 172.21.15.42:0/1878955045 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc7008f680 0x7fcc70119740 secure :-1 s=THROTTLE_DONE pgs=1191 cs=0 l=1 rx=0x7fcc7011aa10 tx=0x7fcc64011160).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:48.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:48.392+0000 7fcc627fc700 1 -- 172.21.15.42:0/1878955045 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc64005c10 con 0x7fcc7008f680 2019-12-05T05:50:48.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:48.392+0000 7fcc77953700 1 -- 172.21.15.42:0/1878955045 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc7012f120 con 0x7fcc7008f680 2019-12-05T05:50:48.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:48.392+0000 7fcc627fc700 1 -- 172.21.15.42:0/1878955045 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcc640127a0 con 0x7fcc7008f680 2019-12-05T05:50:48.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:48.392+0000 7fcc77953700 1 -- 172.21.15.42:0/1878955045 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc70135c70 con 0x7fcc7008f680 2019-12-05T05:50:48.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:48.392+0000 7fcc756ef700 1 --2- 172.21.15.42:0/1878955045 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc7008f680 0x7fcc70119740 secure :-1 s=THROTTLE_DONE pgs=1191 cs=0 l=1 rx=0x7fcc7011aa10 tx=0x7fcc64011160).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:48.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:48.396+0000 7fcc627fc700 1 -- 172.21.15.42:0/1878955045 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fcc64012ac0 con 0x7fcc7008f680 2019-12-05T05:50:48.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:48.396+0000 7fcc756ef700 1 --2- 172.21.15.42:0/1878955045 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc7008f680 0x7fcc70119740 secure :-1 s=THROTTLE_DONE pgs=1191 cs=0 l=1 rx=0x7fcc7011aa10 tx=0x7fcc64011160).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:48.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:48.396+0000 7fcc627fc700 1 --2- 172.21.15.42:0/1878955045 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fcc5803abe0 0x7fcc5803d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:50:48.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:48.396+0000 7fcc627fc700 1 -- 172.21.15.42:0/1878955045 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 8697+0+0 (secure 0 0 0) 0x7fcc6404b310 con 0x7fcc7008f680 2019-12-05T05:50:48.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:48.396+0000 7fcc75ef0700 1 --2- 172.21.15.42:0/1878955045 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fcc5803abe0 0x7fcc5803d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:50:48.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:48.396+0000 7fcc75ef0700 1 --2- 172.21.15.42:0/1878955045 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fcc5803abe0 0x7fcc5803d0f0 crc :-1 s=READY pgs=668 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:50:48.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:48.400+0000 7fcc77953700 1 -- 172.21.15.42:0/1878955045 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc7005d110 con 0x7fcc7008f680 2019-12-05T05:50:48.414 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:48.408+0000 7fcc756ef700 1 --2- 172.21.15.42:0/1878955045 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc7008f680 0x7fcc70119740 secure :-1 s=THROTTLE_DONE pgs=1191 cs=0 l=1 rx=0x7fcc7011aa10 tx=0x7fcc64011160).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:48.415 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:48.408+0000 7fcc627fc700 1 -- 172.21.15.42:0/1878955045 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fcc64012dd0 con 0x7fcc7008f680 2019-12-05T05:50:48.582 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:48.576+0000 7fcc77953700 1 -- 172.21.15.42:0/1878955045 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcc700a7150 con 0x7fcc7008f680 2019-12-05T05:50:48.583 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:48.580+0000 7fcc756ef700 1 --2- 172.21.15.42:0/1878955045 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc7008f680 0x7fcc70119740 secure :-1 s=THROTTLE_DONE pgs=1191 cs=0 l=1 rx=0x7fcc7011aa10 tx=0x7fcc64011160).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:48.583 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:48.580+0000 7fcc627fc700 1 -- 172.21.15.42:0/1878955045 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 188 v188) v1 ==== 95+0+2112 (secure 0 0 0) 0x7fcc640330e0 con 0x7fcc7008f680 2019-12-05T05:50:48.585 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 188 2019-12-05T05:50:48.585 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:50:48.586 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":188,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6245,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/2081111170","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2081111170},{"type":"v1","addr":"172.21.15.40:6833","nonce":2081111170}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":188}],"filesystems":[{"mdsmap":{"epoch":187,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:49:23.679613+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8188},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8188":{"gid":8188,"name":"a","rank":0,"incarnation":184,"state":"up:active","state_seq":33,"addr":"172.21.15.42:6835/2456922143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2456922143},{"type":"v1","addr":"172.21.15.42:6835","nonce":2456922143}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:50:48.588 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:48.584+0000 7fcc77953700 1 -- 172.21.15.42:0/1878955045 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fcc5803abe0 msgr2=0x7fcc5803d0f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:50:48.588 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:48.584+0000 7fcc77953700 1 --2- 172.21.15.42:0/1878955045 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fcc5803abe0 0x7fcc5803d0f0 crc :-1 s=READY pgs=668 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:50:48.589 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:48.584+0000 7fcc77953700 1 -- 172.21.15.42:0/1878955045 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc7008f680 msgr2=0x7fcc70119740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:50:48.589 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:48.584+0000 7fcc77953700 1 --2- 172.21.15.42:0/1878955045 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc7008f680 0x7fcc70119740 secure :-1 s=READY pgs=1191 cs=0 l=1 rx=0x7fcc7011aa10 tx=0x7fcc64011160).stop 2019-12-05T05:50:48.589 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:48.584+0000 7fcc77953700 1 -- 172.21.15.42:0/1878955045 shutdown_connections 2019-12-05T05:50:48.589 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:48.584+0000 7fcc77953700 1 --2- 172.21.15.42:0/1878955045 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fcc5803abe0 0x7fcc5803d0f0 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:50:48.589 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:48.584+0000 7fcc77953700 1 --2- 172.21.15.42:0/1878955045 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcc7008eb70 0x7fcc70135720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:50:48.590 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:48.584+0000 7fcc77953700 1 --2- 172.21.15.42:0/1878955045 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc7008f680 0x7fcc70119740 unknown :-1 s=CLOSED pgs=1191 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:50:48.590 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:48.584+0000 7fcc77953700 1 -- 172.21.15.42:0/1878955045 shutdown_connections 2019-12-05T05:50:48.590 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:48.584+0000 7fcc77953700 1 -- 172.21.15.42:0/1878955045 wait complete. 2019-12-05T05:50:48.590 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:48.584+0000 7fcc77953700 1 -- 172.21.15.42:0/1878955045 >> 172.21.15.42:0/1878955045 conn(0x7fcc700a82a0 msgr2=0x7fcc7009f380 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:50:48.599 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T05:50:50.600 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:50:50.713 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:50.700+0000 7f261ad9c700 1 Processor -- start 2019-12-05T05:50:50.713 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:50.700+0000 7f261ad9c700 1 -- start start 2019-12-05T05:50:50.713 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:50.704+0000 7f261ad9c700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f261408df80 0x7f261409f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:50:50.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:50.704+0000 7f261ad9c700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f261409fc60 con 0x7f26140a7740 2019-12-05T05:50:50.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:50.704+0000 7f261ad9c700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26140a0710 con 0x7f261409f830 2019-12-05T05:50:50.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:50.704+0000 7f261ad9c700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26140a0b00 con 0x7f261408df80 2019-12-05T05:50:50.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:50.704+0000 7f2618b38700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f261408df80 0x7f261409f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:50:50.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:50.704+0000 7f2618b38700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f261408df80 0x7f261409f2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.42:58390/0 (socket says 172.21.15.42:58390) 2019-12-05T05:50:50.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:50.704+0000 7f2618b38700 1 -- 172.21.15.42:0/4195103291 learned_addr learned my addr 172.21.15.42:0/4195103291 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:50:50.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:50.704+0000 7f2619b3a700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f261409f830 0x7f261414efe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:42382/0 (socket says 172.21.15.42:42382) 2019-12-05T05:50:50.715 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:50.704+0000 7f2619339700 1 --1- 172.21.15.42:0/4195103291 >> v1:172.21.15.40:6789/0 conn(0x7f26140a7740 0x7f261407e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:51818/0 (socket says 172.21.15.42:51818) 2019-12-05T05:50:50.715 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:50.704+0000 7f2618b38700 1 -- 172.21.15.42:0/4195103291 >> v1:172.21.15.42:6789/0 conn(0x7f261409f830 legacy=0x7f261414efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:50:50.715 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:50.704+0000 7f2618b38700 1 -- 172.21.15.42:0/4195103291 >> v1:172.21.15.40:6789/0 conn(0x7f26140a7740 legacy=0x7f261407e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:50:50.715 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:50.704+0000 7f2618b38700 1 -- 172.21.15.42:0/4195103291 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f261407ef20 con 0x7f261408df80 2019-12-05T05:50:50.715 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:50.704+0000 7f2618b38700 1 --2- 172.21.15.42:0/4195103291 >> v2:172.21.15.40:3300/0 conn(0x7f261408df80 0x7f261409f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f2604008790 tx=0x7f260400fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:50.715 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:50.704+0000 7f2618b38700 1 --2- 172.21.15.42:0/4195103291 >> v2:172.21.15.40:3300/0 conn(0x7f261408df80 0x7f261409f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f2604008790 tx=0x7f260400fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:50.715 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:50.704+0000 7f2618b38700 1 --2- 172.21.15.42:0/4195103291 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f261408df80 0x7f261409f2e0 secure :-1 s=READY pgs=1193 cs=0 l=1 rx=0x7f2604008790 tx=0x7f260400fcb0).ready entity=mon.0 client_cookie=f4cc10592f5e539c server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:50:50.716 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:50.704+0000 7f2618b38700 1 --2- 172.21.15.42:0/4195103291 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f261408df80 0x7f261409f2e0 secure :-1 s=THROTTLE_DONE pgs=1193 cs=0 l=1 rx=0x7f2604008790 tx=0x7f260400fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:50.716 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:50.704+0000 7f2618b38700 1 --2- 172.21.15.42:0/4195103291 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f261408df80 0x7f261409f2e0 secure :-1 s=THROTTLE_DONE pgs=1193 cs=0 l=1 rx=0x7f2604008790 tx=0x7f260400fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:50.716 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:50.704+0000 7f260bfff700 1 -- 172.21.15.42:0/4195103291 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2604011520 con 0x7f261408df80 2019-12-05T05:50:50.716 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:50.704+0000 7f2618b38700 1 --2- 172.21.15.42:0/4195103291 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f261408df80 0x7f261409f2e0 secure :-1 s=THROTTLE_DONE pgs=1193 cs=0 l=1 rx=0x7f2604008790 tx=0x7f260400fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:50.716 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:50.704+0000 7f260bfff700 1 -- 172.21.15.42:0/4195103291 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2604008ca0 con 0x7f261408df80 2019-12-05T05:50:50.716 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:50.704+0000 7f260bfff700 1 -- 172.21.15.42:0/4195103291 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2604011b00 con 0x7f261408df80 2019-12-05T05:50:50.716 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:50.704+0000 7f261ad9c700 1 -- 172.21.15.42:0/4195103291 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f261408df80 msgr2=0x7f261409f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:50:50.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:50.704+0000 7f261ad9c700 1 --2- 172.21.15.42:0/4195103291 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f261408df80 0x7f261409f2e0 secure :-1 s=READY pgs=1193 cs=0 l=1 rx=0x7f2604008790 tx=0x7f260400fcb0).stop 2019-12-05T05:50:50.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:50.708+0000 7f261ad9c700 1 -- 172.21.15.42:0/4195103291 shutdown_connections 2019-12-05T05:50:50.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:50.708+0000 7f261ad9c700 1 --2- 172.21.15.42:0/4195103291 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f261408df80 0x7f261409f2e0 unknown :-1 s=CLOSED pgs=1193 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:50:50.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:50.708+0000 7f261ad9c700 1 -- 172.21.15.42:0/4195103291 shutdown_connections 2019-12-05T05:50:50.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:50.708+0000 7f261ad9c700 1 -- 172.21.15.42:0/4195103291 wait complete. 2019-12-05T05:50:50.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:50.708+0000 7f261ad9c700 1 -- 172.21.15.42:0/4195103291 >> 172.21.15.42:0/4195103291 conn(0x7f26140aabe0 msgr2=0x7f26140a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:50:50.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:50.708+0000 7f261ad9c700 1 Processor -- start 2019-12-05T05:50:50.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:50.708+0000 7f261ad9c700 1 -- start start 2019-12-05T05:50:50.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:50.712+0000 7f261ad9c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2614119b20 0x7f261407b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:50:50.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:50.712+0000 7f261ad9c700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26140a5510 con 0x7f261407b800 2019-12-05T05:50:50.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:50.712+0000 7f261ad9c700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f261411bb90 con 0x7f26141188c0 2019-12-05T05:50:50.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:50.712+0000 7f261ad9c700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f261411b650 con 0x7f2614119b20 2019-12-05T05:50:50.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:50.712+0000 7f2619339700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f26141188c0 0x7f26140a4d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:42384/0 (socket says 172.21.15.42:42384) 2019-12-05T05:50:50.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:50.712+0000 7f2619339700 1 -- 172.21.15.42:0/1654067869 learned_addr learned my addr 172.21.15.42:0/1654067869 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:50:50.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:50.712+0000 7f2618b38700 1 --2- 172.21.15.42:0/1654067869 >> v2:172.21.15.42:3300/0 conn(0x7f2614119b20 0x7f261407b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:50:50.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:50.712+0000 7f2609ffb700 1 -- 172.21.15.42:0/1654067869 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1448578147 0 0) 0x7f261411bb90 con 0x7f26141188c0 2019-12-05T05:50:50.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:50.712+0000 7f2609ffb700 1 -- 172.21.15.42:0/1654067869 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25f40030b0 con 0x7f26141188c0 2019-12-05T05:50:50.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:50.712+0000 7f2618b38700 1 -- 172.21.15.42:0/1654067869 >> v1:172.21.15.42:6789/0 conn(0x7f26141188c0 legacy=0x7f26140a4d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:50:50.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:50.712+0000 7f2618b38700 1 -- 172.21.15.42:0/1654067869 >> v1:172.21.15.40:6789/0 conn(0x7f261407b800 legacy=0x7f2614138a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:50:50.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:50.712+0000 7f2618b38700 1 -- 172.21.15.42:0/1654067869 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f261411ac30 con 0x7f2614119b20 2019-12-05T05:50:50.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:50.712+0000 7f2618b38700 1 --2- 172.21.15.42:0/1654067869 >> v2:172.21.15.42:3300/0 conn(0x7f2614119b20 0x7f261407b2b0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f2604008790 tx=0x7f2604011560).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:50.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:50.712+0000 7f2618b38700 1 --2- 172.21.15.42:0/1654067869 >> v2:172.21.15.42:3300/0 conn(0x7f2614119b20 0x7f261407b2b0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f2604008790 tx=0x7f2604011560).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:50.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:50.712+0000 7f2618b38700 1 --2- 172.21.15.42:0/1654067869 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2614119b20 0x7f261407b2b0 secure :-1 s=READY pgs=1551 cs=0 l=1 rx=0x7f2604008790 tx=0x7f2604011560).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:50:50.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:50.712+0000 7f2618b38700 1 --2- 172.21.15.42:0/1654067869 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2614119b20 0x7f261407b2b0 secure :-1 s=THROTTLE_DONE pgs=1551 cs=0 l=1 rx=0x7f2604008790 tx=0x7f2604011560).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:50.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:50.712+0000 7f2609ffb700 1 -- 172.21.15.42:0/1654067869 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2604007150 con 0x7f2614119b20 2019-12-05T05:50:50.724 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:50.712+0000 7f2618b38700 1 --2- 172.21.15.42:0/1654067869 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2614119b20 0x7f261407b2b0 secure :-1 s=THROTTLE_DONE pgs=1551 cs=0 l=1 rx=0x7f2604008790 tx=0x7f2604011560).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:50.724 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:50.712+0000 7f2609ffb700 1 -- 172.21.15.42:0/1654067869 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2604011720 con 0x7f2614119b20 2019-12-05T05:50:50.724 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:50.712+0000 7f2618b38700 1 --2- 172.21.15.42:0/1654067869 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2614119b20 0x7f261407b2b0 secure :-1 s=THROTTLE_DONE pgs=1551 cs=0 l=1 rx=0x7f2604008790 tx=0x7f2604011560).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:50.724 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:50.712+0000 7f261ad9c700 1 -- 172.21.15.42:0/1654067869 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2614119f50 con 0x7f2614119b20 2019-12-05T05:50:50.724 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:50.712+0000 7f2609ffb700 1 -- 172.21.15.42:0/1654067869 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2604005730 con 0x7f2614119b20 2019-12-05T05:50:50.724 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:50.712+0000 7f261ad9c700 1 -- 172.21.15.42:0/1654067869 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f261413a290 con 0x7f2614119b20 2019-12-05T05:50:50.725 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:50.712+0000 7f2618b38700 1 --2- 172.21.15.42:0/1654067869 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2614119b20 0x7f261407b2b0 secure :-1 s=THROTTLE_DONE pgs=1551 cs=0 l=1 rx=0x7f2604008790 tx=0x7f2604011560).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:50.725 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:50.716+0000 7f2609ffb700 1 -- 172.21.15.42:0/1654067869 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f2604005a10 con 0x7f2614119b20 2019-12-05T05:50:50.725 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:50.716+0000 7f2618b38700 1 --2- 172.21.15.42:0/1654067869 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2614119b20 0x7f261407b2b0 secure :-1 s=THROTTLE_DONE pgs=1551 cs=0 l=1 rx=0x7f2604008790 tx=0x7f2604011560).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:50.725 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:50.716+0000 7f2609ffb700 1 --2- 172.21.15.42:0/1654067869 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f25f403b250 0x7f25f403d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:50:50.725 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:50.716+0000 7f2609ffb700 1 -- 172.21.15.42:0/1654067869 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8783+0+0 (secure 0 0 0) 0x7f260404b720 con 0x7f2614119b20 2019-12-05T05:50:50.725 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:50.716+0000 7f2619339700 1 --2- 172.21.15.42:0/1654067869 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f25f403b250 0x7f25f403d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:50:50.726 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:50.716+0000 7f2619339700 1 --2- 172.21.15.42:0/1654067869 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f25f403b250 0x7f25f403d760 crc :-1 s=READY pgs=669 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:50:50.726 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:50.716+0000 7f261ad9c700 1 -- 172.21.15.42:0/1654067869 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f261405d110 con 0x7f2614119b20 2019-12-05T05:50:50.727 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:50.724+0000 7f2618b38700 1 --2- 172.21.15.42:0/1654067869 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2614119b20 0x7f261407b2b0 secure :-1 s=THROTTLE_DONE pgs=1551 cs=0 l=1 rx=0x7f2604008790 tx=0x7f2604011560).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:50.728 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:50.724+0000 7f2609ffb700 1 -- 172.21.15.42:0/1654067869 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f2604032780 con 0x7f2614119b20 2019-12-05T05:50:50.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:50.888+0000 7f261ad9c700 1 -- 172.21.15.42:0/1654067869 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f261405b8c0 con 0x7f2614119b20 2019-12-05T05:50:50.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:50.888+0000 7f2618b38700 1 --2- 172.21.15.42:0/1654067869 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2614119b20 0x7f261407b2b0 secure :-1 s=THROTTLE_DONE pgs=1551 cs=0 l=1 rx=0x7f2604008790 tx=0x7f2604011560).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:50.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:50.888+0000 7f2609ffb700 1 -- 172.21.15.42:0/1654067869 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 189 v189) v1 ==== 95+0+1774 (secure 0 0 0) 0x7f26040329e0 con 0x7f2614119b20 2019-12-05T05:50:50.896 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:50:50.897 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":189,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":189,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:50:49.116852+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6245},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6245":{"gid":6245,"name":"b","rank":0,"incarnation":189,"state":"up:replay","state_seq":1,"addr":"172.21.15.40:6833/2081111170","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2081111170},{"type":"v1","addr":"172.21.15.40:6833","nonce":2081111170}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:50:50.897 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 189 2019-12-05T05:50:50.898 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:50.892+0000 7f261ad9c700 1 -- 172.21.15.42:0/1654067869 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f25f403b250 msgr2=0x7f25f403d760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:50:50.898 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:50.892+0000 7f261ad9c700 1 --2- 172.21.15.42:0/1654067869 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f25f403b250 0x7f25f403d760 crc :-1 s=READY pgs=669 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:50:50.898 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:50.892+0000 7f261ad9c700 1 -- 172.21.15.42:0/1654067869 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2614119b20 msgr2=0x7f261407b2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:50:50.899 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:50.892+0000 7f261ad9c700 1 --2- 172.21.15.42:0/1654067869 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2614119b20 0x7f261407b2b0 secure :-1 s=READY pgs=1551 cs=0 l=1 rx=0x7f2604008790 tx=0x7f2604011560).stop 2019-12-05T05:50:50.899 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:50.892+0000 7f261ad9c700 1 -- 172.21.15.42:0/1654067869 shutdown_connections 2019-12-05T05:50:50.899 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:50.892+0000 7f261ad9c700 1 --2- 172.21.15.42:0/1654067869 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f25f403b250 0x7f25f403d760 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:50:50.899 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:50.892+0000 7f261ad9c700 1 --2- 172.21.15.42:0/1654067869 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2614119b20 0x7f261407b2b0 unknown :-1 s=CLOSED pgs=1551 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:50:50.899 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:50.892+0000 7f261ad9c700 1 -- 172.21.15.42:0/1654067869 shutdown_connections 2019-12-05T05:50:50.899 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:50.892+0000 7f261ad9c700 1 -- 172.21.15.42:0/1654067869 wait complete. 2019-12-05T05:50:50.900 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:50.892+0000 7f261ad9c700 1 -- 172.21.15.42:0/1654067869 >> 172.21.15.42:0/1654067869 conn(0x7f26140a97e0 msgr2=0x7f261411dcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:50:50.909 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a down, removed from mdsmap 2019-12-05T05:50:50.909 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2019-12-05T05:50:50.910 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:50:51.008 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:50.996+0000 7fc5eacac700 1 Processor -- start 2019-12-05T05:50:51.008 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:50.996+0000 7fc5eacac700 1 -- start start 2019-12-05T05:50:51.008 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:51.000+0000 7fc5eacac700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc5e408df80 0x7fc5e409f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:50:51.009 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:51.000+0000 7fc5eacac700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc5e409f830 0x7fc5e414efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:50:51.009 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:51.000+0000 7fc5eacac700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5e409fc60 con 0x7fc5e40a7740 2019-12-05T05:50:51.009 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:51.000+0000 7fc5eacac700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5e40a0b00 con 0x7fc5e409f830 2019-12-05T05:50:51.009 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:51.000+0000 7fc5eacac700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5e408e5b0 con 0x7fc5e408df80 2019-12-05T05:50:51.009 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:51.000+0000 7fc5e8a48700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc5e408df80 0x7fc5e409f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:50:51.009 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:51.000+0000 7fc5e8a48700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc5e408df80 0x7fc5e409f2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:42356/0 (socket says 172.21.15.42:42356) 2019-12-05T05:50:51.010 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:51.000+0000 7fc5e8a48700 1 -- 172.21.15.42:0/2016093108 learned_addr learned my addr 172.21.15.42:0/2016093108 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:50:51.010 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:51.000+0000 7fc5e9249700 1 --1- 172.21.15.42:0/2016093108 >> v1:172.21.15.42:6789/0 conn(0x7fc5e40a7740 0x7fc5e407e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:42392/0 (socket says 172.21.15.42:42392) 2019-12-05T05:50:51.010 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:51.000+0000 7fc5e9a4a700 1 --2- 172.21.15.42:0/2016093108 >> v2:172.21.15.40:3300/0 conn(0x7fc5e409f830 0x7fc5e414efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:50:51.010 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:51.000+0000 7fc5e8a48700 1 -- 172.21.15.42:0/2016093108 >> v2:172.21.15.40:3300/0 conn(0x7fc5e409f830 msgr2=0x7fc5e414efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:50:51.010 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:51.000+0000 7fc5e8a48700 1 --2- 172.21.15.42:0/2016093108 >> v2:172.21.15.40:3300/0 conn(0x7fc5e409f830 0x7fc5e414efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:50:51.010 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:51.000+0000 7fc5e8a48700 1 -- 172.21.15.42:0/2016093108 >> v1:172.21.15.42:6789/0 conn(0x7fc5e40a7740 legacy=0x7fc5e407e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:50:51.010 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:51.000+0000 7fc5e8a48700 1 -- 172.21.15.42:0/2016093108 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5e40a0710 con 0x7fc5e408df80 2019-12-05T05:50:51.011 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:51.000+0000 7fc5e8a48700 1 --2- 172.21.15.42:0/2016093108 >> v2:172.21.15.42:3300/0 conn(0x7fc5e408df80 0x7fc5e409f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fc5d8002f30 tx=0x7fc5d800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:51.011 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:51.000+0000 7fc5e8a48700 1 --2- 172.21.15.42:0/2016093108 >> v2:172.21.15.42:3300/0 conn(0x7fc5e408df80 0x7fc5e409f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fc5d8002f30 tx=0x7fc5d800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:51.011 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:51.000+0000 7fc5e8a48700 1 --2- 172.21.15.42:0/2016093108 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc5e408df80 0x7fc5e409f2e0 secure :-1 s=READY pgs=1553 cs=0 l=1 rx=0x7fc5d8002f30 tx=0x7fc5d800fd30).ready entity=mon.1 client_cookie=78e7058aaa9f7f71 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:50:51.011 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:51.000+0000 7fc5e8a48700 1 --2- 172.21.15.42:0/2016093108 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc5e408df80 0x7fc5e409f2e0 secure :-1 s=THROTTLE_DONE pgs=1553 cs=0 l=1 rx=0x7fc5d8002f30 tx=0x7fc5d800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:51.011 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:51.000+0000 7fc5d7fff700 1 -- 172.21.15.42:0/2016093108 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc5d8008c90 con 0x7fc5e408df80 2019-12-05T05:50:51.012 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:51.000+0000 7fc5e8a48700 1 --2- 172.21.15.42:0/2016093108 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc5e408df80 0x7fc5e409f2e0 secure :-1 s=THROTTLE_DONE pgs=1553 cs=0 l=1 rx=0x7fc5d8002f30 tx=0x7fc5d800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:51.012 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:51.000+0000 7fc5e8a48700 1 --2- 172.21.15.42:0/2016093108 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc5e408df80 0x7fc5e409f2e0 secure :-1 s=THROTTLE_DONE pgs=1553 cs=0 l=1 rx=0x7fc5d8002f30 tx=0x7fc5d800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:51.012 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:51.000+0000 7fc5d7fff700 1 -- 172.21.15.42:0/2016093108 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5d80088b0 con 0x7fc5e408df80 2019-12-05T05:50:51.012 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:51.000+0000 7fc5d7fff700 1 -- 172.21.15.42:0/2016093108 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc5d8011ad0 con 0x7fc5e408df80 2019-12-05T05:50:51.012 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:51.000+0000 7fc5eacac700 1 -- 172.21.15.42:0/2016093108 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc5e408df80 msgr2=0x7fc5e409f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:50:51.012 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:51.000+0000 7fc5eacac700 1 --2- 172.21.15.42:0/2016093108 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc5e408df80 0x7fc5e409f2e0 secure :-1 s=READY pgs=1553 cs=0 l=1 rx=0x7fc5d8002f30 tx=0x7fc5d800fd30).stop 2019-12-05T05:50:51.013 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:51.000+0000 7fc5eacac700 1 -- 172.21.15.42:0/2016093108 shutdown_connections 2019-12-05T05:50:51.013 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:51.000+0000 7fc5eacac700 1 --2- 172.21.15.42:0/2016093108 >> v2:172.21.15.40:3300/0 conn(0x7fc5e409f830 0x7fc5e414efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:50:51.013 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:51.000+0000 7fc5eacac700 1 --2- 172.21.15.42:0/2016093108 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc5e408df80 0x7fc5e409f2e0 unknown :-1 s=CLOSED pgs=1553 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:50:51.013 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:51.000+0000 7fc5eacac700 1 -- 172.21.15.42:0/2016093108 shutdown_connections 2019-12-05T05:50:51.013 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:51.000+0000 7fc5eacac700 1 -- 172.21.15.42:0/2016093108 wait complete. 2019-12-05T05:50:51.013 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:51.000+0000 7fc5eacac700 1 -- 172.21.15.42:0/2016093108 >> 172.21.15.42:0/2016093108 conn(0x7fc5e40aabe0 msgr2=0x7fc5e40a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:50:51.017 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:51.004+0000 7fc5eacac700 1 Processor -- start 2019-12-05T05:50:51.017 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:51.004+0000 7fc5eacac700 1 -- start start 2019-12-05T05:50:51.017 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:51.008+0000 7fc5eacac700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc5e40a1fe0 0x7fc5e4064970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:50:51.017 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:51.008+0000 7fc5eacac700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc5e4129c30 0x7fc5e412a060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:50:51.017 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:51.008+0000 7fc5eacac700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5e41240f0 con 0x7fc5e4144a50 2019-12-05T05:50:51.017 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:51.008+0000 7fc5eacac700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5e40650f0 con 0x7fc5e40a1fe0 2019-12-05T05:50:51.018 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:51.008+0000 7fc5eacac700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5e4065390 con 0x7fc5e4129c30 2019-12-05T05:50:51.018 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:51.008+0000 7fc5e8a48700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc5e40a1fe0 0x7fc5e4064970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:50:51.018 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:51.008+0000 7fc5e9249700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fc5e4144a50 0x7fc5e41238f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:42398/0 (socket says 172.21.15.42:42398) 2019-12-05T05:50:51.018 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:51.008+0000 7fc5e9249700 1 -- 172.21.15.42:0/1910128087 learned_addr learned my addr 172.21.15.42:0/1910128087 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:50:51.018 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:51.008+0000 7fc5e9a4a700 1 --2- 172.21.15.42:0/1910128087 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc5e4129c30 0x7fc5e412a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:50:51.018 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:51.008+0000 7fc5d5ffb700 1 -- 172.21.15.42:0/1910128087 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1073934146 0 0) 0x7fc5e41240f0 con 0x7fc5e4144a50 2019-12-05T05:50:51.018 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:51.008+0000 7fc5e8a48700 1 -- 172.21.15.42:0/1910128087 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc5e4129c30 msgr2=0x7fc5e412a060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:50:51.019 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:51.008+0000 7fc5e8a48700 1 --2- 172.21.15.42:0/1910128087 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc5e4129c30 0x7fc5e412a060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:50:51.019 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:51.008+0000 7fc5e8a48700 1 -- 172.21.15.42:0/1910128087 >> v1:172.21.15.42:6789/0 conn(0x7fc5e4144a50 legacy=0x7fc5e41238f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:50:51.019 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:51.008+0000 7fc5e8a48700 1 -- 172.21.15.42:0/1910128087 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5e412a5b0 con 0x7fc5e40a1fe0 2019-12-05T05:50:51.019 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:51.008+0000 7fc5e8a48700 1 --2- 172.21.15.42:0/1910128087 >> v2:172.21.15.42:3300/0 conn(0x7fc5e40a1fe0 0x7fc5e4064970 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fc5e40a9180 tx=0x7fc5d8008e30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:51.019 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:51.008+0000 7fc5e8a48700 1 --2- 172.21.15.42:0/1910128087 >> v2:172.21.15.42:3300/0 conn(0x7fc5e40a1fe0 0x7fc5e4064970 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fc5e40a9180 tx=0x7fc5d8008e30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:51.019 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:51.008+0000 7fc5e8a48700 1 --2- 172.21.15.42:0/1910128087 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc5e40a1fe0 0x7fc5e4064970 secure :-1 s=READY pgs=1555 cs=0 l=1 rx=0x7fc5e40a9180 tx=0x7fc5d8008e30).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:50:51.019 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:51.008+0000 7fc5e8a48700 1 --2- 172.21.15.42:0/1910128087 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc5e40a1fe0 0x7fc5e4064970 secure :-1 s=THROTTLE_DONE pgs=1555 cs=0 l=1 rx=0x7fc5e40a9180 tx=0x7fc5d8008e30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:51.020 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:51.008+0000 7fc5d5ffb700 1 -- 172.21.15.42:0/1910128087 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc5d800f680 con 0x7fc5e40a1fe0 2019-12-05T05:50:51.020 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:51.008+0000 7fc5e8a48700 1 --2- 172.21.15.42:0/1910128087 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc5e40a1fe0 0x7fc5e4064970 secure :-1 s=THROTTLE_DONE pgs=1555 cs=0 l=1 rx=0x7fc5e40a9180 tx=0x7fc5d8008e30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:51.020 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:51.008+0000 7fc5d5ffb700 1 -- 172.21.15.42:0/1910128087 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5d80116f0 con 0x7fc5e40a1fe0 2019-12-05T05:50:51.020 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:51.008+0000 7fc5e8a48700 1 --2- 172.21.15.42:0/1910128087 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc5e40a1fe0 0x7fc5e4064970 secure :-1 s=THROTTLE_DONE pgs=1555 cs=0 l=1 rx=0x7fc5e40a9180 tx=0x7fc5d8008e30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:51.020 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:51.008+0000 7fc5eacac700 1 -- 172.21.15.42:0/1910128087 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5e4131080 con 0x7fc5e40a1fe0 2019-12-05T05:50:51.020 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:51.008+0000 7fc5d5ffb700 1 -- 172.21.15.42:0/1910128087 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc5d8005920 con 0x7fc5e40a1fe0 2019-12-05T05:50:51.020 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:51.008+0000 7fc5eacac700 1 -- 172.21.15.42:0/1910128087 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5e41315d0 con 0x7fc5e40a1fe0 2019-12-05T05:50:51.021 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:51.008+0000 7fc5e8a48700 1 --2- 172.21.15.42:0/1910128087 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc5e40a1fe0 0x7fc5e4064970 secure :-1 s=THROTTLE_DONE pgs=1555 cs=0 l=1 rx=0x7fc5e40a9180 tx=0x7fc5d8008e30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:51.021 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:51.008+0000 7fc5d5ffb700 1 -- 172.21.15.42:0/1910128087 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fc5d8005bc0 con 0x7fc5e40a1fe0 2019-12-05T05:50:51.021 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:51.008+0000 7fc5e8a48700 1 --2- 172.21.15.42:0/1910128087 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc5e40a1fe0 0x7fc5e4064970 secure :-1 s=THROTTLE_DONE pgs=1555 cs=0 l=1 rx=0x7fc5e40a9180 tx=0x7fc5d8008e30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:51.021 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:51.008+0000 7fc5d5ffb700 1 --2- 172.21.15.42:0/1910128087 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc5c403aca0 0x7fc5c403d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:50:51.021 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:51.008+0000 7fc5d5ffb700 1 -- 172.21.15.42:0/1910128087 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8783+0+0 (secure 0 0 0) 0x7fc5d804bb40 con 0x7fc5e40a1fe0 2019-12-05T05:50:51.021 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:51.012+0000 7fc5e9249700 1 --2- 172.21.15.42:0/1910128087 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc5c403aca0 0x7fc5c403d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:50:51.021 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:51.012+0000 7fc5e9249700 1 --2- 172.21.15.42:0/1910128087 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc5c403aca0 0x7fc5c403d1b0 crc :-1 s=READY pgs=670 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:50:51.022 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:51.012+0000 7fc5eacac700 1 -- 172.21.15.42:0/1910128087 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5e405d110 con 0x7fc5e40a1fe0 2019-12-05T05:50:51.025 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:51.020+0000 7fc5e8a48700 1 --2- 172.21.15.42:0/1910128087 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc5e40a1fe0 0x7fc5e4064970 secure :-1 s=THROTTLE_DONE pgs=1555 cs=0 l=1 rx=0x7fc5e40a9180 tx=0x7fc5d8008e30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:51.025 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:51.020+0000 7fc5d5ffb700 1 -- 172.21.15.42:0/1910128087 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fc5d8032ba0 con 0x7fc5e40a1fe0 2019-12-05T05:50:51.186 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:51.180+0000 7fc5eacac700 1 -- 172.21.15.42:0/1910128087 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc5e405b8c0 con 0x7fc5e40a1fe0 2019-12-05T05:50:51.187 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:51.180+0000 7fc5e8a48700 1 --2- 172.21.15.42:0/1910128087 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc5e40a1fe0 0x7fc5e4064970 secure :-1 s=THROTTLE_DONE pgs=1555 cs=0 l=1 rx=0x7fc5e40a9180 tx=0x7fc5d8008e30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:51.187 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:51.180+0000 7fc5d5ffb700 1 -- 172.21.15.42:0/1910128087 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 189 v189) v1 ==== 95+0+1774 (secure 0 0 0) 0x7fc5d8007160 con 0x7fc5e40a1fe0 2019-12-05T05:50:51.187 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 189 2019-12-05T05:50:51.187 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:50:51.188 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":189,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":189,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:50:49.116852+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6245},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6245":{"gid":6245,"name":"b","rank":0,"incarnation":189,"state":"up:replay","state_seq":1,"addr":"172.21.15.40:6833/2081111170","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2081111170},{"type":"v1","addr":"172.21.15.40:6833","nonce":2081111170}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:50:51.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:51.184+0000 7fc5eacac700 1 -- 172.21.15.42:0/1910128087 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc5c403aca0 msgr2=0x7fc5c403d1b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:50:51.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:51.184+0000 7fc5eacac700 1 --2- 172.21.15.42:0/1910128087 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc5c403aca0 0x7fc5c403d1b0 crc :-1 s=READY pgs=670 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:50:51.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:51.184+0000 7fc5eacac700 1 -- 172.21.15.42:0/1910128087 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc5e40a1fe0 msgr2=0x7fc5e4064970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:50:51.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:51.184+0000 7fc5eacac700 1 --2- 172.21.15.42:0/1910128087 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc5e40a1fe0 0x7fc5e4064970 secure :-1 s=READY pgs=1555 cs=0 l=1 rx=0x7fc5e40a9180 tx=0x7fc5d8008e30).stop 2019-12-05T05:50:51.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:51.184+0000 7fc5eacac700 1 -- 172.21.15.42:0/1910128087 shutdown_connections 2019-12-05T05:50:51.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:51.184+0000 7fc5eacac700 1 --2- 172.21.15.42:0/1910128087 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc5c403aca0 0x7fc5c403d1b0 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:50:51.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:51.184+0000 7fc5eacac700 1 --2- 172.21.15.42:0/1910128087 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc5e4129c30 0x7fc5e412a060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:50:51.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:51.184+0000 7fc5eacac700 1 --2- 172.21.15.42:0/1910128087 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc5e40a1fe0 0x7fc5e4064970 unknown :-1 s=CLOSED pgs=1555 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:50:51.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:51.184+0000 7fc5eacac700 1 -- 172.21.15.42:0/1910128087 shutdown_connections 2019-12-05T05:50:51.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:51.184+0000 7fc5eacac700 1 -- 172.21.15.42:0/1910128087 wait complete. 2019-12-05T05:50:51.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:51.184+0000 7fc5eacac700 1 -- 172.21.15.42:0/1910128087 >> 172.21.15.42:0/1910128087 conn(0x7fc5e40a8120 msgr2=0x7fc5e40a6a60 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:50:51.200 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 189, "feature_flags": { "enable_multiple": false, "ever_enabled_multiple": false }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2019-12-05T05:03:35.604738+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 189, "ever_allowed_features": 2, "explicitly_allowed_features": 2, "failed": [], "flags": 18, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_6245": { "addr": "172.21.15.40:6833/2081111170", "addrs": { "addrvec": [ { "addr": "172.21.15.40:6832", "nonce": 2081111170, "type": "v2" }, { "addr": "172.21.15.40:6833", "nonce": 2081111170, "type": "v1" } ] }, "export_targets": [], "features": 4540138292836696063, "flags": 0, "gid": 6245, "incarnation": 189, "name": "b", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 81, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 2, "min_compat_client": "0 (unknown)", "modified": "2019-12-05T05:50:49.116852+0000", "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6245 } } } ], "standbys": [] } 2019-12-05T05:50:53.201 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:50:53.312 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:53.300+0000 7f4e0f06b700 1 Processor -- start 2019-12-05T05:50:53.312 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:53.300+0000 7f4e0f06b700 1 -- start start 2019-12-05T05:50:53.312 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:53.300+0000 7f4e0f06b700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f4e080a7740 0x7f4e0808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:50:53.312 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:53.300+0000 7f4e0f06b700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4e0807e720 0x7f4e0807eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:50:53.312 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:53.300+0000 7f4e0f06b700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f4e0809f2e0 0x7f4e0809f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:50:53.313 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:53.300+0000 7f4e0f06b700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e080a0b00 con 0x7f4e0809f2e0 2019-12-05T05:50:53.313 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:53.300+0000 7f4e0f06b700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e0808e5b0 con 0x7f4e080a7740 2019-12-05T05:50:53.313 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:53.300+0000 7f4e0f06b700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e0808e850 con 0x7f4e0807e720 2019-12-05T05:50:53.313 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:53.300+0000 7f4e0d608700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f4e080a7740 0x7f4e0808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:50:53.313 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:53.300+0000 7f4e0d608700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f4e080a7740 0x7f4e0808df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:42374/0 (socket says 172.21.15.42:42374) 2019-12-05T05:50:53.313 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:53.304+0000 7f4e0d608700 1 -- 172.21.15.42:0/3868152669 learned_addr learned my addr 172.21.15.42:0/3868152669 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:50:53.313 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:53.304+0000 7f4e0ce07700 1 --2- 172.21.15.42:0/3868152669 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4e0807e720 0x7f4e0807eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:50:53.314 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:53.304+0000 7f4e0de09700 1 --2- 172.21.15.42:0/3868152669 >> v2:172.21.15.40:3300/0 conn(0x7f4e0809f2e0 0x7f4e0809f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:50:53.314 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:53.304+0000 7f4e0d608700 1 -- 172.21.15.42:0/3868152669 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4e0807e720 msgr2=0x7f4e0807eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:50:53.314 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:53.304+0000 7f4e0d608700 1 --2- 172.21.15.42:0/3868152669 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4e0807e720 0x7f4e0807eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:50:53.314 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:53.304+0000 7f4e0d608700 1 -- 172.21.15.42:0/3868152669 >> v2:172.21.15.40:3300/0 conn(0x7f4e0809f2e0 msgr2=0x7f4e0809f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:50:53.314 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:53.304+0000 7f4e0d608700 1 --2- 172.21.15.42:0/3868152669 >> v2:172.21.15.40:3300/0 conn(0x7f4e0809f2e0 0x7f4e0809f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:50:53.314 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:53.304+0000 7f4e0d608700 1 -- 172.21.15.42:0/3868152669 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e0807f0a0 con 0x7f4e080a7740 2019-12-05T05:50:53.314 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:53.304+0000 7f4e0de09700 1 --2- 172.21.15.42:0/3868152669 >> v2:172.21.15.40:3300/0 conn(0x7f4e0809f2e0 0x7f4e0809f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T05:50:53.315 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:53.304+0000 7f4e0d608700 1 --2- 172.21.15.42:0/3868152669 >> v2:172.21.15.42:3300/0 conn(0x7f4e080a7740 0x7f4e0808df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f4e0808e770 tx=0x7f4df8010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:53.315 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:53.304+0000 7f4e0d608700 1 --2- 172.21.15.42:0/3868152669 >> v2:172.21.15.42:3300/0 conn(0x7f4e080a7740 0x7f4e0808df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f4e0808e770 tx=0x7f4df8010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:53.315 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:53.304+0000 7f4e0d608700 1 --2- 172.21.15.42:0/3868152669 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4e080a7740 0x7f4e0808df80 secure :-1 s=READY pgs=1556 cs=0 l=1 rx=0x7f4e0808e770 tx=0x7f4df8010130).ready entity=mon.1 client_cookie=32be26fbf8eaf031 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:50:53.315 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:53.304+0000 7f4e0d608700 1 --2- 172.21.15.42:0/3868152669 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4e080a7740 0x7f4e0808df80 secure :-1 s=THROTTLE_DONE pgs=1556 cs=0 l=1 rx=0x7f4e0808e770 tx=0x7f4df8010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:53.315 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:53.304+0000 7f4e07fff700 1 -- 172.21.15.42:0/3868152669 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4df8010de0 con 0x7f4e080a7740 2019-12-05T05:50:53.315 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:53.304+0000 7f4e0d608700 1 --2- 172.21.15.42:0/3868152669 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4e080a7740 0x7f4e0808df80 secure :-1 s=THROTTLE_DONE pgs=1556 cs=0 l=1 rx=0x7f4e0808e770 tx=0x7f4df8010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:53.316 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:53.304+0000 7f4e0d608700 1 --2- 172.21.15.42:0/3868152669 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4e080a7740 0x7f4e0808df80 secure :-1 s=THROTTLE_DONE pgs=1556 cs=0 l=1 rx=0x7f4e0808e770 tx=0x7f4df8010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:53.316 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:53.304+0000 7f4e07fff700 1 -- 172.21.15.42:0/3868152669 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4df8010a00 con 0x7f4e080a7740 2019-12-05T05:50:53.316 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:53.304+0000 7f4e07fff700 1 -- 172.21.15.42:0/3868152669 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4df80114a0 con 0x7f4e080a7740 2019-12-05T05:50:53.316 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:53.304+0000 7f4e0f06b700 1 -- 172.21.15.42:0/3868152669 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4e080a7740 msgr2=0x7f4e0808df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:50:53.316 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:53.304+0000 7f4e0f06b700 1 --2- 172.21.15.42:0/3868152669 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4e080a7740 0x7f4e0808df80 secure :-1 s=READY pgs=1556 cs=0 l=1 rx=0x7f4e0808e770 tx=0x7f4df8010130).stop 2019-12-05T05:50:53.317 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:53.304+0000 7f4e0f06b700 1 -- 172.21.15.42:0/3868152669 shutdown_connections 2019-12-05T05:50:53.317 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:53.304+0000 7f4e0f06b700 1 --2- 172.21.15.42:0/3868152669 >> v2:172.21.15.40:3300/0 conn(0x7f4e0809f2e0 0x7f4e0809f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:50:53.317 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:53.304+0000 7f4e0f06b700 1 --2- 172.21.15.42:0/3868152669 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4e080a7740 0x7f4e0808df80 unknown :-1 s=CLOSED pgs=1556 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:50:53.317 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:53.304+0000 7f4e0f06b700 1 --2- 172.21.15.42:0/3868152669 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4e0807e720 0x7f4e0807eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:50:53.317 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:53.304+0000 7f4e0f06b700 1 -- 172.21.15.42:0/3868152669 shutdown_connections 2019-12-05T05:50:53.317 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:53.304+0000 7f4e0f06b700 1 -- 172.21.15.42:0/3868152669 wait complete. 2019-12-05T05:50:53.317 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:53.304+0000 7f4e0f06b700 1 -- 172.21.15.42:0/3868152669 >> 172.21.15.42:0/3868152669 conn(0x7f4e080aabe0 msgr2=0x7f4e080a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:50:53.321 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:53.308+0000 7f4e0f06b700 1 Processor -- start 2019-12-05T05:50:53.321 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:53.308+0000 7f4e0f06b700 1 -- start start 2019-12-05T05:50:53.321 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:53.308+0000 7f4e0f06b700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4e08064bd0 0x7f4e08142530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:50:53.322 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:53.308+0000 7f4e0f06b700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f4e0811a550 0x7f4e0811a980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:50:53.322 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:53.308+0000 7f4e0f06b700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e080640b0 con 0x7f4e0808ed10 2019-12-05T05:50:53.322 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:53.308+0000 7f4e0f06b700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e0808f140 con 0x7f4e0811a550 2019-12-05T05:50:53.322 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:53.308+0000 7f4e0f06b700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e08142a80 con 0x7f4e08064bd0 2019-12-05T05:50:53.323 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:53.308+0000 7f4e0d608700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f4e0808ed10 0x7f4e080643d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:42416/0 (socket says 172.21.15.42:42416) 2019-12-05T05:50:53.323 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:53.308+0000 7f4e0d608700 1 -- 172.21.15.42:0/3729967416 learned_addr learned my addr 172.21.15.42:0/3729967416 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:50:53.323 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:53.308+0000 7f4e0de09700 1 --2- 172.21.15.42:0/3729967416 >> v2:172.21.15.40:3300/0 conn(0x7f4e0811a550 0x7f4e0811a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:50:53.323 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:53.308+0000 7f4e0ce07700 1 --2- 172.21.15.42:0/3729967416 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4e08064bd0 0x7f4e08142530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:50:53.324 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:53.312+0000 7f4e05ffb700 1 -- 172.21.15.42:0/3729967416 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2072140394 0 0) 0x7f4e080640b0 con 0x7f4e0808ed10 2019-12-05T05:50:53.324 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:53.312+0000 7f4e05ffb700 1 -- 172.21.15.42:0/3729967416 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4de80030b0 con 0x7f4e0808ed10 2019-12-05T05:50:53.324 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:53.312+0000 7f4e0ce07700 1 -- 172.21.15.42:0/3729967416 >> v2:172.21.15.40:3300/0 conn(0x7f4e0811a550 msgr2=0x7f4e0811a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:50:53.324 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:53.312+0000 7f4e0ce07700 1 --2- 172.21.15.42:0/3729967416 >> v2:172.21.15.40:3300/0 conn(0x7f4e0811a550 0x7f4e0811a980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:50:53.324 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:53.312+0000 7f4e0ce07700 1 -- 172.21.15.42:0/3729967416 >> v1:172.21.15.42:6789/0 conn(0x7f4e0808ed10 legacy=0x7f4e080643d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:50:53.324 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:53.312+0000 7f4e0ce07700 1 -- 172.21.15.42:0/3729967416 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e08063190 con 0x7f4e08064bd0 2019-12-05T05:50:53.325 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:53.312+0000 7f4e0ce07700 1 --2- 172.21.15.42:0/3729967416 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4e08064bd0 0x7f4e08142530 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f4e08145680 tx=0x7f4dfc010b20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:53.325 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:53.312+0000 7f4e0ce07700 1 --2- 172.21.15.42:0/3729967416 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4e08064bd0 0x7f4e08142530 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f4e08145680 tx=0x7f4dfc010b20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:53.325 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:53.312+0000 7f4e0ce07700 1 --2- 172.21.15.42:0/3729967416 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4e08064bd0 0x7f4e08142530 secure :-1 s=READY pgs=347 cs=0 l=1 rx=0x7f4e08145680 tx=0x7f4dfc010b20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:50:53.325 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:53.312+0000 7f4e0ce07700 1 --2- 172.21.15.42:0/3729967416 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4e08064bd0 0x7f4e08142530 secure :-1 s=THROTTLE_DONE pgs=347 cs=0 l=1 rx=0x7f4e08145680 tx=0x7f4dfc010b20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:53.325 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:53.312+0000 7f4e05ffb700 1 -- 172.21.15.42:0/3729967416 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4dfc012970 con 0x7f4e08064bd0 2019-12-05T05:50:53.325 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:53.312+0000 7f4e0ce07700 1 --2- 172.21.15.42:0/3729967416 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4e08064bd0 0x7f4e08142530 secure :-1 s=THROTTLE_DONE pgs=347 cs=0 l=1 rx=0x7f4e08145680 tx=0x7f4dfc010b20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:53.325 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:53.312+0000 7f4e0ce07700 1 --2- 172.21.15.42:0/3729967416 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4e08064bd0 0x7f4e08142530 secure :-1 s=THROTTLE_DONE pgs=347 cs=0 l=1 rx=0x7f4e08145680 tx=0x7f4dfc010b20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:53.326 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:53.312+0000 7f4e05ffb700 1 -- 172.21.15.42:0/3729967416 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dfc012590 con 0x7f4e08064bd0 2019-12-05T05:50:53.326 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:53.312+0000 7f4e0f06b700 1 -- 172.21.15.42:0/3729967416 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e0811aed0 con 0x7f4e08064bd0 2019-12-05T05:50:53.326 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:53.312+0000 7f4e05ffb700 1 -- 172.21.15.42:0/3729967416 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4dfc005730 con 0x7f4e08064bd0 2019-12-05T05:50:53.326 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:53.312+0000 7f4e0f06b700 1 -- 172.21.15.42:0/3729967416 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e0811b420 con 0x7f4e08064bd0 2019-12-05T05:50:53.326 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:53.312+0000 7f4e0ce07700 1 --2- 172.21.15.42:0/3729967416 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4e08064bd0 0x7f4e08142530 secure :-1 s=THROTTLE_DONE pgs=347 cs=0 l=1 rx=0x7f4e08145680 tx=0x7f4dfc010b20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:53.326 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:53.312+0000 7f4e05ffb700 1 -- 172.21.15.42:0/3729967416 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f4dfc007180 con 0x7f4e08064bd0 2019-12-05T05:50:53.326 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:53.312+0000 7f4e0ce07700 1 --2- 172.21.15.42:0/3729967416 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4e08064bd0 0x7f4e08142530 secure :-1 s=THROTTLE_DONE pgs=347 cs=0 l=1 rx=0x7f4e08145680 tx=0x7f4dfc010b20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:53.327 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:53.316+0000 7f4e05ffb700 1 --2- 172.21.15.42:0/3729967416 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4de803b230 0x7f4de803d740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:50:53.327 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:53.316+0000 7f4e05ffb700 1 -- 172.21.15.42:0/3729967416 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8783+0+0 (secure 0 0 0) 0x7f4dfc04af30 con 0x7f4e08064bd0 2019-12-05T05:50:53.327 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:53.316+0000 7f4e0d608700 1 --2- 172.21.15.42:0/3729967416 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4de803b230 0x7f4de803d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:50:53.327 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:53.316+0000 7f4e0d608700 1 --2- 172.21.15.42:0/3729967416 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4de803b230 0x7f4de803d740 crc :-1 s=READY pgs=671 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:50:53.327 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:53.316+0000 7f4e0f06b700 1 -- 172.21.15.42:0/3729967416 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e0805d110 con 0x7f4e08064bd0 2019-12-05T05:50:53.331 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:53.324+0000 7f4e0ce07700 1 --2- 172.21.15.42:0/3729967416 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4e08064bd0 0x7f4e08142530 secure :-1 s=THROTTLE_DONE pgs=347 cs=0 l=1 rx=0x7f4e08145680 tx=0x7f4dfc010b20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:53.331 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:53.324+0000 7f4e05ffb700 1 -- 172.21.15.42:0/3729967416 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f4dfc031fb0 con 0x7f4e08064bd0 2019-12-05T05:50:53.498 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:53.492+0000 7f4e0f06b700 1 -- 172.21.15.42:0/3729967416 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4e0805b8c0 con 0x7f4e08064bd0 2019-12-05T05:50:53.499 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:53.492+0000 7f4e0ce07700 1 --2- 172.21.15.42:0/3729967416 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4e08064bd0 0x7f4e08142530 secure :-1 s=THROTTLE_DONE pgs=347 cs=0 l=1 rx=0x7f4e08145680 tx=0x7f4dfc010b20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:53.499 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:53.492+0000 7f4e05ffb700 1 -- 172.21.15.42:0/3729967416 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 190 v190) v1 ==== 95+0+1778 (secure 0 0 0) 0x7f4dfc032210 con 0x7f4e08064bd0 2019-12-05T05:50:53.501 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:50:53.501 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":190,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":190,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:50:52.551183+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6245},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6245":{"gid":6245,"name":"b","rank":0,"incarnation":189,"state":"up:reconnect","state_seq":23,"addr":"172.21.15.40:6833/2081111170","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2081111170},{"type":"v1","addr":"172.21.15.40:6833","nonce":2081111170}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:50:53.501 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 190 2019-12-05T05:50:53.503 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:53.496+0000 7f4e0f06b700 1 -- 172.21.15.42:0/3729967416 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4de803b230 msgr2=0x7f4de803d740 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:50:53.504 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:53.496+0000 7f4e0f06b700 1 --2- 172.21.15.42:0/3729967416 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4de803b230 0x7f4de803d740 crc :-1 s=READY pgs=671 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:50:53.504 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:53.496+0000 7f4e0f06b700 1 -- 172.21.15.42:0/3729967416 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4e08064bd0 msgr2=0x7f4e08142530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:50:53.504 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:53.496+0000 7f4e0f06b700 1 --2- 172.21.15.42:0/3729967416 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4e08064bd0 0x7f4e08142530 secure :-1 s=READY pgs=347 cs=0 l=1 rx=0x7f4e08145680 tx=0x7f4dfc010b20).stop 2019-12-05T05:50:53.504 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:53.500+0000 7f4e0f06b700 1 -- 172.21.15.42:0/3729967416 shutdown_connections 2019-12-05T05:50:53.504 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:53.500+0000 7f4e0f06b700 1 --2- 172.21.15.42:0/3729967416 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4de803b230 0x7f4de803d740 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:50:53.504 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:53.500+0000 7f4e0f06b700 1 --2- 172.21.15.42:0/3729967416 >> v2:172.21.15.40:3300/0 conn(0x7f4e0811a550 0x7f4e0811a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:50:53.504 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:53.500+0000 7f4e0f06b700 1 --2- 172.21.15.42:0/3729967416 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4e08064bd0 0x7f4e08142530 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:50:53.505 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:53.500+0000 7f4e0f06b700 1 -- 172.21.15.42:0/3729967416 shutdown_connections 2019-12-05T05:50:53.505 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:53.500+0000 7f4e0f06b700 1 -- 172.21.15.42:0/3729967416 wait complete. 2019-12-05T05:50:53.505 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:53.500+0000 7f4e0f06b700 1 -- 172.21.15.42:0/3729967416 >> 172.21.15.42:0/3729967416 conn(0x7f4e080a6bb0 msgr2=0x7f4e0811c360 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:50:53.513 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2019-12-05T05:50:55.514 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:50:55.629 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:55.620+0000 7f9477c4a700 1 Processor -- start 2019-12-05T05:50:55.630 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:55.620+0000 7f9477c4a700 1 -- start start 2019-12-05T05:50:55.630 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:55.620+0000 7f9477c4a700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f947008df80 0x7f947009f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:50:55.630 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:55.620+0000 7f9477c4a700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f947009f830 0x7f947014efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:50:55.630 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:55.620+0000 7f9477c4a700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f947009fc60 con 0x7f94700a7740 2019-12-05T05:50:55.630 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:55.620+0000 7f9477c4a700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94700a0b00 con 0x7f947008df80 2019-12-05T05:50:55.631 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:55.620+0000 7f9477c4a700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f947008e5b0 con 0x7f947009f830 2019-12-05T05:50:55.631 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:55.620+0000 7f94761e7700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f94700a7740 0x7f947007e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:42438/0 (socket says 172.21.15.42:42438) 2019-12-05T05:50:55.631 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:55.620+0000 7f94761e7700 1 -- 172.21.15.42:0/2845225670 learned_addr learned my addr 172.21.15.42:0/2845225670 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:50:55.632 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:55.620+0000 7f94769e8700 1 --2- 172.21.15.42:0/2845225670 >> v2:172.21.15.42:3300/0 conn(0x7f947009f830 0x7f947014efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:50:55.632 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:55.620+0000 7f94759e6700 1 --2- 172.21.15.42:0/2845225670 >> v2:172.21.15.40:3300/0 conn(0x7f947008df80 0x7f947009f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:50:55.632 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:55.620+0000 7f94751e5700 1 -- 172.21.15.42:0/2845225670 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2573326654 0 0) 0x7f947009fc60 con 0x7f94700a7740 2019-12-05T05:50:55.632 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:55.620+0000 7f94751e5700 1 -- 172.21.15.42:0/2845225670 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f945c0030b0 con 0x7f94700a7740 2019-12-05T05:50:55.632 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:55.620+0000 7f94751e5700 1 -- 172.21.15.42:0/2845225670 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1950613885 0 0) 0x7f945c0030b0 con 0x7f94700a7740 2019-12-05T05:50:55.633 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:55.620+0000 7f94769e8700 1 -- 172.21.15.42:0/2845225670 >> v2:172.21.15.40:3300/0 conn(0x7f947008df80 msgr2=0x7f947009f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:50:55.633 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:55.620+0000 7f94769e8700 1 --2- 172.21.15.42:0/2845225670 >> v2:172.21.15.40:3300/0 conn(0x7f947008df80 0x7f947009f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:50:55.633 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:55.620+0000 7f94769e8700 1 -- 172.21.15.42:0/2845225670 >> v1:172.21.15.42:6789/0 conn(0x7f94700a7740 legacy=0x7f947007e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:50:55.633 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:55.620+0000 7f94769e8700 1 -- 172.21.15.42:0/2845225670 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94700a0710 con 0x7f947009f830 2019-12-05T05:50:55.634 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:55.620+0000 7f94769e8700 1 --2- 172.21.15.42:0/2845225670 >> v2:172.21.15.42:3300/0 conn(0x7f947009f830 0x7f947014efe0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f946c001990 tx=0x7f946c0019d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:55.634 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:55.620+0000 7f94769e8700 1 --2- 172.21.15.42:0/2845225670 >> v2:172.21.15.42:3300/0 conn(0x7f947009f830 0x7f947014efe0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f946c001990 tx=0x7f946c0019d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:55.634 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:55.620+0000 7f94769e8700 1 --2- 172.21.15.42:0/2845225670 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f947009f830 0x7f947014efe0 secure :-1 s=READY pgs=1559 cs=0 l=1 rx=0x7f946c001990 tx=0x7f946c0019d0).ready entity=mon.1 client_cookie=c9eed31f951750d server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:50:55.635 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:55.620+0000 7f94769e8700 1 --2- 172.21.15.42:0/2845225670 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f947009f830 0x7f947014efe0 secure :-1 s=THROTTLE_DONE pgs=1559 cs=0 l=1 rx=0x7f946c001990 tx=0x7f946c0019d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:55.635 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:55.620+0000 7f94751e5700 1 -- 172.21.15.42:0/2845225670 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f946c001750 con 0x7f947009f830 2019-12-05T05:50:55.635 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:55.620+0000 7f94769e8700 1 --2- 172.21.15.42:0/2845225670 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f947009f830 0x7f947014efe0 secure :-1 s=THROTTLE_DONE pgs=1559 cs=0 l=1 rx=0x7f946c001990 tx=0x7f946c0019d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:55.635 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:55.620+0000 7f94751e5700 1 -- 172.21.15.42:0/2845225670 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f946c001560 con 0x7f947009f830 2019-12-05T05:50:55.635 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:55.620+0000 7f94769e8700 1 --2- 172.21.15.42:0/2845225670 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f947009f830 0x7f947014efe0 secure :-1 s=THROTTLE_DONE pgs=1559 cs=0 l=1 rx=0x7f946c001990 tx=0x7f946c0019d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:55.635 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:55.620+0000 7f9477c4a700 1 -- 172.21.15.42:0/2845225670 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f947009f830 msgr2=0x7f947014efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:50:55.635 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:55.620+0000 7f94751e5700 1 -- 172.21.15.42:0/2845225670 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9468010ce0 con 0x7f947009f830 2019-12-05T05:50:55.636 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:55.620+0000 7f9477c4a700 1 --2- 172.21.15.42:0/2845225670 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f947009f830 0x7f947014efe0 secure :-1 s=READY pgs=1559 cs=0 l=1 rx=0x7f946c001990 tx=0x7f946c0019d0).stop 2019-12-05T05:50:55.636 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:55.624+0000 7f9477c4a700 1 -- 172.21.15.42:0/2845225670 shutdown_connections 2019-12-05T05:50:55.636 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:55.624+0000 7f9477c4a700 1 --2- 172.21.15.42:0/2845225670 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f947009f830 0x7f947014efe0 unknown :-1 s=CLOSED pgs=1559 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:50:55.636 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:55.624+0000 7f9477c4a700 1 --2- 172.21.15.42:0/2845225670 >> v2:172.21.15.40:3300/0 conn(0x7f947008df80 0x7f947009f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:50:55.636 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:55.624+0000 7f9477c4a700 1 -- 172.21.15.42:0/2845225670 shutdown_connections 2019-12-05T05:50:55.636 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:55.624+0000 7f9477c4a700 1 -- 172.21.15.42:0/2845225670 wait complete. 2019-12-05T05:50:55.636 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:55.624+0000 7f9477c4a700 1 -- 172.21.15.42:0/2845225670 >> 172.21.15.42:0/2845225670 conn(0x7f94700aabe0 msgr2=0x7f94700a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:50:55.641 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:55.624+0000 7f9477c4a700 1 Processor -- start 2019-12-05T05:50:55.641 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:55.624+0000 7f9477c4a700 1 -- start start 2019-12-05T05:50:55.641 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:55.628+0000 7f9477c4a700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f94701201d0 0x7f947012d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:50:55.641 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:55.628+0000 7f9477c4a700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f947011bd40 0x7f9470117b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:50:55.641 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:55.628+0000 7f9477c4a700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94701183b0 con 0x7f94701176f0 2019-12-05T05:50:55.642 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:55.628+0000 7f9477c4a700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9470118070 con 0x7f947011bd40 2019-12-05T05:50:55.643 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:55.628+0000 7f9477c4a700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f947011c170 con 0x7f94701201d0 2019-12-05T05:50:55.643 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:55.628+0000 7f94759e6700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f94701176f0 0x7f947011b540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:42444/0 (socket says 172.21.15.42:42444) 2019-12-05T05:50:55.643 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:55.628+0000 7f94759e6700 1 -- 172.21.15.42:0/1627685965 learned_addr learned my addr 172.21.15.42:0/1627685965 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:50:55.643 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:55.628+0000 7f94761e7700 1 --2- 172.21.15.42:0/1627685965 >> v2:172.21.15.42:3300/0 conn(0x7f94701201d0 0x7f947012d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:50:55.644 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:55.628+0000 7f94769e8700 1 --2- 172.21.15.42:0/1627685965 >> v2:172.21.15.40:3300/0 conn(0x7f947011bd40 0x7f9470117b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:50:55.644 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:55.628+0000 7f94761e7700 1 -- 172.21.15.42:0/1627685965 >> v2:172.21.15.40:3300/0 conn(0x7f947011bd40 msgr2=0x7f9470117b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:50:55.644 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:55.628+0000 7f94761e7700 1 --2- 172.21.15.42:0/1627685965 >> v2:172.21.15.40:3300/0 conn(0x7f947011bd40 0x7f9470117b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:50:55.644 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:55.628+0000 7f94761e7700 1 -- 172.21.15.42:0/1627685965 >> v1:172.21.15.42:6789/0 conn(0x7f94701176f0 legacy=0x7f947011b540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:50:55.645 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:55.628+0000 7f9466ffd700 1 -- 172.21.15.42:0/1627685965 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2675243903 0 0) 0x7f94701183b0 con 0x7f94701176f0 2019-12-05T05:50:55.645 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:55.628+0000 7f94761e7700 1 -- 172.21.15.42:0/1627685965 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94701172d0 con 0x7f94701201d0 2019-12-05T05:50:55.645 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:55.628+0000 7f94761e7700 1 --2- 172.21.15.42:0/1627685965 >> v2:172.21.15.42:3300/0 conn(0x7f94701201d0 0x7f947012d7a0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f946c001a50 tx=0x7f946c0109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:55.645 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:55.628+0000 7f94761e7700 1 --2- 172.21.15.42:0/1627685965 >> v2:172.21.15.42:3300/0 conn(0x7f94701201d0 0x7f947012d7a0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f946c001a50 tx=0x7f946c0109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:55.645 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:55.628+0000 7f94761e7700 1 --2- 172.21.15.42:0/1627685965 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f94701201d0 0x7f947012d7a0 secure :-1 s=READY pgs=1561 cs=0 l=1 rx=0x7f946c001a50 tx=0x7f946c0109e0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:50:55.645 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:55.628+0000 7f94761e7700 1 --2- 172.21.15.42:0/1627685965 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f94701201d0 0x7f947012d7a0 secure :-1 s=THROTTLE_DONE pgs=1561 cs=0 l=1 rx=0x7f946c001a50 tx=0x7f946c0109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:55.646 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:55.628+0000 7f9466ffd700 1 -- 172.21.15.42:0/1627685965 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f946c00c890 con 0x7f94701201d0 2019-12-05T05:50:55.646 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:55.628+0000 7f94761e7700 1 --2- 172.21.15.42:0/1627685965 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f94701201d0 0x7f947012d7a0 secure :-1 s=THROTTLE_DONE pgs=1561 cs=0 l=1 rx=0x7f946c001a50 tx=0x7f946c0109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:55.646 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:55.628+0000 7f94761e7700 1 --2- 172.21.15.42:0/1627685965 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f94701201d0 0x7f947012d7a0 secure :-1 s=THROTTLE_DONE pgs=1561 cs=0 l=1 rx=0x7f946c001a50 tx=0x7f946c0109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:55.647 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:55.628+0000 7f9466ffd700 1 -- 172.21.15.42:0/1627685965 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f946c00c4b0 con 0x7f94701201d0 2019-12-05T05:50:55.647 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:55.628+0000 7f9477c4a700 1 -- 172.21.15.42:0/1627685965 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f947012dcf0 con 0x7f94701201d0 2019-12-05T05:50:55.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:55.628+0000 7f9466ffd700 1 -- 172.21.15.42:0/1627685965 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f946c012450 con 0x7f94701201d0 2019-12-05T05:50:55.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:55.628+0000 7f9477c4a700 1 -- 172.21.15.42:0/1627685965 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9470134640 con 0x7f94701201d0 2019-12-05T05:50:55.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:55.628+0000 7f94761e7700 1 --2- 172.21.15.42:0/1627685965 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f94701201d0 0x7f947012d7a0 secure :-1 s=THROTTLE_DONE pgs=1561 cs=0 l=1 rx=0x7f946c001a50 tx=0x7f946c0109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:55.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:55.632+0000 7f9466ffd700 1 -- 172.21.15.42:0/1627685965 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f946c012760 con 0x7f94701201d0 2019-12-05T05:50:55.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:55.632+0000 7f94761e7700 1 --2- 172.21.15.42:0/1627685965 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f94701201d0 0x7f947012d7a0 secure :-1 s=THROTTLE_DONE pgs=1561 cs=0 l=1 rx=0x7f946c001a50 tx=0x7f946c0109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:55.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:55.632+0000 7f9466ffd700 1 --2- 172.21.15.42:0/1627685965 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f945803ad00 0x7f945803d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:50:55.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:55.632+0000 7f9466ffd700 1 -- 172.21.15.42:0/1627685965 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8783+0+0 (secure 0 0 0) 0x7f946c04ac70 con 0x7f94701201d0 2019-12-05T05:50:55.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:55.632+0000 7f94759e6700 1 --2- 172.21.15.42:0/1627685965 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f945803ad00 0x7f945803d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:50:55.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:55.632+0000 7f94759e6700 1 --2- 172.21.15.42:0/1627685965 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f945803ad00 0x7f945803d210 crc :-1 s=READY pgs=672 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:50:55.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:55.636+0000 7f9477c4a700 1 -- 172.21.15.42:0/1627685965 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f947005d110 con 0x7f94701201d0 2019-12-05T05:50:55.650 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:55.644+0000 7f94761e7700 1 --2- 172.21.15.42:0/1627685965 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f94701201d0 0x7f947012d7a0 secure :-1 s=THROTTLE_DONE pgs=1561 cs=0 l=1 rx=0x7f946c001a50 tx=0x7f946c0109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:55.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:55.644+0000 7f9466ffd700 1 -- 172.21.15.42:0/1627685965 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f946c031cf0 con 0x7f94701201d0 2019-12-05T05:50:55.812 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:55.808+0000 7f9477c4a700 1 -- 172.21.15.42:0/1627685965 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f947005b8c0 con 0x7f94701201d0 2019-12-05T05:50:55.813 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:55.808+0000 7f94761e7700 1 --2- 172.21.15.42:0/1627685965 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f94701201d0 0x7f947012d7a0 secure :-1 s=THROTTLE_DONE pgs=1561 cs=0 l=1 rx=0x7f946c001a50 tx=0x7f946c0109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:55.814 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:55.808+0000 7f9466ffd700 1 -- 172.21.15.42:0/1627685965 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 192 v192) v1 ==== 95+0+1775 (secure 0 0 0) 0x7f946c031f50 con 0x7f94701201d0 2019-12-05T05:50:55.814 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 192 2019-12-05T05:50:55.814 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:50:55.815 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":192,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":192,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:50:54.569377+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6245},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6245":{"gid":6245,"name":"b","rank":0,"incarnation":189,"state":"up:active","state_seq":25,"addr":"172.21.15.40:6833/2081111170","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2081111170},{"type":"v1","addr":"172.21.15.40:6833","nonce":2081111170}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:50:55.818 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:55.808+0000 7f9477c4a700 1 -- 172.21.15.42:0/1627685965 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f945803ad00 msgr2=0x7f945803d210 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:50:55.818 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:55.808+0000 7f9477c4a700 1 --2- 172.21.15.42:0/1627685965 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f945803ad00 0x7f945803d210 crc :-1 s=READY pgs=672 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:50:55.818 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:55.808+0000 7f9477c4a700 1 -- 172.21.15.42:0/1627685965 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f94701201d0 msgr2=0x7f947012d7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:50:55.819 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:55.812+0000 7f9477c4a700 1 --2- 172.21.15.42:0/1627685965 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f94701201d0 0x7f947012d7a0 secure :-1 s=READY pgs=1561 cs=0 l=1 rx=0x7f946c001a50 tx=0x7f946c0109e0).stop 2019-12-05T05:50:55.819 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:55.812+0000 7f9477c4a700 1 -- 172.21.15.42:0/1627685965 shutdown_connections 2019-12-05T05:50:55.819 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:55.812+0000 7f9477c4a700 1 --2- 172.21.15.42:0/1627685965 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f945803ad00 0x7f945803d210 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:50:55.820 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:55.812+0000 7f9477c4a700 1 --2- 172.21.15.42:0/1627685965 >> v2:172.21.15.40:3300/0 conn(0x7f947011bd40 0x7f9470117b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:50:55.820 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:55.812+0000 7f9477c4a700 1 --2- 172.21.15.42:0/1627685965 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f94701201d0 0x7f947012d7a0 unknown :-1 s=CLOSED pgs=1561 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:50:55.821 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:55.812+0000 7f9477c4a700 1 -- 172.21.15.42:0/1627685965 shutdown_connections 2019-12-05T05:50:55.821 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:55.812+0000 7f9477c4a700 1 -- 172.21.15.42:0/1627685965 wait complete. 2019-12-05T05:50:55.822 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:55.812+0000 7f9477c4a700 1 -- 172.21.15.42:0/1627685965 >> 172.21.15.42:0/1627685965 conn(0x7f947011d150 msgr2=0x7f9470059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:50:55.826 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b has gained rank=0, replacing gid=8188 2019-12-05T05:50:55.826 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 4 secs before reviving mds.a 2019-12-05T05:50:59.828 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.a 2019-12-05T05:50:59.828 INFO:tasks.ceph.mds.a:Restarting daemon 2019-12-05T05:50:59.828 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2019-12-05T05:50:59.832 INFO:tasks.ceph.mds.a:Started 2019-12-05T05:50:59.832 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:50:59.892 INFO:tasks.ceph.mds.a.smithi042.stdout:starting mds.a at 2019-12-05T05:50:59.943 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:59.932+0000 7fba0ae33700 1 Processor -- start 2019-12-05T05:50:59.944 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:59.932+0000 7fba0ae33700 1 -- start start 2019-12-05T05:50:59.944 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:59.936+0000 7fba0ae33700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fba040a7740 0x7fba0408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:50:59.944 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:59.936+0000 7fba0ae33700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fba0407e720 0x7fba0407eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:50:59.944 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:59.936+0000 7fba0ae33700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba040a0710 con 0x7fba0409f2e0 2019-12-05T05:50:59.944 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:59.936+0000 7fba0ae33700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba040a0b00 con 0x7fba0407e720 2019-12-05T05:50:59.945 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:59.936+0000 7fba0ae33700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba0408e5b0 con 0x7fba040a7740 2019-12-05T05:50:59.945 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:59.936+0000 7fba09bd1700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fba0409f2e0 0x7fba0409f710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:42488/0 (socket says 172.21.15.42:42488) 2019-12-05T05:50:59.945 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:59.936+0000 7fba09bd1700 1 -- 172.21.15.42:0/764440974 learned_addr learned my addr 172.21.15.42:0/764440974 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:50:59.945 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:59.936+0000 7fba08bcf700 1 --2- 172.21.15.42:0/764440974 >> v2:172.21.15.40:3300/0 conn(0x7fba0407e720 0x7fba0407eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:50:59.945 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:59.936+0000 7fba093d0700 1 --2- 172.21.15.42:0/764440974 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fba040a7740 0x7fba0408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:50:59.945 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:59.936+0000 7fb9fbfff700 1 -- 172.21.15.42:0/764440974 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3231133740 0 0) 0x7fba040a0710 con 0x7fba0409f2e0 2019-12-05T05:50:59.945 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:59.936+0000 7fb9fbfff700 1 -- 172.21.15.42:0/764440974 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9f00030b0 con 0x7fba0409f2e0 2019-12-05T05:50:59.945 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:59.936+0000 7fba08bcf700 1 -- 172.21.15.42:0/764440974 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fba040a7740 msgr2=0x7fba0408df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:50:59.946 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:59.936+0000 7fba08bcf700 1 --2- 172.21.15.42:0/764440974 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fba040a7740 0x7fba0408df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:50:59.946 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:59.936+0000 7fba08bcf700 1 -- 172.21.15.42:0/764440974 >> v1:172.21.15.42:6789/0 conn(0x7fba0409f2e0 legacy=0x7fba0409f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:50:59.946 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:59.936+0000 7fba08bcf700 1 -- 172.21.15.42:0/764440974 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba0407f0a0 con 0x7fba0407e720 2019-12-05T05:50:59.946 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:59.936+0000 7fba08bcf700 1 --2- 172.21.15.42:0/764440974 >> v2:172.21.15.40:3300/0 conn(0x7fba0407e720 0x7fba0407eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fb9f4008790 tx=0x7fb9f400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:59.946 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:59.936+0000 7fba08bcf700 1 --2- 172.21.15.42:0/764440974 >> v2:172.21.15.40:3300/0 conn(0x7fba0407e720 0x7fba0407eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fb9f4008790 tx=0x7fb9f400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:59.946 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:59.936+0000 7fba08bcf700 1 --2- 172.21.15.42:0/764440974 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fba0407e720 0x7fba0407eb50 secure :-1 s=READY pgs=1196 cs=0 l=1 rx=0x7fb9f4008790 tx=0x7fb9f400fc70).ready entity=mon.0 client_cookie=d34365f88ead0c86 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:50:59.946 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:59.936+0000 7fba08bcf700 1 --2- 172.21.15.42:0/764440974 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fba0407e720 0x7fba0407eb50 secure :-1 s=THROTTLE_DONE pgs=1196 cs=0 l=1 rx=0x7fb9f4008790 tx=0x7fb9f400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:59.947 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:59.936+0000 7fba08bcf700 1 --2- 172.21.15.42:0/764440974 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fba0407e720 0x7fba0407eb50 secure :-1 s=THROTTLE_DONE pgs=1196 cs=0 l=1 rx=0x7fb9f4008790 tx=0x7fb9f400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:59.947 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:59.936+0000 7fb9fbfff700 1 -- 172.21.15.42:0/764440974 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb9f40112f0 con 0x7fba0407e720 2019-12-05T05:50:59.947 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:59.936+0000 7fba08bcf700 1 --2- 172.21.15.42:0/764440974 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fba0407e720 0x7fba0407eb50 secure :-1 s=THROTTLE_DONE pgs=1196 cs=0 l=1 rx=0x7fb9f4008790 tx=0x7fb9f400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:59.947 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:59.936+0000 7fb9fbfff700 1 -- 172.21.15.42:0/764440974 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9f4008ad0 con 0x7fba0407e720 2019-12-05T05:50:59.947 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:59.936+0000 7fb9fbfff700 1 -- 172.21.15.42:0/764440974 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb9f40118d0 con 0x7fba0407e720 2019-12-05T05:50:59.947 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:59.936+0000 7fba0ae33700 1 -- 172.21.15.42:0/764440974 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fba0407e720 msgr2=0x7fba0407eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:50:59.947 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:59.936+0000 7fba0ae33700 1 --2- 172.21.15.42:0/764440974 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fba0407e720 0x7fba0407eb50 secure :-1 s=READY pgs=1196 cs=0 l=1 rx=0x7fb9f4008790 tx=0x7fb9f400fc70).stop 2019-12-05T05:50:59.948 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:59.936+0000 7fba0ae33700 1 -- 172.21.15.42:0/764440974 shutdown_connections 2019-12-05T05:50:59.948 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:59.936+0000 7fba0ae33700 1 --2- 172.21.15.42:0/764440974 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fba040a7740 0x7fba0408df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:50:59.948 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:59.936+0000 7fba0ae33700 1 --2- 172.21.15.42:0/764440974 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fba0407e720 0x7fba0407eb50 unknown :-1 s=CLOSED pgs=1196 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:50:59.948 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:59.936+0000 7fba0ae33700 1 -- 172.21.15.42:0/764440974 shutdown_connections 2019-12-05T05:50:59.948 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:59.936+0000 7fba0ae33700 1 -- 172.21.15.42:0/764440974 wait complete. 2019-12-05T05:50:59.948 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:59.936+0000 7fba0ae33700 1 -- 172.21.15.42:0/764440974 >> 172.21.15.42:0/764440974 conn(0x7fba040aabe0 msgr2=0x7fba040a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:50:59.952 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:59.940+0000 7fba0ae33700 1 Processor -- start 2019-12-05T05:50:59.952 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:59.940+0000 7fba0ae33700 1 -- start start 2019-12-05T05:50:59.952 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:59.940+0000 7fba0ae33700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fba040a9ad0 0x7fba04119bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:50:59.952 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:59.940+0000 7fba0ae33700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fba04116f00 0x7fba04130290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:50:59.952 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:59.940+0000 7fba0ae33700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fba0411b5f0 0x7fba04065510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:50:59.953 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:59.940+0000 7fba0ae33700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba04117480 con 0x7fba04116f00 2019-12-05T05:50:59.953 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:59.940+0000 7fba0ae33700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba0411b350 con 0x7fba0411b5f0 2019-12-05T05:50:59.953 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:59.940+0000 7fba0ae33700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba0411ba20 con 0x7fba040a9ad0 2019-12-05T05:50:59.953 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:59.940+0000 7fba09bd1700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fba0411b5f0 0x7fba04065510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:50:59.953 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:59.940+0000 7fba09bd1700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fba0411b5f0 0x7fba04065510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:42456/0 (socket says 172.21.15.42:42456) 2019-12-05T05:50:59.953 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:59.940+0000 7fba09bd1700 1 -- 172.21.15.42:0/2033977065 learned_addr learned my addr 172.21.15.42:0/2033977065 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:50:59.954 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:59.940+0000 7fba08bcf700 1 --2- 172.21.15.42:0/2033977065 >> v2:172.21.15.40:3300/0 conn(0x7fba04116f00 0x7fba04130290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:50:59.954 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:59.940+0000 7fba093d0700 1 --2- 172.21.15.42:0/2033977065 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fba040a9ad0 0x7fba04119bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:50:59.954 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:59.944+0000 7fba09bd1700 1 -- 172.21.15.42:0/2033977065 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fba040a9ad0 msgr2=0x7fba04119bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:50:59.954 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:59.944+0000 7fba09bd1700 1 --2- 172.21.15.42:0/2033977065 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fba040a9ad0 0x7fba04119bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:50:59.954 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:59.944+0000 7fba09bd1700 1 -- 172.21.15.42:0/2033977065 >> v2:172.21.15.40:3300/0 conn(0x7fba04116f00 msgr2=0x7fba04130290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:50:59.954 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:59.944+0000 7fba09bd1700 1 --2- 172.21.15.42:0/2033977065 >> v2:172.21.15.40:3300/0 conn(0x7fba04116f00 0x7fba04130290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:50:59.955 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:59.944+0000 7fba09bd1700 1 -- 172.21.15.42:0/2033977065 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba04065a60 con 0x7fba0411b5f0 2019-12-05T05:50:59.955 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:59.944+0000 7fba09bd1700 1 --2- 172.21.15.42:0/2033977065 >> v2:172.21.15.42:3300/0 conn(0x7fba0411b5f0 0x7fba04065510 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fb9fc004720 tx=0x7fb9fc010c10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:59.955 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:59.944+0000 7fba09bd1700 1 --2- 172.21.15.42:0/2033977065 >> v2:172.21.15.42:3300/0 conn(0x7fba0411b5f0 0x7fba04065510 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fb9fc004720 tx=0x7fb9fc010c10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:59.955 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:59.944+0000 7fba09bd1700 1 --2- 172.21.15.42:0/2033977065 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fba0411b5f0 0x7fba04065510 secure :-1 s=READY pgs=1566 cs=0 l=1 rx=0x7fb9fc004720 tx=0x7fb9fc010c10).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:50:59.955 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:59.944+0000 7fba09bd1700 1 --2- 172.21.15.42:0/2033977065 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fba0411b5f0 0x7fba04065510 secure :-1 s=THROTTLE_DONE pgs=1566 cs=0 l=1 rx=0x7fb9fc004720 tx=0x7fb9fc010c10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:59.955 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:59.944+0000 7fba09bd1700 1 --2- 172.21.15.42:0/2033977065 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fba0411b5f0 0x7fba04065510 secure :-1 s=THROTTLE_DONE pgs=1566 cs=0 l=1 rx=0x7fb9fc004720 tx=0x7fb9fc010c10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:59.955 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:59.944+0000 7fba09bd1700 1 --2- 172.21.15.42:0/2033977065 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fba0411b5f0 0x7fba04065510 secure :-1 s=THROTTLE_DONE pgs=1566 cs=0 l=1 rx=0x7fb9fc004720 tx=0x7fb9fc010c10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:59.956 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:59.944+0000 7fb9f9ffb700 1 -- 172.21.15.42:0/2033977065 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb9fc0122d0 con 0x7fba0411b5f0 2019-12-05T05:50:59.956 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:59.944+0000 7fb9f9ffb700 1 -- 172.21.15.42:0/2033977065 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9fc005ae0 con 0x7fba0411b5f0 2019-12-05T05:50:59.956 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:59.944+0000 7fba0ae33700 1 -- 172.21.15.42:0/2033977065 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba041187b0 con 0x7fba0411b5f0 2019-12-05T05:50:59.956 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:59.944+0000 7fb9f9ffb700 1 -- 172.21.15.42:0/2033977065 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb9fc0128b0 con 0x7fba0411b5f0 2019-12-05T05:50:59.956 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:59.944+0000 7fba0ae33700 1 -- 172.21.15.42:0/2033977065 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba04118b40 con 0x7fba0411b5f0 2019-12-05T05:50:59.956 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:59.944+0000 7fba09bd1700 1 --2- 172.21.15.42:0/2033977065 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fba0411b5f0 0x7fba04065510 secure :-1 s=THROTTLE_DONE pgs=1566 cs=0 l=1 rx=0x7fb9fc004720 tx=0x7fb9fc010c10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:59.956 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:59.944+0000 7fb9f9ffb700 1 -- 172.21.15.42:0/2033977065 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fb9fc012bc0 con 0x7fba0411b5f0 2019-12-05T05:50:59.957 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:59.944+0000 7fba09bd1700 1 --2- 172.21.15.42:0/2033977065 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fba0411b5f0 0x7fba04065510 secure :-1 s=THROTTLE_DONE pgs=1566 cs=0 l=1 rx=0x7fb9fc004720 tx=0x7fb9fc010c10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:59.957 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:59.944+0000 7fb9f9ffb700 1 --2- 172.21.15.42:0/2033977065 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb9e403ada0 0x7fb9e403d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:50:59.957 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:59.944+0000 7fb9f9ffb700 1 -- 172.21.15.42:0/2033977065 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8783+0+0 (secure 0 0 0) 0x7fb9fc04b4a0 con 0x7fba0411b5f0 2019-12-05T05:50:59.957 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:59.944+0000 7fba093d0700 1 --2- 172.21.15.42:0/2033977065 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb9e403ada0 0x7fb9e403d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:50:59.957 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:59.944+0000 7fba093d0700 1 --2- 172.21.15.42:0/2033977065 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb9e403ada0 0x7fb9e403d2b0 crc :-1 s=READY pgs=673 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:50:59.958 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:59.948+0000 7fba0ae33700 1 -- 172.21.15.42:0/2033977065 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba0405d110 con 0x7fba0411b5f0 2019-12-05T05:50:59.960 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:59.956+0000 7fba09bd1700 1 --2- 172.21.15.42:0/2033977065 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fba0411b5f0 0x7fba04065510 secure :-1 s=THROTTLE_DONE pgs=1566 cs=0 l=1 rx=0x7fb9fc004720 tx=0x7fb9fc010c10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:50:59.960 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:50:59.956+0000 7fb9f9ffb700 1 -- 172.21.15.42:0/2033977065 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fb9fc032520 con 0x7fba0411b5f0 2019-12-05T05:51:00.135 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:00.132+0000 7fba0ae33700 1 -- 172.21.15.42:0/2033977065 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fba0405b8c0 con 0x7fba0411b5f0 2019-12-05T05:51:00.136 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:00.132+0000 7fba09bd1700 1 --2- 172.21.15.42:0/2033977065 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fba0411b5f0 0x7fba04065510 secure :-1 s=THROTTLE_DONE pgs=1566 cs=0 l=1 rx=0x7fb9fc004720 tx=0x7fb9fc010c10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:00.137 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:00.132+0000 7fb9f9ffb700 1 -- 172.21.15.42:0/2033977065 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 193 v193) v1 ==== 95+0+2103 (secure 0 0 0) 0x7fb9fc032780 con 0x7fba0411b5f0 2019-12-05T05:51:00.137 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 193 2019-12-05T05:51:00.137 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:51:00.137 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":193,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":8430,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6835/5991897","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":5991897},{"type":"v1","addr":"172.21.15.42:6835","nonce":5991897}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":193}],"filesystems":[{"mdsmap":{"epoch":192,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:50:54.569377+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6245},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6245":{"gid":6245,"name":"b","rank":0,"incarnation":189,"state":"up:active","state_seq":25,"addr":"172.21.15.40:6833/2081111170","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2081111170},{"type":"v1","addr":"172.21.15.40:6833","nonce":2081111170}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:51:00.140 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:00.132+0000 7fba0ae33700 1 -- 172.21.15.42:0/2033977065 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb9e403ada0 msgr2=0x7fb9e403d2b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:51:00.140 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:00.132+0000 7fba0ae33700 1 --2- 172.21.15.42:0/2033977065 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb9e403ada0 0x7fb9e403d2b0 crc :-1 s=READY pgs=673 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:51:00.140 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:00.132+0000 7fba0ae33700 1 -- 172.21.15.42:0/2033977065 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fba0411b5f0 msgr2=0x7fba04065510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:51:00.140 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:00.132+0000 7fba0ae33700 1 --2- 172.21.15.42:0/2033977065 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fba0411b5f0 0x7fba04065510 secure :-1 s=READY pgs=1566 cs=0 l=1 rx=0x7fb9fc004720 tx=0x7fb9fc010c10).stop 2019-12-05T05:51:00.140 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:00.136+0000 7fba0ae33700 1 -- 172.21.15.42:0/2033977065 shutdown_connections 2019-12-05T05:51:00.140 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:00.136+0000 7fba0ae33700 1 --2- 172.21.15.42:0/2033977065 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb9e403ada0 0x7fb9e403d2b0 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:51:00.141 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:00.136+0000 7fba0ae33700 1 --2- 172.21.15.42:0/2033977065 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fba0411b5f0 0x7fba04065510 unknown :-1 s=CLOSED pgs=1566 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:51:00.141 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:00.136+0000 7fba0ae33700 1 --2- 172.21.15.42:0/2033977065 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fba040a9ad0 0x7fba04119bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:51:00.141 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:00.136+0000 7fba0ae33700 1 --2- 172.21.15.42:0/2033977065 >> v2:172.21.15.40:3300/0 conn(0x7fba04116f00 0x7fba04130290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:51:00.141 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:00.136+0000 7fba0ae33700 1 -- 172.21.15.42:0/2033977065 shutdown_connections 2019-12-05T05:51:00.141 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:00.136+0000 7fba0ae33700 1 -- 172.21.15.42:0/2033977065 wait complete. 2019-12-05T05:51:00.142 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:00.136+0000 7fba0ae33700 1 -- 172.21.15.42:0/2033977065 >> 172.21.15.42:0/2033977065 conn(0x7fba040a80d0 msgr2=0x7fba04059ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:51:00.150 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a reported in up:standby state 2019-12-05T05:51:00.150 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 3 secs before thrashing 2019-12-05T05:51:03.151 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:51:03.262 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.252+0000 7f8605d6d700 1 Processor -- start 2019-12-05T05:51:03.262 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.252+0000 7f8605d6d700 1 -- start start 2019-12-05T05:51:03.262 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.252+0000 7f8605d6d700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f860009fae0 0x7f860014fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:51:03.262 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.252+0000 7f8605d6d700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86000a0710 con 0x7f86000a7740 2019-12-05T05:51:03.262 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.252+0000 7f8605d6d700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f860007ef20 con 0x7f860008df80 2019-12-05T05:51:03.263 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.252+0000 7f8605d6d700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86000a0b00 con 0x7f860009fae0 2019-12-05T05:51:03.263 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.252+0000 7f85ff7fe700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f860008df80 0x7f860009f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:42528/0 (socket says 172.21.15.42:42528) 2019-12-05T05:51:03.263 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.252+0000 7f85ff7fe700 1 -- 172.21.15.42:0/2256945998 learned_addr learned my addr 172.21.15.42:0/2256945998 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:51:03.263 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.252+0000 7f8604b0b700 1 --2- 172.21.15.42:0/2256945998 >> v2:172.21.15.40:3300/0 conn(0x7f860009fae0 0x7f860014fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:51:03.263 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.252+0000 7f85feffd700 1 -- 172.21.15.42:0/2256945998 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 404701547 0 0) 0x7f860007ef20 con 0x7f860008df80 2019-12-05T05:51:03.263 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.252+0000 7f85feffd700 1 -- 172.21.15.42:0/2256945998 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85e00030b0 con 0x7f860008df80 2019-12-05T05:51:03.263 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.252+0000 7f85feffd700 1 -- 172.21.15.42:0/2256945998 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1089843978 0 0) 0x7f86000a0710 con 0x7f86000a7740 2019-12-05T05:51:03.264 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.252+0000 7f85feffd700 1 -- 172.21.15.42:0/2256945998 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f860007ef20 con 0x7f86000a7740 2019-12-05T05:51:03.264 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.252+0000 7f8604b0b700 1 -- 172.21.15.42:0/2256945998 >> v1:172.21.15.42:6789/0 conn(0x7f860008df80 legacy=0x7f860009f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:51:03.264 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.252+0000 7f8604b0b700 1 -- 172.21.15.42:0/2256945998 >> v1:172.21.15.40:6789/0 conn(0x7f86000a7740 legacy=0x7f860007e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:51:03.264 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.252+0000 7f8604b0b700 1 -- 172.21.15.42:0/2256945998 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86000a9ba0 con 0x7f860009fae0 2019-12-05T05:51:03.264 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.252+0000 7f8604b0b700 1 --2- 172.21.15.42:0/2256945998 >> v2:172.21.15.40:3300/0 conn(0x7f860009fae0 0x7f860014fff0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f85f0008b10 tx=0x7f85f000ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:03.264 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.252+0000 7f8604b0b700 1 --2- 172.21.15.42:0/2256945998 >> v2:172.21.15.40:3300/0 conn(0x7f860009fae0 0x7f860014fff0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f85f0008b10 tx=0x7f85f000ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:03.264 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.252+0000 7f8604b0b700 1 --2- 172.21.15.42:0/2256945998 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f860009fae0 0x7f860014fff0 secure :-1 s=READY pgs=1198 cs=0 l=1 rx=0x7f85f0008b10 tx=0x7f85f000ff70).ready entity=mon.0 client_cookie=f42418e78b3175fb server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:51:03.265 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.252+0000 7f8604b0b700 1 --2- 172.21.15.42:0/2256945998 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f860009fae0 0x7f860014fff0 secure :-1 s=THROTTLE_DONE pgs=1198 cs=0 l=1 rx=0x7f85f0008b10 tx=0x7f85f000ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:03.265 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.252+0000 7f85feffd700 1 -- 172.21.15.42:0/2256945998 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f85f0010970 con 0x7f860009fae0 2019-12-05T05:51:03.265 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.252+0000 7f8604b0b700 1 --2- 172.21.15.42:0/2256945998 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f860009fae0 0x7f860014fff0 secure :-1 s=THROTTLE_DONE pgs=1198 cs=0 l=1 rx=0x7f85f0008b10 tx=0x7f85f000ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:03.265 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.252+0000 7f85feffd700 1 -- 172.21.15.42:0/2256945998 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85f0008b50 con 0x7f860009fae0 2019-12-05T05:51:03.265 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.252+0000 7f8604b0b700 1 --2- 172.21.15.42:0/2256945998 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f860009fae0 0x7f860014fff0 secure :-1 s=THROTTLE_DONE pgs=1198 cs=0 l=1 rx=0x7f85f0008b10 tx=0x7f85f000ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:03.265 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.256+0000 7f8605d6d700 1 -- 172.21.15.42:0/2256945998 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f860009fae0 msgr2=0x7f860014fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:51:03.265 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.256+0000 7f85feffd700 1 -- 172.21.15.42:0/2256945998 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f85f00110c0 con 0x7f860009fae0 2019-12-05T05:51:03.266 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.256+0000 7f8605d6d700 1 --2- 172.21.15.42:0/2256945998 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f860009fae0 0x7f860014fff0 secure :-1 s=READY pgs=1198 cs=0 l=1 rx=0x7f85f0008b10 tx=0x7f85f000ff70).stop 2019-12-05T05:51:03.266 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.256+0000 7f8605d6d700 1 -- 172.21.15.42:0/2256945998 shutdown_connections 2019-12-05T05:51:03.266 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.256+0000 7f8605d6d700 1 --2- 172.21.15.42:0/2256945998 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f860009fae0 0x7f860014fff0 unknown :-1 s=CLOSED pgs=1198 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:51:03.266 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.256+0000 7f8605d6d700 1 -- 172.21.15.42:0/2256945998 shutdown_connections 2019-12-05T05:51:03.266 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.256+0000 7f8605d6d700 1 -- 172.21.15.42:0/2256945998 wait complete. 2019-12-05T05:51:03.266 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.256+0000 7f8605d6d700 1 -- 172.21.15.42:0/2256945998 >> 172.21.15.42:0/2256945998 conn(0x7f86000aabe0 msgr2=0x7f86000a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:51:03.270 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.256+0000 7f8605d6d700 1 Processor -- start 2019-12-05T05:51:03.270 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.256+0000 7f8605d6d700 1 -- start start 2019-12-05T05:51:03.270 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.260+0000 7f8605d6d700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8600116430 0x7f8600120190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:51:03.270 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.260+0000 7f8605d6d700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8600115330 con 0x7f86000a2fd0 2019-12-05T05:51:03.271 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.260+0000 7f8605d6d700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f860012cbc0 con 0x7f86000a41d0 2019-12-05T05:51:03.271 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.260+0000 7f8605d6d700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8600120910 con 0x7f8600116430 2019-12-05T05:51:03.271 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.260+0000 7f85fffff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f86000a41d0 0x7f8600114b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:42532/0 (socket says 172.21.15.42:42532) 2019-12-05T05:51:03.271 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.260+0000 7f85fffff700 1 -- 172.21.15.42:0/3568046210 learned_addr learned my addr 172.21.15.42:0/3568046210 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:51:03.271 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.260+0000 7f85ff7fe700 1 --2- 172.21.15.42:0/3568046210 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8600116430 0x7f8600120190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:51:03.271 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.260+0000 7f85fcff9700 1 -- 172.21.15.42:0/3568046210 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2811484817 0 0) 0x7f860012cbc0 con 0x7f86000a41d0 2019-12-05T05:51:03.272 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.260+0000 7f85fcff9700 1 -- 172.21.15.42:0/3568046210 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85d80030b0 con 0x7f86000a41d0 2019-12-05T05:51:03.272 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.260+0000 7f85fcff9700 1 -- 172.21.15.42:0/3568046210 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1578966369 0 0) 0x7f8600115330 con 0x7f86000a2fd0 2019-12-05T05:51:03.272 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.260+0000 7f85fcff9700 1 -- 172.21.15.42:0/3568046210 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f860012cbc0 con 0x7f86000a2fd0 2019-12-05T05:51:03.272 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.260+0000 7f85ff7fe700 1 -- 172.21.15.42:0/3568046210 >> v1:172.21.15.42:6789/0 conn(0x7f86000a41d0 legacy=0x7f8600114b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:51:03.272 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.260+0000 7f85ff7fe700 1 -- 172.21.15.42:0/3568046210 >> v1:172.21.15.40:6789/0 conn(0x7f86000a2fd0 legacy=0x7f86000a3400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:51:03.272 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.260+0000 7f85ff7fe700 1 -- 172.21.15.42:0/3568046210 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f860012de00 con 0x7f8600116430 2019-12-05T05:51:03.272 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.260+0000 7f85ff7fe700 1 --2- 172.21.15.42:0/3568046210 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8600116430 0x7f8600120190 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f85e8000b80 tx=0x7f85e8010810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:03.273 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.260+0000 7f85ff7fe700 1 --2- 172.21.15.42:0/3568046210 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8600116430 0x7f8600120190 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f85e8000b80 tx=0x7f85e8010810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:03.273 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.260+0000 7f85ff7fe700 1 --2- 172.21.15.42:0/3568046210 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8600116430 0x7f8600120190 secure :-1 s=READY pgs=349 cs=0 l=1 rx=0x7f85e8000b80 tx=0x7f85e8010810).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:51:03.273 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.260+0000 7f85ff7fe700 1 --2- 172.21.15.42:0/3568046210 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8600116430 0x7f8600120190 secure :-1 s=THROTTLE_DONE pgs=349 cs=0 l=1 rx=0x7f85e8000b80 tx=0x7f85e8010810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:03.273 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.260+0000 7f85fcff9700 1 -- 172.21.15.42:0/3568046210 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f85e8005c30 con 0x7f8600116430 2019-12-05T05:51:03.273 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.260+0000 7f85ff7fe700 1 --2- 172.21.15.42:0/3568046210 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8600116430 0x7f8600120190 secure :-1 s=THROTTLE_DONE pgs=349 cs=0 l=1 rx=0x7f85e8000b80 tx=0x7f85e8010810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:03.273 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.260+0000 7f8605d6d700 1 -- 172.21.15.42:0/3568046210 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f860012bbb0 con 0x7f8600116430 2019-12-05T05:51:03.273 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.260+0000 7f85fcff9700 1 -- 172.21.15.42:0/3568046210 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85e8005850 con 0x7f8600116430 2019-12-05T05:51:03.274 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.260+0000 7f85ff7fe700 1 --2- 172.21.15.42:0/3568046210 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8600116430 0x7f8600120190 secure :-1 s=THROTTLE_DONE pgs=349 cs=0 l=1 rx=0x7f85e8000b80 tx=0x7f85e8010810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:03.274 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.260+0000 7f85fcff9700 1 -- 172.21.15.42:0/3568046210 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f85e80064a0 con 0x7f8600116430 2019-12-05T05:51:03.274 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.260+0000 7f8605d6d700 1 -- 172.21.15.42:0/3568046210 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f860012c0e0 con 0x7f8600116430 2019-12-05T05:51:03.274 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.264+0000 7f85ff7fe700 1 --2- 172.21.15.42:0/3568046210 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8600116430 0x7f8600120190 secure :-1 s=THROTTLE_DONE pgs=349 cs=0 l=1 rx=0x7f85e8000b80 tx=0x7f85e8010810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:03.274 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.264+0000 7f85fcff9700 1 -- 172.21.15.42:0/3568046210 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f85e8006780 con 0x7f8600116430 2019-12-05T05:51:03.274 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.264+0000 7f85ff7fe700 1 --2- 172.21.15.42:0/3568046210 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8600116430 0x7f8600120190 secure :-1 s=THROTTLE_DONE pgs=349 cs=0 l=1 rx=0x7f85e8000b80 tx=0x7f85e8010810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:03.274 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.264+0000 7f85fcff9700 1 --2- 172.21.15.42:0/3568046210 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f85d803b4a0 0x7f85d803d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:51:03.275 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.264+0000 7f85fcff9700 1 -- 172.21.15.42:0/3568046210 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8783+0+0 (secure 0 0 0) 0x7f85e804af00 con 0x7f8600116430 2019-12-05T05:51:03.275 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.264+0000 7f85fffff700 1 --2- 172.21.15.42:0/3568046210 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f85d803b4a0 0x7f85d803d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:51:03.275 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.264+0000 7f85fffff700 1 --2- 172.21.15.42:0/3568046210 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f85d803b4a0 0x7f85d803d9b0 crc :-1 s=READY pgs=675 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:51:03.275 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.268+0000 7f8605d6d700 1 -- 172.21.15.42:0/3568046210 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f860005d110 con 0x7f8600116430 2019-12-05T05:51:03.281 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.276+0000 7f85ff7fe700 1 --2- 172.21.15.42:0/3568046210 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8600116430 0x7f8600120190 secure :-1 s=THROTTLE_DONE pgs=349 cs=0 l=1 rx=0x7f85e8000b80 tx=0x7f85e8010810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:03.281 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.276+0000 7f85fcff9700 1 -- 172.21.15.42:0/3568046210 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f85e8031f60 con 0x7f8600116430 2019-12-05T05:51:03.473 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.468+0000 7f8605d6d700 1 -- 172.21.15.42:0/3568046210 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f860005b8c0 con 0x7f8600116430 2019-12-05T05:51:03.474 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.468+0000 7f85ff7fe700 1 --2- 172.21.15.42:0/3568046210 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8600116430 0x7f8600120190 secure :-1 s=THROTTLE_DONE pgs=349 cs=0 l=1 rx=0x7f85e8000b80 tx=0x7f85e8010810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:03.474 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.468+0000 7f85fcff9700 1 -- 172.21.15.42:0/3568046210 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 193 v193) v1 ==== 95+0+2103 (secure 0 0 0) 0x7f85e80321c0 con 0x7f8600116430 2019-12-05T05:51:03.474 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 193 2019-12-05T05:51:03.475 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:51:03.475 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":193,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":8430,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6835/5991897","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":5991897},{"type":"v1","addr":"172.21.15.42:6835","nonce":5991897}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":193}],"filesystems":[{"mdsmap":{"epoch":192,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:50:54.569377+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6245},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6245":{"gid":6245,"name":"b","rank":0,"incarnation":189,"state":"up:active","state_seq":25,"addr":"172.21.15.40:6833/2081111170","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2081111170},{"type":"v1","addr":"172.21.15.40:6833","nonce":2081111170}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:51:03.477 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.472+0000 7f8605d6d700 1 -- 172.21.15.42:0/3568046210 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f85d803b4a0 msgr2=0x7f85d803d9b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:51:03.477 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.472+0000 7f8605d6d700 1 --2- 172.21.15.42:0/3568046210 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f85d803b4a0 0x7f85d803d9b0 crc :-1 s=READY pgs=675 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:51:03.477 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.472+0000 7f8605d6d700 1 -- 172.21.15.42:0/3568046210 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8600116430 msgr2=0x7f8600120190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:51:03.478 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.472+0000 7f8605d6d700 1 --2- 172.21.15.42:0/3568046210 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8600116430 0x7f8600120190 secure :-1 s=READY pgs=349 cs=0 l=1 rx=0x7f85e8000b80 tx=0x7f85e8010810).stop 2019-12-05T05:51:03.478 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.472+0000 7f8605d6d700 1 -- 172.21.15.42:0/3568046210 shutdown_connections 2019-12-05T05:51:03.478 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.472+0000 7f8605d6d700 1 --2- 172.21.15.42:0/3568046210 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f85d803b4a0 0x7f85d803d9b0 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:51:03.478 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.472+0000 7f8605d6d700 1 --2- 172.21.15.42:0/3568046210 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8600116430 0x7f8600120190 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:51:03.478 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.472+0000 7f8605d6d700 1 -- 172.21.15.42:0/3568046210 shutdown_connections 2019-12-05T05:51:03.478 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.472+0000 7f8605d6d700 1 -- 172.21.15.42:0/3568046210 wait complete. 2019-12-05T05:51:03.478 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.472+0000 7f8605d6d700 1 -- 172.21.15.42:0/3568046210 >> 172.21.15.42:0/3568046210 conn(0x7f86000a7740 msgr2=0x7f860007dbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:51:03.487 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.b (rank=0) 2019-12-05T05:51:03.488 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2019-12-05T05:51:03.488 INFO:teuthology.orchestra.run:waiting for 300 2019-12-05T05:51:03.501 INFO:tasks.ceph.mds.b:Stopped 2019-12-05T05:51:03.501 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:51:03.606 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.596+0000 7fbeb0801700 1 Processor -- start 2019-12-05T05:51:03.606 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.596+0000 7fbeb0801700 1 -- start start 2019-12-05T05:51:03.607 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.596+0000 7fbeb0801700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbeac08df80 0x7fbeac09f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:51:03.607 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.596+0000 7fbeb0801700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbeac09fc60 con 0x7fbeac09f830 2019-12-05T05:51:03.607 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.596+0000 7fbeb0801700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbeac0a0710 con 0x7fbeac0a7740 2019-12-05T05:51:03.607 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.596+0000 7fbeb0801700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbeac0a0b00 con 0x7fbeac08df80 2019-12-05T05:51:03.607 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.596+0000 7fbeaad9d700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fbeac0a7740 0x7fbeac07e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:42552/0 (socket says 172.21.15.42:42552) 2019-12-05T05:51:03.607 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.596+0000 7fbeaad9d700 1 -- 172.21.15.42:0/3273769001 learned_addr learned my addr 172.21.15.42:0/3273769001 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:51:03.607 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.596+0000 7fbeaa59c700 1 --2- 172.21.15.42:0/3273769001 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbeac08df80 0x7fbeac09f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:51:03.608 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.596+0000 7fbea9d9b700 1 -- 172.21.15.42:0/3273769001 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1965983353 0 0) 0x7fbeac0a0710 con 0x7fbeac0a7740 2019-12-05T05:51:03.608 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.596+0000 7fbea9d9b700 1 -- 172.21.15.42:0/3273769001 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe940030b0 con 0x7fbeac0a7740 2019-12-05T05:51:03.608 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.600+0000 7fbea9d9b700 1 -- 172.21.15.42:0/3273769001 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 401080812 0 0) 0x7fbeac09fc60 con 0x7fbeac09f830 2019-12-05T05:51:03.608 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.600+0000 7fbea9d9b700 1 -- 172.21.15.42:0/3273769001 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbeac0a0710 con 0x7fbeac09f830 2019-12-05T05:51:03.608 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.600+0000 7fbeaa59c700 1 -- 172.21.15.42:0/3273769001 >> v1:172.21.15.42:6789/0 conn(0x7fbeac0a7740 legacy=0x7fbeac07e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:51:03.608 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.600+0000 7fbeaa59c700 1 -- 172.21.15.42:0/3273769001 >> v1:172.21.15.40:6789/0 conn(0x7fbeac09f830 legacy=0x7fbeac14efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:51:03.608 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.600+0000 7fbeaa59c700 1 -- 172.21.15.42:0/3273769001 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbeac07ef20 con 0x7fbeac08df80 2019-12-05T05:51:03.609 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.600+0000 7fbeaa59c700 1 --2- 172.21.15.42:0/3273769001 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbeac08df80 0x7fbeac09f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fbea0002f30 tx=0x7fbea000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:03.609 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.600+0000 7fbeaa59c700 1 --2- 172.21.15.42:0/3273769001 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbeac08df80 0x7fbeac09f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fbea0002f30 tx=0x7fbea000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:03.609 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.600+0000 7fbeaa59c700 1 --2- 172.21.15.42:0/3273769001 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbeac08df80 0x7fbeac09f2e0 secure :-1 s=READY pgs=350 cs=0 l=1 rx=0x7fbea0002f30 tx=0x7fbea000fbf0).ready entity=mon.2 client_cookie=74f420d0bb5e2fa0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:51:03.609 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.600+0000 7fbeaa59c700 1 --2- 172.21.15.42:0/3273769001 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbeac08df80 0x7fbeac09f2e0 secure :-1 s=THROTTLE_DONE pgs=350 cs=0 l=1 rx=0x7fbea0002f30 tx=0x7fbea000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:03.609 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.600+0000 7fbeaa59c700 1 --2- 172.21.15.42:0/3273769001 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbeac08df80 0x7fbeac09f2e0 secure :-1 s=THROTTLE_DONE pgs=350 cs=0 l=1 rx=0x7fbea0002f30 tx=0x7fbea000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:03.609 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.600+0000 7fbea9d9b700 1 -- 172.21.15.42:0/3273769001 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbea00112b0 con 0x7fbeac08df80 2019-12-05T05:51:03.610 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.600+0000 7fbeaa59c700 1 --2- 172.21.15.42:0/3273769001 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbeac08df80 0x7fbeac09f2e0 secure :-1 s=THROTTLE_DONE pgs=350 cs=0 l=1 rx=0x7fbea0002f30 tx=0x7fbea000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:03.610 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.600+0000 7fbea9d9b700 1 -- 172.21.15.42:0/3273769001 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbea0008bc0 con 0x7fbeac08df80 2019-12-05T05:51:03.610 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.600+0000 7fbea9d9b700 1 -- 172.21.15.42:0/3273769001 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbea0011890 con 0x7fbeac08df80 2019-12-05T05:51:03.610 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.600+0000 7fbeb0801700 1 -- 172.21.15.42:0/3273769001 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbeac08df80 msgr2=0x7fbeac09f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:51:03.610 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.600+0000 7fbeb0801700 1 --2- 172.21.15.42:0/3273769001 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbeac08df80 0x7fbeac09f2e0 secure :-1 s=READY pgs=350 cs=0 l=1 rx=0x7fbea0002f30 tx=0x7fbea000fbf0).stop 2019-12-05T05:51:03.610 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.600+0000 7fbeb0801700 1 -- 172.21.15.42:0/3273769001 shutdown_connections 2019-12-05T05:51:03.610 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.600+0000 7fbeb0801700 1 --2- 172.21.15.42:0/3273769001 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbeac08df80 0x7fbeac09f2e0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:51:03.611 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.600+0000 7fbeb0801700 1 -- 172.21.15.42:0/3273769001 shutdown_connections 2019-12-05T05:51:03.611 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.600+0000 7fbeb0801700 1 -- 172.21.15.42:0/3273769001 wait complete. 2019-12-05T05:51:03.611 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.600+0000 7fbeb0801700 1 -- 172.21.15.42:0/3273769001 >> 172.21.15.42:0/3273769001 conn(0x7fbeac0aabe0 msgr2=0x7fbeac0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:51:03.614 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.604+0000 7fbeb0801700 1 Processor -- start 2019-12-05T05:51:03.614 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.604+0000 7fbeb0801700 1 -- start start 2019-12-05T05:51:03.614 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.604+0000 7fbeb0801700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fbeac1296b0 0x7fbeac131090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:51:03.614 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.604+0000 7fbeb0801700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbeac063f90 con 0x7fbeac129280 2019-12-05T05:51:03.614 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.604+0000 7fbeb0801700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbeac064a30 con 0x7fbeac064600 2019-12-05T05:51:03.615 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.604+0000 7fbeb0801700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbeac0634f0 con 0x7fbeac1296b0 2019-12-05T05:51:03.615 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.604+0000 7fbeaad9d700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fbeac064600 0x7fbeac08eb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:42558/0 (socket says 172.21.15.42:42558) 2019-12-05T05:51:03.615 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.604+0000 7fbeaad9d700 1 -- 172.21.15.42:0/3132399585 learned_addr learned my addr 172.21.15.42:0/3132399585 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:51:03.615 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.604+0000 7fbeab59e700 1 --2- 172.21.15.42:0/3132399585 >> v2:172.21.15.40:3300/0 conn(0x7fbeac1296b0 0x7fbeac131090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:51:03.615 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.604+0000 7fbe9b7fe700 1 -- 172.21.15.42:0/3132399585 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3977103974 0 0) 0x7fbeac064a30 con 0x7fbeac064600 2019-12-05T05:51:03.615 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.604+0000 7fbe9b7fe700 1 -- 172.21.15.42:0/3132399585 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe900030b0 con 0x7fbeac064600 2019-12-05T05:51:03.615 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.604+0000 7fbe9b7fe700 1 -- 172.21.15.42:0/3132399585 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 2494902978 0 0) 0x7fbe900030b0 con 0x7fbeac064600 2019-12-05T05:51:03.616 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.604+0000 7fbe9b7fe700 1 -- 172.21.15.42:0/3132399585 --> v1:172.21.15.42:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbeac064a30 con 0x7fbeac064600 2019-12-05T05:51:03.616 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.604+0000 7fbe9b7fe700 1 -- 172.21.15.42:0/3132399585 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7fbe9c001950 con 0x7fbeac064600 2019-12-05T05:51:03.616 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.604+0000 7fbeab59e700 1 -- 172.21.15.42:0/3132399585 >> v1:172.21.15.42:6789/0 conn(0x7fbeac064600 legacy=0x7fbeac08eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:51:03.616 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.604+0000 7fbeab59e700 1 -- 172.21.15.42:0/3132399585 >> v1:172.21.15.40:6789/0 conn(0x7fbeac129280 legacy=0x7fbeac063790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:51:03.617 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.604+0000 7fbeab59e700 1 -- 172.21.15.42:0/3132399585 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbeac129ae0 con 0x7fbeac1296b0 2019-12-05T05:51:03.617 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.604+0000 7fbeab59e700 1 --2- 172.21.15.42:0/3132399585 >> v2:172.21.15.40:3300/0 conn(0x7fbeac1296b0 0x7fbeac131090 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fbea0008370 tx=0x7fbea000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:03.617 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.604+0000 7fbeab59e700 1 --2- 172.21.15.42:0/3132399585 >> v2:172.21.15.40:3300/0 conn(0x7fbeac1296b0 0x7fbeac131090 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fbea0008370 tx=0x7fbea000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:03.617 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.604+0000 7fbeab59e700 1 --2- 172.21.15.42:0/3132399585 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbeac1296b0 0x7fbeac131090 secure :-1 s=READY pgs=1202 cs=0 l=1 rx=0x7fbea0008370 tx=0x7fbea000fbf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:51:03.617 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.604+0000 7fbeab59e700 1 --2- 172.21.15.42:0/3132399585 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbeac1296b0 0x7fbeac131090 secure :-1 s=THROTTLE_DONE pgs=1202 cs=0 l=1 rx=0x7fbea0008370 tx=0x7fbea000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:03.617 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.604+0000 7fbe9b7fe700 1 -- 172.21.15.42:0/3132399585 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbea4006ac0 con 0x7fbeac1296b0 2019-12-05T05:51:03.618 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.604+0000 7fbeab59e700 1 --2- 172.21.15.42:0/3132399585 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbeac1296b0 0x7fbeac131090 secure :-1 s=THROTTLE_DONE pgs=1202 cs=0 l=1 rx=0x7fbea0008370 tx=0x7fbea000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:03.618 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.604+0000 7fbeab59e700 1 --2- 172.21.15.42:0/3132399585 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbeac1296b0 0x7fbeac131090 secure :-1 s=THROTTLE_DONE pgs=1202 cs=0 l=1 rx=0x7fbea0008370 tx=0x7fbea000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:03.618 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.608+0000 7fbe9b7fe700 1 -- 172.21.15.42:0/3132399585 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbea40066e0 con 0x7fbeac1296b0 2019-12-05T05:51:03.618 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.608+0000 7fbeb0801700 1 -- 172.21.15.42:0/3132399585 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbeac143050 con 0x7fbeac1296b0 2019-12-05T05:51:03.618 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.608+0000 7fbe9b7fe700 1 -- 172.21.15.42:0/3132399585 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbea4003640 con 0x7fbeac1296b0 2019-12-05T05:51:03.618 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.608+0000 7fbeb0801700 1 -- 172.21.15.42:0/3132399585 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbeac1315e0 con 0x7fbeac1296b0 2019-12-05T05:51:03.618 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.608+0000 7fbeab59e700 1 --2- 172.21.15.42:0/3132399585 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbeac1296b0 0x7fbeac131090 secure :-1 s=THROTTLE_DONE pgs=1202 cs=0 l=1 rx=0x7fbea0008370 tx=0x7fbea000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:03.619 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.608+0000 7fbe9b7fe700 1 -- 172.21.15.42:0/3132399585 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fbea4003910 con 0x7fbeac1296b0 2019-12-05T05:51:03.619 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.608+0000 7fbeab59e700 1 --2- 172.21.15.42:0/3132399585 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbeac1296b0 0x7fbeac131090 secure :-1 s=THROTTLE_DONE pgs=1202 cs=0 l=1 rx=0x7fbea0008370 tx=0x7fbea000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:03.619 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.608+0000 7fbe9b7fe700 1 --2- 172.21.15.42:0/3132399585 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbe9003b6d0 0x7fbe9003dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:51:03.619 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.608+0000 7fbe9b7fe700 1 -- 172.21.15.42:0/3132399585 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8783+0+0 (secure 0 0 0) 0x7fbea404a880 con 0x7fbeac1296b0 2019-12-05T05:51:03.619 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.608+0000 7fbeaad9d700 1 --2- 172.21.15.42:0/3132399585 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbe9003b6d0 0x7fbe9003dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:51:03.620 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.608+0000 7fbeaad9d700 1 --2- 172.21.15.42:0/3132399585 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbe9003b6d0 0x7fbe9003dbe0 crc :-1 s=READY pgs=676 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:51:03.620 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.612+0000 7fbeb0801700 1 -- 172.21.15.42:0/3132399585 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbeac05d110 con 0x7fbeac1296b0 2019-12-05T05:51:03.627 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.624+0000 7fbeab59e700 1 --2- 172.21.15.42:0/3132399585 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbeac1296b0 0x7fbeac131090 secure :-1 s=THROTTLE_DONE pgs=1202 cs=0 l=1 rx=0x7fbea0008370 tx=0x7fbea000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:03.628 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.624+0000 7fbe9b7fe700 1 -- 172.21.15.42:0/3132399585 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fbea40318c0 con 0x7fbeac1296b0 2019-12-05T05:51:03.795 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.788+0000 7fbeb0801700 1 -- 172.21.15.42:0/3132399585 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbeac05b8c0 con 0x7fbeac1296b0 2019-12-05T05:51:03.796 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.792+0000 7fbeab59e700 1 --2- 172.21.15.42:0/3132399585 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbeac1296b0 0x7fbeac131090 secure :-1 s=THROTTLE_DONE pgs=1202 cs=0 l=1 rx=0x7fbea0008370 tx=0x7fbea000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:03.796 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.792+0000 7fbe9b7fe700 1 -- 172.21.15.42:0/3132399585 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 193 v193) v1 ==== 95+0+2103 (secure 0 0 0) 0x7fbea4031b20 con 0x7fbeac1296b0 2019-12-05T05:51:03.798 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:51:03.798 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":193,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":8430,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6835/5991897","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":5991897},{"type":"v1","addr":"172.21.15.42:6835","nonce":5991897}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":193}],"filesystems":[{"mdsmap":{"epoch":192,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:50:54.569377+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6245},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6245":{"gid":6245,"name":"b","rank":0,"incarnation":189,"state":"up:active","state_seq":25,"addr":"172.21.15.40:6833/2081111170","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2081111170},{"type":"v1","addr":"172.21.15.40:6833","nonce":2081111170}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:51:03.798 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 193 2019-12-05T05:51:03.801 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.796+0000 7fbeb0801700 1 -- 172.21.15.42:0/3132399585 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbe9003b6d0 msgr2=0x7fbe9003dbe0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:51:03.801 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.796+0000 7fbeb0801700 1 --2- 172.21.15.42:0/3132399585 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbe9003b6d0 0x7fbe9003dbe0 crc :-1 s=READY pgs=676 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:51:03.801 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.796+0000 7fbeb0801700 1 -- 172.21.15.42:0/3132399585 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbeac1296b0 msgr2=0x7fbeac131090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:51:03.801 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.796+0000 7fbeb0801700 1 --2- 172.21.15.42:0/3132399585 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbeac1296b0 0x7fbeac131090 secure :-1 s=READY pgs=1202 cs=0 l=1 rx=0x7fbea0008370 tx=0x7fbea000fbf0).stop 2019-12-05T05:51:03.802 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.796+0000 7fbeb0801700 1 -- 172.21.15.42:0/3132399585 shutdown_connections 2019-12-05T05:51:03.802 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.796+0000 7fbeb0801700 1 --2- 172.21.15.42:0/3132399585 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbe9003b6d0 0x7fbe9003dbe0 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:51:03.802 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.796+0000 7fbeb0801700 1 --2- 172.21.15.42:0/3132399585 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbeac1296b0 0x7fbeac131090 unknown :-1 s=CLOSED pgs=1202 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:51:03.802 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.796+0000 7fbeb0801700 1 -- 172.21.15.42:0/3132399585 shutdown_connections 2019-12-05T05:51:03.802 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.796+0000 7fbeb0801700 1 -- 172.21.15.42:0/3132399585 wait complete. 2019-12-05T05:51:03.802 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:03.796+0000 7fbeb0801700 1 -- 172.21.15.42:0/3132399585 >> 172.21.15.42:0/3132399585 conn(0x7fbeac0a96f0 msgr2=0x7fbeac07e570 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:51:03.811 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T05:51:05.812 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:51:05.912 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:05.900+0000 7f119ae82700 1 Processor -- start 2019-12-05T05:51:05.912 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:05.900+0000 7f119ae82700 1 -- start start 2019-12-05T05:51:05.912 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:05.904+0000 7f119ae82700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f119408df70 0x7f119409f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:51:05.913 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:05.904+0000 7f119ae82700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f119409f820 0x7f119414efd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:51:05.913 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:05.904+0000 7f119ae82700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f119409fc50 con 0x7f11940a7730 2019-12-05T05:51:05.913 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:05.904+0000 7f119ae82700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11940a0af0 con 0x7f119409f820 2019-12-05T05:51:05.913 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:05.904+0000 7f119ae82700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f119408e5a0 con 0x7f119408df70 2019-12-05T05:51:05.913 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:05.904+0000 7f119941f700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f11940a7730 0x7f119407e710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:42580/0 (socket says 172.21.15.42:42580) 2019-12-05T05:51:05.913 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:05.904+0000 7f119941f700 1 -- 172.21.15.42:0/498278800 learned_addr learned my addr 172.21.15.42:0/498278800 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:51:05.913 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:05.904+0000 7f1198c1e700 1 --2- 172.21.15.42:0/498278800 >> v2:172.21.15.42:3300/0 conn(0x7f119408df70 0x7f119409f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:51:05.914 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:05.904+0000 7f1199c20700 1 --2- 172.21.15.42:0/498278800 >> v2:172.21.15.40:3300/0 conn(0x7f119409f820 0x7f119414efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:51:05.914 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:05.904+0000 7f1193fff700 1 -- 172.21.15.42:0/498278800 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2431051593 0 0) 0x7f119409fc50 con 0x7f11940a7730 2019-12-05T05:51:05.914 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:05.904+0000 7f1193fff700 1 -- 172.21.15.42:0/498278800 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f117c0030b0 con 0x7f11940a7730 2019-12-05T05:51:05.914 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:05.904+0000 7f1198c1e700 1 -- 172.21.15.42:0/498278800 >> v2:172.21.15.40:3300/0 conn(0x7f119409f820 msgr2=0x7f119414efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:51:05.914 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:05.904+0000 7f1198c1e700 1 --2- 172.21.15.42:0/498278800 >> v2:172.21.15.40:3300/0 conn(0x7f119409f820 0x7f119414efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:51:05.914 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:05.904+0000 7f1198c1e700 1 -- 172.21.15.42:0/498278800 >> v1:172.21.15.42:6789/0 conn(0x7f11940a7730 legacy=0x7f119407e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:51:05.914 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:05.904+0000 7f1198c1e700 1 -- 172.21.15.42:0/498278800 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11940a0700 con 0x7f119408df70 2019-12-05T05:51:05.915 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:05.904+0000 7f1199c20700 1 --2- 172.21.15.42:0/498278800 >> v2:172.21.15.40:3300/0 conn(0x7f119409f820 0x7f119414efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:51:05.915 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:05.904+0000 7f1198c1e700 1 --2- 172.21.15.42:0/498278800 >> v2:172.21.15.42:3300/0 conn(0x7f119408df70 0x7f119409f2d0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f1188002f30 tx=0x7f118800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:05.915 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:05.904+0000 7f1198c1e700 1 --2- 172.21.15.42:0/498278800 >> v2:172.21.15.42:3300/0 conn(0x7f119408df70 0x7f119409f2d0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f1188002f30 tx=0x7f118800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:05.915 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:05.904+0000 7f1198c1e700 1 --2- 172.21.15.42:0/498278800 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f119408df70 0x7f119409f2d0 secure :-1 s=READY pgs=1573 cs=0 l=1 rx=0x7f1188002f30 tx=0x7f118800fc70).ready entity=mon.1 client_cookie=7665db9fd8d61df1 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:51:05.915 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:05.904+0000 7f1198c1e700 1 --2- 172.21.15.42:0/498278800 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f119408df70 0x7f119409f2d0 secure :-1 s=THROTTLE_DONE pgs=1573 cs=0 l=1 rx=0x7f1188002f30 tx=0x7f118800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:05.915 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:05.904+0000 7f1193fff700 1 -- 172.21.15.42:0/498278800 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1188008bb0 con 0x7f119408df70 2019-12-05T05:51:05.915 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:05.904+0000 7f1198c1e700 1 --2- 172.21.15.42:0/498278800 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f119408df70 0x7f119409f2d0 secure :-1 s=THROTTLE_DONE pgs=1573 cs=0 l=1 rx=0x7f1188002f30 tx=0x7f118800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:05.916 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:05.904+0000 7f1198c1e700 1 --2- 172.21.15.42:0/498278800 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f119408df70 0x7f119409f2d0 secure :-1 s=THROTTLE_DONE pgs=1573 cs=0 l=1 rx=0x7f1188002f30 tx=0x7f118800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:05.916 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:05.904+0000 7f1193fff700 1 -- 172.21.15.42:0/498278800 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11880087d0 con 0x7f119408df70 2019-12-05T05:51:05.916 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:05.904+0000 7f1193fff700 1 -- 172.21.15.42:0/498278800 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1188011820 con 0x7f119408df70 2019-12-05T05:51:05.916 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:05.904+0000 7f119ae82700 1 -- 172.21.15.42:0/498278800 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f119408df70 msgr2=0x7f119409f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:51:05.916 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:05.904+0000 7f119ae82700 1 --2- 172.21.15.42:0/498278800 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f119408df70 0x7f119409f2d0 secure :-1 s=READY pgs=1573 cs=0 l=1 rx=0x7f1188002f30 tx=0x7f118800fc70).stop 2019-12-05T05:51:05.916 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:05.904+0000 7f119ae82700 1 -- 172.21.15.42:0/498278800 shutdown_connections 2019-12-05T05:51:05.916 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:05.904+0000 7f119ae82700 1 --2- 172.21.15.42:0/498278800 >> v2:172.21.15.40:3300/0 conn(0x7f119409f820 0x7f119414efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:51:05.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:05.904+0000 7f119ae82700 1 --2- 172.21.15.42:0/498278800 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f119408df70 0x7f119409f2d0 unknown :-1 s=CLOSED pgs=1573 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:51:05.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:05.904+0000 7f119ae82700 1 -- 172.21.15.42:0/498278800 shutdown_connections 2019-12-05T05:51:05.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:05.904+0000 7f119ae82700 1 -- 172.21.15.42:0/498278800 wait complete. 2019-12-05T05:51:05.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:05.904+0000 7f119ae82700 1 -- 172.21.15.42:0/498278800 >> 172.21.15.42:0/498278800 conn(0x7f11940aabd0 msgr2=0x7f11940a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:51:05.920 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:05.908+0000 7f119ae82700 1 Processor -- start 2019-12-05T05:51:05.920 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:05.908+0000 7f119ae82700 1 -- start start 2019-12-05T05:51:05.920 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:05.912+0000 7f119ae82700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f11940a9920 0x7f119411a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:51:05.920 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:05.912+0000 7f119ae82700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f119411ac00 0x7f11940762e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:51:05.920 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:05.912+0000 7f119ae82700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f119411b030 con 0x7f1194122190 2019-12-05T05:51:05.921 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:05.912+0000 7f119ae82700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11940a9d50 con 0x7f119411ac00 2019-12-05T05:51:05.921 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:05.912+0000 7f119ae82700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11941219b0 con 0x7f11940a9920 2019-12-05T05:51:05.921 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:05.912+0000 7f1198c1e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f11940a9920 0x7f119411a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:51:05.921 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:05.912+0000 7f1198c1e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f11940a9920 0x7f119411a6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:42550/0 (socket says 172.21.15.42:42550) 2019-12-05T05:51:05.921 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:05.912+0000 7f1198c1e700 1 -- 172.21.15.42:0/4091591959 learned_addr learned my addr 172.21.15.42:0/4091591959 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:51:05.921 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:05.912+0000 7f1199c20700 1 --2- 172.21.15.42:0/4091591959 >> v2:172.21.15.40:3300/0 conn(0x7f119411ac00 0x7f11940762e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:51:05.921 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:05.912+0000 7f1198c1e700 1 -- 172.21.15.42:0/4091591959 >> v2:172.21.15.40:3300/0 conn(0x7f119411ac00 msgr2=0x7f11940762e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:51:05.922 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:05.912+0000 7f1198c1e700 1 --2- 172.21.15.42:0/4091591959 >> v2:172.21.15.40:3300/0 conn(0x7f119411ac00 0x7f11940762e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:51:05.922 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:05.912+0000 7f1198c1e700 1 -- 172.21.15.42:0/4091591959 >> v1:172.21.15.40:6789/0 conn(0x7f1194122190 legacy=0x7f11941335b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:51:05.922 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:05.912+0000 7f1198c1e700 1 -- 172.21.15.42:0/4091591959 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1194077840 con 0x7f11940a9920 2019-12-05T05:51:05.922 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:05.912+0000 7f1198c1e700 1 --2- 172.21.15.42:0/4091591959 >> v2:172.21.15.42:3300/0 conn(0x7f11940a9920 0x7f119411a6b0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f1188002f30 tx=0x7f11880118c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:05.922 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:05.912+0000 7f1198c1e700 1 --2- 172.21.15.42:0/4091591959 >> v2:172.21.15.42:3300/0 conn(0x7f11940a9920 0x7f119411a6b0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f1188002f30 tx=0x7f11880118c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:05.922 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:05.912+0000 7f1198c1e700 1 --2- 172.21.15.42:0/4091591959 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f11940a9920 0x7f119411a6b0 secure :-1 s=READY pgs=1574 cs=0 l=1 rx=0x7f1188002f30 tx=0x7f11880118c0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:51:05.923 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:05.912+0000 7f1198c1e700 1 --2- 172.21.15.42:0/4091591959 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f11940a9920 0x7f119411a6b0 secure :-1 s=THROTTLE_DONE pgs=1574 cs=0 l=1 rx=0x7f1188002f30 tx=0x7f11880118c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:05.923 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:05.912+0000 7f1191ffb700 1 -- 172.21.15.42:0/4091591959 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f118800f280 con 0x7f11940a9920 2019-12-05T05:51:05.923 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:05.912+0000 7f1198c1e700 1 --2- 172.21.15.42:0/4091591959 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f11940a9920 0x7f119411a6b0 secure :-1 s=THROTTLE_DONE pgs=1574 cs=0 l=1 rx=0x7f1188002f30 tx=0x7f11880118c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:05.923 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:05.912+0000 7f1191ffb700 1 -- 172.21.15.42:0/4091591959 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1188008db0 con 0x7f11940a9920 2019-12-05T05:51:05.923 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:05.912+0000 7f1198c1e700 1 --2- 172.21.15.42:0/4091591959 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f11940a9920 0x7f119411a6b0 secure :-1 s=THROTTLE_DONE pgs=1574 cs=0 l=1 rx=0x7f1188002f30 tx=0x7f11880118c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:05.923 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:05.912+0000 7f119ae82700 1 -- 172.21.15.42:0/4091591959 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1194077b70 con 0x7f11940a9920 2019-12-05T05:51:05.924 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:05.912+0000 7f1191ffb700 1 -- 172.21.15.42:0/4091591959 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1188005540 con 0x7f11940a9920 2019-12-05T05:51:05.924 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:05.912+0000 7f119ae82700 1 -- 172.21.15.42:0/4091591959 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1194078070 con 0x7f11940a9920 2019-12-05T05:51:05.924 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:05.912+0000 7f1198c1e700 1 --2- 172.21.15.42:0/4091591959 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f11940a9920 0x7f119411a6b0 secure :-1 s=THROTTLE_DONE pgs=1574 cs=0 l=1 rx=0x7f1188002f30 tx=0x7f11880118c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:05.924 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:05.912+0000 7f1191ffb700 1 -- 172.21.15.42:0/4091591959 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f11880130b0 con 0x7f11940a9920 2019-12-05T05:51:05.924 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:05.912+0000 7f1198c1e700 1 --2- 172.21.15.42:0/4091591959 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f11940a9920 0x7f119411a6b0 secure :-1 s=THROTTLE_DONE pgs=1574 cs=0 l=1 rx=0x7f1188002f30 tx=0x7f11880118c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:05.924 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:05.912+0000 7f1191ffb700 1 --2- 172.21.15.42:0/4091591959 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f117403ad40 0x7f117403d250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:51:05.925 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:05.912+0000 7f1191ffb700 1 -- 172.21.15.42:0/4091591959 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8783+0+0 (secure 0 0 0) 0x7f118804b450 con 0x7f11940a9920 2019-12-05T05:51:05.925 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:05.912+0000 7f119941f700 1 --2- 172.21.15.42:0/4091591959 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f117403ad40 0x7f117403d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:51:05.925 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:05.916+0000 7f119941f700 1 --2- 172.21.15.42:0/4091591959 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f117403ad40 0x7f117403d250 crc :-1 s=READY pgs=677 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:51:05.925 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:05.916+0000 7f119ae82700 1 -- 172.21.15.42:0/4091591959 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f119405d100 con 0x7f11940a9920 2019-12-05T05:51:05.929 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:05.924+0000 7f1198c1e700 1 --2- 172.21.15.42:0/4091591959 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f11940a9920 0x7f119411a6b0 secure :-1 s=THROTTLE_DONE pgs=1574 cs=0 l=1 rx=0x7f1188002f30 tx=0x7f11880118c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:05.929 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:05.924+0000 7f1191ffb700 1 -- 172.21.15.42:0/4091591959 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f1188032710 con 0x7f11940a9920 2019-12-05T05:51:06.090 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:06.084+0000 7f119ae82700 1 -- 172.21.15.42:0/4091591959 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f119405b8b0 con 0x7f11940a9920 2019-12-05T05:51:06.090 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:06.084+0000 7f1198c1e700 1 --2- 172.21.15.42:0/4091591959 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f11940a9920 0x7f119411a6b0 secure :-1 s=THROTTLE_DONE pgs=1574 cs=0 l=1 rx=0x7f1188002f30 tx=0x7f11880118c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:06.091 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:06.084+0000 7f1191ffb700 1 -- 172.21.15.42:0/4091591959 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 193 v193) v1 ==== 95+0+2103 (secure 0 0 0) 0x7f11880332c0 con 0x7f11940a9920 2019-12-05T05:51:06.093 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 193 2019-12-05T05:51:06.093 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:51:06.093 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":193,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":8430,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6835/5991897","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":5991897},{"type":"v1","addr":"172.21.15.42:6835","nonce":5991897}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":193}],"filesystems":[{"mdsmap":{"epoch":192,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:50:54.569377+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6245},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6245":{"gid":6245,"name":"b","rank":0,"incarnation":189,"state":"up:active","state_seq":25,"addr":"172.21.15.40:6833/2081111170","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2081111170},{"type":"v1","addr":"172.21.15.40:6833","nonce":2081111170}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:51:06.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:06.092+0000 7f119ae82700 1 -- 172.21.15.42:0/4091591959 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f117403ad40 msgr2=0x7f117403d250 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:51:06.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:06.092+0000 7f119ae82700 1 --2- 172.21.15.42:0/4091591959 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f117403ad40 0x7f117403d250 crc :-1 s=READY pgs=677 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:51:06.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:06.092+0000 7f119ae82700 1 -- 172.21.15.42:0/4091591959 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f11940a9920 msgr2=0x7f119411a6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:51:06.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:06.092+0000 7f119ae82700 1 --2- 172.21.15.42:0/4091591959 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f11940a9920 0x7f119411a6b0 secure :-1 s=READY pgs=1574 cs=0 l=1 rx=0x7f1188002f30 tx=0x7f11880118c0).stop 2019-12-05T05:51:06.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:06.092+0000 7f119ae82700 1 -- 172.21.15.42:0/4091591959 shutdown_connections 2019-12-05T05:51:06.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:06.092+0000 7f119ae82700 1 --2- 172.21.15.42:0/4091591959 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f117403ad40 0x7f117403d250 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:51:06.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:06.092+0000 7f119ae82700 1 --2- 172.21.15.42:0/4091591959 >> v2:172.21.15.40:3300/0 conn(0x7f119411ac00 0x7f11940762e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:51:06.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:06.092+0000 7f119ae82700 1 --2- 172.21.15.42:0/4091591959 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f11940a9920 0x7f119411a6b0 unknown :-1 s=CLOSED pgs=1574 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:51:06.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:06.092+0000 7f119ae82700 1 -- 172.21.15.42:0/4091591959 shutdown_connections 2019-12-05T05:51:06.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:06.092+0000 7f119ae82700 1 -- 172.21.15.42:0/4091591959 wait complete. 2019-12-05T05:51:06.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:06.092+0000 7f119ae82700 1 -- 172.21.15.42:0/4091591959 >> 172.21.15.42:0/4091591959 conn(0x7f11940a8a60 msgr2=0x7f1194059ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:51:06.109 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T05:51:08.110 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:51:08.219 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:08.208+0000 7f8cb34c4700 1 Processor -- start 2019-12-05T05:51:08.219 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:08.208+0000 7f8cb34c4700 1 -- start start 2019-12-05T05:51:08.220 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:08.208+0000 7f8cb34c4700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f8cac0a7740 0x7f8cac08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:51:08.220 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:08.208+0000 7f8cb34c4700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8cac09fae0 0x7f8cac07eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:51:08.220 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:08.208+0000 7f8cb34c4700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cac0a0710 con 0x7f8cac07e720 2019-12-05T05:51:08.220 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:08.208+0000 7f8cb34c4700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8cac0a0b00 con 0x7f8cac09fae0 2019-12-05T05:51:08.220 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:08.208+0000 7f8cb34c4700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8cac08e5b0 con 0x7f8cac0a7740 2019-12-05T05:51:08.220 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:08.212+0000 7f8cb1a61700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f8cac0a7740 0x7f8cac08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:51:08.220 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:08.212+0000 7f8cb1a61700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f8cac0a7740 0x7f8cac08df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:42558/0 (socket says 172.21.15.42:42558) 2019-12-05T05:51:08.221 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:08.212+0000 7f8cb1a61700 1 -- 172.21.15.42:0/4099511765 learned_addr learned my addr 172.21.15.42:0/4099511765 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:51:08.221 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:08.212+0000 7f8cb2262700 1 --2- 172.21.15.42:0/4099511765 >> v2:172.21.15.40:3300/0 conn(0x7f8cac09fae0 0x7f8cac07eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:51:08.221 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:08.212+0000 7f8cb1a61700 1 -- 172.21.15.42:0/4099511765 >> v2:172.21.15.40:3300/0 conn(0x7f8cac09fae0 msgr2=0x7f8cac07eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:51:08.221 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:08.212+0000 7f8cb1a61700 1 --2- 172.21.15.42:0/4099511765 >> v2:172.21.15.40:3300/0 conn(0x7f8cac09fae0 0x7f8cac07eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:51:08.221 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:08.212+0000 7f8cb1a61700 1 -- 172.21.15.42:0/4099511765 >> v1:172.21.15.40:6789/0 conn(0x7f8cac07e720 legacy=0x7f8cac09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:51:08.221 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:08.212+0000 7f8cb1a61700 1 -- 172.21.15.42:0/4099511765 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cac07f0a0 con 0x7f8cac0a7740 2019-12-05T05:51:08.221 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:08.212+0000 7f8cb1a61700 1 --2- 172.21.15.42:0/4099511765 >> v2:172.21.15.42:3300/0 conn(0x7f8cac0a7740 0x7f8cac08df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f8ca8002f30 tx=0x7f8ca800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:08.222 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:08.212+0000 7f8cb1a61700 1 --2- 172.21.15.42:0/4099511765 >> v2:172.21.15.42:3300/0 conn(0x7f8cac0a7740 0x7f8cac08df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f8ca8002f30 tx=0x7f8ca800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:08.222 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:08.212+0000 7f8cb1a61700 1 --2- 172.21.15.42:0/4099511765 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8cac0a7740 0x7f8cac08df80 secure :-1 s=READY pgs=1575 cs=0 l=1 rx=0x7f8ca8002f30 tx=0x7f8ca800fd30).ready entity=mon.1 client_cookie=653e47132260f496 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:51:08.222 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:08.212+0000 7f8cb1a61700 1 --2- 172.21.15.42:0/4099511765 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8cac0a7740 0x7f8cac08df80 secure :-1 s=THROTTLE_DONE pgs=1575 cs=0 l=1 rx=0x7f8ca8002f30 tx=0x7f8ca800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:08.222 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:08.212+0000 7f8cb1a61700 1 --2- 172.21.15.42:0/4099511765 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8cac0a7740 0x7f8cac08df80 secure :-1 s=THROTTLE_DONE pgs=1575 cs=0 l=1 rx=0x7f8ca8002f30 tx=0x7f8ca800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:08.222 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:08.212+0000 7f8cb0a5f700 1 -- 172.21.15.42:0/4099511765 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8ca8008c90 con 0x7f8cac0a7740 2019-12-05T05:51:08.222 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:08.212+0000 7f8cb1a61700 1 --2- 172.21.15.42:0/4099511765 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8cac0a7740 0x7f8cac08df80 secure :-1 s=THROTTLE_DONE pgs=1575 cs=0 l=1 rx=0x7f8ca8002f30 tx=0x7f8ca800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:08.222 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:08.212+0000 7f8cb0a5f700 1 -- 172.21.15.42:0/4099511765 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ca80088b0 con 0x7f8cac0a7740 2019-12-05T05:51:08.223 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:08.212+0000 7f8cb0a5f700 1 -- 172.21.15.42:0/4099511765 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8ca8011ad0 con 0x7f8cac0a7740 2019-12-05T05:51:08.223 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:08.212+0000 7f8cb34c4700 1 -- 172.21.15.42:0/4099511765 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8cac0a7740 msgr2=0x7f8cac08df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:51:08.223 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:08.212+0000 7f8cb34c4700 1 --2- 172.21.15.42:0/4099511765 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8cac0a7740 0x7f8cac08df80 secure :-1 s=READY pgs=1575 cs=0 l=1 rx=0x7f8ca8002f30 tx=0x7f8ca800fd30).stop 2019-12-05T05:51:08.223 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:08.212+0000 7f8cb34c4700 1 -- 172.21.15.42:0/4099511765 shutdown_connections 2019-12-05T05:51:08.223 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:08.212+0000 7f8cb34c4700 1 --2- 172.21.15.42:0/4099511765 >> v2:172.21.15.40:3300/0 conn(0x7f8cac09fae0 0x7f8cac07eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:51:08.223 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:08.212+0000 7f8cb34c4700 1 --2- 172.21.15.42:0/4099511765 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8cac0a7740 0x7f8cac08df80 unknown :-1 s=CLOSED pgs=1575 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:51:08.223 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:08.212+0000 7f8cb34c4700 1 -- 172.21.15.42:0/4099511765 shutdown_connections 2019-12-05T05:51:08.224 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:08.212+0000 7f8cb34c4700 1 -- 172.21.15.42:0/4099511765 wait complete. 2019-12-05T05:51:08.224 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:08.212+0000 7f8cb34c4700 1 -- 172.21.15.42:0/4099511765 >> 172.21.15.42:0/4099511765 conn(0x7f8cac0aabe0 msgr2=0x7f8cac0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:51:08.227 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:08.216+0000 7f8cb34c4700 1 Processor -- start 2019-12-05T05:51:08.227 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:08.216+0000 7f8cb34c4700 1 -- start start 2019-12-05T05:51:08.227 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:08.216+0000 7f8cb34c4700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8cac11b2a0 0x7f8cac118450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:51:08.228 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:08.216+0000 7f8cb34c4700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f8cac0a4f00 0x7f8cac0a5330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:51:08.228 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:08.216+0000 7f8cb34c4700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8cac131920 0x7f8cac131d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:51:08.228 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:08.216+0000 7f8cb34c4700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8cac1313e0 con 0x7f8cac131920 2019-12-05T05:51:08.228 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:08.216+0000 7f8cb34c4700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8cac131680 con 0x7f8cac0a4f00 2019-12-05T05:51:08.228 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:08.216+0000 7f8cb34c4700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cac11a0f0 con 0x7f8cac11b2a0 2019-12-05T05:51:08.228 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:08.216+0000 7f8cb1260700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f8cac0a4f00 0x7f8cac0a5330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:51:08.228 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:08.216+0000 7f8cb1a61700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8cac11b2a0 0x7f8cac118450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:51:08.229 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:08.216+0000 7f8cb2262700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8cac131920 0x7f8cac131d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:51:08.229 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:08.216+0000 7f8cb1260700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f8cac0a4f00 0x7f8cac0a5330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:42566/0 (socket says 172.21.15.42:42566) 2019-12-05T05:51:08.229 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:08.216+0000 7f8cb1260700 1 -- 172.21.15.42:0/2174827676 learned_addr learned my addr 172.21.15.42:0/2174827676 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:51:08.229 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:08.216+0000 7f8cb1260700 1 -- 172.21.15.42:0/2174827676 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8cac11b2a0 msgr2=0x7f8cac118450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:51:08.229 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:08.216+0000 7f8cb1260700 1 --2- 172.21.15.42:0/2174827676 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8cac11b2a0 0x7f8cac118450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:51:08.229 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:08.216+0000 7f8cb1260700 1 -- 172.21.15.42:0/2174827676 >> v2:172.21.15.40:3300/0 conn(0x7f8cac131920 msgr2=0x7f8cac131d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:51:08.229 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:08.216+0000 7f8cb1260700 1 --2- 172.21.15.42:0/2174827676 >> v2:172.21.15.40:3300/0 conn(0x7f8cac131920 0x7f8cac131d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:51:08.230 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:08.216+0000 7f8cb1260700 1 -- 172.21.15.42:0/2174827676 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cac1189a0 con 0x7f8cac0a4f00 2019-12-05T05:51:08.230 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:08.216+0000 7f8cb2262700 1 --2- 172.21.15.42:0/2174827676 >> v2:172.21.15.40:3300/0 conn(0x7f8cac131920 0x7f8cac131d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:51:08.230 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:08.216+0000 7f8cb1a61700 1 --2- 172.21.15.42:0/2174827676 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8cac11b2a0 0x7f8cac118450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:51:08.230 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:08.216+0000 7f8cb1260700 1 --2- 172.21.15.42:0/2174827676 >> v2:172.21.15.42:3300/0 conn(0x7f8cac0a4f00 0x7f8cac0a5330 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f8c9c000e90 tx=0x7f8c9c010a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:08.230 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:08.220+0000 7f8cb1260700 1 --2- 172.21.15.42:0/2174827676 >> v2:172.21.15.42:3300/0 conn(0x7f8cac0a4f00 0x7f8cac0a5330 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f8c9c000e90 tx=0x7f8c9c010a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:08.231 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:08.220+0000 7f8cb1260700 1 --2- 172.21.15.42:0/2174827676 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8cac0a4f00 0x7f8cac0a5330 secure :-1 s=READY pgs=1576 cs=0 l=1 rx=0x7f8c9c000e90 tx=0x7f8c9c010a20).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:51:08.231 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:08.220+0000 7f8cb1260700 1 --2- 172.21.15.42:0/2174827676 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8cac0a4f00 0x7f8cac0a5330 secure :-1 s=THROTTLE_DONE pgs=1576 cs=0 l=1 rx=0x7f8c9c000e90 tx=0x7f8c9c010a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:08.231 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:08.220+0000 7f8ca27fc700 1 -- 172.21.15.42:0/2174827676 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8c9c00caa0 con 0x7f8cac0a4f00 2019-12-05T05:51:08.231 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:08.220+0000 7f8cb1260700 1 --2- 172.21.15.42:0/2174827676 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8cac0a4f00 0x7f8cac0a5330 secure :-1 s=THROTTLE_DONE pgs=1576 cs=0 l=1 rx=0x7f8c9c000e90 tx=0x7f8c9c010a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:08.231 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:08.220+0000 7f8cb1260700 1 --2- 172.21.15.42:0/2174827676 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8cac0a4f00 0x7f8cac0a5330 secure :-1 s=THROTTLE_DONE pgs=1576 cs=0 l=1 rx=0x7f8c9c000e90 tx=0x7f8c9c010a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:08.231 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:08.220+0000 7f8ca27fc700 1 -- 172.21.15.42:0/2174827676 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c9c00c6c0 con 0x7f8cac0a4f00 2019-12-05T05:51:08.231 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:08.220+0000 7f8cb34c4700 1 -- 172.21.15.42:0/2174827676 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cac11a5f0 con 0x7f8cac0a4f00 2019-12-05T05:51:08.232 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:08.220+0000 7f8ca27fc700 1 -- 172.21.15.42:0/2174827676 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8c9c012290 con 0x7f8cac0a4f00 2019-12-05T05:51:08.232 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:08.220+0000 7f8cb34c4700 1 -- 172.21.15.42:0/2174827676 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cac08ecf0 con 0x7f8cac0a4f00 2019-12-05T05:51:08.232 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:08.220+0000 7f8cb1260700 1 --2- 172.21.15.42:0/2174827676 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8cac0a4f00 0x7f8cac0a5330 secure :-1 s=THROTTLE_DONE pgs=1576 cs=0 l=1 rx=0x7f8c9c000e90 tx=0x7f8c9c010a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:08.232 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:08.220+0000 7f8ca27fc700 1 -- 172.21.15.42:0/2174827676 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f8c9c012560 con 0x7f8cac0a4f00 2019-12-05T05:51:08.232 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:08.220+0000 7f8cb1260700 1 --2- 172.21.15.42:0/2174827676 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8cac0a4f00 0x7f8cac0a5330 secure :-1 s=THROTTLE_DONE pgs=1576 cs=0 l=1 rx=0x7f8c9c000e90 tx=0x7f8c9c010a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:08.232 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:08.220+0000 7f8ca27fc700 1 --2- 172.21.15.42:0/2174827676 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8c9403ae60 0x7f8c9403d370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:51:08.232 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:08.220+0000 7f8ca27fc700 1 -- 172.21.15.42:0/2174827676 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8783+0+0 (secure 0 0 0) 0x7f8c9c04b380 con 0x7f8cac0a4f00 2019-12-05T05:51:08.233 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:08.220+0000 7f8cb1a61700 1 --2- 172.21.15.42:0/2174827676 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8c9403ae60 0x7f8c9403d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:51:08.233 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:08.224+0000 7f8cb1a61700 1 --2- 172.21.15.42:0/2174827676 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8c9403ae60 0x7f8c9403d370 crc :-1 s=READY pgs=678 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:51:08.233 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:08.224+0000 7f8cb34c4700 1 -- 172.21.15.42:0/2174827676 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cac05d110 con 0x7f8cac0a4f00 2019-12-05T05:51:08.237 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:08.232+0000 7f8cb1260700 1 --2- 172.21.15.42:0/2174827676 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8cac0a4f00 0x7f8cac0a5330 secure :-1 s=THROTTLE_DONE pgs=1576 cs=0 l=1 rx=0x7f8c9c000e90 tx=0x7f8c9c010a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:08.237 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:08.232+0000 7f8ca27fc700 1 -- 172.21.15.42:0/2174827676 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f8c9c032400 con 0x7f8cac0a4f00 2019-12-05T05:51:08.412 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:08.408+0000 7f8cb34c4700 1 -- 172.21.15.42:0/2174827676 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8cac05b8c0 con 0x7f8cac0a4f00 2019-12-05T05:51:08.413 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:08.408+0000 7f8cb1260700 1 --2- 172.21.15.42:0/2174827676 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8cac0a4f00 0x7f8cac0a5330 secure :-1 s=THROTTLE_DONE pgs=1576 cs=0 l=1 rx=0x7f8c9c000e90 tx=0x7f8c9c010a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:08.413 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:08.408+0000 7f8ca27fc700 1 -- 172.21.15.42:0/2174827676 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 193 v193) v1 ==== 95+0+2103 (secure 0 0 0) 0x7f8c9c032660 con 0x7f8cac0a4f00 2019-12-05T05:51:08.413 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 193 2019-12-05T05:51:08.413 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:51:08.414 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":193,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":8430,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6835/5991897","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":5991897},{"type":"v1","addr":"172.21.15.42:6835","nonce":5991897}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":193}],"filesystems":[{"mdsmap":{"epoch":192,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:50:54.569377+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6245},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6245":{"gid":6245,"name":"b","rank":0,"incarnation":189,"state":"up:active","state_seq":25,"addr":"172.21.15.40:6833/2081111170","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2081111170},{"type":"v1","addr":"172.21.15.40:6833","nonce":2081111170}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:51:08.416 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:08.412+0000 7f8cb34c4700 1 -- 172.21.15.42:0/2174827676 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8c9403ae60 msgr2=0x7f8c9403d370 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:51:08.416 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:08.412+0000 7f8cb34c4700 1 --2- 172.21.15.42:0/2174827676 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8c9403ae60 0x7f8c9403d370 crc :-1 s=READY pgs=678 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:51:08.416 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:08.412+0000 7f8cb34c4700 1 -- 172.21.15.42:0/2174827676 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8cac0a4f00 msgr2=0x7f8cac0a5330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:51:08.416 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:08.412+0000 7f8cb34c4700 1 --2- 172.21.15.42:0/2174827676 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8cac0a4f00 0x7f8cac0a5330 secure :-1 s=READY pgs=1576 cs=0 l=1 rx=0x7f8c9c000e90 tx=0x7f8c9c010a20).stop 2019-12-05T05:51:08.417 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:08.412+0000 7f8cb34c4700 1 -- 172.21.15.42:0/2174827676 shutdown_connections 2019-12-05T05:51:08.417 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:08.412+0000 7f8cb34c4700 1 --2- 172.21.15.42:0/2174827676 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8c9403ae60 0x7f8c9403d370 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:51:08.417 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:08.412+0000 7f8cb34c4700 1 --2- 172.21.15.42:0/2174827676 >> v2:172.21.15.40:3300/0 conn(0x7f8cac131920 0x7f8cac131d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:51:08.417 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:08.412+0000 7f8cb34c4700 1 --2- 172.21.15.42:0/2174827676 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8cac11b2a0 0x7f8cac118450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:51:08.417 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:08.412+0000 7f8cb34c4700 1 --2- 172.21.15.42:0/2174827676 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8cac0a4f00 0x7f8cac0a5330 unknown :-1 s=CLOSED pgs=1576 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:51:08.417 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:08.412+0000 7f8cb34c4700 1 -- 172.21.15.42:0/2174827676 shutdown_connections 2019-12-05T05:51:08.418 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:08.412+0000 7f8cb34c4700 1 -- 172.21.15.42:0/2174827676 wait complete. 2019-12-05T05:51:08.418 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:08.412+0000 7f8cb34c4700 1 -- 172.21.15.42:0/2174827676 >> 172.21.15.42:0/2174827676 conn(0x7f8cac08fda0 msgr2=0x7f8cac09f620 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:51:08.425 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T05:51:10.426 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:51:10.533 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:10.524+0000 7fd9d023b700 1 Processor -- start 2019-12-05T05:51:10.533 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:10.524+0000 7fd9d023b700 1 -- start start 2019-12-05T05:51:10.534 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:10.524+0000 7fd9d023b700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd9c808df80 0x7fd9c809f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:51:10.534 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:10.524+0000 7fd9d023b700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd9c809f830 0x7fd9c814efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:51:10.534 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:10.524+0000 7fd9d023b700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9c809fc60 con 0x7fd9c80a7740 2019-12-05T05:51:10.534 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:10.524+0000 7fd9d023b700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9c80a0b00 con 0x7fd9c809f830 2019-12-05T05:51:10.534 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:10.524+0000 7fd9d023b700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9c808e5b0 con 0x7fd9c808df80 2019-12-05T05:51:10.535 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:10.524+0000 7fd9cdfd7700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd9c808df80 0x7fd9c809f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:51:10.535 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:10.524+0000 7fd9cdfd7700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd9c808df80 0x7fd9c809f2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:42576/0 (socket says 172.21.15.42:42576) 2019-12-05T05:51:10.535 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:10.524+0000 7fd9cdfd7700 1 -- 172.21.15.42:0/1862189720 learned_addr learned my addr 172.21.15.42:0/1862189720 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:51:10.535 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:10.524+0000 7fd9cefd9700 1 --2- 172.21.15.42:0/1862189720 >> v2:172.21.15.40:3300/0 conn(0x7fd9c809f830 0x7fd9c814efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:51:10.535 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:10.524+0000 7fd9cdfd7700 1 -- 172.21.15.42:0/1862189720 >> v2:172.21.15.40:3300/0 conn(0x7fd9c809f830 msgr2=0x7fd9c814efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:51:10.535 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:10.524+0000 7fd9cdfd7700 1 --2- 172.21.15.42:0/1862189720 >> v2:172.21.15.40:3300/0 conn(0x7fd9c809f830 0x7fd9c814efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:51:10.536 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:10.524+0000 7fd9cdfd7700 1 -- 172.21.15.42:0/1862189720 >> v1:172.21.15.40:6789/0 conn(0x7fd9c80a7740 legacy=0x7fd9c807e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:51:10.536 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:10.524+0000 7fd9cdfd7700 1 -- 172.21.15.42:0/1862189720 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9c80a0710 con 0x7fd9c808df80 2019-12-05T05:51:10.536 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:10.524+0000 7fd9cdfd7700 1 --2- 172.21.15.42:0/1862189720 >> v2:172.21.15.42:3300/0 conn(0x7fd9c808df80 0x7fd9c809f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fd9b8002f30 tx=0x7fd9b800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:10.536 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:10.524+0000 7fd9cdfd7700 1 --2- 172.21.15.42:0/1862189720 >> v2:172.21.15.42:3300/0 conn(0x7fd9c808df80 0x7fd9c809f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fd9b8002f30 tx=0x7fd9b800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:10.536 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:10.524+0000 7fd9cdfd7700 1 --2- 172.21.15.42:0/1862189720 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd9c808df80 0x7fd9c809f2e0 secure :-1 s=READY pgs=1577 cs=0 l=1 rx=0x7fd9b8002f30 tx=0x7fd9b800fd30).ready entity=mon.1 client_cookie=24cc35172232c6c server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:51:10.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:10.524+0000 7fd9cdfd7700 1 --2- 172.21.15.42:0/1862189720 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd9c808df80 0x7fd9c809f2e0 secure :-1 s=THROTTLE_DONE pgs=1577 cs=0 l=1 rx=0x7fd9b8002f30 tx=0x7fd9b800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:10.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:10.524+0000 7fd9cd7d6700 1 -- 172.21.15.42:0/1862189720 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd9b8008c90 con 0x7fd9c808df80 2019-12-05T05:51:10.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:10.524+0000 7fd9cdfd7700 1 --2- 172.21.15.42:0/1862189720 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd9c808df80 0x7fd9c809f2e0 secure :-1 s=THROTTLE_DONE pgs=1577 cs=0 l=1 rx=0x7fd9b8002f30 tx=0x7fd9b800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:10.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:10.524+0000 7fd9cdfd7700 1 --2- 172.21.15.42:0/1862189720 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd9c808df80 0x7fd9c809f2e0 secure :-1 s=THROTTLE_DONE pgs=1577 cs=0 l=1 rx=0x7fd9b8002f30 tx=0x7fd9b800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:10.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:10.524+0000 7fd9cd7d6700 1 -- 172.21.15.42:0/1862189720 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9b80088b0 con 0x7fd9c808df80 2019-12-05T05:51:10.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:10.524+0000 7fd9cd7d6700 1 -- 172.21.15.42:0/1862189720 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd9b8011ad0 con 0x7fd9c808df80 2019-12-05T05:51:10.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:10.524+0000 7fd9d023b700 1 -- 172.21.15.42:0/1862189720 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd9c808df80 msgr2=0x7fd9c809f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:51:10.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:10.524+0000 7fd9d023b700 1 --2- 172.21.15.42:0/1862189720 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd9c808df80 0x7fd9c809f2e0 secure :-1 s=READY pgs=1577 cs=0 l=1 rx=0x7fd9b8002f30 tx=0x7fd9b800fd30).stop 2019-12-05T05:51:10.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:10.528+0000 7fd9d023b700 1 -- 172.21.15.42:0/1862189720 shutdown_connections 2019-12-05T05:51:10.539 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:10.528+0000 7fd9d023b700 1 --2- 172.21.15.42:0/1862189720 >> v2:172.21.15.40:3300/0 conn(0x7fd9c809f830 0x7fd9c814efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:51:10.539 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:10.528+0000 7fd9d023b700 1 --2- 172.21.15.42:0/1862189720 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd9c808df80 0x7fd9c809f2e0 unknown :-1 s=CLOSED pgs=1577 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:51:10.539 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:10.528+0000 7fd9d023b700 1 -- 172.21.15.42:0/1862189720 shutdown_connections 2019-12-05T05:51:10.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:10.528+0000 7fd9d023b700 1 -- 172.21.15.42:0/1862189720 wait complete. 2019-12-05T05:51:10.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:10.528+0000 7fd9d023b700 1 -- 172.21.15.42:0/1862189720 >> 172.21.15.42:0/1862189720 conn(0x7fd9c80aabe0 msgr2=0x7fd9c80a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:51:10.547 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:10.528+0000 7fd9d023b700 1 Processor -- start 2019-12-05T05:51:10.547 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:10.532+0000 7fd9d023b700 1 -- start start 2019-12-05T05:51:10.547 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:10.532+0000 7fd9d023b700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd9c80a3a60 0x7fd9c807a930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:51:10.547 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:10.532+0000 7fd9d023b700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd9c80788d0 0x7fd9c8078d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:51:10.548 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:10.532+0000 7fd9d023b700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9c814ed10 con 0x7fd9c8144e30 2019-12-05T05:51:10.548 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:10.532+0000 7fd9d023b700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9c807a230 con 0x7fd9c80a3a60 2019-12-05T05:51:10.548 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:10.532+0000 7fd9d023b700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9c8145370 con 0x7fd9c80788d0 2019-12-05T05:51:10.548 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:10.532+0000 7fd9cdfd7700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fd9c8144e30 0x7fd9c807bf00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:42620/0 (socket says 172.21.15.42:42620) 2019-12-05T05:51:10.548 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:10.532+0000 7fd9cdfd7700 1 -- 172.21.15.42:0/4271179916 learned_addr learned my addr 172.21.15.42:0/4271179916 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:51:10.548 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:10.532+0000 7fd9cefd9700 1 --2- 172.21.15.42:0/4271179916 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd9c80788d0 0x7fd9c8078d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:51:10.548 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:10.532+0000 7fd9ce7d8700 1 --2- 172.21.15.42:0/4271179916 >> v2:172.21.15.40:3300/0 conn(0x7fd9c80a3a60 0x7fd9c807a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:51:10.549 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:10.532+0000 7fd9beffd700 1 -- 172.21.15.42:0/4271179916 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2222837373 0 0) 0x7fd9c814ed10 con 0x7fd9c8144e30 2019-12-05T05:51:10.549 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:10.532+0000 7fd9beffd700 1 -- 172.21.15.42:0/4271179916 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9b00030b0 con 0x7fd9c8144e30 2019-12-05T05:51:10.549 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:10.532+0000 7fd9ce7d8700 1 -- 172.21.15.42:0/4271179916 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd9c80788d0 msgr2=0x7fd9c8078d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:51:10.549 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:10.532+0000 7fd9ce7d8700 1 --2- 172.21.15.42:0/4271179916 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd9c80788d0 0x7fd9c8078d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:51:10.549 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:10.532+0000 7fd9ce7d8700 1 -- 172.21.15.42:0/4271179916 >> v1:172.21.15.42:6789/0 conn(0x7fd9c8144e30 legacy=0x7fd9c807bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:51:10.549 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:10.532+0000 7fd9ce7d8700 1 -- 172.21.15.42:0/4271179916 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9c8130120 con 0x7fd9c80a3a60 2019-12-05T05:51:10.549 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:10.532+0000 7fd9ce7d8700 1 --2- 172.21.15.42:0/4271179916 >> v2:172.21.15.40:3300/0 conn(0x7fd9c80a3a60 0x7fd9c807a930 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fd9c8145530 tx=0x7fd9c400c6a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:10.550 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:10.532+0000 7fd9ce7d8700 1 --2- 172.21.15.42:0/4271179916 >> v2:172.21.15.40:3300/0 conn(0x7fd9c80a3a60 0x7fd9c807a930 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fd9c8145530 tx=0x7fd9c400c6a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:10.550 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:10.532+0000 7fd9ce7d8700 1 --2- 172.21.15.42:0/4271179916 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd9c80a3a60 0x7fd9c807a930 secure :-1 s=READY pgs=1206 cs=0 l=1 rx=0x7fd9c8145530 tx=0x7fd9c400c6a0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:51:10.550 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:10.532+0000 7fd9ce7d8700 1 --2- 172.21.15.42:0/4271179916 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd9c80a3a60 0x7fd9c807a930 secure :-1 s=THROTTLE_DONE pgs=1206 cs=0 l=1 rx=0x7fd9c8145530 tx=0x7fd9c400c6a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:10.550 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:10.532+0000 7fd9ce7d8700 1 --2- 172.21.15.42:0/4271179916 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd9c80a3a60 0x7fd9c807a930 secure :-1 s=THROTTLE_DONE pgs=1206 cs=0 l=1 rx=0x7fd9c8145530 tx=0x7fd9c400c6a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:10.550 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:10.532+0000 7fd9beffd700 1 -- 172.21.15.42:0/4271179916 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd9c40101c0 con 0x7fd9c80a3a60 2019-12-05T05:51:10.551 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:10.532+0000 7fd9ce7d8700 1 --2- 172.21.15.42:0/4271179916 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd9c80a3a60 0x7fd9c807a930 secure :-1 s=THROTTLE_DONE pgs=1206 cs=0 l=1 rx=0x7fd9c8145530 tx=0x7fd9c400c6a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:10.551 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:10.532+0000 7fd9beffd700 1 -- 172.21.15.42:0/4271179916 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9c400cb10 con 0x7fd9c80a3a60 2019-12-05T05:51:10.551 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:10.532+0000 7fd9beffd700 1 -- 172.21.15.42:0/4271179916 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd9c40124d0 con 0x7fd9c80a3a60 2019-12-05T05:51:10.551 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:10.532+0000 7fd9d023b700 1 -- 172.21.15.42:0/4271179916 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9c812f110 con 0x7fd9c80a3a60 2019-12-05T05:51:10.551 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:10.532+0000 7fd9d023b700 1 -- 172.21.15.42:0/4271179916 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9c812f520 con 0x7fd9c80a3a60 2019-12-05T05:51:10.551 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:10.536+0000 7fd9ce7d8700 1 --2- 172.21.15.42:0/4271179916 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd9c80a3a60 0x7fd9c807a930 secure :-1 s=THROTTLE_DONE pgs=1206 cs=0 l=1 rx=0x7fd9c8145530 tx=0x7fd9c400c6a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:10.551 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:10.536+0000 7fd9beffd700 1 -- 172.21.15.42:0/4271179916 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fd9c4013060 con 0x7fd9c80a3a60 2019-12-05T05:51:10.551 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:10.536+0000 7fd9ce7d8700 1 --2- 172.21.15.42:0/4271179916 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd9c80a3a60 0x7fd9c807a930 secure :-1 s=THROTTLE_DONE pgs=1206 cs=0 l=1 rx=0x7fd9c8145530 tx=0x7fd9c400c6a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:10.552 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:10.536+0000 7fd9beffd700 1 --2- 172.21.15.42:0/4271179916 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd9b003b250 0x7fd9b003d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:51:10.552 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:10.536+0000 7fd9beffd700 1 -- 172.21.15.42:0/4271179916 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8783+0+0 (secure 0 0 0) 0x7fd9c404b0c0 con 0x7fd9c80a3a60 2019-12-05T05:51:10.552 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:10.536+0000 7fd9cdfd7700 1 --2- 172.21.15.42:0/4271179916 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd9b003b250 0x7fd9b003d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:51:10.552 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:10.536+0000 7fd9cdfd7700 1 --2- 172.21.15.42:0/4271179916 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd9b003b250 0x7fd9b003d760 crc :-1 s=READY pgs=679 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:51:10.552 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:10.540+0000 7fd9d023b700 1 -- 172.21.15.42:0/4271179916 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9c807bcf0 con 0x7fd9c80a3a60 2019-12-05T05:51:10.553 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:10.548+0000 7fd9ce7d8700 1 --2- 172.21.15.42:0/4271179916 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd9c80a3a60 0x7fd9c807a930 secure :-1 s=THROTTLE_DONE pgs=1206 cs=0 l=1 rx=0x7fd9c8145530 tx=0x7fd9c400c6a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:10.553 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:10.548+0000 7fd9beffd700 1 -- 172.21.15.42:0/4271179916 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fd9c4032100 con 0x7fd9c80a3a60 2019-12-05T05:51:10.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:10.716+0000 7fd9d023b700 1 -- 172.21.15.42:0/4271179916 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd9c805d110 con 0x7fd9c80a3a60 2019-12-05T05:51:10.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:10.716+0000 7fd9ce7d8700 1 --2- 172.21.15.42:0/4271179916 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd9c80a3a60 0x7fd9c807a930 secure :-1 s=THROTTLE_DONE pgs=1206 cs=0 l=1 rx=0x7fd9c8145530 tx=0x7fd9c400c6a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:10.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:10.716+0000 7fd9beffd700 1 -- 172.21.15.42:0/4271179916 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 193 v193) v1 ==== 95+0+2103 (secure 0 0 0) 0x7fd9c4032360 con 0x7fd9c80a3a60 2019-12-05T05:51:10.725 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:51:10.725 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":193,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":8430,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6835/5991897","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":5991897},{"type":"v1","addr":"172.21.15.42:6835","nonce":5991897}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":193}],"filesystems":[{"mdsmap":{"epoch":192,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:50:54.569377+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6245},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6245":{"gid":6245,"name":"b","rank":0,"incarnation":189,"state":"up:active","state_seq":25,"addr":"172.21.15.40:6833/2081111170","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2081111170},{"type":"v1","addr":"172.21.15.40:6833","nonce":2081111170}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:51:10.725 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 193 2019-12-05T05:51:10.728 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:10.724+0000 7fd9d023b700 1 -- 172.21.15.42:0/4271179916 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd9b003b250 msgr2=0x7fd9b003d760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:51:10.728 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:10.724+0000 7fd9d023b700 1 --2- 172.21.15.42:0/4271179916 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd9b003b250 0x7fd9b003d760 crc :-1 s=READY pgs=679 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:51:10.728 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:10.724+0000 7fd9d023b700 1 -- 172.21.15.42:0/4271179916 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd9c80a3a60 msgr2=0x7fd9c807a930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:51:10.728 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:10.724+0000 7fd9d023b700 1 --2- 172.21.15.42:0/4271179916 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd9c80a3a60 0x7fd9c807a930 secure :-1 s=READY pgs=1206 cs=0 l=1 rx=0x7fd9c8145530 tx=0x7fd9c400c6a0).stop 2019-12-05T05:51:10.728 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:10.724+0000 7fd9d023b700 1 -- 172.21.15.42:0/4271179916 shutdown_connections 2019-12-05T05:51:10.728 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:10.724+0000 7fd9d023b700 1 --2- 172.21.15.42:0/4271179916 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd9b003b250 0x7fd9b003d760 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:51:10.729 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:10.724+0000 7fd9d023b700 1 --2- 172.21.15.42:0/4271179916 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd9c80788d0 0x7fd9c8078d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:51:10.729 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:10.724+0000 7fd9d023b700 1 --2- 172.21.15.42:0/4271179916 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd9c80a3a60 0x7fd9c807a930 unknown :-1 s=CLOSED pgs=1206 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:51:10.729 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:10.724+0000 7fd9d023b700 1 -- 172.21.15.42:0/4271179916 shutdown_connections 2019-12-05T05:51:10.729 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:10.724+0000 7fd9d023b700 1 -- 172.21.15.42:0/4271179916 wait complete. 2019-12-05T05:51:10.729 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:10.724+0000 7fd9d023b700 1 -- 172.21.15.42:0/4271179916 >> 172.21.15.42:0/4271179916 conn(0x7fd9c80a9b80 msgr2=0x7fd9c80a6a60 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:51:10.743 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T05:51:11.987 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:51:11.991 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:51:12.745 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:51:12.851 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:12.840+0000 7fe346bd4700 1 Processor -- start 2019-12-05T05:51:12.851 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:12.840+0000 7fe346bd4700 1 -- start start 2019-12-05T05:51:12.852 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:12.840+0000 7fe346bd4700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe3400a7730 0x7fe34008df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:51:12.852 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:12.840+0000 7fe346bd4700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe34009fad0 0x7fe34007eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:51:12.852 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:12.840+0000 7fe346bd4700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3400a0700 con 0x7fe34007e710 2019-12-05T05:51:12.852 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:12.840+0000 7fe346bd4700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3400a0af0 con 0x7fe3400a7730 2019-12-05T05:51:12.852 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:12.840+0000 7fe346bd4700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe34008e5a0 con 0x7fe34009fad0 2019-12-05T05:51:12.852 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:12.840+0000 7fe345972700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe34009fad0 0x7fe34007eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:51:12.853 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:12.840+0000 7fe344970700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fe34007e710 0x7fe34009f2d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:52074/0 (socket says 172.21.15.42:52074) 2019-12-05T05:51:12.853 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:12.844+0000 7fe344970700 1 -- 172.21.15.42:0/183956291 learned_addr learned my addr 172.21.15.42:0/183956291 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:51:12.853 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:12.840+0000 7fe345171700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe3400a7730 0x7fe34008df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:51:12.853 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:12.844+0000 7fe333fff700 1 -- 172.21.15.42:0/183956291 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2879275823 0 0) 0x7fe3400a0700 con 0x7fe34007e710 2019-12-05T05:51:12.853 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:12.844+0000 7fe345171700 1 -- 172.21.15.42:0/183956291 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe34009fad0 msgr2=0x7fe34007eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:51:12.853 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:12.844+0000 7fe345171700 1 --2- 172.21.15.42:0/183956291 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe34009fad0 0x7fe34007eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:51:12.853 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:12.844+0000 7fe345171700 1 -- 172.21.15.42:0/183956291 >> v1:172.21.15.40:6789/0 conn(0x7fe34007e710 legacy=0x7fe34009f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:51:12.854 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:12.844+0000 7fe345171700 1 -- 172.21.15.42:0/183956291 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe34007f090 con 0x7fe3400a7730 2019-12-05T05:51:12.854 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:12.844+0000 7fe345972700 1 --2- 172.21.15.42:0/183956291 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe34009fad0 0x7fe34007eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:51:12.854 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:12.844+0000 7fe345171700 1 --2- 172.21.15.42:0/183956291 >> v2:172.21.15.40:3300/0 conn(0x7fe3400a7730 0x7fe34008df70 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fe334008790 tx=0x7fe33400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:12.854 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:12.844+0000 7fe345171700 1 --2- 172.21.15.42:0/183956291 >> v2:172.21.15.40:3300/0 conn(0x7fe3400a7730 0x7fe34008df70 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fe334008790 tx=0x7fe33400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:12.854 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:12.844+0000 7fe345171700 1 --2- 172.21.15.42:0/183956291 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe3400a7730 0x7fe34008df70 secure :-1 s=READY pgs=1208 cs=0 l=1 rx=0x7fe334008790 tx=0x7fe33400fc70).ready entity=mon.0 client_cookie=7d70280626e8ae4b server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:51:12.854 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:12.844+0000 7fe345171700 1 --2- 172.21.15.42:0/183956291 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe3400a7730 0x7fe34008df70 secure :-1 s=THROTTLE_DONE pgs=1208 cs=0 l=1 rx=0x7fe334008790 tx=0x7fe33400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:12.854 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:12.844+0000 7fe333fff700 1 -- 172.21.15.42:0/183956291 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe334008bb0 con 0x7fe3400a7730 2019-12-05T05:51:12.855 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:12.844+0000 7fe345171700 1 --2- 172.21.15.42:0/183956291 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe3400a7730 0x7fe34008df70 secure :-1 s=THROTTLE_DONE pgs=1208 cs=0 l=1 rx=0x7fe334008790 tx=0x7fe33400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:12.855 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:12.844+0000 7fe345171700 1 --2- 172.21.15.42:0/183956291 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe3400a7730 0x7fe34008df70 secure :-1 s=THROTTLE_DONE pgs=1208 cs=0 l=1 rx=0x7fe334008790 tx=0x7fe33400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:12.855 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:12.844+0000 7fe333fff700 1 -- 172.21.15.42:0/183956291 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3340087d0 con 0x7fe3400a7730 2019-12-05T05:51:12.855 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:12.844+0000 7fe333fff700 1 -- 172.21.15.42:0/183956291 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe334011860 con 0x7fe3400a7730 2019-12-05T05:51:12.855 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:12.844+0000 7fe346bd4700 1 -- 172.21.15.42:0/183956291 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe3400a7730 msgr2=0x7fe34008df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:51:12.855 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:12.844+0000 7fe346bd4700 1 --2- 172.21.15.42:0/183956291 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe3400a7730 0x7fe34008df70 secure :-1 s=READY pgs=1208 cs=0 l=1 rx=0x7fe334008790 tx=0x7fe33400fc70).stop 2019-12-05T05:51:12.855 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:12.844+0000 7fe346bd4700 1 -- 172.21.15.42:0/183956291 shutdown_connections 2019-12-05T05:51:12.856 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:12.844+0000 7fe346bd4700 1 --2- 172.21.15.42:0/183956291 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe34009fad0 0x7fe34007eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:51:12.856 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:12.844+0000 7fe346bd4700 1 --2- 172.21.15.42:0/183956291 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe3400a7730 0x7fe34008df70 unknown :-1 s=CLOSED pgs=1208 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:51:12.856 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:12.844+0000 7fe346bd4700 1 -- 172.21.15.42:0/183956291 shutdown_connections 2019-12-05T05:51:12.856 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:12.844+0000 7fe346bd4700 1 -- 172.21.15.42:0/183956291 wait complete. 2019-12-05T05:51:12.856 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:12.844+0000 7fe346bd4700 1 -- 172.21.15.42:0/183956291 >> 172.21.15.42:0/183956291 conn(0x7fe3400aabd0 msgr2=0x7fe3400a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:51:12.860 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:12.848+0000 7fe346bd4700 1 Processor -- start 2019-12-05T05:51:12.860 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:12.848+0000 7fe346bd4700 1 -- start start 2019-12-05T05:51:12.860 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:12.848+0000 7fe346bd4700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe34008f070 0x7fe34011e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:51:12.861 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:12.848+0000 7fe346bd4700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe34008f730 0x7fe340133590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:51:12.861 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:12.848+0000 7fe346bd4700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe340133ae0 0x7fe3400a4a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:51:12.861 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:12.848+0000 7fe346bd4700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe34011bf90 con 0x7fe34008f070 2019-12-05T05:51:12.861 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:12.848+0000 7fe346bd4700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe34011e590 con 0x7fe340133ae0 2019-12-05T05:51:12.861 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:12.848+0000 7fe346bd4700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe34011e960 con 0x7fe34008f730 2019-12-05T05:51:12.861 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:12.852+0000 7fe345972700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe340133ae0 0x7fe3400a4a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:51:12.861 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:12.852+0000 7fe345171700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe34008f070 0x7fe34011e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:51:12.862 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:12.852+0000 7fe344970700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe34008f730 0x7fe340133590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:51:12.862 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:12.852+0000 7fe345972700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe340133ae0 0x7fe3400a4a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:42602/0 (socket says 172.21.15.42:42602) 2019-12-05T05:51:12.862 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:12.852+0000 7fe345171700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe34008f070 0x7fe34011e040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.42:58646/0 (socket says 172.21.15.42:58646) 2019-12-05T05:51:12.862 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:12.852+0000 7fe345972700 1 -- 172.21.15.42:0/1520395022 learned_addr learned my addr 172.21.15.42:0/1520395022 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:51:12.862 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:12.852+0000 7fe345972700 1 -- 172.21.15.42:0/1520395022 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe34008f730 msgr2=0x7fe340133590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:51:12.862 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:12.852+0000 7fe345972700 1 --2- 172.21.15.42:0/1520395022 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe34008f730 0x7fe340133590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:51:12.862 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:12.852+0000 7fe345972700 1 -- 172.21.15.42:0/1520395022 >> v2:172.21.15.40:3300/0 conn(0x7fe34008f070 msgr2=0x7fe34011e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:51:12.863 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:12.852+0000 7fe345972700 1 --2- 172.21.15.42:0/1520395022 >> v2:172.21.15.40:3300/0 conn(0x7fe34008f070 0x7fe34011e040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:51:12.863 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:12.852+0000 7fe345972700 1 -- 172.21.15.42:0/1520395022 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3401164f0 con 0x7fe340133ae0 2019-12-05T05:51:12.863 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:12.852+0000 7fe345972700 1 --2- 172.21.15.42:0/1520395022 >> v2:172.21.15.42:3300/0 conn(0x7fe340133ae0 0x7fe3400a4a40 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fe3380032b0 tx=0x7fe338010f80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:12.863 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:12.852+0000 7fe345972700 1 --2- 172.21.15.42:0/1520395022 >> v2:172.21.15.42:3300/0 conn(0x7fe340133ae0 0x7fe3400a4a40 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fe3380032b0 tx=0x7fe338010f80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:12.863 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:12.852+0000 7fe345972700 1 --2- 172.21.15.42:0/1520395022 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe340133ae0 0x7fe3400a4a40 secure :-1 s=READY pgs=1579 cs=0 l=1 rx=0x7fe3380032b0 tx=0x7fe338010f80).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:51:12.863 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:12.852+0000 7fe345972700 1 --2- 172.21.15.42:0/1520395022 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe340133ae0 0x7fe3400a4a40 secure :-1 s=THROTTLE_DONE pgs=1579 cs=0 l=1 rx=0x7fe3380032b0 tx=0x7fe338010f80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:12.863 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:12.852+0000 7fe331ffb700 1 -- 172.21.15.42:0/1520395022 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe338007160 con 0x7fe340133ae0 2019-12-05T05:51:12.864 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:12.852+0000 7fe345972700 1 --2- 172.21.15.42:0/1520395022 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe340133ae0 0x7fe3400a4a40 secure :-1 s=THROTTLE_DONE pgs=1579 cs=0 l=1 rx=0x7fe3380032b0 tx=0x7fe338010f80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:12.864 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:12.852+0000 7fe345972700 1 --2- 172.21.15.42:0/1520395022 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe340133ae0 0x7fe3400a4a40 secure :-1 s=THROTTLE_DONE pgs=1579 cs=0 l=1 rx=0x7fe3380032b0 tx=0x7fe338010f80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:12.864 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:12.852+0000 7fe331ffb700 1 -- 172.21.15.42:0/1520395022 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe338005bc0 con 0x7fe340133ae0 2019-12-05T05:51:12.864 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:12.852+0000 7fe346bd4700 1 -- 172.21.15.42:0/1520395022 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3400a4f90 con 0x7fe340133ae0 2019-12-05T05:51:12.864 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:12.852+0000 7fe331ffb700 1 -- 172.21.15.42:0/1520395022 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe338012600 con 0x7fe340133ae0 2019-12-05T05:51:12.864 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:12.852+0000 7fe346bd4700 1 -- 172.21.15.42:0/1520395022 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3400a5320 con 0x7fe340133ae0 2019-12-05T05:51:12.864 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:12.852+0000 7fe345972700 1 --2- 172.21.15.42:0/1520395022 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe340133ae0 0x7fe3400a4a40 secure :-1 s=THROTTLE_DONE pgs=1579 cs=0 l=1 rx=0x7fe3380032b0 tx=0x7fe338010f80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:12.865 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:12.852+0000 7fe331ffb700 1 -- 172.21.15.42:0/1520395022 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fe33800e180 con 0x7fe340133ae0 2019-12-05T05:51:12.865 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:12.852+0000 7fe345972700 1 --2- 172.21.15.42:0/1520395022 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe340133ae0 0x7fe3400a4a40 secure :-1 s=THROTTLE_DONE pgs=1579 cs=0 l=1 rx=0x7fe3380032b0 tx=0x7fe338010f80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:12.865 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:12.856+0000 7fe331ffb700 1 --2- 172.21.15.42:0/1520395022 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe32003ae60 0x7fe32003d370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:51:12.865 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:12.856+0000 7fe331ffb700 1 -- 172.21.15.42:0/1520395022 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8783+0+0 (secure 0 0 0) 0x7fe33804b020 con 0x7fe340133ae0 2019-12-05T05:51:12.865 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:12.856+0000 7fe345171700 1 --2- 172.21.15.42:0/1520395022 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe32003ae60 0x7fe32003d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:51:12.865 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:12.856+0000 7fe345171700 1 --2- 172.21.15.42:0/1520395022 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe32003ae60 0x7fe32003d370 crc :-1 s=READY pgs=680 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:51:12.866 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:12.856+0000 7fe346bd4700 1 -- 172.21.15.42:0/1520395022 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe34005d100 con 0x7fe340133ae0 2019-12-05T05:51:12.866 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:12.860+0000 7fe345972700 1 --2- 172.21.15.42:0/1520395022 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe340133ae0 0x7fe3400a4a40 secure :-1 s=THROTTLE_DONE pgs=1579 cs=0 l=1 rx=0x7fe3380032b0 tx=0x7fe338010f80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:12.866 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:12.860+0000 7fe331ffb700 1 -- 172.21.15.42:0/1520395022 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fe3380320a0 con 0x7fe340133ae0 2019-12-05T05:51:13.028 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:13.024+0000 7fe346bd4700 1 -- 172.21.15.42:0/1520395022 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe34005b8b0 con 0x7fe340133ae0 2019-12-05T05:51:13.029 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:13.024+0000 7fe345972700 1 --2- 172.21.15.42:0/1520395022 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe340133ae0 0x7fe3400a4a40 secure :-1 s=THROTTLE_DONE pgs=1579 cs=0 l=1 rx=0x7fe3380032b0 tx=0x7fe338010f80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:13.029 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:13.024+0000 7fe331ffb700 1 -- 172.21.15.42:0/1520395022 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 193 v193) v1 ==== 95+0+2103 (secure 0 0 0) 0x7fe338032300 con 0x7fe340133ae0 2019-12-05T05:51:13.032 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 193 2019-12-05T05:51:13.032 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:51:13.033 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":193,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":8430,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6835/5991897","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":5991897},{"type":"v1","addr":"172.21.15.42:6835","nonce":5991897}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":193}],"filesystems":[{"mdsmap":{"epoch":192,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:50:54.569377+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6245},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6245":{"gid":6245,"name":"b","rank":0,"incarnation":189,"state":"up:active","state_seq":25,"addr":"172.21.15.40:6833/2081111170","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2081111170},{"type":"v1","addr":"172.21.15.40:6833","nonce":2081111170}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:51:13.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:13.028+0000 7fe346bd4700 1 -- 172.21.15.42:0/1520395022 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe32003ae60 msgr2=0x7fe32003d370 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:51:13.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:13.028+0000 7fe346bd4700 1 --2- 172.21.15.42:0/1520395022 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe32003ae60 0x7fe32003d370 crc :-1 s=READY pgs=680 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:51:13.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:13.028+0000 7fe346bd4700 1 -- 172.21.15.42:0/1520395022 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe340133ae0 msgr2=0x7fe3400a4a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:51:13.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:13.028+0000 7fe346bd4700 1 --2- 172.21.15.42:0/1520395022 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe340133ae0 0x7fe3400a4a40 secure :-1 s=READY pgs=1579 cs=0 l=1 rx=0x7fe3380032b0 tx=0x7fe338010f80).stop 2019-12-05T05:51:13.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:13.028+0000 7fe346bd4700 1 -- 172.21.15.42:0/1520395022 shutdown_connections 2019-12-05T05:51:13.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:13.028+0000 7fe346bd4700 1 --2- 172.21.15.42:0/1520395022 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe32003ae60 0x7fe32003d370 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:51:13.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:13.028+0000 7fe346bd4700 1 --2- 172.21.15.42:0/1520395022 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe340133ae0 0x7fe3400a4a40 unknown :-1 s=CLOSED pgs=1579 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:51:13.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:13.028+0000 7fe346bd4700 1 --2- 172.21.15.42:0/1520395022 >> v2:172.21.15.40:3300/0 conn(0x7fe34008f070 0x7fe34011e040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:51:13.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:13.028+0000 7fe346bd4700 1 --2- 172.21.15.42:0/1520395022 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe34008f730 0x7fe340133590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:51:13.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:13.028+0000 7fe346bd4700 1 -- 172.21.15.42:0/1520395022 shutdown_connections 2019-12-05T05:51:13.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:13.028+0000 7fe346bd4700 1 -- 172.21.15.42:0/1520395022 wait complete. 2019-12-05T05:51:13.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:13.028+0000 7fe346bd4700 1 -- 172.21.15.42:0/1520395022 >> 172.21.15.42:0/1520395022 conn(0x7fe3400aabd0 msgr2=0x7fe3400596f0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:51:13.044 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T05:51:15.045 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:51:15.152 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:15.140+0000 7f4886761700 1 Processor -- start 2019-12-05T05:51:15.152 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:15.144+0000 7f4886761700 1 -- start start 2019-12-05T05:51:15.153 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:15.144+0000 7f4886761700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f48800a7740 0x7f488008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:51:15.153 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:15.144+0000 7f4886761700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48800a0710 con 0x7f488007e720 2019-12-05T05:51:15.153 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:15.144+0000 7f4886761700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48800a9ba0 con 0x7f488009fae0 2019-12-05T05:51:15.153 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:15.144+0000 7f4886761700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48800a0b00 con 0x7f48800a7740 2019-12-05T05:51:15.153 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:15.144+0000 7f48854ff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f488009fae0 0x7f488014efe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:42648/0 (socket says 172.21.15.42:42648) 2019-12-05T05:51:15.153 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:15.144+0000 7f48854ff700 1 -- 172.21.15.42:0/98913877 learned_addr learned my addr 172.21.15.42:0/98913877 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:51:15.154 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:15.144+0000 7f4884cfe700 1 --2- 172.21.15.42:0/98913877 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f48800a7740 0x7f488008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:51:15.154 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:15.144+0000 7f487f7fe700 1 -- 172.21.15.42:0/98913877 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1001428394 0 0) 0x7f48800a9ba0 con 0x7f488009fae0 2019-12-05T05:51:15.154 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:15.144+0000 7f487f7fe700 1 -- 172.21.15.42:0/98913877 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48680030b0 con 0x7f488009fae0 2019-12-05T05:51:15.154 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:15.144+0000 7f487f7fe700 1 -- 172.21.15.42:0/98913877 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2469529183 0 0) 0x7f48800a0710 con 0x7f488007e720 2019-12-05T05:51:15.154 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:15.144+0000 7f487f7fe700 1 -- 172.21.15.42:0/98913877 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48800a9ba0 con 0x7f488007e720 2019-12-05T05:51:15.154 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:15.144+0000 7f4884cfe700 1 -- 172.21.15.42:0/98913877 >> v1:172.21.15.42:6789/0 conn(0x7f488009fae0 legacy=0x7f488014efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:51:15.154 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:15.144+0000 7f487f7fe700 1 -- 172.21.15.42:0/98913877 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 965488396 0 0) 0x7f48680030b0 con 0x7f488009fae0 2019-12-05T05:51:15.155 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:15.144+0000 7f4884cfe700 1 -- 172.21.15.42:0/98913877 >> v1:172.21.15.40:6789/0 conn(0x7f488007e720 legacy=0x7f488009f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:51:15.155 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:15.144+0000 7f4884cfe700 1 -- 172.21.15.42:0/98913877 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f488007eb50 con 0x7f48800a7740 2019-12-05T05:51:15.155 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:15.144+0000 7f4884cfe700 1 --2- 172.21.15.42:0/98913877 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f48800a7740 0x7f488008df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f4870008b10 tx=0x7f4878001950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:15.155 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:15.144+0000 7f4884cfe700 1 --2- 172.21.15.42:0/98913877 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f48800a7740 0x7f488008df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f4870008b10 tx=0x7f4878001950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:15.155 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:15.144+0000 7f4884cfe700 1 --2- 172.21.15.42:0/98913877 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f48800a7740 0x7f488008df80 secure :-1 s=READY pgs=351 cs=0 l=1 rx=0x7f4870008b10 tx=0x7f4878001950).ready entity=mon.2 client_cookie=6835a6eb13ee3411 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:51:15.155 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:15.144+0000 7f4884cfe700 1 --2- 172.21.15.42:0/98913877 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f48800a7740 0x7f488008df80 secure :-1 s=THROTTLE_DONE pgs=351 cs=0 l=1 rx=0x7f4870008b10 tx=0x7f4878001950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:15.156 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:15.144+0000 7f4884cfe700 1 --2- 172.21.15.42:0/98913877 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f48800a7740 0x7f488008df80 secure :-1 s=THROTTLE_DONE pgs=351 cs=0 l=1 rx=0x7f4870008b10 tx=0x7f4878001950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:15.156 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:15.144+0000 7f487f7fe700 1 -- 172.21.15.42:0/98913877 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4878001750 con 0x7f48800a7740 2019-12-05T05:51:15.156 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:15.144+0000 7f4884cfe700 1 --2- 172.21.15.42:0/98913877 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f48800a7740 0x7f488008df80 secure :-1 s=THROTTLE_DONE pgs=351 cs=0 l=1 rx=0x7f4870008b10 tx=0x7f4878001950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:15.156 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:15.144+0000 7f487f7fe700 1 -- 172.21.15.42:0/98913877 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4878001560 con 0x7f48800a7740 2019-12-05T05:51:15.156 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:15.144+0000 7f487f7fe700 1 -- 172.21.15.42:0/98913877 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4870011ac0 con 0x7f48800a7740 2019-12-05T05:51:15.156 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:15.144+0000 7f4886761700 1 -- 172.21.15.42:0/98913877 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f48800a7740 msgr2=0x7f488008df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:51:15.156 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:15.144+0000 7f4886761700 1 --2- 172.21.15.42:0/98913877 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f48800a7740 0x7f488008df80 secure :-1 s=READY pgs=351 cs=0 l=1 rx=0x7f4870008b10 tx=0x7f4878001950).stop 2019-12-05T05:51:15.157 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:15.148+0000 7f4886761700 1 -- 172.21.15.42:0/98913877 shutdown_connections 2019-12-05T05:51:15.157 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:15.148+0000 7f4886761700 1 --2- 172.21.15.42:0/98913877 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f48800a7740 0x7f488008df80 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:51:15.157 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:15.148+0000 7f4886761700 1 -- 172.21.15.42:0/98913877 shutdown_connections 2019-12-05T05:51:15.157 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:15.148+0000 7f4886761700 1 -- 172.21.15.42:0/98913877 wait complete. 2019-12-05T05:51:15.157 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:15.148+0000 7f4886761700 1 -- 172.21.15.42:0/98913877 >> 172.21.15.42:0/98913877 conn(0x7f48800aabe0 msgr2=0x7f48800a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:51:15.170 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:15.148+0000 7f4886761700 1 Processor -- start 2019-12-05T05:51:15.171 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:15.148+0000 7f4886761700 1 -- start start 2019-12-05T05:51:15.171 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:15.148+0000 7f4886761700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f48800a2960 0x7f48801309b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:51:15.171 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:15.148+0000 7f4886761700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f48800a1cd0 0x7f48801387c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:51:15.171 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:15.148+0000 7f4886761700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f488012cda0 con 0x7f4880130f00 2019-12-05T05:51:15.171 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:15.148+0000 7f4886761700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4880130470 con 0x7f48800a2960 2019-12-05T05:51:15.172 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:15.148+0000 7f4886761700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4880130710 con 0x7f48800a1cd0 2019-12-05T05:51:15.172 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:15.148+0000 7f48854ff700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f48800a1cd0 0x7f48801387c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:51:15.172 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:15.148+0000 7f4884cfe700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f48800a2960 0x7f48801309b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:51:15.172 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:15.148+0000 7f4884cfe700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f48800a2960 0x7f48801309b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.42:58660/0 (socket says 172.21.15.42:58660) 2019-12-05T05:51:15.172 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:15.148+0000 7f48854ff700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f48800a1cd0 0x7f48801387c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:43080/0 (socket says 172.21.15.42:43080) 2019-12-05T05:51:15.172 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:15.148+0000 7f4884cfe700 1 -- 172.21.15.42:0/2934564235 learned_addr learned my addr 172.21.15.42:0/2934564235 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:51:15.173 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:15.152+0000 7f4884cfe700 1 -- 172.21.15.42:0/2934564235 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f48800a1cd0 msgr2=0x7f48801387c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:51:15.173 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:15.152+0000 7f4884cfe700 1 --2- 172.21.15.42:0/2934564235 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f48800a1cd0 0x7f48801387c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:51:15.173 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:15.152+0000 7f4884cfe700 1 -- 172.21.15.42:0/2934564235 >> v1:172.21.15.40:6789/0 conn(0x7f4880130f00 legacy=0x7f48800a14d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:51:15.173 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:15.152+0000 7f4884cfe700 1 -- 172.21.15.42:0/2934564235 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48800a2590 con 0x7f48800a2960 2019-12-05T05:51:15.173 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:15.152+0000 7f48854ff700 1 --2- 172.21.15.42:0/2934564235 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f48800a1cd0 0x7f48801387c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:51:15.173 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:15.152+0000 7f4884cfe700 1 --2- 172.21.15.42:0/2934564235 >> v2:172.21.15.40:3300/0 conn(0x7f48800a2960 0x7f48801309b0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f48800a6c80 tx=0x7f4870007d20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:15.174 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:15.152+0000 7f4884cfe700 1 --2- 172.21.15.42:0/2934564235 >> v2:172.21.15.40:3300/0 conn(0x7f48800a2960 0x7f48801309b0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f48800a6c80 tx=0x7f4870007d20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:15.174 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:15.152+0000 7f4884cfe700 1 --2- 172.21.15.42:0/2934564235 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f48800a2960 0x7f48801309b0 secure :-1 s=READY pgs=1211 cs=0 l=1 rx=0x7f48800a6c80 tx=0x7f4870007d20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:51:15.174 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:15.152+0000 7f4884cfe700 1 --2- 172.21.15.42:0/2934564235 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f48800a2960 0x7f48801309b0 secure :-1 s=THROTTLE_DONE pgs=1211 cs=0 l=1 rx=0x7f48800a6c80 tx=0x7f4870007d20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:15.174 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:15.152+0000 7f487d7fa700 1 -- 172.21.15.42:0/2934564235 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f487000f780 con 0x7f48800a2960 2019-12-05T05:51:15.174 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:15.152+0000 7f4884cfe700 1 --2- 172.21.15.42:0/2934564235 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f48800a2960 0x7f48801309b0 secure :-1 s=THROTTLE_DONE pgs=1211 cs=0 l=1 rx=0x7f48800a6c80 tx=0x7f4870007d20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:15.174 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:15.152+0000 7f4884cfe700 1 --2- 172.21.15.42:0/2934564235 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f48800a2960 0x7f48801309b0 secure :-1 s=THROTTLE_DONE pgs=1211 cs=0 l=1 rx=0x7f48800a6c80 tx=0x7f4870007d20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:15.175 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:15.152+0000 7f487d7fa700 1 -- 172.21.15.42:0/2934564235 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48700116e0 con 0x7f48800a2960 2019-12-05T05:51:15.175 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:15.152+0000 7f4886761700 1 -- 172.21.15.42:0/2934564235 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48801429f0 con 0x7f48800a2960 2019-12-05T05:51:15.175 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:15.152+0000 7f487d7fa700 1 -- 172.21.15.42:0/2934564235 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4870005ab0 con 0x7f48800a2960 2019-12-05T05:51:15.175 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:15.152+0000 7f4886761700 1 -- 172.21.15.42:0/2934564235 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48800a2100 con 0x7f48800a2960 2019-12-05T05:51:15.175 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:15.152+0000 7f4884cfe700 1 --2- 172.21.15.42:0/2934564235 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f48800a2960 0x7f48801309b0 secure :-1 s=THROTTLE_DONE pgs=1211 cs=0 l=1 rx=0x7f48800a6c80 tx=0x7f4870007d20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:15.175 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:15.152+0000 7f487d7fa700 1 -- 172.21.15.42:0/2934564235 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f4870007150 con 0x7f48800a2960 2019-12-05T05:51:15.175 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:15.152+0000 7f4884cfe700 1 --2- 172.21.15.42:0/2934564235 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f48800a2960 0x7f48801309b0 secure :-1 s=THROTTLE_DONE pgs=1211 cs=0 l=1 rx=0x7f48800a6c80 tx=0x7f4870007d20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:15.176 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:15.156+0000 7f487d7fa700 1 --2- 172.21.15.42:0/2934564235 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f486003ad00 0x7f486003d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:51:15.176 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:15.156+0000 7f487d7fa700 1 -- 172.21.15.42:0/2934564235 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8783+0+0 (secure 0 0 0) 0x7f487004b440 con 0x7f48800a2960 2019-12-05T05:51:15.176 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:15.156+0000 7f487ffff700 1 --2- 172.21.15.42:0/2934564235 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f486003ad00 0x7f486003d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:51:15.176 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:15.156+0000 7f487ffff700 1 --2- 172.21.15.42:0/2934564235 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f486003ad00 0x7f486003d210 crc :-1 s=READY pgs=681 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:51:15.176 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:15.160+0000 7f4886761700 1 -- 172.21.15.42:0/2934564235 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f488005d110 con 0x7f48800a2960 2019-12-05T05:51:15.177 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:15.168+0000 7f4884cfe700 1 --2- 172.21.15.42:0/2934564235 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f48800a2960 0x7f48801309b0 secure :-1 s=THROTTLE_DONE pgs=1211 cs=0 l=1 rx=0x7f48800a6c80 tx=0x7f4870007d20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:15.177 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:15.168+0000 7f487d7fa700 1 -- 172.21.15.42:0/2934564235 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f4870005d50 con 0x7f48800a2960 2019-12-05T05:51:15.344 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:15.340+0000 7f4886761700 1 -- 172.21.15.42:0/2934564235 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f488005b8c0 con 0x7f48800a2960 2019-12-05T05:51:15.345 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:15.340+0000 7f4884cfe700 1 --2- 172.21.15.42:0/2934564235 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f48800a2960 0x7f48801309b0 secure :-1 s=THROTTLE_DONE pgs=1211 cs=0 l=1 rx=0x7f48800a6c80 tx=0x7f4870007d20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:15.345 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:15.340+0000 7f487d7fa700 1 -- 172.21.15.42:0/2934564235 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 193 v193) v1 ==== 95+0+2103 (secure 0 0 0) 0x7f487000dcb0 con 0x7f48800a2960 2019-12-05T05:51:15.347 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:51:15.347 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":193,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":8430,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6835/5991897","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":5991897},{"type":"v1","addr":"172.21.15.42:6835","nonce":5991897}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":193}],"filesystems":[{"mdsmap":{"epoch":192,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:50:54.569377+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6245},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6245":{"gid":6245,"name":"b","rank":0,"incarnation":189,"state":"up:active","state_seq":25,"addr":"172.21.15.40:6833/2081111170","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2081111170},{"type":"v1","addr":"172.21.15.40:6833","nonce":2081111170}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:51:15.348 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 193 2019-12-05T05:51:15.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:15.344+0000 7f4886761700 1 -- 172.21.15.42:0/2934564235 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f486003ad00 msgr2=0x7f486003d210 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:51:15.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:15.344+0000 7f4886761700 1 --2- 172.21.15.42:0/2934564235 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f486003ad00 0x7f486003d210 crc :-1 s=READY pgs=681 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:51:15.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:15.344+0000 7f4886761700 1 -- 172.21.15.42:0/2934564235 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f48800a2960 msgr2=0x7f48801309b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:51:15.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:15.344+0000 7f4886761700 1 --2- 172.21.15.42:0/2934564235 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f48800a2960 0x7f48801309b0 secure :-1 s=READY pgs=1211 cs=0 l=1 rx=0x7f48800a6c80 tx=0x7f4870007d20).stop 2019-12-05T05:51:15.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:15.344+0000 7f4886761700 1 -- 172.21.15.42:0/2934564235 shutdown_connections 2019-12-05T05:51:15.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:15.344+0000 7f4886761700 1 --2- 172.21.15.42:0/2934564235 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f486003ad00 0x7f486003d210 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:51:15.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:15.344+0000 7f4886761700 1 --2- 172.21.15.42:0/2934564235 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f48800a1cd0 0x7f48801387c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:51:15.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:15.344+0000 7f4886761700 1 --2- 172.21.15.42:0/2934564235 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f48800a2960 0x7f48801309b0 unknown :-1 s=CLOSED pgs=1211 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:51:15.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:15.344+0000 7f4886761700 1 -- 172.21.15.42:0/2934564235 shutdown_connections 2019-12-05T05:51:15.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:15.344+0000 7f4886761700 1 -- 172.21.15.42:0/2934564235 wait complete. 2019-12-05T05:51:15.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:15.344+0000 7f4886761700 1 -- 172.21.15.42:0/2934564235 >> 172.21.15.42:0/2934564235 conn(0x7f488007dc80 msgr2=0x7f48800a7f90 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:51:15.361 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T05:51:17.362 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:51:17.468 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:17.456+0000 7f6ba0d86700 1 Processor -- start 2019-12-05T05:51:17.469 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:17.456+0000 7f6ba0d86700 1 -- start start 2019-12-05T05:51:17.469 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:17.460+0000 7f6ba0d86700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6b9c0a7740 0x7f6b9c08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:51:17.469 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:17.460+0000 7f6ba0d86700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f6b9c07e720 0x7f6b9c07eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:51:17.469 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:17.460+0000 7f6ba0d86700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6b9c09f2e0 0x7f6b9c09f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:51:17.469 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:17.460+0000 7f6ba0d86700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b9c0a0b00 con 0x7f6b9c07e720 2019-12-05T05:51:17.469 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:17.460+0000 7f6ba0d86700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b9c08e5b0 con 0x7f6b9c0a7740 2019-12-05T05:51:17.470 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:17.460+0000 7f6ba0d86700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b9c08e850 con 0x7f6b9c09f2e0 2019-12-05T05:51:17.470 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:17.460+0000 7f6b9ad9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6b9c0a7740 0x7f6b9c08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:51:17.470 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:17.460+0000 7f6b9ad9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6b9c0a7740 0x7f6b9c08df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:42628/0 (socket says 172.21.15.42:42628) 2019-12-05T05:51:17.470 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:17.460+0000 7f6b9ad9d700 1 -- 172.21.15.42:0/692818133 learned_addr learned my addr 172.21.15.42:0/692818133 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:51:17.470 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:17.460+0000 7f6b9b59e700 1 --2- 172.21.15.42:0/692818133 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6b9c09f2e0 0x7f6b9c09f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:51:17.470 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:17.460+0000 7f6b9a59c700 1 --2- 172.21.15.42:0/692818133 >> v2:172.21.15.40:3300/0 conn(0x7f6b9c07e720 0x7f6b9c07eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:51:17.471 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:17.460+0000 7f6b9ad9d700 1 -- 172.21.15.42:0/692818133 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6b9c09f2e0 msgr2=0x7f6b9c09f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:51:17.471 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:17.460+0000 7f6b9ad9d700 1 --2- 172.21.15.42:0/692818133 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6b9c09f2e0 0x7f6b9c09f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:51:17.471 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:17.460+0000 7f6b9ad9d700 1 -- 172.21.15.42:0/692818133 >> v2:172.21.15.40:3300/0 conn(0x7f6b9c07e720 msgr2=0x7f6b9c07eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:51:17.471 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:17.460+0000 7f6b9ad9d700 1 --2- 172.21.15.42:0/692818133 >> v2:172.21.15.40:3300/0 conn(0x7f6b9c07e720 0x7f6b9c07eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:51:17.471 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:17.460+0000 7f6b9ad9d700 1 -- 172.21.15.42:0/692818133 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b9c07f0a0 con 0x7f6b9c0a7740 2019-12-05T05:51:17.471 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:17.460+0000 7f6b9ad9d700 1 --2- 172.21.15.42:0/692818133 >> v2:172.21.15.42:3300/0 conn(0x7f6b9c0a7740 0x7f6b9c08df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f6b9c0a0f50 tx=0x7f6b8c00fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:17.471 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:17.460+0000 7f6b9ad9d700 1 --2- 172.21.15.42:0/692818133 >> v2:172.21.15.42:3300/0 conn(0x7f6b9c0a7740 0x7f6b9c08df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f6b9c0a0f50 tx=0x7f6b8c00fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:17.472 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:17.460+0000 7f6b9ad9d700 1 --2- 172.21.15.42:0/692818133 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6b9c0a7740 0x7f6b9c08df80 secure :-1 s=READY pgs=1581 cs=0 l=1 rx=0x7f6b9c0a0f50 tx=0x7f6b8c00fdb0).ready entity=mon.1 client_cookie=cff186d533d35e5b server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:51:17.472 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:17.460+0000 7f6b9ad9d700 1 --2- 172.21.15.42:0/692818133 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6b9c0a7740 0x7f6b9c08df80 secure :-1 s=THROTTLE_DONE pgs=1581 cs=0 l=1 rx=0x7f6b9c0a0f50 tx=0x7f6b8c00fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:17.472 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:17.460+0000 7f6b99d9b700 1 -- 172.21.15.42:0/692818133 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6b8c008c90 con 0x7f6b9c0a7740 2019-12-05T05:51:17.472 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:17.460+0000 7f6b9ad9d700 1 --2- 172.21.15.42:0/692818133 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6b9c0a7740 0x7f6b9c08df80 secure :-1 s=THROTTLE_DONE pgs=1581 cs=0 l=1 rx=0x7f6b9c0a0f50 tx=0x7f6b8c00fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:17.472 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:17.460+0000 7f6b99d9b700 1 -- 172.21.15.42:0/692818133 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b8c0088b0 con 0x7f6b9c0a7740 2019-12-05T05:51:17.472 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:17.460+0000 7f6b9ad9d700 1 --2- 172.21.15.42:0/692818133 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6b9c0a7740 0x7f6b9c08df80 secure :-1 s=THROTTLE_DONE pgs=1581 cs=0 l=1 rx=0x7f6b9c0a0f50 tx=0x7f6b8c00fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:17.472 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:17.460+0000 7f6b99d9b700 1 -- 172.21.15.42:0/692818133 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6b8c011b50 con 0x7f6b9c0a7740 2019-12-05T05:51:17.473 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:17.460+0000 7f6ba0d86700 1 -- 172.21.15.42:0/692818133 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6b9c0a7740 msgr2=0x7f6b9c08df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:51:17.473 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:17.460+0000 7f6ba0d86700 1 --2- 172.21.15.42:0/692818133 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6b9c0a7740 0x7f6b9c08df80 secure :-1 s=READY pgs=1581 cs=0 l=1 rx=0x7f6b9c0a0f50 tx=0x7f6b8c00fdb0).stop 2019-12-05T05:51:17.473 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:17.460+0000 7f6ba0d86700 1 -- 172.21.15.42:0/692818133 shutdown_connections 2019-12-05T05:51:17.473 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:17.460+0000 7f6ba0d86700 1 --2- 172.21.15.42:0/692818133 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6b9c09f2e0 0x7f6b9c09f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:51:17.473 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:17.460+0000 7f6ba0d86700 1 --2- 172.21.15.42:0/692818133 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6b9c0a7740 0x7f6b9c08df80 unknown :-1 s=CLOSED pgs=1581 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:51:17.473 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:17.460+0000 7f6ba0d86700 1 --2- 172.21.15.42:0/692818133 >> v2:172.21.15.40:3300/0 conn(0x7f6b9c07e720 0x7f6b9c07eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:51:17.473 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:17.464+0000 7f6ba0d86700 1 -- 172.21.15.42:0/692818133 shutdown_connections 2019-12-05T05:51:17.474 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:17.464+0000 7f6ba0d86700 1 -- 172.21.15.42:0/692818133 wait complete. 2019-12-05T05:51:17.474 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:17.464+0000 7f6ba0d86700 1 -- 172.21.15.42:0/692818133 >> 172.21.15.42:0/692818133 conn(0x7f6b9c0aabe0 msgr2=0x7f6b9c0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:51:17.477 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:17.464+0000 7f6ba0d86700 1 Processor -- start 2019-12-05T05:51:17.478 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:17.464+0000 7f6ba0d86700 1 -- start start 2019-12-05T05:51:17.478 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:17.464+0000 7f6ba0d86700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f6b9c121510 0x7f6b9c11d750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:51:17.478 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:17.464+0000 7f6ba0d86700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6b9c131630 0x7f6b9c131a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:51:17.478 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:17.464+0000 7f6ba0d86700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b9c1312c0 con 0x7f6b9c0a3b10 2019-12-05T05:51:17.478 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:17.464+0000 7f6ba0d86700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b9c121ac0 con 0x7f6b9c121510 2019-12-05T05:51:17.478 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:17.464+0000 7f6ba0d86700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b9c123f80 con 0x7f6b9c131630 2019-12-05T05:51:17.479 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:17.464+0000 7f6b9b59e700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6b9c131630 0x7f6b9c131a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:51:17.479 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:17.464+0000 7f6b9a59c700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f6b9c121510 0x7f6b9c11d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:51:17.479 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:17.464+0000 7f6b9b59e700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6b9c131630 0x7f6b9c131a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:43104/0 (socket says 172.21.15.42:43104) 2019-12-05T05:51:17.479 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:17.464+0000 7f6b9b59e700 1 -- 172.21.15.42:0/3833595884 learned_addr learned my addr 172.21.15.42:0/3833595884 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:51:17.479 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:17.468+0000 7f6b9a59c700 1 -- 172.21.15.42:0/3833595884 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6b9c131630 msgr2=0x7f6b9c131a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:51:17.479 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:17.468+0000 7f6b9a59c700 1 --2- 172.21.15.42:0/3833595884 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6b9c131630 0x7f6b9c131a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:51:17.479 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:17.468+0000 7f6b9a59c700 1 -- 172.21.15.42:0/3833595884 >> v1:172.21.15.40:6789/0 conn(0x7f6b9c0a3b10 legacy=0x7f6b9c123780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:51:17.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:17.468+0000 7f6b9a59c700 1 -- 172.21.15.42:0/3833595884 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b9c131fb0 con 0x7f6b9c121510 2019-12-05T05:51:17.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:17.468+0000 7f6b9b59e700 1 --2- 172.21.15.42:0/3833595884 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6b9c131630 0x7f6b9c131a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:51:17.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:17.468+0000 7f6b9a59c700 1 --2- 172.21.15.42:0/3833595884 >> v2:172.21.15.40:3300/0 conn(0x7f6b9c121510 0x7f6b9c11d750 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f6b90002f30 tx=0x7f6b90010af0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:17.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:17.468+0000 7f6b9a59c700 1 --2- 172.21.15.42:0/3833595884 >> v2:172.21.15.40:3300/0 conn(0x7f6b9c121510 0x7f6b9c11d750 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f6b90002f30 tx=0x7f6b90010af0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:17.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:17.468+0000 7f6b9a59c700 1 --2- 172.21.15.42:0/3833595884 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6b9c121510 0x7f6b9c11d750 secure :-1 s=READY pgs=1213 cs=0 l=1 rx=0x7f6b90002f30 tx=0x7f6b90010af0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:51:17.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:17.468+0000 7f6b9a59c700 1 --2- 172.21.15.42:0/3833595884 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6b9c121510 0x7f6b9c11d750 secure :-1 s=THROTTLE_DONE pgs=1213 cs=0 l=1 rx=0x7f6b90002f30 tx=0x7f6b90010af0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:17.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:17.468+0000 7f6b9a59c700 1 --2- 172.21.15.42:0/3833595884 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6b9c121510 0x7f6b9c11d750 secure :-1 s=THROTTLE_DONE pgs=1213 cs=0 l=1 rx=0x7f6b90002f30 tx=0x7f6b90010af0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:17.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:17.468+0000 7f6b9a59c700 1 --2- 172.21.15.42:0/3833595884 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6b9c121510 0x7f6b9c11d750 secure :-1 s=THROTTLE_DONE pgs=1213 cs=0 l=1 rx=0x7f6b90002f30 tx=0x7f6b90010af0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:17.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:17.468+0000 7f6b8b7fe700 1 -- 172.21.15.42:0/3833595884 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6b90012980 con 0x7f6b9c121510 2019-12-05T05:51:17.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:17.468+0000 7f6b8b7fe700 1 -- 172.21.15.42:0/3833595884 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b900125a0 con 0x7f6b9c121510 2019-12-05T05:51:17.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:17.468+0000 7f6ba0d86700 1 -- 172.21.15.42:0/3833595884 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b9c122240 con 0x7f6b9c121510 2019-12-05T05:51:17.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:17.468+0000 7f6b8b7fe700 1 -- 172.21.15.42:0/3833595884 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6b90005730 con 0x7f6b9c121510 2019-12-05T05:51:17.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:17.468+0000 7f6ba0d86700 1 -- 172.21.15.42:0/3833595884 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b9c078530 con 0x7f6b9c121510 2019-12-05T05:51:17.482 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:17.468+0000 7f6b9a59c700 1 --2- 172.21.15.42:0/3833595884 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6b9c121510 0x7f6b9c11d750 secure :-1 s=THROTTLE_DONE pgs=1213 cs=0 l=1 rx=0x7f6b90002f30 tx=0x7f6b90010af0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:17.482 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:17.468+0000 7f6b8b7fe700 1 -- 172.21.15.42:0/3833595884 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f6b90007180 con 0x7f6b9c121510 2019-12-05T05:51:17.482 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:17.468+0000 7f6b9a59c700 1 --2- 172.21.15.42:0/3833595884 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6b9c121510 0x7f6b9c11d750 secure :-1 s=THROTTLE_DONE pgs=1213 cs=0 l=1 rx=0x7f6b90002f30 tx=0x7f6b90010af0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:17.482 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:17.468+0000 7f6b8b7fe700 1 --2- 172.21.15.42:0/3833595884 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6b8003ae40 0x7f6b8003d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:51:17.482 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:17.472+0000 7f6b8b7fe700 1 -- 172.21.15.42:0/3833595884 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8783+0+0 (secure 0 0 0) 0x7f6b9004af50 con 0x7f6b9c121510 2019-12-05T05:51:17.482 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:17.472+0000 7f6b9ad9d700 1 --2- 172.21.15.42:0/3833595884 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6b8003ae40 0x7f6b8003d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:51:17.483 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:17.472+0000 7f6b9ad9d700 1 --2- 172.21.15.42:0/3833595884 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6b8003ae40 0x7f6b8003d350 crc :-1 s=READY pgs=682 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:51:17.483 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:17.472+0000 7f6ba0d86700 1 -- 172.21.15.42:0/3833595884 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b9c121c80 con 0x7f6b9c121510 2019-12-05T05:51:17.488 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:17.480+0000 7f6b9a59c700 1 --2- 172.21.15.42:0/3833595884 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6b9c121510 0x7f6b9c11d750 secure :-1 s=THROTTLE_DONE pgs=1213 cs=0 l=1 rx=0x7f6b90002f30 tx=0x7f6b90010af0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:17.488 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:17.480+0000 7f6b8b7fe700 1 -- 172.21.15.42:0/3833595884 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f6b90031fd0 con 0x7f6b9c121510 2019-12-05T05:51:17.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:17.648+0000 7f6ba0d86700 1 -- 172.21.15.42:0/3833595884 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6b9c05d110 con 0x7f6b9c121510 2019-12-05T05:51:17.654 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:17.648+0000 7f6b9a59c700 1 --2- 172.21.15.42:0/3833595884 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6b9c121510 0x7f6b9c11d750 secure :-1 s=THROTTLE_DONE pgs=1213 cs=0 l=1 rx=0x7f6b90002f30 tx=0x7f6b90010af0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:17.654 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:17.648+0000 7f6b8b7fe700 1 -- 172.21.15.42:0/3833595884 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 193 v193) v1 ==== 95+0+2103 (secure 0 0 0) 0x7f6b90032230 con 0x7f6b9c121510 2019-12-05T05:51:17.656 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 193 2019-12-05T05:51:17.656 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:51:17.656 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":193,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":8430,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6835/5991897","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":5991897},{"type":"v1","addr":"172.21.15.42:6835","nonce":5991897}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":193}],"filesystems":[{"mdsmap":{"epoch":192,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:50:54.569377+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6245},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6245":{"gid":6245,"name":"b","rank":0,"incarnation":189,"state":"up:active","state_seq":25,"addr":"172.21.15.40:6833/2081111170","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2081111170},{"type":"v1","addr":"172.21.15.40:6833","nonce":2081111170}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:51:17.659 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:17.652+0000 7f6ba0d86700 1 -- 172.21.15.42:0/3833595884 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6b8003ae40 msgr2=0x7f6b8003d350 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:51:17.659 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:17.652+0000 7f6ba0d86700 1 --2- 172.21.15.42:0/3833595884 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6b8003ae40 0x7f6b8003d350 crc :-1 s=READY pgs=682 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:51:17.659 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:17.652+0000 7f6ba0d86700 1 -- 172.21.15.42:0/3833595884 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6b9c121510 msgr2=0x7f6b9c11d750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:51:17.659 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:17.652+0000 7f6ba0d86700 1 --2- 172.21.15.42:0/3833595884 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6b9c121510 0x7f6b9c11d750 secure :-1 s=READY pgs=1213 cs=0 l=1 rx=0x7f6b90002f30 tx=0x7f6b90010af0).stop 2019-12-05T05:51:17.659 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:17.652+0000 7f6ba0d86700 1 -- 172.21.15.42:0/3833595884 shutdown_connections 2019-12-05T05:51:17.659 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:17.652+0000 7f6ba0d86700 1 --2- 172.21.15.42:0/3833595884 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6b8003ae40 0x7f6b8003d350 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:51:17.659 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:17.652+0000 7f6ba0d86700 1 --2- 172.21.15.42:0/3833595884 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6b9c131630 0x7f6b9c131a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:51:17.660 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:17.652+0000 7f6ba0d86700 1 --2- 172.21.15.42:0/3833595884 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6b9c121510 0x7f6b9c11d750 unknown :-1 s=CLOSED pgs=1213 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:51:17.660 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:17.652+0000 7f6ba0d86700 1 -- 172.21.15.42:0/3833595884 shutdown_connections 2019-12-05T05:51:17.660 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:17.652+0000 7f6ba0d86700 1 -- 172.21.15.42:0/3833595884 wait complete. 2019-12-05T05:51:17.660 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:17.652+0000 7f6ba0d86700 1 -- 172.21.15.42:0/3833595884 >> 172.21.15.42:0/3833595884 conn(0x7f6b9c0aabe0 msgr2=0x7f6b9c0a9770 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:51:17.669 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T05:51:19.670 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:51:19.765 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:19.756+0000 7f02fe7af700 1 Processor -- start 2019-12-05T05:51:19.766 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:19.756+0000 7f02fe7af700 1 -- start start 2019-12-05T05:51:19.766 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:19.756+0000 7f02fe7af700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f02f80a7740 0x7f02f808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:51:19.766 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:19.756+0000 7f02fe7af700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f02f809fae0 0x7f02f807eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:51:19.766 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:19.756+0000 7f02fe7af700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02f80a0710 con 0x7f02f807e720 2019-12-05T05:51:19.767 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:19.756+0000 7f02fe7af700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02f80a0b00 con 0x7f02f80a7740 2019-12-05T05:51:19.767 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:19.756+0000 7f02fe7af700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02f808e5b0 con 0x7f02f809fae0 2019-12-05T05:51:19.767 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:19.760+0000 7f02fd54d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f02f809fae0 0x7f02f807eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:51:19.767 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:19.760+0000 7f02fd54d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f02f809fae0 0x7f02f807eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:42660/0 (socket says 172.21.15.42:42660) 2019-12-05T05:51:19.767 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:19.760+0000 7f02fd54d700 1 -- 172.21.15.42:0/3191902346 learned_addr learned my addr 172.21.15.42:0/3191902346 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:51:19.768 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:19.760+0000 7f02fcd4c700 1 --2- 172.21.15.42:0/3191902346 >> v2:172.21.15.40:3300/0 conn(0x7f02f80a7740 0x7f02f808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:51:19.768 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:19.760+0000 7f02fd54d700 1 -- 172.21.15.42:0/3191902346 >> v2:172.21.15.40:3300/0 conn(0x7f02f80a7740 msgr2=0x7f02f808df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:51:19.768 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:19.760+0000 7f02fd54d700 1 --2- 172.21.15.42:0/3191902346 >> v2:172.21.15.40:3300/0 conn(0x7f02f80a7740 0x7f02f808df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:51:19.768 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:19.760+0000 7f02fd54d700 1 -- 172.21.15.42:0/3191902346 >> v1:172.21.15.40:6789/0 conn(0x7f02f807e720 legacy=0x7f02f809f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:51:19.768 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:19.760+0000 7f02fd54d700 1 -- 172.21.15.42:0/3191902346 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02f807f0a0 con 0x7f02f809fae0 2019-12-05T05:51:19.768 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:19.760+0000 7f02fd54d700 1 --2- 172.21.15.42:0/3191902346 >> v2:172.21.15.42:3300/0 conn(0x7f02f809fae0 0x7f02f807eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f02f00032b0 tx=0x7f02f00100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:19.769 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:19.760+0000 7f02fd54d700 1 --2- 172.21.15.42:0/3191902346 >> v2:172.21.15.42:3300/0 conn(0x7f02f809fae0 0x7f02f807eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f02f00032b0 tx=0x7f02f00100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:19.769 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:19.760+0000 7f02fd54d700 1 --2- 172.21.15.42:0/3191902346 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f02f809fae0 0x7f02f807eb50 secure :-1 s=READY pgs=1582 cs=0 l=1 rx=0x7f02f00032b0 tx=0x7f02f00100b0).ready entity=mon.1 client_cookie=400b7af3e33d6d2d server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:51:19.769 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:19.760+0000 7f02fd54d700 1 --2- 172.21.15.42:0/3191902346 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f02f809fae0 0x7f02f807eb50 secure :-1 s=THROTTLE_DONE pgs=1582 cs=0 l=1 rx=0x7f02f00032b0 tx=0x7f02f00100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:19.769 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:19.760+0000 7f02fd54d700 1 --2- 172.21.15.42:0/3191902346 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f02f809fae0 0x7f02f807eb50 secure :-1 s=THROTTLE_DONE pgs=1582 cs=0 l=1 rx=0x7f02f00032b0 tx=0x7f02f00100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:19.769 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:19.760+0000 7f02ef7fe700 1 -- 172.21.15.42:0/3191902346 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f02f0010d60 con 0x7f02f809fae0 2019-12-05T05:51:19.769 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:19.760+0000 7f02fd54d700 1 --2- 172.21.15.42:0/3191902346 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f02f809fae0 0x7f02f807eb50 secure :-1 s=THROTTLE_DONE pgs=1582 cs=0 l=1 rx=0x7f02f00032b0 tx=0x7f02f00100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:19.770 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:19.760+0000 7f02ef7fe700 1 -- 172.21.15.42:0/3191902346 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02f0010980 con 0x7f02f809fae0 2019-12-05T05:51:19.770 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:19.760+0000 7f02ef7fe700 1 -- 172.21.15.42:0/3191902346 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f02f00114a0 con 0x7f02f809fae0 2019-12-05T05:51:19.770 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:19.760+0000 7f02fe7af700 1 -- 172.21.15.42:0/3191902346 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f02f809fae0 msgr2=0x7f02f807eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:51:19.770 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:19.760+0000 7f02fe7af700 1 --2- 172.21.15.42:0/3191902346 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f02f809fae0 0x7f02f807eb50 secure :-1 s=READY pgs=1582 cs=0 l=1 rx=0x7f02f00032b0 tx=0x7f02f00100b0).stop 2019-12-05T05:51:19.770 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:19.760+0000 7f02fe7af700 1 -- 172.21.15.42:0/3191902346 shutdown_connections 2019-12-05T05:51:19.770 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:19.760+0000 7f02fe7af700 1 --2- 172.21.15.42:0/3191902346 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f02f809fae0 0x7f02f807eb50 unknown :-1 s=CLOSED pgs=1582 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:51:19.770 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:19.760+0000 7f02fe7af700 1 --2- 172.21.15.42:0/3191902346 >> v2:172.21.15.40:3300/0 conn(0x7f02f80a7740 0x7f02f808df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:51:19.771 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:19.760+0000 7f02fe7af700 1 -- 172.21.15.42:0/3191902346 shutdown_connections 2019-12-05T05:51:19.771 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:19.760+0000 7f02fe7af700 1 -- 172.21.15.42:0/3191902346 wait complete. 2019-12-05T05:51:19.771 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:19.760+0000 7f02fe7af700 1 -- 172.21.15.42:0/3191902346 >> 172.21.15.42:0/3191902346 conn(0x7f02f80aabe0 msgr2=0x7f02f80a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:51:19.775 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:19.760+0000 7f02fe7af700 1 Processor -- start 2019-12-05T05:51:19.775 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:19.760+0000 7f02fe7af700 1 -- start start 2019-12-05T05:51:19.775 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:19.760+0000 7f02fe7af700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f02f811a0e0 0x7f02f8112460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:51:19.775 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:19.760+0000 7f02fe7af700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f02f81129b0 0x7f02f8114070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:51:19.776 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:19.760+0000 7f02fe7af700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02f807bb80 con 0x7f02f80a9ad0 2019-12-05T05:51:19.776 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:19.760+0000 7f02fe7af700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02f8119900 con 0x7f02f811a0e0 2019-12-05T05:51:19.776 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:19.760+0000 7f02fe7af700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02f8119ba0 con 0x7f02f81129b0 2019-12-05T05:51:19.776 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:19.760+0000 7f02fcd4c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f02f811a0e0 0x7f02f8112460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:51:19.776 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:19.760+0000 7f02fcd4c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f02f811a0e0 0x7f02f8112460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:42662/0 (socket says 172.21.15.42:42662) 2019-12-05T05:51:19.776 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:19.760+0000 7f02fcd4c700 1 -- 172.21.15.42:0/1019469059 learned_addr learned my addr 172.21.15.42:0/1019469059 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:51:19.777 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:19.764+0000 7f02effff700 1 --2- 172.21.15.42:0/1019469059 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f02f81129b0 0x7f02f8114070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:51:19.777 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:19.764+0000 7f02fcd4c700 1 -- 172.21.15.42:0/1019469059 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f02f81129b0 msgr2=0x7f02f8114070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:51:19.777 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:19.764+0000 7f02fcd4c700 1 --2- 172.21.15.42:0/1019469059 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f02f81129b0 0x7f02f8114070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:51:19.777 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:19.764+0000 7f02fcd4c700 1 -- 172.21.15.42:0/1019469059 >> v1:172.21.15.40:6789/0 conn(0x7f02f80a9ad0 legacy=0x7f02f807b380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:51:19.777 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:19.764+0000 7f02fcd4c700 1 -- 172.21.15.42:0/1019469059 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02f81145c0 con 0x7f02f811a0e0 2019-12-05T05:51:19.777 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:19.764+0000 7f02fcd4c700 1 --2- 172.21.15.42:0/1019469059 >> v2:172.21.15.42:3300/0 conn(0x7f02f811a0e0 0x7f02f8112460 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f02f8119ac0 tx=0x7f02f40109f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:19.777 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:19.764+0000 7f02fcd4c700 1 --2- 172.21.15.42:0/1019469059 >> v2:172.21.15.42:3300/0 conn(0x7f02f811a0e0 0x7f02f8112460 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f02f8119ac0 tx=0x7f02f40109f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:19.778 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:19.764+0000 7f02fcd4c700 1 --2- 172.21.15.42:0/1019469059 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f02f811a0e0 0x7f02f8112460 secure :-1 s=READY pgs=1583 cs=0 l=1 rx=0x7f02f8119ac0 tx=0x7f02f40109f0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:51:19.778 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:19.764+0000 7f02fcd4c700 1 --2- 172.21.15.42:0/1019469059 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f02f811a0e0 0x7f02f8112460 secure :-1 s=THROTTLE_DONE pgs=1583 cs=0 l=1 rx=0x7f02f8119ac0 tx=0x7f02f40109f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:19.778 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:19.764+0000 7f02fcd4c700 1 --2- 172.21.15.42:0/1019469059 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f02f811a0e0 0x7f02f8112460 secure :-1 s=THROTTLE_DONE pgs=1583 cs=0 l=1 rx=0x7f02f8119ac0 tx=0x7f02f40109f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:19.778 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:19.764+0000 7f02ed7fa700 1 -- 172.21.15.42:0/1019469059 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f02f4005930 con 0x7f02f811a0e0 2019-12-05T05:51:19.778 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:19.764+0000 7f02fcd4c700 1 --2- 172.21.15.42:0/1019469059 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f02f811a0e0 0x7f02f8112460 secure :-1 s=THROTTLE_DONE pgs=1583 cs=0 l=1 rx=0x7f02f8119ac0 tx=0x7f02f40109f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:19.778 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:19.764+0000 7f02ed7fa700 1 -- 172.21.15.42:0/1019469059 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02f4007160 con 0x7f02f811a0e0 2019-12-05T05:51:19.779 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:19.764+0000 7f02fe7af700 1 -- 172.21.15.42:0/1019469059 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02f8114830 con 0x7f02f811a0e0 2019-12-05T05:51:19.779 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:19.764+0000 7f02ed7fa700 1 -- 172.21.15.42:0/1019469059 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f02f40060c0 con 0x7f02f811a0e0 2019-12-05T05:51:19.779 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:19.764+0000 7f02fe7af700 1 -- 172.21.15.42:0/1019469059 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02f812e3d0 con 0x7f02f811a0e0 2019-12-05T05:51:19.779 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:19.764+0000 7f02fcd4c700 1 --2- 172.21.15.42:0/1019469059 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f02f811a0e0 0x7f02f8112460 secure :-1 s=THROTTLE_DONE pgs=1583 cs=0 l=1 rx=0x7f02f8119ac0 tx=0x7f02f40109f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:19.779 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:19.764+0000 7f02ed7fa700 1 -- 172.21.15.42:0/1019469059 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f02f4007a20 con 0x7f02f811a0e0 2019-12-05T05:51:19.779 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:19.764+0000 7f02fcd4c700 1 --2- 172.21.15.42:0/1019469059 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f02f811a0e0 0x7f02f8112460 secure :-1 s=THROTTLE_DONE pgs=1583 cs=0 l=1 rx=0x7f02f8119ac0 tx=0x7f02f40109f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:19.780 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:19.764+0000 7f02ed7fa700 1 --2- 172.21.15.42:0/1019469059 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f02d803adb0 0x7f02d803d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:51:19.780 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:19.764+0000 7f02ed7fa700 1 -- 172.21.15.42:0/1019469059 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8869+0+0 (secure 0 0 0) 0x7f02f404ab10 con 0x7f02f811a0e0 2019-12-05T05:51:19.780 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:19.764+0000 7f02effff700 1 --2- 172.21.15.42:0/1019469059 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f02d803adb0 0x7f02d803d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:51:19.780 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:19.764+0000 7f02effff700 1 --2- 172.21.15.42:0/1019469059 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f02d803adb0 0x7f02d803d2c0 crc :-1 s=READY pgs=683 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:51:19.780 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:19.764+0000 7f02fe7af700 1 -- 172.21.15.42:0/1019469059 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02f805d110 con 0x7f02f811a0e0 2019-12-05T05:51:19.780 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:19.768+0000 7f02fcd4c700 1 --2- 172.21.15.42:0/1019469059 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f02f811a0e0 0x7f02f8112460 secure :-1 s=THROTTLE_DONE pgs=1583 cs=0 l=1 rx=0x7f02f8119ac0 tx=0x7f02f40109f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:19.781 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:19.768+0000 7f02ed7fa700 1 -- 172.21.15.42:0/1019469059 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f02f4031b90 con 0x7f02f811a0e0 2019-12-05T05:51:19.957 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:19.952+0000 7f02fe7af700 1 -- 172.21.15.42:0/1019469059 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f02f805b8c0 con 0x7f02f811a0e0 2019-12-05T05:51:19.958 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:19.952+0000 7f02fcd4c700 1 --2- 172.21.15.42:0/1019469059 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f02f811a0e0 0x7f02f8112460 secure :-1 s=THROTTLE_DONE pgs=1583 cs=0 l=1 rx=0x7f02f8119ac0 tx=0x7f02f40109f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:19.958 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:19.952+0000 7f02ed7fa700 1 -- 172.21.15.42:0/1019469059 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 194 v194) v1 ==== 95+0+1765 (secure 0 0 0) 0x7f02f4031df0 con 0x7f02f811a0e0 2019-12-05T05:51:19.958 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 194 2019-12-05T05:51:19.959 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:51:19.959 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":194,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":194,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:51:19.126731+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8430},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8430":{"gid":8430,"name":"a","rank":0,"incarnation":194,"state":"up:replay","state_seq":1,"addr":"172.21.15.42:6835/5991897","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":5991897},{"type":"v1","addr":"172.21.15.42:6835","nonce":5991897}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:51:19.961 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:19.956+0000 7f02fe7af700 1 -- 172.21.15.42:0/1019469059 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f02d803adb0 msgr2=0x7f02d803d2c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:51:19.961 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:19.956+0000 7f02fe7af700 1 --2- 172.21.15.42:0/1019469059 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f02d803adb0 0x7f02d803d2c0 crc :-1 s=READY pgs=683 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:51:19.961 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:19.956+0000 7f02fe7af700 1 -- 172.21.15.42:0/1019469059 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f02f811a0e0 msgr2=0x7f02f8112460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:51:19.961 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:19.956+0000 7f02fe7af700 1 --2- 172.21.15.42:0/1019469059 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f02f811a0e0 0x7f02f8112460 secure :-1 s=READY pgs=1583 cs=0 l=1 rx=0x7f02f8119ac0 tx=0x7f02f40109f0).stop 2019-12-05T05:51:19.962 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:19.956+0000 7f02fe7af700 1 -- 172.21.15.42:0/1019469059 shutdown_connections 2019-12-05T05:51:19.962 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:19.956+0000 7f02fe7af700 1 --2- 172.21.15.42:0/1019469059 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f02d803adb0 0x7f02d803d2c0 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:51:19.962 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:19.956+0000 7f02fe7af700 1 --2- 172.21.15.42:0/1019469059 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f02f811a0e0 0x7f02f8112460 unknown :-1 s=CLOSED pgs=1583 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:51:19.962 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:19.956+0000 7f02fe7af700 1 --2- 172.21.15.42:0/1019469059 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f02f81129b0 0x7f02f8114070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:51:19.962 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:19.956+0000 7f02fe7af700 1 -- 172.21.15.42:0/1019469059 shutdown_connections 2019-12-05T05:51:19.962 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:19.956+0000 7f02fe7af700 1 -- 172.21.15.42:0/1019469059 wait complete. 2019-12-05T05:51:19.963 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:19.956+0000 7f02fe7af700 1 -- 172.21.15.42:0/1019469059 >> 172.21.15.42:0/1019469059 conn(0x7f02f807d260 msgr2=0x7f02f809f200 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:51:19.974 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b down, removed from mdsmap 2019-12-05T05:51:19.975 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2019-12-05T05:51:19.975 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:51:20.068 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:20.060+0000 7f8bb2c8d700 1 Processor -- start 2019-12-05T05:51:20.068 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:20.060+0000 7f8bb2c8d700 1 -- start start 2019-12-05T05:51:20.068 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:20.060+0000 7f8bb2c8d700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8bac0a7740 0x7f8bac08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:51:20.069 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:20.060+0000 7f8bb2c8d700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8bac09fae0 0x7f8bac07eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:51:20.069 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:20.060+0000 7f8bb2c8d700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bac0a0710 con 0x7f8bac07e720 2019-12-05T05:51:20.069 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:20.060+0000 7f8bb2c8d700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8bac0a0b00 con 0x7f8bac0a7740 2019-12-05T05:51:20.069 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:20.060+0000 7f8bb2c8d700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bac08e5b0 con 0x7f8bac09fae0 2019-12-05T05:51:20.069 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:20.060+0000 7f8bb122a700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8bac0a7740 0x7f8bac08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:51:20.069 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:20.060+0000 7f8bb1a2b700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8bac09fae0 0x7f8bac07eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:51:20.070 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:20.060+0000 7f8bb122a700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8bac0a7740 0x7f8bac08df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.42:58718/0 (socket says 172.21.15.42:58718) 2019-12-05T05:51:20.070 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:20.060+0000 7f8bb122a700 1 -- 172.21.15.42:0/2043108336 learned_addr learned my addr 172.21.15.42:0/2043108336 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:51:20.070 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:20.060+0000 7f8bb1a2b700 1 -- 172.21.15.42:0/2043108336 >> v2:172.21.15.40:3300/0 conn(0x7f8bac0a7740 msgr2=0x7f8bac08df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:51:20.070 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:20.060+0000 7f8bb1a2b700 1 --2- 172.21.15.42:0/2043108336 >> v2:172.21.15.40:3300/0 conn(0x7f8bac0a7740 0x7f8bac08df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:51:20.070 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:20.060+0000 7f8bb1a2b700 1 -- 172.21.15.42:0/2043108336 >> v1:172.21.15.40:6789/0 conn(0x7f8bac07e720 legacy=0x7f8bac09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:51:20.071 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:20.060+0000 7f8bb1a2b700 1 -- 172.21.15.42:0/2043108336 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bac07f0a0 con 0x7f8bac09fae0 2019-12-05T05:51:20.071 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:20.060+0000 7f8bb1a2b700 1 --2- 172.21.15.42:0/2043108336 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8bac09fae0 0x7f8bac07eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f8ba4008b10 tx=0x7f8ba400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:20.071 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:20.060+0000 7f8bb1a2b700 1 --2- 172.21.15.42:0/2043108336 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8bac09fae0 0x7f8bac07eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f8ba4008b10 tx=0x7f8ba400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:20.071 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:20.060+0000 7f8bb1a2b700 1 --2- 172.21.15.42:0/2043108336 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8bac09fae0 0x7f8bac07eb50 secure :-1 s=READY pgs=352 cs=0 l=1 rx=0x7f8ba4008b10 tx=0x7f8ba400fff0).ready entity=mon.2 client_cookie=39cc85f3ddab531e server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:51:20.071 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:20.060+0000 7f8bb1a2b700 1 --2- 172.21.15.42:0/2043108336 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8bac09fae0 0x7f8bac07eb50 secure :-1 s=THROTTLE_DONE pgs=352 cs=0 l=1 rx=0x7f8ba4008b10 tx=0x7f8ba400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:20.071 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:20.060+0000 7f8ba3fff700 1 -- 172.21.15.42:0/2043108336 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8ba4010900 con 0x7f8bac09fae0 2019-12-05T05:51:20.072 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:20.060+0000 7f8bb1a2b700 1 --2- 172.21.15.42:0/2043108336 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8bac09fae0 0x7f8bac07eb50 secure :-1 s=THROTTLE_DONE pgs=352 cs=0 l=1 rx=0x7f8ba4008b10 tx=0x7f8ba400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:20.072 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:20.060+0000 7f8bb1a2b700 1 --2- 172.21.15.42:0/2043108336 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8bac09fae0 0x7f8bac07eb50 secure :-1 s=THROTTLE_DONE pgs=352 cs=0 l=1 rx=0x7f8ba4008b10 tx=0x7f8ba400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:20.072 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:20.060+0000 7f8ba3fff700 1 -- 172.21.15.42:0/2043108336 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ba4008b50 con 0x7f8bac09fae0 2019-12-05T05:51:20.072 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:20.060+0000 7f8ba3fff700 1 -- 172.21.15.42:0/2043108336 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8ba40110c0 con 0x7f8bac09fae0 2019-12-05T05:51:20.072 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:20.060+0000 7f8bb2c8d700 1 -- 172.21.15.42:0/2043108336 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8bac09fae0 msgr2=0x7f8bac07eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:51:20.073 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:20.060+0000 7f8bb2c8d700 1 --2- 172.21.15.42:0/2043108336 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8bac09fae0 0x7f8bac07eb50 secure :-1 s=READY pgs=352 cs=0 l=1 rx=0x7f8ba4008b10 tx=0x7f8ba400fff0).stop 2019-12-05T05:51:20.073 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:20.064+0000 7f8bb2c8d700 1 -- 172.21.15.42:0/2043108336 shutdown_connections 2019-12-05T05:51:20.073 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:20.064+0000 7f8bb2c8d700 1 --2- 172.21.15.42:0/2043108336 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8bac09fae0 0x7f8bac07eb50 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:51:20.073 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:20.064+0000 7f8bb2c8d700 1 --2- 172.21.15.42:0/2043108336 >> v2:172.21.15.40:3300/0 conn(0x7f8bac0a7740 0x7f8bac08df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:51:20.073 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:20.064+0000 7f8bb2c8d700 1 -- 172.21.15.42:0/2043108336 shutdown_connections 2019-12-05T05:51:20.073 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:20.064+0000 7f8bb2c8d700 1 -- 172.21.15.42:0/2043108336 wait complete. 2019-12-05T05:51:20.074 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:20.064+0000 7f8bb2c8d700 1 -- 172.21.15.42:0/2043108336 >> 172.21.15.42:0/2043108336 conn(0x7f8bac0aabe0 msgr2=0x7f8bac0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:51:20.077 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:20.064+0000 7f8bb2c8d700 1 Processor -- start 2019-12-05T05:51:20.077 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:20.064+0000 7f8bb2c8d700 1 -- start start 2019-12-05T05:51:20.078 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:20.064+0000 7f8bb2c8d700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8bac12fc00 0x7f8bac11aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:51:20.078 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:20.064+0000 7f8bb2c8d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f8bac1120f0 0x7f8bac0a4a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:51:20.078 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:20.064+0000 7f8bb2c8d700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bac0a5430 con 0x7f8bac117dd0 2019-12-05T05:51:20.078 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:20.064+0000 7f8bb2c8d700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8bac0a5190 con 0x7f8bac12fc00 2019-12-05T05:51:20.078 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:20.064+0000 7f8bb2c8d700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8bac117b30 con 0x7f8bac1120f0 2019-12-05T05:51:20.079 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:20.064+0000 7f8bb122a700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8bac12fc00 0x7f8bac11aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:51:20.079 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:20.064+0000 7f8bb0a29700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f8bac1120f0 0x7f8bac0a4a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:51:20.079 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:20.064+0000 7f8bb122a700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8bac12fc00 0x7f8bac11aa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.42:58724/0 (socket says 172.21.15.42:58724) 2019-12-05T05:51:20.079 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:20.064+0000 7f8bb122a700 1 -- 172.21.15.42:0/1984626425 learned_addr learned my addr 172.21.15.42:0/1984626425 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:51:20.079 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:20.068+0000 7f8bb0a29700 1 -- 172.21.15.42:0/1984626425 >> v2:172.21.15.40:3300/0 conn(0x7f8bac12fc00 msgr2=0x7f8bac11aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:51:20.079 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:20.068+0000 7f8bb0a29700 1 --2- 172.21.15.42:0/1984626425 >> v2:172.21.15.40:3300/0 conn(0x7f8bac12fc00 0x7f8bac11aa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:51:20.080 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:20.068+0000 7f8bb0a29700 1 -- 172.21.15.42:0/1984626425 >> v1:172.21.15.40:6789/0 conn(0x7f8bac117dd0 legacy=0x7f8bac0a6a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:51:20.080 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:20.068+0000 7f8bb0a29700 1 -- 172.21.15.42:0/1984626425 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bac118200 con 0x7f8bac1120f0 2019-12-05T05:51:20.080 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:20.068+0000 7f8bb122a700 1 --2- 172.21.15.42:0/1984626425 >> v2:172.21.15.40:3300/0 conn(0x7f8bac12fc00 0x7f8bac11aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:51:20.080 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:20.068+0000 7f8bb0a29700 1 --2- 172.21.15.42:0/1984626425 >> v2:172.21.15.42:3300/0 conn(0x7f8bac1120f0 0x7f8bac0a4a50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f8bac0a12a0 tx=0x7f8b9c0108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:20.080 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:20.068+0000 7f8bb0a29700 1 --2- 172.21.15.42:0/1984626425 >> v2:172.21.15.42:3300/0 conn(0x7f8bac1120f0 0x7f8bac0a4a50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f8bac0a12a0 tx=0x7f8b9c0108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:20.080 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:20.068+0000 7f8bb0a29700 1 --2- 172.21.15.42:0/1984626425 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8bac1120f0 0x7f8bac0a4a50 secure :-1 s=READY pgs=1584 cs=0 l=1 rx=0x7f8bac0a12a0 tx=0x7f8b9c0108f0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:51:20.081 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:20.068+0000 7f8bb0a29700 1 --2- 172.21.15.42:0/1984626425 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8bac1120f0 0x7f8bac0a4a50 secure :-1 s=THROTTLE_DONE pgs=1584 cs=0 l=1 rx=0x7f8bac0a12a0 tx=0x7f8b9c0108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:20.081 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:20.068+0000 7f8bb0a29700 1 --2- 172.21.15.42:0/1984626425 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8bac1120f0 0x7f8bac0a4a50 secure :-1 s=THROTTLE_DONE pgs=1584 cs=0 l=1 rx=0x7f8bac0a12a0 tx=0x7f8b9c0108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:20.081 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:20.068+0000 7f8bb0a29700 1 --2- 172.21.15.42:0/1984626425 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8bac1120f0 0x7f8bac0a4a50 secure :-1 s=THROTTLE_DONE pgs=1584 cs=0 l=1 rx=0x7f8bac0a12a0 tx=0x7f8b9c0108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:20.081 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:20.068+0000 7f8ba1ffb700 1 -- 172.21.15.42:0/1984626425 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8b9c003160 con 0x7f8bac1120f0 2019-12-05T05:51:20.081 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:20.068+0000 7f8ba1ffb700 1 -- 172.21.15.42:0/1984626425 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b9c011220 con 0x7f8bac1120f0 2019-12-05T05:51:20.081 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:20.068+0000 7f8bb2c8d700 1 -- 172.21.15.42:0/1984626425 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bac1184d0 con 0x7f8bac1120f0 2019-12-05T05:51:20.082 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:20.068+0000 7f8ba1ffb700 1 -- 172.21.15.42:0/1984626425 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8b9c003740 con 0x7f8bac1120f0 2019-12-05T05:51:20.082 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:20.068+0000 7f8bb2c8d700 1 -- 172.21.15.42:0/1984626425 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bac0a99b0 con 0x7f8bac1120f0 2019-12-05T05:51:20.082 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:20.068+0000 7f8bb0a29700 1 --2- 172.21.15.42:0/1984626425 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8bac1120f0 0x7f8bac0a4a50 secure :-1 s=THROTTLE_DONE pgs=1584 cs=0 l=1 rx=0x7f8bac0a12a0 tx=0x7f8b9c0108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:20.082 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:20.068+0000 7f8ba1ffb700 1 -- 172.21.15.42:0/1984626425 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f8b9c003940 con 0x7f8bac1120f0 2019-12-05T05:51:20.082 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:20.068+0000 7f8bb0a29700 1 --2- 172.21.15.42:0/1984626425 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8bac1120f0 0x7f8bac0a4a50 secure :-1 s=THROTTLE_DONE pgs=1584 cs=0 l=1 rx=0x7f8bac0a12a0 tx=0x7f8b9c0108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:20.082 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:20.068+0000 7f8ba1ffb700 1 --2- 172.21.15.42:0/1984626425 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8b8c03ae00 0x7f8b8c03d310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:51:20.083 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:20.068+0000 7f8ba1ffb700 1 -- 172.21.15.42:0/1984626425 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8869+0+0 (secure 0 0 0) 0x7f8b9c04d370 con 0x7f8bac1120f0 2019-12-05T05:51:20.083 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:20.068+0000 7f8bb122a700 1 --2- 172.21.15.42:0/1984626425 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8b8c03ae00 0x7f8b8c03d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:51:20.083 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:20.068+0000 7f8bb122a700 1 --2- 172.21.15.42:0/1984626425 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8b8c03ae00 0x7f8b8c03d310 crc :-1 s=READY pgs=684 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:51:20.084 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:20.072+0000 7f8bb2c8d700 1 -- 172.21.15.42:0/1984626425 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8bac05d110 con 0x7f8bac1120f0 2019-12-05T05:51:20.084 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:20.076+0000 7f8bb0a29700 1 --2- 172.21.15.42:0/1984626425 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8bac1120f0 0x7f8bac0a4a50 secure :-1 s=THROTTLE_DONE pgs=1584 cs=0 l=1 rx=0x7f8bac0a12a0 tx=0x7f8b9c0108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:20.084 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:20.076+0000 7f8ba1ffb700 1 -- 172.21.15.42:0/1984626425 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f8b9c0343f0 con 0x7f8bac1120f0 2019-12-05T05:51:20.241 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:20.236+0000 7f8bb2c8d700 1 -- 172.21.15.42:0/1984626425 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8bac05b8c0 con 0x7f8bac1120f0 2019-12-05T05:51:20.242 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:20.236+0000 7f8bb0a29700 1 --2- 172.21.15.42:0/1984626425 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8bac1120f0 0x7f8bac0a4a50 secure :-1 s=THROTTLE_DONE pgs=1584 cs=0 l=1 rx=0x7f8bac0a12a0 tx=0x7f8b9c0108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:20.242 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:20.236+0000 7f8ba1ffb700 1 -- 172.21.15.42:0/1984626425 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 194 v194) v1 ==== 95+0+1765 (secure 0 0 0) 0x7f8b9c034650 con 0x7f8bac1120f0 2019-12-05T05:51:20.242 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 194 2019-12-05T05:51:20.242 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:51:20.242 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":194,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":194,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:51:19.126731+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8430},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8430":{"gid":8430,"name":"a","rank":0,"incarnation":194,"state":"up:replay","state_seq":1,"addr":"172.21.15.42:6835/5991897","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":5991897},{"type":"v1","addr":"172.21.15.42:6835","nonce":5991897}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:51:20.245 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:20.240+0000 7f8bb2c8d700 1 -- 172.21.15.42:0/1984626425 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8b8c03ae00 msgr2=0x7f8b8c03d310 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:51:20.245 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:20.240+0000 7f8bb2c8d700 1 --2- 172.21.15.42:0/1984626425 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8b8c03ae00 0x7f8b8c03d310 crc :-1 s=READY pgs=684 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:51:20.245 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:20.240+0000 7f8bb2c8d700 1 -- 172.21.15.42:0/1984626425 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8bac1120f0 msgr2=0x7f8bac0a4a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:51:20.245 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:20.240+0000 7f8bb2c8d700 1 --2- 172.21.15.42:0/1984626425 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8bac1120f0 0x7f8bac0a4a50 secure :-1 s=READY pgs=1584 cs=0 l=1 rx=0x7f8bac0a12a0 tx=0x7f8b9c0108f0).stop 2019-12-05T05:51:20.245 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:20.240+0000 7f8bb2c8d700 1 -- 172.21.15.42:0/1984626425 shutdown_connections 2019-12-05T05:51:20.246 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:20.240+0000 7f8bb2c8d700 1 --2- 172.21.15.42:0/1984626425 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8b8c03ae00 0x7f8b8c03d310 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:51:20.246 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:20.240+0000 7f8bb2c8d700 1 --2- 172.21.15.42:0/1984626425 >> v2:172.21.15.40:3300/0 conn(0x7f8bac12fc00 0x7f8bac11aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:51:20.246 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:20.240+0000 7f8bb2c8d700 1 --2- 172.21.15.42:0/1984626425 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8bac1120f0 0x7f8bac0a4a50 unknown :-1 s=CLOSED pgs=1584 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:51:20.246 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:20.240+0000 7f8bb2c8d700 1 -- 172.21.15.42:0/1984626425 shutdown_connections 2019-12-05T05:51:20.247 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:20.240+0000 7f8bb2c8d700 1 -- 172.21.15.42:0/1984626425 wait complete. 2019-12-05T05:51:20.247 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:20.240+0000 7f8bb2c8d700 1 -- 172.21.15.42:0/1984626425 >> 172.21.15.42:0/1984626425 conn(0x7f8bac0aabe0 msgr2=0x7f8bac118880 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:51:20.254 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 194, "feature_flags": { "enable_multiple": false, "ever_enabled_multiple": false }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2019-12-05T05:03:35.604738+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 194, "ever_allowed_features": 2, "explicitly_allowed_features": 2, "failed": [], "flags": 18, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_8430": { "addr": "172.21.15.42:6835/5991897", "addrs": { "addrvec": [ { "addr": "172.21.15.42:6834", "nonce": 5991897, "type": "v2" }, { "addr": "172.21.15.42:6835", "nonce": 5991897, "type": "v1" } ] }, "export_targets": [], "features": 4540138292836696063, "flags": 0, "gid": 8430, "incarnation": 194, "name": "a", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 82, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 2, "min_compat_client": "0 (unknown)", "modified": "2019-12-05T05:51:19.126731+0000", "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 8430 } } } ], "standbys": [] } 2019-12-05T05:51:22.255 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:51:22.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:22.344+0000 7fd07b591700 1 Processor -- start 2019-12-05T05:51:22.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:22.344+0000 7fd07b591700 1 -- start start 2019-12-05T05:51:22.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:22.344+0000 7fd07b591700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd0740a7740 0x7fd07408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:51:22.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:22.344+0000 7fd07b591700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd07407e720 0x7fd07407eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:51:22.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:22.344+0000 7fd07b591700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd07409f2e0 0x7fd07409f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:51:22.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:22.344+0000 7fd07b591700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0740a0b00 con 0x7fd07407e720 2019-12-05T05:51:22.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:22.344+0000 7fd07b591700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd07408e5b0 con 0x7fd07409f2e0 2019-12-05T05:51:22.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:22.344+0000 7fd07b591700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd07408e850 con 0x7fd0740a7740 2019-12-05T05:51:22.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:22.344+0000 7fd07a32f700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd07409f2e0 0x7fd07409f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:51:22.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:22.344+0000 7fd07a32f700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd07409f2e0 0x7fd07409f710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:42690/0 (socket says 172.21.15.42:42690) 2019-12-05T05:51:22.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:22.344+0000 7fd07a32f700 1 -- 172.21.15.42:0/4231623185 learned_addr learned my addr 172.21.15.42:0/4231623185 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:51:22.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:22.344+0000 7fd07932d700 1 --2- 172.21.15.42:0/4231623185 >> v2:172.21.15.40:3300/0 conn(0x7fd07407e720 0x7fd07407eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:51:22.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:22.344+0000 7fd079b2e700 1 --2- 172.21.15.42:0/4231623185 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd0740a7740 0x7fd07408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:51:22.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:22.344+0000 7fd07a32f700 1 -- 172.21.15.42:0/4231623185 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd0740a7740 msgr2=0x7fd07408df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:51:22.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:22.344+0000 7fd07a32f700 1 --2- 172.21.15.42:0/4231623185 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd0740a7740 0x7fd07408df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:51:22.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:22.344+0000 7fd07a32f700 1 -- 172.21.15.42:0/4231623185 >> v2:172.21.15.40:3300/0 conn(0x7fd07407e720 msgr2=0x7fd07407eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:51:22.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:22.344+0000 7fd07a32f700 1 --2- 172.21.15.42:0/4231623185 >> v2:172.21.15.40:3300/0 conn(0x7fd07407e720 0x7fd07407eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:51:22.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:22.344+0000 7fd07a32f700 1 -- 172.21.15.42:0/4231623185 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd07407f0a0 con 0x7fd07409f2e0 2019-12-05T05:51:22.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:22.344+0000 7fd07a32f700 1 --2- 172.21.15.42:0/4231623185 >> v2:172.21.15.42:3300/0 conn(0x7fd07409f2e0 0x7fd07409f710 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fd07408e770 tx=0x7fd06c010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:22.358 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:22.344+0000 7fd07a32f700 1 --2- 172.21.15.42:0/4231623185 >> v2:172.21.15.42:3300/0 conn(0x7fd07409f2e0 0x7fd07409f710 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fd07408e770 tx=0x7fd06c010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:22.358 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:22.344+0000 7fd07a32f700 1 --2- 172.21.15.42:0/4231623185 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd07409f2e0 0x7fd07409f710 secure :-1 s=READY pgs=1585 cs=0 l=1 rx=0x7fd07408e770 tx=0x7fd06c010130).ready entity=mon.1 client_cookie=1a8b3b64eba43713 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:51:22.358 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:22.344+0000 7fd07a32f700 1 --2- 172.21.15.42:0/4231623185 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd07409f2e0 0x7fd07409f710 secure :-1 s=THROTTLE_DONE pgs=1585 cs=0 l=1 rx=0x7fd07408e770 tx=0x7fd06c010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:22.358 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:22.344+0000 7fd078b2c700 1 -- 172.21.15.42:0/4231623185 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd06c010de0 con 0x7fd07409f2e0 2019-12-05T05:51:22.359 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:22.344+0000 7fd07a32f700 1 --2- 172.21.15.42:0/4231623185 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd07409f2e0 0x7fd07409f710 secure :-1 s=THROTTLE_DONE pgs=1585 cs=0 l=1 rx=0x7fd07408e770 tx=0x7fd06c010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:22.359 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:22.344+0000 7fd07a32f700 1 --2- 172.21.15.42:0/4231623185 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd07409f2e0 0x7fd07409f710 secure :-1 s=THROTTLE_DONE pgs=1585 cs=0 l=1 rx=0x7fd07408e770 tx=0x7fd06c010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:22.359 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:22.344+0000 7fd078b2c700 1 -- 172.21.15.42:0/4231623185 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd06c010a00 con 0x7fd07409f2e0 2019-12-05T05:51:22.359 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:22.348+0000 7fd078b2c700 1 -- 172.21.15.42:0/4231623185 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd06c0114a0 con 0x7fd07409f2e0 2019-12-05T05:51:22.359 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:22.348+0000 7fd07b591700 1 -- 172.21.15.42:0/4231623185 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd07409f2e0 msgr2=0x7fd07409f710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:51:22.359 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:22.348+0000 7fd07b591700 1 --2- 172.21.15.42:0/4231623185 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd07409f2e0 0x7fd07409f710 secure :-1 s=READY pgs=1585 cs=0 l=1 rx=0x7fd07408e770 tx=0x7fd06c010130).stop 2019-12-05T05:51:22.360 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:22.348+0000 7fd07b591700 1 -- 172.21.15.42:0/4231623185 shutdown_connections 2019-12-05T05:51:22.360 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:22.348+0000 7fd07b591700 1 --2- 172.21.15.42:0/4231623185 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd07409f2e0 0x7fd07409f710 unknown :-1 s=CLOSED pgs=1585 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:51:22.360 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:22.348+0000 7fd07b591700 1 --2- 172.21.15.42:0/4231623185 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd0740a7740 0x7fd07408df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:51:22.360 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:22.348+0000 7fd07b591700 1 --2- 172.21.15.42:0/4231623185 >> v2:172.21.15.40:3300/0 conn(0x7fd07407e720 0x7fd07407eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:51:22.360 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:22.348+0000 7fd07b591700 1 -- 172.21.15.42:0/4231623185 shutdown_connections 2019-12-05T05:51:22.361 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:22.348+0000 7fd07b591700 1 -- 172.21.15.42:0/4231623185 wait complete. 2019-12-05T05:51:22.361 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:22.348+0000 7fd07b591700 1 -- 172.21.15.42:0/4231623185 >> 172.21.15.42:0/4231623185 conn(0x7fd0740aabe0 msgr2=0x7fd0740a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:51:22.367 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:22.348+0000 7fd07b591700 1 Processor -- start 2019-12-05T05:51:22.367 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:22.348+0000 7fd07b591700 1 -- start start 2019-12-05T05:51:22.368 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:22.352+0000 7fd07b591700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd0740a5040 0x7fd074136880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:51:22.368 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:22.352+0000 7fd07b591700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0740a1eb0 con 0x7fd0740a1a80 2019-12-05T05:51:22.368 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:22.352+0000 7fd07b591700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0740a5900 con 0x7fd07408eeb0 2019-12-05T05:51:22.369 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:22.352+0000 7fd07b591700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0740a5b30 con 0x7fd0740a5040 2019-12-05T05:51:22.369 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:22.352+0000 7fd079b2e700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fd07408eeb0 0x7fd07407ba60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:42730/0 (socket says 172.21.15.42:42730) 2019-12-05T05:51:22.369 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:22.352+0000 7fd079b2e700 1 -- 172.21.15.42:0/1762354890 learned_addr learned my addr 172.21.15.42:0/1762354890 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:51:22.369 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:22.352+0000 7fd07a32f700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd0740a5040 0x7fd074136880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:51:22.369 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:22.352+0000 7fd06a7fc700 1 -- 172.21.15.42:0/1762354890 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2874586102 0 0) 0x7fd0740a5900 con 0x7fd07408eeb0 2019-12-05T05:51:22.369 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:22.352+0000 7fd06a7fc700 1 -- 172.21.15.42:0/1762354890 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd05c0030b0 con 0x7fd07408eeb0 2019-12-05T05:51:22.370 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:22.352+0000 7fd06a7fc700 1 -- 172.21.15.42:0/1762354890 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 271566333 0 0) 0x7fd070001ec0 con 0x7fd0740a1a80 2019-12-05T05:51:22.370 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:22.352+0000 7fd06a7fc700 1 -- 172.21.15.42:0/1762354890 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd0740a5900 con 0x7fd0740a1a80 2019-12-05T05:51:22.370 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:22.352+0000 7fd07a32f700 1 -- 172.21.15.42:0/1762354890 >> v1:172.21.15.42:6789/0 conn(0x7fd07408eeb0 legacy=0x7fd07407ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:51:22.370 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:22.352+0000 7fd07a32f700 1 -- 172.21.15.42:0/1762354890 >> v1:172.21.15.40:6789/0 conn(0x7fd0740a1a80 legacy=0x7fd0740a4840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:51:22.370 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:22.352+0000 7fd07a32f700 1 -- 172.21.15.42:0/1762354890 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd074138df0 con 0x7fd0740a5040 2019-12-05T05:51:22.370 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:22.352+0000 7fd07a32f700 1 --2- 172.21.15.42:0/1762354890 >> v2:172.21.15.42:3300/0 conn(0x7fd0740a5040 0x7fd074136880 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fd07408e770 tx=0x7fd06c010c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:22.371 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:22.352+0000 7fd07a32f700 1 --2- 172.21.15.42:0/1762354890 >> v2:172.21.15.42:3300/0 conn(0x7fd0740a5040 0x7fd074136880 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fd07408e770 tx=0x7fd06c010c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:22.371 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:22.352+0000 7fd07a32f700 1 --2- 172.21.15.42:0/1762354890 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd0740a5040 0x7fd074136880 secure :-1 s=READY pgs=1587 cs=0 l=1 rx=0x7fd07408e770 tx=0x7fd06c010c30).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:51:22.371 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:22.352+0000 7fd07a32f700 1 --2- 172.21.15.42:0/1762354890 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd0740a5040 0x7fd074136880 secure :-1 s=THROTTLE_DONE pgs=1587 cs=0 l=1 rx=0x7fd07408e770 tx=0x7fd06c010c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:22.371 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:22.352+0000 7fd06a7fc700 1 -- 172.21.15.42:0/1762354890 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd06c011b80 con 0x7fd0740a5040 2019-12-05T05:51:22.371 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:22.352+0000 7fd07a32f700 1 --2- 172.21.15.42:0/1762354890 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd0740a5040 0x7fd074136880 secure :-1 s=THROTTLE_DONE pgs=1587 cs=0 l=1 rx=0x7fd07408e770 tx=0x7fd06c010c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:22.372 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:22.352+0000 7fd06a7fc700 1 -- 172.21.15.42:0/1762354890 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd06c0110c0 con 0x7fd0740a5040 2019-12-05T05:51:22.372 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:22.352+0000 7fd07a32f700 1 --2- 172.21.15.42:0/1762354890 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd0740a5040 0x7fd074136880 secure :-1 s=THROTTLE_DONE pgs=1587 cs=0 l=1 rx=0x7fd07408e770 tx=0x7fd06c010c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:22.372 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:22.352+0000 7fd07b591700 1 -- 172.21.15.42:0/1762354890 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd074136dd0 con 0x7fd0740a5040 2019-12-05T05:51:22.372 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:22.352+0000 7fd06a7fc700 1 -- 172.21.15.42:0/1762354890 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd06c005ab0 con 0x7fd0740a5040 2019-12-05T05:51:22.372 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:22.352+0000 7fd07a32f700 1 -- 172.21.15.42:0/1762354890 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd0740a5040 msgr2=0x7fd074136880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2019-12-05T05:51:22.372 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:22.352+0000 7fd07b591700 1 -- 172.21.15.42:0/1762354890 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd074137320 con 0x7fd0740a5040 2019-12-05T05:51:22.373 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:22.352+0000 7fd07a32f700 1 -- 172.21.15.42:0/1762354890 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd0740a5040 msgr2=0x7fd074136880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2019-12-05T05:51:22.373 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:22.352+0000 7fd07a32f700 1 --2- 172.21.15.42:0/1762354890 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd0740a5040 0x7fd074136880 secure :-1 s=READY pgs=1587 cs=0 l=1 rx=0x7fd07408e770 tx=0x7fd06c010c30).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-12-05T05:51:22.373 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:22.352+0000 7fd07a32f700 1 --2- 172.21.15.42:0/1762354890 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd0740a5040 0x7fd074136880 secure :-1 s=READY pgs=1587 cs=0 l=1 rx=0x7fd07408e770 tx=0x7fd06c010c30).stop 2019-12-05T05:51:22.373 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:22.352+0000 7fd06a7fc700 1 -- 172.21.15.42:0/1762354890 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd0740a5040 msgr2=0x7fd074136880 unknown :-1 s=STATE_CLOSED l=1).mark_down 2019-12-05T05:51:22.373 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:22.352+0000 7fd06a7fc700 1 --2- 172.21.15.42:0/1762354890 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd0740a5040 0x7fd074136880 unknown :-1 s=CLOSED pgs=1587 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:51:22.373 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:22.352+0000 7fd06a7fc700 1 --2- 172.21.15.42:0/1762354890 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd05c004650 0x7fd05c004a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:51:22.374 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:22.352+0000 7fd06a7fc700 1 --2- 172.21.15.42:0/1762354890 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd05c004fd0 0x7fd05c003440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:51:22.374 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:22.352+0000 7fd06a7fc700 1 --2- 172.21.15.42:0/1762354890 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd05c003a50 0x7fd05c00b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:51:22.374 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:22.352+0000 7fd06a7fc700 1 -- 172.21.15.42:0/1762354890 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd05c004190 con 0x7fd05c004650 2019-12-05T05:51:22.374 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:22.352+0000 7fd06a7fc700 1 -- 172.21.15.42:0/1762354890 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd05c00bbc0 con 0x7fd05c003a50 2019-12-05T05:51:22.374 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:22.352+0000 7fd06a7fc700 1 -- 172.21.15.42:0/1762354890 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd05c00bdb0 con 0x7fd05c004fd0 2019-12-05T05:51:22.374 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:22.352+0000 7fd07a32f700 1 --2- 172.21.15.42:0/1762354890 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd05c003a50 0x7fd05c00b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:51:22.375 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:22.352+0000 7fd079b2e700 1 --2- 172.21.15.42:0/1762354890 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd05c004650 0x7fd05c004a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:51:22.375 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:22.352+0000 7fd07932d700 1 --2- 172.21.15.42:0/1762354890 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd05c004fd0 0x7fd05c003440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:51:22.375 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:22.356+0000 7fd07a32f700 1 -- 172.21.15.42:0/1762354890 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd05c004fd0 msgr2=0x7fd05c003440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:51:22.375 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:22.356+0000 7fd07a32f700 1 --2- 172.21.15.42:0/1762354890 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd05c004fd0 0x7fd05c003440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:51:22.375 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:22.356+0000 7fd07a32f700 1 -- 172.21.15.42:0/1762354890 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd05c004650 msgr2=0x7fd05c004a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:51:22.375 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:22.356+0000 7fd07a32f700 1 --2- 172.21.15.42:0/1762354890 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd05c004650 0x7fd05c004a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:51:22.376 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:22.356+0000 7fd07a32f700 1 -- 172.21.15.42:0/1762354890 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+,osdmap=0}) v3 -- 0x7fd05c00c170 con 0x7fd05c003a50 2019-12-05T05:51:22.376 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:22.356+0000 7fd07a32f700 1 --2- 172.21.15.42:0/1762354890 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd05c003a50 0x7fd05c00b670 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fd07408e770 tx=0x7fd06c007560).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:22.376 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:22.356+0000 7fd07a32f700 1 -- 172.21.15.42:0/1762354890 reap_dead start 2019-12-05T05:51:22.376 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:22.356+0000 7fd07a32f700 1 --2- 172.21.15.42:0/1762354890 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd05c003a50 0x7fd05c00b670 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fd07408e770 tx=0x7fd06c007560).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:22.376 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:22.356+0000 7fd07a32f700 1 --2- 172.21.15.42:0/1762354890 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd05c003a50 0x7fd05c00b670 secure :-1 s=READY pgs=1588 cs=0 l=1 rx=0x7fd07408e770 tx=0x7fd06c007560).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:51:22.377 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:22.356+0000 7fd07a32f700 1 --2- 172.21.15.42:0/1762354890 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd05c003a50 0x7fd05c00b670 secure :-1 s=THROTTLE_DONE pgs=1588 cs=0 l=1 rx=0x7fd07408e770 tx=0x7fd06c007560).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:22.377 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:22.356+0000 7fd06a7fc700 1 -- 172.21.15.42:0/1762354890 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd06c00f680 con 0x7fd05c003a50 2019-12-05T05:51:22.377 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:22.356+0000 7fd07a32f700 1 --2- 172.21.15.42:0/1762354890 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd05c003a50 0x7fd05c00b670 secure :-1 s=THROTTLE_DONE pgs=1588 cs=0 l=1 rx=0x7fd07408e770 tx=0x7fd06c007560).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:22.377 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:22.356+0000 7fd06a7fc700 1 -- 172.21.15.42:0/1762354890 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd06c011d80 con 0x7fd05c003a50 2019-12-05T05:51:22.377 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:22.356+0000 7fd07a32f700 1 --2- 172.21.15.42:0/1762354890 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd05c003a50 0x7fd05c00b670 secure :-1 s=THROTTLE_DONE pgs=1588 cs=0 l=1 rx=0x7fd07408e770 tx=0x7fd06c007560).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:22.377 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:22.356+0000 7fd06a7fc700 1 -- 172.21.15.42:0/1762354890 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fd06c00f880 con 0x7fd05c003a50 2019-12-05T05:51:22.378 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:22.356+0000 7fd07a32f700 1 --2- 172.21.15.42:0/1762354890 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd05c003a50 0x7fd05c00b670 secure :-1 s=THROTTLE_DONE pgs=1588 cs=0 l=1 rx=0x7fd07408e770 tx=0x7fd06c007560).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:22.378 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:22.356+0000 7fd06a7fc700 1 --2- 172.21.15.42:0/1762354890 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd05c03ec80 0x7fd05c041190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:51:22.378 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:22.356+0000 7fd06a7fc700 1 -- 172.21.15.42:0/1762354890 <== mon.1 v2:172.21.15.42:3300/0 4 ==== osd_map(82..82 src has 1..82) v4 ==== 8869+0+0 (secure 0 0 0) 0x7fd06c090950 con 0x7fd05c003a50 2019-12-05T05:51:22.378 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:22.356+0000 7fd079b2e700 1 --2- 172.21.15.42:0/1762354890 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd05c03ec80 0x7fd05c041190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:51:22.378 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:22.356+0000 7fd079b2e700 1 --2- 172.21.15.42:0/1762354890 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd05c03ec80 0x7fd05c041190 crc :-1 s=READY pgs=685 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:51:22.378 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:22.360+0000 7fd07b591700 1 -- 172.21.15.42:0/1762354890 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd07405d110 con 0x7fd05c003a50 2019-12-05T05:51:22.379 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:22.368+0000 7fd07a32f700 1 --2- 172.21.15.42:0/1762354890 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd05c003a50 0x7fd05c00b670 secure :-1 s=THROTTLE_DONE pgs=1588 cs=0 l=1 rx=0x7fd07408e770 tx=0x7fd06c007560).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:22.379 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:22.368+0000 7fd06a7fc700 1 -- 172.21.15.42:0/1762354890 <== mon.1 v2:172.21.15.42:3300/0 5 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fd06c00c0d0 con 0x7fd05c003a50 2019-12-05T05:51:22.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:22.524+0000 7fd07b591700 1 -- 172.21.15.42:0/1762354890 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd0740aadf0 con 0x7fd05c003a50 2019-12-05T05:51:22.530 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:22.524+0000 7fd07a32f700 1 --2- 172.21.15.42:0/1762354890 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd05c003a50 0x7fd05c00b670 secure :-1 s=THROTTLE_DONE pgs=1588 cs=0 l=1 rx=0x7fd07408e770 tx=0x7fd06c007560).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:22.530 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:22.524+0000 7fd06a7fc700 1 -- 172.21.15.42:0/1762354890 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 195 v195) v1 ==== 95+0+1768 (secure 0 0 0) 0x7fd06c07cb30 con 0x7fd05c003a50 2019-12-05T05:51:22.532 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:51:22.533 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":195,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":195,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:51:22.159008+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8430},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8430":{"gid":8430,"name":"a","rank":0,"incarnation":194,"state":"up:reconnect","state_seq":7,"addr":"172.21.15.42:6835/5991897","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":5991897},{"type":"v1","addr":"172.21.15.42:6835","nonce":5991897}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:51:22.533 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 195 2019-12-05T05:51:22.535 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:22.528+0000 7fd07b591700 1 -- 172.21.15.42:0/1762354890 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd05c03ec80 msgr2=0x7fd05c041190 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:51:22.535 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:22.528+0000 7fd07b591700 1 --2- 172.21.15.42:0/1762354890 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd05c03ec80 0x7fd05c041190 crc :-1 s=READY pgs=685 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:51:22.536 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:22.528+0000 7fd07b591700 1 -- 172.21.15.42:0/1762354890 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd05c003a50 msgr2=0x7fd05c00b670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:51:22.536 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:22.528+0000 7fd07b591700 1 --2- 172.21.15.42:0/1762354890 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd05c003a50 0x7fd05c00b670 secure :-1 s=READY pgs=1588 cs=0 l=1 rx=0x7fd07408e770 tx=0x7fd06c007560).stop 2019-12-05T05:51:22.536 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:22.528+0000 7fd07b591700 1 -- 172.21.15.42:0/1762354890 shutdown_connections 2019-12-05T05:51:22.536 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:22.528+0000 7fd07b591700 1 --2- 172.21.15.42:0/1762354890 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd05c03ec80 0x7fd05c041190 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:51:22.536 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:22.528+0000 7fd07b591700 1 --2- 172.21.15.42:0/1762354890 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd05c003a50 0x7fd05c00b670 unknown :-1 s=CLOSED pgs=1588 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:51:22.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:22.528+0000 7fd07b591700 1 --2- 172.21.15.42:0/1762354890 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd0740a5040 0x7fd074136880 unknown :-1 s=CLOSED pgs=1587 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:51:22.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:22.528+0000 7fd07b591700 1 -- 172.21.15.42:0/1762354890 shutdown_connections 2019-12-05T05:51:22.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:22.528+0000 7fd07b591700 1 -- 172.21.15.42:0/1762354890 wait complete. 2019-12-05T05:51:22.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:22.528+0000 7fd07b591700 1 -- 172.21.15.42:0/1762354890 >> 172.21.15.42:0/1762354890 conn(0x7fd07407e580 msgr2=0x7fd0740a9790 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:51:22.545 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2019-12-05T05:51:24.545 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:51:24.650 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:24.640+0000 7fc9ab6da700 1 Processor -- start 2019-12-05T05:51:24.650 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:24.640+0000 7fc9ab6da700 1 -- start start 2019-12-05T05:51:24.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:24.640+0000 7fc9ab6da700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc9a409fae0 0x7fc9a414fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:51:24.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:24.640+0000 7fc9ab6da700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9a40a0710 con 0x7fc9a408df80 2019-12-05T05:51:24.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:24.640+0000 7fc9ab6da700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9a407ef20 con 0x7fc9a40a7740 2019-12-05T05:51:24.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:24.640+0000 7fc9ab6da700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9a40a0b00 con 0x7fc9a409fae0 2019-12-05T05:51:24.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:24.640+0000 7fc9aa478700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc9a409fae0 0x7fc9a414fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:51:24.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:24.640+0000 7fc9a9c77700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fc9a40a7740 0x7fc9a407e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:42746/0 (socket says 172.21.15.42:42746) 2019-12-05T05:51:24.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:24.640+0000 7fc9a9c77700 1 -- 172.21.15.42:0/947094005 learned_addr learned my addr 172.21.15.42:0/947094005 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:51:24.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:24.644+0000 7fc9aa478700 1 -- 172.21.15.42:0/947094005 >> v1:172.21.15.42:6789/0 conn(0x7fc9a40a7740 legacy=0x7fc9a407e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:51:24.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:24.644+0000 7fc9a8c75700 1 -- 172.21.15.42:0/947094005 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1293813092 0 0) 0x7fc9a407ef20 con 0x7fc9a40a7740 2019-12-05T05:51:24.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:24.644+0000 7fc9aa478700 1 -- 172.21.15.42:0/947094005 >> v1:172.21.15.40:6789/0 conn(0x7fc9a408df80 legacy=0x7fc9a409f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:51:24.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:24.644+0000 7fc9aa478700 1 -- 172.21.15.42:0/947094005 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9a40a9ba0 con 0x7fc9a409fae0 2019-12-05T05:51:24.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:24.644+0000 7fc9aa478700 1 --2- 172.21.15.42:0/947094005 >> v2:172.21.15.42:3300/0 conn(0x7fc9a409fae0 0x7fc9a414fff0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fc99c008b10 tx=0x7fc99c00ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:24.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:24.644+0000 7fc9aa478700 1 --2- 172.21.15.42:0/947094005 >> v2:172.21.15.42:3300/0 conn(0x7fc9a409fae0 0x7fc9a414fff0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fc99c008b10 tx=0x7fc99c00ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:24.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:24.644+0000 7fc9aa478700 1 --2- 172.21.15.42:0/947094005 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc9a409fae0 0x7fc9a414fff0 secure :-1 s=READY pgs=1590 cs=0 l=1 rx=0x7fc99c008b10 tx=0x7fc99c00ff70).ready entity=mon.1 client_cookie=6fafb88cb43cf8eb server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:51:24.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:24.644+0000 7fc9aa478700 1 --2- 172.21.15.42:0/947094005 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc9a409fae0 0x7fc9a414fff0 secure :-1 s=THROTTLE_DONE pgs=1590 cs=0 l=1 rx=0x7fc99c008b10 tx=0x7fc99c00ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:24.654 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:24.644+0000 7fc9a8c75700 1 -- 172.21.15.42:0/947094005 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc99c010970 con 0x7fc9a409fae0 2019-12-05T05:51:24.654 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:24.644+0000 7fc9aa478700 1 --2- 172.21.15.42:0/947094005 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc9a409fae0 0x7fc9a414fff0 secure :-1 s=THROTTLE_DONE pgs=1590 cs=0 l=1 rx=0x7fc99c008b10 tx=0x7fc99c00ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:24.654 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:24.644+0000 7fc9a8c75700 1 -- 172.21.15.42:0/947094005 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc99c008b50 con 0x7fc9a409fae0 2019-12-05T05:51:24.654 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:24.644+0000 7fc9aa478700 1 --2- 172.21.15.42:0/947094005 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc9a409fae0 0x7fc9a414fff0 secure :-1 s=THROTTLE_DONE pgs=1590 cs=0 l=1 rx=0x7fc99c008b10 tx=0x7fc99c00ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:24.654 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:24.644+0000 7fc9a8c75700 1 -- 172.21.15.42:0/947094005 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc99c0110c0 con 0x7fc9a409fae0 2019-12-05T05:51:24.654 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:24.644+0000 7fc9ab6da700 1 -- 172.21.15.42:0/947094005 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc9a409fae0 msgr2=0x7fc9a414fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:51:24.655 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:24.644+0000 7fc9ab6da700 1 --2- 172.21.15.42:0/947094005 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc9a409fae0 0x7fc9a414fff0 secure :-1 s=READY pgs=1590 cs=0 l=1 rx=0x7fc99c008b10 tx=0x7fc99c00ff70).stop 2019-12-05T05:51:24.655 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:24.644+0000 7fc9ab6da700 1 -- 172.21.15.42:0/947094005 shutdown_connections 2019-12-05T05:51:24.655 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:24.644+0000 7fc9ab6da700 1 --2- 172.21.15.42:0/947094005 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc9a409fae0 0x7fc9a414fff0 unknown :-1 s=CLOSED pgs=1590 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:51:24.655 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:24.644+0000 7fc9ab6da700 1 -- 172.21.15.42:0/947094005 shutdown_connections 2019-12-05T05:51:24.655 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:24.644+0000 7fc9ab6da700 1 -- 172.21.15.42:0/947094005 wait complete. 2019-12-05T05:51:24.655 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:24.644+0000 7fc9ab6da700 1 -- 172.21.15.42:0/947094005 >> 172.21.15.42:0/947094005 conn(0x7fc9a40aabe0 msgr2=0x7fc9a40a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:51:24.659 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:24.644+0000 7fc9ab6da700 1 Processor -- start 2019-12-05T05:51:24.659 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:24.644+0000 7fc9ab6da700 1 -- start start 2019-12-05T05:51:24.659 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:24.648+0000 7fc9ab6da700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc9a411bef0 0x7fc9a40a3460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:51:24.659 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:24.648+0000 7fc9ab6da700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9a4145600 con 0x7fc9a4126fb0 2019-12-05T05:51:24.659 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:24.648+0000 7fc9ab6da700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9a40a2ae0 con 0x7fc9a40a1fc0 2019-12-05T05:51:24.660 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:24.648+0000 7fc9ab6da700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9a40a2f20 con 0x7fc9a411bef0 2019-12-05T05:51:24.660 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:24.648+0000 7fc9a9c77700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc9a411bef0 0x7fc9a40a3460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:51:24.660 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:24.648+0000 7fc9a9c77700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc9a411bef0 0x7fc9a40a3460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:42714/0 (socket says 172.21.15.42:42714) 2019-12-05T05:51:24.660 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:24.648+0000 7fc9a9c77700 1 -- 172.21.15.42:0/4239433846 learned_addr learned my addr 172.21.15.42:0/4239433846 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:51:24.660 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:24.648+0000 7fc9a9c77700 1 -- 172.21.15.42:0/4239433846 >> v1:172.21.15.42:6789/0 conn(0x7fc9a40a1fc0 legacy=0x7fc9a4130640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:51:24.661 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:24.648+0000 7fc9a9c77700 1 -- 172.21.15.42:0/4239433846 >> v1:172.21.15.40:6789/0 conn(0x7fc9a4126fb0 legacy=0x7fc9a4137a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:51:24.661 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:24.648+0000 7fc9a9c77700 1 -- 172.21.15.42:0/4239433846 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9a40a23f0 con 0x7fc9a411bef0 2019-12-05T05:51:24.661 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:24.648+0000 7fc9a9c77700 1 --2- 172.21.15.42:0/4239433846 >> v2:172.21.15.42:3300/0 conn(0x7fc9a411bef0 0x7fc9a40a3460 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fc9a40a3f00 tx=0x7fc9a00108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:24.661 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:24.648+0000 7fc9a9c77700 1 --2- 172.21.15.42:0/4239433846 >> v2:172.21.15.42:3300/0 conn(0x7fc9a411bef0 0x7fc9a40a3460 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fc9a40a3f00 tx=0x7fc9a00108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:24.661 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:24.648+0000 7fc9a9c77700 1 --2- 172.21.15.42:0/4239433846 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc9a411bef0 0x7fc9a40a3460 secure :-1 s=READY pgs=1592 cs=0 l=1 rx=0x7fc9a40a3f00 tx=0x7fc9a00108f0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:51:24.661 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:24.648+0000 7fc9a9c77700 1 --2- 172.21.15.42:0/4239433846 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc9a411bef0 0x7fc9a40a3460 secure :-1 s=THROTTLE_DONE pgs=1592 cs=0 l=1 rx=0x7fc9a40a3f00 tx=0x7fc9a00108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:24.662 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:24.648+0000 7fc99a7fc700 1 -- 172.21.15.42:0/4239433846 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc9a00115e0 con 0x7fc9a411bef0 2019-12-05T05:51:24.662 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:24.648+0000 7fc9a9c77700 1 --2- 172.21.15.42:0/4239433846 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc9a411bef0 0x7fc9a40a3460 secure :-1 s=THROTTLE_DONE pgs=1592 cs=0 l=1 rx=0x7fc9a40a3f00 tx=0x7fc9a00108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:24.662 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:24.648+0000 7fc9a9c77700 1 --2- 172.21.15.42:0/4239433846 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc9a411bef0 0x7fc9a40a3460 secure :-1 s=THROTTLE_DONE pgs=1592 cs=0 l=1 rx=0x7fc9a40a3f00 tx=0x7fc9a00108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:24.662 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:24.648+0000 7fc99a7fc700 1 -- 172.21.15.42:0/4239433846 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9a0011200 con 0x7fc9a411bef0 2019-12-05T05:51:24.662 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:24.648+0000 7fc99a7fc700 1 -- 172.21.15.42:0/4239433846 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc9a0011bc0 con 0x7fc9a411bef0 2019-12-05T05:51:24.662 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:24.648+0000 7fc9ab6da700 1 -- 172.21.15.42:0/4239433846 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9a41273e0 con 0x7fc9a411bef0 2019-12-05T05:51:24.663 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:24.648+0000 7fc9ab6da700 1 -- 172.21.15.42:0/4239433846 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9a41392a0 con 0x7fc9a411bef0 2019-12-05T05:51:24.663 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:24.648+0000 7fc9a9c77700 1 --2- 172.21.15.42:0/4239433846 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc9a411bef0 0x7fc9a40a3460 secure :-1 s=THROTTLE_DONE pgs=1592 cs=0 l=1 rx=0x7fc9a40a3f00 tx=0x7fc9a00108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:24.663 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:24.648+0000 7fc99a7fc700 1 -- 172.21.15.42:0/4239433846 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fc9a000b330 con 0x7fc9a411bef0 2019-12-05T05:51:24.663 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:24.648+0000 7fc9a9c77700 1 --2- 172.21.15.42:0/4239433846 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc9a411bef0 0x7fc9a40a3460 secure :-1 s=THROTTLE_DONE pgs=1592 cs=0 l=1 rx=0x7fc9a40a3f00 tx=0x7fc9a00108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:24.663 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:24.652+0000 7fc99a7fc700 1 --2- 172.21.15.42:0/4239433846 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc98c03ae00 0x7fc98c03d310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:51:24.663 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:24.652+0000 7fc99a7fc700 1 -- 172.21.15.42:0/4239433846 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8869+0+0 (secure 0 0 0) 0x7fc9a004b260 con 0x7fc9a411bef0 2019-12-05T05:51:24.664 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:24.652+0000 7fc9a9476700 1 --2- 172.21.15.42:0/4239433846 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc98c03ae00 0x7fc98c03d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:51:24.664 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:24.652+0000 7fc9a9476700 1 --2- 172.21.15.42:0/4239433846 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc98c03ae00 0x7fc98c03d310 crc :-1 s=READY pgs=686 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:51:24.664 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:24.656+0000 7fc9ab6da700 1 -- 172.21.15.42:0/4239433846 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9a40a2d10 con 0x7fc9a411bef0 2019-12-05T05:51:24.667 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:24.660+0000 7fc9a9c77700 1 --2- 172.21.15.42:0/4239433846 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc9a411bef0 0x7fc9a40a3460 secure :-1 s=THROTTLE_DONE pgs=1592 cs=0 l=1 rx=0x7fc9a40a3f00 tx=0x7fc9a00108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:24.667 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:24.660+0000 7fc99a7fc700 1 -- 172.21.15.42:0/4239433846 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fc9a0032260 con 0x7fc9a411bef0 2019-12-05T05:51:24.852 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:24.844+0000 7fc9ab6da700 1 -- 172.21.15.42:0/4239433846 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc9a405d110 con 0x7fc9a411bef0 2019-12-05T05:51:24.852 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:24.844+0000 7fc9a9c77700 1 --2- 172.21.15.42:0/4239433846 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc9a411bef0 0x7fc9a40a3460 secure :-1 s=THROTTLE_DONE pgs=1592 cs=0 l=1 rx=0x7fc9a40a3f00 tx=0x7fc9a00108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:24.852 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:24.844+0000 7fc99a7fc700 1 -- 172.21.15.42:0/4239433846 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 197 v197) v1 ==== 95+0+1765 (secure 0 0 0) 0x7fc9a00324c0 con 0x7fc9a411bef0 2019-12-05T05:51:24.852 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 197 2019-12-05T05:51:24.853 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:51:24.853 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":197,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":197,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:51:24.172910+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8430},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8430":{"gid":8430,"name":"a","rank":0,"incarnation":194,"state":"up:active","state_seq":9,"addr":"172.21.15.42:6835/5991897","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":5991897},{"type":"v1","addr":"172.21.15.42:6835","nonce":5991897}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:51:24.855 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:24.848+0000 7fc9ab6da700 1 -- 172.21.15.42:0/4239433846 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc98c03ae00 msgr2=0x7fc98c03d310 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:51:24.855 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:24.848+0000 7fc9ab6da700 1 --2- 172.21.15.42:0/4239433846 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc98c03ae00 0x7fc98c03d310 crc :-1 s=READY pgs=686 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:51:24.855 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:24.848+0000 7fc9ab6da700 1 -- 172.21.15.42:0/4239433846 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc9a411bef0 msgr2=0x7fc9a40a3460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:51:24.855 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:24.848+0000 7fc9ab6da700 1 --2- 172.21.15.42:0/4239433846 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc9a411bef0 0x7fc9a40a3460 secure :-1 s=READY pgs=1592 cs=0 l=1 rx=0x7fc9a40a3f00 tx=0x7fc9a00108f0).stop 2019-12-05T05:51:24.855 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:24.848+0000 7fc9ab6da700 1 -- 172.21.15.42:0/4239433846 shutdown_connections 2019-12-05T05:51:24.856 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:24.848+0000 7fc9ab6da700 1 --2- 172.21.15.42:0/4239433846 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc98c03ae00 0x7fc98c03d310 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:51:24.856 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:24.848+0000 7fc9ab6da700 1 --2- 172.21.15.42:0/4239433846 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc9a411bef0 0x7fc9a40a3460 unknown :-1 s=CLOSED pgs=1592 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:51:24.856 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:24.848+0000 7fc9ab6da700 1 -- 172.21.15.42:0/4239433846 shutdown_connections 2019-12-05T05:51:24.856 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:24.848+0000 7fc9ab6da700 1 -- 172.21.15.42:0/4239433846 wait complete. 2019-12-05T05:51:24.856 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:24.848+0000 7fc9ab6da700 1 -- 172.21.15.42:0/4239433846 >> 172.21.15.42:0/4239433846 conn(0x7fc9a40a7740 msgr2=0x7fc9a41190e0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:51:24.866 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a has gained rank=0, replacing gid=6245 2019-12-05T05:51:24.866 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 6 secs before reviving mds.b 2019-12-05T05:51:30.867 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.b 2019-12-05T05:51:30.867 INFO:tasks.ceph.mds.b:Restarting daemon 2019-12-05T05:51:30.868 INFO:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2019-12-05T05:51:30.872 INFO:tasks.ceph.mds.b:Started 2019-12-05T05:51:30.872 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:51:30.945 INFO:tasks.ceph.mds.b.smithi040.stdout:starting mds.b at 2019-12-05T05:51:30.978 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:30.964+0000 7f591e758700 1 Processor -- start 2019-12-05T05:51:30.978 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:30.964+0000 7f591e758700 1 -- start start 2019-12-05T05:51:30.978 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:30.968+0000 7f591e758700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f591808df80 0x7f591809f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:51:30.978 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:30.968+0000 7f591e758700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f591809f830 0x7f591814efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:51:30.978 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:30.968+0000 7f591e758700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f591809fc60 con 0x7f59180a7740 2019-12-05T05:51:30.978 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:30.968+0000 7f591e758700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59180a0b00 con 0x7f591809f830 2019-12-05T05:51:30.979 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:30.968+0000 7f591e758700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f591808e5b0 con 0x7f591808df80 2019-12-05T05:51:30.979 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:30.968+0000 7f591d4f6700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f591809f830 0x7f591814efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:51:30.979 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:30.968+0000 7f591d4f6700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f591809f830 0x7f591814efe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:42748/0 (socket says 172.21.15.42:42748) 2019-12-05T05:51:30.979 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:30.968+0000 7f591d4f6700 1 -- 172.21.15.42:0/2758866325 learned_addr learned my addr 172.21.15.42:0/2758866325 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:51:30.979 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:30.968+0000 7f5917fff700 1 --2- 172.21.15.42:0/2758866325 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f591808df80 0x7f591809f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:51:30.979 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:30.968+0000 7f591d4f6700 1 -- 172.21.15.42:0/2758866325 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f591808df80 msgr2=0x7f591809f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:51:30.979 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:30.968+0000 7f591d4f6700 1 --2- 172.21.15.42:0/2758866325 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f591808df80 0x7f591809f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:51:30.980 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:30.968+0000 7f591d4f6700 1 -- 172.21.15.42:0/2758866325 >> v1:172.21.15.42:6789/0 conn(0x7f59180a7740 legacy=0x7f591807e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:51:30.980 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:30.968+0000 7f591d4f6700 1 -- 172.21.15.42:0/2758866325 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59180a0710 con 0x7f591809f830 2019-12-05T05:51:30.980 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:30.968+0000 7f59177fe700 1 -- 172.21.15.42:0/2758866325 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 534244597 0 0) 0x7f591809fc60 con 0x7f59180a7740 2019-12-05T05:51:30.980 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:30.968+0000 7f591d4f6700 1 --2- 172.21.15.42:0/2758866325 >> v2:172.21.15.42:3300/0 conn(0x7f591809f830 0x7f591814efe0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f5910008b10 tx=0x7f59100100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:30.981 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:30.968+0000 7f591d4f6700 1 --2- 172.21.15.42:0/2758866325 >> v2:172.21.15.42:3300/0 conn(0x7f591809f830 0x7f591814efe0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f5910008b10 tx=0x7f59100100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:30.981 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:30.968+0000 7f591d4f6700 1 --2- 172.21.15.42:0/2758866325 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f591809f830 0x7f591814efe0 secure :-1 s=READY pgs=1596 cs=0 l=1 rx=0x7f5910008b10 tx=0x7f59100100b0).ready entity=mon.1 client_cookie=207f47c706863928 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:51:30.981 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:30.968+0000 7f591d4f6700 1 --2- 172.21.15.42:0/2758866325 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f591809f830 0x7f591814efe0 secure :-1 s=THROTTLE_DONE pgs=1596 cs=0 l=1 rx=0x7f5910008b10 tx=0x7f59100100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:30.981 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:30.968+0000 7f59177fe700 1 -- 172.21.15.42:0/2758866325 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5910010da0 con 0x7f591809f830 2019-12-05T05:51:30.981 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:30.968+0000 7f591d4f6700 1 --2- 172.21.15.42:0/2758866325 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f591809f830 0x7f591814efe0 secure :-1 s=THROTTLE_DONE pgs=1596 cs=0 l=1 rx=0x7f5910008b10 tx=0x7f59100100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:30.982 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:30.968+0000 7f591d4f6700 1 --2- 172.21.15.42:0/2758866325 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f591809f830 0x7f591814efe0 secure :-1 s=THROTTLE_DONE pgs=1596 cs=0 l=1 rx=0x7f5910008b10 tx=0x7f59100100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:30.982 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:30.968+0000 7f59177fe700 1 -- 172.21.15.42:0/2758866325 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59100109c0 con 0x7f591809f830 2019-12-05T05:51:30.982 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:30.968+0000 7f59177fe700 1 -- 172.21.15.42:0/2758866325 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f59100114a0 con 0x7f591809f830 2019-12-05T05:51:30.982 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:30.968+0000 7f591e758700 1 -- 172.21.15.42:0/2758866325 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f591809f830 msgr2=0x7f591814efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:51:30.982 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:30.968+0000 7f591e758700 1 --2- 172.21.15.42:0/2758866325 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f591809f830 0x7f591814efe0 secure :-1 s=READY pgs=1596 cs=0 l=1 rx=0x7f5910008b10 tx=0x7f59100100b0).stop 2019-12-05T05:51:30.982 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:30.968+0000 7f591e758700 1 -- 172.21.15.42:0/2758866325 shutdown_connections 2019-12-05T05:51:30.982 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:30.968+0000 7f591e758700 1 --2- 172.21.15.42:0/2758866325 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f591809f830 0x7f591814efe0 unknown :-1 s=CLOSED pgs=1596 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:51:30.983 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:30.968+0000 7f591e758700 1 --2- 172.21.15.42:0/2758866325 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f591808df80 0x7f591809f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:51:30.983 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:30.968+0000 7f591e758700 1 -- 172.21.15.42:0/2758866325 shutdown_connections 2019-12-05T05:51:30.983 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:30.968+0000 7f591e758700 1 -- 172.21.15.42:0/2758866325 wait complete. 2019-12-05T05:51:30.983 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:30.968+0000 7f591e758700 1 -- 172.21.15.42:0/2758866325 >> 172.21.15.42:0/2758866325 conn(0x7f59180aabe0 msgr2=0x7f59180a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:51:30.983 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:30.972+0000 7f591e758700 1 Processor -- start 2019-12-05T05:51:30.983 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:30.972+0000 7f591e758700 1 -- start start 2019-12-05T05:51:30.986 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:30.972+0000 7f591e758700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5918147fc0 0x7f5918119110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:51:30.986 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:30.972+0000 7f591e758700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f591807b370 0x7f591807b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:51:30.987 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:30.972+0000 7f591e758700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5918119660 con 0x7f591812e4c0 2019-12-05T05:51:30.987 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:30.972+0000 7f591e758700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59181199b0 con 0x7f5918147fc0 2019-12-05T05:51:30.987 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:30.972+0000 7f591e758700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5918119c50 con 0x7f591807b370 2019-12-05T05:51:30.987 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:30.972+0000 7f591ccf5700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5918147fc0 0x7f5918119110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:51:30.987 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:30.972+0000 7f591d4f6700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f591807b370 0x7f591807b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:51:30.987 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:30.972+0000 7f591ccf5700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5918147fc0 0x7f5918119110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.42:58800/0 (socket says 172.21.15.42:58800) 2019-12-05T05:51:30.988 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:30.972+0000 7f591ccf5700 1 -- 172.21.15.42:0/3948361364 learned_addr learned my addr 172.21.15.42:0/3948361364 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:51:30.988 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:30.972+0000 7f591d4f6700 1 --2- 172.21.15.42:0/3948361364 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f591807b370 0x7f591807b7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:43222/0 (socket says 172.21.15.42:43222) 2019-12-05T05:51:30.988 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:30.976+0000 7f591ccf5700 1 -- 172.21.15.42:0/3948361364 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f591807b370 msgr2=0x7f591807b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:51:30.988 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:30.976+0000 7f591ccf5700 1 --2- 172.21.15.42:0/3948361364 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f591807b370 0x7f591807b7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:51:30.988 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:30.976+0000 7f591ccf5700 1 -- 172.21.15.42:0/3948361364 >> v1:172.21.15.40:6789/0 conn(0x7f591812e4c0 legacy=0x7f591812e8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:51:30.988 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:30.976+0000 7f591ccf5700 1 -- 172.21.15.42:0/3948361364 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5918142cb0 con 0x7f5918147fc0 2019-12-05T05:51:30.989 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:30.976+0000 7f591ccf5700 1 --2- 172.21.15.42:0/3948361364 >> v2:172.21.15.40:3300/0 conn(0x7f5918147fc0 0x7f5918119110 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f5908001460 tx=0x7f59080109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:30.989 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:30.976+0000 7f591ccf5700 1 --2- 172.21.15.42:0/3948361364 >> v2:172.21.15.40:3300/0 conn(0x7f5918147fc0 0x7f5918119110 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f5908001460 tx=0x7f59080109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:30.989 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:30.976+0000 7f591ccf5700 1 --2- 172.21.15.42:0/3948361364 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5918147fc0 0x7f5918119110 secure :-1 s=READY pgs=1228 cs=0 l=1 rx=0x7f5908001460 tx=0x7f59080109a0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:51:30.989 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:30.976+0000 7f591ccf5700 1 --2- 172.21.15.42:0/3948361364 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5918147fc0 0x7f5918119110 secure :-1 s=THROTTLE_DONE pgs=1228 cs=0 l=1 rx=0x7f5908001460 tx=0x7f59080109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:30.989 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:30.976+0000 7f59157fa700 1 -- 172.21.15.42:0/3948361364 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f590800ca80 con 0x7f5918147fc0 2019-12-05T05:51:30.990 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:30.976+0000 7f591ccf5700 1 --2- 172.21.15.42:0/3948361364 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5918147fc0 0x7f5918119110 secure :-1 s=THROTTLE_DONE pgs=1228 cs=0 l=1 rx=0x7f5908001460 tx=0x7f59080109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:30.990 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:30.976+0000 7f591ccf5700 1 --2- 172.21.15.42:0/3948361364 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5918147fc0 0x7f5918119110 secure :-1 s=THROTTLE_DONE pgs=1228 cs=0 l=1 rx=0x7f5908001460 tx=0x7f59080109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:30.990 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:30.976+0000 7f59157fa700 1 -- 172.21.15.42:0/3948361364 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f590800c6a0 con 0x7f5918147fc0 2019-12-05T05:51:30.990 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:30.976+0000 7f591e758700 1 -- 172.21.15.42:0/3948361364 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f591807bcf0 con 0x7f5918147fc0 2019-12-05T05:51:30.990 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:30.976+0000 7f59157fa700 1 -- 172.21.15.42:0/3948361364 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5908011160 con 0x7f5918147fc0 2019-12-05T05:51:30.990 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:30.976+0000 7f591e758700 1 -- 172.21.15.42:0/3948361364 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5918135b40 con 0x7f5918147fc0 2019-12-05T05:51:30.990 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:30.976+0000 7f591ccf5700 1 --2- 172.21.15.42:0/3948361364 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5918147fc0 0x7f5918119110 secure :-1 s=THROTTLE_DONE pgs=1228 cs=0 l=1 rx=0x7f5908001460 tx=0x7f59080109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:30.991 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:30.976+0000 7f59157fa700 1 -- 172.21.15.42:0/3948361364 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f5908011470 con 0x7f5918147fc0 2019-12-05T05:51:30.991 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:30.976+0000 7f591ccf5700 1 --2- 172.21.15.42:0/3948361364 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5918147fc0 0x7f5918119110 secure :-1 s=THROTTLE_DONE pgs=1228 cs=0 l=1 rx=0x7f5908001460 tx=0x7f59080109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:30.991 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:30.976+0000 7f59157fa700 1 --2- 172.21.15.42:0/3948361364 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f58f803ae20 0x7f58f803d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:51:30.991 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:30.976+0000 7f59157fa700 1 -- 172.21.15.42:0/3948361364 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8869+0+0 (secure 0 0 0) 0x7f590804b170 con 0x7f5918147fc0 2019-12-05T05:51:30.992 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:30.976+0000 7f5917fff700 1 --2- 172.21.15.42:0/3948361364 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f58f803ae20 0x7f58f803d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:51:30.992 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:30.980+0000 7f5917fff700 1 --2- 172.21.15.42:0/3948361364 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f58f803ae20 0x7f58f803d330 crc :-1 s=READY pgs=687 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:51:30.992 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:30.980+0000 7f591e758700 1 -- 172.21.15.42:0/3948361364 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f591805d110 con 0x7f5918147fc0 2019-12-05T05:51:30.996 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:30.992+0000 7f591ccf5700 1 --2- 172.21.15.42:0/3948361364 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5918147fc0 0x7f5918119110 secure :-1 s=THROTTLE_DONE pgs=1228 cs=0 l=1 rx=0x7f5908001460 tx=0x7f59080109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:30.997 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:30.992+0000 7f59157fa700 1 -- 172.21.15.42:0/3948361364 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f59080321f0 con 0x7f5918147fc0 2019-12-05T05:51:31.165 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:31.160+0000 7f591e758700 1 -- 172.21.15.42:0/3948361364 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f591805b8c0 con 0x7f5918147fc0 2019-12-05T05:51:31.166 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:31.160+0000 7f591ccf5700 1 --2- 172.21.15.42:0/3948361364 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5918147fc0 0x7f5918119110 secure :-1 s=THROTTLE_DONE pgs=1228 cs=0 l=1 rx=0x7f5908001460 tx=0x7f59080109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:51:31.166 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:31.160+0000 7f59157fa700 1 -- 172.21.15.42:0/3948361364 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 198 v198) v1 ==== 95+0+2102 (secure 0 0 0) 0x7f5908032450 con 0x7f5918147fc0 2019-12-05T05:51:31.166 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 198 2019-12-05T05:51:31.166 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:51:31.166 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":198,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6356,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/1523046776","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1523046776},{"type":"v1","addr":"172.21.15.40:6833","nonce":1523046776}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":198}],"filesystems":[{"mdsmap":{"epoch":197,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:51:24.172910+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8430},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8430":{"gid":8430,"name":"a","rank":0,"incarnation":194,"state":"up:active","state_seq":9,"addr":"172.21.15.42:6835/5991897","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":5991897},{"type":"v1","addr":"172.21.15.42:6835","nonce":5991897}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:51:31.169 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:31.164+0000 7f591e758700 1 -- 172.21.15.42:0/3948361364 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f58f803ae20 msgr2=0x7f58f803d330 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:51:31.169 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:31.164+0000 7f591e758700 1 --2- 172.21.15.42:0/3948361364 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f58f803ae20 0x7f58f803d330 crc :-1 s=READY pgs=687 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:51:31.169 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:31.164+0000 7f591e758700 1 -- 172.21.15.42:0/3948361364 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5918147fc0 msgr2=0x7f5918119110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:51:31.169 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:31.164+0000 7f591e758700 1 --2- 172.21.15.42:0/3948361364 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5918147fc0 0x7f5918119110 secure :-1 s=READY pgs=1228 cs=0 l=1 rx=0x7f5908001460 tx=0x7f59080109a0).stop 2019-12-05T05:51:31.169 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:31.164+0000 7f591e758700 1 -- 172.21.15.42:0/3948361364 shutdown_connections 2019-12-05T05:51:31.170 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:31.164+0000 7f591e758700 1 --2- 172.21.15.42:0/3948361364 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f58f803ae20 0x7f58f803d330 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:51:31.170 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:31.164+0000 7f591e758700 1 --2- 172.21.15.42:0/3948361364 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f591807b370 0x7f591807b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:51:31.170 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:31.164+0000 7f591e758700 1 --2- 172.21.15.42:0/3948361364 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5918147fc0 0x7f5918119110 unknown :-1 s=CLOSED pgs=1228 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:51:31.170 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:31.164+0000 7f591e758700 1 -- 172.21.15.42:0/3948361364 shutdown_connections 2019-12-05T05:51:31.170 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:31.164+0000 7f591e758700 1 -- 172.21.15.42:0/3948361364 wait complete. 2019-12-05T05:51:31.170 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:51:31.164+0000 7f591e758700 1 -- 172.21.15.42:0/3948361364 >> 172.21.15.42:0/3948361364 conn(0x7f591807de90 msgr2=0x7f59180a6b40 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:51:31.179 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b reported in up:standby state 2019-12-05T05:51:31.179 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 68 secs before thrashing 2019-12-05T05:51:42.035 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:51:42.038 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:52:12.091 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:52:12.095 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:52:39.180 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:52:39.285 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.271+0000 7fc998c64700 1 Processor -- start 2019-12-05T05:52:39.285 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.275+0000 7fc998c64700 1 -- start start 2019-12-05T05:52:39.286 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.275+0000 7fc998c64700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc9940a7740 0x7fc99408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:52:39.286 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.275+0000 7fc998c64700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc99407e720 0x7fc99407eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:52:39.286 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.275+0000 7fc998c64700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9940a0710 con 0x7fc99409f2e0 2019-12-05T05:52:39.286 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.275+0000 7fc998c64700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9940a0b00 con 0x7fc9940a7740 2019-12-05T05:52:39.286 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.275+0000 7fc998c64700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc99408e5b0 con 0x7fc99407e720 2019-12-05T05:52:39.287 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.275+0000 7fc98a59c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc99407e720 0x7fc99407eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:52:39.287 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.275+0000 7fc99359e700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fc99409f2e0 0x7fc99409f710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:43152/0 (socket says 172.21.15.42:43152) 2019-12-05T05:52:39.287 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.275+0000 7fc992d9d700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc9940a7740 0x7fc99408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:52:39.287 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.275+0000 7fc99359e700 1 -- 172.21.15.42:0/3543988691 learned_addr learned my addr 172.21.15.42:0/3543988691 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:52:39.287 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.275+0000 7fc992d9d700 1 -- 172.21.15.42:0/3543988691 >> v2:172.21.15.42:3300/0 conn(0x7fc99407e720 msgr2=0x7fc99407eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:52:39.288 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.275+0000 7fc992d9d700 1 --2- 172.21.15.42:0/3543988691 >> v2:172.21.15.42:3300/0 conn(0x7fc99407e720 0x7fc99407eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:52:39.288 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.275+0000 7fc992d9d700 1 -- 172.21.15.42:0/3543988691 >> v1:172.21.15.42:6789/0 conn(0x7fc99409f2e0 legacy=0x7fc99409f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:52:39.288 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.275+0000 7fc992d9d700 1 -- 172.21.15.42:0/3543988691 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc99407f0a0 con 0x7fc9940a7740 2019-12-05T05:52:39.288 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.275+0000 7fc98a59c700 1 --2- 172.21.15.42:0/3543988691 >> v2:172.21.15.42:3300/0 conn(0x7fc99407e720 0x7fc99407eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:52:39.288 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.275+0000 7fc99259c700 1 -- 172.21.15.42:0/3543988691 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2140537757 0 0) 0x7fc9940a0710 con 0x7fc99409f2e0 2019-12-05T05:52:39.288 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.275+0000 7fc992d9d700 1 --2- 172.21.15.42:0/3543988691 >> v2:172.21.15.40:3300/0 conn(0x7fc9940a7740 0x7fc99408df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fc9840032b0 tx=0x7fc98400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:52:39.288 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.275+0000 7fc992d9d700 1 --2- 172.21.15.42:0/3543988691 >> v2:172.21.15.40:3300/0 conn(0x7fc9940a7740 0x7fc99408df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fc9840032b0 tx=0x7fc98400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:52:39.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.275+0000 7fc992d9d700 1 --2- 172.21.15.42:0/3543988691 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc9940a7740 0x7fc99408df80 secure :-1 s=READY pgs=1237 cs=0 l=1 rx=0x7fc9840032b0 tx=0x7fc98400fff0).ready entity=mon.0 client_cookie=d764b522cbd0b28c server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:52:39.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.275+0000 7fc992d9d700 1 --2- 172.21.15.42:0/3543988691 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc9940a7740 0x7fc99408df80 secure :-1 s=THROTTLE_DONE pgs=1237 cs=0 l=1 rx=0x7fc9840032b0 tx=0x7fc98400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:52:39.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.275+0000 7fc99259c700 1 -- 172.21.15.42:0/3543988691 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc984011a10 con 0x7fc9940a7740 2019-12-05T05:52:39.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.275+0000 7fc992d9d700 1 --2- 172.21.15.42:0/3543988691 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc9940a7740 0x7fc99408df80 secure :-1 s=THROTTLE_DONE pgs=1237 cs=0 l=1 rx=0x7fc9840032b0 tx=0x7fc98400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:52:39.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.275+0000 7fc992d9d700 1 --2- 172.21.15.42:0/3543988691 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc9940a7740 0x7fc99408df80 secure :-1 s=THROTTLE_DONE pgs=1237 cs=0 l=1 rx=0x7fc9840032b0 tx=0x7fc98400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:52:39.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.275+0000 7fc99259c700 1 -- 172.21.15.42:0/3543988691 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc984011630 con 0x7fc9940a7740 2019-12-05T05:52:39.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.275+0000 7fc99259c700 1 -- 172.21.15.42:0/3543988691 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc98400a160 con 0x7fc9940a7740 2019-12-05T05:52:39.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.275+0000 7fc998c64700 1 -- 172.21.15.42:0/3543988691 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc9940a7740 msgr2=0x7fc99408df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:52:39.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.275+0000 7fc998c64700 1 --2- 172.21.15.42:0/3543988691 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc9940a7740 0x7fc99408df80 secure :-1 s=READY pgs=1237 cs=0 l=1 rx=0x7fc9840032b0 tx=0x7fc98400fff0).stop 2019-12-05T05:52:39.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.279+0000 7fc998c64700 1 -- 172.21.15.42:0/3543988691 shutdown_connections 2019-12-05T05:52:39.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.279+0000 7fc998c64700 1 --2- 172.21.15.42:0/3543988691 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc9940a7740 0x7fc99408df80 unknown :-1 s=CLOSED pgs=1237 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:52:39.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.279+0000 7fc998c64700 1 --2- 172.21.15.42:0/3543988691 >> v2:172.21.15.42:3300/0 conn(0x7fc99407e720 0x7fc99407eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:52:39.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.279+0000 7fc998c64700 1 -- 172.21.15.42:0/3543988691 shutdown_connections 2019-12-05T05:52:39.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.279+0000 7fc998c64700 1 -- 172.21.15.42:0/3543988691 wait complete. 2019-12-05T05:52:39.291 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.279+0000 7fc998c64700 1 -- 172.21.15.42:0/3543988691 >> 172.21.15.42:0/3543988691 conn(0x7fc9940aabe0 msgr2=0x7fc9940a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:52:39.294 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.279+0000 7fc998c64700 1 Processor -- start 2019-12-05T05:52:39.294 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.283+0000 7fc998c64700 1 -- start start 2019-12-05T05:52:39.294 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.283+0000 7fc998c64700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc994116500 0x7fc994142ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:52:39.294 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.283+0000 7fc998c64700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc9941356c0 0x7fc994137b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:52:39.294 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.283+0000 7fc998c64700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc99411b5f0 con 0x7fc994143010 2019-12-05T05:52:39.295 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.283+0000 7fc998c64700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc99411b0b0 con 0x7fc994116500 2019-12-05T05:52:39.295 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.283+0000 7fc998c64700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc99411b350 con 0x7fc9941356c0 2019-12-05T05:52:39.295 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.283+0000 7fc99359e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc9941356c0 0x7fc994137b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:52:39.295 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.283+0000 7fc99359e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc9941356c0 0x7fc994137b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:43120/0 (socket says 172.21.15.42:43120) 2019-12-05T05:52:39.295 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.283+0000 7fc99359e700 1 -- 172.21.15.42:0/825770223 learned_addr learned my addr 172.21.15.42:0/825770223 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:52:39.295 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.283+0000 7fc992d9d700 1 --2- 172.21.15.42:0/825770223 >> v2:172.21.15.40:3300/0 conn(0x7fc994116500 0x7fc994142ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:52:39.295 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.283+0000 7fc99359e700 1 -- 172.21.15.42:0/825770223 >> v2:172.21.15.40:3300/0 conn(0x7fc994116500 msgr2=0x7fc994142ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:52:39.296 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.283+0000 7fc99359e700 1 --2- 172.21.15.42:0/825770223 >> v2:172.21.15.40:3300/0 conn(0x7fc994116500 0x7fc994142ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:52:39.296 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.283+0000 7fc99359e700 1 -- 172.21.15.42:0/825770223 >> v1:172.21.15.42:6789/0 conn(0x7fc994143010 legacy=0x7fc994144930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:52:39.296 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.283+0000 7fc99359e700 1 -- 172.21.15.42:0/825770223 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc994130280 con 0x7fc9941356c0 2019-12-05T05:52:39.296 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.283+0000 7fc98bfff700 1 -- 172.21.15.42:0/825770223 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2573263312 0 0) 0x7fc99411b5f0 con 0x7fc994143010 2019-12-05T05:52:39.296 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.283+0000 7fc99359e700 1 --2- 172.21.15.42:0/825770223 >> v2:172.21.15.42:3300/0 conn(0x7fc9941356c0 0x7fc994137b00 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fc98c0043a0 tx=0x7fc98c010860).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:52:39.296 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.283+0000 7fc99359e700 1 --2- 172.21.15.42:0/825770223 >> v2:172.21.15.42:3300/0 conn(0x7fc9941356c0 0x7fc994137b00 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fc98c0043a0 tx=0x7fc98c010860).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:52:39.297 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.283+0000 7fc99359e700 1 --2- 172.21.15.42:0/825770223 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc9941356c0 0x7fc994137b00 secure :-1 s=READY pgs=1608 cs=0 l=1 rx=0x7fc98c0043a0 tx=0x7fc98c010860).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:52:39.297 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.283+0000 7fc99359e700 1 --2- 172.21.15.42:0/825770223 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc9941356c0 0x7fc994137b00 secure :-1 s=THROTTLE_DONE pgs=1608 cs=0 l=1 rx=0x7fc98c0043a0 tx=0x7fc98c010860).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:52:39.297 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.283+0000 7fc98bfff700 1 -- 172.21.15.42:0/825770223 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc98c011590 con 0x7fc9941356c0 2019-12-05T05:52:39.297 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.283+0000 7fc99359e700 1 --2- 172.21.15.42:0/825770223 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc9941356c0 0x7fc994137b00 secure :-1 s=THROTTLE_DONE pgs=1608 cs=0 l=1 rx=0x7fc98c0043a0 tx=0x7fc98c010860).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:52:39.297 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.283+0000 7fc998c64700 1 -- 172.21.15.42:0/825770223 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc99411ad70 con 0x7fc9941356c0 2019-12-05T05:52:39.299 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.283+0000 7fc98bfff700 1 -- 172.21.15.42:0/825770223 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc98c0111b0 con 0x7fc9941356c0 2019-12-05T05:52:39.299 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.283+0000 7fc99359e700 1 --2- 172.21.15.42:0/825770223 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc9941356c0 0x7fc994137b00 secure :-1 s=THROTTLE_DONE pgs=1608 cs=0 l=1 rx=0x7fc98c0043a0 tx=0x7fc98c010860).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:52:39.299 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.283+0000 7fc98bfff700 1 -- 172.21.15.42:0/825770223 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc98c011b70 con 0x7fc9941356c0 2019-12-05T05:52:39.299 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.283+0000 7fc998c64700 1 -- 172.21.15.42:0/825770223 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc99411a470 con 0x7fc9941356c0 2019-12-05T05:52:39.299 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.283+0000 7fc99359e700 1 --2- 172.21.15.42:0/825770223 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc9941356c0 0x7fc994137b00 secure :-1 s=THROTTLE_DONE pgs=1608 cs=0 l=1 rx=0x7fc98c0043a0 tx=0x7fc98c010860).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:52:39.300 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.283+0000 7fc98bfff700 1 -- 172.21.15.42:0/825770223 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fc98c00b330 con 0x7fc9941356c0 2019-12-05T05:52:39.300 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.283+0000 7fc99359e700 1 --2- 172.21.15.42:0/825770223 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc9941356c0 0x7fc994137b00 secure :-1 s=THROTTLE_DONE pgs=1608 cs=0 l=1 rx=0x7fc98c0043a0 tx=0x7fc98c010860).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:52:39.300 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.287+0000 7fc98bfff700 1 --2- 172.21.15.42:0/825770223 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc97803aca0 0x7fc97803d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:52:39.300 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.287+0000 7fc98bfff700 1 -- 172.21.15.42:0/825770223 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8869+0+0 (secure 0 0 0) 0x7fc98c04b220 con 0x7fc9941356c0 2019-12-05T05:52:39.300 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.287+0000 7fc992d9d700 1 --2- 172.21.15.42:0/825770223 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc97803aca0 0x7fc97803d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:52:39.300 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.287+0000 7fc992d9d700 1 --2- 172.21.15.42:0/825770223 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc97803aca0 0x7fc97803d1b0 crc :-1 s=READY pgs=689 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:52:39.301 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.287+0000 7fc998c64700 1 -- 172.21.15.42:0/825770223 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc99405d110 con 0x7fc9941356c0 2019-12-05T05:52:39.302 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.295+0000 7fc99359e700 1 --2- 172.21.15.42:0/825770223 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc9941356c0 0x7fc994137b00 secure :-1 s=THROTTLE_DONE pgs=1608 cs=0 l=1 rx=0x7fc98c0043a0 tx=0x7fc98c010860).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:52:39.302 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.295+0000 7fc98bfff700 1 -- 172.21.15.42:0/825770223 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fc98c032220 con 0x7fc9941356c0 2019-12-05T05:52:39.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.491+0000 7fc998c64700 1 -- 172.21.15.42:0/825770223 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc99405b8c0 con 0x7fc9941356c0 2019-12-05T05:52:39.496 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.491+0000 7fc99359e700 1 --2- 172.21.15.42:0/825770223 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc9941356c0 0x7fc994137b00 secure :-1 s=THROTTLE_DONE pgs=1608 cs=0 l=1 rx=0x7fc98c0043a0 tx=0x7fc98c010860).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:52:39.496 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.491+0000 7fc98bfff700 1 -- 172.21.15.42:0/825770223 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 198 v198) v1 ==== 95+0+2102 (secure 0 0 0) 0x7fc98c032480 con 0x7fc9941356c0 2019-12-05T05:52:39.497 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 198 2019-12-05T05:52:39.497 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:52:39.497 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":198,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6356,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/1523046776","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1523046776},{"type":"v1","addr":"172.21.15.40:6833","nonce":1523046776}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":198}],"filesystems":[{"mdsmap":{"epoch":197,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:51:24.172910+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8430},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8430":{"gid":8430,"name":"a","rank":0,"incarnation":194,"state":"up:active","state_seq":9,"addr":"172.21.15.42:6835/5991897","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":5991897},{"type":"v1","addr":"172.21.15.42:6835","nonce":5991897}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:52:39.498 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.491+0000 7fc998c64700 1 -- 172.21.15.42:0/825770223 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc97803aca0 msgr2=0x7fc97803d1b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:52:39.499 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.491+0000 7fc998c64700 1 --2- 172.21.15.42:0/825770223 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc97803aca0 0x7fc97803d1b0 crc :-1 s=READY pgs=689 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:52:39.499 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.491+0000 7fc998c64700 1 -- 172.21.15.42:0/825770223 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc9941356c0 msgr2=0x7fc994137b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:52:39.499 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.491+0000 7fc998c64700 1 --2- 172.21.15.42:0/825770223 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc9941356c0 0x7fc994137b00 secure :-1 s=READY pgs=1608 cs=0 l=1 rx=0x7fc98c0043a0 tx=0x7fc98c010860).stop 2019-12-05T05:52:39.500 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.491+0000 7fc998c64700 1 -- 172.21.15.42:0/825770223 shutdown_connections 2019-12-05T05:52:39.501 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.491+0000 7fc998c64700 1 --2- 172.21.15.42:0/825770223 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc97803aca0 0x7fc97803d1b0 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:52:39.501 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.491+0000 7fc998c64700 1 --2- 172.21.15.42:0/825770223 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc9941356c0 0x7fc994137b00 unknown :-1 s=CLOSED pgs=1608 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:52:39.501 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.491+0000 7fc998c64700 1 --2- 172.21.15.42:0/825770223 >> v2:172.21.15.40:3300/0 conn(0x7fc994116500 0x7fc994142ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:52:39.501 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.495+0000 7fc998c64700 1 -- 172.21.15.42:0/825770223 shutdown_connections 2019-12-05T05:52:39.501 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.495+0000 7fc998c64700 1 -- 172.21.15.42:0/825770223 wait complete. 2019-12-05T05:52:39.502 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.495+0000 7fc998c64700 1 -- 172.21.15.42:0/825770223 >> 172.21.15.42:0/825770223 conn(0x7fc99408fde0 msgr2=0x7fc9940a7f90 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:52:39.509 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.a (rank=0) 2019-12-05T05:52:39.509 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2019-12-05T05:52:39.509 INFO:teuthology.orchestra.run:waiting for 300 2019-12-05T05:52:39.522 INFO:tasks.ceph.mds.a:Stopped 2019-12-05T05:52:39.523 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:52:39.685 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.675+0000 7fc204677700 1 Processor -- start 2019-12-05T05:52:39.685 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.675+0000 7fc204677700 1 -- start start 2019-12-05T05:52:39.685 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.675+0000 7fc204677700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc1fc08df80 0x7fc1fc09f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:52:39.686 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.675+0000 7fc204677700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1fc09fc60 con 0x7fc1fc09f830 2019-12-05T05:52:39.686 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.675+0000 7fc204677700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1fc0a0710 con 0x7fc1fc0a7740 2019-12-05T05:52:39.686 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.675+0000 7fc204677700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1fc0a0b00 con 0x7fc1fc08df80 2019-12-05T05:52:39.686 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.675+0000 7fc202413700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc1fc08df80 0x7fc1fc09f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:52:39.686 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.675+0000 7fc202c14700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fc1fc0a7740 0x7fc1fc07e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:43164/0 (socket says 172.21.15.42:43164) 2019-12-05T05:52:39.686 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.675+0000 7fc202c14700 1 -- 172.21.15.42:0/925145359 learned_addr learned my addr 172.21.15.42:0/925145359 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:52:39.687 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.675+0000 7fc203415700 1 -- 172.21.15.42:0/925145359 >> v1:172.21.15.40:6789/0 conn(0x7fc1fc09f830 legacy=0x7fc1fc14efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2019-12-05T05:52:39.687 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.675+0000 7fc203415700 1 -- 172.21.15.42:0/925145359 >> v1:172.21.15.40:6789/0 conn(0x7fc1fc09f830 legacy=0x7fc1fc14efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2019-12-05T05:52:39.687 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.675+0000 7fc203415700 1 --1- 172.21.15.42:0/925145359 >> v1:172.21.15.40:6789/0 conn(0x7fc1fc09f830 0x7fc1fc14efe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify read banner and identify addresses failed 2019-12-05T05:52:39.687 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.675+0000 7fc201c12700 1 -- 172.21.15.42:0/925145359 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1058589018 0 0) 0x7fc1fc0a0710 con 0x7fc1fc0a7740 2019-12-05T05:52:39.687 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.675+0000 7fc202413700 1 -- 172.21.15.42:0/925145359 >> v1:172.21.15.42:6789/0 conn(0x7fc1fc0a7740 legacy=0x7fc1fc07e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:52:39.687 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.675+0000 7fc202413700 1 -- 172.21.15.42:0/925145359 >> v1:172.21.15.40:6789/0 conn(0x7fc1fc09f830 legacy=0x7fc1fc14efe0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2019-12-05T05:52:39.688 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.675+0000 7fc202413700 1 -- 172.21.15.42:0/925145359 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1fc07ef20 con 0x7fc1fc08df80 2019-12-05T05:52:39.688 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.675+0000 7fc202413700 1 --2- 172.21.15.42:0/925145359 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc1fc08df80 0x7fc1fc09f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fc1f00032b0 tx=0x7fc1f000ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:52:39.688 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.675+0000 7fc202413700 1 --2- 172.21.15.42:0/925145359 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc1fc08df80 0x7fc1fc09f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fc1f00032b0 tx=0x7fc1f000ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:52:39.688 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.675+0000 7fc202413700 1 --2- 172.21.15.42:0/925145359 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc1fc08df80 0x7fc1fc09f2e0 secure :-1 s=READY pgs=356 cs=0 l=1 rx=0x7fc1f00032b0 tx=0x7fc1f000ff70).ready entity=mon.2 client_cookie=97d96f4d28d13c35 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:52:39.688 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.679+0000 7fc202413700 1 --2- 172.21.15.42:0/925145359 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc1fc08df80 0x7fc1fc09f2e0 secure :-1 s=THROTTLE_DONE pgs=356 cs=0 l=1 rx=0x7fc1f00032b0 tx=0x7fc1f000ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:52:39.689 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.679+0000 7fc201c12700 1 -- 172.21.15.42:0/925145359 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc1f00109b0 con 0x7fc1fc08df80 2019-12-05T05:52:39.689 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.679+0000 7fc202413700 1 --2- 172.21.15.42:0/925145359 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc1fc08df80 0x7fc1fc09f2e0 secure :-1 s=THROTTLE_DONE pgs=356 cs=0 l=1 rx=0x7fc1f00032b0 tx=0x7fc1f000ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:52:39.689 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.679+0000 7fc201c12700 1 -- 172.21.15.42:0/925145359 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1f0008b50 con 0x7fc1fc08df80 2019-12-05T05:52:39.689 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.679+0000 7fc202413700 1 --2- 172.21.15.42:0/925145359 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc1fc08df80 0x7fc1fc09f2e0 secure :-1 s=THROTTLE_DONE pgs=356 cs=0 l=1 rx=0x7fc1f00032b0 tx=0x7fc1f000ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:52:39.689 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.679+0000 7fc204677700 1 -- 172.21.15.42:0/925145359 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc1fc08df80 msgr2=0x7fc1fc09f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:52:39.689 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.679+0000 7fc201c12700 1 -- 172.21.15.42:0/925145359 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc1f00110c0 con 0x7fc1fc08df80 2019-12-05T05:52:39.690 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.679+0000 7fc204677700 1 --2- 172.21.15.42:0/925145359 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc1fc08df80 0x7fc1fc09f2e0 secure :-1 s=READY pgs=356 cs=0 l=1 rx=0x7fc1f00032b0 tx=0x7fc1f000ff70).stop 2019-12-05T05:52:39.690 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.679+0000 7fc204677700 1 -- 172.21.15.42:0/925145359 shutdown_connections 2019-12-05T05:52:39.690 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.679+0000 7fc204677700 1 --2- 172.21.15.42:0/925145359 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc1fc08df80 0x7fc1fc09f2e0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:52:39.690 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.679+0000 7fc204677700 1 -- 172.21.15.42:0/925145359 shutdown_connections 2019-12-05T05:52:39.690 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.679+0000 7fc204677700 1 -- 172.21.15.42:0/925145359 wait complete. 2019-12-05T05:52:39.690 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.679+0000 7fc204677700 1 -- 172.21.15.42:0/925145359 >> 172.21.15.42:0/925145359 conn(0x7fc1fc0aabe0 msgr2=0x7fc1fc0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:52:39.693 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.679+0000 7fc204677700 1 Processor -- start 2019-12-05T05:52:39.693 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.679+0000 7fc204677700 1 -- start start 2019-12-05T05:52:39.694 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.679+0000 7fc204677700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc1fc1179e0 0x7fc1fc117e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:52:39.694 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.679+0000 7fc204677700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc1fc131a10 0x7fc1fc131e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:52:39.694 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.679+0000 7fc204677700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1fc124340 con 0x7fc1fc08eeb0 2019-12-05T05:52:39.694 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.679+0000 7fc204677700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1fc124cd0 con 0x7fc1fc131a10 2019-12-05T05:52:39.694 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.679+0000 7fc204677700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1fc124790 con 0x7fc1fc1179e0 2019-12-05T05:52:39.694 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.683+0000 7fc202413700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc1fc1179e0 0x7fc1fc117e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:52:39.695 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.683+0000 7fc202413700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc1fc1179e0 0x7fc1fc117e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:43600/0 (socket says 172.21.15.42:43600) 2019-12-05T05:52:39.695 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.683+0000 7fc202413700 1 -- 172.21.15.42:0/615056661 learned_addr learned my addr 172.21.15.42:0/615056661 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:52:39.695 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.683+0000 7fc202c14700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fc1fc08eeb0 0x7fc1fc123740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:52610/0 (socket says 172.21.15.42:52610) 2019-12-05T05:52:39.695 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.683+0000 7fc203415700 1 --2- 172.21.15.42:0/615056661 >> v2:172.21.15.40:3300/0 conn(0x7fc1fc131a10 0x7fc1fc131e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:52:39.695 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.683+0000 7fc202413700 1 -- 172.21.15.42:0/615056661 >> v2:172.21.15.40:3300/0 conn(0x7fc1fc131a10 msgr2=0x7fc1fc131e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:52:39.695 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.683+0000 7fc202413700 1 --2- 172.21.15.42:0/615056661 >> v2:172.21.15.40:3300/0 conn(0x7fc1fc131a10 0x7fc1fc131e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:52:39.695 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.683+0000 7fc202413700 1 -- 172.21.15.42:0/615056661 >> v1:172.21.15.40:6789/0 conn(0x7fc1fc08eeb0 legacy=0x7fc1fc123740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:52:39.695 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.683+0000 7fc202413700 1 -- 172.21.15.42:0/615056661 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1fc118360 con 0x7fc1fc1179e0 2019-12-05T05:52:39.696 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.683+0000 7fc202413700 1 --2- 172.21.15.42:0/615056661 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc1fc1179e0 0x7fc1fc117e10 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fc1f0011310 tx=0x7fc1f0010f20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:52:39.696 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.683+0000 7fc202413700 1 --2- 172.21.15.42:0/615056661 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc1fc1179e0 0x7fc1fc117e10 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fc1f0011310 tx=0x7fc1f0010f20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:52:39.696 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.683+0000 7fc202413700 1 --2- 172.21.15.42:0/615056661 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc1fc1179e0 0x7fc1fc117e10 secure :-1 s=READY pgs=357 cs=0 l=1 rx=0x7fc1f0011310 tx=0x7fc1f0010f20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:52:39.696 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.683+0000 7fc202413700 1 --2- 172.21.15.42:0/615056661 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc1fc1179e0 0x7fc1fc117e10 secure :-1 s=THROTTLE_DONE pgs=357 cs=0 l=1 rx=0x7fc1f0011310 tx=0x7fc1f0010f20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:52:39.696 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.683+0000 7fc1ef7fe700 1 -- 172.21.15.42:0/615056661 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc1f0005cc0 con 0x7fc1fc1179e0 2019-12-05T05:52:39.696 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.683+0000 7fc202413700 1 --2- 172.21.15.42:0/615056661 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc1fc1179e0 0x7fc1fc117e10 secure :-1 s=THROTTLE_DONE pgs=357 cs=0 l=1 rx=0x7fc1f0011310 tx=0x7fc1f0010f20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:52:39.697 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.683+0000 7fc204677700 1 -- 172.21.15.42:0/615056661 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1fc132390 con 0x7fc1fc1179e0 2019-12-05T05:52:39.697 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.683+0000 7fc1ef7fe700 1 -- 172.21.15.42:0/615056661 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1f0010bb0 con 0x7fc1fc1179e0 2019-12-05T05:52:39.697 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.683+0000 7fc202413700 1 --2- 172.21.15.42:0/615056661 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc1fc1179e0 0x7fc1fc117e10 secure :-1 s=THROTTLE_DONE pgs=357 cs=0 l=1 rx=0x7fc1f0011310 tx=0x7fc1f0010f20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:52:39.697 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.683+0000 7fc1ef7fe700 1 -- 172.21.15.42:0/615056661 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc1f0007540 con 0x7fc1fc1179e0 2019-12-05T05:52:39.697 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.683+0000 7fc204677700 1 -- 172.21.15.42:0/615056661 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1fc07dab0 con 0x7fc1fc1179e0 2019-12-05T05:52:39.697 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.683+0000 7fc202413700 1 --2- 172.21.15.42:0/615056661 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc1fc1179e0 0x7fc1fc117e10 secure :-1 s=THROTTLE_DONE pgs=357 cs=0 l=1 rx=0x7fc1f0011310 tx=0x7fc1f0010f20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:52:39.697 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.683+0000 7fc1ef7fe700 1 -- 172.21.15.42:0/615056661 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fc1f000f320 con 0x7fc1fc1179e0 2019-12-05T05:52:39.697 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.683+0000 7fc202413700 1 --2- 172.21.15.42:0/615056661 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc1fc1179e0 0x7fc1fc117e10 secure :-1 s=THROTTLE_DONE pgs=357 cs=0 l=1 rx=0x7fc1f0011310 tx=0x7fc1f0010f20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:52:39.698 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.687+0000 7fc1ef7fe700 1 --2- 172.21.15.42:0/615056661 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc1e403ad80 0x7fc1e403d290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:52:39.698 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.687+0000 7fc1ef7fe700 1 -- 172.21.15.42:0/615056661 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8869+0+0 (secure 0 0 0) 0x7fc1f004b390 con 0x7fc1fc1179e0 2019-12-05T05:52:39.698 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.687+0000 7fc202c14700 1 --2- 172.21.15.42:0/615056661 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc1e403ad80 0x7fc1e403d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:52:39.698 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.687+0000 7fc202c14700 1 --2- 172.21.15.42:0/615056661 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc1e403ad80 0x7fc1e403d290 crc :-1 s=READY pgs=690 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:52:39.699 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.691+0000 7fc204677700 1 -- 172.21.15.42:0/615056661 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1fc05d110 con 0x7fc1fc1179e0 2019-12-05T05:52:39.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.699+0000 7fc202413700 1 --2- 172.21.15.42:0/615056661 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc1fc1179e0 0x7fc1fc117e10 secure :-1 s=THROTTLE_DONE pgs=357 cs=0 l=1 rx=0x7fc1f0011310 tx=0x7fc1f0010f20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:52:39.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.699+0000 7fc1ef7fe700 1 -- 172.21.15.42:0/615056661 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fc1f0032350 con 0x7fc1fc1179e0 2019-12-05T05:52:39.876 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.871+0000 7fc204677700 1 -- 172.21.15.42:0/615056661 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc1fc124570 con 0x7fc1fc1179e0 2019-12-05T05:52:39.876 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.871+0000 7fc202413700 1 --2- 172.21.15.42:0/615056661 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc1fc1179e0 0x7fc1fc117e10 secure :-1 s=THROTTLE_DONE pgs=357 cs=0 l=1 rx=0x7fc1f0011310 tx=0x7fc1f0010f20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:52:39.876 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.871+0000 7fc1ef7fe700 1 -- 172.21.15.42:0/615056661 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 198 v198) v1 ==== 95+0+2102 (secure 0 0 0) 0x7fc1f00325b0 con 0x7fc1fc1179e0 2019-12-05T05:52:39.878 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:52:39.879 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":198,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6356,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/1523046776","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1523046776},{"type":"v1","addr":"172.21.15.40:6833","nonce":1523046776}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":198}],"filesystems":[{"mdsmap":{"epoch":197,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:51:24.172910+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8430},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8430":{"gid":8430,"name":"a","rank":0,"incarnation":194,"state":"up:active","state_seq":9,"addr":"172.21.15.42:6835/5991897","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":5991897},{"type":"v1","addr":"172.21.15.42:6835","nonce":5991897}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:52:39.879 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 198 2019-12-05T05:52:39.881 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.875+0000 7fc204677700 1 -- 172.21.15.42:0/615056661 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc1e403ad80 msgr2=0x7fc1e403d290 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:52:39.881 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.875+0000 7fc204677700 1 --2- 172.21.15.42:0/615056661 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc1e403ad80 0x7fc1e403d290 crc :-1 s=READY pgs=690 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:52:39.882 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.875+0000 7fc204677700 1 -- 172.21.15.42:0/615056661 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc1fc1179e0 msgr2=0x7fc1fc117e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:52:39.882 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.875+0000 7fc204677700 1 --2- 172.21.15.42:0/615056661 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc1fc1179e0 0x7fc1fc117e10 secure :-1 s=READY pgs=357 cs=0 l=1 rx=0x7fc1f0011310 tx=0x7fc1f0010f20).stop 2019-12-05T05:52:39.882 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.875+0000 7fc204677700 1 -- 172.21.15.42:0/615056661 shutdown_connections 2019-12-05T05:52:39.882 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.875+0000 7fc204677700 1 --2- 172.21.15.42:0/615056661 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc1e403ad80 0x7fc1e403d290 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:52:39.882 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.875+0000 7fc204677700 1 --2- 172.21.15.42:0/615056661 >> v2:172.21.15.40:3300/0 conn(0x7fc1fc131a10 0x7fc1fc131e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:52:39.882 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.875+0000 7fc204677700 1 --2- 172.21.15.42:0/615056661 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc1fc1179e0 0x7fc1fc117e10 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:52:39.882 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.875+0000 7fc204677700 1 -- 172.21.15.42:0/615056661 shutdown_connections 2019-12-05T05:52:39.883 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.875+0000 7fc204677700 1 -- 172.21.15.42:0/615056661 wait complete. 2019-12-05T05:52:39.883 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:39.875+0000 7fc204677700 1 -- 172.21.15.42:0/615056661 >> 172.21.15.42:0/615056661 conn(0x7fc1fc07eac0 msgr2=0x7fc1fc0a96e0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:52:39.892 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T05:52:41.893 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:52:42.005 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:41.995+0000 7efeaa84f700 1 Processor -- start 2019-12-05T05:52:42.005 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:41.995+0000 7efeaa84f700 1 -- start start 2019-12-05T05:52:42.005 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:41.995+0000 7efeaa84f700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7efea408df70 0x7efea409f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:52:42.005 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:41.995+0000 7efeaa84f700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7efea409f820 0x7efea414efd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:52:42.005 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:41.995+0000 7efeaa84f700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efea409fc50 con 0x7efea40a7730 2019-12-05T05:52:42.005 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:41.995+0000 7efeaa84f700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efea40a0af0 con 0x7efea409f820 2019-12-05T05:52:42.006 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:41.995+0000 7efeaa84f700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efea408e5a0 con 0x7efea408df70 2019-12-05T05:52:42.006 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:41.995+0000 7efea3fff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7efea408df70 0x7efea409f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:52:42.006 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:41.995+0000 7efea3fff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7efea408df70 0x7efea409f2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:43152/0 (socket says 172.21.15.42:43152) 2019-12-05T05:52:42.006 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:41.995+0000 7efea3fff700 1 -- 172.21.15.42:0/1939559961 learned_addr learned my addr 172.21.15.42:0/1939559961 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:52:42.006 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:41.995+0000 7efea95ed700 1 --2- 172.21.15.42:0/1939559961 >> v2:172.21.15.40:3300/0 conn(0x7efea409f820 0x7efea414efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:52:42.006 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:41.995+0000 7efea3fff700 1 -- 172.21.15.42:0/1939559961 >> v2:172.21.15.40:3300/0 conn(0x7efea409f820 msgr2=0x7efea414efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:52:42.006 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:41.995+0000 7efea3fff700 1 --2- 172.21.15.42:0/1939559961 >> v2:172.21.15.40:3300/0 conn(0x7efea409f820 0x7efea414efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:52:42.007 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:41.995+0000 7efea3fff700 1 -- 172.21.15.42:0/1939559961 >> v1:172.21.15.40:6789/0 conn(0x7efea40a7730 legacy=0x7efea407e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:52:42.007 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:41.995+0000 7efea3fff700 1 -- 172.21.15.42:0/1939559961 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efea40a0700 con 0x7efea408df70 2019-12-05T05:52:42.007 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:41.995+0000 7efea3fff700 1 --2- 172.21.15.42:0/1939559961 >> v2:172.21.15.42:3300/0 conn(0x7efea408df70 0x7efea409f2d0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7efe8c0032b0 tx=0x7efe8c0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:52:42.007 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:41.995+0000 7efea3fff700 1 --2- 172.21.15.42:0/1939559961 >> v2:172.21.15.42:3300/0 conn(0x7efea408df70 0x7efea409f2d0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7efe8c0032b0 tx=0x7efe8c0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:52:42.007 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:41.995+0000 7efea3fff700 1 --2- 172.21.15.42:0/1939559961 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efea408df70 0x7efea409f2d0 secure :-1 s=READY pgs=1610 cs=0 l=1 rx=0x7efe8c0032b0 tx=0x7efe8c0100b0).ready entity=mon.1 client_cookie=e4c87465b58552a6 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:52:42.007 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:41.995+0000 7efea3fff700 1 --2- 172.21.15.42:0/1939559961 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efea408df70 0x7efea409f2d0 secure :-1 s=THROTTLE_DONE pgs=1610 cs=0 l=1 rx=0x7efe8c0032b0 tx=0x7efe8c0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:52:42.007 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:41.995+0000 7efea3fff700 1 --2- 172.21.15.42:0/1939559961 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efea408df70 0x7efea409f2d0 secure :-1 s=THROTTLE_DONE pgs=1610 cs=0 l=1 rx=0x7efe8c0032b0 tx=0x7efe8c0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:52:42.008 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:41.995+0000 7efea3fff700 1 --2- 172.21.15.42:0/1939559961 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efea408df70 0x7efea409f2d0 secure :-1 s=THROTTLE_DONE pgs=1610 cs=0 l=1 rx=0x7efe8c0032b0 tx=0x7efe8c0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:52:42.008 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:41.995+0000 7efea37fe700 1 -- 172.21.15.42:0/1939559961 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efe8c010d60 con 0x7efea408df70 2019-12-05T05:52:42.008 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:41.995+0000 7efea37fe700 1 -- 172.21.15.42:0/1939559961 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe8c010980 con 0x7efea408df70 2019-12-05T05:52:42.008 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:41.995+0000 7efea37fe700 1 -- 172.21.15.42:0/1939559961 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efe8c0114a0 con 0x7efea408df70 2019-12-05T05:52:42.008 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:41.995+0000 7efeaa84f700 1 -- 172.21.15.42:0/1939559961 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efea408df70 msgr2=0x7efea409f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:52:42.008 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:41.995+0000 7efeaa84f700 1 --2- 172.21.15.42:0/1939559961 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efea408df70 0x7efea409f2d0 secure :-1 s=READY pgs=1610 cs=0 l=1 rx=0x7efe8c0032b0 tx=0x7efe8c0100b0).stop 2019-12-05T05:52:42.008 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:41.999+0000 7efeaa84f700 1 -- 172.21.15.42:0/1939559961 shutdown_connections 2019-12-05T05:52:42.009 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:41.999+0000 7efeaa84f700 1 --2- 172.21.15.42:0/1939559961 >> v2:172.21.15.40:3300/0 conn(0x7efea409f820 0x7efea414efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:52:42.009 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:41.999+0000 7efeaa84f700 1 --2- 172.21.15.42:0/1939559961 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efea408df70 0x7efea409f2d0 unknown :-1 s=CLOSED pgs=1610 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:52:42.009 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:41.999+0000 7efeaa84f700 1 -- 172.21.15.42:0/1939559961 shutdown_connections 2019-12-05T05:52:42.009 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:41.999+0000 7efeaa84f700 1 -- 172.21.15.42:0/1939559961 wait complete. 2019-12-05T05:52:42.009 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:41.999+0000 7efeaa84f700 1 -- 172.21.15.42:0/1939559961 >> 172.21.15.42:0/1939559961 conn(0x7efea40aabd0 msgr2=0x7efea40a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:52:42.011 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:41.999+0000 7efeaa84f700 1 Processor -- start 2019-12-05T05:52:42.012 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:41.999+0000 7efeaa84f700 1 -- start start 2019-12-05T05:52:42.012 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:41.999+0000 7efeaa84f700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efea411a640 0x7efea411aa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:52:42.012 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:41.999+0000 7efeaa84f700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efea411d540 con 0x7efea40722b0 2019-12-05T05:52:42.012 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:41.999+0000 7efeaa84f700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efea411c220 con 0x7efea411ece0 2019-12-05T05:52:42.012 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:41.999+0000 7efeaa84f700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efea411afc0 con 0x7efea411a640 2019-12-05T05:52:42.012 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:41.999+0000 7efea8dec700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7efea411ece0 0x7efea411ba20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:43194/0 (socket says 172.21.15.42:43194) 2019-12-05T05:52:42.013 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:41.999+0000 7efea8dec700 1 -- 172.21.15.42:0/4206391819 learned_addr learned my addr 172.21.15.42:0/4206391819 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:52:42.013 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:41.999+0000 7efea95ed700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7efea40722b0 0x7efea40746f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:52638/0 (socket says 172.21.15.42:52638) 2019-12-05T05:52:42.013 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:41.999+0000 7efea3fff700 1 --2- 172.21.15.42:0/4206391819 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efea411a640 0x7efea411aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:52:42.013 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:42.003+0000 7efea17fa700 1 -- 172.21.15.42:0/4206391819 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4104388028 0 0) 0x7efea411d540 con 0x7efea40722b0 2019-12-05T05:52:42.013 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:42.003+0000 7efea17fa700 1 -- 172.21.15.42:0/4206391819 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe880030b0 con 0x7efea40722b0 2019-12-05T05:52:42.013 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:42.003+0000 7efea17fa700 1 -- 172.21.15.42:0/4206391819 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3038769489 0 0) 0x7efea411c220 con 0x7efea411ece0 2019-12-05T05:52:42.013 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:42.003+0000 7efea17fa700 1 -- 172.21.15.42:0/4206391819 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efea411d540 con 0x7efea411ece0 2019-12-05T05:52:42.014 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:42.003+0000 7efea3fff700 1 -- 172.21.15.42:0/4206391819 >> v1:172.21.15.42:6789/0 conn(0x7efea411ece0 legacy=0x7efea411ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:52:42.014 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:42.003+0000 7efea3fff700 1 -- 172.21.15.42:0/4206391819 >> v1:172.21.15.40:6789/0 conn(0x7efea40722b0 legacy=0x7efea40746f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:52:42.014 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:42.003+0000 7efea3fff700 1 -- 172.21.15.42:0/4206391819 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efea411f110 con 0x7efea411a640 2019-12-05T05:52:42.014 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:42.003+0000 7efea3fff700 1 --2- 172.21.15.42:0/4206391819 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efea411a640 0x7efea411aa70 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7efe8c0032b0 tx=0x7efe8c010e80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:52:42.014 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:42.003+0000 7efea3fff700 1 --2- 172.21.15.42:0/4206391819 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efea411a640 0x7efea411aa70 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7efe8c0032b0 tx=0x7efe8c010e80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:52:42.014 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:42.003+0000 7efea3fff700 1 --2- 172.21.15.42:0/4206391819 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efea411a640 0x7efea411aa70 secure :-1 s=READY pgs=358 cs=0 l=1 rx=0x7efe8c0032b0 tx=0x7efe8c010e80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:52:42.014 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:42.003+0000 7efea3fff700 1 --2- 172.21.15.42:0/4206391819 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efea411a640 0x7efea411aa70 secure :-1 s=THROTTLE_DONE pgs=358 cs=0 l=1 rx=0x7efe8c0032b0 tx=0x7efe8c010e80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:52:42.016 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:42.003+0000 7efea3fff700 1 --2- 172.21.15.42:0/4206391819 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efea411a640 0x7efea411aa70 secure :-1 s=THROTTLE_DONE pgs=358 cs=0 l=1 rx=0x7efe8c0032b0 tx=0x7efe8c010e80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:52:42.016 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:42.003+0000 7efea17fa700 1 -- 172.21.15.42:0/4206391819 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efe8c011bb0 con 0x7efea411a640 2019-12-05T05:52:42.016 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:42.003+0000 7efea3fff700 1 --2- 172.21.15.42:0/4206391819 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efea411a640 0x7efea411aa70 secure :-1 s=THROTTLE_DONE pgs=358 cs=0 l=1 rx=0x7efe8c0032b0 tx=0x7efe8c010e80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:52:42.017 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:42.003+0000 7efea17fa700 1 -- 172.21.15.42:0/4206391819 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe8c0110c0 con 0x7efea411a640 2019-12-05T05:52:42.017 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:42.003+0000 7efeaa84f700 1 -- 172.21.15.42:0/4206391819 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efea4075f00 con 0x7efea411a640 2019-12-05T05:52:42.017 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:42.003+0000 7efea17fa700 1 -- 172.21.15.42:0/4206391819 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efe8c005ab0 con 0x7efea411a640 2019-12-05T05:52:42.017 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:42.003+0000 7efeaa84f700 1 -- 172.21.15.42:0/4206391819 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efea4076480 con 0x7efea411a640 2019-12-05T05:52:42.017 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:42.003+0000 7efea3fff700 1 --2- 172.21.15.42:0/4206391819 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efea411a640 0x7efea411aa70 secure :-1 s=THROTTLE_DONE pgs=358 cs=0 l=1 rx=0x7efe8c0032b0 tx=0x7efe8c010e80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:52:42.017 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:42.003+0000 7efea17fa700 1 -- 172.21.15.42:0/4206391819 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7efe8c00f280 con 0x7efea411a640 2019-12-05T05:52:42.017 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:42.003+0000 7efea3fff700 1 --2- 172.21.15.42:0/4206391819 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efea411a640 0x7efea411aa70 secure :-1 s=THROTTLE_DONE pgs=358 cs=0 l=1 rx=0x7efe8c0032b0 tx=0x7efe8c010e80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:52:42.018 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:42.003+0000 7efea17fa700 1 --2- 172.21.15.42:0/4206391819 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7efe8803b470 0x7efe8803d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:52:42.018 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:42.007+0000 7efea17fa700 1 -- 172.21.15.42:0/4206391819 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8869+0+0 (secure 0 0 0) 0x7efe8c04b4c0 con 0x7efea411a640 2019-12-05T05:52:42.018 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:42.007+0000 7efea8dec700 1 --2- 172.21.15.42:0/4206391819 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7efe8803b470 0x7efe8803d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:52:42.018 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:42.007+0000 7efea8dec700 1 --2- 172.21.15.42:0/4206391819 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7efe8803b470 0x7efe8803d980 crc :-1 s=READY pgs=691 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:52:42.018 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:42.011+0000 7efeaa84f700 1 -- 172.21.15.42:0/4206391819 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efea412e3a0 con 0x7efea411a640 2019-12-05T05:52:42.027 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:42.019+0000 7efea3fff700 1 --2- 172.21.15.42:0/4206391819 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efea411a640 0x7efea411aa70 secure :-1 s=THROTTLE_DONE pgs=358 cs=0 l=1 rx=0x7efe8c0032b0 tx=0x7efe8c010e80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:52:42.027 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:42.019+0000 7efea17fa700 1 -- 172.21.15.42:0/4206391819 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7efe8c005dc0 con 0x7efea411a640 2019-12-05T05:52:42.164 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:52:42.169 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:52:42.197 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:42.191+0000 7efeaa84f700 1 -- 172.21.15.42:0/4206391819 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efea405d100 con 0x7efea411a640 2019-12-05T05:52:42.198 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:42.191+0000 7efea3fff700 1 --2- 172.21.15.42:0/4206391819 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efea411a640 0x7efea411aa70 secure :-1 s=THROTTLE_DONE pgs=358 cs=0 l=1 rx=0x7efe8c0032b0 tx=0x7efe8c010e80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:52:42.199 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:42.191+0000 7efea17fa700 1 -- 172.21.15.42:0/4206391819 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 198 v198) v1 ==== 95+0+2102 (secure 0 0 0) 0x7efe8c0327a0 con 0x7efea411a640 2019-12-05T05:52:42.199 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 198 2019-12-05T05:52:42.199 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:52:42.199 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":198,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6356,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/1523046776","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1523046776},{"type":"v1","addr":"172.21.15.40:6833","nonce":1523046776}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":198}],"filesystems":[{"mdsmap":{"epoch":197,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:51:24.172910+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8430},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8430":{"gid":8430,"name":"a","rank":0,"incarnation":194,"state":"up:active","state_seq":9,"addr":"172.21.15.42:6835/5991897","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":5991897},{"type":"v1","addr":"172.21.15.42:6835","nonce":5991897}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:52:42.201 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:42.195+0000 7efeaa84f700 1 -- 172.21.15.42:0/4206391819 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7efe8803b470 msgr2=0x7efe8803d980 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:52:42.201 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:42.195+0000 7efeaa84f700 1 --2- 172.21.15.42:0/4206391819 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7efe8803b470 0x7efe8803d980 crc :-1 s=READY pgs=691 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:52:42.202 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:42.195+0000 7efeaa84f700 1 -- 172.21.15.42:0/4206391819 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efea411a640 msgr2=0x7efea411aa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:52:42.202 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:42.195+0000 7efeaa84f700 1 --2- 172.21.15.42:0/4206391819 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efea411a640 0x7efea411aa70 secure :-1 s=READY pgs=358 cs=0 l=1 rx=0x7efe8c0032b0 tx=0x7efe8c010e80).stop 2019-12-05T05:52:42.202 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:42.195+0000 7efeaa84f700 1 -- 172.21.15.42:0/4206391819 shutdown_connections 2019-12-05T05:52:42.203 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:42.195+0000 7efeaa84f700 1 --2- 172.21.15.42:0/4206391819 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7efe8803b470 0x7efe8803d980 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:52:42.203 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:42.195+0000 7efeaa84f700 1 --2- 172.21.15.42:0/4206391819 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efea411a640 0x7efea411aa70 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:52:42.203 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:42.195+0000 7efeaa84f700 1 -- 172.21.15.42:0/4206391819 shutdown_connections 2019-12-05T05:52:42.204 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:42.195+0000 7efeaa84f700 1 -- 172.21.15.42:0/4206391819 wait complete. 2019-12-05T05:52:42.204 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:42.195+0000 7efeaa84f700 1 -- 172.21.15.42:0/4206391819 >> 172.21.15.42:0/4206391819 conn(0x7efea408fdd0 msgr2=0x7efea41416c0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:52:42.212 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T05:52:44.213 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:52:44.317 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:44.307+0000 7fa57dafd700 1 Processor -- start 2019-12-05T05:52:44.318 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:44.307+0000 7fa57dafd700 1 -- start start 2019-12-05T05:52:44.318 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:44.307+0000 7fa57dafd700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fa57808df70 0x7fa57809f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:52:44.318 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:44.307+0000 7fa57dafd700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa57809fc50 con 0x7fa57809f820 2019-12-05T05:52:44.318 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:44.307+0000 7fa57dafd700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5780a0700 con 0x7fa5780a7730 2019-12-05T05:52:44.318 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:44.307+0000 7fa57dafd700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5780a0af0 con 0x7fa57808df70 2019-12-05T05:52:44.319 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:44.307+0000 7fa5777fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fa57808df70 0x7fa57809f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:52:44.319 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:44.307+0000 7fa5777fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fa57808df70 0x7fa57809f2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:43168/0 (socket says 172.21.15.42:43168) 2019-12-05T05:52:44.319 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:44.307+0000 7fa5777fe700 1 -- 172.21.15.42:0/2486590332 learned_addr learned my addr 172.21.15.42:0/2486590332 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:52:44.319 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:44.307+0000 7fa5777fe700 1 -- 172.21.15.42:0/2486590332 >> v1:172.21.15.42:6789/0 conn(0x7fa5780a7730 legacy=0x7fa57807e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:52:44.319 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:44.307+0000 7fa5777fe700 1 -- 172.21.15.42:0/2486590332 >> v1:172.21.15.40:6789/0 conn(0x7fa57809f820 legacy=0x7fa57814efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:52:44.319 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:44.307+0000 7fa5777fe700 1 -- 172.21.15.42:0/2486590332 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa57807ef10 con 0x7fa57808df70 2019-12-05T05:52:44.319 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:44.307+0000 7fa5777fe700 1 --2- 172.21.15.42:0/2486590332 >> v2:172.21.15.42:3300/0 conn(0x7fa57808df70 0x7fa57809f2d0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fa56c008790 tx=0x7fa56c00fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:52:44.320 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:44.307+0000 7fa5777fe700 1 --2- 172.21.15.42:0/2486590332 >> v2:172.21.15.42:3300/0 conn(0x7fa57808df70 0x7fa57809f2d0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fa56c008790 tx=0x7fa56c00fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:52:44.320 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:44.311+0000 7fa5777fe700 1 --2- 172.21.15.42:0/2486590332 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa57808df70 0x7fa57809f2d0 secure :-1 s=READY pgs=1613 cs=0 l=1 rx=0x7fa56c008790 tx=0x7fa56c00fcb0).ready entity=mon.1 client_cookie=ba36f7cd74b4961a server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:52:44.320 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:44.311+0000 7fa5777fe700 1 --2- 172.21.15.42:0/2486590332 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa57808df70 0x7fa57809f2d0 secure :-1 s=THROTTLE_DONE pgs=1613 cs=0 l=1 rx=0x7fa56c008790 tx=0x7fa56c00fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:52:44.320 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:44.311+0000 7fa5777fe700 1 --2- 172.21.15.42:0/2486590332 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa57808df70 0x7fa57809f2d0 secure :-1 s=THROTTLE_DONE pgs=1613 cs=0 l=1 rx=0x7fa56c008790 tx=0x7fa56c00fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:52:44.320 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:44.311+0000 7fa576ffd700 1 -- 172.21.15.42:0/2486590332 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa56c011520 con 0x7fa57808df70 2019-12-05T05:52:44.320 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:44.311+0000 7fa5777fe700 1 --2- 172.21.15.42:0/2486590332 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa57808df70 0x7fa57809f2d0 secure :-1 s=THROTTLE_DONE pgs=1613 cs=0 l=1 rx=0x7fa56c008790 tx=0x7fa56c00fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:52:44.320 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:44.311+0000 7fa576ffd700 1 -- 172.21.15.42:0/2486590332 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa56c008ca0 con 0x7fa57808df70 2019-12-05T05:52:44.321 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:44.311+0000 7fa576ffd700 1 -- 172.21.15.42:0/2486590332 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa56c011b00 con 0x7fa57808df70 2019-12-05T05:52:44.321 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:44.311+0000 7fa57dafd700 1 -- 172.21.15.42:0/2486590332 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa57808df70 msgr2=0x7fa57809f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:52:44.321 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:44.311+0000 7fa57dafd700 1 --2- 172.21.15.42:0/2486590332 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa57808df70 0x7fa57809f2d0 secure :-1 s=READY pgs=1613 cs=0 l=1 rx=0x7fa56c008790 tx=0x7fa56c00fcb0).stop 2019-12-05T05:52:44.321 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:44.311+0000 7fa57dafd700 1 -- 172.21.15.42:0/2486590332 shutdown_connections 2019-12-05T05:52:44.321 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:44.311+0000 7fa57dafd700 1 --2- 172.21.15.42:0/2486590332 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa57808df70 0x7fa57809f2d0 unknown :-1 s=CLOSED pgs=1613 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:52:44.321 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:44.311+0000 7fa57dafd700 1 -- 172.21.15.42:0/2486590332 shutdown_connections 2019-12-05T05:52:44.322 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:44.311+0000 7fa57dafd700 1 -- 172.21.15.42:0/2486590332 wait complete. 2019-12-05T05:52:44.322 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:44.311+0000 7fa57dafd700 1 -- 172.21.15.42:0/2486590332 >> 172.21.15.42:0/2486590332 conn(0x7fa5780aabd0 msgr2=0x7fa5780a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:52:44.324 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:44.315+0000 7fa57dafd700 1 Processor -- start 2019-12-05T05:52:44.325 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:44.315+0000 7fa57dafd700 1 -- start start 2019-12-05T05:52:44.325 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:44.315+0000 7fa57dafd700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa5780a4830 0x7fa57807c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:52:44.325 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:44.315+0000 7fa57dafd700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa5780a9900 0x7fa5780a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:52:44.325 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:44.315+0000 7fa57dafd700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5780a66d0 con 0x7fa578142a20 2019-12-05T05:52:44.325 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:44.315+0000 7fa57dafd700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5780a9d30 con 0x7fa5780a4830 2019-12-05T05:52:44.325 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:44.315+0000 7fa57dafd700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa578143470 con 0x7fa5780a9900 2019-12-05T05:52:44.325 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:44.315+0000 7fa577fff700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa5780a4830 0x7fa57807c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:52:44.326 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:44.315+0000 7fa5777fe700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa5780a9900 0x7fa5780a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:52:44.326 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:44.315+0000 7fa577fff700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa5780a4830 0x7fa57807c910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.42:59220/0 (socket says 172.21.15.42:59220) 2019-12-05T05:52:44.326 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:44.315+0000 7fa577fff700 1 -- 172.21.15.42:0/3578257735 learned_addr learned my addr 172.21.15.42:0/3578257735 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:52:44.326 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:44.315+0000 7fa5777fe700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa5780a9900 0x7fa5780a6180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:43640/0 (socket says 172.21.15.42:43640) 2019-12-05T05:52:44.326 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:44.315+0000 7fa5777fe700 1 -- 172.21.15.42:0/3578257735 >> v2:172.21.15.40:3300/0 conn(0x7fa5780a4830 msgr2=0x7fa57807c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:52:44.326 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:44.315+0000 7fa5777fe700 1 --2- 172.21.15.42:0/3578257735 >> v2:172.21.15.40:3300/0 conn(0x7fa5780a4830 0x7fa57807c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:52:44.326 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:44.315+0000 7fa5777fe700 1 -- 172.21.15.42:0/3578257735 >> v1:172.21.15.40:6789/0 conn(0x7fa578142a20 legacy=0x7fa578144af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:52:44.327 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:44.315+0000 7fa5777fe700 1 -- 172.21.15.42:0/3578257735 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5781452f0 con 0x7fa5780a9900 2019-12-05T05:52:44.327 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:44.315+0000 7fa5777fe700 1 --2- 172.21.15.42:0/3578257735 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa5780a9900 0x7fa5780a6180 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fa56c008790 tx=0x7fa56c0115e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:52:44.327 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:44.315+0000 7fa5777fe700 1 --2- 172.21.15.42:0/3578257735 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa5780a9900 0x7fa5780a6180 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fa56c008790 tx=0x7fa56c0115e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:52:44.327 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:44.315+0000 7fa5777fe700 1 --2- 172.21.15.42:0/3578257735 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa5780a9900 0x7fa5780a6180 secure :-1 s=READY pgs=359 cs=0 l=1 rx=0x7fa56c008790 tx=0x7fa56c0115e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:52:44.327 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:44.319+0000 7fa5777fe700 1 --2- 172.21.15.42:0/3578257735 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa5780a9900 0x7fa5780a6180 secure :-1 s=THROTTLE_DONE pgs=359 cs=0 l=1 rx=0x7fa56c008790 tx=0x7fa56c0115e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:52:44.327 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:44.319+0000 7fa574ff9700 1 -- 172.21.15.42:0/3578257735 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa56c007150 con 0x7fa5780a9900 2019-12-05T05:52:44.327 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:44.319+0000 7fa5777fe700 1 --2- 172.21.15.42:0/3578257735 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa5780a9900 0x7fa5780a6180 secure :-1 s=THROTTLE_DONE pgs=359 cs=0 l=1 rx=0x7fa56c008790 tx=0x7fa56c0115e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:52:44.328 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:44.319+0000 7fa5777fe700 1 --2- 172.21.15.42:0/3578257735 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa5780a9900 0x7fa5780a6180 secure :-1 s=THROTTLE_DONE pgs=359 cs=0 l=1 rx=0x7fa56c008790 tx=0x7fa56c0115e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:52:44.328 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:44.319+0000 7fa574ff9700 1 -- 172.21.15.42:0/3578257735 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa56c011720 con 0x7fa5780a9900 2019-12-05T05:52:44.328 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:44.319+0000 7fa57dafd700 1 -- 172.21.15.42:0/3578257735 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa578142e50 con 0x7fa5780a9900 2019-12-05T05:52:44.328 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:44.319+0000 7fa574ff9700 1 -- 172.21.15.42:0/3578257735 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa56c005730 con 0x7fa5780a9900 2019-12-05T05:52:44.331 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:44.319+0000 7fa57dafd700 1 -- 172.21.15.42:0/3578257735 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa57812dd80 con 0x7fa5780a9900 2019-12-05T05:52:44.331 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:44.319+0000 7fa5777fe700 1 --2- 172.21.15.42:0/3578257735 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa5780a9900 0x7fa5780a6180 secure :-1 s=THROTTLE_DONE pgs=359 cs=0 l=1 rx=0x7fa56c008790 tx=0x7fa56c0115e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:52:44.333 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:44.319+0000 7fa574ff9700 1 -- 172.21.15.42:0/3578257735 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fa56c0059d0 con 0x7fa5780a9900 2019-12-05T05:52:44.333 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:44.319+0000 7fa5777fe700 1 --2- 172.21.15.42:0/3578257735 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa5780a9900 0x7fa5780a6180 secure :-1 s=THROTTLE_DONE pgs=359 cs=0 l=1 rx=0x7fa56c008790 tx=0x7fa56c0115e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:52:44.333 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:44.319+0000 7fa574ff9700 1 --2- 172.21.15.42:0/3578257735 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa55803acd0 0x7fa55803d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:52:44.333 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:44.319+0000 7fa574ff9700 1 -- 172.21.15.42:0/3578257735 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8869+0+0 (secure 0 0 0) 0x7fa56c04b870 con 0x7fa5780a9900 2019-12-05T05:52:44.333 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:44.319+0000 7fa577fff700 1 --2- 172.21.15.42:0/3578257735 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa55803acd0 0x7fa55803d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:52:44.333 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:44.319+0000 7fa577fff700 1 --2- 172.21.15.42:0/3578257735 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa55803acd0 0x7fa55803d1e0 crc :-1 s=READY pgs=692 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:52:44.334 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:44.319+0000 7fa57dafd700 1 -- 172.21.15.42:0/3578257735 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa57805d100 con 0x7fa5780a9900 2019-12-05T05:52:44.335 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:44.327+0000 7fa5777fe700 1 --2- 172.21.15.42:0/3578257735 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa5780a9900 0x7fa5780a6180 secure :-1 s=THROTTLE_DONE pgs=359 cs=0 l=1 rx=0x7fa56c008790 tx=0x7fa56c0115e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:52:44.335 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:44.331+0000 7fa574ff9700 1 -- 172.21.15.42:0/3578257735 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fa56c0328d0 con 0x7fa5780a9900 2019-12-05T05:52:44.501 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:44.495+0000 7fa57dafd700 1 -- 172.21.15.42:0/3578257735 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa57805b8b0 con 0x7fa5780a9900 2019-12-05T05:52:44.502 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:44.495+0000 7fa5777fe700 1 --2- 172.21.15.42:0/3578257735 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa5780a9900 0x7fa5780a6180 secure :-1 s=THROTTLE_DONE pgs=359 cs=0 l=1 rx=0x7fa56c008790 tx=0x7fa56c0115e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:52:44.503 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:44.495+0000 7fa574ff9700 1 -- 172.21.15.42:0/3578257735 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 198 v198) v1 ==== 95+0+2102 (secure 0 0 0) 0x7fa56c032b30 con 0x7fa5780a9900 2019-12-05T05:52:44.505 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 198 2019-12-05T05:52:44.505 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:52:44.505 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":198,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6356,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/1523046776","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1523046776},{"type":"v1","addr":"172.21.15.40:6833","nonce":1523046776}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":198}],"filesystems":[{"mdsmap":{"epoch":197,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:51:24.172910+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8430},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8430":{"gid":8430,"name":"a","rank":0,"incarnation":194,"state":"up:active","state_seq":9,"addr":"172.21.15.42:6835/5991897","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":5991897},{"type":"v1","addr":"172.21.15.42:6835","nonce":5991897}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:52:44.508 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:44.499+0000 7fa57dafd700 1 -- 172.21.15.42:0/3578257735 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa55803acd0 msgr2=0x7fa55803d1e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:52:44.508 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:44.503+0000 7fa57dafd700 1 --2- 172.21.15.42:0/3578257735 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa55803acd0 0x7fa55803d1e0 crc :-1 s=READY pgs=692 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:52:44.508 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:44.503+0000 7fa57dafd700 1 -- 172.21.15.42:0/3578257735 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa5780a9900 msgr2=0x7fa5780a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:52:44.508 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:44.503+0000 7fa57dafd700 1 --2- 172.21.15.42:0/3578257735 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa5780a9900 0x7fa5780a6180 secure :-1 s=READY pgs=359 cs=0 l=1 rx=0x7fa56c008790 tx=0x7fa56c0115e0).stop 2019-12-05T05:52:44.508 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:44.503+0000 7fa57dafd700 1 -- 172.21.15.42:0/3578257735 shutdown_connections 2019-12-05T05:52:44.508 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:44.503+0000 7fa57dafd700 1 --2- 172.21.15.42:0/3578257735 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa55803acd0 0x7fa55803d1e0 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:52:44.509 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:44.503+0000 7fa57dafd700 1 --2- 172.21.15.42:0/3578257735 >> v2:172.21.15.40:3300/0 conn(0x7fa5780a4830 0x7fa57807c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:52:44.509 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:44.503+0000 7fa57dafd700 1 --2- 172.21.15.42:0/3578257735 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa5780a9900 0x7fa5780a6180 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:52:44.509 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:44.503+0000 7fa57dafd700 1 -- 172.21.15.42:0/3578257735 shutdown_connections 2019-12-05T05:52:44.509 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:44.503+0000 7fa57dafd700 1 -- 172.21.15.42:0/3578257735 wait complete. 2019-12-05T05:52:44.509 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:44.503+0000 7fa57dafd700 1 -- 172.21.15.42:0/3578257735 >> 172.21.15.42:0/3578257735 conn(0x7fa57805a180 msgr2=0x7fa57808eb60 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:52:44.517 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T05:52:46.519 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:52:46.624 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:46.611+0000 7ff93c79f700 1 Processor -- start 2019-12-05T05:52:46.625 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:46.611+0000 7ff93c79f700 1 -- start start 2019-12-05T05:52:46.625 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:46.615+0000 7ff93c79f700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff93408df80 0x7ff93409f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:52:46.625 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:46.615+0000 7ff93c79f700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff93409fc60 con 0x7ff9340a7740 2019-12-05T05:52:46.625 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:46.615+0000 7ff93c79f700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9340a0710 con 0x7ff93409f830 2019-12-05T05:52:46.625 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:46.615+0000 7ff93c79f700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9340a0b00 con 0x7ff93408df80 2019-12-05T05:52:46.625 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:46.615+0000 7ff93a53b700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff93408df80 0x7ff93409f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:52:46.626 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:46.615+0000 7ff93b53d700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7ff93409f830 0x7ff93414efe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:43232/0 (socket says 172.21.15.42:43232) 2019-12-05T05:52:46.626 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:46.615+0000 7ff93b53d700 1 -- 172.21.15.42:0/118690028 learned_addr learned my addr 172.21.15.42:0/118690028 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:52:46.626 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:46.615+0000 7ff93ad3c700 1 --1- 172.21.15.42:0/118690028 >> v1:172.21.15.40:6789/0 conn(0x7ff9340a7740 0x7ff93407e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:52668/0 (socket says 172.21.15.42:52668) 2019-12-05T05:52:46.626 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:46.615+0000 7ff93a53b700 1 -- 172.21.15.42:0/118690028 >> v1:172.21.15.42:6789/0 conn(0x7ff93409f830 legacy=0x7ff93414efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:52:46.626 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:46.615+0000 7ff93a53b700 1 -- 172.21.15.42:0/118690028 >> v1:172.21.15.40:6789/0 conn(0x7ff9340a7740 legacy=0x7ff93407e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:52:46.626 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:46.615+0000 7ff93a53b700 1 -- 172.21.15.42:0/118690028 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff93407ef20 con 0x7ff93408df80 2019-12-05T05:52:46.626 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:46.615+0000 7ff93a53b700 1 --2- 172.21.15.42:0/118690028 >> v2:172.21.15.42:3300/0 conn(0x7ff93408df80 0x7ff93409f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7ff924008790 tx=0x7ff92400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:52:46.627 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:46.615+0000 7ff93a53b700 1 --2- 172.21.15.42:0/118690028 >> v2:172.21.15.42:3300/0 conn(0x7ff93408df80 0x7ff93409f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7ff924008790 tx=0x7ff92400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:52:46.627 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:46.615+0000 7ff93a53b700 1 --2- 172.21.15.42:0/118690028 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff93408df80 0x7ff93409f2e0 secure :-1 s=READY pgs=1615 cs=0 l=1 rx=0x7ff924008790 tx=0x7ff92400fbf0).ready entity=mon.1 client_cookie=493759777cc954be server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:52:46.627 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:46.615+0000 7ff93a53b700 1 --2- 172.21.15.42:0/118690028 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff93408df80 0x7ff93409f2e0 secure :-1 s=THROTTLE_DONE pgs=1615 cs=0 l=1 rx=0x7ff924008790 tx=0x7ff92400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:52:46.627 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:46.615+0000 7ff939d3a700 1 -- 172.21.15.42:0/118690028 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff924011270 con 0x7ff93408df80 2019-12-05T05:52:46.627 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:46.615+0000 7ff93a53b700 1 --2- 172.21.15.42:0/118690028 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff93408df80 0x7ff93409f2e0 secure :-1 s=THROTTLE_DONE pgs=1615 cs=0 l=1 rx=0x7ff924008790 tx=0x7ff92400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:52:46.627 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:46.615+0000 7ff93a53b700 1 --2- 172.21.15.42:0/118690028 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff93408df80 0x7ff93409f2e0 secure :-1 s=THROTTLE_DONE pgs=1615 cs=0 l=1 rx=0x7ff924008790 tx=0x7ff92400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:52:46.627 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:46.615+0000 7ff939d3a700 1 -- 172.21.15.42:0/118690028 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff924008bc0 con 0x7ff93408df80 2019-12-05T05:52:46.628 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:46.615+0000 7ff939d3a700 1 -- 172.21.15.42:0/118690028 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff924011850 con 0x7ff93408df80 2019-12-05T05:52:46.628 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:46.615+0000 7ff93c79f700 1 -- 172.21.15.42:0/118690028 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff93408df80 msgr2=0x7ff93409f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:52:46.628 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:46.615+0000 7ff93c79f700 1 --2- 172.21.15.42:0/118690028 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff93408df80 0x7ff93409f2e0 secure :-1 s=READY pgs=1615 cs=0 l=1 rx=0x7ff924008790 tx=0x7ff92400fbf0).stop 2019-12-05T05:52:46.628 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:46.615+0000 7ff93c79f700 1 -- 172.21.15.42:0/118690028 shutdown_connections 2019-12-05T05:52:46.628 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:46.615+0000 7ff93c79f700 1 --2- 172.21.15.42:0/118690028 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff93408df80 0x7ff93409f2e0 unknown :-1 s=CLOSED pgs=1615 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:52:46.628 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:46.615+0000 7ff93c79f700 1 -- 172.21.15.42:0/118690028 shutdown_connections 2019-12-05T05:52:46.628 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:46.615+0000 7ff93c79f700 1 -- 172.21.15.42:0/118690028 wait complete. 2019-12-05T05:52:46.629 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:46.615+0000 7ff93c79f700 1 -- 172.21.15.42:0/118690028 >> 172.21.15.42:0/118690028 conn(0x7ff9340aabe0 msgr2=0x7ff9340a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:52:46.629 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:46.619+0000 7ff93c79f700 1 Processor -- start 2019-12-05T05:52:46.629 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:46.619+0000 7ff93c79f700 1 -- start start 2019-12-05T05:52:46.632 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:46.619+0000 7ff93c79f700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff93414fbd0 0x7ff9341268a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:52:46.632 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:46.619+0000 7ff93c79f700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff9340a40c0 0x7ff934126df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:52:46.632 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:46.619+0000 7ff93c79f700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff934127340 con 0x7ff9340794f0 2019-12-05T05:52:46.632 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:46.619+0000 7ff93c79f700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff934079a20 con 0x7ff9340a40c0 2019-12-05T05:52:46.632 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:46.619+0000 7ff93c79f700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff934079cc0 con 0x7ff93414fbd0 2019-12-05T05:52:46.632 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:46.619+0000 7ff93a53b700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff93414fbd0 0x7ff9341268a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:52:46.633 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:46.619+0000 7ff93a53b700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff93414fbd0 0x7ff9341268a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:43198/0 (socket says 172.21.15.42:43198) 2019-12-05T05:52:46.633 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:46.619+0000 7ff93a53b700 1 -- 172.21.15.42:0/2726881199 learned_addr learned my addr 172.21.15.42:0/2726881199 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:52:46.633 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:46.619+0000 7ff93b53d700 1 --2- 172.21.15.42:0/2726881199 >> v2:172.21.15.40:3300/0 conn(0x7ff9340a40c0 0x7ff934126df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:52:46.633 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:46.619+0000 7ff93a53b700 1 -- 172.21.15.42:0/2726881199 >> v2:172.21.15.40:3300/0 conn(0x7ff9340a40c0 msgr2=0x7ff934126df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:52:46.633 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:46.619+0000 7ff93a53b700 1 --2- 172.21.15.42:0/2726881199 >> v2:172.21.15.40:3300/0 conn(0x7ff9340a40c0 0x7ff934126df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:52:46.633 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:46.619+0000 7ff93a53b700 1 -- 172.21.15.42:0/2726881199 >> v1:172.21.15.40:6789/0 conn(0x7ff9340794f0 legacy=0x7ff93414f3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:52:46.633 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:46.619+0000 7ff93a53b700 1 -- 172.21.15.42:0/2726881199 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff93407b6b0 con 0x7ff93414fbd0 2019-12-05T05:52:46.634 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:46.619+0000 7ff93b53d700 1 --2- 172.21.15.42:0/2726881199 >> v2:172.21.15.40:3300/0 conn(0x7ff9340a40c0 0x7ff934126df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).send_auth_request state changed! 2019-12-05T05:52:46.634 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:46.619+0000 7ff93a53b700 1 --2- 172.21.15.42:0/2726881199 >> v2:172.21.15.42:3300/0 conn(0x7ff93414fbd0 0x7ff9341268a0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7ff924008790 tx=0x7ff924011dd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:52:46.634 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:46.623+0000 7ff93a53b700 1 --2- 172.21.15.42:0/2726881199 >> v2:172.21.15.42:3300/0 conn(0x7ff93414fbd0 0x7ff9341268a0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7ff924008790 tx=0x7ff924011dd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:52:46.634 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:46.623+0000 7ff93a53b700 1 --2- 172.21.15.42:0/2726881199 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff93414fbd0 0x7ff9341268a0 secure :-1 s=READY pgs=1616 cs=0 l=1 rx=0x7ff924008790 tx=0x7ff924011dd0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:52:46.634 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:46.623+0000 7ff93a53b700 1 --2- 172.21.15.42:0/2726881199 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff93414fbd0 0x7ff9341268a0 secure :-1 s=THROTTLE_DONE pgs=1616 cs=0 l=1 rx=0x7ff924008790 tx=0x7ff924011dd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:52:46.634 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:46.623+0000 7ff92b7fe700 1 -- 172.21.15.42:0/2726881199 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff924007150 con 0x7ff93414fbd0 2019-12-05T05:52:46.634 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:46.623+0000 7ff93a53b700 1 --2- 172.21.15.42:0/2726881199 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff93414fbd0 0x7ff9341268a0 secure :-1 s=THROTTLE_DONE pgs=1616 cs=0 l=1 rx=0x7ff924008790 tx=0x7ff924011dd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:52:46.635 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:46.623+0000 7ff93a53b700 1 --2- 172.21.15.42:0/2726881199 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff93414fbd0 0x7ff9341268a0 secure :-1 s=THROTTLE_DONE pgs=1616 cs=0 l=1 rx=0x7ff924008790 tx=0x7ff924011dd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:52:46.635 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:46.623+0000 7ff92b7fe700 1 -- 172.21.15.42:0/2726881199 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff924011470 con 0x7ff93414fbd0 2019-12-05T05:52:46.635 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:46.623+0000 7ff93c79f700 1 -- 172.21.15.42:0/2726881199 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff934115540 con 0x7ff93414fbd0 2019-12-05T05:52:46.635 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:46.623+0000 7ff92b7fe700 1 -- 172.21.15.42:0/2726881199 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff924005730 con 0x7ff93414fbd0 2019-12-05T05:52:46.635 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:46.623+0000 7ff93c79f700 1 -- 172.21.15.42:0/2726881199 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff93412dc50 con 0x7ff93414fbd0 2019-12-05T05:52:46.635 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:46.623+0000 7ff93a53b700 1 --2- 172.21.15.42:0/2726881199 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff93414fbd0 0x7ff9341268a0 secure :-1 s=THROTTLE_DONE pgs=1616 cs=0 l=1 rx=0x7ff924008790 tx=0x7ff924011dd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:52:46.635 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:46.623+0000 7ff92b7fe700 1 -- 172.21.15.42:0/2726881199 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7ff9240059d0 con 0x7ff93414fbd0 2019-12-05T05:52:46.635 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:46.623+0000 7ff93a53b700 1 --2- 172.21.15.42:0/2726881199 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff93414fbd0 0x7ff9341268a0 secure :-1 s=THROTTLE_DONE pgs=1616 cs=0 l=1 rx=0x7ff924008790 tx=0x7ff924011dd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:52:46.636 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:46.623+0000 7ff92b7fe700 1 --2- 172.21.15.42:0/2726881199 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff91c03aca0 0x7ff91c03d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:52:46.636 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:46.623+0000 7ff92b7fe700 1 -- 172.21.15.42:0/2726881199 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8869+0+0 (secure 0 0 0) 0x7ff92404be30 con 0x7ff93414fbd0 2019-12-05T05:52:46.636 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:46.623+0000 7ff93ad3c700 1 --2- 172.21.15.42:0/2726881199 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff91c03aca0 0x7ff91c03d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:52:46.636 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:46.623+0000 7ff93ad3c700 1 --2- 172.21.15.42:0/2726881199 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff91c03aca0 0x7ff91c03d1b0 crc :-1 s=READY pgs=693 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:52:46.637 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:46.627+0000 7ff93c79f700 1 -- 172.21.15.42:0/2726881199 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff93405d110 con 0x7ff93414fbd0 2019-12-05T05:52:46.643 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:46.635+0000 7ff93a53b700 1 --2- 172.21.15.42:0/2726881199 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff93414fbd0 0x7ff9341268a0 secure :-1 s=THROTTLE_DONE pgs=1616 cs=0 l=1 rx=0x7ff924008790 tx=0x7ff924011dd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:52:46.643 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:46.635+0000 7ff92b7fe700 1 -- 172.21.15.42:0/2726881199 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7ff924030b30 con 0x7ff93414fbd0 2019-12-05T05:52:46.811 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:46.803+0000 7ff93c79f700 1 -- 172.21.15.42:0/2726881199 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff93407eba0 con 0x7ff93414fbd0 2019-12-05T05:52:46.812 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:46.807+0000 7ff93a53b700 1 --2- 172.21.15.42:0/2726881199 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff93414fbd0 0x7ff9341268a0 secure :-1 s=THROTTLE_DONE pgs=1616 cs=0 l=1 rx=0x7ff924008790 tx=0x7ff924011dd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:52:46.812 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:46.807+0000 7ff92b7fe700 1 -- 172.21.15.42:0/2726881199 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 198 v198) v1 ==== 95+0+2102 (secure 0 0 0) 0x7ff924014d50 con 0x7ff93414fbd0 2019-12-05T05:52:46.814 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:52:46.814 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":198,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6356,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/1523046776","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1523046776},{"type":"v1","addr":"172.21.15.40:6833","nonce":1523046776}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":198}],"filesystems":[{"mdsmap":{"epoch":197,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:51:24.172910+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8430},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8430":{"gid":8430,"name":"a","rank":0,"incarnation":194,"state":"up:active","state_seq":9,"addr":"172.21.15.42:6835/5991897","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":5991897},{"type":"v1","addr":"172.21.15.42:6835","nonce":5991897}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:52:46.815 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 198 2019-12-05T05:52:46.816 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:46.811+0000 7ff93c79f700 1 -- 172.21.15.42:0/2726881199 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff91c03aca0 msgr2=0x7ff91c03d1b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:52:46.817 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:46.811+0000 7ff93c79f700 1 --2- 172.21.15.42:0/2726881199 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff91c03aca0 0x7ff91c03d1b0 crc :-1 s=READY pgs=693 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:52:46.817 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:46.811+0000 7ff93c79f700 1 -- 172.21.15.42:0/2726881199 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff93414fbd0 msgr2=0x7ff9341268a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:52:46.817 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:46.811+0000 7ff93c79f700 1 --2- 172.21.15.42:0/2726881199 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff93414fbd0 0x7ff9341268a0 secure :-1 s=READY pgs=1616 cs=0 l=1 rx=0x7ff924008790 tx=0x7ff924011dd0).stop 2019-12-05T05:52:46.817 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:46.811+0000 7ff93c79f700 1 -- 172.21.15.42:0/2726881199 shutdown_connections 2019-12-05T05:52:46.817 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:46.811+0000 7ff93c79f700 1 --2- 172.21.15.42:0/2726881199 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff91c03aca0 0x7ff91c03d1b0 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:52:46.817 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:46.811+0000 7ff93c79f700 1 --2- 172.21.15.42:0/2726881199 >> v2:172.21.15.40:3300/0 conn(0x7ff9340a40c0 0x7ff934126df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:52:46.818 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:46.811+0000 7ff93c79f700 1 --2- 172.21.15.42:0/2726881199 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff93414fbd0 0x7ff9341268a0 unknown :-1 s=CLOSED pgs=1616 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:52:46.818 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:46.811+0000 7ff93c79f700 1 -- 172.21.15.42:0/2726881199 shutdown_connections 2019-12-05T05:52:46.818 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:46.811+0000 7ff93c79f700 1 -- 172.21.15.42:0/2726881199 wait complete. 2019-12-05T05:52:46.818 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:46.811+0000 7ff93c79f700 1 -- 172.21.15.42:0/2726881199 >> 172.21.15.42:0/2726881199 conn(0x7ff9340a82a0 msgr2=0x7ff934111db0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:52:46.830 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T05:52:48.831 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:52:48.950 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:48.935+0000 7f142903a700 1 Processor -- start 2019-12-05T05:52:48.950 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:48.935+0000 7f142903a700 1 -- start start 2019-12-05T05:52:48.950 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:48.939+0000 7f142903a700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f14240a7740 0x7f142408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:52:48.950 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:48.939+0000 7f142903a700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f142409fae0 0x7f142407eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:52:48.950 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:48.939+0000 7f142903a700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14240a0710 con 0x7f142407e720 2019-12-05T05:52:48.950 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:48.939+0000 7f142903a700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14240a0b00 con 0x7f14240a7740 2019-12-05T05:52:48.951 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:48.939+0000 7f142903a700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f142408e5b0 con 0x7f142409fae0 2019-12-05T05:52:48.951 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:48.939+0000 7f1422d9d700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f142407e720 0x7f142409f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:43244/0 (socket says 172.21.15.42:43244) 2019-12-05T05:52:48.951 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:48.939+0000 7f1422d9d700 1 -- 172.21.15.42:0/4234999419 learned_addr learned my addr 172.21.15.42:0/4234999419 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:52:48.951 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:48.939+0000 7f1428839700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f142409fae0 0x7f142407eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:52:48.951 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:48.939+0000 7f142359e700 1 --2- 172.21.15.42:0/4234999419 >> v2:172.21.15.40:3300/0 conn(0x7f14240a7740 0x7f142408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:52:48.951 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:48.939+0000 7f1428839700 1 -- 172.21.15.42:0/4234999419 >> v2:172.21.15.40:3300/0 conn(0x7f14240a7740 msgr2=0x7f142408df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:52:48.951 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:48.939+0000 7f1428839700 1 --2- 172.21.15.42:0/4234999419 >> v2:172.21.15.40:3300/0 conn(0x7f14240a7740 0x7f142408df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:52:48.952 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:48.939+0000 7f1428839700 1 -- 172.21.15.42:0/4234999419 >> v1:172.21.15.42:6789/0 conn(0x7f142407e720 legacy=0x7f142409f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:52:48.952 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:48.939+0000 7f1428839700 1 -- 172.21.15.42:0/4234999419 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f142407f0a0 con 0x7f142409fae0 2019-12-05T05:52:48.952 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:48.939+0000 7f142359e700 1 --2- 172.21.15.42:0/4234999419 >> v2:172.21.15.40:3300/0 conn(0x7f14240a7740 0x7f142408df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:52:48.952 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:48.939+0000 7f1428839700 1 --2- 172.21.15.42:0/4234999419 >> v2:172.21.15.42:3300/0 conn(0x7f142409fae0 0x7f142407eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f141c0032b0 tx=0x7f141c00fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:52:48.970 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:48.939+0000 7f1428839700 1 --2- 172.21.15.42:0/4234999419 >> v2:172.21.15.42:3300/0 conn(0x7f142409fae0 0x7f142407eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f141c0032b0 tx=0x7f141c00fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:52:48.970 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:48.939+0000 7f1428839700 1 --2- 172.21.15.42:0/4234999419 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f142409fae0 0x7f142407eb50 secure :-1 s=READY pgs=1618 cs=0 l=1 rx=0x7f141c0032b0 tx=0x7f141c00fff0).ready entity=mon.1 client_cookie=ecc2c09018d91e9e server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:52:48.970 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:48.939+0000 7f1428839700 1 --2- 172.21.15.42:0/4234999419 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f142409fae0 0x7f142407eb50 secure :-1 s=THROTTLE_DONE pgs=1618 cs=0 l=1 rx=0x7f141c0032b0 tx=0x7f141c00fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:52:48.970 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:48.939+0000 7f1428839700 1 --2- 172.21.15.42:0/4234999419 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f142409fae0 0x7f142407eb50 secure :-1 s=THROTTLE_DONE pgs=1618 cs=0 l=1 rx=0x7f141c0032b0 tx=0x7f141c00fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:52:48.971 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:48.939+0000 7f142259c700 1 -- 172.21.15.42:0/4234999419 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f141c0108c0 con 0x7f142409fae0 2019-12-05T05:52:48.971 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:48.939+0000 7f1428839700 1 --2- 172.21.15.42:0/4234999419 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f142409fae0 0x7f142407eb50 secure :-1 s=THROTTLE_DONE pgs=1618 cs=0 l=1 rx=0x7f141c0032b0 tx=0x7f141c00fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:52:48.971 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:48.939+0000 7f142259c700 1 -- 172.21.15.42:0/4234999419 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f141c008b50 con 0x7f142409fae0 2019-12-05T05:52:48.971 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:48.939+0000 7f142259c700 1 -- 172.21.15.42:0/4234999419 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f141c0110c0 con 0x7f142409fae0 2019-12-05T05:52:48.971 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:48.939+0000 7f142903a700 1 -- 172.21.15.42:0/4234999419 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f142409fae0 msgr2=0x7f142407eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:52:48.971 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:48.939+0000 7f142903a700 1 --2- 172.21.15.42:0/4234999419 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f142409fae0 0x7f142407eb50 secure :-1 s=READY pgs=1618 cs=0 l=1 rx=0x7f141c0032b0 tx=0x7f141c00fff0).stop 2019-12-05T05:52:48.971 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:48.943+0000 7f142903a700 1 -- 172.21.15.42:0/4234999419 shutdown_connections 2019-12-05T05:52:48.972 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:48.943+0000 7f142903a700 1 --2- 172.21.15.42:0/4234999419 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f142409fae0 0x7f142407eb50 unknown :-1 s=CLOSED pgs=1618 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:52:48.972 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:48.943+0000 7f142903a700 1 --2- 172.21.15.42:0/4234999419 >> v2:172.21.15.40:3300/0 conn(0x7f14240a7740 0x7f142408df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:52:48.972 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:48.943+0000 7f142903a700 1 -- 172.21.15.42:0/4234999419 shutdown_connections 2019-12-05T05:52:48.972 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:48.943+0000 7f142903a700 1 -- 172.21.15.42:0/4234999419 wait complete. 2019-12-05T05:52:48.972 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:48.943+0000 7f142903a700 1 -- 172.21.15.42:0/4234999419 >> 172.21.15.42:0/4234999419 conn(0x7f14240aabe0 msgr2=0x7f14240a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:52:48.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:48.943+0000 7f142903a700 1 Processor -- start 2019-12-05T05:52:48.976 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:48.943+0000 7f142903a700 1 -- start start 2019-12-05T05:52:48.976 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:48.947+0000 7f142903a700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f14241219e0 0x7f142407d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:52:48.976 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:48.947+0000 7f142903a700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1424121000 con 0x7f14241161c0 2019-12-05T05:52:48.976 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:48.947+0000 7f142903a700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1424118f10 con 0x7f14241165f0 2019-12-05T05:52:48.976 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:48.947+0000 7f142903a700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1424117f30 con 0x7f14241219e0 2019-12-05T05:52:48.976 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:48.947+0000 7f142359e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f14241219e0 0x7f142407d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:52:48.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:48.947+0000 7f1428839700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f14241165f0 0x7f142412dac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:43254/0 (socket says 172.21.15.42:43254) 2019-12-05T05:52:48.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:48.947+0000 7f142359e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f14241219e0 0x7f142407d930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:43212/0 (socket says 172.21.15.42:43212) 2019-12-05T05:52:48.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:48.947+0000 7f1428839700 1 -- 172.21.15.42:0/2577058261 learned_addr learned my addr 172.21.15.42:0/2577058261 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:52:48.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:48.947+0000 7f142359e700 1 -- 172.21.15.42:0/2577058261 >> v1:172.21.15.42:6789/0 conn(0x7f14241165f0 legacy=0x7f142412dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:52:48.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:48.947+0000 7f142359e700 1 -- 172.21.15.42:0/2577058261 >> v1:172.21.15.40:6789/0 conn(0x7f14241161c0 legacy=0x7f1424118710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:52:48.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:48.947+0000 7f140bfff700 1 -- 172.21.15.42:0/2577058261 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1783636501 0 0) 0x7f1424118f10 con 0x7f14241165f0 2019-12-05T05:52:48.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:48.947+0000 7f142359e700 1 -- 172.21.15.42:0/2577058261 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f142407be90 con 0x7f14241219e0 2019-12-05T05:52:48.978 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:48.947+0000 7f142359e700 1 --2- 172.21.15.42:0/2577058261 >> v2:172.21.15.42:3300/0 conn(0x7f14241219e0 0x7f142407d930 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f1414002f30 tx=0x7f1414010ba0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:52:48.978 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:48.947+0000 7f142359e700 1 --2- 172.21.15.42:0/2577058261 >> v2:172.21.15.42:3300/0 conn(0x7f14241219e0 0x7f142407d930 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f1414002f30 tx=0x7f1414010ba0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:52:48.978 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:48.947+0000 7f142359e700 1 --2- 172.21.15.42:0/2577058261 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f14241219e0 0x7f142407d930 secure :-1 s=READY pgs=1620 cs=0 l=1 rx=0x7f1414002f30 tx=0x7f1414010ba0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:52:48.978 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:48.947+0000 7f142359e700 1 --2- 172.21.15.42:0/2577058261 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f14241219e0 0x7f142407d930 secure :-1 s=THROTTLE_DONE pgs=1620 cs=0 l=1 rx=0x7f1414002f30 tx=0x7f1414010ba0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:52:48.978 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:48.947+0000 7f140bfff700 1 -- 172.21.15.42:0/2577058261 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f141400d0c0 con 0x7f14241219e0 2019-12-05T05:52:48.978 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:48.947+0000 7f142359e700 1 --2- 172.21.15.42:0/2577058261 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f14241219e0 0x7f142407d930 secure :-1 s=THROTTLE_DONE pgs=1620 cs=0 l=1 rx=0x7f1414002f30 tx=0x7f1414010ba0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:52:48.978 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:48.947+0000 7f142359e700 1 --2- 172.21.15.42:0/2577058261 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f14241219e0 0x7f142407d930 secure :-1 s=THROTTLE_DONE pgs=1620 cs=0 l=1 rx=0x7f1414002f30 tx=0x7f1414010ba0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:52:48.978 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:48.947+0000 7f140bfff700 1 -- 172.21.15.42:0/2577058261 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1414007150 con 0x7f14241219e0 2019-12-05T05:52:48.979 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:48.947+0000 7f142903a700 1 -- 172.21.15.42:0/2577058261 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1424119fe0 con 0x7f14241219e0 2019-12-05T05:52:48.979 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:48.947+0000 7f140bfff700 1 -- 172.21.15.42:0/2577058261 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f14140129e0 con 0x7f14241219e0 2019-12-05T05:52:48.979 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:48.947+0000 7f142903a700 1 -- 172.21.15.42:0/2577058261 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1424117b00 con 0x7f14241219e0 2019-12-05T05:52:48.979 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:48.947+0000 7f142359e700 1 --2- 172.21.15.42:0/2577058261 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f14241219e0 0x7f142407d930 secure :-1 s=THROTTLE_DONE pgs=1620 cs=0 l=1 rx=0x7f1414002f30 tx=0x7f1414010ba0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:52:48.979 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:48.951+0000 7f140bfff700 1 -- 172.21.15.42:0/2577058261 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f1414012cf0 con 0x7f14241219e0 2019-12-05T05:52:48.979 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:48.951+0000 7f142359e700 1 --2- 172.21.15.42:0/2577058261 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f14241219e0 0x7f142407d930 secure :-1 s=THROTTLE_DONE pgs=1620 cs=0 l=1 rx=0x7f1414002f30 tx=0x7f1414010ba0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:52:48.979 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:48.951+0000 7f140bfff700 1 --2- 172.21.15.42:0/2577058261 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f140403acd0 0x7f140403d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:52:48.980 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:48.951+0000 7f140bfff700 1 -- 172.21.15.42:0/2577058261 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8869+0+0 (secure 0 0 0) 0x7f141404b260 con 0x7f14241219e0 2019-12-05T05:52:48.980 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:48.951+0000 7f1422d9d700 1 --2- 172.21.15.42:0/2577058261 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f140403acd0 0x7f140403d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:52:48.981 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:48.951+0000 7f1422d9d700 1 --2- 172.21.15.42:0/2577058261 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f140403acd0 0x7f140403d1e0 crc :-1 s=READY pgs=694 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:52:48.981 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:48.951+0000 7f142903a700 1 -- 172.21.15.42:0/2577058261 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f142405d110 con 0x7f14241219e0 2019-12-05T05:52:48.981 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:48.955+0000 7f142359e700 1 --2- 172.21.15.42:0/2577058261 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f14241219e0 0x7f142407d930 secure :-1 s=THROTTLE_DONE pgs=1620 cs=0 l=1 rx=0x7f1414002f30 tx=0x7f1414010ba0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:52:48.981 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:48.955+0000 7f140bfff700 1 -- 172.21.15.42:0/2577058261 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f1414032540 con 0x7f14241219e0 2019-12-05T05:52:49.129 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:49.119+0000 7f142903a700 1 -- 172.21.15.42:0/2577058261 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f142405b8c0 con 0x7f14241219e0 2019-12-05T05:52:49.243 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:49.119+0000 7f142359e700 1 --2- 172.21.15.42:0/2577058261 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f14241219e0 0x7f142407d930 secure :-1 s=THROTTLE_DONE pgs=1620 cs=0 l=1 rx=0x7f1414002f30 tx=0x7f1414010ba0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:52:49.243 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:49.119+0000 7f140bfff700 1 -- 172.21.15.42:0/2577058261 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 198 v198) v1 ==== 95+0+2102 (secure 0 0 0) 0x7f14140330f0 con 0x7f14241219e0 2019-12-05T05:52:49.243 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 198 2019-12-05T05:52:49.244 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:52:49.244 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":198,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6356,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/1523046776","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1523046776},{"type":"v1","addr":"172.21.15.40:6833","nonce":1523046776}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":198}],"filesystems":[{"mdsmap":{"epoch":197,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:51:24.172910+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8430},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8430":{"gid":8430,"name":"a","rank":0,"incarnation":194,"state":"up:active","state_seq":9,"addr":"172.21.15.42:6835/5991897","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":5991897},{"type":"v1","addr":"172.21.15.42:6835","nonce":5991897}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:52:49.246 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:49.127+0000 7f142903a700 1 -- 172.21.15.42:0/2577058261 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f140403acd0 msgr2=0x7f140403d1e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:52:49.246 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:49.127+0000 7f142903a700 1 --2- 172.21.15.42:0/2577058261 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f140403acd0 0x7f140403d1e0 crc :-1 s=READY pgs=694 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:52:49.246 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:49.127+0000 7f142903a700 1 -- 172.21.15.42:0/2577058261 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f14241219e0 msgr2=0x7f142407d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:52:49.246 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:49.127+0000 7f142903a700 1 --2- 172.21.15.42:0/2577058261 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f14241219e0 0x7f142407d930 secure :-1 s=READY pgs=1620 cs=0 l=1 rx=0x7f1414002f30 tx=0x7f1414010ba0).stop 2019-12-05T05:52:49.246 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:49.127+0000 7f142903a700 1 -- 172.21.15.42:0/2577058261 shutdown_connections 2019-12-05T05:52:49.246 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:49.127+0000 7f142903a700 1 --2- 172.21.15.42:0/2577058261 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f140403acd0 0x7f140403d1e0 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:52:49.247 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:49.127+0000 7f142903a700 1 --2- 172.21.15.42:0/2577058261 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f14241219e0 0x7f142407d930 unknown :-1 s=CLOSED pgs=1620 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:52:49.247 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:49.127+0000 7f142903a700 1 -- 172.21.15.42:0/2577058261 shutdown_connections 2019-12-05T05:52:49.247 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:49.127+0000 7f142903a700 1 -- 172.21.15.42:0/2577058261 wait complete. 2019-12-05T05:52:49.247 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:49.127+0000 7f142903a700 1 -- 172.21.15.42:0/2577058261 >> 172.21.15.42:0/2577058261 conn(0x7f14240a7f90 msgr2=0x7f142409f4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:52:49.247 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T05:52:51.249 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:52:51.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:51.339+0000 7f837c864700 1 Processor -- start 2019-12-05T05:52:51.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:51.339+0000 7f837c864700 1 -- start start 2019-12-05T05:52:51.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:51.343+0000 7f837c864700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f837808df80 0x7f837809f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:52:51.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:51.343+0000 7f837c864700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f837809fc60 con 0x7f837809f830 2019-12-05T05:52:51.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:51.343+0000 7f837c864700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83780a0710 con 0x7f83780a7740 2019-12-05T05:52:51.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:51.343+0000 7f837c864700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83780a0b00 con 0x7f837808df80 2019-12-05T05:52:51.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:51.343+0000 7f8376d9d700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f83780a7740 0x7f837807e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:43260/0 (socket says 172.21.15.42:43260) 2019-12-05T05:52:51.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:51.343+0000 7f8376d9d700 1 -- 172.21.15.42:0/1993118218 learned_addr learned my addr 172.21.15.42:0/1993118218 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:52:51.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:51.343+0000 7f837659c700 1 --2- 172.21.15.42:0/1993118218 >> v2:172.21.15.42:3300/0 conn(0x7f837808df80 0x7f837809f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:52:51.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:51.343+0000 7f837659c700 1 -- 172.21.15.42:0/1993118218 >> v1:172.21.15.42:6789/0 conn(0x7f83780a7740 legacy=0x7f837807e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:52:51.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:51.343+0000 7f837659c700 1 -- 172.21.15.42:0/1993118218 >> v1:172.21.15.40:6789/0 conn(0x7f837809f830 legacy=0x7f837814efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:52:51.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:51.343+0000 7f8375d9b700 1 -- 172.21.15.42:0/1993118218 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2514289338 0 0) 0x7f83780a0710 con 0x7f83780a7740 2019-12-05T05:52:51.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:51.343+0000 7f837659c700 1 -- 172.21.15.42:0/1993118218 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f837807ef20 con 0x7f837808df80 2019-12-05T05:52:51.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:51.343+0000 7f837659c700 1 --2- 172.21.15.42:0/1993118218 >> v2:172.21.15.42:3300/0 conn(0x7f837808df80 0x7f837809f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f8360008790 tx=0x7f836000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:52:51.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:51.343+0000 7f837659c700 1 --2- 172.21.15.42:0/1993118218 >> v2:172.21.15.42:3300/0 conn(0x7f837808df80 0x7f837809f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f8360008790 tx=0x7f836000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:52:51.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:51.343+0000 7f837659c700 1 --2- 172.21.15.42:0/1993118218 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f837808df80 0x7f837809f2e0 secure :-1 s=READY pgs=1622 cs=0 l=1 rx=0x7f8360008790 tx=0x7f836000fbf0).ready entity=mon.1 client_cookie=24aa8bfedef05bf9 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:52:51.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:51.343+0000 7f837659c700 1 --2- 172.21.15.42:0/1993118218 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f837808df80 0x7f837809f2e0 secure :-1 s=THROTTLE_DONE pgs=1622 cs=0 l=1 rx=0x7f8360008790 tx=0x7f836000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:52:51.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:51.343+0000 7f8375d9b700 1 -- 172.21.15.42:0/1993118218 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8360011270 con 0x7f837808df80 2019-12-05T05:52:51.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:51.343+0000 7f837659c700 1 --2- 172.21.15.42:0/1993118218 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f837808df80 0x7f837809f2e0 secure :-1 s=THROTTLE_DONE pgs=1622 cs=0 l=1 rx=0x7f8360008790 tx=0x7f836000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:52:51.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:51.343+0000 7f8375d9b700 1 -- 172.21.15.42:0/1993118218 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8360008bc0 con 0x7f837808df80 2019-12-05T05:52:51.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:51.343+0000 7f837659c700 1 --2- 172.21.15.42:0/1993118218 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f837808df80 0x7f837809f2e0 secure :-1 s=THROTTLE_DONE pgs=1622 cs=0 l=1 rx=0x7f8360008790 tx=0x7f836000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:52:51.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:51.343+0000 7f837c864700 1 -- 172.21.15.42:0/1993118218 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f837808df80 msgr2=0x7f837809f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:52:51.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:51.343+0000 7f8375d9b700 1 -- 172.21.15.42:0/1993118218 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8360011850 con 0x7f837808df80 2019-12-05T05:52:51.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:51.343+0000 7f837c864700 1 --2- 172.21.15.42:0/1993118218 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f837808df80 0x7f837809f2e0 secure :-1 s=READY pgs=1622 cs=0 l=1 rx=0x7f8360008790 tx=0x7f836000fbf0).stop 2019-12-05T05:52:51.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:51.343+0000 7f837c864700 1 -- 172.21.15.42:0/1993118218 shutdown_connections 2019-12-05T05:52:51.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:51.343+0000 7f837c864700 1 --2- 172.21.15.42:0/1993118218 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f837808df80 0x7f837809f2e0 unknown :-1 s=CLOSED pgs=1622 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:52:51.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:51.343+0000 7f837c864700 1 -- 172.21.15.42:0/1993118218 shutdown_connections 2019-12-05T05:52:51.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:51.343+0000 7f837c864700 1 -- 172.21.15.42:0/1993118218 wait complete. 2019-12-05T05:52:51.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:51.343+0000 7f837c864700 1 -- 172.21.15.42:0/1993118218 >> 172.21.15.42:0/1993118218 conn(0x7f83780aabe0 msgr2=0x7f83780a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:52:51.359 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:51.347+0000 7f837c864700 1 Processor -- start 2019-12-05T05:52:51.359 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:51.347+0000 7f837c864700 1 -- start start 2019-12-05T05:52:51.359 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:51.347+0000 7f837c864700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f837811db20 0x7f83780a5ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:52:51.359 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:51.347+0000 7f837c864700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f8378122b90 0x7f837811abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:52:51.360 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:51.347+0000 7f837c864700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f837811b110 con 0x7f8378122760 2019-12-05T05:52:51.360 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:51.347+0000 7f837c864700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83780a6b30 con 0x7f8378122b90 2019-12-05T05:52:51.360 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:51.347+0000 7f837c864700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83780a6890 con 0x7f837811db20 2019-12-05T05:52:51.360 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:51.347+0000 7f837759e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f8378122b90 0x7f837811abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:52:51.360 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:51.347+0000 7f8376d9d700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f837811db20 0x7f83780a5ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:52:51.360 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:51.347+0000 7f837659c700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f8378122760 0x7f837811a3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:43268/0 (socket says 172.21.15.42:43268) 2019-12-05T05:52:51.361 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:51.347+0000 7f837659c700 1 -- 172.21.15.42:0/1381094473 learned_addr learned my addr 172.21.15.42:0/1381094473 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:52:51.361 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:51.347+0000 7f837759e700 1 --2- 172.21.15.42:0/1381094473 >> v2:172.21.15.42:3300/0 conn(0x7f8378122b90 0x7f837811abc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:43230/0 (socket says 172.21.15.42:43230) 2019-12-05T05:52:51.361 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:51.347+0000 7f837759e700 1 -- 172.21.15.42:0/1381094473 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f837811db20 msgr2=0x7f83780a5ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:52:51.361 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:51.347+0000 7f837759e700 1 --2- 172.21.15.42:0/1381094473 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f837811db20 0x7f83780a5ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:52:51.361 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:51.347+0000 7f837759e700 1 -- 172.21.15.42:0/1381094473 >> v1:172.21.15.42:6789/0 conn(0x7f8378122760 legacy=0x7f837811a3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:52:51.361 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:51.347+0000 7f836f7fe700 1 -- 172.21.15.42:0/1381094473 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3741327641 0 0) 0x7f837811b110 con 0x7f8378122760 2019-12-05T05:52:51.362 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:51.347+0000 7f837759e700 1 -- 172.21.15.42:0/1381094473 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f837811df50 con 0x7f8378122b90 2019-12-05T05:52:51.362 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:51.347+0000 7f837759e700 1 --2- 172.21.15.42:0/1381094473 >> v2:172.21.15.42:3300/0 conn(0x7f8378122b90 0x7f837811abc0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f837811e530 tx=0x7f8370010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:52:51.362 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:51.347+0000 7f837759e700 1 --2- 172.21.15.42:0/1381094473 >> v2:172.21.15.42:3300/0 conn(0x7f8378122b90 0x7f837811abc0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f837811e530 tx=0x7f8370010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:52:51.362 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:51.347+0000 7f837759e700 1 --2- 172.21.15.42:0/1381094473 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8378122b90 0x7f837811abc0 secure :-1 s=READY pgs=1624 cs=0 l=1 rx=0x7f837811e530 tx=0x7f8370010b40).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:52:51.362 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:51.347+0000 7f837759e700 1 --2- 172.21.15.42:0/1381094473 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8378122b90 0x7f837811abc0 secure :-1 s=THROTTLE_DONE pgs=1624 cs=0 l=1 rx=0x7f837811e530 tx=0x7f8370010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:52:51.362 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:51.347+0000 7f836f7fe700 1 -- 172.21.15.42:0/1381094473 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8370006bd0 con 0x7f8378122b90 2019-12-05T05:52:51.362 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:51.347+0000 7f837759e700 1 --2- 172.21.15.42:0/1381094473 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8378122b90 0x7f837811abc0 secure :-1 s=THROTTLE_DONE pgs=1624 cs=0 l=1 rx=0x7f837811e530 tx=0x7f8370010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:52:51.363 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:51.347+0000 7f837759e700 1 --2- 172.21.15.42:0/1381094473 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8378122b90 0x7f837811abc0 secure :-1 s=THROTTLE_DONE pgs=1624 cs=0 l=1 rx=0x7f837811e530 tx=0x7f8370010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:52:51.363 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:51.347+0000 7f836f7fe700 1 -- 172.21.15.42:0/1381094473 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8370013060 con 0x7f8378122b90 2019-12-05T05:52:51.363 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:51.347+0000 7f836f7fe700 1 -- 172.21.15.42:0/1381094473 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8370002430 con 0x7f8378122b90 2019-12-05T05:52:51.363 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:51.347+0000 7f837c864700 1 -- 172.21.15.42:0/1381094473 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8378120c40 con 0x7f8378122b90 2019-12-05T05:52:51.363 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:51.351+0000 7f837c864700 1 -- 172.21.15.42:0/1381094473 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83780a4840 con 0x7f8378122b90 2019-12-05T05:52:51.363 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:51.351+0000 7f837759e700 1 --2- 172.21.15.42:0/1381094473 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8378122b90 0x7f837811abc0 secure :-1 s=THROTTLE_DONE pgs=1624 cs=0 l=1 rx=0x7f837811e530 tx=0x7f8370010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:52:51.364 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:51.351+0000 7f836f7fe700 1 -- 172.21.15.42:0/1381094473 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f8370002740 con 0x7f8378122b90 2019-12-05T05:52:51.364 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:51.351+0000 7f837759e700 1 --2- 172.21.15.42:0/1381094473 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8378122b90 0x7f837811abc0 secure :-1 s=THROTTLE_DONE pgs=1624 cs=0 l=1 rx=0x7f837811e530 tx=0x7f8370010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:52:51.364 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:51.351+0000 7f836f7fe700 1 --2- 172.21.15.42:0/1381094473 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f835c03abd0 0x7f835c03d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:52:51.364 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:51.351+0000 7f836f7fe700 1 -- 172.21.15.42:0/1381094473 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8869+0+0 (secure 0 0 0) 0x7f837004b280 con 0x7f8378122b90 2019-12-05T05:52:51.364 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:51.351+0000 7f8376d9d700 1 --2- 172.21.15.42:0/1381094473 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f835c03abd0 0x7f835c03d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:52:51.364 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:51.351+0000 7f8376d9d700 1 --2- 172.21.15.42:0/1381094473 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f835c03abd0 0x7f835c03d0e0 crc :-1 s=READY pgs=695 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:52:51.365 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:51.355+0000 7f837c864700 1 -- 172.21.15.42:0/1381094473 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f837805d110 con 0x7f8378122b90 2019-12-05T05:52:51.369 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:51.363+0000 7f837759e700 1 --2- 172.21.15.42:0/1381094473 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8378122b90 0x7f837811abc0 secure :-1 s=THROTTLE_DONE pgs=1624 cs=0 l=1 rx=0x7f837811e530 tx=0x7f8370010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:52:51.369 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:51.363+0000 7f836f7fe700 1 -- 172.21.15.42:0/1381094473 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f8370032300 con 0x7f8378122b90 2019-12-05T05:52:51.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:51.535+0000 7f837c864700 1 -- 172.21.15.42:0/1381094473 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f837805b8c0 con 0x7f8378122b90 2019-12-05T05:52:51.600 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:51.535+0000 7f837759e700 1 --2- 172.21.15.42:0/1381094473 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8378122b90 0x7f837811abc0 secure :-1 s=THROTTLE_DONE pgs=1624 cs=0 l=1 rx=0x7f837811e530 tx=0x7f8370010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:52:51.600 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:51.535+0000 7f836f7fe700 1 -- 172.21.15.42:0/1381094473 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 198 v198) v1 ==== 95+0+2102 (secure 0 0 0) 0x7f8370032560 con 0x7f8378122b90 2019-12-05T05:52:51.600 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 198 2019-12-05T05:52:51.601 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:51.535+0000 7f837c864700 1 -- 172.21.15.42:0/1381094473 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f835c03abd0 msgr2=0x7f835c03d0e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:52:51.601 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:51.535+0000 7f837c864700 1 --2- 172.21.15.42:0/1381094473 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f835c03abd0 0x7f835c03d0e0 crc :-1 s=READY pgs=695 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:52:51.601 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:51.535+0000 7f837c864700 1 -- 172.21.15.42:0/1381094473 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8378122b90 msgr2=0x7f837811abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:52:51.601 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:51.535+0000 7f837c864700 1 --2- 172.21.15.42:0/1381094473 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8378122b90 0x7f837811abc0 secure :-1 s=READY pgs=1624 cs=0 l=1 rx=0x7f837811e530 tx=0x7f8370010b40).stop 2019-12-05T05:52:51.601 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:51.539+0000 7f837c864700 1 -- 172.21.15.42:0/1381094473 shutdown_connections 2019-12-05T05:52:51.601 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:51.539+0000 7f837c864700 1 --2- 172.21.15.42:0/1381094473 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f835c03abd0 0x7f835c03d0e0 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:52:51.601 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:51.539+0000 7f837c864700 1 --2- 172.21.15.42:0/1381094473 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8378122b90 0x7f837811abc0 unknown :-1 s=CLOSED pgs=1624 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:52:51.602 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:51.539+0000 7f837c864700 1 --2- 172.21.15.42:0/1381094473 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f837811db20 0x7f83780a5ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:52:51.602 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:51.539+0000 7f837c864700 1 -- 172.21.15.42:0/1381094473 shutdown_connections 2019-12-05T05:52:51.602 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:51.539+0000 7f837c864700 1 -- 172.21.15.42:0/1381094473 wait complete. 2019-12-05T05:52:51.602 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:51.539+0000 7f837c864700 1 -- 172.21.15.42:0/1381094473 >> 172.21.15.42:0/1381094473 conn(0x7f83780a8f60 msgr2=0x7f83781416d0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:52:51.602 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:52:51.603 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":198,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6356,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/1523046776","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1523046776},{"type":"v1","addr":"172.21.15.40:6833","nonce":1523046776}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":198}],"filesystems":[{"mdsmap":{"epoch":197,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:51:24.172910+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8430},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8430":{"gid":8430,"name":"a","rank":0,"incarnation":194,"state":"up:active","state_seq":9,"addr":"172.21.15.42:6835/5991897","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":5991897},{"type":"v1","addr":"172.21.15.42:6835","nonce":5991897}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:52:51.603 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T05:52:53.604 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:52:53.708 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:53.695+0000 7fe818b1d700 1 Processor -- start 2019-12-05T05:52:53.709 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:53.695+0000 7fe818b1d700 1 -- start start 2019-12-05T05:52:53.709 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:53.699+0000 7fe818b1d700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe8140a7740 0x7fe81408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:52:53.709 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:53.699+0000 7fe818b1d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe81407e720 0x7fe81407eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:52:53.709 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:53.699+0000 7fe818b1d700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe81409f2e0 0x7fe81409f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:52:53.709 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:53.699+0000 7fe818b1d700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8140a0b00 con 0x7fe81409f2e0 2019-12-05T05:52:53.709 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:53.699+0000 7fe818b1d700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe81408e5b0 con 0x7fe81407e720 2019-12-05T05:52:53.709 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:53.699+0000 7fe818b1d700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe81408e850 con 0x7fe8140a7740 2019-12-05T05:52:53.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:53.699+0000 7fe81259c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe81407e720 0x7fe81407eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:52:53.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:53.699+0000 7fe81259c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe81407e720 0x7fe81407eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:43240/0 (socket says 172.21.15.42:43240) 2019-12-05T05:52:53.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:53.699+0000 7fe81259c700 1 -- 172.21.15.42:0/272413592 learned_addr learned my addr 172.21.15.42:0/272413592 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:52:53.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:53.699+0000 7fe812d9d700 1 --2- 172.21.15.42:0/272413592 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe8140a7740 0x7fe81408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:52:53.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:53.699+0000 7fe81359e700 1 --2- 172.21.15.42:0/272413592 >> v2:172.21.15.40:3300/0 conn(0x7fe81409f2e0 0x7fe81409f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:52:53.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:53.699+0000 7fe81259c700 1 -- 172.21.15.42:0/272413592 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe8140a7740 msgr2=0x7fe81408df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:52:53.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:53.699+0000 7fe81259c700 1 --2- 172.21.15.42:0/272413592 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe8140a7740 0x7fe81408df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:52:53.711 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:53.699+0000 7fe81259c700 1 -- 172.21.15.42:0/272413592 >> v2:172.21.15.40:3300/0 conn(0x7fe81409f2e0 msgr2=0x7fe81409f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:52:53.711 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:53.699+0000 7fe81259c700 1 --2- 172.21.15.42:0/272413592 >> v2:172.21.15.40:3300/0 conn(0x7fe81409f2e0 0x7fe81409f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:52:53.711 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:53.699+0000 7fe81259c700 1 -- 172.21.15.42:0/272413592 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe81407f0a0 con 0x7fe81407e720 2019-12-05T05:52:53.711 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:53.699+0000 7fe81359e700 1 --2- 172.21.15.42:0/272413592 >> v2:172.21.15.40:3300/0 conn(0x7fe81409f2e0 0x7fe81409f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).send_auth_request state changed! 2019-12-05T05:52:53.711 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:53.699+0000 7fe81259c700 1 --2- 172.21.15.42:0/272413592 >> v2:172.21.15.42:3300/0 conn(0x7fe81407e720 0x7fe81407eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fe81408e770 tx=0x7fe80800fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:52:53.711 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:53.699+0000 7fe81259c700 1 --2- 172.21.15.42:0/272413592 >> v2:172.21.15.42:3300/0 conn(0x7fe81407e720 0x7fe81407eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fe81408e770 tx=0x7fe80800fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:52:53.711 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:53.699+0000 7fe81259c700 1 --2- 172.21.15.42:0/272413592 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe81407e720 0x7fe81407eb50 secure :-1 s=READY pgs=1625 cs=0 l=1 rx=0x7fe81408e770 tx=0x7fe80800fdb0).ready entity=mon.1 client_cookie=27ffd49bf9bff56c server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:52:53.712 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:53.699+0000 7fe81259c700 1 --2- 172.21.15.42:0/272413592 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe81407e720 0x7fe81407eb50 secure :-1 s=THROTTLE_DONE pgs=1625 cs=0 l=1 rx=0x7fe81408e770 tx=0x7fe80800fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:52:53.712 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:53.699+0000 7fe811d9b700 1 -- 172.21.15.42:0/272413592 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe808008c90 con 0x7fe81407e720 2019-12-05T05:52:53.712 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:53.699+0000 7fe81259c700 1 --2- 172.21.15.42:0/272413592 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe81407e720 0x7fe81407eb50 secure :-1 s=THROTTLE_DONE pgs=1625 cs=0 l=1 rx=0x7fe81408e770 tx=0x7fe80800fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:52:53.712 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:53.699+0000 7fe81259c700 1 --2- 172.21.15.42:0/272413592 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe81407e720 0x7fe81407eb50 secure :-1 s=THROTTLE_DONE pgs=1625 cs=0 l=1 rx=0x7fe81408e770 tx=0x7fe80800fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:52:53.712 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:53.699+0000 7fe811d9b700 1 -- 172.21.15.42:0/272413592 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8080088b0 con 0x7fe81407e720 2019-12-05T05:52:53.712 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:53.699+0000 7fe811d9b700 1 -- 172.21.15.42:0/272413592 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe808011b50 con 0x7fe81407e720 2019-12-05T05:52:53.712 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:53.699+0000 7fe818b1d700 1 -- 172.21.15.42:0/272413592 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe81407e720 msgr2=0x7fe81407eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:52:53.713 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:53.699+0000 7fe818b1d700 1 --2- 172.21.15.42:0/272413592 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe81407e720 0x7fe81407eb50 secure :-1 s=READY pgs=1625 cs=0 l=1 rx=0x7fe81408e770 tx=0x7fe80800fdb0).stop 2019-12-05T05:52:53.713 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:53.703+0000 7fe818b1d700 1 -- 172.21.15.42:0/272413592 shutdown_connections 2019-12-05T05:52:53.713 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:53.703+0000 7fe818b1d700 1 --2- 172.21.15.42:0/272413592 >> v2:172.21.15.40:3300/0 conn(0x7fe81409f2e0 0x7fe81409f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:52:53.713 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:53.703+0000 7fe818b1d700 1 --2- 172.21.15.42:0/272413592 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe8140a7740 0x7fe81408df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:52:53.713 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:53.703+0000 7fe818b1d700 1 --2- 172.21.15.42:0/272413592 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe81407e720 0x7fe81407eb50 unknown :-1 s=CLOSED pgs=1625 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:52:53.713 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:53.703+0000 7fe818b1d700 1 -- 172.21.15.42:0/272413592 shutdown_connections 2019-12-05T05:52:53.713 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:53.703+0000 7fe818b1d700 1 -- 172.21.15.42:0/272413592 wait complete. 2019-12-05T05:52:53.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:53.703+0000 7fe818b1d700 1 -- 172.21.15.42:0/272413592 >> 172.21.15.42:0/272413592 conn(0x7fe8140aabe0 msgr2=0x7fe8140a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:52:53.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:53.703+0000 7fe818b1d700 1 Processor -- start 2019-12-05T05:52:53.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:53.703+0000 7fe818b1d700 1 -- start start 2019-12-05T05:52:53.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:53.703+0000 7fe818b1d700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe814124600 0x7fe814144110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:52:53.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:53.703+0000 7fe818b1d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe814116360 0x7fe814123a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:52:53.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:53.703+0000 7fe818b1d700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe814145460 con 0x7fe81408eb70 2019-12-05T05:52:53.718 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:53.703+0000 7fe818b1d700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe814116790 con 0x7fe814124600 2019-12-05T05:52:53.718 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:53.703+0000 7fe818b1d700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe814124b80 con 0x7fe814116360 2019-12-05T05:52:53.718 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:53.703+0000 7fe81259c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe814116360 0x7fe814123a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:52:53.718 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:53.703+0000 7fe81259c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe814116360 0x7fe814123a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:43246/0 (socket says 172.21.15.42:43246) 2019-12-05T05:52:53.718 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:53.703+0000 7fe81259c700 1 -- 172.21.15.42:0/949736851 learned_addr learned my addr 172.21.15.42:0/949736851 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:52:53.718 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:53.703+0000 7fe812d9d700 1 --2- 172.21.15.42:0/949736851 >> v2:172.21.15.40:3300/0 conn(0x7fe814124600 0x7fe814144110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:52:53.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:53.707+0000 7fe81259c700 1 -- 172.21.15.42:0/949736851 >> v2:172.21.15.40:3300/0 conn(0x7fe814124600 msgr2=0x7fe814144110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:52:53.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:53.707+0000 7fe81259c700 1 --2- 172.21.15.42:0/949736851 >> v2:172.21.15.40:3300/0 conn(0x7fe814124600 0x7fe814144110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:52:53.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:53.707+0000 7fe81259c700 1 -- 172.21.15.42:0/949736851 >> v1:172.21.15.42:6789/0 conn(0x7fe81408eb70 legacy=0x7fe81408efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:52:53.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:53.707+0000 7fe81259c700 1 -- 172.21.15.42:0/949736851 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe814123fe0 con 0x7fe814116360 2019-12-05T05:52:53.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:53.707+0000 7fe812d9d700 1 --2- 172.21.15.42:0/949736851 >> v2:172.21.15.40:3300/0 conn(0x7fe814124600 0x7fe814144110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T05:52:53.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:53.707+0000 7fe81259c700 1 --2- 172.21.15.42:0/949736851 >> v2:172.21.15.42:3300/0 conn(0x7fe814116360 0x7fe814123a90 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fe81408e770 tx=0x7fe808008be0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:52:53.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:53.707+0000 7fe81259c700 1 --2- 172.21.15.42:0/949736851 >> v2:172.21.15.42:3300/0 conn(0x7fe814116360 0x7fe814123a90 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fe81408e770 tx=0x7fe808008be0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:52:53.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:53.707+0000 7fe81259c700 1 --2- 172.21.15.42:0/949736851 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe814116360 0x7fe814123a90 secure :-1 s=READY pgs=1627 cs=0 l=1 rx=0x7fe81408e770 tx=0x7fe808008be0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:52:53.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:53.707+0000 7fe81259c700 1 --2- 172.21.15.42:0/949736851 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe814116360 0x7fe814123a90 secure :-1 s=THROTTLE_DONE pgs=1627 cs=0 l=1 rx=0x7fe81408e770 tx=0x7fe808008be0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:52:53.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:53.707+0000 7fe8037fe700 1 -- 172.21.15.42:0/949736851 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe80800f470 con 0x7fe814116360 2019-12-05T05:52:53.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:53.707+0000 7fe81259c700 1 --2- 172.21.15.42:0/949736851 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe814116360 0x7fe814123a90 secure :-1 s=THROTTLE_DONE pgs=1627 cs=0 l=1 rx=0x7fe81408e770 tx=0x7fe808008be0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:52:53.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:53.707+0000 7fe81259c700 1 --2- 172.21.15.42:0/949736851 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe814116360 0x7fe814123a90 secure :-1 s=THROTTLE_DONE pgs=1627 cs=0 l=1 rx=0x7fe81408e770 tx=0x7fe808008be0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:52:53.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:53.707+0000 7fe8037fe700 1 -- 172.21.15.42:0/949736851 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe808011770 con 0x7fe814116360 2019-12-05T05:52:53.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:53.707+0000 7fe8037fe700 1 -- 172.21.15.42:0/949736851 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe808005730 con 0x7fe814116360 2019-12-05T05:52:53.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:53.707+0000 7fe818b1d700 1 -- 172.21.15.42:0/949736851 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8141224a0 con 0x7fe814116360 2019-12-05T05:52:53.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:53.707+0000 7fe818b1d700 1 -- 172.21.15.42:0/949736851 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe81408f7a0 con 0x7fe814116360 2019-12-05T05:52:53.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:53.707+0000 7fe81259c700 1 --2- 172.21.15.42:0/949736851 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe814116360 0x7fe814123a90 secure :-1 s=THROTTLE_DONE pgs=1627 cs=0 l=1 rx=0x7fe81408e770 tx=0x7fe808008be0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:52:53.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:53.707+0000 7fe8037fe700 1 -- 172.21.15.42:0/949736851 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fe808005a40 con 0x7fe814116360 2019-12-05T05:52:53.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:53.707+0000 7fe81259c700 1 --2- 172.21.15.42:0/949736851 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe814116360 0x7fe814123a90 secure :-1 s=THROTTLE_DONE pgs=1627 cs=0 l=1 rx=0x7fe81408e770 tx=0x7fe808008be0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:52:53.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:53.707+0000 7fe8037fe700 1 --2- 172.21.15.42:0/949736851 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe7f803ae20 0x7fe7f803d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:52:53.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:53.707+0000 7fe8037fe700 1 -- 172.21.15.42:0/949736851 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8869+0+0 (secure 0 0 0) 0x7fe80804b680 con 0x7fe814116360 2019-12-05T05:52:53.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:53.707+0000 7fe812d9d700 1 --2- 172.21.15.42:0/949736851 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe7f803ae20 0x7fe7f803d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:52:53.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:53.707+0000 7fe812d9d700 1 --2- 172.21.15.42:0/949736851 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe7f803ae20 0x7fe7f803d330 crc :-1 s=READY pgs=696 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:52:53.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:53.711+0000 7fe818b1d700 1 -- 172.21.15.42:0/949736851 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe814124d40 con 0x7fe814116360 2019-12-05T05:52:53.726 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:53.719+0000 7fe81259c700 1 --2- 172.21.15.42:0/949736851 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe814116360 0x7fe814123a90 secure :-1 s=THROTTLE_DONE pgs=1627 cs=0 l=1 rx=0x7fe81408e770 tx=0x7fe808008be0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:52:53.726 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:53.719+0000 7fe8037fe700 1 -- 172.21.15.42:0/949736851 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fe808008de0 con 0x7fe814116360 2019-12-05T05:52:53.895 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:53.891+0000 7fe818b1d700 1 -- 172.21.15.42:0/949736851 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe81405d110 con 0x7fe814116360 2019-12-05T05:52:53.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:53.891+0000 7fe81259c700 1 --2- 172.21.15.42:0/949736851 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe814116360 0x7fe814123a90 secure :-1 s=THROTTLE_DONE pgs=1627 cs=0 l=1 rx=0x7fe81408e770 tx=0x7fe808008be0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:52:53.897 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:53.891+0000 7fe8037fe700 1 -- 172.21.15.42:0/949736851 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 198 v198) v1 ==== 95+0+2102 (secure 0 0 0) 0x7fe808005d50 con 0x7fe814116360 2019-12-05T05:52:53.899 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:52:53.899 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":198,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6356,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/1523046776","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1523046776},{"type":"v1","addr":"172.21.15.40:6833","nonce":1523046776}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":198}],"filesystems":[{"mdsmap":{"epoch":197,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:51:24.172910+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8430},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8430":{"gid":8430,"name":"a","rank":0,"incarnation":194,"state":"up:active","state_seq":9,"addr":"172.21.15.42:6835/5991897","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":5991897},{"type":"v1","addr":"172.21.15.42:6835","nonce":5991897}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:52:53.899 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 198 2019-12-05T05:52:53.901 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:53.895+0000 7fe818b1d700 1 -- 172.21.15.42:0/949736851 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe7f803ae20 msgr2=0x7fe7f803d330 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:52:53.901 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:53.895+0000 7fe818b1d700 1 --2- 172.21.15.42:0/949736851 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe7f803ae20 0x7fe7f803d330 crc :-1 s=READY pgs=696 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:52:53.902 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:53.895+0000 7fe818b1d700 1 -- 172.21.15.42:0/949736851 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe814116360 msgr2=0x7fe814123a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:52:53.902 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:53.895+0000 7fe818b1d700 1 --2- 172.21.15.42:0/949736851 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe814116360 0x7fe814123a90 secure :-1 s=READY pgs=1627 cs=0 l=1 rx=0x7fe81408e770 tx=0x7fe808008be0).stop 2019-12-05T05:52:53.902 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:53.895+0000 7fe818b1d700 1 -- 172.21.15.42:0/949736851 shutdown_connections 2019-12-05T05:52:53.902 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:53.895+0000 7fe818b1d700 1 --2- 172.21.15.42:0/949736851 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe7f803ae20 0x7fe7f803d330 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:52:53.902 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:53.895+0000 7fe818b1d700 1 --2- 172.21.15.42:0/949736851 >> v2:172.21.15.40:3300/0 conn(0x7fe814124600 0x7fe814144110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:52:53.902 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:53.895+0000 7fe818b1d700 1 --2- 172.21.15.42:0/949736851 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe814116360 0x7fe814123a90 unknown :-1 s=CLOSED pgs=1627 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:52:53.903 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:53.895+0000 7fe818b1d700 1 -- 172.21.15.42:0/949736851 shutdown_connections 2019-12-05T05:52:53.903 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:53.895+0000 7fe818b1d700 1 -- 172.21.15.42:0/949736851 wait complete. 2019-12-05T05:52:53.903 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:53.895+0000 7fe818b1d700 1 -- 172.21.15.42:0/949736851 >> 172.21.15.42:0/949736851 conn(0x7fe8140a82a0 msgr2=0x7fe81409f460 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:52:53.912 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T05:52:55.913 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:52:56.019 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:56.007+0000 7f481fcdd700 1 Processor -- start 2019-12-05T05:52:56.020 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:56.007+0000 7f481fcdd700 1 -- start start 2019-12-05T05:52:56.020 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:56.011+0000 7f481fcdd700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f48180a7740 0x7f481808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:52:56.020 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:56.011+0000 7f481fcdd700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48180a0710 con 0x7f481809fae0 2019-12-05T05:52:56.020 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:56.011+0000 7f481fcdd700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48180a9ba0 con 0x7f481807e720 2019-12-05T05:52:56.020 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:56.011+0000 7f481fcdd700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48180a0b00 con 0x7f48180a7740 2019-12-05T05:52:56.020 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:56.011+0000 7f481da79700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f481807e720 0x7f481809f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:43290/0 (socket says 172.21.15.42:43290) 2019-12-05T05:52:56.020 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:56.011+0000 7f481da79700 1 -- 172.21.15.42:0/2416688281 learned_addr learned my addr 172.21.15.42:0/2416688281 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:52:56.021 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:56.011+0000 7f481e27a700 1 --2- 172.21.15.42:0/2416688281 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f48180a7740 0x7f481808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:52:56.021 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:56.011+0000 7f481d278700 1 -- 172.21.15.42:0/2416688281 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2647079678 0 0) 0x7f48180a9ba0 con 0x7f481807e720 2019-12-05T05:52:56.021 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:56.011+0000 7f481d278700 1 -- 172.21.15.42:0/2416688281 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48040030b0 con 0x7f481807e720 2019-12-05T05:52:56.021 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:56.011+0000 7f481e27a700 1 -- 172.21.15.42:0/2416688281 >> v1:172.21.15.42:6789/0 conn(0x7f481807e720 legacy=0x7f481809f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:52:56.021 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:56.011+0000 7f481d278700 1 -- 172.21.15.42:0/2416688281 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1362874289 0 0) 0x7f48180a0710 con 0x7f481809fae0 2019-12-05T05:52:56.021 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:56.011+0000 7f481e27a700 1 -- 172.21.15.42:0/2416688281 >> v1:172.21.15.40:6789/0 conn(0x7f481809fae0 legacy=0x7f481814efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:52:56.022 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:56.011+0000 7f481e27a700 1 -- 172.21.15.42:0/2416688281 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f481807eb50 con 0x7f48180a7740 2019-12-05T05:52:56.022 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:56.011+0000 7f481e27a700 1 --2- 172.21.15.42:0/2416688281 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f48180a7740 0x7f481808df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f480c001910 tx=0x7f4810001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:52:56.022 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:56.011+0000 7f481e27a700 1 --2- 172.21.15.42:0/2416688281 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f48180a7740 0x7f481808df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f480c001910 tx=0x7f4810001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:52:56.022 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:56.011+0000 7f481e27a700 1 --2- 172.21.15.42:0/2416688281 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f48180a7740 0x7f481808df80 secure :-1 s=READY pgs=361 cs=0 l=1 rx=0x7f480c001910 tx=0x7f4810001150).ready entity=mon.2 client_cookie=90a5320cd85f7eac server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:52:56.022 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:56.011+0000 7f481e27a700 1 --2- 172.21.15.42:0/2416688281 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f48180a7740 0x7f481808df80 secure :-1 s=THROTTLE_DONE pgs=361 cs=0 l=1 rx=0x7f480c001910 tx=0x7f4810001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:52:56.022 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:56.011+0000 7f481e27a700 1 --2- 172.21.15.42:0/2416688281 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f48180a7740 0x7f481808df80 secure :-1 s=THROTTLE_DONE pgs=361 cs=0 l=1 rx=0x7f480c001910 tx=0x7f4810001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:52:56.022 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:56.011+0000 7f481d278700 1 -- 172.21.15.42:0/2416688281 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f480c0016d0 con 0x7f48180a7740 2019-12-05T05:52:56.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:56.011+0000 7f481e27a700 1 --2- 172.21.15.42:0/2416688281 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f48180a7740 0x7f481808df80 secure :-1 s=THROTTLE_DONE pgs=361 cs=0 l=1 rx=0x7f480c001910 tx=0x7f4810001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:52:56.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:56.011+0000 7f481d278700 1 -- 172.21.15.42:0/2416688281 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f480c0014e0 con 0x7f48180a7740 2019-12-05T05:52:56.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:56.011+0000 7f481d278700 1 -- 172.21.15.42:0/2416688281 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f48140118d0 con 0x7f48180a7740 2019-12-05T05:52:56.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:56.011+0000 7f481fcdd700 1 -- 172.21.15.42:0/2416688281 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f48180a7740 msgr2=0x7f481808df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:52:56.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:56.011+0000 7f481fcdd700 1 --2- 172.21.15.42:0/2416688281 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f48180a7740 0x7f481808df80 secure :-1 s=READY pgs=361 cs=0 l=1 rx=0x7f480c001910 tx=0x7f4810001150).stop 2019-12-05T05:52:56.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:56.011+0000 7f481fcdd700 1 -- 172.21.15.42:0/2416688281 shutdown_connections 2019-12-05T05:52:56.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:56.011+0000 7f481fcdd700 1 --2- 172.21.15.42:0/2416688281 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f48180a7740 0x7f481808df80 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:52:56.024 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:56.011+0000 7f481fcdd700 1 -- 172.21.15.42:0/2416688281 shutdown_connections 2019-12-05T05:52:56.024 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:56.011+0000 7f481fcdd700 1 -- 172.21.15.42:0/2416688281 wait complete. 2019-12-05T05:52:56.024 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:56.011+0000 7f481fcdd700 1 -- 172.21.15.42:0/2416688281 >> 172.21.15.42:0/2416688281 conn(0x7f48180aabe0 msgr2=0x7f48180a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:52:56.026 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:56.015+0000 7f481fcdd700 1 Processor -- start 2019-12-05T05:52:56.026 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:56.015+0000 7f481fcdd700 1 -- start start 2019-12-05T05:52:56.027 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:56.019+0000 7f481fcdd700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4818117f80 0x7f4818130bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:52:56.027 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:56.019+0000 7f481fcdd700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f4818131140 0x7f4818131570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:52:56.027 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:56.019+0000 7f481fcdd700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f481807b7c0 con 0x7f4818123cd0 2019-12-05T05:52:56.027 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:56.019+0000 7f481fcdd700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f481811c270 con 0x7f4818131140 2019-12-05T05:52:56.027 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:56.019+0000 7f481fcdd700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48181183b0 con 0x7f4818117f80 2019-12-05T05:52:56.027 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:56.019+0000 7f481e27a700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f4818123cd0 0x7f481811a550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:43296/0 (socket says 172.21.15.42:43296) 2019-12-05T05:52:56.028 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:56.019+0000 7f481e27a700 1 -- 172.21.15.42:0/3551750651 learned_addr learned my addr 172.21.15.42:0/3551750651 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:52:56.028 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:56.019+0000 7f481ea7b700 1 --2- 172.21.15.42:0/3551750651 >> v2:172.21.15.42:3300/0 conn(0x7f4818131140 0x7f4818131570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:52:56.028 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:56.019+0000 7f481da79700 1 --2- 172.21.15.42:0/3551750651 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4818117f80 0x7f4818130bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:52:56.028 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:56.019+0000 7f480affd700 1 -- 172.21.15.42:0/3551750651 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 348931364 0 0) 0x7f481807b7c0 con 0x7f4818123cd0 2019-12-05T05:52:56.028 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:56.019+0000 7f480affd700 1 -- 172.21.15.42:0/3551750651 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48000030b0 con 0x7f4818123cd0 2019-12-05T05:52:56.028 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:56.019+0000 7f481ea7b700 1 -- 172.21.15.42:0/3551750651 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4818117f80 msgr2=0x7f4818130bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:52:56.028 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:56.019+0000 7f481ea7b700 1 --2- 172.21.15.42:0/3551750651 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4818117f80 0x7f4818130bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:52:56.029 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:56.019+0000 7f481ea7b700 1 -- 172.21.15.42:0/3551750651 >> v1:172.21.15.42:6789/0 conn(0x7f4818123cd0 legacy=0x7f481811a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:52:56.029 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:56.019+0000 7f481ea7b700 1 -- 172.21.15.42:0/3551750651 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4818118c60 con 0x7f4818131140 2019-12-05T05:52:56.029 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:56.019+0000 7f481ea7b700 1 --2- 172.21.15.42:0/3551750651 >> v2:172.21.15.42:3300/0 conn(0x7f4818131140 0x7f4818131570 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f4810000b80 tx=0x7f4810010830).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:52:56.029 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:56.019+0000 7f481ea7b700 1 --2- 172.21.15.42:0/3551750651 >> v2:172.21.15.42:3300/0 conn(0x7f4818131140 0x7f4818131570 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f4810000b80 tx=0x7f4810010830).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:52:56.029 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:56.019+0000 7f481ea7b700 1 --2- 172.21.15.42:0/3551750651 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4818131140 0x7f4818131570 secure :-1 s=READY pgs=1630 cs=0 l=1 rx=0x7f4810000b80 tx=0x7f4810010830).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:52:56.029 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:56.019+0000 7f481ea7b700 1 --2- 172.21.15.42:0/3551750651 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4818131140 0x7f4818131570 secure :-1 s=THROTTLE_DONE pgs=1630 cs=0 l=1 rx=0x7f4810000b80 tx=0x7f4810010830).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:52:56.029 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:56.019+0000 7f480affd700 1 -- 172.21.15.42:0/3551750651 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4810006a40 con 0x7f4818131140 2019-12-05T05:52:56.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:56.019+0000 7f481ea7b700 1 --2- 172.21.15.42:0/3551750651 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4818131140 0x7f4818131570 secure :-1 s=THROTTLE_DONE pgs=1630 cs=0 l=1 rx=0x7f4810000b80 tx=0x7f4810010830).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:52:56.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:56.019+0000 7f481ea7b700 1 --2- 172.21.15.42:0/3551750651 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4818131140 0x7f4818131570 secure :-1 s=THROTTLE_DONE pgs=1630 cs=0 l=1 rx=0x7f4810000b80 tx=0x7f4810010830).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:52:56.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:56.019+0000 7f480affd700 1 -- 172.21.15.42:0/3551750651 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4810006660 con 0x7f4818131140 2019-12-05T05:52:56.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:56.019+0000 7f481fcdd700 1 -- 172.21.15.42:0/3551750651 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f481811ad50 con 0x7f4818131140 2019-12-05T05:52:56.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:56.019+0000 7f480affd700 1 -- 172.21.15.42:0/3551750651 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4810011070 con 0x7f4818131140 2019-12-05T05:52:56.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:56.019+0000 7f481fcdd700 1 -- 172.21.15.42:0/3551750651 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4818119740 con 0x7f4818131140 2019-12-05T05:52:56.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:56.019+0000 7f481ea7b700 1 --2- 172.21.15.42:0/3551750651 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4818131140 0x7f4818131570 secure :-1 s=THROTTLE_DONE pgs=1630 cs=0 l=1 rx=0x7f4810000b80 tx=0x7f4810010830).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:52:56.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:56.019+0000 7f480affd700 1 -- 172.21.15.42:0/3551750651 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f4810011380 con 0x7f4818131140 2019-12-05T05:52:56.032 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:56.019+0000 7f481ea7b700 1 --2- 172.21.15.42:0/3551750651 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4818131140 0x7f4818131570 secure :-1 s=THROTTLE_DONE pgs=1630 cs=0 l=1 rx=0x7f4810000b80 tx=0x7f4810010830).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:52:56.032 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:56.023+0000 7f480affd700 1 --2- 172.21.15.42:0/3551750651 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f480003b2d0 0x7f480003d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:52:56.032 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:56.023+0000 7f480affd700 1 -- 172.21.15.42:0/3551750651 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8869+0+0 (secure 0 0 0) 0x7f481004b040 con 0x7f4818131140 2019-12-05T05:52:56.032 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:56.023+0000 7f481e27a700 1 --2- 172.21.15.42:0/3551750651 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f480003b2d0 0x7f480003d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:52:56.032 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:56.023+0000 7f481e27a700 1 --2- 172.21.15.42:0/3551750651 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f480003b2d0 0x7f480003d7e0 crc :-1 s=READY pgs=697 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:52:56.033 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:56.023+0000 7f481fcdd700 1 -- 172.21.15.42:0/3551750651 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f481805d110 con 0x7f4818131140 2019-12-05T05:52:56.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:56.031+0000 7f481ea7b700 1 --2- 172.21.15.42:0/3551750651 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4818131140 0x7f4818131570 secure :-1 s=THROTTLE_DONE pgs=1630 cs=0 l=1 rx=0x7f4810000b80 tx=0x7f4810010830).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:52:56.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:56.031+0000 7f480affd700 1 -- 172.21.15.42:0/3551750651 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f4810032000 con 0x7f4818131140 2019-12-05T05:52:56.205 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:56.199+0000 7f481fcdd700 1 -- 172.21.15.42:0/3551750651 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f481805b8c0 con 0x7f4818131140 2019-12-05T05:52:56.205 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:56.199+0000 7f481ea7b700 1 --2- 172.21.15.42:0/3551750651 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4818131140 0x7f4818131570 secure :-1 s=THROTTLE_DONE pgs=1630 cs=0 l=1 rx=0x7f4810000b80 tx=0x7f4810010830).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:52:56.205 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:56.199+0000 7f480affd700 1 -- 172.21.15.42:0/3551750651 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 198 v198) v1 ==== 95+0+2102 (secure 0 0 0) 0x7f4810032260 con 0x7f4818131140 2019-12-05T05:52:56.206 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 198 2019-12-05T05:52:56.206 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:52:56.206 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":198,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6356,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/1523046776","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1523046776},{"type":"v1","addr":"172.21.15.40:6833","nonce":1523046776}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":198}],"filesystems":[{"mdsmap":{"epoch":197,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:51:24.172910+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8430},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8430":{"gid":8430,"name":"a","rank":0,"incarnation":194,"state":"up:active","state_seq":9,"addr":"172.21.15.42:6835/5991897","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":5991897},{"type":"v1","addr":"172.21.15.42:6835","nonce":5991897}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:52:56.208 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:56.203+0000 7f481fcdd700 1 -- 172.21.15.42:0/3551750651 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f480003b2d0 msgr2=0x7f480003d7e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:52:56.208 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:56.203+0000 7f481fcdd700 1 --2- 172.21.15.42:0/3551750651 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f480003b2d0 0x7f480003d7e0 crc :-1 s=READY pgs=697 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:52:56.209 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:56.203+0000 7f481fcdd700 1 -- 172.21.15.42:0/3551750651 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4818131140 msgr2=0x7f4818131570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:52:56.209 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:56.203+0000 7f481fcdd700 1 --2- 172.21.15.42:0/3551750651 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4818131140 0x7f4818131570 secure :-1 s=READY pgs=1630 cs=0 l=1 rx=0x7f4810000b80 tx=0x7f4810010830).stop 2019-12-05T05:52:56.209 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:56.203+0000 7f481fcdd700 1 -- 172.21.15.42:0/3551750651 shutdown_connections 2019-12-05T05:52:56.209 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:56.203+0000 7f481fcdd700 1 --2- 172.21.15.42:0/3551750651 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f480003b2d0 0x7f480003d7e0 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:52:56.209 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:56.203+0000 7f481fcdd700 1 --2- 172.21.15.42:0/3551750651 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4818131140 0x7f4818131570 unknown :-1 s=CLOSED pgs=1630 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:52:56.209 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:56.203+0000 7f481fcdd700 1 --2- 172.21.15.42:0/3551750651 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4818117f80 0x7f4818130bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:52:56.209 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:56.203+0000 7f481fcdd700 1 -- 172.21.15.42:0/3551750651 shutdown_connections 2019-12-05T05:52:56.210 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:56.203+0000 7f481fcdd700 1 -- 172.21.15.42:0/3551750651 wait complete. 2019-12-05T05:52:56.210 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:56.203+0000 7f481fcdd700 1 -- 172.21.15.42:0/3551750651 >> 172.21.15.42:0/3551750651 conn(0x7f481811c540 msgr2=0x7f481809f2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:52:56.218 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T05:52:58.219 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:52:58.328 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:58.315+0000 7f0031454700 1 Processor -- start 2019-12-05T05:52:58.328 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:58.315+0000 7f0031454700 1 -- start start 2019-12-05T05:52:58.328 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:58.319+0000 7f0031454700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f002c0a7740 0x7f002c08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:52:58.329 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:58.319+0000 7f0031454700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f002c07e720 0x7f002c07eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:52:58.329 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:58.319+0000 7f0031454700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f002c0a0710 con 0x7f002c09f2e0 2019-12-05T05:52:58.329 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:58.319+0000 7f0031454700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f002c0a0b00 con 0x7f002c0a7740 2019-12-05T05:52:58.329 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:58.319+0000 7f0031454700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f002c08e5b0 con 0x7f002c07e720 2019-12-05T05:52:58.329 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:58.319+0000 7f002affd700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f002c07e720 0x7f002c07eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:52:58.329 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:58.319+0000 7f002b7fe700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f002c0a7740 0x7f002c08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:52:58.330 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:58.319+0000 7f002affd700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f002c07e720 0x7f002c07eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:43734/0 (socket says 172.21.15.42:43734) 2019-12-05T05:52:58.330 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:58.319+0000 7f002b7fe700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f002c0a7740 0x7f002c08df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.42:59314/0 (socket says 172.21.15.42:59314) 2019-12-05T05:52:58.330 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:58.319+0000 7f002affd700 1 -- 172.21.15.42:0/619962859 learned_addr learned my addr 172.21.15.42:0/619962859 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:52:58.330 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:58.319+0000 7f002affd700 1 -- 172.21.15.42:0/619962859 >> v2:172.21.15.40:3300/0 conn(0x7f002c0a7740 msgr2=0x7f002c08df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:52:58.330 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:58.319+0000 7f002affd700 1 --2- 172.21.15.42:0/619962859 >> v2:172.21.15.40:3300/0 conn(0x7f002c0a7740 0x7f002c08df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:52:58.330 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:58.319+0000 7f002affd700 1 -- 172.21.15.42:0/619962859 >> v1:172.21.15.40:6789/0 conn(0x7f002c09f2e0 legacy=0x7f002c09f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:52:58.330 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:58.319+0000 7f002affd700 1 -- 172.21.15.42:0/619962859 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f002c07f0a0 con 0x7f002c07e720 2019-12-05T05:52:58.331 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:58.319+0000 7f002b7fe700 1 --2- 172.21.15.42:0/619962859 >> v2:172.21.15.40:3300/0 conn(0x7f002c0a7740 0x7f002c08df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:52:58.331 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:58.319+0000 7f002affd700 1 --2- 172.21.15.42:0/619962859 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f002c07e720 0x7f002c07eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f0020008790 tx=0x7f002000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:52:58.331 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:58.319+0000 7f002affd700 1 --2- 172.21.15.42:0/619962859 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f002c07e720 0x7f002c07eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f0020008790 tx=0x7f002000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:52:58.332 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:58.319+0000 7f002affd700 1 --2- 172.21.15.42:0/619962859 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f002c07e720 0x7f002c07eb50 secure :-1 s=READY pgs=362 cs=0 l=1 rx=0x7f0020008790 tx=0x7f002000fd30).ready entity=mon.2 client_cookie=8bbbf5a8130c5570 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:52:58.332 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:58.319+0000 7f002affd700 1 --2- 172.21.15.42:0/619962859 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f002c07e720 0x7f002c07eb50 secure :-1 s=THROTTLE_DONE pgs=362 cs=0 l=1 rx=0x7f0020008790 tx=0x7f002000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:52:58.332 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:58.319+0000 7f002affd700 1 --2- 172.21.15.42:0/619962859 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f002c07e720 0x7f002c07eb50 secure :-1 s=THROTTLE_DONE pgs=362 cs=0 l=1 rx=0x7f0020008790 tx=0x7f002000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:52:58.332 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:58.319+0000 7f002a7fc700 1 -- 172.21.15.42:0/619962859 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f00200113b0 con 0x7f002c07e720 2019-12-05T05:52:58.332 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:58.319+0000 7f002affd700 1 --2- 172.21.15.42:0/619962859 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f002c07e720 0x7f002c07eb50 secure :-1 s=THROTTLE_DONE pgs=362 cs=0 l=1 rx=0x7f0020008790 tx=0x7f002000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:52:58.332 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:58.319+0000 7f002a7fc700 1 -- 172.21.15.42:0/619962859 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0020008bb0 con 0x7f002c07e720 2019-12-05T05:52:58.333 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:58.319+0000 7f002a7fc700 1 -- 172.21.15.42:0/619962859 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0020011990 con 0x7f002c07e720 2019-12-05T05:52:58.333 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:58.319+0000 7f0031454700 1 -- 172.21.15.42:0/619962859 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f002c07e720 msgr2=0x7f002c07eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:52:58.333 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:58.319+0000 7f0031454700 1 --2- 172.21.15.42:0/619962859 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f002c07e720 0x7f002c07eb50 secure :-1 s=READY pgs=362 cs=0 l=1 rx=0x7f0020008790 tx=0x7f002000fd30).stop 2019-12-05T05:52:58.333 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:58.323+0000 7f0031454700 1 -- 172.21.15.42:0/619962859 shutdown_connections 2019-12-05T05:52:58.333 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:58.323+0000 7f0031454700 1 --2- 172.21.15.42:0/619962859 >> v2:172.21.15.40:3300/0 conn(0x7f002c0a7740 0x7f002c08df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:52:58.333 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:58.323+0000 7f0031454700 1 --2- 172.21.15.42:0/619962859 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f002c07e720 0x7f002c07eb50 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:52:58.333 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:58.323+0000 7f0031454700 1 -- 172.21.15.42:0/619962859 shutdown_connections 2019-12-05T05:52:58.334 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:58.323+0000 7f0031454700 1 -- 172.21.15.42:0/619962859 wait complete. 2019-12-05T05:52:58.334 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:58.323+0000 7f0031454700 1 -- 172.21.15.42:0/619962859 >> 172.21.15.42:0/619962859 conn(0x7f002c0aabe0 msgr2=0x7f002c0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:52:58.337 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:58.323+0000 7f0031454700 1 Processor -- start 2019-12-05T05:52:58.337 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:58.323+0000 7f0031454700 1 -- start start 2019-12-05T05:52:58.337 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:58.323+0000 7f0031454700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f002c11bf30 0x7f002c07ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:52:58.337 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:58.323+0000 7f0031454700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f002c0a56a0 0x7f002c137a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:52:58.337 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:58.323+0000 7f0031454700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f002c0a1510 con 0x7f002c0a6860 2019-12-05T05:52:58.337 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:58.323+0000 7f0031454700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f002c0a1890 con 0x7f002c0a56a0 2019-12-05T05:52:58.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:58.323+0000 7f0031454700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f002c0a5400 con 0x7f002c11bf30 2019-12-05T05:52:58.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:58.323+0000 7f002bfff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f002c0a56a0 0x7f002c137a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:52:58.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:58.323+0000 7f002b7fe700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f002c11bf30 0x7f002c07ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:52:58.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:58.327+0000 7f002b7fe700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f002c11bf30 0x7f002c07ced0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:43738/0 (socket says 172.21.15.42:43738) 2019-12-05T05:52:58.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:58.327+0000 7f002bfff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f002c0a56a0 0x7f002c137a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:43274/0 (socket says 172.21.15.42:43274) 2019-12-05T05:52:58.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:58.327+0000 7f002b7fe700 1 -- 172.21.15.42:0/526508116 learned_addr learned my addr 172.21.15.42:0/526508116 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:52:58.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:58.327+0000 7f002b7fe700 1 -- 172.21.15.42:0/526508116 >> v2:172.21.15.42:3300/0 conn(0x7f002c0a56a0 msgr2=0x7f002c137a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:52:58.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:58.327+0000 7f002b7fe700 1 --2- 172.21.15.42:0/526508116 >> v2:172.21.15.42:3300/0 conn(0x7f002c0a56a0 0x7f002c137a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:52:58.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:58.327+0000 7f002b7fe700 1 -- 172.21.15.42:0/526508116 >> v1:172.21.15.42:6789/0 conn(0x7f002c0a6860 legacy=0x7f002c07af50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:52:58.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:58.327+0000 7f002b7fe700 1 -- 172.21.15.42:0/526508116 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f002c07b750 con 0x7f002c11bf30 2019-12-05T05:52:58.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:58.327+0000 7f002b7fe700 1 --2- 172.21.15.42:0/526508116 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f002c11bf30 0x7f002c07ced0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f001c002ef0 tx=0x7f001c010d30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:52:58.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:58.327+0000 7f002b7fe700 1 --2- 172.21.15.42:0/526508116 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f002c11bf30 0x7f002c07ced0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f001c002ef0 tx=0x7f001c010d30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:52:58.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:58.327+0000 7f002b7fe700 1 --2- 172.21.15.42:0/526508116 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f002c11bf30 0x7f002c07ced0 secure :-1 s=READY pgs=363 cs=0 l=1 rx=0x7f001c002ef0 tx=0x7f001c010d30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:52:58.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:58.327+0000 7f002b7fe700 1 --2- 172.21.15.42:0/526508116 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f002c11bf30 0x7f002c07ced0 secure :-1 s=THROTTLE_DONE pgs=363 cs=0 l=1 rx=0x7f001c002ef0 tx=0x7f001c010d30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:52:58.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:58.327+0000 7f002b7fe700 1 --2- 172.21.15.42:0/526508116 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f002c11bf30 0x7f002c07ced0 secure :-1 s=THROTTLE_DONE pgs=363 cs=0 l=1 rx=0x7f001c002ef0 tx=0x7f001c010d30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:52:58.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:58.327+0000 7f0013fff700 1 -- 172.21.15.42:0/526508116 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f001c0126b0 con 0x7f002c11bf30 2019-12-05T05:52:58.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:58.327+0000 7f002b7fe700 1 --2- 172.21.15.42:0/526508116 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f002c11bf30 0x7f002c07ced0 secure :-1 s=THROTTLE_DONE pgs=363 cs=0 l=1 rx=0x7f001c002ef0 tx=0x7f001c010d30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:52:58.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:58.327+0000 7f0013fff700 1 -- 172.21.15.42:0/526508116 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f001c007150 con 0x7f002c11bf30 2019-12-05T05:52:58.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:58.327+0000 7f0031454700 1 -- 172.21.15.42:0/526508116 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f002c07d420 con 0x7f002c11bf30 2019-12-05T05:52:58.341 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:58.327+0000 7f0013fff700 1 -- 172.21.15.42:0/526508116 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f001c012c90 con 0x7f002c11bf30 2019-12-05T05:52:58.341 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:58.327+0000 7f0031454700 1 -- 172.21.15.42:0/526508116 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f002c0a4f80 con 0x7f002c11bf30 2019-12-05T05:52:58.341 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:58.327+0000 7f002b7fe700 1 --2- 172.21.15.42:0/526508116 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f002c11bf30 0x7f002c07ced0 secure :-1 s=THROTTLE_DONE pgs=363 cs=0 l=1 rx=0x7f001c002ef0 tx=0x7f001c010d30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:52:58.341 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:58.327+0000 7f0013fff700 1 -- 172.21.15.42:0/526508116 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f001c005540 con 0x7f002c11bf30 2019-12-05T05:52:58.341 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:58.327+0000 7f002b7fe700 1 --2- 172.21.15.42:0/526508116 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f002c11bf30 0x7f002c07ced0 secure :-1 s=THROTTLE_DONE pgs=363 cs=0 l=1 rx=0x7f001c002ef0 tx=0x7f001c010d30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:52:58.342 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:58.331+0000 7f0013fff700 1 --2- 172.21.15.42:0/526508116 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f000c03adf0 0x7f000c03d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:52:58.342 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:58.331+0000 7f0013fff700 1 -- 172.21.15.42:0/526508116 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8869+0+0 (secure 0 0 0) 0x7f001c04b920 con 0x7f002c11bf30 2019-12-05T05:52:58.342 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:58.331+0000 7f002affd700 1 --2- 172.21.15.42:0/526508116 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f000c03adf0 0x7f000c03d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:52:58.342 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:58.331+0000 7f002affd700 1 --2- 172.21.15.42:0/526508116 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f000c03adf0 0x7f000c03d300 crc :-1 s=READY pgs=698 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:52:58.342 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:58.331+0000 7f0031454700 1 -- 172.21.15.42:0/526508116 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f002c05d110 con 0x7f002c11bf30 2019-12-05T05:52:58.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:58.343+0000 7f002b7fe700 1 --2- 172.21.15.42:0/526508116 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f002c11bf30 0x7f002c07ced0 secure :-1 s=THROTTLE_DONE pgs=363 cs=0 l=1 rx=0x7f001c002ef0 tx=0x7f001c010d30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:52:58.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:58.343+0000 7f0013fff700 1 -- 172.21.15.42:0/526508116 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f001c032c00 con 0x7f002c11bf30 2019-12-05T05:52:58.515 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:58.507+0000 7f0031454700 1 -- 172.21.15.42:0/526508116 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f002c05b8c0 con 0x7f002c11bf30 2019-12-05T05:52:58.516 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:58.511+0000 7f002b7fe700 1 --2- 172.21.15.42:0/526508116 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f002c11bf30 0x7f002c07ced0 secure :-1 s=THROTTLE_DONE pgs=363 cs=0 l=1 rx=0x7f001c002ef0 tx=0x7f001c010d30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:52:58.516 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:58.511+0000 7f0013fff700 1 -- 172.21.15.42:0/526508116 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 198 v198) v1 ==== 95+0+2102 (secure 0 0 0) 0x7f001c02ebd0 con 0x7f002c11bf30 2019-12-05T05:52:58.518 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:52:58.519 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":198,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6356,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/1523046776","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1523046776},{"type":"v1","addr":"172.21.15.40:6833","nonce":1523046776}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":198}],"filesystems":[{"mdsmap":{"epoch":197,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:51:24.172910+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8430},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8430":{"gid":8430,"name":"a","rank":0,"incarnation":194,"state":"up:active","state_seq":9,"addr":"172.21.15.42:6835/5991897","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":5991897},{"type":"v1","addr":"172.21.15.42:6835","nonce":5991897}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:52:58.519 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 198 2019-12-05T05:52:58.521 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:58.515+0000 7f0031454700 1 -- 172.21.15.42:0/526508116 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f000c03adf0 msgr2=0x7f000c03d300 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:52:58.521 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:58.515+0000 7f0031454700 1 --2- 172.21.15.42:0/526508116 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f000c03adf0 0x7f000c03d300 crc :-1 s=READY pgs=698 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:52:58.521 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:58.515+0000 7f0031454700 1 -- 172.21.15.42:0/526508116 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f002c11bf30 msgr2=0x7f002c07ced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:52:58.521 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:58.515+0000 7f0031454700 1 --2- 172.21.15.42:0/526508116 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f002c11bf30 0x7f002c07ced0 secure :-1 s=READY pgs=363 cs=0 l=1 rx=0x7f001c002ef0 tx=0x7f001c010d30).stop 2019-12-05T05:52:58.521 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:58.515+0000 7f0031454700 1 -- 172.21.15.42:0/526508116 shutdown_connections 2019-12-05T05:52:58.522 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:58.515+0000 7f0031454700 1 --2- 172.21.15.42:0/526508116 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f000c03adf0 0x7f000c03d300 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:52:58.522 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:58.515+0000 7f0031454700 1 --2- 172.21.15.42:0/526508116 >> v2:172.21.15.42:3300/0 conn(0x7f002c0a56a0 0x7f002c137a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:52:58.522 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:58.515+0000 7f0031454700 1 --2- 172.21.15.42:0/526508116 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f002c11bf30 0x7f002c07ced0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:52:58.522 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:58.515+0000 7f0031454700 1 -- 172.21.15.42:0/526508116 shutdown_connections 2019-12-05T05:52:58.522 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:58.515+0000 7f0031454700 1 -- 172.21.15.42:0/526508116 wait complete. 2019-12-05T05:52:58.522 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:52:58.515+0000 7f0031454700 1 -- 172.21.15.42:0/526508116 >> 172.21.15.42:0/526508116 conn(0x7f002c0a6d70 msgr2=0x7f002c059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:52:58.531 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T05:53:00.534 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:53:00.646 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.627+0000 7f7564cb9700 1 Processor -- start 2019-12-05T05:53:00.647 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.627+0000 7f7564cb9700 1 -- start start 2019-12-05T05:53:00.647 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.631+0000 7f7564cb9700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f75600a7740 0x7f756008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:53:00.647 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.631+0000 7f7564cb9700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f756007e720 0x7f756007eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:53:00.647 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.631+0000 7f7564cb9700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75600a0710 con 0x7f756009f2e0 2019-12-05T05:53:00.647 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.631+0000 7f7564cb9700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75600a0b00 con 0x7f75600a7740 2019-12-05T05:53:00.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.631+0000 7f7564cb9700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f756008e5b0 con 0x7f756007e720 2019-12-05T05:53:00.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.631+0000 7f755ed9d700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f75600a7740 0x7f756008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:53:00.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.631+0000 7f755f59e700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f756009f2e0 0x7f756009f710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:43322/0 (socket says 172.21.15.42:43322) 2019-12-05T05:53:00.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.631+0000 7f755f59e700 1 -- 172.21.15.42:0/4268888023 learned_addr learned my addr 172.21.15.42:0/4268888023 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:53:00.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.631+0000 7f755e59c700 1 --2- 172.21.15.42:0/4268888023 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f756007e720 0x7f756007eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:53:00.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.631+0000 7f755dd9b700 1 -- 172.21.15.42:0/4268888023 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3167310565 0 0) 0x7f75600a0710 con 0x7f756009f2e0 2019-12-05T05:53:00.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.631+0000 7f755ed9d700 1 -- 172.21.15.42:0/4268888023 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f756007e720 msgr2=0x7f756007eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:53:00.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.631+0000 7f755ed9d700 1 --2- 172.21.15.42:0/4268888023 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f756007e720 0x7f756007eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:53:00.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.631+0000 7f755ed9d700 1 -- 172.21.15.42:0/4268888023 >> v1:172.21.15.42:6789/0 conn(0x7f756009f2e0 legacy=0x7f756009f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:53:00.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.631+0000 7f755ed9d700 1 -- 172.21.15.42:0/4268888023 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f756007f0a0 con 0x7f75600a7740 2019-12-05T05:53:00.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.631+0000 7f755e59c700 1 --2- 172.21.15.42:0/4268888023 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f756007e720 0x7f756007eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:53:00.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.631+0000 7f755ed9d700 1 --2- 172.21.15.42:0/4268888023 >> v2:172.21.15.40:3300/0 conn(0x7f75600a7740 0x7f756008df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f7550008790 tx=0x7f755000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:53:00.650 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.631+0000 7f755ed9d700 1 --2- 172.21.15.42:0/4268888023 >> v2:172.21.15.40:3300/0 conn(0x7f75600a7740 0x7f756008df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f7550008790 tx=0x7f755000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:53:00.650 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.631+0000 7f755ed9d700 1 --2- 172.21.15.42:0/4268888023 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f75600a7740 0x7f756008df80 secure :-1 s=READY pgs=1253 cs=0 l=1 rx=0x7f7550008790 tx=0x7f755000fc70).ready entity=mon.0 client_cookie=60844f9a9afcbd68 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:53:00.655 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.631+0000 7f755ed9d700 1 --2- 172.21.15.42:0/4268888023 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f75600a7740 0x7f756008df80 secure :-1 s=THROTTLE_DONE pgs=1253 cs=0 l=1 rx=0x7f7550008790 tx=0x7f755000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:53:00.658 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.631+0000 7f755ed9d700 1 --2- 172.21.15.42:0/4268888023 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f75600a7740 0x7f756008df80 secure :-1 s=THROTTLE_DONE pgs=1253 cs=0 l=1 rx=0x7f7550008790 tx=0x7f755000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:53:00.658 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.631+0000 7f755dd9b700 1 -- 172.21.15.42:0/4268888023 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f75500112f0 con 0x7f75600a7740 2019-12-05T05:53:00.658 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.631+0000 7f755ed9d700 1 --2- 172.21.15.42:0/4268888023 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f75600a7740 0x7f756008df80 secure :-1 s=THROTTLE_DONE pgs=1253 cs=0 l=1 rx=0x7f7550008790 tx=0x7f755000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:53:00.658 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.631+0000 7f755dd9b700 1 -- 172.21.15.42:0/4268888023 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7550008ad0 con 0x7f75600a7740 2019-12-05T05:53:00.658 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.631+0000 7f755dd9b700 1 -- 172.21.15.42:0/4268888023 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f75500118d0 con 0x7f75600a7740 2019-12-05T05:53:00.658 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.631+0000 7f7564cb9700 1 -- 172.21.15.42:0/4268888023 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f75600a7740 msgr2=0x7f756008df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:53:00.659 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.631+0000 7f7564cb9700 1 --2- 172.21.15.42:0/4268888023 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f75600a7740 0x7f756008df80 secure :-1 s=READY pgs=1253 cs=0 l=1 rx=0x7f7550008790 tx=0x7f755000fc70).stop 2019-12-05T05:53:00.659 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.631+0000 7f7564cb9700 1 -- 172.21.15.42:0/4268888023 shutdown_connections 2019-12-05T05:53:00.659 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.631+0000 7f7564cb9700 1 --2- 172.21.15.42:0/4268888023 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f75600a7740 0x7f756008df80 unknown :-1 s=CLOSED pgs=1253 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:53:00.659 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.631+0000 7f7564cb9700 1 --2- 172.21.15.42:0/4268888023 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f756007e720 0x7f756007eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:53:00.659 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.631+0000 7f7564cb9700 1 -- 172.21.15.42:0/4268888023 shutdown_connections 2019-12-05T05:53:00.659 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.631+0000 7f7564cb9700 1 -- 172.21.15.42:0/4268888023 wait complete. 2019-12-05T05:53:00.659 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.631+0000 7f7564cb9700 1 -- 172.21.15.42:0/4268888023 >> 172.21.15.42:0/4268888023 conn(0x7f75600aabe0 msgr2=0x7f75600a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:53:00.660 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.635+0000 7f7564cb9700 1 Processor -- start 2019-12-05T05:53:00.660 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.635+0000 7f7564cb9700 1 -- start start 2019-12-05T05:53:00.660 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.639+0000 7f7564cb9700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f75601243f0 0x7f756011eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:53:00.660 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.639+0000 7f7564cb9700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7560120f10 con 0x7f756011fd40 2019-12-05T05:53:00.660 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.639+0000 7f7564cb9700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7560121380 con 0x7f7560133940 2019-12-05T05:53:00.660 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.639+0000 7f7564cb9700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7560119560 con 0x7f75601243f0 2019-12-05T05:53:00.660 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.639+0000 7f755f59e700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f7560133940 0x7f7560133d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:43328/0 (socket says 172.21.15.42:43328) 2019-12-05T05:53:00.661 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.639+0000 7f755ed9d700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f75601243f0 0x7f756011eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:53:00.661 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.639+0000 7f755f59e700 1 -- 172.21.15.42:0/2026683958 learned_addr learned my addr 172.21.15.42:0/2026683958 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:53:00.661 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.639+0000 7f754f7fe700 1 -- 172.21.15.42:0/2026683958 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3559872466 0 0) 0x7f7560121380 con 0x7f7560133940 2019-12-05T05:53:00.661 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.639+0000 7f754f7fe700 1 -- 172.21.15.42:0/2026683958 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75440030b0 con 0x7f7560133940 2019-12-05T05:53:00.661 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.639+0000 7f755ed9d700 1 -- 172.21.15.42:0/2026683958 >> v1:172.21.15.42:6789/0 conn(0x7f7560133940 legacy=0x7f7560133d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:53:00.661 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.639+0000 7f755ed9d700 1 -- 172.21.15.42:0/2026683958 >> v1:172.21.15.40:6789/0 conn(0x7f756011fd40 legacy=0x7f7560120170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:53:00.661 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.639+0000 7f755ed9d700 1 -- 172.21.15.42:0/2026683958 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f756007a330 con 0x7f75601243f0 2019-12-05T05:53:00.662 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.639+0000 7f755ed9d700 1 --2- 172.21.15.42:0/2026683958 >> v2:172.21.15.40:3300/0 conn(0x7f75601243f0 0x7f756011eab0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f756011f280 tx=0x7f7550011990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:53:00.665 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.639+0000 7f755ed9d700 1 --2- 172.21.15.42:0/2026683958 >> v2:172.21.15.40:3300/0 conn(0x7f75601243f0 0x7f756011eab0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f756011f280 tx=0x7f7550011990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:53:00.665 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.639+0000 7f755ed9d700 1 --2- 172.21.15.42:0/2026683958 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f75601243f0 0x7f756011eab0 secure :-1 s=READY pgs=1255 cs=0 l=1 rx=0x7f756011f280 tx=0x7f7550011990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:53:00.665 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.639+0000 7f755ed9d700 1 --2- 172.21.15.42:0/2026683958 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f75601243f0 0x7f756011eab0 secure :-1 s=THROTTLE_DONE pgs=1255 cs=0 l=1 rx=0x7f756011f280 tx=0x7f7550011990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:53:00.665 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.639+0000 7f754f7fe700 1 -- 172.21.15.42:0/2026683958 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7550007150 con 0x7f75601243f0 2019-12-05T05:53:00.665 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.639+0000 7f755ed9d700 1 --2- 172.21.15.42:0/2026683958 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f75601243f0 0x7f756011eab0 secure :-1 s=THROTTLE_DONE pgs=1255 cs=0 l=1 rx=0x7f756011f280 tx=0x7f7550011990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:53:00.665 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.639+0000 7f755ed9d700 1 --2- 172.21.15.42:0/2026683958 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f75601243f0 0x7f756011eab0 secure :-1 s=THROTTLE_DONE pgs=1255 cs=0 l=1 rx=0x7f756011f280 tx=0x7f7550011990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:53:00.666 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.639+0000 7f754f7fe700 1 -- 172.21.15.42:0/2026683958 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75500114f0 con 0x7f75601243f0 2019-12-05T05:53:00.666 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.639+0000 7f754f7fe700 1 -- 172.21.15.42:0/2026683958 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f755000f550 con 0x7f75601243f0 2019-12-05T05:53:00.666 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.639+0000 7f7564cb9700 1 -- 172.21.15.42:0/2026683958 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7560078310 con 0x7f75601243f0 2019-12-05T05:53:00.666 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.639+0000 7f7564cb9700 1 -- 172.21.15.42:0/2026683958 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7560078890 con 0x7f75601243f0 2019-12-05T05:53:00.666 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.639+0000 7f755ed9d700 1 --2- 172.21.15.42:0/2026683958 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f75601243f0 0x7f756011eab0 secure :-1 s=THROTTLE_DONE pgs=1255 cs=0 l=1 rx=0x7f756011f280 tx=0x7f7550011990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:53:00.667 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.639+0000 7f754f7fe700 1 -- 172.21.15.42:0/2026683958 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f7550005540 con 0x7f75601243f0 2019-12-05T05:53:00.667 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.639+0000 7f755ed9d700 1 --2- 172.21.15.42:0/2026683958 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f75601243f0 0x7f756011eab0 secure :-1 s=THROTTLE_DONE pgs=1255 cs=0 l=1 rx=0x7f756011f280 tx=0x7f7550011990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:53:00.667 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.639+0000 7f754f7fe700 1 --2- 172.21.15.42:0/2026683958 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f754403b2a0 0x7f754403d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:53:00.667 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.643+0000 7f754f7fe700 1 -- 172.21.15.42:0/2026683958 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8955+0+0 (secure 0 0 0) 0x7f755004b650 con 0x7f75601243f0 2019-12-05T05:53:00.667 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.643+0000 7f755e59c700 1 --2- 172.21.15.42:0/2026683958 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f754403b2a0 0x7f754403d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:53:00.668 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.643+0000 7f755e59c700 1 --2- 172.21.15.42:0/2026683958 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f754403b2a0 0x7f754403d7b0 crc :-1 s=READY pgs=699 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:53:00.668 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.643+0000 7f7564cb9700 1 -- 172.21.15.42:0/2026683958 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f756005d110 con 0x7f75601243f0 2019-12-05T05:53:00.669 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.651+0000 7f755ed9d700 1 --2- 172.21.15.42:0/2026683958 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f75601243f0 0x7f756011eab0 secure :-1 s=THROTTLE_DONE pgs=1255 cs=0 l=1 rx=0x7f756011f280 tx=0x7f7550011990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:53:00.669 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.651+0000 7f754f7fe700 1 -- 172.21.15.42:0/2026683958 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f7550007f50 con 0x7f75601243f0 2019-12-05T05:53:00.818 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.811+0000 7f7564cb9700 1 -- 172.21.15.42:0/2026683958 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f756005b8c0 con 0x7f75601243f0 2019-12-05T05:53:00.819 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.815+0000 7f755ed9d700 1 --2- 172.21.15.42:0/2026683958 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f75601243f0 0x7f756011eab0 secure :-1 s=THROTTLE_DONE pgs=1255 cs=0 l=1 rx=0x7f756011f280 tx=0x7f7550011990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:53:00.819 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.815+0000 7f754f7fe700 1 -- 172.21.15.42:0/2026683958 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 199 v199) v1 ==== 95+0+1774 (secure 0 0 0) 0x7f75500334e0 con 0x7f75601243f0 2019-12-05T05:53:00.821 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:53:00.822 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":199,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":199,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:52:59.156877+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6356},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6356":{"gid":6356,"name":"b","rank":0,"incarnation":199,"state":"up:replay","state_seq":1,"addr":"172.21.15.40:6833/1523046776","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1523046776},{"type":"v1","addr":"172.21.15.40:6833","nonce":1523046776}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:53:00.822 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 199 2019-12-05T05:53:00.824 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.819+0000 7f7564cb9700 1 -- 172.21.15.42:0/2026683958 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f754403b2a0 msgr2=0x7f754403d7b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:53:00.824 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.819+0000 7f7564cb9700 1 --2- 172.21.15.42:0/2026683958 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f754403b2a0 0x7f754403d7b0 crc :-1 s=READY pgs=699 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:53:00.824 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.819+0000 7f7564cb9700 1 -- 172.21.15.42:0/2026683958 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f75601243f0 msgr2=0x7f756011eab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:53:00.824 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.819+0000 7f7564cb9700 1 --2- 172.21.15.42:0/2026683958 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f75601243f0 0x7f756011eab0 secure :-1 s=READY pgs=1255 cs=0 l=1 rx=0x7f756011f280 tx=0x7f7550011990).stop 2019-12-05T05:53:00.825 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.819+0000 7f7564cb9700 1 -- 172.21.15.42:0/2026683958 shutdown_connections 2019-12-05T05:53:00.825 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.819+0000 7f7564cb9700 1 --2- 172.21.15.42:0/2026683958 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f754403b2a0 0x7f754403d7b0 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:53:00.825 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.819+0000 7f7564cb9700 1 --2- 172.21.15.42:0/2026683958 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f75601243f0 0x7f756011eab0 unknown :-1 s=CLOSED pgs=1255 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:53:00.825 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.819+0000 7f7564cb9700 1 -- 172.21.15.42:0/2026683958 shutdown_connections 2019-12-05T05:53:00.825 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.819+0000 7f7564cb9700 1 -- 172.21.15.42:0/2026683958 wait complete. 2019-12-05T05:53:00.825 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.819+0000 7f7564cb9700 1 -- 172.21.15.42:0/2026683958 >> 172.21.15.42:0/2026683958 conn(0x7f756007e5a0 msgr2=0x7f75600a9640 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:53:00.836 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a down, removed from mdsmap 2019-12-05T05:53:00.836 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2019-12-05T05:53:00.836 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:53:00.938 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.927+0000 7f2396742700 1 Processor -- start 2019-12-05T05:53:00.938 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.927+0000 7f2396742700 1 -- start start 2019-12-05T05:53:00.939 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.927+0000 7f2396742700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f23900a7740 0x7f239008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:53:00.939 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.927+0000 7f2396742700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f239007e720 0x7f239007eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:53:00.939 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.927+0000 7f2396742700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23900a0710 con 0x7f239009f2e0 2019-12-05T05:53:00.939 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.927+0000 7f2396742700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23900a0b00 con 0x7f239007e720 2019-12-05T05:53:00.939 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.927+0000 7f2396742700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f239008e5b0 con 0x7f23900a7740 2019-12-05T05:53:00.939 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.927+0000 7f238ffff700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f239007e720 0x7f239007eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:53:00.940 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.927+0000 7f238ffff700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f239007e720 0x7f239007eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.42:59344/0 (socket says 172.21.15.42:59344) 2019-12-05T05:53:00.940 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.927+0000 7f238ffff700 1 -- 172.21.15.42:0/249365608 learned_addr learned my addr 172.21.15.42:0/249365608 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:53:00.940 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.927+0000 7f2394cdf700 1 --2- 172.21.15.42:0/249365608 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f23900a7740 0x7f239008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:53:00.940 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.927+0000 7f23954e0700 1 --1- 172.21.15.42:0/249365608 >> v1:172.21.15.40:6789/0 conn(0x7f239009f2e0 0x7f239009f710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:52776/0 (socket says 172.21.15.42:52776) 2019-12-05T05:53:00.940 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.931+0000 7f238ffff700 1 -- 172.21.15.42:0/249365608 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f23900a7740 msgr2=0x7f239008df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:53:00.940 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.931+0000 7f238ffff700 1 --2- 172.21.15.42:0/249365608 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f23900a7740 0x7f239008df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:53:00.941 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.931+0000 7f238ffff700 1 -- 172.21.15.42:0/249365608 >> v1:172.21.15.40:6789/0 conn(0x7f239009f2e0 legacy=0x7f239009f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:53:00.941 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.931+0000 7f238ffff700 1 -- 172.21.15.42:0/249365608 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f239007f0a0 con 0x7f239007e720 2019-12-05T05:53:00.941 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.931+0000 7f238ffff700 1 --2- 172.21.15.42:0/249365608 >> v2:172.21.15.40:3300/0 conn(0x7f239007e720 0x7f239007eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f2378008b10 tx=0x7f23780100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:53:00.942 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.931+0000 7f238ffff700 1 --2- 172.21.15.42:0/249365608 >> v2:172.21.15.40:3300/0 conn(0x7f239007e720 0x7f239007eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f2378008b10 tx=0x7f23780100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:53:00.942 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.931+0000 7f238ffff700 1 --2- 172.21.15.42:0/249365608 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f239007e720 0x7f239007eb50 secure :-1 s=READY pgs=1257 cs=0 l=1 rx=0x7f2378008b10 tx=0x7f23780100b0).ready entity=mon.0 client_cookie=ca8fd987a9e897ef server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:53:00.942 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.931+0000 7f238ffff700 1 --2- 172.21.15.42:0/249365608 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f239007e720 0x7f239007eb50 secure :-1 s=THROTTLE_DONE pgs=1257 cs=0 l=1 rx=0x7f2378008b10 tx=0x7f23780100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:53:00.942 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.931+0000 7f238ffff700 1 --2- 172.21.15.42:0/249365608 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f239007e720 0x7f239007eb50 secure :-1 s=THROTTLE_DONE pgs=1257 cs=0 l=1 rx=0x7f2378008b10 tx=0x7f23780100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:53:00.942 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.931+0000 7f238f7fe700 1 -- 172.21.15.42:0/249365608 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f237800a160 con 0x7f239007e720 2019-12-05T05:53:00.942 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.931+0000 7f238ffff700 1 --2- 172.21.15.42:0/249365608 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f239007e720 0x7f239007eb50 secure :-1 s=THROTTLE_DONE pgs=1257 cs=0 l=1 rx=0x7f2378008b10 tx=0x7f23780100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:53:00.942 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.931+0000 7f238f7fe700 1 -- 172.21.15.42:0/249365608 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2378011a30 con 0x7f239007e720 2019-12-05T05:53:00.943 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.931+0000 7f238f7fe700 1 -- 172.21.15.42:0/249365608 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f237800a550 con 0x7f239007e720 2019-12-05T05:53:00.943 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.931+0000 7f2396742700 1 -- 172.21.15.42:0/249365608 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f239007e720 msgr2=0x7f239007eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:53:00.943 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.931+0000 7f2396742700 1 --2- 172.21.15.42:0/249365608 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f239007e720 0x7f239007eb50 secure :-1 s=READY pgs=1257 cs=0 l=1 rx=0x7f2378008b10 tx=0x7f23780100b0).stop 2019-12-05T05:53:00.943 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.931+0000 7f2396742700 1 -- 172.21.15.42:0/249365608 shutdown_connections 2019-12-05T05:53:00.943 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.931+0000 7f2396742700 1 --2- 172.21.15.42:0/249365608 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f23900a7740 0x7f239008df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:53:00.943 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.931+0000 7f2396742700 1 --2- 172.21.15.42:0/249365608 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f239007e720 0x7f239007eb50 unknown :-1 s=CLOSED pgs=1257 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:53:00.943 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.931+0000 7f2396742700 1 -- 172.21.15.42:0/249365608 shutdown_connections 2019-12-05T05:53:00.944 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.931+0000 7f2396742700 1 -- 172.21.15.42:0/249365608 wait complete. 2019-12-05T05:53:00.944 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.931+0000 7f2396742700 1 -- 172.21.15.42:0/249365608 >> 172.21.15.42:0/249365608 conn(0x7f23900aabe0 msgr2=0x7f23900a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:53:00.948 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.931+0000 7f2396742700 1 Processor -- start 2019-12-05T05:53:00.948 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.935+0000 7f2396742700 1 -- start start 2019-12-05T05:53:00.948 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.935+0000 7f2396742700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2390127550 0x7f23901237b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:53:00.948 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.935+0000 7f2396742700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2390115d90 con 0x7f2390127980 2019-12-05T05:53:00.948 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.935+0000 7f2396742700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2390124500 con 0x7f239011b280 2019-12-05T05:53:00.948 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.935+0000 7f2396742700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2390124730 con 0x7f2390127550 2019-12-05T05:53:00.949 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.935+0000 7f2394cdf700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f239011b280 0x7f2390114090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:43338/0 (socket says 172.21.15.42:43338) 2019-12-05T05:53:00.949 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.935+0000 7f2394cdf700 1 -- 172.21.15.42:0/3481123825 learned_addr learned my addr 172.21.15.42:0/3481123825 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:53:00.949 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.935+0000 7f238ffff700 1 --2- 172.21.15.42:0/3481123825 >> v2:172.21.15.40:3300/0 conn(0x7f2390127550 0x7f23901237b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:53:00.949 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.935+0000 7f238d7fa700 1 -- 172.21.15.42:0/3481123825 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3430314766 0 0) 0x7f2390124500 con 0x7f239011b280 2019-12-05T05:53:00.949 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.935+0000 7f238d7fa700 1 -- 172.21.15.42:0/3481123825 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23740030b0 con 0x7f239011b280 2019-12-05T05:53:00.949 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.935+0000 7f238ffff700 1 -- 172.21.15.42:0/3481123825 >> v1:172.21.15.42:6789/0 conn(0x7f239011b280 legacy=0x7f2390114090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:53:00.949 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.935+0000 7f238d7fa700 1 -- 172.21.15.42:0/3481123825 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1292625864 0 0) 0x7f2390115d90 con 0x7f2390127980 2019-12-05T05:53:00.950 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.935+0000 7f238ffff700 1 -- 172.21.15.42:0/3481123825 >> v1:172.21.15.40:6789/0 conn(0x7f2390127980 legacy=0x7f2390123d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:53:00.950 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.935+0000 7f238ffff700 1 -- 172.21.15.42:0/3481123825 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23901311f0 con 0x7f2390127550 2019-12-05T05:53:00.950 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.935+0000 7f238ffff700 1 --2- 172.21.15.42:0/3481123825 >> v2:172.21.15.40:3300/0 conn(0x7f2390127550 0x7f23901237b0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f2378008b10 tx=0x7f2378011c60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:53:00.950 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.935+0000 7f238ffff700 1 --2- 172.21.15.42:0/3481123825 >> v2:172.21.15.40:3300/0 conn(0x7f2390127550 0x7f23901237b0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f2378008b10 tx=0x7f2378011c60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:53:00.950 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.935+0000 7f238ffff700 1 --2- 172.21.15.42:0/3481123825 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2390127550 0x7f23901237b0 secure :-1 s=READY pgs=1259 cs=0 l=1 rx=0x7f2378008b10 tx=0x7f2378011c60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:53:00.950 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.935+0000 7f238ffff700 1 --2- 172.21.15.42:0/3481123825 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2390127550 0x7f23901237b0 secure :-1 s=THROTTLE_DONE pgs=1259 cs=0 l=1 rx=0x7f2378008b10 tx=0x7f2378011c60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:53:00.950 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.935+0000 7f238d7fa700 1 -- 172.21.15.42:0/3481123825 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f237800ad60 con 0x7f2390127550 2019-12-05T05:53:00.951 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.935+0000 7f238ffff700 1 --2- 172.21.15.42:0/3481123825 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2390127550 0x7f23901237b0 secure :-1 s=THROTTLE_DONE pgs=1259 cs=0 l=1 rx=0x7f2378008b10 tx=0x7f2378011c60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:53:00.951 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.935+0000 7f238ffff700 1 --2- 172.21.15.42:0/3481123825 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2390127550 0x7f23901237b0 secure :-1 s=THROTTLE_DONE pgs=1259 cs=0 l=1 rx=0x7f2378008b10 tx=0x7f2378011c60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:53:00.951 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.935+0000 7f238d7fa700 1 -- 172.21.15.42:0/3481123825 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2378011e10 con 0x7f2390127550 2019-12-05T05:53:00.952 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.935+0000 7f238d7fa700 1 -- 172.21.15.42:0/3481123825 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2378005ca0 con 0x7f2390127550 2019-12-05T05:53:00.952 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.935+0000 7f2396742700 1 -- 172.21.15.42:0/3481123825 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f239012f1d0 con 0x7f2390127550 2019-12-05T05:53:00.952 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.935+0000 7f2396742700 1 -- 172.21.15.42:0/3481123825 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f239012f6a0 con 0x7f2390127550 2019-12-05T05:53:00.952 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.935+0000 7f238ffff700 1 --2- 172.21.15.42:0/3481123825 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2390127550 0x7f23901237b0 secure :-1 s=THROTTLE_DONE pgs=1259 cs=0 l=1 rx=0x7f2378008b10 tx=0x7f2378011c60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:53:00.952 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.939+0000 7f238d7fa700 1 -- 172.21.15.42:0/3481123825 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f2378007150 con 0x7f2390127550 2019-12-05T05:53:00.953 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.939+0000 7f238ffff700 1 --2- 172.21.15.42:0/3481123825 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2390127550 0x7f23901237b0 secure :-1 s=THROTTLE_DONE pgs=1259 cs=0 l=1 rx=0x7f2378008b10 tx=0x7f2378011c60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:53:00.953 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.939+0000 7f238d7fa700 1 --2- 172.21.15.42:0/3481123825 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f237403b230 0x7f237403d740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:53:00.953 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.939+0000 7f238d7fa700 1 -- 172.21.15.42:0/3481123825 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8955+0+0 (secure 0 0 0) 0x7f237804bc60 con 0x7f2390127550 2019-12-05T05:53:00.953 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.939+0000 7f2394cdf700 1 --2- 172.21.15.42:0/3481123825 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f237403b230 0x7f237403d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:53:00.953 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.939+0000 7f2394cdf700 1 --2- 172.21.15.42:0/3481123825 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f237403b230 0x7f237403d740 crc :-1 s=READY pgs=700 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:53:00.953 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.943+0000 7f2396742700 1 -- 172.21.15.42:0/3481123825 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f239005d110 con 0x7f2390127550 2019-12-05T05:53:00.960 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.955+0000 7f238ffff700 1 --2- 172.21.15.42:0/3481123825 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2390127550 0x7f23901237b0 secure :-1 s=THROTTLE_DONE pgs=1259 cs=0 l=1 rx=0x7f2378008b10 tx=0x7f2378011c60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:53:00.960 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:00.955+0000 7f238d7fa700 1 -- 172.21.15.42:0/3481123825 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f2378032ce0 con 0x7f2390127550 2019-12-05T05:53:01.129 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:01.123+0000 7f2396742700 1 -- 172.21.15.42:0/3481123825 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f239005b8c0 con 0x7f2390127550 2019-12-05T05:53:01.131 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:01.123+0000 7f238ffff700 1 --2- 172.21.15.42:0/3481123825 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2390127550 0x7f23901237b0 secure :-1 s=THROTTLE_DONE pgs=1259 cs=0 l=1 rx=0x7f2378008b10 tx=0x7f2378011c60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:53:01.131 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:01.123+0000 7f238d7fa700 1 -- 172.21.15.42:0/3481123825 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 199 v199) v1 ==== 95+0+1774 (secure 0 0 0) 0x7f2378032f40 con 0x7f2390127550 2019-12-05T05:53:01.132 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 199 2019-12-05T05:53:01.132 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:53:01.133 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":199,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":199,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:52:59.156877+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6356},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6356":{"gid":6356,"name":"b","rank":0,"incarnation":199,"state":"up:replay","state_seq":1,"addr":"172.21.15.40:6833/1523046776","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1523046776},{"type":"v1","addr":"172.21.15.40:6833","nonce":1523046776}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:53:01.134 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:01.127+0000 7f2396742700 1 -- 172.21.15.42:0/3481123825 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f237403b230 msgr2=0x7f237403d740 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:53:01.134 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:01.127+0000 7f2396742700 1 --2- 172.21.15.42:0/3481123825 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f237403b230 0x7f237403d740 crc :-1 s=READY pgs=700 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:53:01.136 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:01.127+0000 7f2396742700 1 -- 172.21.15.42:0/3481123825 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2390127550 msgr2=0x7f23901237b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:53:01.137 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:01.127+0000 7f2396742700 1 --2- 172.21.15.42:0/3481123825 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2390127550 0x7f23901237b0 secure :-1 s=READY pgs=1259 cs=0 l=1 rx=0x7f2378008b10 tx=0x7f2378011c60).stop 2019-12-05T05:53:01.137 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:01.127+0000 7f2396742700 1 -- 172.21.15.42:0/3481123825 shutdown_connections 2019-12-05T05:53:01.137 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:01.127+0000 7f2396742700 1 --2- 172.21.15.42:0/3481123825 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f237403b230 0x7f237403d740 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:53:01.137 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:01.127+0000 7f2396742700 1 --2- 172.21.15.42:0/3481123825 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2390127550 0x7f23901237b0 unknown :-1 s=CLOSED pgs=1259 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:53:01.137 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:01.127+0000 7f2396742700 1 -- 172.21.15.42:0/3481123825 shutdown_connections 2019-12-05T05:53:01.138 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:01.127+0000 7f2396742700 1 -- 172.21.15.42:0/3481123825 wait complete. 2019-12-05T05:53:01.140 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:01.127+0000 7f2396742700 1 -- 172.21.15.42:0/3481123825 >> 172.21.15.42:0/3481123825 conn(0x7f23900a8230 msgr2=0x7f23900a9870 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:53:01.155 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 199, "feature_flags": { "enable_multiple": false, "ever_enabled_multiple": false }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2019-12-05T05:03:35.604738+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 199, "ever_allowed_features": 2, "explicitly_allowed_features": 2, "failed": [], "flags": 18, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_6356": { "addr": "172.21.15.40:6833/1523046776", "addrs": { "addrvec": [ { "addr": "172.21.15.40:6832", "nonce": 1523046776, "type": "v2" }, { "addr": "172.21.15.40:6833", "nonce": 1523046776, "type": "v1" } ] }, "export_targets": [], "features": 4540138292836696063, "flags": 0, "gid": 6356, "incarnation": 199, "name": "b", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 83, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 2, "min_compat_client": "0 (unknown)", "modified": "2019-12-05T05:52:59.156877+0000", "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6356 } } } ], "standbys": [] } 2019-12-05T05:53:03.156 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:53:03.259 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:03.247+0000 7f67c9f2a700 1 Processor -- start 2019-12-05T05:53:03.259 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:03.247+0000 7f67c9f2a700 1 -- start start 2019-12-05T05:53:03.260 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:03.251+0000 7f67c9f2a700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f67c40a7740 0x7f67c408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:53:03.260 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:03.251+0000 7f67c9f2a700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f67c407e720 0x7f67c407eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:53:03.260 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:03.251+0000 7f67c9f2a700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f67c409f2e0 0x7f67c409f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:53:03.260 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:03.251+0000 7f67c9f2a700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67c40a0b00 con 0x7f67c407e720 2019-12-05T05:53:03.261 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:03.251+0000 7f67c9f2a700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67c408e5b0 con 0x7f67c40a7740 2019-12-05T05:53:03.261 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:03.251+0000 7f67c9f2a700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67c408e850 con 0x7f67c409f2e0 2019-12-05T05:53:03.261 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:03.251+0000 7f67c3fff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f67c40a7740 0x7f67c408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:53:03.261 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:03.251+0000 7f67c3fff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f67c40a7740 0x7f67c408df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:43316/0 (socket says 172.21.15.42:43316) 2019-12-05T05:53:03.262 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:03.251+0000 7f67c3fff700 1 -- 172.21.15.42:0/1408484373 learned_addr learned my addr 172.21.15.42:0/1408484373 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:53:03.262 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:03.251+0000 7f67c8cc8700 1 --2- 172.21.15.42:0/1408484373 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f67c409f2e0 0x7f67c409f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:53:03.262 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:03.251+0000 7f67bbfff700 1 --2- 172.21.15.42:0/1408484373 >> v2:172.21.15.40:3300/0 conn(0x7f67c407e720 0x7f67c407eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:53:03.262 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:03.251+0000 7f67c3fff700 1 -- 172.21.15.42:0/1408484373 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f67c409f2e0 msgr2=0x7f67c409f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:53:03.262 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:03.251+0000 7f67c3fff700 1 --2- 172.21.15.42:0/1408484373 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f67c409f2e0 0x7f67c409f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:53:03.263 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:03.251+0000 7f67c3fff700 1 -- 172.21.15.42:0/1408484373 >> v2:172.21.15.40:3300/0 conn(0x7f67c407e720 msgr2=0x7f67c407eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:53:03.263 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:03.251+0000 7f67c3fff700 1 --2- 172.21.15.42:0/1408484373 >> v2:172.21.15.40:3300/0 conn(0x7f67c407e720 0x7f67c407eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:53:03.263 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:03.251+0000 7f67c3fff700 1 -- 172.21.15.42:0/1408484373 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67c407f0a0 con 0x7f67c40a7740 2019-12-05T05:53:03.264 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:03.251+0000 7f67c3fff700 1 --2- 172.21.15.42:0/1408484373 >> v2:172.21.15.42:3300/0 conn(0x7f67c40a7740 0x7f67c408df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f67c40a0f50 tx=0x7f67b400fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:53:03.264 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:03.251+0000 7f67c3fff700 1 --2- 172.21.15.42:0/1408484373 >> v2:172.21.15.42:3300/0 conn(0x7f67c40a7740 0x7f67c408df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f67c40a0f50 tx=0x7f67b400fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:53:03.264 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:03.251+0000 7f67c3fff700 1 --2- 172.21.15.42:0/1408484373 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f67c40a7740 0x7f67c408df80 secure :-1 s=READY pgs=1635 cs=0 l=1 rx=0x7f67c40a0f50 tx=0x7f67b400fdb0).ready entity=mon.1 client_cookie=d8e68756a9e81320 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:53:03.264 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:03.251+0000 7f67c3fff700 1 --2- 172.21.15.42:0/1408484373 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f67c40a7740 0x7f67c408df80 secure :-1 s=THROTTLE_DONE pgs=1635 cs=0 l=1 rx=0x7f67c40a0f50 tx=0x7f67b400fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:53:03.265 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:03.251+0000 7f67c37fe700 1 -- 172.21.15.42:0/1408484373 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f67b4008c90 con 0x7f67c40a7740 2019-12-05T05:53:03.265 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:03.251+0000 7f67c3fff700 1 --2- 172.21.15.42:0/1408484373 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f67c40a7740 0x7f67c408df80 secure :-1 s=THROTTLE_DONE pgs=1635 cs=0 l=1 rx=0x7f67c40a0f50 tx=0x7f67b400fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:53:03.265 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:03.251+0000 7f67c37fe700 1 -- 172.21.15.42:0/1408484373 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67b40088b0 con 0x7f67c40a7740 2019-12-05T05:53:03.266 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:03.251+0000 7f67c3fff700 1 --2- 172.21.15.42:0/1408484373 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f67c40a7740 0x7f67c408df80 secure :-1 s=THROTTLE_DONE pgs=1635 cs=0 l=1 rx=0x7f67c40a0f50 tx=0x7f67b400fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:53:03.266 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:03.251+0000 7f67c9f2a700 1 -- 172.21.15.42:0/1408484373 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f67c40a7740 msgr2=0x7f67c408df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:53:03.266 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:03.251+0000 7f67c37fe700 1 -- 172.21.15.42:0/1408484373 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f67b4011b50 con 0x7f67c40a7740 2019-12-05T05:53:03.267 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:03.251+0000 7f67c9f2a700 1 --2- 172.21.15.42:0/1408484373 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f67c40a7740 0x7f67c408df80 secure :-1 s=READY pgs=1635 cs=0 l=1 rx=0x7f67c40a0f50 tx=0x7f67b400fdb0).stop 2019-12-05T05:53:03.267 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:03.251+0000 7f67c9f2a700 1 -- 172.21.15.42:0/1408484373 shutdown_connections 2019-12-05T05:53:03.267 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:03.251+0000 7f67c9f2a700 1 --2- 172.21.15.42:0/1408484373 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f67c409f2e0 0x7f67c409f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:53:03.268 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:03.251+0000 7f67c9f2a700 1 --2- 172.21.15.42:0/1408484373 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f67c40a7740 0x7f67c408df80 unknown :-1 s=CLOSED pgs=1635 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:53:03.268 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:03.251+0000 7f67c9f2a700 1 --2- 172.21.15.42:0/1408484373 >> v2:172.21.15.40:3300/0 conn(0x7f67c407e720 0x7f67c407eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:53:03.268 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:03.251+0000 7f67c9f2a700 1 -- 172.21.15.42:0/1408484373 shutdown_connections 2019-12-05T05:53:03.268 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:03.251+0000 7f67c9f2a700 1 -- 172.21.15.42:0/1408484373 wait complete. 2019-12-05T05:53:03.269 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:03.251+0000 7f67c9f2a700 1 -- 172.21.15.42:0/1408484373 >> 172.21.15.42:0/1408484373 conn(0x7f67c40aabe0 msgr2=0x7f67c40a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:53:03.275 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:03.255+0000 7f67c9f2a700 1 Processor -- start 2019-12-05T05:53:03.275 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:03.255+0000 7f67c9f2a700 1 -- start start 2019-12-05T05:53:03.275 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:03.259+0000 7f67c9f2a700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f67c4122090 0x7f67c4124280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:53:03.276 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:03.259+0000 7f67c9f2a700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f67c4119870 0x7f67c408f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:53:03.276 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:03.259+0000 7f67c9f2a700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67c41247d0 con 0x7f67c4119440 2019-12-05T05:53:03.276 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:03.259+0000 7f67c9f2a700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67c4123580 con 0x7f67c4122090 2019-12-05T05:53:03.277 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:03.259+0000 7f67c9f2a700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67c4123820 con 0x7f67c4119870 2019-12-05T05:53:03.277 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:03.259+0000 7f67bbfff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f67c4119440 0x7f67c408eb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:43362/0 (socket says 172.21.15.42:43362) 2019-12-05T05:53:03.277 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:03.259+0000 7f67bbfff700 1 -- 172.21.15.42:0/1956961353 learned_addr learned my addr 172.21.15.42:0/1956961353 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:53:03.277 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:03.259+0000 7f67c8cc8700 1 --2- 172.21.15.42:0/1956961353 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f67c4119870 0x7f67c408f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:53:03.278 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:03.259+0000 7f67c3fff700 1 --2- 172.21.15.42:0/1956961353 >> v2:172.21.15.40:3300/0 conn(0x7f67c4122090 0x7f67c4124280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:53:03.278 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:03.259+0000 7f67c17fa700 1 -- 172.21.15.42:0/1956961353 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3264951310 0 0) 0x7f67c41247d0 con 0x7f67c4119440 2019-12-05T05:53:03.279 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:03.259+0000 7f67c17fa700 1 -- 172.21.15.42:0/1956961353 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67a80030b0 con 0x7f67c4119440 2019-12-05T05:53:03.279 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:03.259+0000 7f67c8cc8700 1 -- 172.21.15.42:0/1956961353 >> v2:172.21.15.40:3300/0 conn(0x7f67c4122090 msgr2=0x7f67c4124280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:53:03.279 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:03.259+0000 7f67c8cc8700 1 --2- 172.21.15.42:0/1956961353 >> v2:172.21.15.40:3300/0 conn(0x7f67c4122090 0x7f67c4124280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:53:03.280 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:03.259+0000 7f67c8cc8700 1 -- 172.21.15.42:0/1956961353 >> v1:172.21.15.42:6789/0 conn(0x7f67c4119440 legacy=0x7f67c408eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:53:03.280 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:03.259+0000 7f67c8cc8700 1 -- 172.21.15.42:0/1956961353 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67c4120d90 con 0x7f67c4119870 2019-12-05T05:53:03.280 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:03.259+0000 7f67c8cc8700 1 --2- 172.21.15.42:0/1956961353 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f67c4119870 0x7f67c408f370 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f67bc0032b0 tx=0x7f67bc010ca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:53:03.280 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:03.259+0000 7f67c8cc8700 1 --2- 172.21.15.42:0/1956961353 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f67c4119870 0x7f67c408f370 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f67bc0032b0 tx=0x7f67bc010ca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:53:03.281 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:03.259+0000 7f67c8cc8700 1 --2- 172.21.15.42:0/1956961353 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f67c4119870 0x7f67c408f370 secure :-1 s=READY pgs=364 cs=0 l=1 rx=0x7f67bc0032b0 tx=0x7f67bc010ca0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:53:03.281 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:03.259+0000 7f67c8cc8700 1 --2- 172.21.15.42:0/1956961353 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f67c4119870 0x7f67c408f370 secure :-1 s=THROTTLE_DONE pgs=364 cs=0 l=1 rx=0x7f67bc0032b0 tx=0x7f67bc010ca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:53:03.281 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:03.259+0000 7f67c17fa700 1 -- 172.21.15.42:0/1956961353 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f67bc00eaa0 con 0x7f67c4119870 2019-12-05T05:53:03.282 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:03.259+0000 7f67c9f2a700 1 -- 172.21.15.42:0/1956961353 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67c408f8c0 con 0x7f67c4119870 2019-12-05T05:53:03.282 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:03.259+0000 7f67c8cc8700 1 --2- 172.21.15.42:0/1956961353 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f67c4119870 0x7f67c408f370 secure :-1 s=THROTTLE_DONE pgs=364 cs=0 l=1 rx=0x7f67bc0032b0 tx=0x7f67bc010ca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:53:03.282 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:03.259+0000 7f67c17fa700 1 -- 172.21.15.42:0/1956961353 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67bc00e6c0 con 0x7f67c4119870 2019-12-05T05:53:03.282 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:03.259+0000 7f67c8cc8700 1 --2- 172.21.15.42:0/1956961353 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f67c4119870 0x7f67c408f370 secure :-1 s=THROTTLE_DONE pgs=364 cs=0 l=1 rx=0x7f67bc0032b0 tx=0x7f67bc010ca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:53:03.283 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:03.259+0000 7f67c17fa700 1 -- 172.21.15.42:0/1956961353 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f67bc012580 con 0x7f67c4119870 2019-12-05T05:53:03.283 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:03.259+0000 7f67c9f2a700 1 -- 172.21.15.42:0/1956961353 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67c4119ca0 con 0x7f67c4119870 2019-12-05T05:53:03.283 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:03.259+0000 7f67c8cc8700 1 --2- 172.21.15.42:0/1956961353 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f67c4119870 0x7f67c408f370 secure :-1 s=THROTTLE_DONE pgs=364 cs=0 l=1 rx=0x7f67bc0032b0 tx=0x7f67bc010ca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:53:03.283 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:03.263+0000 7f67c17fa700 1 -- 172.21.15.42:0/1956961353 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f67bc00f0c0 con 0x7f67c4119870 2019-12-05T05:53:03.284 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:03.263+0000 7f67c8cc8700 1 --2- 172.21.15.42:0/1956961353 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f67c4119870 0x7f67c408f370 secure :-1 s=THROTTLE_DONE pgs=364 cs=0 l=1 rx=0x7f67bc0032b0 tx=0x7f67bc010ca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:53:03.284 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:03.263+0000 7f67c17fa700 1 --2- 172.21.15.42:0/1956961353 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f67a803b280 0x7f67a803d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:53:03.284 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:03.263+0000 7f67c17fa700 1 -- 172.21.15.42:0/1956961353 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8955+0+0 (secure 0 0 0) 0x7f67bc04b450 con 0x7f67c4119870 2019-12-05T05:53:03.284 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:03.263+0000 7f67c3fff700 1 --2- 172.21.15.42:0/1956961353 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f67a803b280 0x7f67a803d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:53:03.284 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:03.263+0000 7f67c3fff700 1 --2- 172.21.15.42:0/1956961353 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f67a803b280 0x7f67a803d790 crc :-1 s=READY pgs=701 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:53:03.285 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:03.263+0000 7f67c9f2a700 1 -- 172.21.15.42:0/1956961353 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67c405d110 con 0x7f67c4119870 2019-12-05T05:53:03.287 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:03.271+0000 7f67c8cc8700 1 --2- 172.21.15.42:0/1956961353 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f67c4119870 0x7f67c408f370 secure :-1 s=THROTTLE_DONE pgs=364 cs=0 l=1 rx=0x7f67bc0032b0 tx=0x7f67bc010ca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:53:03.287 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:03.271+0000 7f67c17fa700 1 -- 172.21.15.42:0/1956961353 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f67bc032450 con 0x7f67c4119870 2019-12-05T05:53:03.449 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:03.443+0000 7f67c9f2a700 1 -- 172.21.15.42:0/1956961353 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f67c40aadf0 con 0x7f67c4119870 2019-12-05T05:53:03.450 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:03.443+0000 7f67c8cc8700 1 --2- 172.21.15.42:0/1956961353 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f67c4119870 0x7f67c408f370 secure :-1 s=THROTTLE_DONE pgs=364 cs=0 l=1 rx=0x7f67bc0032b0 tx=0x7f67bc010ca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:53:03.450 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:03.443+0000 7f67c17fa700 1 -- 172.21.15.42:0/1956961353 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 200 v200) v1 ==== 95+0+1778 (secure 0 0 0) 0x7f67bc0326b0 con 0x7f67c4119870 2019-12-05T05:53:03.452 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 200 2019-12-05T05:53:03.453 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:53:03.453 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":200,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":200,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:53:02.588896+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6356},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6356":{"gid":6356,"name":"b","rank":0,"incarnation":199,"state":"up:reconnect","state_seq":24,"addr":"172.21.15.40:6833/1523046776","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1523046776},{"type":"v1","addr":"172.21.15.40:6833","nonce":1523046776}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:53:03.455 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:03.447+0000 7f67c9f2a700 1 -- 172.21.15.42:0/1956961353 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f67a803b280 msgr2=0x7f67a803d790 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:53:03.456 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:03.447+0000 7f67c9f2a700 1 --2- 172.21.15.42:0/1956961353 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f67a803b280 0x7f67a803d790 crc :-1 s=READY pgs=701 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:53:03.456 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:03.447+0000 7f67c9f2a700 1 -- 172.21.15.42:0/1956961353 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f67c4119870 msgr2=0x7f67c408f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:53:03.456 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:03.447+0000 7f67c9f2a700 1 --2- 172.21.15.42:0/1956961353 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f67c4119870 0x7f67c408f370 secure :-1 s=READY pgs=364 cs=0 l=1 rx=0x7f67bc0032b0 tx=0x7f67bc010ca0).stop 2019-12-05T05:53:03.456 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:03.447+0000 7f67c9f2a700 1 -- 172.21.15.42:0/1956961353 shutdown_connections 2019-12-05T05:53:03.456 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:03.447+0000 7f67c9f2a700 1 --2- 172.21.15.42:0/1956961353 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f67a803b280 0x7f67a803d790 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:53:03.456 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:03.447+0000 7f67c9f2a700 1 --2- 172.21.15.42:0/1956961353 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f67c4119870 0x7f67c408f370 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:53:03.457 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:03.447+0000 7f67c9f2a700 1 --2- 172.21.15.42:0/1956961353 >> v2:172.21.15.40:3300/0 conn(0x7f67c4122090 0x7f67c4124280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:53:03.457 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:03.451+0000 7f67c9f2a700 1 -- 172.21.15.42:0/1956961353 shutdown_connections 2019-12-05T05:53:03.457 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:03.451+0000 7f67c9f2a700 1 -- 172.21.15.42:0/1956961353 wait complete. 2019-12-05T05:53:03.457 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:03.451+0000 7f67c9f2a700 1 -- 172.21.15.42:0/1956961353 >> 172.21.15.42:0/1956961353 conn(0x7f67c40a7740 msgr2=0x7f67c40a9770 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:53:03.465 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2019-12-05T05:53:05.466 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:53:05.572 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:05.559+0000 7f3636118700 1 Processor -- start 2019-12-05T05:53:05.572 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:05.559+0000 7f3636118700 1 -- start start 2019-12-05T05:53:05.572 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:05.559+0000 7f3636118700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f363008df80 0x7f363009f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:53:05.572 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:05.559+0000 7f3636118700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f363009f830 0x7f363014efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:53:05.572 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:05.559+0000 7f3636118700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f363009fc60 con 0x7f36300a7740 2019-12-05T05:53:05.573 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:05.559+0000 7f3636118700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36300a0b00 con 0x7f363009f830 2019-12-05T05:53:05.573 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:05.559+0000 7f3636118700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f363008e5b0 con 0x7f363008df80 2019-12-05T05:53:05.573 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:05.559+0000 7f362ffff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f36300a7740 0x7f363007e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:43374/0 (socket says 172.21.15.42:43374) 2019-12-05T05:53:05.573 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:05.559+0000 7f362ffff700 1 -- 172.21.15.42:0/1472050406 learned_addr learned my addr 172.21.15.42:0/1472050406 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:53:05.573 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:05.559+0000 7f36277fe700 1 --2- 172.21.15.42:0/1472050406 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f363008df80 0x7f363009f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:53:05.573 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:05.559+0000 7f3634eb6700 1 --2- 172.21.15.42:0/1472050406 >> v2:172.21.15.40:3300/0 conn(0x7f363009f830 0x7f363014efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:53:05.574 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:05.563+0000 7f362f7fe700 1 -- 172.21.15.42:0/1472050406 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 234075405 0 0) 0x7f363009fc60 con 0x7f36300a7740 2019-12-05T05:53:05.574 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:05.563+0000 7f362f7fe700 1 -- 172.21.15.42:0/1472050406 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f361c0030b0 con 0x7f36300a7740 2019-12-05T05:53:05.574 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:05.563+0000 7f36277fe700 1 -- 172.21.15.42:0/1472050406 >> v2:172.21.15.40:3300/0 conn(0x7f363009f830 msgr2=0x7f363014efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:53:05.574 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:05.563+0000 7f36277fe700 1 --2- 172.21.15.42:0/1472050406 >> v2:172.21.15.40:3300/0 conn(0x7f363009f830 0x7f363014efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:53:05.574 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:05.563+0000 7f36277fe700 1 -- 172.21.15.42:0/1472050406 >> v1:172.21.15.42:6789/0 conn(0x7f36300a7740 legacy=0x7f363007e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:53:05.574 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:05.563+0000 7f36277fe700 1 -- 172.21.15.42:0/1472050406 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36300a0710 con 0x7f363008df80 2019-12-05T05:53:05.574 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:05.563+0000 7f36277fe700 1 --2- 172.21.15.42:0/1472050406 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f363008df80 0x7f363009f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f3618008790 tx=0x7f361800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:53:05.575 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:05.563+0000 7f36277fe700 1 --2- 172.21.15.42:0/1472050406 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f363008df80 0x7f363009f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f3618008790 tx=0x7f361800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:53:05.575 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:05.563+0000 7f36277fe700 1 --2- 172.21.15.42:0/1472050406 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f363008df80 0x7f363009f2e0 secure :-1 s=READY pgs=365 cs=0 l=1 rx=0x7f3618008790 tx=0x7f361800fc70).ready entity=mon.2 client_cookie=fe96977c271056c5 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:53:05.575 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:05.563+0000 7f36277fe700 1 --2- 172.21.15.42:0/1472050406 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f363008df80 0x7f363009f2e0 secure :-1 s=THROTTLE_DONE pgs=365 cs=0 l=1 rx=0x7f3618008790 tx=0x7f361800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:53:05.575 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:05.563+0000 7f362f7fe700 1 -- 172.21.15.42:0/1472050406 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3618008bb0 con 0x7f363008df80 2019-12-05T05:53:05.575 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:05.563+0000 7f36277fe700 1 --2- 172.21.15.42:0/1472050406 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f363008df80 0x7f363009f2e0 secure :-1 s=THROTTLE_DONE pgs=365 cs=0 l=1 rx=0x7f3618008790 tx=0x7f361800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:53:05.575 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:05.563+0000 7f362f7fe700 1 -- 172.21.15.42:0/1472050406 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36180087d0 con 0x7f363008df80 2019-12-05T05:53:05.575 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:05.563+0000 7f36277fe700 1 --2- 172.21.15.42:0/1472050406 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f363008df80 0x7f363009f2e0 secure :-1 s=THROTTLE_DONE pgs=365 cs=0 l=1 rx=0x7f3618008790 tx=0x7f361800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:53:05.576 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:05.563+0000 7f362f7fe700 1 -- 172.21.15.42:0/1472050406 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3618011860 con 0x7f363008df80 2019-12-05T05:53:05.576 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:05.563+0000 7f3636118700 1 -- 172.21.15.42:0/1472050406 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f363008df80 msgr2=0x7f363009f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:53:05.576 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:05.563+0000 7f3636118700 1 --2- 172.21.15.42:0/1472050406 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f363008df80 0x7f363009f2e0 secure :-1 s=READY pgs=365 cs=0 l=1 rx=0x7f3618008790 tx=0x7f361800fc70).stop 2019-12-05T05:53:05.576 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:05.563+0000 7f3636118700 1 -- 172.21.15.42:0/1472050406 shutdown_connections 2019-12-05T05:53:05.576 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:05.563+0000 7f3636118700 1 --2- 172.21.15.42:0/1472050406 >> v2:172.21.15.40:3300/0 conn(0x7f363009f830 0x7f363014efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:53:05.582 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:05.563+0000 7f3636118700 1 --2- 172.21.15.42:0/1472050406 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f363008df80 0x7f363009f2e0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:53:05.582 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:05.563+0000 7f3636118700 1 -- 172.21.15.42:0/1472050406 shutdown_connections 2019-12-05T05:53:05.582 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:05.563+0000 7f3636118700 1 -- 172.21.15.42:0/1472050406 wait complete. 2019-12-05T05:53:05.582 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:05.563+0000 7f3636118700 1 -- 172.21.15.42:0/1472050406 >> 172.21.15.42:0/1472050406 conn(0x7f36300aabe0 msgr2=0x7f36300a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:53:05.582 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:05.567+0000 7f3636118700 1 Processor -- start 2019-12-05T05:53:05.582 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:05.567+0000 7f3636118700 1 -- start start 2019-12-05T05:53:05.585 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:05.567+0000 7f3636118700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f363011b6f0 0x7f363011bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:53:05.586 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:05.567+0000 7f3636118700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f3630133d80 0x7f36301341b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:53:05.586 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:05.567+0000 7f3636118700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f363011c070 con 0x7f363005a260 2019-12-05T05:53:05.586 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:05.567+0000 7f3636118700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3630122bd0 con 0x7f3630133d80 2019-12-05T05:53:05.586 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:05.567+0000 7f3636118700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3630122e70 con 0x7f363011b6f0 2019-12-05T05:53:05.586 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:05.567+0000 7f36277fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f363011b6f0 0x7f363011bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:53:05.587 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:05.567+0000 7f36277fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f363011b6f0 0x7f363011bb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:43344/0 (socket says 172.21.15.42:43344) 2019-12-05T05:53:05.587 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:05.567+0000 7f36277fe700 1 -- 172.21.15.42:0/2102128026 learned_addr learned my addr 172.21.15.42:0/2102128026 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:53:05.587 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:05.567+0000 7f3634eb6700 1 --2- 172.21.15.42:0/2102128026 >> v2:172.21.15.40:3300/0 conn(0x7f3630133d80 0x7f36301341b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:53:05.587 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:05.567+0000 7f36277fe700 1 -- 172.21.15.42:0/2102128026 >> v2:172.21.15.40:3300/0 conn(0x7f3630133d80 msgr2=0x7f36301341b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:53:05.587 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:05.567+0000 7f36277fe700 1 --2- 172.21.15.42:0/2102128026 >> v2:172.21.15.40:3300/0 conn(0x7f3630133d80 0x7f36301341b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:53:05.587 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:05.567+0000 7f36277fe700 1 -- 172.21.15.42:0/2102128026 >> v1:172.21.15.42:6789/0 conn(0x7f363005a260 legacy=0x7f36301237e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:53:05.587 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:05.567+0000 7f36277fe700 1 -- 172.21.15.42:0/2102128026 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3630123fe0 con 0x7f363011b6f0 2019-12-05T05:53:05.588 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:05.567+0000 7f36277fe700 1 --2- 172.21.15.42:0/2102128026 >> v2:172.21.15.42:3300/0 conn(0x7f363011b6f0 0x7f363011bb20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f3618008790 tx=0x7f3618011920).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:53:05.588 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:05.567+0000 7f36277fe700 1 --2- 172.21.15.42:0/2102128026 >> v2:172.21.15.42:3300/0 conn(0x7f363011b6f0 0x7f363011bb20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f3618008790 tx=0x7f3618011920).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:53:05.588 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:05.567+0000 7f36277fe700 1 --2- 172.21.15.42:0/2102128026 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f363011b6f0 0x7f363011bb20 secure :-1 s=READY pgs=1641 cs=0 l=1 rx=0x7f3618008790 tx=0x7f3618011920).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:53:05.588 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:05.567+0000 7f36277fe700 1 --2- 172.21.15.42:0/2102128026 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f363011b6f0 0x7f363011bb20 secure :-1 s=THROTTLE_DONE pgs=1641 cs=0 l=1 rx=0x7f3618008790 tx=0x7f3618011920).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:53:05.588 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:05.567+0000 7f362d7fa700 1 -- 172.21.15.42:0/2102128026 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f361800f280 con 0x7f363011b6f0 2019-12-05T05:53:05.588 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:05.567+0000 7f36277fe700 1 --2- 172.21.15.42:0/2102128026 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f363011b6f0 0x7f363011bb20 secure :-1 s=THROTTLE_DONE pgs=1641 cs=0 l=1 rx=0x7f3618008790 tx=0x7f3618011920).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:53:05.588 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:05.567+0000 7f362d7fa700 1 -- 172.21.15.42:0/2102128026 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3618008db0 con 0x7f363011b6f0 2019-12-05T05:53:05.589 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:05.567+0000 7f3636118700 1 -- 172.21.15.42:0/2102128026 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3630134700 con 0x7f363011b6f0 2019-12-05T05:53:05.589 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:05.567+0000 7f36277fe700 1 --2- 172.21.15.42:0/2102128026 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f363011b6f0 0x7f363011bb20 secure :-1 s=THROTTLE_DONE pgs=1641 cs=0 l=1 rx=0x7f3618008790 tx=0x7f3618011920).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:53:05.589 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:05.567+0000 7f362d7fa700 1 -- 172.21.15.42:0/2102128026 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3618005540 con 0x7f363011b6f0 2019-12-05T05:53:05.598 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:05.567+0000 7f3636118700 1 -- 172.21.15.42:0/2102128026 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f363013b140 con 0x7f363011b6f0 2019-12-05T05:53:05.599 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:05.571+0000 7f36277fe700 1 --2- 172.21.15.42:0/2102128026 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f363011b6f0 0x7f363011bb20 secure :-1 s=THROTTLE_DONE pgs=1641 cs=0 l=1 rx=0x7f3618008790 tx=0x7f3618011920).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:53:05.599 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:05.571+0000 7f362d7fa700 1 -- 172.21.15.42:0/2102128026 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f36180057e0 con 0x7f363011b6f0 2019-12-05T05:53:05.599 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:05.571+0000 7f36277fe700 1 --2- 172.21.15.42:0/2102128026 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f363011b6f0 0x7f363011bb20 secure :-1 s=THROTTLE_DONE pgs=1641 cs=0 l=1 rx=0x7f3618008790 tx=0x7f3618011920).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:53:05.599 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:05.571+0000 7f362d7fa700 1 --2- 172.21.15.42:0/2102128026 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f361403ad60 0x7f361403d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:53:05.599 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:05.571+0000 7f362d7fa700 1 -- 172.21.15.42:0/2102128026 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8955+0+0 (secure 0 0 0) 0x7f361804b850 con 0x7f363011b6f0 2019-12-05T05:53:05.599 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:05.571+0000 7f362ffff700 1 --2- 172.21.15.42:0/2102128026 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f361403ad60 0x7f361403d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:53:05.599 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:05.571+0000 7f362ffff700 1 --2- 172.21.15.42:0/2102128026 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f361403ad60 0x7f361403d270 crc :-1 s=READY pgs=702 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:53:05.600 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:05.575+0000 7f3636118700 1 -- 172.21.15.42:0/2102128026 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f363005d110 con 0x7f363011b6f0 2019-12-05T05:53:05.600 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:05.579+0000 7f36277fe700 1 --2- 172.21.15.42:0/2102128026 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f363011b6f0 0x7f363011bb20 secure :-1 s=THROTTLE_DONE pgs=1641 cs=0 l=1 rx=0x7f3618008790 tx=0x7f3618011920).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:53:05.600 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:05.583+0000 7f362d7fa700 1 -- 172.21.15.42:0/2102128026 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f3630123fe0 con 0x7f363011b6f0 2019-12-05T05:53:05.758 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:05.751+0000 7f3636118700 1 -- 172.21.15.42:0/2102128026 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f363005b8c0 con 0x7f363011b6f0 2019-12-05T05:53:05.759 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:05.755+0000 7f36277fe700 1 --2- 172.21.15.42:0/2102128026 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f363011b6f0 0x7f363011bb20 secure :-1 s=THROTTLE_DONE pgs=1641 cs=0 l=1 rx=0x7f3618008790 tx=0x7f3618011920).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:53:05.760 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:05.755+0000 7f362d7fa700 1 -- 172.21.15.42:0/2102128026 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 202 v202) v1 ==== 95+0+1775 (secure 0 0 0) 0x7f36180328b0 con 0x7f363011b6f0 2019-12-05T05:53:05.760 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 202 2019-12-05T05:53:05.760 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:53:05.760 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":202,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":202,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:53:04.600937+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6356},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6356":{"gid":6356,"name":"b","rank":0,"incarnation":199,"state":"up:active","state_seq":26,"addr":"172.21.15.40:6833/1523046776","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1523046776},{"type":"v1","addr":"172.21.15.40:6833","nonce":1523046776}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:53:05.762 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:05.755+0000 7f3636118700 1 -- 172.21.15.42:0/2102128026 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f361403ad60 msgr2=0x7f361403d270 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:53:05.762 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:05.755+0000 7f3636118700 1 --2- 172.21.15.42:0/2102128026 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f361403ad60 0x7f361403d270 crc :-1 s=READY pgs=702 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:53:05.762 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:05.755+0000 7f3636118700 1 -- 172.21.15.42:0/2102128026 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f363011b6f0 msgr2=0x7f363011bb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:53:05.763 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:05.755+0000 7f3636118700 1 --2- 172.21.15.42:0/2102128026 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f363011b6f0 0x7f363011bb20 secure :-1 s=READY pgs=1641 cs=0 l=1 rx=0x7f3618008790 tx=0x7f3618011920).stop 2019-12-05T05:53:05.764 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:05.755+0000 7f3636118700 1 -- 172.21.15.42:0/2102128026 shutdown_connections 2019-12-05T05:53:05.764 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:05.755+0000 7f3636118700 1 --2- 172.21.15.42:0/2102128026 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f361403ad60 0x7f361403d270 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:53:05.764 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:05.755+0000 7f3636118700 1 --2- 172.21.15.42:0/2102128026 >> v2:172.21.15.40:3300/0 conn(0x7f3630133d80 0x7f36301341b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:53:05.764 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:05.755+0000 7f3636118700 1 --2- 172.21.15.42:0/2102128026 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f363011b6f0 0x7f363011bb20 unknown :-1 s=CLOSED pgs=1641 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:53:05.764 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:05.755+0000 7f3636118700 1 -- 172.21.15.42:0/2102128026 shutdown_connections 2019-12-05T05:53:05.764 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:05.755+0000 7f3636118700 1 -- 172.21.15.42:0/2102128026 wait complete. 2019-12-05T05:53:05.765 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:05.755+0000 7f3636118700 1 -- 172.21.15.42:0/2102128026 >> 172.21.15.42:0/2102128026 conn(0x7f36300aabe0 msgr2=0x7f363011c3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:53:05.772 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b has gained rank=0, replacing gid=8430 2019-12-05T05:53:05.772 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 3 secs before reviving mds.a 2019-12-05T05:53:08.773 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.a 2019-12-05T05:53:08.773 INFO:tasks.ceph.mds.a:Restarting daemon 2019-12-05T05:53:08.773 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2019-12-05T05:53:08.777 INFO:tasks.ceph.mds.a:Started 2019-12-05T05:53:08.777 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:53:08.837 INFO:tasks.ceph.mds.a.smithi042.stdout:starting mds.a at 2019-12-05T05:53:08.878 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:08.867+0000 7ff84b8df700 1 Processor -- start 2019-12-05T05:53:08.878 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:08.867+0000 7ff84b8df700 1 -- start start 2019-12-05T05:53:08.879 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:08.867+0000 7ff84b8df700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff8440a7740 0x7ff84408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:53:08.879 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:08.867+0000 7ff84b8df700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff84409fae0 0x7ff84407eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:53:08.879 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:08.867+0000 7ff84b8df700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8440a0710 con 0x7ff84407e720 2019-12-05T05:53:08.879 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:08.867+0000 7ff84b8df700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8440a0b00 con 0x7ff8440a7740 2019-12-05T05:53:08.879 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:08.867+0000 7ff84b8df700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff84408e5b0 con 0x7ff84409fae0 2019-12-05T05:53:08.879 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:08.867+0000 7ff849e7c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff8440a7740 0x7ff84408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:53:08.880 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:08.867+0000 7ff849e7c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff8440a7740 0x7ff84408df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:43372/0 (socket says 172.21.15.42:43372) 2019-12-05T05:53:08.880 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:08.867+0000 7ff849e7c700 1 -- 172.21.15.42:0/2817439503 learned_addr learned my addr 172.21.15.42:0/2817439503 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:53:08.880 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:08.867+0000 7ff84967b700 1 --1- 172.21.15.42:0/2817439503 >> v1:172.21.15.42:6789/0 conn(0x7ff84407e720 0x7ff84409f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:43412/0 (socket says 172.21.15.42:43412) 2019-12-05T05:53:08.880 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:08.867+0000 7ff84a67d700 1 --2- 172.21.15.42:0/2817439503 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff84409fae0 0x7ff84407eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:53:08.880 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:08.867+0000 7ff849e7c700 1 -- 172.21.15.42:0/2817439503 >> v2:172.21.15.42:3300/0 conn(0x7ff8440a7740 msgr2=0x7ff84408df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2019-12-05T05:53:08.880 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:08.867+0000 7ff849e7c700 1 -- 172.21.15.42:0/2817439503 >> v2:172.21.15.42:3300/0 conn(0x7ff8440a7740 msgr2=0x7ff84408df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2019-12-05T05:53:08.881 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:08.867+0000 7ff849e7c700 1 --2- 172.21.15.42:0/2817439503 >> v2:172.21.15.42:3300/0 conn(0x7ff8440a7740 0x7ff84408df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-12-05T05:53:08.881 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:08.867+0000 7ff849e7c700 1 --2- 172.21.15.42:0/2817439503 >> v2:172.21.15.42:3300/0 conn(0x7ff8440a7740 0x7ff84408df80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rx=0 tx=0)._fault waiting 0.200000 2019-12-05T05:53:08.881 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:08.867+0000 7ff84a67d700 1 -- 172.21.15.42:0/2817439503 >> v2:172.21.15.42:3300/0 conn(0x7ff8440a7740 msgr2=0x7ff84408df80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2019-12-05T05:53:08.881 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:08.867+0000 7ff84a67d700 1 --2- 172.21.15.42:0/2817439503 >> v2:172.21.15.42:3300/0 conn(0x7ff8440a7740 0x7ff84408df80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:53:08.881 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:08.867+0000 7ff848e7a700 1 -- 172.21.15.42:0/2817439503 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2829294011 0 0) 0x7ff8440a0710 con 0x7ff84407e720 2019-12-05T05:53:08.881 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:08.867+0000 7ff84a67d700 1 -- 172.21.15.42:0/2817439503 >> v1:172.21.15.42:6789/0 conn(0x7ff84407e720 legacy=0x7ff84409f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:53:08.881 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:08.867+0000 7ff84a67d700 1 -- 172.21.15.42:0/2817439503 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff84407f0a0 con 0x7ff84409fae0 2019-12-05T05:53:08.882 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:08.867+0000 7ff84a67d700 1 --2- 172.21.15.42:0/2817439503 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff84409fae0 0x7ff84407eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7ff834008790 tx=0x7ff83400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:53:08.882 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:08.867+0000 7ff84a67d700 1 --2- 172.21.15.42:0/2817439503 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff84409fae0 0x7ff84407eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7ff834008790 tx=0x7ff83400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:53:08.882 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:08.867+0000 7ff84a67d700 1 --2- 172.21.15.42:0/2817439503 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff84409fae0 0x7ff84407eb50 secure :-1 s=READY pgs=367 cs=0 l=1 rx=0x7ff834008790 tx=0x7ff83400fc70).ready entity=mon.2 client_cookie=6bd8d16f1fdd637 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:53:08.882 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:08.867+0000 7ff84a67d700 1 --2- 172.21.15.42:0/2817439503 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff84409fae0 0x7ff84407eb50 secure :-1 s=THROTTLE_DONE pgs=367 cs=0 l=1 rx=0x7ff834008790 tx=0x7ff83400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:53:08.882 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:08.867+0000 7ff848e7a700 1 -- 172.21.15.42:0/2817439503 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff834008bb0 con 0x7ff84409fae0 2019-12-05T05:53:08.882 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:08.867+0000 7ff84a67d700 1 --2- 172.21.15.42:0/2817439503 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff84409fae0 0x7ff84407eb50 secure :-1 s=THROTTLE_DONE pgs=367 cs=0 l=1 rx=0x7ff834008790 tx=0x7ff83400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:53:08.883 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:08.867+0000 7ff84a67d700 1 --2- 172.21.15.42:0/2817439503 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff84409fae0 0x7ff84407eb50 secure :-1 s=THROTTLE_DONE pgs=367 cs=0 l=1 rx=0x7ff834008790 tx=0x7ff83400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:53:08.883 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:08.867+0000 7ff848e7a700 1 -- 172.21.15.42:0/2817439503 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8340087d0 con 0x7ff84409fae0 2019-12-05T05:53:08.883 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:08.871+0000 7ff848e7a700 1 -- 172.21.15.42:0/2817439503 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff834011860 con 0x7ff84409fae0 2019-12-05T05:53:08.883 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:08.871+0000 7ff84b8df700 1 -- 172.21.15.42:0/2817439503 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff84409fae0 msgr2=0x7ff84407eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:53:08.883 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:08.871+0000 7ff84b8df700 1 --2- 172.21.15.42:0/2817439503 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff84409fae0 0x7ff84407eb50 secure :-1 s=READY pgs=367 cs=0 l=1 rx=0x7ff834008790 tx=0x7ff83400fc70).stop 2019-12-05T05:53:08.884 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:08.871+0000 7ff84b8df700 1 -- 172.21.15.42:0/2817439503 shutdown_connections 2019-12-05T05:53:08.884 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:08.871+0000 7ff84b8df700 1 --2- 172.21.15.42:0/2817439503 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff84409fae0 0x7ff84407eb50 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:53:08.884 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:08.871+0000 7ff84b8df700 1 --2- 172.21.15.42:0/2817439503 >> v2:172.21.15.42:3300/0 conn(0x7ff8440a7740 0x7ff84408df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:53:08.884 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:08.871+0000 7ff84b8df700 1 -- 172.21.15.42:0/2817439503 shutdown_connections 2019-12-05T05:53:08.884 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:08.871+0000 7ff84b8df700 1 -- 172.21.15.42:0/2817439503 wait complete. 2019-12-05T05:53:08.884 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:08.871+0000 7ff84b8df700 1 -- 172.21.15.42:0/2817439503 >> 172.21.15.42:0/2817439503 conn(0x7ff8440aabe0 msgr2=0x7ff8440a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:53:08.885 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:08.871+0000 7ff84b8df700 1 Processor -- start 2019-12-05T05:53:08.885 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:08.871+0000 7ff84b8df700 1 -- start start 2019-12-05T05:53:08.888 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:08.871+0000 7ff84b8df700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff844122020 0x7ff84411a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:53:08.888 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:08.871+0000 7ff84b8df700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff844122450 0x7ff84411f900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:53:08.889 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:08.871+0000 7ff84b8df700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff84407e120 con 0x7ff84411ae90 2019-12-05T05:53:08.889 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:08.871+0000 7ff84b8df700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8440a5d30 con 0x7ff844122020 2019-12-05T05:53:08.889 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:08.871+0000 7ff84b8df700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8440a5fd0 con 0x7ff844122450 2019-12-05T05:53:08.889 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:08.871+0000 7ff84967b700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff844122450 0x7ff84411f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:53:08.889 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:08.871+0000 7ff84967b700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff844122450 0x7ff84411f900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:43380/0 (socket says 172.21.15.42:43380) 2019-12-05T05:53:08.889 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:08.871+0000 7ff84a67d700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7ff84411ae90 0x7ff844130240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:43420/0 (socket says 172.21.15.42:43420) 2019-12-05T05:53:08.890 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:08.871+0000 7ff84967b700 1 -- 172.21.15.42:0/3974680372 learned_addr learned my addr 172.21.15.42:0/3974680372 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:53:08.890 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:08.871+0000 7ff849e7c700 1 --2- 172.21.15.42:0/3974680372 >> v2:172.21.15.40:3300/0 conn(0x7ff844122020 0x7ff84411a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:53:08.890 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:08.875+0000 7ff84967b700 1 -- 172.21.15.42:0/3974680372 >> v2:172.21.15.40:3300/0 conn(0x7ff844122020 msgr2=0x7ff84411a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:53:08.890 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:08.875+0000 7ff84967b700 1 --2- 172.21.15.42:0/3974680372 >> v2:172.21.15.40:3300/0 conn(0x7ff844122020 0x7ff84411a940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:53:08.890 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:08.875+0000 7ff8427fc700 1 -- 172.21.15.42:0/3974680372 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1240451706 0 0) 0x7ff84407e120 con 0x7ff84411ae90 2019-12-05T05:53:08.890 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:08.875+0000 7ff84967b700 1 -- 172.21.15.42:0/3974680372 >> v1:172.21.15.42:6789/0 conn(0x7ff84411ae90 legacy=0x7ff844130240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:53:08.891 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:08.875+0000 7ff84967b700 1 -- 172.21.15.42:0/3974680372 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8440a6270 con 0x7ff844122450 2019-12-05T05:53:08.891 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:08.875+0000 7ff84967b700 1 --2- 172.21.15.42:0/3974680372 >> v2:172.21.15.42:3300/0 conn(0x7ff844122450 0x7ff84411f900 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7ff838000b80 tx=0x7ff838012b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:53:08.891 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:08.875+0000 7ff84967b700 1 --2- 172.21.15.42:0/3974680372 >> v2:172.21.15.42:3300/0 conn(0x7ff844122450 0x7ff84411f900 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7ff838000b80 tx=0x7ff838012b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:53:08.891 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:08.875+0000 7ff84967b700 1 --2- 172.21.15.42:0/3974680372 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff844122450 0x7ff84411f900 secure :-1 s=READY pgs=1646 cs=0 l=1 rx=0x7ff838000b80 tx=0x7ff838012b50).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:53:08.891 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:08.875+0000 7ff84967b700 1 --2- 172.21.15.42:0/3974680372 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff844122450 0x7ff84411f900 secure :-1 s=THROTTLE_DONE pgs=1646 cs=0 l=1 rx=0x7ff838000b80 tx=0x7ff838012b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:53:08.891 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:08.875+0000 7ff8427fc700 1 -- 172.21.15.42:0/3974680372 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff83800ed00 con 0x7ff844122450 2019-12-05T05:53:08.891 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:08.875+0000 7ff84b8df700 1 -- 172.21.15.42:0/3974680372 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8440a5870 con 0x7ff844122450 2019-12-05T05:53:08.892 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:08.875+0000 7ff84967b700 1 --2- 172.21.15.42:0/3974680372 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff844122450 0x7ff84411f900 secure :-1 s=THROTTLE_DONE pgs=1646 cs=0 l=1 rx=0x7ff838000b80 tx=0x7ff838012b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:53:08.892 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:08.875+0000 7ff8427fc700 1 -- 172.21.15.42:0/3974680372 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff83800e920 con 0x7ff844122450 2019-12-05T05:53:08.892 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:08.875+0000 7ff84967b700 1 --2- 172.21.15.42:0/3974680372 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff844122450 0x7ff84411f900 secure :-1 s=THROTTLE_DONE pgs=1646 cs=0 l=1 rx=0x7ff838000b80 tx=0x7ff838012b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:53:08.892 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:08.875+0000 7ff84b8df700 1 -- 172.21.15.42:0/3974680372 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff844120070 con 0x7ff844122450 2019-12-05T05:53:08.893 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:08.875+0000 7ff8427fc700 1 -- 172.21.15.42:0/3974680372 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff83800a9c0 con 0x7ff844122450 2019-12-05T05:53:08.893 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:08.875+0000 7ff84967b700 1 --2- 172.21.15.42:0/3974680372 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff844122450 0x7ff84411f900 secure :-1 s=THROTTLE_DONE pgs=1646 cs=0 l=1 rx=0x7ff838000b80 tx=0x7ff838012b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:53:08.893 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:08.875+0000 7ff8427fc700 1 -- 172.21.15.42:0/3974680372 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7ff83800aca0 con 0x7ff844122450 2019-12-05T05:53:08.893 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:08.875+0000 7ff84967b700 1 --2- 172.21.15.42:0/3974680372 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff844122450 0x7ff84411f900 secure :-1 s=THROTTLE_DONE pgs=1646 cs=0 l=1 rx=0x7ff838000b80 tx=0x7ff838012b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:53:08.893 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:08.875+0000 7ff8427fc700 1 --2- 172.21.15.42:0/3974680372 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff82403aca0 0x7ff82403d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:53:08.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:08.875+0000 7ff8427fc700 1 -- 172.21.15.42:0/3974680372 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8955+0+0 (secure 0 0 0) 0x7ff83804cf10 con 0x7ff844122450 2019-12-05T05:53:08.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:08.875+0000 7ff849e7c700 1 --2- 172.21.15.42:0/3974680372 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff82403aca0 0x7ff82403d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:53:08.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:08.875+0000 7ff849e7c700 1 --2- 172.21.15.42:0/3974680372 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff82403aca0 0x7ff82403d1b0 crc :-1 s=READY pgs=703 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:53:08.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:08.883+0000 7ff84b8df700 1 -- 172.21.15.42:0/3974680372 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff84405d110 con 0x7ff844122450 2019-12-05T05:53:08.895 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:08.891+0000 7ff84967b700 1 --2- 172.21.15.42:0/3974680372 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff844122450 0x7ff84411f900 secure :-1 s=THROTTLE_DONE pgs=1646 cs=0 l=1 rx=0x7ff838000b80 tx=0x7ff838012b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:53:08.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:08.891+0000 7ff8427fc700 1 -- 172.21.15.42:0/3974680372 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7ff838033f70 con 0x7ff844122450 2019-12-05T05:53:09.072 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:09.067+0000 7ff84b8df700 1 -- 172.21.15.42:0/3974680372 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff8440a71e0 con 0x7ff844122450 2019-12-05T05:53:09.073 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:09.067+0000 7ff84967b700 1 --2- 172.21.15.42:0/3974680372 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff844122450 0x7ff84411f900 secure :-1 s=THROTTLE_DONE pgs=1646 cs=0 l=1 rx=0x7ff838000b80 tx=0x7ff838012b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:53:09.073 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:09.067+0000 7ff8427fc700 1 -- 172.21.15.42:0/3974680372 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 202 v202) v1 ==== 95+0+1775 (secure 0 0 0) 0x7ff8380341d0 con 0x7ff844122450 2019-12-05T05:53:09.073 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 202 2019-12-05T05:53:09.073 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:53:09.073 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":202,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":202,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:53:04.600937+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6356},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6356":{"gid":6356,"name":"b","rank":0,"incarnation":199,"state":"up:active","state_seq":26,"addr":"172.21.15.40:6833/1523046776","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1523046776},{"type":"v1","addr":"172.21.15.40:6833","nonce":1523046776}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:53:09.075 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:09.067+0000 7ff84b8df700 1 -- 172.21.15.42:0/3974680372 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff82403aca0 msgr2=0x7ff82403d1b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:53:09.075 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:09.067+0000 7ff84b8df700 1 --2- 172.21.15.42:0/3974680372 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff82403aca0 0x7ff82403d1b0 crc :-1 s=READY pgs=703 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:53:09.075 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:09.067+0000 7ff84b8df700 1 -- 172.21.15.42:0/3974680372 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff844122450 msgr2=0x7ff84411f900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:53:09.075 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:09.067+0000 7ff84b8df700 1 --2- 172.21.15.42:0/3974680372 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff844122450 0x7ff84411f900 secure :-1 s=READY pgs=1646 cs=0 l=1 rx=0x7ff838000b80 tx=0x7ff838012b50).stop 2019-12-05T05:53:09.076 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:09.071+0000 7ff84b8df700 1 -- 172.21.15.42:0/3974680372 shutdown_connections 2019-12-05T05:53:09.076 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:09.071+0000 7ff84b8df700 1 --2- 172.21.15.42:0/3974680372 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff82403aca0 0x7ff82403d1b0 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:53:09.076 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:09.071+0000 7ff84b8df700 1 --2- 172.21.15.42:0/3974680372 >> v2:172.21.15.40:3300/0 conn(0x7ff844122020 0x7ff84411a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:53:09.076 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:09.071+0000 7ff84b8df700 1 --2- 172.21.15.42:0/3974680372 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff844122450 0x7ff84411f900 unknown :-1 s=CLOSED pgs=1646 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:53:09.076 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:09.071+0000 7ff84b8df700 1 -- 172.21.15.42:0/3974680372 shutdown_connections 2019-12-05T05:53:09.077 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:09.071+0000 7ff84b8df700 1 -- 172.21.15.42:0/3974680372 wait complete. 2019-12-05T05:53:09.077 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:09.071+0000 7ff84b8df700 1 -- 172.21.15.42:0/3974680372 >> 172.21.15.42:0/3974680372 conn(0x7ff84411b830 msgr2=0x7ff84408eeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:53:09.085 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2019-12-05T05:53:11.086 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:53:11.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:11.179+0000 7f0cb5d1b700 1 Processor -- start 2019-12-05T05:53:11.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:11.179+0000 7f0cb5d1b700 1 -- start start 2019-12-05T05:53:11.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:11.183+0000 7f0cb5d1b700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0cb008df80 0x7f0cb009f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:53:11.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:11.183+0000 7f0cb5d1b700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0cb009f830 0x7f0cb014efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:53:11.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:11.183+0000 7f0cb5d1b700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cb009fc60 con 0x7f0cb00a7740 2019-12-05T05:53:11.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:11.183+0000 7f0cb5d1b700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0cb00a0b00 con 0x7f0cb008df80 2019-12-05T05:53:11.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:11.183+0000 7f0cb5d1b700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cb008e5b0 con 0x7f0cb009f830 2019-12-05T05:53:11.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:11.183+0000 7f0caf7fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0cb008df80 0x7f0cb009f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:53:11.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:11.183+0000 7f0caf7fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0cb008df80 0x7f0cb009f2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:43392/0 (socket says 172.21.15.42:43392) 2019-12-05T05:53:11.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:11.183+0000 7f0caf7fe700 1 -- 172.21.15.42:0/1624007284 learned_addr learned my addr 172.21.15.42:0/1624007284 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:53:11.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:11.183+0000 7f0cb4ab9700 1 --2- 172.21.15.42:0/1624007284 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0cb009f830 0x7f0cb014efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:53:11.193 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:11.183+0000 7f0caf7fe700 1 -- 172.21.15.42:0/1624007284 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0cb009f830 msgr2=0x7f0cb014efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:53:11.193 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:11.183+0000 7f0caf7fe700 1 --2- 172.21.15.42:0/1624007284 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0cb009f830 0x7f0cb014efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:53:11.193 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:11.183+0000 7f0caf7fe700 1 -- 172.21.15.42:0/1624007284 >> v1:172.21.15.40:6789/0 conn(0x7f0cb00a7740 legacy=0x7f0cb007e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:53:11.193 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:11.183+0000 7f0caf7fe700 1 -- 172.21.15.42:0/1624007284 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cb00a0710 con 0x7f0cb008df80 2019-12-05T05:53:11.193 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:11.183+0000 7f0cb4ab9700 1 --2- 172.21.15.42:0/1624007284 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0cb009f830 0x7f0cb014efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).send_auth_request state changed! 2019-12-05T05:53:11.194 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:11.183+0000 7f0caf7fe700 1 --2- 172.21.15.42:0/1624007284 >> v2:172.21.15.42:3300/0 conn(0x7f0cb008df80 0x7f0cb009f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f0c98008790 tx=0x7f0c9800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:53:11.194 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:11.183+0000 7f0caf7fe700 1 --2- 172.21.15.42:0/1624007284 >> v2:172.21.15.42:3300/0 conn(0x7f0cb008df80 0x7f0cb009f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f0c98008790 tx=0x7f0c9800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:53:11.194 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:11.183+0000 7f0caf7fe700 1 --2- 172.21.15.42:0/1624007284 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0cb008df80 0x7f0cb009f2e0 secure :-1 s=READY pgs=1647 cs=0 l=1 rx=0x7f0c98008790 tx=0x7f0c9800fd30).ready entity=mon.1 client_cookie=c2cd16b6c1eb4c8f server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:53:11.194 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:11.183+0000 7f0caf7fe700 1 --2- 172.21.15.42:0/1624007284 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0cb008df80 0x7f0cb009f2e0 secure :-1 s=THROTTLE_DONE pgs=1647 cs=0 l=1 rx=0x7f0c98008790 tx=0x7f0c9800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:53:11.194 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:11.183+0000 7f0caeffd700 1 -- 172.21.15.42:0/1624007284 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0c98008c90 con 0x7f0cb008df80 2019-12-05T05:53:11.194 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:11.183+0000 7f0caf7fe700 1 --2- 172.21.15.42:0/1624007284 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0cb008df80 0x7f0cb009f2e0 secure :-1 s=THROTTLE_DONE pgs=1647 cs=0 l=1 rx=0x7f0c98008790 tx=0x7f0c9800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:53:11.194 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:11.183+0000 7f0caf7fe700 1 --2- 172.21.15.42:0/1624007284 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0cb008df80 0x7f0cb009f2e0 secure :-1 s=THROTTLE_DONE pgs=1647 cs=0 l=1 rx=0x7f0c98008790 tx=0x7f0c9800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:53:11.195 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:11.183+0000 7f0caeffd700 1 -- 172.21.15.42:0/1624007284 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c980088b0 con 0x7f0cb008df80 2019-12-05T05:53:11.195 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:11.183+0000 7f0caeffd700 1 -- 172.21.15.42:0/1624007284 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0c98011b10 con 0x7f0cb008df80 2019-12-05T05:53:11.195 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:11.183+0000 7f0cb5d1b700 1 -- 172.21.15.42:0/1624007284 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0cb008df80 msgr2=0x7f0cb009f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:53:11.195 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:11.183+0000 7f0cb5d1b700 1 --2- 172.21.15.42:0/1624007284 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0cb008df80 0x7f0cb009f2e0 secure :-1 s=READY pgs=1647 cs=0 l=1 rx=0x7f0c98008790 tx=0x7f0c9800fd30).stop 2019-12-05T05:53:11.195 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:11.183+0000 7f0cb5d1b700 1 -- 172.21.15.42:0/1624007284 shutdown_connections 2019-12-05T05:53:11.195 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:11.183+0000 7f0cb5d1b700 1 --2- 172.21.15.42:0/1624007284 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0cb009f830 0x7f0cb014efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:53:11.196 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:11.183+0000 7f0cb5d1b700 1 --2- 172.21.15.42:0/1624007284 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0cb008df80 0x7f0cb009f2e0 unknown :-1 s=CLOSED pgs=1647 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:53:11.196 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:11.183+0000 7f0cb5d1b700 1 -- 172.21.15.42:0/1624007284 shutdown_connections 2019-12-05T05:53:11.196 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:11.183+0000 7f0cb5d1b700 1 -- 172.21.15.42:0/1624007284 wait complete. 2019-12-05T05:53:11.196 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:11.183+0000 7f0cb5d1b700 1 -- 172.21.15.42:0/1624007284 >> 172.21.15.42:0/1624007284 conn(0x7f0cb00aabe0 msgr2=0x7f0cb00a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:53:11.201 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:11.187+0000 7f0cb5d1b700 1 Processor -- start 2019-12-05T05:53:11.202 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:11.187+0000 7f0cb5d1b700 1 -- start start 2019-12-05T05:53:11.202 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:11.187+0000 7f0cb5d1b700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0cb0119580 0x7f0cb0117b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:53:11.202 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:11.187+0000 7f0cb5d1b700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cb01178c0 con 0x7f0cb011b690 2019-12-05T05:53:11.202 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:11.187+0000 7f0cb5d1b700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cb0131830 con 0x7f0cb01180e0 2019-12-05T05:53:11.202 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:11.187+0000 7f0cb5d1b700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0cb0123c10 con 0x7f0cb0119580 2019-12-05T05:53:11.203 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:11.187+0000 7f0caffff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0cb0119580 0x7f0cb0117b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:53:11.203 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:11.187+0000 7f0cb4ab9700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f0cb01180e0 0x7f0cb0137cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:43438/0 (socket says 172.21.15.42:43438) 2019-12-05T05:53:11.203 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:11.187+0000 7f0cb4ab9700 1 -- 172.21.15.42:0/942777521 learned_addr learned my addr 172.21.15.42:0/942777521 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:53:11.203 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:11.187+0000 7f0caffff700 1 -- 172.21.15.42:0/942777521 >> v1:172.21.15.42:6789/0 conn(0x7f0cb01180e0 legacy=0x7f0cb0137cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:53:11.203 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:11.187+0000 7f0caffff700 1 -- 172.21.15.42:0/942777521 >> v1:172.21.15.40:6789/0 conn(0x7f0cb011b690 legacy=0x7f0cb0131030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:53:11.204 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:11.187+0000 7f0cacff9700 1 -- 172.21.15.42:0/942777521 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2374332362 0 0) 0x7f0cb0131830 con 0x7f0cb01180e0 2019-12-05T05:53:11.204 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:11.187+0000 7f0caffff700 1 -- 172.21.15.42:0/942777521 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cb00a6a70 con 0x7f0cb0119580 2019-12-05T05:53:11.204 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:11.187+0000 7f0caffff700 1 --2- 172.21.15.42:0/942777521 >> v2:172.21.15.42:3300/0 conn(0x7f0cb0119580 0x7f0cb0117b90 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f0ca80017b0 tx=0x7f0c9800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:53:11.204 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:11.187+0000 7f0caffff700 1 --2- 172.21.15.42:0/942777521 >> v2:172.21.15.42:3300/0 conn(0x7f0cb0119580 0x7f0cb0117b90 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f0ca80017b0 tx=0x7f0c9800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:53:11.205 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:11.187+0000 7f0caffff700 1 --2- 172.21.15.42:0/942777521 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0cb0119580 0x7f0cb0117b90 secure :-1 s=READY pgs=1649 cs=0 l=1 rx=0x7f0ca80017b0 tx=0x7f0c9800fd30).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:53:11.205 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:11.191+0000 7f0caffff700 1 --2- 172.21.15.42:0/942777521 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0cb0119580 0x7f0cb0117b90 secure :-1 s=THROTTLE_DONE pgs=1649 cs=0 l=1 rx=0x7f0ca80017b0 tx=0x7f0c9800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:53:11.205 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:11.191+0000 7f0cacff9700 1 -- 172.21.15.42:0/942777521 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0ca8010550 con 0x7f0cb0119580 2019-12-05T05:53:11.205 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:11.191+0000 7f0caffff700 1 --2- 172.21.15.42:0/942777521 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0cb0119580 0x7f0cb0117b90 secure :-1 s=THROTTLE_DONE pgs=1649 cs=0 l=1 rx=0x7f0ca80017b0 tx=0x7f0c9800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:53:11.205 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:11.191+0000 7f0caffff700 1 --2- 172.21.15.42:0/942777521 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0cb0119580 0x7f0cb0117b90 secure :-1 s=THROTTLE_DONE pgs=1649 cs=0 l=1 rx=0x7f0ca80017b0 tx=0x7f0c9800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:53:11.206 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:11.191+0000 7f0cacff9700 1 -- 172.21.15.42:0/942777521 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ca8010170 con 0x7f0cb0119580 2019-12-05T05:53:11.206 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:11.191+0000 7f0cacff9700 1 -- 172.21.15.42:0/942777521 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0ca800cb20 con 0x7f0cb0119580 2019-12-05T05:53:11.206 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:11.191+0000 7f0cb5d1b700 1 -- 172.21.15.42:0/942777521 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cb011aac0 con 0x7f0cb0119580 2019-12-05T05:53:11.206 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:11.191+0000 7f0cb5d1b700 1 -- 172.21.15.42:0/942777521 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cb0139500 con 0x7f0cb0119580 2019-12-05T05:53:11.207 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:11.191+0000 7f0caffff700 1 --2- 172.21.15.42:0/942777521 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0cb0119580 0x7f0cb0117b90 secure :-1 s=THROTTLE_DONE pgs=1649 cs=0 l=1 rx=0x7f0ca80017b0 tx=0x7f0c9800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:53:11.207 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:11.191+0000 7f0cacff9700 1 -- 172.21.15.42:0/942777521 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f0ca80066e0 con 0x7f0cb0119580 2019-12-05T05:53:11.207 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:11.191+0000 7f0caffff700 1 --2- 172.21.15.42:0/942777521 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0cb0119580 0x7f0cb0117b90 secure :-1 s=THROTTLE_DONE pgs=1649 cs=0 l=1 rx=0x7f0ca80017b0 tx=0x7f0c9800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:53:11.207 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:11.191+0000 7f0cacff9700 1 --2- 172.21.15.42:0/942777521 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0c9403acc0 0x7f0c9403d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:53:11.208 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:11.191+0000 7f0cacff9700 1 -- 172.21.15.42:0/942777521 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8955+0+0 (secure 0 0 0) 0x7f0ca804afe0 con 0x7f0cb0119580 2019-12-05T05:53:11.209 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:11.191+0000 7f0caf7fe700 1 --2- 172.21.15.42:0/942777521 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0c9403acc0 0x7f0c9403d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:53:11.209 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:11.191+0000 7f0caf7fe700 1 --2- 172.21.15.42:0/942777521 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0c9403acc0 0x7f0c9403d1d0 crc :-1 s=READY pgs=705 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:53:11.209 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:11.195+0000 7f0cb5d1b700 1 -- 172.21.15.42:0/942777521 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cb005d110 con 0x7f0cb0119580 2019-12-05T05:53:11.210 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:11.199+0000 7f0caffff700 1 --2- 172.21.15.42:0/942777521 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0cb0119580 0x7f0cb0117b90 secure :-1 s=THROTTLE_DONE pgs=1649 cs=0 l=1 rx=0x7f0ca80017b0 tx=0x7f0c9800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:53:11.210 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:11.199+0000 7f0cacff9700 1 -- 172.21.15.42:0/942777521 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f0ca8032000 con 0x7f0cb0119580 2019-12-05T05:53:11.369 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:11.363+0000 7f0cb5d1b700 1 -- 172.21.15.42:0/942777521 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0cb00a71e0 con 0x7f0cb0119580 2019-12-05T05:53:11.370 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:11.363+0000 7f0caffff700 1 --2- 172.21.15.42:0/942777521 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0cb0119580 0x7f0cb0117b90 secure :-1 s=THROTTLE_DONE pgs=1649 cs=0 l=1 rx=0x7f0ca80017b0 tx=0x7f0c9800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:53:11.370 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:11.363+0000 7f0cacff9700 1 -- 172.21.15.42:0/942777521 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 203 v203) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f0ca8032260 con 0x7f0cb0119580 2019-12-05T05:53:11.370 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 203 2019-12-05T05:53:11.370 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:53:11.370 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":203,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":8614,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6835/2511123581","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2511123581},{"type":"v1","addr":"172.21.15.42:6835","nonce":2511123581}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":203}],"filesystems":[{"mdsmap":{"epoch":202,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:53:04.600937+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6356},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6356":{"gid":6356,"name":"b","rank":0,"incarnation":199,"state":"up:active","state_seq":26,"addr":"172.21.15.40:6833/1523046776","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1523046776},{"type":"v1","addr":"172.21.15.40:6833","nonce":1523046776}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:53:11.372 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:11.367+0000 7f0cb5d1b700 1 -- 172.21.15.42:0/942777521 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0c9403acc0 msgr2=0x7f0c9403d1d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:53:11.372 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:11.367+0000 7f0cb5d1b700 1 --2- 172.21.15.42:0/942777521 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0c9403acc0 0x7f0c9403d1d0 crc :-1 s=READY pgs=705 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:53:11.372 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:11.367+0000 7f0cb5d1b700 1 -- 172.21.15.42:0/942777521 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0cb0119580 msgr2=0x7f0cb0117b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:53:11.373 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:11.367+0000 7f0cb5d1b700 1 --2- 172.21.15.42:0/942777521 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0cb0119580 0x7f0cb0117b90 secure :-1 s=READY pgs=1649 cs=0 l=1 rx=0x7f0ca80017b0 tx=0x7f0c9800fd30).stop 2019-12-05T05:53:11.373 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:11.367+0000 7f0cb5d1b700 1 -- 172.21.15.42:0/942777521 shutdown_connections 2019-12-05T05:53:11.373 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:11.367+0000 7f0cb5d1b700 1 --2- 172.21.15.42:0/942777521 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0c9403acc0 0x7f0c9403d1d0 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:53:11.373 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:11.367+0000 7f0cb5d1b700 1 --2- 172.21.15.42:0/942777521 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0cb0119580 0x7f0cb0117b90 unknown :-1 s=CLOSED pgs=1649 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:53:11.373 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:11.367+0000 7f0cb5d1b700 1 -- 172.21.15.42:0/942777521 shutdown_connections 2019-12-05T05:53:11.373 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:11.367+0000 7f0cb5d1b700 1 -- 172.21.15.42:0/942777521 wait complete. 2019-12-05T05:53:11.373 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:53:11.367+0000 7f0cb5d1b700 1 -- 172.21.15.42:0/942777521 >> 172.21.15.42:0/942777521 conn(0x7f0cb00a6d70 msgr2=0x7f0cb011c9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:53:11.388 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a reported in up:standby state 2019-12-05T05:53:11.388 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 92 secs before thrashing 2019-12-05T05:53:12.206 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:53:12.210 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:53:42.253 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:53:42.258 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:54:12.306 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:54:12.309 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:54:42.353 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:54:42.357 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:54:43.390 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:54:43.496 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.486+0000 7f3ea2a6c700 1 Processor -- start 2019-12-05T05:54:43.496 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.486+0000 7f3ea2a6c700 1 -- start start 2019-12-05T05:54:43.497 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.486+0000 7f3ea2a6c700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3e9c08df70 0x7f3e9c09f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:54:43.497 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.486+0000 7f3ea2a6c700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e9c09fc50 con 0x7f3e9c0a7730 2019-12-05T05:54:43.497 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.486+0000 7f3ea2a6c700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e9c0a0700 con 0x7f3e9c09f820 2019-12-05T05:54:43.497 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.486+0000 7f3ea2a6c700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e9c0a0af0 con 0x7f3e9c08df70 2019-12-05T05:54:43.497 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.486+0000 7f3ea180a700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f3e9c09f820 0x7f3e9c14efd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:43766/0 (socket says 172.21.15.42:43766) 2019-12-05T05:54:43.497 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.486+0000 7f3ea180a700 1 -- 172.21.15.42:0/1129851583 learned_addr learned my addr 172.21.15.42:0/1129851583 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:54:43.497 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.486+0000 7f3ea0808700 1 --2- 172.21.15.42:0/1129851583 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3e9c08df70 0x7f3e9c09f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:54:43.498 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.486+0000 7f3e8ffff700 1 -- 172.21.15.42:0/1129851583 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 84205127 0 0) 0x7f3e9c0a0700 con 0x7f3e9c09f820 2019-12-05T05:54:43.498 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.486+0000 7f3e8ffff700 1 -- 172.21.15.42:0/1129851583 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e880030b0 con 0x7f3e9c09f820 2019-12-05T05:54:43.498 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.486+0000 7f3e8ffff700 1 -- 172.21.15.42:0/1129851583 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3468478326 0 0) 0x7f3e9c09fc50 con 0x7f3e9c0a7730 2019-12-05T05:54:43.498 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.486+0000 7f3e8ffff700 1 -- 172.21.15.42:0/1129851583 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e9c0a0700 con 0x7f3e9c0a7730 2019-12-05T05:54:43.498 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.486+0000 7f3ea0808700 1 -- 172.21.15.42:0/1129851583 >> v1:172.21.15.42:6789/0 conn(0x7f3e9c09f820 legacy=0x7f3e9c14efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:54:43.499 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.486+0000 7f3ea0808700 1 -- 172.21.15.42:0/1129851583 >> v1:172.21.15.40:6789/0 conn(0x7f3e9c0a7730 legacy=0x7f3e9c07e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:54:43.499 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.486+0000 7f3ea0808700 1 -- 172.21.15.42:0/1129851583 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e9c07ef10 con 0x7f3e9c08df70 2019-12-05T05:54:43.499 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.486+0000 7f3ea0808700 1 --2- 172.21.15.42:0/1129851583 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3e9c08df70 0x7f3e9c09f2d0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f3e900032b0 tx=0x7f3e9000ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:54:43.499 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.486+0000 7f3ea0808700 1 --2- 172.21.15.42:0/1129851583 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3e9c08df70 0x7f3e9c09f2d0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f3e900032b0 tx=0x7f3e9000ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:54:43.499 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.486+0000 7f3ea0808700 1 --2- 172.21.15.42:0/1129851583 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3e9c08df70 0x7f3e9c09f2d0 secure :-1 s=READY pgs=370 cs=0 l=1 rx=0x7f3e900032b0 tx=0x7f3e9000ff70).ready entity=mon.2 client_cookie=ae90f1cca1a44add server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:54:43.500 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.486+0000 7f3ea0808700 1 --2- 172.21.15.42:0/1129851583 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3e9c08df70 0x7f3e9c09f2d0 secure :-1 s=THROTTLE_DONE pgs=370 cs=0 l=1 rx=0x7f3e900032b0 tx=0x7f3e9000ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:54:43.500 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.486+0000 7f3ea0808700 1 --2- 172.21.15.42:0/1129851583 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3e9c08df70 0x7f3e9c09f2d0 secure :-1 s=THROTTLE_DONE pgs=370 cs=0 l=1 rx=0x7f3e900032b0 tx=0x7f3e9000ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:54:43.500 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.486+0000 7f3e8ffff700 1 -- 172.21.15.42:0/1129851583 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3e90011a10 con 0x7f3e9c08df70 2019-12-05T05:54:43.500 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.486+0000 7f3ea0808700 1 --2- 172.21.15.42:0/1129851583 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3e9c08df70 0x7f3e9c09f2d0 secure :-1 s=THROTTLE_DONE pgs=370 cs=0 l=1 rx=0x7f3e900032b0 tx=0x7f3e9000ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:54:43.500 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.486+0000 7f3e8ffff700 1 -- 172.21.15.42:0/1129851583 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e90011630 con 0x7f3e9c08df70 2019-12-05T05:54:43.500 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.486+0000 7f3e8ffff700 1 -- 172.21.15.42:0/1129851583 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3e9000a160 con 0x7f3e9c08df70 2019-12-05T05:54:43.501 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.486+0000 7f3ea2a6c700 1 -- 172.21.15.42:0/1129851583 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3e9c08df70 msgr2=0x7f3e9c09f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:54:43.501 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.486+0000 7f3ea2a6c700 1 --2- 172.21.15.42:0/1129851583 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3e9c08df70 0x7f3e9c09f2d0 secure :-1 s=READY pgs=370 cs=0 l=1 rx=0x7f3e900032b0 tx=0x7f3e9000ff70).stop 2019-12-05T05:54:43.501 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.490+0000 7f3ea2a6c700 1 -- 172.21.15.42:0/1129851583 shutdown_connections 2019-12-05T05:54:43.501 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.490+0000 7f3ea2a6c700 1 --2- 172.21.15.42:0/1129851583 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3e9c08df70 0x7f3e9c09f2d0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:54:43.501 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.490+0000 7f3ea2a6c700 1 -- 172.21.15.42:0/1129851583 shutdown_connections 2019-12-05T05:54:43.502 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.490+0000 7f3ea2a6c700 1 -- 172.21.15.42:0/1129851583 wait complete. 2019-12-05T05:54:43.502 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.490+0000 7f3ea2a6c700 1 -- 172.21.15.42:0/1129851583 >> 172.21.15.42:0/1129851583 conn(0x7f3e9c0aabd0 msgr2=0x7f3e9c0a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:54:43.512 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.490+0000 7f3ea2a6c700 1 Processor -- start 2019-12-05T05:54:43.513 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.490+0000 7f3ea2a6c700 1 -- start start 2019-12-05T05:54:43.513 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.490+0000 7f3ea2a6c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3e9c063d90 0x7f3e9c08eea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:54:43.513 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.490+0000 7f3ea2a6c700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f3e9c125500 0x7f3e9c125930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:54:43.513 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.490+0000 7f3ea2a6c700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e9c0651a0 con 0x7f3e9c064d70 2019-12-05T05:54:43.514 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.490+0000 7f3ea2a6c700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e9c065520 con 0x7f3e9c125500 2019-12-05T05:54:43.514 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.490+0000 7f3ea2a6c700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e9c0657c0 con 0x7f3e9c063d90 2019-12-05T05:54:43.514 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.490+0000 7f3ea1009700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3e9c063d90 0x7f3e9c08eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:54:43.514 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.490+0000 7f3ea1009700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3e9c063d90 0x7f3e9c08eea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:43732/0 (socket says 172.21.15.42:43732) 2019-12-05T05:54:43.515 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.490+0000 7f3ea1009700 1 -- 172.21.15.42:0/1298874202 learned_addr learned my addr 172.21.15.42:0/1298874202 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:54:43.515 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.490+0000 7f3ea180a700 1 --2- 172.21.15.42:0/1298874202 >> v2:172.21.15.40:3300/0 conn(0x7f3e9c125500 0x7f3e9c125930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:54:43.515 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.494+0000 7f3ea180a700 1 -- 172.21.15.42:0/1298874202 >> v2:172.21.15.42:3300/0 conn(0x7f3e9c063d90 msgr2=0x7f3e9c08eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:54:43.515 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.494+0000 7f3ea180a700 1 --2- 172.21.15.42:0/1298874202 >> v2:172.21.15.42:3300/0 conn(0x7f3e9c063d90 0x7f3e9c08eea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:54:43.516 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.494+0000 7f3ea180a700 1 -- 172.21.15.42:0/1298874202 >> v1:172.21.15.42:6789/0 conn(0x7f3e9c064d70 legacy=0x7f3e9c14fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:54:43.516 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.494+0000 7f3ea180a700 1 -- 172.21.15.42:0/1298874202 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e9c125e80 con 0x7f3e9c125500 2019-12-05T05:54:43.516 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.494+0000 7f3ea180a700 1 --2- 172.21.15.42:0/1298874202 >> v2:172.21.15.40:3300/0 conn(0x7f3e9c125500 0x7f3e9c125930 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f3e9c08f990 tx=0x7f3e94010830).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:54:43.516 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.494+0000 7f3ea180a700 1 --2- 172.21.15.42:0/1298874202 >> v2:172.21.15.40:3300/0 conn(0x7f3e9c125500 0x7f3e9c125930 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f3e9c08f990 tx=0x7f3e94010830).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:54:43.517 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.494+0000 7f3ea180a700 1 --2- 172.21.15.42:0/1298874202 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e9c125500 0x7f3e9c125930 secure :-1 s=READY pgs=1271 cs=0 l=1 rx=0x7f3e9c08f990 tx=0x7f3e94010830).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:54:43.517 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.494+0000 7f3ea180a700 1 --2- 172.21.15.42:0/1298874202 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e9c125500 0x7f3e9c125930 secure :-1 s=THROTTLE_DONE pgs=1271 cs=0 l=1 rx=0x7f3e9c08f990 tx=0x7f3e94010830).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:54:43.517 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.494+0000 7f3e8dffb700 1 -- 172.21.15.42:0/1298874202 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3e940059f0 con 0x7f3e9c125500 2019-12-05T05:54:43.517 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.494+0000 7f3ea180a700 1 --2- 172.21.15.42:0/1298874202 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e9c125500 0x7f3e9c125930 secure :-1 s=THROTTLE_DONE pgs=1271 cs=0 l=1 rx=0x7f3e9c08f990 tx=0x7f3e94010830).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:54:43.518 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.494+0000 7f3e8dffb700 1 -- 172.21.15.42:0/1298874202 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e94011e10 con 0x7f3e9c125500 2019-12-05T05:54:43.518 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.494+0000 7f3ea180a700 1 --2- 172.21.15.42:0/1298874202 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e9c125500 0x7f3e9c125930 secure :-1 s=THROTTLE_DONE pgs=1271 cs=0 l=1 rx=0x7f3e9c08f990 tx=0x7f3e94010830).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:54:43.518 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.494+0000 7f3e8dffb700 1 -- 172.21.15.42:0/1298874202 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3e940060c0 con 0x7f3e9c125500 2019-12-05T05:54:43.518 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.494+0000 7f3ea2a6c700 1 -- 172.21.15.42:0/1298874202 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e9c12fe70 con 0x7f3e9c125500 2019-12-05T05:54:43.519 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.494+0000 7f3ea2a6c700 1 -- 172.21.15.42:0/1298874202 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e9c130340 con 0x7f3e9c125500 2019-12-05T05:54:43.519 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.494+0000 7f3ea180a700 1 --2- 172.21.15.42:0/1298874202 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e9c125500 0x7f3e9c125930 secure :-1 s=THROTTLE_DONE pgs=1271 cs=0 l=1 rx=0x7f3e9c08f990 tx=0x7f3e94010830).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:54:43.519 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.494+0000 7f3e8dffb700 1 -- 172.21.15.42:0/1298874202 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f3e940063a0 con 0x7f3e9c125500 2019-12-05T05:54:43.519 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.494+0000 7f3ea180a700 1 --2- 172.21.15.42:0/1298874202 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e9c125500 0x7f3e9c125930 secure :-1 s=THROTTLE_DONE pgs=1271 cs=0 l=1 rx=0x7f3e9c08f990 tx=0x7f3e94010830).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:54:43.520 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.494+0000 7f3e8dffb700 1 --2- 172.21.15.42:0/1298874202 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3e7c03ae20 0x7f3e7c03d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:54:43.520 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.494+0000 7f3e8dffb700 1 -- 172.21.15.42:0/1298874202 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8955+0+0 (secure 0 0 0) 0x7f3e9404ae60 con 0x7f3e9c125500 2019-12-05T05:54:43.528 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.494+0000 7f3ea1009700 1 --2- 172.21.15.42:0/1298874202 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3e7c03ae20 0x7f3e7c03d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:54:43.528 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.498+0000 7f3ea1009700 1 --2- 172.21.15.42:0/1298874202 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3e7c03ae20 0x7f3e7c03d330 crc :-1 s=READY pgs=707 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:54:43.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.502+0000 7f3ea2a6c700 1 -- 172.21.15.42:0/1298874202 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e9c05d100 con 0x7f3e9c125500 2019-12-05T05:54:43.530 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.510+0000 7f3ea180a700 1 --2- 172.21.15.42:0/1298874202 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e9c125500 0x7f3e9c125930 secure :-1 s=THROTTLE_DONE pgs=1271 cs=0 l=1 rx=0x7f3e9c08f990 tx=0x7f3e94010830).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:54:43.530 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.510+0000 7f3e8dffb700 1 -- 172.21.15.42:0/1298874202 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f3e94067400 con 0x7f3e9c125500 2019-12-05T05:54:43.688 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.682+0000 7f3ea2a6c700 1 -- 172.21.15.42:0/1298874202 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3e9c05b8b0 con 0x7f3e9c125500 2019-12-05T05:54:43.689 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.682+0000 7f3ea180a700 1 --2- 172.21.15.42:0/1298874202 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e9c125500 0x7f3e9c125930 secure :-1 s=THROTTLE_DONE pgs=1271 cs=0 l=1 rx=0x7f3e9c08f990 tx=0x7f3e94010830).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:54:43.689 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.682+0000 7f3e8dffb700 1 -- 172.21.15.42:0/1298874202 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 203 v203) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f3e94031e60 con 0x7f3e9c125500 2019-12-05T05:54:43.700 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 203 2019-12-05T05:54:43.700 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:54:43.700 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":203,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":8614,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6835/2511123581","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2511123581},{"type":"v1","addr":"172.21.15.42:6835","nonce":2511123581}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":203}],"filesystems":[{"mdsmap":{"epoch":202,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:53:04.600937+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6356},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6356":{"gid":6356,"name":"b","rank":0,"incarnation":199,"state":"up:active","state_seq":26,"addr":"172.21.15.40:6833/1523046776","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1523046776},{"type":"v1","addr":"172.21.15.40:6833","nonce":1523046776}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:54:43.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.686+0000 7f3ea2a6c700 1 -- 172.21.15.42:0/1298874202 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3e7c03ae20 msgr2=0x7f3e7c03d330 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:54:43.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.686+0000 7f3ea2a6c700 1 --2- 172.21.15.42:0/1298874202 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3e7c03ae20 0x7f3e7c03d330 crc :-1 s=READY pgs=707 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:54:43.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.686+0000 7f3ea2a6c700 1 -- 172.21.15.42:0/1298874202 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e9c125500 msgr2=0x7f3e9c125930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:54:43.715 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.686+0000 7f3ea2a6c700 1 --2- 172.21.15.42:0/1298874202 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e9c125500 0x7f3e9c125930 secure :-1 s=READY pgs=1271 cs=0 l=1 rx=0x7f3e9c08f990 tx=0x7f3e94010830).stop 2019-12-05T05:54:43.715 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.686+0000 7f3ea2a6c700 1 -- 172.21.15.42:0/1298874202 shutdown_connections 2019-12-05T05:54:43.715 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.686+0000 7f3ea2a6c700 1 --2- 172.21.15.42:0/1298874202 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3e7c03ae20 0x7f3e7c03d330 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:54:43.715 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.686+0000 7f3ea2a6c700 1 --2- 172.21.15.42:0/1298874202 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e9c125500 0x7f3e9c125930 unknown :-1 s=CLOSED pgs=1271 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:54:43.716 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.686+0000 7f3ea2a6c700 1 --2- 172.21.15.42:0/1298874202 >> v2:172.21.15.42:3300/0 conn(0x7f3e9c063d90 0x7f3e9c08eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:54:43.716 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.686+0000 7f3ea2a6c700 1 -- 172.21.15.42:0/1298874202 shutdown_connections 2019-12-05T05:54:43.730 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.686+0000 7f3ea2a6c700 1 -- 172.21.15.42:0/1298874202 wait complete. 2019-12-05T05:54:43.730 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.686+0000 7f3ea2a6c700 1 -- 172.21.15.42:0/1298874202 >> 172.21.15.42:0/1298874202 conn(0x7f3e9c0a0040 msgr2=0x7f3e9c0a4830 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:54:43.731 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.b (rank=0) 2019-12-05T05:54:43.731 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2019-12-05T05:54:43.731 INFO:teuthology.orchestra.run:waiting for 300 2019-12-05T05:54:43.748 INFO:tasks.ceph.mds.b:Stopped 2019-12-05T05:54:43.748 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:54:43.872 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.838+0000 7fe3f21de700 1 Processor -- start 2019-12-05T05:54:43.872 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.838+0000 7fe3f21de700 1 -- start start 2019-12-05T05:54:43.873 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.842+0000 7fe3f21de700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe3ec0a7740 0x7fe3ec08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:54:43.873 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.842+0000 7fe3f21de700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe3ec09fae0 0x7fe3ec07eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:54:43.873 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.842+0000 7fe3f21de700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3ec0a0710 con 0x7fe3ec07e720 2019-12-05T05:54:43.874 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.842+0000 7fe3f21de700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3ec0a0b00 con 0x7fe3ec09fae0 2019-12-05T05:54:43.874 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.842+0000 7fe3f21de700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3ec08e5b0 con 0x7fe3ec0a7740 2019-12-05T05:54:43.874 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.842+0000 7fe3ebfff700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe3ec0a7740 0x7fe3ec08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:54:43.874 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.842+0000 7fe3ebfff700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe3ec0a7740 0x7fe3ec08df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:44206/0 (socket says 172.21.15.42:44206) 2019-12-05T05:54:43.874 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.842+0000 7fe3ebfff700 1 -- 172.21.15.42:0/3207724501 learned_addr learned my addr 172.21.15.42:0/3207724501 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:54:43.875 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.842+0000 7fe3f0f7c700 1 --2- 172.21.15.42:0/3207724501 >> v2:172.21.15.40:3300/0 conn(0x7fe3ec09fae0 0x7fe3ec07eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:54:43.875 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.842+0000 7fe3ebfff700 1 -- 172.21.15.42:0/3207724501 >> v2:172.21.15.40:3300/0 conn(0x7fe3ec09fae0 msgr2=0x7fe3ec07eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:54:43.875 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.842+0000 7fe3ebfff700 1 --2- 172.21.15.42:0/3207724501 >> v2:172.21.15.40:3300/0 conn(0x7fe3ec09fae0 0x7fe3ec07eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:54:43.875 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.842+0000 7fe3ebfff700 1 -- 172.21.15.42:0/3207724501 >> v1:172.21.15.40:6789/0 conn(0x7fe3ec07e720 legacy=0x7fe3ec09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:54:43.876 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.842+0000 7fe3ebfff700 1 -- 172.21.15.42:0/3207724501 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3ec07f0a0 con 0x7fe3ec0a7740 2019-12-05T05:54:43.876 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.842+0000 7fe3ebfff700 1 --2- 172.21.15.42:0/3207724501 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe3ec0a7740 0x7fe3ec08df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fe3dc008b10 tx=0x7fe3dc0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:54:43.876 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.842+0000 7fe3ebfff700 1 --2- 172.21.15.42:0/3207724501 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe3ec0a7740 0x7fe3ec08df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fe3dc008b10 tx=0x7fe3dc0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:54:43.876 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.842+0000 7fe3ebfff700 1 --2- 172.21.15.42:0/3207724501 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe3ec0a7740 0x7fe3ec08df80 secure :-1 s=READY pgs=371 cs=0 l=1 rx=0x7fe3dc008b10 tx=0x7fe3dc0100b0).ready entity=mon.2 client_cookie=64f2623eb658115b server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:54:43.877 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.842+0000 7fe3ebfff700 1 --2- 172.21.15.42:0/3207724501 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe3ec0a7740 0x7fe3ec08df80 secure :-1 s=THROTTLE_DONE pgs=371 cs=0 l=1 rx=0x7fe3dc008b10 tx=0x7fe3dc0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:54:43.877 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.842+0000 7fe3eaffd700 1 -- 172.21.15.42:0/3207724501 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe3dc010da0 con 0x7fe3ec0a7740 2019-12-05T05:54:43.877 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.842+0000 7fe3ebfff700 1 --2- 172.21.15.42:0/3207724501 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe3ec0a7740 0x7fe3ec08df80 secure :-1 s=THROTTLE_DONE pgs=371 cs=0 l=1 rx=0x7fe3dc008b10 tx=0x7fe3dc0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:54:43.877 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.842+0000 7fe3ebfff700 1 --2- 172.21.15.42:0/3207724501 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe3ec0a7740 0x7fe3ec08df80 secure :-1 s=THROTTLE_DONE pgs=371 cs=0 l=1 rx=0x7fe3dc008b10 tx=0x7fe3dc0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:54:43.878 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.842+0000 7fe3eaffd700 1 -- 172.21.15.42:0/3207724501 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3dc0109c0 con 0x7fe3ec0a7740 2019-12-05T05:54:43.878 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.842+0000 7fe3eaffd700 1 -- 172.21.15.42:0/3207724501 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe3dc0114a0 con 0x7fe3ec0a7740 2019-12-05T05:54:43.878 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.842+0000 7fe3f21de700 1 -- 172.21.15.42:0/3207724501 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe3ec0a7740 msgr2=0x7fe3ec08df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:54:43.878 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.842+0000 7fe3f21de700 1 --2- 172.21.15.42:0/3207724501 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe3ec0a7740 0x7fe3ec08df80 secure :-1 s=READY pgs=371 cs=0 l=1 rx=0x7fe3dc008b10 tx=0x7fe3dc0100b0).stop 2019-12-05T05:54:43.879 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.846+0000 7fe3f21de700 1 -- 172.21.15.42:0/3207724501 shutdown_connections 2019-12-05T05:54:43.879 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.846+0000 7fe3f21de700 1 --2- 172.21.15.42:0/3207724501 >> v2:172.21.15.40:3300/0 conn(0x7fe3ec09fae0 0x7fe3ec07eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:54:43.879 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.846+0000 7fe3f21de700 1 --2- 172.21.15.42:0/3207724501 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe3ec0a7740 0x7fe3ec08df80 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:54:43.879 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.846+0000 7fe3f21de700 1 -- 172.21.15.42:0/3207724501 shutdown_connections 2019-12-05T05:54:43.880 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.846+0000 7fe3f21de700 1 -- 172.21.15.42:0/3207724501 wait complete. 2019-12-05T05:54:43.880 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.846+0000 7fe3f21de700 1 -- 172.21.15.42:0/3207724501 >> 172.21.15.42:0/3207724501 conn(0x7fe3ec0aabe0 msgr2=0x7fe3ec0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:54:43.881 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.846+0000 7fe3f21de700 1 Processor -- start 2019-12-05T05:54:43.881 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.846+0000 7fe3f21de700 1 -- start start 2019-12-05T05:54:43.881 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.846+0000 7fe3f21de700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe3ec0a4b20 0x7fe3ec066fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:54:43.881 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.846+0000 7fe3f21de700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe3ec0a68f0 0x7fe3ec130c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:54:43.882 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.846+0000 7fe3f21de700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3ec067530 con 0x7fe3ec0a64c0 2019-12-05T05:54:43.882 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.846+0000 7fe3f21de700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3ec1452d0 con 0x7fe3ec0a68f0 2019-12-05T05:54:43.882 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.846+0000 7fe3f21de700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3ec0a51f0 con 0x7fe3ec0a4b20 2019-12-05T05:54:43.882 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.850+0000 7fe3ebfff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe3ec0a4b20 0x7fe3ec066fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:54:43.883 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.850+0000 7fe3ebfff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe3ec0a4b20 0x7fe3ec066fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:43750/0 (socket says 172.21.15.42:43750) 2019-12-05T05:54:43.883 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.850+0000 7fe3f0f7c700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe3ec0a68f0 0x7fe3ec130c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:54:43.883 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.850+0000 7fe3ebfff700 1 -- 172.21.15.42:0/3917420823 learned_addr learned my addr 172.21.15.42:0/3917420823 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:54:43.883 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.850+0000 7fe3ebfff700 1 -- 172.21.15.42:0/3917420823 >> v2:172.21.15.40:3300/0 conn(0x7fe3ec0a68f0 msgr2=0x7fe3ec130c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:54:43.884 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.850+0000 7fe3ebfff700 1 --2- 172.21.15.42:0/3917420823 >> v2:172.21.15.40:3300/0 conn(0x7fe3ec0a68f0 0x7fe3ec130c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:54:43.884 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.850+0000 7fe3e8ff9700 1 -- 172.21.15.42:0/3917420823 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1893422626 0 0) 0x7fe3ec067530 con 0x7fe3ec0a64c0 2019-12-05T05:54:43.884 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.850+0000 7fe3ebfff700 1 -- 172.21.15.42:0/3917420823 >> v1:172.21.15.42:6789/0 conn(0x7fe3ec0a64c0 legacy=0x7fe3ec130490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:54:43.885 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.850+0000 7fe3ebfff700 1 -- 172.21.15.42:0/3917420823 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3ec0a5490 con 0x7fe3ec0a4b20 2019-12-05T05:54:43.885 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.850+0000 7fe3ebfff700 1 --2- 172.21.15.42:0/3917420823 >> v2:172.21.15.42:3300/0 conn(0x7fe3ec0a4b20 0x7fe3ec066fe0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fe3ec1303b0 tx=0x7fe3dc010d30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:54:43.885 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.850+0000 7fe3ebfff700 1 --2- 172.21.15.42:0/3917420823 >> v2:172.21.15.42:3300/0 conn(0x7fe3ec0a4b20 0x7fe3ec066fe0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fe3ec1303b0 tx=0x7fe3dc010d30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:54:43.885 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.850+0000 7fe3ebfff700 1 --2- 172.21.15.42:0/3917420823 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe3ec0a4b20 0x7fe3ec066fe0 secure :-1 s=READY pgs=1663 cs=0 l=1 rx=0x7fe3ec1303b0 tx=0x7fe3dc010d30).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:54:43.886 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.850+0000 7fe3ebfff700 1 --2- 172.21.15.42:0/3917420823 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe3ec0a4b20 0x7fe3ec066fe0 secure :-1 s=THROTTLE_DONE pgs=1663 cs=0 l=1 rx=0x7fe3ec1303b0 tx=0x7fe3dc010d30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:54:43.886 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.850+0000 7fe3ebfff700 1 --2- 172.21.15.42:0/3917420823 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe3ec0a4b20 0x7fe3ec066fe0 secure :-1 s=THROTTLE_DONE pgs=1663 cs=0 l=1 rx=0x7fe3ec1303b0 tx=0x7fe3dc010d30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:54:43.886 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.850+0000 7fe3ebfff700 1 --2- 172.21.15.42:0/3917420823 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe3ec0a4b20 0x7fe3ec066fe0 secure :-1 s=THROTTLE_DONE pgs=1663 cs=0 l=1 rx=0x7fe3ec1303b0 tx=0x7fe3dc010d30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:54:43.886 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.850+0000 7fe3e8ff9700 1 -- 172.21.15.42:0/3917420823 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe3dc011bb0 con 0x7fe3ec0a4b20 2019-12-05T05:54:43.887 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.850+0000 7fe3e8ff9700 1 -- 172.21.15.42:0/3917420823 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3dc0110c0 con 0x7fe3ec0a4b20 2019-12-05T05:54:43.887 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.850+0000 7fe3f21de700 1 -- 172.21.15.42:0/3917420823 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3ec0742d0 con 0x7fe3ec0a4b20 2019-12-05T05:54:43.887 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.850+0000 7fe3e8ff9700 1 -- 172.21.15.42:0/3917420823 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe3dc005ab0 con 0x7fe3ec0a4b20 2019-12-05T05:54:43.887 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.850+0000 7fe3f21de700 1 -- 172.21.15.42:0/3917420823 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3ec074850 con 0x7fe3ec0a4b20 2019-12-05T05:54:43.888 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.850+0000 7fe3ebfff700 1 --2- 172.21.15.42:0/3917420823 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe3ec0a4b20 0x7fe3ec066fe0 secure :-1 s=THROTTLE_DONE pgs=1663 cs=0 l=1 rx=0x7fe3ec1303b0 tx=0x7fe3dc010d30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:54:43.888 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.850+0000 7fe3e8ff9700 1 -- 172.21.15.42:0/3917420823 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fe3dc005cb0 con 0x7fe3ec0a4b20 2019-12-05T05:54:43.888 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.850+0000 7fe3ebfff700 1 --2- 172.21.15.42:0/3917420823 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe3ec0a4b20 0x7fe3ec066fe0 secure :-1 s=THROTTLE_DONE pgs=1663 cs=0 l=1 rx=0x7fe3ec1303b0 tx=0x7fe3dc010d30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:54:43.888 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.850+0000 7fe3e8ff9700 1 --2- 172.21.15.42:0/3917420823 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe3d003aca0 0x7fe3d003d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:54:43.889 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.850+0000 7fe3e8ff9700 1 -- 172.21.15.42:0/3917420823 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8955+0+0 (secure 0 0 0) 0x7fe3dc04b5e0 con 0x7fe3ec0a4b20 2019-12-05T05:54:43.889 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.854+0000 7fe3eb7fe700 1 --2- 172.21.15.42:0/3917420823 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe3d003aca0 0x7fe3d003d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:54:43.889 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.854+0000 7fe3eb7fe700 1 --2- 172.21.15.42:0/3917420823 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe3d003aca0 0x7fe3d003d1b0 crc :-1 s=READY pgs=708 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:54:43.889 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.854+0000 7fe3f21de700 1 -- 172.21.15.42:0/3917420823 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3ec05d110 con 0x7fe3ec0a4b20 2019-12-05T05:54:43.890 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.862+0000 7fe3ebfff700 1 --2- 172.21.15.42:0/3917420823 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe3ec0a4b20 0x7fe3ec066fe0 secure :-1 s=THROTTLE_DONE pgs=1663 cs=0 l=1 rx=0x7fe3ec1303b0 tx=0x7fe3dc010d30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:54:43.890 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:43.862+0000 7fe3e8ff9700 1 -- 172.21.15.42:0/3917420823 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fe3dc032660 con 0x7fe3ec0a4b20 2019-12-05T05:54:44.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:44.022+0000 7fe3f21de700 1 -- 172.21.15.42:0/3917420823 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe3ec1454c0 con 0x7fe3ec0a4b20 2019-12-05T05:54:44.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:44.026+0000 7fe3ebfff700 1 --2- 172.21.15.42:0/3917420823 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe3ec0a4b20 0x7fe3ec066fe0 secure :-1 s=THROTTLE_DONE pgs=1663 cs=0 l=1 rx=0x7fe3ec1303b0 tx=0x7fe3dc010d30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:54:44.032 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:44.026+0000 7fe3e8ff9700 1 -- 172.21.15.42:0/3917420823 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 203 v203) v1 ==== 95+0+2112 (secure 0 0 0) 0x7fe3dc00f280 con 0x7fe3ec0a4b20 2019-12-05T05:54:44.034 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 203 2019-12-05T05:54:44.034 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:54:44.035 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":203,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":8614,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6835/2511123581","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2511123581},{"type":"v1","addr":"172.21.15.42:6835","nonce":2511123581}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":203}],"filesystems":[{"mdsmap":{"epoch":202,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:53:04.600937+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6356},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6356":{"gid":6356,"name":"b","rank":0,"incarnation":199,"state":"up:active","state_seq":26,"addr":"172.21.15.40:6833/1523046776","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1523046776},{"type":"v1","addr":"172.21.15.40:6833","nonce":1523046776}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:54:44.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:44.030+0000 7fe3f21de700 1 -- 172.21.15.42:0/3917420823 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe3d003aca0 msgr2=0x7fe3d003d1b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:54:44.039 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:44.030+0000 7fe3f21de700 1 --2- 172.21.15.42:0/3917420823 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe3d003aca0 0x7fe3d003d1b0 crc :-1 s=READY pgs=708 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:54:44.039 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:44.030+0000 7fe3f21de700 1 -- 172.21.15.42:0/3917420823 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe3ec0a4b20 msgr2=0x7fe3ec066fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:54:44.039 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:44.030+0000 7fe3f21de700 1 --2- 172.21.15.42:0/3917420823 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe3ec0a4b20 0x7fe3ec066fe0 secure :-1 s=READY pgs=1663 cs=0 l=1 rx=0x7fe3ec1303b0 tx=0x7fe3dc010d30).stop 2019-12-05T05:54:44.039 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:44.030+0000 7fe3f21de700 1 -- 172.21.15.42:0/3917420823 shutdown_connections 2019-12-05T05:54:44.039 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:44.030+0000 7fe3f21de700 1 --2- 172.21.15.42:0/3917420823 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe3d003aca0 0x7fe3d003d1b0 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:54:44.039 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:44.030+0000 7fe3f21de700 1 --2- 172.21.15.42:0/3917420823 >> v2:172.21.15.40:3300/0 conn(0x7fe3ec0a68f0 0x7fe3ec130c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:54:44.039 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:44.030+0000 7fe3f21de700 1 --2- 172.21.15.42:0/3917420823 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe3ec0a4b20 0x7fe3ec066fe0 unknown :-1 s=CLOSED pgs=1663 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:54:44.040 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:44.030+0000 7fe3f21de700 1 -- 172.21.15.42:0/3917420823 shutdown_connections 2019-12-05T05:54:44.040 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:44.030+0000 7fe3f21de700 1 -- 172.21.15.42:0/3917420823 wait complete. 2019-12-05T05:54:44.040 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:44.030+0000 7fe3f21de700 1 -- 172.21.15.42:0/3917420823 >> 172.21.15.42:0/3917420823 conn(0x7fe3ec0aabe0 msgr2=0x7fe3ec059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:54:44.050 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T05:54:46.051 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:54:46.158 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:46.146+0000 7f86a1acb700 1 Processor -- start 2019-12-05T05:54:46.159 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:46.146+0000 7f86a1acb700 1 -- start start 2019-12-05T05:54:46.159 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:46.146+0000 7f86a1acb700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f869c0a7740 0x7f869c08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:54:46.159 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:46.146+0000 7f86a1acb700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f869c0a0710 con 0x7f869c09fae0 2019-12-05T05:54:46.159 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:46.146+0000 7f86a1acb700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f869c0a9ba0 con 0x7f869c07e720 2019-12-05T05:54:46.159 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:46.146+0000 7f86a1acb700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f869c0a0b00 con 0x7f869c0a7740 2019-12-05T05:54:46.159 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:46.146+0000 7f869bfff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f869c0a7740 0x7f869c08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:54:46.159 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:46.150+0000 7f869b7fe700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f869c07e720 0x7f869c09f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:43800/0 (socket says 172.21.15.42:43800) 2019-12-05T05:54:46.160 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:46.150+0000 7f869bfff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f869c0a7740 0x7f869c08df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:43760/0 (socket says 172.21.15.42:43760) 2019-12-05T05:54:46.160 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:46.150+0000 7f869b7fe700 1 -- 172.21.15.42:0/2772205213 learned_addr learned my addr 172.21.15.42:0/2772205213 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:54:46.160 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:46.150+0000 7f869bfff700 1 -- 172.21.15.42:0/2772205213 >> v1:172.21.15.42:6789/0 conn(0x7f869c07e720 legacy=0x7f869c09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:54:46.160 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:46.150+0000 7f869bfff700 1 -- 172.21.15.42:0/2772205213 >> v1:172.21.15.40:6789/0 conn(0x7f869c09fae0 legacy=0x7f869c14efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:54:46.160 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:46.150+0000 7f869bfff700 1 -- 172.21.15.42:0/2772205213 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f869c07eb50 con 0x7f869c0a7740 2019-12-05T05:54:46.160 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:46.150+0000 7f869bfff700 1 --2- 172.21.15.42:0/2772205213 >> v2:172.21.15.42:3300/0 conn(0x7f869c0a7740 0x7f869c08df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f868c008790 tx=0x7f868c00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:54:46.160 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:46.150+0000 7f869bfff700 1 --2- 172.21.15.42:0/2772205213 >> v2:172.21.15.42:3300/0 conn(0x7f869c0a7740 0x7f869c08df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f868c008790 tx=0x7f868c00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:54:46.161 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:46.150+0000 7f869bfff700 1 --2- 172.21.15.42:0/2772205213 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f869c0a7740 0x7f869c08df80 secure :-1 s=READY pgs=1665 cs=0 l=1 rx=0x7f868c008790 tx=0x7f868c00fbf0).ready entity=mon.1 client_cookie=a4b9677dfd901a94 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:54:46.161 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:46.150+0000 7f869bfff700 1 --2- 172.21.15.42:0/2772205213 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f869c0a7740 0x7f869c08df80 secure :-1 s=THROTTLE_DONE pgs=1665 cs=0 l=1 rx=0x7f868c008790 tx=0x7f868c00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:54:46.161 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:46.150+0000 7f869bfff700 1 --2- 172.21.15.42:0/2772205213 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f869c0a7740 0x7f869c08df80 secure :-1 s=THROTTLE_DONE pgs=1665 cs=0 l=1 rx=0x7f868c008790 tx=0x7f868c00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:54:46.161 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:46.150+0000 7f869affd700 1 -- 172.21.15.42:0/2772205213 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f868c011270 con 0x7f869c0a7740 2019-12-05T05:54:46.161 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:46.150+0000 7f869bfff700 1 --2- 172.21.15.42:0/2772205213 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f869c0a7740 0x7f869c08df80 secure :-1 s=THROTTLE_DONE pgs=1665 cs=0 l=1 rx=0x7f868c008790 tx=0x7f868c00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:54:46.161 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:46.150+0000 7f869affd700 1 -- 172.21.15.42:0/2772205213 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f868c008bc0 con 0x7f869c0a7740 2019-12-05T05:54:46.161 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:46.150+0000 7f869affd700 1 -- 172.21.15.42:0/2772205213 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f868c011850 con 0x7f869c0a7740 2019-12-05T05:54:46.162 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:46.150+0000 7f86a1acb700 1 -- 172.21.15.42:0/2772205213 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f869c0a7740 msgr2=0x7f869c08df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:54:46.162 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:46.150+0000 7f86a1acb700 1 --2- 172.21.15.42:0/2772205213 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f869c0a7740 0x7f869c08df80 secure :-1 s=READY pgs=1665 cs=0 l=1 rx=0x7f868c008790 tx=0x7f868c00fbf0).stop 2019-12-05T05:54:46.162 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:46.150+0000 7f86a1acb700 1 -- 172.21.15.42:0/2772205213 shutdown_connections 2019-12-05T05:54:46.162 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:46.150+0000 7f86a1acb700 1 --2- 172.21.15.42:0/2772205213 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f869c0a7740 0x7f869c08df80 unknown :-1 s=CLOSED pgs=1665 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:54:46.162 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:46.150+0000 7f86a1acb700 1 -- 172.21.15.42:0/2772205213 shutdown_connections 2019-12-05T05:54:46.162 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:46.150+0000 7f86a1acb700 1 -- 172.21.15.42:0/2772205213 wait complete. 2019-12-05T05:54:46.162 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:46.150+0000 7f86a1acb700 1 -- 172.21.15.42:0/2772205213 >> 172.21.15.42:0/2772205213 conn(0x7f869c0aabe0 msgr2=0x7f869c0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:54:46.165 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:46.154+0000 7f86a1acb700 1 Processor -- start 2019-12-05T05:54:46.166 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:46.154+0000 7f86a1acb700 1 -- start start 2019-12-05T05:54:46.166 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:46.154+0000 7f86a1acb700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f869c05a210 0x7f869c1154d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:54:46.166 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:46.154+0000 7f86a1acb700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f869c0a2c00 0x7f869c12ade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:54:46.166 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:46.154+0000 7f86a1acb700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f869c0a43e0 con 0x7f869c0a27d0 2019-12-05T05:54:46.166 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:46.154+0000 7f86a1acb700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f869c0a3c00 con 0x7f869c0a2c00 2019-12-05T05:54:46.166 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:46.154+0000 7f86a1acb700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f869c0a3ea0 con 0x7f869c05a210 2019-12-05T05:54:46.167 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:46.154+0000 7f869bfff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f869c05a210 0x7f869c1154d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:54:46.167 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:46.154+0000 7f869bfff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f869c05a210 0x7f869c1154d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:43766/0 (socket says 172.21.15.42:43766) 2019-12-05T05:54:46.167 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:46.154+0000 7f869bfff700 1 -- 172.21.15.42:0/3968457816 learned_addr learned my addr 172.21.15.42:0/3968457816 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:54:46.167 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:46.154+0000 7f86a0869700 1 --2- 172.21.15.42:0/3968457816 >> v2:172.21.15.40:3300/0 conn(0x7f869c0a2c00 0x7f869c12ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:54:46.167 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:46.154+0000 7f869bfff700 1 -- 172.21.15.42:0/3968457816 >> v2:172.21.15.40:3300/0 conn(0x7f869c0a2c00 msgr2=0x7f869c12ade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:54:46.167 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:46.154+0000 7f869bfff700 1 --2- 172.21.15.42:0/3968457816 >> v2:172.21.15.40:3300/0 conn(0x7f869c0a2c00 0x7f869c12ade0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:54:46.168 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:46.154+0000 7f869bfff700 1 -- 172.21.15.42:0/3968457816 >> v1:172.21.15.40:6789/0 conn(0x7f869c0a27d0 legacy=0x7f869c07b900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:54:46.168 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:46.154+0000 7f869bfff700 1 -- 172.21.15.42:0/3968457816 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f869c0a38f0 con 0x7f869c05a210 2019-12-05T05:54:46.168 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:46.154+0000 7f869bfff700 1 --2- 172.21.15.42:0/3968457816 >> v2:172.21.15.42:3300/0 conn(0x7f869c05a210 0x7f869c1154d0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f868c008790 tx=0x7f868c008ef0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:54:46.168 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:46.154+0000 7f869bfff700 1 --2- 172.21.15.42:0/3968457816 >> v2:172.21.15.42:3300/0 conn(0x7f869c05a210 0x7f869c1154d0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f868c008790 tx=0x7f868c008ef0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:54:46.168 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:46.154+0000 7f869bfff700 1 --2- 172.21.15.42:0/3968457816 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f869c05a210 0x7f869c1154d0 secure :-1 s=READY pgs=1666 cs=0 l=1 rx=0x7f868c008790 tx=0x7f868c008ef0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:54:46.168 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:46.158+0000 7f869bfff700 1 --2- 172.21.15.42:0/3968457816 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f869c05a210 0x7f869c1154d0 secure :-1 s=THROTTLE_DONE pgs=1666 cs=0 l=1 rx=0x7f868c008790 tx=0x7f868c008ef0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:54:46.169 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:46.158+0000 7f869bfff700 1 --2- 172.21.15.42:0/3968457816 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f869c05a210 0x7f869c1154d0 secure :-1 s=THROTTLE_DONE pgs=1666 cs=0 l=1 rx=0x7f868c008790 tx=0x7f868c008ef0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:54:46.169 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:46.158+0000 7f8698ff9700 1 -- 172.21.15.42:0/3968457816 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f868c007340 con 0x7f869c05a210 2019-12-05T05:54:46.169 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:46.158+0000 7f869bfff700 1 --2- 172.21.15.42:0/3968457816 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f869c05a210 0x7f869c1154d0 secure :-1 s=THROTTLE_DONE pgs=1666 cs=0 l=1 rx=0x7f868c008790 tx=0x7f868c008ef0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:54:46.169 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:46.158+0000 7f8698ff9700 1 -- 172.21.15.42:0/3968457816 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f868c011470 con 0x7f869c05a210 2019-12-05T05:54:46.169 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:46.158+0000 7f86a1acb700 1 -- 172.21.15.42:0/3968457816 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f869c114180 con 0x7f869c05a210 2019-12-05T05:54:46.169 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:46.158+0000 7f8698ff9700 1 -- 172.21.15.42:0/3968457816 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f868c005540 con 0x7f869c05a210 2019-12-05T05:54:46.169 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:46.158+0000 7f86a1acb700 1 -- 172.21.15.42:0/3968457816 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f869c0a3030 con 0x7f869c05a210 2019-12-05T05:54:46.170 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:46.158+0000 7f869bfff700 1 --2- 172.21.15.42:0/3968457816 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f869c05a210 0x7f869c1154d0 secure :-1 s=THROTTLE_DONE pgs=1666 cs=0 l=1 rx=0x7f868c008790 tx=0x7f868c008ef0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:54:46.170 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:46.158+0000 7f8698ff9700 1 -- 172.21.15.42:0/3968457816 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f868c005850 con 0x7f869c05a210 2019-12-05T05:54:46.170 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:46.158+0000 7f869bfff700 1 --2- 172.21.15.42:0/3968457816 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f869c05a210 0x7f869c1154d0 secure :-1 s=THROTTLE_DONE pgs=1666 cs=0 l=1 rx=0x7f868c008790 tx=0x7f868c008ef0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:54:46.170 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:46.158+0000 7f8698ff9700 1 --2- 172.21.15.42:0/3968457816 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f867c03ada0 0x7f867c03d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:54:46.170 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:46.158+0000 7f8698ff9700 1 -- 172.21.15.42:0/3968457816 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8955+0+0 (secure 0 0 0) 0x7f868c04bd60 con 0x7f869c05a210 2019-12-05T05:54:46.170 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:46.158+0000 7f869b7fe700 1 --2- 172.21.15.42:0/3968457816 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f867c03ada0 0x7f867c03d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:54:46.171 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:46.158+0000 7f869b7fe700 1 --2- 172.21.15.42:0/3968457816 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f867c03ada0 0x7f867c03d2b0 crc :-1 s=READY pgs=709 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:54:46.171 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:46.162+0000 7f86a1acb700 1 -- 172.21.15.42:0/3968457816 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f869c05d110 con 0x7f869c05a210 2019-12-05T05:54:46.175 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:46.166+0000 7f869bfff700 1 --2- 172.21.15.42:0/3968457816 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f869c05a210 0x7f869c1154d0 secure :-1 s=THROTTLE_DONE pgs=1666 cs=0 l=1 rx=0x7f868c008790 tx=0x7f868c008ef0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:54:46.175 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:46.166+0000 7f8698ff9700 1 -- 172.21.15.42:0/3968457816 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f868c032de0 con 0x7f869c05a210 2019-12-05T05:54:46.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:46.334+0000 7f86a1acb700 1 -- 172.21.15.42:0/3968457816 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f869c05b8c0 con 0x7f869c05a210 2019-12-05T05:54:46.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:46.334+0000 7f869bfff700 1 --2- 172.21.15.42:0/3968457816 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f869c05a210 0x7f869c1154d0 secure :-1 s=THROTTLE_DONE pgs=1666 cs=0 l=1 rx=0x7f868c008790 tx=0x7f868c008ef0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:54:46.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:46.334+0000 7f8698ff9700 1 -- 172.21.15.42:0/3968457816 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 203 v203) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f868c033040 con 0x7f869c05a210 2019-12-05T05:54:46.356 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 203 2019-12-05T05:54:46.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:46.338+0000 7f86a1acb700 1 -- 172.21.15.42:0/3968457816 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f867c03ada0 msgr2=0x7f867c03d2b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:54:46.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:46.338+0000 7f86a1acb700 1 --2- 172.21.15.42:0/3968457816 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f867c03ada0 0x7f867c03d2b0 crc :-1 s=READY pgs=709 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:54:46.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:46.338+0000 7f86a1acb700 1 -- 172.21.15.42:0/3968457816 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f869c05a210 msgr2=0x7f869c1154d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:54:46.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:46.338+0000 7f86a1acb700 1 --2- 172.21.15.42:0/3968457816 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f869c05a210 0x7f869c1154d0 secure :-1 s=READY pgs=1666 cs=0 l=1 rx=0x7f868c008790 tx=0x7f868c008ef0).stop 2019-12-05T05:54:46.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:46.338+0000 7f86a1acb700 1 -- 172.21.15.42:0/3968457816 shutdown_connections 2019-12-05T05:54:46.358 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:46.338+0000 7f86a1acb700 1 --2- 172.21.15.42:0/3968457816 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f867c03ada0 0x7f867c03d2b0 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:54:46.359 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:46.338+0000 7f86a1acb700 1 --2- 172.21.15.42:0/3968457816 >> v2:172.21.15.40:3300/0 conn(0x7f869c0a2c00 0x7f869c12ade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:54:46.359 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:46.338+0000 7f86a1acb700 1 --2- 172.21.15.42:0/3968457816 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f869c05a210 0x7f869c1154d0 unknown :-1 s=CLOSED pgs=1666 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:54:46.360 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:46.338+0000 7f86a1acb700 1 -- 172.21.15.42:0/3968457816 shutdown_connections 2019-12-05T05:54:46.360 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:46.338+0000 7f86a1acb700 1 -- 172.21.15.42:0/3968457816 wait complete. 2019-12-05T05:54:46.360 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:46.338+0000 7f86a1acb700 1 -- 172.21.15.42:0/3968457816 >> 172.21.15.42:0/3968457816 conn(0x7f869c0a8400 msgr2=0x7f869c08eb70 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:54:46.361 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:54:46.361 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":203,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":8614,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6835/2511123581","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2511123581},{"type":"v1","addr":"172.21.15.42:6835","nonce":2511123581}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":203}],"filesystems":[{"mdsmap":{"epoch":202,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:53:04.600937+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6356},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6356":{"gid":6356,"name":"b","rank":0,"incarnation":199,"state":"up:active","state_seq":26,"addr":"172.21.15.40:6833/1523046776","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1523046776},{"type":"v1","addr":"172.21.15.40:6833","nonce":1523046776}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:54:46.361 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T05:54:48.362 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:54:48.475 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:48.462+0000 7f330f5d9700 1 Processor -- start 2019-12-05T05:54:48.475 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:48.462+0000 7f330f5d9700 1 -- start start 2019-12-05T05:54:48.476 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:48.462+0000 7f330f5d9700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f33080a7740 0x7f330808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:54:48.476 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:48.462+0000 7f330f5d9700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f330809fae0 0x7f330807eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:54:48.476 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:48.462+0000 7f330f5d9700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33080a0710 con 0x7f330807e720 2019-12-05T05:54:48.476 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:48.462+0000 7f330f5d9700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33080a0b00 con 0x7f33080a7740 2019-12-05T05:54:48.476 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:48.462+0000 7f330f5d9700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f330808e5b0 con 0x7f330809fae0 2019-12-05T05:54:48.476 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:48.462+0000 7f330e377700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f330809fae0 0x7f330807eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:54:48.477 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:48.462+0000 7f330e377700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f330809fae0 0x7f330807eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:43780/0 (socket says 172.21.15.42:43780) 2019-12-05T05:54:48.477 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:48.462+0000 7f330e377700 1 -- 172.21.15.42:0/2841787236 learned_addr learned my addr 172.21.15.42:0/2841787236 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:54:48.477 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:48.462+0000 7f330db76700 1 --2- 172.21.15.42:0/2841787236 >> v2:172.21.15.40:3300/0 conn(0x7f33080a7740 0x7f330808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:54:48.477 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:48.466+0000 7f330e377700 1 -- 172.21.15.42:0/2841787236 >> v2:172.21.15.40:3300/0 conn(0x7f33080a7740 msgr2=0x7f330808df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:54:48.477 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:48.466+0000 7f330e377700 1 --2- 172.21.15.42:0/2841787236 >> v2:172.21.15.40:3300/0 conn(0x7f33080a7740 0x7f330808df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:54:48.477 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:48.466+0000 7f330e377700 1 -- 172.21.15.42:0/2841787236 >> v1:172.21.15.40:6789/0 conn(0x7f330807e720 legacy=0x7f330809f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:54:48.478 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:48.466+0000 7f330cb74700 1 -- 172.21.15.42:0/2841787236 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3491228046 0 0) 0x7f33080a0710 con 0x7f330807e720 2019-12-05T05:54:48.478 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:48.466+0000 7f330e377700 1 -- 172.21.15.42:0/2841787236 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f330807f0a0 con 0x7f330809fae0 2019-12-05T05:54:48.478 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:48.466+0000 7f330e377700 1 --2- 172.21.15.42:0/2841787236 >> v2:172.21.15.42:3300/0 conn(0x7f330809fae0 0x7f330807eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f33000032b0 tx=0x7f33000100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:54:48.478 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:48.466+0000 7f330e377700 1 --2- 172.21.15.42:0/2841787236 >> v2:172.21.15.42:3300/0 conn(0x7f330809fae0 0x7f330807eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f33000032b0 tx=0x7f33000100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:54:48.479 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:48.466+0000 7f330e377700 1 --2- 172.21.15.42:0/2841787236 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f330809fae0 0x7f330807eb50 secure :-1 s=READY pgs=1667 cs=0 l=1 rx=0x7f33000032b0 tx=0x7f33000100b0).ready entity=mon.1 client_cookie=9b6c03c333288bfb server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:54:48.479 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:48.466+0000 7f330e377700 1 --2- 172.21.15.42:0/2841787236 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f330809fae0 0x7f330807eb50 secure :-1 s=THROTTLE_DONE pgs=1667 cs=0 l=1 rx=0x7f33000032b0 tx=0x7f33000100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:54:48.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:48.466+0000 7f330cb74700 1 -- 172.21.15.42:0/2841787236 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3300010d60 con 0x7f330809fae0 2019-12-05T05:54:48.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:48.466+0000 7f330e377700 1 --2- 172.21.15.42:0/2841787236 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f330809fae0 0x7f330807eb50 secure :-1 s=THROTTLE_DONE pgs=1667 cs=0 l=1 rx=0x7f33000032b0 tx=0x7f33000100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:54:48.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:48.466+0000 7f330cb74700 1 -- 172.21.15.42:0/2841787236 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3300010980 con 0x7f330809fae0 2019-12-05T05:54:48.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:48.466+0000 7f330e377700 1 --2- 172.21.15.42:0/2841787236 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f330809fae0 0x7f330807eb50 secure :-1 s=THROTTLE_DONE pgs=1667 cs=0 l=1 rx=0x7f33000032b0 tx=0x7f33000100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:54:48.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:48.466+0000 7f330cb74700 1 -- 172.21.15.42:0/2841787236 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f33000114a0 con 0x7f330809fae0 2019-12-05T05:54:48.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:48.466+0000 7f330f5d9700 1 -- 172.21.15.42:0/2841787236 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f330809fae0 msgr2=0x7f330807eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:54:48.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:48.466+0000 7f330f5d9700 1 --2- 172.21.15.42:0/2841787236 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f330809fae0 0x7f330807eb50 secure :-1 s=READY pgs=1667 cs=0 l=1 rx=0x7f33000032b0 tx=0x7f33000100b0).stop 2019-12-05T05:54:48.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:48.466+0000 7f330f5d9700 1 -- 172.21.15.42:0/2841787236 shutdown_connections 2019-12-05T05:54:48.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:48.466+0000 7f330f5d9700 1 --2- 172.21.15.42:0/2841787236 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f330809fae0 0x7f330807eb50 unknown :-1 s=CLOSED pgs=1667 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:54:48.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:48.466+0000 7f330f5d9700 1 --2- 172.21.15.42:0/2841787236 >> v2:172.21.15.40:3300/0 conn(0x7f33080a7740 0x7f330808df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:54:48.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:48.466+0000 7f330f5d9700 1 -- 172.21.15.42:0/2841787236 shutdown_connections 2019-12-05T05:54:48.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:48.466+0000 7f330f5d9700 1 -- 172.21.15.42:0/2841787236 wait complete. 2019-12-05T05:54:48.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:48.466+0000 7f330f5d9700 1 -- 172.21.15.42:0/2841787236 >> 172.21.15.42:0/2841787236 conn(0x7f33080aabe0 msgr2=0x7f33080a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:54:48.485 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:48.470+0000 7f330f5d9700 1 Processor -- start 2019-12-05T05:54:48.485 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:48.470+0000 7f330f5d9700 1 -- start start 2019-12-05T05:54:48.485 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:48.470+0000 7f330f5d9700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f330805a210 0x7f33080a1d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:54:48.485 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:48.470+0000 7f330f5d9700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f33081138a0 0x7f33080784c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:54:48.485 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:48.470+0000 7f330f5d9700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3308078a10 0x7f3308112550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:54:48.485 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:48.470+0000 7f330f5d9700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3308113e10 con 0x7f330805a210 2019-12-05T05:54:48.486 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:48.470+0000 7f330f5d9700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33081140b0 con 0x7f3308078a10 2019-12-05T05:54:48.486 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:48.470+0000 7f330f5d9700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3308114350 con 0x7f33081138a0 2019-12-05T05:54:48.486 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:48.470+0000 7f330db76700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f330805a210 0x7f33080a1d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:54:48.486 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:48.470+0000 7f330e377700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3308078a10 0x7f3308112550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:54:48.486 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:48.470+0000 7f330db76700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f330805a210 0x7f33080a1d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.42:59830/0 (socket says 172.21.15.42:59830) 2019-12-05T05:54:48.486 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:48.470+0000 7f330e377700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3308078a10 0x7f3308112550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:43786/0 (socket says 172.21.15.42:43786) 2019-12-05T05:54:48.486 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:48.470+0000 7f330db76700 1 -- 172.21.15.42:0/3245835016 learned_addr learned my addr 172.21.15.42:0/3245835016 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:54:48.487 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:48.470+0000 7f330d375700 1 --2- 172.21.15.42:0/3245835016 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f33081138a0 0x7f33080784c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:54:48.487 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:48.470+0000 7f330e377700 1 -- 172.21.15.42:0/3245835016 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f33081138a0 msgr2=0x7f33080784c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:54:48.487 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:48.470+0000 7f330e377700 1 --2- 172.21.15.42:0/3245835016 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f33081138a0 0x7f33080784c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:54:48.487 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:48.470+0000 7f330e377700 1 -- 172.21.15.42:0/3245835016 >> v2:172.21.15.40:3300/0 conn(0x7f330805a210 msgr2=0x7f33080a1d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:54:48.487 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:48.470+0000 7f330e377700 1 --2- 172.21.15.42:0/3245835016 >> v2:172.21.15.40:3300/0 conn(0x7f330805a210 0x7f33080a1d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:54:48.488 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:48.470+0000 7f330e377700 1 -- 172.21.15.42:0/3245835016 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3308112aa0 con 0x7f3308078a10 2019-12-05T05:54:48.488 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:48.470+0000 7f330db76700 1 --2- 172.21.15.42:0/3245835016 >> v2:172.21.15.40:3300/0 conn(0x7f330805a210 0x7f33080a1d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:54:48.488 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:48.470+0000 7f330e377700 1 --2- 172.21.15.42:0/3245835016 >> v2:172.21.15.42:3300/0 conn(0x7f3308078a10 0x7f3308112550 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f3308113fd0 tx=0x7f33000115e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:54:48.488 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:48.470+0000 7f330e377700 1 --2- 172.21.15.42:0/3245835016 >> v2:172.21.15.42:3300/0 conn(0x7f3308078a10 0x7f3308112550 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f3308113fd0 tx=0x7f33000115e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:54:48.488 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:48.470+0000 7f330e377700 1 --2- 172.21.15.42:0/3245835016 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3308078a10 0x7f3308112550 secure :-1 s=READY pgs=1668 cs=0 l=1 rx=0x7f3308113fd0 tx=0x7f33000115e0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:54:48.488 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:48.470+0000 7f330e377700 1 --2- 172.21.15.42:0/3245835016 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3308078a10 0x7f3308112550 secure :-1 s=THROTTLE_DONE pgs=1668 cs=0 l=1 rx=0x7f3308113fd0 tx=0x7f33000115e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:54:48.488 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:48.474+0000 7f32fe7fc700 1 -- 172.21.15.42:0/3245835016 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3300011c40 con 0x7f3308078a10 2019-12-05T05:54:48.489 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:48.474+0000 7f330e377700 1 --2- 172.21.15.42:0/3245835016 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3308078a10 0x7f3308112550 secure :-1 s=THROTTLE_DONE pgs=1668 cs=0 l=1 rx=0x7f3308113fd0 tx=0x7f33000115e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:54:48.489 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:48.474+0000 7f330e377700 1 --2- 172.21.15.42:0/3245835016 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3308078a10 0x7f3308112550 secure :-1 s=THROTTLE_DONE pgs=1668 cs=0 l=1 rx=0x7f3308113fd0 tx=0x7f33000115e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:54:48.489 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:48.474+0000 7f32fe7fc700 1 -- 172.21.15.42:0/3245835016 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33000110c0 con 0x7f3308078a10 2019-12-05T05:54:48.489 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:48.474+0000 7f330f5d9700 1 -- 172.21.15.42:0/3245835016 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3308112d10 con 0x7f3308078a10 2019-12-05T05:54:48.489 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:48.474+0000 7f32fe7fc700 1 -- 172.21.15.42:0/3245835016 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3300005ca0 con 0x7f3308078a10 2019-12-05T05:54:48.489 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:48.474+0000 7f330f5d9700 1 -- 172.21.15.42:0/3245835016 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33080a59e0 con 0x7f3308078a10 2019-12-05T05:54:48.489 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:48.474+0000 7f330e377700 1 --2- 172.21.15.42:0/3245835016 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3308078a10 0x7f3308112550 secure :-1 s=THROTTLE_DONE pgs=1668 cs=0 l=1 rx=0x7f3308113fd0 tx=0x7f33000115e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:54:48.490 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:48.474+0000 7f32fe7fc700 1 -- 172.21.15.42:0/3245835016 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f330000f280 con 0x7f3308078a10 2019-12-05T05:54:48.490 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:48.474+0000 7f330e377700 1 --2- 172.21.15.42:0/3245835016 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3308078a10 0x7f3308112550 secure :-1 s=THROTTLE_DONE pgs=1668 cs=0 l=1 rx=0x7f3308113fd0 tx=0x7f33000115e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:54:48.490 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:48.474+0000 7f32fe7fc700 1 --2- 172.21.15.42:0/3245835016 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f32f003ae00 0x7f32f003d310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:54:48.490 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:48.474+0000 7f32fe7fc700 1 -- 172.21.15.42:0/3245835016 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8955+0+0 (secure 0 0 0) 0x7f330004b540 con 0x7f3308078a10 2019-12-05T05:54:48.490 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:48.474+0000 7f330db76700 1 --2- 172.21.15.42:0/3245835016 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f32f003ae00 0x7f32f003d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:54:48.490 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:48.474+0000 7f330db76700 1 --2- 172.21.15.42:0/3245835016 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f32f003ae00 0x7f32f003d310 crc :-1 s=READY pgs=710 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:54:48.491 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:48.478+0000 7f330f5d9700 1 -- 172.21.15.42:0/3245835016 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f330805d110 con 0x7f3308078a10 2019-12-05T05:54:48.491 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:48.482+0000 7f330e377700 1 --2- 172.21.15.42:0/3245835016 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3308078a10 0x7f3308112550 secure :-1 s=THROTTLE_DONE pgs=1668 cs=0 l=1 rx=0x7f3308113fd0 tx=0x7f33000115e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:54:48.491 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:48.482+0000 7f32fe7fc700 1 -- 172.21.15.42:0/3245835016 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f33000332b0 con 0x7f3308078a10 2019-12-05T05:54:48.657 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:48.650+0000 7f330f5d9700 1 -- 172.21.15.42:0/3245835016 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f330805b8c0 con 0x7f3308078a10 2019-12-05T05:54:48.658 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:48.650+0000 7f330e377700 1 --2- 172.21.15.42:0/3245835016 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3308078a10 0x7f3308112550 secure :-1 s=THROTTLE_DONE pgs=1668 cs=0 l=1 rx=0x7f3308113fd0 tx=0x7f33000115e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:54:48.658 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:48.650+0000 7f32fe7fc700 1 -- 172.21.15.42:0/3245835016 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 203 v203) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f33000334d0 con 0x7f3308078a10 2019-12-05T05:54:48.658 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 203 2019-12-05T05:54:48.659 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:54:48.659 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":203,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":8614,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6835/2511123581","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2511123581},{"type":"v1","addr":"172.21.15.42:6835","nonce":2511123581}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":203}],"filesystems":[{"mdsmap":{"epoch":202,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:53:04.600937+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6356},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6356":{"gid":6356,"name":"b","rank":0,"incarnation":199,"state":"up:active","state_seq":26,"addr":"172.21.15.40:6833/1523046776","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1523046776},{"type":"v1","addr":"172.21.15.40:6833","nonce":1523046776}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:54:48.661 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:48.654+0000 7f330f5d9700 1 -- 172.21.15.42:0/3245835016 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f32f003ae00 msgr2=0x7f32f003d310 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:54:48.661 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:48.654+0000 7f330f5d9700 1 --2- 172.21.15.42:0/3245835016 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f32f003ae00 0x7f32f003d310 crc :-1 s=READY pgs=710 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:54:48.661 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:48.654+0000 7f330f5d9700 1 -- 172.21.15.42:0/3245835016 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3308078a10 msgr2=0x7f3308112550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:54:48.661 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:48.654+0000 7f330f5d9700 1 --2- 172.21.15.42:0/3245835016 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3308078a10 0x7f3308112550 secure :-1 s=READY pgs=1668 cs=0 l=1 rx=0x7f3308113fd0 tx=0x7f33000115e0).stop 2019-12-05T05:54:48.662 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:48.654+0000 7f330f5d9700 1 -- 172.21.15.42:0/3245835016 shutdown_connections 2019-12-05T05:54:48.662 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:48.654+0000 7f330f5d9700 1 --2- 172.21.15.42:0/3245835016 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f32f003ae00 0x7f32f003d310 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:54:48.662 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:48.654+0000 7f330f5d9700 1 --2- 172.21.15.42:0/3245835016 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3308078a10 0x7f3308112550 unknown :-1 s=CLOSED pgs=1668 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:54:48.662 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:48.654+0000 7f330f5d9700 1 --2- 172.21.15.42:0/3245835016 >> v2:172.21.15.40:3300/0 conn(0x7f330805a210 0x7f33080a1d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:54:48.662 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:48.654+0000 7f330f5d9700 1 --2- 172.21.15.42:0/3245835016 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f33081138a0 0x7f33080784c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:54:48.662 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:48.654+0000 7f330f5d9700 1 -- 172.21.15.42:0/3245835016 shutdown_connections 2019-12-05T05:54:48.662 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:48.654+0000 7f330f5d9700 1 -- 172.21.15.42:0/3245835016 wait complete. 2019-12-05T05:54:48.663 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:48.654+0000 7f330f5d9700 1 -- 172.21.15.42:0/3245835016 >> 172.21.15.42:0/3245835016 conn(0x7f33080aabe0 msgr2=0x7f33081170c0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:54:48.671 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T05:54:50.672 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:54:50.786 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:50.766+0000 7fef3dd1a700 1 Processor -- start 2019-12-05T05:54:50.786 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:50.766+0000 7fef3dd1a700 1 -- start start 2019-12-05T05:54:50.787 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:50.770+0000 7fef3dd1a700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fef380a7740 0x7fef3808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:54:50.787 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:50.770+0000 7fef3dd1a700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef380a0710 con 0x7fef3809fae0 2019-12-05T05:54:50.787 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:50.770+0000 7fef3dd1a700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef380a9ba0 con 0x7fef3807e720 2019-12-05T05:54:50.787 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:50.770+0000 7fef3dd1a700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef380a0b00 con 0x7fef380a7740 2019-12-05T05:54:50.787 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:50.770+0000 7fef377fe700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fef3807e720 0x7fef3809f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:43830/0 (socket says 172.21.15.42:43830) 2019-12-05T05:54:50.787 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:50.770+0000 7fef377fe700 1 -- 172.21.15.42:0/3435176392 learned_addr learned my addr 172.21.15.42:0/3435176392 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:54:50.788 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:50.770+0000 7fef37fff700 1 --2- 172.21.15.42:0/3435176392 >> v2:172.21.15.40:3300/0 conn(0x7fef380a7740 0x7fef3808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:54:50.788 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:50.770+0000 7fef36ffd700 1 -- 172.21.15.42:0/3435176392 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 535819254 0 0) 0x7fef380a9ba0 con 0x7fef3807e720 2019-12-05T05:54:50.788 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:50.770+0000 7fef36ffd700 1 -- 172.21.15.42:0/3435176392 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef240030b0 con 0x7fef3807e720 2019-12-05T05:54:50.788 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:50.770+0000 7fef36ffd700 1 -- 172.21.15.42:0/3435176392 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3021662649 0 0) 0x7fef380a0710 con 0x7fef3809fae0 2019-12-05T05:54:50.788 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:50.770+0000 7fef36ffd700 1 -- 172.21.15.42:0/3435176392 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef380a9ba0 con 0x7fef3809fae0 2019-12-05T05:54:50.788 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:50.770+0000 7fef36ffd700 1 -- 172.21.15.42:0/3435176392 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2996527964 0 0) 0x7fef240030b0 con 0x7fef3807e720 2019-12-05T05:54:50.788 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:50.770+0000 7fef36ffd700 1 -- 172.21.15.42:0/3435176392 >> v2:172.21.15.40:3300/0 conn(0x7fef380a7740 msgr2=0x7fef3808df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:54:50.789 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:50.770+0000 7fef36ffd700 1 --2- 172.21.15.42:0/3435176392 >> v2:172.21.15.40:3300/0 conn(0x7fef380a7740 0x7fef3808df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:54:50.789 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:50.770+0000 7fef36ffd700 1 -- 172.21.15.42:0/3435176392 >> v1:172.21.15.40:6789/0 conn(0x7fef3809fae0 legacy=0x7fef3814efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:54:50.789 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:50.770+0000 7fef36ffd700 1 -- 172.21.15.42:0/3435176392 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef3807eb50 con 0x7fef3807e720 2019-12-05T05:54:50.789 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:50.770+0000 7fef36ffd700 1 -- 172.21.15.42:0/3435176392 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7fef28001750 con 0x7fef3807e720 2019-12-05T05:54:50.789 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:50.770+0000 7fef36ffd700 1 -- 172.21.15.42:0/3435176392 >> v1:172.21.15.42:6789/0 conn(0x7fef3807e720 legacy=0x7fef3809f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:54:50.789 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:50.770+0000 7fef36ffd700 1 --2- 172.21.15.42:0/3435176392 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fef24003910 0x7fef24003d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:54:50.789 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:50.770+0000 7fef36ffd700 1 --2- 172.21.15.42:0/3435176392 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fef24004420 0x7fef240048f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:54:50.790 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:50.770+0000 7fef36ffd700 1 --2- 172.21.15.42:0/3435176392 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fef24004fa0 0x7fef2404f700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:54:50.790 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:50.770+0000 7fef36ffd700 1 -- 172.21.15.42:0/3435176392 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef380a0b00 con 0x7fef24004fa0 2019-12-05T05:54:50.790 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:50.770+0000 7fef36ffd700 1 -- 172.21.15.42:0/3435176392 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef2404fec0 con 0x7fef24003910 2019-12-05T05:54:50.790 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:50.770+0000 7fef36ffd700 1 -- 172.21.15.42:0/3435176392 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef24050100 con 0x7fef24004420 2019-12-05T05:54:50.790 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:50.770+0000 7fef37fff700 1 --2- 172.21.15.42:0/3435176392 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fef24003910 0x7fef24003d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:54:50.791 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:50.770+0000 7fef377fe700 1 --2- 172.21.15.42:0/3435176392 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fef24004420 0x7fef240048f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:54:50.791 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:50.770+0000 7fef3cab8700 1 --2- 172.21.15.42:0/3435176392 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fef24004fa0 0x7fef2404f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:54:50.791 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:50.770+0000 7fef377fe700 1 -- 172.21.15.42:0/3435176392 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fef24003910 msgr2=0x7fef24003d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:54:50.791 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:50.770+0000 7fef377fe700 1 --2- 172.21.15.42:0/3435176392 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fef24003910 0x7fef24003d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:54:50.791 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:50.770+0000 7fef377fe700 1 -- 172.21.15.42:0/3435176392 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fef24004fa0 msgr2=0x7fef2404f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:54:50.791 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:50.770+0000 7fef377fe700 1 --2- 172.21.15.42:0/3435176392 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fef24004fa0 0x7fef2404f700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:54:50.792 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:50.770+0000 7fef377fe700 1 -- 172.21.15.42:0/3435176392 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef240503a0 con 0x7fef24004420 2019-12-05T05:54:50.792 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:50.770+0000 7fef3cab8700 1 -- 172.21.15.42:0/3435176392 reap_dead start 2019-12-05T05:54:50.792 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:50.770+0000 7fef37fff700 1 --2- 172.21.15.42:0/3435176392 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fef24003910 0x7fef24003d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:54:50.792 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:50.770+0000 7fef377fe700 1 --2- 172.21.15.42:0/3435176392 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fef24004420 0x7fef240048f0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fef28001520 tx=0x7fef2800c160).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:54:50.792 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:50.770+0000 7fef377fe700 1 --2- 172.21.15.42:0/3435176392 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fef24004420 0x7fef240048f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fef28001520 tx=0x7fef2800c160).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:54:50.792 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:50.770+0000 7fef377fe700 1 --2- 172.21.15.42:0/3435176392 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fef24004420 0x7fef240048f0 secure :-1 s=READY pgs=372 cs=0 l=1 rx=0x7fef28001520 tx=0x7fef2800c160).ready entity=mon.2 client_cookie=7fd011ddeb4b8f85 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:54:50.793 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:50.770+0000 7fef377fe700 1 --2- 172.21.15.42:0/3435176392 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fef24004420 0x7fef240048f0 secure :-1 s=THROTTLE_DONE pgs=372 cs=0 l=1 rx=0x7fef28001520 tx=0x7fef2800c160).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:54:50.793 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:50.770+0000 7fef36ffd700 1 -- 172.21.15.42:0/3435176392 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fef28010350 con 0x7fef24004420 2019-12-05T05:54:50.793 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:50.770+0000 7fef377fe700 1 --2- 172.21.15.42:0/3435176392 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fef24004420 0x7fef240048f0 secure :-1 s=THROTTLE_DONE pgs=372 cs=0 l=1 rx=0x7fef28001520 tx=0x7fef2800c160).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:54:50.793 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:50.774+0000 7fef377fe700 1 --2- 172.21.15.42:0/3435176392 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fef24004420 0x7fef240048f0 secure :-1 s=THROTTLE_DONE pgs=372 cs=0 l=1 rx=0x7fef28001520 tx=0x7fef2800c160).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:54:50.794 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:50.774+0000 7fef36ffd700 1 -- 172.21.15.42:0/3435176392 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef2800ed60 con 0x7fef24004420 2019-12-05T05:54:50.794 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:50.774+0000 7fef36ffd700 1 -- 172.21.15.42:0/3435176392 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fef28005460 con 0x7fef24004420 2019-12-05T05:54:50.794 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:50.774+0000 7fef3dd1a700 1 -- 172.21.15.42:0/3435176392 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fef24004420 msgr2=0x7fef240048f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:54:50.794 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:50.774+0000 7fef3dd1a700 1 --2- 172.21.15.42:0/3435176392 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fef24004420 0x7fef240048f0 secure :-1 s=READY pgs=372 cs=0 l=1 rx=0x7fef28001520 tx=0x7fef2800c160).stop 2019-12-05T05:54:50.794 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:50.774+0000 7fef3dd1a700 1 -- 172.21.15.42:0/3435176392 shutdown_connections 2019-12-05T05:54:50.794 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:50.774+0000 7fef3dd1a700 1 --2- 172.21.15.42:0/3435176392 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fef24004420 0x7fef240048f0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:54:50.794 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:50.774+0000 7fef3dd1a700 1 -- 172.21.15.42:0/3435176392 shutdown_connections 2019-12-05T05:54:50.795 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:50.774+0000 7fef3dd1a700 1 -- 172.21.15.42:0/3435176392 wait complete. 2019-12-05T05:54:50.795 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:50.774+0000 7fef3dd1a700 1 -- 172.21.15.42:0/3435176392 >> 172.21.15.42:0/3435176392 conn(0x7fef380aabe0 msgr2=0x7fef380a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:54:50.795 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:50.774+0000 7fef3dd1a700 1 Processor -- start 2019-12-05T05:54:50.795 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:50.774+0000 7fef3dd1a700 1 -- start start 2019-12-05T05:54:50.795 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:50.778+0000 7fef3dd1a700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fef381442b0 0x7fef381305d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:54:50.796 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:50.778+0000 7fef3dd1a700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef38063630 con 0x7fef3811c5e0 2019-12-05T05:54:50.796 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:50.778+0000 7fef3dd1a700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef38064270 con 0x7fef38130b20 2019-12-05T05:54:50.796 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:50.778+0000 7fef3dd1a700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef38143d70 con 0x7fef381442b0 2019-12-05T05:54:50.796 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:50.778+0000 7fef377fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fef381442b0 0x7fef381305d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:54:50.796 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:50.778+0000 7fef3cab8700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fef38130b20 0x7fef38137c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:43844/0 (socket says 172.21.15.42:43844) 2019-12-05T05:54:50.796 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:50.778+0000 7fef3cab8700 1 -- 172.21.15.42:0/348479087 learned_addr learned my addr 172.21.15.42:0/348479087 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:54:50.796 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:50.778+0000 7fef34ff9700 1 -- 172.21.15.42:0/348479087 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2764894001 0 0) 0x7fef38063630 con 0x7fef3811c5e0 2019-12-05T05:54:50.797 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:50.778+0000 7fef34ff9700 1 -- 172.21.15.42:0/348479087 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef1c0030b0 con 0x7fef3811c5e0 2019-12-05T05:54:50.797 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:50.778+0000 7fef377fe700 1 -- 172.21.15.42:0/348479087 >> v1:172.21.15.42:6789/0 conn(0x7fef38130b20 legacy=0x7fef38137c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:54:50.797 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:50.778+0000 7fef377fe700 1 -- 172.21.15.42:0/348479087 >> v1:172.21.15.40:6789/0 conn(0x7fef3811c5e0 legacy=0x7fef38065190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:54:50.797 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:50.778+0000 7fef377fe700 1 -- 172.21.15.42:0/348479087 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef3805ae30 con 0x7fef381442b0 2019-12-05T05:54:50.797 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:50.778+0000 7fef377fe700 1 --2- 172.21.15.42:0/348479087 >> v2:172.21.15.42:3300/0 conn(0x7fef381442b0 0x7fef381305d0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fef38148b90 tx=0x7fef28001bf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:54:50.797 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:50.778+0000 7fef377fe700 1 --2- 172.21.15.42:0/348479087 >> v2:172.21.15.42:3300/0 conn(0x7fef381442b0 0x7fef381305d0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fef38148b90 tx=0x7fef28001bf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:54:50.797 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:50.778+0000 7fef377fe700 1 --2- 172.21.15.42:0/348479087 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fef381442b0 0x7fef381305d0 secure :-1 s=READY pgs=1671 cs=0 l=1 rx=0x7fef38148b90 tx=0x7fef28001bf0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:54:50.798 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:50.778+0000 7fef377fe700 1 --2- 172.21.15.42:0/348479087 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fef381442b0 0x7fef381305d0 secure :-1 s=THROTTLE_DONE pgs=1671 cs=0 l=1 rx=0x7fef38148b90 tx=0x7fef28001bf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:54:50.798 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:50.778+0000 7fef34ff9700 1 -- 172.21.15.42:0/348479087 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fef28010160 con 0x7fef381442b0 2019-12-05T05:54:50.798 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:50.778+0000 7fef377fe700 1 --2- 172.21.15.42:0/348479087 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fef381442b0 0x7fef381305d0 secure :-1 s=THROTTLE_DONE pgs=1671 cs=0 l=1 rx=0x7fef38148b90 tx=0x7fef28001bf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:54:50.798 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:50.778+0000 7fef377fe700 1 --2- 172.21.15.42:0/348479087 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fef381442b0 0x7fef381305d0 secure :-1 s=THROTTLE_DONE pgs=1671 cs=0 l=1 rx=0x7fef38148b90 tx=0x7fef28001bf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:54:50.798 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:50.778+0000 7fef34ff9700 1 -- 172.21.15.42:0/348479087 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef28010550 con 0x7fef381442b0 2019-12-05T05:54:50.799 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:50.778+0000 7fef3dd1a700 1 -- 172.21.15.42:0/348479087 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef38063ef0 con 0x7fef381442b0 2019-12-05T05:54:50.799 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:50.778+0000 7fef34ff9700 1 -- 172.21.15.42:0/348479087 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fef28005600 con 0x7fef381442b0 2019-12-05T05:54:50.799 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:50.778+0000 7fef3dd1a700 1 -- 172.21.15.42:0/348479087 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef38063190 con 0x7fef381442b0 2019-12-05T05:54:50.800 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:50.778+0000 7fef377fe700 1 --2- 172.21.15.42:0/348479087 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fef381442b0 0x7fef381305d0 secure :-1 s=THROTTLE_DONE pgs=1671 cs=0 l=1 rx=0x7fef38148b90 tx=0x7fef28001bf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:54:50.801 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:50.782+0000 7fef34ff9700 1 -- 172.21.15.42:0/348479087 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fef28005910 con 0x7fef381442b0 2019-12-05T05:54:50.801 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:50.782+0000 7fef377fe700 1 --2- 172.21.15.42:0/348479087 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fef381442b0 0x7fef381305d0 secure :-1 s=THROTTLE_DONE pgs=1671 cs=0 l=1 rx=0x7fef38148b90 tx=0x7fef28001bf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:54:50.801 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:50.782+0000 7fef34ff9700 1 --2- 172.21.15.42:0/348479087 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fef1c03b2a0 0x7fef1c03d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:54:50.801 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:50.782+0000 7fef34ff9700 1 -- 172.21.15.42:0/348479087 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8955+0+0 (secure 0 0 0) 0x7fef2804b370 con 0x7fef381442b0 2019-12-05T05:54:50.801 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:50.782+0000 7fef37fff700 1 --2- 172.21.15.42:0/348479087 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fef1c03b2a0 0x7fef1c03d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:54:50.801 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:50.782+0000 7fef37fff700 1 --2- 172.21.15.42:0/348479087 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fef1c03b2a0 0x7fef1c03d7b0 crc :-1 s=READY pgs=711 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:54:50.801 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:50.782+0000 7fef3dd1a700 1 -- 172.21.15.42:0/348479087 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef3805d110 con 0x7fef381442b0 2019-12-05T05:54:50.802 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:50.790+0000 7fef377fe700 1 --2- 172.21.15.42:0/348479087 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fef381442b0 0x7fef381305d0 secure :-1 s=THROTTLE_DONE pgs=1671 cs=0 l=1 rx=0x7fef38148b90 tx=0x7fef28001bf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:54:50.802 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:50.790+0000 7fef34ff9700 1 -- 172.21.15.42:0/348479087 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fef28001c70 con 0x7fef381442b0 2019-12-05T05:54:50.958 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:50.950+0000 7fef3dd1a700 1 -- 172.21.15.42:0/348479087 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fef3805b8c0 con 0x7fef381442b0 2019-12-05T05:54:50.959 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:50.954+0000 7fef377fe700 1 --2- 172.21.15.42:0/348479087 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fef381442b0 0x7fef381305d0 secure :-1 s=THROTTLE_DONE pgs=1671 cs=0 l=1 rx=0x7fef38148b90 tx=0x7fef28001bf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:54:50.959 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:50.954+0000 7fef34ff9700 1 -- 172.21.15.42:0/348479087 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 203 v203) v1 ==== 95+0+2112 (secure 0 0 0) 0x7fef28005c20 con 0x7fef381442b0 2019-12-05T05:54:50.961 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:54:50.962 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":203,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":8614,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6835/2511123581","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2511123581},{"type":"v1","addr":"172.21.15.42:6835","nonce":2511123581}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":203}],"filesystems":[{"mdsmap":{"epoch":202,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:53:04.600937+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6356},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6356":{"gid":6356,"name":"b","rank":0,"incarnation":199,"state":"up:active","state_seq":26,"addr":"172.21.15.40:6833/1523046776","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1523046776},{"type":"v1","addr":"172.21.15.40:6833","nonce":1523046776}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:54:50.962 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 203 2019-12-05T05:54:50.964 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:50.958+0000 7fef3dd1a700 1 -- 172.21.15.42:0/348479087 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fef1c03b2a0 msgr2=0x7fef1c03d7b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:54:50.964 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:50.958+0000 7fef3dd1a700 1 --2- 172.21.15.42:0/348479087 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fef1c03b2a0 0x7fef1c03d7b0 crc :-1 s=READY pgs=711 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:54:50.965 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:50.958+0000 7fef3dd1a700 1 -- 172.21.15.42:0/348479087 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fef381442b0 msgr2=0x7fef381305d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:54:50.965 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:50.958+0000 7fef3dd1a700 1 --2- 172.21.15.42:0/348479087 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fef381442b0 0x7fef381305d0 secure :-1 s=READY pgs=1671 cs=0 l=1 rx=0x7fef38148b90 tx=0x7fef28001bf0).stop 2019-12-05T05:54:50.965 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:50.958+0000 7fef3dd1a700 1 -- 172.21.15.42:0/348479087 shutdown_connections 2019-12-05T05:54:50.965 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:50.958+0000 7fef3dd1a700 1 --2- 172.21.15.42:0/348479087 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fef1c03b2a0 0x7fef1c03d7b0 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:54:50.965 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:50.958+0000 7fef3dd1a700 1 --2- 172.21.15.42:0/348479087 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fef381442b0 0x7fef381305d0 unknown :-1 s=CLOSED pgs=1671 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:54:50.966 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:50.958+0000 7fef3dd1a700 1 -- 172.21.15.42:0/348479087 shutdown_connections 2019-12-05T05:54:50.966 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:50.958+0000 7fef3dd1a700 1 -- 172.21.15.42:0/348479087 wait complete. 2019-12-05T05:54:50.966 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:50.958+0000 7fef3dd1a700 1 -- 172.21.15.42:0/348479087 >> 172.21.15.42:0/348479087 conn(0x7fef3807df60 msgr2=0x7fef3807e4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:54:50.974 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T05:54:52.975 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:54:53.084 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:53.074+0000 7f2f2a895700 1 Processor -- start 2019-12-05T05:54:53.085 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:53.074+0000 7f2f2a895700 1 -- start start 2019-12-05T05:54:53.085 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:53.074+0000 7f2f2a895700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2f2408df80 0x7f2f2409f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:54:53.085 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:53.074+0000 7f2f2a895700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2f2409f830 0x7f2f2414efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:54:53.085 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:53.074+0000 7f2f2a895700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f2409fc60 con 0x7f2f240a7740 2019-12-05T05:54:53.086 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:53.074+0000 7f2f2a895700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f240a0b00 con 0x7f2f2409f830 2019-12-05T05:54:53.086 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:53.074+0000 7f2f2a895700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f2408e5b0 con 0x7f2f2408df80 2019-12-05T05:54:53.086 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:53.074+0000 7f2f23fff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2f2408df80 0x7f2f2409f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:54:53.086 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:53.074+0000 7f2f23fff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2f2408df80 0x7f2f2409f2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:43816/0 (socket says 172.21.15.42:43816) 2019-12-05T05:54:53.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:53.074+0000 7f2f23fff700 1 -- 172.21.15.42:0/1360448435 learned_addr learned my addr 172.21.15.42:0/1360448435 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:54:53.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:53.074+0000 7f2f29633700 1 --2- 172.21.15.42:0/1360448435 >> v2:172.21.15.40:3300/0 conn(0x7f2f2409f830 0x7f2f2414efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:54:53.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:53.074+0000 7f2f23fff700 1 -- 172.21.15.42:0/1360448435 >> v2:172.21.15.40:3300/0 conn(0x7f2f2409f830 msgr2=0x7f2f2414efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:54:53.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:53.074+0000 7f2f23fff700 1 --2- 172.21.15.42:0/1360448435 >> v2:172.21.15.40:3300/0 conn(0x7f2f2409f830 0x7f2f2414efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:54:53.088 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:53.074+0000 7f2f23fff700 1 -- 172.21.15.42:0/1360448435 >> v1:172.21.15.40:6789/0 conn(0x7f2f240a7740 legacy=0x7f2f2407e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:54:53.089 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:53.074+0000 7f2f23fff700 1 -- 172.21.15.42:0/1360448435 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f240a0710 con 0x7f2f2408df80 2019-12-05T05:54:53.089 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:53.074+0000 7f2f23fff700 1 --2- 172.21.15.42:0/1360448435 >> v2:172.21.15.42:3300/0 conn(0x7f2f2408df80 0x7f2f2409f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f2f18002f30 tx=0x7f2f1800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:54:53.089 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:53.074+0000 7f2f23fff700 1 --2- 172.21.15.42:0/1360448435 >> v2:172.21.15.42:3300/0 conn(0x7f2f2408df80 0x7f2f2409f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f2f18002f30 tx=0x7f2f1800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:54:53.089 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:53.074+0000 7f2f23fff700 1 --2- 172.21.15.42:0/1360448435 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2f2408df80 0x7f2f2409f2e0 secure :-1 s=READY pgs=1674 cs=0 l=1 rx=0x7f2f18002f30 tx=0x7f2f1800fd30).ready entity=mon.1 client_cookie=c12920ace19f0dfc server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:54:53.090 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:53.074+0000 7f2f23fff700 1 --2- 172.21.15.42:0/1360448435 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2f2408df80 0x7f2f2409f2e0 secure :-1 s=THROTTLE_DONE pgs=1674 cs=0 l=1 rx=0x7f2f18002f30 tx=0x7f2f1800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:54:53.090 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:53.074+0000 7f2f23fff700 1 --2- 172.21.15.42:0/1360448435 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2f2408df80 0x7f2f2409f2e0 secure :-1 s=THROTTLE_DONE pgs=1674 cs=0 l=1 rx=0x7f2f18002f30 tx=0x7f2f1800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:54:53.090 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:53.074+0000 7f2f237fe700 1 -- 172.21.15.42:0/1360448435 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2f18008c90 con 0x7f2f2408df80 2019-12-05T05:54:53.090 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:53.074+0000 7f2f23fff700 1 --2- 172.21.15.42:0/1360448435 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2f2408df80 0x7f2f2409f2e0 secure :-1 s=THROTTLE_DONE pgs=1674 cs=0 l=1 rx=0x7f2f18002f30 tx=0x7f2f1800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:54:53.090 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:53.074+0000 7f2f237fe700 1 -- 172.21.15.42:0/1360448435 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f180088b0 con 0x7f2f2408df80 2019-12-05T05:54:53.091 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:53.074+0000 7f2f237fe700 1 -- 172.21.15.42:0/1360448435 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2f18011ad0 con 0x7f2f2408df80 2019-12-05T05:54:53.091 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:53.074+0000 7f2f2a895700 1 -- 172.21.15.42:0/1360448435 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2f2408df80 msgr2=0x7f2f2409f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:54:53.091 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:53.074+0000 7f2f2a895700 1 --2- 172.21.15.42:0/1360448435 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2f2408df80 0x7f2f2409f2e0 secure :-1 s=READY pgs=1674 cs=0 l=1 rx=0x7f2f18002f30 tx=0x7f2f1800fd30).stop 2019-12-05T05:54:53.091 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:53.074+0000 7f2f2a895700 1 -- 172.21.15.42:0/1360448435 shutdown_connections 2019-12-05T05:54:53.092 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:53.074+0000 7f2f2a895700 1 --2- 172.21.15.42:0/1360448435 >> v2:172.21.15.40:3300/0 conn(0x7f2f2409f830 0x7f2f2414efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:54:53.092 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:53.074+0000 7f2f2a895700 1 --2- 172.21.15.42:0/1360448435 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2f2408df80 0x7f2f2409f2e0 unknown :-1 s=CLOSED pgs=1674 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:54:53.092 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:53.074+0000 7f2f2a895700 1 -- 172.21.15.42:0/1360448435 shutdown_connections 2019-12-05T05:54:53.092 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:53.074+0000 7f2f2a895700 1 -- 172.21.15.42:0/1360448435 wait complete. 2019-12-05T05:54:53.093 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:53.074+0000 7f2f2a895700 1 -- 172.21.15.42:0/1360448435 >> 172.21.15.42:0/1360448435 conn(0x7f2f240aabe0 msgr2=0x7f2f240a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:54:53.093 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:53.078+0000 7f2f2a895700 1 Processor -- start 2019-12-05T05:54:53.094 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:53.078+0000 7f2f2a895700 1 -- start start 2019-12-05T05:54:53.100 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:53.078+0000 7f2f2a895700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2f2411ecf0 0x7f2f2411ba30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:54:53.100 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:53.078+0000 7f2f2a895700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2f2411bf80 0x7f2f2411a650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:54:53.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:53.078+0000 7f2f2a895700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f2411d550 con 0x7f2f2411aba0 2019-12-05T05:54:53.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:53.078+0000 7f2f2a895700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f2411c3b0 con 0x7f2f2411bf80 2019-12-05T05:54:53.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:53.078+0000 7f2f2a895700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f2411afd0 con 0x7f2f2411ecf0 2019-12-05T05:54:53.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:53.078+0000 7f2f28e32700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2f2411ecf0 0x7f2f2411ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:54:53.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:53.078+0000 7f2f29633700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f2f2411aba0 0x7f2f2408eb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:43862/0 (socket says 172.21.15.42:43862) 2019-12-05T05:54:53.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:53.078+0000 7f2f28e32700 1 --2- 172.21.15.42:0/1029566391 >> v2:172.21.15.42:3300/0 conn(0x7f2f2411ecf0 0x7f2f2411ba30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:43820/0 (socket says 172.21.15.42:43820) 2019-12-05T05:54:53.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:53.078+0000 7f2f29633700 1 -- 172.21.15.42:0/1029566391 learned_addr learned my addr 172.21.15.42:0/1029566391 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:54:53.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:53.078+0000 7f2f23fff700 1 --2- 172.21.15.42:0/1029566391 >> v2:172.21.15.40:3300/0 conn(0x7f2f2411bf80 0x7f2f2411a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:54:53.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:53.078+0000 7f2f28e32700 1 -- 172.21.15.42:0/1029566391 >> v2:172.21.15.40:3300/0 conn(0x7f2f2411bf80 msgr2=0x7f2f2411a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:54:53.103 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:53.078+0000 7f2f28e32700 1 --2- 172.21.15.42:0/1029566391 >> v2:172.21.15.40:3300/0 conn(0x7f2f2411bf80 0x7f2f2411a650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:54:53.103 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:53.078+0000 7f2f28e32700 1 -- 172.21.15.42:0/1029566391 >> v1:172.21.15.42:6789/0 conn(0x7f2f2411aba0 legacy=0x7f2f2408eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:54:53.103 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:53.078+0000 7f2f28e32700 1 -- 172.21.15.42:0/1029566391 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f2411f120 con 0x7f2f2411ecf0 2019-12-05T05:54:53.103 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:53.078+0000 7f2f217fa700 1 -- 172.21.15.42:0/1029566391 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 370310809 0 0) 0x7f2f2411d550 con 0x7f2f2411aba0 2019-12-05T05:54:53.104 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:53.078+0000 7f2f28e32700 1 --2- 172.21.15.42:0/1029566391 >> v2:172.21.15.42:3300/0 conn(0x7f2f2411ecf0 0x7f2f2411ba30 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f2f14000e90 tx=0x7f2f140108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:54:53.104 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:53.078+0000 7f2f28e32700 1 --2- 172.21.15.42:0/1029566391 >> v2:172.21.15.42:3300/0 conn(0x7f2f2411ecf0 0x7f2f2411ba30 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f2f14000e90 tx=0x7f2f140108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:54:53.105 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:53.078+0000 7f2f28e32700 1 --2- 172.21.15.42:0/1029566391 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2f2411ecf0 0x7f2f2411ba30 secure :-1 s=READY pgs=1676 cs=0 l=1 rx=0x7f2f14000e90 tx=0x7f2f140108f0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:54:53.105 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:53.078+0000 7f2f28e32700 1 --2- 172.21.15.42:0/1029566391 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2f2411ecf0 0x7f2f2411ba30 secure :-1 s=THROTTLE_DONE pgs=1676 cs=0 l=1 rx=0x7f2f14000e90 tx=0x7f2f140108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:54:53.106 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:53.078+0000 7f2f217fa700 1 -- 172.21.15.42:0/1029566391 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2f1400cb90 con 0x7f2f2411ecf0 2019-12-05T05:54:53.106 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:53.078+0000 7f2f28e32700 1 --2- 172.21.15.42:0/1029566391 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2f2411ecf0 0x7f2f2411ba30 secure :-1 s=THROTTLE_DONE pgs=1676 cs=0 l=1 rx=0x7f2f14000e90 tx=0x7f2f140108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:54:53.108 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:53.078+0000 7f2f28e32700 1 --2- 172.21.15.42:0/1029566391 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2f2411ecf0 0x7f2f2411ba30 secure :-1 s=THROTTLE_DONE pgs=1676 cs=0 l=1 rx=0x7f2f14000e90 tx=0x7f2f140108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:54:53.108 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:53.078+0000 7f2f217fa700 1 -- 172.21.15.42:0/1029566391 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f1400c7b0 con 0x7f2f2411ecf0 2019-12-05T05:54:53.108 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:53.078+0000 7f2f2a895700 1 -- 172.21.15.42:0/1029566391 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f2408f370 con 0x7f2f2411ecf0 2019-12-05T05:54:53.109 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:53.078+0000 7f2f217fa700 1 -- 172.21.15.42:0/1029566391 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2f140110b0 con 0x7f2f2411ecf0 2019-12-05T05:54:53.109 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:53.078+0000 7f2f2a895700 1 -- 172.21.15.42:0/1029566391 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f2408f920 con 0x7f2f2411ecf0 2019-12-05T05:54:53.109 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:53.082+0000 7f2f28e32700 1 --2- 172.21.15.42:0/1029566391 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2f2411ecf0 0x7f2f2411ba30 secure :-1 s=THROTTLE_DONE pgs=1676 cs=0 l=1 rx=0x7f2f14000e90 tx=0x7f2f140108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:54:53.109 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:53.082+0000 7f2f217fa700 1 -- 172.21.15.42:0/1029566391 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f2f14011340 con 0x7f2f2411ecf0 2019-12-05T05:54:53.110 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:53.082+0000 7f2f28e32700 1 --2- 172.21.15.42:0/1029566391 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2f2411ecf0 0x7f2f2411ba30 secure :-1 s=THROTTLE_DONE pgs=1676 cs=0 l=1 rx=0x7f2f14000e90 tx=0x7f2f140108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:54:53.110 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:53.082+0000 7f2f217fa700 1 --2- 172.21.15.42:0/1029566391 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2f0403ac50 0x7f2f0403d160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:54:53.110 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:53.082+0000 7f2f217fa700 1 -- 172.21.15.42:0/1029566391 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8955+0+0 (secure 0 0 0) 0x7f2f1404b160 con 0x7f2f2411ecf0 2019-12-05T05:54:53.111 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:53.082+0000 7f2f23fff700 1 --2- 172.21.15.42:0/1029566391 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2f0403ac50 0x7f2f0403d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:54:53.111 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:53.082+0000 7f2f23fff700 1 --2- 172.21.15.42:0/1029566391 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2f0403ac50 0x7f2f0403d160 crc :-1 s=READY pgs=712 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:54:53.111 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:53.086+0000 7f2f2a895700 1 -- 172.21.15.42:0/1029566391 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f2405d110 con 0x7f2f2411ecf0 2019-12-05T05:54:53.111 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:53.094+0000 7f2f28e32700 1 --2- 172.21.15.42:0/1029566391 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2f2411ecf0 0x7f2f2411ba30 secure :-1 s=THROTTLE_DONE pgs=1676 cs=0 l=1 rx=0x7f2f14000e90 tx=0x7f2f140108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:54:53.112 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:53.094+0000 7f2f217fa700 1 -- 172.21.15.42:0/1029566391 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f2f140321e0 con 0x7f2f2411ecf0 2019-12-05T05:54:53.258 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:53.250+0000 7f2f2a895700 1 -- 172.21.15.42:0/1029566391 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2f2405b8c0 con 0x7f2f2411ecf0 2019-12-05T05:54:53.258 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:53.250+0000 7f2f28e32700 1 --2- 172.21.15.42:0/1029566391 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2f2411ecf0 0x7f2f2411ba30 secure :-1 s=THROTTLE_DONE pgs=1676 cs=0 l=1 rx=0x7f2f14000e90 tx=0x7f2f140108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:54:53.259 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:53.250+0000 7f2f217fa700 1 -- 172.21.15.42:0/1029566391 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 203 v203) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f2f14032440 con 0x7f2f2411ecf0 2019-12-05T05:54:53.259 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 203 2019-12-05T05:54:53.259 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:54:53.259 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":203,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":8614,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6835/2511123581","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2511123581},{"type":"v1","addr":"172.21.15.42:6835","nonce":2511123581}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":203}],"filesystems":[{"mdsmap":{"epoch":202,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:53:04.600937+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6356},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6356":{"gid":6356,"name":"b","rank":0,"incarnation":199,"state":"up:active","state_seq":26,"addr":"172.21.15.40:6833/1523046776","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1523046776},{"type":"v1","addr":"172.21.15.40:6833","nonce":1523046776}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:54:53.261 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:53.254+0000 7f2f2a895700 1 -- 172.21.15.42:0/1029566391 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2f0403ac50 msgr2=0x7f2f0403d160 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:54:53.262 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:53.254+0000 7f2f2a895700 1 --2- 172.21.15.42:0/1029566391 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2f0403ac50 0x7f2f0403d160 crc :-1 s=READY pgs=712 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:54:53.262 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:53.254+0000 7f2f2a895700 1 -- 172.21.15.42:0/1029566391 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2f2411ecf0 msgr2=0x7f2f2411ba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:54:53.262 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:53.254+0000 7f2f2a895700 1 --2- 172.21.15.42:0/1029566391 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2f2411ecf0 0x7f2f2411ba30 secure :-1 s=READY pgs=1676 cs=0 l=1 rx=0x7f2f14000e90 tx=0x7f2f140108f0).stop 2019-12-05T05:54:53.262 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:53.254+0000 7f2f2a895700 1 -- 172.21.15.42:0/1029566391 shutdown_connections 2019-12-05T05:54:53.262 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:53.254+0000 7f2f2a895700 1 --2- 172.21.15.42:0/1029566391 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2f0403ac50 0x7f2f0403d160 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:54:53.263 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:53.254+0000 7f2f2a895700 1 --2- 172.21.15.42:0/1029566391 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2f2411ecf0 0x7f2f2411ba30 unknown :-1 s=CLOSED pgs=1676 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:54:53.263 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:53.254+0000 7f2f2a895700 1 --2- 172.21.15.42:0/1029566391 >> v2:172.21.15.40:3300/0 conn(0x7f2f2411bf80 0x7f2f2411a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:54:53.263 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:53.254+0000 7f2f2a895700 1 -- 172.21.15.42:0/1029566391 shutdown_connections 2019-12-05T05:54:53.263 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:53.254+0000 7f2f2a895700 1 -- 172.21.15.42:0/1029566391 wait complete. 2019-12-05T05:54:53.263 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:53.254+0000 7f2f2a895700 1 -- 172.21.15.42:0/1029566391 >> 172.21.15.42:0/1029566391 conn(0x7f2f2408fde0 msgr2=0x7f2f241416d0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:54:53.271 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T05:54:55.272 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:54:55.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:55.370+0000 7fa1739ce700 1 Processor -- start 2019-12-05T05:54:55.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:55.370+0000 7fa1739ce700 1 -- start start 2019-12-05T05:54:55.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:55.370+0000 7fa1739ce700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa16c0a7740 0x7fa16c08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:54:55.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:55.370+0000 7fa1739ce700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa16c09fae0 0x7fa16c07eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:54:55.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:55.370+0000 7fa1739ce700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa16c0a0710 con 0x7fa16c07e720 2019-12-05T05:54:55.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:55.370+0000 7fa1739ce700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa16c0a0b00 con 0x7fa16c0a7740 2019-12-05T05:54:55.404 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:55.370+0000 7fa1739ce700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa16c08e5b0 con 0x7fa16c09fae0 2019-12-05T05:54:55.404 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:55.370+0000 7fa171f6b700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa16c0a7740 0x7fa16c08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:54:55.404 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:55.370+0000 7fa17276c700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa16c09fae0 0x7fa16c07eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:54:55.404 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:55.370+0000 7fa171f6b700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa16c0a7740 0x7fa16c08df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.42:59882/0 (socket says 172.21.15.42:59882) 2019-12-05T05:54:55.404 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:55.370+0000 7fa171f6b700 1 -- 172.21.15.42:0/2296432215 learned_addr learned my addr 172.21.15.42:0/2296432215 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:54:55.404 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:55.370+0000 7fa17176a700 1 --1- 172.21.15.42:0/2296432215 >> v1:172.21.15.40:6789/0 conn(0x7fa16c07e720 0x7fa16c09f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:53314/0 (socket says 172.21.15.42:53314) 2019-12-05T05:54:55.404 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:55.370+0000 7fa171f6b700 1 -- 172.21.15.42:0/2296432215 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa16c09fae0 msgr2=0x7fa16c07eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:54:55.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:55.370+0000 7fa171f6b700 1 --2- 172.21.15.42:0/2296432215 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa16c09fae0 0x7fa16c07eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:54:55.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:55.370+0000 7fa171f6b700 1 -- 172.21.15.42:0/2296432215 >> v1:172.21.15.40:6789/0 conn(0x7fa16c07e720 legacy=0x7fa16c09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:54:55.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:55.370+0000 7fa171f6b700 1 -- 172.21.15.42:0/2296432215 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa16c07f0a0 con 0x7fa16c0a7740 2019-12-05T05:54:55.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:55.370+0000 7fa17276c700 1 --2- 172.21.15.42:0/2296432215 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa16c09fae0 0x7fa16c07eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:54:55.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:55.370+0000 7fa171f6b700 1 --2- 172.21.15.42:0/2296432215 >> v2:172.21.15.40:3300/0 conn(0x7fa16c0a7740 0x7fa16c08df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fa168008790 tx=0x7fa15c001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:54:55.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:55.370+0000 7fa171f6b700 1 --2- 172.21.15.42:0/2296432215 >> v2:172.21.15.40:3300/0 conn(0x7fa16c0a7740 0x7fa16c08df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fa168008790 tx=0x7fa15c001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:54:55.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:55.370+0000 7fa171f6b700 1 --2- 172.21.15.42:0/2296432215 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa16c0a7740 0x7fa16c08df80 secure :-1 s=READY pgs=1281 cs=0 l=1 rx=0x7fa168008790 tx=0x7fa15c001150).ready entity=mon.0 client_cookie=55da1f446f0e75f6 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:54:55.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:55.370+0000 7fa171f6b700 1 --2- 172.21.15.42:0/2296432215 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa16c0a7740 0x7fa16c08df80 secure :-1 s=THROTTLE_DONE pgs=1281 cs=0 l=1 rx=0x7fa168008790 tx=0x7fa15c001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:54:55.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:55.370+0000 7fa171f6b700 1 --2- 172.21.15.42:0/2296432215 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa16c0a7740 0x7fa16c08df80 secure :-1 s=THROTTLE_DONE pgs=1281 cs=0 l=1 rx=0x7fa168008790 tx=0x7fa15c001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:54:55.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:55.370+0000 7fa170f69700 1 -- 172.21.15.42:0/2296432215 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa168008c90 con 0x7fa16c0a7740 2019-12-05T05:54:55.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:55.370+0000 7fa171f6b700 1 --2- 172.21.15.42:0/2296432215 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa16c0a7740 0x7fa16c08df80 secure :-1 s=THROTTLE_DONE pgs=1281 cs=0 l=1 rx=0x7fa168008790 tx=0x7fa15c001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:54:55.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:55.370+0000 7fa170f69700 1 -- 172.21.15.42:0/2296432215 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1680088b0 con 0x7fa16c0a7740 2019-12-05T05:54:55.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:55.370+0000 7fa170f69700 1 -- 172.21.15.42:0/2296432215 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa168011a10 con 0x7fa16c0a7740 2019-12-05T05:54:55.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:55.370+0000 7fa1739ce700 1 -- 172.21.15.42:0/2296432215 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa16c0a7740 msgr2=0x7fa16c08df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:54:55.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:55.370+0000 7fa1739ce700 1 --2- 172.21.15.42:0/2296432215 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa16c0a7740 0x7fa16c08df80 secure :-1 s=READY pgs=1281 cs=0 l=1 rx=0x7fa168008790 tx=0x7fa15c001150).stop 2019-12-05T05:54:55.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:55.370+0000 7fa1739ce700 1 -- 172.21.15.42:0/2296432215 shutdown_connections 2019-12-05T05:54:55.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:55.370+0000 7fa1739ce700 1 --2- 172.21.15.42:0/2296432215 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa16c09fae0 0x7fa16c07eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:54:55.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:55.370+0000 7fa1739ce700 1 --2- 172.21.15.42:0/2296432215 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa16c0a7740 0x7fa16c08df80 unknown :-1 s=CLOSED pgs=1281 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:54:55.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:55.370+0000 7fa1739ce700 1 -- 172.21.15.42:0/2296432215 shutdown_connections 2019-12-05T05:54:55.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:55.370+0000 7fa1739ce700 1 -- 172.21.15.42:0/2296432215 wait complete. 2019-12-05T05:54:55.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:55.370+0000 7fa1739ce700 1 -- 172.21.15.42:0/2296432215 >> 172.21.15.42:0/2296432215 conn(0x7fa16c0aabe0 msgr2=0x7fa16c0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:54:55.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:55.374+0000 7fa1739ce700 1 Processor -- start 2019-12-05T05:54:55.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:55.374+0000 7fa1739ce700 1 -- start start 2019-12-05T05:54:55.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:55.374+0000 7fa1739ce700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa16c07de90 0x7fa16c114e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:54:55.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:55.374+0000 7fa1739ce700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa16c113ee0 0x7fa16c1191c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:54:55.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:55.374+0000 7fa1739ce700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa16c115390 con 0x7fa16c113ab0 2019-12-05T05:54:55.415 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:55.374+0000 7fa1739ce700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa16c0a1a80 con 0x7fa16c113ee0 2019-12-05T05:54:55.415 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:55.374+0000 7fa1739ce700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa16c0a1d20 con 0x7fa16c07de90 2019-12-05T05:54:55.415 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:55.374+0000 7fa171f6b700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa16c07de90 0x7fa16c114e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:54:55.415 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:55.374+0000 7fa17276c700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa16c113ee0 0x7fa16c1191c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:54:55.415 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:55.374+0000 7fa171f6b700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa16c07de90 0x7fa16c114e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:44306/0 (socket says 172.21.15.42:44306) 2019-12-05T05:54:55.416 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:55.374+0000 7fa171f6b700 1 -- 172.21.15.42:0/816148985 learned_addr learned my addr 172.21.15.42:0/816148985 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:54:55.416 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:55.378+0000 7fa171f6b700 1 -- 172.21.15.42:0/816148985 >> v2:172.21.15.40:3300/0 conn(0x7fa16c113ee0 msgr2=0x7fa16c1191c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:54:55.416 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:55.378+0000 7fa171f6b700 1 --2- 172.21.15.42:0/816148985 >> v2:172.21.15.40:3300/0 conn(0x7fa16c113ee0 0x7fa16c1191c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:54:55.416 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:55.378+0000 7fa171f6b700 1 -- 172.21.15.42:0/816148985 >> v1:172.21.15.40:6789/0 conn(0x7fa16c113ab0 legacy=0x7fa16c1189c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:54:55.416 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:55.378+0000 7fa171f6b700 1 -- 172.21.15.42:0/816148985 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa16c11bce0 con 0x7fa16c07de90 2019-12-05T05:54:55.416 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:55.378+0000 7fa171f6b700 1 --2- 172.21.15.42:0/816148985 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa16c07de90 0x7fa16c114e40 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fa168008790 tx=0x7fa168008b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:54:55.416 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:55.378+0000 7fa171f6b700 1 --2- 172.21.15.42:0/816148985 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa16c07de90 0x7fa16c114e40 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fa168008790 tx=0x7fa168008b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:54:55.417 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:55.378+0000 7fa171f6b700 1 --2- 172.21.15.42:0/816148985 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa16c07de90 0x7fa16c114e40 secure :-1 s=READY pgs=373 cs=0 l=1 rx=0x7fa168008790 tx=0x7fa168008b60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:54:55.417 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:55.378+0000 7fa171f6b700 1 --2- 172.21.15.42:0/816148985 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa16c07de90 0x7fa16c114e40 secure :-1 s=THROTTLE_DONE pgs=373 cs=0 l=1 rx=0x7fa168008790 tx=0x7fa168008b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:54:55.417 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:55.378+0000 7fa1627fc700 1 -- 172.21.15.42:0/816148985 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa16800f470 con 0x7fa16c07de90 2019-12-05T05:54:55.417 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:55.378+0000 7fa171f6b700 1 --2- 172.21.15.42:0/816148985 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa16c07de90 0x7fa16c114e40 secure :-1 s=THROTTLE_DONE pgs=373 cs=0 l=1 rx=0x7fa168008790 tx=0x7fa168008b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:54:55.417 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:55.378+0000 7fa171f6b700 1 --2- 172.21.15.42:0/816148985 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa16c07de90 0x7fa16c114e40 secure :-1 s=THROTTLE_DONE pgs=373 cs=0 l=1 rx=0x7fa168008790 tx=0x7fa168008b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:54:55.417 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:55.378+0000 7fa1627fc700 1 -- 172.21.15.42:0/816148985 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa168011630 con 0x7fa16c07de90 2019-12-05T05:54:55.417 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:55.378+0000 7fa1627fc700 1 -- 172.21.15.42:0/816148985 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa168005730 con 0x7fa16c07de90 2019-12-05T05:54:55.418 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:55.378+0000 7fa1739ce700 1 -- 172.21.15.42:0/816148985 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa16c119710 con 0x7fa16c07de90 2019-12-05T05:54:55.418 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:55.378+0000 7fa1739ce700 1 -- 172.21.15.42:0/816148985 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa16c141ec0 con 0x7fa16c07de90 2019-12-05T05:54:55.418 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:55.378+0000 7fa171f6b700 1 --2- 172.21.15.42:0/816148985 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa16c07de90 0x7fa16c114e40 secure :-1 s=THROTTLE_DONE pgs=373 cs=0 l=1 rx=0x7fa168008790 tx=0x7fa168008b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:54:55.418 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:55.378+0000 7fa1627fc700 1 -- 172.21.15.42:0/816148985 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fa1680130b0 con 0x7fa16c07de90 2019-12-05T05:54:55.418 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:55.378+0000 7fa171f6b700 1 --2- 172.21.15.42:0/816148985 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa16c07de90 0x7fa16c114e40 secure :-1 s=THROTTLE_DONE pgs=373 cs=0 l=1 rx=0x7fa168008790 tx=0x7fa168008b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:54:55.418 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:55.378+0000 7fa1627fc700 1 --2- 172.21.15.42:0/816148985 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa15403ad60 0x7fa15403d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:54:55.418 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:55.382+0000 7fa1627fc700 1 -- 172.21.15.42:0/816148985 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8955+0+0 (secure 0 0 0) 0x7fa16804b550 con 0x7fa16c07de90 2019-12-05T05:54:55.419 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:55.382+0000 7fa17176a700 1 --2- 172.21.15.42:0/816148985 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa15403ad60 0x7fa15403d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:54:55.419 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:55.382+0000 7fa17176a700 1 --2- 172.21.15.42:0/816148985 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa15403ad60 0x7fa15403d270 crc :-1 s=READY pgs=713 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:54:55.419 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:55.382+0000 7fa1739ce700 1 -- 172.21.15.42:0/816148985 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa16c05d110 con 0x7fa16c07de90 2019-12-05T05:54:55.419 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:55.394+0000 7fa171f6b700 1 --2- 172.21.15.42:0/816148985 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa16c07de90 0x7fa16c114e40 secure :-1 s=THROTTLE_DONE pgs=373 cs=0 l=1 rx=0x7fa168008790 tx=0x7fa168008b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:54:55.419 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:55.394+0000 7fa1627fc700 1 -- 172.21.15.42:0/816148985 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fa1680301f0 con 0x7fa16c07de90 2019-12-05T05:54:55.571 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:55.562+0000 7fa1739ce700 1 -- 172.21.15.42:0/816148985 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa16c05b8c0 con 0x7fa16c07de90 2019-12-05T05:54:55.571 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:55.570+0000 7fa171f6b700 1 --2- 172.21.15.42:0/816148985 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa16c07de90 0x7fa16c114e40 secure :-1 s=THROTTLE_DONE pgs=373 cs=0 l=1 rx=0x7fa168008790 tx=0x7fa168008b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:54:55.572 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:55.570+0000 7fa1627fc700 1 -- 172.21.15.42:0/816148985 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 203 v203) v1 ==== 95+0+2112 (secure 0 0 0) 0x7fa168038a50 con 0x7fa16c07de90 2019-12-05T05:54:55.574 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 203 2019-12-05T05:54:55.574 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:54:55.574 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":203,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":8614,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6835/2511123581","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2511123581},{"type":"v1","addr":"172.21.15.42:6835","nonce":2511123581}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":203}],"filesystems":[{"mdsmap":{"epoch":202,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:53:04.600937+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6356},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6356":{"gid":6356,"name":"b","rank":0,"incarnation":199,"state":"up:active","state_seq":26,"addr":"172.21.15.40:6833/1523046776","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1523046776},{"type":"v1","addr":"172.21.15.40:6833","nonce":1523046776}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:54:55.576 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:55.570+0000 7fa1739ce700 1 -- 172.21.15.42:0/816148985 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa15403ad60 msgr2=0x7fa15403d270 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:54:55.577 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:55.570+0000 7fa1739ce700 1 --2- 172.21.15.42:0/816148985 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa15403ad60 0x7fa15403d270 crc :-1 s=READY pgs=713 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:54:55.577 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:55.570+0000 7fa1739ce700 1 -- 172.21.15.42:0/816148985 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa16c07de90 msgr2=0x7fa16c114e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:54:55.577 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:55.570+0000 7fa1739ce700 1 --2- 172.21.15.42:0/816148985 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa16c07de90 0x7fa16c114e40 secure :-1 s=READY pgs=373 cs=0 l=1 rx=0x7fa168008790 tx=0x7fa168008b60).stop 2019-12-05T05:54:55.577 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:55.570+0000 7fa1739ce700 1 -- 172.21.15.42:0/816148985 shutdown_connections 2019-12-05T05:54:55.577 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:55.570+0000 7fa1739ce700 1 --2- 172.21.15.42:0/816148985 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa15403ad60 0x7fa15403d270 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:54:55.577 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:55.570+0000 7fa1739ce700 1 --2- 172.21.15.42:0/816148985 >> v2:172.21.15.40:3300/0 conn(0x7fa16c113ee0 0x7fa16c1191c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:54:55.578 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:55.570+0000 7fa1739ce700 1 --2- 172.21.15.42:0/816148985 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa16c07de90 0x7fa16c114e40 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:54:55.578 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:55.570+0000 7fa1739ce700 1 -- 172.21.15.42:0/816148985 shutdown_connections 2019-12-05T05:54:55.578 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:55.570+0000 7fa1739ce700 1 -- 172.21.15.42:0/816148985 wait complete. 2019-12-05T05:54:55.578 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:55.570+0000 7fa1739ce700 1 -- 172.21.15.42:0/816148985 >> 172.21.15.42:0/816148985 conn(0x7fa16c0aabe0 msgr2=0x7fa16c0a7f90 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:54:55.587 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T05:54:57.587 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:54:57.696 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:57.686+0000 7f974a38f700 1 Processor -- start 2019-12-05T05:54:57.697 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:57.686+0000 7f974a38f700 1 -- start start 2019-12-05T05:54:57.697 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:57.686+0000 7f974a38f700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f974408df80 0x7f974409f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:54:57.697 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:57.686+0000 7f974a38f700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f974409fc60 con 0x7f974409f830 2019-12-05T05:54:57.697 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:57.686+0000 7f974a38f700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97440a0710 con 0x7f97440a7740 2019-12-05T05:54:57.697 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:57.686+0000 7f974a38f700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97440a0b00 con 0x7f974408df80 2019-12-05T05:54:57.697 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:57.686+0000 7f974892c700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f97440a7740 0x7f974407e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:43888/0 (socket says 172.21.15.42:43888) 2019-12-05T05:54:57.697 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:57.686+0000 7f974892c700 1 -- 172.21.15.42:0/2391642114 learned_addr learned my addr 172.21.15.42:0/2391642114 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:54:57.698 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:57.686+0000 7f9743fff700 1 --2- 172.21.15.42:0/2391642114 >> v2:172.21.15.40:3300/0 conn(0x7f974408df80 0x7f974409f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:54:57.698 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:57.686+0000 7f97437fe700 1 -- 172.21.15.42:0/2391642114 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3088043024 0 0) 0x7f97440a0710 con 0x7f97440a7740 2019-12-05T05:54:57.698 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:57.686+0000 7f97437fe700 1 -- 172.21.15.42:0/2391642114 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97340030b0 con 0x7f97440a7740 2019-12-05T05:54:57.698 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:57.686+0000 7f97437fe700 1 -- 172.21.15.42:0/2391642114 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3877374570 0 0) 0x7f974409fc60 con 0x7f974409f830 2019-12-05T05:54:57.698 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:57.686+0000 7f97437fe700 1 -- 172.21.15.42:0/2391642114 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97440a0710 con 0x7f974409f830 2019-12-05T05:54:57.698 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:57.686+0000 7f97437fe700 1 -- 172.21.15.42:0/2391642114 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 974161240 0 0) 0x7f97340030b0 con 0x7f97440a7740 2019-12-05T05:54:57.698 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:57.686+0000 7f9743fff700 1 -- 172.21.15.42:0/2391642114 >> v1:172.21.15.42:6789/0 conn(0x7f97440a7740 legacy=0x7f974407e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:54:57.699 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:57.686+0000 7f9743fff700 1 -- 172.21.15.42:0/2391642114 >> v1:172.21.15.40:6789/0 conn(0x7f974409f830 legacy=0x7f974414efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:54:57.699 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:57.686+0000 7f9743fff700 1 -- 172.21.15.42:0/2391642114 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f974407ef20 con 0x7f974408df80 2019-12-05T05:54:57.699 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:57.686+0000 7f9743fff700 1 --2- 172.21.15.42:0/2391642114 >> v2:172.21.15.40:3300/0 conn(0x7f974408df80 0x7f974409f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f9730008790 tx=0x7f9738001950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:54:57.699 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:57.686+0000 7f9743fff700 1 --2- 172.21.15.42:0/2391642114 >> v2:172.21.15.40:3300/0 conn(0x7f974408df80 0x7f974409f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f9730008790 tx=0x7f9738001950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:54:57.699 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:57.686+0000 7f9743fff700 1 --2- 172.21.15.42:0/2391642114 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f974408df80 0x7f974409f2e0 secure :-1 s=READY pgs=1284 cs=0 l=1 rx=0x7f9730008790 tx=0x7f9738001950).ready entity=mon.0 client_cookie=49a1a7450153fb12 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:54:57.699 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:57.686+0000 7f9743fff700 1 --2- 172.21.15.42:0/2391642114 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f974408df80 0x7f974409f2e0 secure :-1 s=THROTTLE_DONE pgs=1284 cs=0 l=1 rx=0x7f9730008790 tx=0x7f9738001950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:54:57.699 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:57.686+0000 7f97437fe700 1 -- 172.21.15.42:0/2391642114 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9738001750 con 0x7f974408df80 2019-12-05T05:54:57.700 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:57.686+0000 7f9743fff700 1 --2- 172.21.15.42:0/2391642114 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f974408df80 0x7f974409f2e0 secure :-1 s=THROTTLE_DONE pgs=1284 cs=0 l=1 rx=0x7f9730008790 tx=0x7f9738001950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:54:57.700 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:57.686+0000 7f9743fff700 1 --2- 172.21.15.42:0/2391642114 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f974408df80 0x7f974409f2e0 secure :-1 s=THROTTLE_DONE pgs=1284 cs=0 l=1 rx=0x7f9730008790 tx=0x7f9738001950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:54:57.700 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:57.686+0000 7f97437fe700 1 -- 172.21.15.42:0/2391642114 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9738001560 con 0x7f974408df80 2019-12-05T05:54:57.700 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:57.686+0000 7f97437fe700 1 -- 172.21.15.42:0/2391642114 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9730011360 con 0x7f974408df80 2019-12-05T05:54:57.700 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:57.686+0000 7f974a38f700 1 -- 172.21.15.42:0/2391642114 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f974408df80 msgr2=0x7f974409f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:54:57.700 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:57.686+0000 7f974a38f700 1 --2- 172.21.15.42:0/2391642114 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f974408df80 0x7f974409f2e0 secure :-1 s=READY pgs=1284 cs=0 l=1 rx=0x7f9730008790 tx=0x7f9738001950).stop 2019-12-05T05:54:57.701 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:57.686+0000 7f974a38f700 1 -- 172.21.15.42:0/2391642114 shutdown_connections 2019-12-05T05:54:57.701 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:57.686+0000 7f974a38f700 1 --2- 172.21.15.42:0/2391642114 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f974408df80 0x7f974409f2e0 unknown :-1 s=CLOSED pgs=1284 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:54:57.701 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:57.686+0000 7f974a38f700 1 -- 172.21.15.42:0/2391642114 shutdown_connections 2019-12-05T05:54:57.701 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:57.686+0000 7f974a38f700 1 -- 172.21.15.42:0/2391642114 wait complete. 2019-12-05T05:54:57.701 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:57.686+0000 7f974a38f700 1 -- 172.21.15.42:0/2391642114 >> 172.21.15.42:0/2391642114 conn(0x7f97440aabe0 msgr2=0x7f97440a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:54:57.705 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:57.690+0000 7f974a38f700 1 Processor -- start 2019-12-05T05:54:57.705 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:57.690+0000 7f974a38f700 1 -- start start 2019-12-05T05:54:57.705 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:57.690+0000 7f974a38f700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f974411c360 0x7f9744130e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:54:57.705 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:57.690+0000 7f974a38f700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f97441357d0 0x7f9744137c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:54:57.705 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:57.690+0000 7f974a38f700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9744130390 con 0x7f97441235c0 2019-12-05T05:54:57.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:57.690+0000 7f974a38f700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f974407c8e0 con 0x7f97441357d0 2019-12-05T05:54:57.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:57.690+0000 7f974a38f700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f974407cb80 con 0x7f974411c360 2019-12-05T05:54:57.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:57.690+0000 7f974892c700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f974411c360 0x7f9744130e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:54:57.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:57.690+0000 7f974912d700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f97441357d0 0x7f9744137c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:54:57.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:57.690+0000 7f974892c700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f974411c360 0x7f9744130e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:44322/0 (socket says 172.21.15.42:44322) 2019-12-05T05:54:57.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:57.690+0000 7f974892c700 1 -- 172.21.15.42:0/3892361025 learned_addr learned my addr 172.21.15.42:0/3892361025 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:54:57.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:57.694+0000 7f974892c700 1 -- 172.21.15.42:0/3892361025 >> v2:172.21.15.40:3300/0 conn(0x7f97441357d0 msgr2=0x7f9744137c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:54:57.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:57.694+0000 7f974892c700 1 --2- 172.21.15.42:0/3892361025 >> v2:172.21.15.40:3300/0 conn(0x7f97441357d0 0x7f9744137c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:54:57.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:57.694+0000 7f974892c700 1 -- 172.21.15.42:0/3892361025 >> v1:172.21.15.40:6789/0 conn(0x7f97441235c0 legacy=0x7f97440a4f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:54:57.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:57.694+0000 7f97417fa700 1 -- 172.21.15.42:0/3892361025 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3762303297 0 0) 0x7f9744130390 con 0x7f97441235c0 2019-12-05T05:54:57.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:57.694+0000 7f974892c700 1 -- 172.21.15.42:0/3892361025 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97440a5730 con 0x7f974411c360 2019-12-05T05:54:57.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:57.694+0000 7f974912d700 1 --2- 172.21.15.42:0/3892361025 >> v2:172.21.15.40:3300/0 conn(0x7f97441357d0 0x7f9744137c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:54:57.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:57.694+0000 7f974892c700 1 --2- 172.21.15.42:0/3892361025 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f974411c360 0x7f9744130e80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f9738000fa0 tx=0x7f9738010a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:54:57.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:57.694+0000 7f974892c700 1 --2- 172.21.15.42:0/3892361025 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f974411c360 0x7f9744130e80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f9738000fa0 tx=0x7f9738010a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:54:57.708 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:57.694+0000 7f974892c700 1 --2- 172.21.15.42:0/3892361025 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f974411c360 0x7f9744130e80 secure :-1 s=READY pgs=374 cs=0 l=1 rx=0x7f9738000fa0 tx=0x7f9738010a90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:54:57.708 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:57.694+0000 7f974892c700 1 --2- 172.21.15.42:0/3892361025 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f974411c360 0x7f9744130e80 secure :-1 s=THROTTLE_DONE pgs=374 cs=0 l=1 rx=0x7f9738000fa0 tx=0x7f9738010a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:54:57.708 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:57.694+0000 7f97417fa700 1 -- 172.21.15.42:0/3892361025 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f973800c890 con 0x7f974411c360 2019-12-05T05:54:57.708 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:57.694+0000 7f974a38f700 1 -- 172.21.15.42:0/3892361025 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97441239f0 con 0x7f974411c360 2019-12-05T05:54:57.708 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:57.694+0000 7f974892c700 1 --2- 172.21.15.42:0/3892361025 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f974411c360 0x7f9744130e80 secure :-1 s=THROTTLE_DONE pgs=374 cs=0 l=1 rx=0x7f9738000fa0 tx=0x7f9738010a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:54:57.708 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:57.694+0000 7f97417fa700 1 -- 172.21.15.42:0/3892361025 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f973800c4b0 con 0x7f974411c360 2019-12-05T05:54:57.708 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:57.694+0000 7f974892c700 1 --2- 172.21.15.42:0/3892361025 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f974411c360 0x7f9744130e80 secure :-1 s=THROTTLE_DONE pgs=374 cs=0 l=1 rx=0x7f9738000fa0 tx=0x7f9738010a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:54:57.709 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:57.694+0000 7f974a38f700 1 -- 172.21.15.42:0/3892361025 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f974407c440 con 0x7f974411c360 2019-12-05T05:54:57.709 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:57.694+0000 7f97417fa700 1 -- 172.21.15.42:0/3892361025 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9738004430 con 0x7f974411c360 2019-12-05T05:54:57.709 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:57.694+0000 7f974892c700 1 --2- 172.21.15.42:0/3892361025 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f974411c360 0x7f9744130e80 secure :-1 s=THROTTLE_DONE pgs=374 cs=0 l=1 rx=0x7f9738000fa0 tx=0x7f9738010a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:54:57.709 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:57.694+0000 7f97417fa700 1 -- 172.21.15.42:0/3892361025 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f97380046d0 con 0x7f974411c360 2019-12-05T05:54:57.709 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:57.694+0000 7f974892c700 1 --2- 172.21.15.42:0/3892361025 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f974411c360 0x7f9744130e80 secure :-1 s=THROTTLE_DONE pgs=374 cs=0 l=1 rx=0x7f9738000fa0 tx=0x7f9738010a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:54:57.709 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:57.698+0000 7f97417fa700 1 --2- 172.21.15.42:0/3892361025 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f972403ac10 0x7f972403d120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:54:57.709 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:57.698+0000 7f97417fa700 1 -- 172.21.15.42:0/3892361025 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8955+0+0 (secure 0 0 0) 0x7f973804add0 con 0x7f974411c360 2019-12-05T05:54:57.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:57.698+0000 7f9743fff700 1 --2- 172.21.15.42:0/3892361025 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f972403ac10 0x7f972403d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:54:57.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:57.698+0000 7f9743fff700 1 --2- 172.21.15.42:0/3892361025 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f972403ac10 0x7f972403d120 crc :-1 s=READY pgs=714 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:54:57.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:57.698+0000 7f974a38f700 1 -- 172.21.15.42:0/3892361025 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f974407cd40 con 0x7f974411c360 2019-12-05T05:54:57.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:57.710+0000 7f974892c700 1 --2- 172.21.15.42:0/3892361025 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f974411c360 0x7f9744130e80 secure :-1 s=THROTTLE_DONE pgs=374 cs=0 l=1 rx=0x7f9738000fa0 tx=0x7f9738010a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:54:57.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:57.710+0000 7f97417fa700 1 -- 172.21.15.42:0/3892361025 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f9738031e30 con 0x7f974411c360 2019-12-05T05:54:57.882 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:57.874+0000 7f974a38f700 1 -- 172.21.15.42:0/3892361025 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f974405d110 con 0x7f974411c360 2019-12-05T05:54:57.883 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:57.882+0000 7f974892c700 1 --2- 172.21.15.42:0/3892361025 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f974411c360 0x7f9744130e80 secure :-1 s=THROTTLE_DONE pgs=374 cs=0 l=1 rx=0x7f9738000fa0 tx=0x7f9738010a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:54:57.884 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:57.882+0000 7f97417fa700 1 -- 172.21.15.42:0/3892361025 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 203 v203) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f9738032090 con 0x7f974411c360 2019-12-05T05:54:57.886 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 203 2019-12-05T05:54:57.886 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:54:57.886 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":203,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":8614,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6835/2511123581","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2511123581},{"type":"v1","addr":"172.21.15.42:6835","nonce":2511123581}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":203}],"filesystems":[{"mdsmap":{"epoch":202,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:53:04.600937+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6356},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6356":{"gid":6356,"name":"b","rank":0,"incarnation":199,"state":"up:active","state_seq":26,"addr":"172.21.15.40:6833/1523046776","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1523046776},{"type":"v1","addr":"172.21.15.40:6833","nonce":1523046776}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:54:57.889 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:57.886+0000 7f974a38f700 1 -- 172.21.15.42:0/3892361025 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f972403ac10 msgr2=0x7f972403d120 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:54:57.889 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:57.886+0000 7f974a38f700 1 --2- 172.21.15.42:0/3892361025 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f972403ac10 0x7f972403d120 crc :-1 s=READY pgs=714 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:54:57.889 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:57.886+0000 7f974a38f700 1 -- 172.21.15.42:0/3892361025 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f974411c360 msgr2=0x7f9744130e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:54:57.889 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:57.886+0000 7f974a38f700 1 --2- 172.21.15.42:0/3892361025 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f974411c360 0x7f9744130e80 secure :-1 s=READY pgs=374 cs=0 l=1 rx=0x7f9738000fa0 tx=0x7f9738010a90).stop 2019-12-05T05:54:57.889 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:57.886+0000 7f974a38f700 1 -- 172.21.15.42:0/3892361025 shutdown_connections 2019-12-05T05:54:57.890 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:57.886+0000 7f974a38f700 1 --2- 172.21.15.42:0/3892361025 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f972403ac10 0x7f972403d120 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:54:57.890 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:57.886+0000 7f974a38f700 1 --2- 172.21.15.42:0/3892361025 >> v2:172.21.15.40:3300/0 conn(0x7f97441357d0 0x7f9744137c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:54:57.890 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:57.886+0000 7f974a38f700 1 --2- 172.21.15.42:0/3892361025 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f974411c360 0x7f9744130e80 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:54:57.890 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:57.886+0000 7f974a38f700 1 -- 172.21.15.42:0/3892361025 shutdown_connections 2019-12-05T05:54:57.890 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:57.886+0000 7f974a38f700 1 -- 172.21.15.42:0/3892361025 wait complete. 2019-12-05T05:54:57.890 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:54:57.886+0000 7f974a38f700 1 -- 172.21.15.42:0/3892361025 >> 172.21.15.42:0/3892361025 conn(0x7f974408fde0 msgr2=0x7f9744059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:54:57.898 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T05:54:59.899 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:55:00.018 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.002+0000 7f64939f3700 1 Processor -- start 2019-12-05T05:55:00.018 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.002+0000 7f64939f3700 1 -- start start 2019-12-05T05:55:00.019 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.002+0000 7f64939f3700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f648c0a7740 0x7f648c08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:55:00.019 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.006+0000 7f64939f3700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f648c07e720 0x7f648c07eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:55:00.019 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.006+0000 7f64939f3700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f648c09f2e0 0x7f648c09f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:55:00.019 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.006+0000 7f64939f3700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f648c0a0b00 con 0x7f648c0a7740 2019-12-05T05:55:00.019 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.006+0000 7f64939f3700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f648c08e5b0 con 0x7f648c09f2e0 2019-12-05T05:55:00.020 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.006+0000 7f64939f3700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f648c08e850 con 0x7f648c07e720 2019-12-05T05:55:00.020 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.006+0000 7f6492791700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f648c09f2e0 0x7f648c09f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:55:00.020 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.006+0000 7f6492791700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f648c09f2e0 0x7f648c09f710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:43884/0 (socket says 172.21.15.42:43884) 2019-12-05T05:55:00.020 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.006+0000 7f6492791700 1 -- 172.21.15.42:0/4209059398 learned_addr learned my addr 172.21.15.42:0/4209059398 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:55:00.020 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.006+0000 7f6491f90700 1 --2- 172.21.15.42:0/4209059398 >> v2:172.21.15.40:3300/0 conn(0x7f648c0a7740 0x7f648c08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:55:00.021 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.006+0000 7f6492791700 1 -- 172.21.15.42:0/4209059398 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f648c07e720 msgr2=0x7f648c07eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:55:00.021 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.006+0000 7f649178f700 1 --2- 172.21.15.42:0/4209059398 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f648c07e720 0x7f648c07eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:55:00.021 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.006+0000 7f6492791700 1 --2- 172.21.15.42:0/4209059398 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f648c07e720 0x7f648c07eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:55:00.021 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.006+0000 7f6492791700 1 -- 172.21.15.42:0/4209059398 >> v2:172.21.15.40:3300/0 conn(0x7f648c0a7740 msgr2=0x7f648c08df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:55:00.021 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.006+0000 7f6492791700 1 --2- 172.21.15.42:0/4209059398 >> v2:172.21.15.40:3300/0 conn(0x7f648c0a7740 0x7f648c08df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:55:00.022 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.006+0000 7f6492791700 1 -- 172.21.15.42:0/4209059398 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f648c07f0a0 con 0x7f648c09f2e0 2019-12-05T05:55:00.022 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.006+0000 7f6491f90700 1 --2- 172.21.15.42:0/4209059398 >> v2:172.21.15.40:3300/0 conn(0x7f648c0a7740 0x7f648c08df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T05:55:00.022 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.006+0000 7f649178f700 1 --2- 172.21.15.42:0/4209059398 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f648c07e720 0x7f648c07eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).send_auth_request state changed! 2019-12-05T05:55:00.022 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.006+0000 7f6492791700 1 --2- 172.21.15.42:0/4209059398 >> v2:172.21.15.42:3300/0 conn(0x7f648c09f2e0 0x7f648c09f710 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f648c08e770 tx=0x7f6484010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:00.022 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.006+0000 7f6492791700 1 --2- 172.21.15.42:0/4209059398 >> v2:172.21.15.42:3300/0 conn(0x7f648c09f2e0 0x7f648c09f710 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f648c08e770 tx=0x7f6484010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:00.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.006+0000 7f6492791700 1 --2- 172.21.15.42:0/4209059398 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f648c09f2e0 0x7f648c09f710 secure :-1 s=READY pgs=1679 cs=0 l=1 rx=0x7f648c08e770 tx=0x7f6484010130).ready entity=mon.1 client_cookie=83f5214814e5c9b6 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:55:00.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.006+0000 7f6492791700 1 --2- 172.21.15.42:0/4209059398 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f648c09f2e0 0x7f648c09f710 secure :-1 s=THROTTLE_DONE pgs=1679 cs=0 l=1 rx=0x7f648c08e770 tx=0x7f6484010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:00.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.006+0000 7f6490f8e700 1 -- 172.21.15.42:0/4209059398 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f64840110c0 con 0x7f648c09f2e0 2019-12-05T05:55:00.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.006+0000 7f6492791700 1 --2- 172.21.15.42:0/4209059398 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f648c09f2e0 0x7f648c09f710 secure :-1 s=THROTTLE_DONE pgs=1679 cs=0 l=1 rx=0x7f648c08e770 tx=0x7f6484010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:00.024 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.006+0000 7f6490f8e700 1 -- 172.21.15.42:0/4209059398 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6484010af0 con 0x7f648c09f2e0 2019-12-05T05:55:00.024 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.006+0000 7f6492791700 1 --2- 172.21.15.42:0/4209059398 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f648c09f2e0 0x7f648c09f710 secure :-1 s=THROTTLE_DONE pgs=1679 cs=0 l=1 rx=0x7f648c08e770 tx=0x7f6484010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:00.024 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.006+0000 7f64939f3700 1 -- 172.21.15.42:0/4209059398 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f648c09f2e0 msgr2=0x7f648c09f710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:55:00.025 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.006+0000 7f6490f8e700 1 -- 172.21.15.42:0/4209059398 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f64840116a0 con 0x7f648c09f2e0 2019-12-05T05:55:00.028 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.006+0000 7f64939f3700 1 --2- 172.21.15.42:0/4209059398 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f648c09f2e0 0x7f648c09f710 secure :-1 s=READY pgs=1679 cs=0 l=1 rx=0x7f648c08e770 tx=0x7f6484010130).stop 2019-12-05T05:55:00.028 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.006+0000 7f64939f3700 1 -- 172.21.15.42:0/4209059398 shutdown_connections 2019-12-05T05:55:00.029 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.006+0000 7f64939f3700 1 --2- 172.21.15.42:0/4209059398 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f648c09f2e0 0x7f648c09f710 unknown :-1 s=CLOSED pgs=1679 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:55:00.029 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.006+0000 7f64939f3700 1 --2- 172.21.15.42:0/4209059398 >> v2:172.21.15.40:3300/0 conn(0x7f648c0a7740 0x7f648c08df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:55:00.029 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.006+0000 7f64939f3700 1 --2- 172.21.15.42:0/4209059398 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f648c07e720 0x7f648c07eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:55:00.029 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.006+0000 7f64939f3700 1 -- 172.21.15.42:0/4209059398 shutdown_connections 2019-12-05T05:55:00.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.006+0000 7f64939f3700 1 -- 172.21.15.42:0/4209059398 wait complete. 2019-12-05T05:55:00.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.006+0000 7f64939f3700 1 -- 172.21.15.42:0/4209059398 >> 172.21.15.42:0/4209059398 conn(0x7f648c0aabe0 msgr2=0x7f648c0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:55:00.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.010+0000 7f64939f3700 1 Processor -- start 2019-12-05T05:55:00.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.010+0000 7f64939f3700 1 -- start start 2019-12-05T05:55:00.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.010+0000 7f64939f3700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f648c07cb90 0x7f648c07b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:55:00.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.010+0000 7f64939f3700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f648c07b800 0x7f648c12cf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:55:00.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.010+0000 7f64939f3700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f648c0a6440 con 0x7f648c12d460 2019-12-05T05:55:00.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.010+0000 7f64939f3700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f648c1429d0 con 0x7f648c07b800 2019-12-05T05:55:00.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.010+0000 7f64939f3700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f648c142c70 con 0x7f648c07cb90 2019-12-05T05:55:00.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.010+0000 7f6491f90700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f648c07cb90 0x7f648c07b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:55:00.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.010+0000 7f649178f700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f648c07b800 0x7f648c12cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:55:00.032 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.010+0000 7f6491f90700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f648c07cb90 0x7f648c07b2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:44354/0 (socket says 172.21.15.42:44354) 2019-12-05T05:55:00.032 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.010+0000 7f6491f90700 1 -- 172.21.15.42:0/1114501100 learned_addr learned my addr 172.21.15.42:0/1114501100 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:55:00.032 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.010+0000 7f64827fc700 1 -- 172.21.15.42:0/1114501100 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2725799604 0 0) 0x7f648c0a6440 con 0x7f648c12d460 2019-12-05T05:55:00.032 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.010+0000 7f6491f90700 1 -- 172.21.15.42:0/1114501100 >> v2:172.21.15.40:3300/0 conn(0x7f648c07b800 msgr2=0x7f648c12cf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:55:00.032 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.010+0000 7f6491f90700 1 --2- 172.21.15.42:0/1114501100 >> v2:172.21.15.40:3300/0 conn(0x7f648c07b800 0x7f648c12cf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:55:00.033 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.010+0000 7f6491f90700 1 -- 172.21.15.42:0/1114501100 >> v1:172.21.15.42:6789/0 conn(0x7f648c12d460 legacy=0x7f648c1303c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:55:00.033 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.010+0000 7f6491f90700 1 -- 172.21.15.42:0/1114501100 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f648c11d380 con 0x7f648c07cb90 2019-12-05T05:55:00.033 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.010+0000 7f6491f90700 1 --2- 172.21.15.42:0/1114501100 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f648c07cb90 0x7f648c07b2b0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f64880077a0 tx=0x7f6488010a00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:00.033 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.010+0000 7f6491f90700 1 --2- 172.21.15.42:0/1114501100 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f648c07cb90 0x7f648c07b2b0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f64880077a0 tx=0x7f6488010a00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:00.033 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.010+0000 7f6491f90700 1 --2- 172.21.15.42:0/1114501100 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f648c07cb90 0x7f648c07b2b0 secure :-1 s=READY pgs=375 cs=0 l=1 rx=0x7f64880077a0 tx=0x7f6488010a00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:55:00.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.010+0000 7f6491f90700 1 --2- 172.21.15.42:0/1114501100 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f648c07cb90 0x7f648c07b2b0 secure :-1 s=THROTTLE_DONE pgs=375 cs=0 l=1 rx=0x7f64880077a0 tx=0x7f6488010a00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:00.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.010+0000 7f64827fc700 1 -- 172.21.15.42:0/1114501100 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6488009150 con 0x7f648c07cb90 2019-12-05T05:55:00.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.010+0000 7f64939f3700 1 -- 172.21.15.42:0/1114501100 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f648c12d890 con 0x7f648c07cb90 2019-12-05T05:55:00.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.010+0000 7f6491f90700 1 --2- 172.21.15.42:0/1114501100 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f648c07cb90 0x7f648c07b2b0 secure :-1 s=THROTTLE_DONE pgs=375 cs=0 l=1 rx=0x7f64880077a0 tx=0x7f6488010a00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:00.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.010+0000 7f64827fc700 1 -- 172.21.15.42:0/1114501100 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6488007b30 con 0x7f648c07cb90 2019-12-05T05:55:00.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.010+0000 7f6491f90700 1 --2- 172.21.15.42:0/1114501100 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f648c07cb90 0x7f648c07b2b0 secure :-1 s=THROTTLE_DONE pgs=375 cs=0 l=1 rx=0x7f64880077a0 tx=0x7f6488010a00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:00.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.010+0000 7f64827fc700 1 -- 172.21.15.42:0/1114501100 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6488009730 con 0x7f648c07cb90 2019-12-05T05:55:00.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.010+0000 7f64939f3700 1 -- 172.21.15.42:0/1114501100 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f648c142530 con 0x7f648c07cb90 2019-12-05T05:55:00.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.014+0000 7f6491f90700 1 --2- 172.21.15.42:0/1114501100 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f648c07cb90 0x7f648c07b2b0 secure :-1 s=THROTTLE_DONE pgs=375 cs=0 l=1 rx=0x7f64880077a0 tx=0x7f6488010a00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:00.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.014+0000 7f64827fc700 1 -- 172.21.15.42:0/1114501100 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f6488007170 con 0x7f648c07cb90 2019-12-05T05:55:00.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.014+0000 7f6491f90700 1 --2- 172.21.15.42:0/1114501100 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f648c07cb90 0x7f648c07b2b0 secure :-1 s=THROTTLE_DONE pgs=375 cs=0 l=1 rx=0x7f64880077a0 tx=0x7f6488010a00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:00.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.014+0000 7f64827fc700 1 --2- 172.21.15.42:0/1114501100 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f647403ac10 0x7f647403d120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:55:00.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.014+0000 7f64827fc700 1 -- 172.21.15.42:0/1114501100 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 9041+0+0 (secure 0 0 0) 0x7f648804abd0 con 0x7f648c07cb90 2019-12-05T05:55:00.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.014+0000 7f649178f700 1 --2- 172.21.15.42:0/1114501100 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f647403ac10 0x7f647403d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:55:00.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.014+0000 7f649178f700 1 --2- 172.21.15.42:0/1114501100 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f647403ac10 0x7f647403d120 crc :-1 s=READY pgs=715 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:55:00.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.018+0000 7f64939f3700 1 -- 172.21.15.42:0/1114501100 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f648c05d110 con 0x7f648c07cb90 2019-12-05T05:55:00.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.022+0000 7f6491f90700 1 --2- 172.21.15.42:0/1114501100 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f648c07cb90 0x7f648c07b2b0 secure :-1 s=THROTTLE_DONE pgs=375 cs=0 l=1 rx=0x7f64880077a0 tx=0x7f6488010a00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:00.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.022+0000 7f64827fc700 1 -- 172.21.15.42:0/1114501100 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f6488031e90 con 0x7f648c07cb90 2019-12-05T05:55:00.201 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.198+0000 7f64939f3700 1 -- 172.21.15.42:0/1114501100 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f648c05b8c0 con 0x7f648c07cb90 2019-12-05T05:55:00.202 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.198+0000 7f6491f90700 1 --2- 172.21.15.42:0/1114501100 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f648c07cb90 0x7f648c07b2b0 secure :-1 s=THROTTLE_DONE pgs=375 cs=0 l=1 rx=0x7f64880077a0 tx=0x7f6488010a00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:00.202 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.198+0000 7f64827fc700 1 -- 172.21.15.42:0/1114501100 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 204 v204) v1 ==== 95+0+1774 (secure 0 0 0) 0x7f648802f1c0 con 0x7f648c07cb90 2019-12-05T05:55:00.204 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:55:00.204 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":204,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":204,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:54:59.209404+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8614},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8614":{"gid":8614,"name":"a","rank":0,"incarnation":204,"state":"up:replay","state_seq":1,"addr":"172.21.15.42:6835/2511123581","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2511123581},{"type":"v1","addr":"172.21.15.42:6835","nonce":2511123581}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:55:00.204 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 204 2019-12-05T05:55:00.206 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.202+0000 7f64939f3700 1 -- 172.21.15.42:0/1114501100 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f647403ac10 msgr2=0x7f647403d120 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:55:00.206 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.202+0000 7f64939f3700 1 --2- 172.21.15.42:0/1114501100 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f647403ac10 0x7f647403d120 crc :-1 s=READY pgs=715 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:55:00.207 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.202+0000 7f64939f3700 1 -- 172.21.15.42:0/1114501100 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f648c07cb90 msgr2=0x7f648c07b2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:55:00.207 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.202+0000 7f64939f3700 1 --2- 172.21.15.42:0/1114501100 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f648c07cb90 0x7f648c07b2b0 secure :-1 s=READY pgs=375 cs=0 l=1 rx=0x7f64880077a0 tx=0x7f6488010a00).stop 2019-12-05T05:55:00.207 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.202+0000 7f64939f3700 1 -- 172.21.15.42:0/1114501100 shutdown_connections 2019-12-05T05:55:00.207 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.202+0000 7f64939f3700 1 --2- 172.21.15.42:0/1114501100 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f647403ac10 0x7f647403d120 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:55:00.207 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.202+0000 7f64939f3700 1 --2- 172.21.15.42:0/1114501100 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f648c07cb90 0x7f648c07b2b0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:55:00.207 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.202+0000 7f64939f3700 1 --2- 172.21.15.42:0/1114501100 >> v2:172.21.15.40:3300/0 conn(0x7f648c07b800 0x7f648c12cf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:55:00.208 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.202+0000 7f64939f3700 1 -- 172.21.15.42:0/1114501100 shutdown_connections 2019-12-05T05:55:00.208 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.202+0000 7f64939f3700 1 -- 172.21.15.42:0/1114501100 wait complete. 2019-12-05T05:55:00.208 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.202+0000 7f64939f3700 1 -- 172.21.15.42:0/1114501100 >> 172.21.15.42:0/1114501100 conn(0x7f648c09f240 msgr2=0x7f648c059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:55:00.218 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b down, removed from mdsmap 2019-12-05T05:55:00.218 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2019-12-05T05:55:00.218 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:55:00.308 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.298+0000 7fc1078f3700 1 Processor -- start 2019-12-05T05:55:00.308 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.298+0000 7fc1078f3700 1 -- start start 2019-12-05T05:55:00.308 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.298+0000 7fc1078f3700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc1000a7740 0x7fc10008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:55:00.309 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.298+0000 7fc1078f3700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc10007e720 0x7fc10007eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:55:00.309 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.298+0000 7fc1078f3700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc10009f2e0 0x7fc10009f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:55:00.309 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.298+0000 7fc1078f3700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1000a0b00 con 0x7fc1000a7740 2019-12-05T05:55:00.309 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.298+0000 7fc1078f3700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc10008e5b0 con 0x7fc10009f2e0 2019-12-05T05:55:00.309 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.298+0000 7fc1078f3700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc10008e850 con 0x7fc10007e720 2019-12-05T05:55:00.309 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.298+0000 7fc106691700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc10009f2e0 0x7fc10009f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:55:00.310 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.298+0000 7fc106691700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc10009f2e0 0x7fc10009f710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:43898/0 (socket says 172.21.15.42:43898) 2019-12-05T05:55:00.310 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.298+0000 7fc106691700 1 -- 172.21.15.42:0/2090873454 learned_addr learned my addr 172.21.15.42:0/2090873454 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:55:00.310 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.298+0000 7fc105e90700 1 --2- 172.21.15.42:0/2090873454 >> v2:172.21.15.40:3300/0 conn(0x7fc1000a7740 0x7fc10008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:55:00.311 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.298+0000 7fc10568f700 1 --2- 172.21.15.42:0/2090873454 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc10007e720 0x7fc10007eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:55:00.311 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.298+0000 7fc106691700 1 -- 172.21.15.42:0/2090873454 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc10007e720 msgr2=0x7fc10007eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:55:00.311 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.298+0000 7fc106691700 1 --2- 172.21.15.42:0/2090873454 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc10007e720 0x7fc10007eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:55:00.311 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.298+0000 7fc106691700 1 -- 172.21.15.42:0/2090873454 >> v2:172.21.15.40:3300/0 conn(0x7fc1000a7740 msgr2=0x7fc10008df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:55:00.311 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.298+0000 7fc106691700 1 --2- 172.21.15.42:0/2090873454 >> v2:172.21.15.40:3300/0 conn(0x7fc1000a7740 0x7fc10008df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:55:00.311 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.298+0000 7fc106691700 1 -- 172.21.15.42:0/2090873454 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc10007f0a0 con 0x7fc10009f2e0 2019-12-05T05:55:00.311 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.298+0000 7fc106691700 1 --2- 172.21.15.42:0/2090873454 >> v2:172.21.15.42:3300/0 conn(0x7fc10009f2e0 0x7fc10009f710 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fc10008e770 tx=0x7fc0f000fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:00.312 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.298+0000 7fc106691700 1 --2- 172.21.15.42:0/2090873454 >> v2:172.21.15.42:3300/0 conn(0x7fc10009f2e0 0x7fc10009f710 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fc10008e770 tx=0x7fc0f000fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:00.312 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.298+0000 7fc106691700 1 --2- 172.21.15.42:0/2090873454 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc10009f2e0 0x7fc10009f710 secure :-1 s=READY pgs=1681 cs=0 l=1 rx=0x7fc10008e770 tx=0x7fc0f000fdb0).ready entity=mon.1 client_cookie=2abf1bfda6917eec server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:55:00.312 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.298+0000 7fc106691700 1 --2- 172.21.15.42:0/2090873454 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc10009f2e0 0x7fc10009f710 secure :-1 s=THROTTLE_DONE pgs=1681 cs=0 l=1 rx=0x7fc10008e770 tx=0x7fc0f000fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:00.312 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.298+0000 7fc104e8e700 1 -- 172.21.15.42:0/2090873454 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc0f0008c90 con 0x7fc10009f2e0 2019-12-05T05:55:00.312 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.298+0000 7fc106691700 1 --2- 172.21.15.42:0/2090873454 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc10009f2e0 0x7fc10009f710 secure :-1 s=THROTTLE_DONE pgs=1681 cs=0 l=1 rx=0x7fc10008e770 tx=0x7fc0f000fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:00.312 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.298+0000 7fc104e8e700 1 -- 172.21.15.42:0/2090873454 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0f00088b0 con 0x7fc10009f2e0 2019-12-05T05:55:00.313 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.298+0000 7fc106691700 1 --2- 172.21.15.42:0/2090873454 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc10009f2e0 0x7fc10009f710 secure :-1 s=THROTTLE_DONE pgs=1681 cs=0 l=1 rx=0x7fc10008e770 tx=0x7fc0f000fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:00.313 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.298+0000 7fc104e8e700 1 -- 172.21.15.42:0/2090873454 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc0f0011b50 con 0x7fc10009f2e0 2019-12-05T05:55:00.313 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.298+0000 7fc1078f3700 1 -- 172.21.15.42:0/2090873454 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc10009f2e0 msgr2=0x7fc10009f710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:55:00.313 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.298+0000 7fc1078f3700 1 --2- 172.21.15.42:0/2090873454 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc10009f2e0 0x7fc10009f710 secure :-1 s=READY pgs=1681 cs=0 l=1 rx=0x7fc10008e770 tx=0x7fc0f000fdb0).stop 2019-12-05T05:55:00.313 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.302+0000 7fc1078f3700 1 -- 172.21.15.42:0/2090873454 shutdown_connections 2019-12-05T05:55:00.314 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.302+0000 7fc1078f3700 1 --2- 172.21.15.42:0/2090873454 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc10009f2e0 0x7fc10009f710 unknown :-1 s=CLOSED pgs=1681 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:55:00.314 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.302+0000 7fc1078f3700 1 --2- 172.21.15.42:0/2090873454 >> v2:172.21.15.40:3300/0 conn(0x7fc1000a7740 0x7fc10008df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:55:00.314 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.302+0000 7fc1078f3700 1 --2- 172.21.15.42:0/2090873454 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc10007e720 0x7fc10007eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:55:00.314 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.302+0000 7fc1078f3700 1 -- 172.21.15.42:0/2090873454 shutdown_connections 2019-12-05T05:55:00.318 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.302+0000 7fc1078f3700 1 -- 172.21.15.42:0/2090873454 wait complete. 2019-12-05T05:55:00.318 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.302+0000 7fc1078f3700 1 -- 172.21.15.42:0/2090873454 >> 172.21.15.42:0/2090873454 conn(0x7fc1000aabe0 msgr2=0x7fc1000a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:55:00.319 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.302+0000 7fc1078f3700 1 Processor -- start 2019-12-05T05:55:00.319 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.302+0000 7fc1078f3700 1 -- start start 2019-12-05T05:55:00.319 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.302+0000 7fc1078f3700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc100122980 0x7fc10011fa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:55:00.319 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.302+0000 7fc1078f3700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc100121850 0x7fc100117f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:55:00.319 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.302+0000 7fc1078f3700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc1001184c0 0x7fc1001188f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:55:00.319 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.302+0000 7fc1078f3700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc100122770 con 0x7fc1001184c0 2019-12-05T05:55:00.320 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.302+0000 7fc1078f3700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1000a6e00 con 0x7fc100121850 2019-12-05T05:55:00.320 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.302+0000 7fc1078f3700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc100121e40 con 0x7fc100122980 2019-12-05T05:55:00.320 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.302+0000 7fc10568f700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc100121850 0x7fc100117f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:55:00.320 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.302+0000 7fc10568f700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc100121850 0x7fc100117f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:43900/0 (socket says 172.21.15.42:43900) 2019-12-05T05:55:00.320 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.302+0000 7fc10568f700 1 -- 172.21.15.42:0/692904427 learned_addr learned my addr 172.21.15.42:0/692904427 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:55:00.320 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.302+0000 7fc105e90700 1 --2- 172.21.15.42:0/692904427 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc100122980 0x7fc10011fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:55:00.321 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.302+0000 7fc106691700 1 --2- 172.21.15.42:0/692904427 >> v2:172.21.15.40:3300/0 conn(0x7fc1001184c0 0x7fc1001188f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:55:00.321 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.302+0000 7fc10568f700 1 -- 172.21.15.42:0/692904427 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc100122980 msgr2=0x7fc10011fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:55:00.321 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.302+0000 7fc10568f700 1 --2- 172.21.15.42:0/692904427 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc100122980 0x7fc10011fa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:55:00.322 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.302+0000 7fc10568f700 1 -- 172.21.15.42:0/692904427 >> v2:172.21.15.40:3300/0 conn(0x7fc1001184c0 msgr2=0x7fc1001188f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:55:00.322 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.302+0000 7fc10568f700 1 --2- 172.21.15.42:0/692904427 >> v2:172.21.15.40:3300/0 conn(0x7fc1001184c0 0x7fc1001188f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:55:00.322 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.302+0000 7fc10568f700 1 -- 172.21.15.42:0/692904427 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1000a70a0 con 0x7fc100121850 2019-12-05T05:55:00.322 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.302+0000 7fc105e90700 1 --2- 172.21.15.42:0/692904427 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc100122980 0x7fc10011fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T05:55:00.322 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.302+0000 7fc10568f700 1 --2- 172.21.15.42:0/692904427 >> v2:172.21.15.42:3300/0 conn(0x7fc100121850 0x7fc100117f70 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fc0f4002f30 tx=0x7fc0f4010ba0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:00.322 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.302+0000 7fc10568f700 1 --2- 172.21.15.42:0/692904427 >> v2:172.21.15.42:3300/0 conn(0x7fc100121850 0x7fc100117f70 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fc0f4002f30 tx=0x7fc0f4010ba0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:00.322 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.302+0000 7fc10568f700 1 --2- 172.21.15.42:0/692904427 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc100121850 0x7fc100117f70 secure :-1 s=READY pgs=1682 cs=0 l=1 rx=0x7fc0f4002f30 tx=0x7fc0f4010ba0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:55:00.323 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.302+0000 7fc10568f700 1 --2- 172.21.15.42:0/692904427 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc100121850 0x7fc100117f70 secure :-1 s=THROTTLE_DONE pgs=1682 cs=0 l=1 rx=0x7fc0f4002f30 tx=0x7fc0f4010ba0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:00.323 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.302+0000 7fc10568f700 1 --2- 172.21.15.42:0/692904427 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc100121850 0x7fc100117f70 secure :-1 s=THROTTLE_DONE pgs=1682 cs=0 l=1 rx=0x7fc0f4002f30 tx=0x7fc0f4010ba0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:00.323 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.302+0000 7fc0fe7fc700 1 -- 172.21.15.42:0/692904427 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc0f40091a0 con 0x7fc100121850 2019-12-05T05:55:00.323 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.302+0000 7fc10568f700 1 --2- 172.21.15.42:0/692904427 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc100121850 0x7fc100117f70 secure :-1 s=THROTTLE_DONE pgs=1682 cs=0 l=1 rx=0x7fc0f4002f30 tx=0x7fc0f4010ba0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:00.323 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.302+0000 7fc0fe7fc700 1 -- 172.21.15.42:0/692904427 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0f4009b70 con 0x7fc100121850 2019-12-05T05:55:00.323 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.302+0000 7fc1078f3700 1 -- 172.21.15.42:0/692904427 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc10011ffa0 con 0x7fc100121850 2019-12-05T05:55:00.323 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.302+0000 7fc0fe7fc700 1 -- 172.21.15.42:0/692904427 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc0f4009780 con 0x7fc100121850 2019-12-05T05:55:00.324 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.302+0000 7fc1078f3700 1 -- 172.21.15.42:0/692904427 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc100115ec0 con 0x7fc100121850 2019-12-05T05:55:00.324 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.302+0000 7fc10568f700 1 --2- 172.21.15.42:0/692904427 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc100121850 0x7fc100117f70 secure :-1 s=THROTTLE_DONE pgs=1682 cs=0 l=1 rx=0x7fc0f4002f30 tx=0x7fc0f4010ba0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:00.324 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.306+0000 7fc0fe7fc700 1 -- 172.21.15.42:0/692904427 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fc0f4007170 con 0x7fc100121850 2019-12-05T05:55:00.324 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.306+0000 7fc10568f700 1 --2- 172.21.15.42:0/692904427 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc100121850 0x7fc100117f70 secure :-1 s=THROTTLE_DONE pgs=1682 cs=0 l=1 rx=0x7fc0f4002f30 tx=0x7fc0f4010ba0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:00.324 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.306+0000 7fc0fe7fc700 1 --2- 172.21.15.42:0/692904427 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc0e003add0 0x7fc0e003d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:55:00.324 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.306+0000 7fc0fe7fc700 1 -- 172.21.15.42:0/692904427 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 9041+0+0 (secure 0 0 0) 0x7fc0f404afb0 con 0x7fc100121850 2019-12-05T05:55:00.324 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.306+0000 7fc1078f3700 1 -- 172.21.15.42:0/692904427 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc10005d110 con 0x7fc100121850 2019-12-05T05:55:00.325 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.306+0000 7fc105e90700 1 --2- 172.21.15.42:0/692904427 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc0e003add0 0x7fc0e003d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:55:00.325 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.306+0000 7fc105e90700 1 --2- 172.21.15.42:0/692904427 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc0e003add0 0x7fc0e003d2e0 crc :-1 s=READY pgs=716 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:55:00.326 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.310+0000 7fc10568f700 1 --2- 172.21.15.42:0/692904427 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc100121850 0x7fc100117f70 secure :-1 s=THROTTLE_DONE pgs=1682 cs=0 l=1 rx=0x7fc0f4002f30 tx=0x7fc0f4010ba0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:00.326 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.310+0000 7fc0fe7fc700 1 -- 172.21.15.42:0/692904427 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fc0f4005540 con 0x7fc100121850 2019-12-05T05:55:00.470 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.466+0000 7fc1078f3700 1 -- 172.21.15.42:0/692904427 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc10005b8c0 con 0x7fc100121850 2019-12-05T05:55:00.470 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.466+0000 7fc10568f700 1 --2- 172.21.15.42:0/692904427 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc100121850 0x7fc100117f70 secure :-1 s=THROTTLE_DONE pgs=1682 cs=0 l=1 rx=0x7fc0f4002f30 tx=0x7fc0f4010ba0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:00.471 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.466+0000 7fc0fe7fc700 1 -- 172.21.15.42:0/692904427 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 204 v204) v1 ==== 95+0+1774 (secure 0 0 0) 0x7fc0f4032030 con 0x7fc100121850 2019-12-05T05:55:00.471 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 204 2019-12-05T05:55:00.472 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:55:00.472 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":204,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":204,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:54:59.209404+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8614},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8614":{"gid":8614,"name":"a","rank":0,"incarnation":204,"state":"up:replay","state_seq":1,"addr":"172.21.15.42:6835/2511123581","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2511123581},{"type":"v1","addr":"172.21.15.42:6835","nonce":2511123581}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:55:00.474 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.470+0000 7fc1078f3700 1 -- 172.21.15.42:0/692904427 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc0e003add0 msgr2=0x7fc0e003d2e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:55:00.474 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.470+0000 7fc1078f3700 1 --2- 172.21.15.42:0/692904427 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc0e003add0 0x7fc0e003d2e0 crc :-1 s=READY pgs=716 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:55:00.475 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.470+0000 7fc1078f3700 1 -- 172.21.15.42:0/692904427 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc100121850 msgr2=0x7fc100117f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:55:00.475 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.470+0000 7fc1078f3700 1 --2- 172.21.15.42:0/692904427 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc100121850 0x7fc100117f70 secure :-1 s=READY pgs=1682 cs=0 l=1 rx=0x7fc0f4002f30 tx=0x7fc0f4010ba0).stop 2019-12-05T05:55:00.475 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.470+0000 7fc1078f3700 1 -- 172.21.15.42:0/692904427 shutdown_connections 2019-12-05T05:55:00.476 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.470+0000 7fc1078f3700 1 --2- 172.21.15.42:0/692904427 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc0e003add0 0x7fc0e003d2e0 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:55:00.476 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.470+0000 7fc1078f3700 1 --2- 172.21.15.42:0/692904427 >> v2:172.21.15.40:3300/0 conn(0x7fc1001184c0 0x7fc1001188f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:55:00.476 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.470+0000 7fc1078f3700 1 --2- 172.21.15.42:0/692904427 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc100122980 0x7fc10011fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:55:00.476 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.470+0000 7fc1078f3700 1 --2- 172.21.15.42:0/692904427 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc100121850 0x7fc100117f70 unknown :-1 s=CLOSED pgs=1682 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:55:00.476 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.470+0000 7fc1078f3700 1 -- 172.21.15.42:0/692904427 shutdown_connections 2019-12-05T05:55:00.476 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.470+0000 7fc1078f3700 1 -- 172.21.15.42:0/692904427 wait complete. 2019-12-05T05:55:00.477 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:00.470+0000 7fc1078f3700 1 -- 172.21.15.42:0/692904427 >> 172.21.15.42:0/692904427 conn(0x7fc10009f3f0 msgr2=0x7fc10011b530 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:55:00.483 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 204, "feature_flags": { "enable_multiple": false, "ever_enabled_multiple": false }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2019-12-05T05:03:35.604738+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 204, "ever_allowed_features": 2, "explicitly_allowed_features": 2, "failed": [], "flags": 18, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_8614": { "addr": "172.21.15.42:6835/2511123581", "addrs": { "addrvec": [ { "addr": "172.21.15.42:6834", "nonce": 2511123581, "type": "v2" }, { "addr": "172.21.15.42:6835", "nonce": 2511123581, "type": "v1" } ] }, "export_targets": [], "features": 4540138292836696063, "flags": 0, "gid": 8614, "incarnation": 204, "name": "a", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 84, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 2, "min_compat_client": "0 (unknown)", "modified": "2019-12-05T05:54:59.209404+0000", "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 8614 } } } ], "standbys": [] } 2019-12-05T05:55:02.484 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:55:02.591 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:02.578+0000 7f0cb888e700 1 Processor -- start 2019-12-05T05:55:02.592 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:02.578+0000 7f0cb888e700 1 -- start start 2019-12-05T05:55:02.592 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:02.578+0000 7f0cb888e700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f0cb40a7740 0x7f0cb408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:55:02.592 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:02.578+0000 7f0cb888e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0cb407e720 0x7f0cb407eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:55:02.600 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:02.578+0000 7f0cb888e700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0cb409f2e0 0x7f0cb409f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:55:02.600 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:02.578+0000 7f0cb888e700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0cb40a0b00 con 0x7f0cb40a7740 2019-12-05T05:55:02.600 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:02.578+0000 7f0cb888e700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0cb408e5b0 con 0x7f0cb407e720 2019-12-05T05:55:02.601 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:02.578+0000 7f0cb888e700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cb408e850 con 0x7f0cb409f2e0 2019-12-05T05:55:02.601 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:02.578+0000 7f0cb259c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0cb407e720 0x7f0cb407eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:55:02.601 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:02.578+0000 7f0cb259c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0cb407e720 0x7f0cb407eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:43916/0 (socket says 172.21.15.42:43916) 2019-12-05T05:55:02.601 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:02.578+0000 7f0cb259c700 1 -- 172.21.15.42:0/1680251906 learned_addr learned my addr 172.21.15.42:0/1680251906 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:55:02.601 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:02.578+0000 7f0cb2d9d700 1 --2- 172.21.15.42:0/1680251906 >> v2:172.21.15.40:3300/0 conn(0x7f0cb40a7740 0x7f0cb408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:55:02.602 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:02.578+0000 7f0cb359e700 1 --2- 172.21.15.42:0/1680251906 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0cb409f2e0 0x7f0cb409f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:55:02.602 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:02.578+0000 7f0cb259c700 1 -- 172.21.15.42:0/1680251906 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0cb409f2e0 msgr2=0x7f0cb409f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:55:02.602 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:02.578+0000 7f0cb259c700 1 --2- 172.21.15.42:0/1680251906 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0cb409f2e0 0x7f0cb409f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:55:02.602 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:02.578+0000 7f0cb259c700 1 -- 172.21.15.42:0/1680251906 >> v2:172.21.15.40:3300/0 conn(0x7f0cb40a7740 msgr2=0x7f0cb408df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:55:02.603 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:02.578+0000 7f0cb259c700 1 --2- 172.21.15.42:0/1680251906 >> v2:172.21.15.40:3300/0 conn(0x7f0cb40a7740 0x7f0cb408df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:55:02.603 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:02.578+0000 7f0cb259c700 1 -- 172.21.15.42:0/1680251906 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cb407f0a0 con 0x7f0cb407e720 2019-12-05T05:55:02.603 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:02.578+0000 7f0cb259c700 1 --2- 172.21.15.42:0/1680251906 >> v2:172.21.15.42:3300/0 conn(0x7f0cb407e720 0x7f0cb407eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f0cb40a0f50 tx=0x7f0ca800fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:02.603 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:02.582+0000 7f0cb259c700 1 --2- 172.21.15.42:0/1680251906 >> v2:172.21.15.42:3300/0 conn(0x7f0cb407e720 0x7f0cb407eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f0cb40a0f50 tx=0x7f0ca800fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:02.604 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:02.582+0000 7f0cb259c700 1 --2- 172.21.15.42:0/1680251906 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0cb407e720 0x7f0cb407eb50 secure :-1 s=READY pgs=1683 cs=0 l=1 rx=0x7f0cb40a0f50 tx=0x7f0ca800fdb0).ready entity=mon.1 client_cookie=fa6896c46765f54b server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:55:02.604 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:02.582+0000 7f0cb259c700 1 --2- 172.21.15.42:0/1680251906 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0cb407e720 0x7f0cb407eb50 secure :-1 s=THROTTLE_DONE pgs=1683 cs=0 l=1 rx=0x7f0cb40a0f50 tx=0x7f0ca800fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:02.604 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:02.582+0000 7f0cb1d9b700 1 -- 172.21.15.42:0/1680251906 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0ca8008c90 con 0x7f0cb407e720 2019-12-05T05:55:02.605 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:02.582+0000 7f0cb259c700 1 --2- 172.21.15.42:0/1680251906 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0cb407e720 0x7f0cb407eb50 secure :-1 s=THROTTLE_DONE pgs=1683 cs=0 l=1 rx=0x7f0cb40a0f50 tx=0x7f0ca800fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:02.606 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:02.582+0000 7f0cb259c700 1 --2- 172.21.15.42:0/1680251906 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0cb407e720 0x7f0cb407eb50 secure :-1 s=THROTTLE_DONE pgs=1683 cs=0 l=1 rx=0x7f0cb40a0f50 tx=0x7f0ca800fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:02.606 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:02.582+0000 7f0cb1d9b700 1 -- 172.21.15.42:0/1680251906 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ca80088b0 con 0x7f0cb407e720 2019-12-05T05:55:02.606 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:02.582+0000 7f0cb1d9b700 1 -- 172.21.15.42:0/1680251906 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0ca8011b50 con 0x7f0cb407e720 2019-12-05T05:55:02.606 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:02.582+0000 7f0cb888e700 1 -- 172.21.15.42:0/1680251906 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0cb407e720 msgr2=0x7f0cb407eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:55:02.607 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:02.582+0000 7f0cb888e700 1 --2- 172.21.15.42:0/1680251906 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0cb407e720 0x7f0cb407eb50 secure :-1 s=READY pgs=1683 cs=0 l=1 rx=0x7f0cb40a0f50 tx=0x7f0ca800fdb0).stop 2019-12-05T05:55:02.607 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:02.582+0000 7f0cb888e700 1 -- 172.21.15.42:0/1680251906 shutdown_connections 2019-12-05T05:55:02.607 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:02.582+0000 7f0cb888e700 1 --2- 172.21.15.42:0/1680251906 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0cb409f2e0 0x7f0cb409f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:55:02.607 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:02.582+0000 7f0cb888e700 1 --2- 172.21.15.42:0/1680251906 >> v2:172.21.15.40:3300/0 conn(0x7f0cb40a7740 0x7f0cb408df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:55:02.608 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:02.582+0000 7f0cb888e700 1 --2- 172.21.15.42:0/1680251906 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0cb407e720 0x7f0cb407eb50 unknown :-1 s=CLOSED pgs=1683 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:55:02.608 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:02.582+0000 7f0cb888e700 1 -- 172.21.15.42:0/1680251906 shutdown_connections 2019-12-05T05:55:02.608 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:02.582+0000 7f0cb888e700 1 -- 172.21.15.42:0/1680251906 wait complete. 2019-12-05T05:55:02.608 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:02.582+0000 7f0cb888e700 1 -- 172.21.15.42:0/1680251906 >> 172.21.15.42:0/1680251906 conn(0x7f0cb40aabe0 msgr2=0x7f0cb40a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:55:02.609 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:02.582+0000 7f0cb888e700 1 Processor -- start 2019-12-05T05:55:02.609 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:02.582+0000 7f0cb888e700 1 -- start start 2019-12-05T05:55:02.609 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:02.582+0000 7f0cb888e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0cb408eeb0 0x7f0cb407bec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:55:02.609 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:02.582+0000 7f0cb888e700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f0cb41197d0 0x7f0cb407adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:55:02.610 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:02.582+0000 7f0cb888e700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0cb411d9a0 0x7f0cb411ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:55:02.611 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:02.582+0000 7f0cb888e700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0cb411e320 con 0x7f0cb41197d0 2019-12-05T05:55:02.611 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:02.582+0000 7f0cb888e700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0cb4119ea0 con 0x7f0cb408eeb0 2019-12-05T05:55:02.611 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:02.582+0000 7f0cb888e700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cb411a140 con 0x7f0cb411d9a0 2019-12-05T05:55:02.612 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:02.582+0000 7f0cb2d9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0cb408eeb0 0x7f0cb407bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:55:02.612 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:02.582+0000 7f0cb2d9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0cb408eeb0 0x7f0cb407bec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:43924/0 (socket says 172.21.15.42:43924) 2019-12-05T05:55:02.612 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:02.582+0000 7f0cb2d9d700 1 -- 172.21.15.42:0/2911471526 learned_addr learned my addr 172.21.15.42:0/2911471526 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:55:02.612 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:02.586+0000 7f0cb359e700 1 --2- 172.21.15.42:0/2911471526 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0cb411d9a0 0x7f0cb411ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:55:02.613 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:02.586+0000 7f0cb259c700 1 --2- 172.21.15.42:0/2911471526 >> v2:172.21.15.40:3300/0 conn(0x7f0cb41197d0 0x7f0cb407adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:55:02.613 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:02.586+0000 7f0cb2d9d700 1 -- 172.21.15.42:0/2911471526 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0cb411d9a0 msgr2=0x7f0cb411ddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:55:02.613 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:02.586+0000 7f0cb2d9d700 1 --2- 172.21.15.42:0/2911471526 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0cb411d9a0 0x7f0cb411ddd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:55:02.613 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:02.586+0000 7f0cb2d9d700 1 -- 172.21.15.42:0/2911471526 >> v2:172.21.15.40:3300/0 conn(0x7f0cb41197d0 msgr2=0x7f0cb407adc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:55:02.613 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:02.586+0000 7f0cb2d9d700 1 --2- 172.21.15.42:0/2911471526 >> v2:172.21.15.40:3300/0 conn(0x7f0cb41197d0 0x7f0cb407adc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:55:02.614 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:02.586+0000 7f0cb2d9d700 1 -- 172.21.15.42:0/2911471526 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cb407b310 con 0x7f0cb408eeb0 2019-12-05T05:55:02.614 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:02.586+0000 7f0cb2d9d700 1 --2- 172.21.15.42:0/2911471526 >> v2:172.21.15.42:3300/0 conn(0x7f0cb408eeb0 0x7f0cb407bec0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f0cb411a060 tx=0x7f0ca4010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:02.614 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:02.586+0000 7f0cb2d9d700 1 --2- 172.21.15.42:0/2911471526 >> v2:172.21.15.42:3300/0 conn(0x7f0cb408eeb0 0x7f0cb407bec0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f0cb411a060 tx=0x7f0ca4010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:02.614 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:02.586+0000 7f0cb2d9d700 1 --2- 172.21.15.42:0/2911471526 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0cb408eeb0 0x7f0cb407bec0 secure :-1 s=READY pgs=1684 cs=0 l=1 rx=0x7f0cb411a060 tx=0x7f0ca4010b60).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:55:02.615 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:02.586+0000 7f0cb2d9d700 1 --2- 172.21.15.42:0/2911471526 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0cb408eeb0 0x7f0cb407bec0 secure :-1 s=THROTTLE_DONE pgs=1684 cs=0 l=1 rx=0x7f0cb411a060 tx=0x7f0ca4010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:02.615 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:02.586+0000 7f0cb2d9d700 1 --2- 172.21.15.42:0/2911471526 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0cb408eeb0 0x7f0cb407bec0 secure :-1 s=THROTTLE_DONE pgs=1684 cs=0 l=1 rx=0x7f0cb411a060 tx=0x7f0ca4010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:02.615 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:02.586+0000 7f0ca37fe700 1 -- 172.21.15.42:0/2911471526 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0ca4005930 con 0x7f0cb408eeb0 2019-12-05T05:55:02.615 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:02.586+0000 7f0cb2d9d700 1 --2- 172.21.15.42:0/2911471526 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0cb408eeb0 0x7f0cb407bec0 secure :-1 s=THROTTLE_DONE pgs=1684 cs=0 l=1 rx=0x7f0cb411a060 tx=0x7f0ca4010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:02.616 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:02.586+0000 7f0ca37fe700 1 -- 172.21.15.42:0/2911471526 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ca4007160 con 0x7f0cb408eeb0 2019-12-05T05:55:02.616 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:02.586+0000 7f0cb888e700 1 -- 172.21.15.42:0/2911471526 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cb4137b80 con 0x7f0cb408eeb0 2019-12-05T05:55:02.616 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:02.586+0000 7f0ca37fe700 1 -- 172.21.15.42:0/2911471526 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0ca40121a0 con 0x7f0cb408eeb0 2019-12-05T05:55:02.616 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:02.586+0000 7f0cb888e700 1 -- 172.21.15.42:0/2911471526 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cb4138130 con 0x7f0cb408eeb0 2019-12-05T05:55:02.616 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:02.586+0000 7f0cb2d9d700 1 --2- 172.21.15.42:0/2911471526 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0cb408eeb0 0x7f0cb407bec0 secure :-1 s=THROTTLE_DONE pgs=1684 cs=0 l=1 rx=0x7f0cb411a060 tx=0x7f0ca4010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:02.616 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:02.586+0000 7f0ca37fe700 1 -- 172.21.15.42:0/2911471526 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f0ca4012470 con 0x7f0cb408eeb0 2019-12-05T05:55:02.617 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:02.586+0000 7f0cb2d9d700 1 --2- 172.21.15.42:0/2911471526 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0cb408eeb0 0x7f0cb407bec0 secure :-1 s=THROTTLE_DONE pgs=1684 cs=0 l=1 rx=0x7f0cb411a060 tx=0x7f0ca4010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:02.617 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:02.586+0000 7f0ca37fe700 1 --2- 172.21.15.42:0/2911471526 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0c9803ae60 0x7f0c9803d370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:55:02.617 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:02.586+0000 7f0ca37fe700 1 -- 172.21.15.42:0/2911471526 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 9041+0+0 (secure 0 0 0) 0x7f0ca404aee0 con 0x7f0cb408eeb0 2019-12-05T05:55:02.617 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:02.586+0000 7f0cb259c700 1 --2- 172.21.15.42:0/2911471526 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0c9803ae60 0x7f0c9803d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:55:02.617 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:02.590+0000 7f0cb259c700 1 --2- 172.21.15.42:0/2911471526 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0c9803ae60 0x7f0c9803d370 crc :-1 s=READY pgs=717 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:55:02.618 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:02.594+0000 7f0cb888e700 1 -- 172.21.15.42:0/2911471526 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cb405d110 con 0x7f0cb408eeb0 2019-12-05T05:55:02.618 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:02.602+0000 7f0cb2d9d700 1 --2- 172.21.15.42:0/2911471526 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0cb408eeb0 0x7f0cb407bec0 secure :-1 s=THROTTLE_DONE pgs=1684 cs=0 l=1 rx=0x7f0cb411a060 tx=0x7f0ca4010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:02.618 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:02.602+0000 7f0ca37fe700 1 -- 172.21.15.42:0/2911471526 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f0ca4031f60 con 0x7f0cb408eeb0 2019-12-05T05:55:02.775 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:02.770+0000 7f0cb888e700 1 -- 172.21.15.42:0/2911471526 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0cb405b8c0 con 0x7f0cb408eeb0 2019-12-05T05:55:02.776 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:02.774+0000 7f0cb2d9d700 1 --2- 172.21.15.42:0/2911471526 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0cb408eeb0 0x7f0cb407bec0 secure :-1 s=THROTTLE_DONE pgs=1684 cs=0 l=1 rx=0x7f0cb411a060 tx=0x7f0ca4010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:02.776 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:02.774+0000 7f0ca37fe700 1 -- 172.21.15.42:0/2911471526 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 205 v205) v1 ==== 95+0+1778 (secure 0 0 0) 0x7f0ca40321c0 con 0x7f0cb408eeb0 2019-12-05T05:55:02.776 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 205 2019-12-05T05:55:02.776 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:55:02.776 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":205,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":205,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:55:02.245421+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8614},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8614":{"gid":8614,"name":"a","rank":0,"incarnation":204,"state":"up:reconnect","state_seq":30,"addr":"172.21.15.42:6835/2511123581","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2511123581},{"type":"v1","addr":"172.21.15.42:6835","nonce":2511123581}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:55:02.778 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:02.774+0000 7f0cb888e700 1 -- 172.21.15.42:0/2911471526 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0c9803ae60 msgr2=0x7f0c9803d370 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:55:02.779 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:02.774+0000 7f0cb888e700 1 --2- 172.21.15.42:0/2911471526 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0c9803ae60 0x7f0c9803d370 crc :-1 s=READY pgs=717 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:55:02.779 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:02.774+0000 7f0cb888e700 1 -- 172.21.15.42:0/2911471526 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0cb408eeb0 msgr2=0x7f0cb407bec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:55:02.779 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:02.774+0000 7f0cb888e700 1 --2- 172.21.15.42:0/2911471526 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0cb408eeb0 0x7f0cb407bec0 secure :-1 s=READY pgs=1684 cs=0 l=1 rx=0x7f0cb411a060 tx=0x7f0ca4010b60).stop 2019-12-05T05:55:02.779 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:02.774+0000 7f0cb888e700 1 -- 172.21.15.42:0/2911471526 shutdown_connections 2019-12-05T05:55:02.779 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:02.774+0000 7f0cb888e700 1 --2- 172.21.15.42:0/2911471526 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0c9803ae60 0x7f0c9803d370 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:55:02.779 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:02.774+0000 7f0cb888e700 1 --2- 172.21.15.42:0/2911471526 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0cb411d9a0 0x7f0cb411ddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:55:02.780 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:02.774+0000 7f0cb888e700 1 --2- 172.21.15.42:0/2911471526 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0cb408eeb0 0x7f0cb407bec0 unknown :-1 s=CLOSED pgs=1684 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:55:02.780 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:02.774+0000 7f0cb888e700 1 --2- 172.21.15.42:0/2911471526 >> v2:172.21.15.40:3300/0 conn(0x7f0cb41197d0 0x7f0cb407adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:55:02.780 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:02.774+0000 7f0cb888e700 1 -- 172.21.15.42:0/2911471526 shutdown_connections 2019-12-05T05:55:02.780 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:02.774+0000 7f0cb888e700 1 -- 172.21.15.42:0/2911471526 wait complete. 2019-12-05T05:55:02.780 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:02.774+0000 7f0cb888e700 1 -- 172.21.15.42:0/2911471526 >> 172.21.15.42:0/2911471526 conn(0x7f0cb411b1d0 msgr2=0x7f0cb4059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:55:02.788 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2019-12-05T05:55:04.789 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:55:04.893 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:04.882+0000 7f9e66832700 1 Processor -- start 2019-12-05T05:55:04.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:04.882+0000 7f9e66832700 1 -- start start 2019-12-05T05:55:04.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:04.886+0000 7f9e66832700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9e600a7740 0x7f9e6008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:55:04.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:04.886+0000 7f9e66832700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f9e6007e720 0x7f9e6007eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:55:04.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:04.886+0000 7f9e66832700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e600a0710 con 0x7f9e6009f2e0 2019-12-05T05:55:04.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:04.886+0000 7f9e66832700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e600a0b00 con 0x7f9e6007e720 2019-12-05T05:55:04.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:04.886+0000 7f9e66832700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e6008e5b0 con 0x7f9e600a7740 2019-12-05T05:55:04.895 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:04.886+0000 7f9e655d0700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f9e6009f2e0 0x7f9e6009f710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:43972/0 (socket says 172.21.15.42:43972) 2019-12-05T05:55:04.895 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:04.886+0000 7f9e655d0700 1 -- 172.21.15.42:0/2805119653 learned_addr learned my addr 172.21.15.42:0/2805119653 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:55:04.895 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:04.886+0000 7f9e64dcf700 1 --2- 172.21.15.42:0/2805119653 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9e600a7740 0x7f9e6008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:55:04.895 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:04.886+0000 7f9e5ffff700 1 --2- 172.21.15.42:0/2805119653 >> v2:172.21.15.40:3300/0 conn(0x7f9e6007e720 0x7f9e6007eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:55:04.895 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:04.886+0000 7f9e5f7fe700 1 -- 172.21.15.42:0/2805119653 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2954757984 0 0) 0x7f9e600a0710 con 0x7f9e6009f2e0 2019-12-05T05:55:04.895 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:04.886+0000 7f9e5f7fe700 1 -- 172.21.15.42:0/2805119653 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e480030b0 con 0x7f9e6009f2e0 2019-12-05T05:55:04.895 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:04.886+0000 7f9e64dcf700 1 -- 172.21.15.42:0/2805119653 >> v2:172.21.15.40:3300/0 conn(0x7f9e6007e720 msgr2=0x7f9e6007eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:55:04.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:04.886+0000 7f9e64dcf700 1 --2- 172.21.15.42:0/2805119653 >> v2:172.21.15.40:3300/0 conn(0x7f9e6007e720 0x7f9e6007eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:55:04.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:04.886+0000 7f9e64dcf700 1 -- 172.21.15.42:0/2805119653 >> v1:172.21.15.42:6789/0 conn(0x7f9e6009f2e0 legacy=0x7f9e6009f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:55:04.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:04.886+0000 7f9e64dcf700 1 -- 172.21.15.42:0/2805119653 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e6007f0a0 con 0x7f9e600a7740 2019-12-05T05:55:04.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:04.886+0000 7f9e64dcf700 1 --2- 172.21.15.42:0/2805119653 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9e600a7740 0x7f9e6008df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f9e50008790 tx=0x7f9e5000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:04.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:04.886+0000 7f9e64dcf700 1 --2- 172.21.15.42:0/2805119653 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9e600a7740 0x7f9e6008df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f9e50008790 tx=0x7f9e5000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:04.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:04.886+0000 7f9e64dcf700 1 --2- 172.21.15.42:0/2805119653 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9e600a7740 0x7f9e6008df80 secure :-1 s=READY pgs=376 cs=0 l=1 rx=0x7f9e50008790 tx=0x7f9e5000fc70).ready entity=mon.2 client_cookie=67e4cf9f1e6413c6 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:55:04.897 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:04.886+0000 7f9e64dcf700 1 --2- 172.21.15.42:0/2805119653 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9e600a7740 0x7f9e6008df80 secure :-1 s=THROTTLE_DONE pgs=376 cs=0 l=1 rx=0x7f9e50008790 tx=0x7f9e5000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:04.897 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:04.886+0000 7f9e5f7fe700 1 -- 172.21.15.42:0/2805119653 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9e500112f0 con 0x7f9e600a7740 2019-12-05T05:55:04.897 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:04.886+0000 7f9e64dcf700 1 --2- 172.21.15.42:0/2805119653 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9e600a7740 0x7f9e6008df80 secure :-1 s=THROTTLE_DONE pgs=376 cs=0 l=1 rx=0x7f9e50008790 tx=0x7f9e5000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:04.897 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:04.886+0000 7f9e64dcf700 1 --2- 172.21.15.42:0/2805119653 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9e600a7740 0x7f9e6008df80 secure :-1 s=THROTTLE_DONE pgs=376 cs=0 l=1 rx=0x7f9e50008790 tx=0x7f9e5000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:04.897 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:04.886+0000 7f9e5f7fe700 1 -- 172.21.15.42:0/2805119653 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e50008ad0 con 0x7f9e600a7740 2019-12-05T05:55:04.897 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:04.886+0000 7f9e5f7fe700 1 -- 172.21.15.42:0/2805119653 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9e500118d0 con 0x7f9e600a7740 2019-12-05T05:55:04.897 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:04.886+0000 7f9e66832700 1 -- 172.21.15.42:0/2805119653 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9e600a7740 msgr2=0x7f9e6008df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:55:04.898 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:04.886+0000 7f9e66832700 1 --2- 172.21.15.42:0/2805119653 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9e600a7740 0x7f9e6008df80 secure :-1 s=READY pgs=376 cs=0 l=1 rx=0x7f9e50008790 tx=0x7f9e5000fc70).stop 2019-12-05T05:55:04.898 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:04.890+0000 7f9e66832700 1 -- 172.21.15.42:0/2805119653 shutdown_connections 2019-12-05T05:55:04.898 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:04.890+0000 7f9e66832700 1 --2- 172.21.15.42:0/2805119653 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9e600a7740 0x7f9e6008df80 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:55:04.898 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:04.890+0000 7f9e66832700 1 --2- 172.21.15.42:0/2805119653 >> v2:172.21.15.40:3300/0 conn(0x7f9e6007e720 0x7f9e6007eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:55:04.898 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:04.890+0000 7f9e66832700 1 -- 172.21.15.42:0/2805119653 shutdown_connections 2019-12-05T05:55:04.898 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:04.890+0000 7f9e66832700 1 -- 172.21.15.42:0/2805119653 wait complete. 2019-12-05T05:55:04.898 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:04.890+0000 7f9e66832700 1 -- 172.21.15.42:0/2805119653 >> 172.21.15.42:0/2805119653 conn(0x7f9e600aabe0 msgr2=0x7f9e600a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:55:04.907 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:04.890+0000 7f9e66832700 1 Processor -- start 2019-12-05T05:55:04.907 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:04.890+0000 7f9e66832700 1 -- start start 2019-12-05T05:55:04.907 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:04.894+0000 7f9e66832700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9e600a3a20 0x7f9e6011af30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:55:04.907 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:04.894+0000 7f9e66832700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f9e6007b830 0x7f9e6007bc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:55:04.908 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:04.894+0000 7f9e66832700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e601199e0 con 0x7f9e601457e0 2019-12-05T05:55:04.908 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:04.894+0000 7f9e66832700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e6007c1b0 con 0x7f9e6007b830 2019-12-05T05:55:04.908 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:04.894+0000 7f9e66832700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e60145dc0 con 0x7f9e600a3a20 2019-12-05T05:55:04.908 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:04.894+0000 7f9e5ffff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f9e601457e0 0x7f9e601191e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:43976/0 (socket says 172.21.15.42:43976) 2019-12-05T05:55:04.913 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:04.894+0000 7f9e5ffff700 1 -- 172.21.15.42:0/286519289 learned_addr learned my addr 172.21.15.42:0/286519289 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:55:04.913 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:04.894+0000 7f9e64dcf700 1 --2- 172.21.15.42:0/286519289 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9e600a3a20 0x7f9e6011af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:55:04.913 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:04.894+0000 7f9e655d0700 1 --2- 172.21.15.42:0/286519289 >> v2:172.21.15.40:3300/0 conn(0x7f9e6007b830 0x7f9e6007bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:55:04.913 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:04.894+0000 7f9e5d7fa700 1 -- 172.21.15.42:0/286519289 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 607688316 0 0) 0x7f9e601199e0 con 0x7f9e601457e0 2019-12-05T05:55:04.914 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:04.894+0000 7f9e5d7fa700 1 -- 172.21.15.42:0/286519289 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e400030b0 con 0x7f9e601457e0 2019-12-05T05:55:04.914 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:04.894+0000 7f9e64dcf700 1 -- 172.21.15.42:0/286519289 >> v2:172.21.15.40:3300/0 conn(0x7f9e6007b830 msgr2=0x7f9e6007bc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:55:04.914 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:04.894+0000 7f9e64dcf700 1 --2- 172.21.15.42:0/286519289 >> v2:172.21.15.40:3300/0 conn(0x7f9e6007b830 0x7f9e6007bc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:55:04.914 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:04.894+0000 7f9e64dcf700 1 -- 172.21.15.42:0/286519289 >> v1:172.21.15.42:6789/0 conn(0x7f9e601457e0 legacy=0x7f9e601191e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:55:04.914 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:04.894+0000 7f9e64dcf700 1 -- 172.21.15.42:0/286519289 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e6011c140 con 0x7f9e600a3a20 2019-12-05T05:55:04.915 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:04.894+0000 7f9e64dcf700 1 --2- 172.21.15.42:0/286519289 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9e600a3a20 0x7f9e6011af30 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f9e50008790 tx=0x7f9e50011760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:04.915 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:04.894+0000 7f9e64dcf700 1 --2- 172.21.15.42:0/286519289 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9e600a3a20 0x7f9e6011af30 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f9e50008790 tx=0x7f9e50011760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:04.915 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:04.894+0000 7f9e64dcf700 1 --2- 172.21.15.42:0/286519289 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9e600a3a20 0x7f9e6011af30 secure :-1 s=READY pgs=377 cs=0 l=1 rx=0x7f9e50008790 tx=0x7f9e50011760).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:55:04.915 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:04.894+0000 7f9e64dcf700 1 --2- 172.21.15.42:0/286519289 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9e600a3a20 0x7f9e6011af30 secure :-1 s=THROTTLE_DONE pgs=377 cs=0 l=1 rx=0x7f9e50008790 tx=0x7f9e50011760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:04.915 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:04.894+0000 7f9e5d7fa700 1 -- 172.21.15.42:0/286519289 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9e50007150 con 0x7f9e600a3a20 2019-12-05T05:55:04.915 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:04.894+0000 7f9e64dcf700 1 --2- 172.21.15.42:0/286519289 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9e600a3a20 0x7f9e6011af30 secure :-1 s=THROTTLE_DONE pgs=377 cs=0 l=1 rx=0x7f9e50008790 tx=0x7f9e50011760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:04.916 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:04.894+0000 7f9e64dcf700 1 --2- 172.21.15.42:0/286519289 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9e600a3a20 0x7f9e6011af30 secure :-1 s=THROTTLE_DONE pgs=377 cs=0 l=1 rx=0x7f9e50008790 tx=0x7f9e50011760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:04.916 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:04.894+0000 7f9e5d7fa700 1 -- 172.21.15.42:0/286519289 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e500114f0 con 0x7f9e600a3a20 2019-12-05T05:55:04.916 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:04.894+0000 7f9e66832700 1 -- 172.21.15.42:0/286519289 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e60146060 con 0x7f9e600a3a20 2019-12-05T05:55:04.916 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:04.894+0000 7f9e5d7fa700 1 -- 172.21.15.42:0/286519289 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9e50008cc0 con 0x7f9e600a3a20 2019-12-05T05:55:04.916 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:04.894+0000 7f9e66832700 1 -- 172.21.15.42:0/286519289 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e60135690 con 0x7f9e600a3a20 2019-12-05T05:55:04.916 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:04.898+0000 7f9e64dcf700 1 --2- 172.21.15.42:0/286519289 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9e600a3a20 0x7f9e6011af30 secure :-1 s=THROTTLE_DONE pgs=377 cs=0 l=1 rx=0x7f9e50008790 tx=0x7f9e50011760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:04.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:04.898+0000 7f9e5d7fa700 1 -- 172.21.15.42:0/286519289 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f9e50005540 con 0x7f9e600a3a20 2019-12-05T05:55:04.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:04.898+0000 7f9e64dcf700 1 --2- 172.21.15.42:0/286519289 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9e600a3a20 0x7f9e6011af30 secure :-1 s=THROTTLE_DONE pgs=377 cs=0 l=1 rx=0x7f9e50008790 tx=0x7f9e50011760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:04.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:04.898+0000 7f9e5d7fa700 1 --2- 172.21.15.42:0/286519289 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9e4003b250 0x7f9e4003d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:55:04.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:04.898+0000 7f9e5d7fa700 1 -- 172.21.15.42:0/286519289 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 9041+0+0 (secure 0 0 0) 0x7f9e5008f6f0 con 0x7f9e600a3a20 2019-12-05T05:55:04.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:04.898+0000 7f9e5ffff700 1 --2- 172.21.15.42:0/286519289 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9e4003b250 0x7f9e4003d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:55:04.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:04.898+0000 7f9e5ffff700 1 --2- 172.21.15.42:0/286519289 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9e4003b250 0x7f9e4003d760 crc :-1 s=READY pgs=718 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:55:04.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:04.898+0000 7f9e66832700 1 -- 172.21.15.42:0/286519289 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e6005d110 con 0x7f9e600a3a20 2019-12-05T05:55:04.918 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:04.910+0000 7f9e64dcf700 1 --2- 172.21.15.42:0/286519289 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9e600a3a20 0x7f9e6011af30 secure :-1 s=THROTTLE_DONE pgs=377 cs=0 l=1 rx=0x7f9e50008790 tx=0x7f9e50011760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:04.918 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:04.910+0000 7f9e5d7fa700 1 -- 172.21.15.42:0/286519289 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f9e50005850 con 0x7f9e600a3a20 2019-12-05T05:55:05.081 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:05.078+0000 7f9e66832700 1 -- 172.21.15.42:0/286519289 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9e6005b8c0 con 0x7f9e600a3a20 2019-12-05T05:55:05.082 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:05.078+0000 7f9e64dcf700 1 --2- 172.21.15.42:0/286519289 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9e600a3a20 0x7f9e6011af30 secure :-1 s=THROTTLE_DONE pgs=377 cs=0 l=1 rx=0x7f9e50008790 tx=0x7f9e50011760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:05.083 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:05.078+0000 7f9e5d7fa700 1 -- 172.21.15.42:0/286519289 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 207 v207) v1 ==== 95+0+1775 (secure 0 0 0) 0x7f9e50005a70 con 0x7f9e600a3a20 2019-12-05T05:55:05.083 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 207 2019-12-05T05:55:05.083 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:55:05.083 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":207,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":207,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:55:04.257118+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8614},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8614":{"gid":8614,"name":"a","rank":0,"incarnation":204,"state":"up:active","state_seq":32,"addr":"172.21.15.42:6835/2511123581","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2511123581},{"type":"v1","addr":"172.21.15.42:6835","nonce":2511123581}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:55:05.085 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:05.082+0000 7f9e66832700 1 -- 172.21.15.42:0/286519289 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9e4003b250 msgr2=0x7f9e4003d760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:55:05.086 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:05.082+0000 7f9e66832700 1 --2- 172.21.15.42:0/286519289 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9e4003b250 0x7f9e4003d760 crc :-1 s=READY pgs=718 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:55:05.086 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:05.082+0000 7f9e66832700 1 -- 172.21.15.42:0/286519289 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9e600a3a20 msgr2=0x7f9e6011af30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:55:05.086 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:05.082+0000 7f9e66832700 1 --2- 172.21.15.42:0/286519289 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9e600a3a20 0x7f9e6011af30 secure :-1 s=READY pgs=377 cs=0 l=1 rx=0x7f9e50008790 tx=0x7f9e50011760).stop 2019-12-05T05:55:05.086 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:05.082+0000 7f9e66832700 1 -- 172.21.15.42:0/286519289 shutdown_connections 2019-12-05T05:55:05.086 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:05.082+0000 7f9e66832700 1 --2- 172.21.15.42:0/286519289 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9e4003b250 0x7f9e4003d760 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:55:05.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:05.082+0000 7f9e66832700 1 --2- 172.21.15.42:0/286519289 >> v2:172.21.15.40:3300/0 conn(0x7f9e6007b830 0x7f9e6007bc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:55:05.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:05.082+0000 7f9e66832700 1 --2- 172.21.15.42:0/286519289 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9e600a3a20 0x7f9e6011af30 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:55:05.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:05.082+0000 7f9e66832700 1 -- 172.21.15.42:0/286519289 shutdown_connections 2019-12-05T05:55:05.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:05.082+0000 7f9e66832700 1 -- 172.21.15.42:0/286519289 wait complete. 2019-12-05T05:55:05.088 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:05.082+0000 7f9e66832700 1 -- 172.21.15.42:0/286519289 >> 172.21.15.42:0/286519289 conn(0x7f9e600a7740 msgr2=0x7f9e6011c540 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:55:05.096 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a has gained rank=0, replacing gid=6356 2019-12-05T05:55:05.097 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 3 secs before reviving mds.b 2019-12-05T05:55:08.098 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.b 2019-12-05T05:55:08.098 INFO:tasks.ceph.mds.b:Restarting daemon 2019-12-05T05:55:08.098 INFO:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2019-12-05T05:55:08.102 INFO:tasks.ceph.mds.b:Started 2019-12-05T05:55:08.102 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:55:08.166 INFO:tasks.ceph.mds.b.smithi040.stdout:starting mds.b at 2019-12-05T05:55:08.239 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:08.202+0000 7f27e261d700 1 Processor -- start 2019-12-05T05:55:08.239 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:08.202+0000 7f27e261d700 1 -- start start 2019-12-05T05:55:08.239 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:08.202+0000 7f27e261d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f27dc0a7740 0x7f27dc08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:55:08.239 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:08.202+0000 7f27e261d700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f27dc07e720 0x7f27dc07eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:55:08.240 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:08.202+0000 7f27e261d700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27dc0a0710 con 0x7f27dc09f2e0 2019-12-05T05:55:08.240 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:08.202+0000 7f27e261d700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27dc0a0b00 con 0x7f27dc07e720 2019-12-05T05:55:08.240 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:08.202+0000 7f27e261d700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27dc08e5b0 con 0x7f27dc0a7740 2019-12-05T05:55:08.240 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:08.202+0000 7f27e0bba700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f27dc0a7740 0x7f27dc08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:55:08.240 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:08.202+0000 7f27dbfff700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f27dc07e720 0x7f27dc07eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:55:08.240 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:08.202+0000 7f27e0bba700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f27dc0a7740 0x7f27dc08df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:43962/0 (socket says 172.21.15.42:43962) 2019-12-05T05:55:08.240 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:08.202+0000 7f27e0bba700 1 -- 172.21.15.42:0/1138830616 learned_addr learned my addr 172.21.15.42:0/1138830616 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:55:08.241 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:08.206+0000 7f27dbfff700 1 -- 172.21.15.42:0/1138830616 >> v2:172.21.15.42:3300/0 conn(0x7f27dc0a7740 msgr2=0x7f27dc08df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:55:08.241 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:08.206+0000 7f27dbfff700 1 --2- 172.21.15.42:0/1138830616 >> v2:172.21.15.42:3300/0 conn(0x7f27dc0a7740 0x7f27dc08df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:55:08.241 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:08.206+0000 7f27dbfff700 1 -- 172.21.15.42:0/1138830616 >> v1:172.21.15.40:6789/0 conn(0x7f27dc09f2e0 legacy=0x7f27dc09f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:55:08.241 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:08.206+0000 7f27dbfff700 1 -- 172.21.15.42:0/1138830616 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27dc07f0a0 con 0x7f27dc07e720 2019-12-05T05:55:08.241 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:08.206+0000 7f27db7fe700 1 -- 172.21.15.42:0/1138830616 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2059907570 0 0) 0x7f27dc0a0710 con 0x7f27dc09f2e0 2019-12-05T05:55:08.241 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:08.206+0000 7f27e0bba700 1 --2- 172.21.15.42:0/1138830616 >> v2:172.21.15.42:3300/0 conn(0x7f27dc0a7740 0x7f27dc08df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:55:08.241 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:08.206+0000 7f27dbfff700 1 --2- 172.21.15.42:0/1138830616 >> v2:172.21.15.40:3300/0 conn(0x7f27dc07e720 0x7f27dc07eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f27d0002f30 tx=0x7f27d000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:08.242 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:08.206+0000 7f27dbfff700 1 --2- 172.21.15.42:0/1138830616 >> v2:172.21.15.40:3300/0 conn(0x7f27dc07e720 0x7f27dc07eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f27d0002f30 tx=0x7f27d000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:08.242 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:08.206+0000 7f27dbfff700 1 --2- 172.21.15.42:0/1138830616 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f27dc07e720 0x7f27dc07eb50 secure :-1 s=READY pgs=1288 cs=0 l=1 rx=0x7f27d0002f30 tx=0x7f27d000fc70).ready entity=mon.0 client_cookie=4c994e671770b3f1 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:55:08.242 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:08.206+0000 7f27dbfff700 1 --2- 172.21.15.42:0/1138830616 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f27dc07e720 0x7f27dc07eb50 secure :-1 s=THROTTLE_DONE pgs=1288 cs=0 l=1 rx=0x7f27d0002f30 tx=0x7f27d000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:08.242 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:08.206+0000 7f27dbfff700 1 --2- 172.21.15.42:0/1138830616 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f27dc07e720 0x7f27dc07eb50 secure :-1 s=THROTTLE_DONE pgs=1288 cs=0 l=1 rx=0x7f27d0002f30 tx=0x7f27d000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:08.242 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:08.206+0000 7f27db7fe700 1 -- 172.21.15.42:0/1138830616 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f27d00112b0 con 0x7f27dc07e720 2019-12-05T05:55:08.242 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:08.206+0000 7f27dbfff700 1 --2- 172.21.15.42:0/1138830616 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f27dc07e720 0x7f27dc07eb50 secure :-1 s=THROTTLE_DONE pgs=1288 cs=0 l=1 rx=0x7f27d0002f30 tx=0x7f27d000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:08.242 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:08.206+0000 7f27db7fe700 1 -- 172.21.15.42:0/1138830616 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27d0008ad0 con 0x7f27dc07e720 2019-12-05T05:55:08.243 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:08.206+0000 7f27db7fe700 1 -- 172.21.15.42:0/1138830616 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f27d0011890 con 0x7f27dc07e720 2019-12-05T05:55:08.243 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:08.206+0000 7f27e261d700 1 -- 172.21.15.42:0/1138830616 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f27dc07e720 msgr2=0x7f27dc07eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:55:08.243 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:08.206+0000 7f27e261d700 1 --2- 172.21.15.42:0/1138830616 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f27dc07e720 0x7f27dc07eb50 secure :-1 s=READY pgs=1288 cs=0 l=1 rx=0x7f27d0002f30 tx=0x7f27d000fc70).stop 2019-12-05T05:55:08.243 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:08.206+0000 7f27e261d700 1 -- 172.21.15.42:0/1138830616 shutdown_connections 2019-12-05T05:55:08.243 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:08.206+0000 7f27e261d700 1 --2- 172.21.15.42:0/1138830616 >> v2:172.21.15.42:3300/0 conn(0x7f27dc0a7740 0x7f27dc08df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:55:08.243 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:08.206+0000 7f27e261d700 1 --2- 172.21.15.42:0/1138830616 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f27dc07e720 0x7f27dc07eb50 unknown :-1 s=CLOSED pgs=1288 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:55:08.244 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:08.206+0000 7f27e261d700 1 -- 172.21.15.42:0/1138830616 shutdown_connections 2019-12-05T05:55:08.244 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:08.206+0000 7f27e261d700 1 -- 172.21.15.42:0/1138830616 wait complete. 2019-12-05T05:55:08.244 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:08.206+0000 7f27e261d700 1 -- 172.21.15.42:0/1138830616 >> 172.21.15.42:0/1138830616 conn(0x7f27dc0aabe0 msgr2=0x7f27dc0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:55:08.244 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:08.210+0000 7f27e261d700 1 Processor -- start 2019-12-05T05:55:08.245 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:08.210+0000 7f27e261d700 1 -- start start 2019-12-05T05:55:08.245 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:08.210+0000 7f27e261d700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f27dc117440 0x7f27dc0a3330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:55:08.246 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:08.210+0000 7f27e261d700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f27dc12d450 0x7f27dc1375c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:55:08.246 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:08.210+0000 7f27e261d700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27dc117e50 con 0x7f27dc119550 2019-12-05T05:55:08.246 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:08.210+0000 7f27e261d700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27dc119f40 con 0x7f27dc117440 2019-12-05T05:55:08.246 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:08.210+0000 7f27e261d700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27dc117910 con 0x7f27dc12d450 2019-12-05T05:55:08.246 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:08.210+0000 7f27e13bb700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f27dc12d450 0x7f27dc1375c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:55:08.247 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:08.210+0000 7f27e0bba700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f27dc117440 0x7f27dc0a3330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:55:08.247 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:08.210+0000 7f27e13bb700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f27dc12d450 0x7f27dc1375c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:44438/0 (socket says 172.21.15.42:44438) 2019-12-05T05:55:08.248 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:08.210+0000 7f27dbfff700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f27dc119550 0x7f27dc12cc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:53446/0 (socket says 172.21.15.42:53446) 2019-12-05T05:55:08.248 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:08.210+0000 7f27e13bb700 1 -- 172.21.15.42:0/1514789814 learned_addr learned my addr 172.21.15.42:0/1514789814 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:55:08.248 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:08.214+0000 7f27e13bb700 1 -- 172.21.15.42:0/1514789814 >> v2:172.21.15.40:3300/0 conn(0x7f27dc117440 msgr2=0x7f27dc0a3330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:55:08.248 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:08.214+0000 7f27e13bb700 1 --2- 172.21.15.42:0/1514789814 >> v2:172.21.15.40:3300/0 conn(0x7f27dc117440 0x7f27dc0a3330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:55:08.249 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:08.214+0000 7f27e13bb700 1 -- 172.21.15.42:0/1514789814 >> v1:172.21.15.40:6789/0 conn(0x7f27dc119550 legacy=0x7f27dc12cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:55:08.249 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:08.214+0000 7f27d97fa700 1 -- 172.21.15.42:0/1514789814 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4195637373 0 0) 0x7f27dc117e50 con 0x7f27dc119550 2019-12-05T05:55:08.249 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:08.214+0000 7f27e13bb700 1 -- 172.21.15.42:0/1514789814 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27dc07e160 con 0x7f27dc12d450 2019-12-05T05:55:08.249 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:08.214+0000 7f27e0bba700 1 --2- 172.21.15.42:0/1514789814 >> v2:172.21.15.40:3300/0 conn(0x7f27dc117440 0x7f27dc0a3330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:55:08.249 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:08.214+0000 7f27e13bb700 1 --2- 172.21.15.42:0/1514789814 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f27dc12d450 0x7f27dc1375c0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f27d4001210 tx=0x7f27d4012d60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:08.249 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:08.214+0000 7f27e13bb700 1 --2- 172.21.15.42:0/1514789814 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f27dc12d450 0x7f27dc1375c0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f27d4001210 tx=0x7f27d4012d60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:08.249 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:08.214+0000 7f27e13bb700 1 --2- 172.21.15.42:0/1514789814 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f27dc12d450 0x7f27dc1375c0 secure :-1 s=READY pgs=380 cs=0 l=1 rx=0x7f27d4001210 tx=0x7f27d4012d60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:55:08.250 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:08.214+0000 7f27e13bb700 1 -- 172.21.15.42:0/1514789814 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f27dc12d450 msgr2=0x7f27dc1375c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2019-12-05T05:55:08.250 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:08.214+0000 7f27e13bb700 1 -- 172.21.15.42:0/1514789814 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f27dc12d450 msgr2=0x7f27dc1375c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2019-12-05T05:55:08.250 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:08.214+0000 7f27e13bb700 1 --2- 172.21.15.42:0/1514789814 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f27dc12d450 0x7f27dc1375c0 secure :-1 s=READY pgs=380 cs=0 l=1 rx=0x7f27d4001210 tx=0x7f27d4012d60).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-12-05T05:55:08.250 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:08.214+0000 7f27e13bb700 1 --2- 172.21.15.42:0/1514789814 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f27dc12d450 0x7f27dc1375c0 secure :-1 s=READY pgs=380 cs=0 l=1 rx=0x7f27d4001210 tx=0x7f27d4012d60).stop 2019-12-05T05:55:08.250 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:08.214+0000 7f27d97fa700 1 -- 172.21.15.42:0/1514789814 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f27dc12d450 msgr2=0x7f27dc1375c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2019-12-05T05:55:08.250 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:08.214+0000 7f27d97fa700 1 --2- 172.21.15.42:0/1514789814 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f27dc12d450 0x7f27dc1375c0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:55:08.250 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:08.214+0000 7f27d97fa700 1 --2- 172.21.15.42:0/1514789814 >> v2:172.21.15.42:3300/0 conn(0x7f27bc006ed0 0x7f27bc0093e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:55:08.251 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:08.214+0000 7f27d97fa700 1 -- 172.21.15.42:0/1514789814 --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27dc117e50 con 0x7f27bc003050 2019-12-05T05:55:08.251 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:08.214+0000 7f27d97fa700 1 -- 172.21.15.42:0/1514789814 --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27bc00aba0 con 0x7f27bc006ed0 2019-12-05T05:55:08.251 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:08.214+0000 7f27dbfff700 1 --2- 172.21.15.42:0/1514789814 >> v2:172.21.15.42:3300/0 conn(0x7f27bc006ed0 0x7f27bc0093e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:55:08.251 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:08.214+0000 7f27dbfff700 1 -- 172.21.15.42:0/1514789814 >> v1:172.21.15.42:6789/0 conn(0x7f27bc003050 legacy=0x7f27bc005520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:55:08.251 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:08.214+0000 7f27d97fa700 1 -- 172.21.15.42:0/1514789814 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1732681922 0 0) 0x7f27dc117e50 con 0x7f27bc003050 2019-12-05T05:55:08.251 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:08.214+0000 7f27dbfff700 1 -- 172.21.15.42:0/1514789814 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27bc00adf0 con 0x7f27bc006ed0 2019-12-05T05:55:08.251 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:08.214+0000 7f27dbfff700 1 --2- 172.21.15.42:0/1514789814 >> v2:172.21.15.42:3300/0 conn(0x7f27bc006ed0 0x7f27bc0093e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f27d0008790 tx=0x7f27d0008d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:08.252 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:08.214+0000 7f27dbfff700 1 --2- 172.21.15.42:0/1514789814 >> v2:172.21.15.42:3300/0 conn(0x7f27bc006ed0 0x7f27bc0093e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f27d0008790 tx=0x7f27d0008d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:08.252 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:08.214+0000 7f27dbfff700 1 --2- 172.21.15.42:0/1514789814 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f27bc006ed0 0x7f27bc0093e0 secure :-1 s=READY pgs=1689 cs=0 l=1 rx=0x7f27d0008790 tx=0x7f27d0008d50).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:55:08.252 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:08.214+0000 7f27dbfff700 1 --2- 172.21.15.42:0/1514789814 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f27bc006ed0 0x7f27bc0093e0 secure :-1 s=THROTTLE_DONE pgs=1689 cs=0 l=1 rx=0x7f27d0008790 tx=0x7f27d0008d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:08.252 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:08.214+0000 7f27d97fa700 1 -- 172.21.15.42:0/1514789814 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f27d0007150 con 0x7f27bc006ed0 2019-12-05T05:55:08.252 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:08.214+0000 7f27dbfff700 1 --2- 172.21.15.42:0/1514789814 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f27bc006ed0 0x7f27bc0093e0 secure :-1 s=THROTTLE_DONE pgs=1689 cs=0 l=1 rx=0x7f27d0008790 tx=0x7f27d0008d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:08.253 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:08.214+0000 7f27dbfff700 1 --2- 172.21.15.42:0/1514789814 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f27bc006ed0 0x7f27bc0093e0 secure :-1 s=THROTTLE_DONE pgs=1689 cs=0 l=1 rx=0x7f27d0008790 tx=0x7f27d0008d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:08.253 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:08.214+0000 7f27d97fa700 1 -- 172.21.15.42:0/1514789814 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27d00114b0 con 0x7f27bc006ed0 2019-12-05T05:55:08.253 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:08.214+0000 7f27e261d700 1 -- 172.21.15.42:0/1514789814 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27dc119980 con 0x7f27bc006ed0 2019-12-05T05:55:08.253 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:08.214+0000 7f27d97fa700 1 -- 172.21.15.42:0/1514789814 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f27d0005730 con 0x7f27bc006ed0 2019-12-05T05:55:08.253 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:08.214+0000 7f27e261d700 1 -- 172.21.15.42:0/1514789814 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27dc08eb50 con 0x7f27bc006ed0 2019-12-05T05:55:08.253 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:08.214+0000 7f27dbfff700 1 --2- 172.21.15.42:0/1514789814 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f27bc006ed0 0x7f27bc0093e0 secure :-1 s=THROTTLE_DONE pgs=1689 cs=0 l=1 rx=0x7f27d0008790 tx=0x7f27d0008d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:08.253 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:08.218+0000 7f27d97fa700 1 -- 172.21.15.42:0/1514789814 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f27d0005a50 con 0x7f27bc006ed0 2019-12-05T05:55:08.254 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:08.218+0000 7f27dbfff700 1 --2- 172.21.15.42:0/1514789814 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f27bc006ed0 0x7f27bc0093e0 secure :-1 s=THROTTLE_DONE pgs=1689 cs=0 l=1 rx=0x7f27d0008790 tx=0x7f27d0008d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:08.254 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:08.218+0000 7f27e261d700 1 -- 172.21.15.42:0/1514789814 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27dc05d110 con 0x7f27bc006ed0 2019-12-05T05:55:08.254 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:08.218+0000 7f27d97fa700 1 --2- 172.21.15.42:0/1514789814 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f27bc041b70 0x7f27bc044080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:55:08.254 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:08.218+0000 7f27d97fa700 1 -- 172.21.15.42:0/1514789814 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 9041+0+0 (secure 0 0 0) 0x7f27d004b6c0 con 0x7f27bc006ed0 2019-12-05T05:55:08.255 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:08.218+0000 7f27e0bba700 1 --2- 172.21.15.42:0/1514789814 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f27bc041b70 0x7f27bc044080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:55:08.255 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:08.218+0000 7f27e0bba700 1 --2- 172.21.15.42:0/1514789814 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f27bc041b70 0x7f27bc044080 crc :-1 s=READY pgs=720 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:55:08.255 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:08.222+0000 7f27dbfff700 1 --2- 172.21.15.42:0/1514789814 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f27bc006ed0 0x7f27bc0093e0 secure :-1 s=THROTTLE_DONE pgs=1689 cs=0 l=1 rx=0x7f27d0008790 tx=0x7f27d0008d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:08.255 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:08.222+0000 7f27d97fa700 1 -- 172.21.15.42:0/1514789814 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f27d0005d60 con 0x7f27bc006ed0 2019-12-05T05:55:08.388 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:08.386+0000 7f27e261d700 1 -- 172.21.15.42:0/1514789814 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f27dc05b8c0 con 0x7f27bc006ed0 2019-12-05T05:55:08.389 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:08.386+0000 7f27dbfff700 1 --2- 172.21.15.42:0/1514789814 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f27bc006ed0 0x7f27bc0093e0 secure :-1 s=THROTTLE_DONE pgs=1689 cs=0 l=1 rx=0x7f27d0008790 tx=0x7f27d0008d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:08.389 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:08.386+0000 7f27d97fa700 1 -- 172.21.15.42:0/1514789814 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 207 v207) v1 ==== 95+0+1775 (secure 0 0 0) 0x7f27d0007f50 con 0x7f27bc006ed0 2019-12-05T05:55:08.390 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 207 2019-12-05T05:55:08.390 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:55:08.390 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":207,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":207,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:55:04.257118+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8614},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8614":{"gid":8614,"name":"a","rank":0,"incarnation":204,"state":"up:active","state_seq":32,"addr":"172.21.15.42:6835/2511123581","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2511123581},{"type":"v1","addr":"172.21.15.42:6835","nonce":2511123581}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:55:08.392 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:08.390+0000 7f27e261d700 1 -- 172.21.15.42:0/1514789814 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f27bc041b70 msgr2=0x7f27bc044080 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:55:08.392 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:08.390+0000 7f27e261d700 1 --2- 172.21.15.42:0/1514789814 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f27bc041b70 0x7f27bc044080 crc :-1 s=READY pgs=720 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:55:08.392 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:08.390+0000 7f27e261d700 1 -- 172.21.15.42:0/1514789814 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f27bc006ed0 msgr2=0x7f27bc0093e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:55:08.392 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:08.390+0000 7f27e261d700 1 --2- 172.21.15.42:0/1514789814 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f27bc006ed0 0x7f27bc0093e0 secure :-1 s=READY pgs=1689 cs=0 l=1 rx=0x7f27d0008790 tx=0x7f27d0008d50).stop 2019-12-05T05:55:08.393 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:08.390+0000 7f27e13bb700 1 -- 172.21.15.42:0/1514789814 reap_dead start 2019-12-05T05:55:08.393 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:08.390+0000 7f27e13bb700 1 -- 172.21.15.42:0/1514789814 reap_dead start 2019-12-05T05:55:08.393 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:08.390+0000 7f27e261d700 1 -- 172.21.15.42:0/1514789814 shutdown_connections 2019-12-05T05:55:08.393 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:08.390+0000 7f27e261d700 1 --2- 172.21.15.42:0/1514789814 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f27bc006ed0 0x7f27bc0093e0 unknown :-1 s=CLOSED pgs=1689 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:55:08.393 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:08.390+0000 7f27e261d700 1 -- 172.21.15.42:0/1514789814 shutdown_connections 2019-12-05T05:55:08.393 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:08.390+0000 7f27e261d700 1 -- 172.21.15.42:0/1514789814 wait complete. 2019-12-05T05:55:08.393 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:08.390+0000 7f27e261d700 1 -- 172.21.15.42:0/1514789814 >> 172.21.15.42:0/1514789814 conn(0x7f27dc08fdc0 msgr2=0x7f27dc09f5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:55:08.401 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2019-12-05T05:55:10.402 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:55:10.499 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:10.490+0000 7fe14dc92700 1 Processor -- start 2019-12-05T05:55:10.500 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:10.490+0000 7fe14dc92700 1 -- start start 2019-12-05T05:55:10.500 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:10.494+0000 7fe14dc92700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe14808df80 0x7fe14809f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:55:10.500 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:10.494+0000 7fe14dc92700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe14809f830 0x7fe14814efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:55:10.500 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:10.494+0000 7fe14dc92700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe14809fc60 con 0x7fe1480a7740 2019-12-05T05:55:10.500 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:10.494+0000 7fe14dc92700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1480a0b00 con 0x7fe14808df80 2019-12-05T05:55:10.501 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:10.494+0000 7fe14dc92700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe14808e5b0 con 0x7fe14809f830 2019-12-05T05:55:10.501 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:10.494+0000 7fe147fff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fe1480a7740 0x7fe14807e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:44038/0 (socket says 172.21.15.42:44038) 2019-12-05T05:55:10.501 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:10.494+0000 7fe147fff700 1 -- 172.21.15.42:0/952355668 learned_addr learned my addr 172.21.15.42:0/952355668 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:55:10.501 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:10.494+0000 7fe14ca30700 1 --2- 172.21.15.42:0/952355668 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe14809f830 0x7fe14814efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:55:10.502 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:10.494+0000 7fe1477fe700 1 --2- 172.21.15.42:0/952355668 >> v2:172.21.15.40:3300/0 conn(0x7fe14808df80 0x7fe14809f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:55:10.502 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:10.494+0000 7fe146ffd700 1 -- 172.21.15.42:0/952355668 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3054770861 0 0) 0x7fe14809fc60 con 0x7fe1480a7740 2019-12-05T05:55:10.502 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:10.494+0000 7fe146ffd700 1 -- 172.21.15.42:0/952355668 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1300030b0 con 0x7fe1480a7740 2019-12-05T05:55:10.502 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:10.494+0000 7fe14ca30700 1 -- 172.21.15.42:0/952355668 >> v2:172.21.15.40:3300/0 conn(0x7fe14808df80 msgr2=0x7fe14809f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:55:10.502 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:10.494+0000 7fe14ca30700 1 --2- 172.21.15.42:0/952355668 >> v2:172.21.15.40:3300/0 conn(0x7fe14808df80 0x7fe14809f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:55:10.502 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:10.494+0000 7fe14ca30700 1 -- 172.21.15.42:0/952355668 >> v1:172.21.15.42:6789/0 conn(0x7fe1480a7740 legacy=0x7fe14807e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:55:10.502 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:10.494+0000 7fe14ca30700 1 -- 172.21.15.42:0/952355668 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1480a0710 con 0x7fe14809f830 2019-12-05T05:55:10.503 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:10.494+0000 7fe14ca30700 1 --2- 172.21.15.42:0/952355668 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe14809f830 0x7fe14814efe0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fe140008b10 tx=0x7fe14000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:10.503 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:10.494+0000 7fe14ca30700 1 --2- 172.21.15.42:0/952355668 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe14809f830 0x7fe14814efe0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fe140008b10 tx=0x7fe14000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:10.503 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:10.494+0000 7fe14ca30700 1 --2- 172.21.15.42:0/952355668 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe14809f830 0x7fe14814efe0 secure :-1 s=READY pgs=381 cs=0 l=1 rx=0x7fe140008b10 tx=0x7fe14000fff0).ready entity=mon.2 client_cookie=448ef8dafaad4d3a server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:55:10.503 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:10.494+0000 7fe14ca30700 1 --2- 172.21.15.42:0/952355668 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe14809f830 0x7fe14814efe0 secure :-1 s=THROTTLE_DONE pgs=381 cs=0 l=1 rx=0x7fe140008b10 tx=0x7fe14000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:10.503 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:10.494+0000 7fe146ffd700 1 -- 172.21.15.42:0/952355668 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe140010900 con 0x7fe14809f830 2019-12-05T05:55:10.503 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:10.494+0000 7fe14ca30700 1 --2- 172.21.15.42:0/952355668 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe14809f830 0x7fe14814efe0 secure :-1 s=THROTTLE_DONE pgs=381 cs=0 l=1 rx=0x7fe140008b10 tx=0x7fe14000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:10.504 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:10.494+0000 7fe14ca30700 1 --2- 172.21.15.42:0/952355668 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe14809f830 0x7fe14814efe0 secure :-1 s=THROTTLE_DONE pgs=381 cs=0 l=1 rx=0x7fe140008b10 tx=0x7fe14000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:10.504 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:10.494+0000 7fe146ffd700 1 -- 172.21.15.42:0/952355668 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe140008b50 con 0x7fe14809f830 2019-12-05T05:55:10.504 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:10.494+0000 7fe146ffd700 1 -- 172.21.15.42:0/952355668 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe1400110c0 con 0x7fe14809f830 2019-12-05T05:55:10.504 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:10.494+0000 7fe14dc92700 1 -- 172.21.15.42:0/952355668 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe14809f830 msgr2=0x7fe14814efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:55:10.504 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:10.494+0000 7fe14dc92700 1 --2- 172.21.15.42:0/952355668 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe14809f830 0x7fe14814efe0 secure :-1 s=READY pgs=381 cs=0 l=1 rx=0x7fe140008b10 tx=0x7fe14000fff0).stop 2019-12-05T05:55:10.504 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:10.494+0000 7fe14dc92700 1 -- 172.21.15.42:0/952355668 shutdown_connections 2019-12-05T05:55:10.504 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:10.494+0000 7fe14dc92700 1 --2- 172.21.15.42:0/952355668 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe14809f830 0x7fe14814efe0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:55:10.505 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:10.494+0000 7fe14dc92700 1 --2- 172.21.15.42:0/952355668 >> v2:172.21.15.40:3300/0 conn(0x7fe14808df80 0x7fe14809f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:55:10.505 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:10.494+0000 7fe14dc92700 1 -- 172.21.15.42:0/952355668 shutdown_connections 2019-12-05T05:55:10.505 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:10.494+0000 7fe14dc92700 1 -- 172.21.15.42:0/952355668 wait complete. 2019-12-05T05:55:10.505 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:10.494+0000 7fe14dc92700 1 -- 172.21.15.42:0/952355668 >> 172.21.15.42:0/952355668 conn(0x7fe1480aabe0 msgr2=0x7fe1480a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:55:10.508 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:10.498+0000 7fe14dc92700 1 Processor -- start 2019-12-05T05:55:10.508 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:10.498+0000 7fe14dc92700 1 -- start start 2019-12-05T05:55:10.508 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:10.498+0000 7fe14dc92700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe1481479e0 0x7fe148146c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:55:10.509 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:10.498+0000 7fe14dc92700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe1480a1d80 0x7fe14812cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:55:10.509 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:10.498+0000 7fe14dc92700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1480a21b0 con 0x7fe148121500 2019-12-05T05:55:10.509 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:10.498+0000 7fe14dc92700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe148146440 con 0x7fe1480a1d80 2019-12-05T05:55:10.509 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:10.498+0000 7fe14dc92700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1481466e0 con 0x7fe1481479e0 2019-12-05T05:55:10.509 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:10.498+0000 7fe147fff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe1481479e0 0x7fe148146c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:55:10.509 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:10.498+0000 7fe147fff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe1481479e0 0x7fe148146c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:44006/0 (socket says 172.21.15.42:44006) 2019-12-05T05:55:10.509 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:10.498+0000 7fe147fff700 1 -- 172.21.15.42:0/3317454962 learned_addr learned my addr 172.21.15.42:0/3317454962 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:55:10.510 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:10.498+0000 7fe1477fe700 1 --2- 172.21.15.42:0/3317454962 >> v2:172.21.15.40:3300/0 conn(0x7fe1480a1d80 0x7fe14812cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:55:10.510 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:10.498+0000 7fe147fff700 1 -- 172.21.15.42:0/3317454962 >> v2:172.21.15.40:3300/0 conn(0x7fe1480a1d80 msgr2=0x7fe14812cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:55:10.510 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:10.498+0000 7fe147fff700 1 --2- 172.21.15.42:0/3317454962 >> v2:172.21.15.40:3300/0 conn(0x7fe1480a1d80 0x7fe14812cea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:55:10.510 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:10.498+0000 7fe147fff700 1 -- 172.21.15.42:0/3317454962 >> v1:172.21.15.40:6789/0 conn(0x7fe148121500 legacy=0x7fe148130460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:55:10.510 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:10.498+0000 7fe147fff700 1 -- 172.21.15.42:0/3317454962 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe14812d3f0 con 0x7fe1481479e0 2019-12-05T05:55:10.510 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:10.498+0000 7fe147fff700 1 --2- 172.21.15.42:0/3317454962 >> v2:172.21.15.42:3300/0 conn(0x7fe1481479e0 0x7fe148146c20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fe138004370 tx=0x7fe138010a00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:10.510 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:10.498+0000 7fe147fff700 1 --2- 172.21.15.42:0/3317454962 >> v2:172.21.15.42:3300/0 conn(0x7fe1481479e0 0x7fe148146c20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fe138004370 tx=0x7fe138010a00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:10.511 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:10.498+0000 7fe147fff700 1 --2- 172.21.15.42:0/3317454962 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe1481479e0 0x7fe148146c20 secure :-1 s=READY pgs=1691 cs=0 l=1 rx=0x7fe138004370 tx=0x7fe138010a00).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:55:10.511 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:10.502+0000 7fe147fff700 1 --2- 172.21.15.42:0/3317454962 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe1481479e0 0x7fe148146c20 secure :-1 s=THROTTLE_DONE pgs=1691 cs=0 l=1 rx=0x7fe138004370 tx=0x7fe138010a00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:10.511 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:10.502+0000 7fe147fff700 1 --2- 172.21.15.42:0/3317454962 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe1481479e0 0x7fe148146c20 secure :-1 s=THROTTLE_DONE pgs=1691 cs=0 l=1 rx=0x7fe138004370 tx=0x7fe138010a00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:10.511 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:10.502+0000 7fe147fff700 1 --2- 172.21.15.42:0/3317454962 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe1481479e0 0x7fe148146c20 secure :-1 s=THROTTLE_DONE pgs=1691 cs=0 l=1 rx=0x7fe138004370 tx=0x7fe138010a00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:10.511 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:10.502+0000 7fe144ff9700 1 -- 172.21.15.42:0/3317454962 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe13800cab0 con 0x7fe1481479e0 2019-12-05T05:55:10.511 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:10.502+0000 7fe144ff9700 1 -- 172.21.15.42:0/3317454962 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe138010250 con 0x7fe1481479e0 2019-12-05T05:55:10.511 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:10.502+0000 7fe144ff9700 1 -- 172.21.15.42:0/3317454962 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe13800d2b0 con 0x7fe1481479e0 2019-12-05T05:55:10.513 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:10.502+0000 7fe14dc92700 1 -- 172.21.15.42:0/3317454962 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe148147220 con 0x7fe1481479e0 2019-12-05T05:55:10.513 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:10.502+0000 7fe14dc92700 1 -- 172.21.15.42:0/3317454962 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe148121930 con 0x7fe1481479e0 2019-12-05T05:55:10.513 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:10.502+0000 7fe147fff700 1 --2- 172.21.15.42:0/3317454962 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe1481479e0 0x7fe148146c20 secure :-1 s=THROTTLE_DONE pgs=1691 cs=0 l=1 rx=0x7fe138004370 tx=0x7fe138010a00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:10.513 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:10.502+0000 7fe144ff9700 1 -- 172.21.15.42:0/3317454962 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fe13800d5c0 con 0x7fe1481479e0 2019-12-05T05:55:10.513 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:10.502+0000 7fe147fff700 1 --2- 172.21.15.42:0/3317454962 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe1481479e0 0x7fe148146c20 secure :-1 s=THROTTLE_DONE pgs=1691 cs=0 l=1 rx=0x7fe138004370 tx=0x7fe138010a00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:10.514 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:10.502+0000 7fe144ff9700 1 --2- 172.21.15.42:0/3317454962 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe12803adc0 0x7fe12803d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:55:10.514 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:10.502+0000 7fe144ff9700 1 -- 172.21.15.42:0/3317454962 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 9041+0+0 (secure 0 0 0) 0x7fe13804ac60 con 0x7fe1481479e0 2019-12-05T05:55:10.514 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:10.502+0000 7fe1477fe700 1 --2- 172.21.15.42:0/3317454962 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe12803adc0 0x7fe12803d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:55:10.514 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:10.502+0000 7fe1477fe700 1 --2- 172.21.15.42:0/3317454962 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe12803adc0 0x7fe12803d2d0 crc :-1 s=READY pgs=721 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:55:10.514 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:10.506+0000 7fe14dc92700 1 -- 172.21.15.42:0/3317454962 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe14805d110 con 0x7fe1481479e0 2019-12-05T05:55:10.516 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:10.514+0000 7fe147fff700 1 --2- 172.21.15.42:0/3317454962 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe1481479e0 0x7fe148146c20 secure :-1 s=THROTTLE_DONE pgs=1691 cs=0 l=1 rx=0x7fe138004370 tx=0x7fe138010a00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:10.516 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:10.514+0000 7fe144ff9700 1 -- 172.21.15.42:0/3317454962 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fe138031ce0 con 0x7fe1481479e0 2019-12-05T05:55:10.681 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:10.674+0000 7fe14dc92700 1 -- 172.21.15.42:0/3317454962 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe14805b8c0 con 0x7fe1481479e0 2019-12-05T05:55:10.681 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:10.678+0000 7fe147fff700 1 --2- 172.21.15.42:0/3317454962 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe1481479e0 0x7fe148146c20 secure :-1 s=THROTTLE_DONE pgs=1691 cs=0 l=1 rx=0x7fe138004370 tx=0x7fe138010a00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:10.682 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:10.678+0000 7fe144ff9700 1 -- 172.21.15.42:0/3317454962 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 207 v207) v1 ==== 95+0+1775 (secure 0 0 0) 0x7fe138031f40 con 0x7fe1481479e0 2019-12-05T05:55:10.682 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 207 2019-12-05T05:55:10.682 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:55:10.682 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":207,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":207,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:55:04.257118+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8614},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8614":{"gid":8614,"name":"a","rank":0,"incarnation":204,"state":"up:active","state_seq":32,"addr":"172.21.15.42:6835/2511123581","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2511123581},{"type":"v1","addr":"172.21.15.42:6835","nonce":2511123581}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:55:10.684 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:10.682+0000 7fe14dc92700 1 -- 172.21.15.42:0/3317454962 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe12803adc0 msgr2=0x7fe12803d2d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:55:10.684 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:10.682+0000 7fe14dc92700 1 --2- 172.21.15.42:0/3317454962 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe12803adc0 0x7fe12803d2d0 crc :-1 s=READY pgs=721 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:55:10.684 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:10.682+0000 7fe14dc92700 1 -- 172.21.15.42:0/3317454962 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe1481479e0 msgr2=0x7fe148146c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:55:10.684 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:10.682+0000 7fe14dc92700 1 --2- 172.21.15.42:0/3317454962 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe1481479e0 0x7fe148146c20 secure :-1 s=READY pgs=1691 cs=0 l=1 rx=0x7fe138004370 tx=0x7fe138010a00).stop 2019-12-05T05:55:10.685 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:10.682+0000 7fe14dc92700 1 -- 172.21.15.42:0/3317454962 shutdown_connections 2019-12-05T05:55:10.685 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:10.682+0000 7fe14dc92700 1 --2- 172.21.15.42:0/3317454962 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe12803adc0 0x7fe12803d2d0 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:55:10.685 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:10.682+0000 7fe14dc92700 1 --2- 172.21.15.42:0/3317454962 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe1481479e0 0x7fe148146c20 unknown :-1 s=CLOSED pgs=1691 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:55:10.685 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:10.682+0000 7fe14dc92700 1 --2- 172.21.15.42:0/3317454962 >> v2:172.21.15.40:3300/0 conn(0x7fe1480a1d80 0x7fe14812cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:55:10.686 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:10.682+0000 7fe14dc92700 1 -- 172.21.15.42:0/3317454962 shutdown_connections 2019-12-05T05:55:10.686 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:10.682+0000 7fe14dc92700 1 -- 172.21.15.42:0/3317454962 wait complete. 2019-12-05T05:55:10.686 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:10.682+0000 7fe14dc92700 1 -- 172.21.15.42:0/3317454962 >> 172.21.15.42:0/3317454962 conn(0x7fe1480a82a0 msgr2=0x7fe148059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:55:10.697 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2019-12-05T05:55:12.401 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:55:12.406 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:55:12.698 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:55:12.803 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:12.794+0000 7f6e58e81700 1 Processor -- start 2019-12-05T05:55:12.803 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:12.794+0000 7f6e58e81700 1 -- start start 2019-12-05T05:55:12.803 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:12.798+0000 7f6e58e81700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6e540a7740 0x7f6e5408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:55:12.803 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:12.798+0000 7f6e58e81700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6e5407e720 0x7f6e5407eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:55:12.803 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:12.798+0000 7f6e58e81700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f6e5409f2e0 0x7f6e5409f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:55:12.804 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:12.798+0000 7f6e58e81700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e540a0b00 con 0x7f6e5409f2e0 2019-12-05T05:55:12.804 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:12.798+0000 7f6e58e81700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e5408e5b0 con 0x7f6e5407e720 2019-12-05T05:55:12.804 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:12.798+0000 7f6e58e81700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e5408e850 con 0x7f6e540a7740 2019-12-05T05:55:12.804 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:12.798+0000 7f6e5259c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6e5407e720 0x7f6e5407eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:55:12.804 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:12.798+0000 7f6e5259c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6e5407e720 0x7f6e5407eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:44016/0 (socket says 172.21.15.42:44016) 2019-12-05T05:55:12.804 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:12.798+0000 7f6e5259c700 1 -- 172.21.15.42:0/213676519 learned_addr learned my addr 172.21.15.42:0/213676519 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:55:12.804 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:12.798+0000 7f6e5359e700 1 --2- 172.21.15.42:0/213676519 >> v2:172.21.15.40:3300/0 conn(0x7f6e5409f2e0 0x7f6e5409f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:55:12.805 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:12.798+0000 7f6e52d9d700 1 --2- 172.21.15.42:0/213676519 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6e540a7740 0x7f6e5408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:55:12.805 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:12.798+0000 7f6e5259c700 1 -- 172.21.15.42:0/213676519 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6e540a7740 msgr2=0x7f6e5408df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:55:12.805 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:12.798+0000 7f6e5259c700 1 --2- 172.21.15.42:0/213676519 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6e540a7740 0x7f6e5408df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:55:12.805 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:12.798+0000 7f6e5259c700 1 -- 172.21.15.42:0/213676519 >> v2:172.21.15.40:3300/0 conn(0x7f6e5409f2e0 msgr2=0x7f6e5409f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:55:12.805 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:12.798+0000 7f6e5259c700 1 --2- 172.21.15.42:0/213676519 >> v2:172.21.15.40:3300/0 conn(0x7f6e5409f2e0 0x7f6e5409f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:55:12.805 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:12.798+0000 7f6e5259c700 1 -- 172.21.15.42:0/213676519 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e5407f0a0 con 0x7f6e5407e720 2019-12-05T05:55:12.805 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:12.798+0000 7f6e5259c700 1 --2- 172.21.15.42:0/213676519 >> v2:172.21.15.42:3300/0 conn(0x7f6e5407e720 0x7f6e5407eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f6e5408e770 tx=0x7f6e44010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:12.806 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:12.798+0000 7f6e5259c700 1 --2- 172.21.15.42:0/213676519 >> v2:172.21.15.42:3300/0 conn(0x7f6e5407e720 0x7f6e5407eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f6e5408e770 tx=0x7f6e44010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:12.806 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:12.798+0000 7f6e5259c700 1 --2- 172.21.15.42:0/213676519 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6e5407e720 0x7f6e5407eb50 secure :-1 s=READY pgs=1692 cs=0 l=1 rx=0x7f6e5408e770 tx=0x7f6e44010130).ready entity=mon.1 client_cookie=5a9c3fbda5ca0a77 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:55:12.806 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:12.798+0000 7f6e5259c700 1 --2- 172.21.15.42:0/213676519 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6e5407e720 0x7f6e5407eb50 secure :-1 s=THROTTLE_DONE pgs=1692 cs=0 l=1 rx=0x7f6e5408e770 tx=0x7f6e44010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:12.806 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:12.798+0000 7f6e51d9b700 1 -- 172.21.15.42:0/213676519 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6e44010de0 con 0x7f6e5407e720 2019-12-05T05:55:12.806 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:12.798+0000 7f6e5259c700 1 --2- 172.21.15.42:0/213676519 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6e5407e720 0x7f6e5407eb50 secure :-1 s=THROTTLE_DONE pgs=1692 cs=0 l=1 rx=0x7f6e5408e770 tx=0x7f6e44010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:12.806 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:12.798+0000 7f6e5259c700 1 --2- 172.21.15.42:0/213676519 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6e5407e720 0x7f6e5407eb50 secure :-1 s=THROTTLE_DONE pgs=1692 cs=0 l=1 rx=0x7f6e5408e770 tx=0x7f6e44010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:12.806 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:12.798+0000 7f6e51d9b700 1 -- 172.21.15.42:0/213676519 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e44010a00 con 0x7f6e5407e720 2019-12-05T05:55:12.807 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:12.798+0000 7f6e51d9b700 1 -- 172.21.15.42:0/213676519 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6e440114a0 con 0x7f6e5407e720 2019-12-05T05:55:12.807 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:12.798+0000 7f6e58e81700 1 -- 172.21.15.42:0/213676519 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6e5407e720 msgr2=0x7f6e5407eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:55:12.807 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:12.798+0000 7f6e58e81700 1 --2- 172.21.15.42:0/213676519 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6e5407e720 0x7f6e5407eb50 secure :-1 s=READY pgs=1692 cs=0 l=1 rx=0x7f6e5408e770 tx=0x7f6e44010130).stop 2019-12-05T05:55:12.807 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:12.798+0000 7f6e58e81700 1 -- 172.21.15.42:0/213676519 shutdown_connections 2019-12-05T05:55:12.807 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:12.798+0000 7f6e58e81700 1 --2- 172.21.15.42:0/213676519 >> v2:172.21.15.40:3300/0 conn(0x7f6e5409f2e0 0x7f6e5409f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:55:12.807 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:12.798+0000 7f6e58e81700 1 --2- 172.21.15.42:0/213676519 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6e540a7740 0x7f6e5408df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:55:12.807 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:12.798+0000 7f6e58e81700 1 --2- 172.21.15.42:0/213676519 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6e5407e720 0x7f6e5407eb50 unknown :-1 s=CLOSED pgs=1692 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:55:12.808 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:12.798+0000 7f6e58e81700 1 -- 172.21.15.42:0/213676519 shutdown_connections 2019-12-05T05:55:12.808 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:12.798+0000 7f6e58e81700 1 -- 172.21.15.42:0/213676519 wait complete. 2019-12-05T05:55:12.808 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:12.798+0000 7f6e58e81700 1 -- 172.21.15.42:0/213676519 >> 172.21.15.42:0/213676519 conn(0x7f6e540aabe0 msgr2=0x7f6e540a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:55:12.812 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:12.802+0000 7f6e58e81700 1 Processor -- start 2019-12-05T05:55:12.812 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:12.802+0000 7f6e58e81700 1 -- start start 2019-12-05T05:55:12.812 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:12.802+0000 7f6e58e81700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6e54122900 0x7f6e5411a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:55:12.812 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:12.802+0000 7f6e58e81700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f6e540a3d80 0x7f6e541243b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:55:12.812 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:12.802+0000 7f6e58e81700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e5411bb60 con 0x7f6e5411b730 2019-12-05T05:55:12.812 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:12.802+0000 7f6e58e81700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e540a3b20 con 0x7f6e540a3d80 2019-12-05T05:55:12.813 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:12.802+0000 7f6e58e81700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e5411c0d0 con 0x7f6e54122900 2019-12-05T05:55:12.813 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:12.802+0000 7f6e52d9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6e54122900 0x7f6e5411a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:55:12.813 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:12.802+0000 7f6e52d9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6e54122900 0x7f6e5411a1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:44024/0 (socket says 172.21.15.42:44024) 2019-12-05T05:55:12.813 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:12.802+0000 7f6e5259c700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f6e5411b730 0x7f6e54130450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:44060/0 (socket says 172.21.15.42:44060) 2019-12-05T05:55:12.813 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:12.802+0000 7f6e52d9d700 1 -- 172.21.15.42:0/3067583583 learned_addr learned my addr 172.21.15.42:0/3067583583 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:55:12.813 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:12.802+0000 7f6e5359e700 1 --2- 172.21.15.42:0/3067583583 >> v2:172.21.15.40:3300/0 conn(0x7f6e540a3d80 0x7f6e541243b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:55:12.813 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:12.802+0000 7f6e52d9d700 1 -- 172.21.15.42:0/3067583583 >> v2:172.21.15.40:3300/0 conn(0x7f6e540a3d80 msgr2=0x7f6e541243b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:55:12.814 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:12.802+0000 7f6e52d9d700 1 --2- 172.21.15.42:0/3067583583 >> v2:172.21.15.40:3300/0 conn(0x7f6e540a3d80 0x7f6e541243b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:55:12.814 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:12.802+0000 7f6e4b7fe700 1 -- 172.21.15.42:0/3067583583 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2546106225 0 0) 0x7f6e5411bb60 con 0x7f6e5411b730 2019-12-05T05:55:12.814 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:12.802+0000 7f6e52d9d700 1 -- 172.21.15.42:0/3067583583 >> v1:172.21.15.42:6789/0 conn(0x7f6e5411b730 legacy=0x7f6e54130450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:55:12.814 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:12.802+0000 7f6e52d9d700 1 -- 172.21.15.42:0/3067583583 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e54124900 con 0x7f6e54122900 2019-12-05T05:55:12.814 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:12.802+0000 7f6e5359e700 1 --2- 172.21.15.42:0/3067583583 >> v2:172.21.15.40:3300/0 conn(0x7f6e540a3d80 0x7f6e541243b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:55:12.814 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:12.802+0000 7f6e52d9d700 1 --2- 172.21.15.42:0/3067583583 >> v2:172.21.15.42:3300/0 conn(0x7f6e54122900 0x7f6e5411a1b0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f6e3c003070 tx=0x7f6e3c010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:12.814 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:12.802+0000 7f6e52d9d700 1 --2- 172.21.15.42:0/3067583583 >> v2:172.21.15.42:3300/0 conn(0x7f6e54122900 0x7f6e5411a1b0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f6e3c003070 tx=0x7f6e3c010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:12.815 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:12.802+0000 7f6e52d9d700 1 --2- 172.21.15.42:0/3067583583 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6e54122900 0x7f6e5411a1b0 secure :-1 s=READY pgs=1694 cs=0 l=1 rx=0x7f6e3c003070 tx=0x7f6e3c010b60).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:55:12.815 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:12.802+0000 7f6e52d9d700 1 --2- 172.21.15.42:0/3067583583 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6e54122900 0x7f6e5411a1b0 secure :-1 s=THROTTLE_DONE pgs=1694 cs=0 l=1 rx=0x7f6e3c003070 tx=0x7f6e3c010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:12.815 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:12.802+0000 7f6e4b7fe700 1 -- 172.21.15.42:0/3067583583 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6e3c012a30 con 0x7f6e54122900 2019-12-05T05:55:12.815 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:12.802+0000 7f6e52d9d700 1 --2- 172.21.15.42:0/3067583583 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6e54122900 0x7f6e5411a1b0 secure :-1 s=THROTTLE_DONE pgs=1694 cs=0 l=1 rx=0x7f6e3c003070 tx=0x7f6e3c010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:12.815 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:12.802+0000 7f6e52d9d700 1 --2- 172.21.15.42:0/3067583583 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6e54122900 0x7f6e5411a1b0 secure :-1 s=THROTTLE_DONE pgs=1694 cs=0 l=1 rx=0x7f6e3c003070 tx=0x7f6e3c010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:12.815 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:12.802+0000 7f6e4b7fe700 1 -- 172.21.15.42:0/3067583583 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e3c012650 con 0x7f6e54122900 2019-12-05T05:55:12.815 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:12.802+0000 7f6e58e81700 1 -- 172.21.15.42:0/3067583583 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e5407e090 con 0x7f6e54122900 2019-12-05T05:55:12.816 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:12.802+0000 7f6e4b7fe700 1 -- 172.21.15.42:0/3067583583 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6e3c011360 con 0x7f6e54122900 2019-12-05T05:55:12.816 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:12.806+0000 7f6e58e81700 1 -- 172.21.15.42:0/3067583583 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e540a41b0 con 0x7f6e54122900 2019-12-05T05:55:12.816 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:12.806+0000 7f6e52d9d700 1 --2- 172.21.15.42:0/3067583583 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6e54122900 0x7f6e5411a1b0 secure :-1 s=THROTTLE_DONE pgs=1694 cs=0 l=1 rx=0x7f6e3c003070 tx=0x7f6e3c010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:12.816 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:12.806+0000 7f6e4b7fe700 1 -- 172.21.15.42:0/3067583583 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f6e3c007170 con 0x7f6e54122900 2019-12-05T05:55:12.816 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:12.806+0000 7f6e52d9d700 1 --2- 172.21.15.42:0/3067583583 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6e54122900 0x7f6e5411a1b0 secure :-1 s=THROTTLE_DONE pgs=1694 cs=0 l=1 rx=0x7f6e3c003070 tx=0x7f6e3c010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:12.816 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:12.806+0000 7f6e4b7fe700 1 --2- 172.21.15.42:0/3067583583 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6e3803ad20 0x7f6e3803d230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:55:12.817 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:12.806+0000 7f6e4b7fe700 1 -- 172.21.15.42:0/3067583583 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 9041+0+0 (secure 0 0 0) 0x7f6e3c04adf0 con 0x7f6e54122900 2019-12-05T05:55:12.817 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:12.806+0000 7f6e5259c700 1 --2- 172.21.15.42:0/3067583583 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6e3803ad20 0x7f6e3803d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:55:12.817 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:12.806+0000 7f6e5259c700 1 --2- 172.21.15.42:0/3067583583 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6e3803ad20 0x7f6e3803d230 crc :-1 s=READY pgs=723 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:55:12.817 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:12.810+0000 7f6e58e81700 1 -- 172.21.15.42:0/3067583583 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e5405d110 con 0x7f6e54122900 2019-12-05T05:55:12.822 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:12.818+0000 7f6e52d9d700 1 --2- 172.21.15.42:0/3067583583 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6e54122900 0x7f6e5411a1b0 secure :-1 s=THROTTLE_DONE pgs=1694 cs=0 l=1 rx=0x7f6e3c003070 tx=0x7f6e3c010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:12.822 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:12.818+0000 7f6e4b7fe700 1 -- 172.21.15.42:0/3067583583 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f6e3c031e70 con 0x7f6e54122900 2019-12-05T05:55:12.991 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:12.990+0000 7f6e58e81700 1 -- 172.21.15.42:0/3067583583 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6e5405b8c0 con 0x7f6e54122900 2019-12-05T05:55:12.992 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:12.990+0000 7f6e52d9d700 1 --2- 172.21.15.42:0/3067583583 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6e54122900 0x7f6e5411a1b0 secure :-1 s=THROTTLE_DONE pgs=1694 cs=0 l=1 rx=0x7f6e3c003070 tx=0x7f6e3c010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:12.992 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:12.990+0000 7f6e4b7fe700 1 -- 172.21.15.42:0/3067583583 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 208 v208) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f6e3c0320d0 con 0x7f6e54122900 2019-12-05T05:55:12.994 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 208 2019-12-05T05:55:12.995 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:55:12.995 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":208,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6461,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/3549993215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3549993215},{"type":"v1","addr":"172.21.15.40:6833","nonce":3549993215}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":208}],"filesystems":[{"mdsmap":{"epoch":207,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:55:04.257118+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8614},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8614":{"gid":8614,"name":"a","rank":0,"incarnation":204,"state":"up:active","state_seq":32,"addr":"172.21.15.42:6835/2511123581","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2511123581},{"type":"v1","addr":"172.21.15.42:6835","nonce":2511123581}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:55:12.999 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:12.994+0000 7f6e58e81700 1 -- 172.21.15.42:0/3067583583 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6e3803ad20 msgr2=0x7f6e3803d230 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:55:12.999 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:12.994+0000 7f6e58e81700 1 --2- 172.21.15.42:0/3067583583 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6e3803ad20 0x7f6e3803d230 crc :-1 s=READY pgs=723 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:55:12.999 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:12.994+0000 7f6e58e81700 1 -- 172.21.15.42:0/3067583583 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6e54122900 msgr2=0x7f6e5411a1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:55:12.999 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:12.994+0000 7f6e58e81700 1 --2- 172.21.15.42:0/3067583583 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6e54122900 0x7f6e5411a1b0 secure :-1 s=READY pgs=1694 cs=0 l=1 rx=0x7f6e3c003070 tx=0x7f6e3c010b60).stop 2019-12-05T05:55:12.999 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:12.994+0000 7f6e58e81700 1 -- 172.21.15.42:0/3067583583 shutdown_connections 2019-12-05T05:55:12.999 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:12.994+0000 7f6e58e81700 1 --2- 172.21.15.42:0/3067583583 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6e3803ad20 0x7f6e3803d230 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:55:13.000 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:12.994+0000 7f6e58e81700 1 --2- 172.21.15.42:0/3067583583 >> v2:172.21.15.40:3300/0 conn(0x7f6e540a3d80 0x7f6e541243b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:55:13.000 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:12.994+0000 7f6e58e81700 1 --2- 172.21.15.42:0/3067583583 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6e54122900 0x7f6e5411a1b0 unknown :-1 s=CLOSED pgs=1694 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:55:13.000 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:12.994+0000 7f6e58e81700 1 -- 172.21.15.42:0/3067583583 shutdown_connections 2019-12-05T05:55:13.000 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:12.994+0000 7f6e58e81700 1 -- 172.21.15.42:0/3067583583 wait complete. 2019-12-05T05:55:13.000 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:12.994+0000 7f6e58e81700 1 -- 172.21.15.42:0/3067583583 >> 172.21.15.42:0/3067583583 conn(0x7f6e540a82a0 msgr2=0x7f6e540a9790 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:55:13.011 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b reported in up:standby state 2019-12-05T05:55:13.011 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 12 secs before thrashing 2019-12-05T05:55:25.012 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:55:25.121 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.114+0000 7f2afeee2700 1 Processor -- start 2019-12-05T05:55:25.121 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.114+0000 7f2afeee2700 1 -- start start 2019-12-05T05:55:25.121 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.114+0000 7f2afeee2700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2af808df80 0x7f2af809f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:55:25.122 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.114+0000 7f2afeee2700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2af809fc60 con 0x7f2af80a7740 2019-12-05T05:55:25.122 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.114+0000 7f2afeee2700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2af80a0710 con 0x7f2af809f830 2019-12-05T05:55:25.122 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.114+0000 7f2afeee2700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2af80a0b00 con 0x7f2af808df80 2019-12-05T05:55:25.122 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.114+0000 7f2afdc80700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f2af809f830 0x7f2af814efe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:44122/0 (socket says 172.21.15.42:44122) 2019-12-05T05:55:25.122 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.114+0000 7f2afdc80700 1 -- 172.21.15.42:0/460209554 learned_addr learned my addr 172.21.15.42:0/460209554 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:55:25.122 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.114+0000 7f2afcc7e700 1 --2- 172.21.15.42:0/460209554 >> v2:172.21.15.40:3300/0 conn(0x7f2af808df80 0x7f2af809f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:55:25.122 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.114+0000 7f2aebfff700 1 -- 172.21.15.42:0/460209554 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4039032090 0 0) 0x7f2af80a0710 con 0x7f2af809f830 2019-12-05T05:55:25.123 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.114+0000 7f2aebfff700 1 -- 172.21.15.42:0/460209554 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ae40030b0 con 0x7f2af809f830 2019-12-05T05:55:25.123 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.114+0000 7f2afcc7e700 1 -- 172.21.15.42:0/460209554 >> v1:172.21.15.42:6789/0 conn(0x7f2af809f830 legacy=0x7f2af814efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:55:25.123 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.114+0000 7f2afcc7e700 1 -- 172.21.15.42:0/460209554 >> v1:172.21.15.40:6789/0 conn(0x7f2af80a7740 legacy=0x7f2af807e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:55:25.123 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.114+0000 7f2aebfff700 1 -- 172.21.15.42:0/460209554 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3045469011 0 0) 0x7f2af809fc60 con 0x7f2af80a7740 2019-12-05T05:55:25.123 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.114+0000 7f2afcc7e700 1 -- 172.21.15.42:0/460209554 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2af807ef20 con 0x7f2af808df80 2019-12-05T05:55:25.123 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.114+0000 7f2afcc7e700 1 --2- 172.21.15.42:0/460209554 >> v2:172.21.15.40:3300/0 conn(0x7f2af808df80 0x7f2af809f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f2af4008790 tx=0x7f2af400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:25.123 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.114+0000 7f2afcc7e700 1 --2- 172.21.15.42:0/460209554 >> v2:172.21.15.40:3300/0 conn(0x7f2af808df80 0x7f2af809f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f2af4008790 tx=0x7f2af400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:25.124 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.114+0000 7f2afcc7e700 1 --2- 172.21.15.42:0/460209554 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2af808df80 0x7f2af809f2e0 secure :-1 s=READY pgs=1292 cs=0 l=1 rx=0x7f2af4008790 tx=0x7f2af400fbf0).ready entity=mon.0 client_cookie=fa1625431f7c22ad server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:55:25.124 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.114+0000 7f2afcc7e700 1 --2- 172.21.15.42:0/460209554 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2af808df80 0x7f2af809f2e0 secure :-1 s=THROTTLE_DONE pgs=1292 cs=0 l=1 rx=0x7f2af4008790 tx=0x7f2af400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:25.124 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.114+0000 7f2aebfff700 1 -- 172.21.15.42:0/460209554 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2af4011270 con 0x7f2af808df80 2019-12-05T05:55:25.124 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.114+0000 7f2afcc7e700 1 --2- 172.21.15.42:0/460209554 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2af808df80 0x7f2af809f2e0 secure :-1 s=THROTTLE_DONE pgs=1292 cs=0 l=1 rx=0x7f2af4008790 tx=0x7f2af400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:25.124 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.114+0000 7f2afcc7e700 1 --2- 172.21.15.42:0/460209554 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2af808df80 0x7f2af809f2e0 secure :-1 s=THROTTLE_DONE pgs=1292 cs=0 l=1 rx=0x7f2af4008790 tx=0x7f2af400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:25.124 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.114+0000 7f2aebfff700 1 -- 172.21.15.42:0/460209554 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2af4008bc0 con 0x7f2af808df80 2019-12-05T05:55:25.124 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.114+0000 7f2aebfff700 1 -- 172.21.15.42:0/460209554 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2af4011850 con 0x7f2af808df80 2019-12-05T05:55:25.125 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.114+0000 7f2afeee2700 1 -- 172.21.15.42:0/460209554 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2af808df80 msgr2=0x7f2af809f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:55:25.125 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.114+0000 7f2afeee2700 1 --2- 172.21.15.42:0/460209554 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2af808df80 0x7f2af809f2e0 secure :-1 s=READY pgs=1292 cs=0 l=1 rx=0x7f2af4008790 tx=0x7f2af400fbf0).stop 2019-12-05T05:55:25.125 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.118+0000 7f2afeee2700 1 -- 172.21.15.42:0/460209554 shutdown_connections 2019-12-05T05:55:25.125 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.118+0000 7f2afeee2700 1 --2- 172.21.15.42:0/460209554 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2af808df80 0x7f2af809f2e0 unknown :-1 s=CLOSED pgs=1292 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:55:25.125 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.118+0000 7f2afeee2700 1 -- 172.21.15.42:0/460209554 shutdown_connections 2019-12-05T05:55:25.125 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.118+0000 7f2afeee2700 1 -- 172.21.15.42:0/460209554 wait complete. 2019-12-05T05:55:25.125 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.118+0000 7f2afeee2700 1 -- 172.21.15.42:0/460209554 >> 172.21.15.42:0/460209554 conn(0x7f2af80aabe0 msgr2=0x7f2af80a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:55:25.128 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.118+0000 7f2afeee2700 1 Processor -- start 2019-12-05T05:55:25.128 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.118+0000 7f2afeee2700 1 -- start start 2019-12-05T05:55:25.128 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.122+0000 7f2afeee2700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2af81247f0 0x7f2af8144810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:55:25.128 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.122+0000 7f2afeee2700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2af80722c0 0x7f2af8074700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:55:25.129 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.122+0000 7f2afeee2700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2af808f1b0 con 0x7f2af8120a60 2019-12-05T05:55:25.129 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.122+0000 7f2afeee2700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2af808f4c0 con 0x7f2af80722c0 2019-12-05T05:55:25.129 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.122+0000 7f2afeee2700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2af808f760 con 0x7f2af81247f0 2019-12-05T05:55:25.129 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.122+0000 7f2afcc7e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2af81247f0 0x7f2af8144810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:55:25.129 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.122+0000 7f2afd47f700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f2af8120a60 0x7f2af8123ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:44124/0 (socket says 172.21.15.42:44124) 2019-12-05T05:55:25.129 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.122+0000 7f2afcc7e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2af81247f0 0x7f2af8144810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:44088/0 (socket says 172.21.15.42:44088) 2019-12-05T05:55:25.129 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.122+0000 7f2afd47f700 1 -- 172.21.15.42:0/1752758765 learned_addr learned my addr 172.21.15.42:0/1752758765 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:55:25.130 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.122+0000 7f2afdc80700 1 --2- 172.21.15.42:0/1752758765 >> v2:172.21.15.40:3300/0 conn(0x7f2af80722c0 0x7f2af8074700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:55:25.130 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.122+0000 7f2afcc7e700 1 -- 172.21.15.42:0/1752758765 >> v2:172.21.15.40:3300/0 conn(0x7f2af80722c0 msgr2=0x7f2af8074700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:55:25.130 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.122+0000 7f2afcc7e700 1 --2- 172.21.15.42:0/1752758765 >> v2:172.21.15.40:3300/0 conn(0x7f2af80722c0 0x7f2af8074700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:55:25.131 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.122+0000 7f2afcc7e700 1 -- 172.21.15.42:0/1752758765 >> v1:172.21.15.42:6789/0 conn(0x7f2af8120a60 legacy=0x7f2af8123ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:55:25.131 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.122+0000 7f2ae9ffb700 1 -- 172.21.15.42:0/1752758765 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2976937368 0 0) 0x7f2af808f1b0 con 0x7f2af8120a60 2019-12-05T05:55:25.131 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.122+0000 7f2afcc7e700 1 -- 172.21.15.42:0/1752758765 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2af8144d60 con 0x7f2af81247f0 2019-12-05T05:55:25.131 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.122+0000 7f2afcc7e700 1 --2- 172.21.15.42:0/1752758765 >> v2:172.21.15.42:3300/0 conn(0x7f2af81247f0 0x7f2af8144810 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f2af80a6610 tx=0x7f2af4008e70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:25.131 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.122+0000 7f2afcc7e700 1 --2- 172.21.15.42:0/1752758765 >> v2:172.21.15.42:3300/0 conn(0x7f2af81247f0 0x7f2af8144810 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f2af80a6610 tx=0x7f2af4008e70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:25.131 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.122+0000 7f2afcc7e700 1 --2- 172.21.15.42:0/1752758765 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2af81247f0 0x7f2af8144810 secure :-1 s=READY pgs=1697 cs=0 l=1 rx=0x7f2af80a6610 tx=0x7f2af4008e70).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:55:25.132 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.122+0000 7f2afcc7e700 1 --2- 172.21.15.42:0/1752758765 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2af81247f0 0x7f2af8144810 secure :-1 s=THROTTLE_DONE pgs=1697 cs=0 l=1 rx=0x7f2af80a6610 tx=0x7f2af4008e70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:25.132 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.122+0000 7f2afcc7e700 1 --2- 172.21.15.42:0/1752758765 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2af81247f0 0x7f2af8144810 secure :-1 s=THROTTLE_DONE pgs=1697 cs=0 l=1 rx=0x7f2af80a6610 tx=0x7f2af4008e70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:25.132 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.122+0000 7f2ae9ffb700 1 -- 172.21.15.42:0/1752758765 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2af400f360 con 0x7f2af81247f0 2019-12-05T05:55:25.132 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.122+0000 7f2afcc7e700 1 --2- 172.21.15.42:0/1752758765 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2af81247f0 0x7f2af8144810 secure :-1 s=THROTTLE_DONE pgs=1697 cs=0 l=1 rx=0x7f2af80a6610 tx=0x7f2af4008e70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:25.132 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.122+0000 7f2ae9ffb700 1 -- 172.21.15.42:0/1752758765 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2af4011470 con 0x7f2af81247f0 2019-12-05T05:55:25.132 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.122+0000 7f2afeee2700 1 -- 172.21.15.42:0/1752758765 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2af8141fe0 con 0x7f2af81247f0 2019-12-05T05:55:25.133 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.122+0000 7f2ae9ffb700 1 -- 172.21.15.42:0/1752758765 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2af4007340 con 0x7f2af81247f0 2019-12-05T05:55:25.133 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.122+0000 7f2afeee2700 1 -- 172.21.15.42:0/1752758765 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2af8118f80 con 0x7f2af81247f0 2019-12-05T05:55:25.133 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.126+0000 7f2afcc7e700 1 --2- 172.21.15.42:0/1752758765 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2af81247f0 0x7f2af8144810 secure :-1 s=THROTTLE_DONE pgs=1697 cs=0 l=1 rx=0x7f2af80a6610 tx=0x7f2af4008e70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:25.133 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.126+0000 7f2ae9ffb700 1 -- 172.21.15.42:0/1752758765 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f2af4005540 con 0x7f2af81247f0 2019-12-05T05:55:25.133 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.126+0000 7f2afcc7e700 1 --2- 172.21.15.42:0/1752758765 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2af81247f0 0x7f2af8144810 secure :-1 s=THROTTLE_DONE pgs=1697 cs=0 l=1 rx=0x7f2af80a6610 tx=0x7f2af4008e70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:25.134 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.126+0000 7f2ae9ffb700 1 --2- 172.21.15.42:0/1752758765 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2ad803acf0 0x7f2ad803d200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:55:25.134 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.126+0000 7f2ae9ffb700 1 -- 172.21.15.42:0/1752758765 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 9041+0+0 (secure 0 0 0) 0x7f2af404bb30 con 0x7f2af81247f0 2019-12-05T05:55:25.134 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.126+0000 7f2afd47f700 1 --2- 172.21.15.42:0/1752758765 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2ad803acf0 0x7f2ad803d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:55:25.134 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.126+0000 7f2afd47f700 1 --2- 172.21.15.42:0/1752758765 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2ad803acf0 0x7f2ad803d200 crc :-1 s=READY pgs=725 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:55:25.135 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.130+0000 7f2afeee2700 1 -- 172.21.15.42:0/1752758765 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2af805d110 con 0x7f2af81247f0 2019-12-05T05:55:25.139 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.138+0000 7f2afcc7e700 1 --2- 172.21.15.42:0/1752758765 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2af81247f0 0x7f2af8144810 secure :-1 s=THROTTLE_DONE pgs=1697 cs=0 l=1 rx=0x7f2af80a6610 tx=0x7f2af4008e70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:25.139 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.138+0000 7f2ae9ffb700 1 -- 172.21.15.42:0/1752758765 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f2af4032b90 con 0x7f2af81247f0 2019-12-05T05:55:25.311 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.306+0000 7f2afeee2700 1 -- 172.21.15.42:0/1752758765 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2af805b8c0 con 0x7f2af81247f0 2019-12-05T05:55:25.323 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.310+0000 7f2afcc7e700 1 --2- 172.21.15.42:0/1752758765 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2af81247f0 0x7f2af8144810 secure :-1 s=THROTTLE_DONE pgs=1697 cs=0 l=1 rx=0x7f2af80a6610 tx=0x7f2af4008e70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:25.323 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.310+0000 7f2ae9ffb700 1 -- 172.21.15.42:0/1752758765 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 208 v208) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f2af40307f0 con 0x7f2af81247f0 2019-12-05T05:55:25.323 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 208 2019-12-05T05:55:25.324 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.310+0000 7f2afeee2700 1 -- 172.21.15.42:0/1752758765 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2ad803acf0 msgr2=0x7f2ad803d200 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:55:25.324 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.310+0000 7f2afeee2700 1 --2- 172.21.15.42:0/1752758765 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2ad803acf0 0x7f2ad803d200 crc :-1 s=READY pgs=725 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:55:25.324 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.310+0000 7f2afeee2700 1 -- 172.21.15.42:0/1752758765 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2af81247f0 msgr2=0x7f2af8144810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:55:25.324 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.310+0000 7f2afeee2700 1 --2- 172.21.15.42:0/1752758765 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2af81247f0 0x7f2af8144810 secure :-1 s=READY pgs=1697 cs=0 l=1 rx=0x7f2af80a6610 tx=0x7f2af4008e70).stop 2019-12-05T05:55:25.325 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.310+0000 7f2afeee2700 1 -- 172.21.15.42:0/1752758765 shutdown_connections 2019-12-05T05:55:25.325 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.310+0000 7f2afeee2700 1 --2- 172.21.15.42:0/1752758765 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2ad803acf0 0x7f2ad803d200 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:55:25.325 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.310+0000 7f2afeee2700 1 --2- 172.21.15.42:0/1752758765 >> v2:172.21.15.40:3300/0 conn(0x7f2af80722c0 0x7f2af8074700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:55:25.325 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.310+0000 7f2afeee2700 1 --2- 172.21.15.42:0/1752758765 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2af81247f0 0x7f2af8144810 unknown :-1 s=CLOSED pgs=1697 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:55:25.325 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.310+0000 7f2afeee2700 1 -- 172.21.15.42:0/1752758765 shutdown_connections 2019-12-05T05:55:25.325 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.310+0000 7f2afeee2700 1 -- 172.21.15.42:0/1752758765 wait complete. 2019-12-05T05:55:25.326 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.310+0000 7f2afeee2700 1 -- 172.21.15.42:0/1752758765 >> 172.21.15.42:0/1752758765 conn(0x7f2af807e790 msgr2=0x7f2af80a9770 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:55:25.326 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:55:25.326 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":208,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6461,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/3549993215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3549993215},{"type":"v1","addr":"172.21.15.40:6833","nonce":3549993215}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":208}],"filesystems":[{"mdsmap":{"epoch":207,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:55:04.257118+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8614},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8614":{"gid":8614,"name":"a","rank":0,"incarnation":204,"state":"up:active","state_seq":32,"addr":"172.21.15.42:6835/2511123581","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2511123581},{"type":"v1","addr":"172.21.15.42:6835","nonce":2511123581}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:55:25.346 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.a (rank=0) 2019-12-05T05:55:25.347 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2019-12-05T05:55:25.347 INFO:teuthology.orchestra.run:waiting for 300 2019-12-05T05:55:25.361 INFO:tasks.ceph.mds.a:Stopped 2019-12-05T05:55:25.361 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:55:25.532 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.522+0000 7fb54fba5700 1 Processor -- start 2019-12-05T05:55:25.532 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.522+0000 7fb54fba5700 1 -- start start 2019-12-05T05:55:25.532 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.526+0000 7fb54fba5700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb54808df80 0x7fb54809f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:55:25.532 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.526+0000 7fb54fba5700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb54809f830 0x7fb54814efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:55:25.532 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.526+0000 7fb54fba5700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb54809fc60 con 0x7fb5480a7740 2019-12-05T05:55:25.533 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.526+0000 7fb54fba5700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5480a0b00 con 0x7fb54809f830 2019-12-05T05:55:25.533 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.526+0000 7fb54fba5700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb54808e5b0 con 0x7fb54808df80 2019-12-05T05:55:25.534 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.526+0000 7fb54e943700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb54809f830 0x7fb54814efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:55:25.534 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.526+0000 7fb54e142700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fb5480a7740 0x7fb54807e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:44134/0 (socket says 172.21.15.42:44134) 2019-12-05T05:55:25.534 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.526+0000 7fb54e142700 1 -- 172.21.15.42:0/1724813589 learned_addr learned my addr 172.21.15.42:0/1724813589 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:55:25.534 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.526+0000 7fb54d941700 1 --2- 172.21.15.42:0/1724813589 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb54808df80 0x7fb54809f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:55:25.534 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.526+0000 7fb54e943700 1 -- 172.21.15.42:0/1724813589 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb54808df80 msgr2=0x7fb54809f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:55:25.534 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.526+0000 7fb54e943700 1 --2- 172.21.15.42:0/1724813589 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb54808df80 0x7fb54809f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:55:25.535 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.526+0000 7fb54e943700 1 -- 172.21.15.42:0/1724813589 >> v1:172.21.15.42:6789/0 conn(0x7fb5480a7740 legacy=0x7fb54807e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:55:25.535 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.526+0000 7fb54d140700 1 -- 172.21.15.42:0/1724813589 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4244955490 0 0) 0x7fb54809fc60 con 0x7fb5480a7740 2019-12-05T05:55:25.536 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.526+0000 7fb54e943700 1 -- 172.21.15.42:0/1724813589 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5480a0710 con 0x7fb54809f830 2019-12-05T05:55:25.536 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.526+0000 7fb54e943700 1 --2- 172.21.15.42:0/1724813589 >> v2:172.21.15.42:3300/0 conn(0x7fb54809f830 0x7fb54814efe0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fb540008b10 tx=0x7fb54000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:25.536 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.526+0000 7fb54e943700 1 --2- 172.21.15.42:0/1724813589 >> v2:172.21.15.42:3300/0 conn(0x7fb54809f830 0x7fb54814efe0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fb540008b10 tx=0x7fb54000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:25.536 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.526+0000 7fb54e943700 1 --2- 172.21.15.42:0/1724813589 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb54809f830 0x7fb54814efe0 secure :-1 s=READY pgs=1699 cs=0 l=1 rx=0x7fb540008b10 tx=0x7fb54000fff0).ready entity=mon.1 client_cookie=b30d19cd760062c1 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:55:25.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.526+0000 7fb54e943700 1 --2- 172.21.15.42:0/1724813589 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb54809f830 0x7fb54814efe0 secure :-1 s=THROTTLE_DONE pgs=1699 cs=0 l=1 rx=0x7fb540008b10 tx=0x7fb54000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:25.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.526+0000 7fb54d140700 1 -- 172.21.15.42:0/1724813589 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb540010900 con 0x7fb54809f830 2019-12-05T05:55:25.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.526+0000 7fb54e943700 1 --2- 172.21.15.42:0/1724813589 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb54809f830 0x7fb54814efe0 secure :-1 s=THROTTLE_DONE pgs=1699 cs=0 l=1 rx=0x7fb540008b10 tx=0x7fb54000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:25.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.526+0000 7fb54d140700 1 -- 172.21.15.42:0/1724813589 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb540008b50 con 0x7fb54809f830 2019-12-05T05:55:25.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.526+0000 7fb54e943700 1 --2- 172.21.15.42:0/1724813589 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb54809f830 0x7fb54814efe0 secure :-1 s=THROTTLE_DONE pgs=1699 cs=0 l=1 rx=0x7fb540008b10 tx=0x7fb54000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:25.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.526+0000 7fb54d140700 1 -- 172.21.15.42:0/1724813589 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb5400110c0 con 0x7fb54809f830 2019-12-05T05:55:25.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.526+0000 7fb54fba5700 1 -- 172.21.15.42:0/1724813589 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb54809f830 msgr2=0x7fb54814efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:55:25.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.526+0000 7fb54fba5700 1 --2- 172.21.15.42:0/1724813589 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb54809f830 0x7fb54814efe0 secure :-1 s=READY pgs=1699 cs=0 l=1 rx=0x7fb540008b10 tx=0x7fb54000fff0).stop 2019-12-05T05:55:25.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.526+0000 7fb54fba5700 1 -- 172.21.15.42:0/1724813589 shutdown_connections 2019-12-05T05:55:25.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.526+0000 7fb54fba5700 1 --2- 172.21.15.42:0/1724813589 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb54809f830 0x7fb54814efe0 unknown :-1 s=CLOSED pgs=1699 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:55:25.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.526+0000 7fb54fba5700 1 --2- 172.21.15.42:0/1724813589 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb54808df80 0x7fb54809f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:55:25.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.530+0000 7fb54fba5700 1 -- 172.21.15.42:0/1724813589 shutdown_connections 2019-12-05T05:55:25.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.530+0000 7fb54fba5700 1 -- 172.21.15.42:0/1724813589 wait complete. 2019-12-05T05:55:25.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.530+0000 7fb54fba5700 1 -- 172.21.15.42:0/1724813589 >> 172.21.15.42:0/1724813589 conn(0x7fb5480aabe0 msgr2=0x7fb5480a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:55:25.542 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.530+0000 7fb54fba5700 1 Processor -- start 2019-12-05T05:55:25.542 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.530+0000 7fb54fba5700 1 -- start start 2019-12-05T05:55:25.543 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.534+0000 7fb54fba5700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb54808eeb0 0x7fb548121a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:55:25.543 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.534+0000 7fb54fba5700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb5481305d0 0x7fb548136e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:55:25.544 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.534+0000 7fb54fba5700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb548122590 con 0x7fb5481301a0 2019-12-05T05:55:25.544 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.534+0000 7fb54fba5700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5481222f0 con 0x7fb5481305d0 2019-12-05T05:55:25.544 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.534+0000 7fb54fba5700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb548121fa0 con 0x7fb54808eeb0 2019-12-05T05:55:25.544 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.534+0000 7fb54e142700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb54808eeb0 0x7fb548121a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:55:25.544 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.534+0000 7fb54e943700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb5481305d0 0x7fb548136e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:55:25.545 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.534+0000 7fb54d941700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fb5481301a0 0x7fb548117a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:53582/0 (socket says 172.21.15.42:53582) 2019-12-05T05:55:25.545 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.534+0000 7fb54d941700 1 -- 172.21.15.42:0/3956241963 learned_addr learned my addr 172.21.15.42:0/3956241963 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:55:25.545 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.534+0000 7fb54e142700 1 --2- 172.21.15.42:0/3956241963 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb54808eeb0 0x7fb548121a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:44568/0 (socket says 172.21.15.42:44568) 2019-12-05T05:55:25.545 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.534+0000 7fb53effd700 1 -- 172.21.15.42:0/3956241963 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 736450469 0 0) 0x7fb548122590 con 0x7fb5481301a0 2019-12-05T05:55:25.545 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.534+0000 7fb53effd700 1 -- 172.21.15.42:0/3956241963 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb5300030b0 con 0x7fb5481301a0 2019-12-05T05:55:25.545 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.534+0000 7fb54e142700 1 -- 172.21.15.42:0/3956241963 >> v2:172.21.15.42:3300/0 conn(0x7fb5481305d0 msgr2=0x7fb548136e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:55:25.545 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.534+0000 7fb54e142700 1 --2- 172.21.15.42:0/3956241963 >> v2:172.21.15.42:3300/0 conn(0x7fb5481305d0 0x7fb548136e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:55:25.545 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.534+0000 7fb54e142700 1 -- 172.21.15.42:0/3956241963 >> v1:172.21.15.40:6789/0 conn(0x7fb5481301a0 legacy=0x7fb548117a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:55:25.546 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.534+0000 7fb54e142700 1 -- 172.21.15.42:0/3956241963 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb548122cb0 con 0x7fb54808eeb0 2019-12-05T05:55:25.546 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.534+0000 7fb54e142700 1 --2- 172.21.15.42:0/3956241963 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb54808eeb0 0x7fb548121a50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fb548117460 tx=0x7fb5440108a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:25.547 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.534+0000 7fb54e142700 1 --2- 172.21.15.42:0/3956241963 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb54808eeb0 0x7fb548121a50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fb548117460 tx=0x7fb5440108a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:25.547 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.534+0000 7fb54e142700 1 --2- 172.21.15.42:0/3956241963 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb54808eeb0 0x7fb548121a50 secure :-1 s=READY pgs=382 cs=0 l=1 rx=0x7fb548117460 tx=0x7fb5440108a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:55:25.547 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.534+0000 7fb54e142700 1 --2- 172.21.15.42:0/3956241963 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb54808eeb0 0x7fb548121a50 secure :-1 s=THROTTLE_DONE pgs=382 cs=0 l=1 rx=0x7fb548117460 tx=0x7fb5440108a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:25.547 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.534+0000 7fb53effd700 1 -- 172.21.15.42:0/3956241963 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb544011590 con 0x7fb54808eeb0 2019-12-05T05:55:25.547 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.534+0000 7fb54e142700 1 --2- 172.21.15.42:0/3956241963 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb54808eeb0 0x7fb548121a50 secure :-1 s=THROTTLE_DONE pgs=382 cs=0 l=1 rx=0x7fb548117460 tx=0x7fb5440108a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:25.547 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.534+0000 7fb53effd700 1 -- 172.21.15.42:0/3956241963 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5440111b0 con 0x7fb54808eeb0 2019-12-05T05:55:25.547 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.534+0000 7fb54e142700 1 --2- 172.21.15.42:0/3956241963 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb54808eeb0 0x7fb548121a50 secure :-1 s=THROTTLE_DONE pgs=382 cs=0 l=1 rx=0x7fb548117460 tx=0x7fb5440108a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:25.548 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.534+0000 7fb54fba5700 1 -- 172.21.15.42:0/3956241963 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb548118210 con 0x7fb54808eeb0 2019-12-05T05:55:25.548 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.534+0000 7fb53effd700 1 -- 172.21.15.42:0/3956241963 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb544011b70 con 0x7fb54808eeb0 2019-12-05T05:55:25.548 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.534+0000 7fb54fba5700 1 -- 172.21.15.42:0/3956241963 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5480a97e0 con 0x7fb54808eeb0 2019-12-05T05:55:25.548 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.534+0000 7fb54e142700 1 --2- 172.21.15.42:0/3956241963 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb54808eeb0 0x7fb548121a50 secure :-1 s=THROTTLE_DONE pgs=382 cs=0 l=1 rx=0x7fb548117460 tx=0x7fb5440108a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:25.548 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.538+0000 7fb53effd700 1 -- 172.21.15.42:0/3956241963 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fb54400b330 con 0x7fb54808eeb0 2019-12-05T05:55:25.548 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.538+0000 7fb54e142700 1 --2- 172.21.15.42:0/3956241963 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb54808eeb0 0x7fb548121a50 secure :-1 s=THROTTLE_DONE pgs=382 cs=0 l=1 rx=0x7fb548117460 tx=0x7fb5440108a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:25.548 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.538+0000 7fb53effd700 1 --2- 172.21.15.42:0/3956241963 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb53003b280 0x7fb53003d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:55:25.549 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.538+0000 7fb53effd700 1 -- 172.21.15.42:0/3956241963 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 9041+0+0 (secure 0 0 0) 0x7fb54404b220 con 0x7fb54808eeb0 2019-12-05T05:55:25.549 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.538+0000 7fb54d941700 1 --2- 172.21.15.42:0/3956241963 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb53003b280 0x7fb53003d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:55:25.549 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.538+0000 7fb54d941700 1 --2- 172.21.15.42:0/3956241963 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb53003b280 0x7fb53003d790 crc :-1 s=READY pgs=726 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:55:25.549 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.538+0000 7fb54fba5700 1 -- 172.21.15.42:0/3956241963 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb54805d110 con 0x7fb54808eeb0 2019-12-05T05:55:25.551 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.546+0000 7fb54e142700 1 --2- 172.21.15.42:0/3956241963 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb54808eeb0 0x7fb548121a50 secure :-1 s=THROTTLE_DONE pgs=382 cs=0 l=1 rx=0x7fb548117460 tx=0x7fb5440108a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:25.551 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.550+0000 7fb53effd700 1 -- 172.21.15.42:0/3956241963 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fb5440321e0 con 0x7fb54808eeb0 2019-12-05T05:55:25.718 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.714+0000 7fb54fba5700 1 -- 172.21.15.42:0/3956241963 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb54805b8c0 con 0x7fb54808eeb0 2019-12-05T05:55:25.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.718+0000 7fb54e142700 1 --2- 172.21.15.42:0/3956241963 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb54808eeb0 0x7fb548121a50 secure :-1 s=THROTTLE_DONE pgs=382 cs=0 l=1 rx=0x7fb548117460 tx=0x7fb5440108a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:25.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.718+0000 7fb53effd700 1 -- 172.21.15.42:0/3956241963 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 208 v208) v1 ==== 95+0+2112 (secure 0 0 0) 0x7fb544032440 con 0x7fb54808eeb0 2019-12-05T05:55:25.722 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 208 2019-12-05T05:55:25.722 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:55:25.723 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":208,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6461,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/3549993215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3549993215},{"type":"v1","addr":"172.21.15.40:6833","nonce":3549993215}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":208}],"filesystems":[{"mdsmap":{"epoch":207,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:55:04.257118+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8614},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8614":{"gid":8614,"name":"a","rank":0,"incarnation":204,"state":"up:active","state_seq":32,"addr":"172.21.15.42:6835/2511123581","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2511123581},{"type":"v1","addr":"172.21.15.42:6835","nonce":2511123581}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:55:25.725 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.722+0000 7fb54fba5700 1 -- 172.21.15.42:0/3956241963 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb53003b280 msgr2=0x7fb53003d790 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:55:25.725 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.722+0000 7fb54fba5700 1 --2- 172.21.15.42:0/3956241963 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb53003b280 0x7fb53003d790 crc :-1 s=READY pgs=726 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:55:25.726 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.722+0000 7fb54fba5700 1 -- 172.21.15.42:0/3956241963 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb54808eeb0 msgr2=0x7fb548121a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:55:25.726 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.722+0000 7fb54fba5700 1 --2- 172.21.15.42:0/3956241963 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb54808eeb0 0x7fb548121a50 secure :-1 s=READY pgs=382 cs=0 l=1 rx=0x7fb548117460 tx=0x7fb5440108a0).stop 2019-12-05T05:55:25.726 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.722+0000 7fb54fba5700 1 -- 172.21.15.42:0/3956241963 shutdown_connections 2019-12-05T05:55:25.726 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.722+0000 7fb54fba5700 1 --2- 172.21.15.42:0/3956241963 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb53003b280 0x7fb53003d790 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:55:25.726 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.722+0000 7fb54fba5700 1 --2- 172.21.15.42:0/3956241963 >> v2:172.21.15.42:3300/0 conn(0x7fb5481305d0 0x7fb548136e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:55:25.726 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.722+0000 7fb54fba5700 1 --2- 172.21.15.42:0/3956241963 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb54808eeb0 0x7fb548121a50 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:55:25.726 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.722+0000 7fb54fba5700 1 -- 172.21.15.42:0/3956241963 shutdown_connections 2019-12-05T05:55:25.727 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.722+0000 7fb54fba5700 1 -- 172.21.15.42:0/3956241963 wait complete. 2019-12-05T05:55:25.727 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:25.722+0000 7fb54fba5700 1 -- 172.21.15.42:0/3956241963 >> 172.21.15.42:0/3956241963 conn(0x7fb54809ffb0 msgr2=0x7fb5481186b0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:55:25.735 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T05:55:27.737 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:55:27.843 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:27.834+0000 7fc2196d5700 1 Processor -- start 2019-12-05T05:55:27.843 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:27.834+0000 7fc2196d5700 1 -- start start 2019-12-05T05:55:27.850 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:27.838+0000 7fc2196d5700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc21408df80 0x7fc21409f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:55:27.850 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:27.838+0000 7fc2196d5700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc21409f830 0x7fc21414efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:55:27.850 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:27.838+0000 7fc2196d5700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc21409fc60 con 0x7fc2140a7740 2019-12-05T05:55:27.851 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:27.838+0000 7fc2196d5700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2140a0b00 con 0x7fc21408df80 2019-12-05T05:55:27.851 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:27.838+0000 7fc2196d5700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc21408e5b0 con 0x7fc21409f830 2019-12-05T05:55:27.851 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:27.838+0000 7fc212ffd700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc21408df80 0x7fc21409f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:55:27.851 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:27.838+0000 7fc212ffd700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc21408df80 0x7fc21409f2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:44116/0 (socket says 172.21.15.42:44116) 2019-12-05T05:55:27.851 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:27.838+0000 7fc212ffd700 1 -- 172.21.15.42:0/3060599353 learned_addr learned my addr 172.21.15.42:0/3060599353 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:55:27.851 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:27.838+0000 7fc2137fe700 1 --1- 172.21.15.42:0/3060599353 >> v1:172.21.15.40:6789/0 conn(0x7fc2140a7740 0x7fc21407e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:53594/0 (socket says 172.21.15.42:53594) 2019-12-05T05:55:27.852 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:27.838+0000 7fc213fff700 1 --2- 172.21.15.42:0/3060599353 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc21409f830 0x7fc21414efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:55:27.852 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:27.838+0000 7fc212ffd700 1 -- 172.21.15.42:0/3060599353 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc21409f830 msgr2=0x7fc21414efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:55:27.852 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:27.838+0000 7fc212ffd700 1 --2- 172.21.15.42:0/3060599353 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc21409f830 0x7fc21414efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:55:27.852 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:27.838+0000 7fc212ffd700 1 -- 172.21.15.42:0/3060599353 >> v1:172.21.15.40:6789/0 conn(0x7fc2140a7740 legacy=0x7fc21407e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:55:27.852 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:27.838+0000 7fc2127fc700 1 -- 172.21.15.42:0/3060599353 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 714441739 0 0) 0x7fc21409fc60 con 0x7fc2140a7740 2019-12-05T05:55:27.852 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:27.838+0000 7fc212ffd700 1 -- 172.21.15.42:0/3060599353 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2140a0710 con 0x7fc21408df80 2019-12-05T05:55:27.852 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:27.838+0000 7fc212ffd700 1 --2- 172.21.15.42:0/3060599353 >> v2:172.21.15.42:3300/0 conn(0x7fc21408df80 0x7fc21409f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fc208008790 tx=0x7fc20800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:27.853 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:27.838+0000 7fc212ffd700 1 --2- 172.21.15.42:0/3060599353 >> v2:172.21.15.42:3300/0 conn(0x7fc21408df80 0x7fc21409f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fc208008790 tx=0x7fc20800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:27.853 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:27.838+0000 7fc212ffd700 1 --2- 172.21.15.42:0/3060599353 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc21408df80 0x7fc21409f2e0 secure :-1 s=READY pgs=1700 cs=0 l=1 rx=0x7fc208008790 tx=0x7fc20800fd30).ready entity=mon.1 client_cookie=e2ba6e5d70613895 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:55:27.853 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:27.838+0000 7fc212ffd700 1 --2- 172.21.15.42:0/3060599353 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc21408df80 0x7fc21409f2e0 secure :-1 s=THROTTLE_DONE pgs=1700 cs=0 l=1 rx=0x7fc208008790 tx=0x7fc20800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:27.853 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:27.838+0000 7fc2127fc700 1 -- 172.21.15.42:0/3060599353 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc208008c90 con 0x7fc21408df80 2019-12-05T05:55:27.854 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:27.838+0000 7fc212ffd700 1 --2- 172.21.15.42:0/3060599353 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc21408df80 0x7fc21409f2e0 secure :-1 s=THROTTLE_DONE pgs=1700 cs=0 l=1 rx=0x7fc208008790 tx=0x7fc20800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:27.854 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:27.838+0000 7fc212ffd700 1 --2- 172.21.15.42:0/3060599353 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc21408df80 0x7fc21409f2e0 secure :-1 s=THROTTLE_DONE pgs=1700 cs=0 l=1 rx=0x7fc208008790 tx=0x7fc20800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:27.854 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:27.838+0000 7fc2127fc700 1 -- 172.21.15.42:0/3060599353 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2080088b0 con 0x7fc21408df80 2019-12-05T05:55:27.854 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:27.838+0000 7fc2127fc700 1 -- 172.21.15.42:0/3060599353 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc208011b10 con 0x7fc21408df80 2019-12-05T05:55:27.854 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:27.838+0000 7fc2196d5700 1 -- 172.21.15.42:0/3060599353 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc21408df80 msgr2=0x7fc21409f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:55:27.854 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:27.838+0000 7fc2196d5700 1 --2- 172.21.15.42:0/3060599353 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc21408df80 0x7fc21409f2e0 secure :-1 s=READY pgs=1700 cs=0 l=1 rx=0x7fc208008790 tx=0x7fc20800fd30).stop 2019-12-05T05:55:27.855 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:27.838+0000 7fc2196d5700 1 -- 172.21.15.42:0/3060599353 shutdown_connections 2019-12-05T05:55:27.856 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:27.838+0000 7fc2196d5700 1 --2- 172.21.15.42:0/3060599353 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc21409f830 0x7fc21414efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:55:27.856 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:27.838+0000 7fc2196d5700 1 --2- 172.21.15.42:0/3060599353 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc21408df80 0x7fc21409f2e0 unknown :-1 s=CLOSED pgs=1700 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:55:27.856 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:27.838+0000 7fc2196d5700 1 -- 172.21.15.42:0/3060599353 shutdown_connections 2019-12-05T05:55:27.856 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:27.838+0000 7fc2196d5700 1 -- 172.21.15.42:0/3060599353 wait complete. 2019-12-05T05:55:27.856 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:27.838+0000 7fc2196d5700 1 -- 172.21.15.42:0/3060599353 >> 172.21.15.42:0/3060599353 conn(0x7fc2140aabe0 msgr2=0x7fc2140a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:55:27.862 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:27.842+0000 7fc2196d5700 1 Processor -- start 2019-12-05T05:55:27.862 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:27.842+0000 7fc2196d5700 1 -- start start 2019-12-05T05:55:27.862 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:27.842+0000 7fc2196d5700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc21411b950 0x7fc2140a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:55:27.862 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:27.842+0000 7fc2196d5700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc2140a52e0 0x7fc2140a5710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:55:27.862 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:27.842+0000 7fc2196d5700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2140a6720 con 0x7fc2141308d0 2019-12-05T05:55:27.862 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:27.842+0000 7fc2196d5700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2140a6a30 con 0x7fc21411b950 2019-12-05T05:55:27.863 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:27.842+0000 7fc2196d5700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc214130390 con 0x7fc2140a52e0 2019-12-05T05:55:27.863 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:27.846+0000 7fc2137fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc21411b950 0x7fc2140a61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:55:27.863 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:27.846+0000 7fc2137fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc21411b950 0x7fc2140a61d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:44122/0 (socket says 172.21.15.42:44122) 2019-12-05T05:55:27.863 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:27.846+0000 7fc2137fe700 1 -- 172.21.15.42:0/308786854 learned_addr learned my addr 172.21.15.42:0/308786854 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:55:27.863 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:27.846+0000 7fc212ffd700 1 --2- 172.21.15.42:0/308786854 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc2140a52e0 0x7fc2140a5710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:55:27.863 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:27.846+0000 7fc212ffd700 1 -- 172.21.15.42:0/308786854 >> v2:172.21.15.42:3300/0 conn(0x7fc21411b950 msgr2=0x7fc2140a61d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:55:27.864 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:27.846+0000 7fc212ffd700 1 --2- 172.21.15.42:0/308786854 >> v2:172.21.15.42:3300/0 conn(0x7fc21411b950 0x7fc2140a61d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:55:27.864 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:27.846+0000 7fc212ffd700 1 -- 172.21.15.42:0/308786854 >> v1:172.21.15.42:6789/0 conn(0x7fc2141308d0 legacy=0x7fc21407c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:55:27.864 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:27.846+0000 7fc1fbfff700 1 -- 172.21.15.42:0/308786854 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1722136583 0 0) 0x7fc2140a6720 con 0x7fc2141308d0 2019-12-05T05:55:27.865 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:27.846+0000 7fc212ffd700 1 -- 172.21.15.42:0/308786854 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2140a4be0 con 0x7fc2140a52e0 2019-12-05T05:55:27.865 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:27.846+0000 7fc2137fe700 1 --2- 172.21.15.42:0/308786854 >> v2:172.21.15.42:3300/0 conn(0x7fc21411b950 0x7fc2140a61d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:55:27.865 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:27.846+0000 7fc212ffd700 1 --2- 172.21.15.42:0/308786854 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc2140a52e0 0x7fc2140a5710 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fc208008790 tx=0x7fc208011920).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:27.865 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:27.846+0000 7fc212ffd700 1 --2- 172.21.15.42:0/308786854 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc2140a52e0 0x7fc2140a5710 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fc208008790 tx=0x7fc208011920).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:27.866 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:27.846+0000 7fc212ffd700 1 --2- 172.21.15.42:0/308786854 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc2140a52e0 0x7fc2140a5710 secure :-1 s=READY pgs=383 cs=0 l=1 rx=0x7fc208008790 tx=0x7fc208011920).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:55:27.866 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:27.846+0000 7fc212ffd700 1 --2- 172.21.15.42:0/308786854 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc2140a52e0 0x7fc2140a5710 secure :-1 s=THROTTLE_DONE pgs=383 cs=0 l=1 rx=0x7fc208008790 tx=0x7fc208011920).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:27.866 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:27.846+0000 7fc1fbfff700 1 -- 172.21.15.42:0/308786854 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc20800f470 con 0x7fc2140a52e0 2019-12-05T05:55:27.866 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:27.846+0000 7fc212ffd700 1 --2- 172.21.15.42:0/308786854 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc2140a52e0 0x7fc2140a5710 secure :-1 s=THROTTLE_DONE pgs=383 cs=0 l=1 rx=0x7fc208008790 tx=0x7fc208011920).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:27.866 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:27.846+0000 7fc1fbfff700 1 -- 172.21.15.42:0/308786854 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc208011730 con 0x7fc2140a52e0 2019-12-05T05:55:27.866 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:27.846+0000 7fc212ffd700 1 --2- 172.21.15.42:0/308786854 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc2140a52e0 0x7fc2140a5710 secure :-1 s=THROTTLE_DONE pgs=383 cs=0 l=1 rx=0x7fc208008790 tx=0x7fc208011920).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:27.866 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:27.846+0000 7fc2196d5700 1 -- 172.21.15.42:0/308786854 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc214119cb0 con 0x7fc2140a52e0 2019-12-05T05:55:27.867 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:27.846+0000 7fc1fbfff700 1 -- 172.21.15.42:0/308786854 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc208005730 con 0x7fc2140a52e0 2019-12-05T05:55:27.867 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:27.846+0000 7fc2196d5700 1 -- 172.21.15.42:0/308786854 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc214130d00 con 0x7fc2140a52e0 2019-12-05T05:55:27.867 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:27.846+0000 7fc212ffd700 1 --2- 172.21.15.42:0/308786854 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc2140a52e0 0x7fc2140a5710 secure :-1 s=THROTTLE_DONE pgs=383 cs=0 l=1 rx=0x7fc208008790 tx=0x7fc208011920).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:27.867 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:27.850+0000 7fc1fbfff700 1 -- 172.21.15.42:0/308786854 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fc2080059d0 con 0x7fc2140a52e0 2019-12-05T05:55:27.867 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:27.850+0000 7fc212ffd700 1 --2- 172.21.15.42:0/308786854 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc2140a52e0 0x7fc2140a5710 secure :-1 s=THROTTLE_DONE pgs=383 cs=0 l=1 rx=0x7fc208008790 tx=0x7fc208011920).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:27.867 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:27.850+0000 7fc1fbfff700 1 --2- 172.21.15.42:0/308786854 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc1f403acc0 0x7fc1f403d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:55:27.868 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:27.850+0000 7fc1fbfff700 1 -- 172.21.15.42:0/308786854 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 9041+0+0 (secure 0 0 0) 0x7fc20804b710 con 0x7fc2140a52e0 2019-12-05T05:55:27.868 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:27.850+0000 7fc2137fe700 1 --2- 172.21.15.42:0/308786854 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc1f403acc0 0x7fc1f403d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:55:27.868 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:27.850+0000 7fc2137fe700 1 --2- 172.21.15.42:0/308786854 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc1f403acc0 0x7fc1f403d1d0 crc :-1 s=READY pgs=728 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:55:27.868 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:27.850+0000 7fc2196d5700 1 -- 172.21.15.42:0/308786854 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc21405d110 con 0x7fc2140a52e0 2019-12-05T05:55:27.869 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:27.862+0000 7fc212ffd700 1 --2- 172.21.15.42:0/308786854 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc2140a52e0 0x7fc2140a5710 secure :-1 s=THROTTLE_DONE pgs=383 cs=0 l=1 rx=0x7fc208008790 tx=0x7fc208011920).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:27.869 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:27.862+0000 7fc1fbfff700 1 -- 172.21.15.42:0/308786854 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fc208032770 con 0x7fc2140a52e0 2019-12-05T05:55:28.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:28.026+0000 7fc2196d5700 1 -- 172.21.15.42:0/308786854 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc21405b8c0 con 0x7fc2140a52e0 2019-12-05T05:55:28.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:28.030+0000 7fc212ffd700 1 --2- 172.21.15.42:0/308786854 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc2140a52e0 0x7fc2140a5710 secure :-1 s=THROTTLE_DONE pgs=383 cs=0 l=1 rx=0x7fc208008790 tx=0x7fc208011920).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:28.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:28.030+0000 7fc1fbfff700 1 -- 172.21.15.42:0/308786854 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 208 v208) v1 ==== 95+0+2112 (secure 0 0 0) 0x7fc2080329d0 con 0x7fc2140a52e0 2019-12-05T05:55:28.033 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 208 2019-12-05T05:55:28.033 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:55:28.034 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":208,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6461,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/3549993215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3549993215},{"type":"v1","addr":"172.21.15.40:6833","nonce":3549993215}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":208}],"filesystems":[{"mdsmap":{"epoch":207,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:55:04.257118+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8614},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8614":{"gid":8614,"name":"a","rank":0,"incarnation":204,"state":"up:active","state_seq":32,"addr":"172.21.15.42:6835/2511123581","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2511123581},{"type":"v1","addr":"172.21.15.42:6835","nonce":2511123581}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:55:28.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:28.034+0000 7fc2196d5700 1 -- 172.21.15.42:0/308786854 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc1f403acc0 msgr2=0x7fc1f403d1d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:55:28.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:28.034+0000 7fc2196d5700 1 --2- 172.21.15.42:0/308786854 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc1f403acc0 0x7fc1f403d1d0 crc :-1 s=READY pgs=728 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:55:28.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:28.034+0000 7fc2196d5700 1 -- 172.21.15.42:0/308786854 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc2140a52e0 msgr2=0x7fc2140a5710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:55:28.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:28.034+0000 7fc2196d5700 1 --2- 172.21.15.42:0/308786854 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc2140a52e0 0x7fc2140a5710 secure :-1 s=READY pgs=383 cs=0 l=1 rx=0x7fc208008790 tx=0x7fc208011920).stop 2019-12-05T05:55:28.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:28.034+0000 7fc2196d5700 1 -- 172.21.15.42:0/308786854 shutdown_connections 2019-12-05T05:55:28.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:28.034+0000 7fc2196d5700 1 --2- 172.21.15.42:0/308786854 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc1f403acc0 0x7fc1f403d1d0 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:55:28.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:28.034+0000 7fc2196d5700 1 --2- 172.21.15.42:0/308786854 >> v2:172.21.15.42:3300/0 conn(0x7fc21411b950 0x7fc2140a61d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:55:28.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:28.034+0000 7fc2196d5700 1 --2- 172.21.15.42:0/308786854 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc2140a52e0 0x7fc2140a5710 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:55:28.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:28.034+0000 7fc2196d5700 1 -- 172.21.15.42:0/308786854 shutdown_connections 2019-12-05T05:55:28.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:28.034+0000 7fc2196d5700 1 -- 172.21.15.42:0/308786854 wait complete. 2019-12-05T05:55:28.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:28.034+0000 7fc2196d5700 1 -- 172.21.15.42:0/308786854 >> 172.21.15.42:0/308786854 conn(0x7fc2140a8120 msgr2=0x7fc21407e790 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:55:28.053 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T05:55:30.055 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:55:30.167 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:30.158+0000 7f0f3dac8700 1 Processor -- start 2019-12-05T05:55:30.167 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:30.158+0000 7f0f3dac8700 1 -- start start 2019-12-05T05:55:30.167 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:30.158+0000 7f0f3dac8700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f0f3808df80 0x7f0f3809f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:55:30.167 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:30.158+0000 7f0f3dac8700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f3809fc60 con 0x7f0f3809f830 2019-12-05T05:55:30.167 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:30.158+0000 7f0f3dac8700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f380a0710 con 0x7f0f380a7740 2019-12-05T05:55:30.168 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:30.158+0000 7f0f3dac8700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f380a0b00 con 0x7f0f3808df80 2019-12-05T05:55:30.168 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:30.158+0000 7f0f37fff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f0f380a7740 0x7f0f3807e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:44170/0 (socket says 172.21.15.42:44170) 2019-12-05T05:55:30.168 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:30.158+0000 7f0f37fff700 1 -- 172.21.15.42:0/966494115 learned_addr learned my addr 172.21.15.42:0/966494115 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:55:30.168 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:30.158+0000 7f0f2f7fe700 1 --2- 172.21.15.42:0/966494115 >> v2:172.21.15.40:3300/0 conn(0x7f0f3808df80 0x7f0f3809f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:55:30.168 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:30.162+0000 7f0f377fe700 1 -- 172.21.15.42:0/966494115 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4067303686 0 0) 0x7f0f380a0710 con 0x7f0f380a7740 2019-12-05T05:55:30.168 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:30.162+0000 7f0f377fe700 1 -- 172.21.15.42:0/966494115 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f240030b0 con 0x7f0f380a7740 2019-12-05T05:55:30.169 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:30.162+0000 7f0f2f7fe700 1 -- 172.21.15.42:0/966494115 >> v1:172.21.15.42:6789/0 conn(0x7f0f380a7740 legacy=0x7f0f3807e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:55:30.169 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:30.162+0000 7f0f2f7fe700 1 -- 172.21.15.42:0/966494115 >> v1:172.21.15.40:6789/0 conn(0x7f0f3809f830 legacy=0x7f0f3814efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:55:30.169 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:30.162+0000 7f0f2f7fe700 1 -- 172.21.15.42:0/966494115 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f3807ef20 con 0x7f0f3808df80 2019-12-05T05:55:30.169 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:30.162+0000 7f0f2f7fe700 1 --2- 172.21.15.42:0/966494115 >> v2:172.21.15.40:3300/0 conn(0x7f0f3808df80 0x7f0f3809f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f0f20008790 tx=0x7f0f2000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:30.169 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:30.162+0000 7f0f2f7fe700 1 --2- 172.21.15.42:0/966494115 >> v2:172.21.15.40:3300/0 conn(0x7f0f3808df80 0x7f0f3809f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f0f20008790 tx=0x7f0f2000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:30.169 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:30.162+0000 7f0f2f7fe700 1 --2- 172.21.15.42:0/966494115 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0f3808df80 0x7f0f3809f2e0 secure :-1 s=READY pgs=1296 cs=0 l=1 rx=0x7f0f20008790 tx=0x7f0f2000fbf0).ready entity=mon.0 client_cookie=820c5a8569e68a62 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:55:30.169 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:30.162+0000 7f0f2f7fe700 1 --2- 172.21.15.42:0/966494115 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0f3808df80 0x7f0f3809f2e0 secure :-1 s=THROTTLE_DONE pgs=1296 cs=0 l=1 rx=0x7f0f20008790 tx=0x7f0f2000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:30.170 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:30.162+0000 7f0f377fe700 1 -- 172.21.15.42:0/966494115 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0f20011270 con 0x7f0f3808df80 2019-12-05T05:55:30.170 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:30.162+0000 7f0f2f7fe700 1 --2- 172.21.15.42:0/966494115 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0f3808df80 0x7f0f3809f2e0 secure :-1 s=THROTTLE_DONE pgs=1296 cs=0 l=1 rx=0x7f0f20008790 tx=0x7f0f2000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:30.170 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:30.162+0000 7f0f377fe700 1 -- 172.21.15.42:0/966494115 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f20008bc0 con 0x7f0f3808df80 2019-12-05T05:55:30.170 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:30.162+0000 7f0f2f7fe700 1 --2- 172.21.15.42:0/966494115 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0f3808df80 0x7f0f3809f2e0 secure :-1 s=THROTTLE_DONE pgs=1296 cs=0 l=1 rx=0x7f0f20008790 tx=0x7f0f2000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:30.170 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:30.162+0000 7f0f3dac8700 1 -- 172.21.15.42:0/966494115 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0f3808df80 msgr2=0x7f0f3809f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:55:30.170 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:30.162+0000 7f0f377fe700 1 -- 172.21.15.42:0/966494115 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0f20011850 con 0x7f0f3808df80 2019-12-05T05:55:30.170 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:30.162+0000 7f0f3dac8700 1 --2- 172.21.15.42:0/966494115 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0f3808df80 0x7f0f3809f2e0 secure :-1 s=READY pgs=1296 cs=0 l=1 rx=0x7f0f20008790 tx=0x7f0f2000fbf0).stop 2019-12-05T05:55:30.171 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:30.162+0000 7f0f3dac8700 1 -- 172.21.15.42:0/966494115 shutdown_connections 2019-12-05T05:55:30.171 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:30.162+0000 7f0f3dac8700 1 --2- 172.21.15.42:0/966494115 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0f3808df80 0x7f0f3809f2e0 unknown :-1 s=CLOSED pgs=1296 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:55:30.171 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:30.162+0000 7f0f3dac8700 1 -- 172.21.15.42:0/966494115 shutdown_connections 2019-12-05T05:55:30.171 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:30.162+0000 7f0f3dac8700 1 -- 172.21.15.42:0/966494115 wait complete. 2019-12-05T05:55:30.171 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:30.162+0000 7f0f3dac8700 1 -- 172.21.15.42:0/966494115 >> 172.21.15.42:0/966494115 conn(0x7f0f380aabe0 msgr2=0x7f0f380a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:55:30.174 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:30.166+0000 7f0f3dac8700 1 Processor -- start 2019-12-05T05:55:30.174 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:30.166+0000 7f0f3dac8700 1 -- start start 2019-12-05T05:55:30.174 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:30.166+0000 7f0f3dac8700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0f38147ff0 0x7f0f380a1c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:55:30.174 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:30.166+0000 7f0f3dac8700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f381479b0 con 0x7f0f3807d890 2019-12-05T05:55:30.175 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:30.166+0000 7f0f3dac8700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f380a2160 con 0x7f0f3807dcc0 2019-12-05T05:55:30.175 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:30.166+0000 7f0f3dac8700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f38145860 con 0x7f0f38147ff0 2019-12-05T05:55:30.175 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:30.166+0000 7f0f37fff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0f38147ff0 0x7f0f380a1c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:55:30.175 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:30.166+0000 7f0f37fff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0f38147ff0 0x7f0f380a1c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:44138/0 (socket says 172.21.15.42:44138) 2019-12-05T05:55:30.175 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:30.170+0000 7f0f37fff700 1 -- 172.21.15.42:0/4074555021 learned_addr learned my addr 172.21.15.42:0/4074555021 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:55:30.175 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:30.170+0000 7f0f357fa700 1 -- 172.21.15.42:0/4074555021 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1082275849 0 0) 0x7f0f381479b0 con 0x7f0f3807d890 2019-12-05T05:55:30.176 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:30.170+0000 7f0f357fa700 1 -- 172.21.15.42:0/4074555021 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f1c0030b0 con 0x7f0f3807d890 2019-12-05T05:55:30.176 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:30.170+0000 7f0f37fff700 1 -- 172.21.15.42:0/4074555021 >> v1:172.21.15.42:6789/0 conn(0x7f0f3807dcc0 legacy=0x7f0f3812fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:55:30.176 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:30.170+0000 7f0f357fa700 1 -- 172.21.15.42:0/4074555021 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3511795841 0 0) 0x7f0f380a2160 con 0x7f0f3807dcc0 2019-12-05T05:55:30.176 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:30.170+0000 7f0f37fff700 1 -- 172.21.15.42:0/4074555021 >> v1:172.21.15.40:6789/0 conn(0x7f0f3807d890 legacy=0x7f0f380a2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:55:30.176 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:30.170+0000 7f0f37fff700 1 -- 172.21.15.42:0/4074555021 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f380a3db0 con 0x7f0f38147ff0 2019-12-05T05:55:30.177 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:30.170+0000 7f0f37fff700 1 --2- 172.21.15.42:0/4074555021 >> v2:172.21.15.42:3300/0 conn(0x7f0f38147ff0 0x7f0f380a1c10 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f0f280013e0 tx=0x7f0f2800c400).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:30.177 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:30.170+0000 7f0f37fff700 1 --2- 172.21.15.42:0/4074555021 >> v2:172.21.15.42:3300/0 conn(0x7f0f38147ff0 0x7f0f380a1c10 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f0f280013e0 tx=0x7f0f2800c400).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:30.177 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:30.170+0000 7f0f37fff700 1 --2- 172.21.15.42:0/4074555021 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0f38147ff0 0x7f0f380a1c10 secure :-1 s=READY pgs=1704 cs=0 l=1 rx=0x7f0f280013e0 tx=0x7f0f2800c400).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:55:30.177 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:30.170+0000 7f0f37fff700 1 --2- 172.21.15.42:0/4074555021 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0f38147ff0 0x7f0f380a1c10 secure :-1 s=THROTTLE_DONE pgs=1704 cs=0 l=1 rx=0x7f0f280013e0 tx=0x7f0f2800c400).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:30.177 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:30.170+0000 7f0f357fa700 1 -- 172.21.15.42:0/4074555021 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0f2800cab0 con 0x7f0f38147ff0 2019-12-05T05:55:30.177 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:30.170+0000 7f0f37fff700 1 --2- 172.21.15.42:0/4074555021 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0f38147ff0 0x7f0f380a1c10 secure :-1 s=THROTTLE_DONE pgs=1704 cs=0 l=1 rx=0x7f0f280013e0 tx=0x7f0f2800c400).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:30.178 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:30.170+0000 7f0f3dac8700 1 -- 172.21.15.42:0/4074555021 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f3807e0f0 con 0x7f0f38147ff0 2019-12-05T05:55:30.178 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:30.170+0000 7f0f357fa700 1 -- 172.21.15.42:0/4074555021 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f28010250 con 0x7f0f38147ff0 2019-12-05T05:55:30.178 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:30.170+0000 7f0f37fff700 1 --2- 172.21.15.42:0/4074555021 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0f38147ff0 0x7f0f380a1c10 secure :-1 s=THROTTLE_DONE pgs=1704 cs=0 l=1 rx=0x7f0f280013e0 tx=0x7f0f2800c400).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:30.178 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:30.170+0000 7f0f357fa700 1 -- 172.21.15.42:0/4074555021 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0f2800d2b0 con 0x7f0f38147ff0 2019-12-05T05:55:30.178 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:30.170+0000 7f0f3dac8700 1 -- 172.21.15.42:0/4074555021 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f380a3360 con 0x7f0f38147ff0 2019-12-05T05:55:30.179 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:30.170+0000 7f0f37fff700 1 --2- 172.21.15.42:0/4074555021 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0f38147ff0 0x7f0f380a1c10 secure :-1 s=THROTTLE_DONE pgs=1704 cs=0 l=1 rx=0x7f0f280013e0 tx=0x7f0f2800c400).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:30.180 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:30.174+0000 7f0f357fa700 1 -- 172.21.15.42:0/4074555021 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f0f2800d550 con 0x7f0f38147ff0 2019-12-05T05:55:30.180 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:30.174+0000 7f0f37fff700 1 --2- 172.21.15.42:0/4074555021 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0f38147ff0 0x7f0f380a1c10 secure :-1 s=THROTTLE_DONE pgs=1704 cs=0 l=1 rx=0x7f0f280013e0 tx=0x7f0f2800c400).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:30.180 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:30.174+0000 7f0f357fa700 1 --2- 172.21.15.42:0/4074555021 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0f1c03b1c0 0x7f0f1c03d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:55:30.180 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:30.174+0000 7f0f357fa700 1 -- 172.21.15.42:0/4074555021 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 9041+0+0 (secure 0 0 0) 0x7f0f2804aa50 con 0x7f0f38147ff0 2019-12-05T05:55:30.180 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:30.174+0000 7f0f2f7fe700 1 --2- 172.21.15.42:0/4074555021 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0f1c03b1c0 0x7f0f1c03d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:55:30.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:30.174+0000 7f0f2f7fe700 1 --2- 172.21.15.42:0/4074555021 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0f1c03b1c0 0x7f0f1c03d6d0 crc :-1 s=READY pgs=729 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:55:30.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:30.174+0000 7f0f3dac8700 1 -- 172.21.15.42:0/4074555021 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f3805d110 con 0x7f0f38147ff0 2019-12-05T05:55:30.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:30.182+0000 7f0f37fff700 1 --2- 172.21.15.42:0/4074555021 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0f38147ff0 0x7f0f380a1c10 secure :-1 s=THROTTLE_DONE pgs=1704 cs=0 l=1 rx=0x7f0f280013e0 tx=0x7f0f2800c400).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:30.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:30.182+0000 7f0f357fa700 1 -- 172.21.15.42:0/4074555021 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f0f2804ade0 con 0x7f0f38147ff0 2019-12-05T05:55:30.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:30.346+0000 7f0f3dac8700 1 -- 172.21.15.42:0/4074555021 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0f380a2910 con 0x7f0f38147ff0 2019-12-05T05:55:30.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:30.346+0000 7f0f37fff700 1 --2- 172.21.15.42:0/4074555021 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0f38147ff0 0x7f0f380a1c10 secure :-1 s=THROTTLE_DONE pgs=1704 cs=0 l=1 rx=0x7f0f280013e0 tx=0x7f0f2800c400).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:30.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:30.346+0000 7f0f357fa700 1 -- 172.21.15.42:0/4074555021 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 208 v208) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f0f28031ad0 con 0x7f0f38147ff0 2019-12-05T05:55:30.349 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 208 2019-12-05T05:55:30.350 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:55:30.350 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":208,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6461,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/3549993215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3549993215},{"type":"v1","addr":"172.21.15.40:6833","nonce":3549993215}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":208}],"filesystems":[{"mdsmap":{"epoch":207,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:55:04.257118+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8614},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8614":{"gid":8614,"name":"a","rank":0,"incarnation":204,"state":"up:active","state_seq":32,"addr":"172.21.15.42:6835/2511123581","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2511123581},{"type":"v1","addr":"172.21.15.42:6835","nonce":2511123581}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:55:30.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:30.350+0000 7f0f3dac8700 1 -- 172.21.15.42:0/4074555021 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0f1c03b1c0 msgr2=0x7f0f1c03d6d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:55:30.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:30.350+0000 7f0f3dac8700 1 --2- 172.21.15.42:0/4074555021 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0f1c03b1c0 0x7f0f1c03d6d0 crc :-1 s=READY pgs=729 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:55:30.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:30.350+0000 7f0f3dac8700 1 -- 172.21.15.42:0/4074555021 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0f38147ff0 msgr2=0x7f0f380a1c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:55:30.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:30.350+0000 7f0f3dac8700 1 --2- 172.21.15.42:0/4074555021 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0f38147ff0 0x7f0f380a1c10 secure :-1 s=READY pgs=1704 cs=0 l=1 rx=0x7f0f280013e0 tx=0x7f0f2800c400).stop 2019-12-05T05:55:30.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:30.350+0000 7f0f3dac8700 1 -- 172.21.15.42:0/4074555021 shutdown_connections 2019-12-05T05:55:30.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:30.350+0000 7f0f3dac8700 1 --2- 172.21.15.42:0/4074555021 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0f1c03b1c0 0x7f0f1c03d6d0 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:55:30.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:30.350+0000 7f0f3dac8700 1 --2- 172.21.15.42:0/4074555021 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0f38147ff0 0x7f0f380a1c10 unknown :-1 s=CLOSED pgs=1704 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:55:30.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:30.350+0000 7f0f3dac8700 1 -- 172.21.15.42:0/4074555021 shutdown_connections 2019-12-05T05:55:30.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:30.350+0000 7f0f3dac8700 1 -- 172.21.15.42:0/4074555021 wait complete. 2019-12-05T05:55:30.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:30.350+0000 7f0f3dac8700 1 -- 172.21.15.42:0/4074555021 >> 172.21.15.42:0/4074555021 conn(0x7f0f3809f040 msgr2=0x7f0f380a9770 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:55:30.361 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T05:55:32.362 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:55:32.468 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:32.458+0000 7fb2a39ba700 1 Processor -- start 2019-12-05T05:55:32.468 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:32.458+0000 7fb2a39ba700 1 -- start start 2019-12-05T05:55:32.469 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:32.462+0000 7fb2a39ba700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb29c09fae0 0x7fb29c14fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:55:32.469 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:32.462+0000 7fb2a39ba700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb29c0a0710 con 0x7fb29c0a7740 2019-12-05T05:55:32.469 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:32.462+0000 7fb2a39ba700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb29c07ef20 con 0x7fb29c08df80 2019-12-05T05:55:32.469 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:32.462+0000 7fb2a39ba700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb29c0a0b00 con 0x7fb29c09fae0 2019-12-05T05:55:32.469 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:32.462+0000 7fb2a2758700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb29c09fae0 0x7fb29c14fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:55:32.469 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:32.462+0000 7fb2a1756700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fb29c08df80 0x7fb29c09f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:44186/0 (socket says 172.21.15.42:44186) 2019-12-05T05:55:32.469 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:32.462+0000 7fb2a2758700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb29c09fae0 0x7fb29c14fff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:44152/0 (socket says 172.21.15.42:44152) 2019-12-05T05:55:32.470 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:32.462+0000 7fb2a1756700 1 -- 172.21.15.42:0/1162433856 learned_addr learned my addr 172.21.15.42:0/1162433856 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:55:32.470 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:32.462+0000 7fb2a2758700 1 -- 172.21.15.42:0/1162433856 >> v1:172.21.15.42:6789/0 conn(0x7fb29c08df80 legacy=0x7fb29c09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:55:32.470 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:32.462+0000 7fb2a2758700 1 -- 172.21.15.42:0/1162433856 >> v1:172.21.15.40:6789/0 conn(0x7fb29c0a7740 legacy=0x7fb29c07e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:55:32.470 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:32.462+0000 7fb2a0f55700 1 -- 172.21.15.42:0/1162433856 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 580560869 0 0) 0x7fb29c07ef20 con 0x7fb29c08df80 2019-12-05T05:55:32.470 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:32.462+0000 7fb2a2758700 1 -- 172.21.15.42:0/1162433856 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb29c0a9ba0 con 0x7fb29c09fae0 2019-12-05T05:55:32.470 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:32.462+0000 7fb2a2758700 1 --2- 172.21.15.42:0/1162433856 >> v2:172.21.15.42:3300/0 conn(0x7fb29c09fae0 0x7fb29c14fff0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fb294008790 tx=0x7fb2980014d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:32.471 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:32.462+0000 7fb2a2758700 1 --2- 172.21.15.42:0/1162433856 >> v2:172.21.15.42:3300/0 conn(0x7fb29c09fae0 0x7fb29c14fff0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fb294008790 tx=0x7fb2980014d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:32.471 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:32.462+0000 7fb2a2758700 1 --2- 172.21.15.42:0/1162433856 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb29c09fae0 0x7fb29c14fff0 secure :-1 s=READY pgs=1706 cs=0 l=1 rx=0x7fb294008790 tx=0x7fb2980014d0).ready entity=mon.1 client_cookie=4e2b6269cbe8076c server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:55:32.471 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:32.462+0000 7fb2a2758700 1 --2- 172.21.15.42:0/1162433856 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb29c09fae0 0x7fb29c14fff0 secure :-1 s=THROTTLE_DONE pgs=1706 cs=0 l=1 rx=0x7fb294008790 tx=0x7fb2980014d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:32.471 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:32.462+0000 7fb2a0f55700 1 -- 172.21.15.42:0/1162433856 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb294008bb0 con 0x7fb29c09fae0 2019-12-05T05:55:32.471 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:32.462+0000 7fb2a2758700 1 --2- 172.21.15.42:0/1162433856 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb29c09fae0 0x7fb29c14fff0 secure :-1 s=THROTTLE_DONE pgs=1706 cs=0 l=1 rx=0x7fb294008790 tx=0x7fb2980014d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:32.471 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:32.462+0000 7fb2a2758700 1 --2- 172.21.15.42:0/1162433856 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb29c09fae0 0x7fb29c14fff0 secure :-1 s=THROTTLE_DONE pgs=1706 cs=0 l=1 rx=0x7fb294008790 tx=0x7fb2980014d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:32.472 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:32.462+0000 7fb2a0f55700 1 -- 172.21.15.42:0/1162433856 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2940087d0 con 0x7fb29c09fae0 2019-12-05T05:55:32.472 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:32.462+0000 7fb2a0f55700 1 -- 172.21.15.42:0/1162433856 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb2940117d0 con 0x7fb29c09fae0 2019-12-05T05:55:32.472 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:32.462+0000 7fb2a39ba700 1 -- 172.21.15.42:0/1162433856 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb29c09fae0 msgr2=0x7fb29c14fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:55:32.472 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:32.462+0000 7fb2a39ba700 1 --2- 172.21.15.42:0/1162433856 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb29c09fae0 0x7fb29c14fff0 secure :-1 s=READY pgs=1706 cs=0 l=1 rx=0x7fb294008790 tx=0x7fb2980014d0).stop 2019-12-05T05:55:32.472 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:32.462+0000 7fb2a39ba700 1 -- 172.21.15.42:0/1162433856 shutdown_connections 2019-12-05T05:55:32.472 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:32.462+0000 7fb2a39ba700 1 --2- 172.21.15.42:0/1162433856 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb29c09fae0 0x7fb29c14fff0 unknown :-1 s=CLOSED pgs=1706 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:55:32.473 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:32.462+0000 7fb2a39ba700 1 -- 172.21.15.42:0/1162433856 shutdown_connections 2019-12-05T05:55:32.473 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:32.462+0000 7fb2a39ba700 1 -- 172.21.15.42:0/1162433856 wait complete. 2019-12-05T05:55:32.473 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:32.462+0000 7fb2a39ba700 1 -- 172.21.15.42:0/1162433856 >> 172.21.15.42:0/1162433856 conn(0x7fb29c0aabe0 msgr2=0x7fb29c0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:55:32.476 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:32.466+0000 7fb2a39ba700 1 Processor -- start 2019-12-05T05:55:32.476 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:32.466+0000 7fb2a39ba700 1 -- start start 2019-12-05T05:55:32.476 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:32.466+0000 7fb2a39ba700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb29c116360 0x7fb29c11bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:55:32.477 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:32.466+0000 7fb2a39ba700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb29c11c210 0x7fb29c0742d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:55:32.477 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:32.466+0000 7fb2a39ba700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb29c121b20 con 0x7fb29c133910 2019-12-05T05:55:32.477 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:32.466+0000 7fb2a39ba700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb29c116790 con 0x7fb29c116360 2019-12-05T05:55:32.477 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:32.466+0000 7fb2a39ba700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb29c07d740 con 0x7fb29c11c210 2019-12-05T05:55:32.477 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:32.466+0000 7fb2a1756700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb29c116360 0x7fb29c11bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:55:32.477 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:32.466+0000 7fb2a1756700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb29c116360 0x7fb29c11bcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:44156/0 (socket says 172.21.15.42:44156) 2019-12-05T05:55:32.477 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:32.466+0000 7fb2a1f57700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fb29c133910 0x7fb29c11aa40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:44192/0 (socket says 172.21.15.42:44192) 2019-12-05T05:55:32.478 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:32.466+0000 7fb2a1756700 1 -- 172.21.15.42:0/3205864439 learned_addr learned my addr 172.21.15.42:0/3205864439 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:55:32.478 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:32.470+0000 7fb2a2758700 1 --2- 172.21.15.42:0/3205864439 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb29c11c210 0x7fb29c0742d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:55:32.478 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:32.470+0000 7fb2a1756700 1 -- 172.21.15.42:0/3205864439 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb29c11c210 msgr2=0x7fb29c0742d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:55:32.478 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:32.470+0000 7fb2a1756700 1 --2- 172.21.15.42:0/3205864439 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb29c11c210 0x7fb29c0742d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:55:32.478 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:32.470+0000 7fb2a1756700 1 -- 172.21.15.42:0/3205864439 >> v1:172.21.15.42:6789/0 conn(0x7fb29c133910 legacy=0x7fb29c11aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:55:32.478 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:32.470+0000 7fb28e7fc700 1 -- 172.21.15.42:0/3205864439 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 389502171 0 0) 0x7fb29c121b20 con 0x7fb29c133910 2019-12-05T05:55:32.478 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:32.470+0000 7fb2a1756700 1 -- 172.21.15.42:0/3205864439 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb29c121170 con 0x7fb29c116360 2019-12-05T05:55:32.479 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:32.470+0000 7fb2a1756700 1 --2- 172.21.15.42:0/3205864439 >> v2:172.21.15.42:3300/0 conn(0x7fb29c116360 0x7fb29c11bcc0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fb290000b80 tx=0x7fb290010940).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:32.479 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:32.470+0000 7fb2a1756700 1 --2- 172.21.15.42:0/3205864439 >> v2:172.21.15.42:3300/0 conn(0x7fb29c116360 0x7fb29c11bcc0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fb290000b80 tx=0x7fb290010940).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:32.479 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:32.470+0000 7fb2a1756700 1 --2- 172.21.15.42:0/3205864439 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb29c116360 0x7fb29c11bcc0 secure :-1 s=READY pgs=1708 cs=0 l=1 rx=0x7fb290000b80 tx=0x7fb290010940).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:55:32.479 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:32.470+0000 7fb2a1756700 1 --2- 172.21.15.42:0/3205864439 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb29c116360 0x7fb29c11bcc0 secure :-1 s=THROTTLE_DONE pgs=1708 cs=0 l=1 rx=0x7fb290000b80 tx=0x7fb290010940).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:32.479 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:32.470+0000 7fb2a1756700 1 --2- 172.21.15.42:0/3205864439 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb29c116360 0x7fb29c11bcc0 secure :-1 s=THROTTLE_DONE pgs=1708 cs=0 l=1 rx=0x7fb290000b80 tx=0x7fb290010940).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:32.479 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:32.470+0000 7fb28e7fc700 1 -- 172.21.15.42:0/3205864439 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb290011670 con 0x7fb29c116360 2019-12-05T05:55:32.479 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:32.470+0000 7fb2a1756700 1 --2- 172.21.15.42:0/3205864439 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb29c116360 0x7fb29c11bcc0 secure :-1 s=THROTTLE_DONE pgs=1708 cs=0 l=1 rx=0x7fb290000b80 tx=0x7fb290010940).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:32.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:32.470+0000 7fb28e7fc700 1 -- 172.21.15.42:0/3205864439 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb290011290 con 0x7fb29c116360 2019-12-05T05:55:32.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:32.470+0000 7fb2a39ba700 1 -- 172.21.15.42:0/3205864439 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb29c07c040 con 0x7fb29c116360 2019-12-05T05:55:32.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:32.470+0000 7fb28e7fc700 1 -- 172.21.15.42:0/3205864439 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb290011c90 con 0x7fb29c116360 2019-12-05T05:55:32.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:32.470+0000 7fb2a39ba700 1 -- 172.21.15.42:0/3205864439 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb29c074a40 con 0x7fb29c116360 2019-12-05T05:55:32.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:32.470+0000 7fb2a1756700 1 --2- 172.21.15.42:0/3205864439 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb29c116360 0x7fb29c11bcc0 secure :-1 s=THROTTLE_DONE pgs=1708 cs=0 l=1 rx=0x7fb290000b80 tx=0x7fb290010940).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:32.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:32.470+0000 7fb28e7fc700 1 -- 172.21.15.42:0/3205864439 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fb29000b330 con 0x7fb29c116360 2019-12-05T05:55:32.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:32.470+0000 7fb2a1756700 1 --2- 172.21.15.42:0/3205864439 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb29c116360 0x7fb29c11bcc0 secure :-1 s=THROTTLE_DONE pgs=1708 cs=0 l=1 rx=0x7fb290000b80 tx=0x7fb290010940).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:32.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:32.470+0000 7fb28e7fc700 1 --2- 172.21.15.42:0/3205864439 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb28403ac60 0x7fb28403d170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:55:32.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:32.470+0000 7fb28e7fc700 1 -- 172.21.15.42:0/3205864439 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 9041+0+0 (secure 0 0 0) 0x7fb29004b240 con 0x7fb29c116360 2019-12-05T05:55:32.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:32.470+0000 7fb2a1f57700 1 --2- 172.21.15.42:0/3205864439 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb28403ac60 0x7fb28403d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:55:32.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:32.470+0000 7fb2a1f57700 1 --2- 172.21.15.42:0/3205864439 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb28403ac60 0x7fb28403d170 crc :-1 s=READY pgs=730 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:55:32.482 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:32.474+0000 7fb2a39ba700 1 -- 172.21.15.42:0/3205864439 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb29c05d110 con 0x7fb29c116360 2019-12-05T05:55:32.483 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:32.482+0000 7fb2a1756700 1 --2- 172.21.15.42:0/3205864439 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb29c116360 0x7fb29c11bcc0 secure :-1 s=THROTTLE_DONE pgs=1708 cs=0 l=1 rx=0x7fb290000b80 tx=0x7fb290010940).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:32.483 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:32.482+0000 7fb28e7fc700 1 -- 172.21.15.42:0/3205864439 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fb2900322c0 con 0x7fb29c116360 2019-12-05T05:55:32.647 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:32.642+0000 7fb2a39ba700 1 -- 172.21.15.42:0/3205864439 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb29c07db20 con 0x7fb29c116360 2019-12-05T05:55:32.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:32.646+0000 7fb2a1756700 1 --2- 172.21.15.42:0/3205864439 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb29c116360 0x7fb29c11bcc0 secure :-1 s=THROTTLE_DONE pgs=1708 cs=0 l=1 rx=0x7fb290000b80 tx=0x7fb290010940).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:32.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:32.646+0000 7fb28e7fc700 1 -- 172.21.15.42:0/3205864439 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 208 v208) v1 ==== 95+0+2112 (secure 0 0 0) 0x7fb290032520 con 0x7fb29c116360 2019-12-05T05:55:32.648 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 208 2019-12-05T05:55:32.648 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:55:32.649 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":208,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6461,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/3549993215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3549993215},{"type":"v1","addr":"172.21.15.40:6833","nonce":3549993215}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":208}],"filesystems":[{"mdsmap":{"epoch":207,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:55:04.257118+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8614},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8614":{"gid":8614,"name":"a","rank":0,"incarnation":204,"state":"up:active","state_seq":32,"addr":"172.21.15.42:6835/2511123581","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2511123581},{"type":"v1","addr":"172.21.15.42:6835","nonce":2511123581}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:55:32.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:32.646+0000 7fb2a39ba700 1 -- 172.21.15.42:0/3205864439 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb28403ac60 msgr2=0x7fb28403d170 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:55:32.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:32.646+0000 7fb2a39ba700 1 --2- 172.21.15.42:0/3205864439 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb28403ac60 0x7fb28403d170 crc :-1 s=READY pgs=730 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:55:32.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:32.646+0000 7fb2a39ba700 1 -- 172.21.15.42:0/3205864439 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb29c116360 msgr2=0x7fb29c11bcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:55:32.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:32.646+0000 7fb2a39ba700 1 --2- 172.21.15.42:0/3205864439 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb29c116360 0x7fb29c11bcc0 secure :-1 s=READY pgs=1708 cs=0 l=1 rx=0x7fb290000b80 tx=0x7fb290010940).stop 2019-12-05T05:55:32.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:32.646+0000 7fb2a39ba700 1 -- 172.21.15.42:0/3205864439 shutdown_connections 2019-12-05T05:55:32.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:32.646+0000 7fb2a39ba700 1 --2- 172.21.15.42:0/3205864439 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb28403ac60 0x7fb28403d170 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:55:32.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:32.646+0000 7fb2a39ba700 1 --2- 172.21.15.42:0/3205864439 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb29c11c210 0x7fb29c0742d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:55:32.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:32.646+0000 7fb2a39ba700 1 --2- 172.21.15.42:0/3205864439 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb29c116360 0x7fb29c11bcc0 unknown :-1 s=CLOSED pgs=1708 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:55:32.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:32.646+0000 7fb2a39ba700 1 -- 172.21.15.42:0/3205864439 shutdown_connections 2019-12-05T05:55:32.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:32.646+0000 7fb2a39ba700 1 -- 172.21.15.42:0/3205864439 wait complete. 2019-12-05T05:55:32.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:32.646+0000 7fb2a39ba700 1 -- 172.21.15.42:0/3205864439 >> 172.21.15.42:0/3205864439 conn(0x7fb29c0a82a0 msgr2=0x7fb29c07e830 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:55:32.661 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T05:55:34.661 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:55:34.770 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:34.758+0000 7f8f2679f700 1 Processor -- start 2019-12-05T05:55:34.770 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:34.758+0000 7f8f2679f700 1 -- start start 2019-12-05T05:55:34.771 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:34.762+0000 7f8f2679f700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8f200a7740 0x7f8f2008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:55:34.771 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:34.762+0000 7f8f2679f700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f200a0710 con 0x7f8f2009fae0 2019-12-05T05:55:34.771 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:34.762+0000 7f8f2679f700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f200a9ba0 con 0x7f8f2007e720 2019-12-05T05:55:34.771 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:34.762+0000 7f8f2679f700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f200a0b00 con 0x7f8f200a7740 2019-12-05T05:55:34.771 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:34.762+0000 7f8f1ffff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f8f2007e720 0x7f8f2009f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:44202/0 (socket says 172.21.15.42:44202) 2019-12-05T05:55:34.772 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:34.762+0000 7f8f1ffff700 1 -- 172.21.15.42:0/2310640374 learned_addr learned my addr 172.21.15.42:0/2310640374 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:55:34.772 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:34.762+0000 7f8f24d3c700 1 --2- 172.21.15.42:0/2310640374 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8f200a7740 0x7f8f2008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:55:34.772 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:34.762+0000 7f8f1f7fe700 1 -- 172.21.15.42:0/2310640374 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3704678348 0 0) 0x7f8f200a9ba0 con 0x7f8f2007e720 2019-12-05T05:55:34.772 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:34.762+0000 7f8f1f7fe700 1 -- 172.21.15.42:0/2310640374 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f080030b0 con 0x7f8f2007e720 2019-12-05T05:55:34.772 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:34.762+0000 7f8f1f7fe700 1 -- 172.21.15.42:0/2310640374 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2955793232 0 0) 0x7f8f200a0710 con 0x7f8f2009fae0 2019-12-05T05:55:34.772 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:34.762+0000 7f8f1f7fe700 1 -- 172.21.15.42:0/2310640374 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f200a9ba0 con 0x7f8f2009fae0 2019-12-05T05:55:34.772 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:34.762+0000 7f8f24d3c700 1 -- 172.21.15.42:0/2310640374 >> v1:172.21.15.42:6789/0 conn(0x7f8f2007e720 legacy=0x7f8f2009f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:55:34.773 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:34.762+0000 7f8f24d3c700 1 -- 172.21.15.42:0/2310640374 >> v1:172.21.15.40:6789/0 conn(0x7f8f2009fae0 legacy=0x7f8f2014efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:55:34.773 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:34.762+0000 7f8f24d3c700 1 -- 172.21.15.42:0/2310640374 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f2007eb50 con 0x7f8f200a7740 2019-12-05T05:55:34.773 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:34.762+0000 7f8f24d3c700 1 --2- 172.21.15.42:0/2310640374 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8f200a7740 0x7f8f2008df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f8f10002f30 tx=0x7f8f1000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:34.773 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:34.762+0000 7f8f24d3c700 1 --2- 172.21.15.42:0/2310640374 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8f200a7740 0x7f8f2008df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f8f10002f30 tx=0x7f8f1000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:34.773 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:34.762+0000 7f8f24d3c700 1 --2- 172.21.15.42:0/2310640374 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8f200a7740 0x7f8f2008df80 secure :-1 s=READY pgs=384 cs=0 l=1 rx=0x7f8f10002f30 tx=0x7f8f1000fbf0).ready entity=mon.2 client_cookie=6c28386d58820f57 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:55:34.773 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:34.762+0000 7f8f24d3c700 1 --2- 172.21.15.42:0/2310640374 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8f200a7740 0x7f8f2008df80 secure :-1 s=THROTTLE_DONE pgs=384 cs=0 l=1 rx=0x7f8f10002f30 tx=0x7f8f1000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:34.773 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:34.762+0000 7f8f1f7fe700 1 -- 172.21.15.42:0/2310640374 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8f100112b0 con 0x7f8f200a7740 2019-12-05T05:55:34.774 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:34.762+0000 7f8f24d3c700 1 --2- 172.21.15.42:0/2310640374 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8f200a7740 0x7f8f2008df80 secure :-1 s=THROTTLE_DONE pgs=384 cs=0 l=1 rx=0x7f8f10002f30 tx=0x7f8f1000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:34.774 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:34.762+0000 7f8f1f7fe700 1 -- 172.21.15.42:0/2310640374 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f10008bc0 con 0x7f8f200a7740 2019-12-05T05:55:34.774 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:34.762+0000 7f8f24d3c700 1 --2- 172.21.15.42:0/2310640374 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8f200a7740 0x7f8f2008df80 secure :-1 s=THROTTLE_DONE pgs=384 cs=0 l=1 rx=0x7f8f10002f30 tx=0x7f8f1000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:34.774 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:34.762+0000 7f8f1f7fe700 1 -- 172.21.15.42:0/2310640374 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8f10011890 con 0x7f8f200a7740 2019-12-05T05:55:34.774 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:34.762+0000 7f8f2679f700 1 -- 172.21.15.42:0/2310640374 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8f200a7740 msgr2=0x7f8f2008df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:55:34.774 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:34.762+0000 7f8f2679f700 1 --2- 172.21.15.42:0/2310640374 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8f200a7740 0x7f8f2008df80 secure :-1 s=READY pgs=384 cs=0 l=1 rx=0x7f8f10002f30 tx=0x7f8f1000fbf0).stop 2019-12-05T05:55:34.774 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:34.766+0000 7f8f2679f700 1 -- 172.21.15.42:0/2310640374 shutdown_connections 2019-12-05T05:55:34.775 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:34.766+0000 7f8f2679f700 1 --2- 172.21.15.42:0/2310640374 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8f200a7740 0x7f8f2008df80 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:55:34.775 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:34.766+0000 7f8f2679f700 1 -- 172.21.15.42:0/2310640374 shutdown_connections 2019-12-05T05:55:34.775 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:34.766+0000 7f8f2679f700 1 -- 172.21.15.42:0/2310640374 wait complete. 2019-12-05T05:55:34.775 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:34.766+0000 7f8f2679f700 1 -- 172.21.15.42:0/2310640374 >> 172.21.15.42:0/2310640374 conn(0x7f8f200aabe0 msgr2=0x7f8f200a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:55:34.775 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:34.766+0000 7f8f2679f700 1 Processor -- start 2019-12-05T05:55:34.775 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:34.766+0000 7f8f2679f700 1 -- start start 2019-12-05T05:55:34.779 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:34.770+0000 7f8f2679f700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8f200a67b0 0x7f8f200a4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:55:34.779 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:34.770+0000 7f8f2679f700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8f20143430 0x7f8f2012cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:55:34.779 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:34.770+0000 7f8f2679f700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f200a4a50 con 0x7f8f201185e0 2019-12-05T05:55:34.779 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:34.770+0000 7f8f2679f700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f20142f30 con 0x7f8f200a67b0 2019-12-05T05:55:34.779 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:34.770+0000 7f8f2679f700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f200a7200 con 0x7f8f20143430 2019-12-05T05:55:34.780 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:34.770+0000 7f8f24d3c700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8f200a67b0 0x7f8f200a4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:55:34.780 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:34.770+0000 7f8f24d3c700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8f200a67b0 0x7f8f200a4f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.42:60216/0 (socket says 172.21.15.42:60216) 2019-12-05T05:55:34.780 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:34.770+0000 7f8f1ffff700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f8f201185e0 0x7f8f20078770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:53648/0 (socket says 172.21.15.42:53648) 2019-12-05T05:55:34.780 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:34.770+0000 7f8f2553d700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8f20143430 0x7f8f2012cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:55:34.780 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:34.770+0000 7f8f24d3c700 1 -- 172.21.15.42:0/2064727224 learned_addr learned my addr 172.21.15.42:0/2064727224 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:55:34.780 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:34.770+0000 7f8f24d3c700 1 -- 172.21.15.42:0/2064727224 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8f20143430 msgr2=0x7f8f2012cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:55:34.780 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:34.770+0000 7f8f24d3c700 1 --2- 172.21.15.42:0/2064727224 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8f20143430 0x7f8f2012cc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:55:34.781 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:34.770+0000 7f8f1d7fa700 1 -- 172.21.15.42:0/2064727224 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3575238577 0 0) 0x7f8f200a4a50 con 0x7f8f201185e0 2019-12-05T05:55:34.781 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:34.770+0000 7f8f24d3c700 1 -- 172.21.15.42:0/2064727224 >> v1:172.21.15.40:6789/0 conn(0x7f8f201185e0 legacy=0x7f8f20078770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:55:34.781 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:34.770+0000 7f8f24d3c700 1 -- 172.21.15.42:0/2064727224 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f200a5ff0 con 0x7f8f200a67b0 2019-12-05T05:55:34.781 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:34.770+0000 7f8f2553d700 1 --2- 172.21.15.42:0/2064727224 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8f20143430 0x7f8f2012cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:55:34.781 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:34.770+0000 7f8f24d3c700 1 --2- 172.21.15.42:0/2064727224 >> v2:172.21.15.40:3300/0 conn(0x7f8f200a67b0 0x7f8f200a4f30 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f8f10002f30 tx=0x7f8f10008bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:34.781 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:34.770+0000 7f8f24d3c700 1 --2- 172.21.15.42:0/2064727224 >> v2:172.21.15.40:3300/0 conn(0x7f8f200a67b0 0x7f8f200a4f30 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f8f10002f30 tx=0x7f8f10008bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:34.782 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:34.770+0000 7f8f24d3c700 1 --2- 172.21.15.42:0/2064727224 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8f200a67b0 0x7f8f200a4f30 secure :-1 s=READY pgs=1302 cs=0 l=1 rx=0x7f8f10002f30 tx=0x7f8f10008bc0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:55:34.782 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:34.770+0000 7f8f24d3c700 1 --2- 172.21.15.42:0/2064727224 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8f200a67b0 0x7f8f200a4f30 secure :-1 s=THROTTLE_DONE pgs=1302 cs=0 l=1 rx=0x7f8f10002f30 tx=0x7f8f10008bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:34.782 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:34.770+0000 7f8f1d7fa700 1 -- 172.21.15.42:0/2064727224 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8f10007150 con 0x7f8f200a67b0 2019-12-05T05:55:34.782 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:34.770+0000 7f8f24d3c700 1 --2- 172.21.15.42:0/2064727224 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8f200a67b0 0x7f8f200a4f30 secure :-1 s=THROTTLE_DONE pgs=1302 cs=0 l=1 rx=0x7f8f10002f30 tx=0x7f8f10008bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:34.782 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:34.770+0000 7f8f24d3c700 1 --2- 172.21.15.42:0/2064727224 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8f200a67b0 0x7f8f200a4f30 secure :-1 s=THROTTLE_DONE pgs=1302 cs=0 l=1 rx=0x7f8f10002f30 tx=0x7f8f10008bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:34.782 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:34.770+0000 7f8f1d7fa700 1 -- 172.21.15.42:0/2064727224 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f100114b0 con 0x7f8f200a67b0 2019-12-05T05:55:34.783 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:34.770+0000 7f8f1d7fa700 1 -- 172.21.15.42:0/2064727224 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8f1000f550 con 0x7f8f200a67b0 2019-12-05T05:55:34.783 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:34.770+0000 7f8f2679f700 1 -- 172.21.15.42:0/2064727224 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f20143860 con 0x7f8f200a67b0 2019-12-05T05:55:34.783 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:34.770+0000 7f8f2679f700 1 -- 172.21.15.42:0/2064727224 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f20078f70 con 0x7f8f200a67b0 2019-12-05T05:55:34.783 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:34.774+0000 7f8f24d3c700 1 --2- 172.21.15.42:0/2064727224 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8f200a67b0 0x7f8f200a4f30 secure :-1 s=THROTTLE_DONE pgs=1302 cs=0 l=1 rx=0x7f8f10002f30 tx=0x7f8f10008bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:34.783 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:34.774+0000 7f8f1d7fa700 1 -- 172.21.15.42:0/2064727224 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f8f10005540 con 0x7f8f200a67b0 2019-12-05T05:55:34.783 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:34.774+0000 7f8f24d3c700 1 --2- 172.21.15.42:0/2064727224 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8f200a67b0 0x7f8f200a4f30 secure :-1 s=THROTTLE_DONE pgs=1302 cs=0 l=1 rx=0x7f8f10002f30 tx=0x7f8f10008bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:34.783 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:34.774+0000 7f8f1d7fa700 1 --2- 172.21.15.42:0/2064727224 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8f0003acd0 0x7f8f0003d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:55:34.784 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:34.774+0000 7f8f1d7fa700 1 -- 172.21.15.42:0/2064727224 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 9041+0+0 (secure 0 0 0) 0x7f8f1004b6d0 con 0x7f8f200a67b0 2019-12-05T05:55:34.784 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:34.774+0000 7f8f1ffff700 1 --2- 172.21.15.42:0/2064727224 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8f0003acd0 0x7f8f0003d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:55:34.784 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:34.774+0000 7f8f1ffff700 1 --2- 172.21.15.42:0/2064727224 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8f0003acd0 0x7f8f0003d1e0 crc :-1 s=READY pgs=731 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:55:34.784 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:34.774+0000 7f8f2679f700 1 -- 172.21.15.42:0/2064727224 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f2005d110 con 0x7f8f200a67b0 2019-12-05T05:55:34.787 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:34.786+0000 7f8f24d3c700 1 --2- 172.21.15.42:0/2064727224 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8f200a67b0 0x7f8f200a4f30 secure :-1 s=THROTTLE_DONE pgs=1302 cs=0 l=1 rx=0x7f8f10002f30 tx=0x7f8f10008bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:34.788 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:34.786+0000 7f8f1d7fa700 1 -- 172.21.15.42:0/2064727224 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f8f10032730 con 0x7f8f200a67b0 2019-12-05T05:55:34.961 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:34.958+0000 7f8f2679f700 1 -- 172.21.15.42:0/2064727224 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8f2005b8c0 con 0x7f8f200a67b0 2019-12-05T05:55:34.962 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:34.958+0000 7f8f24d3c700 1 --2- 172.21.15.42:0/2064727224 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8f200a67b0 0x7f8f200a4f30 secure :-1 s=THROTTLE_DONE pgs=1302 cs=0 l=1 rx=0x7f8f10002f30 tx=0x7f8f10008bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:34.962 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:34.958+0000 7f8f1d7fa700 1 -- 172.21.15.42:0/2064727224 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 208 v208) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f8f10012bf0 con 0x7f8f200a67b0 2019-12-05T05:55:34.964 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:55:34.964 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":208,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6461,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/3549993215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3549993215},{"type":"v1","addr":"172.21.15.40:6833","nonce":3549993215}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":208}],"filesystems":[{"mdsmap":{"epoch":207,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:55:04.257118+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8614},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8614":{"gid":8614,"name":"a","rank":0,"incarnation":204,"state":"up:active","state_seq":32,"addr":"172.21.15.42:6835/2511123581","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2511123581},{"type":"v1","addr":"172.21.15.42:6835","nonce":2511123581}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:55:34.965 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 208 2019-12-05T05:55:34.967 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:34.962+0000 7f8f2679f700 1 -- 172.21.15.42:0/2064727224 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8f0003acd0 msgr2=0x7f8f0003d1e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:55:34.967 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:34.962+0000 7f8f2679f700 1 --2- 172.21.15.42:0/2064727224 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8f0003acd0 0x7f8f0003d1e0 crc :-1 s=READY pgs=731 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:55:34.967 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:34.962+0000 7f8f2679f700 1 -- 172.21.15.42:0/2064727224 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8f200a67b0 msgr2=0x7f8f200a4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:55:34.967 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:34.962+0000 7f8f2679f700 1 --2- 172.21.15.42:0/2064727224 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8f200a67b0 0x7f8f200a4f30 secure :-1 s=READY pgs=1302 cs=0 l=1 rx=0x7f8f10002f30 tx=0x7f8f10008bc0).stop 2019-12-05T05:55:34.967 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:34.962+0000 7f8f2679f700 1 -- 172.21.15.42:0/2064727224 shutdown_connections 2019-12-05T05:55:34.967 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:34.962+0000 7f8f2679f700 1 --2- 172.21.15.42:0/2064727224 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8f0003acd0 0x7f8f0003d1e0 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:55:34.968 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:34.962+0000 7f8f2679f700 1 --2- 172.21.15.42:0/2064727224 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8f20143430 0x7f8f2012cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:55:34.968 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:34.962+0000 7f8f2679f700 1 --2- 172.21.15.42:0/2064727224 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8f200a67b0 0x7f8f200a4f30 unknown :-1 s=CLOSED pgs=1302 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:55:34.968 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:34.962+0000 7f8f2679f700 1 -- 172.21.15.42:0/2064727224 shutdown_connections 2019-12-05T05:55:34.968 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:34.962+0000 7f8f2679f700 1 -- 172.21.15.42:0/2064727224 wait complete. 2019-12-05T05:55:34.968 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:34.962+0000 7f8f2679f700 1 -- 172.21.15.42:0/2064727224 >> 172.21.15.42:0/2064727224 conn(0x7f8f200a8420 msgr2=0x7f8f20059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:55:34.977 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T05:55:36.978 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:55:37.086 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:37.078+0000 7f7725c10700 1 Processor -- start 2019-12-05T05:55:37.086 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:37.078+0000 7f7725c10700 1 -- start start 2019-12-05T05:55:37.086 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:37.078+0000 7f7725c10700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f77200a7740 0x7f772008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:55:37.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:37.078+0000 7f7725c10700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f772009fae0 0x7f772007eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:55:37.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:37.078+0000 7f7725c10700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77200a0710 con 0x7f772007e720 2019-12-05T05:55:37.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:37.078+0000 7f7725c10700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77200a0b00 con 0x7f77200a7740 2019-12-05T05:55:37.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:37.078+0000 7f7725c10700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f772008e5b0 con 0x7f772009fae0 2019-12-05T05:55:37.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:37.078+0000 7f77249ae700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f772009fae0 0x7f772007eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:55:37.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:37.078+0000 7f77249ae700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f772009fae0 0x7f772007eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:44180/0 (socket says 172.21.15.42:44180) 2019-12-05T05:55:37.088 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:37.078+0000 7f77249ae700 1 -- 172.21.15.42:0/1517654819 learned_addr learned my addr 172.21.15.42:0/1517654819 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:55:37.088 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:37.078+0000 7f771f7fe700 1 --1- 172.21.15.42:0/1517654819 >> v1:172.21.15.42:6789/0 conn(0x7f772007e720 0x7f772009f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:44218/0 (socket says 172.21.15.42:44218) 2019-12-05T05:55:37.088 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:37.078+0000 7f771ffff700 1 --2- 172.21.15.42:0/1517654819 >> v2:172.21.15.40:3300/0 conn(0x7f77200a7740 0x7f772008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:55:37.088 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:37.078+0000 7f77249ae700 1 -- 172.21.15.42:0/1517654819 >> v2:172.21.15.40:3300/0 conn(0x7f77200a7740 msgr2=0x7f772008df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:55:37.088 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:37.078+0000 7f77249ae700 1 --2- 172.21.15.42:0/1517654819 >> v2:172.21.15.40:3300/0 conn(0x7f77200a7740 0x7f772008df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:55:37.088 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:37.078+0000 7f77249ae700 1 -- 172.21.15.42:0/1517654819 >> v1:172.21.15.42:6789/0 conn(0x7f772007e720 legacy=0x7f772009f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:55:37.089 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:37.078+0000 7f77249ae700 1 -- 172.21.15.42:0/1517654819 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f772007f0a0 con 0x7f772009fae0 2019-12-05T05:55:37.089 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:37.078+0000 7f771effd700 1 -- 172.21.15.42:0/1517654819 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3152378226 0 0) 0x7f77200a0710 con 0x7f772007e720 2019-12-05T05:55:37.089 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:37.078+0000 7f77249ae700 1 --2- 172.21.15.42:0/1517654819 >> v2:172.21.15.42:3300/0 conn(0x7f772009fae0 0x7f772007eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f7718002f30 tx=0x7f771800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:37.089 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:37.078+0000 7f77249ae700 1 --2- 172.21.15.42:0/1517654819 >> v2:172.21.15.42:3300/0 conn(0x7f772009fae0 0x7f772007eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f7718002f30 tx=0x7f771800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:37.089 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:37.078+0000 7f77249ae700 1 --2- 172.21.15.42:0/1517654819 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f772009fae0 0x7f772007eb50 secure :-1 s=READY pgs=1711 cs=0 l=1 rx=0x7f7718002f30 tx=0x7f771800fd30).ready entity=mon.1 client_cookie=e33eca03d189eabe server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:55:37.089 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:37.078+0000 7f77249ae700 1 --2- 172.21.15.42:0/1517654819 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f772009fae0 0x7f772007eb50 secure :-1 s=THROTTLE_DONE pgs=1711 cs=0 l=1 rx=0x7f7718002f30 tx=0x7f771800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:37.090 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:37.078+0000 7f771effd700 1 -- 172.21.15.42:0/1517654819 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7718008c90 con 0x7f772009fae0 2019-12-05T05:55:37.090 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:37.078+0000 7f77249ae700 1 --2- 172.21.15.42:0/1517654819 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f772009fae0 0x7f772007eb50 secure :-1 s=THROTTLE_DONE pgs=1711 cs=0 l=1 rx=0x7f7718002f30 tx=0x7f771800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:37.090 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:37.078+0000 7f77249ae700 1 --2- 172.21.15.42:0/1517654819 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f772009fae0 0x7f772007eb50 secure :-1 s=THROTTLE_DONE pgs=1711 cs=0 l=1 rx=0x7f7718002f30 tx=0x7f771800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:37.090 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:37.082+0000 7f771effd700 1 -- 172.21.15.42:0/1517654819 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77180088b0 con 0x7f772009fae0 2019-12-05T05:55:37.090 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:37.082+0000 7f771effd700 1 -- 172.21.15.42:0/1517654819 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7718011ad0 con 0x7f772009fae0 2019-12-05T05:55:37.090 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:37.082+0000 7f7725c10700 1 -- 172.21.15.42:0/1517654819 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f772009fae0 msgr2=0x7f772007eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:55:37.090 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:37.082+0000 7f7725c10700 1 --2- 172.21.15.42:0/1517654819 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f772009fae0 0x7f772007eb50 secure :-1 s=READY pgs=1711 cs=0 l=1 rx=0x7f7718002f30 tx=0x7f771800fd30).stop 2019-12-05T05:55:37.091 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:37.082+0000 7f7725c10700 1 -- 172.21.15.42:0/1517654819 shutdown_connections 2019-12-05T05:55:37.091 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:37.082+0000 7f7725c10700 1 --2- 172.21.15.42:0/1517654819 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f772009fae0 0x7f772007eb50 unknown :-1 s=CLOSED pgs=1711 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:55:37.091 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:37.082+0000 7f7725c10700 1 --2- 172.21.15.42:0/1517654819 >> v2:172.21.15.40:3300/0 conn(0x7f77200a7740 0x7f772008df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:55:37.091 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:37.082+0000 7f7725c10700 1 -- 172.21.15.42:0/1517654819 shutdown_connections 2019-12-05T05:55:37.091 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:37.082+0000 7f7725c10700 1 -- 172.21.15.42:0/1517654819 wait complete. 2019-12-05T05:55:37.091 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:37.082+0000 7f7725c10700 1 -- 172.21.15.42:0/1517654819 >> 172.21.15.42:0/1517654819 conn(0x7f77200aabe0 msgr2=0x7f77200a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:55:37.095 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:37.082+0000 7f7725c10700 1 Processor -- start 2019-12-05T05:55:37.095 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:37.082+0000 7f7725c10700 1 -- start start 2019-12-05T05:55:37.095 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:37.086+0000 7f7725c10700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f77200a1850 0x7f7720143ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:55:37.095 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:37.086+0000 7f7725c10700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77201253d0 con 0x7f772012ad90 2019-12-05T05:55:37.095 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:37.086+0000 7f7725c10700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77201377c0 con 0x7f7720145160 2019-12-05T05:55:37.095 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:37.086+0000 7f7725c10700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7720144430 con 0x7f77200a1850 2019-12-05T05:55:37.096 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:37.086+0000 7f77249ae700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f7720145160 0x7f7720124bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:44228/0 (socket says 172.21.15.42:44228) 2019-12-05T05:55:37.096 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:37.086+0000 7f77249ae700 1 -- 172.21.15.42:0/2533346295 learned_addr learned my addr 172.21.15.42:0/2533346295 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:55:37.096 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:37.086+0000 7f771f7fe700 1 --2- 172.21.15.42:0/2533346295 >> v2:172.21.15.40:3300/0 conn(0x7f77200a1850 0x7f7720143ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:55:37.096 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:37.086+0000 7f771cff9700 1 -- 172.21.15.42:0/2533346295 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4292246537 0 0) 0x7f77201377c0 con 0x7f7720145160 2019-12-05T05:55:37.096 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:37.086+0000 7f771cff9700 1 -- 172.21.15.42:0/2533346295 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77000030b0 con 0x7f7720145160 2019-12-05T05:55:37.096 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:37.086+0000 7f771cff9700 1 -- 172.21.15.42:0/2533346295 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 215016819 0 0) 0x7f77201253d0 con 0x7f772012ad90 2019-12-05T05:55:37.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:37.086+0000 7f771cff9700 1 -- 172.21.15.42:0/2533346295 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77201377c0 con 0x7f772012ad90 2019-12-05T05:55:37.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:37.086+0000 7f771cff9700 1 -- 172.21.15.42:0/2533346295 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 2896134887 0 0) 0x7f77000030b0 con 0x7f7720145160 2019-12-05T05:55:37.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:37.086+0000 7f771f7fe700 1 -- 172.21.15.42:0/2533346295 >> v1:172.21.15.42:6789/0 conn(0x7f7720145160 legacy=0x7f7720124bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:55:37.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:37.086+0000 7f771f7fe700 1 -- 172.21.15.42:0/2533346295 >> v1:172.21.15.40:6789/0 conn(0x7f772012ad90 legacy=0x7f772012eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:55:37.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:37.086+0000 7f771f7fe700 1 -- 172.21.15.42:0/2533346295 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7720138a00 con 0x7f77200a1850 2019-12-05T05:55:37.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:37.086+0000 7f771f7fe700 1 --2- 172.21.15.42:0/2533346295 >> v2:172.21.15.40:3300/0 conn(0x7f77200a1850 0x7f7720143ee0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f7718011cf0 tx=0x7f77180082d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:37.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:37.086+0000 7f771f7fe700 1 --2- 172.21.15.42:0/2533346295 >> v2:172.21.15.40:3300/0 conn(0x7f77200a1850 0x7f7720143ee0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f7718011cf0 tx=0x7f77180082d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:37.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:37.086+0000 7f771f7fe700 1 --2- 172.21.15.42:0/2533346295 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f77200a1850 0x7f7720143ee0 secure :-1 s=READY pgs=1304 cs=0 l=1 rx=0x7f7718011cf0 tx=0x7f77180082d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:55:37.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:37.086+0000 7f771f7fe700 1 --2- 172.21.15.42:0/2533346295 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f77200a1850 0x7f7720143ee0 secure :-1 s=THROTTLE_DONE pgs=1304 cs=0 l=1 rx=0x7f7718011cf0 tx=0x7f77180082d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:37.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:37.086+0000 7f771cff9700 1 -- 172.21.15.42:0/2533346295 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7718011de0 con 0x7f77200a1850 2019-12-05T05:55:37.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:37.086+0000 7f771f7fe700 1 --2- 172.21.15.42:0/2533346295 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f77200a1850 0x7f7720143ee0 secure :-1 s=THROTTLE_DONE pgs=1304 cs=0 l=1 rx=0x7f7718011cf0 tx=0x7f77180082d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:37.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:37.086+0000 7f771f7fe700 1 --2- 172.21.15.42:0/2533346295 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f77200a1850 0x7f7720143ee0 secure :-1 s=THROTTLE_DONE pgs=1304 cs=0 l=1 rx=0x7f7718011cf0 tx=0x7f77180082d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:37.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:37.086+0000 7f771cff9700 1 -- 172.21.15.42:0/2533346295 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77180116f0 con 0x7f77200a1850 2019-12-05T05:55:37.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:37.086+0000 7f7725c10700 1 -- 172.21.15.42:0/2533346295 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77201367b0 con 0x7f77200a1850 2019-12-05T05:55:37.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:37.086+0000 7f771cff9700 1 -- 172.21.15.42:0/2533346295 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7714004cc0 con 0x7f77200a1850 2019-12-05T05:55:37.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:37.086+0000 7f7725c10700 1 -- 172.21.15.42:0/2533346295 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7720136cb0 con 0x7f77200a1850 2019-12-05T05:55:37.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:37.090+0000 7f771f7fe700 1 --2- 172.21.15.42:0/2533346295 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f77200a1850 0x7f7720143ee0 secure :-1 s=THROTTLE_DONE pgs=1304 cs=0 l=1 rx=0x7f7718011cf0 tx=0x7f77180082d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:37.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:37.090+0000 7f771cff9700 1 -- 172.21.15.42:0/2533346295 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f77140123c0 con 0x7f77200a1850 2019-12-05T05:55:37.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:37.090+0000 7f771f7fe700 1 --2- 172.21.15.42:0/2533346295 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f77200a1850 0x7f7720143ee0 secure :-1 s=THROTTLE_DONE pgs=1304 cs=0 l=1 rx=0x7f7718011cf0 tx=0x7f77180082d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:37.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:37.090+0000 7f771cff9700 1 --2- 172.21.15.42:0/2533346295 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f770003b450 0x7f770003d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:55:37.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:37.090+0000 7f771cff9700 1 -- 172.21.15.42:0/2533346295 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 9041+0+0 (secure 0 0 0) 0x7f771404ab30 con 0x7f77200a1850 2019-12-05T05:55:37.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:37.090+0000 7f771ffff700 1 --2- 172.21.15.42:0/2533346295 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f770003b450 0x7f770003d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:55:37.100 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:37.090+0000 7f771ffff700 1 --2- 172.21.15.42:0/2533346295 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f770003b450 0x7f770003d960 crc :-1 s=READY pgs=732 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:55:37.100 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:37.094+0000 7f7725c10700 1 -- 172.21.15.42:0/2533346295 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f772005d110 con 0x7f77200a1850 2019-12-05T05:55:37.105 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:37.102+0000 7f771f7fe700 1 --2- 172.21.15.42:0/2533346295 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f77200a1850 0x7f7720143ee0 secure :-1 s=THROTTLE_DONE pgs=1304 cs=0 l=1 rx=0x7f7718011cf0 tx=0x7f77180082d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:37.105 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:37.102+0000 7f771cff9700 1 -- 172.21.15.42:0/2533346295 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f7714031b70 con 0x7f77200a1850 2019-12-05T05:55:37.277 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:37.274+0000 7f7725c10700 1 -- 172.21.15.42:0/2533346295 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f772005b8c0 con 0x7f77200a1850 2019-12-05T05:55:37.278 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:37.274+0000 7f771f7fe700 1 --2- 172.21.15.42:0/2533346295 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f77200a1850 0x7f7720143ee0 secure :-1 s=THROTTLE_DONE pgs=1304 cs=0 l=1 rx=0x7f7718011cf0 tx=0x7f77180082d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:37.278 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:37.274+0000 7f771cff9700 1 -- 172.21.15.42:0/2533346295 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 208 v208) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f7714031dd0 con 0x7f77200a1850 2019-12-05T05:55:37.280 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 208 2019-12-05T05:55:37.281 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:55:37.281 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":208,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6461,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/3549993215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3549993215},{"type":"v1","addr":"172.21.15.40:6833","nonce":3549993215}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":208}],"filesystems":[{"mdsmap":{"epoch":207,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:55:04.257118+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8614},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8614":{"gid":8614,"name":"a","rank":0,"incarnation":204,"state":"up:active","state_seq":32,"addr":"172.21.15.42:6835/2511123581","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2511123581},{"type":"v1","addr":"172.21.15.42:6835","nonce":2511123581}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:55:37.283 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:37.278+0000 7f7725c10700 1 -- 172.21.15.42:0/2533346295 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f770003b450 msgr2=0x7f770003d960 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:55:37.283 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:37.278+0000 7f7725c10700 1 --2- 172.21.15.42:0/2533346295 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f770003b450 0x7f770003d960 crc :-1 s=READY pgs=732 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:55:37.283 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:37.278+0000 7f7725c10700 1 -- 172.21.15.42:0/2533346295 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f77200a1850 msgr2=0x7f7720143ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:55:37.283 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:37.278+0000 7f7725c10700 1 --2- 172.21.15.42:0/2533346295 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f77200a1850 0x7f7720143ee0 secure :-1 s=READY pgs=1304 cs=0 l=1 rx=0x7f7718011cf0 tx=0x7f77180082d0).stop 2019-12-05T05:55:37.284 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:37.278+0000 7f7725c10700 1 -- 172.21.15.42:0/2533346295 shutdown_connections 2019-12-05T05:55:37.284 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:37.278+0000 7f7725c10700 1 --2- 172.21.15.42:0/2533346295 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f770003b450 0x7f770003d960 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:55:37.284 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:37.278+0000 7f7725c10700 1 --2- 172.21.15.42:0/2533346295 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f77200a1850 0x7f7720143ee0 unknown :-1 s=CLOSED pgs=1304 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:55:37.284 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:37.282+0000 7f7725c10700 1 -- 172.21.15.42:0/2533346295 shutdown_connections 2019-12-05T05:55:37.284 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:37.282+0000 7f7725c10700 1 -- 172.21.15.42:0/2533346295 wait complete. 2019-12-05T05:55:37.284 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:37.282+0000 7f7725c10700 1 -- 172.21.15.42:0/2533346295 >> 172.21.15.42:0/2533346295 conn(0x7f77200aabe0 msgr2=0x7f772011d480 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:55:37.293 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T05:55:39.294 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:55:39.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.394+0000 7f4d6f232700 1 Processor -- start 2019-12-05T05:55:39.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.394+0000 7f4d6f232700 1 -- start start 2019-12-05T05:55:39.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.394+0000 7f4d6f232700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4d680a7740 0x7f4d6808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:55:39.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.394+0000 7f4d6f232700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f4d6807e720 0x7f4d6807eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:55:39.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.394+0000 7f4d6f232700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f4d6809f2e0 0x7f4d6809f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:55:39.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.394+0000 7f4d6f232700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d680a0b00 con 0x7f4d6809f2e0 2019-12-05T05:55:39.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.394+0000 7f4d6f232700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d6808e5b0 con 0x7f4d6807e720 2019-12-05T05:55:39.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.394+0000 7f4d6f232700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d6808e850 con 0x7f4d680a7740 2019-12-05T05:55:39.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.394+0000 7f4d6cfce700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f4d6807e720 0x7f4d6807eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:55:39.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.394+0000 7f4d6cfce700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f4d6807e720 0x7f4d6807eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:44202/0 (socket says 172.21.15.42:44202) 2019-12-05T05:55:39.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.394+0000 7f4d6cfce700 1 -- 172.21.15.42:0/2776795578 learned_addr learned my addr 172.21.15.42:0/2776795578 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:55:39.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.394+0000 7f4d6d7cf700 1 --2- 172.21.15.42:0/2776795578 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4d680a7740 0x7f4d6808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:55:39.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.394+0000 7f4d6dfd0700 1 --2- 172.21.15.42:0/2776795578 >> v2:172.21.15.40:3300/0 conn(0x7f4d6809f2e0 0x7f4d6809f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:55:39.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.394+0000 7f4d6cfce700 1 -- 172.21.15.42:0/2776795578 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4d680a7740 msgr2=0x7f4d6808df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:55:39.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.394+0000 7f4d6cfce700 1 --2- 172.21.15.42:0/2776795578 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4d680a7740 0x7f4d6808df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:55:39.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.394+0000 7f4d6cfce700 1 -- 172.21.15.42:0/2776795578 >> v2:172.21.15.40:3300/0 conn(0x7f4d6809f2e0 msgr2=0x7f4d6809f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:55:39.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.394+0000 7f4d6cfce700 1 --2- 172.21.15.42:0/2776795578 >> v2:172.21.15.40:3300/0 conn(0x7f4d6809f2e0 0x7f4d6809f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:55:39.404 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.394+0000 7f4d6cfce700 1 -- 172.21.15.42:0/2776795578 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d6807f0a0 con 0x7f4d6807e720 2019-12-05T05:55:39.404 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.394+0000 7f4d6cfce700 1 --2- 172.21.15.42:0/2776795578 >> v2:172.21.15.42:3300/0 conn(0x7f4d6807e720 0x7f4d6807eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f4d6808e770 tx=0x7f4d5c010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:39.404 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.394+0000 7f4d6cfce700 1 --2- 172.21.15.42:0/2776795578 >> v2:172.21.15.42:3300/0 conn(0x7f4d6807e720 0x7f4d6807eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f4d6808e770 tx=0x7f4d5c010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:39.404 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.394+0000 7f4d6cfce700 1 --2- 172.21.15.42:0/2776795578 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4d6807e720 0x7f4d6807eb50 secure :-1 s=READY pgs=1713 cs=0 l=1 rx=0x7f4d6808e770 tx=0x7f4d5c010130).ready entity=mon.1 client_cookie=78e572035bac83c4 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:55:39.404 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.394+0000 7f4d6cfce700 1 --2- 172.21.15.42:0/2776795578 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4d6807e720 0x7f4d6807eb50 secure :-1 s=THROTTLE_DONE pgs=1713 cs=0 l=1 rx=0x7f4d6808e770 tx=0x7f4d5c010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:39.404 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.394+0000 7f4d6cfce700 1 --2- 172.21.15.42:0/2776795578 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4d6807e720 0x7f4d6807eb50 secure :-1 s=THROTTLE_DONE pgs=1713 cs=0 l=1 rx=0x7f4d6808e770 tx=0x7f4d5c010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:39.404 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.394+0000 7f4d67fff700 1 -- 172.21.15.42:0/2776795578 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4d5c010de0 con 0x7f4d6807e720 2019-12-05T05:55:39.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.394+0000 7f4d6cfce700 1 --2- 172.21.15.42:0/2776795578 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4d6807e720 0x7f4d6807eb50 secure :-1 s=THROTTLE_DONE pgs=1713 cs=0 l=1 rx=0x7f4d6808e770 tx=0x7f4d5c010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:39.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.394+0000 7f4d67fff700 1 -- 172.21.15.42:0/2776795578 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d5c010a00 con 0x7f4d6807e720 2019-12-05T05:55:39.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.394+0000 7f4d67fff700 1 -- 172.21.15.42:0/2776795578 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4d5c0114a0 con 0x7f4d6807e720 2019-12-05T05:55:39.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.394+0000 7f4d6f232700 1 -- 172.21.15.42:0/2776795578 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4d6807e720 msgr2=0x7f4d6807eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:55:39.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.394+0000 7f4d6f232700 1 --2- 172.21.15.42:0/2776795578 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4d6807e720 0x7f4d6807eb50 secure :-1 s=READY pgs=1713 cs=0 l=1 rx=0x7f4d6808e770 tx=0x7f4d5c010130).stop 2019-12-05T05:55:39.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.398+0000 7f4d6f232700 1 -- 172.21.15.42:0/2776795578 shutdown_connections 2019-12-05T05:55:39.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.398+0000 7f4d6f232700 1 --2- 172.21.15.42:0/2776795578 >> v2:172.21.15.40:3300/0 conn(0x7f4d6809f2e0 0x7f4d6809f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:55:39.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.398+0000 7f4d6f232700 1 --2- 172.21.15.42:0/2776795578 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4d680a7740 0x7f4d6808df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:55:39.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.398+0000 7f4d6f232700 1 --2- 172.21.15.42:0/2776795578 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4d6807e720 0x7f4d6807eb50 unknown :-1 s=CLOSED pgs=1713 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:55:39.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.398+0000 7f4d6f232700 1 -- 172.21.15.42:0/2776795578 shutdown_connections 2019-12-05T05:55:39.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.398+0000 7f4d6f232700 1 -- 172.21.15.42:0/2776795578 wait complete. 2019-12-05T05:55:39.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.398+0000 7f4d6f232700 1 -- 172.21.15.42:0/2776795578 >> 172.21.15.42:0/2776795578 conn(0x7f4d680aabe0 msgr2=0x7f4d680a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:55:39.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.398+0000 7f4d6f232700 1 Processor -- start 2019-12-05T05:55:39.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.398+0000 7f4d6f232700 1 -- start start 2019-12-05T05:55:39.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.398+0000 7f4d6f232700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f4d680752f0 0x7f4d68077730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:55:39.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.398+0000 7f4d6f232700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d680a53d0 con 0x7f4d68123650 2019-12-05T05:55:39.411 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.398+0000 7f4d6f232700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d681333a0 con 0x7f4d68116500 2019-12-05T05:55:39.411 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.398+0000 7f4d6f232700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d68124dd0 con 0x7f4d680752f0 2019-12-05T05:55:39.411 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.398+0000 7f4d6dfd0700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f4d680752f0 0x7f4d68077730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:55:39.411 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.398+0000 7f4d6cfce700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f4d68116500 0x7f4d680a4bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:44244/0 (socket says 172.21.15.42:44244) 2019-12-05T05:55:39.411 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.398+0000 7f4d6cfce700 1 -- 172.21.15.42:0/2400065424 learned_addr learned my addr 172.21.15.42:0/2400065424 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:55:39.411 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.398+0000 7f4d6dfd0700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f4d680752f0 0x7f4d68077730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:44208/0 (socket says 172.21.15.42:44208) 2019-12-05T05:55:39.411 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.402+0000 7f4d6dfd0700 1 -- 172.21.15.42:0/2400065424 >> v1:172.21.15.42:6789/0 conn(0x7f4d68116500 legacy=0x7f4d680a4bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:55:39.412 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.402+0000 7f4d65ffb700 1 -- 172.21.15.42:0/2400065424 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2092266248 0 0) 0x7f4d680a53d0 con 0x7f4d68123650 2019-12-05T05:55:39.412 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.402+0000 7f4d6dfd0700 1 -- 172.21.15.42:0/2400065424 >> v1:172.21.15.40:6789/0 conn(0x7f4d68123650 legacy=0x7f4d68124470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:55:39.412 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.402+0000 7f4d6dfd0700 1 -- 172.21.15.42:0/2400065424 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d680a5e70 con 0x7f4d680752f0 2019-12-05T05:55:39.412 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.402+0000 7f4d6dfd0700 1 --2- 172.21.15.42:0/2400065424 >> v2:172.21.15.42:3300/0 conn(0x7f4d680752f0 0x7f4d68077730 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f4d680a7cb0 tx=0x7f4d58002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:39.412 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.402+0000 7f4d6dfd0700 1 --2- 172.21.15.42:0/2400065424 >> v2:172.21.15.42:3300/0 conn(0x7f4d680752f0 0x7f4d68077730 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f4d680a7cb0 tx=0x7f4d58002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:39.412 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.402+0000 7f4d6dfd0700 1 --2- 172.21.15.42:0/2400065424 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4d680752f0 0x7f4d68077730 secure :-1 s=READY pgs=1715 cs=0 l=1 rx=0x7f4d680a7cb0 tx=0x7f4d58002ff0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:55:39.413 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.402+0000 7f4d6dfd0700 1 --2- 172.21.15.42:0/2400065424 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4d680752f0 0x7f4d68077730 secure :-1 s=THROTTLE_DONE pgs=1715 cs=0 l=1 rx=0x7f4d680a7cb0 tx=0x7f4d58002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:39.413 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.402+0000 7f4d6dfd0700 1 --2- 172.21.15.42:0/2400065424 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4d680752f0 0x7f4d68077730 secure :-1 s=THROTTLE_DONE pgs=1715 cs=0 l=1 rx=0x7f4d680a7cb0 tx=0x7f4d58002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:39.413 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.402+0000 7f4d65ffb700 1 -- 172.21.15.42:0/2400065424 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4d60005930 con 0x7f4d680752f0 2019-12-05T05:55:39.413 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.402+0000 7f4d6dfd0700 1 --2- 172.21.15.42:0/2400065424 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4d680752f0 0x7f4d68077730 secure :-1 s=THROTTLE_DONE pgs=1715 cs=0 l=1 rx=0x7f4d680a7cb0 tx=0x7f4d58002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:39.413 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.402+0000 7f4d65ffb700 1 -- 172.21.15.42:0/2400065424 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d60007200 con 0x7f4d680752f0 2019-12-05T05:55:39.413 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.402+0000 7f4d6f232700 1 -- 172.21.15.42:0/2400065424 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d68077c80 con 0x7f4d680752f0 2019-12-05T05:55:39.413 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.402+0000 7f4d65ffb700 1 -- 172.21.15.42:0/2400065424 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4d600060c0 con 0x7f4d680752f0 2019-12-05T05:55:39.414 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.402+0000 7f4d6f232700 1 -- 172.21.15.42:0/2400065424 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d680781d0 con 0x7f4d680752f0 2019-12-05T05:55:39.414 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.402+0000 7f4d6dfd0700 1 --2- 172.21.15.42:0/2400065424 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4d680752f0 0x7f4d68077730 secure :-1 s=THROTTLE_DONE pgs=1715 cs=0 l=1 rx=0x7f4d680a7cb0 tx=0x7f4d58002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:39.414 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.402+0000 7f4d65ffb700 1 -- 172.21.15.42:0/2400065424 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f4d60007bc0 con 0x7f4d680752f0 2019-12-05T05:55:39.414 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.402+0000 7f4d6dfd0700 1 --2- 172.21.15.42:0/2400065424 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4d680752f0 0x7f4d68077730 secure :-1 s=THROTTLE_DONE pgs=1715 cs=0 l=1 rx=0x7f4d680a7cb0 tx=0x7f4d58002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:39.414 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.402+0000 7f4d65ffb700 1 --2- 172.21.15.42:0/2400065424 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4d4803ac60 0x7f4d4803d170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:55:39.414 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.402+0000 7f4d65ffb700 1 -- 172.21.15.42:0/2400065424 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 9127+0+0 (secure 0 0 0) 0x7f4d6004aae0 con 0x7f4d680752f0 2019-12-05T05:55:39.414 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.402+0000 7f4d6d7cf700 1 --2- 172.21.15.42:0/2400065424 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4d4803ac60 0x7f4d4803d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:55:39.415 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.402+0000 7f4d6d7cf700 1 --2- 172.21.15.42:0/2400065424 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4d4803ac60 0x7f4d4803d170 crc :-1 s=READY pgs=734 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:55:39.415 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.410+0000 7f4d6f232700 1 -- 172.21.15.42:0/2400065424 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d6805d110 con 0x7f4d680752f0 2019-12-05T05:55:39.420 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.418+0000 7f4d6dfd0700 1 --2- 172.21.15.42:0/2400065424 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4d680752f0 0x7f4d68077730 secure :-1 s=THROTTLE_DONE pgs=1715 cs=0 l=1 rx=0x7f4d680a7cb0 tx=0x7f4d58002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:39.420 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.418+0000 7f4d65ffb700 1 -- 172.21.15.42:0/2400065424 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f4d60031b60 con 0x7f4d680752f0 2019-12-05T05:55:39.593 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.590+0000 7f4d6f232700 1 -- 172.21.15.42:0/2400065424 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4d6805b8c0 con 0x7f4d680752f0 2019-12-05T05:55:39.594 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.590+0000 7f4d6dfd0700 1 --2- 172.21.15.42:0/2400065424 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4d680752f0 0x7f4d68077730 secure :-1 s=THROTTLE_DONE pgs=1715 cs=0 l=1 rx=0x7f4d680a7cb0 tx=0x7f4d58002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:39.594 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.590+0000 7f4d65ffb700 1 -- 172.21.15.42:0/2400065424 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 209 v209) v1 ==== 95+0+1774 (secure 0 0 0) 0x7f4d60031dc0 con 0x7f4d680752f0 2019-12-05T05:55:39.594 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 209 2019-12-05T05:55:39.594 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:55:39.594 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":209,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":209,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:55:39.222230+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6461},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6461":{"gid":6461,"name":"b","rank":0,"incarnation":209,"state":"up:replay","state_seq":2,"addr":"172.21.15.40:6833/3549993215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3549993215},{"type":"v1","addr":"172.21.15.40:6833","nonce":3549993215}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:55:39.598 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.594+0000 7f4d6f232700 1 -- 172.21.15.42:0/2400065424 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4d4803ac60 msgr2=0x7f4d4803d170 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:55:39.598 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.594+0000 7f4d6f232700 1 --2- 172.21.15.42:0/2400065424 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4d4803ac60 0x7f4d4803d170 crc :-1 s=READY pgs=734 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:55:39.598 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.594+0000 7f4d6f232700 1 -- 172.21.15.42:0/2400065424 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4d680752f0 msgr2=0x7f4d68077730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:55:39.598 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.594+0000 7f4d6f232700 1 --2- 172.21.15.42:0/2400065424 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4d680752f0 0x7f4d68077730 secure :-1 s=READY pgs=1715 cs=0 l=1 rx=0x7f4d680a7cb0 tx=0x7f4d58002ff0).stop 2019-12-05T05:55:39.598 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.594+0000 7f4d6f232700 1 -- 172.21.15.42:0/2400065424 shutdown_connections 2019-12-05T05:55:39.598 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.594+0000 7f4d6f232700 1 --2- 172.21.15.42:0/2400065424 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4d4803ac60 0x7f4d4803d170 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:55:39.599 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.594+0000 7f4d6f232700 1 --2- 172.21.15.42:0/2400065424 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4d680752f0 0x7f4d68077730 unknown :-1 s=CLOSED pgs=1715 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:55:39.599 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.594+0000 7f4d6f232700 1 -- 172.21.15.42:0/2400065424 shutdown_connections 2019-12-05T05:55:39.599 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.594+0000 7f4d6f232700 1 -- 172.21.15.42:0/2400065424 wait complete. 2019-12-05T05:55:39.599 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.594+0000 7f4d6f232700 1 -- 172.21.15.42:0/2400065424 >> 172.21.15.42:0/2400065424 conn(0x7f4d680a89d0 msgr2=0x7f4d680a9790 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:55:39.609 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a down, removed from mdsmap 2019-12-05T05:55:39.609 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2019-12-05T05:55:39.610 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:55:39.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.698+0000 7ff056bf0700 1 Processor -- start 2019-12-05T05:55:39.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.698+0000 7ff056bf0700 1 -- start start 2019-12-05T05:55:39.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.702+0000 7ff056bf0700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff0500a7740 0x7ff05008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:55:39.708 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.702+0000 7ff056bf0700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff05007e720 0x7ff05007eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:55:39.708 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.702+0000 7ff056bf0700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0500a0710 con 0x7ff05009f2e0 2019-12-05T05:55:39.708 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.702+0000 7ff056bf0700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0500a0b00 con 0x7ff05007e720 2019-12-05T05:55:39.708 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.702+0000 7ff056bf0700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff05008e5b0 con 0x7ff0500a7740 2019-12-05T05:55:39.708 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.702+0000 7ff05498c700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff05007e720 0x7ff05007eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:55:39.708 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.702+0000 7ff05598e700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7ff05009f2e0 0x7ff05009f710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:44254/0 (socket says 172.21.15.42:44254) 2019-12-05T05:55:39.709 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.702+0000 7ff05598e700 1 -- 172.21.15.42:0/3235253223 learned_addr learned my addr 172.21.15.42:0/3235253223 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:55:39.709 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.702+0000 7ff05518d700 1 --2- 172.21.15.42:0/3235253223 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff0500a7740 0x7ff05008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:55:39.709 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.702+0000 7ff047fff700 1 -- 172.21.15.42:0/3235253223 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 58790103 0 0) 0x7ff0500a0710 con 0x7ff05009f2e0 2019-12-05T05:55:39.709 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.702+0000 7ff047fff700 1 -- 172.21.15.42:0/3235253223 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff03c0030b0 con 0x7ff05009f2e0 2019-12-05T05:55:39.709 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.702+0000 7ff05518d700 1 -- 172.21.15.42:0/3235253223 >> v2:172.21.15.40:3300/0 conn(0x7ff05007e720 msgr2=0x7ff05007eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:55:39.709 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.702+0000 7ff05518d700 1 --2- 172.21.15.42:0/3235253223 >> v2:172.21.15.40:3300/0 conn(0x7ff05007e720 0x7ff05007eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:55:39.709 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.702+0000 7ff05518d700 1 -- 172.21.15.42:0/3235253223 >> v1:172.21.15.42:6789/0 conn(0x7ff05009f2e0 legacy=0x7ff05009f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:55:39.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.702+0000 7ff05518d700 1 -- 172.21.15.42:0/3235253223 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff05007f0a0 con 0x7ff0500a7740 2019-12-05T05:55:39.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.702+0000 7ff05518d700 1 --2- 172.21.15.42:0/3235253223 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff0500a7740 0x7ff05008df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7ff04c008b10 tx=0x7ff04c00fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:39.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.702+0000 7ff05518d700 1 --2- 172.21.15.42:0/3235253223 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff0500a7740 0x7ff05008df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7ff04c008b10 tx=0x7ff04c00fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:39.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.702+0000 7ff05518d700 1 --2- 172.21.15.42:0/3235253223 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff0500a7740 0x7ff05008df80 secure :-1 s=READY pgs=385 cs=0 l=1 rx=0x7ff04c008b10 tx=0x7ff04c00fff0).ready entity=mon.2 client_cookie=1589de7246af3c58 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:55:39.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.702+0000 7ff05518d700 1 --2- 172.21.15.42:0/3235253223 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff0500a7740 0x7ff05008df80 secure :-1 s=THROTTLE_DONE pgs=385 cs=0 l=1 rx=0x7ff04c008b10 tx=0x7ff04c00fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:39.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.702+0000 7ff047fff700 1 -- 172.21.15.42:0/3235253223 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff04c011a50 con 0x7ff0500a7740 2019-12-05T05:55:39.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.702+0000 7ff05518d700 1 --2- 172.21.15.42:0/3235253223 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff0500a7740 0x7ff05008df80 secure :-1 s=THROTTLE_DONE pgs=385 cs=0 l=1 rx=0x7ff04c008b10 tx=0x7ff04c00fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:39.711 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.702+0000 7ff05518d700 1 --2- 172.21.15.42:0/3235253223 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff0500a7740 0x7ff05008df80 secure :-1 s=THROTTLE_DONE pgs=385 cs=0 l=1 rx=0x7ff04c008b10 tx=0x7ff04c00fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:39.711 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.702+0000 7ff047fff700 1 -- 172.21.15.42:0/3235253223 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff04c011670 con 0x7ff0500a7740 2019-12-05T05:55:39.711 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.702+0000 7ff047fff700 1 -- 172.21.15.42:0/3235253223 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff04c00a350 con 0x7ff0500a7740 2019-12-05T05:55:39.711 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.702+0000 7ff056bf0700 1 -- 172.21.15.42:0/3235253223 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff0500a7740 msgr2=0x7ff05008df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:55:39.711 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.702+0000 7ff056bf0700 1 --2- 172.21.15.42:0/3235253223 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff0500a7740 0x7ff05008df80 secure :-1 s=READY pgs=385 cs=0 l=1 rx=0x7ff04c008b10 tx=0x7ff04c00fff0).stop 2019-12-05T05:55:39.711 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.702+0000 7ff056bf0700 1 -- 172.21.15.42:0/3235253223 shutdown_connections 2019-12-05T05:55:39.712 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.702+0000 7ff056bf0700 1 --2- 172.21.15.42:0/3235253223 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff0500a7740 0x7ff05008df80 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:55:39.712 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.702+0000 7ff056bf0700 1 --2- 172.21.15.42:0/3235253223 >> v2:172.21.15.40:3300/0 conn(0x7ff05007e720 0x7ff05007eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:55:39.712 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.702+0000 7ff056bf0700 1 -- 172.21.15.42:0/3235253223 shutdown_connections 2019-12-05T05:55:39.712 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.702+0000 7ff056bf0700 1 -- 172.21.15.42:0/3235253223 wait complete. 2019-12-05T05:55:39.712 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.702+0000 7ff056bf0700 1 -- 172.21.15.42:0/3235253223 >> 172.21.15.42:0/3235253223 conn(0x7ff0500aabe0 msgr2=0x7ff0500a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:55:39.718 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.706+0000 7ff056bf0700 1 Processor -- start 2019-12-05T05:55:39.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.706+0000 7ff056bf0700 1 -- start start 2019-12-05T05:55:39.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.706+0000 7ff056bf0700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff0500a37e0 0x7ff050129b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:55:39.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.706+0000 7ff056bf0700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff05011b600 0x7ff05011ba30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:55:39.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.706+0000 7ff056bf0700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff05011c220 con 0x7ff0501192c0 2019-12-05T05:55:39.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.706+0000 7ff056bf0700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0500a3c10 con 0x7ff0500a37e0 2019-12-05T05:55:39.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.706+0000 7ff056bf0700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0500a32a0 con 0x7ff05011b600 2019-12-05T05:55:39.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.706+0000 7ff05498c700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff0500a37e0 0x7ff050129b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:55:39.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.706+0000 7ff05498c700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff0500a37e0 0x7ff050129b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.42:60268/0 (socket says 172.21.15.42:60268) 2019-12-05T05:55:39.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.706+0000 7ff05518d700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7ff0501192c0 0x7ff0500a17d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:53696/0 (socket says 172.21.15.42:53696) 2019-12-05T05:55:39.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.706+0000 7ff05598e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff05011b600 0x7ff05011ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:55:39.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.706+0000 7ff05498c700 1 -- 172.21.15.42:0/1751182087 learned_addr learned my addr 172.21.15.42:0/1751182087 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:55:39.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.710+0000 7ff05498c700 1 -- 172.21.15.42:0/1751182087 >> v2:172.21.15.42:3300/0 conn(0x7ff05011b600 msgr2=0x7ff05011ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:55:39.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.710+0000 7ff05498c700 1 --2- 172.21.15.42:0/1751182087 >> v2:172.21.15.42:3300/0 conn(0x7ff05011b600 0x7ff05011ba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:55:39.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.710+0000 7ff05498c700 1 -- 172.21.15.42:0/1751182087 >> v1:172.21.15.40:6789/0 conn(0x7ff0501192c0 legacy=0x7ff0500a17d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:55:39.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.710+0000 7ff045ffb700 1 -- 172.21.15.42:0/1751182087 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1990137850 0 0) 0x7ff05011c220 con 0x7ff0501192c0 2019-12-05T05:55:39.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.710+0000 7ff05498c700 1 -- 172.21.15.42:0/1751182087 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff05011bf80 con 0x7ff0500a37e0 2019-12-05T05:55:39.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.710+0000 7ff05498c700 1 --2- 172.21.15.42:0/1751182087 >> v2:172.21.15.40:3300/0 conn(0x7ff0500a37e0 0x7ff050129b90 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7ff0500a3460 tx=0x7ff040010c40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:39.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.710+0000 7ff05498c700 1 --2- 172.21.15.42:0/1751182087 >> v2:172.21.15.40:3300/0 conn(0x7ff0500a37e0 0x7ff050129b90 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7ff0500a3460 tx=0x7ff040010c40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:39.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.710+0000 7ff05498c700 1 --2- 172.21.15.42:0/1751182087 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff0500a37e0 0x7ff050129b90 secure :-1 s=READY pgs=1307 cs=0 l=1 rx=0x7ff0500a3460 tx=0x7ff040010c40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:55:39.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.710+0000 7ff05498c700 1 --2- 172.21.15.42:0/1751182087 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff0500a37e0 0x7ff050129b90 secure :-1 s=THROTTLE_DONE pgs=1307 cs=0 l=1 rx=0x7ff0500a3460 tx=0x7ff040010c40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:39.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.710+0000 7ff05498c700 1 --2- 172.21.15.42:0/1751182087 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff0500a37e0 0x7ff050129b90 secure :-1 s=THROTTLE_DONE pgs=1307 cs=0 l=1 rx=0x7ff0500a3460 tx=0x7ff040010c40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:39.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.710+0000 7ff045ffb700 1 -- 172.21.15.42:0/1751182087 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff040005930 con 0x7ff0500a37e0 2019-12-05T05:55:39.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.710+0000 7ff05498c700 1 --2- 172.21.15.42:0/1751182087 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff0500a37e0 0x7ff050129b90 secure :-1 s=THROTTLE_DONE pgs=1307 cs=0 l=1 rx=0x7ff0500a3460 tx=0x7ff040010c40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:39.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.710+0000 7ff045ffb700 1 -- 172.21.15.42:0/1751182087 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff040007160 con 0x7ff0500a37e0 2019-12-05T05:55:39.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.710+0000 7ff045ffb700 1 -- 172.21.15.42:0/1751182087 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff040012280 con 0x7ff0500a37e0 2019-12-05T05:55:39.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.710+0000 7ff056bf0700 1 -- 172.21.15.42:0/1751182087 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff05012a0e0 con 0x7ff0500a37e0 2019-12-05T05:55:39.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.710+0000 7ff056bf0700 1 -- 172.21.15.42:0/1751182087 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff05008eb70 con 0x7ff0500a37e0 2019-12-05T05:55:39.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.710+0000 7ff05498c700 1 --2- 172.21.15.42:0/1751182087 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff0500a37e0 0x7ff050129b90 secure :-1 s=THROTTLE_DONE pgs=1307 cs=0 l=1 rx=0x7ff0500a3460 tx=0x7ff040010c40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:39.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.710+0000 7ff045ffb700 1 -- 172.21.15.42:0/1751182087 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7ff040012520 con 0x7ff0500a37e0 2019-12-05T05:55:39.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.710+0000 7ff05498c700 1 --2- 172.21.15.42:0/1751182087 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff0500a37e0 0x7ff050129b90 secure :-1 s=THROTTLE_DONE pgs=1307 cs=0 l=1 rx=0x7ff0500a3460 tx=0x7ff040010c40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:39.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.710+0000 7ff045ffb700 1 --2- 172.21.15.42:0/1751182087 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff03003ac50 0x7ff03003d160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:55:39.724 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.710+0000 7ff045ffb700 1 -- 172.21.15.42:0/1751182087 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 9127+0+0 (secure 0 0 0) 0x7ff04004b3c0 con 0x7ff0500a37e0 2019-12-05T05:55:39.724 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.710+0000 7ff05518d700 1 --2- 172.21.15.42:0/1751182087 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff03003ac50 0x7ff03003d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:55:39.724 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.714+0000 7ff05518d700 1 --2- 172.21.15.42:0/1751182087 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff03003ac50 0x7ff03003d160 crc :-1 s=READY pgs=735 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:55:39.724 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.714+0000 7ff056bf0700 1 -- 172.21.15.42:0/1751182087 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff05005d110 con 0x7ff0500a37e0 2019-12-05T05:55:39.725 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.722+0000 7ff05498c700 1 --2- 172.21.15.42:0/1751182087 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff0500a37e0 0x7ff050129b90 secure :-1 s=THROTTLE_DONE pgs=1307 cs=0 l=1 rx=0x7ff0500a3460 tx=0x7ff040010c40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:39.725 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.722+0000 7ff045ffb700 1 -- 172.21.15.42:0/1751182087 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7ff05012a0e0 con 0x7ff0500a37e0 2019-12-05T05:55:39.893 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.890+0000 7ff056bf0700 1 -- 172.21.15.42:0/1751182087 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff0500a6990 con 0x7ff0500a37e0 2019-12-05T05:55:39.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.890+0000 7ff05498c700 1 --2- 172.21.15.42:0/1751182087 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff0500a37e0 0x7ff050129b90 secure :-1 s=THROTTLE_DONE pgs=1307 cs=0 l=1 rx=0x7ff0500a3460 tx=0x7ff040010c40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:39.895 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.890+0000 7ff045ffb700 1 -- 172.21.15.42:0/1751182087 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 209 v209) v1 ==== 95+0+1774 (secure 0 0 0) 0x7ff040032420 con 0x7ff0500a37e0 2019-12-05T05:55:39.895 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 209 2019-12-05T05:55:39.895 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:55:39.895 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":209,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":209,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:55:39.222230+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6461},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6461":{"gid":6461,"name":"b","rank":0,"incarnation":209,"state":"up:replay","state_seq":2,"addr":"172.21.15.40:6833/3549993215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3549993215},{"type":"v1","addr":"172.21.15.40:6833","nonce":3549993215}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:55:39.897 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.894+0000 7ff056bf0700 1 -- 172.21.15.42:0/1751182087 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff03003ac50 msgr2=0x7ff03003d160 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:55:39.897 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.894+0000 7ff056bf0700 1 --2- 172.21.15.42:0/1751182087 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff03003ac50 0x7ff03003d160 crc :-1 s=READY pgs=735 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:55:39.897 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.894+0000 7ff056bf0700 1 -- 172.21.15.42:0/1751182087 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff0500a37e0 msgr2=0x7ff050129b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:55:39.897 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.894+0000 7ff056bf0700 1 --2- 172.21.15.42:0/1751182087 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff0500a37e0 0x7ff050129b90 secure :-1 s=READY pgs=1307 cs=0 l=1 rx=0x7ff0500a3460 tx=0x7ff040010c40).stop 2019-12-05T05:55:39.898 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.894+0000 7ff056bf0700 1 -- 172.21.15.42:0/1751182087 shutdown_connections 2019-12-05T05:55:39.898 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.894+0000 7ff056bf0700 1 --2- 172.21.15.42:0/1751182087 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff03003ac50 0x7ff03003d160 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:55:39.898 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.894+0000 7ff056bf0700 1 --2- 172.21.15.42:0/1751182087 >> v2:172.21.15.42:3300/0 conn(0x7ff05011b600 0x7ff05011ba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:55:39.898 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.894+0000 7ff056bf0700 1 --2- 172.21.15.42:0/1751182087 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff0500a37e0 0x7ff050129b90 unknown :-1 s=CLOSED pgs=1307 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:55:39.898 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.894+0000 7ff056bf0700 1 -- 172.21.15.42:0/1751182087 shutdown_connections 2019-12-05T05:55:39.898 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.894+0000 7ff056bf0700 1 -- 172.21.15.42:0/1751182087 wait complete. 2019-12-05T05:55:39.898 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:39.894+0000 7ff056bf0700 1 -- 172.21.15.42:0/1751182087 >> 172.21.15.42:0/1751182087 conn(0x7ff0500a7740 msgr2=0x7ff05011c540 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:55:39.913 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 209, "feature_flags": { "enable_multiple": false, "ever_enabled_multiple": false }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2019-12-05T05:03:35.604738+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 209, "ever_allowed_features": 2, "explicitly_allowed_features": 2, "failed": [], "flags": 18, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_6461": { "addr": "172.21.15.40:6833/3549993215", "addrs": { "addrvec": [ { "addr": "172.21.15.40:6832", "nonce": 3549993215, "type": "v2" }, { "addr": "172.21.15.40:6833", "nonce": 3549993215, "type": "v1" } ] }, "export_targets": [], "features": 4540138292836696063, "flags": 0, "gid": 6461, "incarnation": 209, "name": "b", "rank": 0, "state": "up:replay", "state_seq": 2 } }, "last_failure": 0, "last_failure_osd_epoch": 85, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 2, "min_compat_client": "0 (unknown)", "modified": "2019-12-05T05:55:39.222230+0000", "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6461 } } } ], "standbys": [] } 2019-12-05T05:55:41.914 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:55:42.019 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:42.010+0000 7f53cc377700 1 Processor -- start 2019-12-05T05:55:42.020 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:42.010+0000 7f53cc377700 1 -- start start 2019-12-05T05:55:42.020 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:42.014+0000 7f53cc377700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f53c40a7730 0x7f53c408df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:55:42.020 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:42.014+0000 7f53cc377700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f53c409fad0 0x7f53c407eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:55:42.020 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:42.014+0000 7f53cc377700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53c40a0700 con 0x7f53c407e710 2019-12-05T05:55:42.020 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:42.014+0000 7f53cc377700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53c40a0af0 con 0x7f53c40a7730 2019-12-05T05:55:42.020 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:42.014+0000 7f53cc377700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53c408e5a0 con 0x7f53c409fad0 2019-12-05T05:55:42.021 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:42.014+0000 7f53ca914700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f53c40a7730 0x7f53c408df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:55:42.021 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:42.014+0000 7f53ca914700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f53c40a7730 0x7f53c408df70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:44226/0 (socket says 172.21.15.42:44226) 2019-12-05T05:55:42.021 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:42.014+0000 7f53ca914700 1 -- 172.21.15.42:0/2816614700 learned_addr learned my addr 172.21.15.42:0/2816614700 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:55:42.021 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:42.014+0000 7f53cb115700 1 -- 172.21.15.42:0/2816614700 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f53c409fad0 msgr2=0x7f53c407eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2019-12-05T05:55:42.021 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:42.014+0000 7f53cb115700 1 -- 172.21.15.42:0/2816614700 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f53c409fad0 msgr2=0x7f53c407eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2019-12-05T05:55:42.021 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:42.014+0000 7f53cb115700 1 --2- 172.21.15.42:0/2816614700 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f53c409fad0 0x7f53c407eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner read peer banner failed r=-1 ((1) Operation not permitted) 2019-12-05T05:55:42.021 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:42.014+0000 7f53ca914700 1 -- 172.21.15.42:0/2816614700 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f53c409fad0 msgr2=0x7f53c407eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:55:42.022 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:42.014+0000 7f53cb115700 1 --2- 172.21.15.42:0/2816614700 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f53c409fad0 0x7f53c407eb40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rx=0 tx=0)._fault waiting 0.200000 2019-12-05T05:55:42.022 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:42.014+0000 7f53ca914700 1 --2- 172.21.15.42:0/2816614700 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f53c409fad0 0x7f53c407eb40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:55:42.022 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:42.014+0000 7f53ca914700 1 -- 172.21.15.42:0/2816614700 >> v1:172.21.15.42:6789/0 conn(0x7f53c407e710 legacy=0x7f53c409f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:55:42.022 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:42.014+0000 7f53ca914700 1 -- 172.21.15.42:0/2816614700 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53c407f090 con 0x7f53c40a7730 2019-12-05T05:55:42.022 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:42.014+0000 7f53ca914700 1 --2- 172.21.15.42:0/2816614700 >> v2:172.21.15.42:3300/0 conn(0x7f53c40a7730 0x7f53c408df70 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f53b4008b10 tx=0x7f53b40100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:42.022 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:42.014+0000 7f53ca914700 1 --2- 172.21.15.42:0/2816614700 >> v2:172.21.15.42:3300/0 conn(0x7f53c40a7730 0x7f53c408df70 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f53b4008b10 tx=0x7f53b40100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:42.022 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:42.014+0000 7f53ca914700 1 --2- 172.21.15.42:0/2816614700 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f53c40a7730 0x7f53c408df70 secure :-1 s=READY pgs=1718 cs=0 l=1 rx=0x7f53b4008b10 tx=0x7f53b40100b0).ready entity=mon.1 client_cookie=889ff9715ae54559 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:55:42.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:42.014+0000 7f53ca914700 1 --2- 172.21.15.42:0/2816614700 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f53c40a7730 0x7f53c408df70 secure :-1 s=THROTTLE_DONE pgs=1718 cs=0 l=1 rx=0x7f53b4008b10 tx=0x7f53b40100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:42.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:42.014+0000 7f53c9912700 1 -- 172.21.15.42:0/2816614700 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f53b4010da0 con 0x7f53c40a7730 2019-12-05T05:55:42.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:42.014+0000 7f53ca914700 1 --2- 172.21.15.42:0/2816614700 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f53c40a7730 0x7f53c408df70 secure :-1 s=THROTTLE_DONE pgs=1718 cs=0 l=1 rx=0x7f53b4008b10 tx=0x7f53b40100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:42.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:42.014+0000 7f53c9912700 1 -- 172.21.15.42:0/2816614700 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53b40109c0 con 0x7f53c40a7730 2019-12-05T05:55:42.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:42.014+0000 7f53cc377700 1 -- 172.21.15.42:0/2816614700 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f53c40a7730 msgr2=0x7f53c408df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:55:42.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:42.014+0000 7f53cc377700 1 --2- 172.21.15.42:0/2816614700 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f53c40a7730 0x7f53c408df70 secure :-1 s=READY pgs=1718 cs=0 l=1 rx=0x7f53b4008b10 tx=0x7f53b40100b0).stop 2019-12-05T05:55:42.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:42.014+0000 7f53cc377700 1 -- 172.21.15.42:0/2816614700 shutdown_connections 2019-12-05T05:55:42.024 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:42.014+0000 7f53cc377700 1 --2- 172.21.15.42:0/2816614700 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f53c409fad0 0x7f53c407eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:55:42.024 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:42.014+0000 7f53cc377700 1 --2- 172.21.15.42:0/2816614700 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f53c40a7730 0x7f53c408df70 unknown :-1 s=CLOSED pgs=1718 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:55:42.024 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:42.014+0000 7f53cc377700 1 -- 172.21.15.42:0/2816614700 shutdown_connections 2019-12-05T05:55:42.024 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:42.014+0000 7f53cc377700 1 -- 172.21.15.42:0/2816614700 wait complete. 2019-12-05T05:55:42.024 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:42.014+0000 7f53cc377700 1 -- 172.21.15.42:0/2816614700 >> 172.21.15.42:0/2816614700 conn(0x7f53c40aabd0 msgr2=0x7f53c40a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:55:42.029 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:42.018+0000 7f53cc377700 1 Processor -- start 2019-12-05T05:55:42.029 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:42.018+0000 7f53cc377700 1 -- start start 2019-12-05T05:55:42.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:42.018+0000 7f53cc377700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f53c41199d0 0x7f53c411af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:55:42.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:42.018+0000 7f53cc377700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f53c4071370 0x7f53c40717a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:55:42.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:42.018+0000 7f53cc377700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53c4122d50 con 0x7f53c4122460 2019-12-05T05:55:42.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:42.018+0000 7f53cc377700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53c4122890 con 0x7f53c4071370 2019-12-05T05:55:42.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:42.018+0000 7f53cc377700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53c411b560 con 0x7f53c41199d0 2019-12-05T05:55:42.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:42.018+0000 7f53ca914700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f53c41199d0 0x7f53c411af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:55:42.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:42.018+0000 7f53ca914700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f53c41199d0 0x7f53c411af40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:44232/0 (socket says 172.21.15.42:44232) 2019-12-05T05:55:42.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:42.018+0000 7f53ca914700 1 -- 172.21.15.42:0/1090860097 learned_addr learned my addr 172.21.15.42:0/1090860097 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:55:42.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:42.018+0000 7f53cb115700 1 --2- 172.21.15.42:0/1090860097 >> v2:172.21.15.40:3300/0 conn(0x7f53c4071370 0x7f53c40717a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:55:42.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:42.018+0000 7f53ca914700 1 -- 172.21.15.42:0/1090860097 >> v2:172.21.15.40:3300/0 conn(0x7f53c4071370 msgr2=0x7f53c40717a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:55:42.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:42.018+0000 7f53ca914700 1 --2- 172.21.15.42:0/1090860097 >> v2:172.21.15.40:3300/0 conn(0x7f53c4071370 0x7f53c40717a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:55:42.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:42.018+0000 7f53ca914700 1 -- 172.21.15.42:0/1090860097 >> v1:172.21.15.40:6789/0 conn(0x7f53c4122460 legacy=0x7f53c411bd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:55:42.032 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:42.018+0000 7f53ca914700 1 -- 172.21.15.42:0/1090860097 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53c411c550 con 0x7f53c41199d0 2019-12-05T05:55:42.032 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:42.022+0000 7f53ca914700 1 --2- 172.21.15.42:0/1090860097 >> v2:172.21.15.42:3300/0 conn(0x7f53c41199d0 0x7f53c411af40 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f53b4008b10 tx=0x7f53b400f720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:42.032 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:42.022+0000 7f53ca914700 1 --2- 172.21.15.42:0/1090860097 >> v2:172.21.15.42:3300/0 conn(0x7f53c41199d0 0x7f53c411af40 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f53b4008b10 tx=0x7f53b400f720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:42.032 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:42.022+0000 7f53ca914700 1 --2- 172.21.15.42:0/1090860097 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f53c41199d0 0x7f53c411af40 secure :-1 s=READY pgs=1719 cs=0 l=1 rx=0x7f53b4008b10 tx=0x7f53b400f720).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:55:42.032 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:42.022+0000 7f53ca914700 1 --2- 172.21.15.42:0/1090860097 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f53c41199d0 0x7f53c411af40 secure :-1 s=THROTTLE_DONE pgs=1719 cs=0 l=1 rx=0x7f53b4008b10 tx=0x7f53b400f720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:42.032 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:42.022+0000 7f53c37fe700 1 -- 172.21.15.42:0/1090860097 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f53b4005ca0 con 0x7f53c41199d0 2019-12-05T05:55:42.033 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:42.022+0000 7f53ca914700 1 --2- 172.21.15.42:0/1090860097 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f53c41199d0 0x7f53c411af40 secure :-1 s=THROTTLE_DONE pgs=1719 cs=0 l=1 rx=0x7f53b4008b10 tx=0x7f53b400f720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:42.033 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:42.022+0000 7f53ca914700 1 --2- 172.21.15.42:0/1090860097 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f53c41199d0 0x7f53c411af40 secure :-1 s=THROTTLE_DONE pgs=1719 cs=0 l=1 rx=0x7f53b4008b10 tx=0x7f53b400f720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:42.033 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:42.022+0000 7f53c37fe700 1 -- 172.21.15.42:0/1090860097 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53b40058c0 con 0x7f53c41199d0 2019-12-05T05:55:42.033 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:42.022+0000 7f53cc377700 1 -- 172.21.15.42:0/1090860097 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53c4071cf0 con 0x7f53c41199d0 2019-12-05T05:55:42.033 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:42.022+0000 7f53c37fe700 1 -- 172.21.15.42:0/1090860097 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f53b4007540 con 0x7f53c41199d0 2019-12-05T05:55:42.033 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:42.022+0000 7f53cc377700 1 -- 172.21.15.42:0/1090860097 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53c40786b0 con 0x7f53c41199d0 2019-12-05T05:55:42.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:42.022+0000 7f53ca914700 1 --2- 172.21.15.42:0/1090860097 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f53c41199d0 0x7f53c411af40 secure :-1 s=THROTTLE_DONE pgs=1719 cs=0 l=1 rx=0x7f53b4008b10 tx=0x7f53b400f720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:42.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:42.022+0000 7f53c37fe700 1 -- 172.21.15.42:0/1090860097 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f53b400a150 con 0x7f53c41199d0 2019-12-05T05:55:42.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:42.022+0000 7f53ca914700 1 --2- 172.21.15.42:0/1090860097 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f53c41199d0 0x7f53c411af40 secure :-1 s=THROTTLE_DONE pgs=1719 cs=0 l=1 rx=0x7f53b4008b10 tx=0x7f53b400f720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:42.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:42.022+0000 7f53c37fe700 1 --2- 172.21.15.42:0/1090860097 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f53a403ae00 0x7f53a403d310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:55:42.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:42.022+0000 7f53c37fe700 1 -- 172.21.15.42:0/1090860097 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 9127+0+0 (secure 0 0 0) 0x7f53b404b6f0 con 0x7f53c41199d0 2019-12-05T05:55:42.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:42.022+0000 7f53ca113700 1 --2- 172.21.15.42:0/1090860097 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f53a403ae00 0x7f53a403d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:55:42.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:42.022+0000 7f53ca113700 1 --2- 172.21.15.42:0/1090860097 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f53a403ae00 0x7f53a403d310 crc :-1 s=READY pgs=736 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:55:42.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:42.026+0000 7f53cc377700 1 -- 172.21.15.42:0/1090860097 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53c405d100 con 0x7f53c41199d0 2019-12-05T05:55:42.042 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:42.034+0000 7f53ca914700 1 --2- 172.21.15.42:0/1090860097 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f53c41199d0 0x7f53c411af40 secure :-1 s=THROTTLE_DONE pgs=1719 cs=0 l=1 rx=0x7f53b4008b10 tx=0x7f53b400f720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:42.044 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:42.034+0000 7f53c37fe700 1 -- 172.21.15.42:0/1090860097 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f53b4032770 con 0x7f53c41199d0 2019-12-05T05:55:42.206 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:42.202+0000 7f53cc377700 1 -- 172.21.15.42:0/1090860097 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f53c408e800 con 0x7f53c41199d0 2019-12-05T05:55:42.207 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:42.202+0000 7f53ca914700 1 --2- 172.21.15.42:0/1090860097 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f53c41199d0 0x7f53c411af40 secure :-1 s=THROTTLE_DONE pgs=1719 cs=0 l=1 rx=0x7f53b4008b10 tx=0x7f53b400f720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:42.207 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:42.202+0000 7f53c37fe700 1 -- 172.21.15.42:0/1090860097 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 210 v210) v1 ==== 95+0+1778 (secure 0 0 0) 0x7f53b40329d0 con 0x7f53c41199d0 2019-12-05T05:55:42.207 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 210 2019-12-05T05:55:42.208 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:55:42.208 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":210,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":210,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:55:42.033691+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6461},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6461":{"gid":6461,"name":"b","rank":0,"incarnation":209,"state":"up:reconnect","state_seq":10,"addr":"172.21.15.40:6833/3549993215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3549993215},{"type":"v1","addr":"172.21.15.40:6833","nonce":3549993215}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:55:42.209 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:42.206+0000 7f53cc377700 1 -- 172.21.15.42:0/1090860097 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f53a403ae00 msgr2=0x7f53a403d310 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:55:42.209 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:42.206+0000 7f53cc377700 1 --2- 172.21.15.42:0/1090860097 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f53a403ae00 0x7f53a403d310 crc :-1 s=READY pgs=736 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:55:42.209 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:42.206+0000 7f53cc377700 1 -- 172.21.15.42:0/1090860097 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f53c41199d0 msgr2=0x7f53c411af40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:55:42.210 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:42.206+0000 7f53cc377700 1 --2- 172.21.15.42:0/1090860097 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f53c41199d0 0x7f53c411af40 secure :-1 s=READY pgs=1719 cs=0 l=1 rx=0x7f53b4008b10 tx=0x7f53b400f720).stop 2019-12-05T05:55:42.210 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:42.206+0000 7f53cc377700 1 -- 172.21.15.42:0/1090860097 shutdown_connections 2019-12-05T05:55:42.211 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:42.206+0000 7f53cc377700 1 --2- 172.21.15.42:0/1090860097 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f53a403ae00 0x7f53a403d310 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:55:42.211 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:42.206+0000 7f53cc377700 1 --2- 172.21.15.42:0/1090860097 >> v2:172.21.15.40:3300/0 conn(0x7f53c4071370 0x7f53c40717a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:55:42.211 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:42.206+0000 7f53cc377700 1 --2- 172.21.15.42:0/1090860097 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f53c41199d0 0x7f53c411af40 unknown :-1 s=CLOSED pgs=1719 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:55:42.211 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:42.206+0000 7f53cc377700 1 -- 172.21.15.42:0/1090860097 shutdown_connections 2019-12-05T05:55:42.211 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:42.206+0000 7f53cc377700 1 -- 172.21.15.42:0/1090860097 wait complete. 2019-12-05T05:55:42.211 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:42.206+0000 7f53cc377700 1 -- 172.21.15.42:0/1090860097 >> 172.21.15.42:0/1090860097 conn(0x7f53c40a9780 msgr2=0x7f53c40596f0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:55:42.220 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2019-12-05T05:55:42.448 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:55:42.453 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:55:44.220 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:55:44.325 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:44.318+0000 7fa6606a8700 1 Processor -- start 2019-12-05T05:55:44.325 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:44.318+0000 7fa6606a8700 1 -- start start 2019-12-05T05:55:44.325 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:44.318+0000 7fa6606a8700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa65808df80 0x7fa65809f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:55:44.325 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:44.318+0000 7fa6606a8700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fa65809f830 0x7fa65814efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:55:44.326 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:44.318+0000 7fa6606a8700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa65809fc60 con 0x7fa6580a7740 2019-12-05T05:55:44.326 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:44.318+0000 7fa6606a8700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6580a0b00 con 0x7fa65809f830 2019-12-05T05:55:44.326 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:44.318+0000 7fa6606a8700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa65808e5b0 con 0x7fa65808df80 2019-12-05T05:55:44.326 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:44.318+0000 7fa65f446700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fa65809f830 0x7fa65814efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:55:44.326 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:44.318+0000 7fa65f446700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fa65809f830 0x7fa65814efe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:44244/0 (socket says 172.21.15.42:44244) 2019-12-05T05:55:44.328 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:44.318+0000 7fa65f446700 1 -- 172.21.15.42:0/499828483 learned_addr learned my addr 172.21.15.42:0/499828483 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:55:44.328 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:44.318+0000 7fa65ec45700 1 --1- 172.21.15.42:0/499828483 >> v1:172.21.15.42:6789/0 conn(0x7fa6580a7740 0x7fa65807e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:44280/0 (socket says 172.21.15.42:44280) 2019-12-05T05:55:44.328 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:44.318+0000 7fa65e444700 1 --2- 172.21.15.42:0/499828483 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa65808df80 0x7fa65809f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:55:44.328 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:44.318+0000 7fa65f446700 1 -- 172.21.15.42:0/499828483 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa65808df80 msgr2=0x7fa65809f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:55:44.328 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:44.318+0000 7fa65f446700 1 --2- 172.21.15.42:0/499828483 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa65808df80 0x7fa65809f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:55:44.329 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:44.318+0000 7fa65f446700 1 -- 172.21.15.42:0/499828483 >> v1:172.21.15.42:6789/0 conn(0x7fa6580a7740 legacy=0x7fa65807e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:55:44.329 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:44.318+0000 7fa65dc43700 1 -- 172.21.15.42:0/499828483 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 539394169 0 0) 0x7fa65809fc60 con 0x7fa6580a7740 2019-12-05T05:55:44.329 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:44.318+0000 7fa65f446700 1 -- 172.21.15.42:0/499828483 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6580a0710 con 0x7fa65809f830 2019-12-05T05:55:44.330 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:44.318+0000 7fa65f446700 1 --2- 172.21.15.42:0/499828483 >> v2:172.21.15.42:3300/0 conn(0x7fa65809f830 0x7fa65814efe0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fa650008b10 tx=0x7fa6500100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:44.330 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:44.318+0000 7fa65f446700 1 --2- 172.21.15.42:0/499828483 >> v2:172.21.15.42:3300/0 conn(0x7fa65809f830 0x7fa65814efe0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fa650008b10 tx=0x7fa6500100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:44.330 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:44.318+0000 7fa65f446700 1 --2- 172.21.15.42:0/499828483 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa65809f830 0x7fa65814efe0 secure :-1 s=READY pgs=1721 cs=0 l=1 rx=0x7fa650008b10 tx=0x7fa6500100b0).ready entity=mon.1 client_cookie=76af83cd1f2fc23e server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:55:44.330 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:44.318+0000 7fa65f446700 1 --2- 172.21.15.42:0/499828483 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa65809f830 0x7fa65814efe0 secure :-1 s=THROTTLE_DONE pgs=1721 cs=0 l=1 rx=0x7fa650008b10 tx=0x7fa6500100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:44.330 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:44.318+0000 7fa65dc43700 1 -- 172.21.15.42:0/499828483 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa650010da0 con 0x7fa65809f830 2019-12-05T05:55:44.330 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:44.318+0000 7fa65f446700 1 --2- 172.21.15.42:0/499828483 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa65809f830 0x7fa65814efe0 secure :-1 s=THROTTLE_DONE pgs=1721 cs=0 l=1 rx=0x7fa650008b10 tx=0x7fa6500100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:44.331 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:44.318+0000 7fa65f446700 1 --2- 172.21.15.42:0/499828483 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa65809f830 0x7fa65814efe0 secure :-1 s=THROTTLE_DONE pgs=1721 cs=0 l=1 rx=0x7fa650008b10 tx=0x7fa6500100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:44.331 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:44.318+0000 7fa65dc43700 1 -- 172.21.15.42:0/499828483 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6500109c0 con 0x7fa65809f830 2019-12-05T05:55:44.331 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:44.318+0000 7fa65dc43700 1 -- 172.21.15.42:0/499828483 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa6500114a0 con 0x7fa65809f830 2019-12-05T05:55:44.331 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:44.318+0000 7fa6606a8700 1 -- 172.21.15.42:0/499828483 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa65809f830 msgr2=0x7fa65814efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:55:44.331 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:44.318+0000 7fa6606a8700 1 --2- 172.21.15.42:0/499828483 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa65809f830 0x7fa65814efe0 secure :-1 s=READY pgs=1721 cs=0 l=1 rx=0x7fa650008b10 tx=0x7fa6500100b0).stop 2019-12-05T05:55:44.331 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:44.322+0000 7fa6606a8700 1 -- 172.21.15.42:0/499828483 shutdown_connections 2019-12-05T05:55:44.331 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:44.322+0000 7fa6606a8700 1 --2- 172.21.15.42:0/499828483 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa65809f830 0x7fa65814efe0 unknown :-1 s=CLOSED pgs=1721 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:55:44.332 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:44.322+0000 7fa6606a8700 1 --2- 172.21.15.42:0/499828483 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa65808df80 0x7fa65809f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:55:44.332 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:44.322+0000 7fa6606a8700 1 -- 172.21.15.42:0/499828483 shutdown_connections 2019-12-05T05:55:44.332 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:44.322+0000 7fa6606a8700 1 -- 172.21.15.42:0/499828483 wait complete. 2019-12-05T05:55:44.332 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:44.322+0000 7fa6606a8700 1 -- 172.21.15.42:0/499828483 >> 172.21.15.42:0/499828483 conn(0x7fa6580aabe0 msgr2=0x7fa6580a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:55:44.336 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:44.322+0000 7fa6606a8700 1 Processor -- start 2019-12-05T05:55:44.336 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:44.322+0000 7fa6606a8700 1 -- start start 2019-12-05T05:55:44.336 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:44.326+0000 7fa6606a8700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa65811aa30 0x7fa65812cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:55:44.337 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:44.326+0000 7fa6606a8700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa6581242f0 0x7fa658124720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:55:44.337 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:44.326+0000 7fa6606a8700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6580a65a0 con 0x7fa658124c70 2019-12-05T05:55:44.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:44.326+0000 7fa6606a8700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa658123db0 con 0x7fa6581242f0 2019-12-05T05:55:44.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:44.326+0000 7fa6606a8700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa658124050 con 0x7fa65811aa30 2019-12-05T05:55:44.341 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:44.326+0000 7fa65f446700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fa658124c70 0x7fa658130390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:44290/0 (socket says 172.21.15.42:44290) 2019-12-05T05:55:44.341 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:44.326+0000 7fa65f446700 1 -- 172.21.15.42:0/775231242 learned_addr learned my addr 172.21.15.42:0/775231242 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:55:44.341 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:44.326+0000 7fa65e444700 1 --2- 172.21.15.42:0/775231242 >> v2:172.21.15.40:3300/0 conn(0x7fa6581242f0 0x7fa658124720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:55:44.341 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:44.326+0000 7fa65ec45700 1 --2- 172.21.15.42:0/775231242 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa65811aa30 0x7fa65812cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:55:44.341 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:44.326+0000 7fa64f7fe700 1 -- 172.21.15.42:0/775231242 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 692164959 0 0) 0x7fa6580a65a0 con 0x7fa658124c70 2019-12-05T05:55:44.341 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:44.326+0000 7fa64f7fe700 1 -- 172.21.15.42:0/775231242 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6400030b0 con 0x7fa658124c70 2019-12-05T05:55:44.342 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:44.326+0000 7fa65e444700 1 -- 172.21.15.42:0/775231242 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa65811aa30 msgr2=0x7fa65812cdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:55:44.342 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:44.326+0000 7fa65e444700 1 --2- 172.21.15.42:0/775231242 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa65811aa30 0x7fa65812cdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:55:44.342 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:44.326+0000 7fa65e444700 1 -- 172.21.15.42:0/775231242 >> v1:172.21.15.42:6789/0 conn(0x7fa658124c70 legacy=0x7fa658130390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:55:44.342 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:44.326+0000 7fa65e444700 1 -- 172.21.15.42:0/775231242 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa65812d320 con 0x7fa6581242f0 2019-12-05T05:55:44.342 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:44.326+0000 7fa65ec45700 1 --2- 172.21.15.42:0/775231242 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa65811aa30 0x7fa65812cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:55:44.343 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:44.326+0000 7fa65e444700 1 --2- 172.21.15.42:0/775231242 >> v2:172.21.15.40:3300/0 conn(0x7fa6581242f0 0x7fa658124720 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fa648002f30 tx=0x7fa648010b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:44.343 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:44.326+0000 7fa65e444700 1 --2- 172.21.15.42:0/775231242 >> v2:172.21.15.40:3300/0 conn(0x7fa6581242f0 0x7fa658124720 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fa648002f30 tx=0x7fa648010b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:44.344 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:44.326+0000 7fa65e444700 1 --2- 172.21.15.42:0/775231242 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa6581242f0 0x7fa658124720 secure :-1 s=READY pgs=1309 cs=0 l=1 rx=0x7fa648002f30 tx=0x7fa648010b70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:55:44.344 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:44.326+0000 7fa65e444700 1 --2- 172.21.15.42:0/775231242 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa6581242f0 0x7fa658124720 secure :-1 s=THROTTLE_DONE pgs=1309 cs=0 l=1 rx=0x7fa648002f30 tx=0x7fa648010b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:44.344 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:44.326+0000 7fa64f7fe700 1 -- 172.21.15.42:0/775231242 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa648012620 con 0x7fa6581242f0 2019-12-05T05:55:44.344 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:44.326+0000 7fa65e444700 1 --2- 172.21.15.42:0/775231242 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa6581242f0 0x7fa658124720 secure :-1 s=THROTTLE_DONE pgs=1309 cs=0 l=1 rx=0x7fa648002f30 tx=0x7fa648010b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:44.344 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:44.326+0000 7fa65e444700 1 --2- 172.21.15.42:0/775231242 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa6581242f0 0x7fa658124720 secure :-1 s=THROTTLE_DONE pgs=1309 cs=0 l=1 rx=0x7fa648002f30 tx=0x7fa648010b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:44.345 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:44.326+0000 7fa64f7fe700 1 -- 172.21.15.42:0/775231242 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa648007150 con 0x7fa6581242f0 2019-12-05T05:55:44.345 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:44.326+0000 7fa64f7fe700 1 -- 172.21.15.42:0/775231242 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa648012c00 con 0x7fa6581242f0 2019-12-05T05:55:44.345 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:44.326+0000 7fa6606a8700 1 -- 172.21.15.42:0/775231242 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa65811ae60 con 0x7fa6581242f0 2019-12-05T05:55:44.345 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:44.326+0000 7fa6606a8700 1 -- 172.21.15.42:0/775231242 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa658130b90 con 0x7fa6581242f0 2019-12-05T05:55:44.346 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:44.326+0000 7fa65e444700 1 --2- 172.21.15.42:0/775231242 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa6581242f0 0x7fa658124720 secure :-1 s=THROTTLE_DONE pgs=1309 cs=0 l=1 rx=0x7fa648002f30 tx=0x7fa648010b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:44.346 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:44.330+0000 7fa65e444700 1 --2- 172.21.15.42:0/775231242 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa6581242f0 0x7fa658124720 secure :-1 s=THROTTLE_DONE pgs=1309 cs=0 l=1 rx=0x7fa648002f30 tx=0x7fa648010b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:44.346 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:44.330+0000 7fa64f7fe700 1 -- 172.21.15.42:0/775231242 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fa648005540 con 0x7fa6581242f0 2019-12-05T05:55:44.346 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:44.330+0000 7fa64f7fe700 1 --2- 172.21.15.42:0/775231242 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa64003b260 0x7fa64003d770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:55:44.346 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:44.330+0000 7fa64f7fe700 1 -- 172.21.15.42:0/775231242 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 9127+0+0 (secure 0 0 0) 0x7fa64804b150 con 0x7fa6581242f0 2019-12-05T05:55:44.346 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:44.330+0000 7fa65ec45700 1 --2- 172.21.15.42:0/775231242 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa64003b260 0x7fa64003d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:55:44.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:44.330+0000 7fa65ec45700 1 --2- 172.21.15.42:0/775231242 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa64003b260 0x7fa64003d770 crc :-1 s=READY pgs=737 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:55:44.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:44.334+0000 7fa6606a8700 1 -- 172.21.15.42:0/775231242 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa65805d110 con 0x7fa6581242f0 2019-12-05T05:55:44.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:44.342+0000 7fa65e444700 1 --2- 172.21.15.42:0/775231242 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa6581242f0 0x7fa658124720 secure :-1 s=THROTTLE_DONE pgs=1309 cs=0 l=1 rx=0x7fa648002f30 tx=0x7fa648010b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:44.358 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:44.342+0000 7fa64f7fe700 1 -- 172.21.15.42:0/775231242 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fa648032190 con 0x7fa6581242f0 2019-12-05T05:55:44.525 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:44.518+0000 7fa6606a8700 1 -- 172.21.15.42:0/775231242 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa658123b90 con 0x7fa6581242f0 2019-12-05T05:55:44.525 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:44.518+0000 7fa65e444700 1 --2- 172.21.15.42:0/775231242 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa6581242f0 0x7fa658124720 secure :-1 s=THROTTLE_DONE pgs=1309 cs=0 l=1 rx=0x7fa648002f30 tx=0x7fa648010b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:44.526 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:44.518+0000 7fa64f7fe700 1 -- 172.21.15.42:0/775231242 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 212 v212) v1 ==== 95+0+1775 (secure 0 0 0) 0x7fa6480323f0 con 0x7fa6581242f0 2019-12-05T05:55:44.526 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 212 2019-12-05T05:55:44.526 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:55:44.526 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":212,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":212,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:55:44.050934+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6461},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6461":{"gid":6461,"name":"b","rank":0,"incarnation":209,"state":"up:active","state_seq":12,"addr":"172.21.15.40:6833/3549993215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3549993215},{"type":"v1","addr":"172.21.15.40:6833","nonce":3549993215}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:55:44.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:44.522+0000 7fa6606a8700 1 -- 172.21.15.42:0/775231242 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa64003b260 msgr2=0x7fa64003d770 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:55:44.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:44.522+0000 7fa6606a8700 1 --2- 172.21.15.42:0/775231242 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa64003b260 0x7fa64003d770 crc :-1 s=READY pgs=737 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:55:44.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:44.522+0000 7fa6606a8700 1 -- 172.21.15.42:0/775231242 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa6581242f0 msgr2=0x7fa658124720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:55:44.530 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:44.522+0000 7fa6606a8700 1 --2- 172.21.15.42:0/775231242 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa6581242f0 0x7fa658124720 secure :-1 s=READY pgs=1309 cs=0 l=1 rx=0x7fa648002f30 tx=0x7fa648010b70).stop 2019-12-05T05:55:44.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:44.522+0000 7fa6606a8700 1 -- 172.21.15.42:0/775231242 shutdown_connections 2019-12-05T05:55:44.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:44.522+0000 7fa6606a8700 1 --2- 172.21.15.42:0/775231242 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa64003b260 0x7fa64003d770 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:55:44.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:44.522+0000 7fa6606a8700 1 --2- 172.21.15.42:0/775231242 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa65811aa30 0x7fa65812cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:55:44.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:44.522+0000 7fa6606a8700 1 --2- 172.21.15.42:0/775231242 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa6581242f0 0x7fa658124720 unknown :-1 s=CLOSED pgs=1309 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:55:44.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:44.522+0000 7fa6606a8700 1 -- 172.21.15.42:0/775231242 shutdown_connections 2019-12-05T05:55:44.539 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:44.522+0000 7fa6606a8700 1 -- 172.21.15.42:0/775231242 wait complete. 2019-12-05T05:55:44.539 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:44.522+0000 7fa6606a8700 1 -- 172.21.15.42:0/775231242 >> 172.21.15.42:0/775231242 conn(0x7fa65808f880 msgr2=0x7fa658059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:55:44.540 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b has gained rank=0, replacing gid=8614 2019-12-05T05:55:44.540 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 1 secs before reviving mds.a 2019-12-05T05:55:45.541 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.a 2019-12-05T05:55:45.541 INFO:tasks.ceph.mds.a:Restarting daemon 2019-12-05T05:55:45.542 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2019-12-05T05:55:45.545 INFO:tasks.ceph.mds.a:Started 2019-12-05T05:55:45.546 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:55:45.604 INFO:tasks.ceph.mds.a.smithi042.stdout:starting mds.a at 2019-12-05T05:55:45.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:45.638+0000 7fa26abd8700 1 Processor -- start 2019-12-05T05:55:45.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:45.638+0000 7fa26abd8700 1 -- start start 2019-12-05T05:55:45.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:45.638+0000 7fa26abd8700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa2640a7740 0x7fa26408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:55:45.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:45.638+0000 7fa26abd8700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fa26409fae0 0x7fa26407eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:55:45.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:45.638+0000 7fa26abd8700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2640a0710 con 0x7fa26407e720 2019-12-05T05:55:45.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:45.638+0000 7fa26abd8700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2640a0b00 con 0x7fa26409fae0 2019-12-05T05:55:45.657 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:45.638+0000 7fa26abd8700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa26408e5b0 con 0x7fa2640a7740 2019-12-05T05:55:45.657 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:45.638+0000 7fa268974700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fa26407e720 0x7fa26409f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:44316/0 (socket says 172.21.15.42:44316) 2019-12-05T05:55:45.657 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:45.638+0000 7fa268974700 1 -- 172.21.15.42:0/2860360386 learned_addr learned my addr 172.21.15.42:0/2860360386 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:55:45.657 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:45.638+0000 7fa269976700 1 --2- 172.21.15.42:0/2860360386 >> v2:172.21.15.42:3300/0 conn(0x7fa26409fae0 0x7fa26407eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:55:45.658 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:45.638+0000 7fa269175700 1 --2- 172.21.15.42:0/2860360386 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa2640a7740 0x7fa26408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:55:45.658 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:45.638+0000 7fa25bfff700 1 -- 172.21.15.42:0/2860360386 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1109697236 0 0) 0x7fa2640a0710 con 0x7fa26407e720 2019-12-05T05:55:45.658 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:45.638+0000 7fa25bfff700 1 -- 172.21.15.42:0/2860360386 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2500030b0 con 0x7fa26407e720 2019-12-05T05:55:45.658 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:45.638+0000 7fa269976700 1 -- 172.21.15.42:0/2860360386 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa2640a7740 msgr2=0x7fa26408df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:55:45.659 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:45.638+0000 7fa269976700 1 --2- 172.21.15.42:0/2860360386 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa2640a7740 0x7fa26408df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:55:45.659 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:45.638+0000 7fa269976700 1 -- 172.21.15.42:0/2860360386 >> v1:172.21.15.42:6789/0 conn(0x7fa26407e720 legacy=0x7fa26409f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:55:45.659 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:45.638+0000 7fa269976700 1 -- 172.21.15.42:0/2860360386 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa26407f0a0 con 0x7fa26409fae0 2019-12-05T05:55:45.659 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:45.642+0000 7fa269976700 1 --2- 172.21.15.42:0/2860360386 >> v2:172.21.15.42:3300/0 conn(0x7fa26409fae0 0x7fa26407eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fa25c008b10 tx=0x7fa25c00fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:45.659 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:45.642+0000 7fa269976700 1 --2- 172.21.15.42:0/2860360386 >> v2:172.21.15.42:3300/0 conn(0x7fa26409fae0 0x7fa26407eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fa25c008b10 tx=0x7fa25c00fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:45.660 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:45.642+0000 7fa269976700 1 --2- 172.21.15.42:0/2860360386 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa26409fae0 0x7fa26407eb50 secure :-1 s=READY pgs=1728 cs=0 l=1 rx=0x7fa25c008b10 tx=0x7fa25c00fff0).ready entity=mon.1 client_cookie=3e9c279b8168c8f8 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:55:45.660 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:45.642+0000 7fa269976700 1 --2- 172.21.15.42:0/2860360386 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa26409fae0 0x7fa26407eb50 secure :-1 s=THROTTLE_DONE pgs=1728 cs=0 l=1 rx=0x7fa25c008b10 tx=0x7fa25c00fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:45.660 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:45.642+0000 7fa25bfff700 1 -- 172.21.15.42:0/2860360386 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa25c010900 con 0x7fa26409fae0 2019-12-05T05:55:45.660 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:45.642+0000 7fa269976700 1 --2- 172.21.15.42:0/2860360386 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa26409fae0 0x7fa26407eb50 secure :-1 s=THROTTLE_DONE pgs=1728 cs=0 l=1 rx=0x7fa25c008b10 tx=0x7fa25c00fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:45.661 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:45.642+0000 7fa269976700 1 --2- 172.21.15.42:0/2860360386 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa26409fae0 0x7fa26407eb50 secure :-1 s=THROTTLE_DONE pgs=1728 cs=0 l=1 rx=0x7fa25c008b10 tx=0x7fa25c00fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:45.661 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:45.642+0000 7fa25bfff700 1 -- 172.21.15.42:0/2860360386 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa25c008b50 con 0x7fa26409fae0 2019-12-05T05:55:45.661 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:45.642+0000 7fa25bfff700 1 -- 172.21.15.42:0/2860360386 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa25c0110c0 con 0x7fa26409fae0 2019-12-05T05:55:45.661 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:45.642+0000 7fa26abd8700 1 -- 172.21.15.42:0/2860360386 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa26409fae0 msgr2=0x7fa26407eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:55:45.661 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:45.642+0000 7fa26abd8700 1 --2- 172.21.15.42:0/2860360386 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa26409fae0 0x7fa26407eb50 secure :-1 s=READY pgs=1728 cs=0 l=1 rx=0x7fa25c008b10 tx=0x7fa25c00fff0).stop 2019-12-05T05:55:45.661 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:45.642+0000 7fa26abd8700 1 -- 172.21.15.42:0/2860360386 shutdown_connections 2019-12-05T05:55:45.661 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:45.642+0000 7fa26abd8700 1 --2- 172.21.15.42:0/2860360386 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa26409fae0 0x7fa26407eb50 unknown :-1 s=CLOSED pgs=1728 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:55:45.662 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:45.642+0000 7fa26abd8700 1 --2- 172.21.15.42:0/2860360386 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa2640a7740 0x7fa26408df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:55:45.662 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:45.642+0000 7fa26abd8700 1 -- 172.21.15.42:0/2860360386 shutdown_connections 2019-12-05T05:55:45.662 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:45.642+0000 7fa26abd8700 1 -- 172.21.15.42:0/2860360386 wait complete. 2019-12-05T05:55:45.662 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:45.642+0000 7fa26abd8700 1 -- 172.21.15.42:0/2860360386 >> 172.21.15.42:0/2860360386 conn(0x7fa2640aabe0 msgr2=0x7fa2640a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:55:45.662 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:45.642+0000 7fa26abd8700 1 Processor -- start 2019-12-05T05:55:45.662 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:45.642+0000 7fa26abd8700 1 -- start start 2019-12-05T05:55:45.662 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:45.642+0000 7fa26abd8700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa2640a57b0 0x7fa26412cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:55:45.662 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:45.642+0000 7fa26abd8700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2640a5e50 con 0x7fa2640a5380 2019-12-05T05:55:45.663 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:45.642+0000 7fa26abd8700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa26411ab40 con 0x7fa264123010 2019-12-05T05:55:45.663 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:45.642+0000 7fa26abd8700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2640a5be0 con 0x7fa2640a57b0 2019-12-05T05:55:45.663 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:45.646+0000 7fa269175700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fa264123010 0x7fa264119830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:44320/0 (socket says 172.21.15.42:44320) 2019-12-05T05:55:45.663 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:45.646+0000 7fa269175700 1 -- 172.21.15.42:0/3176053367 learned_addr learned my addr 172.21.15.42:0/3176053367 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:55:45.663 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:45.646+0000 7fa269976700 1 --2- 172.21.15.42:0/3176053367 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa2640a57b0 0x7fa26412cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:55:45.663 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:45.646+0000 7fa259ffb700 1 -- 172.21.15.42:0/3176053367 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1061306 0 0) 0x7fa26411ab40 con 0x7fa264123010 2019-12-05T05:55:45.663 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:45.646+0000 7fa259ffb700 1 -- 172.21.15.42:0/3176053367 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2440030b0 con 0x7fa264123010 2019-12-05T05:55:45.664 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:45.646+0000 7fa259ffb700 1 -- 172.21.15.42:0/3176053367 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3168986327 0 0) 0x7fa2640a5e50 con 0x7fa2640a5380 2019-12-05T05:55:45.664 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:45.646+0000 7fa259ffb700 1 -- 172.21.15.42:0/3176053367 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa26411ab40 con 0x7fa2640a5380 2019-12-05T05:55:45.664 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:45.646+0000 7fa259ffb700 1 -- 172.21.15.42:0/3176053367 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 2522487985 0 0) 0x7fa2440030b0 con 0x7fa264123010 2019-12-05T05:55:45.664 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:45.646+0000 7fa269976700 1 -- 172.21.15.42:0/3176053367 >> v1:172.21.15.42:6789/0 conn(0x7fa264123010 legacy=0x7fa264119830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:55:45.664 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:45.646+0000 7fa269976700 1 -- 172.21.15.42:0/3176053367 >> v1:172.21.15.40:6789/0 conn(0x7fa2640a5380 legacy=0x7fa2641294f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:55:45.664 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:45.646+0000 7fa269976700 1 -- 172.21.15.42:0/3176053367 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa264137990 con 0x7fa2640a57b0 2019-12-05T05:55:45.664 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:45.646+0000 7fa269976700 1 --2- 172.21.15.42:0/3176053367 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa2640a57b0 0x7fa26412cd10 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fa2600016d0 tx=0x7fa260002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:45.665 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:45.646+0000 7fa269976700 1 --2- 172.21.15.42:0/3176053367 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa2640a57b0 0x7fa26412cd10 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fa2600016d0 tx=0x7fa260002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:45.665 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:45.646+0000 7fa269976700 1 --2- 172.21.15.42:0/3176053367 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa2640a57b0 0x7fa26412cd10 secure :-1 s=READY pgs=387 cs=0 l=1 rx=0x7fa2600016d0 tx=0x7fa260002ff0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:55:45.665 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:45.646+0000 7fa269976700 1 --2- 172.21.15.42:0/3176053367 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa2640a57b0 0x7fa26412cd10 secure :-1 s=THROTTLE_DONE pgs=387 cs=0 l=1 rx=0x7fa2600016d0 tx=0x7fa260002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:45.665 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:45.646+0000 7fa259ffb700 1 -- 172.21.15.42:0/3176053367 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa260001490 con 0x7fa2640a57b0 2019-12-05T05:55:45.665 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:45.646+0000 7fa269976700 1 --2- 172.21.15.42:0/3176053367 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa2640a57b0 0x7fa26412cd10 secure :-1 s=THROTTLE_DONE pgs=387 cs=0 l=1 rx=0x7fa2600016d0 tx=0x7fa260002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:45.665 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:45.646+0000 7fa269976700 1 --2- 172.21.15.42:0/3176053367 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa2640a57b0 0x7fa26412cd10 secure :-1 s=THROTTLE_DONE pgs=387 cs=0 l=1 rx=0x7fa2600016d0 tx=0x7fa260002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:45.665 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:45.646+0000 7fa259ffb700 1 -- 172.21.15.42:0/3176053367 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2600012a0 con 0x7fa2640a57b0 2019-12-05T05:55:45.666 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:45.646+0000 7fa26abd8700 1 -- 172.21.15.42:0/3176053367 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa264123440 con 0x7fa2640a57b0 2019-12-05T05:55:45.666 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:45.646+0000 7fa259ffb700 1 -- 172.21.15.42:0/3176053367 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa25c005ca0 con 0x7fa2640a57b0 2019-12-05T05:55:45.666 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:45.646+0000 7fa26abd8700 1 -- 172.21.15.42:0/3176053367 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa26412d260 con 0x7fa2640a57b0 2019-12-05T05:55:45.666 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:45.646+0000 7fa269976700 1 --2- 172.21.15.42:0/3176053367 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa2640a57b0 0x7fa26412cd10 secure :-1 s=THROTTLE_DONE pgs=387 cs=0 l=1 rx=0x7fa2600016d0 tx=0x7fa260002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:45.666 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:45.650+0000 7fa259ffb700 1 -- 172.21.15.42:0/3176053367 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fa25c00f280 con 0x7fa2640a57b0 2019-12-05T05:55:45.666 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:45.650+0000 7fa269976700 1 --2- 172.21.15.42:0/3176053367 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa2640a57b0 0x7fa26412cd10 secure :-1 s=THROTTLE_DONE pgs=387 cs=0 l=1 rx=0x7fa2600016d0 tx=0x7fa260002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:45.667 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:45.650+0000 7fa259ffb700 1 --2- 172.21.15.42:0/3176053367 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa24403b410 0x7fa24403d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:55:45.667 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:45.650+0000 7fa259ffb700 1 -- 172.21.15.42:0/3176053367 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 9127+0+0 (secure 0 0 0) 0x7fa25c08eb10 con 0x7fa2640a57b0 2019-12-05T05:55:45.667 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:45.650+0000 7fa269175700 1 --2- 172.21.15.42:0/3176053367 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa24403b410 0x7fa24403d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:55:45.668 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:45.650+0000 7fa269175700 1 --2- 172.21.15.42:0/3176053367 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa24403b410 0x7fa24403d920 crc :-1 s=READY pgs=738 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:55:45.668 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:45.654+0000 7fa26abd8700 1 -- 172.21.15.42:0/3176053367 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa26405d110 con 0x7fa2640a57b0 2019-12-05T05:55:45.668 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:45.662+0000 7fa269976700 1 --2- 172.21.15.42:0/3176053367 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa2640a57b0 0x7fa26412cd10 secure :-1 s=THROTTLE_DONE pgs=387 cs=0 l=1 rx=0x7fa2600016d0 tx=0x7fa260002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:45.669 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:45.662+0000 7fa259ffb700 1 -- 172.21.15.42:0/3176053367 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fa25c075df0 con 0x7fa2640a57b0 2019-12-05T05:55:45.840 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:45.838+0000 7fa26abd8700 1 -- 172.21.15.42:0/3176053367 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa26405b8c0 con 0x7fa2640a57b0 2019-12-05T05:55:45.841 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:45.838+0000 7fa269976700 1 --2- 172.21.15.42:0/3176053367 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa2640a57b0 0x7fa26412cd10 secure :-1 s=THROTTLE_DONE pgs=387 cs=0 l=1 rx=0x7fa2600016d0 tx=0x7fa260002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:45.841 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:45.838+0000 7fa259ffb700 1 -- 172.21.15.42:0/3176053367 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 212 v212) v1 ==== 95+0+1775 (secure 0 0 0) 0x7fa25c0769a0 con 0x7fa2640a57b0 2019-12-05T05:55:45.842 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 212 2019-12-05T05:55:45.842 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:55:45.842 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":212,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":212,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:55:44.050934+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6461},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6461":{"gid":6461,"name":"b","rank":0,"incarnation":209,"state":"up:active","state_seq":12,"addr":"172.21.15.40:6833/3549993215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3549993215},{"type":"v1","addr":"172.21.15.40:6833","nonce":3549993215}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:55:45.845 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:45.842+0000 7fa26abd8700 1 -- 172.21.15.42:0/3176053367 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa24403b410 msgr2=0x7fa24403d920 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:55:45.845 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:45.842+0000 7fa26abd8700 1 --2- 172.21.15.42:0/3176053367 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa24403b410 0x7fa24403d920 crc :-1 s=READY pgs=738 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:55:45.845 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:45.842+0000 7fa26abd8700 1 -- 172.21.15.42:0/3176053367 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa2640a57b0 msgr2=0x7fa26412cd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:55:45.845 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:45.842+0000 7fa26abd8700 1 --2- 172.21.15.42:0/3176053367 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa2640a57b0 0x7fa26412cd10 secure :-1 s=READY pgs=387 cs=0 l=1 rx=0x7fa2600016d0 tx=0x7fa260002ff0).stop 2019-12-05T05:55:45.845 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:45.842+0000 7fa26abd8700 1 -- 172.21.15.42:0/3176053367 shutdown_connections 2019-12-05T05:55:45.845 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:45.842+0000 7fa26abd8700 1 --2- 172.21.15.42:0/3176053367 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa24403b410 0x7fa24403d920 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:55:45.845 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:45.842+0000 7fa26abd8700 1 --2- 172.21.15.42:0/3176053367 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa2640a57b0 0x7fa26412cd10 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:55:45.846 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:45.842+0000 7fa26abd8700 1 -- 172.21.15.42:0/3176053367 shutdown_connections 2019-12-05T05:55:45.846 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:45.842+0000 7fa26abd8700 1 -- 172.21.15.42:0/3176053367 wait complete. 2019-12-05T05:55:45.846 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:45.842+0000 7fa26abd8700 1 -- 172.21.15.42:0/3176053367 >> 172.21.15.42:0/3176053367 conn(0x7fa2640a7f90 msgr2=0x7fa26408eb70 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:55:45.860 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2019-12-05T05:55:47.860 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:55:47.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:47.958+0000 7fcce7b87700 1 Processor -- start 2019-12-05T05:55:47.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:47.958+0000 7fcce7b87700 1 -- start start 2019-12-05T05:55:47.976 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:47.958+0000 7fcce7b87700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcce00a7740 0x7fcce008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:55:47.976 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:47.958+0000 7fcce7b87700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fcce007e720 0x7fcce007eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:55:47.976 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:47.958+0000 7fcce7b87700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcce00a0710 con 0x7fcce009f2e0 2019-12-05T05:55:47.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:47.958+0000 7fcce7b87700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcce00a0b00 con 0x7fcce007e720 2019-12-05T05:55:47.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:47.958+0000 7fcce7b87700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcce008e5b0 con 0x7fcce00a7740 2019-12-05T05:55:47.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:47.958+0000 7fcce5923700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fcce007e720 0x7fcce007eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:55:47.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:47.958+0000 7fcce6925700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fcce009f2e0 0x7fcce009f710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:44342/0 (socket says 172.21.15.42:44342) 2019-12-05T05:55:47.978 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:47.958+0000 7fcce6925700 1 -- 172.21.15.42:0/1873920155 learned_addr learned my addr 172.21.15.42:0/1873920155 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:55:47.978 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:47.958+0000 7fcce6124700 1 --2- 172.21.15.42:0/1873920155 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcce00a7740 0x7fcce008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:55:47.978 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:47.958+0000 7fcce5923700 1 -- 172.21.15.42:0/1873920155 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcce00a7740 msgr2=0x7fcce008df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:55:47.978 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:47.958+0000 7fcce5923700 1 --2- 172.21.15.42:0/1873920155 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcce00a7740 0x7fcce008df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:55:47.978 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:47.958+0000 7fcce5923700 1 -- 172.21.15.42:0/1873920155 >> v1:172.21.15.42:6789/0 conn(0x7fcce009f2e0 legacy=0x7fcce009f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:55:47.979 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:47.958+0000 7fcce5122700 1 -- 172.21.15.42:0/1873920155 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1492846541 0 0) 0x7fcce00a0710 con 0x7fcce009f2e0 2019-12-05T05:55:47.979 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:47.958+0000 7fcce5923700 1 -- 172.21.15.42:0/1873920155 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcce007f0a0 con 0x7fcce007e720 2019-12-05T05:55:47.979 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:47.958+0000 7fcce6124700 1 --2- 172.21.15.42:0/1873920155 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcce00a7740 0x7fcce008df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T05:55:47.979 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:47.958+0000 7fcce5923700 1 --2- 172.21.15.42:0/1873920155 >> v2:172.21.15.42:3300/0 conn(0x7fcce007e720 0x7fcce007eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fccd0008790 tx=0x7fccd000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:47.979 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:47.958+0000 7fcce5923700 1 --2- 172.21.15.42:0/1873920155 >> v2:172.21.15.42:3300/0 conn(0x7fcce007e720 0x7fcce007eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fccd0008790 tx=0x7fccd000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:47.979 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:47.958+0000 7fcce5923700 1 --2- 172.21.15.42:0/1873920155 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcce007e720 0x7fcce007eb50 secure :-1 s=READY pgs=1731 cs=0 l=1 rx=0x7fccd0008790 tx=0x7fccd000fc70).ready entity=mon.1 client_cookie=68f14dae5082322f server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:55:47.980 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:47.958+0000 7fcce5923700 1 --2- 172.21.15.42:0/1873920155 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcce007e720 0x7fcce007eb50 secure :-1 s=THROTTLE_DONE pgs=1731 cs=0 l=1 rx=0x7fccd0008790 tx=0x7fccd000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:47.980 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:47.958+0000 7fcce5122700 1 -- 172.21.15.42:0/1873920155 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fccd00112f0 con 0x7fcce007e720 2019-12-05T05:55:47.980 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:47.958+0000 7fcce5923700 1 --2- 172.21.15.42:0/1873920155 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcce007e720 0x7fcce007eb50 secure :-1 s=THROTTLE_DONE pgs=1731 cs=0 l=1 rx=0x7fccd0008790 tx=0x7fccd000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:47.981 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:47.958+0000 7fcce5122700 1 -- 172.21.15.42:0/1873920155 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccd0008ad0 con 0x7fcce007e720 2019-12-05T05:55:47.981 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:47.958+0000 7fcce5923700 1 --2- 172.21.15.42:0/1873920155 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcce007e720 0x7fcce007eb50 secure :-1 s=THROTTLE_DONE pgs=1731 cs=0 l=1 rx=0x7fccd0008790 tx=0x7fccd000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:47.981 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:47.958+0000 7fcce7b87700 1 -- 172.21.15.42:0/1873920155 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcce007e720 msgr2=0x7fcce007eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:55:47.981 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:47.958+0000 7fcce5122700 1 -- 172.21.15.42:0/1873920155 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fccd00118d0 con 0x7fcce007e720 2019-12-05T05:55:47.981 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:47.958+0000 7fcce7b87700 1 --2- 172.21.15.42:0/1873920155 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcce007e720 0x7fcce007eb50 secure :-1 s=READY pgs=1731 cs=0 l=1 rx=0x7fccd0008790 tx=0x7fccd000fc70).stop 2019-12-05T05:55:47.981 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:47.962+0000 7fcce7b87700 1 -- 172.21.15.42:0/1873920155 shutdown_connections 2019-12-05T05:55:47.981 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:47.962+0000 7fcce7b87700 1 --2- 172.21.15.42:0/1873920155 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcce00a7740 0x7fcce008df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:55:47.982 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:47.962+0000 7fcce7b87700 1 --2- 172.21.15.42:0/1873920155 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcce007e720 0x7fcce007eb50 unknown :-1 s=CLOSED pgs=1731 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:55:47.982 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:47.962+0000 7fcce7b87700 1 -- 172.21.15.42:0/1873920155 shutdown_connections 2019-12-05T05:55:47.982 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:47.962+0000 7fcce7b87700 1 -- 172.21.15.42:0/1873920155 wait complete. 2019-12-05T05:55:47.982 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:47.962+0000 7fcce7b87700 1 -- 172.21.15.42:0/1873920155 >> 172.21.15.42:0/1873920155 conn(0x7fcce00aabe0 msgr2=0x7fcce00a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:55:47.982 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:47.962+0000 7fcce7b87700 1 Processor -- start 2019-12-05T05:55:47.982 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:47.962+0000 7fcce7b87700 1 -- start start 2019-12-05T05:55:47.982 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:47.962+0000 7fcce7b87700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fcce0125650 0x7fcce00a3900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:55:47.983 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:47.962+0000 7fcce7b87700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcce011bb80 0x7fcce011bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:55:47.983 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:47.962+0000 7fcce7b87700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcce00a3e50 con 0x7fcce008eb70 2019-12-05T05:55:47.983 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:47.962+0000 7fcce7b87700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcce011b920 con 0x7fcce0125650 2019-12-05T05:55:47.983 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:47.962+0000 7fcce7b87700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcce00a41f0 con 0x7fcce011bb80 2019-12-05T05:55:47.983 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:47.966+0000 7fcce6124700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fcce0125650 0x7fcce00a3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:55:47.983 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:47.966+0000 7fcce5923700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcce011bb80 0x7fcce011bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:55:47.984 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:47.966+0000 7fcce6124700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fcce0125650 0x7fcce00a3900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:44306/0 (socket says 172.21.15.42:44306) 2019-12-05T05:55:47.984 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:47.966+0000 7fcce6124700 1 -- 172.21.15.42:0/471857226 learned_addr learned my addr 172.21.15.42:0/471857226 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:55:47.984 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:47.966+0000 7fccd6ffd700 1 -- 172.21.15.42:0/471857226 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3169794433 0 0) 0x7fcce00a3e50 con 0x7fcce008eb70 2019-12-05T05:55:47.984 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:47.966+0000 7fcce6124700 1 -- 172.21.15.42:0/471857226 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcce011bb80 msgr2=0x7fcce011bfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:55:47.984 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:47.966+0000 7fcce6124700 1 --2- 172.21.15.42:0/471857226 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcce011bb80 0x7fcce011bfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:55:47.993 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:47.966+0000 7fcce6124700 1 -- 172.21.15.42:0/471857226 >> v1:172.21.15.40:6789/0 conn(0x7fcce008eb70 legacy=0x7fcce008efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:55:47.993 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:47.966+0000 7fcce6124700 1 -- 172.21.15.42:0/471857226 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcce0124790 con 0x7fcce0125650 2019-12-05T05:55:47.994 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:47.966+0000 7fcce6124700 1 --2- 172.21.15.42:0/471857226 >> v2:172.21.15.42:3300/0 conn(0x7fcce0125650 0x7fcce00a3900 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fcce00a43b0 tx=0x7fccdc010d80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:47.994 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:47.966+0000 7fcce6124700 1 --2- 172.21.15.42:0/471857226 >> v2:172.21.15.42:3300/0 conn(0x7fcce0125650 0x7fcce00a3900 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fcce00a43b0 tx=0x7fccdc010d80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:47.994 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:47.966+0000 7fcce6124700 1 --2- 172.21.15.42:0/471857226 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcce0125650 0x7fcce00a3900 secure :-1 s=READY pgs=1732 cs=0 l=1 rx=0x7fcce00a43b0 tx=0x7fccdc010d80).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:55:47.994 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:47.966+0000 7fcce6124700 1 --2- 172.21.15.42:0/471857226 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcce0125650 0x7fcce00a3900 secure :-1 s=THROTTLE_DONE pgs=1732 cs=0 l=1 rx=0x7fcce00a43b0 tx=0x7fccdc010d80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:47.994 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:47.966+0000 7fccd6ffd700 1 -- 172.21.15.42:0/471857226 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fccdc012b90 con 0x7fcce0125650 2019-12-05T05:55:47.995 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:47.966+0000 7fcce6124700 1 --2- 172.21.15.42:0/471857226 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcce0125650 0x7fcce00a3900 secure :-1 s=THROTTLE_DONE pgs=1732 cs=0 l=1 rx=0x7fcce00a43b0 tx=0x7fccdc010d80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:47.995 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:47.966+0000 7fcce7b87700 1 -- 172.21.15.42:0/471857226 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcce0124df0 con 0x7fcce0125650 2019-12-05T05:55:47.995 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:47.966+0000 7fccd6ffd700 1 -- 172.21.15.42:0/471857226 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccdc0127b0 con 0x7fcce0125650 2019-12-05T05:55:47.995 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:47.966+0000 7fcce6124700 1 --2- 172.21.15.42:0/471857226 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcce0125650 0x7fcce00a3900 secure :-1 s=THROTTLE_DONE pgs=1732 cs=0 l=1 rx=0x7fcce00a43b0 tx=0x7fccdc010d80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:47.995 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:47.966+0000 7fccd6ffd700 1 -- 172.21.15.42:0/471857226 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fccdc009340 con 0x7fcce0125650 2019-12-05T05:55:47.995 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:47.966+0000 7fcce7b87700 1 -- 172.21.15.42:0/471857226 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcce008f7a0 con 0x7fcce0125650 2019-12-05T05:55:47.996 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:47.966+0000 7fcce6124700 1 --2- 172.21.15.42:0/471857226 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcce0125650 0x7fcce00a3900 secure :-1 s=THROTTLE_DONE pgs=1732 cs=0 l=1 rx=0x7fcce00a43b0 tx=0x7fccdc010d80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:47.996 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:47.966+0000 7fccd6ffd700 1 -- 172.21.15.42:0/471857226 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fccdc0058c0 con 0x7fcce0125650 2019-12-05T05:55:47.996 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:47.966+0000 7fcce6124700 1 --2- 172.21.15.42:0/471857226 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcce0125650 0x7fcce00a3900 secure :-1 s=THROTTLE_DONE pgs=1732 cs=0 l=1 rx=0x7fcce00a43b0 tx=0x7fccdc010d80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:47.996 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:47.966+0000 7fccd6ffd700 1 --2- 172.21.15.42:0/471857226 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fccc803abf0 0x7fccc803d100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:55:47.996 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:47.966+0000 7fccd6ffd700 1 -- 172.21.15.42:0/471857226 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 9127+0+0 (secure 0 0 0) 0x7fccdc04b480 con 0x7fcce0125650 2019-12-05T05:55:47.996 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:47.970+0000 7fcce5923700 1 --2- 172.21.15.42:0/471857226 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fccc803abf0 0x7fccc803d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:55:47.996 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:47.970+0000 7fcce5923700 1 --2- 172.21.15.42:0/471857226 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fccc803abf0 0x7fccc803d100 crc :-1 s=READY pgs=741 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:55:47.997 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:47.974+0000 7fcce7b87700 1 -- 172.21.15.42:0/471857226 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcce005d110 con 0x7fcce0125650 2019-12-05T05:55:47.997 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:47.982+0000 7fcce6124700 1 --2- 172.21.15.42:0/471857226 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcce0125650 0x7fcce00a3900 secure :-1 s=THROTTLE_DONE pgs=1732 cs=0 l=1 rx=0x7fcce00a43b0 tx=0x7fccdc010d80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:47.998 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:47.982+0000 7fccd6ffd700 1 -- 172.21.15.42:0/471857226 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fccdc032480 con 0x7fcce0125650 2019-12-05T05:55:48.155 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:48.154+0000 7fcce7b87700 1 -- 172.21.15.42:0/471857226 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcce005b8c0 con 0x7fcce0125650 2019-12-05T05:55:48.156 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:48.154+0000 7fcce6124700 1 --2- 172.21.15.42:0/471857226 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcce0125650 0x7fcce00a3900 secure :-1 s=THROTTLE_DONE pgs=1732 cs=0 l=1 rx=0x7fcce00a43b0 tx=0x7fccdc010d80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:55:48.156 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:48.154+0000 7fccd6ffd700 1 -- 172.21.15.42:0/471857226 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+2112 (secure 0 0 0) 0x7fccdc0326e0 con 0x7fcce0125650 2019-12-05T05:55:48.157 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 213 2019-12-05T05:55:48.157 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:55:48.157 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":213,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":8836,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6835/1368848584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1368848584},{"type":"v1","addr":"172.21.15.42:6835","nonce":1368848584}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":213}],"filesystems":[{"mdsmap":{"epoch":212,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:55:44.050934+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6461},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6461":{"gid":6461,"name":"b","rank":0,"incarnation":209,"state":"up:active","state_seq":12,"addr":"172.21.15.40:6833/3549993215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3549993215},{"type":"v1","addr":"172.21.15.40:6833","nonce":3549993215}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:55:48.161 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:48.154+0000 7fcce7b87700 1 -- 172.21.15.42:0/471857226 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fccc803abf0 msgr2=0x7fccc803d100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:55:48.161 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:48.154+0000 7fcce7b87700 1 --2- 172.21.15.42:0/471857226 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fccc803abf0 0x7fccc803d100 crc :-1 s=READY pgs=741 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:55:48.161 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:48.154+0000 7fcce7b87700 1 -- 172.21.15.42:0/471857226 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcce0125650 msgr2=0x7fcce00a3900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:55:48.162 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:48.154+0000 7fcce7b87700 1 --2- 172.21.15.42:0/471857226 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcce0125650 0x7fcce00a3900 secure :-1 s=READY pgs=1732 cs=0 l=1 rx=0x7fcce00a43b0 tx=0x7fccdc010d80).stop 2019-12-05T05:55:48.162 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:48.154+0000 7fcce7b87700 1 -- 172.21.15.42:0/471857226 shutdown_connections 2019-12-05T05:55:48.164 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:48.154+0000 7fcce7b87700 1 --2- 172.21.15.42:0/471857226 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fccc803abf0 0x7fccc803d100 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:55:48.164 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:48.154+0000 7fcce7b87700 1 --2- 172.21.15.42:0/471857226 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcce0125650 0x7fcce00a3900 unknown :-1 s=CLOSED pgs=1732 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:55:48.164 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:48.154+0000 7fcce7b87700 1 --2- 172.21.15.42:0/471857226 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcce011bb80 0x7fcce011bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:55:48.164 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:48.154+0000 7fcce7b87700 1 -- 172.21.15.42:0/471857226 shutdown_connections 2019-12-05T05:55:48.164 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:48.154+0000 7fcce7b87700 1 -- 172.21.15.42:0/471857226 wait complete. 2019-12-05T05:55:48.164 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:55:48.154+0000 7fcce7b87700 1 -- 172.21.15.42:0/471857226 >> 172.21.15.42:0/471857226 conn(0x7fcce00a82a0 msgr2=0x7fcce00a9790 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:55:48.169 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a reported in up:standby state 2019-12-05T05:55:48.169 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 81 secs before thrashing 2019-12-05T05:56:12.498 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:56:12.503 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:56:42.564 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:56:42.568 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:57:09.170 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:57:09.277 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.269+0000 7f8ecbebe700 1 Processor -- start 2019-12-05T05:57:09.278 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.269+0000 7f8ecbebe700 1 -- start start 2019-12-05T05:57:09.278 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.269+0000 7f8ecbebe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f8ec409fae0 0x7f8ec414fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:57:09.278 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.269+0000 7f8ecbebe700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ec40a0710 con 0x7f8ec408df80 2019-12-05T05:57:09.278 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.269+0000 7f8ecbebe700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ec407ef20 con 0x7f8ec40a7740 2019-12-05T05:57:09.278 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.269+0000 7f8ecbebe700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ec40a0b00 con 0x7f8ec409fae0 2019-12-05T05:57:09.278 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.269+0000 7f8ecac5c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f8ec409fae0 0x7f8ec414fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:57:09.279 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.269+0000 7f8eca45b700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f8ec40a7740 0x7f8ec407e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:44668/0 (socket says 172.21.15.42:44668) 2019-12-05T05:57:09.279 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.269+0000 7f8ecac5c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f8ec409fae0 0x7f8ec414fff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:44634/0 (socket says 172.21.15.42:44634) 2019-12-05T05:57:09.279 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.269+0000 7f8eca45b700 1 -- 172.21.15.42:0/2077858126 learned_addr learned my addr 172.21.15.42:0/2077858126 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:57:09.279 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.269+0000 7f8ecac5c700 1 -- 172.21.15.42:0/2077858126 >> v1:172.21.15.42:6789/0 conn(0x7f8ec40a7740 legacy=0x7f8ec407e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:57:09.279 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.269+0000 7f8ecac5c700 1 -- 172.21.15.42:0/2077858126 >> v1:172.21.15.40:6789/0 conn(0x7f8ec408df80 legacy=0x7f8ec409f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:57:09.279 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.269+0000 7f8ecac5c700 1 -- 172.21.15.42:0/2077858126 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ec40a9ba0 con 0x7f8ec409fae0 2019-12-05T05:57:09.279 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.273+0000 7f8ecac5c700 1 --2- 172.21.15.42:0/2077858126 >> v2:172.21.15.42:3300/0 conn(0x7f8ec409fae0 0x7f8ec414fff0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f8ebc008b10 tx=0x7f8ebc00ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:09.280 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.273+0000 7f8ecac5c700 1 --2- 172.21.15.42:0/2077858126 >> v2:172.21.15.42:3300/0 conn(0x7f8ec409fae0 0x7f8ec414fff0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f8ebc008b10 tx=0x7f8ebc00ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:09.280 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.273+0000 7f8ecac5c700 1 --2- 172.21.15.42:0/2077858126 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8ec409fae0 0x7f8ec414fff0 secure :-1 s=READY pgs=1736 cs=0 l=1 rx=0x7f8ebc008b10 tx=0x7f8ebc00ff70).ready entity=mon.1 client_cookie=2a072f865365fd24 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:57:09.280 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.273+0000 7f8ecac5c700 1 --2- 172.21.15.42:0/2077858126 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8ec409fae0 0x7f8ec414fff0 secure :-1 s=THROTTLE_DONE pgs=1736 cs=0 l=1 rx=0x7f8ebc008b10 tx=0x7f8ebc00ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:09.280 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.273+0000 7f8ecac5c700 1 --2- 172.21.15.42:0/2077858126 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8ec409fae0 0x7f8ec414fff0 secure :-1 s=THROTTLE_DONE pgs=1736 cs=0 l=1 rx=0x7f8ebc008b10 tx=0x7f8ebc00ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:09.280 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.273+0000 7f8ecac5c700 1 --2- 172.21.15.42:0/2077858126 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8ec409fae0 0x7f8ec414fff0 secure :-1 s=THROTTLE_DONE pgs=1736 cs=0 l=1 rx=0x7f8ebc008b10 tx=0x7f8ebc00ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:09.281 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.273+0000 7f8ec9459700 1 -- 172.21.15.42:0/2077858126 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8ebc010970 con 0x7f8ec409fae0 2019-12-05T05:57:09.281 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.273+0000 7f8ec9459700 1 -- 172.21.15.42:0/2077858126 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ebc008b50 con 0x7f8ec409fae0 2019-12-05T05:57:09.281 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.273+0000 7f8ec9459700 1 -- 172.21.15.42:0/2077858126 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8ebc0110c0 con 0x7f8ec409fae0 2019-12-05T05:57:09.281 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.273+0000 7f8ecbebe700 1 -- 172.21.15.42:0/2077858126 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8ec409fae0 msgr2=0x7f8ec414fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:57:09.281 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.273+0000 7f8ecbebe700 1 --2- 172.21.15.42:0/2077858126 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8ec409fae0 0x7f8ec414fff0 secure :-1 s=READY pgs=1736 cs=0 l=1 rx=0x7f8ebc008b10 tx=0x7f8ebc00ff70).stop 2019-12-05T05:57:09.281 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.273+0000 7f8ecbebe700 1 -- 172.21.15.42:0/2077858126 shutdown_connections 2019-12-05T05:57:09.281 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.273+0000 7f8ecbebe700 1 --2- 172.21.15.42:0/2077858126 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8ec409fae0 0x7f8ec414fff0 unknown :-1 s=CLOSED pgs=1736 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:57:09.282 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.273+0000 7f8ecbebe700 1 -- 172.21.15.42:0/2077858126 shutdown_connections 2019-12-05T05:57:09.282 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.273+0000 7f8ecbebe700 1 -- 172.21.15.42:0/2077858126 wait complete. 2019-12-05T05:57:09.282 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.273+0000 7f8ecbebe700 1 -- 172.21.15.42:0/2077858126 >> 172.21.15.42:0/2077858126 conn(0x7f8ec40aabe0 msgr2=0x7f8ec40a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:57:09.286 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.273+0000 7f8ecbebe700 1 Processor -- start 2019-12-05T05:57:09.286 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.277+0000 7f8ecbebe700 1 -- start start 2019-12-05T05:57:09.286 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.277+0000 7f8ecbebe700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8ec4061940 0x7f8ec412f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:57:09.286 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.277+0000 7f8ecbebe700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ec4061d70 con 0x7f8ec41236f0 2019-12-05T05:57:09.286 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.277+0000 7f8ecbebe700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ec4065a80 con 0x7f8ec4144c60 2019-12-05T05:57:09.286 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.277+0000 7f8ecbebe700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ec4126d50 con 0x7f8ec4061940 2019-12-05T05:57:09.287 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.277+0000 7f8eca45b700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f8ec4144c60 0x7f8ec4127530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:44674/0 (socket says 172.21.15.42:44674) 2019-12-05T05:57:09.287 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.277+0000 7f8eca45b700 1 -- 172.21.15.42:0/4277212175 learned_addr learned my addr 172.21.15.42:0/4277212175 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:57:09.287 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.277+0000 7f8ecac5c700 1 --2- 172.21.15.42:0/4277212175 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8ec4061940 0x7f8ec412f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:57:09.287 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.277+0000 7f8ebaffd700 1 -- 172.21.15.42:0/4277212175 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3769471544 0 0) 0x7f8ec4065a80 con 0x7f8ec4144c60 2019-12-05T05:57:09.287 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.277+0000 7f8ebaffd700 1 -- 172.21.15.42:0/4277212175 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8eac0030b0 con 0x7f8ec4144c60 2019-12-05T05:57:09.287 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.277+0000 7f8ebaffd700 1 -- 172.21.15.42:0/4277212175 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3052816800 0 0) 0x7f8ec40a0710 con 0x7f8ec41236f0 2019-12-05T05:57:09.287 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.277+0000 7f8ebaffd700 1 -- 172.21.15.42:0/4277212175 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ec4065a80 con 0x7f8ec41236f0 2019-12-05T05:57:09.288 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.277+0000 7f8ecac5c700 1 -- 172.21.15.42:0/4277212175 >> v1:172.21.15.42:6789/0 conn(0x7f8ec4144c60 legacy=0x7f8ec4127530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:57:09.288 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.277+0000 7f8ecac5c700 1 -- 172.21.15.42:0/4277212175 >> v1:172.21.15.40:6789/0 conn(0x7f8ec41236f0 legacy=0x7f8ec4065280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:57:09.288 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.277+0000 7f8ecac5c700 1 -- 172.21.15.42:0/4277212175 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ec41452d0 con 0x7f8ec4061940 2019-12-05T05:57:09.288 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.277+0000 7f8ecac5c700 1 --2- 172.21.15.42:0/4277212175 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8ec4061940 0x7f8ec412f110 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f8ebc008b10 tx=0x7f8ebc0112f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:09.288 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.277+0000 7f8ecac5c700 1 --2- 172.21.15.42:0/4277212175 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8ec4061940 0x7f8ec412f110 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f8ebc008b10 tx=0x7f8ebc0112f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:09.288 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.277+0000 7f8ecac5c700 1 --2- 172.21.15.42:0/4277212175 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8ec4061940 0x7f8ec412f110 secure :-1 s=READY pgs=390 cs=0 l=1 rx=0x7f8ebc008b10 tx=0x7f8ebc0112f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:57:09.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.277+0000 7f8ecac5c700 1 --2- 172.21.15.42:0/4277212175 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8ec4061940 0x7f8ec412f110 secure :-1 s=THROTTLE_DONE pgs=390 cs=0 l=1 rx=0x7f8ebc008b10 tx=0x7f8ebc0112f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:09.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.277+0000 7f8ebaffd700 1 -- 172.21.15.42:0/4277212175 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8ebc005ab0 con 0x7f8ec4061940 2019-12-05T05:57:09.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.277+0000 7f8ecac5c700 1 --2- 172.21.15.42:0/4277212175 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8ec4061940 0x7f8ec412f110 secure :-1 s=THROTTLE_DONE pgs=390 cs=0 l=1 rx=0x7f8ebc008b10 tx=0x7f8ebc0112f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:09.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.277+0000 7f8ecac5c700 1 --2- 172.21.15.42:0/4277212175 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8ec4061940 0x7f8ec412f110 secure :-1 s=THROTTLE_DONE pgs=390 cs=0 l=1 rx=0x7f8ebc008b10 tx=0x7f8ebc0112f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:09.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.277+0000 7f8ebaffd700 1 -- 172.21.15.42:0/4277212175 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ebc010b70 con 0x7f8ec4061940 2019-12-05T05:57:09.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.277+0000 7f8ecbebe700 1 -- 172.21.15.42:0/4277212175 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ec412f660 con 0x7f8ec4061940 2019-12-05T05:57:09.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.277+0000 7f8ebaffd700 1 -- 172.21.15.42:0/4277212175 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8ebc00f680 con 0x7f8ec4061940 2019-12-05T05:57:09.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.277+0000 7f8ecbebe700 1 -- 172.21.15.42:0/4277212175 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ec412fb30 con 0x7f8ec4061940 2019-12-05T05:57:09.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.281+0000 7f8ecac5c700 1 --2- 172.21.15.42:0/4277212175 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8ec4061940 0x7f8ec412f110 secure :-1 s=THROTTLE_DONE pgs=390 cs=0 l=1 rx=0x7f8ebc008b10 tx=0x7f8ebc0112f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:09.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.281+0000 7f8ecac5c700 1 --2- 172.21.15.42:0/4277212175 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8ec4061940 0x7f8ec412f110 secure :-1 s=THROTTLE_DONE pgs=390 cs=0 l=1 rx=0x7f8ebc008b10 tx=0x7f8ebc0112f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:09.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.281+0000 7f8ebaffd700 1 -- 172.21.15.42:0/4277212175 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f8ebc0113d0 con 0x7f8ec4061940 2019-12-05T05:57:09.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.281+0000 7f8ebaffd700 1 --2- 172.21.15.42:0/4277212175 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8eac03b520 0x7f8eac03da30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:57:09.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.281+0000 7f8ebaffd700 1 -- 172.21.15.42:0/4277212175 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 9127+0+0 (secure 0 0 0) 0x7f8ebc04b130 con 0x7f8ec4061940 2019-12-05T05:57:09.291 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.281+0000 7f8eca45b700 1 --2- 172.21.15.42:0/4277212175 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8eac03b520 0x7f8eac03da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:57:09.299 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.281+0000 7f8eca45b700 1 --2- 172.21.15.42:0/4277212175 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8eac03b520 0x7f8eac03da30 crc :-1 s=READY pgs=743 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:57:09.305 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.285+0000 7f8ecbebe700 1 -- 172.21.15.42:0/4277212175 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ec405d110 con 0x7f8ec4061940 2019-12-05T05:57:09.305 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.293+0000 7f8ecac5c700 1 --2- 172.21.15.42:0/4277212175 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8ec4061940 0x7f8ec412f110 secure :-1 s=THROTTLE_DONE pgs=390 cs=0 l=1 rx=0x7f8ebc008b10 tx=0x7f8ebc0112f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:09.305 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.293+0000 7f8ebaffd700 1 -- 172.21.15.42:0/4277212175 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f8ebc011cd0 con 0x7f8ec4061940 2019-12-05T05:57:09.467 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.465+0000 7f8ecbebe700 1 -- 172.21.15.42:0/4277212175 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8ec405b8c0 con 0x7f8ec4061940 2019-12-05T05:57:09.468 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.465+0000 7f8ecac5c700 1 --2- 172.21.15.42:0/4277212175 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8ec4061940 0x7f8ec412f110 secure :-1 s=THROTTLE_DONE pgs=390 cs=0 l=1 rx=0x7f8ebc008b10 tx=0x7f8ebc0112f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:09.468 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.465+0000 7f8ebaffd700 1 -- 172.21.15.42:0/4277212175 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f8ebc032b50 con 0x7f8ec4061940 2019-12-05T05:57:09.471 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 213 2019-12-05T05:57:09.471 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:57:09.471 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":213,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":8836,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6835/1368848584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1368848584},{"type":"v1","addr":"172.21.15.42:6835","nonce":1368848584}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":213}],"filesystems":[{"mdsmap":{"epoch":212,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:55:44.050934+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6461},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6461":{"gid":6461,"name":"b","rank":0,"incarnation":209,"state":"up:active","state_seq":12,"addr":"172.21.15.40:6833/3549993215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3549993215},{"type":"v1","addr":"172.21.15.40:6833","nonce":3549993215}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:57:09.477 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.469+0000 7f8ecbebe700 1 -- 172.21.15.42:0/4277212175 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8eac03b520 msgr2=0x7f8eac03da30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:57:09.477 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.469+0000 7f8ecbebe700 1 --2- 172.21.15.42:0/4277212175 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8eac03b520 0x7f8eac03da30 crc :-1 s=READY pgs=743 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:57:09.477 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.469+0000 7f8ecbebe700 1 -- 172.21.15.42:0/4277212175 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8ec4061940 msgr2=0x7f8ec412f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:57:09.477 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.469+0000 7f8ecbebe700 1 --2- 172.21.15.42:0/4277212175 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8ec4061940 0x7f8ec412f110 secure :-1 s=READY pgs=390 cs=0 l=1 rx=0x7f8ebc008b10 tx=0x7f8ebc0112f0).stop 2019-12-05T05:57:09.477 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.469+0000 7f8ecbebe700 1 -- 172.21.15.42:0/4277212175 shutdown_connections 2019-12-05T05:57:09.478 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.469+0000 7f8ecbebe700 1 --2- 172.21.15.42:0/4277212175 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8eac03b520 0x7f8eac03da30 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:57:09.478 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.469+0000 7f8ecbebe700 1 --2- 172.21.15.42:0/4277212175 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8ec4061940 0x7f8ec412f110 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:57:09.478 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.469+0000 7f8ecbebe700 1 -- 172.21.15.42:0/4277212175 shutdown_connections 2019-12-05T05:57:09.478 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.469+0000 7f8ecbebe700 1 -- 172.21.15.42:0/4277212175 wait complete. 2019-12-05T05:57:09.478 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.469+0000 7f8ecbebe700 1 -- 172.21.15.42:0/4277212175 >> 172.21.15.42:0/4277212175 conn(0x7f8ec40a7740 msgr2=0x7f8ec40a6a60 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:57:09.489 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.b (rank=0) 2019-12-05T05:57:09.492 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2019-12-05T05:57:09.493 INFO:teuthology.orchestra.run:waiting for 300 2019-12-05T05:57:09.521 INFO:tasks.ceph.mds.b:Stopped 2019-12-05T05:57:09.521 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:57:09.626 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.617+0000 7f775e0b1700 1 Processor -- start 2019-12-05T05:57:09.627 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.617+0000 7f775e0b1700 1 -- start start 2019-12-05T05:57:09.627 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.617+0000 7f775e0b1700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f775808df80 0x7f775809f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:57:09.627 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.617+0000 7f775e0b1700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f775809fc60 con 0x7f775809f830 2019-12-05T05:57:09.628 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.617+0000 7f775e0b1700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77580a0710 con 0x7f77580a7740 2019-12-05T05:57:09.628 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.617+0000 7f775e0b1700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77580a0b00 con 0x7f775808df80 2019-12-05T05:57:09.628 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.617+0000 7f7757fff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f77580a7740 0x7f775807e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:44684/0 (socket says 172.21.15.42:44684) 2019-12-05T05:57:09.628 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.617+0000 7f7757fff700 1 -- 172.21.15.42:0/646235499 learned_addr learned my addr 172.21.15.42:0/646235499 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:57:09.629 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.617+0000 7f77577fe700 1 --2- 172.21.15.42:0/646235499 >> v2:172.21.15.40:3300/0 conn(0x7f775808df80 0x7f775809f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:57:09.629 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.617+0000 7f7756ffd700 1 -- 172.21.15.42:0/646235499 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2351291348 0 0) 0x7f77580a0710 con 0x7f77580a7740 2019-12-05T05:57:09.629 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.617+0000 7f7756ffd700 1 -- 172.21.15.42:0/646235499 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77400030b0 con 0x7f77580a7740 2019-12-05T05:57:09.629 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.621+0000 7f7756ffd700 1 -- 172.21.15.42:0/646235499 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1504684194 0 0) 0x7f77400030b0 con 0x7f77580a7740 2019-12-05T05:57:09.629 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.621+0000 7f77577fe700 1 -- 172.21.15.42:0/646235499 >> v1:172.21.15.42:6789/0 conn(0x7f77580a7740 legacy=0x7f775807e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:57:09.630 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.621+0000 7f77577fe700 1 -- 172.21.15.42:0/646235499 >> v1:172.21.15.40:6789/0 conn(0x7f775809f830 legacy=0x7f775814efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:57:09.630 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.621+0000 7f77577fe700 1 -- 172.21.15.42:0/646235499 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f775807ef20 con 0x7f775808df80 2019-12-05T05:57:09.630 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.621+0000 7f77577fe700 1 --2- 172.21.15.42:0/646235499 >> v2:172.21.15.40:3300/0 conn(0x7f775808df80 0x7f775809f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f774c008790 tx=0x7f7748001990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:09.630 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.621+0000 7f77577fe700 1 --2- 172.21.15.42:0/646235499 >> v2:172.21.15.40:3300/0 conn(0x7f775808df80 0x7f775809f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f774c008790 tx=0x7f7748001990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:09.631 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.621+0000 7f77577fe700 1 --2- 172.21.15.42:0/646235499 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f775808df80 0x7f775809f2e0 secure :-1 s=READY pgs=1320 cs=0 l=1 rx=0x7f774c008790 tx=0x7f7748001990).ready entity=mon.0 client_cookie=b0a5ea7776f3af6e server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:57:09.631 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.621+0000 7f77577fe700 1 --2- 172.21.15.42:0/646235499 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f775808df80 0x7f775809f2e0 secure :-1 s=THROTTLE_DONE pgs=1320 cs=0 l=1 rx=0x7f774c008790 tx=0x7f7748001990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:09.631 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.621+0000 7f7756ffd700 1 -- 172.21.15.42:0/646235499 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7748001750 con 0x7f775808df80 2019-12-05T05:57:09.631 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.621+0000 7f77577fe700 1 --2- 172.21.15.42:0/646235499 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f775808df80 0x7f775809f2e0 secure :-1 s=THROTTLE_DONE pgs=1320 cs=0 l=1 rx=0x7f774c008790 tx=0x7f7748001990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:09.632 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.621+0000 7f77577fe700 1 --2- 172.21.15.42:0/646235499 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f775808df80 0x7f775809f2e0 secure :-1 s=THROTTLE_DONE pgs=1320 cs=0 l=1 rx=0x7f774c008790 tx=0x7f7748001990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:09.632 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.621+0000 7f7756ffd700 1 -- 172.21.15.42:0/646235499 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7748001560 con 0x7f775808df80 2019-12-05T05:57:09.632 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.621+0000 7f7756ffd700 1 -- 172.21.15.42:0/646235499 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f774c011360 con 0x7f775808df80 2019-12-05T05:57:09.632 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.621+0000 7f775e0b1700 1 -- 172.21.15.42:0/646235499 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f775808df80 msgr2=0x7f775809f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:57:09.632 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.621+0000 7f775e0b1700 1 --2- 172.21.15.42:0/646235499 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f775808df80 0x7f775809f2e0 secure :-1 s=READY pgs=1320 cs=0 l=1 rx=0x7f774c008790 tx=0x7f7748001990).stop 2019-12-05T05:57:09.633 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.621+0000 7f775e0b1700 1 -- 172.21.15.42:0/646235499 shutdown_connections 2019-12-05T05:57:09.633 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.621+0000 7f775e0b1700 1 --2- 172.21.15.42:0/646235499 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f775808df80 0x7f775809f2e0 unknown :-1 s=CLOSED pgs=1320 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:57:09.633 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.621+0000 7f775e0b1700 1 -- 172.21.15.42:0/646235499 shutdown_connections 2019-12-05T05:57:09.633 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.621+0000 7f775e0b1700 1 -- 172.21.15.42:0/646235499 wait complete. 2019-12-05T05:57:09.634 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.621+0000 7f775e0b1700 1 -- 172.21.15.42:0/646235499 >> 172.21.15.42:0/646235499 conn(0x7f77580aabe0 msgr2=0x7f77580a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:57:09.640 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.625+0000 7f775e0b1700 1 Processor -- start 2019-12-05T05:57:09.640 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.625+0000 7f775e0b1700 1 -- start start 2019-12-05T05:57:09.640 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.625+0000 7f775e0b1700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7758117b90 0x7f7758070cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:57:09.641 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.625+0000 7f775e0b1700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7758071220 con 0x7f7758116500 2019-12-05T05:57:09.641 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.625+0000 7f775e0b1700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7758117670 con 0x7f775811bfd0 2019-12-05T05:57:09.641 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.625+0000 7f775e0b1700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f775811c400 con 0x7f7758117b90 2019-12-05T05:57:09.641 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.625+0000 7f77577fe700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f775811bfd0 0x7f77580704d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:44690/0 (socket says 172.21.15.42:44690) 2019-12-05T05:57:09.642 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.625+0000 7f77577fe700 1 -- 172.21.15.42:0/4132944878 learned_addr learned my addr 172.21.15.42:0/4132944878 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:57:09.642 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.625+0000 7f775ce4f700 1 --2- 172.21.15.42:0/4132944878 >> v2:172.21.15.40:3300/0 conn(0x7f7758117b90 0x7f7758070cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:57:09.642 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.625+0000 7f77577fe700 1 -- 172.21.15.42:0/4132944878 >> v1:172.21.15.42:6789/0 conn(0x7f775811bfd0 legacy=0x7f77580704d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2019-12-05T05:57:09.642 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.625+0000 7f77577fe700 1 -- 172.21.15.42:0/4132944878 >> v1:172.21.15.42:6789/0 conn(0x7f775811bfd0 legacy=0x7f77580704d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2019-12-05T05:57:09.643 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.625+0000 7f77577fe700 1 --1- 172.21.15.42:0/4132944878 >> v1:172.21.15.42:6789/0 conn(0x7f775811bfd0 0x7f77580704d0 :-1 s=OPENED pgs=1739 cs=1 l=1).handle_message read tag failed 2019-12-05T05:57:09.643 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.625+0000 7f77577fe700 1 --1- 172.21.15.42:0/4132944878 >> v1:172.21.15.42:6789/0 conn(0x7f775811bfd0 0x7f77580704d0 :-1 s=OPENED pgs=1739 cs=1 l=1).fault on lossy channel, failing 2019-12-05T05:57:09.643 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.625+0000 7f7754ff9700 1 -- 172.21.15.42:0/4132944878 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3953654474 0 0) 0x7f7758071220 con 0x7f7758116500 2019-12-05T05:57:09.643 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.625+0000 7f7754ff9700 1 -- 172.21.15.42:0/4132944878 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77380030b0 con 0x7f7758116500 2019-12-05T05:57:09.643 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.625+0000 7f775ce4f700 1 -- 172.21.15.42:0/4132944878 >> v1:172.21.15.42:6789/0 conn(0x7f775811bfd0 legacy=0x7f77580704d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2019-12-05T05:57:09.644 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.625+0000 7f775ce4f700 1 -- 172.21.15.42:0/4132944878 >> v1:172.21.15.40:6789/0 conn(0x7f7758116500 legacy=0x7f775811b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:57:09.644 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.625+0000 7f775ce4f700 1 -- 172.21.15.42:0/4132944878 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f775811abf0 con 0x7f7758117b90 2019-12-05T05:57:09.644 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.625+0000 7f775ce4f700 1 --2- 172.21.15.42:0/4132944878 >> v2:172.21.15.40:3300/0 conn(0x7f7758117b90 0x7f7758070cd0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f7750000f00 tx=0x7f7750010a80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:09.644 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.625+0000 7f775ce4f700 1 --2- 172.21.15.42:0/4132944878 >> v2:172.21.15.40:3300/0 conn(0x7f7758117b90 0x7f7758070cd0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f7750000f00 tx=0x7f7750010a80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:09.645 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.625+0000 7f775ce4f700 1 --2- 172.21.15.42:0/4132944878 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7758117b90 0x7f7758070cd0 secure :-1 s=READY pgs=1322 cs=0 l=1 rx=0x7f7750000f00 tx=0x7f7750010a80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:57:09.645 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.625+0000 7f775ce4f700 1 --2- 172.21.15.42:0/4132944878 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7758117b90 0x7f7758070cd0 secure :-1 s=THROTTLE_DONE pgs=1322 cs=0 l=1 rx=0x7f7750000f00 tx=0x7f7750010a80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:09.645 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.625+0000 7f7754ff9700 1 -- 172.21.15.42:0/4132944878 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7750002220 con 0x7f7758117b90 2019-12-05T05:57:09.645 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.625+0000 7f775ce4f700 1 --2- 172.21.15.42:0/4132944878 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7758117b90 0x7f7758070cd0 secure :-1 s=THROTTLE_DONE pgs=1322 cs=0 l=1 rx=0x7f7750000f00 tx=0x7f7750010a80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:09.645 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.625+0000 7f775ce4f700 1 --2- 172.21.15.42:0/4132944878 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7758117b90 0x7f7758070cd0 secure :-1 s=THROTTLE_DONE pgs=1322 cs=0 l=1 rx=0x7f7750000f00 tx=0x7f7750010a80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:09.646 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.625+0000 7f7754ff9700 1 -- 172.21.15.42:0/4132944878 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7750006ad0 con 0x7f7758117b90 2019-12-05T05:57:09.646 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.625+0000 7f7754ff9700 1 -- 172.21.15.42:0/4132944878 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7750002800 con 0x7f7758117b90 2019-12-05T05:57:09.646 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.625+0000 7f775e0b1700 1 -- 172.21.15.42:0/4132944878 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7758117fc0 con 0x7f7758117b90 2019-12-05T05:57:09.646 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.625+0000 7f775e0b1700 1 -- 172.21.15.42:0/4132944878 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7758119f80 con 0x7f7758117b90 2019-12-05T05:57:09.647 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.629+0000 7f775ce4f700 1 --2- 172.21.15.42:0/4132944878 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7758117b90 0x7f7758070cd0 secure :-1 s=THROTTLE_DONE pgs=1322 cs=0 l=1 rx=0x7f7750000f00 tx=0x7f7750010a80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:09.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.629+0000 7f7754ff9700 1 -- 172.21.15.42:0/4132944878 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f7750002ad0 con 0x7f7758117b90 2019-12-05T05:57:09.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.629+0000 7f775ce4f700 1 --2- 172.21.15.42:0/4132944878 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7758117b90 0x7f7758070cd0 secure :-1 s=THROTTLE_DONE pgs=1322 cs=0 l=1 rx=0x7f7750000f00 tx=0x7f7750010a80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:09.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.629+0000 7f7754ff9700 1 --2- 172.21.15.42:0/4132944878 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f773803b1e0 0x7f773803d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:57:09.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.629+0000 7f7754ff9700 1 -- 172.21.15.42:0/4132944878 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 9127+0+0 (secure 0 0 0) 0x7f775004b3a0 con 0x7f7758117b90 2019-12-05T05:57:09.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.629+0000 7f7757fff700 1 --2- 172.21.15.42:0/4132944878 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f773803b1e0 0x7f773803d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:57:09.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.629+0000 7f7757fff700 1 --2- 172.21.15.42:0/4132944878 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f773803b1e0 0x7f773803d6f0 crc :-1 s=READY pgs=744 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:57:09.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.633+0000 7f775e0b1700 1 -- 172.21.15.42:0/4132944878 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f775805d110 con 0x7f7758117b90 2019-12-05T05:57:09.650 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.641+0000 7f775ce4f700 1 --2- 172.21.15.42:0/4132944878 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7758117b90 0x7f7758070cd0 secure :-1 s=THROTTLE_DONE pgs=1322 cs=0 l=1 rx=0x7f7750000f00 tx=0x7f7750010a80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:09.650 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.641+0000 7f7754ff9700 1 -- 172.21.15.42:0/4132944878 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f7750002de0 con 0x7f7758117b90 2019-12-05T05:57:09.814 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.809+0000 7f775e0b1700 1 -- 172.21.15.42:0/4132944878 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f775811bb10 con 0x7f7758117b90 2019-12-05T05:57:09.815 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.813+0000 7f775ce4f700 1 --2- 172.21.15.42:0/4132944878 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7758117b90 0x7f7758070cd0 secure :-1 s=THROTTLE_DONE pgs=1322 cs=0 l=1 rx=0x7f7750000f00 tx=0x7f7750010a80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:09.815 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.813+0000 7f7754ff9700 1 -- 172.21.15.42:0/4132944878 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f7750032420 con 0x7f7758117b90 2019-12-05T05:57:09.817 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 213 2019-12-05T05:57:09.817 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:57:09.818 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":213,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":8836,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6835/1368848584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1368848584},{"type":"v1","addr":"172.21.15.42:6835","nonce":1368848584}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":213}],"filesystems":[{"mdsmap":{"epoch":212,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:55:44.050934+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6461},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6461":{"gid":6461,"name":"b","rank":0,"incarnation":209,"state":"up:active","state_seq":12,"addr":"172.21.15.40:6833/3549993215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3549993215},{"type":"v1","addr":"172.21.15.40:6833","nonce":3549993215}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:57:09.820 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.817+0000 7f775e0b1700 1 -- 172.21.15.42:0/4132944878 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f773803b1e0 msgr2=0x7f773803d6f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:57:09.820 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.817+0000 7f775e0b1700 1 --2- 172.21.15.42:0/4132944878 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f773803b1e0 0x7f773803d6f0 crc :-1 s=READY pgs=744 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:57:09.820 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.817+0000 7f775e0b1700 1 -- 172.21.15.42:0/4132944878 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7758117b90 msgr2=0x7f7758070cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:57:09.821 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.817+0000 7f775e0b1700 1 --2- 172.21.15.42:0/4132944878 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7758117b90 0x7f7758070cd0 secure :-1 s=READY pgs=1322 cs=0 l=1 rx=0x7f7750000f00 tx=0x7f7750010a80).stop 2019-12-05T05:57:09.821 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.817+0000 7f775e0b1700 1 -- 172.21.15.42:0/4132944878 shutdown_connections 2019-12-05T05:57:09.821 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.817+0000 7f775e0b1700 1 --2- 172.21.15.42:0/4132944878 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f773803b1e0 0x7f773803d6f0 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:57:09.821 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.817+0000 7f775e0b1700 1 --2- 172.21.15.42:0/4132944878 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7758117b90 0x7f7758070cd0 unknown :-1 s=CLOSED pgs=1322 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:57:09.821 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.817+0000 7f775e0b1700 1 -- 172.21.15.42:0/4132944878 shutdown_connections 2019-12-05T05:57:09.821 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.817+0000 7f775e0b1700 1 -- 172.21.15.42:0/4132944878 wait complete. 2019-12-05T05:57:09.822 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:09.817+0000 7f775e0b1700 1 -- 172.21.15.42:0/4132944878 >> 172.21.15.42:0/4132944878 conn(0x7f775809f240 msgr2=0x7f77580a98f0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:57:09.830 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T05:57:11.831 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:57:11.936 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:11.925+0000 7fc672cea700 1 Processor -- start 2019-12-05T05:57:11.937 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:11.925+0000 7fc672cea700 1 -- start start 2019-12-05T05:57:11.937 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:11.929+0000 7fc672cea700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc66c0a7740 0x7fc66c08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:57:11.937 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:11.929+0000 7fc672cea700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc66c07e720 0x7fc66c07eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:57:11.937 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:11.929+0000 7fc672cea700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc66c09f2e0 0x7fc66c09f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:57:11.938 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:11.929+0000 7fc672cea700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc66c0a0b00 con 0x7fc66c07e720 2019-12-05T05:57:11.938 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:11.929+0000 7fc672cea700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc66c08e5b0 con 0x7fc66c09f2e0 2019-12-05T05:57:11.938 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:11.929+0000 7fc672cea700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc66c08e850 con 0x7fc66c0a7740 2019-12-05T05:57:11.938 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:11.929+0000 7fc671a88700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc66c09f2e0 0x7fc66c09f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:57:11.938 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:11.929+0000 7fc671a88700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc66c09f2e0 0x7fc66c09f710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:44668/0 (socket says 172.21.15.42:44668) 2019-12-05T05:57:11.938 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:11.929+0000 7fc671a88700 1 -- 172.21.15.42:0/1331507272 learned_addr learned my addr 172.21.15.42:0/1331507272 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:57:11.939 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:11.929+0000 7fc671287700 1 --2- 172.21.15.42:0/1331507272 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc66c0a7740 0x7fc66c08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:57:11.939 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:11.929+0000 7fc670a86700 1 --2- 172.21.15.42:0/1331507272 >> v2:172.21.15.40:3300/0 conn(0x7fc66c07e720 0x7fc66c07eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:57:11.939 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:11.929+0000 7fc671a88700 1 -- 172.21.15.42:0/1331507272 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc66c0a7740 msgr2=0x7fc66c08df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:57:11.939 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:11.929+0000 7fc671a88700 1 --2- 172.21.15.42:0/1331507272 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc66c0a7740 0x7fc66c08df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:57:11.939 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:11.929+0000 7fc671a88700 1 -- 172.21.15.42:0/1331507272 >> v2:172.21.15.40:3300/0 conn(0x7fc66c07e720 msgr2=0x7fc66c07eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:57:11.939 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:11.929+0000 7fc671a88700 1 --2- 172.21.15.42:0/1331507272 >> v2:172.21.15.40:3300/0 conn(0x7fc66c07e720 0x7fc66c07eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:57:11.940 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:11.929+0000 7fc671a88700 1 -- 172.21.15.42:0/1331507272 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc66c07f0a0 con 0x7fc66c09f2e0 2019-12-05T05:57:11.940 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:11.929+0000 7fc670a86700 1 --2- 172.21.15.42:0/1331507272 >> v2:172.21.15.40:3300/0 conn(0x7fc66c07e720 0x7fc66c07eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T05:57:11.940 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:11.929+0000 7fc671a88700 1 --2- 172.21.15.42:0/1331507272 >> v2:172.21.15.42:3300/0 conn(0x7fc66c09f2e0 0x7fc66c09f710 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fc66c08e770 tx=0x7fc664010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:11.940 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:11.929+0000 7fc671a88700 1 --2- 172.21.15.42:0/1331507272 >> v2:172.21.15.42:3300/0 conn(0x7fc66c09f2e0 0x7fc66c09f710 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fc66c08e770 tx=0x7fc664010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:11.949 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:11.929+0000 7fc671a88700 1 --2- 172.21.15.42:0/1331507272 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc66c09f2e0 0x7fc66c09f710 secure :-1 s=READY pgs=1740 cs=0 l=1 rx=0x7fc66c08e770 tx=0x7fc664010130).ready entity=mon.1 client_cookie=ab4522a862bafb14 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:57:11.949 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:11.929+0000 7fc671a88700 1 --2- 172.21.15.42:0/1331507272 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc66c09f2e0 0x7fc66c09f710 secure :-1 s=THROTTLE_DONE pgs=1740 cs=0 l=1 rx=0x7fc66c08e770 tx=0x7fc664010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:11.949 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:11.929+0000 7fc671a88700 1 --2- 172.21.15.42:0/1331507272 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc66c09f2e0 0x7fc66c09f710 secure :-1 s=THROTTLE_DONE pgs=1740 cs=0 l=1 rx=0x7fc66c08e770 tx=0x7fc664010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:11.949 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:11.929+0000 7fc663fff700 1 -- 172.21.15.42:0/1331507272 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc664010de0 con 0x7fc66c09f2e0 2019-12-05T05:57:11.950 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:11.929+0000 7fc671a88700 1 --2- 172.21.15.42:0/1331507272 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc66c09f2e0 0x7fc66c09f710 secure :-1 s=THROTTLE_DONE pgs=1740 cs=0 l=1 rx=0x7fc66c08e770 tx=0x7fc664010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:11.950 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:11.929+0000 7fc663fff700 1 -- 172.21.15.42:0/1331507272 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc664010a00 con 0x7fc66c09f2e0 2019-12-05T05:57:11.950 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:11.929+0000 7fc663fff700 1 -- 172.21.15.42:0/1331507272 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc6640114a0 con 0x7fc66c09f2e0 2019-12-05T05:57:11.950 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:11.929+0000 7fc672cea700 1 -- 172.21.15.42:0/1331507272 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc66c09f2e0 msgr2=0x7fc66c09f710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:57:11.950 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:11.929+0000 7fc672cea700 1 --2- 172.21.15.42:0/1331507272 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc66c09f2e0 0x7fc66c09f710 secure :-1 s=READY pgs=1740 cs=0 l=1 rx=0x7fc66c08e770 tx=0x7fc664010130).stop 2019-12-05T05:57:11.950 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:11.933+0000 7fc672cea700 1 -- 172.21.15.42:0/1331507272 shutdown_connections 2019-12-05T05:57:11.951 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:11.933+0000 7fc672cea700 1 --2- 172.21.15.42:0/1331507272 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc66c09f2e0 0x7fc66c09f710 unknown :-1 s=CLOSED pgs=1740 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:57:11.951 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:11.933+0000 7fc672cea700 1 --2- 172.21.15.42:0/1331507272 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc66c0a7740 0x7fc66c08df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:57:11.951 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:11.933+0000 7fc672cea700 1 --2- 172.21.15.42:0/1331507272 >> v2:172.21.15.40:3300/0 conn(0x7fc66c07e720 0x7fc66c07eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:57:11.951 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:11.933+0000 7fc672cea700 1 -- 172.21.15.42:0/1331507272 shutdown_connections 2019-12-05T05:57:11.951 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:11.933+0000 7fc672cea700 1 -- 172.21.15.42:0/1331507272 wait complete. 2019-12-05T05:57:11.951 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:11.933+0000 7fc672cea700 1 -- 172.21.15.42:0/1331507272 >> 172.21.15.42:0/1331507272 conn(0x7fc66c0aabe0 msgr2=0x7fc66c0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:57:11.955 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:11.933+0000 7fc672cea700 1 Processor -- start 2019-12-05T05:57:11.955 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:11.933+0000 7fc672cea700 1 -- start start 2019-12-05T05:57:11.955 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:11.937+0000 7fc672cea700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc66c11e0c0 0x7fc66c11a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:57:11.955 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:11.937+0000 7fc672cea700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc66c1228a0 0x7fc66c0a5840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:57:11.955 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:11.937+0000 7fc672cea700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc66c122cd0 con 0x7fc66c0a5d90 2019-12-05T05:57:11.956 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:11.937+0000 7fc672cea700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc66c122fb0 con 0x7fc66c1228a0 2019-12-05T05:57:11.956 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:11.937+0000 7fc672cea700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc66c123250 con 0x7fc66c11e0c0 2019-12-05T05:57:11.956 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:11.937+0000 7fc670a86700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc66c1228a0 0x7fc66c0a5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:57:11.956 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:11.937+0000 7fc670a86700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc66c1228a0 0x7fc66c0a5840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:44672/0 (socket says 172.21.15.42:44672) 2019-12-05T05:57:11.956 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:11.937+0000 7fc670a86700 1 -- 172.21.15.42:0/1569719961 learned_addr learned my addr 172.21.15.42:0/1569719961 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:57:11.956 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:11.937+0000 7fc671287700 1 --2- 172.21.15.42:0/1569719961 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc66c11e0c0 0x7fc66c11a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:57:11.957 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:11.937+0000 7fc670a86700 1 -- 172.21.15.42:0/1569719961 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc66c11e0c0 msgr2=0x7fc66c11a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:57:11.957 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:11.937+0000 7fc670a86700 1 --2- 172.21.15.42:0/1569719961 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc66c11e0c0 0x7fc66c11a830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:57:11.957 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:11.937+0000 7fc670a86700 1 -- 172.21.15.42:0/1569719961 >> v1:172.21.15.40:6789/0 conn(0x7fc66c0a5d90 legacy=0x7fc66c133a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:57:11.957 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:11.937+0000 7fc670a86700 1 -- 172.21.15.42:0/1569719961 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc66c0a61c0 con 0x7fc66c1228a0 2019-12-05T05:57:11.957 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:11.937+0000 7fc670a86700 1 --2- 172.21.15.42:0/1569719961 >> v2:172.21.15.42:3300/0 conn(0x7fc66c1228a0 0x7fc66c0a5840 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fc66c123170 tx=0x7fc65c010a80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:11.957 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:11.937+0000 7fc670a86700 1 --2- 172.21.15.42:0/1569719961 >> v2:172.21.15.42:3300/0 conn(0x7fc66c1228a0 0x7fc66c0a5840 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fc66c123170 tx=0x7fc65c010a80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:11.958 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:11.937+0000 7fc670a86700 1 --2- 172.21.15.42:0/1569719961 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc66c1228a0 0x7fc66c0a5840 secure :-1 s=READY pgs=1741 cs=0 l=1 rx=0x7fc66c123170 tx=0x7fc65c010a80).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:57:11.958 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:11.937+0000 7fc670a86700 1 --2- 172.21.15.42:0/1569719961 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc66c1228a0 0x7fc66c0a5840 secure :-1 s=THROTTLE_DONE pgs=1741 cs=0 l=1 rx=0x7fc66c123170 tx=0x7fc65c010a80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:11.958 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:11.937+0000 7fc661ffb700 1 -- 172.21.15.42:0/1569719961 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc65c00a3c0 con 0x7fc66c1228a0 2019-12-05T05:57:11.958 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:11.937+0000 7fc670a86700 1 --2- 172.21.15.42:0/1569719961 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc66c1228a0 0x7fc66c0a5840 secure :-1 s=THROTTLE_DONE pgs=1741 cs=0 l=1 rx=0x7fc66c123170 tx=0x7fc65c010a80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:11.958 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:11.937+0000 7fc670a86700 1 --2- 172.21.15.42:0/1569719961 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc66c1228a0 0x7fc66c0a5840 secure :-1 s=THROTTLE_DONE pgs=1741 cs=0 l=1 rx=0x7fc66c123170 tx=0x7fc65c010a80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:11.958 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:11.937+0000 7fc661ffb700 1 -- 172.21.15.42:0/1569719961 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc65c007be0 con 0x7fc66c1228a0 2019-12-05T05:57:11.959 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:11.937+0000 7fc661ffb700 1 -- 172.21.15.42:0/1569719961 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc65c00a9e0 con 0x7fc66c1228a0 2019-12-05T05:57:11.959 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:11.937+0000 7fc672cea700 1 -- 172.21.15.42:0/1569719961 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc66c134230 con 0x7fc66c1228a0 2019-12-05T05:57:11.959 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:11.937+0000 7fc672cea700 1 -- 172.21.15.42:0/1569719961 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc66c077300 con 0x7fc66c1228a0 2019-12-05T05:57:11.959 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:11.937+0000 7fc670a86700 1 --2- 172.21.15.42:0/1569719961 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc66c1228a0 0x7fc66c0a5840 secure :-1 s=THROTTLE_DONE pgs=1741 cs=0 l=1 rx=0x7fc66c123170 tx=0x7fc65c010a80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:11.959 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:11.941+0000 7fc661ffb700 1 -- 172.21.15.42:0/1569719961 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fc65c007170 con 0x7fc66c1228a0 2019-12-05T05:57:11.960 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:11.941+0000 7fc670a86700 1 --2- 172.21.15.42:0/1569719961 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc66c1228a0 0x7fc66c0a5840 secure :-1 s=THROTTLE_DONE pgs=1741 cs=0 l=1 rx=0x7fc66c123170 tx=0x7fc65c010a80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:11.960 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:11.941+0000 7fc661ffb700 1 --2- 172.21.15.42:0/1569719961 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc64c03ade0 0x7fc64c03d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:57:11.960 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:11.941+0000 7fc661ffb700 1 -- 172.21.15.42:0/1569719961 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 9127+0+0 (secure 0 0 0) 0x7fc65c04afd0 con 0x7fc66c1228a0 2019-12-05T05:57:11.960 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:11.941+0000 7fc671287700 1 --2- 172.21.15.42:0/1569719961 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc64c03ade0 0x7fc64c03d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:57:11.961 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:11.941+0000 7fc671287700 1 --2- 172.21.15.42:0/1569719961 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc64c03ade0 0x7fc64c03d2f0 crc :-1 s=READY pgs=745 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:57:11.961 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:11.941+0000 7fc672cea700 1 -- 172.21.15.42:0/1569719961 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc66c05d110 con 0x7fc66c1228a0 2019-12-05T05:57:11.961 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:11.949+0000 7fc670a86700 1 --2- 172.21.15.42:0/1569719961 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc66c1228a0 0x7fc66c0a5840 secure :-1 s=THROTTLE_DONE pgs=1741 cs=0 l=1 rx=0x7fc66c123170 tx=0x7fc65c010a80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:11.961 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:11.949+0000 7fc661ffb700 1 -- 172.21.15.42:0/1569719961 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fc65c032050 con 0x7fc66c1228a0 2019-12-05T05:57:12.124 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:12.121+0000 7fc672cea700 1 -- 172.21.15.42:0/1569719961 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc66c05b8c0 con 0x7fc66c1228a0 2019-12-05T05:57:12.128 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:12.121+0000 7fc670a86700 1 --2- 172.21.15.42:0/1569719961 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc66c1228a0 0x7fc66c0a5840 secure :-1 s=THROTTLE_DONE pgs=1741 cs=0 l=1 rx=0x7fc66c123170 tx=0x7fc65c010a80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:12.128 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:12.121+0000 7fc661ffb700 1 -- 172.21.15.42:0/1569719961 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+2112 (secure 0 0 0) 0x7fc65c0322b0 con 0x7fc66c1228a0 2019-12-05T05:57:12.128 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 213 2019-12-05T05:57:12.128 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:57:12.129 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":213,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":8836,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6835/1368848584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1368848584},{"type":"v1","addr":"172.21.15.42:6835","nonce":1368848584}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":213}],"filesystems":[{"mdsmap":{"epoch":212,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:55:44.050934+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6461},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6461":{"gid":6461,"name":"b","rank":0,"incarnation":209,"state":"up:active","state_seq":12,"addr":"172.21.15.40:6833/3549993215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3549993215},{"type":"v1","addr":"172.21.15.40:6833","nonce":3549993215}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:57:12.132 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:12.129+0000 7fc672cea700 1 -- 172.21.15.42:0/1569719961 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc64c03ade0 msgr2=0x7fc64c03d2f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:57:12.132 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:12.129+0000 7fc672cea700 1 --2- 172.21.15.42:0/1569719961 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc64c03ade0 0x7fc64c03d2f0 crc :-1 s=READY pgs=745 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:57:12.132 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:12.129+0000 7fc672cea700 1 -- 172.21.15.42:0/1569719961 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc66c1228a0 msgr2=0x7fc66c0a5840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:57:12.132 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:12.129+0000 7fc672cea700 1 --2- 172.21.15.42:0/1569719961 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc66c1228a0 0x7fc66c0a5840 secure :-1 s=READY pgs=1741 cs=0 l=1 rx=0x7fc66c123170 tx=0x7fc65c010a80).stop 2019-12-05T05:57:12.133 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:12.129+0000 7fc672cea700 1 -- 172.21.15.42:0/1569719961 shutdown_connections 2019-12-05T05:57:12.133 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:12.129+0000 7fc672cea700 1 --2- 172.21.15.42:0/1569719961 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc64c03ade0 0x7fc64c03d2f0 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:57:12.133 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:12.129+0000 7fc672cea700 1 --2- 172.21.15.42:0/1569719961 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc66c11e0c0 0x7fc66c11a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:57:12.133 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:12.129+0000 7fc672cea700 1 --2- 172.21.15.42:0/1569719961 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc66c1228a0 0x7fc66c0a5840 unknown :-1 s=CLOSED pgs=1741 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:57:12.133 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:12.129+0000 7fc672cea700 1 -- 172.21.15.42:0/1569719961 shutdown_connections 2019-12-05T05:57:12.133 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:12.129+0000 7fc672cea700 1 -- 172.21.15.42:0/1569719961 wait complete. 2019-12-05T05:57:12.133 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:12.129+0000 7fc672cea700 1 -- 172.21.15.42:0/1569719961 >> 172.21.15.42:0/1569719961 conn(0x7fc66c07e5d0 msgr2=0x7fc66c0a9790 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:57:12.144 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T05:57:12.624 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:57:12.630 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:57:14.145 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:57:14.250 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:14.241+0000 7f5a11547700 1 Processor -- start 2019-12-05T05:57:14.250 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:14.241+0000 7f5a11547700 1 -- start start 2019-12-05T05:57:14.250 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:14.241+0000 7f5a11547700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5a0c09fae0 0x7f5a0c14fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:57:14.251 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:14.241+0000 7f5a11547700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a0c0a0710 con 0x7f5a0c0a7740 2019-12-05T05:57:14.251 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:14.241+0000 7f5a11547700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a0c07ef20 con 0x7f5a0c08df80 2019-12-05T05:57:14.251 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:14.241+0000 7f5a11547700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a0c0a0b00 con 0x7f5a0c09fae0 2019-12-05T05:57:14.251 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:14.241+0000 7f5a0affd700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f5a0c08df80 0x7f5a0c09f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:44720/0 (socket says 172.21.15.42:44720) 2019-12-05T05:57:14.252 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:14.241+0000 7f5a0affd700 1 -- 172.21.15.42:0/2346679780 learned_addr learned my addr 172.21.15.42:0/2346679780 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:57:14.264 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:14.241+0000 7f5a0bfff700 1 --2- 172.21.15.42:0/2346679780 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5a0c09fae0 0x7f5a0c14fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:57:14.264 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:14.241+0000 7f5a0a7fc700 1 -- 172.21.15.42:0/2346679780 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 205353446 0 0) 0x7f5a0c07ef20 con 0x7f5a0c08df80 2019-12-05T05:57:14.264 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:14.241+0000 7f5a0a7fc700 1 -- 172.21.15.42:0/2346679780 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59f80030b0 con 0x7f5a0c08df80 2019-12-05T05:57:14.264 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:14.241+0000 7f5a0a7fc700 1 -- 172.21.15.42:0/2346679780 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1080999682 0 0) 0x7f5a0c0a0710 con 0x7f5a0c0a7740 2019-12-05T05:57:14.264 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:14.241+0000 7f5a0a7fc700 1 -- 172.21.15.42:0/2346679780 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a0c07ef20 con 0x7f5a0c0a7740 2019-12-05T05:57:14.265 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:14.241+0000 7f5a0a7fc700 1 -- 172.21.15.42:0/2346679780 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 557502096 0 0) 0x7f59f80030b0 con 0x7f5a0c08df80 2019-12-05T05:57:14.265 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:14.241+0000 7f5a0bfff700 1 -- 172.21.15.42:0/2346679780 >> v1:172.21.15.42:6789/0 conn(0x7f5a0c08df80 legacy=0x7f5a0c09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:57:14.265 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:14.241+0000 7f5a0bfff700 1 -- 172.21.15.42:0/2346679780 >> v1:172.21.15.40:6789/0 conn(0x7f5a0c0a7740 legacy=0x7f5a0c07e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:57:14.265 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:14.241+0000 7f5a0bfff700 1 -- 172.21.15.42:0/2346679780 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a0c0a9ba0 con 0x7f5a0c09fae0 2019-12-05T05:57:14.265 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:14.241+0000 7f5a0bfff700 1 --2- 172.21.15.42:0/2346679780 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5a0c09fae0 0x7f5a0c14fff0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f5a04008b10 tx=0x7f59f4001950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:14.265 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:14.241+0000 7f5a0bfff700 1 --2- 172.21.15.42:0/2346679780 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5a0c09fae0 0x7f5a0c14fff0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f5a04008b10 tx=0x7f59f4001950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:14.265 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:14.241+0000 7f5a0bfff700 1 --2- 172.21.15.42:0/2346679780 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5a0c09fae0 0x7f5a0c14fff0 secure :-1 s=READY pgs=391 cs=0 l=1 rx=0x7f5a04008b10 tx=0x7f59f4001950).ready entity=mon.2 client_cookie=1bfebbc0f1b66728 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:57:14.266 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:14.245+0000 7f5a0bfff700 1 --2- 172.21.15.42:0/2346679780 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5a0c09fae0 0x7f5a0c14fff0 secure :-1 s=THROTTLE_DONE pgs=391 cs=0 l=1 rx=0x7f5a04008b10 tx=0x7f59f4001950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:14.266 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:14.245+0000 7f5a0a7fc700 1 -- 172.21.15.42:0/2346679780 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f59f4001750 con 0x7f5a0c09fae0 2019-12-05T05:57:14.266 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:14.245+0000 7f5a0bfff700 1 --2- 172.21.15.42:0/2346679780 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5a0c09fae0 0x7f5a0c14fff0 secure :-1 s=THROTTLE_DONE pgs=391 cs=0 l=1 rx=0x7f5a04008b10 tx=0x7f59f4001950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:14.266 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:14.245+0000 7f5a0a7fc700 1 -- 172.21.15.42:0/2346679780 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59f4001560 con 0x7f5a0c09fae0 2019-12-05T05:57:14.266 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:14.245+0000 7f5a0bfff700 1 --2- 172.21.15.42:0/2346679780 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5a0c09fae0 0x7f5a0c14fff0 secure :-1 s=THROTTLE_DONE pgs=391 cs=0 l=1 rx=0x7f5a04008b10 tx=0x7f59f4001950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:14.266 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:14.245+0000 7f5a11547700 1 -- 172.21.15.42:0/2346679780 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5a0c09fae0 msgr2=0x7f5a0c14fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:57:14.266 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:14.245+0000 7f5a0a7fc700 1 -- 172.21.15.42:0/2346679780 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5a04010a60 con 0x7f5a0c09fae0 2019-12-05T05:57:14.267 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:14.245+0000 7f5a11547700 1 --2- 172.21.15.42:0/2346679780 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5a0c09fae0 0x7f5a0c14fff0 secure :-1 s=READY pgs=391 cs=0 l=1 rx=0x7f5a04008b10 tx=0x7f59f4001950).stop 2019-12-05T05:57:14.267 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:14.245+0000 7f5a11547700 1 -- 172.21.15.42:0/2346679780 shutdown_connections 2019-12-05T05:57:14.267 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:14.245+0000 7f5a11547700 1 --2- 172.21.15.42:0/2346679780 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5a0c09fae0 0x7f5a0c14fff0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:57:14.267 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:14.245+0000 7f5a11547700 1 -- 172.21.15.42:0/2346679780 shutdown_connections 2019-12-05T05:57:14.274 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:14.245+0000 7f5a11547700 1 -- 172.21.15.42:0/2346679780 wait complete. 2019-12-05T05:57:14.274 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:14.245+0000 7f5a11547700 1 -- 172.21.15.42:0/2346679780 >> 172.21.15.42:0/2346679780 conn(0x7f5a0c0aabe0 msgr2=0x7f5a0c0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:57:14.277 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:14.249+0000 7f5a11547700 1 Processor -- start 2019-12-05T05:57:14.278 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:14.249+0000 7f5a11547700 1 -- start start 2019-12-05T05:57:14.278 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:14.249+0000 7f5a11547700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5a0c07e830 0x7f5a0c12ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:57:14.278 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:14.249+0000 7f5a11547700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a0c07cea0 con 0x7f5a0c0a39d0 2019-12-05T05:57:14.278 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:14.249+0000 7f5a11547700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a0c07d0d0 con 0x7f5a0c0a6c30 2019-12-05T05:57:14.278 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:14.249+0000 7f5a11547700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a0c0a3ee0 con 0x7f5a0c07e830 2019-12-05T05:57:14.279 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:14.249+0000 7f5a0bfff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f5a0c0a6c30 0x7f5a0c138c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:44730/0 (socket says 172.21.15.42:44730) 2019-12-05T05:57:14.280 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:14.249+0000 7f5a0bfff700 1 -- 172.21.15.42:0/2123117720 learned_addr learned my addr 172.21.15.42:0/2123117720 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:57:14.280 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:14.249+0000 7f5a0affd700 1 --2- 172.21.15.42:0/2123117720 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5a0c07e830 0x7f5a0c12ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:57:14.280 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:14.249+0000 7f5a0affd700 1 -- 172.21.15.42:0/2123117720 >> v1:172.21.15.42:6789/0 conn(0x7f5a0c0a6c30 legacy=0x7f5a0c138c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:57:14.280 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:14.249+0000 7f5a0affd700 1 -- 172.21.15.42:0/2123117720 >> v1:172.21.15.40:6789/0 conn(0x7f5a0c0a39d0 legacy=0x7f5a0c12e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:57:14.280 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:14.249+0000 7f5a0affd700 1 -- 172.21.15.42:0/2123117720 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a0c08fb00 con 0x7f5a0c07e830 2019-12-05T05:57:14.280 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:14.249+0000 7f5a0affd700 1 --2- 172.21.15.42:0/2123117720 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5a0c07e830 0x7f5a0c12ebe0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f59f4000fa0 tx=0x7f59f400c410).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:14.281 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:14.249+0000 7f5a0affd700 1 --2- 172.21.15.42:0/2123117720 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5a0c07e830 0x7f5a0c12ebe0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f59f4000fa0 tx=0x7f59f400c410).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:14.281 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:14.249+0000 7f5a0affd700 1 --2- 172.21.15.42:0/2123117720 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5a0c07e830 0x7f5a0c12ebe0 secure :-1 s=READY pgs=392 cs=0 l=1 rx=0x7f59f4000fa0 tx=0x7f59f400c410).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:57:14.281 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:14.257+0000 7f5a0affd700 1 --2- 172.21.15.42:0/2123117720 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5a0c07e830 0x7f5a0c12ebe0 secure :-1 s=THROTTLE_DONE pgs=392 cs=0 l=1 rx=0x7f59f4000fa0 tx=0x7f59f400c410).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:14.281 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:14.257+0000 7f5a0affd700 1 --2- 172.21.15.42:0/2123117720 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5a0c07e830 0x7f5a0c12ebe0 secure :-1 s=THROTTLE_DONE pgs=392 cs=0 l=1 rx=0x7f59f4000fa0 tx=0x7f59f400c410).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:14.281 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:14.257+0000 7f5a0affd700 1 --2- 172.21.15.42:0/2123117720 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5a0c07e830 0x7f5a0c12ebe0 secure :-1 s=THROTTLE_DONE pgs=392 cs=0 l=1 rx=0x7f59f4000fa0 tx=0x7f59f400c410).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:14.281 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:14.257+0000 7f5a03fff700 1 -- 172.21.15.42:0/2123117720 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f59f4011740 con 0x7f5a0c07e830 2019-12-05T05:57:14.282 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:14.257+0000 7f5a03fff700 1 -- 172.21.15.42:0/2123117720 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59f4011360 con 0x7f5a0c07e830 2019-12-05T05:57:14.282 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:14.257+0000 7f5a11547700 1 -- 172.21.15.42:0/2123117720 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a0c07d300 con 0x7f5a0c07e830 2019-12-05T05:57:14.282 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:14.257+0000 7f5a03fff700 1 -- 172.21.15.42:0/2123117720 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f59f4011d20 con 0x7f5a0c07e830 2019-12-05T05:57:14.282 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:14.257+0000 7f5a11547700 1 -- 172.21.15.42:0/2123117720 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a0c13a480 con 0x7f5a0c07e830 2019-12-05T05:57:14.282 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:14.257+0000 7f5a0affd700 1 --2- 172.21.15.42:0/2123117720 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5a0c07e830 0x7f5a0c12ebe0 secure :-1 s=THROTTLE_DONE pgs=392 cs=0 l=1 rx=0x7f59f4000fa0 tx=0x7f59f400c410).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:14.282 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:14.257+0000 7f5a03fff700 1 -- 172.21.15.42:0/2123117720 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f59f4007180 con 0x7f5a0c07e830 2019-12-05T05:57:14.283 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:14.257+0000 7f5a0affd700 1 --2- 172.21.15.42:0/2123117720 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5a0c07e830 0x7f5a0c12ebe0 secure :-1 s=THROTTLE_DONE pgs=392 cs=0 l=1 rx=0x7f59f4000fa0 tx=0x7f59f400c410).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:14.283 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:14.257+0000 7f5a03fff700 1 --2- 172.21.15.42:0/2123117720 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f59f003ad00 0x7f59f003d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:57:14.283 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:14.257+0000 7f5a03fff700 1 -- 172.21.15.42:0/2123117720 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 9127+0+0 (secure 0 0 0) 0x7f59f404ada0 con 0x7f5a0c07e830 2019-12-05T05:57:14.283 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:14.257+0000 7f5a0b7fe700 1 --2- 172.21.15.42:0/2123117720 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f59f003ad00 0x7f59f003d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:57:14.284 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:14.257+0000 7f5a0b7fe700 1 --2- 172.21.15.42:0/2123117720 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f59f003ad00 0x7f59f003d210 crc :-1 s=READY pgs=746 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:57:14.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:14.273+0000 7f5a11547700 1 -- 172.21.15.42:0/2123117720 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a0c09f710 con 0x7f5a0c07e830 2019-12-05T05:57:14.296 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:14.281+0000 7f5a0affd700 1 --2- 172.21.15.42:0/2123117720 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5a0c07e830 0x7f5a0c12ebe0 secure :-1 s=THROTTLE_DONE pgs=392 cs=0 l=1 rx=0x7f59f4000fa0 tx=0x7f59f400c410).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:14.296 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:14.281+0000 7f5a03fff700 1 -- 172.21.15.42:0/2123117720 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f59f4031d60 con 0x7f5a0c07e830 2019-12-05T05:57:14.463 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:14.449+0000 7f5a11547700 1 -- 172.21.15.42:0/2123117720 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5a0c05d110 con 0x7f5a0c07e830 2019-12-05T05:57:14.463 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:14.449+0000 7f5a0affd700 1 --2- 172.21.15.42:0/2123117720 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5a0c07e830 0x7f5a0c12ebe0 secure :-1 s=THROTTLE_DONE pgs=392 cs=0 l=1 rx=0x7f59f4000fa0 tx=0x7f59f400c410).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:14.464 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:14.449+0000 7f5a03fff700 1 -- 172.21.15.42:0/2123117720 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f59f4031fc0 con 0x7f5a0c07e830 2019-12-05T05:57:14.464 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 213 2019-12-05T05:57:14.465 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:14.453+0000 7f5a11547700 1 -- 172.21.15.42:0/2123117720 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f59f003ad00 msgr2=0x7f59f003d210 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:57:14.465 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:14.453+0000 7f5a11547700 1 --2- 172.21.15.42:0/2123117720 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f59f003ad00 0x7f59f003d210 crc :-1 s=READY pgs=746 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:57:14.465 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:14.453+0000 7f5a11547700 1 -- 172.21.15.42:0/2123117720 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5a0c07e830 msgr2=0x7f5a0c12ebe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:57:14.465 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:14.453+0000 7f5a11547700 1 --2- 172.21.15.42:0/2123117720 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5a0c07e830 0x7f5a0c12ebe0 secure :-1 s=READY pgs=392 cs=0 l=1 rx=0x7f59f4000fa0 tx=0x7f59f400c410).stop 2019-12-05T05:57:14.465 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:14.453+0000 7f5a11547700 1 -- 172.21.15.42:0/2123117720 shutdown_connections 2019-12-05T05:57:14.465 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:14.453+0000 7f5a11547700 1 --2- 172.21.15.42:0/2123117720 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f59f003ad00 0x7f59f003d210 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:57:14.466 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:14.453+0000 7f5a11547700 1 --2- 172.21.15.42:0/2123117720 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5a0c07e830 0x7f5a0c12ebe0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:57:14.466 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:14.453+0000 7f5a11547700 1 -- 172.21.15.42:0/2123117720 shutdown_connections 2019-12-05T05:57:14.466 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:14.453+0000 7f5a11547700 1 -- 172.21.15.42:0/2123117720 wait complete. 2019-12-05T05:57:14.466 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:14.453+0000 7f5a11547700 1 -- 172.21.15.42:0/2123117720 >> 172.21.15.42:0/2123117720 conn(0x7f5a0c059ef0 msgr2=0x7f5a0c1189c0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:57:14.467 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:57:14.467 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":213,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":8836,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6835/1368848584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1368848584},{"type":"v1","addr":"172.21.15.42:6835","nonce":1368848584}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":213}],"filesystems":[{"mdsmap":{"epoch":212,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:55:44.050934+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6461},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6461":{"gid":6461,"name":"b","rank":0,"incarnation":209,"state":"up:active","state_seq":12,"addr":"172.21.15.40:6833/3549993215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3549993215},{"type":"v1","addr":"172.21.15.40:6833","nonce":3549993215}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:57:14.468 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T05:57:16.469 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:57:16.579 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:16.569+0000 7f81346ef700 1 Processor -- start 2019-12-05T05:57:16.579 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:16.569+0000 7f81346ef700 1 -- start start 2019-12-05T05:57:16.580 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:16.569+0000 7f81346ef700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f812c08df80 0x7f812c09f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:57:16.580 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:16.569+0000 7f81346ef700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f812c09f830 0x7f812c14efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:57:16.580 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:16.569+0000 7f81346ef700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f812c09fc60 con 0x7f812c0a7740 2019-12-05T05:57:16.580 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:16.569+0000 7f81346ef700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f812c0a0b00 con 0x7f812c08df80 2019-12-05T05:57:16.580 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:16.569+0000 7f81346ef700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f812c08e5b0 con 0x7f812c09f830 2019-12-05T05:57:16.581 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:16.569+0000 7f813248b700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f812c08df80 0x7f812c09f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:57:16.581 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:16.569+0000 7f813348d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f812c09f830 0x7f812c14efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:57:16.581 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:16.569+0000 7f813248b700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f812c08df80 0x7f812c09f2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.42:60748/0 (socket says 172.21.15.42:60748) 2019-12-05T05:57:16.581 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:16.569+0000 7f813248b700 1 -- 172.21.15.42:0/3489287304 learned_addr learned my addr 172.21.15.42:0/3489287304 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:57:16.581 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:16.569+0000 7f813348d700 1 -- 172.21.15.42:0/3489287304 >> v2:172.21.15.40:3300/0 conn(0x7f812c08df80 msgr2=0x7f812c09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:57:16.581 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:16.569+0000 7f813348d700 1 --2- 172.21.15.42:0/3489287304 >> v2:172.21.15.40:3300/0 conn(0x7f812c08df80 0x7f812c09f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:57:16.582 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:16.569+0000 7f813348d700 1 -- 172.21.15.42:0/3489287304 >> v1:172.21.15.40:6789/0 conn(0x7f812c0a7740 legacy=0x7f812c07e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:57:16.583 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:16.569+0000 7f813348d700 1 -- 172.21.15.42:0/3489287304 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f812c0a0710 con 0x7f812c09f830 2019-12-05T05:57:16.583 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:16.569+0000 7f813248b700 1 --2- 172.21.15.42:0/3489287304 >> v2:172.21.15.40:3300/0 conn(0x7f812c08df80 0x7f812c09f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:57:16.583 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:16.569+0000 7f813348d700 1 --2- 172.21.15.42:0/3489287304 >> v2:172.21.15.42:3300/0 conn(0x7f812c09f830 0x7f812c14efe0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f8124002f30 tx=0x7f812400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:16.583 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:16.569+0000 7f813348d700 1 --2- 172.21.15.42:0/3489287304 >> v2:172.21.15.42:3300/0 conn(0x7f812c09f830 0x7f812c14efe0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f8124002f30 tx=0x7f812400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:16.583 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:16.573+0000 7f813348d700 1 --2- 172.21.15.42:0/3489287304 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f812c09f830 0x7f812c14efe0 secure :-1 s=READY pgs=1744 cs=0 l=1 rx=0x7f8124002f30 tx=0x7f812400fc70).ready entity=mon.1 client_cookie=a232181e04d5063c server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:57:16.584 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:16.573+0000 7f813348d700 1 --2- 172.21.15.42:0/3489287304 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f812c09f830 0x7f812c14efe0 secure :-1 s=THROTTLE_DONE pgs=1744 cs=0 l=1 rx=0x7f8124002f30 tx=0x7f812400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:16.584 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:16.573+0000 7f8131c8a700 1 -- 172.21.15.42:0/3489287304 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8124008bb0 con 0x7f812c09f830 2019-12-05T05:57:16.584 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:16.573+0000 7f813348d700 1 --2- 172.21.15.42:0/3489287304 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f812c09f830 0x7f812c14efe0 secure :-1 s=THROTTLE_DONE pgs=1744 cs=0 l=1 rx=0x7f8124002f30 tx=0x7f812400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:16.585 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:16.573+0000 7f8131c8a700 1 -- 172.21.15.42:0/3489287304 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81240087d0 con 0x7f812c09f830 2019-12-05T05:57:16.585 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:16.573+0000 7f813348d700 1 --2- 172.21.15.42:0/3489287304 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f812c09f830 0x7f812c14efe0 secure :-1 s=THROTTLE_DONE pgs=1744 cs=0 l=1 rx=0x7f8124002f30 tx=0x7f812400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:16.585 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:16.573+0000 7f8131c8a700 1 -- 172.21.15.42:0/3489287304 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8124011820 con 0x7f812c09f830 2019-12-05T05:57:16.585 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:16.573+0000 7f81346ef700 1 -- 172.21.15.42:0/3489287304 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f812c09f830 msgr2=0x7f812c14efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:57:16.585 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:16.573+0000 7f81346ef700 1 --2- 172.21.15.42:0/3489287304 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f812c09f830 0x7f812c14efe0 secure :-1 s=READY pgs=1744 cs=0 l=1 rx=0x7f8124002f30 tx=0x7f812400fc70).stop 2019-12-05T05:57:16.586 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:16.573+0000 7f81346ef700 1 -- 172.21.15.42:0/3489287304 shutdown_connections 2019-12-05T05:57:16.595 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:16.573+0000 7f81346ef700 1 --2- 172.21.15.42:0/3489287304 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f812c09f830 0x7f812c14efe0 unknown :-1 s=CLOSED pgs=1744 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:57:16.595 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:16.573+0000 7f81346ef700 1 --2- 172.21.15.42:0/3489287304 >> v2:172.21.15.40:3300/0 conn(0x7f812c08df80 0x7f812c09f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:57:16.595 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:16.573+0000 7f81346ef700 1 -- 172.21.15.42:0/3489287304 shutdown_connections 2019-12-05T05:57:16.595 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:16.573+0000 7f81346ef700 1 -- 172.21.15.42:0/3489287304 wait complete. 2019-12-05T05:57:16.596 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:16.573+0000 7f81346ef700 1 -- 172.21.15.42:0/3489287304 >> 172.21.15.42:0/3489287304 conn(0x7f812c0aabe0 msgr2=0x7f812c0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:57:16.596 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:16.573+0000 7f81346ef700 1 Processor -- start 2019-12-05T05:57:16.596 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:16.577+0000 7f81346ef700 1 -- start start 2019-12-05T05:57:16.600 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:16.577+0000 7f81346ef700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f812c124fd0 0x7f812c125400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:57:16.600 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:16.577+0000 7f81346ef700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f812c129d50 0x7f812c12c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:57:16.600 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:16.577+0000 7f81346ef700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f812c08f040 con 0x7f812c122070 2019-12-05T05:57:16.600 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:16.577+0000 7f81346ef700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f812c124a90 con 0x7f812c124fd0 2019-12-05T05:57:16.601 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:16.577+0000 7f81346ef700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f812c124d30 con 0x7f812c129d50 2019-12-05T05:57:16.601 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:16.577+0000 7f813248b700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f812c124fd0 0x7f812c125400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:57:16.601 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:16.577+0000 7f813348d700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f812c129d50 0x7f812c12c190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:57:16.601 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:16.577+0000 7f813248b700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f812c124fd0 0x7f812c125400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:44708/0 (socket says 172.21.15.42:44708) 2019-12-05T05:57:16.601 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:16.577+0000 7f813248b700 1 -- 172.21.15.42:0/2917070773 learned_addr learned my addr 172.21.15.42:0/2917070773 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:57:16.601 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:16.577+0000 7f813348d700 1 --2- 172.21.15.42:0/2917070773 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f812c129d50 0x7f812c12c190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:45172/0 (socket says 172.21.15.42:45172) 2019-12-05T05:57:16.602 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:16.577+0000 7f8132c8c700 1 --1- 172.21.15.42:0/2917070773 >> v1:172.21.15.40:6789/0 conn(0x7f812c122070 0x7f812c1143c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:54182/0 (socket says 172.21.15.42:54182) 2019-12-05T05:57:16.602 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:16.577+0000 7f811f7fe700 1 -- 172.21.15.42:0/2917070773 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3961705219 0 0) 0x7f812c08f040 con 0x7f812c122070 2019-12-05T05:57:16.602 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:16.577+0000 7f813348d700 1 -- 172.21.15.42:0/2917070773 >> v2:172.21.15.42:3300/0 conn(0x7f812c124fd0 msgr2=0x7f812c125400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:57:16.602 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:16.577+0000 7f813348d700 1 --2- 172.21.15.42:0/2917070773 >> v2:172.21.15.42:3300/0 conn(0x7f812c124fd0 0x7f812c125400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:57:16.602 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:16.577+0000 7f813348d700 1 -- 172.21.15.42:0/2917070773 >> v1:172.21.15.40:6789/0 conn(0x7f812c122070 legacy=0x7f812c1143c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:57:16.604 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:16.577+0000 7f813348d700 1 -- 172.21.15.42:0/2917070773 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f812c111fa0 con 0x7f812c129d50 2019-12-05T05:57:16.604 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:16.577+0000 7f813348d700 1 --2- 172.21.15.42:0/2917070773 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f812c129d50 0x7f812c12c190 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f8124002f30 tx=0x7f8124008a40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:16.604 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:16.577+0000 7f813348d700 1 --2- 172.21.15.42:0/2917070773 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f812c129d50 0x7f812c12c190 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f8124002f30 tx=0x7f8124008a40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:16.604 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:16.577+0000 7f813348d700 1 --2- 172.21.15.42:0/2917070773 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f812c129d50 0x7f812c12c190 secure :-1 s=READY pgs=393 cs=0 l=1 rx=0x7f8124002f30 tx=0x7f8124008a40).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:57:16.604 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:16.577+0000 7f813348d700 1 --2- 172.21.15.42:0/2917070773 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f812c129d50 0x7f812c12c190 secure :-1 s=THROTTLE_DONE pgs=393 cs=0 l=1 rx=0x7f8124002f30 tx=0x7f8124008a40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:16.605 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:16.577+0000 7f811f7fe700 1 -- 172.21.15.42:0/2917070773 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f812400f490 con 0x7f812c129d50 2019-12-05T05:57:16.605 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:16.577+0000 7f813348d700 1 --2- 172.21.15.42:0/2917070773 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f812c129d50 0x7f812c12c190 secure :-1 s=THROTTLE_DONE pgs=393 cs=0 l=1 rx=0x7f8124002f30 tx=0x7f8124008a40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:16.605 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:16.577+0000 7f81346ef700 1 -- 172.21.15.42:0/2917070773 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f812c121350 con 0x7f812c129d50 2019-12-05T05:57:16.606 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:16.577+0000 7f811f7fe700 1 -- 172.21.15.42:0/2917070773 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8124008db0 con 0x7f812c129d50 2019-12-05T05:57:16.606 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:16.577+0000 7f813348d700 1 --2- 172.21.15.42:0/2917070773 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f812c129d50 0x7f812c12c190 secure :-1 s=THROTTLE_DONE pgs=393 cs=0 l=1 rx=0x7f8124002f30 tx=0x7f8124008a40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:16.606 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:16.577+0000 7f811f7fe700 1 -- 172.21.15.42:0/2917070773 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8124005920 con 0x7f812c129d50 2019-12-05T05:57:16.606 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:16.577+0000 7f81346ef700 1 -- 172.21.15.42:0/2917070773 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f812c08fa60 con 0x7f812c129d50 2019-12-05T05:57:16.606 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:16.581+0000 7f813348d700 1 --2- 172.21.15.42:0/2917070773 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f812c129d50 0x7f812c12c190 secure :-1 s=THROTTLE_DONE pgs=393 cs=0 l=1 rx=0x7f8124002f30 tx=0x7f8124008a40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:16.606 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:16.581+0000 7f811f7fe700 1 -- 172.21.15.42:0/2917070773 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f81240130b0 con 0x7f812c129d50 2019-12-05T05:57:16.607 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:16.581+0000 7f813348d700 1 --2- 172.21.15.42:0/2917070773 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f812c129d50 0x7f812c12c190 secure :-1 s=THROTTLE_DONE pgs=393 cs=0 l=1 rx=0x7f8124002f30 tx=0x7f8124008a40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:16.607 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:16.581+0000 7f811f7fe700 1 --2- 172.21.15.42:0/2917070773 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f811403ac00 0x7f811403d110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:57:16.607 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:16.581+0000 7f811f7fe700 1 -- 172.21.15.42:0/2917070773 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 9127+0+0 (secure 0 0 0) 0x7f812404b420 con 0x7f812c129d50 2019-12-05T05:57:16.607 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:16.581+0000 7f8132c8c700 1 --2- 172.21.15.42:0/2917070773 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f811403ac00 0x7f811403d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:57:16.607 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:16.581+0000 7f8132c8c700 1 --2- 172.21.15.42:0/2917070773 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f811403ac00 0x7f811403d110 crc :-1 s=READY pgs=747 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:57:16.607 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:16.585+0000 7f81346ef700 1 -- 172.21.15.42:0/2917070773 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f812c05d110 con 0x7f812c129d50 2019-12-05T05:57:16.608 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:16.597+0000 7f813348d700 1 --2- 172.21.15.42:0/2917070773 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f812c129d50 0x7f812c12c190 secure :-1 s=THROTTLE_DONE pgs=393 cs=0 l=1 rx=0x7f8124002f30 tx=0x7f8124008a40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:16.608 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:16.597+0000 7f811f7fe700 1 -- 172.21.15.42:0/2917070773 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f8124005bc0 con 0x7f812c129d50 2019-12-05T05:57:16.774 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:16.761+0000 7f81346ef700 1 -- 172.21.15.42:0/2917070773 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f812c1454c0 con 0x7f812c129d50 2019-12-05T05:57:16.775 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:16.765+0000 7f813348d700 1 --2- 172.21.15.42:0/2917070773 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f812c129d50 0x7f812c12c190 secure :-1 s=THROTTLE_DONE pgs=393 cs=0 l=1 rx=0x7f8124002f30 tx=0x7f8124008a40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:16.775 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:16.765+0000 7f811f7fe700 1 -- 172.21.15.42:0/2917070773 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f81240331f0 con 0x7f812c129d50 2019-12-05T05:57:16.775 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 213 2019-12-05T05:57:16.775 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:16.769+0000 7f81346ef700 1 -- 172.21.15.42:0/2917070773 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f811403ac00 msgr2=0x7f811403d110 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:57:16.775 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:16.769+0000 7f81346ef700 1 --2- 172.21.15.42:0/2917070773 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f811403ac00 0x7f811403d110 crc :-1 s=READY pgs=747 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:57:16.775 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:16.769+0000 7f81346ef700 1 -- 172.21.15.42:0/2917070773 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f812c129d50 msgr2=0x7f812c12c190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:57:16.776 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:16.769+0000 7f81346ef700 1 --2- 172.21.15.42:0/2917070773 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f812c129d50 0x7f812c12c190 secure :-1 s=READY pgs=393 cs=0 l=1 rx=0x7f8124002f30 tx=0x7f8124008a40).stop 2019-12-05T05:57:16.776 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:16.769+0000 7f81346ef700 1 -- 172.21.15.42:0/2917070773 shutdown_connections 2019-12-05T05:57:16.776 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:16.769+0000 7f81346ef700 1 --2- 172.21.15.42:0/2917070773 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f811403ac00 0x7f811403d110 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:57:16.776 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:16.769+0000 7f81346ef700 1 --2- 172.21.15.42:0/2917070773 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f812c129d50 0x7f812c12c190 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:57:16.776 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:16.769+0000 7f81346ef700 1 --2- 172.21.15.42:0/2917070773 >> v2:172.21.15.42:3300/0 conn(0x7f812c124fd0 0x7f812c125400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:57:16.777 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:16.769+0000 7f81346ef700 1 -- 172.21.15.42:0/2917070773 shutdown_connections 2019-12-05T05:57:16.777 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:16.769+0000 7f81346ef700 1 -- 172.21.15.42:0/2917070773 wait complete. 2019-12-05T05:57:16.777 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:16.769+0000 7f81346ef700 1 -- 172.21.15.42:0/2917070773 >> 172.21.15.42:0/2917070773 conn(0x7f812c141400 msgr2=0x7f812c07da60 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:57:16.777 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:57:16.777 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":213,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":8836,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6835/1368848584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1368848584},{"type":"v1","addr":"172.21.15.42:6835","nonce":1368848584}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":213}],"filesystems":[{"mdsmap":{"epoch":212,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:55:44.050934+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6461},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6461":{"gid":6461,"name":"b","rank":0,"incarnation":209,"state":"up:active","state_seq":12,"addr":"172.21.15.40:6833/3549993215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3549993215},{"type":"v1","addr":"172.21.15.40:6833","nonce":3549993215}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:57:16.783 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T05:57:18.784 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:57:18.893 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:18.881+0000 7f8068e63700 1 Processor -- start 2019-12-05T05:57:18.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:18.885+0000 7f8068e63700 1 -- start start 2019-12-05T05:57:18.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:18.885+0000 7f8068e63700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f806408df80 0x7f806409f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:57:18.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:18.885+0000 7f8068e63700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f806409f830 0x7f806414efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:57:18.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:18.885+0000 7f8068e63700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f806409fc60 con 0x7f80640a7740 2019-12-05T05:57:18.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:18.885+0000 7f8068e63700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80640a0b00 con 0x7f806408df80 2019-12-05T05:57:18.895 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:18.885+0000 7f8068e63700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f806408e5b0 con 0x7f806409f830 2019-12-05T05:57:18.895 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:18.885+0000 7f806259c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f806408df80 0x7f806409f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:57:18.895 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:18.885+0000 7f8062d9d700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f80640a7740 0x7f806407e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:44750/0 (socket says 172.21.15.42:44750) 2019-12-05T05:57:18.895 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:18.885+0000 7f8062d9d700 1 -- 172.21.15.42:0/3666840407 learned_addr learned my addr 172.21.15.42:0/3666840407 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:57:18.895 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:18.885+0000 7f806359e700 1 --2- 172.21.15.42:0/3666840407 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f806409f830 0x7f806414efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:57:18.895 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:18.885+0000 7f806259c700 1 -- 172.21.15.42:0/3666840407 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f806409f830 msgr2=0x7f806414efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:57:18.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:18.885+0000 7f806259c700 1 --2- 172.21.15.42:0/3666840407 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f806409f830 0x7f806414efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:57:18.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:18.885+0000 7f806259c700 1 -- 172.21.15.42:0/3666840407 >> v1:172.21.15.42:6789/0 conn(0x7f80640a7740 legacy=0x7f806407e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:57:18.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:18.885+0000 7f806259c700 1 -- 172.21.15.42:0/3666840407 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80640a0710 con 0x7f806408df80 2019-12-05T05:57:18.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:18.885+0000 7f8061d9b700 1 -- 172.21.15.42:0/3666840407 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 781859169 0 0) 0x7f806409fc60 con 0x7f80640a7740 2019-12-05T05:57:18.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:18.885+0000 7f806259c700 1 --2- 172.21.15.42:0/3666840407 >> v2:172.21.15.42:3300/0 conn(0x7f806408df80 0x7f806409f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f804c008790 tx=0x7f804c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:18.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:18.885+0000 7f806259c700 1 --2- 172.21.15.42:0/3666840407 >> v2:172.21.15.42:3300/0 conn(0x7f806408df80 0x7f806409f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f804c008790 tx=0x7f804c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:18.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:18.885+0000 7f806259c700 1 --2- 172.21.15.42:0/3666840407 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f806408df80 0x7f806409f2e0 secure :-1 s=READY pgs=1746 cs=0 l=1 rx=0x7f804c008790 tx=0x7f804c00fc70).ready entity=mon.1 client_cookie=279a3ea860a6c07b server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:57:18.897 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:18.885+0000 7f806259c700 1 --2- 172.21.15.42:0/3666840407 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f806408df80 0x7f806409f2e0 secure :-1 s=THROTTLE_DONE pgs=1746 cs=0 l=1 rx=0x7f804c008790 tx=0x7f804c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:18.897 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:18.885+0000 7f8061d9b700 1 -- 172.21.15.42:0/3666840407 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f804c008bb0 con 0x7f806408df80 2019-12-05T05:57:18.898 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:18.885+0000 7f806259c700 1 --2- 172.21.15.42:0/3666840407 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f806408df80 0x7f806409f2e0 secure :-1 s=THROTTLE_DONE pgs=1746 cs=0 l=1 rx=0x7f804c008790 tx=0x7f804c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:18.898 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:18.885+0000 7f8061d9b700 1 -- 172.21.15.42:0/3666840407 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f804c0087d0 con 0x7f806408df80 2019-12-05T05:57:18.898 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:18.885+0000 7f806259c700 1 --2- 172.21.15.42:0/3666840407 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f806408df80 0x7f806409f2e0 secure :-1 s=THROTTLE_DONE pgs=1746 cs=0 l=1 rx=0x7f804c008790 tx=0x7f804c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:18.898 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:18.885+0000 7f8061d9b700 1 -- 172.21.15.42:0/3666840407 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f804c011860 con 0x7f806408df80 2019-12-05T05:57:18.898 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:18.885+0000 7f8068e63700 1 -- 172.21.15.42:0/3666840407 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f806408df80 msgr2=0x7f806409f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:57:18.898 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:18.885+0000 7f8068e63700 1 --2- 172.21.15.42:0/3666840407 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f806408df80 0x7f806409f2e0 secure :-1 s=READY pgs=1746 cs=0 l=1 rx=0x7f804c008790 tx=0x7f804c00fc70).stop 2019-12-05T05:57:18.898 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:18.889+0000 7f8068e63700 1 -- 172.21.15.42:0/3666840407 shutdown_connections 2019-12-05T05:57:18.899 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:18.889+0000 7f8068e63700 1 --2- 172.21.15.42:0/3666840407 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f806409f830 0x7f806414efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:57:18.899 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:18.889+0000 7f8068e63700 1 --2- 172.21.15.42:0/3666840407 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f806408df80 0x7f806409f2e0 unknown :-1 s=CLOSED pgs=1746 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:57:18.899 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:18.889+0000 7f8068e63700 1 -- 172.21.15.42:0/3666840407 shutdown_connections 2019-12-05T05:57:18.899 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:18.889+0000 7f8068e63700 1 -- 172.21.15.42:0/3666840407 wait complete. 2019-12-05T05:57:18.900 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:18.889+0000 7f8068e63700 1 -- 172.21.15.42:0/3666840407 >> 172.21.15.42:0/3666840407 conn(0x7f80640aabe0 msgr2=0x7f80640a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:57:18.904 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:18.889+0000 7f8068e63700 1 Processor -- start 2019-12-05T05:57:18.905 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:18.889+0000 7f8068e63700 1 -- start start 2019-12-05T05:57:18.905 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:18.893+0000 7f8068e63700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f80640a3960 0x7f806411a930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:57:18.905 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:18.893+0000 7f8068e63700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8064123c90 0x7f8064118430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:57:18.905 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:18.893+0000 7f8068e63700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8064142530 con 0x7f8064078b00 2019-12-05T05:57:18.905 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:18.893+0000 7f8068e63700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80641240c0 con 0x7f80640a3960 2019-12-05T05:57:18.906 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:18.893+0000 7f8068e63700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80641234b0 con 0x7f8064123c90 2019-12-05T05:57:18.906 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:18.893+0000 7f8062d9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f80640a3960 0x7f806411a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:57:18.906 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:18.893+0000 7f806359e700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f8064078b00 0x7f80641266f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:44758/0 (socket says 172.21.15.42:44758) 2019-12-05T05:57:18.906 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:18.893+0000 7f806359e700 1 -- 172.21.15.42:0/1841130576 learned_addr learned my addr 172.21.15.42:0/1841130576 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:57:18.907 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:18.893+0000 7f806259c700 1 --2- 172.21.15.42:0/1841130576 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8064123c90 0x7f8064118430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:57:18.907 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:18.893+0000 7f8062d9d700 1 -- 172.21.15.42:0/1841130576 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8064123c90 msgr2=0x7f8064118430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:57:18.908 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:18.893+0000 7f8062d9d700 1 --2- 172.21.15.42:0/1841130576 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8064123c90 0x7f8064118430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:57:18.908 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:18.893+0000 7f8062d9d700 1 -- 172.21.15.42:0/1841130576 >> v1:172.21.15.42:6789/0 conn(0x7f8064078b00 legacy=0x7f80641266f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:57:18.908 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:18.893+0000 7f8062d9d700 1 -- 172.21.15.42:0/1841130576 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8064118980 con 0x7f80640a3960 2019-12-05T05:57:18.908 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:18.893+0000 7f805b7fe700 1 -- 172.21.15.42:0/1841130576 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 406571509 0 0) 0x7f8064142530 con 0x7f8064078b00 2019-12-05T05:57:18.908 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:18.893+0000 7f8062d9d700 1 --2- 172.21.15.42:0/1841130576 >> v2:172.21.15.42:3300/0 conn(0x7f80640a3960 0x7f806411a930 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f8064123670 tx=0x7f80540108d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:18.909 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:18.893+0000 7f8062d9d700 1 --2- 172.21.15.42:0/1841130576 >> v2:172.21.15.42:3300/0 conn(0x7f80640a3960 0x7f806411a930 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f8064123670 tx=0x7f80540108d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:18.909 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:18.893+0000 7f8062d9d700 1 --2- 172.21.15.42:0/1841130576 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f80640a3960 0x7f806411a930 secure :-1 s=READY pgs=1748 cs=0 l=1 rx=0x7f8064123670 tx=0x7f80540108d0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:57:18.910 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:18.893+0000 7f8062d9d700 1 --2- 172.21.15.42:0/1841130576 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f80640a3960 0x7f806411a930 secure :-1 s=THROTTLE_DONE pgs=1748 cs=0 l=1 rx=0x7f8064123670 tx=0x7f80540108d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:18.910 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:18.893+0000 7f805b7fe700 1 -- 172.21.15.42:0/1841130576 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f80540115c0 con 0x7f80640a3960 2019-12-05T05:57:18.910 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:18.893+0000 7f8062d9d700 1 --2- 172.21.15.42:0/1841130576 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f80640a3960 0x7f806411a930 secure :-1 s=THROTTLE_DONE pgs=1748 cs=0 l=1 rx=0x7f8064123670 tx=0x7f80540108d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:18.910 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:18.893+0000 7f805b7fe700 1 -- 172.21.15.42:0/1841130576 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80540111e0 con 0x7f80640a3960 2019-12-05T05:57:18.911 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:18.893+0000 7f8062d9d700 1 --2- 172.21.15.42:0/1841130576 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f80640a3960 0x7f806411a930 secure :-1 s=THROTTLE_DONE pgs=1748 cs=0 l=1 rx=0x7f8064123670 tx=0x7f80540108d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:18.911 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:18.893+0000 7f8068e63700 1 -- 172.21.15.42:0/1841130576 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8064123130 con 0x7f80640a3960 2019-12-05T05:57:18.911 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:18.893+0000 7f805b7fe700 1 -- 172.21.15.42:0/1841130576 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8054011ba0 con 0x7f80640a3960 2019-12-05T05:57:18.912 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:18.893+0000 7f8068e63700 1 -- 172.21.15.42:0/1841130576 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8064078f30 con 0x7f80640a3960 2019-12-05T05:57:18.912 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:18.897+0000 7f8062d9d700 1 --2- 172.21.15.42:0/1841130576 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f80640a3960 0x7f806411a930 secure :-1 s=THROTTLE_DONE pgs=1748 cs=0 l=1 rx=0x7f8064123670 tx=0x7f80540108d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:18.912 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:18.897+0000 7f805b7fe700 1 -- 172.21.15.42:0/1841130576 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f805400b330 con 0x7f80640a3960 2019-12-05T05:57:18.912 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:18.897+0000 7f8062d9d700 1 --2- 172.21.15.42:0/1841130576 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f80640a3960 0x7f806411a930 secure :-1 s=THROTTLE_DONE pgs=1748 cs=0 l=1 rx=0x7f8064123670 tx=0x7f80540108d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:18.912 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:18.897+0000 7f805b7fe700 1 --2- 172.21.15.42:0/1841130576 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f804803ad00 0x7f804803d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:57:18.912 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:18.897+0000 7f805b7fe700 1 -- 172.21.15.42:0/1841130576 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 9127+0+0 (secure 0 0 0) 0x7f805404b270 con 0x7f80640a3960 2019-12-05T05:57:18.912 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:18.897+0000 7f806259c700 1 --2- 172.21.15.42:0/1841130576 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f804803ad00 0x7f804803d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:57:18.913 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:18.897+0000 7f806259c700 1 --2- 172.21.15.42:0/1841130576 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f804803ad00 0x7f804803d210 crc :-1 s=READY pgs=748 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:57:18.913 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:18.897+0000 7f8068e63700 1 -- 172.21.15.42:0/1841130576 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f806405d110 con 0x7f80640a3960 2019-12-05T05:57:18.913 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:18.905+0000 7f8062d9d700 1 --2- 172.21.15.42:0/1841130576 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f80640a3960 0x7f806411a930 secure :-1 s=THROTTLE_DONE pgs=1748 cs=0 l=1 rx=0x7f8064123670 tx=0x7f80540108d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:18.914 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:18.905+0000 7f805b7fe700 1 -- 172.21.15.42:0/1841130576 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f8054032210 con 0x7f80640a3960 2019-12-05T05:57:19.073 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:19.069+0000 7f8068e63700 1 -- 172.21.15.42:0/1841130576 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f806405b8c0 con 0x7f80640a3960 2019-12-05T05:57:19.073 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:19.069+0000 7f8062d9d700 1 --2- 172.21.15.42:0/1841130576 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f80640a3960 0x7f806411a930 secure :-1 s=THROTTLE_DONE pgs=1748 cs=0 l=1 rx=0x7f8064123670 tx=0x7f80540108d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:19.074 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:19.069+0000 7f805b7fe700 1 -- 172.21.15.42:0/1841130576 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f8054032470 con 0x7f80640a3960 2019-12-05T05:57:19.076 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:57:19.076 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":213,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":8836,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6835/1368848584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1368848584},{"type":"v1","addr":"172.21.15.42:6835","nonce":1368848584}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":213}],"filesystems":[{"mdsmap":{"epoch":212,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:55:44.050934+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6461},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6461":{"gid":6461,"name":"b","rank":0,"incarnation":209,"state":"up:active","state_seq":12,"addr":"172.21.15.40:6833/3549993215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3549993215},{"type":"v1","addr":"172.21.15.40:6833","nonce":3549993215}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:57:19.076 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 213 2019-12-05T05:57:19.079 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:19.073+0000 7f8068e63700 1 -- 172.21.15.42:0/1841130576 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f804803ad00 msgr2=0x7f804803d210 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:57:19.079 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:19.073+0000 7f8068e63700 1 --2- 172.21.15.42:0/1841130576 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f804803ad00 0x7f804803d210 crc :-1 s=READY pgs=748 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:57:19.079 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:19.073+0000 7f8068e63700 1 -- 172.21.15.42:0/1841130576 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f80640a3960 msgr2=0x7f806411a930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:57:19.079 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:19.073+0000 7f8068e63700 1 --2- 172.21.15.42:0/1841130576 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f80640a3960 0x7f806411a930 secure :-1 s=READY pgs=1748 cs=0 l=1 rx=0x7f8064123670 tx=0x7f80540108d0).stop 2019-12-05T05:57:19.080 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:19.073+0000 7f8068e63700 1 -- 172.21.15.42:0/1841130576 shutdown_connections 2019-12-05T05:57:19.080 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:19.073+0000 7f8068e63700 1 --2- 172.21.15.42:0/1841130576 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f804803ad00 0x7f804803d210 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:57:19.080 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:19.073+0000 7f8068e63700 1 --2- 172.21.15.42:0/1841130576 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f80640a3960 0x7f806411a930 unknown :-1 s=CLOSED pgs=1748 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:57:19.080 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:19.073+0000 7f8068e63700 1 --2- 172.21.15.42:0/1841130576 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8064123c90 0x7f8064118430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:57:19.080 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:19.073+0000 7f8068e63700 1 -- 172.21.15.42:0/1841130576 shutdown_connections 2019-12-05T05:57:19.080 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:19.073+0000 7f8068e63700 1 -- 172.21.15.42:0/1841130576 wait complete. 2019-12-05T05:57:19.081 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:19.073+0000 7f8068e63700 1 -- 172.21.15.42:0/1841130576 >> 172.21.15.42:0/1841130576 conn(0x7f806408fde0 msgr2=0x7f80640a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:57:19.089 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T05:57:21.090 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:57:21.193 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:21.185+0000 7f03e10ca700 1 Processor -- start 2019-12-05T05:57:21.194 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:21.185+0000 7f03e10ca700 1 -- start start 2019-12-05T05:57:21.194 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:21.185+0000 7f03e10ca700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f03dc0a7740 0x7f03dc08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:57:21.194 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:21.185+0000 7f03e10ca700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f03dc07e720 0x7f03dc07eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:57:21.194 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:21.185+0000 7f03e10ca700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f03dc09f2e0 0x7f03dc09f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:57:21.194 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:21.185+0000 7f03e10ca700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03dc0a0b00 con 0x7f03dc09f2e0 2019-12-05T05:57:21.195 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:21.185+0000 7f03e10ca700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03dc08e5b0 con 0x7f03dc07e720 2019-12-05T05:57:21.195 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:21.185+0000 7f03e10ca700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03dc08e850 con 0x7f03dc0a7740 2019-12-05T05:57:21.195 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:21.185+0000 7f03dad9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f03dc07e720 0x7f03dc07eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:57:21.195 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:21.185+0000 7f03dad9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f03dc07e720 0x7f03dc07eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:44726/0 (socket says 172.21.15.42:44726) 2019-12-05T05:57:21.195 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:21.185+0000 7f03dad9d700 1 -- 172.21.15.42:0/789309140 learned_addr learned my addr 172.21.15.42:0/789309140 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:57:21.195 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:21.185+0000 7f03e08c9700 1 --2- 172.21.15.42:0/789309140 >> v2:172.21.15.40:3300/0 conn(0x7f03dc09f2e0 0x7f03dc09f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:57:21.195 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:21.185+0000 7f03db59e700 1 --2- 172.21.15.42:0/789309140 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f03dc0a7740 0x7f03dc08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:57:21.196 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:21.185+0000 7f03dad9d700 1 -- 172.21.15.42:0/789309140 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f03dc0a7740 msgr2=0x7f03dc08df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:57:21.196 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:21.185+0000 7f03dad9d700 1 --2- 172.21.15.42:0/789309140 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f03dc0a7740 0x7f03dc08df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:57:21.196 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:21.185+0000 7f03dad9d700 1 -- 172.21.15.42:0/789309140 >> v2:172.21.15.40:3300/0 conn(0x7f03dc09f2e0 msgr2=0x7f03dc09f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:57:21.196 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:21.185+0000 7f03dad9d700 1 --2- 172.21.15.42:0/789309140 >> v2:172.21.15.40:3300/0 conn(0x7f03dc09f2e0 0x7f03dc09f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:57:21.196 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:21.185+0000 7f03dad9d700 1 -- 172.21.15.42:0/789309140 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03dc07f0a0 con 0x7f03dc07e720 2019-12-05T05:57:21.196 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:21.185+0000 7f03dad9d700 1 --2- 172.21.15.42:0/789309140 >> v2:172.21.15.42:3300/0 conn(0x7f03dc07e720 0x7f03dc07eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f03dc08e770 tx=0x7f03d000fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:21.196 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:21.185+0000 7f03dad9d700 1 --2- 172.21.15.42:0/789309140 >> v2:172.21.15.42:3300/0 conn(0x7f03dc07e720 0x7f03dc07eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f03dc08e770 tx=0x7f03d000fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:21.197 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:21.185+0000 7f03dad9d700 1 --2- 172.21.15.42:0/789309140 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f03dc07e720 0x7f03dc07eb50 secure :-1 s=READY pgs=1749 cs=0 l=1 rx=0x7f03dc08e770 tx=0x7f03d000fdb0).ready entity=mon.1 client_cookie=e016e50dfdd7d665 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:57:21.197 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:21.189+0000 7f03dad9d700 1 --2- 172.21.15.42:0/789309140 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f03dc07e720 0x7f03dc07eb50 secure :-1 s=THROTTLE_DONE pgs=1749 cs=0 l=1 rx=0x7f03dc08e770 tx=0x7f03d000fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:21.197 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:21.189+0000 7f03da59c700 1 -- 172.21.15.42:0/789309140 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f03d0008c90 con 0x7f03dc07e720 2019-12-05T05:57:21.197 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:21.189+0000 7f03dad9d700 1 --2- 172.21.15.42:0/789309140 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f03dc07e720 0x7f03dc07eb50 secure :-1 s=THROTTLE_DONE pgs=1749 cs=0 l=1 rx=0x7f03dc08e770 tx=0x7f03d000fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:21.197 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:21.189+0000 7f03dad9d700 1 --2- 172.21.15.42:0/789309140 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f03dc07e720 0x7f03dc07eb50 secure :-1 s=THROTTLE_DONE pgs=1749 cs=0 l=1 rx=0x7f03dc08e770 tx=0x7f03d000fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:21.197 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:21.189+0000 7f03da59c700 1 -- 172.21.15.42:0/789309140 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03d00088b0 con 0x7f03dc07e720 2019-12-05T05:57:21.197 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:21.189+0000 7f03da59c700 1 -- 172.21.15.42:0/789309140 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f03d0011b50 con 0x7f03dc07e720 2019-12-05T05:57:21.198 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:21.189+0000 7f03e10ca700 1 -- 172.21.15.42:0/789309140 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f03dc07e720 msgr2=0x7f03dc07eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:57:21.198 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:21.189+0000 7f03e10ca700 1 --2- 172.21.15.42:0/789309140 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f03dc07e720 0x7f03dc07eb50 secure :-1 s=READY pgs=1749 cs=0 l=1 rx=0x7f03dc08e770 tx=0x7f03d000fdb0).stop 2019-12-05T05:57:21.198 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:21.189+0000 7f03e10ca700 1 -- 172.21.15.42:0/789309140 shutdown_connections 2019-12-05T05:57:21.198 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:21.189+0000 7f03e10ca700 1 --2- 172.21.15.42:0/789309140 >> v2:172.21.15.40:3300/0 conn(0x7f03dc09f2e0 0x7f03dc09f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:57:21.198 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:21.189+0000 7f03e10ca700 1 --2- 172.21.15.42:0/789309140 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f03dc0a7740 0x7f03dc08df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:57:21.198 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:21.189+0000 7f03e10ca700 1 --2- 172.21.15.42:0/789309140 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f03dc07e720 0x7f03dc07eb50 unknown :-1 s=CLOSED pgs=1749 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:57:21.198 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:21.189+0000 7f03e10ca700 1 -- 172.21.15.42:0/789309140 shutdown_connections 2019-12-05T05:57:21.199 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:21.189+0000 7f03e10ca700 1 -- 172.21.15.42:0/789309140 wait complete. 2019-12-05T05:57:21.199 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:21.189+0000 7f03e10ca700 1 -- 172.21.15.42:0/789309140 >> 172.21.15.42:0/789309140 conn(0x7f03dc0aabe0 msgr2=0x7f03dc0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:57:21.202 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:21.189+0000 7f03e10ca700 1 Processor -- start 2019-12-05T05:57:21.202 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:21.189+0000 7f03e10ca700 1 -- start start 2019-12-05T05:57:21.202 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:21.193+0000 7f03e10ca700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f03dc1225b0 0x7f03dc116360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:57:21.203 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:21.193+0000 7f03e10ca700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03dc119eb0 con 0x7f03dc119a80 2019-12-05T05:57:21.203 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:21.193+0000 7f03e10ca700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03dc078b10 con 0x7f03dc0a5d00 2019-12-05T05:57:21.203 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:21.193+0000 7f03e10ca700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03dc122bd0 con 0x7f03dc1225b0 2019-12-05T05:57:21.203 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:21.193+0000 7f03e08c9700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f03dc0a5d00 0x7f03dc0762f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:44770/0 (socket says 172.21.15.42:44770) 2019-12-05T05:57:21.203 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:21.193+0000 7f03e08c9700 1 -- 172.21.15.42:0/3285594037 learned_addr learned my addr 172.21.15.42:0/3285594037 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:57:21.203 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:21.193+0000 7f03db59e700 1 --2- 172.21.15.42:0/3285594037 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f03dc1225b0 0x7f03dc116360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:57:21.203 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:21.193+0000 7f03c3fff700 1 -- 172.21.15.42:0/3285594037 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1669607759 0 0) 0x7f03dc078b10 con 0x7f03dc0a5d00 2019-12-05T05:57:21.204 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:21.193+0000 7f03c3fff700 1 -- 172.21.15.42:0/3285594037 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03bc0030b0 con 0x7f03dc0a5d00 2019-12-05T05:57:21.204 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:21.193+0000 7f03c3fff700 1 -- 172.21.15.42:0/3285594037 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 149823643 0 0) 0x7f03dc119eb0 con 0x7f03dc119a80 2019-12-05T05:57:21.204 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:21.193+0000 7f03c3fff700 1 -- 172.21.15.42:0/3285594037 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03dc078b10 con 0x7f03dc119a80 2019-12-05T05:57:21.204 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:21.193+0000 7f03db59e700 1 -- 172.21.15.42:0/3285594037 >> v1:172.21.15.42:6789/0 conn(0x7f03dc0a5d00 legacy=0x7f03dc0762f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:57:21.204 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:21.193+0000 7f03db59e700 1 -- 172.21.15.42:0/3285594037 >> v1:172.21.15.40:6789/0 conn(0x7f03dc119a80 legacy=0x7f03dc0a5500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:57:21.204 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:21.193+0000 7f03db59e700 1 -- 172.21.15.42:0/3285594037 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03dc079d50 con 0x7f03dc1225b0 2019-12-05T05:57:21.204 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:21.193+0000 7f03db59e700 1 --2- 172.21.15.42:0/3285594037 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f03dc1225b0 0x7f03dc116360 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f03cc002f30 tx=0x7f03cc010a10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:21.205 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:21.193+0000 7f03db59e700 1 --2- 172.21.15.42:0/3285594037 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f03dc1225b0 0x7f03dc116360 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f03cc002f30 tx=0x7f03cc010a10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:21.205 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:21.193+0000 7f03db59e700 1 --2- 172.21.15.42:0/3285594037 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f03dc1225b0 0x7f03dc116360 secure :-1 s=READY pgs=396 cs=0 l=1 rx=0x7f03cc002f30 tx=0x7f03cc010a10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:57:21.205 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:21.193+0000 7f03db59e700 1 --2- 172.21.15.42:0/3285594037 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f03dc1225b0 0x7f03dc116360 secure :-1 s=THROTTLE_DONE pgs=396 cs=0 l=1 rx=0x7f03cc002f30 tx=0x7f03cc010a10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:21.205 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:21.193+0000 7f03c3fff700 1 -- 172.21.15.42:0/3285594037 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f03cc005920 con 0x7f03dc1225b0 2019-12-05T05:57:21.205 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:21.193+0000 7f03db59e700 1 --2- 172.21.15.42:0/3285594037 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f03dc1225b0 0x7f03dc116360 secure :-1 s=THROTTLE_DONE pgs=396 cs=0 l=1 rx=0x7f03cc002f30 tx=0x7f03cc010a10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:21.205 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:21.193+0000 7f03db59e700 1 --2- 172.21.15.42:0/3285594037 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f03dc1225b0 0x7f03dc116360 secure :-1 s=THROTTLE_DONE pgs=396 cs=0 l=1 rx=0x7f03cc002f30 tx=0x7f03cc010a10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:21.205 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:21.193+0000 7f03c3fff700 1 -- 172.21.15.42:0/3285594037 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03cc005540 con 0x7f03dc1225b0 2019-12-05T05:57:21.206 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:21.193+0000 7f03e10ca700 1 -- 172.21.15.42:0/3285594037 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03dc077b00 con 0x7f03dc1225b0 2019-12-05T05:57:21.206 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:21.193+0000 7f03c3fff700 1 -- 172.21.15.42:0/3285594037 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f03cc012540 con 0x7f03dc1225b0 2019-12-05T05:57:21.206 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:21.193+0000 7f03e10ca700 1 -- 172.21.15.42:0/3285594037 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03dc078080 con 0x7f03dc1225b0 2019-12-05T05:57:21.206 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:21.197+0000 7f03db59e700 1 --2- 172.21.15.42:0/3285594037 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f03dc1225b0 0x7f03dc116360 secure :-1 s=THROTTLE_DONE pgs=396 cs=0 l=1 rx=0x7f03cc002f30 tx=0x7f03cc010a10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:21.206 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:21.197+0000 7f03c3fff700 1 -- 172.21.15.42:0/3285594037 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f03cc007140 con 0x7f03dc1225b0 2019-12-05T05:57:21.206 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:21.197+0000 7f03db59e700 1 --2- 172.21.15.42:0/3285594037 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f03dc1225b0 0x7f03dc116360 secure :-1 s=THROTTLE_DONE pgs=396 cs=0 l=1 rx=0x7f03cc002f30 tx=0x7f03cc010a10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:21.206 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:21.197+0000 7f03c3fff700 1 --2- 172.21.15.42:0/3285594037 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f03bc03b470 0x7f03bc03d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:57:21.207 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:21.197+0000 7f03c3fff700 1 -- 172.21.15.42:0/3285594037 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 9127+0+0 (secure 0 0 0) 0x7f03cc04ac00 con 0x7f03dc1225b0 2019-12-05T05:57:21.207 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:21.197+0000 7f03dad9d700 1 --2- 172.21.15.42:0/3285594037 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f03bc03b470 0x7f03bc03d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:57:21.207 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:21.197+0000 7f03dad9d700 1 --2- 172.21.15.42:0/3285594037 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f03bc03b470 0x7f03bc03d980 crc :-1 s=READY pgs=749 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:57:21.207 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:21.201+0000 7f03e10ca700 1 -- 172.21.15.42:0/3285594037 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03dc05d110 con 0x7f03dc1225b0 2019-12-05T05:57:21.214 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:21.209+0000 7f03db59e700 1 --2- 172.21.15.42:0/3285594037 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f03dc1225b0 0x7f03dc116360 secure :-1 s=THROTTLE_DONE pgs=396 cs=0 l=1 rx=0x7f03cc002f30 tx=0x7f03cc010a10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:21.214 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:21.209+0000 7f03c3fff700 1 -- 172.21.15.42:0/3285594037 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f03cc031ea0 con 0x7f03dc1225b0 2019-12-05T05:57:21.384 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:21.381+0000 7f03e10ca700 1 -- 172.21.15.42:0/3285594037 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f03dc05b8c0 con 0x7f03dc1225b0 2019-12-05T05:57:21.386 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:21.381+0000 7f03db59e700 1 --2- 172.21.15.42:0/3285594037 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f03dc1225b0 0x7f03dc116360 secure :-1 s=THROTTLE_DONE pgs=396 cs=0 l=1 rx=0x7f03cc002f30 tx=0x7f03cc010a10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:21.386 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:21.381+0000 7f03c3fff700 1 -- 172.21.15.42:0/3285594037 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f03cc032a50 con 0x7f03dc1225b0 2019-12-05T05:57:21.388 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 213 2019-12-05T05:57:21.388 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:57:21.388 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":213,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":8836,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6835/1368848584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1368848584},{"type":"v1","addr":"172.21.15.42:6835","nonce":1368848584}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":213}],"filesystems":[{"mdsmap":{"epoch":212,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:55:44.050934+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6461},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6461":{"gid":6461,"name":"b","rank":0,"incarnation":209,"state":"up:active","state_seq":12,"addr":"172.21.15.40:6833/3549993215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3549993215},{"type":"v1","addr":"172.21.15.40:6833","nonce":3549993215}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:57:21.391 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:21.385+0000 7f03e10ca700 1 -- 172.21.15.42:0/3285594037 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f03bc03b470 msgr2=0x7f03bc03d980 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:57:21.391 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:21.385+0000 7f03e10ca700 1 --2- 172.21.15.42:0/3285594037 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f03bc03b470 0x7f03bc03d980 crc :-1 s=READY pgs=749 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:57:21.391 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:21.385+0000 7f03e10ca700 1 -- 172.21.15.42:0/3285594037 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f03dc1225b0 msgr2=0x7f03dc116360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:57:21.392 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:21.385+0000 7f03e10ca700 1 --2- 172.21.15.42:0/3285594037 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f03dc1225b0 0x7f03dc116360 secure :-1 s=READY pgs=396 cs=0 l=1 rx=0x7f03cc002f30 tx=0x7f03cc010a10).stop 2019-12-05T05:57:21.392 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:21.385+0000 7f03e10ca700 1 -- 172.21.15.42:0/3285594037 shutdown_connections 2019-12-05T05:57:21.392 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:21.385+0000 7f03e10ca700 1 --2- 172.21.15.42:0/3285594037 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f03bc03b470 0x7f03bc03d980 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:57:21.392 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:21.385+0000 7f03e10ca700 1 --2- 172.21.15.42:0/3285594037 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f03dc1225b0 0x7f03dc116360 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:57:21.392 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:21.385+0000 7f03e10ca700 1 -- 172.21.15.42:0/3285594037 shutdown_connections 2019-12-05T05:57:21.392 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:21.385+0000 7f03e10ca700 1 -- 172.21.15.42:0/3285594037 wait complete. 2019-12-05T05:57:21.392 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:21.385+0000 7f03e10ca700 1 -- 172.21.15.42:0/3285594037 >> 172.21.15.42:0/3285594037 conn(0x7f03dc0a8fc0 msgr2=0x7f03dc0a9790 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:57:21.401 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T05:57:23.402 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:57:23.508 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:23.501+0000 7f91497a3700 1 Processor -- start 2019-12-05T05:57:23.509 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:23.501+0000 7f91497a3700 1 -- start start 2019-12-05T05:57:23.509 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:23.501+0000 7f91497a3700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f91440a7740 0x7f914408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:57:23.509 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:23.501+0000 7f91497a3700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f914409fae0 0x7f914407eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:57:23.509 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:23.501+0000 7f91497a3700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91440a0710 con 0x7f914407e720 2019-12-05T05:57:23.509 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:23.501+0000 7f91497a3700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91440a0b00 con 0x7f914409fae0 2019-12-05T05:57:23.510 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:23.501+0000 7f91497a3700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f914408e5b0 con 0x7f91440a7740 2019-12-05T05:57:23.510 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:23.501+0000 7f91437fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f91440a7740 0x7f914408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:57:23.510 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:23.501+0000 7f91437fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f91440a7740 0x7f914408df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:44740/0 (socket says 172.21.15.42:44740) 2019-12-05T05:57:23.510 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:23.501+0000 7f91437fe700 1 -- 172.21.15.42:0/1596595411 learned_addr learned my addr 172.21.15.42:0/1596595411 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:57:23.510 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:23.501+0000 7f9143fff700 1 --2- 172.21.15.42:0/1596595411 >> v2:172.21.15.40:3300/0 conn(0x7f914409fae0 0x7f914407eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:57:23.510 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:23.501+0000 7f91437fe700 1 -- 172.21.15.42:0/1596595411 >> v2:172.21.15.40:3300/0 conn(0x7f914409fae0 msgr2=0x7f914407eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:57:23.510 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:23.501+0000 7f91437fe700 1 --2- 172.21.15.42:0/1596595411 >> v2:172.21.15.40:3300/0 conn(0x7f914409fae0 0x7f914407eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:57:23.511 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:23.501+0000 7f91437fe700 1 -- 172.21.15.42:0/1596595411 >> v1:172.21.15.40:6789/0 conn(0x7f914407e720 legacy=0x7f914409f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:57:23.511 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:23.501+0000 7f91437fe700 1 -- 172.21.15.42:0/1596595411 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f914407f0a0 con 0x7f91440a7740 2019-12-05T05:57:23.511 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:23.501+0000 7f91437fe700 1 --2- 172.21.15.42:0/1596595411 >> v2:172.21.15.42:3300/0 conn(0x7f91440a7740 0x7f914408df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f912c002f30 tx=0x7f912c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:23.511 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:23.501+0000 7f91437fe700 1 --2- 172.21.15.42:0/1596595411 >> v2:172.21.15.42:3300/0 conn(0x7f91440a7740 0x7f914408df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f912c002f30 tx=0x7f912c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:23.511 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:23.501+0000 7f91437fe700 1 --2- 172.21.15.42:0/1596595411 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f91440a7740 0x7f914408df80 secure :-1 s=READY pgs=1751 cs=0 l=1 rx=0x7f912c002f30 tx=0x7f912c00fd30).ready entity=mon.1 client_cookie=d47daaea3accd1bd server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:57:23.511 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:23.501+0000 7f91437fe700 1 --2- 172.21.15.42:0/1596595411 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f91440a7740 0x7f914408df80 secure :-1 s=THROTTLE_DONE pgs=1751 cs=0 l=1 rx=0x7f912c002f30 tx=0x7f912c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:23.511 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:23.501+0000 7f91437fe700 1 --2- 172.21.15.42:0/1596595411 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f91440a7740 0x7f914408df80 secure :-1 s=THROTTLE_DONE pgs=1751 cs=0 l=1 rx=0x7f912c002f30 tx=0x7f912c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:23.512 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:23.501+0000 7f91427fc700 1 -- 172.21.15.42:0/1596595411 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f912c008c90 con 0x7f91440a7740 2019-12-05T05:57:23.512 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:23.501+0000 7f91437fe700 1 --2- 172.21.15.42:0/1596595411 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f91440a7740 0x7f914408df80 secure :-1 s=THROTTLE_DONE pgs=1751 cs=0 l=1 rx=0x7f912c002f30 tx=0x7f912c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:23.512 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:23.501+0000 7f91427fc700 1 -- 172.21.15.42:0/1596595411 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f912c0088b0 con 0x7f91440a7740 2019-12-05T05:57:23.512 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:23.501+0000 7f91427fc700 1 -- 172.21.15.42:0/1596595411 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f912c011ad0 con 0x7f91440a7740 2019-12-05T05:57:23.512 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:23.505+0000 7f91497a3700 1 -- 172.21.15.42:0/1596595411 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f91440a7740 msgr2=0x7f914408df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:57:23.512 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:23.505+0000 7f91497a3700 1 --2- 172.21.15.42:0/1596595411 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f91440a7740 0x7f914408df80 secure :-1 s=READY pgs=1751 cs=0 l=1 rx=0x7f912c002f30 tx=0x7f912c00fd30).stop 2019-12-05T05:57:23.512 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:23.505+0000 7f91497a3700 1 -- 172.21.15.42:0/1596595411 shutdown_connections 2019-12-05T05:57:23.513 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:23.505+0000 7f91497a3700 1 --2- 172.21.15.42:0/1596595411 >> v2:172.21.15.40:3300/0 conn(0x7f914409fae0 0x7f914407eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:57:23.513 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:23.505+0000 7f91497a3700 1 --2- 172.21.15.42:0/1596595411 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f91440a7740 0x7f914408df80 unknown :-1 s=CLOSED pgs=1751 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:57:23.513 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:23.505+0000 7f91497a3700 1 -- 172.21.15.42:0/1596595411 shutdown_connections 2019-12-05T05:57:23.513 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:23.505+0000 7f91497a3700 1 -- 172.21.15.42:0/1596595411 wait complete. 2019-12-05T05:57:23.513 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:23.505+0000 7f91497a3700 1 -- 172.21.15.42:0/1596595411 >> 172.21.15.42:0/1596595411 conn(0x7f91440aabe0 msgr2=0x7f91440a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:57:23.517 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:23.505+0000 7f91497a3700 1 Processor -- start 2019-12-05T05:57:23.517 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:23.505+0000 7f91497a3700 1 -- start start 2019-12-05T05:57:23.517 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:23.509+0000 7f91497a3700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f9144116500 0x7f9144118130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:57:23.517 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:23.509+0000 7f91497a3700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f9144133c10 0x7f9144134040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:57:23.517 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:23.509+0000 7f91497a3700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f914408eb50 0x7f914408ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:57:23.517 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:23.509+0000 7f91497a3700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9144134590 con 0x7f9144116500 2019-12-05T05:57:23.518 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:23.509+0000 7f91497a3700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9144118ae0 con 0x7f9144133c10 2019-12-05T05:57:23.518 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:23.509+0000 7f91497a3700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91441336d0 con 0x7f914408eb50 2019-12-05T05:57:23.518 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:23.509+0000 7f9142ffd700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f9144133c10 0x7f9144134040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:57:23.518 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:23.509+0000 7f9142ffd700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f9144133c10 0x7f9144134040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:44746/0 (socket says 172.21.15.42:44746) 2019-12-05T05:57:23.518 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:23.509+0000 7f9142ffd700 1 -- 172.21.15.42:0/3456775796 learned_addr learned my addr 172.21.15.42:0/3456775796 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:57:23.518 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:23.509+0000 7f91437fe700 1 --2- 172.21.15.42:0/3456775796 >> v2:172.21.15.40:3300/0 conn(0x7f9144116500 0x7f9144118130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:57:23.518 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:23.509+0000 7f9143fff700 1 --2- 172.21.15.42:0/3456775796 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f914408eb50 0x7f914408ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:57:23.519 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:23.509+0000 7f9142ffd700 1 -- 172.21.15.42:0/3456775796 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f914408eb50 msgr2=0x7f914408ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:57:23.519 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:23.509+0000 7f9142ffd700 1 --2- 172.21.15.42:0/3456775796 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f914408eb50 0x7f914408ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:57:23.519 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:23.509+0000 7f9142ffd700 1 -- 172.21.15.42:0/3456775796 >> v2:172.21.15.40:3300/0 conn(0x7f9144116500 msgr2=0x7f9144118130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:57:23.519 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:23.509+0000 7f9142ffd700 1 --2- 172.21.15.42:0/3456775796 >> v2:172.21.15.40:3300/0 conn(0x7f9144116500 0x7f9144118130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:57:23.519 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:23.509+0000 7f9142ffd700 1 -- 172.21.15.42:0/3456775796 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9144145370 con 0x7f9144133c10 2019-12-05T05:57:23.519 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:23.509+0000 7f9143fff700 1 --2- 172.21.15.42:0/3456775796 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f914408eb50 0x7f914408ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T05:57:23.519 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:23.509+0000 7f9142ffd700 1 --2- 172.21.15.42:0/3456775796 >> v2:172.21.15.42:3300/0 conn(0x7f9144133c10 0x7f9144134040 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f9144133890 tx=0x7f9134010da0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:23.520 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:23.509+0000 7f9142ffd700 1 --2- 172.21.15.42:0/3456775796 >> v2:172.21.15.42:3300/0 conn(0x7f9144133c10 0x7f9144134040 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f9144133890 tx=0x7f9134010da0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:23.520 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:23.509+0000 7f9142ffd700 1 --2- 172.21.15.42:0/3456775796 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9144133c10 0x7f9144134040 secure :-1 s=READY pgs=1752 cs=0 l=1 rx=0x7f9144133890 tx=0x7f9134010da0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:57:23.520 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:23.509+0000 7f9142ffd700 1 --2- 172.21.15.42:0/3456775796 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9144133c10 0x7f9144134040 secure :-1 s=THROTTLE_DONE pgs=1752 cs=0 l=1 rx=0x7f9144133890 tx=0x7f9134010da0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:23.520 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:23.509+0000 7f9142ffd700 1 --2- 172.21.15.42:0/3456775796 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9144133c10 0x7f9144134040 secure :-1 s=THROTTLE_DONE pgs=1752 cs=0 l=1 rx=0x7f9144133890 tx=0x7f9134010da0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:23.520 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:23.509+0000 7f9142ffd700 1 --2- 172.21.15.42:0/3456775796 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9144133c10 0x7f9144134040 secure :-1 s=THROTTLE_DONE pgs=1752 cs=0 l=1 rx=0x7f9144133890 tx=0x7f9134010da0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:23.520 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:23.509+0000 7f913bfff700 1 -- 172.21.15.42:0/3456775796 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f913400ca40 con 0x7f9144133c10 2019-12-05T05:57:23.520 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:23.509+0000 7f913bfff700 1 -- 172.21.15.42:0/3456775796 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91340101c0 con 0x7f9144133c10 2019-12-05T05:57:23.521 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:23.509+0000 7f91497a3700 1 -- 172.21.15.42:0/3456775796 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9144118680 con 0x7f9144133c10 2019-12-05T05:57:23.521 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:23.509+0000 7f913bfff700 1 -- 172.21.15.42:0/3456775796 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f91340125d0 con 0x7f9144133c10 2019-12-05T05:57:23.521 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:23.509+0000 7f91497a3700 1 -- 172.21.15.42:0/3456775796 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f914408f4d0 con 0x7f9144133c10 2019-12-05T05:57:23.521 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:23.509+0000 7f9142ffd700 1 --2- 172.21.15.42:0/3456775796 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9144133c10 0x7f9144134040 secure :-1 s=THROTTLE_DONE pgs=1752 cs=0 l=1 rx=0x7f9144133890 tx=0x7f9134010da0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:23.521 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:23.509+0000 7f913bfff700 1 -- 172.21.15.42:0/3456775796 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f91340128a0 con 0x7f9144133c10 2019-12-05T05:57:23.521 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:23.509+0000 7f9142ffd700 1 --2- 172.21.15.42:0/3456775796 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9144133c10 0x7f9144134040 secure :-1 s=THROTTLE_DONE pgs=1752 cs=0 l=1 rx=0x7f9144133890 tx=0x7f9134010da0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:23.521 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:23.513+0000 7f913bfff700 1 --2- 172.21.15.42:0/3456775796 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f911c03ae60 0x7f911c03d370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:57:23.522 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:23.513+0000 7f913bfff700 1 -- 172.21.15.42:0/3456775796 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 9127+0+0 (secure 0 0 0) 0x7f913404b4e0 con 0x7f9144133c10 2019-12-05T05:57:23.522 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:23.513+0000 7f91437fe700 1 --2- 172.21.15.42:0/3456775796 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f911c03ae60 0x7f911c03d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:57:23.522 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:23.513+0000 7f91437fe700 1 --2- 172.21.15.42:0/3456775796 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f911c03ae60 0x7f911c03d370 crc :-1 s=READY pgs=750 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:57:23.522 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:23.517+0000 7f91497a3700 1 -- 172.21.15.42:0/3456775796 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f914405d110 con 0x7f9144133c10 2019-12-05T05:57:23.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:23.525+0000 7f9142ffd700 1 --2- 172.21.15.42:0/3456775796 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9144133c10 0x7f9144134040 secure :-1 s=THROTTLE_DONE pgs=1752 cs=0 l=1 rx=0x7f9144133890 tx=0x7f9134010da0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:23.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:23.525+0000 7f913bfff700 1 -- 172.21.15.42:0/3456775796 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f9134032560 con 0x7f9144133c10 2019-12-05T05:57:23.693 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:23.689+0000 7f91497a3700 1 -- 172.21.15.42:0/3456775796 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f914405b8c0 con 0x7f9144133c10 2019-12-05T05:57:23.694 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:23.689+0000 7f9142ffd700 1 --2- 172.21.15.42:0/3456775796 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9144133c10 0x7f9144134040 secure :-1 s=THROTTLE_DONE pgs=1752 cs=0 l=1 rx=0x7f9144133890 tx=0x7f9134010da0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:23.694 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:23.689+0000 7f913bfff700 1 -- 172.21.15.42:0/3456775796 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f91340327c0 con 0x7f9144133c10 2019-12-05T05:57:23.696 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:57:23.696 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":213,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":8836,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6835/1368848584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1368848584},{"type":"v1","addr":"172.21.15.42:6835","nonce":1368848584}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":213}],"filesystems":[{"mdsmap":{"epoch":212,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:55:44.050934+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6461},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6461":{"gid":6461,"name":"b","rank":0,"incarnation":209,"state":"up:active","state_seq":12,"addr":"172.21.15.40:6833/3549993215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3549993215},{"type":"v1","addr":"172.21.15.40:6833","nonce":3549993215}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:57:23.696 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 213 2019-12-05T05:57:23.698 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:23.693+0000 7f91497a3700 1 -- 172.21.15.42:0/3456775796 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f911c03ae60 msgr2=0x7f911c03d370 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:57:23.699 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:23.693+0000 7f91497a3700 1 --2- 172.21.15.42:0/3456775796 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f911c03ae60 0x7f911c03d370 crc :-1 s=READY pgs=750 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:57:23.699 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:23.693+0000 7f91497a3700 1 -- 172.21.15.42:0/3456775796 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9144133c10 msgr2=0x7f9144134040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:57:23.699 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:23.693+0000 7f91497a3700 1 --2- 172.21.15.42:0/3456775796 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9144133c10 0x7f9144134040 secure :-1 s=READY pgs=1752 cs=0 l=1 rx=0x7f9144133890 tx=0x7f9134010da0).stop 2019-12-05T05:57:23.699 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:23.693+0000 7f91497a3700 1 -- 172.21.15.42:0/3456775796 shutdown_connections 2019-12-05T05:57:23.699 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:23.693+0000 7f91497a3700 1 --2- 172.21.15.42:0/3456775796 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f911c03ae60 0x7f911c03d370 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:57:23.699 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:23.693+0000 7f91497a3700 1 --2- 172.21.15.42:0/3456775796 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f914408eb50 0x7f914408ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:57:23.700 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:23.693+0000 7f91497a3700 1 --2- 172.21.15.42:0/3456775796 >> v2:172.21.15.40:3300/0 conn(0x7f9144116500 0x7f9144118130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:57:23.700 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:23.693+0000 7f91497a3700 1 --2- 172.21.15.42:0/3456775796 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9144133c10 0x7f9144134040 unknown :-1 s=CLOSED pgs=1752 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:57:23.700 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:23.693+0000 7f91497a3700 1 -- 172.21.15.42:0/3456775796 shutdown_connections 2019-12-05T05:57:23.700 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:23.693+0000 7f91497a3700 1 -- 172.21.15.42:0/3456775796 wait complete. 2019-12-05T05:57:23.700 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:23.693+0000 7f91497a3700 1 -- 172.21.15.42:0/3456775796 >> 172.21.15.42:0/3456775796 conn(0x7f91440a7740 msgr2=0x7f914409f2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:57:23.709 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T05:57:25.711 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:57:25.802 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:25.797+0000 7fb2d720d700 1 Processor -- start 2019-12-05T05:57:25.803 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:25.797+0000 7fb2d720d700 1 -- start start 2019-12-05T05:57:25.803 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:25.797+0000 7fb2d720d700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb2d00a7740 0x7fb2d008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:57:25.803 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:25.797+0000 7fb2d720d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb2d009fae0 0x7fb2d007eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:57:25.803 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:25.797+0000 7fb2d720d700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2d00a0710 con 0x7fb2d007e720 2019-12-05T05:57:25.803 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:25.797+0000 7fb2d720d700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2d00a0b00 con 0x7fb2d00a7740 2019-12-05T05:57:25.803 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:25.797+0000 7fb2d720d700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2d008e5b0 con 0x7fb2d009fae0 2019-12-05T05:57:25.803 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:25.797+0000 7fb2d5fab700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb2d009fae0 0x7fb2d007eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:57:25.804 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:25.797+0000 7fb2d4fa9700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fb2d007e720 0x7fb2d009f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:44808/0 (socket says 172.21.15.42:44808) 2019-12-05T05:57:25.804 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:25.797+0000 7fb2d4fa9700 1 -- 172.21.15.42:0/2084776307 learned_addr learned my addr 172.21.15.42:0/2084776307 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:57:25.804 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:25.797+0000 7fb2d57aa700 1 --2- 172.21.15.42:0/2084776307 >> v2:172.21.15.40:3300/0 conn(0x7fb2d00a7740 0x7fb2d008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:57:25.804 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:25.797+0000 7fb2d5fab700 1 -- 172.21.15.42:0/2084776307 >> v2:172.21.15.40:3300/0 conn(0x7fb2d00a7740 msgr2=0x7fb2d008df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:57:25.804 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:25.797+0000 7fb2d5fab700 1 --2- 172.21.15.42:0/2084776307 >> v2:172.21.15.40:3300/0 conn(0x7fb2d00a7740 0x7fb2d008df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:57:25.804 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:25.797+0000 7fb2d5fab700 1 -- 172.21.15.42:0/2084776307 >> v1:172.21.15.42:6789/0 conn(0x7fb2d007e720 legacy=0x7fb2d009f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:57:25.804 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:25.797+0000 7fb2c7fff700 1 -- 172.21.15.42:0/2084776307 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2248834409 0 0) 0x7fb2d00a0710 con 0x7fb2d007e720 2019-12-05T05:57:25.805 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:25.797+0000 7fb2d5fab700 1 -- 172.21.15.42:0/2084776307 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2d007f0a0 con 0x7fb2d009fae0 2019-12-05T05:57:25.805 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:25.797+0000 7fb2d5fab700 1 --2- 172.21.15.42:0/2084776307 >> v2:172.21.15.42:3300/0 conn(0x7fb2d009fae0 0x7fb2d007eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fb2c8002f30 tx=0x7fb2c800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:25.805 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:25.797+0000 7fb2d5fab700 1 --2- 172.21.15.42:0/2084776307 >> v2:172.21.15.42:3300/0 conn(0x7fb2d009fae0 0x7fb2d007eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fb2c8002f30 tx=0x7fb2c800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:25.805 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:25.797+0000 7fb2d5fab700 1 --2- 172.21.15.42:0/2084776307 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb2d009fae0 0x7fb2d007eb50 secure :-1 s=READY pgs=1754 cs=0 l=1 rx=0x7fb2c8002f30 tx=0x7fb2c800fc70).ready entity=mon.1 client_cookie=aaa627145a0985a9 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:57:25.805 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:25.797+0000 7fb2d5fab700 1 --2- 172.21.15.42:0/2084776307 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb2d009fae0 0x7fb2d007eb50 secure :-1 s=THROTTLE_DONE pgs=1754 cs=0 l=1 rx=0x7fb2c8002f30 tx=0x7fb2c800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:25.805 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:25.797+0000 7fb2c7fff700 1 -- 172.21.15.42:0/2084776307 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb2c8008bb0 con 0x7fb2d009fae0 2019-12-05T05:57:25.805 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:25.797+0000 7fb2d5fab700 1 --2- 172.21.15.42:0/2084776307 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb2d009fae0 0x7fb2d007eb50 secure :-1 s=THROTTLE_DONE pgs=1754 cs=0 l=1 rx=0x7fb2c8002f30 tx=0x7fb2c800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:25.806 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:25.797+0000 7fb2c7fff700 1 -- 172.21.15.42:0/2084776307 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2c80087d0 con 0x7fb2d009fae0 2019-12-05T05:57:25.806 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:25.797+0000 7fb2d5fab700 1 --2- 172.21.15.42:0/2084776307 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb2d009fae0 0x7fb2d007eb50 secure :-1 s=THROTTLE_DONE pgs=1754 cs=0 l=1 rx=0x7fb2c8002f30 tx=0x7fb2c800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:25.806 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:25.797+0000 7fb2c7fff700 1 -- 172.21.15.42:0/2084776307 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb2c8011820 con 0x7fb2d009fae0 2019-12-05T05:57:25.806 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:25.797+0000 7fb2d720d700 1 -- 172.21.15.42:0/2084776307 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb2d009fae0 msgr2=0x7fb2d007eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:57:25.806 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:25.797+0000 7fb2d720d700 1 --2- 172.21.15.42:0/2084776307 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb2d009fae0 0x7fb2d007eb50 secure :-1 s=READY pgs=1754 cs=0 l=1 rx=0x7fb2c8002f30 tx=0x7fb2c800fc70).stop 2019-12-05T05:57:25.806 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:25.797+0000 7fb2d720d700 1 -- 172.21.15.42:0/2084776307 shutdown_connections 2019-12-05T05:57:25.806 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:25.797+0000 7fb2d720d700 1 --2- 172.21.15.42:0/2084776307 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb2d009fae0 0x7fb2d007eb50 unknown :-1 s=CLOSED pgs=1754 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:57:25.807 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:25.797+0000 7fb2d720d700 1 --2- 172.21.15.42:0/2084776307 >> v2:172.21.15.40:3300/0 conn(0x7fb2d00a7740 0x7fb2d008df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:57:25.807 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:25.797+0000 7fb2d720d700 1 -- 172.21.15.42:0/2084776307 shutdown_connections 2019-12-05T05:57:25.807 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:25.797+0000 7fb2d720d700 1 -- 172.21.15.42:0/2084776307 wait complete. 2019-12-05T05:57:25.807 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:25.797+0000 7fb2d720d700 1 -- 172.21.15.42:0/2084776307 >> 172.21.15.42:0/2084776307 conn(0x7fb2d00aabe0 msgr2=0x7fb2d00a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:57:25.813 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:25.801+0000 7fb2d720d700 1 Processor -- start 2019-12-05T05:57:25.813 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:25.801+0000 7fb2d720d700 1 -- start start 2019-12-05T05:57:25.813 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:25.801+0000 7fb2d720d700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb2d01190e0 0x7fb2d00a17b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:57:25.813 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:25.801+0000 7fb2d720d700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2d00a1d00 con 0x7fb2d0118ba0 2019-12-05T05:57:25.813 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:25.801+0000 7fb2d720d700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2d0119510 con 0x7fb2d01301f0 2019-12-05T05:57:25.813 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:25.801+0000 7fb2d720d700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2d00a2040 con 0x7fb2d01190e0 2019-12-05T05:57:25.814 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:25.801+0000 7fb2d57aa700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fb2d01301f0 0x7fb2d011b520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:44812/0 (socket says 172.21.15.42:44812) 2019-12-05T05:57:25.814 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:25.801+0000 7fb2d57aa700 1 -- 172.21.15.42:0/1811720214 learned_addr learned my addr 172.21.15.42:0/1811720214 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:57:25.814 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:25.801+0000 7fb2d5fab700 1 --2- 172.21.15.42:0/1811720214 >> v2:172.21.15.40:3300/0 conn(0x7fb2d01190e0 0x7fb2d00a17b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:57:25.814 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:25.801+0000 7fb2c5ffb700 1 -- 172.21.15.42:0/1811720214 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1291336946 0 0) 0x7fb2d0119510 con 0x7fb2d01301f0 2019-12-05T05:57:25.814 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:25.801+0000 7fb2c5ffb700 1 -- 172.21.15.42:0/1811720214 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2b00030b0 con 0x7fb2d01301f0 2019-12-05T05:57:25.814 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:25.801+0000 7fb2c5ffb700 1 -- 172.21.15.42:0/1811720214 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 2294843601 0 0) 0x7fb2b00030b0 con 0x7fb2d01301f0 2019-12-05T05:57:25.814 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:25.801+0000 7fb2c5ffb700 1 -- 172.21.15.42:0/1811720214 --> v1:172.21.15.42:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb2d0119510 con 0x7fb2d01301f0 2019-12-05T05:57:25.815 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:25.801+0000 7fb2c5ffb700 1 -- 172.21.15.42:0/1811720214 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7fb2cc001910 con 0x7fb2d01301f0 2019-12-05T05:57:25.815 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:25.801+0000 7fb2c5ffb700 1 -- 172.21.15.42:0/1811720214 >> v2:172.21.15.40:3300/0 conn(0x7fb2d01190e0 msgr2=0x7fb2d00a17b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:57:25.815 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:25.801+0000 7fb2c5ffb700 1 --2- 172.21.15.42:0/1811720214 >> v2:172.21.15.40:3300/0 conn(0x7fb2d01190e0 0x7fb2d00a17b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:57:25.815 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:25.801+0000 7fb2c5ffb700 1 -- 172.21.15.42:0/1811720214 >> v1:172.21.15.42:6789/0 conn(0x7fb2d01301f0 legacy=0x7fb2d011b520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:57:25.815 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:25.801+0000 7fb2c5ffb700 1 -- 172.21.15.42:0/1811720214 >> v1:172.21.15.40:6789/0 conn(0x7fb2d0118ba0 legacy=0x7fb2d0135630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:57:25.815 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:25.801+0000 7fb2c5ffb700 1 --2- 172.21.15.42:0/1811720214 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb2b00038a0 0x7fb2b0003d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:57:25.815 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:25.801+0000 7fb2c5ffb700 1 --2- 172.21.15.42:0/1811720214 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb2b0004420 0x7fb2b0004850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:57:25.815 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:25.801+0000 7fb2c5ffb700 1 --2- 172.21.15.42:0/1811720214 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb2b0004e90 0x7fb2b000b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:57:25.816 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:25.801+0000 7fb2c5ffb700 1 -- 172.21.15.42:0/1811720214 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2d00a2040 con 0x7fb2b0004420 2019-12-05T05:57:25.816 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:25.801+0000 7fb2c5ffb700 1 -- 172.21.15.42:0/1811720214 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2b000bd80 con 0x7fb2b0004e90 2019-12-05T05:57:25.816 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:25.801+0000 7fb2c5ffb700 1 -- 172.21.15.42:0/1811720214 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2b000bfc0 con 0x7fb2b00038a0 2019-12-05T05:57:25.816 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:25.801+0000 7fb2d5fab700 1 --2- 172.21.15.42:0/1811720214 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb2b0004e90 0x7fb2b000b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:57:25.823 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:25.801+0000 7fb2d4fa9700 1 --2- 172.21.15.42:0/1811720214 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb2b0004420 0x7fb2b0004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:57:25.823 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:25.801+0000 7fb2d57aa700 1 --2- 172.21.15.42:0/1811720214 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb2b00038a0 0x7fb2b0003d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:57:25.829 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:25.801+0000 7fb2d5fab700 1 -- 172.21.15.42:0/1811720214 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb2b00038a0 msgr2=0x7fb2b0003d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:57:25.829 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:25.801+0000 7fb2d5fab700 1 --2- 172.21.15.42:0/1811720214 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb2b00038a0 0x7fb2b0003d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:57:25.830 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:25.801+0000 7fb2d5fab700 1 -- 172.21.15.42:0/1811720214 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb2b0004420 msgr2=0x7fb2b0004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:57:25.830 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:25.801+0000 7fb2d5fab700 1 --2- 172.21.15.42:0/1811720214 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb2b0004420 0x7fb2b0004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:57:25.830 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:25.801+0000 7fb2d5fab700 1 -- 172.21.15.42:0/1811720214 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2d007dfe0 con 0x7fb2b0004e90 2019-12-05T05:57:25.830 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:25.801+0000 7fb2d57aa700 1 --2- 172.21.15.42:0/1811720214 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb2b00038a0 0x7fb2b0003d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).send_auth_request state changed! 2019-12-05T05:57:25.830 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:25.801+0000 7fb2d4fa9700 1 --2- 172.21.15.42:0/1811720214 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb2b0004420 0x7fb2b0004850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T05:57:25.830 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:25.801+0000 7fb2d720d700 1 -- 172.21.15.42:0/1811720214 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2d008eb50 con 0x7fb2b0004e90 2019-12-05T05:57:25.830 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:25.801+0000 7fb2d720d700 1 -- 172.21.15.42:0/1811720214 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2d008f020 con 0x7fb2b0004e90 2019-12-05T05:57:25.831 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:25.801+0000 7fb2d5fab700 1 --2- 172.21.15.42:0/1811720214 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb2b0004e90 0x7fb2b000b670 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fb2c8002f30 tx=0x7fb2c8011950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:25.831 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:25.801+0000 7fb2d5fab700 1 -- 172.21.15.42:0/1811720214 reap_dead start 2019-12-05T05:57:25.831 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:25.801+0000 7fb2d5fab700 1 --2- 172.21.15.42:0/1811720214 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb2b0004e90 0x7fb2b000b670 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fb2c8002f30 tx=0x7fb2c8011950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:25.831 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:25.801+0000 7fb2d5fab700 1 --2- 172.21.15.42:0/1811720214 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb2b0004e90 0x7fb2b000b670 secure :-1 s=READY pgs=1756 cs=0 l=1 rx=0x7fb2c8002f30 tx=0x7fb2c8011950).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:57:25.831 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:25.801+0000 7fb2d5fab700 1 --2- 172.21.15.42:0/1811720214 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb2b0004e90 0x7fb2b000b670 secure :-1 s=THROTTLE_DONE pgs=1756 cs=0 l=1 rx=0x7fb2c8002f30 tx=0x7fb2c8011950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:25.831 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:25.801+0000 7fb2d5fab700 1 --2- 172.21.15.42:0/1811720214 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb2b0004e90 0x7fb2b000b670 secure :-1 s=THROTTLE_DONE pgs=1756 cs=0 l=1 rx=0x7fb2c8002f30 tx=0x7fb2c8011950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:25.831 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:25.801+0000 7fb2c5ffb700 1 -- 172.21.15.42:0/1811720214 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb2c800f530 con 0x7fb2b0004e90 2019-12-05T05:57:25.832 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:25.801+0000 7fb2d5fab700 1 --2- 172.21.15.42:0/1811720214 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb2b0004e90 0x7fb2b000b670 secure :-1 s=THROTTLE_DONE pgs=1756 cs=0 l=1 rx=0x7fb2c8002f30 tx=0x7fb2c8011950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:25.832 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:25.801+0000 7fb2c5ffb700 1 -- 172.21.15.42:0/1811720214 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2c8008db0 con 0x7fb2b0004e90 2019-12-05T05:57:25.832 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:25.801+0000 7fb2c5ffb700 1 -- 172.21.15.42:0/1811720214 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb2c8005540 con 0x7fb2b0004e90 2019-12-05T05:57:25.832 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:25.801+0000 7fb2d5fab700 1 --2- 172.21.15.42:0/1811720214 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb2b0004e90 0x7fb2b000b670 secure :-1 s=THROTTLE_DONE pgs=1756 cs=0 l=1 rx=0x7fb2c8002f30 tx=0x7fb2c8011950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:25.832 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:25.805+0000 7fb2c5ffb700 1 -- 172.21.15.42:0/1811720214 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fb2c80130a0 con 0x7fb2b0004e90 2019-12-05T05:57:25.832 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:25.805+0000 7fb2d5fab700 1 --2- 172.21.15.42:0/1811720214 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb2b0004e90 0x7fb2b000b670 secure :-1 s=THROTTLE_DONE pgs=1756 cs=0 l=1 rx=0x7fb2c8002f30 tx=0x7fb2c8011950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:25.832 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:25.805+0000 7fb2c5ffb700 1 --2- 172.21.15.42:0/1811720214 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb2b003ead0 0x7fb2b0040fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:57:25.833 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:25.805+0000 7fb2c5ffb700 1 -- 172.21.15.42:0/1811720214 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 9213+0+0 (secure 0 0 0) 0x7fb2c804b6d0 con 0x7fb2b0004e90 2019-12-05T05:57:25.833 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:25.805+0000 7fb2d57aa700 1 --2- 172.21.15.42:0/1811720214 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb2b003ead0 0x7fb2b0040fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:57:25.833 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:25.805+0000 7fb2d57aa700 1 --2- 172.21.15.42:0/1811720214 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb2b003ead0 0x7fb2b0040fe0 crc :-1 s=READY pgs=751 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:57:25.834 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:25.813+0000 7fb2d720d700 1 -- 172.21.15.42:0/1811720214 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2a8003200 con 0x7fb2b0004e90 2019-12-05T05:57:25.834 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:25.817+0000 7fb2d5fab700 1 --2- 172.21.15.42:0/1811720214 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb2b0004e90 0x7fb2b000b670 secure :-1 s=THROTTLE_DONE pgs=1756 cs=0 l=1 rx=0x7fb2c8002f30 tx=0x7fb2c8011950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:25.834 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:25.817+0000 7fb2c5ffb700 1 -- 172.21.15.42:0/1811720214 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fb2c80329b0 con 0x7fb2b0004e90 2019-12-05T05:57:25.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:25.969+0000 7fb2d720d700 1 -- 172.21.15.42:0/1811720214 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb2a8003630 con 0x7fb2b0004e90 2019-12-05T05:57:25.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:25.969+0000 7fb2d5fab700 1 --2- 172.21.15.42:0/1811720214 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb2b0004e90 0x7fb2b000b670 secure :-1 s=THROTTLE_DONE pgs=1756 cs=0 l=1 rx=0x7fb2c8002f30 tx=0x7fb2c8011950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:25.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:25.969+0000 7fb2c5ffb700 1 -- 172.21.15.42:0/1811720214 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 214 v214) v1 ==== 95+0+1774 (secure 0 0 0) 0x7fb2c8033560 con 0x7fb2b0004e90 2019-12-05T05:57:25.975 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 214 2019-12-05T05:57:25.976 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:57:25.976 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":214,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":214,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:57:24.254998+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8836},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8836":{"gid":8836,"name":"a","rank":0,"incarnation":214,"state":"up:replay","state_seq":1,"addr":"172.21.15.42:6835/1368848584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1368848584},{"type":"v1","addr":"172.21.15.42:6835","nonce":1368848584}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:57:25.980 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:25.973+0000 7fb2d720d700 1 -- 172.21.15.42:0/1811720214 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb2b003ead0 msgr2=0x7fb2b0040fe0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:57:25.980 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:25.973+0000 7fb2d720d700 1 --2- 172.21.15.42:0/1811720214 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb2b003ead0 0x7fb2b0040fe0 crc :-1 s=READY pgs=751 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:57:25.980 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:25.973+0000 7fb2d720d700 1 -- 172.21.15.42:0/1811720214 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb2b0004e90 msgr2=0x7fb2b000b670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:57:25.981 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:25.973+0000 7fb2d720d700 1 --2- 172.21.15.42:0/1811720214 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb2b0004e90 0x7fb2b000b670 secure :-1 s=READY pgs=1756 cs=0 l=1 rx=0x7fb2c8002f30 tx=0x7fb2c8011950).stop 2019-12-05T05:57:25.981 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:25.973+0000 7fb2d720d700 1 -- 172.21.15.42:0/1811720214 shutdown_connections 2019-12-05T05:57:25.981 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:25.973+0000 7fb2d720d700 1 --2- 172.21.15.42:0/1811720214 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb2b003ead0 0x7fb2b0040fe0 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:57:25.981 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:25.973+0000 7fb2d720d700 1 --2- 172.21.15.42:0/1811720214 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb2b0004e90 0x7fb2b000b670 unknown :-1 s=CLOSED pgs=1756 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:57:25.981 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:25.973+0000 7fb2d720d700 1 -- 172.21.15.42:0/1811720214 shutdown_connections 2019-12-05T05:57:25.981 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:25.973+0000 7fb2d720d700 1 -- 172.21.15.42:0/1811720214 wait complete. 2019-12-05T05:57:25.981 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:25.973+0000 7fb2d720d700 1 -- 172.21.15.42:0/1811720214 >> 172.21.15.42:0/1811720214 conn(0x7fb2d00a82a0 msgr2=0x7fb2d00a9790 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:57:25.987 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b down, removed from mdsmap 2019-12-05T05:57:25.987 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2019-12-05T05:57:25.987 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:57:26.085 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:26.077+0000 7f806e8af700 1 Processor -- start 2019-12-05T05:57:26.085 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:26.077+0000 7f806e8af700 1 -- start start 2019-12-05T05:57:26.085 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:26.077+0000 7f806e8af700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f80680a7740 0x7f806808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:57:26.085 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:26.077+0000 7f806e8af700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f806807e720 0x7f806807eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:57:26.086 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:26.077+0000 7f806e8af700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80680a0710 con 0x7f806809f2e0 2019-12-05T05:57:26.086 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:26.077+0000 7f806e8af700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80680a0b00 con 0x7f80680a7740 2019-12-05T05:57:26.086 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:26.077+0000 7f806e8af700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f806808e5b0 con 0x7f806807e720 2019-12-05T05:57:26.086 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:26.077+0000 7f806ce4c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f80680a7740 0x7f806808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:57:26.086 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:26.077+0000 7f806ce4c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f80680a7740 0x7f806808df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:44788/0 (socket says 172.21.15.42:44788) 2019-12-05T05:57:26.086 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:26.077+0000 7f806ce4c700 1 -- 172.21.15.42:0/2422150880 learned_addr learned my addr 172.21.15.42:0/2422150880 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:57:26.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:26.081+0000 7f8067fff700 1 --2- 172.21.15.42:0/2422150880 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f806807e720 0x7f806807eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:57:26.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:26.081+0000 7f806ce4c700 1 -- 172.21.15.42:0/2422150880 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f806807e720 msgr2=0x7f806807eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:57:26.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:26.081+0000 7f806ce4c700 1 --2- 172.21.15.42:0/2422150880 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f806807e720 0x7f806807eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:57:26.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:26.081+0000 7f806ce4c700 1 -- 172.21.15.42:0/2422150880 >> v1:172.21.15.42:6789/0 conn(0x7f806809f2e0 legacy=0x7f806809f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:57:26.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:26.081+0000 7f806ce4c700 1 -- 172.21.15.42:0/2422150880 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f806807f0a0 con 0x7f80680a7740 2019-12-05T05:57:26.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:26.081+0000 7f806ce4c700 1 --2- 172.21.15.42:0/2422150880 >> v2:172.21.15.42:3300/0 conn(0x7f80680a7740 0x7f806808df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f8058008b10 tx=0x7f80580100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:26.088 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:26.081+0000 7f806ce4c700 1 --2- 172.21.15.42:0/2422150880 >> v2:172.21.15.42:3300/0 conn(0x7f80680a7740 0x7f806808df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f8058008b10 tx=0x7f80580100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:26.088 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:26.081+0000 7f806ce4c700 1 --2- 172.21.15.42:0/2422150880 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f80680a7740 0x7f806808df80 secure :-1 s=READY pgs=1758 cs=0 l=1 rx=0x7f8058008b10 tx=0x7f80580100b0).ready entity=mon.1 client_cookie=1186ea5f1aeeb017 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:57:26.088 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:26.081+0000 7f806ce4c700 1 --2- 172.21.15.42:0/2422150880 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f80680a7740 0x7f806808df80 secure :-1 s=THROTTLE_DONE pgs=1758 cs=0 l=1 rx=0x7f8058008b10 tx=0x7f80580100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:26.088 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:26.081+0000 7f806ce4c700 1 --2- 172.21.15.42:0/2422150880 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f80680a7740 0x7f806808df80 secure :-1 s=THROTTLE_DONE pgs=1758 cs=0 l=1 rx=0x7f8058008b10 tx=0x7f80580100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:26.088 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:26.081+0000 7f806ce4c700 1 --2- 172.21.15.42:0/2422150880 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f80680a7740 0x7f806808df80 secure :-1 s=THROTTLE_DONE pgs=1758 cs=0 l=1 rx=0x7f8058008b10 tx=0x7f80580100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:26.089 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:26.081+0000 7f80677fe700 1 -- 172.21.15.42:0/2422150880 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f805800a160 con 0x7f80680a7740 2019-12-05T05:57:26.089 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:26.081+0000 7f80677fe700 1 -- 172.21.15.42:0/2422150880 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8058011a30 con 0x7f80680a7740 2019-12-05T05:57:26.089 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:26.081+0000 7f80677fe700 1 -- 172.21.15.42:0/2422150880 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f805800a550 con 0x7f80680a7740 2019-12-05T05:57:26.089 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:26.081+0000 7f806e8af700 1 -- 172.21.15.42:0/2422150880 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f80680a7740 msgr2=0x7f806808df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:57:26.089 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:26.081+0000 7f806e8af700 1 --2- 172.21.15.42:0/2422150880 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f80680a7740 0x7f806808df80 secure :-1 s=READY pgs=1758 cs=0 l=1 rx=0x7f8058008b10 tx=0x7f80580100b0).stop 2019-12-05T05:57:26.089 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:26.081+0000 7f806e8af700 1 -- 172.21.15.42:0/2422150880 shutdown_connections 2019-12-05T05:57:26.089 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:26.081+0000 7f806e8af700 1 --2- 172.21.15.42:0/2422150880 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f80680a7740 0x7f806808df80 unknown :-1 s=CLOSED pgs=1758 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:57:26.090 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:26.081+0000 7f806e8af700 1 --2- 172.21.15.42:0/2422150880 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f806807e720 0x7f806807eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:57:26.090 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:26.081+0000 7f806e8af700 1 -- 172.21.15.42:0/2422150880 shutdown_connections 2019-12-05T05:57:26.090 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:26.081+0000 7f806e8af700 1 -- 172.21.15.42:0/2422150880 wait complete. 2019-12-05T05:57:26.090 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:26.081+0000 7f806e8af700 1 -- 172.21.15.42:0/2422150880 >> 172.21.15.42:0/2422150880 conn(0x7f80680aabe0 msgr2=0x7f80680a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:57:26.094 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:26.085+0000 7f806e8af700 1 Processor -- start 2019-12-05T05:57:26.094 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:26.085+0000 7f806e8af700 1 -- start start 2019-12-05T05:57:26.094 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:26.085+0000 7f806e8af700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f8068123b40 0x7f80680a3bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:57:26.094 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:26.085+0000 7f806e8af700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f806807b7a0 0x7f806807bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:57:26.095 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:26.085+0000 7f806e8af700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f806807c120 con 0x7f80680a5f10 2019-12-05T05:57:26.095 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:26.085+0000 7f806e8af700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80681432e0 con 0x7f806807b7a0 2019-12-05T05:57:26.095 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:26.085+0000 7f806e8af700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8068143580 con 0x7f8068123b40 2019-12-05T05:57:26.095 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:26.085+0000 7f8067fff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f8068123b40 0x7f80680a3bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:57:26.095 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:26.085+0000 7f8067fff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f8068123b40 0x7f80680a3bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:44796/0 (socket says 172.21.15.42:44796) 2019-12-05T05:57:26.095 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:26.085+0000 7f8067fff700 1 -- 172.21.15.42:0/1862985285 learned_addr learned my addr 172.21.15.42:0/1862985285 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:57:26.096 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:26.085+0000 7f806d64d700 1 --2- 172.21.15.42:0/1862985285 >> v2:172.21.15.40:3300/0 conn(0x7f806807b7a0 0x7f806807bbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:57:26.096 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:26.085+0000 7f8067fff700 1 -- 172.21.15.42:0/1862985285 >> v2:172.21.15.40:3300/0 conn(0x7f806807b7a0 msgr2=0x7f806807bbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:57:26.096 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:26.085+0000 7f8067fff700 1 --2- 172.21.15.42:0/1862985285 >> v2:172.21.15.40:3300/0 conn(0x7f806807b7a0 0x7f806807bbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:57:26.096 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:26.085+0000 7f8067fff700 1 -- 172.21.15.42:0/1862985285 >> v1:172.21.15.40:6789/0 conn(0x7f80680a5f10 legacy=0x7f8068123340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:57:26.096 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:26.085+0000 7f8067fff700 1 -- 172.21.15.42:0/1862985285 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8068142f30 con 0x7f8068123b40 2019-12-05T05:57:26.096 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:26.085+0000 7f8067fff700 1 --2- 172.21.15.42:0/1862985285 >> v2:172.21.15.42:3300/0 conn(0x7f8068123b40 0x7f80680a3bf0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f8050007980 tx=0x7f8050010b80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:26.096 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:26.085+0000 7f8067fff700 1 --2- 172.21.15.42:0/1862985285 >> v2:172.21.15.42:3300/0 conn(0x7f8068123b40 0x7f80680a3bf0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f8050007980 tx=0x7f8050010b80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:26.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:26.085+0000 7f8067fff700 1 --2- 172.21.15.42:0/1862985285 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8068123b40 0x7f80680a3bf0 secure :-1 s=READY pgs=1759 cs=0 l=1 rx=0x7f8050007980 tx=0x7f8050010b80).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:57:26.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:26.085+0000 7f8067fff700 1 --2- 172.21.15.42:0/1862985285 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8068123b40 0x7f80680a3bf0 secure :-1 s=THROTTLE_DONE pgs=1759 cs=0 l=1 rx=0x7f8050007980 tx=0x7f8050010b80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:26.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:26.085+0000 7f8067fff700 1 --2- 172.21.15.42:0/1862985285 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8068123b40 0x7f80680a3bf0 secure :-1 s=THROTTLE_DONE pgs=1759 cs=0 l=1 rx=0x7f8050007980 tx=0x7f8050010b80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:26.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:26.085+0000 7f80657fa700 1 -- 172.21.15.42:0/1862985285 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f80500059c0 con 0x7f8068123b40 2019-12-05T05:57:26.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:26.085+0000 7f8067fff700 1 --2- 172.21.15.42:0/1862985285 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8068123b40 0x7f80680a3bf0 secure :-1 s=THROTTLE_DONE pgs=1759 cs=0 l=1 rx=0x7f8050007980 tx=0x7f8050010b80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:26.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:26.085+0000 7f80657fa700 1 -- 172.21.15.42:0/1862985285 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80500055e0 con 0x7f8068123b40 2019-12-05T05:57:26.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:26.085+0000 7f80657fa700 1 -- 172.21.15.42:0/1862985285 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f805000e4f0 con 0x7f8068123b40 2019-12-05T05:57:26.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:26.085+0000 7f806e8af700 1 -- 172.21.15.42:0/1862985285 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80680a6bd0 con 0x7f8068123b40 2019-12-05T05:57:26.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:26.085+0000 7f806e8af700 1 -- 172.21.15.42:0/1862985285 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80680a9640 con 0x7f8068123b40 2019-12-05T05:57:26.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:26.089+0000 7f8067fff700 1 --2- 172.21.15.42:0/1862985285 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8068123b40 0x7f80680a3bf0 secure :-1 s=THROTTLE_DONE pgs=1759 cs=0 l=1 rx=0x7f8050007980 tx=0x7f8050010b80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:26.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:26.089+0000 7f80657fa700 1 -- 172.21.15.42:0/1862985285 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f8050007180 con 0x7f8068123b40 2019-12-05T05:57:26.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:26.089+0000 7f8067fff700 1 --2- 172.21.15.42:0/1862985285 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8068123b40 0x7f80680a3bf0 secure :-1 s=THROTTLE_DONE pgs=1759 cs=0 l=1 rx=0x7f8050007980 tx=0x7f8050010b80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:26.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:26.089+0000 7f80657fa700 1 --2- 172.21.15.42:0/1862985285 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f804c03ad60 0x7f804c03d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:57:26.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:26.089+0000 7f80657fa700 1 -- 172.21.15.42:0/1862985285 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 9213+0+0 (secure 0 0 0) 0x7f805004b150 con 0x7f8068123b40 2019-12-05T05:57:26.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:26.089+0000 7f806ce4c700 1 --2- 172.21.15.42:0/1862985285 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f804c03ad60 0x7f804c03d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:57:26.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:26.089+0000 7f806ce4c700 1 --2- 172.21.15.42:0/1862985285 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f804c03ad60 0x7f804c03d270 crc :-1 s=READY pgs=752 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:57:26.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:26.093+0000 7f806e8af700 1 -- 172.21.15.42:0/1862985285 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f806805d110 con 0x7f8068123b40 2019-12-05T05:57:26.106 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:26.101+0000 7f8067fff700 1 --2- 172.21.15.42:0/1862985285 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8068123b40 0x7f80680a3bf0 secure :-1 s=THROTTLE_DONE pgs=1759 cs=0 l=1 rx=0x7f8050007980 tx=0x7f8050010b80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:26.107 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:26.101+0000 7f80657fa700 1 -- 172.21.15.42:0/1862985285 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f80500321d0 con 0x7f8068123b40 2019-12-05T05:57:26.264 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:26.261+0000 7f806e8af700 1 -- 172.21.15.42:0/1862985285 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f80680a9cb0 con 0x7f8068123b40 2019-12-05T05:57:26.265 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:26.261+0000 7f8067fff700 1 --2- 172.21.15.42:0/1862985285 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8068123b40 0x7f80680a3bf0 secure :-1 s=THROTTLE_DONE pgs=1759 cs=0 l=1 rx=0x7f8050007980 tx=0x7f8050010b80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:26.265 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:26.261+0000 7f80657fa700 1 -- 172.21.15.42:0/1862985285 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 214 v214) v1 ==== 95+0+1774 (secure 0 0 0) 0x7f8050032430 con 0x7f8068123b40 2019-12-05T05:57:26.265 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 214 2019-12-05T05:57:26.265 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:57:26.266 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":214,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":214,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:57:24.254998+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8836},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8836":{"gid":8836,"name":"a","rank":0,"incarnation":214,"state":"up:replay","state_seq":1,"addr":"172.21.15.42:6835/1368848584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1368848584},{"type":"v1","addr":"172.21.15.42:6835","nonce":1368848584}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:57:26.267 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:26.265+0000 7f806e8af700 1 -- 172.21.15.42:0/1862985285 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f804c03ad60 msgr2=0x7f804c03d270 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:57:26.267 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:26.265+0000 7f806e8af700 1 --2- 172.21.15.42:0/1862985285 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f804c03ad60 0x7f804c03d270 crc :-1 s=READY pgs=752 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:57:26.267 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:26.265+0000 7f806e8af700 1 -- 172.21.15.42:0/1862985285 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8068123b40 msgr2=0x7f80680a3bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:57:26.267 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:26.265+0000 7f806e8af700 1 --2- 172.21.15.42:0/1862985285 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8068123b40 0x7f80680a3bf0 secure :-1 s=READY pgs=1759 cs=0 l=1 rx=0x7f8050007980 tx=0x7f8050010b80).stop 2019-12-05T05:57:26.268 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:26.265+0000 7f806e8af700 1 -- 172.21.15.42:0/1862985285 shutdown_connections 2019-12-05T05:57:26.268 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:26.265+0000 7f806e8af700 1 --2- 172.21.15.42:0/1862985285 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f804c03ad60 0x7f804c03d270 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:57:26.269 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:26.265+0000 7f806e8af700 1 --2- 172.21.15.42:0/1862985285 >> v2:172.21.15.40:3300/0 conn(0x7f806807b7a0 0x7f806807bbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:57:26.269 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:26.265+0000 7f806e8af700 1 --2- 172.21.15.42:0/1862985285 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8068123b40 0x7f80680a3bf0 unknown :-1 s=CLOSED pgs=1759 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:57:26.269 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:26.265+0000 7f806e8af700 1 -- 172.21.15.42:0/1862985285 shutdown_connections 2019-12-05T05:57:26.269 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:26.265+0000 7f806e8af700 1 -- 172.21.15.42:0/1862985285 wait complete. 2019-12-05T05:57:26.269 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:26.265+0000 7f806e8af700 1 -- 172.21.15.42:0/1862985285 >> 172.21.15.42:0/1862985285 conn(0x7f806809f350 msgr2=0x7f806807ca30 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:57:26.277 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 214, "feature_flags": { "enable_multiple": false, "ever_enabled_multiple": false }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2019-12-05T05:03:35.604738+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 214, "ever_allowed_features": 2, "explicitly_allowed_features": 2, "failed": [], "flags": 18, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_8836": { "addr": "172.21.15.42:6835/1368848584", "addrs": { "addrvec": [ { "addr": "172.21.15.42:6834", "nonce": 1368848584, "type": "v2" }, { "addr": "172.21.15.42:6835", "nonce": 1368848584, "type": "v1" } ] }, "export_targets": [], "features": 4540138292836696063, "flags": 0, "gid": 8836, "incarnation": 214, "name": "a", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 86, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 2, "min_compat_client": "0 (unknown)", "modified": "2019-12-05T05:57:24.254998+0000", "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 8836 } } } ], "standbys": [] } 2019-12-05T05:57:28.279 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:57:28.378 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:28.369+0000 7f48db67e700 1 Processor -- start 2019-12-05T05:57:28.378 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:28.369+0000 7f48db67e700 1 -- start start 2019-12-05T05:57:28.379 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:28.369+0000 7f48db67e700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f48d408df70 0x7f48d409f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:57:28.379 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:28.369+0000 7f48db67e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f48d409f820 0x7f48d414efd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:57:28.379 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:28.369+0000 7f48db67e700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48d409fc50 con 0x7f48d40a7730 2019-12-05T05:57:28.379 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:28.369+0000 7f48db67e700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48d40a0af0 con 0x7f48d409f820 2019-12-05T05:57:28.379 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:28.369+0000 7f48db67e700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48d408e5a0 con 0x7f48d408df70 2019-12-05T05:57:28.379 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:28.369+0000 7f48da41c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f48d409f820 0x7f48d414efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:57:28.379 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:28.369+0000 7f48da41c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f48d409f820 0x7f48d414efd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:44808/0 (socket says 172.21.15.42:44808) 2019-12-05T05:57:28.380 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:28.369+0000 7f48da41c700 1 -- 172.21.15.42:0/879920842 learned_addr learned my addr 172.21.15.42:0/879920842 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:57:28.380 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:28.369+0000 7f48d941a700 1 --2- 172.21.15.42:0/879920842 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f48d408df70 0x7f48d409f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:57:28.380 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:28.373+0000 7f48da41c700 1 -- 172.21.15.42:0/879920842 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f48d408df70 msgr2=0x7f48d409f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:57:28.380 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:28.373+0000 7f48da41c700 1 --2- 172.21.15.42:0/879920842 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f48d408df70 0x7f48d409f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:57:28.380 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:28.373+0000 7f48da41c700 1 -- 172.21.15.42:0/879920842 >> v1:172.21.15.40:6789/0 conn(0x7f48d40a7730 legacy=0x7f48d407e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:57:28.380 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:28.373+0000 7f48d8c19700 1 -- 172.21.15.42:0/879920842 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1558814807 0 0) 0x7f48d409fc50 con 0x7f48d40a7730 2019-12-05T05:57:28.380 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:28.373+0000 7f48da41c700 1 -- 172.21.15.42:0/879920842 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48d40a0700 con 0x7f48d409f820 2019-12-05T05:57:28.381 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:28.373+0000 7f48d941a700 1 --2- 172.21.15.42:0/879920842 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f48d408df70 0x7f48d409f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:57:28.381 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:28.373+0000 7f48da41c700 1 --2- 172.21.15.42:0/879920842 >> v2:172.21.15.42:3300/0 conn(0x7f48d409f820 0x7f48d414efd0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f48cc008b10 tx=0x7f48cc0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:28.381 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:28.373+0000 7f48da41c700 1 --2- 172.21.15.42:0/879920842 >> v2:172.21.15.42:3300/0 conn(0x7f48d409f820 0x7f48d414efd0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f48cc008b10 tx=0x7f48cc0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:28.381 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:28.373+0000 7f48da41c700 1 --2- 172.21.15.42:0/879920842 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f48d409f820 0x7f48d414efd0 secure :-1 s=READY pgs=1760 cs=0 l=1 rx=0x7f48cc008b10 tx=0x7f48cc0100b0).ready entity=mon.1 client_cookie=3f9ddd942c05cefb server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:57:28.381 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:28.373+0000 7f48da41c700 1 --2- 172.21.15.42:0/879920842 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f48d409f820 0x7f48d414efd0 secure :-1 s=THROTTLE_DONE pgs=1760 cs=0 l=1 rx=0x7f48cc008b10 tx=0x7f48cc0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:28.381 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:28.373+0000 7f48d8c19700 1 -- 172.21.15.42:0/879920842 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f48cc010da0 con 0x7f48d409f820 2019-12-05T05:57:28.381 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:28.373+0000 7f48da41c700 1 --2- 172.21.15.42:0/879920842 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f48d409f820 0x7f48d414efd0 secure :-1 s=THROTTLE_DONE pgs=1760 cs=0 l=1 rx=0x7f48cc008b10 tx=0x7f48cc0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:28.382 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:28.373+0000 7f48da41c700 1 --2- 172.21.15.42:0/879920842 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f48d409f820 0x7f48d414efd0 secure :-1 s=THROTTLE_DONE pgs=1760 cs=0 l=1 rx=0x7f48cc008b10 tx=0x7f48cc0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:28.382 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:28.373+0000 7f48d8c19700 1 -- 172.21.15.42:0/879920842 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48cc0109c0 con 0x7f48d409f820 2019-12-05T05:57:28.382 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:28.373+0000 7f48d8c19700 1 -- 172.21.15.42:0/879920842 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f48cc0114a0 con 0x7f48d409f820 2019-12-05T05:57:28.382 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:28.373+0000 7f48db67e700 1 -- 172.21.15.42:0/879920842 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f48d409f820 msgr2=0x7f48d414efd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:57:28.382 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:28.373+0000 7f48db67e700 1 --2- 172.21.15.42:0/879920842 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f48d409f820 0x7f48d414efd0 secure :-1 s=READY pgs=1760 cs=0 l=1 rx=0x7f48cc008b10 tx=0x7f48cc0100b0).stop 2019-12-05T05:57:28.382 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:28.373+0000 7f48db67e700 1 -- 172.21.15.42:0/879920842 shutdown_connections 2019-12-05T05:57:28.382 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:28.373+0000 7f48db67e700 1 --2- 172.21.15.42:0/879920842 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f48d409f820 0x7f48d414efd0 unknown :-1 s=CLOSED pgs=1760 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:57:28.383 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:28.373+0000 7f48db67e700 1 --2- 172.21.15.42:0/879920842 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f48d408df70 0x7f48d409f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:57:28.383 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:28.373+0000 7f48db67e700 1 -- 172.21.15.42:0/879920842 shutdown_connections 2019-12-05T05:57:28.383 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:28.373+0000 7f48db67e700 1 -- 172.21.15.42:0/879920842 wait complete. 2019-12-05T05:57:28.383 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:28.373+0000 7f48db67e700 1 -- 172.21.15.42:0/879920842 >> 172.21.15.42:0/879920842 conn(0x7f48d40aabd0 msgr2=0x7f48d40a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:57:28.386 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:28.377+0000 7f48db67e700 1 Processor -- start 2019-12-05T05:57:28.386 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:28.377+0000 7f48db67e700 1 -- start start 2019-12-05T05:57:28.386 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:28.377+0000 7f48db67e700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f48d407df40 0x7f48d4122af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:57:28.386 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:28.377+0000 7f48db67e700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f48d41174d0 0x7f48d4117900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:57:28.387 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:28.377+0000 7f48db67e700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48d4130310 con 0x7f48d41191a0 2019-12-05T05:57:28.387 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:28.377+0000 7f48db67e700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48d4117e50 con 0x7f48d407df40 2019-12-05T05:57:28.387 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:28.377+0000 7f48db67e700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48d4121740 con 0x7f48d41174d0 2019-12-05T05:57:28.387 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:28.377+0000 7f48d941a700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f48d41174d0 0x7f48d4117900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:57:28.387 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:28.377+0000 7f48d9c1b700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f48d407df40 0x7f48d4122af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:57:28.387 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:28.377+0000 7f48d941a700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f48d41174d0 0x7f48d4117900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:45280/0 (socket says 172.21.15.42:45280) 2019-12-05T05:57:28.387 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:28.377+0000 7f48d941a700 1 -- 172.21.15.42:0/2177282956 learned_addr learned my addr 172.21.15.42:0/2177282956 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:57:28.388 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:28.377+0000 7f48d9c1b700 1 -- 172.21.15.42:0/2177282956 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f48d41174d0 msgr2=0x7f48d4117900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:57:28.388 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:28.377+0000 7f48d9c1b700 1 --2- 172.21.15.42:0/2177282956 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f48d41174d0 0x7f48d4117900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:57:28.388 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:28.377+0000 7f48d9c1b700 1 -- 172.21.15.42:0/2177282956 >> v1:172.21.15.40:6789/0 conn(0x7f48d41191a0 legacy=0x7f48d41195d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:57:28.388 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:28.377+0000 7f48d9c1b700 1 -- 172.21.15.42:0/2177282956 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48d41220d0 con 0x7f48d407df40 2019-12-05T05:57:28.388 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:28.377+0000 7f48d941a700 1 --2- 172.21.15.42:0/2177282956 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f48d41174d0 0x7f48d4117900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:57:28.388 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:28.377+0000 7f48d9c1b700 1 --2- 172.21.15.42:0/2177282956 >> v2:172.21.15.40:3300/0 conn(0x7f48d407df40 0x7f48d4122af0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f48d4121900 tx=0x7f48c8010870).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:28.388 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:28.377+0000 7f48d9c1b700 1 --2- 172.21.15.42:0/2177282956 >> v2:172.21.15.40:3300/0 conn(0x7f48d407df40 0x7f48d4122af0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f48d4121900 tx=0x7f48c8010870).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:28.389 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:28.377+0000 7f48d9c1b700 1 --2- 172.21.15.42:0/2177282956 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f48d407df40 0x7f48d4122af0 secure :-1 s=READY pgs=1336 cs=0 l=1 rx=0x7f48d4121900 tx=0x7f48c8010870).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:57:28.389 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:28.381+0000 7f48d9c1b700 1 --2- 172.21.15.42:0/2177282956 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f48d407df40 0x7f48d4122af0 secure :-1 s=THROTTLE_DONE pgs=1336 cs=0 l=1 rx=0x7f48d4121900 tx=0x7f48c8010870).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:28.389 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:28.381+0000 7f48d9c1b700 1 --2- 172.21.15.42:0/2177282956 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f48d407df40 0x7f48d4122af0 secure :-1 s=THROTTLE_DONE pgs=1336 cs=0 l=1 rx=0x7f48d4121900 tx=0x7f48c8010870).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:28.389 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:28.381+0000 7f48d9c1b700 1 --2- 172.21.15.42:0/2177282956 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f48d407df40 0x7f48d4122af0 secure :-1 s=THROTTLE_DONE pgs=1336 cs=0 l=1 rx=0x7f48d4121900 tx=0x7f48c8010870).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:28.389 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:28.381+0000 7f48c67fc700 1 -- 172.21.15.42:0/2177282956 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f48c8006a40 con 0x7f48d407df40 2019-12-05T05:57:28.389 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:28.381+0000 7f48c67fc700 1 -- 172.21.15.42:0/2177282956 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48c8006660 con 0x7f48d407df40 2019-12-05T05:57:28.390 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:28.381+0000 7f48c67fc700 1 -- 172.21.15.42:0/2177282956 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f48c8011070 con 0x7f48d407df40 2019-12-05T05:57:28.390 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:28.381+0000 7f48db67e700 1 -- 172.21.15.42:0/2177282956 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48d41247d0 con 0x7f48d407df40 2019-12-05T05:57:28.390 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:28.381+0000 7f48db67e700 1 -- 172.21.15.42:0/2177282956 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48d4121310 con 0x7f48d407df40 2019-12-05T05:57:28.390 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:28.381+0000 7f48d9c1b700 1 --2- 172.21.15.42:0/2177282956 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f48d407df40 0x7f48d4122af0 secure :-1 s=THROTTLE_DONE pgs=1336 cs=0 l=1 rx=0x7f48d4121900 tx=0x7f48c8010870).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:28.390 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:28.381+0000 7f48c67fc700 1 -- 172.21.15.42:0/2177282956 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f48c8011340 con 0x7f48d407df40 2019-12-05T05:57:28.390 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:28.381+0000 7f48d9c1b700 1 --2- 172.21.15.42:0/2177282956 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f48d407df40 0x7f48d4122af0 secure :-1 s=THROTTLE_DONE pgs=1336 cs=0 l=1 rx=0x7f48d4121900 tx=0x7f48c8010870).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:28.391 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:28.381+0000 7f48c67fc700 1 --2- 172.21.15.42:0/2177282956 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f48bc03ace0 0x7f48bc03d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:57:28.391 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:28.381+0000 7f48c67fc700 1 -- 172.21.15.42:0/2177282956 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 9213+0+0 (secure 0 0 0) 0x7f48c804b120 con 0x7f48d407df40 2019-12-05T05:57:28.391 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:28.381+0000 7f48d941a700 1 --2- 172.21.15.42:0/2177282956 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f48bc03ace0 0x7f48bc03d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:57:28.392 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:28.381+0000 7f48d941a700 1 --2- 172.21.15.42:0/2177282956 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f48bc03ace0 0x7f48bc03d1f0 crc :-1 s=READY pgs=753 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:57:28.392 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:28.385+0000 7f48db67e700 1 -- 172.21.15.42:0/2177282956 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48d405d100 con 0x7f48d407df40 2019-12-05T05:57:28.399 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:28.397+0000 7f48d9c1b700 1 --2- 172.21.15.42:0/2177282956 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f48d407df40 0x7f48d4122af0 secure :-1 s=THROTTLE_DONE pgs=1336 cs=0 l=1 rx=0x7f48d4121900 tx=0x7f48c8010870).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:28.399 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:28.397+0000 7f48c67fc700 1 -- 172.21.15.42:0/2177282956 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f48c8032140 con 0x7f48d407df40 2019-12-05T05:57:28.567 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:28.561+0000 7f48db67e700 1 -- 172.21.15.42:0/2177282956 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f48d405b8b0 con 0x7f48d407df40 2019-12-05T05:57:28.568 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:28.565+0000 7f48d9c1b700 1 --2- 172.21.15.42:0/2177282956 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f48d407df40 0x7f48d4122af0 secure :-1 s=THROTTLE_DONE pgs=1336 cs=0 l=1 rx=0x7f48d4121900 tx=0x7f48c8010870).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:28.568 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:28.565+0000 7f48c67fc700 1 -- 172.21.15.42:0/2177282956 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 216 v216) v1 ==== 95+0+1775 (secure 0 0 0) 0x7f48c80323a0 con 0x7f48d407df40 2019-12-05T05:57:28.570 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:57:28.570 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":216,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":216,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:57:28.279088+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8836},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8836":{"gid":8836,"name":"a","rank":0,"incarnation":214,"state":"up:rejoin","state_seq":28,"addr":"172.21.15.42:6835/1368848584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1368848584},{"type":"v1","addr":"172.21.15.42:6835","nonce":1368848584}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:57:28.570 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 216 2019-12-05T05:57:28.573 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:28.569+0000 7f48db67e700 1 -- 172.21.15.42:0/2177282956 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f48bc03ace0 msgr2=0x7f48bc03d1f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:57:28.573 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:28.569+0000 7f48db67e700 1 --2- 172.21.15.42:0/2177282956 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f48bc03ace0 0x7f48bc03d1f0 crc :-1 s=READY pgs=753 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:57:28.573 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:28.569+0000 7f48db67e700 1 -- 172.21.15.42:0/2177282956 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f48d407df40 msgr2=0x7f48d4122af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:57:28.573 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:28.569+0000 7f48db67e700 1 --2- 172.21.15.42:0/2177282956 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f48d407df40 0x7f48d4122af0 secure :-1 s=READY pgs=1336 cs=0 l=1 rx=0x7f48d4121900 tx=0x7f48c8010870).stop 2019-12-05T05:57:28.573 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:28.569+0000 7f48db67e700 1 -- 172.21.15.42:0/2177282956 shutdown_connections 2019-12-05T05:57:28.573 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:28.569+0000 7f48db67e700 1 --2- 172.21.15.42:0/2177282956 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f48bc03ace0 0x7f48bc03d1f0 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:57:28.574 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:28.569+0000 7f48db67e700 1 --2- 172.21.15.42:0/2177282956 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f48d407df40 0x7f48d4122af0 unknown :-1 s=CLOSED pgs=1336 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:57:28.574 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:28.569+0000 7f48db67e700 1 --2- 172.21.15.42:0/2177282956 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f48d41174d0 0x7f48d4117900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:57:28.574 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:28.569+0000 7f48db67e700 1 -- 172.21.15.42:0/2177282956 shutdown_connections 2019-12-05T05:57:28.574 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:28.569+0000 7f48db67e700 1 -- 172.21.15.42:0/2177282956 wait complete. 2019-12-05T05:57:28.574 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:28.569+0000 7f48db67e700 1 -- 172.21.15.42:0/2177282956 >> 172.21.15.42:0/2177282956 conn(0x7f48d4067070 msgr2=0x7f48d40a6b30 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:57:28.582 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2019-12-05T05:57:30.583 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:57:30.718 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:30.685+0000 7ff831e59700 1 Processor -- start 2019-12-05T05:57:30.718 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:30.685+0000 7ff831e59700 1 -- start start 2019-12-05T05:57:30.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:30.689+0000 7ff831e59700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff82c0a7730 0x7ff82c08df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:57:30.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:30.689+0000 7ff831e59700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff82c09fad0 0x7ff82c07eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:57:30.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:30.689+0000 7ff831e59700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff82c0a0700 con 0x7ff82c07e710 2019-12-05T05:57:30.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:30.689+0000 7ff831e59700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff82c0a0af0 con 0x7ff82c09fad0 2019-12-05T05:57:30.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:30.689+0000 7ff831e59700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff82c08e5a0 con 0x7ff82c0a7730 2019-12-05T05:57:30.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:30.689+0000 7ff82bfff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff82c0a7730 0x7ff82c08df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:57:30.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:30.689+0000 7ff82bfff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff82c0a7730 0x7ff82c08df70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:44832/0 (socket says 172.21.15.42:44832) 2019-12-05T05:57:30.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:30.689+0000 7ff82bfff700 1 -- 172.21.15.42:0/2673561239 learned_addr learned my addr 172.21.15.42:0/2673561239 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:57:30.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:30.689+0000 7ff830bf7700 1 --2- 172.21.15.42:0/2673561239 >> v2:172.21.15.40:3300/0 conn(0x7ff82c09fad0 0x7ff82c07eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:57:30.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:30.689+0000 7ff82bfff700 1 -- 172.21.15.42:0/2673561239 >> v2:172.21.15.40:3300/0 conn(0x7ff82c09fad0 msgr2=0x7ff82c07eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:57:30.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:30.689+0000 7ff82bfff700 1 --2- 172.21.15.42:0/2673561239 >> v2:172.21.15.40:3300/0 conn(0x7ff82c09fad0 0x7ff82c07eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:57:30.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:30.689+0000 7ff82bfff700 1 -- 172.21.15.42:0/2673561239 >> v1:172.21.15.40:6789/0 conn(0x7ff82c07e710 legacy=0x7ff82c09f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:57:30.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:30.689+0000 7ff82bfff700 1 -- 172.21.15.42:0/2673561239 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff82c07f090 con 0x7ff82c0a7730 2019-12-05T05:57:30.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:30.689+0000 7ff830bf7700 1 --2- 172.21.15.42:0/2673561239 >> v2:172.21.15.40:3300/0 conn(0x7ff82c09fad0 0x7ff82c07eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:57:30.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:30.689+0000 7ff82bfff700 1 --2- 172.21.15.42:0/2673561239 >> v2:172.21.15.42:3300/0 conn(0x7ff82c0a7730 0x7ff82c08df70 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7ff81c002f30 tx=0x7ff81c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:30.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:30.689+0000 7ff82bfff700 1 --2- 172.21.15.42:0/2673561239 >> v2:172.21.15.42:3300/0 conn(0x7ff82c0a7730 0x7ff82c08df70 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7ff81c002f30 tx=0x7ff81c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:30.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:30.689+0000 7ff82bfff700 1 --2- 172.21.15.42:0/2673561239 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff82c0a7730 0x7ff82c08df70 secure :-1 s=READY pgs=1761 cs=0 l=1 rx=0x7ff81c002f30 tx=0x7ff81c00fd30).ready entity=mon.1 client_cookie=8b399daaba582d23 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:57:30.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:30.689+0000 7ff82bfff700 1 --2- 172.21.15.42:0/2673561239 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff82c0a7730 0x7ff82c08df70 secure :-1 s=THROTTLE_DONE pgs=1761 cs=0 l=1 rx=0x7ff81c002f30 tx=0x7ff81c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:30.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:30.689+0000 7ff82bfff700 1 --2- 172.21.15.42:0/2673561239 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff82c0a7730 0x7ff82c08df70 secure :-1 s=THROTTLE_DONE pgs=1761 cs=0 l=1 rx=0x7ff81c002f30 tx=0x7ff81c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:30.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:30.689+0000 7ff82affd700 1 -- 172.21.15.42:0/2673561239 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff81c008c90 con 0x7ff82c0a7730 2019-12-05T05:57:30.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:30.689+0000 7ff82bfff700 1 --2- 172.21.15.42:0/2673561239 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff82c0a7730 0x7ff82c08df70 secure :-1 s=THROTTLE_DONE pgs=1761 cs=0 l=1 rx=0x7ff81c002f30 tx=0x7ff81c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:30.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:30.689+0000 7ff82affd700 1 -- 172.21.15.42:0/2673561239 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff81c0088b0 con 0x7ff82c0a7730 2019-12-05T05:57:30.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:30.689+0000 7ff82affd700 1 -- 172.21.15.42:0/2673561239 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff81c011ad0 con 0x7ff82c0a7730 2019-12-05T05:57:30.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:30.689+0000 7ff831e59700 1 -- 172.21.15.42:0/2673561239 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff82c0a7730 msgr2=0x7ff82c08df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:57:30.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:30.689+0000 7ff831e59700 1 --2- 172.21.15.42:0/2673561239 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff82c0a7730 0x7ff82c08df70 secure :-1 s=READY pgs=1761 cs=0 l=1 rx=0x7ff81c002f30 tx=0x7ff81c00fd30).stop 2019-12-05T05:57:30.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:30.689+0000 7ff831e59700 1 -- 172.21.15.42:0/2673561239 shutdown_connections 2019-12-05T05:57:30.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:30.689+0000 7ff831e59700 1 --2- 172.21.15.42:0/2673561239 >> v2:172.21.15.40:3300/0 conn(0x7ff82c09fad0 0x7ff82c07eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:57:30.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:30.689+0000 7ff831e59700 1 --2- 172.21.15.42:0/2673561239 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff82c0a7730 0x7ff82c08df70 unknown :-1 s=CLOSED pgs=1761 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:57:30.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:30.689+0000 7ff831e59700 1 -- 172.21.15.42:0/2673561239 shutdown_connections 2019-12-05T05:57:30.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:30.689+0000 7ff831e59700 1 -- 172.21.15.42:0/2673561239 wait complete. 2019-12-05T05:57:30.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:30.689+0000 7ff831e59700 1 -- 172.21.15.42:0/2673561239 >> 172.21.15.42:0/2673561239 conn(0x7ff82c0aabd0 msgr2=0x7ff82c0a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:57:30.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:30.693+0000 7ff831e59700 1 Processor -- start 2019-12-05T05:57:30.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:30.693+0000 7ff831e59700 1 -- start start 2019-12-05T05:57:30.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:30.693+0000 7ff831e59700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff82c08ee80 0x7ff82c065800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:57:30.724 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:30.693+0000 7ff831e59700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff82c063180 con 0x7ff82c063e00 2019-12-05T05:57:30.724 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:30.693+0000 7ff831e59700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff82c0a3cd0 con 0x7ff82c0a38a0 2019-12-05T05:57:30.724 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:30.693+0000 7ff831e59700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff82c0633b0 con 0x7ff82c08ee80 2019-12-05T05:57:30.724 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:30.693+0000 7ff82bfff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff82c08ee80 0x7ff82c065800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:57:30.724 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:30.693+0000 7ff830bf7700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7ff82c0a38a0 0x7ff82c1316c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:44880/0 (socket says 172.21.15.42:44880) 2019-12-05T05:57:30.724 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:30.693+0000 7ff830bf7700 1 -- 172.21.15.42:0/1866410899 learned_addr learned my addr 172.21.15.42:0/1866410899 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:57:30.724 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:30.693+0000 7ff828ff9700 1 -- 172.21.15.42:0/1866410899 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2794295990 0 0) 0x7ff82c0a3cd0 con 0x7ff82c0a38a0 2019-12-05T05:57:30.725 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:30.693+0000 7ff828ff9700 1 -- 172.21.15.42:0/1866410899 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8100030b0 con 0x7ff82c0a38a0 2019-12-05T05:57:30.725 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:30.693+0000 7ff828ff9700 1 -- 172.21.15.42:0/1866410899 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 200035835 0 0) 0x7ff82c063180 con 0x7ff82c063e00 2019-12-05T05:57:30.725 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:30.693+0000 7ff828ff9700 1 -- 172.21.15.42:0/1866410899 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff82c0a3cd0 con 0x7ff82c063e00 2019-12-05T05:57:30.725 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:30.693+0000 7ff82bfff700 1 -- 172.21.15.42:0/1866410899 >> v1:172.21.15.42:6789/0 conn(0x7ff82c0a38a0 legacy=0x7ff82c1316c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:57:30.725 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:30.693+0000 7ff82bfff700 1 -- 172.21.15.42:0/1866410899 >> v1:172.21.15.40:6789/0 conn(0x7ff82c063e00 legacy=0x7ff82c126cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:57:30.725 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:30.693+0000 7ff82bfff700 1 -- 172.21.15.42:0/1866410899 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff82c0650e0 con 0x7ff82c08ee80 2019-12-05T05:57:30.726 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:30.693+0000 7ff82bfff700 1 --2- 172.21.15.42:0/1866410899 >> v2:172.21.15.42:3300/0 conn(0x7ff82c08ee80 0x7ff82c065800 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7ff81c002f30 tx=0x7ff81c008ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:30.726 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:30.693+0000 7ff82bfff700 1 --2- 172.21.15.42:0/1866410899 >> v2:172.21.15.42:3300/0 conn(0x7ff82c08ee80 0x7ff82c065800 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7ff81c002f30 tx=0x7ff81c008ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:30.726 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:30.693+0000 7ff82bfff700 1 --2- 172.21.15.42:0/1866410899 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff82c08ee80 0x7ff82c065800 secure :-1 s=READY pgs=1763 cs=0 l=1 rx=0x7ff81c002f30 tx=0x7ff81c008ae0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:57:30.726 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:30.693+0000 7ff82bfff700 1 --2- 172.21.15.42:0/1866410899 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff82c08ee80 0x7ff82c065800 secure :-1 s=THROTTLE_DONE pgs=1763 cs=0 l=1 rx=0x7ff81c002f30 tx=0x7ff81c008ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:30.726 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:30.693+0000 7ff82bfff700 1 --2- 172.21.15.42:0/1866410899 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff82c08ee80 0x7ff82c065800 secure :-1 s=THROTTLE_DONE pgs=1763 cs=0 l=1 rx=0x7ff81c002f30 tx=0x7ff81c008ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:30.726 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:30.693+0000 7ff828ff9700 1 -- 172.21.15.42:0/1866410899 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff81c00f280 con 0x7ff82c08ee80 2019-12-05T05:57:30.726 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:30.693+0000 7ff82bfff700 1 --2- 172.21.15.42:0/1866410899 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff82c08ee80 0x7ff82c065800 secure :-1 s=THROTTLE_DONE pgs=1763 cs=0 l=1 rx=0x7ff81c002f30 tx=0x7ff81c008ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:30.727 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:30.693+0000 7ff828ff9700 1 -- 172.21.15.42:0/1866410899 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff81c0116f0 con 0x7ff82c08ee80 2019-12-05T05:57:30.727 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:30.693+0000 7ff831e59700 1 -- 172.21.15.42:0/1866410899 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff82c0a3f00 con 0x7ff82c08ee80 2019-12-05T05:57:30.727 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:30.693+0000 7ff828ff9700 1 -- 172.21.15.42:0/1866410899 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff81c005540 con 0x7ff82c08ee80 2019-12-05T05:57:30.727 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:30.693+0000 7ff831e59700 1 -- 172.21.15.42:0/1866410899 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff82c132ed0 con 0x7ff82c08ee80 2019-12-05T05:57:30.727 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:30.697+0000 7ff82bfff700 1 --2- 172.21.15.42:0/1866410899 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff82c08ee80 0x7ff82c065800 secure :-1 s=THROTTLE_DONE pgs=1763 cs=0 l=1 rx=0x7ff81c002f30 tx=0x7ff81c008ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:30.727 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:30.697+0000 7ff828ff9700 1 -- 172.21.15.42:0/1866410899 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7ff81c005740 con 0x7ff82c08ee80 2019-12-05T05:57:30.727 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:30.697+0000 7ff82bfff700 1 --2- 172.21.15.42:0/1866410899 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff82c08ee80 0x7ff82c065800 secure :-1 s=THROTTLE_DONE pgs=1763 cs=0 l=1 rx=0x7ff81c002f30 tx=0x7ff81c008ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:30.727 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:30.697+0000 7ff828ff9700 1 --2- 172.21.15.42:0/1866410899 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff81003b470 0x7ff81003d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:57:30.732 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:30.697+0000 7ff828ff9700 1 -- 172.21.15.42:0/1866410899 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 9213+0+0 (secure 0 0 0) 0x7ff81c04b6b0 con 0x7ff82c08ee80 2019-12-05T05:57:30.732 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:30.697+0000 7ff82b7fe700 1 --2- 172.21.15.42:0/1866410899 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff81003b470 0x7ff81003d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:57:30.732 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:30.697+0000 7ff82b7fe700 1 --2- 172.21.15.42:0/1866410899 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff81003b470 0x7ff81003d980 crc :-1 s=READY pgs=754 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:57:30.732 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:30.701+0000 7ff831e59700 1 -- 172.21.15.42:0/1866410899 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff82c05d100 con 0x7ff82c08ee80 2019-12-05T05:57:30.732 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:30.709+0000 7ff82bfff700 1 --2- 172.21.15.42:0/1866410899 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff82c08ee80 0x7ff82c065800 secure :-1 s=THROTTLE_DONE pgs=1763 cs=0 l=1 rx=0x7ff81c002f30 tx=0x7ff81c008ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:30.732 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:30.709+0000 7ff828ff9700 1 -- 172.21.15.42:0/1866410899 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7ff81c007f10 con 0x7ff82c08ee80 2019-12-05T05:57:30.875 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:30.873+0000 7ff831e59700 1 -- 172.21.15.42:0/1866410899 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff82c05b8b0 con 0x7ff82c08ee80 2019-12-05T05:57:30.876 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:30.873+0000 7ff82bfff700 1 --2- 172.21.15.42:0/1866410899 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff82c08ee80 0x7ff82c065800 secure :-1 s=THROTTLE_DONE pgs=1763 cs=0 l=1 rx=0x7ff81c002f30 tx=0x7ff81c008ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:30.876 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:30.873+0000 7ff828ff9700 1 -- 172.21.15.42:0/1866410899 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 217 v217) v1 ==== 95+0+1775 (secure 0 0 0) 0x7ff81c032730 con 0x7ff82c08ee80 2019-12-05T05:57:30.876 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 217 2019-12-05T05:57:30.877 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:57:30.877 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":217,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":217,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:57:29.289849+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8836},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8836":{"gid":8836,"name":"a","rank":0,"incarnation":214,"state":"up:active","state_seq":29,"addr":"172.21.15.42:6835/1368848584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1368848584},{"type":"v1","addr":"172.21.15.42:6835","nonce":1368848584}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:57:30.879 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:30.873+0000 7ff831e59700 1 -- 172.21.15.42:0/1866410899 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff81003b470 msgr2=0x7ff81003d980 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:57:30.879 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:30.873+0000 7ff831e59700 1 --2- 172.21.15.42:0/1866410899 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff81003b470 0x7ff81003d980 crc :-1 s=READY pgs=754 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:57:30.879 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:30.873+0000 7ff831e59700 1 -- 172.21.15.42:0/1866410899 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff82c08ee80 msgr2=0x7ff82c065800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:57:30.879 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:30.873+0000 7ff831e59700 1 --2- 172.21.15.42:0/1866410899 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff82c08ee80 0x7ff82c065800 secure :-1 s=READY pgs=1763 cs=0 l=1 rx=0x7ff81c002f30 tx=0x7ff81c008ae0).stop 2019-12-05T05:57:30.880 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:30.873+0000 7ff831e59700 1 -- 172.21.15.42:0/1866410899 shutdown_connections 2019-12-05T05:57:30.880 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:30.873+0000 7ff831e59700 1 --2- 172.21.15.42:0/1866410899 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff81003b470 0x7ff81003d980 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:57:30.880 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:30.873+0000 7ff831e59700 1 --2- 172.21.15.42:0/1866410899 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff82c08ee80 0x7ff82c065800 unknown :-1 s=CLOSED pgs=1763 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:57:30.880 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:30.873+0000 7ff831e59700 1 -- 172.21.15.42:0/1866410899 shutdown_connections 2019-12-05T05:57:30.881 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:30.873+0000 7ff831e59700 1 -- 172.21.15.42:0/1866410899 wait complete. 2019-12-05T05:57:30.881 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:30.873+0000 7ff831e59700 1 -- 172.21.15.42:0/1866410899 >> 172.21.15.42:0/1866410899 conn(0x7ff82c1162c0 msgr2=0x7ff82c0596f0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:57:30.889 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a has gained rank=0, replacing gid=6461 2019-12-05T05:57:30.889 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 6 secs before reviving mds.b 2019-12-05T05:57:36.890 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.b 2019-12-05T05:57:36.890 INFO:tasks.ceph.mds.b:Restarting daemon 2019-12-05T05:57:36.890 INFO:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2019-12-05T05:57:36.900 INFO:tasks.ceph.mds.b:Started 2019-12-05T05:57:36.901 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:57:36.962 INFO:tasks.ceph.mds.b.smithi040.stdout:starting mds.b at 2019-12-05T05:57:37.009 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:37.001+0000 7f27364c4700 1 Processor -- start 2019-12-05T05:57:37.009 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:37.001+0000 7f27364c4700 1 -- start start 2019-12-05T05:57:37.010 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:37.001+0000 7f27364c4700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f27300a7740 0x7f273008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:57:37.010 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:37.001+0000 7f27364c4700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f273009fae0 0x7f273007eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:57:37.010 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:37.001+0000 7f27364c4700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27300a0710 con 0x7f273007e720 2019-12-05T05:57:37.010 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:37.001+0000 7f27364c4700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27300a0b00 con 0x7f27300a7740 2019-12-05T05:57:37.010 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:37.001+0000 7f27364c4700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f273008e5b0 con 0x7f273009fae0 2019-12-05T05:57:37.010 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:37.001+0000 7f2735262700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f273009fae0 0x7f273007eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:57:37.010 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:37.001+0000 7f272ffff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f273007e720 0x7f273009f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:44910/0 (socket says 172.21.15.42:44910) 2019-12-05T05:57:37.011 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:37.001+0000 7f272ffff700 1 -- 172.21.15.42:0/1435883956 learned_addr learned my addr 172.21.15.42:0/1435883956 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:57:37.011 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:37.001+0000 7f2735262700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f273009fae0 0x7f273007eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:44874/0 (socket says 172.21.15.42:44874) 2019-12-05T05:57:37.011 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:37.001+0000 7f2734a61700 1 --2- 172.21.15.42:0/1435883956 >> v2:172.21.15.40:3300/0 conn(0x7f27300a7740 0x7f273008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:57:37.011 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:37.001+0000 7f2734a61700 1 -- 172.21.15.42:0/1435883956 >> v2:172.21.15.40:3300/0 conn(0x7f27300a7740 msgr2=0x7f273008df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2019-12-05T05:57:37.011 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:37.001+0000 7f2734a61700 1 -- 172.21.15.42:0/1435883956 >> v2:172.21.15.40:3300/0 conn(0x7f27300a7740 msgr2=0x7f273008df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2019-12-05T05:57:37.011 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:37.001+0000 7f2734a61700 1 --2- 172.21.15.42:0/1435883956 >> v2:172.21.15.40:3300/0 conn(0x7f27300a7740 0x7f273008df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-12-05T05:57:37.011 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:37.001+0000 7f2734a61700 1 --2- 172.21.15.42:0/1435883956 >> v2:172.21.15.40:3300/0 conn(0x7f27300a7740 0x7f273008df80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rx=0 tx=0)._fault waiting 0.200000 2019-12-05T05:57:37.012 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:37.001+0000 7f2735262700 1 -- 172.21.15.42:0/1435883956 >> v2:172.21.15.40:3300/0 conn(0x7f27300a7740 msgr2=0x7f273008df80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2019-12-05T05:57:37.012 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:37.001+0000 7f2735262700 1 --2- 172.21.15.42:0/1435883956 >> v2:172.21.15.40:3300/0 conn(0x7f27300a7740 0x7f273008df80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:57:37.012 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:37.001+0000 7f2735262700 1 -- 172.21.15.42:0/1435883956 >> v1:172.21.15.42:6789/0 conn(0x7f273007e720 legacy=0x7f273009f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:57:37.012 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:37.001+0000 7f2735262700 1 -- 172.21.15.42:0/1435883956 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f273007f0a0 con 0x7f273009fae0 2019-12-05T05:57:37.012 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:37.001+0000 7f2735262700 1 --2- 172.21.15.42:0/1435883956 >> v2:172.21.15.42:3300/0 conn(0x7f273009fae0 0x7f273007eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f2728002f30 tx=0x7f272800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:37.012 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:37.001+0000 7f2735262700 1 --2- 172.21.15.42:0/1435883956 >> v2:172.21.15.42:3300/0 conn(0x7f273009fae0 0x7f273007eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f2728002f30 tx=0x7f272800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:37.012 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:37.001+0000 7f2735262700 1 --2- 172.21.15.42:0/1435883956 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f273009fae0 0x7f273007eb50 secure :-1 s=READY pgs=1766 cs=0 l=1 rx=0x7f2728002f30 tx=0x7f272800fc70).ready entity=mon.1 client_cookie=e1371eef4ec6f73c server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:57:37.013 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:37.001+0000 7f2735262700 1 --2- 172.21.15.42:0/1435883956 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f273009fae0 0x7f273007eb50 secure :-1 s=THROTTLE_DONE pgs=1766 cs=0 l=1 rx=0x7f2728002f30 tx=0x7f272800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:37.013 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:37.001+0000 7f272f7fe700 1 -- 172.21.15.42:0/1435883956 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2728008bb0 con 0x7f273009fae0 2019-12-05T05:57:37.013 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:37.001+0000 7f2735262700 1 --2- 172.21.15.42:0/1435883956 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f273009fae0 0x7f273007eb50 secure :-1 s=THROTTLE_DONE pgs=1766 cs=0 l=1 rx=0x7f2728002f30 tx=0x7f272800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:37.013 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:37.001+0000 7f2735262700 1 --2- 172.21.15.42:0/1435883956 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f273009fae0 0x7f273007eb50 secure :-1 s=THROTTLE_DONE pgs=1766 cs=0 l=1 rx=0x7f2728002f30 tx=0x7f272800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:37.013 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:37.001+0000 7f272f7fe700 1 -- 172.21.15.42:0/1435883956 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27280087d0 con 0x7f273009fae0 2019-12-05T05:57:37.013 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:37.001+0000 7f272f7fe700 1 -- 172.21.15.42:0/1435883956 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2728011820 con 0x7f273009fae0 2019-12-05T05:57:37.013 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:37.001+0000 7f27364c4700 1 -- 172.21.15.42:0/1435883956 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f273009fae0 msgr2=0x7f273007eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:57:37.014 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:37.001+0000 7f27364c4700 1 --2- 172.21.15.42:0/1435883956 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f273009fae0 0x7f273007eb50 secure :-1 s=READY pgs=1766 cs=0 l=1 rx=0x7f2728002f30 tx=0x7f272800fc70).stop 2019-12-05T05:57:37.014 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:37.005+0000 7f27364c4700 1 -- 172.21.15.42:0/1435883956 shutdown_connections 2019-12-05T05:57:37.014 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:37.005+0000 7f27364c4700 1 --2- 172.21.15.42:0/1435883956 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f273009fae0 0x7f273007eb50 unknown :-1 s=CLOSED pgs=1766 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:57:37.014 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:37.005+0000 7f27364c4700 1 --2- 172.21.15.42:0/1435883956 >> v2:172.21.15.40:3300/0 conn(0x7f27300a7740 0x7f273008df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:57:37.014 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:37.005+0000 7f27364c4700 1 -- 172.21.15.42:0/1435883956 shutdown_connections 2019-12-05T05:57:37.014 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:37.005+0000 7f27364c4700 1 -- 172.21.15.42:0/1435883956 wait complete. 2019-12-05T05:57:37.014 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:37.005+0000 7f27364c4700 1 -- 172.21.15.42:0/1435883956 >> 172.21.15.42:0/1435883956 conn(0x7f27300aabe0 msgr2=0x7f27300a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:57:37.018 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:37.005+0000 7f27364c4700 1 Processor -- start 2019-12-05T05:57:37.018 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:37.005+0000 7f27364c4700 1 -- start start 2019-12-05T05:57:37.018 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:37.005+0000 7f27364c4700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f273011a650 0x7f2730116ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:57:37.018 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:37.005+0000 7f27364c4700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f27300742e0 0x7f2730076720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:57:37.018 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:37.005+0000 7f27364c4700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27300a6a70 con 0x7f27301161c0 2019-12-05T05:57:37.018 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:37.005+0000 7f27364c4700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27300a6ea0 con 0x7f273011a650 2019-12-05T05:57:37.018 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:37.005+0000 7f27364c4700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27300a7140 con 0x7f27300742e0 2019-12-05T05:57:37.019 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:37.005+0000 7f2735262700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f27300742e0 0x7f2730076720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:57:37.019 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:37.005+0000 7f272ffff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f27301161c0 0x7f273011bd20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:44918/0 (socket says 172.21.15.42:44918) 2019-12-05T05:57:37.019 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:37.005+0000 7f272ffff700 1 -- 172.21.15.42:0/1400171576 learned_addr learned my addr 172.21.15.42:0/1400171576 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:57:37.019 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:37.005+0000 7f2734a61700 1 --2- 172.21.15.42:0/1400171576 >> v2:172.21.15.40:3300/0 conn(0x7f273011a650 0x7f2730116ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:57:37.019 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:37.009+0000 7f2735262700 1 -- 172.21.15.42:0/1400171576 >> v2:172.21.15.40:3300/0 conn(0x7f273011a650 msgr2=0x7f2730116ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:57:37.019 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:37.009+0000 7f2735262700 1 --2- 172.21.15.42:0/1400171576 >> v2:172.21.15.40:3300/0 conn(0x7f273011a650 0x7f2730116ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:57:37.019 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:37.009+0000 7f2735262700 1 -- 172.21.15.42:0/1400171576 >> v1:172.21.15.42:6789/0 conn(0x7f27301161c0 legacy=0x7f273011bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:57:37.020 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:37.009+0000 7f2735262700 1 -- 172.21.15.42:0/1400171576 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f273011f250 con 0x7f27300742e0 2019-12-05T05:57:37.020 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:37.009+0000 7f272d7fa700 1 -- 172.21.15.42:0/1400171576 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2941038107 0 0) 0x7f27300a6a70 con 0x7f27301161c0 2019-12-05T05:57:37.020 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:37.009+0000 7f2735262700 1 --2- 172.21.15.42:0/1400171576 >> v2:172.21.15.42:3300/0 conn(0x7f27300742e0 0x7f2730076720 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f27301171a0 tx=0x7f2728008a40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:37.020 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:37.009+0000 7f2735262700 1 --2- 172.21.15.42:0/1400171576 >> v2:172.21.15.42:3300/0 conn(0x7f27300742e0 0x7f2730076720 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f27301171a0 tx=0x7f2728008a40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:37.020 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:37.009+0000 7f2735262700 1 --2- 172.21.15.42:0/1400171576 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f27300742e0 0x7f2730076720 secure :-1 s=READY pgs=1768 cs=0 l=1 rx=0x7f27301171a0 tx=0x7f2728008a40).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:57:37.021 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:37.009+0000 7f2735262700 1 --2- 172.21.15.42:0/1400171576 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f27300742e0 0x7f2730076720 secure :-1 s=THROTTLE_DONE pgs=1768 cs=0 l=1 rx=0x7f27301171a0 tx=0x7f2728008a40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:37.021 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:37.009+0000 7f272d7fa700 1 -- 172.21.15.42:0/1400171576 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2728005540 con 0x7f27300742e0 2019-12-05T05:57:37.021 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:37.009+0000 7f2735262700 1 --2- 172.21.15.42:0/1400171576 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f27300742e0 0x7f2730076720 secure :-1 s=THROTTLE_DONE pgs=1768 cs=0 l=1 rx=0x7f27301171a0 tx=0x7f2728008a40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:37.021 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:37.009+0000 7f2735262700 1 --2- 172.21.15.42:0/1400171576 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f27300742e0 0x7f2730076720 secure :-1 s=THROTTLE_DONE pgs=1768 cs=0 l=1 rx=0x7f27301171a0 tx=0x7f2728008a40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:37.021 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:37.009+0000 7f272d7fa700 1 -- 172.21.15.42:0/1400171576 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2728008db0 con 0x7f27300742e0 2019-12-05T05:57:37.021 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:37.009+0000 7f27364c4700 1 -- 172.21.15.42:0/1400171576 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2730076c70 con 0x7f27300742e0 2019-12-05T05:57:37.021 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:37.009+0000 7f272d7fa700 1 -- 172.21.15.42:0/1400171576 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2728005b20 con 0x7f27300742e0 2019-12-05T05:57:37.022 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:37.009+0000 7f27364c4700 1 -- 172.21.15.42:0/1400171576 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27300771f0 con 0x7f27300742e0 2019-12-05T05:57:37.022 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:37.009+0000 7f2735262700 1 --2- 172.21.15.42:0/1400171576 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f27300742e0 0x7f2730076720 secure :-1 s=THROTTLE_DONE pgs=1768 cs=0 l=1 rx=0x7f27301171a0 tx=0x7f2728008a40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:37.022 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:37.009+0000 7f272d7fa700 1 -- 172.21.15.42:0/1400171576 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f27280130b0 con 0x7f27300742e0 2019-12-05T05:57:37.022 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:37.009+0000 7f2735262700 1 --2- 172.21.15.42:0/1400171576 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f27300742e0 0x7f2730076720 secure :-1 s=THROTTLE_DONE pgs=1768 cs=0 l=1 rx=0x7f27301171a0 tx=0x7f2728008a40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:37.022 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:37.009+0000 7f272d7fa700 1 --2- 172.21.15.42:0/1400171576 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f271403ac00 0x7f271403d110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:57:37.022 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:37.009+0000 7f272d7fa700 1 -- 172.21.15.42:0/1400171576 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 9213+0+0 (secure 0 0 0) 0x7f272804b7e0 con 0x7f27300742e0 2019-12-05T05:57:37.022 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:37.009+0000 7f2734a61700 1 --2- 172.21.15.42:0/1400171576 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f271403ac00 0x7f271403d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:57:37.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:37.013+0000 7f2734a61700 1 --2- 172.21.15.42:0/1400171576 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f271403ac00 0x7f271403d110 crc :-1 s=READY pgs=755 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:57:37.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:37.017+0000 7f27364c4700 1 -- 172.21.15.42:0/1400171576 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27300a6ca0 con 0x7f27300742e0 2019-12-05T05:57:37.028 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:37.025+0000 7f2735262700 1 --2- 172.21.15.42:0/1400171576 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f27300742e0 0x7f2730076720 secure :-1 s=THROTTLE_DONE pgs=1768 cs=0 l=1 rx=0x7f27301171a0 tx=0x7f2728008a40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:37.029 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:37.025+0000 7f272d7fa700 1 -- 172.21.15.42:0/1400171576 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f2728032840 con 0x7f27300742e0 2019-12-05T05:57:37.198 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:37.193+0000 7f27364c4700 1 -- 172.21.15.42:0/1400171576 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f273005d110 con 0x7f27300742e0 2019-12-05T05:57:37.199 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:37.193+0000 7f2735262700 1 --2- 172.21.15.42:0/1400171576 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f27300742e0 0x7f2730076720 secure :-1 s=THROTTLE_DONE pgs=1768 cs=0 l=1 rx=0x7f27301171a0 tx=0x7f2728008a40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:37.199 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:37.193+0000 7f272d7fa700 1 -- 172.21.15.42:0/1400171576 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 217 v217) v1 ==== 95+0+1775 (secure 0 0 0) 0x7f2728005dc0 con 0x7f27300742e0 2019-12-05T05:57:37.199 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 217 2019-12-05T05:57:37.199 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:57:37.200 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":217,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":217,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:57:29.289849+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8836},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8836":{"gid":8836,"name":"a","rank":0,"incarnation":214,"state":"up:active","state_seq":29,"addr":"172.21.15.42:6835/1368848584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1368848584},{"type":"v1","addr":"172.21.15.42:6835","nonce":1368848584}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:57:37.202 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:37.197+0000 7f27364c4700 1 -- 172.21.15.42:0/1400171576 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f271403ac00 msgr2=0x7f271403d110 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:57:37.202 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:37.197+0000 7f27364c4700 1 --2- 172.21.15.42:0/1400171576 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f271403ac00 0x7f271403d110 crc :-1 s=READY pgs=755 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:57:37.202 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:37.197+0000 7f27364c4700 1 -- 172.21.15.42:0/1400171576 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f27300742e0 msgr2=0x7f2730076720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:57:37.202 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:37.197+0000 7f27364c4700 1 --2- 172.21.15.42:0/1400171576 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f27300742e0 0x7f2730076720 secure :-1 s=READY pgs=1768 cs=0 l=1 rx=0x7f27301171a0 tx=0x7f2728008a40).stop 2019-12-05T05:57:37.202 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:37.197+0000 7f27364c4700 1 -- 172.21.15.42:0/1400171576 shutdown_connections 2019-12-05T05:57:37.203 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:37.197+0000 7f27364c4700 1 --2- 172.21.15.42:0/1400171576 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f271403ac00 0x7f271403d110 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:57:37.203 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:37.197+0000 7f27364c4700 1 --2- 172.21.15.42:0/1400171576 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f27300742e0 0x7f2730076720 unknown :-1 s=CLOSED pgs=1768 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:57:37.203 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:37.197+0000 7f27364c4700 1 --2- 172.21.15.42:0/1400171576 >> v2:172.21.15.40:3300/0 conn(0x7f273011a650 0x7f2730116ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:57:37.203 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:37.197+0000 7f27364c4700 1 -- 172.21.15.42:0/1400171576 shutdown_connections 2019-12-05T05:57:37.203 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:37.197+0000 7f27364c4700 1 -- 172.21.15.42:0/1400171576 wait complete. 2019-12-05T05:57:37.203 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:37.197+0000 7f27364c4700 1 -- 172.21.15.42:0/1400171576 >> 172.21.15.42:0/1400171576 conn(0x7f27300a82a0 msgr2=0x7f273009f700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:57:37.211 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2019-12-05T05:57:39.212 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:57:39.333 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:39.321+0000 7f2d98220700 1 Processor -- start 2019-12-05T05:57:39.333 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:39.321+0000 7f2d98220700 1 -- start start 2019-12-05T05:57:39.333 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:39.321+0000 7f2d98220700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2d900a7740 0x7f2d9008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:57:39.334 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:39.321+0000 7f2d98220700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d900a0710 con 0x7f2d9007e720 2019-12-05T05:57:39.334 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:39.321+0000 7f2d98220700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d900a9ba0 con 0x7f2d9009fae0 2019-12-05T05:57:39.334 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:39.321+0000 7f2d98220700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d900a0b00 con 0x7f2d900a7740 2019-12-05T05:57:39.334 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:39.321+0000 7f2d96fbe700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f2d9009fae0 0x7f2d9014efe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:44940/0 (socket says 172.21.15.42:44940) 2019-12-05T05:57:39.335 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:39.321+0000 7f2d96fbe700 1 -- 172.21.15.42:0/127706930 learned_addr learned my addr 172.21.15.42:0/127706930 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:57:39.335 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:39.321+0000 7f2d967bd700 1 --2- 172.21.15.42:0/127706930 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2d900a7740 0x7f2d9008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:57:39.335 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:39.321+0000 7f2d957bb700 1 -- 172.21.15.42:0/127706930 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1278559189 0 0) 0x7f2d900a9ba0 con 0x7f2d9009fae0 2019-12-05T05:57:39.335 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:39.321+0000 7f2d957bb700 1 -- 172.21.15.42:0/127706930 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d7c0030b0 con 0x7f2d9009fae0 2019-12-05T05:57:39.336 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:39.325+0000 7f2d957bb700 1 -- 172.21.15.42:0/127706930 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2774381242 0 0) 0x7f2d900a0710 con 0x7f2d9007e720 2019-12-05T05:57:39.336 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:39.325+0000 7f2d957bb700 1 -- 172.21.15.42:0/127706930 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d900a9ba0 con 0x7f2d9007e720 2019-12-05T05:57:39.336 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:39.325+0000 7f2d957bb700 1 -- 172.21.15.42:0/127706930 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 783786644 0 0) 0x7f2d7c0030b0 con 0x7f2d9009fae0 2019-12-05T05:57:39.336 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:39.325+0000 7f2d957bb700 1 -- 172.21.15.42:0/127706930 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2d900a7740 msgr2=0x7f2d9008df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:57:39.337 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:39.325+0000 7f2d957bb700 1 --2- 172.21.15.42:0/127706930 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2d900a7740 0x7f2d9008df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:57:39.337 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:39.325+0000 7f2d957bb700 1 -- 172.21.15.42:0/127706930 >> v1:172.21.15.40:6789/0 conn(0x7f2d9007e720 legacy=0x7f2d9009f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:57:39.337 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:39.325+0000 7f2d957bb700 1 -- 172.21.15.42:0/127706930 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d9007eb50 con 0x7f2d9009fae0 2019-12-05T05:57:39.337 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:39.325+0000 7f2d957bb700 1 -- 172.21.15.42:0/127706930 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7f2d88001750 con 0x7f2d9009fae0 2019-12-05T05:57:39.337 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:39.325+0000 7f2d957bb700 1 -- 172.21.15.42:0/127706930 >> v1:172.21.15.42:6789/0 conn(0x7f2d9009fae0 legacy=0x7f2d9014efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:57:39.337 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:39.325+0000 7f2d957bb700 1 --2- 172.21.15.42:0/127706930 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2d7c003940 0x7f2d7c003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:57:39.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:39.325+0000 7f2d957bb700 1 --2- 172.21.15.42:0/127706930 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2d7c004420 0x7f2d7c004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:57:39.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:39.325+0000 7f2d957bb700 1 --2- 172.21.15.42:0/127706930 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2d7c004e90 0x7f2d7c04f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:57:39.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:39.325+0000 7f2d957bb700 1 -- 172.21.15.42:0/127706930 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d900a0b00 con 0x7f2d7c004420 2019-12-05T05:57:39.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:39.325+0000 7f2d957bb700 1 -- 172.21.15.42:0/127706930 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d7c04fcc0 con 0x7f2d7c004e90 2019-12-05T05:57:39.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:39.325+0000 7f2d957bb700 1 -- 172.21.15.42:0/127706930 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d7c04ff00 con 0x7f2d7c003940 2019-12-05T05:57:39.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:39.325+0000 7f2d96fbe700 1 --2- 172.21.15.42:0/127706930 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2d7c004e90 0x7f2d7c04f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:57:39.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:39.325+0000 7f2d95fbc700 1 --2- 172.21.15.42:0/127706930 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2d7c004420 0x7f2d7c004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:57:39.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:39.325+0000 7f2d967bd700 1 --2- 172.21.15.42:0/127706930 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2d7c003940 0x7f2d7c003d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:57:39.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:39.325+0000 7f2d95fbc700 1 -- 172.21.15.42:0/127706930 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2d7c003940 msgr2=0x7f2d7c003d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:57:39.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:39.325+0000 7f2d95fbc700 1 --2- 172.21.15.42:0/127706930 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2d7c003940 0x7f2d7c003d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:57:39.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:39.325+0000 7f2d95fbc700 1 -- 172.21.15.42:0/127706930 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2d7c004e90 msgr2=0x7f2d7c04f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:57:39.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:39.325+0000 7f2d95fbc700 1 --2- 172.21.15.42:0/127706930 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2d7c004e90 0x7f2d7c04f680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:57:39.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:39.325+0000 7f2d95fbc700 1 -- 172.21.15.42:0/127706930 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d7c050200 con 0x7f2d7c004420 2019-12-05T05:57:39.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:39.325+0000 7f2d96fbe700 1 --2- 172.21.15.42:0/127706930 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2d7c004e90 0x7f2d7c04f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:57:39.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:39.325+0000 7f2d95fbc700 1 --2- 172.21.15.42:0/127706930 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2d7c004420 0x7f2d7c004850 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f2d80000fe0 tx=0x7f2d8000be70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:39.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:39.325+0000 7f2d95fbc700 1 --2- 172.21.15.42:0/127706930 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2d7c004420 0x7f2d7c004850 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f2d80000fe0 tx=0x7f2d8000be70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:39.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:39.325+0000 7f2d95fbc700 1 --2- 172.21.15.42:0/127706930 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2d7c004420 0x7f2d7c004850 secure :-1 s=READY pgs=1342 cs=0 l=1 rx=0x7f2d80000fe0 tx=0x7f2d8000be70).ready entity=mon.0 client_cookie=b3347fabaa846f15 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:57:39.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:39.325+0000 7f2d95fbc700 1 --2- 172.21.15.42:0/127706930 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2d7c004420 0x7f2d7c004850 secure :-1 s=THROTTLE_DONE pgs=1342 cs=0 l=1 rx=0x7f2d80000fe0 tx=0x7f2d8000be70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:39.347 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:39.325+0000 7f2d957bb700 1 -- 172.21.15.42:0/127706930 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2d80010be0 con 0x7f2d7c004420 2019-12-05T05:57:39.347 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:39.325+0000 7f2d95fbc700 1 --2- 172.21.15.42:0/127706930 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2d7c004420 0x7f2d7c004850 secure :-1 s=THROTTLE_DONE pgs=1342 cs=0 l=1 rx=0x7f2d80000fe0 tx=0x7f2d8000be70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:39.347 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:39.325+0000 7f2d957bb700 1 -- 172.21.15.42:0/127706930 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d80010800 con 0x7f2d7c004420 2019-12-05T05:57:39.347 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:39.325+0000 7f2d95fbc700 1 --2- 172.21.15.42:0/127706930 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2d7c004420 0x7f2d7c004850 secure :-1 s=THROTTLE_DONE pgs=1342 cs=0 l=1 rx=0x7f2d80000fe0 tx=0x7f2d8000be70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:39.347 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:39.325+0000 7f2d957bb700 1 -- 172.21.15.42:0/127706930 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2d8000c5b0 con 0x7f2d7c004420 2019-12-05T05:57:39.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:39.325+0000 7f2d98220700 1 -- 172.21.15.42:0/127706930 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2d7c004420 msgr2=0x7f2d7c004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:57:39.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:39.325+0000 7f2d98220700 1 --2- 172.21.15.42:0/127706930 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2d7c004420 0x7f2d7c004850 secure :-1 s=READY pgs=1342 cs=0 l=1 rx=0x7f2d80000fe0 tx=0x7f2d8000be70).stop 2019-12-05T05:57:39.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:39.325+0000 7f2d96fbe700 1 -- 172.21.15.42:0/127706930 reap_dead start 2019-12-05T05:57:39.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:39.325+0000 7f2d98220700 1 -- 172.21.15.42:0/127706930 shutdown_connections 2019-12-05T05:57:39.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:39.325+0000 7f2d98220700 1 -- 172.21.15.42:0/127706930 shutdown_connections 2019-12-05T05:57:39.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:39.325+0000 7f2d98220700 1 -- 172.21.15.42:0/127706930 wait complete. 2019-12-05T05:57:39.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:39.325+0000 7f2d98220700 1 -- 172.21.15.42:0/127706930 >> 172.21.15.42:0/127706930 conn(0x7f2d900aabe0 msgr2=0x7f2d900a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:57:39.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:39.329+0000 7f2d98220700 1 Processor -- start 2019-12-05T05:57:39.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:39.329+0000 7f2d98220700 1 -- start start 2019-12-05T05:57:39.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:39.329+0000 7f2d98220700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2d90122690 0x7f2d90075ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:57:39.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:39.329+0000 7f2d98220700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d90142020 con 0x7f2d9011dc90 2019-12-05T05:57:39.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:39.329+0000 7f2d98220700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d9011e780 con 0x7f2d9011f2a0 2019-12-05T05:57:39.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:39.329+0000 7f2d98220700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d9011e240 con 0x7f2d90122690 2019-12-05T05:57:39.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:39.329+0000 7f2d967bd700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f2d9011dc90 0x7f2d9011eaa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:54388/0 (socket says 172.21.15.42:54388) 2019-12-05T05:57:39.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:39.329+0000 7f2d967bd700 1 -- 172.21.15.42:0/3501800857 learned_addr learned my addr 172.21.15.42:0/3501800857 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:57:39.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:39.329+0000 7f2d96fbe700 1 --2- 172.21.15.42:0/3501800857 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2d90122690 0x7f2d90075ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:57:39.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:39.329+0000 7f2d86ffd700 1 -- 172.21.15.42:0/3501800857 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4069257979 0 0) 0x7f2d90142020 con 0x7f2d9011dc90 2019-12-05T05:57:39.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:39.329+0000 7f2d86ffd700 1 -- 172.21.15.42:0/3501800857 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d780030b0 con 0x7f2d9011dc90 2019-12-05T05:57:39.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:39.329+0000 7f2d86ffd700 1 -- 172.21.15.42:0/3501800857 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4136783823 0 0) 0x7f2d9011e780 con 0x7f2d9011f2a0 2019-12-05T05:57:39.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:39.329+0000 7f2d86ffd700 1 -- 172.21.15.42:0/3501800857 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d90142020 con 0x7f2d9011f2a0 2019-12-05T05:57:39.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:39.329+0000 7f2d96fbe700 1 -- 172.21.15.42:0/3501800857 >> v1:172.21.15.42:6789/0 conn(0x7f2d9011f2a0 legacy=0x7f2d900722c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:57:39.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:39.329+0000 7f2d96fbe700 1 -- 172.21.15.42:0/3501800857 >> v1:172.21.15.40:6789/0 conn(0x7f2d9011dc90 legacy=0x7f2d9011eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:57:39.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:39.329+0000 7f2d96fbe700 1 -- 172.21.15.42:0/3501800857 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d90122ac0 con 0x7f2d90122690 2019-12-05T05:57:39.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:39.333+0000 7f2d96fbe700 1 --2- 172.21.15.42:0/3501800857 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2d90122690 0x7f2d90075ae0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f2d9011d3b0 tx=0x7f2d88001880).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:39.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:39.333+0000 7f2d96fbe700 1 --2- 172.21.15.42:0/3501800857 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2d90122690 0x7f2d90075ae0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f2d9011d3b0 tx=0x7f2d88001880).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:39.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:39.333+0000 7f2d96fbe700 1 --2- 172.21.15.42:0/3501800857 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2d90122690 0x7f2d90075ae0 secure :-1 s=READY pgs=398 cs=0 l=1 rx=0x7f2d9011d3b0 tx=0x7f2d88001880).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:57:39.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:39.333+0000 7f2d96fbe700 1 --2- 172.21.15.42:0/3501800857 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2d90122690 0x7f2d90075ae0 secure :-1 s=THROTTLE_DONE pgs=398 cs=0 l=1 rx=0x7f2d9011d3b0 tx=0x7f2d88001880).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:39.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:39.333+0000 7f2d86ffd700 1 -- 172.21.15.42:0/3501800857 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2d8800ebd0 con 0x7f2d90122690 2019-12-05T05:57:39.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:39.333+0000 7f2d96fbe700 1 --2- 172.21.15.42:0/3501800857 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2d90122690 0x7f2d90075ae0 secure :-1 s=THROTTLE_DONE pgs=398 cs=0 l=1 rx=0x7f2d9011d3b0 tx=0x7f2d88001880).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:39.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:39.333+0000 7f2d96fbe700 1 --2- 172.21.15.42:0/3501800857 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2d90122690 0x7f2d90075ae0 secure :-1 s=THROTTLE_DONE pgs=398 cs=0 l=1 rx=0x7f2d9011d3b0 tx=0x7f2d88001880).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:39.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:39.333+0000 7f2d86ffd700 1 -- 172.21.15.42:0/3501800857 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d8800e7f0 con 0x7f2d90122690 2019-12-05T05:57:39.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:39.333+0000 7f2d86ffd700 1 -- 172.21.15.42:0/3501800857 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2d880127f0 con 0x7f2d90122690 2019-12-05T05:57:39.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:39.333+0000 7f2d98220700 1 -- 172.21.15.42:0/3501800857 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d9011cab0 con 0x7f2d90122690 2019-12-05T05:57:39.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:39.333+0000 7f2d98220700 1 -- 172.21.15.42:0/3501800857 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d90123420 con 0x7f2d90122690 2019-12-05T05:57:39.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:39.333+0000 7f2d96fbe700 1 --2- 172.21.15.42:0/3501800857 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2d90122690 0x7f2d90075ae0 secure :-1 s=THROTTLE_DONE pgs=398 cs=0 l=1 rx=0x7f2d9011d3b0 tx=0x7f2d88001880).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:39.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:39.333+0000 7f2d86ffd700 1 -- 172.21.15.42:0/3501800857 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f2d880129f0 con 0x7f2d90122690 2019-12-05T05:57:39.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:39.333+0000 7f2d96fbe700 1 --2- 172.21.15.42:0/3501800857 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2d90122690 0x7f2d90075ae0 secure :-1 s=THROTTLE_DONE pgs=398 cs=0 l=1 rx=0x7f2d9011d3b0 tx=0x7f2d88001880).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:39.358 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:39.333+0000 7f2d86ffd700 1 --2- 172.21.15.42:0/3501800857 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2d7803b450 0x7f2d7803d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:57:39.358 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:39.333+0000 7f2d86ffd700 1 -- 172.21.15.42:0/3501800857 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 9213+0+0 (secure 0 0 0) 0x7f2d880120c0 con 0x7f2d90122690 2019-12-05T05:57:39.358 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:39.337+0000 7f2d967bd700 1 --2- 172.21.15.42:0/3501800857 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2d7803b450 0x7f2d7803d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:57:39.358 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:39.337+0000 7f2d967bd700 1 --2- 172.21.15.42:0/3501800857 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2d7803b450 0x7f2d7803d960 crc :-1 s=READY pgs=756 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:57:39.358 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:39.337+0000 7f2d98220700 1 -- 172.21.15.42:0/3501800857 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d9005d110 con 0x7f2d90122690 2019-12-05T05:57:39.358 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:39.345+0000 7f2d96fbe700 1 --2- 172.21.15.42:0/3501800857 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2d90122690 0x7f2d90075ae0 secure :-1 s=THROTTLE_DONE pgs=398 cs=0 l=1 rx=0x7f2d9011d3b0 tx=0x7f2d88001880).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:39.358 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:39.345+0000 7f2d86ffd700 1 -- 172.21.15.42:0/3501800857 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f2d88031d60 con 0x7f2d90122690 2019-12-05T05:57:39.527 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:39.521+0000 7f2d98220700 1 -- 172.21.15.42:0/3501800857 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2d9005b8c0 con 0x7f2d90122690 2019-12-05T05:57:39.528 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:39.525+0000 7f2d96fbe700 1 --2- 172.21.15.42:0/3501800857 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2d90122690 0x7f2d90075ae0 secure :-1 s=THROTTLE_DONE pgs=398 cs=0 l=1 rx=0x7f2d9011d3b0 tx=0x7f2d88001880).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:39.528 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:39.525+0000 7f2d86ffd700 1 -- 172.21.15.42:0/3501800857 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 217 v217) v1 ==== 95+0+1775 (secure 0 0 0) 0x7f2d88031fc0 con 0x7f2d90122690 2019-12-05T05:57:39.530 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:57:39.530 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":217,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":217,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:57:29.289849+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8836},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8836":{"gid":8836,"name":"a","rank":0,"incarnation":214,"state":"up:active","state_seq":29,"addr":"172.21.15.42:6835/1368848584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1368848584},{"type":"v1","addr":"172.21.15.42:6835","nonce":1368848584}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:57:39.530 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 217 2019-12-05T05:57:39.533 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:39.529+0000 7f2d98220700 1 -- 172.21.15.42:0/3501800857 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2d7803b450 msgr2=0x7f2d7803d960 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:57:39.533 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:39.529+0000 7f2d98220700 1 --2- 172.21.15.42:0/3501800857 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2d7803b450 0x7f2d7803d960 crc :-1 s=READY pgs=756 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:57:39.533 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:39.529+0000 7f2d98220700 1 -- 172.21.15.42:0/3501800857 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2d90122690 msgr2=0x7f2d90075ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:57:39.533 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:39.529+0000 7f2d98220700 1 --2- 172.21.15.42:0/3501800857 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2d90122690 0x7f2d90075ae0 secure :-1 s=READY pgs=398 cs=0 l=1 rx=0x7f2d9011d3b0 tx=0x7f2d88001880).stop 2019-12-05T05:57:39.534 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:39.529+0000 7f2d98220700 1 -- 172.21.15.42:0/3501800857 shutdown_connections 2019-12-05T05:57:39.534 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:39.529+0000 7f2d98220700 1 --2- 172.21.15.42:0/3501800857 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2d7803b450 0x7f2d7803d960 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:57:39.534 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:39.529+0000 7f2d98220700 1 --2- 172.21.15.42:0/3501800857 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2d90122690 0x7f2d90075ae0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:57:39.534 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:39.529+0000 7f2d98220700 1 -- 172.21.15.42:0/3501800857 shutdown_connections 2019-12-05T05:57:39.535 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:39.529+0000 7f2d98220700 1 -- 172.21.15.42:0/3501800857 wait complete. 2019-12-05T05:57:39.535 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:39.529+0000 7f2d98220700 1 -- 172.21.15.42:0/3501800857 >> 172.21.15.42:0/3501800857 conn(0x7f2d900a7740 msgr2=0x7f2d900a6b10 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:57:39.554 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2019-12-05T05:57:41.556 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:57:41.669 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:41.661+0000 7f0a7b621700 1 Processor -- start 2019-12-05T05:57:41.669 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:41.661+0000 7f0a7b621700 1 -- start start 2019-12-05T05:57:41.669 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:41.661+0000 7f0a7b621700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0a740a7740 0x7f0a7408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:57:41.670 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:41.661+0000 7f0a7b621700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a740a0710 con 0x7f0a7407e720 2019-12-05T05:57:41.670 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:41.661+0000 7f0a7b621700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a740a9ba0 con 0x7f0a7409fae0 2019-12-05T05:57:41.670 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:41.661+0000 7f0a7b621700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a740a0b00 con 0x7f0a740a7740 2019-12-05T05:57:41.670 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:41.661+0000 7f0a7a3bf700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f0a7409fae0 0x7f0a7414efe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:44962/0 (socket says 172.21.15.42:44962) 2019-12-05T05:57:41.670 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:41.661+0000 7f0a7a3bf700 1 -- 172.21.15.42:0/3280480679 learned_addr learned my addr 172.21.15.42:0/3280480679 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:57:41.670 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:41.661+0000 7f0a79bbe700 1 --2- 172.21.15.42:0/3280480679 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0a740a7740 0x7f0a7408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:57:41.670 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:41.661+0000 7f0a78bbc700 1 -- 172.21.15.42:0/3280480679 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1024476406 0 0) 0x7f0a740a9ba0 con 0x7f0a7409fae0 2019-12-05T05:57:41.671 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:41.661+0000 7f0a78bbc700 1 -- 172.21.15.42:0/3280480679 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a5c0030b0 con 0x7f0a7409fae0 2019-12-05T05:57:41.671 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:41.661+0000 7f0a79bbe700 1 -- 172.21.15.42:0/3280480679 >> v1:172.21.15.42:6789/0 conn(0x7f0a7409fae0 legacy=0x7f0a7414efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:57:41.671 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:41.661+0000 7f0a79bbe700 1 -- 172.21.15.42:0/3280480679 >> v1:172.21.15.40:6789/0 conn(0x7f0a7407e720 legacy=0x7f0a7409f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:57:41.671 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:41.661+0000 7f0a78bbc700 1 -- 172.21.15.42:0/3280480679 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2746832090 0 0) 0x7f0a740a0710 con 0x7f0a7407e720 2019-12-05T05:57:41.671 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:41.661+0000 7f0a79bbe700 1 -- 172.21.15.42:0/3280480679 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a7407eb50 con 0x7f0a740a7740 2019-12-05T05:57:41.671 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:41.661+0000 7f0a79bbe700 1 --2- 172.21.15.42:0/3280480679 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0a740a7740 0x7f0a7408df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f0a6c0032b0 tx=0x7f0a6c00ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:41.671 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:41.661+0000 7f0a79bbe700 1 --2- 172.21.15.42:0/3280480679 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0a740a7740 0x7f0a7408df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f0a6c0032b0 tx=0x7f0a6c00ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:41.672 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:41.661+0000 7f0a79bbe700 1 --2- 172.21.15.42:0/3280480679 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0a740a7740 0x7f0a7408df80 secure :-1 s=READY pgs=399 cs=0 l=1 rx=0x7f0a6c0032b0 tx=0x7f0a6c00ff70).ready entity=mon.2 client_cookie=6a1c4dc527fbff08 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:57:41.672 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:41.665+0000 7f0a79bbe700 1 --2- 172.21.15.42:0/3280480679 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0a740a7740 0x7f0a7408df80 secure :-1 s=THROTTLE_DONE pgs=399 cs=0 l=1 rx=0x7f0a6c0032b0 tx=0x7f0a6c00ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:41.672 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:41.665+0000 7f0a78bbc700 1 -- 172.21.15.42:0/3280480679 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0a6c011a10 con 0x7f0a740a7740 2019-12-05T05:57:41.672 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:41.665+0000 7f0a79bbe700 1 --2- 172.21.15.42:0/3280480679 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0a740a7740 0x7f0a7408df80 secure :-1 s=THROTTLE_DONE pgs=399 cs=0 l=1 rx=0x7f0a6c0032b0 tx=0x7f0a6c00ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:41.672 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:41.665+0000 7f0a79bbe700 1 --2- 172.21.15.42:0/3280480679 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0a740a7740 0x7f0a7408df80 secure :-1 s=THROTTLE_DONE pgs=399 cs=0 l=1 rx=0x7f0a6c0032b0 tx=0x7f0a6c00ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:41.672 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:41.665+0000 7f0a78bbc700 1 -- 172.21.15.42:0/3280480679 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a6c011630 con 0x7f0a740a7740 2019-12-05T05:57:41.672 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:41.665+0000 7f0a78bbc700 1 -- 172.21.15.42:0/3280480679 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0a6c00a160 con 0x7f0a740a7740 2019-12-05T05:57:41.673 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:41.665+0000 7f0a7b621700 1 -- 172.21.15.42:0/3280480679 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0a740a7740 msgr2=0x7f0a7408df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:57:41.673 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:41.665+0000 7f0a7b621700 1 --2- 172.21.15.42:0/3280480679 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0a740a7740 0x7f0a7408df80 secure :-1 s=READY pgs=399 cs=0 l=1 rx=0x7f0a6c0032b0 tx=0x7f0a6c00ff70).stop 2019-12-05T05:57:41.673 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:41.665+0000 7f0a7b621700 1 -- 172.21.15.42:0/3280480679 shutdown_connections 2019-12-05T05:57:41.673 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:41.665+0000 7f0a7b621700 1 --2- 172.21.15.42:0/3280480679 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0a740a7740 0x7f0a7408df80 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:57:41.673 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:41.665+0000 7f0a7b621700 1 -- 172.21.15.42:0/3280480679 shutdown_connections 2019-12-05T05:57:41.673 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:41.665+0000 7f0a7b621700 1 -- 172.21.15.42:0/3280480679 wait complete. 2019-12-05T05:57:41.674 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:41.665+0000 7f0a7b621700 1 -- 172.21.15.42:0/3280480679 >> 172.21.15.42:0/3280480679 conn(0x7f0a740aabe0 msgr2=0x7f0a740a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:57:41.678 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:41.665+0000 7f0a7b621700 1 Processor -- start 2019-12-05T05:57:41.678 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:41.665+0000 7f0a7b621700 1 -- start start 2019-12-05T05:57:41.678 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:41.669+0000 7f0a7b621700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f0a740a3720 0x7f0a740a3b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:57:41.678 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:41.669+0000 7f0a7b621700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a740a40a0 con 0x7f0a74065020 2019-12-05T05:57:41.678 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:41.669+0000 7f0a7b621700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a7407a510 con 0x7f0a74125640 2019-12-05T05:57:41.679 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:41.669+0000 7f0a7b621700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a741294e0 con 0x7f0a740a3720 2019-12-05T05:57:41.679 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:41.669+0000 7f0a7a3bf700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f0a74125640 0x7f0a74125a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:44968/0 (socket says 172.21.15.42:44968) 2019-12-05T05:57:41.679 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:41.669+0000 7f0a7a3bf700 1 -- 172.21.15.42:0/736828500 learned_addr learned my addr 172.21.15.42:0/736828500 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:57:41.679 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:41.669+0000 7f0a793bd700 1 --2- 172.21.15.42:0/736828500 >> v2:172.21.15.40:3300/0 conn(0x7f0a740a3720 0x7f0a740a3b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:57:41.679 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:41.669+0000 7f0a727fc700 1 -- 172.21.15.42:0/736828500 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3505519013 0 0) 0x7f0a7407a510 con 0x7f0a74125640 2019-12-05T05:57:41.679 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:41.669+0000 7f0a727fc700 1 -- 172.21.15.42:0/736828500 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a540030b0 con 0x7f0a74125640 2019-12-05T05:57:41.679 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:41.669+0000 7f0a793bd700 1 -- 172.21.15.42:0/736828500 >> v1:172.21.15.42:6789/0 conn(0x7f0a74125640 legacy=0x7f0a74125a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:57:41.680 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:41.669+0000 7f0a793bd700 1 -- 172.21.15.42:0/736828500 >> v1:172.21.15.40:6789/0 conn(0x7f0a74065020 legacy=0x7f0a74128b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:57:41.680 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:41.669+0000 7f0a793bd700 1 -- 172.21.15.42:0/736828500 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a74132fe0 con 0x7f0a740a3720 2019-12-05T05:57:41.680 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:41.669+0000 7f0a793bd700 1 --2- 172.21.15.42:0/736828500 >> v2:172.21.15.40:3300/0 conn(0x7f0a740a3720 0x7f0a740a3b50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f0a640014f0 tx=0x7f0a64001530).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:41.680 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:41.669+0000 7f0a793bd700 1 --2- 172.21.15.42:0/736828500 >> v2:172.21.15.40:3300/0 conn(0x7f0a740a3720 0x7f0a740a3b50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f0a640014f0 tx=0x7f0a64001530).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:41.680 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:41.669+0000 7f0a793bd700 1 --2- 172.21.15.42:0/736828500 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0a740a3720 0x7f0a740a3b50 secure :-1 s=READY pgs=1346 cs=0 l=1 rx=0x7f0a640014f0 tx=0x7f0a64001530).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:57:41.680 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:41.669+0000 7f0a793bd700 1 --2- 172.21.15.42:0/736828500 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0a740a3720 0x7f0a740a3b50 secure :-1 s=THROTTLE_DONE pgs=1346 cs=0 l=1 rx=0x7f0a640014f0 tx=0x7f0a64001530).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:41.681 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:41.669+0000 7f0a727fc700 1 -- 172.21.15.42:0/736828500 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0a6800cb70 con 0x7f0a740a3720 2019-12-05T05:57:41.681 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:41.669+0000 7f0a793bd700 1 --2- 172.21.15.42:0/736828500 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0a740a3720 0x7f0a740a3b50 secure :-1 s=THROTTLE_DONE pgs=1346 cs=0 l=1 rx=0x7f0a640014f0 tx=0x7f0a64001530).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:41.681 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:41.669+0000 7f0a793bd700 1 --2- 172.21.15.42:0/736828500 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0a740a3720 0x7f0a740a3b50 secure :-1 s=THROTTLE_DONE pgs=1346 cs=0 l=1 rx=0x7f0a640014f0 tx=0x7f0a64001530).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:41.681 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:41.669+0000 7f0a727fc700 1 -- 172.21.15.42:0/736828500 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a6800c790 con 0x7f0a740a3720 2019-12-05T05:57:41.681 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:41.669+0000 7f0a727fc700 1 -- 172.21.15.42:0/736828500 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0a68010f40 con 0x7f0a740a3720 2019-12-05T05:57:41.681 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:41.669+0000 7f0a7b621700 1 -- 172.21.15.42:0/736828500 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a74130fc0 con 0x7f0a740a3720 2019-12-05T05:57:41.681 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:41.669+0000 7f0a7b621700 1 -- 172.21.15.42:0/736828500 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a74131510 con 0x7f0a740a3720 2019-12-05T05:57:41.682 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:41.673+0000 7f0a793bd700 1 --2- 172.21.15.42:0/736828500 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0a740a3720 0x7f0a740a3b50 secure :-1 s=THROTTLE_DONE pgs=1346 cs=0 l=1 rx=0x7f0a640014f0 tx=0x7f0a64001530).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:41.682 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:41.673+0000 7f0a727fc700 1 -- 172.21.15.42:0/736828500 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f0a68011250 con 0x7f0a740a3720 2019-12-05T05:57:41.682 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:41.673+0000 7f0a793bd700 1 --2- 172.21.15.42:0/736828500 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0a740a3720 0x7f0a740a3b50 secure :-1 s=THROTTLE_DONE pgs=1346 cs=0 l=1 rx=0x7f0a640014f0 tx=0x7f0a64001530).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:41.682 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:41.673+0000 7f0a727fc700 1 --2- 172.21.15.42:0/736828500 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0a5403b270 0x7f0a5403d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:57:41.682 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:41.673+0000 7f0a727fc700 1 -- 172.21.15.42:0/736828500 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 9213+0+0 (secure 0 0 0) 0x7f0a68011dc0 con 0x7f0a740a3720 2019-12-05T05:57:41.682 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:41.673+0000 7f0a79bbe700 1 --2- 172.21.15.42:0/736828500 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0a5403b270 0x7f0a5403d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:57:41.682 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:41.673+0000 7f0a79bbe700 1 --2- 172.21.15.42:0/736828500 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0a5403b270 0x7f0a5403d780 crc :-1 s=READY pgs=758 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:57:41.683 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:41.677+0000 7f0a7b621700 1 -- 172.21.15.42:0/736828500 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a7405d110 con 0x7f0a740a3720 2019-12-05T05:57:41.690 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:41.685+0000 7f0a793bd700 1 --2- 172.21.15.42:0/736828500 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0a740a3720 0x7f0a740a3b50 secure :-1 s=THROTTLE_DONE pgs=1346 cs=0 l=1 rx=0x7f0a640014f0 tx=0x7f0a64001530).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:41.690 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:41.685+0000 7f0a727fc700 1 -- 172.21.15.42:0/736828500 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f0a680321e0 con 0x7f0a740a3720 2019-12-05T05:57:41.881 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:41.877+0000 7f0a7b621700 1 -- 172.21.15.42:0/736828500 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0a7405b8c0 con 0x7f0a740a3720 2019-12-05T05:57:41.882 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:41.877+0000 7f0a793bd700 1 --2- 172.21.15.42:0/736828500 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0a740a3720 0x7f0a740a3b50 secure :-1 s=THROTTLE_DONE pgs=1346 cs=0 l=1 rx=0x7f0a640014f0 tx=0x7f0a64001530).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:57:41.883 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:41.877+0000 7f0a727fc700 1 -- 172.21.15.42:0/736828500 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 218 v218) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f0a68032440 con 0x7f0a740a3720 2019-12-05T05:57:41.883 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 218 2019-12-05T05:57:41.883 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:57:41.883 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":218,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":8940,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/1082666540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1082666540},{"type":"v1","addr":"172.21.15.40:6833","nonce":1082666540}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":218}],"filesystems":[{"mdsmap":{"epoch":217,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:57:29.289849+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8836},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8836":{"gid":8836,"name":"a","rank":0,"incarnation":214,"state":"up:active","state_seq":29,"addr":"172.21.15.42:6835/1368848584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1368848584},{"type":"v1","addr":"172.21.15.42:6835","nonce":1368848584}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:57:41.886 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:41.881+0000 7f0a7b621700 1 -- 172.21.15.42:0/736828500 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0a5403b270 msgr2=0x7f0a5403d780 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:57:41.887 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:41.881+0000 7f0a7b621700 1 --2- 172.21.15.42:0/736828500 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0a5403b270 0x7f0a5403d780 crc :-1 s=READY pgs=758 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:57:41.887 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:41.881+0000 7f0a7b621700 1 -- 172.21.15.42:0/736828500 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0a740a3720 msgr2=0x7f0a740a3b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:57:41.887 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:41.881+0000 7f0a7b621700 1 --2- 172.21.15.42:0/736828500 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0a740a3720 0x7f0a740a3b50 secure :-1 s=READY pgs=1346 cs=0 l=1 rx=0x7f0a640014f0 tx=0x7f0a64001530).stop 2019-12-05T05:57:41.887 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:41.881+0000 7f0a7b621700 1 -- 172.21.15.42:0/736828500 shutdown_connections 2019-12-05T05:57:41.887 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:41.881+0000 7f0a7b621700 1 --2- 172.21.15.42:0/736828500 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0a5403b270 0x7f0a5403d780 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:57:41.887 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:41.881+0000 7f0a7b621700 1 --2- 172.21.15.42:0/736828500 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0a740a3720 0x7f0a740a3b50 unknown :-1 s=CLOSED pgs=1346 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:57:41.888 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:41.881+0000 7f0a7b621700 1 -- 172.21.15.42:0/736828500 shutdown_connections 2019-12-05T05:57:41.888 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:41.881+0000 7f0a7b621700 1 -- 172.21.15.42:0/736828500 wait complete. 2019-12-05T05:57:41.888 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:57:41.881+0000 7f0a7b621700 1 -- 172.21.15.42:0/736828500 >> 172.21.15.42:0/736828500 conn(0x7f0a740a6eb0 msgr2=0x7f0a740a7f90 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:57:41.899 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b reported in up:standby state 2019-12-05T05:57:41.900 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 107 secs before thrashing 2019-12-05T05:57:42.718 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:57:42.728 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:58:12.799 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:58:12.802 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:58:42.847 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:58:42.851 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:59:12.924 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:59:12.928 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:59:28.901 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:59:29.005 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:28.996+0000 7facf44db700 1 Processor -- start 2019-12-05T05:59:29.005 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:28.996+0000 7facf44db700 1 -- start start 2019-12-05T05:59:29.006 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:28.996+0000 7facf44db700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7facec0a7740 0x7facec08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:59:29.006 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:28.996+0000 7facf44db700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7facec07e720 0x7facec07eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:59:29.006 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:28.996+0000 7facf44db700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facec0a0710 con 0x7facec09f2e0 2019-12-05T05:59:29.006 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:28.996+0000 7facf44db700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7facec0a0b00 con 0x7facec0a7740 2019-12-05T05:59:29.006 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:28.996+0000 7facf44db700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facec08e5b0 con 0x7facec07e720 2019-12-05T05:59:29.006 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:28.996+0000 7facf2a78700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7facec0a7740 0x7facec08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:59:29.006 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:28.996+0000 7facf3279700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7facec09f2e0 0x7facec09f710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:45452/0 (socket says 172.21.15.42:45452) 2019-12-05T05:59:29.007 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:28.996+0000 7facf3279700 1 -- 172.21.15.42:0/3839436580 learned_addr learned my addr 172.21.15.42:0/3839436580 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:59:29.007 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:28.996+0000 7facf2277700 1 --2- 172.21.15.42:0/3839436580 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7facec07e720 0x7facec07eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:59:29.007 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:28.996+0000 7facf2a78700 1 -- 172.21.15.42:0/3839436580 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7facec07e720 msgr2=0x7facec07eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:59:29.007 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:28.996+0000 7facf2a78700 1 --2- 172.21.15.42:0/3839436580 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7facec07e720 0x7facec07eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:59:29.016 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:28.996+0000 7facf2a78700 1 -- 172.21.15.42:0/3839436580 >> v1:172.21.15.42:6789/0 conn(0x7facec09f2e0 legacy=0x7facec09f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:59:29.017 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:28.996+0000 7facf2a78700 1 -- 172.21.15.42:0/3839436580 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facec07f0a0 con 0x7facec0a7740 2019-12-05T05:59:29.017 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:28.996+0000 7facf2a78700 1 --2- 172.21.15.42:0/3839436580 >> v2:172.21.15.42:3300/0 conn(0x7facec0a7740 0x7facec08df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7face8008790 tx=0x7face800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:29.017 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:28.996+0000 7facf2a78700 1 --2- 172.21.15.42:0/3839436580 >> v2:172.21.15.42:3300/0 conn(0x7facec0a7740 0x7facec08df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7face8008790 tx=0x7face800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:29.017 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:28.996+0000 7facf2a78700 1 --2- 172.21.15.42:0/3839436580 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7facec0a7740 0x7facec08df80 secure :-1 s=READY pgs=1780 cs=0 l=1 rx=0x7face8008790 tx=0x7face800fc70).ready entity=mon.1 client_cookie=120d64d8127a029b server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:59:29.017 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:28.996+0000 7facf2a78700 1 --2- 172.21.15.42:0/3839436580 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7facec0a7740 0x7facec08df80 secure :-1 s=THROTTLE_DONE pgs=1780 cs=0 l=1 rx=0x7face8008790 tx=0x7face800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:29.017 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:28.996+0000 7facf1a76700 1 -- 172.21.15.42:0/3839436580 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7face80112f0 con 0x7facec0a7740 2019-12-05T05:59:29.018 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:28.996+0000 7facf2a78700 1 --2- 172.21.15.42:0/3839436580 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7facec0a7740 0x7facec08df80 secure :-1 s=THROTTLE_DONE pgs=1780 cs=0 l=1 rx=0x7face8008790 tx=0x7face800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:29.018 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.000+0000 7facf1a76700 1 -- 172.21.15.42:0/3839436580 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7face8008ad0 con 0x7facec0a7740 2019-12-05T05:59:29.018 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.000+0000 7facf2a78700 1 --2- 172.21.15.42:0/3839436580 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7facec0a7740 0x7facec08df80 secure :-1 s=THROTTLE_DONE pgs=1780 cs=0 l=1 rx=0x7face8008790 tx=0x7face800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:29.019 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.000+0000 7facf1a76700 1 -- 172.21.15.42:0/3839436580 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7face80118d0 con 0x7facec0a7740 2019-12-05T05:59:29.019 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.000+0000 7facf44db700 1 -- 172.21.15.42:0/3839436580 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7facec0a7740 msgr2=0x7facec08df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:59:29.019 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.000+0000 7facf44db700 1 --2- 172.21.15.42:0/3839436580 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7facec0a7740 0x7facec08df80 secure :-1 s=READY pgs=1780 cs=0 l=1 rx=0x7face8008790 tx=0x7face800fc70).stop 2019-12-05T05:59:29.019 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.000+0000 7facf44db700 1 -- 172.21.15.42:0/3839436580 shutdown_connections 2019-12-05T05:59:29.019 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.000+0000 7facf44db700 1 --2- 172.21.15.42:0/3839436580 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7facec0a7740 0x7facec08df80 unknown :-1 s=CLOSED pgs=1780 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:59:29.019 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.000+0000 7facf44db700 1 --2- 172.21.15.42:0/3839436580 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7facec07e720 0x7facec07eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:59:29.019 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.000+0000 7facf44db700 1 -- 172.21.15.42:0/3839436580 shutdown_connections 2019-12-05T05:59:29.020 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.000+0000 7facf44db700 1 -- 172.21.15.42:0/3839436580 wait complete. 2019-12-05T05:59:29.020 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.000+0000 7facf44db700 1 -- 172.21.15.42:0/3839436580 >> 172.21.15.42:0/3839436580 conn(0x7facec0aabe0 msgr2=0x7facec0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:59:29.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.000+0000 7facf44db700 1 Processor -- start 2019-12-05T05:59:29.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.000+0000 7facf44db700 1 -- start start 2019-12-05T05:59:29.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.004+0000 7facf44db700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7facec07bc80 0x7facec12d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:59:29.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.004+0000 7facf44db700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7facec0a5d50 0x7facec0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:59:29.024 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.004+0000 7facf44db700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facec0a66d0 con 0x7facec07c0b0 2019-12-05T05:59:29.024 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.004+0000 7facf44db700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7facec0a7110 con 0x7facec0a5d50 2019-12-05T05:59:29.024 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.004+0000 7facf44db700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facec0a6e70 con 0x7facec07bc80 2019-12-05T05:59:29.024 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.004+0000 7facf2277700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7facec07c0b0 0x7facec12d760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:45456/0 (socket says 172.21.15.42:45456) 2019-12-05T05:59:29.024 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.004+0000 7facf2277700 1 -- 172.21.15.42:0/1749679683 learned_addr learned my addr 172.21.15.42:0/1749679683 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:59:29.024 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.004+0000 7facf2a78700 1 --2- 172.21.15.42:0/1749679683 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7facec07bc80 0x7facec12d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:59:29.025 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.004+0000 7facf3279700 1 --2- 172.21.15.42:0/1749679683 >> v2:172.21.15.40:3300/0 conn(0x7facec0a5d50 0x7facec0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:59:29.025 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.004+0000 7face37fe700 1 -- 172.21.15.42:0/1749679683 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1267741778 0 0) 0x7facec0a66d0 con 0x7facec07c0b0 2019-12-05T05:59:29.025 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.004+0000 7face37fe700 1 -- 172.21.15.42:0/1749679683 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7facd40030b0 con 0x7facec07c0b0 2019-12-05T05:59:29.025 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.004+0000 7facf2a78700 1 -- 172.21.15.42:0/1749679683 >> v2:172.21.15.40:3300/0 conn(0x7facec0a5d50 msgr2=0x7facec0a6180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:59:29.025 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.004+0000 7facf2a78700 1 --2- 172.21.15.42:0/1749679683 >> v2:172.21.15.40:3300/0 conn(0x7facec0a5d50 0x7facec0a6180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:59:29.025 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.004+0000 7facf2a78700 1 -- 172.21.15.42:0/1749679683 >> v1:172.21.15.42:6789/0 conn(0x7facec07c0b0 legacy=0x7facec12d760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:59:29.025 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.004+0000 7facf2a78700 1 -- 172.21.15.42:0/1749679683 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facec0a6900 con 0x7facec07bc80 2019-12-05T05:59:29.026 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.004+0000 7facf2a78700 1 --2- 172.21.15.42:0/1749679683 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7facec07bc80 0x7facec12d210 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7face8008790 tx=0x7face8007a70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:29.026 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.004+0000 7facf2a78700 1 --2- 172.21.15.42:0/1749679683 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7facec07bc80 0x7facec12d210 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7face8008790 tx=0x7face8007a70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:29.026 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.004+0000 7facf2a78700 1 --2- 172.21.15.42:0/1749679683 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7facec07bc80 0x7facec12d210 secure :-1 s=READY pgs=402 cs=0 l=1 rx=0x7face8008790 tx=0x7face8007a70).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:59:29.026 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.004+0000 7facf2a78700 1 --2- 172.21.15.42:0/1749679683 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7facec07bc80 0x7facec12d210 secure :-1 s=THROTTLE_DONE pgs=402 cs=0 l=1 rx=0x7face8008790 tx=0x7face8007a70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:29.026 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.004+0000 7face37fe700 1 -- 172.21.15.42:0/1749679683 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7face800f360 con 0x7facec07bc80 2019-12-05T05:59:29.026 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.004+0000 7facf2a78700 1 --2- 172.21.15.42:0/1749679683 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7facec07bc80 0x7facec12d210 secure :-1 s=THROTTLE_DONE pgs=402 cs=0 l=1 rx=0x7face8008790 tx=0x7face8007a70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:29.026 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.004+0000 7facf44db700 1 -- 172.21.15.42:0/1749679683 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facec08eb70 con 0x7facec07bc80 2019-12-05T05:59:29.027 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.004+0000 7facf2a78700 1 --2- 172.21.15.42:0/1749679683 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7facec07bc80 0x7facec12d210 secure :-1 s=THROTTLE_DONE pgs=402 cs=0 l=1 rx=0x7face8008790 tx=0x7face8007a70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:29.027 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.004+0000 7face37fe700 1 -- 172.21.15.42:0/1749679683 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7face80114f0 con 0x7facec07bc80 2019-12-05T05:59:29.027 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.004+0000 7face37fe700 1 -- 172.21.15.42:0/1749679683 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7face80167a0 con 0x7facec07bc80 2019-12-05T05:59:29.027 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.004+0000 7facf44db700 1 -- 172.21.15.42:0/1749679683 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facec08f0f0 con 0x7facec07bc80 2019-12-05T05:59:29.027 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.004+0000 7facf2a78700 1 --2- 172.21.15.42:0/1749679683 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7facec07bc80 0x7facec12d210 secure :-1 s=THROTTLE_DONE pgs=402 cs=0 l=1 rx=0x7face8008790 tx=0x7face8007a70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:29.027 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.008+0000 7face37fe700 1 -- 172.21.15.42:0/1749679683 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7face8007170 con 0x7facec07bc80 2019-12-05T05:59:29.027 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.008+0000 7facf2a78700 1 --2- 172.21.15.42:0/1749679683 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7facec07bc80 0x7facec12d210 secure :-1 s=THROTTLE_DONE pgs=402 cs=0 l=1 rx=0x7face8008790 tx=0x7face8007a70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:29.028 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.008+0000 7face37fe700 1 --2- 172.21.15.42:0/1749679683 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7facd403b270 0x7facd403d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:59:29.028 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.008+0000 7face37fe700 1 -- 172.21.15.42:0/1749679683 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 9213+0+0 (secure 0 0 0) 0x7face804d830 con 0x7facec07bc80 2019-12-05T05:59:29.033 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.008+0000 7facf2277700 1 --2- 172.21.15.42:0/1749679683 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7facd403b270 0x7facd403d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:59:29.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.008+0000 7facf2277700 1 --2- 172.21.15.42:0/1749679683 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7facd403b270 0x7facd403d780 crc :-1 s=READY pgs=760 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:59:29.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.012+0000 7facf44db700 1 -- 172.21.15.42:0/1749679683 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facec05d110 con 0x7facec07bc80 2019-12-05T05:59:29.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.020+0000 7facf2a78700 1 --2- 172.21.15.42:0/1749679683 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7facec07bc80 0x7facec12d210 secure :-1 s=THROTTLE_DONE pgs=402 cs=0 l=1 rx=0x7face8008790 tx=0x7face8007a70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:29.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.020+0000 7face37fe700 1 -- 172.21.15.42:0/1749679683 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7face80347f0 con 0x7facec07bc80 2019-12-05T05:59:29.193 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.188+0000 7facf44db700 1 -- 172.21.15.42:0/1749679683 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7facec05b8c0 con 0x7facec07bc80 2019-12-05T05:59:29.193 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.188+0000 7facf2a78700 1 --2- 172.21.15.42:0/1749679683 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7facec07bc80 0x7facec12d210 secure :-1 s=THROTTLE_DONE pgs=402 cs=0 l=1 rx=0x7face8008790 tx=0x7face8007a70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:29.193 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.188+0000 7face37fe700 1 -- 172.21.15.42:0/1749679683 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 218 v218) v1 ==== 95+0+2112 (secure 0 0 0) 0x7face8032440 con 0x7facec07bc80 2019-12-05T05:59:29.196 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:59:29.196 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":218,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":8940,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/1082666540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1082666540},{"type":"v1","addr":"172.21.15.40:6833","nonce":1082666540}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":218}],"filesystems":[{"mdsmap":{"epoch":217,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:57:29.289849+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8836},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8836":{"gid":8836,"name":"a","rank":0,"incarnation":214,"state":"up:active","state_seq":29,"addr":"172.21.15.42:6835/1368848584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1368848584},{"type":"v1","addr":"172.21.15.42:6835","nonce":1368848584}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:59:29.196 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 218 2019-12-05T05:59:29.198 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.192+0000 7facf44db700 1 -- 172.21.15.42:0/1749679683 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7facd403b270 msgr2=0x7facd403d780 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:59:29.199 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.192+0000 7facf44db700 1 --2- 172.21.15.42:0/1749679683 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7facd403b270 0x7facd403d780 crc :-1 s=READY pgs=760 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:59:29.199 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.192+0000 7facf44db700 1 -- 172.21.15.42:0/1749679683 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7facec07bc80 msgr2=0x7facec12d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:59:29.199 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.192+0000 7facf44db700 1 --2- 172.21.15.42:0/1749679683 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7facec07bc80 0x7facec12d210 secure :-1 s=READY pgs=402 cs=0 l=1 rx=0x7face8008790 tx=0x7face8007a70).stop 2019-12-05T05:59:29.199 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.192+0000 7facf44db700 1 -- 172.21.15.42:0/1749679683 shutdown_connections 2019-12-05T05:59:29.200 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.192+0000 7facf44db700 1 --2- 172.21.15.42:0/1749679683 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7facd403b270 0x7facd403d780 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:59:29.200 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.192+0000 7facf44db700 1 --2- 172.21.15.42:0/1749679683 >> v2:172.21.15.40:3300/0 conn(0x7facec0a5d50 0x7facec0a6180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:59:29.200 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.192+0000 7facf44db700 1 --2- 172.21.15.42:0/1749679683 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7facec07bc80 0x7facec12d210 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:59:29.200 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.192+0000 7facf44db700 1 -- 172.21.15.42:0/1749679683 shutdown_connections 2019-12-05T05:59:29.200 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.192+0000 7facf44db700 1 -- 172.21.15.42:0/1749679683 wait complete. 2019-12-05T05:59:29.200 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.192+0000 7facf44db700 1 -- 172.21.15.42:0/1749679683 >> 172.21.15.42:0/1749679683 conn(0x7facec0a9c30 msgr2=0x7facec11c6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:59:29.208 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.a (rank=0) 2019-12-05T05:59:29.208 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2019-12-05T05:59:29.208 INFO:teuthology.orchestra.run:waiting for 300 2019-12-05T05:59:29.224 INFO:tasks.ceph.mds.a:Stopped 2019-12-05T05:59:29.225 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:59:29.391 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.380+0000 7f7c16a2f700 1 Processor -- start 2019-12-05T05:59:29.392 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.380+0000 7f7c16a2f700 1 -- start start 2019-12-05T05:59:29.392 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.384+0000 7f7c16a2f700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f7c1008df80 0x7f7c1009f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:59:29.392 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.384+0000 7f7c16a2f700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7c1009f830 0x7f7c1014efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:59:29.393 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.384+0000 7f7c16a2f700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c1009fc60 con 0x7f7c100a7740 2019-12-05T05:59:29.393 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.384+0000 7f7c16a2f700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c100a0b00 con 0x7f7c1009f830 2019-12-05T05:59:29.393 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.384+0000 7f7c16a2f700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c1008e5b0 con 0x7f7c1008df80 2019-12-05T05:59:29.393 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.384+0000 7f7c0ffff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f7c1008df80 0x7f7c1009f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:59:29.393 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.384+0000 7f7c14fcc700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f7c100a7740 0x7f7c1007e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:45466/0 (socket says 172.21.15.42:45466) 2019-12-05T05:59:29.393 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.384+0000 7f7c14fcc700 1 -- 172.21.15.42:0/3445207948 learned_addr learned my addr 172.21.15.42:0/3445207948 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:59:29.394 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.384+0000 7f7c157cd700 1 --2- 172.21.15.42:0/3445207948 >> v2:172.21.15.40:3300/0 conn(0x7f7c1009f830 0x7f7c1014efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:59:29.394 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.384+0000 7f7c0ffff700 1 -- 172.21.15.42:0/3445207948 >> v2:172.21.15.40:3300/0 conn(0x7f7c1009f830 msgr2=0x7f7c1014efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:59:29.395 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.384+0000 7f7c0ffff700 1 --2- 172.21.15.42:0/3445207948 >> v2:172.21.15.40:3300/0 conn(0x7f7c1009f830 0x7f7c1014efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:59:29.395 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.384+0000 7f7c0f7fe700 1 -- 172.21.15.42:0/3445207948 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1580234500 0 0) 0x7f7c1009fc60 con 0x7f7c100a7740 2019-12-05T05:59:29.395 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.384+0000 7f7c0ffff700 1 -- 172.21.15.42:0/3445207948 >> v1:172.21.15.42:6789/0 conn(0x7f7c100a7740 legacy=0x7f7c1007e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:59:29.395 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.384+0000 7f7c0ffff700 1 -- 172.21.15.42:0/3445207948 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c100a0710 con 0x7f7c1008df80 2019-12-05T05:59:29.395 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.384+0000 7f7c0ffff700 1 --2- 172.21.15.42:0/3445207948 >> v2:172.21.15.42:3300/0 conn(0x7f7c1008df80 0x7f7c1009f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f7c00002f30 tx=0x7f7c0000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:29.395 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.384+0000 7f7c0ffff700 1 --2- 172.21.15.42:0/3445207948 >> v2:172.21.15.42:3300/0 conn(0x7f7c1008df80 0x7f7c1009f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f7c00002f30 tx=0x7f7c0000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:29.396 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.384+0000 7f7c0ffff700 1 --2- 172.21.15.42:0/3445207948 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7c1008df80 0x7f7c1009f2e0 secure :-1 s=READY pgs=1783 cs=0 l=1 rx=0x7f7c00002f30 tx=0x7f7c0000fc70).ready entity=mon.1 client_cookie=5bcf26f08acfb7ca server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:59:29.396 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.384+0000 7f7c0ffff700 1 --2- 172.21.15.42:0/3445207948 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7c1008df80 0x7f7c1009f2e0 secure :-1 s=THROTTLE_DONE pgs=1783 cs=0 l=1 rx=0x7f7c00002f30 tx=0x7f7c0000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:29.396 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.384+0000 7f7c0ffff700 1 --2- 172.21.15.42:0/3445207948 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7c1008df80 0x7f7c1009f2e0 secure :-1 s=THROTTLE_DONE pgs=1783 cs=0 l=1 rx=0x7f7c00002f30 tx=0x7f7c0000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:29.396 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.384+0000 7f7c0f7fe700 1 -- 172.21.15.42:0/3445207948 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7c00008bb0 con 0x7f7c1008df80 2019-12-05T05:59:29.396 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.384+0000 7f7c0ffff700 1 --2- 172.21.15.42:0/3445207948 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7c1008df80 0x7f7c1009f2e0 secure :-1 s=THROTTLE_DONE pgs=1783 cs=0 l=1 rx=0x7f7c00002f30 tx=0x7f7c0000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:29.397 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.384+0000 7f7c0f7fe700 1 -- 172.21.15.42:0/3445207948 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c000087d0 con 0x7f7c1008df80 2019-12-05T05:59:29.397 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.384+0000 7f7c0f7fe700 1 -- 172.21.15.42:0/3445207948 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7c00011820 con 0x7f7c1008df80 2019-12-05T05:59:29.400 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.384+0000 7f7c16a2f700 1 -- 172.21.15.42:0/3445207948 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7c1008df80 msgr2=0x7f7c1009f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:59:29.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.384+0000 7f7c16a2f700 1 --2- 172.21.15.42:0/3445207948 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7c1008df80 0x7f7c1009f2e0 secure :-1 s=READY pgs=1783 cs=0 l=1 rx=0x7f7c00002f30 tx=0x7f7c0000fc70).stop 2019-12-05T05:59:29.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.384+0000 7f7c16a2f700 1 -- 172.21.15.42:0/3445207948 shutdown_connections 2019-12-05T05:59:29.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.384+0000 7f7c16a2f700 1 --2- 172.21.15.42:0/3445207948 >> v2:172.21.15.40:3300/0 conn(0x7f7c1009f830 0x7f7c1014efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:59:29.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.384+0000 7f7c16a2f700 1 --2- 172.21.15.42:0/3445207948 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7c1008df80 0x7f7c1009f2e0 unknown :-1 s=CLOSED pgs=1783 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:59:29.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.384+0000 7f7c16a2f700 1 -- 172.21.15.42:0/3445207948 shutdown_connections 2019-12-05T05:59:29.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.384+0000 7f7c16a2f700 1 -- 172.21.15.42:0/3445207948 wait complete. 2019-12-05T05:59:29.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.384+0000 7f7c16a2f700 1 -- 172.21.15.42:0/3445207948 >> 172.21.15.42:0/3445207948 conn(0x7f7c100aabe0 msgr2=0x7f7c100a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:59:29.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.388+0000 7f7c16a2f700 1 Processor -- start 2019-12-05T05:59:29.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.388+0000 7f7c16a2f700 1 -- start start 2019-12-05T05:59:29.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.388+0000 7f7c16a2f700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7c10116500 0x7f7c10126710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:59:29.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.388+0000 7f7c16a2f700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7c10126c60 0x7f7c10127090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:59:29.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.388+0000 7f7c16a2f700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c1007ba80 con 0x7f7c1011a500 2019-12-05T05:59:29.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.388+0000 7f7c16a2f700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c1011c2b0 con 0x7f7c10116500 2019-12-05T05:59:29.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.388+0000 7f7c16a2f700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c1011a9d0 con 0x7f7c10126c60 2019-12-05T05:59:29.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.388+0000 7f7c157cd700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7c10126c60 0x7f7c10127090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:59:29.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.388+0000 7f7c0ffff700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7c10116500 0x7f7c10126710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:59:29.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.388+0000 7f7c0ffff700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7c10116500 0x7f7c10126710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.42:33250/0 (socket says 172.21.15.42:33250) 2019-12-05T05:59:29.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.388+0000 7f7c157cd700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7c10126c60 0x7f7c10127090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:45900/0 (socket says 172.21.15.42:45900) 2019-12-05T05:59:29.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.388+0000 7f7c0ffff700 1 -- 172.21.15.42:0/142203439 learned_addr learned my addr 172.21.15.42:0/142203439 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:59:29.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.388+0000 7f7c157cd700 1 -- 172.21.15.42:0/142203439 >> v2:172.21.15.40:3300/0 conn(0x7f7c10116500 msgr2=0x7f7c10126710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:59:29.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.388+0000 7f7c157cd700 1 --2- 172.21.15.42:0/142203439 >> v2:172.21.15.40:3300/0 conn(0x7f7c10116500 0x7f7c10126710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:59:29.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.388+0000 7f7c157cd700 1 -- 172.21.15.42:0/142203439 >> v1:172.21.15.40:6789/0 conn(0x7f7c1011a500 legacy=0x7f7c10118f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:59:29.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.388+0000 7f7c157cd700 1 -- 172.21.15.42:0/142203439 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c1007bdf0 con 0x7f7c10126c60 2019-12-05T05:59:29.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.388+0000 7f7c0ffff700 1 --2- 172.21.15.42:0/142203439 >> v2:172.21.15.40:3300/0 conn(0x7f7c10116500 0x7f7c10126710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:59:29.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.392+0000 7f7c157cd700 1 --2- 172.21.15.42:0/142203439 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7c10126c60 0x7f7c10127090 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f7bf8001090 tx=0x7f7bf80010d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:29.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.392+0000 7f7c157cd700 1 --2- 172.21.15.42:0/142203439 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7c10126c60 0x7f7c10127090 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f7bf8001090 tx=0x7f7bf80010d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:29.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.392+0000 7f7c157cd700 1 --2- 172.21.15.42:0/142203439 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7c10126c60 0x7f7c10127090 secure :-1 s=READY pgs=403 cs=0 l=1 rx=0x7f7bf8001090 tx=0x7f7bf80010d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:59:29.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.392+0000 7f7c157cd700 1 --2- 172.21.15.42:0/142203439 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7c10126c60 0x7f7c10127090 secure :-1 s=THROTTLE_DONE pgs=403 cs=0 l=1 rx=0x7f7bf8001090 tx=0x7f7bf80010d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:29.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.392+0000 7f7c0d7fa700 1 -- 172.21.15.42:0/142203439 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7c08005ca0 con 0x7f7c10126c60 2019-12-05T05:59:29.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.392+0000 7f7c157cd700 1 --2- 172.21.15.42:0/142203439 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7c10126c60 0x7f7c10127090 secure :-1 s=THROTTLE_DONE pgs=403 cs=0 l=1 rx=0x7f7bf8001090 tx=0x7f7bf80010d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:29.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.392+0000 7f7c157cd700 1 --2- 172.21.15.42:0/142203439 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7c10126c60 0x7f7c10127090 secure :-1 s=THROTTLE_DONE pgs=403 cs=0 l=1 rx=0x7f7bf8001090 tx=0x7f7bf80010d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:29.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.392+0000 7f7c0d7fa700 1 -- 172.21.15.42:0/142203439 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c080058c0 con 0x7f7c10126c60 2019-12-05T05:59:29.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.392+0000 7f7c16a2f700 1 -- 172.21.15.42:0/142203439 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c10118a50 con 0x7f7c10126c60 2019-12-05T05:59:29.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.392+0000 7f7c0d7fa700 1 -- 172.21.15.42:0/142203439 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7c08012ae0 con 0x7f7c10126c60 2019-12-05T05:59:29.413 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.392+0000 7f7c16a2f700 1 -- 172.21.15.42:0/142203439 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c10119d70 con 0x7f7c10126c60 2019-12-05T05:59:29.413 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.392+0000 7f7c157cd700 1 --2- 172.21.15.42:0/142203439 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7c10126c60 0x7f7c10127090 secure :-1 s=THROTTLE_DONE pgs=403 cs=0 l=1 rx=0x7f7bf8001090 tx=0x7f7bf80010d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:29.413 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.392+0000 7f7c0d7fa700 1 -- 172.21.15.42:0/142203439 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f7c08012ce0 con 0x7f7c10126c60 2019-12-05T05:59:29.413 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.392+0000 7f7c157cd700 1 --2- 172.21.15.42:0/142203439 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7c10126c60 0x7f7c10127090 secure :-1 s=THROTTLE_DONE pgs=403 cs=0 l=1 rx=0x7f7bf8001090 tx=0x7f7bf80010d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:29.413 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.392+0000 7f7c0d7fa700 1 --2- 172.21.15.42:0/142203439 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f7bf403ae10 0x7f7bf403d320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:59:29.414 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.392+0000 7f7c0d7fa700 1 -- 172.21.15.42:0/142203439 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 9213+0+0 (secure 0 0 0) 0x7f7c0804b4d0 con 0x7f7c10126c60 2019-12-05T05:59:29.414 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.392+0000 7f7c14fcc700 1 --2- 172.21.15.42:0/142203439 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f7bf403ae10 0x7f7bf403d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:59:29.414 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.396+0000 7f7c14fcc700 1 --2- 172.21.15.42:0/142203439 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f7bf403ae10 0x7f7bf403d320 crc :-1 s=READY pgs=761 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:59:29.414 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.400+0000 7f7c16a2f700 1 -- 172.21.15.42:0/142203439 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c100aae10 con 0x7f7c10126c60 2019-12-05T05:59:29.415 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.408+0000 7f7c157cd700 1 --2- 172.21.15.42:0/142203439 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7c10126c60 0x7f7c10127090 secure :-1 s=THROTTLE_DONE pgs=403 cs=0 l=1 rx=0x7f7bf8001090 tx=0x7f7bf80010d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:29.415 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.408+0000 7f7c0d7fa700 1 -- 172.21.15.42:0/142203439 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f7c1007bdf0 con 0x7f7c10126c60 2019-12-05T05:59:29.585 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.580+0000 7f7c16a2f700 1 -- 172.21.15.42:0/142203439 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7c1005d110 con 0x7f7c10126c60 2019-12-05T05:59:29.585 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.580+0000 7f7c157cd700 1 --2- 172.21.15.42:0/142203439 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7c10126c60 0x7f7c10127090 secure :-1 s=THROTTLE_DONE pgs=403 cs=0 l=1 rx=0x7f7bf8001090 tx=0x7f7bf80010d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:29.585 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.580+0000 7f7c0d7fa700 1 -- 172.21.15.42:0/142203439 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 218 v218) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f7c080324f0 con 0x7f7c10126c60 2019-12-05T05:59:29.586 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:59:29.587 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":218,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":8940,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/1082666540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1082666540},{"type":"v1","addr":"172.21.15.40:6833","nonce":1082666540}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":218}],"filesystems":[{"mdsmap":{"epoch":217,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:57:29.289849+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8836},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8836":{"gid":8836,"name":"a","rank":0,"incarnation":214,"state":"up:active","state_seq":29,"addr":"172.21.15.42:6835/1368848584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1368848584},{"type":"v1","addr":"172.21.15.42:6835","nonce":1368848584}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:59:29.587 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 218 2019-12-05T05:59:29.589 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.584+0000 7f7c16a2f700 1 -- 172.21.15.42:0/142203439 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f7bf403ae10 msgr2=0x7f7bf403d320 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:59:29.589 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.584+0000 7f7c16a2f700 1 --2- 172.21.15.42:0/142203439 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f7bf403ae10 0x7f7bf403d320 crc :-1 s=READY pgs=761 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:59:29.589 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.584+0000 7f7c16a2f700 1 -- 172.21.15.42:0/142203439 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7c10126c60 msgr2=0x7f7c10127090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:59:29.590 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.584+0000 7f7c16a2f700 1 --2- 172.21.15.42:0/142203439 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7c10126c60 0x7f7c10127090 secure :-1 s=READY pgs=403 cs=0 l=1 rx=0x7f7bf8001090 tx=0x7f7bf80010d0).stop 2019-12-05T05:59:29.590 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.584+0000 7f7c16a2f700 1 -- 172.21.15.42:0/142203439 shutdown_connections 2019-12-05T05:59:29.590 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.584+0000 7f7c16a2f700 1 --2- 172.21.15.42:0/142203439 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f7bf403ae10 0x7f7bf403d320 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:59:29.590 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.584+0000 7f7c16a2f700 1 --2- 172.21.15.42:0/142203439 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7c10126c60 0x7f7c10127090 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:59:29.590 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.584+0000 7f7c16a2f700 1 --2- 172.21.15.42:0/142203439 >> v2:172.21.15.40:3300/0 conn(0x7f7c10116500 0x7f7c10126710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:59:29.590 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.584+0000 7f7c16a2f700 1 -- 172.21.15.42:0/142203439 shutdown_connections 2019-12-05T05:59:29.590 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.584+0000 7f7c16a2f700 1 -- 172.21.15.42:0/142203439 wait complete. 2019-12-05T05:59:29.591 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:29.584+0000 7f7c16a2f700 1 -- 172.21.15.42:0/142203439 >> 172.21.15.42:0/142203439 conn(0x7f7c1008fde0 msgr2=0x7f7c10059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:59:29.599 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T05:59:31.600 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:59:31.705 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:31.696+0000 7f6f5445a700 1 Processor -- start 2019-12-05T05:59:31.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:31.696+0000 7f6f5445a700 1 -- start start 2019-12-05T05:59:31.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:31.696+0000 7f6f5445a700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f6f4c0a7740 0x7f6f4c08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:59:31.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:31.696+0000 7f6f5445a700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6f4c07e720 0x7f6f4c07eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:59:31.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:31.696+0000 7f6f5445a700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f4c0a0710 con 0x7f6f4c09f2e0 2019-12-05T05:59:31.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:31.696+0000 7f6f5445a700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f4c0a0b00 con 0x7f6f4c0a7740 2019-12-05T05:59:31.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:31.696+0000 7f6f5445a700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f4c08e5b0 con 0x7f6f4c07e720 2019-12-05T05:59:31.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:31.696+0000 7f6f521f6700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6f4c07e720 0x7f6f4c07eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:59:31.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:31.696+0000 7f6f521f6700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6f4c07e720 0x7f6f4c07eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:45448/0 (socket says 172.21.15.42:45448) 2019-12-05T05:59:31.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:31.696+0000 7f6f521f6700 1 -- 172.21.15.42:0/1017021544 learned_addr learned my addr 172.21.15.42:0/1017021544 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:59:31.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:31.696+0000 7f6f529f7700 1 --2- 172.21.15.42:0/1017021544 >> v2:172.21.15.40:3300/0 conn(0x7f6f4c0a7740 0x7f6f4c08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:59:31.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:31.696+0000 7f6f521f6700 1 -- 172.21.15.42:0/1017021544 >> v2:172.21.15.40:3300/0 conn(0x7f6f4c0a7740 msgr2=0x7f6f4c08df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:59:31.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:31.696+0000 7f6f521f6700 1 --2- 172.21.15.42:0/1017021544 >> v2:172.21.15.40:3300/0 conn(0x7f6f4c0a7740 0x7f6f4c08df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:59:31.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:31.696+0000 7f6f521f6700 1 -- 172.21.15.42:0/1017021544 >> v1:172.21.15.40:6789/0 conn(0x7f6f4c09f2e0 legacy=0x7f6f4c09f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:59:31.708 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:31.696+0000 7f6f521f6700 1 -- 172.21.15.42:0/1017021544 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f4c07f0a0 con 0x7f6f4c07e720 2019-12-05T05:59:31.708 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:31.696+0000 7f6f521f6700 1 --2- 172.21.15.42:0/1017021544 >> v2:172.21.15.42:3300/0 conn(0x7f6f4c07e720 0x7f6f4c07eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f6f3c002f30 tx=0x7f6f3c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:31.708 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:31.696+0000 7f6f521f6700 1 --2- 172.21.15.42:0/1017021544 >> v2:172.21.15.42:3300/0 conn(0x7f6f4c07e720 0x7f6f4c07eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f6f3c002f30 tx=0x7f6f3c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:31.708 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:31.696+0000 7f6f521f6700 1 --2- 172.21.15.42:0/1017021544 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6f4c07e720 0x7f6f4c07eb50 secure :-1 s=READY pgs=1784 cs=0 l=1 rx=0x7f6f3c002f30 tx=0x7f6f3c00fd30).ready entity=mon.1 client_cookie=c3ec04706881f1d4 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:59:31.708 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:31.696+0000 7f6f521f6700 1 --2- 172.21.15.42:0/1017021544 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6f4c07e720 0x7f6f4c07eb50 secure :-1 s=THROTTLE_DONE pgs=1784 cs=0 l=1 rx=0x7f6f3c002f30 tx=0x7f6f3c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:31.708 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:31.696+0000 7f6f519f5700 1 -- 172.21.15.42:0/1017021544 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6f3c011370 con 0x7f6f4c07e720 2019-12-05T05:59:31.708 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:31.696+0000 7f6f521f6700 1 --2- 172.21.15.42:0/1017021544 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6f4c07e720 0x7f6f4c07eb50 secure :-1 s=THROTTLE_DONE pgs=1784 cs=0 l=1 rx=0x7f6f3c002f30 tx=0x7f6f3c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:31.709 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:31.696+0000 7f6f521f6700 1 --2- 172.21.15.42:0/1017021544 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6f4c07e720 0x7f6f4c07eb50 secure :-1 s=THROTTLE_DONE pgs=1784 cs=0 l=1 rx=0x7f6f3c002f30 tx=0x7f6f3c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:31.709 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:31.696+0000 7f6f519f5700 1 -- 172.21.15.42:0/1017021544 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f3c008bb0 con 0x7f6f4c07e720 2019-12-05T05:59:31.709 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:31.696+0000 7f6f519f5700 1 -- 172.21.15.42:0/1017021544 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6f3c011950 con 0x7f6f4c07e720 2019-12-05T05:59:31.709 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:31.696+0000 7f6f5445a700 1 -- 172.21.15.42:0/1017021544 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6f4c07e720 msgr2=0x7f6f4c07eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:59:31.709 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:31.696+0000 7f6f5445a700 1 --2- 172.21.15.42:0/1017021544 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6f4c07e720 0x7f6f4c07eb50 secure :-1 s=READY pgs=1784 cs=0 l=1 rx=0x7f6f3c002f30 tx=0x7f6f3c00fd30).stop 2019-12-05T05:59:31.709 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:31.700+0000 7f6f5445a700 1 -- 172.21.15.42:0/1017021544 shutdown_connections 2019-12-05T05:59:31.709 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:31.700+0000 7f6f5445a700 1 --2- 172.21.15.42:0/1017021544 >> v2:172.21.15.40:3300/0 conn(0x7f6f4c0a7740 0x7f6f4c08df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:59:31.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:31.700+0000 7f6f5445a700 1 --2- 172.21.15.42:0/1017021544 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6f4c07e720 0x7f6f4c07eb50 unknown :-1 s=CLOSED pgs=1784 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:59:31.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:31.700+0000 7f6f5445a700 1 -- 172.21.15.42:0/1017021544 shutdown_connections 2019-12-05T05:59:31.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:31.700+0000 7f6f5445a700 1 -- 172.21.15.42:0/1017021544 wait complete. 2019-12-05T05:59:31.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:31.700+0000 7f6f5445a700 1 -- 172.21.15.42:0/1017021544 >> 172.21.15.42:0/1017021544 conn(0x7f6f4c0aabe0 msgr2=0x7f6f4c0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:59:31.713 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:31.700+0000 7f6f5445a700 1 Processor -- start 2019-12-05T05:59:31.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:31.704+0000 7f6f5445a700 1 -- start start 2019-12-05T05:59:31.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:31.704+0000 7f6f5445a700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f6f4c1156c0 0x7f6f4c117e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:59:31.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:31.704+0000 7f6f5445a700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6f4c1183c0 0x7f6f4c12a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:59:31.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:31.704+0000 7f6f5445a700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f4c0789d0 con 0x7f6f4c12ad00 2019-12-05T05:59:31.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:31.704+0000 7f6f5445a700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f4c12b130 con 0x7f6f4c1156c0 2019-12-05T05:59:31.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:31.704+0000 7f6f5445a700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f4c0781f0 con 0x7f6f4c1183c0 2019-12-05T05:59:31.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:31.704+0000 7f6f529f7700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f6f4c1156c0 0x7f6f4c117e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:59:31.715 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:31.704+0000 7f6f521f6700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6f4c1183c0 0x7f6f4c12a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:59:31.716 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:31.704+0000 7f6f531f8700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f6f4c12ad00 0x7f6f4c072fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:54934/0 (socket says 172.21.15.42:54934) 2019-12-05T05:59:31.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:31.704+0000 7f6f531f8700 1 -- 172.21.15.42:0/2271751087 learned_addr learned my addr 172.21.15.42:0/2271751087 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:59:31.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:31.704+0000 7f6f529f7700 1 -- 172.21.15.42:0/2271751087 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6f4c1183c0 msgr2=0x7f6f4c12a7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:59:31.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:31.704+0000 7f6f529f7700 1 --2- 172.21.15.42:0/2271751087 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6f4c1183c0 0x7f6f4c12a7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:59:31.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:31.704+0000 7f6f529f7700 1 -- 172.21.15.42:0/2271751087 >> v1:172.21.15.40:6789/0 conn(0x7f6f4c12ad00 legacy=0x7f6f4c072fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:59:31.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:31.704+0000 7f6f529f7700 1 -- 172.21.15.42:0/2271751087 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f4c0737a0 con 0x7f6f4c1156c0 2019-12-05T05:59:31.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:31.704+0000 7f6f529f7700 1 --2- 172.21.15.42:0/2271751087 >> v2:172.21.15.40:3300/0 conn(0x7f6f4c1156c0 0x7f6f4c117e70 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f6f4c0783b0 tx=0x7f6f480109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:31.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:31.704+0000 7f6f529f7700 1 --2- 172.21.15.42:0/2271751087 >> v2:172.21.15.40:3300/0 conn(0x7f6f4c1156c0 0x7f6f4c117e70 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f6f4c0783b0 tx=0x7f6f480109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:31.718 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:31.704+0000 7f6f529f7700 1 --2- 172.21.15.42:0/2271751087 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6f4c1156c0 0x7f6f4c117e70 secure :-1 s=READY pgs=1361 cs=0 l=1 rx=0x7f6f4c0783b0 tx=0x7f6f480109e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:59:31.718 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:31.704+0000 7f6f529f7700 1 --2- 172.21.15.42:0/2271751087 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6f4c1156c0 0x7f6f4c117e70 secure :-1 s=THROTTLE_DONE pgs=1361 cs=0 l=1 rx=0x7f6f4c0783b0 tx=0x7f6f480109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:31.718 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:31.704+0000 7f6f437fe700 1 -- 172.21.15.42:0/2271751087 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6f48005920 con 0x7f6f4c1156c0 2019-12-05T05:59:31.718 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:31.704+0000 7f6f529f7700 1 --2- 172.21.15.42:0/2271751087 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6f4c1156c0 0x7f6f4c117e70 secure :-1 s=THROTTLE_DONE pgs=1361 cs=0 l=1 rx=0x7f6f4c0783b0 tx=0x7f6f480109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:31.718 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:31.704+0000 7f6f529f7700 1 --2- 172.21.15.42:0/2271751087 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6f4c1156c0 0x7f6f4c117e70 secure :-1 s=THROTTLE_DONE pgs=1361 cs=0 l=1 rx=0x7f6f4c0783b0 tx=0x7f6f480109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:31.718 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:31.704+0000 7f6f437fe700 1 -- 172.21.15.42:0/2271751087 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f48005540 con 0x7f6f4c1156c0 2019-12-05T05:59:31.718 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:31.704+0000 7f6f5445a700 1 -- 172.21.15.42:0/2271751087 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f4c131690 con 0x7f6f4c1156c0 2019-12-05T05:59:31.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:31.704+0000 7f6f437fe700 1 -- 172.21.15.42:0/2271751087 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6f48012450 con 0x7f6f4c1156c0 2019-12-05T05:59:31.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:31.704+0000 7f6f5445a700 1 -- 172.21.15.42:0/2271751087 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f4c131c40 con 0x7f6f4c1156c0 2019-12-05T05:59:31.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:31.708+0000 7f6f529f7700 1 --2- 172.21.15.42:0/2271751087 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6f4c1156c0 0x7f6f4c117e70 secure :-1 s=THROTTLE_DONE pgs=1361 cs=0 l=1 rx=0x7f6f4c0783b0 tx=0x7f6f480109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:31.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:31.708+0000 7f6f529f7700 1 --2- 172.21.15.42:0/2271751087 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6f4c1156c0 0x7f6f4c117e70 secure :-1 s=THROTTLE_DONE pgs=1361 cs=0 l=1 rx=0x7f6f4c0783b0 tx=0x7f6f480109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:31.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:31.708+0000 7f6f437fe700 1 -- 172.21.15.42:0/2271751087 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f6f48007180 con 0x7f6f4c1156c0 2019-12-05T05:59:31.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:31.708+0000 7f6f437fe700 1 --2- 172.21.15.42:0/2271751087 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6f3403ace0 0x7f6f3403d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:59:31.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:31.708+0000 7f6f437fe700 1 -- 172.21.15.42:0/2271751087 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 9213+0+0 (secure 0 0 0) 0x7f6f4804aca0 con 0x7f6f4c1156c0 2019-12-05T05:59:31.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:31.708+0000 7f6f521f6700 1 --2- 172.21.15.42:0/2271751087 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6f3403ace0 0x7f6f3403d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:59:31.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:31.708+0000 7f6f521f6700 1 --2- 172.21.15.42:0/2271751087 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6f3403ace0 0x7f6f3403d1f0 crc :-1 s=READY pgs=762 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:59:31.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:31.708+0000 7f6f5445a700 1 -- 172.21.15.42:0/2271751087 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f4c05d110 con 0x7f6f4c1156c0 2019-12-05T05:59:31.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:31.720+0000 7f6f529f7700 1 --2- 172.21.15.42:0/2271751087 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6f4c1156c0 0x7f6f4c117e70 secure :-1 s=THROTTLE_DONE pgs=1361 cs=0 l=1 rx=0x7f6f4c0783b0 tx=0x7f6f480109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:31.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:31.720+0000 7f6f437fe700 1 -- 172.21.15.42:0/2271751087 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f6f48031ce0 con 0x7f6f4c1156c0 2019-12-05T05:59:31.895 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:31.892+0000 7f6f5445a700 1 -- 172.21.15.42:0/2271751087 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6f4c05b8c0 con 0x7f6f4c1156c0 2019-12-05T05:59:31.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:31.892+0000 7f6f529f7700 1 --2- 172.21.15.42:0/2271751087 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6f4c1156c0 0x7f6f4c117e70 secure :-1 s=THROTTLE_DONE pgs=1361 cs=0 l=1 rx=0x7f6f4c0783b0 tx=0x7f6f480109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:31.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:31.892+0000 7f6f437fe700 1 -- 172.21.15.42:0/2271751087 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 218 v218) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f6f48031f40 con 0x7f6f4c1156c0 2019-12-05T05:59:31.898 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 218 2019-12-05T05:59:31.899 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:59:31.899 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":218,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":8940,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/1082666540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1082666540},{"type":"v1","addr":"172.21.15.40:6833","nonce":1082666540}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":218}],"filesystems":[{"mdsmap":{"epoch":217,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:57:29.289849+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8836},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8836":{"gid":8836,"name":"a","rank":0,"incarnation":214,"state":"up:active","state_seq":29,"addr":"172.21.15.42:6835/1368848584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1368848584},{"type":"v1","addr":"172.21.15.42:6835","nonce":1368848584}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:59:31.901 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:31.896+0000 7f6f5445a700 1 -- 172.21.15.42:0/2271751087 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6f3403ace0 msgr2=0x7f6f3403d1f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:59:31.901 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:31.896+0000 7f6f5445a700 1 --2- 172.21.15.42:0/2271751087 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6f3403ace0 0x7f6f3403d1f0 crc :-1 s=READY pgs=762 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:59:31.902 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:31.896+0000 7f6f5445a700 1 -- 172.21.15.42:0/2271751087 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6f4c1156c0 msgr2=0x7f6f4c117e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:59:31.902 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:31.896+0000 7f6f5445a700 1 --2- 172.21.15.42:0/2271751087 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6f4c1156c0 0x7f6f4c117e70 secure :-1 s=READY pgs=1361 cs=0 l=1 rx=0x7f6f4c0783b0 tx=0x7f6f480109e0).stop 2019-12-05T05:59:31.902 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:31.896+0000 7f6f5445a700 1 -- 172.21.15.42:0/2271751087 shutdown_connections 2019-12-05T05:59:31.902 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:31.896+0000 7f6f5445a700 1 --2- 172.21.15.42:0/2271751087 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6f3403ace0 0x7f6f3403d1f0 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:59:31.902 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:31.896+0000 7f6f5445a700 1 --2- 172.21.15.42:0/2271751087 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6f4c1156c0 0x7f6f4c117e70 unknown :-1 s=CLOSED pgs=1361 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:59:31.902 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:31.896+0000 7f6f5445a700 1 --2- 172.21.15.42:0/2271751087 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6f4c1183c0 0x7f6f4c12a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:59:31.902 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:31.896+0000 7f6f5445a700 1 -- 172.21.15.42:0/2271751087 shutdown_connections 2019-12-05T05:59:31.903 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:31.896+0000 7f6f5445a700 1 -- 172.21.15.42:0/2271751087 wait complete. 2019-12-05T05:59:31.903 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:31.896+0000 7f6f5445a700 1 -- 172.21.15.42:0/2271751087 >> 172.21.15.42:0/2271751087 conn(0x7f6f4c08fdc0 msgr2=0x7f6f4c0a97e0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:59:31.912 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T05:59:33.914 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:59:34.042 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:34.028+0000 7fc0c9353700 1 Processor -- start 2019-12-05T05:59:34.043 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:34.028+0000 7fc0c9353700 1 -- start start 2019-12-05T05:59:34.043 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:34.032+0000 7fc0c9353700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc0c40a7740 0x7fc0c408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:59:34.043 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:34.032+0000 7fc0c9353700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc0c407e720 0x7fc0c407eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:59:34.043 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:34.032+0000 7fc0c9353700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0c40a0710 con 0x7fc0c409f2e0 2019-12-05T05:59:34.044 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:34.032+0000 7fc0c9353700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0c40a0b00 con 0x7fc0c40a7740 2019-12-05T05:59:34.044 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:34.032+0000 7fc0c9353700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0c408e5b0 con 0x7fc0c407e720 2019-12-05T05:59:34.044 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:34.032+0000 7fc0c2ffd700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc0c407e720 0x7fc0c407eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:59:34.047 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:34.032+0000 7fc0c37fe700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc0c40a7740 0x7fc0c408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:59:34.047 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:34.032+0000 7fc0c2ffd700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc0c407e720 0x7fc0c407eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:45930/0 (socket says 172.21.15.42:45930) 2019-12-05T05:59:34.047 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:34.032+0000 7fc0c2ffd700 1 -- 172.21.15.42:0/275145971 learned_addr learned my addr 172.21.15.42:0/275145971 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:59:34.048 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:34.032+0000 7fc0c3fff700 1 --1- 172.21.15.42:0/275145971 >> v1:172.21.15.40:6789/0 conn(0x7fc0c409f2e0 0x7fc0c409f710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:54946/0 (socket says 172.21.15.42:54946) 2019-12-05T05:59:34.048 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:34.032+0000 7fc0c2ffd700 1 -- 172.21.15.42:0/275145971 >> v2:172.21.15.40:3300/0 conn(0x7fc0c40a7740 msgr2=0x7fc0c408df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:59:34.048 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:34.032+0000 7fc0c2ffd700 1 --2- 172.21.15.42:0/275145971 >> v2:172.21.15.40:3300/0 conn(0x7fc0c40a7740 0x7fc0c408df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:59:34.048 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:34.032+0000 7fc0c2ffd700 1 -- 172.21.15.42:0/275145971 >> v1:172.21.15.40:6789/0 conn(0x7fc0c409f2e0 legacy=0x7fc0c409f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:59:34.049 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:34.032+0000 7fc0c2ffd700 1 -- 172.21.15.42:0/275145971 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0c407f0a0 con 0x7fc0c407e720 2019-12-05T05:59:34.049 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:34.032+0000 7fc0c37fe700 1 --2- 172.21.15.42:0/275145971 >> v2:172.21.15.40:3300/0 conn(0x7fc0c40a7740 0x7fc0c408df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:59:34.049 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:34.032+0000 7fc0c2ffd700 1 --2- 172.21.15.42:0/275145971 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc0c407e720 0x7fc0c407eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fc0b8008790 tx=0x7fc0b800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:34.049 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:34.032+0000 7fc0c2ffd700 1 --2- 172.21.15.42:0/275145971 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc0c407e720 0x7fc0c407eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fc0b8008790 tx=0x7fc0b800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:34.050 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:34.032+0000 7fc0c2ffd700 1 --2- 172.21.15.42:0/275145971 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc0c407e720 0x7fc0c407eb50 secure :-1 s=READY pgs=404 cs=0 l=1 rx=0x7fc0b8008790 tx=0x7fc0b800fd30).ready entity=mon.2 client_cookie=c18040b59032ebf0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:59:34.050 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:34.032+0000 7fc0c2ffd700 1 --2- 172.21.15.42:0/275145971 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc0c407e720 0x7fc0c407eb50 secure :-1 s=THROTTLE_DONE pgs=404 cs=0 l=1 rx=0x7fc0b8008790 tx=0x7fc0b800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:34.050 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:34.032+0000 7fc0c2ffd700 1 --2- 172.21.15.42:0/275145971 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc0c407e720 0x7fc0c407eb50 secure :-1 s=THROTTLE_DONE pgs=404 cs=0 l=1 rx=0x7fc0b8008790 tx=0x7fc0b800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:34.051 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:34.032+0000 7fc0c27fc700 1 -- 172.21.15.42:0/275145971 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc0b80113b0 con 0x7fc0c407e720 2019-12-05T05:59:34.051 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:34.032+0000 7fc0c2ffd700 1 --2- 172.21.15.42:0/275145971 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc0c407e720 0x7fc0c407eb50 secure :-1 s=THROTTLE_DONE pgs=404 cs=0 l=1 rx=0x7fc0b8008790 tx=0x7fc0b800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:34.051 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:34.032+0000 7fc0c27fc700 1 -- 172.21.15.42:0/275145971 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0b8008bb0 con 0x7fc0c407e720 2019-12-05T05:59:34.051 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:34.032+0000 7fc0c27fc700 1 -- 172.21.15.42:0/275145971 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc0b8011990 con 0x7fc0c407e720 2019-12-05T05:59:34.052 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:34.032+0000 7fc0c9353700 1 -- 172.21.15.42:0/275145971 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc0c407e720 msgr2=0x7fc0c407eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:59:34.052 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:34.032+0000 7fc0c9353700 1 --2- 172.21.15.42:0/275145971 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc0c407e720 0x7fc0c407eb50 secure :-1 s=READY pgs=404 cs=0 l=1 rx=0x7fc0b8008790 tx=0x7fc0b800fd30).stop 2019-12-05T05:59:34.052 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:34.036+0000 7fc0c9353700 1 -- 172.21.15.42:0/275145971 shutdown_connections 2019-12-05T05:59:34.053 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:34.036+0000 7fc0c9353700 1 --2- 172.21.15.42:0/275145971 >> v2:172.21.15.40:3300/0 conn(0x7fc0c40a7740 0x7fc0c408df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:59:34.053 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:34.036+0000 7fc0c9353700 1 --2- 172.21.15.42:0/275145971 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc0c407e720 0x7fc0c407eb50 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:59:34.053 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:34.036+0000 7fc0c9353700 1 -- 172.21.15.42:0/275145971 shutdown_connections 2019-12-05T05:59:34.054 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:34.036+0000 7fc0c9353700 1 -- 172.21.15.42:0/275145971 wait complete. 2019-12-05T05:59:34.054 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:34.036+0000 7fc0c9353700 1 -- 172.21.15.42:0/275145971 >> 172.21.15.42:0/275145971 conn(0x7fc0c40aabe0 msgr2=0x7fc0c40a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:59:34.075 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:34.036+0000 7fc0c9353700 1 Processor -- start 2019-12-05T05:59:34.075 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:34.036+0000 7fc0c9353700 1 -- start start 2019-12-05T05:59:34.076 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:34.040+0000 7fc0c9353700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc0c411bf30 0x7fc0c4141a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:59:34.076 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:34.040+0000 7fc0c9353700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc0c407bf30 0x7fc0c407c360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:59:34.076 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:34.040+0000 7fc0c9353700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0c4141f70 con 0x7fc0c4133960 2019-12-05T05:59:34.077 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:34.040+0000 7fc0c9353700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0c407ca00 con 0x7fc0c411bf30 2019-12-05T05:59:34.077 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:34.040+0000 7fc0c9353700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0c407cca0 con 0x7fc0c407bf30 2019-12-05T05:59:34.077 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:34.040+0000 7fc0c37fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc0c411bf30 0x7fc0c4141a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:59:34.077 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:34.040+0000 7fc0c37fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc0c411bf30 0x7fc0c4141a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:45470/0 (socket says 172.21.15.42:45470) 2019-12-05T05:59:34.078 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:34.040+0000 7fc0c37fe700 1 -- 172.21.15.42:0/3295288621 learned_addr learned my addr 172.21.15.42:0/3295288621 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:59:34.078 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:34.040+0000 7fc0c2ffd700 1 --2- 172.21.15.42:0/3295288621 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc0c407bf30 0x7fc0c407c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:59:34.078 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:34.040+0000 7fc0abfff700 1 -- 172.21.15.42:0/3295288621 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3374658205 0 0) 0x7fc0c4141f70 con 0x7fc0c4133960 2019-12-05T05:59:34.078 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:34.040+0000 7fc0abfff700 1 -- 172.21.15.42:0/3295288621 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0a40030b0 con 0x7fc0c4133960 2019-12-05T05:59:34.079 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:34.040+0000 7fc0c37fe700 1 -- 172.21.15.42:0/3295288621 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc0c407bf30 msgr2=0x7fc0c407c360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:59:34.079 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:34.040+0000 7fc0c37fe700 1 --2- 172.21.15.42:0/3295288621 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc0c407bf30 0x7fc0c407c360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:59:34.079 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:34.040+0000 7fc0c37fe700 1 -- 172.21.15.42:0/3295288621 >> v1:172.21.15.40:6789/0 conn(0x7fc0c4133960 legacy=0x7fc0c4133d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:59:34.079 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:34.040+0000 7fc0c37fe700 1 -- 172.21.15.42:0/3295288621 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0c41169a0 con 0x7fc0c411bf30 2019-12-05T05:59:34.079 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:34.040+0000 7fc0c2ffd700 1 --2- 172.21.15.42:0/3295288621 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc0c407bf30 0x7fc0c407c360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:59:34.080 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:34.040+0000 7fc0c37fe700 1 --2- 172.21.15.42:0/3295288621 >> v2:172.21.15.42:3300/0 conn(0x7fc0c411bf30 0x7fc0c4141a20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fc0c407d760 tx=0x7fc0b4010d60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:34.080 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:34.040+0000 7fc0c37fe700 1 --2- 172.21.15.42:0/3295288621 >> v2:172.21.15.42:3300/0 conn(0x7fc0c411bf30 0x7fc0c4141a20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fc0c407d760 tx=0x7fc0b4010d60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:34.080 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:34.040+0000 7fc0c37fe700 1 --2- 172.21.15.42:0/3295288621 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc0c411bf30 0x7fc0c4141a20 secure :-1 s=READY pgs=1788 cs=0 l=1 rx=0x7fc0c407d760 tx=0x7fc0b4010d60).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:59:34.081 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:34.040+0000 7fc0c37fe700 1 --2- 172.21.15.42:0/3295288621 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc0c411bf30 0x7fc0c4141a20 secure :-1 s=THROTTLE_DONE pgs=1788 cs=0 l=1 rx=0x7fc0c407d760 tx=0x7fc0b4010d60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:34.081 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:34.040+0000 7fc0abfff700 1 -- 172.21.15.42:0/3295288621 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc0b40126a0 con 0x7fc0c411bf30 2019-12-05T05:59:34.082 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:34.040+0000 7fc0c37fe700 1 --2- 172.21.15.42:0/3295288621 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc0c411bf30 0x7fc0c4141a20 secure :-1 s=THROTTLE_DONE pgs=1788 cs=0 l=1 rx=0x7fc0c407d760 tx=0x7fc0b4010d60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:34.082 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:34.040+0000 7fc0c37fe700 1 --2- 172.21.15.42:0/3295288621 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc0c411bf30 0x7fc0c4141a20 secure :-1 s=THROTTLE_DONE pgs=1788 cs=0 l=1 rx=0x7fc0c407d760 tx=0x7fc0b4010d60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:34.082 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:34.040+0000 7fc0abfff700 1 -- 172.21.15.42:0/3295288621 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0b4007150 con 0x7fc0c411bf30 2019-12-05T05:59:34.082 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:34.040+0000 7fc0c9353700 1 -- 172.21.15.42:0/3295288621 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0c40742d0 con 0x7fc0c411bf30 2019-12-05T05:59:34.083 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:34.040+0000 7fc0abfff700 1 -- 172.21.15.42:0/3295288621 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc0b4012cc0 con 0x7fc0c411bf30 2019-12-05T05:59:34.083 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:34.040+0000 7fc0c9353700 1 -- 172.21.15.42:0/3295288621 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0c4074820 con 0x7fc0c411bf30 2019-12-05T05:59:34.083 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:34.040+0000 7fc0c37fe700 1 --2- 172.21.15.42:0/3295288621 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc0c411bf30 0x7fc0c4141a20 secure :-1 s=THROTTLE_DONE pgs=1788 cs=0 l=1 rx=0x7fc0c407d760 tx=0x7fc0b4010d60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:34.083 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:34.044+0000 7fc0c37fe700 1 --2- 172.21.15.42:0/3295288621 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc0c411bf30 0x7fc0c4141a20 secure :-1 s=THROTTLE_DONE pgs=1788 cs=0 l=1 rx=0x7fc0c407d760 tx=0x7fc0b4010d60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:34.084 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:34.044+0000 7fc0abfff700 1 -- 172.21.15.42:0/3295288621 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fc0b4005540 con 0x7fc0c411bf30 2019-12-05T05:59:34.084 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:34.044+0000 7fc0abfff700 1 --2- 172.21.15.42:0/3295288621 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc0a403b260 0x7fc0a403d770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:59:34.084 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:34.044+0000 7fc0abfff700 1 -- 172.21.15.42:0/3295288621 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 9213+0+0 (secure 0 0 0) 0x7fc0b404b8f0 con 0x7fc0c411bf30 2019-12-05T05:59:34.086 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:34.044+0000 7fc0c2ffd700 1 --2- 172.21.15.42:0/3295288621 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc0a403b260 0x7fc0a403d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:59:34.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:34.044+0000 7fc0c2ffd700 1 --2- 172.21.15.42:0/3295288621 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc0a403b260 0x7fc0a403d770 crc :-1 s=READY pgs=763 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:59:34.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:34.044+0000 7fc0c9353700 1 -- 172.21.15.42:0/3295288621 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0c405d110 con 0x7fc0c411bf30 2019-12-05T05:59:34.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:34.052+0000 7fc0c37fe700 1 --2- 172.21.15.42:0/3295288621 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc0c411bf30 0x7fc0c4141a20 secure :-1 s=THROTTLE_DONE pgs=1788 cs=0 l=1 rx=0x7fc0c407d760 tx=0x7fc0b4010d60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:34.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:34.052+0000 7fc0abfff700 1 -- 172.21.15.42:0/3295288621 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fc0b4032b90 con 0x7fc0c411bf30 2019-12-05T05:59:34.216 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:34.212+0000 7fc0c9353700 1 -- 172.21.15.42:0/3295288621 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc0c405b8c0 con 0x7fc0c411bf30 2019-12-05T05:59:34.217 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:34.212+0000 7fc0c37fe700 1 --2- 172.21.15.42:0/3295288621 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc0c411bf30 0x7fc0c4141a20 secure :-1 s=THROTTLE_DONE pgs=1788 cs=0 l=1 rx=0x7fc0c407d760 tx=0x7fc0b4010d60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:34.217 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:34.212+0000 7fc0abfff700 1 -- 172.21.15.42:0/3295288621 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 218 v218) v1 ==== 95+0+2112 (secure 0 0 0) 0x7fc0b4033740 con 0x7fc0c411bf30 2019-12-05T05:59:34.217 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 218 2019-12-05T05:59:34.217 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:59:34.217 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":218,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":8940,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/1082666540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1082666540},{"type":"v1","addr":"172.21.15.40:6833","nonce":1082666540}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":218}],"filesystems":[{"mdsmap":{"epoch":217,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:57:29.289849+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8836},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8836":{"gid":8836,"name":"a","rank":0,"incarnation":214,"state":"up:active","state_seq":29,"addr":"172.21.15.42:6835/1368848584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1368848584},{"type":"v1","addr":"172.21.15.42:6835","nonce":1368848584}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:59:34.218 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:34.212+0000 7fc0c9353700 1 -- 172.21.15.42:0/3295288621 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc0a403b260 msgr2=0x7fc0a403d770 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:59:34.219 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:34.212+0000 7fc0c9353700 1 --2- 172.21.15.42:0/3295288621 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc0a403b260 0x7fc0a403d770 crc :-1 s=READY pgs=763 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:59:34.219 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:34.212+0000 7fc0c9353700 1 -- 172.21.15.42:0/3295288621 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc0c411bf30 msgr2=0x7fc0c4141a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:59:34.219 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:34.212+0000 7fc0c9353700 1 --2- 172.21.15.42:0/3295288621 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc0c411bf30 0x7fc0c4141a20 secure :-1 s=READY pgs=1788 cs=0 l=1 rx=0x7fc0c407d760 tx=0x7fc0b4010d60).stop 2019-12-05T05:59:34.222 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:34.216+0000 7fc0c9353700 1 -- 172.21.15.42:0/3295288621 shutdown_connections 2019-12-05T05:59:34.223 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:34.216+0000 7fc0c9353700 1 --2- 172.21.15.42:0/3295288621 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc0a403b260 0x7fc0a403d770 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:59:34.223 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:34.216+0000 7fc0c9353700 1 --2- 172.21.15.42:0/3295288621 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc0c411bf30 0x7fc0c4141a20 unknown :-1 s=CLOSED pgs=1788 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:59:34.223 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:34.216+0000 7fc0c9353700 1 --2- 172.21.15.42:0/3295288621 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc0c407bf30 0x7fc0c407c360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:59:34.223 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:34.216+0000 7fc0c9353700 1 -- 172.21.15.42:0/3295288621 shutdown_connections 2019-12-05T05:59:34.223 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:34.216+0000 7fc0c9353700 1 -- 172.21.15.42:0/3295288621 wait complete. 2019-12-05T05:59:34.223 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:34.216+0000 7fc0c9353700 1 -- 172.21.15.42:0/3295288621 >> 172.21.15.42:0/3295288621 conn(0x7fc0c40a6d70 msgr2=0x7fc0c4059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:59:34.234 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T05:59:36.235 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:59:36.343 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:36.328+0000 7fc309524700 1 Processor -- start 2019-12-05T05:59:36.343 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:36.328+0000 7fc309524700 1 -- start start 2019-12-05T05:59:36.344 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:36.332+0000 7fc309524700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc3040a7740 0x7fc30408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:59:36.344 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:36.332+0000 7fc309524700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc30407e720 0x7fc30407eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:59:36.344 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:36.332+0000 7fc309524700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3040a0710 con 0x7fc30409f2e0 2019-12-05T05:59:36.344 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:36.332+0000 7fc309524700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3040a0b00 con 0x7fc30407e720 2019-12-05T05:59:36.344 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:36.332+0000 7fc309524700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc30408e5b0 con 0x7fc3040a7740 2019-12-05T05:59:36.344 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:36.332+0000 7fc302ffd700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc30407e720 0x7fc30407eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:59:36.345 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:36.332+0000 7fc302ffd700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc30407e720 0x7fc30407eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:45482/0 (socket says 172.21.15.42:45482) 2019-12-05T05:59:36.345 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:36.332+0000 7fc302ffd700 1 -- 172.21.15.42:0/3684460889 learned_addr learned my addr 172.21.15.42:0/3684460889 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:59:36.345 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:36.332+0000 7fc3037fe700 1 --2- 172.21.15.42:0/3684460889 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc3040a7740 0x7fc30408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:59:36.345 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:36.332+0000 7fc302ffd700 1 -- 172.21.15.42:0/3684460889 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc3040a7740 msgr2=0x7fc30408df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:59:36.345 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:36.332+0000 7fc302ffd700 1 --2- 172.21.15.42:0/3684460889 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc3040a7740 0x7fc30408df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:59:36.345 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:36.332+0000 7fc302ffd700 1 -- 172.21.15.42:0/3684460889 >> v1:172.21.15.42:6789/0 conn(0x7fc30409f2e0 legacy=0x7fc30409f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:59:36.345 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:36.332+0000 7fc302ffd700 1 -- 172.21.15.42:0/3684460889 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc30407f0a0 con 0x7fc30407e720 2019-12-05T05:59:36.346 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:36.332+0000 7fc302ffd700 1 --2- 172.21.15.42:0/3684460889 >> v2:172.21.15.42:3300/0 conn(0x7fc30407e720 0x7fc30407eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fc2ec008790 tx=0x7fc2ec00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:36.346 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:36.332+0000 7fc302ffd700 1 --2- 172.21.15.42:0/3684460889 >> v2:172.21.15.42:3300/0 conn(0x7fc30407e720 0x7fc30407eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fc2ec008790 tx=0x7fc2ec00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:36.346 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:36.332+0000 7fc302ffd700 1 --2- 172.21.15.42:0/3684460889 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc30407e720 0x7fc30407eb50 secure :-1 s=READY pgs=1790 cs=0 l=1 rx=0x7fc2ec008790 tx=0x7fc2ec00fd30).ready entity=mon.1 client_cookie=3241488d6f001840 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:59:36.346 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:36.332+0000 7fc302ffd700 1 --2- 172.21.15.42:0/3684460889 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc30407e720 0x7fc30407eb50 secure :-1 s=THROTTLE_DONE pgs=1790 cs=0 l=1 rx=0x7fc2ec008790 tx=0x7fc2ec00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:36.346 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:36.332+0000 7fc302ffd700 1 --2- 172.21.15.42:0/3684460889 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc30407e720 0x7fc30407eb50 secure :-1 s=THROTTLE_DONE pgs=1790 cs=0 l=1 rx=0x7fc2ec008790 tx=0x7fc2ec00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:36.346 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:36.332+0000 7fc3027fc700 1 -- 172.21.15.42:0/3684460889 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc2ec0113b0 con 0x7fc30407e720 2019-12-05T05:59:36.346 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:36.332+0000 7fc302ffd700 1 --2- 172.21.15.42:0/3684460889 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc30407e720 0x7fc30407eb50 secure :-1 s=THROTTLE_DONE pgs=1790 cs=0 l=1 rx=0x7fc2ec008790 tx=0x7fc2ec00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:36.347 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:36.332+0000 7fc3027fc700 1 -- 172.21.15.42:0/3684460889 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2ec008bb0 con 0x7fc30407e720 2019-12-05T05:59:36.347 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:36.332+0000 7fc3027fc700 1 -- 172.21.15.42:0/3684460889 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc2ec011990 con 0x7fc30407e720 2019-12-05T05:59:36.347 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:36.332+0000 7fc309524700 1 -- 172.21.15.42:0/3684460889 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc30407e720 msgr2=0x7fc30407eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:59:36.347 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:36.332+0000 7fc309524700 1 --2- 172.21.15.42:0/3684460889 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc30407e720 0x7fc30407eb50 secure :-1 s=READY pgs=1790 cs=0 l=1 rx=0x7fc2ec008790 tx=0x7fc2ec00fd30).stop 2019-12-05T05:59:36.347 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:36.336+0000 7fc309524700 1 -- 172.21.15.42:0/3684460889 shutdown_connections 2019-12-05T05:59:36.347 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:36.336+0000 7fc309524700 1 --2- 172.21.15.42:0/3684460889 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc3040a7740 0x7fc30408df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:59:36.347 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:36.336+0000 7fc309524700 1 --2- 172.21.15.42:0/3684460889 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc30407e720 0x7fc30407eb50 unknown :-1 s=CLOSED pgs=1790 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:59:36.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:36.336+0000 7fc309524700 1 -- 172.21.15.42:0/3684460889 shutdown_connections 2019-12-05T05:59:36.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:36.336+0000 7fc309524700 1 -- 172.21.15.42:0/3684460889 wait complete. 2019-12-05T05:59:36.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:36.336+0000 7fc309524700 1 -- 172.21.15.42:0/3684460889 >> 172.21.15.42:0/3684460889 conn(0x7fc3040aabe0 msgr2=0x7fc3040a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:59:36.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:36.336+0000 7fc309524700 1 Processor -- start 2019-12-05T05:59:36.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:36.336+0000 7fc309524700 1 -- start start 2019-12-05T05:59:36.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:36.336+0000 7fc309524700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc30411ee90 0x7fc30411bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:59:36.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:36.336+0000 7fc309524700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc30412f2c0 0x7fc30412f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:59:36.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:36.336+0000 7fc309524700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc304117c00 con 0x7fc304118660 2019-12-05T05:59:36.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:36.336+0000 7fc309524700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3041197e0 con 0x7fc30411ee90 2019-12-05T05:59:36.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:36.336+0000 7fc309524700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc30411f2c0 con 0x7fc30412f2c0 2019-12-05T05:59:36.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:36.336+0000 7fc303fff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc30412f2c0 0x7fc30412f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:59:36.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:36.336+0000 7fc303fff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc30412f2c0 0x7fc30412f6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:45486/0 (socket says 172.21.15.42:45486) 2019-12-05T05:59:36.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:36.336+0000 7fc303fff700 1 -- 172.21.15.42:0/1795693653 learned_addr learned my addr 172.21.15.42:0/1795693653 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:59:36.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:36.336+0000 7fc302ffd700 1 --2- 172.21.15.42:0/1795693653 >> v2:172.21.15.40:3300/0 conn(0x7fc30411ee90 0x7fc30411bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:59:36.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:36.336+0000 7fc303fff700 1 -- 172.21.15.42:0/1795693653 >> v2:172.21.15.40:3300/0 conn(0x7fc30411ee90 msgr2=0x7fc30411bbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:59:36.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:36.336+0000 7fc303fff700 1 --2- 172.21.15.42:0/1795693653 >> v2:172.21.15.40:3300/0 conn(0x7fc30411ee90 0x7fc30411bbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:59:36.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:36.336+0000 7fc303fff700 1 -- 172.21.15.42:0/1795693653 >> v1:172.21.15.40:6789/0 conn(0x7fc304118660 legacy=0x7fc304117400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:59:36.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:36.336+0000 7fc303fff700 1 -- 172.21.15.42:0/1795693653 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc30412fc40 con 0x7fc30412f2c0 2019-12-05T05:59:36.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:36.336+0000 7fc303fff700 1 --2- 172.21.15.42:0/1795693653 >> v2:172.21.15.42:3300/0 conn(0x7fc30412f2c0 0x7fc30412f6f0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fc30411fd20 tx=0x7fc2fc010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:36.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:36.336+0000 7fc303fff700 1 --2- 172.21.15.42:0/1795693653 >> v2:172.21.15.42:3300/0 conn(0x7fc30412f2c0 0x7fc30412f6f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fc30411fd20 tx=0x7fc2fc010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:36.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:36.336+0000 7fc303fff700 1 --2- 172.21.15.42:0/1795693653 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc30412f2c0 0x7fc30412f6f0 secure :-1 s=READY pgs=1791 cs=0 l=1 rx=0x7fc30411fd20 tx=0x7fc2fc010b40).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:59:36.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:36.336+0000 7fc303fff700 1 --2- 172.21.15.42:0/1795693653 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc30412f2c0 0x7fc30412f6f0 secure :-1 s=THROTTLE_DONE pgs=1791 cs=0 l=1 rx=0x7fc30411fd20 tx=0x7fc2fc010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:36.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:36.336+0000 7fc2fbfff700 1 -- 172.21.15.42:0/1795693653 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc2fc006ac0 con 0x7fc30412f2c0 2019-12-05T05:59:36.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:36.340+0000 7fc303fff700 1 --2- 172.21.15.42:0/1795693653 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc30412f2c0 0x7fc30412f6f0 secure :-1 s=THROTTLE_DONE pgs=1791 cs=0 l=1 rx=0x7fc30411fd20 tx=0x7fc2fc010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:36.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:36.340+0000 7fc309524700 1 -- 172.21.15.42:0/1795693653 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc304077300 con 0x7fc30412f2c0 2019-12-05T05:59:36.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:36.340+0000 7fc2fbfff700 1 -- 172.21.15.42:0/1795693653 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2fc0066e0 con 0x7fc30412f2c0 2019-12-05T05:59:36.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:36.340+0000 7fc303fff700 1 --2- 172.21.15.42:0/1795693653 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc30412f2c0 0x7fc30412f6f0 secure :-1 s=THROTTLE_DONE pgs=1791 cs=0 l=1 rx=0x7fc30411fd20 tx=0x7fc2fc010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:36.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:36.340+0000 7fc2fbfff700 1 -- 172.21.15.42:0/1795693653 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc2fc002710 con 0x7fc30412f2c0 2019-12-05T05:59:36.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:36.340+0000 7fc309524700 1 -- 172.21.15.42:0/1795693653 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc304077800 con 0x7fc30412f2c0 2019-12-05T05:59:36.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:36.340+0000 7fc303fff700 1 --2- 172.21.15.42:0/1795693653 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc30412f2c0 0x7fc30412f6f0 secure :-1 s=THROTTLE_DONE pgs=1791 cs=0 l=1 rx=0x7fc30411fd20 tx=0x7fc2fc010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:36.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:36.340+0000 7fc2fbfff700 1 -- 172.21.15.42:0/1795693653 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fc2fc002a30 con 0x7fc30412f2c0 2019-12-05T05:59:36.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:36.340+0000 7fc303fff700 1 --2- 172.21.15.42:0/1795693653 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc30412f2c0 0x7fc30412f6f0 secure :-1 s=THROTTLE_DONE pgs=1791 cs=0 l=1 rx=0x7fc30411fd20 tx=0x7fc2fc010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:36.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:36.340+0000 7fc2fbfff700 1 --2- 172.21.15.42:0/1795693653 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc2e803ae00 0x7fc2e803d310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:59:36.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:36.340+0000 7fc2fbfff700 1 -- 172.21.15.42:0/1795693653 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 9213+0+0 (secure 0 0 0) 0x7fc2fc04b2b0 con 0x7fc30412f2c0 2019-12-05T05:59:36.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:36.340+0000 7fc3037fe700 1 --2- 172.21.15.42:0/1795693653 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc2e803ae00 0x7fc2e803d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:59:36.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:36.340+0000 7fc3037fe700 1 --2- 172.21.15.42:0/1795693653 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc2e803ae00 0x7fc2e803d310 crc :-1 s=READY pgs=764 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:59:36.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:36.348+0000 7fc309524700 1 -- 172.21.15.42:0/1795693653 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc30405d110 con 0x7fc30412f2c0 2019-12-05T05:59:36.360 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:36.356+0000 7fc303fff700 1 --2- 172.21.15.42:0/1795693653 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc30412f2c0 0x7fc30412f6f0 secure :-1 s=THROTTLE_DONE pgs=1791 cs=0 l=1 rx=0x7fc30411fd20 tx=0x7fc2fc010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:36.360 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:36.356+0000 7fc2fbfff700 1 -- 172.21.15.42:0/1795693653 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fc2fc032310 con 0x7fc30412f2c0 2019-12-05T05:59:36.530 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:36.524+0000 7fc309524700 1 -- 172.21.15.42:0/1795693653 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc30405b8c0 con 0x7fc30412f2c0 2019-12-05T05:59:36.536 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:36.524+0000 7fc303fff700 1 --2- 172.21.15.42:0/1795693653 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc30412f2c0 0x7fc30412f6f0 secure :-1 s=THROTTLE_DONE pgs=1791 cs=0 l=1 rx=0x7fc30411fd20 tx=0x7fc2fc010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:36.536 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:36.524+0000 7fc2fbfff700 1 -- 172.21.15.42:0/1795693653 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 218 v218) v1 ==== 95+0+2112 (secure 0 0 0) 0x7fc2fc032570 con 0x7fc30412f2c0 2019-12-05T05:59:36.536 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 218 2019-12-05T05:59:36.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:36.528+0000 7fc309524700 1 -- 172.21.15.42:0/1795693653 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc2e803ae00 msgr2=0x7fc2e803d310 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:59:36.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:36.528+0000 7fc309524700 1 --2- 172.21.15.42:0/1795693653 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc2e803ae00 0x7fc2e803d310 crc :-1 s=READY pgs=764 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:59:36.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:36.528+0000 7fc309524700 1 -- 172.21.15.42:0/1795693653 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc30412f2c0 msgr2=0x7fc30412f6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:59:36.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:36.528+0000 7fc309524700 1 --2- 172.21.15.42:0/1795693653 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc30412f2c0 0x7fc30412f6f0 secure :-1 s=READY pgs=1791 cs=0 l=1 rx=0x7fc30411fd20 tx=0x7fc2fc010b40).stop 2019-12-05T05:59:36.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:36.528+0000 7fc309524700 1 -- 172.21.15.42:0/1795693653 shutdown_connections 2019-12-05T05:59:36.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:36.528+0000 7fc309524700 1 --2- 172.21.15.42:0/1795693653 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc2e803ae00 0x7fc2e803d310 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:59:36.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:36.528+0000 7fc309524700 1 --2- 172.21.15.42:0/1795693653 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc30412f2c0 0x7fc30412f6f0 unknown :-1 s=CLOSED pgs=1791 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:59:36.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:36.528+0000 7fc309524700 1 --2- 172.21.15.42:0/1795693653 >> v2:172.21.15.40:3300/0 conn(0x7fc30411ee90 0x7fc30411bbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:59:36.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:36.528+0000 7fc309524700 1 -- 172.21.15.42:0/1795693653 shutdown_connections 2019-12-05T05:59:36.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:36.528+0000 7fc309524700 1 -- 172.21.15.42:0/1795693653 wait complete. 2019-12-05T05:59:36.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:36.528+0000 7fc309524700 1 -- 172.21.15.42:0/1795693653 >> 172.21.15.42:0/1795693653 conn(0x7fc3040a9870 msgr2=0x7fc30411d8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:59:36.538 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:59:36.538 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":218,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":8940,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/1082666540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1082666540},{"type":"v1","addr":"172.21.15.40:6833","nonce":1082666540}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":218}],"filesystems":[{"mdsmap":{"epoch":217,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:57:29.289849+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8836},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8836":{"gid":8836,"name":"a","rank":0,"incarnation":214,"state":"up:active","state_seq":29,"addr":"172.21.15.42:6835/1368848584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1368848584},{"type":"v1","addr":"172.21.15.42:6835","nonce":1368848584}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:59:36.557 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T05:59:38.558 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:59:38.668 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:38.656+0000 7f74b2b82700 1 Processor -- start 2019-12-05T05:59:38.669 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:38.656+0000 7f74b2b82700 1 -- start start 2019-12-05T05:59:38.669 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:38.660+0000 7f74b2b82700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f74ac0a7740 0x7f74ac08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:59:38.669 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:38.660+0000 7f74b2b82700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74ac0a0710 con 0x7f74ac07e720 2019-12-05T05:59:38.669 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:38.660+0000 7f74b2b82700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74ac0a9ba0 con 0x7f74ac09fae0 2019-12-05T05:59:38.669 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:38.660+0000 7f74b2b82700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74ac0a0b00 con 0x7f74ac0a7740 2019-12-05T05:59:38.670 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:38.660+0000 7f74b111f700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f74ac0a7740 0x7f74ac08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:59:38.670 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:38.660+0000 7f74b111f700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f74ac0a7740 0x7f74ac08df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:45496/0 (socket says 172.21.15.42:45496) 2019-12-05T05:59:38.670 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:38.660+0000 7f74b111f700 1 -- 172.21.15.42:0/193706431 learned_addr learned my addr 172.21.15.42:0/193706431 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:59:38.671 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:38.660+0000 7f74a3fff700 1 -- 172.21.15.42:0/193706431 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1782553201 0 0) 0x7f74ac0a0710 con 0x7f74ac07e720 2019-12-05T05:59:38.671 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:38.660+0000 7f74b111f700 1 -- 172.21.15.42:0/193706431 >> v1:172.21.15.42:6789/0 conn(0x7f74ac09fae0 legacy=0x7f74ac14efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:59:38.671 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:38.660+0000 7f74b111f700 1 -- 172.21.15.42:0/193706431 >> v1:172.21.15.40:6789/0 conn(0x7f74ac07e720 legacy=0x7f74ac09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:59:38.671 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:38.660+0000 7f74b111f700 1 -- 172.21.15.42:0/193706431 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74ac07eb50 con 0x7f74ac0a7740 2019-12-05T05:59:38.672 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:38.660+0000 7f74b111f700 1 --2- 172.21.15.42:0/193706431 >> v2:172.21.15.42:3300/0 conn(0x7f74ac0a7740 0x7f74ac08df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f74a8008790 tx=0x7f74a800fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:38.672 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:38.660+0000 7f74b111f700 1 --2- 172.21.15.42:0/193706431 >> v2:172.21.15.42:3300/0 conn(0x7f74ac0a7740 0x7f74ac08df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f74a8008790 tx=0x7f74a800fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:38.672 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:38.660+0000 7f74b111f700 1 --2- 172.21.15.42:0/193706431 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f74ac0a7740 0x7f74ac08df80 secure :-1 s=READY pgs=1793 cs=0 l=1 rx=0x7f74a8008790 tx=0x7f74a800fcb0).ready entity=mon.1 client_cookie=7d0dc18cb006e247 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:59:38.672 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:38.660+0000 7f74b111f700 1 --2- 172.21.15.42:0/193706431 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f74ac0a7740 0x7f74ac08df80 secure :-1 s=THROTTLE_DONE pgs=1793 cs=0 l=1 rx=0x7f74a8008790 tx=0x7f74a800fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:38.672 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:38.660+0000 7f74a3fff700 1 -- 172.21.15.42:0/193706431 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f74a8011520 con 0x7f74ac0a7740 2019-12-05T05:59:38.673 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:38.660+0000 7f74b111f700 1 --2- 172.21.15.42:0/193706431 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f74ac0a7740 0x7f74ac08df80 secure :-1 s=THROTTLE_DONE pgs=1793 cs=0 l=1 rx=0x7f74a8008790 tx=0x7f74a800fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:38.681 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:38.660+0000 7f74a3fff700 1 -- 172.21.15.42:0/193706431 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74a8008ca0 con 0x7f74ac0a7740 2019-12-05T05:59:38.682 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:38.660+0000 7f74b111f700 1 --2- 172.21.15.42:0/193706431 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f74ac0a7740 0x7f74ac08df80 secure :-1 s=THROTTLE_DONE pgs=1793 cs=0 l=1 rx=0x7f74a8008790 tx=0x7f74a800fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:38.682 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:38.660+0000 7f74a3fff700 1 -- 172.21.15.42:0/193706431 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f74a8011b00 con 0x7f74ac0a7740 2019-12-05T05:59:38.682 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:38.660+0000 7f74b2b82700 1 -- 172.21.15.42:0/193706431 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f74ac0a7740 msgr2=0x7f74ac08df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:59:38.682 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:38.660+0000 7f74b2b82700 1 --2- 172.21.15.42:0/193706431 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f74ac0a7740 0x7f74ac08df80 secure :-1 s=READY pgs=1793 cs=0 l=1 rx=0x7f74a8008790 tx=0x7f74a800fcb0).stop 2019-12-05T05:59:38.683 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:38.664+0000 7f74b2b82700 1 -- 172.21.15.42:0/193706431 shutdown_connections 2019-12-05T05:59:38.684 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:38.664+0000 7f74b2b82700 1 --2- 172.21.15.42:0/193706431 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f74ac0a7740 0x7f74ac08df80 unknown :-1 s=CLOSED pgs=1793 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:59:38.685 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:38.664+0000 7f74b2b82700 1 -- 172.21.15.42:0/193706431 shutdown_connections 2019-12-05T05:59:38.685 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:38.664+0000 7f74b2b82700 1 -- 172.21.15.42:0/193706431 wait complete. 2019-12-05T05:59:38.685 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:38.664+0000 7f74b2b82700 1 -- 172.21.15.42:0/193706431 >> 172.21.15.42:0/193706431 conn(0x7f74ac0aabe0 msgr2=0x7f74ac0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:59:38.688 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:38.664+0000 7f74b2b82700 1 Processor -- start 2019-12-05T05:59:38.689 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:38.664+0000 7f74b2b82700 1 -- start start 2019-12-05T05:59:38.689 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:38.664+0000 7f74b2b82700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f74ac07bb10 0x7f74ac1189d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:59:38.689 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:38.668+0000 7f74b2b82700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74ac11a0f0 con 0x7f74ac119250 2019-12-05T05:59:38.689 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:38.668+0000 7f74b2b82700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74ac119ab0 con 0x7f74ac119680 2019-12-05T05:59:38.689 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:38.668+0000 7f74b2b82700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74ac11c530 con 0x7f74ac07bb10 2019-12-05T05:59:38.689 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:38.668+0000 7f74b1920700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f74ac119680 0x7f74ac08f370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:45542/0 (socket says 172.21.15.42:45542) 2019-12-05T05:59:38.705 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:38.668+0000 7f74b1920700 1 -- 172.21.15.42:0/1397908246 learned_addr learned my addr 172.21.15.42:0/1397908246 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:59:38.705 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:38.668+0000 7f74b111f700 1 --2- 172.21.15.42:0/1397908246 >> v2:172.21.15.40:3300/0 conn(0x7f74ac07bb10 0x7f74ac1189d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:59:38.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:38.668+0000 7f74a1ffb700 1 -- 172.21.15.42:0/1397908246 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 564646752 0 0) 0x7f74ac119ab0 con 0x7f74ac119680 2019-12-05T05:59:38.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:38.668+0000 7f74a1ffb700 1 -- 172.21.15.42:0/1397908246 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f748c0030b0 con 0x7f74ac119680 2019-12-05T05:59:38.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:38.668+0000 7f74b111f700 1 -- 172.21.15.42:0/1397908246 >> v1:172.21.15.42:6789/0 conn(0x7f74ac119680 legacy=0x7f74ac08f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:59:38.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:38.668+0000 7f74b111f700 1 -- 172.21.15.42:0/1397908246 >> v1:172.21.15.40:6789/0 conn(0x7f74ac119250 legacy=0x7f74ac08eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:59:38.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:38.668+0000 7f74a1ffb700 1 -- 172.21.15.42:0/1397908246 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1277435524 0 0) 0x7f74ac11a0f0 con 0x7f74ac119250 2019-12-05T05:59:38.708 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:38.668+0000 7f74b111f700 1 -- 172.21.15.42:0/1397908246 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74ac12f950 con 0x7f74ac07bb10 2019-12-05T05:59:38.708 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:38.668+0000 7f74b111f700 1 --2- 172.21.15.42:0/1397908246 >> v2:172.21.15.40:3300/0 conn(0x7f74ac07bb10 0x7f74ac1189d0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f74ac118fb0 tx=0x7f74a8008f50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:38.708 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:38.668+0000 7f74b111f700 1 --2- 172.21.15.42:0/1397908246 >> v2:172.21.15.40:3300/0 conn(0x7f74ac07bb10 0x7f74ac1189d0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f74ac118fb0 tx=0x7f74a8008f50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:38.708 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:38.668+0000 7f74b111f700 1 --2- 172.21.15.42:0/1397908246 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f74ac07bb10 0x7f74ac1189d0 secure :-1 s=READY pgs=1368 cs=0 l=1 rx=0x7f74ac118fb0 tx=0x7f74a8008f50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:59:38.709 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:38.668+0000 7f74b111f700 1 --2- 172.21.15.42:0/1397908246 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f74ac07bb10 0x7f74ac1189d0 secure :-1 s=THROTTLE_DONE pgs=1368 cs=0 l=1 rx=0x7f74ac118fb0 tx=0x7f74a8008f50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:38.709 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:38.668+0000 7f74a1ffb700 1 -- 172.21.15.42:0/1397908246 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f74a8007340 con 0x7f74ac07bb10 2019-12-05T05:59:38.709 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:38.668+0000 7f74b111f700 1 --2- 172.21.15.42:0/1397908246 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f74ac07bb10 0x7f74ac1189d0 secure :-1 s=THROTTLE_DONE pgs=1368 cs=0 l=1 rx=0x7f74ac118fb0 tx=0x7f74a8008f50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:38.709 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:38.668+0000 7f74b111f700 1 --2- 172.21.15.42:0/1397908246 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f74ac07bb10 0x7f74ac1189d0 secure :-1 s=THROTTLE_DONE pgs=1368 cs=0 l=1 rx=0x7f74ac118fb0 tx=0x7f74a8008f50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:38.709 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:38.668+0000 7f74a1ffb700 1 -- 172.21.15.42:0/1397908246 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74a8011720 con 0x7f74ac07bb10 2019-12-05T05:59:38.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:38.668+0000 7f74b2b82700 1 -- 172.21.15.42:0/1397908246 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74ac12d930 con 0x7f74ac07bb10 2019-12-05T05:59:38.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:38.668+0000 7f74a1ffb700 1 -- 172.21.15.42:0/1397908246 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f74a8005540 con 0x7f74ac07bb10 2019-12-05T05:59:38.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:38.668+0000 7f74b2b82700 1 -- 172.21.15.42:0/1397908246 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74ac12deb0 con 0x7f74ac07bb10 2019-12-05T05:59:38.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:38.668+0000 7f74b111f700 1 --2- 172.21.15.42:0/1397908246 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f74ac07bb10 0x7f74ac1189d0 secure :-1 s=THROTTLE_DONE pgs=1368 cs=0 l=1 rx=0x7f74ac118fb0 tx=0x7f74a8008f50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:38.711 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:38.672+0000 7f74b111f700 1 --2- 172.21.15.42:0/1397908246 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f74ac07bb10 0x7f74ac1189d0 secure :-1 s=THROTTLE_DONE pgs=1368 cs=0 l=1 rx=0x7f74ac118fb0 tx=0x7f74a8008f50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:38.711 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:38.672+0000 7f74a1ffb700 1 -- 172.21.15.42:0/1397908246 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f74a8005790 con 0x7f74ac07bb10 2019-12-05T05:59:38.711 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:38.672+0000 7f74a1ffb700 1 --2- 172.21.15.42:0/1397908246 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f748c03b230 0x7f748c03d740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:59:38.711 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:38.672+0000 7f74a1ffb700 1 -- 172.21.15.42:0/1397908246 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 9213+0+0 (secure 0 0 0) 0x7f74a804b6a0 con 0x7f74ac07bb10 2019-12-05T05:59:38.712 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:38.672+0000 7f74b091e700 1 --2- 172.21.15.42:0/1397908246 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f748c03b230 0x7f748c03d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:59:38.712 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:38.672+0000 7f74b091e700 1 --2- 172.21.15.42:0/1397908246 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f748c03b230 0x7f748c03d740 crc :-1 s=READY pgs=765 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:59:38.712 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:38.676+0000 7f74b2b82700 1 -- 172.21.15.42:0/1397908246 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74ac05d110 con 0x7f74ac07bb10 2019-12-05T05:59:38.713 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:38.684+0000 7f74b111f700 1 --2- 172.21.15.42:0/1397908246 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f74ac07bb10 0x7f74ac1189d0 secure :-1 s=THROTTLE_DONE pgs=1368 cs=0 l=1 rx=0x7f74ac118fb0 tx=0x7f74a8008f50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:38.713 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:38.684+0000 7f74a1ffb700 1 -- 172.21.15.42:0/1397908246 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f74a80326e0 con 0x7f74ac07bb10 2019-12-05T05:59:38.860 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:38.856+0000 7f74b2b82700 1 -- 172.21.15.42:0/1397908246 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f74ac05b8c0 con 0x7f74ac07bb10 2019-12-05T05:59:38.861 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:38.856+0000 7f74b111f700 1 --2- 172.21.15.42:0/1397908246 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f74ac07bb10 0x7f74ac1189d0 secure :-1 s=THROTTLE_DONE pgs=1368 cs=0 l=1 rx=0x7f74ac118fb0 tx=0x7f74a8008f50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:38.861 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:38.856+0000 7f74a1ffb700 1 -- 172.21.15.42:0/1397908246 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 218 v218) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f74a8032940 con 0x7f74ac07bb10 2019-12-05T05:59:38.861 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 218 2019-12-05T05:59:38.862 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:59:38.862 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":218,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":8940,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/1082666540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1082666540},{"type":"v1","addr":"172.21.15.40:6833","nonce":1082666540}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":218}],"filesystems":[{"mdsmap":{"epoch":217,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:57:29.289849+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8836},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8836":{"gid":8836,"name":"a","rank":0,"incarnation":214,"state":"up:active","state_seq":29,"addr":"172.21.15.42:6835/1368848584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1368848584},{"type":"v1","addr":"172.21.15.42:6835","nonce":1368848584}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:59:38.864 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:38.860+0000 7f74b2b82700 1 -- 172.21.15.42:0/1397908246 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f748c03b230 msgr2=0x7f748c03d740 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:59:38.864 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:38.860+0000 7f74b2b82700 1 --2- 172.21.15.42:0/1397908246 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f748c03b230 0x7f748c03d740 crc :-1 s=READY pgs=765 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:59:38.864 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:38.860+0000 7f74b2b82700 1 -- 172.21.15.42:0/1397908246 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f74ac07bb10 msgr2=0x7f74ac1189d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:59:38.865 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:38.860+0000 7f74b2b82700 1 --2- 172.21.15.42:0/1397908246 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f74ac07bb10 0x7f74ac1189d0 secure :-1 s=READY pgs=1368 cs=0 l=1 rx=0x7f74ac118fb0 tx=0x7f74a8008f50).stop 2019-12-05T05:59:38.865 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:38.860+0000 7f74b2b82700 1 -- 172.21.15.42:0/1397908246 shutdown_connections 2019-12-05T05:59:38.865 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:38.860+0000 7f74b2b82700 1 --2- 172.21.15.42:0/1397908246 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f748c03b230 0x7f748c03d740 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:59:38.865 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:38.860+0000 7f74b2b82700 1 --2- 172.21.15.42:0/1397908246 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f74ac07bb10 0x7f74ac1189d0 unknown :-1 s=CLOSED pgs=1368 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:59:38.865 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:38.860+0000 7f74b2b82700 1 -- 172.21.15.42:0/1397908246 shutdown_connections 2019-12-05T05:59:38.865 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:38.860+0000 7f74b2b82700 1 -- 172.21.15.42:0/1397908246 wait complete. 2019-12-05T05:59:38.866 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:38.860+0000 7f74b2b82700 1 -- 172.21.15.42:0/1397908246 >> 172.21.15.42:0/1397908246 conn(0x7f74ac08fda0 msgr2=0x7f74ac09f460 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:59:38.877 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T05:59:40.880 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:59:40.993 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:40.984+0000 7fbf1607d700 1 Processor -- start 2019-12-05T05:59:40.994 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:40.984+0000 7fbf1607d700 1 -- start start 2019-12-05T05:59:40.994 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:40.984+0000 7fbf1607d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fbf1008df80 0x7fbf1009f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:59:40.994 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:40.984+0000 7fbf1607d700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf1009fc60 con 0x7fbf1009f830 2019-12-05T05:59:40.994 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:40.984+0000 7fbf1607d700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf100a0710 con 0x7fbf100a7740 2019-12-05T05:59:40.994 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:40.984+0000 7fbf1607d700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf100a0b00 con 0x7fbf1008df80 2019-12-05T05:59:40.995 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:40.984+0000 7fbf0f7fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fbf1008df80 0x7fbf1009f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:59:40.995 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:40.984+0000 7fbf0ffff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fbf100a7740 0x7fbf1007e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:45548/0 (socket says 172.21.15.42:45548) 2019-12-05T05:59:40.995 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:40.984+0000 7fbf0f7fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fbf1008df80 0x7fbf1009f2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:45512/0 (socket says 172.21.15.42:45512) 2019-12-05T05:59:40.995 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:40.984+0000 7fbf0ffff700 1 -- 172.21.15.42:0/2554290233 learned_addr learned my addr 172.21.15.42:0/2554290233 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:59:40.995 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:40.984+0000 7fbf0f7fe700 1 -- 172.21.15.42:0/2554290233 >> v1:172.21.15.42:6789/0 conn(0x7fbf100a7740 legacy=0x7fbf1007e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:59:40.995 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:40.984+0000 7fbf0f7fe700 1 -- 172.21.15.42:0/2554290233 >> v1:172.21.15.40:6789/0 conn(0x7fbf1009f830 legacy=0x7fbf1014efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:59:40.995 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:40.988+0000 7fbf0f7fe700 1 -- 172.21.15.42:0/2554290233 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf1007ef20 con 0x7fbf1008df80 2019-12-05T05:59:40.996 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:40.988+0000 7fbf0f7fe700 1 --2- 172.21.15.42:0/2554290233 >> v2:172.21.15.42:3300/0 conn(0x7fbf1008df80 0x7fbf1009f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fbf04008790 tx=0x7fbf0400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:40.996 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:40.988+0000 7fbf0f7fe700 1 --2- 172.21.15.42:0/2554290233 >> v2:172.21.15.42:3300/0 conn(0x7fbf1008df80 0x7fbf1009f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fbf04008790 tx=0x7fbf0400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:40.996 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:40.988+0000 7fbf0f7fe700 1 --2- 172.21.15.42:0/2554290233 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbf1008df80 0x7fbf1009f2e0 secure :-1 s=READY pgs=1796 cs=0 l=1 rx=0x7fbf04008790 tx=0x7fbf0400fbf0).ready entity=mon.1 client_cookie=fdf5a4d99c610ba7 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:59:40.996 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:40.988+0000 7fbf0f7fe700 1 --2- 172.21.15.42:0/2554290233 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbf1008df80 0x7fbf1009f2e0 secure :-1 s=THROTTLE_DONE pgs=1796 cs=0 l=1 rx=0x7fbf04008790 tx=0x7fbf0400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:40.996 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:40.988+0000 7fbf0f7fe700 1 --2- 172.21.15.42:0/2554290233 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbf1008df80 0x7fbf1009f2e0 secure :-1 s=THROTTLE_DONE pgs=1796 cs=0 l=1 rx=0x7fbf04008790 tx=0x7fbf0400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:40.997 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:40.988+0000 7fbf0effd700 1 -- 172.21.15.42:0/2554290233 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbf04011270 con 0x7fbf1008df80 2019-12-05T05:59:40.997 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:40.988+0000 7fbf0f7fe700 1 --2- 172.21.15.42:0/2554290233 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbf1008df80 0x7fbf1009f2e0 secure :-1 s=THROTTLE_DONE pgs=1796 cs=0 l=1 rx=0x7fbf04008790 tx=0x7fbf0400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:41.005 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:40.988+0000 7fbf0effd700 1 -- 172.21.15.42:0/2554290233 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf04008bc0 con 0x7fbf1008df80 2019-12-05T05:59:41.005 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:40.988+0000 7fbf0effd700 1 -- 172.21.15.42:0/2554290233 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbf04011850 con 0x7fbf1008df80 2019-12-05T05:59:41.005 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:40.988+0000 7fbf1607d700 1 -- 172.21.15.42:0/2554290233 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbf1008df80 msgr2=0x7fbf1009f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:59:41.005 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:40.988+0000 7fbf1607d700 1 --2- 172.21.15.42:0/2554290233 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbf1008df80 0x7fbf1009f2e0 secure :-1 s=READY pgs=1796 cs=0 l=1 rx=0x7fbf04008790 tx=0x7fbf0400fbf0).stop 2019-12-05T05:59:41.005 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:40.988+0000 7fbf1607d700 1 -- 172.21.15.42:0/2554290233 shutdown_connections 2019-12-05T05:59:41.006 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:40.988+0000 7fbf1607d700 1 --2- 172.21.15.42:0/2554290233 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbf1008df80 0x7fbf1009f2e0 unknown :-1 s=CLOSED pgs=1796 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:59:41.006 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:40.988+0000 7fbf1607d700 1 -- 172.21.15.42:0/2554290233 shutdown_connections 2019-12-05T05:59:41.006 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:40.988+0000 7fbf1607d700 1 -- 172.21.15.42:0/2554290233 wait complete. 2019-12-05T05:59:41.006 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:40.988+0000 7fbf1607d700 1 -- 172.21.15.42:0/2554290233 >> 172.21.15.42:0/2554290233 conn(0x7fbf100aabe0 msgr2=0x7fbf100a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:59:41.014 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:40.992+0000 7fbf1607d700 1 Processor -- start 2019-12-05T05:59:41.014 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:40.992+0000 7fbf1607d700 1 -- start start 2019-12-05T05:59:41.015 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:40.992+0000 7fbf1607d700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbf1011beb0 0x7fbf100a5600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:59:41.015 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:40.992+0000 7fbf1607d700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf100647e0 con 0x7fbf100a4840 2019-12-05T05:59:41.015 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:40.992+0000 7fbf1607d700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf100a52c0 con 0x7fbf100a5b50 2019-12-05T05:59:41.015 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:40.992+0000 7fbf1607d700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf10128b60 con 0x7fbf1011beb0 2019-12-05T05:59:41.015 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:40.992+0000 7fbf14e1b700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fbf100a5b50 0x7fbf1012ffb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:45558/0 (socket says 172.21.15.42:45558) 2019-12-05T05:59:41.015 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:40.992+0000 7fbf14e1b700 1 -- 172.21.15.42:0/3142134089 learned_addr learned my addr 172.21.15.42:0/3142134089 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:59:41.016 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:40.992+0000 7fbf0f7fe700 1 --2- 172.21.15.42:0/3142134089 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbf1011beb0 0x7fbf100a5600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:59:41.016 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:40.992+0000 7fbf0cff9700 1 -- 172.21.15.42:0/3142134089 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 109813564 0 0) 0x7fbf100a52c0 con 0x7fbf100a5b50 2019-12-05T05:59:41.016 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:40.992+0000 7fbf0cff9700 1 -- 172.21.15.42:0/3142134089 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbef00030b0 con 0x7fbf100a5b50 2019-12-05T05:59:41.016 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:40.992+0000 7fbf0cff9700 1 -- 172.21.15.42:0/3142134089 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1285302325 0 0) 0x7fbf100647e0 con 0x7fbf100a4840 2019-12-05T05:59:41.016 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:40.992+0000 7fbf0cff9700 1 -- 172.21.15.42:0/3142134089 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf100a52c0 con 0x7fbf100a4840 2019-12-05T05:59:41.016 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:40.992+0000 7fbf0f7fe700 1 -- 172.21.15.42:0/3142134089 >> v1:172.21.15.42:6789/0 conn(0x7fbf100a5b50 legacy=0x7fbf1012ffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:59:41.017 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:40.992+0000 7fbf0f7fe700 1 -- 172.21.15.42:0/3142134089 >> v1:172.21.15.40:6789/0 conn(0x7fbf100a4840 legacy=0x7fbf10129290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:59:41.017 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:40.992+0000 7fbf0f7fe700 1 -- 172.21.15.42:0/3142134089 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf100a5f80 con 0x7fbf1011beb0 2019-12-05T05:59:41.017 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:40.992+0000 7fbf0f7fe700 1 --2- 172.21.15.42:0/3142134089 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbf1011beb0 0x7fbf100a5600 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fbf04008790 tx=0x7fbf04011d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:41.017 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:40.992+0000 7fbf0f7fe700 1 --2- 172.21.15.42:0/3142134089 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbf1011beb0 0x7fbf100a5600 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fbf04008790 tx=0x7fbf04011d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:41.017 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:40.992+0000 7fbf0f7fe700 1 --2- 172.21.15.42:0/3142134089 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbf1011beb0 0x7fbf100a5600 secure :-1 s=READY pgs=405 cs=0 l=1 rx=0x7fbf04008790 tx=0x7fbf04011d50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:59:41.018 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:40.992+0000 7fbf0f7fe700 1 --2- 172.21.15.42:0/3142134089 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbf1011beb0 0x7fbf100a5600 secure :-1 s=THROTTLE_DONE pgs=405 cs=0 l=1 rx=0x7fbf04008790 tx=0x7fbf04011d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:41.018 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:40.992+0000 7fbf0cff9700 1 -- 172.21.15.42:0/3142134089 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbf04007150 con 0x7fbf1011beb0 2019-12-05T05:59:41.019 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:40.992+0000 7fbf0f7fe700 1 --2- 172.21.15.42:0/3142134089 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbf1011beb0 0x7fbf100a5600 secure :-1 s=THROTTLE_DONE pgs=405 cs=0 l=1 rx=0x7fbf04008790 tx=0x7fbf04011d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:41.019 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:40.992+0000 7fbf0f7fe700 1 --2- 172.21.15.42:0/3142134089 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbf1011beb0 0x7fbf100a5600 secure :-1 s=THROTTLE_DONE pgs=405 cs=0 l=1 rx=0x7fbf04008790 tx=0x7fbf04011d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:41.019 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:40.992+0000 7fbf0cff9700 1 -- 172.21.15.42:0/3142134089 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf04011470 con 0x7fbf1011beb0 2019-12-05T05:59:41.022 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:40.992+0000 7fbf1607d700 1 -- 172.21.15.42:0/3142134089 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf100a1a20 con 0x7fbf1011beb0 2019-12-05T05:59:41.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:40.996+0000 7fbf0cff9700 1 -- 172.21.15.42:0/3142134089 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbf04005730 con 0x7fbf1011beb0 2019-12-05T05:59:41.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:40.996+0000 7fbf1607d700 1 -- 172.21.15.42:0/3142134089 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf101317c0 con 0x7fbf1011beb0 2019-12-05T05:59:41.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:40.996+0000 7fbf0f7fe700 1 --2- 172.21.15.42:0/3142134089 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbf1011beb0 0x7fbf100a5600 secure :-1 s=THROTTLE_DONE pgs=405 cs=0 l=1 rx=0x7fbf04008790 tx=0x7fbf04011d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:41.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:40.996+0000 7fbf0cff9700 1 -- 172.21.15.42:0/3142134089 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fbf04005a00 con 0x7fbf1011beb0 2019-12-05T05:59:41.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:40.996+0000 7fbf0f7fe700 1 --2- 172.21.15.42:0/3142134089 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbf1011beb0 0x7fbf100a5600 secure :-1 s=THROTTLE_DONE pgs=405 cs=0 l=1 rx=0x7fbf04008790 tx=0x7fbf04011d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:41.024 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:40.996+0000 7fbf0cff9700 1 --2- 172.21.15.42:0/3142134089 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbef003b410 0x7fbef003d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:59:41.024 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:40.996+0000 7fbf0cff9700 1 -- 172.21.15.42:0/3142134089 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 9213+0+0 (secure 0 0 0) 0x7fbf0404bc40 con 0x7fbf1011beb0 2019-12-05T05:59:41.024 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:40.996+0000 7fbf0ffff700 1 --2- 172.21.15.42:0/3142134089 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbef003b410 0x7fbef003d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:59:41.025 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:40.996+0000 7fbf0ffff700 1 --2- 172.21.15.42:0/3142134089 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbef003b410 0x7fbef003d920 crc :-1 s=READY pgs=766 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:59:41.025 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:41.000+0000 7fbf1607d700 1 -- 172.21.15.42:0/3142134089 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf1005d110 con 0x7fbf1011beb0 2019-12-05T05:59:41.025 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:41.008+0000 7fbf0f7fe700 1 --2- 172.21.15.42:0/3142134089 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbf1011beb0 0x7fbf100a5600 secure :-1 s=THROTTLE_DONE pgs=405 cs=0 l=1 rx=0x7fbf04008790 tx=0x7fbf04011d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:41.025 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:41.008+0000 7fbf0cff9700 1 -- 172.21.15.42:0/3142134089 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fbf04032cc0 con 0x7fbf1011beb0 2019-12-05T05:59:41.184 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:41.180+0000 7fbf1607d700 1 -- 172.21.15.42:0/3142134089 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbf1005b8c0 con 0x7fbf1011beb0 2019-12-05T05:59:41.197 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:41.180+0000 7fbf0f7fe700 1 --2- 172.21.15.42:0/3142134089 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbf1011beb0 0x7fbf100a5600 secure :-1 s=THROTTLE_DONE pgs=405 cs=0 l=1 rx=0x7fbf04008790 tx=0x7fbf04011d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:41.197 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:41.180+0000 7fbf0cff9700 1 -- 172.21.15.42:0/3142134089 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 218 v218) v1 ==== 95+0+2112 (secure 0 0 0) 0x7fbf04032f20 con 0x7fbf1011beb0 2019-12-05T05:59:41.198 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 218 2019-12-05T05:59:41.198 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:41.184+0000 7fbf1607d700 1 -- 172.21.15.42:0/3142134089 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbef003b410 msgr2=0x7fbef003d920 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:59:41.198 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:41.184+0000 7fbf1607d700 1 --2- 172.21.15.42:0/3142134089 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbef003b410 0x7fbef003d920 crc :-1 s=READY pgs=766 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:59:41.198 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:41.184+0000 7fbf1607d700 1 -- 172.21.15.42:0/3142134089 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbf1011beb0 msgr2=0x7fbf100a5600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:59:41.198 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:41.184+0000 7fbf1607d700 1 --2- 172.21.15.42:0/3142134089 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbf1011beb0 0x7fbf100a5600 secure :-1 s=READY pgs=405 cs=0 l=1 rx=0x7fbf04008790 tx=0x7fbf04011d50).stop 2019-12-05T05:59:41.198 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:41.184+0000 7fbf1607d700 1 -- 172.21.15.42:0/3142134089 shutdown_connections 2019-12-05T05:59:41.198 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:41.184+0000 7fbf1607d700 1 --2- 172.21.15.42:0/3142134089 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbef003b410 0x7fbef003d920 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:59:41.199 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:41.184+0000 7fbf1607d700 1 --2- 172.21.15.42:0/3142134089 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbf1011beb0 0x7fbf100a5600 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:59:41.199 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:41.184+0000 7fbf1607d700 1 -- 172.21.15.42:0/3142134089 shutdown_connections 2019-12-05T05:59:41.199 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:41.184+0000 7fbf1607d700 1 -- 172.21.15.42:0/3142134089 wait complete. 2019-12-05T05:59:41.199 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:41.184+0000 7fbf1607d700 1 -- 172.21.15.42:0/3142134089 >> 172.21.15.42:0/3142134089 conn(0x7fbf100aabe0 msgr2=0x7fbf100a97e0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:59:41.199 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:59:41.199 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":218,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":8940,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/1082666540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1082666540},{"type":"v1","addr":"172.21.15.40:6833","nonce":1082666540}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":218}],"filesystems":[{"mdsmap":{"epoch":217,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:57:29.289849+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8836},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8836":{"gid":8836,"name":"a","rank":0,"incarnation":214,"state":"up:active","state_seq":29,"addr":"172.21.15.42:6835/1368848584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1368848584},{"type":"v1","addr":"172.21.15.42:6835","nonce":1368848584}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:59:41.200 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T05:59:42.985 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:59:42.991 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T05:59:43.201 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:59:43.333 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:43.308+0000 7f155d797700 1 Processor -- start 2019-12-05T05:59:43.334 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:43.308+0000 7f155d797700 1 -- start start 2019-12-05T05:59:43.334 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:43.312+0000 7f155d797700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f155808df80 0x7f155809f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:59:43.334 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:43.312+0000 7f155d797700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f155809f830 0x7f155814efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:59:43.334 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:43.312+0000 7f155d797700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f155809fc60 con 0x7f15580a7740 2019-12-05T05:59:43.334 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:43.312+0000 7f155d797700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15580a0b00 con 0x7f155809f830 2019-12-05T05:59:43.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:43.312+0000 7f155d797700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f155808e5b0 con 0x7f155808df80 2019-12-05T05:59:43.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:43.312+0000 7f1556ffd700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f155808df80 0x7f155809f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:59:43.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:43.312+0000 7f1557fff700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f155809f830 0x7f155814efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:59:43.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:43.312+0000 7f1556ffd700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f155808df80 0x7f155809f2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:45996/0 (socket says 172.21.15.42:45996) 2019-12-05T05:59:43.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:43.312+0000 7f1556ffd700 1 -- 172.21.15.42:0/2526447072 learned_addr learned my addr 172.21.15.42:0/2526447072 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:59:43.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:43.312+0000 7f15567fc700 1 -- 172.21.15.42:0/2526447072 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1911495475 0 0) 0x7f155809fc60 con 0x7f15580a7740 2019-12-05T05:59:43.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:43.312+0000 7f1556ffd700 1 -- 172.21.15.42:0/2526447072 >> v2:172.21.15.40:3300/0 conn(0x7f155809f830 msgr2=0x7f155814efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:59:43.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:43.312+0000 7f1556ffd700 1 --2- 172.21.15.42:0/2526447072 >> v2:172.21.15.40:3300/0 conn(0x7f155809f830 0x7f155814efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:59:43.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:43.312+0000 7f1556ffd700 1 -- 172.21.15.42:0/2526447072 >> v1:172.21.15.40:6789/0 conn(0x7f15580a7740 legacy=0x7f155807e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:59:43.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:43.312+0000 7f1556ffd700 1 -- 172.21.15.42:0/2526447072 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15580a0710 con 0x7f155808df80 2019-12-05T05:59:43.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:43.312+0000 7f1556ffd700 1 --2- 172.21.15.42:0/2526447072 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f155808df80 0x7f155809f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f154c008790 tx=0x7f154c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:43.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:43.312+0000 7f1556ffd700 1 --2- 172.21.15.42:0/2526447072 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f155808df80 0x7f155809f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f154c008790 tx=0x7f154c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:43.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:43.312+0000 7f1556ffd700 1 --2- 172.21.15.42:0/2526447072 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f155808df80 0x7f155809f2e0 secure :-1 s=READY pgs=406 cs=0 l=1 rx=0x7f154c008790 tx=0x7f154c00fd30).ready entity=mon.2 client_cookie=7f0820c9e122a2f0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:59:43.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:43.312+0000 7f1556ffd700 1 --2- 172.21.15.42:0/2526447072 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f155808df80 0x7f155809f2e0 secure :-1 s=THROTTLE_DONE pgs=406 cs=0 l=1 rx=0x7f154c008790 tx=0x7f154c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:43.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:43.312+0000 7f15567fc700 1 -- 172.21.15.42:0/2526447072 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f154c008c90 con 0x7f155808df80 2019-12-05T05:59:43.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:43.312+0000 7f1556ffd700 1 --2- 172.21.15.42:0/2526447072 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f155808df80 0x7f155809f2e0 secure :-1 s=THROTTLE_DONE pgs=406 cs=0 l=1 rx=0x7f154c008790 tx=0x7f154c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:43.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:43.312+0000 7f15567fc700 1 -- 172.21.15.42:0/2526447072 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f154c0088b0 con 0x7f155808df80 2019-12-05T05:59:43.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:43.312+0000 7f1556ffd700 1 --2- 172.21.15.42:0/2526447072 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f155808df80 0x7f155809f2e0 secure :-1 s=THROTTLE_DONE pgs=406 cs=0 l=1 rx=0x7f154c008790 tx=0x7f154c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:43.341 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:43.312+0000 7f155d797700 1 -- 172.21.15.42:0/2526447072 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f155808df80 msgr2=0x7f155809f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:59:43.341 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:43.312+0000 7f15567fc700 1 -- 172.21.15.42:0/2526447072 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f154c011b10 con 0x7f155808df80 2019-12-05T05:59:43.341 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:43.312+0000 7f155d797700 1 --2- 172.21.15.42:0/2526447072 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f155808df80 0x7f155809f2e0 secure :-1 s=READY pgs=406 cs=0 l=1 rx=0x7f154c008790 tx=0x7f154c00fd30).stop 2019-12-05T05:59:43.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:43.316+0000 7f155d797700 1 -- 172.21.15.42:0/2526447072 shutdown_connections 2019-12-05T05:59:43.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:43.316+0000 7f155d797700 1 --2- 172.21.15.42:0/2526447072 >> v2:172.21.15.40:3300/0 conn(0x7f155809f830 0x7f155814efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:59:43.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:43.316+0000 7f155d797700 1 --2- 172.21.15.42:0/2526447072 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f155808df80 0x7f155809f2e0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:59:43.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:43.316+0000 7f155d797700 1 -- 172.21.15.42:0/2526447072 shutdown_connections 2019-12-05T05:59:43.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:43.316+0000 7f155d797700 1 -- 172.21.15.42:0/2526447072 wait complete. 2019-12-05T05:59:43.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:43.316+0000 7f155d797700 1 -- 172.21.15.42:0/2526447072 >> 172.21.15.42:0/2526447072 conn(0x7f15580aabe0 msgr2=0x7f15580a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:59:43.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:43.320+0000 7f155d797700 1 Processor -- start 2019-12-05T05:59:43.359 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:43.320+0000 7f155d797700 1 -- start start 2019-12-05T05:59:43.359 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:43.320+0000 7f155d797700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f1558116500 0x7f155811a440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:59:43.359 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:43.320+0000 7f155d797700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f1558078890 0x7f1558078cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:59:43.359 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:43.320+0000 7f155d797700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f15580a35a0 0x7f15580a39d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:59:43.359 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:43.320+0000 7f155d797700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1558079210 con 0x7f1558078890 2019-12-05T05:59:43.359 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:43.320+0000 7f155d797700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15580a3f20 con 0x7f1558116500 2019-12-05T05:59:43.360 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:43.320+0000 7f155d797700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1558119c60 con 0x7f15580a35a0 2019-12-05T05:59:43.360 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:43.320+0000 7f15577fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f1558116500 0x7f155811a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:59:43.360 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:43.320+0000 7f15577fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f1558116500 0x7f155811a440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:45534/0 (socket says 172.21.15.42:45534) 2019-12-05T05:59:43.360 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:43.320+0000 7f15577fe700 1 -- 172.21.15.42:0/1102042401 learned_addr learned my addr 172.21.15.42:0/1102042401 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:59:43.360 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:43.320+0000 7f1556ffd700 1 --2- 172.21.15.42:0/1102042401 >> v2:172.21.15.40:3300/0 conn(0x7f1558078890 0x7f1558078cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:59:43.360 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:43.320+0000 7f1557fff700 1 --2- 172.21.15.42:0/1102042401 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f15580a35a0 0x7f15580a39d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:59:43.361 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:43.320+0000 7f15577fe700 1 -- 172.21.15.42:0/1102042401 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f15580a35a0 msgr2=0x7f15580a39d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:59:43.361 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:43.320+0000 7f15577fe700 1 --2- 172.21.15.42:0/1102042401 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f15580a35a0 0x7f15580a39d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:59:43.361 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:43.320+0000 7f15577fe700 1 -- 172.21.15.42:0/1102042401 >> v2:172.21.15.40:3300/0 conn(0x7f1558078890 msgr2=0x7f1558078cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:59:43.361 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:43.320+0000 7f15577fe700 1 --2- 172.21.15.42:0/1102042401 >> v2:172.21.15.40:3300/0 conn(0x7f1558078890 0x7f1558078cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:59:43.361 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:43.320+0000 7f15577fe700 1 -- 172.21.15.42:0/1102042401 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f155811a990 con 0x7f1558116500 2019-12-05T05:59:43.361 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:43.320+0000 7f1556ffd700 1 --2- 172.21.15.42:0/1102042401 >> v2:172.21.15.40:3300/0 conn(0x7f1558078890 0x7f1558078cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:59:43.361 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:43.320+0000 7f15577fe700 1 --2- 172.21.15.42:0/1102042401 >> v2:172.21.15.42:3300/0 conn(0x7f1558116500 0x7f155811a440 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f1548000e90 tx=0x7f1548010a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:43.362 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:43.320+0000 7f15577fe700 1 --2- 172.21.15.42:0/1102042401 >> v2:172.21.15.42:3300/0 conn(0x7f1558116500 0x7f155811a440 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f1548000e90 tx=0x7f1548010a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:43.362 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:43.320+0000 7f15577fe700 1 --2- 172.21.15.42:0/1102042401 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1558116500 0x7f155811a440 secure :-1 s=READY pgs=1799 cs=0 l=1 rx=0x7f1548000e90 tx=0x7f1548010a20).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:59:43.362 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:43.320+0000 7f15577fe700 1 --2- 172.21.15.42:0/1102042401 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1558116500 0x7f155811a440 secure :-1 s=THROTTLE_DONE pgs=1799 cs=0 l=1 rx=0x7f1548000e90 tx=0x7f1548010a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:43.362 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:43.320+0000 7f153ffff700 1 -- 172.21.15.42:0/1102042401 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f154800ca70 con 0x7f1558116500 2019-12-05T05:59:43.363 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:43.320+0000 7f15577fe700 1 --2- 172.21.15.42:0/1102042401 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1558116500 0x7f155811a440 secure :-1 s=THROTTLE_DONE pgs=1799 cs=0 l=1 rx=0x7f1548000e90 tx=0x7f1548010a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:43.363 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:43.320+0000 7f15577fe700 1 --2- 172.21.15.42:0/1102042401 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1558116500 0x7f155811a440 secure :-1 s=THROTTLE_DONE pgs=1799 cs=0 l=1 rx=0x7f1548000e90 tx=0x7f1548010a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:43.364 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:43.320+0000 7f153ffff700 1 -- 172.21.15.42:0/1102042401 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f154800c690 con 0x7f1558116500 2019-12-05T05:59:43.364 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:43.320+0000 7f155d797700 1 -- 172.21.15.42:0/1102042401 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15581185b0 con 0x7f1558116500 2019-12-05T05:59:43.364 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:43.320+0000 7f153ffff700 1 -- 172.21.15.42:0/1102042401 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1548012290 con 0x7f1558116500 2019-12-05T05:59:43.364 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:43.320+0000 7f155d797700 1 -- 172.21.15.42:0/1102042401 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f155808eb70 con 0x7f1558116500 2019-12-05T05:59:43.364 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:43.324+0000 7f15577fe700 1 --2- 172.21.15.42:0/1102042401 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1558116500 0x7f155811a440 secure :-1 s=THROTTLE_DONE pgs=1799 cs=0 l=1 rx=0x7f1548000e90 tx=0x7f1548010a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:43.364 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:43.324+0000 7f153ffff700 1 -- 172.21.15.42:0/1102042401 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f1548012560 con 0x7f1558116500 2019-12-05T05:59:43.365 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:43.324+0000 7f15577fe700 1 --2- 172.21.15.42:0/1102042401 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1558116500 0x7f155811a440 secure :-1 s=THROTTLE_DONE pgs=1799 cs=0 l=1 rx=0x7f1548000e90 tx=0x7f1548010a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:43.365 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:43.324+0000 7f153ffff700 1 --2- 172.21.15.42:0/1102042401 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f153803ae90 0x7f153803d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:59:43.365 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:43.324+0000 7f153ffff700 1 -- 172.21.15.42:0/1102042401 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 9213+0+0 (secure 0 0 0) 0x7f154804b300 con 0x7f1558116500 2019-12-05T05:59:43.365 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:43.324+0000 7f1556ffd700 1 --2- 172.21.15.42:0/1102042401 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f153803ae90 0x7f153803d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:59:43.365 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:43.324+0000 7f1556ffd700 1 --2- 172.21.15.42:0/1102042401 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f153803ae90 0x7f153803d3a0 crc :-1 s=READY pgs=768 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:59:43.366 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:43.324+0000 7f155d797700 1 -- 172.21.15.42:0/1102042401 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f155805d110 con 0x7f1558116500 2019-12-05T05:59:43.366 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:43.332+0000 7f15577fe700 1 --2- 172.21.15.42:0/1102042401 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1558116500 0x7f155811a440 secure :-1 s=THROTTLE_DONE pgs=1799 cs=0 l=1 rx=0x7f1548000e90 tx=0x7f1548010a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:43.366 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:43.332+0000 7f153ffff700 1 -- 172.21.15.42:0/1102042401 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f1548032380 con 0x7f1558116500 2019-12-05T05:59:43.511 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:43.504+0000 7f155d797700 1 -- 172.21.15.42:0/1102042401 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f155805b8c0 con 0x7f1558116500 2019-12-05T05:59:43.512 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:43.508+0000 7f15577fe700 1 --2- 172.21.15.42:0/1102042401 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1558116500 0x7f155811a440 secure :-1 s=THROTTLE_DONE pgs=1799 cs=0 l=1 rx=0x7f1548000e90 tx=0x7f1548010a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:43.513 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:43.508+0000 7f153ffff700 1 -- 172.21.15.42:0/1102042401 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 218 v218) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f15480325e0 con 0x7f1558116500 2019-12-05T05:59:43.515 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 218 2019-12-05T05:59:43.515 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:59:43.516 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":218,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":8940,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/1082666540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1082666540},{"type":"v1","addr":"172.21.15.40:6833","nonce":1082666540}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":218}],"filesystems":[{"mdsmap":{"epoch":217,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:57:29.289849+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8836},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8836":{"gid":8836,"name":"a","rank":0,"incarnation":214,"state":"up:active","state_seq":29,"addr":"172.21.15.42:6835/1368848584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1368848584},{"type":"v1","addr":"172.21.15.42:6835","nonce":1368848584}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:59:43.520 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:43.508+0000 7f155d797700 1 -- 172.21.15.42:0/1102042401 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f153803ae90 msgr2=0x7f153803d3a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:59:43.521 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:43.508+0000 7f155d797700 1 --2- 172.21.15.42:0/1102042401 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f153803ae90 0x7f153803d3a0 crc :-1 s=READY pgs=768 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:59:43.521 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:43.508+0000 7f155d797700 1 -- 172.21.15.42:0/1102042401 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1558116500 msgr2=0x7f155811a440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:59:43.521 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:43.508+0000 7f155d797700 1 --2- 172.21.15.42:0/1102042401 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1558116500 0x7f155811a440 secure :-1 s=READY pgs=1799 cs=0 l=1 rx=0x7f1548000e90 tx=0x7f1548010a20).stop 2019-12-05T05:59:43.522 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:43.508+0000 7f155d797700 1 -- 172.21.15.42:0/1102042401 shutdown_connections 2019-12-05T05:59:43.522 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:43.508+0000 7f155d797700 1 --2- 172.21.15.42:0/1102042401 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f153803ae90 0x7f153803d3a0 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:59:43.523 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:43.508+0000 7f155d797700 1 --2- 172.21.15.42:0/1102042401 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f15580a35a0 0x7f15580a39d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:59:43.523 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:43.508+0000 7f155d797700 1 --2- 172.21.15.42:0/1102042401 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1558116500 0x7f155811a440 unknown :-1 s=CLOSED pgs=1799 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:59:43.523 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:43.508+0000 7f155d797700 1 --2- 172.21.15.42:0/1102042401 >> v2:172.21.15.40:3300/0 conn(0x7f1558078890 0x7f1558078cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:59:43.524 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:43.508+0000 7f155d797700 1 -- 172.21.15.42:0/1102042401 shutdown_connections 2019-12-05T05:59:43.524 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:43.508+0000 7f155d797700 1 -- 172.21.15.42:0/1102042401 wait complete. 2019-12-05T05:59:43.524 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:43.508+0000 7f155d797700 1 -- 172.21.15.42:0/1102042401 >> 172.21.15.42:0/1102042401 conn(0x7f15580a9c30 msgr2=0x7f155807e4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:59:43.526 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T05:59:45.527 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:59:45.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.628+0000 7f48c17e4700 1 Processor -- start 2019-12-05T05:59:45.654 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.628+0000 7f48c17e4700 1 -- start start 2019-12-05T05:59:45.654 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.628+0000 7f48c17e4700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f48bc0a7740 0x7f48bc08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:59:45.654 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.628+0000 7f48c17e4700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f48bc09fae0 0x7f48bc07eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:59:45.654 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.628+0000 7f48c17e4700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48bc0a0710 con 0x7f48bc07e720 2019-12-05T05:59:45.654 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.628+0000 7f48c17e4700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48bc0a0b00 con 0x7f48bc09fae0 2019-12-05T05:59:45.655 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.628+0000 7f48c17e4700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48bc08e5b0 con 0x7f48bc0a7740 2019-12-05T05:59:45.655 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.628+0000 7f48bbfff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f48bc09fae0 0x7f48bc07eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:59:45.655 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.628+0000 7f48baffd700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f48bc07e720 0x7f48bc09f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:55026/0 (socket says 172.21.15.42:55026) 2019-12-05T05:59:45.655 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.628+0000 7f48bbfff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f48bc09fae0 0x7f48bc07eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:45552/0 (socket says 172.21.15.42:45552) 2019-12-05T05:59:45.656 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.628+0000 7f48baffd700 1 -- 172.21.15.42:0/3673624713 learned_addr learned my addr 172.21.15.42:0/3673624713 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:59:45.656 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.628+0000 7f48bb7fe700 1 --2- 172.21.15.42:0/3673624713 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f48bc0a7740 0x7f48bc08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:59:45.662 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.632+0000 7f48baffd700 1 -- 172.21.15.42:0/3673624713 >> v1:172.21.15.40:6789/0 conn(0x7f48bc07e720 legacy=0x7f48bc09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2019-12-05T05:59:45.663 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.632+0000 7f48baffd700 1 -- 172.21.15.42:0/3673624713 >> v1:172.21.15.40:6789/0 conn(0x7f48bc07e720 legacy=0x7f48bc09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2019-12-05T05:59:45.663 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.632+0000 7f48baffd700 1 --1- 172.21.15.42:0/3673624713 >> v1:172.21.15.40:6789/0 conn(0x7f48bc07e720 0x7f48bc09f2e0 :-1 s=OPENED pgs=1374 cs=1 l=1).handle_message read tag failed 2019-12-05T05:59:45.663 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.632+0000 7f48baffd700 1 --1- 172.21.15.42:0/3673624713 >> v1:172.21.15.40:6789/0 conn(0x7f48bc07e720 0x7f48bc09f2e0 :-1 s=OPENED pgs=1374 cs=1 l=1).fault on lossy channel, failing 2019-12-05T05:59:45.663 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.632+0000 7f48bbfff700 1 -- 172.21.15.42:0/3673624713 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f48bc0a7740 msgr2=0x7f48bc08df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:59:45.664 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.632+0000 7f48bbfff700 1 --2- 172.21.15.42:0/3673624713 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f48bc0a7740 0x7f48bc08df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:59:45.664 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.632+0000 7f48bbfff700 1 -- 172.21.15.42:0/3673624713 >> v1:172.21.15.40:6789/0 conn(0x7f48bc07e720 legacy=0x7f48bc09f2e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2019-12-05T05:59:45.664 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.632+0000 7f48bbfff700 1 -- 172.21.15.42:0/3673624713 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48bc07f0a0 con 0x7f48bc09fae0 2019-12-05T05:59:45.664 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.632+0000 7f48bbfff700 1 --2- 172.21.15.42:0/3673624713 >> v2:172.21.15.42:3300/0 conn(0x7f48bc09fae0 0x7f48bc07eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f48b40032b0 tx=0x7f48b400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:45.664 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.632+0000 7f48bbfff700 1 --2- 172.21.15.42:0/3673624713 >> v2:172.21.15.42:3300/0 conn(0x7f48bc09fae0 0x7f48bc07eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f48b40032b0 tx=0x7f48b400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:45.664 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.632+0000 7f48bbfff700 1 --2- 172.21.15.42:0/3673624713 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f48bc09fae0 0x7f48bc07eb50 secure :-1 s=READY pgs=1800 cs=0 l=1 rx=0x7f48b40032b0 tx=0x7f48b400fff0).ready entity=mon.1 client_cookie=f5b7aff00624798e server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:59:45.665 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.632+0000 7f48bbfff700 1 --2- 172.21.15.42:0/3673624713 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f48bc09fae0 0x7f48bc07eb50 secure :-1 s=THROTTLE_DONE pgs=1800 cs=0 l=1 rx=0x7f48b40032b0 tx=0x7f48b400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:45.665 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.632+0000 7f48ba7fc700 1 -- 172.21.15.42:0/3673624713 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f48b40108c0 con 0x7f48bc09fae0 2019-12-05T05:59:45.665 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.632+0000 7f48bbfff700 1 --2- 172.21.15.42:0/3673624713 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f48bc09fae0 0x7f48bc07eb50 secure :-1 s=THROTTLE_DONE pgs=1800 cs=0 l=1 rx=0x7f48b40032b0 tx=0x7f48b400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:45.665 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.632+0000 7f48bbfff700 1 --2- 172.21.15.42:0/3673624713 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f48bc09fae0 0x7f48bc07eb50 secure :-1 s=THROTTLE_DONE pgs=1800 cs=0 l=1 rx=0x7f48b40032b0 tx=0x7f48b400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:45.665 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.632+0000 7f48ba7fc700 1 -- 172.21.15.42:0/3673624713 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48b4008b50 con 0x7f48bc09fae0 2019-12-05T05:59:45.665 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.632+0000 7f48ba7fc700 1 -- 172.21.15.42:0/3673624713 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f48b40110c0 con 0x7f48bc09fae0 2019-12-05T05:59:45.665 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.632+0000 7f48c17e4700 1 -- 172.21.15.42:0/3673624713 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f48bc09fae0 msgr2=0x7f48bc07eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:59:45.666 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.632+0000 7f48c17e4700 1 --2- 172.21.15.42:0/3673624713 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f48bc09fae0 0x7f48bc07eb50 secure :-1 s=READY pgs=1800 cs=0 l=1 rx=0x7f48b40032b0 tx=0x7f48b400fff0).stop 2019-12-05T05:59:45.666 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.632+0000 7f48c17e4700 1 -- 172.21.15.42:0/3673624713 shutdown_connections 2019-12-05T05:59:45.666 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.632+0000 7f48c17e4700 1 --2- 172.21.15.42:0/3673624713 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f48bc09fae0 0x7f48bc07eb50 unknown :-1 s=CLOSED pgs=1800 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:59:45.666 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.632+0000 7f48c17e4700 1 --2- 172.21.15.42:0/3673624713 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f48bc0a7740 0x7f48bc08df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:59:45.666 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.632+0000 7f48c17e4700 1 -- 172.21.15.42:0/3673624713 shutdown_connections 2019-12-05T05:59:45.666 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.632+0000 7f48c17e4700 1 -- 172.21.15.42:0/3673624713 wait complete. 2019-12-05T05:59:45.666 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.632+0000 7f48c17e4700 1 -- 172.21.15.42:0/3673624713 >> 172.21.15.42:0/3673624713 conn(0x7f48bc0aabe0 msgr2=0x7f48bc0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:59:45.667 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.632+0000 7f48c17e4700 1 Processor -- start 2019-12-05T05:59:45.667 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.632+0000 7f48c17e4700 1 -- start start 2019-12-05T05:59:45.667 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.636+0000 7f48c17e4700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f48bc12e420 0x7f48bc11d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:59:45.667 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.636+0000 7f48c17e4700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48bc0a4f60 con 0x7f48bc11ba40 2019-12-05T05:59:45.667 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.636+0000 7f48c17e4700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48bc076af0 con 0x7f48bc11d2b0 2019-12-05T05:59:45.667 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.636+0000 7f48c17e4700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48bc11bfa0 con 0x7f48bc12e420 2019-12-05T05:59:45.667 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.636+0000 7f48bb7fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f48bc12e420 0x7f48bc11d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:59:45.668 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.636+0000 7f48baffd700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f48bc11d2b0 0x7f48bc11a740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:45594/0 (socket says 172.21.15.42:45594) 2019-12-05T05:59:45.668 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.636+0000 7f48baffd700 1 -- 172.21.15.42:0/1290131330 learned_addr learned my addr 172.21.15.42:0/1290131330 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:59:45.668 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.636+0000 7f48a3fff700 1 -- 172.21.15.42:0/1290131330 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3161630148 0 0) 0x7f48bc0a4f60 con 0x7f48bc11ba40 2019-12-05T05:59:45.668 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.636+0000 7f48a3fff700 1 -- 172.21.15.42:0/1290131330 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f489c0030b0 con 0x7f48bc11ba40 2019-12-05T05:59:45.668 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.636+0000 7f48a3fff700 1 -- 172.21.15.42:0/1290131330 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 779684632 0 0) 0x7f48bc076af0 con 0x7f48bc11d2b0 2019-12-05T05:59:45.668 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.636+0000 7f48bb7fe700 1 -- 172.21.15.42:0/1290131330 >> v1:172.21.15.42:6789/0 conn(0x7f48bc11d2b0 legacy=0x7f48bc11a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:59:45.668 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.636+0000 7f48bb7fe700 1 -- 172.21.15.42:0/1290131330 >> v1:172.21.15.40:6789/0 conn(0x7f48bc11ba40 legacy=0x7f48bc0742d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:59:45.669 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.636+0000 7f48bb7fe700 1 -- 172.21.15.42:0/1290131330 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48bc077d30 con 0x7f48bc12e420 2019-12-05T05:59:45.669 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.636+0000 7f48bb7fe700 1 --2- 172.21.15.42:0/1290131330 >> v2:172.21.15.42:3300/0 conn(0x7f48bc12e420 0x7f48bc11d920 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f48ac002f30 tx=0x7f48ac010960).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:45.669 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.636+0000 7f48bb7fe700 1 --2- 172.21.15.42:0/1290131330 >> v2:172.21.15.42:3300/0 conn(0x7f48bc12e420 0x7f48bc11d920 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f48ac002f30 tx=0x7f48ac010960).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:45.669 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.636+0000 7f48bb7fe700 1 --2- 172.21.15.42:0/1290131330 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f48bc12e420 0x7f48bc11d920 secure :-1 s=READY pgs=1802 cs=0 l=1 rx=0x7f48ac002f30 tx=0x7f48ac010960).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:59:45.669 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.636+0000 7f48bb7fe700 1 --2- 172.21.15.42:0/1290131330 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f48bc12e420 0x7f48bc11d920 secure :-1 s=THROTTLE_DONE pgs=1802 cs=0 l=1 rx=0x7f48ac002f30 tx=0x7f48ac010960).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:45.669 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.636+0000 7f48a3fff700 1 -- 172.21.15.42:0/1290131330 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f48ac001700 con 0x7f48bc12e420 2019-12-05T05:59:45.670 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.636+0000 7f48bb7fe700 1 --2- 172.21.15.42:0/1290131330 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f48bc12e420 0x7f48bc11d920 secure :-1 s=THROTTLE_DONE pgs=1802 cs=0 l=1 rx=0x7f48ac002f30 tx=0x7f48ac010960).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:45.670 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.636+0000 7f48bb7fe700 1 --2- 172.21.15.42:0/1290131330 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f48bc12e420 0x7f48bc11d920 secure :-1 s=THROTTLE_DONE pgs=1802 cs=0 l=1 rx=0x7f48ac002f30 tx=0x7f48ac010960).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:45.670 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.636+0000 7f48a3fff700 1 -- 172.21.15.42:0/1290131330 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48ac001320 con 0x7f48bc12e420 2019-12-05T05:59:45.670 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.636+0000 7f48c17e4700 1 -- 172.21.15.42:0/1290131330 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48bc075ae0 con 0x7f48bc12e420 2019-12-05T05:59:45.670 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.636+0000 7f48a3fff700 1 -- 172.21.15.42:0/1290131330 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f48ac00d160 con 0x7f48bc12e420 2019-12-05T05:59:45.671 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.636+0000 7f48c17e4700 1 -- 172.21.15.42:0/1290131330 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48bc076060 con 0x7f48bc12e420 2019-12-05T05:59:45.671 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.636+0000 7f48bb7fe700 1 --2- 172.21.15.42:0/1290131330 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f48bc12e420 0x7f48bc11d920 secure :-1 s=THROTTLE_DONE pgs=1802 cs=0 l=1 rx=0x7f48ac002f30 tx=0x7f48ac010960).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:45.671 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.640+0000 7f48a3fff700 1 -- 172.21.15.42:0/1290131330 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f48ac007170 con 0x7f48bc12e420 2019-12-05T05:59:45.671 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.640+0000 7f48bb7fe700 1 --2- 172.21.15.42:0/1290131330 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f48bc12e420 0x7f48bc11d920 secure :-1 s=THROTTLE_DONE pgs=1802 cs=0 l=1 rx=0x7f48ac002f30 tx=0x7f48ac010960).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:45.671 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.640+0000 7f48a3fff700 1 --2- 172.21.15.42:0/1290131330 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f489c03b180 0x7f489c03d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:59:45.671 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.640+0000 7f48a3fff700 1 -- 172.21.15.42:0/1290131330 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9299+0+0 (secure 0 0 0) 0x7f48ac04aa20 con 0x7f48bc12e420 2019-12-05T05:59:45.672 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.640+0000 7f48baffd700 1 --2- 172.21.15.42:0/1290131330 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f489c03b180 0x7f489c03d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:59:45.672 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.640+0000 7f48baffd700 1 --2- 172.21.15.42:0/1290131330 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f489c03b180 0x7f489c03d690 crc :-1 s=READY pgs=770 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:59:45.672 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.644+0000 7f48c17e4700 1 -- 172.21.15.42:0/1290131330 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48bc05d110 con 0x7f48bc12e420 2019-12-05T05:59:45.672 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.652+0000 7f48bb7fe700 1 --2- 172.21.15.42:0/1290131330 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f48bc12e420 0x7f48bc11d920 secure :-1 s=THROTTLE_DONE pgs=1802 cs=0 l=1 rx=0x7f48ac002f30 tx=0x7f48ac010960).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:45.673 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.652+0000 7f48a3fff700 1 -- 172.21.15.42:0/1290131330 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f48ac031aa0 con 0x7f48bc12e420 2019-12-05T05:59:45.814 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.808+0000 7f48c17e4700 1 -- 172.21.15.42:0/1290131330 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f48bc05b8c0 con 0x7f48bc12e420 2019-12-05T05:59:45.815 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.808+0000 7f48bb7fe700 1 --2- 172.21.15.42:0/1290131330 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f48bc12e420 0x7f48bc11d920 secure :-1 s=THROTTLE_DONE pgs=1802 cs=0 l=1 rx=0x7f48ac002f30 tx=0x7f48ac010960).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:45.815 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.812+0000 7f48a3fff700 1 -- 172.21.15.42:0/1290131330 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 219 v219) v1 ==== 95+0+1774 (secure 0 0 0) 0x7f48ac031d00 con 0x7f48bc12e420 2019-12-05T05:59:45.815 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 219 2019-12-05T05:59:45.816 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:59:45.816 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":219,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":219,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:59:44.295892+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":87,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8940},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8940":{"gid":8940,"name":"b","rank":0,"incarnation":219,"state":"up:replay","state_seq":2,"addr":"172.21.15.40:6833/1082666540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1082666540},{"type":"v1","addr":"172.21.15.40:6833","nonce":1082666540}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:59:45.818 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.812+0000 7f48c17e4700 1 -- 172.21.15.42:0/1290131330 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f489c03b180 msgr2=0x7f489c03d690 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:59:45.818 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.812+0000 7f48c17e4700 1 --2- 172.21.15.42:0/1290131330 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f489c03b180 0x7f489c03d690 crc :-1 s=READY pgs=770 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:59:45.818 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.812+0000 7f48c17e4700 1 -- 172.21.15.42:0/1290131330 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f48bc12e420 msgr2=0x7f48bc11d920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:59:45.818 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.812+0000 7f48c17e4700 1 --2- 172.21.15.42:0/1290131330 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f48bc12e420 0x7f48bc11d920 secure :-1 s=READY pgs=1802 cs=0 l=1 rx=0x7f48ac002f30 tx=0x7f48ac010960).stop 2019-12-05T05:59:45.818 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.812+0000 7f48c17e4700 1 -- 172.21.15.42:0/1290131330 shutdown_connections 2019-12-05T05:59:45.819 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.812+0000 7f48c17e4700 1 --2- 172.21.15.42:0/1290131330 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f489c03b180 0x7f489c03d690 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:59:45.819 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.812+0000 7f48c17e4700 1 --2- 172.21.15.42:0/1290131330 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f48bc12e420 0x7f48bc11d920 unknown :-1 s=CLOSED pgs=1802 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:59:45.819 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.812+0000 7f48c17e4700 1 -- 172.21.15.42:0/1290131330 shutdown_connections 2019-12-05T05:59:45.819 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.812+0000 7f48c17e4700 1 -- 172.21.15.42:0/1290131330 wait complete. 2019-12-05T05:59:45.819 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.812+0000 7f48c17e4700 1 -- 172.21.15.42:0/1290131330 >> 172.21.15.42:0/1290131330 conn(0x7f48bc0a99c0 msgr2=0x7f48bc141680 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:59:45.827 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a down, removed from mdsmap 2019-12-05T05:59:45.828 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2019-12-05T05:59:45.828 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:59:45.925 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.916+0000 7f88dcb68700 1 Processor -- start 2019-12-05T05:59:45.926 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.916+0000 7f88dcb68700 1 -- start start 2019-12-05T05:59:45.926 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.916+0000 7f88dcb68700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f88d808df80 0x7f88d809f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:59:45.926 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.916+0000 7f88dcb68700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88d809fc60 con 0x7f88d809f830 2019-12-05T05:59:45.926 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.916+0000 7f88dcb68700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88d80a0710 con 0x7f88d80a7740 2019-12-05T05:59:45.927 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.916+0000 7f88dcb68700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88d80a0b00 con 0x7f88d808df80 2019-12-05T05:59:45.927 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.916+0000 7f88d6d9d700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f88d80a7740 0x7f88d807e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:45600/0 (socket says 172.21.15.42:45600) 2019-12-05T05:59:45.927 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.916+0000 7f88d6d9d700 1 -- 172.21.15.42:0/1744709486 learned_addr learned my addr 172.21.15.42:0/1744709486 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:59:45.927 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.916+0000 7f88cffff700 1 --2- 172.21.15.42:0/1744709486 >> v2:172.21.15.40:3300/0 conn(0x7f88d808df80 0x7f88d809f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:59:45.928 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.916+0000 7f88d659c700 1 -- 172.21.15.42:0/1744709486 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3377334088 0 0) 0x7f88d80a0710 con 0x7f88d80a7740 2019-12-05T05:59:45.928 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.916+0000 7f88d659c700 1 -- 172.21.15.42:0/1744709486 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88c40030b0 con 0x7f88d80a7740 2019-12-05T05:59:45.928 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.916+0000 7f88d659c700 1 -- 172.21.15.42:0/1744709486 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 943880311 0 0) 0x7f88d809fc60 con 0x7f88d809f830 2019-12-05T05:59:45.928 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.916+0000 7f88d659c700 1 -- 172.21.15.42:0/1744709486 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88d80a0710 con 0x7f88d809f830 2019-12-05T05:59:45.929 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.916+0000 7f88d659c700 1 -- 172.21.15.42:0/1744709486 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2165248906 0 0) 0x7f88c40030b0 con 0x7f88d80a7740 2019-12-05T05:59:45.929 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.916+0000 7f88cffff700 1 -- 172.21.15.42:0/1744709486 >> v1:172.21.15.42:6789/0 conn(0x7f88d80a7740 legacy=0x7f88d807e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:59:45.929 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.916+0000 7f88cffff700 1 -- 172.21.15.42:0/1744709486 >> v1:172.21.15.40:6789/0 conn(0x7f88d809f830 legacy=0x7f88d814efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:59:45.929 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.920+0000 7f88cffff700 1 -- 172.21.15.42:0/1744709486 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88d807ef20 con 0x7f88d808df80 2019-12-05T05:59:45.930 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.920+0000 7f88cffff700 1 --2- 172.21.15.42:0/1744709486 >> v2:172.21.15.40:3300/0 conn(0x7f88d808df80 0x7f88d809f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f88c0008790 tx=0x7f88c8001950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:45.930 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.920+0000 7f88cffff700 1 --2- 172.21.15.42:0/1744709486 >> v2:172.21.15.40:3300/0 conn(0x7f88d808df80 0x7f88d809f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f88c0008790 tx=0x7f88c8001950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:45.930 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.920+0000 7f88cffff700 1 --2- 172.21.15.42:0/1744709486 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f88d808df80 0x7f88d809f2e0 secure :-1 s=READY pgs=1377 cs=0 l=1 rx=0x7f88c0008790 tx=0x7f88c8001950).ready entity=mon.0 client_cookie=cda32699eded59f0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:59:45.931 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.920+0000 7f88cffff700 1 --2- 172.21.15.42:0/1744709486 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f88d808df80 0x7f88d809f2e0 secure :-1 s=THROTTLE_DONE pgs=1377 cs=0 l=1 rx=0x7f88c0008790 tx=0x7f88c8001950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:45.931 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.920+0000 7f88cffff700 1 --2- 172.21.15.42:0/1744709486 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f88d808df80 0x7f88d809f2e0 secure :-1 s=THROTTLE_DONE pgs=1377 cs=0 l=1 rx=0x7f88c0008790 tx=0x7f88c8001950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:45.931 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.920+0000 7f88d659c700 1 -- 172.21.15.42:0/1744709486 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f88c8001750 con 0x7f88d808df80 2019-12-05T05:59:45.931 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.920+0000 7f88cffff700 1 --2- 172.21.15.42:0/1744709486 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f88d808df80 0x7f88d809f2e0 secure :-1 s=THROTTLE_DONE pgs=1377 cs=0 l=1 rx=0x7f88c0008790 tx=0x7f88c8001950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:45.932 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.920+0000 7f88d659c700 1 -- 172.21.15.42:0/1744709486 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88c8001560 con 0x7f88d808df80 2019-12-05T05:59:45.932 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.920+0000 7f88d659c700 1 -- 172.21.15.42:0/1744709486 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f88c0011360 con 0x7f88d808df80 2019-12-05T05:59:45.932 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.920+0000 7f88dcb68700 1 -- 172.21.15.42:0/1744709486 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f88d808df80 msgr2=0x7f88d809f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:59:45.932 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.920+0000 7f88dcb68700 1 --2- 172.21.15.42:0/1744709486 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f88d808df80 0x7f88d809f2e0 secure :-1 s=READY pgs=1377 cs=0 l=1 rx=0x7f88c0008790 tx=0x7f88c8001950).stop 2019-12-05T05:59:45.933 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.920+0000 7f88dcb68700 1 -- 172.21.15.42:0/1744709486 shutdown_connections 2019-12-05T05:59:45.933 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.920+0000 7f88dcb68700 1 --2- 172.21.15.42:0/1744709486 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f88d808df80 0x7f88d809f2e0 unknown :-1 s=CLOSED pgs=1377 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:59:45.933 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.920+0000 7f88dcb68700 1 -- 172.21.15.42:0/1744709486 shutdown_connections 2019-12-05T05:59:45.933 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.920+0000 7f88dcb68700 1 -- 172.21.15.42:0/1744709486 wait complete. 2019-12-05T05:59:45.934 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.920+0000 7f88dcb68700 1 -- 172.21.15.42:0/1744709486 >> 172.21.15.42:0/1744709486 conn(0x7f88d80aabe0 msgr2=0x7f88d80a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:59:45.940 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.920+0000 7f88dcb68700 1 Processor -- start 2019-12-05T05:59:45.940 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.920+0000 7f88dcb68700 1 -- start start 2019-12-05T05:59:45.940 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.924+0000 7f88dcb68700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f88d80a6820 0x7f88d80a5720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:59:45.940 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.924+0000 7f88dcb68700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88d812cfd0 con 0x7f88d811b290 2019-12-05T05:59:45.941 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.924+0000 7f88dcb68700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88d812deb0 con 0x7f88d812da80 2019-12-05T05:59:45.941 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.924+0000 7f88dcb68700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88d812d2a0 con 0x7f88d80a6820 2019-12-05T05:59:45.942 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.924+0000 7f88d6d9d700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f88d811b290 0x7f88d811a1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:55046/0 (socket says 172.21.15.42:55046) 2019-12-05T05:59:45.942 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.924+0000 7f88d6d9d700 1 -- 172.21.15.42:0/4190851225 learned_addr learned my addr 172.21.15.42:0/4190851225 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:59:45.942 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.924+0000 7f88d759e700 1 --2- 172.21.15.42:0/4190851225 >> v2:172.21.15.40:3300/0 conn(0x7f88d80a6820 0x7f88d80a5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:59:45.943 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.924+0000 7f88cf7fe700 1 -- 172.21.15.42:0/4190851225 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4027302807 0 0) 0x7f88d812cfd0 con 0x7f88d811b290 2019-12-05T05:59:45.943 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.924+0000 7f88cf7fe700 1 -- 172.21.15.42:0/4190851225 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88bc0030b0 con 0x7f88d811b290 2019-12-05T05:59:45.943 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.924+0000 7f88cf7fe700 1 -- 172.21.15.42:0/4190851225 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4043157606 0 0) 0x7f88d812deb0 con 0x7f88d812da80 2019-12-05T05:59:45.943 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.924+0000 7f88cf7fe700 1 -- 172.21.15.42:0/4190851225 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88d812cfd0 con 0x7f88d812da80 2019-12-05T05:59:45.944 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.924+0000 7f88d759e700 1 -- 172.21.15.42:0/4190851225 >> v1:172.21.15.42:6789/0 conn(0x7f88d812da80 legacy=0x7f88d80a6020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:59:45.944 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.924+0000 7f88d759e700 1 -- 172.21.15.42:0/4190851225 >> v1:172.21.15.40:6789/0 conn(0x7f88d811b290 legacy=0x7f88d811a1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:59:45.944 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.924+0000 7f88d759e700 1 -- 172.21.15.42:0/4190851225 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88d813aa10 con 0x7f88d80a6820 2019-12-05T05:59:45.944 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.924+0000 7f88d759e700 1 --2- 172.21.15.42:0/4190851225 >> v2:172.21.15.40:3300/0 conn(0x7f88d80a6820 0x7f88d80a5720 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f88d0001210 tx=0x7f88d0010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:45.944 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.924+0000 7f88d759e700 1 --2- 172.21.15.42:0/4190851225 >> v2:172.21.15.40:3300/0 conn(0x7f88d80a6820 0x7f88d80a5720 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f88d0001210 tx=0x7f88d0010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:45.945 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.924+0000 7f88d759e700 1 --2- 172.21.15.42:0/4190851225 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f88d80a6820 0x7f88d80a5720 secure :-1 s=READY pgs=1379 cs=0 l=1 rx=0x7f88d0001210 tx=0x7f88d0010aa0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:59:45.945 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.924+0000 7f88d759e700 1 --2- 172.21.15.42:0/4190851225 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f88d80a6820 0x7f88d80a5720 secure :-1 s=THROTTLE_DONE pgs=1379 cs=0 l=1 rx=0x7f88d0001210 tx=0x7f88d0010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:45.945 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.924+0000 7f88cf7fe700 1 -- 172.21.15.42:0/4190851225 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f88d0006ac0 con 0x7f88d80a6820 2019-12-05T05:59:45.945 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.924+0000 7f88d759e700 1 --2- 172.21.15.42:0/4190851225 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f88d80a6820 0x7f88d80a5720 secure :-1 s=THROTTLE_DONE pgs=1379 cs=0 l=1 rx=0x7f88d0001210 tx=0x7f88d0010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:45.946 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.924+0000 7f88d759e700 1 --2- 172.21.15.42:0/4190851225 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f88d80a6820 0x7f88d80a5720 secure :-1 s=THROTTLE_DONE pgs=1379 cs=0 l=1 rx=0x7f88d0001210 tx=0x7f88d0010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:45.946 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.924+0000 7f88cf7fe700 1 -- 172.21.15.42:0/4190851225 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88d00066e0 con 0x7f88d80a6820 2019-12-05T05:59:45.946 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.924+0000 7f88dcb68700 1 -- 172.21.15.42:0/4190851225 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88d81389f0 con 0x7f88d80a6820 2019-12-05T05:59:45.947 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.924+0000 7f88cf7fe700 1 -- 172.21.15.42:0/4190851225 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f88d00127c0 con 0x7f88d80a6820 2019-12-05T05:59:45.947 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.924+0000 7f88dcb68700 1 -- 172.21.15.42:0/4190851225 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88d8138f40 con 0x7f88d80a6820 2019-12-05T05:59:45.947 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.924+0000 7f88d759e700 1 --2- 172.21.15.42:0/4190851225 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f88d80a6820 0x7f88d80a5720 secure :-1 s=THROTTLE_DONE pgs=1379 cs=0 l=1 rx=0x7f88d0001210 tx=0x7f88d0010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:45.947 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.924+0000 7f88cf7fe700 1 -- 172.21.15.42:0/4190851225 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f88d0012a90 con 0x7f88d80a6820 2019-12-05T05:59:45.948 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.924+0000 7f88d759e700 1 --2- 172.21.15.42:0/4190851225 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f88d80a6820 0x7f88d80a5720 secure :-1 s=THROTTLE_DONE pgs=1379 cs=0 l=1 rx=0x7f88d0001210 tx=0x7f88d0010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:45.948 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.924+0000 7f88cf7fe700 1 --2- 172.21.15.42:0/4190851225 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f88bc03b470 0x7f88bc03d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:59:45.948 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.924+0000 7f88cf7fe700 1 -- 172.21.15.42:0/4190851225 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9299+0+0 (secure 0 0 0) 0x7f88d004b110 con 0x7f88d80a6820 2019-12-05T05:59:45.948 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.924+0000 7f88d6d9d700 1 --2- 172.21.15.42:0/4190851225 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f88bc03b470 0x7f88bc03d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:59:45.948 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.928+0000 7f88d6d9d700 1 --2- 172.21.15.42:0/4190851225 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f88bc03b470 0x7f88bc03d980 crc :-1 s=READY pgs=771 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:59:45.949 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.932+0000 7f88dcb68700 1 -- 172.21.15.42:0/4190851225 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88d805d110 con 0x7f88d80a6820 2019-12-05T05:59:45.950 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.936+0000 7f88d759e700 1 --2- 172.21.15.42:0/4190851225 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f88d80a6820 0x7f88d80a5720 secure :-1 s=THROTTLE_DONE pgs=1379 cs=0 l=1 rx=0x7f88d0001210 tx=0x7f88d0010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:45.950 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:45.936+0000 7f88cf7fe700 1 -- 172.21.15.42:0/4190851225 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f88d0032150 con 0x7f88d80a6820 2019-12-05T05:59:46.100 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:46.096+0000 7f88dcb68700 1 -- 172.21.15.42:0/4190851225 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f88d805b8c0 con 0x7f88d80a6820 2019-12-05T05:59:46.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:46.096+0000 7f88d759e700 1 --2- 172.21.15.42:0/4190851225 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f88d80a6820 0x7f88d80a5720 secure :-1 s=THROTTLE_DONE pgs=1379 cs=0 l=1 rx=0x7f88d0001210 tx=0x7f88d0010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:46.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:46.096+0000 7f88cf7fe700 1 -- 172.21.15.42:0/4190851225 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 219 v219) v1 ==== 95+0+1774 (secure 0 0 0) 0x7f88d0012da0 con 0x7f88d80a6820 2019-12-05T05:59:46.101 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 219 2019-12-05T05:59:46.101 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:59:46.102 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":219,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":219,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:59:44.295892+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":87,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8940},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8940":{"gid":8940,"name":"b","rank":0,"incarnation":219,"state":"up:replay","state_seq":2,"addr":"172.21.15.40:6833/1082666540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1082666540},{"type":"v1","addr":"172.21.15.40:6833","nonce":1082666540}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:59:46.104 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:46.100+0000 7f88dcb68700 1 -- 172.21.15.42:0/4190851225 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f88bc03b470 msgr2=0x7f88bc03d980 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:59:46.104 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:46.100+0000 7f88dcb68700 1 --2- 172.21.15.42:0/4190851225 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f88bc03b470 0x7f88bc03d980 crc :-1 s=READY pgs=771 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:59:46.104 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:46.100+0000 7f88dcb68700 1 -- 172.21.15.42:0/4190851225 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f88d80a6820 msgr2=0x7f88d80a5720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:59:46.104 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:46.100+0000 7f88dcb68700 1 --2- 172.21.15.42:0/4190851225 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f88d80a6820 0x7f88d80a5720 secure :-1 s=READY pgs=1379 cs=0 l=1 rx=0x7f88d0001210 tx=0x7f88d0010aa0).stop 2019-12-05T05:59:46.104 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:46.100+0000 7f88dcb68700 1 -- 172.21.15.42:0/4190851225 shutdown_connections 2019-12-05T05:59:46.105 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:46.100+0000 7f88dcb68700 1 --2- 172.21.15.42:0/4190851225 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f88bc03b470 0x7f88bc03d980 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:59:46.105 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:46.100+0000 7f88dcb68700 1 --2- 172.21.15.42:0/4190851225 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f88d80a6820 0x7f88d80a5720 unknown :-1 s=CLOSED pgs=1379 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:59:46.105 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:46.100+0000 7f88dcb68700 1 -- 172.21.15.42:0/4190851225 shutdown_connections 2019-12-05T05:59:46.105 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:46.100+0000 7f88dcb68700 1 -- 172.21.15.42:0/4190851225 wait complete. 2019-12-05T05:59:46.105 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:46.100+0000 7f88dcb68700 1 -- 172.21.15.42:0/4190851225 >> 172.21.15.42:0/4190851225 conn(0x7f88d80a7740 msgr2=0x7f88d80a9770 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:59:46.114 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 219, "feature_flags": { "enable_multiple": false, "ever_enabled_multiple": false }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2019-12-05T05:03:35.604738+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 219, "ever_allowed_features": 2, "explicitly_allowed_features": 2, "failed": [], "flags": 18, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_8940": { "addr": "172.21.15.40:6833/1082666540", "addrs": { "addrvec": [ { "addr": "172.21.15.40:6832", "nonce": 1082666540, "type": "v2" }, { "addr": "172.21.15.40:6833", "nonce": 1082666540, "type": "v1" } ] }, "export_targets": [], "features": 4540138292836696063, "flags": 0, "gid": 8940, "incarnation": 219, "name": "b", "rank": 0, "state": "up:replay", "state_seq": 2 } }, "last_failure": 0, "last_failure_osd_epoch": 87, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 2, "min_compat_client": "0 (unknown)", "modified": "2019-12-05T05:59:44.295892+0000", "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 8940 } } } ], "standbys": [] } 2019-12-05T05:59:48.115 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:59:48.228 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:48.216+0000 7f733bdd3700 1 Processor -- start 2019-12-05T05:59:48.229 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:48.216+0000 7f733bdd3700 1 -- start start 2019-12-05T05:59:48.229 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:48.216+0000 7f733bdd3700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f733409fae0 0x7f733414fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:59:48.229 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:48.216+0000 7f733bdd3700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73340a0710 con 0x7f733408df80 2019-12-05T05:59:48.229 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:48.216+0000 7f733bdd3700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f733407ef20 con 0x7f73340a7740 2019-12-05T05:59:48.230 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:48.216+0000 7f733bdd3700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73340a0b00 con 0x7f733409fae0 2019-12-05T05:59:48.230 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:48.216+0000 7f733a370700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f73340a7740 0x7f733407e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:45618/0 (socket says 172.21.15.42:45618) 2019-12-05T05:59:48.230 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:48.216+0000 7f733a370700 1 -- 172.21.15.42:0/214949210 learned_addr learned my addr 172.21.15.42:0/214949210 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:59:48.230 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:48.220+0000 7f733ab71700 1 --2- 172.21.15.42:0/214949210 >> v2:172.21.15.40:3300/0 conn(0x7f733409fae0 0x7f733414fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:59:48.230 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:48.220+0000 7f733936e700 1 -- 172.21.15.42:0/214949210 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1248998241 0 0) 0x7f733407ef20 con 0x7f73340a7740 2019-12-05T05:59:48.230 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:48.220+0000 7f733936e700 1 -- 172.21.15.42:0/214949210 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73200030b0 con 0x7f73340a7740 2019-12-05T05:59:48.230 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:48.220+0000 7f733ab71700 1 -- 172.21.15.42:0/214949210 >> v1:172.21.15.42:6789/0 conn(0x7f73340a7740 legacy=0x7f733407e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:59:48.231 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:48.220+0000 7f733ab71700 1 -- 172.21.15.42:0/214949210 >> v1:172.21.15.40:6789/0 conn(0x7f733408df80 legacy=0x7f733409f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:59:48.231 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:48.220+0000 7f733ab71700 1 -- 172.21.15.42:0/214949210 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73340a9ba0 con 0x7f733409fae0 2019-12-05T05:59:48.231 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:48.220+0000 7f733ab71700 1 --2- 172.21.15.42:0/214949210 >> v2:172.21.15.40:3300/0 conn(0x7f733409fae0 0x7f733414fff0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f732c008790 tx=0x7f732c00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:48.231 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:48.220+0000 7f733ab71700 1 --2- 172.21.15.42:0/214949210 >> v2:172.21.15.40:3300/0 conn(0x7f733409fae0 0x7f733414fff0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f732c008790 tx=0x7f732c00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:48.231 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:48.220+0000 7f733ab71700 1 --2- 172.21.15.42:0/214949210 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f733409fae0 0x7f733414fff0 secure :-1 s=READY pgs=1381 cs=0 l=1 rx=0x7f732c008790 tx=0x7f732c00fbf0).ready entity=mon.0 client_cookie=8ad8dc589b73550c server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:59:48.231 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:48.220+0000 7f733ab71700 1 --2- 172.21.15.42:0/214949210 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f733409fae0 0x7f733414fff0 secure :-1 s=THROTTLE_DONE pgs=1381 cs=0 l=1 rx=0x7f732c008790 tx=0x7f732c00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:48.231 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:48.220+0000 7f733936e700 1 -- 172.21.15.42:0/214949210 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f732c008bb0 con 0x7f733409fae0 2019-12-05T05:59:48.232 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:48.220+0000 7f733ab71700 1 --2- 172.21.15.42:0/214949210 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f733409fae0 0x7f733414fff0 secure :-1 s=THROTTLE_DONE pgs=1381 cs=0 l=1 rx=0x7f732c008790 tx=0x7f732c00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:48.232 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:48.220+0000 7f733ab71700 1 --2- 172.21.15.42:0/214949210 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f733409fae0 0x7f733414fff0 secure :-1 s=THROTTLE_DONE pgs=1381 cs=0 l=1 rx=0x7f732c008790 tx=0x7f732c00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:48.232 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:48.220+0000 7f733936e700 1 -- 172.21.15.42:0/214949210 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f732c0087d0 con 0x7f733409fae0 2019-12-05T05:59:48.232 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:48.220+0000 7f733936e700 1 -- 172.21.15.42:0/214949210 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f732c0118d0 con 0x7f733409fae0 2019-12-05T05:59:48.232 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:48.220+0000 7f733bdd3700 1 -- 172.21.15.42:0/214949210 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f733409fae0 msgr2=0x7f733414fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:59:48.232 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:48.220+0000 7f733bdd3700 1 --2- 172.21.15.42:0/214949210 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f733409fae0 0x7f733414fff0 secure :-1 s=READY pgs=1381 cs=0 l=1 rx=0x7f732c008790 tx=0x7f732c00fbf0).stop 2019-12-05T05:59:48.232 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:48.220+0000 7f733bdd3700 1 -- 172.21.15.42:0/214949210 shutdown_connections 2019-12-05T05:59:48.233 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:48.220+0000 7f733bdd3700 1 --2- 172.21.15.42:0/214949210 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f733409fae0 0x7f733414fff0 unknown :-1 s=CLOSED pgs=1381 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:59:48.233 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:48.220+0000 7f733bdd3700 1 -- 172.21.15.42:0/214949210 shutdown_connections 2019-12-05T05:59:48.233 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:48.220+0000 7f733bdd3700 1 -- 172.21.15.42:0/214949210 wait complete. 2019-12-05T05:59:48.233 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:48.220+0000 7f733bdd3700 1 -- 172.21.15.42:0/214949210 >> 172.21.15.42:0/214949210 conn(0x7f73340aabe0 msgr2=0x7f73340a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:59:48.233 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:48.224+0000 7f733bdd3700 1 Processor -- start 2019-12-05T05:59:48.233 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:48.224+0000 7f733bdd3700 1 -- start start 2019-12-05T05:59:48.234 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:48.224+0000 7f733bdd3700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f733407cb30 0x7f733412ce30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:59:48.234 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:48.224+0000 7f733bdd3700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f733412d380 0x7f7334137840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:59:48.237 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:48.224+0000 7f733bdd3700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f733407bb00 con 0x7f733411c2e0 2019-12-05T05:59:48.237 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:48.224+0000 7f733bdd3700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73340a5030 con 0x7f733412d380 2019-12-05T05:59:48.237 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:48.224+0000 7f733bdd3700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73340a52d0 con 0x7f733407cb30 2019-12-05T05:59:48.237 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:48.224+0000 7f733ab71700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f733412d380 0x7f7334137840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:59:48.237 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:48.224+0000 7f733ab71700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f733412d380 0x7f7334137840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:45588/0 (socket says 172.21.15.42:45588) 2019-12-05T05:59:48.237 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:48.224+0000 7f733ab71700 1 -- 172.21.15.42:0/3267535185 learned_addr learned my addr 172.21.15.42:0/3267535185 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:59:48.238 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:48.224+0000 7f7339b6f700 1 --2- 172.21.15.42:0/3267535185 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f733407cb30 0x7f733412ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:59:48.238 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:48.224+0000 7f732affd700 1 -- 172.21.15.42:0/3267535185 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2866354385 0 0) 0x7f733407bb00 con 0x7f733411c2e0 2019-12-05T05:59:48.238 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:48.224+0000 7f732affd700 1 -- 172.21.15.42:0/3267535185 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f731c0030b0 con 0x7f733411c2e0 2019-12-05T05:59:48.238 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:48.224+0000 7f733ab71700 1 -- 172.21.15.42:0/3267535185 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f733407cb30 msgr2=0x7f733412ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:59:48.238 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:48.224+0000 7f733ab71700 1 --2- 172.21.15.42:0/3267535185 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f733407cb30 0x7f733412ce30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:59:48.238 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:48.224+0000 7f733ab71700 1 -- 172.21.15.42:0/3267535185 >> v1:172.21.15.40:6789/0 conn(0x7f733411c2e0 legacy=0x7f73340a6b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:59:48.238 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:48.224+0000 7f733ab71700 1 -- 172.21.15.42:0/3267535185 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73340a4c50 con 0x7f733412d380 2019-12-05T05:59:48.239 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:48.228+0000 7f733ab71700 1 --2- 172.21.15.42:0/3267535185 >> v2:172.21.15.42:3300/0 conn(0x7f733412d380 0x7f7334137840 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f73340a51f0 tx=0x7f732c011e90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:48.239 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:48.228+0000 7f733ab71700 1 --2- 172.21.15.42:0/3267535185 >> v2:172.21.15.42:3300/0 conn(0x7f733412d380 0x7f7334137840 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f73340a51f0 tx=0x7f732c011e90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:48.239 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:48.228+0000 7f733ab71700 1 --2- 172.21.15.42:0/3267535185 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f733412d380 0x7f7334137840 secure :-1 s=READY pgs=1806 cs=0 l=1 rx=0x7f73340a51f0 tx=0x7f732c011e90).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:59:48.239 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:48.228+0000 7f733ab71700 1 --2- 172.21.15.42:0/3267535185 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f733412d380 0x7f7334137840 secure :-1 s=THROTTLE_DONE pgs=1806 cs=0 l=1 rx=0x7f73340a51f0 tx=0x7f732c011e90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:48.239 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:48.228+0000 7f733ab71700 1 --2- 172.21.15.42:0/3267535185 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f733412d380 0x7f7334137840 secure :-1 s=THROTTLE_DONE pgs=1806 cs=0 l=1 rx=0x7f73340a51f0 tx=0x7f732c011e90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:48.239 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:48.228+0000 7f732affd700 1 -- 172.21.15.42:0/3267535185 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f732c00f280 con 0x7f733412d380 2019-12-05T05:59:48.239 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:48.228+0000 7f733ab71700 1 --2- 172.21.15.42:0/3267535185 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f733412d380 0x7f7334137840 secure :-1 s=THROTTLE_DONE pgs=1806 cs=0 l=1 rx=0x7f73340a51f0 tx=0x7f732c011e90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:48.240 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:48.228+0000 7f732affd700 1 -- 172.21.15.42:0/3267535185 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f732c008db0 con 0x7f733412d380 2019-12-05T05:59:48.240 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:48.228+0000 7f733bdd3700 1 -- 172.21.15.42:0/3267535185 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73340a5570 con 0x7f733412d380 2019-12-05T05:59:48.240 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:48.228+0000 7f732affd700 1 -- 172.21.15.42:0/3267535185 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f732c005540 con 0x7f733412d380 2019-12-05T05:59:48.240 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:48.228+0000 7f733bdd3700 1 -- 172.21.15.42:0/3267535185 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f733407c410 con 0x7f733412d380 2019-12-05T05:59:48.240 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:48.228+0000 7f733ab71700 1 --2- 172.21.15.42:0/3267535185 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f733412d380 0x7f7334137840 secure :-1 s=THROTTLE_DONE pgs=1806 cs=0 l=1 rx=0x7f73340a51f0 tx=0x7f732c011e90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:48.240 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:48.228+0000 7f732affd700 1 -- 172.21.15.42:0/3267535185 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f732c005850 con 0x7f733412d380 2019-12-05T05:59:48.240 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:48.228+0000 7f733ab71700 1 --2- 172.21.15.42:0/3267535185 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f733412d380 0x7f7334137840 secure :-1 s=THROTTLE_DONE pgs=1806 cs=0 l=1 rx=0x7f73340a51f0 tx=0x7f732c011e90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:48.241 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:48.228+0000 7f732affd700 1 --2- 172.21.15.42:0/3267535185 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f731c03b220 0x7f731c03d730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:59:48.241 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:48.228+0000 7f732affd700 1 -- 172.21.15.42:0/3267535185 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9299+0+0 (secure 0 0 0) 0x7f732c012060 con 0x7f733412d380 2019-12-05T05:59:48.241 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:48.228+0000 7f733a370700 1 --2- 172.21.15.42:0/3267535185 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f731c03b220 0x7f731c03d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:59:48.241 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:48.232+0000 7f733a370700 1 --2- 172.21.15.42:0/3267535185 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f731c03b220 0x7f731c03d730 crc :-1 s=READY pgs=772 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:59:48.241 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:48.232+0000 7f733bdd3700 1 -- 172.21.15.42:0/3267535185 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f733405d110 con 0x7f733412d380 2019-12-05T05:59:48.244 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:48.236+0000 7f733ab71700 1 --2- 172.21.15.42:0/3267535185 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f733412d380 0x7f7334137840 secure :-1 s=THROTTLE_DONE pgs=1806 cs=0 l=1 rx=0x7f73340a51f0 tx=0x7f732c011e90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:48.244 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:48.236+0000 7f732affd700 1 -- 172.21.15.42:0/3267535185 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f732c032780 con 0x7f733412d380 2019-12-05T05:59:48.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:48.404+0000 7f733bdd3700 1 -- 172.21.15.42:0/3267535185 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f733405b8c0 con 0x7f733412d380 2019-12-05T05:59:48.412 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:48.404+0000 7f733ab71700 1 --2- 172.21.15.42:0/3267535185 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f733412d380 0x7f7334137840 secure :-1 s=THROTTLE_DONE pgs=1806 cs=0 l=1 rx=0x7f73340a51f0 tx=0x7f732c011e90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:48.412 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:48.404+0000 7f732affd700 1 -- 172.21.15.42:0/3267535185 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 221 v221) v1 ==== 95+0+1775 (secure 0 0 0) 0x7f732c0329e0 con 0x7f733412d380 2019-12-05T05:59:48.412 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 221 2019-12-05T05:59:48.413 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:59:48.413 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":221,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":221,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:59:48.320144+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":87,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8940},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8940":{"gid":8940,"name":"b","rank":0,"incarnation":219,"state":"up:rejoin","state_seq":35,"addr":"172.21.15.40:6833/1082666540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1082666540},{"type":"v1","addr":"172.21.15.40:6833","nonce":1082666540}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:59:48.415 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:48.408+0000 7f733bdd3700 1 -- 172.21.15.42:0/3267535185 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f731c03b220 msgr2=0x7f731c03d730 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:59:48.415 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:48.408+0000 7f733bdd3700 1 --2- 172.21.15.42:0/3267535185 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f731c03b220 0x7f731c03d730 crc :-1 s=READY pgs=772 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:59:48.415 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:48.408+0000 7f733bdd3700 1 -- 172.21.15.42:0/3267535185 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f733412d380 msgr2=0x7f7334137840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:59:48.415 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:48.408+0000 7f733bdd3700 1 --2- 172.21.15.42:0/3267535185 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f733412d380 0x7f7334137840 secure :-1 s=READY pgs=1806 cs=0 l=1 rx=0x7f73340a51f0 tx=0x7f732c011e90).stop 2019-12-05T05:59:48.416 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:48.408+0000 7f733bdd3700 1 -- 172.21.15.42:0/3267535185 shutdown_connections 2019-12-05T05:59:48.416 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:48.408+0000 7f733bdd3700 1 --2- 172.21.15.42:0/3267535185 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f731c03b220 0x7f731c03d730 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:59:48.416 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:48.408+0000 7f733bdd3700 1 --2- 172.21.15.42:0/3267535185 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f733412d380 0x7f7334137840 unknown :-1 s=CLOSED pgs=1806 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:59:48.417 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:48.408+0000 7f733bdd3700 1 --2- 172.21.15.42:0/3267535185 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f733407cb30 0x7f733412ce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:59:48.417 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:48.408+0000 7f733bdd3700 1 -- 172.21.15.42:0/3267535185 shutdown_connections 2019-12-05T05:59:48.417 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:48.408+0000 7f733bdd3700 1 -- 172.21.15.42:0/3267535185 wait complete. 2019-12-05T05:59:48.417 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:48.408+0000 7f733bdd3700 1 -- 172.21.15.42:0/3267535185 >> 172.21.15.42:0/3267535185 conn(0x7f73340a8030 msgr2=0x7f7334059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:59:48.433 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2019-12-05T05:59:50.434 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:59:50.545 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:50.536+0000 7facbb0c3700 1 Processor -- start 2019-12-05T05:59:50.546 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:50.536+0000 7facbb0c3700 1 -- start start 2019-12-05T05:59:50.546 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:50.536+0000 7facbb0c3700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7facb40a7740 0x7facb408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:59:50.551 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:50.536+0000 7facbb0c3700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7facb409fae0 0x7facb407eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:59:50.551 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:50.536+0000 7facbb0c3700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facb40a0710 con 0x7facb407e720 2019-12-05T05:59:50.551 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:50.536+0000 7facbb0c3700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7facb40a0b00 con 0x7facb409fae0 2019-12-05T05:59:50.552 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:50.536+0000 7facbb0c3700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facb408e5b0 con 0x7facb40a7740 2019-12-05T05:59:50.552 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:50.536+0000 7facb9e61700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7facb409fae0 0x7facb407eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:59:50.553 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:50.536+0000 7facb9660700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7facb40a7740 0x7facb408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:59:50.553 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:50.536+0000 7facb9660700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7facb40a7740 0x7facb408df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:46074/0 (socket says 172.21.15.42:46074) 2019-12-05T05:59:50.553 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:50.536+0000 7facb9660700 1 -- 172.21.15.42:0/699615962 learned_addr learned my addr 172.21.15.42:0/699615962 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:59:50.553 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:50.536+0000 7facb9e61700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7facb409fae0 0x7facb407eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:45612/0 (socket says 172.21.15.42:45612) 2019-12-05T05:59:50.553 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:50.536+0000 7facb9e61700 1 -- 172.21.15.42:0/699615962 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7facb40a7740 msgr2=0x7facb408df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:59:50.554 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:50.536+0000 7facb9e61700 1 --2- 172.21.15.42:0/699615962 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7facb40a7740 0x7facb408df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:59:50.554 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:50.536+0000 7facb9e61700 1 -- 172.21.15.42:0/699615962 >> v1:172.21.15.40:6789/0 conn(0x7facb407e720 legacy=0x7facb409f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:59:50.554 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:50.536+0000 7facb9e61700 1 -- 172.21.15.42:0/699615962 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facb407f0a0 con 0x7facb409fae0 2019-12-05T05:59:50.554 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:50.536+0000 7facb9660700 1 --2- 172.21.15.42:0/699615962 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7facb40a7740 0x7facb408df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:59:50.554 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:50.536+0000 7facb9e61700 1 --2- 172.21.15.42:0/699615962 >> v2:172.21.15.42:3300/0 conn(0x7facb409fae0 0x7facb407eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7facac008b10 tx=0x7facac00fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:50.557 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:50.536+0000 7facb9e61700 1 --2- 172.21.15.42:0/699615962 >> v2:172.21.15.42:3300/0 conn(0x7facb409fae0 0x7facb407eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7facac008b10 tx=0x7facac00fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:50.557 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:50.536+0000 7facb9e61700 1 --2- 172.21.15.42:0/699615962 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7facb409fae0 0x7facb407eb50 secure :-1 s=READY pgs=1807 cs=0 l=1 rx=0x7facac008b10 tx=0x7facac00fff0).ready entity=mon.1 client_cookie=c85f772a45964e29 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:59:50.557 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:50.536+0000 7facb9e61700 1 --2- 172.21.15.42:0/699615962 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7facb409fae0 0x7facb407eb50 secure :-1 s=THROTTLE_DONE pgs=1807 cs=0 l=1 rx=0x7facac008b10 tx=0x7facac00fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:50.558 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:50.536+0000 7facb9e61700 1 --2- 172.21.15.42:0/699615962 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7facb409fae0 0x7facb407eb50 secure :-1 s=THROTTLE_DONE pgs=1807 cs=0 l=1 rx=0x7facac008b10 tx=0x7facac00fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:50.558 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:50.536+0000 7facabfff700 1 -- 172.21.15.42:0/699615962 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7facac010900 con 0x7facb409fae0 2019-12-05T05:59:50.558 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:50.536+0000 7facb9e61700 1 --2- 172.21.15.42:0/699615962 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7facb409fae0 0x7facb407eb50 secure :-1 s=THROTTLE_DONE pgs=1807 cs=0 l=1 rx=0x7facac008b10 tx=0x7facac00fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:50.558 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:50.536+0000 7facabfff700 1 -- 172.21.15.42:0/699615962 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facac008b50 con 0x7facb409fae0 2019-12-05T05:59:50.558 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:50.536+0000 7facabfff700 1 -- 172.21.15.42:0/699615962 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7facac0110c0 con 0x7facb409fae0 2019-12-05T05:59:50.559 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:50.536+0000 7facbb0c3700 1 -- 172.21.15.42:0/699615962 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7facb409fae0 msgr2=0x7facb407eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:59:50.559 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:50.536+0000 7facbb0c3700 1 --2- 172.21.15.42:0/699615962 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7facb409fae0 0x7facb407eb50 secure :-1 s=READY pgs=1807 cs=0 l=1 rx=0x7facac008b10 tx=0x7facac00fff0).stop 2019-12-05T05:59:50.559 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:50.540+0000 7facbb0c3700 1 -- 172.21.15.42:0/699615962 shutdown_connections 2019-12-05T05:59:50.559 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:50.540+0000 7facbb0c3700 1 --2- 172.21.15.42:0/699615962 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7facb409fae0 0x7facb407eb50 unknown :-1 s=CLOSED pgs=1807 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:59:50.559 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:50.540+0000 7facbb0c3700 1 --2- 172.21.15.42:0/699615962 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7facb40a7740 0x7facb408df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:59:50.559 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:50.540+0000 7facbb0c3700 1 -- 172.21.15.42:0/699615962 shutdown_connections 2019-12-05T05:59:50.559 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:50.540+0000 7facbb0c3700 1 -- 172.21.15.42:0/699615962 wait complete. 2019-12-05T05:59:50.560 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:50.540+0000 7facbb0c3700 1 -- 172.21.15.42:0/699615962 >> 172.21.15.42:0/699615962 conn(0x7facb40aabe0 msgr2=0x7facb40a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:59:50.565 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:50.540+0000 7facbb0c3700 1 Processor -- start 2019-12-05T05:59:50.565 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:50.540+0000 7facbb0c3700 1 -- start start 2019-12-05T05:59:50.566 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:50.544+0000 7facbb0c3700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7facb41302c0 0x7facb4135700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:59:50.566 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:50.544+0000 7facbb0c3700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facb407d110 con 0x7facb40a6bc0 2019-12-05T05:59:50.566 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:50.544+0000 7facbb0c3700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facb411c3c0 con 0x7facb41306f0 2019-12-05T05:59:50.566 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:50.544+0000 7facbb0c3700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7facb407c810 con 0x7facb41302c0 2019-12-05T05:59:50.567 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:50.544+0000 7facb9660700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7facb40a6bc0 0x7facb411e0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:55092/0 (socket says 172.21.15.42:55092) 2019-12-05T05:59:50.567 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:50.544+0000 7facb9660700 1 -- 172.21.15.42:0/1865277412 learned_addr learned my addr 172.21.15.42:0/1865277412 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:59:50.567 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:50.544+0000 7facb8e5f700 1 --2- 172.21.15.42:0/1865277412 >> v2:172.21.15.42:3300/0 conn(0x7facb41302c0 0x7facb4135700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:59:50.567 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:50.544+0000 7faca9ffb700 1 -- 172.21.15.42:0/1865277412 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 400401039 0 0) 0x7facb407d110 con 0x7facb40a6bc0 2019-12-05T05:59:50.567 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:50.544+0000 7faca9ffb700 1 -- 172.21.15.42:0/1865277412 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac940030b0 con 0x7facb40a6bc0 2019-12-05T05:59:50.567 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:50.544+0000 7faca9ffb700 1 -- 172.21.15.42:0/1865277412 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1246149754 0 0) 0x7facb411c3c0 con 0x7facb41306f0 2019-12-05T05:59:50.567 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:50.544+0000 7faca9ffb700 1 -- 172.21.15.42:0/1865277412 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7facb407d110 con 0x7facb41306f0 2019-12-05T05:59:50.568 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:50.544+0000 7facb8e5f700 1 -- 172.21.15.42:0/1865277412 >> v1:172.21.15.42:6789/0 conn(0x7facb41306f0 legacy=0x7facb4137c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:59:50.568 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:50.544+0000 7facb8e5f700 1 -- 172.21.15.42:0/1865277412 >> v1:172.21.15.40:6789/0 conn(0x7facb40a6bc0 legacy=0x7facb411e0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:59:50.568 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:50.544+0000 7facb8e5f700 1 -- 172.21.15.42:0/1865277412 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facb407b5d0 con 0x7facb41302c0 2019-12-05T05:59:50.568 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:50.544+0000 7facb8e5f700 1 --2- 172.21.15.42:0/1865277412 >> v2:172.21.15.42:3300/0 conn(0x7facb41302c0 0x7facb4135700 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7faca4001430 tx=0x7faca400c320).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:50.568 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:50.544+0000 7facb8e5f700 1 --2- 172.21.15.42:0/1865277412 >> v2:172.21.15.42:3300/0 conn(0x7facb41302c0 0x7facb4135700 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7faca4001430 tx=0x7faca400c320).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:50.568 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:50.544+0000 7facb8e5f700 1 --2- 172.21.15.42:0/1865277412 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7facb41302c0 0x7facb4135700 secure :-1 s=READY pgs=1809 cs=0 l=1 rx=0x7faca4001430 tx=0x7faca400c320).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:59:50.569 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:50.544+0000 7facb8e5f700 1 --2- 172.21.15.42:0/1865277412 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7facb41302c0 0x7facb4135700 secure :-1 s=THROTTLE_DONE pgs=1809 cs=0 l=1 rx=0x7faca4001430 tx=0x7faca400c320).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:50.569 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:50.544+0000 7faca9ffb700 1 -- 172.21.15.42:0/1865277412 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faca400cb70 con 0x7facb41302c0 2019-12-05T05:59:50.569 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:50.544+0000 7facb8e5f700 1 --2- 172.21.15.42:0/1865277412 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7facb41302c0 0x7facb4135700 secure :-1 s=THROTTLE_DONE pgs=1809 cs=0 l=1 rx=0x7faca4001430 tx=0x7faca400c320).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:50.569 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:50.544+0000 7facb8e5f700 1 --2- 172.21.15.42:0/1865277412 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7facb41302c0 0x7facb4135700 secure :-1 s=THROTTLE_DONE pgs=1809 cs=0 l=1 rx=0x7faca4001430 tx=0x7faca400c320).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:50.569 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:50.544+0000 7faca9ffb700 1 -- 172.21.15.42:0/1865277412 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faca400c790 con 0x7facb41302c0 2019-12-05T05:59:50.569 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:50.544+0000 7facbb0c3700 1 -- 172.21.15.42:0/1865277412 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facb40a6ff0 con 0x7facb41302c0 2019-12-05T05:59:50.569 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:50.544+0000 7faca9ffb700 1 -- 172.21.15.42:0/1865277412 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faca4011070 con 0x7facb41302c0 2019-12-05T05:59:50.570 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:50.544+0000 7facbb0c3700 1 -- 172.21.15.42:0/1865277412 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facb407d9c0 con 0x7facb41302c0 2019-12-05T05:59:50.570 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:50.548+0000 7facb8e5f700 1 --2- 172.21.15.42:0/1865277412 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7facb41302c0 0x7facb4135700 secure :-1 s=THROTTLE_DONE pgs=1809 cs=0 l=1 rx=0x7faca4001430 tx=0x7faca400c320).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:50.570 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:50.548+0000 7faca9ffb700 1 -- 172.21.15.42:0/1865277412 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7faca4011340 con 0x7facb41302c0 2019-12-05T05:59:50.570 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:50.548+0000 7facb8e5f700 1 --2- 172.21.15.42:0/1865277412 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7facb41302c0 0x7facb4135700 secure :-1 s=THROTTLE_DONE pgs=1809 cs=0 l=1 rx=0x7faca4001430 tx=0x7faca400c320).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:50.570 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:50.548+0000 7faca9ffb700 1 --2- 172.21.15.42:0/1865277412 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fac9403b450 0x7fac9403d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:59:50.577 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:50.548+0000 7faca9ffb700 1 -- 172.21.15.42:0/1865277412 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9299+0+0 (secure 0 0 0) 0x7faca404b1e0 con 0x7facb41302c0 2019-12-05T05:59:50.578 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:50.548+0000 7facb9660700 1 --2- 172.21.15.42:0/1865277412 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fac9403b450 0x7fac9403d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:59:50.578 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:50.548+0000 7facb9660700 1 --2- 172.21.15.42:0/1865277412 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fac9403b450 0x7fac9403d960 crc :-1 s=READY pgs=773 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:59:50.578 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:50.548+0000 7facbb0c3700 1 -- 172.21.15.42:0/1865277412 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facb405d110 con 0x7facb41302c0 2019-12-05T05:59:50.578 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:50.556+0000 7facb8e5f700 1 --2- 172.21.15.42:0/1865277412 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7facb41302c0 0x7facb4135700 secure :-1 s=THROTTLE_DONE pgs=1809 cs=0 l=1 rx=0x7faca4001430 tx=0x7faca400c320).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:50.578 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:50.556+0000 7faca9ffb700 1 -- 172.21.15.42:0/1865277412 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7faca4032260 con 0x7facb41302c0 2019-12-05T05:59:50.729 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:50.724+0000 7facbb0c3700 1 -- 172.21.15.42:0/1865277412 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7facb405b8c0 con 0x7facb41302c0 2019-12-05T05:59:50.729 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:50.724+0000 7facb8e5f700 1 --2- 172.21.15.42:0/1865277412 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7facb41302c0 0x7facb4135700 secure :-1 s=THROTTLE_DONE pgs=1809 cs=0 l=1 rx=0x7faca4001430 tx=0x7faca400c320).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:50.729 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:50.724+0000 7faca9ffb700 1 -- 172.21.15.42:0/1865277412 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 222 v222) v1 ==== 95+0+1775 (secure 0 0 0) 0x7faca40324c0 con 0x7facb41302c0 2019-12-05T05:59:50.730 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 222 2019-12-05T05:59:50.730 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:59:50.730 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":222,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":222,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:59:49.329977+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":87,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8940},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8940":{"gid":8940,"name":"b","rank":0,"incarnation":219,"state":"up:active","state_seq":36,"addr":"172.21.15.40:6833/1082666540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1082666540},{"type":"v1","addr":"172.21.15.40:6833","nonce":1082666540}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:59:50.732 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:50.728+0000 7facbb0c3700 1 -- 172.21.15.42:0/1865277412 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fac9403b450 msgr2=0x7fac9403d960 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:59:50.732 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:50.728+0000 7facbb0c3700 1 --2- 172.21.15.42:0/1865277412 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fac9403b450 0x7fac9403d960 crc :-1 s=READY pgs=773 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:59:50.732 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:50.728+0000 7facbb0c3700 1 -- 172.21.15.42:0/1865277412 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7facb41302c0 msgr2=0x7facb4135700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:59:50.733 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:50.728+0000 7facbb0c3700 1 --2- 172.21.15.42:0/1865277412 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7facb41302c0 0x7facb4135700 secure :-1 s=READY pgs=1809 cs=0 l=1 rx=0x7faca4001430 tx=0x7faca400c320).stop 2019-12-05T05:59:50.733 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:50.728+0000 7facbb0c3700 1 -- 172.21.15.42:0/1865277412 shutdown_connections 2019-12-05T05:59:50.733 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:50.728+0000 7facbb0c3700 1 --2- 172.21.15.42:0/1865277412 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fac9403b450 0x7fac9403d960 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:59:50.733 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:50.728+0000 7facbb0c3700 1 --2- 172.21.15.42:0/1865277412 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7facb41302c0 0x7facb4135700 unknown :-1 s=CLOSED pgs=1809 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:59:50.733 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:50.728+0000 7facbb0c3700 1 -- 172.21.15.42:0/1865277412 shutdown_connections 2019-12-05T05:59:50.733 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:50.728+0000 7facbb0c3700 1 -- 172.21.15.42:0/1865277412 wait complete. 2019-12-05T05:59:50.733 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:50.728+0000 7facbb0c3700 1 -- 172.21.15.42:0/1865277412 >> 172.21.15.42:0/1865277412 conn(0x7facb40a9820 msgr2=0x7facb409f620 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:59:50.746 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b has gained rank=0, replacing gid=8836 2019-12-05T05:59:50.747 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 4 secs before reviving mds.a 2019-12-05T05:59:54.748 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.a 2019-12-05T05:59:54.748 INFO:tasks.ceph.mds.a:Restarting daemon 2019-12-05T05:59:54.748 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2019-12-05T05:59:54.752 INFO:tasks.ceph.mds.a:Started 2019-12-05T05:59:54.752 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:59:54.812 INFO:tasks.ceph.mds.a.smithi042.stdout:starting mds.a at 2019-12-05T05:59:54.868 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:54.856+0000 7eff23e6d700 1 Processor -- start 2019-12-05T05:59:54.868 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:54.856+0000 7eff23e6d700 1 -- start start 2019-12-05T05:59:54.868 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:54.860+0000 7eff23e6d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7eff1c08df70 0x7eff1c09f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:59:54.868 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:54.860+0000 7eff23e6d700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff1c09fc50 con 0x7eff1c0a7730 2019-12-05T05:59:54.868 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:54.860+0000 7eff23e6d700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff1c0a0700 con 0x7eff1c09f820 2019-12-05T05:59:54.869 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:54.860+0000 7eff23e6d700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff1c0a0af0 con 0x7eff1c08df70 2019-12-05T05:59:54.869 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:54.860+0000 7eff2240a700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7eff1c0a7730 0x7eff1c07e710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:55136/0 (socket says 172.21.15.42:55136) 2019-12-05T05:59:54.869 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:54.860+0000 7eff2240a700 1 -- 172.21.15.42:0/93299399 learned_addr learned my addr 172.21.15.42:0/93299399 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:59:54.869 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:54.860+0000 7eff21c09700 1 --2- 172.21.15.42:0/93299399 >> v2:172.21.15.42:3300/0 conn(0x7eff1c08df70 0x7eff1c09f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:59:54.869 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:54.860+0000 7eff21408700 1 -- 172.21.15.42:0/93299399 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2093326814 0 0) 0x7eff1c09fc50 con 0x7eff1c0a7730 2019-12-05T05:59:54.869 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:54.860+0000 7eff21408700 1 -- 172.21.15.42:0/93299399 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff040030b0 con 0x7eff1c0a7730 2019-12-05T05:59:54.869 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:54.860+0000 7eff21c09700 1 -- 172.21.15.42:0/93299399 >> v1:172.21.15.42:6789/0 conn(0x7eff1c09f820 legacy=0x7eff1c14efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:59:54.870 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:54.860+0000 7eff21408700 1 -- 172.21.15.42:0/93299399 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 116052489 0 0) 0x7eff1c0a0700 con 0x7eff1c09f820 2019-12-05T05:59:54.870 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:54.860+0000 7eff21c09700 1 -- 172.21.15.42:0/93299399 >> v1:172.21.15.40:6789/0 conn(0x7eff1c0a7730 legacy=0x7eff1c07e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:59:54.870 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:54.860+0000 7eff21c09700 1 -- 172.21.15.42:0/93299399 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff1c07ef10 con 0x7eff1c08df70 2019-12-05T05:59:54.870 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:54.860+0000 7eff21c09700 1 --2- 172.21.15.42:0/93299399 >> v2:172.21.15.42:3300/0 conn(0x7eff1c08df70 0x7eff1c09f2d0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7eff10008790 tx=0x7eff1000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:54.870 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:54.860+0000 7eff21c09700 1 --2- 172.21.15.42:0/93299399 >> v2:172.21.15.42:3300/0 conn(0x7eff1c08df70 0x7eff1c09f2d0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7eff10008790 tx=0x7eff1000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:54.871 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:54.860+0000 7eff21c09700 1 --2- 172.21.15.42:0/93299399 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7eff1c08df70 0x7eff1c09f2d0 secure :-1 s=READY pgs=1813 cs=0 l=1 rx=0x7eff10008790 tx=0x7eff1000fbf0).ready entity=mon.1 client_cookie=2d7ce788909678ea server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:59:54.871 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:54.860+0000 7eff21c09700 1 --2- 172.21.15.42:0/93299399 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7eff1c08df70 0x7eff1c09f2d0 secure :-1 s=THROTTLE_DONE pgs=1813 cs=0 l=1 rx=0x7eff10008790 tx=0x7eff1000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:54.871 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:54.860+0000 7eff21408700 1 -- 172.21.15.42:0/93299399 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7eff10011270 con 0x7eff1c08df70 2019-12-05T05:59:54.871 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:54.860+0000 7eff21c09700 1 --2- 172.21.15.42:0/93299399 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7eff1c08df70 0x7eff1c09f2d0 secure :-1 s=THROTTLE_DONE pgs=1813 cs=0 l=1 rx=0x7eff10008790 tx=0x7eff1000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:54.871 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:54.860+0000 7eff21408700 1 -- 172.21.15.42:0/93299399 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff10008bc0 con 0x7eff1c08df70 2019-12-05T05:59:54.871 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:54.860+0000 7eff21c09700 1 --2- 172.21.15.42:0/93299399 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7eff1c08df70 0x7eff1c09f2d0 secure :-1 s=THROTTLE_DONE pgs=1813 cs=0 l=1 rx=0x7eff10008790 tx=0x7eff1000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:54.872 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:54.860+0000 7eff21408700 1 -- 172.21.15.42:0/93299399 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7eff10011850 con 0x7eff1c08df70 2019-12-05T05:59:54.872 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:54.860+0000 7eff23e6d700 1 -- 172.21.15.42:0/93299399 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7eff1c08df70 msgr2=0x7eff1c09f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:59:54.872 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:54.860+0000 7eff23e6d700 1 --2- 172.21.15.42:0/93299399 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7eff1c08df70 0x7eff1c09f2d0 secure :-1 s=READY pgs=1813 cs=0 l=1 rx=0x7eff10008790 tx=0x7eff1000fbf0).stop 2019-12-05T05:59:54.872 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:54.860+0000 7eff23e6d700 1 -- 172.21.15.42:0/93299399 shutdown_connections 2019-12-05T05:59:54.873 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:54.860+0000 7eff23e6d700 1 --2- 172.21.15.42:0/93299399 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7eff1c08df70 0x7eff1c09f2d0 unknown :-1 s=CLOSED pgs=1813 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:59:54.873 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:54.860+0000 7eff23e6d700 1 -- 172.21.15.42:0/93299399 shutdown_connections 2019-12-05T05:59:54.874 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:54.860+0000 7eff23e6d700 1 -- 172.21.15.42:0/93299399 wait complete. 2019-12-05T05:59:54.874 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:54.860+0000 7eff23e6d700 1 -- 172.21.15.42:0/93299399 >> 172.21.15.42:0/93299399 conn(0x7eff1c0aabd0 msgr2=0x7eff1c0a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:59:54.878 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:54.864+0000 7eff23e6d700 1 Processor -- start 2019-12-05T05:59:54.878 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:54.864+0000 7eff23e6d700 1 -- start start 2019-12-05T05:59:54.878 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:54.864+0000 7eff23e6d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7eff1c12f1a0 0x7eff1c12f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:59:54.878 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:54.864+0000 7eff23e6d700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff1c132c20 con 0x7eff1c1221d0 2019-12-05T05:59:54.879 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:54.864+0000 7eff23e6d700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff1c12fb20 con 0x7eff1c0742d0 2019-12-05T05:59:54.879 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:54.864+0000 7eff23e6d700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff1c133010 con 0x7eff1c12f1a0 2019-12-05T05:59:54.879 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:54.864+0000 7eff21c09700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7eff1c12f1a0 0x7eff1c12f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:59:54.879 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:54.864+0000 7eff21c09700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7eff1c12f1a0 0x7eff1c12f5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:45666/0 (socket says 172.21.15.42:45666) 2019-12-05T05:59:54.880 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:54.864+0000 7eff21c09700 1 -- 172.21.15.42:0/2170964202 learned_addr learned my addr 172.21.15.42:0/2170964202 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:59:54.880 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:54.864+0000 7eff21c09700 1 -- 172.21.15.42:0/2170964202 >> v1:172.21.15.42:6789/0 conn(0x7eff1c0742d0 legacy=0x7eff1c076710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:59:54.880 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:54.864+0000 7eff21c09700 1 -- 172.21.15.42:0/2170964202 >> v1:172.21.15.40:6789/0 conn(0x7eff1c1221d0 legacy=0x7eff1c132420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:59:54.880 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:54.864+0000 7eff21c09700 1 -- 172.21.15.42:0/2170964202 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff1c079f40 con 0x7eff1c12f1a0 2019-12-05T05:59:54.881 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:54.864+0000 7eff21c09700 1 --2- 172.21.15.42:0/2170964202 >> v2:172.21.15.42:3300/0 conn(0x7eff1c12f1a0 0x7eff1c12f5d0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7eff1c122710 tx=0x7eff10008df0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:54.881 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:54.864+0000 7eff21c09700 1 --2- 172.21.15.42:0/2170964202 >> v2:172.21.15.42:3300/0 conn(0x7eff1c12f1a0 0x7eff1c12f5d0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7eff1c122710 tx=0x7eff10008df0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:54.881 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:54.864+0000 7eff21c09700 1 --2- 172.21.15.42:0/2170964202 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7eff1c12f1a0 0x7eff1c12f5d0 secure :-1 s=READY pgs=1815 cs=0 l=1 rx=0x7eff1c122710 tx=0x7eff10008df0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:59:54.881 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:54.864+0000 7eff21c09700 1 --2- 172.21.15.42:0/2170964202 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7eff1c12f1a0 0x7eff1c12f5d0 secure :-1 s=THROTTLE_DONE pgs=1815 cs=0 l=1 rx=0x7eff1c122710 tx=0x7eff10008df0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:54.882 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:54.864+0000 7eff21c09700 1 --2- 172.21.15.42:0/2170964202 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7eff1c12f1a0 0x7eff1c12f5d0 secure :-1 s=THROTTLE_DONE pgs=1815 cs=0 l=1 rx=0x7eff1c122710 tx=0x7eff10008df0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:54.882 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:54.864+0000 7eff1affd700 1 -- 172.21.15.42:0/2170964202 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7eff10007150 con 0x7eff1c12f1a0 2019-12-05T05:59:54.882 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:54.864+0000 7eff21c09700 1 --2- 172.21.15.42:0/2170964202 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7eff1c12f1a0 0x7eff1c12f5d0 secure :-1 s=THROTTLE_DONE pgs=1815 cs=0 l=1 rx=0x7eff1c122710 tx=0x7eff10008df0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:54.882 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:54.868+0000 7eff1affd700 1 -- 172.21.15.42:0/2170964202 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff10011470 con 0x7eff1c12f1a0 2019-12-05T05:59:54.883 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:54.868+0000 7eff23e6d700 1 -- 172.21.15.42:0/2170964202 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff1c077f20 con 0x7eff1c12f1a0 2019-12-05T05:59:54.883 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:54.868+0000 7eff1affd700 1 -- 172.21.15.42:0/2170964202 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7eff1000f550 con 0x7eff1c12f1a0 2019-12-05T05:59:54.883 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:54.868+0000 7eff23e6d700 1 -- 172.21.15.42:0/2170964202 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff1c078420 con 0x7eff1c12f1a0 2019-12-05T05:59:54.883 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:54.868+0000 7eff21c09700 1 --2- 172.21.15.42:0/2170964202 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7eff1c12f1a0 0x7eff1c12f5d0 secure :-1 s=THROTTLE_DONE pgs=1815 cs=0 l=1 rx=0x7eff1c122710 tx=0x7eff10008df0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:54.885 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:54.868+0000 7eff1affd700 1 -- 172.21.15.42:0/2170964202 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7eff10005540 con 0x7eff1c12f1a0 2019-12-05T05:59:54.885 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:54.868+0000 7eff21c09700 1 --2- 172.21.15.42:0/2170964202 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7eff1c12f1a0 0x7eff1c12f5d0 secure :-1 s=THROTTLE_DONE pgs=1815 cs=0 l=1 rx=0x7eff1c122710 tx=0x7eff10008df0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:54.885 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:54.868+0000 7eff1affd700 1 --2- 172.21.15.42:0/2170964202 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7efefc03ad30 0x7efefc03d240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:59:54.885 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:54.868+0000 7eff1affd700 1 -- 172.21.15.42:0/2170964202 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9299+0+0 (secure 0 0 0) 0x7eff1004bcb0 con 0x7eff1c12f1a0 2019-12-05T05:59:54.886 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:54.868+0000 7eff2240a700 1 --2- 172.21.15.42:0/2170964202 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7efefc03ad30 0x7efefc03d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:59:54.886 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:54.868+0000 7eff2240a700 1 --2- 172.21.15.42:0/2170964202 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7efefc03ad30 0x7efefc03d240 crc :-1 s=READY pgs=775 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:59:54.887 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:54.872+0000 7eff23e6d700 1 -- 172.21.15.42:0/2170964202 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff1c05d100 con 0x7eff1c12f1a0 2019-12-05T05:59:54.888 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:54.880+0000 7eff21c09700 1 --2- 172.21.15.42:0/2170964202 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7eff1c12f1a0 0x7eff1c12f5d0 secure :-1 s=THROTTLE_DONE pgs=1815 cs=0 l=1 rx=0x7eff1c122710 tx=0x7eff10008df0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:54.888 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:54.880+0000 7eff1affd700 1 -- 172.21.15.42:0/2170964202 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7eff100309d0 con 0x7eff1c12f1a0 2019-12-05T05:59:55.063 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:55.056+0000 7eff23e6d700 1 -- 172.21.15.42:0/2170964202 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7eff1c05b8b0 con 0x7eff1c12f1a0 2019-12-05T05:59:55.064 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:55.060+0000 7eff21c09700 1 --2- 172.21.15.42:0/2170964202 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7eff1c12f1a0 0x7eff1c12f5d0 secure :-1 s=THROTTLE_DONE pgs=1815 cs=0 l=1 rx=0x7eff1c122710 tx=0x7eff10008df0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:55.064 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:55.060+0000 7eff1affd700 1 -- 172.21.15.42:0/2170964202 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 222 v222) v1 ==== 95+0+1775 (secure 0 0 0) 0x7eff10031900 con 0x7eff1c12f1a0 2019-12-05T05:59:55.064 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 222 2019-12-05T05:59:55.064 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:59:55.064 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":222,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":222,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:59:49.329977+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":87,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8940},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8940":{"gid":8940,"name":"b","rank":0,"incarnation":219,"state":"up:active","state_seq":36,"addr":"172.21.15.40:6833/1082666540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1082666540},{"type":"v1","addr":"172.21.15.40:6833","nonce":1082666540}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:59:55.066 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:55.060+0000 7eff23e6d700 1 -- 172.21.15.42:0/2170964202 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7efefc03ad30 msgr2=0x7efefc03d240 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:59:55.067 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:55.060+0000 7eff23e6d700 1 --2- 172.21.15.42:0/2170964202 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7efefc03ad30 0x7efefc03d240 crc :-1 s=READY pgs=775 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:59:55.067 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:55.060+0000 7eff23e6d700 1 -- 172.21.15.42:0/2170964202 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7eff1c12f1a0 msgr2=0x7eff1c12f5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:59:55.067 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:55.060+0000 7eff23e6d700 1 --2- 172.21.15.42:0/2170964202 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7eff1c12f1a0 0x7eff1c12f5d0 secure :-1 s=READY pgs=1815 cs=0 l=1 rx=0x7eff1c122710 tx=0x7eff10008df0).stop 2019-12-05T05:59:55.067 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:55.060+0000 7eff23e6d700 1 -- 172.21.15.42:0/2170964202 shutdown_connections 2019-12-05T05:59:55.067 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:55.060+0000 7eff23e6d700 1 --2- 172.21.15.42:0/2170964202 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7efefc03ad30 0x7efefc03d240 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:59:55.067 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:55.060+0000 7eff23e6d700 1 --2- 172.21.15.42:0/2170964202 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7eff1c12f1a0 0x7eff1c12f5d0 unknown :-1 s=CLOSED pgs=1815 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:59:55.068 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:55.060+0000 7eff23e6d700 1 -- 172.21.15.42:0/2170964202 shutdown_connections 2019-12-05T05:59:55.068 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:55.060+0000 7eff23e6d700 1 -- 172.21.15.42:0/2170964202 wait complete. 2019-12-05T05:59:55.068 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:55.060+0000 7eff23e6d700 1 -- 172.21.15.42:0/2170964202 >> 172.21.15.42:0/2170964202 conn(0x7eff1c0a8fc0 msgr2=0x7eff1c11d610 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:59:55.077 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2019-12-05T05:59:57.080 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:59:57.197 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.188+0000 7f3fd4dea700 1 Processor -- start 2019-12-05T05:59:57.198 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.188+0000 7f3fd4dea700 1 -- start start 2019-12-05T05:59:57.198 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.188+0000 7f3fd4dea700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f3fd00a7740 0x7f3fd008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:59:57.198 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.188+0000 7f3fd4dea700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fd00a0710 con 0x7f3fd007e720 2019-12-05T05:59:57.198 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.188+0000 7f3fd4dea700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fd00a9ba0 con 0x7f3fd009fae0 2019-12-05T05:59:57.198 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.188+0000 7f3fd4dea700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3fd00a0b00 con 0x7f3fd00a7740 2019-12-05T05:59:57.198 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.188+0000 7f3fcf59e700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f3fd009fae0 0x7f3fd014efe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:45732/0 (socket says 172.21.15.42:45732) 2019-12-05T05:59:57.198 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.188+0000 7f3fcf59e700 1 -- 172.21.15.42:0/3575843226 learned_addr learned my addr 172.21.15.42:0/3575843226 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:59:57.199 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.188+0000 7f3fced9d700 1 --2- 172.21.15.42:0/3575843226 >> v2:172.21.15.40:3300/0 conn(0x7f3fd00a7740 0x7f3fd008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:59:57.199 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.188+0000 7f3fcdd9b700 1 -- 172.21.15.42:0/3575843226 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1451666533 0 0) 0x7f3fd00a9ba0 con 0x7f3fd009fae0 2019-12-05T05:59:57.199 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.188+0000 7f3fcdd9b700 1 -- 172.21.15.42:0/3575843226 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3fb80030b0 con 0x7f3fd009fae0 2019-12-05T05:59:57.199 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.188+0000 7f3fced9d700 1 -- 172.21.15.42:0/3575843226 >> v1:172.21.15.42:6789/0 conn(0x7f3fd009fae0 legacy=0x7f3fd014efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:59:57.199 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.188+0000 7f3fced9d700 1 -- 172.21.15.42:0/3575843226 >> v1:172.21.15.40:6789/0 conn(0x7f3fd007e720 legacy=0x7f3fd009f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:59:57.199 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.188+0000 7f3fced9d700 1 -- 172.21.15.42:0/3575843226 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fd007eb50 con 0x7f3fd00a7740 2019-12-05T05:59:57.199 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.188+0000 7f3fced9d700 1 --2- 172.21.15.42:0/3575843226 >> v2:172.21.15.40:3300/0 conn(0x7f3fd00a7740 0x7f3fd008df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f3fc0008790 tx=0x7f3fc000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:57.200 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.188+0000 7f3fced9d700 1 --2- 172.21.15.42:0/3575843226 >> v2:172.21.15.40:3300/0 conn(0x7f3fd00a7740 0x7f3fd008df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f3fc0008790 tx=0x7f3fc000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:57.200 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.188+0000 7f3fced9d700 1 --2- 172.21.15.42:0/3575843226 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3fd00a7740 0x7f3fd008df80 secure :-1 s=READY pgs=1393 cs=0 l=1 rx=0x7f3fc0008790 tx=0x7f3fc000fbf0).ready entity=mon.0 client_cookie=3eaa00241b377ba1 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:59:57.200 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.188+0000 7f3fced9d700 1 --2- 172.21.15.42:0/3575843226 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3fd00a7740 0x7f3fd008df80 secure :-1 s=THROTTLE_DONE pgs=1393 cs=0 l=1 rx=0x7f3fc0008790 tx=0x7f3fc000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:57.200 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.188+0000 7f3fcdd9b700 1 -- 172.21.15.42:0/3575843226 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3fc0011270 con 0x7f3fd00a7740 2019-12-05T05:59:57.200 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.188+0000 7f3fced9d700 1 --2- 172.21.15.42:0/3575843226 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3fd00a7740 0x7f3fd008df80 secure :-1 s=THROTTLE_DONE pgs=1393 cs=0 l=1 rx=0x7f3fc0008790 tx=0x7f3fc000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:57.201 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.188+0000 7f3fcdd9b700 1 -- 172.21.15.42:0/3575843226 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fc0008bc0 con 0x7f3fd00a7740 2019-12-05T05:59:57.201 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.188+0000 7f3fced9d700 1 --2- 172.21.15.42:0/3575843226 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3fd00a7740 0x7f3fd008df80 secure :-1 s=THROTTLE_DONE pgs=1393 cs=0 l=1 rx=0x7f3fc0008790 tx=0x7f3fc000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:57.201 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.188+0000 7f3fcdd9b700 1 -- 172.21.15.42:0/3575843226 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3fc0011850 con 0x7f3fd00a7740 2019-12-05T05:59:57.201 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.188+0000 7f3fd4dea700 1 -- 172.21.15.42:0/3575843226 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3fd00a7740 msgr2=0x7f3fd008df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:59:57.201 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.188+0000 7f3fd4dea700 1 --2- 172.21.15.42:0/3575843226 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3fd00a7740 0x7f3fd008df80 secure :-1 s=READY pgs=1393 cs=0 l=1 rx=0x7f3fc0008790 tx=0x7f3fc000fbf0).stop 2019-12-05T05:59:57.201 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.192+0000 7f3fd4dea700 1 -- 172.21.15.42:0/3575843226 shutdown_connections 2019-12-05T05:59:57.201 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.192+0000 7f3fd4dea700 1 --2- 172.21.15.42:0/3575843226 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3fd00a7740 0x7f3fd008df80 unknown :-1 s=CLOSED pgs=1393 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:59:57.202 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.192+0000 7f3fd4dea700 1 -- 172.21.15.42:0/3575843226 shutdown_connections 2019-12-05T05:59:57.202 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.192+0000 7f3fd4dea700 1 -- 172.21.15.42:0/3575843226 wait complete. 2019-12-05T05:59:57.202 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.192+0000 7f3fd4dea700 1 -- 172.21.15.42:0/3575843226 >> 172.21.15.42:0/3575843226 conn(0x7f3fd00aabe0 msgr2=0x7f3fd00a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:59:57.204 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.196+0000 7f3fd4dea700 1 Processor -- start 2019-12-05T05:59:57.204 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.196+0000 7f3fd4dea700 1 -- start start 2019-12-05T05:59:57.204 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.196+0000 7f3fd4dea700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3fd011bfe0 0x7f3fd0124440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:59:57.205 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.196+0000 7f3fd4dea700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3fd011f790 0x7f3fd011fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:59:57.205 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.196+0000 7f3fd4dea700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fd011f420 con 0x7f3fd0121070 2019-12-05T05:59:57.205 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.196+0000 7f3fd4dea700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3fd0120740 con 0x7f3fd011f790 2019-12-05T05:59:57.205 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.196+0000 7f3fd4dea700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fd011a410 con 0x7f3fd011bfe0 2019-12-05T05:59:57.205 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.196+0000 7f3fced9d700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f3fd0121070 0x7f3fd007d970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:45736/0 (socket says 172.21.15.42:45736) 2019-12-05T05:59:57.205 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.196+0000 7f3fced9d700 1 -- 172.21.15.42:0/47591493 learned_addr learned my addr 172.21.15.42:0/47591493 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:59:57.205 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.196+0000 7f3fcf59e700 1 --2- 172.21.15.42:0/47591493 >> v2:172.21.15.42:3300/0 conn(0x7f3fd011f790 0x7f3fd011fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:59:57.206 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.196+0000 7f3fce59c700 1 --2- 172.21.15.42:0/47591493 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3fd011bfe0 0x7f3fd0124440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:59:57.206 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.196+0000 7f3fbf7fe700 1 -- 172.21.15.42:0/47591493 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2896108411 0 0) 0x7f3fd011f420 con 0x7f3fd0121070 2019-12-05T05:59:57.206 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.196+0000 7f3fbf7fe700 1 -- 172.21.15.42:0/47591493 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3fb40030b0 con 0x7f3fd0121070 2019-12-05T05:59:57.206 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.196+0000 7f3fce59c700 1 -- 172.21.15.42:0/47591493 >> v2:172.21.15.42:3300/0 conn(0x7f3fd011f790 msgr2=0x7f3fd011fbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:59:57.206 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.196+0000 7f3fce59c700 1 --2- 172.21.15.42:0/47591493 >> v2:172.21.15.42:3300/0 conn(0x7f3fd011f790 0x7f3fd011fbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:59:57.206 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.196+0000 7f3fce59c700 1 -- 172.21.15.42:0/47591493 >> v1:172.21.15.42:6789/0 conn(0x7f3fd0121070 legacy=0x7f3fd007d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:59:57.207 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.196+0000 7f3fce59c700 1 -- 172.21.15.42:0/47591493 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fd008eb70 con 0x7f3fd011bfe0 2019-12-05T05:59:57.207 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.200+0000 7f3fce59c700 1 --2- 172.21.15.42:0/47591493 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3fd011bfe0 0x7f3fd0124440 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f3fc4000e90 tx=0x7f3fc40108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:57.207 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.200+0000 7f3fce59c700 1 --2- 172.21.15.42:0/47591493 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3fd011bfe0 0x7f3fd0124440 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f3fc4000e90 tx=0x7f3fc40108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:57.207 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.200+0000 7f3fce59c700 1 --2- 172.21.15.42:0/47591493 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3fd011bfe0 0x7f3fd0124440 secure :-1 s=READY pgs=407 cs=0 l=1 rx=0x7f3fc4000e90 tx=0x7f3fc40108f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:59:57.207 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.200+0000 7f3fce59c700 1 --2- 172.21.15.42:0/47591493 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3fd011bfe0 0x7f3fd0124440 secure :-1 s=THROTTLE_DONE pgs=407 cs=0 l=1 rx=0x7f3fc4000e90 tx=0x7f3fc40108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:57.208 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.200+0000 7f3fbf7fe700 1 -- 172.21.15.42:0/47591493 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3fc400c990 con 0x7f3fd011bfe0 2019-12-05T05:59:57.208 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.200+0000 7f3fce59c700 1 --2- 172.21.15.42:0/47591493 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3fd011bfe0 0x7f3fd0124440 secure :-1 s=THROTTLE_DONE pgs=407 cs=0 l=1 rx=0x7f3fc4000e90 tx=0x7f3fc40108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:57.208 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.200+0000 7f3fd4dea700 1 -- 172.21.15.42:0/47591493 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fd008ee60 con 0x7f3fd011bfe0 2019-12-05T05:59:57.208 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.200+0000 7f3fbf7fe700 1 -- 172.21.15.42:0/47591493 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fc400c5b0 con 0x7f3fd011bfe0 2019-12-05T05:59:57.208 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.200+0000 7f3fce59c700 1 --2- 172.21.15.42:0/47591493 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3fd011bfe0 0x7f3fd0124440 secure :-1 s=THROTTLE_DONE pgs=407 cs=0 l=1 rx=0x7f3fc4000e90 tx=0x7f3fc40108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:57.208 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.200+0000 7f3fbf7fe700 1 -- 172.21.15.42:0/47591493 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3fc4010f00 con 0x7f3fd011bfe0 2019-12-05T05:59:57.208 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.200+0000 7f3fd4dea700 1 -- 172.21.15.42:0/47591493 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fd008f360 con 0x7f3fd011bfe0 2019-12-05T05:59:57.210 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.200+0000 7f3fce59c700 1 --2- 172.21.15.42:0/47591493 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3fd011bfe0 0x7f3fd0124440 secure :-1 s=THROTTLE_DONE pgs=407 cs=0 l=1 rx=0x7f3fc4000e90 tx=0x7f3fc40108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:57.210 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.200+0000 7f3fbf7fe700 1 -- 172.21.15.42:0/47591493 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f3fc4007430 con 0x7f3fd011bfe0 2019-12-05T05:59:57.210 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.200+0000 7f3fce59c700 1 --2- 172.21.15.42:0/47591493 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3fd011bfe0 0x7f3fd0124440 secure :-1 s=THROTTLE_DONE pgs=407 cs=0 l=1 rx=0x7f3fc4000e90 tx=0x7f3fc40108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:57.210 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.200+0000 7f3fbf7fe700 1 --2- 172.21.15.42:0/47591493 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3fb403b220 0x7f3fb403d730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:59:57.210 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.200+0000 7f3fbf7fe700 1 -- 172.21.15.42:0/47591493 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9299+0+0 (secure 0 0 0) 0x7f3fc404b130 con 0x7f3fd011bfe0 2019-12-05T05:59:57.210 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.200+0000 7f3fced9d700 1 --2- 172.21.15.42:0/47591493 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3fb403b220 0x7f3fb403d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:59:57.211 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.200+0000 7f3fd4dea700 1 -- 172.21.15.42:0/47591493 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fd008fa80 con 0x7f3fd011bfe0 2019-12-05T05:59:57.211 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.200+0000 7f3fced9d700 1 --2- 172.21.15.42:0/47591493 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3fb403b220 0x7f3fb403d730 crc :-1 s=READY pgs=777 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:59:57.215 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.212+0000 7f3fce59c700 1 --2- 172.21.15.42:0/47591493 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3fd011bfe0 0x7f3fd0124440 secure :-1 s=THROTTLE_DONE pgs=407 cs=0 l=1 rx=0x7f3fc4000e90 tx=0x7f3fc40108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:57.216 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.212+0000 7f3fbf7fe700 1 -- 172.21.15.42:0/47591493 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f3fc4032190 con 0x7f3fd011bfe0 2019-12-05T05:59:57.392 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.388+0000 7f3fd4dea700 1 -- 172.21.15.42:0/47591493 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3fd005d110 con 0x7f3fd011bfe0 2019-12-05T05:59:57.393 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.388+0000 7f3fce59c700 1 --2- 172.21.15.42:0/47591493 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3fd011bfe0 0x7f3fd0124440 secure :-1 s=THROTTLE_DONE pgs=407 cs=0 l=1 rx=0x7f3fc4000e90 tx=0x7f3fc40108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:57.393 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.388+0000 7f3fbf7fe700 1 -- 172.21.15.42:0/47591493 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 223 v223) v1 ==== 95+0+2109 (secure 0 0 0) 0x7f3fc40323f0 con 0x7f3fd011bfe0 2019-12-05T05:59:57.393 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 223 2019-12-05T05:59:57.393 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:59:57.394 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":223,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":9061,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6835/448869597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":448869597},{"type":"v1","addr":"172.21.15.42:6835","nonce":448869597}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":223}],"filesystems":[{"mdsmap":{"epoch":222,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:59:49.329977+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":87,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8940},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8940":{"gid":8940,"name":"b","rank":0,"incarnation":219,"state":"up:active","state_seq":36,"addr":"172.21.15.40:6833/1082666540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1082666540},{"type":"v1","addr":"172.21.15.40:6833","nonce":1082666540}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:59:57.396 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.388+0000 7f3fd4dea700 1 -- 172.21.15.42:0/47591493 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3fb403b220 msgr2=0x7f3fb403d730 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:59:57.396 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.388+0000 7f3fd4dea700 1 --2- 172.21.15.42:0/47591493 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3fb403b220 0x7f3fb403d730 crc :-1 s=READY pgs=777 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:59:57.396 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.388+0000 7f3fd4dea700 1 -- 172.21.15.42:0/47591493 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3fd011bfe0 msgr2=0x7f3fd0124440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:59:57.396 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.388+0000 7f3fd4dea700 1 --2- 172.21.15.42:0/47591493 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3fd011bfe0 0x7f3fd0124440 secure :-1 s=READY pgs=407 cs=0 l=1 rx=0x7f3fc4000e90 tx=0x7f3fc40108f0).stop 2019-12-05T05:59:57.397 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.392+0000 7f3fd4dea700 1 -- 172.21.15.42:0/47591493 shutdown_connections 2019-12-05T05:59:57.397 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.392+0000 7f3fd4dea700 1 --2- 172.21.15.42:0/47591493 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3fb403b220 0x7f3fb403d730 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:59:57.397 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.392+0000 7f3fd4dea700 1 --2- 172.21.15.42:0/47591493 >> v2:172.21.15.42:3300/0 conn(0x7f3fd011f790 0x7f3fd011fbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:59:57.398 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.392+0000 7f3fd4dea700 1 --2- 172.21.15.42:0/47591493 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3fd011bfe0 0x7f3fd0124440 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:59:57.398 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.392+0000 7f3fd4dea700 1 -- 172.21.15.42:0/47591493 shutdown_connections 2019-12-05T05:59:57.398 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.392+0000 7f3fd4dea700 1 -- 172.21.15.42:0/47591493 wait complete. 2019-12-05T05:59:57.398 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.392+0000 7f3fd4dea700 1 -- 172.21.15.42:0/47591493 >> 172.21.15.42:0/47591493 conn(0x7f3fd007e4f0 msgr2=0x7f3fd00a9770 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:59:57.405 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a reported in up:standby state 2019-12-05T05:59:57.405 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:59:57.514 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.500+0000 7f74f85b8700 1 Processor -- start 2019-12-05T05:59:57.515 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.500+0000 7f74f85b8700 1 -- start start 2019-12-05T05:59:57.515 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.504+0000 7f74f85b8700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f74f00a7740 0x7f74f008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:59:57.515 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.504+0000 7f74f85b8700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f74f009fae0 0x7f74f007eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:59:57.515 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.504+0000 7f74f85b8700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74f00a0710 con 0x7f74f007e720 2019-12-05T05:59:57.515 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.504+0000 7f74f85b8700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74f00a0b00 con 0x7f74f00a7740 2019-12-05T05:59:57.516 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.504+0000 7f74f85b8700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74f008e5b0 con 0x7f74f009fae0 2019-12-05T05:59:57.516 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.504+0000 7f74f7356700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f74f009fae0 0x7f74f007eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:59:57.516 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.504+0000 7f74f7356700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f74f009fae0 0x7f74f007eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:45710/0 (socket says 172.21.15.42:45710) 2019-12-05T05:59:57.516 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.504+0000 7f74f7356700 1 -- 172.21.15.42:0/2404509987 learned_addr learned my addr 172.21.15.42:0/2404509987 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:59:57.516 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.504+0000 7f74f6b55700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f74f00a7740 0x7f74f008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:59:57.516 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.504+0000 7f74f7356700 1 -- 172.21.15.42:0/2404509987 >> v2:172.21.15.40:3300/0 conn(0x7f74f00a7740 msgr2=0x7f74f008df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T05:59:57.516 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.504+0000 7f74f7356700 1 --2- 172.21.15.42:0/2404509987 >> v2:172.21.15.40:3300/0 conn(0x7f74f00a7740 0x7f74f008df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:59:57.517 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.504+0000 7f74f7356700 1 -- 172.21.15.42:0/2404509987 >> v1:172.21.15.42:6789/0 conn(0x7f74f007e720 legacy=0x7f74f009f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:59:57.517 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.504+0000 7f74f7356700 1 -- 172.21.15.42:0/2404509987 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74f007f0a0 con 0x7f74f009fae0 2019-12-05T05:59:57.517 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.504+0000 7f74f7356700 1 --2- 172.21.15.42:0/2404509987 >> v2:172.21.15.42:3300/0 conn(0x7f74f009fae0 0x7f74f007eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f74e80032b0 tx=0x7f74e80100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:57.517 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.504+0000 7f74f7356700 1 --2- 172.21.15.42:0/2404509987 >> v2:172.21.15.42:3300/0 conn(0x7f74f009fae0 0x7f74f007eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f74e80032b0 tx=0x7f74e80100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:57.517 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.504+0000 7f74f7356700 1 --2- 172.21.15.42:0/2404509987 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f74f009fae0 0x7f74f007eb50 secure :-1 s=READY pgs=1819 cs=0 l=1 rx=0x7f74e80032b0 tx=0x7f74e80100b0).ready entity=mon.1 client_cookie=33867426b24e406d server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:59:57.518 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.504+0000 7f74f7356700 1 --2- 172.21.15.42:0/2404509987 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f74f009fae0 0x7f74f007eb50 secure :-1 s=THROTTLE_DONE pgs=1819 cs=0 l=1 rx=0x7f74e80032b0 tx=0x7f74e80100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:57.518 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.504+0000 7f74f5b53700 1 -- 172.21.15.42:0/2404509987 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f74e8010d60 con 0x7f74f009fae0 2019-12-05T05:59:57.519 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.504+0000 7f74f7356700 1 --2- 172.21.15.42:0/2404509987 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f74f009fae0 0x7f74f007eb50 secure :-1 s=THROTTLE_DONE pgs=1819 cs=0 l=1 rx=0x7f74e80032b0 tx=0x7f74e80100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:57.519 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.504+0000 7f74f5b53700 1 -- 172.21.15.42:0/2404509987 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74e8010980 con 0x7f74f009fae0 2019-12-05T05:59:57.519 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.504+0000 7f74f7356700 1 --2- 172.21.15.42:0/2404509987 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f74f009fae0 0x7f74f007eb50 secure :-1 s=THROTTLE_DONE pgs=1819 cs=0 l=1 rx=0x7f74e80032b0 tx=0x7f74e80100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:57.519 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.504+0000 7f74f85b8700 1 -- 172.21.15.42:0/2404509987 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f74f009fae0 msgr2=0x7f74f007eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:59:57.520 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.504+0000 7f74f5b53700 1 -- 172.21.15.42:0/2404509987 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f74e80114a0 con 0x7f74f009fae0 2019-12-05T05:59:57.520 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.504+0000 7f74f85b8700 1 --2- 172.21.15.42:0/2404509987 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f74f009fae0 0x7f74f007eb50 secure :-1 s=READY pgs=1819 cs=0 l=1 rx=0x7f74e80032b0 tx=0x7f74e80100b0).stop 2019-12-05T05:59:57.520 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.504+0000 7f74f85b8700 1 -- 172.21.15.42:0/2404509987 shutdown_connections 2019-12-05T05:59:57.520 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.504+0000 7f74f85b8700 1 --2- 172.21.15.42:0/2404509987 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f74f009fae0 0x7f74f007eb50 unknown :-1 s=CLOSED pgs=1819 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:59:57.520 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.504+0000 7f74f85b8700 1 --2- 172.21.15.42:0/2404509987 >> v2:172.21.15.40:3300/0 conn(0x7f74f00a7740 0x7f74f008df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T05:59:57.521 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.504+0000 7f74f85b8700 1 -- 172.21.15.42:0/2404509987 shutdown_connections 2019-12-05T05:59:57.521 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.504+0000 7f74f85b8700 1 -- 172.21.15.42:0/2404509987 wait complete. 2019-12-05T05:59:57.521 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.504+0000 7f74f85b8700 1 -- 172.21.15.42:0/2404509987 >> 172.21.15.42:0/2404509987 conn(0x7f74f00aabe0 msgr2=0x7f74f00a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:59:57.521 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.508+0000 7f74f85b8700 1 Processor -- start 2019-12-05T05:59:57.521 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.508+0000 7f74f85b8700 1 -- start start 2019-12-05T05:59:57.523 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.508+0000 7f74f85b8700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f74f0119730 0x7f74f0063d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:59:57.524 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.508+0000 7f74f85b8700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f74f0064fe0 0x7f74f0118420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:59:57.525 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.508+0000 7f74f85b8700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74f0065410 con 0x7f74f0118970 2019-12-05T05:59:57.525 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.508+0000 7f74f85b8700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74f0118da0 con 0x7f74f0119730 2019-12-05T05:59:57.528 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.508+0000 7f74f85b8700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74f01307a0 con 0x7f74f0064fe0 2019-12-05T05:59:57.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.508+0000 7f74f6b55700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f74f0119730 0x7f74f0063d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:59:57.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.508+0000 7f74f6354700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f74f0064fe0 0x7f74f0118420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:59:57.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.508+0000 7f74f7356700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f74f0118970 0x7f74f007c7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:55194/0 (socket says 172.21.15.42:55194) 2019-12-05T05:59:57.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.508+0000 7f74f7356700 1 -- 172.21.15.42:0/2537239456 learned_addr learned my addr 172.21.15.42:0/2537239456 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:59:57.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.508+0000 7f74f6b55700 1 -- 172.21.15.42:0/2537239456 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f74f0064fe0 msgr2=0x7f74f0118420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:59:57.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.508+0000 7f74e77fe700 1 -- 172.21.15.42:0/2537239456 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 180361103 0 0) 0x7f74f0065410 con 0x7f74f0118970 2019-12-05T05:59:57.530 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.508+0000 7f74f6b55700 1 --2- 172.21.15.42:0/2537239456 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f74f0064fe0 0x7f74f0118420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:59:57.530 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.508+0000 7f74f6b55700 1 -- 172.21.15.42:0/2537239456 >> v1:172.21.15.40:6789/0 conn(0x7f74f0118970 legacy=0x7f74f007c7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:59:57.530 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.508+0000 7f74f6b55700 1 -- 172.21.15.42:0/2537239456 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74f007cfb0 con 0x7f74f0119730 2019-12-05T05:59:57.531 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.508+0000 7f74f6b55700 1 --2- 172.21.15.42:0/2537239456 >> v2:172.21.15.40:3300/0 conn(0x7f74f0119730 0x7f74f0063d40 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f74ec002f30 tx=0x7f74ec0109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:57.531 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.508+0000 7f74f6b55700 1 --2- 172.21.15.42:0/2537239456 >> v2:172.21.15.40:3300/0 conn(0x7f74f0119730 0x7f74f0063d40 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f74ec002f30 tx=0x7f74ec0109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:57.531 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.508+0000 7f74f6b55700 1 --2- 172.21.15.42:0/2537239456 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f74f0119730 0x7f74f0063d40 secure :-1 s=READY pgs=1395 cs=0 l=1 rx=0x7f74ec002f30 tx=0x7f74ec0109e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:59:57.536 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.512+0000 7f74f6b55700 1 --2- 172.21.15.42:0/2537239456 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f74f0119730 0x7f74f0063d40 secure :-1 s=THROTTLE_DONE pgs=1395 cs=0 l=1 rx=0x7f74ec002f30 tx=0x7f74ec0109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:57.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.512+0000 7f74e77fe700 1 -- 172.21.15.42:0/2537239456 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f74ec005920 con 0x7f74f0119730 2019-12-05T05:59:57.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.512+0000 7f74f6b55700 1 --2- 172.21.15.42:0/2537239456 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f74f0119730 0x7f74f0063d40 secure :-1 s=THROTTLE_DONE pgs=1395 cs=0 l=1 rx=0x7f74ec002f30 tx=0x7f74ec0109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:57.539 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.512+0000 7f74f6b55700 1 --2- 172.21.15.42:0/2537239456 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f74f0119730 0x7f74f0063d40 secure :-1 s=THROTTLE_DONE pgs=1395 cs=0 l=1 rx=0x7f74ec002f30 tx=0x7f74ec0109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:57.539 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.512+0000 7f74e77fe700 1 -- 172.21.15.42:0/2537239456 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74ec005540 con 0x7f74f0119730 2019-12-05T05:59:57.539 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.512+0000 7f74f85b8700 1 -- 172.21.15.42:0/2537239456 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74f0137bf0 con 0x7f74f0119730 2019-12-05T05:59:57.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.512+0000 7f74e77fe700 1 -- 172.21.15.42:0/2537239456 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f74ec012510 con 0x7f74f0119730 2019-12-05T05:59:57.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.512+0000 7f74f85b8700 1 -- 172.21.15.42:0/2537239456 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74f0138000 con 0x7f74f0119730 2019-12-05T05:59:57.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.512+0000 7f74f6b55700 1 --2- 172.21.15.42:0/2537239456 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f74f0119730 0x7f74f0063d40 secure :-1 s=THROTTLE_DONE pgs=1395 cs=0 l=1 rx=0x7f74ec002f30 tx=0x7f74ec0109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:57.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.512+0000 7f74e77fe700 1 -- 172.21.15.42:0/2537239456 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f74ec012830 con 0x7f74f0119730 2019-12-05T05:59:57.541 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.512+0000 7f74f6b55700 1 --2- 172.21.15.42:0/2537239456 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f74f0119730 0x7f74f0063d40 secure :-1 s=THROTTLE_DONE pgs=1395 cs=0 l=1 rx=0x7f74ec002f30 tx=0x7f74ec0109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:57.541 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.512+0000 7f74e77fe700 1 --2- 172.21.15.42:0/2537239456 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f74d803ac90 0x7f74d803d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:59:57.541 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.512+0000 7f74e77fe700 1 -- 172.21.15.42:0/2537239456 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9299+0+0 (secure 0 0 0) 0x7f74ec04ab60 con 0x7f74f0119730 2019-12-05T05:59:57.544 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.512+0000 7f74f6354700 1 --2- 172.21.15.42:0/2537239456 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f74d803ac90 0x7f74d803d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:59:57.544 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.512+0000 7f74f6354700 1 --2- 172.21.15.42:0/2537239456 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f74d803ac90 0x7f74d803d1a0 crc :-1 s=READY pgs=778 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:59:57.544 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.516+0000 7f74f85b8700 1 -- 172.21.15.42:0/2537239456 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74f005d110 con 0x7f74f0119730 2019-12-05T05:59:57.544 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.524+0000 7f74f6b55700 1 --2- 172.21.15.42:0/2537239456 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f74f0119730 0x7f74f0063d40 secure :-1 s=THROTTLE_DONE pgs=1395 cs=0 l=1 rx=0x7f74ec002f30 tx=0x7f74ec0109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:57.545 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.524+0000 7f74e77fe700 1 -- 172.21.15.42:0/2537239456 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f74ec031bc0 con 0x7f74f0119730 2019-12-05T05:59:57.713 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.708+0000 7f74f85b8700 1 -- 172.21.15.42:0/2537239456 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f74f005b8c0 con 0x7f74f0119730 2019-12-05T05:59:57.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.708+0000 7f74f6b55700 1 --2- 172.21.15.42:0/2537239456 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f74f0119730 0x7f74f0063d40 secure :-1 s=THROTTLE_DONE pgs=1395 cs=0 l=1 rx=0x7f74ec002f30 tx=0x7f74ec0109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:57.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.708+0000 7f74e77fe700 1 -- 172.21.15.42:0/2537239456 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 223 v223) v1 ==== 95+0+2109 (secure 0 0 0) 0x7f74ec031e20 con 0x7f74f0119730 2019-12-05T05:59:57.716 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 223 2019-12-05T05:59:57.716 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:59:57.717 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":223,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":9061,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6835/448869597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":448869597},{"type":"v1","addr":"172.21.15.42:6835","nonce":448869597}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":223}],"filesystems":[{"mdsmap":{"epoch":222,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:59:49.329977+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":87,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8940},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8940":{"gid":8940,"name":"b","rank":0,"incarnation":219,"state":"up:active","state_seq":36,"addr":"172.21.15.40:6833/1082666540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1082666540},{"type":"v1","addr":"172.21.15.40:6833","nonce":1082666540}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:59:57.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.712+0000 7f74f85b8700 1 -- 172.21.15.42:0/2537239456 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f74d803ac90 msgr2=0x7f74d803d1a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:59:57.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.712+0000 7f74f85b8700 1 --2- 172.21.15.42:0/2537239456 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f74d803ac90 0x7f74d803d1a0 crc :-1 s=READY pgs=778 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:59:57.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.712+0000 7f74f85b8700 1 -- 172.21.15.42:0/2537239456 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f74f0119730 msgr2=0x7f74f0063d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:59:57.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.712+0000 7f74f85b8700 1 --2- 172.21.15.42:0/2537239456 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f74f0119730 0x7f74f0063d40 secure :-1 s=READY pgs=1395 cs=0 l=1 rx=0x7f74ec002f30 tx=0x7f74ec0109e0).stop 2019-12-05T05:59:57.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.712+0000 7f74f85b8700 1 -- 172.21.15.42:0/2537239456 shutdown_connections 2019-12-05T05:59:57.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.712+0000 7f74f85b8700 1 --2- 172.21.15.42:0/2537239456 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f74d803ac90 0x7f74d803d1a0 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:59:57.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.712+0000 7f74f85b8700 1 --2- 172.21.15.42:0/2537239456 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f74f0119730 0x7f74f0063d40 unknown :-1 s=CLOSED pgs=1395 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:59:57.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.712+0000 7f74f85b8700 1 --2- 172.21.15.42:0/2537239456 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f74f0064fe0 0x7f74f0118420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:59:57.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.712+0000 7f74f85b8700 1 -- 172.21.15.42:0/2537239456 shutdown_connections 2019-12-05T05:59:57.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.712+0000 7f74f85b8700 1 -- 172.21.15.42:0/2537239456 wait complete. 2019-12-05T05:59:57.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.712+0000 7f74f85b8700 1 -- 172.21.15.42:0/2537239456 >> 172.21.15.42:0/2537239456 conn(0x7f74f009f0b0 msgr2=0x7f74f00a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:59:57.730 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.b (rank=0) 2019-12-05T05:59:57.730 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2019-12-05T05:59:57.731 INFO:teuthology.orchestra.run:waiting for 300 2019-12-05T05:59:57.748 INFO:tasks.ceph.mds.b:Stopped 2019-12-05T05:59:57.748 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T05:59:57.855 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.840+0000 7f5c2075c700 1 Processor -- start 2019-12-05T05:59:57.855 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.840+0000 7f5c2075c700 1 -- start start 2019-12-05T05:59:57.855 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.840+0000 7f5c2075c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5c1808df80 0x7f5c1809f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T05:59:57.856 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.840+0000 7f5c2075c700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c1809fc60 con 0x7f5c1809f830 2019-12-05T05:59:57.856 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.840+0000 7f5c2075c700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c180a0710 con 0x7f5c180a7740 2019-12-05T05:59:57.856 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.840+0000 7f5c2075c700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c180a0b00 con 0x7f5c1808df80 2019-12-05T05:59:57.856 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.840+0000 7f5c1e4f8700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5c1808df80 0x7f5c1809f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:59:57.856 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.840+0000 7f5c1ecf9700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f5c180a7740 0x7f5c1807e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:45764/0 (socket says 172.21.15.42:45764) 2019-12-05T05:59:57.856 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.840+0000 7f5c1ecf9700 1 -- 172.21.15.42:0/3009505514 learned_addr learned my addr 172.21.15.42:0/3009505514 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T05:59:57.857 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.844+0000 7f5c1e4f8700 1 -- 172.21.15.42:0/3009505514 >> v1:172.21.15.42:6789/0 conn(0x7f5c180a7740 legacy=0x7f5c1807e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:59:57.857 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.844+0000 7f5c1e4f8700 1 -- 172.21.15.42:0/3009505514 >> v1:172.21.15.40:6789/0 conn(0x7f5c1809f830 legacy=0x7f5c1814efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:59:57.857 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.844+0000 7f5c1e4f8700 1 -- 172.21.15.42:0/3009505514 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c1807ef20 con 0x7f5c1808df80 2019-12-05T05:59:57.857 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.844+0000 7f5c1e4f8700 1 --2- 172.21.15.42:0/3009505514 >> v2:172.21.15.42:3300/0 conn(0x7f5c1808df80 0x7f5c1809f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f5c04008790 tx=0x7f5c0400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:57.857 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.844+0000 7f5c1e4f8700 1 --2- 172.21.15.42:0/3009505514 >> v2:172.21.15.42:3300/0 conn(0x7f5c1808df80 0x7f5c1809f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f5c04008790 tx=0x7f5c0400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:57.857 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.844+0000 7f5c1e4f8700 1 --2- 172.21.15.42:0/3009505514 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5c1808df80 0x7f5c1809f2e0 secure :-1 s=READY pgs=1821 cs=0 l=1 rx=0x7f5c04008790 tx=0x7f5c0400fbf0).ready entity=mon.1 client_cookie=1ee70c7eaa2efb10 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:59:57.857 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.848+0000 7f5c1e4f8700 1 --2- 172.21.15.42:0/3009505514 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5c1808df80 0x7f5c1809f2e0 secure :-1 s=THROTTLE_DONE pgs=1821 cs=0 l=1 rx=0x7f5c04008790 tx=0x7f5c0400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:57.858 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.848+0000 7f5c1e4f8700 1 --2- 172.21.15.42:0/3009505514 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5c1808df80 0x7f5c1809f2e0 secure :-1 s=THROTTLE_DONE pgs=1821 cs=0 l=1 rx=0x7f5c04008790 tx=0x7f5c0400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:57.858 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.848+0000 7f5c1dcf7700 1 -- 172.21.15.42:0/3009505514 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5c04011270 con 0x7f5c1808df80 2019-12-05T05:59:57.858 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.848+0000 7f5c1e4f8700 1 --2- 172.21.15.42:0/3009505514 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5c1808df80 0x7f5c1809f2e0 secure :-1 s=THROTTLE_DONE pgs=1821 cs=0 l=1 rx=0x7f5c04008790 tx=0x7f5c0400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:57.858 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.848+0000 7f5c1dcf7700 1 -- 172.21.15.42:0/3009505514 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c04008bc0 con 0x7f5c1808df80 2019-12-05T05:59:57.858 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.848+0000 7f5c1dcf7700 1 -- 172.21.15.42:0/3009505514 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5c04011850 con 0x7f5c1808df80 2019-12-05T05:59:57.858 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.848+0000 7f5c2075c700 1 -- 172.21.15.42:0/3009505514 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5c1808df80 msgr2=0x7f5c1809f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:59:57.858 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.848+0000 7f5c2075c700 1 --2- 172.21.15.42:0/3009505514 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5c1808df80 0x7f5c1809f2e0 secure :-1 s=READY pgs=1821 cs=0 l=1 rx=0x7f5c04008790 tx=0x7f5c0400fbf0).stop 2019-12-05T05:59:57.859 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.848+0000 7f5c2075c700 1 -- 172.21.15.42:0/3009505514 shutdown_connections 2019-12-05T05:59:57.859 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.848+0000 7f5c2075c700 1 --2- 172.21.15.42:0/3009505514 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5c1808df80 0x7f5c1809f2e0 unknown :-1 s=CLOSED pgs=1821 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:59:57.859 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.848+0000 7f5c2075c700 1 -- 172.21.15.42:0/3009505514 shutdown_connections 2019-12-05T05:59:57.859 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.848+0000 7f5c2075c700 1 -- 172.21.15.42:0/3009505514 wait complete. 2019-12-05T05:59:57.859 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.848+0000 7f5c2075c700 1 -- 172.21.15.42:0/3009505514 >> 172.21.15.42:0/3009505514 conn(0x7f5c180aabe0 msgr2=0x7f5c180a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:59:57.862 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.852+0000 7f5c2075c700 1 Processor -- start 2019-12-05T05:59:57.862 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.852+0000 7f5c2075c700 1 -- start start 2019-12-05T05:59:57.863 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.852+0000 7f5c2075c700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5c18118d90 0x7f5c1811c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:59:57.863 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.852+0000 7f5c2075c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5c1807d0c0 0x7f5c1807d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:59:57.863 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.852+0000 7f5c2075c700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c1811c740 con 0x7f5c1808eb70 2019-12-05T05:59:57.863 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.852+0000 7f5c2075c700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c1807c920 con 0x7f5c18118d90 2019-12-05T05:59:57.863 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.852+0000 7f5c2075c700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c1807cb80 con 0x7f5c1807d0c0 2019-12-05T05:59:57.863 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.852+0000 7f5c1e4f8700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5c1807d0c0 0x7f5c1807d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:59:57.864 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.852+0000 7f5c1e4f8700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5c1807d0c0 0x7f5c1807d4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:45734/0 (socket says 172.21.15.42:45734) 2019-12-05T05:59:57.864 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.852+0000 7f5c1e4f8700 1 -- 172.21.15.42:0/3887011445 learned_addr learned my addr 172.21.15.42:0/3887011445 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T05:59:57.864 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.852+0000 7f5c1ecf9700 1 --2- 172.21.15.42:0/3887011445 >> v2:172.21.15.40:3300/0 conn(0x7f5c18118d90 0x7f5c1811c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:59:57.864 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.856+0000 7f5c1e4f8700 1 -- 172.21.15.42:0/3887011445 >> v2:172.21.15.40:3300/0 conn(0x7f5c18118d90 msgr2=0x7f5c1811c1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:59:57.864 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.856+0000 7f5c1e4f8700 1 --2- 172.21.15.42:0/3887011445 >> v2:172.21.15.40:3300/0 conn(0x7f5c18118d90 0x7f5c1811c1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:59:57.864 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.856+0000 7f5c1e4f8700 1 -- 172.21.15.42:0/3887011445 >> v1:172.21.15.40:6789/0 conn(0x7f5c1808eb70 legacy=0x7f5c1808efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:59:57.865 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.856+0000 7f5c1e4f8700 1 -- 172.21.15.42:0/3887011445 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c18116eb0 con 0x7f5c1807d0c0 2019-12-05T05:59:57.865 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.856+0000 7f5c1ecf9700 1 --2- 172.21.15.42:0/3887011445 >> v2:172.21.15.40:3300/0 conn(0x7f5c18118d90 0x7f5c1811c1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T05:59:57.865 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.856+0000 7f5c1e4f8700 1 --2- 172.21.15.42:0/3887011445 >> v2:172.21.15.42:3300/0 conn(0x7f5c1807d0c0 0x7f5c1807d4f0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f5c04008790 tx=0x7f5c04011dd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:57.865 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.856+0000 7f5c1e4f8700 1 --2- 172.21.15.42:0/3887011445 >> v2:172.21.15.42:3300/0 conn(0x7f5c1807d0c0 0x7f5c1807d4f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f5c04008790 tx=0x7f5c04011dd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:57.865 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.856+0000 7f5c1e4f8700 1 --2- 172.21.15.42:0/3887011445 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5c1807d0c0 0x7f5c1807d4f0 secure :-1 s=READY pgs=1822 cs=0 l=1 rx=0x7f5c04008790 tx=0x7f5c04011dd0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:59:57.865 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.856+0000 7f5c1e4f8700 1 --2- 172.21.15.42:0/3887011445 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5c1807d0c0 0x7f5c1807d4f0 secure :-1 s=THROTTLE_DONE pgs=1822 cs=0 l=1 rx=0x7f5c04008790 tx=0x7f5c04011dd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:57.865 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.856+0000 7f5c177fe700 1 -- 172.21.15.42:0/3887011445 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5c04007150 con 0x7f5c1807d0c0 2019-12-05T05:59:57.865 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.856+0000 7f5c1e4f8700 1 --2- 172.21.15.42:0/3887011445 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5c1807d0c0 0x7f5c1807d4f0 secure :-1 s=THROTTLE_DONE pgs=1822 cs=0 l=1 rx=0x7f5c04008790 tx=0x7f5c04011dd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:57.866 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.856+0000 7f5c1e4f8700 1 --2- 172.21.15.42:0/3887011445 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5c1807d0c0 0x7f5c1807d4f0 secure :-1 s=THROTTLE_DONE pgs=1822 cs=0 l=1 rx=0x7f5c04008790 tx=0x7f5c04011dd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:57.866 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.856+0000 7f5c177fe700 1 -- 172.21.15.42:0/3887011445 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c04011470 con 0x7f5c1807d0c0 2019-12-05T05:59:57.866 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.856+0000 7f5c177fe700 1 -- 172.21.15.42:0/3887011445 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5c04005730 con 0x7f5c1807d0c0 2019-12-05T05:59:57.866 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.856+0000 7f5c2075c700 1 -- 172.21.15.42:0/3887011445 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c1807da40 con 0x7f5c1807d0c0 2019-12-05T05:59:57.866 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.856+0000 7f5c2075c700 1 -- 172.21.15.42:0/3887011445 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c1808f7a0 con 0x7f5c1807d0c0 2019-12-05T05:59:57.866 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.856+0000 7f5c1e4f8700 1 --2- 172.21.15.42:0/3887011445 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5c1807d0c0 0x7f5c1807d4f0 secure :-1 s=THROTTLE_DONE pgs=1822 cs=0 l=1 rx=0x7f5c04008790 tx=0x7f5c04011dd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:57.867 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.856+0000 7f5c177fe700 1 -- 172.21.15.42:0/3887011445 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f5c04005a10 con 0x7f5c1807d0c0 2019-12-05T05:59:57.867 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.856+0000 7f5c1e4f8700 1 --2- 172.21.15.42:0/3887011445 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5c1807d0c0 0x7f5c1807d4f0 secure :-1 s=THROTTLE_DONE pgs=1822 cs=0 l=1 rx=0x7f5c04008790 tx=0x7f5c04011dd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:57.867 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.856+0000 7f5c177fe700 1 --2- 172.21.15.42:0/3887011445 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5c0003ad60 0x7f5c0003d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T05:59:57.867 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.856+0000 7f5c177fe700 1 -- 172.21.15.42:0/3887011445 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9299+0+0 (secure 0 0 0) 0x7f5c0404be40 con 0x7f5c1807d0c0 2019-12-05T05:59:57.867 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.856+0000 7f5c1ecf9700 1 --2- 172.21.15.42:0/3887011445 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5c0003ad60 0x7f5c0003d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T05:59:57.867 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.856+0000 7f5c1ecf9700 1 --2- 172.21.15.42:0/3887011445 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5c0003ad60 0x7f5c0003d270 crc :-1 s=READY pgs=779 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T05:59:57.876 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.872+0000 7f5c2075c700 1 -- 172.21.15.42:0/3887011445 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c1805d110 con 0x7f5c1807d0c0 2019-12-05T05:59:57.882 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.876+0000 7f5c1e4f8700 1 --2- 172.21.15.42:0/3887011445 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5c1807d0c0 0x7f5c1807d4f0 secure :-1 s=THROTTLE_DONE pgs=1822 cs=0 l=1 rx=0x7f5c04008790 tx=0x7f5c04011dd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:57.882 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:57.876+0000 7f5c177fe700 1 -- 172.21.15.42:0/3887011445 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f5c04032ea0 con 0x7f5c1807d0c0 2019-12-05T05:59:58.048 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:58.044+0000 7f5c2075c700 1 -- 172.21.15.42:0/3887011445 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5c1807c700 con 0x7f5c1807d0c0 2019-12-05T05:59:58.049 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:58.044+0000 7f5c1e4f8700 1 --2- 172.21.15.42:0/3887011445 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5c1807d0c0 0x7f5c1807d4f0 secure :-1 s=THROTTLE_DONE pgs=1822 cs=0 l=1 rx=0x7f5c04008790 tx=0x7f5c04011dd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T05:59:58.049 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:58.044+0000 7f5c177fe700 1 -- 172.21.15.42:0/3887011445 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 223 v223) v1 ==== 95+0+2109 (secure 0 0 0) 0x7f5c04033100 con 0x7f5c1807d0c0 2019-12-05T05:59:58.049 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 223 2019-12-05T05:59:58.050 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T05:59:58.050 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":223,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":9061,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6835/448869597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":448869597},{"type":"v1","addr":"172.21.15.42:6835","nonce":448869597}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":223}],"filesystems":[{"mdsmap":{"epoch":222,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:59:49.329977+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":87,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8940},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8940":{"gid":8940,"name":"b","rank":0,"incarnation":219,"state":"up:active","state_seq":36,"addr":"172.21.15.40:6833/1082666540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1082666540},{"type":"v1","addr":"172.21.15.40:6833","nonce":1082666540}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T05:59:58.054 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:58.048+0000 7f5c2075c700 1 -- 172.21.15.42:0/3887011445 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5c0003ad60 msgr2=0x7f5c0003d270 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:59:58.054 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:58.048+0000 7f5c2075c700 1 --2- 172.21.15.42:0/3887011445 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5c0003ad60 0x7f5c0003d270 crc :-1 s=READY pgs=779 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:59:58.054 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:58.048+0000 7f5c2075c700 1 -- 172.21.15.42:0/3887011445 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5c1807d0c0 msgr2=0x7f5c1807d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T05:59:58.054 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:58.048+0000 7f5c2075c700 1 --2- 172.21.15.42:0/3887011445 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5c1807d0c0 0x7f5c1807d4f0 secure :-1 s=READY pgs=1822 cs=0 l=1 rx=0x7f5c04008790 tx=0x7f5c04011dd0).stop 2019-12-05T05:59:58.054 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:58.048+0000 7f5c2075c700 1 -- 172.21.15.42:0/3887011445 shutdown_connections 2019-12-05T05:59:58.055 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:58.048+0000 7f5c2075c700 1 --2- 172.21.15.42:0/3887011445 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5c0003ad60 0x7f5c0003d270 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:59:58.055 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:58.048+0000 7f5c2075c700 1 --2- 172.21.15.42:0/3887011445 >> v2:172.21.15.40:3300/0 conn(0x7f5c18118d90 0x7f5c1811c1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:59:58.055 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:58.048+0000 7f5c2075c700 1 --2- 172.21.15.42:0/3887011445 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5c1807d0c0 0x7f5c1807d4f0 unknown :-1 s=CLOSED pgs=1822 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T05:59:58.055 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:58.048+0000 7f5c2075c700 1 -- 172.21.15.42:0/3887011445 shutdown_connections 2019-12-05T05:59:58.055 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:58.048+0000 7f5c2075c700 1 -- 172.21.15.42:0/3887011445 wait complete. 2019-12-05T05:59:58.055 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T05:59:58.048+0000 7f5c2075c700 1 -- 172.21.15.42:0/3887011445 >> 172.21.15.42:0/3887011445 conn(0x7f5c1808fda0 msgr2=0x7f5c1811cba0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T05:59:58.062 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:00:00.063 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:00:00.172 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:00.160+0000 7fb6eadb1700 1 Processor -- start 2019-12-05T06:00:00.172 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:00.160+0000 7fb6eadb1700 1 -- start start 2019-12-05T06:00:00.172 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:00.164+0000 7fb6eadb1700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb6e409fae0 0x7fb6e414fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:00:00.173 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:00.164+0000 7fb6eadb1700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6e40a0710 con 0x7fb6e40a7740 2019-12-05T06:00:00.173 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:00.164+0000 7fb6eadb1700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6e407ef20 con 0x7fb6e408df80 2019-12-05T06:00:00.173 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:00.164+0000 7fb6eadb1700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6e40a0b00 con 0x7fb6e409fae0 2019-12-05T06:00:00.173 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:00.164+0000 7fb6e8b4d700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fb6e408df80 0x7fb6e409f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:45784/0 (socket says 172.21.15.42:45784) 2019-12-05T06:00:00.174 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:00.164+0000 7fb6e8b4d700 1 -- 172.21.15.42:0/544991034 learned_addr learned my addr 172.21.15.42:0/544991034 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:00:00.174 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:00.164+0000 7fb6e9b4f700 1 --2- 172.21.15.42:0/544991034 >> v2:172.21.15.40:3300/0 conn(0x7fb6e409fae0 0x7fb6e414fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:00:00.174 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:00.164+0000 7fb6dbfff700 1 -- 172.21.15.42:0/544991034 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 965705831 0 0) 0x7fb6e407ef20 con 0x7fb6e408df80 2019-12-05T06:00:00.174 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:00.164+0000 7fb6dbfff700 1 -- 172.21.15.42:0/544991034 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6d00030b0 con 0x7fb6e408df80 2019-12-05T06:00:00.174 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:00.164+0000 7fb6e9b4f700 1 -- 172.21.15.42:0/544991034 >> v1:172.21.15.42:6789/0 conn(0x7fb6e408df80 legacy=0x7fb6e409f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:00:00.174 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:00.164+0000 7fb6e9b4f700 1 -- 172.21.15.42:0/544991034 >> v1:172.21.15.40:6789/0 conn(0x7fb6e40a7740 legacy=0x7fb6e407e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:00:00.175 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:00.164+0000 7fb6dbfff700 1 -- 172.21.15.42:0/544991034 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2935452473 0 0) 0x7fb6e40a0710 con 0x7fb6e40a7740 2019-12-05T06:00:00.175 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:00.164+0000 7fb6e9b4f700 1 -- 172.21.15.42:0/544991034 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6e40a9ba0 con 0x7fb6e409fae0 2019-12-05T06:00:00.175 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:00.164+0000 7fb6e9b4f700 1 --2- 172.21.15.42:0/544991034 >> v2:172.21.15.40:3300/0 conn(0x7fb6e409fae0 0x7fb6e414fff0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fb6dc008b10 tx=0x7fb6dc00ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:00.175 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:00.164+0000 7fb6e9b4f700 1 --2- 172.21.15.42:0/544991034 >> v2:172.21.15.40:3300/0 conn(0x7fb6e409fae0 0x7fb6e414fff0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fb6dc008b10 tx=0x7fb6dc00ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:00.175 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:00.164+0000 7fb6e9b4f700 1 --2- 172.21.15.42:0/544991034 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb6e409fae0 0x7fb6e414fff0 secure :-1 s=READY pgs=1399 cs=0 l=1 rx=0x7fb6dc008b10 tx=0x7fb6dc00ff70).ready entity=mon.0 client_cookie=20f490f12cddddc4 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:00:00.176 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:00.164+0000 7fb6e9b4f700 1 --2- 172.21.15.42:0/544991034 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb6e409fae0 0x7fb6e414fff0 secure :-1 s=THROTTLE_DONE pgs=1399 cs=0 l=1 rx=0x7fb6dc008b10 tx=0x7fb6dc00ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:00.176 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:00.164+0000 7fb6dbfff700 1 -- 172.21.15.42:0/544991034 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb6dc010970 con 0x7fb6e409fae0 2019-12-05T06:00:00.178 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:00.164+0000 7fb6e9b4f700 1 --2- 172.21.15.42:0/544991034 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb6e409fae0 0x7fb6e414fff0 secure :-1 s=THROTTLE_DONE pgs=1399 cs=0 l=1 rx=0x7fb6dc008b10 tx=0x7fb6dc00ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:00.178 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:00.164+0000 7fb6e9b4f700 1 --2- 172.21.15.42:0/544991034 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb6e409fae0 0x7fb6e414fff0 secure :-1 s=THROTTLE_DONE pgs=1399 cs=0 l=1 rx=0x7fb6dc008b10 tx=0x7fb6dc00ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:00.178 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:00.164+0000 7fb6dbfff700 1 -- 172.21.15.42:0/544991034 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6dc008b50 con 0x7fb6e409fae0 2019-12-05T06:00:00.178 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:00.164+0000 7fb6dbfff700 1 -- 172.21.15.42:0/544991034 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb6dc0110c0 con 0x7fb6e409fae0 2019-12-05T06:00:00.178 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:00.164+0000 7fb6eadb1700 1 -- 172.21.15.42:0/544991034 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb6e409fae0 msgr2=0x7fb6e414fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:00:00.179 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:00.164+0000 7fb6eadb1700 1 --2- 172.21.15.42:0/544991034 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb6e409fae0 0x7fb6e414fff0 secure :-1 s=READY pgs=1399 cs=0 l=1 rx=0x7fb6dc008b10 tx=0x7fb6dc00ff70).stop 2019-12-05T06:00:00.179 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:00.164+0000 7fb6eadb1700 1 -- 172.21.15.42:0/544991034 shutdown_connections 2019-12-05T06:00:00.179 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:00.164+0000 7fb6eadb1700 1 --2- 172.21.15.42:0/544991034 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb6e409fae0 0x7fb6e414fff0 unknown :-1 s=CLOSED pgs=1399 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:00:00.179 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:00.168+0000 7fb6eadb1700 1 -- 172.21.15.42:0/544991034 shutdown_connections 2019-12-05T06:00:00.179 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:00.168+0000 7fb6eadb1700 1 -- 172.21.15.42:0/544991034 wait complete. 2019-12-05T06:00:00.180 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:00.168+0000 7fb6eadb1700 1 -- 172.21.15.42:0/544991034 >> 172.21.15.42:0/544991034 conn(0x7fb6e40aabe0 msgr2=0x7fb6e40a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:00:00.184 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:00.168+0000 7fb6eadb1700 1 Processor -- start 2019-12-05T06:00:00.184 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:00.168+0000 7fb6eadb1700 1 -- start start 2019-12-05T06:00:00.184 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:00.172+0000 7fb6eadb1700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb6e41257d0 0x7fb6e4122210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:00:00.185 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:00.172+0000 7fb6eadb1700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb6e4111db0 0x7fb6e4115290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:00:00.185 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:00.172+0000 7fb6eadb1700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb6e4143da0 0x7fb6e41157e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:00:00.185 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:00.172+0000 7fb6eadb1700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6e4125ce0 con 0x7fb6e4111db0 2019-12-05T06:00:00.185 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:00.172+0000 7fb6eadb1700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6e4125f80 con 0x7fb6e41257d0 2019-12-05T06:00:00.185 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:00.172+0000 7fb6eadb1700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6e4126220 con 0x7fb6e4143da0 2019-12-05T06:00:00.185 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:00.172+0000 7fb6e934e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb6e41257d0 0x7fb6e4122210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:00:00.186 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:00.172+0000 7fb6e8b4d700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb6e4111db0 0x7fb6e4115290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:00:00.186 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:00.172+0000 7fb6e8b4d700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb6e4111db0 0x7fb6e4115290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.42:33570/0 (socket says 172.21.15.42:33570) 2019-12-05T06:00:00.186 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:00.172+0000 7fb6e934e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb6e41257d0 0x7fb6e4122210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:45752/0 (socket says 172.21.15.42:45752) 2019-12-05T06:00:00.186 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:00.172+0000 7fb6e8b4d700 1 -- 172.21.15.42:0/2851350762 learned_addr learned my addr 172.21.15.42:0/2851350762 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:00:00.188 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:00.172+0000 7fb6e9b4f700 1 --2- 172.21.15.42:0/2851350762 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb6e4143da0 0x7fb6e41157e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:00:00.188 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:00.172+0000 7fb6e934e700 1 -- 172.21.15.42:0/2851350762 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb6e4143da0 msgr2=0x7fb6e41157e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:00:00.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:00.172+0000 7fb6e934e700 1 --2- 172.21.15.42:0/2851350762 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb6e4143da0 0x7fb6e41157e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:00:00.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:00.172+0000 7fb6e934e700 1 -- 172.21.15.42:0/2851350762 >> v2:172.21.15.40:3300/0 conn(0x7fb6e4111db0 msgr2=0x7fb6e4115290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:00:00.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:00.172+0000 7fb6e934e700 1 --2- 172.21.15.42:0/2851350762 >> v2:172.21.15.40:3300/0 conn(0x7fb6e4111db0 0x7fb6e4115290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:00:00.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:00.172+0000 7fb6e934e700 1 -- 172.21.15.42:0/2851350762 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6e4143810 con 0x7fb6e41257d0 2019-12-05T06:00:00.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:00.172+0000 7fb6e8b4d700 1 --2- 172.21.15.42:0/2851350762 >> v2:172.21.15.40:3300/0 conn(0x7fb6e4111db0 0x7fb6e4115290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:00:00.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:00.172+0000 7fb6e9b4f700 1 --2- 172.21.15.42:0/2851350762 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb6e4143da0 0x7fb6e41157e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:00:00.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:00.172+0000 7fb6e934e700 1 --2- 172.21.15.42:0/2851350762 >> v2:172.21.15.42:3300/0 conn(0x7fb6e41257d0 0x7fb6e4122210 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fb6e4125ea0 tx=0x7fb6e00108c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:00.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:00.172+0000 7fb6e934e700 1 --2- 172.21.15.42:0/2851350762 >> v2:172.21.15.42:3300/0 conn(0x7fb6e41257d0 0x7fb6e4122210 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fb6e4125ea0 tx=0x7fb6e00108c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:00.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:00.172+0000 7fb6e934e700 1 --2- 172.21.15.42:0/2851350762 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb6e41257d0 0x7fb6e4122210 secure :-1 s=READY pgs=1824 cs=0 l=1 rx=0x7fb6e4125ea0 tx=0x7fb6e00108c0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:00:00.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:00.172+0000 7fb6e934e700 1 --2- 172.21.15.42:0/2851350762 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb6e41257d0 0x7fb6e4122210 secure :-1 s=THROTTLE_DONE pgs=1824 cs=0 l=1 rx=0x7fb6e4125ea0 tx=0x7fb6e00108c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:00.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:00.172+0000 7fb6d9ffb700 1 -- 172.21.15.42:0/2851350762 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb6e0006a40 con 0x7fb6e41257d0 2019-12-05T06:00:00.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:00.172+0000 7fb6e934e700 1 --2- 172.21.15.42:0/2851350762 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb6e41257d0 0x7fb6e4122210 secure :-1 s=THROTTLE_DONE pgs=1824 cs=0 l=1 rx=0x7fb6e4125ea0 tx=0x7fb6e00108c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:00.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:00.172+0000 7fb6e934e700 1 --2- 172.21.15.42:0/2851350762 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb6e41257d0 0x7fb6e4122210 secure :-1 s=THROTTLE_DONE pgs=1824 cs=0 l=1 rx=0x7fb6e4125ea0 tx=0x7fb6e00108c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:00.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:00.172+0000 7fb6d9ffb700 1 -- 172.21.15.42:0/2851350762 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6e0006660 con 0x7fb6e41257d0 2019-12-05T06:00:00.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:00.172+0000 7fb6eadb1700 1 -- 172.21.15.42:0/2851350762 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6e41264c0 con 0x7fb6e41257d0 2019-12-05T06:00:00.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:00.172+0000 7fb6d9ffb700 1 -- 172.21.15.42:0/2851350762 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb6e00110c0 con 0x7fb6e41257d0 2019-12-05T06:00:00.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:00.172+0000 7fb6eadb1700 1 -- 172.21.15.42:0/2851350762 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6e4122980 con 0x7fb6e41257d0 2019-12-05T06:00:00.193 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:00.172+0000 7fb6e934e700 1 --2- 172.21.15.42:0/2851350762 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb6e41257d0 0x7fb6e4122210 secure :-1 s=THROTTLE_DONE pgs=1824 cs=0 l=1 rx=0x7fb6e4125ea0 tx=0x7fb6e00108c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:00.193 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:00.172+0000 7fb6e934e700 1 --2- 172.21.15.42:0/2851350762 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb6e41257d0 0x7fb6e4122210 secure :-1 s=THROTTLE_DONE pgs=1824 cs=0 l=1 rx=0x7fb6e4125ea0 tx=0x7fb6e00108c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:00.193 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:00.172+0000 7fb6d9ffb700 1 -- 172.21.15.42:0/2851350762 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fb6e0011390 con 0x7fb6e41257d0 2019-12-05T06:00:00.194 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:00.176+0000 7fb6d9ffb700 1 --2- 172.21.15.42:0/2851350762 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb6c403aec0 0x7fb6c403d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:00:00.194 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:00.176+0000 7fb6d9ffb700 1 -- 172.21.15.42:0/2851350762 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9299+0+0 (secure 0 0 0) 0x7fb6e004b040 con 0x7fb6e41257d0 2019-12-05T06:00:00.194 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:00.176+0000 7fb6e8b4d700 1 --2- 172.21.15.42:0/2851350762 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb6c403aec0 0x7fb6c403d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:00:00.194 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:00.176+0000 7fb6e8b4d700 1 --2- 172.21.15.42:0/2851350762 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb6c403aec0 0x7fb6c403d3d0 crc :-1 s=READY pgs=780 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:00:00.195 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:00.176+0000 7fb6eadb1700 1 -- 172.21.15.42:0/2851350762 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6e405d110 con 0x7fb6e41257d0 2019-12-05T06:00:00.195 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:00.184+0000 7fb6e934e700 1 --2- 172.21.15.42:0/2851350762 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb6e41257d0 0x7fb6e4122210 secure :-1 s=THROTTLE_DONE pgs=1824 cs=0 l=1 rx=0x7fb6e4125ea0 tx=0x7fb6e00108c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:00.196 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:00.184+0000 7fb6d9ffb700 1 -- 172.21.15.42:0/2851350762 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fb6e00320a0 con 0x7fb6e41257d0 2019-12-05T06:00:00.359 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:00.352+0000 7fb6eadb1700 1 -- 172.21.15.42:0/2851350762 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb6e405b8c0 con 0x7fb6e41257d0 2019-12-05T06:00:00.359 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:00.356+0000 7fb6e934e700 1 --2- 172.21.15.42:0/2851350762 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb6e41257d0 0x7fb6e4122210 secure :-1 s=THROTTLE_DONE pgs=1824 cs=0 l=1 rx=0x7fb6e4125ea0 tx=0x7fb6e00108c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:00.359 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:00.356+0000 7fb6d9ffb700 1 -- 172.21.15.42:0/2851350762 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 223 v223) v1 ==== 95+0+2109 (secure 0 0 0) 0x7fb6e0032300 con 0x7fb6e41257d0 2019-12-05T06:00:00.360 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 223 2019-12-05T06:00:00.360 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:00:00.361 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":223,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":9061,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6835/448869597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":448869597},{"type":"v1","addr":"172.21.15.42:6835","nonce":448869597}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":223}],"filesystems":[{"mdsmap":{"epoch":222,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:59:49.329977+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":87,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8940},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8940":{"gid":8940,"name":"b","rank":0,"incarnation":219,"state":"up:active","state_seq":36,"addr":"172.21.15.40:6833/1082666540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1082666540},{"type":"v1","addr":"172.21.15.40:6833","nonce":1082666540}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:00:00.363 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:00.356+0000 7fb6eadb1700 1 -- 172.21.15.42:0/2851350762 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb6c403aec0 msgr2=0x7fb6c403d3d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:00:00.363 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:00.356+0000 7fb6eadb1700 1 --2- 172.21.15.42:0/2851350762 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb6c403aec0 0x7fb6c403d3d0 crc :-1 s=READY pgs=780 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:00:00.363 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:00.356+0000 7fb6eadb1700 1 -- 172.21.15.42:0/2851350762 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb6e41257d0 msgr2=0x7fb6e4122210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:00:00.364 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:00.356+0000 7fb6eadb1700 1 --2- 172.21.15.42:0/2851350762 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb6e41257d0 0x7fb6e4122210 secure :-1 s=READY pgs=1824 cs=0 l=1 rx=0x7fb6e4125ea0 tx=0x7fb6e00108c0).stop 2019-12-05T06:00:00.364 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:00.356+0000 7fb6eadb1700 1 -- 172.21.15.42:0/2851350762 shutdown_connections 2019-12-05T06:00:00.364 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:00.356+0000 7fb6eadb1700 1 --2- 172.21.15.42:0/2851350762 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb6c403aec0 0x7fb6c403d3d0 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:00:00.364 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:00.356+0000 7fb6eadb1700 1 --2- 172.21.15.42:0/2851350762 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb6e4143da0 0x7fb6e41157e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:00:00.364 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:00.356+0000 7fb6eadb1700 1 --2- 172.21.15.42:0/2851350762 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb6e41257d0 0x7fb6e4122210 unknown :-1 s=CLOSED pgs=1824 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:00:00.365 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:00.356+0000 7fb6eadb1700 1 --2- 172.21.15.42:0/2851350762 >> v2:172.21.15.40:3300/0 conn(0x7fb6e4111db0 0x7fb6e4115290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:00:00.365 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:00.356+0000 7fb6eadb1700 1 -- 172.21.15.42:0/2851350762 shutdown_connections 2019-12-05T06:00:00.365 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:00.356+0000 7fb6eadb1700 1 -- 172.21.15.42:0/2851350762 wait complete. 2019-12-05T06:00:00.365 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:00.356+0000 7fb6eadb1700 1 -- 172.21.15.42:0/2851350762 >> 172.21.15.42:0/2851350762 conn(0x7fb6e40a7740 msgr2=0x7fb6e407e650 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:00:00.372 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:00:02.373 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:00:02.479 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:02.468+0000 7f3d3176a700 1 Processor -- start 2019-12-05T06:00:02.479 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:02.468+0000 7f3d3176a700 1 -- start start 2019-12-05T06:00:02.479 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:02.472+0000 7f3d3176a700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f3d2c0a7740 0x7f3d2c08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:00:02.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:02.472+0000 7f3d3176a700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3d2c07e720 0x7f3d2c07eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:00:02.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:02.472+0000 7f3d3176a700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d2c0a0710 con 0x7f3d2c09f2e0 2019-12-05T06:00:02.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:02.472+0000 7f3d3176a700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d2c0a0b00 con 0x7f3d2c0a7740 2019-12-05T06:00:02.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:02.472+0000 7f3d3176a700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d2c08e5b0 con 0x7f3d2c07e720 2019-12-05T06:00:02.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:02.472+0000 7f3d2affd700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3d2c07e720 0x7f3d2c07eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:00:02.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:02.472+0000 7f3d2affd700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3d2c07e720 0x7f3d2c07eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:45768/0 (socket says 172.21.15.42:45768) 2019-12-05T06:00:02.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:02.472+0000 7f3d2affd700 1 -- 172.21.15.42:0/613719972 learned_addr learned my addr 172.21.15.42:0/613719972 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:00:02.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:02.472+0000 7f3d2b7fe700 1 --2- 172.21.15.42:0/613719972 >> v2:172.21.15.40:3300/0 conn(0x7f3d2c0a7740 0x7f3d2c08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:00:02.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:02.472+0000 7f3d2affd700 1 -- 172.21.15.42:0/613719972 >> v2:172.21.15.40:3300/0 conn(0x7f3d2c0a7740 msgr2=0x7f3d2c08df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:00:02.482 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:02.472+0000 7f3d2affd700 1 --2- 172.21.15.42:0/613719972 >> v2:172.21.15.40:3300/0 conn(0x7f3d2c0a7740 0x7f3d2c08df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:00:02.482 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:02.472+0000 7f3d2affd700 1 -- 172.21.15.42:0/613719972 >> v1:172.21.15.40:6789/0 conn(0x7f3d2c09f2e0 legacy=0x7f3d2c09f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:00:02.482 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:02.472+0000 7f3d2affd700 1 -- 172.21.15.42:0/613719972 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d2c07f0a0 con 0x7f3d2c07e720 2019-12-05T06:00:02.482 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:02.472+0000 7f3d2affd700 1 --2- 172.21.15.42:0/613719972 >> v2:172.21.15.42:3300/0 conn(0x7f3d2c07e720 0x7f3d2c07eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f3d14002f30 tx=0x7f3d1400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:02.482 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:02.472+0000 7f3d2affd700 1 --2- 172.21.15.42:0/613719972 >> v2:172.21.15.42:3300/0 conn(0x7f3d2c07e720 0x7f3d2c07eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f3d14002f30 tx=0x7f3d1400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:02.483 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:02.472+0000 7f3d2affd700 1 --2- 172.21.15.42:0/613719972 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3d2c07e720 0x7f3d2c07eb50 secure :-1 s=READY pgs=1825 cs=0 l=1 rx=0x7f3d14002f30 tx=0x7f3d1400fd30).ready entity=mon.1 client_cookie=8d7a31476c8c31d5 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:00:02.483 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:02.472+0000 7f3d2affd700 1 --2- 172.21.15.42:0/613719972 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3d2c07e720 0x7f3d2c07eb50 secure :-1 s=THROTTLE_DONE pgs=1825 cs=0 l=1 rx=0x7f3d14002f30 tx=0x7f3d1400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:02.483 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:02.472+0000 7f3d2a7fc700 1 -- 172.21.15.42:0/613719972 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3d14011370 con 0x7f3d2c07e720 2019-12-05T06:00:02.483 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:02.472+0000 7f3d2affd700 1 --2- 172.21.15.42:0/613719972 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3d2c07e720 0x7f3d2c07eb50 secure :-1 s=THROTTLE_DONE pgs=1825 cs=0 l=1 rx=0x7f3d14002f30 tx=0x7f3d1400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:02.483 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:02.472+0000 7f3d2a7fc700 1 -- 172.21.15.42:0/613719972 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d14008bb0 con 0x7f3d2c07e720 2019-12-05T06:00:02.484 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:02.472+0000 7f3d2affd700 1 --2- 172.21.15.42:0/613719972 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3d2c07e720 0x7f3d2c07eb50 secure :-1 s=THROTTLE_DONE pgs=1825 cs=0 l=1 rx=0x7f3d14002f30 tx=0x7f3d1400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:02.484 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:02.472+0000 7f3d3176a700 1 -- 172.21.15.42:0/613719972 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3d2c07e720 msgr2=0x7f3d2c07eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:00:02.484 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:02.472+0000 7f3d3176a700 1 --2- 172.21.15.42:0/613719972 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3d2c07e720 0x7f3d2c07eb50 secure :-1 s=READY pgs=1825 cs=0 l=1 rx=0x7f3d14002f30 tx=0x7f3d1400fd30).stop 2019-12-05T06:00:02.484 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:02.472+0000 7f3d2a7fc700 1 -- 172.21.15.42:0/613719972 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3d14011950 con 0x7f3d2c07e720 2019-12-05T06:00:02.485 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:02.472+0000 7f3d3176a700 1 -- 172.21.15.42:0/613719972 shutdown_connections 2019-12-05T06:00:02.485 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:02.472+0000 7f3d3176a700 1 --2- 172.21.15.42:0/613719972 >> v2:172.21.15.40:3300/0 conn(0x7f3d2c0a7740 0x7f3d2c08df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:00:02.485 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:02.472+0000 7f3d3176a700 1 --2- 172.21.15.42:0/613719972 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3d2c07e720 0x7f3d2c07eb50 unknown :-1 s=CLOSED pgs=1825 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:00:02.485 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:02.472+0000 7f3d3176a700 1 -- 172.21.15.42:0/613719972 shutdown_connections 2019-12-05T06:00:02.485 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:02.472+0000 7f3d3176a700 1 -- 172.21.15.42:0/613719972 wait complete. 2019-12-05T06:00:02.486 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:02.472+0000 7f3d3176a700 1 -- 172.21.15.42:0/613719972 >> 172.21.15.42:0/613719972 conn(0x7f3d2c0aabe0 msgr2=0x7f3d2c0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:00:02.489 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:02.476+0000 7f3d3176a700 1 Processor -- start 2019-12-05T06:00:02.490 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:02.476+0000 7f3d3176a700 1 -- start start 2019-12-05T06:00:02.490 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:02.476+0000 7f3d3176a700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3d2c11b580 0x7f3d2c116360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:00:02.490 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:02.476+0000 7f3d3176a700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3d2c07c550 0x7f3d2c08f350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:00:02.490 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:02.476+0000 7f3d3176a700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d2c07d930 con 0x7f3d2c11c1a0 2019-12-05T06:00:02.490 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:02.476+0000 7f3d3176a700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d2c11b9b0 con 0x7f3d2c07c550 2019-12-05T06:00:02.491 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:02.476+0000 7f3d3176a700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d2c11bc60 con 0x7f3d2c11b580 2019-12-05T06:00:02.491 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:02.476+0000 7f3d2bfff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3d2c07c550 0x7f3d2c08f350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:00:02.491 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:02.476+0000 7f3d2b7fe700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f3d2c11c1a0 0x7f3d2c08eb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:55248/0 (socket says 172.21.15.42:55248) 2019-12-05T06:00:02.491 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:02.476+0000 7f3d2b7fe700 1 -- 172.21.15.42:0/4109791521 learned_addr learned my addr 172.21.15.42:0/4109791521 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:00:02.492 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:02.476+0000 7f3d2affd700 1 --2- 172.21.15.42:0/4109791521 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3d2c11b580 0x7f3d2c116360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:00:02.492 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:02.476+0000 7f3d2bfff700 1 -- 172.21.15.42:0/4109791521 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3d2c11b580 msgr2=0x7f3d2c116360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:00:02.492 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:02.476+0000 7f3d23fff700 1 -- 172.21.15.42:0/4109791521 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2328362380 0 0) 0x7f3d2c07d930 con 0x7f3d2c11c1a0 2019-12-05T06:00:02.492 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:02.476+0000 7f3d2bfff700 1 --2- 172.21.15.42:0/4109791521 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3d2c11b580 0x7f3d2c116360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:00:02.492 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:02.476+0000 7f3d2bfff700 1 -- 172.21.15.42:0/4109791521 >> v1:172.21.15.40:6789/0 conn(0x7f3d2c11c1a0 legacy=0x7f3d2c08eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:00:02.493 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:02.476+0000 7f3d2bfff700 1 -- 172.21.15.42:0/4109791521 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d2c08f8a0 con 0x7f3d2c07c550 2019-12-05T06:00:02.493 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:02.476+0000 7f3d2bfff700 1 --2- 172.21.15.42:0/4109791521 >> v2:172.21.15.42:3300/0 conn(0x7f3d2c07c550 0x7f3d2c08f350 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f3d2c07bf30 tx=0x7f3d240107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:02.493 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:02.476+0000 7f3d2bfff700 1 --2- 172.21.15.42:0/4109791521 >> v2:172.21.15.42:3300/0 conn(0x7f3d2c07c550 0x7f3d2c08f350 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f3d2c07bf30 tx=0x7f3d240107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:02.493 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:02.476+0000 7f3d2bfff700 1 --2- 172.21.15.42:0/4109791521 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3d2c07c550 0x7f3d2c08f350 secure :-1 s=READY pgs=1826 cs=0 l=1 rx=0x7f3d2c07bf30 tx=0x7f3d240107c0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:00:02.493 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:02.476+0000 7f3d2bfff700 1 --2- 172.21.15.42:0/4109791521 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3d2c07c550 0x7f3d2c08f350 secure :-1 s=THROTTLE_DONE pgs=1826 cs=0 l=1 rx=0x7f3d2c07bf30 tx=0x7f3d240107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:02.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:02.476+0000 7f3d23fff700 1 -- 172.21.15.42:0/4109791521 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3d24006a40 con 0x7f3d2c07c550 2019-12-05T06:00:02.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:02.476+0000 7f3d2bfff700 1 --2- 172.21.15.42:0/4109791521 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3d2c07c550 0x7f3d2c08f350 secure :-1 s=THROTTLE_DONE pgs=1826 cs=0 l=1 rx=0x7f3d2c07bf30 tx=0x7f3d240107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:02.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:02.476+0000 7f3d23fff700 1 -- 172.21.15.42:0/4109791521 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d24006660 con 0x7f3d2c07c550 2019-12-05T06:00:02.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:02.476+0000 7f3d2bfff700 1 --2- 172.21.15.42:0/4109791521 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3d2c07c550 0x7f3d2c08f350 secure :-1 s=THROTTLE_DONE pgs=1826 cs=0 l=1 rx=0x7f3d2c07bf30 tx=0x7f3d240107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:02.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:02.480+0000 7f3d23fff700 1 -- 172.21.15.42:0/4109791521 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3d24010f80 con 0x7f3d2c07c550 2019-12-05T06:00:02.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:02.480+0000 7f3d3176a700 1 -- 172.21.15.42:0/4109791521 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d2c11a5f0 con 0x7f3d2c07c550 2019-12-05T06:00:02.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:02.480+0000 7f3d3176a700 1 -- 172.21.15.42:0/4109791521 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d2c07d5b0 con 0x7f3d2c07c550 2019-12-05T06:00:02.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:02.480+0000 7f3d2bfff700 1 --2- 172.21.15.42:0/4109791521 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3d2c07c550 0x7f3d2c08f350 secure :-1 s=THROTTLE_DONE pgs=1826 cs=0 l=1 rx=0x7f3d2c07bf30 tx=0x7f3d240107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:02.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:02.480+0000 7f3d23fff700 1 -- 172.21.15.42:0/4109791521 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f3d24011260 con 0x7f3d2c07c550 2019-12-05T06:00:02.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:02.480+0000 7f3d2bfff700 1 --2- 172.21.15.42:0/4109791521 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3d2c07c550 0x7f3d2c08f350 secure :-1 s=THROTTLE_DONE pgs=1826 cs=0 l=1 rx=0x7f3d2c07bf30 tx=0x7f3d240107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:02.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:02.480+0000 7f3d23fff700 1 --2- 172.21.15.42:0/4109791521 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3d1003ac00 0x7f3d1003d110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:00:02.496 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:02.480+0000 7f3d23fff700 1 -- 172.21.15.42:0/4109791521 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9299+0+0 (secure 0 0 0) 0x7f3d2404b140 con 0x7f3d2c07c550 2019-12-05T06:00:02.496 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:02.480+0000 7f3d2b7fe700 1 --2- 172.21.15.42:0/4109791521 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3d1003ac00 0x7f3d1003d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:00:02.496 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:02.480+0000 7f3d2b7fe700 1 --2- 172.21.15.42:0/4109791521 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3d1003ac00 0x7f3d1003d110 crc :-1 s=READY pgs=781 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:00:02.496 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:02.484+0000 7f3d3176a700 1 -- 172.21.15.42:0/4109791521 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d2c05d110 con 0x7f3d2c07c550 2019-12-05T06:00:02.497 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:02.492+0000 7f3d2bfff700 1 --2- 172.21.15.42:0/4109791521 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3d2c07c550 0x7f3d2c08f350 secure :-1 s=THROTTLE_DONE pgs=1826 cs=0 l=1 rx=0x7f3d2c07bf30 tx=0x7f3d240107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:02.497 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:02.492+0000 7f3d23fff700 1 -- 172.21.15.42:0/4109791521 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f3d24032140 con 0x7f3d2c07c550 2019-12-05T06:00:02.659 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:02.656+0000 7f3d3176a700 1 -- 172.21.15.42:0/4109791521 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3d2c05b8c0 con 0x7f3d2c07c550 2019-12-05T06:00:02.660 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:02.656+0000 7f3d2bfff700 1 --2- 172.21.15.42:0/4109791521 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3d2c07c550 0x7f3d2c08f350 secure :-1 s=THROTTLE_DONE pgs=1826 cs=0 l=1 rx=0x7f3d2c07bf30 tx=0x7f3d240107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:02.661 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:02.656+0000 7f3d23fff700 1 -- 172.21.15.42:0/4109791521 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 223 v223) v1 ==== 95+0+2109 (secure 0 0 0) 0x7f3d240323a0 con 0x7f3d2c07c550 2019-12-05T06:00:02.661 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 223 2019-12-05T06:00:02.662 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:00:02.662 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":223,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":9061,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6835/448869597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":448869597},{"type":"v1","addr":"172.21.15.42:6835","nonce":448869597}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":223}],"filesystems":[{"mdsmap":{"epoch":222,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:59:49.329977+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":87,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8940},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8940":{"gid":8940,"name":"b","rank":0,"incarnation":219,"state":"up:active","state_seq":36,"addr":"172.21.15.40:6833/1082666540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1082666540},{"type":"v1","addr":"172.21.15.40:6833","nonce":1082666540}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:00:02.663 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:02.656+0000 7f3d3176a700 1 -- 172.21.15.42:0/4109791521 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3d1003ac00 msgr2=0x7f3d1003d110 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:00:02.664 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:02.656+0000 7f3d3176a700 1 --2- 172.21.15.42:0/4109791521 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3d1003ac00 0x7f3d1003d110 crc :-1 s=READY pgs=781 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:00:02.664 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:02.656+0000 7f3d3176a700 1 -- 172.21.15.42:0/4109791521 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3d2c07c550 msgr2=0x7f3d2c08f350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:00:02.664 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:02.656+0000 7f3d3176a700 1 --2- 172.21.15.42:0/4109791521 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3d2c07c550 0x7f3d2c08f350 secure :-1 s=READY pgs=1826 cs=0 l=1 rx=0x7f3d2c07bf30 tx=0x7f3d240107c0).stop 2019-12-05T06:00:02.665 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:02.656+0000 7f3d3176a700 1 -- 172.21.15.42:0/4109791521 shutdown_connections 2019-12-05T06:00:02.665 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:02.656+0000 7f3d3176a700 1 --2- 172.21.15.42:0/4109791521 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3d1003ac00 0x7f3d1003d110 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:00:02.665 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:02.656+0000 7f3d3176a700 1 --2- 172.21.15.42:0/4109791521 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3d2c07c550 0x7f3d2c08f350 unknown :-1 s=CLOSED pgs=1826 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:00:02.666 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:02.656+0000 7f3d3176a700 1 --2- 172.21.15.42:0/4109791521 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3d2c11b580 0x7f3d2c116360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:00:02.666 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:02.656+0000 7f3d3176a700 1 -- 172.21.15.42:0/4109791521 shutdown_connections 2019-12-05T06:00:02.666 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:02.656+0000 7f3d3176a700 1 -- 172.21.15.42:0/4109791521 wait complete. 2019-12-05T06:00:02.666 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:02.656+0000 7f3d3176a700 1 -- 172.21.15.42:0/4109791521 >> 172.21.15.42:0/4109791521 conn(0x7f3d2c0a7f90 msgr2=0x7f3d2c059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:00:02.672 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:00:04.673 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:00:04.777 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:04.768+0000 7fb8f89f1700 1 Processor -- start 2019-12-05T06:00:04.777 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:04.768+0000 7fb8f89f1700 1 -- start start 2019-12-05T06:00:04.777 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:04.768+0000 7fb8f89f1700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb8f40a7740 0x7fb8f408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:00:04.777 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:04.768+0000 7fb8f89f1700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb8f409fae0 0x7fb8f407eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:00:04.778 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:04.768+0000 7fb8f89f1700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8f40a0710 con 0x7fb8f407e720 2019-12-05T06:00:04.778 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:04.768+0000 7fb8f89f1700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8f40a0b00 con 0x7fb8f40a7740 2019-12-05T06:00:04.778 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:04.768+0000 7fb8f89f1700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8f408e5b0 con 0x7fb8f409fae0 2019-12-05T06:00:04.778 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:04.768+0000 7fb8f2d9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb8f40a7740 0x7fb8f408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:00:04.778 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:04.768+0000 7fb8f2d9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb8f40a7740 0x7fb8f408df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:45782/0 (socket says 172.21.15.42:45782) 2019-12-05T06:00:04.778 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:04.768+0000 7fb8f2d9d700 1 -- 172.21.15.42:0/3301069679 learned_addr learned my addr 172.21.15.42:0/3301069679 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:00:04.779 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:04.768+0000 7fb8f359e700 1 --2- 172.21.15.42:0/3301069679 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb8f409fae0 0x7fb8f407eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:00:04.779 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:04.768+0000 7fb8f2d9d700 1 -- 172.21.15.42:0/3301069679 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb8f409fae0 msgr2=0x7fb8f407eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:00:04.779 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:04.768+0000 7fb8f2d9d700 1 --2- 172.21.15.42:0/3301069679 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb8f409fae0 0x7fb8f407eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:00:04.779 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:04.768+0000 7fb8f2d9d700 1 -- 172.21.15.42:0/3301069679 >> v1:172.21.15.42:6789/0 conn(0x7fb8f407e720 legacy=0x7fb8f409f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:00:04.779 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:04.768+0000 7fb8f2d9d700 1 -- 172.21.15.42:0/3301069679 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8f407f0a0 con 0x7fb8f40a7740 2019-12-05T06:00:04.779 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:04.768+0000 7fb8f2d9d700 1 --2- 172.21.15.42:0/3301069679 >> v2:172.21.15.42:3300/0 conn(0x7fb8f40a7740 0x7fb8f408df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fb8e4008790 tx=0x7fb8e400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:04.779 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:04.768+0000 7fb8f2d9d700 1 --2- 172.21.15.42:0/3301069679 >> v2:172.21.15.42:3300/0 conn(0x7fb8f40a7740 0x7fb8f408df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fb8e4008790 tx=0x7fb8e400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:04.780 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:04.772+0000 7fb8f2d9d700 1 --2- 172.21.15.42:0/3301069679 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb8f40a7740 0x7fb8f408df80 secure :-1 s=READY pgs=1828 cs=0 l=1 rx=0x7fb8e4008790 tx=0x7fb8e400fd30).ready entity=mon.1 client_cookie=a290c9c6951d9ad2 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:00:04.780 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:04.772+0000 7fb8f2d9d700 1 --2- 172.21.15.42:0/3301069679 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb8f40a7740 0x7fb8f408df80 secure :-1 s=THROTTLE_DONE pgs=1828 cs=0 l=1 rx=0x7fb8e4008790 tx=0x7fb8e400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:04.780 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:04.772+0000 7fb8f1d9b700 1 -- 172.21.15.42:0/3301069679 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb8e4008c90 con 0x7fb8f40a7740 2019-12-05T06:00:04.780 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:04.772+0000 7fb8f2d9d700 1 --2- 172.21.15.42:0/3301069679 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb8f40a7740 0x7fb8f408df80 secure :-1 s=THROTTLE_DONE pgs=1828 cs=0 l=1 rx=0x7fb8e4008790 tx=0x7fb8e400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:04.780 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:04.772+0000 7fb8f1d9b700 1 -- 172.21.15.42:0/3301069679 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8e40088b0 con 0x7fb8f40a7740 2019-12-05T06:00:04.780 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:04.772+0000 7fb8f2d9d700 1 --2- 172.21.15.42:0/3301069679 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb8f40a7740 0x7fb8f408df80 secure :-1 s=THROTTLE_DONE pgs=1828 cs=0 l=1 rx=0x7fb8e4008790 tx=0x7fb8e400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:04.780 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:04.772+0000 7fb8f1d9b700 1 -- 172.21.15.42:0/3301069679 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb8e4011b10 con 0x7fb8f40a7740 2019-12-05T06:00:04.781 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:04.772+0000 7fb8f89f1700 1 -- 172.21.15.42:0/3301069679 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb8f40a7740 msgr2=0x7fb8f408df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:00:04.781 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:04.772+0000 7fb8f89f1700 1 --2- 172.21.15.42:0/3301069679 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb8f40a7740 0x7fb8f408df80 secure :-1 s=READY pgs=1828 cs=0 l=1 rx=0x7fb8e4008790 tx=0x7fb8e400fd30).stop 2019-12-05T06:00:04.781 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:04.772+0000 7fb8f89f1700 1 -- 172.21.15.42:0/3301069679 shutdown_connections 2019-12-05T06:00:04.781 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:04.772+0000 7fb8f89f1700 1 --2- 172.21.15.42:0/3301069679 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb8f409fae0 0x7fb8f407eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:00:04.781 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:04.772+0000 7fb8f89f1700 1 --2- 172.21.15.42:0/3301069679 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb8f40a7740 0x7fb8f408df80 unknown :-1 s=CLOSED pgs=1828 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:00:04.781 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:04.772+0000 7fb8f89f1700 1 -- 172.21.15.42:0/3301069679 shutdown_connections 2019-12-05T06:00:04.781 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:04.772+0000 7fb8f89f1700 1 -- 172.21.15.42:0/3301069679 wait complete. 2019-12-05T06:00:04.782 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:04.772+0000 7fb8f89f1700 1 -- 172.21.15.42:0/3301069679 >> 172.21.15.42:0/3301069679 conn(0x7fb8f40aabe0 msgr2=0x7fb8f40a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:00:04.788 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:04.772+0000 7fb8f89f1700 1 Processor -- start 2019-12-05T06:00:04.788 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:04.776+0000 7fb8f89f1700 1 -- start start 2019-12-05T06:00:04.788 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:04.776+0000 7fb8f89f1700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb8f407c790 0x7fb8f4144eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:00:04.788 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:04.776+0000 7fb8f89f1700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8f40a5b10 con 0x7fb8f40a4ae0 2019-12-05T06:00:04.789 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:04.776+0000 7fb8f89f1700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8f411d040 con 0x7fb8f411cc10 2019-12-05T06:00:04.789 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:04.776+0000 7fb8f89f1700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8f40a5d40 con 0x7fb8f407c790 2019-12-05T06:00:04.789 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:04.776+0000 7fb8f259c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb8f407c790 0x7fb8f4144eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:00:04.789 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:04.776+0000 7fb8f359e700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fb8f411cc10 0x7fb8f41357b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:45828/0 (socket says 172.21.15.42:45828) 2019-12-05T06:00:04.789 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:04.776+0000 7fb8f359e700 1 -- 172.21.15.42:0/406524701 learned_addr learned my addr 172.21.15.42:0/406524701 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:00:04.790 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:04.776+0000 7fb8eb7fe700 1 -- 172.21.15.42:0/406524701 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 897398598 0 0) 0x7fb8f40a5b10 con 0x7fb8f40a4ae0 2019-12-05T06:00:04.790 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:04.776+0000 7fb8eb7fe700 1 -- 172.21.15.42:0/406524701 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8d80030b0 con 0x7fb8f40a4ae0 2019-12-05T06:00:04.790 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:04.776+0000 7fb8f259c700 1 -- 172.21.15.42:0/406524701 >> v1:172.21.15.42:6789/0 conn(0x7fb8f411cc10 legacy=0x7fb8f41357b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:00:04.790 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:04.776+0000 7fb8eb7fe700 1 -- 172.21.15.42:0/406524701 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3679326287 0 0) 0x7fb8f411d040 con 0x7fb8f411cc10 2019-12-05T06:00:04.790 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:04.776+0000 7fb8f259c700 1 -- 172.21.15.42:0/406524701 >> v1:172.21.15.40:6789/0 conn(0x7fb8f40a4ae0 legacy=0x7fb8f407bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:00:04.790 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:04.776+0000 7fb8f259c700 1 -- 172.21.15.42:0/406524701 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8f411d270 con 0x7fb8f407c790 2019-12-05T06:00:04.791 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:04.776+0000 7fb8f259c700 1 --2- 172.21.15.42:0/406524701 >> v2:172.21.15.42:3300/0 conn(0x7fb8f407c790 0x7fb8f4144eb0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fb8f40a0a20 tx=0x7fb8dc00c320).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:04.791 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:04.776+0000 7fb8f259c700 1 --2- 172.21.15.42:0/406524701 >> v2:172.21.15.42:3300/0 conn(0x7fb8f407c790 0x7fb8f4144eb0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fb8f40a0a20 tx=0x7fb8dc00c320).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:04.791 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:04.776+0000 7fb8f259c700 1 --2- 172.21.15.42:0/406524701 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb8f407c790 0x7fb8f4144eb0 secure :-1 s=READY pgs=1830 cs=0 l=1 rx=0x7fb8f40a0a20 tx=0x7fb8dc00c320).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:00:04.791 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:04.776+0000 7fb8f259c700 1 --2- 172.21.15.42:0/406524701 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb8f407c790 0x7fb8f4144eb0 secure :-1 s=THROTTLE_DONE pgs=1830 cs=0 l=1 rx=0x7fb8f40a0a20 tx=0x7fb8dc00c320).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:04.791 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:04.776+0000 7fb8eb7fe700 1 -- 172.21.15.42:0/406524701 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb8dc00cb70 con 0x7fb8f407c790 2019-12-05T06:00:04.791 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:04.776+0000 7fb8f259c700 1 --2- 172.21.15.42:0/406524701 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb8f407c790 0x7fb8f4144eb0 secure :-1 s=THROTTLE_DONE pgs=1830 cs=0 l=1 rx=0x7fb8f40a0a20 tx=0x7fb8dc00c320).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:04.792 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:04.776+0000 7fb8eb7fe700 1 -- 172.21.15.42:0/406524701 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8dc00c790 con 0x7fb8f407c790 2019-12-05T06:00:04.792 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:04.776+0000 7fb8f259c700 1 --2- 172.21.15.42:0/406524701 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb8f407c790 0x7fb8f4144eb0 secure :-1 s=THROTTLE_DONE pgs=1830 cs=0 l=1 rx=0x7fb8f40a0a20 tx=0x7fb8dc00c320).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:04.792 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:04.776+0000 7fb8f89f1700 1 -- 172.21.15.42:0/406524701 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8f4136fc0 con 0x7fb8f407c790 2019-12-05T06:00:04.792 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:04.776+0000 7fb8eb7fe700 1 -- 172.21.15.42:0/406524701 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb8dc011070 con 0x7fb8f407c790 2019-12-05T06:00:04.793 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:04.776+0000 7fb8f89f1700 1 -- 172.21.15.42:0/406524701 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8f4137540 con 0x7fb8f407c790 2019-12-05T06:00:04.793 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:04.780+0000 7fb8f259c700 1 --2- 172.21.15.42:0/406524701 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb8f407c790 0x7fb8f4144eb0 secure :-1 s=THROTTLE_DONE pgs=1830 cs=0 l=1 rx=0x7fb8f40a0a20 tx=0x7fb8dc00c320).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:04.793 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:04.780+0000 7fb8eb7fe700 1 -- 172.21.15.42:0/406524701 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fb8dc011390 con 0x7fb8f407c790 2019-12-05T06:00:04.793 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:04.780+0000 7fb8f259c700 1 --2- 172.21.15.42:0/406524701 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb8f407c790 0x7fb8f4144eb0 secure :-1 s=THROTTLE_DONE pgs=1830 cs=0 l=1 rx=0x7fb8f40a0a20 tx=0x7fb8dc00c320).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:04.793 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:04.780+0000 7fb8eb7fe700 1 --2- 172.21.15.42:0/406524701 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb8d803b1c0 0x7fb8d803d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:00:04.793 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:04.780+0000 7fb8eb7fe700 1 -- 172.21.15.42:0/406524701 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9299+0+0 (secure 0 0 0) 0x7fb8dc04b160 con 0x7fb8f407c790 2019-12-05T06:00:04.793 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:04.780+0000 7fb8f2d9d700 1 --2- 172.21.15.42:0/406524701 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb8d803b1c0 0x7fb8d803d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:00:04.794 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:04.780+0000 7fb8f2d9d700 1 --2- 172.21.15.42:0/406524701 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb8d803b1c0 0x7fb8d803d6d0 crc :-1 s=READY pgs=782 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:00:04.794 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:04.784+0000 7fb8f89f1700 1 -- 172.21.15.42:0/406524701 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8f405d110 con 0x7fb8f407c790 2019-12-05T06:00:04.794 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:04.788+0000 7fb8f259c700 1 --2- 172.21.15.42:0/406524701 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb8f407c790 0x7fb8f4144eb0 secure :-1 s=THROTTLE_DONE pgs=1830 cs=0 l=1 rx=0x7fb8f40a0a20 tx=0x7fb8dc00c320).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:04.794 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:04.788+0000 7fb8eb7fe700 1 -- 172.21.15.42:0/406524701 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fb8dc0321c0 con 0x7fb8f407c790 2019-12-05T06:00:04.964 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:04.960+0000 7fb8f89f1700 1 -- 172.21.15.42:0/406524701 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb8f405b8c0 con 0x7fb8f407c790 2019-12-05T06:00:04.965 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:04.960+0000 7fb8f259c700 1 --2- 172.21.15.42:0/406524701 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb8f407c790 0x7fb8f4144eb0 secure :-1 s=THROTTLE_DONE pgs=1830 cs=0 l=1 rx=0x7fb8f40a0a20 tx=0x7fb8dc00c320).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:04.965 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:04.960+0000 7fb8eb7fe700 1 -- 172.21.15.42:0/406524701 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 223 v223) v1 ==== 95+0+2109 (secure 0 0 0) 0x7fb8dc032420 con 0x7fb8f407c790 2019-12-05T06:00:04.968 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:00:04.968 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":223,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":9061,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6835/448869597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":448869597},{"type":"v1","addr":"172.21.15.42:6835","nonce":448869597}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":223}],"filesystems":[{"mdsmap":{"epoch":222,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:59:49.329977+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":87,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8940},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8940":{"gid":8940,"name":"b","rank":0,"incarnation":219,"state":"up:active","state_seq":36,"addr":"172.21.15.40:6833/1082666540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1082666540},{"type":"v1","addr":"172.21.15.40:6833","nonce":1082666540}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:00:04.968 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 223 2019-12-05T06:00:04.970 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:04.964+0000 7fb8f89f1700 1 -- 172.21.15.42:0/406524701 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb8d803b1c0 msgr2=0x7fb8d803d6d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:00:04.970 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:04.964+0000 7fb8f89f1700 1 --2- 172.21.15.42:0/406524701 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb8d803b1c0 0x7fb8d803d6d0 crc :-1 s=READY pgs=782 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:00:04.970 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:04.964+0000 7fb8f89f1700 1 -- 172.21.15.42:0/406524701 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb8f407c790 msgr2=0x7fb8f4144eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:00:04.970 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:04.964+0000 7fb8f89f1700 1 --2- 172.21.15.42:0/406524701 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb8f407c790 0x7fb8f4144eb0 secure :-1 s=READY pgs=1830 cs=0 l=1 rx=0x7fb8f40a0a20 tx=0x7fb8dc00c320).stop 2019-12-05T06:00:04.970 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:04.964+0000 7fb8f89f1700 1 -- 172.21.15.42:0/406524701 shutdown_connections 2019-12-05T06:00:04.971 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:04.964+0000 7fb8f89f1700 1 --2- 172.21.15.42:0/406524701 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb8d803b1c0 0x7fb8d803d6d0 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:00:04.971 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:04.964+0000 7fb8f89f1700 1 --2- 172.21.15.42:0/406524701 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb8f407c790 0x7fb8f4144eb0 unknown :-1 s=CLOSED pgs=1830 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:00:04.971 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:04.964+0000 7fb8f89f1700 1 -- 172.21.15.42:0/406524701 shutdown_connections 2019-12-05T06:00:04.971 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:04.964+0000 7fb8f89f1700 1 -- 172.21.15.42:0/406524701 wait complete. 2019-12-05T06:00:04.971 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:04.964+0000 7fb8f89f1700 1 -- 172.21.15.42:0/406524701 >> 172.21.15.42:0/406524701 conn(0x7fb8f40a80b0 msgr2=0x7fb8f411d8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:00:04.983 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:00:06.984 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:00:07.090 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:07.080+0000 7f25e88af700 1 Processor -- start 2019-12-05T06:00:07.091 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:07.080+0000 7f25e88af700 1 -- start start 2019-12-05T06:00:07.091 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:07.084+0000 7f25e88af700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f25e408df80 0x7f25e409f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:00:07.091 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:07.084+0000 7f25e88af700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25e409fc60 con 0x7f25e40a7740 2019-12-05T06:00:07.091 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:07.084+0000 7f25e88af700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25e40a0710 con 0x7f25e409f830 2019-12-05T06:00:07.091 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:07.084+0000 7f25e88af700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25e40a0b00 con 0x7f25e408df80 2019-12-05T06:00:07.091 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:07.084+0000 7f25e259c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f25e408df80 0x7f25e409f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:00:07.092 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:07.084+0000 7f25e359e700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f25e409f830 0x7f25e414efe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:45836/0 (socket says 172.21.15.42:45836) 2019-12-05T06:00:07.092 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:07.084+0000 7f25e359e700 1 -- 172.21.15.42:0/1403410574 learned_addr learned my addr 172.21.15.42:0/1403410574 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:00:07.092 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:07.084+0000 7f25e259c700 1 -- 172.21.15.42:0/1403410574 >> v1:172.21.15.42:6789/0 conn(0x7f25e409f830 legacy=0x7f25e414efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:00:07.092 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:07.084+0000 7f25e259c700 1 -- 172.21.15.42:0/1403410574 >> v1:172.21.15.40:6789/0 conn(0x7f25e40a7740 legacy=0x7f25e407e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:00:07.092 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:07.084+0000 7f25e1d9b700 1 -- 172.21.15.42:0/1403410574 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 335784414 0 0) 0x7f25e40a0710 con 0x7f25e409f830 2019-12-05T06:00:07.092 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:07.084+0000 7f25e259c700 1 -- 172.21.15.42:0/1403410574 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25e407ef20 con 0x7f25e408df80 2019-12-05T06:00:07.092 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:07.084+0000 7f25e259c700 1 --2- 172.21.15.42:0/1403410574 >> v2:172.21.15.42:3300/0 conn(0x7f25e408df80 0x7f25e409f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f25d4008b10 tx=0x7f25cc001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:07.093 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:07.084+0000 7f25e259c700 1 --2- 172.21.15.42:0/1403410574 >> v2:172.21.15.42:3300/0 conn(0x7f25e408df80 0x7f25e409f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f25d4008b10 tx=0x7f25cc001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:07.093 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:07.084+0000 7f25e259c700 1 --2- 172.21.15.42:0/1403410574 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f25e408df80 0x7f25e409f2e0 secure :-1 s=READY pgs=1832 cs=0 l=1 rx=0x7f25d4008b10 tx=0x7f25cc001150).ready entity=mon.1 client_cookie=26c7a0b1dceff2e2 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:00:07.093 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:07.084+0000 7f25e259c700 1 --2- 172.21.15.42:0/1403410574 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f25e408df80 0x7f25e409f2e0 secure :-1 s=THROTTLE_DONE pgs=1832 cs=0 l=1 rx=0x7f25d4008b10 tx=0x7f25cc001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:07.093 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:07.084+0000 7f25e259c700 1 --2- 172.21.15.42:0/1403410574 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f25e408df80 0x7f25e409f2e0 secure :-1 s=THROTTLE_DONE pgs=1832 cs=0 l=1 rx=0x7f25d4008b10 tx=0x7f25cc001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:07.093 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:07.084+0000 7f25e1d9b700 1 -- 172.21.15.42:0/1403410574 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f25d4010870 con 0x7f25e408df80 2019-12-05T06:00:07.093 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:07.084+0000 7f25e259c700 1 --2- 172.21.15.42:0/1403410574 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f25e408df80 0x7f25e409f2e0 secure :-1 s=THROTTLE_DONE pgs=1832 cs=0 l=1 rx=0x7f25d4008b10 tx=0x7f25cc001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:07.093 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:07.084+0000 7f25e1d9b700 1 -- 172.21.15.42:0/1403410574 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25d4008b50 con 0x7f25e408df80 2019-12-05T06:00:07.094 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:07.084+0000 7f25e1d9b700 1 -- 172.21.15.42:0/1403410574 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f25d40110c0 con 0x7f25e408df80 2019-12-05T06:00:07.094 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:07.084+0000 7f25e88af700 1 -- 172.21.15.42:0/1403410574 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f25e408df80 msgr2=0x7f25e409f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:00:07.094 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:07.084+0000 7f25e88af700 1 --2- 172.21.15.42:0/1403410574 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f25e408df80 0x7f25e409f2e0 secure :-1 s=READY pgs=1832 cs=0 l=1 rx=0x7f25d4008b10 tx=0x7f25cc001150).stop 2019-12-05T06:00:07.094 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:07.084+0000 7f25e88af700 1 -- 172.21.15.42:0/1403410574 shutdown_connections 2019-12-05T06:00:07.094 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:07.084+0000 7f25e88af700 1 --2- 172.21.15.42:0/1403410574 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f25e408df80 0x7f25e409f2e0 unknown :-1 s=CLOSED pgs=1832 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:00:07.094 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:07.084+0000 7f25e88af700 1 -- 172.21.15.42:0/1403410574 shutdown_connections 2019-12-05T06:00:07.094 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:07.084+0000 7f25e88af700 1 -- 172.21.15.42:0/1403410574 wait complete. 2019-12-05T06:00:07.095 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:07.084+0000 7f25e88af700 1 -- 172.21.15.42:0/1403410574 >> 172.21.15.42:0/1403410574 conn(0x7f25e40aabe0 msgr2=0x7f25e40a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:00:07.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:07.088+0000 7f25e88af700 1 Processor -- start 2019-12-05T06:00:07.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:07.088+0000 7f25e88af700 1 -- start start 2019-12-05T06:00:07.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:07.088+0000 7f25e88af700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f25e4122fa0 0x7f25e4117560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:00:07.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:07.088+0000 7f25e88af700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f25e41191b0 0x7f25e413a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:00:07.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:07.088+0000 7f25e88af700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25e407e1f0 con 0x7f25e40a5ef0 2019-12-05T06:00:07.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:07.088+0000 7f25e88af700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25e4117dc0 con 0x7f25e41191b0 2019-12-05T06:00:07.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:07.088+0000 7f25e88af700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25e4118340 con 0x7f25e4122fa0 2019-12-05T06:00:07.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:07.088+0000 7f25e259c700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f25e40a5ef0 0x7f25e41185e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:45838/0 (socket says 172.21.15.42:45838) 2019-12-05T06:00:07.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:07.088+0000 7f25e2d9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f25e4122fa0 0x7f25e4117560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:00:07.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:07.088+0000 7f25e259c700 1 -- 172.21.15.42:0/3018405508 learned_addr learned my addr 172.21.15.42:0/3018405508 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:00:07.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:07.088+0000 7f25e359e700 1 --2- 172.21.15.42:0/3018405508 >> v2:172.21.15.40:3300/0 conn(0x7f25e41191b0 0x7f25e413a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:00:07.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:07.088+0000 7f25db7fe700 1 -- 172.21.15.42:0/3018405508 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 108972583 0 0) 0x7f25e409fc60 con 0x7f25e40a5ef0 2019-12-05T06:00:07.100 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:07.088+0000 7f25db7fe700 1 -- 172.21.15.42:0/3018405508 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25c80030b0 con 0x7f25e40a5ef0 2019-12-05T06:00:07.100 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:07.088+0000 7f25e2d9d700 1 -- 172.21.15.42:0/3018405508 >> v2:172.21.15.40:3300/0 conn(0x7f25e41191b0 msgr2=0x7f25e413a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:00:07.100 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:07.088+0000 7f25e2d9d700 1 --2- 172.21.15.42:0/3018405508 >> v2:172.21.15.40:3300/0 conn(0x7f25e41191b0 0x7f25e413a8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:00:07.100 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:07.088+0000 7f25e2d9d700 1 -- 172.21.15.42:0/3018405508 >> v1:172.21.15.42:6789/0 conn(0x7f25e40a5ef0 legacy=0x7f25e41185e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:00:07.100 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:07.088+0000 7f25e2d9d700 1 -- 172.21.15.42:0/3018405508 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25e4117ab0 con 0x7f25e4122fa0 2019-12-05T06:00:07.100 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:07.088+0000 7f25e2d9d700 1 --2- 172.21.15.42:0/3018405508 >> v2:172.21.15.42:3300/0 conn(0x7f25e4122fa0 0x7f25e4117560 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f25cc000b80 tx=0x7f25cc0109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:07.100 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:07.088+0000 7f25e2d9d700 1 --2- 172.21.15.42:0/3018405508 >> v2:172.21.15.42:3300/0 conn(0x7f25e4122fa0 0x7f25e4117560 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f25cc000b80 tx=0x7f25cc0109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:07.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:07.088+0000 7f25e2d9d700 1 --2- 172.21.15.42:0/3018405508 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f25e4122fa0 0x7f25e4117560 secure :-1 s=READY pgs=1834 cs=0 l=1 rx=0x7f25cc000b80 tx=0x7f25cc0109a0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:00:07.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:07.088+0000 7f25e2d9d700 1 --2- 172.21.15.42:0/3018405508 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f25e4122fa0 0x7f25e4117560 secure :-1 s=THROTTLE_DONE pgs=1834 cs=0 l=1 rx=0x7f25cc000b80 tx=0x7f25cc0109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:07.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:07.088+0000 7f25e2d9d700 1 --2- 172.21.15.42:0/3018405508 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f25e4122fa0 0x7f25e4117560 secure :-1 s=THROTTLE_DONE pgs=1834 cs=0 l=1 rx=0x7f25cc000b80 tx=0x7f25cc0109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:07.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:07.088+0000 7f25db7fe700 1 -- 172.21.15.42:0/3018405508 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f25cc00ca70 con 0x7f25e4122fa0 2019-12-05T06:00:07.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:07.088+0000 7f25e2d9d700 1 --2- 172.21.15.42:0/3018405508 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f25e4122fa0 0x7f25e4117560 secure :-1 s=THROTTLE_DONE pgs=1834 cs=0 l=1 rx=0x7f25cc000b80 tx=0x7f25cc0109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:07.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:07.088+0000 7f25db7fe700 1 -- 172.21.15.42:0/3018405508 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25cc00c690 con 0x7f25e4122fa0 2019-12-05T06:00:07.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:07.088+0000 7f25db7fe700 1 -- 172.21.15.42:0/3018405508 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f25cc0111a0 con 0x7f25e4122fa0 2019-12-05T06:00:07.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:07.088+0000 7f25e88af700 1 -- 172.21.15.42:0/3018405508 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25e41195e0 con 0x7f25e4122fa0 2019-12-05T06:00:07.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:07.088+0000 7f25e88af700 1 -- 172.21.15.42:0/3018405508 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25e413b060 con 0x7f25e4122fa0 2019-12-05T06:00:07.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:07.092+0000 7f25e2d9d700 1 --2- 172.21.15.42:0/3018405508 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f25e4122fa0 0x7f25e4117560 secure :-1 s=THROTTLE_DONE pgs=1834 cs=0 l=1 rx=0x7f25cc000b80 tx=0x7f25cc0109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:07.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:07.092+0000 7f25db7fe700 1 -- 172.21.15.42:0/3018405508 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f25cc0114b0 con 0x7f25e4122fa0 2019-12-05T06:00:07.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:07.092+0000 7f25e2d9d700 1 --2- 172.21.15.42:0/3018405508 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f25e4122fa0 0x7f25e4117560 secure :-1 s=THROTTLE_DONE pgs=1834 cs=0 l=1 rx=0x7f25cc000b80 tx=0x7f25cc0109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:07.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:07.092+0000 7f25db7fe700 1 --2- 172.21.15.42:0/3018405508 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f25c803b280 0x7f25c803d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:00:07.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:07.092+0000 7f25db7fe700 1 -- 172.21.15.42:0/3018405508 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9299+0+0 (secure 0 0 0) 0x7f25cc04b170 con 0x7f25e4122fa0 2019-12-05T06:00:07.103 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:07.092+0000 7f25e259c700 1 --2- 172.21.15.42:0/3018405508 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f25c803b280 0x7f25c803d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:00:07.103 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:07.092+0000 7f25e259c700 1 --2- 172.21.15.42:0/3018405508 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f25c803b280 0x7f25c803d790 crc :-1 s=READY pgs=783 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:00:07.103 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:07.096+0000 7f25e88af700 1 -- 172.21.15.42:0/3018405508 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25e405d110 con 0x7f25e4122fa0 2019-12-05T06:00:07.109 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:07.104+0000 7f25e2d9d700 1 --2- 172.21.15.42:0/3018405508 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f25e4122fa0 0x7f25e4117560 secure :-1 s=THROTTLE_DONE pgs=1834 cs=0 l=1 rx=0x7f25cc000b80 tx=0x7f25cc0109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:07.109 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:07.104+0000 7f25db7fe700 1 -- 172.21.15.42:0/3018405508 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f25cc0321f0 con 0x7f25e4122fa0 2019-12-05T06:00:07.274 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:07.264+0000 7f25e88af700 1 -- 172.21.15.42:0/3018405508 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f25e405b8c0 con 0x7f25e4122fa0 2019-12-05T06:00:07.274 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:07.264+0000 7f25e2d9d700 1 --2- 172.21.15.42:0/3018405508 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f25e4122fa0 0x7f25e4117560 secure :-1 s=THROTTLE_DONE pgs=1834 cs=0 l=1 rx=0x7f25cc000b80 tx=0x7f25cc0109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:07.274 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:07.264+0000 7f25db7fe700 1 -- 172.21.15.42:0/3018405508 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 223 v223) v1 ==== 95+0+2109 (secure 0 0 0) 0x7f25cc032450 con 0x7f25e4122fa0 2019-12-05T06:00:07.274 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 223 2019-12-05T06:00:07.275 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:07.268+0000 7f25e88af700 1 -- 172.21.15.42:0/3018405508 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f25c803b280 msgr2=0x7f25c803d790 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:00:07.275 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:07.268+0000 7f25e88af700 1 --2- 172.21.15.42:0/3018405508 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f25c803b280 0x7f25c803d790 crc :-1 s=READY pgs=783 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:00:07.275 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:07.268+0000 7f25e88af700 1 -- 172.21.15.42:0/3018405508 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f25e4122fa0 msgr2=0x7f25e4117560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:00:07.275 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:07.268+0000 7f25e88af700 1 --2- 172.21.15.42:0/3018405508 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f25e4122fa0 0x7f25e4117560 secure :-1 s=READY pgs=1834 cs=0 l=1 rx=0x7f25cc000b80 tx=0x7f25cc0109a0).stop 2019-12-05T06:00:07.275 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:07.268+0000 7f25e88af700 1 -- 172.21.15.42:0/3018405508 shutdown_connections 2019-12-05T06:00:07.275 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:07.268+0000 7f25e88af700 1 --2- 172.21.15.42:0/3018405508 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f25c803b280 0x7f25c803d790 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:00:07.275 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:07.268+0000 7f25e88af700 1 --2- 172.21.15.42:0/3018405508 >> v2:172.21.15.40:3300/0 conn(0x7f25e41191b0 0x7f25e413a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:00:07.276 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:07.268+0000 7f25e88af700 1 --2- 172.21.15.42:0/3018405508 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f25e4122fa0 0x7f25e4117560 unknown :-1 s=CLOSED pgs=1834 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:00:07.276 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:07.268+0000 7f25e88af700 1 -- 172.21.15.42:0/3018405508 shutdown_connections 2019-12-05T06:00:07.276 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:07.268+0000 7f25e88af700 1 -- 172.21.15.42:0/3018405508 wait complete. 2019-12-05T06:00:07.276 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:07.268+0000 7f25e88af700 1 -- 172.21.15.42:0/3018405508 >> 172.21.15.42:0/3018405508 conn(0x7f25e40a82a0 msgr2=0x7f25e409f220 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:00:07.276 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:00:07.277 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":223,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":9061,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6835/448869597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":448869597},{"type":"v1","addr":"172.21.15.42:6835","nonce":448869597}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":223}],"filesystems":[{"mdsmap":{"epoch":222,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:59:49.329977+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":87,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8940},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8940":{"gid":8940,"name":"b","rank":0,"incarnation":219,"state":"up:active","state_seq":36,"addr":"172.21.15.40:6833/1082666540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1082666540},{"type":"v1","addr":"172.21.15.40:6833","nonce":1082666540}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:00:07.283 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:00:09.284 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:00:09.400 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:09.372+0000 7f6ff3550700 1 Processor -- start 2019-12-05T06:00:09.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:09.372+0000 7f6ff3550700 1 -- start start 2019-12-05T06:00:09.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:09.376+0000 7f6ff3550700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6fec0a7740 0x7f6fec08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:00:09.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:09.376+0000 7f6ff3550700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6fec09fae0 0x7f6fec07eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:00:09.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:09.376+0000 7f6ff3550700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fec0a0710 con 0x7f6fec07e720 2019-12-05T06:00:09.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:09.376+0000 7f6ff3550700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fec0a0b00 con 0x7f6fec09fae0 2019-12-05T06:00:09.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:09.376+0000 7f6ff3550700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fec08e5b0 con 0x7f6fec0a7740 2019-12-05T06:00:09.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:09.376+0000 7f6ff22ee700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6fec09fae0 0x7f6fec07eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:00:09.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:09.376+0000 7f6ff12ec700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f6fec07e720 0x7f6fec09f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:55286/0 (socket says 172.21.15.42:55286) 2019-12-05T06:00:09.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:09.376+0000 7f6ff12ec700 1 -- 172.21.15.42:0/4158207180 learned_addr learned my addr 172.21.15.42:0/4158207180 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:00:09.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:09.376+0000 7f6ff1aed700 1 --2- 172.21.15.42:0/4158207180 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6fec0a7740 0x7f6fec08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:00:09.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:09.376+0000 7f6ff0aeb700 1 -- 172.21.15.42:0/4158207180 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4107439379 0 0) 0x7f6fec0a0710 con 0x7f6fec07e720 2019-12-05T06:00:09.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:09.376+0000 7f6ff0aeb700 1 -- 172.21.15.42:0/4158207180 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fd80030b0 con 0x7f6fec07e720 2019-12-05T06:00:09.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:09.376+0000 7f6ff22ee700 1 -- 172.21.15.42:0/4158207180 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6fec0a7740 msgr2=0x7f6fec08df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:00:09.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:09.376+0000 7f6ff22ee700 1 --2- 172.21.15.42:0/4158207180 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6fec0a7740 0x7f6fec08df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:00:09.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:09.376+0000 7f6ff22ee700 1 -- 172.21.15.42:0/4158207180 >> v1:172.21.15.40:6789/0 conn(0x7f6fec07e720 legacy=0x7f6fec09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:00:09.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:09.376+0000 7f6ff22ee700 1 -- 172.21.15.42:0/4158207180 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fec07f0a0 con 0x7f6fec09fae0 2019-12-05T06:00:09.404 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:09.376+0000 7f6ff22ee700 1 --2- 172.21.15.42:0/4158207180 >> v2:172.21.15.42:3300/0 conn(0x7f6fec09fae0 0x7f6fec07eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f6fe4008b10 tx=0x7f6fe400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:09.404 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:09.376+0000 7f6ff22ee700 1 --2- 172.21.15.42:0/4158207180 >> v2:172.21.15.42:3300/0 conn(0x7f6fec09fae0 0x7f6fec07eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f6fe4008b10 tx=0x7f6fe400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:09.404 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:09.376+0000 7f6ff22ee700 1 --2- 172.21.15.42:0/4158207180 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6fec09fae0 0x7f6fec07eb50 secure :-1 s=READY pgs=1835 cs=0 l=1 rx=0x7f6fe4008b10 tx=0x7f6fe400fff0).ready entity=mon.1 client_cookie=bb83b3ce191772ba server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:00:09.404 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:09.376+0000 7f6ff22ee700 1 --2- 172.21.15.42:0/4158207180 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6fec09fae0 0x7f6fec07eb50 secure :-1 s=THROTTLE_DONE pgs=1835 cs=0 l=1 rx=0x7f6fe4008b10 tx=0x7f6fe400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:09.404 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:09.376+0000 7f6ff0aeb700 1 -- 172.21.15.42:0/4158207180 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6fe4010900 con 0x7f6fec09fae0 2019-12-05T06:00:09.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:09.376+0000 7f6ff22ee700 1 --2- 172.21.15.42:0/4158207180 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6fec09fae0 0x7f6fec07eb50 secure :-1 s=THROTTLE_DONE pgs=1835 cs=0 l=1 rx=0x7f6fe4008b10 tx=0x7f6fe400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:09.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:09.376+0000 7f6ff22ee700 1 --2- 172.21.15.42:0/4158207180 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6fec09fae0 0x7f6fec07eb50 secure :-1 s=THROTTLE_DONE pgs=1835 cs=0 l=1 rx=0x7f6fe4008b10 tx=0x7f6fe400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:09.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:09.376+0000 7f6ff0aeb700 1 -- 172.21.15.42:0/4158207180 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fe4008b50 con 0x7f6fec09fae0 2019-12-05T06:00:09.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:09.376+0000 7f6ff0aeb700 1 -- 172.21.15.42:0/4158207180 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6fe40110c0 con 0x7f6fec09fae0 2019-12-05T06:00:09.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:09.376+0000 7f6ff3550700 1 -- 172.21.15.42:0/4158207180 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6fec09fae0 msgr2=0x7f6fec07eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:00:09.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:09.376+0000 7f6ff3550700 1 --2- 172.21.15.42:0/4158207180 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6fec09fae0 0x7f6fec07eb50 secure :-1 s=READY pgs=1835 cs=0 l=1 rx=0x7f6fe4008b10 tx=0x7f6fe400fff0).stop 2019-12-05T06:00:09.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:09.376+0000 7f6ff3550700 1 -- 172.21.15.42:0/4158207180 shutdown_connections 2019-12-05T06:00:09.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:09.376+0000 7f6ff3550700 1 --2- 172.21.15.42:0/4158207180 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6fec09fae0 0x7f6fec07eb50 unknown :-1 s=CLOSED pgs=1835 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:00:09.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:09.376+0000 7f6ff3550700 1 --2- 172.21.15.42:0/4158207180 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6fec0a7740 0x7f6fec08df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:00:09.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:09.376+0000 7f6ff3550700 1 -- 172.21.15.42:0/4158207180 shutdown_connections 2019-12-05T06:00:09.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:09.376+0000 7f6ff3550700 1 -- 172.21.15.42:0/4158207180 wait complete. 2019-12-05T06:00:09.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:09.376+0000 7f6ff3550700 1 -- 172.21.15.42:0/4158207180 >> 172.21.15.42:0/4158207180 conn(0x7f6fec0aabe0 msgr2=0x7f6fec0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:00:09.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:09.380+0000 7f6ff3550700 1 Processor -- start 2019-12-05T06:00:09.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:09.380+0000 7f6ff3550700 1 -- start start 2019-12-05T06:00:09.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:09.380+0000 7f6ff3550700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6fec11b800 0x7f6fec07d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:00:09.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:09.380+0000 7f6ff3550700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6fec1357d0 0x7f6fec137c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:00:09.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:09.380+0000 7f6ff3550700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fec11a490 con 0x7f6fec124430 2019-12-05T06:00:09.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:09.380+0000 7f6ff3550700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fec120b30 con 0x7f6fec11b800 2019-12-05T06:00:09.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:09.380+0000 7f6ff3550700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fec11bcd0 con 0x7f6fec1357d0 2019-12-05T06:00:09.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:09.380+0000 7f6ff12ec700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f6fec124430 0x7f6fec1229e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:45854/0 (socket says 172.21.15.42:45854) 2019-12-05T06:00:09.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:09.380+0000 7f6ff12ec700 1 -- 172.21.15.42:0/1424899839 learned_addr learned my addr 172.21.15.42:0/1424899839 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:00:09.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:09.380+0000 7f6ff1aed700 1 --2- 172.21.15.42:0/1424899839 >> v2:172.21.15.42:3300/0 conn(0x7f6fec11b800 0x7f6fec07d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:00:09.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:09.380+0000 7f6ff22ee700 1 --2- 172.21.15.42:0/1424899839 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6fec1357d0 0x7f6fec137c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:00:09.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:09.384+0000 7f6fde7fc700 1 -- 172.21.15.42:0/1424899839 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 183107288 0 0) 0x7f6fec11a490 con 0x7f6fec124430 2019-12-05T06:00:09.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:09.384+0000 7f6fde7fc700 1 -- 172.21.15.42:0/1424899839 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fd40030b0 con 0x7f6fec124430 2019-12-05T06:00:09.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:09.384+0000 7f6ff1aed700 1 -- 172.21.15.42:0/1424899839 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6fec1357d0 msgr2=0x7f6fec137c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:00:09.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:09.384+0000 7f6ff1aed700 1 --2- 172.21.15.42:0/1424899839 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6fec1357d0 0x7f6fec137c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:00:09.411 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:09.384+0000 7f6ff1aed700 1 -- 172.21.15.42:0/1424899839 >> v1:172.21.15.42:6789/0 conn(0x7f6fec124430 legacy=0x7f6fec1229e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:00:09.411 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:09.384+0000 7f6ff1aed700 1 -- 172.21.15.42:0/1424899839 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fec1231e0 con 0x7f6fec11b800 2019-12-05T06:00:09.411 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:09.384+0000 7f6ff1aed700 1 --2- 172.21.15.42:0/1424899839 >> v2:172.21.15.42:3300/0 conn(0x7f6fec11b800 0x7f6fec07d900 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f6fe0001b50 tx=0x7f6fe00016a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:09.411 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:09.384+0000 7f6ff1aed700 1 --2- 172.21.15.42:0/1424899839 >> v2:172.21.15.42:3300/0 conn(0x7f6fec11b800 0x7f6fec07d900 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f6fe0001b50 tx=0x7f6fe00016a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:09.411 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:09.384+0000 7f6ff1aed700 1 --2- 172.21.15.42:0/1424899839 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6fec11b800 0x7f6fec07d900 secure :-1 s=READY pgs=1837 cs=0 l=1 rx=0x7f6fe0001b50 tx=0x7f6fe00016a0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:00:09.412 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:09.384+0000 7f6ff1aed700 1 --2- 172.21.15.42:0/1424899839 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6fec11b800 0x7f6fec07d900 secure :-1 s=THROTTLE_DONE pgs=1837 cs=0 l=1 rx=0x7f6fe0001b50 tx=0x7f6fe00016a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:09.412 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:09.384+0000 7f6ff1aed700 1 --2- 172.21.15.42:0/1424899839 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6fec11b800 0x7f6fec07d900 secure :-1 s=THROTTLE_DONE pgs=1837 cs=0 l=1 rx=0x7f6fe0001b50 tx=0x7f6fe00016a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:09.412 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:09.384+0000 7f6ff1aed700 1 --2- 172.21.15.42:0/1424899839 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6fec11b800 0x7f6fec07d900 secure :-1 s=THROTTLE_DONE pgs=1837 cs=0 l=1 rx=0x7f6fe0001b50 tx=0x7f6fe00016a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:09.412 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:09.384+0000 7f6fde7fc700 1 -- 172.21.15.42:0/1424899839 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6fe0001950 con 0x7f6fec11b800 2019-12-05T06:00:09.412 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:09.384+0000 7f6fde7fc700 1 -- 172.21.15.42:0/1424899839 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fe0001760 con 0x7f6fec11b800 2019-12-05T06:00:09.413 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:09.384+0000 7f6ff3550700 1 -- 172.21.15.42:0/1424899839 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fec07de50 con 0x7f6fec11b800 2019-12-05T06:00:09.413 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:09.384+0000 7f6fde7fc700 1 -- 172.21.15.42:0/1424899839 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6fe8005b20 con 0x7f6fec11b800 2019-12-05T06:00:09.413 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:09.384+0000 7f6ff3550700 1 -- 172.21.15.42:0/1424899839 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fec123600 con 0x7f6fec11b800 2019-12-05T06:00:09.413 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:09.384+0000 7f6ff1aed700 1 --2- 172.21.15.42:0/1424899839 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6fec11b800 0x7f6fec07d900 secure :-1 s=THROTTLE_DONE pgs=1837 cs=0 l=1 rx=0x7f6fe0001b50 tx=0x7f6fe00016a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:09.413 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:09.384+0000 7f6fde7fc700 1 -- 172.21.15.42:0/1424899839 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f6fe800a250 con 0x7f6fec11b800 2019-12-05T06:00:09.414 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:09.384+0000 7f6ff1aed700 1 --2- 172.21.15.42:0/1424899839 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6fec11b800 0x7f6fec07d900 secure :-1 s=THROTTLE_DONE pgs=1837 cs=0 l=1 rx=0x7f6fe0001b50 tx=0x7f6fe00016a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:09.414 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:09.384+0000 7f6fde7fc700 1 --2- 172.21.15.42:0/1424899839 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6fd403b270 0x7f6fd403d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:00:09.414 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:09.384+0000 7f6fde7fc700 1 -- 172.21.15.42:0/1424899839 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9299+0+0 (secure 0 0 0) 0x7f6fe804a880 con 0x7f6fec11b800 2019-12-05T06:00:09.414 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:09.388+0000 7f6ff12ec700 1 --2- 172.21.15.42:0/1424899839 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6fd403b270 0x7f6fd403d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:00:09.414 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:09.388+0000 7f6ff12ec700 1 --2- 172.21.15.42:0/1424899839 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6fd403b270 0x7f6fd403d780 crc :-1 s=READY pgs=784 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:00:09.414 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:09.388+0000 7f6ff3550700 1 -- 172.21.15.42:0/1424899839 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fec05d110 con 0x7f6fec11b800 2019-12-05T06:00:09.415 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:09.396+0000 7f6ff1aed700 1 --2- 172.21.15.42:0/1424899839 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6fec11b800 0x7f6fec07d900 secure :-1 s=THROTTLE_DONE pgs=1837 cs=0 l=1 rx=0x7f6fe0001b50 tx=0x7f6fe00016a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:09.415 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:09.396+0000 7f6fde7fc700 1 -- 172.21.15.42:0/1424899839 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f6fe8031900 con 0x7f6fec11b800 2019-12-05T06:00:09.573 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:09.568+0000 7f6ff3550700 1 -- 172.21.15.42:0/1424899839 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6fec05b8c0 con 0x7f6fec11b800 2019-12-05T06:00:09.574 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:09.568+0000 7f6ff1aed700 1 --2- 172.21.15.42:0/1424899839 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6fec11b800 0x7f6fec07d900 secure :-1 s=THROTTLE_DONE pgs=1837 cs=0 l=1 rx=0x7f6fe0001b50 tx=0x7f6fe00016a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:09.574 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:09.568+0000 7f6fde7fc700 1 -- 172.21.15.42:0/1424899839 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 223 v223) v1 ==== 95+0+2109 (secure 0 0 0) 0x7f6fe8031b60 con 0x7f6fec11b800 2019-12-05T06:00:09.574 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 223 2019-12-05T06:00:09.575 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:00:09.575 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":223,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":9061,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6835/448869597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":448869597},{"type":"v1","addr":"172.21.15.42:6835","nonce":448869597}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":223}],"filesystems":[{"mdsmap":{"epoch":222,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:59:49.329977+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":87,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8940},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8940":{"gid":8940,"name":"b","rank":0,"incarnation":219,"state":"up:active","state_seq":36,"addr":"172.21.15.40:6833/1082666540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1082666540},{"type":"v1","addr":"172.21.15.40:6833","nonce":1082666540}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:00:09.578 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:09.572+0000 7f6ff3550700 1 -- 172.21.15.42:0/1424899839 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6fd403b270 msgr2=0x7f6fd403d780 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:00:09.578 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:09.572+0000 7f6ff3550700 1 --2- 172.21.15.42:0/1424899839 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6fd403b270 0x7f6fd403d780 crc :-1 s=READY pgs=784 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:00:09.578 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:09.572+0000 7f6ff3550700 1 -- 172.21.15.42:0/1424899839 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6fec11b800 msgr2=0x7f6fec07d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:00:09.578 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:09.572+0000 7f6ff3550700 1 --2- 172.21.15.42:0/1424899839 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6fec11b800 0x7f6fec07d900 secure :-1 s=READY pgs=1837 cs=0 l=1 rx=0x7f6fe0001b50 tx=0x7f6fe00016a0).stop 2019-12-05T06:00:09.578 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:09.572+0000 7f6ff3550700 1 -- 172.21.15.42:0/1424899839 shutdown_connections 2019-12-05T06:00:09.578 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:09.572+0000 7f6ff3550700 1 --2- 172.21.15.42:0/1424899839 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6fd403b270 0x7f6fd403d780 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:00:09.578 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:09.572+0000 7f6ff3550700 1 --2- 172.21.15.42:0/1424899839 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6fec1357d0 0x7f6fec137c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:00:09.579 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:09.572+0000 7f6ff3550700 1 --2- 172.21.15.42:0/1424899839 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6fec11b800 0x7f6fec07d900 unknown :-1 s=CLOSED pgs=1837 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:00:09.579 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:09.572+0000 7f6ff3550700 1 -- 172.21.15.42:0/1424899839 shutdown_connections 2019-12-05T06:00:09.579 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:09.572+0000 7f6ff3550700 1 -- 172.21.15.42:0/1424899839 wait complete. 2019-12-05T06:00:09.579 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:09.572+0000 7f6ff3550700 1 -- 172.21.15.42:0/1424899839 >> 172.21.15.42:0/1424899839 conn(0x7f6fec07e5d0 msgr2=0x7f6fec11c610 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:00:09.586 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:00:11.587 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:00:11.692 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:11.680+0000 7f3b3455f700 1 Processor -- start 2019-12-05T06:00:11.692 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:11.680+0000 7f3b3455f700 1 -- start start 2019-12-05T06:00:11.692 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:11.684+0000 7f3b3455f700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3b2c08df80 0x7f3b2c09f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:00:11.693 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:11.684+0000 7f3b3455f700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f3b2c09f830 0x7f3b2c14efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:00:11.693 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:11.684+0000 7f3b3455f700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b2c09fc60 con 0x7f3b2c0a7740 2019-12-05T06:00:11.693 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:11.684+0000 7f3b3455f700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b2c0a0b00 con 0x7f3b2c09f830 2019-12-05T06:00:11.693 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:11.684+0000 7f3b3455f700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b2c08e5b0 con 0x7f3b2c08df80 2019-12-05T06:00:11.693 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:11.684+0000 7f3b332fd700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f3b2c09f830 0x7f3b2c14efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:00:11.693 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:11.684+0000 7f3b332fd700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f3b2c09f830 0x7f3b2c14efe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.42:33644/0 (socket says 172.21.15.42:33644) 2019-12-05T06:00:11.694 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:11.684+0000 7f3b322fb700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3b2c08df80 0x7f3b2c09f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:00:11.694 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:11.684+0000 7f3b332fd700 1 -- 172.21.15.42:0/179499395 learned_addr learned my addr 172.21.15.42:0/179499395 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:00:11.694 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:11.684+0000 7f3b322fb700 1 -- 172.21.15.42:0/179499395 >> v2:172.21.15.40:3300/0 conn(0x7f3b2c09f830 msgr2=0x7f3b2c14efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:00:11.694 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:11.684+0000 7f3b322fb700 1 --2- 172.21.15.42:0/179499395 >> v2:172.21.15.40:3300/0 conn(0x7f3b2c09f830 0x7f3b2c14efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:00:11.694 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:11.684+0000 7f3b322fb700 1 -- 172.21.15.42:0/179499395 >> v1:172.21.15.40:6789/0 conn(0x7f3b2c0a7740 legacy=0x7f3b2c07e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:00:11.694 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:11.684+0000 7f3b322fb700 1 -- 172.21.15.42:0/179499395 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b2c0a0710 con 0x7f3b2c08df80 2019-12-05T06:00:11.694 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:11.684+0000 7f3b31afa700 1 -- 172.21.15.42:0/179499395 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1305554866 0 0) 0x7f3b2c09fc60 con 0x7f3b2c0a7740 2019-12-05T06:00:11.694 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:11.684+0000 7f3b332fd700 1 --2- 172.21.15.42:0/179499395 >> v2:172.21.15.40:3300/0 conn(0x7f3b2c09f830 0x7f3b2c14efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:00:11.695 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:11.684+0000 7f3b322fb700 1 --2- 172.21.15.42:0/179499395 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3b2c08df80 0x7f3b2c09f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f3b20008790 tx=0x7f3b2000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:11.695 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:11.684+0000 7f3b322fb700 1 --2- 172.21.15.42:0/179499395 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3b2c08df80 0x7f3b2c09f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f3b20008790 tx=0x7f3b2000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:11.695 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:11.684+0000 7f3b322fb700 1 --2- 172.21.15.42:0/179499395 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3b2c08df80 0x7f3b2c09f2e0 secure :-1 s=READY pgs=408 cs=0 l=1 rx=0x7f3b20008790 tx=0x7f3b2000fc70).ready entity=mon.2 client_cookie=36cd2b42d47285e1 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:00:11.695 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:11.684+0000 7f3b322fb700 1 --2- 172.21.15.42:0/179499395 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3b2c08df80 0x7f3b2c09f2e0 secure :-1 s=THROTTLE_DONE pgs=408 cs=0 l=1 rx=0x7f3b20008790 tx=0x7f3b2000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:11.695 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:11.684+0000 7f3b31afa700 1 -- 172.21.15.42:0/179499395 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3b20008bb0 con 0x7f3b2c08df80 2019-12-05T06:00:11.695 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:11.684+0000 7f3b322fb700 1 --2- 172.21.15.42:0/179499395 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3b2c08df80 0x7f3b2c09f2e0 secure :-1 s=THROTTLE_DONE pgs=408 cs=0 l=1 rx=0x7f3b20008790 tx=0x7f3b2000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:11.695 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:11.684+0000 7f3b322fb700 1 --2- 172.21.15.42:0/179499395 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3b2c08df80 0x7f3b2c09f2e0 secure :-1 s=THROTTLE_DONE pgs=408 cs=0 l=1 rx=0x7f3b20008790 tx=0x7f3b2000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:11.696 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:11.684+0000 7f3b31afa700 1 -- 172.21.15.42:0/179499395 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b200087d0 con 0x7f3b2c08df80 2019-12-05T06:00:11.696 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:11.684+0000 7f3b31afa700 1 -- 172.21.15.42:0/179499395 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3b20011860 con 0x7f3b2c08df80 2019-12-05T06:00:11.696 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:11.684+0000 7f3b3455f700 1 -- 172.21.15.42:0/179499395 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3b2c08df80 msgr2=0x7f3b2c09f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:00:11.696 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:11.684+0000 7f3b3455f700 1 --2- 172.21.15.42:0/179499395 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3b2c08df80 0x7f3b2c09f2e0 secure :-1 s=READY pgs=408 cs=0 l=1 rx=0x7f3b20008790 tx=0x7f3b2000fc70).stop 2019-12-05T06:00:11.696 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:11.688+0000 7f3b3455f700 1 -- 172.21.15.42:0/179499395 shutdown_connections 2019-12-05T06:00:11.696 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:11.688+0000 7f3b3455f700 1 --2- 172.21.15.42:0/179499395 >> v2:172.21.15.40:3300/0 conn(0x7f3b2c09f830 0x7f3b2c14efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:00:11.696 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:11.688+0000 7f3b3455f700 1 --2- 172.21.15.42:0/179499395 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3b2c08df80 0x7f3b2c09f2e0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:00:11.697 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:11.688+0000 7f3b3455f700 1 -- 172.21.15.42:0/179499395 shutdown_connections 2019-12-05T06:00:11.697 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:11.688+0000 7f3b3455f700 1 -- 172.21.15.42:0/179499395 wait complete. 2019-12-05T06:00:11.697 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:11.688+0000 7f3b3455f700 1 -- 172.21.15.42:0/179499395 >> 172.21.15.42:0/179499395 conn(0x7f3b2c0aabe0 msgr2=0x7f3b2c0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:00:11.701 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:11.688+0000 7f3b3455f700 1 Processor -- start 2019-12-05T06:00:11.701 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:11.688+0000 7f3b3455f700 1 -- start start 2019-12-05T06:00:11.701 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:11.688+0000 7f3b3455f700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3b2c12cf40 0x7f3b2c130df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:00:11.702 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:11.688+0000 7f3b3455f700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b2c131340 con 0x7f3b2c124580 2019-12-05T06:00:11.702 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:11.688+0000 7f3b3455f700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b2c0a6c90 con 0x7f3b2c0a6860 2019-12-05T06:00:11.702 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:11.688+0000 7f3b3455f700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b2c130610 con 0x7f3b2c12cf40 2019-12-05T06:00:11.702 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:11.692+0000 7f3b32afc700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3b2c12cf40 0x7f3b2c130df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:00:11.702 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:11.692+0000 7f3b332fd700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f3b2c124580 0x7f3b2c137950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:55310/0 (socket says 172.21.15.42:55310) 2019-12-05T06:00:11.702 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:11.692+0000 7f3b322fb700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f3b2c0a6860 0x7f3b2c122c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:45872/0 (socket says 172.21.15.42:45872) 2019-12-05T06:00:11.702 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:11.692+0000 7f3b32afc700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3b2c12cf40 0x7f3b2c130df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:45830/0 (socket says 172.21.15.42:45830) 2019-12-05T06:00:11.703 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:11.692+0000 7f3b332fd700 1 -- 172.21.15.42:0/842545110 learned_addr learned my addr 172.21.15.42:0/842545110 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:00:11.703 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:11.692+0000 7f3b1f7fe700 1 -- 172.21.15.42:0/842545110 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2681565625 0 0) 0x7f3b2c131340 con 0x7f3b2c124580 2019-12-05T06:00:11.703 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:11.692+0000 7f3b1f7fe700 1 -- 172.21.15.42:0/842545110 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b140030b0 con 0x7f3b2c124580 2019-12-05T06:00:11.703 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:11.692+0000 7f3b32afc700 1 -- 172.21.15.42:0/842545110 >> v1:172.21.15.42:6789/0 conn(0x7f3b2c0a6860 legacy=0x7f3b2c122c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:00:11.703 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:11.692+0000 7f3b1f7fe700 1 -- 172.21.15.42:0/842545110 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 389918960 0 0) 0x7f3b2c0a6c90 con 0x7f3b2c0a6860 2019-12-05T06:00:11.703 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:11.692+0000 7f3b32afc700 1 -- 172.21.15.42:0/842545110 >> v1:172.21.15.40:6789/0 conn(0x7f3b2c124580 legacy=0x7f3b2c137950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:00:11.703 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:11.692+0000 7f3b32afc700 1 -- 172.21.15.42:0/842545110 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b2c1249b0 con 0x7f3b2c12cf40 2019-12-05T06:00:11.704 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:11.692+0000 7f3b32afc700 1 --2- 172.21.15.42:0/842545110 >> v2:172.21.15.42:3300/0 conn(0x7f3b2c12cf40 0x7f3b2c130df0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f3b28000b80 tx=0x7f3b280129d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:11.704 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:11.692+0000 7f3b32afc700 1 --2- 172.21.15.42:0/842545110 >> v2:172.21.15.42:3300/0 conn(0x7f3b2c12cf40 0x7f3b2c130df0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f3b28000b80 tx=0x7f3b280129d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:11.704 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:11.692+0000 7f3b32afc700 1 --2- 172.21.15.42:0/842545110 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3b2c12cf40 0x7f3b2c130df0 secure :-1 s=READY pgs=1839 cs=0 l=1 rx=0x7f3b28000b80 tx=0x7f3b280129d0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:00:11.704 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:11.692+0000 7f3b32afc700 1 --2- 172.21.15.42:0/842545110 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3b2c12cf40 0x7f3b2c130df0 secure :-1 s=THROTTLE_DONE pgs=1839 cs=0 l=1 rx=0x7f3b28000b80 tx=0x7f3b280129d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:11.704 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:11.692+0000 7f3b1f7fe700 1 -- 172.21.15.42:0/842545110 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3b2800a970 con 0x7f3b2c12cf40 2019-12-05T06:00:11.704 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:11.692+0000 7f3b32afc700 1 --2- 172.21.15.42:0/842545110 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3b2c12cf40 0x7f3b2c130df0 secure :-1 s=THROTTLE_DONE pgs=1839 cs=0 l=1 rx=0x7f3b28000b80 tx=0x7f3b280129d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:11.705 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:11.692+0000 7f3b32afc700 1 --2- 172.21.15.42:0/842545110 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3b2c12cf40 0x7f3b2c130df0 secure :-1 s=THROTTLE_DONE pgs=1839 cs=0 l=1 rx=0x7f3b28000b80 tx=0x7f3b280129d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:11.705 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:11.692+0000 7f3b1f7fe700 1 -- 172.21.15.42:0/842545110 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b2800a590 con 0x7f3b2c12cf40 2019-12-05T06:00:11.705 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:11.692+0000 7f3b3455f700 1 -- 172.21.15.42:0/842545110 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b2c124ca0 con 0x7f3b2c12cf40 2019-12-05T06:00:11.705 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:11.692+0000 7f3b1f7fe700 1 -- 172.21.15.42:0/842545110 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3b28014480 con 0x7f3b2c12cf40 2019-12-05T06:00:11.705 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:11.692+0000 7f3b3455f700 1 -- 172.21.15.42:0/842545110 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b2c08eb70 con 0x7f3b2c12cf40 2019-12-05T06:00:11.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:11.692+0000 7f3b32afc700 1 --2- 172.21.15.42:0/842545110 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3b2c12cf40 0x7f3b2c130df0 secure :-1 s=THROTTLE_DONE pgs=1839 cs=0 l=1 rx=0x7f3b28000b80 tx=0x7f3b280129d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:11.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:11.696+0000 7f3b1f7fe700 1 -- 172.21.15.42:0/842545110 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f3b28014790 con 0x7f3b2c12cf40 2019-12-05T06:00:11.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:11.696+0000 7f3b32afc700 1 --2- 172.21.15.42:0/842545110 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3b2c12cf40 0x7f3b2c130df0 secure :-1 s=THROTTLE_DONE pgs=1839 cs=0 l=1 rx=0x7f3b28000b80 tx=0x7f3b280129d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:11.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:11.696+0000 7f3b1f7fe700 1 --2- 172.21.15.42:0/842545110 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3b1403b230 0x7f3b1403d740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:00:11.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:11.696+0000 7f3b1f7fe700 1 -- 172.21.15.42:0/842545110 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9299+0+0 (secure 0 0 0) 0x7f3b2804cf70 con 0x7f3b2c12cf40 2019-12-05T06:00:11.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:11.696+0000 7f3b322fb700 1 --2- 172.21.15.42:0/842545110 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3b1403b230 0x7f3b1403d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:00:11.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:11.696+0000 7f3b322fb700 1 --2- 172.21.15.42:0/842545110 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3b1403b230 0x7f3b1403d740 crc :-1 s=READY pgs=785 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:00:11.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:11.700+0000 7f3b3455f700 1 -- 172.21.15.42:0/842545110 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b2c05d110 con 0x7f3b2c12cf40 2019-12-05T06:00:11.709 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:11.704+0000 7f3b32afc700 1 --2- 172.21.15.42:0/842545110 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3b2c12cf40 0x7f3b2c130df0 secure :-1 s=THROTTLE_DONE pgs=1839 cs=0 l=1 rx=0x7f3b28000b80 tx=0x7f3b280129d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:11.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:11.704+0000 7f3b1f7fe700 1 -- 172.21.15.42:0/842545110 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f3b28033ff0 con 0x7f3b2c12cf40 2019-12-05T06:00:11.872 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:11.868+0000 7f3b3455f700 1 -- 172.21.15.42:0/842545110 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3b2c05b8c0 con 0x7f3b2c12cf40 2019-12-05T06:00:11.874 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:11.868+0000 7f3b32afc700 1 --2- 172.21.15.42:0/842545110 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3b2c12cf40 0x7f3b2c130df0 secure :-1 s=THROTTLE_DONE pgs=1839 cs=0 l=1 rx=0x7f3b28000b80 tx=0x7f3b280129d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:11.874 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:11.868+0000 7f3b1f7fe700 1 -- 172.21.15.42:0/842545110 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 223 v223) v1 ==== 95+0+2109 (secure 0 0 0) 0x7f3b28034250 con 0x7f3b2c12cf40 2019-12-05T06:00:11.875 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 223 2019-12-05T06:00:11.875 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:00:11.875 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":223,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":9061,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6835/448869597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":448869597},{"type":"v1","addr":"172.21.15.42:6835","nonce":448869597}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":223}],"filesystems":[{"mdsmap":{"epoch":222,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:59:49.329977+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":87,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8940},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8940":{"gid":8940,"name":"b","rank":0,"incarnation":219,"state":"up:active","state_seq":36,"addr":"172.21.15.40:6833/1082666540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1082666540},{"type":"v1","addr":"172.21.15.40:6833","nonce":1082666540}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:00:11.878 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:11.872+0000 7f3b3455f700 1 -- 172.21.15.42:0/842545110 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3b1403b230 msgr2=0x7f3b1403d740 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:00:11.878 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:11.872+0000 7f3b3455f700 1 --2- 172.21.15.42:0/842545110 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3b1403b230 0x7f3b1403d740 crc :-1 s=READY pgs=785 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:00:11.878 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:11.872+0000 7f3b3455f700 1 -- 172.21.15.42:0/842545110 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3b2c12cf40 msgr2=0x7f3b2c130df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:00:11.878 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:11.872+0000 7f3b3455f700 1 --2- 172.21.15.42:0/842545110 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3b2c12cf40 0x7f3b2c130df0 secure :-1 s=READY pgs=1839 cs=0 l=1 rx=0x7f3b28000b80 tx=0x7f3b280129d0).stop 2019-12-05T06:00:11.878 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:11.872+0000 7f3b3455f700 1 -- 172.21.15.42:0/842545110 shutdown_connections 2019-12-05T06:00:11.879 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:11.872+0000 7f3b3455f700 1 --2- 172.21.15.42:0/842545110 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3b1403b230 0x7f3b1403d740 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:00:11.879 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:11.872+0000 7f3b3455f700 1 --2- 172.21.15.42:0/842545110 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3b2c12cf40 0x7f3b2c130df0 unknown :-1 s=CLOSED pgs=1839 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:00:11.879 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:11.872+0000 7f3b3455f700 1 -- 172.21.15.42:0/842545110 shutdown_connections 2019-12-05T06:00:11.879 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:11.872+0000 7f3b3455f700 1 -- 172.21.15.42:0/842545110 wait complete. 2019-12-05T06:00:11.880 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:11.872+0000 7f3b3455f700 1 -- 172.21.15.42:0/842545110 >> 172.21.15.42:0/842545110 conn(0x7f3b2c066fe0 msgr2=0x7f3b2c11c8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:00:11.888 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:00:13.034 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:00:13.038 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:00:13.889 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:00:13.996 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:13.984+0000 7fa34fd66700 1 Processor -- start 2019-12-05T06:00:13.996 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:13.984+0000 7fa34fd66700 1 -- start start 2019-12-05T06:00:13.996 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:13.988+0000 7fa34fd66700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa34808df80 0x7fa34809f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:00:13.996 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:13.988+0000 7fa34fd66700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa34809f830 0x7fa34814efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:00:13.997 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:13.988+0000 7fa34fd66700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa34809fc60 con 0x7fa3480a7740 2019-12-05T06:00:13.997 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:13.988+0000 7fa34fd66700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3480a0b00 con 0x7fa34808df80 2019-12-05T06:00:13.997 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:13.988+0000 7fa34fd66700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa34808e5b0 con 0x7fa34809f830 2019-12-05T06:00:13.997 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:13.988+0000 7fa34db02700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa34808df80 0x7fa34809f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:00:13.997 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:13.988+0000 7fa34db02700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa34808df80 0x7fa34809f2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.42:33656/0 (socket says 172.21.15.42:33656) 2019-12-05T06:00:13.997 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:13.988+0000 7fa34db02700 1 -- 172.21.15.42:0/107967991 learned_addr learned my addr 172.21.15.42:0/107967991 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:00:13.998 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:13.988+0000 7fa34eb04700 1 --2- 172.21.15.42:0/107967991 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa34809f830 0x7fa34814efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:00:13.998 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:13.988+0000 7fa34db02700 1 -- 172.21.15.42:0/107967991 >> v2:172.21.15.40:3300/0 conn(0x7fa34808df80 msgr2=0x7fa34809f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2019-12-05T06:00:13.998 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:13.988+0000 7fa34db02700 1 -- 172.21.15.42:0/107967991 >> v2:172.21.15.40:3300/0 conn(0x7fa34808df80 msgr2=0x7fa34809f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2019-12-05T06:00:13.998 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:13.988+0000 7fa34db02700 1 --2- 172.21.15.42:0/107967991 >> v2:172.21.15.40:3300/0 conn(0x7fa34808df80 0x7fa34809f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-12-05T06:00:13.998 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:13.988+0000 7fa34db02700 1 --2- 172.21.15.42:0/107967991 >> v2:172.21.15.40:3300/0 conn(0x7fa34808df80 0x7fa34809f2e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rx=0 tx=0)._fault waiting 0.200000 2019-12-05T06:00:13.998 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:13.988+0000 7fa34eb04700 1 -- 172.21.15.42:0/107967991 >> v2:172.21.15.40:3300/0 conn(0x7fa34808df80 msgr2=0x7fa34809f2e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2019-12-05T06:00:13.998 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:13.988+0000 7fa34eb04700 1 --2- 172.21.15.42:0/107967991 >> v2:172.21.15.40:3300/0 conn(0x7fa34808df80 0x7fa34809f2e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:00:13.999 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:13.988+0000 7fa34eb04700 1 -- 172.21.15.42:0/107967991 >> v1:172.21.15.40:6789/0 conn(0x7fa3480a7740 legacy=0x7fa34807e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:00:13.999 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:13.988+0000 7fa34eb04700 1 -- 172.21.15.42:0/107967991 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3480a0710 con 0x7fa34809f830 2019-12-05T06:00:13.999 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:13.988+0000 7fa34d301700 1 -- 172.21.15.42:0/107967991 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1050686326 0 0) 0x7fa34809fc60 con 0x7fa3480a7740 2019-12-05T06:00:13.999 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:13.988+0000 7fa34eb04700 1 --2- 172.21.15.42:0/107967991 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa34809f830 0x7fa34814efe0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fa340008b10 tx=0x7fa34000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:13.999 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:13.988+0000 7fa34eb04700 1 --2- 172.21.15.42:0/107967991 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa34809f830 0x7fa34814efe0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fa340008b10 tx=0x7fa34000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:13.999 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:13.988+0000 7fa34eb04700 1 --2- 172.21.15.42:0/107967991 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa34809f830 0x7fa34814efe0 secure :-1 s=READY pgs=409 cs=0 l=1 rx=0x7fa340008b10 tx=0x7fa34000fff0).ready entity=mon.2 client_cookie=6c526ee37e942f9e server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:00:13.999 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:13.988+0000 7fa34eb04700 1 --2- 172.21.15.42:0/107967991 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa34809f830 0x7fa34814efe0 secure :-1 s=THROTTLE_DONE pgs=409 cs=0 l=1 rx=0x7fa340008b10 tx=0x7fa34000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:14.000 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:13.988+0000 7fa34eb04700 1 --2- 172.21.15.42:0/107967991 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa34809f830 0x7fa34814efe0 secure :-1 s=THROTTLE_DONE pgs=409 cs=0 l=1 rx=0x7fa340008b10 tx=0x7fa34000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:14.000 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:13.988+0000 7fa34d301700 1 -- 172.21.15.42:0/107967991 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa340010900 con 0x7fa34809f830 2019-12-05T06:00:14.000 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:13.988+0000 7fa34eb04700 1 --2- 172.21.15.42:0/107967991 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa34809f830 0x7fa34814efe0 secure :-1 s=THROTTLE_DONE pgs=409 cs=0 l=1 rx=0x7fa340008b10 tx=0x7fa34000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:14.000 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:13.988+0000 7fa34d301700 1 -- 172.21.15.42:0/107967991 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa340008b50 con 0x7fa34809f830 2019-12-05T06:00:14.001 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:13.988+0000 7fa34d301700 1 -- 172.21.15.42:0/107967991 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa3400110c0 con 0x7fa34809f830 2019-12-05T06:00:14.001 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:13.988+0000 7fa34fd66700 1 -- 172.21.15.42:0/107967991 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa34809f830 msgr2=0x7fa34814efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:00:14.001 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:13.988+0000 7fa34fd66700 1 --2- 172.21.15.42:0/107967991 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa34809f830 0x7fa34814efe0 secure :-1 s=READY pgs=409 cs=0 l=1 rx=0x7fa340008b10 tx=0x7fa34000fff0).stop 2019-12-05T06:00:14.001 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:13.988+0000 7fa34fd66700 1 -- 172.21.15.42:0/107967991 shutdown_connections 2019-12-05T06:00:14.001 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:13.988+0000 7fa34fd66700 1 --2- 172.21.15.42:0/107967991 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa34809f830 0x7fa34814efe0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:00:14.001 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:13.988+0000 7fa34fd66700 1 --2- 172.21.15.42:0/107967991 >> v2:172.21.15.40:3300/0 conn(0x7fa34808df80 0x7fa34809f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:00:14.002 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:13.992+0000 7fa34fd66700 1 -- 172.21.15.42:0/107967991 shutdown_connections 2019-12-05T06:00:14.002 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:13.992+0000 7fa34fd66700 1 -- 172.21.15.42:0/107967991 wait complete. 2019-12-05T06:00:14.002 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:13.992+0000 7fa34fd66700 1 -- 172.21.15.42:0/107967991 >> 172.21.15.42:0/107967991 conn(0x7fa3480aabe0 msgr2=0x7fa3480a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:00:14.005 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:13.992+0000 7fa34fd66700 1 Processor -- start 2019-12-05T06:00:14.005 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:13.992+0000 7fa34fd66700 1 -- start start 2019-12-05T06:00:14.006 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:13.992+0000 7fa34fd66700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fa34811b880 0x7fa3480a5b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:00:14.006 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:13.992+0000 7fa34fd66700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa34811ac60 con 0x7fa3480a6e60 2019-12-05T06:00:14.006 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:13.992+0000 7fa34fd66700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa348120450 con 0x7fa3480a60e0 2019-12-05T06:00:14.006 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:13.992+0000 7fa34fd66700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa34811a6d0 con 0x7fa34811b880 2019-12-05T06:00:14.006 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:13.992+0000 7fa34eb04700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fa3480a60e0 0x7fa348137920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:45884/0 (socket says 172.21.15.42:45884) 2019-12-05T06:00:14.006 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:13.992+0000 7fa34e303700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fa34811b880 0x7fa3480a5b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:00:14.006 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:13.992+0000 7fa34eb04700 1 -- 172.21.15.42:0/2346498514 learned_addr learned my addr 172.21.15.42:0/2346498514 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:00:14.007 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:13.996+0000 7fa33effd700 1 -- 172.21.15.42:0/2346498514 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3288196674 0 0) 0x7fa34811ac60 con 0x7fa3480a6e60 2019-12-05T06:00:14.007 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:13.996+0000 7fa33effd700 1 -- 172.21.15.42:0/2346498514 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3300030b0 con 0x7fa3480a6e60 2019-12-05T06:00:14.007 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:13.996+0000 7fa34e303700 1 -- 172.21.15.42:0/2346498514 >> v1:172.21.15.42:6789/0 conn(0x7fa3480a60e0 legacy=0x7fa348137920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:00:14.007 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:13.996+0000 7fa33effd700 1 -- 172.21.15.42:0/2346498514 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4079622197 0 0) 0x7fa348120450 con 0x7fa3480a60e0 2019-12-05T06:00:14.007 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:13.996+0000 7fa34e303700 1 -- 172.21.15.42:0/2346498514 >> v1:172.21.15.40:6789/0 conn(0x7fa3480a6e60 legacy=0x7fa34807ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:00:14.007 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:13.996+0000 7fa34e303700 1 -- 172.21.15.42:0/2346498514 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa34807b470 con 0x7fa34811b880 2019-12-05T06:00:14.007 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:13.996+0000 7fa34e303700 1 --2- 172.21.15.42:0/2346498514 >> v2:172.21.15.42:3300/0 conn(0x7fa34811b880 0x7fa3480a5b90 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fa344000e90 tx=0x7fa34400c2f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:14.008 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:13.996+0000 7fa34e303700 1 --2- 172.21.15.42:0/2346498514 >> v2:172.21.15.42:3300/0 conn(0x7fa34811b880 0x7fa3480a5b90 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fa344000e90 tx=0x7fa34400c2f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:14.008 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:13.996+0000 7fa34e303700 1 --2- 172.21.15.42:0/2346498514 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa34811b880 0x7fa3480a5b90 secure :-1 s=READY pgs=1841 cs=0 l=1 rx=0x7fa344000e90 tx=0x7fa34400c2f0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:00:14.008 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:13.996+0000 7fa34e303700 1 --2- 172.21.15.42:0/2346498514 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa34811b880 0x7fa3480a5b90 secure :-1 s=THROTTLE_DONE pgs=1841 cs=0 l=1 rx=0x7fa344000e90 tx=0x7fa34400c2f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:14.008 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:13.996+0000 7fa33effd700 1 -- 172.21.15.42:0/2346498514 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa34400cb90 con 0x7fa34811b880 2019-12-05T06:00:14.008 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:13.996+0000 7fa34e303700 1 --2- 172.21.15.42:0/2346498514 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa34811b880 0x7fa3480a5b90 secure :-1 s=THROTTLE_DONE pgs=1841 cs=0 l=1 rx=0x7fa344000e90 tx=0x7fa34400c2f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:14.008 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:13.996+0000 7fa34fd66700 1 -- 172.21.15.42:0/2346498514 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa348139130 con 0x7fa34811b880 2019-12-05T06:00:14.008 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:13.996+0000 7fa34e303700 1 --2- 172.21.15.42:0/2346498514 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa34811b880 0x7fa3480a5b90 secure :-1 s=THROTTLE_DONE pgs=1841 cs=0 l=1 rx=0x7fa344000e90 tx=0x7fa34400c2f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:14.009 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:13.996+0000 7fa33effd700 1 -- 172.21.15.42:0/2346498514 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa34400c7b0 con 0x7fa34811b880 2019-12-05T06:00:14.009 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:13.996+0000 7fa33effd700 1 -- 172.21.15.42:0/2346498514 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa344011070 con 0x7fa34811b880 2019-12-05T06:00:14.009 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:13.996+0000 7fa34fd66700 1 -- 172.21.15.42:0/2346498514 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa348139660 con 0x7fa34811b880 2019-12-05T06:00:14.009 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:13.996+0000 7fa34e303700 1 --2- 172.21.15.42:0/2346498514 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa34811b880 0x7fa3480a5b90 secure :-1 s=THROTTLE_DONE pgs=1841 cs=0 l=1 rx=0x7fa344000e90 tx=0x7fa34400c2f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:14.009 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:13.996+0000 7fa33effd700 1 -- 172.21.15.42:0/2346498514 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fa344011350 con 0x7fa34811b880 2019-12-05T06:00:14.009 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:13.996+0000 7fa34e303700 1 --2- 172.21.15.42:0/2346498514 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa34811b880 0x7fa3480a5b90 secure :-1 s=THROTTLE_DONE pgs=1841 cs=0 l=1 rx=0x7fa344000e90 tx=0x7fa34400c2f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:14.009 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:13.996+0000 7fa33effd700 1 --2- 172.21.15.42:0/2346498514 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa33003b190 0x7fa33003d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:00:14.010 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:13.996+0000 7fa33effd700 1 -- 172.21.15.42:0/2346498514 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9299+0+0 (secure 0 0 0) 0x7fa34404b180 con 0x7fa34811b880 2019-12-05T06:00:14.010 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:14.000+0000 7fa34db02700 1 --2- 172.21.15.42:0/2346498514 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa33003b190 0x7fa33003d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:00:14.010 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:14.000+0000 7fa34db02700 1 --2- 172.21.15.42:0/2346498514 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa33003b190 0x7fa33003d6a0 crc :-1 s=READY pgs=786 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:00:14.010 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:14.000+0000 7fa34fd66700 1 -- 172.21.15.42:0/2346498514 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa34805d110 con 0x7fa34811b880 2019-12-05T06:00:14.013 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:14.008+0000 7fa34e303700 1 --2- 172.21.15.42:0/2346498514 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa34811b880 0x7fa3480a5b90 secure :-1 s=THROTTLE_DONE pgs=1841 cs=0 l=1 rx=0x7fa344000e90 tx=0x7fa34400c2f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:14.013 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:14.008+0000 7fa33effd700 1 -- 172.21.15.42:0/2346498514 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fa3440321e0 con 0x7fa34811b880 2019-12-05T06:00:14.187 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:14.180+0000 7fa34fd66700 1 -- 172.21.15.42:0/2346498514 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa34805b8c0 con 0x7fa34811b880 2019-12-05T06:00:14.187 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:14.184+0000 7fa34e303700 1 --2- 172.21.15.42:0/2346498514 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa34811b880 0x7fa3480a5b90 secure :-1 s=THROTTLE_DONE pgs=1841 cs=0 l=1 rx=0x7fa344000e90 tx=0x7fa34400c2f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:14.187 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:14.184+0000 7fa33effd700 1 -- 172.21.15.42:0/2346498514 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 223 v223) v1 ==== 95+0+2109 (secure 0 0 0) 0x7fa344032440 con 0x7fa34811b880 2019-12-05T06:00:14.189 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:00:14.189 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":223,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":9061,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6835/448869597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":448869597},{"type":"v1","addr":"172.21.15.42:6835","nonce":448869597}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":223}],"filesystems":[{"mdsmap":{"epoch":222,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T05:59:49.329977+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":87,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8940},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8940":{"gid":8940,"name":"b","rank":0,"incarnation":219,"state":"up:active","state_seq":36,"addr":"172.21.15.40:6833/1082666540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1082666540},{"type":"v1","addr":"172.21.15.40:6833","nonce":1082666540}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:00:14.189 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 223 2019-12-05T06:00:14.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:14.188+0000 7fa34fd66700 1 -- 172.21.15.42:0/2346498514 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa33003b190 msgr2=0x7fa33003d6a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:00:14.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:14.188+0000 7fa34fd66700 1 --2- 172.21.15.42:0/2346498514 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa33003b190 0x7fa33003d6a0 crc :-1 s=READY pgs=786 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:00:14.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:14.188+0000 7fa34fd66700 1 -- 172.21.15.42:0/2346498514 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa34811b880 msgr2=0x7fa3480a5b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:00:14.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:14.188+0000 7fa34fd66700 1 --2- 172.21.15.42:0/2346498514 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa34811b880 0x7fa3480a5b90 secure :-1 s=READY pgs=1841 cs=0 l=1 rx=0x7fa344000e90 tx=0x7fa34400c2f0).stop 2019-12-05T06:00:14.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:14.188+0000 7fa34fd66700 1 -- 172.21.15.42:0/2346498514 shutdown_connections 2019-12-05T06:00:14.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:14.188+0000 7fa34fd66700 1 --2- 172.21.15.42:0/2346498514 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa33003b190 0x7fa33003d6a0 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:00:14.193 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:14.188+0000 7fa34fd66700 1 --2- 172.21.15.42:0/2346498514 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa34811b880 0x7fa3480a5b90 unknown :-1 s=CLOSED pgs=1841 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:00:14.193 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:14.188+0000 7fa34fd66700 1 -- 172.21.15.42:0/2346498514 shutdown_connections 2019-12-05T06:00:14.193 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:14.188+0000 7fa34fd66700 1 -- 172.21.15.42:0/2346498514 wait complete. 2019-12-05T06:00:14.193 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:14.188+0000 7fa34fd66700 1 -- 172.21.15.42:0/2346498514 >> 172.21.15.42:0/2346498514 conn(0x7fa34811d080 msgr2=0x7fa34807e650 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:00:14.202 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:00:16.203 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:00:16.314 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.304+0000 7f055c7ca700 1 Processor -- start 2019-12-05T06:00:16.314 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.304+0000 7f055c7ca700 1 -- start start 2019-12-05T06:00:16.314 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.308+0000 7f055c7ca700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f05540a7730 0x7f055408df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:00:16.315 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.308+0000 7f055c7ca700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f055407e710 0x7f055407eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:00:16.315 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.308+0000 7f055c7ca700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f055409f2d0 0x7f055409f700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:00:16.315 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.308+0000 7f055c7ca700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05540a0af0 con 0x7f055409f2d0 2019-12-05T06:00:16.315 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.308+0000 7f055c7ca700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f055408e5a0 con 0x7f05540a7730 2019-12-05T06:00:16.315 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.308+0000 7f055c7ca700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f055408e840 con 0x7f055407e710 2019-12-05T06:00:16.315 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.308+0000 7f055ad67700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f05540a7730 0x7f055408df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:00:16.316 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.308+0000 7f055ad67700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f05540a7730 0x7f055408df70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:45866/0 (socket says 172.21.15.42:45866) 2019-12-05T06:00:16.316 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.308+0000 7f055ad67700 1 -- 172.21.15.42:0/2710923776 learned_addr learned my addr 172.21.15.42:0/2710923776 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:00:16.316 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.308+0000 7f055b568700 1 --2- 172.21.15.42:0/2710923776 >> v2:172.21.15.40:3300/0 conn(0x7f055409f2d0 0x7f055409f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:00:16.316 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.308+0000 7f055a566700 1 --2- 172.21.15.42:0/2710923776 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f055407e710 0x7f055407eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:00:16.316 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.308+0000 7f055ad67700 1 -- 172.21.15.42:0/2710923776 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f055407e710 msgr2=0x7f055407eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:00:16.316 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.308+0000 7f055ad67700 1 --2- 172.21.15.42:0/2710923776 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f055407e710 0x7f055407eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:00:16.316 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.308+0000 7f055ad67700 1 -- 172.21.15.42:0/2710923776 >> v2:172.21.15.40:3300/0 conn(0x7f055409f2d0 msgr2=0x7f055409f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:00:16.317 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.308+0000 7f055ad67700 1 --2- 172.21.15.42:0/2710923776 >> v2:172.21.15.40:3300/0 conn(0x7f055409f2d0 0x7f055409f700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:00:16.317 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.308+0000 7f055ad67700 1 -- 172.21.15.42:0/2710923776 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f055407f090 con 0x7f05540a7730 2019-12-05T06:00:16.317 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.308+0000 7f055ad67700 1 --2- 172.21.15.42:0/2710923776 >> v2:172.21.15.42:3300/0 conn(0x7f05540a7730 0x7f055408df70 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f055408e760 tx=0x7f054c010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:16.317 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.308+0000 7f055ad67700 1 --2- 172.21.15.42:0/2710923776 >> v2:172.21.15.42:3300/0 conn(0x7f05540a7730 0x7f055408df70 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f055408e760 tx=0x7f054c010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:16.317 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.308+0000 7f055ad67700 1 --2- 172.21.15.42:0/2710923776 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f05540a7730 0x7f055408df70 secure :-1 s=READY pgs=1842 cs=0 l=1 rx=0x7f055408e760 tx=0x7f054c010130).ready entity=mon.1 client_cookie=25cd23f6e6b432d3 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:00:16.317 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.308+0000 7f055ad67700 1 --2- 172.21.15.42:0/2710923776 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f05540a7730 0x7f055408df70 secure :-1 s=THROTTLE_DONE pgs=1842 cs=0 l=1 rx=0x7f055408e760 tx=0x7f054c010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:16.317 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.308+0000 7f055ad67700 1 --2- 172.21.15.42:0/2710923776 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f05540a7730 0x7f055408df70 secure :-1 s=THROTTLE_DONE pgs=1842 cs=0 l=1 rx=0x7f055408e760 tx=0x7f054c010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:16.318 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.308+0000 7f0559d65700 1 -- 172.21.15.42:0/2710923776 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f054c010de0 con 0x7f05540a7730 2019-12-05T06:00:16.318 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.308+0000 7f055ad67700 1 --2- 172.21.15.42:0/2710923776 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f05540a7730 0x7f055408df70 secure :-1 s=THROTTLE_DONE pgs=1842 cs=0 l=1 rx=0x7f055408e760 tx=0x7f054c010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:16.318 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.308+0000 7f0559d65700 1 -- 172.21.15.42:0/2710923776 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f054c010a00 con 0x7f05540a7730 2019-12-05T06:00:16.318 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.308+0000 7f0559d65700 1 -- 172.21.15.42:0/2710923776 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f054c0114a0 con 0x7f05540a7730 2019-12-05T06:00:16.318 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.308+0000 7f055c7ca700 1 -- 172.21.15.42:0/2710923776 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f05540a7730 msgr2=0x7f055408df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:00:16.318 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.308+0000 7f055c7ca700 1 --2- 172.21.15.42:0/2710923776 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f05540a7730 0x7f055408df70 secure :-1 s=READY pgs=1842 cs=0 l=1 rx=0x7f055408e760 tx=0x7f054c010130).stop 2019-12-05T06:00:16.318 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.308+0000 7f055c7ca700 1 -- 172.21.15.42:0/2710923776 shutdown_connections 2019-12-05T06:00:16.319 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.308+0000 7f055c7ca700 1 --2- 172.21.15.42:0/2710923776 >> v2:172.21.15.40:3300/0 conn(0x7f055409f2d0 0x7f055409f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:00:16.319 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.308+0000 7f055c7ca700 1 --2- 172.21.15.42:0/2710923776 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f05540a7730 0x7f055408df70 unknown :-1 s=CLOSED pgs=1842 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:00:16.319 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.308+0000 7f055c7ca700 1 --2- 172.21.15.42:0/2710923776 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f055407e710 0x7f055407eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:00:16.319 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.308+0000 7f055c7ca700 1 -- 172.21.15.42:0/2710923776 shutdown_connections 2019-12-05T06:00:16.319 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.308+0000 7f055c7ca700 1 -- 172.21.15.42:0/2710923776 wait complete. 2019-12-05T06:00:16.319 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.308+0000 7f055c7ca700 1 -- 172.21.15.42:0/2710923776 >> 172.21.15.42:0/2710923776 conn(0x7f05540aabd0 msgr2=0x7f05540a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:00:16.322 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.308+0000 7f055c7ca700 1 Processor -- start 2019-12-05T06:00:16.322 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.308+0000 7f055c7ca700 1 -- start start 2019-12-05T06:00:16.323 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.312+0000 7f055c7ca700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f0554064880 0x7f055411bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:00:16.323 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.312+0000 7f055c7ca700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0554065ab0 con 0x7f055411b3c0 2019-12-05T06:00:16.323 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.312+0000 7f055c7ca700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0554078f30 con 0x7f05540742d0 2019-12-05T06:00:16.323 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.312+0000 7f055c7ca700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0554065ce0 con 0x7f0554064880 2019-12-05T06:00:16.323 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.312+0000 7f055b568700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f05540742d0 0x7f0554076710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:45914/0 (socket says 172.21.15.42:45914) 2019-12-05T06:00:16.323 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.312+0000 7f055b568700 1 -- 172.21.15.42:0/259783107 learned_addr learned my addr 172.21.15.42:0/259783107 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:00:16.324 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.312+0000 7f055a566700 1 --2- 172.21.15.42:0/259783107 >> v2:172.21.15.40:3300/0 conn(0x7f0554064880 0x7f055411bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:00:16.324 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.312+0000 7f05477fe700 1 -- 172.21.15.42:0/259783107 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 766754642 0 0) 0x7f0554078f30 con 0x7f05540742d0 2019-12-05T06:00:16.324 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.312+0000 7f05477fe700 1 -- 172.21.15.42:0/259783107 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f053c0030b0 con 0x7f05540742d0 2019-12-05T06:00:16.324 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.312+0000 7f055a566700 1 -- 172.21.15.42:0/259783107 >> v1:172.21.15.42:6789/0 conn(0x7f05540742d0 legacy=0x7f0554076710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:00:16.324 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.312+0000 7f05477fe700 1 -- 172.21.15.42:0/259783107 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 2459809767 0 0) 0x7f053c0030b0 con 0x7f05540742d0 2019-12-05T06:00:16.324 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.312+0000 7f055a566700 1 -- 172.21.15.42:0/259783107 >> v1:172.21.15.40:6789/0 conn(0x7f055411b3c0 legacy=0x7f0554133440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:00:16.324 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.312+0000 7f055a566700 1 -- 172.21.15.42:0/259783107 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f055407a170 con 0x7f0554064880 2019-12-05T06:00:16.325 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.312+0000 7f055a566700 1 --2- 172.21.15.42:0/259783107 >> v2:172.21.15.40:3300/0 conn(0x7f0554064880 0x7f055411bb10 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f0550001690 tx=0x7f0550002f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:16.325 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.312+0000 7f055a566700 1 --2- 172.21.15.42:0/259783107 >> v2:172.21.15.40:3300/0 conn(0x7f0554064880 0x7f055411bb10 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f0550001690 tx=0x7f0550002f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:16.325 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.312+0000 7f055a566700 1 --2- 172.21.15.42:0/259783107 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0554064880 0x7f055411bb10 secure :-1 s=READY pgs=1414 cs=0 l=1 rx=0x7f0550001690 tx=0x7f0550002f70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:00:16.325 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.312+0000 7f055a566700 1 --2- 172.21.15.42:0/259783107 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0554064880 0x7f055411bb10 secure :-1 s=THROTTLE_DONE pgs=1414 cs=0 l=1 rx=0x7f0550001690 tx=0x7f0550002f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:16.325 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.312+0000 7f05477fe700 1 -- 172.21.15.42:0/259783107 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0550001490 con 0x7f0554064880 2019-12-05T06:00:16.325 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.312+0000 7f055c7ca700 1 -- 172.21.15.42:0/259783107 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0554077f20 con 0x7f0554064880 2019-12-05T06:00:16.325 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.312+0000 7f055a566700 1 --2- 172.21.15.42:0/259783107 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0554064880 0x7f055411bb10 secure :-1 s=THROTTLE_DONE pgs=1414 cs=0 l=1 rx=0x7f0550001690 tx=0x7f0550002f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:16.326 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.312+0000 7f05477fe700 1 -- 172.21.15.42:0/259783107 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05500012a0 con 0x7f0554064880 2019-12-05T06:00:16.326 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.312+0000 7f055a566700 1 --2- 172.21.15.42:0/259783107 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0554064880 0x7f055411bb10 secure :-1 s=THROTTLE_DONE pgs=1414 cs=0 l=1 rx=0x7f0550001690 tx=0x7f0550002f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:16.326 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.312+0000 7f05477fe700 1 -- 172.21.15.42:0/259783107 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0548005b20 con 0x7f0554064880 2019-12-05T06:00:16.326 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.312+0000 7f055c7ca700 1 -- 172.21.15.42:0/259783107 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0554078420 con 0x7f0554064880 2019-12-05T06:00:16.326 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.312+0000 7f055a566700 1 --2- 172.21.15.42:0/259783107 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0554064880 0x7f055411bb10 secure :-1 s=THROTTLE_DONE pgs=1414 cs=0 l=1 rx=0x7f0550001690 tx=0x7f0550002f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:16.326 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.312+0000 7f05477fe700 1 -- 172.21.15.42:0/259783107 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f0548007ae0 con 0x7f0554064880 2019-12-05T06:00:16.326 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.312+0000 7f055a566700 1 --2- 172.21.15.42:0/259783107 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0554064880 0x7f055411bb10 secure :-1 s=THROTTLE_DONE pgs=1414 cs=0 l=1 rx=0x7f0550001690 tx=0x7f0550002f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:16.327 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.312+0000 7f05477fe700 1 --2- 172.21.15.42:0/259783107 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f053c03b1b0 0x7f053c03d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:00:16.327 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.312+0000 7f05477fe700 1 -- 172.21.15.42:0/259783107 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9385+0+0 (secure 0 0 0) 0x7f054804a6c0 con 0x7f0554064880 2019-12-05T06:00:16.327 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.312+0000 7f055ad67700 1 --2- 172.21.15.42:0/259783107 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f053c03b1b0 0x7f053c03d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:00:16.327 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.316+0000 7f055ad67700 1 --2- 172.21.15.42:0/259783107 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f053c03b1b0 0x7f053c03d6c0 crc :-1 s=READY pgs=787 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:00:16.327 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.316+0000 7f055c7ca700 1 -- 172.21.15.42:0/259783107 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f055405d100 con 0x7f0554064880 2019-12-05T06:00:16.333 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.324+0000 7f055a566700 1 --2- 172.21.15.42:0/259783107 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0554064880 0x7f055411bb10 secure :-1 s=THROTTLE_DONE pgs=1414 cs=0 l=1 rx=0x7f0550001690 tx=0x7f0550002f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:16.333 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.324+0000 7f05477fe700 1 -- 172.21.15.42:0/259783107 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f0548005d70 con 0x7f0554064880 2019-12-05T06:00:16.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.524+0000 7f055c7ca700 1 -- 172.21.15.42:0/259783107 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f05540a5ff0 con 0x7f0554064880 2019-12-05T06:00:16.530 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.524+0000 7f055a566700 1 --2- 172.21.15.42:0/259783107 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0554064880 0x7f055411bb10 secure :-1 s=THROTTLE_DONE pgs=1414 cs=0 l=1 rx=0x7f0550001690 tx=0x7f0550002f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:16.530 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.524+0000 7f05477fe700 1 -- 172.21.15.42:0/259783107 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 224 v224) v1 ==== 95+0+1771 (secure 0 0 0) 0x7f0548031740 con 0x7f0554064880 2019-12-05T06:00:16.532 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 224 2019-12-05T06:00:16.532 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:00:16.533 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":224,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":224,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:00:14.305676+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":88,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9061},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9061":{"gid":9061,"name":"a","rank":0,"incarnation":224,"state":"up:replay","state_seq":1,"addr":"172.21.15.42:6835/448869597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":448869597},{"type":"v1","addr":"172.21.15.42:6835","nonce":448869597}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:00:16.535 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.528+0000 7f055c7ca700 1 -- 172.21.15.42:0/259783107 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f053c03b1b0 msgr2=0x7f053c03d6c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:00:16.535 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.528+0000 7f055c7ca700 1 --2- 172.21.15.42:0/259783107 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f053c03b1b0 0x7f053c03d6c0 crc :-1 s=READY pgs=787 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:00:16.535 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.528+0000 7f055c7ca700 1 -- 172.21.15.42:0/259783107 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0554064880 msgr2=0x7f055411bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:00:16.535 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.528+0000 7f055c7ca700 1 --2- 172.21.15.42:0/259783107 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0554064880 0x7f055411bb10 secure :-1 s=READY pgs=1414 cs=0 l=1 rx=0x7f0550001690 tx=0x7f0550002f70).stop 2019-12-05T06:00:16.535 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.528+0000 7f055c7ca700 1 -- 172.21.15.42:0/259783107 shutdown_connections 2019-12-05T06:00:16.535 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.528+0000 7f055c7ca700 1 --2- 172.21.15.42:0/259783107 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f053c03b1b0 0x7f053c03d6c0 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:00:16.536 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.528+0000 7f055c7ca700 1 --2- 172.21.15.42:0/259783107 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0554064880 0x7f055411bb10 unknown :-1 s=CLOSED pgs=1414 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:00:16.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.528+0000 7f055c7ca700 1 -- 172.21.15.42:0/259783107 shutdown_connections 2019-12-05T06:00:16.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.528+0000 7f055c7ca700 1 -- 172.21.15.42:0/259783107 wait complete. 2019-12-05T06:00:16.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.528+0000 7f055c7ca700 1 -- 172.21.15.42:0/259783107 >> 172.21.15.42:0/259783107 conn(0x7f05540a6ac0 msgr2=0x7f055411c320 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:00:16.548 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b down, removed from mdsmap 2019-12-05T06:00:16.549 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2019-12-05T06:00:16.549 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:00:16.637 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.628+0000 7ff8442db700 1 Processor -- start 2019-12-05T06:00:16.638 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.628+0000 7ff8442db700 1 -- start start 2019-12-05T06:00:16.638 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.628+0000 7ff8442db700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff83c0a7740 0x7ff83c08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:00:16.638 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.628+0000 7ff8442db700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff83c0a0710 con 0x7ff83c09fae0 2019-12-05T06:00:16.638 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.628+0000 7ff8442db700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff83c0a9ba0 con 0x7ff83c07e720 2019-12-05T06:00:16.639 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.628+0000 7ff8442db700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff83c0a0b00 con 0x7ff83c0a7740 2019-12-05T06:00:16.639 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.628+0000 7ff842077700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7ff83c07e720 0x7ff83c09f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:45920/0 (socket says 172.21.15.42:45920) 2019-12-05T06:00:16.639 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.628+0000 7ff842077700 1 -- 172.21.15.42:0/757219124 learned_addr learned my addr 172.21.15.42:0/757219124 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:00:16.639 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.628+0000 7ff842878700 1 --2- 172.21.15.42:0/757219124 >> v2:172.21.15.40:3300/0 conn(0x7ff83c0a7740 0x7ff83c08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:00:16.639 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.628+0000 7ff841876700 1 -- 172.21.15.42:0/757219124 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1588439868 0 0) 0x7ff83c0a9ba0 con 0x7ff83c07e720 2019-12-05T06:00:16.640 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.628+0000 7ff841876700 1 -- 172.21.15.42:0/757219124 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8240030b0 con 0x7ff83c07e720 2019-12-05T06:00:16.640 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.628+0000 7ff841876700 1 -- 172.21.15.42:0/757219124 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 860066892 0 0) 0x7ff83c0a0710 con 0x7ff83c09fae0 2019-12-05T06:00:16.640 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.628+0000 7ff841876700 1 -- 172.21.15.42:0/757219124 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff83c0a9ba0 con 0x7ff83c09fae0 2019-12-05T06:00:16.640 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.628+0000 7ff841876700 1 -- 172.21.15.42:0/757219124 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3679542081 0 0) 0x7ff8240030b0 con 0x7ff83c07e720 2019-12-05T06:00:16.640 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.628+0000 7ff841876700 1 -- 172.21.15.42:0/757219124 >> v2:172.21.15.40:3300/0 conn(0x7ff83c0a7740 msgr2=0x7ff83c08df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:00:16.641 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.628+0000 7ff841876700 1 --2- 172.21.15.42:0/757219124 >> v2:172.21.15.40:3300/0 conn(0x7ff83c0a7740 0x7ff83c08df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:00:16.641 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.628+0000 7ff841876700 1 -- 172.21.15.42:0/757219124 >> v1:172.21.15.40:6789/0 conn(0x7ff83c09fae0 legacy=0x7ff83c14efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:00:16.641 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.628+0000 7ff841876700 1 -- 172.21.15.42:0/757219124 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff83c07eb50 con 0x7ff83c07e720 2019-12-05T06:00:16.641 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.628+0000 7ff841876700 1 -- 172.21.15.42:0/757219124 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7ff830001ad0 con 0x7ff83c07e720 2019-12-05T06:00:16.642 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.628+0000 7ff841876700 1 -- 172.21.15.42:0/757219124 >> v1:172.21.15.42:6789/0 conn(0x7ff83c07e720 legacy=0x7ff83c09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:00:16.642 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.628+0000 7ff841876700 1 --2- 172.21.15.42:0/757219124 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff824003910 0x7ff824003d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:00:16.642 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.632+0000 7ff841876700 1 --2- 172.21.15.42:0/757219124 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff824004420 0x7ff8240048f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:00:16.642 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.632+0000 7ff842878700 1 --2- 172.21.15.42:0/757219124 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff824003910 0x7ff824003d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:00:16.642 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.632+0000 7ff841876700 1 --2- 172.21.15.42:0/757219124 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff824004fa0 0x7ff82404f700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:00:16.642 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.632+0000 7ff841876700 1 -- 172.21.15.42:0/757219124 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff83c0a0b00 con 0x7ff824004420 2019-12-05T06:00:16.642 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.632+0000 7ff841876700 1 -- 172.21.15.42:0/757219124 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff82404fec0 con 0x7ff824003910 2019-12-05T06:00:16.643 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.632+0000 7ff841876700 1 -- 172.21.15.42:0/757219124 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff824050100 con 0x7ff824004fa0 2019-12-05T06:00:16.643 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.632+0000 7ff842878700 1 -- 172.21.15.42:0/757219124 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff824004fa0 msgr2=0x7ff82404f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:00:16.643 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.632+0000 7ff842077700 1 --2- 172.21.15.42:0/757219124 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff824004420 0x7ff8240048f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:00:16.643 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.632+0000 7ff843079700 1 --2- 172.21.15.42:0/757219124 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff824004fa0 0x7ff82404f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:00:16.643 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.632+0000 7ff842878700 1 --2- 172.21.15.42:0/757219124 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff824004fa0 0x7ff82404f700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:00:16.643 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.632+0000 7ff842878700 1 -- 172.21.15.42:0/757219124 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff824004420 msgr2=0x7ff8240048f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:00:16.643 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.632+0000 7ff842878700 1 --2- 172.21.15.42:0/757219124 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff824004420 0x7ff8240048f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:00:16.644 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.632+0000 7ff842878700 1 -- 172.21.15.42:0/757219124 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8240503a0 con 0x7ff824003910 2019-12-05T06:00:16.644 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.632+0000 7ff843079700 1 -- 172.21.15.42:0/757219124 reap_dead start 2019-12-05T06:00:16.644 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.632+0000 7ff842878700 1 --2- 172.21.15.42:0/757219124 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff824003910 0x7ff824003d60 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7ff834002f30 tx=0x7ff83400ffe0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:16.644 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.632+0000 7ff842878700 1 --2- 172.21.15.42:0/757219124 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff824003910 0x7ff824003d60 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7ff834002f30 tx=0x7ff83400ffe0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:16.644 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.632+0000 7ff842878700 1 --2- 172.21.15.42:0/757219124 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff824003910 0x7ff824003d60 secure :-1 s=READY pgs=1845 cs=0 l=1 rx=0x7ff834002f30 tx=0x7ff83400ffe0).ready entity=mon.1 client_cookie=dfa4f4ab0ee5f1f8 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:00:16.644 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.632+0000 7ff842878700 1 --2- 172.21.15.42:0/757219124 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff824003910 0x7ff824003d60 secure :-1 s=THROTTLE_DONE pgs=1845 cs=0 l=1 rx=0x7ff834002f30 tx=0x7ff83400ffe0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:16.645 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.632+0000 7ff842878700 1 --2- 172.21.15.42:0/757219124 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff824003910 0x7ff824003d60 secure :-1 s=THROTTLE_DONE pgs=1845 cs=0 l=1 rx=0x7ff834002f30 tx=0x7ff83400ffe0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:16.645 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.632+0000 7ff841876700 1 -- 172.21.15.42:0/757219124 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff834010d10 con 0x7ff824003910 2019-12-05T06:00:16.645 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.632+0000 7ff842878700 1 --2- 172.21.15.42:0/757219124 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff824003910 0x7ff824003d60 secure :-1 s=THROTTLE_DONE pgs=1845 cs=0 l=1 rx=0x7ff834002f30 tx=0x7ff83400ffe0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:16.645 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.632+0000 7ff841876700 1 -- 172.21.15.42:0/757219124 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff834010930 con 0x7ff824003910 2019-12-05T06:00:16.645 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.632+0000 7ff841876700 1 -- 172.21.15.42:0/757219124 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff834008bd0 con 0x7ff824003910 2019-12-05T06:00:16.645 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.632+0000 7ff8442db700 1 -- 172.21.15.42:0/757219124 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff824003910 msgr2=0x7ff824003d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:00:16.646 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.632+0000 7ff8442db700 1 --2- 172.21.15.42:0/757219124 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff824003910 0x7ff824003d60 secure :-1 s=READY pgs=1845 cs=0 l=1 rx=0x7ff834002f30 tx=0x7ff83400ffe0).stop 2019-12-05T06:00:16.646 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.632+0000 7ff8442db700 1 -- 172.21.15.42:0/757219124 shutdown_connections 2019-12-05T06:00:16.646 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.632+0000 7ff8442db700 1 --2- 172.21.15.42:0/757219124 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff824003910 0x7ff824003d60 unknown :-1 s=CLOSED pgs=1845 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:00:16.646 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.632+0000 7ff8442db700 1 -- 172.21.15.42:0/757219124 shutdown_connections 2019-12-05T06:00:16.646 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.632+0000 7ff8442db700 1 -- 172.21.15.42:0/757219124 wait complete. 2019-12-05T06:00:16.646 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.632+0000 7ff8442db700 1 -- 172.21.15.42:0/757219124 >> 172.21.15.42:0/757219124 conn(0x7ff83c0aabe0 msgr2=0x7ff83c0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:00:16.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.632+0000 7ff8442db700 1 Processor -- start 2019-12-05T06:00:16.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.632+0000 7ff8442db700 1 -- start start 2019-12-05T06:00:16.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.636+0000 7ff8442db700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff83c117700 0x7ff83c11e550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:00:16.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.636+0000 7ff8442db700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff83c120430 con 0x7ff83c120000 2019-12-05T06:00:16.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.636+0000 7ff8442db700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff83c12dce0 con 0x7ff83c09f620 2019-12-05T06:00:16.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.636+0000 7ff8442db700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff83c11f860 con 0x7ff83c117700 2019-12-05T06:00:16.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.636+0000 7ff842878700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7ff83c09f620 0x7ff83c116f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:45930/0 (socket says 172.21.15.42:45930) 2019-12-05T06:00:16.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.636+0000 7ff842878700 1 -- 172.21.15.42:0/3545603258 learned_addr learned my addr 172.21.15.42:0/3545603258 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:00:16.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.636+0000 7ff83b7fe700 1 -- 172.21.15.42:0/3545603258 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2778414702 0 0) 0x7ff83c12dce0 con 0x7ff83c09f620 2019-12-05T06:00:16.654 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.636+0000 7ff842077700 1 --2- 172.21.15.42:0/3545603258 >> v2:172.21.15.40:3300/0 conn(0x7ff83c117700 0x7ff83c11e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:00:16.654 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.636+0000 7ff83b7fe700 1 -- 172.21.15.42:0/3545603258 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff81c0030b0 con 0x7ff83c09f620 2019-12-05T06:00:16.654 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.636+0000 7ff83b7fe700 1 -- 172.21.15.42:0/3545603258 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 2198016799 0 0) 0x7ff81c0030b0 con 0x7ff83c09f620 2019-12-05T06:00:16.654 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.636+0000 7ff83b7fe700 1 -- 172.21.15.42:0/3545603258 --> v1:172.21.15.42:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff83c12dce0 con 0x7ff83c09f620 2019-12-05T06:00:16.654 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.636+0000 7ff83b7fe700 1 -- 172.21.15.42:0/3545603258 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7ff834008df0 con 0x7ff83c09f620 2019-12-05T06:00:16.654 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.636+0000 7ff83b7fe700 1 -- 172.21.15.42:0/3545603258 >> v2:172.21.15.40:3300/0 conn(0x7ff83c117700 msgr2=0x7ff83c11e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:00:16.654 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.636+0000 7ff83b7fe700 1 --2- 172.21.15.42:0/3545603258 >> v2:172.21.15.40:3300/0 conn(0x7ff83c117700 0x7ff83c11e550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:00:16.655 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.636+0000 7ff83b7fe700 1 -- 172.21.15.42:0/3545603258 >> v1:172.21.15.42:6789/0 conn(0x7ff83c09f620 legacy=0x7ff83c116f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:00:16.655 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.636+0000 7ff83b7fe700 1 -- 172.21.15.42:0/3545603258 >> v1:172.21.15.40:6789/0 conn(0x7ff83c120000 legacy=0x7ff83c12d4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:00:16.655 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.636+0000 7ff83b7fe700 1 --2- 172.21.15.42:0/3545603258 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff81c0038a0 0x7ff81c003d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:00:16.655 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.636+0000 7ff83b7fe700 1 --2- 172.21.15.42:0/3545603258 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff81c004420 0x7ff81c004870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:00:16.655 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.636+0000 7ff83b7fe700 1 --2- 172.21.15.42:0/3545603258 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff81c004ed0 0x7ff81c00d880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:00:16.655 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.636+0000 7ff83b7fe700 1 -- 172.21.15.42:0/3545603258 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff83c11f860 con 0x7ff81c0038a0 2019-12-05T06:00:16.655 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.636+0000 7ff83b7fe700 1 -- 172.21.15.42:0/3545603258 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff81c009690 con 0x7ff81c004ed0 2019-12-05T06:00:16.655 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.636+0000 7ff83b7fe700 1 -- 172.21.15.42:0/3545603258 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff81c0098d0 con 0x7ff81c004420 2019-12-05T06:00:16.656 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.636+0000 7ff843079700 1 --2- 172.21.15.42:0/3545603258 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff81c004ed0 0x7ff81c00d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:00:16.656 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.636+0000 7ff842077700 1 --2- 172.21.15.42:0/3545603258 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff81c004420 0x7ff81c004870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:00:16.656 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.636+0000 7ff842878700 1 --2- 172.21.15.42:0/3545603258 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff81c0038a0 0x7ff81c003d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:00:16.656 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.636+0000 7ff843079700 1 -- 172.21.15.42:0/3545603258 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff81c004420 msgr2=0x7ff81c004870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:00:16.656 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.636+0000 7ff843079700 1 --2- 172.21.15.42:0/3545603258 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff81c004420 0x7ff81c004870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:00:16.656 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.636+0000 7ff843079700 1 -- 172.21.15.42:0/3545603258 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff81c0038a0 msgr2=0x7ff81c003d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:00:16.657 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.636+0000 7ff843079700 1 --2- 172.21.15.42:0/3545603258 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff81c0038a0 0x7ff81c003d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:00:16.657 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.636+0000 7ff843079700 1 -- 172.21.15.42:0/3545603258 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff83c12df10 con 0x7ff81c004ed0 2019-12-05T06:00:16.657 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.636+0000 7ff8442db700 1 -- 172.21.15.42:0/3545603258 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff83c12e180 con 0x7ff81c004ed0 2019-12-05T06:00:16.657 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.636+0000 7ff8442db700 1 -- 172.21.15.42:0/3545603258 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff83c118730 con 0x7ff81c004ed0 2019-12-05T06:00:16.657 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.636+0000 7ff843079700 1 --2- 172.21.15.42:0/3545603258 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff81c004ed0 0x7ff81c00d880 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7ff82c0011d0 tx=0x7ff82c010760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:16.658 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.636+0000 7ff843079700 1 -- 172.21.15.42:0/3545603258 reap_dead start 2019-12-05T06:00:16.658 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.636+0000 7ff843079700 1 --2- 172.21.15.42:0/3545603258 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff81c004ed0 0x7ff81c00d880 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7ff82c0011d0 tx=0x7ff82c010760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:16.658 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.636+0000 7ff843079700 1 --2- 172.21.15.42:0/3545603258 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff81c004ed0 0x7ff81c00d880 secure :-1 s=READY pgs=1847 cs=0 l=1 rx=0x7ff82c0011d0 tx=0x7ff82c010760).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:00:16.658 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.636+0000 7ff843079700 1 --2- 172.21.15.42:0/3545603258 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff81c004ed0 0x7ff81c00d880 secure :-1 s=THROTTLE_DONE pgs=1847 cs=0 l=1 rx=0x7ff82c0011d0 tx=0x7ff82c010760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:16.658 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.636+0000 7ff83b7fe700 1 -- 172.21.15.42:0/3545603258 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff82c006b30 con 0x7ff81c004ed0 2019-12-05T06:00:16.658 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.636+0000 7ff843079700 1 --2- 172.21.15.42:0/3545603258 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff81c004ed0 0x7ff81c00d880 secure :-1 s=THROTTLE_DONE pgs=1847 cs=0 l=1 rx=0x7ff82c0011d0 tx=0x7ff82c010760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:16.659 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.636+0000 7ff83b7fe700 1 -- 172.21.15.42:0/3545603258 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff82c006750 con 0x7ff81c004ed0 2019-12-05T06:00:16.659 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.636+0000 7ff843079700 1 --2- 172.21.15.42:0/3545603258 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff81c004ed0 0x7ff81c00d880 secure :-1 s=THROTTLE_DONE pgs=1847 cs=0 l=1 rx=0x7ff82c0011d0 tx=0x7ff82c010760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:16.659 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.636+0000 7ff83b7fe700 1 -- 172.21.15.42:0/3545603258 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff82c007640 con 0x7ff81c004ed0 2019-12-05T06:00:16.659 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.636+0000 7ff843079700 1 --2- 172.21.15.42:0/3545603258 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff81c004ed0 0x7ff81c00d880 secure :-1 s=THROTTLE_DONE pgs=1847 cs=0 l=1 rx=0x7ff82c0011d0 tx=0x7ff82c010760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:16.660 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.636+0000 7ff83b7fe700 1 -- 172.21.15.42:0/3545603258 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7ff82c007840 con 0x7ff81c004ed0 2019-12-05T06:00:16.660 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.636+0000 7ff843079700 1 --2- 172.21.15.42:0/3545603258 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff81c004ed0 0x7ff81c00d880 secure :-1 s=THROTTLE_DONE pgs=1847 cs=0 l=1 rx=0x7ff82c0011d0 tx=0x7ff82c010760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:16.660 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.640+0000 7ff83b7fe700 1 --2- 172.21.15.42:0/3545603258 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff81c042e50 0x7ff81c045360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:00:16.660 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.640+0000 7ff83b7fe700 1 -- 172.21.15.42:0/3545603258 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9385+0+0 (secure 0 0 0) 0x7ff82c04ac70 con 0x7ff81c004ed0 2019-12-05T06:00:16.660 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.640+0000 7ff842878700 1 --2- 172.21.15.42:0/3545603258 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff81c042e50 0x7ff81c045360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:00:16.660 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.640+0000 7ff842878700 1 --2- 172.21.15.42:0/3545603258 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff81c042e50 0x7ff81c045360 crc :-1 s=READY pgs=788 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:00:16.660 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.644+0000 7ff8442db700 1 -- 172.21.15.42:0/3545603258 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff818003200 con 0x7ff81c004ed0 2019-12-05T06:00:16.661 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.648+0000 7ff843079700 1 --2- 172.21.15.42:0/3545603258 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff81c004ed0 0x7ff81c00d880 secure :-1 s=THROTTLE_DONE pgs=1847 cs=0 l=1 rx=0x7ff82c0011d0 tx=0x7ff82c010760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:16.661 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.648+0000 7ff83b7fe700 1 -- 172.21.15.42:0/3545603258 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7ff82c031cf0 con 0x7ff81c004ed0 2019-12-05T06:00:16.813 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.808+0000 7ff8442db700 1 -- 172.21.15.42:0/3545603258 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff818003630 con 0x7ff81c004ed0 2019-12-05T06:00:16.814 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.808+0000 7ff843079700 1 --2- 172.21.15.42:0/3545603258 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff81c004ed0 0x7ff81c00d880 secure :-1 s=THROTTLE_DONE pgs=1847 cs=0 l=1 rx=0x7ff82c0011d0 tx=0x7ff82c010760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:16.814 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.808+0000 7ff83b7fe700 1 -- 172.21.15.42:0/3545603258 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 224 v224) v1 ==== 95+0+1771 (secure 0 0 0) 0x7ff82c031f50 con 0x7ff81c004ed0 2019-12-05T06:00:16.815 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 224 2019-12-05T06:00:16.815 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:00:16.815 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":224,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":224,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:00:14.305676+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":88,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9061},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9061":{"gid":9061,"name":"a","rank":0,"incarnation":224,"state":"up:replay","state_seq":1,"addr":"172.21.15.42:6835/448869597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":448869597},{"type":"v1","addr":"172.21.15.42:6835","nonce":448869597}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:00:16.818 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.812+0000 7ff8442db700 1 -- 172.21.15.42:0/3545603258 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff81c042e50 msgr2=0x7ff81c045360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:00:16.818 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.812+0000 7ff8442db700 1 --2- 172.21.15.42:0/3545603258 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff81c042e50 0x7ff81c045360 crc :-1 s=READY pgs=788 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:00:16.818 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.812+0000 7ff8442db700 1 -- 172.21.15.42:0/3545603258 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff81c004ed0 msgr2=0x7ff81c00d880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:00:16.818 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.812+0000 7ff8442db700 1 --2- 172.21.15.42:0/3545603258 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff81c004ed0 0x7ff81c00d880 secure :-1 s=READY pgs=1847 cs=0 l=1 rx=0x7ff82c0011d0 tx=0x7ff82c010760).stop 2019-12-05T06:00:16.819 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.812+0000 7ff8442db700 1 -- 172.21.15.42:0/3545603258 shutdown_connections 2019-12-05T06:00:16.820 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.812+0000 7ff8442db700 1 --2- 172.21.15.42:0/3545603258 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff81c042e50 0x7ff81c045360 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:00:16.820 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.812+0000 7ff8442db700 1 --2- 172.21.15.42:0/3545603258 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff81c004ed0 0x7ff81c00d880 unknown :-1 s=CLOSED pgs=1847 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:00:16.820 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.812+0000 7ff8442db700 1 -- 172.21.15.42:0/3545603258 shutdown_connections 2019-12-05T06:00:16.820 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.812+0000 7ff8442db700 1 -- 172.21.15.42:0/3545603258 wait complete. 2019-12-05T06:00:16.820 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:16.812+0000 7ff8442db700 1 -- 172.21.15.42:0/3545603258 >> 172.21.15.42:0/3545603258 conn(0x7ff83c07e650 msgr2=0x7ff83c059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:00:16.832 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 224, "feature_flags": { "enable_multiple": false, "ever_enabled_multiple": false }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2019-12-05T05:03:35.604738+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 224, "ever_allowed_features": 2, "explicitly_allowed_features": 2, "failed": [], "flags": 18, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_9061": { "addr": "172.21.15.42:6835/448869597", "addrs": { "addrvec": [ { "addr": "172.21.15.42:6834", "nonce": 448869597, "type": "v2" }, { "addr": "172.21.15.42:6835", "nonce": 448869597, "type": "v1" } ] }, "export_targets": [], "features": 4540138292836696063, "flags": 0, "gid": 9061, "incarnation": 224, "name": "a", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 88, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 2, "min_compat_client": "0 (unknown)", "modified": "2019-12-05T06:00:14.305676+0000", "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 9061 } } } ], "standbys": [] } 2019-12-05T06:00:18.834 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:00:18.943 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:18.932+0000 7f5bf6ad0700 1 Processor -- start 2019-12-05T06:00:18.944 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:18.932+0000 7f5bf6ad0700 1 -- start start 2019-12-05T06:00:18.944 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:18.936+0000 7f5bf6ad0700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5bf00a7740 0x7f5bf008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:00:18.944 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:18.936+0000 7f5bf6ad0700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5bf009fae0 0x7f5bf007eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:00:18.945 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:18.936+0000 7f5bf6ad0700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bf00a0710 con 0x7f5bf007e720 2019-12-05T06:00:18.946 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:18.936+0000 7f5bf6ad0700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5bf00a0b00 con 0x7f5bf00a7740 2019-12-05T06:00:18.946 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:18.936+0000 7f5bf6ad0700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bf008e5b0 con 0x7f5bf009fae0 2019-12-05T06:00:18.946 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:18.936+0000 7f5bf506d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5bf00a7740 0x7f5bf008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:00:18.947 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:18.936+0000 7f5bf506d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5bf00a7740 0x7f5bf008df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:45912/0 (socket says 172.21.15.42:45912) 2019-12-05T06:00:18.947 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:18.936+0000 7f5bf506d700 1 -- 172.21.15.42:0/264114843 learned_addr learned my addr 172.21.15.42:0/264114843 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:00:18.947 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:18.936+0000 7f5bf586e700 1 --2- 172.21.15.42:0/264114843 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5bf009fae0 0x7f5bf007eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:00:18.947 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:18.936+0000 7f5bf506d700 1 -- 172.21.15.42:0/264114843 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5bf009fae0 msgr2=0x7f5bf007eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:00:18.948 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:18.936+0000 7f5bf506d700 1 --2- 172.21.15.42:0/264114843 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5bf009fae0 0x7f5bf007eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:00:18.949 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:18.936+0000 7f5bf506d700 1 -- 172.21.15.42:0/264114843 >> v1:172.21.15.40:6789/0 conn(0x7f5bf007e720 legacy=0x7f5bf009f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:00:18.949 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:18.936+0000 7f5be3fff700 1 -- 172.21.15.42:0/264114843 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1775978121 0 0) 0x7f5bf00a0710 con 0x7f5bf007e720 2019-12-05T06:00:18.949 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:18.936+0000 7f5bf506d700 1 -- 172.21.15.42:0/264114843 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bf007f0a0 con 0x7f5bf00a7740 2019-12-05T06:00:18.950 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:18.936+0000 7f5bf506d700 1 --2- 172.21.15.42:0/264114843 >> v2:172.21.15.42:3300/0 conn(0x7f5bf00a7740 0x7f5bf008df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f5bec008790 tx=0x7f5bec00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:18.950 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:18.936+0000 7f5bf506d700 1 --2- 172.21.15.42:0/264114843 >> v2:172.21.15.42:3300/0 conn(0x7f5bf00a7740 0x7f5bf008df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f5bec008790 tx=0x7f5bec00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:18.950 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:18.936+0000 7f5bf506d700 1 --2- 172.21.15.42:0/264114843 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5bf00a7740 0x7f5bf008df80 secure :-1 s=READY pgs=1848 cs=0 l=1 rx=0x7f5bec008790 tx=0x7f5bec00fd30).ready entity=mon.1 client_cookie=74311936b61d83d server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:00:18.950 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:18.936+0000 7f5bf506d700 1 --2- 172.21.15.42:0/264114843 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5bf00a7740 0x7f5bf008df80 secure :-1 s=THROTTLE_DONE pgs=1848 cs=0 l=1 rx=0x7f5bec008790 tx=0x7f5bec00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:18.950 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:18.936+0000 7f5be3fff700 1 -- 172.21.15.42:0/264114843 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5bec008c90 con 0x7f5bf00a7740 2019-12-05T06:00:18.951 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:18.936+0000 7f5bf506d700 1 --2- 172.21.15.42:0/264114843 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5bf00a7740 0x7f5bf008df80 secure :-1 s=THROTTLE_DONE pgs=1848 cs=0 l=1 rx=0x7f5bec008790 tx=0x7f5bec00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:18.951 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:18.936+0000 7f5bf506d700 1 --2- 172.21.15.42:0/264114843 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5bf00a7740 0x7f5bf008df80 secure :-1 s=THROTTLE_DONE pgs=1848 cs=0 l=1 rx=0x7f5bec008790 tx=0x7f5bec00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:18.951 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:18.936+0000 7f5be3fff700 1 -- 172.21.15.42:0/264114843 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bec0088b0 con 0x7f5bf00a7740 2019-12-05T06:00:18.951 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:18.936+0000 7f5be3fff700 1 -- 172.21.15.42:0/264114843 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5bec011b10 con 0x7f5bf00a7740 2019-12-05T06:00:18.952 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:18.936+0000 7f5bf6ad0700 1 -- 172.21.15.42:0/264114843 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5bf00a7740 msgr2=0x7f5bf008df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:00:18.952 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:18.936+0000 7f5bf6ad0700 1 --2- 172.21.15.42:0/264114843 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5bf00a7740 0x7f5bf008df80 secure :-1 s=READY pgs=1848 cs=0 l=1 rx=0x7f5bec008790 tx=0x7f5bec00fd30).stop 2019-12-05T06:00:18.952 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:18.936+0000 7f5bf6ad0700 1 -- 172.21.15.42:0/264114843 shutdown_connections 2019-12-05T06:00:18.952 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:18.936+0000 7f5bf6ad0700 1 --2- 172.21.15.42:0/264114843 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5bf009fae0 0x7f5bf007eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:00:18.953 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:18.936+0000 7f5bf6ad0700 1 --2- 172.21.15.42:0/264114843 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5bf00a7740 0x7f5bf008df80 unknown :-1 s=CLOSED pgs=1848 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:00:18.953 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:18.936+0000 7f5bf6ad0700 1 -- 172.21.15.42:0/264114843 shutdown_connections 2019-12-05T06:00:18.953 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:18.936+0000 7f5bf6ad0700 1 -- 172.21.15.42:0/264114843 wait complete. 2019-12-05T06:00:18.953 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:18.936+0000 7f5bf6ad0700 1 -- 172.21.15.42:0/264114843 >> 172.21.15.42:0/264114843 conn(0x7f5bf00aabe0 msgr2=0x7f5bf00a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:00:18.960 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:18.940+0000 7f5bf6ad0700 1 Processor -- start 2019-12-05T06:00:18.960 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:18.940+0000 7f5bf6ad0700 1 -- start start 2019-12-05T06:00:18.960 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:18.940+0000 7f5bf6ad0700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5bf01190c0 0x7f5bf011d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:00:18.961 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:18.940+0000 7f5bf6ad0700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5bf0132720 0x7f5bf0132b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:00:18.961 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:18.940+0000 7f5bf6ad0700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bf011daf0 con 0x7f5bf0116570 2019-12-05T06:00:18.961 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:18.940+0000 7f5bf6ad0700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5bf01198e0 con 0x7f5bf01190c0 2019-12-05T06:00:18.961 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:18.940+0000 7f5bf6ad0700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5bf01162d0 con 0x7f5bf0132720 2019-12-05T06:00:18.962 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:18.940+0000 7f5bf586e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5bf0132720 0x7f5bf0132b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:00:18.962 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:18.940+0000 7f5bf586e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5bf0132720 0x7f5bf0132b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:45916/0 (socket says 172.21.15.42:45916) 2019-12-05T06:00:18.962 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:18.940+0000 7f5bf586e700 1 -- 172.21.15.42:0/3189984740 learned_addr learned my addr 172.21.15.42:0/3189984740 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:00:18.962 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:18.940+0000 7f5bf506d700 1 --2- 172.21.15.42:0/3189984740 >> v2:172.21.15.40:3300/0 conn(0x7f5bf01190c0 0x7f5bf011d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:00:18.963 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:18.940+0000 7f5bf586e700 1 -- 172.21.15.42:0/3189984740 >> v2:172.21.15.40:3300/0 conn(0x7f5bf01190c0 msgr2=0x7f5bf011d5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:00:18.963 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:18.940+0000 7f5bf586e700 1 --2- 172.21.15.42:0/3189984740 >> v2:172.21.15.40:3300/0 conn(0x7f5bf01190c0 0x7f5bf011d5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:00:18.963 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:18.940+0000 7f5bf586e700 1 -- 172.21.15.42:0/3189984740 >> v1:172.21.15.40:6789/0 conn(0x7f5bf0116570 legacy=0x7f5bf01169a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:00:18.963 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:18.940+0000 7f5bf586e700 1 -- 172.21.15.42:0/3189984740 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bf0118020 con 0x7f5bf0132720 2019-12-05T06:00:18.963 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:18.940+0000 7f5bf586e700 1 --2- 172.21.15.42:0/3189984740 >> v2:172.21.15.42:3300/0 conn(0x7f5bf0132720 0x7f5bf0132b50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f5be80032b0 tx=0x7f5be8010e40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:18.964 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:18.940+0000 7f5bf586e700 1 --2- 172.21.15.42:0/3189984740 >> v2:172.21.15.42:3300/0 conn(0x7f5bf0132720 0x7f5bf0132b50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f5be80032b0 tx=0x7f5be8010e40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:18.964 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:18.944+0000 7f5bf586e700 1 --2- 172.21.15.42:0/3189984740 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5bf0132720 0x7f5bf0132b50 secure :-1 s=READY pgs=1849 cs=0 l=1 rx=0x7f5be80032b0 tx=0x7f5be8010e40).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:00:18.964 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:18.944+0000 7f5bf586e700 1 --2- 172.21.15.42:0/3189984740 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5bf0132720 0x7f5bf0132b50 secure :-1 s=THROTTLE_DONE pgs=1849 cs=0 l=1 rx=0x7f5be80032b0 tx=0x7f5be8010e40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:18.964 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:18.944+0000 7f5bf586e700 1 --2- 172.21.15.42:0/3189984740 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5bf0132720 0x7f5bf0132b50 secure :-1 s=THROTTLE_DONE pgs=1849 cs=0 l=1 rx=0x7f5be80032b0 tx=0x7f5be8010e40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:18.965 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:18.944+0000 7f5bf586e700 1 --2- 172.21.15.42:0/3189984740 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5bf0132720 0x7f5bf0132b50 secure :-1 s=THROTTLE_DONE pgs=1849 cs=0 l=1 rx=0x7f5be80032b0 tx=0x7f5be8010e40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:18.965 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:18.944+0000 7f5be1ffb700 1 -- 172.21.15.42:0/3189984740 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5be8012ae0 con 0x7f5bf0132720 2019-12-05T06:00:18.965 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:18.944+0000 7f5be1ffb700 1 -- 172.21.15.42:0/3189984740 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5be800e160 con 0x7f5bf0132720 2019-12-05T06:00:18.965 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:18.944+0000 7f5bf6ad0700 1 -- 172.21.15.42:0/3189984740 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bf01194f0 con 0x7f5bf0132720 2019-12-05T06:00:18.965 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:18.944+0000 7f5be1ffb700 1 -- 172.21.15.42:0/3189984740 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5be8005ab0 con 0x7f5bf0132720 2019-12-05T06:00:18.966 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:18.944+0000 7f5bf6ad0700 1 -- 172.21.15.42:0/3189984740 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bf0133240 con 0x7f5bf0132720 2019-12-05T06:00:18.966 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:18.944+0000 7f5bf586e700 1 --2- 172.21.15.42:0/3189984740 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5bf0132720 0x7f5bf0132b50 secure :-1 s=THROTTLE_DONE pgs=1849 cs=0 l=1 rx=0x7f5be80032b0 tx=0x7f5be8010e40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:18.966 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:18.944+0000 7f5be1ffb700 1 -- 172.21.15.42:0/3189984740 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f5be8007180 con 0x7f5bf0132720 2019-12-05T06:00:18.966 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:18.944+0000 7f5bf586e700 1 --2- 172.21.15.42:0/3189984740 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5bf0132720 0x7f5bf0132b50 secure :-1 s=THROTTLE_DONE pgs=1849 cs=0 l=1 rx=0x7f5be80032b0 tx=0x7f5be8010e40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:18.967 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:18.944+0000 7f5be1ffb700 1 --2- 172.21.15.42:0/3189984740 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5bd003ae10 0x7f5bd003d320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:00:18.967 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:18.944+0000 7f5be1ffb700 1 -- 172.21.15.42:0/3189984740 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9385+0+0 (secure 0 0 0) 0x7f5be804afa0 con 0x7f5bf0132720 2019-12-05T06:00:18.967 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:18.944+0000 7f5bf506d700 1 --2- 172.21.15.42:0/3189984740 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5bd003ae10 0x7f5bd003d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:00:18.967 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:18.944+0000 7f5bf506d700 1 --2- 172.21.15.42:0/3189984740 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5bd003ae10 0x7f5bd003d320 crc :-1 s=READY pgs=789 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:00:18.967 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:18.948+0000 7f5bf6ad0700 1 -- 172.21.15.42:0/3189984740 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5bf005d110 con 0x7f5bf0132720 2019-12-05T06:00:18.968 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:18.956+0000 7f5bf586e700 1 --2- 172.21.15.42:0/3189984740 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5bf0132720 0x7f5bf0132b50 secure :-1 s=THROTTLE_DONE pgs=1849 cs=0 l=1 rx=0x7f5be80032b0 tx=0x7f5be8010e40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:18.968 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:18.956+0000 7f5be1ffb700 1 -- 172.21.15.42:0/3189984740 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f5be8032020 con 0x7f5bf0132720 2019-12-05T06:00:19.138 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:19.132+0000 7f5bf6ad0700 1 -- 172.21.15.42:0/3189984740 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5bf005b8c0 con 0x7f5bf0132720 2019-12-05T06:00:19.140 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:19.132+0000 7f5bf586e700 1 --2- 172.21.15.42:0/3189984740 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5bf0132720 0x7f5bf0132b50 secure :-1 s=THROTTLE_DONE pgs=1849 cs=0 l=1 rx=0x7f5be80032b0 tx=0x7f5be8010e40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:19.140 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:19.132+0000 7f5be1ffb700 1 -- 172.21.15.42:0/3189984740 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 226 v226) v1 ==== 95+0+1771 (secure 0 0 0) 0x7f5be8032280 con 0x7f5bf0132720 2019-12-05T06:00:19.140 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 226 2019-12-05T06:00:19.140 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:00:19.141 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":226,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":226,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:00:18.594336+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":88,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9061},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9061":{"gid":9061,"name":"a","rank":0,"incarnation":224,"state":"up:rejoin","state_seq":8,"addr":"172.21.15.42:6835/448869597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":448869597},{"type":"v1","addr":"172.21.15.42:6835","nonce":448869597}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:00:19.142 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:19.136+0000 7f5bf6ad0700 1 -- 172.21.15.42:0/3189984740 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5bd003ae10 msgr2=0x7f5bd003d320 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:00:19.143 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:19.136+0000 7f5bf6ad0700 1 --2- 172.21.15.42:0/3189984740 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5bd003ae10 0x7f5bd003d320 crc :-1 s=READY pgs=789 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:00:19.143 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:19.136+0000 7f5bf6ad0700 1 -- 172.21.15.42:0/3189984740 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5bf0132720 msgr2=0x7f5bf0132b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:00:19.143 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:19.136+0000 7f5bf6ad0700 1 --2- 172.21.15.42:0/3189984740 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5bf0132720 0x7f5bf0132b50 secure :-1 s=READY pgs=1849 cs=0 l=1 rx=0x7f5be80032b0 tx=0x7f5be8010e40).stop 2019-12-05T06:00:19.143 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:19.136+0000 7f5bf6ad0700 1 -- 172.21.15.42:0/3189984740 shutdown_connections 2019-12-05T06:00:19.143 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:19.136+0000 7f5bf6ad0700 1 --2- 172.21.15.42:0/3189984740 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5bd003ae10 0x7f5bd003d320 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:00:19.144 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:19.136+0000 7f5bf6ad0700 1 --2- 172.21.15.42:0/3189984740 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5bf0132720 0x7f5bf0132b50 unknown :-1 s=CLOSED pgs=1849 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:00:19.144 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:19.136+0000 7f5bf6ad0700 1 --2- 172.21.15.42:0/3189984740 >> v2:172.21.15.40:3300/0 conn(0x7f5bf01190c0 0x7f5bf011d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:00:19.144 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:19.136+0000 7f5bf6ad0700 1 -- 172.21.15.42:0/3189984740 shutdown_connections 2019-12-05T06:00:19.144 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:19.136+0000 7f5bf6ad0700 1 -- 172.21.15.42:0/3189984740 wait complete. 2019-12-05T06:00:19.144 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:19.136+0000 7f5bf6ad0700 1 -- 172.21.15.42:0/3189984740 >> 172.21.15.42:0/3189984740 conn(0x7f5bf00aabe0 msgr2=0x7f5bf0141360 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:00:19.159 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2019-12-05T06:00:21.160 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:00:21.272 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:21.260+0000 7fdc706a3700 1 Processor -- start 2019-12-05T06:00:21.273 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:21.260+0000 7fdc706a3700 1 -- start start 2019-12-05T06:00:21.273 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:21.264+0000 7fdc706a3700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fdc680a7740 0x7fdc6808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:00:21.273 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:21.264+0000 7fdc706a3700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdc6807e720 0x7fdc6807eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:00:21.273 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:21.264+0000 7fdc706a3700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc680a0710 con 0x7fdc6809f2e0 2019-12-05T06:00:21.273 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:21.264+0000 7fdc706a3700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc680a0b00 con 0x7fdc680a7740 2019-12-05T06:00:21.273 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:21.264+0000 7fdc706a3700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc6808e5b0 con 0x7fdc6807e720 2019-12-05T06:00:21.274 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:21.264+0000 7fdc6ec40700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fdc680a7740 0x7fdc6808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:00:21.274 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:21.264+0000 7fdc6ec40700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fdc680a7740 0x7fdc6808df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:45930/0 (socket says 172.21.15.42:45930) 2019-12-05T06:00:21.274 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:21.264+0000 7fdc6ec40700 1 -- 172.21.15.42:0/1250025802 learned_addr learned my addr 172.21.15.42:0/1250025802 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:00:21.274 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:21.264+0000 7fdc6e43f700 1 --2- 172.21.15.42:0/1250025802 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdc6807e720 0x7fdc6807eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:00:21.274 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:21.264+0000 7fdc6ec40700 1 -- 172.21.15.42:0/1250025802 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdc6807e720 msgr2=0x7fdc6807eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:00:21.274 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:21.264+0000 7fdc6ec40700 1 --2- 172.21.15.42:0/1250025802 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdc6807e720 0x7fdc6807eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:00:21.274 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:21.264+0000 7fdc6ec40700 1 -- 172.21.15.42:0/1250025802 >> v1:172.21.15.40:6789/0 conn(0x7fdc6809f2e0 legacy=0x7fdc6809f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:00:21.275 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:21.264+0000 7fdc6ec40700 1 -- 172.21.15.42:0/1250025802 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc6807f0a0 con 0x7fdc680a7740 2019-12-05T06:00:21.275 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:21.264+0000 7fdc6ec40700 1 --2- 172.21.15.42:0/1250025802 >> v2:172.21.15.42:3300/0 conn(0x7fdc680a7740 0x7fdc6808df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fdc64008790 tx=0x7fdc6400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:21.275 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:21.264+0000 7fdc6ec40700 1 --2- 172.21.15.42:0/1250025802 >> v2:172.21.15.42:3300/0 conn(0x7fdc680a7740 0x7fdc6808df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fdc64008790 tx=0x7fdc6400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:21.275 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:21.264+0000 7fdc6ec40700 1 --2- 172.21.15.42:0/1250025802 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdc680a7740 0x7fdc6808df80 secure :-1 s=READY pgs=1850 cs=0 l=1 rx=0x7fdc64008790 tx=0x7fdc6400fd30).ready entity=mon.1 client_cookie=8eabc77924bcf1dd server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:00:21.275 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:21.264+0000 7fdc6ec40700 1 --2- 172.21.15.42:0/1250025802 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdc680a7740 0x7fdc6808df80 secure :-1 s=THROTTLE_DONE pgs=1850 cs=0 l=1 rx=0x7fdc64008790 tx=0x7fdc6400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:21.275 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:21.264+0000 7fdc6ec40700 1 --2- 172.21.15.42:0/1250025802 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdc680a7740 0x7fdc6808df80 secure :-1 s=THROTTLE_DONE pgs=1850 cs=0 l=1 rx=0x7fdc64008790 tx=0x7fdc6400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:21.276 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:21.264+0000 7fdc6dc3e700 1 -- 172.21.15.42:0/1250025802 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdc640113b0 con 0x7fdc680a7740 2019-12-05T06:00:21.276 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:21.264+0000 7fdc6ec40700 1 --2- 172.21.15.42:0/1250025802 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdc680a7740 0x7fdc6808df80 secure :-1 s=THROTTLE_DONE pgs=1850 cs=0 l=1 rx=0x7fdc64008790 tx=0x7fdc6400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:21.276 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:21.264+0000 7fdc6dc3e700 1 -- 172.21.15.42:0/1250025802 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc64008bb0 con 0x7fdc680a7740 2019-12-05T06:00:21.276 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:21.264+0000 7fdc6dc3e700 1 -- 172.21.15.42:0/1250025802 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdc64011990 con 0x7fdc680a7740 2019-12-05T06:00:21.276 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:21.264+0000 7fdc706a3700 1 -- 172.21.15.42:0/1250025802 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdc680a7740 msgr2=0x7fdc6808df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:00:21.277 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:21.264+0000 7fdc706a3700 1 --2- 172.21.15.42:0/1250025802 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdc680a7740 0x7fdc6808df80 secure :-1 s=READY pgs=1850 cs=0 l=1 rx=0x7fdc64008790 tx=0x7fdc6400fd30).stop 2019-12-05T06:00:21.277 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:21.268+0000 7fdc706a3700 1 -- 172.21.15.42:0/1250025802 shutdown_connections 2019-12-05T06:00:21.277 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:21.268+0000 7fdc706a3700 1 --2- 172.21.15.42:0/1250025802 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdc680a7740 0x7fdc6808df80 unknown :-1 s=CLOSED pgs=1850 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:00:21.277 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:21.268+0000 7fdc706a3700 1 --2- 172.21.15.42:0/1250025802 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdc6807e720 0x7fdc6807eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:00:21.277 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:21.268+0000 7fdc706a3700 1 -- 172.21.15.42:0/1250025802 shutdown_connections 2019-12-05T06:00:21.277 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:21.268+0000 7fdc706a3700 1 -- 172.21.15.42:0/1250025802 wait complete. 2019-12-05T06:00:21.284 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:21.268+0000 7fdc706a3700 1 -- 172.21.15.42:0/1250025802 >> 172.21.15.42:0/1250025802 conn(0x7fdc680aabe0 msgr2=0x7fdc680a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:00:21.288 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:21.268+0000 7fdc706a3700 1 Processor -- start 2019-12-05T06:00:21.288 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:21.268+0000 7fdc706a3700 1 -- start start 2019-12-05T06:00:21.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:21.268+0000 7fdc706a3700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fdc681170e0 0x7fdc6811af90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:00:21.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:21.268+0000 7fdc706a3700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fdc68117af0 0x7fdc68117f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:00:21.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:21.268+0000 7fdc706a3700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc681214f0 con 0x7fdc681239d0 2019-12-05T06:00:21.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:21.268+0000 7fdc706a3700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc681175b0 con 0x7fdc681170e0 2019-12-05T06:00:21.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:21.268+0000 7fdc706a3700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc68117850 con 0x7fdc68117af0 2019-12-05T06:00:21.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:21.272+0000 7fdc6e43f700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fdc681239d0 0x7fdc6807af50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:45976/0 (socket says 172.21.15.42:45976) 2019-12-05T06:00:21.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:21.272+0000 7fdc6e43f700 1 -- 172.21.15.42:0/1646539956 learned_addr learned my addr 172.21.15.42:0/1646539956 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:00:21.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:21.272+0000 7fdc6f441700 1 --2- 172.21.15.42:0/1646539956 >> v2:172.21.15.42:3300/0 conn(0x7fdc68117af0 0x7fdc68117f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:00:21.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:21.272+0000 7fdc6ec40700 1 --2- 172.21.15.42:0/1646539956 >> v2:172.21.15.40:3300/0 conn(0x7fdc681170e0 0x7fdc6811af90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:00:21.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:21.272+0000 7fdc6f441700 1 -- 172.21.15.42:0/1646539956 >> v2:172.21.15.40:3300/0 conn(0x7fdc681170e0 msgr2=0x7fdc6811af90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:00:21.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:21.272+0000 7fdc6f441700 1 --2- 172.21.15.42:0/1646539956 >> v2:172.21.15.40:3300/0 conn(0x7fdc681170e0 0x7fdc6811af90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:00:21.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:21.272+0000 7fdc5f7fe700 1 -- 172.21.15.42:0/1646539956 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3237438030 0 0) 0x7fdc681214f0 con 0x7fdc681239d0 2019-12-05T06:00:21.291 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:21.272+0000 7fdc6f441700 1 -- 172.21.15.42:0/1646539956 >> v1:172.21.15.42:6789/0 conn(0x7fdc681239d0 legacy=0x7fdc6807af50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:00:21.291 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:21.272+0000 7fdc6f441700 1 -- 172.21.15.42:0/1646539956 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc68118470 con 0x7fdc68117af0 2019-12-05T06:00:21.291 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:21.272+0000 7fdc6f441700 1 --2- 172.21.15.42:0/1646539956 >> v2:172.21.15.42:3300/0 conn(0x7fdc68117af0 0x7fdc68117f20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fdc60001210 tx=0x7fdc60010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:21.296 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:21.272+0000 7fdc6f441700 1 --2- 172.21.15.42:0/1646539956 >> v2:172.21.15.42:3300/0 conn(0x7fdc68117af0 0x7fdc68117f20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fdc60001210 tx=0x7fdc60010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:21.296 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:21.272+0000 7fdc6f441700 1 --2- 172.21.15.42:0/1646539956 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdc68117af0 0x7fdc68117f20 secure :-1 s=READY pgs=1852 cs=0 l=1 rx=0x7fdc60001210 tx=0x7fdc60010b40).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:00:21.296 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:21.272+0000 7fdc6f441700 1 --2- 172.21.15.42:0/1646539956 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdc68117af0 0x7fdc68117f20 secure :-1 s=THROTTLE_DONE pgs=1852 cs=0 l=1 rx=0x7fdc60001210 tx=0x7fdc60010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:21.296 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:21.272+0000 7fdc5f7fe700 1 -- 172.21.15.42:0/1646539956 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdc60006dc0 con 0x7fdc68117af0 2019-12-05T06:00:21.296 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:21.272+0000 7fdc6f441700 1 --2- 172.21.15.42:0/1646539956 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdc68117af0 0x7fdc68117f20 secure :-1 s=THROTTLE_DONE pgs=1852 cs=0 l=1 rx=0x7fdc60001210 tx=0x7fdc60010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:21.296 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:21.272+0000 7fdc6f441700 1 --2- 172.21.15.42:0/1646539956 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdc68117af0 0x7fdc68117f20 secure :-1 s=THROTTLE_DONE pgs=1852 cs=0 l=1 rx=0x7fdc60001210 tx=0x7fdc60010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:21.296 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:21.272+0000 7fdc5f7fe700 1 -- 172.21.15.42:0/1646539956 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc600069e0 con 0x7fdc68117af0 2019-12-05T06:00:21.297 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:21.272+0000 7fdc5f7fe700 1 -- 172.21.15.42:0/1646539956 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdc60002600 con 0x7fdc68117af0 2019-12-05T06:00:21.297 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:21.272+0000 7fdc706a3700 1 -- 172.21.15.42:0/1646539956 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc6808eb70 con 0x7fdc68117af0 2019-12-05T06:00:21.297 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:21.272+0000 7fdc706a3700 1 -- 172.21.15.42:0/1646539956 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc6808f0c0 con 0x7fdc68117af0 2019-12-05T06:00:21.297 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:21.272+0000 7fdc6f441700 1 --2- 172.21.15.42:0/1646539956 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdc68117af0 0x7fdc68117f20 secure :-1 s=THROTTLE_DONE pgs=1852 cs=0 l=1 rx=0x7fdc60001210 tx=0x7fdc60010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:21.297 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:21.272+0000 7fdc5f7fe700 1 -- 172.21.15.42:0/1646539956 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fdc60002910 con 0x7fdc68117af0 2019-12-05T06:00:21.297 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:21.272+0000 7fdc6f441700 1 --2- 172.21.15.42:0/1646539956 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdc68117af0 0x7fdc68117f20 secure :-1 s=THROTTLE_DONE pgs=1852 cs=0 l=1 rx=0x7fdc60001210 tx=0x7fdc60010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:21.297 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:21.272+0000 7fdc5f7fe700 1 --2- 172.21.15.42:0/1646539956 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fdc5003acf0 0x7fdc5003d200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:00:21.298 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:21.272+0000 7fdc5f7fe700 1 -- 172.21.15.42:0/1646539956 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9385+0+0 (secure 0 0 0) 0x7fdc6004b460 con 0x7fdc68117af0 2019-12-05T06:00:21.298 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:21.272+0000 7fdc6ec40700 1 --2- 172.21.15.42:0/1646539956 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fdc5003acf0 0x7fdc5003d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:00:21.298 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:21.276+0000 7fdc6ec40700 1 --2- 172.21.15.42:0/1646539956 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fdc5003acf0 0x7fdc5003d200 crc :-1 s=READY pgs=790 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:00:21.298 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:21.276+0000 7fdc706a3700 1 -- 172.21.15.42:0/1646539956 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc6805d110 con 0x7fdc68117af0 2019-12-05T06:00:21.299 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:21.284+0000 7fdc6f441700 1 --2- 172.21.15.42:0/1646539956 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdc68117af0 0x7fdc68117f20 secure :-1 s=THROTTLE_DONE pgs=1852 cs=0 l=1 rx=0x7fdc60001210 tx=0x7fdc60010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:21.299 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:21.284+0000 7fdc5f7fe700 1 -- 172.21.15.42:0/1646539956 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fdc600324e0 con 0x7fdc68117af0 2019-12-05T06:00:21.465 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:21.460+0000 7fdc706a3700 1 -- 172.21.15.42:0/1646539956 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdc6805b8c0 con 0x7fdc68117af0 2019-12-05T06:00:21.466 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:21.460+0000 7fdc6f441700 1 --2- 172.21.15.42:0/1646539956 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdc68117af0 0x7fdc68117f20 secure :-1 s=THROTTLE_DONE pgs=1852 cs=0 l=1 rx=0x7fdc60001210 tx=0x7fdc60010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:21.466 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:21.460+0000 7fdc5f7fe700 1 -- 172.21.15.42:0/1646539956 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 227 v227) v1 ==== 95+0+1771 (secure 0 0 0) 0x7fdc60032740 con 0x7fdc68117af0 2019-12-05T06:00:21.466 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 227 2019-12-05T06:00:21.467 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:00:21.467 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":227,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":227,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:00:19.604634+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":88,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9061},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9061":{"gid":9061,"name":"a","rank":0,"incarnation":224,"state":"up:active","state_seq":9,"addr":"172.21.15.42:6835/448869597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":448869597},{"type":"v1","addr":"172.21.15.42:6835","nonce":448869597}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:00:21.469 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:21.464+0000 7fdc706a3700 1 -- 172.21.15.42:0/1646539956 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fdc5003acf0 msgr2=0x7fdc5003d200 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:00:21.469 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:21.464+0000 7fdc706a3700 1 --2- 172.21.15.42:0/1646539956 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fdc5003acf0 0x7fdc5003d200 crc :-1 s=READY pgs=790 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:00:21.469 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:21.464+0000 7fdc706a3700 1 -- 172.21.15.42:0/1646539956 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdc68117af0 msgr2=0x7fdc68117f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:00:21.469 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:21.464+0000 7fdc706a3700 1 --2- 172.21.15.42:0/1646539956 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdc68117af0 0x7fdc68117f20 secure :-1 s=READY pgs=1852 cs=0 l=1 rx=0x7fdc60001210 tx=0x7fdc60010b40).stop 2019-12-05T06:00:21.469 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:21.464+0000 7fdc706a3700 1 -- 172.21.15.42:0/1646539956 shutdown_connections 2019-12-05T06:00:21.470 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:21.464+0000 7fdc706a3700 1 --2- 172.21.15.42:0/1646539956 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fdc5003acf0 0x7fdc5003d200 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:00:21.470 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:21.464+0000 7fdc706a3700 1 --2- 172.21.15.42:0/1646539956 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdc68117af0 0x7fdc68117f20 unknown :-1 s=CLOSED pgs=1852 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:00:21.470 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:21.464+0000 7fdc706a3700 1 --2- 172.21.15.42:0/1646539956 >> v2:172.21.15.40:3300/0 conn(0x7fdc681170e0 0x7fdc6811af90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:00:21.470 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:21.464+0000 7fdc706a3700 1 -- 172.21.15.42:0/1646539956 shutdown_connections 2019-12-05T06:00:21.470 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:21.464+0000 7fdc706a3700 1 -- 172.21.15.42:0/1646539956 wait complete. 2019-12-05T06:00:21.470 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:21.464+0000 7fdc706a3700 1 -- 172.21.15.42:0/1646539956 >> 172.21.15.42:0/1646539956 conn(0x7fdc680aabe0 msgr2=0x7fdc6809f460 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:00:21.483 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a has gained rank=0, replacing gid=8940 2019-12-05T06:00:21.483 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 2 secs before reviving mds.b 2019-12-05T06:00:23.485 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.b 2019-12-05T06:00:23.485 INFO:tasks.ceph.mds.b:Restarting daemon 2019-12-05T06:00:23.485 INFO:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2019-12-05T06:00:23.489 INFO:tasks.ceph.mds.b:Started 2019-12-05T06:00:23.489 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:00:23.550 INFO:tasks.ceph.mds.b.smithi040.stdout:starting mds.b at 2019-12-05T06:00:23.616 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:23.584+0000 7f200cf79700 1 Processor -- start 2019-12-05T06:00:23.617 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:23.584+0000 7f200cf79700 1 -- start start 2019-12-05T06:00:23.617 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:23.584+0000 7f200cf79700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f20080a7740 0x7f200808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:00:23.617 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:23.584+0000 7f200cf79700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f200807e720 0x7f200807eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:00:23.618 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:23.584+0000 7f200cf79700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f200809f2e0 0x7f200809f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:00:23.618 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:23.584+0000 7f200cf79700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20080a0b00 con 0x7f200807e720 2019-12-05T06:00:23.618 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:23.584+0000 7f200cf79700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f200808e5b0 con 0x7f200809f2e0 2019-12-05T06:00:23.618 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:23.584+0000 7f200cf79700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f200808e850 con 0x7f20080a7740 2019-12-05T06:00:23.618 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:23.584+0000 7f200759e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f200809f2e0 0x7f200809f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:00:23.619 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:23.584+0000 7f200759e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f200809f2e0 0x7f200809f710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:45954/0 (socket says 172.21.15.42:45954) 2019-12-05T06:00:23.619 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:23.584+0000 7f200759e700 1 -- 172.21.15.42:0/3168605675 learned_addr learned my addr 172.21.15.42:0/3168605675 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:00:23.619 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:23.584+0000 7f200659c700 1 --2- 172.21.15.42:0/3168605675 >> v2:172.21.15.40:3300/0 conn(0x7f200807e720 0x7f200807eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:00:23.619 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:23.584+0000 7f2006d9d700 1 --2- 172.21.15.42:0/3168605675 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f20080a7740 0x7f200808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:00:23.619 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:23.588+0000 7f200759e700 1 -- 172.21.15.42:0/3168605675 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f20080a7740 msgr2=0x7f200808df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:00:23.620 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:23.588+0000 7f200759e700 1 --2- 172.21.15.42:0/3168605675 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f20080a7740 0x7f200808df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:00:23.620 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:23.588+0000 7f200759e700 1 -- 172.21.15.42:0/3168605675 >> v2:172.21.15.40:3300/0 conn(0x7f200807e720 msgr2=0x7f200807eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:00:23.620 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:23.588+0000 7f200759e700 1 --2- 172.21.15.42:0/3168605675 >> v2:172.21.15.40:3300/0 conn(0x7f200807e720 0x7f200807eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:00:23.620 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:23.588+0000 7f200759e700 1 -- 172.21.15.42:0/3168605675 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f200807f0a0 con 0x7f200809f2e0 2019-12-05T06:00:23.620 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:23.588+0000 7f200659c700 1 --2- 172.21.15.42:0/3168605675 >> v2:172.21.15.40:3300/0 conn(0x7f200807e720 0x7f200807eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:00:23.621 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:23.588+0000 7f200759e700 1 --2- 172.21.15.42:0/3168605675 >> v2:172.21.15.42:3300/0 conn(0x7f200809f2e0 0x7f200809f710 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f200808e770 tx=0x7f2000010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:23.621 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:23.588+0000 7f200759e700 1 --2- 172.21.15.42:0/3168605675 >> v2:172.21.15.42:3300/0 conn(0x7f200809f2e0 0x7f200809f710 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f200808e770 tx=0x7f2000010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:23.621 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:23.588+0000 7f200759e700 1 --2- 172.21.15.42:0/3168605675 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f200809f2e0 0x7f200809f710 secure :-1 s=READY pgs=1854 cs=0 l=1 rx=0x7f200808e770 tx=0x7f2000010130).ready entity=mon.1 client_cookie=7ab19ecddb9d17a5 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:00:23.621 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:23.588+0000 7f200759e700 1 --2- 172.21.15.42:0/3168605675 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f200809f2e0 0x7f200809f710 secure :-1 s=THROTTLE_DONE pgs=1854 cs=0 l=1 rx=0x7f200808e770 tx=0x7f2000010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:23.621 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:23.588+0000 7f2005d9b700 1 -- 172.21.15.42:0/3168605675 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2000010de0 con 0x7f200809f2e0 2019-12-05T06:00:23.621 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:23.588+0000 7f200759e700 1 --2- 172.21.15.42:0/3168605675 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f200809f2e0 0x7f200809f710 secure :-1 s=THROTTLE_DONE pgs=1854 cs=0 l=1 rx=0x7f200808e770 tx=0x7f2000010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:23.621 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:23.588+0000 7f200759e700 1 --2- 172.21.15.42:0/3168605675 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f200809f2e0 0x7f200809f710 secure :-1 s=THROTTLE_DONE pgs=1854 cs=0 l=1 rx=0x7f200808e770 tx=0x7f2000010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:23.622 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:23.588+0000 7f2005d9b700 1 -- 172.21.15.42:0/3168605675 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2000010a00 con 0x7f200809f2e0 2019-12-05T06:00:23.622 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:23.588+0000 7f2005d9b700 1 -- 172.21.15.42:0/3168605675 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f20000114a0 con 0x7f200809f2e0 2019-12-05T06:00:23.622 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:23.588+0000 7f200cf79700 1 -- 172.21.15.42:0/3168605675 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f200809f2e0 msgr2=0x7f200809f710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:00:23.622 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:23.588+0000 7f200cf79700 1 --2- 172.21.15.42:0/3168605675 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f200809f2e0 0x7f200809f710 secure :-1 s=READY pgs=1854 cs=0 l=1 rx=0x7f200808e770 tx=0x7f2000010130).stop 2019-12-05T06:00:23.622 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:23.588+0000 7f200cf79700 1 -- 172.21.15.42:0/3168605675 shutdown_connections 2019-12-05T06:00:23.622 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:23.588+0000 7f200cf79700 1 --2- 172.21.15.42:0/3168605675 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f200809f2e0 0x7f200809f710 unknown :-1 s=CLOSED pgs=1854 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:00:23.622 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:23.588+0000 7f200cf79700 1 --2- 172.21.15.42:0/3168605675 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f20080a7740 0x7f200808df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:00:23.623 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:23.588+0000 7f200cf79700 1 --2- 172.21.15.42:0/3168605675 >> v2:172.21.15.40:3300/0 conn(0x7f200807e720 0x7f200807eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:00:23.623 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:23.588+0000 7f200cf79700 1 -- 172.21.15.42:0/3168605675 shutdown_connections 2019-12-05T06:00:23.623 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:23.588+0000 7f200cf79700 1 -- 172.21.15.42:0/3168605675 wait complete. 2019-12-05T06:00:23.623 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:23.588+0000 7f200cf79700 1 -- 172.21.15.42:0/3168605675 >> 172.21.15.42:0/3168605675 conn(0x7f20080aabe0 msgr2=0x7f20080a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:00:23.623 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:23.588+0000 7f200cf79700 1 Processor -- start 2019-12-05T06:00:23.623 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:23.588+0000 7f200cf79700 1 -- start start 2019-12-05T06:00:23.623 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:23.592+0000 7f200cf79700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2008125130 0x7f20080a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:00:23.623 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:23.592+0000 7f200cf79700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2008142470 0x7f200812ace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:00:23.624 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:23.592+0000 7f200cf79700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f200811d270 con 0x7f200812b230 2019-12-05T06:00:23.624 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:23.592+0000 7f200cf79700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f200812b660 con 0x7f2008125130 2019-12-05T06:00:23.624 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:23.592+0000 7f200cf79700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20080a6bc0 con 0x7f2008142470 2019-12-05T06:00:23.624 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:23.592+0000 7f200759e700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f200812b230 0x7f200812e2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:45998/0 (socket says 172.21.15.42:45998) 2019-12-05T06:00:23.624 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:23.592+0000 7f200759e700 1 -- 172.21.15.42:0/835180520 learned_addr learned my addr 172.21.15.42:0/835180520 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:00:23.625 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:23.592+0000 7f200659c700 1 --2- 172.21.15.42:0/835180520 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2008142470 0x7f200812ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:00:23.625 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:23.592+0000 7f2006d9d700 1 --2- 172.21.15.42:0/835180520 >> v2:172.21.15.40:3300/0 conn(0x7f2008125130 0x7f20080a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:00:23.625 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:23.592+0000 7f1ff77fe700 1 -- 172.21.15.42:0/835180520 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 502611467 0 0) 0x7f200811d270 con 0x7f200812b230 2019-12-05T06:00:23.625 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:23.592+0000 7f1ff77fe700 1 -- 172.21.15.42:0/835180520 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1fec0030b0 con 0x7f200812b230 2019-12-05T06:00:23.625 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:23.592+0000 7f200659c700 1 -- 172.21.15.42:0/835180520 >> v2:172.21.15.40:3300/0 conn(0x7f2008125130 msgr2=0x7f20080a4d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:00:23.625 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:23.592+0000 7f200659c700 1 --2- 172.21.15.42:0/835180520 >> v2:172.21.15.40:3300/0 conn(0x7f2008125130 0x7f20080a4d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:00:23.626 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:23.592+0000 7f200659c700 1 -- 172.21.15.42:0/835180520 >> v1:172.21.15.42:6789/0 conn(0x7f200812b230 legacy=0x7f200812e2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:00:23.626 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:23.592+0000 7f200659c700 1 -- 172.21.15.42:0/835180520 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20080a5450 con 0x7f2008142470 2019-12-05T06:00:23.626 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:23.592+0000 7f2006d9d700 1 --2- 172.21.15.42:0/835180520 >> v2:172.21.15.40:3300/0 conn(0x7f2008125130 0x7f20080a4d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:00:23.626 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:23.592+0000 7f200659c700 1 --2- 172.21.15.42:0/835180520 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2008142470 0x7f200812ace0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f1ffc002f30 tx=0x7f1ffc010a00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:23.626 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:23.592+0000 7f200659c700 1 --2- 172.21.15.42:0/835180520 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2008142470 0x7f200812ace0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f1ffc002f30 tx=0x7f1ffc010a00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:23.626 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:23.592+0000 7f200659c700 1 --2- 172.21.15.42:0/835180520 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2008142470 0x7f200812ace0 secure :-1 s=READY pgs=414 cs=0 l=1 rx=0x7f1ffc002f30 tx=0x7f1ffc010a00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:00:23.626 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:23.592+0000 7f200659c700 1 --2- 172.21.15.42:0/835180520 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2008142470 0x7f200812ace0 secure :-1 s=THROTTLE_DONE pgs=414 cs=0 l=1 rx=0x7f1ffc002f30 tx=0x7f1ffc010a00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:23.627 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:23.592+0000 7f1ff77fe700 1 -- 172.21.15.42:0/835180520 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1ffc005840 con 0x7f2008142470 2019-12-05T06:00:23.627 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:23.592+0000 7f200659c700 1 --2- 172.21.15.42:0/835180520 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2008142470 0x7f200812ace0 secure :-1 s=THROTTLE_DONE pgs=414 cs=0 l=1 rx=0x7f1ffc002f30 tx=0x7f1ffc010a00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:23.627 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:23.592+0000 7f1ff77fe700 1 -- 172.21.15.42:0/835180520 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ffc007160 con 0x7f2008142470 2019-12-05T06:00:23.627 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:23.592+0000 7f200659c700 1 --2- 172.21.15.42:0/835180520 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2008142470 0x7f200812ace0 secure :-1 s=THROTTLE_DONE pgs=414 cs=0 l=1 rx=0x7f1ffc002f30 tx=0x7f1ffc010a00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:23.627 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:23.592+0000 7f200cf79700 1 -- 172.21.15.42:0/835180520 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20081428a0 con 0x7f2008142470 2019-12-05T06:00:23.627 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:23.592+0000 7f1ff77fe700 1 -- 172.21.15.42:0/835180520 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1ffc0060c0 con 0x7f2008142470 2019-12-05T06:00:23.627 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:23.592+0000 7f200cf79700 1 -- 172.21.15.42:0/835180520 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f200812eaa0 con 0x7f2008142470 2019-12-05T06:00:23.628 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:23.596+0000 7f200659c700 1 --2- 172.21.15.42:0/835180520 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2008142470 0x7f200812ace0 secure :-1 s=THROTTLE_DONE pgs=414 cs=0 l=1 rx=0x7f1ffc002f30 tx=0x7f1ffc010a00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:23.628 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:23.596+0000 7f1ff77fe700 1 -- 172.21.15.42:0/835180520 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f1ffc006360 con 0x7f2008142470 2019-12-05T06:00:23.628 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:23.596+0000 7f200659c700 1 --2- 172.21.15.42:0/835180520 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2008142470 0x7f200812ace0 secure :-1 s=THROTTLE_DONE pgs=414 cs=0 l=1 rx=0x7f1ffc002f30 tx=0x7f1ffc010a00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:23.628 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:23.596+0000 7f1ff77fe700 1 --2- 172.21.15.42:0/835180520 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f1fec03b270 0x7f1fec03d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:00:23.628 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:23.596+0000 7f1ff77fe700 1 -- 172.21.15.42:0/835180520 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9385+0+0 (secure 0 0 0) 0x7f1ffc04aef0 con 0x7f2008142470 2019-12-05T06:00:23.628 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:23.596+0000 7f2006d9d700 1 --2- 172.21.15.42:0/835180520 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f1fec03b270 0x7f1fec03d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:00:23.628 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:23.596+0000 7f2006d9d700 1 --2- 172.21.15.42:0/835180520 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f1fec03b270 0x7f1fec03d780 crc :-1 s=READY pgs=791 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:00:23.629 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:23.600+0000 7f200cf79700 1 -- 172.21.15.42:0/835180520 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f200805d110 con 0x7f2008142470 2019-12-05T06:00:23.629 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:23.608+0000 7f200659c700 1 --2- 172.21.15.42:0/835180520 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2008142470 0x7f200812ace0 secure :-1 s=THROTTLE_DONE pgs=414 cs=0 l=1 rx=0x7f1ffc002f30 tx=0x7f1ffc010a00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:23.629 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:23.608+0000 7f1ff77fe700 1 -- 172.21.15.42:0/835180520 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f1ffc031f50 con 0x7f2008142470 2019-12-05T06:00:23.795 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:23.780+0000 7f200cf79700 1 -- 172.21.15.42:0/835180520 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f200805b8c0 con 0x7f2008142470 2019-12-05T06:00:23.795 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:23.780+0000 7f200659c700 1 --2- 172.21.15.42:0/835180520 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2008142470 0x7f200812ace0 secure :-1 s=THROTTLE_DONE pgs=414 cs=0 l=1 rx=0x7f1ffc002f30 tx=0x7f1ffc010a00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:23.795 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:23.780+0000 7f1ff77fe700 1 -- 172.21.15.42:0/835180520 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 227 v227) v1 ==== 95+0+1771 (secure 0 0 0) 0x7f1ffc0321b0 con 0x7f2008142470 2019-12-05T06:00:23.795 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 227 2019-12-05T06:00:23.796 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:23.784+0000 7f200cf79700 1 -- 172.21.15.42:0/835180520 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f1fec03b270 msgr2=0x7f1fec03d780 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:00:23.796 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:23.784+0000 7f200cf79700 1 --2- 172.21.15.42:0/835180520 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f1fec03b270 0x7f1fec03d780 crc :-1 s=READY pgs=791 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:00:23.796 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:23.784+0000 7f200cf79700 1 -- 172.21.15.42:0/835180520 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2008142470 msgr2=0x7f200812ace0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:00:23.796 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:23.784+0000 7f200cf79700 1 --2- 172.21.15.42:0/835180520 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2008142470 0x7f200812ace0 secure :-1 s=READY pgs=414 cs=0 l=1 rx=0x7f1ffc002f30 tx=0x7f1ffc010a00).stop 2019-12-05T06:00:23.796 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:23.784+0000 7f200cf79700 1 -- 172.21.15.42:0/835180520 shutdown_connections 2019-12-05T06:00:23.797 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:23.784+0000 7f200cf79700 1 --2- 172.21.15.42:0/835180520 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f1fec03b270 0x7f1fec03d780 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:00:23.797 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:23.784+0000 7f200cf79700 1 --2- 172.21.15.42:0/835180520 >> v2:172.21.15.40:3300/0 conn(0x7f2008125130 0x7f20080a4d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:00:23.797 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:23.784+0000 7f200cf79700 1 --2- 172.21.15.42:0/835180520 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2008142470 0x7f200812ace0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:00:23.797 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:23.784+0000 7f200cf79700 1 -- 172.21.15.42:0/835180520 shutdown_connections 2019-12-05T06:00:23.798 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:23.784+0000 7f200cf79700 1 -- 172.21.15.42:0/835180520 wait complete. 2019-12-05T06:00:23.798 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:23.784+0000 7f200cf79700 1 -- 172.21.15.42:0/835180520 >> 172.21.15.42:0/835180520 conn(0x7f200807de00 msgr2=0x7f20080a9790 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:00:23.798 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:00:23.798 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":227,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":227,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:00:19.604634+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":88,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9061},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9061":{"gid":9061,"name":"a","rank":0,"incarnation":224,"state":"up:active","state_seq":9,"addr":"172.21.15.42:6835/448869597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":448869597},{"type":"v1","addr":"172.21.15.42:6835","nonce":448869597}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:00:23.813 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2019-12-05T06:00:25.813 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:00:25.912 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:25.904+0000 7f2f815d6700 1 Processor -- start 2019-12-05T06:00:25.913 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:25.904+0000 7f2f815d6700 1 -- start start 2019-12-05T06:00:25.913 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:25.904+0000 7f2f815d6700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2f7c08df80 0x7f2f7c09f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:00:25.913 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:25.904+0000 7f2f815d6700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2f7c09f830 0x7f2f7c14efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:00:25.913 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:25.904+0000 7f2f815d6700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f7c09fc60 con 0x7f2f7c0a7740 2019-12-05T06:00:25.913 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:25.904+0000 7f2f815d6700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f7c0a0b00 con 0x7f2f7c08df80 2019-12-05T06:00:25.913 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:25.904+0000 7f2f815d6700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f7c08e5b0 con 0x7f2f7c09f830 2019-12-05T06:00:25.914 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:25.904+0000 7f2f7b7fe700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f2f7c0a7740 0x7f2f7c07e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:46008/0 (socket says 172.21.15.42:46008) 2019-12-05T06:00:25.914 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:25.904+0000 7f2f7b7fe700 1 -- 172.21.15.42:0/1531692329 learned_addr learned my addr 172.21.15.42:0/1531692329 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:00:25.914 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:25.904+0000 7f2f7bfff700 1 --2- 172.21.15.42:0/1531692329 >> v2:172.21.15.42:3300/0 conn(0x7f2f7c09f830 0x7f2f7c14efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:00:25.914 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:25.904+0000 7f2f7affd700 1 --2- 172.21.15.42:0/1531692329 >> v2:172.21.15.40:3300/0 conn(0x7f2f7c08df80 0x7f2f7c09f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:00:25.914 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:25.904+0000 7f2f7a7fc700 1 -- 172.21.15.42:0/1531692329 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 334216022 0 0) 0x7f2f7c09fc60 con 0x7f2f7c0a7740 2019-12-05T06:00:25.914 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:25.904+0000 7f2f7a7fc700 1 -- 172.21.15.42:0/1531692329 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f640030b0 con 0x7f2f7c0a7740 2019-12-05T06:00:25.914 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:25.904+0000 7f2f7bfff700 1 -- 172.21.15.42:0/1531692329 >> v2:172.21.15.40:3300/0 conn(0x7f2f7c08df80 msgr2=0x7f2f7c09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:00:25.915 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:25.904+0000 7f2f7bfff700 1 --2- 172.21.15.42:0/1531692329 >> v2:172.21.15.40:3300/0 conn(0x7f2f7c08df80 0x7f2f7c09f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:00:25.915 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:25.904+0000 7f2f7bfff700 1 -- 172.21.15.42:0/1531692329 >> v1:172.21.15.42:6789/0 conn(0x7f2f7c0a7740 legacy=0x7f2f7c07e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:00:25.915 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:25.904+0000 7f2f7bfff700 1 -- 172.21.15.42:0/1531692329 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f7c0a0710 con 0x7f2f7c09f830 2019-12-05T06:00:25.915 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:25.904+0000 7f2f7affd700 1 --2- 172.21.15.42:0/1531692329 >> v2:172.21.15.40:3300/0 conn(0x7f2f7c08df80 0x7f2f7c09f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:00:25.915 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:25.904+0000 7f2f7bfff700 1 --2- 172.21.15.42:0/1531692329 >> v2:172.21.15.42:3300/0 conn(0x7f2f7c09f830 0x7f2f7c14efe0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f2f740032b0 tx=0x7f2f7400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:25.915 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:25.904+0000 7f2f7bfff700 1 --2- 172.21.15.42:0/1531692329 >> v2:172.21.15.42:3300/0 conn(0x7f2f7c09f830 0x7f2f7c14efe0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f2f740032b0 tx=0x7f2f7400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:25.915 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:25.904+0000 7f2f7bfff700 1 --2- 172.21.15.42:0/1531692329 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2f7c09f830 0x7f2f7c14efe0 secure :-1 s=READY pgs=1857 cs=0 l=1 rx=0x7f2f740032b0 tx=0x7f2f7400fff0).ready entity=mon.1 client_cookie=e151d93a83d573ab server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:00:25.916 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:25.904+0000 7f2f7bfff700 1 --2- 172.21.15.42:0/1531692329 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2f7c09f830 0x7f2f7c14efe0 secure :-1 s=THROTTLE_DONE pgs=1857 cs=0 l=1 rx=0x7f2f740032b0 tx=0x7f2f7400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:25.916 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:25.904+0000 7f2f7a7fc700 1 -- 172.21.15.42:0/1531692329 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2f740108c0 con 0x7f2f7c09f830 2019-12-05T06:00:25.916 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:25.904+0000 7f2f7bfff700 1 --2- 172.21.15.42:0/1531692329 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2f7c09f830 0x7f2f7c14efe0 secure :-1 s=THROTTLE_DONE pgs=1857 cs=0 l=1 rx=0x7f2f740032b0 tx=0x7f2f7400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:25.916 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:25.904+0000 7f2f7bfff700 1 --2- 172.21.15.42:0/1531692329 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2f7c09f830 0x7f2f7c14efe0 secure :-1 s=THROTTLE_DONE pgs=1857 cs=0 l=1 rx=0x7f2f740032b0 tx=0x7f2f7400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:25.928 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:25.904+0000 7f2f7a7fc700 1 -- 172.21.15.42:0/1531692329 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f74008b50 con 0x7f2f7c09f830 2019-12-05T06:00:25.929 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:25.904+0000 7f2f7a7fc700 1 -- 172.21.15.42:0/1531692329 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2f740110c0 con 0x7f2f7c09f830 2019-12-05T06:00:25.929 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:25.904+0000 7f2f815d6700 1 -- 172.21.15.42:0/1531692329 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2f7c09f830 msgr2=0x7f2f7c14efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:00:25.929 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:25.904+0000 7f2f815d6700 1 --2- 172.21.15.42:0/1531692329 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2f7c09f830 0x7f2f7c14efe0 secure :-1 s=READY pgs=1857 cs=0 l=1 rx=0x7f2f740032b0 tx=0x7f2f7400fff0).stop 2019-12-05T06:00:25.929 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:25.908+0000 7f2f815d6700 1 -- 172.21.15.42:0/1531692329 shutdown_connections 2019-12-05T06:00:25.930 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:25.908+0000 7f2f815d6700 1 --2- 172.21.15.42:0/1531692329 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2f7c09f830 0x7f2f7c14efe0 unknown :-1 s=CLOSED pgs=1857 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:00:25.930 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:25.908+0000 7f2f815d6700 1 --2- 172.21.15.42:0/1531692329 >> v2:172.21.15.40:3300/0 conn(0x7f2f7c08df80 0x7f2f7c09f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:00:25.930 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:25.908+0000 7f2f815d6700 1 -- 172.21.15.42:0/1531692329 shutdown_connections 2019-12-05T06:00:25.930 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:25.908+0000 7f2f815d6700 1 -- 172.21.15.42:0/1531692329 wait complete. 2019-12-05T06:00:25.930 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:25.908+0000 7f2f815d6700 1 -- 172.21.15.42:0/1531692329 >> 172.21.15.42:0/1531692329 conn(0x7f2f7c0aabe0 msgr2=0x7f2f7c0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:00:25.934 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:25.908+0000 7f2f815d6700 1 Processor -- start 2019-12-05T06:00:25.934 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:25.908+0000 7f2f815d6700 1 -- start start 2019-12-05T06:00:25.935 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:25.912+0000 7f2f815d6700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2f7c116500 0x7f2f7c07b050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:00:25.935 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:25.912+0000 7f2f815d6700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2f7c120b70 0x7f2f7c120fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:00:25.935 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:25.912+0000 7f2f815d6700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2f7c08f1b0 0x7f2f7c08f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:00:25.935 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:25.912+0000 7f2f815d6700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f7c11a6a0 con 0x7f2f7c120b70 2019-12-05T06:00:25.935 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:25.912+0000 7f2f815d6700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f7c11b680 con 0x7f2f7c116500 2019-12-05T06:00:25.935 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:25.912+0000 7f2f815d6700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f7c11b920 con 0x7f2f7c08f1b0 2019-12-05T06:00:25.935 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:25.912+0000 7f2f7b7fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2f7c116500 0x7f2f7c07b050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:00:25.936 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:25.912+0000 7f2f7affd700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2f7c120b70 0x7f2f7c120fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:00:25.936 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:25.912+0000 7f2f7b7fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2f7c116500 0x7f2f7c07b050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:45976/0 (socket says 172.21.15.42:45976) 2019-12-05T06:00:25.936 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:25.912+0000 7f2f7b7fe700 1 -- 172.21.15.42:0/4251328766 learned_addr learned my addr 172.21.15.42:0/4251328766 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:00:25.936 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:25.912+0000 7f2f7bfff700 1 --2- 172.21.15.42:0/4251328766 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2f7c08f1b0 0x7f2f7c08f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:00:25.936 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:25.912+0000 7f2f7b7fe700 1 -- 172.21.15.42:0/4251328766 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2f7c08f1b0 msgr2=0x7f2f7c08f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:00:25.936 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:25.912+0000 7f2f7b7fe700 1 --2- 172.21.15.42:0/4251328766 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2f7c08f1b0 0x7f2f7c08f5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:00:25.936 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:25.912+0000 7f2f7b7fe700 1 -- 172.21.15.42:0/4251328766 >> v2:172.21.15.40:3300/0 conn(0x7f2f7c120b70 msgr2=0x7f2f7c120fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:00:25.937 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:25.912+0000 7f2f7b7fe700 1 --2- 172.21.15.42:0/4251328766 >> v2:172.21.15.40:3300/0 conn(0x7f2f7c120b70 0x7f2f7c120fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:00:25.937 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:25.912+0000 7f2f7b7fe700 1 -- 172.21.15.42:0/4251328766 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f7c07b5a0 con 0x7f2f7c116500 2019-12-05T06:00:25.937 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:25.912+0000 7f2f7b7fe700 1 --2- 172.21.15.42:0/4251328766 >> v2:172.21.15.42:3300/0 conn(0x7f2f7c116500 0x7f2f7c07b050 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f2f7c11b840 tx=0x7f2f6c00c450).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:25.937 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:25.912+0000 7f2f7b7fe700 1 --2- 172.21.15.42:0/4251328766 >> v2:172.21.15.42:3300/0 conn(0x7f2f7c116500 0x7f2f7c07b050 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f2f7c11b840 tx=0x7f2f6c00c450).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:25.937 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:25.912+0000 7f2f7b7fe700 1 --2- 172.21.15.42:0/4251328766 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2f7c116500 0x7f2f7c07b050 secure :-1 s=READY pgs=1858 cs=0 l=1 rx=0x7f2f7c11b840 tx=0x7f2f6c00c450).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:00:25.937 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:25.912+0000 7f2f7b7fe700 1 --2- 172.21.15.42:0/4251328766 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2f7c116500 0x7f2f7c07b050 secure :-1 s=THROTTLE_DONE pgs=1858 cs=0 l=1 rx=0x7f2f7c11b840 tx=0x7f2f6c00c450).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:25.937 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:25.912+0000 7f2f7b7fe700 1 --2- 172.21.15.42:0/4251328766 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2f7c116500 0x7f2f7c07b050 secure :-1 s=THROTTLE_DONE pgs=1858 cs=0 l=1 rx=0x7f2f7c11b840 tx=0x7f2f6c00c450).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:25.938 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:25.912+0000 7f2f7b7fe700 1 --2- 172.21.15.42:0/4251328766 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2f7c116500 0x7f2f7c07b050 secure :-1 s=THROTTLE_DONE pgs=1858 cs=0 l=1 rx=0x7f2f7c11b840 tx=0x7f2f6c00c450).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:25.938 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:25.912+0000 7f2f63fff700 1 -- 172.21.15.42:0/4251328766 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2f6c00c6c0 con 0x7f2f7c116500 2019-12-05T06:00:25.938 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:25.912+0000 7f2f63fff700 1 -- 172.21.15.42:0/4251328766 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f6c010170 con 0x7f2f7c116500 2019-12-05T06:00:25.938 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:25.912+0000 7f2f815d6700 1 -- 172.21.15.42:0/4251328766 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f7c11be60 con 0x7f2f7c116500 2019-12-05T06:00:25.938 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:25.912+0000 7f2f63fff700 1 -- 172.21.15.42:0/4251328766 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2f6c00cca0 con 0x7f2f7c116500 2019-12-05T06:00:25.938 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:25.912+0000 7f2f815d6700 1 -- 172.21.15.42:0/4251328766 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f7c124250 con 0x7f2f7c116500 2019-12-05T06:00:25.938 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:25.912+0000 7f2f7b7fe700 1 --2- 172.21.15.42:0/4251328766 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2f7c116500 0x7f2f7c07b050 secure :-1 s=THROTTLE_DONE pgs=1858 cs=0 l=1 rx=0x7f2f7c11b840 tx=0x7f2f6c00c450).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:25.939 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:25.912+0000 7f2f63fff700 1 -- 172.21.15.42:0/4251328766 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f2f6c0060a0 con 0x7f2f7c116500 2019-12-05T06:00:25.939 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:25.912+0000 7f2f7b7fe700 1 --2- 172.21.15.42:0/4251328766 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2f7c116500 0x7f2f7c07b050 secure :-1 s=THROTTLE_DONE pgs=1858 cs=0 l=1 rx=0x7f2f7c11b840 tx=0x7f2f6c00c450).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:25.939 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:25.916+0000 7f2f63fff700 1 --2- 172.21.15.42:0/4251328766 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2f5c03aee0 0x7f2f5c03d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:00:25.939 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:25.916+0000 7f2f63fff700 1 -- 172.21.15.42:0/4251328766 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9385+0+0 (secure 0 0 0) 0x7f2f6c04aa50 con 0x7f2f7c116500 2019-12-05T06:00:25.939 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:25.916+0000 7f2f7affd700 1 --2- 172.21.15.42:0/4251328766 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2f5c03aee0 0x7f2f5c03d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:00:25.939 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:25.916+0000 7f2f7affd700 1 --2- 172.21.15.42:0/4251328766 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2f5c03aee0 0x7f2f5c03d3f0 crc :-1 s=READY pgs=792 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:00:25.939 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:25.920+0000 7f2f815d6700 1 -- 172.21.15.42:0/4251328766 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f7c05d110 con 0x7f2f7c116500 2019-12-05T06:00:25.940 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:25.924+0000 7f2f7b7fe700 1 --2- 172.21.15.42:0/4251328766 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2f7c116500 0x7f2f7c07b050 secure :-1 s=THROTTLE_DONE pgs=1858 cs=0 l=1 rx=0x7f2f7c11b840 tx=0x7f2f6c00c450).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:25.940 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:25.924+0000 7f2f63fff700 1 -- 172.21.15.42:0/4251328766 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f2f6c031ad0 con 0x7f2f7c116500 2019-12-05T06:00:26.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:26.092+0000 7f2f815d6700 1 -- 172.21.15.42:0/4251328766 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2f7c05b8c0 con 0x7f2f7c116500 2019-12-05T06:00:26.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:26.092+0000 7f2f7b7fe700 1 --2- 172.21.15.42:0/4251328766 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2f7c116500 0x7f2f7c07b050 secure :-1 s=THROTTLE_DONE pgs=1858 cs=0 l=1 rx=0x7f2f7c11b840 tx=0x7f2f6c00c450).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:26.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:26.092+0000 7f2f63fff700 1 -- 172.21.15.42:0/4251328766 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 227 v227) v1 ==== 95+0+1771 (secure 0 0 0) 0x7f2f6c031d30 con 0x7f2f7c116500 2019-12-05T06:00:26.102 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 227 2019-12-05T06:00:26.102 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:00:26.102 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":227,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":227,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:00:19.604634+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":88,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9061},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9061":{"gid":9061,"name":"a","rank":0,"incarnation":224,"state":"up:active","state_seq":9,"addr":"172.21.15.42:6835/448869597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":448869597},{"type":"v1","addr":"172.21.15.42:6835","nonce":448869597}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:00:26.104 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:26.096+0000 7f2f815d6700 1 -- 172.21.15.42:0/4251328766 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2f5c03aee0 msgr2=0x7f2f5c03d3f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:00:26.104 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:26.096+0000 7f2f815d6700 1 --2- 172.21.15.42:0/4251328766 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2f5c03aee0 0x7f2f5c03d3f0 crc :-1 s=READY pgs=792 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:00:26.105 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:26.096+0000 7f2f815d6700 1 -- 172.21.15.42:0/4251328766 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2f7c116500 msgr2=0x7f2f7c07b050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:00:26.105 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:26.096+0000 7f2f815d6700 1 --2- 172.21.15.42:0/4251328766 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2f7c116500 0x7f2f7c07b050 secure :-1 s=READY pgs=1858 cs=0 l=1 rx=0x7f2f7c11b840 tx=0x7f2f6c00c450).stop 2019-12-05T06:00:26.105 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:26.096+0000 7f2f815d6700 1 -- 172.21.15.42:0/4251328766 shutdown_connections 2019-12-05T06:00:26.105 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:26.096+0000 7f2f815d6700 1 --2- 172.21.15.42:0/4251328766 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2f5c03aee0 0x7f2f5c03d3f0 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:00:26.108 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:26.096+0000 7f2f815d6700 1 --2- 172.21.15.42:0/4251328766 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2f7c08f1b0 0x7f2f7c08f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:00:26.108 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:26.096+0000 7f2f815d6700 1 --2- 172.21.15.42:0/4251328766 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2f7c116500 0x7f2f7c07b050 unknown :-1 s=CLOSED pgs=1858 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:00:26.108 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:26.096+0000 7f2f815d6700 1 --2- 172.21.15.42:0/4251328766 >> v2:172.21.15.40:3300/0 conn(0x7f2f7c120b70 0x7f2f7c120fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:00:26.109 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:26.096+0000 7f2f815d6700 1 -- 172.21.15.42:0/4251328766 shutdown_connections 2019-12-05T06:00:26.109 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:26.096+0000 7f2f815d6700 1 -- 172.21.15.42:0/4251328766 wait complete. 2019-12-05T06:00:26.109 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:26.096+0000 7f2f815d6700 1 -- 172.21.15.42:0/4251328766 >> 172.21.15.42:0/4251328766 conn(0x7f2f7c11c610 msgr2=0x7f2f7c07e560 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:00:26.110 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2019-12-05T06:00:28.111 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:00:28.326 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:28.208+0000 7f381dd2e700 1 Processor -- start 2019-12-05T06:00:28.326 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:28.208+0000 7f381dd2e700 1 -- start start 2019-12-05T06:00:28.326 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:28.208+0000 7f381dd2e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f38180a7740 0x7f381808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:00:28.326 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:28.208+0000 7f381dd2e700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f381809fae0 0x7f381807eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:00:28.327 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:28.208+0000 7f381dd2e700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38180a0710 con 0x7f381807e720 2019-12-05T06:00:28.327 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:28.208+0000 7f381dd2e700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38180a0b00 con 0x7f38180a7740 2019-12-05T06:00:28.327 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:28.208+0000 7f381dd2e700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f381808e5b0 con 0x7f381809fae0 2019-12-05T06:00:28.327 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:28.208+0000 7f3817fff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f38180a7740 0x7f381808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:00:28.327 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:28.208+0000 7f38177fe700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f381807e720 0x7f381809f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:46032/0 (socket says 172.21.15.42:46032) 2019-12-05T06:00:28.327 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:28.208+0000 7f38177fe700 1 -- 172.21.15.42:0/3775611179 learned_addr learned my addr 172.21.15.42:0/3775611179 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:00:28.327 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:28.208+0000 7f381cacc700 1 --2- 172.21.15.42:0/3775611179 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f381809fae0 0x7f381807eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:00:28.328 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:28.208+0000 7f3817fff700 1 -- 172.21.15.42:0/3775611179 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f381809fae0 msgr2=0x7f381807eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:00:28.328 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:28.208+0000 7f3817fff700 1 --2- 172.21.15.42:0/3775611179 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f381809fae0 0x7f381807eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:00:28.328 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:28.212+0000 7f3817fff700 1 -- 172.21.15.42:0/3775611179 >> v1:172.21.15.42:6789/0 conn(0x7f381807e720 legacy=0x7f381809f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:00:28.328 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:28.212+0000 7f3816ffd700 1 -- 172.21.15.42:0/3775611179 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 528864610 0 0) 0x7f38180a0710 con 0x7f381807e720 2019-12-05T06:00:28.328 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:28.212+0000 7f3817fff700 1 -- 172.21.15.42:0/3775611179 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f381807f0a0 con 0x7f38180a7740 2019-12-05T06:00:28.328 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:28.212+0000 7f3817fff700 1 --2- 172.21.15.42:0/3775611179 >> v2:172.21.15.42:3300/0 conn(0x7f38180a7740 0x7f381808df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f3808008b10 tx=0x7f380800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:28.328 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:28.212+0000 7f3817fff700 1 --2- 172.21.15.42:0/3775611179 >> v2:172.21.15.42:3300/0 conn(0x7f38180a7740 0x7f381808df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f3808008b10 tx=0x7f380800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:28.329 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:28.212+0000 7f3817fff700 1 --2- 172.21.15.42:0/3775611179 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f38180a7740 0x7f381808df80 secure :-1 s=READY pgs=1860 cs=0 l=1 rx=0x7f3808008b10 tx=0x7f380800fff0).ready entity=mon.1 client_cookie=988a064db922248f server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:00:28.329 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:28.212+0000 7f3817fff700 1 --2- 172.21.15.42:0/3775611179 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f38180a7740 0x7f381808df80 secure :-1 s=THROTTLE_DONE pgs=1860 cs=0 l=1 rx=0x7f3808008b10 tx=0x7f380800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:28.329 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:28.212+0000 7f3816ffd700 1 -- 172.21.15.42:0/3775611179 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3808010900 con 0x7f38180a7740 2019-12-05T06:00:28.329 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:28.212+0000 7f3817fff700 1 --2- 172.21.15.42:0/3775611179 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f38180a7740 0x7f381808df80 secure :-1 s=THROTTLE_DONE pgs=1860 cs=0 l=1 rx=0x7f3808008b10 tx=0x7f380800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:28.329 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:28.212+0000 7f3817fff700 1 --2- 172.21.15.42:0/3775611179 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f38180a7740 0x7f381808df80 secure :-1 s=THROTTLE_DONE pgs=1860 cs=0 l=1 rx=0x7f3808008b10 tx=0x7f380800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:28.329 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:28.212+0000 7f3816ffd700 1 -- 172.21.15.42:0/3775611179 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3808008b50 con 0x7f38180a7740 2019-12-05T06:00:28.330 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:28.212+0000 7f3816ffd700 1 -- 172.21.15.42:0/3775611179 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f38080110c0 con 0x7f38180a7740 2019-12-05T06:00:28.330 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:28.212+0000 7f381dd2e700 1 -- 172.21.15.42:0/3775611179 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f38180a7740 msgr2=0x7f381808df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:00:28.330 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:28.212+0000 7f381dd2e700 1 --2- 172.21.15.42:0/3775611179 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f38180a7740 0x7f381808df80 secure :-1 s=READY pgs=1860 cs=0 l=1 rx=0x7f3808008b10 tx=0x7f380800fff0).stop 2019-12-05T06:00:28.330 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:28.212+0000 7f381dd2e700 1 -- 172.21.15.42:0/3775611179 shutdown_connections 2019-12-05T06:00:28.330 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:28.212+0000 7f381dd2e700 1 --2- 172.21.15.42:0/3775611179 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f381809fae0 0x7f381807eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:00:28.330 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:28.212+0000 7f381dd2e700 1 --2- 172.21.15.42:0/3775611179 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f38180a7740 0x7f381808df80 unknown :-1 s=CLOSED pgs=1860 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:00:28.330 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:28.212+0000 7f381dd2e700 1 -- 172.21.15.42:0/3775611179 shutdown_connections 2019-12-05T06:00:28.330 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:28.212+0000 7f381dd2e700 1 -- 172.21.15.42:0/3775611179 wait complete. 2019-12-05T06:00:28.331 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:28.212+0000 7f381dd2e700 1 -- 172.21.15.42:0/3775611179 >> 172.21.15.42:0/3775611179 conn(0x7f38180aabe0 msgr2=0x7f38180a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:00:28.333 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:28.324+0000 7f381dd2e700 1 Processor -- start 2019-12-05T06:00:28.333 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:28.324+0000 7f381dd2e700 1 -- start start 2019-12-05T06:00:28.333 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:28.324+0000 7f381dd2e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3818122650 0x7f3818122a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:00:28.334 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:28.324+0000 7f381dd2e700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3818121e50 con 0x7f38180a38c0 2019-12-05T06:00:28.334 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:28.324+0000 7f381dd2e700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f381806eed0 con 0x7f381806e2a0 2019-12-05T06:00:28.334 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:28.324+0000 7f381dd2e700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38180a3cf0 con 0x7f3818122650 2019-12-05T06:00:28.334 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:28.324+0000 7f38177fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3818122650 0x7f3818122a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:00:28.334 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:28.324+0000 7f38177fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3818122650 0x7f3818122a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:46002/0 (socket says 172.21.15.42:46002) 2019-12-05T06:00:28.334 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:28.324+0000 7f381cacc700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f381806e2a0 0x7f381806e6d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:46042/0 (socket says 172.21.15.42:46042) 2019-12-05T06:00:28.335 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:28.324+0000 7f38177fe700 1 -- 172.21.15.42:0/2939997428 learned_addr learned my addr 172.21.15.42:0/2939997428 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:00:28.335 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:28.324+0000 7f3814ff9700 1 -- 172.21.15.42:0/2939997428 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2462666088 0 0) 0x7f38080110c0 con 0x7f38180a38c0 2019-12-05T06:00:28.335 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:28.328+0000 7f3814ff9700 1 -- 172.21.15.42:0/2939997428 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37f80030b0 con 0x7f38180a38c0 2019-12-05T06:00:28.335 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:28.328+0000 7f38177fe700 1 -- 172.21.15.42:0/2939997428 >> v1:172.21.15.42:6789/0 conn(0x7f381806e2a0 legacy=0x7f381806e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:00:28.335 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:28.328+0000 7f3814ff9700 1 -- 172.21.15.42:0/2939997428 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1728738831 0 0) 0x7f381806eed0 con 0x7f381806e2a0 2019-12-05T06:00:28.335 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:28.328+0000 7f38177fe700 1 -- 172.21.15.42:0/2939997428 >> v1:172.21.15.40:6789/0 conn(0x7f38180a38c0 legacy=0x7f381811d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:00:28.335 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:28.328+0000 7f38177fe700 1 -- 172.21.15.42:0/2939997428 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3818119180 con 0x7f3818122650 2019-12-05T06:00:28.336 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:28.328+0000 7f38177fe700 1 --2- 172.21.15.42:0/2939997428 >> v2:172.21.15.42:3300/0 conn(0x7f3818122650 0x7f3818122a80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f380c0043a0 tx=0x7f380c0108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:28.336 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:28.328+0000 7f38177fe700 1 --2- 172.21.15.42:0/2939997428 >> v2:172.21.15.42:3300/0 conn(0x7f3818122650 0x7f3818122a80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f380c0043a0 tx=0x7f380c0108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:28.336 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:28.328+0000 7f38177fe700 1 --2- 172.21.15.42:0/2939997428 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3818122650 0x7f3818122a80 secure :-1 s=READY pgs=1862 cs=0 l=1 rx=0x7f380c0043a0 tx=0x7f380c0108f0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:00:28.336 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:28.328+0000 7f38177fe700 1 --2- 172.21.15.42:0/2939997428 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3818122650 0x7f3818122a80 secure :-1 s=THROTTLE_DONE pgs=1862 cs=0 l=1 rx=0x7f380c0043a0 tx=0x7f380c0108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:28.336 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:28.328+0000 7f3814ff9700 1 -- 172.21.15.42:0/2939997428 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f380c011710 con 0x7f3818122650 2019-12-05T06:00:28.336 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:28.328+0000 7f381dd2e700 1 -- 172.21.15.42:0/2939997428 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3818076520 con 0x7f3818122650 2019-12-05T06:00:28.336 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:28.328+0000 7f381dd2e700 1 -- 172.21.15.42:0/2939997428 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3818076aa0 con 0x7f3818122650 2019-12-05T06:00:28.337 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:28.328+0000 7f38177fe700 1 --2- 172.21.15.42:0/2939997428 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3818122650 0x7f3818122a80 secure :-1 s=THROTTLE_DONE pgs=1862 cs=0 l=1 rx=0x7f380c0043a0 tx=0x7f380c0108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:28.337 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:28.328+0000 7f38177fe700 1 --2- 172.21.15.42:0/2939997428 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3818122650 0x7f3818122a80 secure :-1 s=THROTTLE_DONE pgs=1862 cs=0 l=1 rx=0x7f380c0043a0 tx=0x7f380c0108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:28.337 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:28.328+0000 7f3814ff9700 1 -- 172.21.15.42:0/2939997428 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f380c011330 con 0x7f3818122650 2019-12-05T06:00:28.337 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:28.328+0000 7f3814ff9700 1 -- 172.21.15.42:0/2939997428 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f380c011cf0 con 0x7f3818122650 2019-12-05T06:00:28.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:28.328+0000 7f38177fe700 1 --2- 172.21.15.42:0/2939997428 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3818122650 0x7f3818122a80 secure :-1 s=THROTTLE_DONE pgs=1862 cs=0 l=1 rx=0x7f380c0043a0 tx=0x7f380c0108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:28.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:28.328+0000 7f3814ff9700 1 -- 172.21.15.42:0/2939997428 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f380c00b330 con 0x7f3818122650 2019-12-05T06:00:28.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:28.328+0000 7f38177fe700 1 --2- 172.21.15.42:0/2939997428 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3818122650 0x7f3818122a80 secure :-1 s=THROTTLE_DONE pgs=1862 cs=0 l=1 rx=0x7f380c0043a0 tx=0x7f380c0108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:28.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:28.332+0000 7f3814ff9700 1 --2- 172.21.15.42:0/2939997428 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f37f803b1b0 0x7f37f803d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:00:28.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:28.332+0000 7f3814ff9700 1 -- 172.21.15.42:0/2939997428 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9385+0+0 (secure 0 0 0) 0x7f380c04b160 con 0x7f3818122650 2019-12-05T06:00:28.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:28.332+0000 7f3817fff700 1 --2- 172.21.15.42:0/2939997428 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f37f803b1b0 0x7f37f803d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:00:28.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:28.332+0000 7f3817fff700 1 --2- 172.21.15.42:0/2939997428 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f37f803b1b0 0x7f37f803d6c0 crc :-1 s=READY pgs=794 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:00:28.375 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:28.372+0000 7f381dd2e700 1 -- 172.21.15.42:0/2939997428 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3818131700 con 0x7f3818122650 2019-12-05T06:00:28.383 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:28.376+0000 7f38177fe700 1 --2- 172.21.15.42:0/2939997428 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3818122650 0x7f3818122a80 secure :-1 s=THROTTLE_DONE pgs=1862 cs=0 l=1 rx=0x7f380c0043a0 tx=0x7f380c0108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:28.383 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:28.380+0000 7f3814ff9700 1 -- 172.21.15.42:0/2939997428 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f3818119180 con 0x7f3818122650 2019-12-05T06:00:28.552 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:28.548+0000 7f381dd2e700 1 -- 172.21.15.42:0/2939997428 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f381805d110 con 0x7f3818122650 2019-12-05T06:00:28.553 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:28.548+0000 7f38177fe700 1 --2- 172.21.15.42:0/2939997428 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3818122650 0x7f3818122a80 secure :-1 s=THROTTLE_DONE pgs=1862 cs=0 l=1 rx=0x7f380c0043a0 tx=0x7f380c0108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:28.553 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:28.548+0000 7f3814ff9700 1 -- 172.21.15.42:0/2939997428 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 228 v228) v1 ==== 95+0+2108 (secure 0 0 0) 0x7f380c032180 con 0x7f3818122650 2019-12-05T06:00:28.555 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:00:28.555 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":228,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6653,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/2780168808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2780168808},{"type":"v1","addr":"172.21.15.40:6833","nonce":2780168808}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":228}],"filesystems":[{"mdsmap":{"epoch":227,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:00:19.604634+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":88,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9061},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9061":{"gid":9061,"name":"a","rank":0,"incarnation":224,"state":"up:active","state_seq":9,"addr":"172.21.15.42:6835/448869597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":448869597},{"type":"v1","addr":"172.21.15.42:6835","nonce":448869597}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:00:28.555 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 228 2019-12-05T06:00:28.561 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:28.552+0000 7f381dd2e700 1 -- 172.21.15.42:0/2939997428 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f37f803b1b0 msgr2=0x7f37f803d6c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:00:28.561 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:28.552+0000 7f381dd2e700 1 --2- 172.21.15.42:0/2939997428 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f37f803b1b0 0x7f37f803d6c0 crc :-1 s=READY pgs=794 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:00:28.561 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:28.552+0000 7f381dd2e700 1 -- 172.21.15.42:0/2939997428 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3818122650 msgr2=0x7f3818122a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:00:28.561 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:28.552+0000 7f381dd2e700 1 --2- 172.21.15.42:0/2939997428 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3818122650 0x7f3818122a80 secure :-1 s=READY pgs=1862 cs=0 l=1 rx=0x7f380c0043a0 tx=0x7f380c0108f0).stop 2019-12-05T06:00:28.561 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:28.552+0000 7f381dd2e700 1 -- 172.21.15.42:0/2939997428 shutdown_connections 2019-12-05T06:00:28.562 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:28.552+0000 7f381dd2e700 1 --2- 172.21.15.42:0/2939997428 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f37f803b1b0 0x7f37f803d6c0 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:00:28.562 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:28.552+0000 7f381dd2e700 1 --2- 172.21.15.42:0/2939997428 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3818122650 0x7f3818122a80 unknown :-1 s=CLOSED pgs=1862 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:00:28.562 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:28.552+0000 7f381dd2e700 1 -- 172.21.15.42:0/2939997428 shutdown_connections 2019-12-05T06:00:28.562 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:28.552+0000 7f381dd2e700 1 -- 172.21.15.42:0/2939997428 wait complete. 2019-12-05T06:00:28.562 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:28.552+0000 7f381dd2e700 1 -- 172.21.15.42:0/2939997428 >> 172.21.15.42:0/2939997428 conn(0x7f38180a7740 msgr2=0x7f3818059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:00:28.567 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b reported in up:standby state 2019-12-05T06:00:28.568 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 17 secs before thrashing 2019-12-05T06:00:43.084 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:00:43.090 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:00:45.569 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:00:45.674 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:45.663+0000 7fe5329e0700 1 Processor -- start 2019-12-05T06:00:45.675 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:45.663+0000 7fe5329e0700 1 -- start start 2019-12-05T06:00:45.675 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:45.663+0000 7fe5329e0700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe52c0a7740 0x7fe52c08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:00:45.675 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:45.667+0000 7fe5329e0700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe52c09fae0 0x7fe52c07eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:00:45.675 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:45.667+0000 7fe5329e0700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe52c0a0710 con 0x7fe52c07e720 2019-12-05T06:00:45.675 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:45.667+0000 7fe5329e0700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe52c0a0b00 con 0x7fe52c0a7740 2019-12-05T06:00:45.675 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:45.667+0000 7fe5329e0700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe52c08e5b0 con 0x7fe52c09fae0 2019-12-05T06:00:45.676 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:45.667+0000 7fe53177e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe52c09fae0 0x7fe52c07eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:00:45.676 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:45.667+0000 7fe530f7d700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe52c0a7740 0x7fe52c08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:00:45.676 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:45.667+0000 7fe53177e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe52c09fae0 0x7fe52c07eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:46094/0 (socket says 172.21.15.42:46094) 2019-12-05T06:00:45.676 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:45.667+0000 7fe53177e700 1 -- 172.21.15.42:0/2423290907 learned_addr learned my addr 172.21.15.42:0/2423290907 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:00:45.676 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:45.667+0000 7fe53177e700 1 -- 172.21.15.42:0/2423290907 >> v2:172.21.15.40:3300/0 conn(0x7fe52c0a7740 msgr2=0x7fe52c08df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:00:45.676 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:45.667+0000 7fe53177e700 1 --2- 172.21.15.42:0/2423290907 >> v2:172.21.15.40:3300/0 conn(0x7fe52c0a7740 0x7fe52c08df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:00:45.676 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:45.667+0000 7fe53177e700 1 -- 172.21.15.42:0/2423290907 >> v1:172.21.15.40:6789/0 conn(0x7fe52c07e720 legacy=0x7fe52c09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:00:45.677 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:45.667+0000 7fe53177e700 1 -- 172.21.15.42:0/2423290907 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe52c07f0a0 con 0x7fe52c09fae0 2019-12-05T06:00:45.677 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:45.667+0000 7fe53177e700 1 --2- 172.21.15.42:0/2423290907 >> v2:172.21.15.42:3300/0 conn(0x7fe52c09fae0 0x7fe52c07eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fe524002f30 tx=0x7fe52400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:45.677 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:45.667+0000 7fe53177e700 1 --2- 172.21.15.42:0/2423290907 >> v2:172.21.15.42:3300/0 conn(0x7fe52c09fae0 0x7fe52c07eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fe524002f30 tx=0x7fe52400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:45.677 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:45.667+0000 7fe53177e700 1 --2- 172.21.15.42:0/2423290907 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe52c09fae0 0x7fe52c07eb50 secure :-1 s=READY pgs=1863 cs=0 l=1 rx=0x7fe524002f30 tx=0x7fe52400fd30).ready entity=mon.1 client_cookie=8f79fa9d648368be server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:00:45.677 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:45.667+0000 7fe53177e700 1 --2- 172.21.15.42:0/2423290907 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe52c09fae0 0x7fe52c07eb50 secure :-1 s=THROTTLE_DONE pgs=1863 cs=0 l=1 rx=0x7fe524002f30 tx=0x7fe52400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:45.677 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:45.667+0000 7fe53177e700 1 --2- 172.21.15.42:0/2423290907 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe52c09fae0 0x7fe52c07eb50 secure :-1 s=THROTTLE_DONE pgs=1863 cs=0 l=1 rx=0x7fe524002f30 tx=0x7fe52400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:45.678 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:45.667+0000 7fe52b7fe700 1 -- 172.21.15.42:0/2423290907 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe524008c90 con 0x7fe52c09fae0 2019-12-05T06:00:45.678 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:45.667+0000 7fe53177e700 1 --2- 172.21.15.42:0/2423290907 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe52c09fae0 0x7fe52c07eb50 secure :-1 s=THROTTLE_DONE pgs=1863 cs=0 l=1 rx=0x7fe524002f30 tx=0x7fe52400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:45.678 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:45.667+0000 7fe52b7fe700 1 -- 172.21.15.42:0/2423290907 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5240088b0 con 0x7fe52c09fae0 2019-12-05T06:00:45.678 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:45.667+0000 7fe52b7fe700 1 -- 172.21.15.42:0/2423290907 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe524011ad0 con 0x7fe52c09fae0 2019-12-05T06:00:45.678 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:45.667+0000 7fe5329e0700 1 -- 172.21.15.42:0/2423290907 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe52c09fae0 msgr2=0x7fe52c07eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:00:45.678 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:45.667+0000 7fe5329e0700 1 --2- 172.21.15.42:0/2423290907 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe52c09fae0 0x7fe52c07eb50 secure :-1 s=READY pgs=1863 cs=0 l=1 rx=0x7fe524002f30 tx=0x7fe52400fd30).stop 2019-12-05T06:00:45.679 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:45.667+0000 7fe5329e0700 1 -- 172.21.15.42:0/2423290907 shutdown_connections 2019-12-05T06:00:45.679 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:45.667+0000 7fe5329e0700 1 --2- 172.21.15.42:0/2423290907 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe52c09fae0 0x7fe52c07eb50 unknown :-1 s=CLOSED pgs=1863 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:00:45.679 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:45.667+0000 7fe5329e0700 1 --2- 172.21.15.42:0/2423290907 >> v2:172.21.15.40:3300/0 conn(0x7fe52c0a7740 0x7fe52c08df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:00:45.679 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:45.667+0000 7fe5329e0700 1 -- 172.21.15.42:0/2423290907 shutdown_connections 2019-12-05T06:00:45.679 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:45.667+0000 7fe5329e0700 1 -- 172.21.15.42:0/2423290907 wait complete. 2019-12-05T06:00:45.679 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:45.667+0000 7fe5329e0700 1 -- 172.21.15.42:0/2423290907 >> 172.21.15.42:0/2423290907 conn(0x7fe52c0aabe0 msgr2=0x7fe52c0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:00:45.682 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:45.671+0000 7fe5329e0700 1 Processor -- start 2019-12-05T06:00:45.682 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:45.671+0000 7fe5329e0700 1 -- start start 2019-12-05T06:00:45.682 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:45.671+0000 7fe5329e0700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe52c08f150 0x7fe52c0a4a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:00:45.682 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:45.671+0000 7fe5329e0700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe52c142960 0x7fe52c133d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:00:45.683 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:45.671+0000 7fe5329e0700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe52c0a5500 con 0x7fe52c142530 2019-12-05T06:00:45.683 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:45.671+0000 7fe5329e0700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe52c0a5980 con 0x7fe52c142960 2019-12-05T06:00:45.683 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:45.671+0000 7fe5329e0700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe52c0a5c20 con 0x7fe52c08f150 2019-12-05T06:00:45.683 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:45.671+0000 7fe530f7d700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe52c08f150 0x7fe52c0a4a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:00:45.683 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:45.671+0000 7fe53177e700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe52c142960 0x7fe52c133d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:00:45.683 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:45.671+0000 7fe530f7d700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe52c08f150 0x7fe52c0a4a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:46564/0 (socket says 172.21.15.42:46564) 2019-12-05T06:00:45.683 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:45.671+0000 7fe530f7d700 1 -- 172.21.15.42:0/2942306374 learned_addr learned my addr 172.21.15.42:0/2942306374 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:00:45.684 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:45.671+0000 7fe52bfff700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fe52c142530 0x7fe52c07b830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:55578/0 (socket says 172.21.15.42:55578) 2019-12-05T06:00:45.684 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:45.675+0000 7fe530f7d700 1 -- 172.21.15.42:0/2942306374 >> v2:172.21.15.40:3300/0 conn(0x7fe52c142960 msgr2=0x7fe52c133d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:00:45.684 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:45.675+0000 7fe530f7d700 1 --2- 172.21.15.42:0/2942306374 >> v2:172.21.15.40:3300/0 conn(0x7fe52c142960 0x7fe52c133d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:00:45.684 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:45.675+0000 7fe530f7d700 1 -- 172.21.15.42:0/2942306374 >> v1:172.21.15.40:6789/0 conn(0x7fe52c142530 legacy=0x7fe52c07b830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:00:45.684 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:45.675+0000 7fe530f7d700 1 -- 172.21.15.42:0/2942306374 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe52c0a6160 con 0x7fe52c08f150 2019-12-05T06:00:45.684 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:45.675+0000 7fe53177e700 1 --2- 172.21.15.42:0/2942306374 >> v2:172.21.15.40:3300/0 conn(0x7fe52c142960 0x7fe52c133d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:00:45.684 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:45.675+0000 7fe530f7d700 1 --2- 172.21.15.42:0/2942306374 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe52c08f150 0x7fe52c0a4a50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fe51c0033f0 tx=0x7fe51c010ee0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:45.685 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:45.675+0000 7fe530f7d700 1 --2- 172.21.15.42:0/2942306374 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe52c08f150 0x7fe52c0a4a50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fe51c0033f0 tx=0x7fe51c010ee0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:45.685 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:45.675+0000 7fe530f7d700 1 --2- 172.21.15.42:0/2942306374 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe52c08f150 0x7fe52c0a4a50 secure :-1 s=READY pgs=415 cs=0 l=1 rx=0x7fe51c0033f0 tx=0x7fe51c010ee0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:00:45.685 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:45.675+0000 7fe530f7d700 1 --2- 172.21.15.42:0/2942306374 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe52c08f150 0x7fe52c0a4a50 secure :-1 s=THROTTLE_DONE pgs=415 cs=0 l=1 rx=0x7fe51c0033f0 tx=0x7fe51c010ee0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:45.685 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:45.675+0000 7fe5297fa700 1 -- 172.21.15.42:0/2942306374 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe51c011cc0 con 0x7fe52c08f150 2019-12-05T06:00:45.685 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:45.675+0000 7fe530f7d700 1 --2- 172.21.15.42:0/2942306374 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe52c08f150 0x7fe52c0a4a50 secure :-1 s=THROTTLE_DONE pgs=415 cs=0 l=1 rx=0x7fe51c0033f0 tx=0x7fe51c010ee0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:45.685 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:45.675+0000 7fe5297fa700 1 -- 172.21.15.42:0/2942306374 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe51c0118e0 con 0x7fe52c08f150 2019-12-05T06:00:45.685 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:45.675+0000 7fe530f7d700 1 --2- 172.21.15.42:0/2942306374 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe52c08f150 0x7fe52c0a4a50 secure :-1 s=THROTTLE_DONE pgs=415 cs=0 l=1 rx=0x7fe51c0033f0 tx=0x7fe51c010ee0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:45.686 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:45.675+0000 7fe5329e0700 1 -- 172.21.15.42:0/2942306374 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe52c0a4fa0 con 0x7fe52c08f150 2019-12-05T06:00:45.686 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:45.675+0000 7fe5297fa700 1 -- 172.21.15.42:0/2942306374 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe51c005ca0 con 0x7fe52c08f150 2019-12-05T06:00:45.686 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:45.675+0000 7fe5329e0700 1 -- 172.21.15.42:0/2942306374 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe52c142d90 con 0x7fe52c08f150 2019-12-05T06:00:45.686 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:45.675+0000 7fe530f7d700 1 --2- 172.21.15.42:0/2942306374 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe52c08f150 0x7fe52c0a4a50 secure :-1 s=THROTTLE_DONE pgs=415 cs=0 l=1 rx=0x7fe51c0033f0 tx=0x7fe51c010ee0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:45.687 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:45.675+0000 7fe5297fa700 1 -- 172.21.15.42:0/2942306374 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fe51c007180 con 0x7fe52c08f150 2019-12-05T06:00:45.687 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:45.675+0000 7fe530f7d700 1 --2- 172.21.15.42:0/2942306374 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe52c08f150 0x7fe52c0a4a50 secure :-1 s=THROTTLE_DONE pgs=415 cs=0 l=1 rx=0x7fe51c0033f0 tx=0x7fe51c010ee0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:45.687 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:45.679+0000 7fe5297fa700 1 --2- 172.21.15.42:0/2942306374 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe51003aca0 0x7fe51003d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:00:45.687 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:45.679+0000 7fe5297fa700 1 -- 172.21.15.42:0/2942306374 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9385+0+0 (secure 0 0 0) 0x7fe51c04afa0 con 0x7fe52c08f150 2019-12-05T06:00:45.688 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:45.679+0000 7fe52bfff700 1 --2- 172.21.15.42:0/2942306374 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe51003aca0 0x7fe51003d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:00:45.688 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:45.679+0000 7fe52bfff700 1 --2- 172.21.15.42:0/2942306374 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe51003aca0 0x7fe51003d1b0 crc :-1 s=READY pgs=796 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:00:45.688 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:45.679+0000 7fe5329e0700 1 -- 172.21.15.42:0/2942306374 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe52c05d110 con 0x7fe52c08f150 2019-12-05T06:00:45.695 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:45.687+0000 7fe530f7d700 1 --2- 172.21.15.42:0/2942306374 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe52c08f150 0x7fe52c0a4a50 secure :-1 s=THROTTLE_DONE pgs=415 cs=0 l=1 rx=0x7fe51c0033f0 tx=0x7fe51c010ee0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:45.695 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:45.687+0000 7fe5297fa700 1 -- 172.21.15.42:0/2942306374 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fe51c032000 con 0x7fe52c08f150 2019-12-05T06:00:45.881 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:45.875+0000 7fe5329e0700 1 -- 172.21.15.42:0/2942306374 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe52c05b8c0 con 0x7fe52c08f150 2019-12-05T06:00:45.881 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:45.875+0000 7fe530f7d700 1 --2- 172.21.15.42:0/2942306374 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe52c08f150 0x7fe52c0a4a50 secure :-1 s=THROTTLE_DONE pgs=415 cs=0 l=1 rx=0x7fe51c0033f0 tx=0x7fe51c010ee0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:45.882 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:45.875+0000 7fe5297fa700 1 -- 172.21.15.42:0/2942306374 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 228 v228) v1 ==== 95+0+2108 (secure 0 0 0) 0x7fe51c032260 con 0x7fe52c08f150 2019-12-05T06:00:45.884 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 228 2019-12-05T06:00:45.884 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:00:45.884 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":228,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6653,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/2780168808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2780168808},{"type":"v1","addr":"172.21.15.40:6833","nonce":2780168808}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":228}],"filesystems":[{"mdsmap":{"epoch":227,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:00:19.604634+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":88,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9061},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9061":{"gid":9061,"name":"a","rank":0,"incarnation":224,"state":"up:active","state_seq":9,"addr":"172.21.15.42:6835/448869597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":448869597},{"type":"v1","addr":"172.21.15.42:6835","nonce":448869597}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:00:45.886 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:45.879+0000 7fe5329e0700 1 -- 172.21.15.42:0/2942306374 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe51003aca0 msgr2=0x7fe51003d1b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:00:45.887 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:45.879+0000 7fe5329e0700 1 --2- 172.21.15.42:0/2942306374 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe51003aca0 0x7fe51003d1b0 crc :-1 s=READY pgs=796 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:00:45.887 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:45.879+0000 7fe5329e0700 1 -- 172.21.15.42:0/2942306374 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe52c08f150 msgr2=0x7fe52c0a4a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:00:45.887 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:45.879+0000 7fe5329e0700 1 --2- 172.21.15.42:0/2942306374 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe52c08f150 0x7fe52c0a4a50 secure :-1 s=READY pgs=415 cs=0 l=1 rx=0x7fe51c0033f0 tx=0x7fe51c010ee0).stop 2019-12-05T06:00:45.887 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:45.879+0000 7fe5329e0700 1 -- 172.21.15.42:0/2942306374 shutdown_connections 2019-12-05T06:00:45.887 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:45.879+0000 7fe5329e0700 1 --2- 172.21.15.42:0/2942306374 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe51003aca0 0x7fe51003d1b0 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:00:45.887 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:45.879+0000 7fe5329e0700 1 --2- 172.21.15.42:0/2942306374 >> v2:172.21.15.40:3300/0 conn(0x7fe52c142960 0x7fe52c133d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:00:45.887 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:45.879+0000 7fe5329e0700 1 --2- 172.21.15.42:0/2942306374 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe52c08f150 0x7fe52c0a4a50 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:00:45.888 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:45.879+0000 7fe5329e0700 1 -- 172.21.15.42:0/2942306374 shutdown_connections 2019-12-05T06:00:45.888 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:45.883+0000 7fe5329e0700 1 -- 172.21.15.42:0/2942306374 wait complete. 2019-12-05T06:00:45.888 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:45.883+0000 7fe5329e0700 1 -- 172.21.15.42:0/2942306374 >> 172.21.15.42:0/2942306374 conn(0x7fe52c0a7f90 msgr2=0x7fe52c118730 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:00:45.897 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.a (rank=0) 2019-12-05T06:00:45.898 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2019-12-05T06:00:45.898 INFO:teuthology.orchestra.run:waiting for 300 2019-12-05T06:00:45.908 INFO:tasks.ceph.mds.a:Stopped 2019-12-05T06:00:45.908 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:00:46.078 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:46.067+0000 7f3e442d3700 1 Processor -- start 2019-12-05T06:00:46.079 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:46.067+0000 7f3e442d3700 1 -- start start 2019-12-05T06:00:46.079 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:46.067+0000 7f3e442d3700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3e3c0a7740 0x7f3e3c08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:00:46.079 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:46.067+0000 7f3e442d3700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f3e3c09fae0 0x7f3e3c07eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:00:46.079 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:46.067+0000 7f3e442d3700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e3c0a0710 con 0x7f3e3c07e720 2019-12-05T06:00:46.079 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:46.067+0000 7f3e442d3700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e3c0a0b00 con 0x7f3e3c09fae0 2019-12-05T06:00:46.079 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:46.067+0000 7f3e442d3700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e3c08e5b0 con 0x7f3e3c0a7740 2019-12-05T06:00:46.080 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:46.067+0000 7f3e4206f700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f3e3c07e720 0x7f3e3c09f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:46150/0 (socket says 172.21.15.42:46150) 2019-12-05T06:00:46.080 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:46.067+0000 7f3e4206f700 1 -- 172.21.15.42:0/2211782411 learned_addr learned my addr 172.21.15.42:0/2211782411 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:00:46.080 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:46.067+0000 7f3e42870700 1 --2- 172.21.15.42:0/2211782411 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3e3c0a7740 0x7f3e3c08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:00:46.080 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:46.067+0000 7f3e43071700 1 --2- 172.21.15.42:0/2211782411 >> v2:172.21.15.40:3300/0 conn(0x7f3e3c09fae0 0x7f3e3c07eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:00:46.080 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:46.071+0000 7f3e4186e700 1 -- 172.21.15.42:0/2211782411 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2361693439 0 0) 0x7f3e3c0a0710 con 0x7f3e3c07e720 2019-12-05T06:00:46.080 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:46.071+0000 7f3e4186e700 1 -- 172.21.15.42:0/2211782411 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e240030b0 con 0x7f3e3c07e720 2019-12-05T06:00:46.081 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:46.071+0000 7f3e42870700 1 -- 172.21.15.42:0/2211782411 >> v2:172.21.15.40:3300/0 conn(0x7f3e3c09fae0 msgr2=0x7f3e3c07eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:00:46.081 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:46.071+0000 7f3e42870700 1 --2- 172.21.15.42:0/2211782411 >> v2:172.21.15.40:3300/0 conn(0x7f3e3c09fae0 0x7f3e3c07eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:00:46.081 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:46.071+0000 7f3e42870700 1 -- 172.21.15.42:0/2211782411 >> v1:172.21.15.42:6789/0 conn(0x7f3e3c07e720 legacy=0x7f3e3c09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:00:46.081 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:46.071+0000 7f3e42870700 1 -- 172.21.15.42:0/2211782411 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e3c07f0a0 con 0x7f3e3c0a7740 2019-12-05T06:00:46.081 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:46.071+0000 7f3e43071700 1 --2- 172.21.15.42:0/2211782411 >> v2:172.21.15.40:3300/0 conn(0x7f3e3c09fae0 0x7f3e3c07eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:00:46.081 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:46.071+0000 7f3e42870700 1 --2- 172.21.15.42:0/2211782411 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3e3c0a7740 0x7f3e3c08df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f3e34008b10 tx=0x7f3e3400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:46.081 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:46.071+0000 7f3e42870700 1 --2- 172.21.15.42:0/2211782411 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3e3c0a7740 0x7f3e3c08df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f3e34008b10 tx=0x7f3e3400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:46.082 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:46.071+0000 7f3e42870700 1 --2- 172.21.15.42:0/2211782411 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3e3c0a7740 0x7f3e3c08df80 secure :-1 s=READY pgs=416 cs=0 l=1 rx=0x7f3e34008b10 tx=0x7f3e3400fff0).ready entity=mon.2 client_cookie=6fd7d5be46c5ff89 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:00:46.082 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:46.071+0000 7f3e42870700 1 --2- 172.21.15.42:0/2211782411 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3e3c0a7740 0x7f3e3c08df80 secure :-1 s=THROTTLE_DONE pgs=416 cs=0 l=1 rx=0x7f3e34008b10 tx=0x7f3e3400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:46.082 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:46.071+0000 7f3e4186e700 1 -- 172.21.15.42:0/2211782411 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3e34010900 con 0x7f3e3c0a7740 2019-12-05T06:00:46.082 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:46.071+0000 7f3e42870700 1 --2- 172.21.15.42:0/2211782411 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3e3c0a7740 0x7f3e3c08df80 secure :-1 s=THROTTLE_DONE pgs=416 cs=0 l=1 rx=0x7f3e34008b10 tx=0x7f3e3400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:46.082 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:46.071+0000 7f3e4186e700 1 -- 172.21.15.42:0/2211782411 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e34008b50 con 0x7f3e3c0a7740 2019-12-05T06:00:46.082 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:46.071+0000 7f3e42870700 1 --2- 172.21.15.42:0/2211782411 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3e3c0a7740 0x7f3e3c08df80 secure :-1 s=THROTTLE_DONE pgs=416 cs=0 l=1 rx=0x7f3e34008b10 tx=0x7f3e3400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:46.082 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:46.071+0000 7f3e4186e700 1 -- 172.21.15.42:0/2211782411 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3e340110c0 con 0x7f3e3c0a7740 2019-12-05T06:00:46.083 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:46.071+0000 7f3e442d3700 1 -- 172.21.15.42:0/2211782411 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3e3c0a7740 msgr2=0x7f3e3c08df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:00:46.083 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:46.071+0000 7f3e442d3700 1 --2- 172.21.15.42:0/2211782411 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3e3c0a7740 0x7f3e3c08df80 secure :-1 s=READY pgs=416 cs=0 l=1 rx=0x7f3e34008b10 tx=0x7f3e3400fff0).stop 2019-12-05T06:00:46.083 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:46.071+0000 7f3e442d3700 1 -- 172.21.15.42:0/2211782411 shutdown_connections 2019-12-05T06:00:46.083 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:46.071+0000 7f3e442d3700 1 --2- 172.21.15.42:0/2211782411 >> v2:172.21.15.40:3300/0 conn(0x7f3e3c09fae0 0x7f3e3c07eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:00:46.083 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:46.071+0000 7f3e442d3700 1 --2- 172.21.15.42:0/2211782411 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3e3c0a7740 0x7f3e3c08df80 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:00:46.083 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:46.071+0000 7f3e442d3700 1 -- 172.21.15.42:0/2211782411 shutdown_connections 2019-12-05T06:00:46.083 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:46.071+0000 7f3e442d3700 1 -- 172.21.15.42:0/2211782411 wait complete. 2019-12-05T06:00:46.084 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:46.071+0000 7f3e442d3700 1 -- 172.21.15.42:0/2211782411 >> 172.21.15.42:0/2211782411 conn(0x7f3e3c0aabe0 msgr2=0x7f3e3c0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:00:46.086 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:46.075+0000 7f3e442d3700 1 Processor -- start 2019-12-05T06:00:46.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:46.075+0000 7f3e442d3700 1 -- start start 2019-12-05T06:00:46.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:46.075+0000 7f3e442d3700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3e3c11a960 0x7f3e3c116360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:00:46.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:46.075+0000 7f3e442d3700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3e3c0a23b0 0x7f3e3c133ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:00:46.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:46.075+0000 7f3e442d3700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e3c134030 con 0x7f3e3c11b0d0 2019-12-05T06:00:46.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:46.075+0000 7f3e442d3700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e3c11b7b0 con 0x7f3e3c11a960 2019-12-05T06:00:46.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:46.075+0000 7f3e442d3700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e3c11ba50 con 0x7f3e3c0a23b0 2019-12-05T06:00:46.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:46.075+0000 7f3e42870700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3e3c11a960 0x7f3e3c116360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:00:46.088 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:46.075+0000 7f3e42870700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3e3c11a960 0x7f3e3c116360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:46120/0 (socket says 172.21.15.42:46120) 2019-12-05T06:00:46.088 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:46.075+0000 7f3e42870700 1 -- 172.21.15.42:0/1154540183 learned_addr learned my addr 172.21.15.42:0/1154540183 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:00:46.088 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:46.075+0000 7f3e43071700 1 --2- 172.21.15.42:0/1154540183 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3e3c0a23b0 0x7f3e3c133ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:00:46.088 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:46.075+0000 7f3e3b7fe700 1 -- 172.21.15.42:0/1154540183 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2978602662 0 0) 0x7f3e3c134030 con 0x7f3e3c11b0d0 2019-12-05T06:00:46.088 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:46.075+0000 7f3e3b7fe700 1 -- 172.21.15.42:0/1154540183 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e1c0030b0 con 0x7f3e3c11b0d0 2019-12-05T06:00:46.088 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:46.075+0000 7f3e42870700 1 -- 172.21.15.42:0/1154540183 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3e3c0a23b0 msgr2=0x7f3e3c133ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:00:46.089 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:46.075+0000 7f3e42870700 1 --2- 172.21.15.42:0/1154540183 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3e3c0a23b0 0x7f3e3c133ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:00:46.089 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:46.075+0000 7f3e42870700 1 -- 172.21.15.42:0/1154540183 >> v1:172.21.15.40:6789/0 conn(0x7f3e3c11b0d0 legacy=0x7f3e3c0a1bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:00:46.089 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:46.075+0000 7f3e42870700 1 -- 172.21.15.42:0/1154540183 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e3c11b500 con 0x7f3e3c11a960 2019-12-05T06:00:46.089 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:46.075+0000 7f3e42870700 1 --2- 172.21.15.42:0/1154540183 >> v2:172.21.15.42:3300/0 conn(0x7f3e3c11a960 0x7f3e3c116360 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f3e3c11b970 tx=0x7f3e34007960).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:46.089 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:46.075+0000 7f3e42870700 1 --2- 172.21.15.42:0/1154540183 >> v2:172.21.15.42:3300/0 conn(0x7f3e3c11a960 0x7f3e3c116360 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f3e3c11b970 tx=0x7f3e34007960).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:46.089 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:46.075+0000 7f3e42870700 1 --2- 172.21.15.42:0/1154540183 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3e3c11a960 0x7f3e3c116360 secure :-1 s=READY pgs=1865 cs=0 l=1 rx=0x7f3e3c11b970 tx=0x7f3e34007960).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:00:46.089 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:46.075+0000 7f3e42870700 1 --2- 172.21.15.42:0/1154540183 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3e3c11a960 0x7f3e3c116360 secure :-1 s=THROTTLE_DONE pgs=1865 cs=0 l=1 rx=0x7f3e3c11b970 tx=0x7f3e34007960).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:46.090 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:46.075+0000 7f3e3b7fe700 1 -- 172.21.15.42:0/1154540183 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3e34005ab0 con 0x7f3e3c11a960 2019-12-05T06:00:46.090 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:46.075+0000 7f3e442d3700 1 -- 172.21.15.42:0/1154540183 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e3c134260 con 0x7f3e3c11a960 2019-12-05T06:00:46.090 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:46.075+0000 7f3e42870700 1 --2- 172.21.15.42:0/1154540183 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3e3c11a960 0x7f3e3c116360 secure :-1 s=THROTTLE_DONE pgs=1865 cs=0 l=1 rx=0x7f3e3c11b970 tx=0x7f3e34007960).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:46.090 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:46.075+0000 7f3e3b7fe700 1 -- 172.21.15.42:0/1154540183 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e34010b00 con 0x7f3e3c11a960 2019-12-05T06:00:46.091 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:46.075+0000 7f3e42870700 1 --2- 172.21.15.42:0/1154540183 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3e3c11a960 0x7f3e3c116360 secure :-1 s=THROTTLE_DONE pgs=1865 cs=0 l=1 rx=0x7f3e3c11b970 tx=0x7f3e34007960).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:46.091 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:46.075+0000 7f3e3b7fe700 1 -- 172.21.15.42:0/1154540183 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3e3400f470 con 0x7f3e3c11a960 2019-12-05T06:00:46.091 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:46.075+0000 7f3e442d3700 1 -- 172.21.15.42:0/1154540183 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e3c077300 con 0x7f3e3c11a960 2019-12-05T06:00:46.091 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:46.079+0000 7f3e42870700 1 --2- 172.21.15.42:0/1154540183 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3e3c11a960 0x7f3e3c116360 secure :-1 s=THROTTLE_DONE pgs=1865 cs=0 l=1 rx=0x7f3e3c11b970 tx=0x7f3e34007960).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:46.091 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:46.079+0000 7f3e3b7fe700 1 -- 172.21.15.42:0/1154540183 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f3e3400f710 con 0x7f3e3c11a960 2019-12-05T06:00:46.092 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:46.079+0000 7f3e42870700 1 --2- 172.21.15.42:0/1154540183 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3e3c11a960 0x7f3e3c116360 secure :-1 s=THROTTLE_DONE pgs=1865 cs=0 l=1 rx=0x7f3e3c11b970 tx=0x7f3e34007960).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:46.092 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:46.079+0000 7f3e3b7fe700 1 --2- 172.21.15.42:0/1154540183 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3e1c03b250 0x7f3e1c03d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:00:46.092 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:46.079+0000 7f3e3b7fe700 1 -- 172.21.15.42:0/1154540183 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9385+0+0 (secure 0 0 0) 0x7f3e3408efc0 con 0x7f3e3c11a960 2019-12-05T06:00:46.092 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:46.079+0000 7f3e4206f700 1 --2- 172.21.15.42:0/1154540183 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3e1c03b250 0x7f3e1c03d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:00:46.092 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:46.079+0000 7f3e4206f700 1 --2- 172.21.15.42:0/1154540183 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3e1c03b250 0x7f3e1c03d760 crc :-1 s=READY pgs=797 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:00:46.093 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:46.083+0000 7f3e442d3700 1 -- 172.21.15.42:0/1154540183 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e3c05d110 con 0x7f3e3c11a960 2019-12-05T06:00:46.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:46.091+0000 7f3e42870700 1 --2- 172.21.15.42:0/1154540183 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3e3c11a960 0x7f3e3c116360 secure :-1 s=THROTTLE_DONE pgs=1865 cs=0 l=1 rx=0x7f3e3c11b970 tx=0x7f3e34007960).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:46.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:46.091+0000 7f3e3b7fe700 1 -- 172.21.15.42:0/1154540183 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f3e34008290 con 0x7f3e3c11a960 2019-12-05T06:00:46.269 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:46.263+0000 7f3e442d3700 1 -- 172.21.15.42:0/1154540183 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3e3c05b8c0 con 0x7f3e3c11a960 2019-12-05T06:00:46.269 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:46.263+0000 7f3e42870700 1 --2- 172.21.15.42:0/1154540183 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3e3c11a960 0x7f3e3c116360 secure :-1 s=THROTTLE_DONE pgs=1865 cs=0 l=1 rx=0x7f3e3c11b970 tx=0x7f3e34007960).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:46.269 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:46.263+0000 7f3e3b7fe700 1 -- 172.21.15.42:0/1154540183 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 228 v228) v1 ==== 95+0+2108 (secure 0 0 0) 0x7f3e34076e30 con 0x7f3e3c11a960 2019-12-05T06:00:46.272 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 228 2019-12-05T06:00:46.272 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:00:46.272 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":228,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6653,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/2780168808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2780168808},{"type":"v1","addr":"172.21.15.40:6833","nonce":2780168808}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":228}],"filesystems":[{"mdsmap":{"epoch":227,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:00:19.604634+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":88,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9061},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9061":{"gid":9061,"name":"a","rank":0,"incarnation":224,"state":"up:active","state_seq":9,"addr":"172.21.15.42:6835/448869597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":448869597},{"type":"v1","addr":"172.21.15.42:6835","nonce":448869597}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:00:46.275 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:46.267+0000 7f3e442d3700 1 -- 172.21.15.42:0/1154540183 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3e1c03b250 msgr2=0x7f3e1c03d760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:00:46.275 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:46.267+0000 7f3e442d3700 1 --2- 172.21.15.42:0/1154540183 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3e1c03b250 0x7f3e1c03d760 crc :-1 s=READY pgs=797 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:00:46.275 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:46.267+0000 7f3e442d3700 1 -- 172.21.15.42:0/1154540183 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3e3c11a960 msgr2=0x7f3e3c116360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:00:46.275 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:46.267+0000 7f3e442d3700 1 --2- 172.21.15.42:0/1154540183 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3e3c11a960 0x7f3e3c116360 secure :-1 s=READY pgs=1865 cs=0 l=1 rx=0x7f3e3c11b970 tx=0x7f3e34007960).stop 2019-12-05T06:00:46.275 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:46.267+0000 7f3e442d3700 1 -- 172.21.15.42:0/1154540183 shutdown_connections 2019-12-05T06:00:46.275 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:46.267+0000 7f3e442d3700 1 --2- 172.21.15.42:0/1154540183 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3e1c03b250 0x7f3e1c03d760 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:00:46.275 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:46.267+0000 7f3e442d3700 1 --2- 172.21.15.42:0/1154540183 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3e3c0a23b0 0x7f3e3c133ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:00:46.276 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:46.267+0000 7f3e442d3700 1 --2- 172.21.15.42:0/1154540183 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3e3c11a960 0x7f3e3c116360 unknown :-1 s=CLOSED pgs=1865 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:00:46.276 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:46.267+0000 7f3e442d3700 1 -- 172.21.15.42:0/1154540183 shutdown_connections 2019-12-05T06:00:46.276 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:46.267+0000 7f3e442d3700 1 -- 172.21.15.42:0/1154540183 wait complete. 2019-12-05T06:00:46.276 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:46.267+0000 7f3e442d3700 1 -- 172.21.15.42:0/1154540183 >> 172.21.15.42:0/1154540183 conn(0x7f3e3c07e4f0 msgr2=0x7f3e3c11dac0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:00:46.285 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:00:48.286 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:00:48.389 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:48.379+0000 7f3570774700 1 Processor -- start 2019-12-05T06:00:48.389 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:48.379+0000 7f3570774700 1 -- start start 2019-12-05T06:00:48.389 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:48.379+0000 7f3570774700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f35680a7730 0x7f356808df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:00:48.389 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:48.379+0000 7f3570774700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f356807e710 0x7f356807eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:00:48.389 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:48.379+0000 7f3570774700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35680a0700 con 0x7f356809f2d0 2019-12-05T06:00:48.389 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:48.379+0000 7f3570774700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35680a0af0 con 0x7f35680a7730 2019-12-05T06:00:48.390 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:48.379+0000 7f3570774700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f356808e5a0 con 0x7f356807e710 2019-12-05T06:00:48.390 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:48.379+0000 7f356e510700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f356807e710 0x7f356807eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:00:48.390 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:48.379+0000 7f356ed11700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f35680a7730 0x7f356808df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:00:48.390 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:48.379+0000 7f356e510700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f356807e710 0x7f356807eb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:46596/0 (socket says 172.21.15.42:46596) 2019-12-05T06:00:48.390 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:48.379+0000 7f356e510700 1 -- 172.21.15.42:0/3146529356 learned_addr learned my addr 172.21.15.42:0/3146529356 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:00:48.390 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:48.379+0000 7f356e510700 1 -- 172.21.15.42:0/3146529356 >> v2:172.21.15.40:3300/0 conn(0x7f35680a7730 msgr2=0x7f356808df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:00:48.391 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:48.379+0000 7f356e510700 1 --2- 172.21.15.42:0/3146529356 >> v2:172.21.15.40:3300/0 conn(0x7f35680a7730 0x7f356808df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:00:48.391 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:48.379+0000 7f356e510700 1 -- 172.21.15.42:0/3146529356 >> v1:172.21.15.40:6789/0 conn(0x7f356809f2d0 legacy=0x7f356809f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:00:48.391 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:48.379+0000 7f356e510700 1 -- 172.21.15.42:0/3146529356 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f356807f090 con 0x7f356807e710 2019-12-05T06:00:48.391 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:48.379+0000 7f356e510700 1 --2- 172.21.15.42:0/3146529356 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f356807e710 0x7f356807eb40 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f3558008b10 tx=0x7f35580100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:48.391 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:48.379+0000 7f356e510700 1 --2- 172.21.15.42:0/3146529356 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f356807e710 0x7f356807eb40 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f3558008b10 tx=0x7f35580100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:48.391 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:48.379+0000 7f356e510700 1 --2- 172.21.15.42:0/3146529356 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f356807e710 0x7f356807eb40 secure :-1 s=READY pgs=417 cs=0 l=1 rx=0x7f3558008b10 tx=0x7f35580100b0).ready entity=mon.2 client_cookie=20d974d7c5a6b1c9 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:00:48.391 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:48.379+0000 7f356e510700 1 --2- 172.21.15.42:0/3146529356 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f356807e710 0x7f356807eb40 secure :-1 s=THROTTLE_DONE pgs=417 cs=0 l=1 rx=0x7f3558008b10 tx=0x7f35580100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:48.392 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:48.379+0000 7f356dd0f700 1 -- 172.21.15.42:0/3146529356 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f355800a160 con 0x7f356807e710 2019-12-05T06:00:48.392 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:48.379+0000 7f356e510700 1 --2- 172.21.15.42:0/3146529356 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f356807e710 0x7f356807eb40 secure :-1 s=THROTTLE_DONE pgs=417 cs=0 l=1 rx=0x7f3558008b10 tx=0x7f35580100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:48.392 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:48.379+0000 7f356dd0f700 1 -- 172.21.15.42:0/3146529356 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3558011a30 con 0x7f356807e710 2019-12-05T06:00:48.392 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:48.379+0000 7f356e510700 1 --2- 172.21.15.42:0/3146529356 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f356807e710 0x7f356807eb40 secure :-1 s=THROTTLE_DONE pgs=417 cs=0 l=1 rx=0x7f3558008b10 tx=0x7f35580100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:48.392 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:48.379+0000 7f3570774700 1 -- 172.21.15.42:0/3146529356 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f356807e710 msgr2=0x7f356807eb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:00:48.392 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:48.379+0000 7f356dd0f700 1 -- 172.21.15.42:0/3146529356 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f355800a550 con 0x7f356807e710 2019-12-05T06:00:48.392 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:48.379+0000 7f3570774700 1 --2- 172.21.15.42:0/3146529356 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f356807e710 0x7f356807eb40 secure :-1 s=READY pgs=417 cs=0 l=1 rx=0x7f3558008b10 tx=0x7f35580100b0).stop 2019-12-05T06:00:48.393 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:48.383+0000 7f3570774700 1 -- 172.21.15.42:0/3146529356 shutdown_connections 2019-12-05T06:00:48.393 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:48.383+0000 7f3570774700 1 --2- 172.21.15.42:0/3146529356 >> v2:172.21.15.40:3300/0 conn(0x7f35680a7730 0x7f356808df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:00:48.393 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:48.383+0000 7f3570774700 1 --2- 172.21.15.42:0/3146529356 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f356807e710 0x7f356807eb40 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:00:48.393 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:48.383+0000 7f3570774700 1 -- 172.21.15.42:0/3146529356 shutdown_connections 2019-12-05T06:00:48.393 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:48.383+0000 7f3570774700 1 -- 172.21.15.42:0/3146529356 wait complete. 2019-12-05T06:00:48.393 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:48.383+0000 7f3570774700 1 -- 172.21.15.42:0/3146529356 >> 172.21.15.42:0/3146529356 conn(0x7f35680aabd0 msgr2=0x7f35680a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:00:48.396 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:48.383+0000 7f3570774700 1 Processor -- start 2019-12-05T06:00:48.397 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:48.383+0000 7f3570774700 1 -- start start 2019-12-05T06:00:48.397 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:48.383+0000 7f3570774700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f356808eea0 0x7f3568122220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:00:48.398 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:48.383+0000 7f3570774700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f35681181f0 0x7f356807b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:00:48.398 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:48.383+0000 7f3570774700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3568123670 con 0x7f356807afb0 2019-12-05T06:00:48.398 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:48.383+0000 7f3570774700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3568123a10 con 0x7f356808eea0 2019-12-05T06:00:48.398 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:48.383+0000 7f3570774700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3568123cb0 con 0x7f35681181f0 2019-12-05T06:00:48.398 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:48.383+0000 7f356f512700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f35681181f0 0x7f356807b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:00:48.398 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:48.383+0000 7f356ed11700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f356808eea0 0x7f3568122220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:00:48.398 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:48.383+0000 7f356e510700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f356807afb0 0x7f35681179f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:55616/0 (socket says 172.21.15.42:55616) 2019-12-05T06:00:48.399 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:48.383+0000 7f356e510700 1 -- 172.21.15.42:0/3375784845 learned_addr learned my addr 172.21.15.42:0/3375784845 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:00:48.399 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:48.383+0000 7f356f512700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f35681181f0 0x7f356807b3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:46140/0 (socket says 172.21.15.42:46140) 2019-12-05T06:00:48.399 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:48.387+0000 7f355f7fe700 1 -- 172.21.15.42:0/3375784845 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 973500518 0 0) 0x7f3568123670 con 0x7f356807afb0 2019-12-05T06:00:48.399 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:48.387+0000 7f356f512700 1 -- 172.21.15.42:0/3375784845 >> v2:172.21.15.40:3300/0 conn(0x7f356808eea0 msgr2=0x7f3568122220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:00:48.399 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:48.387+0000 7f356f512700 1 --2- 172.21.15.42:0/3375784845 >> v2:172.21.15.40:3300/0 conn(0x7f356808eea0 0x7f3568122220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:00:48.399 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:48.387+0000 7f356f512700 1 -- 172.21.15.42:0/3375784845 >> v1:172.21.15.40:6789/0 conn(0x7f356807afb0 legacy=0x7f35681179f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:00:48.400 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:48.387+0000 7f356f512700 1 -- 172.21.15.42:0/3375784845 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3568123f50 con 0x7f35681181f0 2019-12-05T06:00:48.400 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:48.387+0000 7f356f512700 1 --2- 172.21.15.42:0/3375784845 >> v2:172.21.15.42:3300/0 conn(0x7f35681181f0 0x7f356807b3e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f3560000e90 tx=0x7f35600107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:48.400 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:48.387+0000 7f356f512700 1 --2- 172.21.15.42:0/3375784845 >> v2:172.21.15.42:3300/0 conn(0x7f35681181f0 0x7f356807b3e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f3560000e90 tx=0x7f35600107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:48.400 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:48.387+0000 7f356f512700 1 --2- 172.21.15.42:0/3375784845 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f35681181f0 0x7f356807b3e0 secure :-1 s=READY pgs=1866 cs=0 l=1 rx=0x7f3560000e90 tx=0x7f35600107c0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:00:48.400 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:48.387+0000 7f356f512700 1 --2- 172.21.15.42:0/3375784845 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f35681181f0 0x7f356807b3e0 secure :-1 s=THROTTLE_DONE pgs=1866 cs=0 l=1 rx=0x7f3560000e90 tx=0x7f35600107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:48.400 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:48.387+0000 7f355f7fe700 1 -- 172.21.15.42:0/3375784845 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3560006a40 con 0x7f35681181f0 2019-12-05T06:00:48.400 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:48.387+0000 7f356f512700 1 --2- 172.21.15.42:0/3375784845 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f35681181f0 0x7f356807b3e0 secure :-1 s=THROTTLE_DONE pgs=1866 cs=0 l=1 rx=0x7f3560000e90 tx=0x7f35600107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:48.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:48.387+0000 7f355f7fe700 1 -- 172.21.15.42:0/3375784845 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3560006660 con 0x7f35681181f0 2019-12-05T06:00:48.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:48.387+0000 7f356f512700 1 --2- 172.21.15.42:0/3375784845 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f35681181f0 0x7f356807b3e0 secure :-1 s=THROTTLE_DONE pgs=1866 cs=0 l=1 rx=0x7f3560000e90 tx=0x7f35600107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:48.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:48.387+0000 7f3570774700 1 -- 172.21.15.42:0/3375784845 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3568122770 con 0x7f35681181f0 2019-12-05T06:00:48.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:48.387+0000 7f355f7fe700 1 -- 172.21.15.42:0/3375784845 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3560011000 con 0x7f35681181f0 2019-12-05T06:00:48.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:48.387+0000 7f3570774700 1 -- 172.21.15.42:0/3375784845 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3568138c40 con 0x7f35681181f0 2019-12-05T06:00:48.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:48.387+0000 7f356f512700 1 --2- 172.21.15.42:0/3375784845 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f35681181f0 0x7f356807b3e0 secure :-1 s=THROTTLE_DONE pgs=1866 cs=0 l=1 rx=0x7f3560000e90 tx=0x7f35600107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:48.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:48.387+0000 7f355f7fe700 1 -- 172.21.15.42:0/3375784845 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f3560011320 con 0x7f35681181f0 2019-12-05T06:00:48.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:48.387+0000 7f356f512700 1 --2- 172.21.15.42:0/3375784845 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f35681181f0 0x7f356807b3e0 secure :-1 s=THROTTLE_DONE pgs=1866 cs=0 l=1 rx=0x7f3560000e90 tx=0x7f35600107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:48.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:48.387+0000 7f355f7fe700 1 --2- 172.21.15.42:0/3375784845 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f354803abf0 0x7f354803d100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:00:48.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:48.387+0000 7f355f7fe700 1 -- 172.21.15.42:0/3375784845 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9385+0+0 (secure 0 0 0) 0x7f356004b0a0 con 0x7f35681181f0 2019-12-05T06:00:48.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:48.391+0000 7f356ed11700 1 --2- 172.21.15.42:0/3375784845 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f354803abf0 0x7f354803d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:00:48.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:48.391+0000 7f356ed11700 1 --2- 172.21.15.42:0/3375784845 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f354803abf0 0x7f354803d100 crc :-1 s=READY pgs=798 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:00:48.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:48.395+0000 7f3570774700 1 -- 172.21.15.42:0/3375784845 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f356805d100 con 0x7f35681181f0 2019-12-05T06:00:48.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:48.403+0000 7f356f512700 1 --2- 172.21.15.42:0/3375784845 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f35681181f0 0x7f356807b3e0 secure :-1 s=THROTTLE_DONE pgs=1866 cs=0 l=1 rx=0x7f3560000e90 tx=0x7f35600107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:48.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:48.403+0000 7f355f7fe700 1 -- 172.21.15.42:0/3375784845 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f3560032100 con 0x7f35681181f0 2019-12-05T06:00:48.574 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:48.567+0000 7f3570774700 1 -- 172.21.15.42:0/3375784845 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f356805b8b0 con 0x7f35681181f0 2019-12-05T06:00:48.575 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:48.571+0000 7f356f512700 1 --2- 172.21.15.42:0/3375784845 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f35681181f0 0x7f356807b3e0 secure :-1 s=THROTTLE_DONE pgs=1866 cs=0 l=1 rx=0x7f3560000e90 tx=0x7f35600107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:48.575 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:48.571+0000 7f355f7fe700 1 -- 172.21.15.42:0/3375784845 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 228 v228) v1 ==== 95+0+2108 (secure 0 0 0) 0x7f3560032360 con 0x7f35681181f0 2019-12-05T06:00:48.578 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 228 2019-12-05T06:00:48.578 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:00:48.578 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":228,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6653,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/2780168808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2780168808},{"type":"v1","addr":"172.21.15.40:6833","nonce":2780168808}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":228}],"filesystems":[{"mdsmap":{"epoch":227,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:00:19.604634+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":88,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9061},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9061":{"gid":9061,"name":"a","rank":0,"incarnation":224,"state":"up:active","state_seq":9,"addr":"172.21.15.42:6835/448869597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":448869597},{"type":"v1","addr":"172.21.15.42:6835","nonce":448869597}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:00:48.583 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:48.575+0000 7f3570774700 1 -- 172.21.15.42:0/3375784845 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f354803abf0 msgr2=0x7f354803d100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:00:48.583 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:48.575+0000 7f3570774700 1 --2- 172.21.15.42:0/3375784845 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f354803abf0 0x7f354803d100 crc :-1 s=READY pgs=798 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:00:48.583 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:48.575+0000 7f3570774700 1 -- 172.21.15.42:0/3375784845 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f35681181f0 msgr2=0x7f356807b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:00:48.584 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:48.575+0000 7f3570774700 1 --2- 172.21.15.42:0/3375784845 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f35681181f0 0x7f356807b3e0 secure :-1 s=READY pgs=1866 cs=0 l=1 rx=0x7f3560000e90 tx=0x7f35600107c0).stop 2019-12-05T06:00:48.584 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:48.575+0000 7f3570774700 1 -- 172.21.15.42:0/3375784845 shutdown_connections 2019-12-05T06:00:48.584 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:48.575+0000 7f3570774700 1 --2- 172.21.15.42:0/3375784845 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f354803abf0 0x7f354803d100 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:00:48.584 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:48.575+0000 7f3570774700 1 --2- 172.21.15.42:0/3375784845 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f35681181f0 0x7f356807b3e0 unknown :-1 s=CLOSED pgs=1866 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:00:48.584 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:48.575+0000 7f3570774700 1 --2- 172.21.15.42:0/3375784845 >> v2:172.21.15.40:3300/0 conn(0x7f356808eea0 0x7f3568122220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:00:48.584 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:48.575+0000 7f3570774700 1 -- 172.21.15.42:0/3375784845 shutdown_connections 2019-12-05T06:00:48.585 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:48.575+0000 7f3570774700 1 -- 172.21.15.42:0/3375784845 wait complete. 2019-12-05T06:00:48.585 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:48.575+0000 7f3570774700 1 -- 172.21.15.42:0/3375784845 >> 172.21.15.42:0/3375784845 conn(0x7f35680a9630 msgr2=0x7f356809f610 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:00:48.598 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:00:50.599 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:00:50.709 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:50.699+0000 7f63c6632700 1 Processor -- start 2019-12-05T06:00:50.709 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:50.699+0000 7f63c6632700 1 -- start start 2019-12-05T06:00:50.709 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:50.699+0000 7f63c6632700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f63c00a7740 0x7f63c008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:00:50.709 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:50.699+0000 7f63c6632700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f63c007e720 0x7f63c007eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:00:50.709 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:50.699+0000 7f63c6632700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63c00a0710 con 0x7f63c009f2e0 2019-12-05T06:00:50.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:50.699+0000 7f63c6632700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63c00a0b00 con 0x7f63c00a7740 2019-12-05T06:00:50.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:50.699+0000 7f63c6632700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63c008e5b0 con 0x7f63c007e720 2019-12-05T06:00:50.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:50.699+0000 7f63c4bcf700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f63c00a7740 0x7f63c008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:00:50.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:50.699+0000 7f63c4bcf700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f63c00a7740 0x7f63c008df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:46148/0 (socket says 172.21.15.42:46148) 2019-12-05T06:00:50.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:50.699+0000 7f63c4bcf700 1 -- 172.21.15.42:0/1828121274 learned_addr learned my addr 172.21.15.42:0/1828121274 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:00:50.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:50.699+0000 7f63c53d0700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f63c009f2e0 0x7f63c009f710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:46188/0 (socket says 172.21.15.42:46188) 2019-12-05T06:00:50.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:50.699+0000 7f63bffff700 1 --2- 172.21.15.42:0/1828121274 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f63c007e720 0x7f63c007eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:00:50.711 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:50.699+0000 7f63c4bcf700 1 -- 172.21.15.42:0/1828121274 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f63c007e720 msgr2=0x7f63c007eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:00:50.711 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:50.699+0000 7f63c4bcf700 1 --2- 172.21.15.42:0/1828121274 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f63c007e720 0x7f63c007eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:00:50.711 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:50.699+0000 7f63c4bcf700 1 -- 172.21.15.42:0/1828121274 >> v1:172.21.15.42:6789/0 conn(0x7f63c009f2e0 legacy=0x7f63c009f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:00:50.711 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:50.699+0000 7f63bf7fe700 1 -- 172.21.15.42:0/1828121274 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4014136169 0 0) 0x7f63c00a0710 con 0x7f63c009f2e0 2019-12-05T06:00:50.711 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:50.699+0000 7f63c4bcf700 1 -- 172.21.15.42:0/1828121274 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63c007f0a0 con 0x7f63c00a7740 2019-12-05T06:00:50.711 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:50.699+0000 7f63c4bcf700 1 --2- 172.21.15.42:0/1828121274 >> v2:172.21.15.42:3300/0 conn(0x7f63c00a7740 0x7f63c008df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f63b0008b10 tx=0x7f63b00100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:50.711 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:50.699+0000 7f63c4bcf700 1 --2- 172.21.15.42:0/1828121274 >> v2:172.21.15.42:3300/0 conn(0x7f63c00a7740 0x7f63c008df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f63b0008b10 tx=0x7f63b00100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:50.712 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:50.699+0000 7f63c4bcf700 1 --2- 172.21.15.42:0/1828121274 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f63c00a7740 0x7f63c008df80 secure :-1 s=READY pgs=1868 cs=0 l=1 rx=0x7f63b0008b10 tx=0x7f63b00100b0).ready entity=mon.1 client_cookie=ba678621db92f839 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:00:50.712 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:50.699+0000 7f63c4bcf700 1 --2- 172.21.15.42:0/1828121274 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f63c00a7740 0x7f63c008df80 secure :-1 s=THROTTLE_DONE pgs=1868 cs=0 l=1 rx=0x7f63b0008b10 tx=0x7f63b00100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:50.712 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:50.699+0000 7f63bf7fe700 1 -- 172.21.15.42:0/1828121274 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f63b000a160 con 0x7f63c00a7740 2019-12-05T06:00:50.712 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:50.699+0000 7f63c4bcf700 1 --2- 172.21.15.42:0/1828121274 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f63c00a7740 0x7f63c008df80 secure :-1 s=THROTTLE_DONE pgs=1868 cs=0 l=1 rx=0x7f63b0008b10 tx=0x7f63b00100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:50.713 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:50.699+0000 7f63bf7fe700 1 -- 172.21.15.42:0/1828121274 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63b0011a30 con 0x7f63c00a7740 2019-12-05T06:00:50.713 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:50.699+0000 7f63c4bcf700 1 --2- 172.21.15.42:0/1828121274 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f63c00a7740 0x7f63c008df80 secure :-1 s=THROTTLE_DONE pgs=1868 cs=0 l=1 rx=0x7f63b0008b10 tx=0x7f63b00100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:50.713 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:50.699+0000 7f63c6632700 1 -- 172.21.15.42:0/1828121274 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f63c00a7740 msgr2=0x7f63c008df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:00:50.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:50.699+0000 7f63c6632700 1 --2- 172.21.15.42:0/1828121274 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f63c00a7740 0x7f63c008df80 secure :-1 s=READY pgs=1868 cs=0 l=1 rx=0x7f63b0008b10 tx=0x7f63b00100b0).stop 2019-12-05T06:00:50.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:50.699+0000 7f63bf7fe700 1 -- 172.21.15.42:0/1828121274 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f63b000a550 con 0x7f63c00a7740 2019-12-05T06:00:50.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:50.703+0000 7f63c6632700 1 -- 172.21.15.42:0/1828121274 shutdown_connections 2019-12-05T06:00:50.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:50.703+0000 7f63c6632700 1 --2- 172.21.15.42:0/1828121274 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f63c00a7740 0x7f63c008df80 unknown :-1 s=CLOSED pgs=1868 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:00:50.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:50.703+0000 7f63c6632700 1 --2- 172.21.15.42:0/1828121274 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f63c007e720 0x7f63c007eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:00:50.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:50.703+0000 7f63c6632700 1 -- 172.21.15.42:0/1828121274 shutdown_connections 2019-12-05T06:00:50.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:50.703+0000 7f63c6632700 1 -- 172.21.15.42:0/1828121274 wait complete. 2019-12-05T06:00:50.715 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:50.703+0000 7f63c6632700 1 -- 172.21.15.42:0/1828121274 >> 172.21.15.42:0/1828121274 conn(0x7f63c00aabe0 msgr2=0x7f63c00a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:00:50.718 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:50.703+0000 7f63c6632700 1 Processor -- start 2019-12-05T06:00:50.718 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:50.703+0000 7f63c6632700 1 -- start start 2019-12-05T06:00:50.718 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:50.703+0000 7f63c6632700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f63c0113b60 0x7f63c007afe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:00:50.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:50.703+0000 7f63c6632700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63c01134e0 con 0x7f63c0143010 2019-12-05T06:00:50.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:50.703+0000 7f63c6632700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63c007bb30 con 0x7f63c007b530 2019-12-05T06:00:50.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:50.703+0000 7f63c6632700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63c0142ad0 con 0x7f63c0113b60 2019-12-05T06:00:50.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:50.703+0000 7f63c53d0700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f63c007b530 0x7f63c012c3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:46192/0 (socket says 172.21.15.42:46192) 2019-12-05T06:00:50.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:50.703+0000 7f63c53d0700 1 -- 172.21.15.42:0/1495317350 learned_addr learned my addr 172.21.15.42:0/1495317350 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:00:50.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:50.703+0000 7f63bffff700 1 --2- 172.21.15.42:0/1495317350 >> v2:172.21.15.40:3300/0 conn(0x7f63c0113b60 0x7f63c007afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:00:50.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:50.707+0000 7f63bd7fa700 1 -- 172.21.15.42:0/1495317350 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2509577566 0 0) 0x7f63c007bb30 con 0x7f63c007b530 2019-12-05T06:00:50.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:50.707+0000 7f63bd7fa700 1 -- 172.21.15.42:0/1495317350 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63a40030b0 con 0x7f63c007b530 2019-12-05T06:00:50.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:50.707+0000 7f63bd7fa700 1 -- 172.21.15.42:0/1495317350 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 549840146 0 0) 0x7f63c01134e0 con 0x7f63c0143010 2019-12-05T06:00:50.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:50.707+0000 7f63bffff700 1 -- 172.21.15.42:0/1495317350 >> v1:172.21.15.42:6789/0 conn(0x7f63c007b530 legacy=0x7f63c012c3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:00:50.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:50.707+0000 7f63bffff700 1 -- 172.21.15.42:0/1495317350 >> v1:172.21.15.40:6789/0 conn(0x7f63c0143010 legacy=0x7f63c01121f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:00:50.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:50.707+0000 7f63bffff700 1 -- 172.21.15.42:0/1495317350 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63c008f050 con 0x7f63c0113b60 2019-12-05T06:00:50.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:50.707+0000 7f63bffff700 1 --2- 172.21.15.42:0/1495317350 >> v2:172.21.15.40:3300/0 conn(0x7f63c0113b60 0x7f63c007afe0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f63a8002f30 tx=0x7f63a8010af0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:50.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:50.707+0000 7f63bffff700 1 --2- 172.21.15.42:0/1495317350 >> v2:172.21.15.40:3300/0 conn(0x7f63c0113b60 0x7f63c007afe0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f63a8002f30 tx=0x7f63a8010af0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:50.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:50.707+0000 7f63bffff700 1 --2- 172.21.15.42:0/1495317350 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f63c0113b60 0x7f63c007afe0 secure :-1 s=READY pgs=1434 cs=0 l=1 rx=0x7f63a8002f30 tx=0x7f63a8010af0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:00:50.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:50.707+0000 7f63bffff700 1 --2- 172.21.15.42:0/1495317350 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f63c0113b60 0x7f63c007afe0 secure :-1 s=THROTTLE_DONE pgs=1434 cs=0 l=1 rx=0x7f63a8002f30 tx=0x7f63a8010af0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:50.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:50.707+0000 7f63bd7fa700 1 -- 172.21.15.42:0/1495317350 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f63a80125a0 con 0x7f63c0113b60 2019-12-05T06:00:50.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:50.707+0000 7f63bffff700 1 --2- 172.21.15.42:0/1495317350 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f63c0113b60 0x7f63c007afe0 secure :-1 s=THROTTLE_DONE pgs=1434 cs=0 l=1 rx=0x7f63a8002f30 tx=0x7f63a8010af0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:50.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:50.707+0000 7f63bffff700 1 --2- 172.21.15.42:0/1495317350 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f63c0113b60 0x7f63c007afe0 secure :-1 s=THROTTLE_DONE pgs=1434 cs=0 l=1 rx=0x7f63a8002f30 tx=0x7f63a8010af0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:50.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:50.707+0000 7f63bd7fa700 1 -- 172.21.15.42:0/1495317350 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63a8007150 con 0x7f63c0113b60 2019-12-05T06:00:50.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:50.707+0000 7f63c6632700 1 -- 172.21.15.42:0/1495317350 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63c0142610 con 0x7f63c0113b60 2019-12-05T06:00:50.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:50.707+0000 7f63bd7fa700 1 -- 172.21.15.42:0/1495317350 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f63a8012b80 con 0x7f63c0113b60 2019-12-05T06:00:50.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:50.707+0000 7f63c6632700 1 -- 172.21.15.42:0/1495317350 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63c012dbc0 con 0x7f63c0113b60 2019-12-05T06:00:50.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:50.707+0000 7f63bffff700 1 --2- 172.21.15.42:0/1495317350 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f63c0113b60 0x7f63c007afe0 secure :-1 s=THROTTLE_DONE pgs=1434 cs=0 l=1 rx=0x7f63a8002f30 tx=0x7f63a8010af0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:50.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:50.707+0000 7f63bd7fa700 1 -- 172.21.15.42:0/1495317350 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f63a8005540 con 0x7f63c0113b60 2019-12-05T06:00:50.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:50.707+0000 7f63bffff700 1 --2- 172.21.15.42:0/1495317350 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f63c0113b60 0x7f63c007afe0 secure :-1 s=THROTTLE_DONE pgs=1434 cs=0 l=1 rx=0x7f63a8002f30 tx=0x7f63a8010af0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:50.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:50.711+0000 7f63bd7fa700 1 --2- 172.21.15.42:0/1495317350 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f63a403b1e0 0x7f63a403d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:00:50.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:50.711+0000 7f63bd7fa700 1 -- 172.21.15.42:0/1495317350 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9385+0+0 (secure 0 0 0) 0x7f63a804af50 con 0x7f63c0113b60 2019-12-05T06:00:50.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:50.711+0000 7f63c4bcf700 1 --2- 172.21.15.42:0/1495317350 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f63a403b1e0 0x7f63a403d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:00:50.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:50.711+0000 7f63c4bcf700 1 --2- 172.21.15.42:0/1495317350 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f63a403b1e0 0x7f63a403d6f0 crc :-1 s=READY pgs=799 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:00:50.724 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:50.715+0000 7f63c6632700 1 -- 172.21.15.42:0/1495317350 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63c005d110 con 0x7f63c0113b60 2019-12-05T06:00:50.729 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:50.723+0000 7f63bffff700 1 --2- 172.21.15.42:0/1495317350 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f63c0113b60 0x7f63c007afe0 secure :-1 s=THROTTLE_DONE pgs=1434 cs=0 l=1 rx=0x7f63a8002f30 tx=0x7f63a8010af0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:50.730 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:50.723+0000 7f63bd7fa700 1 -- 172.21.15.42:0/1495317350 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f63a8031fd0 con 0x7f63c0113b60 2019-12-05T06:00:50.903 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:50.895+0000 7f63c6632700 1 -- 172.21.15.42:0/1495317350 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f63c005b8c0 con 0x7f63c0113b60 2019-12-05T06:00:50.903 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:50.899+0000 7f63bffff700 1 --2- 172.21.15.42:0/1495317350 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f63c0113b60 0x7f63c007afe0 secure :-1 s=THROTTLE_DONE pgs=1434 cs=0 l=1 rx=0x7f63a8002f30 tx=0x7f63a8010af0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:50.904 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:50.899+0000 7f63bd7fa700 1 -- 172.21.15.42:0/1495317350 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 228 v228) v1 ==== 95+0+2108 (secure 0 0 0) 0x7f63a8032230 con 0x7f63c0113b60 2019-12-05T06:00:50.910 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:00:50.910 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":228,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6653,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/2780168808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2780168808},{"type":"v1","addr":"172.21.15.40:6833","nonce":2780168808}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":228}],"filesystems":[{"mdsmap":{"epoch":227,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:00:19.604634+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":88,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9061},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9061":{"gid":9061,"name":"a","rank":0,"incarnation":224,"state":"up:active","state_seq":9,"addr":"172.21.15.42:6835/448869597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":448869597},{"type":"v1","addr":"172.21.15.42:6835","nonce":448869597}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:00:50.910 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 228 2019-12-05T06:00:50.910 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:50.903+0000 7f63c6632700 1 -- 172.21.15.42:0/1495317350 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f63a403b1e0 msgr2=0x7f63a403d6f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:00:50.911 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:50.903+0000 7f63c6632700 1 --2- 172.21.15.42:0/1495317350 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f63a403b1e0 0x7f63a403d6f0 crc :-1 s=READY pgs=799 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:00:50.911 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:50.903+0000 7f63c6632700 1 -- 172.21.15.42:0/1495317350 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f63c0113b60 msgr2=0x7f63c007afe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:00:50.911 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:50.903+0000 7f63c6632700 1 --2- 172.21.15.42:0/1495317350 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f63c0113b60 0x7f63c007afe0 secure :-1 s=READY pgs=1434 cs=0 l=1 rx=0x7f63a8002f30 tx=0x7f63a8010af0).stop 2019-12-05T06:00:50.911 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:50.903+0000 7f63c6632700 1 -- 172.21.15.42:0/1495317350 shutdown_connections 2019-12-05T06:00:50.911 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:50.903+0000 7f63c6632700 1 --2- 172.21.15.42:0/1495317350 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f63a403b1e0 0x7f63a403d6f0 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:00:50.911 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:50.903+0000 7f63c6632700 1 --2- 172.21.15.42:0/1495317350 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f63c0113b60 0x7f63c007afe0 unknown :-1 s=CLOSED pgs=1434 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:00:50.911 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:50.903+0000 7f63c6632700 1 -- 172.21.15.42:0/1495317350 shutdown_connections 2019-12-05T06:00:50.912 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:50.903+0000 7f63c6632700 1 -- 172.21.15.42:0/1495317350 wait complete. 2019-12-05T06:00:50.912 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:50.903+0000 7f63c6632700 1 -- 172.21.15.42:0/1495317350 >> 172.21.15.42:0/1495317350 conn(0x7f63c00a7740 msgr2=0x7f63c007e610 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:00:50.920 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:00:52.920 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:00:53.029 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:53.015+0000 7f6919e27700 1 Processor -- start 2019-12-05T06:00:53.029 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:53.015+0000 7f6919e27700 1 -- start start 2019-12-05T06:00:53.029 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:53.015+0000 7f6919e27700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f69140a7740 0x7f691408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:00:53.029 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:53.015+0000 7f6919e27700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f691407e720 0x7f691407eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:00:53.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:53.015+0000 7f6919e27700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69140a0710 con 0x7f691409f2e0 2019-12-05T06:00:53.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:53.015+0000 7f6919e27700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69140a0b00 con 0x7f691407e720 2019-12-05T06:00:53.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:53.015+0000 7f6919e27700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f691408e5b0 con 0x7f69140a7740 2019-12-05T06:00:53.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:53.015+0000 7f6913fff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f69140a7740 0x7f691408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:00:53.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:53.015+0000 7f6913fff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f69140a7740 0x7f691408df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:46164/0 (socket says 172.21.15.42:46164) 2019-12-05T06:00:53.033 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:53.015+0000 7f6913fff700 1 -- 172.21.15.42:0/3777254051 learned_addr learned my addr 172.21.15.42:0/3777254051 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:00:53.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:53.015+0000 7f6918bc5700 1 --1- 172.21.15.42:0/3777254051 >> v1:172.21.15.42:6789/0 conn(0x7f691409f2e0 0x7f691409f710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:46204/0 (socket says 172.21.15.42:46204) 2019-12-05T06:00:53.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:53.015+0000 7f69137fe700 1 --2- 172.21.15.42:0/3777254051 >> v2:172.21.15.40:3300/0 conn(0x7f691407e720 0x7f691407eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:00:53.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:53.015+0000 7f6913fff700 1 -- 172.21.15.42:0/3777254051 >> v2:172.21.15.40:3300/0 conn(0x7f691407e720 msgr2=0x7f691407eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:00:53.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:53.015+0000 7f6913fff700 1 --2- 172.21.15.42:0/3777254051 >> v2:172.21.15.40:3300/0 conn(0x7f691407e720 0x7f691407eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:00:53.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:53.015+0000 7f6913fff700 1 -- 172.21.15.42:0/3777254051 >> v1:172.21.15.42:6789/0 conn(0x7f691409f2e0 legacy=0x7f691409f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:00:53.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:53.015+0000 7f6913fff700 1 -- 172.21.15.42:0/3777254051 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f691407f0a0 con 0x7f69140a7740 2019-12-05T06:00:53.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:53.015+0000 7f69137fe700 1 --2- 172.21.15.42:0/3777254051 >> v2:172.21.15.40:3300/0 conn(0x7f691407e720 0x7f691407eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:00:53.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:53.015+0000 7f6913fff700 1 --2- 172.21.15.42:0/3777254051 >> v2:172.21.15.42:3300/0 conn(0x7f69140a7740 0x7f691408df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f68fc002f30 tx=0x7f68fc00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:53.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:53.015+0000 7f6913fff700 1 --2- 172.21.15.42:0/3777254051 >> v2:172.21.15.42:3300/0 conn(0x7f69140a7740 0x7f691408df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f68fc002f30 tx=0x7f68fc00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:53.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:53.015+0000 7f6913fff700 1 --2- 172.21.15.42:0/3777254051 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f69140a7740 0x7f691408df80 secure :-1 s=READY pgs=1871 cs=0 l=1 rx=0x7f68fc002f30 tx=0x7f68fc00fd30).ready entity=mon.1 client_cookie=8737dc632beeb3dc server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:00:53.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:53.015+0000 7f6913fff700 1 --2- 172.21.15.42:0/3777254051 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f69140a7740 0x7f691408df80 secure :-1 s=THROTTLE_DONE pgs=1871 cs=0 l=1 rx=0x7f68fc002f30 tx=0x7f68fc00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:53.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:53.015+0000 7f6912ffd700 1 -- 172.21.15.42:0/3777254051 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f68fc011370 con 0x7f69140a7740 2019-12-05T06:00:53.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:53.015+0000 7f6913fff700 1 --2- 172.21.15.42:0/3777254051 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f69140a7740 0x7f691408df80 secure :-1 s=THROTTLE_DONE pgs=1871 cs=0 l=1 rx=0x7f68fc002f30 tx=0x7f68fc00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:53.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:53.015+0000 7f6912ffd700 1 -- 172.21.15.42:0/3777254051 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68fc008bb0 con 0x7f69140a7740 2019-12-05T06:00:53.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:53.015+0000 7f6913fff700 1 --2- 172.21.15.42:0/3777254051 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f69140a7740 0x7f691408df80 secure :-1 s=THROTTLE_DONE pgs=1871 cs=0 l=1 rx=0x7f68fc002f30 tx=0x7f68fc00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:53.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:53.019+0000 7f6919e27700 1 -- 172.21.15.42:0/3777254051 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f69140a7740 msgr2=0x7f691408df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:00:53.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:53.019+0000 7f6912ffd700 1 -- 172.21.15.42:0/3777254051 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f68fc011950 con 0x7f69140a7740 2019-12-05T06:00:53.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:53.019+0000 7f6919e27700 1 --2- 172.21.15.42:0/3777254051 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f69140a7740 0x7f691408df80 secure :-1 s=READY pgs=1871 cs=0 l=1 rx=0x7f68fc002f30 tx=0x7f68fc00fd30).stop 2019-12-05T06:00:53.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:53.019+0000 7f6919e27700 1 -- 172.21.15.42:0/3777254051 shutdown_connections 2019-12-05T06:00:53.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:53.019+0000 7f6919e27700 1 --2- 172.21.15.42:0/3777254051 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f69140a7740 0x7f691408df80 unknown :-1 s=CLOSED pgs=1871 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:00:53.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:53.019+0000 7f6919e27700 1 --2- 172.21.15.42:0/3777254051 >> v2:172.21.15.40:3300/0 conn(0x7f691407e720 0x7f691407eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:00:53.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:53.019+0000 7f6919e27700 1 -- 172.21.15.42:0/3777254051 shutdown_connections 2019-12-05T06:00:53.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:53.019+0000 7f6919e27700 1 -- 172.21.15.42:0/3777254051 wait complete. 2019-12-05T06:00:53.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:53.019+0000 7f6919e27700 1 -- 172.21.15.42:0/3777254051 >> 172.21.15.42:0/3777254051 conn(0x7f69140aabe0 msgr2=0x7f69140a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:00:53.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:53.019+0000 7f6919e27700 1 Processor -- start 2019-12-05T06:00:53.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:53.019+0000 7f6919e27700 1 -- start start 2019-12-05T06:00:53.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:53.019+0000 7f6919e27700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f691411da70 0x7f6914132760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:00:53.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:53.019+0000 7f6919e27700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6914132cb0 0x7f6914116780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:00:53.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:53.019+0000 7f6919e27700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6914116cd0 0x7f691411a310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:00:53.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:53.019+0000 7f6919e27700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f691411dea0 con 0x7f691411da70 2019-12-05T06:00:53.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:53.019+0000 7f6919e27700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69141332e0 con 0x7f6914116cd0 2019-12-05T06:00:53.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:53.019+0000 7f6919e27700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6914133580 con 0x7f6914132cb0 2019-12-05T06:00:53.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:53.019+0000 7f6918bc5700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6914116cd0 0x7f691411a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:00:53.039 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:53.023+0000 7f69137fe700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6914132cb0 0x7f6914116780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:00:53.039 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:53.023+0000 7f6918bc5700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6914116cd0 0x7f691411a310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:46172/0 (socket says 172.21.15.42:46172) 2019-12-05T06:00:53.039 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:53.023+0000 7f6918bc5700 1 -- 172.21.15.42:0/70381968 learned_addr learned my addr 172.21.15.42:0/70381968 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:00:53.039 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:53.023+0000 7f6913fff700 1 --2- 172.21.15.42:0/70381968 >> v2:172.21.15.40:3300/0 conn(0x7f691411da70 0x7f6914132760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:00:53.039 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:53.023+0000 7f69137fe700 1 -- 172.21.15.42:0/70381968 >> v2:172.21.15.42:3300/0 conn(0x7f6914116cd0 msgr2=0x7f691411a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:00:53.039 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:53.023+0000 7f69137fe700 1 --2- 172.21.15.42:0/70381968 >> v2:172.21.15.42:3300/0 conn(0x7f6914116cd0 0x7f691411a310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:00:53.040 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:53.023+0000 7f69137fe700 1 -- 172.21.15.42:0/70381968 >> v2:172.21.15.40:3300/0 conn(0x7f691411da70 msgr2=0x7f6914132760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:00:53.040 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:53.023+0000 7f69137fe700 1 --2- 172.21.15.42:0/70381968 >> v2:172.21.15.40:3300/0 conn(0x7f691411da70 0x7f6914132760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:00:53.040 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:53.023+0000 7f69137fe700 1 -- 172.21.15.42:0/70381968 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6914117100 con 0x7f6914132cb0 2019-12-05T06:00:53.040 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:53.023+0000 7f6913fff700 1 --2- 172.21.15.42:0/70381968 >> v2:172.21.15.40:3300/0 conn(0x7f691411da70 0x7f6914132760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:00:53.040 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:53.023+0000 7f69137fe700 1 --2- 172.21.15.42:0/70381968 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6914132cb0 0x7f6914116780 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f69040032b0 tx=0x7f6904010d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:53.040 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:53.023+0000 7f69137fe700 1 --2- 172.21.15.42:0/70381968 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6914132cb0 0x7f6914116780 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f69040032b0 tx=0x7f6904010d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:53.040 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:53.023+0000 7f69137fe700 1 --2- 172.21.15.42:0/70381968 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6914132cb0 0x7f6914116780 secure :-1 s=READY pgs=418 cs=0 l=1 rx=0x7f69040032b0 tx=0x7f6904010d50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:00:53.041 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:53.023+0000 7f69137fe700 1 --2- 172.21.15.42:0/70381968 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6914132cb0 0x7f6914116780 secure :-1 s=THROTTLE_DONE pgs=418 cs=0 l=1 rx=0x7f69040032b0 tx=0x7f6904010d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:53.041 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:53.023+0000 7f6910ff9700 1 -- 172.21.15.42:0/70381968 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6904007350 con 0x7f6914132cb0 2019-12-05T06:00:53.041 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:53.023+0000 7f69137fe700 1 --2- 172.21.15.42:0/70381968 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6914132cb0 0x7f6914116780 secure :-1 s=THROTTLE_DONE pgs=418 cs=0 l=1 rx=0x7f69040032b0 tx=0x7f6904010d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:53.041 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:53.023+0000 7f69137fe700 1 --2- 172.21.15.42:0/70381968 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6914132cb0 0x7f6914116780 secure :-1 s=THROTTLE_DONE pgs=418 cs=0 l=1 rx=0x7f69040032b0 tx=0x7f6904010d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:53.041 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:53.023+0000 7f6910ff9700 1 -- 172.21.15.42:0/70381968 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6904005cb0 con 0x7f6914132cb0 2019-12-05T06:00:53.041 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:53.023+0000 7f6919e27700 1 -- 172.21.15.42:0/70381968 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f691411a860 con 0x7f6914132cb0 2019-12-05T06:00:53.041 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:53.023+0000 7f6910ff9700 1 -- 172.21.15.42:0/70381968 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6904012600 con 0x7f6914132cb0 2019-12-05T06:00:53.042 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:53.023+0000 7f6919e27700 1 -- 172.21.15.42:0/70381968 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f691411ad90 con 0x7f6914132cb0 2019-12-05T06:00:53.043 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:53.023+0000 7f69137fe700 1 --2- 172.21.15.42:0/70381968 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6914132cb0 0x7f6914116780 secure :-1 s=THROTTLE_DONE pgs=418 cs=0 l=1 rx=0x7f69040032b0 tx=0x7f6904010d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:53.043 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:53.023+0000 7f69137fe700 1 --2- 172.21.15.42:0/70381968 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6914132cb0 0x7f6914116780 secure :-1 s=THROTTLE_DONE pgs=418 cs=0 l=1 rx=0x7f69040032b0 tx=0x7f6904010d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:53.043 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:53.023+0000 7f6910ff9700 1 -- 172.21.15.42:0/70381968 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f69040128d0 con 0x7f6914132cb0 2019-12-05T06:00:53.043 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:53.023+0000 7f6910ff9700 1 --2- 172.21.15.42:0/70381968 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f68ec03ae60 0x7f68ec03d370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:00:53.044 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:53.023+0000 7f6910ff9700 1 -- 172.21.15.42:0/70381968 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9385+0+0 (secure 0 0 0) 0x7f690404b290 con 0x7f6914132cb0 2019-12-05T06:00:53.044 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:53.023+0000 7f6913fff700 1 --2- 172.21.15.42:0/70381968 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f68ec03ae60 0x7f68ec03d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:00:53.044 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:53.027+0000 7f6913fff700 1 --2- 172.21.15.42:0/70381968 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f68ec03ae60 0x7f68ec03d370 crc :-1 s=READY pgs=800 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:00:53.044 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:53.031+0000 7f6919e27700 1 -- 172.21.15.42:0/70381968 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69141330e0 con 0x7f6914132cb0 2019-12-05T06:00:53.048 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:53.039+0000 7f69137fe700 1 --2- 172.21.15.42:0/70381968 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6914132cb0 0x7f6914116780 secure :-1 s=THROTTLE_DONE pgs=418 cs=0 l=1 rx=0x7f69040032b0 tx=0x7f6904010d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:53.048 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:53.039+0000 7f6910ff9700 1 -- 172.21.15.42:0/70381968 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f69040322d0 con 0x7f6914132cb0 2019-12-05T06:00:53.213 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:53.207+0000 7f6919e27700 1 -- 172.21.15.42:0/70381968 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f691405d110 con 0x7f6914132cb0 2019-12-05T06:00:53.214 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:53.207+0000 7f69137fe700 1 --2- 172.21.15.42:0/70381968 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6914132cb0 0x7f6914116780 secure :-1 s=THROTTLE_DONE pgs=418 cs=0 l=1 rx=0x7f69040032b0 tx=0x7f6904010d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:53.214 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:53.207+0000 7f6910ff9700 1 -- 172.21.15.42:0/70381968 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 228 v228) v1 ==== 95+0+2108 (secure 0 0 0) 0x7f6904032530 con 0x7f6914132cb0 2019-12-05T06:00:53.216 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 228 2019-12-05T06:00:53.216 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:00:53.217 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":228,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6653,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/2780168808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2780168808},{"type":"v1","addr":"172.21.15.40:6833","nonce":2780168808}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":228}],"filesystems":[{"mdsmap":{"epoch":227,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:00:19.604634+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":88,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9061},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9061":{"gid":9061,"name":"a","rank":0,"incarnation":224,"state":"up:active","state_seq":9,"addr":"172.21.15.42:6835/448869597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":448869597},{"type":"v1","addr":"172.21.15.42:6835","nonce":448869597}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:00:53.220 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:53.215+0000 7f6919e27700 1 -- 172.21.15.42:0/70381968 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f68ec03ae60 msgr2=0x7f68ec03d370 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:00:53.220 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:53.215+0000 7f6919e27700 1 --2- 172.21.15.42:0/70381968 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f68ec03ae60 0x7f68ec03d370 crc :-1 s=READY pgs=800 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:00:53.220 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:53.215+0000 7f6919e27700 1 -- 172.21.15.42:0/70381968 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6914132cb0 msgr2=0x7f6914116780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:00:53.220 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:53.215+0000 7f6919e27700 1 --2- 172.21.15.42:0/70381968 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6914132cb0 0x7f6914116780 secure :-1 s=READY pgs=418 cs=0 l=1 rx=0x7f69040032b0 tx=0x7f6904010d50).stop 2019-12-05T06:00:53.220 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:53.215+0000 7f6919e27700 1 -- 172.21.15.42:0/70381968 shutdown_connections 2019-12-05T06:00:53.221 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:53.215+0000 7f6919e27700 1 --2- 172.21.15.42:0/70381968 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f68ec03ae60 0x7f68ec03d370 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:00:53.221 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:53.215+0000 7f6919e27700 1 --2- 172.21.15.42:0/70381968 >> v2:172.21.15.42:3300/0 conn(0x7f6914116cd0 0x7f691411a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:00:53.221 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:53.215+0000 7f6919e27700 1 --2- 172.21.15.42:0/70381968 >> v2:172.21.15.40:3300/0 conn(0x7f691411da70 0x7f6914132760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:00:53.221 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:53.215+0000 7f6919e27700 1 --2- 172.21.15.42:0/70381968 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6914132cb0 0x7f6914116780 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:00:53.221 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:53.215+0000 7f6919e27700 1 -- 172.21.15.42:0/70381968 shutdown_connections 2019-12-05T06:00:53.221 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:53.215+0000 7f6919e27700 1 -- 172.21.15.42:0/70381968 wait complete. 2019-12-05T06:00:53.221 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:53.215+0000 7f6919e27700 1 -- 172.21.15.42:0/70381968 >> 172.21.15.42:0/70381968 conn(0x7f691408eb50 msgr2=0x7f691409f460 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:00:53.233 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:00:55.234 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:00:55.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:55.339+0000 7fbaf8f72700 1 Processor -- start 2019-12-05T06:00:55.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:55.339+0000 7fbaf8f72700 1 -- start start 2019-12-05T06:00:55.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:55.339+0000 7fbaf8f72700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fbaf408df80 0x7fbaf409f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:00:55.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:55.339+0000 7fbaf8f72700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbaf409fc60 con 0x7fbaf409f830 2019-12-05T06:00:55.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:55.339+0000 7fbaf8f72700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbaf40a0710 con 0x7fbaf40a7740 2019-12-05T06:00:55.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:55.339+0000 7fbaf8f72700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbaf40a0b00 con 0x7fbaf408df80 2019-12-05T06:00:55.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:55.339+0000 7fbaf259c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fbaf408df80 0x7fbaf409f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:00:55.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:55.339+0000 7fbaf2d9d700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fbaf40a7740 0x7fbaf407e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:46222/0 (socket says 172.21.15.42:46222) 2019-12-05T06:00:55.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:55.339+0000 7fbaf2d9d700 1 -- 172.21.15.42:0/3947805642 learned_addr learned my addr 172.21.15.42:0/3947805642 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:00:55.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:55.343+0000 7fbaf259c700 1 -- 172.21.15.42:0/3947805642 >> v1:172.21.15.42:6789/0 conn(0x7fbaf40a7740 legacy=0x7fbaf407e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:00:55.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:55.343+0000 7fbaf259c700 1 -- 172.21.15.42:0/3947805642 >> v1:172.21.15.40:6789/0 conn(0x7fbaf409f830 legacy=0x7fbaf414efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:00:55.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:55.343+0000 7fbaf1d9b700 1 -- 172.21.15.42:0/3947805642 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2320267977 0 0) 0x7fbaf40a0710 con 0x7fbaf40a7740 2019-12-05T06:00:55.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:55.343+0000 7fbaf259c700 1 -- 172.21.15.42:0/3947805642 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbaf407ef20 con 0x7fbaf408df80 2019-12-05T06:00:55.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:55.343+0000 7fbaf259c700 1 --2- 172.21.15.42:0/3947805642 >> v2:172.21.15.42:3300/0 conn(0x7fbaf408df80 0x7fbaf409f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fbadc008790 tx=0x7fbadc00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:55.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:55.343+0000 7fbaf259c700 1 --2- 172.21.15.42:0/3947805642 >> v2:172.21.15.42:3300/0 conn(0x7fbaf408df80 0x7fbaf409f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fbadc008790 tx=0x7fbadc00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:55.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:55.343+0000 7fbaf259c700 1 --2- 172.21.15.42:0/3947805642 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbaf408df80 0x7fbaf409f2e0 secure :-1 s=READY pgs=1878 cs=0 l=1 rx=0x7fbadc008790 tx=0x7fbadc00fbf0).ready entity=mon.1 client_cookie=231126a869bdcbd4 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:00:55.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:55.343+0000 7fbaf259c700 1 --2- 172.21.15.42:0/3947805642 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbaf408df80 0x7fbaf409f2e0 secure :-1 s=THROTTLE_DONE pgs=1878 cs=0 l=1 rx=0x7fbadc008790 tx=0x7fbadc00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:55.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:55.343+0000 7fbaf1d9b700 1 -- 172.21.15.42:0/3947805642 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbadc011270 con 0x7fbaf408df80 2019-12-05T06:00:55.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:55.343+0000 7fbaf259c700 1 --2- 172.21.15.42:0/3947805642 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbaf408df80 0x7fbaf409f2e0 secure :-1 s=THROTTLE_DONE pgs=1878 cs=0 l=1 rx=0x7fbadc008790 tx=0x7fbadc00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:55.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:55.343+0000 7fbaf259c700 1 --2- 172.21.15.42:0/3947805642 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbaf408df80 0x7fbaf409f2e0 secure :-1 s=THROTTLE_DONE pgs=1878 cs=0 l=1 rx=0x7fbadc008790 tx=0x7fbadc00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:55.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:55.343+0000 7fbaf1d9b700 1 -- 172.21.15.42:0/3947805642 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbadc008bc0 con 0x7fbaf408df80 2019-12-05T06:00:55.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:55.343+0000 7fbaf1d9b700 1 -- 172.21.15.42:0/3947805642 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbadc011850 con 0x7fbaf408df80 2019-12-05T06:00:55.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:55.343+0000 7fbaf8f72700 1 -- 172.21.15.42:0/3947805642 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbaf408df80 msgr2=0x7fbaf409f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:00:55.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:55.343+0000 7fbaf8f72700 1 --2- 172.21.15.42:0/3947805642 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbaf408df80 0x7fbaf409f2e0 secure :-1 s=READY pgs=1878 cs=0 l=1 rx=0x7fbadc008790 tx=0x7fbadc00fbf0).stop 2019-12-05T06:00:55.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:55.343+0000 7fbaf8f72700 1 -- 172.21.15.42:0/3947805642 shutdown_connections 2019-12-05T06:00:55.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:55.343+0000 7fbaf8f72700 1 --2- 172.21.15.42:0/3947805642 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbaf408df80 0x7fbaf409f2e0 unknown :-1 s=CLOSED pgs=1878 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:00:55.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:55.343+0000 7fbaf8f72700 1 -- 172.21.15.42:0/3947805642 shutdown_connections 2019-12-05T06:00:55.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:55.343+0000 7fbaf8f72700 1 -- 172.21.15.42:0/3947805642 wait complete. 2019-12-05T06:00:55.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:55.343+0000 7fbaf8f72700 1 -- 172.21.15.42:0/3947805642 >> 172.21.15.42:0/3947805642 conn(0x7fbaf40aabe0 msgr2=0x7fbaf40a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:00:55.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:55.343+0000 7fbaf8f72700 1 Processor -- start 2019-12-05T06:00:55.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:55.343+0000 7fbaf8f72700 1 -- start start 2019-12-05T06:00:55.358 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:55.343+0000 7fbaf8f72700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fbaf4116500 0x7fbaf4145fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:00:55.358 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:55.343+0000 7fbaf8f72700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbaf4146500 con 0x7fbaf411a770 2019-12-05T06:00:55.359 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:55.343+0000 7fbaf8f72700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbaf4130530 con 0x7fbaf4147fd0 2019-12-05T06:00:55.359 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:55.343+0000 7fbaf8f72700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbaf41467d0 con 0x7fbaf4116500 2019-12-05T06:00:55.359 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:55.343+0000 7fbaf259c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fbaf4116500 0x7fbaf4145fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:00:55.359 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:55.343+0000 7fbaf359e700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fbaf4147fd0 0x7fbaf4137980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:46230/0 (socket says 172.21.15.42:46230) 2019-12-05T06:00:55.359 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:55.343+0000 7fbaf359e700 1 -- 172.21.15.42:0/2237543583 learned_addr learned my addr 172.21.15.42:0/2237543583 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:00:55.359 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:55.343+0000 7fbaf259c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fbaf4116500 0x7fbaf4145fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:46192/0 (socket says 172.21.15.42:46192) 2019-12-05T06:00:55.359 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:55.347+0000 7fbaf259c700 1 -- 172.21.15.42:0/2237543583 >> v1:172.21.15.42:6789/0 conn(0x7fbaf4147fd0 legacy=0x7fbaf4137980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:00:55.360 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:55.347+0000 7fbaeb7fe700 1 -- 172.21.15.42:0/2237543583 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1281120859 0 0) 0x7fbaf4130530 con 0x7fbaf4147fd0 2019-12-05T06:00:55.360 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:55.347+0000 7fbaf259c700 1 -- 172.21.15.42:0/2237543583 >> v1:172.21.15.40:6789/0 conn(0x7fbaf411a770 legacy=0x7fbaf411b880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:00:55.360 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:55.347+0000 7fbaf259c700 1 -- 172.21.15.42:0/2237543583 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbaf411c080 con 0x7fbaf4116500 2019-12-05T06:00:55.360 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:55.347+0000 7fbaf259c700 1 --2- 172.21.15.42:0/2237543583 >> v2:172.21.15.42:3300/0 conn(0x7fbaf4116500 0x7fbaf4145fb0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fbaf40a4660 tx=0x7fbadc008e50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:55.360 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:55.347+0000 7fbaf259c700 1 --2- 172.21.15.42:0/2237543583 >> v2:172.21.15.42:3300/0 conn(0x7fbaf4116500 0x7fbaf4145fb0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fbaf40a4660 tx=0x7fbadc008e50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:55.360 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:55.347+0000 7fbaf259c700 1 --2- 172.21.15.42:0/2237543583 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbaf4116500 0x7fbaf4145fb0 secure :-1 s=READY pgs=1880 cs=0 l=1 rx=0x7fbaf40a4660 tx=0x7fbadc008e50).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:00:55.360 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:55.347+0000 7fbaf259c700 1 --2- 172.21.15.42:0/2237543583 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbaf4116500 0x7fbaf4145fb0 secure :-1 s=THROTTLE_DONE pgs=1880 cs=0 l=1 rx=0x7fbaf40a4660 tx=0x7fbadc008e50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:55.361 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:55.347+0000 7fbaeb7fe700 1 -- 172.21.15.42:0/2237543583 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbadc007340 con 0x7fbaf4116500 2019-12-05T06:00:55.361 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:55.347+0000 7fbaf259c700 1 --2- 172.21.15.42:0/2237543583 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbaf4116500 0x7fbaf4145fb0 secure :-1 s=THROTTLE_DONE pgs=1880 cs=0 l=1 rx=0x7fbaf40a4660 tx=0x7fbadc008e50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:55.361 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:55.347+0000 7fbaeb7fe700 1 -- 172.21.15.42:0/2237543583 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbadc011470 con 0x7fbaf4116500 2019-12-05T06:00:55.361 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:55.347+0000 7fbaf259c700 1 --2- 172.21.15.42:0/2237543583 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbaf4116500 0x7fbaf4145fb0 secure :-1 s=THROTTLE_DONE pgs=1880 cs=0 l=1 rx=0x7fbaf40a4660 tx=0x7fbadc008e50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:55.361 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:55.347+0000 7fbaf8f72700 1 -- 172.21.15.42:0/2237543583 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbaf4148400 con 0x7fbaf4116500 2019-12-05T06:00:55.361 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:55.347+0000 7fbaeb7fe700 1 -- 172.21.15.42:0/2237543583 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbadc005540 con 0x7fbaf4116500 2019-12-05T06:00:55.361 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:55.347+0000 7fbaf8f72700 1 -- 172.21.15.42:0/2237543583 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbaf4139190 con 0x7fbaf4116500 2019-12-05T06:00:55.365 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:55.347+0000 7fbaf259c700 1 --2- 172.21.15.42:0/2237543583 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbaf4116500 0x7fbaf4145fb0 secure :-1 s=THROTTLE_DONE pgs=1880 cs=0 l=1 rx=0x7fbaf40a4660 tx=0x7fbadc008e50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:55.366 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:55.347+0000 7fbaeb7fe700 1 -- 172.21.15.42:0/2237543583 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fbadc0057e0 con 0x7fbaf4116500 2019-12-05T06:00:55.366 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:55.347+0000 7fbaf259c700 1 --2- 172.21.15.42:0/2237543583 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbaf4116500 0x7fbaf4145fb0 secure :-1 s=THROTTLE_DONE pgs=1880 cs=0 l=1 rx=0x7fbaf40a4660 tx=0x7fbadc008e50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:55.374 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:55.347+0000 7fbaeb7fe700 1 --2- 172.21.15.42:0/2237543583 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbad803abf0 0x7fbad803d100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:00:55.374 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:55.347+0000 7fbaeb7fe700 1 -- 172.21.15.42:0/2237543583 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9385+0+0 (secure 0 0 0) 0x7fbadc04bcf0 con 0x7fbaf4116500 2019-12-05T06:00:55.374 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:55.347+0000 7fbaf2d9d700 1 --2- 172.21.15.42:0/2237543583 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbad803abf0 0x7fbad803d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:00:55.375 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:55.347+0000 7fbaf2d9d700 1 --2- 172.21.15.42:0/2237543583 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbad803abf0 0x7fbad803d100 crc :-1 s=READY pgs=801 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:00:55.376 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:55.355+0000 7fbaf8f72700 1 -- 172.21.15.42:0/2237543583 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbaf405d110 con 0x7fbaf4116500 2019-12-05T06:00:55.376 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:55.363+0000 7fbaf259c700 1 --2- 172.21.15.42:0/2237543583 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbaf4116500 0x7fbaf4145fb0 secure :-1 s=THROTTLE_DONE pgs=1880 cs=0 l=1 rx=0x7fbaf40a4660 tx=0x7fbadc008e50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:55.376 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:55.363+0000 7fbaeb7fe700 1 -- 172.21.15.42:0/2237543583 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fbadc032d50 con 0x7fbaf4116500 2019-12-05T06:00:55.544 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:55.539+0000 7fbaf8f72700 1 -- 172.21.15.42:0/2237543583 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbaf405b8c0 con 0x7fbaf4116500 2019-12-05T06:00:55.545 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:55.539+0000 7fbaf259c700 1 --2- 172.21.15.42:0/2237543583 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbaf4116500 0x7fbaf4145fb0 secure :-1 s=THROTTLE_DONE pgs=1880 cs=0 l=1 rx=0x7fbaf40a4660 tx=0x7fbadc008e50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:55.545 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:55.539+0000 7fbaeb7fe700 1 -- 172.21.15.42:0/2237543583 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 228 v228) v1 ==== 95+0+2108 (secure 0 0 0) 0x7fbadc011e40 con 0x7fbaf4116500 2019-12-05T06:00:55.547 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 228 2019-12-05T06:00:55.547 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:00:55.547 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":228,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6653,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/2780168808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2780168808},{"type":"v1","addr":"172.21.15.40:6833","nonce":2780168808}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":228}],"filesystems":[{"mdsmap":{"epoch":227,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:00:19.604634+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":88,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9061},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9061":{"gid":9061,"name":"a","rank":0,"incarnation":224,"state":"up:active","state_seq":9,"addr":"172.21.15.42:6835/448869597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":448869597},{"type":"v1","addr":"172.21.15.42:6835","nonce":448869597}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:00:55.551 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:55.543+0000 7fbaf8f72700 1 -- 172.21.15.42:0/2237543583 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbad803abf0 msgr2=0x7fbad803d100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:00:55.551 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:55.543+0000 7fbaf8f72700 1 --2- 172.21.15.42:0/2237543583 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbad803abf0 0x7fbad803d100 crc :-1 s=READY pgs=801 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:00:55.552 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:55.543+0000 7fbaf8f72700 1 -- 172.21.15.42:0/2237543583 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbaf4116500 msgr2=0x7fbaf4145fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:00:55.552 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:55.543+0000 7fbaf8f72700 1 --2- 172.21.15.42:0/2237543583 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbaf4116500 0x7fbaf4145fb0 secure :-1 s=READY pgs=1880 cs=0 l=1 rx=0x7fbaf40a4660 tx=0x7fbadc008e50).stop 2019-12-05T06:00:55.552 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:55.543+0000 7fbaf8f72700 1 -- 172.21.15.42:0/2237543583 shutdown_connections 2019-12-05T06:00:55.552 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:55.547+0000 7fbaf8f72700 1 --2- 172.21.15.42:0/2237543583 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbad803abf0 0x7fbad803d100 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:00:55.552 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:55.547+0000 7fbaf8f72700 1 --2- 172.21.15.42:0/2237543583 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbaf4116500 0x7fbaf4145fb0 unknown :-1 s=CLOSED pgs=1880 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:00:55.552 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:55.547+0000 7fbaf8f72700 1 -- 172.21.15.42:0/2237543583 shutdown_connections 2019-12-05T06:00:55.552 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:55.547+0000 7fbaf8f72700 1 -- 172.21.15.42:0/2237543583 wait complete. 2019-12-05T06:00:55.553 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:55.547+0000 7fbaf8f72700 1 -- 172.21.15.42:0/2237543583 >> 172.21.15.42:0/2237543583 conn(0x7fbaf409f510 msgr2=0x7fbaf407e720 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:00:55.563 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:00:57.564 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:00:57.669 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:57.659+0000 7f8a1a434700 1 Processor -- start 2019-12-05T06:00:57.670 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:57.659+0000 7f8a1a434700 1 -- start start 2019-12-05T06:00:57.670 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:57.659+0000 7f8a1a434700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8a140a7740 0x7f8a1408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:00:57.670 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:57.659+0000 7f8a1a434700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8a1409fae0 0x7f8a1407eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:00:57.670 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:57.659+0000 7f8a1a434700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a140a0710 con 0x7f8a1407e720 2019-12-05T06:00:57.670 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:57.659+0000 7f8a1a434700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a140a0b00 con 0x7f8a140a7740 2019-12-05T06:00:57.670 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:57.659+0000 7f8a1a434700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a1408e5b0 con 0x7f8a1409fae0 2019-12-05T06:00:57.670 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:57.659+0000 7f8a0bfff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f8a1407e720 0x7f8a1409f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:46242/0 (socket says 172.21.15.42:46242) 2019-12-05T06:00:57.671 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:57.659+0000 7f8a0bfff700 1 -- 172.21.15.42:0/3740397990 learned_addr learned my addr 172.21.15.42:0/3740397990 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:00:57.671 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:57.659+0000 7f8a191d2700 1 --2- 172.21.15.42:0/3740397990 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8a1409fae0 0x7f8a1407eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:00:57.671 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:57.659+0000 7f8a189d1700 1 --2- 172.21.15.42:0/3740397990 >> v2:172.21.15.40:3300/0 conn(0x7f8a140a7740 0x7f8a1408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:00:57.671 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:57.659+0000 7f8a0b7fe700 1 -- 172.21.15.42:0/3740397990 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2250488057 0 0) 0x7f8a140a0710 con 0x7f8a1407e720 2019-12-05T06:00:57.671 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:57.659+0000 7f8a0b7fe700 1 -- 172.21.15.42:0/3740397990 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89fc0030b0 con 0x7f8a1407e720 2019-12-05T06:00:57.671 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:57.659+0000 7f8a191d2700 1 -- 172.21.15.42:0/3740397990 >> v2:172.21.15.40:3300/0 conn(0x7f8a140a7740 msgr2=0x7f8a1408df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:00:57.671 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:57.659+0000 7f8a191d2700 1 --2- 172.21.15.42:0/3740397990 >> v2:172.21.15.40:3300/0 conn(0x7f8a140a7740 0x7f8a1408df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:00:57.672 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:57.659+0000 7f8a191d2700 1 -- 172.21.15.42:0/3740397990 >> v1:172.21.15.42:6789/0 conn(0x7f8a1407e720 legacy=0x7f8a1409f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:00:57.672 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:57.659+0000 7f8a191d2700 1 -- 172.21.15.42:0/3740397990 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a1407f0a0 con 0x7f8a1409fae0 2019-12-05T06:00:57.672 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:57.659+0000 7f8a191d2700 1 --2- 172.21.15.42:0/3740397990 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8a1409fae0 0x7f8a1407eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f8a0c008b10 tx=0x7f8a0c00fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:57.672 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:57.659+0000 7f8a191d2700 1 --2- 172.21.15.42:0/3740397990 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8a1409fae0 0x7f8a1407eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f8a0c008b10 tx=0x7f8a0c00fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:57.673 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:57.663+0000 7f8a191d2700 1 --2- 172.21.15.42:0/3740397990 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8a1409fae0 0x7f8a1407eb50 secure :-1 s=READY pgs=419 cs=0 l=1 rx=0x7f8a0c008b10 tx=0x7f8a0c00fff0).ready entity=mon.2 client_cookie=ca75c75ccb135666 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:00:57.673 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:57.663+0000 7f8a191d2700 1 --2- 172.21.15.42:0/3740397990 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8a1409fae0 0x7f8a1407eb50 secure :-1 s=THROTTLE_DONE pgs=419 cs=0 l=1 rx=0x7f8a0c008b10 tx=0x7f8a0c00fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:57.673 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:57.663+0000 7f8a191d2700 1 --2- 172.21.15.42:0/3740397990 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8a1409fae0 0x7f8a1407eb50 secure :-1 s=THROTTLE_DONE pgs=419 cs=0 l=1 rx=0x7f8a0c008b10 tx=0x7f8a0c00fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:57.673 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:57.663+0000 7f8a0b7fe700 1 -- 172.21.15.42:0/3740397990 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8a0c010900 con 0x7f8a1409fae0 2019-12-05T06:00:57.673 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:57.663+0000 7f8a191d2700 1 --2- 172.21.15.42:0/3740397990 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8a1409fae0 0x7f8a1407eb50 secure :-1 s=THROTTLE_DONE pgs=419 cs=0 l=1 rx=0x7f8a0c008b10 tx=0x7f8a0c00fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:57.674 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:57.663+0000 7f8a0b7fe700 1 -- 172.21.15.42:0/3740397990 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a0c008b50 con 0x7f8a1409fae0 2019-12-05T06:00:57.674 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:57.663+0000 7f8a0b7fe700 1 -- 172.21.15.42:0/3740397990 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8a0c0110c0 con 0x7f8a1409fae0 2019-12-05T06:00:57.674 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:57.663+0000 7f8a1a434700 1 -- 172.21.15.42:0/3740397990 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8a1409fae0 msgr2=0x7f8a1407eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:00:57.674 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:57.663+0000 7f8a1a434700 1 --2- 172.21.15.42:0/3740397990 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8a1409fae0 0x7f8a1407eb50 secure :-1 s=READY pgs=419 cs=0 l=1 rx=0x7f8a0c008b10 tx=0x7f8a0c00fff0).stop 2019-12-05T06:00:57.674 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:57.663+0000 7f8a1a434700 1 -- 172.21.15.42:0/3740397990 shutdown_connections 2019-12-05T06:00:57.674 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:57.663+0000 7f8a1a434700 1 --2- 172.21.15.42:0/3740397990 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8a1409fae0 0x7f8a1407eb50 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:00:57.674 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:57.663+0000 7f8a1a434700 1 --2- 172.21.15.42:0/3740397990 >> v2:172.21.15.40:3300/0 conn(0x7f8a140a7740 0x7f8a1408df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:00:57.675 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:57.663+0000 7f8a1a434700 1 -- 172.21.15.42:0/3740397990 shutdown_connections 2019-12-05T06:00:57.675 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:57.663+0000 7f8a1a434700 1 -- 172.21.15.42:0/3740397990 wait complete. 2019-12-05T06:00:57.675 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:57.663+0000 7f8a1a434700 1 -- 172.21.15.42:0/3740397990 >> 172.21.15.42:0/3740397990 conn(0x7f8a140aabe0 msgr2=0x7f8a140a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:00:57.679 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:57.663+0000 7f8a1a434700 1 Processor -- start 2019-12-05T06:00:57.680 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:57.663+0000 7f8a1a434700 1 -- start start 2019-12-05T06:00:57.680 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:57.667+0000 7f8a1a434700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8a141321f0 0x7f8a1408eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:00:57.680 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:57.667+0000 7f8a1a434700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8a140a5160 0x7f8a141389f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:00:57.680 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:57.667+0000 7f8a1a434700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a1408f0c0 con 0x7f8a140a4d30 2019-12-05T06:00:57.680 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:57.667+0000 7f8a1a434700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a140a5830 con 0x7f8a140a5160 2019-12-05T06:00:57.680 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:57.667+0000 7f8a1a434700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a1411ae90 con 0x7f8a141321f0 2019-12-05T06:00:57.680 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:57.667+0000 7f8a189d1700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8a141321f0 0x7f8a1408eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:00:57.681 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:57.667+0000 7f8a191d2700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8a140a5160 0x7f8a141389f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:00:57.681 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:57.667+0000 7f8a189d1700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8a141321f0 0x7f8a1408eb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:46674/0 (socket says 172.21.15.42:46674) 2019-12-05T06:00:57.681 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:57.667+0000 7f8a189d1700 1 -- 172.21.15.42:0/2801050562 learned_addr learned my addr 172.21.15.42:0/2801050562 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:00:57.681 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:57.667+0000 7f8a189d1700 1 -- 172.21.15.42:0/2801050562 >> v2:172.21.15.40:3300/0 conn(0x7f8a140a5160 msgr2=0x7f8a141389f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:00:57.681 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:57.667+0000 7f8a189d1700 1 --2- 172.21.15.42:0/2801050562 >> v2:172.21.15.40:3300/0 conn(0x7f8a140a5160 0x7f8a141389f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:00:57.681 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:57.667+0000 7f8a189d1700 1 -- 172.21.15.42:0/2801050562 >> v1:172.21.15.40:6789/0 conn(0x7f8a140a4d30 legacy=0x7f8a1411b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:00:57.681 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:57.667+0000 7f8a097fa700 1 -- 172.21.15.42:0/2801050562 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1475135093 0 0) 0x7f8a1408f0c0 con 0x7f8a140a4d30 2019-12-05T06:00:57.682 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:57.667+0000 7f8a189d1700 1 -- 172.21.15.42:0/2801050562 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a1411bbd0 con 0x7f8a141321f0 2019-12-05T06:00:57.682 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:57.667+0000 7f8a191d2700 1 --2- 172.21.15.42:0/2801050562 >> v2:172.21.15.40:3300/0 conn(0x7f8a140a5160 0x7f8a141389f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:00:57.682 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:57.667+0000 7f8a189d1700 1 --2- 172.21.15.42:0/2801050562 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8a141321f0 0x7f8a1408eb70 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f8a10003070 tx=0x7f8a10010ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:57.682 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:57.667+0000 7f8a189d1700 1 --2- 172.21.15.42:0/2801050562 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8a141321f0 0x7f8a1408eb70 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f8a10003070 tx=0x7f8a10010ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:57.682 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:57.667+0000 7f8a189d1700 1 --2- 172.21.15.42:0/2801050562 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8a141321f0 0x7f8a1408eb70 secure :-1 s=READY pgs=420 cs=0 l=1 rx=0x7f8a10003070 tx=0x7f8a10010ae0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:00:57.682 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:57.667+0000 7f8a189d1700 1 --2- 172.21.15.42:0/2801050562 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8a141321f0 0x7f8a1408eb70 secure :-1 s=THROTTLE_DONE pgs=420 cs=0 l=1 rx=0x7f8a10003070 tx=0x7f8a10010ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:57.683 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:57.667+0000 7f8a097fa700 1 -- 172.21.15.42:0/2801050562 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8a10005930 con 0x7f8a141321f0 2019-12-05T06:00:57.683 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:57.667+0000 7f8a189d1700 1 --2- 172.21.15.42:0/2801050562 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8a141321f0 0x7f8a1408eb70 secure :-1 s=THROTTLE_DONE pgs=420 cs=0 l=1 rx=0x7f8a10003070 tx=0x7f8a10010ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:57.683 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:57.667+0000 7f8a189d1700 1 --2- 172.21.15.42:0/2801050562 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8a141321f0 0x7f8a1408eb70 secure :-1 s=THROTTLE_DONE pgs=420 cs=0 l=1 rx=0x7f8a10003070 tx=0x7f8a10010ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:57.683 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:57.667+0000 7f8a097fa700 1 -- 172.21.15.42:0/2801050562 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a10007160 con 0x7f8a141321f0 2019-12-05T06:00:57.684 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:57.667+0000 7f8a097fa700 1 -- 172.21.15.42:0/2801050562 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8a10012160 con 0x7f8a141321f0 2019-12-05T06:00:57.684 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:57.667+0000 7f8a1a434700 1 -- 172.21.15.42:0/2801050562 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a141315a0 con 0x7f8a141321f0 2019-12-05T06:00:57.684 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:57.667+0000 7f8a1a434700 1 -- 172.21.15.42:0/2801050562 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a1408f8c0 con 0x7f8a141321f0 2019-12-05T06:00:57.684 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:57.671+0000 7f8a189d1700 1 --2- 172.21.15.42:0/2801050562 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8a141321f0 0x7f8a1408eb70 secure :-1 s=THROTTLE_DONE pgs=420 cs=0 l=1 rx=0x7f8a10003070 tx=0x7f8a10010ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:57.684 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:57.671+0000 7f8a097fa700 1 -- 172.21.15.42:0/2801050562 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f8a10012430 con 0x7f8a141321f0 2019-12-05T06:00:57.685 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:57.671+0000 7f8a189d1700 1 --2- 172.21.15.42:0/2801050562 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8a141321f0 0x7f8a1408eb70 secure :-1 s=THROTTLE_DONE pgs=420 cs=0 l=1 rx=0x7f8a10003070 tx=0x7f8a10010ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:57.685 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:57.671+0000 7f8a097fa700 1 --2- 172.21.15.42:0/2801050562 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f89f403ad10 0x7f89f403d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:00:57.685 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:57.671+0000 7f8a097fa700 1 -- 172.21.15.42:0/2801050562 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9385+0+0 (secure 0 0 0) 0x7f8a1004aea0 con 0x7f8a141321f0 2019-12-05T06:00:57.685 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:57.671+0000 7f8a0bfff700 1 --2- 172.21.15.42:0/2801050562 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f89f403ad10 0x7f89f403d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:00:57.685 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:57.671+0000 7f8a0bfff700 1 --2- 172.21.15.42:0/2801050562 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f89f403ad10 0x7f89f403d220 crc :-1 s=READY pgs=802 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:00:57.685 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:57.675+0000 7f8a1a434700 1 -- 172.21.15.42:0/2801050562 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a1405d110 con 0x7f8a141321f0 2019-12-05T06:00:57.689 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:57.683+0000 7f8a189d1700 1 --2- 172.21.15.42:0/2801050562 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8a141321f0 0x7f8a1408eb70 secure :-1 s=THROTTLE_DONE pgs=420 cs=0 l=1 rx=0x7f8a10003070 tx=0x7f8a10010ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:57.689 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:57.683+0000 7f8a097fa700 1 -- 172.21.15.42:0/2801050562 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f8a10031e80 con 0x7f8a141321f0 2019-12-05T06:00:57.862 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:57.855+0000 7f8a1a434700 1 -- 172.21.15.42:0/2801050562 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8a140a8cb0 con 0x7f8a141321f0 2019-12-05T06:00:57.862 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:57.855+0000 7f8a189d1700 1 --2- 172.21.15.42:0/2801050562 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8a141321f0 0x7f8a1408eb70 secure :-1 s=THROTTLE_DONE pgs=420 cs=0 l=1 rx=0x7f8a10003070 tx=0x7f8a10010ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:57.863 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:57.855+0000 7f8a097fa700 1 -- 172.21.15.42:0/2801050562 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 228 v228) v1 ==== 95+0+2108 (secure 0 0 0) 0x7f8a100320e0 con 0x7f8a141321f0 2019-12-05T06:00:57.864 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 228 2019-12-05T06:00:57.865 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:00:57.865 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":228,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6653,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/2780168808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2780168808},{"type":"v1","addr":"172.21.15.40:6833","nonce":2780168808}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":228}],"filesystems":[{"mdsmap":{"epoch":227,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:00:19.604634+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":88,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9061},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9061":{"gid":9061,"name":"a","rank":0,"incarnation":224,"state":"up:active","state_seq":9,"addr":"172.21.15.42:6835/448869597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":448869597},{"type":"v1","addr":"172.21.15.42:6835","nonce":448869597}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:00:57.867 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:57.859+0000 7f8a1a434700 1 -- 172.21.15.42:0/2801050562 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f89f403ad10 msgr2=0x7f89f403d220 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:00:57.867 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:57.859+0000 7f8a1a434700 1 --2- 172.21.15.42:0/2801050562 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f89f403ad10 0x7f89f403d220 crc :-1 s=READY pgs=802 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:00:57.867 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:57.859+0000 7f8a1a434700 1 -- 172.21.15.42:0/2801050562 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8a141321f0 msgr2=0x7f8a1408eb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:00:57.868 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:57.859+0000 7f8a1a434700 1 --2- 172.21.15.42:0/2801050562 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8a141321f0 0x7f8a1408eb70 secure :-1 s=READY pgs=420 cs=0 l=1 rx=0x7f8a10003070 tx=0x7f8a10010ae0).stop 2019-12-05T06:00:57.868 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:57.859+0000 7f8a1a434700 1 -- 172.21.15.42:0/2801050562 shutdown_connections 2019-12-05T06:00:57.868 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:57.859+0000 7f8a1a434700 1 --2- 172.21.15.42:0/2801050562 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f89f403ad10 0x7f89f403d220 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:00:57.868 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:57.859+0000 7f8a1a434700 1 --2- 172.21.15.42:0/2801050562 >> v2:172.21.15.40:3300/0 conn(0x7f8a140a5160 0x7f8a141389f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:00:57.868 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:57.859+0000 7f8a1a434700 1 --2- 172.21.15.42:0/2801050562 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8a141321f0 0x7f8a1408eb70 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:00:57.868 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:57.863+0000 7f8a1a434700 1 -- 172.21.15.42:0/2801050562 shutdown_connections 2019-12-05T06:00:57.868 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:57.863+0000 7f8a1a434700 1 -- 172.21.15.42:0/2801050562 wait complete. 2019-12-05T06:00:57.869 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:57.863+0000 7f8a1a434700 1 -- 172.21.15.42:0/2801050562 >> 172.21.15.42:0/2801050562 conn(0x7f8a140a81f0 msgr2=0x7f8a1407e580 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:00:57.877 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:00:59.877 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:00:59.990 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:59.979+0000 7fe28e708700 1 Processor -- start 2019-12-05T06:00:59.990 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:59.979+0000 7fe28e708700 1 -- start start 2019-12-05T06:00:59.991 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:59.979+0000 7fe28e708700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe2880a7740 0x7fe28808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:00:59.991 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:59.979+0000 7fe28e708700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe28809fae0 0x7fe28807eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:00:59.991 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:59.979+0000 7fe28e708700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2880a0710 con 0x7fe28807e720 2019-12-05T06:00:59.991 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:59.979+0000 7fe28e708700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2880a0b00 con 0x7fe28809fae0 2019-12-05T06:00:59.991 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:59.979+0000 7fe28e708700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe28808e5b0 con 0x7fe2880a7740 2019-12-05T06:00:59.991 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:59.979+0000 7fe28cca5700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe2880a7740 0x7fe28808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:00:59.991 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:59.979+0000 7fe28d4a6700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe28809fae0 0x7fe28807eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:00:59.992 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:59.979+0000 7fe28cca5700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe2880a7740 0x7fe28808df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:46684/0 (socket says 172.21.15.42:46684) 2019-12-05T06:00:59.992 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:59.979+0000 7fe28cca5700 1 -- 172.21.15.42:0/3872174135 learned_addr learned my addr 172.21.15.42:0/3872174135 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:00:59.992 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:59.983+0000 7fe28d4a6700 1 -- 172.21.15.42:0/3872174135 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe2880a7740 msgr2=0x7fe28808df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:00:59.992 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:59.983+0000 7fe28d4a6700 1 --2- 172.21.15.42:0/3872174135 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe2880a7740 0x7fe28808df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:00:59.992 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:59.983+0000 7fe28d4a6700 1 -- 172.21.15.42:0/3872174135 >> v1:172.21.15.40:6789/0 conn(0x7fe28807e720 legacy=0x7fe28809f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:00:59.992 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:59.983+0000 7fe2877fe700 1 -- 172.21.15.42:0/3872174135 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1879822577 0 0) 0x7fe2880a0710 con 0x7fe28807e720 2019-12-05T06:00:59.992 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:59.983+0000 7fe28d4a6700 1 -- 172.21.15.42:0/3872174135 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe28807f0a0 con 0x7fe28809fae0 2019-12-05T06:00:59.993 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:59.983+0000 7fe28cca5700 1 --2- 172.21.15.42:0/3872174135 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe2880a7740 0x7fe28808df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:00:59.993 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:59.983+0000 7fe28d4a6700 1 --2- 172.21.15.42:0/3872174135 >> v2:172.21.15.40:3300/0 conn(0x7fe28809fae0 0x7fe28807eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fe280008b10 tx=0x7fe28000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:59.993 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:59.983+0000 7fe28d4a6700 1 --2- 172.21.15.42:0/3872174135 >> v2:172.21.15.40:3300/0 conn(0x7fe28809fae0 0x7fe28807eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fe280008b10 tx=0x7fe28000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:59.993 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:59.983+0000 7fe28d4a6700 1 --2- 172.21.15.42:0/3872174135 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe28809fae0 0x7fe28807eb50 secure :-1 s=READY pgs=1440 cs=0 l=1 rx=0x7fe280008b10 tx=0x7fe28000fff0).ready entity=mon.0 client_cookie=398fd8e2bf9198f6 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:00:59.993 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:59.983+0000 7fe28d4a6700 1 --2- 172.21.15.42:0/3872174135 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe28809fae0 0x7fe28807eb50 secure :-1 s=THROTTLE_DONE pgs=1440 cs=0 l=1 rx=0x7fe280008b10 tx=0x7fe28000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:59.993 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:59.983+0000 7fe28d4a6700 1 --2- 172.21.15.42:0/3872174135 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe28809fae0 0x7fe28807eb50 secure :-1 s=THROTTLE_DONE pgs=1440 cs=0 l=1 rx=0x7fe280008b10 tx=0x7fe28000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:59.993 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:59.983+0000 7fe2877fe700 1 -- 172.21.15.42:0/3872174135 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe280010900 con 0x7fe28809fae0 2019-12-05T06:00:59.994 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:59.983+0000 7fe28d4a6700 1 --2- 172.21.15.42:0/3872174135 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe28809fae0 0x7fe28807eb50 secure :-1 s=THROTTLE_DONE pgs=1440 cs=0 l=1 rx=0x7fe280008b10 tx=0x7fe28000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:00:59.994 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:59.983+0000 7fe2877fe700 1 -- 172.21.15.42:0/3872174135 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe280008b50 con 0x7fe28809fae0 2019-12-05T06:00:59.994 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:59.983+0000 7fe2877fe700 1 -- 172.21.15.42:0/3872174135 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe2800110c0 con 0x7fe28809fae0 2019-12-05T06:00:59.994 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:59.983+0000 7fe28e708700 1 -- 172.21.15.42:0/3872174135 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe28809fae0 msgr2=0x7fe28807eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:00:59.994 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:59.983+0000 7fe28e708700 1 --2- 172.21.15.42:0/3872174135 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe28809fae0 0x7fe28807eb50 secure :-1 s=READY pgs=1440 cs=0 l=1 rx=0x7fe280008b10 tx=0x7fe28000fff0).stop 2019-12-05T06:00:59.994 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:59.983+0000 7fe28e708700 1 -- 172.21.15.42:0/3872174135 shutdown_connections 2019-12-05T06:00:59.994 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:59.983+0000 7fe28e708700 1 --2- 172.21.15.42:0/3872174135 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe28809fae0 0x7fe28807eb50 unknown :-1 s=CLOSED pgs=1440 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:00:59.995 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:59.983+0000 7fe28e708700 1 --2- 172.21.15.42:0/3872174135 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe2880a7740 0x7fe28808df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:00:59.995 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:59.983+0000 7fe28e708700 1 -- 172.21.15.42:0/3872174135 shutdown_connections 2019-12-05T06:00:59.995 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:59.983+0000 7fe28e708700 1 -- 172.21.15.42:0/3872174135 wait complete. 2019-12-05T06:00:59.995 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:59.983+0000 7fe28e708700 1 -- 172.21.15.42:0/3872174135 >> 172.21.15.42:0/3872174135 conn(0x7fe2880aabe0 msgr2=0x7fe2880a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:00:59.998 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:59.987+0000 7fe28e708700 1 Processor -- start 2019-12-05T06:00:59.999 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:59.987+0000 7fe28e708700 1 -- start start 2019-12-05T06:00:59.999 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:59.987+0000 7fe28e708700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe28807da60 0x7fe2881448c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:00:59.999 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:59.987+0000 7fe28e708700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe28807de90 0x7fe288144e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:00:59.999 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:59.987+0000 7fe28e708700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe288142e20 con 0x7fe2881433b0 2019-12-05T06:00:59.999 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:59.987+0000 7fe28e708700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe288143ef0 con 0x7fe28807de90 2019-12-05T06:00:59.999 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:59.987+0000 7fe28e708700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe28811b9c0 con 0x7fe28807da60 2019-12-05T06:00:59.999 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:59.987+0000 7fe28cca5700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fe2881433b0 0x7fe288120e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:46262/0 (socket says 172.21.15.42:46262) 2019-12-05T06:01:00.000 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:59.987+0000 7fe28cca5700 1 -- 172.21.15.42:0/1824347059 learned_addr learned my addr 172.21.15.42:0/1824347059 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:01:00.000 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:59.987+0000 7fe287fff700 1 --2- 172.21.15.42:0/1824347059 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe28807da60 0x7fe2881448c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:01:00.000 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:59.987+0000 7fe28d4a6700 1 --2- 172.21.15.42:0/1824347059 >> v2:172.21.15.40:3300/0 conn(0x7fe28807de90 0x7fe288144e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:01:00.000 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:59.987+0000 7fe2857fa700 1 -- 172.21.15.42:0/1824347059 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4267969854 0 0) 0x7fe288142e20 con 0x7fe2881433b0 2019-12-05T06:01:00.000 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:59.987+0000 7fe2857fa700 1 -- 172.21.15.42:0/1824347059 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2640030b0 con 0x7fe2881433b0 2019-12-05T06:01:00.000 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:59.987+0000 7fe28d4a6700 1 -- 172.21.15.42:0/1824347059 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe28807da60 msgr2=0x7fe2881448c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:01:00.000 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:59.987+0000 7fe28d4a6700 1 --2- 172.21.15.42:0/1824347059 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe28807da60 0x7fe2881448c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:01:00.001 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:59.987+0000 7fe28d4a6700 1 -- 172.21.15.42:0/1824347059 >> v1:172.21.15.42:6789/0 conn(0x7fe2881433b0 legacy=0x7fe288120e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:01:00.001 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:59.987+0000 7fe28d4a6700 1 -- 172.21.15.42:0/1824347059 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe288135590 con 0x7fe28807de90 2019-12-05T06:01:00.001 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:59.987+0000 7fe28d4a6700 1 --2- 172.21.15.42:0/1824347059 >> v2:172.21.15.40:3300/0 conn(0x7fe28807de90 0x7fe288144e10 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fe280008b10 tx=0x7fe280007a60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:00.001 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:59.987+0000 7fe28d4a6700 1 --2- 172.21.15.42:0/1824347059 >> v2:172.21.15.40:3300/0 conn(0x7fe28807de90 0x7fe288144e10 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fe280008b10 tx=0x7fe280007a60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:00.001 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:59.987+0000 7fe28d4a6700 1 --2- 172.21.15.42:0/1824347059 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe28807de90 0x7fe288144e10 secure :-1 s=READY pgs=1441 cs=0 l=1 rx=0x7fe280008b10 tx=0x7fe280007a60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:01:00.001 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:59.987+0000 7fe28d4a6700 1 --2- 172.21.15.42:0/1824347059 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe28807de90 0x7fe288144e10 secure :-1 s=THROTTLE_DONE pgs=1441 cs=0 l=1 rx=0x7fe280008b10 tx=0x7fe280007a60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:00.002 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:59.987+0000 7fe2857fa700 1 -- 172.21.15.42:0/1824347059 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe280005ab0 con 0x7fe28807de90 2019-12-05T06:01:00.002 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:59.987+0000 7fe28d4a6700 1 --2- 172.21.15.42:0/1824347059 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe28807de90 0x7fe288144e10 secure :-1 s=THROTTLE_DONE pgs=1441 cs=0 l=1 rx=0x7fe280008b10 tx=0x7fe280007a60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:00.002 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:59.987+0000 7fe28d4a6700 1 --2- 172.21.15.42:0/1824347059 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe28807de90 0x7fe288144e10 secure :-1 s=THROTTLE_DONE pgs=1441 cs=0 l=1 rx=0x7fe280008b10 tx=0x7fe280007a60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:00.002 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:59.987+0000 7fe2857fa700 1 -- 172.21.15.42:0/1824347059 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe280010b00 con 0x7fe28807de90 2019-12-05T06:01:00.002 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:59.987+0000 7fe28e708700 1 -- 172.21.15.42:0/1824347059 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe288134580 con 0x7fe28807de90 2019-12-05T06:01:00.002 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:59.991+0000 7fe2857fa700 1 -- 172.21.15.42:0/1824347059 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe28000f680 con 0x7fe28807de90 2019-12-05T06:01:00.003 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:59.991+0000 7fe28e708700 1 -- 172.21.15.42:0/1824347059 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe288134b00 con 0x7fe28807de90 2019-12-05T06:01:00.003 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:59.991+0000 7fe28d4a6700 1 --2- 172.21.15.42:0/1824347059 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe28807de90 0x7fe288144e10 secure :-1 s=THROTTLE_DONE pgs=1441 cs=0 l=1 rx=0x7fe280008b10 tx=0x7fe280007a60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:00.003 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:59.991+0000 7fe2857fa700 1 -- 172.21.15.42:0/1824347059 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fe2800113d0 con 0x7fe28807de90 2019-12-05T06:01:00.003 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:59.991+0000 7fe28d4a6700 1 --2- 172.21.15.42:0/1824347059 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe28807de90 0x7fe288144e10 secure :-1 s=THROTTLE_DONE pgs=1441 cs=0 l=1 rx=0x7fe280008b10 tx=0x7fe280007a60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:00.003 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:59.991+0000 7fe2857fa700 1 --2- 172.21.15.42:0/1824347059 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe26403b260 0x7fe26403d770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:01:00.003 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:59.991+0000 7fe2857fa700 1 -- 172.21.15.42:0/1824347059 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9471+0+0 (secure 0 0 0) 0x7fe28004ade0 con 0x7fe28807de90 2019-12-05T06:01:00.004 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:59.991+0000 7fe28cca5700 1 --2- 172.21.15.42:0/1824347059 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe26403b260 0x7fe26403d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:01:00.004 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:59.991+0000 7fe28cca5700 1 --2- 172.21.15.42:0/1824347059 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe26403b260 0x7fe26403d770 crc :-1 s=READY pgs=803 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:01:00.004 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:59.995+0000 7fe28e708700 1 -- 172.21.15.42:0/1824347059 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe28805d110 con 0x7fe28807de90 2019-12-05T06:01:00.007 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:59.999+0000 7fe28d4a6700 1 --2- 172.21.15.42:0/1824347059 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe28807de90 0x7fe288144e10 secure :-1 s=THROTTLE_DONE pgs=1441 cs=0 l=1 rx=0x7fe280008b10 tx=0x7fe280007a60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:00.007 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:00:59.999+0000 7fe2857fa700 1 -- 172.21.15.42:0/1824347059 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fe28000f990 con 0x7fe28807de90 2019-12-05T06:01:00.170 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:00.163+0000 7fe28e708700 1 -- 172.21.15.42:0/1824347059 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe28805b8c0 con 0x7fe28807de90 2019-12-05T06:01:00.170 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:00.163+0000 7fe28d4a6700 1 --2- 172.21.15.42:0/1824347059 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe28807de90 0x7fe288144e10 secure :-1 s=THROTTLE_DONE pgs=1441 cs=0 l=1 rx=0x7fe280008b10 tx=0x7fe280007a60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:00.171 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:00.163+0000 7fe2857fa700 1 -- 172.21.15.42:0/1824347059 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 229 v229) v1 ==== 95+0+1774 (secure 0 0 0) 0x7fe280008290 con 0x7fe28807de90 2019-12-05T06:01:00.171 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:01:00.171 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":229,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":229,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:00:59.319594+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":89,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6653},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6653":{"gid":6653,"name":"b","rank":0,"incarnation":229,"state":"up:replay","state_seq":2,"addr":"172.21.15.40:6833/2780168808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2780168808},{"type":"v1","addr":"172.21.15.40:6833","nonce":2780168808}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:01:00.172 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 229 2019-12-05T06:01:00.173 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:00.167+0000 7fe28e708700 1 -- 172.21.15.42:0/1824347059 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe26403b260 msgr2=0x7fe26403d770 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:01:00.174 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:00.167+0000 7fe28e708700 1 --2- 172.21.15.42:0/1824347059 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe26403b260 0x7fe26403d770 crc :-1 s=READY pgs=803 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:01:00.174 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:00.167+0000 7fe28e708700 1 -- 172.21.15.42:0/1824347059 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe28807de90 msgr2=0x7fe288144e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:01:00.174 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:00.167+0000 7fe28e708700 1 --2- 172.21.15.42:0/1824347059 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe28807de90 0x7fe288144e10 secure :-1 s=READY pgs=1441 cs=0 l=1 rx=0x7fe280008b10 tx=0x7fe280007a60).stop 2019-12-05T06:01:00.175 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:00.167+0000 7fe28e708700 1 -- 172.21.15.42:0/1824347059 shutdown_connections 2019-12-05T06:01:00.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:00.167+0000 7fe28e708700 1 --2- 172.21.15.42:0/1824347059 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe26403b260 0x7fe26403d770 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:01:00.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:00.167+0000 7fe28e708700 1 --2- 172.21.15.42:0/1824347059 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe28807de90 0x7fe288144e10 unknown :-1 s=CLOSED pgs=1441 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:01:00.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:00.167+0000 7fe28e708700 1 --2- 172.21.15.42:0/1824347059 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe28807da60 0x7fe2881448c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:01:00.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:00.167+0000 7fe28e708700 1 -- 172.21.15.42:0/1824347059 shutdown_connections 2019-12-05T06:01:00.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:00.167+0000 7fe28e708700 1 -- 172.21.15.42:0/1824347059 wait complete. 2019-12-05T06:01:00.184 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:00.167+0000 7fe28e708700 1 -- 172.21.15.42:0/1824347059 >> 172.21.15.42:0/1824347059 conn(0x7fe28807e4f0 msgr2=0x7fe28811c610 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:01:00.185 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a down, removed from mdsmap 2019-12-05T06:01:00.185 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2019-12-05T06:01:00.185 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:01:00.291 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:00.279+0000 7fb37a731700 1 Processor -- start 2019-12-05T06:01:00.292 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:00.279+0000 7fb37a731700 1 -- start start 2019-12-05T06:01:00.292 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:00.283+0000 7fb37a731700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb3740a7740 0x7fb37408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:01:00.292 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:00.283+0000 7fb37a731700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb37409fae0 0x7fb37407eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:01:00.292 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:00.283+0000 7fb37a731700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3740a0710 con 0x7fb37407e720 2019-12-05T06:01:00.292 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:00.283+0000 7fb37a731700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3740a0b00 con 0x7fb3740a7740 2019-12-05T06:01:00.293 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:00.283+0000 7fb37a731700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb37408e5b0 con 0x7fb37409fae0 2019-12-05T06:01:00.293 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:00.283+0000 7fb3794cf700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb37409fae0 0x7fb37407eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:01:00.293 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:00.283+0000 7fb3794cf700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb37409fae0 0x7fb37407eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:46236/0 (socket says 172.21.15.42:46236) 2019-12-05T06:01:00.293 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:00.283+0000 7fb3794cf700 1 -- 172.21.15.42:0/599201102 learned_addr learned my addr 172.21.15.42:0/599201102 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:01:00.293 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:00.283+0000 7fb373fff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fb37407e720 0x7fb37409f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:46270/0 (socket says 172.21.15.42:46270) 2019-12-05T06:01:00.293 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:00.283+0000 7fb378cce700 1 --2- 172.21.15.42:0/599201102 >> v2:172.21.15.40:3300/0 conn(0x7fb3740a7740 0x7fb37408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:01:00.293 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:00.283+0000 7fb3794cf700 1 -- 172.21.15.42:0/599201102 >> v2:172.21.15.40:3300/0 conn(0x7fb3740a7740 msgr2=0x7fb37408df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:01:00.294 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:00.283+0000 7fb3794cf700 1 --2- 172.21.15.42:0/599201102 >> v2:172.21.15.40:3300/0 conn(0x7fb3740a7740 0x7fb37408df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:01:00.294 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:00.283+0000 7fb3794cf700 1 -- 172.21.15.42:0/599201102 >> v1:172.21.15.42:6789/0 conn(0x7fb37407e720 legacy=0x7fb37409f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:01:00.294 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:00.283+0000 7fb3794cf700 1 -- 172.21.15.42:0/599201102 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb37407f0a0 con 0x7fb37409fae0 2019-12-05T06:01:00.294 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:00.283+0000 7fb3794cf700 1 --2- 172.21.15.42:0/599201102 >> v2:172.21.15.42:3300/0 conn(0x7fb37409fae0 0x7fb37407eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fb35c001110 tx=0x7fb35c001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:00.294 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:00.283+0000 7fb3794cf700 1 --2- 172.21.15.42:0/599201102 >> v2:172.21.15.42:3300/0 conn(0x7fb37409fae0 0x7fb37407eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fb35c001110 tx=0x7fb35c001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:00.294 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:00.283+0000 7fb3794cf700 1 --2- 172.21.15.42:0/599201102 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb37409fae0 0x7fb37407eb50 secure :-1 s=READY pgs=1884 cs=0 l=1 rx=0x7fb35c001110 tx=0x7fb35c001150).ready entity=mon.1 client_cookie=56b62bda491adc34 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:01:00.294 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:00.283+0000 7fb3794cf700 1 --2- 172.21.15.42:0/599201102 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb37409fae0 0x7fb37407eb50 secure :-1 s=THROTTLE_DONE pgs=1884 cs=0 l=1 rx=0x7fb35c001110 tx=0x7fb35c001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:00.295 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:00.283+0000 7fb3737fe700 1 -- 172.21.15.42:0/599201102 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb36c008c90 con 0x7fb37409fae0 2019-12-05T06:01:00.295 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:00.283+0000 7fb3794cf700 1 --2- 172.21.15.42:0/599201102 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb37409fae0 0x7fb37407eb50 secure :-1 s=THROTTLE_DONE pgs=1884 cs=0 l=1 rx=0x7fb35c001110 tx=0x7fb35c001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:00.295 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:00.283+0000 7fb3794cf700 1 --2- 172.21.15.42:0/599201102 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb37409fae0 0x7fb37407eb50 secure :-1 s=THROTTLE_DONE pgs=1884 cs=0 l=1 rx=0x7fb35c001110 tx=0x7fb35c001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:00.295 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:00.283+0000 7fb3737fe700 1 -- 172.21.15.42:0/599201102 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb36c0088b0 con 0x7fb37409fae0 2019-12-05T06:01:00.295 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:00.283+0000 7fb3737fe700 1 -- 172.21.15.42:0/599201102 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb36c011920 con 0x7fb37409fae0 2019-12-05T06:01:00.295 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:00.283+0000 7fb37a731700 1 -- 172.21.15.42:0/599201102 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb37409fae0 msgr2=0x7fb37407eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:01:00.295 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:00.283+0000 7fb37a731700 1 --2- 172.21.15.42:0/599201102 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb37409fae0 0x7fb37407eb50 secure :-1 s=READY pgs=1884 cs=0 l=1 rx=0x7fb35c001110 tx=0x7fb35c001150).stop 2019-12-05T06:01:00.296 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:00.283+0000 7fb37a731700 1 -- 172.21.15.42:0/599201102 shutdown_connections 2019-12-05T06:01:00.296 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:00.283+0000 7fb37a731700 1 --2- 172.21.15.42:0/599201102 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb37409fae0 0x7fb37407eb50 unknown :-1 s=CLOSED pgs=1884 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:01:00.296 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:00.283+0000 7fb37a731700 1 --2- 172.21.15.42:0/599201102 >> v2:172.21.15.40:3300/0 conn(0x7fb3740a7740 0x7fb37408df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:01:00.296 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:00.283+0000 7fb37a731700 1 -- 172.21.15.42:0/599201102 shutdown_connections 2019-12-05T06:01:00.296 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:00.283+0000 7fb37a731700 1 -- 172.21.15.42:0/599201102 wait complete. 2019-12-05T06:01:00.296 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:00.283+0000 7fb37a731700 1 -- 172.21.15.42:0/599201102 >> 172.21.15.42:0/599201102 conn(0x7fb3740aabe0 msgr2=0x7fb3740a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:01:00.300 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:00.287+0000 7fb37a731700 1 Processor -- start 2019-12-05T06:01:00.300 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:00.287+0000 7fb37a731700 1 -- start start 2019-12-05T06:01:00.300 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:00.287+0000 7fb37a731700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb374143070 0x7fb374144330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:01:00.300 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:00.287+0000 7fb37a731700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb3741244d0 0x7fb37407c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:01:00.300 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:00.287+0000 7fb37a731700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb374144880 con 0x7fb37407c0a0 2019-12-05T06:01:00.301 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:00.287+0000 7fb37a731700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb374143580 con 0x7fb374143070 2019-12-05T06:01:00.301 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:00.287+0000 7fb37a731700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb374123790 con 0x7fb3741244d0 2019-12-05T06:01:00.301 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:00.287+0000 7fb373fff700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fb37407c0a0 0x7fb374123cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:55716/0 (socket says 172.21.15.42:55716) 2019-12-05T06:01:00.301 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:00.287+0000 7fb3794cf700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb3741244d0 0x7fb37407c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:01:00.301 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:00.287+0000 7fb378cce700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb374143070 0x7fb374144330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:01:00.301 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:00.287+0000 7fb373fff700 1 -- 172.21.15.42:0/2534622880 learned_addr learned my addr 172.21.15.42:0/2534622880 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:01:00.302 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:00.287+0000 7fb3717fa700 1 -- 172.21.15.42:0/2534622880 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3844732550 0 0) 0x7fb374144880 con 0x7fb37407c0a0 2019-12-05T06:01:00.302 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:00.287+0000 7fb3717fa700 1 -- 172.21.15.42:0/2534622880 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3580030b0 con 0x7fb37407c0a0 2019-12-05T06:01:00.302 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:00.287+0000 7fb378cce700 1 -- 172.21.15.42:0/2534622880 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb3741244d0 msgr2=0x7fb37407c4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:01:00.302 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:00.287+0000 7fb378cce700 1 --2- 172.21.15.42:0/2534622880 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb3741244d0 0x7fb37407c4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:01:00.302 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:00.287+0000 7fb378cce700 1 -- 172.21.15.42:0/2534622880 >> v1:172.21.15.40:6789/0 conn(0x7fb37407c0a0 legacy=0x7fb374123cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:01:00.302 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:00.287+0000 7fb378cce700 1 -- 172.21.15.42:0/2534622880 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb374143910 con 0x7fb374143070 2019-12-05T06:01:00.303 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:00.287+0000 7fb378cce700 1 --2- 172.21.15.42:0/2534622880 >> v2:172.21.15.40:3300/0 conn(0x7fb374143070 0x7fb374144330 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fb374143740 tx=0x7fb364010e10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:00.303 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:00.287+0000 7fb378cce700 1 --2- 172.21.15.42:0/2534622880 >> v2:172.21.15.40:3300/0 conn(0x7fb374143070 0x7fb374144330 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fb374143740 tx=0x7fb364010e10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:00.303 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:00.287+0000 7fb378cce700 1 --2- 172.21.15.42:0/2534622880 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb374143070 0x7fb374144330 secure :-1 s=READY pgs=1443 cs=0 l=1 rx=0x7fb374143740 tx=0x7fb364010e10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:01:00.303 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:00.291+0000 7fb378cce700 1 --2- 172.21.15.42:0/2534622880 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb374143070 0x7fb374144330 secure :-1 s=THROTTLE_DONE pgs=1443 cs=0 l=1 rx=0x7fb374143740 tx=0x7fb364010e10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:00.303 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:00.291+0000 7fb3717fa700 1 -- 172.21.15.42:0/2534622880 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb364005ca0 con 0x7fb374143070 2019-12-05T06:01:00.304 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:00.291+0000 7fb378cce700 1 --2- 172.21.15.42:0/2534622880 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb374143070 0x7fb374144330 secure :-1 s=THROTTLE_DONE pgs=1443 cs=0 l=1 rx=0x7fb374143740 tx=0x7fb364010e10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:00.304 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:00.291+0000 7fb3717fa700 1 -- 172.21.15.42:0/2534622880 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3640058c0 con 0x7fb374143070 2019-12-05T06:01:00.304 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:00.291+0000 7fb37a731700 1 -- 172.21.15.42:0/2534622880 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb37412e1a0 con 0x7fb374143070 2019-12-05T06:01:00.304 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:00.291+0000 7fb378cce700 1 --2- 172.21.15.42:0/2534622880 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb374143070 0x7fb374144330 secure :-1 s=THROTTLE_DONE pgs=1443 cs=0 l=1 rx=0x7fb374143740 tx=0x7fb364010e10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:00.305 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:00.291+0000 7fb3717fa700 1 -- 172.21.15.42:0/2534622880 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb364011bf0 con 0x7fb374143070 2019-12-05T06:01:00.305 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:00.291+0000 7fb37a731700 1 -- 172.21.15.42:0/2534622880 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb37412e720 con 0x7fb374143070 2019-12-05T06:01:00.305 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:00.291+0000 7fb378cce700 1 --2- 172.21.15.42:0/2534622880 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb374143070 0x7fb374144330 secure :-1 s=THROTTLE_DONE pgs=1443 cs=0 l=1 rx=0x7fb374143740 tx=0x7fb364010e10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:00.305 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:00.291+0000 7fb3717fa700 1 -- 172.21.15.42:0/2534622880 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fb36400c3f0 con 0x7fb374143070 2019-12-05T06:01:00.306 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:00.291+0000 7fb378cce700 1 --2- 172.21.15.42:0/2534622880 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb374143070 0x7fb374144330 secure :-1 s=THROTTLE_DONE pgs=1443 cs=0 l=1 rx=0x7fb374143740 tx=0x7fb364010e10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:00.306 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:00.291+0000 7fb3717fa700 1 --2- 172.21.15.42:0/2534622880 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb35803b260 0x7fb35803d770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:01:00.306 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:00.291+0000 7fb3717fa700 1 -- 172.21.15.42:0/2534622880 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9471+0+0 (secure 0 0 0) 0x7fb36404b430 con 0x7fb374143070 2019-12-05T06:01:00.306 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:00.291+0000 7fb373fff700 1 --2- 172.21.15.42:0/2534622880 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb35803b260 0x7fb35803d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:01:00.306 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:00.291+0000 7fb373fff700 1 --2- 172.21.15.42:0/2534622880 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb35803b260 0x7fb35803d770 crc :-1 s=READY pgs=804 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:01:00.307 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:00.299+0000 7fb37a731700 1 -- 172.21.15.42:0/2534622880 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb37405d110 con 0x7fb374143070 2019-12-05T06:01:00.312 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:00.307+0000 7fb378cce700 1 --2- 172.21.15.42:0/2534622880 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb374143070 0x7fb374144330 secure :-1 s=THROTTLE_DONE pgs=1443 cs=0 l=1 rx=0x7fb374143740 tx=0x7fb364010e10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:00.312 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:00.307+0000 7fb3717fa700 1 -- 172.21.15.42:0/2534622880 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fb364032430 con 0x7fb374143070 2019-12-05T06:01:00.485 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:00.479+0000 7fb37a731700 1 -- 172.21.15.42:0/2534622880 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb37405b8c0 con 0x7fb374143070 2019-12-05T06:01:00.486 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:00.479+0000 7fb378cce700 1 --2- 172.21.15.42:0/2534622880 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb374143070 0x7fb374144330 secure :-1 s=THROTTLE_DONE pgs=1443 cs=0 l=1 rx=0x7fb374143740 tx=0x7fb364010e10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:00.488 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:00.479+0000 7fb3717fa700 1 -- 172.21.15.42:0/2534622880 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 229 v229) v1 ==== 95+0+1774 (secure 0 0 0) 0x7fb364032690 con 0x7fb374143070 2019-12-05T06:01:00.489 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 229 2019-12-05T06:01:00.489 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:01:00.489 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":229,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":229,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:00:59.319594+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":89,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6653},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6653":{"gid":6653,"name":"b","rank":0,"incarnation":229,"state":"up:replay","state_seq":2,"addr":"172.21.15.40:6833/2780168808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2780168808},{"type":"v1","addr":"172.21.15.40:6833","nonce":2780168808}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:01:00.491 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:00.483+0000 7fb37a731700 1 -- 172.21.15.42:0/2534622880 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb35803b260 msgr2=0x7fb35803d770 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:01:00.491 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:00.483+0000 7fb37a731700 1 --2- 172.21.15.42:0/2534622880 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb35803b260 0x7fb35803d770 crc :-1 s=READY pgs=804 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:01:00.491 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:00.483+0000 7fb37a731700 1 -- 172.21.15.42:0/2534622880 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb374143070 msgr2=0x7fb374144330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:01:00.491 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:00.483+0000 7fb37a731700 1 --2- 172.21.15.42:0/2534622880 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb374143070 0x7fb374144330 secure :-1 s=READY pgs=1443 cs=0 l=1 rx=0x7fb374143740 tx=0x7fb364010e10).stop 2019-12-05T06:01:00.492 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:00.483+0000 7fb37a731700 1 -- 172.21.15.42:0/2534622880 shutdown_connections 2019-12-05T06:01:00.492 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:00.483+0000 7fb37a731700 1 --2- 172.21.15.42:0/2534622880 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb35803b260 0x7fb35803d770 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:01:00.492 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:00.483+0000 7fb37a731700 1 --2- 172.21.15.42:0/2534622880 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb3741244d0 0x7fb37407c4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:01:00.492 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:00.483+0000 7fb37a731700 1 --2- 172.21.15.42:0/2534622880 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb374143070 0x7fb374144330 unknown :-1 s=CLOSED pgs=1443 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:01:00.492 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:00.483+0000 7fb37a731700 1 -- 172.21.15.42:0/2534622880 shutdown_connections 2019-12-05T06:01:00.493 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:00.483+0000 7fb37a731700 1 -- 172.21.15.42:0/2534622880 wait complete. 2019-12-05T06:01:00.493 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:00.483+0000 7fb37a731700 1 -- 172.21.15.42:0/2534622880 >> 172.21.15.42:0/2534622880 conn(0x7fb3741160b0 msgr2=0x7fb3740a7f90 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:01:00.501 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 229, "feature_flags": { "enable_multiple": false, "ever_enabled_multiple": false }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2019-12-05T05:03:35.604738+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 229, "ever_allowed_features": 2, "explicitly_allowed_features": 2, "failed": [], "flags": 18, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_6653": { "addr": "172.21.15.40:6833/2780168808", "addrs": { "addrvec": [ { "addr": "172.21.15.40:6832", "nonce": 2780168808, "type": "v2" }, { "addr": "172.21.15.40:6833", "nonce": 2780168808, "type": "v1" } ] }, "export_targets": [], "features": 4540138292836696063, "flags": 0, "gid": 6653, "incarnation": 229, "name": "b", "rank": 0, "state": "up:replay", "state_seq": 2 } }, "last_failure": 0, "last_failure_osd_epoch": 89, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 2, "min_compat_client": "0 (unknown)", "modified": "2019-12-05T06:00:59.319594+0000", "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6653 } } } ], "standbys": [] } 2019-12-05T06:01:02.503 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:01:02.613 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:02.603+0000 7f8749092700 1 Processor -- start 2019-12-05T06:01:02.613 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:02.603+0000 7f8749092700 1 -- start start 2019-12-05T06:01:02.613 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:02.603+0000 7f8749092700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f874408df80 0x7f874409f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:01:02.614 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:02.603+0000 7f8749092700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f874409f830 0x7f874414efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:01:02.614 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:02.603+0000 7f8749092700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f874409fc60 con 0x7f87440a7740 2019-12-05T06:01:02.614 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:02.603+0000 7f8749092700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87440a0b00 con 0x7f874409f830 2019-12-05T06:01:02.614 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:02.603+0000 7f8749092700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f874408e5b0 con 0x7f874408df80 2019-12-05T06:01:02.614 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:02.603+0000 7f8742d9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f874408df80 0x7f874409f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:01:02.614 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:02.603+0000 7f8742d9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f874408df80 0x7f874409f2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:46256/0 (socket says 172.21.15.42:46256) 2019-12-05T06:01:02.614 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:02.603+0000 7f8742d9d700 1 -- 172.21.15.42:0/2846381242 learned_addr learned my addr 172.21.15.42:0/2846381242 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:01:02.615 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:02.603+0000 7f8748891700 1 --2- 172.21.15.42:0/2846381242 >> v2:172.21.15.40:3300/0 conn(0x7f874409f830 0x7f874414efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:01:02.615 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:02.603+0000 7f8742d9d700 1 -- 172.21.15.42:0/2846381242 >> v2:172.21.15.40:3300/0 conn(0x7f874409f830 msgr2=0x7f874414efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:01:02.615 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:02.603+0000 7f8742d9d700 1 --2- 172.21.15.42:0/2846381242 >> v2:172.21.15.40:3300/0 conn(0x7f874409f830 0x7f874414efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:01:02.615 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:02.603+0000 7f8742d9d700 1 -- 172.21.15.42:0/2846381242 >> v1:172.21.15.40:6789/0 conn(0x7f87440a7740 legacy=0x7f874407e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:01:02.615 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:02.603+0000 7f8742d9d700 1 -- 172.21.15.42:0/2846381242 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87440a0710 con 0x7f874408df80 2019-12-05T06:01:02.615 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:02.603+0000 7f8742d9d700 1 --2- 172.21.15.42:0/2846381242 >> v2:172.21.15.42:3300/0 conn(0x7f874408df80 0x7f874409f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f87380032b0 tx=0x7f87380100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:02.615 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:02.603+0000 7f8742d9d700 1 --2- 172.21.15.42:0/2846381242 >> v2:172.21.15.42:3300/0 conn(0x7f874408df80 0x7f874409f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f87380032b0 tx=0x7f87380100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:02.616 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:02.603+0000 7f8742d9d700 1 --2- 172.21.15.42:0/2846381242 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f874408df80 0x7f874409f2e0 secure :-1 s=READY pgs=1885 cs=0 l=1 rx=0x7f87380032b0 tx=0x7f87380100b0).ready entity=mon.1 client_cookie=a7465ca75bc2f60b server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:01:02.616 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:02.603+0000 7f8742d9d700 1 --2- 172.21.15.42:0/2846381242 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f874408df80 0x7f874409f2e0 secure :-1 s=THROTTLE_DONE pgs=1885 cs=0 l=1 rx=0x7f87380032b0 tx=0x7f87380100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:02.616 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:02.603+0000 7f874259c700 1 -- 172.21.15.42:0/2846381242 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8738010d60 con 0x7f874408df80 2019-12-05T06:01:02.616 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:02.603+0000 7f8742d9d700 1 --2- 172.21.15.42:0/2846381242 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f874408df80 0x7f874409f2e0 secure :-1 s=THROTTLE_DONE pgs=1885 cs=0 l=1 rx=0x7f87380032b0 tx=0x7f87380100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:02.616 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:02.603+0000 7f8742d9d700 1 --2- 172.21.15.42:0/2846381242 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f874408df80 0x7f874409f2e0 secure :-1 s=THROTTLE_DONE pgs=1885 cs=0 l=1 rx=0x7f87380032b0 tx=0x7f87380100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:02.616 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:02.603+0000 7f874259c700 1 -- 172.21.15.42:0/2846381242 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8738010980 con 0x7f874408df80 2019-12-05T06:01:02.616 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:02.603+0000 7f874259c700 1 -- 172.21.15.42:0/2846381242 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f87380114a0 con 0x7f874408df80 2019-12-05T06:01:02.617 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:02.603+0000 7f8749092700 1 -- 172.21.15.42:0/2846381242 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f874408df80 msgr2=0x7f874409f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:01:02.617 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:02.603+0000 7f8749092700 1 --2- 172.21.15.42:0/2846381242 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f874408df80 0x7f874409f2e0 secure :-1 s=READY pgs=1885 cs=0 l=1 rx=0x7f87380032b0 tx=0x7f87380100b0).stop 2019-12-05T06:01:02.617 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:02.607+0000 7f8749092700 1 -- 172.21.15.42:0/2846381242 shutdown_connections 2019-12-05T06:01:02.617 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:02.607+0000 7f8749092700 1 --2- 172.21.15.42:0/2846381242 >> v2:172.21.15.40:3300/0 conn(0x7f874409f830 0x7f874414efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:01:02.617 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:02.607+0000 7f8749092700 1 --2- 172.21.15.42:0/2846381242 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f874408df80 0x7f874409f2e0 unknown :-1 s=CLOSED pgs=1885 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:01:02.617 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:02.607+0000 7f8749092700 1 -- 172.21.15.42:0/2846381242 shutdown_connections 2019-12-05T06:01:02.617 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:02.607+0000 7f8749092700 1 -- 172.21.15.42:0/2846381242 wait complete. 2019-12-05T06:01:02.618 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:02.607+0000 7f8749092700 1 -- 172.21.15.42:0/2846381242 >> 172.21.15.42:0/2846381242 conn(0x7f87440aabe0 msgr2=0x7f87440a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:01:02.622 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:02.607+0000 7f8749092700 1 Processor -- start 2019-12-05T06:01:02.623 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:02.607+0000 7f8749092700 1 -- start start 2019-12-05T06:01:02.623 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:02.611+0000 7f8749092700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8744063fa0 0x7f8744112560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:01:02.623 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:02.611+0000 7f8749092700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f8744125950 0x7f8744064d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:01:02.623 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:02.611+0000 7f8749092700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8744114080 0x7f87441144b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:01:02.632 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:02.611+0000 7f8749092700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8744065b70 con 0x7f8744114080 2019-12-05T06:01:02.632 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:02.611+0000 7f8749092700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87440637c0 con 0x7f8744125950 2019-12-05T06:01:02.632 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:02.611+0000 7f8749092700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8744063a60 con 0x7f8744063fa0 2019-12-05T06:01:02.632 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:02.611+0000 7f8742d9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f8744125950 0x7f8744064d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:01:02.632 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:02.611+0000 7f8742d9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f8744125950 0x7f8744064d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:46260/0 (socket says 172.21.15.42:46260) 2019-12-05T06:01:02.633 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:02.611+0000 7f8742d9d700 1 -- 172.21.15.42:0/4118179820 learned_addr learned my addr 172.21.15.42:0/4118179820 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:01:02.633 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:02.611+0000 7f8748891700 1 --2- 172.21.15.42:0/4118179820 >> v2:172.21.15.40:3300/0 conn(0x7f8744114080 0x7f87441144b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:01:02.633 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:02.611+0000 7f874359e700 1 --2- 172.21.15.42:0/4118179820 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8744063fa0 0x7f8744112560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:01:02.633 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:02.611+0000 7f8742d9d700 1 -- 172.21.15.42:0/4118179820 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8744063fa0 msgr2=0x7f8744112560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:01:02.633 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:02.611+0000 7f8742d9d700 1 --2- 172.21.15.42:0/4118179820 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8744063fa0 0x7f8744112560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:01:02.633 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:02.611+0000 7f8742d9d700 1 -- 172.21.15.42:0/4118179820 >> v2:172.21.15.40:3300/0 conn(0x7f8744114080 msgr2=0x7f87441144b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:01:02.633 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:02.611+0000 7f8742d9d700 1 --2- 172.21.15.42:0/4118179820 >> v2:172.21.15.40:3300/0 conn(0x7f8744114080 0x7f87441144b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:01:02.634 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:02.611+0000 7f8742d9d700 1 -- 172.21.15.42:0/4118179820 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87440634b0 con 0x7f8744125950 2019-12-05T06:01:02.634 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:02.611+0000 7f8742d9d700 1 --2- 172.21.15.42:0/4118179820 >> v2:172.21.15.42:3300/0 conn(0x7f8744125950 0x7f8744064d90 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f8744063980 tx=0x7f8738010f60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:02.634 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:02.611+0000 7f8742d9d700 1 --2- 172.21.15.42:0/4118179820 >> v2:172.21.15.42:3300/0 conn(0x7f8744125950 0x7f8744064d90 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f8744063980 tx=0x7f8738010f60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:02.634 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:02.611+0000 7f8742d9d700 1 --2- 172.21.15.42:0/4118179820 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8744125950 0x7f8744064d90 secure :-1 s=READY pgs=1886 cs=0 l=1 rx=0x7f8744063980 tx=0x7f8738010f60).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:01:02.634 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:02.611+0000 7f8742d9d700 1 --2- 172.21.15.42:0/4118179820 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8744125950 0x7f8744064d90 secure :-1 s=THROTTLE_DONE pgs=1886 cs=0 l=1 rx=0x7f8744063980 tx=0x7f8738010f60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:02.634 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:02.611+0000 7f872bfff700 1 -- 172.21.15.42:0/4118179820 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8738011d40 con 0x7f8744125950 2019-12-05T06:01:02.635 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:02.611+0000 7f8742d9d700 1 --2- 172.21.15.42:0/4118179820 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8744125950 0x7f8744064d90 secure :-1 s=THROTTLE_DONE pgs=1886 cs=0 l=1 rx=0x7f8744063980 tx=0x7f8738010f60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:02.635 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:02.611+0000 7f872bfff700 1 -- 172.21.15.42:0/4118179820 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87380110c0 con 0x7f8744125950 2019-12-05T06:01:02.635 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:02.611+0000 7f8742d9d700 1 --2- 172.21.15.42:0/4118179820 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8744125950 0x7f8744064d90 secure :-1 s=THROTTLE_DONE pgs=1886 cs=0 l=1 rx=0x7f8744063980 tx=0x7f8738010f60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:02.635 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:02.611+0000 7f872bfff700 1 -- 172.21.15.42:0/4118179820 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8738005ca0 con 0x7f8744125950 2019-12-05T06:01:02.635 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:02.611+0000 7f8749092700 1 -- 172.21.15.42:0/4118179820 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87440652e0 con 0x7f8744125950 2019-12-05T06:01:02.635 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:02.611+0000 7f8749092700 1 -- 172.21.15.42:0/4118179820 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8744125d80 con 0x7f8744125950 2019-12-05T06:01:02.635 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:02.611+0000 7f8742d9d700 1 --2- 172.21.15.42:0/4118179820 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8744125950 0x7f8744064d90 secure :-1 s=THROTTLE_DONE pgs=1886 cs=0 l=1 rx=0x7f8744063980 tx=0x7f8738010f60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:02.636 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:02.611+0000 7f872bfff700 1 -- 172.21.15.42:0/4118179820 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f873800f280 con 0x7f8744125950 2019-12-05T06:01:02.636 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:02.611+0000 7f8742d9d700 1 --2- 172.21.15.42:0/4118179820 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8744125950 0x7f8744064d90 secure :-1 s=THROTTLE_DONE pgs=1886 cs=0 l=1 rx=0x7f8744063980 tx=0x7f8738010f60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:02.636 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:02.615+0000 7f872bfff700 1 --2- 172.21.15.42:0/4118179820 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f872403ae40 0x7f872403d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:01:02.636 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:02.615+0000 7f872bfff700 1 -- 172.21.15.42:0/4118179820 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9471+0+0 (secure 0 0 0) 0x7f873804b7c0 con 0x7f8744125950 2019-12-05T06:01:02.636 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:02.615+0000 7f874359e700 1 --2- 172.21.15.42:0/4118179820 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f872403ae40 0x7f872403d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:01:02.636 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:02.615+0000 7f874359e700 1 --2- 172.21.15.42:0/4118179820 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f872403ae40 0x7f872403d350 crc :-1 s=READY pgs=805 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:01:02.637 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:02.619+0000 7f8749092700 1 -- 172.21.15.42:0/4118179820 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f874405d110 con 0x7f8744125950 2019-12-05T06:01:02.638 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:02.623+0000 7f8742d9d700 1 --2- 172.21.15.42:0/4118179820 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8744125950 0x7f8744064d90 secure :-1 s=THROTTLE_DONE pgs=1886 cs=0 l=1 rx=0x7f8744063980 tx=0x7f8738010f60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:02.638 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:02.623+0000 7f872bfff700 1 -- 172.21.15.42:0/4118179820 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f8738032780 con 0x7f8744125950 2019-12-05T06:01:02.805 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:02.799+0000 7f8749092700 1 -- 172.21.15.42:0/4118179820 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f874405b8c0 con 0x7f8744125950 2019-12-05T06:01:02.805 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:02.799+0000 7f8742d9d700 1 --2- 172.21.15.42:0/4118179820 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8744125950 0x7f8744064d90 secure :-1 s=THROTTLE_DONE pgs=1886 cs=0 l=1 rx=0x7f8744063980 tx=0x7f8738010f60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:02.805 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:02.799+0000 7f872bfff700 1 -- 172.21.15.42:0/4118179820 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 230 v230) v1 ==== 95+0+1778 (secure 0 0 0) 0x7f87380329e0 con 0x7f8744125950 2019-12-05T06:01:02.805 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 230 2019-12-05T06:01:02.806 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:01:02.806 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":230,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":230,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:01:02.408885+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":89,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6653},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6653":{"gid":6653,"name":"b","rank":0,"incarnation":229,"state":"up:reconnect","state_seq":11,"addr":"172.21.15.40:6833/2780168808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2780168808},{"type":"v1","addr":"172.21.15.40:6833","nonce":2780168808}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:01:02.808 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:02.799+0000 7f8749092700 1 -- 172.21.15.42:0/4118179820 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f872403ae40 msgr2=0x7f872403d350 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:01:02.808 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:02.799+0000 7f8749092700 1 --2- 172.21.15.42:0/4118179820 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f872403ae40 0x7f872403d350 crc :-1 s=READY pgs=805 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:01:02.808 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:02.803+0000 7f8749092700 1 -- 172.21.15.42:0/4118179820 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8744125950 msgr2=0x7f8744064d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:01:02.808 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:02.803+0000 7f8749092700 1 --2- 172.21.15.42:0/4118179820 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8744125950 0x7f8744064d90 secure :-1 s=READY pgs=1886 cs=0 l=1 rx=0x7f8744063980 tx=0x7f8738010f60).stop 2019-12-05T06:01:02.809 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:02.803+0000 7f8749092700 1 -- 172.21.15.42:0/4118179820 shutdown_connections 2019-12-05T06:01:02.809 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:02.803+0000 7f8749092700 1 --2- 172.21.15.42:0/4118179820 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f872403ae40 0x7f872403d350 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:01:02.809 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:02.803+0000 7f8749092700 1 --2- 172.21.15.42:0/4118179820 >> v2:172.21.15.40:3300/0 conn(0x7f8744114080 0x7f87441144b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:01:02.809 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:02.803+0000 7f8749092700 1 --2- 172.21.15.42:0/4118179820 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8744063fa0 0x7f8744112560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:01:02.809 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:02.803+0000 7f8749092700 1 --2- 172.21.15.42:0/4118179820 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8744125950 0x7f8744064d90 unknown :-1 s=CLOSED pgs=1886 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:01:02.809 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:02.803+0000 7f8749092700 1 -- 172.21.15.42:0/4118179820 shutdown_connections 2019-12-05T06:01:02.810 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:02.803+0000 7f8749092700 1 -- 172.21.15.42:0/4118179820 wait complete. 2019-12-05T06:01:02.810 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:02.803+0000 7f8749092700 1 -- 172.21.15.42:0/4118179820 >> 172.21.15.42:0/4118179820 conn(0x7f87440aabe0 msgr2=0x7f874407db50 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:01:02.818 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2019-12-05T06:01:04.820 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:01:04.947 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:04.923+0000 7f2abb476700 1 Processor -- start 2019-12-05T06:01:04.948 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:04.923+0000 7f2abb476700 1 -- start start 2019-12-05T06:01:04.948 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:04.923+0000 7f2abb476700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2ab408df70 0x7f2ab409f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:01:04.948 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:04.923+0000 7f2abb476700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ab409fc50 con 0x7f2ab40a7730 2019-12-05T06:01:04.948 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:04.923+0000 7f2abb476700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ab40a0700 con 0x7f2ab409f820 2019-12-05T06:01:04.949 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:04.923+0000 7f2abb476700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ab40a0af0 con 0x7f2ab408df70 2019-12-05T06:01:04.949 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:04.923+0000 7f2ab9212700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2ab408df70 0x7f2ab409f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:01:04.950 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:04.923+0000 7f2aba214700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f2ab409f820 0x7f2ab414efd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:46314/0 (socket says 172.21.15.42:46314) 2019-12-05T06:01:04.950 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:04.923+0000 7f2ab9212700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2ab408df70 0x7f2ab409f2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:46278/0 (socket says 172.21.15.42:46278) 2019-12-05T06:01:04.950 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:04.923+0000 7f2aba214700 1 -- 172.21.15.42:0/150869176 learned_addr learned my addr 172.21.15.42:0/150869176 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:01:04.951 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:04.923+0000 7f2ab9212700 1 -- 172.21.15.42:0/150869176 >> v1:172.21.15.42:6789/0 conn(0x7f2ab409f820 legacy=0x7f2ab414efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:01:04.951 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:04.923+0000 7f2ab9212700 1 -- 172.21.15.42:0/150869176 >> v1:172.21.15.40:6789/0 conn(0x7f2ab40a7730 legacy=0x7f2ab407e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:01:04.951 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:04.923+0000 7f2ab8a11700 1 -- 172.21.15.42:0/150869176 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3970594715 0 0) 0x7f2ab409fc50 con 0x7f2ab40a7730 2019-12-05T06:01:04.951 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:04.923+0000 7f2ab9212700 1 -- 172.21.15.42:0/150869176 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ab407ef10 con 0x7f2ab408df70 2019-12-05T06:01:04.952 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:04.923+0000 7f2ab9212700 1 --2- 172.21.15.42:0/150869176 >> v2:172.21.15.42:3300/0 conn(0x7f2ab408df70 0x7f2ab409f2d0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f2aa4008790 tx=0x7f2aa400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:04.952 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:04.927+0000 7f2ab9212700 1 --2- 172.21.15.42:0/150869176 >> v2:172.21.15.42:3300/0 conn(0x7f2ab408df70 0x7f2ab409f2d0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f2aa4008790 tx=0x7f2aa400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:04.952 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:04.927+0000 7f2ab9212700 1 --2- 172.21.15.42:0/150869176 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2ab408df70 0x7f2ab409f2d0 secure :-1 s=READY pgs=1888 cs=0 l=1 rx=0x7f2aa4008790 tx=0x7f2aa400fbf0).ready entity=mon.1 client_cookie=14cd24dbfd299295 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:01:04.953 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:04.927+0000 7f2ab9212700 1 --2- 172.21.15.42:0/150869176 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2ab408df70 0x7f2ab409f2d0 secure :-1 s=THROTTLE_DONE pgs=1888 cs=0 l=1 rx=0x7f2aa4008790 tx=0x7f2aa400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:04.954 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:04.927+0000 7f2ab8a11700 1 -- 172.21.15.42:0/150869176 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2aa4011270 con 0x7f2ab408df70 2019-12-05T06:01:04.954 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:04.927+0000 7f2ab9212700 1 --2- 172.21.15.42:0/150869176 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2ab408df70 0x7f2ab409f2d0 secure :-1 s=THROTTLE_DONE pgs=1888 cs=0 l=1 rx=0x7f2aa4008790 tx=0x7f2aa400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:04.954 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:04.927+0000 7f2ab9212700 1 --2- 172.21.15.42:0/150869176 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2ab408df70 0x7f2ab409f2d0 secure :-1 s=THROTTLE_DONE pgs=1888 cs=0 l=1 rx=0x7f2aa4008790 tx=0x7f2aa400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:04.955 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:04.927+0000 7f2ab8a11700 1 -- 172.21.15.42:0/150869176 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2aa4008bc0 con 0x7f2ab408df70 2019-12-05T06:01:04.955 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:04.927+0000 7f2ab8a11700 1 -- 172.21.15.42:0/150869176 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2aa4011850 con 0x7f2ab408df70 2019-12-05T06:01:04.955 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:04.927+0000 7f2abb476700 1 -- 172.21.15.42:0/150869176 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2ab408df70 msgr2=0x7f2ab409f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:01:04.955 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:04.927+0000 7f2abb476700 1 --2- 172.21.15.42:0/150869176 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2ab408df70 0x7f2ab409f2d0 secure :-1 s=READY pgs=1888 cs=0 l=1 rx=0x7f2aa4008790 tx=0x7f2aa400fbf0).stop 2019-12-05T06:01:04.956 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:04.927+0000 7f2abb476700 1 -- 172.21.15.42:0/150869176 shutdown_connections 2019-12-05T06:01:04.956 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:04.927+0000 7f2abb476700 1 --2- 172.21.15.42:0/150869176 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2ab408df70 0x7f2ab409f2d0 unknown :-1 s=CLOSED pgs=1888 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:01:04.956 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:04.927+0000 7f2abb476700 1 -- 172.21.15.42:0/150869176 shutdown_connections 2019-12-05T06:01:04.957 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:04.927+0000 7f2abb476700 1 -- 172.21.15.42:0/150869176 wait complete. 2019-12-05T06:01:04.957 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:04.927+0000 7f2abb476700 1 -- 172.21.15.42:0/150869176 >> 172.21.15.42:0/150869176 conn(0x7f2ab40aabd0 msgr2=0x7f2ab40a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:01:04.957 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:04.927+0000 7f2abb476700 1 Processor -- start 2019-12-05T06:01:04.958 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:04.927+0000 7f2abb476700 1 -- start start 2019-12-05T06:01:04.958 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:04.931+0000 7f2abb476700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2ab41476b0 0x7f2ab4147ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:01:04.958 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:04.931+0000 7f2abb476700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ab40a68e0 con 0x7f2ab411ebb0 2019-12-05T06:01:04.959 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:04.931+0000 7f2abb476700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ab41464d0 con 0x7f2ab411bc90 2019-12-05T06:01:04.959 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:04.931+0000 7f2abb476700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ab4145f90 con 0x7f2ab41476b0 2019-12-05T06:01:04.959 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:04.931+0000 7f2ab9212700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2ab41476b0 0x7f2ab4147ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:01:04.959 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:04.931+0000 7f2ab9a13700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f2ab411bc90 0x7f2ab40a60e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:46318/0 (socket says 172.21.15.42:46318) 2019-12-05T06:01:04.960 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:04.931+0000 7f2ab9212700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2ab41476b0 0x7f2ab4147ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:46282/0 (socket says 172.21.15.42:46282) 2019-12-05T06:01:04.961 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:04.931+0000 7f2ab9a13700 1 -- 172.21.15.42:0/3355985589 learned_addr learned my addr 172.21.15.42:0/3355985589 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:01:04.961 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:04.931+0000 7f2ab9212700 1 -- 172.21.15.42:0/3355985589 >> v1:172.21.15.42:6789/0 conn(0x7f2ab411bc90 legacy=0x7f2ab40a60e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:01:04.961 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:04.931+0000 7f2ab9212700 1 -- 172.21.15.42:0/3355985589 >> v1:172.21.15.40:6789/0 conn(0x7f2ab411ebb0 legacy=0x7f2ab411efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:01:04.961 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:04.931+0000 7f2aaa7fc700 1 -- 172.21.15.42:0/3355985589 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 916917510 0 0) 0x7f2ab41464d0 con 0x7f2ab411bc90 2019-12-05T06:01:04.962 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:04.931+0000 7f2ab9212700 1 -- 172.21.15.42:0/3355985589 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ab4145b10 con 0x7f2ab41476b0 2019-12-05T06:01:04.962 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:04.931+0000 7f2ab9212700 1 --2- 172.21.15.42:0/3355985589 >> v2:172.21.15.42:3300/0 conn(0x7f2ab41476b0 0x7f2ab4147ae0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f2aa4008790 tx=0x7f2aa4008df0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:04.962 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:04.931+0000 7f2ab9212700 1 --2- 172.21.15.42:0/3355985589 >> v2:172.21.15.42:3300/0 conn(0x7f2ab41476b0 0x7f2ab4147ae0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f2aa4008790 tx=0x7f2aa4008df0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:04.963 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:04.931+0000 7f2ab9212700 1 --2- 172.21.15.42:0/3355985589 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2ab41476b0 0x7f2ab4147ae0 secure :-1 s=READY pgs=1890 cs=0 l=1 rx=0x7f2aa4008790 tx=0x7f2aa4008df0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:01:04.963 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:04.931+0000 7f2ab9212700 1 --2- 172.21.15.42:0/3355985589 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2ab41476b0 0x7f2ab4147ae0 secure :-1 s=THROTTLE_DONE pgs=1890 cs=0 l=1 rx=0x7f2aa4008790 tx=0x7f2aa4008df0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:04.964 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:04.931+0000 7f2aaa7fc700 1 -- 172.21.15.42:0/3355985589 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2aa4007340 con 0x7f2ab41476b0 2019-12-05T06:01:04.964 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:04.931+0000 7f2ab9212700 1 --2- 172.21.15.42:0/3355985589 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2ab41476b0 0x7f2ab4147ae0 secure :-1 s=THROTTLE_DONE pgs=1890 cs=0 l=1 rx=0x7f2aa4008790 tx=0x7f2aa4008df0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:04.964 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:04.931+0000 7f2abb476700 1 -- 172.21.15.42:0/3355985589 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ab411f7e0 con 0x7f2ab41476b0 2019-12-05T06:01:04.965 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:04.931+0000 7f2aaa7fc700 1 -- 172.21.15.42:0/3355985589 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2aa4011470 con 0x7f2ab41476b0 2019-12-05T06:01:04.965 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:04.931+0000 7f2ab9212700 1 --2- 172.21.15.42:0/3355985589 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2ab41476b0 0x7f2ab4147ae0 secure :-1 s=THROTTLE_DONE pgs=1890 cs=0 l=1 rx=0x7f2aa4008790 tx=0x7f2aa4008df0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:04.965 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:04.931+0000 7f2aaa7fc700 1 -- 172.21.15.42:0/3355985589 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2aa4005540 con 0x7f2ab41476b0 2019-12-05T06:01:04.965 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:04.931+0000 7f2abb476700 1 -- 172.21.15.42:0/3355985589 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ab4126e30 con 0x7f2ab41476b0 2019-12-05T06:01:04.966 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:04.931+0000 7f2ab9212700 1 --2- 172.21.15.42:0/3355985589 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2ab41476b0 0x7f2ab4147ae0 secure :-1 s=THROTTLE_DONE pgs=1890 cs=0 l=1 rx=0x7f2aa4008790 tx=0x7f2aa4008df0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:04.966 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:04.931+0000 7f2aaa7fc700 1 -- 172.21.15.42:0/3355985589 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f2aa40057e0 con 0x7f2ab41476b0 2019-12-05T06:01:04.966 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:04.931+0000 7f2ab9212700 1 --2- 172.21.15.42:0/3355985589 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2ab41476b0 0x7f2ab4147ae0 secure :-1 s=THROTTLE_DONE pgs=1890 cs=0 l=1 rx=0x7f2aa4008790 tx=0x7f2aa4008df0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:04.966 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:04.931+0000 7f2aaa7fc700 1 --2- 172.21.15.42:0/3355985589 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2a9c03abc0 0x7f2a9c03d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:01:04.967 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:04.931+0000 7f2aaa7fc700 1 -- 172.21.15.42:0/3355985589 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9471+0+0 (secure 0 0 0) 0x7f2aa404bd50 con 0x7f2ab41476b0 2019-12-05T06:01:04.967 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:04.931+0000 7f2ab9a13700 1 --2- 172.21.15.42:0/3355985589 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2a9c03abc0 0x7f2a9c03d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:01:04.967 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:04.935+0000 7f2ab9a13700 1 --2- 172.21.15.42:0/3355985589 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2a9c03abc0 0x7f2a9c03d0d0 crc :-1 s=READY pgs=806 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:01:04.967 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:04.939+0000 7f2abb476700 1 -- 172.21.15.42:0/3355985589 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ab405d100 con 0x7f2ab41476b0 2019-12-05T06:01:04.969 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:04.943+0000 7f2ab9212700 1 --2- 172.21.15.42:0/3355985589 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2ab41476b0 0x7f2ab4147ae0 secure :-1 s=THROTTLE_DONE pgs=1890 cs=0 l=1 rx=0x7f2aa4008790 tx=0x7f2aa4008df0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:04.969 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:04.943+0000 7f2aaa7fc700 1 -- 172.21.15.42:0/3355985589 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f2aa4032db0 con 0x7f2ab41476b0 2019-12-05T06:01:05.132 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:05.127+0000 7f2abb476700 1 -- 172.21.15.42:0/3355985589 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2ab405b8b0 con 0x7f2ab41476b0 2019-12-05T06:01:05.133 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:05.127+0000 7f2ab9212700 1 --2- 172.21.15.42:0/3355985589 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2ab41476b0 0x7f2ab4147ae0 secure :-1 s=THROTTLE_DONE pgs=1890 cs=0 l=1 rx=0x7f2aa4008790 tx=0x7f2aa4008df0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:05.133 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:05.127+0000 7f2aaa7fc700 1 -- 172.21.15.42:0/3355985589 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 232 v232) v1 ==== 95+0+1775 (secure 0 0 0) 0x7f2aa4033010 con 0x7f2ab41476b0 2019-12-05T06:01:05.133 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 232 2019-12-05T06:01:05.133 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:01:05.134 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":232,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":232,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:01:04.420072+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":89,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6653},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6653":{"gid":6653,"name":"b","rank":0,"incarnation":229,"state":"up:active","state_seq":13,"addr":"172.21.15.40:6833/2780168808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2780168808},{"type":"v1","addr":"172.21.15.40:6833","nonce":2780168808}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:01:05.136 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:05.131+0000 7f2abb476700 1 -- 172.21.15.42:0/3355985589 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2a9c03abc0 msgr2=0x7f2a9c03d0d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:01:05.136 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:05.131+0000 7f2abb476700 1 --2- 172.21.15.42:0/3355985589 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2a9c03abc0 0x7f2a9c03d0d0 crc :-1 s=READY pgs=806 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:01:05.136 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:05.131+0000 7f2abb476700 1 -- 172.21.15.42:0/3355985589 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2ab41476b0 msgr2=0x7f2ab4147ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:01:05.136 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:05.131+0000 7f2abb476700 1 --2- 172.21.15.42:0/3355985589 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2ab41476b0 0x7f2ab4147ae0 secure :-1 s=READY pgs=1890 cs=0 l=1 rx=0x7f2aa4008790 tx=0x7f2aa4008df0).stop 2019-12-05T06:01:05.137 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:05.131+0000 7f2abb476700 1 -- 172.21.15.42:0/3355985589 shutdown_connections 2019-12-05T06:01:05.137 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:05.131+0000 7f2abb476700 1 --2- 172.21.15.42:0/3355985589 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2a9c03abc0 0x7f2a9c03d0d0 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:01:05.137 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:05.131+0000 7f2abb476700 1 --2- 172.21.15.42:0/3355985589 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2ab41476b0 0x7f2ab4147ae0 unknown :-1 s=CLOSED pgs=1890 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:01:05.137 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:05.131+0000 7f2abb476700 1 -- 172.21.15.42:0/3355985589 shutdown_connections 2019-12-05T06:01:05.138 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:05.131+0000 7f2abb476700 1 -- 172.21.15.42:0/3355985589 wait complete. 2019-12-05T06:01:05.138 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:05.131+0000 7f2abb476700 1 -- 172.21.15.42:0/3355985589 >> 172.21.15.42:0/3355985589 conn(0x7f2ab40a8290 msgr2=0x7f2ab407e4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:01:05.147 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b has gained rank=0, replacing gid=9061 2019-12-05T06:01:05.147 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 2 secs before reviving mds.a 2019-12-05T06:01:07.148 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.a 2019-12-05T06:01:07.148 INFO:tasks.ceph.mds.a:Restarting daemon 2019-12-05T06:01:07.148 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2019-12-05T06:01:07.170 INFO:tasks.ceph.mds.a:Started 2019-12-05T06:01:07.171 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:01:07.229 INFO:tasks.ceph.mds.a.smithi042.stdout:starting mds.a at 2019-12-05T06:01:07.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:07.279+0000 7fa3ada89700 1 Processor -- start 2019-12-05T06:01:07.291 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:07.279+0000 7fa3ada89700 1 -- start start 2019-12-05T06:01:07.291 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:07.279+0000 7fa3ada89700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa3a808df80 0x7fa3a809f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:01:07.291 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:07.279+0000 7fa3ada89700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fa3a809f830 0x7fa3a814efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:01:07.291 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:07.279+0000 7fa3ada89700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3a809fc60 con 0x7fa3a80a7740 2019-12-05T06:01:07.291 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:07.279+0000 7fa3ada89700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3a80a0b00 con 0x7fa3a809f830 2019-12-05T06:01:07.292 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:07.279+0000 7fa3ada89700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3a808e5b0 con 0x7fa3a808df80 2019-12-05T06:01:07.292 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:07.283+0000 7fa3ac827700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fa3a809f830 0x7fa3a814efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:01:07.292 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:07.283+0000 7fa3a7fff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fa3a80a7740 0x7fa3a807e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:46354/0 (socket says 172.21.15.42:46354) 2019-12-05T06:01:07.292 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:07.283+0000 7fa3a7fff700 1 -- 172.21.15.42:0/5801556 learned_addr learned my addr 172.21.15.42:0/5801556 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:01:07.292 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:07.283+0000 7fa3a77fe700 1 --2- 172.21.15.42:0/5801556 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa3a808df80 0x7fa3a809f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:01:07.292 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:07.283+0000 7fa3ac827700 1 -- 172.21.15.42:0/5801556 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa3a808df80 msgr2=0x7fa3a809f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:01:07.293 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:07.283+0000 7fa3ac827700 1 --2- 172.21.15.42:0/5801556 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa3a808df80 0x7fa3a809f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:01:07.293 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:07.283+0000 7fa3a6ffd700 1 -- 172.21.15.42:0/5801556 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 156115752 0 0) 0x7fa3a809fc60 con 0x7fa3a80a7740 2019-12-05T06:01:07.293 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:07.283+0000 7fa3ac827700 1 -- 172.21.15.42:0/5801556 >> v1:172.21.15.42:6789/0 conn(0x7fa3a80a7740 legacy=0x7fa3a807e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:01:07.293 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:07.283+0000 7fa3ac827700 1 -- 172.21.15.42:0/5801556 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3a80a0710 con 0x7fa3a809f830 2019-12-05T06:01:07.293 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:07.283+0000 7fa3ac827700 1 --2- 172.21.15.42:0/5801556 >> v2:172.21.15.42:3300/0 conn(0x7fa3a809f830 0x7fa3a814efe0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fa3a0008b10 tx=0x7fa3a000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:07.293 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:07.283+0000 7fa3ac827700 1 --2- 172.21.15.42:0/5801556 >> v2:172.21.15.42:3300/0 conn(0x7fa3a809f830 0x7fa3a814efe0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fa3a0008b10 tx=0x7fa3a000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:07.293 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:07.283+0000 7fa3ac827700 1 --2- 172.21.15.42:0/5801556 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa3a809f830 0x7fa3a814efe0 secure :-1 s=READY pgs=1895 cs=0 l=1 rx=0x7fa3a0008b10 tx=0x7fa3a000fff0).ready entity=mon.1 client_cookie=2b291572ffdabbd0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:01:07.294 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:07.283+0000 7fa3ac827700 1 --2- 172.21.15.42:0/5801556 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa3a809f830 0x7fa3a814efe0 secure :-1 s=THROTTLE_DONE pgs=1895 cs=0 l=1 rx=0x7fa3a0008b10 tx=0x7fa3a000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:07.294 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:07.283+0000 7fa3a6ffd700 1 -- 172.21.15.42:0/5801556 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa3a0010900 con 0x7fa3a809f830 2019-12-05T06:01:07.294 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:07.283+0000 7fa3ac827700 1 --2- 172.21.15.42:0/5801556 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa3a809f830 0x7fa3a814efe0 secure :-1 s=THROTTLE_DONE pgs=1895 cs=0 l=1 rx=0x7fa3a0008b10 tx=0x7fa3a000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:07.294 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:07.283+0000 7fa3ac827700 1 --2- 172.21.15.42:0/5801556 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa3a809f830 0x7fa3a814efe0 secure :-1 s=THROTTLE_DONE pgs=1895 cs=0 l=1 rx=0x7fa3a0008b10 tx=0x7fa3a000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:07.294 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:07.283+0000 7fa3a6ffd700 1 -- 172.21.15.42:0/5801556 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3a0008b50 con 0x7fa3a809f830 2019-12-05T06:01:07.295 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:07.283+0000 7fa3a6ffd700 1 -- 172.21.15.42:0/5801556 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa3a00110c0 con 0x7fa3a809f830 2019-12-05T06:01:07.295 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:07.283+0000 7fa3ada89700 1 -- 172.21.15.42:0/5801556 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa3a809f830 msgr2=0x7fa3a814efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:01:07.296 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:07.283+0000 7fa3ada89700 1 --2- 172.21.15.42:0/5801556 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa3a809f830 0x7fa3a814efe0 secure :-1 s=READY pgs=1895 cs=0 l=1 rx=0x7fa3a0008b10 tx=0x7fa3a000fff0).stop 2019-12-05T06:01:07.296 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:07.283+0000 7fa3ada89700 1 -- 172.21.15.42:0/5801556 shutdown_connections 2019-12-05T06:01:07.296 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:07.283+0000 7fa3ada89700 1 --2- 172.21.15.42:0/5801556 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa3a809f830 0x7fa3a814efe0 unknown :-1 s=CLOSED pgs=1895 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:01:07.296 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:07.283+0000 7fa3ada89700 1 --2- 172.21.15.42:0/5801556 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa3a808df80 0x7fa3a809f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:01:07.296 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:07.283+0000 7fa3ada89700 1 -- 172.21.15.42:0/5801556 shutdown_connections 2019-12-05T06:01:07.296 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:07.283+0000 7fa3ada89700 1 -- 172.21.15.42:0/5801556 wait complete. 2019-12-05T06:01:07.297 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:07.283+0000 7fa3ada89700 1 -- 172.21.15.42:0/5801556 >> 172.21.15.42:0/5801556 conn(0x7fa3a80aabe0 msgr2=0x7fa3a80a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:01:07.297 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:07.287+0000 7fa3ada89700 1 Processor -- start 2019-12-05T06:01:07.298 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:07.287+0000 7fa3ada89700 1 -- start start 2019-12-05T06:01:07.301 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:07.287+0000 7fa3ada89700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fa3a812a4d0 0x7fa3a81482e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:01:07.301 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:07.287+0000 7fa3ada89700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3a807bfa0 con 0x7fa3a81434a0 2019-12-05T06:01:07.301 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:07.287+0000 7fa3ada89700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3a8136580 con 0x7fa3a81191f0 2019-12-05T06:01:07.301 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:07.287+0000 7fa3ada89700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3a81454c0 con 0x7fa3a812a4d0 2019-12-05T06:01:07.301 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:07.287+0000 7fa3a77fe700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fa3a81191f0 0x7fa3a8129cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:46360/0 (socket says 172.21.15.42:46360) 2019-12-05T06:01:07.302 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:07.287+0000 7fa3a77fe700 1 -- 172.21.15.42:0/3892498628 learned_addr learned my addr 172.21.15.42:0/3892498628 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:01:07.302 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:07.287+0000 7fa3ac827700 1 --2- 172.21.15.42:0/3892498628 >> v2:172.21.15.42:3300/0 conn(0x7fa3a812a4d0 0x7fa3a81482e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:01:07.302 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:07.287+0000 7fa3ac827700 1 -- 172.21.15.42:0/3892498628 >> v1:172.21.15.42:6789/0 conn(0x7fa3a81191f0 legacy=0x7fa3a8129cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:01:07.302 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:07.287+0000 7fa3ac827700 1 -- 172.21.15.42:0/3892498628 >> v1:172.21.15.40:6789/0 conn(0x7fa3a81434a0 legacy=0x7fa3a807b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:01:07.302 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:07.287+0000 7fa3a4ff9700 1 -- 172.21.15.42:0/3892498628 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3590742380 0 0) 0x7fa3a8136580 con 0x7fa3a81191f0 2019-12-05T06:01:07.302 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:07.287+0000 7fa3ac827700 1 -- 172.21.15.42:0/3892498628 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3a81377c0 con 0x7fa3a812a4d0 2019-12-05T06:01:07.303 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:07.287+0000 7fa3ac827700 1 --2- 172.21.15.42:0/3892498628 >> v2:172.21.15.42:3300/0 conn(0x7fa3a812a4d0 0x7fa3a81482e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fa3a0008b10 tx=0x7fa3a0008a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:07.303 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:07.287+0000 7fa3ac827700 1 --2- 172.21.15.42:0/3892498628 >> v2:172.21.15.42:3300/0 conn(0x7fa3a812a4d0 0x7fa3a81482e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fa3a0008b10 tx=0x7fa3a0008a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:07.303 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:07.287+0000 7fa3ac827700 1 --2- 172.21.15.42:0/3892498628 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa3a812a4d0 0x7fa3a81482e0 secure :-1 s=READY pgs=1897 cs=0 l=1 rx=0x7fa3a0008b10 tx=0x7fa3a0008a20).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:01:07.303 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:07.287+0000 7fa3ac827700 1 --2- 172.21.15.42:0/3892498628 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa3a812a4d0 0x7fa3a81482e0 secure :-1 s=THROTTLE_DONE pgs=1897 cs=0 l=1 rx=0x7fa3a0008b10 tx=0x7fa3a0008a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:07.303 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:07.287+0000 7fa3a4ff9700 1 -- 172.21.15.42:0/3892498628 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa3a0005ab0 con 0x7fa3a812a4d0 2019-12-05T06:01:07.303 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:07.287+0000 7fa3ac827700 1 --2- 172.21.15.42:0/3892498628 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa3a812a4d0 0x7fa3a81482e0 secure :-1 s=THROTTLE_DONE pgs=1897 cs=0 l=1 rx=0x7fa3a0008b10 tx=0x7fa3a0008a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:07.303 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:07.287+0000 7fa3ac827700 1 --2- 172.21.15.42:0/3892498628 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa3a812a4d0 0x7fa3a81482e0 secure :-1 s=THROTTLE_DONE pgs=1897 cs=0 l=1 rx=0x7fa3a0008b10 tx=0x7fa3a0008a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:07.304 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:07.287+0000 7fa3a4ff9700 1 -- 172.21.15.42:0/3892498628 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3a0010b00 con 0x7fa3a812a4d0 2019-12-05T06:01:07.304 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:07.287+0000 7fa3ada89700 1 -- 172.21.15.42:0/3892498628 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3a8135570 con 0x7fa3a812a4d0 2019-12-05T06:01:07.305 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:07.287+0000 7fa3a4ff9700 1 -- 172.21.15.42:0/3892498628 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa3a000f680 con 0x7fa3a812a4d0 2019-12-05T06:01:07.305 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:07.287+0000 7fa3ada89700 1 -- 172.21.15.42:0/3892498628 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3a8135920 con 0x7fa3a812a4d0 2019-12-05T06:01:07.306 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:07.291+0000 7fa3ac827700 1 --2- 172.21.15.42:0/3892498628 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa3a812a4d0 0x7fa3a81482e0 secure :-1 s=THROTTLE_DONE pgs=1897 cs=0 l=1 rx=0x7fa3a0008b10 tx=0x7fa3a0008a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:07.306 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:07.291+0000 7fa3a4ff9700 1 -- 172.21.15.42:0/3892498628 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fa3a000f8d0 con 0x7fa3a812a4d0 2019-12-05T06:01:07.306 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:07.291+0000 7fa3ac827700 1 --2- 172.21.15.42:0/3892498628 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa3a812a4d0 0x7fa3a81482e0 secure :-1 s=THROTTLE_DONE pgs=1897 cs=0 l=1 rx=0x7fa3a0008b10 tx=0x7fa3a0008a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:07.306 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:07.291+0000 7fa3a4ff9700 1 --2- 172.21.15.42:0/3892498628 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa38803aca0 0x7fa38803d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:01:07.306 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:07.291+0000 7fa3a4ff9700 1 -- 172.21.15.42:0/3892498628 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9471+0+0 (secure 0 0 0) 0x7fa3a008f2e0 con 0x7fa3a812a4d0 2019-12-05T06:01:07.306 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:07.291+0000 7fa3a7fff700 1 --2- 172.21.15.42:0/3892498628 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa38803aca0 0x7fa38803d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:01:07.306 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:07.291+0000 7fa3a7fff700 1 --2- 172.21.15.42:0/3892498628 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa38803aca0 0x7fa38803d1b0 crc :-1 s=READY pgs=807 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:01:07.307 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:07.295+0000 7fa3ada89700 1 -- 172.21.15.42:0/3892498628 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3a805d110 con 0x7fa3a812a4d0 2019-12-05T06:01:07.310 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:07.303+0000 7fa3ac827700 1 --2- 172.21.15.42:0/3892498628 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa3a812a4d0 0x7fa3a81482e0 secure :-1 s=THROTTLE_DONE pgs=1897 cs=0 l=1 rx=0x7fa3a0008b10 tx=0x7fa3a0008a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:07.310 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:07.303+0000 7fa3a4ff9700 1 -- 172.21.15.42:0/3892498628 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fa3a00765a0 con 0x7fa3a812a4d0 2019-12-05T06:01:07.482 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:07.475+0000 7fa3ada89700 1 -- 172.21.15.42:0/3892498628 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa3a805b8c0 con 0x7fa3a812a4d0 2019-12-05T06:01:07.484 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:07.479+0000 7fa3ac827700 1 --2- 172.21.15.42:0/3892498628 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa3a812a4d0 0x7fa3a81482e0 secure :-1 s=THROTTLE_DONE pgs=1897 cs=0 l=1 rx=0x7fa3a0008b10 tx=0x7fa3a0008a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:07.484 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:07.479+0000 7fa3a4ff9700 1 -- 172.21.15.42:0/3892498628 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 232 v232) v1 ==== 95+0+1775 (secure 0 0 0) 0x7fa3a0077150 con 0x7fa3a812a4d0 2019-12-05T06:01:07.484 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 232 2019-12-05T06:01:07.485 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:01:07.485 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":232,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":232,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:01:04.420072+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":89,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6653},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6653":{"gid":6653,"name":"b","rank":0,"incarnation":229,"state":"up:active","state_seq":13,"addr":"172.21.15.40:6833/2780168808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2780168808},{"type":"v1","addr":"172.21.15.40:6833","nonce":2780168808}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:01:07.489 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:07.479+0000 7fa3ada89700 1 -- 172.21.15.42:0/3892498628 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa38803aca0 msgr2=0x7fa38803d1b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:01:07.490 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:07.479+0000 7fa3ada89700 1 --2- 172.21.15.42:0/3892498628 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa38803aca0 0x7fa38803d1b0 crc :-1 s=READY pgs=807 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:01:07.490 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:07.479+0000 7fa3ada89700 1 -- 172.21.15.42:0/3892498628 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa3a812a4d0 msgr2=0x7fa3a81482e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:01:07.490 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:07.479+0000 7fa3ada89700 1 --2- 172.21.15.42:0/3892498628 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa3a812a4d0 0x7fa3a81482e0 secure :-1 s=READY pgs=1897 cs=0 l=1 rx=0x7fa3a0008b10 tx=0x7fa3a0008a20).stop 2019-12-05T06:01:07.490 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:07.479+0000 7fa3ada89700 1 -- 172.21.15.42:0/3892498628 shutdown_connections 2019-12-05T06:01:07.490 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:07.479+0000 7fa3ada89700 1 --2- 172.21.15.42:0/3892498628 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa38803aca0 0x7fa38803d1b0 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:01:07.491 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:07.479+0000 7fa3ada89700 1 --2- 172.21.15.42:0/3892498628 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa3a812a4d0 0x7fa3a81482e0 unknown :-1 s=CLOSED pgs=1897 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:01:07.491 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:07.479+0000 7fa3ada89700 1 -- 172.21.15.42:0/3892498628 shutdown_connections 2019-12-05T06:01:07.491 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:07.479+0000 7fa3ada89700 1 -- 172.21.15.42:0/3892498628 wait complete. 2019-12-05T06:01:07.491 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:07.479+0000 7fa3ada89700 1 -- 172.21.15.42:0/3892498628 >> 172.21.15.42:0/3892498628 conn(0x7fa3a80aabe0 msgr2=0x7fa3a811c470 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:01:07.496 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2019-12-05T06:01:09.497 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:01:09.598 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:09.587+0000 7f9534575700 1 Processor -- start 2019-12-05T06:01:09.599 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:09.587+0000 7f9534575700 1 -- start start 2019-12-05T06:01:09.599 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:09.591+0000 7f9534575700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f952c0a7740 0x7f952c08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:01:09.599 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:09.591+0000 7f9534575700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f952c07e720 0x7f952c07eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:01:09.599 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:09.591+0000 7f9534575700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f952c09f2e0 0x7f952c09f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:01:09.599 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:09.591+0000 7f9534575700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f952c0a0b00 con 0x7f952c0a7740 2019-12-05T06:01:09.600 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:09.591+0000 7f9534575700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f952c08e5b0 con 0x7f952c07e720 2019-12-05T06:01:09.600 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:09.591+0000 7f9534575700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f952c08e850 con 0x7f952c09f2e0 2019-12-05T06:01:09.600 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:09.591+0000 7f9532311700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f952c07e720 0x7f952c07eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:01:09.600 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:09.591+0000 7f9532311700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f952c07e720 0x7f952c07eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:46344/0 (socket says 172.21.15.42:46344) 2019-12-05T06:01:09.601 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:09.591+0000 7f9532311700 1 -- 172.21.15.42:0/3519929732 learned_addr learned my addr 172.21.15.42:0/3519929732 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:01:09.608 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:09.591+0000 7f9532b12700 1 --2- 172.21.15.42:0/3519929732 >> v2:172.21.15.40:3300/0 conn(0x7f952c0a7740 0x7f952c08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:01:09.608 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:09.591+0000 7f9533313700 1 --2- 172.21.15.42:0/3519929732 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f952c09f2e0 0x7f952c09f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:01:09.608 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:09.591+0000 7f9532311700 1 -- 172.21.15.42:0/3519929732 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f952c09f2e0 msgr2=0x7f952c09f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:01:09.609 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:09.591+0000 7f9532311700 1 --2- 172.21.15.42:0/3519929732 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f952c09f2e0 0x7f952c09f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:01:09.609 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:09.591+0000 7f9532311700 1 -- 172.21.15.42:0/3519929732 >> v2:172.21.15.40:3300/0 conn(0x7f952c0a7740 msgr2=0x7f952c08df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:01:09.609 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:09.591+0000 7f9532311700 1 --2- 172.21.15.42:0/3519929732 >> v2:172.21.15.40:3300/0 conn(0x7f952c0a7740 0x7f952c08df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:01:09.609 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:09.591+0000 7f9532311700 1 -- 172.21.15.42:0/3519929732 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f952c07f0a0 con 0x7f952c07e720 2019-12-05T06:01:09.609 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:09.591+0000 7f9532311700 1 --2- 172.21.15.42:0/3519929732 >> v2:172.21.15.42:3300/0 conn(0x7f952c07e720 0x7f952c07eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f952c0a0f50 tx=0x7f951c00fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:09.609 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:09.591+0000 7f9532311700 1 --2- 172.21.15.42:0/3519929732 >> v2:172.21.15.42:3300/0 conn(0x7f952c07e720 0x7f952c07eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f952c0a0f50 tx=0x7f951c00fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:09.609 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:09.591+0000 7f9532311700 1 --2- 172.21.15.42:0/3519929732 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f952c07e720 0x7f952c07eb50 secure :-1 s=READY pgs=1898 cs=0 l=1 rx=0x7f952c0a0f50 tx=0x7f951c00fdb0).ready entity=mon.1 client_cookie=ac42c7b0777fc868 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:01:09.610 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:09.591+0000 7f9532311700 1 --2- 172.21.15.42:0/3519929732 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f952c07e720 0x7f952c07eb50 secure :-1 s=THROTTLE_DONE pgs=1898 cs=0 l=1 rx=0x7f952c0a0f50 tx=0x7f951c00fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:09.610 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:09.591+0000 7f9532311700 1 --2- 172.21.15.42:0/3519929732 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f952c07e720 0x7f952c07eb50 secure :-1 s=THROTTLE_DONE pgs=1898 cs=0 l=1 rx=0x7f952c0a0f50 tx=0x7f951c00fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:09.610 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:09.591+0000 7f9531b10700 1 -- 172.21.15.42:0/3519929732 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f951c008c90 con 0x7f952c07e720 2019-12-05T06:01:09.610 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:09.591+0000 7f9532311700 1 --2- 172.21.15.42:0/3519929732 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f952c07e720 0x7f952c07eb50 secure :-1 s=THROTTLE_DONE pgs=1898 cs=0 l=1 rx=0x7f952c0a0f50 tx=0x7f951c00fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:09.611 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:09.591+0000 7f9531b10700 1 -- 172.21.15.42:0/3519929732 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f951c0088b0 con 0x7f952c07e720 2019-12-05T06:01:09.611 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:09.591+0000 7f9531b10700 1 -- 172.21.15.42:0/3519929732 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f951c011b50 con 0x7f952c07e720 2019-12-05T06:01:09.611 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:09.591+0000 7f9534575700 1 -- 172.21.15.42:0/3519929732 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f952c07e720 msgr2=0x7f952c07eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:01:09.611 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:09.591+0000 7f9534575700 1 --2- 172.21.15.42:0/3519929732 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f952c07e720 0x7f952c07eb50 secure :-1 s=READY pgs=1898 cs=0 l=1 rx=0x7f952c0a0f50 tx=0x7f951c00fdb0).stop 2019-12-05T06:01:09.612 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:09.591+0000 7f9534575700 1 -- 172.21.15.42:0/3519929732 shutdown_connections 2019-12-05T06:01:09.612 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:09.591+0000 7f9534575700 1 --2- 172.21.15.42:0/3519929732 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f952c09f2e0 0x7f952c09f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:01:09.612 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:09.591+0000 7f9534575700 1 --2- 172.21.15.42:0/3519929732 >> v2:172.21.15.40:3300/0 conn(0x7f952c0a7740 0x7f952c08df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:01:09.612 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:09.591+0000 7f9534575700 1 --2- 172.21.15.42:0/3519929732 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f952c07e720 0x7f952c07eb50 unknown :-1 s=CLOSED pgs=1898 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:01:09.612 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:09.591+0000 7f9534575700 1 -- 172.21.15.42:0/3519929732 shutdown_connections 2019-12-05T06:01:09.612 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:09.591+0000 7f9534575700 1 -- 172.21.15.42:0/3519929732 wait complete. 2019-12-05T06:01:09.612 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:09.591+0000 7f9534575700 1 -- 172.21.15.42:0/3519929732 >> 172.21.15.42:0/3519929732 conn(0x7f952c0aabe0 msgr2=0x7f952c0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:01:09.616 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:09.595+0000 7f9534575700 1 Processor -- start 2019-12-05T06:01:09.616 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:09.595+0000 7f9534575700 1 -- start start 2019-12-05T06:01:09.617 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:09.595+0000 7f9534575700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f952c065160 0x7f952c0a2570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:01:09.617 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:09.595+0000 7f9534575700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f952c126a40 0x7f952c126e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:01:09.617 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:09.595+0000 7f9534575700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f952c064c20 con 0x7f952c0a5e50 2019-12-05T06:01:09.617 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:09.595+0000 7f9534575700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f952c0a2ba0 con 0x7f952c126a40 2019-12-05T06:01:09.618 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:09.595+0000 7f9534575700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f952c0a2e40 con 0x7f952c065160 2019-12-05T06:01:09.618 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:09.595+0000 7f9533313700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f952c126a40 0x7f952c126e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:01:09.618 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:09.595+0000 7f9533313700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f952c126a40 0x7f952c126e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:46352/0 (socket says 172.21.15.42:46352) 2019-12-05T06:01:09.618 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:09.595+0000 7f9533313700 1 -- 172.21.15.42:0/1445820101 learned_addr learned my addr 172.21.15.42:0/1445820101 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:01:09.619 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:09.595+0000 7f9532b12700 1 --2- 172.21.15.42:0/1445820101 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f952c065160 0x7f952c0a2570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:01:09.619 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:09.595+0000 7f9533313700 1 -- 172.21.15.42:0/1445820101 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f952c065160 msgr2=0x7f952c0a2570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:01:09.619 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:09.595+0000 7f9533313700 1 --2- 172.21.15.42:0/1445820101 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f952c065160 0x7f952c0a2570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:01:09.619 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:09.595+0000 7f9533313700 1 -- 172.21.15.42:0/1445820101 >> v1:172.21.15.40:6789/0 conn(0x7f952c0a5e50 legacy=0x7f952c064420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:01:09.620 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:09.595+0000 7f9533313700 1 -- 172.21.15.42:0/1445820101 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f952c1303b0 con 0x7f952c126a40 2019-12-05T06:01:09.620 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:09.595+0000 7f9533313700 1 --2- 172.21.15.42:0/1445820101 >> v2:172.21.15.42:3300/0 conn(0x7f952c126a40 0x7f952c126e70 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f952c0a2d60 tx=0x7f95240109b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:09.620 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:09.595+0000 7f9533313700 1 --2- 172.21.15.42:0/1445820101 >> v2:172.21.15.42:3300/0 conn(0x7f952c126a40 0x7f952c126e70 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f952c0a2d60 tx=0x7f95240109b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:09.620 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:09.595+0000 7f9533313700 1 --2- 172.21.15.42:0/1445820101 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f952c126a40 0x7f952c126e70 secure :-1 s=READY pgs=1899 cs=0 l=1 rx=0x7f952c0a2d60 tx=0x7f95240109b0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:01:09.621 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:09.595+0000 7f9533313700 1 --2- 172.21.15.42:0/1445820101 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f952c126a40 0x7f952c126e70 secure :-1 s=THROTTLE_DONE pgs=1899 cs=0 l=1 rx=0x7f952c0a2d60 tx=0x7f95240109b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:09.621 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:09.595+0000 7f9533313700 1 --2- 172.21.15.42:0/1445820101 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f952c126a40 0x7f952c126e70 secure :-1 s=THROTTLE_DONE pgs=1899 cs=0 l=1 rx=0x7f952c0a2d60 tx=0x7f95240109b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:09.622 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:09.595+0000 7f95237fe700 1 -- 172.21.15.42:0/1445820101 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9524005930 con 0x7f952c126a40 2019-12-05T06:01:09.622 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:09.595+0000 7f9533313700 1 --2- 172.21.15.42:0/1445820101 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f952c126a40 0x7f952c126e70 secure :-1 s=THROTTLE_DONE pgs=1899 cs=0 l=1 rx=0x7f952c0a2d60 tx=0x7f95240109b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:09.622 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:09.595+0000 7f95237fe700 1 -- 172.21.15.42:0/1445820101 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9524007200 con 0x7f952c126a40 2019-12-05T06:01:09.622 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:09.595+0000 7f9534575700 1 -- 172.21.15.42:0/1445820101 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f952c065b20 con 0x7f952c126a40 2019-12-05T06:01:09.623 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:09.595+0000 7f95237fe700 1 -- 172.21.15.42:0/1445820101 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f95240060c0 con 0x7f952c126a40 2019-12-05T06:01:09.623 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:09.595+0000 7f9534575700 1 -- 172.21.15.42:0/1445820101 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f952c1273c0 con 0x7f952c126a40 2019-12-05T06:01:09.624 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:09.599+0000 7f9533313700 1 --2- 172.21.15.42:0/1445820101 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f952c126a40 0x7f952c126e70 secure :-1 s=THROTTLE_DONE pgs=1899 cs=0 l=1 rx=0x7f952c0a2d60 tx=0x7f95240109b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:09.624 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:09.599+0000 7f95237fe700 1 -- 172.21.15.42:0/1445820101 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f95240063d0 con 0x7f952c126a40 2019-12-05T06:01:09.624 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:09.599+0000 7f9533313700 1 --2- 172.21.15.42:0/1445820101 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f952c126a40 0x7f952c126e70 secure :-1 s=THROTTLE_DONE pgs=1899 cs=0 l=1 rx=0x7f952c0a2d60 tx=0x7f95240109b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:09.626 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:09.599+0000 7f95237fe700 1 --2- 172.21.15.42:0/1445820101 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f951403adb0 0x7f951403d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:01:09.626 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:09.599+0000 7f95237fe700 1 -- 172.21.15.42:0/1445820101 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9471+0+0 (secure 0 0 0) 0x7f952404acf0 con 0x7f952c126a40 2019-12-05T06:01:09.626 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:09.599+0000 7f9532b12700 1 --2- 172.21.15.42:0/1445820101 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f951403adb0 0x7f951403d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:01:09.626 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:09.599+0000 7f9532b12700 1 --2- 172.21.15.42:0/1445820101 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f951403adb0 0x7f951403d2c0 crc :-1 s=READY pgs=809 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:01:09.626 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:09.603+0000 7f9534575700 1 -- 172.21.15.42:0/1445820101 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f952c05d110 con 0x7f952c126a40 2019-12-05T06:01:09.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:09.611+0000 7f9533313700 1 --2- 172.21.15.42:0/1445820101 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f952c126a40 0x7f952c126e70 secure :-1 s=THROTTLE_DONE pgs=1899 cs=0 l=1 rx=0x7f952c0a2d60 tx=0x7f95240109b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:09.650 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:09.611+0000 7f95237fe700 1 -- 172.21.15.42:0/1445820101 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f9524031d70 con 0x7f952c126a40 2019-12-05T06:01:09.790 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:09.783+0000 7f9534575700 1 -- 172.21.15.42:0/1445820101 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f952c05b8c0 con 0x7f952c126a40 2019-12-05T06:01:09.791 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:09.783+0000 7f9533313700 1 --2- 172.21.15.42:0/1445820101 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f952c126a40 0x7f952c126e70 secure :-1 s=THROTTLE_DONE pgs=1899 cs=0 l=1 rx=0x7f952c0a2d60 tx=0x7f95240109b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:09.791 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:09.783+0000 7f95237fe700 1 -- 172.21.15.42:0/1445820101 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 233 v233) v1 ==== 95+0+2109 (secure 0 0 0) 0x7f9524031fd0 con 0x7f952c126a40 2019-12-05T06:01:09.791 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 233 2019-12-05T06:01:09.792 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:01:09.792 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":233,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":9292,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6835/613535662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":613535662},{"type":"v1","addr":"172.21.15.42:6835","nonce":613535662}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":233}],"filesystems":[{"mdsmap":{"epoch":232,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:01:04.420072+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":89,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6653},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6653":{"gid":6653,"name":"b","rank":0,"incarnation":229,"state":"up:active","state_seq":13,"addr":"172.21.15.40:6833/2780168808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2780168808},{"type":"v1","addr":"172.21.15.40:6833","nonce":2780168808}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:01:09.794 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:09.787+0000 7f9534575700 1 -- 172.21.15.42:0/1445820101 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f951403adb0 msgr2=0x7f951403d2c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:01:09.794 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:09.787+0000 7f9534575700 1 --2- 172.21.15.42:0/1445820101 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f951403adb0 0x7f951403d2c0 crc :-1 s=READY pgs=809 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:01:09.794 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:09.787+0000 7f9534575700 1 -- 172.21.15.42:0/1445820101 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f952c126a40 msgr2=0x7f952c126e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:01:09.794 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:09.787+0000 7f9534575700 1 --2- 172.21.15.42:0/1445820101 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f952c126a40 0x7f952c126e70 secure :-1 s=READY pgs=1899 cs=0 l=1 rx=0x7f952c0a2d60 tx=0x7f95240109b0).stop 2019-12-05T06:01:09.794 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:09.787+0000 7f9534575700 1 -- 172.21.15.42:0/1445820101 shutdown_connections 2019-12-05T06:01:09.795 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:09.787+0000 7f9534575700 1 --2- 172.21.15.42:0/1445820101 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f951403adb0 0x7f951403d2c0 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:01:09.795 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:09.787+0000 7f9534575700 1 --2- 172.21.15.42:0/1445820101 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f952c126a40 0x7f952c126e70 unknown :-1 s=CLOSED pgs=1899 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:01:09.795 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:09.787+0000 7f9534575700 1 --2- 172.21.15.42:0/1445820101 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f952c065160 0x7f952c0a2570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:01:09.795 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:09.787+0000 7f9534575700 1 -- 172.21.15.42:0/1445820101 shutdown_connections 2019-12-05T06:01:09.795 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:09.787+0000 7f9534575700 1 -- 172.21.15.42:0/1445820101 wait complete. 2019-12-05T06:01:09.795 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:09.787+0000 7f9534575700 1 -- 172.21.15.42:0/1445820101 >> 172.21.15.42:0/1445820101 conn(0x7f952c0a8a80 msgr2=0x7f952c09f690 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:01:09.803 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a reported in up:standby state 2019-12-05T06:01:09.804 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 14 secs before thrashing 2019-12-05T06:01:13.124 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:01:13.128 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:01:23.805 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:01:23.915 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:23.903+0000 7f9de8879700 1 Processor -- start 2019-12-05T06:01:23.915 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:23.903+0000 7f9de8879700 1 -- start start 2019-12-05T06:01:23.916 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:23.903+0000 7f9de8879700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9de40a7740 0x7f9de408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:01:23.916 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:23.903+0000 7f9de8879700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f9de407e720 0x7f9de407eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:01:23.916 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:23.903+0000 7f9de8879700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9de40a0710 con 0x7f9de409f2e0 2019-12-05T06:01:23.916 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:23.903+0000 7f9de8879700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9de40a0b00 con 0x7f9de407e720 2019-12-05T06:01:23.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:23.903+0000 7f9de8879700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9de408e5b0 con 0x7f9de40a7740 2019-12-05T06:01:23.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:23.903+0000 7f9de359e700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f9de409f2e0 0x7f9de409f710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:46482/0 (socket says 172.21.15.42:46482) 2019-12-05T06:01:23.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:23.903+0000 7f9de359e700 1 -- 172.21.15.42:0/2349500946 learned_addr learned my addr 172.21.15.42:0/2349500946 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:01:23.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:23.903+0000 7f9de259c700 1 --2- 172.21.15.42:0/2349500946 >> v2:172.21.15.40:3300/0 conn(0x7f9de407e720 0x7f9de407eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:01:23.918 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:23.903+0000 7f9de2d9d700 1 --2- 172.21.15.42:0/2349500946 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9de40a7740 0x7f9de408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:01:23.918 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:23.903+0000 7f9de1d9b700 1 -- 172.21.15.42:0/2349500946 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 562297325 0 0) 0x7f9de40a0710 con 0x7f9de409f2e0 2019-12-05T06:01:23.918 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:23.903+0000 7f9de1d9b700 1 -- 172.21.15.42:0/2349500946 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9dcc0030b0 con 0x7f9de409f2e0 2019-12-05T06:01:23.918 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:23.903+0000 7f9de1d9b700 1 -- 172.21.15.42:0/2349500946 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 445638531 0 0) 0x7f9dcc0030b0 con 0x7f9de409f2e0 2019-12-05T06:01:23.919 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:23.903+0000 7f9de2d9d700 1 -- 172.21.15.42:0/2349500946 >> v2:172.21.15.40:3300/0 conn(0x7f9de407e720 msgr2=0x7f9de407eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:01:23.919 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:23.903+0000 7f9de2d9d700 1 --2- 172.21.15.42:0/2349500946 >> v2:172.21.15.40:3300/0 conn(0x7f9de407e720 0x7f9de407eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:01:23.919 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:23.903+0000 7f9de2d9d700 1 -- 172.21.15.42:0/2349500946 >> v1:172.21.15.42:6789/0 conn(0x7f9de409f2e0 legacy=0x7f9de409f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:01:23.919 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:23.903+0000 7f9de2d9d700 1 -- 172.21.15.42:0/2349500946 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9de407f0a0 con 0x7f9de40a7740 2019-12-05T06:01:23.920 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:23.903+0000 7f9de259c700 1 --2- 172.21.15.42:0/2349500946 >> v2:172.21.15.40:3300/0 conn(0x7f9de407e720 0x7f9de407eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:01:23.922 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:23.907+0000 7f9de2d9d700 1 --2- 172.21.15.42:0/2349500946 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9de40a7740 0x7f9de408df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f9dd4008b10 tx=0x7f9ddc0019d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:23.922 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:23.907+0000 7f9de2d9d700 1 --2- 172.21.15.42:0/2349500946 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9de40a7740 0x7f9de408df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f9dd4008b10 tx=0x7f9ddc0019d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:23.922 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:23.907+0000 7f9de2d9d700 1 --2- 172.21.15.42:0/2349500946 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9de40a7740 0x7f9de408df80 secure :-1 s=READY pgs=421 cs=0 l=1 rx=0x7f9dd4008b10 tx=0x7f9ddc0019d0).ready entity=mon.2 client_cookie=c97436bc339d6579 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:01:23.923 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:23.907+0000 7f9de2d9d700 1 --2- 172.21.15.42:0/2349500946 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9de40a7740 0x7f9de408df80 secure :-1 s=THROTTLE_DONE pgs=421 cs=0 l=1 rx=0x7f9dd4008b10 tx=0x7f9ddc0019d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:23.923 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:23.907+0000 7f9de1d9b700 1 -- 172.21.15.42:0/2349500946 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9ddc001750 con 0x7f9de40a7740 2019-12-05T06:01:23.923 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:23.907+0000 7f9de2d9d700 1 --2- 172.21.15.42:0/2349500946 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9de40a7740 0x7f9de408df80 secure :-1 s=THROTTLE_DONE pgs=421 cs=0 l=1 rx=0x7f9dd4008b10 tx=0x7f9ddc0019d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:23.923 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:23.907+0000 7f9de2d9d700 1 --2- 172.21.15.42:0/2349500946 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9de40a7740 0x7f9de408df80 secure :-1 s=THROTTLE_DONE pgs=421 cs=0 l=1 rx=0x7f9dd4008b10 tx=0x7f9ddc0019d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:23.924 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:23.907+0000 7f9de1d9b700 1 -- 172.21.15.42:0/2349500946 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ddc001560 con 0x7f9de40a7740 2019-12-05T06:01:23.924 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:23.907+0000 7f9de1d9b700 1 -- 172.21.15.42:0/2349500946 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9dd4011b40 con 0x7f9de40a7740 2019-12-05T06:01:23.924 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:23.907+0000 7f9de8879700 1 -- 172.21.15.42:0/2349500946 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9de40a7740 msgr2=0x7f9de408df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:01:23.924 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:23.907+0000 7f9de8879700 1 --2- 172.21.15.42:0/2349500946 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9de40a7740 0x7f9de408df80 secure :-1 s=READY pgs=421 cs=0 l=1 rx=0x7f9dd4008b10 tx=0x7f9ddc0019d0).stop 2019-12-05T06:01:23.925 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:23.907+0000 7f9de8879700 1 -- 172.21.15.42:0/2349500946 shutdown_connections 2019-12-05T06:01:23.925 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:23.907+0000 7f9de8879700 1 --2- 172.21.15.42:0/2349500946 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9de40a7740 0x7f9de408df80 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:01:23.925 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:23.907+0000 7f9de8879700 1 --2- 172.21.15.42:0/2349500946 >> v2:172.21.15.40:3300/0 conn(0x7f9de407e720 0x7f9de407eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:01:23.925 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:23.907+0000 7f9de8879700 1 -- 172.21.15.42:0/2349500946 shutdown_connections 2019-12-05T06:01:23.926 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:23.907+0000 7f9de8879700 1 -- 172.21.15.42:0/2349500946 wait complete. 2019-12-05T06:01:23.926 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:23.907+0000 7f9de8879700 1 -- 172.21.15.42:0/2349500946 >> 172.21.15.42:0/2349500946 conn(0x7f9de40aabe0 msgr2=0x7f9de40a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:01:23.934 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:23.911+0000 7f9de8879700 1 Processor -- start 2019-12-05T06:01:23.935 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:23.911+0000 7f9de8879700 1 -- start start 2019-12-05T06:01:23.935 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:23.911+0000 7f9de8879700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f9de4143500 0x7f9de41447f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:01:23.935 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:23.911+0000 7f9de8879700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9de411ba50 0x7f9de411be80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:01:23.935 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:23.911+0000 7f9de8879700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9de4144d40 con 0x7f9de41243d0 2019-12-05T06:01:23.936 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:23.911+0000 7f9de8879700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9de4145040 con 0x7f9de4143500 2019-12-05T06:01:23.936 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:23.911+0000 7f9de8879700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9de4142620 con 0x7f9de411ba50 2019-12-05T06:01:23.938 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:23.911+0000 7f9de2d9d700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f9de41243d0 0x7f9de407da80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:46484/0 (socket says 172.21.15.42:46484) 2019-12-05T06:01:23.938 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:23.911+0000 7f9de2d9d700 1 -- 172.21.15.42:0/3744808066 learned_addr learned my addr 172.21.15.42:0/3744808066 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:01:23.938 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:23.911+0000 7f9de259c700 1 --2- 172.21.15.42:0/3744808066 >> v2:172.21.15.40:3300/0 conn(0x7f9de4143500 0x7f9de41447f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:01:23.938 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:23.911+0000 7f9de359e700 1 --2- 172.21.15.42:0/3744808066 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9de411ba50 0x7f9de411be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:01:23.939 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:23.911+0000 7f9dd37fe700 1 -- 172.21.15.42:0/3744808066 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3777091053 0 0) 0x7f9de4144d40 con 0x7f9de41243d0 2019-12-05T06:01:23.939 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:23.911+0000 7f9dd37fe700 1 -- 172.21.15.42:0/3744808066 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9dc80030b0 con 0x7f9de41243d0 2019-12-05T06:01:23.939 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:23.911+0000 7f9de259c700 1 -- 172.21.15.42:0/3744808066 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9de411ba50 msgr2=0x7f9de411be80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:01:23.939 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:23.911+0000 7f9de259c700 1 --2- 172.21.15.42:0/3744808066 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9de411ba50 0x7f9de411be80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:01:23.940 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:23.911+0000 7f9de259c700 1 -- 172.21.15.42:0/3744808066 >> v1:172.21.15.42:6789/0 conn(0x7f9de41243d0 legacy=0x7f9de407da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:01:23.940 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:23.911+0000 7f9de259c700 1 -- 172.21.15.42:0/3744808066 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9de4130340 con 0x7f9de4143500 2019-12-05T06:01:23.940 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:23.915+0000 7f9de259c700 1 --2- 172.21.15.42:0/3744808066 >> v2:172.21.15.40:3300/0 conn(0x7f9de4143500 0x7f9de41447f0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f9dd8002f30 tx=0x7f9dd8010ca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:23.940 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:23.915+0000 7f9de259c700 1 --2- 172.21.15.42:0/3744808066 >> v2:172.21.15.40:3300/0 conn(0x7f9de4143500 0x7f9de41447f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f9dd8002f30 tx=0x7f9dd8010ca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:23.942 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:23.915+0000 7f9de259c700 1 --2- 172.21.15.42:0/3744808066 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9de4143500 0x7f9de41447f0 secure :-1 s=READY pgs=1452 cs=0 l=1 rx=0x7f9dd8002f30 tx=0x7f9dd8010ca0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:01:23.942 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:23.915+0000 7f9de259c700 1 --2- 172.21.15.42:0/3744808066 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9de4143500 0x7f9de41447f0 secure :-1 s=THROTTLE_DONE pgs=1452 cs=0 l=1 rx=0x7f9dd8002f30 tx=0x7f9dd8010ca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:23.942 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:23.915+0000 7f9dd37fe700 1 -- 172.21.15.42:0/3744808066 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9dd8005840 con 0x7f9de4143500 2019-12-05T06:01:23.942 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:23.915+0000 7f9de259c700 1 --2- 172.21.15.42:0/3744808066 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9de4143500 0x7f9de41447f0 secure :-1 s=THROTTLE_DONE pgs=1452 cs=0 l=1 rx=0x7f9dd8002f30 tx=0x7f9dd8010ca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:23.943 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:23.915+0000 7f9de259c700 1 --2- 172.21.15.42:0/3744808066 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9de4143500 0x7f9de41447f0 secure :-1 s=THROTTLE_DONE pgs=1452 cs=0 l=1 rx=0x7f9dd8002f30 tx=0x7f9dd8010ca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:23.943 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:23.915+0000 7f9dd37fe700 1 -- 172.21.15.42:0/3744808066 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9dd8007160 con 0x7f9de4143500 2019-12-05T06:01:23.943 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:23.915+0000 7f9de8879700 1 -- 172.21.15.42:0/3744808066 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9de4137790 con 0x7f9de4143500 2019-12-05T06:01:23.944 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:23.915+0000 7f9dd37fe700 1 -- 172.21.15.42:0/3744808066 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9dd8012360 con 0x7f9de4143500 2019-12-05T06:01:23.944 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:23.915+0000 7f9de8879700 1 -- 172.21.15.42:0/3744808066 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9de4137af0 con 0x7f9de4143500 2019-12-05T06:01:23.944 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:23.915+0000 7f9de259c700 1 --2- 172.21.15.42:0/3744808066 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9de4143500 0x7f9de41447f0 secure :-1 s=THROTTLE_DONE pgs=1452 cs=0 l=1 rx=0x7f9dd8002f30 tx=0x7f9dd8010ca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:23.945 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:23.915+0000 7f9dd37fe700 1 -- 172.21.15.42:0/3744808066 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f9dd8012670 con 0x7f9de4143500 2019-12-05T06:01:23.945 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:23.915+0000 7f9de259c700 1 --2- 172.21.15.42:0/3744808066 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9de4143500 0x7f9de41447f0 secure :-1 s=THROTTLE_DONE pgs=1452 cs=0 l=1 rx=0x7f9dd8002f30 tx=0x7f9dd8010ca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:23.945 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:23.915+0000 7f9dd37fe700 1 --2- 172.21.15.42:0/3744808066 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9dc803b280 0x7f9dc803d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:01:23.945 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:23.915+0000 7f9dd37fe700 1 -- 172.21.15.42:0/3744808066 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9471+0+0 (secure 0 0 0) 0x7f9dd804b7c0 con 0x7f9de4143500 2019-12-05T06:01:23.945 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:23.915+0000 7f9de2d9d700 1 --2- 172.21.15.42:0/3744808066 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9dc803b280 0x7f9dc803d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:01:23.945 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:23.915+0000 7f9de2d9d700 1 --2- 172.21.15.42:0/3744808066 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9dc803b280 0x7f9dc803d790 crc :-1 s=READY pgs=810 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:01:23.946 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:23.919+0000 7f9de8879700 1 -- 172.21.15.42:0/3744808066 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9de405d110 con 0x7f9de4143500 2019-12-05T06:01:23.946 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:23.927+0000 7f9de259c700 1 --2- 172.21.15.42:0/3744808066 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9de4143500 0x7f9de41447f0 secure :-1 s=THROTTLE_DONE pgs=1452 cs=0 l=1 rx=0x7f9dd8002f30 tx=0x7f9dd8010ca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:23.946 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:23.927+0000 7f9dd37fe700 1 -- 172.21.15.42:0/3744808066 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f9dd8032840 con 0x7f9de4143500 2019-12-05T06:01:24.106 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:24.099+0000 7f9de8879700 1 -- 172.21.15.42:0/3744808066 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9de405b8c0 con 0x7f9de4143500 2019-12-05T06:01:24.106 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:24.099+0000 7f9de259c700 1 --2- 172.21.15.42:0/3744808066 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9de4143500 0x7f9de41447f0 secure :-1 s=THROTTLE_DONE pgs=1452 cs=0 l=1 rx=0x7f9dd8002f30 tx=0x7f9dd8010ca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:24.107 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:24.099+0000 7f9dd37fe700 1 -- 172.21.15.42:0/3744808066 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 233 v233) v1 ==== 95+0+2109 (secure 0 0 0) 0x7f9dd8032aa0 con 0x7f9de4143500 2019-12-05T06:01:24.109 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 233 2019-12-05T06:01:24.109 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:01:24.111 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":233,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":9292,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6835/613535662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":613535662},{"type":"v1","addr":"172.21.15.42:6835","nonce":613535662}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":233}],"filesystems":[{"mdsmap":{"epoch":232,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:01:04.420072+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":89,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6653},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6653":{"gid":6653,"name":"b","rank":0,"incarnation":229,"state":"up:active","state_seq":13,"addr":"172.21.15.40:6833/2780168808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2780168808},{"type":"v1","addr":"172.21.15.40:6833","nonce":2780168808}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:01:24.112 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:24.107+0000 7f9de8879700 1 -- 172.21.15.42:0/3744808066 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9dc803b280 msgr2=0x7f9dc803d790 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:01:24.112 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:24.107+0000 7f9de8879700 1 --2- 172.21.15.42:0/3744808066 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9dc803b280 0x7f9dc803d790 crc :-1 s=READY pgs=810 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:01:24.112 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:24.107+0000 7f9de8879700 1 -- 172.21.15.42:0/3744808066 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9de4143500 msgr2=0x7f9de41447f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:01:24.113 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:24.107+0000 7f9de8879700 1 --2- 172.21.15.42:0/3744808066 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9de4143500 0x7f9de41447f0 secure :-1 s=READY pgs=1452 cs=0 l=1 rx=0x7f9dd8002f30 tx=0x7f9dd8010ca0).stop 2019-12-05T06:01:24.113 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:24.107+0000 7f9de8879700 1 -- 172.21.15.42:0/3744808066 shutdown_connections 2019-12-05T06:01:24.113 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:24.107+0000 7f9de8879700 1 --2- 172.21.15.42:0/3744808066 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9dc803b280 0x7f9dc803d790 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:01:24.113 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:24.107+0000 7f9de8879700 1 --2- 172.21.15.42:0/3744808066 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9de411ba50 0x7f9de411be80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:01:24.113 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:24.107+0000 7f9de8879700 1 --2- 172.21.15.42:0/3744808066 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9de4143500 0x7f9de41447f0 unknown :-1 s=CLOSED pgs=1452 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:01:24.113 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:24.107+0000 7f9de8879700 1 -- 172.21.15.42:0/3744808066 shutdown_connections 2019-12-05T06:01:24.113 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:24.107+0000 7f9de8879700 1 -- 172.21.15.42:0/3744808066 wait complete. 2019-12-05T06:01:24.114 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:24.107+0000 7f9de8879700 1 -- 172.21.15.42:0/3744808066 >> 172.21.15.42:0/3744808066 conn(0x7f9de40a7f90 msgr2=0x7f9de40a9790 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:01:24.122 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.b (rank=0) 2019-12-05T06:01:24.123 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2019-12-05T06:01:24.123 INFO:teuthology.orchestra.run:waiting for 300 2019-12-05T06:01:24.137 INFO:tasks.ceph.mds.b:Stopped 2019-12-05T06:01:24.137 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:01:24.240 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:24.227+0000 7f1d7a233700 1 Processor -- start 2019-12-05T06:01:24.241 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:24.227+0000 7f1d7a233700 1 -- start start 2019-12-05T06:01:24.241 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:24.227+0000 7f1d7a233700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f1d740a7740 0x7f1d7408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:01:24.241 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:24.227+0000 7f1d7a233700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d740a0710 con 0x7f1d7407e720 2019-12-05T06:01:24.241 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:24.227+0000 7f1d7a233700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d740a9ba0 con 0x7f1d7409fae0 2019-12-05T06:01:24.241 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:24.227+0000 7f1d7a233700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d740a0b00 con 0x7f1d740a7740 2019-12-05T06:01:24.241 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:24.227+0000 7f1d73fff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f1d740a7740 0x7f1d7408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:01:24.241 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:24.227+0000 7f1d78fd1700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f1d7409fae0 0x7f1d7414efe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:46496/0 (socket says 172.21.15.42:46496) 2019-12-05T06:01:24.242 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:24.227+0000 7f1d73fff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f1d740a7740 0x7f1d7408df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:46454/0 (socket says 172.21.15.42:46454) 2019-12-05T06:01:24.242 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:24.227+0000 7f1d78fd1700 1 -- 172.21.15.42:0/1948674456 learned_addr learned my addr 172.21.15.42:0/1948674456 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:01:24.242 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:24.231+0000 7f1d73fff700 1 -- 172.21.15.42:0/1948674456 >> v1:172.21.15.42:6789/0 conn(0x7f1d7409fae0 legacy=0x7f1d7414efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:01:24.242 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:24.231+0000 7f1d72ffd700 1 -- 172.21.15.42:0/1948674456 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4014260233 0 0) 0x7f1d740a0710 con 0x7f1d7407e720 2019-12-05T06:01:24.242 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:24.231+0000 7f1d73fff700 1 -- 172.21.15.42:0/1948674456 >> v1:172.21.15.40:6789/0 conn(0x7f1d7407e720 legacy=0x7f1d7409f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:01:24.242 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:24.231+0000 7f1d73fff700 1 -- 172.21.15.42:0/1948674456 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d7407eb50 con 0x7f1d740a7740 2019-12-05T06:01:24.242 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:24.231+0000 7f1d73fff700 1 --2- 172.21.15.42:0/1948674456 >> v2:172.21.15.42:3300/0 conn(0x7f1d740a7740 0x7f1d7408df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f1d64008790 tx=0x7f1d6400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:24.243 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:24.231+0000 7f1d73fff700 1 --2- 172.21.15.42:0/1948674456 >> v2:172.21.15.42:3300/0 conn(0x7f1d740a7740 0x7f1d7408df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f1d64008790 tx=0x7f1d6400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:24.243 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:24.231+0000 7f1d73fff700 1 --2- 172.21.15.42:0/1948674456 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1d740a7740 0x7f1d7408df80 secure :-1 s=READY pgs=1907 cs=0 l=1 rx=0x7f1d64008790 tx=0x7f1d6400fbf0).ready entity=mon.1 client_cookie=645b0fab9b58f942 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:01:24.243 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:24.231+0000 7f1d73fff700 1 --2- 172.21.15.42:0/1948674456 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1d740a7740 0x7f1d7408df80 secure :-1 s=THROTTLE_DONE pgs=1907 cs=0 l=1 rx=0x7f1d64008790 tx=0x7f1d6400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:24.243 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:24.231+0000 7f1d72ffd700 1 -- 172.21.15.42:0/1948674456 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1d64011270 con 0x7f1d740a7740 2019-12-05T06:01:24.243 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:24.231+0000 7f1d73fff700 1 --2- 172.21.15.42:0/1948674456 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1d740a7740 0x7f1d7408df80 secure :-1 s=THROTTLE_DONE pgs=1907 cs=0 l=1 rx=0x7f1d64008790 tx=0x7f1d6400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:24.243 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:24.231+0000 7f1d73fff700 1 --2- 172.21.15.42:0/1948674456 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1d740a7740 0x7f1d7408df80 secure :-1 s=THROTTLE_DONE pgs=1907 cs=0 l=1 rx=0x7f1d64008790 tx=0x7f1d6400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:24.243 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:24.231+0000 7f1d72ffd700 1 -- 172.21.15.42:0/1948674456 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d64008bc0 con 0x7f1d740a7740 2019-12-05T06:01:24.244 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:24.231+0000 7f1d72ffd700 1 -- 172.21.15.42:0/1948674456 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1d64011850 con 0x7f1d740a7740 2019-12-05T06:01:24.244 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:24.231+0000 7f1d7a233700 1 -- 172.21.15.42:0/1948674456 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1d740a7740 msgr2=0x7f1d7408df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:01:24.244 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:24.231+0000 7f1d7a233700 1 --2- 172.21.15.42:0/1948674456 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1d740a7740 0x7f1d7408df80 secure :-1 s=READY pgs=1907 cs=0 l=1 rx=0x7f1d64008790 tx=0x7f1d6400fbf0).stop 2019-12-05T06:01:24.244 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:24.231+0000 7f1d7a233700 1 -- 172.21.15.42:0/1948674456 shutdown_connections 2019-12-05T06:01:24.244 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:24.231+0000 7f1d7a233700 1 --2- 172.21.15.42:0/1948674456 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1d740a7740 0x7f1d7408df80 unknown :-1 s=CLOSED pgs=1907 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:01:24.244 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:24.231+0000 7f1d7a233700 1 -- 172.21.15.42:0/1948674456 shutdown_connections 2019-12-05T06:01:24.245 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:24.231+0000 7f1d7a233700 1 -- 172.21.15.42:0/1948674456 wait complete. 2019-12-05T06:01:24.245 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:24.231+0000 7f1d7a233700 1 -- 172.21.15.42:0/1948674456 >> 172.21.15.42:0/1948674456 conn(0x7f1d740aabe0 msgr2=0x7f1d740a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:01:24.245 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:24.235+0000 7f1d7a233700 1 Processor -- start 2019-12-05T06:01:24.245 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:24.235+0000 7f1d7a233700 1 -- start start 2019-12-05T06:01:24.249 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:24.235+0000 7f1d7a233700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f1d740a6b50 0x7f1d7413c060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:01:24.250 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:24.235+0000 7f1d7a233700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d7411b690 con 0x7f1d741189d0 2019-12-05T06:01:24.250 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:24.235+0000 7f1d7a233700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d740a6f80 con 0x7f1d7411c320 2019-12-05T06:01:24.250 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:24.235+0000 7f1d7a233700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d74118e00 con 0x7f1d740a6b50 2019-12-05T06:01:24.250 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:24.235+0000 7f1d73fff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f1d7411c320 0x7f1d7411ae90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:46498/0 (socket says 172.21.15.42:46498) 2019-12-05T06:01:24.251 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:24.235+0000 7f1d73fff700 1 -- 172.21.15.42:0/1434305313 learned_addr learned my addr 172.21.15.42:0/1434305313 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:01:24.251 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:24.235+0000 7f1d78fd1700 1 --2- 172.21.15.42:0/1434305313 >> v2:172.21.15.42:3300/0 conn(0x7f1d740a6b50 0x7f1d7413c060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:01:24.251 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:24.235+0000 7f1d70ff9700 1 -- 172.21.15.42:0/1434305313 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1394924505 0 0) 0x7f1d740a6f80 con 0x7f1d7411c320 2019-12-05T06:01:24.251 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:24.235+0000 7f1d70ff9700 1 -- 172.21.15.42:0/1434305313 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d580030b0 con 0x7f1d7411c320 2019-12-05T06:01:24.251 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:24.235+0000 7f1d70ff9700 1 -- 172.21.15.42:0/1434305313 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1125107024 0 0) 0x7f1d7411b690 con 0x7f1d741189d0 2019-12-05T06:01:24.252 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:24.235+0000 7f1d70ff9700 1 -- 172.21.15.42:0/1434305313 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d740a6f80 con 0x7f1d741189d0 2019-12-05T06:01:24.252 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:24.235+0000 7f1d78fd1700 1 -- 172.21.15.42:0/1434305313 >> v1:172.21.15.42:6789/0 conn(0x7f1d7411c320 legacy=0x7f1d7411ae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:01:24.252 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:24.235+0000 7f1d78fd1700 1 -- 172.21.15.42:0/1434305313 >> v1:172.21.15.40:6789/0 conn(0x7f1d741189d0 legacy=0x7f1d74138840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:01:24.252 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:24.235+0000 7f1d78fd1700 1 -- 172.21.15.42:0/1434305313 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d7407d9e0 con 0x7f1d740a6b50 2019-12-05T06:01:24.252 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:24.235+0000 7f1d78fd1700 1 --2- 172.21.15.42:0/1434305313 >> v2:172.21.15.42:3300/0 conn(0x7f1d740a6b50 0x7f1d7413c060 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f1d74122140 tx=0x7f1d6c0107b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:24.252 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:24.235+0000 7f1d78fd1700 1 --2- 172.21.15.42:0/1434305313 >> v2:172.21.15.42:3300/0 conn(0x7f1d740a6b50 0x7f1d7413c060 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f1d74122140 tx=0x7f1d6c0107b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:24.252 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:24.235+0000 7f1d78fd1700 1 --2- 172.21.15.42:0/1434305313 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1d740a6b50 0x7f1d7413c060 secure :-1 s=READY pgs=1909 cs=0 l=1 rx=0x7f1d74122140 tx=0x7f1d6c0107b0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:01:24.253 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:24.235+0000 7f1d78fd1700 1 --2- 172.21.15.42:0/1434305313 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1d740a6b50 0x7f1d7413c060 secure :-1 s=THROTTLE_DONE pgs=1909 cs=0 l=1 rx=0x7f1d74122140 tx=0x7f1d6c0107b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:24.253 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:24.235+0000 7f1d70ff9700 1 -- 172.21.15.42:0/1434305313 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1d6c011340 con 0x7f1d740a6b50 2019-12-05T06:01:24.253 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:24.235+0000 7f1d78fd1700 1 --2- 172.21.15.42:0/1434305313 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1d740a6b50 0x7f1d7413c060 secure :-1 s=THROTTLE_DONE pgs=1909 cs=0 l=1 rx=0x7f1d74122140 tx=0x7f1d6c0107b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:24.253 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:24.235+0000 7f1d7a233700 1 -- 172.21.15.42:0/1434305313 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d7407cfe0 con 0x7f1d740a6b50 2019-12-05T06:01:24.253 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:24.235+0000 7f1d70ff9700 1 -- 172.21.15.42:0/1434305313 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d6c010f60 con 0x7f1d740a6b50 2019-12-05T06:01:24.253 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:24.235+0000 7f1d78fd1700 1 --2- 172.21.15.42:0/1434305313 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1d740a6b50 0x7f1d7413c060 secure :-1 s=THROTTLE_DONE pgs=1909 cs=0 l=1 rx=0x7f1d74122140 tx=0x7f1d6c0107b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:24.253 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:24.235+0000 7f1d70ff9700 1 -- 172.21.15.42:0/1434305313 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1d6c011920 con 0x7f1d740a6b50 2019-12-05T06:01:24.254 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:24.235+0000 7f1d7a233700 1 -- 172.21.15.42:0/1434305313 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d7413c5b0 con 0x7f1d740a6b50 2019-12-05T06:01:24.254 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:24.239+0000 7f1d78fd1700 1 --2- 172.21.15.42:0/1434305313 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1d740a6b50 0x7f1d7413c060 secure :-1 s=THROTTLE_DONE pgs=1909 cs=0 l=1 rx=0x7f1d74122140 tx=0x7f1d6c0107b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:24.254 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:24.239+0000 7f1d70ff9700 1 -- 172.21.15.42:0/1434305313 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f1d6c00b330 con 0x7f1d740a6b50 2019-12-05T06:01:24.254 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:24.239+0000 7f1d78fd1700 1 --2- 172.21.15.42:0/1434305313 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1d740a6b50 0x7f1d7413c060 secure :-1 s=THROTTLE_DONE pgs=1909 cs=0 l=1 rx=0x7f1d74122140 tx=0x7f1d6c0107b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:24.254 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:24.239+0000 7f1d70ff9700 1 --2- 172.21.15.42:0/1434305313 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f1d5803b420 0x7f1d5803d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:01:24.254 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:24.239+0000 7f1d70ff9700 1 -- 172.21.15.42:0/1434305313 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9471+0+0 (secure 0 0 0) 0x7f1d6c04ae30 con 0x7f1d740a6b50 2019-12-05T06:01:24.255 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:24.239+0000 7f1d73fff700 1 --2- 172.21.15.42:0/1434305313 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f1d5803b420 0x7f1d5803d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:01:24.255 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:24.239+0000 7f1d73fff700 1 --2- 172.21.15.42:0/1434305313 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f1d5803b420 0x7f1d5803d930 crc :-1 s=READY pgs=811 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:01:24.255 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:24.243+0000 7f1d7a233700 1 -- 172.21.15.42:0/1434305313 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d7405d110 con 0x7f1d740a6b50 2019-12-05T06:01:24.256 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:24.251+0000 7f1d78fd1700 1 --2- 172.21.15.42:0/1434305313 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1d740a6b50 0x7f1d7413c060 secure :-1 s=THROTTLE_DONE pgs=1909 cs=0 l=1 rx=0x7f1d74122140 tx=0x7f1d6c0107b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:24.257 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:24.251+0000 7f1d70ff9700 1 -- 172.21.15.42:0/1434305313 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f1d6c031eb0 con 0x7f1d740a6b50 2019-12-05T06:01:24.428 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:24.423+0000 7f1d7a233700 1 -- 172.21.15.42:0/1434305313 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1d7405b8c0 con 0x7f1d740a6b50 2019-12-05T06:01:24.429 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:24.423+0000 7f1d78fd1700 1 --2- 172.21.15.42:0/1434305313 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1d740a6b50 0x7f1d7413c060 secure :-1 s=THROTTLE_DONE pgs=1909 cs=0 l=1 rx=0x7f1d74122140 tx=0x7f1d6c0107b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:24.429 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:24.423+0000 7f1d70ff9700 1 -- 172.21.15.42:0/1434305313 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 233 v233) v1 ==== 95+0+2109 (secure 0 0 0) 0x7f1d6c011b20 con 0x7f1d740a6b50 2019-12-05T06:01:24.431 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 233 2019-12-05T06:01:24.431 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:01:24.432 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":233,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":9292,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6835/613535662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":613535662},{"type":"v1","addr":"172.21.15.42:6835","nonce":613535662}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":233}],"filesystems":[{"mdsmap":{"epoch":232,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:01:04.420072+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":89,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6653},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6653":{"gid":6653,"name":"b","rank":0,"incarnation":229,"state":"up:active","state_seq":13,"addr":"172.21.15.40:6833/2780168808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2780168808},{"type":"v1","addr":"172.21.15.40:6833","nonce":2780168808}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:01:24.435 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:24.427+0000 7f1d7a233700 1 -- 172.21.15.42:0/1434305313 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f1d5803b420 msgr2=0x7f1d5803d930 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:01:24.435 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:24.427+0000 7f1d7a233700 1 --2- 172.21.15.42:0/1434305313 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f1d5803b420 0x7f1d5803d930 crc :-1 s=READY pgs=811 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:01:24.435 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:24.427+0000 7f1d7a233700 1 -- 172.21.15.42:0/1434305313 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1d740a6b50 msgr2=0x7f1d7413c060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:01:24.435 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:24.427+0000 7f1d7a233700 1 --2- 172.21.15.42:0/1434305313 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1d740a6b50 0x7f1d7413c060 secure :-1 s=READY pgs=1909 cs=0 l=1 rx=0x7f1d74122140 tx=0x7f1d6c0107b0).stop 2019-12-05T06:01:24.436 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:24.427+0000 7f1d7a233700 1 -- 172.21.15.42:0/1434305313 shutdown_connections 2019-12-05T06:01:24.436 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:24.427+0000 7f1d7a233700 1 --2- 172.21.15.42:0/1434305313 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f1d5803b420 0x7f1d5803d930 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:01:24.436 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:24.427+0000 7f1d7a233700 1 --2- 172.21.15.42:0/1434305313 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1d740a6b50 0x7f1d7413c060 unknown :-1 s=CLOSED pgs=1909 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:01:24.437 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:24.427+0000 7f1d7a233700 1 -- 172.21.15.42:0/1434305313 shutdown_connections 2019-12-05T06:01:24.437 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:24.427+0000 7f1d7a233700 1 -- 172.21.15.42:0/1434305313 wait complete. 2019-12-05T06:01:24.437 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:24.427+0000 7f1d7a233700 1 -- 172.21.15.42:0/1434305313 >> 172.21.15.42:0/1434305313 conn(0x7f1d7408fda0 msgr2=0x7f1d7409f460 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:01:24.443 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:01:26.444 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:01:26.547 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:26.535+0000 7fa085533700 1 Processor -- start 2019-12-05T06:01:26.547 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:26.535+0000 7fa085533700 1 -- start start 2019-12-05T06:01:26.547 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:26.539+0000 7fa085533700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fa08009fae0 0x7fa08014fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:01:26.547 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:26.539+0000 7fa085533700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0800a0710 con 0x7fa08008df80 2019-12-05T06:01:26.547 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:26.539+0000 7fa085533700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa08007ef20 con 0x7fa0800a7740 2019-12-05T06:01:26.548 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:26.539+0000 7fa085533700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0800a0b00 con 0x7fa08009fae0 2019-12-05T06:01:26.548 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:26.539+0000 7fa07f7fe700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fa0800a7740 0x7fa08007e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:46516/0 (socket says 172.21.15.42:46516) 2019-12-05T06:01:26.548 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:26.539+0000 7fa07f7fe700 1 -- 172.21.15.42:0/656697426 learned_addr learned my addr 172.21.15.42:0/656697426 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:01:26.548 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:26.539+0000 7fa07ffff700 1 --2- 172.21.15.42:0/656697426 >> v2:172.21.15.42:3300/0 conn(0x7fa08009fae0 0x7fa08014fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:01:26.548 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:26.539+0000 7fa07e7fc700 1 -- 172.21.15.42:0/656697426 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 450331345 0 0) 0x7fa08007ef20 con 0x7fa0800a7740 2019-12-05T06:01:26.548 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:26.539+0000 7fa07e7fc700 1 -- 172.21.15.42:0/656697426 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0680030b0 con 0x7fa0800a7740 2019-12-05T06:01:26.548 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:26.539+0000 7fa07e7fc700 1 -- 172.21.15.42:0/656697426 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 856075666 0 0) 0x7fa0680030b0 con 0x7fa0800a7740 2019-12-05T06:01:26.549 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:26.539+0000 7fa07ffff700 1 -- 172.21.15.42:0/656697426 >> v1:172.21.15.42:6789/0 conn(0x7fa0800a7740 legacy=0x7fa08007e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:01:26.549 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:26.539+0000 7fa07ffff700 1 -- 172.21.15.42:0/656697426 >> v1:172.21.15.40:6789/0 conn(0x7fa08008df80 legacy=0x7fa08009f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:01:26.549 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:26.539+0000 7fa07ffff700 1 -- 172.21.15.42:0/656697426 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0800a9ba0 con 0x7fa08009fae0 2019-12-05T06:01:26.549 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:26.539+0000 7fa07ffff700 1 --2- 172.21.15.42:0/656697426 >> v2:172.21.15.42:3300/0 conn(0x7fa08009fae0 0x7fa08014fff0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fa074001110 tx=0x7fa074001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:26.549 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:26.539+0000 7fa07ffff700 1 --2- 172.21.15.42:0/656697426 >> v2:172.21.15.42:3300/0 conn(0x7fa08009fae0 0x7fa08014fff0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fa074001110 tx=0x7fa074001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:26.549 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:26.539+0000 7fa07ffff700 1 --2- 172.21.15.42:0/656697426 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa08009fae0 0x7fa08014fff0 secure :-1 s=READY pgs=1911 cs=0 l=1 rx=0x7fa074001110 tx=0x7fa074001150).ready entity=mon.1 client_cookie=e82ca82b76fa9f5d server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:01:26.549 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:26.539+0000 7fa07ffff700 1 --2- 172.21.15.42:0/656697426 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa08009fae0 0x7fa08014fff0 secure :-1 s=THROTTLE_DONE pgs=1911 cs=0 l=1 rx=0x7fa074001110 tx=0x7fa074001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:26.550 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:26.539+0000 7fa07ffff700 1 --2- 172.21.15.42:0/656697426 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa08009fae0 0x7fa08014fff0 secure :-1 s=THROTTLE_DONE pgs=1911 cs=0 l=1 rx=0x7fa074001110 tx=0x7fa074001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:26.550 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:26.539+0000 7fa07e7fc700 1 -- 172.21.15.42:0/656697426 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa070001750 con 0x7fa08009fae0 2019-12-05T06:01:26.550 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:26.539+0000 7fa07ffff700 1 --2- 172.21.15.42:0/656697426 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa08009fae0 0x7fa08014fff0 secure :-1 s=THROTTLE_DONE pgs=1911 cs=0 l=1 rx=0x7fa074001110 tx=0x7fa074001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:26.550 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:26.539+0000 7fa07e7fc700 1 -- 172.21.15.42:0/656697426 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa070001560 con 0x7fa08009fae0 2019-12-05T06:01:26.550 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:26.539+0000 7fa07e7fc700 1 -- 172.21.15.42:0/656697426 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa0780109e0 con 0x7fa08009fae0 2019-12-05T06:01:26.550 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:26.539+0000 7fa085533700 1 -- 172.21.15.42:0/656697426 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa08009fae0 msgr2=0x7fa08014fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:01:26.550 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:26.539+0000 7fa085533700 1 --2- 172.21.15.42:0/656697426 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa08009fae0 0x7fa08014fff0 secure :-1 s=READY pgs=1911 cs=0 l=1 rx=0x7fa074001110 tx=0x7fa074001150).stop 2019-12-05T06:01:26.551 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:26.539+0000 7fa085533700 1 -- 172.21.15.42:0/656697426 shutdown_connections 2019-12-05T06:01:26.551 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:26.539+0000 7fa085533700 1 --2- 172.21.15.42:0/656697426 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa08009fae0 0x7fa08014fff0 unknown :-1 s=CLOSED pgs=1911 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:01:26.551 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:26.539+0000 7fa085533700 1 -- 172.21.15.42:0/656697426 shutdown_connections 2019-12-05T06:01:26.551 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:26.539+0000 7fa085533700 1 -- 172.21.15.42:0/656697426 wait complete. 2019-12-05T06:01:26.551 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:26.539+0000 7fa085533700 1 -- 172.21.15.42:0/656697426 >> 172.21.15.42:0/656697426 conn(0x7fa0800aabe0 msgr2=0x7fa0800a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:01:26.559 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:26.543+0000 7fa085533700 1 Processor -- start 2019-12-05T06:01:26.560 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:26.543+0000 7fa085533700 1 -- start start 2019-12-05T06:01:26.560 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:26.543+0000 7fa085533700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa080119f60 0x7fa08012da20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:01:26.560 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:26.543+0000 7fa085533700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa08012df70 con 0x7fa08011c280 2019-12-05T06:01:26.560 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:26.543+0000 7fa085533700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa08007d910 con 0x7fa08012d1e0 2019-12-05T06:01:26.561 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:26.543+0000 7fa085533700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa08007bb50 con 0x7fa080119f60 2019-12-05T06:01:26.561 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:26.543+0000 7fa07ffff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fa08012d1e0 0x7fa08007d110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:46524/0 (socket says 172.21.15.42:46524) 2019-12-05T06:01:26.561 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:26.543+0000 7fa07ffff700 1 -- 172.21.15.42:0/1299792007 learned_addr learned my addr 172.21.15.42:0/1299792007 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:01:26.561 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:26.543+0000 7fa07effd700 1 --2- 172.21.15.42:0/1299792007 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa080119f60 0x7fa08012da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:01:26.562 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:26.547+0000 7fa067fff700 1 -- 172.21.15.42:0/1299792007 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3961266306 0 0) 0x7fa08007d910 con 0x7fa08012d1e0 2019-12-05T06:01:26.562 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:26.547+0000 7fa067fff700 1 -- 172.21.15.42:0/1299792007 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0600030b0 con 0x7fa08012d1e0 2019-12-05T06:01:26.562 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:26.547+0000 7fa067fff700 1 -- 172.21.15.42:0/1299792007 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2676747365 0 0) 0x7fa08012df70 con 0x7fa08011c280 2019-12-05T06:01:26.562 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:26.547+0000 7fa067fff700 1 -- 172.21.15.42:0/1299792007 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa08007d910 con 0x7fa08011c280 2019-12-05T06:01:26.563 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:26.547+0000 7fa067fff700 1 -- 172.21.15.42:0/1299792007 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 3155661317 0 0) 0x7fa0600030b0 con 0x7fa08012d1e0 2019-12-05T06:01:26.563 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:26.547+0000 7fa067fff700 1 -- 172.21.15.42:0/1299792007 --> v1:172.21.15.42:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa08012df70 con 0x7fa08012d1e0 2019-12-05T06:01:26.564 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:26.547+0000 7fa067fff700 1 -- 172.21.15.42:0/1299792007 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7fa078008810 con 0x7fa08012d1e0 2019-12-05T06:01:26.564 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:26.547+0000 7fa067fff700 1 -- 172.21.15.42:0/1299792007 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa080119f60 msgr2=0x7fa08012da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:01:26.564 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:26.547+0000 7fa067fff700 1 --2- 172.21.15.42:0/1299792007 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa080119f60 0x7fa08012da20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:01:26.564 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:26.547+0000 7fa067fff700 1 -- 172.21.15.42:0/1299792007 >> v1:172.21.15.42:6789/0 conn(0x7fa08012d1e0 legacy=0x7fa08007d110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:01:26.565 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:26.547+0000 7fa067fff700 1 -- 172.21.15.42:0/1299792007 >> v1:172.21.15.40:6789/0 conn(0x7fa08011c280 legacy=0x7fa08007bdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:01:26.565 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:26.547+0000 7fa067fff700 1 --2- 172.21.15.42:0/1299792007 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa060003ab0 0x7fa060004420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:01:26.565 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:26.547+0000 7fa067fff700 1 --2- 172.21.15.42:0/1299792007 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa060003f70 0x7fa0600049d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:01:26.566 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:26.547+0000 7fa067fff700 1 --2- 172.21.15.42:0/1299792007 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa0600095d0 0x7fa06000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:01:26.566 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:26.547+0000 7fa067fff700 1 -- 172.21.15.42:0/1299792007 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa08007bb50 con 0x7fa060003f70 2019-12-05T06:01:26.567 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:26.547+0000 7fa067fff700 1 -- 172.21.15.42:0/1299792007 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa06000bf60 con 0x7fa060003ab0 2019-12-05T06:01:26.567 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:26.547+0000 7fa067fff700 1 -- 172.21.15.42:0/1299792007 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa06000c120 con 0x7fa0600095d0 2019-12-05T06:01:26.567 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:26.547+0000 7fa07f7fe700 1 --2- 172.21.15.42:0/1299792007 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa060003ab0 0x7fa060004420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:01:26.568 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:26.547+0000 7fa07effd700 1 --2- 172.21.15.42:0/1299792007 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa080119f60 0x7fa08012da20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:01:26.568 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:26.547+0000 7fa07ffff700 1 --2- 172.21.15.42:0/1299792007 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa0600095d0 0x7fa06000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:01:26.568 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:26.547+0000 7fa07effd700 1 --2- 172.21.15.42:0/1299792007 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa060003f70 0x7fa0600049d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:01:26.569 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:26.547+0000 7fa07f7fe700 1 -- 172.21.15.42:0/1299792007 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa0600095d0 msgr2=0x7fa06000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:01:26.569 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:26.547+0000 7fa07f7fe700 1 --2- 172.21.15.42:0/1299792007 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa0600095d0 0x7fa06000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:01:26.569 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:26.547+0000 7fa07f7fe700 1 -- 172.21.15.42:0/1299792007 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa060003f70 msgr2=0x7fa0600049d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:01:26.569 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:26.547+0000 7fa07f7fe700 1 --2- 172.21.15.42:0/1299792007 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa060003f70 0x7fa0600049d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:01:26.569 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:26.547+0000 7fa07f7fe700 1 -- 172.21.15.42:0/1299792007 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa060003690 con 0x7fa060003ab0 2019-12-05T06:01:26.570 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:26.547+0000 7fa085533700 1 -- 172.21.15.42:0/1299792007 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa080119f60 con 0x7fa060003ab0 2019-12-05T06:01:26.570 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:26.547+0000 7fa07effd700 1 --2- 172.21.15.42:0/1299792007 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa060003f70 0x7fa0600049d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:01:26.570 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:26.547+0000 7fa07f7fe700 1 --2- 172.21.15.42:0/1299792007 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa060003ab0 0x7fa060004420 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fa070001350 tx=0x7fa070010b90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:26.570 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:26.547+0000 7fa085533700 1 -- 172.21.15.42:0/1299792007 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa08012da20 con 0x7fa060003ab0 2019-12-05T06:01:26.571 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:26.547+0000 7fa07f7fe700 1 --2- 172.21.15.42:0/1299792007 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa060003ab0 0x7fa060004420 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fa070001350 tx=0x7fa070010b90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:26.571 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:26.547+0000 7fa07f7fe700 1 --2- 172.21.15.42:0/1299792007 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa060003ab0 0x7fa060004420 secure :-1 s=READY pgs=1913 cs=0 l=1 rx=0x7fa070001350 tx=0x7fa070010b90).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:01:26.571 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:26.547+0000 7fa07f7fe700 1 --2- 172.21.15.42:0/1299792007 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa060003ab0 0x7fa060004420 secure :-1 s=THROTTLE_DONE pgs=1913 cs=0 l=1 rx=0x7fa070001350 tx=0x7fa070010b90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:26.571 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:26.547+0000 7fa067fff700 1 -- 172.21.15.42:0/1299792007 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa07000cb70 con 0x7fa060003ab0 2019-12-05T06:01:26.572 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:26.547+0000 7fa07f7fe700 1 --2- 172.21.15.42:0/1299792007 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa060003ab0 0x7fa060004420 secure :-1 s=THROTTLE_DONE pgs=1913 cs=0 l=1 rx=0x7fa070001350 tx=0x7fa070010b90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:26.572 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:26.547+0000 7fa07f7fe700 1 --2- 172.21.15.42:0/1299792007 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa060003ab0 0x7fa060004420 secure :-1 s=THROTTLE_DONE pgs=1913 cs=0 l=1 rx=0x7fa070001350 tx=0x7fa070010b90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:26.572 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:26.547+0000 7fa067fff700 1 -- 172.21.15.42:0/1299792007 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa07000c790 con 0x7fa060003ab0 2019-12-05T06:01:26.572 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:26.547+0000 7fa067fff700 1 -- 172.21.15.42:0/1299792007 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa070004620 con 0x7fa060003ab0 2019-12-05T06:01:26.572 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:26.547+0000 7fa07f7fe700 1 --2- 172.21.15.42:0/1299792007 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa060003ab0 0x7fa060004420 secure :-1 s=THROTTLE_DONE pgs=1913 cs=0 l=1 rx=0x7fa070001350 tx=0x7fa070010b90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:26.572 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:26.551+0000 7fa067fff700 1 -- 172.21.15.42:0/1299792007 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fa070004820 con 0x7fa060003ab0 2019-12-05T06:01:26.573 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:26.551+0000 7fa07f7fe700 1 --2- 172.21.15.42:0/1299792007 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa060003ab0 0x7fa060004420 secure :-1 s=THROTTLE_DONE pgs=1913 cs=0 l=1 rx=0x7fa070001350 tx=0x7fa070010b90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:26.573 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:26.551+0000 7fa067fff700 1 --2- 172.21.15.42:0/1299792007 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa0600461e0 0x7fa0600486f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:01:26.573 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:26.551+0000 7fa067fff700 1 -- 172.21.15.42:0/1299792007 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9471+0+0 (secure 0 0 0) 0x7fa07004aef0 con 0x7fa060003ab0 2019-12-05T06:01:26.573 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:26.551+0000 7fa07effd700 1 --2- 172.21.15.42:0/1299792007 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa0600461e0 0x7fa0600486f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:01:26.573 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:26.551+0000 7fa07effd700 1 --2- 172.21.15.42:0/1299792007 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa0600461e0 0x7fa0600486f0 crc :-1 s=READY pgs=812 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:01:26.574 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:26.551+0000 7fa085533700 1 -- 172.21.15.42:0/1299792007 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa058003200 con 0x7fa060003ab0 2019-12-05T06:01:26.576 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:26.559+0000 7fa07f7fe700 1 --2- 172.21.15.42:0/1299792007 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa060003ab0 0x7fa060004420 secure :-1 s=THROTTLE_DONE pgs=1913 cs=0 l=1 rx=0x7fa070001350 tx=0x7fa070010b90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:26.576 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:26.559+0000 7fa067fff700 1 -- 172.21.15.42:0/1299792007 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fa070013060 con 0x7fa060003ab0 2019-12-05T06:01:26.737 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:26.731+0000 7fa085533700 1 -- 172.21.15.42:0/1299792007 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa058003630 con 0x7fa060003ab0 2019-12-05T06:01:26.737 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:26.731+0000 7fa07f7fe700 1 --2- 172.21.15.42:0/1299792007 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa060003ab0 0x7fa060004420 secure :-1 s=THROTTLE_DONE pgs=1913 cs=0 l=1 rx=0x7fa070001350 tx=0x7fa070010b90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:26.738 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:26.731+0000 7fa067fff700 1 -- 172.21.15.42:0/1299792007 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 233 v233) v1 ==== 95+0+2109 (secure 0 0 0) 0x7fa070031f70 con 0x7fa060003ab0 2019-12-05T06:01:26.740 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 233 2019-12-05T06:01:26.740 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:01:26.740 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":233,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":9292,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6835/613535662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":613535662},{"type":"v1","addr":"172.21.15.42:6835","nonce":613535662}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":233}],"filesystems":[{"mdsmap":{"epoch":232,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:01:04.420072+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":89,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6653},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6653":{"gid":6653,"name":"b","rank":0,"incarnation":229,"state":"up:active","state_seq":13,"addr":"172.21.15.40:6833/2780168808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2780168808},{"type":"v1","addr":"172.21.15.40:6833","nonce":2780168808}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:01:26.743 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:26.739+0000 7fa085533700 1 -- 172.21.15.42:0/1299792007 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa0600461e0 msgr2=0x7fa0600486f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:01:26.744 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:26.739+0000 7fa085533700 1 --2- 172.21.15.42:0/1299792007 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa0600461e0 0x7fa0600486f0 crc :-1 s=READY pgs=812 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:01:26.744 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:26.739+0000 7fa085533700 1 -- 172.21.15.42:0/1299792007 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa060003ab0 msgr2=0x7fa060004420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:01:26.744 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:26.739+0000 7fa07ffff700 1 -- 172.21.15.42:0/1299792007 reap_dead start 2019-12-05T06:01:26.744 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:26.739+0000 7fa085533700 1 --2- 172.21.15.42:0/1299792007 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa060003ab0 0x7fa060004420 secure :-1 s=READY pgs=1913 cs=0 l=1 rx=0x7fa070001350 tx=0x7fa070010b90).stop 2019-12-05T06:01:26.744 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:26.739+0000 7fa085533700 1 -- 172.21.15.42:0/1299792007 shutdown_connections 2019-12-05T06:01:26.745 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:26.739+0000 7fa085533700 1 --2- 172.21.15.42:0/1299792007 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa060003ab0 0x7fa060004420 unknown :-1 s=CLOSED pgs=1913 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:01:26.746 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:26.739+0000 7fa085533700 1 -- 172.21.15.42:0/1299792007 shutdown_connections 2019-12-05T06:01:26.746 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:26.739+0000 7fa085533700 1 -- 172.21.15.42:0/1299792007 wait complete. 2019-12-05T06:01:26.746 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:26.739+0000 7fa085533700 1 -- 172.21.15.42:0/1299792007 >> 172.21.15.42:0/1299792007 conn(0x7fa0800a82a0 msgr2=0x7fa080059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:01:26.758 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:01:28.759 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:01:28.872 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:28.851+0000 7f5bcfe25700 1 Processor -- start 2019-12-05T06:01:28.873 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:28.851+0000 7f5bcfe25700 1 -- start start 2019-12-05T06:01:28.873 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:28.855+0000 7f5bcfe25700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5bc809fae0 0x7f5bc814fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:01:28.873 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:28.855+0000 7f5bcfe25700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bc80a0710 con 0x7f5bc808df80 2019-12-05T06:01:28.873 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:28.855+0000 7f5bcfe25700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bc807ef20 con 0x7f5bc80a7740 2019-12-05T06:01:28.873 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:28.855+0000 7f5bcfe25700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bc80a0b00 con 0x7f5bc809fae0 2019-12-05T06:01:28.873 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:28.855+0000 7f5bce3c2700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f5bc80a7740 0x7f5bc807e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:46538/0 (socket says 172.21.15.42:46538) 2019-12-05T06:01:28.874 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:28.855+0000 7f5bce3c2700 1 -- 172.21.15.42:0/2604914598 learned_addr learned my addr 172.21.15.42:0/2604914598 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:01:28.874 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:28.855+0000 7f5bcebc3700 1 --2- 172.21.15.42:0/2604914598 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5bc809fae0 0x7f5bc814fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:01:28.874 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:28.855+0000 7f5bcd3c0700 1 -- 172.21.15.42:0/2604914598 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3222037860 0 0) 0x7f5bc807ef20 con 0x7f5bc80a7740 2019-12-05T06:01:28.882 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:28.855+0000 7f5bcd3c0700 1 -- 172.21.15.42:0/2604914598 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5bb40030b0 con 0x7f5bc80a7740 2019-12-05T06:01:28.882 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:28.855+0000 7f5bcd3c0700 1 -- 172.21.15.42:0/2604914598 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 793231207 0 0) 0x7f5bb40030b0 con 0x7f5bc80a7740 2019-12-05T06:01:28.882 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:28.855+0000 7f5bcd3c0700 1 -- 172.21.15.42:0/2604914598 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5bc809fae0 msgr2=0x7f5bc814fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:01:28.882 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:28.855+0000 7f5bcd3c0700 1 --2- 172.21.15.42:0/2604914598 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5bc809fae0 0x7f5bc814fff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:01:28.883 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:28.855+0000 7f5bcd3c0700 1 -- 172.21.15.42:0/2604914598 >> v1:172.21.15.40:6789/0 conn(0x7f5bc808df80 legacy=0x7f5bc809f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:01:28.883 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:28.855+0000 7f5bcd3c0700 1 -- 172.21.15.42:0/2604914598 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bc80a9ba0 con 0x7f5bc80a7740 2019-12-05T06:01:28.883 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:28.855+0000 7f5bcd3c0700 1 -- 172.21.15.42:0/2604914598 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7f5bc4001750 con 0x7f5bc80a7740 2019-12-05T06:01:28.883 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:28.855+0000 7f5bcd3c0700 1 -- 172.21.15.42:0/2604914598 >> v1:172.21.15.42:6789/0 conn(0x7f5bc80a7740 legacy=0x7f5bc807e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:01:28.883 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:28.855+0000 7f5bcd3c0700 1 --2- 172.21.15.42:0/2604914598 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5bb4003710 0x7f5bb4003b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:01:28.883 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:28.855+0000 7f5bcd3c0700 1 --2- 172.21.15.42:0/2604914598 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5bb4004420 0x7f5bb4004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:01:28.883 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:28.855+0000 7f5bcd3c0700 1 --2- 172.21.15.42:0/2604914598 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5bb4004da0 0x7f5bb400b670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:01:28.884 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:28.855+0000 7f5bcd3c0700 1 -- 172.21.15.42:0/2604914598 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bc80a0b00 con 0x7f5bb4004420 2019-12-05T06:01:28.884 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:28.855+0000 7f5bce3c2700 1 --2- 172.21.15.42:0/2604914598 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5bb4003710 0x7f5bb4003b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:01:28.884 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:28.855+0000 7f5bcd3c0700 1 -- 172.21.15.42:0/2604914598 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bb4005200 con 0x7f5bb4003710 2019-12-05T06:01:28.884 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:28.855+0000 7f5bcd3c0700 1 -- 172.21.15.42:0/2604914598 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bb404fc00 con 0x7f5bb4004da0 2019-12-05T06:01:28.884 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:28.855+0000 7f5bcebc3700 1 --2- 172.21.15.42:0/2604914598 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5bb4004da0 0x7f5bb400b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:01:28.884 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:28.855+0000 7f5bcdbc1700 1 --2- 172.21.15.42:0/2604914598 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5bb4004420 0x7f5bb4004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:01:28.884 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:28.855+0000 7f5bce3c2700 1 -- 172.21.15.42:0/2604914598 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5bb4004da0 msgr2=0x7f5bb400b670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:01:28.885 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:28.855+0000 7f5bce3c2700 1 --2- 172.21.15.42:0/2604914598 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5bb4004da0 0x7f5bb400b670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:01:28.885 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:28.855+0000 7f5bce3c2700 1 -- 172.21.15.42:0/2604914598 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5bb4004420 msgr2=0x7f5bb4004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:01:28.885 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:28.855+0000 7f5bce3c2700 1 --2- 172.21.15.42:0/2604914598 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5bb4004420 0x7f5bb4004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:01:28.885 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:28.855+0000 7f5bce3c2700 1 -- 172.21.15.42:0/2604914598 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bb404ff00 con 0x7f5bb4003710 2019-12-05T06:01:28.885 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:28.855+0000 7f5bcebc3700 1 --2- 172.21.15.42:0/2604914598 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5bb4004da0 0x7f5bb400b670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:01:28.885 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:28.855+0000 7f5bce3c2700 1 --2- 172.21.15.42:0/2604914598 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5bb4003710 0x7f5bb4003b80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f5bc4001950 tx=0x7f5bc400c240).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:28.885 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:28.855+0000 7f5bce3c2700 1 --2- 172.21.15.42:0/2604914598 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5bb4003710 0x7f5bb4003b80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f5bc4001950 tx=0x7f5bc400c240).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:28.886 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:28.855+0000 7f5bce3c2700 1 --2- 172.21.15.42:0/2604914598 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5bb4003710 0x7f5bb4003b80 secure :-1 s=READY pgs=1915 cs=0 l=1 rx=0x7f5bc4001950 tx=0x7f5bc400c240).ready entity=mon.1 client_cookie=7aca4dc504481024 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:01:28.886 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:28.855+0000 7f5bce3c2700 1 -- 172.21.15.42:0/2604914598 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5bb4003710 msgr2=0x7f5bb4003b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2019-12-05T06:01:28.886 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:28.855+0000 7f5bce3c2700 1 -- 172.21.15.42:0/2604914598 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5bb4003710 msgr2=0x7f5bb4003b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2019-12-05T06:01:28.886 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:28.855+0000 7f5bce3c2700 1 --2- 172.21.15.42:0/2604914598 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5bb4003710 0x7f5bb4003b80 secure :-1 s=READY pgs=1915 cs=0 l=1 rx=0x7f5bc4001950 tx=0x7f5bc400c240).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-12-05T06:01:28.886 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:28.855+0000 7f5bce3c2700 1 --2- 172.21.15.42:0/2604914598 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5bb4003710 0x7f5bb4003b80 secure :-1 s=READY pgs=1915 cs=0 l=1 rx=0x7f5bc4001950 tx=0x7f5bc400c240).stop 2019-12-05T06:01:28.886 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:28.855+0000 7f5bcebc3700 1 -- 172.21.15.42:0/2604914598 reap_dead start 2019-12-05T06:01:28.887 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:28.855+0000 7f5bcd3c0700 1 -- 172.21.15.42:0/2604914598 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5bb4003710 msgr2=0x7f5bb4003b80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2019-12-05T06:01:28.887 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:28.855+0000 7f5bcd3c0700 1 --2- 172.21.15.42:0/2604914598 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5bb4003710 0x7f5bb4003b80 unknown :-1 s=CLOSED pgs=1915 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:01:28.887 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:28.855+0000 7f5bcd3c0700 1 --2- 172.21.15.42:0/2604914598 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5bb4004420 0x7f5bb4004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:01:28.887 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:28.855+0000 7f5bcd3c0700 1 --2- 172.21.15.42:0/2604914598 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5bb4009660 0x7f5bb40501d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:01:28.887 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:28.855+0000 7f5bcd3c0700 1 --2- 172.21.15.42:0/2604914598 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5bb4050720 0x7f5bb4052b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:01:28.887 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:28.855+0000 7f5bcd3c0700 1 -- 172.21.15.42:0/2604914598 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bb4053210 con 0x7f5bb4050720 2019-12-05T06:01:28.887 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:28.855+0000 7f5bcd3c0700 1 -- 172.21.15.42:0/2604914598 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bb4053400 con 0x7f5bb4004420 2019-12-05T06:01:28.888 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:28.855+0000 7f5bcd3c0700 1 -- 172.21.15.42:0/2604914598 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bb4053640 con 0x7f5bb4009660 2019-12-05T06:01:28.888 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:28.855+0000 7f5bce3c2700 1 --2- 172.21.15.42:0/2604914598 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5bb4004420 0x7f5bb4004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:01:28.888 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:28.855+0000 7f5bcdbc1700 1 --2- 172.21.15.42:0/2604914598 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5bb4009660 0x7f5bb40501d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:01:28.888 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:28.855+0000 7f5bcebc3700 1 --2- 172.21.15.42:0/2604914598 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5bb4050720 0x7f5bb4052b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:01:28.888 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:28.855+0000 7f5bce3c2700 1 -- 172.21.15.42:0/2604914598 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5bb4009660 msgr2=0x7f5bb40501d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:01:28.888 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:28.855+0000 7f5bce3c2700 1 --2- 172.21.15.42:0/2604914598 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5bb4009660 0x7f5bb40501d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:01:28.888 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:28.855+0000 7f5bce3c2700 1 -- 172.21.15.42:0/2604914598 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5bb4050720 msgr2=0x7f5bb4052b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:01:28.889 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:28.855+0000 7f5bce3c2700 1 --2- 172.21.15.42:0/2604914598 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5bb4050720 0x7f5bb4052b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:01:28.889 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:28.855+0000 7f5bce3c2700 1 -- 172.21.15.42:0/2604914598 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=2+}) v3 -- 0x7f5bb4003460 con 0x7f5bb4004420 2019-12-05T06:01:28.890 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:28.855+0000 7f5bcebc3700 1 --2- 172.21.15.42:0/2604914598 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5bb4050720 0x7f5bb4052b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).send_auth_request state changed! 2019-12-05T06:01:28.890 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:28.855+0000 7f5bce3c2700 1 --2- 172.21.15.42:0/2604914598 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5bb4004420 0x7f5bb4004850 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f5bc4001950 tx=0x7f5bc4008150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:28.890 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:28.855+0000 7f5bce3c2700 1 --2- 172.21.15.42:0/2604914598 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5bb4004420 0x7f5bb4004850 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f5bc4001950 tx=0x7f5bc4008150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:28.890 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:28.855+0000 7f5bce3c2700 1 --2- 172.21.15.42:0/2604914598 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5bb4004420 0x7f5bb4004850 secure :-1 s=READY pgs=1916 cs=0 l=1 rx=0x7f5bc4001950 tx=0x7f5bc4008150).ready entity=mon.1 client_cookie=1d465337f2ea15de server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:01:28.890 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:28.855+0000 7f5bce3c2700 1 --2- 172.21.15.42:0/2604914598 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5bb4004420 0x7f5bb4004850 secure :-1 s=THROTTLE_DONE pgs=1916 cs=0 l=1 rx=0x7f5bc4001950 tx=0x7f5bc4008150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:28.890 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:28.859+0000 7f5bcd3c0700 1 -- 172.21.15.42:0/2604914598 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5bc4011630 con 0x7f5bb4004420 2019-12-05T06:01:28.891 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:28.859+0000 7f5bce3c2700 1 --2- 172.21.15.42:0/2604914598 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5bb4004420 0x7f5bb4004850 secure :-1 s=THROTTLE_DONE pgs=1916 cs=0 l=1 rx=0x7f5bc4001950 tx=0x7f5bc4008150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:28.891 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:28.859+0000 7f5bcd3c0700 1 -- 172.21.15.42:0/2604914598 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bc4011250 con 0x7f5bb4004420 2019-12-05T06:01:28.892 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:28.859+0000 7f5bcfe25700 1 -- 172.21.15.42:0/2604914598 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5bb4004420 msgr2=0x7f5bb4004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:01:28.892 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:28.859+0000 7f5bcfe25700 1 --2- 172.21.15.42:0/2604914598 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5bb4004420 0x7f5bb4004850 secure :-1 s=READY pgs=1916 cs=0 l=1 rx=0x7f5bc4001950 tx=0x7f5bc4008150).stop 2019-12-05T06:01:28.892 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:28.859+0000 7f5bcfe25700 1 -- 172.21.15.42:0/2604914598 shutdown_connections 2019-12-05T06:01:28.892 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:28.859+0000 7f5bcfe25700 1 --2- 172.21.15.42:0/2604914598 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5bb4050720 0x7f5bb4052b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:01:28.892 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:28.859+0000 7f5bcfe25700 1 --2- 172.21.15.42:0/2604914598 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5bb4009660 0x7f5bb40501d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:01:28.892 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:28.859+0000 7f5bcfe25700 1 --2- 172.21.15.42:0/2604914598 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5bb4004420 0x7f5bb4004850 unknown :-1 s=CLOSED pgs=1916 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:01:28.892 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:28.859+0000 7f5bcfe25700 1 -- 172.21.15.42:0/2604914598 shutdown_connections 2019-12-05T06:01:28.893 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:28.859+0000 7f5bcfe25700 1 -- 172.21.15.42:0/2604914598 wait complete. 2019-12-05T06:01:28.893 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:28.859+0000 7f5bcfe25700 1 -- 172.21.15.42:0/2604914598 >> 172.21.15.42:0/2604914598 conn(0x7f5bc80aabe0 msgr2=0x7f5bc80a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:01:28.893 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:28.859+0000 7f5bcfe25700 1 Processor -- start 2019-12-05T06:01:28.893 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:28.859+0000 7f5bcfe25700 1 -- start start 2019-12-05T06:01:28.893 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:28.863+0000 7f5bcfe25700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5bc80a3ab0 0x7f5bc8124420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:01:28.893 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:28.863+0000 7f5bcfe25700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5bc80a2950 0x7f5bc80742d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:01:28.893 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:28.863+0000 7f5bcfe25700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bc80a6ba0 con 0x7f5bc80a6770 2019-12-05T06:01:28.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:28.863+0000 7f5bcfe25700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5bc80a4030 con 0x7f5bc80a3ab0 2019-12-05T06:01:28.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:28.863+0000 7f5bcfe25700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5bc80a42d0 con 0x7f5bc80a2950 2019-12-05T06:01:28.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:28.863+0000 7f5bcebc3700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5bc80a2950 0x7f5bc80742d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:01:28.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:28.863+0000 7f5bcebc3700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5bc80a2950 0x7f5bc80742d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:46522/0 (socket says 172.21.15.42:46522) 2019-12-05T06:01:28.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:28.863+0000 7f5bcebc3700 1 -- 172.21.15.42:0/1290905719 learned_addr learned my addr 172.21.15.42:0/1290905719 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:01:28.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:28.863+0000 7f5bce3c2700 1 --2- 172.21.15.42:0/1290905719 >> v2:172.21.15.40:3300/0 conn(0x7f5bc80a3ab0 0x7f5bc8124420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:01:28.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:28.863+0000 7f5bcebc3700 1 -- 172.21.15.42:0/1290905719 >> v2:172.21.15.40:3300/0 conn(0x7f5bc80a3ab0 msgr2=0x7f5bc8124420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:01:28.895 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:28.863+0000 7f5bcebc3700 1 --2- 172.21.15.42:0/1290905719 >> v2:172.21.15.40:3300/0 conn(0x7f5bc80a3ab0 0x7f5bc8124420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:01:28.895 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:28.863+0000 7f5bcebc3700 1 -- 172.21.15.42:0/1290905719 >> v1:172.21.15.40:6789/0 conn(0x7f5bc80a6770 legacy=0x7f5bc80a2150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:01:28.895 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:28.863+0000 7f5bcebc3700 1 -- 172.21.15.42:0/1290905719 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bc8075830 con 0x7f5bc80a2950 2019-12-05T06:01:28.895 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:28.863+0000 7f5bcebc3700 1 --2- 172.21.15.42:0/1290905719 >> v2:172.21.15.42:3300/0 conn(0x7f5bc80a2950 0x7f5bc80742d0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f5bc4001520 tx=0x7f5bc0010b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:28.895 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:28.863+0000 7f5bcebc3700 1 --2- 172.21.15.42:0/1290905719 >> v2:172.21.15.42:3300/0 conn(0x7f5bc80a2950 0x7f5bc80742d0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f5bc4001520 tx=0x7f5bc0010b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:28.895 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:28.863+0000 7f5bcebc3700 1 --2- 172.21.15.42:0/1290905719 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5bc80a2950 0x7f5bc80742d0 secure :-1 s=READY pgs=1917 cs=0 l=1 rx=0x7f5bc4001520 tx=0x7f5bc0010b00).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:01:28.895 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:28.863+0000 7f5bcebc3700 1 --2- 172.21.15.42:0/1290905719 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5bc80a2950 0x7f5bc80742d0 secure :-1 s=THROTTLE_DONE pgs=1917 cs=0 l=1 rx=0x7f5bc4001520 tx=0x7f5bc0010b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:28.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:28.863+0000 7f5bbeffd700 1 -- 172.21.15.42:0/1290905719 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5bc0005ca0 con 0x7f5bc80a2950 2019-12-05T06:01:28.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:28.863+0000 7f5bcebc3700 1 --2- 172.21.15.42:0/1290905719 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5bc80a2950 0x7f5bc80742d0 secure :-1 s=THROTTLE_DONE pgs=1917 cs=0 l=1 rx=0x7f5bc4001520 tx=0x7f5bc0010b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:28.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:28.863+0000 7f5bbeffd700 1 -- 172.21.15.42:0/1290905719 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bc00058c0 con 0x7f5bc80a2950 2019-12-05T06:01:28.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:28.863+0000 7f5bcebc3700 1 --2- 172.21.15.42:0/1290905719 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5bc80a2950 0x7f5bc80742d0 secure :-1 s=THROTTLE_DONE pgs=1917 cs=0 l=1 rx=0x7f5bc4001520 tx=0x7f5bc0010b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:28.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:28.863+0000 7f5bcfe25700 1 -- 172.21.15.42:0/1290905719 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bc8074820 con 0x7f5bc80a2950 2019-12-05T06:01:28.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:28.863+0000 7f5bbeffd700 1 -- 172.21.15.42:0/1290905719 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5bc0012860 con 0x7f5bc80a2950 2019-12-05T06:01:28.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:28.863+0000 7f5bcfe25700 1 -- 172.21.15.42:0/1290905719 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bc8074dd0 con 0x7f5bc80a2950 2019-12-05T06:01:28.897 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:28.863+0000 7f5bcebc3700 1 --2- 172.21.15.42:0/1290905719 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5bc80a2950 0x7f5bc80742d0 secure :-1 s=THROTTLE_DONE pgs=1917 cs=0 l=1 rx=0x7f5bc4001520 tx=0x7f5bc0010b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:28.897 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:28.863+0000 7f5bbeffd700 1 -- 172.21.15.42:0/1290905719 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f5bc0012a60 con 0x7f5bc80a2950 2019-12-05T06:01:28.897 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:28.863+0000 7f5bcebc3700 1 --2- 172.21.15.42:0/1290905719 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5bc80a2950 0x7f5bc80742d0 secure :-1 s=THROTTLE_DONE pgs=1917 cs=0 l=1 rx=0x7f5bc4001520 tx=0x7f5bc0010b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:28.897 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:28.863+0000 7f5bbeffd700 1 --2- 172.21.15.42:0/1290905719 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5bb003adf0 0x7f5bb003d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:01:28.897 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:28.867+0000 7f5bbeffd700 1 -- 172.21.15.42:0/1290905719 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9471+0+0 (secure 0 0 0) 0x7f5bc004ad20 con 0x7f5bc80a2950 2019-12-05T06:01:28.897 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:28.867+0000 7f5bce3c2700 1 --2- 172.21.15.42:0/1290905719 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5bb003adf0 0x7f5bb003d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:01:28.897 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:28.867+0000 7f5bce3c2700 1 --2- 172.21.15.42:0/1290905719 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5bb003adf0 0x7f5bb003d300 crc :-1 s=READY pgs=813 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:01:28.898 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:28.867+0000 7f5bcfe25700 1 -- 172.21.15.42:0/1290905719 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5bc805d110 con 0x7f5bc80a2950 2019-12-05T06:01:28.898 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:28.875+0000 7f5bcebc3700 1 --2- 172.21.15.42:0/1290905719 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5bc80a2950 0x7f5bc80742d0 secure :-1 s=THROTTLE_DONE pgs=1917 cs=0 l=1 rx=0x7f5bc4001520 tx=0x7f5bc0010b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:28.899 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:28.875+0000 7f5bbeffd700 1 -- 172.21.15.42:0/1290905719 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f5bc0012d70 con 0x7f5bc80a2950 2019-12-05T06:01:29.050 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:29.043+0000 7f5bcfe25700 1 -- 172.21.15.42:0/1290905719 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5bc805b8c0 con 0x7f5bc80a2950 2019-12-05T06:01:29.051 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:29.043+0000 7f5bcebc3700 1 --2- 172.21.15.42:0/1290905719 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5bc80a2950 0x7f5bc80742d0 secure :-1 s=THROTTLE_DONE pgs=1917 cs=0 l=1 rx=0x7f5bc4001520 tx=0x7f5bc0010b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:29.051 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:29.047+0000 7f5bbeffd700 1 -- 172.21.15.42:0/1290905719 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 233 v233) v1 ==== 95+0+2109 (secure 0 0 0) 0x7f5bc0031d80 con 0x7f5bc80a2950 2019-12-05T06:01:29.051 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 233 2019-12-05T06:01:29.052 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:01:29.052 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":233,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":9292,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6835/613535662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":613535662},{"type":"v1","addr":"172.21.15.42:6835","nonce":613535662}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":233}],"filesystems":[{"mdsmap":{"epoch":232,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:01:04.420072+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":89,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6653},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6653":{"gid":6653,"name":"b","rank":0,"incarnation":229,"state":"up:active","state_seq":13,"addr":"172.21.15.40:6833/2780168808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2780168808},{"type":"v1","addr":"172.21.15.40:6833","nonce":2780168808}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:01:29.055 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:29.047+0000 7f5bcfe25700 1 -- 172.21.15.42:0/1290905719 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5bb003adf0 msgr2=0x7f5bb003d300 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:01:29.055 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:29.047+0000 7f5bcfe25700 1 --2- 172.21.15.42:0/1290905719 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5bb003adf0 0x7f5bb003d300 crc :-1 s=READY pgs=813 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:01:29.055 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:29.047+0000 7f5bcfe25700 1 -- 172.21.15.42:0/1290905719 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5bc80a2950 msgr2=0x7f5bc80742d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:01:29.055 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:29.047+0000 7f5bcfe25700 1 --2- 172.21.15.42:0/1290905719 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5bc80a2950 0x7f5bc80742d0 secure :-1 s=READY pgs=1917 cs=0 l=1 rx=0x7f5bc4001520 tx=0x7f5bc0010b00).stop 2019-12-05T06:01:29.055 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:29.047+0000 7f5bcfe25700 1 -- 172.21.15.42:0/1290905719 shutdown_connections 2019-12-05T06:01:29.056 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:29.047+0000 7f5bcfe25700 1 --2- 172.21.15.42:0/1290905719 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5bb003adf0 0x7f5bb003d300 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:01:29.056 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:29.047+0000 7f5bcfe25700 1 --2- 172.21.15.42:0/1290905719 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5bc80a2950 0x7f5bc80742d0 unknown :-1 s=CLOSED pgs=1917 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:01:29.056 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:29.047+0000 7f5bcfe25700 1 --2- 172.21.15.42:0/1290905719 >> v2:172.21.15.40:3300/0 conn(0x7f5bc80a3ab0 0x7f5bc8124420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:01:29.056 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:29.047+0000 7f5bcfe25700 1 -- 172.21.15.42:0/1290905719 shutdown_connections 2019-12-05T06:01:29.056 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:29.047+0000 7f5bcfe25700 1 -- 172.21.15.42:0/1290905719 wait complete. 2019-12-05T06:01:29.056 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:29.047+0000 7f5bcfe25700 1 -- 172.21.15.42:0/1290905719 >> 172.21.15.42:0/1290905719 conn(0x7f5bc80a01e0 msgr2=0x7f5bc8059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:01:29.063 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:01:31.065 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:01:31.173 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:31.159+0000 7ff8f8ed3700 1 Processor -- start 2019-12-05T06:01:31.173 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:31.159+0000 7ff8f8ed3700 1 -- start start 2019-12-05T06:01:31.173 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:31.163+0000 7ff8f8ed3700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff8f40a7740 0x7ff8f408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:01:31.174 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:31.163+0000 7ff8f8ed3700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff8f407e720 0x7ff8f407eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:01:31.174 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:31.163+0000 7ff8f8ed3700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8f40a0710 con 0x7ff8f409f2e0 2019-12-05T06:01:31.174 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:31.163+0000 7ff8f8ed3700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8f40a0b00 con 0x7ff8f40a7740 2019-12-05T06:01:31.175 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:31.163+0000 7ff8f8ed3700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8f408e5b0 con 0x7ff8f407e720 2019-12-05T06:01:31.176 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:31.163+0000 7ff8f359e700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7ff8f409f2e0 0x7ff8f409f710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:46572/0 (socket says 172.21.15.42:46572) 2019-12-05T06:01:31.176 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:31.163+0000 7ff8f359e700 1 -- 172.21.15.42:0/3057929826 learned_addr learned my addr 172.21.15.42:0/3057929826 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:01:31.176 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:31.163+0000 7ff8f2d9d700 1 --2- 172.21.15.42:0/3057929826 >> v2:172.21.15.40:3300/0 conn(0x7ff8f40a7740 0x7ff8f408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:01:31.177 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:31.163+0000 7ff8f259c700 1 --2- 172.21.15.42:0/3057929826 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff8f407e720 0x7ff8f407eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:01:31.177 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:31.163+0000 7ff8f1d9b700 1 -- 172.21.15.42:0/3057929826 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3626180989 0 0) 0x7ff8f40a0710 con 0x7ff8f409f2e0 2019-12-05T06:01:31.177 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:31.163+0000 7ff8f1d9b700 1 -- 172.21.15.42:0/3057929826 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8e00030b0 con 0x7ff8f409f2e0 2019-12-05T06:01:31.177 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:31.163+0000 7ff8f259c700 1 -- 172.21.15.42:0/3057929826 >> v2:172.21.15.40:3300/0 conn(0x7ff8f40a7740 msgr2=0x7ff8f408df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:01:31.177 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:31.163+0000 7ff8f259c700 1 --2- 172.21.15.42:0/3057929826 >> v2:172.21.15.40:3300/0 conn(0x7ff8f40a7740 0x7ff8f408df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:01:31.178 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:31.163+0000 7ff8f259c700 1 -- 172.21.15.42:0/3057929826 >> v1:172.21.15.42:6789/0 conn(0x7ff8f409f2e0 legacy=0x7ff8f409f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:01:31.178 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:31.163+0000 7ff8f259c700 1 -- 172.21.15.42:0/3057929826 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8f407f0a0 con 0x7ff8f407e720 2019-12-05T06:01:31.178 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:31.163+0000 7ff8f2d9d700 1 --2- 172.21.15.42:0/3057929826 >> v2:172.21.15.40:3300/0 conn(0x7ff8f40a7740 0x7ff8f408df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:01:31.178 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:31.163+0000 7ff8f259c700 1 --2- 172.21.15.42:0/3057929826 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff8f407e720 0x7ff8f407eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7ff8dc008790 tx=0x7ff8dc00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:31.179 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:31.163+0000 7ff8f259c700 1 --2- 172.21.15.42:0/3057929826 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff8f407e720 0x7ff8f407eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7ff8dc008790 tx=0x7ff8dc00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:31.179 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:31.163+0000 7ff8f259c700 1 --2- 172.21.15.42:0/3057929826 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff8f407e720 0x7ff8f407eb50 secure :-1 s=READY pgs=425 cs=0 l=1 rx=0x7ff8dc008790 tx=0x7ff8dc00fc70).ready entity=mon.2 client_cookie=9544acf34a5cfd8b server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:01:31.179 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:31.163+0000 7ff8f259c700 1 --2- 172.21.15.42:0/3057929826 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff8f407e720 0x7ff8f407eb50 secure :-1 s=THROTTLE_DONE pgs=425 cs=0 l=1 rx=0x7ff8dc008790 tx=0x7ff8dc00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:31.179 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:31.163+0000 7ff8f1d9b700 1 -- 172.21.15.42:0/3057929826 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff8dc0112f0 con 0x7ff8f407e720 2019-12-05T06:01:31.179 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:31.163+0000 7ff8f259c700 1 --2- 172.21.15.42:0/3057929826 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff8f407e720 0x7ff8f407eb50 secure :-1 s=THROTTLE_DONE pgs=425 cs=0 l=1 rx=0x7ff8dc008790 tx=0x7ff8dc00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:31.179 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:31.163+0000 7ff8f1d9b700 1 -- 172.21.15.42:0/3057929826 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8dc008ad0 con 0x7ff8f407e720 2019-12-05T06:01:31.180 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:31.163+0000 7ff8f259c700 1 --2- 172.21.15.42:0/3057929826 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff8f407e720 0x7ff8f407eb50 secure :-1 s=THROTTLE_DONE pgs=425 cs=0 l=1 rx=0x7ff8dc008790 tx=0x7ff8dc00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:31.180 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:31.163+0000 7ff8f1d9b700 1 -- 172.21.15.42:0/3057929826 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff8dc0118d0 con 0x7ff8f407e720 2019-12-05T06:01:31.180 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:31.163+0000 7ff8f8ed3700 1 -- 172.21.15.42:0/3057929826 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff8f407e720 msgr2=0x7ff8f407eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:01:31.180 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:31.163+0000 7ff8f8ed3700 1 --2- 172.21.15.42:0/3057929826 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff8f407e720 0x7ff8f407eb50 secure :-1 s=READY pgs=425 cs=0 l=1 rx=0x7ff8dc008790 tx=0x7ff8dc00fc70).stop 2019-12-05T06:01:31.180 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:31.163+0000 7ff8f8ed3700 1 -- 172.21.15.42:0/3057929826 shutdown_connections 2019-12-05T06:01:31.180 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:31.163+0000 7ff8f8ed3700 1 --2- 172.21.15.42:0/3057929826 >> v2:172.21.15.40:3300/0 conn(0x7ff8f40a7740 0x7ff8f408df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:01:31.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:31.163+0000 7ff8f8ed3700 1 --2- 172.21.15.42:0/3057929826 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff8f407e720 0x7ff8f407eb50 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:01:31.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:31.163+0000 7ff8f8ed3700 1 -- 172.21.15.42:0/3057929826 shutdown_connections 2019-12-05T06:01:31.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:31.163+0000 7ff8f8ed3700 1 -- 172.21.15.42:0/3057929826 wait complete. 2019-12-05T06:01:31.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:31.163+0000 7ff8f8ed3700 1 -- 172.21.15.42:0/3057929826 >> 172.21.15.42:0/3057929826 conn(0x7ff8f40aabe0 msgr2=0x7ff8f40a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:01:31.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:31.167+0000 7ff8f8ed3700 1 Processor -- start 2019-12-05T06:01:31.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:31.167+0000 7ff8f8ed3700 1 -- start start 2019-12-05T06:01:31.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:31.167+0000 7ff8f8ed3700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff8f41179d0 0x7ff8f411bad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:01:31.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:31.167+0000 7ff8f8ed3700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff8f411a360 0x7ff8f412f280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:01:31.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:31.167+0000 7ff8f8ed3700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8f411a790 con 0x7ff8f4118650 2019-12-05T06:01:31.194 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:31.167+0000 7ff8f8ed3700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8f4118b60 con 0x7ff8f41179d0 2019-12-05T06:01:31.194 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:31.167+0000 7ff8f8ed3700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8f4118e00 con 0x7ff8f411a360 2019-12-05T06:01:31.194 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:31.167+0000 7ff8f259c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff8f41179d0 0x7ff8f411bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:01:31.194 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:31.167+0000 7ff8f259c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff8f41179d0 0x7ff8f411bad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:46540/0 (socket says 172.21.15.42:46540) 2019-12-05T06:01:31.194 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:31.167+0000 7ff8f259c700 1 -- 172.21.15.42:0/1144095199 learned_addr learned my addr 172.21.15.42:0/1144095199 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:01:31.194 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:31.167+0000 7ff8f359e700 1 --2- 172.21.15.42:0/1144095199 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff8f411a360 0x7ff8f412f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:01:31.202 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:31.167+0000 7ff8f259c700 1 -- 172.21.15.42:0/1144095199 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff8f411a360 msgr2=0x7ff8f412f280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:01:31.202 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:31.167+0000 7ff8f259c700 1 --2- 172.21.15.42:0/1144095199 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff8f411a360 0x7ff8f412f280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:01:31.202 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:31.167+0000 7ff8f259c700 1 -- 172.21.15.42:0/1144095199 >> v1:172.21.15.42:6789/0 conn(0x7ff8f4118650 legacy=0x7ff8f41171d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:01:31.203 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:31.167+0000 7ff8f259c700 1 -- 172.21.15.42:0/1144095199 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8f412f7d0 con 0x7ff8f41179d0 2019-12-05T06:01:31.204 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:31.167+0000 7ff8f259c700 1 --2- 172.21.15.42:0/1144095199 >> v2:172.21.15.42:3300/0 conn(0x7ff8f41179d0 0x7ff8f411bad0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7ff8dc008790 tx=0x7ff8dc011c60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:31.204 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:31.167+0000 7ff8f259c700 1 --2- 172.21.15.42:0/1144095199 >> v2:172.21.15.42:3300/0 conn(0x7ff8f41179d0 0x7ff8f411bad0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7ff8dc008790 tx=0x7ff8dc011c60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:31.205 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:31.171+0000 7ff8f259c700 1 --2- 172.21.15.42:0/1144095199 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff8f41179d0 0x7ff8f411bad0 secure :-1 s=READY pgs=1920 cs=0 l=1 rx=0x7ff8dc008790 tx=0x7ff8dc011c60).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:01:31.205 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:31.171+0000 7ff8f259c700 1 --2- 172.21.15.42:0/1144095199 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff8f41179d0 0x7ff8f411bad0 secure :-1 s=THROTTLE_DONE pgs=1920 cs=0 l=1 rx=0x7ff8dc008790 tx=0x7ff8dc011c60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:31.205 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:31.171+0000 7ff8f259c700 1 --2- 172.21.15.42:0/1144095199 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff8f41179d0 0x7ff8f411bad0 secure :-1 s=THROTTLE_DONE pgs=1920 cs=0 l=1 rx=0x7ff8dc008790 tx=0x7ff8dc011c60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:31.205 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:31.171+0000 7ff8eb7fe700 1 -- 172.21.15.42:0/1144095199 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff8dc007150 con 0x7ff8f41179d0 2019-12-05T06:01:31.207 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:31.171+0000 7ff8f259c700 1 --2- 172.21.15.42:0/1144095199 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff8f41179d0 0x7ff8f411bad0 secure :-1 s=THROTTLE_DONE pgs=1920 cs=0 l=1 rx=0x7ff8dc008790 tx=0x7ff8dc011c60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:31.207 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:31.171+0000 7ff8eb7fe700 1 -- 172.21.15.42:0/1144095199 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8dc0114f0 con 0x7ff8f41179d0 2019-12-05T06:01:31.207 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:31.171+0000 7ff8f8ed3700 1 -- 172.21.15.42:0/1144095199 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8f412fa60 con 0x7ff8f41179d0 2019-12-05T06:01:31.207 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:31.171+0000 7ff8eb7fe700 1 -- 172.21.15.42:0/1144095199 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff8dc008cc0 con 0x7ff8f41179d0 2019-12-05T06:01:31.207 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:31.171+0000 7ff8f8ed3700 1 -- 172.21.15.42:0/1144095199 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8f4077300 con 0x7ff8f41179d0 2019-12-05T06:01:31.207 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:31.171+0000 7ff8f259c700 1 --2- 172.21.15.42:0/1144095199 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff8f41179d0 0x7ff8f411bad0 secure :-1 s=THROTTLE_DONE pgs=1920 cs=0 l=1 rx=0x7ff8dc008790 tx=0x7ff8dc011c60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:31.208 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:31.171+0000 7ff8eb7fe700 1 -- 172.21.15.42:0/1144095199 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7ff8dc005540 con 0x7ff8f41179d0 2019-12-05T06:01:31.208 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:31.171+0000 7ff8f259c700 1 --2- 172.21.15.42:0/1144095199 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff8f41179d0 0x7ff8f411bad0 secure :-1 s=THROTTLE_DONE pgs=1920 cs=0 l=1 rx=0x7ff8dc008790 tx=0x7ff8dc011c60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:31.208 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:31.171+0000 7ff8eb7fe700 1 --2- 172.21.15.42:0/1144095199 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff8d803ace0 0x7ff8d803d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:01:31.208 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:31.171+0000 7ff8eb7fe700 1 -- 172.21.15.42:0/1144095199 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9471+0+0 (secure 0 0 0) 0x7ff8dc04b5c0 con 0x7ff8f41179d0 2019-12-05T06:01:31.208 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:31.171+0000 7ff8f2d9d700 1 --2- 172.21.15.42:0/1144095199 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff8d803ace0 0x7ff8d803d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:01:31.208 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:31.171+0000 7ff8f2d9d700 1 --2- 172.21.15.42:0/1144095199 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff8d803ace0 0x7ff8d803d1f0 crc :-1 s=READY pgs=814 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:01:31.208 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:31.175+0000 7ff8f8ed3700 1 -- 172.21.15.42:0/1144095199 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8f405d110 con 0x7ff8f41179d0 2019-12-05T06:01:31.209 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:31.187+0000 7ff8f259c700 1 --2- 172.21.15.42:0/1144095199 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff8f41179d0 0x7ff8f411bad0 secure :-1 s=THROTTLE_DONE pgs=1920 cs=0 l=1 rx=0x7ff8dc008790 tx=0x7ff8dc011c60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:31.209 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:31.187+0000 7ff8eb7fe700 1 -- 172.21.15.42:0/1144095199 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7ff8dc007f50 con 0x7ff8f41179d0 2019-12-05T06:01:31.360 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:31.355+0000 7ff8f8ed3700 1 -- 172.21.15.42:0/1144095199 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff8f405b8c0 con 0x7ff8f41179d0 2019-12-05T06:01:31.361 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:31.355+0000 7ff8f259c700 1 --2- 172.21.15.42:0/1144095199 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff8f41179d0 0x7ff8f411bad0 secure :-1 s=THROTTLE_DONE pgs=1920 cs=0 l=1 rx=0x7ff8dc008790 tx=0x7ff8dc011c60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:31.362 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:31.355+0000 7ff8eb7fe700 1 -- 172.21.15.42:0/1144095199 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 233 v233) v1 ==== 95+0+2109 (secure 0 0 0) 0x7ff8dc0328a0 con 0x7ff8f41179d0 2019-12-05T06:01:31.362 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 233 2019-12-05T06:01:31.362 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:01:31.362 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":233,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":9292,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6835/613535662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":613535662},{"type":"v1","addr":"172.21.15.42:6835","nonce":613535662}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":233}],"filesystems":[{"mdsmap":{"epoch":232,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:01:04.420072+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":89,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6653},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6653":{"gid":6653,"name":"b","rank":0,"incarnation":229,"state":"up:active","state_seq":13,"addr":"172.21.15.40:6833/2780168808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2780168808},{"type":"v1","addr":"172.21.15.40:6833","nonce":2780168808}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:01:31.365 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:31.359+0000 7ff8f8ed3700 1 -- 172.21.15.42:0/1144095199 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff8d803ace0 msgr2=0x7ff8d803d1f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:01:31.365 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:31.359+0000 7ff8f8ed3700 1 --2- 172.21.15.42:0/1144095199 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff8d803ace0 0x7ff8d803d1f0 crc :-1 s=READY pgs=814 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:01:31.366 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:31.359+0000 7ff8f8ed3700 1 -- 172.21.15.42:0/1144095199 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff8f41179d0 msgr2=0x7ff8f411bad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:01:31.366 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:31.359+0000 7ff8f8ed3700 1 --2- 172.21.15.42:0/1144095199 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff8f41179d0 0x7ff8f411bad0 secure :-1 s=READY pgs=1920 cs=0 l=1 rx=0x7ff8dc008790 tx=0x7ff8dc011c60).stop 2019-12-05T06:01:31.366 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:31.359+0000 7ff8f8ed3700 1 -- 172.21.15.42:0/1144095199 shutdown_connections 2019-12-05T06:01:31.366 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:31.359+0000 7ff8f8ed3700 1 --2- 172.21.15.42:0/1144095199 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff8d803ace0 0x7ff8d803d1f0 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:01:31.367 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:31.359+0000 7ff8f8ed3700 1 --2- 172.21.15.42:0/1144095199 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff8f411a360 0x7ff8f412f280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:01:31.367 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:31.359+0000 7ff8f8ed3700 1 --2- 172.21.15.42:0/1144095199 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff8f41179d0 0x7ff8f411bad0 unknown :-1 s=CLOSED pgs=1920 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:01:31.367 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:31.359+0000 7ff8f8ed3700 1 -- 172.21.15.42:0/1144095199 shutdown_connections 2019-12-05T06:01:31.367 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:31.359+0000 7ff8f8ed3700 1 -- 172.21.15.42:0/1144095199 wait complete. 2019-12-05T06:01:31.367 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:31.359+0000 7ff8f8ed3700 1 -- 172.21.15.42:0/1144095199 >> 172.21.15.42:0/1144095199 conn(0x7ff8f407e4f0 msgr2=0x7ff8f40a9640 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:01:31.374 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:01:33.375 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:01:33.484 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:33.471+0000 7f1a90b9b700 1 Processor -- start 2019-12-05T06:01:33.487 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:33.471+0000 7f1a90b9b700 1 -- start start 2019-12-05T06:01:33.488 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:33.471+0000 7f1a90b9b700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f1a8c08df80 0x7f1a8c09f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:01:33.488 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:33.471+0000 7f1a90b9b700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1a8c09f830 0x7f1a8c14efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:01:33.488 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:33.471+0000 7f1a90b9b700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a8c09fc60 con 0x7f1a8c0a7740 2019-12-05T06:01:33.488 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:33.471+0000 7f1a90b9b700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a8c0a0b00 con 0x7f1a8c08df80 2019-12-05T06:01:33.488 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:33.471+0000 7f1a90b9b700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a8c08e5b0 con 0x7f1a8c09f830 2019-12-05T06:01:33.488 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:33.471+0000 7f1a8a59c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f1a8c08df80 0x7f1a8c09f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:01:33.489 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:33.471+0000 7f1a8ad9d700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f1a8c0a7740 0x7f1a8c07e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:46584/0 (socket says 172.21.15.42:46584) 2019-12-05T06:01:33.489 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:33.471+0000 7f1a8ad9d700 1 -- 172.21.15.42:0/2399638691 learned_addr learned my addr 172.21.15.42:0/2399638691 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:01:33.489 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:33.471+0000 7f1a8b59e700 1 --2- 172.21.15.42:0/2399638691 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1a8c09f830 0x7f1a8c14efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:01:33.489 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:33.471+0000 7f1a8a59c700 1 -- 172.21.15.42:0/2399638691 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1a8c09f830 msgr2=0x7f1a8c14efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:01:33.489 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:33.471+0000 7f1a8a59c700 1 --2- 172.21.15.42:0/2399638691 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1a8c09f830 0x7f1a8c14efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:01:33.489 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:33.471+0000 7f1a8a59c700 1 -- 172.21.15.42:0/2399638691 >> v1:172.21.15.42:6789/0 conn(0x7f1a8c0a7740 legacy=0x7f1a8c07e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:01:33.489 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:33.471+0000 7f1a89d9b700 1 -- 172.21.15.42:0/2399638691 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 242022655 0 0) 0x7f1a8c09fc60 con 0x7f1a8c0a7740 2019-12-05T06:01:33.490 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:33.475+0000 7f1a8a59c700 1 -- 172.21.15.42:0/2399638691 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a8c0a0710 con 0x7f1a8c08df80 2019-12-05T06:01:33.490 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:33.475+0000 7f1a8a59c700 1 --2- 172.21.15.42:0/2399638691 >> v2:172.21.15.42:3300/0 conn(0x7f1a8c08df80 0x7f1a8c09f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f1a7c008790 tx=0x7f1a7c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:33.490 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:33.475+0000 7f1a8a59c700 1 --2- 172.21.15.42:0/2399638691 >> v2:172.21.15.42:3300/0 conn(0x7f1a8c08df80 0x7f1a8c09f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f1a7c008790 tx=0x7f1a7c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:33.490 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:33.475+0000 7f1a8a59c700 1 --2- 172.21.15.42:0/2399638691 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1a8c08df80 0x7f1a8c09f2e0 secure :-1 s=READY pgs=1922 cs=0 l=1 rx=0x7f1a7c008790 tx=0x7f1a7c00fc70).ready entity=mon.1 client_cookie=2992bb511a45465b server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:01:33.490 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:33.475+0000 7f1a8a59c700 1 -- 172.21.15.42:0/2399638691 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1a8c08df80 msgr2=0x7f1a8c09f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2019-12-05T06:01:33.490 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:33.475+0000 7f1a8a59c700 1 -- 172.21.15.42:0/2399638691 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1a8c08df80 msgr2=0x7f1a8c09f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2019-12-05T06:01:33.490 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:33.475+0000 7f1a8a59c700 1 --2- 172.21.15.42:0/2399638691 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1a8c08df80 0x7f1a8c09f2e0 secure :-1 s=READY pgs=1922 cs=0 l=1 rx=0x7f1a7c008790 tx=0x7f1a7c00fc70).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-12-05T06:01:33.491 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:33.475+0000 7f1a8a59c700 1 --2- 172.21.15.42:0/2399638691 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1a8c08df80 0x7f1a8c09f2e0 secure :-1 s=READY pgs=1922 cs=0 l=1 rx=0x7f1a7c008790 tx=0x7f1a7c00fc70).stop 2019-12-05T06:01:33.491 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:33.475+0000 7f1a89d9b700 1 -- 172.21.15.42:0/2399638691 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1a8c08df80 msgr2=0x7f1a8c09f2e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2019-12-05T06:01:33.491 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:33.475+0000 7f1a89d9b700 1 --2- 172.21.15.42:0/2399638691 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1a8c08df80 0x7f1a8c09f2e0 unknown :-1 s=CLOSED pgs=1922 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:01:33.491 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:33.475+0000 7f1a89d9b700 1 --2- 172.21.15.42:0/2399638691 >> v2:172.21.15.40:3300/0 conn(0x7f1a7804aee0 0x7f1a7804d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:01:33.491 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:33.475+0000 7f1a89d9b700 1 -- 172.21.15.42:0/2399638691 --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a8c09fc60 con 0x7f1a78047060 2019-12-05T06:01:33.491 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:33.475+0000 7f1a89d9b700 1 -- 172.21.15.42:0/2399638691 --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a7804ebb0 con 0x7f1a7804aee0 2019-12-05T06:01:33.492 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:33.475+0000 7f1a8a59c700 1 --2- 172.21.15.42:0/2399638691 >> v2:172.21.15.40:3300/0 conn(0x7f1a7804aee0 0x7f1a7804d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:01:33.492 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:33.475+0000 7f1a89d9b700 1 -- 172.21.15.42:0/2399638691 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1955686865 0 0) 0x7f1a8c09fc60 con 0x7f1a78047060 2019-12-05T06:01:33.492 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:33.475+0000 7f1a89d9b700 1 -- 172.21.15.42:0/2399638691 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a7804dbf0 con 0x7f1a78047060 2019-12-05T06:01:33.492 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:33.475+0000 7f1a8a59c700 1 -- 172.21.15.42:0/2399638691 >> v1:172.21.15.40:6789/0 conn(0x7f1a78047060 legacy=0x7f1a78049530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:01:33.492 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:33.475+0000 7f1a8a59c700 1 -- 172.21.15.42:0/2399638691 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a7804ee60 con 0x7f1a7804aee0 2019-12-05T06:01:33.493 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:33.475+0000 7f1a8a59c700 1 --2- 172.21.15.42:0/2399638691 >> v2:172.21.15.40:3300/0 conn(0x7f1a7804aee0 0x7f1a7804d3f0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f1a7c008790 tx=0x7f1a7c00f500).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:33.493 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:33.475+0000 7f1a8a59c700 1 --2- 172.21.15.42:0/2399638691 >> v2:172.21.15.40:3300/0 conn(0x7f1a7804aee0 0x7f1a7804d3f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f1a7c008790 tx=0x7f1a7c00f500).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:33.493 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:33.475+0000 7f1a8a59c700 1 --2- 172.21.15.42:0/2399638691 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1a7804aee0 0x7f1a7804d3f0 secure :-1 s=READY pgs=1460 cs=0 l=1 rx=0x7f1a7c008790 tx=0x7f1a7c00f500).ready entity=mon.0 client_cookie=60beb6f60ec27097 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:01:33.493 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:33.475+0000 7f1a8a59c700 1 --2- 172.21.15.42:0/2399638691 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1a7804aee0 0x7f1a7804d3f0 secure :-1 s=THROTTLE_DONE pgs=1460 cs=0 l=1 rx=0x7f1a7c008790 tx=0x7f1a7c00f500).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:33.493 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:33.475+0000 7f1a89d9b700 1 -- 172.21.15.42:0/2399638691 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1a7c008ca0 con 0x7f1a7804aee0 2019-12-05T06:01:33.493 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:33.475+0000 7f1a8a59c700 1 --2- 172.21.15.42:0/2399638691 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1a7804aee0 0x7f1a7804d3f0 secure :-1 s=THROTTLE_DONE pgs=1460 cs=0 l=1 rx=0x7f1a7c008790 tx=0x7f1a7c00f500).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:33.493 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:33.475+0000 7f1a8a59c700 1 --2- 172.21.15.42:0/2399638691 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1a7804aee0 0x7f1a7804d3f0 secure :-1 s=THROTTLE_DONE pgs=1460 cs=0 l=1 rx=0x7f1a7c008790 tx=0x7f1a7c00f500).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:33.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:33.475+0000 7f1a89d9b700 1 -- 172.21.15.42:0/2399638691 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a7c0088c0 con 0x7f1a7804aee0 2019-12-05T06:01:33.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:33.475+0000 7f1a89d9b700 1 -- 172.21.15.42:0/2399638691 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1a7c011a50 con 0x7f1a7804aee0 2019-12-05T06:01:33.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:33.475+0000 7f1a90b9b700 1 -- 172.21.15.42:0/2399638691 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1a7804aee0 msgr2=0x7f1a7804d3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:01:33.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:33.475+0000 7f1a90b9b700 1 --2- 172.21.15.42:0/2399638691 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1a7804aee0 0x7f1a7804d3f0 secure :-1 s=READY pgs=1460 cs=0 l=1 rx=0x7f1a7c008790 tx=0x7f1a7c00f500).stop 2019-12-05T06:01:33.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:33.475+0000 7f1a8b59e700 1 -- 172.21.15.42:0/2399638691 reap_dead start 2019-12-05T06:01:33.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:33.475+0000 7f1a90b9b700 1 -- 172.21.15.42:0/2399638691 shutdown_connections 2019-12-05T06:01:33.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:33.475+0000 7f1a90b9b700 1 --2- 172.21.15.42:0/2399638691 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1a7804aee0 0x7f1a7804d3f0 unknown :-1 s=CLOSED pgs=1460 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:01:33.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:33.475+0000 7f1a90b9b700 1 --2- 172.21.15.42:0/2399638691 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1a8c08df80 0x7f1a8c09f2e0 unknown :-1 s=CLOSED pgs=1922 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:01:33.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:33.475+0000 7f1a90b9b700 1 -- 172.21.15.42:0/2399638691 shutdown_connections 2019-12-05T06:01:33.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:33.475+0000 7f1a90b9b700 1 -- 172.21.15.42:0/2399638691 wait complete. 2019-12-05T06:01:33.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:33.475+0000 7f1a90b9b700 1 -- 172.21.15.42:0/2399638691 >> 172.21.15.42:0/2399638691 conn(0x7f1a8c0aabe0 msgr2=0x7f1a8c0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:01:33.498 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:33.479+0000 7f1a90b9b700 1 Processor -- start 2019-12-05T06:01:33.498 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:33.479+0000 7f1a90b9b700 1 -- start start 2019-12-05T06:01:33.499 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:33.479+0000 7f1a90b9b700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f1a8c0a4250 0x7f1a8c127cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:01:33.499 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:33.479+0000 7f1a90b9b700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1a8c0656b0 0x7f1a8c063d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:01:33.499 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:33.479+0000 7f1a90b9b700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a8c128200 con 0x7f1a8c0642a0 2019-12-05T06:01:33.499 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:33.479+0000 7f1a90b9b700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a8c0a23e0 con 0x7f1a8c0a4250 2019-12-05T06:01:33.499 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:33.479+0000 7f1a90b9b700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a8c0a2680 con 0x7f1a8c0656b0 2019-12-05T06:01:33.499 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:33.479+0000 7f1a8ad9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f1a8c0a4250 0x7f1a8c127cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:01:33.499 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:33.479+0000 7f1a8b59e700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f1a8c0642a0 0x7f1a8c08eb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:46596/0 (socket says 172.21.15.42:46596) 2019-12-05T06:01:33.500 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:33.479+0000 7f1a8b59e700 1 -- 172.21.15.42:0/4242322862 learned_addr learned my addr 172.21.15.42:0/4242322862 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:01:33.500 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:33.483+0000 7f1a8a59c700 1 --2- 172.21.15.42:0/4242322862 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1a8c0656b0 0x7f1a8c063d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:01:33.500 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:33.483+0000 7f1a8ad9d700 1 -- 172.21.15.42:0/4242322862 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1a8c0656b0 msgr2=0x7f1a8c063d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:01:33.500 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:33.483+0000 7f1a8ad9d700 1 --2- 172.21.15.42:0/4242322862 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1a8c0656b0 0x7f1a8c063d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:01:33.500 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:33.483+0000 7f1a8ad9d700 1 -- 172.21.15.42:0/4242322862 >> v1:172.21.15.42:6789/0 conn(0x7f1a8c0642a0 legacy=0x7f1a8c08eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:01:33.500 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:33.483+0000 7f1a8ad9d700 1 -- 172.21.15.42:0/4242322862 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a8c0a2bc0 con 0x7f1a8c0a4250 2019-12-05T06:01:33.500 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:33.483+0000 7f1a837fe700 1 -- 172.21.15.42:0/4242322862 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4077128837 0 0) 0x7f1a8c128200 con 0x7f1a8c0642a0 2019-12-05T06:01:33.501 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:33.483+0000 7f1a8ad9d700 1 --2- 172.21.15.42:0/4242322862 >> v2:172.21.15.42:3300/0 conn(0x7f1a8c0a4250 0x7f1a8c127cb0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f1a8c128430 tx=0x7f1a7400c370).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:33.501 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:33.483+0000 7f1a8ad9d700 1 --2- 172.21.15.42:0/4242322862 >> v2:172.21.15.42:3300/0 conn(0x7f1a8c0a4250 0x7f1a8c127cb0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f1a8c128430 tx=0x7f1a7400c370).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:33.501 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:33.483+0000 7f1a8ad9d700 1 --2- 172.21.15.42:0/4242322862 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1a8c0a4250 0x7f1a8c127cb0 secure :-1 s=READY pgs=1924 cs=0 l=1 rx=0x7f1a8c128430 tx=0x7f1a7400c370).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:01:33.501 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:33.483+0000 7f1a8ad9d700 1 --2- 172.21.15.42:0/4242322862 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1a8c0a4250 0x7f1a8c127cb0 secure :-1 s=THROTTLE_DONE pgs=1924 cs=0 l=1 rx=0x7f1a8c128430 tx=0x7f1a7400c370).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:33.501 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:33.483+0000 7f1a837fe700 1 -- 172.21.15.42:0/4242322862 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1a74011600 con 0x7f1a8c0a4250 2019-12-05T06:01:33.501 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:33.483+0000 7f1a8ad9d700 1 --2- 172.21.15.42:0/4242322862 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1a8c0a4250 0x7f1a8c127cb0 secure :-1 s=THROTTLE_DONE pgs=1924 cs=0 l=1 rx=0x7f1a8c128430 tx=0x7f1a7400c370).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:33.502 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:33.483+0000 7f1a837fe700 1 -- 172.21.15.42:0/4242322862 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a74011220 con 0x7f1a8c0a4250 2019-12-05T06:01:33.502 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:33.483+0000 7f1a8ad9d700 1 --2- 172.21.15.42:0/4242322862 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1a8c0a4250 0x7f1a8c127cb0 secure :-1 s=THROTTLE_DONE pgs=1924 cs=0 l=1 rx=0x7f1a8c128430 tx=0x7f1a7400c370).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:33.502 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:33.483+0000 7f1a837fe700 1 -- 172.21.15.42:0/4242322862 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1a74011be0 con 0x7f1a8c0a4250 2019-12-05T06:01:33.502 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:33.483+0000 7f1a90b9b700 1 -- 172.21.15.42:0/4242322862 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a8c08f370 con 0x7f1a8c0a4250 2019-12-05T06:01:33.503 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:33.483+0000 7f1a90b9b700 1 -- 172.21.15.42:0/4242322862 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a8c08f870 con 0x7f1a8c0a4250 2019-12-05T06:01:33.503 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:33.483+0000 7f1a8ad9d700 1 --2- 172.21.15.42:0/4242322862 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1a8c0a4250 0x7f1a8c127cb0 secure :-1 s=THROTTLE_DONE pgs=1924 cs=0 l=1 rx=0x7f1a8c128430 tx=0x7f1a7400c370).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:33.503 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:33.483+0000 7f1a837fe700 1 -- 172.21.15.42:0/4242322862 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f1a74006460 con 0x7f1a8c0a4250 2019-12-05T06:01:33.503 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:33.483+0000 7f1a8ad9d700 1 --2- 172.21.15.42:0/4242322862 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1a8c0a4250 0x7f1a8c127cb0 secure :-1 s=THROTTLE_DONE pgs=1924 cs=0 l=1 rx=0x7f1a8c128430 tx=0x7f1a7400c370).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:33.503 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:33.483+0000 7f1a837fe700 1 --2- 172.21.15.42:0/4242322862 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f1a7003abc0 0x7f1a7003d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:01:33.503 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:33.483+0000 7f1a837fe700 1 -- 172.21.15.42:0/4242322862 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9471+0+0 (secure 0 0 0) 0x7f1a7404af10 con 0x7f1a8c0a4250 2019-12-05T06:01:33.503 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:33.487+0000 7f1a8a59c700 1 --2- 172.21.15.42:0/4242322862 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f1a7003abc0 0x7f1a7003d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:01:33.504 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:33.487+0000 7f1a8a59c700 1 --2- 172.21.15.42:0/4242322862 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f1a7003abc0 0x7f1a7003d0d0 crc :-1 s=READY pgs=815 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:01:33.504 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:33.487+0000 7f1a90b9b700 1 -- 172.21.15.42:0/4242322862 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a8c05d110 con 0x7f1a8c0a4250 2019-12-05T06:01:33.504 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:33.495+0000 7f1a8ad9d700 1 --2- 172.21.15.42:0/4242322862 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1a8c0a4250 0x7f1a8c127cb0 secure :-1 s=THROTTLE_DONE pgs=1924 cs=0 l=1 rx=0x7f1a8c128430 tx=0x7f1a7400c370).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:33.505 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:33.495+0000 7f1a837fe700 1 -- 172.21.15.42:0/4242322862 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f1a74031f10 con 0x7f1a8c0a4250 2019-12-05T06:01:33.673 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:33.667+0000 7f1a90b9b700 1 -- 172.21.15.42:0/4242322862 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1a8c05b8c0 con 0x7f1a8c0a4250 2019-12-05T06:01:33.673 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:33.667+0000 7f1a8ad9d700 1 --2- 172.21.15.42:0/4242322862 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1a8c0a4250 0x7f1a8c127cb0 secure :-1 s=THROTTLE_DONE pgs=1924 cs=0 l=1 rx=0x7f1a8c128430 tx=0x7f1a7400c370).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:33.674 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:33.667+0000 7f1a837fe700 1 -- 172.21.15.42:0/4242322862 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 233 v233) v1 ==== 95+0+2109 (secure 0 0 0) 0x7f1a74032170 con 0x7f1a8c0a4250 2019-12-05T06:01:33.674 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 233 2019-12-05T06:01:33.674 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:01:33.674 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":233,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":9292,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6835/613535662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":613535662},{"type":"v1","addr":"172.21.15.42:6835","nonce":613535662}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":233}],"filesystems":[{"mdsmap":{"epoch":232,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:01:04.420072+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":89,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6653},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6653":{"gid":6653,"name":"b","rank":0,"incarnation":229,"state":"up:active","state_seq":13,"addr":"172.21.15.40:6833/2780168808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2780168808},{"type":"v1","addr":"172.21.15.40:6833","nonce":2780168808}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:01:33.676 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:33.667+0000 7f1a90b9b700 1 -- 172.21.15.42:0/4242322862 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f1a7003abc0 msgr2=0x7f1a7003d0d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:01:33.676 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:33.667+0000 7f1a90b9b700 1 --2- 172.21.15.42:0/4242322862 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f1a7003abc0 0x7f1a7003d0d0 crc :-1 s=READY pgs=815 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:01:33.676 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:33.667+0000 7f1a90b9b700 1 -- 172.21.15.42:0/4242322862 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1a8c0a4250 msgr2=0x7f1a8c127cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:01:33.676 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:33.667+0000 7f1a90b9b700 1 --2- 172.21.15.42:0/4242322862 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1a8c0a4250 0x7f1a8c127cb0 secure :-1 s=READY pgs=1924 cs=0 l=1 rx=0x7f1a8c128430 tx=0x7f1a7400c370).stop 2019-12-05T06:01:33.677 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:33.667+0000 7f1a90b9b700 1 -- 172.21.15.42:0/4242322862 shutdown_connections 2019-12-05T06:01:33.677 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:33.667+0000 7f1a90b9b700 1 --2- 172.21.15.42:0/4242322862 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f1a7003abc0 0x7f1a7003d0d0 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:01:33.677 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:33.667+0000 7f1a90b9b700 1 --2- 172.21.15.42:0/4242322862 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1a8c0a4250 0x7f1a8c127cb0 unknown :-1 s=CLOSED pgs=1924 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:01:33.677 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:33.667+0000 7f1a90b9b700 1 --2- 172.21.15.42:0/4242322862 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1a8c0656b0 0x7f1a8c063d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:01:33.677 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:33.671+0000 7f1a90b9b700 1 -- 172.21.15.42:0/4242322862 shutdown_connections 2019-12-05T06:01:33.677 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:33.671+0000 7f1a90b9b700 1 -- 172.21.15.42:0/4242322862 wait complete. 2019-12-05T06:01:33.677 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:33.671+0000 7f1a90b9b700 1 -- 172.21.15.42:0/4242322862 >> 172.21.15.42:0/4242322862 conn(0x7f1a8c0aabe0 msgr2=0x7f1a8c0a9640 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:01:33.685 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:01:35.686 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:01:35.795 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:35.783+0000 7f18185ff700 1 Processor -- start 2019-12-05T06:01:35.795 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:35.783+0000 7f18185ff700 1 -- start start 2019-12-05T06:01:35.795 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:35.783+0000 7f18185ff700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f181008df80 0x7f181009f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:01:35.795 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:35.783+0000 7f18185ff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f181009f830 0x7f181014efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:01:35.796 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:35.783+0000 7f18185ff700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f181009fc60 con 0x7f18100a7740 2019-12-05T06:01:35.796 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:35.783+0000 7f18185ff700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18100a0b00 con 0x7f181008df80 2019-12-05T06:01:35.796 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:35.783+0000 7f18185ff700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f181008e5b0 con 0x7f181009f830 2019-12-05T06:01:35.796 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:35.783+0000 7f181739d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f181009f830 0x7f181014efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:01:35.796 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:35.783+0000 7f181639b700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f181008df80 0x7f181009f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:01:35.797 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:35.783+0000 7f181739d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f181009f830 0x7f181014efe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:46570/0 (socket says 172.21.15.42:46570) 2019-12-05T06:01:35.797 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:35.783+0000 7f181739d700 1 -- 172.21.15.42:0/3757957374 learned_addr learned my addr 172.21.15.42:0/3757957374 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:01:35.797 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:35.783+0000 7f181739d700 1 -- 172.21.15.42:0/3757957374 >> v2:172.21.15.40:3300/0 conn(0x7f181008df80 msgr2=0x7f181009f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:01:35.797 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:35.783+0000 7f181739d700 1 --2- 172.21.15.42:0/3757957374 >> v2:172.21.15.40:3300/0 conn(0x7f181008df80 0x7f181009f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:01:35.797 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:35.787+0000 7f181739d700 1 -- 172.21.15.42:0/3757957374 >> v1:172.21.15.40:6789/0 conn(0x7f18100a7740 legacy=0x7f181007e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:01:35.797 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:35.787+0000 7f181739d700 1 -- 172.21.15.42:0/3757957374 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18100a0710 con 0x7f181009f830 2019-12-05T06:01:35.798 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:35.787+0000 7f1815b9a700 1 -- 172.21.15.42:0/3757957374 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 199960389 0 0) 0x7f181009fc60 con 0x7f18100a7740 2019-12-05T06:01:35.798 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:35.787+0000 7f181739d700 1 --2- 172.21.15.42:0/3757957374 >> v2:172.21.15.42:3300/0 conn(0x7f181009f830 0x7f181014efe0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f1808002f30 tx=0x7f180800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:35.798 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:35.787+0000 7f181739d700 1 --2- 172.21.15.42:0/3757957374 >> v2:172.21.15.42:3300/0 conn(0x7f181009f830 0x7f181014efe0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f1808002f30 tx=0x7f180800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:35.798 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:35.787+0000 7f181739d700 1 --2- 172.21.15.42:0/3757957374 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f181009f830 0x7f181014efe0 secure :-1 s=READY pgs=1925 cs=0 l=1 rx=0x7f1808002f30 tx=0x7f180800fd30).ready entity=mon.1 client_cookie=6e213cd4d7be4dc8 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:01:35.798 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:35.787+0000 7f181739d700 1 --2- 172.21.15.42:0/3757957374 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f181009f830 0x7f181014efe0 secure :-1 s=THROTTLE_DONE pgs=1925 cs=0 l=1 rx=0x7f1808002f30 tx=0x7f180800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:35.798 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:35.787+0000 7f1815b9a700 1 -- 172.21.15.42:0/3757957374 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1808008c90 con 0x7f181009f830 2019-12-05T06:01:35.799 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:35.787+0000 7f181739d700 1 --2- 172.21.15.42:0/3757957374 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f181009f830 0x7f181014efe0 secure :-1 s=THROTTLE_DONE pgs=1925 cs=0 l=1 rx=0x7f1808002f30 tx=0x7f180800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:35.799 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:35.787+0000 7f181739d700 1 --2- 172.21.15.42:0/3757957374 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f181009f830 0x7f181014efe0 secure :-1 s=THROTTLE_DONE pgs=1925 cs=0 l=1 rx=0x7f1808002f30 tx=0x7f180800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:35.799 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:35.787+0000 7f1815b9a700 1 -- 172.21.15.42:0/3757957374 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18080088b0 con 0x7f181009f830 2019-12-05T06:01:35.799 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:35.787+0000 7f1815b9a700 1 -- 172.21.15.42:0/3757957374 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1808011ad0 con 0x7f181009f830 2019-12-05T06:01:35.799 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:35.787+0000 7f18185ff700 1 -- 172.21.15.42:0/3757957374 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f181009f830 msgr2=0x7f181014efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:01:35.799 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:35.787+0000 7f18185ff700 1 --2- 172.21.15.42:0/3757957374 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f181009f830 0x7f181014efe0 secure :-1 s=READY pgs=1925 cs=0 l=1 rx=0x7f1808002f30 tx=0x7f180800fd30).stop 2019-12-05T06:01:35.800 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:35.787+0000 7f18185ff700 1 -- 172.21.15.42:0/3757957374 shutdown_connections 2019-12-05T06:01:35.800 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:35.787+0000 7f18185ff700 1 --2- 172.21.15.42:0/3757957374 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f181009f830 0x7f181014efe0 unknown :-1 s=CLOSED pgs=1925 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:01:35.800 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:35.787+0000 7f18185ff700 1 --2- 172.21.15.42:0/3757957374 >> v2:172.21.15.40:3300/0 conn(0x7f181008df80 0x7f181009f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:01:35.800 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:35.787+0000 7f18185ff700 1 -- 172.21.15.42:0/3757957374 shutdown_connections 2019-12-05T06:01:35.800 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:35.787+0000 7f18185ff700 1 -- 172.21.15.42:0/3757957374 wait complete. 2019-12-05T06:01:35.800 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:35.787+0000 7f18185ff700 1 -- 172.21.15.42:0/3757957374 >> 172.21.15.42:0/3757957374 conn(0x7f18100aabe0 msgr2=0x7f18100a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:01:35.801 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:35.791+0000 7f18185ff700 1 Processor -- start 2019-12-05T06:01:35.801 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:35.791+0000 7f18185ff700 1 -- start start 2019-12-05T06:01:35.804 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:35.791+0000 7f18185ff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f181011b830 0x7f181008eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:01:35.804 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:35.791+0000 7f18185ff700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f181012af70 0x7f181012b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:01:35.804 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:35.791+0000 7f18185ff700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f181008f0c0 con 0x7f181012dff0 2019-12-05T06:01:35.804 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:35.791+0000 7f18185ff700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f181011c040 con 0x7f181011b830 2019-12-05T06:01:35.804 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:35.791+0000 7f18185ff700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f181012aa30 con 0x7f181012af70 2019-12-05T06:01:35.804 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:35.791+0000 7f1816b9c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f181011b830 0x7f181008eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:01:35.805 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:35.791+0000 7f181739d700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f181012dff0 0x7f181012e420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:56054/0 (socket says 172.21.15.42:56054) 2019-12-05T06:01:35.805 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:35.791+0000 7f181639b700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f181012af70 0x7f181012b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:01:35.805 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:35.791+0000 7f181739d700 1 -- 172.21.15.42:0/1713684387 learned_addr learned my addr 172.21.15.42:0/1713684387 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:01:35.805 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:35.791+0000 7f18037fe700 1 -- 172.21.15.42:0/1713684387 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 91634348 0 0) 0x7f181008f0c0 con 0x7f181012dff0 2019-12-05T06:01:35.805 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:35.791+0000 7f18037fe700 1 -- 172.21.15.42:0/1713684387 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17f80030b0 con 0x7f181012dff0 2019-12-05T06:01:35.805 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:35.791+0000 7f1816b9c700 1 -- 172.21.15.42:0/1713684387 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f181012af70 msgr2=0x7f181012b3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:01:35.806 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:35.791+0000 7f1816b9c700 1 --2- 172.21.15.42:0/1713684387 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f181012af70 0x7f181012b3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:01:35.806 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:35.791+0000 7f1816b9c700 1 -- 172.21.15.42:0/1713684387 >> v1:172.21.15.40:6789/0 conn(0x7f181012dff0 legacy=0x7f181012e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:01:35.806 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:35.791+0000 7f1816b9c700 1 -- 172.21.15.42:0/1713684387 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f181012b8f0 con 0x7f181011b830 2019-12-05T06:01:35.806 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:35.791+0000 7f181639b700 1 --2- 172.21.15.42:0/1713684387 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f181012af70 0x7f181012b3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:01:35.806 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:35.791+0000 7f1816b9c700 1 --2- 172.21.15.42:0/1713684387 >> v2:172.21.15.42:3300/0 conn(0x7f181011b830 0x7f181008eb70 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f180c000b80 tx=0x7f180c010810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:35.806 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:35.791+0000 7f1816b9c700 1 --2- 172.21.15.42:0/1713684387 >> v2:172.21.15.42:3300/0 conn(0x7f181011b830 0x7f181008eb70 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f180c000b80 tx=0x7f180c010810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:35.806 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:35.791+0000 7f1816b9c700 1 --2- 172.21.15.42:0/1713684387 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f181011b830 0x7f181008eb70 secure :-1 s=READY pgs=1926 cs=0 l=1 rx=0x7f180c000b80 tx=0x7f180c010810).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:01:35.807 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:35.791+0000 7f1816b9c700 1 --2- 172.21.15.42:0/1713684387 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f181011b830 0x7f181008eb70 secure :-1 s=THROTTLE_DONE pgs=1926 cs=0 l=1 rx=0x7f180c000b80 tx=0x7f180c010810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:35.807 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:35.791+0000 7f18037fe700 1 -- 172.21.15.42:0/1713684387 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f180c006a40 con 0x7f181011b830 2019-12-05T06:01:35.807 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:35.791+0000 7f1816b9c700 1 --2- 172.21.15.42:0/1713684387 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f181011b830 0x7f181008eb70 secure :-1 s=THROTTLE_DONE pgs=1926 cs=0 l=1 rx=0x7f180c000b80 tx=0x7f180c010810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:35.807 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:35.791+0000 7f18037fe700 1 -- 172.21.15.42:0/1713684387 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f180c006660 con 0x7f181011b830 2019-12-05T06:01:35.807 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:35.791+0000 7f1816b9c700 1 --2- 172.21.15.42:0/1713684387 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f181011b830 0x7f181008eb70 secure :-1 s=THROTTLE_DONE pgs=1926 cs=0 l=1 rx=0x7f180c000b80 tx=0x7f180c010810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:35.807 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:35.791+0000 7f18185ff700 1 -- 172.21.15.42:0/1713684387 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f181011bc60 con 0x7f181011b830 2019-12-05T06:01:35.808 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:35.791+0000 7f18037fe700 1 -- 172.21.15.42:0/1713684387 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f180c011050 con 0x7f181011b830 2019-12-05T06:01:35.808 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:35.791+0000 7f18185ff700 1 -- 172.21.15.42:0/1713684387 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f181013edd0 con 0x7f181011b830 2019-12-05T06:01:35.808 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:35.795+0000 7f1816b9c700 1 --2- 172.21.15.42:0/1713684387 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f181011b830 0x7f181008eb70 secure :-1 s=THROTTLE_DONE pgs=1926 cs=0 l=1 rx=0x7f180c000b80 tx=0x7f180c010810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:35.808 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:35.795+0000 7f18037fe700 1 -- 172.21.15.42:0/1713684387 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f180c011370 con 0x7f181011b830 2019-12-05T06:01:35.808 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:35.795+0000 7f1816b9c700 1 --2- 172.21.15.42:0/1713684387 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f181011b830 0x7f181008eb70 secure :-1 s=THROTTLE_DONE pgs=1926 cs=0 l=1 rx=0x7f180c000b80 tx=0x7f180c010810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:35.808 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:35.795+0000 7f18037fe700 1 --2- 172.21.15.42:0/1713684387 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f17f803b220 0x7f17f803d730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:01:35.809 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:35.795+0000 7f18037fe700 1 -- 172.21.15.42:0/1713684387 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9471+0+0 (secure 0 0 0) 0x7f180c04b140 con 0x7f181011b830 2019-12-05T06:01:35.809 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:35.795+0000 7f181639b700 1 --2- 172.21.15.42:0/1713684387 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f17f803b220 0x7f17f803d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:01:35.809 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:35.795+0000 7f181639b700 1 --2- 172.21.15.42:0/1713684387 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f17f803b220 0x7f17f803d730 crc :-1 s=READY pgs=816 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:01:35.809 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:35.799+0000 7f18185ff700 1 -- 172.21.15.42:0/1713684387 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f181005d110 con 0x7f181011b830 2019-12-05T06:01:35.813 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:35.807+0000 7f1816b9c700 1 --2- 172.21.15.42:0/1713684387 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f181011b830 0x7f181008eb70 secure :-1 s=THROTTLE_DONE pgs=1926 cs=0 l=1 rx=0x7f180c000b80 tx=0x7f180c010810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:35.813 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:35.807+0000 7f18037fe700 1 -- 172.21.15.42:0/1713684387 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f180c0320e0 con 0x7f181011b830 2019-12-05T06:01:35.983 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:35.979+0000 7f18185ff700 1 -- 172.21.15.42:0/1713684387 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f181005b8c0 con 0x7f181011b830 2019-12-05T06:01:35.984 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:35.979+0000 7f1816b9c700 1 --2- 172.21.15.42:0/1713684387 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f181011b830 0x7f181008eb70 secure :-1 s=THROTTLE_DONE pgs=1926 cs=0 l=1 rx=0x7f180c000b80 tx=0x7f180c010810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:35.984 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:35.979+0000 7f18037fe700 1 -- 172.21.15.42:0/1713684387 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 233 v233) v1 ==== 95+0+2109 (secure 0 0 0) 0x7f180c032340 con 0x7f181011b830 2019-12-05T06:01:35.986 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:01:35.986 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":233,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":9292,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6835/613535662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":613535662},{"type":"v1","addr":"172.21.15.42:6835","nonce":613535662}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":233}],"filesystems":[{"mdsmap":{"epoch":232,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:01:04.420072+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":89,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6653},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6653":{"gid":6653,"name":"b","rank":0,"incarnation":229,"state":"up:active","state_seq":13,"addr":"172.21.15.40:6833/2780168808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2780168808},{"type":"v1","addr":"172.21.15.40:6833","nonce":2780168808}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:01:35.986 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 233 2019-12-05T06:01:35.989 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:35.983+0000 7f18185ff700 1 -- 172.21.15.42:0/1713684387 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f17f803b220 msgr2=0x7f17f803d730 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:01:35.989 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:35.983+0000 7f18185ff700 1 --2- 172.21.15.42:0/1713684387 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f17f803b220 0x7f17f803d730 crc :-1 s=READY pgs=816 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:01:35.989 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:35.983+0000 7f18185ff700 1 -- 172.21.15.42:0/1713684387 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f181011b830 msgr2=0x7f181008eb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:01:35.989 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:35.983+0000 7f18185ff700 1 --2- 172.21.15.42:0/1713684387 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f181011b830 0x7f181008eb70 secure :-1 s=READY pgs=1926 cs=0 l=1 rx=0x7f180c000b80 tx=0x7f180c010810).stop 2019-12-05T06:01:35.990 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:35.983+0000 7f18185ff700 1 -- 172.21.15.42:0/1713684387 shutdown_connections 2019-12-05T06:01:35.990 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:35.983+0000 7f18185ff700 1 --2- 172.21.15.42:0/1713684387 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f17f803b220 0x7f17f803d730 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:01:35.990 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:35.983+0000 7f18185ff700 1 --2- 172.21.15.42:0/1713684387 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f181011b830 0x7f181008eb70 unknown :-1 s=CLOSED pgs=1926 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:01:35.990 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:35.983+0000 7f18185ff700 1 --2- 172.21.15.42:0/1713684387 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f181012af70 0x7f181012b3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:01:35.990 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:35.983+0000 7f18185ff700 1 -- 172.21.15.42:0/1713684387 shutdown_connections 2019-12-05T06:01:35.990 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:35.983+0000 7f18185ff700 1 -- 172.21.15.42:0/1713684387 wait complete. 2019-12-05T06:01:35.990 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:35.983+0000 7f18185ff700 1 -- 172.21.15.42:0/1713684387 >> 172.21.15.42:0/1713684387 conn(0x7f18100a9870 msgr2=0x7f181011d7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:01:35.999 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:01:38.013 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:01:38.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:38.139+0000 7f2947743700 1 Processor -- start 2019-12-05T06:01:38.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:38.139+0000 7f2947743700 1 -- start start 2019-12-05T06:01:38.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:38.139+0000 7f2947743700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f29400a7740 0x7f294008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:01:38.185 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:38.139+0000 7f2947743700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f294007e720 0x7f294007eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:01:38.186 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:38.139+0000 7f2947743700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29400a0710 con 0x7f294009f2e0 2019-12-05T06:01:38.186 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:38.139+0000 7f2947743700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29400a0b00 con 0x7f294007e720 2019-12-05T06:01:38.186 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:38.139+0000 7f2947743700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f294008e5b0 con 0x7f29400a7740 2019-12-05T06:01:38.187 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:38.143+0000 7f29464e1700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f294009f2e0 0x7f294009f710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:46624/0 (socket says 172.21.15.42:46624) 2019-12-05T06:01:38.187 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:38.143+0000 7f2945ce0700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f29400a7740 0x7f294008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:01:38.187 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:38.143+0000 7f29454df700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f294007e720 0x7f294007eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:01:38.187 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:38.143+0000 7f29464e1700 1 -- 172.21.15.42:0/2091309728 learned_addr learned my addr 172.21.15.42:0/2091309728 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:01:38.187 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:38.143+0000 7f2944cde700 1 -- 172.21.15.42:0/2091309728 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1951681850 0 0) 0x7f29400a0710 con 0x7f294009f2e0 2019-12-05T06:01:38.187 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:38.143+0000 7f2944cde700 1 -- 172.21.15.42:0/2091309728 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f292c0030b0 con 0x7f294009f2e0 2019-12-05T06:01:38.188 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:38.143+0000 7f2945ce0700 1 -- 172.21.15.42:0/2091309728 >> v2:172.21.15.40:3300/0 conn(0x7f294007e720 msgr2=0x7f294007eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:01:38.188 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:38.143+0000 7f2945ce0700 1 --2- 172.21.15.42:0/2091309728 >> v2:172.21.15.40:3300/0 conn(0x7f294007e720 0x7f294007eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:01:38.188 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:38.143+0000 7f2945ce0700 1 -- 172.21.15.42:0/2091309728 >> v1:172.21.15.42:6789/0 conn(0x7f294009f2e0 legacy=0x7f294009f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:01:38.188 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:38.143+0000 7f2945ce0700 1 -- 172.21.15.42:0/2091309728 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f294007f0a0 con 0x7f29400a7740 2019-12-05T06:01:38.188 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:38.143+0000 7f29454df700 1 --2- 172.21.15.42:0/2091309728 >> v2:172.21.15.40:3300/0 conn(0x7f294007e720 0x7f294007eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:01:38.188 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:38.143+0000 7f2945ce0700 1 --2- 172.21.15.42:0/2091309728 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f29400a7740 0x7f294008df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f293c008790 tx=0x7f293c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:38.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:38.143+0000 7f2945ce0700 1 --2- 172.21.15.42:0/2091309728 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f29400a7740 0x7f294008df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f293c008790 tx=0x7f293c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:38.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:38.143+0000 7f2945ce0700 1 --2- 172.21.15.42:0/2091309728 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f29400a7740 0x7f294008df80 secure :-1 s=READY pgs=426 cs=0 l=1 rx=0x7f293c008790 tx=0x7f293c00fc70).ready entity=mon.2 client_cookie=ed513aee2aa6985c server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:01:38.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:38.143+0000 7f2945ce0700 1 --2- 172.21.15.42:0/2091309728 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f29400a7740 0x7f294008df80 secure :-1 s=THROTTLE_DONE pgs=426 cs=0 l=1 rx=0x7f293c008790 tx=0x7f293c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:38.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:38.143+0000 7f2944cde700 1 -- 172.21.15.42:0/2091309728 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f293c0112f0 con 0x7f29400a7740 2019-12-05T06:01:38.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:38.143+0000 7f2945ce0700 1 --2- 172.21.15.42:0/2091309728 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f29400a7740 0x7f294008df80 secure :-1 s=THROTTLE_DONE pgs=426 cs=0 l=1 rx=0x7f293c008790 tx=0x7f293c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:38.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:38.143+0000 7f2945ce0700 1 --2- 172.21.15.42:0/2091309728 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f29400a7740 0x7f294008df80 secure :-1 s=THROTTLE_DONE pgs=426 cs=0 l=1 rx=0x7f293c008790 tx=0x7f293c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:38.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:38.143+0000 7f2944cde700 1 -- 172.21.15.42:0/2091309728 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f293c008ad0 con 0x7f29400a7740 2019-12-05T06:01:38.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:38.143+0000 7f2944cde700 1 -- 172.21.15.42:0/2091309728 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f293c0118d0 con 0x7f29400a7740 2019-12-05T06:01:38.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:38.143+0000 7f2947743700 1 -- 172.21.15.42:0/2091309728 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f29400a7740 msgr2=0x7f294008df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:01:38.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:38.143+0000 7f2947743700 1 --2- 172.21.15.42:0/2091309728 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f29400a7740 0x7f294008df80 secure :-1 s=READY pgs=426 cs=0 l=1 rx=0x7f293c008790 tx=0x7f293c00fc70).stop 2019-12-05T06:01:38.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:38.143+0000 7f2947743700 1 -- 172.21.15.42:0/2091309728 shutdown_connections 2019-12-05T06:01:38.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:38.143+0000 7f2947743700 1 --2- 172.21.15.42:0/2091309728 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f29400a7740 0x7f294008df80 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:01:38.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:38.143+0000 7f2947743700 1 --2- 172.21.15.42:0/2091309728 >> v2:172.21.15.40:3300/0 conn(0x7f294007e720 0x7f294007eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:01:38.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:38.143+0000 7f2947743700 1 -- 172.21.15.42:0/2091309728 shutdown_connections 2019-12-05T06:01:38.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:38.143+0000 7f2947743700 1 -- 172.21.15.42:0/2091309728 wait complete. 2019-12-05T06:01:38.193 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:38.143+0000 7f2947743700 1 -- 172.21.15.42:0/2091309728 >> 172.21.15.42:0/2091309728 conn(0x7f29400aabe0 msgr2=0x7f29400a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:01:38.193 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:38.147+0000 7f2947743700 1 Processor -- start 2019-12-05T06:01:38.193 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:38.147+0000 7f2947743700 1 -- start start 2019-12-05T06:01:38.193 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:38.147+0000 7f2947743700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f29401309e0 0x7f294011a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:01:38.194 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:38.147+0000 7f2947743700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2940135620 0x7f2940137a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:01:38.194 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:38.147+0000 7f2947743700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f294011ad10 con 0x7f2940117c90 2019-12-05T06:01:38.194 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:38.147+0000 7f2947743700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2940118240 con 0x7f2940135620 2019-12-05T06:01:38.194 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:38.147+0000 7f2947743700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29401184e0 con 0x7f29401309e0 2019-12-05T06:01:38.195 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:38.151+0000 7f29464e1700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2940135620 0x7f2940137a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:01:38.195 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:38.151+0000 7f29464e1700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2940135620 0x7f2940137a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:46592/0 (socket says 172.21.15.42:46592) 2019-12-05T06:01:38.195 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:38.151+0000 7f29464e1700 1 -- 172.21.15.42:0/68896999 learned_addr learned my addr 172.21.15.42:0/68896999 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:01:38.195 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:38.151+0000 7f29454df700 1 --2- 172.21.15.42:0/68896999 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f29401309e0 0x7f294011a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:01:38.196 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:38.151+0000 7f29464e1700 1 -- 172.21.15.42:0/68896999 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f29401309e0 msgr2=0x7f294011a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:01:38.196 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:38.151+0000 7f29367fc700 1 -- 172.21.15.42:0/68896999 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3126507571 0 0) 0x7f294011ad10 con 0x7f2940117c90 2019-12-05T06:01:38.196 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:38.151+0000 7f29464e1700 1 --2- 172.21.15.42:0/68896999 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f29401309e0 0x7f294011a7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:01:38.197 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:38.151+0000 7f29464e1700 1 -- 172.21.15.42:0/68896999 >> v1:172.21.15.42:6789/0 conn(0x7f2940117c90 legacy=0x7f29401301e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:01:38.197 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:38.151+0000 7f29464e1700 1 -- 172.21.15.42:0/68896999 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f294011a4c0 con 0x7f2940135620 2019-12-05T06:01:38.197 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:38.151+0000 7f29454df700 1 --2- 172.21.15.42:0/68896999 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f29401309e0 0x7f294011a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:01:38.198 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:38.151+0000 7f29464e1700 1 --2- 172.21.15.42:0/68896999 >> v2:172.21.15.42:3300/0 conn(0x7f2940135620 0x7f2940137a60 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f294011af40 tx=0x7f2938010b90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:38.198 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:38.151+0000 7f29464e1700 1 --2- 172.21.15.42:0/68896999 >> v2:172.21.15.42:3300/0 conn(0x7f2940135620 0x7f2940137a60 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f294011af40 tx=0x7f2938010b90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:38.198 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:38.151+0000 7f29464e1700 1 --2- 172.21.15.42:0/68896999 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2940135620 0x7f2940137a60 secure :-1 s=READY pgs=1929 cs=0 l=1 rx=0x7f294011af40 tx=0x7f2938010b90).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:01:38.198 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:38.151+0000 7f29464e1700 1 --2- 172.21.15.42:0/68896999 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2940135620 0x7f2940137a60 secure :-1 s=THROTTLE_DONE pgs=1929 cs=0 l=1 rx=0x7f294011af40 tx=0x7f2938010b90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:38.199 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:38.151+0000 7f29367fc700 1 -- 172.21.15.42:0/68896999 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2938012210 con 0x7f2940135620 2019-12-05T06:01:38.199 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:38.151+0000 7f29464e1700 1 --2- 172.21.15.42:0/68896999 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2940135620 0x7f2940137a60 secure :-1 s=THROTTLE_DONE pgs=1929 cs=0 l=1 rx=0x7f294011af40 tx=0x7f2938010b90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:38.199 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:38.151+0000 7f2947743700 1 -- 172.21.15.42:0/68896999 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29400a9ca0 con 0x7f2940135620 2019-12-05T06:01:38.199 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:38.151+0000 7f29367fc700 1 -- 172.21.15.42:0/68896999 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2938005ae0 con 0x7f2940135620 2019-12-05T06:01:38.199 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:38.151+0000 7f29464e1700 1 --2- 172.21.15.42:0/68896999 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2940135620 0x7f2940137a60 secure :-1 s=THROTTLE_DONE pgs=1929 cs=0 l=1 rx=0x7f294011af40 tx=0x7f2938010b90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:38.200 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:38.151+0000 7f29367fc700 1 -- 172.21.15.42:0/68896999 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f29380127f0 con 0x7f2940135620 2019-12-05T06:01:38.200 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:38.151+0000 7f2947743700 1 -- 172.21.15.42:0/68896999 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f294007d770 con 0x7f2940135620 2019-12-05T06:01:38.200 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:38.151+0000 7f29464e1700 1 --2- 172.21.15.42:0/68896999 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2940135620 0x7f2940137a60 secure :-1 s=THROTTLE_DONE pgs=1929 cs=0 l=1 rx=0x7f294011af40 tx=0x7f2938010b90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:38.200 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:38.151+0000 7f29367fc700 1 -- 172.21.15.42:0/68896999 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f2938012b50 con 0x7f2940135620 2019-12-05T06:01:38.201 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:38.151+0000 7f29464e1700 1 --2- 172.21.15.42:0/68896999 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2940135620 0x7f2940137a60 secure :-1 s=THROTTLE_DONE pgs=1929 cs=0 l=1 rx=0x7f294011af40 tx=0x7f2938010b90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:38.201 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:38.151+0000 7f29367fc700 1 --2- 172.21.15.42:0/68896999 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f292803ac10 0x7f292803d120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:01:38.201 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:38.151+0000 7f29367fc700 1 -- 172.21.15.42:0/68896999 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9471+0+0 (secure 0 0 0) 0x7f2938013060 con 0x7f2940135620 2019-12-05T06:01:38.201 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:38.151+0000 7f2945ce0700 1 --2- 172.21.15.42:0/68896999 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f292803ac10 0x7f292803d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:01:38.201 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:38.155+0000 7f2945ce0700 1 --2- 172.21.15.42:0/68896999 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f292803ac10 0x7f292803d120 crc :-1 s=READY pgs=817 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:01:38.201 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:38.155+0000 7f2947743700 1 -- 172.21.15.42:0/68896999 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f294005d110 con 0x7f2940135620 2019-12-05T06:01:38.201 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:38.159+0000 7f29464e1700 1 --2- 172.21.15.42:0/68896999 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2940135620 0x7f2940137a60 secure :-1 s=THROTTLE_DONE pgs=1929 cs=0 l=1 rx=0x7f294011af40 tx=0x7f2938010b90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:38.202 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:38.159+0000 7f29367fc700 1 -- 172.21.15.42:0/68896999 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f2938032540 con 0x7f2940135620 2019-12-05T06:01:38.331 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:38.323+0000 7f2947743700 1 -- 172.21.15.42:0/68896999 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f294005b8c0 con 0x7f2940135620 2019-12-05T06:01:38.331 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:38.327+0000 7f29464e1700 1 --2- 172.21.15.42:0/68896999 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2940135620 0x7f2940137a60 secure :-1 s=THROTTLE_DONE pgs=1929 cs=0 l=1 rx=0x7f294011af40 tx=0x7f2938010b90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:38.331 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:38.327+0000 7f29367fc700 1 -- 172.21.15.42:0/68896999 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 233 v233) v1 ==== 95+0+2109 (secure 0 0 0) 0x7f29380327a0 con 0x7f2940135620 2019-12-05T06:01:38.334 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:01:38.334 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":233,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":9292,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6835/613535662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":613535662},{"type":"v1","addr":"172.21.15.42:6835","nonce":613535662}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":233}],"filesystems":[{"mdsmap":{"epoch":232,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:01:04.420072+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":89,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6653},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6653":{"gid":6653,"name":"b","rank":0,"incarnation":229,"state":"up:active","state_seq":13,"addr":"172.21.15.40:6833/2780168808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2780168808},{"type":"v1","addr":"172.21.15.40:6833","nonce":2780168808}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:01:38.335 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 233 2019-12-05T06:01:38.337 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:38.331+0000 7f2947743700 1 -- 172.21.15.42:0/68896999 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f292803ac10 msgr2=0x7f292803d120 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:01:38.337 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:38.331+0000 7f2947743700 1 --2- 172.21.15.42:0/68896999 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f292803ac10 0x7f292803d120 crc :-1 s=READY pgs=817 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:01:38.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:38.331+0000 7f2947743700 1 -- 172.21.15.42:0/68896999 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2940135620 msgr2=0x7f2940137a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:01:38.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:38.331+0000 7f2947743700 1 --2- 172.21.15.42:0/68896999 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2940135620 0x7f2940137a60 secure :-1 s=READY pgs=1929 cs=0 l=1 rx=0x7f294011af40 tx=0x7f2938010b90).stop 2019-12-05T06:01:38.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:38.331+0000 7f2947743700 1 -- 172.21.15.42:0/68896999 shutdown_connections 2019-12-05T06:01:38.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:38.331+0000 7f2947743700 1 --2- 172.21.15.42:0/68896999 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f292803ac10 0x7f292803d120 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:01:38.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:38.331+0000 7f2947743700 1 --2- 172.21.15.42:0/68896999 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2940135620 0x7f2940137a60 unknown :-1 s=CLOSED pgs=1929 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:01:38.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:38.331+0000 7f2947743700 1 --2- 172.21.15.42:0/68896999 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f29401309e0 0x7f294011a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:01:38.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:38.331+0000 7f2947743700 1 -- 172.21.15.42:0/68896999 shutdown_connections 2019-12-05T06:01:38.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:38.331+0000 7f2947743700 1 -- 172.21.15.42:0/68896999 wait complete. 2019-12-05T06:01:38.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:38.331+0000 7f2947743700 1 -- 172.21.15.42:0/68896999 >> 172.21.15.42:0/68896999 conn(0x7f294007e4f0 msgr2=0x7f294009f6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:01:38.352 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:01:40.353 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:01:40.447 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.431+0000 7fb926b08700 1 Processor -- start 2019-12-05T06:01:40.448 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.431+0000 7fb926b08700 1 -- start start 2019-12-05T06:01:40.448 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.435+0000 7fb926b08700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb9200a7730 0x7fb92008df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:01:40.448 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.435+0000 7fb926b08700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb92007e710 0x7fb92007eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:01:40.448 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.435+0000 7fb926b08700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9200a0700 con 0x7fb92009f2d0 2019-12-05T06:01:40.448 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.435+0000 7fb926b08700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9200a0af0 con 0x7fb9200a7730 2019-12-05T06:01:40.449 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.435+0000 7fb926b08700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb92008e5a0 con 0x7fb92007e710 2019-12-05T06:01:40.449 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.435+0000 7fb9248a4700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb92007e710 0x7fb92007eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:01:40.449 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.435+0000 7fb9250a5700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb9200a7730 0x7fb92008df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:01:40.449 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.435+0000 7fb9248a4700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb92007e710 0x7fb92007eb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:47080/0 (socket says 172.21.15.42:47080) 2019-12-05T06:01:40.449 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.435+0000 7fb9248a4700 1 -- 172.21.15.42:0/3810079595 learned_addr learned my addr 172.21.15.42:0/3810079595 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:01:40.450 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.435+0000 7fb9258a6700 1 --1- 172.21.15.42:0/3810079595 >> v1:172.21.15.40:6789/0 conn(0x7fb92009f2d0 0x7fb92009f700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:56094/0 (socket says 172.21.15.42:56094) 2019-12-05T06:01:40.451 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.435+0000 7fb9248a4700 1 -- 172.21.15.42:0/3810079595 >> v2:172.21.15.40:3300/0 conn(0x7fb9200a7730 msgr2=0x7fb92008df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:01:40.451 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.435+0000 7fb9248a4700 1 --2- 172.21.15.42:0/3810079595 >> v2:172.21.15.40:3300/0 conn(0x7fb9200a7730 0x7fb92008df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:01:40.452 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.435+0000 7fb9248a4700 1 -- 172.21.15.42:0/3810079595 >> v1:172.21.15.40:6789/0 conn(0x7fb92009f2d0 legacy=0x7fb92009f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:01:40.452 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.435+0000 7fb9248a4700 1 -- 172.21.15.42:0/3810079595 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb92007f090 con 0x7fb92007e710 2019-12-05T06:01:40.452 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.435+0000 7fb9250a5700 1 --2- 172.21.15.42:0/3810079595 >> v2:172.21.15.40:3300/0 conn(0x7fb9200a7730 0x7fb92008df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:01:40.452 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.435+0000 7fb9248a4700 1 --2- 172.21.15.42:0/3810079595 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb92007e710 0x7fb92007eb40 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fb910008790 tx=0x7fb91000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:40.452 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.435+0000 7fb9248a4700 1 --2- 172.21.15.42:0/3810079595 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb92007e710 0x7fb92007eb40 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fb910008790 tx=0x7fb91000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:40.452 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.435+0000 7fb9248a4700 1 --2- 172.21.15.42:0/3810079595 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb92007e710 0x7fb92007eb40 secure :-1 s=READY pgs=427 cs=0 l=1 rx=0x7fb910008790 tx=0x7fb91000fd30).ready entity=mon.2 client_cookie=2f9e1681a9377a13 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:01:40.453 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.435+0000 7fb9248a4700 1 --2- 172.21.15.42:0/3810079595 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb92007e710 0x7fb92007eb40 secure :-1 s=THROTTLE_DONE pgs=427 cs=0 l=1 rx=0x7fb910008790 tx=0x7fb91000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:40.453 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.435+0000 7fb917fff700 1 -- 172.21.15.42:0/3810079595 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb9100113b0 con 0x7fb92007e710 2019-12-05T06:01:40.453 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.435+0000 7fb9248a4700 1 --2- 172.21.15.42:0/3810079595 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb92007e710 0x7fb92007eb40 secure :-1 s=THROTTLE_DONE pgs=427 cs=0 l=1 rx=0x7fb910008790 tx=0x7fb91000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:40.453 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.435+0000 7fb917fff700 1 -- 172.21.15.42:0/3810079595 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb910008bb0 con 0x7fb92007e710 2019-12-05T06:01:40.453 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.435+0000 7fb9248a4700 1 --2- 172.21.15.42:0/3810079595 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb92007e710 0x7fb92007eb40 secure :-1 s=THROTTLE_DONE pgs=427 cs=0 l=1 rx=0x7fb910008790 tx=0x7fb91000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:40.453 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.435+0000 7fb926b08700 1 -- 172.21.15.42:0/3810079595 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb92007e710 msgr2=0x7fb92007eb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:01:40.453 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.435+0000 7fb917fff700 1 -- 172.21.15.42:0/3810079595 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb910011990 con 0x7fb92007e710 2019-12-05T06:01:40.454 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.435+0000 7fb926b08700 1 --2- 172.21.15.42:0/3810079595 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb92007e710 0x7fb92007eb40 secure :-1 s=READY pgs=427 cs=0 l=1 rx=0x7fb910008790 tx=0x7fb91000fd30).stop 2019-12-05T06:01:40.454 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.435+0000 7fb926b08700 1 -- 172.21.15.42:0/3810079595 shutdown_connections 2019-12-05T06:01:40.454 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.435+0000 7fb926b08700 1 --2- 172.21.15.42:0/3810079595 >> v2:172.21.15.40:3300/0 conn(0x7fb9200a7730 0x7fb92008df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:01:40.454 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.435+0000 7fb926b08700 1 --2- 172.21.15.42:0/3810079595 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb92007e710 0x7fb92007eb40 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:01:40.454 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.435+0000 7fb926b08700 1 -- 172.21.15.42:0/3810079595 shutdown_connections 2019-12-05T06:01:40.454 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.435+0000 7fb926b08700 1 -- 172.21.15.42:0/3810079595 wait complete. 2019-12-05T06:01:40.454 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.435+0000 7fb926b08700 1 -- 172.21.15.42:0/3810079595 >> 172.21.15.42:0/3810079595 conn(0x7fb9200aabd0 msgr2=0x7fb9200a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:01:40.455 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.439+0000 7fb926b08700 1 Processor -- start 2019-12-05T06:01:40.455 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.439+0000 7fb926b08700 1 -- start start 2019-12-05T06:01:40.455 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.439+0000 7fb926b08700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb9201384f0 0x7fb920116f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:01:40.455 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.439+0000 7fb926b08700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb920138920 0x7fb920117460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:01:40.455 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.439+0000 7fb926b08700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb920119e30 con 0x7fb920124b00 2019-12-05T06:01:40.455 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.439+0000 7fb926b08700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb920124f30 con 0x7fb9201384f0 2019-12-05T06:01:40.455 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.439+0000 7fb926b08700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb920122bf0 con 0x7fb920138920 2019-12-05T06:01:40.456 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.439+0000 7fb9248a4700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb9201384f0 0x7fb920116f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:01:40.456 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.439+0000 7fb9248a4700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb9201384f0 0x7fb920116f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:46620/0 (socket says 172.21.15.42:46620) 2019-12-05T06:01:40.456 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.439+0000 7fb9248a4700 1 -- 172.21.15.42:0/1410978725 learned_addr learned my addr 172.21.15.42:0/1410978725 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:01:40.456 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.439+0000 7fb9258a6700 1 --2- 172.21.15.42:0/1410978725 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb920138920 0x7fb920117460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:01:40.456 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.439+0000 7fb9248a4700 1 -- 172.21.15.42:0/1410978725 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb920138920 msgr2=0x7fb920117460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:01:40.456 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.439+0000 7fb9248a4700 1 --2- 172.21.15.42:0/1410978725 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb920138920 0x7fb920117460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:01:40.457 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.439+0000 7fb9248a4700 1 -- 172.21.15.42:0/1410978725 >> v1:172.21.15.40:6789/0 conn(0x7fb920124b00 legacy=0x7fb9201222e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:01:40.457 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.439+0000 7fb9248a4700 1 -- 172.21.15.42:0/1410978725 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb920140160 con 0x7fb9201384f0 2019-12-05T06:01:40.457 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.439+0000 7fb9248a4700 1 --2- 172.21.15.42:0/1410978725 >> v2:172.21.15.42:3300/0 conn(0x7fb9201384f0 0x7fb920116f10 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fb910008790 tx=0x7fb910011f60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:40.457 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.439+0000 7fb9248a4700 1 --2- 172.21.15.42:0/1410978725 >> v2:172.21.15.42:3300/0 conn(0x7fb9201384f0 0x7fb920116f10 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fb910008790 tx=0x7fb910011f60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:40.457 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.439+0000 7fb9248a4700 1 --2- 172.21.15.42:0/1410978725 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb9201384f0 0x7fb920116f10 secure :-1 s=READY pgs=1930 cs=0 l=1 rx=0x7fb910008790 tx=0x7fb910011f60).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:01:40.457 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.439+0000 7fb9248a4700 1 --2- 172.21.15.42:0/1410978725 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb9201384f0 0x7fb920116f10 secure :-1 s=THROTTLE_DONE pgs=1930 cs=0 l=1 rx=0x7fb910008790 tx=0x7fb910011f60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:40.457 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.439+0000 7fb915ffb700 1 -- 172.21.15.42:0/1410978725 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb910007340 con 0x7fb9201384f0 2019-12-05T06:01:40.457 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.439+0000 7fb9248a4700 1 --2- 172.21.15.42:0/1410978725 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb9201384f0 0x7fb920116f10 secure :-1 s=THROTTLE_DONE pgs=1930 cs=0 l=1 rx=0x7fb910008790 tx=0x7fb910011f60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:40.458 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.439+0000 7fb915ffb700 1 -- 172.21.15.42:0/1410978725 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9100115b0 con 0x7fb9201384f0 2019-12-05T06:01:40.458 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.439+0000 7fb9248a4700 1 --2- 172.21.15.42:0/1410978725 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb9201384f0 0x7fb920116f10 secure :-1 s=THROTTLE_DONE pgs=1930 cs=0 l=1 rx=0x7fb910008790 tx=0x7fb910011f60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:40.458 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.439+0000 7fb926b08700 1 -- 172.21.15.42:0/1410978725 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb92013f150 con 0x7fb9201384f0 2019-12-05T06:01:40.458 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.439+0000 7fb915ffb700 1 -- 172.21.15.42:0/1410978725 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb910005540 con 0x7fb9201384f0 2019-12-05T06:01:40.458 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.439+0000 7fb926b08700 1 -- 172.21.15.42:0/1410978725 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb92013f650 con 0x7fb9201384f0 2019-12-05T06:01:40.458 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.439+0000 7fb9248a4700 1 --2- 172.21.15.42:0/1410978725 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb9201384f0 0x7fb920116f10 secure :-1 s=THROTTLE_DONE pgs=1930 cs=0 l=1 rx=0x7fb910008790 tx=0x7fb910011f60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:40.458 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.439+0000 7fb915ffb700 1 -- 172.21.15.42:0/1410978725 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fb910005820 con 0x7fb9201384f0 2019-12-05T06:01:40.459 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.439+0000 7fb9248a4700 1 --2- 172.21.15.42:0/1410978725 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb9201384f0 0x7fb920116f10 secure :-1 s=THROTTLE_DONE pgs=1930 cs=0 l=1 rx=0x7fb910008790 tx=0x7fb910011f60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:40.459 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.443+0000 7fb915ffb700 1 --2- 172.21.15.42:0/1410978725 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb90003ade0 0x7fb90003d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:01:40.459 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.443+0000 7fb915ffb700 1 -- 172.21.15.42:0/1410978725 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9557+0+0 (secure 0 0 0) 0x7fb91008f230 con 0x7fb9201384f0 2019-12-05T06:01:40.459 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.443+0000 7fb9250a5700 1 --2- 172.21.15.42:0/1410978725 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb90003ade0 0x7fb90003d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:01:40.459 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.443+0000 7fb9250a5700 1 --2- 172.21.15.42:0/1410978725 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb90003ade0 0x7fb90003d2f0 crc :-1 s=READY pgs=818 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:01:40.460 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.447+0000 7fb926b08700 1 -- 172.21.15.42:0/1410978725 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb92005d100 con 0x7fb9201384f0 2019-12-05T06:01:40.460 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.455+0000 7fb9248a4700 1 --2- 172.21.15.42:0/1410978725 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb9201384f0 0x7fb920116f10 secure :-1 s=THROTTLE_DONE pgs=1930 cs=0 l=1 rx=0x7fb910008790 tx=0x7fb910011f60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:40.461 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.455+0000 7fb915ffb700 1 -- 172.21.15.42:0/1410978725 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fb910076290 con 0x7fb9201384f0 2019-12-05T06:01:40.621 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.615+0000 7fb926b08700 1 -- 172.21.15.42:0/1410978725 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb92005b8b0 con 0x7fb9201384f0 2019-12-05T06:01:40.621 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.615+0000 7fb9248a4700 1 --2- 172.21.15.42:0/1410978725 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb9201384f0 0x7fb920116f10 secure :-1 s=THROTTLE_DONE pgs=1930 cs=0 l=1 rx=0x7fb910008790 tx=0x7fb910011f60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:40.622 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.615+0000 7fb915ffb700 1 -- 172.21.15.42:0/1410978725 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 234 v234) v1 ==== 95+0+1771 (secure 0 0 0) 0x7fb910007f50 con 0x7fb9201384f0 2019-12-05T06:01:40.622 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 234 2019-12-05T06:01:40.622 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:01:40.622 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":234,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":234,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:01:39.358486+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":90,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9292},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9292":{"gid":9292,"name":"a","rank":0,"incarnation":234,"state":"up:replay","state_seq":1,"addr":"172.21.15.42:6835/613535662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":613535662},{"type":"v1","addr":"172.21.15.42:6835","nonce":613535662}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:01:40.630 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.619+0000 7fb926b08700 1 -- 172.21.15.42:0/1410978725 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb90003ade0 msgr2=0x7fb90003d2f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:01:40.630 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.619+0000 7fb926b08700 1 --2- 172.21.15.42:0/1410978725 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb90003ade0 0x7fb90003d2f0 crc :-1 s=READY pgs=818 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:01:40.630 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.619+0000 7fb926b08700 1 -- 172.21.15.42:0/1410978725 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb9201384f0 msgr2=0x7fb920116f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:01:40.630 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.619+0000 7fb926b08700 1 --2- 172.21.15.42:0/1410978725 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb9201384f0 0x7fb920116f10 secure :-1 s=READY pgs=1930 cs=0 l=1 rx=0x7fb910008790 tx=0x7fb910011f60).stop 2019-12-05T06:01:40.631 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.619+0000 7fb926b08700 1 -- 172.21.15.42:0/1410978725 shutdown_connections 2019-12-05T06:01:40.631 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.619+0000 7fb926b08700 1 --2- 172.21.15.42:0/1410978725 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb90003ade0 0x7fb90003d2f0 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:01:40.631 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.619+0000 7fb926b08700 1 --2- 172.21.15.42:0/1410978725 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb920138920 0x7fb920117460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:01:40.631 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.619+0000 7fb926b08700 1 --2- 172.21.15.42:0/1410978725 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb9201384f0 0x7fb920116f10 unknown :-1 s=CLOSED pgs=1930 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:01:40.631 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.619+0000 7fb926b08700 1 -- 172.21.15.42:0/1410978725 shutdown_connections 2019-12-05T06:01:40.631 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.619+0000 7fb926b08700 1 -- 172.21.15.42:0/1410978725 wait complete. 2019-12-05T06:01:40.631 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.619+0000 7fb926b08700 1 -- 172.21.15.42:0/1410978725 >> 172.21.15.42:0/1410978725 conn(0x7fb9200a6d20 msgr2=0x7fb9200a9700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:01:40.636 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b down, removed from mdsmap 2019-12-05T06:01:40.636 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2019-12-05T06:01:40.637 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:01:40.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.711+0000 7f417ce55700 1 Processor -- start 2019-12-05T06:01:40.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.711+0000 7f417ce55700 1 -- start start 2019-12-05T06:01:40.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.711+0000 7f417ce55700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f41780a7740 0x7f417808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:01:40.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.711+0000 7f417ce55700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f417809fae0 0x7f417807eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:01:40.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.711+0000 7f417ce55700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41780a0710 con 0x7f417807e720 2019-12-05T06:01:40.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.711+0000 7f417ce55700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41780a0b00 con 0x7f41780a7740 2019-12-05T06:01:40.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.711+0000 7f417ce55700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f417808e5b0 con 0x7f417809fae0 2019-12-05T06:01:40.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.711+0000 7f4176d9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f41780a7740 0x7f417808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:01:40.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.711+0000 7f4176d9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f41780a7740 0x7f417808df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:46626/0 (socket says 172.21.15.42:46626) 2019-12-05T06:01:40.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.711+0000 7f4176d9d700 1 -- 172.21.15.42:0/144672029 learned_addr learned my addr 172.21.15.42:0/144672029 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:01:40.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.711+0000 7f416ffff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f417807e720 0x7f417809f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:46666/0 (socket says 172.21.15.42:46666) 2019-12-05T06:01:40.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.711+0000 7f417759e700 1 --2- 172.21.15.42:0/144672029 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f417809fae0 0x7f417807eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:01:40.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.711+0000 7f4176d9d700 1 -- 172.21.15.42:0/144672029 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f417809fae0 msgr2=0x7f417807eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:01:40.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.711+0000 7f4176d9d700 1 --2- 172.21.15.42:0/144672029 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f417809fae0 0x7f417807eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:01:40.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.711+0000 7f4176d9d700 1 -- 172.21.15.42:0/144672029 >> v1:172.21.15.42:6789/0 conn(0x7f417807e720 legacy=0x7f417809f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:01:40.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.711+0000 7f417659c700 1 -- 172.21.15.42:0/144672029 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3294824975 0 0) 0x7f41780a0710 con 0x7f417807e720 2019-12-05T06:01:40.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.711+0000 7f4176d9d700 1 -- 172.21.15.42:0/144672029 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f417807f0a0 con 0x7f41780a7740 2019-12-05T06:01:40.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.711+0000 7f4176d9d700 1 --2- 172.21.15.42:0/144672029 >> v2:172.21.15.42:3300/0 conn(0x7f41780a7740 0x7f417808df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f4168008790 tx=0x7f416800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:40.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.711+0000 7f4176d9d700 1 --2- 172.21.15.42:0/144672029 >> v2:172.21.15.42:3300/0 conn(0x7f41780a7740 0x7f417808df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f4168008790 tx=0x7f416800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:40.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.711+0000 7f4176d9d700 1 --2- 172.21.15.42:0/144672029 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f41780a7740 0x7f417808df80 secure :-1 s=READY pgs=1932 cs=0 l=1 rx=0x7f4168008790 tx=0x7f416800fd30).ready entity=mon.1 client_cookie=e60d24f439b948a4 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:01:40.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.711+0000 7f4176d9d700 1 --2- 172.21.15.42:0/144672029 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f41780a7740 0x7f417808df80 secure :-1 s=THROTTLE_DONE pgs=1932 cs=0 l=1 rx=0x7f4168008790 tx=0x7f416800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:40.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.711+0000 7f417659c700 1 -- 172.21.15.42:0/144672029 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4168008c90 con 0x7f41780a7740 2019-12-05T06:01:40.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.711+0000 7f4176d9d700 1 --2- 172.21.15.42:0/144672029 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f41780a7740 0x7f417808df80 secure :-1 s=THROTTLE_DONE pgs=1932 cs=0 l=1 rx=0x7f4168008790 tx=0x7f416800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:40.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.711+0000 7f4176d9d700 1 --2- 172.21.15.42:0/144672029 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f41780a7740 0x7f417808df80 secure :-1 s=THROTTLE_DONE pgs=1932 cs=0 l=1 rx=0x7f4168008790 tx=0x7f416800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:40.724 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.711+0000 7f417659c700 1 -- 172.21.15.42:0/144672029 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41680088b0 con 0x7f41780a7740 2019-12-05T06:01:40.724 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.711+0000 7f417659c700 1 -- 172.21.15.42:0/144672029 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4168011b10 con 0x7f41780a7740 2019-12-05T06:01:40.724 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.711+0000 7f417ce55700 1 -- 172.21.15.42:0/144672029 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f41780a7740 msgr2=0x7f417808df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:01:40.724 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.711+0000 7f417ce55700 1 --2- 172.21.15.42:0/144672029 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f41780a7740 0x7f417808df80 secure :-1 s=READY pgs=1932 cs=0 l=1 rx=0x7f4168008790 tx=0x7f416800fd30).stop 2019-12-05T06:01:40.724 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.711+0000 7f417ce55700 1 -- 172.21.15.42:0/144672029 shutdown_connections 2019-12-05T06:01:40.724 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.711+0000 7f417ce55700 1 --2- 172.21.15.42:0/144672029 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f417809fae0 0x7f417807eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:01:40.725 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.715+0000 7f417ce55700 1 --2- 172.21.15.42:0/144672029 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f41780a7740 0x7f417808df80 unknown :-1 s=CLOSED pgs=1932 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:01:40.725 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.715+0000 7f417ce55700 1 -- 172.21.15.42:0/144672029 shutdown_connections 2019-12-05T06:01:40.725 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.715+0000 7f417ce55700 1 -- 172.21.15.42:0/144672029 wait complete. 2019-12-05T06:01:40.725 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.715+0000 7f417ce55700 1 -- 172.21.15.42:0/144672029 >> 172.21.15.42:0/144672029 conn(0x7f41780aabe0 msgr2=0x7f41780a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:01:40.725 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.715+0000 7f417ce55700 1 Processor -- start 2019-12-05T06:01:40.725 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.715+0000 7f417ce55700 1 -- start start 2019-12-05T06:01:40.730 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.715+0000 7f417ce55700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f41781156d0 0x7f4178115b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:01:40.730 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.715+0000 7f417ce55700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4178064420 con 0x7f41781272d0 2019-12-05T06:01:40.730 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.715+0000 7f417ce55700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4178126cb0 con 0x7f417808eeb0 2019-12-05T06:01:40.730 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.715+0000 7f417ce55700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4178064180 con 0x7f41781156d0 2019-12-05T06:01:40.731 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.715+0000 7f4176d9d700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f417808eeb0 0x7f4178063800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:46670/0 (socket says 172.21.15.42:46670) 2019-12-05T06:01:40.731 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.715+0000 7f4176d9d700 1 -- 172.21.15.42:0/1232534390 learned_addr learned my addr 172.21.15.42:0/1232534390 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:01:40.731 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.715+0000 7f416ffff700 1 --2- 172.21.15.42:0/1232534390 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f41781156d0 0x7f4178115b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:01:40.731 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.715+0000 7f416f7fe700 1 -- 172.21.15.42:0/1232534390 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 714218504 0 0) 0x7f4178126cb0 con 0x7f417808eeb0 2019-12-05T06:01:40.731 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.715+0000 7f416f7fe700 1 -- 172.21.15.42:0/1232534390 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f415c0030b0 con 0x7f417808eeb0 2019-12-05T06:01:40.731 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.715+0000 7f416f7fe700 1 -- 172.21.15.42:0/1232534390 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 3016610041 0 0) 0x7f415c0030b0 con 0x7f417808eeb0 2019-12-05T06:01:40.732 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.715+0000 7f416f7fe700 1 -- 172.21.15.42:0/1232534390 --> v1:172.21.15.42:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4178126cb0 con 0x7f417808eeb0 2019-12-05T06:01:40.732 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.715+0000 7f416f7fe700 1 -- 172.21.15.42:0/1232534390 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7f4168011db0 con 0x7f417808eeb0 2019-12-05T06:01:40.732 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.715+0000 7f416f7fe700 1 -- 172.21.15.42:0/1232534390 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f41781156d0 msgr2=0x7f4178115b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:01:40.732 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.715+0000 7f416f7fe700 1 --2- 172.21.15.42:0/1232534390 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f41781156d0 0x7f4178115b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:01:40.732 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.715+0000 7f416f7fe700 1 -- 172.21.15.42:0/1232534390 >> v1:172.21.15.42:6789/0 conn(0x7f417808eeb0 legacy=0x7f4178063800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:01:40.732 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.715+0000 7f416f7fe700 1 -- 172.21.15.42:0/1232534390 >> v1:172.21.15.40:6789/0 conn(0x7f41781272d0 legacy=0x7f41781316c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:01:40.732 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.715+0000 7f416f7fe700 1 --2- 172.21.15.42:0/1232534390 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f415c0038a0 0x7f415c003d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:01:40.733 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.715+0000 7f416f7fe700 1 --2- 172.21.15.42:0/1232534390 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f415c004420 0x7f415c004850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:01:40.733 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.715+0000 7f416f7fe700 1 --2- 172.21.15.42:0/1232534390 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f415c004ed0 0x7f415c00b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:01:40.733 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.715+0000 7f416f7fe700 1 -- 172.21.15.42:0/1232534390 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4178064180 con 0x7f415c004420 2019-12-05T06:01:40.733 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.715+0000 7f416f7fe700 1 -- 172.21.15.42:0/1232534390 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f415c00bbc0 con 0x7f415c0038a0 2019-12-05T06:01:40.733 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.715+0000 7f416f7fe700 1 -- 172.21.15.42:0/1232534390 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f415c00bd80 con 0x7f415c004ed0 2019-12-05T06:01:40.733 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.715+0000 7f4176d9d700 1 --2- 172.21.15.42:0/1232534390 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f415c0038a0 0x7f415c003d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:01:40.733 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.715+0000 7f416ffff700 1 --2- 172.21.15.42:0/1232534390 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f415c004420 0x7f415c004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:01:40.734 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.715+0000 7f4176d9d700 1 -- 172.21.15.42:0/1232534390 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f415c004ed0 msgr2=0x7f415c00b670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:01:40.734 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.715+0000 7f417759e700 1 --2- 172.21.15.42:0/1232534390 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f415c004ed0 0x7f415c00b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:01:40.734 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.715+0000 7f4176d9d700 1 --2- 172.21.15.42:0/1232534390 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f415c004ed0 0x7f415c00b670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:01:40.734 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.715+0000 7f4176d9d700 1 -- 172.21.15.42:0/1232534390 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f415c004420 msgr2=0x7f415c004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:01:40.734 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.715+0000 7f4176d9d700 1 --2- 172.21.15.42:0/1232534390 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f415c004420 0x7f415c004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:01:40.734 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.715+0000 7f4176d9d700 1 -- 172.21.15.42:0/1232534390 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4178065980 con 0x7f415c0038a0 2019-12-05T06:01:40.734 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.715+0000 7f417ce55700 1 -- 172.21.15.42:0/1232534390 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4178127700 con 0x7f415c0038a0 2019-12-05T06:01:40.735 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.715+0000 7f417ce55700 1 -- 172.21.15.42:0/1232534390 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4178127a60 con 0x7f415c0038a0 2019-12-05T06:01:40.735 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.715+0000 7f4176d9d700 1 --2- 172.21.15.42:0/1232534390 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f415c0038a0 0x7f415c003d10 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f4168007930 tx=0x7f4168008f50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:40.735 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.715+0000 7f4176d9d700 1 --2- 172.21.15.42:0/1232534390 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f415c0038a0 0x7f415c003d10 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f4168007930 tx=0x7f4168008f50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:40.735 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.715+0000 7f4176d9d700 1 --2- 172.21.15.42:0/1232534390 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f415c0038a0 0x7f415c003d10 secure :-1 s=READY pgs=1934 cs=0 l=1 rx=0x7f4168007930 tx=0x7f4168008f50).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:01:40.735 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.715+0000 7f4176d9d700 1 --2- 172.21.15.42:0/1232534390 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f415c0038a0 0x7f415c003d10 secure :-1 s=THROTTLE_DONE pgs=1934 cs=0 l=1 rx=0x7f4168007930 tx=0x7f4168008f50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:40.735 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.715+0000 7f416f7fe700 1 -- 172.21.15.42:0/1232534390 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f41680076c0 con 0x7f415c0038a0 2019-12-05T06:01:40.735 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.715+0000 7f4176d9d700 1 --2- 172.21.15.42:0/1232534390 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f415c0038a0 0x7f415c003d10 secure :-1 s=THROTTLE_DONE pgs=1934 cs=0 l=1 rx=0x7f4168007930 tx=0x7f4168008f50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:40.736 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.715+0000 7f416f7fe700 1 -- 172.21.15.42:0/1232534390 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4168007350 con 0x7f415c0038a0 2019-12-05T06:01:40.736 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.715+0000 7f4176d9d700 1 --2- 172.21.15.42:0/1232534390 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f415c0038a0 0x7f415c003d10 secure :-1 s=THROTTLE_DONE pgs=1934 cs=0 l=1 rx=0x7f4168007930 tx=0x7f4168008f50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:40.736 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.715+0000 7f416f7fe700 1 -- 172.21.15.42:0/1232534390 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4168005b30 con 0x7f415c0038a0 2019-12-05T06:01:40.736 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.715+0000 7f4176d9d700 1 --2- 172.21.15.42:0/1232534390 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f415c0038a0 0x7f415c003d10 secure :-1 s=THROTTLE_DONE pgs=1934 cs=0 l=1 rx=0x7f4168007930 tx=0x7f4168008f50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:40.737 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.719+0000 7f416f7fe700 1 -- 172.21.15.42:0/1232534390 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f4168010440 con 0x7f415c0038a0 2019-12-05T06:01:40.737 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.719+0000 7f4176d9d700 1 --2- 172.21.15.42:0/1232534390 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f415c0038a0 0x7f415c003d10 secure :-1 s=THROTTLE_DONE pgs=1934 cs=0 l=1 rx=0x7f4168007930 tx=0x7f4168008f50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:40.737 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.719+0000 7f416f7fe700 1 --2- 172.21.15.42:0/1232534390 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f415c043c80 0x7f415c046190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:01:40.737 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.719+0000 7f416f7fe700 1 -- 172.21.15.42:0/1232534390 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9557+0+0 (secure 0 0 0) 0x7f416804be80 con 0x7f415c0038a0 2019-12-05T06:01:40.737 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.719+0000 7f416ffff700 1 --2- 172.21.15.42:0/1232534390 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f415c043c80 0x7f415c046190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:01:40.737 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.719+0000 7f417ce55700 1 -- 172.21.15.42:0/1232534390 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4150003200 con 0x7f415c0038a0 2019-12-05T06:01:40.737 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.719+0000 7f416ffff700 1 --2- 172.21.15.42:0/1232534390 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f415c043c80 0x7f415c046190 crc :-1 s=READY pgs=819 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:01:40.738 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.723+0000 7f4176d9d700 1 --2- 172.21.15.42:0/1232534390 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f415c0038a0 0x7f415c003d10 secure :-1 s=THROTTLE_DONE pgs=1934 cs=0 l=1 rx=0x7f4168007930 tx=0x7f4168008f50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:40.738 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.723+0000 7f416f7fe700 1 -- 172.21.15.42:0/1232534390 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f4168010de0 con 0x7f415c0038a0 2019-12-05T06:01:40.938 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.923+0000 7f417ce55700 1 -- 172.21.15.42:0/1232534390 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4150003630 con 0x7f415c0038a0 2019-12-05T06:01:40.938 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.927+0000 7f4176d9d700 1 --2- 172.21.15.42:0/1232534390 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f415c0038a0 0x7f415c003d10 secure :-1 s=THROTTLE_DONE pgs=1934 cs=0 l=1 rx=0x7f4168007930 tx=0x7f4168008f50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:40.939 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.927+0000 7f416f7fe700 1 -- 172.21.15.42:0/1232534390 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 234 v234) v1 ==== 95+0+1771 (secure 0 0 0) 0x7f4168032ee0 con 0x7f415c0038a0 2019-12-05T06:01:40.939 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 234 2019-12-05T06:01:40.939 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.927+0000 7f417ce55700 1 -- 172.21.15.42:0/1232534390 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f415c043c80 msgr2=0x7f415c046190 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:01:40.939 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.927+0000 7f417ce55700 1 --2- 172.21.15.42:0/1232534390 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f415c043c80 0x7f415c046190 crc :-1 s=READY pgs=819 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:01:40.939 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.927+0000 7f417ce55700 1 -- 172.21.15.42:0/1232534390 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f415c0038a0 msgr2=0x7f415c003d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:01:40.939 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.927+0000 7f417759e700 1 -- 172.21.15.42:0/1232534390 reap_dead start 2019-12-05T06:01:40.939 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.927+0000 7f417ce55700 1 --2- 172.21.15.42:0/1232534390 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f415c0038a0 0x7f415c003d10 secure :-1 s=READY pgs=1934 cs=0 l=1 rx=0x7f4168007930 tx=0x7f4168008f50).stop 2019-12-05T06:01:40.940 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.927+0000 7f417ce55700 1 -- 172.21.15.42:0/1232534390 shutdown_connections 2019-12-05T06:01:40.940 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.927+0000 7f417ce55700 1 --2- 172.21.15.42:0/1232534390 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f415c0038a0 0x7f415c003d10 unknown :-1 s=CLOSED pgs=1934 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:01:40.940 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.927+0000 7f417ce55700 1 -- 172.21.15.42:0/1232534390 shutdown_connections 2019-12-05T06:01:40.940 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.927+0000 7f417ce55700 1 -- 172.21.15.42:0/1232534390 wait complete. 2019-12-05T06:01:40.940 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:40.927+0000 7f417ce55700 1 -- 172.21.15.42:0/1232534390 >> 172.21.15.42:0/1232534390 conn(0x7f417807ea50 msgr2=0x7f41780a69e0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:01:40.940 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:01:40.941 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":234,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":234,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:01:39.358486+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":90,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9292},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9292":{"gid":9292,"name":"a","rank":0,"incarnation":234,"state":"up:replay","state_seq":1,"addr":"172.21.15.42:6835/613535662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":613535662},{"type":"v1","addr":"172.21.15.42:6835","nonce":613535662}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:01:40.944 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 234, "feature_flags": { "enable_multiple": false, "ever_enabled_multiple": false }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2019-12-05T05:03:35.604738+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 234, "ever_allowed_features": 2, "explicitly_allowed_features": 2, "failed": [], "flags": 18, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_9292": { "addr": "172.21.15.42:6835/613535662", "addrs": { "addrvec": [ { "addr": "172.21.15.42:6834", "nonce": 613535662, "type": "v2" }, { "addr": "172.21.15.42:6835", "nonce": 613535662, "type": "v1" } ] }, "export_targets": [], "features": 4540138292836696063, "flags": 0, "gid": 9292, "incarnation": 234, "name": "a", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 90, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 2, "min_compat_client": "0 (unknown)", "modified": "2019-12-05T06:01:39.358486+0000", "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 9292 } } } ], "standbys": [] } 2019-12-05T06:01:42.945 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:01:43.052 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:43.039+0000 7feaa9607700 1 Processor -- start 2019-12-05T06:01:43.052 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:43.039+0000 7feaa9607700 1 -- start start 2019-12-05T06:01:43.052 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:43.039+0000 7feaa9607700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7feaa40a7740 0x7feaa408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:01:43.052 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:43.039+0000 7feaa9607700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7feaa409fae0 0x7feaa407eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:01:43.053 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:43.039+0000 7feaa9607700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feaa40a0710 con 0x7feaa407e720 2019-12-05T06:01:43.053 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:43.039+0000 7feaa9607700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feaa40a0b00 con 0x7feaa40a7740 2019-12-05T06:01:43.053 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:43.039+0000 7feaa9607700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feaa408e5b0 con 0x7feaa409fae0 2019-12-05T06:01:43.053 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:43.043+0000 7feaa3fff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7feaa409fae0 0x7feaa407eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:01:43.053 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:43.043+0000 7feaa3fff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7feaa409fae0 0x7feaa407eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:46656/0 (socket says 172.21.15.42:46656) 2019-12-05T06:01:43.053 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:43.043+0000 7feaa3fff700 1 -- 172.21.15.42:0/908307897 learned_addr learned my addr 172.21.15.42:0/908307897 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:01:43.053 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:43.043+0000 7feaa37fe700 1 --2- 172.21.15.42:0/908307897 >> v2:172.21.15.40:3300/0 conn(0x7feaa40a7740 0x7feaa408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:01:43.054 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:43.043+0000 7feaa3fff700 1 -- 172.21.15.42:0/908307897 >> v2:172.21.15.40:3300/0 conn(0x7feaa40a7740 msgr2=0x7feaa408df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:01:43.054 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:43.043+0000 7feaa3fff700 1 --2- 172.21.15.42:0/908307897 >> v2:172.21.15.40:3300/0 conn(0x7feaa40a7740 0x7feaa408df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:01:43.054 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:43.043+0000 7feaa3fff700 1 -- 172.21.15.42:0/908307897 >> v1:172.21.15.40:6789/0 conn(0x7feaa407e720 legacy=0x7feaa409f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:01:43.054 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:43.043+0000 7feaa3fff700 1 -- 172.21.15.42:0/908307897 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feaa407f0a0 con 0x7feaa409fae0 2019-12-05T06:01:43.054 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:43.043+0000 7feaa3fff700 1 --2- 172.21.15.42:0/908307897 >> v2:172.21.15.42:3300/0 conn(0x7feaa409fae0 0x7feaa407eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fea9c0032b0 tx=0x7fea9c0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:43.054 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:43.043+0000 7feaa3fff700 1 --2- 172.21.15.42:0/908307897 >> v2:172.21.15.42:3300/0 conn(0x7feaa409fae0 0x7feaa407eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fea9c0032b0 tx=0x7fea9c0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:43.054 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:43.043+0000 7feaa3fff700 1 --2- 172.21.15.42:0/908307897 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feaa409fae0 0x7feaa407eb50 secure :-1 s=READY pgs=1935 cs=0 l=1 rx=0x7fea9c0032b0 tx=0x7fea9c0100b0).ready entity=mon.1 client_cookie=4b2305e1242329b5 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:01:43.055 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:43.043+0000 7feaa3fff700 1 --2- 172.21.15.42:0/908307897 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feaa409fae0 0x7feaa407eb50 secure :-1 s=THROTTLE_DONE pgs=1935 cs=0 l=1 rx=0x7fea9c0032b0 tx=0x7fea9c0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:43.055 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:43.043+0000 7feaa27fc700 1 -- 172.21.15.42:0/908307897 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fea9c010d60 con 0x7feaa409fae0 2019-12-05T06:01:43.055 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:43.043+0000 7feaa3fff700 1 --2- 172.21.15.42:0/908307897 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feaa409fae0 0x7feaa407eb50 secure :-1 s=THROTTLE_DONE pgs=1935 cs=0 l=1 rx=0x7fea9c0032b0 tx=0x7fea9c0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:43.055 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:43.043+0000 7feaa3fff700 1 --2- 172.21.15.42:0/908307897 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feaa409fae0 0x7feaa407eb50 secure :-1 s=THROTTLE_DONE pgs=1935 cs=0 l=1 rx=0x7fea9c0032b0 tx=0x7fea9c0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:43.055 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:43.043+0000 7feaa27fc700 1 -- 172.21.15.42:0/908307897 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea9c010980 con 0x7feaa409fae0 2019-12-05T06:01:43.055 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:43.043+0000 7feaa27fc700 1 -- 172.21.15.42:0/908307897 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fea9c0114a0 con 0x7feaa409fae0 2019-12-05T06:01:43.056 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:43.043+0000 7feaa9607700 1 -- 172.21.15.42:0/908307897 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feaa409fae0 msgr2=0x7feaa407eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:01:43.056 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:43.043+0000 7feaa9607700 1 --2- 172.21.15.42:0/908307897 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feaa409fae0 0x7feaa407eb50 secure :-1 s=READY pgs=1935 cs=0 l=1 rx=0x7fea9c0032b0 tx=0x7fea9c0100b0).stop 2019-12-05T06:01:43.056 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:43.043+0000 7feaa9607700 1 -- 172.21.15.42:0/908307897 shutdown_connections 2019-12-05T06:01:43.056 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:43.043+0000 7feaa9607700 1 --2- 172.21.15.42:0/908307897 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feaa409fae0 0x7feaa407eb50 unknown :-1 s=CLOSED pgs=1935 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:01:43.056 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:43.043+0000 7feaa9607700 1 --2- 172.21.15.42:0/908307897 >> v2:172.21.15.40:3300/0 conn(0x7feaa40a7740 0x7feaa408df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:01:43.057 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:43.043+0000 7feaa9607700 1 -- 172.21.15.42:0/908307897 shutdown_connections 2019-12-05T06:01:43.057 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:43.043+0000 7feaa9607700 1 -- 172.21.15.42:0/908307897 wait complete. 2019-12-05T06:01:43.057 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:43.043+0000 7feaa9607700 1 -- 172.21.15.42:0/908307897 >> 172.21.15.42:0/908307897 conn(0x7feaa40aabe0 msgr2=0x7feaa40a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:01:43.061 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:43.047+0000 7feaa9607700 1 Processor -- start 2019-12-05T06:01:43.061 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:43.047+0000 7feaa9607700 1 -- start start 2019-12-05T06:01:43.061 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:43.047+0000 7feaa9607700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7feaa4121860 0x7feaa411a500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:01:43.061 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:43.047+0000 7feaa9607700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7feaa40783f0 0x7feaa4078820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:01:43.061 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:43.047+0000 7feaa9607700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feaa411aa50 con 0x7feaa40a4f00 2019-12-05T06:01:43.062 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:43.047+0000 7feaa9607700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feaa40a49c0 con 0x7feaa40783f0 2019-12-05T06:01:43.062 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:43.047+0000 7feaa9607700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feaa40a4c60 con 0x7feaa4121860 2019-12-05T06:01:43.062 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:43.047+0000 7feaa2ffd700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7feaa40a4f00 0x7feaa40a5e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:46698/0 (socket says 172.21.15.42:46698) 2019-12-05T06:01:43.062 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:43.047+0000 7feaa2ffd700 1 -- 172.21.15.42:0/3455139773 learned_addr learned my addr 172.21.15.42:0/3455139773 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:01:43.062 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:43.047+0000 7feaa37fe700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7feaa4121860 0x7feaa411a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:01:43.062 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:43.047+0000 7feaa3fff700 1 --2- 172.21.15.42:0/3455139773 >> v2:172.21.15.40:3300/0 conn(0x7feaa40783f0 0x7feaa4078820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:01:43.063 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:43.051+0000 7fea9bfff700 1 -- 172.21.15.42:0/3455139773 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2157941965 0 0) 0x7feaa411aa50 con 0x7feaa40a4f00 2019-12-05T06:01:43.063 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:43.051+0000 7fea9bfff700 1 -- 172.21.15.42:0/3455139773 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea880030b0 con 0x7feaa40a4f00 2019-12-05T06:01:43.063 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:43.051+0000 7feaa3fff700 1 -- 172.21.15.42:0/3455139773 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7feaa4121860 msgr2=0x7feaa411a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:01:43.063 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:43.051+0000 7feaa3fff700 1 --2- 172.21.15.42:0/3455139773 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7feaa4121860 0x7feaa411a500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:01:43.063 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:43.051+0000 7feaa3fff700 1 -- 172.21.15.42:0/3455139773 >> v1:172.21.15.42:6789/0 conn(0x7feaa40a4f00 legacy=0x7feaa40a5e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:01:43.063 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:43.051+0000 7feaa3fff700 1 -- 172.21.15.42:0/3455139773 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feaa40a6640 con 0x7feaa40783f0 2019-12-05T06:01:43.063 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:43.051+0000 7feaa3fff700 1 --2- 172.21.15.42:0/3455139773 >> v2:172.21.15.40:3300/0 conn(0x7feaa40783f0 0x7feaa4078820 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7feaa40a59c0 tx=0x7fea9c010f00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:43.064 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:43.051+0000 7feaa37fe700 1 --2- 172.21.15.42:0/3455139773 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7feaa4121860 0x7feaa411a500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:01:43.064 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:43.051+0000 7feaa3fff700 1 --2- 172.21.15.42:0/3455139773 >> v2:172.21.15.40:3300/0 conn(0x7feaa40783f0 0x7feaa4078820 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7feaa40a59c0 tx=0x7fea9c010f00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:43.064 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:43.051+0000 7feaa3fff700 1 --2- 172.21.15.42:0/3455139773 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feaa40783f0 0x7feaa4078820 secure :-1 s=READY pgs=1469 cs=0 l=1 rx=0x7feaa40a59c0 tx=0x7fea9c010f00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:01:43.064 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:43.051+0000 7feaa3fff700 1 --2- 172.21.15.42:0/3455139773 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feaa40783f0 0x7feaa4078820 secure :-1 s=THROTTLE_DONE pgs=1469 cs=0 l=1 rx=0x7feaa40a59c0 tx=0x7fea9c010f00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:43.064 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:43.051+0000 7fea9bfff700 1 -- 172.21.15.42:0/3455139773 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fea9c011bc0 con 0x7feaa40783f0 2019-12-05T06:01:43.064 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:43.051+0000 7feaa3fff700 1 --2- 172.21.15.42:0/3455139773 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feaa40783f0 0x7feaa4078820 secure :-1 s=THROTTLE_DONE pgs=1469 cs=0 l=1 rx=0x7feaa40a59c0 tx=0x7fea9c010f00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:43.064 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:43.051+0000 7fea9bfff700 1 -- 172.21.15.42:0/3455139773 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea9c0110c0 con 0x7feaa40783f0 2019-12-05T06:01:43.065 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:43.051+0000 7feaa3fff700 1 --2- 172.21.15.42:0/3455139773 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feaa40783f0 0x7feaa4078820 secure :-1 s=THROTTLE_DONE pgs=1469 cs=0 l=1 rx=0x7feaa40a59c0 tx=0x7fea9c010f00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:43.065 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:43.051+0000 7feaa9607700 1 -- 172.21.15.42:0/3455139773 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feaa4078d70 con 0x7feaa40783f0 2019-12-05T06:01:43.065 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:43.051+0000 7fea9bfff700 1 -- 172.21.15.42:0/3455139773 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fea9c005ab0 con 0x7feaa40783f0 2019-12-05T06:01:43.065 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:43.051+0000 7feaa9607700 1 -- 172.21.15.42:0/3455139773 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feaa4137cc0 con 0x7feaa40783f0 2019-12-05T06:01:43.065 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:43.051+0000 7feaa3fff700 1 --2- 172.21.15.42:0/3455139773 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feaa40783f0 0x7feaa4078820 secure :-1 s=THROTTLE_DONE pgs=1469 cs=0 l=1 rx=0x7feaa40a59c0 tx=0x7fea9c010f00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:43.065 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:43.051+0000 7fea9bfff700 1 -- 172.21.15.42:0/3455139773 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fea9c00f280 con 0x7feaa40783f0 2019-12-05T06:01:43.065 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:43.051+0000 7feaa3fff700 1 --2- 172.21.15.42:0/3455139773 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feaa40783f0 0x7feaa4078820 secure :-1 s=THROTTLE_DONE pgs=1469 cs=0 l=1 rx=0x7feaa40a59c0 tx=0x7fea9c010f00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:43.066 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:43.055+0000 7fea9bfff700 1 --2- 172.21.15.42:0/3455139773 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fea8803b260 0x7fea8803d770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:01:43.066 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:43.055+0000 7fea9bfff700 1 -- 172.21.15.42:0/3455139773 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9557+0+0 (secure 0 0 0) 0x7fea9c04b4d0 con 0x7feaa40783f0 2019-12-05T06:01:43.066 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:43.055+0000 7feaa37fe700 1 --2- 172.21.15.42:0/3455139773 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fea8803b260 0x7fea8803d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:01:43.066 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:43.055+0000 7feaa37fe700 1 --2- 172.21.15.42:0/3455139773 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fea8803b260 0x7fea8803d770 crc :-1 s=READY pgs=820 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:01:43.068 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:43.055+0000 7feaa9607700 1 -- 172.21.15.42:0/3455139773 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feaa405d110 con 0x7feaa40783f0 2019-12-05T06:01:43.071 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:43.067+0000 7feaa3fff700 1 --2- 172.21.15.42:0/3455139773 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feaa40783f0 0x7feaa4078820 secure :-1 s=THROTTLE_DONE pgs=1469 cs=0 l=1 rx=0x7feaa40a59c0 tx=0x7fea9c010f00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:43.072 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:43.067+0000 7fea9bfff700 1 -- 172.21.15.42:0/3455139773 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fea9c032790 con 0x7feaa40783f0 2019-12-05T06:01:43.185 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:01:43.188 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:01:43.235 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:43.231+0000 7feaa9607700 1 -- 172.21.15.42:0/3455139773 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7feaa405b8c0 con 0x7feaa40783f0 2019-12-05T06:01:43.236 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:43.231+0000 7feaa3fff700 1 --2- 172.21.15.42:0/3455139773 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feaa40783f0 0x7feaa4078820 secure :-1 s=THROTTLE_DONE pgs=1469 cs=0 l=1 rx=0x7feaa40a59c0 tx=0x7fea9c010f00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:43.236 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:43.231+0000 7fea9bfff700 1 -- 172.21.15.42:0/3455139773 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 235 v235) v1 ==== 95+0+1775 (secure 0 0 0) 0x7fea9c005d00 con 0x7feaa40783f0 2019-12-05T06:01:43.238 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:01:43.238 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":235,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":235,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:01:42.389598+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":90,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9292},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9292":{"gid":9292,"name":"a","rank":0,"incarnation":234,"state":"up:reconnect","state_seq":10,"addr":"172.21.15.42:6835/613535662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":613535662},{"type":"v1","addr":"172.21.15.42:6835","nonce":613535662}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:01:43.239 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 235 2019-12-05T06:01:43.241 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:43.235+0000 7feaa9607700 1 -- 172.21.15.42:0/3455139773 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fea8803b260 msgr2=0x7fea8803d770 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:01:43.241 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:43.235+0000 7feaa9607700 1 --2- 172.21.15.42:0/3455139773 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fea8803b260 0x7fea8803d770 crc :-1 s=READY pgs=820 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:01:43.241 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:43.235+0000 7feaa9607700 1 -- 172.21.15.42:0/3455139773 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feaa40783f0 msgr2=0x7feaa4078820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:01:43.241 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:43.235+0000 7feaa9607700 1 --2- 172.21.15.42:0/3455139773 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feaa40783f0 0x7feaa4078820 secure :-1 s=READY pgs=1469 cs=0 l=1 rx=0x7feaa40a59c0 tx=0x7fea9c010f00).stop 2019-12-05T06:01:43.241 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:43.235+0000 7feaa9607700 1 -- 172.21.15.42:0/3455139773 shutdown_connections 2019-12-05T06:01:43.241 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:43.235+0000 7feaa9607700 1 --2- 172.21.15.42:0/3455139773 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fea8803b260 0x7fea8803d770 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:01:43.242 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:43.235+0000 7feaa9607700 1 --2- 172.21.15.42:0/3455139773 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feaa40783f0 0x7feaa4078820 unknown :-1 s=CLOSED pgs=1469 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:01:43.242 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:43.235+0000 7feaa9607700 1 --2- 172.21.15.42:0/3455139773 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7feaa4121860 0x7feaa411a500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:01:43.242 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:43.235+0000 7feaa9607700 1 -- 172.21.15.42:0/3455139773 shutdown_connections 2019-12-05T06:01:43.242 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:43.235+0000 7feaa9607700 1 -- 172.21.15.42:0/3455139773 wait complete. 2019-12-05T06:01:43.242 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:43.235+0000 7feaa9607700 1 -- 172.21.15.42:0/3455139773 >> 172.21.15.42:0/3455139773 conn(0x7feaa409ffb0 msgr2=0x7feaa409f2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:01:43.251 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2019-12-05T06:01:45.252 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:01:45.360 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:45.351+0000 7f9e7fe97700 1 Processor -- start 2019-12-05T06:01:45.360 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:45.351+0000 7f9e7fe97700 1 -- start start 2019-12-05T06:01:45.360 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:45.351+0000 7f9e7fe97700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f9e7809fae0 0x7f9e7814fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:01:45.361 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:45.351+0000 7f9e7fe97700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e780a0710 con 0x7f9e7808df80 2019-12-05T06:01:45.361 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:45.351+0000 7f9e7fe97700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e7807ef20 con 0x7f9e780a7740 2019-12-05T06:01:45.361 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:45.351+0000 7f9e7fe97700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e780a0b00 con 0x7f9e7809fae0 2019-12-05T06:01:45.361 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:45.351+0000 7f9e7ec35700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f9e7809fae0 0x7f9e7814fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:01:45.361 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:45.351+0000 7f9e7ec35700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f9e7809fae0 0x7f9e7814fff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:46676/0 (socket says 172.21.15.42:46676) 2019-12-05T06:01:45.361 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:45.351+0000 7f9e7ec35700 1 -- 172.21.15.42:0/1364615243 learned_addr learned my addr 172.21.15.42:0/1364615243 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:01:45.361 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:45.351+0000 7f9e7ec35700 1 -- 172.21.15.42:0/1364615243 >> v1:172.21.15.42:6789/0 conn(0x7f9e780a7740 legacy=0x7f9e7807e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:01:45.362 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:45.351+0000 7f9e7ec35700 1 -- 172.21.15.42:0/1364615243 >> v1:172.21.15.40:6789/0 conn(0x7f9e7808df80 legacy=0x7f9e7809f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:01:45.362 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:45.351+0000 7f9e7ec35700 1 -- 172.21.15.42:0/1364615243 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e780a9ba0 con 0x7f9e7809fae0 2019-12-05T06:01:45.362 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:45.351+0000 7f9e7ec35700 1 --2- 172.21.15.42:0/1364615243 >> v2:172.21.15.42:3300/0 conn(0x7f9e7809fae0 0x7f9e7814fff0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f9e70008b10 tx=0x7f9e70010030).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:45.362 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:45.351+0000 7f9e7ec35700 1 --2- 172.21.15.42:0/1364615243 >> v2:172.21.15.42:3300/0 conn(0x7f9e7809fae0 0x7f9e7814fff0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f9e70008b10 tx=0x7f9e70010030).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:45.362 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:45.351+0000 7f9e7ec35700 1 --2- 172.21.15.42:0/1364615243 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9e7809fae0 0x7f9e7814fff0 secure :-1 s=READY pgs=1938 cs=0 l=1 rx=0x7f9e70008b10 tx=0x7f9e70010030).ready entity=mon.1 client_cookie=8b0768e47ad399b4 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:01:45.362 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:45.351+0000 7f9e7ec35700 1 --2- 172.21.15.42:0/1364615243 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9e7809fae0 0x7f9e7814fff0 secure :-1 s=THROTTLE_DONE pgs=1938 cs=0 l=1 rx=0x7f9e70008b10 tx=0x7f9e70010030).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:45.362 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:45.351+0000 7f9e7d432700 1 -- 172.21.15.42:0/1364615243 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9e700110c0 con 0x7f9e7809fae0 2019-12-05T06:01:45.363 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:45.351+0000 7f9e7ec35700 1 --2- 172.21.15.42:0/1364615243 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9e7809fae0 0x7f9e7814fff0 secure :-1 s=THROTTLE_DONE pgs=1938 cs=0 l=1 rx=0x7f9e70008b10 tx=0x7f9e70010030).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:45.363 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:45.351+0000 7f9e7ec35700 1 --2- 172.21.15.42:0/1364615243 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9e7809fae0 0x7f9e7814fff0 secure :-1 s=THROTTLE_DONE pgs=1938 cs=0 l=1 rx=0x7f9e70008b10 tx=0x7f9e70010030).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:45.363 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:45.351+0000 7f9e7d432700 1 -- 172.21.15.42:0/1364615243 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e70010a30 con 0x7f9e7809fae0 2019-12-05T06:01:45.363 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:45.351+0000 7f9e7d432700 1 -- 172.21.15.42:0/1364615243 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9e700114b0 con 0x7f9e7809fae0 2019-12-05T06:01:45.363 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:45.351+0000 7f9e7fe97700 1 -- 172.21.15.42:0/1364615243 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9e7809fae0 msgr2=0x7f9e7814fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:01:45.363 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:45.351+0000 7f9e7fe97700 1 --2- 172.21.15.42:0/1364615243 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9e7809fae0 0x7f9e7814fff0 secure :-1 s=READY pgs=1938 cs=0 l=1 rx=0x7f9e70008b10 tx=0x7f9e70010030).stop 2019-12-05T06:01:45.363 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:45.355+0000 7f9e7fe97700 1 -- 172.21.15.42:0/1364615243 shutdown_connections 2019-12-05T06:01:45.364 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:45.355+0000 7f9e7fe97700 1 --2- 172.21.15.42:0/1364615243 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9e7809fae0 0x7f9e7814fff0 unknown :-1 s=CLOSED pgs=1938 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:01:45.364 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:45.355+0000 7f9e7fe97700 1 -- 172.21.15.42:0/1364615243 shutdown_connections 2019-12-05T06:01:45.364 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:45.355+0000 7f9e7fe97700 1 -- 172.21.15.42:0/1364615243 wait complete. 2019-12-05T06:01:45.364 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:45.355+0000 7f9e7fe97700 1 -- 172.21.15.42:0/1364615243 >> 172.21.15.42:0/1364615243 conn(0x7f9e780aabe0 msgr2=0x7f9e780a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:01:45.367 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:45.355+0000 7f9e7fe97700 1 Processor -- start 2019-12-05T06:01:45.367 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:45.355+0000 7f9e7fe97700 1 -- start start 2019-12-05T06:01:45.367 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:45.355+0000 7f9e7fe97700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f9e7807d420 0x7f9e780a2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:01:45.367 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:45.355+0000 7f9e7fe97700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9e78123650 0x7f9e78123a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:01:45.367 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:45.355+0000 7f9e7fe97700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e78127040 con 0x7f9e78126c10 2019-12-05T06:01:45.368 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:45.355+0000 7f9e7fe97700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e781273c0 con 0x7f9e7807d420 2019-12-05T06:01:45.368 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:45.355+0000 7f9e7fe97700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e78127660 con 0x7f9e78123650 2019-12-05T06:01:45.368 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:45.355+0000 7f9e7ec35700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9e78123650 0x7f9e78123a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:01:45.368 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:45.355+0000 7f9e7e434700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f9e7807d420 0x7f9e780a2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:01:45.368 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:45.355+0000 7f9e7ec35700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9e78123650 0x7f9e78123a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:47148/0 (socket says 172.21.15.42:47148) 2019-12-05T06:01:45.368 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:45.355+0000 7f9e7e434700 1 --2- 172.21.15.42:0/2033758442 >> v2:172.21.15.40:3300/0 conn(0x7f9e7807d420 0x7f9e780a2ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.42:34496/0 (socket says 172.21.15.42:34496) 2019-12-05T06:01:45.369 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:45.355+0000 7f9e7ec35700 1 -- 172.21.15.42:0/2033758442 learned_addr learned my addr 172.21.15.42:0/2033758442 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:01:45.369 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:45.359+0000 7f9e7ec35700 1 -- 172.21.15.42:0/2033758442 >> v2:172.21.15.40:3300/0 conn(0x7f9e7807d420 msgr2=0x7f9e780a2ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:01:45.369 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:45.359+0000 7f9e7ec35700 1 --2- 172.21.15.42:0/2033758442 >> v2:172.21.15.40:3300/0 conn(0x7f9e7807d420 0x7f9e780a2ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:01:45.369 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:45.359+0000 7f9e7ec35700 1 -- 172.21.15.42:0/2033758442 >> v1:172.21.15.40:6789/0 conn(0x7f9e78126c10 legacy=0x7f9e78117d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:01:45.369 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:45.359+0000 7f9e6affd700 1 -- 172.21.15.42:0/2033758442 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3787303780 0 0) 0x7f9e78127040 con 0x7f9e78126c10 2019-12-05T06:01:45.369 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:45.359+0000 7f9e7ec35700 1 -- 172.21.15.42:0/2033758442 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e78127900 con 0x7f9e78123650 2019-12-05T06:01:45.369 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:45.359+0000 7f9e7e434700 1 --2- 172.21.15.42:0/2033758442 >> v2:172.21.15.40:3300/0 conn(0x7f9e7807d420 0x7f9e780a2ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:01:45.370 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:45.359+0000 7f9e7ec35700 1 --2- 172.21.15.42:0/2033758442 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9e78123650 0x7f9e78123a80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f9e70008b10 tx=0x7f9e70010ce0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:45.370 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:45.359+0000 7f9e7ec35700 1 --2- 172.21.15.42:0/2033758442 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9e78123650 0x7f9e78123a80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f9e70008b10 tx=0x7f9e70010ce0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:45.370 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:45.359+0000 7f9e7ec35700 1 --2- 172.21.15.42:0/2033758442 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9e78123650 0x7f9e78123a80 secure :-1 s=READY pgs=428 cs=0 l=1 rx=0x7f9e70008b10 tx=0x7f9e70010ce0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:01:45.370 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:45.359+0000 7f9e7ec35700 1 --2- 172.21.15.42:0/2033758442 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9e78123650 0x7f9e78123a80 secure :-1 s=THROTTLE_DONE pgs=428 cs=0 l=1 rx=0x7f9e70008b10 tx=0x7f9e70010ce0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:45.370 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:45.359+0000 7f9e6affd700 1 -- 172.21.15.42:0/2033758442 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9e70011d10 con 0x7f9e78123650 2019-12-05T06:01:45.371 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:45.359+0000 7f9e7ec35700 1 --2- 172.21.15.42:0/2033758442 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9e78123650 0x7f9e78123a80 secure :-1 s=THROTTLE_DONE pgs=428 cs=0 l=1 rx=0x7f9e70008b10 tx=0x7f9e70010ce0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:45.371 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:45.359+0000 7f9e7ec35700 1 --2- 172.21.15.42:0/2033758442 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9e78123650 0x7f9e78123a80 secure :-1 s=THROTTLE_DONE pgs=428 cs=0 l=1 rx=0x7f9e70008b10 tx=0x7f9e70010ce0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:45.371 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:45.359+0000 7f9e6affd700 1 -- 172.21.15.42:0/2033758442 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e70010e10 con 0x7f9e78123650 2019-12-05T06:01:45.371 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:45.359+0000 7f9e7fe97700 1 -- 172.21.15.42:0/2033758442 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e7812e060 con 0x7f9e78123650 2019-12-05T06:01:45.371 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:45.359+0000 7f9e6affd700 1 -- 172.21.15.42:0/2033758442 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9e70005ca0 con 0x7f9e78123650 2019-12-05T06:01:45.371 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:45.359+0000 7f9e7fe97700 1 -- 172.21.15.42:0/2033758442 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e7812e5e0 con 0x7f9e78123650 2019-12-05T06:01:45.371 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:45.359+0000 7f9e7ec35700 1 --2- 172.21.15.42:0/2033758442 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9e78123650 0x7f9e78123a80 secure :-1 s=THROTTLE_DONE pgs=428 cs=0 l=1 rx=0x7f9e70008b10 tx=0x7f9e70010ce0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:45.372 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:45.359+0000 7f9e6affd700 1 -- 172.21.15.42:0/2033758442 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f9e7000f280 con 0x7f9e78123650 2019-12-05T06:01:45.372 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:45.359+0000 7f9e7ec35700 1 --2- 172.21.15.42:0/2033758442 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9e78123650 0x7f9e78123a80 secure :-1 s=THROTTLE_DONE pgs=428 cs=0 l=1 rx=0x7f9e70008b10 tx=0x7f9e70010ce0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:45.372 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:45.359+0000 7f9e6affd700 1 --2- 172.21.15.42:0/2033758442 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9e6003aca0 0x7f9e6003d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:01:45.372 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:45.363+0000 7f9e6affd700 1 -- 172.21.15.42:0/2033758442 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9557+0+0 (secure 0 0 0) 0x7f9e7004b670 con 0x7f9e78123650 2019-12-05T06:01:45.372 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:45.363+0000 7f9e7e434700 1 --2- 172.21.15.42:0/2033758442 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9e6003aca0 0x7f9e6003d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:01:45.373 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:45.363+0000 7f9e7e434700 1 --2- 172.21.15.42:0/2033758442 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9e6003aca0 0x7f9e6003d1b0 crc :-1 s=READY pgs=821 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:01:45.373 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:45.367+0000 7f9e7fe97700 1 -- 172.21.15.42:0/2033758442 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e7805d110 con 0x7f9e78123650 2019-12-05T06:01:45.381 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:45.375+0000 7f9e7ec35700 1 --2- 172.21.15.42:0/2033758442 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9e78123650 0x7f9e78123a80 secure :-1 s=THROTTLE_DONE pgs=428 cs=0 l=1 rx=0x7f9e70008b10 tx=0x7f9e70010ce0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:45.381 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:45.375+0000 7f9e6affd700 1 -- 172.21.15.42:0/2033758442 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f9e700334e0 con 0x7f9e78123650 2019-12-05T06:01:45.551 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:45.547+0000 7f9e7fe97700 1 -- 172.21.15.42:0/2033758442 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9e7805b8c0 con 0x7f9e78123650 2019-12-05T06:01:45.552 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:45.547+0000 7f9e7ec35700 1 --2- 172.21.15.42:0/2033758442 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9e78123650 0x7f9e78123a80 secure :-1 s=THROTTLE_DONE pgs=428 cs=0 l=1 rx=0x7f9e70008b10 tx=0x7f9e70010ce0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:45.552 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:45.547+0000 7f9e6affd700 1 -- 172.21.15.42:0/2033758442 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 237 v237) v1 ==== 95+0+1772 (secure 0 0 0) 0x7f9e7000f590 con 0x7f9e78123650 2019-12-05T06:01:45.554 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 237 2019-12-05T06:01:45.555 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:01:45.555 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":237,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":237,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:01:44.401238+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":90,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9292},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9292":{"gid":9292,"name":"a","rank":0,"incarnation":234,"state":"up:active","state_seq":12,"addr":"172.21.15.42:6835/613535662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":613535662},{"type":"v1","addr":"172.21.15.42:6835","nonce":613535662}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:01:45.558 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:45.551+0000 7f9e7fe97700 1 -- 172.21.15.42:0/2033758442 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9e6003aca0 msgr2=0x7f9e6003d1b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:01:45.558 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:45.551+0000 7f9e7fe97700 1 --2- 172.21.15.42:0/2033758442 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9e6003aca0 0x7f9e6003d1b0 crc :-1 s=READY pgs=821 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:01:45.558 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:45.551+0000 7f9e7fe97700 1 -- 172.21.15.42:0/2033758442 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9e78123650 msgr2=0x7f9e78123a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:01:45.558 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:45.551+0000 7f9e7fe97700 1 --2- 172.21.15.42:0/2033758442 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9e78123650 0x7f9e78123a80 secure :-1 s=READY pgs=428 cs=0 l=1 rx=0x7f9e70008b10 tx=0x7f9e70010ce0).stop 2019-12-05T06:01:45.558 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:45.551+0000 7f9e7fe97700 1 -- 172.21.15.42:0/2033758442 shutdown_connections 2019-12-05T06:01:45.558 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:45.551+0000 7f9e7fe97700 1 --2- 172.21.15.42:0/2033758442 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9e6003aca0 0x7f9e6003d1b0 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:01:45.559 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:45.551+0000 7f9e7fe97700 1 --2- 172.21.15.42:0/2033758442 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9e78123650 0x7f9e78123a80 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:01:45.559 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:45.551+0000 7f9e7fe97700 1 --2- 172.21.15.42:0/2033758442 >> v2:172.21.15.40:3300/0 conn(0x7f9e7807d420 0x7f9e780a2ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:01:45.559 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:45.551+0000 7f9e7fe97700 1 -- 172.21.15.42:0/2033758442 shutdown_connections 2019-12-05T06:01:45.559 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:45.551+0000 7f9e7fe97700 1 -- 172.21.15.42:0/2033758442 wait complete. 2019-12-05T06:01:45.559 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:45.551+0000 7f9e7fe97700 1 -- 172.21.15.42:0/2033758442 >> 172.21.15.42:0/2033758442 conn(0x7f9e780a6fd0 msgr2=0x7f9e78111db0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:01:45.568 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a has gained rank=0, replacing gid=6653 2019-12-05T06:01:45.568 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 6 secs before reviving mds.b 2019-12-05T06:01:51.569 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.b 2019-12-05T06:01:51.569 INFO:tasks.ceph.mds.b:Restarting daemon 2019-12-05T06:01:51.570 INFO:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2019-12-05T06:01:51.574 INFO:tasks.ceph.mds.b:Started 2019-12-05T06:01:51.574 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:01:51.636 INFO:tasks.ceph.mds.b.smithi040.stdout:starting mds.b at 2019-12-05T06:01:51.682 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:51.671+0000 7fe6b7cff700 1 Processor -- start 2019-12-05T06:01:51.682 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:51.671+0000 7fe6b7cff700 1 -- start start 2019-12-05T06:01:51.682 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:51.671+0000 7fe6b7cff700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe6b008df70 0x7fe6b009f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:01:51.683 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:51.671+0000 7fe6b7cff700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6b009fc50 con 0x7fe6b00a7730 2019-12-05T06:01:51.683 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:51.671+0000 7fe6b7cff700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6b00a0700 con 0x7fe6b009f820 2019-12-05T06:01:51.683 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:51.671+0000 7fe6b7cff700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6b00a0af0 con 0x7fe6b008df70 2019-12-05T06:01:51.683 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:51.671+0000 7fe6b6a9d700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fe6b009f820 0x7fe6b014efd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:46760/0 (socket says 172.21.15.42:46760) 2019-12-05T06:01:51.683 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:51.671+0000 7fe6b6a9d700 1 -- 172.21.15.42:0/2042890407 learned_addr learned my addr 172.21.15.42:0/2042890407 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:01:51.684 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:51.671+0000 7fe6b5a9b700 1 --2- 172.21.15.42:0/2042890407 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe6b008df70 0x7fe6b009f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:01:51.684 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:51.675+0000 7fe6b529a700 1 -- 172.21.15.42:0/2042890407 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1988822858 0 0) 0x7fe6b00a0700 con 0x7fe6b009f820 2019-12-05T06:01:51.684 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:51.675+0000 7fe6b529a700 1 -- 172.21.15.42:0/2042890407 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe69c0030b0 con 0x7fe6b009f820 2019-12-05T06:01:51.684 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:51.675+0000 7fe6b529a700 1 -- 172.21.15.42:0/2042890407 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3783231549 0 0) 0x7fe6b009fc50 con 0x7fe6b00a7730 2019-12-05T06:01:51.684 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:51.675+0000 7fe6b529a700 1 -- 172.21.15.42:0/2042890407 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6b00a0700 con 0x7fe6b00a7730 2019-12-05T06:01:51.684 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:51.675+0000 7fe6b5a9b700 1 -- 172.21.15.42:0/2042890407 >> v1:172.21.15.42:6789/0 conn(0x7fe6b009f820 legacy=0x7fe6b014efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:01:51.685 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:51.675+0000 7fe6b5a9b700 1 -- 172.21.15.42:0/2042890407 >> v1:172.21.15.40:6789/0 conn(0x7fe6b00a7730 legacy=0x7fe6b007e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:01:51.685 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:51.675+0000 7fe6b5a9b700 1 -- 172.21.15.42:0/2042890407 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6b007ef10 con 0x7fe6b008df70 2019-12-05T06:01:51.685 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:51.675+0000 7fe6b5a9b700 1 --2- 172.21.15.42:0/2042890407 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe6b008df70 0x7fe6b009f2d0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fe6a0002f30 tx=0x7fe6a000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:51.685 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:51.675+0000 7fe6b5a9b700 1 --2- 172.21.15.42:0/2042890407 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe6b008df70 0x7fe6b009f2d0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fe6a0002f30 tx=0x7fe6a000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:51.685 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:51.675+0000 7fe6b5a9b700 1 --2- 172.21.15.42:0/2042890407 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe6b008df70 0x7fe6b009f2d0 secure :-1 s=READY pgs=429 cs=0 l=1 rx=0x7fe6a0002f30 tx=0x7fe6a000fbf0).ready entity=mon.2 client_cookie=1875c2a2d4a406dc server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:01:51.686 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:51.675+0000 7fe6b5a9b700 1 --2- 172.21.15.42:0/2042890407 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe6b008df70 0x7fe6b009f2d0 secure :-1 s=THROTTLE_DONE pgs=429 cs=0 l=1 rx=0x7fe6a0002f30 tx=0x7fe6a000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:51.686 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:51.675+0000 7fe6b529a700 1 -- 172.21.15.42:0/2042890407 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe6a00112b0 con 0x7fe6b008df70 2019-12-05T06:01:51.686 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:51.675+0000 7fe6b5a9b700 1 --2- 172.21.15.42:0/2042890407 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe6b008df70 0x7fe6b009f2d0 secure :-1 s=THROTTLE_DONE pgs=429 cs=0 l=1 rx=0x7fe6a0002f30 tx=0x7fe6a000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:51.686 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:51.675+0000 7fe6b5a9b700 1 --2- 172.21.15.42:0/2042890407 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe6b008df70 0x7fe6b009f2d0 secure :-1 s=THROTTLE_DONE pgs=429 cs=0 l=1 rx=0x7fe6a0002f30 tx=0x7fe6a000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:51.686 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:51.675+0000 7fe6b529a700 1 -- 172.21.15.42:0/2042890407 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6a0008bc0 con 0x7fe6b008df70 2019-12-05T06:01:51.686 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:51.675+0000 7fe6b529a700 1 -- 172.21.15.42:0/2042890407 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe6a0011890 con 0x7fe6b008df70 2019-12-05T06:01:51.686 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:51.675+0000 7fe6b7cff700 1 -- 172.21.15.42:0/2042890407 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe6b008df70 msgr2=0x7fe6b009f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:01:51.687 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:51.675+0000 7fe6b7cff700 1 --2- 172.21.15.42:0/2042890407 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe6b008df70 0x7fe6b009f2d0 secure :-1 s=READY pgs=429 cs=0 l=1 rx=0x7fe6a0002f30 tx=0x7fe6a000fbf0).stop 2019-12-05T06:01:51.687 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:51.675+0000 7fe6b7cff700 1 -- 172.21.15.42:0/2042890407 shutdown_connections 2019-12-05T06:01:51.687 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:51.675+0000 7fe6b7cff700 1 --2- 172.21.15.42:0/2042890407 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe6b008df70 0x7fe6b009f2d0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:01:51.687 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:51.675+0000 7fe6b7cff700 1 -- 172.21.15.42:0/2042890407 shutdown_connections 2019-12-05T06:01:51.687 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:51.675+0000 7fe6b7cff700 1 -- 172.21.15.42:0/2042890407 wait complete. 2019-12-05T06:01:51.687 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:51.675+0000 7fe6b7cff700 1 -- 172.21.15.42:0/2042890407 >> 172.21.15.42:0/2042890407 conn(0x7fe6b00aabd0 msgr2=0x7fe6b00a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:01:51.692 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:51.679+0000 7fe6b7cff700 1 Processor -- start 2019-12-05T06:01:51.692 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:51.679+0000 7fe6b7cff700 1 -- start start 2019-12-05T06:01:51.692 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:51.679+0000 7fe6b7cff700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe6b0144960 0x7fe6b00a3170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:01:51.692 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:51.679+0000 7fe6b7cff700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe6b0122760 0x7fe6b0122b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:01:51.693 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:51.679+0000 7fe6b7cff700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6b007a810 con 0x7fe6b007ab50 2019-12-05T06:01:51.693 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:51.679+0000 7fe6b7cff700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6b00a36c0 con 0x7fe6b0144960 2019-12-05T06:01:51.693 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:51.679+0000 7fe6b7cff700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6b0144d90 con 0x7fe6b0122760 2019-12-05T06:01:51.693 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:51.679+0000 7fe6b6a9d700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fe6b007ab50 0x7fe6b01255b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:46766/0 (socket says 172.21.15.42:46766) 2019-12-05T06:01:51.693 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:51.679+0000 7fe6b6a9d700 1 -- 172.21.15.42:0/1410350506 learned_addr learned my addr 172.21.15.42:0/1410350506 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:01:51.693 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:51.679+0000 7fe6b5a9b700 1 --2- 172.21.15.42:0/1410350506 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe6b0122760 0x7fe6b0122b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:01:51.693 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:51.679+0000 7fe6b629c700 1 --2- 172.21.15.42:0/1410350506 >> v2:172.21.15.40:3300/0 conn(0x7fe6b0144960 0x7fe6b00a3170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:01:51.694 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:51.679+0000 7fe6a6ffd700 1 -- 172.21.15.42:0/1410350506 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3893665310 0 0) 0x7fe6b007a810 con 0x7fe6b007ab50 2019-12-05T06:01:51.694 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:51.679+0000 7fe6a6ffd700 1 -- 172.21.15.42:0/1410350506 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6980030b0 con 0x7fe6b007ab50 2019-12-05T06:01:51.694 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:51.679+0000 7fe6b5a9b700 1 -- 172.21.15.42:0/1410350506 >> v2:172.21.15.40:3300/0 conn(0x7fe6b0144960 msgr2=0x7fe6b00a3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:01:51.694 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:51.679+0000 7fe6b5a9b700 1 --2- 172.21.15.42:0/1410350506 >> v2:172.21.15.40:3300/0 conn(0x7fe6b0144960 0x7fe6b00a3170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:01:51.694 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:51.679+0000 7fe6b5a9b700 1 -- 172.21.15.42:0/1410350506 >> v1:172.21.15.42:6789/0 conn(0x7fe6b007ab50 legacy=0x7fe6b01255b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:01:51.694 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:51.679+0000 7fe6b5a9b700 1 -- 172.21.15.42:0/1410350506 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6b0143d70 con 0x7fe6b0122760 2019-12-05T06:01:51.695 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:51.679+0000 7fe6b5a9b700 1 --2- 172.21.15.42:0/1410350506 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe6b0122760 0x7fe6b0122b90 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fe6a0002f30 tx=0x7fe6a00116a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:51.695 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:51.679+0000 7fe6b5a9b700 1 --2- 172.21.15.42:0/1410350506 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe6b0122760 0x7fe6b0122b90 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fe6a0002f30 tx=0x7fe6a00116a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:51.695 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:51.679+0000 7fe6b5a9b700 1 --2- 172.21.15.42:0/1410350506 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe6b0122760 0x7fe6b0122b90 secure :-1 s=READY pgs=430 cs=0 l=1 rx=0x7fe6a0002f30 tx=0x7fe6a00116a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:01:51.695 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:51.679+0000 7fe6b5a9b700 1 --2- 172.21.15.42:0/1410350506 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe6b0122760 0x7fe6b0122b90 secure :-1 s=THROTTLE_DONE pgs=430 cs=0 l=1 rx=0x7fe6a0002f30 tx=0x7fe6a00116a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:51.695 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:51.679+0000 7fe6a6ffd700 1 -- 172.21.15.42:0/1410350506 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe6a0008dd0 con 0x7fe6b0122760 2019-12-05T06:01:51.695 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:51.679+0000 7fe6b5a9b700 1 --2- 172.21.15.42:0/1410350506 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe6b0122760 0x7fe6b0122b90 secure :-1 s=THROTTLE_DONE pgs=430 cs=0 l=1 rx=0x7fe6a0002f30 tx=0x7fe6a00116a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:51.696 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:51.679+0000 7fe6b5a9b700 1 --2- 172.21.15.42:0/1410350506 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe6b0122760 0x7fe6b0122b90 secure :-1 s=THROTTLE_DONE pgs=430 cs=0 l=1 rx=0x7fe6a0002f30 tx=0x7fe6a00116a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:51.696 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:51.679+0000 7fe6a6ffd700 1 -- 172.21.15.42:0/1410350506 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6a00114b0 con 0x7fe6b0122760 2019-12-05T06:01:51.696 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:51.679+0000 7fe6b7cff700 1 -- 172.21.15.42:0/1410350506 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6b007af80 con 0x7fe6b0122760 2019-12-05T06:01:51.696 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:51.679+0000 7fe6a6ffd700 1 -- 172.21.15.42:0/1410350506 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe6a000f360 con 0x7fe6b0122760 2019-12-05T06:01:51.696 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:51.679+0000 7fe6b7cff700 1 -- 172.21.15.42:0/1410350506 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6b007b500 con 0x7fe6b0122760 2019-12-05T06:01:51.697 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:51.683+0000 7fe6b5a9b700 1 --2- 172.21.15.42:0/1410350506 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe6b0122760 0x7fe6b0122b90 secure :-1 s=THROTTLE_DONE pgs=430 cs=0 l=1 rx=0x7fe6a0002f30 tx=0x7fe6a00116a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:51.697 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:51.683+0000 7fe6a6ffd700 1 -- 172.21.15.42:0/1410350506 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fe6a0005540 con 0x7fe6b0122760 2019-12-05T06:01:51.697 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:51.683+0000 7fe6b5a9b700 1 --2- 172.21.15.42:0/1410350506 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe6b0122760 0x7fe6b0122b90 secure :-1 s=THROTTLE_DONE pgs=430 cs=0 l=1 rx=0x7fe6a0002f30 tx=0x7fe6a00116a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:51.697 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:51.683+0000 7fe6a6ffd700 1 --2- 172.21.15.42:0/1410350506 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe69803b2a0 0x7fe69803d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:01:51.697 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:51.683+0000 7fe6a6ffd700 1 -- 172.21.15.42:0/1410350506 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9557+0+0 (secure 0 0 0) 0x7fe6a004b590 con 0x7fe6b0122760 2019-12-05T06:01:51.697 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:51.683+0000 7fe6b629c700 1 --2- 172.21.15.42:0/1410350506 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe69803b2a0 0x7fe69803d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:01:51.699 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:51.683+0000 7fe6b629c700 1 --2- 172.21.15.42:0/1410350506 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe69803b2a0 0x7fe69803d7b0 crc :-1 s=READY pgs=822 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:01:51.700 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:51.687+0000 7fe6b7cff700 1 -- 172.21.15.42:0/1410350506 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6b005d100 con 0x7fe6b0122760 2019-12-05T06:01:51.700 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:51.695+0000 7fe6b5a9b700 1 --2- 172.21.15.42:0/1410350506 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe6b0122760 0x7fe6b0122b90 secure :-1 s=THROTTLE_DONE pgs=430 cs=0 l=1 rx=0x7fe6a0002f30 tx=0x7fe6a00116a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:51.701 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:51.695+0000 7fe6a6ffd700 1 -- 172.21.15.42:0/1410350506 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fe6a0032610 con 0x7fe6b0122760 2019-12-05T06:01:51.881 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:51.875+0000 7fe6b7cff700 1 -- 172.21.15.42:0/1410350506 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe6b005b8b0 con 0x7fe6b0122760 2019-12-05T06:01:51.882 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:51.875+0000 7fe6b5a9b700 1 --2- 172.21.15.42:0/1410350506 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe6b0122760 0x7fe6b0122b90 secure :-1 s=THROTTLE_DONE pgs=430 cs=0 l=1 rx=0x7fe6a0002f30 tx=0x7fe6a00116a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:51.882 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:51.875+0000 7fe6a6ffd700 1 -- 172.21.15.42:0/1410350506 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 237 v237) v1 ==== 95+0+1772 (secure 0 0 0) 0x7fe6a0032870 con 0x7fe6b0122760 2019-12-05T06:01:51.885 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 237 2019-12-05T06:01:51.885 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:01:51.885 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":237,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":237,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:01:44.401238+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":90,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9292},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9292":{"gid":9292,"name":"a","rank":0,"incarnation":234,"state":"up:active","state_seq":12,"addr":"172.21.15.42:6835/613535662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":613535662},{"type":"v1","addr":"172.21.15.42:6835","nonce":613535662}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:01:51.891 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:51.883+0000 7fe6b7cff700 1 -- 172.21.15.42:0/1410350506 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe69803b2a0 msgr2=0x7fe69803d7b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:01:51.891 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:51.883+0000 7fe6b7cff700 1 --2- 172.21.15.42:0/1410350506 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe69803b2a0 0x7fe69803d7b0 crc :-1 s=READY pgs=822 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:01:51.891 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:51.883+0000 7fe6b7cff700 1 -- 172.21.15.42:0/1410350506 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe6b0122760 msgr2=0x7fe6b0122b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:01:51.891 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:51.883+0000 7fe6b7cff700 1 --2- 172.21.15.42:0/1410350506 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe6b0122760 0x7fe6b0122b90 secure :-1 s=READY pgs=430 cs=0 l=1 rx=0x7fe6a0002f30 tx=0x7fe6a00116a0).stop 2019-12-05T06:01:51.892 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:51.883+0000 7fe6b7cff700 1 -- 172.21.15.42:0/1410350506 shutdown_connections 2019-12-05T06:01:51.892 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:51.883+0000 7fe6b7cff700 1 --2- 172.21.15.42:0/1410350506 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe69803b2a0 0x7fe69803d7b0 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:01:51.892 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:51.883+0000 7fe6b7cff700 1 --2- 172.21.15.42:0/1410350506 >> v2:172.21.15.40:3300/0 conn(0x7fe6b0144960 0x7fe6b00a3170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:01:51.892 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:51.883+0000 7fe6b7cff700 1 --2- 172.21.15.42:0/1410350506 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe6b0122760 0x7fe6b0122b90 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:01:51.892 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:51.883+0000 7fe6b7cff700 1 -- 172.21.15.42:0/1410350506 shutdown_connections 2019-12-05T06:01:51.892 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:51.883+0000 7fe6b7cff700 1 -- 172.21.15.42:0/1410350506 wait complete. 2019-12-05T06:01:51.893 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:51.883+0000 7fe6b7cff700 1 -- 172.21.15.42:0/1410350506 >> 172.21.15.42:0/1410350506 conn(0x7fe6b00a9ad0 msgr2=0x7fe6b007e440 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:01:51.904 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2019-12-05T06:01:53.904 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:01:54.021 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:54.011+0000 7f25b273a700 1 Processor -- start 2019-12-05T06:01:54.021 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:54.011+0000 7f25b273a700 1 -- start start 2019-12-05T06:01:54.022 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:54.011+0000 7f25b273a700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f25ac09fad0 0x7f25ac14ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:01:54.022 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:54.011+0000 7f25b273a700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25ac0a0700 con 0x7f25ac0a7730 2019-12-05T06:01:54.022 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:54.011+0000 7f25b273a700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25ac07ef10 con 0x7f25ac08df70 2019-12-05T06:01:54.022 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:54.011+0000 7f25b273a700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25ac0a0af0 con 0x7f25ac09fad0 2019-12-05T06:01:54.022 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:54.011+0000 7f25abfff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f25ac08df70 0x7f25ac09f2d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:46786/0 (socket says 172.21.15.42:46786) 2019-12-05T06:01:54.022 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:54.011+0000 7f25abfff700 1 -- 172.21.15.42:0/4177970683 learned_addr learned my addr 172.21.15.42:0/4177970683 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:01:54.022 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:54.011+0000 7f25b14d8700 1 --2- 172.21.15.42:0/4177970683 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f25ac09fad0 0x7f25ac14ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:01:54.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:54.011+0000 7f25ab7fe700 1 -- 172.21.15.42:0/4177970683 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3223667463 0 0) 0x7f25ac07ef10 con 0x7f25ac08df70 2019-12-05T06:01:54.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:54.011+0000 7f25ab7fe700 1 -- 172.21.15.42:0/4177970683 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f259c0030b0 con 0x7f25ac08df70 2019-12-05T06:01:54.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:54.011+0000 7f25ab7fe700 1 -- 172.21.15.42:0/4177970683 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1150059844 0 0) 0x7f25ac0a0700 con 0x7f25ac0a7730 2019-12-05T06:01:54.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:54.011+0000 7f25b14d8700 1 -- 172.21.15.42:0/4177970683 >> v1:172.21.15.42:6789/0 conn(0x7f25ac08df70 legacy=0x7f25ac09f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:01:54.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:54.011+0000 7f25b14d8700 1 -- 172.21.15.42:0/4177970683 >> v1:172.21.15.40:6789/0 conn(0x7f25ac0a7730 legacy=0x7f25ac07e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:01:54.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:54.011+0000 7f25b14d8700 1 -- 172.21.15.42:0/4177970683 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25ac0a9b90 con 0x7f25ac09fad0 2019-12-05T06:01:54.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:54.011+0000 7f25b14d8700 1 --2- 172.21.15.42:0/4177970683 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f25ac09fad0 0x7f25ac14ffe0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f25a40032b0 tx=0x7f25a400ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:54.024 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:54.011+0000 7f25b14d8700 1 --2- 172.21.15.42:0/4177970683 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f25ac09fad0 0x7f25ac14ffe0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f25a40032b0 tx=0x7f25a400ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:54.024 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:54.011+0000 7f25b14d8700 1 --2- 172.21.15.42:0/4177970683 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f25ac09fad0 0x7f25ac14ffe0 secure :-1 s=READY pgs=431 cs=0 l=1 rx=0x7f25a40032b0 tx=0x7f25a400ff70).ready entity=mon.2 client_cookie=8447758ccdd13357 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:01:54.024 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:54.011+0000 7f25b14d8700 1 --2- 172.21.15.42:0/4177970683 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f25ac09fad0 0x7f25ac14ffe0 secure :-1 s=THROTTLE_DONE pgs=431 cs=0 l=1 rx=0x7f25a40032b0 tx=0x7f25a400ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:54.024 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:54.011+0000 7f25ab7fe700 1 -- 172.21.15.42:0/4177970683 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f25a40109b0 con 0x7f25ac09fad0 2019-12-05T06:01:54.024 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:54.011+0000 7f25b14d8700 1 --2- 172.21.15.42:0/4177970683 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f25ac09fad0 0x7f25ac14ffe0 secure :-1 s=THROTTLE_DONE pgs=431 cs=0 l=1 rx=0x7f25a40032b0 tx=0x7f25a400ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:54.024 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:54.011+0000 7f25ab7fe700 1 -- 172.21.15.42:0/4177970683 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25a4008b50 con 0x7f25ac09fad0 2019-12-05T06:01:54.024 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:54.011+0000 7f25b14d8700 1 --2- 172.21.15.42:0/4177970683 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f25ac09fad0 0x7f25ac14ffe0 secure :-1 s=THROTTLE_DONE pgs=431 cs=0 l=1 rx=0x7f25a40032b0 tx=0x7f25a400ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:54.025 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:54.011+0000 7f25ab7fe700 1 -- 172.21.15.42:0/4177970683 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f25a40110c0 con 0x7f25ac09fad0 2019-12-05T06:01:54.025 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:54.015+0000 7f25b273a700 1 -- 172.21.15.42:0/4177970683 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f25ac09fad0 msgr2=0x7f25ac14ffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:01:54.025 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:54.015+0000 7f25b273a700 1 --2- 172.21.15.42:0/4177970683 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f25ac09fad0 0x7f25ac14ffe0 secure :-1 s=READY pgs=431 cs=0 l=1 rx=0x7f25a40032b0 tx=0x7f25a400ff70).stop 2019-12-05T06:01:54.025 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:54.015+0000 7f25b273a700 1 -- 172.21.15.42:0/4177970683 shutdown_connections 2019-12-05T06:01:54.025 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:54.015+0000 7f25b273a700 1 --2- 172.21.15.42:0/4177970683 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f25ac09fad0 0x7f25ac14ffe0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:01:54.025 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:54.015+0000 7f25b273a700 1 -- 172.21.15.42:0/4177970683 shutdown_connections 2019-12-05T06:01:54.025 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:54.015+0000 7f25b273a700 1 -- 172.21.15.42:0/4177970683 wait complete. 2019-12-05T06:01:54.026 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:54.015+0000 7f25b273a700 1 -- 172.21.15.42:0/4177970683 >> 172.21.15.42:0/4177970683 conn(0x7f25ac0aabd0 msgr2=0x7f25ac0a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:01:54.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:54.015+0000 7f25b273a700 1 Processor -- start 2019-12-05T06:01:54.040 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:54.015+0000 7f25b273a700 1 -- start start 2019-12-05T06:01:54.040 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:54.019+0000 7f25b273a700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f25ac11afa0 0x7f25ac08f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:01:54.040 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:54.019+0000 7f25b273a700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f25ac1316f0 0x7f25ac131b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:01:54.041 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:54.019+0000 7f25b273a700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25ac11bbf0 con 0x7f25ac118900 2019-12-05T06:01:54.041 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:54.019+0000 7f25b273a700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25ac11ad00 con 0x7f25ac11afa0 2019-12-05T06:01:54.041 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:54.019+0000 7f25b0cd7700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f25ac11afa0 0x7f25ac08f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:01:54.041 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:54.019+0000 7f25b0cd7700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f25ac11afa0 0x7f25ac08f3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:46752/0 (socket says 172.21.15.42:46752) 2019-12-05T06:01:54.041 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:54.019+0000 7f25b0cd7700 1 -- 172.21.15.42:0/1584755941 learned_addr learned my addr 172.21.15.42:0/1584755941 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:01:54.041 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:54.019+0000 7f25b273a700 1 -- 172.21.15.42:0/1584755941 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25ac11b6b0 con 0x7f25ac1316f0 2019-12-05T06:01:54.041 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:54.019+0000 7f25abfff700 1 --2- 172.21.15.42:0/1584755941 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f25ac1316f0 0x7f25ac131b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:01:54.042 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:54.019+0000 7f25b0cd7700 1 -- 172.21.15.42:0/1584755941 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f25ac1316f0 msgr2=0x7f25ac131b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:01:54.042 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:54.019+0000 7f25b0cd7700 1 --2- 172.21.15.42:0/1584755941 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f25ac1316f0 0x7f25ac131b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:01:54.042 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:54.019+0000 7f25b0cd7700 1 -- 172.21.15.42:0/1584755941 >> v1:172.21.15.42:6789/0 conn(0x7f25ac118900 legacy=0x7f25ac118d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:01:54.042 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:54.019+0000 7f25a97fa700 1 -- 172.21.15.42:0/1584755941 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2919357898 0 0) 0x7f25a4010da0 con 0x7f25ac118900 2019-12-05T06:01:54.042 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:54.019+0000 7f25b0cd7700 1 -- 172.21.15.42:0/1584755941 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25ac11b3d0 con 0x7f25ac11afa0 2019-12-05T06:01:54.042 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:54.019+0000 7f25abfff700 1 --2- 172.21.15.42:0/1584755941 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f25ac1316f0 0x7f25ac131b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:01:54.043 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:54.019+0000 7f25b0cd7700 1 --2- 172.21.15.42:0/1584755941 >> v2:172.21.15.42:3300/0 conn(0x7f25ac11afa0 0x7f25ac08f3e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f25ac11aec0 tx=0x7f25a00109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:54.043 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:54.019+0000 7f25b0cd7700 1 --2- 172.21.15.42:0/1584755941 >> v2:172.21.15.42:3300/0 conn(0x7f25ac11afa0 0x7f25ac08f3e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f25ac11aec0 tx=0x7f25a00109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:54.043 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:54.019+0000 7f25b0cd7700 1 --2- 172.21.15.42:0/1584755941 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f25ac11afa0 0x7f25ac08f3e0 secure :-1 s=READY pgs=1945 cs=0 l=1 rx=0x7f25ac11aec0 tx=0x7f25a00109a0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:01:54.043 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:54.019+0000 7f25b0cd7700 1 --2- 172.21.15.42:0/1584755941 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f25ac11afa0 0x7f25ac08f3e0 secure :-1 s=THROTTLE_DONE pgs=1945 cs=0 l=1 rx=0x7f25ac11aec0 tx=0x7f25a00109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:54.043 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:54.019+0000 7f25b0cd7700 1 --2- 172.21.15.42:0/1584755941 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f25ac11afa0 0x7f25ac08f3e0 secure :-1 s=THROTTLE_DONE pgs=1945 cs=0 l=1 rx=0x7f25ac11aec0 tx=0x7f25a00109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:54.043 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:54.019+0000 7f25a97fa700 1 -- 172.21.15.42:0/1584755941 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f25a000caa0 con 0x7f25ac11afa0 2019-12-05T06:01:54.043 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:54.019+0000 7f25b0cd7700 1 --2- 172.21.15.42:0/1584755941 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f25ac11afa0 0x7f25ac08f3e0 secure :-1 s=THROTTLE_DONE pgs=1945 cs=0 l=1 rx=0x7f25ac11aec0 tx=0x7f25a00109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:54.044 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:54.019+0000 7f25a97fa700 1 -- 172.21.15.42:0/1584755941 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25a000c6c0 con 0x7f25ac11afa0 2019-12-05T06:01:54.044 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:54.019+0000 7f25a97fa700 1 -- 172.21.15.42:0/1584755941 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f25a0011120 con 0x7f25ac11afa0 2019-12-05T06:01:54.044 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:54.019+0000 7f25b273a700 1 -- 172.21.15.42:0/1584755941 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25ac08f930 con 0x7f25ac11afa0 2019-12-05T06:01:54.044 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:54.019+0000 7f25b273a700 1 -- 172.21.15.42:0/1584755941 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25ac118500 con 0x7f25ac11afa0 2019-12-05T06:01:54.044 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:54.023+0000 7f25b0cd7700 1 --2- 172.21.15.42:0/1584755941 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f25ac11afa0 0x7f25ac08f3e0 secure :-1 s=THROTTLE_DONE pgs=1945 cs=0 l=1 rx=0x7f25ac11aec0 tx=0x7f25a00109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:54.044 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:54.023+0000 7f25a97fa700 1 -- 172.21.15.42:0/1584755941 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f25a00113f0 con 0x7f25ac11afa0 2019-12-05T06:01:54.053 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:54.023+0000 7f25b0cd7700 1 --2- 172.21.15.42:0/1584755941 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f25ac11afa0 0x7f25ac08f3e0 secure :-1 s=THROTTLE_DONE pgs=1945 cs=0 l=1 rx=0x7f25ac11aec0 tx=0x7f25a00109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:54.053 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:54.023+0000 7f25a97fa700 1 --2- 172.21.15.42:0/1584755941 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f258c03abe0 0x7f258c03d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:01:54.053 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:54.023+0000 7f25a97fa700 1 -- 172.21.15.42:0/1584755941 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9557+0+0 (secure 0 0 0) 0x7f25a004b180 con 0x7f25ac11afa0 2019-12-05T06:01:54.053 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:54.023+0000 7f25abfff700 1 --2- 172.21.15.42:0/1584755941 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f258c03abe0 0x7f258c03d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:01:54.053 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:54.023+0000 7f25abfff700 1 --2- 172.21.15.42:0/1584755941 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f258c03abe0 0x7f258c03d0f0 crc :-1 s=READY pgs=823 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:01:54.053 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:54.027+0000 7f25b273a700 1 -- 172.21.15.42:0/1584755941 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25ac05d100 con 0x7f25ac11afa0 2019-12-05T06:01:54.054 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:54.031+0000 7f25b0cd7700 1 --2- 172.21.15.42:0/1584755941 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f25ac11afa0 0x7f25ac08f3e0 secure :-1 s=THROTTLE_DONE pgs=1945 cs=0 l=1 rx=0x7f25ac11aec0 tx=0x7f25a00109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:54.054 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:54.031+0000 7f25a97fa700 1 -- 172.21.15.42:0/1584755941 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f25a0032200 con 0x7f25ac11afa0 2019-12-05T06:01:54.198 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:54.191+0000 7f25b273a700 1 -- 172.21.15.42:0/1584755941 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f25ac05b8b0 con 0x7f25ac11afa0 2019-12-05T06:01:54.199 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:54.195+0000 7f25b0cd7700 1 --2- 172.21.15.42:0/1584755941 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f25ac11afa0 0x7f25ac08f3e0 secure :-1 s=THROTTLE_DONE pgs=1945 cs=0 l=1 rx=0x7f25ac11aec0 tx=0x7f25a00109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:54.200 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:54.195+0000 7f25a97fa700 1 -- 172.21.15.42:0/1584755941 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 237 v237) v1 ==== 95+0+1772 (secure 0 0 0) 0x7f25a0032460 con 0x7f25ac11afa0 2019-12-05T06:01:54.200 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 237 2019-12-05T06:01:54.200 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:01:54.200 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":237,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":237,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:01:44.401238+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":90,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9292},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9292":{"gid":9292,"name":"a","rank":0,"incarnation":234,"state":"up:active","state_seq":12,"addr":"172.21.15.42:6835/613535662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":613535662},{"type":"v1","addr":"172.21.15.42:6835","nonce":613535662}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:01:54.202 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:54.195+0000 7f25b273a700 1 -- 172.21.15.42:0/1584755941 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f258c03abe0 msgr2=0x7f258c03d0f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:01:54.202 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:54.195+0000 7f25b273a700 1 --2- 172.21.15.42:0/1584755941 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f258c03abe0 0x7f258c03d0f0 crc :-1 s=READY pgs=823 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:01:54.202 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:54.195+0000 7f25b273a700 1 -- 172.21.15.42:0/1584755941 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f25ac11afa0 msgr2=0x7f25ac08f3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:01:54.202 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:54.195+0000 7f25b273a700 1 --2- 172.21.15.42:0/1584755941 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f25ac11afa0 0x7f25ac08f3e0 secure :-1 s=READY pgs=1945 cs=0 l=1 rx=0x7f25ac11aec0 tx=0x7f25a00109a0).stop 2019-12-05T06:01:54.203 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:54.195+0000 7f25b273a700 1 -- 172.21.15.42:0/1584755941 shutdown_connections 2019-12-05T06:01:54.203 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:54.195+0000 7f25b273a700 1 --2- 172.21.15.42:0/1584755941 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f258c03abe0 0x7f258c03d0f0 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:01:54.203 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:54.195+0000 7f25b273a700 1 --2- 172.21.15.42:0/1584755941 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f25ac11afa0 0x7f25ac08f3e0 unknown :-1 s=CLOSED pgs=1945 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:01:54.203 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:54.195+0000 7f25b273a700 1 --2- 172.21.15.42:0/1584755941 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f25ac1316f0 0x7f25ac131b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:01:54.204 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:54.195+0000 7f25b273a700 1 -- 172.21.15.42:0/1584755941 shutdown_connections 2019-12-05T06:01:54.204 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:54.195+0000 7f25b273a700 1 -- 172.21.15.42:0/1584755941 wait complete. 2019-12-05T06:01:54.205 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:54.195+0000 7f25b273a700 1 -- 172.21.15.42:0/1584755941 >> 172.21.15.42:0/1584755941 conn(0x7f25ac0a7f80 msgr2=0x7f25ac059ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:01:54.211 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2019-12-05T06:01:56.212 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:01:56.317 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:56.303+0000 7f720b11c700 1 Processor -- start 2019-12-05T06:01:56.317 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:56.303+0000 7f720b11c700 1 -- start start 2019-12-05T06:01:56.317 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:56.307+0000 7f720b11c700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f720408df80 0x7f720409f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:01:56.318 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:56.307+0000 7f720b11c700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f720409f830 0x7f720414efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:01:56.318 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:56.307+0000 7f720b11c700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f720409fc60 con 0x7f72040a7740 2019-12-05T06:01:56.318 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:56.307+0000 7f720b11c700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72040a0b00 con 0x7f720408df80 2019-12-05T06:01:56.318 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:56.307+0000 7f720b11c700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f720408e5b0 con 0x7f720409f830 2019-12-05T06:01:56.318 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:56.307+0000 7f72096b9700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f72040a7740 0x7f720407e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:46810/0 (socket says 172.21.15.42:46810) 2019-12-05T06:01:56.318 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:56.307+0000 7f72096b9700 1 -- 172.21.15.42:0/2720205142 learned_addr learned my addr 172.21.15.42:0/2720205142 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:01:56.319 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:56.307+0000 7f7208eb8700 1 --2- 172.21.15.42:0/2720205142 >> v2:172.21.15.40:3300/0 conn(0x7f720408df80 0x7f720409f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:01:56.319 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:56.307+0000 7f7209eba700 1 --2- 172.21.15.42:0/2720205142 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f720409f830 0x7f720414efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:01:56.319 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:56.307+0000 7f71fbfff700 1 -- 172.21.15.42:0/2720205142 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1438192430 0 0) 0x7f720409fc60 con 0x7f72040a7740 2019-12-05T06:01:56.319 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:56.307+0000 7f71fbfff700 1 -- 172.21.15.42:0/2720205142 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71f00030b0 con 0x7f72040a7740 2019-12-05T06:01:56.320 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:56.307+0000 7f7209eba700 1 -- 172.21.15.42:0/2720205142 >> v2:172.21.15.40:3300/0 conn(0x7f720408df80 msgr2=0x7f720409f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:01:56.321 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:56.307+0000 7f7209eba700 1 --2- 172.21.15.42:0/2720205142 >> v2:172.21.15.40:3300/0 conn(0x7f720408df80 0x7f720409f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:01:56.321 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:56.307+0000 7f71fbfff700 1 -- 172.21.15.42:0/2720205142 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1775593284 0 0) 0x7f71f00030b0 con 0x7f72040a7740 2019-12-05T06:01:56.321 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:56.307+0000 7f7209eba700 1 -- 172.21.15.42:0/2720205142 >> v1:172.21.15.42:6789/0 conn(0x7f72040a7740 legacy=0x7f720407e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:01:56.321 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:56.307+0000 7f7209eba700 1 -- 172.21.15.42:0/2720205142 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72040a0710 con 0x7f720409f830 2019-12-05T06:01:56.321 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:56.307+0000 7f7208eb8700 1 --2- 172.21.15.42:0/2720205142 >> v2:172.21.15.40:3300/0 conn(0x7f720408df80 0x7f720409f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:01:56.322 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:56.307+0000 7f7209eba700 1 --2- 172.21.15.42:0/2720205142 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f720409f830 0x7f720414efe0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f71fc008790 tx=0x7f7200001820).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:56.322 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:56.307+0000 7f7209eba700 1 --2- 172.21.15.42:0/2720205142 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f720409f830 0x7f720414efe0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f71fc008790 tx=0x7f7200001820).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:56.322 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:56.307+0000 7f7209eba700 1 --2- 172.21.15.42:0/2720205142 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f720409f830 0x7f720414efe0 secure :-1 s=READY pgs=432 cs=0 l=1 rx=0x7f71fc008790 tx=0x7f7200001820).ready entity=mon.2 client_cookie=129d2187c9949e0f server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:01:56.322 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:56.307+0000 7f7209eba700 1 --2- 172.21.15.42:0/2720205142 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f720409f830 0x7f720414efe0 secure :-1 s=THROTTLE_DONE pgs=432 cs=0 l=1 rx=0x7f71fc008790 tx=0x7f7200001820).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:56.322 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:56.307+0000 7f7209eba700 1 --2- 172.21.15.42:0/2720205142 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f720409f830 0x7f720414efe0 secure :-1 s=THROTTLE_DONE pgs=432 cs=0 l=1 rx=0x7f71fc008790 tx=0x7f7200001820).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:56.322 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:56.307+0000 7f71fbfff700 1 -- 172.21.15.42:0/2720205142 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7200001ad0 con 0x7f720409f830 2019-12-05T06:01:56.323 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:56.307+0000 7f7209eba700 1 --2- 172.21.15.42:0/2720205142 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f720409f830 0x7f720414efe0 secure :-1 s=THROTTLE_DONE pgs=432 cs=0 l=1 rx=0x7f71fc008790 tx=0x7f7200001820).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:56.323 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:56.307+0000 7f71fbfff700 1 -- 172.21.15.42:0/2720205142 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72000018e0 con 0x7f720409f830 2019-12-05T06:01:56.323 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:56.307+0000 7f71fbfff700 1 -- 172.21.15.42:0/2720205142 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f71fc008da0 con 0x7f720409f830 2019-12-05T06:01:56.323 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:56.307+0000 7f720b11c700 1 -- 172.21.15.42:0/2720205142 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f720409f830 msgr2=0x7f720414efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:01:56.323 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:56.307+0000 7f720b11c700 1 --2- 172.21.15.42:0/2720205142 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f720409f830 0x7f720414efe0 secure :-1 s=READY pgs=432 cs=0 l=1 rx=0x7f71fc008790 tx=0x7f7200001820).stop 2019-12-05T06:01:56.324 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:56.311+0000 7f720b11c700 1 -- 172.21.15.42:0/2720205142 shutdown_connections 2019-12-05T06:01:56.327 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:56.311+0000 7f720b11c700 1 --2- 172.21.15.42:0/2720205142 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f720409f830 0x7f720414efe0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:01:56.327 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:56.311+0000 7f720b11c700 1 --2- 172.21.15.42:0/2720205142 >> v2:172.21.15.40:3300/0 conn(0x7f720408df80 0x7f720409f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:01:56.327 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:56.311+0000 7f720b11c700 1 -- 172.21.15.42:0/2720205142 shutdown_connections 2019-12-05T06:01:56.327 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:56.311+0000 7f720b11c700 1 -- 172.21.15.42:0/2720205142 wait complete. 2019-12-05T06:01:56.328 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:56.311+0000 7f720b11c700 1 -- 172.21.15.42:0/2720205142 >> 172.21.15.42:0/2720205142 conn(0x7f72040aabe0 msgr2=0x7f72040a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:01:56.367 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:56.311+0000 7f720b11c700 1 Processor -- start 2019-12-05T06:01:56.367 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:56.311+0000 7f720b11c700 1 -- start start 2019-12-05T06:01:56.367 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:56.315+0000 7f720b11c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f7204117dc0 0x7f7204118f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:01:56.367 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:56.315+0000 7f720b11c700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7204119760 con 0x7f720408eb70 2019-12-05T06:01:56.367 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:56.315+0000 7f720b11c700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f720408f7a0 con 0x7f7204116500 2019-12-05T06:01:56.367 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:56.315+0000 7f720b11c700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72041194d0 con 0x7f7204117dc0 2019-12-05T06:01:56.368 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:56.315+0000 7f7208eb8700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f7204117dc0 0x7f7204118f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:01:56.368 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:56.315+0000 7f7208eb8700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f7204117dc0 0x7f7204118f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:46778/0 (socket says 172.21.15.42:46778) 2019-12-05T06:01:56.368 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:56.315+0000 7f7208eb8700 1 -- 172.21.15.42:0/384445973 learned_addr learned my addr 172.21.15.42:0/384445973 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:01:56.368 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:56.315+0000 7f7208eb8700 1 -- 172.21.15.42:0/384445973 >> v1:172.21.15.42:6789/0 conn(0x7f7204116500 legacy=0x7f72041175c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:01:56.368 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:56.315+0000 7f7208eb8700 1 -- 172.21.15.42:0/384445973 >> v1:172.21.15.40:6789/0 conn(0x7f720408eb70 legacy=0x7f720408efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:01:56.368 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:56.315+0000 7f7208eb8700 1 -- 172.21.15.42:0/384445973 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7204133c00 con 0x7f7204117dc0 2019-12-05T06:01:56.369 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:56.315+0000 7f7208eb8700 1 --2- 172.21.15.42:0/384445973 >> v2:172.21.15.42:3300/0 conn(0x7f7204117dc0 0x7f7204118f80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f71f4002ef0 tx=0x7f71f4010d30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:56.369 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:56.315+0000 7f7208eb8700 1 --2- 172.21.15.42:0/384445973 >> v2:172.21.15.42:3300/0 conn(0x7f7204117dc0 0x7f7204118f80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f71f4002ef0 tx=0x7f71f4010d30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:56.369 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:56.315+0000 7f7208eb8700 1 --2- 172.21.15.42:0/384445973 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7204117dc0 0x7f7204118f80 secure :-1 s=READY pgs=1948 cs=0 l=1 rx=0x7f71f4002ef0 tx=0x7f71f4010d30).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:01:56.369 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:56.315+0000 7f7208eb8700 1 --2- 172.21.15.42:0/384445973 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7204117dc0 0x7f7204118f80 secure :-1 s=THROTTLE_DONE pgs=1948 cs=0 l=1 rx=0x7f71f4002ef0 tx=0x7f71f4010d30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:56.369 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:56.315+0000 7f7208eb8700 1 --2- 172.21.15.42:0/384445973 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7204117dc0 0x7f7204118f80 secure :-1 s=THROTTLE_DONE pgs=1948 cs=0 l=1 rx=0x7f71f4002ef0 tx=0x7f71f4010d30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:56.369 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:56.315+0000 7f7208eb8700 1 --2- 172.21.15.42:0/384445973 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7204117dc0 0x7f7204118f80 secure :-1 s=THROTTLE_DONE pgs=1948 cs=0 l=1 rx=0x7f71f4002ef0 tx=0x7f71f4010d30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:56.369 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:56.315+0000 7f71f9ffb700 1 -- 172.21.15.42:0/384445973 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f71f4005930 con 0x7f7204117dc0 2019-12-05T06:01:56.370 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:56.315+0000 7f71f9ffb700 1 -- 172.21.15.42:0/384445973 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71f4007160 con 0x7f7204117dc0 2019-12-05T06:01:56.370 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:56.315+0000 7f720b11c700 1 -- 172.21.15.42:0/384445973 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7204133ed0 con 0x7f7204117dc0 2019-12-05T06:01:56.370 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:56.315+0000 7f71f9ffb700 1 -- 172.21.15.42:0/384445973 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f71f400e410 con 0x7f7204117dc0 2019-12-05T06:01:56.370 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:56.315+0000 7f720b11c700 1 -- 172.21.15.42:0/384445973 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7204134400 con 0x7f7204117dc0 2019-12-05T06:01:56.370 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:56.315+0000 7f7208eb8700 1 --2- 172.21.15.42:0/384445973 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7204117dc0 0x7f7204118f80 secure :-1 s=THROTTLE_DONE pgs=1948 cs=0 l=1 rx=0x7f71f4002ef0 tx=0x7f71f4010d30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:56.370 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:56.319+0000 7f71f9ffb700 1 -- 172.21.15.42:0/384445973 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f71f4010180 con 0x7f7204117dc0 2019-12-05T06:01:56.370 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:56.319+0000 7f7208eb8700 1 --2- 172.21.15.42:0/384445973 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7204117dc0 0x7f7204118f80 secure :-1 s=THROTTLE_DONE pgs=1948 cs=0 l=1 rx=0x7f71f4002ef0 tx=0x7f71f4010d30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:56.371 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:56.319+0000 7f71f9ffb700 1 --2- 172.21.15.42:0/384445973 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f71e403ad60 0x7f71e403d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:01:56.371 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:56.319+0000 7f71f9ffb700 1 -- 172.21.15.42:0/384445973 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9557+0+0 (secure 0 0 0) 0x7f71f404b430 con 0x7f7204117dc0 2019-12-05T06:01:56.371 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:56.319+0000 7f72096b9700 1 --2- 172.21.15.42:0/384445973 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f71e403ad60 0x7f71e403d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:01:56.371 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:56.319+0000 7f72096b9700 1 --2- 172.21.15.42:0/384445973 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f71e403ad60 0x7f71e403d270 crc :-1 s=READY pgs=824 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:01:56.371 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:56.319+0000 7f720b11c700 1 -- 172.21.15.42:0/384445973 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f720408f9d0 con 0x7f7204117dc0 2019-12-05T06:01:56.371 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:56.327+0000 7f7208eb8700 1 --2- 172.21.15.42:0/384445973 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7204117dc0 0x7f7204118f80 secure :-1 s=THROTTLE_DONE pgs=1948 cs=0 l=1 rx=0x7f71f4002ef0 tx=0x7f71f4010d30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:56.371 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:56.327+0000 7f71f9ffb700 1 -- 172.21.15.42:0/384445973 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f71f40324b0 con 0x7f7204117dc0 2019-12-05T06:01:56.500 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:56.495+0000 7f720b11c700 1 -- 172.21.15.42:0/384445973 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f720405d110 con 0x7f7204117dc0 2019-12-05T06:01:56.501 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:56.495+0000 7f7208eb8700 1 --2- 172.21.15.42:0/384445973 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7204117dc0 0x7f7204118f80 secure :-1 s=THROTTLE_DONE pgs=1948 cs=0 l=1 rx=0x7f71f4002ef0 tx=0x7f71f4010d30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:01:56.501 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:56.495+0000 7f71f9ffb700 1 -- 172.21.15.42:0/384445973 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 238 v238) v1 ==== 95+0+2109 (secure 0 0 0) 0x7f71f4032710 con 0x7f7204117dc0 2019-12-05T06:01:56.501 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 238 2019-12-05T06:01:56.502 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:01:56.502 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":238,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":9381,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/3789081777","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3789081777},{"type":"v1","addr":"172.21.15.40:6833","nonce":3789081777}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":238}],"filesystems":[{"mdsmap":{"epoch":237,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:01:44.401238+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":90,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9292},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9292":{"gid":9292,"name":"a","rank":0,"incarnation":234,"state":"up:active","state_seq":12,"addr":"172.21.15.42:6835/613535662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":613535662},{"type":"v1","addr":"172.21.15.42:6835","nonce":613535662}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:01:56.504 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:56.499+0000 7f720b11c700 1 -- 172.21.15.42:0/384445973 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f71e403ad60 msgr2=0x7f71e403d270 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:01:56.504 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:56.499+0000 7f720b11c700 1 --2- 172.21.15.42:0/384445973 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f71e403ad60 0x7f71e403d270 crc :-1 s=READY pgs=824 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:01:56.504 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:56.499+0000 7f720b11c700 1 -- 172.21.15.42:0/384445973 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7204117dc0 msgr2=0x7f7204118f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:01:56.504 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:56.499+0000 7f720b11c700 1 --2- 172.21.15.42:0/384445973 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7204117dc0 0x7f7204118f80 secure :-1 s=READY pgs=1948 cs=0 l=1 rx=0x7f71f4002ef0 tx=0x7f71f4010d30).stop 2019-12-05T06:01:56.504 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:56.499+0000 7f720b11c700 1 -- 172.21.15.42:0/384445973 shutdown_connections 2019-12-05T06:01:56.504 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:56.499+0000 7f720b11c700 1 --2- 172.21.15.42:0/384445973 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f71e403ad60 0x7f71e403d270 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:01:56.505 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:56.499+0000 7f720b11c700 1 --2- 172.21.15.42:0/384445973 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7204117dc0 0x7f7204118f80 unknown :-1 s=CLOSED pgs=1948 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:01:56.505 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:56.499+0000 7f720b11c700 1 -- 172.21.15.42:0/384445973 shutdown_connections 2019-12-05T06:01:56.505 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:56.499+0000 7f720b11c700 1 -- 172.21.15.42:0/384445973 wait complete. 2019-12-05T06:01:56.505 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:01:56.499+0000 7f720b11c700 1 -- 172.21.15.42:0/384445973 >> 172.21.15.42:0/384445973 conn(0x7f72040a9790 msgr2=0x7f7204059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:01:56.514 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b reported in up:standby state 2019-12-05T06:01:56.514 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 114 secs before thrashing 2019-12-05T06:02:13.232 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:02:13.235 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:02:43.318 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:02:43.322 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:03:13.369 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:03:13.380 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:03:43.419 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:03:43.428 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:03:50.516 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:03:50.628 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:50.622+0000 7fa249205700 1 Processor -- start 2019-12-05T06:03:50.628 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:50.622+0000 7fa249205700 1 -- start start 2019-12-05T06:03:50.628 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:50.622+0000 7fa249205700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa2440a7740 0x7fa24408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:03:50.628 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:50.622+0000 7fa249205700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fa24409fae0 0x7fa24407eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:03:50.629 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:50.622+0000 7fa249205700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2440a0710 con 0x7fa24407e720 2019-12-05T06:03:50.629 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:50.622+0000 7fa249205700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2440a0b00 con 0x7fa2440a7740 2019-12-05T06:03:50.629 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:50.622+0000 7fa249205700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa24408e5b0 con 0x7fa24409fae0 2019-12-05T06:03:50.629 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:50.622+0000 7fa248a04700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fa24409fae0 0x7fa24407eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:03:50.629 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:50.622+0000 7fa248a04700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fa24409fae0 0x7fa24407eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:47330/0 (socket says 172.21.15.42:47330) 2019-12-05T06:03:50.630 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:50.622+0000 7fa248a04700 1 -- 172.21.15.42:0/3204700742 learned_addr learned my addr 172.21.15.42:0/3204700742 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:03:50.630 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:50.622+0000 7fa24359e700 1 --2- 172.21.15.42:0/3204700742 >> v2:172.21.15.40:3300/0 conn(0x7fa2440a7740 0x7fa24408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:03:50.630 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:50.622+0000 7fa248a04700 1 -- 172.21.15.42:0/3204700742 >> v2:172.21.15.40:3300/0 conn(0x7fa2440a7740 msgr2=0x7fa24408df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:03:50.630 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:50.622+0000 7fa248a04700 1 --2- 172.21.15.42:0/3204700742 >> v2:172.21.15.40:3300/0 conn(0x7fa2440a7740 0x7fa24408df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:03:50.630 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:50.622+0000 7fa248a04700 1 -- 172.21.15.42:0/3204700742 >> v1:172.21.15.40:6789/0 conn(0x7fa24407e720 legacy=0x7fa24409f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:03:50.630 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:50.622+0000 7fa248a04700 1 -- 172.21.15.42:0/3204700742 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa24407f0a0 con 0x7fa24409fae0 2019-12-05T06:03:50.630 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:50.622+0000 7fa248a04700 1 --2- 172.21.15.42:0/3204700742 >> v2:172.21.15.42:3300/0 conn(0x7fa24409fae0 0x7fa24407eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fa23c0032b0 tx=0x7fa23c0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:03:50.631 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:50.622+0000 7fa248a04700 1 --2- 172.21.15.42:0/3204700742 >> v2:172.21.15.42:3300/0 conn(0x7fa24409fae0 0x7fa24407eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fa23c0032b0 tx=0x7fa23c0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:03:50.631 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:50.622+0000 7fa248a04700 1 --2- 172.21.15.42:0/3204700742 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa24409fae0 0x7fa24407eb50 secure :-1 s=READY pgs=1953 cs=0 l=1 rx=0x7fa23c0032b0 tx=0x7fa23c0100b0).ready entity=mon.1 client_cookie=a8949657c3e4f5e7 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:03:50.631 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:50.622+0000 7fa248a04700 1 --2- 172.21.15.42:0/3204700742 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa24409fae0 0x7fa24407eb50 secure :-1 s=THROTTLE_DONE pgs=1953 cs=0 l=1 rx=0x7fa23c0032b0 tx=0x7fa23c0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:03:50.632 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:50.622+0000 7fa24259c700 1 -- 172.21.15.42:0/3204700742 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa23c010d60 con 0x7fa24409fae0 2019-12-05T06:03:50.632 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:50.622+0000 7fa248a04700 1 --2- 172.21.15.42:0/3204700742 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa24409fae0 0x7fa24407eb50 secure :-1 s=THROTTLE_DONE pgs=1953 cs=0 l=1 rx=0x7fa23c0032b0 tx=0x7fa23c0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:03:50.632 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:50.622+0000 7fa248a04700 1 --2- 172.21.15.42:0/3204700742 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa24409fae0 0x7fa24407eb50 secure :-1 s=THROTTLE_DONE pgs=1953 cs=0 l=1 rx=0x7fa23c0032b0 tx=0x7fa23c0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:03:50.632 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:50.622+0000 7fa24259c700 1 -- 172.21.15.42:0/3204700742 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa23c010980 con 0x7fa24409fae0 2019-12-05T06:03:50.632 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:50.622+0000 7fa24259c700 1 -- 172.21.15.42:0/3204700742 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa23c0114a0 con 0x7fa24409fae0 2019-12-05T06:03:50.633 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:50.622+0000 7fa249205700 1 -- 172.21.15.42:0/3204700742 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa24409fae0 msgr2=0x7fa24407eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:03:50.633 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:50.622+0000 7fa249205700 1 --2- 172.21.15.42:0/3204700742 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa24409fae0 0x7fa24407eb50 secure :-1 s=READY pgs=1953 cs=0 l=1 rx=0x7fa23c0032b0 tx=0x7fa23c0100b0).stop 2019-12-05T06:03:50.633 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:50.622+0000 7fa249205700 1 -- 172.21.15.42:0/3204700742 shutdown_connections 2019-12-05T06:03:50.633 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:50.622+0000 7fa249205700 1 --2- 172.21.15.42:0/3204700742 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa24409fae0 0x7fa24407eb50 unknown :-1 s=CLOSED pgs=1953 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:03:50.633 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:50.622+0000 7fa249205700 1 --2- 172.21.15.42:0/3204700742 >> v2:172.21.15.40:3300/0 conn(0x7fa2440a7740 0x7fa24408df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:03:50.633 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:50.622+0000 7fa249205700 1 -- 172.21.15.42:0/3204700742 shutdown_connections 2019-12-05T06:03:50.633 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:50.626+0000 7fa249205700 1 -- 172.21.15.42:0/3204700742 wait complete. 2019-12-05T06:03:50.634 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:50.626+0000 7fa249205700 1 -- 172.21.15.42:0/3204700742 >> 172.21.15.42:0/3204700742 conn(0x7fa2440aabe0 msgr2=0x7fa2440a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:03:50.638 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:50.626+0000 7fa249205700 1 Processor -- start 2019-12-05T06:03:50.638 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:50.626+0000 7fa249205700 1 -- start start 2019-12-05T06:03:50.639 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:50.626+0000 7fa249205700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fa2440a15c0 0x7fa244123810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:03:50.639 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:50.626+0000 7fa249205700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa244144f80 con 0x7fa2440a2910 2019-12-05T06:03:50.639 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:50.626+0000 7fa249205700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa244145400 con 0x7fa244123d60 2019-12-05T06:03:50.640 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:50.626+0000 7fa249205700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa24414ebb0 con 0x7fa2440a15c0 2019-12-05T06:03:50.640 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:50.626+0000 7fa24359e700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fa2440a2910 0x7fa244144780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:56810/0 (socket says 172.21.15.42:56810) 2019-12-05T06:03:50.640 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:50.626+0000 7fa24359e700 1 -- 172.21.15.42:0/3024375951 learned_addr learned my addr 172.21.15.42:0/3024375951 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:03:50.640 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:50.626+0000 7fa242d9d700 1 --2- 172.21.15.42:0/3024375951 >> v2:172.21.15.42:3300/0 conn(0x7fa2440a15c0 0x7fa244123810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:03:50.640 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:50.630+0000 7fa22bfff700 1 -- 172.21.15.42:0/3024375951 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3347413771 0 0) 0x7fa244144f80 con 0x7fa2440a2910 2019-12-05T06:03:50.641 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:50.630+0000 7fa22bfff700 1 -- 172.21.15.42:0/3024375951 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2240030b0 con 0x7fa2440a2910 2019-12-05T06:03:50.641 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:50.630+0000 7fa242d9d700 1 -- 172.21.15.42:0/3024375951 >> v1:172.21.15.42:6789/0 conn(0x7fa244123d60 legacy=0x7fa24412e220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:03:50.641 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:50.630+0000 7fa242d9d700 1 -- 172.21.15.42:0/3024375951 >> v1:172.21.15.40:6789/0 conn(0x7fa2440a2910 legacy=0x7fa244144780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:03:50.641 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:50.630+0000 7fa242d9d700 1 -- 172.21.15.42:0/3024375951 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2440a4060 con 0x7fa2440a15c0 2019-12-05T06:03:50.641 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:50.630+0000 7fa242d9d700 1 --2- 172.21.15.42:0/3024375951 >> v2:172.21.15.42:3300/0 conn(0x7fa2440a15c0 0x7fa244123810 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fa238001430 tx=0x7fa23800c320).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:03:50.641 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:50.630+0000 7fa242d9d700 1 --2- 172.21.15.42:0/3024375951 >> v2:172.21.15.42:3300/0 conn(0x7fa2440a15c0 0x7fa244123810 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fa238001430 tx=0x7fa23800c320).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:03:50.641 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:50.630+0000 7fa242d9d700 1 --2- 172.21.15.42:0/3024375951 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa2440a15c0 0x7fa244123810 secure :-1 s=READY pgs=1955 cs=0 l=1 rx=0x7fa238001430 tx=0x7fa23800c320).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:03:50.642 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:50.630+0000 7fa242d9d700 1 --2- 172.21.15.42:0/3024375951 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa2440a15c0 0x7fa244123810 secure :-1 s=THROTTLE_DONE pgs=1955 cs=0 l=1 rx=0x7fa238001430 tx=0x7fa23800c320).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:03:50.642 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:50.630+0000 7fa22bfff700 1 -- 172.21.15.42:0/3024375951 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa23800cb70 con 0x7fa2440a15c0 2019-12-05T06:03:50.642 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:50.630+0000 7fa242d9d700 1 --2- 172.21.15.42:0/3024375951 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa2440a15c0 0x7fa244123810 secure :-1 s=THROTTLE_DONE pgs=1955 cs=0 l=1 rx=0x7fa238001430 tx=0x7fa23800c320).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:03:50.643 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:50.630+0000 7fa242d9d700 1 --2- 172.21.15.42:0/3024375951 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa2440a15c0 0x7fa244123810 secure :-1 s=THROTTLE_DONE pgs=1955 cs=0 l=1 rx=0x7fa238001430 tx=0x7fa23800c320).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:03:50.643 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:50.630+0000 7fa22bfff700 1 -- 172.21.15.42:0/3024375951 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa23800c790 con 0x7fa2440a15c0 2019-12-05T06:03:50.643 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:50.630+0000 7fa249205700 1 -- 172.21.15.42:0/3024375951 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa24408eb50 con 0x7fa2440a15c0 2019-12-05T06:03:50.643 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:50.630+0000 7fa22bfff700 1 -- 172.21.15.42:0/3024375951 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa238011030 con 0x7fa2440a15c0 2019-12-05T06:03:50.643 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:50.630+0000 7fa249205700 1 -- 172.21.15.42:0/3024375951 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa24408f100 con 0x7fa2440a15c0 2019-12-05T06:03:50.643 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:50.630+0000 7fa242d9d700 1 --2- 172.21.15.42:0/3024375951 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa2440a15c0 0x7fa244123810 secure :-1 s=THROTTLE_DONE pgs=1955 cs=0 l=1 rx=0x7fa238001430 tx=0x7fa23800c320).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:03:50.644 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:50.630+0000 7fa22bfff700 1 -- 172.21.15.42:0/3024375951 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fa2380112d0 con 0x7fa2440a15c0 2019-12-05T06:03:50.644 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:50.630+0000 7fa242d9d700 1 --2- 172.21.15.42:0/3024375951 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa2440a15c0 0x7fa244123810 secure :-1 s=THROTTLE_DONE pgs=1955 cs=0 l=1 rx=0x7fa238001430 tx=0x7fa23800c320).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:03:50.644 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:50.630+0000 7fa22bfff700 1 --2- 172.21.15.42:0/3024375951 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa22403b1e0 0x7fa22403d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:03:50.644 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:50.630+0000 7fa22bfff700 1 -- 172.21.15.42:0/3024375951 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9557+0+0 (secure 0 0 0) 0x7fa23804b230 con 0x7fa2440a15c0 2019-12-05T06:03:50.644 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:50.630+0000 7fa24359e700 1 --2- 172.21.15.42:0/3024375951 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa22403b1e0 0x7fa22403d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:03:50.644 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:50.634+0000 7fa24359e700 1 --2- 172.21.15.42:0/3024375951 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa22403b1e0 0x7fa22403d6f0 crc :-1 s=READY pgs=831 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:03:50.645 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:50.634+0000 7fa249205700 1 -- 172.21.15.42:0/3024375951 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa24405d110 con 0x7fa2440a15c0 2019-12-05T06:03:50.646 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:50.642+0000 7fa242d9d700 1 --2- 172.21.15.42:0/3024375951 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa2440a15c0 0x7fa244123810 secure :-1 s=THROTTLE_DONE pgs=1955 cs=0 l=1 rx=0x7fa238001430 tx=0x7fa23800c320).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:03:50.646 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:50.642+0000 7fa22bfff700 1 -- 172.21.15.42:0/3024375951 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fa238032290 con 0x7fa2440a15c0 2019-12-05T06:03:50.831 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:50.830+0000 7fa249205700 1 -- 172.21.15.42:0/3024375951 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa24405b8c0 con 0x7fa2440a15c0 2019-12-05T06:03:50.832 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:50.830+0000 7fa242d9d700 1 --2- 172.21.15.42:0/3024375951 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa2440a15c0 0x7fa244123810 secure :-1 s=THROTTLE_DONE pgs=1955 cs=0 l=1 rx=0x7fa238001430 tx=0x7fa23800c320).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:03:50.833 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:50.830+0000 7fa22bfff700 1 -- 172.21.15.42:0/3024375951 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 238 v238) v1 ==== 95+0+2109 (secure 0 0 0) 0x7fa2380324f0 con 0x7fa2440a15c0 2019-12-05T06:03:50.833 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 238 2019-12-05T06:03:50.833 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:03:50.833 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":238,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":9381,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/3789081777","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3789081777},{"type":"v1","addr":"172.21.15.40:6833","nonce":3789081777}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":238}],"filesystems":[{"mdsmap":{"epoch":237,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:01:44.401238+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":90,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9292},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9292":{"gid":9292,"name":"a","rank":0,"incarnation":234,"state":"up:active","state_seq":12,"addr":"172.21.15.42:6835/613535662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":613535662},{"type":"v1","addr":"172.21.15.42:6835","nonce":613535662}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:03:50.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:50.830+0000 7fa249205700 1 -- 172.21.15.42:0/3024375951 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa22403b1e0 msgr2=0x7fa22403d6f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:03:50.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:50.830+0000 7fa249205700 1 --2- 172.21.15.42:0/3024375951 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa22403b1e0 0x7fa22403d6f0 crc :-1 s=READY pgs=831 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:03:50.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:50.830+0000 7fa249205700 1 -- 172.21.15.42:0/3024375951 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa2440a15c0 msgr2=0x7fa244123810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:03:50.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:50.830+0000 7fa249205700 1 --2- 172.21.15.42:0/3024375951 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa2440a15c0 0x7fa244123810 secure :-1 s=READY pgs=1955 cs=0 l=1 rx=0x7fa238001430 tx=0x7fa23800c320).stop 2019-12-05T06:03:50.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:50.830+0000 7fa249205700 1 -- 172.21.15.42:0/3024375951 shutdown_connections 2019-12-05T06:03:50.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:50.830+0000 7fa249205700 1 --2- 172.21.15.42:0/3024375951 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa22403b1e0 0x7fa22403d6f0 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:03:50.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:50.830+0000 7fa249205700 1 --2- 172.21.15.42:0/3024375951 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa2440a15c0 0x7fa244123810 unknown :-1 s=CLOSED pgs=1955 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:03:50.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:50.830+0000 7fa249205700 1 -- 172.21.15.42:0/3024375951 shutdown_connections 2019-12-05T06:03:50.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:50.834+0000 7fa249205700 1 -- 172.21.15.42:0/3024375951 wait complete. 2019-12-05T06:03:50.837 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:50.834+0000 7fa249205700 1 -- 172.21.15.42:0/3024375951 >> 172.21.15.42:0/3024375951 conn(0x7fa2440a7f90 msgr2=0x7fa244116220 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:03:50.846 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.a (rank=0) 2019-12-05T06:03:50.847 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2019-12-05T06:03:50.847 INFO:teuthology.orchestra.run:waiting for 300 2019-12-05T06:03:50.856 INFO:tasks.ceph.mds.a:Stopped 2019-12-05T06:03:50.856 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:03:51.022 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:51.010+0000 7f9e154ee700 1 Processor -- start 2019-12-05T06:03:51.022 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:51.014+0000 7f9e154ee700 1 -- start start 2019-12-05T06:03:51.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:51.014+0000 7f9e154ee700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9e1008df80 0x7f9e1009f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:03:51.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:51.014+0000 7f9e154ee700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e1009fc60 con 0x7f9e1009f830 2019-12-05T06:03:51.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:51.014+0000 7f9e154ee700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e100a0710 con 0x7f9e100a7740 2019-12-05T06:03:51.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:51.014+0000 7f9e154ee700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e100a0b00 con 0x7f9e1008df80 2019-12-05T06:03:51.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:51.014+0000 7f9e0f7fe700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f9e100a7740 0x7f9e1007e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:47386/0 (socket says 172.21.15.42:47386) 2019-12-05T06:03:51.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:51.014+0000 7f9e0f7fe700 1 -- 172.21.15.42:0/3513554517 learned_addr learned my addr 172.21.15.42:0/3513554517 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:03:51.024 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:51.014+0000 7f9e0effd700 1 --2- 172.21.15.42:0/3513554517 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9e1008df80 0x7f9e1009f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:03:51.024 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:51.014+0000 7f9e0e7fc700 1 -- 172.21.15.42:0/3513554517 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 100637553 0 0) 0x7f9e100a0710 con 0x7f9e100a7740 2019-12-05T06:03:51.024 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:51.014+0000 7f9e0e7fc700 1 -- 172.21.15.42:0/3513554517 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9df80030b0 con 0x7f9e100a7740 2019-12-05T06:03:51.024 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:51.014+0000 7f9e0e7fc700 1 -- 172.21.15.42:0/3513554517 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1996181994 0 0) 0x7f9df80030b0 con 0x7f9e100a7740 2019-12-05T06:03:51.024 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:51.014+0000 7f9e0e7fc700 1 -- 172.21.15.42:0/3513554517 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9e1008df80 msgr2=0x7f9e1009f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:03:51.024 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:51.014+0000 7f9e0e7fc700 1 --2- 172.21.15.42:0/3513554517 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9e1008df80 0x7f9e1009f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:03:51.024 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:51.014+0000 7f9e0e7fc700 1 -- 172.21.15.42:0/3513554517 >> v1:172.21.15.40:6789/0 conn(0x7f9e1009f830 legacy=0x7f9e1014efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:03:51.025 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:51.014+0000 7f9e0e7fc700 1 -- 172.21.15.42:0/3513554517 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e1007ef20 con 0x7f9e100a7740 2019-12-05T06:03:51.025 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:51.014+0000 7f9e0e7fc700 1 -- 172.21.15.42:0/3513554517 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7f9e00001ad0 con 0x7f9e100a7740 2019-12-05T06:03:51.025 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:51.014+0000 7f9e0e7fc700 1 -- 172.21.15.42:0/3513554517 >> v1:172.21.15.42:6789/0 conn(0x7f9e100a7740 legacy=0x7f9e1007e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:03:51.025 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:51.014+0000 7f9e0e7fc700 1 --2- 172.21.15.42:0/3513554517 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9df8003710 0x7f9df8003b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:03:51.025 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:51.014+0000 7f9e0e7fc700 1 --2- 172.21.15.42:0/3513554517 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9df8004420 0x7f9df8004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:03:51.025 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:51.014+0000 7f9e0e7fc700 1 --2- 172.21.15.42:0/3513554517 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9df8004da0 0x7f9df800b670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:03:51.025 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:51.014+0000 7f9e0e7fc700 1 -- 172.21.15.42:0/3513554517 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e100a0b00 con 0x7f9df8004da0 2019-12-05T06:03:51.026 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:51.014+0000 7f9e0f7fe700 1 --2- 172.21.15.42:0/3513554517 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9df8003710 0x7f9df8003b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:03:51.026 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:51.014+0000 7f9e0e7fc700 1 -- 172.21.15.42:0/3513554517 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9df80051d0 con 0x7f9df8003710 2019-12-05T06:03:51.026 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:51.014+0000 7f9e0e7fc700 1 -- 172.21.15.42:0/3513554517 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9df804fbd0 con 0x7f9df8004420 2019-12-05T06:03:51.026 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:51.014+0000 7f9e0effd700 1 --2- 172.21.15.42:0/3513554517 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9df8004420 0x7f9df8004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:03:51.026 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:51.014+0000 7f9e0ffff700 1 --2- 172.21.15.42:0/3513554517 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9df8004da0 0x7f9df800b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:03:51.026 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:51.014+0000 7f9e0f7fe700 1 -- 172.21.15.42:0/3513554517 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9df8004420 msgr2=0x7f9df8004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:03:51.026 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:51.014+0000 7f9e0f7fe700 1 --2- 172.21.15.42:0/3513554517 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9df8004420 0x7f9df8004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:03:51.027 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:51.014+0000 7f9e0f7fe700 1 -- 172.21.15.42:0/3513554517 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9df8004da0 msgr2=0x7f9df800b670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:03:51.027 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:51.014+0000 7f9e0f7fe700 1 --2- 172.21.15.42:0/3513554517 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9df8004da0 0x7f9df800b670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:03:51.027 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:51.018+0000 7f9e0f7fe700 1 -- 172.21.15.42:0/3513554517 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9df804fed0 con 0x7f9df8003710 2019-12-05T06:03:51.027 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:51.018+0000 7f9e0ffff700 1 --2- 172.21.15.42:0/3513554517 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9df8004da0 0x7f9df800b670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:03:51.027 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:51.018+0000 7f9e0f7fe700 1 --2- 172.21.15.42:0/3513554517 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9df8003710 0x7f9df8003b80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f9e000017e0 tx=0x7f9e0000c560).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:03:51.027 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:51.018+0000 7f9e0f7fe700 1 --2- 172.21.15.42:0/3513554517 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9df8003710 0x7f9df8003b80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f9e000017e0 tx=0x7f9e0000c560).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:03:51.028 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:51.018+0000 7f9e0f7fe700 1 --2- 172.21.15.42:0/3513554517 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9df8003710 0x7f9df8003b80 secure :-1 s=READY pgs=1957 cs=0 l=1 rx=0x7f9e000017e0 tx=0x7f9e0000c560).ready entity=mon.1 client_cookie=faa16139876a5a57 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:03:51.028 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:51.018+0000 7f9e0f7fe700 1 --2- 172.21.15.42:0/3513554517 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9df8003710 0x7f9df8003b80 secure :-1 s=THROTTLE_DONE pgs=1957 cs=0 l=1 rx=0x7f9e000017e0 tx=0x7f9e0000c560).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:03:51.028 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:51.018+0000 7f9e0e7fc700 1 -- 172.21.15.42:0/3513554517 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9e0000b2b0 con 0x7f9df8003710 2019-12-05T06:03:51.028 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:51.018+0000 7f9e0f7fe700 1 --2- 172.21.15.42:0/3513554517 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9df8003710 0x7f9df8003b80 secure :-1 s=THROTTLE_DONE pgs=1957 cs=0 l=1 rx=0x7f9e000017e0 tx=0x7f9e0000c560).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:03:51.028 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:51.018+0000 7f9e0f7fe700 1 --2- 172.21.15.42:0/3513554517 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9df8003710 0x7f9df8003b80 secure :-1 s=THROTTLE_DONE pgs=1957 cs=0 l=1 rx=0x7f9e000017e0 tx=0x7f9e0000c560).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:03:51.028 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:51.018+0000 7f9e0e7fc700 1 -- 172.21.15.42:0/3513554517 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e0000ac60 con 0x7f9df8003710 2019-12-05T06:03:51.028 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:51.018+0000 7f9e0e7fc700 1 -- 172.21.15.42:0/3513554517 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9e00010150 con 0x7f9df8003710 2019-12-05T06:03:51.029 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:51.018+0000 7f9e154ee700 1 -- 172.21.15.42:0/3513554517 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9df8003710 msgr2=0x7f9df8003b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:03:51.029 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:51.018+0000 7f9e154ee700 1 --2- 172.21.15.42:0/3513554517 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9df8003710 0x7f9df8003b80 secure :-1 s=READY pgs=1957 cs=0 l=1 rx=0x7f9e000017e0 tx=0x7f9e0000c560).stop 2019-12-05T06:03:51.029 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:51.018+0000 7f9e0ffff700 1 -- 172.21.15.42:0/3513554517 reap_dead start 2019-12-05T06:03:51.029 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:51.018+0000 7f9e154ee700 1 -- 172.21.15.42:0/3513554517 shutdown_connections 2019-12-05T06:03:51.029 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:51.018+0000 7f9e154ee700 1 -- 172.21.15.42:0/3513554517 shutdown_connections 2019-12-05T06:03:51.029 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:51.018+0000 7f9e154ee700 1 -- 172.21.15.42:0/3513554517 wait complete. 2019-12-05T06:03:51.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:51.018+0000 7f9e154ee700 1 -- 172.21.15.42:0/3513554517 >> 172.21.15.42:0/3513554517 conn(0x7f9e100aabe0 msgr2=0x7f9e100a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:03:51.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:51.022+0000 7f9e154ee700 1 Processor -- start 2019-12-05T06:03:51.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:51.022+0000 7f9e154ee700 1 -- start start 2019-12-05T06:03:51.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:51.022+0000 7f9e154ee700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9e10078e80 0x7f9e10077d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:03:51.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:51.022+0000 7f9e154ee700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f9e100a3000 0x7f9e100a3430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:03:51.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:51.022+0000 7f9e154ee700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e100a3c40 con 0x7f9e1011c860 2019-12-05T06:03:51.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:51.022+0000 7f9e154ee700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e100a3980 con 0x7f9e100a3000 2019-12-05T06:03:51.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:51.022+0000 7f9e154ee700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e10078be0 con 0x7f9e10078e80 2019-12-05T06:03:51.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:51.022+0000 7f9e0f7fe700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f9e1011c860 0x7f9e10079e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:47398/0 (socket says 172.21.15.42:47398) 2019-12-05T06:03:51.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:51.022+0000 7f9e0f7fe700 1 -- 172.21.15.42:0/129702070 learned_addr learned my addr 172.21.15.42:0/129702070 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:03:51.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:51.022+0000 7f9e0ffff700 1 --2- 172.21.15.42:0/129702070 >> v2:172.21.15.42:3300/0 conn(0x7f9e100a3000 0x7f9e100a3430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:03:51.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:51.022+0000 7f9e0effd700 1 --2- 172.21.15.42:0/129702070 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9e10078e80 0x7f9e10077d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:03:51.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:51.022+0000 7f9df7fff700 1 -- 172.21.15.42:0/129702070 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3565137211 0 0) 0x7f9e100a3c40 con 0x7f9e1011c860 2019-12-05T06:03:51.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:51.022+0000 7f9df7fff700 1 -- 172.21.15.42:0/129702070 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9df00030b0 con 0x7f9e1011c860 2019-12-05T06:03:51.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:51.022+0000 7f9e0ffff700 1 -- 172.21.15.42:0/129702070 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9e10078e80 msgr2=0x7f9e10077d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:03:51.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:51.022+0000 7f9e0ffff700 1 --2- 172.21.15.42:0/129702070 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9e10078e80 0x7f9e10077d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:03:51.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:51.022+0000 7f9e0ffff700 1 -- 172.21.15.42:0/129702070 >> v1:172.21.15.42:6789/0 conn(0x7f9e1011c860 legacy=0x7f9e10079e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:03:51.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:51.022+0000 7f9e0ffff700 1 -- 172.21.15.42:0/129702070 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e10078870 con 0x7f9e100a3000 2019-12-05T06:03:51.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:51.022+0000 7f9e0ffff700 1 --2- 172.21.15.42:0/129702070 >> v2:172.21.15.42:3300/0 conn(0x7f9e100a3000 0x7f9e100a3430 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f9e100a14d0 tx=0x7f9e080108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:03:51.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:51.022+0000 7f9e0ffff700 1 --2- 172.21.15.42:0/129702070 >> v2:172.21.15.42:3300/0 conn(0x7f9e100a3000 0x7f9e100a3430 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f9e100a14d0 tx=0x7f9e080108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:03:51.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:51.022+0000 7f9e0ffff700 1 --2- 172.21.15.42:0/129702070 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9e100a3000 0x7f9e100a3430 secure :-1 s=READY pgs=1959 cs=0 l=1 rx=0x7f9e100a14d0 tx=0x7f9e080108f0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:03:51.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:51.022+0000 7f9e0ffff700 1 --2- 172.21.15.42:0/129702070 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9e100a3000 0x7f9e100a3430 secure :-1 s=THROTTLE_DONE pgs=1959 cs=0 l=1 rx=0x7f9e100a14d0 tx=0x7f9e080108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:03:51.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:51.022+0000 7f9df7fff700 1 -- 172.21.15.42:0/129702070 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9e0800c960 con 0x7f9e100a3000 2019-12-05T06:03:51.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:51.022+0000 7f9e0ffff700 1 --2- 172.21.15.42:0/129702070 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9e100a3000 0x7f9e100a3430 secure :-1 s=THROTTLE_DONE pgs=1959 cs=0 l=1 rx=0x7f9e100a14d0 tx=0x7f9e080108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:03:51.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:51.022+0000 7f9e154ee700 1 -- 172.21.15.42:0/129702070 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e100782d0 con 0x7f9e100a3000 2019-12-05T06:03:51.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:51.022+0000 7f9df7fff700 1 -- 172.21.15.42:0/129702070 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e0800c580 con 0x7f9e100a3000 2019-12-05T06:03:51.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:51.022+0000 7f9e0ffff700 1 --2- 172.21.15.42:0/129702070 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9e100a3000 0x7f9e100a3430 secure :-1 s=THROTTLE_DONE pgs=1959 cs=0 l=1 rx=0x7f9e100a14d0 tx=0x7f9e080108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:03:51.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:51.022+0000 7f9df7fff700 1 -- 172.21.15.42:0/129702070 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9e08010e80 con 0x7f9e100a3000 2019-12-05T06:03:51.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:51.022+0000 7f9e154ee700 1 -- 172.21.15.42:0/129702070 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e100a4cc0 con 0x7f9e100a3000 2019-12-05T06:03:51.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:51.026+0000 7f9e0ffff700 1 --2- 172.21.15.42:0/129702070 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9e100a3000 0x7f9e100a3430 secure :-1 s=THROTTLE_DONE pgs=1959 cs=0 l=1 rx=0x7f9e100a14d0 tx=0x7f9e080108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:03:51.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:51.026+0000 7f9df7fff700 1 -- 172.21.15.42:0/129702070 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f9e08007430 con 0x7f9e100a3000 2019-12-05T06:03:51.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:51.026+0000 7f9e0ffff700 1 --2- 172.21.15.42:0/129702070 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9e100a3000 0x7f9e100a3430 secure :-1 s=THROTTLE_DONE pgs=1959 cs=0 l=1 rx=0x7f9e100a14d0 tx=0x7f9e080108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:03:51.039 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:51.026+0000 7f9df7fff700 1 --2- 172.21.15.42:0/129702070 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9df003b2b0 0x7f9df003d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:03:51.039 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:51.026+0000 7f9df7fff700 1 -- 172.21.15.42:0/129702070 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9557+0+0 (secure 0 0 0) 0x7f9e0804af50 con 0x7f9e100a3000 2019-12-05T06:03:51.039 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:51.026+0000 7f9e0f7fe700 1 --2- 172.21.15.42:0/129702070 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9df003b2b0 0x7f9df003d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:03:51.039 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:51.026+0000 7f9e0f7fe700 1 --2- 172.21.15.42:0/129702070 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9df003b2b0 0x7f9df003d7c0 crc :-1 s=READY pgs=832 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:03:51.039 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:51.026+0000 7f9e154ee700 1 -- 172.21.15.42:0/129702070 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e1005d110 con 0x7f9e100a3000 2019-12-05T06:03:51.040 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:51.034+0000 7f9e0ffff700 1 --2- 172.21.15.42:0/129702070 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9e100a3000 0x7f9e100a3430 secure :-1 s=THROTTLE_DONE pgs=1959 cs=0 l=1 rx=0x7f9e100a14d0 tx=0x7f9e080108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:03:51.040 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:51.034+0000 7f9df7fff700 1 -- 172.21.15.42:0/129702070 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f9e08031ef0 con 0x7f9e100a3000 2019-12-05T06:03:51.207 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:51.202+0000 7f9e154ee700 1 -- 172.21.15.42:0/129702070 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9e1005b8c0 con 0x7f9e100a3000 2019-12-05T06:03:51.208 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:51.206+0000 7f9e0ffff700 1 --2- 172.21.15.42:0/129702070 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9e100a3000 0x7f9e100a3430 secure :-1 s=THROTTLE_DONE pgs=1959 cs=0 l=1 rx=0x7f9e100a14d0 tx=0x7f9e080108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:03:51.208 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:51.206+0000 7f9df7fff700 1 -- 172.21.15.42:0/129702070 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 238 v238) v1 ==== 95+0+2109 (secure 0 0 0) 0x7f9e08032150 con 0x7f9e100a3000 2019-12-05T06:03:51.209 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:03:51.210 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":238,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":9381,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/3789081777","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3789081777},{"type":"v1","addr":"172.21.15.40:6833","nonce":3789081777}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":238}],"filesystems":[{"mdsmap":{"epoch":237,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:01:44.401238+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":90,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9292},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9292":{"gid":9292,"name":"a","rank":0,"incarnation":234,"state":"up:active","state_seq":12,"addr":"172.21.15.42:6835/613535662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":613535662},{"type":"v1","addr":"172.21.15.42:6835","nonce":613535662}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:03:51.210 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 238 2019-12-05T06:03:51.212 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:51.210+0000 7f9e154ee700 1 -- 172.21.15.42:0/129702070 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9df003b2b0 msgr2=0x7f9df003d7c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:03:51.212 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:51.210+0000 7f9e154ee700 1 --2- 172.21.15.42:0/129702070 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9df003b2b0 0x7f9df003d7c0 crc :-1 s=READY pgs=832 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:03:51.213 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:51.210+0000 7f9e154ee700 1 -- 172.21.15.42:0/129702070 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9e100a3000 msgr2=0x7f9e100a3430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:03:51.213 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:51.210+0000 7f9e154ee700 1 --2- 172.21.15.42:0/129702070 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9e100a3000 0x7f9e100a3430 secure :-1 s=READY pgs=1959 cs=0 l=1 rx=0x7f9e100a14d0 tx=0x7f9e080108f0).stop 2019-12-05T06:03:51.213 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:51.210+0000 7f9e154ee700 1 -- 172.21.15.42:0/129702070 shutdown_connections 2019-12-05T06:03:51.213 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:51.210+0000 7f9e154ee700 1 --2- 172.21.15.42:0/129702070 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9df003b2b0 0x7f9df003d7c0 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:03:51.213 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:51.210+0000 7f9e154ee700 1 --2- 172.21.15.42:0/129702070 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9e100a3000 0x7f9e100a3430 unknown :-1 s=CLOSED pgs=1959 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:03:51.213 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:51.210+0000 7f9e154ee700 1 --2- 172.21.15.42:0/129702070 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9e10078e80 0x7f9e10077d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:03:51.213 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:51.210+0000 7f9e154ee700 1 -- 172.21.15.42:0/129702070 shutdown_connections 2019-12-05T06:03:51.214 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:51.210+0000 7f9e154ee700 1 -- 172.21.15.42:0/129702070 wait complete. 2019-12-05T06:03:51.214 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:51.210+0000 7f9e154ee700 1 -- 172.21.15.42:0/129702070 >> 172.21.15.42:0/129702070 conn(0x7f9e100a9640 msgr2=0x7f9e1009f240 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:03:51.222 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:03:53.223 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:03:53.327 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:53.318+0000 7f24799bd700 1 Processor -- start 2019-12-05T06:03:53.327 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:53.318+0000 7f24799bd700 1 -- start start 2019-12-05T06:03:53.327 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:53.318+0000 7f24799bd700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f24740a7740 0x7f247408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:03:53.328 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:53.318+0000 7f24799bd700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f247407e720 0x7f247407eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:03:53.328 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:53.318+0000 7f24799bd700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f247409f2e0 0x7f247409f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:03:53.328 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:53.322+0000 7f24799bd700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24740a0b00 con 0x7f247407e720 2019-12-05T06:03:53.328 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:53.322+0000 7f24799bd700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f247408e5b0 con 0x7f24740a7740 2019-12-05T06:03:53.328 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:53.322+0000 7f24799bd700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f247408e850 con 0x7f247409f2e0 2019-12-05T06:03:53.328 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:53.322+0000 7f24737fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f24740a7740 0x7f247408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:03:53.329 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:53.322+0000 7f2472ffd700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f247407e720 0x7f247407eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:03:53.329 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:53.322+0000 7f24737fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f24740a7740 0x7f247408df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:47376/0 (socket says 172.21.15.42:47376) 2019-12-05T06:03:53.329 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:53.322+0000 7f24737fe700 1 -- 172.21.15.42:0/2897600956 learned_addr learned my addr 172.21.15.42:0/2897600956 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:03:53.329 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:53.322+0000 7f2473fff700 1 --2- 172.21.15.42:0/2897600956 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f247409f2e0 0x7f247409f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:03:53.329 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:53.322+0000 7f24737fe700 1 -- 172.21.15.42:0/2897600956 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f247409f2e0 msgr2=0x7f247409f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:03:53.329 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:53.322+0000 7f24737fe700 1 --2- 172.21.15.42:0/2897600956 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f247409f2e0 0x7f247409f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:03:53.329 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:53.322+0000 7f24737fe700 1 -- 172.21.15.42:0/2897600956 >> v2:172.21.15.40:3300/0 conn(0x7f247407e720 msgr2=0x7f247407eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:03:53.330 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:53.322+0000 7f24737fe700 1 --2- 172.21.15.42:0/2897600956 >> v2:172.21.15.40:3300/0 conn(0x7f247407e720 0x7f247407eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:03:53.330 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:53.322+0000 7f24737fe700 1 -- 172.21.15.42:0/2897600956 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f247407f0a0 con 0x7f24740a7740 2019-12-05T06:03:53.330 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:53.322+0000 7f24737fe700 1 --2- 172.21.15.42:0/2897600956 >> v2:172.21.15.42:3300/0 conn(0x7f24740a7740 0x7f247408df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f24740a0f50 tx=0x7f246800fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:03:53.330 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:53.322+0000 7f24737fe700 1 --2- 172.21.15.42:0/2897600956 >> v2:172.21.15.42:3300/0 conn(0x7f24740a7740 0x7f247408df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f24740a0f50 tx=0x7f246800fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:03:53.330 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:53.322+0000 7f24737fe700 1 --2- 172.21.15.42:0/2897600956 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f24740a7740 0x7f247408df80 secure :-1 s=READY pgs=1960 cs=0 l=1 rx=0x7f24740a0f50 tx=0x7f246800fdb0).ready entity=mon.1 client_cookie=d14d45078aac31d9 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:03:53.330 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:53.322+0000 7f24737fe700 1 --2- 172.21.15.42:0/2897600956 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f24740a7740 0x7f247408df80 secure :-1 s=THROTTLE_DONE pgs=1960 cs=0 l=1 rx=0x7f24740a0f50 tx=0x7f246800fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:03:53.330 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:53.322+0000 7f24727fc700 1 -- 172.21.15.42:0/2897600956 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2468008c90 con 0x7f24740a7740 2019-12-05T06:03:53.331 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:53.322+0000 7f24737fe700 1 --2- 172.21.15.42:0/2897600956 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f24740a7740 0x7f247408df80 secure :-1 s=THROTTLE_DONE pgs=1960 cs=0 l=1 rx=0x7f24740a0f50 tx=0x7f246800fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:03:53.331 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:53.322+0000 7f24737fe700 1 --2- 172.21.15.42:0/2897600956 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f24740a7740 0x7f247408df80 secure :-1 s=THROTTLE_DONE pgs=1960 cs=0 l=1 rx=0x7f24740a0f50 tx=0x7f246800fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:03:53.331 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:53.322+0000 7f24727fc700 1 -- 172.21.15.42:0/2897600956 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24680088b0 con 0x7f24740a7740 2019-12-05T06:03:53.331 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:53.322+0000 7f24727fc700 1 -- 172.21.15.42:0/2897600956 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2468011b50 con 0x7f24740a7740 2019-12-05T06:03:53.331 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:53.322+0000 7f24799bd700 1 -- 172.21.15.42:0/2897600956 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f24740a7740 msgr2=0x7f247408df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:03:53.331 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:53.322+0000 7f24799bd700 1 --2- 172.21.15.42:0/2897600956 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f24740a7740 0x7f247408df80 secure :-1 s=READY pgs=1960 cs=0 l=1 rx=0x7f24740a0f50 tx=0x7f246800fdb0).stop 2019-12-05T06:03:53.331 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:53.322+0000 7f24799bd700 1 -- 172.21.15.42:0/2897600956 shutdown_connections 2019-12-05T06:03:53.332 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:53.322+0000 7f24799bd700 1 --2- 172.21.15.42:0/2897600956 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f247409f2e0 0x7f247409f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:03:53.332 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:53.322+0000 7f24799bd700 1 --2- 172.21.15.42:0/2897600956 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f24740a7740 0x7f247408df80 unknown :-1 s=CLOSED pgs=1960 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:03:53.332 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:53.322+0000 7f24799bd700 1 --2- 172.21.15.42:0/2897600956 >> v2:172.21.15.40:3300/0 conn(0x7f247407e720 0x7f247407eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:03:53.332 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:53.322+0000 7f24799bd700 1 -- 172.21.15.42:0/2897600956 shutdown_connections 2019-12-05T06:03:53.333 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:53.322+0000 7f24799bd700 1 -- 172.21.15.42:0/2897600956 wait complete. 2019-12-05T06:03:53.333 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:53.322+0000 7f24799bd700 1 -- 172.21.15.42:0/2897600956 >> 172.21.15.42:0/2897600956 conn(0x7f24740aabe0 msgr2=0x7f24740a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:03:53.337 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:53.326+0000 7f24799bd700 1 Processor -- start 2019-12-05T06:03:53.337 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:53.326+0000 7f24799bd700 1 -- start start 2019-12-05T06:03:53.337 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:53.326+0000 7f24799bd700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f24740a56a0 0x7f24740a5ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:03:53.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:53.326+0000 7f24799bd700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2474130130 0x7f2474130560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:03:53.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:53.326+0000 7f24799bd700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24740a6020 con 0x7f24741214e0 2019-12-05T06:03:53.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:53.326+0000 7f24799bd700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2474133870 con 0x7f2474130130 2019-12-05T06:03:53.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:53.326+0000 7f24799bd700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2474133b10 con 0x7f24740a56a0 2019-12-05T06:03:53.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:53.326+0000 7f24737fe700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f24741214e0 0x7f2474133db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:47418/0 (socket says 172.21.15.42:47418) 2019-12-05T06:03:53.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:53.326+0000 7f24737fe700 1 -- 172.21.15.42:0/4141626251 learned_addr learned my addr 172.21.15.42:0/4141626251 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:03:53.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:53.326+0000 7f2473fff700 1 --2- 172.21.15.42:0/4141626251 >> v2:172.21.15.42:3300/0 conn(0x7f2474130130 0x7f2474130560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:03:53.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:53.326+0000 7f2472ffd700 1 --2- 172.21.15.42:0/4141626251 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f24740a56a0 0x7f24740a5ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:03:53.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:53.330+0000 7f245bfff700 1 -- 172.21.15.42:0/4141626251 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 512270529 0 0) 0x7f24740a6020 con 0x7f24741214e0 2019-12-05T06:03:53.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:53.330+0000 7f245bfff700 1 -- 172.21.15.42:0/4141626251 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24540030b0 con 0x7f24741214e0 2019-12-05T06:03:53.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:53.330+0000 7f2472ffd700 1 -- 172.21.15.42:0/4141626251 >> v2:172.21.15.42:3300/0 conn(0x7f2474130130 msgr2=0x7f2474130560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:03:53.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:53.330+0000 7f2472ffd700 1 --2- 172.21.15.42:0/4141626251 >> v2:172.21.15.42:3300/0 conn(0x7f2474130130 0x7f2474130560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:03:53.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:53.330+0000 7f2472ffd700 1 -- 172.21.15.42:0/4141626251 >> v1:172.21.15.42:6789/0 conn(0x7f24741214e0 legacy=0x7f2474133db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:03:53.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:53.330+0000 7f2472ffd700 1 -- 172.21.15.42:0/4141626251 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2474130ab0 con 0x7f24740a56a0 2019-12-05T06:03:53.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:53.330+0000 7f2473fff700 1 --2- 172.21.15.42:0/4141626251 >> v2:172.21.15.42:3300/0 conn(0x7f2474130130 0x7f2474130560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:03:53.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:53.330+0000 7f2472ffd700 1 --2- 172.21.15.42:0/4141626251 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f24740a56a0 0x7f24740a5ad0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f247411ad50 tx=0x7f2464010b20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:03:53.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:53.330+0000 7f2472ffd700 1 --2- 172.21.15.42:0/4141626251 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f24740a56a0 0x7f24740a5ad0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f247411ad50 tx=0x7f2464010b20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:03:53.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:53.330+0000 7f2472ffd700 1 --2- 172.21.15.42:0/4141626251 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f24740a56a0 0x7f24740a5ad0 secure :-1 s=READY pgs=436 cs=0 l=1 rx=0x7f247411ad50 tx=0x7f2464010b20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:03:53.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:53.330+0000 7f2472ffd700 1 --2- 172.21.15.42:0/4141626251 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f24740a56a0 0x7f24740a5ad0 secure :-1 s=THROTTLE_DONE pgs=436 cs=0 l=1 rx=0x7f247411ad50 tx=0x7f2464010b20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:03:53.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:53.330+0000 7f245bfff700 1 -- 172.21.15.42:0/4141626251 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2464012970 con 0x7f24740a56a0 2019-12-05T06:03:53.341 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:53.330+0000 7f2472ffd700 1 --2- 172.21.15.42:0/4141626251 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f24740a56a0 0x7f24740a5ad0 secure :-1 s=THROTTLE_DONE pgs=436 cs=0 l=1 rx=0x7f247411ad50 tx=0x7f2464010b20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:03:53.341 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:53.330+0000 7f2472ffd700 1 --2- 172.21.15.42:0/4141626251 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f24740a56a0 0x7f24740a5ad0 secure :-1 s=THROTTLE_DONE pgs=436 cs=0 l=1 rx=0x7f247411ad50 tx=0x7f2464010b20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:03:53.341 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:53.330+0000 7f245bfff700 1 -- 172.21.15.42:0/4141626251 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2464012590 con 0x7f24740a56a0 2019-12-05T06:03:53.341 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:53.330+0000 7f24799bd700 1 -- 172.21.15.42:0/4141626251 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2474130da0 con 0x7f24740a56a0 2019-12-05T06:03:53.341 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:53.330+0000 7f245bfff700 1 -- 172.21.15.42:0/4141626251 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2464005730 con 0x7f24740a56a0 2019-12-05T06:03:53.341 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:53.330+0000 7f24799bd700 1 -- 172.21.15.42:0/4141626251 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2474077300 con 0x7f24740a56a0 2019-12-05T06:03:53.342 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:53.330+0000 7f2472ffd700 1 --2- 172.21.15.42:0/4141626251 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f24740a56a0 0x7f24740a5ad0 secure :-1 s=THROTTLE_DONE pgs=436 cs=0 l=1 rx=0x7f247411ad50 tx=0x7f2464010b20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:03:53.342 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:53.330+0000 7f245bfff700 1 -- 172.21.15.42:0/4141626251 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f2464007180 con 0x7f24740a56a0 2019-12-05T06:03:53.342 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:53.330+0000 7f2472ffd700 1 --2- 172.21.15.42:0/4141626251 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f24740a56a0 0x7f24740a5ad0 secure :-1 s=THROTTLE_DONE pgs=436 cs=0 l=1 rx=0x7f247411ad50 tx=0x7f2464010b20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:03:53.342 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:53.330+0000 7f245bfff700 1 --2- 172.21.15.42:0/4141626251 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f245403b260 0x7f245403d770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:03:53.342 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:53.330+0000 7f245bfff700 1 -- 172.21.15.42:0/4141626251 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9557+0+0 (secure 0 0 0) 0x7f246404af30 con 0x7f24740a56a0 2019-12-05T06:03:53.342 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:53.330+0000 7f24737fe700 1 --2- 172.21.15.42:0/4141626251 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f245403b260 0x7f245403d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:03:53.342 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:53.330+0000 7f24737fe700 1 -- 172.21.15.42:0/4141626251 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f245403b260 msgr2=0x7f245403d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2019-12-05T06:03:53.343 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:53.330+0000 7f24737fe700 1 -- 172.21.15.42:0/4141626251 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f245403b260 msgr2=0x7f245403d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2019-12-05T06:03:53.343 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:53.330+0000 7f24737fe700 1 --2- 172.21.15.42:0/4141626251 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f245403b260 0x7f245403d770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-12-05T06:03:53.343 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:53.334+0000 7f24737fe700 1 --2- 172.21.15.42:0/4141626251 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f245403b260 0x7f245403d770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rx=0 tx=0)._fault waiting 0.200000 2019-12-05T06:03:53.344 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:53.334+0000 7f24799bd700 1 -- 172.21.15.42:0/4141626251 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f247405d110 con 0x7f24740a56a0 2019-12-05T06:03:53.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:53.346+0000 7f2472ffd700 1 --2- 172.21.15.42:0/4141626251 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f24740a56a0 0x7f24740a5ad0 secure :-1 s=THROTTLE_DONE pgs=436 cs=0 l=1 rx=0x7f247411ad50 tx=0x7f2464010b20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:03:53.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:53.346+0000 7f245bfff700 1 -- 172.21.15.42:0/4141626251 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f2464031fb0 con 0x7f24740a56a0 2019-12-05T06:03:53.519 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:53.514+0000 7f24799bd700 1 -- 172.21.15.42:0/4141626251 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f247405b8c0 con 0x7f24740a56a0 2019-12-05T06:03:53.520 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:53.518+0000 7f2472ffd700 1 --2- 172.21.15.42:0/4141626251 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f24740a56a0 0x7f24740a5ad0 secure :-1 s=THROTTLE_DONE pgs=436 cs=0 l=1 rx=0x7f247411ad50 tx=0x7f2464010b20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:03:53.520 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:53.518+0000 7f245bfff700 1 -- 172.21.15.42:0/4141626251 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 238 v238) v1 ==== 95+0+2109 (secure 0 0 0) 0x7f2464032210 con 0x7f24740a56a0 2019-12-05T06:03:53.522 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:03:53.522 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":238,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":9381,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/3789081777","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3789081777},{"type":"v1","addr":"172.21.15.40:6833","nonce":3789081777}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":238}],"filesystems":[{"mdsmap":{"epoch":237,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:01:44.401238+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":90,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9292},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9292":{"gid":9292,"name":"a","rank":0,"incarnation":234,"state":"up:active","state_seq":12,"addr":"172.21.15.42:6835/613535662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":613535662},{"type":"v1","addr":"172.21.15.42:6835","nonce":613535662}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:03:53.523 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 238 2019-12-05T06:03:53.525 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:53.522+0000 7f24799bd700 1 -- 172.21.15.42:0/4141626251 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f245403b260 msgr2=0x7f245403d770 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2019-12-05T06:03:53.525 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:53.522+0000 7f24799bd700 1 --2- 172.21.15.42:0/4141626251 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f245403b260 0x7f245403d770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:03:53.525 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:53.522+0000 7f24799bd700 1 -- 172.21.15.42:0/4141626251 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f24740a56a0 msgr2=0x7f24740a5ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:03:53.525 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:53.522+0000 7f24799bd700 1 --2- 172.21.15.42:0/4141626251 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f24740a56a0 0x7f24740a5ad0 secure :-1 s=READY pgs=436 cs=0 l=1 rx=0x7f247411ad50 tx=0x7f2464010b20).stop 2019-12-05T06:03:53.525 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:53.522+0000 7f24799bd700 1 -- 172.21.15.42:0/4141626251 shutdown_connections 2019-12-05T06:03:53.526 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:53.522+0000 7f24799bd700 1 --2- 172.21.15.42:0/4141626251 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f245403b260 0x7f245403d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:03:53.526 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:53.522+0000 7f24799bd700 1 --2- 172.21.15.42:0/4141626251 >> v2:172.21.15.42:3300/0 conn(0x7f2474130130 0x7f2474130560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:03:53.527 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:53.522+0000 7f24799bd700 1 --2- 172.21.15.42:0/4141626251 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f24740a56a0 0x7f24740a5ad0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:03:53.527 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:53.522+0000 7f24799bd700 1 -- 172.21.15.42:0/4141626251 shutdown_connections 2019-12-05T06:03:53.528 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:53.522+0000 7f24799bd700 1 -- 172.21.15.42:0/4141626251 wait complete. 2019-12-05T06:03:53.528 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:53.522+0000 7f24799bd700 1 -- 172.21.15.42:0/4141626251 >> 172.21.15.42:0/4141626251 conn(0x7f24740aabe0 msgr2=0x7f24740a9790 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:03:53.536 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:03:55.536 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:03:55.641 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:55.630+0000 7f4e24806700 1 Processor -- start 2019-12-05T06:03:55.641 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:55.630+0000 7f4e24806700 1 -- start start 2019-12-05T06:03:55.641 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:55.634+0000 7f4e24806700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f4e200a7740 0x7f4e2008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:03:55.641 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:55.634+0000 7f4e24806700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4e2009fae0 0x7f4e2007eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:03:55.641 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:55.634+0000 7f4e24806700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e200a0710 con 0x7f4e2007e720 2019-12-05T06:03:55.642 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:55.634+0000 7f4e24806700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e200a0b00 con 0x7f4e200a7740 2019-12-05T06:03:55.642 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:55.634+0000 7f4e24806700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e2008e5b0 con 0x7f4e2009fae0 2019-12-05T06:03:55.642 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:55.634+0000 7f4e1e59c700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f4e2007e720 0x7f4e2009f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:47430/0 (socket says 172.21.15.42:47430) 2019-12-05T06:03:55.642 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:55.634+0000 7f4e1e59c700 1 -- 172.21.15.42:0/2007867539 learned_addr learned my addr 172.21.15.42:0/2007867539 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:03:55.642 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:55.634+0000 7f4e1f59e700 1 --2- 172.21.15.42:0/2007867539 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4e2009fae0 0x7f4e2007eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:03:55.642 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:55.634+0000 7f4e1ed9d700 1 --2- 172.21.15.42:0/2007867539 >> v2:172.21.15.40:3300/0 conn(0x7f4e200a7740 0x7f4e2008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:03:55.642 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:55.634+0000 7f4e1dd9b700 1 -- 172.21.15.42:0/2007867539 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1294150704 0 0) 0x7f4e200a0710 con 0x7f4e2007e720 2019-12-05T06:03:55.643 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:55.634+0000 7f4e1dd9b700 1 -- 172.21.15.42:0/2007867539 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e0c0030b0 con 0x7f4e2007e720 2019-12-05T06:03:55.643 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:55.634+0000 7f4e1ed9d700 1 -- 172.21.15.42:0/2007867539 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4e2009fae0 msgr2=0x7f4e2007eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:03:55.643 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:55.634+0000 7f4e1ed9d700 1 --2- 172.21.15.42:0/2007867539 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4e2009fae0 0x7f4e2007eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:03:55.643 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:55.634+0000 7f4e1ed9d700 1 -- 172.21.15.42:0/2007867539 >> v1:172.21.15.42:6789/0 conn(0x7f4e2007e720 legacy=0x7f4e2009f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:03:55.643 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:55.634+0000 7f4e1ed9d700 1 -- 172.21.15.42:0/2007867539 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e2007f0a0 con 0x7f4e200a7740 2019-12-05T06:03:55.643 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:55.634+0000 7f4e1f59e700 1 --2- 172.21.15.42:0/2007867539 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4e2009fae0 0x7f4e2007eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:03:55.643 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:55.634+0000 7f4e1ed9d700 1 --2- 172.21.15.42:0/2007867539 >> v2:172.21.15.40:3300/0 conn(0x7f4e200a7740 0x7f4e2008df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f4e10008790 tx=0x7f4e1000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:03:55.644 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:55.634+0000 7f4e1ed9d700 1 --2- 172.21.15.42:0/2007867539 >> v2:172.21.15.40:3300/0 conn(0x7f4e200a7740 0x7f4e2008df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f4e10008790 tx=0x7f4e1000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:03:55.644 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:55.634+0000 7f4e1ed9d700 1 --2- 172.21.15.42:0/2007867539 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e200a7740 0x7f4e2008df80 secure :-1 s=READY pgs=1487 cs=0 l=1 rx=0x7f4e10008790 tx=0x7f4e1000fc70).ready entity=mon.0 client_cookie=82d9d6b2cbd7b8dc server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:03:55.644 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:55.634+0000 7f4e1ed9d700 1 --2- 172.21.15.42:0/2007867539 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e200a7740 0x7f4e2008df80 secure :-1 s=THROTTLE_DONE pgs=1487 cs=0 l=1 rx=0x7f4e10008790 tx=0x7f4e1000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:03:55.644 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:55.634+0000 7f4e1dd9b700 1 -- 172.21.15.42:0/2007867539 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4e10008bb0 con 0x7f4e200a7740 2019-12-05T06:03:55.644 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:55.634+0000 7f4e1ed9d700 1 --2- 172.21.15.42:0/2007867539 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e200a7740 0x7f4e2008df80 secure :-1 s=THROTTLE_DONE pgs=1487 cs=0 l=1 rx=0x7f4e10008790 tx=0x7f4e1000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:03:55.644 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:55.638+0000 7f4e1dd9b700 1 -- 172.21.15.42:0/2007867539 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e100087d0 con 0x7f4e200a7740 2019-12-05T06:03:55.645 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:55.638+0000 7f4e1ed9d700 1 --2- 172.21.15.42:0/2007867539 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e200a7740 0x7f4e2008df80 secure :-1 s=THROTTLE_DONE pgs=1487 cs=0 l=1 rx=0x7f4e10008790 tx=0x7f4e1000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:03:55.645 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:55.638+0000 7f4e1dd9b700 1 -- 172.21.15.42:0/2007867539 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4e10011860 con 0x7f4e200a7740 2019-12-05T06:03:55.645 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:55.638+0000 7f4e24806700 1 -- 172.21.15.42:0/2007867539 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e200a7740 msgr2=0x7f4e2008df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:03:55.645 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:55.638+0000 7f4e24806700 1 --2- 172.21.15.42:0/2007867539 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e200a7740 0x7f4e2008df80 secure :-1 s=READY pgs=1487 cs=0 l=1 rx=0x7f4e10008790 tx=0x7f4e1000fc70).stop 2019-12-05T06:03:55.645 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:55.638+0000 7f4e24806700 1 -- 172.21.15.42:0/2007867539 shutdown_connections 2019-12-05T06:03:55.645 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:55.638+0000 7f4e24806700 1 --2- 172.21.15.42:0/2007867539 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4e2009fae0 0x7f4e2007eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:03:55.646 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:55.638+0000 7f4e24806700 1 --2- 172.21.15.42:0/2007867539 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e200a7740 0x7f4e2008df80 unknown :-1 s=CLOSED pgs=1487 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:03:55.646 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:55.638+0000 7f4e24806700 1 -- 172.21.15.42:0/2007867539 shutdown_connections 2019-12-05T06:03:55.646 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:55.638+0000 7f4e24806700 1 -- 172.21.15.42:0/2007867539 wait complete. 2019-12-05T06:03:55.646 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:55.638+0000 7f4e24806700 1 -- 172.21.15.42:0/2007867539 >> 172.21.15.42:0/2007867539 conn(0x7f4e200aabe0 msgr2=0x7f4e200a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:03:55.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:55.638+0000 7f4e24806700 1 Processor -- start 2019-12-05T06:03:55.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:55.638+0000 7f4e24806700 1 -- start start 2019-12-05T06:03:55.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:55.642+0000 7f4e24806700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4e20118850 0x7f4e2007b230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:03:55.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:55.642+0000 7f4e24806700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f4e2007b780 0x7f4e20137b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:03:55.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:55.642+0000 7f4e24806700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e2011bc40 con 0x7f4e20124660 2019-12-05T06:03:55.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:55.642+0000 7f4e24806700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e2011d8b0 con 0x7f4e2007b780 2019-12-05T06:03:55.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:55.642+0000 7f4e24806700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e2011b1c0 con 0x7f4e20118850 2019-12-05T06:03:55.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:55.642+0000 7f4e1e59c700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4e20118850 0x7f4e2007b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:03:55.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:55.642+0000 7f4e1f59e700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f4e2007b780 0x7f4e20137b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:03:55.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:55.642+0000 7f4e1f59e700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f4e2007b780 0x7f4e20137b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.42:35216/0 (socket says 172.21.15.42:35216) 2019-12-05T06:03:55.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:55.642+0000 7f4e1e59c700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4e20118850 0x7f4e2007b230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:47864/0 (socket says 172.21.15.42:47864) 2019-12-05T06:03:55.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:55.642+0000 7f4e1f59e700 1 -- 172.21.15.42:0/900622224 learned_addr learned my addr 172.21.15.42:0/900622224 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:03:55.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:55.642+0000 7f4e1f59e700 1 -- 172.21.15.42:0/900622224 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4e20118850 msgr2=0x7f4e2007b230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:03:55.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:55.642+0000 7f4e1f59e700 1 --2- 172.21.15.42:0/900622224 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4e20118850 0x7f4e2007b230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:03:55.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:55.642+0000 7f4e1f59e700 1 -- 172.21.15.42:0/900622224 >> v1:172.21.15.40:6789/0 conn(0x7f4e20124660 legacy=0x7f4e20130700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:03:55.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:55.642+0000 7f4e1f59e700 1 -- 172.21.15.42:0/900622224 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e20130f00 con 0x7f4e2007b780 2019-12-05T06:03:55.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:55.642+0000 7f4e1e59c700 1 --2- 172.21.15.42:0/900622224 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4e20118850 0x7f4e2007b230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:03:55.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:55.642+0000 7f4e1f59e700 1 --2- 172.21.15.42:0/900622224 >> v2:172.21.15.40:3300/0 conn(0x7f4e2007b780 0x7f4e20137b50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f4e2011be90 tx=0x7f4e18010f00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:03:55.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:55.642+0000 7f4e1f59e700 1 --2- 172.21.15.42:0/900622224 >> v2:172.21.15.40:3300/0 conn(0x7f4e2007b780 0x7f4e20137b50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f4e2011be90 tx=0x7f4e18010f00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:03:55.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:55.642+0000 7f4e1f59e700 1 --2- 172.21.15.42:0/900622224 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e2007b780 0x7f4e20137b50 secure :-1 s=READY pgs=1489 cs=0 l=1 rx=0x7f4e2011be90 tx=0x7f4e18010f00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:03:55.654 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:55.642+0000 7f4e1f59e700 1 --2- 172.21.15.42:0/900622224 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e2007b780 0x7f4e20137b50 secure :-1 s=THROTTLE_DONE pgs=1489 cs=0 l=1 rx=0x7f4e2011be90 tx=0x7f4e18010f00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:03:55.654 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:55.642+0000 7f4e177fe700 1 -- 172.21.15.42:0/900622224 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4e180058c0 con 0x7f4e2007b780 2019-12-05T06:03:55.654 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:55.642+0000 7f4e1f59e700 1 --2- 172.21.15.42:0/900622224 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e2007b780 0x7f4e20137b50 secure :-1 s=THROTTLE_DONE pgs=1489 cs=0 l=1 rx=0x7f4e2011be90 tx=0x7f4e18010f00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:03:55.654 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:55.642+0000 7f4e1f59e700 1 --2- 172.21.15.42:0/900622224 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e2007b780 0x7f4e20137b50 secure :-1 s=THROTTLE_DONE pgs=1489 cs=0 l=1 rx=0x7f4e2011be90 tx=0x7f4e18010f00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:03:55.654 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:55.642+0000 7f4e177fe700 1 -- 172.21.15.42:0/900622224 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e1800e160 con 0x7f4e2007b780 2019-12-05T06:03:55.654 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:55.642+0000 7f4e24806700 1 -- 172.21.15.42:0/900622224 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e2011c0f0 con 0x7f4e2007b780 2019-12-05T06:03:55.654 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:55.642+0000 7f4e177fe700 1 -- 172.21.15.42:0/900622224 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4e1800c850 con 0x7f4e2007b780 2019-12-05T06:03:55.655 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:55.642+0000 7f4e24806700 1 -- 172.21.15.42:0/900622224 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e201380a0 con 0x7f4e2007b780 2019-12-05T06:03:55.655 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:55.642+0000 7f4e1f59e700 1 --2- 172.21.15.42:0/900622224 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e2007b780 0x7f4e20137b50 secure :-1 s=THROTTLE_DONE pgs=1489 cs=0 l=1 rx=0x7f4e2011be90 tx=0x7f4e18010f00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:03:55.655 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:55.646+0000 7f4e177fe700 1 -- 172.21.15.42:0/900622224 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f4e1800cb60 con 0x7f4e2007b780 2019-12-05T06:03:55.655 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:55.646+0000 7f4e1f59e700 1 --2- 172.21.15.42:0/900622224 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e2007b780 0x7f4e20137b50 secure :-1 s=THROTTLE_DONE pgs=1489 cs=0 l=1 rx=0x7f4e2011be90 tx=0x7f4e18010f00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:03:55.655 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:55.646+0000 7f4e177fe700 1 --2- 172.21.15.42:0/900622224 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4e0403ae70 0x7f4e0403d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:03:55.655 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:55.646+0000 7f4e177fe700 1 -- 172.21.15.42:0/900622224 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9557+0+0 (secure 0 0 0) 0x7f4e1804afb0 con 0x7f4e2007b780 2019-12-05T06:03:55.655 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:55.646+0000 7f4e1ed9d700 1 --2- 172.21.15.42:0/900622224 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4e0403ae70 0x7f4e0403d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:03:55.656 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:55.646+0000 7f4e1ed9d700 1 --2- 172.21.15.42:0/900622224 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4e0403ae70 0x7f4e0403d380 crc :-1 s=READY pgs=833 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:03:55.656 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:55.646+0000 7f4e24806700 1 -- 172.21.15.42:0/900622224 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e2005d110 con 0x7f4e2007b780 2019-12-05T06:03:55.660 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:55.658+0000 7f4e1f59e700 1 --2- 172.21.15.42:0/900622224 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e2007b780 0x7f4e20137b50 secure :-1 s=THROTTLE_DONE pgs=1489 cs=0 l=1 rx=0x7f4e2011be90 tx=0x7f4e18010f00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:03:55.660 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:55.658+0000 7f4e177fe700 1 -- 172.21.15.42:0/900622224 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f4e18032290 con 0x7f4e2007b780 2019-12-05T06:03:55.825 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:55.822+0000 7f4e24806700 1 -- 172.21.15.42:0/900622224 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4e200a9ed0 con 0x7f4e2007b780 2019-12-05T06:03:55.826 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:55.822+0000 7f4e1f59e700 1 --2- 172.21.15.42:0/900622224 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e2007b780 0x7f4e20137b50 secure :-1 s=THROTTLE_DONE pgs=1489 cs=0 l=1 rx=0x7f4e2011be90 tx=0x7f4e18010f00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:03:55.826 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:55.822+0000 7f4e177fe700 1 -- 172.21.15.42:0/900622224 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 238 v238) v1 ==== 95+0+2109 (secure 0 0 0) 0x7f4e18032e40 con 0x7f4e2007b780 2019-12-05T06:03:55.828 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:03:55.828 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":238,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":9381,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/3789081777","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3789081777},{"type":"v1","addr":"172.21.15.40:6833","nonce":3789081777}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":238}],"filesystems":[{"mdsmap":{"epoch":237,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:01:44.401238+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":90,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9292},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9292":{"gid":9292,"name":"a","rank":0,"incarnation":234,"state":"up:active","state_seq":12,"addr":"172.21.15.42:6835/613535662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":613535662},{"type":"v1","addr":"172.21.15.42:6835","nonce":613535662}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:03:55.829 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 238 2019-12-05T06:03:55.831 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:55.826+0000 7f4e24806700 1 -- 172.21.15.42:0/900622224 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4e0403ae70 msgr2=0x7f4e0403d380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:03:55.831 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:55.826+0000 7f4e24806700 1 --2- 172.21.15.42:0/900622224 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4e0403ae70 0x7f4e0403d380 crc :-1 s=READY pgs=833 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:03:55.831 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:55.826+0000 7f4e24806700 1 -- 172.21.15.42:0/900622224 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e2007b780 msgr2=0x7f4e20137b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:03:55.831 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:55.826+0000 7f4e24806700 1 --2- 172.21.15.42:0/900622224 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e2007b780 0x7f4e20137b50 secure :-1 s=READY pgs=1489 cs=0 l=1 rx=0x7f4e2011be90 tx=0x7f4e18010f00).stop 2019-12-05T06:03:55.831 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:55.826+0000 7f4e24806700 1 -- 172.21.15.42:0/900622224 shutdown_connections 2019-12-05T06:03:55.832 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:55.826+0000 7f4e24806700 1 --2- 172.21.15.42:0/900622224 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4e0403ae70 0x7f4e0403d380 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:03:55.832 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:55.826+0000 7f4e24806700 1 --2- 172.21.15.42:0/900622224 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e2007b780 0x7f4e20137b50 unknown :-1 s=CLOSED pgs=1489 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:03:55.832 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:55.826+0000 7f4e24806700 1 --2- 172.21.15.42:0/900622224 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4e20118850 0x7f4e2007b230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:03:55.832 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:55.826+0000 7f4e24806700 1 -- 172.21.15.42:0/900622224 shutdown_connections 2019-12-05T06:03:55.833 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:55.826+0000 7f4e24806700 1 -- 172.21.15.42:0/900622224 wait complete. 2019-12-05T06:03:55.833 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:55.826+0000 7f4e24806700 1 -- 172.21.15.42:0/900622224 >> 172.21.15.42:0/900622224 conn(0x7f4e2008fda0 msgr2=0x7f4e200a9640 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:03:55.841 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:03:57.841 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:03:57.957 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:57.938+0000 7ff84e249700 1 Processor -- start 2019-12-05T06:03:57.957 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:57.938+0000 7ff84e249700 1 -- start start 2019-12-05T06:03:57.957 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:57.938+0000 7ff84e249700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff84809fae0 0x7ff84814fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:03:57.957 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:57.938+0000 7ff84e249700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8480a0710 con 0x7ff84808df80 2019-12-05T06:03:57.957 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:57.938+0000 7ff84e249700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff84807ef20 con 0x7ff8480a7740 2019-12-05T06:03:57.958 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:57.938+0000 7ff84e249700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8480a0b00 con 0x7ff84809fae0 2019-12-05T06:03:57.958 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:57.938+0000 7ff84cfe7700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff84809fae0 0x7ff84814fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:03:57.958 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:57.938+0000 7ff847fff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7ff8480a7740 0x7ff84807e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:47444/0 (socket says 172.21.15.42:47444) 2019-12-05T06:03:57.958 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:57.942+0000 7ff847fff700 1 -- 172.21.15.42:0/2360637343 learned_addr learned my addr 172.21.15.42:0/2360637343 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:03:57.958 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:57.942+0000 7ff8477fe700 1 -- 172.21.15.42:0/2360637343 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1616341244 0 0) 0x7ff8480a0710 con 0x7ff84808df80 2019-12-05T06:03:57.958 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:57.942+0000 7ff84cfe7700 1 -- 172.21.15.42:0/2360637343 >> v1:172.21.15.42:6789/0 conn(0x7ff8480a7740 legacy=0x7ff84807e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:03:57.958 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:57.942+0000 7ff84cfe7700 1 -- 172.21.15.42:0/2360637343 >> v1:172.21.15.40:6789/0 conn(0x7ff84808df80 legacy=0x7ff84809f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:03:57.959 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:57.942+0000 7ff84cfe7700 1 -- 172.21.15.42:0/2360637343 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8480a9ba0 con 0x7ff84809fae0 2019-12-05T06:03:57.959 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:57.942+0000 7ff84cfe7700 1 --2- 172.21.15.42:0/2360637343 >> v2:172.21.15.42:3300/0 conn(0x7ff84809fae0 0x7ff84814fff0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7ff840008b10 tx=0x7ff838001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:03:57.959 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:57.942+0000 7ff84cfe7700 1 --2- 172.21.15.42:0/2360637343 >> v2:172.21.15.42:3300/0 conn(0x7ff84809fae0 0x7ff84814fff0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7ff840008b10 tx=0x7ff838001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:03:57.959 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:57.942+0000 7ff84cfe7700 1 --2- 172.21.15.42:0/2360637343 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff84809fae0 0x7ff84814fff0 secure :-1 s=READY pgs=1964 cs=0 l=1 rx=0x7ff840008b10 tx=0x7ff838001150).ready entity=mon.1 client_cookie=146d5e36ded3b781 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:03:57.959 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:57.942+0000 7ff84cfe7700 1 --2- 172.21.15.42:0/2360637343 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff84809fae0 0x7ff84814fff0 secure :-1 s=THROTTLE_DONE pgs=1964 cs=0 l=1 rx=0x7ff840008b10 tx=0x7ff838001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:03:57.959 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:57.942+0000 7ff8477fe700 1 -- 172.21.15.42:0/2360637343 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff840010870 con 0x7ff84809fae0 2019-12-05T06:03:57.959 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:57.942+0000 7ff84cfe7700 1 --2- 172.21.15.42:0/2360637343 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff84809fae0 0x7ff84814fff0 secure :-1 s=THROTTLE_DONE pgs=1964 cs=0 l=1 rx=0x7ff840008b10 tx=0x7ff838001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:03:57.960 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:57.942+0000 7ff84cfe7700 1 --2- 172.21.15.42:0/2360637343 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff84809fae0 0x7ff84814fff0 secure :-1 s=THROTTLE_DONE pgs=1964 cs=0 l=1 rx=0x7ff840008b10 tx=0x7ff838001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:03:57.960 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:57.942+0000 7ff8477fe700 1 -- 172.21.15.42:0/2360637343 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff840008b50 con 0x7ff84809fae0 2019-12-05T06:03:57.960 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:57.942+0000 7ff8477fe700 1 -- 172.21.15.42:0/2360637343 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff8400110c0 con 0x7ff84809fae0 2019-12-05T06:03:57.960 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:57.942+0000 7ff84e249700 1 -- 172.21.15.42:0/2360637343 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff84809fae0 msgr2=0x7ff84814fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:03:57.960 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:57.942+0000 7ff84e249700 1 --2- 172.21.15.42:0/2360637343 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff84809fae0 0x7ff84814fff0 secure :-1 s=READY pgs=1964 cs=0 l=1 rx=0x7ff840008b10 tx=0x7ff838001150).stop 2019-12-05T06:03:57.960 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:57.942+0000 7ff84e249700 1 -- 172.21.15.42:0/2360637343 shutdown_connections 2019-12-05T06:03:57.960 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:57.942+0000 7ff84e249700 1 --2- 172.21.15.42:0/2360637343 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff84809fae0 0x7ff84814fff0 unknown :-1 s=CLOSED pgs=1964 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:03:57.961 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:57.942+0000 7ff84e249700 1 -- 172.21.15.42:0/2360637343 shutdown_connections 2019-12-05T06:03:57.961 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:57.942+0000 7ff84e249700 1 -- 172.21.15.42:0/2360637343 wait complete. 2019-12-05T06:03:57.961 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:57.942+0000 7ff84e249700 1 -- 172.21.15.42:0/2360637343 >> 172.21.15.42:0/2360637343 conn(0x7ff8480aabe0 msgr2=0x7ff8480a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:03:57.961 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:57.946+0000 7ff84e249700 1 Processor -- start 2019-12-05T06:03:57.961 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:57.946+0000 7ff84e249700 1 -- start start 2019-12-05T06:03:57.962 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:57.946+0000 7ff84e249700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff848124540 0x7ff848131500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:03:57.963 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:57.946+0000 7ff84e249700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff84812e9e0 0x7ff8480742d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:03:57.963 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:57.946+0000 7ff84e249700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff848124970 con 0x7ff848131a50 2019-12-05T06:03:57.963 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:57.946+0000 7ff84e249700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff848132080 con 0x7ff848124540 2019-12-05T06:03:57.963 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:57.946+0000 7ff84e249700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff848132320 con 0x7ff84812e9e0 2019-12-05T06:03:57.978 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:57.946+0000 7ff84cfe7700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff84812e9e0 0x7ff8480742d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:03:57.978 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:57.946+0000 7ff84cfe7700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff84812e9e0 0x7ff8480742d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:47416/0 (socket says 172.21.15.42:47416) 2019-12-05T06:03:57.979 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:57.946+0000 7ff84cfe7700 1 -- 172.21.15.42:0/3456396917 learned_addr learned my addr 172.21.15.42:0/3456396917 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:03:57.979 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:57.946+0000 7ff847fff700 1 --2- 172.21.15.42:0/3456396917 >> v2:172.21.15.40:3300/0 conn(0x7ff848124540 0x7ff848131500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:03:57.979 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:57.950+0000 7ff84cfe7700 1 -- 172.21.15.42:0/3456396917 >> v2:172.21.15.40:3300/0 conn(0x7ff848124540 msgr2=0x7ff848131500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:03:57.979 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:57.950+0000 7ff84cfe7700 1 --2- 172.21.15.42:0/3456396917 >> v2:172.21.15.40:3300/0 conn(0x7ff848124540 0x7ff848131500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:03:57.979 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:57.950+0000 7ff84cfe7700 1 -- 172.21.15.42:0/3456396917 >> v1:172.21.15.42:6789/0 conn(0x7ff848131a50 legacy=0x7ff84812e1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:03:57.979 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:57.950+0000 7ff84cfe7700 1 -- 172.21.15.42:0/3456396917 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff848075830 con 0x7ff84812e9e0 2019-12-05T06:03:57.980 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:57.950+0000 7ff84cfe7700 1 --2- 172.21.15.42:0/3456396917 >> v2:172.21.15.42:3300/0 conn(0x7ff84812e9e0 0x7ff8480742d0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7ff848117aa0 tx=0x7ff840010f10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:03:57.980 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:57.950+0000 7ff84cfe7700 1 --2- 172.21.15.42:0/3456396917 >> v2:172.21.15.42:3300/0 conn(0x7ff84812e9e0 0x7ff8480742d0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7ff848117aa0 tx=0x7ff840010f10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:03:57.980 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:57.950+0000 7ff84cfe7700 1 --2- 172.21.15.42:0/3456396917 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff84812e9e0 0x7ff8480742d0 secure :-1 s=READY pgs=1966 cs=0 l=1 rx=0x7ff848117aa0 tx=0x7ff840010f10).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:03:57.980 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:57.950+0000 7ff84cfe7700 1 --2- 172.21.15.42:0/3456396917 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff84812e9e0 0x7ff8480742d0 secure :-1 s=THROTTLE_DONE pgs=1966 cs=0 l=1 rx=0x7ff848117aa0 tx=0x7ff840010f10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:03:57.980 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:57.950+0000 7ff8457fa700 1 -- 172.21.15.42:0/3456396917 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff840005cc0 con 0x7ff84812e9e0 2019-12-05T06:03:57.980 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:57.950+0000 7ff84cfe7700 1 --2- 172.21.15.42:0/3456396917 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff84812e9e0 0x7ff8480742d0 secure :-1 s=THROTTLE_DONE pgs=1966 cs=0 l=1 rx=0x7ff848117aa0 tx=0x7ff840010f10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:03:57.981 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:57.950+0000 7ff84cfe7700 1 --2- 172.21.15.42:0/3456396917 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff84812e9e0 0x7ff8480742d0 secure :-1 s=THROTTLE_DONE pgs=1966 cs=0 l=1 rx=0x7ff848117aa0 tx=0x7ff840010f10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:03:57.981 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:57.950+0000 7ff8457fa700 1 -- 172.21.15.42:0/3456396917 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff840010a70 con 0x7ff84812e9e0 2019-12-05T06:03:57.981 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:57.950+0000 7ff8457fa700 1 -- 172.21.15.42:0/3456396917 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff840007540 con 0x7ff84812e9e0 2019-12-05T06:03:57.981 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:57.950+0000 7ff84e249700 1 -- 172.21.15.42:0/3456396917 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff848074820 con 0x7ff84812e9e0 2019-12-05T06:03:57.981 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:57.950+0000 7ff84e249700 1 -- 172.21.15.42:0/3456396917 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff848074dd0 con 0x7ff84812e9e0 2019-12-05T06:03:57.981 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:57.950+0000 7ff84cfe7700 1 --2- 172.21.15.42:0/3456396917 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff84812e9e0 0x7ff8480742d0 secure :-1 s=THROTTLE_DONE pgs=1966 cs=0 l=1 rx=0x7ff848117aa0 tx=0x7ff840010f10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:03:57.981 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:57.950+0000 7ff8457fa700 1 -- 172.21.15.42:0/3456396917 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7ff84000f220 con 0x7ff84812e9e0 2019-12-05T06:03:57.981 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:57.950+0000 7ff84cfe7700 1 --2- 172.21.15.42:0/3456396917 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff84812e9e0 0x7ff8480742d0 secure :-1 s=THROTTLE_DONE pgs=1966 cs=0 l=1 rx=0x7ff848117aa0 tx=0x7ff840010f10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:03:57.982 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:57.950+0000 7ff8457fa700 1 --2- 172.21.15.42:0/3456396917 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff82c03ae70 0x7ff82c03d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:03:57.982 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:57.950+0000 7ff8457fa700 1 -- 172.21.15.42:0/3456396917 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9557+0+0 (secure 0 0 0) 0x7ff84004b020 con 0x7ff84812e9e0 2019-12-05T06:03:57.982 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:57.950+0000 7ff847fff700 1 --2- 172.21.15.42:0/3456396917 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff82c03ae70 0x7ff82c03d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:03:57.993 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:57.954+0000 7ff847fff700 1 --2- 172.21.15.42:0/3456396917 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff82c03ae70 0x7ff82c03d380 crc :-1 s=READY pgs=834 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:03:57.999 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:57.954+0000 7ff84e249700 1 -- 172.21.15.42:0/3456396917 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff848131e80 con 0x7ff84812e9e0 2019-12-05T06:03:57.999 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:57.954+0000 7ff84cfe7700 1 -- 172.21.15.42:0/3456396917 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff84812e9e0 msgr2=0x7ff8480742d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2019-12-05T06:03:57.999 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:57.954+0000 7ff84cfe7700 1 -- 172.21.15.42:0/3456396917 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff84812e9e0 msgr2=0x7ff8480742d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2019-12-05T06:03:58.000 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:57.954+0000 7ff84cfe7700 1 --2- 172.21.15.42:0/3456396917 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff84812e9e0 0x7ff8480742d0 secure :-1 s=READY pgs=1966 cs=0 l=1 rx=0x7ff848117aa0 tx=0x7ff840010f10).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-12-05T06:03:58.000 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:57.954+0000 7ff84cfe7700 1 --2- 172.21.15.42:0/3456396917 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff84812e9e0 0x7ff8480742d0 secure :-1 s=READY pgs=1966 cs=0 l=1 rx=0x7ff848117aa0 tx=0x7ff840010f10).stop 2019-12-05T06:03:58.000 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:57.954+0000 7ff8457fa700 1 -- 172.21.15.42:0/3456396917 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff84812e9e0 msgr2=0x7ff8480742d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2019-12-05T06:03:58.000 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:57.954+0000 7ff8457fa700 1 --2- 172.21.15.42:0/3456396917 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff84812e9e0 0x7ff8480742d0 unknown :-1 s=CLOSED pgs=1966 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:03:58.000 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:57.954+0000 7ff8457fa700 1 --2- 172.21.15.42:0/3456396917 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff82c043930 0x7ff82c045d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:03:58.000 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:57.954+0000 7ff8457fa700 1 --2- 172.21.15.42:0/3456396917 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff82c046430 0x7ff82c048940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:03:58.001 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:57.954+0000 7ff8457fa700 1 --2- 172.21.15.42:0/3456396917 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff82c049120 0x7ff82c04b630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:03:58.001 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:57.954+0000 7ff8457fa700 1 -- 172.21.15.42:0/3456396917 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff82c04be50 con 0x7ff82c043930 2019-12-05T06:03:58.001 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:57.954+0000 7ff8457fa700 1 -- 172.21.15.42:0/3456396917 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff82c04c040 con 0x7ff82c049120 2019-12-05T06:03:58.001 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:57.954+0000 7ff8457fa700 1 -- 172.21.15.42:0/3456396917 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff82c04c280 con 0x7ff82c046430 2019-12-05T06:03:58.001 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:57.954+0000 7ff847fff700 1 --2- 172.21.15.42:0/3456396917 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff82c049120 0x7ff82c04b630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:03:58.001 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:57.954+0000 7ff84cfe7700 1 --2- 172.21.15.42:0/3456396917 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff82c046430 0x7ff82c048940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:03:58.001 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:57.954+0000 7ff83f7fe700 1 --2- 172.21.15.42:0/3456396917 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff82c043930 0x7ff82c045d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:03:58.002 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:57.958+0000 7ff847fff700 1 -- 172.21.15.42:0/3456396917 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff82c046430 msgr2=0x7ff82c048940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:03:58.002 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:57.958+0000 7ff847fff700 1 --2- 172.21.15.42:0/3456396917 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff82c046430 0x7ff82c048940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:03:58.002 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:57.958+0000 7ff847fff700 1 -- 172.21.15.42:0/3456396917 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff82c043930 msgr2=0x7ff82c045d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:03:58.002 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:57.958+0000 7ff847fff700 1 --2- 172.21.15.42:0/3456396917 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff82c043930 0x7ff82c045d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:03:58.002 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:57.958+0000 7ff847fff700 1 -- 172.21.15.42:0/3456396917 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7ff82c04c4c0 con 0x7ff82c049120 2019-12-05T06:03:58.002 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:57.958+0000 7ff847fff700 1 -- 172.21.15.42:0/3456396917 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff838012140 con 0x7ff82c049120 2019-12-05T06:03:58.002 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:57.958+0000 7ff83f7fe700 1 --2- 172.21.15.42:0/3456396917 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff82c043930 0x7ff82c045d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:03:58.003 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:57.958+0000 7ff84cfe7700 1 --2- 172.21.15.42:0/3456396917 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff82c046430 0x7ff82c048940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:03:58.003 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:57.958+0000 7ff84cfe7700 1 -- 172.21.15.42:0/3456396917 reap_dead start 2019-12-05T06:03:58.003 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:57.958+0000 7ff847fff700 1 --2- 172.21.15.42:0/3456396917 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff82c049120 0x7ff82c04b630 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7ff838000b80 tx=0x7ff838012700).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:03:58.003 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:57.958+0000 7ff847fff700 1 --2- 172.21.15.42:0/3456396917 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff82c049120 0x7ff82c04b630 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7ff838000b80 tx=0x7ff838012700).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:03:58.003 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:57.958+0000 7ff847fff700 1 --2- 172.21.15.42:0/3456396917 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff82c049120 0x7ff82c04b630 secure :-1 s=READY pgs=1967 cs=0 l=1 rx=0x7ff838000b80 tx=0x7ff838012700).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:03:58.004 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:57.966+0000 7ff847fff700 1 --2- 172.21.15.42:0/3456396917 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff82c049120 0x7ff82c04b630 secure :-1 s=THROTTLE_DONE pgs=1967 cs=0 l=1 rx=0x7ff838000b80 tx=0x7ff838012700).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:03:58.004 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:57.966+0000 7ff847fff700 1 --2- 172.21.15.42:0/3456396917 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff82c049120 0x7ff82c04b630 secure :-1 s=THROTTLE_DONE pgs=1967 cs=0 l=1 rx=0x7ff838000b80 tx=0x7ff838012700).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:03:58.004 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:57.966+0000 7ff8457fa700 1 -- 172.21.15.42:0/3456396917 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff838012140 con 0x7ff82c049120 2019-12-05T06:03:58.004 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:57.966+0000 7ff8457fa700 1 -- 172.21.15.42:0/3456396917 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff83800b160 con 0x7ff82c049120 2019-12-05T06:03:58.004 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:57.966+0000 7ff847fff700 1 --2- 172.21.15.42:0/3456396917 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff82c049120 0x7ff82c04b630 secure :-1 s=THROTTLE_DONE pgs=1967 cs=0 l=1 rx=0x7ff838000b80 tx=0x7ff838012700).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:03:58.004 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:57.966+0000 7ff8457fa700 1 -- 172.21.15.42:0/3456396917 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7ff838004640 con 0x7ff82c049120 2019-12-05T06:03:58.004 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:57.970+0000 7ff847fff700 1 --2- 172.21.15.42:0/3456396917 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff82c049120 0x7ff82c04b630 secure :-1 s=THROTTLE_DONE pgs=1967 cs=0 l=1 rx=0x7ff838000b80 tx=0x7ff838012700).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:03:58.005 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:57.970+0000 7ff8457fa700 1 -- 172.21.15.42:0/3456396917 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7ff8380307a0 con 0x7ff82c049120 2019-12-05T06:03:58.151 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:58.150+0000 7ff84e249700 1 -- 172.21.15.42:0/3456396917 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff84805d110 con 0x7ff82c049120 2019-12-05T06:03:58.152 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:58.150+0000 7ff847fff700 1 --2- 172.21.15.42:0/3456396917 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff82c049120 0x7ff82c04b630 secure :-1 s=THROTTLE_DONE pgs=1967 cs=0 l=1 rx=0x7ff838000b80 tx=0x7ff838012700).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:03:58.152 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:58.150+0000 7ff8457fa700 1 -- 172.21.15.42:0/3456396917 <== mon.1 v2:172.21.15.42:3300/0 5 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 238 v238) v1 ==== 95+0+2109 (secure 0 0 0) 0x7ff8380309c0 con 0x7ff82c049120 2019-12-05T06:03:58.154 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 238 2019-12-05T06:03:58.154 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:03:58.154 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":238,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":9381,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/3789081777","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3789081777},{"type":"v1","addr":"172.21.15.40:6833","nonce":3789081777}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":238}],"filesystems":[{"mdsmap":{"epoch":237,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:01:44.401238+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":90,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9292},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9292":{"gid":9292,"name":"a","rank":0,"incarnation":234,"state":"up:active","state_seq":12,"addr":"172.21.15.42:6835/613535662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":613535662},{"type":"v1","addr":"172.21.15.42:6835","nonce":613535662}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:03:58.157 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:58.154+0000 7ff84e249700 1 -- 172.21.15.42:0/3456396917 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff82c03ae70 msgr2=0x7ff82c03d380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:03:58.157 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:58.154+0000 7ff84e249700 1 --2- 172.21.15.42:0/3456396917 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff82c03ae70 0x7ff82c03d380 crc :-1 s=READY pgs=834 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:03:58.157 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:58.154+0000 7ff84e249700 1 -- 172.21.15.42:0/3456396917 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff82c049120 msgr2=0x7ff82c04b630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:03:58.157 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:58.154+0000 7ff84e249700 1 --2- 172.21.15.42:0/3456396917 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff82c049120 0x7ff82c04b630 secure :-1 s=READY pgs=1967 cs=0 l=1 rx=0x7ff838000b80 tx=0x7ff838012700).stop 2019-12-05T06:03:58.158 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:58.154+0000 7ff84e249700 1 -- 172.21.15.42:0/3456396917 shutdown_connections 2019-12-05T06:03:58.158 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:58.154+0000 7ff84e249700 1 --2- 172.21.15.42:0/3456396917 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff82c049120 0x7ff82c04b630 unknown :-1 s=CLOSED pgs=1967 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:03:58.158 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:58.154+0000 7ff84e249700 1 --2- 172.21.15.42:0/3456396917 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff84812e9e0 0x7ff8480742d0 unknown :-1 s=CLOSED pgs=1966 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:03:58.158 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:58.154+0000 7ff84e249700 1 --2- 172.21.15.42:0/3456396917 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff82c03ae70 0x7ff82c03d380 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:03:58.159 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:58.154+0000 7ff84e249700 1 -- 172.21.15.42:0/3456396917 shutdown_connections 2019-12-05T06:03:58.159 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:58.154+0000 7ff84e249700 1 -- 172.21.15.42:0/3456396917 wait complete. 2019-12-05T06:03:58.159 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:03:58.154+0000 7ff84e249700 1 -- 172.21.15.42:0/3456396917 >> 172.21.15.42:0/3456396917 conn(0x7ff8480a6bb0 msgr2=0x7ff84807e4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:03:58.167 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:04:00.168 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:04:00.274 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:00.266+0000 7fbc6408e700 1 Processor -- start 2019-12-05T06:04:00.274 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:00.266+0000 7fbc6408e700 1 -- start start 2019-12-05T06:04:00.274 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:00.266+0000 7fbc6408e700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fbc5c0a7740 0x7fbc5c08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:04:00.274 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:00.266+0000 7fbc6408e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fbc5c07e720 0x7fbc5c07eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:04:00.275 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:00.266+0000 7fbc6408e700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc5c0a0710 con 0x7fbc5c09f2e0 2019-12-05T06:04:00.275 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:00.266+0000 7fbc6408e700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc5c0a0b00 con 0x7fbc5c0a7740 2019-12-05T06:04:00.275 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:00.266+0000 7fbc6408e700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc5c08e5b0 con 0x7fbc5c07e720 2019-12-05T06:04:00.275 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:00.266+0000 7fbc61e2a700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fbc5c07e720 0x7fbc5c07eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:04:00.275 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:00.266+0000 7fbc61e2a700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fbc5c07e720 0x7fbc5c07eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:47426/0 (socket says 172.21.15.42:47426) 2019-12-05T06:04:00.275 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:00.266+0000 7fbc61e2a700 1 -- 172.21.15.42:0/3962119491 learned_addr learned my addr 172.21.15.42:0/3962119491 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:04:00.276 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:00.266+0000 7fbc6262b700 1 --2- 172.21.15.42:0/3962119491 >> v2:172.21.15.40:3300/0 conn(0x7fbc5c0a7740 0x7fbc5c08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:04:00.276 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:00.270+0000 7fbc61e2a700 1 -- 172.21.15.42:0/3962119491 >> v2:172.21.15.40:3300/0 conn(0x7fbc5c0a7740 msgr2=0x7fbc5c08df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:04:00.276 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:00.270+0000 7fbc61e2a700 1 --2- 172.21.15.42:0/3962119491 >> v2:172.21.15.40:3300/0 conn(0x7fbc5c0a7740 0x7fbc5c08df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:04:00.276 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:00.270+0000 7fbc61e2a700 1 -- 172.21.15.42:0/3962119491 >> v1:172.21.15.40:6789/0 conn(0x7fbc5c09f2e0 legacy=0x7fbc5c09f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:04:00.276 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:00.270+0000 7fbc61e2a700 1 -- 172.21.15.42:0/3962119491 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc5c07f0a0 con 0x7fbc5c07e720 2019-12-05T06:04:00.276 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:00.270+0000 7fbc6262b700 1 --2- 172.21.15.42:0/3962119491 >> v2:172.21.15.40:3300/0 conn(0x7fbc5c0a7740 0x7fbc5c08df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).send_auth_request state changed! 2019-12-05T06:04:00.277 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:00.270+0000 7fbc61e2a700 1 --2- 172.21.15.42:0/3962119491 >> v2:172.21.15.42:3300/0 conn(0x7fbc5c07e720 0x7fbc5c07eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fbc4c002f30 tx=0x7fbc4c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:04:00.277 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:00.270+0000 7fbc61e2a700 1 --2- 172.21.15.42:0/3962119491 >> v2:172.21.15.42:3300/0 conn(0x7fbc5c07e720 0x7fbc5c07eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fbc4c002f30 tx=0x7fbc4c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:04:00.277 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:00.270+0000 7fbc61e2a700 1 --2- 172.21.15.42:0/3962119491 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbc5c07e720 0x7fbc5c07eb50 secure :-1 s=READY pgs=1968 cs=0 l=1 rx=0x7fbc4c002f30 tx=0x7fbc4c00fd30).ready entity=mon.1 client_cookie=d967fd6b8ec0b311 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:04:00.277 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:00.270+0000 7fbc61e2a700 1 --2- 172.21.15.42:0/3962119491 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbc5c07e720 0x7fbc5c07eb50 secure :-1 s=THROTTLE_DONE pgs=1968 cs=0 l=1 rx=0x7fbc4c002f30 tx=0x7fbc4c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:04:00.277 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:00.270+0000 7fbc61629700 1 -- 172.21.15.42:0/3962119491 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbc4c011370 con 0x7fbc5c07e720 2019-12-05T06:04:00.277 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:00.270+0000 7fbc61e2a700 1 --2- 172.21.15.42:0/3962119491 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbc5c07e720 0x7fbc5c07eb50 secure :-1 s=THROTTLE_DONE pgs=1968 cs=0 l=1 rx=0x7fbc4c002f30 tx=0x7fbc4c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:04:00.278 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:00.270+0000 7fbc61629700 1 -- 172.21.15.42:0/3962119491 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc4c008bb0 con 0x7fbc5c07e720 2019-12-05T06:04:00.278 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:00.270+0000 7fbc61e2a700 1 --2- 172.21.15.42:0/3962119491 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbc5c07e720 0x7fbc5c07eb50 secure :-1 s=THROTTLE_DONE pgs=1968 cs=0 l=1 rx=0x7fbc4c002f30 tx=0x7fbc4c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:04:00.278 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:00.270+0000 7fbc61629700 1 -- 172.21.15.42:0/3962119491 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbc4c011950 con 0x7fbc5c07e720 2019-12-05T06:04:00.278 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:00.270+0000 7fbc6408e700 1 -- 172.21.15.42:0/3962119491 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbc5c07e720 msgr2=0x7fbc5c07eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:04:00.278 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:00.270+0000 7fbc6408e700 1 --2- 172.21.15.42:0/3962119491 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbc5c07e720 0x7fbc5c07eb50 secure :-1 s=READY pgs=1968 cs=0 l=1 rx=0x7fbc4c002f30 tx=0x7fbc4c00fd30).stop 2019-12-05T06:04:00.278 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:00.270+0000 7fbc6408e700 1 -- 172.21.15.42:0/3962119491 shutdown_connections 2019-12-05T06:04:00.278 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:00.270+0000 7fbc6408e700 1 --2- 172.21.15.42:0/3962119491 >> v2:172.21.15.40:3300/0 conn(0x7fbc5c0a7740 0x7fbc5c08df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:04:00.278 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:00.270+0000 7fbc6408e700 1 --2- 172.21.15.42:0/3962119491 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbc5c07e720 0x7fbc5c07eb50 unknown :-1 s=CLOSED pgs=1968 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:04:00.279 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:00.270+0000 7fbc6408e700 1 -- 172.21.15.42:0/3962119491 shutdown_connections 2019-12-05T06:04:00.279 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:00.270+0000 7fbc6408e700 1 -- 172.21.15.42:0/3962119491 wait complete. 2019-12-05T06:04:00.279 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:00.270+0000 7fbc6408e700 1 -- 172.21.15.42:0/3962119491 >> 172.21.15.42:0/3962119491 conn(0x7fbc5c0aabe0 msgr2=0x7fbc5c0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:04:00.282 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:00.270+0000 7fbc6408e700 1 Processor -- start 2019-12-05T06:04:00.282 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:00.270+0000 7fbc6408e700 1 -- start start 2019-12-05T06:04:00.282 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:00.274+0000 7fbc6408e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fbc5c130520 0x7fbc5c119650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:04:00.282 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:00.274+0000 7fbc6408e700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbc5c07c880 0x7fbc5c07ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:04:00.283 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:00.274+0000 7fbc6408e700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fbc5c0a4dc0 0x7fbc5c0a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:04:00.283 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:00.274+0000 7fbc6408e700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc5c07c340 con 0x7fbc5c0a4dc0 2019-12-05T06:04:00.283 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:00.274+0000 7fbc6408e700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc5c07c5e0 con 0x7fbc5c130520 2019-12-05T06:04:00.283 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:00.274+0000 7fbc6408e700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc5c131360 con 0x7fbc5c07c880 2019-12-05T06:04:00.283 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:00.274+0000 7fbc6262b700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fbc5c130520 0x7fbc5c119650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:04:00.283 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:00.274+0000 7fbc6262b700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fbc5c130520 0x7fbc5c119650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:47432/0 (socket says 172.21.15.42:47432) 2019-12-05T06:04:00.283 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:00.274+0000 7fbc6262b700 1 -- 172.21.15.42:0/1084505422 learned_addr learned my addr 172.21.15.42:0/1084505422 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:04:00.284 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:00.274+0000 7fbc62e2c700 1 --2- 172.21.15.42:0/1084505422 >> v2:172.21.15.40:3300/0 conn(0x7fbc5c0a4dc0 0x7fbc5c0a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:04:00.284 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:00.274+0000 7fbc61e2a700 1 --2- 172.21.15.42:0/1084505422 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbc5c07c880 0x7fbc5c07ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:04:00.284 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:00.274+0000 7fbc6262b700 1 -- 172.21.15.42:0/1084505422 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbc5c07c880 msgr2=0x7fbc5c07ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:04:00.284 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:00.274+0000 7fbc6262b700 1 --2- 172.21.15.42:0/1084505422 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbc5c07c880 0x7fbc5c07ccb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:04:00.284 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:00.274+0000 7fbc6262b700 1 -- 172.21.15.42:0/1084505422 >> v2:172.21.15.40:3300/0 conn(0x7fbc5c0a4dc0 msgr2=0x7fbc5c0a51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:04:00.285 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:00.274+0000 7fbc6262b700 1 --2- 172.21.15.42:0/1084505422 >> v2:172.21.15.40:3300/0 conn(0x7fbc5c0a4dc0 0x7fbc5c0a51f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:04:00.285 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:00.274+0000 7fbc6262b700 1 -- 172.21.15.42:0/1084505422 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc5c0a5740 con 0x7fbc5c130520 2019-12-05T06:04:00.285 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:00.274+0000 7fbc61e2a700 1 --2- 172.21.15.42:0/1084505422 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbc5c07c880 0x7fbc5c07ccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).send_auth_request state changed! 2019-12-05T06:04:00.285 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:00.274+0000 7fbc6262b700 1 --2- 172.21.15.42:0/1084505422 >> v2:172.21.15.42:3300/0 conn(0x7fbc5c130520 0x7fbc5c119650 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fbc58002f30 tx=0x7fbc58010950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:04:00.285 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:00.274+0000 7fbc6262b700 1 --2- 172.21.15.42:0/1084505422 >> v2:172.21.15.42:3300/0 conn(0x7fbc5c130520 0x7fbc5c119650 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fbc58002f30 tx=0x7fbc58010950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:04:00.286 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:00.274+0000 7fbc6262b700 1 --2- 172.21.15.42:0/1084505422 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbc5c130520 0x7fbc5c119650 secure :-1 s=READY pgs=1969 cs=0 l=1 rx=0x7fbc58002f30 tx=0x7fbc58010950).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:04:00.286 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:00.274+0000 7fbc6262b700 1 --2- 172.21.15.42:0/1084505422 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbc5c130520 0x7fbc5c119650 secure :-1 s=THROTTLE_DONE pgs=1969 cs=0 l=1 rx=0x7fbc58002f30 tx=0x7fbc58010950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:04:00.286 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:00.274+0000 7fbc52ffd700 1 -- 172.21.15.42:0/1084505422 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbc58009af0 con 0x7fbc5c130520 2019-12-05T06:04:00.286 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:00.274+0000 7fbc6262b700 1 --2- 172.21.15.42:0/1084505422 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbc5c130520 0x7fbc5c119650 secure :-1 s=THROTTLE_DONE pgs=1969 cs=0 l=1 rx=0x7fbc58002f30 tx=0x7fbc58010950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:04:00.286 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:00.274+0000 7fbc6262b700 1 --2- 172.21.15.42:0/1084505422 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbc5c130520 0x7fbc5c119650 secure :-1 s=THROTTLE_DONE pgs=1969 cs=0 l=1 rx=0x7fbc58002f30 tx=0x7fbc58010950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:04:00.286 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:00.274+0000 7fbc52ffd700 1 -- 172.21.15.42:0/1084505422 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc58007d30 con 0x7fbc5c130520 2019-12-05T06:04:00.287 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:00.274+0000 7fbc6408e700 1 -- 172.21.15.42:0/1084505422 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc5c11c420 con 0x7fbc5c130520 2019-12-05T06:04:00.287 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:00.274+0000 7fbc52ffd700 1 -- 172.21.15.42:0/1084505422 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbc58009390 con 0x7fbc5c130520 2019-12-05T06:04:00.287 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:00.274+0000 7fbc6408e700 1 -- 172.21.15.42:0/1084505422 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc5c11a5f0 con 0x7fbc5c130520 2019-12-05T06:04:00.287 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:00.274+0000 7fbc6262b700 1 --2- 172.21.15.42:0/1084505422 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbc5c130520 0x7fbc5c119650 secure :-1 s=THROTTLE_DONE pgs=1969 cs=0 l=1 rx=0x7fbc58002f30 tx=0x7fbc58010950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:04:00.287 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:00.274+0000 7fbc52ffd700 1 -- 172.21.15.42:0/1084505422 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fbc58009590 con 0x7fbc5c130520 2019-12-05T06:04:00.287 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:00.274+0000 7fbc6262b700 1 --2- 172.21.15.42:0/1084505422 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbc5c130520 0x7fbc5c119650 secure :-1 s=THROTTLE_DONE pgs=1969 cs=0 l=1 rx=0x7fbc58002f30 tx=0x7fbc58010950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:04:00.287 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:00.278+0000 7fbc52ffd700 1 --2- 172.21.15.42:0/1084505422 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbc4403ada0 0x7fbc4403d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:04:00.288 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:00.278+0000 7fbc52ffd700 1 -- 172.21.15.42:0/1084505422 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9557+0+0 (secure 0 0 0) 0x7fbc5804a9d0 con 0x7fbc5c130520 2019-12-05T06:04:00.288 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:00.278+0000 7fbc61e2a700 1 --2- 172.21.15.42:0/1084505422 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbc4403ada0 0x7fbc4403d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:04:00.288 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:00.278+0000 7fbc61e2a700 1 --2- 172.21.15.42:0/1084505422 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbc4403ada0 0x7fbc4403d2b0 crc :-1 s=READY pgs=835 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:04:00.288 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:00.282+0000 7fbc6408e700 1 -- 172.21.15.42:0/1084505422 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc5c05d110 con 0x7fbc5c130520 2019-12-05T06:04:00.294 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:00.290+0000 7fbc6262b700 1 --2- 172.21.15.42:0/1084505422 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbc5c130520 0x7fbc5c119650 secure :-1 s=THROTTLE_DONE pgs=1969 cs=0 l=1 rx=0x7fbc58002f30 tx=0x7fbc58010950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:04:00.295 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:00.290+0000 7fbc52ffd700 1 -- 172.21.15.42:0/1084505422 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fbc58031a50 con 0x7fbc5c130520 2019-12-05T06:04:00.460 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:00.458+0000 7fbc6408e700 1 -- 172.21.15.42:0/1084505422 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbc5c05b8c0 con 0x7fbc5c130520 2019-12-05T06:04:00.462 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:00.458+0000 7fbc6262b700 1 --2- 172.21.15.42:0/1084505422 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbc5c130520 0x7fbc5c119650 secure :-1 s=THROTTLE_DONE pgs=1969 cs=0 l=1 rx=0x7fbc58002f30 tx=0x7fbc58010950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:04:00.462 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:00.458+0000 7fbc52ffd700 1 -- 172.21.15.42:0/1084505422 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 238 v238) v1 ==== 95+0+2109 (secure 0 0 0) 0x7fbc58031cb0 con 0x7fbc5c130520 2019-12-05T06:04:00.462 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 238 2019-12-05T06:04:00.462 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:04:00.462 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":238,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":9381,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/3789081777","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3789081777},{"type":"v1","addr":"172.21.15.40:6833","nonce":3789081777}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":238}],"filesystems":[{"mdsmap":{"epoch":237,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:01:44.401238+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":90,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9292},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9292":{"gid":9292,"name":"a","rank":0,"incarnation":234,"state":"up:active","state_seq":12,"addr":"172.21.15.42:6835/613535662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":613535662},{"type":"v1","addr":"172.21.15.42:6835","nonce":613535662}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:04:00.464 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:00.462+0000 7fbc6408e700 1 -- 172.21.15.42:0/1084505422 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbc4403ada0 msgr2=0x7fbc4403d2b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:04:00.464 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:00.462+0000 7fbc6408e700 1 --2- 172.21.15.42:0/1084505422 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbc4403ada0 0x7fbc4403d2b0 crc :-1 s=READY pgs=835 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:04:00.464 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:00.462+0000 7fbc6408e700 1 -- 172.21.15.42:0/1084505422 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbc5c130520 msgr2=0x7fbc5c119650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:04:00.464 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:00.462+0000 7fbc6408e700 1 --2- 172.21.15.42:0/1084505422 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbc5c130520 0x7fbc5c119650 secure :-1 s=READY pgs=1969 cs=0 l=1 rx=0x7fbc58002f30 tx=0x7fbc58010950).stop 2019-12-05T06:04:00.465 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:00.462+0000 7fbc6408e700 1 -- 172.21.15.42:0/1084505422 shutdown_connections 2019-12-05T06:04:00.465 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:00.462+0000 7fbc6408e700 1 --2- 172.21.15.42:0/1084505422 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbc4403ada0 0x7fbc4403d2b0 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:04:00.465 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:00.462+0000 7fbc6408e700 1 --2- 172.21.15.42:0/1084505422 >> v2:172.21.15.40:3300/0 conn(0x7fbc5c0a4dc0 0x7fbc5c0a51f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:04:00.465 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:00.462+0000 7fbc6408e700 1 --2- 172.21.15.42:0/1084505422 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbc5c130520 0x7fbc5c119650 unknown :-1 s=CLOSED pgs=1969 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:04:00.465 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:00.462+0000 7fbc6408e700 1 --2- 172.21.15.42:0/1084505422 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbc5c07c880 0x7fbc5c07ccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:04:00.465 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:00.462+0000 7fbc6408e700 1 -- 172.21.15.42:0/1084505422 shutdown_connections 2019-12-05T06:04:00.465 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:00.462+0000 7fbc6408e700 1 -- 172.21.15.42:0/1084505422 wait complete. 2019-12-05T06:04:00.466 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:00.462+0000 7fbc6408e700 1 -- 172.21.15.42:0/1084505422 >> 172.21.15.42:0/1084505422 conn(0x7fbc5c08fda0 msgr2=0x7fbc5c0a9800 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:04:00.482 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:04:02.483 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:04:02.592 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:02.578+0000 7f2094734700 1 Processor -- start 2019-12-05T06:04:02.592 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:02.578+0000 7f2094734700 1 -- start start 2019-12-05T06:04:02.592 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:02.578+0000 7f2094734700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f208c0a7740 0x7f208c08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:04:02.592 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:02.578+0000 7f2094734700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f208c09fae0 0x7f208c07eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:04:02.593 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:02.578+0000 7f2094734700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f208c0a0710 con 0x7f208c07e720 2019-12-05T06:04:02.593 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:02.578+0000 7f2094734700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f208c0a0b00 con 0x7f208c09fae0 2019-12-05T06:04:02.593 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:02.578+0000 7f2094734700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f208c08e5b0 con 0x7f208c0a7740 2019-12-05T06:04:02.593 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:02.578+0000 7f2092cd1700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f208c0a7740 0x7f208c08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:04:02.593 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:02.578+0000 7f2092cd1700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f208c0a7740 0x7f208c08df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:47442/0 (socket says 172.21.15.42:47442) 2019-12-05T06:04:02.594 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:02.578+0000 7f2092cd1700 1 -- 172.21.15.42:0/3346524500 learned_addr learned my addr 172.21.15.42:0/3346524500 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:04:02.594 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:02.578+0000 7f20934d2700 1 --2- 172.21.15.42:0/3346524500 >> v2:172.21.15.40:3300/0 conn(0x7f208c09fae0 0x7f208c07eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:04:02.594 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:02.578+0000 7f2092cd1700 1 -- 172.21.15.42:0/3346524500 >> v2:172.21.15.40:3300/0 conn(0x7f208c09fae0 msgr2=0x7f208c07eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:04:02.594 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:02.578+0000 7f2092cd1700 1 --2- 172.21.15.42:0/3346524500 >> v2:172.21.15.40:3300/0 conn(0x7f208c09fae0 0x7f208c07eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:04:02.594 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:02.578+0000 7f2092cd1700 1 -- 172.21.15.42:0/3346524500 >> v1:172.21.15.40:6789/0 conn(0x7f208c07e720 legacy=0x7f208c09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:04:02.594 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:02.578+0000 7f2092cd1700 1 -- 172.21.15.42:0/3346524500 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f208c07f0a0 con 0x7f208c0a7740 2019-12-05T06:04:02.595 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:02.578+0000 7f2092cd1700 1 --2- 172.21.15.42:0/3346524500 >> v2:172.21.15.42:3300/0 conn(0x7f208c0a7740 0x7f208c08df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f2080002f30 tx=0x7f208000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:04:02.595 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:02.578+0000 7f2092cd1700 1 --2- 172.21.15.42:0/3346524500 >> v2:172.21.15.42:3300/0 conn(0x7f208c0a7740 0x7f208c08df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f2080002f30 tx=0x7f208000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:04:02.595 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:02.578+0000 7f2092cd1700 1 --2- 172.21.15.42:0/3346524500 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f208c0a7740 0x7f208c08df80 secure :-1 s=READY pgs=1970 cs=0 l=1 rx=0x7f2080002f30 tx=0x7f208000fd30).ready entity=mon.1 client_cookie=e6dc2910e4ab50db server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:04:02.595 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:02.578+0000 7f2092cd1700 1 --2- 172.21.15.42:0/3346524500 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f208c0a7740 0x7f208c08df80 secure :-1 s=THROTTLE_DONE pgs=1970 cs=0 l=1 rx=0x7f2080002f30 tx=0x7f208000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:04:02.595 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:02.578+0000 7f2092cd1700 1 --2- 172.21.15.42:0/3346524500 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f208c0a7740 0x7f208c08df80 secure :-1 s=THROTTLE_DONE pgs=1970 cs=0 l=1 rx=0x7f2080002f30 tx=0x7f208000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:04:02.595 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:02.578+0000 7f2091ccf700 1 -- 172.21.15.42:0/3346524500 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2080008c90 con 0x7f208c0a7740 2019-12-05T06:04:02.595 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:02.578+0000 7f2092cd1700 1 --2- 172.21.15.42:0/3346524500 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f208c0a7740 0x7f208c08df80 secure :-1 s=THROTTLE_DONE pgs=1970 cs=0 l=1 rx=0x7f2080002f30 tx=0x7f208000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:04:02.596 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:02.578+0000 7f2091ccf700 1 -- 172.21.15.42:0/3346524500 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20800088b0 con 0x7f208c0a7740 2019-12-05T06:04:02.596 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:02.578+0000 7f2091ccf700 1 -- 172.21.15.42:0/3346524500 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2080011ad0 con 0x7f208c0a7740 2019-12-05T06:04:02.596 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:02.578+0000 7f2094734700 1 -- 172.21.15.42:0/3346524500 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f208c0a7740 msgr2=0x7f208c08df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:04:02.596 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:02.578+0000 7f2094734700 1 --2- 172.21.15.42:0/3346524500 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f208c0a7740 0x7f208c08df80 secure :-1 s=READY pgs=1970 cs=0 l=1 rx=0x7f2080002f30 tx=0x7f208000fd30).stop 2019-12-05T06:04:02.597 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:02.582+0000 7f2094734700 1 -- 172.21.15.42:0/3346524500 shutdown_connections 2019-12-05T06:04:02.597 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:02.582+0000 7f2094734700 1 --2- 172.21.15.42:0/3346524500 >> v2:172.21.15.40:3300/0 conn(0x7f208c09fae0 0x7f208c07eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:04:02.597 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:02.582+0000 7f2094734700 1 --2- 172.21.15.42:0/3346524500 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f208c0a7740 0x7f208c08df80 unknown :-1 s=CLOSED pgs=1970 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:04:02.597 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:02.582+0000 7f2094734700 1 -- 172.21.15.42:0/3346524500 shutdown_connections 2019-12-05T06:04:02.597 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:02.582+0000 7f2094734700 1 -- 172.21.15.42:0/3346524500 wait complete. 2019-12-05T06:04:02.597 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:02.582+0000 7f2094734700 1 -- 172.21.15.42:0/3346524500 >> 172.21.15.42:0/3346524500 conn(0x7f208c0aabe0 msgr2=0x7f208c0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:04:02.597 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:02.582+0000 7f2094734700 1 Processor -- start 2019-12-05T06:04:02.598 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:02.582+0000 7f2094734700 1 -- start start 2019-12-05T06:04:02.598 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:02.582+0000 7f2094734700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f208c11d9c0 0x7f208c12e290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:04:02.598 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:02.582+0000 7f2094734700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f208c12e7e0 0x7f208c1220c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:04:02.598 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:02.582+0000 7f2094734700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f208c11b6a0 con 0x7f208c122610 2019-12-05T06:04:02.598 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:02.582+0000 7f2094734700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f208c11ddf0 con 0x7f208c12e7e0 2019-12-05T06:04:02.599 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:02.582+0000 7f2094734700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f208c11aaa0 con 0x7f208c11d9c0 2019-12-05T06:04:02.599 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:02.582+0000 7f2092cd1700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f208c11d9c0 0x7f208c12e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:04:02.599 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:02.582+0000 7f20934d2700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f208c122610 0x7f208c0742d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:47488/0 (socket says 172.21.15.42:47488) 2019-12-05T06:04:02.599 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:02.582+0000 7f20934d2700 1 -- 172.21.15.42:0/3794249741 learned_addr learned my addr 172.21.15.42:0/3794249741 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:04:02.599 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:02.582+0000 7f20924d0700 1 --2- 172.21.15.42:0/3794249741 >> v2:172.21.15.40:3300/0 conn(0x7f208c12e7e0 0x7f208c1220c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:04:02.599 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:02.586+0000 7f2092cd1700 1 -- 172.21.15.42:0/3794249741 >> v2:172.21.15.40:3300/0 conn(0x7f208c12e7e0 msgr2=0x7f208c1220c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:04:02.600 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:02.586+0000 7f2092cd1700 1 --2- 172.21.15.42:0/3794249741 >> v2:172.21.15.40:3300/0 conn(0x7f208c12e7e0 0x7f208c1220c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:04:02.600 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:02.586+0000 7f2092cd1700 1 -- 172.21.15.42:0/3794249741 >> v1:172.21.15.42:6789/0 conn(0x7f208c122610 legacy=0x7f208c0742d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:04:02.600 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:02.586+0000 7f2092cd1700 1 -- 172.21.15.42:0/3794249741 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f208c11a7e0 con 0x7f208c11d9c0 2019-12-05T06:04:02.600 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:02.586+0000 7f208b7fe700 1 -- 172.21.15.42:0/3794249741 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1823740167 0 0) 0x7f208c11b6a0 con 0x7f208c122610 2019-12-05T06:04:02.601 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:02.586+0000 7f2092cd1700 1 --2- 172.21.15.42:0/3794249741 >> v2:172.21.15.42:3300/0 conn(0x7f208c11d9c0 0x7f208c12e290 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f2080002f30 tx=0x7f2080008b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:04:02.601 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:02.586+0000 7f2092cd1700 1 --2- 172.21.15.42:0/3794249741 >> v2:172.21.15.42:3300/0 conn(0x7f208c11d9c0 0x7f208c12e290 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f2080002f30 tx=0x7f2080008b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:04:02.601 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:02.586+0000 7f2092cd1700 1 --2- 172.21.15.42:0/3794249741 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f208c11d9c0 0x7f208c12e290 secure :-1 s=READY pgs=1972 cs=0 l=1 rx=0x7f2080002f30 tx=0x7f2080008b60).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:04:02.601 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:02.586+0000 7f2092cd1700 1 --2- 172.21.15.42:0/3794249741 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f208c11d9c0 0x7f208c12e290 secure :-1 s=THROTTLE_DONE pgs=1972 cs=0 l=1 rx=0x7f2080002f30 tx=0x7f2080008b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:04:02.601 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:02.586+0000 7f208b7fe700 1 -- 172.21.15.42:0/3794249741 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f208000f490 con 0x7f208c11d9c0 2019-12-05T06:04:02.601 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:02.586+0000 7f2094734700 1 -- 172.21.15.42:0/3794249741 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f208c12ec10 con 0x7f208c11d9c0 2019-12-05T06:04:02.601 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:02.586+0000 7f2092cd1700 1 --2- 172.21.15.42:0/3794249741 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f208c11d9c0 0x7f208c12e290 secure :-1 s=THROTTLE_DONE pgs=1972 cs=0 l=1 rx=0x7f2080002f30 tx=0x7f2080008b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:04:02.602 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:02.586+0000 7f208b7fe700 1 -- 172.21.15.42:0/3794249741 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20800116f0 con 0x7f208c11d9c0 2019-12-05T06:04:02.602 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:02.586+0000 7f2092cd1700 1 --2- 172.21.15.42:0/3794249741 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f208c11d9c0 0x7f208c12e290 secure :-1 s=THROTTLE_DONE pgs=1972 cs=0 l=1 rx=0x7f2080002f30 tx=0x7f2080008b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:04:02.602 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:02.586+0000 7f208b7fe700 1 -- 172.21.15.42:0/3794249741 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2080005730 con 0x7f208c11d9c0 2019-12-05T06:04:02.602 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:02.586+0000 7f2094734700 1 -- 172.21.15.42:0/3794249741 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f208c11c030 con 0x7f208c11d9c0 2019-12-05T06:04:02.602 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:02.586+0000 7f2092cd1700 1 --2- 172.21.15.42:0/3794249741 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f208c11d9c0 0x7f208c12e290 secure :-1 s=THROTTLE_DONE pgs=1972 cs=0 l=1 rx=0x7f2080002f30 tx=0x7f2080008b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:04:02.602 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:02.586+0000 7f208b7fe700 1 -- 172.21.15.42:0/3794249741 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f20800059d0 con 0x7f208c11d9c0 2019-12-05T06:04:02.602 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:02.586+0000 7f2092cd1700 1 --2- 172.21.15.42:0/3794249741 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f208c11d9c0 0x7f208c12e290 secure :-1 s=THROTTLE_DONE pgs=1972 cs=0 l=1 rx=0x7f2080002f30 tx=0x7f2080008b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:04:02.603 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:02.586+0000 7f208b7fe700 1 --2- 172.21.15.42:0/3794249741 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f206c03ac10 0x7f206c03d120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:04:02.603 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:02.586+0000 7f208b7fe700 1 -- 172.21.15.42:0/3794249741 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9557+0+0 (secure 0 0 0) 0x7f208004b9c0 con 0x7f208c11d9c0 2019-12-05T06:04:02.603 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:02.586+0000 7f20924d0700 1 --2- 172.21.15.42:0/3794249741 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f206c03ac10 0x7f206c03d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:04:02.603 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:02.586+0000 7f20924d0700 1 --2- 172.21.15.42:0/3794249741 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f206c03ac10 0x7f206c03d120 crc :-1 s=READY pgs=836 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:04:02.604 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:02.594+0000 7f2094734700 1 -- 172.21.15.42:0/3794249741 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f208c05d110 con 0x7f208c11d9c0 2019-12-05T06:04:02.604 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:02.598+0000 7f2092cd1700 1 --2- 172.21.15.42:0/3794249741 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f208c11d9c0 0x7f208c12e290 secure :-1 s=THROTTLE_DONE pgs=1972 cs=0 l=1 rx=0x7f2080002f30 tx=0x7f2080008b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:04:02.609 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:02.598+0000 7f208b7fe700 1 -- 172.21.15.42:0/3794249741 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f208c11a7e0 con 0x7f208c11d9c0 2019-12-05T06:04:02.765 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:02.762+0000 7f2094734700 1 -- 172.21.15.42:0/3794249741 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f208c05b8c0 con 0x7f208c11d9c0 2019-12-05T06:04:02.765 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:02.762+0000 7f2092cd1700 1 --2- 172.21.15.42:0/3794249741 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f208c11d9c0 0x7f208c12e290 secure :-1 s=THROTTLE_DONE pgs=1972 cs=0 l=1 rx=0x7f2080002f30 tx=0x7f2080008b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:04:02.766 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:02.762+0000 7f208b7fe700 1 -- 172.21.15.42:0/3794249741 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 238 v238) v1 ==== 95+0+2109 (secure 0 0 0) 0x7f2080033850 con 0x7f208c11d9c0 2019-12-05T06:04:02.766 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 238 2019-12-05T06:04:02.766 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:04:02.767 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":238,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":9381,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/3789081777","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3789081777},{"type":"v1","addr":"172.21.15.40:6833","nonce":3789081777}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":238}],"filesystems":[{"mdsmap":{"epoch":237,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:01:44.401238+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":90,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9292},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9292":{"gid":9292,"name":"a","rank":0,"incarnation":234,"state":"up:active","state_seq":12,"addr":"172.21.15.42:6835/613535662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":613535662},{"type":"v1","addr":"172.21.15.42:6835","nonce":613535662}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:04:02.769 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:02.766+0000 7f2094734700 1 -- 172.21.15.42:0/3794249741 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f206c03ac10 msgr2=0x7f206c03d120 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:04:02.770 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:02.766+0000 7f2094734700 1 --2- 172.21.15.42:0/3794249741 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f206c03ac10 0x7f206c03d120 crc :-1 s=READY pgs=836 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:04:02.770 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:02.766+0000 7f2094734700 1 -- 172.21.15.42:0/3794249741 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f208c11d9c0 msgr2=0x7f208c12e290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:04:02.770 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:02.766+0000 7f2094734700 1 --2- 172.21.15.42:0/3794249741 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f208c11d9c0 0x7f208c12e290 secure :-1 s=READY pgs=1972 cs=0 l=1 rx=0x7f2080002f30 tx=0x7f2080008b60).stop 2019-12-05T06:04:02.770 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:02.766+0000 7f2094734700 1 -- 172.21.15.42:0/3794249741 shutdown_connections 2019-12-05T06:04:02.771 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:02.766+0000 7f2094734700 1 --2- 172.21.15.42:0/3794249741 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f206c03ac10 0x7f206c03d120 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:04:02.771 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:02.766+0000 7f2094734700 1 --2- 172.21.15.42:0/3794249741 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f208c11d9c0 0x7f208c12e290 unknown :-1 s=CLOSED pgs=1972 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:04:02.771 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:02.766+0000 7f2094734700 1 --2- 172.21.15.42:0/3794249741 >> v2:172.21.15.40:3300/0 conn(0x7f208c12e7e0 0x7f208c1220c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:04:02.771 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:02.766+0000 7f2094734700 1 -- 172.21.15.42:0/3794249741 shutdown_connections 2019-12-05T06:04:02.771 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:02.766+0000 7f2094734700 1 -- 172.21.15.42:0/3794249741 wait complete. 2019-12-05T06:04:02.772 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:02.766+0000 7f2094734700 1 -- 172.21.15.42:0/3794249741 >> 172.21.15.42:0/3794249741 conn(0x7f208c0aabe0 msgr2=0x7f208c08eb50 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:04:02.781 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:04:04.783 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:04:04.891 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:04.882+0000 7fca869b6700 1 Processor -- start 2019-12-05T06:04:04.891 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:04.882+0000 7fca869b6700 1 -- start start 2019-12-05T06:04:04.891 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:04.882+0000 7fca869b6700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fca800a7740 0x7fca8008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:04:04.891 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:04.882+0000 7fca869b6700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fca8007e720 0x7fca8007eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:04:04.891 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:04.886+0000 7fca869b6700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca800a0710 con 0x7fca8009f2e0 2019-12-05T06:04:04.892 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:04.886+0000 7fca869b6700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca800a0b00 con 0x7fca800a7740 2019-12-05T06:04:04.892 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:04.886+0000 7fca869b6700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca8008e5b0 con 0x7fca8007e720 2019-12-05T06:04:04.892 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:04.886+0000 7fca84f53700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fca800a7740 0x7fca8008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:04:04.892 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:04.886+0000 7fca85754700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fca8009f2e0 0x7fca8009f710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:47502/0 (socket says 172.21.15.42:47502) 2019-12-05T06:04:04.892 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:04.886+0000 7fca85754700 1 -- 172.21.15.42:0/1453899565 learned_addr learned my addr 172.21.15.42:0/1453899565 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:04:04.892 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:04.886+0000 7fca7ffff700 1 --2- 172.21.15.42:0/1453899565 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fca8007e720 0x7fca8007eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:04:04.892 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:04.886+0000 7fca84f53700 1 -- 172.21.15.42:0/1453899565 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fca8007e720 msgr2=0x7fca8007eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:04:04.893 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:04.886+0000 7fca84f53700 1 --2- 172.21.15.42:0/1453899565 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fca8007e720 0x7fca8007eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:04:04.893 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:04.886+0000 7fca84f53700 1 -- 172.21.15.42:0/1453899565 >> v1:172.21.15.42:6789/0 conn(0x7fca8009f2e0 legacy=0x7fca8009f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:04:04.893 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:04.886+0000 7fca84f53700 1 -- 172.21.15.42:0/1453899565 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca8007f0a0 con 0x7fca800a7740 2019-12-05T06:04:04.893 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:04.886+0000 7fca84f53700 1 --2- 172.21.15.42:0/1453899565 >> v2:172.21.15.42:3300/0 conn(0x7fca800a7740 0x7fca8008df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fca68008790 tx=0x7fca780014d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:04:04.893 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:04.886+0000 7fca84f53700 1 --2- 172.21.15.42:0/1453899565 >> v2:172.21.15.42:3300/0 conn(0x7fca800a7740 0x7fca8008df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fca68008790 tx=0x7fca780014d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:04:04.893 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:04.886+0000 7fca84f53700 1 --2- 172.21.15.42:0/1453899565 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fca800a7740 0x7fca8008df80 secure :-1 s=READY pgs=1974 cs=0 l=1 rx=0x7fca68008790 tx=0x7fca780014d0).ready entity=mon.1 client_cookie=c6289bd90f80737a server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:04:04.893 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:04.886+0000 7fca84f53700 1 --2- 172.21.15.42:0/1453899565 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fca800a7740 0x7fca8008df80 secure :-1 s=THROTTLE_DONE pgs=1974 cs=0 l=1 rx=0x7fca68008790 tx=0x7fca780014d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:04:04.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:04.886+0000 7fca84f53700 1 --2- 172.21.15.42:0/1453899565 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fca800a7740 0x7fca8008df80 secure :-1 s=THROTTLE_DONE pgs=1974 cs=0 l=1 rx=0x7fca68008790 tx=0x7fca780014d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:04:04.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:04.886+0000 7fca7f7fe700 1 -- 172.21.15.42:0/1453899565 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fca68008bb0 con 0x7fca800a7740 2019-12-05T06:04:04.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:04.886+0000 7fca84f53700 1 --2- 172.21.15.42:0/1453899565 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fca800a7740 0x7fca8008df80 secure :-1 s=THROTTLE_DONE pgs=1974 cs=0 l=1 rx=0x7fca68008790 tx=0x7fca780014d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:04:04.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:04.886+0000 7fca7f7fe700 1 -- 172.21.15.42:0/1453899565 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca680087d0 con 0x7fca800a7740 2019-12-05T06:04:04.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:04.886+0000 7fca7f7fe700 1 -- 172.21.15.42:0/1453899565 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fca68011760 con 0x7fca800a7740 2019-12-05T06:04:04.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:04.886+0000 7fca869b6700 1 -- 172.21.15.42:0/1453899565 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fca800a7740 msgr2=0x7fca8008df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:04:04.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:04.886+0000 7fca869b6700 1 --2- 172.21.15.42:0/1453899565 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fca800a7740 0x7fca8008df80 secure :-1 s=READY pgs=1974 cs=0 l=1 rx=0x7fca68008790 tx=0x7fca780014d0).stop 2019-12-05T06:04:04.895 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:04.886+0000 7fca869b6700 1 -- 172.21.15.42:0/1453899565 shutdown_connections 2019-12-05T06:04:04.895 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:04.886+0000 7fca869b6700 1 --2- 172.21.15.42:0/1453899565 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fca800a7740 0x7fca8008df80 unknown :-1 s=CLOSED pgs=1974 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:04:04.895 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:04.886+0000 7fca869b6700 1 --2- 172.21.15.42:0/1453899565 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fca8007e720 0x7fca8007eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:04:04.895 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:04.886+0000 7fca869b6700 1 -- 172.21.15.42:0/1453899565 shutdown_connections 2019-12-05T06:04:04.895 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:04.886+0000 7fca869b6700 1 -- 172.21.15.42:0/1453899565 wait complete. 2019-12-05T06:04:04.895 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:04.886+0000 7fca869b6700 1 -- 172.21.15.42:0/1453899565 >> 172.21.15.42:0/1453899565 conn(0x7fca800aabe0 msgr2=0x7fca800a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:04:04.898 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:04.890+0000 7fca869b6700 1 Processor -- start 2019-12-05T06:04:04.898 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:04.890+0000 7fca869b6700 1 -- start start 2019-12-05T06:04:04.899 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:04.890+0000 7fca869b6700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fca8007b7c0 0x7fca80120900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:04:04.899 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:04.890+0000 7fca869b6700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fca80135850 0x7fca80137c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:04:04.899 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:04.890+0000 7fca869b6700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca80120e50 con 0x7fca8012cef0 2019-12-05T06:04:04.899 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:04.890+0000 7fca869b6700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca8007bff0 con 0x7fca8007b7c0 2019-12-05T06:04:04.899 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:04.890+0000 7fca869b6700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca8007c290 con 0x7fca80135850 2019-12-05T06:04:04.899 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:04.890+0000 7fca84f53700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fca8007b7c0 0x7fca80120900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:04:04.900 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:04.890+0000 7fca7ffff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fca8012cef0 0x7fca8012d320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:47508/0 (socket says 172.21.15.42:47508) 2019-12-05T06:04:04.900 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:04.890+0000 7fca7ffff700 1 -- 172.21.15.42:0/3163280020 learned_addr learned my addr 172.21.15.42:0/3163280020 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:04:04.900 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:04.890+0000 7fca85754700 1 --2- 172.21.15.42:0/3163280020 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fca80135850 0x7fca80137c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:04:04.900 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:04.890+0000 7fca85754700 1 -- 172.21.15.42:0/3163280020 >> v2:172.21.15.42:3300/0 conn(0x7fca8007b7c0 msgr2=0x7fca80120900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:04:04.900 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:04.890+0000 7fca85754700 1 --2- 172.21.15.42:0/3163280020 >> v2:172.21.15.42:3300/0 conn(0x7fca8007b7c0 0x7fca80120900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:04:04.900 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:04.890+0000 7fca85754700 1 -- 172.21.15.42:0/3163280020 >> v1:172.21.15.42:6789/0 conn(0x7fca8012cef0 legacy=0x7fca8012d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:04:04.900 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:04.890+0000 7fca85754700 1 -- 172.21.15.42:0/3163280020 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca8011ba10 con 0x7fca80135850 2019-12-05T06:04:04.901 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:04.890+0000 7fca84f53700 1 --2- 172.21.15.42:0/3163280020 >> v2:172.21.15.42:3300/0 conn(0x7fca8007b7c0 0x7fca80120900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:04:04.901 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:04.890+0000 7fca85754700 1 --2- 172.21.15.42:0/3163280020 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fca80135850 0x7fca80137c90 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fca78004720 tx=0x7fca78010d80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:04:04.901 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:04.890+0000 7fca85754700 1 --2- 172.21.15.42:0/3163280020 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fca80135850 0x7fca80137c90 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fca78004720 tx=0x7fca78010d80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:04:04.901 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:04.890+0000 7fca85754700 1 --2- 172.21.15.42:0/3163280020 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fca80135850 0x7fca80137c90 secure :-1 s=READY pgs=437 cs=0 l=1 rx=0x7fca78004720 tx=0x7fca78010d80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:04:04.901 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:04.890+0000 7fca85754700 1 --2- 172.21.15.42:0/3163280020 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fca80135850 0x7fca80137c90 secure :-1 s=THROTTLE_DONE pgs=437 cs=0 l=1 rx=0x7fca78004720 tx=0x7fca78010d80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:04:04.901 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:04.890+0000 7fca7d7fa700 1 -- 172.21.15.42:0/3163280020 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fca7800cc30 con 0x7fca80135850 2019-12-05T06:04:04.901 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:04.890+0000 7fca85754700 1 --2- 172.21.15.42:0/3163280020 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fca80135850 0x7fca80137c90 secure :-1 s=THROTTLE_DONE pgs=437 cs=0 l=1 rx=0x7fca78004720 tx=0x7fca78010d80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:04:04.902 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:04.890+0000 7fca7d7fa700 1 -- 172.21.15.42:0/3163280020 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca7800c850 con 0x7fca80135850 2019-12-05T06:04:04.902 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:04.890+0000 7fca85754700 1 --2- 172.21.15.42:0/3163280020 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fca80135850 0x7fca80137c90 secure :-1 s=THROTTLE_DONE pgs=437 cs=0 l=1 rx=0x7fca78004720 tx=0x7fca78010d80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:04:04.902 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:04.890+0000 7fca869b6700 1 -- 172.21.15.42:0/3163280020 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca8007c530 con 0x7fca80135850 2019-12-05T06:04:04.902 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:04.890+0000 7fca7d7fa700 1 -- 172.21.15.42:0/3163280020 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fca78004b90 con 0x7fca80135850 2019-12-05T06:04:04.902 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:04.890+0000 7fca869b6700 1 -- 172.21.15.42:0/3163280020 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca8007d5d0 con 0x7fca80135850 2019-12-05T06:04:04.902 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:04.894+0000 7fca85754700 1 --2- 172.21.15.42:0/3163280020 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fca80135850 0x7fca80137c90 secure :-1 s=THROTTLE_DONE pgs=437 cs=0 l=1 rx=0x7fca78004720 tx=0x7fca78010d80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:04:04.902 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:04.894+0000 7fca7d7fa700 1 -- 172.21.15.42:0/3163280020 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fca780127b0 con 0x7fca80135850 2019-12-05T06:04:04.903 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:04.894+0000 7fca85754700 1 --2- 172.21.15.42:0/3163280020 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fca80135850 0x7fca80137c90 secure :-1 s=THROTTLE_DONE pgs=437 cs=0 l=1 rx=0x7fca78004720 tx=0x7fca78010d80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:04:04.903 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:04.894+0000 7fca7d7fa700 1 --2- 172.21.15.42:0/3163280020 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fca6403ae20 0x7fca6403d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:04:04.903 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:04.894+0000 7fca7d7fa700 1 -- 172.21.15.42:0/3163280020 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9643+0+0 (secure 0 0 0) 0x7fca7804b330 con 0x7fca80135850 2019-12-05T06:04:04.903 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:04.894+0000 7fca84f53700 1 --2- 172.21.15.42:0/3163280020 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fca6403ae20 0x7fca6403d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:04:04.903 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:04.894+0000 7fca84f53700 1 --2- 172.21.15.42:0/3163280020 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fca6403ae20 0x7fca6403d330 crc :-1 s=READY pgs=837 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:04:04.904 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:04.898+0000 7fca869b6700 1 -- 172.21.15.42:0/3163280020 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca8005d110 con 0x7fca80135850 2019-12-05T06:04:04.911 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:04.906+0000 7fca85754700 1 --2- 172.21.15.42:0/3163280020 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fca80135850 0x7fca80137c90 secure :-1 s=THROTTLE_DONE pgs=437 cs=0 l=1 rx=0x7fca78004720 tx=0x7fca78010d80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:04:04.911 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:04.910+0000 7fca7d7fa700 1 -- 172.21.15.42:0/3163280020 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fca78032390 con 0x7fca80135850 2019-12-05T06:04:05.084 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:05.082+0000 7fca869b6700 1 -- 172.21.15.42:0/3163280020 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fca8005b8c0 con 0x7fca80135850 2019-12-05T06:04:05.084 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:05.082+0000 7fca85754700 1 --2- 172.21.15.42:0/3163280020 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fca80135850 0x7fca80137c90 secure :-1 s=THROTTLE_DONE pgs=437 cs=0 l=1 rx=0x7fca78004720 tx=0x7fca78010d80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:04:05.085 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:05.082+0000 7fca7d7fa700 1 -- 172.21.15.42:0/3163280020 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 239 v239) v1 ==== 95+0+1774 (secure 0 0 0) 0x7fca780325f0 con 0x7fca80135850 2019-12-05T06:04:05.087 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 239 2019-12-05T06:04:05.087 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:04:05.087 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":239,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":239,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:04:04.416716+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":91,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9381},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9381":{"gid":9381,"name":"b","rank":0,"incarnation":239,"state":"up:replay","state_seq":2,"addr":"172.21.15.40:6833/3789081777","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3789081777},{"type":"v1","addr":"172.21.15.40:6833","nonce":3789081777}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:04:05.091 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:05.086+0000 7fca869b6700 1 -- 172.21.15.42:0/3163280020 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fca6403ae20 msgr2=0x7fca6403d330 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:04:05.091 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:05.086+0000 7fca869b6700 1 --2- 172.21.15.42:0/3163280020 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fca6403ae20 0x7fca6403d330 crc :-1 s=READY pgs=837 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:04:05.091 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:05.086+0000 7fca869b6700 1 -- 172.21.15.42:0/3163280020 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fca80135850 msgr2=0x7fca80137c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:04:05.091 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:05.086+0000 7fca869b6700 1 --2- 172.21.15.42:0/3163280020 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fca80135850 0x7fca80137c90 secure :-1 s=READY pgs=437 cs=0 l=1 rx=0x7fca78004720 tx=0x7fca78010d80).stop 2019-12-05T06:04:05.091 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:05.086+0000 7fca869b6700 1 -- 172.21.15.42:0/3163280020 shutdown_connections 2019-12-05T06:04:05.091 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:05.086+0000 7fca869b6700 1 --2- 172.21.15.42:0/3163280020 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fca6403ae20 0x7fca6403d330 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:04:05.092 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:05.086+0000 7fca869b6700 1 --2- 172.21.15.42:0/3163280020 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fca80135850 0x7fca80137c90 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:04:05.092 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:05.086+0000 7fca869b6700 1 --2- 172.21.15.42:0/3163280020 >> v2:172.21.15.42:3300/0 conn(0x7fca8007b7c0 0x7fca80120900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:04:05.092 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:05.086+0000 7fca869b6700 1 -- 172.21.15.42:0/3163280020 shutdown_connections 2019-12-05T06:04:05.092 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:05.086+0000 7fca869b6700 1 -- 172.21.15.42:0/3163280020 wait complete. 2019-12-05T06:04:05.092 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:05.086+0000 7fca869b6700 1 -- 172.21.15.42:0/3163280020 >> 172.21.15.42:0/3163280020 conn(0x7fca800aabe0 msgr2=0x7fca80059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:04:05.103 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a down, removed from mdsmap 2019-12-05T06:04:05.104 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2019-12-05T06:04:05.104 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:04:05.203 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:05.194+0000 7f2d124c0700 1 Processor -- start 2019-12-05T06:04:05.204 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:05.194+0000 7f2d124c0700 1 -- start start 2019-12-05T06:04:05.204 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:05.198+0000 7f2d124c0700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2d0c09fae0 0x7f2d0c14fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:04:05.204 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:05.198+0000 7f2d124c0700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d0c0a0710 con 0x7f2d0c08df80 2019-12-05T06:04:05.204 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:05.198+0000 7f2d124c0700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d0c07ef20 con 0x7f2d0c0a7740 2019-12-05T06:04:05.204 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:05.198+0000 7f2d124c0700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d0c0a0b00 con 0x7f2d0c09fae0 2019-12-05T06:04:05.204 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:05.198+0000 7f2d10a5d700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f2d0c0a7740 0x7f2d0c07e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:47512/0 (socket says 172.21.15.42:47512) 2019-12-05T06:04:05.205 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:05.198+0000 7f2d10a5d700 1 -- 172.21.15.42:0/441186316 learned_addr learned my addr 172.21.15.42:0/441186316 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:04:05.205 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:05.198+0000 7f2d1125e700 1 --2- 172.21.15.42:0/441186316 >> v2:172.21.15.40:3300/0 conn(0x7f2d0c09fae0 0x7f2d0c14fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:04:05.205 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:05.198+0000 7f2d0b7fe700 1 -- 172.21.15.42:0/441186316 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3458722747 0 0) 0x7f2d0c0a0710 con 0x7f2d0c08df80 2019-12-05T06:04:05.205 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:05.198+0000 7f2d0b7fe700 1 -- 172.21.15.42:0/441186316 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2cf80030b0 con 0x7f2d0c08df80 2019-12-05T06:04:05.205 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:05.198+0000 7f2d0b7fe700 1 -- 172.21.15.42:0/441186316 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1496352948 0 0) 0x7f2d0c07ef20 con 0x7f2d0c0a7740 2019-12-05T06:04:05.205 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:05.198+0000 7f2d0b7fe700 1 -- 172.21.15.42:0/441186316 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d0c0a0710 con 0x7f2d0c0a7740 2019-12-05T06:04:05.205 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:05.198+0000 7f2d1125e700 1 -- 172.21.15.42:0/441186316 >> v1:172.21.15.42:6789/0 conn(0x7f2d0c0a7740 legacy=0x7f2d0c07e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:04:05.206 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:05.198+0000 7f2d1125e700 1 -- 172.21.15.42:0/441186316 >> v1:172.21.15.40:6789/0 conn(0x7f2d0c08df80 legacy=0x7f2d0c09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:04:05.206 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:05.198+0000 7f2d1125e700 1 -- 172.21.15.42:0/441186316 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d0c0a9ba0 con 0x7f2d0c09fae0 2019-12-05T06:04:05.206 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:05.198+0000 7f2d1125e700 1 --2- 172.21.15.42:0/441186316 >> v2:172.21.15.40:3300/0 conn(0x7f2d0c09fae0 0x7f2d0c14fff0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f2d04008b10 tx=0x7f2d0400ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:04:05.206 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:05.198+0000 7f2d1125e700 1 --2- 172.21.15.42:0/441186316 >> v2:172.21.15.40:3300/0 conn(0x7f2d0c09fae0 0x7f2d0c14fff0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f2d04008b10 tx=0x7f2d0400ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:04:05.206 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:05.198+0000 7f2d1125e700 1 --2- 172.21.15.42:0/441186316 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2d0c09fae0 0x7f2d0c14fff0 secure :-1 s=READY pgs=1494 cs=0 l=1 rx=0x7f2d04008b10 tx=0x7f2d0400ff70).ready entity=mon.0 client_cookie=c236cd4faeea5f2f server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:04:05.206 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:05.198+0000 7f2d1125e700 1 --2- 172.21.15.42:0/441186316 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2d0c09fae0 0x7f2d0c14fff0 secure :-1 s=THROTTLE_DONE pgs=1494 cs=0 l=1 rx=0x7f2d04008b10 tx=0x7f2d0400ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:04:05.207 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:05.198+0000 7f2d0b7fe700 1 -- 172.21.15.42:0/441186316 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2d04010970 con 0x7f2d0c09fae0 2019-12-05T06:04:05.207 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:05.198+0000 7f2d1125e700 1 --2- 172.21.15.42:0/441186316 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2d0c09fae0 0x7f2d0c14fff0 secure :-1 s=THROTTLE_DONE pgs=1494 cs=0 l=1 rx=0x7f2d04008b10 tx=0x7f2d0400ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:04:05.207 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:05.198+0000 7f2d1125e700 1 --2- 172.21.15.42:0/441186316 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2d0c09fae0 0x7f2d0c14fff0 secure :-1 s=THROTTLE_DONE pgs=1494 cs=0 l=1 rx=0x7f2d04008b10 tx=0x7f2d0400ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:04:05.207 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:05.198+0000 7f2d0b7fe700 1 -- 172.21.15.42:0/441186316 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d04008b50 con 0x7f2d0c09fae0 2019-12-05T06:04:05.207 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:05.198+0000 7f2d0b7fe700 1 -- 172.21.15.42:0/441186316 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2d040110c0 con 0x7f2d0c09fae0 2019-12-05T06:04:05.207 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:05.198+0000 7f2d124c0700 1 -- 172.21.15.42:0/441186316 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2d0c09fae0 msgr2=0x7f2d0c14fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:04:05.207 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:05.198+0000 7f2d124c0700 1 --2- 172.21.15.42:0/441186316 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2d0c09fae0 0x7f2d0c14fff0 secure :-1 s=READY pgs=1494 cs=0 l=1 rx=0x7f2d04008b10 tx=0x7f2d0400ff70).stop 2019-12-05T06:04:05.208 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:05.198+0000 7f2d124c0700 1 -- 172.21.15.42:0/441186316 shutdown_connections 2019-12-05T06:04:05.208 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:05.198+0000 7f2d124c0700 1 --2- 172.21.15.42:0/441186316 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2d0c09fae0 0x7f2d0c14fff0 unknown :-1 s=CLOSED pgs=1494 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:04:05.208 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:05.198+0000 7f2d124c0700 1 -- 172.21.15.42:0/441186316 shutdown_connections 2019-12-05T06:04:05.208 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:05.198+0000 7f2d124c0700 1 -- 172.21.15.42:0/441186316 wait complete. 2019-12-05T06:04:05.208 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:05.198+0000 7f2d124c0700 1 -- 172.21.15.42:0/441186316 >> 172.21.15.42:0/441186316 conn(0x7f2d0c0aabe0 msgr2=0x7f2d0c0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:04:05.211 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:05.202+0000 7f2d124c0700 1 Processor -- start 2019-12-05T06:04:05.212 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:05.202+0000 7f2d124c0700 1 -- start start 2019-12-05T06:04:05.212 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:05.202+0000 7f2d124c0700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2d0c127740 0x7f2d0c0a6a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:04:05.212 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:05.202+0000 7f2d124c0700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2d0c1434e0 0x7f2d0c143910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:04:05.212 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:05.202+0000 7f2d124c0700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d0c0a6fa0 con 0x7f2d0c07c3f0 2019-12-05T06:04:05.212 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:05.202+0000 7f2d124c0700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d0c07c150 con 0x7f2d0c1434e0 2019-12-05T06:04:05.212 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:05.202+0000 7f2d124c0700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d0c142610 con 0x7f2d0c127740 2019-12-05T06:04:05.213 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:05.202+0000 7f2d0bfff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2d0c1434e0 0x7f2d0c143910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:04:05.213 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:05.202+0000 7f2d1125e700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f2d0c07c3f0 0x7f2d0c08eb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:56962/0 (socket says 172.21.15.42:56962) 2019-12-05T06:04:05.213 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:05.202+0000 7f2d1125e700 1 -- 172.21.15.42:0/4065544149 learned_addr learned my addr 172.21.15.42:0/4065544149 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:04:05.213 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:05.202+0000 7f2d10a5d700 1 --2- 172.21.15.42:0/4065544149 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2d0c127740 0x7f2d0c0a6a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:04:05.213 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:05.202+0000 7f2d097fa700 1 -- 172.21.15.42:0/4065544149 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2419402528 0 0) 0x7f2d0c0a6fa0 con 0x7f2d0c07c3f0 2019-12-05T06:04:05.213 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:05.202+0000 7f2d097fa700 1 -- 172.21.15.42:0/4065544149 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2cf00030b0 con 0x7f2d0c07c3f0 2019-12-05T06:04:05.213 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:05.202+0000 7f2d0bfff700 1 -- 172.21.15.42:0/4065544149 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2d0c127740 msgr2=0x7f2d0c0a6a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:04:05.214 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:05.202+0000 7f2d0bfff700 1 --2- 172.21.15.42:0/4065544149 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2d0c127740 0x7f2d0c0a6a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:04:05.214 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:05.202+0000 7f2d0bfff700 1 -- 172.21.15.42:0/4065544149 >> v1:172.21.15.40:6789/0 conn(0x7f2d0c07c3f0 legacy=0x7f2d0c08eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:04:05.214 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:05.202+0000 7f2d0bfff700 1 -- 172.21.15.42:0/4065544149 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d0c07d4b0 con 0x7f2d0c1434e0 2019-12-05T06:04:05.214 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:05.206+0000 7f2d0bfff700 1 --2- 172.21.15.42:0/4065544149 >> v2:172.21.15.42:3300/0 conn(0x7f2d0c1434e0 0x7f2d0c143910 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f2d0c07c070 tx=0x7f2cf40108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:04:05.214 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:05.206+0000 7f2d0bfff700 1 --2- 172.21.15.42:0/4065544149 >> v2:172.21.15.42:3300/0 conn(0x7f2d0c1434e0 0x7f2d0c143910 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f2d0c07c070 tx=0x7f2cf40108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:04:05.214 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:05.206+0000 7f2d0bfff700 1 --2- 172.21.15.42:0/4065544149 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2d0c1434e0 0x7f2d0c143910 secure :-1 s=READY pgs=1977 cs=0 l=1 rx=0x7f2d0c07c070 tx=0x7f2cf40108f0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:04:05.215 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:05.206+0000 7f2d0bfff700 1 --2- 172.21.15.42:0/4065544149 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2d0c1434e0 0x7f2d0c143910 secure :-1 s=THROTTLE_DONE pgs=1977 cs=0 l=1 rx=0x7f2d0c07c070 tx=0x7f2cf40108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:04:05.215 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:05.206+0000 7f2d0bfff700 1 --2- 172.21.15.42:0/4065544149 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2d0c1434e0 0x7f2d0c143910 secure :-1 s=THROTTLE_DONE pgs=1977 cs=0 l=1 rx=0x7f2d0c07c070 tx=0x7f2cf40108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:04:05.215 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:05.206+0000 7f2d0bfff700 1 --2- 172.21.15.42:0/4065544149 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2d0c1434e0 0x7f2d0c143910 secure :-1 s=THROTTLE_DONE pgs=1977 cs=0 l=1 rx=0x7f2d0c07c070 tx=0x7f2cf40108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:04:05.215 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:05.206+0000 7f2d097fa700 1 -- 172.21.15.42:0/4065544149 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2cf400cb90 con 0x7f2d0c1434e0 2019-12-05T06:04:05.215 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:05.206+0000 7f2d097fa700 1 -- 172.21.15.42:0/4065544149 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cf400c7b0 con 0x7f2d0c1434e0 2019-12-05T06:04:05.215 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:05.206+0000 7f2d124c0700 1 -- 172.21.15.42:0/4065544149 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d0c07c820 con 0x7f2d0c1434e0 2019-12-05T06:04:05.216 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:05.206+0000 7f2d097fa700 1 -- 172.21.15.42:0/4065544149 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2cf400d6a0 con 0x7f2d0c1434e0 2019-12-05T06:04:05.216 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:05.206+0000 7f2d124c0700 1 -- 172.21.15.42:0/4065544149 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d0c07cc30 con 0x7f2d0c1434e0 2019-12-05T06:04:05.216 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:05.206+0000 7f2d0bfff700 1 --2- 172.21.15.42:0/4065544149 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2d0c1434e0 0x7f2d0c143910 secure :-1 s=THROTTLE_DONE pgs=1977 cs=0 l=1 rx=0x7f2d0c07c070 tx=0x7f2cf40108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:04:05.216 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:05.206+0000 7f2d097fa700 1 -- 172.21.15.42:0/4065544149 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f2cf400d8a0 con 0x7f2d0c1434e0 2019-12-05T06:04:05.216 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:05.206+0000 7f2d0bfff700 1 --2- 172.21.15.42:0/4065544149 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2d0c1434e0 0x7f2d0c143910 secure :-1 s=THROTTLE_DONE pgs=1977 cs=0 l=1 rx=0x7f2d0c07c070 tx=0x7f2cf40108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:04:05.216 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:05.206+0000 7f2d097fa700 1 --2- 172.21.15.42:0/4065544149 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2cf003b320 0x7f2cf003d830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:04:05.216 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:05.206+0000 7f2d097fa700 1 -- 172.21.15.42:0/4065544149 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9643+0+0 (secure 0 0 0) 0x7f2cf404b1e0 con 0x7f2d0c1434e0 2019-12-05T06:04:05.217 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:05.206+0000 7f2d10a5d700 1 --2- 172.21.15.42:0/4065544149 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2cf003b320 0x7f2cf003d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:04:05.217 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:05.206+0000 7f2d10a5d700 1 --2- 172.21.15.42:0/4065544149 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2cf003b320 0x7f2cf003d830 crc :-1 s=READY pgs=838 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:04:05.217 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:05.210+0000 7f2d124c0700 1 -- 172.21.15.42:0/4065544149 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d0c05d110 con 0x7f2d0c1434e0 2019-12-05T06:04:05.222 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:05.218+0000 7f2d0bfff700 1 --2- 172.21.15.42:0/4065544149 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2d0c1434e0 0x7f2d0c143910 secure :-1 s=THROTTLE_DONE pgs=1977 cs=0 l=1 rx=0x7f2d0c07c070 tx=0x7f2cf40108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:04:05.223 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:05.218+0000 7f2d097fa700 1 -- 172.21.15.42:0/4065544149 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f2cf4032260 con 0x7f2d0c1434e0 2019-12-05T06:04:05.396 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:05.394+0000 7f2d124c0700 1 -- 172.21.15.42:0/4065544149 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2d0c0a90e0 con 0x7f2d0c1434e0 2019-12-05T06:04:05.397 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:05.394+0000 7f2d0bfff700 1 --2- 172.21.15.42:0/4065544149 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2d0c1434e0 0x7f2d0c143910 secure :-1 s=THROTTLE_DONE pgs=1977 cs=0 l=1 rx=0x7f2d0c07c070 tx=0x7f2cf40108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:04:05.397 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:05.394+0000 7f2d097fa700 1 -- 172.21.15.42:0/4065544149 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 239 v239) v1 ==== 95+0+1774 (secure 0 0 0) 0x7f2cf40324c0 con 0x7f2d0c1434e0 2019-12-05T06:04:05.397 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 239 2019-12-05T06:04:05.398 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:04:05.398 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":239,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":239,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:04:04.416716+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":91,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9381},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9381":{"gid":9381,"name":"b","rank":0,"incarnation":239,"state":"up:replay","state_seq":2,"addr":"172.21.15.40:6833/3789081777","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3789081777},{"type":"v1","addr":"172.21.15.40:6833","nonce":3789081777}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:04:05.400 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:05.398+0000 7f2d124c0700 1 -- 172.21.15.42:0/4065544149 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2cf003b320 msgr2=0x7f2cf003d830 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:04:05.400 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:05.398+0000 7f2d124c0700 1 --2- 172.21.15.42:0/4065544149 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2cf003b320 0x7f2cf003d830 crc :-1 s=READY pgs=838 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:04:05.400 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:05.398+0000 7f2d124c0700 1 -- 172.21.15.42:0/4065544149 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2d0c1434e0 msgr2=0x7f2d0c143910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:04:05.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:05.398+0000 7f2d124c0700 1 --2- 172.21.15.42:0/4065544149 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2d0c1434e0 0x7f2d0c143910 secure :-1 s=READY pgs=1977 cs=0 l=1 rx=0x7f2d0c07c070 tx=0x7f2cf40108f0).stop 2019-12-05T06:04:05.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:05.398+0000 7f2d124c0700 1 -- 172.21.15.42:0/4065544149 shutdown_connections 2019-12-05T06:04:05.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:05.398+0000 7f2d124c0700 1 --2- 172.21.15.42:0/4065544149 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2cf003b320 0x7f2cf003d830 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:04:05.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:05.398+0000 7f2d124c0700 1 --2- 172.21.15.42:0/4065544149 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2d0c127740 0x7f2d0c0a6a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:04:05.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:05.398+0000 7f2d124c0700 1 --2- 172.21.15.42:0/4065544149 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2d0c1434e0 0x7f2d0c143910 unknown :-1 s=CLOSED pgs=1977 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:04:05.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:05.398+0000 7f2d124c0700 1 -- 172.21.15.42:0/4065544149 shutdown_connections 2019-12-05T06:04:05.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:05.398+0000 7f2d124c0700 1 -- 172.21.15.42:0/4065544149 wait complete. 2019-12-05T06:04:05.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:05.398+0000 7f2d124c0700 1 -- 172.21.15.42:0/4065544149 >> 172.21.15.42:0/4065544149 conn(0x7f2d0c0a80a0 msgr2=0x7f2d0c111db0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:04:05.410 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 239, "feature_flags": { "enable_multiple": false, "ever_enabled_multiple": false }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2019-12-05T05:03:35.604738+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 239, "ever_allowed_features": 2, "explicitly_allowed_features": 2, "failed": [], "flags": 18, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_9381": { "addr": "172.21.15.40:6833/3789081777", "addrs": { "addrvec": [ { "addr": "172.21.15.40:6832", "nonce": 3789081777, "type": "v2" }, { "addr": "172.21.15.40:6833", "nonce": 3789081777, "type": "v1" } ] }, "export_targets": [], "features": 4540138292836696063, "flags": 0, "gid": 9381, "incarnation": 239, "name": "b", "rank": 0, "state": "up:replay", "state_seq": 2 } }, "last_failure": 0, "last_failure_osd_epoch": 91, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 2, "min_compat_client": "0 (unknown)", "modified": "2019-12-05T06:04:04.416716+0000", "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 9381 } } } ], "standbys": [] } 2019-12-05T06:04:07.412 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:04:07.524 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:07.514+0000 7f123d303700 1 Processor -- start 2019-12-05T06:04:07.524 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:07.514+0000 7f123d303700 1 -- start start 2019-12-05T06:04:07.525 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:07.518+0000 7f123d303700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f123809fae0 0x7f123814fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:04:07.525 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:07.518+0000 7f123d303700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12380a0710 con 0x7f12380a7740 2019-12-05T06:04:07.525 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:07.518+0000 7f123d303700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f123807ef20 con 0x7f123808df80 2019-12-05T06:04:07.525 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:07.518+0000 7f123d303700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12380a0b00 con 0x7f123809fae0 2019-12-05T06:04:07.525 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:07.518+0000 7f1237fff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f123809fae0 0x7f123814fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:04:07.526 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:07.518+0000 7f1236ffd700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f123808df80 0x7f123809f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:47532/0 (socket says 172.21.15.42:47532) 2019-12-05T06:04:07.526 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:07.518+0000 7f1236ffd700 1 -- 172.21.15.42:0/3672934908 learned_addr learned my addr 172.21.15.42:0/3672934908 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:04:07.526 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:07.518+0000 7f1237fff700 1 -- 172.21.15.42:0/3672934908 >> v1:172.21.15.42:6789/0 conn(0x7f123808df80 legacy=0x7f123809f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:04:07.526 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:07.518+0000 7f1237fff700 1 -- 172.21.15.42:0/3672934908 >> v1:172.21.15.40:6789/0 conn(0x7f12380a7740 legacy=0x7f123807e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:04:07.526 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:07.518+0000 7f1237fff700 1 -- 172.21.15.42:0/3672934908 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12380a9ba0 con 0x7f123809fae0 2019-12-05T06:04:07.527 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:07.518+0000 7f12367fc700 1 -- 172.21.15.42:0/3672934908 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1399865476 0 0) 0x7f12380a0710 con 0x7f12380a7740 2019-12-05T06:04:07.527 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:07.518+0000 7f1237fff700 1 --2- 172.21.15.42:0/3672934908 >> v2:172.21.15.42:3300/0 conn(0x7f123809fae0 0x7f123814fff0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f1230008b10 tx=0x7f123000ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:04:07.527 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:07.518+0000 7f1237fff700 1 --2- 172.21.15.42:0/3672934908 >> v2:172.21.15.42:3300/0 conn(0x7f123809fae0 0x7f123814fff0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f1230008b10 tx=0x7f123000ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:04:07.527 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:07.518+0000 7f1237fff700 1 --2- 172.21.15.42:0/3672934908 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f123809fae0 0x7f123814fff0 secure :-1 s=READY pgs=1979 cs=0 l=1 rx=0x7f1230008b10 tx=0x7f123000ff70).ready entity=mon.1 client_cookie=37aa0b589af64d7a server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:04:07.527 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:07.518+0000 7f1237fff700 1 --2- 172.21.15.42:0/3672934908 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f123809fae0 0x7f123814fff0 secure :-1 s=THROTTLE_DONE pgs=1979 cs=0 l=1 rx=0x7f1230008b10 tx=0x7f123000ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:04:07.528 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:07.518+0000 7f12367fc700 1 -- 172.21.15.42:0/3672934908 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1230010970 con 0x7f123809fae0 2019-12-05T06:04:07.528 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:07.518+0000 7f1237fff700 1 --2- 172.21.15.42:0/3672934908 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f123809fae0 0x7f123814fff0 secure :-1 s=THROTTLE_DONE pgs=1979 cs=0 l=1 rx=0x7f1230008b10 tx=0x7f123000ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:04:07.528 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:07.518+0000 7f1237fff700 1 --2- 172.21.15.42:0/3672934908 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f123809fae0 0x7f123814fff0 secure :-1 s=THROTTLE_DONE pgs=1979 cs=0 l=1 rx=0x7f1230008b10 tx=0x7f123000ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:04:07.528 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:07.518+0000 7f12367fc700 1 -- 172.21.15.42:0/3672934908 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1230008b50 con 0x7f123809fae0 2019-12-05T06:04:07.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:07.518+0000 7f12367fc700 1 -- 172.21.15.42:0/3672934908 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f12300110c0 con 0x7f123809fae0 2019-12-05T06:04:07.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:07.518+0000 7f123d303700 1 -- 172.21.15.42:0/3672934908 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f123809fae0 msgr2=0x7f123814fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:04:07.530 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:07.518+0000 7f123d303700 1 --2- 172.21.15.42:0/3672934908 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f123809fae0 0x7f123814fff0 secure :-1 s=READY pgs=1979 cs=0 l=1 rx=0x7f1230008b10 tx=0x7f123000ff70).stop 2019-12-05T06:04:07.530 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:07.518+0000 7f123d303700 1 -- 172.21.15.42:0/3672934908 shutdown_connections 2019-12-05T06:04:07.530 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:07.518+0000 7f123d303700 1 --2- 172.21.15.42:0/3672934908 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f123809fae0 0x7f123814fff0 unknown :-1 s=CLOSED pgs=1979 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:04:07.530 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:07.518+0000 7f123d303700 1 -- 172.21.15.42:0/3672934908 shutdown_connections 2019-12-05T06:04:07.530 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:07.518+0000 7f123d303700 1 -- 172.21.15.42:0/3672934908 wait complete. 2019-12-05T06:04:07.531 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:07.518+0000 7f123d303700 1 -- 172.21.15.42:0/3672934908 >> 172.21.15.42:0/3672934908 conn(0x7f12380aabe0 msgr2=0x7f12380a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:04:07.535 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:07.522+0000 7f123d303700 1 Processor -- start 2019-12-05T06:04:07.535 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:07.522+0000 7f123d303700 1 -- start start 2019-12-05T06:04:07.535 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:07.522+0000 7f123d303700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1238115b00 0x7f123812bd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:04:07.535 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:07.522+0000 7f123d303700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1238142530 con 0x7f1238143010 2019-12-05T06:04:07.536 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:07.522+0000 7f123d303700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1238124660 con 0x7f1238116c90 2019-12-05T06:04:07.536 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:07.522+0000 7f123d303700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1238143520 con 0x7f1238115b00 2019-12-05T06:04:07.536 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:07.522+0000 7f1236ffd700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f1238116c90 0x7f1238115300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:47538/0 (socket says 172.21.15.42:47538) 2019-12-05T06:04:07.536 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:07.522+0000 7f1236ffd700 1 -- 172.21.15.42:0/3255332135 learned_addr learned my addr 172.21.15.42:0/3255332135 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:04:07.536 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:07.522+0000 7f1237fff700 1 --2- 172.21.15.42:0/3255332135 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1238115b00 0x7f123812bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:04:07.536 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:07.526+0000 7f121ffff700 1 -- 172.21.15.42:0/3255332135 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 887133288 0 0) 0x7f1238124660 con 0x7f1238116c90 2019-12-05T06:04:07.536 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:07.526+0000 7f121ffff700 1 -- 172.21.15.42:0/3255332135 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12180030b0 con 0x7f1238116c90 2019-12-05T06:04:07.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:07.526+0000 7f121ffff700 1 -- 172.21.15.42:0/3255332135 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4243042374 0 0) 0x7f1238142530 con 0x7f1238143010 2019-12-05T06:04:07.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:07.526+0000 7f121ffff700 1 -- 172.21.15.42:0/3255332135 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1238124660 con 0x7f1238143010 2019-12-05T06:04:07.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:07.526+0000 7f121ffff700 1 -- 172.21.15.42:0/3255332135 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 3366258870 0 0) 0x7f12180030b0 con 0x7f1238116c90 2019-12-05T06:04:07.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:07.526+0000 7f1237fff700 1 -- 172.21.15.42:0/3255332135 >> v1:172.21.15.42:6789/0 conn(0x7f1238116c90 legacy=0x7f1238115300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:04:07.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:07.526+0000 7f1237fff700 1 -- 172.21.15.42:0/3255332135 >> v1:172.21.15.40:6789/0 conn(0x7f1238143010 legacy=0x7f1238123e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:04:07.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:07.526+0000 7f1237fff700 1 -- 172.21.15.42:0/3255332135 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12380a6180 con 0x7f1238115b00 2019-12-05T06:04:07.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:07.526+0000 7f1237fff700 1 --2- 172.21.15.42:0/3255332135 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1238115b00 0x7f123812bd90 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f122c001740 tx=0x7f122c001b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:04:07.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:07.526+0000 7f1237fff700 1 --2- 172.21.15.42:0/3255332135 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1238115b00 0x7f123812bd90 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f122c001740 tx=0x7f122c001b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:04:07.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:07.526+0000 7f1237fff700 1 --2- 172.21.15.42:0/3255332135 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1238115b00 0x7f123812bd90 secure :-1 s=READY pgs=438 cs=0 l=1 rx=0x7f122c001740 tx=0x7f122c001b70).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:04:07.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:07.526+0000 7f1237fff700 1 --2- 172.21.15.42:0/3255332135 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1238115b00 0x7f123812bd90 secure :-1 s=THROTTLE_DONE pgs=438 cs=0 l=1 rx=0x7f122c001740 tx=0x7f122c001b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:04:07.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:07.526+0000 7f121ffff700 1 -- 172.21.15.42:0/3255332135 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f122c001970 con 0x7f1238115b00 2019-12-05T06:04:07.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:07.526+0000 7f1237fff700 1 --2- 172.21.15.42:0/3255332135 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1238115b00 0x7f123812bd90 secure :-1 s=THROTTLE_DONE pgs=438 cs=0 l=1 rx=0x7f122c001740 tx=0x7f122c001b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:04:07.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:07.526+0000 7f123d303700 1 -- 172.21.15.42:0/3255332135 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f123812c2e0 con 0x7f1238115b00 2019-12-05T06:04:07.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:07.526+0000 7f121ffff700 1 -- 172.21.15.42:0/3255332135 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f122c001780 con 0x7f1238115b00 2019-12-05T06:04:07.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:07.526+0000 7f1237fff700 1 --2- 172.21.15.42:0/3255332135 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1238115b00 0x7f123812bd90 secure :-1 s=THROTTLE_DONE pgs=438 cs=0 l=1 rx=0x7f122c001740 tx=0x7f122c001b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:04:07.539 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:07.526+0000 7f121ffff700 1 -- 172.21.15.42:0/3255332135 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1230005ca0 con 0x7f1238115b00 2019-12-05T06:04:07.539 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:07.526+0000 7f123d303700 1 -- 172.21.15.42:0/3255332135 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f123812c7b0 con 0x7f1238115b00 2019-12-05T06:04:07.539 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:07.526+0000 7f1237fff700 1 --2- 172.21.15.42:0/3255332135 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1238115b00 0x7f123812bd90 secure :-1 s=THROTTLE_DONE pgs=438 cs=0 l=1 rx=0x7f122c001740 tx=0x7f122c001b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:04:07.539 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:07.526+0000 7f121ffff700 1 -- 172.21.15.42:0/3255332135 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f123000f490 con 0x7f1238115b00 2019-12-05T06:04:07.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:07.526+0000 7f1237fff700 1 --2- 172.21.15.42:0/3255332135 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1238115b00 0x7f123812bd90 secure :-1 s=THROTTLE_DONE pgs=438 cs=0 l=1 rx=0x7f122c001740 tx=0x7f122c001b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:04:07.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:07.530+0000 7f121ffff700 1 --2- 172.21.15.42:0/3255332135 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f121803b420 0x7f121803d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:04:07.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:07.530+0000 7f121ffff700 1 -- 172.21.15.42:0/3255332135 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9643+0+0 (secure 0 0 0) 0x7f123004abb0 con 0x7f1238115b00 2019-12-05T06:04:07.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:07.530+0000 7f12377fe700 1 --2- 172.21.15.42:0/3255332135 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f121803b420 0x7f121803d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:04:07.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:07.530+0000 7f12377fe700 1 --2- 172.21.15.42:0/3255332135 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f121803b420 0x7f121803d930 crc :-1 s=READY pgs=839 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:04:07.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:07.530+0000 7f123d303700 1 -- 172.21.15.42:0/3255332135 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f123805d110 con 0x7f1238115b00 2019-12-05T06:04:07.543 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:07.542+0000 7f1237fff700 1 --2- 172.21.15.42:0/3255332135 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1238115b00 0x7f123812bd90 secure :-1 s=THROTTLE_DONE pgs=438 cs=0 l=1 rx=0x7f122c001740 tx=0x7f122c001b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:04:07.543 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:07.542+0000 7f121ffff700 1 -- 172.21.15.42:0/3255332135 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f1230031c10 con 0x7f1238115b00 2019-12-05T06:04:07.709 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:07.706+0000 7f123d303700 1 -- 172.21.15.42:0/3255332135 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f123805b8c0 con 0x7f1238115b00 2019-12-05T06:04:07.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:07.706+0000 7f1237fff700 1 --2- 172.21.15.42:0/3255332135 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1238115b00 0x7f123812bd90 secure :-1 s=THROTTLE_DONE pgs=438 cs=0 l=1 rx=0x7f122c001740 tx=0x7f122c001b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:04:07.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:07.706+0000 7f121ffff700 1 -- 172.21.15.42:0/3255332135 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 239 v239) v1 ==== 95+0+1774 (secure 0 0 0) 0x7f1230031e70 con 0x7f1238115b00 2019-12-05T06:04:07.712 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:04:07.712 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":239,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":239,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:04:04.416716+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":91,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9381},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9381":{"gid":9381,"name":"b","rank":0,"incarnation":239,"state":"up:replay","state_seq":2,"addr":"172.21.15.40:6833/3789081777","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3789081777},{"type":"v1","addr":"172.21.15.40:6833","nonce":3789081777}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:04:07.713 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 239 2019-12-05T06:04:07.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:07.710+0000 7f123d303700 1 -- 172.21.15.42:0/3255332135 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f121803b420 msgr2=0x7f121803d930 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:04:07.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:07.710+0000 7f123d303700 1 --2- 172.21.15.42:0/3255332135 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f121803b420 0x7f121803d930 crc :-1 s=READY pgs=839 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:04:07.718 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:07.710+0000 7f123d303700 1 -- 172.21.15.42:0/3255332135 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1238115b00 msgr2=0x7f123812bd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:04:07.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:07.710+0000 7f123d303700 1 --2- 172.21.15.42:0/3255332135 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1238115b00 0x7f123812bd90 secure :-1 s=READY pgs=438 cs=0 l=1 rx=0x7f122c001740 tx=0x7f122c001b70).stop 2019-12-05T06:04:07.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:07.710+0000 7f123d303700 1 -- 172.21.15.42:0/3255332135 shutdown_connections 2019-12-05T06:04:07.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:07.710+0000 7f123d303700 1 --2- 172.21.15.42:0/3255332135 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f121803b420 0x7f121803d930 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:04:07.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:07.710+0000 7f123d303700 1 --2- 172.21.15.42:0/3255332135 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1238115b00 0x7f123812bd90 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:04:07.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:07.714+0000 7f123d303700 1 -- 172.21.15.42:0/3255332135 shutdown_connections 2019-12-05T06:04:07.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:07.714+0000 7f123d303700 1 -- 172.21.15.42:0/3255332135 wait complete. 2019-12-05T06:04:07.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:07.714+0000 7f123d303700 1 -- 172.21.15.42:0/3255332135 >> 172.21.15.42:0/3255332135 conn(0x7f12380aabe0 msgr2=0x7f1238059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:04:07.725 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2019-12-05T06:04:09.726 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:04:09.844 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:09.830+0000 7f83a7789700 1 Processor -- start 2019-12-05T06:04:09.845 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:09.830+0000 7f83a7789700 1 -- start start 2019-12-05T06:04:09.845 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:09.830+0000 7f83a7789700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f83a008df80 0x7f83a009f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:04:09.845 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:09.830+0000 7f83a7789700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f83a009f830 0x7f83a014efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:04:09.845 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:09.830+0000 7f83a7789700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83a009fc60 con 0x7f83a00a7740 2019-12-05T06:04:09.846 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:09.830+0000 7f83a7789700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83a00a0b00 con 0x7f83a008df80 2019-12-05T06:04:09.846 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:09.830+0000 7f83a7789700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83a008e5b0 con 0x7f83a009f830 2019-12-05T06:04:09.852 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:09.834+0000 7f83a5d26700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f83a00a7740 0x7f83a007e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:47550/0 (socket says 172.21.15.42:47550) 2019-12-05T06:04:09.852 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:09.834+0000 7f83a5d26700 1 -- 172.21.15.42:0/841528870 learned_addr learned my addr 172.21.15.42:0/841528870 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:04:09.852 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:09.834+0000 7f83a6527700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f83a009f830 0x7f83a014efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:04:09.852 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:09.834+0000 7f83a5525700 1 --2- 172.21.15.42:0/841528870 >> v2:172.21.15.40:3300/0 conn(0x7f83a008df80 0x7f83a009f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:04:09.852 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:09.834+0000 7f83a4d24700 1 -- 172.21.15.42:0/841528870 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 819782866 0 0) 0x7f83a009fc60 con 0x7f83a00a7740 2019-12-05T06:04:09.853 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:09.834+0000 7f83a6527700 1 -- 172.21.15.42:0/841528870 >> v2:172.21.15.40:3300/0 conn(0x7f83a008df80 msgr2=0x7f83a009f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:04:09.853 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:09.834+0000 7f83a6527700 1 --2- 172.21.15.42:0/841528870 >> v2:172.21.15.40:3300/0 conn(0x7f83a008df80 0x7f83a009f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:04:09.853 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:09.834+0000 7f83a6527700 1 -- 172.21.15.42:0/841528870 >> v1:172.21.15.42:6789/0 conn(0x7f83a00a7740 legacy=0x7f83a007e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:04:09.853 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:09.834+0000 7f83a6527700 1 -- 172.21.15.42:0/841528870 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83a00a0710 con 0x7f83a009f830 2019-12-05T06:04:09.853 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:09.834+0000 7f83a6527700 1 --2- 172.21.15.42:0/841528870 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f83a009f830 0x7f83a014efe0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f8398008b10 tx=0x7f839800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:04:09.854 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:09.834+0000 7f83a6527700 1 --2- 172.21.15.42:0/841528870 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f83a009f830 0x7f83a014efe0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f8398008b10 tx=0x7f839800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:04:09.854 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:09.834+0000 7f83a6527700 1 --2- 172.21.15.42:0/841528870 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f83a009f830 0x7f83a014efe0 secure :-1 s=READY pgs=439 cs=0 l=1 rx=0x7f8398008b10 tx=0x7f839800fff0).ready entity=mon.2 client_cookie=a40b535cb6f123b2 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:04:09.854 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:09.834+0000 7f83a6527700 1 --2- 172.21.15.42:0/841528870 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f83a009f830 0x7f83a014efe0 secure :-1 s=THROTTLE_DONE pgs=439 cs=0 l=1 rx=0x7f8398008b10 tx=0x7f839800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:04:09.854 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:09.834+0000 7f83a6527700 1 --2- 172.21.15.42:0/841528870 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f83a009f830 0x7f83a014efe0 secure :-1 s=THROTTLE_DONE pgs=439 cs=0 l=1 rx=0x7f8398008b10 tx=0x7f839800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:04:09.855 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:09.834+0000 7f83a4d24700 1 -- 172.21.15.42:0/841528870 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8398010900 con 0x7f83a009f830 2019-12-05T06:04:09.855 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:09.834+0000 7f83a6527700 1 --2- 172.21.15.42:0/841528870 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f83a009f830 0x7f83a014efe0 secure :-1 s=THROTTLE_DONE pgs=439 cs=0 l=1 rx=0x7f8398008b10 tx=0x7f839800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:04:09.855 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:09.834+0000 7f83a4d24700 1 -- 172.21.15.42:0/841528870 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8398008b50 con 0x7f83a009f830 2019-12-05T06:04:09.855 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:09.834+0000 7f83a4d24700 1 -- 172.21.15.42:0/841528870 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f83980110c0 con 0x7f83a009f830 2019-12-05T06:04:09.855 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:09.834+0000 7f83a7789700 1 -- 172.21.15.42:0/841528870 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f83a009f830 msgr2=0x7f83a014efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:04:09.855 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:09.834+0000 7f83a7789700 1 --2- 172.21.15.42:0/841528870 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f83a009f830 0x7f83a014efe0 secure :-1 s=READY pgs=439 cs=0 l=1 rx=0x7f8398008b10 tx=0x7f839800fff0).stop 2019-12-05T06:04:09.855 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:09.834+0000 7f83a7789700 1 -- 172.21.15.42:0/841528870 shutdown_connections 2019-12-05T06:04:09.856 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:09.834+0000 7f83a7789700 1 --2- 172.21.15.42:0/841528870 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f83a009f830 0x7f83a014efe0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:04:09.856 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:09.834+0000 7f83a7789700 1 --2- 172.21.15.42:0/841528870 >> v2:172.21.15.40:3300/0 conn(0x7f83a008df80 0x7f83a009f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:04:09.856 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:09.834+0000 7f83a7789700 1 -- 172.21.15.42:0/841528870 shutdown_connections 2019-12-05T06:04:09.856 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:09.834+0000 7f83a7789700 1 -- 172.21.15.42:0/841528870 wait complete. 2019-12-05T06:04:09.856 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:09.834+0000 7f83a7789700 1 -- 172.21.15.42:0/841528870 >> 172.21.15.42:0/841528870 conn(0x7f83a00aabe0 msgr2=0x7f83a00a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:04:09.856 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:09.838+0000 7f83a7789700 1 Processor -- start 2019-12-05T06:04:09.869 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:09.838+0000 7f83a7789700 1 -- start start 2019-12-05T06:04:09.869 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:09.838+0000 7f83a7789700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f83a012ce70 0x7f83a0130d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:04:09.869 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:09.838+0000 7f83a7789700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f83a0123310 0x7f83a007ce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:04:09.870 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:09.838+0000 7f83a7789700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f83a007d3a0 0x7f83a01209b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:04:09.870 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:09.838+0000 7f83a7789700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83a007c300 con 0x7f83a0123310 2019-12-05T06:04:09.870 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:09.838+0000 7f83a7789700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83a0130540 con 0x7f83a007d3a0 2019-12-05T06:04:09.870 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:09.838+0000 7f83a7789700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83a01307e0 con 0x7f83a012ce70 2019-12-05T06:04:09.870 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:09.838+0000 7f83a6527700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f83a007d3a0 0x7f83a01209b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:04:09.870 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:09.838+0000 7f83a6527700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f83a007d3a0 0x7f83a01209b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:47522/0 (socket says 172.21.15.42:47522) 2019-12-05T06:04:09.871 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:09.838+0000 7f83a6527700 1 -- 172.21.15.42:0/2999431242 learned_addr learned my addr 172.21.15.42:0/2999431242 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:04:09.871 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:09.838+0000 7f83a5525700 1 --2- 172.21.15.42:0/2999431242 >> v2:172.21.15.40:3300/0 conn(0x7f83a0123310 0x7f83a007ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:04:09.871 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:09.838+0000 7f83a5d26700 1 --2- 172.21.15.42:0/2999431242 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f83a012ce70 0x7f83a0130d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:04:09.871 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:09.842+0000 7f83a6527700 1 -- 172.21.15.42:0/2999431242 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f83a012ce70 msgr2=0x7f83a0130d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:04:09.871 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:09.842+0000 7f83a6527700 1 --2- 172.21.15.42:0/2999431242 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f83a012ce70 0x7f83a0130d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:04:09.871 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:09.842+0000 7f83a6527700 1 -- 172.21.15.42:0/2999431242 >> v2:172.21.15.40:3300/0 conn(0x7f83a0123310 msgr2=0x7f83a007ce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:04:09.871 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:09.842+0000 7f83a6527700 1 --2- 172.21.15.42:0/2999431242 >> v2:172.21.15.40:3300/0 conn(0x7f83a0123310 0x7f83a007ce50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:04:09.872 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:09.842+0000 7f83a6527700 1 -- 172.21.15.42:0/2999431242 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83a0131270 con 0x7f83a007d3a0 2019-12-05T06:04:09.873 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:09.842+0000 7f83a6527700 1 --2- 172.21.15.42:0/2999431242 >> v2:172.21.15.42:3300/0 conn(0x7f83a007d3a0 0x7f83a01209b0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f83a011a5b0 tx=0x7f8398007ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:04:09.873 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:09.842+0000 7f83a6527700 1 --2- 172.21.15.42:0/2999431242 >> v2:172.21.15.42:3300/0 conn(0x7f83a007d3a0 0x7f83a01209b0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f83a011a5b0 tx=0x7f8398007ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:04:09.873 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:09.842+0000 7f83a6527700 1 --2- 172.21.15.42:0/2999431242 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f83a007d3a0 0x7f83a01209b0 secure :-1 s=READY pgs=1982 cs=0 l=1 rx=0x7f83a011a5b0 tx=0x7f8398007ae0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:04:09.873 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:09.842+0000 7f83a6527700 1 --2- 172.21.15.42:0/2999431242 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f83a007d3a0 0x7f83a01209b0 secure :-1 s=THROTTLE_DONE pgs=1982 cs=0 l=1 rx=0x7f83a011a5b0 tx=0x7f8398007ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:04:09.874 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:09.842+0000 7f83967fc700 1 -- 172.21.15.42:0/2999431242 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8398005ab0 con 0x7f83a007d3a0 2019-12-05T06:04:09.874 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:09.842+0000 7f83a6527700 1 --2- 172.21.15.42:0/2999431242 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f83a007d3a0 0x7f83a01209b0 secure :-1 s=THROTTLE_DONE pgs=1982 cs=0 l=1 rx=0x7f83a011a5b0 tx=0x7f8398007ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:04:09.874 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:09.842+0000 7f83a6527700 1 --2- 172.21.15.42:0/2999431242 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f83a007d3a0 0x7f83a01209b0 secure :-1 s=THROTTLE_DONE pgs=1982 cs=0 l=1 rx=0x7f83a011a5b0 tx=0x7f8398007ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:04:09.874 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:09.842+0000 7f83967fc700 1 -- 172.21.15.42:0/2999431242 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8398010b00 con 0x7f83a007d3a0 2019-12-05T06:04:09.875 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:09.842+0000 7f83a7789700 1 -- 172.21.15.42:0/2999431242 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83a007d7d0 con 0x7f83a007d3a0 2019-12-05T06:04:09.875 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:09.842+0000 7f83967fc700 1 -- 172.21.15.42:0/2999431242 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f839800f680 con 0x7f83a007d3a0 2019-12-05T06:04:09.875 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:09.842+0000 7f83a7789700 1 -- 172.21.15.42:0/2999431242 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83a0123740 con 0x7f83a007d3a0 2019-12-05T06:04:09.878 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:09.842+0000 7f83a6527700 1 --2- 172.21.15.42:0/2999431242 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f83a007d3a0 0x7f83a01209b0 secure :-1 s=THROTTLE_DONE pgs=1982 cs=0 l=1 rx=0x7f83a011a5b0 tx=0x7f8398007ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:04:09.878 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:09.842+0000 7f83967fc700 1 -- 172.21.15.42:0/2999431242 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f839800f880 con 0x7f83a007d3a0 2019-12-05T06:04:09.878 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:09.842+0000 7f83a6527700 1 --2- 172.21.15.42:0/2999431242 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f83a007d3a0 0x7f83a01209b0 secure :-1 s=THROTTLE_DONE pgs=1982 cs=0 l=1 rx=0x7f83a011a5b0 tx=0x7f8398007ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:04:09.878 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:09.842+0000 7f83967fc700 1 --2- 172.21.15.42:0/2999431242 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f838803ad10 0x7f838803d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:04:09.879 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:09.842+0000 7f83967fc700 1 -- 172.21.15.42:0/2999431242 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9643+0+0 (secure 0 0 0) 0x7f839804b120 con 0x7f83a007d3a0 2019-12-05T06:04:09.879 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:09.842+0000 7f83a5d26700 1 --2- 172.21.15.42:0/2999431242 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f838803ad10 0x7f838803d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:04:09.879 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:09.846+0000 7f83a5d26700 1 --2- 172.21.15.42:0/2999431242 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f838803ad10 0x7f838803d220 crc :-1 s=READY pgs=840 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:04:09.879 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:09.846+0000 7f83a7789700 1 -- 172.21.15.42:0/2999431242 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83a005d110 con 0x7f83a007d3a0 2019-12-05T06:04:09.880 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:09.854+0000 7f83a6527700 1 --2- 172.21.15.42:0/2999431242 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f83a007d3a0 0x7f83a01209b0 secure :-1 s=THROTTLE_DONE pgs=1982 cs=0 l=1 rx=0x7f83a011a5b0 tx=0x7f8398007ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:04:09.880 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:09.854+0000 7f83967fc700 1 -- 172.21.15.42:0/2999431242 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f8398032400 con 0x7f83a007d3a0 2019-12-05T06:04:10.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:10.026+0000 7f83a7789700 1 -- 172.21.15.42:0/2999431242 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f83a005b8c0 con 0x7f83a007d3a0 2019-12-05T06:04:10.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:10.026+0000 7f83a6527700 1 --2- 172.21.15.42:0/2999431242 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f83a007d3a0 0x7f83a01209b0 secure :-1 s=THROTTLE_DONE pgs=1982 cs=0 l=1 rx=0x7f83a011a5b0 tx=0x7f8398007ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:04:10.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:10.026+0000 7f83967fc700 1 -- 172.21.15.42:0/2999431242 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 242 v242) v1 ==== 95+0+1775 (secure 0 0 0) 0x7f8398032fb0 con 0x7f83a007d3a0 2019-12-05T06:04:10.031 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 242 2019-12-05T06:04:10.032 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:04:10.032 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":242,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":242,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:04:09.851907+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":91,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9381},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9381":{"gid":9381,"name":"b","rank":0,"incarnation":239,"state":"up:active","state_seq":37,"addr":"172.21.15.40:6833/3789081777","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3789081777},{"type":"v1","addr":"172.21.15.40:6833","nonce":3789081777}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:04:10.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:10.030+0000 7f83a7789700 1 -- 172.21.15.42:0/2999431242 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f838803ad10 msgr2=0x7f838803d220 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:04:10.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:10.030+0000 7f83a7789700 1 --2- 172.21.15.42:0/2999431242 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f838803ad10 0x7f838803d220 crc :-1 s=READY pgs=840 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:04:10.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:10.030+0000 7f83a7789700 1 -- 172.21.15.42:0/2999431242 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f83a007d3a0 msgr2=0x7f83a01209b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:04:10.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:10.030+0000 7f83a7789700 1 --2- 172.21.15.42:0/2999431242 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f83a007d3a0 0x7f83a01209b0 secure :-1 s=READY pgs=1982 cs=0 l=1 rx=0x7f83a011a5b0 tx=0x7f8398007ae0).stop 2019-12-05T06:04:10.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:10.030+0000 7f83a7789700 1 -- 172.21.15.42:0/2999431242 shutdown_connections 2019-12-05T06:04:10.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:10.030+0000 7f83a7789700 1 --2- 172.21.15.42:0/2999431242 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f838803ad10 0x7f838803d220 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:04:10.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:10.030+0000 7f83a7789700 1 --2- 172.21.15.42:0/2999431242 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f83a007d3a0 0x7f83a01209b0 unknown :-1 s=CLOSED pgs=1982 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:04:10.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:10.030+0000 7f83a7789700 1 --2- 172.21.15.42:0/2999431242 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f83a012ce70 0x7f83a0130d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:04:10.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:10.030+0000 7f83a7789700 1 --2- 172.21.15.42:0/2999431242 >> v2:172.21.15.40:3300/0 conn(0x7f83a0123310 0x7f83a007ce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:04:10.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:10.030+0000 7f83a7789700 1 -- 172.21.15.42:0/2999431242 shutdown_connections 2019-12-05T06:04:10.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:10.030+0000 7f83a7789700 1 -- 172.21.15.42:0/2999431242 wait complete. 2019-12-05T06:04:10.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:10.030+0000 7f83a7789700 1 -- 172.21.15.42:0/2999431242 >> 172.21.15.42:0/2999431242 conn(0x7f83a00a7740 msgr2=0x7f83a0059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:04:10.043 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b has gained rank=0, replacing gid=9292 2019-12-05T06:04:10.044 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 5 secs before reviving mds.a 2019-12-05T06:04:13.484 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:04:13.490 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:04:15.047 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.a 2019-12-05T06:04:15.048 INFO:tasks.ceph.mds.a:Restarting daemon 2019-12-05T06:04:15.048 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2019-12-05T06:04:15.052 INFO:tasks.ceph.mds.a:Started 2019-12-05T06:04:15.052 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:04:15.110 INFO:tasks.ceph.mds.a.smithi042.stdout:starting mds.a at 2019-12-05T06:04:15.157 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:15.150+0000 7ff6ff133700 1 Processor -- start 2019-12-05T06:04:15.157 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:15.150+0000 7ff6ff133700 1 -- start start 2019-12-05T06:04:15.158 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:15.150+0000 7ff6ff133700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff6f80a7740 0x7ff6f808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:04:15.158 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:15.150+0000 7ff6ff133700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff6f807e720 0x7ff6f807eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:04:15.158 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:15.150+0000 7ff6ff133700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6f80a0710 con 0x7ff6f809f2e0 2019-12-05T06:04:15.158 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:15.150+0000 7ff6ff133700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6f80a0b00 con 0x7ff6f807e720 2019-12-05T06:04:15.159 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:15.150+0000 7ff6ff133700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6f808e5b0 con 0x7ff6f80a7740 2019-12-05T06:04:15.159 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:15.150+0000 7ff6fd6d0700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff6f80a7740 0x7ff6f808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:04:15.160 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:15.150+0000 7ff6fd6d0700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff6f80a7740 0x7ff6f808df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:47572/0 (socket says 172.21.15.42:47572) 2019-12-05T06:04:15.160 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:15.150+0000 7ff6fd6d0700 1 -- 172.21.15.42:0/988184912 learned_addr learned my addr 172.21.15.42:0/988184912 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:04:15.160 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:15.150+0000 7ff6fcecf700 1 --2- 172.21.15.42:0/988184912 >> v2:172.21.15.40:3300/0 conn(0x7ff6f807e720 0x7ff6f807eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:04:15.160 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:15.150+0000 7ff6fd6d0700 1 -- 172.21.15.42:0/988184912 >> v2:172.21.15.40:3300/0 conn(0x7ff6f807e720 msgr2=0x7ff6f807eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:04:15.160 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:15.150+0000 7ff6fd6d0700 1 --2- 172.21.15.42:0/988184912 >> v2:172.21.15.40:3300/0 conn(0x7ff6f807e720 0x7ff6f807eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:04:15.161 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:15.150+0000 7ff6fd6d0700 1 -- 172.21.15.42:0/988184912 >> v1:172.21.15.40:6789/0 conn(0x7ff6f809f2e0 legacy=0x7ff6f809f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:04:15.161 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:15.150+0000 7ff6fd6d0700 1 -- 172.21.15.42:0/988184912 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6f807f0a0 con 0x7ff6f80a7740 2019-12-05T06:04:15.161 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:15.150+0000 7ff6fd6d0700 1 --2- 172.21.15.42:0/988184912 >> v2:172.21.15.42:3300/0 conn(0x7ff6f80a7740 0x7ff6f808df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7ff6f4002f30 tx=0x7ff6f400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:04:15.161 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:15.150+0000 7ff6fd6d0700 1 --2- 172.21.15.42:0/988184912 >> v2:172.21.15.42:3300/0 conn(0x7ff6f80a7740 0x7ff6f808df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7ff6f4002f30 tx=0x7ff6f400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:04:15.162 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:15.150+0000 7ff6fd6d0700 1 --2- 172.21.15.42:0/988184912 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff6f80a7740 0x7ff6f808df80 secure :-1 s=READY pgs=1986 cs=0 l=1 rx=0x7ff6f4002f30 tx=0x7ff6f400fd30).ready entity=mon.1 client_cookie=6e8ffc9f23e61c9d server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:04:15.162 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:15.150+0000 7ff6fd6d0700 1 --2- 172.21.15.42:0/988184912 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff6f80a7740 0x7ff6f808df80 secure :-1 s=THROTTLE_DONE pgs=1986 cs=0 l=1 rx=0x7ff6f4002f30 tx=0x7ff6f400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:04:15.162 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:15.150+0000 7ff6fd6d0700 1 --2- 172.21.15.42:0/988184912 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff6f80a7740 0x7ff6f808df80 secure :-1 s=THROTTLE_DONE pgs=1986 cs=0 l=1 rx=0x7ff6f4002f30 tx=0x7ff6f400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:04:15.162 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:15.150+0000 7ff6effff700 1 -- 172.21.15.42:0/988184912 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff6f4011370 con 0x7ff6f80a7740 2019-12-05T06:04:15.162 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:15.150+0000 7ff6fd6d0700 1 --2- 172.21.15.42:0/988184912 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff6f80a7740 0x7ff6f808df80 secure :-1 s=THROTTLE_DONE pgs=1986 cs=0 l=1 rx=0x7ff6f4002f30 tx=0x7ff6f400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:04:15.163 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:15.150+0000 7ff6effff700 1 -- 172.21.15.42:0/988184912 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6f4008bb0 con 0x7ff6f80a7740 2019-12-05T06:04:15.163 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:15.150+0000 7ff6effff700 1 -- 172.21.15.42:0/988184912 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff6f4011950 con 0x7ff6f80a7740 2019-12-05T06:04:15.163 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:15.150+0000 7ff6ff133700 1 -- 172.21.15.42:0/988184912 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff6f80a7740 msgr2=0x7ff6f808df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:04:15.163 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:15.150+0000 7ff6ff133700 1 --2- 172.21.15.42:0/988184912 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff6f80a7740 0x7ff6f808df80 secure :-1 s=READY pgs=1986 cs=0 l=1 rx=0x7ff6f4002f30 tx=0x7ff6f400fd30).stop 2019-12-05T06:04:15.164 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:15.154+0000 7ff6ff133700 1 -- 172.21.15.42:0/988184912 shutdown_connections 2019-12-05T06:04:15.164 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:15.154+0000 7ff6ff133700 1 --2- 172.21.15.42:0/988184912 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff6f80a7740 0x7ff6f808df80 unknown :-1 s=CLOSED pgs=1986 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:04:15.164 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:15.154+0000 7ff6ff133700 1 --2- 172.21.15.42:0/988184912 >> v2:172.21.15.40:3300/0 conn(0x7ff6f807e720 0x7ff6f807eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:04:15.164 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:15.154+0000 7ff6ff133700 1 -- 172.21.15.42:0/988184912 shutdown_connections 2019-12-05T06:04:15.165 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:15.154+0000 7ff6ff133700 1 -- 172.21.15.42:0/988184912 wait complete. 2019-12-05T06:04:15.165 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:15.154+0000 7ff6ff133700 1 -- 172.21.15.42:0/988184912 >> 172.21.15.42:0/988184912 conn(0x7ff6f80aabe0 msgr2=0x7ff6f80a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:04:15.172 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:15.154+0000 7ff6ff133700 1 Processor -- start 2019-12-05T06:04:15.173 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:15.154+0000 7ff6ff133700 1 -- start start 2019-12-05T06:04:15.173 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:15.158+0000 7ff6ff133700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff6f8145120 0x7ff6f8078b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:04:15.173 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:15.158+0000 7ff6ff133700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6f8079050 con 0x7ff6f81284b0 2019-12-05T06:04:15.174 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:15.158+0000 7ff6ff133700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6f8131500 con 0x7ff6f8147110 2019-12-05T06:04:15.174 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:15.158+0000 7ff6ff133700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6f807a240 con 0x7ff6f8145120 2019-12-05T06:04:15.174 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:15.158+0000 7ff6fd6d0700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7ff6f8147110 0x7ff6f807a5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:47616/0 (socket says 172.21.15.42:47616) 2019-12-05T06:04:15.174 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:15.158+0000 7ff6fd6d0700 1 -- 172.21.15.42:0/663185721 learned_addr learned my addr 172.21.15.42:0/663185721 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:04:15.175 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:15.158+0000 7ff6fcecf700 1 --2- 172.21.15.42:0/663185721 >> v2:172.21.15.40:3300/0 conn(0x7ff6f8145120 0x7ff6f8078b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:04:15.176 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:15.158+0000 7ff6edffb700 1 -- 172.21.15.42:0/663185721 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1147416784 0 0) 0x7ff6f8131500 con 0x7ff6f8147110 2019-12-05T06:04:15.176 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:15.158+0000 7ff6edffb700 1 -- 172.21.15.42:0/663185721 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6d80030b0 con 0x7ff6f8147110 2019-12-05T06:04:15.177 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:15.158+0000 7ff6fcecf700 1 -- 172.21.15.42:0/663185721 >> v1:172.21.15.42:6789/0 conn(0x7ff6f8147110 legacy=0x7ff6f807a5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:04:15.177 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:15.158+0000 7ff6edffb700 1 -- 172.21.15.42:0/663185721 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1417959708 0 0) 0x7ff6f8079050 con 0x7ff6f81284b0 2019-12-05T06:04:15.178 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:15.158+0000 7ff6fcecf700 1 -- 172.21.15.42:0/663185721 >> v1:172.21.15.40:6789/0 conn(0x7ff6f81284b0 legacy=0x7ff6f81288e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:04:15.178 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:15.158+0000 7ff6fcecf700 1 -- 172.21.15.42:0/663185721 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6f8132740 con 0x7ff6f8145120 2019-12-05T06:04:15.179 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:15.158+0000 7ff6fcecf700 1 --2- 172.21.15.42:0/663185721 >> v2:172.21.15.40:3300/0 conn(0x7ff6f8145120 0x7ff6f8078b00 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7ff6f4007670 tx=0x7ff6f40082d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:04:15.179 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:15.158+0000 7ff6fcecf700 1 --2- 172.21.15.42:0/663185721 >> v2:172.21.15.40:3300/0 conn(0x7ff6f8145120 0x7ff6f8078b00 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7ff6f4007670 tx=0x7ff6f40082d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:04:15.179 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:15.158+0000 7ff6fcecf700 1 --2- 172.21.15.42:0/663185721 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff6f8145120 0x7ff6f8078b00 secure :-1 s=READY pgs=1503 cs=0 l=1 rx=0x7ff6f4007670 tx=0x7ff6f40082d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:04:15.179 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:15.158+0000 7ff6fcecf700 1 --2- 172.21.15.42:0/663185721 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff6f8145120 0x7ff6f8078b00 secure :-1 s=THROTTLE_DONE pgs=1503 cs=0 l=1 rx=0x7ff6f4007670 tx=0x7ff6f40082d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:04:15.180 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:15.158+0000 7ff6edffb700 1 -- 172.21.15.42:0/663185721 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff6f4011370 con 0x7ff6f8145120 2019-12-05T06:04:15.180 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:15.158+0000 7ff6fcecf700 1 --2- 172.21.15.42:0/663185721 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff6f8145120 0x7ff6f8078b00 secure :-1 s=THROTTLE_DONE pgs=1503 cs=0 l=1 rx=0x7ff6f4007670 tx=0x7ff6f40082d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:04:15.180 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:15.158+0000 7ff6edffb700 1 -- 172.21.15.42:0/663185721 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6f4011570 con 0x7ff6f8145120 2019-12-05T06:04:15.180 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:15.158+0000 7ff6fcecf700 1 --2- 172.21.15.42:0/663185721 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff6f8145120 0x7ff6f8078b00 secure :-1 s=THROTTLE_DONE pgs=1503 cs=0 l=1 rx=0x7ff6f4007670 tx=0x7ff6f40082d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:04:15.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:15.158+0000 7ff6ff133700 1 -- 172.21.15.42:0/663185721 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6f81304f0 con 0x7ff6f8145120 2019-12-05T06:04:15.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:15.158+0000 7ff6edffb700 1 -- 172.21.15.42:0/663185721 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff6e8005b20 con 0x7ff6f8145120 2019-12-05T06:04:15.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:15.158+0000 7ff6ff133700 1 -- 172.21.15.42:0/663185721 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6f8130a70 con 0x7ff6f8145120 2019-12-05T06:04:15.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:15.158+0000 7ff6fcecf700 1 --2- 172.21.15.42:0/663185721 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff6f8145120 0x7ff6f8078b00 secure :-1 s=THROTTLE_DONE pgs=1503 cs=0 l=1 rx=0x7ff6f4007670 tx=0x7ff6f40082d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:04:15.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:15.162+0000 7ff6edffb700 1 -- 172.21.15.42:0/663185721 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7ff6e8007ae0 con 0x7ff6f8145120 2019-12-05T06:04:15.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:15.162+0000 7ff6fcecf700 1 --2- 172.21.15.42:0/663185721 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff6f8145120 0x7ff6f8078b00 secure :-1 s=THROTTLE_DONE pgs=1503 cs=0 l=1 rx=0x7ff6f4007670 tx=0x7ff6f40082d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:04:15.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:15.162+0000 7ff6edffb700 1 --2- 172.21.15.42:0/663185721 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff6d803b160 0x7ff6d803d670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:04:15.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:15.162+0000 7ff6edffb700 1 -- 172.21.15.42:0/663185721 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9643+0+0 (secure 0 0 0) 0x7ff6e804a6c0 con 0x7ff6f8145120 2019-12-05T06:04:15.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:15.162+0000 7ff6fd6d0700 1 --2- 172.21.15.42:0/663185721 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff6d803b160 0x7ff6d803d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:04:15.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:15.162+0000 7ff6fd6d0700 1 --2- 172.21.15.42:0/663185721 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff6d803b160 0x7ff6d803d670 crc :-1 s=READY pgs=842 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:04:15.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:15.166+0000 7ff6ff133700 1 -- 172.21.15.42:0/663185721 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6f805d110 con 0x7ff6f8145120 2019-12-05T06:04:15.184 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:15.174+0000 7ff6fcecf700 1 --2- 172.21.15.42:0/663185721 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff6f8145120 0x7ff6f8078b00 secure :-1 s=THROTTLE_DONE pgs=1503 cs=0 l=1 rx=0x7ff6f4007670 tx=0x7ff6f40082d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:04:15.184 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:15.174+0000 7ff6edffb700 1 -- 172.21.15.42:0/663185721 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7ff6e8031740 con 0x7ff6f8145120 2019-12-05T06:04:15.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:15.346+0000 7ff6ff133700 1 -- 172.21.15.42:0/663185721 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff6f805b8c0 con 0x7ff6f8145120 2019-12-05T06:04:15.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:15.346+0000 7ff6fcecf700 1 --2- 172.21.15.42:0/663185721 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff6f8145120 0x7ff6f8078b00 secure :-1 s=THROTTLE_DONE pgs=1503 cs=0 l=1 rx=0x7ff6f4007670 tx=0x7ff6f40082d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:04:15.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:15.346+0000 7ff6edffb700 1 -- 172.21.15.42:0/663185721 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 242 v242) v1 ==== 95+0+1775 (secure 0 0 0) 0x7ff6e80319a0 con 0x7ff6f8145120 2019-12-05T06:04:15.351 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 242 2019-12-05T06:04:15.351 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:04:15.351 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":242,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":242,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:04:09.851907+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":91,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9381},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9381":{"gid":9381,"name":"b","rank":0,"incarnation":239,"state":"up:active","state_seq":37,"addr":"172.21.15.40:6833/3789081777","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3789081777},{"type":"v1","addr":"172.21.15.40:6833","nonce":3789081777}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:04:15.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:15.350+0000 7ff6ff133700 1 -- 172.21.15.42:0/663185721 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff6d803b160 msgr2=0x7ff6d803d670 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:04:15.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:15.350+0000 7ff6ff133700 1 --2- 172.21.15.42:0/663185721 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff6d803b160 0x7ff6d803d670 crc :-1 s=READY pgs=842 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:04:15.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:15.350+0000 7ff6ff133700 1 -- 172.21.15.42:0/663185721 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff6f8145120 msgr2=0x7ff6f8078b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:04:15.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:15.350+0000 7ff6ff133700 1 --2- 172.21.15.42:0/663185721 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff6f8145120 0x7ff6f8078b00 secure :-1 s=READY pgs=1503 cs=0 l=1 rx=0x7ff6f4007670 tx=0x7ff6f40082d0).stop 2019-12-05T06:04:15.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:15.350+0000 7ff6ff133700 1 -- 172.21.15.42:0/663185721 shutdown_connections 2019-12-05T06:04:15.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:15.350+0000 7ff6ff133700 1 --2- 172.21.15.42:0/663185721 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff6d803b160 0x7ff6d803d670 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:04:15.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:15.350+0000 7ff6ff133700 1 --2- 172.21.15.42:0/663185721 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff6f8145120 0x7ff6f8078b00 unknown :-1 s=CLOSED pgs=1503 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:04:15.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:15.350+0000 7ff6ff133700 1 -- 172.21.15.42:0/663185721 shutdown_connections 2019-12-05T06:04:15.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:15.350+0000 7ff6ff133700 1 -- 172.21.15.42:0/663185721 wait complete. 2019-12-05T06:04:15.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:15.350+0000 7ff6ff133700 1 -- 172.21.15.42:0/663185721 >> 172.21.15.42:0/663185721 conn(0x7ff6f80a9800 msgr2=0x7ff6f809f620 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:04:15.367 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2019-12-05T06:04:17.368 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:04:17.479 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:17.470+0000 7f912e0f3700 1 Processor -- start 2019-12-05T06:04:17.479 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:17.470+0000 7f912e0f3700 1 -- start start 2019-12-05T06:04:17.479 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:17.470+0000 7f912e0f3700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f91280a7740 0x7f912808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:04:17.479 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:17.474+0000 7f912e0f3700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f912809fae0 0x7f912807eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:04:17.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:17.474+0000 7f912e0f3700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91280a0710 con 0x7f912807e720 2019-12-05T06:04:17.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:17.474+0000 7f912e0f3700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91280a0b00 con 0x7f912809fae0 2019-12-05T06:04:17.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:17.474+0000 7f912e0f3700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f912808e5b0 con 0x7f91280a7740 2019-12-05T06:04:17.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:17.474+0000 7f9127fff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f91280a7740 0x7f912808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:04:17.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:17.474+0000 7f9127fff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f91280a7740 0x7f912808df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:47590/0 (socket says 172.21.15.42:47590) 2019-12-05T06:04:17.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:17.474+0000 7f9127fff700 1 -- 172.21.15.42:0/2303600670 learned_addr learned my addr 172.21.15.42:0/2303600670 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:04:17.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:17.474+0000 7f912ce91700 1 --2- 172.21.15.42:0/2303600670 >> v2:172.21.15.40:3300/0 conn(0x7f912809fae0 0x7f912807eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:04:17.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:17.474+0000 7f9127fff700 1 -- 172.21.15.42:0/2303600670 >> v2:172.21.15.40:3300/0 conn(0x7f912809fae0 msgr2=0x7f912807eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:04:17.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:17.474+0000 7f9127fff700 1 --2- 172.21.15.42:0/2303600670 >> v2:172.21.15.40:3300/0 conn(0x7f912809fae0 0x7f912807eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:04:17.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:17.474+0000 7f9127fff700 1 -- 172.21.15.42:0/2303600670 >> v1:172.21.15.42:6789/0 conn(0x7f912807e720 legacy=0x7f912809f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:04:17.482 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:17.474+0000 7f9127fff700 1 -- 172.21.15.42:0/2303600670 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f912807f0a0 con 0x7f91280a7740 2019-12-05T06:04:17.482 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:17.474+0000 7f9127fff700 1 --2- 172.21.15.42:0/2303600670 >> v2:172.21.15.42:3300/0 conn(0x7f91280a7740 0x7f912808df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f9118002f30 tx=0x7f911800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:04:17.482 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:17.474+0000 7f9127fff700 1 --2- 172.21.15.42:0/2303600670 >> v2:172.21.15.42:3300/0 conn(0x7f91280a7740 0x7f912808df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f9118002f30 tx=0x7f911800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:04:17.482 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:17.474+0000 7f9127fff700 1 --2- 172.21.15.42:0/2303600670 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f91280a7740 0x7f912808df80 secure :-1 s=READY pgs=1989 cs=0 l=1 rx=0x7f9118002f30 tx=0x7f911800fd30).ready entity=mon.1 client_cookie=c0ac363ad18af288 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:04:17.482 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:17.474+0000 7f9127fff700 1 --2- 172.21.15.42:0/2303600670 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f91280a7740 0x7f912808df80 secure :-1 s=THROTTLE_DONE pgs=1989 cs=0 l=1 rx=0x7f9118002f30 tx=0x7f911800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:04:17.482 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:17.474+0000 7f91277fe700 1 -- 172.21.15.42:0/2303600670 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9118008c90 con 0x7f91280a7740 2019-12-05T06:04:17.483 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:17.474+0000 7f9127fff700 1 --2- 172.21.15.42:0/2303600670 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f91280a7740 0x7f912808df80 secure :-1 s=THROTTLE_DONE pgs=1989 cs=0 l=1 rx=0x7f9118002f30 tx=0x7f911800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:04:17.483 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:17.474+0000 7f9127fff700 1 --2- 172.21.15.42:0/2303600670 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f91280a7740 0x7f912808df80 secure :-1 s=THROTTLE_DONE pgs=1989 cs=0 l=1 rx=0x7f9118002f30 tx=0x7f911800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:04:17.483 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:17.474+0000 7f91277fe700 1 -- 172.21.15.42:0/2303600670 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91180088b0 con 0x7f91280a7740 2019-12-05T06:04:17.483 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:17.474+0000 7f91277fe700 1 -- 172.21.15.42:0/2303600670 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9118011ad0 con 0x7f91280a7740 2019-12-05T06:04:17.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:17.474+0000 7f912e0f3700 1 -- 172.21.15.42:0/2303600670 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f91280a7740 msgr2=0x7f912808df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:04:17.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:17.474+0000 7f912e0f3700 1 --2- 172.21.15.42:0/2303600670 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f91280a7740 0x7f912808df80 secure :-1 s=READY pgs=1989 cs=0 l=1 rx=0x7f9118002f30 tx=0x7f911800fd30).stop 2019-12-05T06:04:17.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:17.474+0000 7f912e0f3700 1 -- 172.21.15.42:0/2303600670 shutdown_connections 2019-12-05T06:04:17.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:17.474+0000 7f912e0f3700 1 --2- 172.21.15.42:0/2303600670 >> v2:172.21.15.40:3300/0 conn(0x7f912809fae0 0x7f912807eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:04:17.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:17.474+0000 7f912e0f3700 1 --2- 172.21.15.42:0/2303600670 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f91280a7740 0x7f912808df80 unknown :-1 s=CLOSED pgs=1989 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:04:17.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:17.474+0000 7f912e0f3700 1 -- 172.21.15.42:0/2303600670 shutdown_connections 2019-12-05T06:04:17.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:17.474+0000 7f912e0f3700 1 -- 172.21.15.42:0/2303600670 wait complete. 2019-12-05T06:04:17.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:17.474+0000 7f912e0f3700 1 -- 172.21.15.42:0/2303600670 >> 172.21.15.42:0/2303600670 conn(0x7f91280aabe0 msgr2=0x7f91280a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:04:17.498 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:17.478+0000 7f912e0f3700 1 Processor -- start 2019-12-05T06:04:17.499 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:17.478+0000 7f912e0f3700 1 -- start start 2019-12-05T06:04:17.499 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:17.478+0000 7f912e0f3700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9128143ee0 0x7f9128142530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:04:17.499 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:17.478+0000 7f912e0f3700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f912814e140 0x7f912814e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:04:17.499 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:17.478+0000 7f912e0f3700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f912814eac0 con 0x7f9128123ab0 2019-12-05T06:04:17.499 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:17.478+0000 7f912e0f3700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9128142cb0 con 0x7f912814e140 2019-12-05T06:04:17.499 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:17.478+0000 7f912e0f3700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9128142f50 con 0x7f9128143ee0 2019-12-05T06:04:17.500 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:17.478+0000 7f912ce91700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f9128123ab0 0x7f9128123ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:47640/0 (socket says 172.21.15.42:47640) 2019-12-05T06:04:17.500 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:17.478+0000 7f912ce91700 1 -- 172.21.15.42:0/2522879166 learned_addr learned my addr 172.21.15.42:0/2522879166 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:04:17.500 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:17.478+0000 7f9127fff700 1 --2- 172.21.15.42:0/2522879166 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9128143ee0 0x7f9128142530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:04:17.500 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:17.478+0000 7f911f7fe700 1 --2- 172.21.15.42:0/2522879166 >> v2:172.21.15.40:3300/0 conn(0x7f912814e140 0x7f912814e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:04:17.508 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:17.478+0000 7f91257fa700 1 -- 172.21.15.42:0/2522879166 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2558083548 0 0) 0x7f912814eac0 con 0x7f9128123ab0 2019-12-05T06:04:17.509 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:17.478+0000 7f91257fa700 1 -- 172.21.15.42:0/2522879166 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f910c0030b0 con 0x7f9128123ab0 2019-12-05T06:04:17.509 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:17.478+0000 7f9127fff700 1 -- 172.21.15.42:0/2522879166 >> v2:172.21.15.40:3300/0 conn(0x7f912814e140 msgr2=0x7f912814e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:04:17.509 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:17.478+0000 7f9127fff700 1 --2- 172.21.15.42:0/2522879166 >> v2:172.21.15.40:3300/0 conn(0x7f912814e140 0x7f912814e570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:04:17.509 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:17.478+0000 7f9127fff700 1 -- 172.21.15.42:0/2522879166 >> v1:172.21.15.42:6789/0 conn(0x7f9128123ab0 legacy=0x7f9128123ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:04:17.509 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:17.478+0000 7f9127fff700 1 -- 172.21.15.42:0/2522879166 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9128143770 con 0x7f9128143ee0 2019-12-05T06:04:17.509 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:17.478+0000 7f9127fff700 1 --2- 172.21.15.42:0/2522879166 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9128143ee0 0x7f9128142530 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f9118002f30 tx=0x7f9118008b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:04:17.509 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:17.478+0000 7f9127fff700 1 --2- 172.21.15.42:0/2522879166 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9128143ee0 0x7f9128142530 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f9118002f30 tx=0x7f9118008b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:04:17.510 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:17.478+0000 7f9127fff700 1 --2- 172.21.15.42:0/2522879166 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9128143ee0 0x7f9128142530 secure :-1 s=READY pgs=441 cs=0 l=1 rx=0x7f9118002f30 tx=0x7f9118008b60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:04:17.510 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:17.478+0000 7f9127fff700 1 --2- 172.21.15.42:0/2522879166 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9128143ee0 0x7f9128142530 secure :-1 s=THROTTLE_DONE pgs=441 cs=0 l=1 rx=0x7f9118002f30 tx=0x7f9118008b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:04:17.510 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:17.478+0000 7f91257fa700 1 -- 172.21.15.42:0/2522879166 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f911800f490 con 0x7f9128143ee0 2019-12-05T06:04:17.510 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:17.482+0000 7f9127fff700 1 --2- 172.21.15.42:0/2522879166 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9128143ee0 0x7f9128142530 secure :-1 s=THROTTLE_DONE pgs=441 cs=0 l=1 rx=0x7f9118002f30 tx=0x7f9118008b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:04:17.510 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:17.482+0000 7f91257fa700 1 -- 172.21.15.42:0/2522879166 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91180116f0 con 0x7f9128143ee0 2019-12-05T06:04:17.510 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:17.482+0000 7f9127fff700 1 --2- 172.21.15.42:0/2522879166 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9128143ee0 0x7f9128142530 secure :-1 s=THROTTLE_DONE pgs=441 cs=0 l=1 rx=0x7f9118002f30 tx=0x7f9118008b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:04:17.510 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:17.482+0000 7f91257fa700 1 -- 172.21.15.42:0/2522879166 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9118005730 con 0x7f9128143ee0 2019-12-05T06:04:17.511 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:17.482+0000 7f912e0f3700 1 -- 172.21.15.42:0/2522879166 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f912808eb50 con 0x7f9128143ee0 2019-12-05T06:04:17.511 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:17.482+0000 7f912e0f3700 1 -- 172.21.15.42:0/2522879166 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f912808f100 con 0x7f9128143ee0 2019-12-05T06:04:17.511 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:17.482+0000 7f9127fff700 1 --2- 172.21.15.42:0/2522879166 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9128143ee0 0x7f9128142530 secure :-1 s=THROTTLE_DONE pgs=441 cs=0 l=1 rx=0x7f9118002f30 tx=0x7f9118008b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:04:17.511 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:17.482+0000 7f91257fa700 1 -- 172.21.15.42:0/2522879166 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f91180059d0 con 0x7f9128143ee0 2019-12-05T06:04:17.511 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:17.482+0000 7f9127fff700 1 --2- 172.21.15.42:0/2522879166 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9128143ee0 0x7f9128142530 secure :-1 s=THROTTLE_DONE pgs=441 cs=0 l=1 rx=0x7f9118002f30 tx=0x7f9118008b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:04:17.511 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:17.482+0000 7f91257fa700 1 --2- 172.21.15.42:0/2522879166 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f910c03d460 0x7f910c03f970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:04:17.511 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:17.482+0000 7f91257fa700 1 -- 172.21.15.42:0/2522879166 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9643+0+0 (secure 0 0 0) 0x7f911804b920 con 0x7f9128143ee0 2019-12-05T06:04:17.512 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:17.482+0000 7f911f7fe700 1 --2- 172.21.15.42:0/2522879166 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f910c03d460 0x7f910c03f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:04:17.512 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:17.482+0000 7f911f7fe700 1 --2- 172.21.15.42:0/2522879166 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f910c03d460 0x7f910c03f970 crc :-1 s=READY pgs=843 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:04:17.512 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:17.486+0000 7f912e0f3700 1 -- 172.21.15.42:0/2522879166 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f912805d110 con 0x7f9128143ee0 2019-12-05T06:04:17.513 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:17.498+0000 7f9127fff700 1 --2- 172.21.15.42:0/2522879166 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9128143ee0 0x7f9128142530 secure :-1 s=THROTTLE_DONE pgs=441 cs=0 l=1 rx=0x7f9118002f30 tx=0x7f9118008b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:04:17.513 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:17.498+0000 7f91257fa700 1 -- 172.21.15.42:0/2522879166 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f9118005ce0 con 0x7f9128143ee0 2019-12-05T06:04:17.680 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:17.678+0000 7f912e0f3700 1 -- 172.21.15.42:0/2522879166 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f912805b8c0 con 0x7f9128143ee0 2019-12-05T06:04:17.681 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:17.678+0000 7f9127fff700 1 --2- 172.21.15.42:0/2522879166 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9128143ee0 0x7f9128142530 secure :-1 s=THROTTLE_DONE pgs=441 cs=0 l=1 rx=0x7f9118002f30 tx=0x7f9118008b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:04:17.681 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:17.678+0000 7f91257fa700 1 -- 172.21.15.42:0/2522879166 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 242 v242) v1 ==== 95+0+1775 (secure 0 0 0) 0x7f9118032be0 con 0x7f9128143ee0 2019-12-05T06:04:17.681 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 242 2019-12-05T06:04:17.681 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:04:17.682 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":242,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":242,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:04:09.851907+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":91,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9381},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9381":{"gid":9381,"name":"b","rank":0,"incarnation":239,"state":"up:active","state_seq":37,"addr":"172.21.15.40:6833/3789081777","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3789081777},{"type":"v1","addr":"172.21.15.40:6833","nonce":3789081777}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:04:17.684 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:17.682+0000 7f912e0f3700 1 -- 172.21.15.42:0/2522879166 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f910c03d460 msgr2=0x7f910c03f970 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:04:17.684 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:17.682+0000 7f912e0f3700 1 --2- 172.21.15.42:0/2522879166 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f910c03d460 0x7f910c03f970 crc :-1 s=READY pgs=843 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:04:17.684 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:17.682+0000 7f912e0f3700 1 -- 172.21.15.42:0/2522879166 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9128143ee0 msgr2=0x7f9128142530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:04:17.684 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:17.682+0000 7f912e0f3700 1 --2- 172.21.15.42:0/2522879166 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9128143ee0 0x7f9128142530 secure :-1 s=READY pgs=441 cs=0 l=1 rx=0x7f9118002f30 tx=0x7f9118008b60).stop 2019-12-05T06:04:17.684 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:17.682+0000 7f912e0f3700 1 -- 172.21.15.42:0/2522879166 shutdown_connections 2019-12-05T06:04:17.684 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:17.682+0000 7f912e0f3700 1 --2- 172.21.15.42:0/2522879166 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f910c03d460 0x7f910c03f970 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:04:17.685 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:17.682+0000 7f912e0f3700 1 --2- 172.21.15.42:0/2522879166 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9128143ee0 0x7f9128142530 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:04:17.685 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:17.682+0000 7f912e0f3700 1 --2- 172.21.15.42:0/2522879166 >> v2:172.21.15.40:3300/0 conn(0x7f912814e140 0x7f912814e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:04:17.685 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:17.682+0000 7f912e0f3700 1 -- 172.21.15.42:0/2522879166 shutdown_connections 2019-12-05T06:04:17.685 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:17.682+0000 7f912e0f3700 1 -- 172.21.15.42:0/2522879166 wait complete. 2019-12-05T06:04:17.685 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:17.682+0000 7f912e0f3700 1 -- 172.21.15.42:0/2522879166 >> 172.21.15.42:0/2522879166 conn(0x7f91280a89d0 msgr2=0x7f912809f620 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:04:17.694 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2019-12-05T06:04:19.695 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:04:19.800 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:19.790+0000 7f00fdb6d700 1 Processor -- start 2019-12-05T06:04:19.800 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:19.790+0000 7f00fdb6d700 1 -- start start 2019-12-05T06:04:19.800 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:19.794+0000 7f00fdb6d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f00f808df80 0x7f00f809f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:04:19.801 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:19.794+0000 7f00fdb6d700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f00f809f830 0x7f00f814efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:04:19.801 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:19.794+0000 7f00fdb6d700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00f809fc60 con 0x7f00f80a7740 2019-12-05T06:04:19.801 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:19.794+0000 7f00fdb6d700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00f80a0b00 con 0x7f00f809f830 2019-12-05T06:04:19.801 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:19.794+0000 7f00fdb6d700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00f808e5b0 con 0x7f00f808df80 2019-12-05T06:04:19.801 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:19.794+0000 7f00f7fff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f00f80a7740 0x7f00f807e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:47654/0 (socket says 172.21.15.42:47654) 2019-12-05T06:04:19.802 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:19.794+0000 7f00f7fff700 1 -- 172.21.15.42:0/2357387388 learned_addr learned my addr 172.21.15.42:0/2357387388 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:04:19.802 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:19.794+0000 7f00f77fe700 1 --2- 172.21.15.42:0/2357387388 >> v2:172.21.15.42:3300/0 conn(0x7f00f808df80 0x7f00f809f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:04:19.802 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:19.794+0000 7f00fc90b700 1 --2- 172.21.15.42:0/2357387388 >> v2:172.21.15.40:3300/0 conn(0x7f00f809f830 0x7f00f814efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:04:19.802 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:19.794+0000 7f00f6ffd700 1 -- 172.21.15.42:0/2357387388 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 409869615 0 0) 0x7f00f809fc60 con 0x7f00f80a7740 2019-12-05T06:04:19.802 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:19.794+0000 7f00f6ffd700 1 -- 172.21.15.42:0/2357387388 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00e40030b0 con 0x7f00f80a7740 2019-12-05T06:04:19.802 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:19.794+0000 7f00f77fe700 1 -- 172.21.15.42:0/2357387388 >> v2:172.21.15.40:3300/0 conn(0x7f00f809f830 msgr2=0x7f00f814efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:04:19.803 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:19.794+0000 7f00f77fe700 1 --2- 172.21.15.42:0/2357387388 >> v2:172.21.15.40:3300/0 conn(0x7f00f809f830 0x7f00f814efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:04:19.803 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:19.794+0000 7f00f77fe700 1 -- 172.21.15.42:0/2357387388 >> v1:172.21.15.42:6789/0 conn(0x7f00f80a7740 legacy=0x7f00f807e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:04:19.803 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:19.794+0000 7f00f77fe700 1 -- 172.21.15.42:0/2357387388 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00f80a0710 con 0x7f00f808df80 2019-12-05T06:04:19.804 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:19.794+0000 7f00f77fe700 1 --2- 172.21.15.42:0/2357387388 >> v2:172.21.15.42:3300/0 conn(0x7f00f808df80 0x7f00f809f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f00e0002f30 tx=0x7f00e000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:04:19.804 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:19.794+0000 7f00f77fe700 1 --2- 172.21.15.42:0/2357387388 >> v2:172.21.15.42:3300/0 conn(0x7f00f808df80 0x7f00f809f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f00e0002f30 tx=0x7f00e000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:04:19.804 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:19.794+0000 7f00f77fe700 1 --2- 172.21.15.42:0/2357387388 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f00f808df80 0x7f00f809f2e0 secure :-1 s=READY pgs=1992 cs=0 l=1 rx=0x7f00e0002f30 tx=0x7f00e000fc70).ready entity=mon.1 client_cookie=1fb416b651ae566d server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:04:19.804 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:19.794+0000 7f00f77fe700 1 --2- 172.21.15.42:0/2357387388 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f00f808df80 0x7f00f809f2e0 secure :-1 s=THROTTLE_DONE pgs=1992 cs=0 l=1 rx=0x7f00e0002f30 tx=0x7f00e000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:04:19.805 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:19.794+0000 7f00f6ffd700 1 -- 172.21.15.42:0/2357387388 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f00e0008bb0 con 0x7f00f808df80 2019-12-05T06:04:19.805 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:19.794+0000 7f00f77fe700 1 --2- 172.21.15.42:0/2357387388 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f00f808df80 0x7f00f809f2e0 secure :-1 s=THROTTLE_DONE pgs=1992 cs=0 l=1 rx=0x7f00e0002f30 tx=0x7f00e000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:04:19.805 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:19.794+0000 7f00f77fe700 1 --2- 172.21.15.42:0/2357387388 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f00f808df80 0x7f00f809f2e0 secure :-1 s=THROTTLE_DONE pgs=1992 cs=0 l=1 rx=0x7f00e0002f30 tx=0x7f00e000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:04:19.805 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:19.794+0000 7f00f6ffd700 1 -- 172.21.15.42:0/2357387388 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00e00087d0 con 0x7f00f808df80 2019-12-05T06:04:19.806 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:19.794+0000 7f00f6ffd700 1 -- 172.21.15.42:0/2357387388 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f00e0011820 con 0x7f00f808df80 2019-12-05T06:04:19.806 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:19.794+0000 7f00fdb6d700 1 -- 172.21.15.42:0/2357387388 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f00f808df80 msgr2=0x7f00f809f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:04:19.806 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:19.794+0000 7f00fdb6d700 1 --2- 172.21.15.42:0/2357387388 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f00f808df80 0x7f00f809f2e0 secure :-1 s=READY pgs=1992 cs=0 l=1 rx=0x7f00e0002f30 tx=0x7f00e000fc70).stop 2019-12-05T06:04:19.806 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:19.794+0000 7f00fdb6d700 1 -- 172.21.15.42:0/2357387388 shutdown_connections 2019-12-05T06:04:19.807 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:19.794+0000 7f00fdb6d700 1 --2- 172.21.15.42:0/2357387388 >> v2:172.21.15.40:3300/0 conn(0x7f00f809f830 0x7f00f814efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:04:19.807 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:19.794+0000 7f00fdb6d700 1 --2- 172.21.15.42:0/2357387388 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f00f808df80 0x7f00f809f2e0 unknown :-1 s=CLOSED pgs=1992 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:04:19.807 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:19.794+0000 7f00fdb6d700 1 -- 172.21.15.42:0/2357387388 shutdown_connections 2019-12-05T06:04:19.807 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:19.794+0000 7f00fdb6d700 1 -- 172.21.15.42:0/2357387388 wait complete. 2019-12-05T06:04:19.808 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:19.794+0000 7f00fdb6d700 1 -- 172.21.15.42:0/2357387388 >> 172.21.15.42:0/2357387388 conn(0x7f00f80aabe0 msgr2=0x7f00f80a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:04:19.812 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:19.798+0000 7f00fdb6d700 1 Processor -- start 2019-12-05T06:04:19.812 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:19.798+0000 7f00fdb6d700 1 -- start start 2019-12-05T06:04:19.812 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:19.802+0000 7f00fdb6d700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f00f8122360 0x7f00f8123b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:04:19.812 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:19.802+0000 7f00fdb6d700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00f807b950 con 0x7f00f8131740 2019-12-05T06:04:19.812 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:19.802+0000 7f00f7fff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f00f808eeb0 0x7f00f807b150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:47660/0 (socket says 172.21.15.42:47660) 2019-12-05T06:04:19.813 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:19.802+0000 7f00f7fff700 1 -- 172.21.15.42:0/2726817394 learned_addr learned my addr 172.21.15.42:0/2726817394 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:04:19.813 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:19.802+0000 7f00f77fe700 1 --2- 172.21.15.42:0/2726817394 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f00f8122360 0x7f00f8123b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:04:19.813 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:19.802+0000 7f00fdb6d700 1 -- 172.21.15.42:0/2726817394 --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00f813aaa0 con 0x7f00f808eeb0 2019-12-05T06:04:19.814 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:19.802+0000 7f00fdb6d700 1 -- 172.21.15.42:0/2726817394 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00f81228e0 con 0x7f00f8122360 2019-12-05T06:04:19.814 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:19.802+0000 7f00f77fe700 1 -- 172.21.15.42:0/2726817394 >> v1:172.21.15.42:6789/0 conn(0x7f00f808eeb0 legacy=0x7f00f807b150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:04:19.814 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:19.802+0000 7f00f77fe700 1 -- 172.21.15.42:0/2726817394 >> v1:172.21.15.40:6789/0 conn(0x7f00f8131740 legacy=0x7f00f8131b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2019-12-05T06:04:19.815 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:19.802+0000 7f00f77fe700 1 -- 172.21.15.42:0/2726817394 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00f813bce0 con 0x7f00f8122360 2019-12-05T06:04:19.815 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:19.802+0000 7f00f77fe700 1 --2- 172.21.15.42:0/2726817394 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f00f8122360 0x7f00f8123b00 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f00e0002f30 tx=0x7f00e00118e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:04:19.815 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:19.802+0000 7f00f77fe700 1 --2- 172.21.15.42:0/2726817394 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f00f8122360 0x7f00f8123b00 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f00e0002f30 tx=0x7f00e00118e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:04:19.815 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:19.802+0000 7f00f77fe700 1 --2- 172.21.15.42:0/2726817394 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f00f8122360 0x7f00f8123b00 secure :-1 s=READY pgs=442 cs=0 l=1 rx=0x7f00e0002f30 tx=0x7f00e00118e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:04:19.816 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:19.806+0000 7f00f77fe700 1 --2- 172.21.15.42:0/2726817394 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f00f8122360 0x7f00f8123b00 secure :-1 s=THROTTLE_DONE pgs=442 cs=0 l=1 rx=0x7f00e0002f30 tx=0x7f00e00118e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:04:19.816 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:19.806+0000 7f00f77fe700 1 --2- 172.21.15.42:0/2726817394 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f00f8122360 0x7f00f8123b00 secure :-1 s=THROTTLE_DONE pgs=442 cs=0 l=1 rx=0x7f00e0002f30 tx=0x7f00e00118e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:04:19.816 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:19.806+0000 7f00f4ff9700 1 -- 172.21.15.42:0/2726817394 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f00e000f280 con 0x7f00f8122360 2019-12-05T06:04:19.816 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:19.806+0000 7f00f77fe700 1 --2- 172.21.15.42:0/2726817394 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f00f8122360 0x7f00f8123b00 secure :-1 s=THROTTLE_DONE pgs=442 cs=0 l=1 rx=0x7f00e0002f30 tx=0x7f00e00118e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:04:19.817 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:19.806+0000 7f00f4ff9700 1 -- 172.21.15.42:0/2726817394 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00e0008db0 con 0x7f00f8122360 2019-12-05T06:04:19.817 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:19.806+0000 7f00fdb6d700 1 -- 172.21.15.42:0/2726817394 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00f8139a90 con 0x7f00f8122360 2019-12-05T06:04:19.817 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:19.806+0000 7f00f4ff9700 1 -- 172.21.15.42:0/2726817394 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f00e0005540 con 0x7f00f8122360 2019-12-05T06:04:19.817 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:19.806+0000 7f00fdb6d700 1 -- 172.21.15.42:0/2726817394 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00f813a010 con 0x7f00f8122360 2019-12-05T06:04:19.818 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:19.806+0000 7f00f77fe700 1 --2- 172.21.15.42:0/2726817394 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f00f8122360 0x7f00f8123b00 secure :-1 s=THROTTLE_DONE pgs=442 cs=0 l=1 rx=0x7f00e0002f30 tx=0x7f00e00118e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:04:19.818 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:19.806+0000 7f00f4ff9700 1 -- 172.21.15.42:0/2726817394 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f00e00130b0 con 0x7f00f8122360 2019-12-05T06:04:19.818 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:19.806+0000 7f00f77fe700 1 --2- 172.21.15.42:0/2726817394 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f00f8122360 0x7f00f8123b00 secure :-1 s=THROTTLE_DONE pgs=442 cs=0 l=1 rx=0x7f00e0002f30 tx=0x7f00e00118e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:04:19.818 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:19.806+0000 7f00f4ff9700 1 --2- 172.21.15.42:0/2726817394 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f00dc03ad80 0x7f00dc03d290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:04:19.818 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:19.806+0000 7f00f4ff9700 1 -- 172.21.15.42:0/2726817394 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9643+0+0 (secure 0 0 0) 0x7f00e004b560 con 0x7f00f8122360 2019-12-05T06:04:19.819 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:19.806+0000 7f00f7fff700 1 --2- 172.21.15.42:0/2726817394 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f00dc03ad80 0x7f00dc03d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:04:19.819 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:19.806+0000 7f00f7fff700 1 --2- 172.21.15.42:0/2726817394 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f00dc03ad80 0x7f00dc03d290 crc :-1 s=READY pgs=845 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:04:19.826 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:19.822+0000 7f00fdb6d700 1 -- 172.21.15.42:0/2726817394 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00f805d110 con 0x7f00f8122360 2019-12-05T06:04:19.841 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:19.834+0000 7f00f77fe700 1 --2- 172.21.15.42:0/2726817394 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f00f8122360 0x7f00f8123b00 secure :-1 s=THROTTLE_DONE pgs=442 cs=0 l=1 rx=0x7f00e0002f30 tx=0x7f00e00118e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:04:19.841 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:19.834+0000 7f00f4ff9700 1 -- 172.21.15.42:0/2726817394 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f00e0007160 con 0x7f00f8122360 2019-12-05T06:04:20.007 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:20.006+0000 7f00fdb6d700 1 -- 172.21.15.42:0/2726817394 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f00f805b8c0 con 0x7f00f8122360 2019-12-05T06:04:20.008 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:20.006+0000 7f00f77fe700 1 --2- 172.21.15.42:0/2726817394 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f00f8122360 0x7f00f8123b00 secure :-1 s=THROTTLE_DONE pgs=442 cs=0 l=1 rx=0x7f00e0002f30 tx=0x7f00e00118e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:04:20.008 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:20.006+0000 7f00f4ff9700 1 -- 172.21.15.42:0/2726817394 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 243 v243) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f00e0031290 con 0x7f00f8122360 2019-12-05T06:04:20.010 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 243 2019-12-05T06:04:20.011 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:04:20.011 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":243,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6821,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/2253267554","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2253267554},{"type":"v1","addr":"172.21.15.42:6835","nonce":2253267554}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":243}],"filesystems":[{"mdsmap":{"epoch":242,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:04:09.851907+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":91,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9381},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9381":{"gid":9381,"name":"b","rank":0,"incarnation":239,"state":"up:active","state_seq":37,"addr":"172.21.15.40:6833/3789081777","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3789081777},{"type":"v1","addr":"172.21.15.40:6833","nonce":3789081777}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:04:20.013 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:20.010+0000 7f00fdb6d700 1 -- 172.21.15.42:0/2726817394 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f00dc03ad80 msgr2=0x7f00dc03d290 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:04:20.013 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:20.010+0000 7f00fdb6d700 1 --2- 172.21.15.42:0/2726817394 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f00dc03ad80 0x7f00dc03d290 crc :-1 s=READY pgs=845 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:04:20.014 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:20.010+0000 7f00fdb6d700 1 -- 172.21.15.42:0/2726817394 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f00f8122360 msgr2=0x7f00f8123b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:04:20.014 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:20.010+0000 7f00fdb6d700 1 --2- 172.21.15.42:0/2726817394 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f00f8122360 0x7f00f8123b00 secure :-1 s=READY pgs=442 cs=0 l=1 rx=0x7f00e0002f30 tx=0x7f00e00118e0).stop 2019-12-05T06:04:20.014 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:20.010+0000 7f00fdb6d700 1 -- 172.21.15.42:0/2726817394 shutdown_connections 2019-12-05T06:04:20.014 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:20.010+0000 7f00fdb6d700 1 --2- 172.21.15.42:0/2726817394 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f00dc03ad80 0x7f00dc03d290 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:04:20.014 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:20.010+0000 7f00fdb6d700 1 --2- 172.21.15.42:0/2726817394 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f00f8122360 0x7f00f8123b00 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:04:20.015 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:20.010+0000 7f00fdb6d700 1 -- 172.21.15.42:0/2726817394 shutdown_connections 2019-12-05T06:04:20.016 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:20.010+0000 7f00fdb6d700 1 -- 172.21.15.42:0/2726817394 wait complete. 2019-12-05T06:04:20.016 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:04:20.010+0000 7f00fdb6d700 1 -- 172.21.15.42:0/2726817394 >> 172.21.15.42:0/2726817394 conn(0x7f00f80a9b60 msgr2=0x7f00f811c540 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:04:20.023 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a reported in up:standby state 2019-12-05T06:04:20.023 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 55 secs before thrashing 2019-12-05T06:04:43.532 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:04:43.538 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:05:13.661 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:05:13.665 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:05:15.025 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:05:15.128 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.117+0000 7f5cec7a3700 1 Processor -- start 2019-12-05T06:05:15.129 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.121+0000 7f5cec7a3700 1 -- start start 2019-12-05T06:05:15.129 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.121+0000 7f5cec7a3700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5ce40a7740 0x7f5ce408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:05:15.129 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.121+0000 7f5cec7a3700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5ce409fae0 0x7f5ce407eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:05:15.129 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.121+0000 7f5cec7a3700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ce40a0710 con 0x7f5ce407e720 2019-12-05T06:05:15.129 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.121+0000 7f5cec7a3700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ce40a0b00 con 0x7f5ce409fae0 2019-12-05T06:05:15.129 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.121+0000 7f5cec7a3700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ce408e5b0 con 0x7f5ce40a7740 2019-12-05T06:05:15.129 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.121+0000 7f5cead40700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5ce40a7740 0x7f5ce408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:05:15.130 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.121+0000 7f5cead40700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5ce40a7740 0x7f5ce408df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:47772/0 (socket says 172.21.15.42:47772) 2019-12-05T06:05:15.130 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.121+0000 7f5cead40700 1 -- 172.21.15.42:0/617153962 learned_addr learned my addr 172.21.15.42:0/617153962 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:05:15.130 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.121+0000 7f5ceb541700 1 --2- 172.21.15.42:0/617153962 >> v2:172.21.15.40:3300/0 conn(0x7f5ce409fae0 0x7f5ce407eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:05:15.130 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.121+0000 7f5cead40700 1 -- 172.21.15.42:0/617153962 >> v2:172.21.15.40:3300/0 conn(0x7f5ce409fae0 msgr2=0x7f5ce407eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:05:15.130 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.121+0000 7f5cead40700 1 --2- 172.21.15.42:0/617153962 >> v2:172.21.15.40:3300/0 conn(0x7f5ce409fae0 0x7f5ce407eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:05:15.130 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.121+0000 7f5cead40700 1 -- 172.21.15.42:0/617153962 >> v1:172.21.15.42:6789/0 conn(0x7f5ce407e720 legacy=0x7f5ce409f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:05:15.130 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.121+0000 7f5cead40700 1 -- 172.21.15.42:0/617153962 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ce407f0a0 con 0x7f5ce40a7740 2019-12-05T06:05:15.131 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.121+0000 7f5cead40700 1 --2- 172.21.15.42:0/617153962 >> v2:172.21.15.42:3300/0 conn(0x7f5ce40a7740 0x7f5ce408df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f5ce0002f30 tx=0x7f5ce000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:15.131 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.121+0000 7f5cead40700 1 --2- 172.21.15.42:0/617153962 >> v2:172.21.15.42:3300/0 conn(0x7f5ce40a7740 0x7f5ce408df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f5ce0002f30 tx=0x7f5ce000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:15.131 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.121+0000 7f5cead40700 1 --2- 172.21.15.42:0/617153962 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5ce40a7740 0x7f5ce408df80 secure :-1 s=READY pgs=1997 cs=0 l=1 rx=0x7f5ce0002f30 tx=0x7f5ce000fd30).ready entity=mon.1 client_cookie=acb20713051782d3 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:05:15.131 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.121+0000 7f5cead40700 1 --2- 172.21.15.42:0/617153962 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5ce40a7740 0x7f5ce408df80 secure :-1 s=THROTTLE_DONE pgs=1997 cs=0 l=1 rx=0x7f5ce0002f30 tx=0x7f5ce000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:15.131 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.121+0000 7f5ce9d3e700 1 -- 172.21.15.42:0/617153962 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5ce0008c90 con 0x7f5ce40a7740 2019-12-05T06:05:15.131 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.121+0000 7f5cead40700 1 --2- 172.21.15.42:0/617153962 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5ce40a7740 0x7f5ce408df80 secure :-1 s=THROTTLE_DONE pgs=1997 cs=0 l=1 rx=0x7f5ce0002f30 tx=0x7f5ce000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:15.131 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.121+0000 7f5cead40700 1 --2- 172.21.15.42:0/617153962 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5ce40a7740 0x7f5ce408df80 secure :-1 s=THROTTLE_DONE pgs=1997 cs=0 l=1 rx=0x7f5ce0002f30 tx=0x7f5ce000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:15.132 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.121+0000 7f5ce9d3e700 1 -- 172.21.15.42:0/617153962 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ce00088b0 con 0x7f5ce40a7740 2019-12-05T06:05:15.132 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.121+0000 7f5ce9d3e700 1 -- 172.21.15.42:0/617153962 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5ce0011ad0 con 0x7f5ce40a7740 2019-12-05T06:05:15.132 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.121+0000 7f5cec7a3700 1 -- 172.21.15.42:0/617153962 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5ce40a7740 msgr2=0x7f5ce408df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:05:15.132 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.121+0000 7f5cec7a3700 1 --2- 172.21.15.42:0/617153962 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5ce40a7740 0x7f5ce408df80 secure :-1 s=READY pgs=1997 cs=0 l=1 rx=0x7f5ce0002f30 tx=0x7f5ce000fd30).stop 2019-12-05T06:05:15.132 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.125+0000 7f5cec7a3700 1 -- 172.21.15.42:0/617153962 shutdown_connections 2019-12-05T06:05:15.132 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.125+0000 7f5cec7a3700 1 --2- 172.21.15.42:0/617153962 >> v2:172.21.15.40:3300/0 conn(0x7f5ce409fae0 0x7f5ce407eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:05:15.132 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.125+0000 7f5cec7a3700 1 --2- 172.21.15.42:0/617153962 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5ce40a7740 0x7f5ce408df80 unknown :-1 s=CLOSED pgs=1997 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:05:15.133 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.125+0000 7f5cec7a3700 1 -- 172.21.15.42:0/617153962 shutdown_connections 2019-12-05T06:05:15.133 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.125+0000 7f5cec7a3700 1 -- 172.21.15.42:0/617153962 wait complete. 2019-12-05T06:05:15.133 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.125+0000 7f5cec7a3700 1 -- 172.21.15.42:0/617153962 >> 172.21.15.42:0/617153962 conn(0x7f5ce40aabe0 msgr2=0x7f5ce40a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:05:15.135 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.125+0000 7f5cec7a3700 1 Processor -- start 2019-12-05T06:05:15.135 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.125+0000 7f5cec7a3700 1 -- start start 2019-12-05T06:05:15.135 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.129+0000 7f5cec7a3700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5ce412ad90 0x7f5ce4065730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:05:15.136 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.129+0000 7f5cec7a3700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5ce40a64c0 0x7f5ce40a68f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:05:15.136 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.129+0000 7f5cec7a3700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ce40a6e40 con 0x7f5ce408ee90 2019-12-05T06:05:15.136 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.129+0000 7f5cec7a3700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ce4063530 con 0x7f5ce412ad90 2019-12-05T06:05:15.136 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.129+0000 7f5cec7a3700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ce4063c30 con 0x7f5ce40a64c0 2019-12-05T06:05:15.136 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.129+0000 7f5ceb541700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5ce40a64c0 0x7f5ce40a68f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:05:15.136 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.129+0000 7f5cea53f700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5ce412ad90 0x7f5ce4065730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:05:15.136 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.129+0000 7f5ceb541700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5ce40a64c0 0x7f5ce40a68f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:48246/0 (socket says 172.21.15.42:48246) 2019-12-05T06:05:15.137 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.129+0000 7f5ceb541700 1 -- 172.21.15.42:0/1026673503 learned_addr learned my addr 172.21.15.42:0/1026673503 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:05:15.137 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.129+0000 7f5ceb541700 1 -- 172.21.15.42:0/1026673503 >> v2:172.21.15.40:3300/0 conn(0x7f5ce412ad90 msgr2=0x7f5ce4065730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:05:15.137 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.129+0000 7f5ceb541700 1 --2- 172.21.15.42:0/1026673503 >> v2:172.21.15.40:3300/0 conn(0x7f5ce412ad90 0x7f5ce4065730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:05:15.137 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.129+0000 7f5ceb541700 1 -- 172.21.15.42:0/1026673503 >> v1:172.21.15.40:6789/0 conn(0x7f5ce408ee90 legacy=0x7f5ce412a590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:05:15.137 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.129+0000 7f5ceb541700 1 -- 172.21.15.42:0/1026673503 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ce4063800 con 0x7f5ce40a64c0 2019-12-05T06:05:15.137 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.129+0000 7f5ceb541700 1 --2- 172.21.15.42:0/1026673503 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5ce40a64c0 0x7f5ce40a68f0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f5cdc0032b0 tx=0x7f5cdc010ca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:15.137 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.129+0000 7f5ceb541700 1 --2- 172.21.15.42:0/1026673503 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5ce40a64c0 0x7f5ce40a68f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f5cdc0032b0 tx=0x7f5cdc010ca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:15.137 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.129+0000 7f5ceb541700 1 --2- 172.21.15.42:0/1026673503 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5ce40a64c0 0x7f5ce40a68f0 secure :-1 s=READY pgs=445 cs=0 l=1 rx=0x7f5cdc0032b0 tx=0x7f5cdc010ca0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:05:15.138 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.129+0000 7f5ceb541700 1 --2- 172.21.15.42:0/1026673503 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5ce40a64c0 0x7f5ce40a68f0 secure :-1 s=THROTTLE_DONE pgs=445 cs=0 l=1 rx=0x7f5cdc0032b0 tx=0x7f5cdc010ca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:15.138 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.129+0000 7f5ceb541700 1 --2- 172.21.15.42:0/1026673503 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5ce40a64c0 0x7f5ce40a68f0 secure :-1 s=THROTTLE_DONE pgs=445 cs=0 l=1 rx=0x7f5cdc0032b0 tx=0x7f5cdc010ca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:15.138 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.129+0000 7f5ceb541700 1 --2- 172.21.15.42:0/1026673503 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5ce40a64c0 0x7f5ce40a68f0 secure :-1 s=THROTTLE_DONE pgs=445 cs=0 l=1 rx=0x7f5cdc0032b0 tx=0x7f5cdc010ca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:15.138 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.129+0000 7f5cdb7fe700 1 -- 172.21.15.42:0/1026673503 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5cdc00eaa0 con 0x7f5ce40a64c0 2019-12-05T06:05:15.138 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.129+0000 7f5cdb7fe700 1 -- 172.21.15.42:0/1026673503 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cdc00e6c0 con 0x7f5ce40a64c0 2019-12-05T06:05:15.138 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.129+0000 7f5cec7a3700 1 -- 172.21.15.42:0/1026673503 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ce4131580 con 0x7f5ce40a64c0 2019-12-05T06:05:15.138 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.129+0000 7f5cdb7fe700 1 -- 172.21.15.42:0/1026673503 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5cdc012580 con 0x7f5ce40a64c0 2019-12-05T06:05:15.139 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.129+0000 7f5cec7a3700 1 -- 172.21.15.42:0/1026673503 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ce4131b30 con 0x7f5ce40a64c0 2019-12-05T06:05:15.139 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.129+0000 7f5ceb541700 1 --2- 172.21.15.42:0/1026673503 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5ce40a64c0 0x7f5ce40a68f0 secure :-1 s=THROTTLE_DONE pgs=445 cs=0 l=1 rx=0x7f5cdc0032b0 tx=0x7f5cdc010ca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:15.140 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.133+0000 7f5ceb541700 1 --2- 172.21.15.42:0/1026673503 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5ce40a64c0 0x7f5ce40a68f0 secure :-1 s=THROTTLE_DONE pgs=445 cs=0 l=1 rx=0x7f5cdc0032b0 tx=0x7f5cdc010ca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:15.140 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.133+0000 7f5cdb7fe700 1 -- 172.21.15.42:0/1026673503 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f5cdc00f0c0 con 0x7f5ce40a64c0 2019-12-05T06:05:15.140 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.133+0000 7f5cdb7fe700 1 --2- 172.21.15.42:0/1026673503 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5ccc03ade0 0x7f5ccc03d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:05:15.140 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.133+0000 7f5cdb7fe700 1 -- 172.21.15.42:0/1026673503 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9643+0+0 (secure 0 0 0) 0x7f5cdc04b530 con 0x7f5ce40a64c0 2019-12-05T06:05:15.140 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.133+0000 7f5cead40700 1 --2- 172.21.15.42:0/1026673503 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5ccc03ade0 0x7f5ccc03d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:05:15.140 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.133+0000 7f5cead40700 1 --2- 172.21.15.42:0/1026673503 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5ccc03ade0 0x7f5ccc03d2f0 crc :-1 s=READY pgs=846 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:05:15.141 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.133+0000 7f5cec7a3700 1 -- 172.21.15.42:0/1026673503 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ce405d110 con 0x7f5ce40a64c0 2019-12-05T06:05:15.148 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.145+0000 7f5ceb541700 1 --2- 172.21.15.42:0/1026673503 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5ce40a64c0 0x7f5ce40a68f0 secure :-1 s=THROTTLE_DONE pgs=445 cs=0 l=1 rx=0x7f5cdc0032b0 tx=0x7f5cdc010ca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:15.149 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.145+0000 7f5cdb7fe700 1 -- 172.21.15.42:0/1026673503 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f5cdc032570 con 0x7f5ce40a64c0 2019-12-05T06:05:15.316 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.313+0000 7f5cec7a3700 1 -- 172.21.15.42:0/1026673503 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5ce405b8c0 con 0x7f5ce40a64c0 2019-12-05T06:05:15.317 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.313+0000 7f5ceb541700 1 --2- 172.21.15.42:0/1026673503 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5ce40a64c0 0x7f5ce40a68f0 secure :-1 s=THROTTLE_DONE pgs=445 cs=0 l=1 rx=0x7f5cdc0032b0 tx=0x7f5cdc010ca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:15.317 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.313+0000 7f5cdb7fe700 1 -- 172.21.15.42:0/1026673503 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 243 v243) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f5cdc0327d0 con 0x7f5ce40a64c0 2019-12-05T06:05:15.319 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:05:15.319 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":243,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6821,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/2253267554","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2253267554},{"type":"v1","addr":"172.21.15.42:6835","nonce":2253267554}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":243}],"filesystems":[{"mdsmap":{"epoch":242,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:04:09.851907+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":91,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9381},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9381":{"gid":9381,"name":"b","rank":0,"incarnation":239,"state":"up:active","state_seq":37,"addr":"172.21.15.40:6833/3789081777","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3789081777},{"type":"v1","addr":"172.21.15.40:6833","nonce":3789081777}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:05:15.319 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 243 2019-12-05T06:05:15.321 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.317+0000 7f5cec7a3700 1 -- 172.21.15.42:0/1026673503 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5ccc03ade0 msgr2=0x7f5ccc03d2f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:05:15.321 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.317+0000 7f5cec7a3700 1 --2- 172.21.15.42:0/1026673503 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5ccc03ade0 0x7f5ccc03d2f0 crc :-1 s=READY pgs=846 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:05:15.321 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.317+0000 7f5cec7a3700 1 -- 172.21.15.42:0/1026673503 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5ce40a64c0 msgr2=0x7f5ce40a68f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:05:15.322 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.317+0000 7f5cec7a3700 1 --2- 172.21.15.42:0/1026673503 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5ce40a64c0 0x7f5ce40a68f0 secure :-1 s=READY pgs=445 cs=0 l=1 rx=0x7f5cdc0032b0 tx=0x7f5cdc010ca0).stop 2019-12-05T06:05:15.322 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.317+0000 7f5cec7a3700 1 -- 172.21.15.42:0/1026673503 shutdown_connections 2019-12-05T06:05:15.322 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.317+0000 7f5cec7a3700 1 --2- 172.21.15.42:0/1026673503 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5ccc03ade0 0x7f5ccc03d2f0 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:05:15.322 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.317+0000 7f5cec7a3700 1 --2- 172.21.15.42:0/1026673503 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5ce40a64c0 0x7f5ce40a68f0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:05:15.322 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.317+0000 7f5cec7a3700 1 --2- 172.21.15.42:0/1026673503 >> v2:172.21.15.40:3300/0 conn(0x7f5ce412ad90 0x7f5ce4065730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:05:15.322 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.317+0000 7f5cec7a3700 1 -- 172.21.15.42:0/1026673503 shutdown_connections 2019-12-05T06:05:15.322 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.317+0000 7f5cec7a3700 1 -- 172.21.15.42:0/1026673503 wait complete. 2019-12-05T06:05:15.323 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.317+0000 7f5cec7a3700 1 -- 172.21.15.42:0/1026673503 >> 172.21.15.42:0/1026673503 conn(0x7f5ce40aabe0 msgr2=0x7f5ce4059d50 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:05:15.334 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.b (rank=0) 2019-12-05T06:05:15.335 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2019-12-05T06:05:15.335 INFO:teuthology.orchestra.run:waiting for 300 2019-12-05T06:05:15.352 INFO:tasks.ceph.mds.b:Stopped 2019-12-05T06:05:15.352 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:05:15.454 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.445+0000 7f91b6926700 1 Processor -- start 2019-12-05T06:05:15.454 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.445+0000 7f91b6926700 1 -- start start 2019-12-05T06:05:15.454 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.445+0000 7f91b6926700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f91b009fae0 0x7f91b014fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:05:15.455 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.445+0000 7f91b6926700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91b00a0710 con 0x7f91b00a7740 2019-12-05T06:05:15.455 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.445+0000 7f91b6926700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91b007ef20 con 0x7f91b008df80 2019-12-05T06:05:15.455 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.445+0000 7f91b6926700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91b00a0b00 con 0x7f91b009fae0 2019-12-05T06:05:15.455 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.445+0000 7f91affff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f91b008df80 0x7f91b009f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:47826/0 (socket says 172.21.15.42:47826) 2019-12-05T06:05:15.455 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.445+0000 7f91affff700 1 -- 172.21.15.42:0/2027917218 learned_addr learned my addr 172.21.15.42:0/2027917218 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:05:15.455 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.445+0000 7f91b56c4700 1 --2- 172.21.15.42:0/2027917218 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f91b009fae0 0x7f91b014fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:05:15.455 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.445+0000 7f91b56c4700 1 -- 172.21.15.42:0/2027917218 >> v1:172.21.15.42:6789/0 conn(0x7f91b008df80 legacy=0x7f91b009f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:05:15.456 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.445+0000 7f91b56c4700 1 -- 172.21.15.42:0/2027917218 >> v1:172.21.15.40:6789/0 conn(0x7f91b00a7740 legacy=0x7f91b007e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:05:15.456 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.445+0000 7f91af7fe700 1 -- 172.21.15.42:0/2027917218 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1038759584 0 0) 0x7f91b007ef20 con 0x7f91b008df80 2019-12-05T06:05:15.456 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.445+0000 7f91b56c4700 1 -- 172.21.15.42:0/2027917218 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91b00a9ba0 con 0x7f91b009fae0 2019-12-05T06:05:15.456 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.445+0000 7f91b56c4700 1 --2- 172.21.15.42:0/2027917218 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f91b009fae0 0x7f91b014fff0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f91a8002f30 tx=0x7f91a800fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:15.456 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.445+0000 7f91b56c4700 1 --2- 172.21.15.42:0/2027917218 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f91b009fae0 0x7f91b014fff0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f91a8002f30 tx=0x7f91a800fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:15.456 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.445+0000 7f91b56c4700 1 --2- 172.21.15.42:0/2027917218 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f91b009fae0 0x7f91b014fff0 secure :-1 s=READY pgs=446 cs=0 l=1 rx=0x7f91a8002f30 tx=0x7f91a800fbf0).ready entity=mon.2 client_cookie=d1136cd0604f8cb8 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:05:15.456 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.449+0000 7f91b56c4700 1 --2- 172.21.15.42:0/2027917218 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f91b009fae0 0x7f91b014fff0 secure :-1 s=THROTTLE_DONE pgs=446 cs=0 l=1 rx=0x7f91a8002f30 tx=0x7f91a800fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:15.457 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.449+0000 7f91b56c4700 1 --2- 172.21.15.42:0/2027917218 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f91b009fae0 0x7f91b014fff0 secure :-1 s=THROTTLE_DONE pgs=446 cs=0 l=1 rx=0x7f91a8002f30 tx=0x7f91a800fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:15.457 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.449+0000 7f91af7fe700 1 -- 172.21.15.42:0/2027917218 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f91a8008bb0 con 0x7f91b009fae0 2019-12-05T06:05:15.457 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.449+0000 7f91b56c4700 1 --2- 172.21.15.42:0/2027917218 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f91b009fae0 0x7f91b014fff0 secure :-1 s=THROTTLE_DONE pgs=446 cs=0 l=1 rx=0x7f91a8002f30 tx=0x7f91a800fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:15.457 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.449+0000 7f91af7fe700 1 -- 172.21.15.42:0/2027917218 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91a80087d0 con 0x7f91b009fae0 2019-12-05T06:05:15.457 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.449+0000 7f91af7fe700 1 -- 172.21.15.42:0/2027917218 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f91a8011910 con 0x7f91b009fae0 2019-12-05T06:05:15.457 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.449+0000 7f91b6926700 1 -- 172.21.15.42:0/2027917218 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f91b009fae0 msgr2=0x7f91b014fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:05:15.457 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.449+0000 7f91b6926700 1 --2- 172.21.15.42:0/2027917218 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f91b009fae0 0x7f91b014fff0 secure :-1 s=READY pgs=446 cs=0 l=1 rx=0x7f91a8002f30 tx=0x7f91a800fbf0).stop 2019-12-05T06:05:15.458 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.449+0000 7f91b6926700 1 -- 172.21.15.42:0/2027917218 shutdown_connections 2019-12-05T06:05:15.458 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.449+0000 7f91b6926700 1 --2- 172.21.15.42:0/2027917218 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f91b009fae0 0x7f91b014fff0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:05:15.458 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.449+0000 7f91b6926700 1 -- 172.21.15.42:0/2027917218 shutdown_connections 2019-12-05T06:05:15.458 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.449+0000 7f91b6926700 1 -- 172.21.15.42:0/2027917218 wait complete. 2019-12-05T06:05:15.458 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.449+0000 7f91b6926700 1 -- 172.21.15.42:0/2027917218 >> 172.21.15.42:0/2027917218 conn(0x7f91b00aabe0 msgr2=0x7f91b00a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:05:15.461 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.453+0000 7f91b6926700 1 Processor -- start 2019-12-05T06:05:15.461 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.453+0000 7f91b6926700 1 -- start start 2019-12-05T06:05:15.461 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.453+0000 7f91b6926700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f91b0146f70 0x7f91b0148060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:05:15.461 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.453+0000 7f91b6926700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f91b01485b0 0x7f91b01280f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:05:15.461 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.453+0000 7f91b6926700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91b01425c0 con 0x7f91b005a210 2019-12-05T06:05:15.462 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.453+0000 7f91b6926700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91b0146a30 con 0x7f91b01485b0 2019-12-05T06:05:15.462 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.453+0000 7f91b6926700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91b0146cd0 con 0x7f91b0146f70 2019-12-05T06:05:15.462 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.453+0000 7f91b56c4700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f91b01485b0 0x7f91b01280f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:05:15.462 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.453+0000 7f91b4ec3700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f91b005a210 0x7f91b0145860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:57270/0 (socket says 172.21.15.42:57270) 2019-12-05T06:05:15.462 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.453+0000 7f91b4ec3700 1 -- 172.21.15.42:0/2916526541 learned_addr learned my addr 172.21.15.42:0/2916526541 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:05:15.462 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.453+0000 7f91affff700 1 --2- 172.21.15.42:0/2916526541 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f91b0146f70 0x7f91b0148060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:05:15.462 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.453+0000 7f91ad7fa700 1 -- 172.21.15.42:0/2916526541 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3003221057 0 0) 0x7f91b01425c0 con 0x7f91b005a210 2019-12-05T06:05:15.463 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.453+0000 7f91b56c4700 1 -- 172.21.15.42:0/2916526541 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f91b0146f70 msgr2=0x7f91b0148060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:05:15.463 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.453+0000 7f91b56c4700 1 --2- 172.21.15.42:0/2916526541 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f91b0146f70 0x7f91b0148060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:05:15.463 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.453+0000 7f91b56c4700 1 -- 172.21.15.42:0/2916526541 >> v1:172.21.15.40:6789/0 conn(0x7f91b005a210 legacy=0x7f91b0145860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:05:15.463 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.453+0000 7f91b56c4700 1 -- 172.21.15.42:0/2916526541 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91b0146770 con 0x7f91b01485b0 2019-12-05T06:05:15.463 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.453+0000 7f91b56c4700 1 --2- 172.21.15.42:0/2916526541 >> v2:172.21.15.40:3300/0 conn(0x7f91b01485b0 0x7f91b01280f0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f91b0146bf0 tx=0x7f91a8008a40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:15.463 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.453+0000 7f91b56c4700 1 --2- 172.21.15.42:0/2916526541 >> v2:172.21.15.40:3300/0 conn(0x7f91b01485b0 0x7f91b01280f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f91b0146bf0 tx=0x7f91a8008a40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:15.463 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.453+0000 7f91b56c4700 1 --2- 172.21.15.42:0/2916526541 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f91b01485b0 0x7f91b01280f0 secure :-1 s=READY pgs=1512 cs=0 l=1 rx=0x7f91b0146bf0 tx=0x7f91a8008a40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:05:15.464 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.453+0000 7f91b56c4700 1 --2- 172.21.15.42:0/2916526541 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f91b01485b0 0x7f91b01280f0 secure :-1 s=THROTTLE_DONE pgs=1512 cs=0 l=1 rx=0x7f91b0146bf0 tx=0x7f91a8008a40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:15.464 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.457+0000 7f91ad7fa700 1 -- 172.21.15.42:0/2916526541 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f91a8005540 con 0x7f91b01485b0 2019-12-05T06:05:15.464 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.457+0000 7f91b56c4700 1 --2- 172.21.15.42:0/2916526541 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f91b01485b0 0x7f91b01280f0 secure :-1 s=THROTTLE_DONE pgs=1512 cs=0 l=1 rx=0x7f91b0146bf0 tx=0x7f91a8008a40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:15.464 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.457+0000 7f91b56c4700 1 --2- 172.21.15.42:0/2916526541 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f91b01485b0 0x7f91b01280f0 secure :-1 s=THROTTLE_DONE pgs=1512 cs=0 l=1 rx=0x7f91b0146bf0 tx=0x7f91a8008a40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:15.464 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.457+0000 7f91ad7fa700 1 -- 172.21.15.42:0/2916526541 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91a8008db0 con 0x7f91b01485b0 2019-12-05T06:05:15.464 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.457+0000 7f91ad7fa700 1 -- 172.21.15.42:0/2916526541 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f91a8005b20 con 0x7f91b01485b0 2019-12-05T06:05:15.464 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.457+0000 7f91b6926700 1 -- 172.21.15.42:0/2916526541 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91b0128640 con 0x7f91b01485b0 2019-12-05T06:05:15.465 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.457+0000 7f91b6926700 1 -- 172.21.15.42:0/2916526541 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91b0135ac0 con 0x7f91b01485b0 2019-12-05T06:05:15.465 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.457+0000 7f91b56c4700 1 --2- 172.21.15.42:0/2916526541 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f91b01485b0 0x7f91b01280f0 secure :-1 s=THROTTLE_DONE pgs=1512 cs=0 l=1 rx=0x7f91b0146bf0 tx=0x7f91a8008a40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:15.466 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.457+0000 7f91ad7fa700 1 -- 172.21.15.42:0/2916526541 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f91a80130b0 con 0x7f91b01485b0 2019-12-05T06:05:15.466 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.457+0000 7f91b56c4700 1 --2- 172.21.15.42:0/2916526541 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f91b01485b0 0x7f91b01280f0 secure :-1 s=THROTTLE_DONE pgs=1512 cs=0 l=1 rx=0x7f91b0146bf0 tx=0x7f91a8008a40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:15.466 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.457+0000 7f91ad7fa700 1 --2- 172.21.15.42:0/2916526541 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f919403ad10 0x7f919403d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:05:15.466 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.457+0000 7f91ad7fa700 1 -- 172.21.15.42:0/2916526541 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9643+0+0 (secure 0 0 0) 0x7f91a804b6f0 con 0x7f91b01485b0 2019-12-05T06:05:15.466 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.457+0000 7f91b4ec3700 1 --2- 172.21.15.42:0/2916526541 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f919403ad10 0x7f919403d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:05:15.467 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.457+0000 7f91b4ec3700 1 --2- 172.21.15.42:0/2916526541 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f919403ad10 0x7f919403d220 crc :-1 s=READY pgs=847 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:05:15.467 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.457+0000 7f91b6926700 1 -- 172.21.15.42:0/2916526541 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91b005d110 con 0x7f91b01485b0 2019-12-05T06:05:15.468 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.465+0000 7f91b56c4700 1 --2- 172.21.15.42:0/2916526541 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f91b01485b0 0x7f91b01280f0 secure :-1 s=THROTTLE_DONE pgs=1512 cs=0 l=1 rx=0x7f91b0146bf0 tx=0x7f91a8008a40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:15.468 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.465+0000 7f91ad7fa700 1 -- 172.21.15.42:0/2916526541 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f91b0146770 con 0x7f91b01485b0 2019-12-05T06:05:15.632 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.629+0000 7f91b6926700 1 -- 172.21.15.42:0/2916526541 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f91b005b8c0 con 0x7f91b01485b0 2019-12-05T06:05:15.632 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.629+0000 7f91b56c4700 1 --2- 172.21.15.42:0/2916526541 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f91b01485b0 0x7f91b01280f0 secure :-1 s=THROTTLE_DONE pgs=1512 cs=0 l=1 rx=0x7f91b0146bf0 tx=0x7f91a8008a40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:15.633 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.629+0000 7f91ad7fa700 1 -- 172.21.15.42:0/2916526541 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 243 v243) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f91a8007160 con 0x7f91b01485b0 2019-12-05T06:05:15.635 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 243 2019-12-05T06:05:15.635 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:05:15.635 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":243,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6821,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/2253267554","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2253267554},{"type":"v1","addr":"172.21.15.42:6835","nonce":2253267554}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":243}],"filesystems":[{"mdsmap":{"epoch":242,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:04:09.851907+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":91,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9381},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9381":{"gid":9381,"name":"b","rank":0,"incarnation":239,"state":"up:active","state_seq":37,"addr":"172.21.15.40:6833/3789081777","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3789081777},{"type":"v1","addr":"172.21.15.40:6833","nonce":3789081777}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:05:15.637 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.633+0000 7f91b6926700 1 -- 172.21.15.42:0/2916526541 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f919403ad10 msgr2=0x7f919403d220 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:05:15.638 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.633+0000 7f91b6926700 1 --2- 172.21.15.42:0/2916526541 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f919403ad10 0x7f919403d220 crc :-1 s=READY pgs=847 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:05:15.638 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.633+0000 7f91b6926700 1 -- 172.21.15.42:0/2916526541 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f91b01485b0 msgr2=0x7f91b01280f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:05:15.638 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.633+0000 7f91b6926700 1 --2- 172.21.15.42:0/2916526541 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f91b01485b0 0x7f91b01280f0 secure :-1 s=READY pgs=1512 cs=0 l=1 rx=0x7f91b0146bf0 tx=0x7f91a8008a40).stop 2019-12-05T06:05:15.638 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.633+0000 7f91b6926700 1 -- 172.21.15.42:0/2916526541 shutdown_connections 2019-12-05T06:05:15.638 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.633+0000 7f91b6926700 1 --2- 172.21.15.42:0/2916526541 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f919403ad10 0x7f919403d220 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:05:15.638 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.633+0000 7f91b6926700 1 --2- 172.21.15.42:0/2916526541 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f91b01485b0 0x7f91b01280f0 unknown :-1 s=CLOSED pgs=1512 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:05:15.638 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.633+0000 7f91b6926700 1 --2- 172.21.15.42:0/2916526541 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f91b0146f70 0x7f91b0148060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:05:15.639 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.633+0000 7f91b6926700 1 -- 172.21.15.42:0/2916526541 shutdown_connections 2019-12-05T06:05:15.639 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.633+0000 7f91b6926700 1 -- 172.21.15.42:0/2916526541 wait complete. 2019-12-05T06:05:15.639 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:15.633+0000 7f91b6926700 1 -- 172.21.15.42:0/2916526541 >> 172.21.15.42:0/2916526541 conn(0x7f91b007da70 msgr2=0x7f91b011cab0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:05:15.648 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:05:17.649 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:05:17.754 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:17.745+0000 7f4abed50700 1 Processor -- start 2019-12-05T06:05:17.754 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:17.745+0000 7f4abed50700 1 -- start start 2019-12-05T06:05:17.754 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:17.745+0000 7f4abed50700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f4ab808df80 0x7f4ab809f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:05:17.754 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:17.745+0000 7f4abed50700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f4ab809f830 0x7f4ab814efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:05:17.755 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:17.745+0000 7f4abed50700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ab809fc60 con 0x7f4ab80a7740 2019-12-05T06:05:17.755 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:17.745+0000 7f4abed50700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ab80a0b00 con 0x7f4ab809f830 2019-12-05T06:05:17.756 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:17.745+0000 7f4abed50700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ab808e5b0 con 0x7f4ab808df80 2019-12-05T06:05:17.756 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:17.745+0000 7f4abcaec700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f4ab808df80 0x7f4ab809f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:05:17.756 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:17.745+0000 7f4abcaec700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f4ab808df80 0x7f4ab809f2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:47808/0 (socket says 172.21.15.42:47808) 2019-12-05T06:05:17.756 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:17.745+0000 7f4abd2ed700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f4ab80a7740 0x7f4ab807e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:47844/0 (socket says 172.21.15.42:47844) 2019-12-05T06:05:17.756 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:17.745+0000 7f4abcaec700 1 -- 172.21.15.42:0/2472060986 learned_addr learned my addr 172.21.15.42:0/2472060986 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:05:17.756 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:17.745+0000 7f4abdaee700 1 --2- 172.21.15.42:0/2472060986 >> v2:172.21.15.40:3300/0 conn(0x7f4ab809f830 0x7f4ab814efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:05:17.756 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:17.745+0000 7f4abcaec700 1 -- 172.21.15.42:0/2472060986 >> v2:172.21.15.40:3300/0 conn(0x7f4ab809f830 msgr2=0x7f4ab814efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:05:17.757 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:17.745+0000 7f4abcaec700 1 --2- 172.21.15.42:0/2472060986 >> v2:172.21.15.40:3300/0 conn(0x7f4ab809f830 0x7f4ab814efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:05:17.757 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:17.745+0000 7f4abcaec700 1 -- 172.21.15.42:0/2472060986 >> v1:172.21.15.42:6789/0 conn(0x7f4ab80a7740 legacy=0x7f4ab807e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:05:17.757 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:17.745+0000 7f4ab7fff700 1 -- 172.21.15.42:0/2472060986 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4112161262 0 0) 0x7f4ab809fc60 con 0x7f4ab80a7740 2019-12-05T06:05:17.757 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:17.745+0000 7f4abcaec700 1 -- 172.21.15.42:0/2472060986 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ab80a0710 con 0x7f4ab808df80 2019-12-05T06:05:17.757 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:17.749+0000 7f4abcaec700 1 --2- 172.21.15.42:0/2472060986 >> v2:172.21.15.42:3300/0 conn(0x7f4ab808df80 0x7f4ab809f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f4aa00032b0 tx=0x7f4aa00100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:17.757 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:17.749+0000 7f4abcaec700 1 --2- 172.21.15.42:0/2472060986 >> v2:172.21.15.42:3300/0 conn(0x7f4ab808df80 0x7f4ab809f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f4aa00032b0 tx=0x7f4aa00100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:17.757 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:17.749+0000 7f4abcaec700 1 --2- 172.21.15.42:0/2472060986 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4ab808df80 0x7f4ab809f2e0 secure :-1 s=READY pgs=2000 cs=0 l=1 rx=0x7f4aa00032b0 tx=0x7f4aa00100b0).ready entity=mon.1 client_cookie=92057e0f051a7ea9 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:05:17.758 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:17.749+0000 7f4abcaec700 1 --2- 172.21.15.42:0/2472060986 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4ab808df80 0x7f4ab809f2e0 secure :-1 s=THROTTLE_DONE pgs=2000 cs=0 l=1 rx=0x7f4aa00032b0 tx=0x7f4aa00100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:17.758 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:17.749+0000 7f4ab7fff700 1 -- 172.21.15.42:0/2472060986 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4aa0010d60 con 0x7f4ab808df80 2019-12-05T06:05:17.758 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:17.749+0000 7f4abcaec700 1 --2- 172.21.15.42:0/2472060986 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4ab808df80 0x7f4ab809f2e0 secure :-1 s=THROTTLE_DONE pgs=2000 cs=0 l=1 rx=0x7f4aa00032b0 tx=0x7f4aa00100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:17.758 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:17.749+0000 7f4abcaec700 1 --2- 172.21.15.42:0/2472060986 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4ab808df80 0x7f4ab809f2e0 secure :-1 s=THROTTLE_DONE pgs=2000 cs=0 l=1 rx=0x7f4aa00032b0 tx=0x7f4aa00100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:17.758 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:17.749+0000 7f4ab7fff700 1 -- 172.21.15.42:0/2472060986 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4aa0010980 con 0x7f4ab808df80 2019-12-05T06:05:17.759 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:17.749+0000 7f4ab7fff700 1 -- 172.21.15.42:0/2472060986 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4aa00114a0 con 0x7f4ab808df80 2019-12-05T06:05:17.759 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:17.749+0000 7f4abed50700 1 -- 172.21.15.42:0/2472060986 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4ab808df80 msgr2=0x7f4ab809f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:05:17.759 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:17.749+0000 7f4abed50700 1 --2- 172.21.15.42:0/2472060986 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4ab808df80 0x7f4ab809f2e0 secure :-1 s=READY pgs=2000 cs=0 l=1 rx=0x7f4aa00032b0 tx=0x7f4aa00100b0).stop 2019-12-05T06:05:17.759 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:17.749+0000 7f4abed50700 1 -- 172.21.15.42:0/2472060986 shutdown_connections 2019-12-05T06:05:17.759 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:17.749+0000 7f4abed50700 1 --2- 172.21.15.42:0/2472060986 >> v2:172.21.15.40:3300/0 conn(0x7f4ab809f830 0x7f4ab814efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:05:17.759 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:17.749+0000 7f4abed50700 1 --2- 172.21.15.42:0/2472060986 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4ab808df80 0x7f4ab809f2e0 unknown :-1 s=CLOSED pgs=2000 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:05:17.759 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:17.749+0000 7f4abed50700 1 -- 172.21.15.42:0/2472060986 shutdown_connections 2019-12-05T06:05:17.760 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:17.749+0000 7f4abed50700 1 -- 172.21.15.42:0/2472060986 wait complete. 2019-12-05T06:05:17.760 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:17.749+0000 7f4abed50700 1 -- 172.21.15.42:0/2472060986 >> 172.21.15.42:0/2472060986 conn(0x7f4ab80aabe0 msgr2=0x7f4ab80a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:05:17.763 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:17.749+0000 7f4abed50700 1 Processor -- start 2019-12-05T06:05:17.763 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:17.753+0000 7f4abed50700 1 -- start start 2019-12-05T06:05:17.764 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:17.753+0000 7f4abed50700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f4ab8067320 0x7f4ab80a5f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:05:17.764 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:17.753+0000 7f4abed50700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4ab81336e0 0x7f4ab80a2810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:05:17.764 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:17.753+0000 7f4abed50700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ab80a1510 con 0x7f4ab8129c30 2019-12-05T06:05:17.764 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:17.753+0000 7f4abed50700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ab80a3120 con 0x7f4ab8067320 2019-12-05T06:05:17.764 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:17.753+0000 7f4abed50700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ab80a59e0 con 0x7f4ab81336e0 2019-12-05T06:05:17.765 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:17.753+0000 7f4abcaec700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4ab81336e0 0x7f4ab80a2810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:05:17.765 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:17.753+0000 7f4abd2ed700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f4ab8067320 0x7f4ab80a5f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:05:17.765 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:17.753+0000 7f4abcaec700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4ab81336e0 0x7f4ab80a2810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:48280/0 (socket says 172.21.15.42:48280) 2019-12-05T06:05:17.766 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:17.753+0000 7f4abd2ed700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f4ab8067320 0x7f4ab80a5f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.42:35628/0 (socket says 172.21.15.42:35628) 2019-12-05T06:05:17.766 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:17.753+0000 7f4abcaec700 1 -- 172.21.15.42:0/502139689 learned_addr learned my addr 172.21.15.42:0/502139689 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:05:17.766 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:17.753+0000 7f4abd2ed700 1 -- 172.21.15.42:0/502139689 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4ab81336e0 msgr2=0x7f4ab80a2810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:05:17.766 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:17.753+0000 7f4abd2ed700 1 --2- 172.21.15.42:0/502139689 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4ab81336e0 0x7f4ab80a2810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:05:17.766 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:17.753+0000 7f4abd2ed700 1 -- 172.21.15.42:0/502139689 >> v1:172.21.15.40:6789/0 conn(0x7f4ab8129c30 legacy=0x7f4ab812a060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:05:17.766 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:17.753+0000 7f4ab5ffb700 1 -- 172.21.15.42:0/502139689 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 464633031 0 0) 0x7f4ab80a1510 con 0x7f4ab8129c30 2019-12-05T06:05:17.766 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:17.753+0000 7f4abd2ed700 1 -- 172.21.15.42:0/502139689 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ab80a6470 con 0x7f4ab8067320 2019-12-05T06:05:17.767 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:17.753+0000 7f4abcaec700 1 --2- 172.21.15.42:0/502139689 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4ab81336e0 0x7f4ab80a2810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:05:17.767 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:17.753+0000 7f4abd2ed700 1 --2- 172.21.15.42:0/502139689 >> v2:172.21.15.40:3300/0 conn(0x7f4ab8067320 0x7f4ab80a5f20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f4ab80a1740 tx=0x7f4aa8010870).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:17.767 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:17.753+0000 7f4abd2ed700 1 --2- 172.21.15.42:0/502139689 >> v2:172.21.15.40:3300/0 conn(0x7f4ab8067320 0x7f4ab80a5f20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f4ab80a1740 tx=0x7f4aa8010870).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:17.767 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:17.753+0000 7f4abd2ed700 1 --2- 172.21.15.42:0/502139689 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ab8067320 0x7f4ab80a5f20 secure :-1 s=READY pgs=1514 cs=0 l=1 rx=0x7f4ab80a1740 tx=0x7f4aa8010870).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:05:17.767 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:17.753+0000 7f4abd2ed700 1 --2- 172.21.15.42:0/502139689 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ab8067320 0x7f4ab80a5f20 secure :-1 s=THROTTLE_DONE pgs=1514 cs=0 l=1 rx=0x7f4ab80a1740 tx=0x7f4aa8010870).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:17.767 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:17.753+0000 7f4ab5ffb700 1 -- 172.21.15.42:0/502139689 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4aa8006850 con 0x7f4ab8067320 2019-12-05T06:05:17.767 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:17.753+0000 7f4abd2ed700 1 --2- 172.21.15.42:0/502139689 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ab8067320 0x7f4ab80a5f20 secure :-1 s=THROTTLE_DONE pgs=1514 cs=0 l=1 rx=0x7f4ab80a1740 tx=0x7f4aa8010870).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:17.768 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:17.753+0000 7f4ab5ffb700 1 -- 172.21.15.42:0/502139689 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4aa8006470 con 0x7f4ab8067320 2019-12-05T06:05:17.768 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:17.753+0000 7f4abd2ed700 1 --2- 172.21.15.42:0/502139689 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ab8067320 0x7f4ab80a5f20 secure :-1 s=THROTTLE_DONE pgs=1514 cs=0 l=1 rx=0x7f4ab80a1740 tx=0x7f4aa8010870).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:17.768 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:17.753+0000 7f4abed50700 1 -- 172.21.15.42:0/502139689 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ab80a2d60 con 0x7f4ab8067320 2019-12-05T06:05:17.768 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:17.753+0000 7f4ab5ffb700 1 -- 172.21.15.42:0/502139689 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4aa8011180 con 0x7f4ab8067320 2019-12-05T06:05:17.768 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:17.753+0000 7f4abed50700 1 -- 172.21.15.42:0/502139689 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ab80762f0 con 0x7f4ab8067320 2019-12-05T06:05:17.768 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:17.757+0000 7f4abd2ed700 1 --2- 172.21.15.42:0/502139689 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ab8067320 0x7f4ab80a5f20 secure :-1 s=THROTTLE_DONE pgs=1514 cs=0 l=1 rx=0x7f4ab80a1740 tx=0x7f4aa8010870).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:17.769 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:17.757+0000 7f4ab5ffb700 1 -- 172.21.15.42:0/502139689 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f4aa80114a0 con 0x7f4ab8067320 2019-12-05T06:05:17.769 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:17.757+0000 7f4abd2ed700 1 --2- 172.21.15.42:0/502139689 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ab8067320 0x7f4ab80a5f20 secure :-1 s=THROTTLE_DONE pgs=1514 cs=0 l=1 rx=0x7f4ab80a1740 tx=0x7f4aa8010870).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:17.769 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:17.757+0000 7f4ab5ffb700 1 --2- 172.21.15.42:0/502139689 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4a9c03ac60 0x7f4a9c03d170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:05:17.769 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:17.757+0000 7f4ab5ffb700 1 -- 172.21.15.42:0/502139689 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9643+0+0 (secure 0 0 0) 0x7f4aa804b0c0 con 0x7f4ab8067320 2019-12-05T06:05:17.769 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:17.757+0000 7f4abcaec700 1 --2- 172.21.15.42:0/502139689 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4a9c03ac60 0x7f4a9c03d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:05:17.769 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:17.757+0000 7f4abcaec700 1 --2- 172.21.15.42:0/502139689 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4a9c03ac60 0x7f4a9c03d170 crc :-1 s=READY pgs=848 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:05:17.770 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:17.761+0000 7f4abed50700 1 -- 172.21.15.42:0/502139689 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ab805d110 con 0x7f4ab8067320 2019-12-05T06:05:17.775 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:17.769+0000 7f4abd2ed700 1 --2- 172.21.15.42:0/502139689 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ab8067320 0x7f4ab80a5f20 secure :-1 s=THROTTLE_DONE pgs=1514 cs=0 l=1 rx=0x7f4ab80a1740 tx=0x7f4aa8010870).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:17.775 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:17.773+0000 7f4ab5ffb700 1 -- 172.21.15.42:0/502139689 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f4aa8032080 con 0x7f4ab8067320 2019-12-05T06:05:17.949 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:17.945+0000 7f4abed50700 1 -- 172.21.15.42:0/502139689 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4ab805b8c0 con 0x7f4ab8067320 2019-12-05T06:05:17.950 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:17.945+0000 7f4abd2ed700 1 --2- 172.21.15.42:0/502139689 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ab8067320 0x7f4ab80a5f20 secure :-1 s=THROTTLE_DONE pgs=1514 cs=0 l=1 rx=0x7f4ab80a1740 tx=0x7f4aa8010870).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:17.950 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:17.945+0000 7f4ab5ffb700 1 -- 172.21.15.42:0/502139689 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 243 v243) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f4aa80322e0 con 0x7f4ab8067320 2019-12-05T06:05:17.953 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:05:17.953 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":243,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6821,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/2253267554","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2253267554},{"type":"v1","addr":"172.21.15.42:6835","nonce":2253267554}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":243}],"filesystems":[{"mdsmap":{"epoch":242,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:04:09.851907+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":91,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9381},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9381":{"gid":9381,"name":"b","rank":0,"incarnation":239,"state":"up:active","state_seq":37,"addr":"172.21.15.40:6833/3789081777","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3789081777},{"type":"v1","addr":"172.21.15.40:6833","nonce":3789081777}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:05:17.953 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 243 2019-12-05T06:05:17.955 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:17.949+0000 7f4abed50700 1 -- 172.21.15.42:0/502139689 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4a9c03ac60 msgr2=0x7f4a9c03d170 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:05:17.955 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:17.949+0000 7f4abed50700 1 --2- 172.21.15.42:0/502139689 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4a9c03ac60 0x7f4a9c03d170 crc :-1 s=READY pgs=848 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:05:17.955 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:17.949+0000 7f4abed50700 1 -- 172.21.15.42:0/502139689 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ab8067320 msgr2=0x7f4ab80a5f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:05:17.956 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:17.949+0000 7f4abed50700 1 --2- 172.21.15.42:0/502139689 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ab8067320 0x7f4ab80a5f20 secure :-1 s=READY pgs=1514 cs=0 l=1 rx=0x7f4ab80a1740 tx=0x7f4aa8010870).stop 2019-12-05T06:05:17.956 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:17.949+0000 7f4abed50700 1 -- 172.21.15.42:0/502139689 shutdown_connections 2019-12-05T06:05:17.956 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:17.949+0000 7f4abed50700 1 --2- 172.21.15.42:0/502139689 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4a9c03ac60 0x7f4a9c03d170 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:05:17.956 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:17.949+0000 7f4abed50700 1 --2- 172.21.15.42:0/502139689 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ab8067320 0x7f4ab80a5f20 unknown :-1 s=CLOSED pgs=1514 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:05:17.956 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:17.949+0000 7f4abed50700 1 --2- 172.21.15.42:0/502139689 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4ab81336e0 0x7f4ab80a2810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:05:17.956 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:17.949+0000 7f4abed50700 1 -- 172.21.15.42:0/502139689 shutdown_connections 2019-12-05T06:05:17.957 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:17.949+0000 7f4abed50700 1 -- 172.21.15.42:0/502139689 wait complete. 2019-12-05T06:05:17.957 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:17.949+0000 7f4abed50700 1 -- 172.21.15.42:0/502139689 >> 172.21.15.42:0/502139689 conn(0x7f4ab80a82a0 msgr2=0x7f4ab811ca60 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:05:17.965 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:05:19.980 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:05:20.088 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:20.077+0000 7fda0045c700 1 Processor -- start 2019-12-05T06:05:20.088 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:20.077+0000 7fda0045c700 1 -- start start 2019-12-05T06:05:20.088 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:20.081+0000 7fda0045c700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd9f80a7740 0x7fd9f808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:05:20.088 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:20.081+0000 7fda0045c700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd9f807e720 0x7fd9f807eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:05:20.089 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:20.081+0000 7fda0045c700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9f80a0710 con 0x7fd9f809f2e0 2019-12-05T06:05:20.089 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:20.081+0000 7fda0045c700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9f80a0b00 con 0x7fd9f80a7740 2019-12-05T06:05:20.089 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:20.081+0000 7fda0045c700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9f808e5b0 con 0x7fd9f807e720 2019-12-05T06:05:20.089 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:20.081+0000 7fd9fe1f8700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd9f807e720 0x7fd9f807eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:05:20.089 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:20.081+0000 7fd9fe9f9700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd9f80a7740 0x7fd9f808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:05:20.089 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:20.081+0000 7fd9fe1f8700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd9f807e720 0x7fd9f807eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:48290/0 (socket says 172.21.15.42:48290) 2019-12-05T06:05:20.089 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:20.081+0000 7fd9fe1f8700 1 -- 172.21.15.42:0/823085069 learned_addr learned my addr 172.21.15.42:0/823085069 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:05:20.090 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:20.081+0000 7fd9fe1f8700 1 -- 172.21.15.42:0/823085069 >> v2:172.21.15.40:3300/0 conn(0x7fd9f80a7740 msgr2=0x7fd9f808df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:05:20.090 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:20.081+0000 7fd9fe1f8700 1 --2- 172.21.15.42:0/823085069 >> v2:172.21.15.40:3300/0 conn(0x7fd9f80a7740 0x7fd9f808df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:05:20.090 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:20.081+0000 7fd9fe1f8700 1 -- 172.21.15.42:0/823085069 >> v1:172.21.15.40:6789/0 conn(0x7fd9f809f2e0 legacy=0x7fd9f809f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:05:20.090 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:20.081+0000 7fd9fd9f7700 1 -- 172.21.15.42:0/823085069 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4121532167 0 0) 0x7fd9f80a0710 con 0x7fd9f809f2e0 2019-12-05T06:05:20.090 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:20.081+0000 7fd9fe1f8700 1 -- 172.21.15.42:0/823085069 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9f807f0a0 con 0x7fd9f807e720 2019-12-05T06:05:20.090 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:20.081+0000 7fd9fe1f8700 1 --2- 172.21.15.42:0/823085069 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd9f807e720 0x7fd9f807eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fd9ec008790 tx=0x7fd9ec00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:20.090 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:20.081+0000 7fd9fe1f8700 1 --2- 172.21.15.42:0/823085069 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd9f807e720 0x7fd9f807eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fd9ec008790 tx=0x7fd9ec00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:20.091 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:20.081+0000 7fd9fe1f8700 1 --2- 172.21.15.42:0/823085069 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd9f807e720 0x7fd9f807eb50 secure :-1 s=READY pgs=447 cs=0 l=1 rx=0x7fd9ec008790 tx=0x7fd9ec00fd30).ready entity=mon.2 client_cookie=d9115dfaa2200ab4 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:05:20.091 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:20.081+0000 7fd9fe1f8700 1 --2- 172.21.15.42:0/823085069 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd9f807e720 0x7fd9f807eb50 secure :-1 s=THROTTLE_DONE pgs=447 cs=0 l=1 rx=0x7fd9ec008790 tx=0x7fd9ec00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:20.091 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:20.081+0000 7fd9fd9f7700 1 -- 172.21.15.42:0/823085069 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd9ec0113b0 con 0x7fd9f807e720 2019-12-05T06:05:20.091 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:20.081+0000 7fd9fe1f8700 1 --2- 172.21.15.42:0/823085069 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd9f807e720 0x7fd9f807eb50 secure :-1 s=THROTTLE_DONE pgs=447 cs=0 l=1 rx=0x7fd9ec008790 tx=0x7fd9ec00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:20.092 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:20.081+0000 7fd9fe1f8700 1 --2- 172.21.15.42:0/823085069 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd9f807e720 0x7fd9f807eb50 secure :-1 s=THROTTLE_DONE pgs=447 cs=0 l=1 rx=0x7fd9ec008790 tx=0x7fd9ec00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:20.092 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:20.081+0000 7fd9fd9f7700 1 -- 172.21.15.42:0/823085069 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9ec008bb0 con 0x7fd9f807e720 2019-12-05T06:05:20.092 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:20.081+0000 7fd9fd9f7700 1 -- 172.21.15.42:0/823085069 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd9ec011990 con 0x7fd9f807e720 2019-12-05T06:05:20.092 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:20.081+0000 7fda0045c700 1 -- 172.21.15.42:0/823085069 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd9f807e720 msgr2=0x7fd9f807eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:05:20.092 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:20.081+0000 7fda0045c700 1 --2- 172.21.15.42:0/823085069 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd9f807e720 0x7fd9f807eb50 secure :-1 s=READY pgs=447 cs=0 l=1 rx=0x7fd9ec008790 tx=0x7fd9ec00fd30).stop 2019-12-05T06:05:20.092 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:20.081+0000 7fda0045c700 1 -- 172.21.15.42:0/823085069 shutdown_connections 2019-12-05T06:05:20.092 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:20.081+0000 7fda0045c700 1 --2- 172.21.15.42:0/823085069 >> v2:172.21.15.40:3300/0 conn(0x7fd9f80a7740 0x7fd9f808df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:05:20.093 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:20.081+0000 7fda0045c700 1 --2- 172.21.15.42:0/823085069 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd9f807e720 0x7fd9f807eb50 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:05:20.093 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:20.081+0000 7fda0045c700 1 -- 172.21.15.42:0/823085069 shutdown_connections 2019-12-05T06:05:20.093 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:20.081+0000 7fda0045c700 1 -- 172.21.15.42:0/823085069 wait complete. 2019-12-05T06:05:20.093 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:20.081+0000 7fda0045c700 1 -- 172.21.15.42:0/823085069 >> 172.21.15.42:0/823085069 conn(0x7fd9f80aabe0 msgr2=0x7fd9f80a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:05:20.096 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:20.085+0000 7fda0045c700 1 Processor -- start 2019-12-05T06:05:20.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:20.085+0000 7fda0045c700 1 -- start start 2019-12-05T06:05:20.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:20.085+0000 7fda0045c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd9f805a210 0x7fd9f8115510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:05:20.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:20.085+0000 7fda0045c700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd9f8127110 0x7fd9f8127540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:05:20.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:20.085+0000 7fda0045c700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9f8115f20 con 0x7fd9f812a590 2019-12-05T06:05:20.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:20.085+0000 7fda0045c700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9f812a050 con 0x7fd9f805a210 2019-12-05T06:05:20.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:20.085+0000 7fda0045c700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9f812a2f0 con 0x7fd9f8127110 2019-12-05T06:05:20.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:20.085+0000 7fd9fe9f9700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd9f805a210 0x7fd9f8115510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:05:20.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:20.085+0000 7fd9fe9f9700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd9f805a210 0x7fd9f8115510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:47832/0 (socket says 172.21.15.42:47832) 2019-12-05T06:05:20.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:20.085+0000 7fd9fe9f9700 1 -- 172.21.15.42:0/1049797797 learned_addr learned my addr 172.21.15.42:0/1049797797 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:05:20.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:20.085+0000 7fd9fe1f8700 1 --2- 172.21.15.42:0/1049797797 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd9f8127110 0x7fd9f8127540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:05:20.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:20.085+0000 7fd9eb7fe700 1 -- 172.21.15.42:0/1049797797 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3676787309 0 0) 0x7fd9f8115f20 con 0x7fd9f812a590 2019-12-05T06:05:20.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:20.085+0000 7fd9eb7fe700 1 -- 172.21.15.42:0/1049797797 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9e00030b0 con 0x7fd9f812a590 2019-12-05T06:05:20.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:20.089+0000 7fd9fe1f8700 1 -- 172.21.15.42:0/1049797797 >> v2:172.21.15.42:3300/0 conn(0x7fd9f805a210 msgr2=0x7fd9f8115510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:05:20.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:20.089+0000 7fd9fe1f8700 1 --2- 172.21.15.42:0/1049797797 >> v2:172.21.15.42:3300/0 conn(0x7fd9f805a210 0x7fd9f8115510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:05:20.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:20.089+0000 7fd9fe1f8700 1 -- 172.21.15.42:0/1049797797 >> v1:172.21.15.40:6789/0 conn(0x7fd9f812a590 legacy=0x7fd9f808eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:05:20.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:20.089+0000 7fd9fe1f8700 1 -- 172.21.15.42:0/1049797797 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9f81269f0 con 0x7fd9f8127110 2019-12-05T06:05:20.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:20.089+0000 7fd9fe9f9700 1 --2- 172.21.15.42:0/1049797797 >> v2:172.21.15.42:3300/0 conn(0x7fd9f805a210 0x7fd9f8115510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:05:20.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:20.089+0000 7fd9fe1f8700 1 --2- 172.21.15.42:0/1049797797 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd9f8127110 0x7fd9f8127540 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fd9ec008790 tx=0x7fd9ec008bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:20.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:20.089+0000 7fd9fe1f8700 1 --2- 172.21.15.42:0/1049797797 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd9f8127110 0x7fd9f8127540 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fd9ec008790 tx=0x7fd9ec008bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:20.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:20.089+0000 7fd9fe1f8700 1 --2- 172.21.15.42:0/1049797797 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd9f8127110 0x7fd9f8127540 secure :-1 s=READY pgs=448 cs=0 l=1 rx=0x7fd9ec008790 tx=0x7fd9ec008bb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:05:20.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:20.089+0000 7fd9fe1f8700 1 --2- 172.21.15.42:0/1049797797 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd9f8127110 0x7fd9f8127540 secure :-1 s=THROTTLE_DONE pgs=448 cs=0 l=1 rx=0x7fd9ec008790 tx=0x7fd9ec008bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:20.100 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:20.089+0000 7fd9eb7fe700 1 -- 172.21.15.42:0/1049797797 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd9ec007150 con 0x7fd9f8127110 2019-12-05T06:05:20.100 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:20.089+0000 7fd9fe1f8700 1 --2- 172.21.15.42:0/1049797797 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd9f8127110 0x7fd9f8127540 secure :-1 s=THROTTLE_DONE pgs=448 cs=0 l=1 rx=0x7fd9ec008790 tx=0x7fd9ec008bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:20.100 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:20.089+0000 7fd9fe1f8700 1 --2- 172.21.15.42:0/1049797797 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd9f8127110 0x7fd9f8127540 secure :-1 s=THROTTLE_DONE pgs=448 cs=0 l=1 rx=0x7fd9ec008790 tx=0x7fd9ec008bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:20.100 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:20.089+0000 7fd9eb7fe700 1 -- 172.21.15.42:0/1049797797 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9ec0115b0 con 0x7fd9f8127110 2019-12-05T06:05:20.100 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:20.089+0000 7fda0045c700 1 -- 172.21.15.42:0/1049797797 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9f8114750 con 0x7fd9f8127110 2019-12-05T06:05:20.100 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:20.089+0000 7fd9eb7fe700 1 -- 172.21.15.42:0/1049797797 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd9ec005730 con 0x7fd9f8127110 2019-12-05T06:05:20.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:20.089+0000 7fda0045c700 1 -- 172.21.15.42:0/1049797797 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9f812a9c0 con 0x7fd9f8127110 2019-12-05T06:05:20.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:20.089+0000 7fd9fe1f8700 1 --2- 172.21.15.42:0/1049797797 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd9f8127110 0x7fd9f8127540 secure :-1 s=THROTTLE_DONE pgs=448 cs=0 l=1 rx=0x7fd9ec008790 tx=0x7fd9ec008bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:20.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:20.089+0000 7fd9eb7fe700 1 -- 172.21.15.42:0/1049797797 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fd9ec005a40 con 0x7fd9f8127110 2019-12-05T06:05:20.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:20.089+0000 7fd9fe1f8700 1 --2- 172.21.15.42:0/1049797797 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd9f8127110 0x7fd9f8127540 secure :-1 s=THROTTLE_DONE pgs=448 cs=0 l=1 rx=0x7fd9ec008790 tx=0x7fd9ec008bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:20.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:20.089+0000 7fd9eb7fe700 1 --2- 172.21.15.42:0/1049797797 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd9e003b2a0 0x7fd9e003d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:05:20.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:20.089+0000 7fd9eb7fe700 1 -- 172.21.15.42:0/1049797797 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9643+0+0 (secure 0 0 0) 0x7fd9ec04b3b0 con 0x7fd9f8127110 2019-12-05T06:05:20.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:20.089+0000 7fd9fe9f9700 1 --2- 172.21.15.42:0/1049797797 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd9e003b2a0 0x7fd9e003d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:05:20.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:20.089+0000 7fd9fe9f9700 1 --2- 172.21.15.42:0/1049797797 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd9e003b2a0 0x7fd9e003d7b0 crc :-1 s=READY pgs=849 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:05:20.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:20.093+0000 7fda0045c700 1 -- 172.21.15.42:0/1049797797 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9f805d110 con 0x7fd9f8127110 2019-12-05T06:05:20.108 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:20.105+0000 7fd9fe1f8700 1 --2- 172.21.15.42:0/1049797797 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd9f8127110 0x7fd9f8127540 secure :-1 s=THROTTLE_DONE pgs=448 cs=0 l=1 rx=0x7fd9ec008790 tx=0x7fd9ec008bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:20.109 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:20.105+0000 7fd9eb7fe700 1 -- 172.21.15.42:0/1049797797 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fd9ec005d50 con 0x7fd9f8127110 2019-12-05T06:05:20.278 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:20.273+0000 7fda0045c700 1 -- 172.21.15.42:0/1049797797 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd9f805b8c0 con 0x7fd9f8127110 2019-12-05T06:05:20.279 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:20.273+0000 7fd9fe1f8700 1 --2- 172.21.15.42:0/1049797797 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd9f8127110 0x7fd9f8127540 secure :-1 s=THROTTLE_DONE pgs=448 cs=0 l=1 rx=0x7fd9ec008790 tx=0x7fd9ec008bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:20.279 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:20.273+0000 7fd9eb7fe700 1 -- 172.21.15.42:0/1049797797 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 243 v243) v1 ==== 95+0+2112 (secure 0 0 0) 0x7fd9ec012cf0 con 0x7fd9f8127110 2019-12-05T06:05:20.281 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:05:20.281 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":243,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6821,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/2253267554","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2253267554},{"type":"v1","addr":"172.21.15.42:6835","nonce":2253267554}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":243}],"filesystems":[{"mdsmap":{"epoch":242,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:04:09.851907+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":91,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9381},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9381":{"gid":9381,"name":"b","rank":0,"incarnation":239,"state":"up:active","state_seq":37,"addr":"172.21.15.40:6833/3789081777","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3789081777},{"type":"v1","addr":"172.21.15.40:6833","nonce":3789081777}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:05:20.281 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 243 2019-12-05T06:05:20.285 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:20.277+0000 7fda0045c700 1 -- 172.21.15.42:0/1049797797 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd9e003b2a0 msgr2=0x7fd9e003d7b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:05:20.285 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:20.277+0000 7fda0045c700 1 --2- 172.21.15.42:0/1049797797 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd9e003b2a0 0x7fd9e003d7b0 crc :-1 s=READY pgs=849 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:05:20.285 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:20.277+0000 7fda0045c700 1 -- 172.21.15.42:0/1049797797 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd9f8127110 msgr2=0x7fd9f8127540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:05:20.285 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:20.277+0000 7fda0045c700 1 --2- 172.21.15.42:0/1049797797 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd9f8127110 0x7fd9f8127540 secure :-1 s=READY pgs=448 cs=0 l=1 rx=0x7fd9ec008790 tx=0x7fd9ec008bb0).stop 2019-12-05T06:05:20.285 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:20.281+0000 7fda0045c700 1 -- 172.21.15.42:0/1049797797 shutdown_connections 2019-12-05T06:05:20.285 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:20.281+0000 7fda0045c700 1 --2- 172.21.15.42:0/1049797797 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd9e003b2a0 0x7fd9e003d7b0 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:05:20.286 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:20.281+0000 7fda0045c700 1 --2- 172.21.15.42:0/1049797797 >> v2:172.21.15.42:3300/0 conn(0x7fd9f805a210 0x7fd9f8115510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:05:20.286 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:20.281+0000 7fda0045c700 1 --2- 172.21.15.42:0/1049797797 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd9f8127110 0x7fd9f8127540 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:05:20.286 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:20.281+0000 7fda0045c700 1 -- 172.21.15.42:0/1049797797 shutdown_connections 2019-12-05T06:05:20.286 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:20.281+0000 7fda0045c700 1 -- 172.21.15.42:0/1049797797 wait complete. 2019-12-05T06:05:20.286 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:20.281+0000 7fda0045c700 1 -- 172.21.15.42:0/1049797797 >> 172.21.15.42:0/1049797797 conn(0x7fd9f80a9640 msgr2=0x7fd9f809f190 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:05:20.294 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:05:22.295 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:05:22.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:22.389+0000 7f81e2f68700 1 Processor -- start 2019-12-05T06:05:22.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:22.389+0000 7f81e2f68700 1 -- start start 2019-12-05T06:05:22.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:22.393+0000 7f81e2f68700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f81dc0a7730 0x7f81dc08df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:05:22.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:22.393+0000 7f81e2f68700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f81dc07e710 0x7f81dc07eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:05:22.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:22.393+0000 7f81e2f68700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f81dc09f2d0 0x7f81dc09f700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:05:22.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:22.393+0000 7f81e2f68700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81dc0a0af0 con 0x7f81dc09f2d0 2019-12-05T06:05:22.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:22.393+0000 7f81e2f68700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81dc08e5a0 con 0x7f81dc0a7730 2019-12-05T06:05:22.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:22.393+0000 7f81e2f68700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81dc08e840 con 0x7f81dc07e710 2019-12-05T06:05:22.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:22.393+0000 7f81e1505700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f81dc0a7730 0x7f81dc08df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:05:22.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:22.393+0000 7f81e1505700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f81dc0a7730 0x7f81dc08df70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:47840/0 (socket says 172.21.15.42:47840) 2019-12-05T06:05:22.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:22.393+0000 7f81e1505700 1 -- 172.21.15.42:0/2541778430 learned_addr learned my addr 172.21.15.42:0/2541778430 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:05:22.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:22.393+0000 7f81e0d04700 1 --2- 172.21.15.42:0/2541778430 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f81dc07e710 0x7f81dc07eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:05:22.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:22.393+0000 7f81e1d06700 1 --2- 172.21.15.42:0/2541778430 >> v2:172.21.15.40:3300/0 conn(0x7f81dc09f2d0 0x7f81dc09f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:05:22.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:22.393+0000 7f81e1505700 1 -- 172.21.15.42:0/2541778430 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f81dc07e710 msgr2=0x7f81dc07eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:05:22.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:22.393+0000 7f81e1505700 1 --2- 172.21.15.42:0/2541778430 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f81dc07e710 0x7f81dc07eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:05:22.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:22.393+0000 7f81e1505700 1 -- 172.21.15.42:0/2541778430 >> v2:172.21.15.40:3300/0 conn(0x7f81dc09f2d0 msgr2=0x7f81dc09f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:05:22.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:22.393+0000 7f81e1505700 1 --2- 172.21.15.42:0/2541778430 >> v2:172.21.15.40:3300/0 conn(0x7f81dc09f2d0 0x7f81dc09f700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:05:22.404 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:22.393+0000 7f81e1505700 1 -- 172.21.15.42:0/2541778430 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81dc07f090 con 0x7f81dc0a7730 2019-12-05T06:05:22.404 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:22.393+0000 7f81e1d06700 1 --2- 172.21.15.42:0/2541778430 >> v2:172.21.15.40:3300/0 conn(0x7f81dc09f2d0 0x7f81dc09f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:05:22.404 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:22.393+0000 7f81e1505700 1 --2- 172.21.15.42:0/2541778430 >> v2:172.21.15.42:3300/0 conn(0x7f81dc0a7730 0x7f81dc08df70 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f81dc08e760 tx=0x7f81d800fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:22.404 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:22.393+0000 7f81e1505700 1 --2- 172.21.15.42:0/2541778430 >> v2:172.21.15.42:3300/0 conn(0x7f81dc0a7730 0x7f81dc08df70 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f81dc08e760 tx=0x7f81d800fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:22.404 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:22.393+0000 7f81e1505700 1 --2- 172.21.15.42:0/2541778430 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f81dc0a7730 0x7f81dc08df70 secure :-1 s=READY pgs=2004 cs=0 l=1 rx=0x7f81dc08e760 tx=0x7f81d800fdb0).ready entity=mon.1 client_cookie=fd2bc4ef94719334 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:05:22.404 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:22.393+0000 7f81e1505700 1 --2- 172.21.15.42:0/2541778430 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f81dc0a7730 0x7f81dc08df70 secure :-1 s=THROTTLE_DONE pgs=2004 cs=0 l=1 rx=0x7f81dc08e760 tx=0x7f81d800fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:22.404 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:22.393+0000 7f81e1505700 1 --2- 172.21.15.42:0/2541778430 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f81dc0a7730 0x7f81dc08df70 secure :-1 s=THROTTLE_DONE pgs=2004 cs=0 l=1 rx=0x7f81dc08e760 tx=0x7f81d800fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:22.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:22.393+0000 7f81d3fff700 1 -- 172.21.15.42:0/2541778430 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f81d8008c90 con 0x7f81dc0a7730 2019-12-05T06:05:22.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:22.393+0000 7f81e1505700 1 --2- 172.21.15.42:0/2541778430 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f81dc0a7730 0x7f81dc08df70 secure :-1 s=THROTTLE_DONE pgs=2004 cs=0 l=1 rx=0x7f81dc08e760 tx=0x7f81d800fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:22.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:22.393+0000 7f81d3fff700 1 -- 172.21.15.42:0/2541778430 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81d80088b0 con 0x7f81dc0a7730 2019-12-05T06:05:22.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:22.393+0000 7f81d3fff700 1 -- 172.21.15.42:0/2541778430 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f81d8011b50 con 0x7f81dc0a7730 2019-12-05T06:05:22.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:22.393+0000 7f81e2f68700 1 -- 172.21.15.42:0/2541778430 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f81dc0a7730 msgr2=0x7f81dc08df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:05:22.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:22.393+0000 7f81e2f68700 1 --2- 172.21.15.42:0/2541778430 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f81dc0a7730 0x7f81dc08df70 secure :-1 s=READY pgs=2004 cs=0 l=1 rx=0x7f81dc08e760 tx=0x7f81d800fdb0).stop 2019-12-05T06:05:22.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:22.393+0000 7f81e2f68700 1 -- 172.21.15.42:0/2541778430 shutdown_connections 2019-12-05T06:05:22.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:22.397+0000 7f81e2f68700 1 --2- 172.21.15.42:0/2541778430 >> v2:172.21.15.40:3300/0 conn(0x7f81dc09f2d0 0x7f81dc09f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:05:22.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:22.397+0000 7f81e2f68700 1 --2- 172.21.15.42:0/2541778430 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f81dc0a7730 0x7f81dc08df70 unknown :-1 s=CLOSED pgs=2004 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:05:22.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:22.397+0000 7f81e2f68700 1 --2- 172.21.15.42:0/2541778430 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f81dc07e710 0x7f81dc07eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:05:22.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:22.397+0000 7f81e2f68700 1 -- 172.21.15.42:0/2541778430 shutdown_connections 2019-12-05T06:05:22.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:22.397+0000 7f81e2f68700 1 -- 172.21.15.42:0/2541778430 wait complete. 2019-12-05T06:05:22.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:22.397+0000 7f81e2f68700 1 -- 172.21.15.42:0/2541778430 >> 172.21.15.42:0/2541778430 conn(0x7f81dc0aabd0 msgr2=0x7f81dc0a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:05:22.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:22.397+0000 7f81e2f68700 1 Processor -- start 2019-12-05T06:05:22.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:22.397+0000 7f81e2f68700 1 -- start start 2019-12-05T06:05:22.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:22.401+0000 7f81e2f68700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f81dc07cbf0 0x7f81dc07d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:05:22.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:22.401+0000 7f81e2f68700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f81dc060000 0x7f81dc060430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:05:22.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:22.401+0000 7f81e2f68700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81dc144b60 con 0x7f81dc0a3a50 2019-12-05T06:05:22.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:22.401+0000 7f81e2f68700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81dc144e70 con 0x7f81dc07cbf0 2019-12-05T06:05:22.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:22.401+0000 7f81e2f68700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81dc145110 con 0x7f81dc060000 2019-12-05T06:05:22.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:22.401+0000 7f81e1d06700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f81dc060000 0x7f81dc060430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:05:22.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:22.401+0000 7f81e1d06700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f81dc060000 0x7f81dc060430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:47850/0 (socket says 172.21.15.42:47850) 2019-12-05T06:05:22.411 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:22.401+0000 7f81e1d06700 1 -- 172.21.15.42:0/2083190573 learned_addr learned my addr 172.21.15.42:0/2083190573 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:05:22.411 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:22.401+0000 7f81e0d04700 1 --2- 172.21.15.42:0/2083190573 >> v2:172.21.15.40:3300/0 conn(0x7f81dc07cbf0 0x7f81dc07d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:05:22.411 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:22.401+0000 7f81e1d06700 1 -- 172.21.15.42:0/2083190573 >> v2:172.21.15.40:3300/0 conn(0x7f81dc07cbf0 msgr2=0x7f81dc07d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:05:22.411 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:22.401+0000 7f81e1d06700 1 --2- 172.21.15.42:0/2083190573 >> v2:172.21.15.40:3300/0 conn(0x7f81dc07cbf0 0x7f81dc07d020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:05:22.411 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:22.401+0000 7f81e1d06700 1 -- 172.21.15.42:0/2083190573 >> v1:172.21.15.40:6789/0 conn(0x7f81dc0a3a50 legacy=0x7f81dc1441b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:05:22.411 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:22.401+0000 7f81e1d06700 1 -- 172.21.15.42:0/2083190573 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81dc060980 con 0x7f81dc060000 2019-12-05T06:05:22.412 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:22.401+0000 7f81e1d06700 1 --2- 172.21.15.42:0/2083190573 >> v2:172.21.15.42:3300/0 conn(0x7f81dc060000 0x7f81dc060430 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f81d40032b0 tx=0x7f81d4010b80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:22.412 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:22.401+0000 7f81e1d06700 1 --2- 172.21.15.42:0/2083190573 >> v2:172.21.15.42:3300/0 conn(0x7f81dc060000 0x7f81dc060430 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f81d40032b0 tx=0x7f81d4010b80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:22.412 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:22.401+0000 7f81e1d06700 1 --2- 172.21.15.42:0/2083190573 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f81dc060000 0x7f81dc060430 secure :-1 s=READY pgs=2005 cs=0 l=1 rx=0x7f81d40032b0 tx=0x7f81d4010b80).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:05:22.412 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:22.401+0000 7f81e1d06700 1 --2- 172.21.15.42:0/2083190573 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f81dc060000 0x7f81dc060430 secure :-1 s=THROTTLE_DONE pgs=2005 cs=0 l=1 rx=0x7f81d40032b0 tx=0x7f81d4010b80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:22.412 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:22.401+0000 7f81e1d06700 1 --2- 172.21.15.42:0/2083190573 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f81dc060000 0x7f81dc060430 secure :-1 s=THROTTLE_DONE pgs=2005 cs=0 l=1 rx=0x7f81d40032b0 tx=0x7f81d4010b80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:22.412 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:22.401+0000 7f81d1ffb700 1 -- 172.21.15.42:0/2083190573 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f81d4012070 con 0x7f81dc060000 2019-12-05T06:05:22.412 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:22.401+0000 7f81e1d06700 1 --2- 172.21.15.42:0/2083190573 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f81dc060000 0x7f81dc060430 secure :-1 s=THROTTLE_DONE pgs=2005 cs=0 l=1 rx=0x7f81d40032b0 tx=0x7f81d4010b80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:22.413 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:22.401+0000 7f81d1ffb700 1 -- 172.21.15.42:0/2083190573 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81d400f0c0 con 0x7f81dc060000 2019-12-05T06:05:22.413 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:22.401+0000 7f81e2f68700 1 -- 172.21.15.42:0/2083190573 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81dc1389d0 con 0x7f81dc060000 2019-12-05T06:05:22.413 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:22.401+0000 7f81d1ffb700 1 -- 172.21.15.42:0/2083190573 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f81d400e390 con 0x7f81dc060000 2019-12-05T06:05:22.413 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:22.401+0000 7f81e2f68700 1 -- 172.21.15.42:0/2083190573 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81dc138f00 con 0x7f81dc060000 2019-12-05T06:05:22.413 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:22.401+0000 7f81e1d06700 1 --2- 172.21.15.42:0/2083190573 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f81dc060000 0x7f81dc060430 secure :-1 s=THROTTLE_DONE pgs=2005 cs=0 l=1 rx=0x7f81d40032b0 tx=0x7f81d4010b80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:22.413 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:22.405+0000 7f81d1ffb700 1 -- 172.21.15.42:0/2083190573 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f81d4007180 con 0x7f81dc060000 2019-12-05T06:05:22.413 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:22.405+0000 7f81e1d06700 1 --2- 172.21.15.42:0/2083190573 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f81dc060000 0x7f81dc060430 secure :-1 s=THROTTLE_DONE pgs=2005 cs=0 l=1 rx=0x7f81d40032b0 tx=0x7f81d4010b80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:22.414 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:22.405+0000 7f81d1ffb700 1 --2- 172.21.15.42:0/2083190573 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f81bc03ad60 0x7f81bc03d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:05:22.414 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:22.405+0000 7f81d1ffb700 1 -- 172.21.15.42:0/2083190573 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9643+0+0 (secure 0 0 0) 0x7f81d404adf0 con 0x7f81dc060000 2019-12-05T06:05:22.414 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:22.405+0000 7f81e1505700 1 --2- 172.21.15.42:0/2083190573 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f81bc03ad60 0x7f81bc03d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:05:22.414 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:22.405+0000 7f81e1505700 1 --2- 172.21.15.42:0/2083190573 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f81bc03ad60 0x7f81bc03d270 crc :-1 s=READY pgs=850 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:05:22.415 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:22.405+0000 7f81e2f68700 1 -- 172.21.15.42:0/2083190573 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81dc05d100 con 0x7f81dc060000 2019-12-05T06:05:22.418 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:22.413+0000 7f81e1d06700 1 --2- 172.21.15.42:0/2083190573 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f81dc060000 0x7f81dc060430 secure :-1 s=THROTTLE_DONE pgs=2005 cs=0 l=1 rx=0x7f81d40032b0 tx=0x7f81d4010b80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:22.418 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:22.413+0000 7f81d1ffb700 1 -- 172.21.15.42:0/2083190573 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f81d40320d0 con 0x7f81dc060000 2019-12-05T06:05:22.585 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:22.581+0000 7f81e2f68700 1 -- 172.21.15.42:0/2083190573 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f81dc05b8b0 con 0x7f81dc060000 2019-12-05T06:05:22.585 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:22.581+0000 7f81e1d06700 1 --2- 172.21.15.42:0/2083190573 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f81dc060000 0x7f81dc060430 secure :-1 s=THROTTLE_DONE pgs=2005 cs=0 l=1 rx=0x7f81d40032b0 tx=0x7f81d4010b80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:22.586 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:22.581+0000 7f81d1ffb700 1 -- 172.21.15.42:0/2083190573 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 243 v243) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f81d4007490 con 0x7f81dc060000 2019-12-05T06:05:22.586 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 243 2019-12-05T06:05:22.586 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:05:22.586 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":243,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6821,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/2253267554","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2253267554},{"type":"v1","addr":"172.21.15.42:6835","nonce":2253267554}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":243}],"filesystems":[{"mdsmap":{"epoch":242,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:04:09.851907+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":91,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9381},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9381":{"gid":9381,"name":"b","rank":0,"incarnation":239,"state":"up:active","state_seq":37,"addr":"172.21.15.40:6833/3789081777","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3789081777},{"type":"v1","addr":"172.21.15.40:6833","nonce":3789081777}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:05:22.588 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:22.585+0000 7f81e2f68700 1 -- 172.21.15.42:0/2083190573 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f81bc03ad60 msgr2=0x7f81bc03d270 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:05:22.588 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:22.585+0000 7f81e2f68700 1 --2- 172.21.15.42:0/2083190573 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f81bc03ad60 0x7f81bc03d270 crc :-1 s=READY pgs=850 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:05:22.589 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:22.585+0000 7f81e2f68700 1 -- 172.21.15.42:0/2083190573 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f81dc060000 msgr2=0x7f81dc060430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:05:22.589 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:22.585+0000 7f81e2f68700 1 --2- 172.21.15.42:0/2083190573 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f81dc060000 0x7f81dc060430 secure :-1 s=READY pgs=2005 cs=0 l=1 rx=0x7f81d40032b0 tx=0x7f81d4010b80).stop 2019-12-05T06:05:22.589 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:22.585+0000 7f81e2f68700 1 -- 172.21.15.42:0/2083190573 shutdown_connections 2019-12-05T06:05:22.589 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:22.585+0000 7f81e2f68700 1 --2- 172.21.15.42:0/2083190573 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f81bc03ad60 0x7f81bc03d270 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:05:22.589 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:22.585+0000 7f81e2f68700 1 --2- 172.21.15.42:0/2083190573 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f81dc060000 0x7f81dc060430 unknown :-1 s=CLOSED pgs=2005 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:05:22.589 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:22.585+0000 7f81e2f68700 1 --2- 172.21.15.42:0/2083190573 >> v2:172.21.15.40:3300/0 conn(0x7f81dc07cbf0 0x7f81dc07d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:05:22.590 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:22.585+0000 7f81e2f68700 1 -- 172.21.15.42:0/2083190573 shutdown_connections 2019-12-05T06:05:22.590 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:22.585+0000 7f81e2f68700 1 -- 172.21.15.42:0/2083190573 wait complete. 2019-12-05T06:05:22.590 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:22.585+0000 7f81e2f68700 1 -- 172.21.15.42:0/2083190573 >> 172.21.15.42:0/2083190573 conn(0x7f81dc07e4e0 msgr2=0x7f81dc11c940 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:05:22.600 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:05:24.601 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:05:24.709 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:24.701+0000 7f205dc5d700 1 Processor -- start 2019-12-05T06:05:24.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:24.701+0000 7f205dc5d700 1 -- start start 2019-12-05T06:05:24.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:24.701+0000 7f205dc5d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f20580a7740 0x7f205808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:05:24.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:24.701+0000 7f205dc5d700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f205809fae0 0x7f205807eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:05:24.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:24.701+0000 7f205dc5d700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20580a0710 con 0x7f205807e720 2019-12-05T06:05:24.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:24.701+0000 7f205dc5d700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20580a0b00 con 0x7f205809fae0 2019-12-05T06:05:24.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:24.701+0000 7f205dc5d700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f205808e5b0 con 0x7f20580a7740 2019-12-05T06:05:24.711 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:24.701+0000 7f2057fff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f20580a7740 0x7f205808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:05:24.711 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:24.701+0000 7f2057fff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f20580a7740 0x7f205808df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:47856/0 (socket says 172.21.15.42:47856) 2019-12-05T06:05:24.711 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:24.701+0000 7f2057fff700 1 -- 172.21.15.42:0/1167131633 learned_addr learned my addr 172.21.15.42:0/1167131633 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:05:24.711 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:24.701+0000 7f205c9fb700 1 --2- 172.21.15.42:0/1167131633 >> v2:172.21.15.40:3300/0 conn(0x7f205809fae0 0x7f205807eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:05:24.711 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:24.701+0000 7f2057fff700 1 -- 172.21.15.42:0/1167131633 >> v2:172.21.15.40:3300/0 conn(0x7f205809fae0 msgr2=0x7f205807eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:05:24.711 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:24.701+0000 7f2057fff700 1 --2- 172.21.15.42:0/1167131633 >> v2:172.21.15.40:3300/0 conn(0x7f205809fae0 0x7f205807eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:05:24.711 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:24.701+0000 7f2057fff700 1 -- 172.21.15.42:0/1167131633 >> v1:172.21.15.40:6789/0 conn(0x7f205807e720 legacy=0x7f205809f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:05:24.712 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:24.701+0000 7f2057fff700 1 -- 172.21.15.42:0/1167131633 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f205807f0a0 con 0x7f20580a7740 2019-12-05T06:05:24.712 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:24.701+0000 7f2057fff700 1 --2- 172.21.15.42:0/1167131633 >> v2:172.21.15.42:3300/0 conn(0x7f20580a7740 0x7f205808df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f2048002f30 tx=0x7f204800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:24.712 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:24.701+0000 7f2057fff700 1 --2- 172.21.15.42:0/1167131633 >> v2:172.21.15.42:3300/0 conn(0x7f20580a7740 0x7f205808df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f2048002f30 tx=0x7f204800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:24.712 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:24.701+0000 7f2057fff700 1 --2- 172.21.15.42:0/1167131633 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f20580a7740 0x7f205808df80 secure :-1 s=READY pgs=2006 cs=0 l=1 rx=0x7f2048002f30 tx=0x7f204800fd30).ready entity=mon.1 client_cookie=6f1c268c05de82af server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:05:24.712 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:24.701+0000 7f2057fff700 1 --2- 172.21.15.42:0/1167131633 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f20580a7740 0x7f205808df80 secure :-1 s=THROTTLE_DONE pgs=2006 cs=0 l=1 rx=0x7f2048002f30 tx=0x7f204800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:24.712 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:24.701+0000 7f2057fff700 1 --2- 172.21.15.42:0/1167131633 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f20580a7740 0x7f205808df80 secure :-1 s=THROTTLE_DONE pgs=2006 cs=0 l=1 rx=0x7f2048002f30 tx=0x7f204800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:24.713 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:24.701+0000 7f2057fff700 1 --2- 172.21.15.42:0/1167131633 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f20580a7740 0x7f205808df80 secure :-1 s=THROTTLE_DONE pgs=2006 cs=0 l=1 rx=0x7f2048002f30 tx=0x7f204800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:24.713 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:24.701+0000 7f2056ffd700 1 -- 172.21.15.42:0/1167131633 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2048008c90 con 0x7f20580a7740 2019-12-05T06:05:24.713 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:24.701+0000 7f2056ffd700 1 -- 172.21.15.42:0/1167131633 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20480088b0 con 0x7f20580a7740 2019-12-05T06:05:24.713 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:24.701+0000 7f2056ffd700 1 -- 172.21.15.42:0/1167131633 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2048011ad0 con 0x7f20580a7740 2019-12-05T06:05:24.713 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:24.701+0000 7f205dc5d700 1 -- 172.21.15.42:0/1167131633 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f20580a7740 msgr2=0x7f205808df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:05:24.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:24.701+0000 7f205dc5d700 1 --2- 172.21.15.42:0/1167131633 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f20580a7740 0x7f205808df80 secure :-1 s=READY pgs=2006 cs=0 l=1 rx=0x7f2048002f30 tx=0x7f204800fd30).stop 2019-12-05T06:05:24.715 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:24.705+0000 7f205dc5d700 1 -- 172.21.15.42:0/1167131633 shutdown_connections 2019-12-05T06:05:24.715 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:24.705+0000 7f205dc5d700 1 --2- 172.21.15.42:0/1167131633 >> v2:172.21.15.40:3300/0 conn(0x7f205809fae0 0x7f205807eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:05:24.715 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:24.705+0000 7f205dc5d700 1 --2- 172.21.15.42:0/1167131633 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f20580a7740 0x7f205808df80 unknown :-1 s=CLOSED pgs=2006 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:05:24.715 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:24.705+0000 7f205dc5d700 1 -- 172.21.15.42:0/1167131633 shutdown_connections 2019-12-05T06:05:24.716 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:24.705+0000 7f205dc5d700 1 -- 172.21.15.42:0/1167131633 wait complete. 2019-12-05T06:05:24.716 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:24.705+0000 7f205dc5d700 1 -- 172.21.15.42:0/1167131633 >> 172.21.15.42:0/1167131633 conn(0x7f20580aabe0 msgr2=0x7f20580a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:05:24.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:24.705+0000 7f205dc5d700 1 Processor -- start 2019-12-05T06:05:24.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:24.705+0000 7f205dc5d700 1 -- start start 2019-12-05T06:05:24.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:24.705+0000 7f205dc5d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2058136a00 0x7f2058138e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:05:24.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:24.705+0000 7f205dc5d700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2058131470 con 0x7f2058116500 2019-12-05T06:05:24.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:24.705+0000 7f205dc5d700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f205807dd70 con 0x7f2058119240 2019-12-05T06:05:24.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:24.705+0000 7f205dc5d700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f205807c450 con 0x7f2058136a00 2019-12-05T06:05:24.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:24.705+0000 7f205c9fb700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2058136a00 0x7f2058138e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:05:24.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:24.709+0000 7f205c9fb700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2058136a00 0x7f2058138e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:47864/0 (socket says 172.21.15.42:47864) 2019-12-05T06:05:24.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:24.709+0000 7f205c9fb700 1 -- 172.21.15.42:0/3182767014 learned_addr learned my addr 172.21.15.42:0/3182767014 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:05:24.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:24.709+0000 7f205c9fb700 1 -- 172.21.15.42:0/3182767014 >> v1:172.21.15.42:6789/0 conn(0x7f2058119240 legacy=0x7f2058119670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:05:24.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:24.709+0000 7f205c9fb700 1 -- 172.21.15.42:0/3182767014 >> v1:172.21.15.40:6789/0 conn(0x7f2058116500 legacy=0x7f2058130c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:05:24.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:24.709+0000 7f205c9fb700 1 -- 172.21.15.42:0/3182767014 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f205807bfb0 con 0x7f2058136a00 2019-12-05T06:05:24.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:24.709+0000 7f205c9fb700 1 --2- 172.21.15.42:0/3182767014 >> v2:172.21.15.42:3300/0 conn(0x7f2058136a00 0x7f2058138e40 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f20500032b0 tx=0x7f2050010c80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:24.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:24.709+0000 7f205c9fb700 1 --2- 172.21.15.42:0/3182767014 >> v2:172.21.15.42:3300/0 conn(0x7f2058136a00 0x7f2058138e40 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f20500032b0 tx=0x7f2050010c80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:24.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:24.709+0000 7f205c9fb700 1 --2- 172.21.15.42:0/3182767014 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2058136a00 0x7f2058138e40 secure :-1 s=READY pgs=2008 cs=0 l=1 rx=0x7f20500032b0 tx=0x7f2050010c80).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:05:24.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:24.709+0000 7f205c9fb700 1 --2- 172.21.15.42:0/3182767014 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2058136a00 0x7f2058138e40 secure :-1 s=THROTTLE_DONE pgs=2008 cs=0 l=1 rx=0x7f20500032b0 tx=0x7f2050010c80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:24.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:24.709+0000 7f205c9fb700 1 --2- 172.21.15.42:0/3182767014 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2058136a00 0x7f2058138e40 secure :-1 s=THROTTLE_DONE pgs=2008 cs=0 l=1 rx=0x7f20500032b0 tx=0x7f2050010c80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:24.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:24.709+0000 7f2054ff9700 1 -- 172.21.15.42:0/3182767014 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2050012a50 con 0x7f2058136a00 2019-12-05T06:05:24.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:24.709+0000 7f205c9fb700 1 --2- 172.21.15.42:0/3182767014 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2058136a00 0x7f2058138e40 secure :-1 s=THROTTLE_DONE pgs=2008 cs=0 l=1 rx=0x7f20500032b0 tx=0x7f2050010c80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:24.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:24.709+0000 7f2054ff9700 1 -- 172.21.15.42:0/3182767014 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2050012670 con 0x7f2058136a00 2019-12-05T06:05:24.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:24.709+0000 7f2054ff9700 1 -- 172.21.15.42:0/3182767014 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f205000e760 con 0x7f2058136a00 2019-12-05T06:05:24.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:24.709+0000 7f205dc5d700 1 -- 172.21.15.42:0/3182767014 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f205811a5f0 con 0x7f2058136a00 2019-12-05T06:05:24.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:24.709+0000 7f205dc5d700 1 -- 172.21.15.42:0/3182767014 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2058139390 con 0x7f2058136a00 2019-12-05T06:05:24.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:24.709+0000 7f205c9fb700 1 --2- 172.21.15.42:0/3182767014 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2058136a00 0x7f2058138e40 secure :-1 s=THROTTLE_DONE pgs=2008 cs=0 l=1 rx=0x7f20500032b0 tx=0x7f2050010c80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:24.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:24.709+0000 7f2054ff9700 1 -- 172.21.15.42:0/3182767014 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f2050013060 con 0x7f2058136a00 2019-12-05T06:05:24.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:24.709+0000 7f205c9fb700 1 --2- 172.21.15.42:0/3182767014 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2058136a00 0x7f2058138e40 secure :-1 s=THROTTLE_DONE pgs=2008 cs=0 l=1 rx=0x7f20500032b0 tx=0x7f2050010c80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:24.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:24.709+0000 7f2054ff9700 1 --2- 172.21.15.42:0/3182767014 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f203c03adc0 0x7f203c03d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:05:24.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:24.709+0000 7f2054ff9700 1 -- 172.21.15.42:0/3182767014 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9643+0+0 (secure 0 0 0) 0x7f205004b380 con 0x7f2058136a00 2019-12-05T06:05:24.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:24.709+0000 7f2057fff700 1 --2- 172.21.15.42:0/3182767014 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f203c03adc0 0x7f203c03d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:05:24.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:24.713+0000 7f2057fff700 1 --2- 172.21.15.42:0/3182767014 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f203c03adc0 0x7f203c03d2d0 crc :-1 s=READY pgs=851 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:05:24.724 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:24.717+0000 7f205dc5d700 1 -- 172.21.15.42:0/3182767014 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f205805d110 con 0x7f2058136a00 2019-12-05T06:05:24.730 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:24.725+0000 7f205c9fb700 1 --2- 172.21.15.42:0/3182767014 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2058136a00 0x7f2058138e40 secure :-1 s=THROTTLE_DONE pgs=2008 cs=0 l=1 rx=0x7f20500032b0 tx=0x7f2050010c80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:24.731 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:24.725+0000 7f2054ff9700 1 -- 172.21.15.42:0/3182767014 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f2050032400 con 0x7f2058136a00 2019-12-05T06:05:24.902 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:24.897+0000 7f205dc5d700 1 -- 172.21.15.42:0/3182767014 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f205805b8c0 con 0x7f2058136a00 2019-12-05T06:05:24.903 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:24.897+0000 7f205c9fb700 1 --2- 172.21.15.42:0/3182767014 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2058136a00 0x7f2058138e40 secure :-1 s=THROTTLE_DONE pgs=2008 cs=0 l=1 rx=0x7f20500032b0 tx=0x7f2050010c80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:24.903 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:24.897+0000 7f2054ff9700 1 -- 172.21.15.42:0/3182767014 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 243 v243) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f2050032660 con 0x7f2058136a00 2019-12-05T06:05:24.903 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 243 2019-12-05T06:05:24.903 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:05:24.903 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":243,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6821,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/2253267554","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2253267554},{"type":"v1","addr":"172.21.15.42:6835","nonce":2253267554}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":243}],"filesystems":[{"mdsmap":{"epoch":242,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:04:09.851907+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":91,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9381},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9381":{"gid":9381,"name":"b","rank":0,"incarnation":239,"state":"up:active","state_seq":37,"addr":"172.21.15.40:6833/3789081777","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3789081777},{"type":"v1","addr":"172.21.15.40:6833","nonce":3789081777}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:05:24.906 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:24.901+0000 7f205dc5d700 1 -- 172.21.15.42:0/3182767014 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f203c03adc0 msgr2=0x7f203c03d2d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:05:24.907 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:24.901+0000 7f205dc5d700 1 --2- 172.21.15.42:0/3182767014 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f203c03adc0 0x7f203c03d2d0 crc :-1 s=READY pgs=851 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:05:24.908 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:24.901+0000 7f205dc5d700 1 -- 172.21.15.42:0/3182767014 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2058136a00 msgr2=0x7f2058138e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:05:24.909 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:24.901+0000 7f205dc5d700 1 --2- 172.21.15.42:0/3182767014 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2058136a00 0x7f2058138e40 secure :-1 s=READY pgs=2008 cs=0 l=1 rx=0x7f20500032b0 tx=0x7f2050010c80).stop 2019-12-05T06:05:24.909 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:24.901+0000 7f205dc5d700 1 -- 172.21.15.42:0/3182767014 shutdown_connections 2019-12-05T06:05:24.909 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:24.901+0000 7f205dc5d700 1 --2- 172.21.15.42:0/3182767014 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f203c03adc0 0x7f203c03d2d0 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:05:24.909 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:24.901+0000 7f205dc5d700 1 --2- 172.21.15.42:0/3182767014 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2058136a00 0x7f2058138e40 unknown :-1 s=CLOSED pgs=2008 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:05:24.909 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:24.901+0000 7f205dc5d700 1 -- 172.21.15.42:0/3182767014 shutdown_connections 2019-12-05T06:05:24.909 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:24.901+0000 7f205dc5d700 1 -- 172.21.15.42:0/3182767014 wait complete. 2019-12-05T06:05:24.910 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:24.901+0000 7f205dc5d700 1 -- 172.21.15.42:0/3182767014 >> 172.21.15.42:0/3182767014 conn(0x7f20580a9840 msgr2=0x7f205809f620 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:05:24.916 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:05:26.918 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:05:27.024 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:27.013+0000 7f36c4f6c700 1 Processor -- start 2019-12-05T06:05:27.024 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:27.013+0000 7f36c4f6c700 1 -- start start 2019-12-05T06:05:27.024 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:27.017+0000 7f36c4f6c700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f36c00a7740 0x7f36c008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:05:27.024 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:27.017+0000 7f36c4f6c700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f36c007e720 0x7f36c007eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:05:27.024 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:27.017+0000 7f36c4f6c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f36c009f2e0 0x7f36c009f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:05:27.025 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:27.017+0000 7f36c4f6c700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36c00a0b00 con 0x7f36c007e720 2019-12-05T06:05:27.025 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:27.017+0000 7f36c4f6c700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36c008e5b0 con 0x7f36c009f2e0 2019-12-05T06:05:27.025 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:27.017+0000 7f36c4f6c700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36c008e850 con 0x7f36c00a7740 2019-12-05T06:05:27.025 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:27.017+0000 7f36bf59e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f36c009f2e0 0x7f36c009f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:05:27.025 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:27.017+0000 7f36bf59e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f36c009f2e0 0x7f36c009f710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:47872/0 (socket says 172.21.15.42:47872) 2019-12-05T06:05:27.025 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:27.017+0000 7f36bf59e700 1 -- 172.21.15.42:0/146112803 learned_addr learned my addr 172.21.15.42:0/146112803 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:05:27.026 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:27.017+0000 7f36bed9d700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f36c00a7740 0x7f36c008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:05:27.026 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:27.017+0000 7f36be59c700 1 --2- 172.21.15.42:0/146112803 >> v2:172.21.15.40:3300/0 conn(0x7f36c007e720 0x7f36c007eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:05:27.026 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:27.017+0000 7f36bf59e700 1 -- 172.21.15.42:0/146112803 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f36c00a7740 msgr2=0x7f36c008df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:05:27.028 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:27.017+0000 7f36bf59e700 1 --2- 172.21.15.42:0/146112803 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f36c00a7740 0x7f36c008df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:05:27.029 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:27.017+0000 7f36bf59e700 1 -- 172.21.15.42:0/146112803 >> v2:172.21.15.40:3300/0 conn(0x7f36c007e720 msgr2=0x7f36c007eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:05:27.029 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:27.017+0000 7f36bf59e700 1 --2- 172.21.15.42:0/146112803 >> v2:172.21.15.40:3300/0 conn(0x7f36c007e720 0x7f36c007eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:05:27.029 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:27.017+0000 7f36bf59e700 1 -- 172.21.15.42:0/146112803 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36c007f0a0 con 0x7f36c009f2e0 2019-12-05T06:05:27.029 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:27.017+0000 7f36bf59e700 1 --2- 172.21.15.42:0/146112803 >> v2:172.21.15.42:3300/0 conn(0x7f36c009f2e0 0x7f36c009f710 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f36c008e770 tx=0x7f36b800fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:27.029 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:27.017+0000 7f36bf59e700 1 --2- 172.21.15.42:0/146112803 >> v2:172.21.15.42:3300/0 conn(0x7f36c009f2e0 0x7f36c009f710 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f36c008e770 tx=0x7f36b800fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:27.029 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:27.017+0000 7f36bf59e700 1 --2- 172.21.15.42:0/146112803 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f36c009f2e0 0x7f36c009f710 secure :-1 s=READY pgs=2009 cs=0 l=1 rx=0x7f36c008e770 tx=0x7f36b800fdb0).ready entity=mon.1 client_cookie=2b2d823544c971e8 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:05:27.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:27.017+0000 7f36bf59e700 1 --2- 172.21.15.42:0/146112803 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f36c009f2e0 0x7f36c009f710 secure :-1 s=THROTTLE_DONE pgs=2009 cs=0 l=1 rx=0x7f36c008e770 tx=0x7f36b800fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:27.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:27.017+0000 7f36bdd9b700 1 -- 172.21.15.42:0/146112803 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f36b8008c90 con 0x7f36c009f2e0 2019-12-05T06:05:27.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:27.017+0000 7f36bf59e700 1 --2- 172.21.15.42:0/146112803 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f36c009f2e0 0x7f36c009f710 secure :-1 s=THROTTLE_DONE pgs=2009 cs=0 l=1 rx=0x7f36c008e770 tx=0x7f36b800fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:27.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:27.017+0000 7f36bf59e700 1 --2- 172.21.15.42:0/146112803 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f36c009f2e0 0x7f36c009f710 secure :-1 s=THROTTLE_DONE pgs=2009 cs=0 l=1 rx=0x7f36c008e770 tx=0x7f36b800fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:27.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:27.017+0000 7f36bdd9b700 1 -- 172.21.15.42:0/146112803 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36b80088b0 con 0x7f36c009f2e0 2019-12-05T06:05:27.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:27.017+0000 7f36bdd9b700 1 -- 172.21.15.42:0/146112803 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f36b8011b50 con 0x7f36c009f2e0 2019-12-05T06:05:27.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:27.017+0000 7f36c4f6c700 1 -- 172.21.15.42:0/146112803 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f36c009f2e0 msgr2=0x7f36c009f710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:05:27.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:27.017+0000 7f36c4f6c700 1 --2- 172.21.15.42:0/146112803 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f36c009f2e0 0x7f36c009f710 secure :-1 s=READY pgs=2009 cs=0 l=1 rx=0x7f36c008e770 tx=0x7f36b800fdb0).stop 2019-12-05T06:05:27.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:27.017+0000 7f36c4f6c700 1 -- 172.21.15.42:0/146112803 shutdown_connections 2019-12-05T06:05:27.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:27.017+0000 7f36c4f6c700 1 --2- 172.21.15.42:0/146112803 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f36c009f2e0 0x7f36c009f710 unknown :-1 s=CLOSED pgs=2009 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:05:27.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:27.017+0000 7f36c4f6c700 1 --2- 172.21.15.42:0/146112803 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f36c00a7740 0x7f36c008df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:05:27.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:27.017+0000 7f36c4f6c700 1 --2- 172.21.15.42:0/146112803 >> v2:172.21.15.40:3300/0 conn(0x7f36c007e720 0x7f36c007eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:05:27.032 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:27.017+0000 7f36c4f6c700 1 -- 172.21.15.42:0/146112803 shutdown_connections 2019-12-05T06:05:27.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:27.017+0000 7f36c4f6c700 1 -- 172.21.15.42:0/146112803 wait complete. 2019-12-05T06:05:27.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:27.017+0000 7f36c4f6c700 1 -- 172.21.15.42:0/146112803 >> 172.21.15.42:0/146112803 conn(0x7f36c00aabe0 msgr2=0x7f36c00a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:05:27.039 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:27.021+0000 7f36c4f6c700 1 Processor -- start 2019-12-05T06:05:27.039 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:27.021+0000 7f36c4f6c700 1 -- start start 2019-12-05T06:05:27.039 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:27.021+0000 7f36c4f6c700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f36c0121620 0x7f36c011a600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:05:27.039 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:27.021+0000 7f36c4f6c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f36c0059c10 0x7f36c008f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:05:27.039 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:27.021+0000 7f36c4f6c700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36c011ab50 con 0x7f36c011b130 2019-12-05T06:05:27.040 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:27.021+0000 7f36c4f6c700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36c01208a0 con 0x7f36c0059c10 2019-12-05T06:05:27.040 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:27.021+0000 7f36c4f6c700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36c0121120 con 0x7f36c0121620 2019-12-05T06:05:27.040 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:27.021+0000 7f36bf59e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f36c0059c10 0x7f36c008f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:05:27.040 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:27.021+0000 7f36bf59e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f36c0059c10 0x7f36c008f370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:47876/0 (socket says 172.21.15.42:47876) 2019-12-05T06:05:27.040 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:27.021+0000 7f36bf59e700 1 -- 172.21.15.42:0/67820106 learned_addr learned my addr 172.21.15.42:0/67820106 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:05:27.040 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:27.021+0000 7f36be59c700 1 --2- 172.21.15.42:0/67820106 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f36c0121620 0x7f36c011a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:05:27.040 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:27.025+0000 7f36bf59e700 1 -- 172.21.15.42:0/67820106 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f36c0121620 msgr2=0x7f36c011a600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:05:27.041 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:27.025+0000 7f36bf59e700 1 --2- 172.21.15.42:0/67820106 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f36c0121620 0x7f36c011a600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:05:27.041 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:27.025+0000 7f36bf59e700 1 -- 172.21.15.42:0/67820106 >> v1:172.21.15.42:6789/0 conn(0x7f36c011b130 legacy=0x7f36c008eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:05:27.041 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:27.025+0000 7f36bf59e700 1 -- 172.21.15.42:0/67820106 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36c011f750 con 0x7f36c0059c10 2019-12-05T06:05:27.041 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:27.025+0000 7f36af7fe700 1 -- 172.21.15.42:0/67820106 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 611006382 0 0) 0x7f36c011ab50 con 0x7f36c011b130 2019-12-05T06:05:27.041 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:27.025+0000 7f36bf59e700 1 --2- 172.21.15.42:0/67820106 >> v2:172.21.15.42:3300/0 conn(0x7f36c0059c10 0x7f36c008f370 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f36c008e770 tx=0x7f36b8008b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:27.041 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:27.025+0000 7f36bf59e700 1 --2- 172.21.15.42:0/67820106 >> v2:172.21.15.42:3300/0 conn(0x7f36c0059c10 0x7f36c008f370 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f36c008e770 tx=0x7f36b8008b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:27.042 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:27.025+0000 7f36bf59e700 1 --2- 172.21.15.42:0/67820106 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f36c0059c10 0x7f36c008f370 secure :-1 s=READY pgs=2011 cs=0 l=1 rx=0x7f36c008e770 tx=0x7f36b8008b40).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:05:27.042 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:27.025+0000 7f36bf59e700 1 --2- 172.21.15.42:0/67820106 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f36c0059c10 0x7f36c008f370 secure :-1 s=THROTTLE_DONE pgs=2011 cs=0 l=1 rx=0x7f36c008e770 tx=0x7f36b8008b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:27.042 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:27.025+0000 7f36af7fe700 1 -- 172.21.15.42:0/67820106 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f36b800f470 con 0x7f36c0059c10 2019-12-05T06:05:27.042 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:27.025+0000 7f36bf59e700 1 --2- 172.21.15.42:0/67820106 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f36c0059c10 0x7f36c008f370 secure :-1 s=THROTTLE_DONE pgs=2011 cs=0 l=1 rx=0x7f36c008e770 tx=0x7f36b8008b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:27.043 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:27.025+0000 7f36bf59e700 1 --2- 172.21.15.42:0/67820106 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f36c0059c10 0x7f36c008f370 secure :-1 s=THROTTLE_DONE pgs=2011 cs=0 l=1 rx=0x7f36c008e770 tx=0x7f36b8008b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:27.043 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:27.025+0000 7f36af7fe700 1 -- 172.21.15.42:0/67820106 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36b8011770 con 0x7f36c0059c10 2019-12-05T06:05:27.043 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:27.025+0000 7f36c4f6c700 1 -- 172.21.15.42:0/67820106 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36c008f8c0 con 0x7f36c0059c10 2019-12-05T06:05:27.043 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:27.025+0000 7f36af7fe700 1 -- 172.21.15.42:0/67820106 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f36b8005730 con 0x7f36c0059c10 2019-12-05T06:05:27.043 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:27.025+0000 7f36c4f6c700 1 -- 172.21.15.42:0/67820106 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36c011fa60 con 0x7f36c0059c10 2019-12-05T06:05:27.043 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:27.025+0000 7f36bf59e700 1 --2- 172.21.15.42:0/67820106 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f36c0059c10 0x7f36c008f370 secure :-1 s=THROTTLE_DONE pgs=2011 cs=0 l=1 rx=0x7f36c008e770 tx=0x7f36b8008b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:27.043 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:27.025+0000 7f36af7fe700 1 -- 172.21.15.42:0/67820106 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f36b8005a40 con 0x7f36c0059c10 2019-12-05T06:05:27.044 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:27.025+0000 7f36bf59e700 1 --2- 172.21.15.42:0/67820106 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f36c0059c10 0x7f36c008f370 secure :-1 s=THROTTLE_DONE pgs=2011 cs=0 l=1 rx=0x7f36c008e770 tx=0x7f36b8008b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:27.044 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:27.025+0000 7f36af7fe700 1 --2- 172.21.15.42:0/67820106 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f36a403ad10 0x7f36a403d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:05:27.044 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:27.025+0000 7f36af7fe700 1 -- 172.21.15.42:0/67820106 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9643+0+0 (secure 0 0 0) 0x7f36b808f2f0 con 0x7f36c0059c10 2019-12-05T06:05:27.044 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:27.025+0000 7f36bed9d700 1 --2- 172.21.15.42:0/67820106 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f36a403ad10 0x7f36a403d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:05:27.044 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:27.029+0000 7f36bed9d700 1 --2- 172.21.15.42:0/67820106 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f36a403ad10 0x7f36a403d220 crc :-1 s=READY pgs=852 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:05:27.044 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:27.029+0000 7f36c4f6c700 1 -- 172.21.15.42:0/67820106 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36c005d110 con 0x7f36c0059c10 2019-12-05T06:05:27.045 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:27.037+0000 7f36bf59e700 1 --2- 172.21.15.42:0/67820106 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f36c0059c10 0x7f36c008f370 secure :-1 s=THROTTLE_DONE pgs=2011 cs=0 l=1 rx=0x7f36c008e770 tx=0x7f36b8008b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:27.046 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:27.037+0000 7f36af7fe700 1 -- 172.21.15.42:0/67820106 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f36b8077200 con 0x7f36c0059c10 2019-12-05T06:05:27.209 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:27.205+0000 7f36c4f6c700 1 -- 172.21.15.42:0/67820106 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f36c005b8c0 con 0x7f36c0059c10 2019-12-05T06:05:27.210 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:27.205+0000 7f36bf59e700 1 --2- 172.21.15.42:0/67820106 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f36c0059c10 0x7f36c008f370 secure :-1 s=THROTTLE_DONE pgs=2011 cs=0 l=1 rx=0x7f36c008e770 tx=0x7f36b8008b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:27.211 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:27.205+0000 7f36af7fe700 1 -- 172.21.15.42:0/67820106 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 243 v243) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f36b8010380 con 0x7f36c0059c10 2019-12-05T06:05:27.211 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 243 2019-12-05T06:05:27.211 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:05:27.211 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":243,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6821,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/2253267554","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2253267554},{"type":"v1","addr":"172.21.15.42:6835","nonce":2253267554}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":243}],"filesystems":[{"mdsmap":{"epoch":242,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:04:09.851907+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":91,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9381},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9381":{"gid":9381,"name":"b","rank":0,"incarnation":239,"state":"up:active","state_seq":37,"addr":"172.21.15.40:6833/3789081777","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3789081777},{"type":"v1","addr":"172.21.15.40:6833","nonce":3789081777}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:05:27.214 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:27.209+0000 7f36c4f6c700 1 -- 172.21.15.42:0/67820106 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f36a403ad10 msgr2=0x7f36a403d220 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:05:27.214 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:27.209+0000 7f36c4f6c700 1 --2- 172.21.15.42:0/67820106 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f36a403ad10 0x7f36a403d220 crc :-1 s=READY pgs=852 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:05:27.215 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:27.209+0000 7f36c4f6c700 1 -- 172.21.15.42:0/67820106 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f36c0059c10 msgr2=0x7f36c008f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:05:27.215 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:27.209+0000 7f36c4f6c700 1 --2- 172.21.15.42:0/67820106 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f36c0059c10 0x7f36c008f370 secure :-1 s=READY pgs=2011 cs=0 l=1 rx=0x7f36c008e770 tx=0x7f36b8008b40).stop 2019-12-05T06:05:27.215 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:27.209+0000 7f36c4f6c700 1 -- 172.21.15.42:0/67820106 shutdown_connections 2019-12-05T06:05:27.215 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:27.209+0000 7f36c4f6c700 1 --2- 172.21.15.42:0/67820106 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f36a403ad10 0x7f36a403d220 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:05:27.215 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:27.209+0000 7f36c4f6c700 1 --2- 172.21.15.42:0/67820106 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f36c0059c10 0x7f36c008f370 unknown :-1 s=CLOSED pgs=2011 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:05:27.215 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:27.209+0000 7f36c4f6c700 1 --2- 172.21.15.42:0/67820106 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f36c0121620 0x7f36c011a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:05:27.215 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:27.209+0000 7f36c4f6c700 1 -- 172.21.15.42:0/67820106 shutdown_connections 2019-12-05T06:05:27.216 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:27.209+0000 7f36c4f6c700 1 -- 172.21.15.42:0/67820106 wait complete. 2019-12-05T06:05:27.216 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:27.209+0000 7f36c4f6c700 1 -- 172.21.15.42:0/67820106 >> 172.21.15.42:0/67820106 conn(0x7f36c007e770 msgr2=0x7f36c009f380 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:05:27.230 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:05:29.231 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:05:29.335 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.325+0000 7f6538cee700 1 Processor -- start 2019-12-05T06:05:29.335 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.325+0000 7f6538cee700 1 -- start start 2019-12-05T06:05:29.336 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.329+0000 7f6538cee700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f65340a7740 0x7f653408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:05:29.336 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.329+0000 7f6538cee700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f653407e720 0x7f653407eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:05:29.336 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.329+0000 7f6538cee700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65340a0710 con 0x7f653409f2e0 2019-12-05T06:05:29.336 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.329+0000 7f6538cee700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65340a0b00 con 0x7f65340a7740 2019-12-05T06:05:29.336 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.329+0000 7f6538cee700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f653408e5b0 con 0x7f653407e720 2019-12-05T06:05:29.336 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.329+0000 7f6532d9d700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f65340a7740 0x7f653408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:05:29.336 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.329+0000 7f6532d9d700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f65340a7740 0x7f653408df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.42:35698/0 (socket says 172.21.15.42:35698) 2019-12-05T06:05:29.337 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.329+0000 7f6532d9d700 1 -- 172.21.15.42:0/198162224 learned_addr learned my addr 172.21.15.42:0/198162224 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:05:29.337 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.329+0000 7f653359e700 1 --1- 172.21.15.42:0/198162224 >> v1:172.21.15.40:6789/0 conn(0x7f653409f2e0 0x7f653409f710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:57364/0 (socket says 172.21.15.42:57364) 2019-12-05T06:05:29.337 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.329+0000 7f653259c700 1 --2- 172.21.15.42:0/198162224 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f653407e720 0x7f653407eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:05:29.337 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.329+0000 7f6532d9d700 1 -- 172.21.15.42:0/198162224 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f653407e720 msgr2=0x7f653407eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:05:29.337 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.329+0000 7f6532d9d700 1 --2- 172.21.15.42:0/198162224 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f653407e720 0x7f653407eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:05:29.337 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.329+0000 7f6532d9d700 1 -- 172.21.15.42:0/198162224 >> v1:172.21.15.40:6789/0 conn(0x7f653409f2e0 legacy=0x7f653409f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:05:29.337 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.329+0000 7f6532d9d700 1 -- 172.21.15.42:0/198162224 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f653407f0a0 con 0x7f65340a7740 2019-12-05T06:05:29.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.329+0000 7f6531d9b700 1 -- 172.21.15.42:0/198162224 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 503026653 0 0) 0x7f65340a0710 con 0x7f653409f2e0 2019-12-05T06:05:29.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.329+0000 7f6532d9d700 1 --2- 172.21.15.42:0/198162224 >> v2:172.21.15.40:3300/0 conn(0x7f65340a7740 0x7f653408df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f6524008b10 tx=0x7f65240100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:29.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.329+0000 7f6532d9d700 1 --2- 172.21.15.42:0/198162224 >> v2:172.21.15.40:3300/0 conn(0x7f65340a7740 0x7f653408df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f6524008b10 tx=0x7f65240100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:29.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.329+0000 7f6532d9d700 1 --2- 172.21.15.42:0/198162224 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f65340a7740 0x7f653408df80 secure :-1 s=READY pgs=1522 cs=0 l=1 rx=0x7f6524008b10 tx=0x7f65240100b0).ready entity=mon.0 client_cookie=b5ce54e6a31945ce server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:05:29.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.329+0000 7f6532d9d700 1 --2- 172.21.15.42:0/198162224 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f65340a7740 0x7f653408df80 secure :-1 s=THROTTLE_DONE pgs=1522 cs=0 l=1 rx=0x7f6524008b10 tx=0x7f65240100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:29.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.329+0000 7f6531d9b700 1 -- 172.21.15.42:0/198162224 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f652400a160 con 0x7f65340a7740 2019-12-05T06:05:29.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.329+0000 7f6532d9d700 1 --2- 172.21.15.42:0/198162224 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f65340a7740 0x7f653408df80 secure :-1 s=THROTTLE_DONE pgs=1522 cs=0 l=1 rx=0x7f6524008b10 tx=0x7f65240100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:29.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.329+0000 7f6532d9d700 1 --2- 172.21.15.42:0/198162224 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f65340a7740 0x7f653408df80 secure :-1 s=THROTTLE_DONE pgs=1522 cs=0 l=1 rx=0x7f6524008b10 tx=0x7f65240100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:29.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.329+0000 7f6531d9b700 1 -- 172.21.15.42:0/198162224 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6524011a30 con 0x7f65340a7740 2019-12-05T06:05:29.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.329+0000 7f6531d9b700 1 -- 172.21.15.42:0/198162224 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f652400a550 con 0x7f65340a7740 2019-12-05T06:05:29.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.329+0000 7f6538cee700 1 -- 172.21.15.42:0/198162224 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f65340a7740 msgr2=0x7f653408df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:05:29.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.329+0000 7f6538cee700 1 --2- 172.21.15.42:0/198162224 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f65340a7740 0x7f653408df80 secure :-1 s=READY pgs=1522 cs=0 l=1 rx=0x7f6524008b10 tx=0x7f65240100b0).stop 2019-12-05T06:05:29.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.329+0000 7f6538cee700 1 -- 172.21.15.42:0/198162224 shutdown_connections 2019-12-05T06:05:29.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.329+0000 7f6538cee700 1 --2- 172.21.15.42:0/198162224 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f65340a7740 0x7f653408df80 unknown :-1 s=CLOSED pgs=1522 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:05:29.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.329+0000 7f6538cee700 1 --2- 172.21.15.42:0/198162224 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f653407e720 0x7f653407eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:05:29.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.329+0000 7f6538cee700 1 -- 172.21.15.42:0/198162224 shutdown_connections 2019-12-05T06:05:29.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.329+0000 7f6538cee700 1 -- 172.21.15.42:0/198162224 wait complete. 2019-12-05T06:05:29.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.329+0000 7f6538cee700 1 -- 172.21.15.42:0/198162224 >> 172.21.15.42:0/198162224 conn(0x7f65340aabe0 msgr2=0x7f65340a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:05:29.344 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.333+0000 7f6538cee700 1 Processor -- start 2019-12-05T06:05:29.345 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.333+0000 7f6538cee700 1 -- start start 2019-12-05T06:05:29.345 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.333+0000 7f6538cee700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6534133420 0x7f65341161c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:05:29.345 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.333+0000 7f6538cee700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f653411b520 0x7f653411ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:05:29.345 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.333+0000 7f6538cee700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f653411e340 0x7f653411fa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:05:29.345 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.333+0000 7f6538cee700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f653411afe0 con 0x7f653411b520 2019-12-05T06:05:29.345 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.333+0000 7f6538cee700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f653411b280 con 0x7f6534133420 2019-12-05T06:05:29.346 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.333+0000 7f6538cee700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6534116710 con 0x7f653411e340 2019-12-05T06:05:29.346 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.333+0000 7f6532d9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6534133420 0x7f65341161c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:05:29.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.333+0000 7f6532d9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6534133420 0x7f65341161c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:47888/0 (socket says 172.21.15.42:47888) 2019-12-05T06:05:29.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.333+0000 7f6532d9d700 1 -- 172.21.15.42:0/3380991777 learned_addr learned my addr 172.21.15.42:0/3380991777 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:05:29.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.333+0000 7f653359e700 1 --2- 172.21.15.42:0/3380991777 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f653411e340 0x7f653411fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:05:29.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.333+0000 7f653259c700 1 --2- 172.21.15.42:0/3380991777 >> v2:172.21.15.40:3300/0 conn(0x7f653411b520 0x7f653411ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:05:29.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.333+0000 7f6532d9d700 1 -- 172.21.15.42:0/3380991777 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f653411e340 msgr2=0x7f653411fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:05:29.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.333+0000 7f6532d9d700 1 --2- 172.21.15.42:0/3380991777 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f653411e340 0x7f653411fa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:05:29.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.333+0000 7f6532d9d700 1 -- 172.21.15.42:0/3380991777 >> v2:172.21.15.40:3300/0 conn(0x7f653411b520 msgr2=0x7f653411ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:05:29.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.333+0000 7f6532d9d700 1 --2- 172.21.15.42:0/3380991777 >> v2:172.21.15.40:3300/0 conn(0x7f653411b520 0x7f653411ddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:05:29.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.333+0000 7f6532d9d700 1 -- 172.21.15.42:0/3380991777 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f653411a720 con 0x7f6534133420 2019-12-05T06:05:29.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.333+0000 7f6532d9d700 1 --2- 172.21.15.42:0/3380991777 >> v2:172.21.15.42:3300/0 conn(0x7f6534133420 0x7f65341161c0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f6524008b10 tx=0x7f652400ab60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:29.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.333+0000 7f6532d9d700 1 --2- 172.21.15.42:0/3380991777 >> v2:172.21.15.42:3300/0 conn(0x7f6534133420 0x7f65341161c0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f6524008b10 tx=0x7f652400ab60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:29.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.333+0000 7f6532d9d700 1 --2- 172.21.15.42:0/3380991777 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6534133420 0x7f65341161c0 secure :-1 s=READY pgs=2012 cs=0 l=1 rx=0x7f6524008b10 tx=0x7f652400ab60).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:05:29.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.337+0000 7f6532d9d700 1 --2- 172.21.15.42:0/3380991777 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6534133420 0x7f65341161c0 secure :-1 s=THROTTLE_DONE pgs=2012 cs=0 l=1 rx=0x7f6524008b10 tx=0x7f652400ab60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:29.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.337+0000 7f6532d9d700 1 --2- 172.21.15.42:0/3380991777 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6534133420 0x7f65341161c0 secure :-1 s=THROTTLE_DONE pgs=2012 cs=0 l=1 rx=0x7f6524008b10 tx=0x7f652400ab60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:29.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.337+0000 7f65237fe700 1 -- 172.21.15.42:0/3380991777 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6524005ab0 con 0x7f6534133420 2019-12-05T06:05:29.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.337+0000 7f6532d9d700 1 --2- 172.21.15.42:0/3380991777 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6534133420 0x7f65341161c0 secure :-1 s=THROTTLE_DONE pgs=2012 cs=0 l=1 rx=0x7f6524008b10 tx=0x7f652400ab60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:29.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.337+0000 7f65237fe700 1 -- 172.21.15.42:0/3380991777 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6524011e10 con 0x7f6534133420 2019-12-05T06:05:29.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.337+0000 7f65237fe700 1 -- 172.21.15.42:0/3380991777 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6524007340 con 0x7f6534133420 2019-12-05T06:05:29.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.337+0000 7f6538cee700 1 -- 172.21.15.42:0/3380991777 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f653411b950 con 0x7f6534133420 2019-12-05T06:05:29.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.337+0000 7f6538cee700 1 -- 172.21.15.42:0/3380991777 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f653411ffe0 con 0x7f6534133420 2019-12-05T06:05:29.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.337+0000 7f6532d9d700 1 --2- 172.21.15.42:0/3380991777 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6534133420 0x7f65341161c0 secure :-1 s=THROTTLE_DONE pgs=2012 cs=0 l=1 rx=0x7f6524008b10 tx=0x7f652400ab60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:29.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.337+0000 7f65237fe700 1 -- 172.21.15.42:0/3380991777 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f6524007650 con 0x7f6534133420 2019-12-05T06:05:29.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.337+0000 7f6532d9d700 1 --2- 172.21.15.42:0/3380991777 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6534133420 0x7f65341161c0 secure :-1 s=THROTTLE_DONE pgs=2012 cs=0 l=1 rx=0x7f6524008b10 tx=0x7f652400ab60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:29.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.337+0000 7f65237fe700 1 --2- 172.21.15.42:0/3380991777 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f651803aef0 0x7f651803d400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:05:29.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.337+0000 7f65237fe700 1 -- 172.21.15.42:0/3380991777 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9643+0+0 (secure 0 0 0) 0x7f652408fd30 con 0x7f6534133420 2019-12-05T06:05:29.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.337+0000 7f653259c700 1 --2- 172.21.15.42:0/3380991777 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f651803aef0 0x7f651803d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:05:29.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.337+0000 7f653259c700 1 --2- 172.21.15.42:0/3380991777 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f651803aef0 0x7f651803d400 crc :-1 s=READY pgs=853 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:05:29.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.341+0000 7f6538cee700 1 -- 172.21.15.42:0/3380991777 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f653405d110 con 0x7f6534133420 2019-12-05T06:05:29.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.349+0000 7f6532d9d700 1 --2- 172.21.15.42:0/3380991777 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6534133420 0x7f65341161c0 secure :-1 s=THROTTLE_DONE pgs=2012 cs=0 l=1 rx=0x7f6524008b10 tx=0x7f652400ab60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:29.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.349+0000 7f65237fe700 1 -- 172.21.15.42:0/3380991777 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f652400ade0 con 0x7f6534133420 2019-12-05T06:05:29.550 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.545+0000 7f6538cee700 1 -- 172.21.15.42:0/3380991777 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f653405b8c0 con 0x7f6534133420 2019-12-05T06:05:29.550 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.545+0000 7f6532d9d700 1 --2- 172.21.15.42:0/3380991777 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6534133420 0x7f65341161c0 secure :-1 s=THROTTLE_DONE pgs=2012 cs=0 l=1 rx=0x7f6524008b10 tx=0x7f652400ab60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:29.550 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.545+0000 7f65237fe700 1 -- 172.21.15.42:0/3380991777 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 244 v244) v1 ==== 95+0+1774 (secure 0 0 0) 0x7f6524077010 con 0x7f6534133420 2019-12-05T06:05:29.551 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 244 2019-12-05T06:05:29.552 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:05:29.552 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":244,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":244,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:05:29.443166+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":92,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6821},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6821":{"gid":6821,"name":"a","rank":0,"incarnation":244,"state":"up:replay","state_seq":2,"addr":"172.21.15.42:6835/2253267554","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2253267554},{"type":"v1","addr":"172.21.15.42:6835","nonce":2253267554}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:05:29.553 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.549+0000 7f6538cee700 1 -- 172.21.15.42:0/3380991777 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f651803aef0 msgr2=0x7f651803d400 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:05:29.553 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.549+0000 7f6538cee700 1 --2- 172.21.15.42:0/3380991777 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f651803aef0 0x7f651803d400 crc :-1 s=READY pgs=853 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:05:29.553 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.549+0000 7f6538cee700 1 -- 172.21.15.42:0/3380991777 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6534133420 msgr2=0x7f65341161c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:05:29.554 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.549+0000 7f6538cee700 1 --2- 172.21.15.42:0/3380991777 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6534133420 0x7f65341161c0 secure :-1 s=READY pgs=2012 cs=0 l=1 rx=0x7f6524008b10 tx=0x7f652400ab60).stop 2019-12-05T06:05:29.554 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.549+0000 7f6538cee700 1 -- 172.21.15.42:0/3380991777 shutdown_connections 2019-12-05T06:05:29.554 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.549+0000 7f6538cee700 1 --2- 172.21.15.42:0/3380991777 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f651803aef0 0x7f651803d400 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:05:29.554 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.549+0000 7f6538cee700 1 --2- 172.21.15.42:0/3380991777 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f653411e340 0x7f653411fa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:05:29.554 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.549+0000 7f6538cee700 1 --2- 172.21.15.42:0/3380991777 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6534133420 0x7f65341161c0 unknown :-1 s=CLOSED pgs=2012 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:05:29.554 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.549+0000 7f6538cee700 1 --2- 172.21.15.42:0/3380991777 >> v2:172.21.15.40:3300/0 conn(0x7f653411b520 0x7f653411ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:05:29.555 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.549+0000 7f6538cee700 1 -- 172.21.15.42:0/3380991777 shutdown_connections 2019-12-05T06:05:29.555 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.549+0000 7f6538cee700 1 -- 172.21.15.42:0/3380991777 wait complete. 2019-12-05T06:05:29.555 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.549+0000 7f6538cee700 1 -- 172.21.15.42:0/3380991777 >> 172.21.15.42:0/3380991777 conn(0x7f65340a7740 msgr2=0x7f653409f620 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:05:29.566 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b down, removed from mdsmap 2019-12-05T06:05:29.566 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2019-12-05T06:05:29.566 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:05:29.657 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.649+0000 7fb378afd700 1 Processor -- start 2019-12-05T06:05:29.657 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.649+0000 7fb378afd700 1 -- start start 2019-12-05T06:05:29.657 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.649+0000 7fb378afd700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb37409fae0 0x7fb37414fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:05:29.657 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.649+0000 7fb378afd700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3740a0710 con 0x7fb37408df80 2019-12-05T06:05:29.657 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.649+0000 7fb378afd700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb37407ef20 con 0x7fb3740a7740 2019-12-05T06:05:29.658 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.649+0000 7fb378afd700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3740a0b00 con 0x7fb37409fae0 2019-12-05T06:05:29.658 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.649+0000 7fb372d9d700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fb3740a7740 0x7fb37407e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:47950/0 (socket says 172.21.15.42:47950) 2019-12-05T06:05:29.658 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.649+0000 7fb372d9d700 1 -- 172.21.15.42:0/47593546 learned_addr learned my addr 172.21.15.42:0/47593546 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:05:29.659 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.649+0000 7fb37359e700 1 --2- 172.21.15.42:0/47593546 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb37409fae0 0x7fb37414fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:05:29.659 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.649+0000 7fb371d9b700 1 -- 172.21.15.42:0/47593546 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1138492724 0 0) 0x7fb37407ef20 con 0x7fb3740a7740 2019-12-05T06:05:29.659 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.649+0000 7fb371d9b700 1 -- 172.21.15.42:0/47593546 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3600030b0 con 0x7fb3740a7740 2019-12-05T06:05:29.659 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.649+0000 7fb371d9b700 1 -- 172.21.15.42:0/47593546 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 9093371 0 0) 0x7fb3600030b0 con 0x7fb3740a7740 2019-12-05T06:05:29.659 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.649+0000 7fb371d9b700 1 -- 172.21.15.42:0/47593546 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb37409fae0 msgr2=0x7fb37414fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:05:29.659 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.649+0000 7fb371d9b700 1 --2- 172.21.15.42:0/47593546 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb37409fae0 0x7fb37414fff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:05:29.659 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.649+0000 7fb371d9b700 1 -- 172.21.15.42:0/47593546 >> v1:172.21.15.40:6789/0 conn(0x7fb37408df80 legacy=0x7fb37409f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:05:29.660 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.649+0000 7fb371d9b700 1 -- 172.21.15.42:0/47593546 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3740a9ba0 con 0x7fb3740a7740 2019-12-05T06:05:29.660 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.649+0000 7fb371d9b700 1 -- 172.21.15.42:0/47593546 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7fb36c001ad0 con 0x7fb3740a7740 2019-12-05T06:05:29.660 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.649+0000 7fb371d9b700 1 -- 172.21.15.42:0/47593546 >> v1:172.21.15.42:6789/0 conn(0x7fb3740a7740 legacy=0x7fb37407e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:05:29.660 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.649+0000 7fb371d9b700 1 --2- 172.21.15.42:0/47593546 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb360003710 0x7fb360003b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:05:29.660 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.649+0000 7fb371d9b700 1 --2- 172.21.15.42:0/47593546 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb360004420 0x7fb360004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:05:29.660 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.649+0000 7fb371d9b700 1 --2- 172.21.15.42:0/47593546 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb360004da0 0x7fb36000b670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:05:29.661 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.649+0000 7fb371d9b700 1 -- 172.21.15.42:0/47593546 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3740a0b00 con 0x7fb360003710 2019-12-05T06:05:29.661 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.649+0000 7fb371d9b700 1 -- 172.21.15.42:0/47593546 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb360005200 con 0x7fb360004da0 2019-12-05T06:05:29.661 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.649+0000 7fb371d9b700 1 -- 172.21.15.42:0/47593546 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb36004fc00 con 0x7fb360004420 2019-12-05T06:05:29.661 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.649+0000 7fb37359e700 1 --2- 172.21.15.42:0/47593546 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb360004da0 0x7fb36000b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:05:29.661 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.649+0000 7fb37259c700 1 --2- 172.21.15.42:0/47593546 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb360004420 0x7fb360004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:05:29.661 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.649+0000 7fb372d9d700 1 --2- 172.21.15.42:0/47593546 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb360003710 0x7fb360003b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:05:29.661 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.649+0000 7fb37259c700 1 -- 172.21.15.42:0/47593546 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb360004da0 msgr2=0x7fb36000b670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:05:29.661 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.649+0000 7fb37259c700 1 --2- 172.21.15.42:0/47593546 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb360004da0 0x7fb36000b670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:05:29.662 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.649+0000 7fb37259c700 1 -- 172.21.15.42:0/47593546 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb360003710 msgr2=0x7fb360003b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:05:29.662 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.649+0000 7fb37259c700 1 --2- 172.21.15.42:0/47593546 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb360003710 0x7fb360003b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:05:29.662 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.649+0000 7fb37259c700 1 -- 172.21.15.42:0/47593546 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb36004ff00 con 0x7fb360004420 2019-12-05T06:05:29.662 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.649+0000 7fb37359e700 1 --2- 172.21.15.42:0/47593546 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb360004da0 0x7fb36000b670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:05:29.662 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.649+0000 7fb37259c700 1 --2- 172.21.15.42:0/47593546 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb360004420 0x7fb360004850 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fb35c001000 tx=0x7fb35c00fcd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:29.662 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.649+0000 7fb37259c700 1 --2- 172.21.15.42:0/47593546 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb360004420 0x7fb360004850 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fb35c001000 tx=0x7fb35c00fcd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:29.662 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.649+0000 7fb37259c700 1 --2- 172.21.15.42:0/47593546 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb360004420 0x7fb360004850 secure :-1 s=READY pgs=449 cs=0 l=1 rx=0x7fb35c001000 tx=0x7fb35c00fcd0).ready entity=mon.2 client_cookie=fba30c452c6f816 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:05:29.662 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.649+0000 7fb37259c700 1 --2- 172.21.15.42:0/47593546 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb360004420 0x7fb360004850 secure :-1 s=THROTTLE_DONE pgs=449 cs=0 l=1 rx=0x7fb35c001000 tx=0x7fb35c00fcd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:29.663 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.649+0000 7fb371d9b700 1 -- 172.21.15.42:0/47593546 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb35c003540 con 0x7fb360004420 2019-12-05T06:05:29.663 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.649+0000 7fb37259c700 1 --2- 172.21.15.42:0/47593546 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb360004420 0x7fb360004850 secure :-1 s=THROTTLE_DONE pgs=449 cs=0 l=1 rx=0x7fb35c001000 tx=0x7fb35c00fcd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:29.663 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.649+0000 7fb371d9b700 1 -- 172.21.15.42:0/47593546 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb35c003160 con 0x7fb360004420 2019-12-05T06:05:29.663 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.649+0000 7fb37259c700 1 --2- 172.21.15.42:0/47593546 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb360004420 0x7fb360004850 secure :-1 s=THROTTLE_DONE pgs=449 cs=0 l=1 rx=0x7fb35c001000 tx=0x7fb35c00fcd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:29.663 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.649+0000 7fb371d9b700 1 -- 172.21.15.42:0/47593546 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb35c003b20 con 0x7fb360004420 2019-12-05T06:05:29.663 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.649+0000 7fb378afd700 1 -- 172.21.15.42:0/47593546 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb360004420 msgr2=0x7fb360004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:05:29.663 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.649+0000 7fb378afd700 1 --2- 172.21.15.42:0/47593546 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb360004420 0x7fb360004850 secure :-1 s=READY pgs=449 cs=0 l=1 rx=0x7fb35c001000 tx=0x7fb35c00fcd0).stop 2019-12-05T06:05:29.664 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.649+0000 7fb37359e700 1 -- 172.21.15.42:0/47593546 reap_dead start 2019-12-05T06:05:29.664 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.653+0000 7fb378afd700 1 -- 172.21.15.42:0/47593546 shutdown_connections 2019-12-05T06:05:29.664 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.653+0000 7fb378afd700 1 -- 172.21.15.42:0/47593546 shutdown_connections 2019-12-05T06:05:29.664 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.653+0000 7fb378afd700 1 -- 172.21.15.42:0/47593546 wait complete. 2019-12-05T06:05:29.664 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.653+0000 7fb378afd700 1 -- 172.21.15.42:0/47593546 >> 172.21.15.42:0/47593546 conn(0x7fb3740aabe0 msgr2=0x7fb3740a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:05:29.664 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.653+0000 7fb378afd700 1 Processor -- start 2019-12-05T06:05:29.664 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.653+0000 7fb378afd700 1 -- start start 2019-12-05T06:05:29.667 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.653+0000 7fb378afd700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb374116500 0x7fb3740a1ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:05:29.668 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.653+0000 7fb378afd700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb37411ad30 0x7fb37411b160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:05:29.668 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.653+0000 7fb378afd700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb37408eb70 0x7fb37408efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:05:29.668 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.653+0000 7fb378afd700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb37411c440 con 0x7fb37408eb70 2019-12-05T06:05:29.668 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.653+0000 7fb378afd700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3741196a0 con 0x7fb374116500 2019-12-05T06:05:29.668 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.653+0000 7fb378afd700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb374119940 con 0x7fb37411ad30 2019-12-05T06:05:29.669 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.653+0000 7fb372d9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb374116500 0x7fb3740a1ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:05:29.669 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.653+0000 7fb372d9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb374116500 0x7fb3740a1ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:47924/0 (socket says 172.21.15.42:47924) 2019-12-05T06:05:29.669 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.653+0000 7fb37259c700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb37411ad30 0x7fb37411b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:05:29.669 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.653+0000 7fb372d9d700 1 -- 172.21.15.42:0/318951777 learned_addr learned my addr 172.21.15.42:0/318951777 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:05:29.669 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.653+0000 7fb37359e700 1 --2- 172.21.15.42:0/318951777 >> v2:172.21.15.40:3300/0 conn(0x7fb37408eb70 0x7fb37408efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:05:29.670 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.653+0000 7fb372d9d700 1 -- 172.21.15.42:0/318951777 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb37411ad30 msgr2=0x7fb37411b160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:05:29.670 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.653+0000 7fb372d9d700 1 --2- 172.21.15.42:0/318951777 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb37411ad30 0x7fb37411b160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:05:29.670 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.653+0000 7fb372d9d700 1 -- 172.21.15.42:0/318951777 >> v2:172.21.15.40:3300/0 conn(0x7fb37408eb70 msgr2=0x7fb37408efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:05:29.670 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.653+0000 7fb372d9d700 1 --2- 172.21.15.42:0/318951777 >> v2:172.21.15.40:3300/0 conn(0x7fb37408eb70 0x7fb37408efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:05:29.670 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.653+0000 7fb372d9d700 1 -- 172.21.15.42:0/318951777 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb374148880 con 0x7fb374116500 2019-12-05T06:05:29.670 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.653+0000 7fb37359e700 1 --2- 172.21.15.42:0/318951777 >> v2:172.21.15.40:3300/0 conn(0x7fb37408eb70 0x7fb37408efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:05:29.670 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.653+0000 7fb37259c700 1 --2- 172.21.15.42:0/318951777 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb37411ad30 0x7fb37411b160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:05:29.671 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.653+0000 7fb372d9d700 1 --2- 172.21.15.42:0/318951777 >> v2:172.21.15.42:3300/0 conn(0x7fb374116500 0x7fb3740a1ac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fb374119860 tx=0x7fb36c005670).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:29.671 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.653+0000 7fb372d9d700 1 --2- 172.21.15.42:0/318951777 >> v2:172.21.15.42:3300/0 conn(0x7fb374116500 0x7fb3740a1ac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fb374119860 tx=0x7fb36c005670).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:29.671 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.653+0000 7fb372d9d700 1 --2- 172.21.15.42:0/318951777 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb374116500 0x7fb3740a1ac0 secure :-1 s=READY pgs=2014 cs=0 l=1 rx=0x7fb374119860 tx=0x7fb36c005670).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:05:29.671 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.653+0000 7fb372d9d700 1 --2- 172.21.15.42:0/318951777 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb374116500 0x7fb3740a1ac0 secure :-1 s=THROTTLE_DONE pgs=2014 cs=0 l=1 rx=0x7fb374119860 tx=0x7fb36c005670).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:29.671 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.653+0000 7fb36b7fe700 1 -- 172.21.15.42:0/318951777 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb36c011d20 con 0x7fb374116500 2019-12-05T06:05:29.671 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.653+0000 7fb372d9d700 1 --2- 172.21.15.42:0/318951777 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb374116500 0x7fb3740a1ac0 secure :-1 s=THROTTLE_DONE pgs=2014 cs=0 l=1 rx=0x7fb374119860 tx=0x7fb36c005670).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:29.671 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.653+0000 7fb372d9d700 1 --2- 172.21.15.42:0/318951777 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb374116500 0x7fb3740a1ac0 secure :-1 s=THROTTLE_DONE pgs=2014 cs=0 l=1 rx=0x7fb374119860 tx=0x7fb36c005670).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:29.672 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.653+0000 7fb36b7fe700 1 -- 172.21.15.42:0/318951777 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb36c011940 con 0x7fb374116500 2019-12-05T06:05:29.672 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.653+0000 7fb378afd700 1 -- 172.21.15.42:0/318951777 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb374119e80 con 0x7fb374116500 2019-12-05T06:05:29.672 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.653+0000 7fb36b7fe700 1 -- 172.21.15.42:0/318951777 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb36c008cd0 con 0x7fb374116500 2019-12-05T06:05:29.672 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.653+0000 7fb378afd700 1 -- 172.21.15.42:0/318951777 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb37408f4f0 con 0x7fb374116500 2019-12-05T06:05:29.672 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.657+0000 7fb372d9d700 1 --2- 172.21.15.42:0/318951777 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb374116500 0x7fb3740a1ac0 secure :-1 s=THROTTLE_DONE pgs=2014 cs=0 l=1 rx=0x7fb374119860 tx=0x7fb36c005670).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:29.672 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.657+0000 7fb36b7fe700 1 -- 172.21.15.42:0/318951777 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fb36c00b250 con 0x7fb374116500 2019-12-05T06:05:29.673 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.657+0000 7fb372d9d700 1 --2- 172.21.15.42:0/318951777 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb374116500 0x7fb3740a1ac0 secure :-1 s=THROTTLE_DONE pgs=2014 cs=0 l=1 rx=0x7fb374119860 tx=0x7fb36c005670).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:29.673 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.657+0000 7fb36b7fe700 1 --2- 172.21.15.42:0/318951777 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb35803ad20 0x7fb35803d230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:05:29.673 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.657+0000 7fb36b7fe700 1 -- 172.21.15.42:0/318951777 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9729+0+0 (secure 0 0 0) 0x7fb36c04b2b0 con 0x7fb374116500 2019-12-05T06:05:29.673 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.657+0000 7fb37259c700 1 --2- 172.21.15.42:0/318951777 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb35803ad20 0x7fb35803d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:05:29.673 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.657+0000 7fb37259c700 1 --2- 172.21.15.42:0/318951777 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb35803ad20 0x7fb35803d230 crc :-1 s=READY pgs=854 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:05:29.673 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.661+0000 7fb378afd700 1 -- 172.21.15.42:0/318951777 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb37405d110 con 0x7fb374116500 2019-12-05T06:05:29.674 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.665+0000 7fb372d9d700 1 --2- 172.21.15.42:0/318951777 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb374116500 0x7fb3740a1ac0 secure :-1 s=THROTTLE_DONE pgs=2014 cs=0 l=1 rx=0x7fb374119860 tx=0x7fb36c005670).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:29.674 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.665+0000 7fb36b7fe700 1 -- 172.21.15.42:0/318951777 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fb36c001540 con 0x7fb374116500 2019-12-05T06:05:29.833 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.829+0000 7fb378afd700 1 -- 172.21.15.42:0/318951777 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb37405b8c0 con 0x7fb374116500 2019-12-05T06:05:29.833 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.829+0000 7fb372d9d700 1 --2- 172.21.15.42:0/318951777 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb374116500 0x7fb3740a1ac0 secure :-1 s=THROTTLE_DONE pgs=2014 cs=0 l=1 rx=0x7fb374119860 tx=0x7fb36c005670).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:29.834 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.829+0000 7fb36b7fe700 1 -- 172.21.15.42:0/318951777 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 244 v244) v1 ==== 95+0+1774 (secure 0 0 0) 0x7fb36c032570 con 0x7fb374116500 2019-12-05T06:05:29.835 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 244 2019-12-05T06:05:29.835 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:05:29.835 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":244,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":244,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:05:29.443166+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":92,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6821},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6821":{"gid":6821,"name":"a","rank":0,"incarnation":244,"state":"up:replay","state_seq":2,"addr":"172.21.15.42:6835/2253267554","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2253267554},{"type":"v1","addr":"172.21.15.42:6835","nonce":2253267554}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:05:29.837 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.833+0000 7fb378afd700 1 -- 172.21.15.42:0/318951777 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb35803ad20 msgr2=0x7fb35803d230 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:05:29.837 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.833+0000 7fb378afd700 1 --2- 172.21.15.42:0/318951777 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb35803ad20 0x7fb35803d230 crc :-1 s=READY pgs=854 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:05:29.837 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.833+0000 7fb378afd700 1 -- 172.21.15.42:0/318951777 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb374116500 msgr2=0x7fb3740a1ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:05:29.837 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.833+0000 7fb378afd700 1 --2- 172.21.15.42:0/318951777 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb374116500 0x7fb3740a1ac0 secure :-1 s=READY pgs=2014 cs=0 l=1 rx=0x7fb374119860 tx=0x7fb36c005670).stop 2019-12-05T06:05:29.837 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.833+0000 7fb378afd700 1 -- 172.21.15.42:0/318951777 shutdown_connections 2019-12-05T06:05:29.837 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.833+0000 7fb378afd700 1 --2- 172.21.15.42:0/318951777 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb35803ad20 0x7fb35803d230 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:05:29.837 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.833+0000 7fb378afd700 1 --2- 172.21.15.42:0/318951777 >> v2:172.21.15.40:3300/0 conn(0x7fb37408eb70 0x7fb37408efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:05:29.838 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.833+0000 7fb378afd700 1 --2- 172.21.15.42:0/318951777 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb374116500 0x7fb3740a1ac0 unknown :-1 s=CLOSED pgs=2014 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:05:29.838 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.833+0000 7fb378afd700 1 --2- 172.21.15.42:0/318951777 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb37411ad30 0x7fb37411b160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:05:29.838 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.833+0000 7fb378afd700 1 -- 172.21.15.42:0/318951777 shutdown_connections 2019-12-05T06:05:29.838 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.833+0000 7fb378afd700 1 -- 172.21.15.42:0/318951777 wait complete. 2019-12-05T06:05:29.838 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:29.833+0000 7fb378afd700 1 -- 172.21.15.42:0/318951777 >> 172.21.15.42:0/318951777 conn(0x7fb37409ffb0 msgr2=0x7fb3740a7f90 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:05:29.846 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 244, "feature_flags": { "enable_multiple": false, "ever_enabled_multiple": false }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2019-12-05T05:03:35.604738+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 244, "ever_allowed_features": 2, "explicitly_allowed_features": 2, "failed": [], "flags": 18, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_6821": { "addr": "172.21.15.42:6835/2253267554", "addrs": { "addrvec": [ { "addr": "172.21.15.42:6834", "nonce": 2253267554, "type": "v2" }, { "addr": "172.21.15.42:6835", "nonce": 2253267554, "type": "v1" } ] }, "export_targets": [], "features": 4540138292836696063, "flags": 0, "gid": 6821, "incarnation": 244, "name": "a", "rank": 0, "state": "up:replay", "state_seq": 2 } }, "last_failure": 0, "last_failure_osd_epoch": 92, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 2, "min_compat_client": "0 (unknown)", "modified": "2019-12-05T06:05:29.443166+0000", "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6821 } } } ], "standbys": [] } 2019-12-05T06:05:31.847 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:05:31.926 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:31.917+0000 7f48f16a8700 1 Processor -- start 2019-12-05T06:05:31.926 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:31.917+0000 7f48f16a8700 1 -- start start 2019-12-05T06:05:31.927 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:31.921+0000 7f48f16a8700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f48ec08df80 0x7f48ec09f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:05:31.927 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:31.921+0000 7f48f16a8700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48ec09fc60 con 0x7f48ec0a7740 2019-12-05T06:05:31.927 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:31.921+0000 7f48f16a8700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48ec0a0710 con 0x7f48ec09f830 2019-12-05T06:05:31.927 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:31.921+0000 7f48f16a8700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48ec0a0b00 con 0x7f48ec08df80 2019-12-05T06:05:31.927 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:31.921+0000 7f48eaffd700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f48ec08df80 0x7f48ec09f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:05:31.927 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:31.921+0000 7f48eaffd700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f48ec08df80 0x7f48ec09f2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:47934/0 (socket says 172.21.15.42:47934) 2019-12-05T06:05:31.929 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:31.921+0000 7f48eaffd700 1 -- 172.21.15.42:0/2360594506 learned_addr learned my addr 172.21.15.42:0/2360594506 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:05:31.931 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:31.921+0000 7f48eaffd700 1 -- 172.21.15.42:0/2360594506 >> v1:172.21.15.42:6789/0 conn(0x7f48ec09f830 legacy=0x7f48ec14efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:05:31.931 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:31.921+0000 7f48eaffd700 1 -- 172.21.15.42:0/2360594506 >> v1:172.21.15.40:6789/0 conn(0x7f48ec0a7740 legacy=0x7f48ec07e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:05:31.931 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:31.921+0000 7f48eaffd700 1 -- 172.21.15.42:0/2360594506 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48ec07ef20 con 0x7f48ec08df80 2019-12-05T06:05:31.931 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:31.921+0000 7f48eaffd700 1 --2- 172.21.15.42:0/2360594506 >> v2:172.21.15.42:3300/0 conn(0x7f48ec08df80 0x7f48ec09f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f48e0008790 tx=0x7f48e000fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:31.932 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:31.921+0000 7f48eaffd700 1 --2- 172.21.15.42:0/2360594506 >> v2:172.21.15.42:3300/0 conn(0x7f48ec08df80 0x7f48ec09f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f48e0008790 tx=0x7f48e000fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:31.932 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:31.921+0000 7f48eaffd700 1 --2- 172.21.15.42:0/2360594506 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f48ec08df80 0x7f48ec09f2e0 secure :-1 s=READY pgs=2016 cs=0 l=1 rx=0x7f48e0008790 tx=0x7f48e000fcb0).ready entity=mon.1 client_cookie=697c65a125fac6e server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:05:31.933 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:31.921+0000 7f48eaffd700 1 --2- 172.21.15.42:0/2360594506 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f48ec08df80 0x7f48ec09f2e0 secure :-1 s=THROTTLE_DONE pgs=2016 cs=0 l=1 rx=0x7f48e0008790 tx=0x7f48e000fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:31.939 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:31.921+0000 7f48ea7fc700 1 -- 172.21.15.42:0/2360594506 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f48e0011520 con 0x7f48ec08df80 2019-12-05T06:05:31.939 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:31.921+0000 7f48eaffd700 1 --2- 172.21.15.42:0/2360594506 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f48ec08df80 0x7f48ec09f2e0 secure :-1 s=THROTTLE_DONE pgs=2016 cs=0 l=1 rx=0x7f48e0008790 tx=0x7f48e000fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:31.940 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:31.921+0000 7f48ea7fc700 1 -- 172.21.15.42:0/2360594506 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48e0008ca0 con 0x7f48ec08df80 2019-12-05T06:05:31.940 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:31.921+0000 7f48eaffd700 1 --2- 172.21.15.42:0/2360594506 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f48ec08df80 0x7f48ec09f2e0 secure :-1 s=THROTTLE_DONE pgs=2016 cs=0 l=1 rx=0x7f48e0008790 tx=0x7f48e000fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:31.940 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:31.921+0000 7f48ea7fc700 1 -- 172.21.15.42:0/2360594506 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f48e0011b00 con 0x7f48ec08df80 2019-12-05T06:05:31.940 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:31.921+0000 7f48f16a8700 1 -- 172.21.15.42:0/2360594506 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f48ec08df80 msgr2=0x7f48ec09f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:05:31.940 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:31.921+0000 7f48f16a8700 1 --2- 172.21.15.42:0/2360594506 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f48ec08df80 0x7f48ec09f2e0 secure :-1 s=READY pgs=2016 cs=0 l=1 rx=0x7f48e0008790 tx=0x7f48e000fcb0).stop 2019-12-05T06:05:31.940 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:31.921+0000 7f48f16a8700 1 -- 172.21.15.42:0/2360594506 shutdown_connections 2019-12-05T06:05:31.941 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:31.921+0000 7f48f16a8700 1 --2- 172.21.15.42:0/2360594506 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f48ec08df80 0x7f48ec09f2e0 unknown :-1 s=CLOSED pgs=2016 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:05:31.941 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:31.921+0000 7f48f16a8700 1 -- 172.21.15.42:0/2360594506 shutdown_connections 2019-12-05T06:05:31.941 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:31.921+0000 7f48f16a8700 1 -- 172.21.15.42:0/2360594506 wait complete. 2019-12-05T06:05:31.941 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:31.921+0000 7f48f16a8700 1 -- 172.21.15.42:0/2360594506 >> 172.21.15.42:0/2360594506 conn(0x7f48ec0aabe0 msgr2=0x7f48ec0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:05:31.945 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:31.921+0000 7f48f16a8700 1 Processor -- start 2019-12-05T06:05:31.946 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:31.921+0000 7f48f16a8700 1 -- start start 2019-12-05T06:05:31.946 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:31.925+0000 7f48f16a8700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f48ec127200 0x7f48ec12ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:05:31.946 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:31.925+0000 7f48f16a8700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48ec065c50 con 0x7f48ec126dd0 2019-12-05T06:05:31.946 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:31.925+0000 7f48f16a8700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48ec05fff0 con 0x7f48ec067320 2019-12-05T06:05:31.947 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:31.925+0000 7f48f16a8700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48ec07bde0 con 0x7f48ec127200 2019-12-05T06:05:31.947 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:31.925+0000 7f48eb7fe700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f48ec067320 0x7f48ec05f7f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:47976/0 (socket says 172.21.15.42:47976) 2019-12-05T06:05:31.947 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:31.925+0000 7f48eb7fe700 1 -- 172.21.15.42:0/4042839098 learned_addr learned my addr 172.21.15.42:0/4042839098 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:05:31.947 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:31.925+0000 7f48ebfff700 1 --2- 172.21.15.42:0/4042839098 >> v2:172.21.15.42:3300/0 conn(0x7f48ec127200 0x7f48ec12ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:05:31.947 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:31.925+0000 7f48ebfff700 1 -- 172.21.15.42:0/4042839098 >> v1:172.21.15.42:6789/0 conn(0x7f48ec067320 legacy=0x7f48ec05f7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:05:31.947 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:31.925+0000 7f48ebfff700 1 -- 172.21.15.42:0/4042839098 >> v1:172.21.15.40:6789/0 conn(0x7f48ec126dd0 legacy=0x7f48ec0612b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:05:31.948 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:31.925+0000 7f48ebfff700 1 -- 172.21.15.42:0/4042839098 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48ec127630 con 0x7f48ec127200 2019-12-05T06:05:31.948 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:31.925+0000 7f48ebfff700 1 --2- 172.21.15.42:0/4042839098 >> v2:172.21.15.42:3300/0 conn(0x7f48ec127200 0x7f48ec12ed10 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f48e4001210 tx=0x7f48e4010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:31.948 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:31.925+0000 7f48ebfff700 1 --2- 172.21.15.42:0/4042839098 >> v2:172.21.15.42:3300/0 conn(0x7f48ec127200 0x7f48ec12ed10 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f48e4001210 tx=0x7f48e4010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:31.948 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:31.925+0000 7f48ebfff700 1 --2- 172.21.15.42:0/4042839098 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f48ec127200 0x7f48ec12ed10 secure :-1 s=READY pgs=2018 cs=0 l=1 rx=0x7f48e4001210 tx=0x7f48e4010ac0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:05:31.949 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:31.925+0000 7f48ebfff700 1 --2- 172.21.15.42:0/4042839098 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f48ec127200 0x7f48ec12ed10 secure :-1 s=THROTTLE_DONE pgs=2018 cs=0 l=1 rx=0x7f48e4001210 tx=0x7f48e4010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:31.949 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:31.925+0000 7f48ebfff700 1 --2- 172.21.15.42:0/4042839098 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f48ec127200 0x7f48ec12ed10 secure :-1 s=THROTTLE_DONE pgs=2018 cs=0 l=1 rx=0x7f48e4001210 tx=0x7f48e4010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:31.949 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:31.925+0000 7f48d3fff700 1 -- 172.21.15.42:0/4042839098 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f48e4006ac0 con 0x7f48ec127200 2019-12-05T06:05:31.949 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:31.925+0000 7f48ebfff700 1 --2- 172.21.15.42:0/4042839098 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f48ec127200 0x7f48ec12ed10 secure :-1 s=THROTTLE_DONE pgs=2018 cs=0 l=1 rx=0x7f48e4001210 tx=0x7f48e4010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:31.951 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:31.925+0000 7f48d3fff700 1 -- 172.21.15.42:0/4042839098 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48e40066e0 con 0x7f48ec127200 2019-12-05T06:05:31.951 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:31.925+0000 7f48f16a8700 1 -- 172.21.15.42:0/4042839098 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48ec061ab0 con 0x7f48ec127200 2019-12-05T06:05:31.951 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:31.925+0000 7f48d3fff700 1 -- 172.21.15.42:0/4042839098 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f48e4002820 con 0x7f48ec127200 2019-12-05T06:05:31.951 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:31.925+0000 7f48f16a8700 1 -- 172.21.15.42:0/4042839098 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48ec12f260 con 0x7f48ec127200 2019-12-05T06:05:31.952 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:31.925+0000 7f48ebfff700 1 --2- 172.21.15.42:0/4042839098 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f48ec127200 0x7f48ec12ed10 secure :-1 s=THROTTLE_DONE pgs=2018 cs=0 l=1 rx=0x7f48e4001210 tx=0x7f48e4010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:31.952 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:31.925+0000 7f48d3fff700 1 -- 172.21.15.42:0/4042839098 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f48e4002b30 con 0x7f48ec127200 2019-12-05T06:05:31.952 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:31.925+0000 7f48ebfff700 1 --2- 172.21.15.42:0/4042839098 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f48ec127200 0x7f48ec12ed10 secure :-1 s=THROTTLE_DONE pgs=2018 cs=0 l=1 rx=0x7f48e4001210 tx=0x7f48e4010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:31.952 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:31.925+0000 7f48d3fff700 1 --2- 172.21.15.42:0/4042839098 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f48cc03ac50 0x7f48cc03d160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:05:31.952 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:31.925+0000 7f48d3fff700 1 -- 172.21.15.42:0/4042839098 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9729+0+0 (secure 0 0 0) 0x7f48e404b210 con 0x7f48ec127200 2019-12-05T06:05:31.954 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:31.925+0000 7f48eb7fe700 1 --2- 172.21.15.42:0/4042839098 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f48cc03ac50 0x7f48cc03d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:05:31.954 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:31.925+0000 7f48eb7fe700 1 --2- 172.21.15.42:0/4042839098 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f48cc03ac50 0x7f48cc03d160 crc :-1 s=READY pgs=855 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:05:31.954 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:31.933+0000 7f48f16a8700 1 -- 172.21.15.42:0/4042839098 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48ec05d110 con 0x7f48ec127200 2019-12-05T06:05:31.954 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:31.937+0000 7f48ebfff700 1 --2- 172.21.15.42:0/4042839098 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f48ec127200 0x7f48ec12ed10 secure :-1 s=THROTTLE_DONE pgs=2018 cs=0 l=1 rx=0x7f48e4001210 tx=0x7f48e4010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:31.954 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:31.937+0000 7f48d3fff700 1 -- 172.21.15.42:0/4042839098 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f48e4032290 con 0x7f48ec127200 2019-12-05T06:05:32.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:32.093+0000 7f48f16a8700 1 -- 172.21.15.42:0/4042839098 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f48ec05b8c0 con 0x7f48ec127200 2019-12-05T06:05:32.104 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:32.097+0000 7f48ebfff700 1 --2- 172.21.15.42:0/4042839098 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f48ec127200 0x7f48ec12ed10 secure :-1 s=THROTTLE_DONE pgs=2018 cs=0 l=1 rx=0x7f48e4001210 tx=0x7f48e4010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:32.104 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:32.097+0000 7f48d3fff700 1 -- 172.21.15.42:0/4042839098 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 245 v245) v1 ==== 95+0+1778 (secure 0 0 0) 0x7f48e40324f0 con 0x7f48ec127200 2019-12-05T06:05:32.104 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 245 2019-12-05T06:05:32.104 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:32.097+0000 7f48f16a8700 1 -- 172.21.15.42:0/4042839098 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f48cc03ac50 msgr2=0x7f48cc03d160 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:05:32.105 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:32.097+0000 7f48f16a8700 1 --2- 172.21.15.42:0/4042839098 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f48cc03ac50 0x7f48cc03d160 crc :-1 s=READY pgs=855 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:05:32.105 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:32.097+0000 7f48f16a8700 1 -- 172.21.15.42:0/4042839098 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f48ec127200 msgr2=0x7f48ec12ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:05:32.105 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:32.097+0000 7f48f16a8700 1 --2- 172.21.15.42:0/4042839098 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f48ec127200 0x7f48ec12ed10 secure :-1 s=READY pgs=2018 cs=0 l=1 rx=0x7f48e4001210 tx=0x7f48e4010ac0).stop 2019-12-05T06:05:32.105 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:32.097+0000 7f48f16a8700 1 -- 172.21.15.42:0/4042839098 shutdown_connections 2019-12-05T06:05:32.105 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:32.097+0000 7f48f16a8700 1 --2- 172.21.15.42:0/4042839098 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f48cc03ac50 0x7f48cc03d160 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:05:32.106 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:32.097+0000 7f48f16a8700 1 --2- 172.21.15.42:0/4042839098 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f48ec127200 0x7f48ec12ed10 unknown :-1 s=CLOSED pgs=2018 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:05:32.106 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:32.097+0000 7f48f16a8700 1 -- 172.21.15.42:0/4042839098 shutdown_connections 2019-12-05T06:05:32.106 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:32.097+0000 7f48f16a8700 1 -- 172.21.15.42:0/4042839098 wait complete. 2019-12-05T06:05:32.106 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:32.097+0000 7f48f16a8700 1 -- 172.21.15.42:0/4042839098 >> 172.21.15.42:0/4042839098 conn(0x7f48ec0a9770 msgr2=0x7f48ec08eb70 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:05:32.107 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:05:32.107 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":245,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":245,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:05:31.985992+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":92,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6821},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6821":{"gid":6821,"name":"a","rank":0,"incarnation":244,"state":"up:reconnect","state_seq":21,"addr":"172.21.15.42:6835/2253267554","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2253267554},{"type":"v1","addr":"172.21.15.42:6835","nonce":2253267554}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:05:32.113 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2019-12-05T06:05:34.114 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:05:34.225 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.217+0000 7fb3bc75e700 1 Processor -- start 2019-12-05T06:05:34.225 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.217+0000 7fb3bc75e700 1 -- start start 2019-12-05T06:05:34.226 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.217+0000 7fb3bc75e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb3b40a7740 0x7fb3b408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:05:34.226 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.217+0000 7fb3bc75e700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb3b407e720 0x7fb3b407eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:05:34.226 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.217+0000 7fb3bc75e700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3b40a0710 con 0x7fb3b409f2e0 2019-12-05T06:05:34.226 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.217+0000 7fb3bc75e700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3b40a0b00 con 0x7fb3b407e720 2019-12-05T06:05:34.226 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.217+0000 7fb3bc75e700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3b408e5b0 con 0x7fb3b40a7740 2019-12-05T06:05:34.226 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.217+0000 7fb3bacfb700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb3b40a7740 0x7fb3b408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:05:34.226 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.217+0000 7fb3bacfb700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb3b40a7740 0x7fb3b408df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:47952/0 (socket says 172.21.15.42:47952) 2019-12-05T06:05:34.227 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.217+0000 7fb3bacfb700 1 -- 172.21.15.42:0/4069765683 learned_addr learned my addr 172.21.15.42:0/4069765683 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:05:34.227 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.217+0000 7fb3ba4fa700 1 --2- 172.21.15.42:0/4069765683 >> v2:172.21.15.40:3300/0 conn(0x7fb3b407e720 0x7fb3b407eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:05:34.227 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.217+0000 7fb3bacfb700 1 -- 172.21.15.42:0/4069765683 >> v2:172.21.15.40:3300/0 conn(0x7fb3b407e720 msgr2=0x7fb3b407eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:05:34.227 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.217+0000 7fb3bacfb700 1 --2- 172.21.15.42:0/4069765683 >> v2:172.21.15.40:3300/0 conn(0x7fb3b407e720 0x7fb3b407eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:05:34.227 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.217+0000 7fb3bacfb700 1 -- 172.21.15.42:0/4069765683 >> v1:172.21.15.42:6789/0 conn(0x7fb3b409f2e0 legacy=0x7fb3b409f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:05:34.227 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.217+0000 7fb3b9cf9700 1 -- 172.21.15.42:0/4069765683 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2792203418 0 0) 0x7fb3b40a0710 con 0x7fb3b409f2e0 2019-12-05T06:05:34.227 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.217+0000 7fb3bacfb700 1 -- 172.21.15.42:0/4069765683 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3b407f0a0 con 0x7fb3b40a7740 2019-12-05T06:05:34.228 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.217+0000 7fb3ba4fa700 1 --2- 172.21.15.42:0/4069765683 >> v2:172.21.15.40:3300/0 conn(0x7fb3b407e720 0x7fb3b407eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:05:34.228 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.217+0000 7fb3bacfb700 1 --2- 172.21.15.42:0/4069765683 >> v2:172.21.15.42:3300/0 conn(0x7fb3b40a7740 0x7fb3b408df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fb3b0002f30 tx=0x7fb3b000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:34.228 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.217+0000 7fb3bacfb700 1 --2- 172.21.15.42:0/4069765683 >> v2:172.21.15.42:3300/0 conn(0x7fb3b40a7740 0x7fb3b408df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fb3b0002f30 tx=0x7fb3b000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:34.228 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.217+0000 7fb3bacfb700 1 --2- 172.21.15.42:0/4069765683 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb3b40a7740 0x7fb3b408df80 secure :-1 s=READY pgs=2020 cs=0 l=1 rx=0x7fb3b0002f30 tx=0x7fb3b000fd30).ready entity=mon.1 client_cookie=62590ba57eb4bb77 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:05:34.228 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.217+0000 7fb3bacfb700 1 --2- 172.21.15.42:0/4069765683 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb3b40a7740 0x7fb3b408df80 secure :-1 s=THROTTLE_DONE pgs=2020 cs=0 l=1 rx=0x7fb3b0002f30 tx=0x7fb3b000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:34.228 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.217+0000 7fb3b9cf9700 1 -- 172.21.15.42:0/4069765683 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb3b0011370 con 0x7fb3b40a7740 2019-12-05T06:05:34.228 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.217+0000 7fb3bacfb700 1 --2- 172.21.15.42:0/4069765683 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb3b40a7740 0x7fb3b408df80 secure :-1 s=THROTTLE_DONE pgs=2020 cs=0 l=1 rx=0x7fb3b0002f30 tx=0x7fb3b000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:34.229 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.217+0000 7fb3bacfb700 1 --2- 172.21.15.42:0/4069765683 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb3b40a7740 0x7fb3b408df80 secure :-1 s=THROTTLE_DONE pgs=2020 cs=0 l=1 rx=0x7fb3b0002f30 tx=0x7fb3b000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:34.229 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.217+0000 7fb3b9cf9700 1 -- 172.21.15.42:0/4069765683 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3b0008bb0 con 0x7fb3b40a7740 2019-12-05T06:05:34.229 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.217+0000 7fb3b9cf9700 1 -- 172.21.15.42:0/4069765683 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb3b0011950 con 0x7fb3b40a7740 2019-12-05T06:05:34.229 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.217+0000 7fb3bc75e700 1 -- 172.21.15.42:0/4069765683 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb3b40a7740 msgr2=0x7fb3b408df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:05:34.229 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.221+0000 7fb3bc75e700 1 --2- 172.21.15.42:0/4069765683 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb3b40a7740 0x7fb3b408df80 secure :-1 s=READY pgs=2020 cs=0 l=1 rx=0x7fb3b0002f30 tx=0x7fb3b000fd30).stop 2019-12-05T06:05:34.229 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.221+0000 7fb3bc75e700 1 -- 172.21.15.42:0/4069765683 shutdown_connections 2019-12-05T06:05:34.229 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.221+0000 7fb3bc75e700 1 --2- 172.21.15.42:0/4069765683 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb3b40a7740 0x7fb3b408df80 unknown :-1 s=CLOSED pgs=2020 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:05:34.230 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.221+0000 7fb3bc75e700 1 --2- 172.21.15.42:0/4069765683 >> v2:172.21.15.40:3300/0 conn(0x7fb3b407e720 0x7fb3b407eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:05:34.230 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.221+0000 7fb3bc75e700 1 -- 172.21.15.42:0/4069765683 shutdown_connections 2019-12-05T06:05:34.230 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.221+0000 7fb3bc75e700 1 -- 172.21.15.42:0/4069765683 wait complete. 2019-12-05T06:05:34.230 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.221+0000 7fb3bc75e700 1 -- 172.21.15.42:0/4069765683 >> 172.21.15.42:0/4069765683 conn(0x7fb3b40aabe0 msgr2=0x7fb3b40a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:05:34.233 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.221+0000 7fb3bc75e700 1 Processor -- start 2019-12-05T06:05:34.233 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.221+0000 7fb3bc75e700 1 -- start start 2019-12-05T06:05:34.233 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.225+0000 7fb3bc75e700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb3b411ac60 0x7fb3b4073af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:05:34.234 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.225+0000 7fb3bc75e700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3b4074040 con 0x7fb3b4117b30 2019-12-05T06:05:34.234 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.225+0000 7fb3bc75e700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3b411b090 con 0x7fb3b4116500 2019-12-05T06:05:34.234 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.225+0000 7fb3bc75e700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3b411b480 con 0x7fb3b411ac60 2019-12-05T06:05:34.234 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.225+0000 7fb3ba4fa700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fb3b4116500 0x7fb3b40732f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:47998/0 (socket says 172.21.15.42:47998) 2019-12-05T06:05:34.234 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.225+0000 7fb3ba4fa700 1 -- 172.21.15.42:0/3924364520 learned_addr learned my addr 172.21.15.42:0/3924364520 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:05:34.234 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.225+0000 7fb3bb4fc700 1 --2- 172.21.15.42:0/3924364520 >> v2:172.21.15.40:3300/0 conn(0x7fb3b411ac60 0x7fb3b4073af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:05:34.235 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.225+0000 7fb3ab7fe700 1 -- 172.21.15.42:0/3924364520 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 804418446 0 0) 0x7fb3b411b090 con 0x7fb3b4116500 2019-12-05T06:05:34.235 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.225+0000 7fb3ab7fe700 1 -- 172.21.15.42:0/3924364520 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb39c0030b0 con 0x7fb3b4116500 2019-12-05T06:05:34.235 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.225+0000 7fb3bb4fc700 1 -- 172.21.15.42:0/3924364520 >> v1:172.21.15.42:6789/0 conn(0x7fb3b4116500 legacy=0x7fb3b40732f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:05:34.235 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.225+0000 7fb3bb4fc700 1 -- 172.21.15.42:0/3924364520 >> v1:172.21.15.40:6789/0 conn(0x7fb3b4117b30 legacy=0x7fb3b411bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:05:34.235 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.225+0000 7fb3ab7fe700 1 -- 172.21.15.42:0/3924364520 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3039118759 0 0) 0x7fb3b4074040 con 0x7fb3b4117b30 2019-12-05T06:05:34.235 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.225+0000 7fb3bb4fc700 1 -- 172.21.15.42:0/3924364520 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3b4117f90 con 0x7fb3b411ac60 2019-12-05T06:05:34.235 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.225+0000 7fb3bb4fc700 1 --2- 172.21.15.42:0/3924364520 >> v2:172.21.15.40:3300/0 conn(0x7fb3b411ac60 0x7fb3b4073af0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fb3ac001210 tx=0x7fb3ac010ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:34.236 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.225+0000 7fb3bb4fc700 1 --2- 172.21.15.42:0/3924364520 >> v2:172.21.15.40:3300/0 conn(0x7fb3b411ac60 0x7fb3b4073af0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fb3ac001210 tx=0x7fb3ac010ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:34.236 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.225+0000 7fb3bb4fc700 1 --2- 172.21.15.42:0/3924364520 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb3b411ac60 0x7fb3b4073af0 secure :-1 s=READY pgs=1528 cs=0 l=1 rx=0x7fb3ac001210 tx=0x7fb3ac010ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:05:34.236 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.225+0000 7fb3bb4fc700 1 --2- 172.21.15.42:0/3924364520 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb3b411ac60 0x7fb3b4073af0 secure :-1 s=THROTTLE_DONE pgs=1528 cs=0 l=1 rx=0x7fb3ac001210 tx=0x7fb3ac010ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:34.236 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.225+0000 7fb3ab7fe700 1 -- 172.21.15.42:0/3924364520 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb3ac002220 con 0x7fb3b411ac60 2019-12-05T06:05:34.236 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.225+0000 7fb3bb4fc700 1 --2- 172.21.15.42:0/3924364520 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb3b411ac60 0x7fb3b4073af0 secure :-1 s=THROTTLE_DONE pgs=1528 cs=0 l=1 rx=0x7fb3ac001210 tx=0x7fb3ac010ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:34.236 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.225+0000 7fb3ab7fe700 1 -- 172.21.15.42:0/3924364520 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3ac006ad0 con 0x7fb3b411ac60 2019-12-05T06:05:34.237 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.225+0000 7fb3bb4fc700 1 --2- 172.21.15.42:0/3924364520 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb3b411ac60 0x7fb3b4073af0 secure :-1 s=THROTTLE_DONE pgs=1528 cs=0 l=1 rx=0x7fb3ac001210 tx=0x7fb3ac010ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:34.237 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.225+0000 7fb3bc75e700 1 -- 172.21.15.42:0/3924364520 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3b407b570 con 0x7fb3b411ac60 2019-12-05T06:05:34.237 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.225+0000 7fb3ab7fe700 1 -- 172.21.15.42:0/3924364520 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb3ac002800 con 0x7fb3b411ac60 2019-12-05T06:05:34.237 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.225+0000 7fb3bc75e700 1 -- 172.21.15.42:0/3924364520 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3b407bac0 con 0x7fb3b411ac60 2019-12-05T06:05:34.237 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.225+0000 7fb3bb4fc700 1 --2- 172.21.15.42:0/3924364520 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb3b411ac60 0x7fb3b4073af0 secure :-1 s=THROTTLE_DONE pgs=1528 cs=0 l=1 rx=0x7fb3ac001210 tx=0x7fb3ac010ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:34.237 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.225+0000 7fb3ab7fe700 1 -- 172.21.15.42:0/3924364520 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fb3ac002b60 con 0x7fb3b411ac60 2019-12-05T06:05:34.238 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.225+0000 7fb3bb4fc700 1 --2- 172.21.15.42:0/3924364520 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb3b411ac60 0x7fb3b4073af0 secure :-1 s=THROTTLE_DONE pgs=1528 cs=0 l=1 rx=0x7fb3ac001210 tx=0x7fb3ac010ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:34.238 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.229+0000 7fb3ab7fe700 1 --2- 172.21.15.42:0/3924364520 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb39c03b1c0 0x7fb39c03d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:05:34.238 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.229+0000 7fb3ab7fe700 1 -- 172.21.15.42:0/3924364520 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9729+0+0 (secure 0 0 0) 0x7fb3ac04b420 con 0x7fb3b411ac60 2019-12-05T06:05:34.238 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.229+0000 7fb3bacfb700 1 --2- 172.21.15.42:0/3924364520 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb39c03b1c0 0x7fb39c03d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:05:34.238 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.229+0000 7fb3bacfb700 1 --2- 172.21.15.42:0/3924364520 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb39c03b1c0 0x7fb39c03d6d0 crc :-1 s=READY pgs=856 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:05:34.238 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.233+0000 7fb3bc75e700 1 -- 172.21.15.42:0/3924364520 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3b405d110 con 0x7fb3b411ac60 2019-12-05T06:05:34.248 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.241+0000 7fb3bb4fc700 1 --2- 172.21.15.42:0/3924364520 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb3b411ac60 0x7fb3b4073af0 secure :-1 s=THROTTLE_DONE pgs=1528 cs=0 l=1 rx=0x7fb3ac001210 tx=0x7fb3ac010ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:34.249 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.241+0000 7fb3ab7fe700 1 -- 172.21.15.42:0/3924364520 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fb3ac032480 con 0x7fb3b411ac60 2019-12-05T06:05:34.414 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.409+0000 7fb3bc75e700 1 -- 172.21.15.42:0/3924364520 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb3b405b8c0 con 0x7fb3b411ac60 2019-12-05T06:05:34.415 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.413+0000 7fb3bb4fc700 1 --2- 172.21.15.42:0/3924364520 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb3b411ac60 0x7fb3b4073af0 secure :-1 s=THROTTLE_DONE pgs=1528 cs=0 l=1 rx=0x7fb3ac001210 tx=0x7fb3ac010ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:34.415 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.413+0000 7fb3ab7fe700 1 -- 172.21.15.42:0/3924364520 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 247 v247) v1 ==== 95+0+1775 (secure 0 0 0) 0x7fb3ac0326e0 con 0x7fb3b411ac60 2019-12-05T06:05:34.417 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 247 2019-12-05T06:05:34.418 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:05:34.418 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":247,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":247,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:05:33.997161+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":92,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6821},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6821":{"gid":6821,"name":"a","rank":0,"incarnation":244,"state":"up:active","state_seq":23,"addr":"172.21.15.42:6835/2253267554","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2253267554},{"type":"v1","addr":"172.21.15.42:6835","nonce":2253267554}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:05:34.421 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.417+0000 7fb3bc75e700 1 -- 172.21.15.42:0/3924364520 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb39c03b1c0 msgr2=0x7fb39c03d6d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:05:34.421 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.417+0000 7fb3bc75e700 1 --2- 172.21.15.42:0/3924364520 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb39c03b1c0 0x7fb39c03d6d0 crc :-1 s=READY pgs=856 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:05:34.421 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.417+0000 7fb3bc75e700 1 -- 172.21.15.42:0/3924364520 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb3b411ac60 msgr2=0x7fb3b4073af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:05:34.421 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.417+0000 7fb3bc75e700 1 --2- 172.21.15.42:0/3924364520 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb3b411ac60 0x7fb3b4073af0 secure :-1 s=READY pgs=1528 cs=0 l=1 rx=0x7fb3ac001210 tx=0x7fb3ac010ad0).stop 2019-12-05T06:05:34.422 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.417+0000 7fb3bc75e700 1 -- 172.21.15.42:0/3924364520 shutdown_connections 2019-12-05T06:05:34.422 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.417+0000 7fb3bc75e700 1 --2- 172.21.15.42:0/3924364520 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb39c03b1c0 0x7fb39c03d6d0 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:05:34.422 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.417+0000 7fb3bc75e700 1 --2- 172.21.15.42:0/3924364520 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb3b411ac60 0x7fb3b4073af0 unknown :-1 s=CLOSED pgs=1528 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:05:34.422 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.417+0000 7fb3bc75e700 1 -- 172.21.15.42:0/3924364520 shutdown_connections 2019-12-05T06:05:34.422 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.417+0000 7fb3bc75e700 1 -- 172.21.15.42:0/3924364520 wait complete. 2019-12-05T06:05:34.422 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.417+0000 7fb3bc75e700 1 -- 172.21.15.42:0/3924364520 >> 172.21.15.42:0/3924364520 conn(0x7fb3b409faf0 msgr2=0x7fb3b40a9790 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:05:34.433 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a has gained rank=0, replacing gid=9381 2019-12-05T06:05:34.433 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 0 secs before reviving mds.b 2019-12-05T06:05:34.433 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.b 2019-12-05T06:05:34.433 INFO:tasks.ceph.mds.b:Restarting daemon 2019-12-05T06:05:34.433 INFO:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2019-12-05T06:05:34.437 INFO:tasks.ceph.mds.b:Started 2019-12-05T06:05:34.437 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:05:34.492 INFO:tasks.ceph.mds.b.smithi040.stdout:starting mds.b at 2019-12-05T06:05:34.526 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.517+0000 7f33fbb21700 1 Processor -- start 2019-12-05T06:05:34.526 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.517+0000 7f33fbb21700 1 -- start start 2019-12-05T06:05:34.526 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.517+0000 7f33fbb21700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f33f40a7740 0x7f33f408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:05:34.526 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.517+0000 7f33fbb21700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f33f407e720 0x7f33f407eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:05:34.526 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.517+0000 7f33fbb21700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f33f409f2e0 0x7f33f409f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:05:34.527 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.517+0000 7f33fbb21700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33f40a0b00 con 0x7f33f40a7740 2019-12-05T06:05:34.527 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.517+0000 7f33fbb21700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33f408e5b0 con 0x7f33f409f2e0 2019-12-05T06:05:34.527 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.517+0000 7f33fbb21700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33f408e850 con 0x7f33f407e720 2019-12-05T06:05:34.527 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.517+0000 7f33fa8bf700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f33f409f2e0 0x7f33f409f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:05:34.527 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.517+0000 7f33fa8bf700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f33f409f2e0 0x7f33f409f710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:47972/0 (socket says 172.21.15.42:47972) 2019-12-05T06:05:34.527 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.517+0000 7f33fa8bf700 1 -- 172.21.15.42:0/441476149 learned_addr learned my addr 172.21.15.42:0/441476149 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:05:34.527 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.517+0000 7f33f98bd700 1 --2- 172.21.15.42:0/441476149 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f33f407e720 0x7f33f407eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:05:34.528 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.517+0000 7f33fa0be700 1 --2- 172.21.15.42:0/441476149 >> v2:172.21.15.40:3300/0 conn(0x7f33f40a7740 0x7f33f408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:05:34.528 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.517+0000 7f33fa8bf700 1 -- 172.21.15.42:0/441476149 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f33f407e720 msgr2=0x7f33f407eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:05:34.528 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.517+0000 7f33fa8bf700 1 --2- 172.21.15.42:0/441476149 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f33f407e720 0x7f33f407eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:05:34.528 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.517+0000 7f33fa8bf700 1 -- 172.21.15.42:0/441476149 >> v2:172.21.15.40:3300/0 conn(0x7f33f40a7740 msgr2=0x7f33f408df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:05:34.528 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.517+0000 7f33fa8bf700 1 --2- 172.21.15.42:0/441476149 >> v2:172.21.15.40:3300/0 conn(0x7f33f40a7740 0x7f33f408df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:05:34.528 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.521+0000 7f33fa8bf700 1 -- 172.21.15.42:0/441476149 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33f407f0a0 con 0x7f33f409f2e0 2019-12-05T06:05:34.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.521+0000 7f33fa8bf700 1 --2- 172.21.15.42:0/441476149 >> v2:172.21.15.42:3300/0 conn(0x7f33f409f2e0 0x7f33f409f710 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f33f408e770 tx=0x7f33ec00fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:34.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.521+0000 7f33fa8bf700 1 --2- 172.21.15.42:0/441476149 >> v2:172.21.15.42:3300/0 conn(0x7f33f409f2e0 0x7f33f409f710 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f33f408e770 tx=0x7f33ec00fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:34.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.521+0000 7f33fa8bf700 1 --2- 172.21.15.42:0/441476149 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f33f409f2e0 0x7f33f409f710 secure :-1 s=READY pgs=2024 cs=0 l=1 rx=0x7f33f408e770 tx=0x7f33ec00fdb0).ready entity=mon.1 client_cookie=a480dcc3fe3ad296 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:05:34.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.521+0000 7f33fa8bf700 1 --2- 172.21.15.42:0/441476149 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f33f409f2e0 0x7f33f409f710 secure :-1 s=THROTTLE_DONE pgs=2024 cs=0 l=1 rx=0x7f33f408e770 tx=0x7f33ec00fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:34.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.521+0000 7f33fa8bf700 1 --2- 172.21.15.42:0/441476149 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f33f409f2e0 0x7f33f409f710 secure :-1 s=THROTTLE_DONE pgs=2024 cs=0 l=1 rx=0x7f33f408e770 tx=0x7f33ec00fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:34.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.521+0000 7f33f90bc700 1 -- 172.21.15.42:0/441476149 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f33ec008c90 con 0x7f33f409f2e0 2019-12-05T06:05:34.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.521+0000 7f33fa8bf700 1 --2- 172.21.15.42:0/441476149 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f33f409f2e0 0x7f33f409f710 secure :-1 s=THROTTLE_DONE pgs=2024 cs=0 l=1 rx=0x7f33f408e770 tx=0x7f33ec00fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:34.530 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.521+0000 7f33f90bc700 1 -- 172.21.15.42:0/441476149 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33ec0088b0 con 0x7f33f409f2e0 2019-12-05T06:05:34.530 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.521+0000 7f33f90bc700 1 -- 172.21.15.42:0/441476149 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f33ec011b50 con 0x7f33f409f2e0 2019-12-05T06:05:34.530 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.521+0000 7f33fbb21700 1 -- 172.21.15.42:0/441476149 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f33f409f2e0 msgr2=0x7f33f409f710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:05:34.530 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.521+0000 7f33fbb21700 1 --2- 172.21.15.42:0/441476149 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f33f409f2e0 0x7f33f409f710 secure :-1 s=READY pgs=2024 cs=0 l=1 rx=0x7f33f408e770 tx=0x7f33ec00fdb0).stop 2019-12-05T06:05:34.530 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.521+0000 7f33fbb21700 1 -- 172.21.15.42:0/441476149 shutdown_connections 2019-12-05T06:05:34.539 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.521+0000 7f33fbb21700 1 --2- 172.21.15.42:0/441476149 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f33f409f2e0 0x7f33f409f710 unknown :-1 s=CLOSED pgs=2024 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:05:34.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.521+0000 7f33fbb21700 1 --2- 172.21.15.42:0/441476149 >> v2:172.21.15.40:3300/0 conn(0x7f33f40a7740 0x7f33f408df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:05:34.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.521+0000 7f33fbb21700 1 --2- 172.21.15.42:0/441476149 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f33f407e720 0x7f33f407eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:05:34.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.521+0000 7f33fbb21700 1 -- 172.21.15.42:0/441476149 shutdown_connections 2019-12-05T06:05:34.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.521+0000 7f33fbb21700 1 -- 172.21.15.42:0/441476149 wait complete. 2019-12-05T06:05:34.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.521+0000 7f33fbb21700 1 -- 172.21.15.42:0/441476149 >> 172.21.15.42:0/441476149 conn(0x7f33f40aabe0 msgr2=0x7f33f40a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:05:34.545 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.521+0000 7f33fbb21700 1 Processor -- start 2019-12-05T06:05:34.545 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.521+0000 7f33fbb21700 1 -- start start 2019-12-05T06:05:34.545 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.525+0000 7f33fbb21700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f33f4123130 0x7f33f4123560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:05:34.545 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.525+0000 7f33fbb21700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f33f40722c0 0x7f33f4074700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:05:34.546 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.525+0000 7f33fbb21700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33f408f100 con 0x7f33f4144810 2019-12-05T06:05:34.546 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.525+0000 7f33fbb21700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33f4122d40 con 0x7f33f40722c0 2019-12-05T06:05:34.546 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.525+0000 7f33fbb21700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33f4142030 con 0x7f33f4123130 2019-12-05T06:05:34.546 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.525+0000 7f33fa8bf700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f33f40722c0 0x7f33f4074700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:05:34.546 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.525+0000 7f33fa8bf700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f33f40722c0 0x7f33f4074700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:47978/0 (socket says 172.21.15.42:47978) 2019-12-05T06:05:34.546 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.525+0000 7f33fa8bf700 1 -- 172.21.15.42:0/634274023 learned_addr learned my addr 172.21.15.42:0/634274023 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:05:34.546 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.525+0000 7f33f98bd700 1 --2- 172.21.15.42:0/634274023 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f33f4123130 0x7f33f4123560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:05:34.547 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.525+0000 7f33fa8bf700 1 -- 172.21.15.42:0/634274023 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f33f4123130 msgr2=0x7f33f4123560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:05:34.547 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.525+0000 7f33fa8bf700 1 --2- 172.21.15.42:0/634274023 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f33f4123130 0x7f33f4123560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:05:34.547 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.525+0000 7f33fa8bf700 1 -- 172.21.15.42:0/634274023 >> v1:172.21.15.40:6789/0 conn(0x7f33f4144810 legacy=0x7f33f4121420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:05:34.547 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.525+0000 7f33fa8bf700 1 -- 172.21.15.42:0/634274023 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33f4123ab0 con 0x7f33f40722c0 2019-12-05T06:05:34.547 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.525+0000 7f33fa8bf700 1 --2- 172.21.15.42:0/634274023 >> v2:172.21.15.42:3300/0 conn(0x7f33f40722c0 0x7f33f4074700 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f33f408e770 tx=0x7f33ec008b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:34.547 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.525+0000 7f33fa8bf700 1 --2- 172.21.15.42:0/634274023 >> v2:172.21.15.42:3300/0 conn(0x7f33f40722c0 0x7f33f4074700 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f33f408e770 tx=0x7f33ec008b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:34.548 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.525+0000 7f33fa8bf700 1 --2- 172.21.15.42:0/634274023 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f33f40722c0 0x7f33f4074700 secure :-1 s=READY pgs=2025 cs=0 l=1 rx=0x7f33f408e770 tx=0x7f33ec008b60).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:05:34.548 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.525+0000 7f33fa8bf700 1 --2- 172.21.15.42:0/634274023 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f33f40722c0 0x7f33f4074700 secure :-1 s=THROTTLE_DONE pgs=2025 cs=0 l=1 rx=0x7f33f408e770 tx=0x7f33ec008b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:34.548 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.525+0000 7f33fa8bf700 1 --2- 172.21.15.42:0/634274023 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f33f40722c0 0x7f33f4074700 secure :-1 s=THROTTLE_DONE pgs=2025 cs=0 l=1 rx=0x7f33f408e770 tx=0x7f33ec008b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:34.548 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.525+0000 7f33eaffd700 1 -- 172.21.15.42:0/634274023 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f33ec00f680 con 0x7f33f40722c0 2019-12-05T06:05:34.548 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.525+0000 7f33fa8bf700 1 --2- 172.21.15.42:0/634274023 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f33f40722c0 0x7f33f4074700 secure :-1 s=THROTTLE_DONE pgs=2025 cs=0 l=1 rx=0x7f33f408e770 tx=0x7f33ec008b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:34.548 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.525+0000 7f33eaffd700 1 -- 172.21.15.42:0/634274023 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33ec011770 con 0x7f33f40722c0 2019-12-05T06:05:34.548 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.525+0000 7f33fbb21700 1 -- 172.21.15.42:0/634274023 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33f4144c40 con 0x7f33f40722c0 2019-12-05T06:05:34.549 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.525+0000 7f33eaffd700 1 -- 172.21.15.42:0/634274023 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f33ec005920 con 0x7f33f40722c0 2019-12-05T06:05:34.549 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.525+0000 7f33fbb21700 1 -- 172.21.15.42:0/634274023 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33f4074c50 con 0x7f33f40722c0 2019-12-05T06:05:34.549 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.525+0000 7f33fa8bf700 1 --2- 172.21.15.42:0/634274023 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f33f40722c0 0x7f33f4074700 secure :-1 s=THROTTLE_DONE pgs=2025 cs=0 l=1 rx=0x7f33f408e770 tx=0x7f33ec008b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:34.549 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.525+0000 7f33eaffd700 1 -- 172.21.15.42:0/634274023 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f33ec005c30 con 0x7f33f40722c0 2019-12-05T06:05:34.549 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.525+0000 7f33fa8bf700 1 --2- 172.21.15.42:0/634274023 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f33f40722c0 0x7f33f4074700 secure :-1 s=THROTTLE_DONE pgs=2025 cs=0 l=1 rx=0x7f33f408e770 tx=0x7f33ec008b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:34.549 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.529+0000 7f33eaffd700 1 --2- 172.21.15.42:0/634274023 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f33dc03ae30 0x7f33dc03d340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:05:34.549 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.529+0000 7f33eaffd700 1 -- 172.21.15.42:0/634274023 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9729+0+0 (secure 0 0 0) 0x7f33ec08f380 con 0x7f33f40722c0 2019-12-05T06:05:34.550 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.529+0000 7f33fa0be700 1 --2- 172.21.15.42:0/634274023 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f33dc03ae30 0x7f33dc03d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:05:34.550 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.529+0000 7f33fa0be700 1 --2- 172.21.15.42:0/634274023 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f33dc03ae30 0x7f33dc03d340 crc :-1 s=READY pgs=857 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:05:34.550 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.533+0000 7f33fbb21700 1 -- 172.21.15.42:0/634274023 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33f405d110 con 0x7f33f40722c0 2019-12-05T06:05:34.550 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.541+0000 7f33fa8bf700 1 --2- 172.21.15.42:0/634274023 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f33f40722c0 0x7f33f4074700 secure :-1 s=THROTTLE_DONE pgs=2025 cs=0 l=1 rx=0x7f33f408e770 tx=0x7f33ec008b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:34.550 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.541+0000 7f33eaffd700 1 -- 172.21.15.42:0/634274023 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f33ec077210 con 0x7f33f40722c0 2019-12-05T06:05:34.736 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.733+0000 7f33fbb21700 1 -- 172.21.15.42:0/634274023 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f33f405b8c0 con 0x7f33f40722c0 2019-12-05T06:05:34.737 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.733+0000 7f33fa8bf700 1 --2- 172.21.15.42:0/634274023 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f33f40722c0 0x7f33f4074700 secure :-1 s=THROTTLE_DONE pgs=2025 cs=0 l=1 rx=0x7f33f408e770 tx=0x7f33ec008b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:34.737 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.733+0000 7f33eaffd700 1 -- 172.21.15.42:0/634274023 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 247 v247) v1 ==== 95+0+1775 (secure 0 0 0) 0x7f33ec011e20 con 0x7f33f40722c0 2019-12-05T06:05:34.737 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 247 2019-12-05T06:05:34.738 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:05:34.738 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":247,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":247,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:05:33.997161+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":92,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6821},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6821":{"gid":6821,"name":"a","rank":0,"incarnation":244,"state":"up:active","state_seq":23,"addr":"172.21.15.42:6835/2253267554","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2253267554},{"type":"v1","addr":"172.21.15.42:6835","nonce":2253267554}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:05:34.740 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.737+0000 7f33fbb21700 1 -- 172.21.15.42:0/634274023 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f33dc03ae30 msgr2=0x7f33dc03d340 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:05:34.740 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.737+0000 7f33fbb21700 1 --2- 172.21.15.42:0/634274023 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f33dc03ae30 0x7f33dc03d340 crc :-1 s=READY pgs=857 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:05:34.740 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.737+0000 7f33fbb21700 1 -- 172.21.15.42:0/634274023 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f33f40722c0 msgr2=0x7f33f4074700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:05:34.740 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.737+0000 7f33fbb21700 1 --2- 172.21.15.42:0/634274023 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f33f40722c0 0x7f33f4074700 secure :-1 s=READY pgs=2025 cs=0 l=1 rx=0x7f33f408e770 tx=0x7f33ec008b60).stop 2019-12-05T06:05:34.741 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.737+0000 7f33fbb21700 1 -- 172.21.15.42:0/634274023 shutdown_connections 2019-12-05T06:05:34.741 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.737+0000 7f33fbb21700 1 --2- 172.21.15.42:0/634274023 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f33dc03ae30 0x7f33dc03d340 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:05:34.741 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.737+0000 7f33fbb21700 1 --2- 172.21.15.42:0/634274023 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f33f40722c0 0x7f33f4074700 unknown :-1 s=CLOSED pgs=2025 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:05:34.741 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.737+0000 7f33fbb21700 1 --2- 172.21.15.42:0/634274023 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f33f4123130 0x7f33f4123560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:05:34.742 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.737+0000 7f33fbb21700 1 -- 172.21.15.42:0/634274023 shutdown_connections 2019-12-05T06:05:34.742 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.737+0000 7f33fbb21700 1 -- 172.21.15.42:0/634274023 wait complete. 2019-12-05T06:05:34.742 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:34.737+0000 7f33fbb21700 1 -- 172.21.15.42:0/634274023 >> 172.21.15.42:0/634274023 conn(0x7f33f407e590 msgr2=0x7f33f40a9640 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:05:34.750 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2019-12-05T06:05:36.751 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:05:36.856 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:36.849+0000 7ffbe746a700 1 Processor -- start 2019-12-05T06:05:36.856 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:36.849+0000 7ffbe746a700 1 -- start start 2019-12-05T06:05:36.856 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:36.849+0000 7ffbe746a700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ffbe008df70 0x7ffbe009f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:05:36.856 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:36.849+0000 7ffbe746a700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbe009fc50 con 0x7ffbe00a7730 2019-12-05T06:05:36.857 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:36.849+0000 7ffbe746a700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbe00a0700 con 0x7ffbe009f820 2019-12-05T06:05:36.857 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:36.849+0000 7ffbe746a700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffbe00a0af0 con 0x7ffbe008df70 2019-12-05T06:05:36.857 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:36.849+0000 7ffbe6208700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7ffbe009f820 0x7ffbe014efd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:48036/0 (socket says 172.21.15.42:48036) 2019-12-05T06:05:36.857 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:36.849+0000 7ffbe6208700 1 -- 172.21.15.42:0/2401641936 learned_addr learned my addr 172.21.15.42:0/2401641936 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:05:36.857 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:36.849+0000 7ffbe5206700 1 --2- 172.21.15.42:0/2401641936 >> v2:172.21.15.40:3300/0 conn(0x7ffbe008df70 0x7ffbe009f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:05:36.857 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:36.849+0000 7ffbe4a05700 1 -- 172.21.15.42:0/2401641936 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1321333879 0 0) 0x7ffbe00a0700 con 0x7ffbe009f820 2019-12-05T06:05:36.857 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:36.849+0000 7ffbe4a05700 1 -- 172.21.15.42:0/2401641936 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffbcc0030b0 con 0x7ffbe009f820 2019-12-05T06:05:36.858 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:36.849+0000 7ffbe4a05700 1 -- 172.21.15.42:0/2401641936 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2723119677 0 0) 0x7ffbcc0030b0 con 0x7ffbe009f820 2019-12-05T06:05:36.858 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:36.849+0000 7ffbe5206700 1 -- 172.21.15.42:0/2401641936 >> v1:172.21.15.42:6789/0 conn(0x7ffbe009f820 legacy=0x7ffbe014efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:05:36.858 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:36.849+0000 7ffbe5206700 1 -- 172.21.15.42:0/2401641936 >> v1:172.21.15.40:6789/0 conn(0x7ffbe00a7730 legacy=0x7ffbe007e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:05:36.858 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:36.849+0000 7ffbe5206700 1 -- 172.21.15.42:0/2401641936 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbe007ef10 con 0x7ffbe008df70 2019-12-05T06:05:36.858 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:36.849+0000 7ffbe5206700 1 --2- 172.21.15.42:0/2401641936 >> v2:172.21.15.40:3300/0 conn(0x7ffbe008df70 0x7ffbe009f2d0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7ffbd4008b10 tx=0x7ffbd8001990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:36.858 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:36.849+0000 7ffbe5206700 1 --2- 172.21.15.42:0/2401641936 >> v2:172.21.15.40:3300/0 conn(0x7ffbe008df70 0x7ffbe009f2d0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7ffbd4008b10 tx=0x7ffbd8001990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:36.858 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:36.849+0000 7ffbe5206700 1 --2- 172.21.15.42:0/2401641936 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffbe008df70 0x7ffbe009f2d0 secure :-1 s=READY pgs=1534 cs=0 l=1 rx=0x7ffbd4008b10 tx=0x7ffbd8001990).ready entity=mon.0 client_cookie=aa812d20e371aaa0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:05:36.859 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:36.849+0000 7ffbe5206700 1 --2- 172.21.15.42:0/2401641936 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffbe008df70 0x7ffbe009f2d0 secure :-1 s=THROTTLE_DONE pgs=1534 cs=0 l=1 rx=0x7ffbd4008b10 tx=0x7ffbd8001990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:36.859 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:36.849+0000 7ffbe4a05700 1 -- 172.21.15.42:0/2401641936 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ffbd8001750 con 0x7ffbe008df70 2019-12-05T06:05:36.859 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:36.849+0000 7ffbe5206700 1 --2- 172.21.15.42:0/2401641936 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffbe008df70 0x7ffbe009f2d0 secure :-1 s=THROTTLE_DONE pgs=1534 cs=0 l=1 rx=0x7ffbd4008b10 tx=0x7ffbd8001990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:36.859 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:36.849+0000 7ffbe5206700 1 --2- 172.21.15.42:0/2401641936 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffbe008df70 0x7ffbe009f2d0 secure :-1 s=THROTTLE_DONE pgs=1534 cs=0 l=1 rx=0x7ffbd4008b10 tx=0x7ffbd8001990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:36.859 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:36.849+0000 7ffbe4a05700 1 -- 172.21.15.42:0/2401641936 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbd8001560 con 0x7ffbe008df70 2019-12-05T06:05:36.860 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:36.849+0000 7ffbe4a05700 1 -- 172.21.15.42:0/2401641936 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ffbd4011ac0 con 0x7ffbe008df70 2019-12-05T06:05:36.860 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:36.849+0000 7ffbe746a700 1 -- 172.21.15.42:0/2401641936 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffbe008df70 msgr2=0x7ffbe009f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:05:36.860 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:36.849+0000 7ffbe746a700 1 --2- 172.21.15.42:0/2401641936 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffbe008df70 0x7ffbe009f2d0 secure :-1 s=READY pgs=1534 cs=0 l=1 rx=0x7ffbd4008b10 tx=0x7ffbd8001990).stop 2019-12-05T06:05:36.860 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:36.849+0000 7ffbe746a700 1 -- 172.21.15.42:0/2401641936 shutdown_connections 2019-12-05T06:05:36.860 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:36.849+0000 7ffbe746a700 1 --2- 172.21.15.42:0/2401641936 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffbe008df70 0x7ffbe009f2d0 unknown :-1 s=CLOSED pgs=1534 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:05:36.860 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:36.853+0000 7ffbe746a700 1 -- 172.21.15.42:0/2401641936 shutdown_connections 2019-12-05T06:05:36.861 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:36.853+0000 7ffbe746a700 1 -- 172.21.15.42:0/2401641936 wait complete. 2019-12-05T06:05:36.861 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:36.853+0000 7ffbe746a700 1 -- 172.21.15.42:0/2401641936 >> 172.21.15.42:0/2401641936 conn(0x7ffbe00aabd0 msgr2=0x7ffbe00a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:05:36.863 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:36.853+0000 7ffbe746a700 1 Processor -- start 2019-12-05T06:05:36.864 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:36.853+0000 7ffbe746a700 1 -- start start 2019-12-05T06:05:36.864 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:36.853+0000 7ffbe746a700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ffbe01161b0 0x7ffbe012ff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:05:36.864 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:36.853+0000 7ffbe746a700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ffbe0130480 0x7ffbe0118220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:05:36.864 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:36.853+0000 7ffbe746a700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbe0141fd0 con 0x7ffbe0118770 2019-12-05T06:05:36.864 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:36.853+0000 7ffbe746a700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffbe0119780 con 0x7ffbe01161b0 2019-12-05T06:05:36.864 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:36.853+0000 7ffbe746a700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffbe0119a20 con 0x7ffbe0130480 2019-12-05T06:05:36.865 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:36.853+0000 7ffbe5206700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ffbe0130480 0x7ffbe0118220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:05:36.865 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:36.853+0000 7ffbe5206700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ffbe0130480 0x7ffbe0118220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:48004/0 (socket says 172.21.15.42:48004) 2019-12-05T06:05:36.865 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:36.853+0000 7ffbe5206700 1 -- 172.21.15.42:0/3251942088 learned_addr learned my addr 172.21.15.42:0/3251942088 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:05:36.865 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:36.853+0000 7ffbe5a07700 1 --2- 172.21.15.42:0/3251942088 >> v2:172.21.15.40:3300/0 conn(0x7ffbe01161b0 0x7ffbe012ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:05:36.865 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:36.853+0000 7ffbe5206700 1 -- 172.21.15.42:0/3251942088 >> v2:172.21.15.40:3300/0 conn(0x7ffbe01161b0 msgr2=0x7ffbe012ff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:05:36.865 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:36.853+0000 7ffbe5206700 1 --2- 172.21.15.42:0/3251942088 >> v2:172.21.15.40:3300/0 conn(0x7ffbe01161b0 0x7ffbe012ff30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:05:36.865 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:36.853+0000 7ffbe5206700 1 -- 172.21.15.42:0/3251942088 >> v1:172.21.15.40:6789/0 conn(0x7ffbe0118770 legacy=0x7ffbe01334f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:05:36.866 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:36.853+0000 7ffbe5206700 1 -- 172.21.15.42:0/3251942088 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbe008fa30 con 0x7ffbe0130480 2019-12-05T06:05:36.866 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:36.853+0000 7ffbe5a07700 1 --2- 172.21.15.42:0/3251942088 >> v2:172.21.15.40:3300/0 conn(0x7ffbe01161b0 0x7ffbe012ff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:05:36.866 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:36.853+0000 7ffbe5206700 1 --2- 172.21.15.42:0/3251942088 >> v2:172.21.15.42:3300/0 conn(0x7ffbe0130480 0x7ffbe0118220 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7ffbe005d360 tx=0x7ffbd4007e40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:36.866 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:36.853+0000 7ffbe5206700 1 --2- 172.21.15.42:0/3251942088 >> v2:172.21.15.42:3300/0 conn(0x7ffbe0130480 0x7ffbe0118220 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7ffbe005d360 tx=0x7ffbd4007e40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:36.866 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:36.853+0000 7ffbe5206700 1 --2- 172.21.15.42:0/3251942088 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ffbe0130480 0x7ffbe0118220 secure :-1 s=READY pgs=2027 cs=0 l=1 rx=0x7ffbe005d360 tx=0x7ffbd4007e40).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:05:36.866 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:36.853+0000 7ffbe5206700 1 --2- 172.21.15.42:0/3251942088 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ffbe0130480 0x7ffbe0118220 secure :-1 s=THROTTLE_DONE pgs=2027 cs=0 l=1 rx=0x7ffbe005d360 tx=0x7ffbd4007e40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:36.866 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:36.853+0000 7ffbd27fc700 1 -- 172.21.15.42:0/3251942088 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ffbd400f8d0 con 0x7ffbe0130480 2019-12-05T06:05:36.867 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:36.857+0000 7ffbe5206700 1 --2- 172.21.15.42:0/3251942088 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ffbe0130480 0x7ffbe0118220 secure :-1 s=THROTTLE_DONE pgs=2027 cs=0 l=1 rx=0x7ffbe005d360 tx=0x7ffbd4007e40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:36.867 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:36.857+0000 7ffbe5206700 1 --2- 172.21.15.42:0/3251942088 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ffbe0130480 0x7ffbe0118220 secure :-1 s=THROTTLE_DONE pgs=2027 cs=0 l=1 rx=0x7ffbe005d360 tx=0x7ffbd4007e40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:36.867 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:36.857+0000 7ffbd27fc700 1 -- 172.21.15.42:0/3251942088 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbd40116e0 con 0x7ffbe0130480 2019-12-05T06:05:36.867 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:36.857+0000 7ffbe746a700 1 -- 172.21.15.42:0/3251942088 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbe0119f60 con 0x7ffbe0130480 2019-12-05T06:05:36.867 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:36.857+0000 7ffbd27fc700 1 -- 172.21.15.42:0/3251942088 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ffbd4005ca0 con 0x7ffbe0130480 2019-12-05T06:05:36.867 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:36.857+0000 7ffbe746a700 1 -- 172.21.15.42:0/3251942088 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbe0118ba0 con 0x7ffbe0130480 2019-12-05T06:05:36.867 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:36.857+0000 7ffbe5206700 1 --2- 172.21.15.42:0/3251942088 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ffbe0130480 0x7ffbe0118220 secure :-1 s=THROTTLE_DONE pgs=2027 cs=0 l=1 rx=0x7ffbe005d360 tx=0x7ffbd4007e40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:36.868 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:36.857+0000 7ffbd27fc700 1 -- 172.21.15.42:0/3251942088 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7ffbd4007150 con 0x7ffbe0130480 2019-12-05T06:05:36.868 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:36.857+0000 7ffbe5206700 1 --2- 172.21.15.42:0/3251942088 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ffbe0130480 0x7ffbe0118220 secure :-1 s=THROTTLE_DONE pgs=2027 cs=0 l=1 rx=0x7ffbe005d360 tx=0x7ffbd4007e40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:36.868 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:36.857+0000 7ffbd27fc700 1 --2- 172.21.15.42:0/3251942088 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ffbc803adb0 0x7ffbc803d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:05:36.868 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:36.857+0000 7ffbd27fc700 1 -- 172.21.15.42:0/3251942088 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9729+0+0 (secure 0 0 0) 0x7ffbd404b550 con 0x7ffbe0130480 2019-12-05T06:05:36.868 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:36.857+0000 7ffbe5a07700 1 --2- 172.21.15.42:0/3251942088 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ffbc803adb0 0x7ffbc803d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:05:36.868 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:36.857+0000 7ffbe5a07700 1 --2- 172.21.15.42:0/3251942088 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ffbc803adb0 0x7ffbc803d2c0 crc :-1 s=READY pgs=858 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:05:36.869 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:36.865+0000 7ffbe746a700 1 -- 172.21.15.42:0/3251942088 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbe005d100 con 0x7ffbe0130480 2019-12-05T06:05:36.874 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:36.869+0000 7ffbe5206700 1 --2- 172.21.15.42:0/3251942088 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ffbe0130480 0x7ffbe0118220 secure :-1 s=THROTTLE_DONE pgs=2027 cs=0 l=1 rx=0x7ffbe005d360 tx=0x7ffbd4007e40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:36.874 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:36.869+0000 7ffbd27fc700 1 -- 172.21.15.42:0/3251942088 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7ffbe008fa30 con 0x7ffbe0130480 2019-12-05T06:05:37.044 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:37.037+0000 7ffbe746a700 1 -- 172.21.15.42:0/3251942088 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffbe005b8b0 con 0x7ffbe0130480 2019-12-05T06:05:37.044 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:37.037+0000 7ffbe5206700 1 --2- 172.21.15.42:0/3251942088 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ffbe0130480 0x7ffbe0118220 secure :-1 s=THROTTLE_DONE pgs=2027 cs=0 l=1 rx=0x7ffbe005d360 tx=0x7ffbd4007e40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:37.044 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:37.037+0000 7ffbd27fc700 1 -- 172.21.15.42:0/3251942088 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 247 v247) v1 ==== 95+0+1775 (secure 0 0 0) 0x7ffbd40301c0 con 0x7ffbe0130480 2019-12-05T06:05:37.044 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 247 2019-12-05T06:05:37.044 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:05:37.045 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":247,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":247,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:05:33.997161+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":92,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6821},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6821":{"gid":6821,"name":"a","rank":0,"incarnation":244,"state":"up:active","state_seq":23,"addr":"172.21.15.42:6835/2253267554","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2253267554},{"type":"v1","addr":"172.21.15.42:6835","nonce":2253267554}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:05:37.046 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:37.041+0000 7ffbe746a700 1 -- 172.21.15.42:0/3251942088 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ffbc803adb0 msgr2=0x7ffbc803d2c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:05:37.046 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:37.041+0000 7ffbe746a700 1 --2- 172.21.15.42:0/3251942088 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ffbc803adb0 0x7ffbc803d2c0 crc :-1 s=READY pgs=858 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:05:37.046 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:37.041+0000 7ffbe746a700 1 -- 172.21.15.42:0/3251942088 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ffbe0130480 msgr2=0x7ffbe0118220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:05:37.046 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:37.041+0000 7ffbe746a700 1 --2- 172.21.15.42:0/3251942088 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ffbe0130480 0x7ffbe0118220 secure :-1 s=READY pgs=2027 cs=0 l=1 rx=0x7ffbe005d360 tx=0x7ffbd4007e40).stop 2019-12-05T06:05:37.046 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:37.041+0000 7ffbe746a700 1 -- 172.21.15.42:0/3251942088 shutdown_connections 2019-12-05T06:05:37.046 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:37.041+0000 7ffbe746a700 1 --2- 172.21.15.42:0/3251942088 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ffbc803adb0 0x7ffbc803d2c0 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:05:37.047 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:37.041+0000 7ffbe746a700 1 --2- 172.21.15.42:0/3251942088 >> v2:172.21.15.40:3300/0 conn(0x7ffbe01161b0 0x7ffbe012ff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:05:37.047 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:37.041+0000 7ffbe746a700 1 --2- 172.21.15.42:0/3251942088 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ffbe0130480 0x7ffbe0118220 unknown :-1 s=CLOSED pgs=2027 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:05:37.047 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:37.041+0000 7ffbe746a700 1 -- 172.21.15.42:0/3251942088 shutdown_connections 2019-12-05T06:05:37.047 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:37.041+0000 7ffbe746a700 1 -- 172.21.15.42:0/3251942088 wait complete. 2019-12-05T06:05:37.047 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:37.041+0000 7ffbe746a700 1 -- 172.21.15.42:0/3251942088 >> 172.21.15.42:0/3251942088 conn(0x7ffbe008fd90 msgr2=0x7ffbe0144800 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:05:37.055 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2019-12-05T06:05:39.055 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:05:39.160 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:39.149+0000 7f8d55b82700 1 Processor -- start 2019-12-05T06:05:39.160 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:39.149+0000 7f8d55b82700 1 -- start start 2019-12-05T06:05:39.160 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:39.149+0000 7f8d55b82700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8d5008df80 0x7f8d5009f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:05:39.161 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:39.149+0000 7f8d55b82700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d5009fc60 con 0x7f8d500a7740 2019-12-05T06:05:39.161 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:39.149+0000 7f8d55b82700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d500a0710 con 0x7f8d5009f830 2019-12-05T06:05:39.161 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:39.149+0000 7f8d55b82700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d500a0b00 con 0x7f8d5008df80 2019-12-05T06:05:39.161 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:39.149+0000 7f8d54920700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f8d5009f830 0x7f8d5014efe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:48068/0 (socket says 172.21.15.42:48068) 2019-12-05T06:05:39.161 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:39.149+0000 7f8d54920700 1 -- 172.21.15.42:0/1952471545 learned_addr learned my addr 172.21.15.42:0/1952471545 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:05:39.161 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:39.149+0000 7f8d477fe700 1 --2- 172.21.15.42:0/1952471545 >> v2:172.21.15.40:3300/0 conn(0x7f8d5008df80 0x7f8d5009f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:05:39.161 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:39.149+0000 7f8d4f7fe700 1 -- 172.21.15.42:0/1952471545 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2172503228 0 0) 0x7f8d500a0710 con 0x7f8d5009f830 2019-12-05T06:05:39.162 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:39.149+0000 7f8d4f7fe700 1 -- 172.21.15.42:0/1952471545 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d3c0030b0 con 0x7f8d5009f830 2019-12-05T06:05:39.162 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:39.149+0000 7f8d4f7fe700 1 -- 172.21.15.42:0/1952471545 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3846492120 0 0) 0x7f8d3c0030b0 con 0x7f8d5009f830 2019-12-05T06:05:39.162 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:39.149+0000 7f8d4f7fe700 1 -- 172.21.15.42:0/1952471545 >> v2:172.21.15.40:3300/0 conn(0x7f8d5008df80 msgr2=0x7f8d5009f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:05:39.163 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:39.149+0000 7f8d4f7fe700 1 --2- 172.21.15.42:0/1952471545 >> v2:172.21.15.40:3300/0 conn(0x7f8d5008df80 0x7f8d5009f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:05:39.163 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:39.153+0000 7f8d4f7fe700 1 -- 172.21.15.42:0/1952471545 >> v1:172.21.15.40:6789/0 conn(0x7f8d500a7740 legacy=0x7f8d5007e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:05:39.163 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:39.153+0000 7f8d4f7fe700 1 -- 172.21.15.42:0/1952471545 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d5007ef20 con 0x7f8d5009f830 2019-12-05T06:05:39.163 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:39.153+0000 7f8d4f7fe700 1 -- 172.21.15.42:0/1952471545 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7f8d48001ad0 con 0x7f8d5009f830 2019-12-05T06:05:39.164 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:39.153+0000 7f8d4f7fe700 1 -- 172.21.15.42:0/1952471545 >> v1:172.21.15.42:6789/0 conn(0x7f8d5009f830 legacy=0x7f8d5014efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:05:39.164 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:39.153+0000 7f8d4f7fe700 1 --2- 172.21.15.42:0/1952471545 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8d3c0036e0 0x7f8d3c003b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:05:39.164 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:39.153+0000 7f8d4f7fe700 1 --2- 172.21.15.42:0/1952471545 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8d3c004420 0x7f8d3c004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:05:39.164 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:39.153+0000 7f8d4f7fe700 1 --2- 172.21.15.42:0/1952471545 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8d3c004da0 0x7f8d3c00b670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:05:39.164 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:39.153+0000 7f8d4f7fe700 1 -- 172.21.15.42:0/1952471545 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d500a0b00 con 0x7f8d3c0036e0 2019-12-05T06:05:39.164 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:39.153+0000 7f8d477fe700 1 --2- 172.21.15.42:0/1952471545 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8d3c004420 0x7f8d3c004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:05:39.165 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:39.153+0000 7f8d4f7fe700 1 -- 172.21.15.42:0/1952471545 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d3c04fc90 con 0x7f8d3c004420 2019-12-05T06:05:39.165 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:39.153+0000 7f8d4f7fe700 1 -- 172.21.15.42:0/1952471545 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d3c04fed0 con 0x7f8d3c004da0 2019-12-05T06:05:39.165 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:39.153+0000 7f8d4ffff700 1 --2- 172.21.15.42:0/1952471545 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8d3c0036e0 0x7f8d3c003b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:05:39.165 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:39.153+0000 7f8d54920700 1 --2- 172.21.15.42:0/1952471545 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8d3c004da0 0x7f8d3c00b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:05:39.165 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:39.153+0000 7f8d477fe700 1 -- 172.21.15.42:0/1952471545 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8d3c004da0 msgr2=0x7f8d3c00b670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:05:39.165 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:39.153+0000 7f8d477fe700 1 --2- 172.21.15.42:0/1952471545 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8d3c004da0 0x7f8d3c00b670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:05:39.165 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:39.153+0000 7f8d477fe700 1 -- 172.21.15.42:0/1952471545 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8d3c0036e0 msgr2=0x7f8d3c003b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:05:39.166 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:39.153+0000 7f8d477fe700 1 --2- 172.21.15.42:0/1952471545 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8d3c0036e0 0x7f8d3c003b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:05:39.166 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:39.153+0000 7f8d54920700 1 -- 172.21.15.42:0/1952471545 reap_dead start 2019-12-05T06:05:39.166 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:39.153+0000 7f8d477fe700 1 -- 172.21.15.42:0/1952471545 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d3c0501d0 con 0x7f8d3c004420 2019-12-05T06:05:39.166 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:39.153+0000 7f8d477fe700 1 --2- 172.21.15.42:0/1952471545 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8d3c004420 0x7f8d3c004850 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f8d38002f30 tx=0x7f8d38007c80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:39.166 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:39.153+0000 7f8d477fe700 1 --2- 172.21.15.42:0/1952471545 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8d3c004420 0x7f8d3c004850 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f8d38002f30 tx=0x7f8d38007c80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:39.166 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:39.153+0000 7f8d477fe700 1 --2- 172.21.15.42:0/1952471545 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8d3c004420 0x7f8d3c004850 secure :-1 s=READY pgs=2029 cs=0 l=1 rx=0x7f8d38002f30 tx=0x7f8d38007c80).ready entity=mon.1 client_cookie=e7930b2a2f3b77d0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:05:39.166 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:39.153+0000 7f8d477fe700 1 --2- 172.21.15.42:0/1952471545 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8d3c004420 0x7f8d3c004850 secure :-1 s=THROTTLE_DONE pgs=2029 cs=0 l=1 rx=0x7f8d38002f30 tx=0x7f8d38007c80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:39.167 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:39.153+0000 7f8d4f7fe700 1 -- 172.21.15.42:0/1952471545 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8d38010c90 con 0x7f8d3c004420 2019-12-05T06:05:39.173 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:39.153+0000 7f8d477fe700 1 --2- 172.21.15.42:0/1952471545 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8d3c004420 0x7f8d3c004850 secure :-1 s=THROTTLE_DONE pgs=2029 cs=0 l=1 rx=0x7f8d38002f30 tx=0x7f8d38007c80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:39.173 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:39.153+0000 7f8d4f7fe700 1 -- 172.21.15.42:0/1952471545 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d380108b0 con 0x7f8d3c004420 2019-12-05T06:05:39.173 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:39.153+0000 7f8d477fe700 1 --2- 172.21.15.42:0/1952471545 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8d3c004420 0x7f8d3c004850 secure :-1 s=THROTTLE_DONE pgs=2029 cs=0 l=1 rx=0x7f8d38002f30 tx=0x7f8d38007c80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:39.173 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:39.153+0000 7f8d4f7fe700 1 -- 172.21.15.42:0/1952471545 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8d38008bb0 con 0x7f8d3c004420 2019-12-05T06:05:39.173 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:39.153+0000 7f8d55b82700 1 -- 172.21.15.42:0/1952471545 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8d3c004420 msgr2=0x7f8d3c004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:05:39.173 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:39.153+0000 7f8d55b82700 1 --2- 172.21.15.42:0/1952471545 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8d3c004420 0x7f8d3c004850 secure :-1 s=READY pgs=2029 cs=0 l=1 rx=0x7f8d38002f30 tx=0x7f8d38007c80).stop 2019-12-05T06:05:39.173 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:39.153+0000 7f8d55b82700 1 -- 172.21.15.42:0/1952471545 shutdown_connections 2019-12-05T06:05:39.174 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:39.153+0000 7f8d55b82700 1 --2- 172.21.15.42:0/1952471545 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8d3c004420 0x7f8d3c004850 unknown :-1 s=CLOSED pgs=2029 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:05:39.174 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:39.153+0000 7f8d55b82700 1 -- 172.21.15.42:0/1952471545 shutdown_connections 2019-12-05T06:05:39.174 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:39.153+0000 7f8d55b82700 1 -- 172.21.15.42:0/1952471545 wait complete. 2019-12-05T06:05:39.174 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:39.153+0000 7f8d55b82700 1 -- 172.21.15.42:0/1952471545 >> 172.21.15.42:0/1952471545 conn(0x7f8d500aabe0 msgr2=0x7f8d500a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:05:39.178 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:39.157+0000 7f8d55b82700 1 Processor -- start 2019-12-05T06:05:39.178 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:39.157+0000 7f8d55b82700 1 -- start start 2019-12-05T06:05:39.179 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:39.157+0000 7f8d55b82700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8d5009f620 0x7f8d5011b710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:05:39.179 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:39.157+0000 7f8d55b82700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8d5011a770 0x7f8d5011aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:05:39.179 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:39.157+0000 7f8d55b82700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d5011bfd0 con 0x7f8d500a3470 2019-12-05T06:05:39.179 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:39.157+0000 7f8d55b82700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d5011a230 con 0x7f8d5011a770 2019-12-05T06:05:39.179 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:39.157+0000 7f8d55b82700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d5011a4d0 con 0x7f8d5009f620 2019-12-05T06:05:39.179 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:39.161+0000 7f8d4ffff700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8d5009f620 0x7f8d5011b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:05:39.180 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:39.161+0000 7f8d4ffff700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8d5009f620 0x7f8d5011b710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:48504/0 (socket says 172.21.15.42:48504) 2019-12-05T06:05:39.180 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:39.161+0000 7f8d477fe700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8d5011a770 0x7f8d5011aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:05:39.180 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:39.161+0000 7f8d4ffff700 1 -- 172.21.15.42:0/3370568692 learned_addr learned my addr 172.21.15.42:0/3370568692 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:05:39.180 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:39.161+0000 7f8d4ffff700 1 -- 172.21.15.42:0/3370568692 >> v2:172.21.15.40:3300/0 conn(0x7f8d5011a770 msgr2=0x7f8d5011aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:05:39.180 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:39.161+0000 7f8d4ffff700 1 --2- 172.21.15.42:0/3370568692 >> v2:172.21.15.40:3300/0 conn(0x7f8d5011a770 0x7f8d5011aba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:05:39.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:39.161+0000 7f8d4ffff700 1 -- 172.21.15.42:0/3370568692 >> v1:172.21.15.40:6789/0 conn(0x7f8d500a3470 legacy=0x7f8d50122aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:05:39.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:39.161+0000 7f8d4ffff700 1 -- 172.21.15.42:0/3370568692 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d5011bc60 con 0x7f8d5009f620 2019-12-05T06:05:39.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:39.161+0000 7f8d4ffff700 1 --2- 172.21.15.42:0/3370568692 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8d5009f620 0x7f8d5011b710 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f8d40000fe0 tx=0x7f8d40010a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:39.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:39.161+0000 7f8d4ffff700 1 --2- 172.21.15.42:0/3370568692 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8d5009f620 0x7f8d5011b710 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f8d40000fe0 tx=0x7f8d40010a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:39.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:39.161+0000 7f8d4ffff700 1 --2- 172.21.15.42:0/3370568692 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8d5009f620 0x7f8d5011b710 secure :-1 s=READY pgs=450 cs=0 l=1 rx=0x7f8d40000fe0 tx=0x7f8d40010a20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:05:39.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:39.161+0000 7f8d4ffff700 1 --2- 172.21.15.42:0/3370568692 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8d5009f620 0x7f8d5011b710 secure :-1 s=THROTTLE_DONE pgs=450 cs=0 l=1 rx=0x7f8d40000fe0 tx=0x7f8d40010a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:39.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:39.161+0000 7f8d4d7fa700 1 -- 172.21.15.42:0/3370568692 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8d4000c6c0 con 0x7f8d5009f620 2019-12-05T06:05:39.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:39.161+0000 7f8d4ffff700 1 --2- 172.21.15.42:0/3370568692 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8d5009f620 0x7f8d5011b710 secure :-1 s=THROTTLE_DONE pgs=450 cs=0 l=1 rx=0x7f8d40000fe0 tx=0x7f8d40010a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:39.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:39.161+0000 7f8d4ffff700 1 --2- 172.21.15.42:0/3370568692 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8d5009f620 0x7f8d5011b710 secure :-1 s=THROTTLE_DONE pgs=450 cs=0 l=1 rx=0x7f8d40000fe0 tx=0x7f8d40010a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:39.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:39.161+0000 7f8d4d7fa700 1 -- 172.21.15.42:0/3370568692 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d40001b60 con 0x7f8d5009f620 2019-12-05T06:05:39.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:39.161+0000 7f8d55b82700 1 -- 172.21.15.42:0/3370568692 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d50134910 con 0x7f8d5009f620 2019-12-05T06:05:39.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:39.161+0000 7f8d4d7fa700 1 -- 172.21.15.42:0/3370568692 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8d4000cca0 con 0x7f8d5009f620 2019-12-05T06:05:39.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:39.161+0000 7f8d55b82700 1 -- 172.21.15.42:0/3370568692 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d500a38a0 con 0x7f8d5009f620 2019-12-05T06:05:39.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:39.161+0000 7f8d4ffff700 1 --2- 172.21.15.42:0/3370568692 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8d5009f620 0x7f8d5011b710 secure :-1 s=THROTTLE_DONE pgs=450 cs=0 l=1 rx=0x7f8d40000fe0 tx=0x7f8d40010a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:39.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:39.161+0000 7f8d4d7fa700 1 -- 172.21.15.42:0/3370568692 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f8d4000d4b0 con 0x7f8d5009f620 2019-12-05T06:05:39.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:39.161+0000 7f8d55b82700 1 -- 172.21.15.42:0/3370568692 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d5005d110 con 0x7f8d5009f620 2019-12-05T06:05:39.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:39.161+0000 7f8d4ffff700 1 --2- 172.21.15.42:0/3370568692 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8d5009f620 0x7f8d5011b710 secure :-1 s=THROTTLE_DONE pgs=450 cs=0 l=1 rx=0x7f8d40000fe0 tx=0x7f8d40010a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:39.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:39.165+0000 7f8d4d7fa700 1 --2- 172.21.15.42:0/3370568692 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8d3403ad30 0x7f8d3403d240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:05:39.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:39.165+0000 7f8d4d7fa700 1 -- 172.21.15.42:0/3370568692 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9729+0+0 (secure 0 0 0) 0x7f8d4004b270 con 0x7f8d5009f620 2019-12-05T06:05:39.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:39.165+0000 7f8d477fe700 1 --2- 172.21.15.42:0/3370568692 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8d3403ad30 0x7f8d3403d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:05:39.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:39.165+0000 7f8d477fe700 1 --2- 172.21.15.42:0/3370568692 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8d3403ad30 0x7f8d3403d240 crc :-1 s=READY pgs=860 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:05:39.184 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:39.173+0000 7f8d4ffff700 1 --2- 172.21.15.42:0/3370568692 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8d5009f620 0x7f8d5011b710 secure :-1 s=THROTTLE_DONE pgs=450 cs=0 l=1 rx=0x7f8d40000fe0 tx=0x7f8d40010a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:39.184 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:39.173+0000 7f8d4d7fa700 1 -- 172.21.15.42:0/3370568692 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f8d400324d0 con 0x7f8d5009f620 2019-12-05T06:05:39.343 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:39.337+0000 7f8d55b82700 1 -- 172.21.15.42:0/3370568692 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8d500a8cb0 con 0x7f8d5009f620 2019-12-05T06:05:39.344 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:39.341+0000 7f8d4ffff700 1 --2- 172.21.15.42:0/3370568692 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8d5009f620 0x7f8d5011b710 secure :-1 s=THROTTLE_DONE pgs=450 cs=0 l=1 rx=0x7f8d40000fe0 tx=0x7f8d40010a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:05:39.344 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:39.341+0000 7f8d4d7fa700 1 -- 172.21.15.42:0/3370568692 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 248 v248) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f8d40033080 con 0x7f8d5009f620 2019-12-05T06:05:39.346 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:05:39.346 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":248,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":9573,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/2968227232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2968227232},{"type":"v1","addr":"172.21.15.40:6833","nonce":2968227232}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":248}],"filesystems":[{"mdsmap":{"epoch":247,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:05:33.997161+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":92,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6821},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6821":{"gid":6821,"name":"a","rank":0,"incarnation":244,"state":"up:active","state_seq":23,"addr":"172.21.15.42:6835/2253267554","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2253267554},{"type":"v1","addr":"172.21.15.42:6835","nonce":2253267554}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:05:39.347 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 248 2019-12-05T06:05:39.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:39.345+0000 7f8d55b82700 1 -- 172.21.15.42:0/3370568692 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8d3403ad30 msgr2=0x7f8d3403d240 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:05:39.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:39.345+0000 7f8d55b82700 1 --2- 172.21.15.42:0/3370568692 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8d3403ad30 0x7f8d3403d240 crc :-1 s=READY pgs=860 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:05:39.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:39.345+0000 7f8d55b82700 1 -- 172.21.15.42:0/3370568692 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8d5009f620 msgr2=0x7f8d5011b710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:05:39.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:39.345+0000 7f8d55b82700 1 --2- 172.21.15.42:0/3370568692 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8d5009f620 0x7f8d5011b710 secure :-1 s=READY pgs=450 cs=0 l=1 rx=0x7f8d40000fe0 tx=0x7f8d40010a20).stop 2019-12-05T06:05:39.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:39.345+0000 7f8d55b82700 1 -- 172.21.15.42:0/3370568692 shutdown_connections 2019-12-05T06:05:39.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:39.345+0000 7f8d55b82700 1 --2- 172.21.15.42:0/3370568692 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8d3403ad30 0x7f8d3403d240 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:05:39.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:39.345+0000 7f8d55b82700 1 --2- 172.21.15.42:0/3370568692 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8d5009f620 0x7f8d5011b710 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:05:39.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:39.345+0000 7f8d55b82700 1 --2- 172.21.15.42:0/3370568692 >> v2:172.21.15.40:3300/0 conn(0x7f8d5011a770 0x7f8d5011aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:05:39.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:39.345+0000 7f8d55b82700 1 -- 172.21.15.42:0/3370568692 shutdown_connections 2019-12-05T06:05:39.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:39.345+0000 7f8d55b82700 1 -- 172.21.15.42:0/3370568692 wait complete. 2019-12-05T06:05:39.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:05:39.345+0000 7f8d55b82700 1 -- 172.21.15.42:0/3370568692 >> 172.21.15.42:0/3370568692 conn(0x7f8d500a9800 msgr2=0x7f8d5008eb70 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:05:39.358 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b reported in up:standby state 2019-12-05T06:05:39.359 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 62 secs before thrashing 2019-12-05T06:05:43.698 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:05:43.702 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:06:13.746 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:06:13.750 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:06:41.364 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:06:41.474 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.464+0000 7fa2a8680700 1 Processor -- start 2019-12-05T06:06:41.474 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.464+0000 7fa2a8680700 1 -- start start 2019-12-05T06:06:41.474 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.464+0000 7fa2a8680700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa2a00a7740 0x7fa2a008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:06:41.475 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.464+0000 7fa2a8680700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fa2a007e720 0x7fa2a007eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:06:41.475 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.464+0000 7fa2a8680700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2a00a0710 con 0x7fa2a009f2e0 2019-12-05T06:06:41.475 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.464+0000 7fa2a8680700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2a00a0b00 con 0x7fa2a007e720 2019-12-05T06:06:41.475 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.464+0000 7fa2a8680700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2a008e5b0 con 0x7fa2a00a7740 2019-12-05T06:06:41.475 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.464+0000 7fa2a641c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fa2a007e720 0x7fa2a007eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:41.475 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.464+0000 7fa2a641c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fa2a007e720 0x7fa2a007eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:48298/0 (socket says 172.21.15.42:48298) 2019-12-05T06:06:41.475 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.464+0000 7fa2a641c700 1 -- 172.21.15.42:0/1773050274 learned_addr learned my addr 172.21.15.42:0/1773050274 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:06:41.476 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.464+0000 7fa2a6c1d700 1 --2- 172.21.15.42:0/1773050274 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa2a00a7740 0x7fa2a008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:41.476 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.464+0000 7fa2a641c700 1 -- 172.21.15.42:0/1773050274 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa2a00a7740 msgr2=0x7fa2a008df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:06:41.476 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.464+0000 7fa2a641c700 1 --2- 172.21.15.42:0/1773050274 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa2a00a7740 0x7fa2a008df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:06:41.477 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.464+0000 7fa2a641c700 1 -- 172.21.15.42:0/1773050274 >> v1:172.21.15.42:6789/0 conn(0x7fa2a009f2e0 legacy=0x7fa2a009f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:41.477 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.464+0000 7fa2a641c700 1 -- 172.21.15.42:0/1773050274 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2a007f0a0 con 0x7fa2a007e720 2019-12-05T06:06:41.477 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.464+0000 7fa2a641c700 1 --2- 172.21.15.42:0/1773050274 >> v2:172.21.15.42:3300/0 conn(0x7fa2a007e720 0x7fa2a007eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fa290008790 tx=0x7fa29000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:41.478 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.464+0000 7fa2a641c700 1 --2- 172.21.15.42:0/1773050274 >> v2:172.21.15.42:3300/0 conn(0x7fa2a007e720 0x7fa2a007eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fa290008790 tx=0x7fa29000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:41.478 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.464+0000 7fa2a641c700 1 --2- 172.21.15.42:0/1773050274 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa2a007e720 0x7fa2a007eb50 secure :-1 s=READY pgs=2031 cs=0 l=1 rx=0x7fa290008790 tx=0x7fa29000fd30).ready entity=mon.1 client_cookie=90841a3bdf9ea920 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:41.478 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.464+0000 7fa2a641c700 1 --2- 172.21.15.42:0/1773050274 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa2a007e720 0x7fa2a007eb50 secure :-1 s=THROTTLE_DONE pgs=2031 cs=0 l=1 rx=0x7fa290008790 tx=0x7fa29000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:41.478 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.464+0000 7fa2a5c1b700 1 -- 172.21.15.42:0/1773050274 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa2900113b0 con 0x7fa2a007e720 2019-12-05T06:06:41.478 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.468+0000 7fa2a641c700 1 --2- 172.21.15.42:0/1773050274 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa2a007e720 0x7fa2a007eb50 secure :-1 s=THROTTLE_DONE pgs=2031 cs=0 l=1 rx=0x7fa290008790 tx=0x7fa29000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:41.478 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.468+0000 7fa2a641c700 1 --2- 172.21.15.42:0/1773050274 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa2a007e720 0x7fa2a007eb50 secure :-1 s=THROTTLE_DONE pgs=2031 cs=0 l=1 rx=0x7fa290008790 tx=0x7fa29000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:41.479 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.468+0000 7fa2a5c1b700 1 -- 172.21.15.42:0/1773050274 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa290008bb0 con 0x7fa2a007e720 2019-12-05T06:06:41.479 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.468+0000 7fa2a5c1b700 1 -- 172.21.15.42:0/1773050274 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa290011990 con 0x7fa2a007e720 2019-12-05T06:06:41.479 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.468+0000 7fa2a8680700 1 -- 172.21.15.42:0/1773050274 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa2a007e720 msgr2=0x7fa2a007eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:41.479 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.468+0000 7fa2a8680700 1 --2- 172.21.15.42:0/1773050274 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa2a007e720 0x7fa2a007eb50 secure :-1 s=READY pgs=2031 cs=0 l=1 rx=0x7fa290008790 tx=0x7fa29000fd30).stop 2019-12-05T06:06:41.479 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.468+0000 7fa2a8680700 1 -- 172.21.15.42:0/1773050274 shutdown_connections 2019-12-05T06:06:41.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.468+0000 7fa2a8680700 1 --2- 172.21.15.42:0/1773050274 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa2a00a7740 0x7fa2a008df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:06:41.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.468+0000 7fa2a8680700 1 --2- 172.21.15.42:0/1773050274 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa2a007e720 0x7fa2a007eb50 unknown :-1 s=CLOSED pgs=2031 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:41.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.468+0000 7fa2a8680700 1 -- 172.21.15.42:0/1773050274 shutdown_connections 2019-12-05T06:06:41.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.468+0000 7fa2a8680700 1 -- 172.21.15.42:0/1773050274 wait complete. 2019-12-05T06:06:41.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.468+0000 7fa2a8680700 1 -- 172.21.15.42:0/1773050274 >> 172.21.15.42:0/1773050274 conn(0x7fa2a00aabe0 msgr2=0x7fa2a00a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:06:41.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.468+0000 7fa2a8680700 1 Processor -- start 2019-12-05T06:06:41.486 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.468+0000 7fa2a8680700 1 -- start start 2019-12-05T06:06:41.486 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.472+0000 7fa2a8680700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa2a0116500 0x7fa2a00a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:41.486 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.472+0000 7fa2a8680700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa2a007c710 0x7fa2a007cb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:41.486 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.472+0000 7fa2a8680700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2a00a5b30 con 0x7fa2a007be10 2019-12-05T06:06:41.487 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.472+0000 7fa2a8680700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2a007d090 con 0x7fa2a007c710 2019-12-05T06:06:41.488 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.472+0000 7fa2a8680700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2a0133880 con 0x7fa2a0116500 2019-12-05T06:06:41.488 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.472+0000 7fa2a6c1d700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa2a0116500 0x7fa2a00a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:41.488 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.472+0000 7fa2a6c1d700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa2a0116500 0x7fa2a00a51c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:48768/0 (socket says 172.21.15.42:48768) 2019-12-05T06:06:41.488 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.472+0000 7fa2a6c1d700 1 -- 172.21.15.42:0/568062459 learned_addr learned my addr 172.21.15.42:0/568062459 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:06:41.488 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.472+0000 7fa2a741e700 1 --2- 172.21.15.42:0/568062459 >> v2:172.21.15.40:3300/0 conn(0x7fa2a007c710 0x7fa2a007cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:41.488 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.472+0000 7fa2a741e700 1 -- 172.21.15.42:0/568062459 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa2a0116500 msgr2=0x7fa2a00a51c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:41.489 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.472+0000 7fa2a741e700 1 --2- 172.21.15.42:0/568062459 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa2a0116500 0x7fa2a00a51c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:41.489 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.472+0000 7fa2a741e700 1 -- 172.21.15.42:0/568062459 >> v1:172.21.15.40:6789/0 conn(0x7fa2a007be10 legacy=0x7fa2a01340e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:41.489 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.472+0000 7fa2a741e700 1 -- 172.21.15.42:0/568062459 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2a00752e0 con 0x7fa2a007c710 2019-12-05T06:06:41.490 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.472+0000 7fa2977fe700 1 -- 172.21.15.42:0/568062459 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2740857600 0 0) 0x7fa2a00a5b30 con 0x7fa2a007be10 2019-12-05T06:06:41.490 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.472+0000 7fa2a6c1d700 1 --2- 172.21.15.42:0/568062459 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa2a0116500 0x7fa2a00a51c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:06:41.490 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.472+0000 7fa2a741e700 1 --2- 172.21.15.42:0/568062459 >> v2:172.21.15.40:3300/0 conn(0x7fa2a007c710 0x7fa2a007cb40 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fa2a0133a40 tx=0x7fa298010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:41.490 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.472+0000 7fa2a741e700 1 --2- 172.21.15.42:0/568062459 >> v2:172.21.15.40:3300/0 conn(0x7fa2a007c710 0x7fa2a007cb40 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fa2a0133a40 tx=0x7fa298010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:41.490 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.472+0000 7fa2a741e700 1 --2- 172.21.15.42:0/568062459 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa2a007c710 0x7fa2a007cb40 secure :-1 s=READY pgs=1543 cs=0 l=1 rx=0x7fa2a0133a40 tx=0x7fa298010b40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:41.490 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.472+0000 7fa2a741e700 1 --2- 172.21.15.42:0/568062459 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa2a007c710 0x7fa2a007cb40 secure :-1 s=THROTTLE_DONE pgs=1543 cs=0 l=1 rx=0x7fa2a0133a40 tx=0x7fa298010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:41.491 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.472+0000 7fa2977fe700 1 -- 172.21.15.42:0/568062459 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa298006bd0 con 0x7fa2a007c710 2019-12-05T06:06:41.491 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.472+0000 7fa2a741e700 1 --2- 172.21.15.42:0/568062459 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa2a007c710 0x7fa2a007cb40 secure :-1 s=THROTTLE_DONE pgs=1543 cs=0 l=1 rx=0x7fa2a0133a40 tx=0x7fa298010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:41.491 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.472+0000 7fa2977fe700 1 -- 172.21.15.42:0/568062459 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa298013060 con 0x7fa2a007c710 2019-12-05T06:06:41.491 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.472+0000 7fa2a741e700 1 --2- 172.21.15.42:0/568062459 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa2a007c710 0x7fa2a007cb40 secure :-1 s=THROTTLE_DONE pgs=1543 cs=0 l=1 rx=0x7fa2a0133a40 tx=0x7fa298010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:41.491 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.472+0000 7fa2a8680700 1 -- 172.21.15.42:0/568062459 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2a00742d0 con 0x7fa2a007c710 2019-12-05T06:06:41.491 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.472+0000 7fa2977fe700 1 -- 172.21.15.42:0/568062459 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa298002410 con 0x7fa2a007c710 2019-12-05T06:06:41.492 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.472+0000 7fa2a8680700 1 -- 172.21.15.42:0/568062459 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2a0074850 con 0x7fa2a007c710 2019-12-05T06:06:41.492 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.472+0000 7fa2a741e700 1 -- 172.21.15.42:0/568062459 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa2a007c710 msgr2=0x7fa2a007cb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2019-12-05T06:06:41.492 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.472+0000 7fa2a741e700 1 -- 172.21.15.42:0/568062459 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa2a007c710 msgr2=0x7fa2a007cb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2019-12-05T06:06:41.492 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.472+0000 7fa2a741e700 1 --2- 172.21.15.42:0/568062459 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa2a007c710 0x7fa2a007cb40 secure :-1 s=READY pgs=1543 cs=0 l=1 rx=0x7fa2a0133a40 tx=0x7fa298010b40).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-12-05T06:06:41.492 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.472+0000 7fa2a741e700 1 --2- 172.21.15.42:0/568062459 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa2a007c710 0x7fa2a007cb40 secure :-1 s=READY pgs=1543 cs=0 l=1 rx=0x7fa2a0133a40 tx=0x7fa298010b40).stop 2019-12-05T06:06:41.493 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.472+0000 7fa2977fe700 1 -- 172.21.15.42:0/568062459 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa2a007c710 msgr2=0x7fa2a007cb40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2019-12-05T06:06:41.493 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.472+0000 7fa2977fe700 1 --2- 172.21.15.42:0/568062459 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa2a007c710 0x7fa2a007cb40 unknown :-1 s=CLOSED pgs=1543 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:41.493 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.472+0000 7fa2977fe700 1 --2- 172.21.15.42:0/568062459 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa288002fd0 0x7fa288005450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:41.493 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.472+0000 7fa2977fe700 1 --2- 172.21.15.42:0/568062459 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa288005a50 0x7fa288007f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:41.493 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.472+0000 7fa2977fe700 1 --2- 172.21.15.42:0/568062459 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa288008690 0x7fa28800ab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:41.493 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.472+0000 7fa2977fe700 1 -- 172.21.15.42:0/568062459 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa28800b110 con 0x7fa288002fd0 2019-12-05T06:06:41.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.472+0000 7fa2977fe700 1 -- 172.21.15.42:0/568062459 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa28800b300 con 0x7fa288005a50 2019-12-05T06:06:41.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.472+0000 7fa2977fe700 1 -- 172.21.15.42:0/568062459 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa28800b540 con 0x7fa288008690 2019-12-05T06:06:41.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.472+0000 7fa2a641c700 1 --2- 172.21.15.42:0/568062459 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa288005a50 0x7fa288007f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:41.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.472+0000 7fa2a6c1d700 1 --2- 172.21.15.42:0/568062459 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa288002fd0 0x7fa288005450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:41.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.472+0000 7fa2a741e700 1 --2- 172.21.15.42:0/568062459 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa288008690 0x7fa28800ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:41.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.472+0000 7fa2a6c1d700 1 -- 172.21.15.42:0/568062459 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa288008690 msgr2=0x7fa28800ab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:41.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.472+0000 7fa2a6c1d700 1 --2- 172.21.15.42:0/568062459 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa288008690 0x7fa28800ab10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:41.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.472+0000 7fa2a6c1d700 1 -- 172.21.15.42:0/568062459 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa288005a50 msgr2=0x7fa288007f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:41.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.472+0000 7fa2a6c1d700 1 --2- 172.21.15.42:0/568062459 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa288005a50 0x7fa288007f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:41.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.472+0000 7fa2a6c1d700 1 -- 172.21.15.42:0/568062459 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+,osdmap=0}) v3 -- 0x7fa28800b8a0 con 0x7fa288002fd0 2019-12-05T06:06:41.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.472+0000 7fa2a6c1d700 1 --2- 172.21.15.42:0/568062459 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa288002fd0 0x7fa288005450 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fa29c003070 tx=0x7fa29c010cf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:41.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.472+0000 7fa2a6c1d700 1 --2- 172.21.15.42:0/568062459 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa288002fd0 0x7fa288005450 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fa29c003070 tx=0x7fa29c010cf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:41.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.472+0000 7fa2a6c1d700 1 --2- 172.21.15.42:0/568062459 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa288002fd0 0x7fa288005450 secure :-1 s=READY pgs=1544 cs=0 l=1 rx=0x7fa29c003070 tx=0x7fa29c010cf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:41.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.476+0000 7fa2a6c1d700 1 --2- 172.21.15.42:0/568062459 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa288002fd0 0x7fa288005450 secure :-1 s=THROTTLE_DONE pgs=1544 cs=0 l=1 rx=0x7fa29c003070 tx=0x7fa29c010cf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:41.496 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.476+0000 7fa2977fe700 1 -- 172.21.15.42:0/568062459 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa29c012a50 con 0x7fa288002fd0 2019-12-05T06:06:41.496 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.476+0000 7fa2a6c1d700 1 --2- 172.21.15.42:0/568062459 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa288002fd0 0x7fa288005450 secure :-1 s=THROTTLE_DONE pgs=1544 cs=0 l=1 rx=0x7fa29c003070 tx=0x7fa29c010cf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:41.496 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.476+0000 7fa2977fe700 1 -- 172.21.15.42:0/568062459 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa29c012670 con 0x7fa288002fd0 2019-12-05T06:06:41.496 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.476+0000 7fa2a6c1d700 1 --2- 172.21.15.42:0/568062459 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa288002fd0 0x7fa288005450 secure :-1 s=THROTTLE_DONE pgs=1544 cs=0 l=1 rx=0x7fa29c003070 tx=0x7fa29c010cf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:41.496 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.476+0000 7fa2977fe700 1 -- 172.21.15.42:0/568062459 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fa29c012c90 con 0x7fa288002fd0 2019-12-05T06:06:41.496 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.476+0000 7fa2a6c1d700 1 --2- 172.21.15.42:0/568062459 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa288002fd0 0x7fa288005450 secure :-1 s=THROTTLE_DONE pgs=1544 cs=0 l=1 rx=0x7fa29c003070 tx=0x7fa29c010cf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:41.497 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.476+0000 7fa2977fe700 1 --2- 172.21.15.42:0/568062459 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa2880438d0 0x7fa288045de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:41.497 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.476+0000 7fa2977fe700 1 -- 172.21.15.42:0/568062459 <== mon.0 v2:172.21.15.40:3300/0 4 ==== osd_map(92..92 src has 1..92) v4 ==== 9729+0+0 (secure 0 0 0) 0x7fa29c04a910 con 0x7fa288002fd0 2019-12-05T06:06:41.497 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.476+0000 7fa2a641c700 1 --2- 172.21.15.42:0/568062459 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa2880438d0 0x7fa288045de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:41.497 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.476+0000 7fa2a641c700 1 --2- 172.21.15.42:0/568062459 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa2880438d0 0x7fa288045de0 crc :-1 s=READY pgs=862 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:41.498 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.480+0000 7fa2a8680700 1 -- 172.21.15.42:0/568062459 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2a005d110 con 0x7fa288002fd0 2019-12-05T06:06:41.498 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.488+0000 7fa2a6c1d700 1 --2- 172.21.15.42:0/568062459 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa288002fd0 0x7fa288005450 secure :-1 s=THROTTLE_DONE pgs=1544 cs=0 l=1 rx=0x7fa29c003070 tx=0x7fa29c010cf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:41.498 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.488+0000 7fa2977fe700 1 -- 172.21.15.42:0/568062459 <== mon.0 v2:172.21.15.40:3300/0 5 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fa29c031990 con 0x7fa288002fd0 2019-12-05T06:06:41.689 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.676+0000 7fa2a8680700 1 -- 172.21.15.42:0/568062459 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa2a005b8c0 con 0x7fa288002fd0 2019-12-05T06:06:41.690 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.680+0000 7fa2a6c1d700 1 --2- 172.21.15.42:0/568062459 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa288002fd0 0x7fa288005450 secure :-1 s=THROTTLE_DONE pgs=1544 cs=0 l=1 rx=0x7fa29c003070 tx=0x7fa29c010cf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:41.690 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.680+0000 7fa2977fe700 1 -- 172.21.15.42:0/568062459 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 248 v248) v1 ==== 95+0+2112 (secure 0 0 0) 0x7fa29c031bf0 con 0x7fa288002fd0 2019-12-05T06:06:41.690 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 248 2019-12-05T06:06:41.690 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.684+0000 7fa2a8680700 1 -- 172.21.15.42:0/568062459 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa2880438d0 msgr2=0x7fa288045de0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:41.691 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.684+0000 7fa2a8680700 1 --2- 172.21.15.42:0/568062459 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa2880438d0 0x7fa288045de0 crc :-1 s=READY pgs=862 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:41.691 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.684+0000 7fa2a8680700 1 -- 172.21.15.42:0/568062459 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa288002fd0 msgr2=0x7fa288005450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:41.691 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.684+0000 7fa2a8680700 1 --2- 172.21.15.42:0/568062459 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa288002fd0 0x7fa288005450 secure :-1 s=READY pgs=1544 cs=0 l=1 rx=0x7fa29c003070 tx=0x7fa29c010cf0).stop 2019-12-05T06:06:41.691 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.684+0000 7fa2a741e700 1 -- 172.21.15.42:0/568062459 reap_dead start 2019-12-05T06:06:41.691 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.684+0000 7fa2a741e700 1 -- 172.21.15.42:0/568062459 reap_dead start 2019-12-05T06:06:41.691 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.684+0000 7fa2a8680700 1 -- 172.21.15.42:0/568062459 shutdown_connections 2019-12-05T06:06:41.692 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.684+0000 7fa2a8680700 1 --2- 172.21.15.42:0/568062459 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa2a007c710 0x7fa2a007cb40 unknown :-1 s=CLOSED pgs=1543 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:41.692 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.684+0000 7fa2a8680700 1 -- 172.21.15.42:0/568062459 shutdown_connections 2019-12-05T06:06:41.692 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.684+0000 7fa2a8680700 1 -- 172.21.15.42:0/568062459 wait complete. 2019-12-05T06:06:41.692 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.684+0000 7fa2a8680700 1 -- 172.21.15.42:0/568062459 >> 172.21.15.42:0/568062459 conn(0x7fa2a00a9870 msgr2=0x7fa2a011c610 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:06:41.692 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:06:41.692 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":248,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":9573,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/2968227232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2968227232},{"type":"v1","addr":"172.21.15.40:6833","nonce":2968227232}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":248}],"filesystems":[{"mdsmap":{"epoch":247,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:05:33.997161+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":92,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6821},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6821":{"gid":6821,"name":"a","rank":0,"incarnation":244,"state":"up:active","state_seq":23,"addr":"172.21.15.42:6835/2253267554","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2253267554},{"type":"v1","addr":"172.21.15.42:6835","nonce":2253267554}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:06:41.700 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.a (rank=0) 2019-12-05T06:06:41.700 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2019-12-05T06:06:41.700 INFO:teuthology.orchestra.run:waiting for 300 2019-12-05T06:06:41.709 INFO:tasks.ceph.mds.a:Stopped 2019-12-05T06:06:41.710 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:06:41.882 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.868+0000 7f427cbbc700 1 Processor -- start 2019-12-05T06:06:41.882 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.868+0000 7f427cbbc700 1 -- start start 2019-12-05T06:06:41.883 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.872+0000 7f427cbbc700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f427808df80 0x7f427809f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:06:41.883 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.872+0000 7f427cbbc700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f427809f830 0x7f427814efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:06:41.883 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.872+0000 7f427cbbc700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f427809fc60 con 0x7f42780a7740 2019-12-05T06:06:41.883 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.872+0000 7f427cbbc700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42780a0b00 con 0x7f427808df80 2019-12-05T06:06:41.883 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.872+0000 7f427cbbc700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f427808e5b0 con 0x7f427809f830 2019-12-05T06:06:41.883 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.872+0000 7f426ffff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f427808df80 0x7f427809f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:41.883 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.872+0000 7f426ffff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f427808df80 0x7f427809f2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:48320/0 (socket says 172.21.15.42:48320) 2019-12-05T06:06:41.884 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.872+0000 7f426ffff700 1 -- 172.21.15.42:0/2080559363 learned_addr learned my addr 172.21.15.42:0/2080559363 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:06:41.884 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.872+0000 7f427759e700 1 --2- 172.21.15.42:0/2080559363 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f427809f830 0x7f427814efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:41.884 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.872+0000 7f426ffff700 1 -- 172.21.15.42:0/2080559363 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f427809f830 msgr2=0x7f427814efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:06:41.884 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.872+0000 7f426ffff700 1 --2- 172.21.15.42:0/2080559363 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f427809f830 0x7f427814efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:06:41.885 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.872+0000 7f426ffff700 1 -- 172.21.15.42:0/2080559363 >> v1:172.21.15.40:6789/0 conn(0x7f42780a7740 legacy=0x7f427807e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:41.885 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.872+0000 7f426ffff700 1 -- 172.21.15.42:0/2080559363 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42780a0710 con 0x7f427808df80 2019-12-05T06:06:41.885 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.872+0000 7f426ffff700 1 --2- 172.21.15.42:0/2080559363 >> v2:172.21.15.42:3300/0 conn(0x7f427808df80 0x7f427809f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f4260008790 tx=0x7f4268001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:41.885 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.872+0000 7f426ffff700 1 --2- 172.21.15.42:0/2080559363 >> v2:172.21.15.42:3300/0 conn(0x7f427808df80 0x7f427809f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f4260008790 tx=0x7f4268001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:41.886 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.872+0000 7f426ffff700 1 --2- 172.21.15.42:0/2080559363 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f427808df80 0x7f427809f2e0 secure :-1 s=READY pgs=2032 cs=0 l=1 rx=0x7f4260008790 tx=0x7f4268001150).ready entity=mon.1 client_cookie=3edd903c8cdbac6b server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:41.886 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.872+0000 7f426ffff700 1 --2- 172.21.15.42:0/2080559363 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f427808df80 0x7f427809f2e0 secure :-1 s=THROTTLE_DONE pgs=2032 cs=0 l=1 rx=0x7f4260008790 tx=0x7f4268001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:41.887 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.872+0000 7f426ffff700 1 --2- 172.21.15.42:0/2080559363 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f427808df80 0x7f427809f2e0 secure :-1 s=THROTTLE_DONE pgs=2032 cs=0 l=1 rx=0x7f4260008790 tx=0x7f4268001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:41.887 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.872+0000 7f427659c700 1 -- 172.21.15.42:0/2080559363 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4260008c90 con 0x7f427808df80 2019-12-05T06:06:41.887 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.872+0000 7f426ffff700 1 --2- 172.21.15.42:0/2080559363 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f427808df80 0x7f427809f2e0 secure :-1 s=THROTTLE_DONE pgs=2032 cs=0 l=1 rx=0x7f4260008790 tx=0x7f4268001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:41.887 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.872+0000 7f427659c700 1 -- 172.21.15.42:0/2080559363 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42600088b0 con 0x7f427808df80 2019-12-05T06:06:41.887 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.872+0000 7f427659c700 1 -- 172.21.15.42:0/2080559363 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4260011a10 con 0x7f427808df80 2019-12-05T06:06:41.888 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.872+0000 7f427cbbc700 1 -- 172.21.15.42:0/2080559363 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f427808df80 msgr2=0x7f427809f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:41.888 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.872+0000 7f427cbbc700 1 --2- 172.21.15.42:0/2080559363 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f427808df80 0x7f427809f2e0 secure :-1 s=READY pgs=2032 cs=0 l=1 rx=0x7f4260008790 tx=0x7f4268001150).stop 2019-12-05T06:06:41.888 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.876+0000 7f427cbbc700 1 -- 172.21.15.42:0/2080559363 shutdown_connections 2019-12-05T06:06:41.888 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.876+0000 7f427cbbc700 1 --2- 172.21.15.42:0/2080559363 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f427809f830 0x7f427814efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:06:41.889 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.876+0000 7f427cbbc700 1 --2- 172.21.15.42:0/2080559363 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f427808df80 0x7f427809f2e0 unknown :-1 s=CLOSED pgs=2032 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:41.889 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.876+0000 7f427cbbc700 1 -- 172.21.15.42:0/2080559363 shutdown_connections 2019-12-05T06:06:41.890 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.876+0000 7f427cbbc700 1 -- 172.21.15.42:0/2080559363 wait complete. 2019-12-05T06:06:41.890 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.876+0000 7f427cbbc700 1 -- 172.21.15.42:0/2080559363 >> 172.21.15.42:0/2080559363 conn(0x7f42780aabe0 msgr2=0x7f42780a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:06:41.910 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.876+0000 7f427cbbc700 1 Processor -- start 2019-12-05T06:06:41.910 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.876+0000 7f427cbbc700 1 -- start start 2019-12-05T06:06:41.910 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.880+0000 7f427cbbc700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4278120550 0x7f4278077d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:41.910 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.880+0000 7f427cbbc700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f4278078260 0x7f4278117850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:41.911 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.880+0000 7f427cbbc700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4278117da0 con 0x7f427811b930 2019-12-05T06:06:41.926 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.880+0000 7f427cbbc700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f427811be30 con 0x7f4278078260 2019-12-05T06:06:41.927 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.880+0000 7f427cbbc700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f427811c0d0 con 0x7f4278120550 2019-12-05T06:06:41.927 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.880+0000 7f4276d9d700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f427811b930 0x7f427811a930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:48362/0 (socket says 172.21.15.42:48362) 2019-12-05T06:06:41.927 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.880+0000 7f4276d9d700 1 -- 172.21.15.42:0/1055334167 learned_addr learned my addr 172.21.15.42:0/1055334167 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:06:41.928 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.880+0000 7f426ffff700 1 --2- 172.21.15.42:0/1055334167 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4278120550 0x7f4278077d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:41.928 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.880+0000 7f427759e700 1 --2- 172.21.15.42:0/1055334167 >> v2:172.21.15.40:3300/0 conn(0x7f4278078260 0x7f4278117850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:41.928 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.880+0000 7f426f7fe700 1 -- 172.21.15.42:0/1055334167 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1360258500 0 0) 0x7f4278117da0 con 0x7f427811b930 2019-12-05T06:06:41.928 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.880+0000 7f426f7fe700 1 -- 172.21.15.42:0/1055334167 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f425c0030b0 con 0x7f427811b930 2019-12-05T06:06:41.932 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.880+0000 7f426ffff700 1 -- 172.21.15.42:0/1055334167 >> v2:172.21.15.40:3300/0 conn(0x7f4278078260 msgr2=0x7f4278117850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:41.932 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.880+0000 7f426ffff700 1 --2- 172.21.15.42:0/1055334167 >> v2:172.21.15.40:3300/0 conn(0x7f4278078260 0x7f4278117850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:41.932 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.880+0000 7f426ffff700 1 -- 172.21.15.42:0/1055334167 >> v1:172.21.15.42:6789/0 conn(0x7f427811b930 legacy=0x7f427811a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:41.932 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.880+0000 7f426ffff700 1 -- 172.21.15.42:0/1055334167 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4278117fd0 con 0x7f4278120550 2019-12-05T06:06:41.933 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.880+0000 7f427759e700 1 --2- 172.21.15.42:0/1055334167 >> v2:172.21.15.40:3300/0 conn(0x7f4278078260 0x7f4278117850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:06:41.933 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.880+0000 7f426ffff700 1 --2- 172.21.15.42:0/1055334167 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4278120550 0x7f4278077d10 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f4260008790 tx=0x7f4260008b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:41.933 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.880+0000 7f426ffff700 1 --2- 172.21.15.42:0/1055334167 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4278120550 0x7f4278077d10 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f4260008790 tx=0x7f4260008b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:41.933 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.880+0000 7f426ffff700 1 --2- 172.21.15.42:0/1055334167 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4278120550 0x7f4278077d10 secure :-1 s=READY pgs=451 cs=0 l=1 rx=0x7f4260008790 tx=0x7f4260008b60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:41.933 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.880+0000 7f426ffff700 1 --2- 172.21.15.42:0/1055334167 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4278120550 0x7f4278077d10 secure :-1 s=THROTTLE_DONE pgs=451 cs=0 l=1 rx=0x7f4260008790 tx=0x7f4260008b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:41.934 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.880+0000 7f426f7fe700 1 -- 172.21.15.42:0/1055334167 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f426000f470 con 0x7f4278120550 2019-12-05T06:06:41.934 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.880+0000 7f426ffff700 1 --2- 172.21.15.42:0/1055334167 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4278120550 0x7f4278077d10 secure :-1 s=THROTTLE_DONE pgs=451 cs=0 l=1 rx=0x7f4260008790 tx=0x7f4260008b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:41.934 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.880+0000 7f427cbbc700 1 -- 172.21.15.42:0/1055334167 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4278134830 con 0x7f4278120550 2019-12-05T06:06:41.934 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.880+0000 7f426f7fe700 1 -- 172.21.15.42:0/1055334167 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4260011630 con 0x7f4278120550 2019-12-05T06:06:41.934 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.880+0000 7f426ffff700 1 --2- 172.21.15.42:0/1055334167 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4278120550 0x7f4278077d10 secure :-1 s=THROTTLE_DONE pgs=451 cs=0 l=1 rx=0x7f4260008790 tx=0x7f4260008b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:41.934 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.880+0000 7f426f7fe700 1 -- 172.21.15.42:0/1055334167 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4260005730 con 0x7f4278120550 2019-12-05T06:06:41.934 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.880+0000 7f427cbbc700 1 -- 172.21.15.42:0/1055334167 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4278134d30 con 0x7f4278120550 2019-12-05T06:06:41.935 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.884+0000 7f426ffff700 1 --2- 172.21.15.42:0/1055334167 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4278120550 0x7f4278077d10 secure :-1 s=THROTTLE_DONE pgs=451 cs=0 l=1 rx=0x7f4260008790 tx=0x7f4260008b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:41.935 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.884+0000 7f426f7fe700 1 -- 172.21.15.42:0/1055334167 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f42600130b0 con 0x7f4278120550 2019-12-05T06:06:41.935 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.884+0000 7f426ffff700 1 --2- 172.21.15.42:0/1055334167 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4278120550 0x7f4278077d10 secure :-1 s=THROTTLE_DONE pgs=451 cs=0 l=1 rx=0x7f4260008790 tx=0x7f4260008b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:41.935 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.884+0000 7f426f7fe700 1 --2- 172.21.15.42:0/1055334167 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f425c03b210 0x7f425c03d720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:41.935 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.884+0000 7f426f7fe700 1 -- 172.21.15.42:0/1055334167 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9729+0+0 (secure 0 0 0) 0x7f426004b670 con 0x7f4278120550 2019-12-05T06:06:41.935 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.884+0000 7f4276d9d700 1 --2- 172.21.15.42:0/1055334167 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f425c03b210 0x7f425c03d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:41.935 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.884+0000 7f4276d9d700 1 --2- 172.21.15.42:0/1055334167 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f425c03b210 0x7f425c03d720 crc :-1 s=READY pgs=863 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:41.936 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.884+0000 7f427cbbc700 1 -- 172.21.15.42:0/1055334167 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f427805d110 con 0x7f4278120550 2019-12-05T06:06:41.936 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.896+0000 7f426ffff700 1 --2- 172.21.15.42:0/1055334167 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4278120550 0x7f4278077d10 secure :-1 s=THROTTLE_DONE pgs=451 cs=0 l=1 rx=0x7f4260008790 tx=0x7f4260008b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:41.936 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:41.896+0000 7f426f7fe700 1 -- 172.21.15.42:0/1055334167 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f4260008de0 con 0x7f4278120550 2019-12-05T06:06:42.066 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:42.060+0000 7f427cbbc700 1 -- 172.21.15.42:0/1055334167 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f427805b8c0 con 0x7f4278120550 2019-12-05T06:06:42.067 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:42.060+0000 7f426ffff700 1 --2- 172.21.15.42:0/1055334167 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4278120550 0x7f4278077d10 secure :-1 s=THROTTLE_DONE pgs=451 cs=0 l=1 rx=0x7f4260008790 tx=0x7f4260008b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:42.067 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:42.060+0000 7f426f7fe700 1 -- 172.21.15.42:0/1055334167 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 248 v248) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f4260032930 con 0x7f4278120550 2019-12-05T06:06:42.069 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 248 2019-12-05T06:06:42.070 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:06:42.070 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":248,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":9573,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/2968227232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2968227232},{"type":"v1","addr":"172.21.15.40:6833","nonce":2968227232}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":248}],"filesystems":[{"mdsmap":{"epoch":247,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:05:33.997161+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":92,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6821},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6821":{"gid":6821,"name":"a","rank":0,"incarnation":244,"state":"up:active","state_seq":23,"addr":"172.21.15.42:6835/2253267554","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2253267554},{"type":"v1","addr":"172.21.15.42:6835","nonce":2253267554}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:06:42.073 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:42.068+0000 7f427cbbc700 1 -- 172.21.15.42:0/1055334167 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f425c03b210 msgr2=0x7f425c03d720 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:42.074 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:42.068+0000 7f427cbbc700 1 --2- 172.21.15.42:0/1055334167 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f425c03b210 0x7f425c03d720 crc :-1 s=READY pgs=863 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:42.074 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:42.068+0000 7f427cbbc700 1 -- 172.21.15.42:0/1055334167 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4278120550 msgr2=0x7f4278077d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:42.074 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:42.068+0000 7f427cbbc700 1 --2- 172.21.15.42:0/1055334167 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4278120550 0x7f4278077d10 secure :-1 s=READY pgs=451 cs=0 l=1 rx=0x7f4260008790 tx=0x7f4260008b60).stop 2019-12-05T06:06:42.076 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:42.068+0000 7f427cbbc700 1 -- 172.21.15.42:0/1055334167 shutdown_connections 2019-12-05T06:06:42.076 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:42.068+0000 7f427cbbc700 1 --2- 172.21.15.42:0/1055334167 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f425c03b210 0x7f425c03d720 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:42.076 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:42.068+0000 7f427cbbc700 1 --2- 172.21.15.42:0/1055334167 >> v2:172.21.15.40:3300/0 conn(0x7f4278078260 0x7f4278117850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:42.076 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:42.068+0000 7f427cbbc700 1 --2- 172.21.15.42:0/1055334167 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4278120550 0x7f4278077d10 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:42.076 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:42.068+0000 7f427cbbc700 1 -- 172.21.15.42:0/1055334167 shutdown_connections 2019-12-05T06:06:42.077 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:42.068+0000 7f427cbbc700 1 -- 172.21.15.42:0/1055334167 wait complete. 2019-12-05T06:06:42.077 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:42.068+0000 7f427cbbc700 1 -- 172.21.15.42:0/1055334167 >> 172.21.15.42:0/1055334167 conn(0x7f42780a8250 msgr2=0x7f4278059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:06:42.102 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:06:43.794 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:06:43.809 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:06:44.103 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:06:44.211 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:44.200+0000 7f6be76ed700 1 Processor -- start 2019-12-05T06:06:44.211 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:44.200+0000 7f6be76ed700 1 -- start start 2019-12-05T06:06:44.212 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:44.200+0000 7f6be76ed700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f6be009fae0 0x7f6be014fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:06:44.212 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:44.204+0000 7f6be76ed700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6be00a0710 con 0x7f6be00a7740 2019-12-05T06:06:44.212 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:44.204+0000 7f6be76ed700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6be007ef20 con 0x7f6be008df80 2019-12-05T06:06:44.212 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:44.204+0000 7f6be76ed700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6be00a0b00 con 0x7f6be009fae0 2019-12-05T06:06:44.212 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:44.204+0000 7f6be5489700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f6be008df80 0x7f6be009f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:48378/0 (socket says 172.21.15.42:48378) 2019-12-05T06:06:44.212 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:44.204+0000 7f6be5489700 1 -- 172.21.15.42:0/443371415 learned_addr learned my addr 172.21.15.42:0/443371415 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:06:44.213 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:44.204+0000 7f6be648b700 1 --2- 172.21.15.42:0/443371415 >> v2:172.21.15.40:3300/0 conn(0x7f6be009fae0 0x7f6be014fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:44.213 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:44.204+0000 7f6be4c88700 1 -- 172.21.15.42:0/443371415 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1858572263 0 0) 0x7f6be007ef20 con 0x7f6be008df80 2019-12-05T06:06:44.213 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:44.204+0000 7f6be4c88700 1 -- 172.21.15.42:0/443371415 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6bcc0030b0 con 0x7f6be008df80 2019-12-05T06:06:44.213 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:44.204+0000 7f6be648b700 1 -- 172.21.15.42:0/443371415 >> v1:172.21.15.42:6789/0 conn(0x7f6be008df80 legacy=0x7f6be009f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:44.213 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:44.204+0000 7f6be648b700 1 -- 172.21.15.42:0/443371415 >> v1:172.21.15.40:6789/0 conn(0x7f6be00a7740 legacy=0x7f6be007e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:44.213 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:44.204+0000 7f6be648b700 1 -- 172.21.15.42:0/443371415 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6be00a9ba0 con 0x7f6be009fae0 2019-12-05T06:06:44.213 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:44.204+0000 7f6be648b700 1 --2- 172.21.15.42:0/443371415 >> v2:172.21.15.40:3300/0 conn(0x7f6be009fae0 0x7f6be014fff0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f6bd8008790 tx=0x7f6bd800fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:44.214 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:44.204+0000 7f6be648b700 1 --2- 172.21.15.42:0/443371415 >> v2:172.21.15.40:3300/0 conn(0x7f6be009fae0 0x7f6be014fff0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f6bd8008790 tx=0x7f6bd800fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:44.214 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:44.204+0000 7f6be648b700 1 --2- 172.21.15.42:0/443371415 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6be009fae0 0x7f6be014fff0 secure :-1 s=READY pgs=1547 cs=0 l=1 rx=0x7f6bd8008790 tx=0x7f6bd800fbf0).ready entity=mon.0 client_cookie=352d7d96a66b0ad9 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:44.214 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:44.204+0000 7f6be648b700 1 --2- 172.21.15.42:0/443371415 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6be009fae0 0x7f6be014fff0 secure :-1 s=THROTTLE_DONE pgs=1547 cs=0 l=1 rx=0x7f6bd8008790 tx=0x7f6bd800fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:44.214 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:44.204+0000 7f6be4c88700 1 -- 172.21.15.42:0/443371415 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6bd8008bb0 con 0x7f6be009fae0 2019-12-05T06:06:44.214 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:44.204+0000 7f6be648b700 1 --2- 172.21.15.42:0/443371415 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6be009fae0 0x7f6be014fff0 secure :-1 s=THROTTLE_DONE pgs=1547 cs=0 l=1 rx=0x7f6bd8008790 tx=0x7f6bd800fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:44.214 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:44.204+0000 7f6be648b700 1 --2- 172.21.15.42:0/443371415 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6be009fae0 0x7f6be014fff0 secure :-1 s=THROTTLE_DONE pgs=1547 cs=0 l=1 rx=0x7f6bd8008790 tx=0x7f6bd800fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:44.214 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:44.204+0000 7f6be4c88700 1 -- 172.21.15.42:0/443371415 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bd80087d0 con 0x7f6be009fae0 2019-12-05T06:06:44.215 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:44.204+0000 7f6be4c88700 1 -- 172.21.15.42:0/443371415 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6bd80118d0 con 0x7f6be009fae0 2019-12-05T06:06:44.215 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:44.204+0000 7f6be76ed700 1 -- 172.21.15.42:0/443371415 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6be009fae0 msgr2=0x7f6be014fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:44.215 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:44.204+0000 7f6be76ed700 1 --2- 172.21.15.42:0/443371415 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6be009fae0 0x7f6be014fff0 secure :-1 s=READY pgs=1547 cs=0 l=1 rx=0x7f6bd8008790 tx=0x7f6bd800fbf0).stop 2019-12-05T06:06:44.215 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:44.204+0000 7f6be76ed700 1 -- 172.21.15.42:0/443371415 shutdown_connections 2019-12-05T06:06:44.216 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:44.204+0000 7f6be76ed700 1 --2- 172.21.15.42:0/443371415 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6be009fae0 0x7f6be014fff0 unknown :-1 s=CLOSED pgs=1547 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:44.216 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:44.204+0000 7f6be76ed700 1 -- 172.21.15.42:0/443371415 shutdown_connections 2019-12-05T06:06:44.216 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:44.204+0000 7f6be76ed700 1 -- 172.21.15.42:0/443371415 wait complete. 2019-12-05T06:06:44.216 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:44.204+0000 7f6be76ed700 1 -- 172.21.15.42:0/443371415 >> 172.21.15.42:0/443371415 conn(0x7f6be00aabe0 msgr2=0x7f6be00a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:06:44.221 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:44.208+0000 7f6be76ed700 1 Processor -- start 2019-12-05T06:06:44.221 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:44.208+0000 7f6be76ed700 1 -- start start 2019-12-05T06:06:44.221 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:44.208+0000 7f6be76ed700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6be0067320 0x7f6be0144d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:44.221 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:44.208+0000 7f6be76ed700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6be0143230 0x7f6be0143660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:44.222 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:44.208+0000 7f6be76ed700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6be0142f40 con 0x7f6be007a790 2019-12-05T06:06:44.228 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:44.208+0000 7f6be76ed700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6be0142760 con 0x7f6be0143230 2019-12-05T06:06:44.229 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:44.208+0000 7f6be76ed700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6be0142a00 con 0x7f6be0067320 2019-12-05T06:06:44.229 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:44.208+0000 7f6be5489700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6be0143230 0x7f6be0143660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:44.229 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:44.208+0000 7f6be5489700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6be0143230 0x7f6be0143660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:48344/0 (socket says 172.21.15.42:48344) 2019-12-05T06:06:44.229 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:44.208+0000 7f6be5489700 1 -- 172.21.15.42:0/798718205 learned_addr learned my addr 172.21.15.42:0/798718205 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:06:44.229 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:44.208+0000 7f6be5c8a700 1 --2- 172.21.15.42:0/798718205 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6be0067320 0x7f6be0144d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:44.229 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:44.212+0000 7f6be5489700 1 -- 172.21.15.42:0/798718205 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6be0067320 msgr2=0x7f6be0144d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:44.229 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:44.212+0000 7f6be5489700 1 --2- 172.21.15.42:0/798718205 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6be0067320 0x7f6be0144d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:44.230 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:44.212+0000 7f6be5489700 1 -- 172.21.15.42:0/798718205 >> v1:172.21.15.40:6789/0 conn(0x7f6be007a790 legacy=0x7f6be008eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:44.230 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:44.212+0000 7f6bd27fc700 1 -- 172.21.15.42:0/798718205 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2900985676 0 0) 0x7f6bd80110c0 con 0x7f6be007a790 2019-12-05T06:06:44.230 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:44.212+0000 7f6be5489700 1 -- 172.21.15.42:0/798718205 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6be0114220 con 0x7f6be0143230 2019-12-05T06:06:44.230 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:44.212+0000 7f6be5489700 1 --2- 172.21.15.42:0/798718205 >> v2:172.21.15.42:3300/0 conn(0x7f6be0143230 0x7f6be0143660 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f6be0142920 tx=0x7f6bd4010ca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:44.230 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:44.212+0000 7f6be5489700 1 --2- 172.21.15.42:0/798718205 >> v2:172.21.15.42:3300/0 conn(0x7f6be0143230 0x7f6be0143660 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f6be0142920 tx=0x7f6bd4010ca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:44.230 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:44.212+0000 7f6be5489700 1 --2- 172.21.15.42:0/798718205 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6be0143230 0x7f6be0143660 secure :-1 s=READY pgs=2035 cs=0 l=1 rx=0x7f6be0142920 tx=0x7f6bd4010ca0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:44.231 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:44.212+0000 7f6be5489700 1 --2- 172.21.15.42:0/798718205 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6be0143230 0x7f6be0143660 secure :-1 s=THROTTLE_DONE pgs=2035 cs=0 l=1 rx=0x7f6be0142920 tx=0x7f6bd4010ca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:44.231 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:44.212+0000 7f6bd27fc700 1 -- 172.21.15.42:0/798718205 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6bd4005bc0 con 0x7f6be0143230 2019-12-05T06:06:44.241 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:44.212+0000 7f6be5489700 1 --2- 172.21.15.42:0/798718205 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6be0143230 0x7f6be0143660 secure :-1 s=THROTTLE_DONE pgs=2035 cs=0 l=1 rx=0x7f6be0142920 tx=0x7f6bd4010ca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:44.242 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:44.212+0000 7f6be5489700 1 --2- 172.21.15.42:0/798718205 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6be0143230 0x7f6be0143660 secure :-1 s=THROTTLE_DONE pgs=2035 cs=0 l=1 rx=0x7f6be0142920 tx=0x7f6bd4010ca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:44.242 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:44.212+0000 7f6bd27fc700 1 -- 172.21.15.42:0/798718205 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bd40057e0 con 0x7f6be0143230 2019-12-05T06:06:44.242 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:44.212+0000 7f6be76ed700 1 -- 172.21.15.42:0/798718205 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6be0145250 con 0x7f6be0143230 2019-12-05T06:06:44.242 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:44.212+0000 7f6bd27fc700 1 -- 172.21.15.42:0/798718205 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6bd4012890 con 0x7f6be0143230 2019-12-05T06:06:44.242 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:44.212+0000 7f6be76ed700 1 -- 172.21.15.42:0/798718205 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6be007abc0 con 0x7f6be0143230 2019-12-05T06:06:44.242 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:44.212+0000 7f6be5489700 1 --2- 172.21.15.42:0/798718205 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6be0143230 0x7f6be0143660 secure :-1 s=THROTTLE_DONE pgs=2035 cs=0 l=1 rx=0x7f6be0142920 tx=0x7f6bd4010ca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:44.243 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:44.212+0000 7f6bd27fc700 1 -- 172.21.15.42:0/798718205 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f6bd4012a90 con 0x7f6be0143230 2019-12-05T06:06:44.243 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:44.212+0000 7f6be5489700 1 --2- 172.21.15.42:0/798718205 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6be0143230 0x7f6be0143660 secure :-1 s=THROTTLE_DONE pgs=2035 cs=0 l=1 rx=0x7f6be0142920 tx=0x7f6bd4010ca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:44.249 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:44.212+0000 7f6bd27fc700 1 --2- 172.21.15.42:0/798718205 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6bc803ac90 0x7f6bc803d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:44.249 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:44.212+0000 7f6bd27fc700 1 -- 172.21.15.42:0/798718205 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9729+0+0 (secure 0 0 0) 0x7f6bd404b470 con 0x7f6be0143230 2019-12-05T06:06:44.249 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:44.212+0000 7f6be5c8a700 1 --2- 172.21.15.42:0/798718205 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6bc803ac90 0x7f6bc803d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:44.249 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:44.216+0000 7f6be5c8a700 1 --2- 172.21.15.42:0/798718205 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6bc803ac90 0x7f6bc803d1a0 crc :-1 s=READY pgs=864 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:44.250 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:44.216+0000 7f6be76ed700 1 -- 172.21.15.42:0/798718205 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6be005d110 con 0x7f6be0143230 2019-12-05T06:06:44.250 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:44.228+0000 7f6be5489700 1 --2- 172.21.15.42:0/798718205 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6be0143230 0x7f6be0143660 secure :-1 s=THROTTLE_DONE pgs=2035 cs=0 l=1 rx=0x7f6be0142920 tx=0x7f6bd4010ca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:44.250 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:44.228+0000 7f6bd27fc700 1 -- 172.21.15.42:0/798718205 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f6bd40324f0 con 0x7f6be0143230 2019-12-05T06:06:44.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:44.396+0000 7f6be76ed700 1 -- 172.21.15.42:0/798718205 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6be005b8c0 con 0x7f6be0143230 2019-12-05T06:06:44.404 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:44.396+0000 7f6be5489700 1 --2- 172.21.15.42:0/798718205 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6be0143230 0x7f6be0143660 secure :-1 s=THROTTLE_DONE pgs=2035 cs=0 l=1 rx=0x7f6be0142920 tx=0x7f6bd4010ca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:44.404 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:44.396+0000 7f6bd27fc700 1 -- 172.21.15.42:0/798718205 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 248 v248) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f6bd4012da0 con 0x7f6be0143230 2019-12-05T06:06:44.404 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 248 2019-12-05T06:06:44.404 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:06:44.404 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":248,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":9573,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/2968227232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2968227232},{"type":"v1","addr":"172.21.15.40:6833","nonce":2968227232}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":248}],"filesystems":[{"mdsmap":{"epoch":247,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:05:33.997161+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":92,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6821},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6821":{"gid":6821,"name":"a","rank":0,"incarnation":244,"state":"up:active","state_seq":23,"addr":"172.21.15.42:6835/2253267554","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2253267554},{"type":"v1","addr":"172.21.15.42:6835","nonce":2253267554}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:06:44.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:44.400+0000 7f6be76ed700 1 -- 172.21.15.42:0/798718205 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6bc803ac90 msgr2=0x7f6bc803d1a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:44.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:44.400+0000 7f6be76ed700 1 --2- 172.21.15.42:0/798718205 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6bc803ac90 0x7f6bc803d1a0 crc :-1 s=READY pgs=864 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:44.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:44.400+0000 7f6be76ed700 1 -- 172.21.15.42:0/798718205 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6be0143230 msgr2=0x7f6be0143660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:44.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:44.400+0000 7f6be76ed700 1 --2- 172.21.15.42:0/798718205 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6be0143230 0x7f6be0143660 secure :-1 s=READY pgs=2035 cs=0 l=1 rx=0x7f6be0142920 tx=0x7f6bd4010ca0).stop 2019-12-05T06:06:44.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:44.400+0000 7f6be76ed700 1 -- 172.21.15.42:0/798718205 shutdown_connections 2019-12-05T06:06:44.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:44.400+0000 7f6be76ed700 1 --2- 172.21.15.42:0/798718205 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6bc803ac90 0x7f6bc803d1a0 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:44.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:44.400+0000 7f6be76ed700 1 --2- 172.21.15.42:0/798718205 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6be0067320 0x7f6be0144d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:44.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:44.400+0000 7f6be76ed700 1 --2- 172.21.15.42:0/798718205 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6be0143230 0x7f6be0143660 unknown :-1 s=CLOSED pgs=2035 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:44.424 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:44.400+0000 7f6be76ed700 1 -- 172.21.15.42:0/798718205 shutdown_connections 2019-12-05T06:06:44.424 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:44.400+0000 7f6be76ed700 1 -- 172.21.15.42:0/798718205 wait complete. 2019-12-05T06:06:44.425 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:44.400+0000 7f6be76ed700 1 -- 172.21.15.42:0/798718205 >> 172.21.15.42:0/798718205 conn(0x7f6be00a7740 msgr2=0x7f6be01196c0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:06:44.426 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:06:46.428 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:06:46.535 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:46.524+0000 7fb0bd9c6700 1 Processor -- start 2019-12-05T06:06:46.535 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:46.524+0000 7fb0bd9c6700 1 -- start start 2019-12-05T06:06:46.535 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:46.524+0000 7fb0bd9c6700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb0b80a7740 0x7fb0b808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:06:46.535 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:46.524+0000 7fb0bd9c6700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb0b809fae0 0x7fb0b807eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:06:46.536 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:46.524+0000 7fb0bd9c6700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0b80a0710 con 0x7fb0b807e720 2019-12-05T06:06:46.536 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:46.524+0000 7fb0bd9c6700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0b80a0b00 con 0x7fb0b809fae0 2019-12-05T06:06:46.536 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:46.524+0000 7fb0bd9c6700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0b808e5b0 con 0x7fb0b80a7740 2019-12-05T06:06:46.536 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:46.524+0000 7fb0b7fff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb0b809fae0 0x7fb0b807eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:46.536 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:46.524+0000 7fb0b7fff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb0b809fae0 0x7fb0b807eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:48358/0 (socket says 172.21.15.42:48358) 2019-12-05T06:06:46.536 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:46.524+0000 7fb0b7fff700 1 -- 172.21.15.42:0/2741584147 learned_addr learned my addr 172.21.15.42:0/2741584147 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:06:46.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:46.524+0000 7fb0b77fe700 1 --2- 172.21.15.42:0/2741584147 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb0b80a7740 0x7fb0b808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:46.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:46.524+0000 7fb0b7fff700 1 -- 172.21.15.42:0/2741584147 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb0b80a7740 msgr2=0x7fb0b808df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:06:46.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:46.524+0000 7fb0b7fff700 1 --2- 172.21.15.42:0/2741584147 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb0b80a7740 0x7fb0b808df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:06:46.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:46.524+0000 7fb0b7fff700 1 -- 172.21.15.42:0/2741584147 >> v1:172.21.15.40:6789/0 conn(0x7fb0b807e720 legacy=0x7fb0b809f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:46.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:46.524+0000 7fb0b7fff700 1 -- 172.21.15.42:0/2741584147 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0b807f0a0 con 0x7fb0b809fae0 2019-12-05T06:06:46.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:46.524+0000 7fb0b7fff700 1 --2- 172.21.15.42:0/2741584147 >> v2:172.21.15.42:3300/0 conn(0x7fb0b809fae0 0x7fb0b807eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fb0b0008b10 tx=0x7fb0b00100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:46.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:46.524+0000 7fb0b7fff700 1 --2- 172.21.15.42:0/2741584147 >> v2:172.21.15.42:3300/0 conn(0x7fb0b809fae0 0x7fb0b807eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fb0b0008b10 tx=0x7fb0b00100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:46.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:46.524+0000 7fb0b7fff700 1 --2- 172.21.15.42:0/2741584147 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb0b809fae0 0x7fb0b807eb50 secure :-1 s=READY pgs=2036 cs=0 l=1 rx=0x7fb0b0008b10 tx=0x7fb0b00100b0).ready entity=mon.1 client_cookie=3270352dffa6a055 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:46.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:46.524+0000 7fb0b7fff700 1 --2- 172.21.15.42:0/2741584147 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb0b809fae0 0x7fb0b807eb50 secure :-1 s=THROTTLE_DONE pgs=2036 cs=0 l=1 rx=0x7fb0b0008b10 tx=0x7fb0b00100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:46.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:46.524+0000 7fb0b67fc700 1 -- 172.21.15.42:0/2741584147 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb0b0010da0 con 0x7fb0b809fae0 2019-12-05T06:06:46.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:46.524+0000 7fb0b7fff700 1 --2- 172.21.15.42:0/2741584147 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb0b809fae0 0x7fb0b807eb50 secure :-1 s=THROTTLE_DONE pgs=2036 cs=0 l=1 rx=0x7fb0b0008b10 tx=0x7fb0b00100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:46.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:46.524+0000 7fb0b7fff700 1 --2- 172.21.15.42:0/2741584147 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb0b809fae0 0x7fb0b807eb50 secure :-1 s=THROTTLE_DONE pgs=2036 cs=0 l=1 rx=0x7fb0b0008b10 tx=0x7fb0b00100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:46.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:46.524+0000 7fb0b67fc700 1 -- 172.21.15.42:0/2741584147 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0b00109c0 con 0x7fb0b809fae0 2019-12-05T06:06:46.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:46.524+0000 7fb0b67fc700 1 -- 172.21.15.42:0/2741584147 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb0b00114a0 con 0x7fb0b809fae0 2019-12-05T06:06:46.539 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:46.524+0000 7fb0bd9c6700 1 -- 172.21.15.42:0/2741584147 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb0b809fae0 msgr2=0x7fb0b807eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:46.539 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:46.524+0000 7fb0bd9c6700 1 --2- 172.21.15.42:0/2741584147 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb0b809fae0 0x7fb0b807eb50 secure :-1 s=READY pgs=2036 cs=0 l=1 rx=0x7fb0b0008b10 tx=0x7fb0b00100b0).stop 2019-12-05T06:06:46.539 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:46.528+0000 7fb0bd9c6700 1 -- 172.21.15.42:0/2741584147 shutdown_connections 2019-12-05T06:06:46.539 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:46.528+0000 7fb0bd9c6700 1 --2- 172.21.15.42:0/2741584147 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb0b809fae0 0x7fb0b807eb50 unknown :-1 s=CLOSED pgs=2036 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:46.539 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:46.528+0000 7fb0bd9c6700 1 --2- 172.21.15.42:0/2741584147 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb0b80a7740 0x7fb0b808df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:06:46.539 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:46.528+0000 7fb0bd9c6700 1 -- 172.21.15.42:0/2741584147 shutdown_connections 2019-12-05T06:06:46.539 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:46.528+0000 7fb0bd9c6700 1 -- 172.21.15.42:0/2741584147 wait complete. 2019-12-05T06:06:46.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:46.528+0000 7fb0bd9c6700 1 -- 172.21.15.42:0/2741584147 >> 172.21.15.42:0/2741584147 conn(0x7fb0b80aabe0 msgr2=0x7fb0b80a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:06:46.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:46.528+0000 7fb0bd9c6700 1 Processor -- start 2019-12-05T06:06:46.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:46.528+0000 7fb0bd9c6700 1 -- start start 2019-12-05T06:06:46.544 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:46.532+0000 7fb0bd9c6700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb0b811ead0 0x7fb0b811d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:46.544 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:46.532+0000 7fb0bd9c6700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb0b80722c0 0x7fb0b8074700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:46.544 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:46.532+0000 7fb0bd9c6700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0b812e240 con 0x7fb0b811bc40 2019-12-05T06:06:46.544 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:46.532+0000 7fb0bd9c6700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0b811f050 con 0x7fb0b811ead0 2019-12-05T06:06:46.544 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:46.532+0000 7fb0bd9c6700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0b81193c0 con 0x7fb0b80722c0 2019-12-05T06:06:46.545 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:46.532+0000 7fb0b6ffd700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb0b811ead0 0x7fb0b811d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:46.545 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:46.532+0000 7fb0b6ffd700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb0b811ead0 0x7fb0b811d0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:48362/0 (socket says 172.21.15.42:48362) 2019-12-05T06:06:46.545 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:46.532+0000 7fb0b6ffd700 1 -- 172.21.15.42:0/301197105 learned_addr learned my addr 172.21.15.42:0/301197105 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:06:46.545 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:46.532+0000 7fb0b7fff700 1 --2- 172.21.15.42:0/301197105 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb0b80722c0 0x7fb0b8074700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:46.545 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:46.532+0000 7fb0b6ffd700 1 -- 172.21.15.42:0/301197105 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb0b80722c0 msgr2=0x7fb0b8074700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:46.545 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:46.532+0000 7fb0b6ffd700 1 --2- 172.21.15.42:0/301197105 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb0b80722c0 0x7fb0b8074700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:46.546 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:46.532+0000 7fb0b6ffd700 1 -- 172.21.15.42:0/301197105 >> v1:172.21.15.40:6789/0 conn(0x7fb0b811bc40 legacy=0x7fb0b8119660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:46.546 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:46.532+0000 7fb0b6ffd700 1 -- 172.21.15.42:0/301197105 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0b80a4f90 con 0x7fb0b811ead0 2019-12-05T06:06:46.546 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:46.532+0000 7fb0b6ffd700 1 --2- 172.21.15.42:0/301197105 >> v2:172.21.15.42:3300/0 conn(0x7fb0b811ead0 0x7fb0b811d0a0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fb0b811f210 tx=0x7fb0a80109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:46.546 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:46.532+0000 7fb0b6ffd700 1 --2- 172.21.15.42:0/301197105 >> v2:172.21.15.42:3300/0 conn(0x7fb0b811ead0 0x7fb0b811d0a0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fb0b811f210 tx=0x7fb0a80109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:46.547 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:46.532+0000 7fb0b6ffd700 1 --2- 172.21.15.42:0/301197105 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb0b811ead0 0x7fb0b811d0a0 secure :-1 s=READY pgs=2037 cs=0 l=1 rx=0x7fb0b811f210 tx=0x7fb0a80109a0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:46.547 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:46.532+0000 7fb0b6ffd700 1 --2- 172.21.15.42:0/301197105 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb0b811ead0 0x7fb0b811d0a0 secure :-1 s=THROTTLE_DONE pgs=2037 cs=0 l=1 rx=0x7fb0b811f210 tx=0x7fb0a80109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:46.547 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:46.532+0000 7fb0affff700 1 -- 172.21.15.42:0/301197105 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb0a800caa0 con 0x7fb0b811ead0 2019-12-05T06:06:46.547 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:46.532+0000 7fb0b6ffd700 1 --2- 172.21.15.42:0/301197105 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb0b811ead0 0x7fb0b811d0a0 secure :-1 s=THROTTLE_DONE pgs=2037 cs=0 l=1 rx=0x7fb0b811f210 tx=0x7fb0a80109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:46.547 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:46.532+0000 7fb0b6ffd700 1 --2- 172.21.15.42:0/301197105 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb0b811ead0 0x7fb0b811d0a0 secure :-1 s=THROTTLE_DONE pgs=2037 cs=0 l=1 rx=0x7fb0b811f210 tx=0x7fb0a80109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:46.547 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:46.532+0000 7fb0affff700 1 -- 172.21.15.42:0/301197105 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0a800c6c0 con 0x7fb0b811ead0 2019-12-05T06:06:46.547 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:46.532+0000 7fb0bd9c6700 1 -- 172.21.15.42:0/301197105 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0b8074c50 con 0x7fb0b811ead0 2019-12-05T06:06:46.548 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:46.532+0000 7fb0affff700 1 -- 172.21.15.42:0/301197105 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb0a8012210 con 0x7fb0b811ead0 2019-12-05T06:06:46.548 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:46.532+0000 7fb0bd9c6700 1 -- 172.21.15.42:0/301197105 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0b80751a0 con 0x7fb0b811ead0 2019-12-05T06:06:46.548 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:46.532+0000 7fb0b6ffd700 1 --2- 172.21.15.42:0/301197105 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb0b811ead0 0x7fb0b811d0a0 secure :-1 s=THROTTLE_DONE pgs=2037 cs=0 l=1 rx=0x7fb0b811f210 tx=0x7fb0a80109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:46.548 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:46.532+0000 7fb0affff700 1 -- 172.21.15.42:0/301197105 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fb0a8012520 con 0x7fb0b811ead0 2019-12-05T06:06:46.549 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:46.532+0000 7fb0b6ffd700 1 --2- 172.21.15.42:0/301197105 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb0b811ead0 0x7fb0b811d0a0 secure :-1 s=THROTTLE_DONE pgs=2037 cs=0 l=1 rx=0x7fb0b811f210 tx=0x7fb0a80109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:46.549 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:46.532+0000 7fb0affff700 1 --2- 172.21.15.42:0/301197105 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb09003ae70 0x7fb09003d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:46.549 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:46.532+0000 7fb0affff700 1 -- 172.21.15.42:0/301197105 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9729+0+0 (secure 0 0 0) 0x7fb0a804b270 con 0x7fb0b811ead0 2019-12-05T06:06:46.549 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:46.536+0000 7fb0b77fe700 1 --2- 172.21.15.42:0/301197105 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb09003ae70 0x7fb09003d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:46.549 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:46.536+0000 7fb0b77fe700 1 --2- 172.21.15.42:0/301197105 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb09003ae70 0x7fb09003d380 crc :-1 s=READY pgs=865 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:46.550 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:46.540+0000 7fb0bd9c6700 1 -- 172.21.15.42:0/301197105 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0b8119e60 con 0x7fb0b811ead0 2019-12-05T06:06:46.551 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:46.544+0000 7fb0b6ffd700 1 --2- 172.21.15.42:0/301197105 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb0b811ead0 0x7fb0b811d0a0 secure :-1 s=THROTTLE_DONE pgs=2037 cs=0 l=1 rx=0x7fb0b811f210 tx=0x7fb0a80109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:46.551 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:46.544+0000 7fb0affff700 1 -- 172.21.15.42:0/301197105 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fb0a80322f0 con 0x7fb0b811ead0 2019-12-05T06:06:46.715 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:46.712+0000 7fb0bd9c6700 1 -- 172.21.15.42:0/301197105 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb0b805d110 con 0x7fb0b811ead0 2019-12-05T06:06:46.716 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:46.712+0000 7fb0b6ffd700 1 --2- 172.21.15.42:0/301197105 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb0b811ead0 0x7fb0b811d0a0 secure :-1 s=THROTTLE_DONE pgs=2037 cs=0 l=1 rx=0x7fb0b811f210 tx=0x7fb0a80109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:46.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:46.712+0000 7fb0affff700 1 -- 172.21.15.42:0/301197105 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 248 v248) v1 ==== 95+0+2112 (secure 0 0 0) 0x7fb0a8032550 con 0x7fb0b811ead0 2019-12-05T06:06:46.717 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 248 2019-12-05T06:06:46.717 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:06:46.717 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":248,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":9573,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/2968227232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2968227232},{"type":"v1","addr":"172.21.15.40:6833","nonce":2968227232}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":248}],"filesystems":[{"mdsmap":{"epoch":247,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:05:33.997161+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":92,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6821},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6821":{"gid":6821,"name":"a","rank":0,"incarnation":244,"state":"up:active","state_seq":23,"addr":"172.21.15.42:6835/2253267554","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2253267554},{"type":"v1","addr":"172.21.15.42:6835","nonce":2253267554}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:06:46.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:46.712+0000 7fb0bd9c6700 1 -- 172.21.15.42:0/301197105 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb09003ae70 msgr2=0x7fb09003d380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:46.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:46.712+0000 7fb0bd9c6700 1 --2- 172.21.15.42:0/301197105 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb09003ae70 0x7fb09003d380 crc :-1 s=READY pgs=865 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:46.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:46.712+0000 7fb0bd9c6700 1 -- 172.21.15.42:0/301197105 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb0b811ead0 msgr2=0x7fb0b811d0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:46.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:46.712+0000 7fb0bd9c6700 1 --2- 172.21.15.42:0/301197105 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb0b811ead0 0x7fb0b811d0a0 secure :-1 s=READY pgs=2037 cs=0 l=1 rx=0x7fb0b811f210 tx=0x7fb0a80109a0).stop 2019-12-05T06:06:46.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:46.712+0000 7fb0bd9c6700 1 -- 172.21.15.42:0/301197105 shutdown_connections 2019-12-05T06:06:46.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:46.712+0000 7fb0bd9c6700 1 --2- 172.21.15.42:0/301197105 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb09003ae70 0x7fb09003d380 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:46.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:46.712+0000 7fb0bd9c6700 1 --2- 172.21.15.42:0/301197105 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb0b80722c0 0x7fb0b8074700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:46.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:46.712+0000 7fb0bd9c6700 1 --2- 172.21.15.42:0/301197105 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb0b811ead0 0x7fb0b811d0a0 unknown :-1 s=CLOSED pgs=2037 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:46.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:46.712+0000 7fb0bd9c6700 1 -- 172.21.15.42:0/301197105 shutdown_connections 2019-12-05T06:06:46.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:46.712+0000 7fb0bd9c6700 1 -- 172.21.15.42:0/301197105 wait complete. 2019-12-05T06:06:46.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:46.712+0000 7fb0bd9c6700 1 -- 172.21.15.42:0/301197105 >> 172.21.15.42:0/301197105 conn(0x7fb0b80a7740 msgr2=0x7fb0b811d6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:06:46.728 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:06:48.730 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:06:48.848 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:48.828+0000 7f337956f700 1 Processor -- start 2019-12-05T06:06:48.848 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:48.828+0000 7f337956f700 1 -- start start 2019-12-05T06:06:48.848 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:48.832+0000 7f337956f700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f337409fae0 0x7f337414fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:06:48.848 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:48.832+0000 7f337956f700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33740a0710 con 0x7f33740a7740 2019-12-05T06:06:48.849 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:48.832+0000 7f337956f700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f337407ef20 con 0x7f337408df80 2019-12-05T06:06:48.849 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:48.832+0000 7f337956f700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33740a0b00 con 0x7f337409fae0 2019-12-05T06:06:48.849 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:48.832+0000 7f3372ffd700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f337408df80 0x7f337409f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:48414/0 (socket says 172.21.15.42:48414) 2019-12-05T06:06:48.849 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:48.832+0000 7f3372ffd700 1 -- 172.21.15.42:0/1699520471 learned_addr learned my addr 172.21.15.42:0/1699520471 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:06:48.849 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:48.832+0000 7f3373fff700 1 --2- 172.21.15.42:0/1699520471 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f337409fae0 0x7f337414fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:48.849 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:48.832+0000 7f33727fc700 1 -- 172.21.15.42:0/1699520471 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1031071437 0 0) 0x7f337407ef20 con 0x7f337408df80 2019-12-05T06:06:48.849 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:48.832+0000 7f33727fc700 1 -- 172.21.15.42:0/1699520471 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33600030b0 con 0x7f337408df80 2019-12-05T06:06:48.850 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:48.832+0000 7f33727fc700 1 -- 172.21.15.42:0/1699520471 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1610663798 0 0) 0x7f33740a0710 con 0x7f33740a7740 2019-12-05T06:06:48.850 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:48.832+0000 7f3373fff700 1 -- 172.21.15.42:0/1699520471 >> v1:172.21.15.42:6789/0 conn(0x7f337408df80 legacy=0x7f337409f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:48.850 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:48.832+0000 7f3373fff700 1 -- 172.21.15.42:0/1699520471 >> v1:172.21.15.40:6789/0 conn(0x7f33740a7740 legacy=0x7f337407e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:48.850 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:48.832+0000 7f3373fff700 1 -- 172.21.15.42:0/1699520471 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33740a9ba0 con 0x7f337409fae0 2019-12-05T06:06:48.850 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:48.832+0000 7f3373fff700 1 --2- 172.21.15.42:0/1699520471 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f337409fae0 0x7f337414fff0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f336c0032b0 tx=0x7f336c00ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:48.850 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:48.832+0000 7f3373fff700 1 --2- 172.21.15.42:0/1699520471 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f337409fae0 0x7f337414fff0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f336c0032b0 tx=0x7f336c00ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:48.850 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:48.832+0000 7f3373fff700 1 --2- 172.21.15.42:0/1699520471 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f337409fae0 0x7f337414fff0 secure :-1 s=READY pgs=452 cs=0 l=1 rx=0x7f336c0032b0 tx=0x7f336c00ff70).ready entity=mon.2 client_cookie=3153837f7e6ef740 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:48.851 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:48.832+0000 7f3373fff700 1 --2- 172.21.15.42:0/1699520471 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f337409fae0 0x7f337414fff0 secure :-1 s=THROTTLE_DONE pgs=452 cs=0 l=1 rx=0x7f336c0032b0 tx=0x7f336c00ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:48.851 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:48.832+0000 7f3373fff700 1 --2- 172.21.15.42:0/1699520471 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f337409fae0 0x7f337414fff0 secure :-1 s=THROTTLE_DONE pgs=452 cs=0 l=1 rx=0x7f336c0032b0 tx=0x7f336c00ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:48.851 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:48.832+0000 7f3373fff700 1 --2- 172.21.15.42:0/1699520471 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f337409fae0 0x7f337414fff0 secure :-1 s=THROTTLE_DONE pgs=452 cs=0 l=1 rx=0x7f336c0032b0 tx=0x7f336c00ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:48.851 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:48.832+0000 7f33727fc700 1 -- 172.21.15.42:0/1699520471 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f336c0109b0 con 0x7f337409fae0 2019-12-05T06:06:48.851 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:48.832+0000 7f33727fc700 1 -- 172.21.15.42:0/1699520471 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f336c008b50 con 0x7f337409fae0 2019-12-05T06:06:48.851 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:48.832+0000 7f33727fc700 1 -- 172.21.15.42:0/1699520471 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f336c0110c0 con 0x7f337409fae0 2019-12-05T06:06:48.851 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:48.832+0000 7f337956f700 1 -- 172.21.15.42:0/1699520471 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f337409fae0 msgr2=0x7f337414fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:48.852 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:48.832+0000 7f337956f700 1 --2- 172.21.15.42:0/1699520471 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f337409fae0 0x7f337414fff0 secure :-1 s=READY pgs=452 cs=0 l=1 rx=0x7f336c0032b0 tx=0x7f336c00ff70).stop 2019-12-05T06:06:48.852 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:48.836+0000 7f337956f700 1 -- 172.21.15.42:0/1699520471 shutdown_connections 2019-12-05T06:06:48.852 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:48.836+0000 7f337956f700 1 --2- 172.21.15.42:0/1699520471 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f337409fae0 0x7f337414fff0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:48.853 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:48.836+0000 7f337956f700 1 -- 172.21.15.42:0/1699520471 shutdown_connections 2019-12-05T06:06:48.853 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:48.836+0000 7f337956f700 1 -- 172.21.15.42:0/1699520471 wait complete. 2019-12-05T06:06:48.853 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:48.836+0000 7f337956f700 1 -- 172.21.15.42:0/1699520471 >> 172.21.15.42:0/1699520471 conn(0x7f33740aabe0 msgr2=0x7f33740a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:06:48.853 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:48.836+0000 7f337956f700 1 Processor -- start 2019-12-05T06:06:48.853 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:48.836+0000 7f337956f700 1 -- start start 2019-12-05T06:06:48.854 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:48.840+0000 7f337956f700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3374118e90 0x7f33741474a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:48.854 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:48.840+0000 7f337956f700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f3374148680 0x7f337407d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:48.854 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:48.840+0000 7f337956f700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f337407df40 con 0x7f3374148250 2019-12-05T06:06:48.854 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:48.840+0000 7f337956f700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3374147b60 con 0x7f3374148680 2019-12-05T06:06:48.854 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:48.840+0000 7f337956f700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3374147e00 con 0x7f3374118e90 2019-12-05T06:06:48.854 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:48.840+0000 7f33737fe700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3374118e90 0x7f33741474a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:48.855 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:48.840+0000 7f3373fff700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f3374148680 0x7f337407d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:48.855 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:48.840+0000 7f33737fe700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3374118e90 0x7f33741474a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:48846/0 (socket says 172.21.15.42:48846) 2019-12-05T06:06:48.855 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:48.840+0000 7f33737fe700 1 -- 172.21.15.42:0/3568681769 learned_addr learned my addr 172.21.15.42:0/3568681769 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:06:48.855 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:48.840+0000 7f33737fe700 1 -- 172.21.15.42:0/3568681769 >> v2:172.21.15.40:3300/0 conn(0x7f3374148680 msgr2=0x7f337407d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:48.856 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:48.840+0000 7f33737fe700 1 --2- 172.21.15.42:0/3568681769 >> v2:172.21.15.40:3300/0 conn(0x7f3374148680 0x7f337407d9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:48.856 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:48.840+0000 7f33737fe700 1 -- 172.21.15.42:0/3568681769 >> v1:172.21.15.40:6789/0 conn(0x7f3374148250 legacy=0x7f3374113a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:48.856 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:48.840+0000 7f33737fe700 1 -- 172.21.15.42:0/3568681769 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f337407e170 con 0x7f3374118e90 2019-12-05T06:06:48.856 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:48.840+0000 7f33737fe700 1 --2- 172.21.15.42:0/3568681769 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3374118e90 0x7f33741474a0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f3364000b80 tx=0x7f33640191b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:48.856 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:48.840+0000 7f33737fe700 1 --2- 172.21.15.42:0/3568681769 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3374118e90 0x7f33741474a0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f3364000b80 tx=0x7f33640191b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:48.856 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:48.840+0000 7f33737fe700 1 --2- 172.21.15.42:0/3568681769 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3374118e90 0x7f33741474a0 secure :-1 s=READY pgs=453 cs=0 l=1 rx=0x7f3364000b80 tx=0x7f33640191b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:48.857 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:48.840+0000 7f33737fe700 1 --2- 172.21.15.42:0/3568681769 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3374118e90 0x7f33741474a0 secure :-1 s=THROTTLE_DONE pgs=453 cs=0 l=1 rx=0x7f3364000b80 tx=0x7f33640191b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:48.857 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:48.840+0000 7f336bfff700 1 -- 172.21.15.42:0/3568681769 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f336401aaf0 con 0x7f3374118e90 2019-12-05T06:06:48.857 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:48.840+0000 7f33737fe700 1 --2- 172.21.15.42:0/3568681769 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3374118e90 0x7f33741474a0 secure :-1 s=THROTTLE_DONE pgs=453 cs=0 l=1 rx=0x7f3364000b80 tx=0x7f33640191b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:48.857 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:48.840+0000 7f336bfff700 1 -- 172.21.15.42:0/3568681769 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3364010b90 con 0x7f3374118e90 2019-12-05T06:06:48.857 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:48.840+0000 7f33737fe700 1 --2- 172.21.15.42:0/3568681769 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3374118e90 0x7f33741474a0 secure :-1 s=THROTTLE_DONE pgs=453 cs=0 l=1 rx=0x7f3364000b80 tx=0x7f33640191b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:48.857 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:48.840+0000 7f337956f700 1 -- 172.21.15.42:0/3568681769 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3374114260 con 0x7f3374118e90 2019-12-05T06:06:48.858 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:48.840+0000 7f336bfff700 1 -- 172.21.15.42:0/3568681769 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3364003350 con 0x7f3374118e90 2019-12-05T06:06:48.858 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:48.840+0000 7f337956f700 1 -- 172.21.15.42:0/3568681769 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3374129b40 con 0x7f3374118e90 2019-12-05T06:06:48.858 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:48.840+0000 7f33737fe700 1 --2- 172.21.15.42:0/3568681769 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3374118e90 0x7f33741474a0 secure :-1 s=THROTTLE_DONE pgs=453 cs=0 l=1 rx=0x7f3364000b80 tx=0x7f33640191b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:48.858 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:48.844+0000 7f336bfff700 1 -- 172.21.15.42:0/3568681769 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f3364014bd0 con 0x7f3374118e90 2019-12-05T06:06:48.858 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:48.844+0000 7f33737fe700 1 --2- 172.21.15.42:0/3568681769 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3374118e90 0x7f33741474a0 secure :-1 s=THROTTLE_DONE pgs=453 cs=0 l=1 rx=0x7f3364000b80 tx=0x7f33640191b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:48.859 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:48.844+0000 7f336bfff700 1 --2- 172.21.15.42:0/3568681769 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f335803aca0 0x7f335803d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:48.859 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:48.844+0000 7f336bfff700 1 -- 172.21.15.42:0/3568681769 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9729+0+0 (secure 0 0 0) 0x7f3364053460 con 0x7f3374118e90 2019-12-05T06:06:48.860 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:48.844+0000 7f3372ffd700 1 --2- 172.21.15.42:0/3568681769 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f335803aca0 0x7f335803d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:48.860 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:48.844+0000 7f3372ffd700 1 --2- 172.21.15.42:0/3568681769 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f335803aca0 0x7f335803d1b0 crc :-1 s=READY pgs=866 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:48.860 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:48.844+0000 7f337956f700 1 -- 172.21.15.42:0/3568681769 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f337405d110 con 0x7f3374118e90 2019-12-05T06:06:48.860 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:48.852+0000 7f33737fe700 1 --2- 172.21.15.42:0/3568681769 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3374118e90 0x7f33741474a0 secure :-1 s=THROTTLE_DONE pgs=453 cs=0 l=1 rx=0x7f3364000b80 tx=0x7f33640191b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:48.860 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:48.852+0000 7f336bfff700 1 -- 172.21.15.42:0/3568681769 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f336403a4c0 con 0x7f3374118e90 2019-12-05T06:06:49.024 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:49.016+0000 7f337956f700 1 -- 172.21.15.42:0/3568681769 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f337405b8c0 con 0x7f3374118e90 2019-12-05T06:06:49.024 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:49.020+0000 7f33737fe700 1 --2- 172.21.15.42:0/3568681769 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3374118e90 0x7f33741474a0 secure :-1 s=THROTTLE_DONE pgs=453 cs=0 l=1 rx=0x7f3364000b80 tx=0x7f33640191b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:49.025 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:49.020+0000 7f336bfff700 1 -- 172.21.15.42:0/3568681769 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 248 v248) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f336403a720 con 0x7f3374118e90 2019-12-05T06:06:49.026 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 248 2019-12-05T06:06:49.026 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:06:49.027 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":248,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":9573,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/2968227232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2968227232},{"type":"v1","addr":"172.21.15.40:6833","nonce":2968227232}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":248}],"filesystems":[{"mdsmap":{"epoch":247,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:05:33.997161+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":92,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6821},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6821":{"gid":6821,"name":"a","rank":0,"incarnation":244,"state":"up:active","state_seq":23,"addr":"172.21.15.42:6835/2253267554","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2253267554},{"type":"v1","addr":"172.21.15.42:6835","nonce":2253267554}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:06:49.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:49.024+0000 7f337956f700 1 -- 172.21.15.42:0/3568681769 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f335803aca0 msgr2=0x7f335803d1b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:49.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:49.024+0000 7f337956f700 1 --2- 172.21.15.42:0/3568681769 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f335803aca0 0x7f335803d1b0 crc :-1 s=READY pgs=866 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:49.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:49.024+0000 7f337956f700 1 -- 172.21.15.42:0/3568681769 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3374118e90 msgr2=0x7f33741474a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:49.032 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:49.024+0000 7f337956f700 1 --2- 172.21.15.42:0/3568681769 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3374118e90 0x7f33741474a0 secure :-1 s=READY pgs=453 cs=0 l=1 rx=0x7f3364000b80 tx=0x7f33640191b0).stop 2019-12-05T06:06:49.032 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:49.024+0000 7f337956f700 1 -- 172.21.15.42:0/3568681769 shutdown_connections 2019-12-05T06:06:49.032 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:49.024+0000 7f337956f700 1 --2- 172.21.15.42:0/3568681769 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f335803aca0 0x7f335803d1b0 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:49.033 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:49.024+0000 7f337956f700 1 --2- 172.21.15.42:0/3568681769 >> v2:172.21.15.40:3300/0 conn(0x7f3374148680 0x7f337407d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:49.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:49.024+0000 7f337956f700 1 --2- 172.21.15.42:0/3568681769 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3374118e90 0x7f33741474a0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:49.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:49.024+0000 7f337956f700 1 -- 172.21.15.42:0/3568681769 shutdown_connections 2019-12-05T06:06:49.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:49.024+0000 7f337956f700 1 -- 172.21.15.42:0/3568681769 wait complete. 2019-12-05T06:06:49.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:49.024+0000 7f337956f700 1 -- 172.21.15.42:0/3568681769 >> 172.21.15.42:0/3568681769 conn(0x7f33740a8040 msgr2=0x7f3374059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:06:49.039 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:06:51.041 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:06:51.152 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:51.140+0000 7fe6fd31d700 1 Processor -- start 2019-12-05T06:06:51.152 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:51.140+0000 7fe6fd31d700 1 -- start start 2019-12-05T06:06:51.152 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:51.140+0000 7fe6fd31d700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe6f80a7740 0x7fe6f808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:06:51.153 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:51.140+0000 7fe6fd31d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe6f807e720 0x7fe6f807eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:06:51.153 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:51.140+0000 7fe6fd31d700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe6f809f2e0 0x7fe6f809f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:06:51.153 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:51.140+0000 7fe6fd31d700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6f80a0b00 con 0x7fe6f80a7740 2019-12-05T06:06:51.153 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:51.140+0000 7fe6fd31d700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6f808e5b0 con 0x7fe6f807e720 2019-12-05T06:06:51.153 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:51.140+0000 7fe6fd31d700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6f808e850 con 0x7fe6f809f2e0 2019-12-05T06:06:51.153 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:51.140+0000 7fe6f6ffd700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe6f807e720 0x7fe6f807eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:51.153 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:51.140+0000 7fe6f6ffd700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe6f807e720 0x7fe6f807eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:48390/0 (socket says 172.21.15.42:48390) 2019-12-05T06:06:51.154 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:51.140+0000 7fe6f6ffd700 1 -- 172.21.15.42:0/464945178 learned_addr learned my addr 172.21.15.42:0/464945178 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:06:51.154 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:51.140+0000 7fe6f7fff700 1 --2- 172.21.15.42:0/464945178 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe6f809f2e0 0x7fe6f809f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:51.154 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:51.140+0000 7fe6f77fe700 1 --2- 172.21.15.42:0/464945178 >> v2:172.21.15.40:3300/0 conn(0x7fe6f80a7740 0x7fe6f808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:51.154 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:51.144+0000 7fe6f7fff700 1 -- 172.21.15.42:0/464945178 >> v2:172.21.15.42:3300/0 conn(0x7fe6f807e720 msgr2=0x7fe6f807eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:06:51.154 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:51.144+0000 7fe6f7fff700 1 --2- 172.21.15.42:0/464945178 >> v2:172.21.15.42:3300/0 conn(0x7fe6f807e720 0x7fe6f807eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:06:51.154 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:51.144+0000 7fe6f7fff700 1 -- 172.21.15.42:0/464945178 >> v2:172.21.15.40:3300/0 conn(0x7fe6f80a7740 msgr2=0x7fe6f808df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:06:51.155 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:51.144+0000 7fe6f7fff700 1 --2- 172.21.15.42:0/464945178 >> v2:172.21.15.40:3300/0 conn(0x7fe6f80a7740 0x7fe6f808df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:06:51.156 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:51.144+0000 7fe6f7fff700 1 -- 172.21.15.42:0/464945178 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6f807f0a0 con 0x7fe6f809f2e0 2019-12-05T06:06:51.158 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:51.144+0000 7fe6f7fff700 1 --2- 172.21.15.42:0/464945178 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe6f809f2e0 0x7fe6f809f710 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fe6f00032b0 tx=0x7fe6f0010070).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:51.158 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:51.144+0000 7fe6f7fff700 1 --2- 172.21.15.42:0/464945178 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe6f809f2e0 0x7fe6f809f710 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fe6f00032b0 tx=0x7fe6f0010070).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:51.158 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:51.144+0000 7fe6f7fff700 1 --2- 172.21.15.42:0/464945178 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe6f809f2e0 0x7fe6f809f710 secure :-1 s=READY pgs=454 cs=0 l=1 rx=0x7fe6f00032b0 tx=0x7fe6f0010070).ready entity=mon.2 client_cookie=c56da93aaf4f3569 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:51.158 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:51.144+0000 7fe6f7fff700 1 --2- 172.21.15.42:0/464945178 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe6f809f2e0 0x7fe6f809f710 secure :-1 s=THROTTLE_DONE pgs=454 cs=0 l=1 rx=0x7fe6f00032b0 tx=0x7fe6f0010070).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:51.158 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:51.144+0000 7fe6f7fff700 1 --2- 172.21.15.42:0/464945178 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe6f809f2e0 0x7fe6f809f710 secure :-1 s=THROTTLE_DONE pgs=454 cs=0 l=1 rx=0x7fe6f00032b0 tx=0x7fe6f0010070).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:51.158 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:51.144+0000 7fe6f7fff700 1 --2- 172.21.15.42:0/464945178 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe6f809f2e0 0x7fe6f809f710 secure :-1 s=THROTTLE_DONE pgs=454 cs=0 l=1 rx=0x7fe6f00032b0 tx=0x7fe6f0010070).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:51.159 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:51.144+0000 7fe6f67fc700 1 -- 172.21.15.42:0/464945178 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe6f0010940 con 0x7fe6f809f2e0 2019-12-05T06:06:51.159 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:51.144+0000 7fe6f67fc700 1 -- 172.21.15.42:0/464945178 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6f0008b50 con 0x7fe6f809f2e0 2019-12-05T06:06:51.159 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:51.144+0000 7fe6f67fc700 1 -- 172.21.15.42:0/464945178 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe6f00110c0 con 0x7fe6f809f2e0 2019-12-05T06:06:51.159 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:51.144+0000 7fe6fd31d700 1 -- 172.21.15.42:0/464945178 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe6f809f2e0 msgr2=0x7fe6f809f710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:51.159 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:51.144+0000 7fe6fd31d700 1 --2- 172.21.15.42:0/464945178 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe6f809f2e0 0x7fe6f809f710 secure :-1 s=READY pgs=454 cs=0 l=1 rx=0x7fe6f00032b0 tx=0x7fe6f0010070).stop 2019-12-05T06:06:51.160 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:51.144+0000 7fe6fd31d700 1 -- 172.21.15.42:0/464945178 shutdown_connections 2019-12-05T06:06:51.160 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:51.144+0000 7fe6fd31d700 1 --2- 172.21.15.42:0/464945178 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe6f809f2e0 0x7fe6f809f710 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:51.160 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:51.144+0000 7fe6fd31d700 1 --2- 172.21.15.42:0/464945178 >> v2:172.21.15.40:3300/0 conn(0x7fe6f80a7740 0x7fe6f808df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:06:51.160 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:51.144+0000 7fe6fd31d700 1 --2- 172.21.15.42:0/464945178 >> v2:172.21.15.42:3300/0 conn(0x7fe6f807e720 0x7fe6f807eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:06:51.160 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:51.144+0000 7fe6fd31d700 1 -- 172.21.15.42:0/464945178 shutdown_connections 2019-12-05T06:06:51.160 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:51.144+0000 7fe6fd31d700 1 -- 172.21.15.42:0/464945178 wait complete. 2019-12-05T06:06:51.160 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:51.144+0000 7fe6fd31d700 1 -- 172.21.15.42:0/464945178 >> 172.21.15.42:0/464945178 conn(0x7fe6f80aabe0 msgr2=0x7fe6f80a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:06:51.163 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:51.148+0000 7fe6fd31d700 1 Processor -- start 2019-12-05T06:06:51.164 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:51.148+0000 7fe6fd31d700 1 -- start start 2019-12-05T06:06:51.164 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:51.148+0000 7fe6fd31d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe6f80a4c40 0x7fe6f80a5070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:51.164 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:51.148+0000 7fe6fd31d700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6f8139750 con 0x7fe6f8131ec0 2019-12-05T06:06:51.164 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:51.148+0000 7fe6fd31d700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6f813a990 con 0x7fe6f81312f0 2019-12-05T06:06:51.164 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:51.148+0000 7fe6fd31d700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6f811bc20 con 0x7fe6f80a4c40 2019-12-05T06:06:51.164 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:51.148+0000 7fe6f77fe700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fe6f81312f0 0x7fe6f8123ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:48432/0 (socket says 172.21.15.42:48432) 2019-12-05T06:06:51.165 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:51.148+0000 7fe6f7fff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe6f80a4c40 0x7fe6f80a5070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:51.165 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:51.148+0000 7fe6f77fe700 1 -- 172.21.15.42:0/483761871 learned_addr learned my addr 172.21.15.42:0/483761871 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:06:51.165 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:51.148+0000 7fe6effff700 1 -- 172.21.15.42:0/483761871 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 190016168 0 0) 0x7fe6f8139750 con 0x7fe6f8131ec0 2019-12-05T06:06:51.166 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:51.148+0000 7fe6f7fff700 1 -- 172.21.15.42:0/483761871 >> v1:172.21.15.42:6789/0 conn(0x7fe6f81312f0 legacy=0x7fe6f8123ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:51.166 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:51.148+0000 7fe6f7fff700 1 -- 172.21.15.42:0/483761871 >> v1:172.21.15.40:6789/0 conn(0x7fe6f8131ec0 legacy=0x7fe6f811b420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:51.166 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:51.148+0000 7fe6f7fff700 1 -- 172.21.15.42:0/483761871 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6f813bbd0 con 0x7fe6f80a4c40 2019-12-05T06:06:51.166 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:51.148+0000 7fe6f7fff700 1 --2- 172.21.15.42:0/483761871 >> v2:172.21.15.42:3300/0 conn(0x7fe6f80a4c40 0x7fe6f80a5070 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fe6f805bbf0 tx=0x7fe6e8007720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:51.167 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:51.152+0000 7fe6f7fff700 1 --2- 172.21.15.42:0/483761871 >> v2:172.21.15.42:3300/0 conn(0x7fe6f80a4c40 0x7fe6f80a5070 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fe6f805bbf0 tx=0x7fe6e8007720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:51.167 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:51.152+0000 7fe6f7fff700 1 --2- 172.21.15.42:0/483761871 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe6f80a4c40 0x7fe6f80a5070 secure :-1 s=READY pgs=2040 cs=0 l=1 rx=0x7fe6f805bbf0 tx=0x7fe6e8007720).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:51.167 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:51.152+0000 7fe6f7fff700 1 --2- 172.21.15.42:0/483761871 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe6f80a4c40 0x7fe6f80a5070 secure :-1 s=THROTTLE_DONE pgs=2040 cs=0 l=1 rx=0x7fe6f805bbf0 tx=0x7fe6e8007720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:51.167 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:51.152+0000 7fe6effff700 1 -- 172.21.15.42:0/483761871 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe6f0005ab0 con 0x7fe6f80a4c40 2019-12-05T06:06:51.167 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:51.152+0000 7fe6f7fff700 1 --2- 172.21.15.42:0/483761871 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe6f80a4c40 0x7fe6f80a5070 secure :-1 s=THROTTLE_DONE pgs=2040 cs=0 l=1 rx=0x7fe6f805bbf0 tx=0x7fe6e8007720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:51.167 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:51.152+0000 7fe6f7fff700 1 --2- 172.21.15.42:0/483761871 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe6f80a4c40 0x7fe6f80a5070 secure :-1 s=THROTTLE_DONE pgs=2040 cs=0 l=1 rx=0x7fe6f805bbf0 tx=0x7fe6e8007720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:51.167 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:51.152+0000 7fe6effff700 1 -- 172.21.15.42:0/483761871 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6f0010b40 con 0x7fe6f80a4c40 2019-12-05T06:06:51.168 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:51.152+0000 7fe6fd31d700 1 -- 172.21.15.42:0/483761871 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6f8139980 con 0x7fe6f80a4c40 2019-12-05T06:06:51.168 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:51.152+0000 7fe6effff700 1 -- 172.21.15.42:0/483761871 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe6f000f470 con 0x7fe6f80a4c40 2019-12-05T06:06:51.168 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:51.152+0000 7fe6fd31d700 1 -- 172.21.15.42:0/483761871 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6f8139f00 con 0x7fe6f80a4c40 2019-12-05T06:06:51.168 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:51.152+0000 7fe6f7fff700 1 --2- 172.21.15.42:0/483761871 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe6f80a4c40 0x7fe6f80a5070 secure :-1 s=THROTTLE_DONE pgs=2040 cs=0 l=1 rx=0x7fe6f805bbf0 tx=0x7fe6e8007720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:51.168 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:51.152+0000 7fe6effff700 1 -- 172.21.15.42:0/483761871 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fe6f000f710 con 0x7fe6f80a4c40 2019-12-05T06:06:51.168 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:51.152+0000 7fe6f7fff700 1 --2- 172.21.15.42:0/483761871 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe6f80a4c40 0x7fe6f80a5070 secure :-1 s=THROTTLE_DONE pgs=2040 cs=0 l=1 rx=0x7fe6f805bbf0 tx=0x7fe6e8007720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:51.168 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:51.152+0000 7fe6effff700 1 --2- 172.21.15.42:0/483761871 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe6dc03ac40 0x7fe6dc03d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:51.169 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:51.152+0000 7fe6effff700 1 -- 172.21.15.42:0/483761871 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9729+0+0 (secure 0 0 0) 0x7fe6f008f000 con 0x7fe6f80a4c40 2019-12-05T06:06:51.169 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:51.152+0000 7fe6f77fe700 1 --2- 172.21.15.42:0/483761871 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe6dc03ac40 0x7fe6dc03d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:51.169 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:51.152+0000 7fe6f77fe700 1 --2- 172.21.15.42:0/483761871 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe6dc03ac40 0x7fe6dc03d150 crc :-1 s=READY pgs=867 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:51.169 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:51.156+0000 7fe6fd31d700 1 -- 172.21.15.42:0/483761871 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6f8121830 con 0x7fe6f80a4c40 2019-12-05T06:06:51.170 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:51.160+0000 7fe6f7fff700 1 --2- 172.21.15.42:0/483761871 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe6f80a4c40 0x7fe6f80a5070 secure :-1 s=THROTTLE_DONE pgs=2040 cs=0 l=1 rx=0x7fe6f805bbf0 tx=0x7fe6e8007720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:51.170 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:51.160+0000 7fe6effff700 1 -- 172.21.15.42:0/483761871 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fe6f0076060 con 0x7fe6f80a4c40 2019-12-05T06:06:51.326 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:51.320+0000 7fe6fd31d700 1 -- 172.21.15.42:0/483761871 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe6f805d110 con 0x7fe6f80a4c40 2019-12-05T06:06:51.327 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:51.324+0000 7fe6f7fff700 1 --2- 172.21.15.42:0/483761871 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe6f80a4c40 0x7fe6f80a5070 secure :-1 s=THROTTLE_DONE pgs=2040 cs=0 l=1 rx=0x7fe6f805bbf0 tx=0x7fe6e8007720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:51.327 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:51.324+0000 7fe6effff700 1 -- 172.21.15.42:0/483761871 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 248 v248) v1 ==== 95+0+2112 (secure 0 0 0) 0x7fe6f00762c0 con 0x7fe6f80a4c40 2019-12-05T06:06:51.330 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 248 2019-12-05T06:06:51.330 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:06:51.330 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":248,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":9573,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/2968227232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2968227232},{"type":"v1","addr":"172.21.15.40:6833","nonce":2968227232}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":248}],"filesystems":[{"mdsmap":{"epoch":247,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:05:33.997161+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":92,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6821},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6821":{"gid":6821,"name":"a","rank":0,"incarnation":244,"state":"up:active","state_seq":23,"addr":"172.21.15.42:6835/2253267554","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2253267554},{"type":"v1","addr":"172.21.15.42:6835","nonce":2253267554}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:06:51.334 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:51.328+0000 7fe6fd31d700 1 -- 172.21.15.42:0/483761871 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe6dc03ac40 msgr2=0x7fe6dc03d150 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:51.334 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:51.328+0000 7fe6fd31d700 1 --2- 172.21.15.42:0/483761871 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe6dc03ac40 0x7fe6dc03d150 crc :-1 s=READY pgs=867 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:51.334 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:51.328+0000 7fe6fd31d700 1 -- 172.21.15.42:0/483761871 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe6f80a4c40 msgr2=0x7fe6f80a5070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:51.334 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:51.328+0000 7fe6fd31d700 1 --2- 172.21.15.42:0/483761871 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe6f80a4c40 0x7fe6f80a5070 secure :-1 s=READY pgs=2040 cs=0 l=1 rx=0x7fe6f805bbf0 tx=0x7fe6e8007720).stop 2019-12-05T06:06:51.334 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:51.328+0000 7fe6fd31d700 1 -- 172.21.15.42:0/483761871 shutdown_connections 2019-12-05T06:06:51.334 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:51.328+0000 7fe6fd31d700 1 --2- 172.21.15.42:0/483761871 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe6dc03ac40 0x7fe6dc03d150 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:51.335 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:51.328+0000 7fe6fd31d700 1 --2- 172.21.15.42:0/483761871 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe6f80a4c40 0x7fe6f80a5070 unknown :-1 s=CLOSED pgs=2040 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:51.335 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:51.328+0000 7fe6fd31d700 1 -- 172.21.15.42:0/483761871 shutdown_connections 2019-12-05T06:06:51.335 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:51.328+0000 7fe6fd31d700 1 -- 172.21.15.42:0/483761871 wait complete. 2019-12-05T06:06:51.335 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:51.328+0000 7fe6fd31d700 1 -- 172.21.15.42:0/483761871 >> 172.21.15.42:0/483761871 conn(0x7fe6f80aabe0 msgr2=0x7fe6f80a9790 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:06:51.349 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:06:53.351 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:06:53.459 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:53.448+0000 7fa8af18b700 1 Processor -- start 2019-12-05T06:06:53.460 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:53.448+0000 7fa8af18b700 1 -- start start 2019-12-05T06:06:53.460 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:53.448+0000 7fa8af18b700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fa8a80a7740 0x7fa8a808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:06:53.460 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:53.448+0000 7fa8af18b700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8a80a0710 con 0x7fa8a809fae0 2019-12-05T06:06:53.460 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:53.448+0000 7fa8af18b700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8a80a9ba0 con 0x7fa8a807e720 2019-12-05T06:06:53.461 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:53.448+0000 7fa8af18b700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8a80a0b00 con 0x7fa8a80a7740 2019-12-05T06:06:53.461 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:53.448+0000 7fa8ad728700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fa8a80a7740 0x7fa8a808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:53.461 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:53.448+0000 7fa8acf27700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fa8a807e720 0x7fa8a809f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:48442/0 (socket says 172.21.15.42:48442) 2019-12-05T06:06:53.461 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:53.448+0000 7fa8ad728700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fa8a80a7740 0x7fa8a808df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:48406/0 (socket says 172.21.15.42:48406) 2019-12-05T06:06:53.461 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:53.448+0000 7fa8acf27700 1 -- 172.21.15.42:0/81415482 learned_addr learned my addr 172.21.15.42:0/81415482 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:06:53.462 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:53.452+0000 7fa89bfff700 1 -- 172.21.15.42:0/81415482 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2800687714 0 0) 0x7fa8a80a9ba0 con 0x7fa8a807e720 2019-12-05T06:06:53.462 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:53.452+0000 7fa89bfff700 1 -- 172.21.15.42:0/81415482 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8940030b0 con 0x7fa8a807e720 2019-12-05T06:06:53.462 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:53.452+0000 7fa8ad728700 1 -- 172.21.15.42:0/81415482 >> v1:172.21.15.42:6789/0 conn(0x7fa8a807e720 legacy=0x7fa8a809f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:53.462 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:53.452+0000 7fa8ad728700 1 -- 172.21.15.42:0/81415482 >> v1:172.21.15.40:6789/0 conn(0x7fa8a809fae0 legacy=0x7fa8a814efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:53.463 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:53.452+0000 7fa89bfff700 1 -- 172.21.15.42:0/81415482 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 217907556 0 0) 0x7fa8a80a0710 con 0x7fa8a809fae0 2019-12-05T06:06:53.463 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:53.452+0000 7fa8ad728700 1 -- 172.21.15.42:0/81415482 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8a807eb50 con 0x7fa8a80a7740 2019-12-05T06:06:53.463 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:53.452+0000 7fa8ad728700 1 --2- 172.21.15.42:0/81415482 >> v2:172.21.15.42:3300/0 conn(0x7fa8a80a7740 0x7fa8a808df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fa89c008790 tx=0x7fa89c00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:53.463 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:53.452+0000 7fa8ad728700 1 --2- 172.21.15.42:0/81415482 >> v2:172.21.15.42:3300/0 conn(0x7fa8a80a7740 0x7fa8a808df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fa89c008790 tx=0x7fa89c00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:53.465 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:53.452+0000 7fa8ad728700 1 --2- 172.21.15.42:0/81415482 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa8a80a7740 0x7fa8a808df80 secure :-1 s=READY pgs=2044 cs=0 l=1 rx=0x7fa89c008790 tx=0x7fa89c00fbf0).ready entity=mon.1 client_cookie=6af50e085105dc44 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:53.465 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:53.452+0000 7fa8ad728700 1 --2- 172.21.15.42:0/81415482 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa8a80a7740 0x7fa8a808df80 secure :-1 s=THROTTLE_DONE pgs=2044 cs=0 l=1 rx=0x7fa89c008790 tx=0x7fa89c00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:53.465 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:53.452+0000 7fa89bfff700 1 -- 172.21.15.42:0/81415482 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa89c011270 con 0x7fa8a80a7740 2019-12-05T06:06:53.465 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:53.452+0000 7fa8ad728700 1 --2- 172.21.15.42:0/81415482 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa8a80a7740 0x7fa8a808df80 secure :-1 s=THROTTLE_DONE pgs=2044 cs=0 l=1 rx=0x7fa89c008790 tx=0x7fa89c00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:53.465 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:53.452+0000 7fa8ad728700 1 --2- 172.21.15.42:0/81415482 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa8a80a7740 0x7fa8a808df80 secure :-1 s=THROTTLE_DONE pgs=2044 cs=0 l=1 rx=0x7fa89c008790 tx=0x7fa89c00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:53.466 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:53.452+0000 7fa89bfff700 1 -- 172.21.15.42:0/81415482 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa89c008bc0 con 0x7fa8a80a7740 2019-12-05T06:06:53.466 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:53.452+0000 7fa89bfff700 1 -- 172.21.15.42:0/81415482 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa89c011850 con 0x7fa8a80a7740 2019-12-05T06:06:53.466 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:53.452+0000 7fa8af18b700 1 -- 172.21.15.42:0/81415482 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa8a80a7740 msgr2=0x7fa8a808df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:53.466 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:53.452+0000 7fa8af18b700 1 --2- 172.21.15.42:0/81415482 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa8a80a7740 0x7fa8a808df80 secure :-1 s=READY pgs=2044 cs=0 l=1 rx=0x7fa89c008790 tx=0x7fa89c00fbf0).stop 2019-12-05T06:06:53.467 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:53.452+0000 7fa8af18b700 1 -- 172.21.15.42:0/81415482 shutdown_connections 2019-12-05T06:06:53.467 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:53.452+0000 7fa8af18b700 1 --2- 172.21.15.42:0/81415482 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa8a80a7740 0x7fa8a808df80 unknown :-1 s=CLOSED pgs=2044 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:53.467 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:53.452+0000 7fa8af18b700 1 -- 172.21.15.42:0/81415482 shutdown_connections 2019-12-05T06:06:53.468 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:53.452+0000 7fa8af18b700 1 -- 172.21.15.42:0/81415482 wait complete. 2019-12-05T06:06:53.468 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:53.452+0000 7fa8af18b700 1 -- 172.21.15.42:0/81415482 >> 172.21.15.42:0/81415482 conn(0x7fa8a80aabe0 msgr2=0x7fa8a80a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:06:53.472 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:53.456+0000 7fa8af18b700 1 Processor -- start 2019-12-05T06:06:53.472 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:53.456+0000 7fa8af18b700 1 -- start start 2019-12-05T06:06:53.472 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:53.456+0000 7fa8af18b700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa8a811ad90 0x7fa8a807a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:53.472 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:53.456+0000 7fa8af18b700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8a80a4390 con 0x7fa8a80789a0 2019-12-05T06:06:53.472 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:53.456+0000 7fa8af18b700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8a807b2d0 con 0x7fa8a8078570 2019-12-05T06:06:53.472 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:53.456+0000 7fa8af18b700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8a8079220 con 0x7fa8a811ad90 2019-12-05T06:06:53.473 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:53.456+0000 7fa8acf27700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fa8a8078570 0x7fa8a8122f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:48450/0 (socket says 172.21.15.42:48450) 2019-12-05T06:06:53.473 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:53.456+0000 7fa8acf27700 1 -- 172.21.15.42:0/3667528479 learned_addr learned my addr 172.21.15.42:0/3667528479 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:06:53.473 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:53.456+0000 7fa8ad728700 1 --2- 172.21.15.42:0/3667528479 >> v2:172.21.15.40:3300/0 conn(0x7fa8a811ad90 0x7fa8a807a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:53.473 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:53.456+0000 7fa899ffb700 1 -- 172.21.15.42:0/3667528479 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2615713418 0 0) 0x7fa8a807b2d0 con 0x7fa8a8078570 2019-12-05T06:06:53.473 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:53.456+0000 7fa899ffb700 1 -- 172.21.15.42:0/3667528479 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8880030b0 con 0x7fa8a8078570 2019-12-05T06:06:53.473 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:53.456+0000 7fa8ad728700 1 -- 172.21.15.42:0/3667528479 >> v1:172.21.15.42:6789/0 conn(0x7fa8a8078570 legacy=0x7fa8a8122f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:53.473 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:53.456+0000 7fa8ad728700 1 -- 172.21.15.42:0/3667528479 >> v1:172.21.15.40:6789/0 conn(0x7fa8a80789a0 legacy=0x7fa8a812d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:53.474 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:53.456+0000 7fa8ad728700 1 -- 172.21.15.42:0/3667528479 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8a8123760 con 0x7fa8a811ad90 2019-12-05T06:06:53.474 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:53.456+0000 7fa8ad728700 1 --2- 172.21.15.42:0/3667528479 >> v2:172.21.15.40:3300/0 conn(0x7fa8a811ad90 0x7fa8a807a100 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fa89c008790 tx=0x7fa89c008df0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:53.474 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:53.456+0000 7fa8ad728700 1 --2- 172.21.15.42:0/3667528479 >> v2:172.21.15.40:3300/0 conn(0x7fa8a811ad90 0x7fa8a807a100 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fa89c008790 tx=0x7fa89c008df0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:53.474 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:53.456+0000 7fa8ad728700 1 --2- 172.21.15.42:0/3667528479 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa8a811ad90 0x7fa8a807a100 secure :-1 s=READY pgs=1559 cs=0 l=1 rx=0x7fa89c008790 tx=0x7fa89c008df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:53.475 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:53.456+0000 7fa8ad728700 1 --2- 172.21.15.42:0/3667528479 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa8a811ad90 0x7fa8a807a100 secure :-1 s=THROTTLE_DONE pgs=1559 cs=0 l=1 rx=0x7fa89c008790 tx=0x7fa89c008df0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:53.475 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:53.456+0000 7fa899ffb700 1 -- 172.21.15.42:0/3667528479 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa89c007150 con 0x7fa8a811ad90 2019-12-05T06:06:53.475 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:53.456+0000 7fa8af18b700 1 -- 172.21.15.42:0/3667528479 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8a807a650 con 0x7fa8a811ad90 2019-12-05T06:06:53.475 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:53.456+0000 7fa8ad728700 1 --2- 172.21.15.42:0/3667528479 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa8a811ad90 0x7fa8a807a100 secure :-1 s=THROTTLE_DONE pgs=1559 cs=0 l=1 rx=0x7fa89c008790 tx=0x7fa89c008df0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:53.476 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:53.456+0000 7fa899ffb700 1 -- 172.21.15.42:0/3667528479 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa89c011470 con 0x7fa8a811ad90 2019-12-05T06:06:53.476 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:53.456+0000 7fa8ad728700 1 --2- 172.21.15.42:0/3667528479 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa8a811ad90 0x7fa8a807a100 secure :-1 s=THROTTLE_DONE pgs=1559 cs=0 l=1 rx=0x7fa89c008790 tx=0x7fa89c008df0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:53.476 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:53.456+0000 7fa8af18b700 1 -- 172.21.15.42:0/3667528479 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8a812f180 con 0x7fa8a811ad90 2019-12-05T06:06:53.476 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:53.456+0000 7fa899ffb700 1 -- 172.21.15.42:0/3667528479 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa89c00f550 con 0x7fa8a811ad90 2019-12-05T06:06:53.476 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:53.460+0000 7fa8ad728700 1 --2- 172.21.15.42:0/3667528479 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa8a811ad90 0x7fa8a807a100 secure :-1 s=THROTTLE_DONE pgs=1559 cs=0 l=1 rx=0x7fa89c008790 tx=0x7fa89c008df0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:53.476 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:53.460+0000 7fa899ffb700 1 -- 172.21.15.42:0/3667528479 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fa89c005540 con 0x7fa8a811ad90 2019-12-05T06:06:53.476 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:53.460+0000 7fa8ad728700 1 --2- 172.21.15.42:0/3667528479 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa8a811ad90 0x7fa8a807a100 secure :-1 s=THROTTLE_DONE pgs=1559 cs=0 l=1 rx=0x7fa89c008790 tx=0x7fa89c008df0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:53.477 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:53.460+0000 7fa899ffb700 1 --2- 172.21.15.42:0/3667528479 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa88803b210 0x7fa88803d720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:53.477 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:53.460+0000 7fa899ffb700 1 -- 172.21.15.42:0/3667528479 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9729+0+0 (secure 0 0 0) 0x7fa89c04bbc0 con 0x7fa8a811ad90 2019-12-05T06:06:53.477 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:53.460+0000 7fa8acf27700 1 --2- 172.21.15.42:0/3667528479 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa88803b210 0x7fa88803d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:53.477 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:53.460+0000 7fa8acf27700 1 --2- 172.21.15.42:0/3667528479 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa88803b210 0x7fa88803d720 crc :-1 s=READY pgs=868 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:53.477 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:53.464+0000 7fa8af18b700 1 -- 172.21.15.42:0/3667528479 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8a805d110 con 0x7fa8a811ad90 2019-12-05T06:06:53.478 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:53.472+0000 7fa8ad728700 1 --2- 172.21.15.42:0/3667528479 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa8a811ad90 0x7fa8a807a100 secure :-1 s=THROTTLE_DONE pgs=1559 cs=0 l=1 rx=0x7fa89c008790 tx=0x7fa89c008df0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:53.478 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:53.472+0000 7fa899ffb700 1 -- 172.21.15.42:0/3667528479 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fa89c032c20 con 0x7fa8a811ad90 2019-12-05T06:06:53.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:53.644+0000 7fa8af18b700 1 -- 172.21.15.42:0/3667528479 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa8a807e180 con 0x7fa8a811ad90 2019-12-05T06:06:53.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:53.648+0000 7fa8ad728700 1 --2- 172.21.15.42:0/3667528479 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa8a811ad90 0x7fa8a807a100 secure :-1 s=THROTTLE_DONE pgs=1559 cs=0 l=1 rx=0x7fa89c008790 tx=0x7fa89c008df0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:53.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:53.648+0000 7fa899ffb700 1 -- 172.21.15.42:0/3667528479 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 248 v248) v1 ==== 95+0+2112 (secure 0 0 0) 0x7fa89c032970 con 0x7fa8a811ad90 2019-12-05T06:06:53.654 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 248 2019-12-05T06:06:53.654 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:06:53.654 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":248,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":9573,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/2968227232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2968227232},{"type":"v1","addr":"172.21.15.40:6833","nonce":2968227232}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":248}],"filesystems":[{"mdsmap":{"epoch":247,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:05:33.997161+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":92,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6821},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6821":{"gid":6821,"name":"a","rank":0,"incarnation":244,"state":"up:active","state_seq":23,"addr":"172.21.15.42:6835/2253267554","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2253267554},{"type":"v1","addr":"172.21.15.42:6835","nonce":2253267554}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:06:53.657 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:53.652+0000 7fa8af18b700 1 -- 172.21.15.42:0/3667528479 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa88803b210 msgr2=0x7fa88803d720 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:53.657 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:53.652+0000 7fa8af18b700 1 --2- 172.21.15.42:0/3667528479 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa88803b210 0x7fa88803d720 crc :-1 s=READY pgs=868 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:53.657 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:53.652+0000 7fa8af18b700 1 -- 172.21.15.42:0/3667528479 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa8a811ad90 msgr2=0x7fa8a807a100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:53.657 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:53.652+0000 7fa8af18b700 1 --2- 172.21.15.42:0/3667528479 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa8a811ad90 0x7fa8a807a100 secure :-1 s=READY pgs=1559 cs=0 l=1 rx=0x7fa89c008790 tx=0x7fa89c008df0).stop 2019-12-05T06:06:53.658 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:53.652+0000 7fa8af18b700 1 -- 172.21.15.42:0/3667528479 shutdown_connections 2019-12-05T06:06:53.658 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:53.652+0000 7fa8af18b700 1 --2- 172.21.15.42:0/3667528479 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa88803b210 0x7fa88803d720 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:53.658 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:53.652+0000 7fa8af18b700 1 --2- 172.21.15.42:0/3667528479 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa8a811ad90 0x7fa8a807a100 unknown :-1 s=CLOSED pgs=1559 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:53.658 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:53.652+0000 7fa8af18b700 1 -- 172.21.15.42:0/3667528479 shutdown_connections 2019-12-05T06:06:53.658 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:53.652+0000 7fa8af18b700 1 -- 172.21.15.42:0/3667528479 wait complete. 2019-12-05T06:06:53.658 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:53.652+0000 7fa8af18b700 1 -- 172.21.15.42:0/3667528479 >> 172.21.15.42:0/3667528479 conn(0x7fa8a807ee20 msgr2=0x7fa8a808ef30 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:06:53.667 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:06:55.668 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:06:55.783 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:55.772+0000 7fab6ec4e700 1 Processor -- start 2019-12-05T06:06:55.783 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:55.772+0000 7fab6ec4e700 1 -- start start 2019-12-05T06:06:55.783 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:55.776+0000 7fab6ec4e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fab6808df80 0x7fab6809f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:06:55.783 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:55.776+0000 7fab6ec4e700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fab6809f830 0x7fab6814efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:06:55.784 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:55.776+0000 7fab6ec4e700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab6809fc60 con 0x7fab680a7740 2019-12-05T06:06:55.784 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:55.776+0000 7fab6ec4e700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab680a0b00 con 0x7fab6808df80 2019-12-05T06:06:55.784 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:55.776+0000 7fab6ec4e700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab6808e5b0 con 0x7fab6809f830 2019-12-05T06:06:55.784 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:55.776+0000 7fab6c9ea700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fab6808df80 0x7fab6809f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:55.784 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:55.776+0000 7fab6c9ea700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fab6808df80 0x7fab6809f2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:48422/0 (socket says 172.21.15.42:48422) 2019-12-05T06:06:55.784 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:55.776+0000 7fab6d1eb700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fab680a7740 0x7fab6807e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:48458/0 (socket says 172.21.15.42:48458) 2019-12-05T06:06:55.784 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:55.776+0000 7fab6c9ea700 1 -- 172.21.15.42:0/2442249154 learned_addr learned my addr 172.21.15.42:0/2442249154 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:06:55.785 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:55.776+0000 7fab6d9ec700 1 --2- 172.21.15.42:0/2442249154 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fab6809f830 0x7fab6814efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:55.785 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:55.776+0000 7fab6c9ea700 1 -- 172.21.15.42:0/2442249154 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fab6809f830 msgr2=0x7fab6814efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:06:55.785 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:55.776+0000 7fab6c9ea700 1 --2- 172.21.15.42:0/2442249154 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fab6809f830 0x7fab6814efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:06:55.789 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:55.776+0000 7fab6c9ea700 1 -- 172.21.15.42:0/2442249154 >> v1:172.21.15.42:6789/0 conn(0x7fab680a7740 legacy=0x7fab6807e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:55.789 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:55.776+0000 7fab6c9ea700 1 -- 172.21.15.42:0/2442249154 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab680a0710 con 0x7fab6808df80 2019-12-05T06:06:55.789 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:55.776+0000 7fab6d9ec700 1 --2- 172.21.15.42:0/2442249154 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fab6809f830 0x7fab6814efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).send_auth_request state changed! 2019-12-05T06:06:55.790 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:55.776+0000 7fab6c9ea700 1 --2- 172.21.15.42:0/2442249154 >> v2:172.21.15.42:3300/0 conn(0x7fab6808df80 0x7fab6809f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fab5c008790 tx=0x7fab5c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:55.790 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:55.776+0000 7fab6c9ea700 1 --2- 172.21.15.42:0/2442249154 >> v2:172.21.15.42:3300/0 conn(0x7fab6808df80 0x7fab6809f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fab5c008790 tx=0x7fab5c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:55.790 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:55.776+0000 7fab6c9ea700 1 --2- 172.21.15.42:0/2442249154 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fab6808df80 0x7fab6809f2e0 secure :-1 s=READY pgs=2047 cs=0 l=1 rx=0x7fab5c008790 tx=0x7fab5c00fd30).ready entity=mon.1 client_cookie=c257db7aa2b878bf server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:55.790 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:55.776+0000 7fab6c9ea700 1 --2- 172.21.15.42:0/2442249154 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fab6808df80 0x7fab6809f2e0 secure :-1 s=THROTTLE_DONE pgs=2047 cs=0 l=1 rx=0x7fab5c008790 tx=0x7fab5c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:55.790 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:55.776+0000 7fab5bfff700 1 -- 172.21.15.42:0/2442249154 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fab5c008c90 con 0x7fab6808df80 2019-12-05T06:06:55.790 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:55.776+0000 7fab6c9ea700 1 --2- 172.21.15.42:0/2442249154 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fab6808df80 0x7fab6809f2e0 secure :-1 s=THROTTLE_DONE pgs=2047 cs=0 l=1 rx=0x7fab5c008790 tx=0x7fab5c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:55.790 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:55.776+0000 7fab6c9ea700 1 --2- 172.21.15.42:0/2442249154 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fab6808df80 0x7fab6809f2e0 secure :-1 s=THROTTLE_DONE pgs=2047 cs=0 l=1 rx=0x7fab5c008790 tx=0x7fab5c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:55.791 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:55.776+0000 7fab5bfff700 1 -- 172.21.15.42:0/2442249154 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab5c0088b0 con 0x7fab6808df80 2019-12-05T06:06:55.791 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:55.776+0000 7fab5bfff700 1 -- 172.21.15.42:0/2442249154 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fab5c011b10 con 0x7fab6808df80 2019-12-05T06:06:55.791 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:55.776+0000 7fab6ec4e700 1 -- 172.21.15.42:0/2442249154 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fab6808df80 msgr2=0x7fab6809f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:55.791 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:55.776+0000 7fab6ec4e700 1 --2- 172.21.15.42:0/2442249154 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fab6808df80 0x7fab6809f2e0 secure :-1 s=READY pgs=2047 cs=0 l=1 rx=0x7fab5c008790 tx=0x7fab5c00fd30).stop 2019-12-05T06:06:55.791 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:55.776+0000 7fab6ec4e700 1 -- 172.21.15.42:0/2442249154 shutdown_connections 2019-12-05T06:06:55.791 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:55.776+0000 7fab6ec4e700 1 --2- 172.21.15.42:0/2442249154 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fab6809f830 0x7fab6814efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:06:55.792 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:55.776+0000 7fab6ec4e700 1 --2- 172.21.15.42:0/2442249154 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fab6808df80 0x7fab6809f2e0 unknown :-1 s=CLOSED pgs=2047 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:55.792 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:55.776+0000 7fab6ec4e700 1 -- 172.21.15.42:0/2442249154 shutdown_connections 2019-12-05T06:06:55.792 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:55.776+0000 7fab6ec4e700 1 -- 172.21.15.42:0/2442249154 wait complete. 2019-12-05T06:06:55.792 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:55.776+0000 7fab6ec4e700 1 -- 172.21.15.42:0/2442249154 >> 172.21.15.42:0/2442249154 conn(0x7fab680aabe0 msgr2=0x7fab680a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:06:55.795 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:55.780+0000 7fab6ec4e700 1 Processor -- start 2019-12-05T06:06:55.795 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:55.780+0000 7fab6ec4e700 1 -- start start 2019-12-05T06:06:55.795 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:55.780+0000 7fab6ec4e700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fab680a9ad0 0x7fab6807b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:55.795 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:55.780+0000 7fab6ec4e700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fab6807bb10 0x7fab6812dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:55.795 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:55.780+0000 7fab6ec4e700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab6805f190 con 0x7fab680616a0 2019-12-05T06:06:55.796 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:55.780+0000 7fab6ec4e700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab6805eec0 con 0x7fab680a9ad0 2019-12-05T06:06:55.796 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:55.780+0000 7fab6ec4e700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab68061160 con 0x7fab6807bb10 2019-12-05T06:06:55.796 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:55.780+0000 7fab6d1eb700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fab680616a0 0x7fab680a3950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:48464/0 (socket says 172.21.15.42:48464) 2019-12-05T06:06:55.796 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:55.780+0000 7fab6d1eb700 1 -- 172.21.15.42:0/1287373649 learned_addr learned my addr 172.21.15.42:0/1287373649 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:06:55.796 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:55.780+0000 7fab6d9ec700 1 --2- 172.21.15.42:0/1287373649 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fab6807bb10 0x7fab6812dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:55.796 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:55.780+0000 7fab6c9ea700 1 --2- 172.21.15.42:0/1287373649 >> v2:172.21.15.40:3300/0 conn(0x7fab680a9ad0 0x7fab6807b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:55.796 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:55.780+0000 7fab59ffb700 1 -- 172.21.15.42:0/1287373649 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 751538898 0 0) 0x7fab6805f190 con 0x7fab680616a0 2019-12-05T06:06:55.797 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:55.780+0000 7fab59ffb700 1 -- 172.21.15.42:0/1287373649 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab480030b0 con 0x7fab680616a0 2019-12-05T06:06:55.797 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:55.780+0000 7fab6d9ec700 1 -- 172.21.15.42:0/1287373649 >> v2:172.21.15.40:3300/0 conn(0x7fab680a9ad0 msgr2=0x7fab6807b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:55.797 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:55.780+0000 7fab6d9ec700 1 --2- 172.21.15.42:0/1287373649 >> v2:172.21.15.40:3300/0 conn(0x7fab680a9ad0 0x7fab6807b5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:55.797 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:55.780+0000 7fab6d9ec700 1 -- 172.21.15.42:0/1287373649 >> v1:172.21.15.42:6789/0 conn(0x7fab680616a0 legacy=0x7fab680a3950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:55.797 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:55.780+0000 7fab59ffb700 1 -- 172.21.15.42:0/1287373649 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 1576441277 0 0) 0x7fab480030b0 con 0x7fab680616a0 2019-12-05T06:06:55.797 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:55.780+0000 7fab6d9ec700 1 -- 172.21.15.42:0/1287373649 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab6814ed10 con 0x7fab6807bb10 2019-12-05T06:06:55.797 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:55.780+0000 7fab6d9ec700 1 --2- 172.21.15.42:0/1287373649 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fab6807bb10 0x7fab6812dde0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fab64001c50 tx=0x7fab64001c90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:55.798 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:55.780+0000 7fab6d9ec700 1 --2- 172.21.15.42:0/1287373649 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fab6807bb10 0x7fab6812dde0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fab64001c50 tx=0x7fab64001c90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:55.798 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:55.780+0000 7fab6d9ec700 1 --2- 172.21.15.42:0/1287373649 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fab6807bb10 0x7fab6812dde0 secure :-1 s=READY pgs=455 cs=0 l=1 rx=0x7fab64001c50 tx=0x7fab64001c90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:55.798 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:55.780+0000 7fab6d9ec700 1 --2- 172.21.15.42:0/1287373649 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fab6807bb10 0x7fab6812dde0 secure :-1 s=THROTTLE_DONE pgs=455 cs=0 l=1 rx=0x7fab64001c50 tx=0x7fab64001c90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:55.798 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:55.784+0000 7fab59ffb700 1 -- 172.21.15.42:0/1287373649 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fab64001a10 con 0x7fab6807bb10 2019-12-05T06:06:55.798 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:55.784+0000 7fab6d9ec700 1 --2- 172.21.15.42:0/1287373649 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fab6807bb10 0x7fab6812dde0 secure :-1 s=THROTTLE_DONE pgs=455 cs=0 l=1 rx=0x7fab64001c50 tx=0x7fab64001c90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:55.798 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:55.784+0000 7fab6d9ec700 1 --2- 172.21.15.42:0/1287373649 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fab6807bb10 0x7fab6812dde0 secure :-1 s=THROTTLE_DONE pgs=455 cs=0 l=1 rx=0x7fab64001c50 tx=0x7fab64001c90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:55.798 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:55.784+0000 7fab59ffb700 1 -- 172.21.15.42:0/1287373649 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab64001820 con 0x7fab6807bb10 2019-12-05T06:06:55.799 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:55.784+0000 7fab6ec4e700 1 -- 172.21.15.42:0/1287373649 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab6805f890 con 0x7fab6807bb10 2019-12-05T06:06:55.799 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:55.784+0000 7fab59ffb700 1 -- 172.21.15.42:0/1287373649 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fab6000e570 con 0x7fab6807bb10 2019-12-05T06:06:55.799 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:55.784+0000 7fab6ec4e700 1 -- 172.21.15.42:0/1287373649 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab680a1510 con 0x7fab6807bb10 2019-12-05T06:06:55.799 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:55.784+0000 7fab6d9ec700 1 --2- 172.21.15.42:0/1287373649 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fab6807bb10 0x7fab6812dde0 secure :-1 s=THROTTLE_DONE pgs=455 cs=0 l=1 rx=0x7fab64001c50 tx=0x7fab64001c90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:55.799 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:55.784+0000 7fab59ffb700 1 -- 172.21.15.42:0/1287373649 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fab6000e840 con 0x7fab6807bb10 2019-12-05T06:06:55.813 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:55.784+0000 7fab6d9ec700 1 --2- 172.21.15.42:0/1287373649 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fab6807bb10 0x7fab6812dde0 secure :-1 s=THROTTLE_DONE pgs=455 cs=0 l=1 rx=0x7fab64001c50 tx=0x7fab64001c90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:55.813 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:55.784+0000 7fab59ffb700 1 --2- 172.21.15.42:0/1287373649 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fab4803b280 0x7fab4803d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:55.813 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:55.784+0000 7fab59ffb700 1 -- 172.21.15.42:0/1287373649 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9729+0+0 (secure 0 0 0) 0x7fab6004aaf0 con 0x7fab6807bb10 2019-12-05T06:06:55.813 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:55.784+0000 7fab6d1eb700 1 --2- 172.21.15.42:0/1287373649 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fab4803b280 0x7fab4803d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:55.829 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:55.784+0000 7fab6d1eb700 1 --2- 172.21.15.42:0/1287373649 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fab4803b280 0x7fab4803d790 crc :-1 s=READY pgs=869 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:55.829 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:55.788+0000 7fab6ec4e700 1 -- 172.21.15.42:0/1287373649 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab6805d110 con 0x7fab6807bb10 2019-12-05T06:06:55.830 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:55.800+0000 7fab6d9ec700 1 --2- 172.21.15.42:0/1287373649 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fab6807bb10 0x7fab6812dde0 secure :-1 s=THROTTLE_DONE pgs=455 cs=0 l=1 rx=0x7fab64001c50 tx=0x7fab64001c90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:55.830 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:55.800+0000 7fab59ffb700 1 -- 172.21.15.42:0/1287373649 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fab60031b10 con 0x7fab6807bb10 2019-12-05T06:06:55.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:55.968+0000 7fab6ec4e700 1 -- 172.21.15.42:0/1287373649 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fab6805b8c0 con 0x7fab6807bb10 2019-12-05T06:06:55.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:55.968+0000 7fab6d9ec700 1 --2- 172.21.15.42:0/1287373649 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fab6807bb10 0x7fab6812dde0 secure :-1 s=THROTTLE_DONE pgs=455 cs=0 l=1 rx=0x7fab64001c50 tx=0x7fab64001c90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:55.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:55.972+0000 7fab59ffb700 1 -- 172.21.15.42:0/1287373649 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 248 v248) v1 ==== 95+0+2112 (secure 0 0 0) 0x7fab60031d70 con 0x7fab6807bb10 2019-12-05T06:06:55.978 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 248 2019-12-05T06:06:55.978 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:06:55.978 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":248,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":9573,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/2968227232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2968227232},{"type":"v1","addr":"172.21.15.40:6833","nonce":2968227232}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":248}],"filesystems":[{"mdsmap":{"epoch":247,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:05:33.997161+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":92,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6821},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6821":{"gid":6821,"name":"a","rank":0,"incarnation":244,"state":"up:active","state_seq":23,"addr":"172.21.15.42:6835/2253267554","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2253267554},{"type":"v1","addr":"172.21.15.42:6835","nonce":2253267554}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:06:55.980 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:55.976+0000 7fab6ec4e700 1 -- 172.21.15.42:0/1287373649 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fab4803b280 msgr2=0x7fab4803d790 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:55.980 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:55.976+0000 7fab6ec4e700 1 --2- 172.21.15.42:0/1287373649 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fab4803b280 0x7fab4803d790 crc :-1 s=READY pgs=869 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:55.980 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:55.976+0000 7fab6ec4e700 1 -- 172.21.15.42:0/1287373649 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fab6807bb10 msgr2=0x7fab6812dde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:55.981 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:55.976+0000 7fab6ec4e700 1 --2- 172.21.15.42:0/1287373649 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fab6807bb10 0x7fab6812dde0 secure :-1 s=READY pgs=455 cs=0 l=1 rx=0x7fab64001c50 tx=0x7fab64001c90).stop 2019-12-05T06:06:55.981 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:55.976+0000 7fab6ec4e700 1 -- 172.21.15.42:0/1287373649 shutdown_connections 2019-12-05T06:06:55.981 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:55.976+0000 7fab6ec4e700 1 --2- 172.21.15.42:0/1287373649 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fab4803b280 0x7fab4803d790 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:55.981 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:55.976+0000 7fab6ec4e700 1 --2- 172.21.15.42:0/1287373649 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fab6807bb10 0x7fab6812dde0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:55.981 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:55.976+0000 7fab6ec4e700 1 --2- 172.21.15.42:0/1287373649 >> v2:172.21.15.40:3300/0 conn(0x7fab680a9ad0 0x7fab6807b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:55.982 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:55.976+0000 7fab6ec4e700 1 -- 172.21.15.42:0/1287373649 shutdown_connections 2019-12-05T06:06:55.982 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:55.976+0000 7fab6ec4e700 1 -- 172.21.15.42:0/1287373649 wait complete. 2019-12-05T06:06:55.982 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:55.976+0000 7fab6ec4e700 1 -- 172.21.15.42:0/1287373649 >> 172.21.15.42:0/1287373649 conn(0x7fab6807cf10 msgr2=0x7fab680a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:06:55.990 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:06:57.990 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:06:58.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:58.088+0000 7f9a03121700 1 Processor -- start 2019-12-05T06:06:58.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:58.088+0000 7f9a03121700 1 -- start start 2019-12-05T06:06:58.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:58.092+0000 7f9a03121700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f99fc08df80 0x7f99fc09f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:06:58.100 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:58.092+0000 7f9a03121700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99fc09fc60 con 0x7f99fc0a7740 2019-12-05T06:06:58.100 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:58.092+0000 7f9a03121700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99fc0a0710 con 0x7f99fc09f830 2019-12-05T06:06:58.100 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:58.092+0000 7f9a03121700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99fc0a0b00 con 0x7f99fc08df80 2019-12-05T06:06:58.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:58.092+0000 7f9a00ebd700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f99fc08df80 0x7f99fc09f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:58.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:58.092+0000 7f9a00ebd700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f99fc08df80 0x7f99fc09f2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:48436/0 (socket says 172.21.15.42:48436) 2019-12-05T06:06:58.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:58.092+0000 7f9a00ebd700 1 -- 172.21.15.42:0/1079607217 learned_addr learned my addr 172.21.15.42:0/1079607217 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:06:58.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:58.092+0000 7f9a00ebd700 1 -- 172.21.15.42:0/1079607217 >> v1:172.21.15.42:6789/0 conn(0x7f99fc09f830 legacy=0x7f99fc14efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:58.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:58.092+0000 7f9a00ebd700 1 -- 172.21.15.42:0/1079607217 >> v1:172.21.15.40:6789/0 conn(0x7f99fc0a7740 legacy=0x7f99fc07e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:58.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:58.092+0000 7f9a00ebd700 1 -- 172.21.15.42:0/1079607217 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99fc07ef20 con 0x7f99fc08df80 2019-12-05T06:06:58.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:58.092+0000 7f9a00ebd700 1 --2- 172.21.15.42:0/1079607217 >> v2:172.21.15.42:3300/0 conn(0x7f99fc08df80 0x7f99fc09f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f99ec008790 tx=0x7f99ec00fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:58.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:58.092+0000 7f9a00ebd700 1 --2- 172.21.15.42:0/1079607217 >> v2:172.21.15.42:3300/0 conn(0x7f99fc08df80 0x7f99fc09f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f99ec008790 tx=0x7f99ec00fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:58.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:58.092+0000 7f9a00ebd700 1 --2- 172.21.15.42:0/1079607217 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f99fc08df80 0x7f99fc09f2e0 secure :-1 s=READY pgs=2050 cs=0 l=1 rx=0x7f99ec008790 tx=0x7f99ec00fcb0).ready entity=mon.1 client_cookie=b1fe21a1feae6298 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:58.103 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:58.092+0000 7f9a00ebd700 1 --2- 172.21.15.42:0/1079607217 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f99fc08df80 0x7f99fc09f2e0 secure :-1 s=THROTTLE_DONE pgs=2050 cs=0 l=1 rx=0x7f99ec008790 tx=0x7f99ec00fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:58.103 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:58.092+0000 7f99f3fff700 1 -- 172.21.15.42:0/1079607217 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f99ec011520 con 0x7f99fc08df80 2019-12-05T06:06:58.103 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:58.092+0000 7f9a00ebd700 1 --2- 172.21.15.42:0/1079607217 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f99fc08df80 0x7f99fc09f2e0 secure :-1 s=THROTTLE_DONE pgs=2050 cs=0 l=1 rx=0x7f99ec008790 tx=0x7f99ec00fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:58.103 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:58.092+0000 7f99f3fff700 1 -- 172.21.15.42:0/1079607217 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99ec008ca0 con 0x7f99fc08df80 2019-12-05T06:06:58.103 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:58.092+0000 7f9a00ebd700 1 --2- 172.21.15.42:0/1079607217 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f99fc08df80 0x7f99fc09f2e0 secure :-1 s=THROTTLE_DONE pgs=2050 cs=0 l=1 rx=0x7f99ec008790 tx=0x7f99ec00fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:58.104 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:58.092+0000 7f99f3fff700 1 -- 172.21.15.42:0/1079607217 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f99ec011b00 con 0x7f99fc08df80 2019-12-05T06:06:58.104 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:58.092+0000 7f9a03121700 1 -- 172.21.15.42:0/1079607217 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f99fc08df80 msgr2=0x7f99fc09f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:58.104 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:58.092+0000 7f9a03121700 1 --2- 172.21.15.42:0/1079607217 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f99fc08df80 0x7f99fc09f2e0 secure :-1 s=READY pgs=2050 cs=0 l=1 rx=0x7f99ec008790 tx=0x7f99ec00fcb0).stop 2019-12-05T06:06:58.104 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:58.092+0000 7f9a03121700 1 -- 172.21.15.42:0/1079607217 shutdown_connections 2019-12-05T06:06:58.104 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:58.092+0000 7f9a03121700 1 --2- 172.21.15.42:0/1079607217 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f99fc08df80 0x7f99fc09f2e0 unknown :-1 s=CLOSED pgs=2050 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:58.105 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:58.092+0000 7f9a03121700 1 -- 172.21.15.42:0/1079607217 shutdown_connections 2019-12-05T06:06:58.105 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:58.092+0000 7f9a03121700 1 -- 172.21.15.42:0/1079607217 wait complete. 2019-12-05T06:06:58.105 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:58.092+0000 7f9a03121700 1 -- 172.21.15.42:0/1079607217 >> 172.21.15.42:0/1079607217 conn(0x7f99fc0aabe0 msgr2=0x7f99fc0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:06:58.109 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:58.096+0000 7f9a03121700 1 Processor -- start 2019-12-05T06:06:58.109 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:58.096+0000 7f9a03121700 1 -- start start 2019-12-05T06:06:58.109 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:58.096+0000 7f9a03121700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f99fc0a2a20 0x7f99fc0a3bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:58.109 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:58.096+0000 7f9a03121700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f99fc0a9ab0 0x7f99fc12cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:58.109 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:58.096+0000 7f9a03121700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99fc079730 con 0x7f99fc0a6a20 2019-12-05T06:06:58.110 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:58.096+0000 7f9a03121700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99fc0a1a90 con 0x7f99fc0a9ab0 2019-12-05T06:06:58.110 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:58.096+0000 7f9a03121700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99fc0a2780 con 0x7f99fc0a2a20 2019-12-05T06:06:58.110 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:58.096+0000 7f9a01ebf700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f99fc0a9ab0 0x7f99fc12cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:58.110 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:58.096+0000 7f9a016be700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f99fc0a2a20 0x7f99fc0a3bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:58.110 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:58.096+0000 7f9a01ebf700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f99fc0a9ab0 0x7f99fc12cf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.42:36260/0 (socket says 172.21.15.42:36260) 2019-12-05T06:06:58.110 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:58.096+0000 7f9a01ebf700 1 -- 172.21.15.42:0/2769744599 learned_addr learned my addr 172.21.15.42:0/2769744599 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:06:58.110 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:58.100+0000 7f9a01ebf700 1 -- 172.21.15.42:0/2769744599 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f99fc0a2a20 msgr2=0x7f99fc0a3bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:58.111 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:58.100+0000 7f9a01ebf700 1 --2- 172.21.15.42:0/2769744599 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f99fc0a2a20 0x7f99fc0a3bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:58.111 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:58.100+0000 7f9a01ebf700 1 -- 172.21.15.42:0/2769744599 >> v1:172.21.15.40:6789/0 conn(0x7f99fc0a6a20 legacy=0x7f99fc078f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:58.111 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:58.100+0000 7f99f1ffb700 1 -- 172.21.15.42:0/2769744599 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 686533470 0 0) 0x7f99fc079730 con 0x7f99fc0a6a20 2019-12-05T06:06:58.111 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:58.100+0000 7f9a01ebf700 1 -- 172.21.15.42:0/2769744599 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99fc0a1ef0 con 0x7f99fc0a9ab0 2019-12-05T06:06:58.111 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:58.100+0000 7f9a016be700 1 --2- 172.21.15.42:0/2769744599 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f99fc0a2a20 0x7f99fc0a3bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:06:58.111 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:58.100+0000 7f9a01ebf700 1 --2- 172.21.15.42:0/2769744599 >> v2:172.21.15.40:3300/0 conn(0x7f99fc0a9ab0 0x7f99fc12cf00 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f99fc0a39b0 tx=0x7f99f4012d80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:58.111 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:58.100+0000 7f9a01ebf700 1 --2- 172.21.15.42:0/2769744599 >> v2:172.21.15.40:3300/0 conn(0x7f99fc0a9ab0 0x7f99fc12cf00 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f99fc0a39b0 tx=0x7f99f4012d80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:58.112 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:58.100+0000 7f9a01ebf700 1 --2- 172.21.15.42:0/2769744599 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f99fc0a9ab0 0x7f99fc12cf00 secure :-1 s=READY pgs=1563 cs=0 l=1 rx=0x7f99fc0a39b0 tx=0x7f99f4012d80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:58.112 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:58.100+0000 7f9a01ebf700 1 --2- 172.21.15.42:0/2769744599 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f99fc0a9ab0 0x7f99fc12cf00 secure :-1 s=THROTTLE_DONE pgs=1563 cs=0 l=1 rx=0x7f99fc0a39b0 tx=0x7f99f4012d80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:58.112 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:58.100+0000 7f99f1ffb700 1 -- 172.21.15.42:0/2769744599 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f99f40143c0 con 0x7f99fc0a9ab0 2019-12-05T06:06:58.112 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:58.100+0000 7f9a01ebf700 1 --2- 172.21.15.42:0/2769744599 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f99fc0a9ab0 0x7f99fc12cf00 secure :-1 s=THROTTLE_DONE pgs=1563 cs=0 l=1 rx=0x7f99fc0a39b0 tx=0x7f99f4012d80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:58.112 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:58.100+0000 7f9a01ebf700 1 --2- 172.21.15.42:0/2769744599 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f99fc0a9ab0 0x7f99fc12cf00 secure :-1 s=THROTTLE_DONE pgs=1563 cs=0 l=1 rx=0x7f99fc0a39b0 tx=0x7f99f4012d80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:58.113 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:58.100+0000 7f99f1ffb700 1 -- 172.21.15.42:0/2769744599 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99f4006ad0 con 0x7f99fc0a9ab0 2019-12-05T06:06:58.113 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:58.100+0000 7f9a03121700 1 -- 172.21.15.42:0/2769744599 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99fc142530 con 0x7f99fc0a9ab0 2019-12-05T06:06:58.113 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:58.100+0000 7f99f1ffb700 1 -- 172.21.15.42:0/2769744599 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f99f40149e0 con 0x7f99fc0a9ab0 2019-12-05T06:06:58.113 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:58.100+0000 7f9a03121700 1 -- 172.21.15.42:0/2769744599 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99fc0a2460 con 0x7f99fc0a9ab0 2019-12-05T06:06:58.113 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:58.100+0000 7f9a01ebf700 1 --2- 172.21.15.42:0/2769744599 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f99fc0a9ab0 0x7f99fc12cf00 secure :-1 s=THROTTLE_DONE pgs=1563 cs=0 l=1 rx=0x7f99fc0a39b0 tx=0x7f99f4012d80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:58.113 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:58.100+0000 7f99f1ffb700 1 -- 172.21.15.42:0/2769744599 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f99f4014cf0 con 0x7f99fc0a9ab0 2019-12-05T06:06:58.113 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:58.100+0000 7f9a01ebf700 1 --2- 172.21.15.42:0/2769744599 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f99fc0a9ab0 0x7f99fc12cf00 secure :-1 s=THROTTLE_DONE pgs=1563 cs=0 l=1 rx=0x7f99fc0a39b0 tx=0x7f99f4012d80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:58.114 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:58.100+0000 7f99f1ffb700 1 --2- 172.21.15.42:0/2769744599 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f99dc03ac00 0x7f99dc03d110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:58.114 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:58.100+0000 7f99f1ffb700 1 -- 172.21.15.42:0/2769744599 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9729+0+0 (secure 0 0 0) 0x7f99f404d960 con 0x7f99fc0a9ab0 2019-12-05T06:06:58.114 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:58.100+0000 7f9a016be700 1 --2- 172.21.15.42:0/2769744599 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f99dc03ac00 0x7f99dc03d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:58.114 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:58.104+0000 7f9a016be700 1 --2- 172.21.15.42:0/2769744599 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f99dc03ac00 0x7f99dc03d110 crc :-1 s=READY pgs=870 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:58.114 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:58.104+0000 7f9a03121700 1 -- 172.21.15.42:0/2769744599 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99fc05d110 con 0x7f99fc0a9ab0 2019-12-05T06:06:58.119 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:58.112+0000 7f9a01ebf700 1 --2- 172.21.15.42:0/2769744599 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f99fc0a9ab0 0x7f99fc12cf00 secure :-1 s=THROTTLE_DONE pgs=1563 cs=0 l=1 rx=0x7f99fc0a39b0 tx=0x7f99f4012d80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:58.119 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:58.112+0000 7f99f1ffb700 1 -- 172.21.15.42:0/2769744599 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f99f40349e0 con 0x7f99fc0a9ab0 2019-12-05T06:06:58.288 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:58.280+0000 7f9a03121700 1 -- 172.21.15.42:0/2769744599 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f99fc1505d0 con 0x7f99fc0a9ab0 2019-12-05T06:06:58.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:58.284+0000 7f9a01ebf700 1 --2- 172.21.15.42:0/2769744599 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f99fc0a9ab0 0x7f99fc12cf00 secure :-1 s=THROTTLE_DONE pgs=1563 cs=0 l=1 rx=0x7f99fc0a39b0 tx=0x7f99f4012d80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:58.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:58.284+0000 7f99f1ffb700 1 -- 172.21.15.42:0/2769744599 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 248 v248) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f99f4034c40 con 0x7f99fc0a9ab0 2019-12-05T06:06:58.290 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:06:58.291 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":248,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":9573,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/2968227232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2968227232},{"type":"v1","addr":"172.21.15.40:6833","nonce":2968227232}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":248}],"filesystems":[{"mdsmap":{"epoch":247,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:05:33.997161+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":92,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6821},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6821":{"gid":6821,"name":"a","rank":0,"incarnation":244,"state":"up:active","state_seq":23,"addr":"172.21.15.42:6835/2253267554","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2253267554},{"type":"v1","addr":"172.21.15.42:6835","nonce":2253267554}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:06:58.291 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 248 2019-12-05T06:06:58.322 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:58.288+0000 7f9a03121700 1 -- 172.21.15.42:0/2769744599 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f99dc03ac00 msgr2=0x7f99dc03d110 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:58.322 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:58.288+0000 7f9a03121700 1 --2- 172.21.15.42:0/2769744599 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f99dc03ac00 0x7f99dc03d110 crc :-1 s=READY pgs=870 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:58.323 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:58.288+0000 7f9a03121700 1 -- 172.21.15.42:0/2769744599 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f99fc0a9ab0 msgr2=0x7f99fc12cf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:58.323 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:58.288+0000 7f9a03121700 1 --2- 172.21.15.42:0/2769744599 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f99fc0a9ab0 0x7f99fc12cf00 secure :-1 s=READY pgs=1563 cs=0 l=1 rx=0x7f99fc0a39b0 tx=0x7f99f4012d80).stop 2019-12-05T06:06:58.323 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:58.288+0000 7f9a03121700 1 -- 172.21.15.42:0/2769744599 shutdown_connections 2019-12-05T06:06:58.323 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:58.288+0000 7f9a03121700 1 --2- 172.21.15.42:0/2769744599 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f99dc03ac00 0x7f99dc03d110 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:58.323 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:58.288+0000 7f9a03121700 1 --2- 172.21.15.42:0/2769744599 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f99fc0a9ab0 0x7f99fc12cf00 unknown :-1 s=CLOSED pgs=1563 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:58.323 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:58.288+0000 7f9a03121700 1 --2- 172.21.15.42:0/2769744599 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f99fc0a2a20 0x7f99fc0a3bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:58.324 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:58.288+0000 7f9a03121700 1 -- 172.21.15.42:0/2769744599 shutdown_connections 2019-12-05T06:06:58.324 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:58.288+0000 7f9a03121700 1 -- 172.21.15.42:0/2769744599 wait complete. 2019-12-05T06:06:58.324 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:06:58.288+0000 7f9a03121700 1 -- 172.21.15.42:0/2769744599 >> 172.21.15.42:0/2769744599 conn(0x7f99fc07de70 msgr2=0x7f99fc08eb70 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:06:58.325 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:07:00.326 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:07:00.442 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.432+0000 7f2778695700 1 Processor -- start 2019-12-05T06:07:00.442 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.432+0000 7f2778695700 1 -- start start 2019-12-05T06:07:00.442 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.432+0000 7f2778695700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f27700a7740 0x7f277008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:07:00.442 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.432+0000 7f2778695700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f277009fae0 0x7f277007eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:07:00.442 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.432+0000 7f2778695700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27700a0710 con 0x7f277007e720 2019-12-05T06:07:00.444 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.432+0000 7f2778695700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27700a0b00 con 0x7f27700a7740 2019-12-05T06:07:00.444 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.432+0000 7f2778695700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f277008e5b0 con 0x7f277009fae0 2019-12-05T06:07:00.444 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.432+0000 7f2776c32700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f27700a7740 0x7f277008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:00.445 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.432+0000 7f2776c32700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f27700a7740 0x7f277008df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:48448/0 (socket says 172.21.15.42:48448) 2019-12-05T06:07:00.445 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.432+0000 7f2776c32700 1 -- 172.21.15.42:0/4261865796 learned_addr learned my addr 172.21.15.42:0/4261865796 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:07:00.445 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.432+0000 7f2777433700 1 --2- 172.21.15.42:0/4261865796 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f277009fae0 0x7f277007eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:00.445 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.432+0000 7f2776c32700 1 -- 172.21.15.42:0/4261865796 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f277009fae0 msgr2=0x7f277007eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:07:00.446 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.432+0000 7f2776c32700 1 --2- 172.21.15.42:0/4261865796 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f277009fae0 0x7f277007eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:07:00.446 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.432+0000 7f2776c32700 1 -- 172.21.15.42:0/4261865796 >> v1:172.21.15.40:6789/0 conn(0x7f277007e720 legacy=0x7f277009f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:00.446 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.432+0000 7f2776c32700 1 -- 172.21.15.42:0/4261865796 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f277007f0a0 con 0x7f27700a7740 2019-12-05T06:07:00.446 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.432+0000 7f2776c32700 1 --2- 172.21.15.42:0/4261865796 >> v2:172.21.15.42:3300/0 conn(0x7f27700a7740 0x7f277008df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f276c008790 tx=0x7f276c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:00.447 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.432+0000 7f2776c32700 1 --2- 172.21.15.42:0/4261865796 >> v2:172.21.15.42:3300/0 conn(0x7f27700a7740 0x7f277008df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f276c008790 tx=0x7f276c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:00.447 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.432+0000 7f2776c32700 1 --2- 172.21.15.42:0/4261865796 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f27700a7740 0x7f277008df80 secure :-1 s=READY pgs=2051 cs=0 l=1 rx=0x7f276c008790 tx=0x7f276c00fd30).ready entity=mon.1 client_cookie=3cead4724698c262 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:07:00.447 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.432+0000 7f2776c32700 1 --2- 172.21.15.42:0/4261865796 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f27700a7740 0x7f277008df80 secure :-1 s=THROTTLE_DONE pgs=2051 cs=0 l=1 rx=0x7f276c008790 tx=0x7f276c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:00.447 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.432+0000 7f2775c30700 1 -- 172.21.15.42:0/4261865796 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f276c008c90 con 0x7f27700a7740 2019-12-05T06:07:00.448 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.436+0000 7f2776c32700 1 --2- 172.21.15.42:0/4261865796 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f27700a7740 0x7f277008df80 secure :-1 s=THROTTLE_DONE pgs=2051 cs=0 l=1 rx=0x7f276c008790 tx=0x7f276c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:00.460 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.436+0000 7f2776c32700 1 --2- 172.21.15.42:0/4261865796 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f27700a7740 0x7f277008df80 secure :-1 s=THROTTLE_DONE pgs=2051 cs=0 l=1 rx=0x7f276c008790 tx=0x7f276c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:00.461 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.436+0000 7f2775c30700 1 -- 172.21.15.42:0/4261865796 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f276c0088b0 con 0x7f27700a7740 2019-12-05T06:07:00.461 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.436+0000 7f2775c30700 1 -- 172.21.15.42:0/4261865796 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f276c011b10 con 0x7f27700a7740 2019-12-05T06:07:00.461 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.436+0000 7f2778695700 1 -- 172.21.15.42:0/4261865796 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f27700a7740 msgr2=0x7f277008df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:00.461 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.436+0000 7f2778695700 1 --2- 172.21.15.42:0/4261865796 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f27700a7740 0x7f277008df80 secure :-1 s=READY pgs=2051 cs=0 l=1 rx=0x7f276c008790 tx=0x7f276c00fd30).stop 2019-12-05T06:07:00.462 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.436+0000 7f2778695700 1 -- 172.21.15.42:0/4261865796 shutdown_connections 2019-12-05T06:07:00.462 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.436+0000 7f2778695700 1 --2- 172.21.15.42:0/4261865796 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f277009fae0 0x7f277007eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:07:00.462 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.436+0000 7f2778695700 1 --2- 172.21.15.42:0/4261865796 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f27700a7740 0x7f277008df80 unknown :-1 s=CLOSED pgs=2051 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:00.462 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.436+0000 7f2778695700 1 -- 172.21.15.42:0/4261865796 shutdown_connections 2019-12-05T06:07:00.463 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.436+0000 7f2778695700 1 -- 172.21.15.42:0/4261865796 wait complete. 2019-12-05T06:07:00.463 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.436+0000 7f2778695700 1 -- 172.21.15.42:0/4261865796 >> 172.21.15.42:0/4261865796 conn(0x7f27700aabe0 msgr2=0x7f27700a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:07:00.473 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.436+0000 7f2778695700 1 Processor -- start 2019-12-05T06:07:00.473 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.436+0000 7f2778695700 1 -- start start 2019-12-05T06:07:00.474 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.440+0000 7f2778695700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f27700a4ae0 0x7f2770144eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:07:00.474 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.440+0000 7f2778695700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f277011cc10 0x7f277011d040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:07:00.475 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.440+0000 7f2778695700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27700a5b10 con 0x7f277007c790 2019-12-05T06:07:00.475 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.440+0000 7f2778695700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27700a5d40 con 0x7f27700a4ae0 2019-12-05T06:07:00.475 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.440+0000 7f2778695700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27700a55d0 con 0x7f277011cc10 2019-12-05T06:07:00.475 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.440+0000 7f2776c32700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f27700a4ae0 0x7f2770144eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:00.475 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.440+0000 7f2776431700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f277007c790 0x7f277007bb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:48494/0 (socket says 172.21.15.42:48494) 2019-12-05T06:07:00.475 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.440+0000 7f2776431700 1 -- 172.21.15.42:0/1394435222 learned_addr learned my addr 172.21.15.42:0/1394435222 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:07:00.476 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.440+0000 7f2777433700 1 --2- 172.21.15.42:0/1394435222 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f277011cc10 0x7f277011d040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:00.476 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.440+0000 7f27637fe700 1 -- 172.21.15.42:0/1394435222 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1598474111 0 0) 0x7f27700a5b10 con 0x7f277007c790 2019-12-05T06:07:00.476 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.440+0000 7f2776c32700 1 -- 172.21.15.42:0/1394435222 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f277011cc10 msgr2=0x7f277011d040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:00.476 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.440+0000 7f2776c32700 1 --2- 172.21.15.42:0/1394435222 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f277011cc10 0x7f277011d040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:00.476 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.440+0000 7f2776c32700 1 -- 172.21.15.42:0/1394435222 >> v1:172.21.15.42:6789/0 conn(0x7f277007c790 legacy=0x7f277007bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:00.476 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.440+0000 7f2776c32700 1 -- 172.21.15.42:0/1394435222 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27701367c0 con 0x7f27700a4ae0 2019-12-05T06:07:00.477 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.440+0000 7f2777433700 1 --2- 172.21.15.42:0/1394435222 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f277011cc10 0x7f277011d040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:07:00.477 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.440+0000 7f2776c32700 1 --2- 172.21.15.42:0/1394435222 >> v2:172.21.15.40:3300/0 conn(0x7f27700a4ae0 0x7f2770144eb0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f27700a5790 tx=0x7f276c008ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:00.477 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.440+0000 7f2776c32700 1 --2- 172.21.15.42:0/1394435222 >> v2:172.21.15.40:3300/0 conn(0x7f27700a4ae0 0x7f2770144eb0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f27700a5790 tx=0x7f276c008ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:00.477 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.440+0000 7f2776c32700 1 --2- 172.21.15.42:0/1394435222 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f27700a4ae0 0x7f2770144eb0 secure :-1 s=READY pgs=1565 cs=0 l=1 rx=0x7f27700a5790 tx=0x7f276c008ae0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:07:00.477 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.440+0000 7f2776c32700 1 --2- 172.21.15.42:0/1394435222 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f27700a4ae0 0x7f2770144eb0 secure :-1 s=THROTTLE_DONE pgs=1565 cs=0 l=1 rx=0x7f27700a5790 tx=0x7f276c008ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:00.477 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.440+0000 7f2776c32700 1 --2- 172.21.15.42:0/1394435222 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f27700a4ae0 0x7f2770144eb0 secure :-1 s=THROTTLE_DONE pgs=1565 cs=0 l=1 rx=0x7f27700a5790 tx=0x7f276c008ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:00.479 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.440+0000 7f27637fe700 1 -- 172.21.15.42:0/1394435222 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f276c00f470 con 0x7f27700a4ae0 2019-12-05T06:07:00.479 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.440+0000 7f2776c32700 1 --2- 172.21.15.42:0/1394435222 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f27700a4ae0 0x7f2770144eb0 secure :-1 s=THROTTLE_DONE pgs=1565 cs=0 l=1 rx=0x7f27700a5790 tx=0x7f276c008ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:00.479 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.440+0000 7f27637fe700 1 -- 172.21.15.42:0/1394435222 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f276c011730 con 0x7f27700a4ae0 2019-12-05T06:07:00.479 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.440+0000 7f2778695700 1 -- 172.21.15.42:0/1394435222 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27701357b0 con 0x7f27700a4ae0 2019-12-05T06:07:00.479 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.440+0000 7f27637fe700 1 -- 172.21.15.42:0/1394435222 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f276c005730 con 0x7f27700a4ae0 2019-12-05T06:07:00.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.440+0000 7f2778695700 1 -- 172.21.15.42:0/1394435222 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2770135d60 con 0x7f27700a4ae0 2019-12-05T06:07:00.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.440+0000 7f2776c32700 1 --2- 172.21.15.42:0/1394435222 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f27700a4ae0 0x7f2770144eb0 secure :-1 s=THROTTLE_DONE pgs=1565 cs=0 l=1 rx=0x7f27700a5790 tx=0x7f276c008ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:00.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.444+0000 7f27637fe700 1 -- 172.21.15.42:0/1394435222 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f276c0059c0 con 0x7f27700a4ae0 2019-12-05T06:07:00.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.444+0000 7f2776c32700 1 --2- 172.21.15.42:0/1394435222 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f27700a4ae0 0x7f2770144eb0 secure :-1 s=THROTTLE_DONE pgs=1565 cs=0 l=1 rx=0x7f27700a5790 tx=0x7f276c008ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:00.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.444+0000 7f27637fe700 1 --2- 172.21.15.42:0/1394435222 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f275803ac80 0x7f275803d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:07:00.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.444+0000 7f27637fe700 1 -- 172.21.15.42:0/1394435222 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9815+0+0 (secure 0 0 0) 0x7f276c04b630 con 0x7f27700a4ae0 2019-12-05T06:07:00.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.444+0000 7f2776431700 1 --2- 172.21.15.42:0/1394435222 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f275803ac80 0x7f275803d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:00.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.444+0000 7f2776431700 1 --2- 172.21.15.42:0/1394435222 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f275803ac80 0x7f275803d190 crc :-1 s=READY pgs=871 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:07:00.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.448+0000 7f2778695700 1 -- 172.21.15.42:0/1394435222 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27700a53c0 con 0x7f27700a4ae0 2019-12-05T06:07:00.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.452+0000 7f2776c32700 1 --2- 172.21.15.42:0/1394435222 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f27700a4ae0 0x7f2770144eb0 secure :-1 s=THROTTLE_DONE pgs=1565 cs=0 l=1 rx=0x7f27700a5790 tx=0x7f276c008ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:00.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.452+0000 7f27637fe700 1 -- 172.21.15.42:0/1394435222 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f276c032670 con 0x7f27700a4ae0 2019-12-05T06:07:00.634 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.620+0000 7f2778695700 1 -- 172.21.15.42:0/1394435222 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f277005b8c0 con 0x7f27700a4ae0 2019-12-05T06:07:00.634 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.620+0000 7f2776c32700 1 --2- 172.21.15.42:0/1394435222 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f27700a4ae0 0x7f2770144eb0 secure :-1 s=THROTTLE_DONE pgs=1565 cs=0 l=1 rx=0x7f27700a5790 tx=0x7f276c008ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:00.634 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.620+0000 7f27637fe700 1 -- 172.21.15.42:0/1394435222 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 249 v249) v1 ==== 95+0+1774 (secure 0 0 0) 0x7f276c0328d0 con 0x7f27700a4ae0 2019-12-05T06:07:00.635 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 249 2019-12-05T06:07:00.635 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.624+0000 7f2778695700 1 -- 172.21.15.42:0/1394435222 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f275803ac80 msgr2=0x7f275803d190 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:00.635 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.624+0000 7f2778695700 1 --2- 172.21.15.42:0/1394435222 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f275803ac80 0x7f275803d190 crc :-1 s=READY pgs=871 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:00.636 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.624+0000 7f2778695700 1 -- 172.21.15.42:0/1394435222 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f27700a4ae0 msgr2=0x7f2770144eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:00.636 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.624+0000 7f2778695700 1 --2- 172.21.15.42:0/1394435222 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f27700a4ae0 0x7f2770144eb0 secure :-1 s=READY pgs=1565 cs=0 l=1 rx=0x7f27700a5790 tx=0x7f276c008ae0).stop 2019-12-05T06:07:00.636 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.624+0000 7f2778695700 1 -- 172.21.15.42:0/1394435222 shutdown_connections 2019-12-05T06:07:00.636 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.624+0000 7f2778695700 1 --2- 172.21.15.42:0/1394435222 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f275803ac80 0x7f275803d190 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:00.636 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.624+0000 7f2778695700 1 --2- 172.21.15.42:0/1394435222 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f277011cc10 0x7f277011d040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:00.636 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.624+0000 7f2778695700 1 --2- 172.21.15.42:0/1394435222 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f27700a4ae0 0x7f2770144eb0 unknown :-1 s=CLOSED pgs=1565 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:00.637 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.624+0000 7f2778695700 1 -- 172.21.15.42:0/1394435222 shutdown_connections 2019-12-05T06:07:00.637 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.624+0000 7f2778695700 1 -- 172.21.15.42:0/1394435222 wait complete. 2019-12-05T06:07:00.637 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.624+0000 7f2778695700 1 -- 172.21.15.42:0/1394435222 >> 172.21.15.42:0/1394435222 conn(0x7f27700a80b0 msgr2=0x7f277011d8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:07:00.637 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:07:00.637 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":249,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":249,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:06:59.470781+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":93,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9573},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9573":{"gid":9573,"name":"b","rank":0,"incarnation":249,"state":"up:replay","state_seq":2,"addr":"172.21.15.40:6833/2968227232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2968227232},{"type":"v1","addr":"172.21.15.40:6833","nonce":2968227232}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:07:00.648 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a down, removed from mdsmap 2019-12-05T06:07:00.648 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2019-12-05T06:07:00.648 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:07:00.753 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.740+0000 7f6c6faf5700 1 Processor -- start 2019-12-05T06:07:00.753 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.740+0000 7f6c6faf5700 1 -- start start 2019-12-05T06:07:00.754 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.744+0000 7f6c6faf5700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f6c680a7740 0x7f6c6808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:07:00.754 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.744+0000 7f6c6faf5700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6c6809fae0 0x7f6c6807eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:07:00.754 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.744+0000 7f6c6faf5700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c680a0710 con 0x7f6c6807e720 2019-12-05T06:07:00.754 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.744+0000 7f6c6faf5700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c680a0b00 con 0x7f6c680a7740 2019-12-05T06:07:00.754 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.744+0000 7f6c6faf5700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c6808e5b0 con 0x7f6c6809fae0 2019-12-05T06:07:00.754 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.744+0000 7f6c6d891700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f6c6807e720 0x7f6c6809f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:48500/0 (socket says 172.21.15.42:48500) 2019-12-05T06:07:00.754 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.744+0000 7f6c6d891700 1 -- 172.21.15.42:0/2061488246 learned_addr learned my addr 172.21.15.42:0/2061488246 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:07:00.755 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.744+0000 7f6c6e893700 1 --2- 172.21.15.42:0/2061488246 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6c6809fae0 0x7f6c6807eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:00.755 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.744+0000 7f6c6e092700 1 --2- 172.21.15.42:0/2061488246 >> v2:172.21.15.40:3300/0 conn(0x7f6c680a7740 0x7f6c6808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:00.755 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.744+0000 7f6c6d090700 1 -- 172.21.15.42:0/2061488246 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1882945527 0 0) 0x7f6c680a0710 con 0x7f6c6807e720 2019-12-05T06:07:00.755 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.744+0000 7f6c6d090700 1 -- 172.21.15.42:0/2061488246 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c540030b0 con 0x7f6c6807e720 2019-12-05T06:07:00.755 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.744+0000 7f6c6e092700 1 -- 172.21.15.42:0/2061488246 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6c6809fae0 msgr2=0x7f6c6807eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:07:00.755 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.744+0000 7f6c6e092700 1 --2- 172.21.15.42:0/2061488246 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6c6809fae0 0x7f6c6807eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:07:00.755 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.744+0000 7f6c6e092700 1 -- 172.21.15.42:0/2061488246 >> v1:172.21.15.42:6789/0 conn(0x7f6c6807e720 legacy=0x7f6c6809f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:00.756 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.744+0000 7f6c6e092700 1 -- 172.21.15.42:0/2061488246 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c6807f0a0 con 0x7f6c680a7740 2019-12-05T06:07:00.756 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.744+0000 7f6c6e092700 1 --2- 172.21.15.42:0/2061488246 >> v2:172.21.15.40:3300/0 conn(0x7f6c680a7740 0x7f6c6808df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f6c64008790 tx=0x7f6c6400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:00.756 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.744+0000 7f6c6e092700 1 --2- 172.21.15.42:0/2061488246 >> v2:172.21.15.40:3300/0 conn(0x7f6c680a7740 0x7f6c6808df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f6c64008790 tx=0x7f6c6400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:00.756 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.744+0000 7f6c6e092700 1 --2- 172.21.15.42:0/2061488246 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6c680a7740 0x7f6c6808df80 secure :-1 s=READY pgs=1566 cs=0 l=1 rx=0x7f6c64008790 tx=0x7f6c6400fc70).ready entity=mon.0 client_cookie=ccfccd5e1e2dd9ed server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:07:00.771 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.744+0000 7f6c6e092700 1 --2- 172.21.15.42:0/2061488246 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6c680a7740 0x7f6c6808df80 secure :-1 s=THROTTLE_DONE pgs=1566 cs=0 l=1 rx=0x7f6c64008790 tx=0x7f6c6400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:00.772 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.744+0000 7f6c6d090700 1 -- 172.21.15.42:0/2061488246 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6c64008bb0 con 0x7f6c680a7740 2019-12-05T06:07:00.772 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.744+0000 7f6c6e092700 1 --2- 172.21.15.42:0/2061488246 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6c680a7740 0x7f6c6808df80 secure :-1 s=THROTTLE_DONE pgs=1566 cs=0 l=1 rx=0x7f6c64008790 tx=0x7f6c6400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:00.772 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.744+0000 7f6c6d090700 1 -- 172.21.15.42:0/2061488246 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c640087d0 con 0x7f6c680a7740 2019-12-05T06:07:00.774 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.744+0000 7f6c6e092700 1 --2- 172.21.15.42:0/2061488246 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6c680a7740 0x7f6c6808df80 secure :-1 s=THROTTLE_DONE pgs=1566 cs=0 l=1 rx=0x7f6c64008790 tx=0x7f6c6400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:00.774 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.744+0000 7f6c6d090700 1 -- 172.21.15.42:0/2061488246 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6c64011860 con 0x7f6c680a7740 2019-12-05T06:07:00.774 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.744+0000 7f6c6faf5700 1 -- 172.21.15.42:0/2061488246 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6c680a7740 msgr2=0x7f6c6808df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:00.775 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.744+0000 7f6c6faf5700 1 --2- 172.21.15.42:0/2061488246 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6c680a7740 0x7f6c6808df80 secure :-1 s=READY pgs=1566 cs=0 l=1 rx=0x7f6c64008790 tx=0x7f6c6400fc70).stop 2019-12-05T06:07:00.775 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.744+0000 7f6c6faf5700 1 -- 172.21.15.42:0/2061488246 shutdown_connections 2019-12-05T06:07:00.775 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.744+0000 7f6c6faf5700 1 --2- 172.21.15.42:0/2061488246 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6c6809fae0 0x7f6c6807eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:07:00.775 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.744+0000 7f6c6faf5700 1 --2- 172.21.15.42:0/2061488246 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6c680a7740 0x7f6c6808df80 unknown :-1 s=CLOSED pgs=1566 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:00.776 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.744+0000 7f6c6faf5700 1 -- 172.21.15.42:0/2061488246 shutdown_connections 2019-12-05T06:07:00.776 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.744+0000 7f6c6faf5700 1 -- 172.21.15.42:0/2061488246 wait complete. 2019-12-05T06:07:00.777 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.744+0000 7f6c6faf5700 1 -- 172.21.15.42:0/2061488246 >> 172.21.15.42:0/2061488246 conn(0x7f6c680aabe0 msgr2=0x7f6c680a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:07:00.783 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.748+0000 7f6c6faf5700 1 Processor -- start 2019-12-05T06:07:00.783 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.748+0000 7f6c6faf5700 1 -- start start 2019-12-05T06:07:00.784 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.752+0000 7f6c6faf5700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f6c68144810 0x7f6c6808f450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:07:00.784 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.752+0000 7f6c6faf5700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6c6811b820 0x7f6c6811bc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:07:00.785 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.752+0000 7f6c6faf5700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c68131200 con 0x7f6c680a3760 2019-12-05T06:07:00.785 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.752+0000 7f6c6faf5700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c6811c1a0 con 0x7f6c68144810 2019-12-05T06:07:00.785 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.752+0000 7f6c6faf5700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c68119450 con 0x7f6c6811b820 2019-12-05T06:07:00.785 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.752+0000 7f6c6d891700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6c6811b820 0x7f6c6811bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:00.787 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.752+0000 7f6c6e092700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f6c68144810 0x7f6c6808f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:00.787 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.752+0000 7f6c6e092700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f6c68144810 0x7f6c6808f450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.42:36284/0 (socket says 172.21.15.42:36284) 2019-12-05T06:07:00.787 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.752+0000 7f6c6e092700 1 -- 172.21.15.42:0/4264954247 learned_addr learned my addr 172.21.15.42:0/4264954247 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:07:00.787 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.752+0000 7f6c6d891700 1 --2- 172.21.15.42:0/4264954247 >> v2:172.21.15.42:3300/0 conn(0x7f6c6811b820 0x7f6c6811bc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:48470/0 (socket says 172.21.15.42:48470) 2019-12-05T06:07:00.788 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.752+0000 7f6c6d891700 1 -- 172.21.15.42:0/4264954247 >> v2:172.21.15.40:3300/0 conn(0x7f6c68144810 msgr2=0x7f6c6808f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:00.788 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.752+0000 7f6c6d891700 1 --2- 172.21.15.42:0/4264954247 >> v2:172.21.15.40:3300/0 conn(0x7f6c68144810 0x7f6c6808f450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:00.788 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.752+0000 7f6c6d891700 1 -- 172.21.15.42:0/4264954247 >> v1:172.21.15.40:6789/0 conn(0x7f6c680a3760 legacy=0x7f6c680a3b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:00.788 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.752+0000 7f6c6d891700 1 -- 172.21.15.42:0/4264954247 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c6811aa60 con 0x7f6c6811b820 2019-12-05T06:07:00.789 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.752+0000 7f6c6e092700 1 --2- 172.21.15.42:0/4264954247 >> v2:172.21.15.40:3300/0 conn(0x7f6c68144810 0x7f6c6808f450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:07:00.789 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.752+0000 7f6c6d891700 1 --2- 172.21.15.42:0/4264954247 >> v2:172.21.15.42:3300/0 conn(0x7f6c6811b820 0x7f6c6811bc50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f6c5c0043a0 tx=0x7f6c5c0108b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:00.789 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.752+0000 7f6c6d891700 1 --2- 172.21.15.42:0/4264954247 >> v2:172.21.15.42:3300/0 conn(0x7f6c6811b820 0x7f6c6811bc50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f6c5c0043a0 tx=0x7f6c5c0108b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:00.790 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.752+0000 7f6c6d891700 1 --2- 172.21.15.42:0/4264954247 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6c6811b820 0x7f6c6811bc50 secure :-1 s=READY pgs=2054 cs=0 l=1 rx=0x7f6c5c0043a0 tx=0x7f6c5c0108b0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:07:00.790 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.752+0000 7f6c6d891700 1 --2- 172.21.15.42:0/4264954247 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6c6811b820 0x7f6c6811bc50 secure :-1 s=THROTTLE_DONE pgs=2054 cs=0 l=1 rx=0x7f6c5c0043a0 tx=0x7f6c5c0108b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:00.790 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.752+0000 7f6c6d891700 1 --2- 172.21.15.42:0/4264954247 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6c6811b820 0x7f6c6811bc50 secure :-1 s=THROTTLE_DONE pgs=2054 cs=0 l=1 rx=0x7f6c5c0043a0 tx=0x7f6c5c0108b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:00.790 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.752+0000 7f6c6d891700 1 --2- 172.21.15.42:0/4264954247 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6c6811b820 0x7f6c6811bc50 secure :-1 s=THROTTLE_DONE pgs=2054 cs=0 l=1 rx=0x7f6c5c0043a0 tx=0x7f6c5c0108b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:00.791 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.752+0000 7f6c5affd700 1 -- 172.21.15.42:0/4264954247 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6c5c005b40 con 0x7f6c6811b820 2019-12-05T06:07:00.791 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.752+0000 7f6c5affd700 1 -- 172.21.15.42:0/4264954247 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c5c005760 con 0x7f6c6811b820 2019-12-05T06:07:00.791 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.752+0000 7f6c6faf5700 1 -- 172.21.15.42:0/4264954247 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c6808f9a0 con 0x7f6c6811b820 2019-12-05T06:07:00.791 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.752+0000 7f6c5affd700 1 -- 172.21.15.42:0/4264954247 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6c5c0062b0 con 0x7f6c6811b820 2019-12-05T06:07:00.792 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.752+0000 7f6c6faf5700 1 -- 172.21.15.42:0/4264954247 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c68144c40 con 0x7f6c6811b820 2019-12-05T06:07:00.792 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.752+0000 7f6c6d891700 1 --2- 172.21.15.42:0/4264954247 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6c6811b820 0x7f6c6811bc50 secure :-1 s=THROTTLE_DONE pgs=2054 cs=0 l=1 rx=0x7f6c5c0043a0 tx=0x7f6c5c0108b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:00.792 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.752+0000 7f6c5affd700 1 -- 172.21.15.42:0/4264954247 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f6c5c0065c0 con 0x7f6c6811b820 2019-12-05T06:07:00.792 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.752+0000 7f6c6d891700 1 --2- 172.21.15.42:0/4264954247 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6c6811b820 0x7f6c6811bc50 secure :-1 s=THROTTLE_DONE pgs=2054 cs=0 l=1 rx=0x7f6c5c0043a0 tx=0x7f6c5c0108b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:00.793 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.756+0000 7f6c5affd700 1 --2- 172.21.15.42:0/4264954247 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6c5003ae40 0x7f6c5003d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:07:00.793 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.756+0000 7f6c5affd700 1 -- 172.21.15.42:0/4264954247 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9815+0+0 (secure 0 0 0) 0x7f6c5c04b030 con 0x7f6c6811b820 2019-12-05T06:07:00.793 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.756+0000 7f6c6e092700 1 --2- 172.21.15.42:0/4264954247 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6c5003ae40 0x7f6c5003d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:00.794 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.756+0000 7f6c6e092700 1 --2- 172.21.15.42:0/4264954247 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6c5003ae40 0x7f6c5003d350 crc :-1 s=READY pgs=872 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:07:00.794 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.756+0000 7f6c6faf5700 1 -- 172.21.15.42:0/4264954247 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c68142020 con 0x7f6c6811b820 2019-12-05T06:07:00.795 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.764+0000 7f6c6d891700 1 --2- 172.21.15.42:0/4264954247 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6c6811b820 0x7f6c6811bc50 secure :-1 s=THROTTLE_DONE pgs=2054 cs=0 l=1 rx=0x7f6c5c0043a0 tx=0x7f6c5c0108b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:00.795 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.764+0000 7f6c5affd700 1 -- 172.21.15.42:0/4264954247 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f6c5c032050 con 0x7f6c6811b820 2019-12-05T06:07:00.941 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.936+0000 7f6c6faf5700 1 -- 172.21.15.42:0/4264954247 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6c6805d110 con 0x7f6c6811b820 2019-12-05T06:07:00.942 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.936+0000 7f6c6d891700 1 --2- 172.21.15.42:0/4264954247 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6c6811b820 0x7f6c6811bc50 secure :-1 s=THROTTLE_DONE pgs=2054 cs=0 l=1 rx=0x7f6c5c0043a0 tx=0x7f6c5c0108b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:00.942 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.936+0000 7f6c5affd700 1 -- 172.21.15.42:0/4264954247 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 249 v249) v1 ==== 95+0+1774 (secure 0 0 0) 0x7f6c5c0322b0 con 0x7f6c6811b820 2019-12-05T06:07:00.942 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 249 2019-12-05T06:07:00.942 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:07:00.948 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":249,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":249,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:06:59.470781+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":93,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9573},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9573":{"gid":9573,"name":"b","rank":0,"incarnation":249,"state":"up:replay","state_seq":2,"addr":"172.21.15.40:6833/2968227232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2968227232},{"type":"v1","addr":"172.21.15.40:6833","nonce":2968227232}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:07:00.950 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.940+0000 7f6c6faf5700 1 -- 172.21.15.42:0/4264954247 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6c5003ae40 msgr2=0x7f6c5003d350 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:00.950 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.940+0000 7f6c6faf5700 1 --2- 172.21.15.42:0/4264954247 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6c5003ae40 0x7f6c5003d350 crc :-1 s=READY pgs=872 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:00.950 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.940+0000 7f6c6faf5700 1 -- 172.21.15.42:0/4264954247 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6c6811b820 msgr2=0x7f6c6811bc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:00.950 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.940+0000 7f6c6faf5700 1 --2- 172.21.15.42:0/4264954247 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6c6811b820 0x7f6c6811bc50 secure :-1 s=READY pgs=2054 cs=0 l=1 rx=0x7f6c5c0043a0 tx=0x7f6c5c0108b0).stop 2019-12-05T06:07:00.951 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.940+0000 7f6c6faf5700 1 -- 172.21.15.42:0/4264954247 shutdown_connections 2019-12-05T06:07:00.951 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.940+0000 7f6c6faf5700 1 --2- 172.21.15.42:0/4264954247 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6c5003ae40 0x7f6c5003d350 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:00.951 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.940+0000 7f6c6faf5700 1 --2- 172.21.15.42:0/4264954247 >> v2:172.21.15.40:3300/0 conn(0x7f6c68144810 0x7f6c6808f450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:00.951 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.940+0000 7f6c6faf5700 1 --2- 172.21.15.42:0/4264954247 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6c6811b820 0x7f6c6811bc50 unknown :-1 s=CLOSED pgs=2054 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:00.951 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.940+0000 7f6c6faf5700 1 -- 172.21.15.42:0/4264954247 shutdown_connections 2019-12-05T06:07:00.953 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.940+0000 7f6c6faf5700 1 -- 172.21.15.42:0/4264954247 wait complete. 2019-12-05T06:07:00.953 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:00.940+0000 7f6c6faf5700 1 -- 172.21.15.42:0/4264954247 >> 172.21.15.42:0/4264954247 conn(0x7f6c680a7740 msgr2=0x7f6c6809f570 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:07:00.955 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 249, "feature_flags": { "enable_multiple": false, "ever_enabled_multiple": false }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2019-12-05T05:03:35.604738+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 249, "ever_allowed_features": 2, "explicitly_allowed_features": 2, "failed": [], "flags": 18, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_9573": { "addr": "172.21.15.40:6833/2968227232", "addrs": { "addrvec": [ { "addr": "172.21.15.40:6832", "nonce": 2968227232, "type": "v2" }, { "addr": "172.21.15.40:6833", "nonce": 2968227232, "type": "v1" } ] }, "export_targets": [], "features": 4540138292836696063, "flags": 0, "gid": 9573, "incarnation": 249, "name": "b", "rank": 0, "state": "up:replay", "state_seq": 2 } }, "last_failure": 0, "last_failure_osd_epoch": 93, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 2, "min_compat_client": "0 (unknown)", "modified": "2019-12-05T06:06:59.470781+0000", "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 9573 } } } ], "standbys": [] } 2019-12-05T06:07:02.957 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:07:03.066 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:03.052+0000 7ff844c56700 1 Processor -- start 2019-12-05T06:07:03.066 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:03.052+0000 7ff844c56700 1 -- start start 2019-12-05T06:07:03.067 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:03.052+0000 7ff844c56700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff84008df80 0x7ff84009f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:07:03.067 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:03.052+0000 7ff844c56700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff84009fc60 con 0x7ff8400a7740 2019-12-05T06:07:03.067 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:03.052+0000 7ff844c56700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8400a0710 con 0x7ff84009f830 2019-12-05T06:07:03.067 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:03.052+0000 7ff844c56700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8400a0b00 con 0x7ff84008df80 2019-12-05T06:07:03.067 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:03.052+0000 7ff83f59e700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7ff84009f830 0x7ff84014efe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:48520/0 (socket says 172.21.15.42:48520) 2019-12-05T06:07:03.067 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:03.052+0000 7ff83f59e700 1 -- 172.21.15.42:0/3122360624 learned_addr learned my addr 172.21.15.42:0/3122360624 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:07:03.068 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:03.052+0000 7ff83e59c700 1 --2- 172.21.15.42:0/3122360624 >> v2:172.21.15.40:3300/0 conn(0x7ff84008df80 0x7ff84009f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:03.069 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:03.052+0000 7ff83dd9b700 1 -- 172.21.15.42:0/3122360624 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 237545805 0 0) 0x7ff8400a0710 con 0x7ff84009f830 2019-12-05T06:07:03.069 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:03.052+0000 7ff83dd9b700 1 -- 172.21.15.42:0/3122360624 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8200030b0 con 0x7ff84009f830 2019-12-05T06:07:03.069 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:03.052+0000 7ff83e59c700 1 -- 172.21.15.42:0/3122360624 >> v1:172.21.15.42:6789/0 conn(0x7ff84009f830 legacy=0x7ff84014efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:03.069 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:03.052+0000 7ff83e59c700 1 -- 172.21.15.42:0/3122360624 >> v1:172.21.15.40:6789/0 conn(0x7ff8400a7740 legacy=0x7ff84007e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:03.069 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:03.052+0000 7ff83dd9b700 1 -- 172.21.15.42:0/3122360624 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1299864851 0 0) 0x7ff84009fc60 con 0x7ff8400a7740 2019-12-05T06:07:03.069 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:03.052+0000 7ff83e59c700 1 -- 172.21.15.42:0/3122360624 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff84007ef20 con 0x7ff84008df80 2019-12-05T06:07:03.069 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:03.052+0000 7ff83e59c700 1 --2- 172.21.15.42:0/3122360624 >> v2:172.21.15.40:3300/0 conn(0x7ff84008df80 0x7ff84009f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7ff828008790 tx=0x7ff82800fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:03.070 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:03.052+0000 7ff83e59c700 1 --2- 172.21.15.42:0/3122360624 >> v2:172.21.15.40:3300/0 conn(0x7ff84008df80 0x7ff84009f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7ff828008790 tx=0x7ff82800fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:03.070 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:03.052+0000 7ff83e59c700 1 --2- 172.21.15.42:0/3122360624 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff84008df80 0x7ff84009f2e0 secure :-1 s=READY pgs=1569 cs=0 l=1 rx=0x7ff828008790 tx=0x7ff82800fbf0).ready entity=mon.0 client_cookie=474428487a520e28 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:07:03.070 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:03.052+0000 7ff83e59c700 1 --2- 172.21.15.42:0/3122360624 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff84008df80 0x7ff84009f2e0 secure :-1 s=THROTTLE_DONE pgs=1569 cs=0 l=1 rx=0x7ff828008790 tx=0x7ff82800fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:03.070 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:03.052+0000 7ff83dd9b700 1 -- 172.21.15.42:0/3122360624 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff828011270 con 0x7ff84008df80 2019-12-05T06:07:03.070 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:03.052+0000 7ff83e59c700 1 --2- 172.21.15.42:0/3122360624 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff84008df80 0x7ff84009f2e0 secure :-1 s=THROTTLE_DONE pgs=1569 cs=0 l=1 rx=0x7ff828008790 tx=0x7ff82800fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:03.070 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:03.052+0000 7ff83e59c700 1 --2- 172.21.15.42:0/3122360624 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff84008df80 0x7ff84009f2e0 secure :-1 s=THROTTLE_DONE pgs=1569 cs=0 l=1 rx=0x7ff828008790 tx=0x7ff82800fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:03.070 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:03.052+0000 7ff83dd9b700 1 -- 172.21.15.42:0/3122360624 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff828008bc0 con 0x7ff84008df80 2019-12-05T06:07:03.071 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:03.052+0000 7ff83dd9b700 1 -- 172.21.15.42:0/3122360624 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff828011850 con 0x7ff84008df80 2019-12-05T06:07:03.071 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:03.056+0000 7ff844c56700 1 -- 172.21.15.42:0/3122360624 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff84008df80 msgr2=0x7ff84009f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:03.071 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:03.056+0000 7ff844c56700 1 --2- 172.21.15.42:0/3122360624 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff84008df80 0x7ff84009f2e0 secure :-1 s=READY pgs=1569 cs=0 l=1 rx=0x7ff828008790 tx=0x7ff82800fbf0).stop 2019-12-05T06:07:03.071 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:03.056+0000 7ff844c56700 1 -- 172.21.15.42:0/3122360624 shutdown_connections 2019-12-05T06:07:03.071 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:03.056+0000 7ff844c56700 1 --2- 172.21.15.42:0/3122360624 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff84008df80 0x7ff84009f2e0 unknown :-1 s=CLOSED pgs=1569 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:03.071 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:03.056+0000 7ff844c56700 1 -- 172.21.15.42:0/3122360624 shutdown_connections 2019-12-05T06:07:03.071 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:03.056+0000 7ff844c56700 1 -- 172.21.15.42:0/3122360624 wait complete. 2019-12-05T06:07:03.072 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:03.056+0000 7ff844c56700 1 -- 172.21.15.42:0/3122360624 >> 172.21.15.42:0/3122360624 conn(0x7ff8400aabe0 msgr2=0x7ff8400a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:07:03.072 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:03.056+0000 7ff844c56700 1 Processor -- start 2019-12-05T06:07:03.072 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:03.056+0000 7ff844c56700 1 -- start start 2019-12-05T06:07:03.072 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:03.056+0000 7ff844c56700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff840147f10 0x7ff8401242e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:07:03.072 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:03.056+0000 7ff844c56700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff84007dda0 0x7ff84012ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:07:03.072 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:03.056+0000 7ff844c56700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8401476c0 con 0x7ff840146190 2019-12-05T06:07:03.072 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:03.056+0000 7ff844c56700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff840146b30 con 0x7ff84007dda0 2019-12-05T06:07:03.073 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:03.056+0000 7ff844c56700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8401479d0 con 0x7ff840147f10 2019-12-05T06:07:03.073 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:03.056+0000 7ff83ed9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff840147f10 0x7ff8401242e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:03.073 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:03.056+0000 7ff83ed9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff840147f10 0x7ff8401242e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:48484/0 (socket says 172.21.15.42:48484) 2019-12-05T06:07:03.073 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:03.056+0000 7ff83ed9d700 1 -- 172.21.15.42:0/2709614165 learned_addr learned my addr 172.21.15.42:0/2709614165 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:07:03.073 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:03.056+0000 7ff83f59e700 1 --2- 172.21.15.42:0/2709614165 >> v2:172.21.15.40:3300/0 conn(0x7ff84007dda0 0x7ff84012ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:03.073 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:03.060+0000 7ff83ed9d700 1 -- 172.21.15.42:0/2709614165 >> v2:172.21.15.40:3300/0 conn(0x7ff84007dda0 msgr2=0x7ff84012ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:03.073 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:03.060+0000 7ff83ed9d700 1 --2- 172.21.15.42:0/2709614165 >> v2:172.21.15.40:3300/0 conn(0x7ff84007dda0 0x7ff84012ecf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:03.074 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:03.060+0000 7ff83ed9d700 1 -- 172.21.15.42:0/2709614165 >> v1:172.21.15.40:6789/0 conn(0x7ff840146190 legacy=0x7ff8400a3ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:03.074 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:03.060+0000 7ff83ed9d700 1 -- 172.21.15.42:0/2709614165 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff840145860 con 0x7ff840147f10 2019-12-05T06:07:03.074 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:03.060+0000 7ff83ed9d700 1 --2- 172.21.15.42:0/2709614165 >> v2:172.21.15.42:3300/0 conn(0x7ff840147f10 0x7ff8401242e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7ff830000f00 tx=0x7ff83000c6c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:03.074 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:03.060+0000 7ff83ed9d700 1 --2- 172.21.15.42:0/2709614165 >> v2:172.21.15.42:3300/0 conn(0x7ff840147f10 0x7ff8401242e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7ff830000f00 tx=0x7ff83000c6c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:03.074 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:03.060+0000 7ff83ed9d700 1 --2- 172.21.15.42:0/2709614165 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff840147f10 0x7ff8401242e0 secure :-1 s=READY pgs=2056 cs=0 l=1 rx=0x7ff830000f00 tx=0x7ff83000c6c0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:07:03.074 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:03.060+0000 7ff83ed9d700 1 --2- 172.21.15.42:0/2709614165 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff840147f10 0x7ff8401242e0 secure :-1 s=THROTTLE_DONE pgs=2056 cs=0 l=1 rx=0x7ff830000f00 tx=0x7ff83000c6c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:03.074 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:03.060+0000 7ff83ed9d700 1 --2- 172.21.15.42:0/2709614165 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff840147f10 0x7ff8401242e0 secure :-1 s=THROTTLE_DONE pgs=2056 cs=0 l=1 rx=0x7ff830000f00 tx=0x7ff83000c6c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:03.075 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:03.060+0000 7ff8377fe700 1 -- 172.21.15.42:0/2709614165 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff83000ca40 con 0x7ff840147f10 2019-12-05T06:07:03.075 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:03.060+0000 7ff83ed9d700 1 --2- 172.21.15.42:0/2709614165 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff840147f10 0x7ff8401242e0 secure :-1 s=THROTTLE_DONE pgs=2056 cs=0 l=1 rx=0x7ff830000f00 tx=0x7ff83000c6c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:03.075 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:03.060+0000 7ff8377fe700 1 -- 172.21.15.42:0/2709614165 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8300101c0 con 0x7ff840147f10 2019-12-05T06:07:03.075 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:03.060+0000 7ff8377fe700 1 -- 172.21.15.42:0/2709614165 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff830012510 con 0x7ff840147f10 2019-12-05T06:07:03.075 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:03.060+0000 7ff844c56700 1 -- 172.21.15.42:0/2709614165 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8400a42e0 con 0x7ff840147f10 2019-12-05T06:07:03.075 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:03.060+0000 7ff844c56700 1 -- 172.21.15.42:0/2709614165 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff840148370 con 0x7ff840147f10 2019-12-05T06:07:03.075 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:03.060+0000 7ff83ed9d700 1 --2- 172.21.15.42:0/2709614165 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff840147f10 0x7ff8401242e0 secure :-1 s=THROTTLE_DONE pgs=2056 cs=0 l=1 rx=0x7ff830000f00 tx=0x7ff83000c6c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:03.075 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:03.060+0000 7ff83ed9d700 1 --2- 172.21.15.42:0/2709614165 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff840147f10 0x7ff8401242e0 secure :-1 s=THROTTLE_DONE pgs=2056 cs=0 l=1 rx=0x7ff830000f00 tx=0x7ff83000c6c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:03.076 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:03.060+0000 7ff8377fe700 1 -- 172.21.15.42:0/2709614165 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7ff830012820 con 0x7ff840147f10 2019-12-05T06:07:03.076 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:03.060+0000 7ff8377fe700 1 --2- 172.21.15.42:0/2709614165 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff81803ad80 0x7ff81803d290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:07:03.077 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:03.060+0000 7ff8377fe700 1 -- 172.21.15.42:0/2709614165 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9815+0+0 (secure 0 0 0) 0x7ff83004b470 con 0x7ff840147f10 2019-12-05T06:07:03.077 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:03.060+0000 7ff83e59c700 1 --2- 172.21.15.42:0/2709614165 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff81803ad80 0x7ff81803d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:03.077 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:03.064+0000 7ff83e59c700 1 --2- 172.21.15.42:0/2709614165 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff81803ad80 0x7ff81803d290 crc :-1 s=READY pgs=873 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:07:03.077 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:03.068+0000 7ff844c56700 1 -- 172.21.15.42:0/2709614165 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff84005d110 con 0x7ff840147f10 2019-12-05T06:07:03.081 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:03.076+0000 7ff83ed9d700 1 --2- 172.21.15.42:0/2709614165 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff840147f10 0x7ff8401242e0 secure :-1 s=THROTTLE_DONE pgs=2056 cs=0 l=1 rx=0x7ff830000f00 tx=0x7ff83000c6c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:03.081 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:03.076+0000 7ff8377fe700 1 -- 172.21.15.42:0/2709614165 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7ff8300324b0 con 0x7ff840147f10 2019-12-05T06:07:03.249 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:03.244+0000 7ff844c56700 1 -- 172.21.15.42:0/2709614165 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff84005b8c0 con 0x7ff840147f10 2019-12-05T06:07:03.250 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:03.244+0000 7ff83ed9d700 1 --2- 172.21.15.42:0/2709614165 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff840147f10 0x7ff8401242e0 secure :-1 s=THROTTLE_DONE pgs=2056 cs=0 l=1 rx=0x7ff830000f00 tx=0x7ff83000c6c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:03.250 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:03.244+0000 7ff8377fe700 1 -- 172.21.15.42:0/2709614165 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 250 v250) v1 ==== 95+0+1778 (secure 0 0 0) 0x7ff830032710 con 0x7ff840147f10 2019-12-05T06:07:03.250 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 250 2019-12-05T06:07:03.251 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:07:03.251 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":250,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":250,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:07:02.508482+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":93,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9573},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9573":{"gid":9573,"name":"b","rank":0,"incarnation":249,"state":"up:reconnect","state_seq":23,"addr":"172.21.15.40:6833/2968227232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2968227232},{"type":"v1","addr":"172.21.15.40:6833","nonce":2968227232}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:07:03.253 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:03.248+0000 7ff844c56700 1 -- 172.21.15.42:0/2709614165 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff81803ad80 msgr2=0x7ff81803d290 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:03.253 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:03.248+0000 7ff844c56700 1 --2- 172.21.15.42:0/2709614165 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff81803ad80 0x7ff81803d290 crc :-1 s=READY pgs=873 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:03.253 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:03.248+0000 7ff844c56700 1 -- 172.21.15.42:0/2709614165 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff840147f10 msgr2=0x7ff8401242e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:03.253 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:03.248+0000 7ff844c56700 1 --2- 172.21.15.42:0/2709614165 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff840147f10 0x7ff8401242e0 secure :-1 s=READY pgs=2056 cs=0 l=1 rx=0x7ff830000f00 tx=0x7ff83000c6c0).stop 2019-12-05T06:07:03.253 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:03.248+0000 7ff844c56700 1 -- 172.21.15.42:0/2709614165 shutdown_connections 2019-12-05T06:07:03.253 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:03.248+0000 7ff844c56700 1 --2- 172.21.15.42:0/2709614165 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff81803ad80 0x7ff81803d290 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:03.254 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:03.248+0000 7ff844c56700 1 --2- 172.21.15.42:0/2709614165 >> v2:172.21.15.40:3300/0 conn(0x7ff84007dda0 0x7ff84012ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:03.254 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:03.248+0000 7ff844c56700 1 --2- 172.21.15.42:0/2709614165 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff840147f10 0x7ff8401242e0 unknown :-1 s=CLOSED pgs=2056 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:03.254 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:03.248+0000 7ff844c56700 1 -- 172.21.15.42:0/2709614165 shutdown_connections 2019-12-05T06:07:03.254 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:03.248+0000 7ff844c56700 1 -- 172.21.15.42:0/2709614165 wait complete. 2019-12-05T06:07:03.254 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:03.248+0000 7ff844c56700 1 -- 172.21.15.42:0/2709614165 >> 172.21.15.42:0/2709614165 conn(0x7ff8400a7740 msgr2=0x7ff84009f380 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:07:03.263 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2019-12-05T06:07:05.276 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:07:05.414 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:05.372+0000 7ff697642700 1 Processor -- start 2019-12-05T06:07:05.415 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:05.372+0000 7ff697642700 1 -- start start 2019-12-05T06:07:05.415 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:05.372+0000 7ff697642700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff6900a7740 0x7ff69008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:07:05.415 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:05.372+0000 7ff697642700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff69007e720 0x7ff69007eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:07:05.415 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:05.372+0000 7ff697642700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6900a0710 con 0x7ff69009f2e0 2019-12-05T06:07:05.415 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:05.372+0000 7ff697642700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6900a0b00 con 0x7ff69007e720 2019-12-05T06:07:05.415 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:05.372+0000 7ff697642700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff69008e5b0 con 0x7ff6900a7740 2019-12-05T06:07:05.416 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:05.372+0000 7ff695bdf700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff6900a7740 0x7ff69008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:05.416 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:05.372+0000 7ff695bdf700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff6900a7740 0x7ff69008df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:48496/0 (socket says 172.21.15.42:48496) 2019-12-05T06:07:05.416 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:05.372+0000 7ff695bdf700 1 -- 172.21.15.42:0/1974645219 learned_addr learned my addr 172.21.15.42:0/1974645219 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:07:05.416 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:05.372+0000 7ff6953de700 1 --2- 172.21.15.42:0/1974645219 >> v2:172.21.15.40:3300/0 conn(0x7ff69007e720 0x7ff69007eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:05.416 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:05.372+0000 7ff695bdf700 1 -- 172.21.15.42:0/1974645219 >> v2:172.21.15.40:3300/0 conn(0x7ff69007e720 msgr2=0x7ff69007eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:07:05.416 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:05.372+0000 7ff695bdf700 1 --2- 172.21.15.42:0/1974645219 >> v2:172.21.15.40:3300/0 conn(0x7ff69007e720 0x7ff69007eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:07:05.417 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:05.372+0000 7ff695bdf700 1 -- 172.21.15.42:0/1974645219 >> v1:172.21.15.40:6789/0 conn(0x7ff69009f2e0 legacy=0x7ff69009f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:05.417 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:05.372+0000 7ff695bdf700 1 -- 172.21.15.42:0/1974645219 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff69007f0a0 con 0x7ff6900a7740 2019-12-05T06:07:05.417 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:05.372+0000 7ff695bdf700 1 --2- 172.21.15.42:0/1974645219 >> v2:172.21.15.42:3300/0 conn(0x7ff6900a7740 0x7ff69008df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7ff68c002f30 tx=0x7ff68c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:05.418 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:05.372+0000 7ff695bdf700 1 --2- 172.21.15.42:0/1974645219 >> v2:172.21.15.42:3300/0 conn(0x7ff6900a7740 0x7ff69008df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7ff68c002f30 tx=0x7ff68c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:05.418 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:05.372+0000 7ff695bdf700 1 --2- 172.21.15.42:0/1974645219 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff6900a7740 0x7ff69008df80 secure :-1 s=READY pgs=2057 cs=0 l=1 rx=0x7ff68c002f30 tx=0x7ff68c00fd30).ready entity=mon.1 client_cookie=e20528b37955585d server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:07:05.418 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:05.372+0000 7ff695bdf700 1 --2- 172.21.15.42:0/1974645219 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff6900a7740 0x7ff69008df80 secure :-1 s=THROTTLE_DONE pgs=2057 cs=0 l=1 rx=0x7ff68c002f30 tx=0x7ff68c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:05.418 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:05.372+0000 7ff695bdf700 1 --2- 172.21.15.42:0/1974645219 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff6900a7740 0x7ff69008df80 secure :-1 s=THROTTLE_DONE pgs=2057 cs=0 l=1 rx=0x7ff68c002f30 tx=0x7ff68c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:05.418 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:05.372+0000 7ff694bdd700 1 -- 172.21.15.42:0/1974645219 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff68c011370 con 0x7ff6900a7740 2019-12-05T06:07:05.418 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:05.372+0000 7ff695bdf700 1 --2- 172.21.15.42:0/1974645219 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff6900a7740 0x7ff69008df80 secure :-1 s=THROTTLE_DONE pgs=2057 cs=0 l=1 rx=0x7ff68c002f30 tx=0x7ff68c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:05.418 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:05.372+0000 7ff694bdd700 1 -- 172.21.15.42:0/1974645219 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff68c008bb0 con 0x7ff6900a7740 2019-12-05T06:07:05.419 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:05.376+0000 7ff697642700 1 -- 172.21.15.42:0/1974645219 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff6900a7740 msgr2=0x7ff69008df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:05.419 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:05.372+0000 7ff694bdd700 1 -- 172.21.15.42:0/1974645219 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff68c011950 con 0x7ff6900a7740 2019-12-05T06:07:05.419 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:05.376+0000 7ff697642700 1 --2- 172.21.15.42:0/1974645219 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff6900a7740 0x7ff69008df80 secure :-1 s=READY pgs=2057 cs=0 l=1 rx=0x7ff68c002f30 tx=0x7ff68c00fd30).stop 2019-12-05T06:07:05.419 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:05.376+0000 7ff697642700 1 -- 172.21.15.42:0/1974645219 shutdown_connections 2019-12-05T06:07:05.419 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:05.376+0000 7ff697642700 1 --2- 172.21.15.42:0/1974645219 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff6900a7740 0x7ff69008df80 unknown :-1 s=CLOSED pgs=2057 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:05.419 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:05.376+0000 7ff697642700 1 --2- 172.21.15.42:0/1974645219 >> v2:172.21.15.40:3300/0 conn(0x7ff69007e720 0x7ff69007eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:07:05.420 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:05.376+0000 7ff697642700 1 -- 172.21.15.42:0/1974645219 shutdown_connections 2019-12-05T06:07:05.420 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:05.376+0000 7ff697642700 1 -- 172.21.15.42:0/1974645219 wait complete. 2019-12-05T06:07:05.420 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:05.376+0000 7ff697642700 1 -- 172.21.15.42:0/1974645219 >> 172.21.15.42:0/1974645219 conn(0x7ff6900aabe0 msgr2=0x7ff6900a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:07:05.420 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:05.376+0000 7ff697642700 1 Processor -- start 2019-12-05T06:07:05.420 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:05.376+0000 7ff697642700 1 -- start start 2019-12-05T06:07:05.420 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:05.376+0000 7ff697642700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff690116500 0x7ff690118500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:07:05.420 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:05.376+0000 7ff697642700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff690118a50 0x7ff69007ad00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:07:05.421 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:05.376+0000 7ff697642700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff69007b250 0x7ff690118f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:07:05.421 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:05.376+0000 7ff697642700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff690117a60 con 0x7ff690116500 2019-12-05T06:07:05.421 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:05.376+0000 7ff697642700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff690117d00 con 0x7ff690118a50 2019-12-05T06:07:05.421 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:05.376+0000 7ff697642700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff690117fa0 con 0x7ff69007b250 2019-12-05T06:07:05.421 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:05.376+0000 7ff6953de700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff690118a50 0x7ff69007ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:05.421 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:05.376+0000 7ff6953de700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff690118a50 0x7ff69007ad00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:48500/0 (socket says 172.21.15.42:48500) 2019-12-05T06:07:05.422 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:05.380+0000 7ff6953de700 1 -- 172.21.15.42:0/1265505932 learned_addr learned my addr 172.21.15.42:0/1265505932 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:07:05.422 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:05.380+0000 7ff695bdf700 1 --2- 172.21.15.42:0/1265505932 >> v2:172.21.15.40:3300/0 conn(0x7ff690116500 0x7ff690118500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:05.422 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:05.380+0000 7ff6963e0700 1 --2- 172.21.15.42:0/1265505932 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff69007b250 0x7ff690118f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:05.422 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:05.380+0000 7ff6953de700 1 -- 172.21.15.42:0/1265505932 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff69007b250 msgr2=0x7ff690118f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:05.422 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:05.380+0000 7ff6953de700 1 --2- 172.21.15.42:0/1265505932 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff69007b250 0x7ff690118f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:05.422 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:05.380+0000 7ff6953de700 1 -- 172.21.15.42:0/1265505932 >> v2:172.21.15.40:3300/0 conn(0x7ff690116500 msgr2=0x7ff690118500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:05.422 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:05.380+0000 7ff6953de700 1 --2- 172.21.15.42:0/1265505932 >> v2:172.21.15.40:3300/0 conn(0x7ff690116500 0x7ff690118500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:05.423 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:05.380+0000 7ff6953de700 1 -- 172.21.15.42:0/1265505932 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6901176c0 con 0x7ff690118a50 2019-12-05T06:07:05.423 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:05.380+0000 7ff6963e0700 1 --2- 172.21.15.42:0/1265505932 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff69007b250 0x7ff690118f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:07:05.423 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:05.380+0000 7ff6953de700 1 --2- 172.21.15.42:0/1265505932 >> v2:172.21.15.42:3300/0 conn(0x7ff690118a50 0x7ff69007ad00 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7ff690117ec0 tx=0x7ff684011040).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:05.423 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:05.380+0000 7ff6953de700 1 --2- 172.21.15.42:0/1265505932 >> v2:172.21.15.42:3300/0 conn(0x7ff690118a50 0x7ff69007ad00 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7ff690117ec0 tx=0x7ff684011040).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:05.423 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:05.380+0000 7ff6953de700 1 --2- 172.21.15.42:0/1265505932 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff690118a50 0x7ff69007ad00 secure :-1 s=READY pgs=2058 cs=0 l=1 rx=0x7ff690117ec0 tx=0x7ff684011040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:07:05.423 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:05.380+0000 7ff6953de700 1 --2- 172.21.15.42:0/1265505932 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff690118a50 0x7ff69007ad00 secure :-1 s=THROTTLE_DONE pgs=2058 cs=0 l=1 rx=0x7ff690117ec0 tx=0x7ff684011040).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:05.424 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:05.380+0000 7ff6827fc700 1 -- 172.21.15.42:0/1265505932 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff684007350 con 0x7ff690118a50 2019-12-05T06:07:05.424 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:05.380+0000 7ff6953de700 1 --2- 172.21.15.42:0/1265505932 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff690118a50 0x7ff69007ad00 secure :-1 s=THROTTLE_DONE pgs=2058 cs=0 l=1 rx=0x7ff690117ec0 tx=0x7ff684011040).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:05.424 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:05.380+0000 7ff6953de700 1 --2- 172.21.15.42:0/1265505932 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff690118a50 0x7ff69007ad00 secure :-1 s=THROTTLE_DONE pgs=2058 cs=0 l=1 rx=0x7ff690117ec0 tx=0x7ff684011040).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:05.424 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:05.380+0000 7ff6827fc700 1 -- 172.21.15.42:0/1265505932 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff684005cb0 con 0x7ff690118a50 2019-12-05T06:07:05.424 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:05.380+0000 7ff697642700 1 -- 172.21.15.42:0/1265505932 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6901194d0 con 0x7ff690118a50 2019-12-05T06:07:05.424 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:05.380+0000 7ff6827fc700 1 -- 172.21.15.42:0/1265505932 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff684012870 con 0x7ff690118a50 2019-12-05T06:07:05.424 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:05.380+0000 7ff697642700 1 -- 172.21.15.42:0/1265505932 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff69008eb50 con 0x7ff690118a50 2019-12-05T06:07:05.425 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:05.380+0000 7ff6953de700 1 --2- 172.21.15.42:0/1265505932 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff690118a50 0x7ff69007ad00 secure :-1 s=THROTTLE_DONE pgs=2058 cs=0 l=1 rx=0x7ff690117ec0 tx=0x7ff684011040).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:05.425 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:05.380+0000 7ff6827fc700 1 -- 172.21.15.42:0/1265505932 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7ff68400e180 con 0x7ff690118a50 2019-12-05T06:07:05.425 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:05.380+0000 7ff6953de700 1 --2- 172.21.15.42:0/1265505932 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff690118a50 0x7ff69007ad00 secure :-1 s=THROTTLE_DONE pgs=2058 cs=0 l=1 rx=0x7ff690117ec0 tx=0x7ff684011040).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:05.425 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:05.380+0000 7ff6827fc700 1 --2- 172.21.15.42:0/1265505932 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff67803aef0 0x7ff67803d400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:07:05.425 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:05.380+0000 7ff6827fc700 1 -- 172.21.15.42:0/1265505932 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9815+0+0 (secure 0 0 0) 0x7ff68404b0b0 con 0x7ff690118a50 2019-12-05T06:07:05.425 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:05.380+0000 7ff695bdf700 1 --2- 172.21.15.42:0/1265505932 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff67803aef0 0x7ff67803d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:05.425 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:05.380+0000 7ff695bdf700 1 --2- 172.21.15.42:0/1265505932 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff67803aef0 0x7ff67803d400 crc :-1 s=READY pgs=874 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:07:05.426 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:05.388+0000 7ff697642700 1 -- 172.21.15.42:0/1265505932 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff690118160 con 0x7ff690118a50 2019-12-05T06:07:05.426 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:05.396+0000 7ff6953de700 1 --2- 172.21.15.42:0/1265505932 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff690118a50 0x7ff69007ad00 secure :-1 s=THROTTLE_DONE pgs=2058 cs=0 l=1 rx=0x7ff690117ec0 tx=0x7ff684011040).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:05.426 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:05.396+0000 7ff6827fc700 1 -- 172.21.15.42:0/1265505932 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7ff684032130 con 0x7ff690118a50 2019-12-05T06:07:05.572 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:05.568+0000 7ff697642700 1 -- 172.21.15.42:0/1265505932 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff69005d110 con 0x7ff690118a50 2019-12-05T06:07:05.573 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:05.568+0000 7ff6953de700 1 --2- 172.21.15.42:0/1265505932 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff690118a50 0x7ff69007ad00 secure :-1 s=THROTTLE_DONE pgs=2058 cs=0 l=1 rx=0x7ff690117ec0 tx=0x7ff684011040).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:05.574 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:05.568+0000 7ff6827fc700 1 -- 172.21.15.42:0/1265505932 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 252 v252) v1 ==== 95+0+1775 (secure 0 0 0) 0x7ff684032390 con 0x7ff690118a50 2019-12-05T06:07:05.574 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 252 2019-12-05T06:07:05.574 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:07:05.574 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":252,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":252,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:07:04.524680+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":93,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9573},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9573":{"gid":9573,"name":"b","rank":0,"incarnation":249,"state":"up:active","state_seq":25,"addr":"172.21.15.40:6833/2968227232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2968227232},{"type":"v1","addr":"172.21.15.40:6833","nonce":2968227232}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:07:05.577 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:05.572+0000 7ff697642700 1 -- 172.21.15.42:0/1265505932 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff67803aef0 msgr2=0x7ff67803d400 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:05.577 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:05.572+0000 7ff697642700 1 --2- 172.21.15.42:0/1265505932 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff67803aef0 0x7ff67803d400 crc :-1 s=READY pgs=874 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:05.577 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:05.572+0000 7ff697642700 1 -- 172.21.15.42:0/1265505932 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff690118a50 msgr2=0x7ff69007ad00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:05.577 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:05.572+0000 7ff697642700 1 --2- 172.21.15.42:0/1265505932 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff690118a50 0x7ff69007ad00 secure :-1 s=READY pgs=2058 cs=0 l=1 rx=0x7ff690117ec0 tx=0x7ff684011040).stop 2019-12-05T06:07:05.577 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:05.572+0000 7ff697642700 1 -- 172.21.15.42:0/1265505932 shutdown_connections 2019-12-05T06:07:05.577 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:05.572+0000 7ff697642700 1 --2- 172.21.15.42:0/1265505932 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff67803aef0 0x7ff67803d400 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:05.578 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:05.572+0000 7ff697642700 1 --2- 172.21.15.42:0/1265505932 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff69007b250 0x7ff690118f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:05.578 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:05.572+0000 7ff697642700 1 --2- 172.21.15.42:0/1265505932 >> v2:172.21.15.40:3300/0 conn(0x7ff690116500 0x7ff690118500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:05.578 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:05.572+0000 7ff697642700 1 --2- 172.21.15.42:0/1265505932 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff690118a50 0x7ff69007ad00 unknown :-1 s=CLOSED pgs=2058 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:05.578 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:05.572+0000 7ff697642700 1 -- 172.21.15.42:0/1265505932 shutdown_connections 2019-12-05T06:07:05.578 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:05.572+0000 7ff697642700 1 -- 172.21.15.42:0/1265505932 wait complete. 2019-12-05T06:07:05.578 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:05.572+0000 7ff697642700 1 -- 172.21.15.42:0/1265505932 >> 172.21.15.42:0/1265505932 conn(0x7ff6900a8a70 msgr2=0x7ff69009f580 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:07:05.586 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b has gained rank=0, replacing gid=6821 2019-12-05T06:07:05.587 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 3 secs before reviving mds.a 2019-12-05T06:07:08.587 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.a 2019-12-05T06:07:08.587 INFO:tasks.ceph.mds.a:Restarting daemon 2019-12-05T06:07:08.588 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2019-12-05T06:07:08.595 INFO:tasks.ceph.mds.a:Started 2019-12-05T06:07:08.595 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:07:08.656 INFO:tasks.ceph.mds.a.smithi042.stdout:starting mds.a at 2019-12-05T06:07:08.705 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:08.688+0000 7f66f439b700 1 Processor -- start 2019-12-05T06:07:08.705 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:08.688+0000 7f66f439b700 1 -- start start 2019-12-05T06:07:08.705 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:08.688+0000 7f66f439b700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f66ec0a7740 0x7f66ec08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:07:08.705 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:08.688+0000 7f66f439b700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f66ec07e720 0x7f66ec07eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:07:08.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:08.688+0000 7f66f439b700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f66ec09f2e0 0x7f66ec09f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:07:08.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:08.688+0000 7f66f439b700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66ec0a0b00 con 0x7f66ec09f2e0 2019-12-05T06:07:08.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:08.688+0000 7f66f439b700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66ec08e5b0 con 0x7f66ec07e720 2019-12-05T06:07:08.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:08.688+0000 7f66f439b700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66ec08e850 con 0x7f66ec0a7740 2019-12-05T06:07:08.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:08.688+0000 7f66f2137700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f66ec07e720 0x7f66ec07eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:08.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:08.688+0000 7f66f2137700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f66ec07e720 0x7f66ec07eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:48542/0 (socket says 172.21.15.42:48542) 2019-12-05T06:07:08.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:08.688+0000 7f66f2137700 1 -- 172.21.15.42:0/905577868 learned_addr learned my addr 172.21.15.42:0/905577868 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:07:08.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:08.688+0000 7f66f2938700 1 --2- 172.21.15.42:0/905577868 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f66ec0a7740 0x7f66ec08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:08.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:08.688+0000 7f66f3139700 1 --2- 172.21.15.42:0/905577868 >> v2:172.21.15.40:3300/0 conn(0x7f66ec09f2e0 0x7f66ec09f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:08.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:08.688+0000 7f66f2137700 1 -- 172.21.15.42:0/905577868 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f66ec0a7740 msgr2=0x7f66ec08df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:07:08.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:08.688+0000 7f66f2137700 1 --2- 172.21.15.42:0/905577868 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f66ec0a7740 0x7f66ec08df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:07:08.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:08.688+0000 7f66f2137700 1 -- 172.21.15.42:0/905577868 >> v2:172.21.15.40:3300/0 conn(0x7f66ec09f2e0 msgr2=0x7f66ec09f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:07:08.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:08.688+0000 7f66f2137700 1 --2- 172.21.15.42:0/905577868 >> v2:172.21.15.40:3300/0 conn(0x7f66ec09f2e0 0x7f66ec09f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:07:08.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:08.688+0000 7f66f2137700 1 -- 172.21.15.42:0/905577868 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66ec07f0a0 con 0x7f66ec07e720 2019-12-05T06:07:08.708 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:08.688+0000 7f66f2938700 1 --2- 172.21.15.42:0/905577868 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f66ec0a7740 0x7f66ec08df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:07:08.708 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:08.688+0000 7f66f3139700 1 --2- 172.21.15.42:0/905577868 >> v2:172.21.15.40:3300/0 conn(0x7f66ec09f2e0 0x7f66ec09f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:07:08.708 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:08.688+0000 7f66f2137700 1 --2- 172.21.15.42:0/905577868 >> v2:172.21.15.42:3300/0 conn(0x7f66ec07e720 0x7f66ec07eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f66ec08e770 tx=0x7f66dc010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:08.708 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:08.688+0000 7f66f2137700 1 --2- 172.21.15.42:0/905577868 >> v2:172.21.15.42:3300/0 conn(0x7f66ec07e720 0x7f66ec07eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f66ec08e770 tx=0x7f66dc010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:08.708 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:08.688+0000 7f66f2137700 1 --2- 172.21.15.42:0/905577868 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f66ec07e720 0x7f66ec07eb50 secure :-1 s=READY pgs=2063 cs=0 l=1 rx=0x7f66ec08e770 tx=0x7f66dc010130).ready entity=mon.1 client_cookie=567a5080168e244d server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:07:08.708 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:08.688+0000 7f66f2137700 1 --2- 172.21.15.42:0/905577868 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f66ec07e720 0x7f66ec07eb50 secure :-1 s=THROTTLE_DONE pgs=2063 cs=0 l=1 rx=0x7f66ec08e770 tx=0x7f66dc010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:08.708 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:08.688+0000 7f66f2137700 1 --2- 172.21.15.42:0/905577868 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f66ec07e720 0x7f66ec07eb50 secure :-1 s=THROTTLE_DONE pgs=2063 cs=0 l=1 rx=0x7f66ec08e770 tx=0x7f66dc010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:08.709 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:08.688+0000 7f66f1936700 1 -- 172.21.15.42:0/905577868 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f66dc010de0 con 0x7f66ec07e720 2019-12-05T06:07:08.709 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:08.692+0000 7f66f2137700 1 --2- 172.21.15.42:0/905577868 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f66ec07e720 0x7f66ec07eb50 secure :-1 s=THROTTLE_DONE pgs=2063 cs=0 l=1 rx=0x7f66ec08e770 tx=0x7f66dc010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:08.709 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:08.692+0000 7f66f1936700 1 -- 172.21.15.42:0/905577868 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66dc010a00 con 0x7f66ec07e720 2019-12-05T06:07:08.709 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:08.692+0000 7f66f1936700 1 -- 172.21.15.42:0/905577868 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f66dc0114a0 con 0x7f66ec07e720 2019-12-05T06:07:08.709 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:08.692+0000 7f66f439b700 1 -- 172.21.15.42:0/905577868 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f66ec07e720 msgr2=0x7f66ec07eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:08.709 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:08.692+0000 7f66f439b700 1 --2- 172.21.15.42:0/905577868 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f66ec07e720 0x7f66ec07eb50 secure :-1 s=READY pgs=2063 cs=0 l=1 rx=0x7f66ec08e770 tx=0x7f66dc010130).stop 2019-12-05T06:07:08.709 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:08.692+0000 7f66f439b700 1 -- 172.21.15.42:0/905577868 shutdown_connections 2019-12-05T06:07:08.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:08.692+0000 7f66f439b700 1 --2- 172.21.15.42:0/905577868 >> v2:172.21.15.40:3300/0 conn(0x7f66ec09f2e0 0x7f66ec09f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:07:08.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:08.692+0000 7f66f439b700 1 --2- 172.21.15.42:0/905577868 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f66ec0a7740 0x7f66ec08df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:07:08.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:08.692+0000 7f66f439b700 1 --2- 172.21.15.42:0/905577868 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f66ec07e720 0x7f66ec07eb50 unknown :-1 s=CLOSED pgs=2063 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:08.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:08.692+0000 7f66f439b700 1 -- 172.21.15.42:0/905577868 shutdown_connections 2019-12-05T06:07:08.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:08.692+0000 7f66f439b700 1 -- 172.21.15.42:0/905577868 wait complete. 2019-12-05T06:07:08.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:08.692+0000 7f66f439b700 1 -- 172.21.15.42:0/905577868 >> 172.21.15.42:0/905577868 conn(0x7f66ec0aabe0 msgr2=0x7f66ec0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:07:08.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:08.692+0000 7f66f439b700 1 Processor -- start 2019-12-05T06:07:08.711 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:08.692+0000 7f66f439b700 1 -- start start 2019-12-05T06:07:08.711 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:08.696+0000 7f66f439b700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f66ec131a80 0x7f66ec131eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:07:08.711 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:08.696+0000 7f66f439b700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66ec121bd0 con 0x7f66ec124870 2019-12-05T06:07:08.711 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:08.696+0000 7f66f439b700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66ec11e540 con 0x7f66ec071430 2019-12-05T06:07:08.711 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:08.696+0000 7f66f439b700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66ec122320 con 0x7f66ec131a80 2019-12-05T06:07:08.711 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:08.696+0000 7f66f3139700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f66ec071430 0x7f66ec071860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:48588/0 (socket says 172.21.15.42:48588) 2019-12-05T06:07:08.712 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:08.696+0000 7f66f3139700 1 -- 172.21.15.42:0/2438515852 learned_addr learned my addr 172.21.15.42:0/2438515852 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:07:08.712 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:08.696+0000 7f66f2137700 1 --2- 172.21.15.42:0/2438515852 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f66ec131a80 0x7f66ec131eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:08.712 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:08.696+0000 7f66f2137700 1 -- 172.21.15.42:0/2438515852 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f66ec131a80 msgr2=0x7f66ec131eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2019-12-05T06:07:08.712 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:08.696+0000 7f66f2137700 1 -- 172.21.15.42:0/2438515852 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f66ec131a80 msgr2=0x7f66ec131eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2019-12-05T06:07:08.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:08.696+0000 7f66f2137700 1 --2- 172.21.15.42:0/2438515852 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f66ec131a80 0x7f66ec131eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-12-05T06:07:08.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:08.696+0000 7f66f2137700 1 --2- 172.21.15.42:0/2438515852 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f66ec131a80 0x7f66ec131eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rx=0 tx=0)._fault waiting 0.200000 2019-12-05T06:07:08.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:08.696+0000 7f66eb7fe700 1 -- 172.21.15.42:0/2438515852 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 823852252 0 0) 0x7f66ec11e540 con 0x7f66ec071430 2019-12-05T06:07:08.715 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:08.696+0000 7f66eb7fe700 1 -- 172.21.15.42:0/2438515852 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66d00030b0 con 0x7f66ec071430 2019-12-05T06:07:08.715 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:08.696+0000 7f66eb7fe700 1 -- 172.21.15.42:0/2438515852 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3395814710 0 0) 0x7f66ec121bd0 con 0x7f66ec124870 2019-12-05T06:07:08.715 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:08.696+0000 7f66eb7fe700 1 -- 172.21.15.42:0/2438515852 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66ec11e540 con 0x7f66ec124870 2019-12-05T06:07:08.715 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:08.696+0000 7f66eb7fe700 1 -- 172.21.15.42:0/2438515852 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 523405264 0 0) 0x7f66d00030b0 con 0x7f66ec071430 2019-12-05T06:07:08.715 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:08.696+0000 7f66eb7fe700 1 -- 172.21.15.42:0/2438515852 --> v1:172.21.15.42:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f66ec121bd0 con 0x7f66ec071430 2019-12-05T06:07:08.715 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:08.696+0000 7f66eb7fe700 1 -- 172.21.15.42:0/2438515852 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7f66e4001510 con 0x7f66ec071430 2019-12-05T06:07:08.716 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:08.696+0000 7f66eb7fe700 1 -- 172.21.15.42:0/2438515852 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f66ec131a80 msgr2=0x7f66ec131eb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2019-12-05T06:07:08.716 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:08.696+0000 7f66eb7fe700 1 --2- 172.21.15.42:0/2438515852 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f66ec131a80 0x7f66ec131eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:08.716 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:08.696+0000 7f66eb7fe700 1 -- 172.21.15.42:0/2438515852 >> v1:172.21.15.42:6789/0 conn(0x7f66ec071430 legacy=0x7f66ec071860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:08.716 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:08.696+0000 7f66eb7fe700 1 -- 172.21.15.42:0/2438515852 >> v1:172.21.15.40:6789/0 conn(0x7f66ec124870 legacy=0x7f66ec120d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:08.716 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:08.696+0000 7f66eb7fe700 1 --2- 172.21.15.42:0/2438515852 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f66d0003ab0 0x7f66d0004420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:07:08.716 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:08.696+0000 7f66eb7fe700 1 --2- 172.21.15.42:0/2438515852 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f66d0003fc0 0x7f66d00049d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:07:08.716 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:08.696+0000 7f66eb7fe700 1 --2- 172.21.15.42:0/2438515852 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f66d0004f90 0x7f66d000b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:07:08.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:08.696+0000 7f66eb7fe700 1 -- 172.21.15.42:0/2438515852 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66ec122320 con 0x7f66d0003ab0 2019-12-05T06:07:08.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:08.696+0000 7f66eb7fe700 1 -- 172.21.15.42:0/2438515852 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66d000bd20 con 0x7f66d0004f90 2019-12-05T06:07:08.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:08.696+0000 7f66eb7fe700 1 -- 172.21.15.42:0/2438515852 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66d000bf60 con 0x7f66d0003fc0 2019-12-05T06:07:08.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:08.696+0000 7f66f3139700 1 --2- 172.21.15.42:0/2438515852 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f66d0004f90 0x7f66d000b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:08.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:08.696+0000 7f66f2137700 1 --2- 172.21.15.42:0/2438515852 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f66d0003fc0 0x7f66d00049d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:08.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:08.696+0000 7f66f2938700 1 --2- 172.21.15.42:0/2438515852 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f66d0003ab0 0x7f66d0004420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:08.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:08.696+0000 7f66f3139700 1 -- 172.21.15.42:0/2438515852 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f66d0003fc0 msgr2=0x7f66d00049d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:08.718 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:08.696+0000 7f66f3139700 1 --2- 172.21.15.42:0/2438515852 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f66d0003fc0 0x7f66d00049d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:08.718 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:08.696+0000 7f66f3139700 1 -- 172.21.15.42:0/2438515852 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f66d0003ab0 msgr2=0x7f66d0004420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:08.718 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:08.696+0000 7f66f3139700 1 --2- 172.21.15.42:0/2438515852 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f66d0003ab0 0x7f66d0004420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:08.718 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:08.696+0000 7f66f3139700 1 -- 172.21.15.42:0/2438515852 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66ec132400 con 0x7f66d0004f90 2019-12-05T06:07:08.718 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:08.700+0000 7f66f439b700 1 -- 172.21.15.42:0/2438515852 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66ec072060 con 0x7f66d0004f90 2019-12-05T06:07:08.718 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:08.700+0000 7f66f3139700 1 --2- 172.21.15.42:0/2438515852 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f66d0004f90 0x7f66d000b6b0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f66e4007950 tx=0x7f66e40113d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:08.718 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:08.700+0000 7f66f439b700 1 -- 172.21.15.42:0/2438515852 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66ec131a80 con 0x7f66d0004f90 2019-12-05T06:07:08.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:08.700+0000 7f66f3139700 1 --2- 172.21.15.42:0/2438515852 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f66d0004f90 0x7f66d000b6b0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f66e4007950 tx=0x7f66e40113d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:08.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:08.700+0000 7f66f3139700 1 --2- 172.21.15.42:0/2438515852 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f66d0004f90 0x7f66d000b6b0 secure :-1 s=READY pgs=2065 cs=0 l=1 rx=0x7f66e4007950 tx=0x7f66e40113d0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:07:08.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:08.700+0000 7f66f3139700 1 --2- 172.21.15.42:0/2438515852 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f66d0004f90 0x7f66d000b6b0 secure :-1 s=THROTTLE_DONE pgs=2065 cs=0 l=1 rx=0x7f66e4007950 tx=0x7f66e40113d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:08.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:08.700+0000 7f66eb7fe700 1 -- 172.21.15.42:0/2438515852 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f66e400b440 con 0x7f66d0004f90 2019-12-05T06:07:08.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:08.700+0000 7f66f3139700 1 --2- 172.21.15.42:0/2438515852 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f66d0004f90 0x7f66d000b6b0 secure :-1 s=THROTTLE_DONE pgs=2065 cs=0 l=1 rx=0x7f66e4007950 tx=0x7f66e40113d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:08.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:08.700+0000 7f66f3139700 1 --2- 172.21.15.42:0/2438515852 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f66d0004f90 0x7f66d000b6b0 secure :-1 s=THROTTLE_DONE pgs=2065 cs=0 l=1 rx=0x7f66e4007950 tx=0x7f66e40113d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:08.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:08.700+0000 7f66eb7fe700 1 -- 172.21.15.42:0/2438515852 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66e4011ce0 con 0x7f66d0004f90 2019-12-05T06:07:08.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:08.700+0000 7f66eb7fe700 1 -- 172.21.15.42:0/2438515852 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f66e4005540 con 0x7f66d0004f90 2019-12-05T06:07:08.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:08.700+0000 7f66f3139700 1 --2- 172.21.15.42:0/2438515852 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f66d0004f90 0x7f66d000b6b0 secure :-1 s=THROTTLE_DONE pgs=2065 cs=0 l=1 rx=0x7f66e4007950 tx=0x7f66e40113d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:08.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:08.700+0000 7f66eb7fe700 1 -- 172.21.15.42:0/2438515852 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f66e4005740 con 0x7f66d0004f90 2019-12-05T06:07:08.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:08.700+0000 7f66f3139700 1 --2- 172.21.15.42:0/2438515852 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f66d0004f90 0x7f66d000b6b0 secure :-1 s=THROTTLE_DONE pgs=2065 cs=0 l=1 rx=0x7f66e4007950 tx=0x7f66e40113d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:08.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:08.700+0000 7f66eb7fe700 1 --2- 172.21.15.42:0/2438515852 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f66d0043da0 0x7f66d00462b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:07:08.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:08.700+0000 7f66eb7fe700 1 -- 172.21.15.42:0/2438515852 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9815+0+0 (secure 0 0 0) 0x7f66e404b350 con 0x7f66d0004f90 2019-12-05T06:07:08.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:08.700+0000 7f66f2938700 1 --2- 172.21.15.42:0/2438515852 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f66d0043da0 0x7f66d00462b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:08.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:08.700+0000 7f66f2938700 1 --2- 172.21.15.42:0/2438515852 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f66d0043da0 0x7f66d00462b0 crc :-1 s=READY pgs=875 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:07:08.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:08.704+0000 7f66f439b700 1 -- 172.21.15.42:0/2438515852 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66c4003200 con 0x7f66d0004f90 2019-12-05T06:07:08.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:08.712+0000 7f66f3139700 1 --2- 172.21.15.42:0/2438515852 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f66d0004f90 0x7f66d000b6b0 secure :-1 s=THROTTLE_DONE pgs=2065 cs=0 l=1 rx=0x7f66e4007950 tx=0x7f66e40113d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:08.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:08.712+0000 7f66eb7fe700 1 -- 172.21.15.42:0/2438515852 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f66e40323d0 con 0x7f66d0004f90 2019-12-05T06:07:08.889 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:08.884+0000 7f66f439b700 1 -- 172.21.15.42:0/2438515852 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f66c4003630 con 0x7f66d0004f90 2019-12-05T06:07:08.890 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:08.884+0000 7f66f3139700 1 --2- 172.21.15.42:0/2438515852 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f66d0004f90 0x7f66d000b6b0 secure :-1 s=THROTTLE_DONE pgs=2065 cs=0 l=1 rx=0x7f66e4007950 tx=0x7f66e40113d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:08.890 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:08.884+0000 7f66eb7fe700 1 -- 172.21.15.42:0/2438515852 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 252 v252) v1 ==== 95+0+1775 (secure 0 0 0) 0x7f66e4032630 con 0x7f66d0004f90 2019-12-05T06:07:08.890 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 252 2019-12-05T06:07:08.894 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:07:08.894 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":252,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":252,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:07:04.524680+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":93,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9573},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9573":{"gid":9573,"name":"b","rank":0,"incarnation":249,"state":"up:active","state_seq":25,"addr":"172.21.15.40:6833/2968227232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2968227232},{"type":"v1","addr":"172.21.15.40:6833","nonce":2968227232}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:07:08.895 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:08.888+0000 7f66f439b700 1 -- 172.21.15.42:0/2438515852 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f66d0043da0 msgr2=0x7f66d00462b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:08.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:08.888+0000 7f66f439b700 1 --2- 172.21.15.42:0/2438515852 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f66d0043da0 0x7f66d00462b0 crc :-1 s=READY pgs=875 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:08.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:08.888+0000 7f66f439b700 1 -- 172.21.15.42:0/2438515852 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f66d0004f90 msgr2=0x7f66d000b6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:08.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:08.888+0000 7f66f3139700 1 -- 172.21.15.42:0/2438515852 reap_dead start 2019-12-05T06:07:08.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:08.888+0000 7f66f439b700 1 --2- 172.21.15.42:0/2438515852 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f66d0004f90 0x7f66d000b6b0 secure :-1 s=READY pgs=2065 cs=0 l=1 rx=0x7f66e4007950 tx=0x7f66e40113d0).stop 2019-12-05T06:07:08.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:08.888+0000 7f66f439b700 1 -- 172.21.15.42:0/2438515852 shutdown_connections 2019-12-05T06:07:08.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:08.888+0000 7f66f439b700 1 --2- 172.21.15.42:0/2438515852 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f66d0004f90 0x7f66d000b6b0 unknown :-1 s=CLOSED pgs=2065 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:08.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:08.888+0000 7f66f439b700 1 -- 172.21.15.42:0/2438515852 shutdown_connections 2019-12-05T06:07:08.897 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:08.888+0000 7f66f439b700 1 -- 172.21.15.42:0/2438515852 wait complete. 2019-12-05T06:07:08.897 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:08.888+0000 7f66f439b700 1 -- 172.21.15.42:0/2438515852 >> 172.21.15.42:0/2438515852 conn(0x7f66ec09faf0 msgr2=0x7f66ec0a9790 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:07:08.902 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2019-12-05T06:07:10.907 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:07:11.026 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:11.016+0000 7f8d3d5df700 1 Processor -- start 2019-12-05T06:07:11.027 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:11.016+0000 7f8d3d5df700 1 -- start start 2019-12-05T06:07:11.027 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:11.016+0000 7f8d3d5df700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8d380a7740 0x7f8d3808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:07:11.027 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:11.016+0000 7f8d3d5df700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d380a0710 con 0x7f8d3807e720 2019-12-05T06:07:11.027 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:11.016+0000 7f8d3d5df700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d380a9ba0 con 0x7f8d3809fae0 2019-12-05T06:07:11.027 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:11.016+0000 7f8d3d5df700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d380a0b00 con 0x7f8d380a7740 2019-12-05T06:07:11.027 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:11.016+0000 7f8d37fff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f8d3809fae0 0x7f8d3814efe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:48620/0 (socket says 172.21.15.42:48620) 2019-12-05T06:07:11.028 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:11.016+0000 7f8d37fff700 1 -- 172.21.15.42:0/1904866066 learned_addr learned my addr 172.21.15.42:0/1904866066 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:07:11.028 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:11.016+0000 7f8d377fe700 1 --2- 172.21.15.42:0/1904866066 >> v2:172.21.15.40:3300/0 conn(0x7f8d380a7740 0x7f8d3808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:11.028 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:11.016+0000 7f8d37fff700 1 -- 172.21.15.42:0/1904866066 >> v1:172.21.15.42:6789/0 conn(0x7f8d3809fae0 legacy=0x7f8d3814efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2019-12-05T06:07:11.028 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:11.016+0000 7f8d37fff700 1 -- 172.21.15.42:0/1904866066 >> v1:172.21.15.42:6789/0 conn(0x7f8d3809fae0 legacy=0x7f8d3814efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2019-12-05T06:07:11.028 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:11.016+0000 7f8d37fff700 1 --1- 172.21.15.42:0/1904866066 >> v1:172.21.15.42:6789/0 conn(0x7f8d3809fae0 0x7f8d3814efe0 :-1 s=OPENED pgs=2067 cs=1 l=1).handle_message read tag failed 2019-12-05T06:07:11.028 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:11.016+0000 7f8d37fff700 1 --1- 172.21.15.42:0/1904866066 >> v1:172.21.15.42:6789/0 conn(0x7f8d3809fae0 0x7f8d3814efe0 :-1 s=OPENED pgs=2067 cs=1 l=1).fault on lossy channel, failing 2019-12-05T06:07:11.028 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:11.016+0000 7f8d367fc700 1 -- 172.21.15.42:0/1904866066 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3666742945 0 0) 0x7f8d380a0710 con 0x7f8d3807e720 2019-12-05T06:07:11.029 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:11.016+0000 7f8d367fc700 1 -- 172.21.15.42:0/1904866066 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d240030b0 con 0x7f8d3807e720 2019-12-05T06:07:11.029 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:11.016+0000 7f8d377fe700 1 -- 172.21.15.42:0/1904866066 >> v1:172.21.15.42:6789/0 conn(0x7f8d3809fae0 legacy=0x7f8d3814efe0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2019-12-05T06:07:11.029 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:11.016+0000 7f8d377fe700 1 -- 172.21.15.42:0/1904866066 >> v1:172.21.15.40:6789/0 conn(0x7f8d3807e720 legacy=0x7f8d3809f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:11.029 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:11.016+0000 7f8d377fe700 1 -- 172.21.15.42:0/1904866066 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d3807eb50 con 0x7f8d380a7740 2019-12-05T06:07:11.029 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:11.016+0000 7f8d377fe700 1 --2- 172.21.15.42:0/1904866066 >> v2:172.21.15.40:3300/0 conn(0x7f8d380a7740 0x7f8d3808df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f8d28002f30 tx=0x7f8d2800fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:11.029 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:11.020+0000 7f8d377fe700 1 --2- 172.21.15.42:0/1904866066 >> v2:172.21.15.40:3300/0 conn(0x7f8d380a7740 0x7f8d3808df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f8d28002f30 tx=0x7f8d2800fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:11.029 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:11.020+0000 7f8d377fe700 1 --2- 172.21.15.42:0/1904866066 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8d380a7740 0x7f8d3808df80 secure :-1 s=READY pgs=1576 cs=0 l=1 rx=0x7f8d28002f30 tx=0x7f8d2800fbf0).ready entity=mon.0 client_cookie=a03eaa42e33ff048 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:07:11.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:11.020+0000 7f8d377fe700 1 --2- 172.21.15.42:0/1904866066 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8d380a7740 0x7f8d3808df80 secure :-1 s=THROTTLE_DONE pgs=1576 cs=0 l=1 rx=0x7f8d28002f30 tx=0x7f8d2800fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:11.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:11.020+0000 7f8d367fc700 1 -- 172.21.15.42:0/1904866066 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8d28012060 con 0x7f8d380a7740 2019-12-05T06:07:11.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:11.020+0000 7f8d377fe700 1 --2- 172.21.15.42:0/1904866066 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8d380a7740 0x7f8d3808df80 secure :-1 s=THROTTLE_DONE pgs=1576 cs=0 l=1 rx=0x7f8d28002f30 tx=0x7f8d2800fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:11.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:11.020+0000 7f8d377fe700 1 --2- 172.21.15.42:0/1904866066 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8d380a7740 0x7f8d3808df80 secure :-1 s=THROTTLE_DONE pgs=1576 cs=0 l=1 rx=0x7f8d28002f30 tx=0x7f8d2800fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:11.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:11.020+0000 7f8d367fc700 1 -- 172.21.15.42:0/1904866066 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d280110c0 con 0x7f8d380a7740 2019-12-05T06:07:11.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:11.020+0000 7f8d367fc700 1 -- 172.21.15.42:0/1904866066 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8d28008da0 con 0x7f8d380a7740 2019-12-05T06:07:11.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:11.020+0000 7f8d3d5df700 1 -- 172.21.15.42:0/1904866066 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8d380a7740 msgr2=0x7f8d3808df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:11.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:11.020+0000 7f8d3d5df700 1 --2- 172.21.15.42:0/1904866066 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8d380a7740 0x7f8d3808df80 secure :-1 s=READY pgs=1576 cs=0 l=1 rx=0x7f8d28002f30 tx=0x7f8d2800fbf0).stop 2019-12-05T06:07:11.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:11.020+0000 7f8d3d5df700 1 -- 172.21.15.42:0/1904866066 shutdown_connections 2019-12-05T06:07:11.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:11.020+0000 7f8d3d5df700 1 --2- 172.21.15.42:0/1904866066 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8d380a7740 0x7f8d3808df80 unknown :-1 s=CLOSED pgs=1576 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:11.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:11.020+0000 7f8d3d5df700 1 -- 172.21.15.42:0/1904866066 shutdown_connections 2019-12-05T06:07:11.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:11.020+0000 7f8d3d5df700 1 -- 172.21.15.42:0/1904866066 wait complete. 2019-12-05T06:07:11.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:11.020+0000 7f8d3d5df700 1 -- 172.21.15.42:0/1904866066 >> 172.21.15.42:0/1904866066 conn(0x7f8d380aabe0 msgr2=0x7f8d380a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:07:11.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:11.024+0000 7f8d3d5df700 1 Processor -- start 2019-12-05T06:07:11.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:11.024+0000 7f8d3d5df700 1 -- start start 2019-12-05T06:07:11.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:11.024+0000 7f8d3d5df700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8d38117dd0 0x7f8d38116360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:07:11.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:11.024+0000 7f8d3d5df700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f8d380732f0 0x7f8d38073720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:07:11.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:11.024+0000 7f8d3d5df700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d3811b740 con 0x7f8d3811bf90 2019-12-05T06:07:11.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:11.024+0000 7f8d3d5df700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d3811b4d0 con 0x7f8d38117dd0 2019-12-05T06:07:11.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:11.024+0000 7f8d3d5df700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d3811ba50 con 0x7f8d380732f0 2019-12-05T06:07:11.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:11.024+0000 7f8d37fff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f8d380732f0 0x7f8d38073720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:11.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:11.024+0000 7f8d37fff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f8d380732f0 0x7f8d38073720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:48588/0 (socket says 172.21.15.42:48588) 2019-12-05T06:07:11.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:11.024+0000 7f8d377fe700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8d38117dd0 0x7f8d38116360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:11.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:11.024+0000 7f8d37fff700 1 -- 172.21.15.42:0/583040664 learned_addr learned my addr 172.21.15.42:0/583040664 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:07:11.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:11.024+0000 7f8d37fff700 1 -- 172.21.15.42:0/583040664 >> v2:172.21.15.40:3300/0 conn(0x7f8d38117dd0 msgr2=0x7f8d38116360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:11.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:11.024+0000 7f8d37fff700 1 --2- 172.21.15.42:0/583040664 >> v2:172.21.15.40:3300/0 conn(0x7f8d38117dd0 0x7f8d38116360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:11.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:11.024+0000 7f8d37fff700 1 -- 172.21.15.42:0/583040664 >> v1:172.21.15.42:6789/0 conn(0x7f8d3811bf90 legacy=0x7f8d3811acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:11.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:11.024+0000 7f8d37fff700 1 -- 172.21.15.42:0/583040664 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d3811a140 con 0x7f8d380732f0 2019-12-05T06:07:11.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:11.024+0000 7f8d377fe700 1 --2- 172.21.15.42:0/583040664 >> v2:172.21.15.40:3300/0 conn(0x7f8d38117dd0 0x7f8d38116360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:07:11.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:11.024+0000 7f8d37fff700 1 --2- 172.21.15.42:0/583040664 >> v2:172.21.15.42:3300/0 conn(0x7f8d380732f0 0x7f8d38073720 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f8d30004720 tx=0x7f8d30010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:11.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:11.024+0000 7f8d37fff700 1 --2- 172.21.15.42:0/583040664 >> v2:172.21.15.42:3300/0 conn(0x7f8d380732f0 0x7f8d38073720 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f8d30004720 tx=0x7f8d30010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:11.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:11.024+0000 7f8d37fff700 1 --2- 172.21.15.42:0/583040664 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8d380732f0 0x7f8d38073720 secure :-1 s=READY pgs=2069 cs=0 l=1 rx=0x7f8d30004720 tx=0x7f8d30010b60).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:07:11.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:11.028+0000 7f8d37fff700 1 --2- 172.21.15.42:0/583040664 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8d380732f0 0x7f8d38073720 secure :-1 s=THROTTLE_DONE pgs=2069 cs=0 l=1 rx=0x7f8d30004720 tx=0x7f8d30010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:11.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:11.028+0000 7f8d2ffff700 1 -- 172.21.15.42:0/583040664 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8d3000d0c0 con 0x7f8d380732f0 2019-12-05T06:07:11.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:11.028+0000 7f8d37fff700 1 --2- 172.21.15.42:0/583040664 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8d380732f0 0x7f8d38073720 secure :-1 s=THROTTLE_DONE pgs=2069 cs=0 l=1 rx=0x7f8d30004720 tx=0x7f8d30010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:11.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:11.028+0000 7f8d37fff700 1 --2- 172.21.15.42:0/583040664 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8d380732f0 0x7f8d38073720 secure :-1 s=THROTTLE_DONE pgs=2069 cs=0 l=1 rx=0x7f8d30004720 tx=0x7f8d30010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:11.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:11.028+0000 7f8d2ffff700 1 -- 172.21.15.42:0/583040664 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d30013060 con 0x7f8d380732f0 2019-12-05T06:07:11.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:11.028+0000 7f8d3d5df700 1 -- 172.21.15.42:0/583040664 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d3811c3c0 con 0x7f8d380732f0 2019-12-05T06:07:11.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:11.028+0000 7f8d2ffff700 1 -- 172.21.15.42:0/583040664 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8d3000d6a0 con 0x7f8d380732f0 2019-12-05T06:07:11.039 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:11.028+0000 7f8d3d5df700 1 -- 172.21.15.42:0/583040664 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d38073c70 con 0x7f8d380732f0 2019-12-05T06:07:11.039 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:11.028+0000 7f8d37fff700 1 --2- 172.21.15.42:0/583040664 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8d380732f0 0x7f8d38073720 secure :-1 s=THROTTLE_DONE pgs=2069 cs=0 l=1 rx=0x7f8d30004720 tx=0x7f8d30010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:11.039 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:11.028+0000 7f8d2ffff700 1 -- 172.21.15.42:0/583040664 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f8d3000d970 con 0x7f8d380732f0 2019-12-05T06:07:11.039 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:11.028+0000 7f8d37fff700 1 --2- 172.21.15.42:0/583040664 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8d380732f0 0x7f8d38073720 secure :-1 s=THROTTLE_DONE pgs=2069 cs=0 l=1 rx=0x7f8d30004720 tx=0x7f8d30010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:11.039 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:11.028+0000 7f8d2ffff700 1 --2- 172.21.15.42:0/583040664 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8d1c03ae40 0x7f8d1c03d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:07:11.039 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:11.028+0000 7f8d2ffff700 1 -- 172.21.15.42:0/583040664 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9815+0+0 (secure 0 0 0) 0x7f8d3004aed0 con 0x7f8d380732f0 2019-12-05T06:07:11.040 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:11.028+0000 7f8d377fe700 1 --2- 172.21.15.42:0/583040664 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8d1c03ae40 0x7f8d1c03d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:11.040 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:11.028+0000 7f8d377fe700 1 --2- 172.21.15.42:0/583040664 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8d1c03ae40 0x7f8d1c03d350 crc :-1 s=READY pgs=876 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:07:11.040 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:11.032+0000 7f8d3d5df700 1 -- 172.21.15.42:0/583040664 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d3805d110 con 0x7f8d380732f0 2019-12-05T06:07:11.043 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:11.036+0000 7f8d37fff700 1 --2- 172.21.15.42:0/583040664 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8d380732f0 0x7f8d38073720 secure :-1 s=THROTTLE_DONE pgs=2069 cs=0 l=1 rx=0x7f8d30004720 tx=0x7f8d30010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:11.044 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:11.036+0000 7f8d2ffff700 1 -- 172.21.15.42:0/583040664 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f8d30031f50 con 0x7f8d380732f0 2019-12-05T06:07:11.220 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:11.216+0000 7f8d3d5df700 1 -- 172.21.15.42:0/583040664 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8d3805b8c0 con 0x7f8d380732f0 2019-12-05T06:07:11.221 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:11.216+0000 7f8d37fff700 1 --2- 172.21.15.42:0/583040664 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8d380732f0 0x7f8d38073720 secure :-1 s=THROTTLE_DONE pgs=2069 cs=0 l=1 rx=0x7f8d30004720 tx=0x7f8d30010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:11.221 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:11.216+0000 7f8d2ffff700 1 -- 172.21.15.42:0/583040664 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 252 v252) v1 ==== 95+0+1775 (secure 0 0 0) 0x7f8d300321b0 con 0x7f8d380732f0 2019-12-05T06:07:11.222 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 252 2019-12-05T06:07:11.222 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:07:11.222 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":252,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":252,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:07:04.524680+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":93,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9573},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9573":{"gid":9573,"name":"b","rank":0,"incarnation":249,"state":"up:active","state_seq":25,"addr":"172.21.15.40:6833/2968227232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2968227232},{"type":"v1","addr":"172.21.15.40:6833","nonce":2968227232}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:07:11.225 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:11.220+0000 7f8d3d5df700 1 -- 172.21.15.42:0/583040664 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8d1c03ae40 msgr2=0x7f8d1c03d350 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:11.225 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:11.220+0000 7f8d3d5df700 1 --2- 172.21.15.42:0/583040664 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8d1c03ae40 0x7f8d1c03d350 crc :-1 s=READY pgs=876 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:11.226 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:11.220+0000 7f8d3d5df700 1 -- 172.21.15.42:0/583040664 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8d380732f0 msgr2=0x7f8d38073720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:11.226 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:11.220+0000 7f8d3d5df700 1 --2- 172.21.15.42:0/583040664 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8d380732f0 0x7f8d38073720 secure :-1 s=READY pgs=2069 cs=0 l=1 rx=0x7f8d30004720 tx=0x7f8d30010b60).stop 2019-12-05T06:07:11.226 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:11.220+0000 7f8d3d5df700 1 -- 172.21.15.42:0/583040664 shutdown_connections 2019-12-05T06:07:11.226 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:11.220+0000 7f8d3d5df700 1 --2- 172.21.15.42:0/583040664 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8d1c03ae40 0x7f8d1c03d350 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:11.227 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:11.220+0000 7f8d3d5df700 1 --2- 172.21.15.42:0/583040664 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8d380732f0 0x7f8d38073720 unknown :-1 s=CLOSED pgs=2069 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:11.227 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:11.220+0000 7f8d3d5df700 1 --2- 172.21.15.42:0/583040664 >> v2:172.21.15.40:3300/0 conn(0x7f8d38117dd0 0x7f8d38116360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:11.227 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:11.220+0000 7f8d3d5df700 1 -- 172.21.15.42:0/583040664 shutdown_connections 2019-12-05T06:07:11.227 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:11.220+0000 7f8d3d5df700 1 -- 172.21.15.42:0/583040664 wait complete. 2019-12-05T06:07:11.228 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:11.220+0000 7f8d3d5df700 1 -- 172.21.15.42:0/583040664 >> 172.21.15.42:0/583040664 conn(0x7f8d3807e720 msgr2=0x7f8d38059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:07:11.235 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2019-12-05T06:07:13.236 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:07:13.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:13.328+0000 7f4f9a45c700 1 Processor -- start 2019-12-05T06:07:13.341 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:13.328+0000 7f4f9a45c700 1 -- start start 2019-12-05T06:07:13.341 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:13.332+0000 7f4f9a45c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f4f940a7740 0x7f4f9408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:07:13.341 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:13.332+0000 7f4f9a45c700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f940a0710 con 0x7f4f9407e720 2019-12-05T06:07:13.341 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:13.332+0000 7f4f9a45c700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f940a9ba0 con 0x7f4f9409fae0 2019-12-05T06:07:13.341 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:13.332+0000 7f4f9a45c700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f940a0b00 con 0x7f4f940a7740 2019-12-05T06:07:13.341 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:13.332+0000 7f4f989f9700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f4f940a7740 0x7f4f9408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:13.342 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:13.332+0000 7f4f989f9700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f4f940a7740 0x7f4f9408df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:48596/0 (socket says 172.21.15.42:48596) 2019-12-05T06:07:13.342 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:13.332+0000 7f4f989f9700 1 -- 172.21.15.42:0/971330480 learned_addr learned my addr 172.21.15.42:0/971330480 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:07:13.342 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:13.332+0000 7f4f989f9700 1 -- 172.21.15.42:0/971330480 >> v1:172.21.15.42:6789/0 conn(0x7f4f9409fae0 legacy=0x7f4f9414efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:13.342 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:13.332+0000 7f4f989f9700 1 -- 172.21.15.42:0/971330480 >> v1:172.21.15.40:6789/0 conn(0x7f4f9407e720 legacy=0x7f4f9409f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:13.342 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:13.332+0000 7f4f989f9700 1 -- 172.21.15.42:0/971330480 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f9407eb50 con 0x7f4f940a7740 2019-12-05T06:07:13.343 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:13.332+0000 7f4f989f9700 1 --2- 172.21.15.42:0/971330480 >> v2:172.21.15.42:3300/0 conn(0x7f4f940a7740 0x7f4f9408df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f4f84008790 tx=0x7f4f8400fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:13.343 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:13.332+0000 7f4f989f9700 1 --2- 172.21.15.42:0/971330480 >> v2:172.21.15.42:3300/0 conn(0x7f4f940a7740 0x7f4f9408df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f4f84008790 tx=0x7f4f8400fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:13.343 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:13.332+0000 7f4f989f9700 1 --2- 172.21.15.42:0/971330480 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4f940a7740 0x7f4f9408df80 secure :-1 s=READY pgs=2071 cs=0 l=1 rx=0x7f4f84008790 tx=0x7f4f8400fcb0).ready entity=mon.1 client_cookie=faef4648fa257fa2 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:07:13.343 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:13.332+0000 7f4f989f9700 1 --2- 172.21.15.42:0/971330480 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4f940a7740 0x7f4f9408df80 secure :-1 s=THROTTLE_DONE pgs=2071 cs=0 l=1 rx=0x7f4f84008790 tx=0x7f4f8400fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:13.343 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:13.332+0000 7f4f937fe700 1 -- 172.21.15.42:0/971330480 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4f84011520 con 0x7f4f940a7740 2019-12-05T06:07:13.343 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:13.332+0000 7f4f989f9700 1 --2- 172.21.15.42:0/971330480 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4f940a7740 0x7f4f9408df80 secure :-1 s=THROTTLE_DONE pgs=2071 cs=0 l=1 rx=0x7f4f84008790 tx=0x7f4f8400fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:13.343 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:13.332+0000 7f4f937fe700 1 -- 172.21.15.42:0/971330480 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f84008ca0 con 0x7f4f940a7740 2019-12-05T06:07:13.344 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:13.332+0000 7f4f989f9700 1 --2- 172.21.15.42:0/971330480 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4f940a7740 0x7f4f9408df80 secure :-1 s=THROTTLE_DONE pgs=2071 cs=0 l=1 rx=0x7f4f84008790 tx=0x7f4f8400fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:13.344 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:13.332+0000 7f4f937fe700 1 -- 172.21.15.42:0/971330480 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4f84011b00 con 0x7f4f940a7740 2019-12-05T06:07:13.344 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:13.332+0000 7f4f9a45c700 1 -- 172.21.15.42:0/971330480 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4f940a7740 msgr2=0x7f4f9408df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:13.344 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:13.332+0000 7f4f9a45c700 1 --2- 172.21.15.42:0/971330480 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4f940a7740 0x7f4f9408df80 secure :-1 s=READY pgs=2071 cs=0 l=1 rx=0x7f4f84008790 tx=0x7f4f8400fcb0).stop 2019-12-05T06:07:13.344 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:13.332+0000 7f4f9a45c700 1 -- 172.21.15.42:0/971330480 shutdown_connections 2019-12-05T06:07:13.344 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:13.332+0000 7f4f9a45c700 1 --2- 172.21.15.42:0/971330480 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4f940a7740 0x7f4f9408df80 unknown :-1 s=CLOSED pgs=2071 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:13.344 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:13.336+0000 7f4f9a45c700 1 -- 172.21.15.42:0/971330480 shutdown_connections 2019-12-05T06:07:13.345 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:13.336+0000 7f4f9a45c700 1 -- 172.21.15.42:0/971330480 wait complete. 2019-12-05T06:07:13.345 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:13.336+0000 7f4f9a45c700 1 -- 172.21.15.42:0/971330480 >> 172.21.15.42:0/971330480 conn(0x7f4f940aabe0 msgr2=0x7f4f940a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:07:13.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:13.336+0000 7f4f9a45c700 1 Processor -- start 2019-12-05T06:07:13.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:13.336+0000 7f4f9a45c700 1 -- start start 2019-12-05T06:07:13.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:13.336+0000 7f4f9a45c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f4f940a3310 0x7f4f94126520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:07:13.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:13.336+0000 7f4f9a45c700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f940a2c10 con 0x7f4f94118a50 2019-12-05T06:07:13.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:13.336+0000 7f4f9a45c700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f940a25d0 con 0x7f4f94145dc0 2019-12-05T06:07:13.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:13.336+0000 7f4f9a45c700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f940a3af0 con 0x7f4f940a3310 2019-12-05T06:07:13.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:13.340+0000 7f4f93fff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f4f94145dc0 0x7f4f94125d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:48642/0 (socket says 172.21.15.42:48642) 2019-12-05T06:07:13.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:13.340+0000 7f4f93fff700 1 -- 172.21.15.42:0/1545719582 learned_addr learned my addr 172.21.15.42:0/1545719582 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:07:13.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:13.340+0000 7f4f991fa700 1 --2- 172.21.15.42:0/1545719582 >> v2:172.21.15.42:3300/0 conn(0x7f4f940a3310 0x7f4f94126520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:13.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:13.340+0000 7f4f917fa700 1 -- 172.21.15.42:0/1545719582 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4181400766 0 0) 0x7f4f940a25d0 con 0x7f4f94145dc0 2019-12-05T06:07:13.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:13.340+0000 7f4f917fa700 1 -- 172.21.15.42:0/1545719582 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f780030b0 con 0x7f4f94145dc0 2019-12-05T06:07:13.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:13.340+0000 7f4f991fa700 1 -- 172.21.15.42:0/1545719582 >> v1:172.21.15.42:6789/0 conn(0x7f4f94145dc0 legacy=0x7f4f94125d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:13.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:13.340+0000 7f4f917fa700 1 -- 172.21.15.42:0/1545719582 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1205816531 0 0) 0x7f4f940a2c10 con 0x7f4f94118a50 2019-12-05T06:07:13.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:13.340+0000 7f4f991fa700 1 -- 172.21.15.42:0/1545719582 >> v1:172.21.15.40:6789/0 conn(0x7f4f94118a50 legacy=0x7f4f940a1dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:13.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:13.340+0000 7f4f991fa700 1 -- 172.21.15.42:0/1545719582 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f941461f0 con 0x7f4f940a3310 2019-12-05T06:07:13.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:13.340+0000 7f4f991fa700 1 --2- 172.21.15.42:0/1545719582 >> v2:172.21.15.42:3300/0 conn(0x7f4f940a3310 0x7f4f94126520 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f4f8c001210 tx=0x7f4f8c010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:13.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:13.340+0000 7f4f991fa700 1 --2- 172.21.15.42:0/1545719582 >> v2:172.21.15.42:3300/0 conn(0x7f4f940a3310 0x7f4f94126520 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f4f8c001210 tx=0x7f4f8c010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:13.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:13.340+0000 7f4f991fa700 1 --2- 172.21.15.42:0/1545719582 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4f940a3310 0x7f4f94126520 secure :-1 s=READY pgs=2073 cs=0 l=1 rx=0x7f4f8c001210 tx=0x7f4f8c010ac0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:07:13.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:13.340+0000 7f4f991fa700 1 --2- 172.21.15.42:0/1545719582 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4f940a3310 0x7f4f94126520 secure :-1 s=THROTTLE_DONE pgs=2073 cs=0 l=1 rx=0x7f4f8c001210 tx=0x7f4f8c010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:13.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:13.340+0000 7f4f917fa700 1 -- 172.21.15.42:0/1545719582 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4f8c006ac0 con 0x7f4f940a3310 2019-12-05T06:07:13.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:13.340+0000 7f4f991fa700 1 --2- 172.21.15.42:0/1545719582 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4f940a3310 0x7f4f94126520 secure :-1 s=THROTTLE_DONE pgs=2073 cs=0 l=1 rx=0x7f4f8c001210 tx=0x7f4f8c010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:13.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:13.340+0000 7f4f917fa700 1 -- 172.21.15.42:0/1545719582 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f8c0066e0 con 0x7f4f940a3310 2019-12-05T06:07:13.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:13.340+0000 7f4f991fa700 1 --2- 172.21.15.42:0/1545719582 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4f940a3310 0x7f4f94126520 secure :-1 s=THROTTLE_DONE pgs=2073 cs=0 l=1 rx=0x7f4f8c001210 tx=0x7f4f8c010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:13.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:13.340+0000 7f4f917fa700 1 -- 172.21.15.42:0/1545719582 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4f8c002820 con 0x7f4f940a3310 2019-12-05T06:07:13.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:13.340+0000 7f4f9a45c700 1 -- 172.21.15.42:0/1545719582 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f94126a70 con 0x7f4f940a3310 2019-12-05T06:07:13.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:13.340+0000 7f4f9a45c700 1 -- 172.21.15.42:0/1545719582 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f941467f0 con 0x7f4f940a3310 2019-12-05T06:07:13.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:13.340+0000 7f4f991fa700 1 --2- 172.21.15.42:0/1545719582 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4f940a3310 0x7f4f94126520 secure :-1 s=THROTTLE_DONE pgs=2073 cs=0 l=1 rx=0x7f4f8c001210 tx=0x7f4f8c010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:13.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:13.340+0000 7f4f917fa700 1 -- 172.21.15.42:0/1545719582 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f4f8c002b80 con 0x7f4f940a3310 2019-12-05T06:07:13.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:13.340+0000 7f4f991fa700 1 --2- 172.21.15.42:0/1545719582 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4f940a3310 0x7f4f94126520 secure :-1 s=THROTTLE_DONE pgs=2073 cs=0 l=1 rx=0x7f4f8c001210 tx=0x7f4f8c010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:13.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:13.340+0000 7f4f917fa700 1 --2- 172.21.15.42:0/1545719582 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4f7803b1e0 0x7f4f7803d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:07:13.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:13.340+0000 7f4f917fa700 1 -- 172.21.15.42:0/1545719582 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9815+0+0 (secure 0 0 0) 0x7f4f8c04b0e0 con 0x7f4f940a3310 2019-12-05T06:07:13.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:13.340+0000 7f4f989f9700 1 --2- 172.21.15.42:0/1545719582 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4f7803b1e0 0x7f4f7803d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:13.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:13.344+0000 7f4f989f9700 1 --2- 172.21.15.42:0/1545719582 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4f7803b1e0 0x7f4f7803d6f0 crc :-1 s=READY pgs=878 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:07:13.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:13.344+0000 7f4f9a45c700 1 -- 172.21.15.42:0/1545719582 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f9405d110 con 0x7f4f940a3310 2019-12-05T06:07:13.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:13.352+0000 7f4f991fa700 1 --2- 172.21.15.42:0/1545719582 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4f940a3310 0x7f4f94126520 secure :-1 s=THROTTLE_DONE pgs=2073 cs=0 l=1 rx=0x7f4f8c001210 tx=0x7f4f8c010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:13.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:13.352+0000 7f4f917fa700 1 -- 172.21.15.42:0/1545719582 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f4f8c0323a0 con 0x7f4f940a3310 2019-12-05T06:07:13.522 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:13.516+0000 7f4f9a45c700 1 -- 172.21.15.42:0/1545719582 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4f9405b8c0 con 0x7f4f940a3310 2019-12-05T06:07:13.523 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:13.516+0000 7f4f991fa700 1 --2- 172.21.15.42:0/1545719582 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4f940a3310 0x7f4f94126520 secure :-1 s=THROTTLE_DONE pgs=2073 cs=0 l=1 rx=0x7f4f8c001210 tx=0x7f4f8c010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:13.524 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:13.516+0000 7f4f917fa700 1 -- 172.21.15.42:0/1545719582 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 253 v253) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f4f8c032f50 con 0x7f4f940a3310 2019-12-05T06:07:13.524 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 253 2019-12-05T06:07:13.524 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:07:13.524 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":253,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6929,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/2636699468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2636699468},{"type":"v1","addr":"172.21.15.42:6835","nonce":2636699468}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":253}],"filesystems":[{"mdsmap":{"epoch":252,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:07:04.524680+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":93,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9573},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9573":{"gid":9573,"name":"b","rank":0,"incarnation":249,"state":"up:active","state_seq":25,"addr":"172.21.15.40:6833/2968227232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2968227232},{"type":"v1","addr":"172.21.15.40:6833","nonce":2968227232}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:07:13.526 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:13.520+0000 7f4f9a45c700 1 -- 172.21.15.42:0/1545719582 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4f7803b1e0 msgr2=0x7f4f7803d6f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:13.527 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:13.520+0000 7f4f9a45c700 1 --2- 172.21.15.42:0/1545719582 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4f7803b1e0 0x7f4f7803d6f0 crc :-1 s=READY pgs=878 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:13.527 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:13.520+0000 7f4f9a45c700 1 -- 172.21.15.42:0/1545719582 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4f940a3310 msgr2=0x7f4f94126520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:13.528 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:13.520+0000 7f4f9a45c700 1 --2- 172.21.15.42:0/1545719582 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4f940a3310 0x7f4f94126520 secure :-1 s=READY pgs=2073 cs=0 l=1 rx=0x7f4f8c001210 tx=0x7f4f8c010ac0).stop 2019-12-05T06:07:13.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:13.520+0000 7f4f9a45c700 1 -- 172.21.15.42:0/1545719582 shutdown_connections 2019-12-05T06:07:13.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:13.520+0000 7f4f9a45c700 1 --2- 172.21.15.42:0/1545719582 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4f7803b1e0 0x7f4f7803d6f0 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:13.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:13.520+0000 7f4f9a45c700 1 --2- 172.21.15.42:0/1545719582 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4f940a3310 0x7f4f94126520 unknown :-1 s=CLOSED pgs=2073 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:13.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:13.520+0000 7f4f9a45c700 1 -- 172.21.15.42:0/1545719582 shutdown_connections 2019-12-05T06:07:13.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:13.520+0000 7f4f9a45c700 1 -- 172.21.15.42:0/1545719582 wait complete. 2019-12-05T06:07:13.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:13.520+0000 7f4f9a45c700 1 -- 172.21.15.42:0/1545719582 >> 172.21.15.42:0/1545719582 conn(0x7f4f9407ee20 msgr2=0x7f4f94059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:07:13.536 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a reported in up:standby state 2019-12-05T06:07:13.536 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 41 secs before thrashing 2019-12-05T06:07:13.846 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:07:13.860 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:07:43.926 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:07:43.930 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:07:54.538 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:07:54.645 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.632+0000 7fe2335bc700 1 Processor -- start 2019-12-05T06:07:54.646 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.636+0000 7fe2335bc700 1 -- start start 2019-12-05T06:07:54.646 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.636+0000 7fe2335bc700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe22c0a7730 0x7fe22c08df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:07:54.646 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.636+0000 7fe2335bc700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe22c07e710 0x7fe22c07eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:07:54.646 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.636+0000 7fe2335bc700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe22c0a0700 con 0x7fe22c09f2d0 2019-12-05T06:07:54.646 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.636+0000 7fe2335bc700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe22c0a0af0 con 0x7fe22c07e710 2019-12-05T06:07:54.646 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.636+0000 7fe2335bc700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe22c08e5a0 con 0x7fe22c0a7730 2019-12-05T06:07:54.647 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.636+0000 7fe231358700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe22c07e710 0x7fe22c07eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:54.647 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.636+0000 7fe231358700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe22c07e710 0x7fe22c07eb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:48754/0 (socket says 172.21.15.42:48754) 2019-12-05T06:07:54.647 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.636+0000 7fe231358700 1 -- 172.21.15.42:0/2262751471 learned_addr learned my addr 172.21.15.42:0/2262751471 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:07:54.647 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.636+0000 7fe231b59700 1 --2- 172.21.15.42:0/2262751471 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe22c0a7730 0x7fe22c08df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:54.647 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.636+0000 7fe231358700 1 -- 172.21.15.42:0/2262751471 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe22c0a7730 msgr2=0x7fe22c08df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:07:54.647 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.636+0000 7fe231358700 1 --2- 172.21.15.42:0/2262751471 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe22c0a7730 0x7fe22c08df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:07:54.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.636+0000 7fe231358700 1 -- 172.21.15.42:0/2262751471 >> v1:172.21.15.40:6789/0 conn(0x7fe22c09f2d0 legacy=0x7fe22c09f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:54.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.636+0000 7fe231358700 1 -- 172.21.15.42:0/2262751471 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe22c07f090 con 0x7fe22c07e710 2019-12-05T06:07:54.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.636+0000 7fe231358700 1 --2- 172.21.15.42:0/2262751471 >> v2:172.21.15.42:3300/0 conn(0x7fe22c07e710 0x7fe22c07eb40 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fe21c008790 tx=0x7fe21c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:54.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.636+0000 7fe231358700 1 --2- 172.21.15.42:0/2262751471 >> v2:172.21.15.42:3300/0 conn(0x7fe22c07e710 0x7fe22c07eb40 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fe21c008790 tx=0x7fe21c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:54.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.636+0000 7fe231358700 1 --2- 172.21.15.42:0/2262751471 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe22c07e710 0x7fe22c07eb40 secure :-1 s=READY pgs=2076 cs=0 l=1 rx=0x7fe21c008790 tx=0x7fe21c00fd30).ready entity=mon.1 client_cookie=537ce8ec8d4db619 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:07:54.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.636+0000 7fe231358700 1 --2- 172.21.15.42:0/2262751471 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe22c07e710 0x7fe22c07eb40 secure :-1 s=THROTTLE_DONE pgs=2076 cs=0 l=1 rx=0x7fe21c008790 tx=0x7fe21c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:54.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.636+0000 7fe231358700 1 --2- 172.21.15.42:0/2262751471 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe22c07e710 0x7fe22c07eb40 secure :-1 s=THROTTLE_DONE pgs=2076 cs=0 l=1 rx=0x7fe21c008790 tx=0x7fe21c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:54.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.636+0000 7fe231358700 1 --2- 172.21.15.42:0/2262751471 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe22c07e710 0x7fe22c07eb40 secure :-1 s=THROTTLE_DONE pgs=2076 cs=0 l=1 rx=0x7fe21c008790 tx=0x7fe21c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:54.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.636+0000 7fe230b57700 1 -- 172.21.15.42:0/2262751471 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe21c0113b0 con 0x7fe22c07e710 2019-12-05T06:07:54.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.636+0000 7fe230b57700 1 -- 172.21.15.42:0/2262751471 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe21c008bb0 con 0x7fe22c07e710 2019-12-05T06:07:54.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.636+0000 7fe230b57700 1 -- 172.21.15.42:0/2262751471 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe21c011990 con 0x7fe22c07e710 2019-12-05T06:07:54.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.636+0000 7fe2335bc700 1 -- 172.21.15.42:0/2262751471 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe22c07e710 msgr2=0x7fe22c07eb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:54.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.636+0000 7fe2335bc700 1 --2- 172.21.15.42:0/2262751471 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe22c07e710 0x7fe22c07eb40 secure :-1 s=READY pgs=2076 cs=0 l=1 rx=0x7fe21c008790 tx=0x7fe21c00fd30).stop 2019-12-05T06:07:54.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.636+0000 7fe2335bc700 1 -- 172.21.15.42:0/2262751471 shutdown_connections 2019-12-05T06:07:54.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.636+0000 7fe2335bc700 1 --2- 172.21.15.42:0/2262751471 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe22c0a7730 0x7fe22c08df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:07:54.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.636+0000 7fe2335bc700 1 --2- 172.21.15.42:0/2262751471 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe22c07e710 0x7fe22c07eb40 unknown :-1 s=CLOSED pgs=2076 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:54.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.636+0000 7fe2335bc700 1 -- 172.21.15.42:0/2262751471 shutdown_connections 2019-12-05T06:07:54.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.636+0000 7fe2335bc700 1 -- 172.21.15.42:0/2262751471 wait complete. 2019-12-05T06:07:54.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.636+0000 7fe2335bc700 1 -- 172.21.15.42:0/2262751471 >> 172.21.15.42:0/2262751471 conn(0x7fe22c0aabd0 msgr2=0x7fe22c0a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:07:54.656 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.640+0000 7fe2335bc700 1 Processor -- start 2019-12-05T06:07:54.656 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.640+0000 7fe2335bc700 1 -- start start 2019-12-05T06:07:54.656 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.644+0000 7fe2335bc700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe22c0a9ac0 0x7fe22c0a49c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:07:54.657 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.644+0000 7fe2335bc700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe22c0793e0 0x7fe22c079810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:07:54.657 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.644+0000 7fe2335bc700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe22c0a4f10 con 0x7fe22c0a1b80 2019-12-05T06:07:54.657 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.644+0000 7fe2335bc700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe22c0a6c40 con 0x7fe22c0793e0 2019-12-05T06:07:54.657 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.644+0000 7fe2335bc700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe22c0a6ee0 con 0x7fe22c0a9ac0 2019-12-05T06:07:54.658 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.644+0000 7fe231b59700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe22c0a9ac0 0x7fe22c0a49c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:54.658 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.644+0000 7fe231b59700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe22c0a9ac0 0x7fe22c0a49c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:48758/0 (socket says 172.21.15.42:48758) 2019-12-05T06:07:54.658 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.644+0000 7fe231b59700 1 -- 172.21.15.42:0/3776582904 learned_addr learned my addr 172.21.15.42:0/3776582904 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:07:54.658 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.644+0000 7fe23235a700 1 --2- 172.21.15.42:0/3776582904 >> v2:172.21.15.40:3300/0 conn(0x7fe22c0793e0 0x7fe22c079810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:54.658 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.644+0000 7fe231b59700 1 -- 172.21.15.42:0/3776582904 >> v2:172.21.15.40:3300/0 conn(0x7fe22c0793e0 msgr2=0x7fe22c079810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:54.658 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.644+0000 7fe231b59700 1 --2- 172.21.15.42:0/3776582904 >> v2:172.21.15.40:3300/0 conn(0x7fe22c0793e0 0x7fe22c079810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:54.659 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.644+0000 7fe231b59700 1 -- 172.21.15.42:0/3776582904 >> v1:172.21.15.40:6789/0 conn(0x7fe22c0a1b80 legacy=0x7fe22c0a5580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:54.659 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.644+0000 7fe231b59700 1 -- 172.21.15.42:0/3776582904 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe22c132500 con 0x7fe22c0a9ac0 2019-12-05T06:07:54.660 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.644+0000 7fe231b59700 1 --2- 172.21.15.42:0/3776582904 >> v2:172.21.15.42:3300/0 conn(0x7fe22c0a9ac0 0x7fe22c0a49c0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fe228003070 tx=0x7fe228010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:54.660 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.644+0000 7fe231b59700 1 --2- 172.21.15.42:0/3776582904 >> v2:172.21.15.42:3300/0 conn(0x7fe22c0a9ac0 0x7fe22c0a49c0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fe228003070 tx=0x7fe228010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:54.660 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.644+0000 7fe231b59700 1 --2- 172.21.15.42:0/3776582904 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe22c0a9ac0 0x7fe22c0a49c0 secure :-1 s=READY pgs=2077 cs=0 l=1 rx=0x7fe228003070 tx=0x7fe228010b60).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:07:54.660 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.644+0000 7fe231b59700 1 --2- 172.21.15.42:0/3776582904 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe22c0a9ac0 0x7fe22c0a49c0 secure :-1 s=THROTTLE_DONE pgs=2077 cs=0 l=1 rx=0x7fe228003070 tx=0x7fe228010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:54.661 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.644+0000 7fe2227fc700 1 -- 172.21.15.42:0/3776582904 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe228005540 con 0x7fe22c0a9ac0 2019-12-05T06:07:54.661 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.644+0000 7fe231b59700 1 --2- 172.21.15.42:0/3776582904 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe22c0a9ac0 0x7fe22c0a49c0 secure :-1 s=THROTTLE_DONE pgs=2077 cs=0 l=1 rx=0x7fe228003070 tx=0x7fe228010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:54.661 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.644+0000 7fe231b59700 1 --2- 172.21.15.42:0/3776582904 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe22c0a9ac0 0x7fe22c0a49c0 secure :-1 s=THROTTLE_DONE pgs=2077 cs=0 l=1 rx=0x7fe228003070 tx=0x7fe228010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:54.661 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.644+0000 7fe2227fc700 1 -- 172.21.15.42:0/3776582904 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe228007150 con 0x7fe22c0a9ac0 2019-12-05T06:07:54.661 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.644+0000 7fe2227fc700 1 -- 172.21.15.42:0/3776582904 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe228005b20 con 0x7fe22c0a9ac0 2019-12-05T06:07:54.661 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.644+0000 7fe2335bc700 1 -- 172.21.15.42:0/3776582904 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe22c1314f0 con 0x7fe22c0a9ac0 2019-12-05T06:07:54.661 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.644+0000 7fe2335bc700 1 -- 172.21.15.42:0/3776582904 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe22c131a20 con 0x7fe22c0a9ac0 2019-12-05T06:07:54.662 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.644+0000 7fe231b59700 1 --2- 172.21.15.42:0/3776582904 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe22c0a9ac0 0x7fe22c0a49c0 secure :-1 s=THROTTLE_DONE pgs=2077 cs=0 l=1 rx=0x7fe228003070 tx=0x7fe228010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:54.662 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.644+0000 7fe2227fc700 1 -- 172.21.15.42:0/3776582904 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fe2280123c0 con 0x7fe22c0a9ac0 2019-12-05T06:07:54.662 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.644+0000 7fe231b59700 1 --2- 172.21.15.42:0/3776582904 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe22c0a9ac0 0x7fe22c0a49c0 secure :-1 s=THROTTLE_DONE pgs=2077 cs=0 l=1 rx=0x7fe228003070 tx=0x7fe228010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:54.662 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.644+0000 7fe2227fc700 1 --2- 172.21.15.42:0/3776582904 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe21403adb0 0x7fe21403d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:07:54.662 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.644+0000 7fe2227fc700 1 -- 172.21.15.42:0/3776582904 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9815+0+0 (secure 0 0 0) 0x7fe22804adc0 con 0x7fe22c0a9ac0 2019-12-05T06:07:54.662 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.648+0000 7fe231358700 1 --2- 172.21.15.42:0/3776582904 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe21403adb0 0x7fe21403d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:54.662 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.648+0000 7fe231358700 1 --2- 172.21.15.42:0/3776582904 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe21403adb0 0x7fe21403d2c0 crc :-1 s=READY pgs=879 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:07:54.663 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.648+0000 7fe2335bc700 1 -- 172.21.15.42:0/3776582904 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe22c05d100 con 0x7fe22c0a9ac0 2019-12-05T06:07:54.663 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.652+0000 7fe231b59700 1 --2- 172.21.15.42:0/3776582904 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe22c0a9ac0 0x7fe22c0a49c0 secure :-1 s=THROTTLE_DONE pgs=2077 cs=0 l=1 rx=0x7fe228003070 tx=0x7fe228010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:54.663 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.652+0000 7fe2227fc700 1 -- 172.21.15.42:0/3776582904 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fe228031e40 con 0x7fe22c0a9ac0 2019-12-05T06:07:54.837 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.832+0000 7fe2335bc700 1 -- 172.21.15.42:0/3776582904 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe22c05b8b0 con 0x7fe22c0a9ac0 2019-12-05T06:07:54.838 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.832+0000 7fe231b59700 1 --2- 172.21.15.42:0/3776582904 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe22c0a9ac0 0x7fe22c0a49c0 secure :-1 s=THROTTLE_DONE pgs=2077 cs=0 l=1 rx=0x7fe228003070 tx=0x7fe228010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:54.838 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.832+0000 7fe2227fc700 1 -- 172.21.15.42:0/3776582904 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 253 v253) v1 ==== 95+0+2112 (secure 0 0 0) 0x7fe2280320a0 con 0x7fe22c0a9ac0 2019-12-05T06:07:54.839 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 253 2019-12-05T06:07:54.839 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:07:54.839 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":253,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6929,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/2636699468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2636699468},{"type":"v1","addr":"172.21.15.42:6835","nonce":2636699468}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":253}],"filesystems":[{"mdsmap":{"epoch":252,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:07:04.524680+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":93,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9573},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9573":{"gid":9573,"name":"b","rank":0,"incarnation":249,"state":"up:active","state_seq":25,"addr":"172.21.15.40:6833/2968227232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2968227232},{"type":"v1","addr":"172.21.15.40:6833","nonce":2968227232}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:07:54.842 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.836+0000 7fe2335bc700 1 -- 172.21.15.42:0/3776582904 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe21403adb0 msgr2=0x7fe21403d2c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:54.842 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.836+0000 7fe2335bc700 1 --2- 172.21.15.42:0/3776582904 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe21403adb0 0x7fe21403d2c0 crc :-1 s=READY pgs=879 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:54.842 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.836+0000 7fe2335bc700 1 -- 172.21.15.42:0/3776582904 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe22c0a9ac0 msgr2=0x7fe22c0a49c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:54.842 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.836+0000 7fe2335bc700 1 --2- 172.21.15.42:0/3776582904 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe22c0a9ac0 0x7fe22c0a49c0 secure :-1 s=READY pgs=2077 cs=0 l=1 rx=0x7fe228003070 tx=0x7fe228010b60).stop 2019-12-05T06:07:54.843 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.836+0000 7fe2335bc700 1 -- 172.21.15.42:0/3776582904 shutdown_connections 2019-12-05T06:07:54.843 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.836+0000 7fe2335bc700 1 --2- 172.21.15.42:0/3776582904 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe21403adb0 0x7fe21403d2c0 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:54.843 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.836+0000 7fe2335bc700 1 --2- 172.21.15.42:0/3776582904 >> v2:172.21.15.40:3300/0 conn(0x7fe22c0793e0 0x7fe22c079810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:54.843 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.836+0000 7fe2335bc700 1 --2- 172.21.15.42:0/3776582904 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe22c0a9ac0 0x7fe22c0a49c0 unknown :-1 s=CLOSED pgs=2077 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:54.843 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.836+0000 7fe2335bc700 1 -- 172.21.15.42:0/3776582904 shutdown_connections 2019-12-05T06:07:54.843 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.836+0000 7fe2335bc700 1 -- 172.21.15.42:0/3776582904 wait complete. 2019-12-05T06:07:54.844 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.836+0000 7fe2335bc700 1 -- 172.21.15.42:0/3776582904 >> 172.21.15.42:0/3776582904 conn(0x7fe22c09f3b0 msgr2=0x7fe22c08eeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:07:54.855 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.b (rank=0) 2019-12-05T06:07:54.856 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2019-12-05T06:07:54.856 INFO:teuthology.orchestra.run:waiting for 300 2019-12-05T06:07:54.872 INFO:tasks.ceph.mds.b:Stopped 2019-12-05T06:07:54.873 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:07:54.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.964+0000 7f125049e700 1 Processor -- start 2019-12-05T06:07:54.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.964+0000 7f125049e700 1 -- start start 2019-12-05T06:07:54.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.964+0000 7f125049e700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f12480a7730 0x7f124808df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:07:54.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.964+0000 7f125049e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f124807e710 0x7f124807eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:07:54.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.964+0000 7f125049e700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12480a0700 con 0x7f124809f2d0 2019-12-05T06:07:54.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.964+0000 7f125049e700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12480a0af0 con 0x7f124807e710 2019-12-05T06:07:54.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.964+0000 7f125049e700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f124808e5a0 con 0x7f12480a7730 2019-12-05T06:07:54.976 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.964+0000 7f124e23a700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f124807e710 0x7f124807eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:54.976 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.964+0000 7f124e23a700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f124807e710 0x7f124807eb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:48770/0 (socket says 172.21.15.42:48770) 2019-12-05T06:07:54.976 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.964+0000 7f124e23a700 1 -- 172.21.15.42:0/1616427032 learned_addr learned my addr 172.21.15.42:0/1616427032 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:07:54.976 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.964+0000 7f124ea3b700 1 --2- 172.21.15.42:0/1616427032 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f12480a7730 0x7f124808df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:54.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.964+0000 7f124e23a700 1 -- 172.21.15.42:0/1616427032 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f12480a7730 msgr2=0x7f124808df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:07:54.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.964+0000 7f124e23a700 1 --2- 172.21.15.42:0/1616427032 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f12480a7730 0x7f124808df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:07:54.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.964+0000 7f124e23a700 1 -- 172.21.15.42:0/1616427032 >> v1:172.21.15.42:6789/0 conn(0x7f124809f2d0 legacy=0x7f124809f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:54.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.964+0000 7f124e23a700 1 -- 172.21.15.42:0/1616427032 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f124807f090 con 0x7f124807e710 2019-12-05T06:07:54.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.964+0000 7f124e23a700 1 --2- 172.21.15.42:0/1616427032 >> v2:172.21.15.42:3300/0 conn(0x7f124807e710 0x7f124807eb40 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f1238008790 tx=0x7f123800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:54.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.964+0000 7f124e23a700 1 --2- 172.21.15.42:0/1616427032 >> v2:172.21.15.42:3300/0 conn(0x7f124807e710 0x7f124807eb40 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f1238008790 tx=0x7f123800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:54.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.964+0000 7f124e23a700 1 --2- 172.21.15.42:0/1616427032 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f124807e710 0x7f124807eb40 secure :-1 s=READY pgs=2079 cs=0 l=1 rx=0x7f1238008790 tx=0x7f123800fd30).ready entity=mon.1 client_cookie=3dd8a0223a90b436 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:07:54.978 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.964+0000 7f124e23a700 1 --2- 172.21.15.42:0/1616427032 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f124807e710 0x7f124807eb40 secure :-1 s=THROTTLE_DONE pgs=2079 cs=0 l=1 rx=0x7f1238008790 tx=0x7f123800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:54.978 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.968+0000 7f124da39700 1 -- 172.21.15.42:0/1616427032 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f12380113b0 con 0x7f124807e710 2019-12-05T06:07:54.978 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.968+0000 7f124e23a700 1 --2- 172.21.15.42:0/1616427032 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f124807e710 0x7f124807eb40 secure :-1 s=THROTTLE_DONE pgs=2079 cs=0 l=1 rx=0x7f1238008790 tx=0x7f123800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:54.978 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.968+0000 7f124e23a700 1 --2- 172.21.15.42:0/1616427032 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f124807e710 0x7f124807eb40 secure :-1 s=THROTTLE_DONE pgs=2079 cs=0 l=1 rx=0x7f1238008790 tx=0x7f123800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:54.978 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.968+0000 7f124da39700 1 -- 172.21.15.42:0/1616427032 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1238008bb0 con 0x7f124807e710 2019-12-05T06:07:54.978 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.968+0000 7f124da39700 1 -- 172.21.15.42:0/1616427032 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1238011990 con 0x7f124807e710 2019-12-05T06:07:54.979 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.968+0000 7f125049e700 1 -- 172.21.15.42:0/1616427032 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f124807e710 msgr2=0x7f124807eb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:54.979 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.968+0000 7f125049e700 1 --2- 172.21.15.42:0/1616427032 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f124807e710 0x7f124807eb40 secure :-1 s=READY pgs=2079 cs=0 l=1 rx=0x7f1238008790 tx=0x7f123800fd30).stop 2019-12-05T06:07:54.979 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.968+0000 7f125049e700 1 -- 172.21.15.42:0/1616427032 shutdown_connections 2019-12-05T06:07:54.979 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.968+0000 7f125049e700 1 --2- 172.21.15.42:0/1616427032 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f12480a7730 0x7f124808df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:07:54.979 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.968+0000 7f125049e700 1 --2- 172.21.15.42:0/1616427032 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f124807e710 0x7f124807eb40 unknown :-1 s=CLOSED pgs=2079 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:54.979 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.968+0000 7f125049e700 1 -- 172.21.15.42:0/1616427032 shutdown_connections 2019-12-05T06:07:54.979 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.968+0000 7f125049e700 1 -- 172.21.15.42:0/1616427032 wait complete. 2019-12-05T06:07:54.980 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.968+0000 7f125049e700 1 -- 172.21.15.42:0/1616427032 >> 172.21.15.42:0/1616427032 conn(0x7f12480aabd0 msgr2=0x7f12480a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:07:54.983 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.972+0000 7f125049e700 1 Processor -- start 2019-12-05T06:07:54.983 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.972+0000 7f125049e700 1 -- start start 2019-12-05T06:07:54.983 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.972+0000 7f125049e700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f124811a170 0x7f12480a4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:07:54.983 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.972+0000 7f125049e700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f12480a9ac0 0x7f1248117a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:07:54.984 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.972+0000 7f125049e700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f124811ae60 con 0x7f1248117f70 2019-12-05T06:07:54.984 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.972+0000 7f125049e700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f124811a680 con 0x7f124811a170 2019-12-05T06:07:54.984 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.972+0000 7f125049e700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f124811a920 con 0x7f12480a9ac0 2019-12-05T06:07:54.984 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.972+0000 7f124ea3b700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f124811a170 0x7f12480a4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:54.984 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.972+0000 7f124f23c700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f1248117f70 0x7f124807ccb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:48814/0 (socket says 172.21.15.42:48814) 2019-12-05T06:07:54.984 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.972+0000 7f124f23c700 1 -- 172.21.15.42:0/1686825547 learned_addr learned my addr 172.21.15.42:0/1686825547 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:07:54.984 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.972+0000 7f124e23a700 1 --2- 172.21.15.42:0/1686825547 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f12480a9ac0 0x7f1248117a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:54.985 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.972+0000 7f123f7fe700 1 -- 172.21.15.42:0/1686825547 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3549084898 0 0) 0x7f12480a0700 con 0x7f1248117f70 2019-12-05T06:07:54.985 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.972+0000 7f123f7fe700 1 -- 172.21.15.42:0/1686825547 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12300030b0 con 0x7f1248117f70 2019-12-05T06:07:54.985 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.972+0000 7f124ea3b700 1 -- 172.21.15.42:0/1686825547 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f12480a9ac0 msgr2=0x7f1248117a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:54.985 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.972+0000 7f124ea3b700 1 --2- 172.21.15.42:0/1686825547 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f12480a9ac0 0x7f1248117a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:54.985 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.972+0000 7f124ea3b700 1 -- 172.21.15.42:0/1686825547 >> v1:172.21.15.42:6789/0 conn(0x7f1248117f70 legacy=0x7f124807ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:54.985 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.972+0000 7f124ea3b700 1 -- 172.21.15.42:0/1686825547 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f124811bb40 con 0x7f124811a170 2019-12-05T06:07:54.985 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.972+0000 7f124ea3b700 1 --2- 172.21.15.42:0/1686825547 >> v2:172.21.15.40:3300/0 conn(0x7f124811a170 0x7f12480a4ea0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f124811a840 tx=0x7f1244010a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:54.986 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.972+0000 7f124ea3b700 1 --2- 172.21.15.42:0/1686825547 >> v2:172.21.15.40:3300/0 conn(0x7f124811a170 0x7f12480a4ea0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f124811a840 tx=0x7f1244010a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:54.986 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.972+0000 7f124ea3b700 1 --2- 172.21.15.42:0/1686825547 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f124811a170 0x7f12480a4ea0 secure :-1 s=READY pgs=1583 cs=0 l=1 rx=0x7f124811a840 tx=0x7f1244010a90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:07:54.986 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.972+0000 7f124ea3b700 1 --2- 172.21.15.42:0/1686825547 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f124811a170 0x7f12480a4ea0 secure :-1 s=THROTTLE_DONE pgs=1583 cs=0 l=1 rx=0x7f124811a840 tx=0x7f1244010a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:54.986 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.972+0000 7f123f7fe700 1 -- 172.21.15.42:0/1686825547 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1244012960 con 0x7f124811a170 2019-12-05T06:07:54.986 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.972+0000 7f124ea3b700 1 --2- 172.21.15.42:0/1686825547 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f124811a170 0x7f12480a4ea0 secure :-1 s=THROTTLE_DONE pgs=1583 cs=0 l=1 rx=0x7f124811a840 tx=0x7f1244010a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:54.986 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.976+0000 7f123f7fe700 1 -- 172.21.15.42:0/1686825547 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1244012580 con 0x7f124811a170 2019-12-05T06:07:54.986 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.976+0000 7f124ea3b700 1 --2- 172.21.15.42:0/1686825547 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f124811a170 0x7f12480a4ea0 secure :-1 s=THROTTLE_DONE pgs=1583 cs=0 l=1 rx=0x7f124811a840 tx=0x7f1244010a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:54.987 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.976+0000 7f125049e700 1 -- 172.21.15.42:0/1686825547 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12481183a0 con 0x7f124811a170 2019-12-05T06:07:54.987 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.976+0000 7f123f7fe700 1 -- 172.21.15.42:0/1686825547 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1244005540 con 0x7f124811a170 2019-12-05T06:07:54.987 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.976+0000 7f125049e700 1 -- 172.21.15.42:0/1686825547 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f124807d4b0 con 0x7f124811a170 2019-12-05T06:07:54.987 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.976+0000 7f124ea3b700 1 --2- 172.21.15.42:0/1686825547 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f124811a170 0x7f12480a4ea0 secure :-1 s=THROTTLE_DONE pgs=1583 cs=0 l=1 rx=0x7f124811a840 tx=0x7f1244010a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:54.987 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.976+0000 7f123f7fe700 1 -- 172.21.15.42:0/1686825547 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f1244005860 con 0x7f124811a170 2019-12-05T06:07:54.987 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.976+0000 7f124ea3b700 1 --2- 172.21.15.42:0/1686825547 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f124811a170 0x7f12480a4ea0 secure :-1 s=THROTTLE_DONE pgs=1583 cs=0 l=1 rx=0x7f124811a840 tx=0x7f1244010a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:54.987 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.976+0000 7f123f7fe700 1 --2- 172.21.15.42:0/1686825547 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f123003b1f0 0x7f123003d700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:07:54.988 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.976+0000 7f123f7fe700 1 -- 172.21.15.42:0/1686825547 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9815+0+0 (secure 0 0 0) 0x7f124404af00 con 0x7f124811a170 2019-12-05T06:07:54.988 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.976+0000 7f124e23a700 1 --2- 172.21.15.42:0/1686825547 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f123003b1f0 0x7f123003d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:54.988 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.976+0000 7f124e23a700 1 --2- 172.21.15.42:0/1686825547 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f123003b1f0 0x7f123003d700 crc :-1 s=READY pgs=880 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:07:54.988 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.980+0000 7f125049e700 1 -- 172.21.15.42:0/1686825547 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f124805d100 con 0x7f124811a170 2019-12-05T06:07:54.994 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.988+0000 7f124ea3b700 1 --2- 172.21.15.42:0/1686825547 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f124811a170 0x7f12480a4ea0 secure :-1 s=THROTTLE_DONE pgs=1583 cs=0 l=1 rx=0x7f124811a840 tx=0x7f1244010a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:54.995 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:54.988+0000 7f123f7fe700 1 -- 172.21.15.42:0/1686825547 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f1244031f00 con 0x7f124811a170 2019-12-05T06:07:55.164 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:55.160+0000 7f125049e700 1 -- 172.21.15.42:0/1686825547 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f124805b8b0 con 0x7f124811a170 2019-12-05T06:07:55.165 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:55.160+0000 7f124ea3b700 1 --2- 172.21.15.42:0/1686825547 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f124811a170 0x7f12480a4ea0 secure :-1 s=THROTTLE_DONE pgs=1583 cs=0 l=1 rx=0x7f124811a840 tx=0x7f1244010a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:55.165 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:55.160+0000 7f123f7fe700 1 -- 172.21.15.42:0/1686825547 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 253 v253) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f1244032160 con 0x7f124811a170 2019-12-05T06:07:55.167 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 253 2019-12-05T06:07:55.167 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:07:55.167 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":253,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6929,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/2636699468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2636699468},{"type":"v1","addr":"172.21.15.42:6835","nonce":2636699468}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":253}],"filesystems":[{"mdsmap":{"epoch":252,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:07:04.524680+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":93,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9573},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9573":{"gid":9573,"name":"b","rank":0,"incarnation":249,"state":"up:active","state_seq":25,"addr":"172.21.15.40:6833/2968227232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2968227232},{"type":"v1","addr":"172.21.15.40:6833","nonce":2968227232}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:07:55.170 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:55.164+0000 7f125049e700 1 -- 172.21.15.42:0/1686825547 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f123003b1f0 msgr2=0x7f123003d700 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:55.170 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:55.164+0000 7f125049e700 1 --2- 172.21.15.42:0/1686825547 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f123003b1f0 0x7f123003d700 crc :-1 s=READY pgs=880 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:55.170 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:55.164+0000 7f125049e700 1 -- 172.21.15.42:0/1686825547 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f124811a170 msgr2=0x7f12480a4ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:55.170 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:55.164+0000 7f125049e700 1 --2- 172.21.15.42:0/1686825547 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f124811a170 0x7f12480a4ea0 secure :-1 s=READY pgs=1583 cs=0 l=1 rx=0x7f124811a840 tx=0x7f1244010a90).stop 2019-12-05T06:07:55.170 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:55.164+0000 7f125049e700 1 -- 172.21.15.42:0/1686825547 shutdown_connections 2019-12-05T06:07:55.170 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:55.164+0000 7f125049e700 1 --2- 172.21.15.42:0/1686825547 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f123003b1f0 0x7f123003d700 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:55.170 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:55.164+0000 7f125049e700 1 --2- 172.21.15.42:0/1686825547 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f124811a170 0x7f12480a4ea0 unknown :-1 s=CLOSED pgs=1583 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:55.171 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:55.164+0000 7f125049e700 1 --2- 172.21.15.42:0/1686825547 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f12480a9ac0 0x7f1248117a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:55.171 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:55.164+0000 7f125049e700 1 -- 172.21.15.42:0/1686825547 shutdown_connections 2019-12-05T06:07:55.171 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:55.164+0000 7f125049e700 1 -- 172.21.15.42:0/1686825547 wait complete. 2019-12-05T06:07:55.171 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:55.164+0000 7f125049e700 1 -- 172.21.15.42:0/1686825547 >> 172.21.15.42:0/1686825547 conn(0x7f12480a7730 msgr2=0x7f124811bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:07:55.180 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:07:57.181 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:07:57.291 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:57.280+0000 7f9a2f761700 1 Processor -- start 2019-12-05T06:07:57.291 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:57.280+0000 7f9a2f761700 1 -- start start 2019-12-05T06:07:57.292 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:57.280+0000 7f9a2f761700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9a280a7740 0x7f9a2808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:07:57.292 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:57.280+0000 7f9a2f761700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f9a2807e720 0x7f9a2807eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:07:57.292 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:57.280+0000 7f9a2f761700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f9a2809f2e0 0x7f9a2809f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:07:57.292 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:57.280+0000 7f9a2f761700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a280a0b00 con 0x7f9a2809f2e0 2019-12-05T06:07:57.292 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:57.280+0000 7f9a2f761700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a2808e5b0 con 0x7f9a2807e720 2019-12-05T06:07:57.292 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:57.280+0000 7f9a2f761700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a2808e850 con 0x7f9a280a7740 2019-12-05T06:07:57.292 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:57.284+0000 7f9a2d4fd700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f9a2807e720 0x7f9a2807eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:57.293 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:57.284+0000 7f9a2e4ff700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f9a2809f2e0 0x7f9a2809f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:57.293 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:57.284+0000 7f9a2d4fd700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f9a2807e720 0x7f9a2807eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:48788/0 (socket says 172.21.15.42:48788) 2019-12-05T06:07:57.293 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:57.284+0000 7f9a2dcfe700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9a280a7740 0x7f9a2808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:57.293 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:57.284+0000 7f9a2d4fd700 1 -- 172.21.15.42:0/2958683127 learned_addr learned my addr 172.21.15.42:0/2958683127 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:07:57.293 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:57.284+0000 7f9a2d4fd700 1 -- 172.21.15.42:0/2958683127 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9a280a7740 msgr2=0x7f9a2808df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:07:57.293 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:57.284+0000 7f9a2d4fd700 1 --2- 172.21.15.42:0/2958683127 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9a280a7740 0x7f9a2808df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:07:57.293 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:57.284+0000 7f9a2d4fd700 1 -- 172.21.15.42:0/2958683127 >> v2:172.21.15.40:3300/0 conn(0x7f9a2809f2e0 msgr2=0x7f9a2809f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:07:57.294 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:57.284+0000 7f9a2d4fd700 1 --2- 172.21.15.42:0/2958683127 >> v2:172.21.15.40:3300/0 conn(0x7f9a2809f2e0 0x7f9a2809f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:07:57.294 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:57.284+0000 7f9a2d4fd700 1 -- 172.21.15.42:0/2958683127 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a2807f0a0 con 0x7f9a2807e720 2019-12-05T06:07:57.294 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:57.284+0000 7f9a2d4fd700 1 --2- 172.21.15.42:0/2958683127 >> v2:172.21.15.42:3300/0 conn(0x7f9a2807e720 0x7f9a2807eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f9a2808e770 tx=0x7f9a18010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:57.294 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:57.284+0000 7f9a2d4fd700 1 --2- 172.21.15.42:0/2958683127 >> v2:172.21.15.42:3300/0 conn(0x7f9a2807e720 0x7f9a2807eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f9a2808e770 tx=0x7f9a18010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:57.294 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:57.284+0000 7f9a2d4fd700 1 --2- 172.21.15.42:0/2958683127 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9a2807e720 0x7f9a2807eb50 secure :-1 s=READY pgs=2081 cs=0 l=1 rx=0x7f9a2808e770 tx=0x7f9a18010130).ready entity=mon.1 client_cookie=29000519dc9e74df server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:07:57.294 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:57.284+0000 7f9a2d4fd700 1 --2- 172.21.15.42:0/2958683127 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9a2807e720 0x7f9a2807eb50 secure :-1 s=THROTTLE_DONE pgs=2081 cs=0 l=1 rx=0x7f9a2808e770 tx=0x7f9a18010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:57.294 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:57.284+0000 7f9a2d4fd700 1 --2- 172.21.15.42:0/2958683127 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9a2807e720 0x7f9a2807eb50 secure :-1 s=THROTTLE_DONE pgs=2081 cs=0 l=1 rx=0x7f9a2808e770 tx=0x7f9a18010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:57.295 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:57.284+0000 7f9a2d4fd700 1 --2- 172.21.15.42:0/2958683127 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9a2807e720 0x7f9a2807eb50 secure :-1 s=THROTTLE_DONE pgs=2081 cs=0 l=1 rx=0x7f9a2808e770 tx=0x7f9a18010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:57.295 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:57.284+0000 7f9a2ccfc700 1 -- 172.21.15.42:0/2958683127 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9a18010de0 con 0x7f9a2807e720 2019-12-05T06:07:57.295 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:57.284+0000 7f9a2ccfc700 1 -- 172.21.15.42:0/2958683127 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a18010a00 con 0x7f9a2807e720 2019-12-05T06:07:57.295 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:57.284+0000 7f9a2ccfc700 1 -- 172.21.15.42:0/2958683127 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9a180114a0 con 0x7f9a2807e720 2019-12-05T06:07:57.295 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:57.284+0000 7f9a2f761700 1 -- 172.21.15.42:0/2958683127 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9a2807e720 msgr2=0x7f9a2807eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:57.295 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:57.284+0000 7f9a2f761700 1 --2- 172.21.15.42:0/2958683127 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9a2807e720 0x7f9a2807eb50 secure :-1 s=READY pgs=2081 cs=0 l=1 rx=0x7f9a2808e770 tx=0x7f9a18010130).stop 2019-12-05T06:07:57.295 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:57.284+0000 7f9a2f761700 1 -- 172.21.15.42:0/2958683127 shutdown_connections 2019-12-05T06:07:57.295 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:57.284+0000 7f9a2f761700 1 --2- 172.21.15.42:0/2958683127 >> v2:172.21.15.40:3300/0 conn(0x7f9a2809f2e0 0x7f9a2809f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:07:57.296 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:57.284+0000 7f9a2f761700 1 --2- 172.21.15.42:0/2958683127 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9a280a7740 0x7f9a2808df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:07:57.296 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:57.284+0000 7f9a2f761700 1 --2- 172.21.15.42:0/2958683127 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9a2807e720 0x7f9a2807eb50 unknown :-1 s=CLOSED pgs=2081 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:57.296 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:57.284+0000 7f9a2f761700 1 -- 172.21.15.42:0/2958683127 shutdown_connections 2019-12-05T06:07:57.296 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:57.284+0000 7f9a2f761700 1 -- 172.21.15.42:0/2958683127 wait complete. 2019-12-05T06:07:57.296 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:57.284+0000 7f9a2f761700 1 -- 172.21.15.42:0/2958683127 >> 172.21.15.42:0/2958683127 conn(0x7f9a280aabe0 msgr2=0x7f9a280a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:07:57.300 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:57.288+0000 7f9a2f761700 1 Processor -- start 2019-12-05T06:07:57.301 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:57.288+0000 7f9a2f761700 1 -- start start 2019-12-05T06:07:57.301 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:57.288+0000 7f9a2f761700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f9a2808f7d0 0x7f9a28118eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:07:57.301 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:57.288+0000 7f9a2f761700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f9a280722c0 0x7f9a28074700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:07:57.301 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:57.288+0000 7f9a2f761700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a280a4590 con 0x7f9a2811eed0 2019-12-05T06:07:57.301 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:57.288+0000 7f9a2f761700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a28144990 con 0x7f9a2808f7d0 2019-12-05T06:07:57.301 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:57.288+0000 7f9a2f761700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a28144c30 con 0x7f9a280722c0 2019-12-05T06:07:57.302 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:57.288+0000 7f9a2e4ff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f9a280722c0 0x7f9a28074700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:57.302 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:57.288+0000 7f9a2e4ff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f9a280722c0 0x7f9a28074700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:48796/0 (socket says 172.21.15.42:48796) 2019-12-05T06:07:57.302 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:57.288+0000 7f9a2e4ff700 1 -- 172.21.15.42:0/248600399 learned_addr learned my addr 172.21.15.42:0/248600399 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:07:57.302 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:57.288+0000 7f9a2dcfe700 1 --2- 172.21.15.42:0/248600399 >> v2:172.21.15.40:3300/0 conn(0x7f9a2808f7d0 0x7f9a28118eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:57.302 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:57.288+0000 7f9a2e4ff700 1 -- 172.21.15.42:0/248600399 >> v2:172.21.15.40:3300/0 conn(0x7f9a2808f7d0 msgr2=0x7f9a28118eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:57.302 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:57.288+0000 7f9a2e4ff700 1 --2- 172.21.15.42:0/248600399 >> v2:172.21.15.40:3300/0 conn(0x7f9a2808f7d0 0x7f9a28118eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:57.302 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:57.288+0000 7f9a2e4ff700 1 -- 172.21.15.42:0/248600399 >> v1:172.21.15.40:6789/0 conn(0x7f9a2811eed0 legacy=0x7f9a2811f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:57.303 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:57.288+0000 7f9a2e4ff700 1 -- 172.21.15.42:0/248600399 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a281219d0 con 0x7f9a280722c0 2019-12-05T06:07:57.303 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:57.288+0000 7f9a2e4ff700 1 --2- 172.21.15.42:0/248600399 >> v2:172.21.15.42:3300/0 conn(0x7f9a280722c0 0x7f9a28074700 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f9a20002f30 tx=0x7f9a20010980).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:57.303 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:57.288+0000 7f9a2e4ff700 1 --2- 172.21.15.42:0/248600399 >> v2:172.21.15.42:3300/0 conn(0x7f9a280722c0 0x7f9a28074700 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f9a20002f30 tx=0x7f9a20010980).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:57.303 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:57.288+0000 7f9a2e4ff700 1 --2- 172.21.15.42:0/248600399 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9a280722c0 0x7f9a28074700 secure :-1 s=READY pgs=2082 cs=0 l=1 rx=0x7f9a20002f30 tx=0x7f9a20010980).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:07:57.303 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:57.288+0000 7f9a2e4ff700 1 --2- 172.21.15.42:0/248600399 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9a280722c0 0x7f9a28074700 secure :-1 s=THROTTLE_DONE pgs=2082 cs=0 l=1 rx=0x7f9a20002f30 tx=0x7f9a20010980).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:57.303 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:57.288+0000 7f9a2e4ff700 1 --2- 172.21.15.42:0/248600399 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9a280722c0 0x7f9a28074700 secure :-1 s=THROTTLE_DONE pgs=2082 cs=0 l=1 rx=0x7f9a20002f30 tx=0x7f9a20010980).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:57.303 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:57.288+0000 7f9a1e7fc700 1 -- 172.21.15.42:0/248600399 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9a20005930 con 0x7f9a280722c0 2019-12-05T06:07:57.304 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:57.288+0000 7f9a2e4ff700 1 --2- 172.21.15.42:0/248600399 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9a280722c0 0x7f9a28074700 secure :-1 s=THROTTLE_DONE pgs=2082 cs=0 l=1 rx=0x7f9a20002f30 tx=0x7f9a20010980).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:57.304 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:57.288+0000 7f9a1e7fc700 1 -- 172.21.15.42:0/248600399 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a20007200 con 0x7f9a280722c0 2019-12-05T06:07:57.304 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:57.288+0000 7f9a2f761700 1 -- 172.21.15.42:0/248600399 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a28074c50 con 0x7f9a280722c0 2019-12-05T06:07:57.304 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:57.288+0000 7f9a2f761700 1 -- 172.21.15.42:0/248600399 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a280751d0 con 0x7f9a280722c0 2019-12-05T06:07:57.304 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:57.288+0000 7f9a1e7fc700 1 -- 172.21.15.42:0/248600399 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9a200060c0 con 0x7f9a280722c0 2019-12-05T06:07:57.305 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:57.292+0000 7f9a2e4ff700 1 --2- 172.21.15.42:0/248600399 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9a280722c0 0x7f9a28074700 secure :-1 s=THROTTLE_DONE pgs=2082 cs=0 l=1 rx=0x7f9a20002f30 tx=0x7f9a20010980).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:57.305 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:57.292+0000 7f9a1e7fc700 1 -- 172.21.15.42:0/248600399 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f9a200063d0 con 0x7f9a280722c0 2019-12-05T06:07:57.305 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:57.292+0000 7f9a2e4ff700 1 --2- 172.21.15.42:0/248600399 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9a280722c0 0x7f9a28074700 secure :-1 s=THROTTLE_DONE pgs=2082 cs=0 l=1 rx=0x7f9a20002f30 tx=0x7f9a20010980).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:57.305 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:57.292+0000 7f9a1e7fc700 1 --2- 172.21.15.42:0/248600399 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9a0803ad10 0x7f9a0803d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:07:57.305 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:57.292+0000 7f9a1e7fc700 1 -- 172.21.15.42:0/248600399 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9815+0+0 (secure 0 0 0) 0x7f9a2004acb0 con 0x7f9a280722c0 2019-12-05T06:07:57.305 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:57.292+0000 7f9a2dcfe700 1 --2- 172.21.15.42:0/248600399 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9a0803ad10 0x7f9a0803d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:57.305 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:57.292+0000 7f9a2dcfe700 1 --2- 172.21.15.42:0/248600399 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9a0803ad10 0x7f9a0803d220 crc :-1 s=READY pgs=881 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:07:57.306 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:57.296+0000 7f9a2f761700 1 -- 172.21.15.42:0/248600399 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a04003200 con 0x7f9a280722c0 2019-12-05T06:07:57.310 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:57.304+0000 7f9a2e4ff700 1 --2- 172.21.15.42:0/248600399 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9a280722c0 0x7f9a28074700 secure :-1 s=THROTTLE_DONE pgs=2082 cs=0 l=1 rx=0x7f9a20002f30 tx=0x7f9a20010980).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:57.310 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:57.304+0000 7f9a1e7fc700 1 -- 172.21.15.42:0/248600399 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f9a20031d30 con 0x7f9a280722c0 2019-12-05T06:07:57.485 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:57.480+0000 7f9a2f761700 1 -- 172.21.15.42:0/248600399 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9a04003630 con 0x7f9a280722c0 2019-12-05T06:07:57.486 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:57.480+0000 7f9a2e4ff700 1 --2- 172.21.15.42:0/248600399 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9a280722c0 0x7f9a28074700 secure :-1 s=THROTTLE_DONE pgs=2082 cs=0 l=1 rx=0x7f9a20002f30 tx=0x7f9a20010980).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:57.486 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:57.480+0000 7f9a1e7fc700 1 -- 172.21.15.42:0/248600399 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 253 v253) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f9a20031f90 con 0x7f9a280722c0 2019-12-05T06:07:57.487 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 253 2019-12-05T06:07:57.487 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:07:57.487 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":253,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6929,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/2636699468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2636699468},{"type":"v1","addr":"172.21.15.42:6835","nonce":2636699468}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":253}],"filesystems":[{"mdsmap":{"epoch":252,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:07:04.524680+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":93,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9573},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9573":{"gid":9573,"name":"b","rank":0,"incarnation":249,"state":"up:active","state_seq":25,"addr":"172.21.15.40:6833/2968227232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2968227232},{"type":"v1","addr":"172.21.15.40:6833","nonce":2968227232}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:07:57.491 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:57.484+0000 7f9a2f761700 1 -- 172.21.15.42:0/248600399 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9a0803ad10 msgr2=0x7f9a0803d220 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:57.491 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:57.484+0000 7f9a2f761700 1 --2- 172.21.15.42:0/248600399 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9a0803ad10 0x7f9a0803d220 crc :-1 s=READY pgs=881 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:57.491 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:57.484+0000 7f9a2f761700 1 -- 172.21.15.42:0/248600399 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9a280722c0 msgr2=0x7f9a28074700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:57.491 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:57.484+0000 7f9a2f761700 1 --2- 172.21.15.42:0/248600399 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9a280722c0 0x7f9a28074700 secure :-1 s=READY pgs=2082 cs=0 l=1 rx=0x7f9a20002f30 tx=0x7f9a20010980).stop 2019-12-05T06:07:57.491 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:57.484+0000 7f9a2f761700 1 -- 172.21.15.42:0/248600399 shutdown_connections 2019-12-05T06:07:57.492 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:57.484+0000 7f9a2f761700 1 --2- 172.21.15.42:0/248600399 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9a0803ad10 0x7f9a0803d220 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:57.492 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:57.484+0000 7f9a2f761700 1 --2- 172.21.15.42:0/248600399 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9a280722c0 0x7f9a28074700 unknown :-1 s=CLOSED pgs=2082 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:57.492 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:57.484+0000 7f9a2f761700 1 --2- 172.21.15.42:0/248600399 >> v2:172.21.15.40:3300/0 conn(0x7f9a2808f7d0 0x7f9a28118eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:57.492 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:57.484+0000 7f9a2f761700 1 -- 172.21.15.42:0/248600399 shutdown_connections 2019-12-05T06:07:57.492 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:57.484+0000 7f9a2f761700 1 -- 172.21.15.42:0/248600399 wait complete. 2019-12-05T06:07:57.492 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:57.484+0000 7f9a2f761700 1 -- 172.21.15.42:0/248600399 >> 172.21.15.42:0/248600399 conn(0x7f9a280aabe0 msgr2=0x7f9a280a9790 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:07:57.504 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:07:59.505 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:07:59.614 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:59.604+0000 7f25446ac700 1 Processor -- start 2019-12-05T06:07:59.614 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:59.604+0000 7f25446ac700 1 -- start start 2019-12-05T06:07:59.614 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:59.604+0000 7f25446ac700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f253c0a7740 0x7f253c08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:07:59.614 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:59.604+0000 7f25446ac700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f253c07e720 0x7f253c07eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:07:59.615 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:59.604+0000 7f25446ac700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f253c0a0710 con 0x7f253c09f2e0 2019-12-05T06:07:59.615 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:59.604+0000 7f25446ac700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f253c0a0b00 con 0x7f253c07e720 2019-12-05T06:07:59.615 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:59.604+0000 7f25446ac700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f253c08e5b0 con 0x7f253c0a7740 2019-12-05T06:07:59.615 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:59.604+0000 7f2542c49700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f253c0a7740 0x7f253c08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:59.616 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:59.604+0000 7f2542c49700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f253c0a7740 0x7f253c08df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:48804/0 (socket says 172.21.15.42:48804) 2019-12-05T06:07:59.616 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:59.604+0000 7f2542c49700 1 -- 172.21.15.42:0/2651729802 learned_addr learned my addr 172.21.15.42:0/2651729802 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:07:59.616 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:59.604+0000 7f2542448700 1 --2- 172.21.15.42:0/2651729802 >> v2:172.21.15.40:3300/0 conn(0x7f253c07e720 0x7f253c07eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:59.616 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:59.604+0000 7f2542c49700 1 -- 172.21.15.42:0/2651729802 >> v2:172.21.15.40:3300/0 conn(0x7f253c07e720 msgr2=0x7f253c07eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:07:59.616 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:59.604+0000 7f2542c49700 1 --2- 172.21.15.42:0/2651729802 >> v2:172.21.15.40:3300/0 conn(0x7f253c07e720 0x7f253c07eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:07:59.616 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:59.604+0000 7f2542c49700 1 -- 172.21.15.42:0/2651729802 >> v1:172.21.15.40:6789/0 conn(0x7f253c09f2e0 legacy=0x7f253c09f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:59.617 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:59.604+0000 7f2542c49700 1 -- 172.21.15.42:0/2651729802 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f253c07f0a0 con 0x7f253c0a7740 2019-12-05T06:07:59.617 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:59.604+0000 7f2542c49700 1 --2- 172.21.15.42:0/2651729802 >> v2:172.21.15.42:3300/0 conn(0x7f253c0a7740 0x7f253c08df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f2538002f30 tx=0x7f253800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:59.617 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:59.604+0000 7f2542c49700 1 --2- 172.21.15.42:0/2651729802 >> v2:172.21.15.42:3300/0 conn(0x7f253c0a7740 0x7f253c08df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f2538002f30 tx=0x7f253800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:59.617 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:59.604+0000 7f2542c49700 1 --2- 172.21.15.42:0/2651729802 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f253c0a7740 0x7f253c08df80 secure :-1 s=READY pgs=2083 cs=0 l=1 rx=0x7f2538002f30 tx=0x7f253800fd30).ready entity=mon.1 client_cookie=6c1849b4acbe62cc server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:07:59.618 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:59.608+0000 7f2542c49700 1 --2- 172.21.15.42:0/2651729802 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f253c0a7740 0x7f253c08df80 secure :-1 s=THROTTLE_DONE pgs=2083 cs=0 l=1 rx=0x7f2538002f30 tx=0x7f253800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:59.618 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:59.608+0000 7f2542c49700 1 --2- 172.21.15.42:0/2651729802 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f253c0a7740 0x7f253c08df80 secure :-1 s=THROTTLE_DONE pgs=2083 cs=0 l=1 rx=0x7f2538002f30 tx=0x7f253800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:59.618 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:59.608+0000 7f2541c47700 1 -- 172.21.15.42:0/2651729802 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2538011370 con 0x7f253c0a7740 2019-12-05T06:07:59.618 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:59.608+0000 7f2542c49700 1 --2- 172.21.15.42:0/2651729802 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f253c0a7740 0x7f253c08df80 secure :-1 s=THROTTLE_DONE pgs=2083 cs=0 l=1 rx=0x7f2538002f30 tx=0x7f253800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:59.618 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:59.608+0000 7f2541c47700 1 -- 172.21.15.42:0/2651729802 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2538008bb0 con 0x7f253c0a7740 2019-12-05T06:07:59.618 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:59.608+0000 7f2541c47700 1 -- 172.21.15.42:0/2651729802 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2538011950 con 0x7f253c0a7740 2019-12-05T06:07:59.618 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:59.608+0000 7f25446ac700 1 -- 172.21.15.42:0/2651729802 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f253c0a7740 msgr2=0x7f253c08df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:59.619 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:59.608+0000 7f25446ac700 1 --2- 172.21.15.42:0/2651729802 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f253c0a7740 0x7f253c08df80 secure :-1 s=READY pgs=2083 cs=0 l=1 rx=0x7f2538002f30 tx=0x7f253800fd30).stop 2019-12-05T06:07:59.619 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:59.608+0000 7f25446ac700 1 -- 172.21.15.42:0/2651729802 shutdown_connections 2019-12-05T06:07:59.619 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:59.608+0000 7f25446ac700 1 --2- 172.21.15.42:0/2651729802 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f253c0a7740 0x7f253c08df80 unknown :-1 s=CLOSED pgs=2083 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:59.619 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:59.608+0000 7f25446ac700 1 --2- 172.21.15.42:0/2651729802 >> v2:172.21.15.40:3300/0 conn(0x7f253c07e720 0x7f253c07eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:07:59.619 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:59.608+0000 7f25446ac700 1 -- 172.21.15.42:0/2651729802 shutdown_connections 2019-12-05T06:07:59.619 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:59.608+0000 7f25446ac700 1 -- 172.21.15.42:0/2651729802 wait complete. 2019-12-05T06:07:59.619 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:59.608+0000 7f25446ac700 1 -- 172.21.15.42:0/2651729802 >> 172.21.15.42:0/2651729802 conn(0x7f253c0aabe0 msgr2=0x7f253c0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:07:59.623 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:59.612+0000 7f25446ac700 1 Processor -- start 2019-12-05T06:07:59.623 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:59.612+0000 7f25446ac700 1 -- start start 2019-12-05T06:07:59.623 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:59.612+0000 7f25446ac700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f253c116ed0 0x7f253c118320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:07:59.623 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:59.612+0000 7f25446ac700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f253c118870 0x7f253c138a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:07:59.623 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:59.612+0000 7f25446ac700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f253c121000 con 0x7f253c123690 2019-12-05T06:07:59.623 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:59.612+0000 7f25446ac700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f253c119a20 con 0x7f253c118870 2019-12-05T06:07:59.623 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:59.612+0000 7f25446ac700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f253c119cc0 con 0x7f253c116ed0 2019-12-05T06:07:59.624 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:59.612+0000 7f254344a700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f253c118870 0x7f253c138a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:59.624 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:59.612+0000 7f254344a700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f253c118870 0x7f253c138a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:48814/0 (socket says 172.21.15.42:48814) 2019-12-05T06:07:59.624 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:59.612+0000 7f254344a700 1 -- 172.21.15.42:0/4087028651 learned_addr learned my addr 172.21.15.42:0/4087028651 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:07:59.624 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:59.612+0000 7f2542448700 1 --2- 172.21.15.42:0/4087028651 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f253c116ed0 0x7f253c118320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:59.624 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:59.612+0000 7f254344a700 1 -- 172.21.15.42:0/4087028651 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f253c116ed0 msgr2=0x7f253c118320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:59.624 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:59.612+0000 7f254344a700 1 --2- 172.21.15.42:0/4087028651 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f253c116ed0 0x7f253c118320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:59.624 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:59.612+0000 7f254344a700 1 -- 172.21.15.42:0/4087028651 >> v1:172.21.15.42:6789/0 conn(0x7f253c123690 legacy=0x7f253c131e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:59.625 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:59.612+0000 7f254344a700 1 -- 172.21.15.42:0/4087028651 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f253c118ca0 con 0x7f253c118870 2019-12-05T06:07:59.625 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:59.612+0000 7f254344a700 1 --2- 172.21.15.42:0/4087028651 >> v2:172.21.15.42:3300/0 conn(0x7f253c118870 0x7f253c138a70 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f253c119be0 tx=0x7f2534010b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:59.625 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:59.612+0000 7f254344a700 1 --2- 172.21.15.42:0/4087028651 >> v2:172.21.15.42:3300/0 conn(0x7f253c118870 0x7f253c138a70 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f253c119be0 tx=0x7f2534010b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:59.625 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:59.612+0000 7f254344a700 1 --2- 172.21.15.42:0/4087028651 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f253c118870 0x7f253c138a70 secure :-1 s=READY pgs=2085 cs=0 l=1 rx=0x7f253c119be0 tx=0x7f2534010b70).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:07:59.625 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:59.612+0000 7f254344a700 1 --2- 172.21.15.42:0/4087028651 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f253c118870 0x7f253c138a70 secure :-1 s=THROTTLE_DONE pgs=2085 cs=0 l=1 rx=0x7f253c119be0 tx=0x7f2534010b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:59.625 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:59.612+0000 7f25337fe700 1 -- 172.21.15.42:0/4087028651 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2534006dc0 con 0x7f253c118870 2019-12-05T06:07:59.626 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:59.612+0000 7f254344a700 1 --2- 172.21.15.42:0/4087028651 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f253c118870 0x7f253c138a70 secure :-1 s=THROTTLE_DONE pgs=2085 cs=0 l=1 rx=0x7f253c119be0 tx=0x7f2534010b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:59.626 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:59.612+0000 7f254344a700 1 --2- 172.21.15.42:0/4087028651 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f253c118870 0x7f253c138a70 secure :-1 s=THROTTLE_DONE pgs=2085 cs=0 l=1 rx=0x7f253c119be0 tx=0x7f2534010b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:59.626 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:59.612+0000 7f25337fe700 1 -- 172.21.15.42:0/4087028651 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25340069e0 con 0x7f253c118870 2019-12-05T06:07:59.626 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:59.612+0000 7f25337fe700 1 -- 172.21.15.42:0/4087028651 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2534002620 con 0x7f253c118870 2019-12-05T06:07:59.626 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:59.612+0000 7f25446ac700 1 -- 172.21.15.42:0/4087028651 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f253c121900 con 0x7f253c118870 2019-12-05T06:07:59.627 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:59.612+0000 7f25446ac700 1 -- 172.21.15.42:0/4087028651 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f253c138fc0 con 0x7f253c118870 2019-12-05T06:07:59.627 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:59.616+0000 7f254344a700 1 --2- 172.21.15.42:0/4087028651 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f253c118870 0x7f253c138a70 secure :-1 s=THROTTLE_DONE pgs=2085 cs=0 l=1 rx=0x7f253c119be0 tx=0x7f2534010b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:59.627 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:59.616+0000 7f254344a700 1 --2- 172.21.15.42:0/4087028651 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f253c118870 0x7f253c138a70 secure :-1 s=THROTTLE_DONE pgs=2085 cs=0 l=1 rx=0x7f253c119be0 tx=0x7f2534010b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:59.627 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:59.616+0000 7f25337fe700 1 -- 172.21.15.42:0/4087028651 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f2534002940 con 0x7f253c118870 2019-12-05T06:07:59.627 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:59.616+0000 7f25337fe700 1 --2- 172.21.15.42:0/4087028651 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f252403ade0 0x7f252403d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:07:59.627 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:59.616+0000 7f25337fe700 1 -- 172.21.15.42:0/4087028651 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9815+0+0 (secure 0 0 0) 0x7f253404b6e0 con 0x7f253c118870 2019-12-05T06:07:59.628 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:59.616+0000 7f2542c49700 1 --2- 172.21.15.42:0/4087028651 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f252403ade0 0x7f252403d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:59.628 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:59.616+0000 7f2542c49700 1 --2- 172.21.15.42:0/4087028651 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f252403ade0 0x7f252403d2f0 crc :-1 s=READY pgs=882 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:07:59.629 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:59.620+0000 7f25446ac700 1 -- 172.21.15.42:0/4087028651 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f253c05d110 con 0x7f253c118870 2019-12-05T06:07:59.630 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:59.624+0000 7f254344a700 1 --2- 172.21.15.42:0/4087028651 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f253c118870 0x7f253c138a70 secure :-1 s=THROTTLE_DONE pgs=2085 cs=0 l=1 rx=0x7f253c119be0 tx=0x7f2534010b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:59.631 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:59.624+0000 7f25337fe700 1 -- 172.21.15.42:0/4087028651 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f253c118ca0 con 0x7f253c118870 2019-12-05T06:07:59.802 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:59.792+0000 7f25446ac700 1 -- 172.21.15.42:0/4087028651 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f253c117500 con 0x7f253c118870 2019-12-05T06:07:59.802 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:59.796+0000 7f254344a700 1 --2- 172.21.15.42:0/4087028651 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f253c118870 0x7f253c138a70 secure :-1 s=THROTTLE_DONE pgs=2085 cs=0 l=1 rx=0x7f253c119be0 tx=0x7f2534010b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:59.802 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:59.796+0000 7f25337fe700 1 -- 172.21.15.42:0/4087028651 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 253 v253) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f2534013060 con 0x7f253c118870 2019-12-05T06:07:59.803 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:07:59.803 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":253,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6929,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/2636699468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2636699468},{"type":"v1","addr":"172.21.15.42:6835","nonce":2636699468}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":253}],"filesystems":[{"mdsmap":{"epoch":252,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:07:04.524680+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":93,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9573},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9573":{"gid":9573,"name":"b","rank":0,"incarnation":249,"state":"up:active","state_seq":25,"addr":"172.21.15.40:6833/2968227232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2968227232},{"type":"v1","addr":"172.21.15.40:6833","nonce":2968227232}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:07:59.803 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 253 2019-12-05T06:07:59.810 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:59.800+0000 7f25446ac700 1 -- 172.21.15.42:0/4087028651 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f252403ade0 msgr2=0x7f252403d2f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:59.810 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:59.800+0000 7f25446ac700 1 --2- 172.21.15.42:0/4087028651 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f252403ade0 0x7f252403d2f0 crc :-1 s=READY pgs=882 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:59.810 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:59.800+0000 7f25446ac700 1 -- 172.21.15.42:0/4087028651 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f253c118870 msgr2=0x7f253c138a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:59.810 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:59.800+0000 7f25446ac700 1 --2- 172.21.15.42:0/4087028651 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f253c118870 0x7f253c138a70 secure :-1 s=READY pgs=2085 cs=0 l=1 rx=0x7f253c119be0 tx=0x7f2534010b70).stop 2019-12-05T06:07:59.810 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:59.800+0000 7f25446ac700 1 -- 172.21.15.42:0/4087028651 shutdown_connections 2019-12-05T06:07:59.811 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:59.800+0000 7f25446ac700 1 --2- 172.21.15.42:0/4087028651 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f252403ade0 0x7f252403d2f0 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:59.811 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:59.800+0000 7f25446ac700 1 --2- 172.21.15.42:0/4087028651 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f253c118870 0x7f253c138a70 unknown :-1 s=CLOSED pgs=2085 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:59.812 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:59.800+0000 7f25446ac700 1 --2- 172.21.15.42:0/4087028651 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f253c116ed0 0x7f253c118320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:59.813 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:59.800+0000 7f25446ac700 1 -- 172.21.15.42:0/4087028651 shutdown_connections 2019-12-05T06:07:59.813 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:59.800+0000 7f25446ac700 1 -- 172.21.15.42:0/4087028651 wait complete. 2019-12-05T06:07:59.813 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:07:59.800+0000 7f25446ac700 1 -- 172.21.15.42:0/4087028651 >> 172.21.15.42:0/4087028651 conn(0x7f253c0a97e0 msgr2=0x7f253c09f580 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:07:59.815 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:08:01.816 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:08:01.923 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:01.912+0000 7f2d89ccb700 1 Processor -- start 2019-12-05T06:08:01.923 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:01.912+0000 7f2d89ccb700 1 -- start start 2019-12-05T06:08:01.923 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:01.912+0000 7f2d89ccb700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2d840a7740 0x7f2d8408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:08:01.923 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:01.912+0000 7f2d89ccb700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2d8407e720 0x7f2d8407eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:08:01.924 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:01.912+0000 7f2d89ccb700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2d8409f2e0 0x7f2d8409f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:08:01.924 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:01.912+0000 7f2d89ccb700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d840a0b00 con 0x7f2d840a7740 2019-12-05T06:08:01.924 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:01.912+0000 7f2d89ccb700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d8408e5b0 con 0x7f2d8407e720 2019-12-05T06:08:01.924 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:01.912+0000 7f2d89ccb700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d8408e850 con 0x7f2d8409f2e0 2019-12-05T06:08:01.925 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:01.912+0000 7f2d837fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2d8407e720 0x7f2d8407eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:01.930 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:01.912+0000 7f2d837fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2d8407e720 0x7f2d8407eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:48824/0 (socket says 172.21.15.42:48824) 2019-12-05T06:08:01.930 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:01.912+0000 7f2d837fe700 1 -- 172.21.15.42:0/579299746 learned_addr learned my addr 172.21.15.42:0/579299746 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:08:01.931 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:01.912+0000 7f2d83fff700 1 --2- 172.21.15.42:0/579299746 >> v2:172.21.15.40:3300/0 conn(0x7f2d840a7740 0x7f2d8408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:01.931 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:01.912+0000 7f2d88a69700 1 --2- 172.21.15.42:0/579299746 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2d8409f2e0 0x7f2d8409f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:01.931 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:01.912+0000 7f2d837fe700 1 -- 172.21.15.42:0/579299746 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2d8409f2e0 msgr2=0x7f2d8409f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:08:01.931 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:01.912+0000 7f2d837fe700 1 --2- 172.21.15.42:0/579299746 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2d8409f2e0 0x7f2d8409f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:08:01.931 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:01.912+0000 7f2d837fe700 1 -- 172.21.15.42:0/579299746 >> v2:172.21.15.40:3300/0 conn(0x7f2d840a7740 msgr2=0x7f2d8408df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:08:01.931 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:01.912+0000 7f2d837fe700 1 --2- 172.21.15.42:0/579299746 >> v2:172.21.15.40:3300/0 conn(0x7f2d840a7740 0x7f2d8408df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:08:01.931 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:01.912+0000 7f2d837fe700 1 -- 172.21.15.42:0/579299746 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d8407f0a0 con 0x7f2d8407e720 2019-12-05T06:08:01.932 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:01.912+0000 7f2d83fff700 1 --2- 172.21.15.42:0/579299746 >> v2:172.21.15.40:3300/0 conn(0x7f2d840a7740 0x7f2d8408df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:08:01.932 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:01.912+0000 7f2d837fe700 1 --2- 172.21.15.42:0/579299746 >> v2:172.21.15.42:3300/0 conn(0x7f2d8407e720 0x7f2d8407eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f2d840a0f50 tx=0x7f2d6c00fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:01.932 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:01.912+0000 7f2d837fe700 1 --2- 172.21.15.42:0/579299746 >> v2:172.21.15.42:3300/0 conn(0x7f2d8407e720 0x7f2d8407eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f2d840a0f50 tx=0x7f2d6c00fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:01.932 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:01.912+0000 7f2d837fe700 1 --2- 172.21.15.42:0/579299746 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2d8407e720 0x7f2d8407eb50 secure :-1 s=READY pgs=2086 cs=0 l=1 rx=0x7f2d840a0f50 tx=0x7f2d6c00fdb0).ready entity=mon.1 client_cookie=2c338a7626c118bf server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:08:01.932 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:01.912+0000 7f2d837fe700 1 --2- 172.21.15.42:0/579299746 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2d8407e720 0x7f2d8407eb50 secure :-1 s=THROTTLE_DONE pgs=2086 cs=0 l=1 rx=0x7f2d840a0f50 tx=0x7f2d6c00fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:01.932 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:01.912+0000 7f2d82ffd700 1 -- 172.21.15.42:0/579299746 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2d6c008c90 con 0x7f2d8407e720 2019-12-05T06:08:01.932 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:01.912+0000 7f2d837fe700 1 --2- 172.21.15.42:0/579299746 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2d8407e720 0x7f2d8407eb50 secure :-1 s=THROTTLE_DONE pgs=2086 cs=0 l=1 rx=0x7f2d840a0f50 tx=0x7f2d6c00fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:01.933 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:01.912+0000 7f2d837fe700 1 --2- 172.21.15.42:0/579299746 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2d8407e720 0x7f2d8407eb50 secure :-1 s=THROTTLE_DONE pgs=2086 cs=0 l=1 rx=0x7f2d840a0f50 tx=0x7f2d6c00fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:01.933 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:01.912+0000 7f2d82ffd700 1 -- 172.21.15.42:0/579299746 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d6c0088b0 con 0x7f2d8407e720 2019-12-05T06:08:01.934 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:01.912+0000 7f2d82ffd700 1 -- 172.21.15.42:0/579299746 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2d6c011b50 con 0x7f2d8407e720 2019-12-05T06:08:01.934 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:01.912+0000 7f2d89ccb700 1 -- 172.21.15.42:0/579299746 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2d8407e720 msgr2=0x7f2d8407eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:01.934 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:01.912+0000 7f2d89ccb700 1 --2- 172.21.15.42:0/579299746 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2d8407e720 0x7f2d8407eb50 secure :-1 s=READY pgs=2086 cs=0 l=1 rx=0x7f2d840a0f50 tx=0x7f2d6c00fdb0).stop 2019-12-05T06:08:01.934 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:01.916+0000 7f2d89ccb700 1 -- 172.21.15.42:0/579299746 shutdown_connections 2019-12-05T06:08:01.934 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:01.916+0000 7f2d89ccb700 1 --2- 172.21.15.42:0/579299746 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2d8409f2e0 0x7f2d8409f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:08:01.935 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:01.916+0000 7f2d89ccb700 1 --2- 172.21.15.42:0/579299746 >> v2:172.21.15.40:3300/0 conn(0x7f2d840a7740 0x7f2d8408df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:08:01.937 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:01.916+0000 7f2d89ccb700 1 --2- 172.21.15.42:0/579299746 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2d8407e720 0x7f2d8407eb50 unknown :-1 s=CLOSED pgs=2086 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:01.937 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:01.916+0000 7f2d89ccb700 1 -- 172.21.15.42:0/579299746 shutdown_connections 2019-12-05T06:08:01.937 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:01.916+0000 7f2d89ccb700 1 -- 172.21.15.42:0/579299746 wait complete. 2019-12-05T06:08:01.937 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:01.916+0000 7f2d89ccb700 1 -- 172.21.15.42:0/579299746 >> 172.21.15.42:0/579299746 conn(0x7f2d840aabe0 msgr2=0x7f2d840a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:08:01.937 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:01.916+0000 7f2d89ccb700 1 Processor -- start 2019-12-05T06:08:01.937 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:01.916+0000 7f2d89ccb700 1 -- start start 2019-12-05T06:08:01.946 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:01.920+0000 7f2d89ccb700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2d840a4f80 0x7f2d840653c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:08:01.947 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:01.920+0000 7f2d89ccb700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2d84065910 0x7f2d840762f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:08:01.947 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:01.920+0000 7f2d89ccb700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d8411a3d0 con 0x7f2d8412ee30 2019-12-05T06:08:01.947 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:01.920+0000 7f2d89ccb700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d8411ae80 con 0x7f2d840a4f80 2019-12-05T06:08:01.947 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:01.920+0000 7f2d89ccb700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d840a53b0 con 0x7f2d84065910 2019-12-05T06:08:01.947 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:01.920+0000 7f2d83fff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f2d8412ee30 0x7f2d840a6900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:48868/0 (socket says 172.21.15.42:48868) 2019-12-05T06:08:01.947 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:01.920+0000 7f2d83fff700 1 -- 172.21.15.42:0/3104841097 learned_addr learned my addr 172.21.15.42:0/3104841097 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:08:01.948 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:01.920+0000 7f2d88a69700 1 --2- 172.21.15.42:0/3104841097 >> v2:172.21.15.42:3300/0 conn(0x7f2d84065910 0x7f2d840762f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:01.948 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:01.920+0000 7f2d837fe700 1 --2- 172.21.15.42:0/3104841097 >> v2:172.21.15.40:3300/0 conn(0x7f2d840a4f80 0x7f2d840653c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:01.948 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:01.920+0000 7f2d80ff9700 1 -- 172.21.15.42:0/3104841097 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3943495275 0 0) 0x7f2d8411a3d0 con 0x7f2d8412ee30 2019-12-05T06:08:01.948 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:01.920+0000 7f2d80ff9700 1 -- 172.21.15.42:0/3104841097 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d680030b0 con 0x7f2d8412ee30 2019-12-05T06:08:01.948 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:01.920+0000 7f2d88a69700 1 -- 172.21.15.42:0/3104841097 >> v2:172.21.15.40:3300/0 conn(0x7f2d840a4f80 msgr2=0x7f2d840653c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:01.948 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:01.920+0000 7f2d88a69700 1 --2- 172.21.15.42:0/3104841097 >> v2:172.21.15.40:3300/0 conn(0x7f2d840a4f80 0x7f2d840653c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:01.948 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:01.920+0000 7f2d88a69700 1 -- 172.21.15.42:0/3104841097 >> v1:172.21.15.42:6789/0 conn(0x7f2d8412ee30 legacy=0x7f2d840a6900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:01.949 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:01.920+0000 7f2d88a69700 1 -- 172.21.15.42:0/3104841097 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d840a6520 con 0x7f2d84065910 2019-12-05T06:08:01.949 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:01.920+0000 7f2d88a69700 1 --2- 172.21.15.42:0/3104841097 >> v2:172.21.15.42:3300/0 conn(0x7f2d84065910 0x7f2d840762f0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f2d7c002f30 tx=0x7f2d7c010990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:01.949 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:01.920+0000 7f2d88a69700 1 --2- 172.21.15.42:0/3104841097 >> v2:172.21.15.42:3300/0 conn(0x7f2d84065910 0x7f2d840762f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f2d7c002f30 tx=0x7f2d7c010990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:01.949 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:01.920+0000 7f2d88a69700 1 --2- 172.21.15.42:0/3104841097 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2d84065910 0x7f2d840762f0 secure :-1 s=READY pgs=2088 cs=0 l=1 rx=0x7f2d7c002f30 tx=0x7f2d7c010990).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:08:01.949 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:01.920+0000 7f2d88a69700 1 --2- 172.21.15.42:0/3104841097 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2d84065910 0x7f2d840762f0 secure :-1 s=THROTTLE_DONE pgs=2088 cs=0 l=1 rx=0x7f2d7c002f30 tx=0x7f2d7c010990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:01.949 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:01.920+0000 7f2d80ff9700 1 -- 172.21.15.42:0/3104841097 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2d7c005930 con 0x7f2d84065910 2019-12-05T06:08:01.949 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:01.920+0000 7f2d88a69700 1 --2- 172.21.15.42:0/3104841097 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2d84065910 0x7f2d840762f0 secure :-1 s=THROTTLE_DONE pgs=2088 cs=0 l=1 rx=0x7f2d7c002f30 tx=0x7f2d7c010990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:01.950 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:01.920+0000 7f2d80ff9700 1 -- 172.21.15.42:0/3104841097 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d7c007160 con 0x7f2d84065910 2019-12-05T06:08:01.950 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:01.920+0000 7f2d89ccb700 1 -- 172.21.15.42:0/3104841097 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d84076840 con 0x7f2d84065910 2019-12-05T06:08:01.950 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:01.920+0000 7f2d88a69700 1 --2- 172.21.15.42:0/3104841097 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2d84065910 0x7f2d840762f0 secure :-1 s=THROTTLE_DONE pgs=2088 cs=0 l=1 rx=0x7f2d7c002f30 tx=0x7f2d7c010990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:01.950 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:01.920+0000 7f2d80ff9700 1 -- 172.21.15.42:0/3104841097 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2d7c0060c0 con 0x7f2d84065910 2019-12-05T06:08:01.950 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:01.920+0000 7f2d89ccb700 1 -- 172.21.15.42:0/3104841097 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d84076b70 con 0x7f2d84065910 2019-12-05T06:08:01.950 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:01.920+0000 7f2d88a69700 1 --2- 172.21.15.42:0/3104841097 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2d84065910 0x7f2d840762f0 secure :-1 s=THROTTLE_DONE pgs=2088 cs=0 l=1 rx=0x7f2d7c002f30 tx=0x7f2d7c010990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:01.950 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:01.920+0000 7f2d80ff9700 1 -- 172.21.15.42:0/3104841097 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f2d7c0063a0 con 0x7f2d84065910 2019-12-05T06:08:01.951 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:01.920+0000 7f2d88a69700 1 --2- 172.21.15.42:0/3104841097 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2d84065910 0x7f2d840762f0 secure :-1 s=THROTTLE_DONE pgs=2088 cs=0 l=1 rx=0x7f2d7c002f30 tx=0x7f2d7c010990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:01.951 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:01.920+0000 7f2d80ff9700 1 --2- 172.21.15.42:0/3104841097 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2d6803b230 0x7f2d6803d740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:08:01.951 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:01.920+0000 7f2d80ff9700 1 -- 172.21.15.42:0/3104841097 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9815+0+0 (secure 0 0 0) 0x7f2d7c04ac00 con 0x7f2d84065910 2019-12-05T06:08:01.951 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:01.924+0000 7f2d83fff700 1 --2- 172.21.15.42:0/3104841097 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2d6803b230 0x7f2d6803d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:01.951 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:01.924+0000 7f2d83fff700 1 --2- 172.21.15.42:0/3104841097 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2d6803b230 0x7f2d6803d740 crc :-1 s=READY pgs=883 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:08:01.951 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:01.928+0000 7f2d89ccb700 1 -- 172.21.15.42:0/3104841097 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d8405d110 con 0x7f2d84065910 2019-12-05T06:08:01.951 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:01.932+0000 7f2d88a69700 1 --2- 172.21.15.42:0/3104841097 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2d84065910 0x7f2d840762f0 secure :-1 s=THROTTLE_DONE pgs=2088 cs=0 l=1 rx=0x7f2d7c002f30 tx=0x7f2d7c010990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:01.952 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:01.932+0000 7f2d80ff9700 1 -- 172.21.15.42:0/3104841097 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f2d7c031ba0 con 0x7f2d84065910 2019-12-05T06:08:02.104 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:02.096+0000 7f2d89ccb700 1 -- 172.21.15.42:0/3104841097 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2d8405b8c0 con 0x7f2d84065910 2019-12-05T06:08:02.104 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:02.100+0000 7f2d88a69700 1 --2- 172.21.15.42:0/3104841097 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2d84065910 0x7f2d840762f0 secure :-1 s=THROTTLE_DONE pgs=2088 cs=0 l=1 rx=0x7f2d7c002f30 tx=0x7f2d7c010990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:02.105 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:02.100+0000 7f2d80ff9700 1 -- 172.21.15.42:0/3104841097 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 253 v253) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f2d7c031e00 con 0x7f2d84065910 2019-12-05T06:08:02.106 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 253 2019-12-05T06:08:02.106 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:08:02.106 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":253,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6929,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/2636699468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2636699468},{"type":"v1","addr":"172.21.15.42:6835","nonce":2636699468}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":253}],"filesystems":[{"mdsmap":{"epoch":252,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:07:04.524680+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":93,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9573},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9573":{"gid":9573,"name":"b","rank":0,"incarnation":249,"state":"up:active","state_seq":25,"addr":"172.21.15.40:6833/2968227232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2968227232},{"type":"v1","addr":"172.21.15.40:6833","nonce":2968227232}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:08:02.110 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:02.104+0000 7f2d89ccb700 1 -- 172.21.15.42:0/3104841097 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2d6803b230 msgr2=0x7f2d6803d740 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:02.110 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:02.104+0000 7f2d89ccb700 1 --2- 172.21.15.42:0/3104841097 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2d6803b230 0x7f2d6803d740 crc :-1 s=READY pgs=883 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:02.110 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:02.104+0000 7f2d89ccb700 1 -- 172.21.15.42:0/3104841097 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2d84065910 msgr2=0x7f2d840762f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:02.111 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:02.104+0000 7f2d89ccb700 1 --2- 172.21.15.42:0/3104841097 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2d84065910 0x7f2d840762f0 secure :-1 s=READY pgs=2088 cs=0 l=1 rx=0x7f2d7c002f30 tx=0x7f2d7c010990).stop 2019-12-05T06:08:02.111 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:02.104+0000 7f2d89ccb700 1 -- 172.21.15.42:0/3104841097 shutdown_connections 2019-12-05T06:08:02.111 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:02.104+0000 7f2d89ccb700 1 --2- 172.21.15.42:0/3104841097 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2d6803b230 0x7f2d6803d740 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:02.111 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:02.104+0000 7f2d89ccb700 1 --2- 172.21.15.42:0/3104841097 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2d84065910 0x7f2d840762f0 unknown :-1 s=CLOSED pgs=2088 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:02.111 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:02.104+0000 7f2d89ccb700 1 --2- 172.21.15.42:0/3104841097 >> v2:172.21.15.40:3300/0 conn(0x7f2d840a4f80 0x7f2d840653c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:02.111 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:02.104+0000 7f2d89ccb700 1 -- 172.21.15.42:0/3104841097 shutdown_connections 2019-12-05T06:08:02.111 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:02.104+0000 7f2d89ccb700 1 -- 172.21.15.42:0/3104841097 wait complete. 2019-12-05T06:08:02.112 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:02.104+0000 7f2d89ccb700 1 -- 172.21.15.42:0/3104841097 >> 172.21.15.42:0/3104841097 conn(0x7f2d840aabe0 msgr2=0x7f2d8409f6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:08:02.123 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:08:04.124 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:08:04.235 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:04.216+0000 7f61fe851700 1 Processor -- start 2019-12-05T06:08:04.235 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:04.216+0000 7f61fe851700 1 -- start start 2019-12-05T06:08:04.235 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:04.220+0000 7f61fe851700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f61f808df70 0x7f61f809f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:08:04.235 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:04.220+0000 7f61fe851700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f61f809f820 0x7f61f814efd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:08:04.236 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:04.220+0000 7f61fe851700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61f809fc50 con 0x7f61f80a7730 2019-12-05T06:08:04.236 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:04.220+0000 7f61fe851700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61f80a0af0 con 0x7f61f809f820 2019-12-05T06:08:04.236 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:04.220+0000 7f61fe851700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61f808e5a0 con 0x7f61f808df70 2019-12-05T06:08:04.236 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:04.220+0000 7f61fcdee700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f61f80a7730 0x7f61f807e710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:48876/0 (socket says 172.21.15.42:48876) 2019-12-05T06:08:04.240 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:04.220+0000 7f61fcdee700 1 -- 172.21.15.42:0/60971674 learned_addr learned my addr 172.21.15.42:0/60971674 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:08:04.240 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:04.220+0000 7f61f7fff700 1 --2- 172.21.15.42:0/60971674 >> v2:172.21.15.42:3300/0 conn(0x7f61f808df70 0x7f61f809f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:04.241 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:04.220+0000 7f61fd5ef700 1 --2- 172.21.15.42:0/60971674 >> v2:172.21.15.40:3300/0 conn(0x7f61f809f820 0x7f61f814efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:04.241 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:04.220+0000 7f61f7fff700 1 -- 172.21.15.42:0/60971674 >> v2:172.21.15.40:3300/0 conn(0x7f61f809f820 msgr2=0x7f61f814efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:08:04.241 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:04.220+0000 7f61f7fff700 1 --2- 172.21.15.42:0/60971674 >> v2:172.21.15.40:3300/0 conn(0x7f61f809f820 0x7f61f814efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:08:04.241 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:04.220+0000 7f61f7fff700 1 -- 172.21.15.42:0/60971674 >> v1:172.21.15.42:6789/0 conn(0x7f61f80a7730 legacy=0x7f61f807e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:04.241 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:04.220+0000 7f61f77fe700 1 -- 172.21.15.42:0/60971674 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1881892994 0 0) 0x7f61f809fc50 con 0x7f61f80a7730 2019-12-05T06:08:04.241 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:04.220+0000 7f61f7fff700 1 -- 172.21.15.42:0/60971674 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61f80a0700 con 0x7f61f808df70 2019-12-05T06:08:04.241 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:04.220+0000 7f61f7fff700 1 --2- 172.21.15.42:0/60971674 >> v2:172.21.15.42:3300/0 conn(0x7f61f808df70 0x7f61f809f2d0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f61ec002f30 tx=0x7f61ec00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:04.242 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:04.220+0000 7f61f7fff700 1 --2- 172.21.15.42:0/60971674 >> v2:172.21.15.42:3300/0 conn(0x7f61f808df70 0x7f61f809f2d0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f61ec002f30 tx=0x7f61ec00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:04.242 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:04.220+0000 7f61f7fff700 1 --2- 172.21.15.42:0/60971674 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f61f808df70 0x7f61f809f2d0 secure :-1 s=READY pgs=2090 cs=0 l=1 rx=0x7f61ec002f30 tx=0x7f61ec00fc70).ready entity=mon.1 client_cookie=2bc76575308ee306 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:08:04.242 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:04.220+0000 7f61f7fff700 1 --2- 172.21.15.42:0/60971674 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f61f808df70 0x7f61f809f2d0 secure :-1 s=THROTTLE_DONE pgs=2090 cs=0 l=1 rx=0x7f61ec002f30 tx=0x7f61ec00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:04.242 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:04.220+0000 7f61f77fe700 1 -- 172.21.15.42:0/60971674 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f61ec008bb0 con 0x7f61f808df70 2019-12-05T06:08:04.242 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:04.220+0000 7f61f7fff700 1 --2- 172.21.15.42:0/60971674 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f61f808df70 0x7f61f809f2d0 secure :-1 s=THROTTLE_DONE pgs=2090 cs=0 l=1 rx=0x7f61ec002f30 tx=0x7f61ec00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:04.242 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:04.220+0000 7f61f7fff700 1 --2- 172.21.15.42:0/60971674 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f61f808df70 0x7f61f809f2d0 secure :-1 s=THROTTLE_DONE pgs=2090 cs=0 l=1 rx=0x7f61ec002f30 tx=0x7f61ec00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:04.243 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:04.220+0000 7f61f77fe700 1 -- 172.21.15.42:0/60971674 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61ec0087d0 con 0x7f61f808df70 2019-12-05T06:08:04.243 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:04.220+0000 7f61f77fe700 1 -- 172.21.15.42:0/60971674 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f61ec011820 con 0x7f61f808df70 2019-12-05T06:08:04.243 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:04.220+0000 7f61fe851700 1 -- 172.21.15.42:0/60971674 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f61f808df70 msgr2=0x7f61f809f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:04.243 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:04.220+0000 7f61fe851700 1 --2- 172.21.15.42:0/60971674 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f61f808df70 0x7f61f809f2d0 secure :-1 s=READY pgs=2090 cs=0 l=1 rx=0x7f61ec002f30 tx=0x7f61ec00fc70).stop 2019-12-05T06:08:04.243 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:04.220+0000 7f61fe851700 1 -- 172.21.15.42:0/60971674 shutdown_connections 2019-12-05T06:08:04.243 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:04.224+0000 7f61fe851700 1 --2- 172.21.15.42:0/60971674 >> v2:172.21.15.40:3300/0 conn(0x7f61f809f820 0x7f61f814efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:08:04.243 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:04.224+0000 7f61fe851700 1 --2- 172.21.15.42:0/60971674 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f61f808df70 0x7f61f809f2d0 unknown :-1 s=CLOSED pgs=2090 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:04.244 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:04.224+0000 7f61fe851700 1 -- 172.21.15.42:0/60971674 shutdown_connections 2019-12-05T06:08:04.244 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:04.224+0000 7f61fe851700 1 -- 172.21.15.42:0/60971674 wait complete. 2019-12-05T06:08:04.244 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:04.224+0000 7f61fe851700 1 -- 172.21.15.42:0/60971674 >> 172.21.15.42:0/60971674 conn(0x7f61f80aabd0 msgr2=0x7f61f80a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:08:04.244 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:04.224+0000 7f61fe851700 1 Processor -- start 2019-12-05T06:08:04.244 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:04.224+0000 7f61fe851700 1 -- start start 2019-12-05T06:08:04.244 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:04.224+0000 7f61fe851700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f61f80a1500 0x7f61f80a1930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:08:04.244 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:04.224+0000 7f61fe851700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61f807d4c0 con 0x7f61f80a1e80 2019-12-05T06:08:04.245 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:04.224+0000 7f61fe851700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61f8119ee0 con 0x7f61f808f670 2019-12-05T06:08:04.245 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:04.224+0000 7f61fe851700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61f80a4220 con 0x7f61f80a1500 2019-12-05T06:08:04.245 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:04.224+0000 7f61f7fff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f61f80a1500 0x7f61f80a1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:04.245 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:04.224+0000 7f61f7fff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f61f80a1500 0x7f61f80a1930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:48844/0 (socket says 172.21.15.42:48844) 2019-12-05T06:08:04.245 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:04.224+0000 7f61f7fff700 1 -- 172.21.15.42:0/3434929991 learned_addr learned my addr 172.21.15.42:0/3434929991 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:08:04.245 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:04.228+0000 7f61f7fff700 1 -- 172.21.15.42:0/3434929991 >> v1:172.21.15.42:6789/0 conn(0x7f61f808f670 legacy=0x7f61f8118770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:04.245 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:04.228+0000 7f61f7fff700 1 -- 172.21.15.42:0/3434929991 >> v1:172.21.15.40:6789/0 conn(0x7f61f80a1e80 legacy=0x7f61f8126cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:04.246 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:04.228+0000 7f61f57fa700 1 -- 172.21.15.42:0/3434929991 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3310532995 0 0) 0x7f61f807d4c0 con 0x7f61f80a1e80 2019-12-05T06:08:04.246 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:04.228+0000 7f61f7fff700 1 -- 172.21.15.42:0/3434929991 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61f8118f70 con 0x7f61f80a1500 2019-12-05T06:08:04.246 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:04.228+0000 7f61f7fff700 1 --2- 172.21.15.42:0/3434929991 >> v2:172.21.15.42:3300/0 conn(0x7f61f80a1500 0x7f61f80a1930 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f61f811cff0 tx=0x7f61ec0089c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:04.246 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:04.228+0000 7f61f7fff700 1 --2- 172.21.15.42:0/3434929991 >> v2:172.21.15.42:3300/0 conn(0x7f61f80a1500 0x7f61f80a1930 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f61f811cff0 tx=0x7f61ec0089c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:04.246 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:04.228+0000 7f61f7fff700 1 --2- 172.21.15.42:0/3434929991 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f61f80a1500 0x7f61f80a1930 secure :-1 s=READY pgs=2092 cs=0 l=1 rx=0x7f61f811cff0 tx=0x7f61ec0089c0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:08:04.246 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:04.228+0000 7f61f7fff700 1 --2- 172.21.15.42:0/3434929991 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f61f80a1500 0x7f61f80a1930 secure :-1 s=THROTTLE_DONE pgs=2092 cs=0 l=1 rx=0x7f61f811cff0 tx=0x7f61ec0089c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:04.247 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:04.228+0000 7f61f57fa700 1 -- 172.21.15.42:0/3434929991 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f61ec00f490 con 0x7f61f80a1500 2019-12-05T06:08:04.247 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:04.228+0000 7f61f7fff700 1 --2- 172.21.15.42:0/3434929991 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f61f80a1500 0x7f61f80a1930 secure :-1 s=THROTTLE_DONE pgs=2092 cs=0 l=1 rx=0x7f61f811cff0 tx=0x7f61ec0089c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:04.247 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:04.228+0000 7f61f57fa700 1 -- 172.21.15.42:0/3434929991 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61ec008db0 con 0x7f61f80a1500 2019-12-05T06:08:04.247 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:04.228+0000 7f61f7fff700 1 --2- 172.21.15.42:0/3434929991 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f61f80a1500 0x7f61f80a1930 secure :-1 s=THROTTLE_DONE pgs=2092 cs=0 l=1 rx=0x7f61f811cff0 tx=0x7f61ec0089c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:04.247 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:04.228+0000 7f61fe851700 1 -- 172.21.15.42:0/3434929991 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61f80a4430 con 0x7f61f80a1500 2019-12-05T06:08:04.247 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:04.228+0000 7f61f57fa700 1 -- 172.21.15.42:0/3434929991 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f61ec005920 con 0x7f61f80a1500 2019-12-05T06:08:04.247 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:04.228+0000 7f61fe851700 1 -- 172.21.15.42:0/3434929991 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61f80a3d30 con 0x7f61f80a1500 2019-12-05T06:08:04.248 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:04.228+0000 7f61f7fff700 1 --2- 172.21.15.42:0/3434929991 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f61f80a1500 0x7f61f80a1930 secure :-1 s=THROTTLE_DONE pgs=2092 cs=0 l=1 rx=0x7f61f811cff0 tx=0x7f61ec0089c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:04.248 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:04.228+0000 7f61f57fa700 1 -- 172.21.15.42:0/3434929991 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f61ec0130b0 con 0x7f61f80a1500 2019-12-05T06:08:04.248 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:04.228+0000 7f61f7fff700 1 --2- 172.21.15.42:0/3434929991 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f61f80a1500 0x7f61f80a1930 secure :-1 s=THROTTLE_DONE pgs=2092 cs=0 l=1 rx=0x7f61f811cff0 tx=0x7f61ec0089c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:04.248 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:04.228+0000 7f61f57fa700 1 --2- 172.21.15.42:0/3434929991 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f61d803ac20 0x7f61d803d130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:08:04.248 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:04.228+0000 7f61f57fa700 1 -- 172.21.15.42:0/3434929991 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9815+0+0 (secure 0 0 0) 0x7f61ec04b960 con 0x7f61f80a1500 2019-12-05T06:08:04.264 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:04.228+0000 7f61fcdee700 1 --2- 172.21.15.42:0/3434929991 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f61d803ac20 0x7f61d803d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:04.265 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:04.232+0000 7f61fcdee700 1 --2- 172.21.15.42:0/3434929991 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f61d803ac20 0x7f61d803d130 crc :-1 s=READY pgs=884 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:08:04.266 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:04.232+0000 7f61fe851700 1 -- 172.21.15.42:0/3434929991 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61f805d100 con 0x7f61f80a1500 2019-12-05T06:08:04.266 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:04.240+0000 7f61f7fff700 1 --2- 172.21.15.42:0/3434929991 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f61f80a1500 0x7f61f80a1930 secure :-1 s=THROTTLE_DONE pgs=2092 cs=0 l=1 rx=0x7f61f811cff0 tx=0x7f61ec0089c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:04.266 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:04.240+0000 7f61f57fa700 1 -- 172.21.15.42:0/3434929991 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f61f80a4430 con 0x7f61f80a1500 2019-12-05T06:08:04.417 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:04.412+0000 7f61fe851700 1 -- 172.21.15.42:0/3434929991 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f61f805b8b0 con 0x7f61f80a1500 2019-12-05T06:08:04.418 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:04.412+0000 7f61f7fff700 1 --2- 172.21.15.42:0/3434929991 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f61f80a1500 0x7f61f80a1930 secure :-1 s=THROTTLE_DONE pgs=2092 cs=0 l=1 rx=0x7f61f811cff0 tx=0x7f61ec0089c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:04.418 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:04.412+0000 7f61f57fa700 1 -- 172.21.15.42:0/3434929991 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 253 v253) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f61ec0329c0 con 0x7f61f80a1500 2019-12-05T06:08:04.418 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 253 2019-12-05T06:08:04.418 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:08:04.419 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":253,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6929,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/2636699468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2636699468},{"type":"v1","addr":"172.21.15.42:6835","nonce":2636699468}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":253}],"filesystems":[{"mdsmap":{"epoch":252,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:07:04.524680+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":93,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9573},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9573":{"gid":9573,"name":"b","rank":0,"incarnation":249,"state":"up:active","state_seq":25,"addr":"172.21.15.40:6833/2968227232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2968227232},{"type":"v1","addr":"172.21.15.40:6833","nonce":2968227232}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:08:04.421 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:04.416+0000 7f61fe851700 1 -- 172.21.15.42:0/3434929991 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f61d803ac20 msgr2=0x7f61d803d130 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:04.421 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:04.416+0000 7f61fe851700 1 --2- 172.21.15.42:0/3434929991 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f61d803ac20 0x7f61d803d130 crc :-1 s=READY pgs=884 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:04.421 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:04.416+0000 7f61fe851700 1 -- 172.21.15.42:0/3434929991 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f61f80a1500 msgr2=0x7f61f80a1930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:04.421 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:04.416+0000 7f61fe851700 1 --2- 172.21.15.42:0/3434929991 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f61f80a1500 0x7f61f80a1930 secure :-1 s=READY pgs=2092 cs=0 l=1 rx=0x7f61f811cff0 tx=0x7f61ec0089c0).stop 2019-12-05T06:08:04.421 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:04.416+0000 7f61fe851700 1 -- 172.21.15.42:0/3434929991 shutdown_connections 2019-12-05T06:08:04.421 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:04.416+0000 7f61fe851700 1 --2- 172.21.15.42:0/3434929991 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f61d803ac20 0x7f61d803d130 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:04.422 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:04.416+0000 7f61fe851700 1 --2- 172.21.15.42:0/3434929991 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f61f80a1500 0x7f61f80a1930 unknown :-1 s=CLOSED pgs=2092 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:04.422 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:04.416+0000 7f61fe851700 1 -- 172.21.15.42:0/3434929991 shutdown_connections 2019-12-05T06:08:04.422 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:04.416+0000 7f61fe851700 1 -- 172.21.15.42:0/3434929991 wait complete. 2019-12-05T06:08:04.422 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:04.416+0000 7f61fe851700 1 -- 172.21.15.42:0/3434929991 >> 172.21.15.42:0/3434929991 conn(0x7f61f80aabd0 msgr2=0x7f61f807ea10 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:08:04.430 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:08:06.431 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:08:06.536 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:06.528+0000 7fcfebbc5700 1 Processor -- start 2019-12-05T06:08:06.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:06.528+0000 7fcfebbc5700 1 -- start start 2019-12-05T06:08:06.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:06.528+0000 7fcfebbc5700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fcfe409fad0 0x7fcfe414ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:08:06.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:06.528+0000 7fcfebbc5700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfe40a0700 con 0x7fcfe40a7730 2019-12-05T06:08:06.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:06.528+0000 7fcfebbc5700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfe407ef10 con 0x7fcfe408df70 2019-12-05T06:08:06.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:06.528+0000 7fcfebbc5700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcfe40a0af0 con 0x7fcfe409fad0 2019-12-05T06:08:06.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:06.528+0000 7fcfea963700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fcfe409fad0 0x7fcfe414ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:06.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:06.528+0000 7fcfea963700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fcfe409fad0 0x7fcfe414ffe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:48862/0 (socket says 172.21.15.42:48862) 2019-12-05T06:08:06.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:06.528+0000 7fcfea963700 1 -- 172.21.15.42:0/2913145377 learned_addr learned my addr 172.21.15.42:0/2913145377 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:08:06.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:06.528+0000 7fcfe9961700 1 --1- 172.21.15.42:0/2913145377 >> v1:172.21.15.42:6789/0 conn(0x7fcfe408df70 0x7fcfe409f2d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:48898/0 (socket says 172.21.15.42:48898) 2019-12-05T06:08:06.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:06.528+0000 7fcfea963700 1 -- 172.21.15.42:0/2913145377 >> v1:172.21.15.42:6789/0 conn(0x7fcfe408df70 legacy=0x7fcfe409f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:06.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:06.528+0000 7fcfea963700 1 -- 172.21.15.42:0/2913145377 >> v1:172.21.15.40:6789/0 conn(0x7fcfe40a7730 legacy=0x7fcfe407e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:06.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:06.528+0000 7fcfea963700 1 -- 172.21.15.42:0/2913145377 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfe40a9b90 con 0x7fcfe409fad0 2019-12-05T06:08:06.539 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:06.528+0000 7fcfea963700 1 --2- 172.21.15.42:0/2913145377 >> v2:172.21.15.42:3300/0 conn(0x7fcfe409fad0 0x7fcfe414ffe0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fcfdc008b10 tx=0x7fcfdc010030).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:06.539 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:06.528+0000 7fcfea963700 1 --2- 172.21.15.42:0/2913145377 >> v2:172.21.15.42:3300/0 conn(0x7fcfe409fad0 0x7fcfe414ffe0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fcfdc008b10 tx=0x7fcfdc010030).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:06.539 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:06.528+0000 7fcfea963700 1 --2- 172.21.15.42:0/2913145377 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcfe409fad0 0x7fcfe414ffe0 secure :-1 s=READY pgs=2095 cs=0 l=1 rx=0x7fcfdc008b10 tx=0x7fcfdc010030).ready entity=mon.1 client_cookie=71548c5f064a427 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:08:06.539 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:06.528+0000 7fcfea963700 1 --2- 172.21.15.42:0/2913145377 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcfe409fad0 0x7fcfe414ffe0 secure :-1 s=THROTTLE_DONE pgs=2095 cs=0 l=1 rx=0x7fcfdc008b10 tx=0x7fcfdc010030).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:06.539 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:06.528+0000 7fcfe9160700 1 -- 172.21.15.42:0/2913145377 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcfdc0110c0 con 0x7fcfe409fad0 2019-12-05T06:08:06.539 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:06.528+0000 7fcfea963700 1 --2- 172.21.15.42:0/2913145377 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcfe409fad0 0x7fcfe414ffe0 secure :-1 s=THROTTLE_DONE pgs=2095 cs=0 l=1 rx=0x7fcfdc008b10 tx=0x7fcfdc010030).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:06.539 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:06.528+0000 7fcfea963700 1 --2- 172.21.15.42:0/2913145377 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcfe409fad0 0x7fcfe414ffe0 secure :-1 s=THROTTLE_DONE pgs=2095 cs=0 l=1 rx=0x7fcfdc008b10 tx=0x7fcfdc010030).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:06.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:06.528+0000 7fcfe9160700 1 -- 172.21.15.42:0/2913145377 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfdc010a30 con 0x7fcfe409fad0 2019-12-05T06:08:06.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:06.528+0000 7fcfe9160700 1 -- 172.21.15.42:0/2913145377 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcfdc0114b0 con 0x7fcfe409fad0 2019-12-05T06:08:06.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:06.528+0000 7fcfebbc5700 1 -- 172.21.15.42:0/2913145377 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcfe409fad0 msgr2=0x7fcfe414ffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:06.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:06.528+0000 7fcfebbc5700 1 --2- 172.21.15.42:0/2913145377 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcfe409fad0 0x7fcfe414ffe0 secure :-1 s=READY pgs=2095 cs=0 l=1 rx=0x7fcfdc008b10 tx=0x7fcfdc010030).stop 2019-12-05T06:08:06.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:06.532+0000 7fcfebbc5700 1 -- 172.21.15.42:0/2913145377 shutdown_connections 2019-12-05T06:08:06.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:06.532+0000 7fcfebbc5700 1 --2- 172.21.15.42:0/2913145377 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcfe409fad0 0x7fcfe414ffe0 unknown :-1 s=CLOSED pgs=2095 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:06.541 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:06.532+0000 7fcfebbc5700 1 -- 172.21.15.42:0/2913145377 shutdown_connections 2019-12-05T06:08:06.541 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:06.532+0000 7fcfebbc5700 1 -- 172.21.15.42:0/2913145377 wait complete. 2019-12-05T06:08:06.541 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:06.532+0000 7fcfebbc5700 1 -- 172.21.15.42:0/2913145377 >> 172.21.15.42:0/2913145377 conn(0x7fcfe40aabd0 msgr2=0x7fcfe40a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:08:06.544 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:06.532+0000 7fcfebbc5700 1 Processor -- start 2019-12-05T06:08:06.544 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:06.532+0000 7fcfebbc5700 1 -- start start 2019-12-05T06:08:06.544 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:06.536+0000 7fcfebbc5700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fcfe407c070 0x7fcfe407d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:08:06.545 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:06.536+0000 7fcfebbc5700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcfe41145a0 0x7fcfe41149d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:08:06.545 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:06.536+0000 7fcfebbc5700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfe4119ea0 con 0x7fcfe41226e0 2019-12-05T06:08:06.545 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:06.536+0000 7fcfebbc5700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcfe407c9a0 con 0x7fcfe407c070 2019-12-05T06:08:06.545 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:06.536+0000 7fcfebbc5700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe4119960 con 0x7fcfe41145a0 2019-12-05T06:08:06.545 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:06.536+0000 7fcfea963700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fcfe41226e0 0x7fcfe4125ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:48906/0 (socket says 172.21.15.42:48906) 2019-12-05T06:08:06.545 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:06.536+0000 7fcfea963700 1 -- 172.21.15.42:0/4031681502 learned_addr learned my addr 172.21.15.42:0/4031681502 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:08:06.545 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:06.536+0000 7fcfea162700 1 --2- 172.21.15.42:0/4031681502 >> v2:172.21.15.40:3300/0 conn(0x7fcfe407c070 0x7fcfe407d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:06.546 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:06.536+0000 7fcfe9961700 1 --2- 172.21.15.42:0/4031681502 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcfe41145a0 0x7fcfe41149d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:06.546 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:06.536+0000 7fcfdaffd700 1 -- 172.21.15.42:0/4031681502 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1941425162 0 0) 0x7fcfdc0087f0 con 0x7fcfe41226e0 2019-12-05T06:08:06.546 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:06.536+0000 7fcfdaffd700 1 -- 172.21.15.42:0/4031681502 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcfcc0030b0 con 0x7fcfe41226e0 2019-12-05T06:08:06.546 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:06.536+0000 7fcfea162700 1 -- 172.21.15.42:0/4031681502 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcfe41145a0 msgr2=0x7fcfe41149d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:06.547 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:06.536+0000 7fcfea162700 1 --2- 172.21.15.42:0/4031681502 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcfe41145a0 0x7fcfe41149d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:06.547 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:06.536+0000 7fcfea162700 1 -- 172.21.15.42:0/4031681502 >> v1:172.21.15.42:6789/0 conn(0x7fcfe41226e0 legacy=0x7fcfe4125ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:06.547 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:06.536+0000 7fcfea162700 1 -- 172.21.15.42:0/4031681502 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfe407c5f0 con 0x7fcfe407c070 2019-12-05T06:08:06.548 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:06.536+0000 7fcfea162700 1 --2- 172.21.15.42:0/4031681502 >> v2:172.21.15.40:3300/0 conn(0x7fcfe407c070 0x7fcfe407d960 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fcfe40a97b0 tx=0x7fcfe000c320).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:06.548 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:06.536+0000 7fcfea162700 1 --2- 172.21.15.42:0/4031681502 >> v2:172.21.15.40:3300/0 conn(0x7fcfe407c070 0x7fcfe407d960 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fcfe40a97b0 tx=0x7fcfe000c320).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:06.548 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:06.536+0000 7fcfea162700 1 --2- 172.21.15.42:0/4031681502 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcfe407c070 0x7fcfe407d960 secure :-1 s=READY pgs=1588 cs=0 l=1 rx=0x7fcfe40a97b0 tx=0x7fcfe000c320).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:08:06.548 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:06.536+0000 7fcfea162700 1 --2- 172.21.15.42:0/4031681502 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcfe407c070 0x7fcfe407d960 secure :-1 s=THROTTLE_DONE pgs=1588 cs=0 l=1 rx=0x7fcfe40a97b0 tx=0x7fcfe000c320).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:06.549 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:06.536+0000 7fcfdaffd700 1 -- 172.21.15.42:0/4031681502 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcfe000ca80 con 0x7fcfe407c070 2019-12-05T06:08:06.549 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:06.536+0000 7fcfea162700 1 --2- 172.21.15.42:0/4031681502 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcfe407c070 0x7fcfe407d960 secure :-1 s=THROTTLE_DONE pgs=1588 cs=0 l=1 rx=0x7fcfe40a97b0 tx=0x7fcfe000c320).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:06.549 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:06.536+0000 7fcfea162700 1 --2- 172.21.15.42:0/4031681502 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcfe407c070 0x7fcfe407d960 secure :-1 s=THROTTLE_DONE pgs=1588 cs=0 l=1 rx=0x7fcfe40a97b0 tx=0x7fcfe000c320).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:06.549 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:06.536+0000 7fcfdaffd700 1 -- 172.21.15.42:0/4031681502 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfe000c6a0 con 0x7fcfe407c070 2019-12-05T06:08:06.549 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:06.536+0000 7fcfebbc5700 1 -- 172.21.15.42:0/4031681502 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfe40a4a40 con 0x7fcfe407c070 2019-12-05T06:08:06.549 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:06.536+0000 7fcfdaffd700 1 -- 172.21.15.42:0/4031681502 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcfe00110e0 con 0x7fcfe407c070 2019-12-05T06:08:06.550 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:06.536+0000 7fcfebbc5700 1 -- 172.21.15.42:0/4031681502 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfe4122b10 con 0x7fcfe407c070 2019-12-05T06:08:06.550 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:06.536+0000 7fcfea162700 1 --2- 172.21.15.42:0/4031681502 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcfe407c070 0x7fcfe407d960 secure :-1 s=THROTTLE_DONE pgs=1588 cs=0 l=1 rx=0x7fcfe40a97b0 tx=0x7fcfe000c320).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:06.550 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:06.536+0000 7fcfdaffd700 1 -- 172.21.15.42:0/4031681502 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fcfe00113b0 con 0x7fcfe407c070 2019-12-05T06:08:06.550 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:06.536+0000 7fcfea162700 1 --2- 172.21.15.42:0/4031681502 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcfe407c070 0x7fcfe407d960 secure :-1 s=THROTTLE_DONE pgs=1588 cs=0 l=1 rx=0x7fcfe40a97b0 tx=0x7fcfe000c320).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:06.551 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:06.540+0000 7fcfdaffd700 1 --2- 172.21.15.42:0/4031681502 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fcfcc03b2f0 0x7fcfcc03d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:08:06.551 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:06.540+0000 7fcfdaffd700 1 -- 172.21.15.42:0/4031681502 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9815+0+0 (secure 0 0 0) 0x7fcfe004b190 con 0x7fcfe407c070 2019-12-05T06:08:06.551 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:06.540+0000 7fcfe9961700 1 --2- 172.21.15.42:0/4031681502 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fcfcc03b2f0 0x7fcfcc03d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:06.551 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:06.540+0000 7fcfe9961700 1 --2- 172.21.15.42:0/4031681502 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fcfcc03b2f0 0x7fcfcc03d800 crc :-1 s=READY pgs=885 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:08:06.552 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:06.544+0000 7fcfebbc5700 1 -- 172.21.15.42:0/4031681502 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfe405d100 con 0x7fcfe407c070 2019-12-05T06:08:06.556 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:06.552+0000 7fcfea162700 1 --2- 172.21.15.42:0/4031681502 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcfe407c070 0x7fcfe407d960 secure :-1 s=THROTTLE_DONE pgs=1588 cs=0 l=1 rx=0x7fcfe40a97b0 tx=0x7fcfe000c320).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:06.556 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:06.552+0000 7fcfdaffd700 1 -- 172.21.15.42:0/4031681502 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fcfe0032190 con 0x7fcfe407c070 2019-12-05T06:08:06.726 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:06.720+0000 7fcfebbc5700 1 -- 172.21.15.42:0/4031681502 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcfe405b8b0 con 0x7fcfe407c070 2019-12-05T06:08:06.727 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:06.720+0000 7fcfea162700 1 --2- 172.21.15.42:0/4031681502 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcfe407c070 0x7fcfe407d960 secure :-1 s=THROTTLE_DONE pgs=1588 cs=0 l=1 rx=0x7fcfe40a97b0 tx=0x7fcfe000c320).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:06.727 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:06.720+0000 7fcfdaffd700 1 -- 172.21.15.42:0/4031681502 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 253 v253) v1 ==== 95+0+2112 (secure 0 0 0) 0x7fcfe00323f0 con 0x7fcfe407c070 2019-12-05T06:08:06.729 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 253 2019-12-05T06:08:06.729 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:08:06.729 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":253,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6929,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/2636699468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2636699468},{"type":"v1","addr":"172.21.15.42:6835","nonce":2636699468}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":253}],"filesystems":[{"mdsmap":{"epoch":252,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:07:04.524680+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":93,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9573},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9573":{"gid":9573,"name":"b","rank":0,"incarnation":249,"state":"up:active","state_seq":25,"addr":"172.21.15.40:6833/2968227232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2968227232},{"type":"v1","addr":"172.21.15.40:6833","nonce":2968227232}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:08:06.731 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:06.724+0000 7fcfebbc5700 1 -- 172.21.15.42:0/4031681502 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fcfcc03b2f0 msgr2=0x7fcfcc03d800 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:06.732 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:06.724+0000 7fcfebbc5700 1 --2- 172.21.15.42:0/4031681502 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fcfcc03b2f0 0x7fcfcc03d800 crc :-1 s=READY pgs=885 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:06.732 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:06.724+0000 7fcfebbc5700 1 -- 172.21.15.42:0/4031681502 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcfe407c070 msgr2=0x7fcfe407d960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:06.732 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:06.724+0000 7fcfebbc5700 1 --2- 172.21.15.42:0/4031681502 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcfe407c070 0x7fcfe407d960 secure :-1 s=READY pgs=1588 cs=0 l=1 rx=0x7fcfe40a97b0 tx=0x7fcfe000c320).stop 2019-12-05T06:08:06.732 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:06.724+0000 7fcfebbc5700 1 -- 172.21.15.42:0/4031681502 shutdown_connections 2019-12-05T06:08:06.732 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:06.724+0000 7fcfebbc5700 1 --2- 172.21.15.42:0/4031681502 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fcfcc03b2f0 0x7fcfcc03d800 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:06.732 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:06.724+0000 7fcfebbc5700 1 --2- 172.21.15.42:0/4031681502 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcfe407c070 0x7fcfe407d960 unknown :-1 s=CLOSED pgs=1588 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:06.733 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:06.724+0000 7fcfebbc5700 1 --2- 172.21.15.42:0/4031681502 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcfe41145a0 0x7fcfe41149d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:06.733 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:06.728+0000 7fcfebbc5700 1 -- 172.21.15.42:0/4031681502 shutdown_connections 2019-12-05T06:08:06.733 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:06.728+0000 7fcfebbc5700 1 -- 172.21.15.42:0/4031681502 wait complete. 2019-12-05T06:08:06.733 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:06.728+0000 7fcfebbc5700 1 -- 172.21.15.42:0/4031681502 >> 172.21.15.42:0/4031681502 conn(0x7fcfe407e820 msgr2=0x7fcfe40596f0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:08:06.748 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:08:08.749 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:08:08.854 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:08.844+0000 7f5d7a868700 1 Processor -- start 2019-12-05T06:08:08.855 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:08.844+0000 7f5d7a868700 1 -- start start 2019-12-05T06:08:08.855 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:08.844+0000 7f5d7a868700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5d740a7740 0x7f5d7408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:08:08.855 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:08.844+0000 7f5d7a868700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5d7409fae0 0x7f5d7407eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:08:08.855 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:08.844+0000 7f5d7a868700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d740a0710 con 0x7f5d7407e720 2019-12-05T06:08:08.855 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:08.844+0000 7f5d7a868700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d740a0b00 con 0x7f5d740a7740 2019-12-05T06:08:08.855 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:08.844+0000 7f5d7a868700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d7408e5b0 con 0x7f5d7409fae0 2019-12-05T06:08:08.856 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:08.844+0000 7f5d79606700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5d7409fae0 0x7f5d7407eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:08.856 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:08.844+0000 7f5d79606700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5d7409fae0 0x7f5d7407eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:48878/0 (socket says 172.21.15.42:48878) 2019-12-05T06:08:08.856 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:08.844+0000 7f5d79606700 1 -- 172.21.15.42:0/1819868862 learned_addr learned my addr 172.21.15.42:0/1819868862 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:08:08.856 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:08.844+0000 7f5d78e05700 1 --2- 172.21.15.42:0/1819868862 >> v2:172.21.15.40:3300/0 conn(0x7f5d740a7740 0x7f5d7408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:08.856 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:08.844+0000 7f5d737fe700 1 -- 172.21.15.42:0/1819868862 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 957347461 0 0) 0x7f5d740a0710 con 0x7f5d7407e720 2019-12-05T06:08:08.856 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:08.844+0000 7f5d737fe700 1 -- 172.21.15.42:0/1819868862 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d5c0030b0 con 0x7f5d7407e720 2019-12-05T06:08:08.857 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:08.848+0000 7f5d79606700 1 -- 172.21.15.42:0/1819868862 >> v2:172.21.15.40:3300/0 conn(0x7f5d740a7740 msgr2=0x7f5d7408df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:08:08.857 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:08.848+0000 7f5d79606700 1 --2- 172.21.15.42:0/1819868862 >> v2:172.21.15.40:3300/0 conn(0x7f5d740a7740 0x7f5d7408df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:08:08.857 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:08.848+0000 7f5d79606700 1 -- 172.21.15.42:0/1819868862 >> v1:172.21.15.40:6789/0 conn(0x7f5d7407e720 legacy=0x7f5d7409f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:08.857 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:08.848+0000 7f5d79606700 1 -- 172.21.15.42:0/1819868862 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d7407f0a0 con 0x7f5d7409fae0 2019-12-05T06:08:08.857 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:08.848+0000 7f5d79606700 1 --2- 172.21.15.42:0/1819868862 >> v2:172.21.15.42:3300/0 conn(0x7f5d7409fae0 0x7f5d7407eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f5d6c002f30 tx=0x7f5d6c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:08.858 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:08.848+0000 7f5d79606700 1 --2- 172.21.15.42:0/1819868862 >> v2:172.21.15.42:3300/0 conn(0x7f5d7409fae0 0x7f5d7407eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f5d6c002f30 tx=0x7f5d6c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:08.858 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:08.848+0000 7f5d79606700 1 --2- 172.21.15.42:0/1819868862 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5d7409fae0 0x7f5d7407eb50 secure :-1 s=READY pgs=2097 cs=0 l=1 rx=0x7f5d6c002f30 tx=0x7f5d6c00fd30).ready entity=mon.1 client_cookie=13b170be7dfc686b server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:08:08.858 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:08.848+0000 7f5d79606700 1 --2- 172.21.15.42:0/1819868862 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5d7409fae0 0x7f5d7407eb50 secure :-1 s=THROTTLE_DONE pgs=2097 cs=0 l=1 rx=0x7f5d6c002f30 tx=0x7f5d6c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:08.858 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:08.848+0000 7f5d737fe700 1 -- 172.21.15.42:0/1819868862 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5d6c008c90 con 0x7f5d7409fae0 2019-12-05T06:08:08.858 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:08.848+0000 7f5d79606700 1 --2- 172.21.15.42:0/1819868862 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5d7409fae0 0x7f5d7407eb50 secure :-1 s=THROTTLE_DONE pgs=2097 cs=0 l=1 rx=0x7f5d6c002f30 tx=0x7f5d6c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:08.858 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:08.848+0000 7f5d79606700 1 --2- 172.21.15.42:0/1819868862 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5d7409fae0 0x7f5d7407eb50 secure :-1 s=THROTTLE_DONE pgs=2097 cs=0 l=1 rx=0x7f5d6c002f30 tx=0x7f5d6c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:08.859 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:08.848+0000 7f5d737fe700 1 -- 172.21.15.42:0/1819868862 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d6c0088b0 con 0x7f5d7409fae0 2019-12-05T06:08:08.859 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:08.848+0000 7f5d737fe700 1 -- 172.21.15.42:0/1819868862 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5d6c011ad0 con 0x7f5d7409fae0 2019-12-05T06:08:08.859 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:08.848+0000 7f5d7a868700 1 -- 172.21.15.42:0/1819868862 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5d7409fae0 msgr2=0x7f5d7407eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:08.859 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:08.848+0000 7f5d7a868700 1 --2- 172.21.15.42:0/1819868862 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5d7409fae0 0x7f5d7407eb50 secure :-1 s=READY pgs=2097 cs=0 l=1 rx=0x7f5d6c002f30 tx=0x7f5d6c00fd30).stop 2019-12-05T06:08:08.859 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:08.848+0000 7f5d7a868700 1 -- 172.21.15.42:0/1819868862 shutdown_connections 2019-12-05T06:08:08.859 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:08.848+0000 7f5d7a868700 1 --2- 172.21.15.42:0/1819868862 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5d7409fae0 0x7f5d7407eb50 unknown :-1 s=CLOSED pgs=2097 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:08.859 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:08.848+0000 7f5d7a868700 1 --2- 172.21.15.42:0/1819868862 >> v2:172.21.15.40:3300/0 conn(0x7f5d740a7740 0x7f5d7408df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:08:08.860 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:08.848+0000 7f5d7a868700 1 -- 172.21.15.42:0/1819868862 shutdown_connections 2019-12-05T06:08:08.860 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:08.848+0000 7f5d7a868700 1 -- 172.21.15.42:0/1819868862 wait complete. 2019-12-05T06:08:08.860 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:08.848+0000 7f5d7a868700 1 -- 172.21.15.42:0/1819868862 >> 172.21.15.42:0/1819868862 conn(0x7f5d740aabe0 msgr2=0x7f5d740a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:08:08.863 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:08.852+0000 7f5d7a868700 1 Processor -- start 2019-12-05T06:08:08.863 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:08.852+0000 7f5d7a868700 1 -- start start 2019-12-05T06:08:08.863 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:08.852+0000 7f5d7a868700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5d7411f740 0x7f5d74124230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:08:08.864 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:08.852+0000 7f5d7a868700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5d741336c0 0x7f5d74133af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:08:08.864 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:08.852+0000 7f5d7a868700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d7411a7a0 con 0x7f5d7411a370 2019-12-05T06:08:08.864 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:08.852+0000 7f5d7a868700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d7411aa80 con 0x7f5d7411f740 2019-12-05T06:08:08.864 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:08.852+0000 7f5d7a868700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d7411ad20 con 0x7f5d741336c0 2019-12-05T06:08:08.864 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:08.852+0000 7f5d79606700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5d741336c0 0x7f5d74133af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:08.864 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:08.852+0000 7f5d79606700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5d741336c0 0x7f5d74133af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:48886/0 (socket says 172.21.15.42:48886) 2019-12-05T06:08:08.865 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:08.852+0000 7f5d79606700 1 -- 172.21.15.42:0/3200048348 learned_addr learned my addr 172.21.15.42:0/3200048348 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:08:08.865 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:08.852+0000 7f5d78e05700 1 --2- 172.21.15.42:0/3200048348 >> v2:172.21.15.40:3300/0 conn(0x7f5d7411f740 0x7f5d74124230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:08.865 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:08.852+0000 7f5d79606700 1 -- 172.21.15.42:0/3200048348 >> v2:172.21.15.40:3300/0 conn(0x7f5d7411f740 msgr2=0x7f5d74124230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:08.865 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:08.852+0000 7f5d79606700 1 --2- 172.21.15.42:0/3200048348 >> v2:172.21.15.40:3300/0 conn(0x7f5d7411f740 0x7f5d74124230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:08.865 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:08.852+0000 7f5d79606700 1 -- 172.21.15.42:0/3200048348 >> v1:172.21.15.40:6789/0 conn(0x7f5d7411a370 legacy=0x7f5d74121370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:08.865 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:08.852+0000 7f5d79606700 1 -- 172.21.15.42:0/3200048348 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d74134040 con 0x7f5d741336c0 2019-12-05T06:08:08.866 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:08.852+0000 7f5d79606700 1 --2- 172.21.15.42:0/3200048348 >> v2:172.21.15.42:3300/0 conn(0x7f5d741336c0 0x7f5d74133af0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f5d6c002f30 tx=0x7f5d6c008d90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:08.866 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:08.852+0000 7f5d79606700 1 --2- 172.21.15.42:0/3200048348 >> v2:172.21.15.42:3300/0 conn(0x7f5d741336c0 0x7f5d74133af0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f5d6c002f30 tx=0x7f5d6c008d90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:08.866 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:08.852+0000 7f5d79606700 1 --2- 172.21.15.42:0/3200048348 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5d741336c0 0x7f5d74133af0 secure :-1 s=READY pgs=2098 cs=0 l=1 rx=0x7f5d6c002f30 tx=0x7f5d6c008d90).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:08:08.866 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:08.852+0000 7f5d79606700 1 --2- 172.21.15.42:0/3200048348 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5d741336c0 0x7f5d74133af0 secure :-1 s=THROTTLE_DONE pgs=2098 cs=0 l=1 rx=0x7f5d6c002f30 tx=0x7f5d6c008d90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:08.867 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:08.852+0000 7f5d79606700 1 --2- 172.21.15.42:0/3200048348 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5d741336c0 0x7f5d74133af0 secure :-1 s=THROTTLE_DONE pgs=2098 cs=0 l=1 rx=0x7f5d6c002f30 tx=0x7f5d6c008d90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:08.867 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:08.852+0000 7f5d79606700 1 --2- 172.21.15.42:0/3200048348 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5d741336c0 0x7f5d74133af0 secure :-1 s=THROTTLE_DONE pgs=2098 cs=0 l=1 rx=0x7f5d6c002f30 tx=0x7f5d6c008d90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:08.867 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:08.852+0000 7f5d717fa700 1 -- 172.21.15.42:0/3200048348 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5d6c00f470 con 0x7f5d741336c0 2019-12-05T06:08:08.867 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:08.852+0000 7f5d717fa700 1 -- 172.21.15.42:0/3200048348 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d6c0116f0 con 0x7f5d741336c0 2019-12-05T06:08:08.867 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:08.852+0000 7f5d7a868700 1 -- 172.21.15.42:0/3200048348 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d74134310 con 0x7f5d741336c0 2019-12-05T06:08:08.867 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:08.852+0000 7f5d717fa700 1 -- 172.21.15.42:0/3200048348 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5d6c005730 con 0x7f5d741336c0 2019-12-05T06:08:08.867 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:08.852+0000 7f5d7a868700 1 -- 172.21.15.42:0/3200048348 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d74077300 con 0x7f5d741336c0 2019-12-05T06:08:08.868 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:08.856+0000 7f5d79606700 1 --2- 172.21.15.42:0/3200048348 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5d741336c0 0x7f5d74133af0 secure :-1 s=THROTTLE_DONE pgs=2098 cs=0 l=1 rx=0x7f5d6c002f30 tx=0x7f5d6c008d90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:08.868 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:08.856+0000 7f5d717fa700 1 -- 172.21.15.42:0/3200048348 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f5d6c005a40 con 0x7f5d741336c0 2019-12-05T06:08:08.868 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:08.856+0000 7f5d79606700 1 --2- 172.21.15.42:0/3200048348 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5d741336c0 0x7f5d74133af0 secure :-1 s=THROTTLE_DONE pgs=2098 cs=0 l=1 rx=0x7f5d6c002f30 tx=0x7f5d6c008d90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:08.868 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:08.856+0000 7f5d717fa700 1 --2- 172.21.15.42:0/3200048348 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5d5403ae00 0x7f5d5403d310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:08:08.868 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:08.856+0000 7f5d717fa700 1 -- 172.21.15.42:0/3200048348 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9815+0+0 (secure 0 0 0) 0x7f5d6c04b7b0 con 0x7f5d741336c0 2019-12-05T06:08:08.868 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:08.856+0000 7f5d78e05700 1 --2- 172.21.15.42:0/3200048348 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5d5403ae00 0x7f5d5403d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:08.868 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:08.856+0000 7f5d78e05700 1 --2- 172.21.15.42:0/3200048348 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5d5403ae00 0x7f5d5403d310 crc :-1 s=READY pgs=886 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:08:08.869 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:08.860+0000 7f5d7a868700 1 -- 172.21.15.42:0/3200048348 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d74124ce0 con 0x7f5d741336c0 2019-12-05T06:08:08.874 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:08.868+0000 7f5d79606700 1 --2- 172.21.15.42:0/3200048348 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5d741336c0 0x7f5d74133af0 secure :-1 s=THROTTLE_DONE pgs=2098 cs=0 l=1 rx=0x7f5d6c002f30 tx=0x7f5d6c008d90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:08.874 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:08.868+0000 7f5d717fa700 1 -- 172.21.15.42:0/3200048348 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f5d6c007770 con 0x7f5d741336c0 2019-12-05T06:08:09.043 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:09.036+0000 7f5d7a868700 1 -- 172.21.15.42:0/3200048348 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5d7405d110 con 0x7f5d741336c0 2019-12-05T06:08:09.044 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:09.040+0000 7f5d79606700 1 --2- 172.21.15.42:0/3200048348 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5d741336c0 0x7f5d74133af0 secure :-1 s=THROTTLE_DONE pgs=2098 cs=0 l=1 rx=0x7f5d6c002f30 tx=0x7f5d6c008d90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:09.044 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:09.040+0000 7f5d717fa700 1 -- 172.21.15.42:0/3200048348 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 253 v253) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f5d6c033680 con 0x7f5d741336c0 2019-12-05T06:08:09.045 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 253 2019-12-05T06:08:09.045 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:08:09.045 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":253,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6929,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/2636699468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2636699468},{"type":"v1","addr":"172.21.15.42:6835","nonce":2636699468}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":253}],"filesystems":[{"mdsmap":{"epoch":252,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:07:04.524680+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":93,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9573},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9573":{"gid":9573,"name":"b","rank":0,"incarnation":249,"state":"up:active","state_seq":25,"addr":"172.21.15.40:6833/2968227232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2968227232},{"type":"v1","addr":"172.21.15.40:6833","nonce":2968227232}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:08:09.047 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:09.040+0000 7f5d7a868700 1 -- 172.21.15.42:0/3200048348 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5d5403ae00 msgr2=0x7f5d5403d310 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:09.047 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:09.040+0000 7f5d7a868700 1 --2- 172.21.15.42:0/3200048348 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5d5403ae00 0x7f5d5403d310 crc :-1 s=READY pgs=886 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:09.047 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:09.040+0000 7f5d7a868700 1 -- 172.21.15.42:0/3200048348 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5d741336c0 msgr2=0x7f5d74133af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:09.047 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:09.040+0000 7f5d7a868700 1 --2- 172.21.15.42:0/3200048348 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5d741336c0 0x7f5d74133af0 secure :-1 s=READY pgs=2098 cs=0 l=1 rx=0x7f5d6c002f30 tx=0x7f5d6c008d90).stop 2019-12-05T06:08:09.047 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:09.040+0000 7f5d7a868700 1 -- 172.21.15.42:0/3200048348 shutdown_connections 2019-12-05T06:08:09.047 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:09.040+0000 7f5d7a868700 1 --2- 172.21.15.42:0/3200048348 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5d5403ae00 0x7f5d5403d310 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:09.047 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:09.040+0000 7f5d7a868700 1 --2- 172.21.15.42:0/3200048348 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5d741336c0 0x7f5d74133af0 unknown :-1 s=CLOSED pgs=2098 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:09.048 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:09.040+0000 7f5d7a868700 1 --2- 172.21.15.42:0/3200048348 >> v2:172.21.15.40:3300/0 conn(0x7f5d7411f740 0x7f5d74124230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:09.048 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:09.040+0000 7f5d7a868700 1 -- 172.21.15.42:0/3200048348 shutdown_connections 2019-12-05T06:08:09.048 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:09.040+0000 7f5d7a868700 1 -- 172.21.15.42:0/3200048348 wait complete. 2019-12-05T06:08:09.048 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:09.040+0000 7f5d7a868700 1 -- 172.21.15.42:0/3200048348 >> 172.21.15.42:0/3200048348 conn(0x7f5d740a82a0 msgr2=0x7f5d7409f380 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:08:09.056 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:08:11.058 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:08:11.154 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.144+0000 7fa6fc239700 1 Processor -- start 2019-12-05T06:08:11.154 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.144+0000 7fa6fc239700 1 -- start start 2019-12-05T06:08:11.155 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.144+0000 7fa6fc239700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa6f409fad0 0x7fa6f414ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:08:11.155 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.144+0000 7fa6fc239700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6f40a0700 con 0x7fa6f408df70 2019-12-05T06:08:11.155 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.144+0000 7fa6fc239700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6f407ef10 con 0x7fa6f40a7730 2019-12-05T06:08:11.155 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.144+0000 7fa6fc239700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6f40a0af0 con 0x7fa6f409fad0 2019-12-05T06:08:11.155 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.144+0000 7fa6fa7d6700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fa6f40a7730 0x7fa6f407e710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:48948/0 (socket says 172.21.15.42:48948) 2019-12-05T06:08:11.156 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.144+0000 7fa6fa7d6700 1 -- 172.21.15.42:0/3352162547 learned_addr learned my addr 172.21.15.42:0/3352162547 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:08:11.156 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.144+0000 7fa6fafd7700 1 --2- 172.21.15.42:0/3352162547 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa6f409fad0 0x7fa6f414ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:11.156 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.144+0000 7fa6f97d4700 1 -- 172.21.15.42:0/3352162547 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3858801447 0 0) 0x7fa6f407ef10 con 0x7fa6f40a7730 2019-12-05T06:08:11.156 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.144+0000 7fa6f97d4700 1 -- 172.21.15.42:0/3352162547 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6e00030b0 con 0x7fa6f40a7730 2019-12-05T06:08:11.156 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.144+0000 7fa6f97d4700 1 -- 172.21.15.42:0/3352162547 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4001763822 0 0) 0x7fa6f40a0700 con 0x7fa6f408df70 2019-12-05T06:08:11.157 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.144+0000 7fa6f97d4700 1 -- 172.21.15.42:0/3352162547 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6f407ef10 con 0x7fa6f408df70 2019-12-05T06:08:11.157 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.144+0000 7fa6f97d4700 1 -- 172.21.15.42:0/3352162547 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3747243328 0 0) 0x7fa6e00030b0 con 0x7fa6f40a7730 2019-12-05T06:08:11.157 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.144+0000 7fa6f97d4700 1 -- 172.21.15.42:0/3352162547 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa6f409fad0 msgr2=0x7fa6f414ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:08:11.157 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.144+0000 7fa6f97d4700 1 --2- 172.21.15.42:0/3352162547 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa6f409fad0 0x7fa6f414ffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:08:11.157 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.144+0000 7fa6f97d4700 1 -- 172.21.15.42:0/3352162547 >> v1:172.21.15.40:6789/0 conn(0x7fa6f408df70 legacy=0x7fa6f409f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:11.168 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.144+0000 7fa6f97d4700 1 -- 172.21.15.42:0/3352162547 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6f40a9b90 con 0x7fa6f40a7730 2019-12-05T06:08:11.168 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.144+0000 7fa6f97d4700 1 -- 172.21.15.42:0/3352162547 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7fa6f0001750 con 0x7fa6f40a7730 2019-12-05T06:08:11.168 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.144+0000 7fa6f97d4700 1 -- 172.21.15.42:0/3352162547 >> v1:172.21.15.42:6789/0 conn(0x7fa6f40a7730 legacy=0x7fa6f407e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:11.168 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.144+0000 7fa6f97d4700 1 --2- 172.21.15.42:0/3352162547 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa6e0003940 0x7fa6e0003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:08:11.169 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.144+0000 7fa6f97d4700 1 --2- 172.21.15.42:0/3352162547 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa6e0004420 0x7fa6e00048d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:08:11.169 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.144+0000 7fa6f97d4700 1 --2- 172.21.15.42:0/3352162547 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa6e0004f30 0x7fa6e004f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:08:11.169 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.144+0000 7fa6f97d4700 1 -- 172.21.15.42:0/3352162547 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6f40a0af0 con 0x7fa6e0003940 2019-12-05T06:08:11.169 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.144+0000 7fa6f97d4700 1 -- 172.21.15.42:0/3352162547 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6e004fbd0 con 0x7fa6e0004420 2019-12-05T06:08:11.169 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.144+0000 7fa6f97d4700 1 -- 172.21.15.42:0/3352162547 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6e004fde0 con 0x7fa6e0004f30 2019-12-05T06:08:11.169 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.144+0000 7fa6f9fd5700 1 --2- 172.21.15.42:0/3352162547 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa6e0004420 0x7fa6e00048d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:11.169 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.144+0000 7fa6fa7d6700 1 --2- 172.21.15.42:0/3352162547 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa6e0003940 0x7fa6e0003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:11.170 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.144+0000 7fa6fafd7700 1 --2- 172.21.15.42:0/3352162547 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa6e0004f30 0x7fa6e004f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:11.170 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.144+0000 7fa6f9fd5700 1 -- 172.21.15.42:0/3352162547 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa6e0004f30 msgr2=0x7fa6e004f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:08:11.170 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.144+0000 7fa6f9fd5700 1 --2- 172.21.15.42:0/3352162547 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa6e0004f30 0x7fa6e004f680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:08:11.170 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.144+0000 7fa6f9fd5700 1 -- 172.21.15.42:0/3352162547 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa6e0003940 msgr2=0x7fa6e0003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:08:11.170 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.144+0000 7fa6f9fd5700 1 --2- 172.21.15.42:0/3352162547 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa6e0003940 0x7fa6e0003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:08:11.170 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.144+0000 7fa6f9fd5700 1 -- 172.21.15.42:0/3352162547 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6e00500e0 con 0x7fa6e0004420 2019-12-05T06:08:11.170 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.144+0000 7fa6f9fd5700 1 --2- 172.21.15.42:0/3352162547 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa6e0004420 0x7fa6e00048d0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fa6e4000fe0 tx=0x7fa6e400be70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:11.171 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.144+0000 7fa6f9fd5700 1 --2- 172.21.15.42:0/3352162547 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa6e0004420 0x7fa6e00048d0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fa6e4000fe0 tx=0x7fa6e400be70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:11.171 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.144+0000 7fa6f9fd5700 1 --2- 172.21.15.42:0/3352162547 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa6e0004420 0x7fa6e00048d0 secure :-1 s=READY pgs=2101 cs=0 l=1 rx=0x7fa6e4000fe0 tx=0x7fa6e400be70).ready entity=mon.1 client_cookie=9a00f7b7030ca969 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:08:11.171 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.148+0000 7fa6f9fd5700 1 --2- 172.21.15.42:0/3352162547 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa6e0004420 0x7fa6e00048d0 secure :-1 s=THROTTLE_DONE pgs=2101 cs=0 l=1 rx=0x7fa6e4000fe0 tx=0x7fa6e400be70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:11.171 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.148+0000 7fa6f97d4700 1 -- 172.21.15.42:0/3352162547 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa6e4010be0 con 0x7fa6e0004420 2019-12-05T06:08:11.171 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.148+0000 7fa6f9fd5700 1 --2- 172.21.15.42:0/3352162547 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa6e0004420 0x7fa6e00048d0 secure :-1 s=THROTTLE_DONE pgs=2101 cs=0 l=1 rx=0x7fa6e4000fe0 tx=0x7fa6e400be70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:11.171 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.148+0000 7fa6f9fd5700 1 --2- 172.21.15.42:0/3352162547 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa6e0004420 0x7fa6e00048d0 secure :-1 s=THROTTLE_DONE pgs=2101 cs=0 l=1 rx=0x7fa6e4000fe0 tx=0x7fa6e400be70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:11.172 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.148+0000 7fa6f97d4700 1 -- 172.21.15.42:0/3352162547 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6e4010800 con 0x7fa6e0004420 2019-12-05T06:08:11.172 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.148+0000 7fa6f97d4700 1 -- 172.21.15.42:0/3352162547 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa6e400c5b0 con 0x7fa6e0004420 2019-12-05T06:08:11.172 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.148+0000 7fa6fc239700 1 -- 172.21.15.42:0/3352162547 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa6e0004420 msgr2=0x7fa6e00048d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:11.172 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.148+0000 7fa6fc239700 1 --2- 172.21.15.42:0/3352162547 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa6e0004420 0x7fa6e00048d0 secure :-1 s=READY pgs=2101 cs=0 l=1 rx=0x7fa6e4000fe0 tx=0x7fa6e400be70).stop 2019-12-05T06:08:11.172 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.148+0000 7fa6fafd7700 1 -- 172.21.15.42:0/3352162547 reap_dead start 2019-12-05T06:08:11.173 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.148+0000 7fa6fc239700 1 -- 172.21.15.42:0/3352162547 shutdown_connections 2019-12-05T06:08:11.173 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.148+0000 7fa6fc239700 1 -- 172.21.15.42:0/3352162547 shutdown_connections 2019-12-05T06:08:11.173 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.148+0000 7fa6fc239700 1 -- 172.21.15.42:0/3352162547 wait complete. 2019-12-05T06:08:11.173 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.148+0000 7fa6fc239700 1 -- 172.21.15.42:0/3352162547 >> 172.21.15.42:0/3352162547 conn(0x7fa6f40aabd0 msgr2=0x7fa6f40a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:08:11.173 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.148+0000 7fa6fc239700 1 Processor -- start 2019-12-05T06:08:11.173 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.148+0000 7fa6fc239700 1 -- start start 2019-12-05T06:08:11.180 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.148+0000 7fa6fc239700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fa6f40a3a50 0x7fa6f4129960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:08:11.180 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.148+0000 7fa6fc239700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6f40a2f10 con 0x7fa6f40a1740 2019-12-05T06:08:11.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.148+0000 7fa6fc239700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6f411fc60 con 0x7fa6f40a5f90 2019-12-05T06:08:11.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.148+0000 7fa6fc239700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6f40a2b20 con 0x7fa6f40a3a50 2019-12-05T06:08:11.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.148+0000 7fa6f9fd5700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fa6f40a5f90 0x7fa6f4126140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:48962/0 (socket says 172.21.15.42:48962) 2019-12-05T06:08:11.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.148+0000 7fa6f9fd5700 1 -- 172.21.15.42:0/2382079600 learned_addr learned my addr 172.21.15.42:0/2382079600 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:08:11.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.148+0000 7fa6fafd7700 1 --2- 172.21.15.42:0/2382079600 >> v2:172.21.15.42:3300/0 conn(0x7fa6f40a3a50 0x7fa6f4129960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:11.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.148+0000 7fa6eaffd700 1 -- 172.21.15.42:0/2382079600 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3887895613 0 0) 0x7fa6f411fc60 con 0x7fa6f40a5f90 2019-12-05T06:08:11.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.148+0000 7fa6eaffd700 1 -- 172.21.15.42:0/2382079600 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6dc0030b0 con 0x7fa6f40a5f90 2019-12-05T06:08:11.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.148+0000 7fa6eaffd700 1 -- 172.21.15.42:0/2382079600 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 201943717 0 0) 0x7fa6dc0030b0 con 0x7fa6f40a5f90 2019-12-05T06:08:11.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.148+0000 7fa6fafd7700 1 -- 172.21.15.42:0/2382079600 >> v1:172.21.15.42:6789/0 conn(0x7fa6f40a5f90 legacy=0x7fa6f4126140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:11.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.148+0000 7fa6fafd7700 1 -- 172.21.15.42:0/2382079600 >> v1:172.21.15.40:6789/0 conn(0x7fa6f40a1740 legacy=0x7fa6f40a3250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:11.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.148+0000 7fa6fafd7700 1 -- 172.21.15.42:0/2382079600 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6f40a6ad0 con 0x7fa6f40a3a50 2019-12-05T06:08:11.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.152+0000 7fa6fafd7700 1 --2- 172.21.15.42:0/2382079600 >> v2:172.21.15.42:3300/0 conn(0x7fa6f40a3a50 0x7fa6f4129960 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fa6e4001bc0 tx=0x7fa6e4010480).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:11.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.152+0000 7fa6fafd7700 1 --2- 172.21.15.42:0/2382079600 >> v2:172.21.15.42:3300/0 conn(0x7fa6f40a3a50 0x7fa6f4129960 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fa6e4001bc0 tx=0x7fa6e4010480).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:11.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.152+0000 7fa6fafd7700 1 --2- 172.21.15.42:0/2382079600 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa6f40a3a50 0x7fa6f4129960 secure :-1 s=READY pgs=2103 cs=0 l=1 rx=0x7fa6e4001bc0 tx=0x7fa6e4010480).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:08:11.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.152+0000 7fa6fafd7700 1 --2- 172.21.15.42:0/2382079600 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa6f40a3a50 0x7fa6f4129960 secure :-1 s=THROTTLE_DONE pgs=2103 cs=0 l=1 rx=0x7fa6e4001bc0 tx=0x7fa6e4010480).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:11.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.152+0000 7fa6fafd7700 1 --2- 172.21.15.42:0/2382079600 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa6f40a3a50 0x7fa6f4129960 secure :-1 s=THROTTLE_DONE pgs=2103 cs=0 l=1 rx=0x7fa6e4001bc0 tx=0x7fa6e4010480).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:11.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.152+0000 7fa6fafd7700 1 --2- 172.21.15.42:0/2382079600 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa6f40a3a50 0x7fa6f4129960 secure :-1 s=THROTTLE_DONE pgs=2103 cs=0 l=1 rx=0x7fa6e4001bc0 tx=0x7fa6e4010480).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:11.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.152+0000 7fa6eaffd700 1 -- 172.21.15.42:0/2382079600 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa6e4001340 con 0x7fa6f40a3a50 2019-12-05T06:08:11.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.152+0000 7fa6eaffd700 1 -- 172.21.15.42:0/2382079600 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6e4010de0 con 0x7fa6f40a3a50 2019-12-05T06:08:11.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.152+0000 7fa6eaffd700 1 -- 172.21.15.42:0/2382079600 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa6ec007350 con 0x7fa6f40a3a50 2019-12-05T06:08:11.184 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.152+0000 7fa6fc239700 1 -- 172.21.15.42:0/2382079600 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6f4129eb0 con 0x7fa6f40a3a50 2019-12-05T06:08:11.184 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.152+0000 7fa6fc239700 1 -- 172.21.15.42:0/2382079600 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6f412a380 con 0x7fa6f40a3a50 2019-12-05T06:08:11.184 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.152+0000 7fa6fafd7700 1 --2- 172.21.15.42:0/2382079600 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa6f40a3a50 0x7fa6f4129960 secure :-1 s=THROTTLE_DONE pgs=2103 cs=0 l=1 rx=0x7fa6e4001bc0 tx=0x7fa6e4010480).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:11.184 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.152+0000 7fa6eaffd700 1 -- 172.21.15.42:0/2382079600 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fa6ec007550 con 0x7fa6f40a3a50 2019-12-05T06:08:11.184 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.152+0000 7fa6fafd7700 1 --2- 172.21.15.42:0/2382079600 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa6f40a3a50 0x7fa6f4129960 secure :-1 s=THROTTLE_DONE pgs=2103 cs=0 l=1 rx=0x7fa6e4001bc0 tx=0x7fa6e4010480).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:11.184 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.152+0000 7fa6eaffd700 1 --2- 172.21.15.42:0/2382079600 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa6dc03b210 0x7fa6dc03d720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:08:11.185 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.152+0000 7fa6eaffd700 1 -- 172.21.15.42:0/2382079600 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9901+0+0 (secure 0 0 0) 0x7fa6ec04a810 con 0x7fa6f40a3a50 2019-12-05T06:08:11.185 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.152+0000 7fa6fa7d6700 1 --2- 172.21.15.42:0/2382079600 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa6dc03b210 0x7fa6dc03d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:11.185 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.152+0000 7fa6fa7d6700 1 --2- 172.21.15.42:0/2382079600 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa6dc03b210 0x7fa6dc03d720 crc :-1 s=READY pgs=887 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:08:11.185 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.160+0000 7fa6fc239700 1 -- 172.21.15.42:0/2382079600 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6f405d100 con 0x7fa6f40a3a50 2019-12-05T06:08:11.185 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.164+0000 7fa6fafd7700 1 --2- 172.21.15.42:0/2382079600 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa6f40a3a50 0x7fa6f4129960 secure :-1 s=THROTTLE_DONE pgs=2103 cs=0 l=1 rx=0x7fa6e4001bc0 tx=0x7fa6e4010480).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:11.186 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.164+0000 7fa6eaffd700 1 -- 172.21.15.42:0/2382079600 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fa6ec00b170 con 0x7fa6f40a3a50 2019-12-05T06:08:11.328 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.324+0000 7fa6fc239700 1 -- 172.21.15.42:0/2382079600 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa6f405b8b0 con 0x7fa6f40a3a50 2019-12-05T06:08:11.329 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.324+0000 7fa6fafd7700 1 --2- 172.21.15.42:0/2382079600 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa6f40a3a50 0x7fa6f4129960 secure :-1 s=THROTTLE_DONE pgs=2103 cs=0 l=1 rx=0x7fa6e4001bc0 tx=0x7fa6e4010480).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:11.329 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.324+0000 7fa6eaffd700 1 -- 172.21.15.42:0/2382079600 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 254 v254) v1 ==== 95+0+1774 (secure 0 0 0) 0x7fa6ec031890 con 0x7fa6f40a3a50 2019-12-05T06:08:11.329 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 254 2019-12-05T06:08:11.329 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:08:11.330 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":254,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":254,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:08:09.493328+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":94,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6929},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6929":{"gid":6929,"name":"a","rank":0,"incarnation":254,"state":"up:replay","state_seq":2,"addr":"172.21.15.42:6835/2636699468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2636699468},{"type":"v1","addr":"172.21.15.42:6835","nonce":2636699468}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:08:11.332 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.324+0000 7fa6fc239700 1 -- 172.21.15.42:0/2382079600 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa6dc03b210 msgr2=0x7fa6dc03d720 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:11.332 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.324+0000 7fa6fc239700 1 --2- 172.21.15.42:0/2382079600 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa6dc03b210 0x7fa6dc03d720 crc :-1 s=READY pgs=887 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:11.332 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.324+0000 7fa6fc239700 1 -- 172.21.15.42:0/2382079600 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa6f40a3a50 msgr2=0x7fa6f4129960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:11.332 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.324+0000 7fa6fc239700 1 --2- 172.21.15.42:0/2382079600 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa6f40a3a50 0x7fa6f4129960 secure :-1 s=READY pgs=2103 cs=0 l=1 rx=0x7fa6e4001bc0 tx=0x7fa6e4010480).stop 2019-12-05T06:08:11.332 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.324+0000 7fa6fc239700 1 -- 172.21.15.42:0/2382079600 shutdown_connections 2019-12-05T06:08:11.332 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.324+0000 7fa6fc239700 1 --2- 172.21.15.42:0/2382079600 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa6dc03b210 0x7fa6dc03d720 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:11.333 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.324+0000 7fa6fc239700 1 --2- 172.21.15.42:0/2382079600 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa6f40a3a50 0x7fa6f4129960 unknown :-1 s=CLOSED pgs=2103 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:11.333 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.328+0000 7fa6fc239700 1 -- 172.21.15.42:0/2382079600 shutdown_connections 2019-12-05T06:08:11.333 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.328+0000 7fa6fc239700 1 -- 172.21.15.42:0/2382079600 wait complete. 2019-12-05T06:08:11.333 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.328+0000 7fa6fc239700 1 -- 172.21.15.42:0/2382079600 >> 172.21.15.42:0/2382079600 conn(0x7fa6f409f170 msgr2=0x7fa6f409f650 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:08:11.341 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b down, removed from mdsmap 2019-12-05T06:08:11.341 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2019-12-05T06:08:11.341 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:08:11.432 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.420+0000 7f3bfb2b0700 1 Processor -- start 2019-12-05T06:08:11.432 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.420+0000 7f3bfb2b0700 1 -- start start 2019-12-05T06:08:11.432 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.424+0000 7f3bfb2b0700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f3bf40a7740 0x7f3bf408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:08:11.432 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.424+0000 7f3bfb2b0700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3bf407e720 0x7f3bf407eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:08:11.432 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.424+0000 7f3bfb2b0700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bf40a0710 con 0x7f3bf409f2e0 2019-12-05T06:08:11.433 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.424+0000 7f3bfb2b0700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3bf40a0b00 con 0x7f3bf40a7740 2019-12-05T06:08:11.433 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.424+0000 7f3bfb2b0700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bf408e5b0 con 0x7f3bf407e720 2019-12-05T06:08:11.433 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.424+0000 7f3bf984d700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f3bf40a7740 0x7f3bf408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:11.433 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.424+0000 7f3bf904c700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3bf407e720 0x7f3bf407eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:11.433 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.424+0000 7f3bf984d700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f3bf40a7740 0x7f3bf408df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.42:36746/0 (socket says 172.21.15.42:36746) 2019-12-05T06:08:11.434 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.424+0000 7f3bf984d700 1 -- 172.21.15.42:0/2563442954 learned_addr learned my addr 172.21.15.42:0/2563442954 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:08:11.434 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.424+0000 7f3bf984d700 1 -- 172.21.15.42:0/2563442954 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3bf407e720 msgr2=0x7f3bf407eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:08:11.434 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.424+0000 7f3bf984d700 1 --2- 172.21.15.42:0/2563442954 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3bf407e720 0x7f3bf407eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:08:11.434 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.424+0000 7f3bf984d700 1 -- 172.21.15.42:0/2563442954 >> v1:172.21.15.40:6789/0 conn(0x7f3bf409f2e0 legacy=0x7f3bf409f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:11.434 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.424+0000 7f3bf984d700 1 -- 172.21.15.42:0/2563442954 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bf407f0a0 con 0x7f3bf40a7740 2019-12-05T06:08:11.435 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.424+0000 7f3bf984d700 1 --2- 172.21.15.42:0/2563442954 >> v2:172.21.15.40:3300/0 conn(0x7f3bf40a7740 0x7f3bf408df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f3bec008b10 tx=0x7f3bec0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:11.435 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.424+0000 7f3bf984d700 1 --2- 172.21.15.42:0/2563442954 >> v2:172.21.15.40:3300/0 conn(0x7f3bf40a7740 0x7f3bf408df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f3bec008b10 tx=0x7f3bec0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:11.435 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.424+0000 7f3bf984d700 1 --2- 172.21.15.42:0/2563442954 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3bf40a7740 0x7f3bf408df80 secure :-1 s=READY pgs=1594 cs=0 l=1 rx=0x7f3bec008b10 tx=0x7f3bec0100b0).ready entity=mon.0 client_cookie=e308e320accf03e9 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:08:11.435 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.424+0000 7f3bf984d700 1 --2- 172.21.15.42:0/2563442954 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3bf40a7740 0x7f3bf408df80 secure :-1 s=THROTTLE_DONE pgs=1594 cs=0 l=1 rx=0x7f3bec008b10 tx=0x7f3bec0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:11.435 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.424+0000 7f3bf884b700 1 -- 172.21.15.42:0/2563442954 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3bec00a160 con 0x7f3bf40a7740 2019-12-05T06:08:11.435 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.424+0000 7f3bf984d700 1 --2- 172.21.15.42:0/2563442954 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3bf40a7740 0x7f3bf408df80 secure :-1 s=THROTTLE_DONE pgs=1594 cs=0 l=1 rx=0x7f3bec008b10 tx=0x7f3bec0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:11.436 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.424+0000 7f3bf984d700 1 --2- 172.21.15.42:0/2563442954 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3bf40a7740 0x7f3bf408df80 secure :-1 s=THROTTLE_DONE pgs=1594 cs=0 l=1 rx=0x7f3bec008b10 tx=0x7f3bec0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:11.436 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.424+0000 7f3bf884b700 1 -- 172.21.15.42:0/2563442954 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bec011a30 con 0x7f3bf40a7740 2019-12-05T06:08:11.436 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.424+0000 7f3bf884b700 1 -- 172.21.15.42:0/2563442954 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3bec00a550 con 0x7f3bf40a7740 2019-12-05T06:08:11.436 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.424+0000 7f3bfb2b0700 1 -- 172.21.15.42:0/2563442954 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3bf40a7740 msgr2=0x7f3bf408df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:11.437 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.424+0000 7f3bfb2b0700 1 --2- 172.21.15.42:0/2563442954 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3bf40a7740 0x7f3bf408df80 secure :-1 s=READY pgs=1594 cs=0 l=1 rx=0x7f3bec008b10 tx=0x7f3bec0100b0).stop 2019-12-05T06:08:11.437 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.424+0000 7f3bfb2b0700 1 -- 172.21.15.42:0/2563442954 shutdown_connections 2019-12-05T06:08:11.437 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.424+0000 7f3bfb2b0700 1 --2- 172.21.15.42:0/2563442954 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3bf40a7740 0x7f3bf408df80 unknown :-1 s=CLOSED pgs=1594 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:11.437 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.424+0000 7f3bfb2b0700 1 --2- 172.21.15.42:0/2563442954 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3bf407e720 0x7f3bf407eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:08:11.437 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.424+0000 7f3bfb2b0700 1 -- 172.21.15.42:0/2563442954 shutdown_connections 2019-12-05T06:08:11.437 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.424+0000 7f3bfb2b0700 1 -- 172.21.15.42:0/2563442954 wait complete. 2019-12-05T06:08:11.438 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.424+0000 7f3bfb2b0700 1 -- 172.21.15.42:0/2563442954 >> 172.21.15.42:0/2563442954 conn(0x7f3bf40aabe0 msgr2=0x7f3bf40a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:08:11.442 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.428+0000 7f3bfb2b0700 1 Processor -- start 2019-12-05T06:08:11.442 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.428+0000 7f3bfb2b0700 1 -- start start 2019-12-05T06:08:11.442 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.428+0000 7f3bfb2b0700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f3bf407b3f0 0x7f3bf4123690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:08:11.442 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.428+0000 7f3bfb2b0700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3bf4118980 0x7f3bf4120b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:08:11.442 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.428+0000 7f3bfb2b0700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bf4123be0 con 0x7f3bf41210c0 2019-12-05T06:08:11.443 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.428+0000 7f3bfb2b0700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3bf4117d40 con 0x7f3bf407b3f0 2019-12-05T06:08:11.443 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.428+0000 7f3bfb2b0700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bf4117250 con 0x7f3bf4118980 2019-12-05T06:08:11.443 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.428+0000 7f3bf904c700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3bf4118980 0x7f3bf4120b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:11.443 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.428+0000 7f3bf904c700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3bf4118980 0x7f3bf4120b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:49402/0 (socket says 172.21.15.42:49402) 2019-12-05T06:08:11.443 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.428+0000 7f3bf904c700 1 -- 172.21.15.42:0/1878536708 learned_addr learned my addr 172.21.15.42:0/1878536708 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:08:11.443 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.428+0000 7f3bf984d700 1 --2- 172.21.15.42:0/1878536708 >> v2:172.21.15.40:3300/0 conn(0x7f3bf407b3f0 0x7f3bf4123690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:11.444 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.428+0000 7f3bf904c700 1 -- 172.21.15.42:0/1878536708 >> v2:172.21.15.40:3300/0 conn(0x7f3bf407b3f0 msgr2=0x7f3bf4123690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:11.444 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.428+0000 7f3bf904c700 1 --2- 172.21.15.42:0/1878536708 >> v2:172.21.15.40:3300/0 conn(0x7f3bf407b3f0 0x7f3bf4123690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:11.444 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.428+0000 7f3bf904c700 1 -- 172.21.15.42:0/1878536708 >> v1:172.21.15.40:6789/0 conn(0x7f3bf41210c0 legacy=0x7f3bf408eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:11.444 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.428+0000 7f3bf904c700 1 -- 172.21.15.42:0/1878536708 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bf4116ed0 con 0x7f3bf4118980 2019-12-05T06:08:11.444 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.428+0000 7f3bf904c700 1 --2- 172.21.15.42:0/1878536708 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3bf4118980 0x7f3bf4120b70 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f3be8003070 tx=0x7f3be8010bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:11.444 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.428+0000 7f3bf904c700 1 --2- 172.21.15.42:0/1878536708 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3bf4118980 0x7f3bf4120b70 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f3be8003070 tx=0x7f3be8010bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:11.445 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.428+0000 7f3bf904c700 1 --2- 172.21.15.42:0/1878536708 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3bf4118980 0x7f3bf4120b70 secure :-1 s=READY pgs=460 cs=0 l=1 rx=0x7f3be8003070 tx=0x7f3be8010bc0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:08:11.445 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.428+0000 7f3bf904c700 1 --2- 172.21.15.42:0/1878536708 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3bf4118980 0x7f3bf4120b70 secure :-1 s=THROTTLE_DONE pgs=460 cs=0 l=1 rx=0x7f3be8003070 tx=0x7f3be8010bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:11.445 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.428+0000 7f3bf904c700 1 --2- 172.21.15.42:0/1878536708 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3bf4118980 0x7f3bf4120b70 secure :-1 s=THROTTLE_DONE pgs=460 cs=0 l=1 rx=0x7f3be8003070 tx=0x7f3be8010bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:11.445 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.428+0000 7f3bf27fc700 1 -- 172.21.15.42:0/1878536708 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3be8005840 con 0x7f3bf4118980 2019-12-05T06:08:11.445 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.428+0000 7f3bf904c700 1 --2- 172.21.15.42:0/1878536708 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3bf4118980 0x7f3bf4120b70 secure :-1 s=THROTTLE_DONE pgs=460 cs=0 l=1 rx=0x7f3be8003070 tx=0x7f3be8010bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:11.445 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.428+0000 7f3bf27fc700 1 -- 172.21.15.42:0/1878536708 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3be8007160 con 0x7f3bf4118980 2019-12-05T06:08:11.445 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.428+0000 7f3bfb2b0700 1 -- 172.21.15.42:0/1878536708 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bf411a500 con 0x7f3bf4118980 2019-12-05T06:08:11.446 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.428+0000 7f3bf27fc700 1 -- 172.21.15.42:0/1878536708 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3be8012240 con 0x7f3bf4118980 2019-12-05T06:08:11.446 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.432+0000 7f3bfb2b0700 1 -- 172.21.15.42:0/1878536708 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bf4119d50 con 0x7f3bf4118980 2019-12-05T06:08:11.446 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.432+0000 7f3bf904c700 1 --2- 172.21.15.42:0/1878536708 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3bf4118980 0x7f3bf4120b70 secure :-1 s=THROTTLE_DONE pgs=460 cs=0 l=1 rx=0x7f3be8003070 tx=0x7f3be8010bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:11.446 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.432+0000 7f3bf27fc700 1 -- 172.21.15.42:0/1878536708 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f3be8012550 con 0x7f3bf4118980 2019-12-05T06:08:11.446 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.432+0000 7f3bf904c700 1 --2- 172.21.15.42:0/1878536708 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3bf4118980 0x7f3bf4120b70 secure :-1 s=THROTTLE_DONE pgs=460 cs=0 l=1 rx=0x7f3be8003070 tx=0x7f3be8010bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:11.446 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.432+0000 7f3bf27fc700 1 --2- 172.21.15.42:0/1878536708 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3bd403ae40 0x7f3bd403d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:08:11.447 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.432+0000 7f3bf27fc700 1 -- 172.21.15.42:0/1878536708 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9901+0+0 (secure 0 0 0) 0x7f3be804b1a0 con 0x7f3bf4118980 2019-12-05T06:08:11.447 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.432+0000 7f3bf984d700 1 --2- 172.21.15.42:0/1878536708 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3bd403ae40 0x7f3bd403d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:11.447 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.432+0000 7f3bf984d700 1 --2- 172.21.15.42:0/1878536708 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3bd403ae40 0x7f3bd403d350 crc :-1 s=READY pgs=888 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:08:11.447 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.436+0000 7f3bfb2b0700 1 -- 172.21.15.42:0/1878536708 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bf405d110 con 0x7f3bf4118980 2019-12-05T06:08:11.451 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.444+0000 7f3bf904c700 1 --2- 172.21.15.42:0/1878536708 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3bf4118980 0x7f3bf4120b70 secure :-1 s=THROTTLE_DONE pgs=460 cs=0 l=1 rx=0x7f3be8003070 tx=0x7f3be8010bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:11.451 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.444+0000 7f3bf27fc700 1 -- 172.21.15.42:0/1878536708 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f3be80321c0 con 0x7f3bf4118980 2019-12-05T06:08:11.611 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.604+0000 7f3bfb2b0700 1 -- 172.21.15.42:0/1878536708 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3bf405b8c0 con 0x7f3bf4118980 2019-12-05T06:08:11.612 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.608+0000 7f3bf904c700 1 --2- 172.21.15.42:0/1878536708 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3bf4118980 0x7f3bf4120b70 secure :-1 s=THROTTLE_DONE pgs=460 cs=0 l=1 rx=0x7f3be8003070 tx=0x7f3be8010bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:11.612 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.608+0000 7f3bf27fc700 1 -- 172.21.15.42:0/1878536708 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 254 v254) v1 ==== 95+0+1774 (secure 0 0 0) 0x7f3be8032420 con 0x7f3bf4118980 2019-12-05T06:08:11.614 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 254 2019-12-05T06:08:11.614 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:08:11.614 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":254,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":254,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:08:09.493328+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":94,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6929},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6929":{"gid":6929,"name":"a","rank":0,"incarnation":254,"state":"up:replay","state_seq":2,"addr":"172.21.15.42:6835/2636699468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2636699468},{"type":"v1","addr":"172.21.15.42:6835","nonce":2636699468}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:08:11.617 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.612+0000 7f3bfb2b0700 1 -- 172.21.15.42:0/1878536708 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3bd403ae40 msgr2=0x7f3bd403d350 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:11.617 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.612+0000 7f3bfb2b0700 1 --2- 172.21.15.42:0/1878536708 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3bd403ae40 0x7f3bd403d350 crc :-1 s=READY pgs=888 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:11.617 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.612+0000 7f3bfb2b0700 1 -- 172.21.15.42:0/1878536708 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3bf4118980 msgr2=0x7f3bf4120b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:11.617 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.612+0000 7f3bfb2b0700 1 --2- 172.21.15.42:0/1878536708 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3bf4118980 0x7f3bf4120b70 secure :-1 s=READY pgs=460 cs=0 l=1 rx=0x7f3be8003070 tx=0x7f3be8010bc0).stop 2019-12-05T06:08:11.617 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.612+0000 7f3bfb2b0700 1 -- 172.21.15.42:0/1878536708 shutdown_connections 2019-12-05T06:08:11.617 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.612+0000 7f3bfb2b0700 1 --2- 172.21.15.42:0/1878536708 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3bd403ae40 0x7f3bd403d350 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:11.618 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.612+0000 7f3bfb2b0700 1 --2- 172.21.15.42:0/1878536708 >> v2:172.21.15.40:3300/0 conn(0x7f3bf407b3f0 0x7f3bf4123690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:11.618 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.612+0000 7f3bfb2b0700 1 --2- 172.21.15.42:0/1878536708 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3bf4118980 0x7f3bf4120b70 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:11.618 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.612+0000 7f3bfb2b0700 1 -- 172.21.15.42:0/1878536708 shutdown_connections 2019-12-05T06:08:11.618 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.612+0000 7f3bfb2b0700 1 -- 172.21.15.42:0/1878536708 wait complete. 2019-12-05T06:08:11.618 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:11.612+0000 7f3bfb2b0700 1 -- 172.21.15.42:0/1878536708 >> 172.21.15.42:0/1878536708 conn(0x7f3bf40aabe0 msgr2=0x7f3bf409f460 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:08:11.626 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 254, "feature_flags": { "enable_multiple": false, "ever_enabled_multiple": false }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2019-12-05T05:03:35.604738+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 254, "ever_allowed_features": 2, "explicitly_allowed_features": 2, "failed": [], "flags": 18, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_6929": { "addr": "172.21.15.42:6835/2636699468", "addrs": { "addrvec": [ { "addr": "172.21.15.42:6834", "nonce": 2636699468, "type": "v2" }, { "addr": "172.21.15.42:6835", "nonce": 2636699468, "type": "v1" } ] }, "export_targets": [], "features": 4540138292836696063, "flags": 0, "gid": 6929, "incarnation": 254, "name": "a", "rank": 0, "state": "up:replay", "state_seq": 2 } }, "last_failure": 0, "last_failure_osd_epoch": 94, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 2, "min_compat_client": "0 (unknown)", "modified": "2019-12-05T06:08:09.493328+0000", "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6929 } } } ], "standbys": [] } 2019-12-05T06:08:13.633 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:08:13.745 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:13.736+0000 7f0121131700 1 Processor -- start 2019-12-05T06:08:13.746 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:13.736+0000 7f0121131700 1 -- start start 2019-12-05T06:08:13.746 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:13.736+0000 7f0121131700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f011c0a7740 0x7f011c08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:08:13.746 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:13.736+0000 7f0121131700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f011c07e720 0x7f011c07eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:08:13.746 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:13.736+0000 7f0121131700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f011c0a0710 con 0x7f011c09f2e0 2019-12-05T06:08:13.746 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:13.736+0000 7f0121131700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f011c0a0b00 con 0x7f011c07e720 2019-12-05T06:08:13.746 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:13.736+0000 7f0121131700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f011c08e5b0 con 0x7f011c0a7740 2019-12-05T06:08:13.747 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:13.736+0000 7f011ad9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f011c07e720 0x7f011c07eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:13.747 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:13.736+0000 7f011ad9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f011c07e720 0x7f011c07eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:48952/0 (socket says 172.21.15.42:48952) 2019-12-05T06:08:13.747 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:13.736+0000 7f011ad9d700 1 -- 172.21.15.42:0/1192873534 learned_addr learned my addr 172.21.15.42:0/1192873534 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:08:13.747 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:13.736+0000 7f011b59e700 1 --2- 172.21.15.42:0/1192873534 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f011c0a7740 0x7f011c08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:13.747 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:13.736+0000 7f011ad9d700 1 -- 172.21.15.42:0/1192873534 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f011c0a7740 msgr2=0x7f011c08df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:08:13.748 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:13.736+0000 7f011ad9d700 1 --2- 172.21.15.42:0/1192873534 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f011c0a7740 0x7f011c08df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:08:13.748 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:13.736+0000 7f011ad9d700 1 -- 172.21.15.42:0/1192873534 >> v1:172.21.15.40:6789/0 conn(0x7f011c09f2e0 legacy=0x7f011c09f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:13.748 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:13.736+0000 7f011ad9d700 1 -- 172.21.15.42:0/1192873534 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f011c07f0a0 con 0x7f011c07e720 2019-12-05T06:08:13.748 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:13.736+0000 7f011ad9d700 1 --2- 172.21.15.42:0/1192873534 >> v2:172.21.15.42:3300/0 conn(0x7f011c07e720 0x7f011c07eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f0104008790 tx=0x7f010400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:13.748 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:13.736+0000 7f011ad9d700 1 --2- 172.21.15.42:0/1192873534 >> v2:172.21.15.42:3300/0 conn(0x7f011c07e720 0x7f011c07eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f0104008790 tx=0x7f010400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:13.751 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:13.736+0000 7f011ad9d700 1 --2- 172.21.15.42:0/1192873534 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f011c07e720 0x7f011c07eb50 secure :-1 s=READY pgs=2107 cs=0 l=1 rx=0x7f0104008790 tx=0x7f010400fd30).ready entity=mon.1 client_cookie=ae73bec828183df2 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:08:13.751 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:13.736+0000 7f011ad9d700 1 --2- 172.21.15.42:0/1192873534 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f011c07e720 0x7f011c07eb50 secure :-1 s=THROTTLE_DONE pgs=2107 cs=0 l=1 rx=0x7f0104008790 tx=0x7f010400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:13.751 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:13.736+0000 7f011ad9d700 1 --2- 172.21.15.42:0/1192873534 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f011c07e720 0x7f011c07eb50 secure :-1 s=THROTTLE_DONE pgs=2107 cs=0 l=1 rx=0x7f0104008790 tx=0x7f010400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:13.752 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:13.736+0000 7f011ad9d700 1 --2- 172.21.15.42:0/1192873534 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f011c07e720 0x7f011c07eb50 secure :-1 s=THROTTLE_DONE pgs=2107 cs=0 l=1 rx=0x7f0104008790 tx=0x7f010400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:13.752 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:13.736+0000 7f011a59c700 1 -- 172.21.15.42:0/1192873534 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f01040113b0 con 0x7f011c07e720 2019-12-05T06:08:13.752 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:13.736+0000 7f011a59c700 1 -- 172.21.15.42:0/1192873534 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0104008bb0 con 0x7f011c07e720 2019-12-05T06:08:13.752 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:13.736+0000 7f011a59c700 1 -- 172.21.15.42:0/1192873534 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0104011990 con 0x7f011c07e720 2019-12-05T06:08:13.752 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:13.736+0000 7f0121131700 1 -- 172.21.15.42:0/1192873534 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f011c07e720 msgr2=0x7f011c07eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:13.752 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:13.736+0000 7f0121131700 1 --2- 172.21.15.42:0/1192873534 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f011c07e720 0x7f011c07eb50 secure :-1 s=READY pgs=2107 cs=0 l=1 rx=0x7f0104008790 tx=0x7f010400fd30).stop 2019-12-05T06:08:13.752 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:13.740+0000 7f0121131700 1 -- 172.21.15.42:0/1192873534 shutdown_connections 2019-12-05T06:08:13.753 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:13.740+0000 7f0121131700 1 --2- 172.21.15.42:0/1192873534 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f011c0a7740 0x7f011c08df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:08:13.753 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:13.740+0000 7f0121131700 1 --2- 172.21.15.42:0/1192873534 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f011c07e720 0x7f011c07eb50 unknown :-1 s=CLOSED pgs=2107 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:13.753 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:13.740+0000 7f0121131700 1 -- 172.21.15.42:0/1192873534 shutdown_connections 2019-12-05T06:08:13.753 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:13.740+0000 7f0121131700 1 -- 172.21.15.42:0/1192873534 wait complete. 2019-12-05T06:08:13.753 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:13.740+0000 7f0121131700 1 -- 172.21.15.42:0/1192873534 >> 172.21.15.42:0/1192873534 conn(0x7f011c0aabe0 msgr2=0x7f011c0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:08:13.757 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:13.740+0000 7f0121131700 1 Processor -- start 2019-12-05T06:08:13.757 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:13.740+0000 7f0121131700 1 -- start start 2019-12-05T06:08:13.757 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:13.744+0000 7f0121131700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f011c136940 0x7f011c138d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:08:13.757 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:13.744+0000 7f0121131700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f011c11f0d0 con 0x7f011c12cf30 2019-12-05T06:08:13.757 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:13.744+0000 7f0121131700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f011c124940 con 0x7f011c124510 2019-12-05T06:08:13.758 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:13.744+0000 7f0121131700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f011c1304f0 con 0x7f011c136940 2019-12-05T06:08:13.758 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:13.744+0000 7f011ad9d700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f011c124510 0x7f011c130c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:48992/0 (socket says 172.21.15.42:48992) 2019-12-05T06:08:13.759 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:13.744+0000 7f011ad9d700 1 -- 172.21.15.42:0/3140407055 learned_addr learned my addr 172.21.15.42:0/3140407055 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:08:13.759 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:13.744+0000 7f0120930700 1 --2- 172.21.15.42:0/3140407055 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f011c136940 0x7f011c138d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:13.759 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:13.744+0000 7f0113fff700 1 -- 172.21.15.42:0/3140407055 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3321292501 0 0) 0x7f011c124940 con 0x7f011c124510 2019-12-05T06:08:13.760 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:13.744+0000 7f0113fff700 1 -- 172.21.15.42:0/3140407055 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01000030b0 con 0x7f011c124510 2019-12-05T06:08:13.767 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:13.744+0000 7f0113fff700 1 -- 172.21.15.42:0/3140407055 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 214069966 0 0) 0x7f011c11f0d0 con 0x7f011c12cf30 2019-12-05T06:08:13.767 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:13.744+0000 7f0113fff700 1 -- 172.21.15.42:0/3140407055 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f011c124940 con 0x7f011c12cf30 2019-12-05T06:08:13.767 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:13.744+0000 7f0120930700 1 -- 172.21.15.42:0/3140407055 >> v1:172.21.15.42:6789/0 conn(0x7f011c124510 legacy=0x7f011c130c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:13.767 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:13.744+0000 7f0120930700 1 -- 172.21.15.42:0/3140407055 >> v1:172.21.15.40:6789/0 conn(0x7f011c12cf30 legacy=0x7f011c07d180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:13.767 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:13.744+0000 7f0120930700 1 -- 172.21.15.42:0/3140407055 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f011c12dca0 con 0x7f011c136940 2019-12-05T06:08:13.768 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:13.744+0000 7f0120930700 1 --2- 172.21.15.42:0/3140407055 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f011c136940 0x7f011c138d80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f0114001210 tx=0x7f0114010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:13.768 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:13.744+0000 7f0120930700 1 --2- 172.21.15.42:0/3140407055 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f011c136940 0x7f011c138d80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f0114001210 tx=0x7f0114010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:13.768 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:13.744+0000 7f0120930700 1 --2- 172.21.15.42:0/3140407055 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f011c136940 0x7f011c138d80 secure :-1 s=READY pgs=461 cs=0 l=1 rx=0x7f0114001210 tx=0x7f0114010ac0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:08:13.768 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:13.744+0000 7f0120930700 1 --2- 172.21.15.42:0/3140407055 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f011c136940 0x7f011c138d80 secure :-1 s=THROTTLE_DONE pgs=461 cs=0 l=1 rx=0x7f0114001210 tx=0x7f0114010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:13.768 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:13.744+0000 7f0113fff700 1 -- 172.21.15.42:0/3140407055 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0114006ac0 con 0x7f011c136940 2019-12-05T06:08:13.769 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:13.744+0000 7f0120930700 1 --2- 172.21.15.42:0/3140407055 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f011c136940 0x7f011c138d80 secure :-1 s=THROTTLE_DONE pgs=461 cs=0 l=1 rx=0x7f0114001210 tx=0x7f0114010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:13.769 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:13.744+0000 7f0120930700 1 --2- 172.21.15.42:0/3140407055 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f011c136940 0x7f011c138d80 secure :-1 s=THROTTLE_DONE pgs=461 cs=0 l=1 rx=0x7f0114001210 tx=0x7f0114010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:13.769 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:13.744+0000 7f0113fff700 1 -- 172.21.15.42:0/3140407055 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01140066e0 con 0x7f011c136940 2019-12-05T06:08:13.769 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:13.744+0000 7f0121131700 1 -- 172.21.15.42:0/3140407055 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f011c1392d0 con 0x7f011c136940 2019-12-05T06:08:13.772 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:13.744+0000 7f0113fff700 1 -- 172.21.15.42:0/3140407055 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0114002820 con 0x7f011c136940 2019-12-05T06:08:13.772 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:13.744+0000 7f0121131700 1 -- 172.21.15.42:0/3140407055 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f011c139880 con 0x7f011c136940 2019-12-05T06:08:13.772 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:13.748+0000 7f0120930700 1 --2- 172.21.15.42:0/3140407055 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f011c136940 0x7f011c138d80 secure :-1 s=THROTTLE_DONE pgs=461 cs=0 l=1 rx=0x7f0114001210 tx=0x7f0114010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:13.772 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:13.748+0000 7f0113fff700 1 -- 172.21.15.42:0/3140407055 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f0114002af0 con 0x7f011c136940 2019-12-05T06:08:13.772 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:13.748+0000 7f0120930700 1 --2- 172.21.15.42:0/3140407055 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f011c136940 0x7f011c138d80 secure :-1 s=THROTTLE_DONE pgs=461 cs=0 l=1 rx=0x7f0114001210 tx=0x7f0114010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:13.773 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:13.748+0000 7f0113fff700 1 --2- 172.21.15.42:0/3140407055 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f010003b4b0 0x7f010003d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:08:13.773 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:13.748+0000 7f0113fff700 1 -- 172.21.15.42:0/3140407055 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9901+0+0 (secure 0 0 0) 0x7f011404b0a0 con 0x7f011c136940 2019-12-05T06:08:13.773 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:13.748+0000 7f011b59e700 1 --2- 172.21.15.42:0/3140407055 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f010003b4b0 0x7f010003d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:13.773 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:13.748+0000 7f011b59e700 1 --2- 172.21.15.42:0/3140407055 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f010003b4b0 0x7f010003d9c0 crc :-1 s=READY pgs=889 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:08:13.773 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:13.752+0000 7f0121131700 1 -- 172.21.15.42:0/3140407055 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f011c05d110 con 0x7f011c136940 2019-12-05T06:08:13.774 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:13.760+0000 7f0120930700 1 --2- 172.21.15.42:0/3140407055 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f011c136940 0x7f011c138d80 secure :-1 s=THROTTLE_DONE pgs=461 cs=0 l=1 rx=0x7f0114001210 tx=0x7f0114010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:13.775 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:13.760+0000 7f0113fff700 1 -- 172.21.15.42:0/3140407055 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f0114032120 con 0x7f011c136940 2019-12-05T06:08:13.938 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:13.932+0000 7f0121131700 1 -- 172.21.15.42:0/3140407055 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f011c0a71b0 con 0x7f011c136940 2019-12-05T06:08:13.939 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:13.932+0000 7f0120930700 1 --2- 172.21.15.42:0/3140407055 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f011c136940 0x7f011c138d80 secure :-1 s=THROTTLE_DONE pgs=461 cs=0 l=1 rx=0x7f0114001210 tx=0x7f0114010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:13.939 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:13.932+0000 7f0113fff700 1 -- 172.21.15.42:0/3140407055 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 256 v256) v1 ==== 95+0+1775 (secure 0 0 0) 0x7f0114032380 con 0x7f011c136940 2019-12-05T06:08:13.939 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 256 2019-12-05T06:08:13.939 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:08:13.940 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":256,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":256,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:08:13.887768+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":94,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6929},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6929":{"gid":6929,"name":"a","rank":0,"incarnation":254,"state":"up:rejoin","state_seq":18,"addr":"172.21.15.42:6835/2636699468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2636699468},{"type":"v1","addr":"172.21.15.42:6835","nonce":2636699468}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:08:13.942 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:13.936+0000 7f0121131700 1 -- 172.21.15.42:0/3140407055 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f010003b4b0 msgr2=0x7f010003d9c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:13.942 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:13.936+0000 7f0121131700 1 --2- 172.21.15.42:0/3140407055 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f010003b4b0 0x7f010003d9c0 crc :-1 s=READY pgs=889 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:13.943 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:13.936+0000 7f0121131700 1 -- 172.21.15.42:0/3140407055 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f011c136940 msgr2=0x7f011c138d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:13.943 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:13.936+0000 7f0121131700 1 --2- 172.21.15.42:0/3140407055 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f011c136940 0x7f011c138d80 secure :-1 s=READY pgs=461 cs=0 l=1 rx=0x7f0114001210 tx=0x7f0114010ac0).stop 2019-12-05T06:08:13.943 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:13.936+0000 7f0121131700 1 -- 172.21.15.42:0/3140407055 shutdown_connections 2019-12-05T06:08:13.943 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:13.936+0000 7f0121131700 1 --2- 172.21.15.42:0/3140407055 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f010003b4b0 0x7f010003d9c0 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:13.943 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:13.936+0000 7f0121131700 1 --2- 172.21.15.42:0/3140407055 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f011c136940 0x7f011c138d80 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:13.943 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:13.936+0000 7f0121131700 1 -- 172.21.15.42:0/3140407055 shutdown_connections 2019-12-05T06:08:13.943 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:13.936+0000 7f0121131700 1 -- 172.21.15.42:0/3140407055 wait complete. 2019-12-05T06:08:13.944 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:13.936+0000 7f0121131700 1 -- 172.21.15.42:0/3140407055 >> 172.21.15.42:0/3140407055 conn(0x7f011c0a9870 msgr2=0x7f011c11c610 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:08:13.949 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2019-12-05T06:08:13.965 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:08:13.969 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:08:15.950 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:08:16.066 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:16.056+0000 7f8e16663700 1 Processor -- start 2019-12-05T06:08:16.066 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:16.056+0000 7f8e16663700 1 -- start start 2019-12-05T06:08:16.066 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:16.056+0000 7f8e16663700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8e1009fae0 0x7f8e1014fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:08:16.066 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:16.056+0000 7f8e16663700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e100a0710 con 0x7f8e100a7740 2019-12-05T06:08:16.066 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:16.056+0000 7f8e16663700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e1007ef20 con 0x7f8e1008df80 2019-12-05T06:08:16.067 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:16.056+0000 7f8e16663700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e100a0b00 con 0x7f8e1009fae0 2019-12-05T06:08:16.067 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:16.056+0000 7f8e0ffff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f8e1008df80 0x7f8e1009f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:49008/0 (socket says 172.21.15.42:49008) 2019-12-05T06:08:16.067 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:16.056+0000 7f8e0ffff700 1 -- 172.21.15.42:0/2701358620 learned_addr learned my addr 172.21.15.42:0/2701358620 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:08:16.067 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:16.056+0000 7f8e15401700 1 --2- 172.21.15.42:0/2701358620 >> v2:172.21.15.40:3300/0 conn(0x7f8e1009fae0 0x7f8e1014fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:16.067 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:16.056+0000 7f8e0f7fe700 1 -- 172.21.15.42:0/2701358620 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3728790747 0 0) 0x7f8e1007ef20 con 0x7f8e1008df80 2019-12-05T06:08:16.067 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:16.056+0000 7f8e0f7fe700 1 -- 172.21.15.42:0/2701358620 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e000030b0 con 0x7f8e1008df80 2019-12-05T06:08:16.067 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:16.056+0000 7f8e15401700 1 -- 172.21.15.42:0/2701358620 >> v1:172.21.15.42:6789/0 conn(0x7f8e1008df80 legacy=0x7f8e1009f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:16.068 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:16.056+0000 7f8e15401700 1 -- 172.21.15.42:0/2701358620 >> v1:172.21.15.40:6789/0 conn(0x7f8e100a7740 legacy=0x7f8e1007e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:16.068 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:16.056+0000 7f8e15401700 1 -- 172.21.15.42:0/2701358620 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e100a9ba0 con 0x7f8e1009fae0 2019-12-05T06:08:16.068 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:16.056+0000 7f8e15401700 1 --2- 172.21.15.42:0/2701358620 >> v2:172.21.15.40:3300/0 conn(0x7f8e1009fae0 0x7f8e1014fff0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f8e08008b10 tx=0x7f8e0800ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:16.068 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:16.060+0000 7f8e15401700 1 --2- 172.21.15.42:0/2701358620 >> v2:172.21.15.40:3300/0 conn(0x7f8e1009fae0 0x7f8e1014fff0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f8e08008b10 tx=0x7f8e0800ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:16.068 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:16.060+0000 7f8e15401700 1 --2- 172.21.15.42:0/2701358620 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e1009fae0 0x7f8e1014fff0 secure :-1 s=READY pgs=1600 cs=0 l=1 rx=0x7f8e08008b10 tx=0x7f8e0800ff70).ready entity=mon.0 client_cookie=8e7cdaf11422fa5d server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:08:16.068 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:16.060+0000 7f8e15401700 1 --2- 172.21.15.42:0/2701358620 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e1009fae0 0x7f8e1014fff0 secure :-1 s=THROTTLE_DONE pgs=1600 cs=0 l=1 rx=0x7f8e08008b10 tx=0x7f8e0800ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:16.069 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:16.060+0000 7f8e0f7fe700 1 -- 172.21.15.42:0/2701358620 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8e08010970 con 0x7f8e1009fae0 2019-12-05T06:08:16.069 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:16.060+0000 7f8e15401700 1 --2- 172.21.15.42:0/2701358620 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e1009fae0 0x7f8e1014fff0 secure :-1 s=THROTTLE_DONE pgs=1600 cs=0 l=1 rx=0x7f8e08008b10 tx=0x7f8e0800ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:16.069 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:16.060+0000 7f8e0f7fe700 1 -- 172.21.15.42:0/2701358620 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e08008b50 con 0x7f8e1009fae0 2019-12-05T06:08:16.069 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:16.060+0000 7f8e15401700 1 --2- 172.21.15.42:0/2701358620 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e1009fae0 0x7f8e1014fff0 secure :-1 s=THROTTLE_DONE pgs=1600 cs=0 l=1 rx=0x7f8e08008b10 tx=0x7f8e0800ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:16.069 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:16.060+0000 7f8e0f7fe700 1 -- 172.21.15.42:0/2701358620 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8e080110c0 con 0x7f8e1009fae0 2019-12-05T06:08:16.069 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:16.060+0000 7f8e16663700 1 -- 172.21.15.42:0/2701358620 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e1009fae0 msgr2=0x7f8e1014fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:16.069 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:16.060+0000 7f8e16663700 1 --2- 172.21.15.42:0/2701358620 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e1009fae0 0x7f8e1014fff0 secure :-1 s=READY pgs=1600 cs=0 l=1 rx=0x7f8e08008b10 tx=0x7f8e0800ff70).stop 2019-12-05T06:08:16.070 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:16.060+0000 7f8e16663700 1 -- 172.21.15.42:0/2701358620 shutdown_connections 2019-12-05T06:08:16.070 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:16.060+0000 7f8e16663700 1 --2- 172.21.15.42:0/2701358620 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e1009fae0 0x7f8e1014fff0 unknown :-1 s=CLOSED pgs=1600 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:16.070 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:16.060+0000 7f8e16663700 1 -- 172.21.15.42:0/2701358620 shutdown_connections 2019-12-05T06:08:16.070 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:16.060+0000 7f8e16663700 1 -- 172.21.15.42:0/2701358620 wait complete. 2019-12-05T06:08:16.070 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:16.060+0000 7f8e16663700 1 -- 172.21.15.42:0/2701358620 >> 172.21.15.42:0/2701358620 conn(0x7f8e100aabe0 msgr2=0x7f8e100a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:08:16.074 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:16.060+0000 7f8e16663700 1 Processor -- start 2019-12-05T06:08:16.074 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:16.060+0000 7f8e16663700 1 -- start start 2019-12-05T06:08:16.074 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:16.064+0000 7f8e16663700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8e100a6210 0x7f8e100a6640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:08:16.075 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:16.064+0000 7f8e16663700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e1011a560 con 0x7f8e10125cc0 2019-12-05T06:08:16.075 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:16.064+0000 7f8e16663700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e1007c690 con 0x7f8e1014fd80 2019-12-05T06:08:16.075 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:16.064+0000 7f8e16663700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e1007c150 con 0x7f8e100a6210 2019-12-05T06:08:16.076 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:16.064+0000 7f8e14c00700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f8e1014fd80 0x7f8e1007ca80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:49012/0 (socket says 172.21.15.42:49012) 2019-12-05T06:08:16.076 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:16.064+0000 7f8e14c00700 1 -- 172.21.15.42:0/777746559 learned_addr learned my addr 172.21.15.42:0/777746559 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:08:16.076 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:16.064+0000 7f8e0ffff700 1 --2- 172.21.15.42:0/777746559 >> v2:172.21.15.40:3300/0 conn(0x7f8e100a6210 0x7f8e100a6640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:16.076 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:16.064+0000 7f8e0d7fa700 1 -- 172.21.15.42:0/777746559 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3479705447 0 0) 0x7f8e1007c690 con 0x7f8e1014fd80 2019-12-05T06:08:16.076 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:16.064+0000 7f8e0d7fa700 1 -- 172.21.15.42:0/777746559 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8df00030b0 con 0x7f8e1014fd80 2019-12-05T06:08:16.076 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:16.064+0000 7f8e0ffff700 1 -- 172.21.15.42:0/777746559 >> v1:172.21.15.42:6789/0 conn(0x7f8e1014fd80 legacy=0x7f8e1007ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:16.077 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:16.064+0000 7f8e0d7fa700 1 -- 172.21.15.42:0/777746559 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 2584105087 0 0) 0x7f8df00030b0 con 0x7f8e1014fd80 2019-12-05T06:08:16.077 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:16.064+0000 7f8e0ffff700 1 -- 172.21.15.42:0/777746559 >> v1:172.21.15.40:6789/0 conn(0x7f8e10125cc0 legacy=0x7f8e101260f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:16.077 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:16.064+0000 7f8e0ffff700 1 -- 172.21.15.42:0/777746559 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e101268f0 con 0x7f8e100a6210 2019-12-05T06:08:16.077 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:16.064+0000 7f8e0ffff700 1 --2- 172.21.15.42:0/777746559 >> v2:172.21.15.40:3300/0 conn(0x7f8e100a6210 0x7f8e100a6640 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f8e08010380 tx=0x7f8e0800ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:16.077 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:16.064+0000 7f8e0ffff700 1 --2- 172.21.15.42:0/777746559 >> v2:172.21.15.40:3300/0 conn(0x7f8e100a6210 0x7f8e100a6640 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f8e08010380 tx=0x7f8e0800ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:16.077 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:16.064+0000 7f8e0ffff700 1 --2- 172.21.15.42:0/777746559 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e100a6210 0x7f8e100a6640 secure :-1 s=READY pgs=1602 cs=0 l=1 rx=0x7f8e08010380 tx=0x7f8e0800ff70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:08:16.078 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:16.064+0000 7f8e0ffff700 1 --2- 172.21.15.42:0/777746559 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e100a6210 0x7f8e100a6640 secure :-1 s=THROTTLE_DONE pgs=1602 cs=0 l=1 rx=0x7f8e08010380 tx=0x7f8e0800ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:16.078 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:16.064+0000 7f8e0d7fa700 1 -- 172.21.15.42:0/777746559 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8e04001a60 con 0x7f8e100a6210 2019-12-05T06:08:16.078 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:16.064+0000 7f8e0ffff700 1 --2- 172.21.15.42:0/777746559 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e100a6210 0x7f8e100a6640 secure :-1 s=THROTTLE_DONE pgs=1602 cs=0 l=1 rx=0x7f8e08010380 tx=0x7f8e0800ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:16.078 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:16.064+0000 7f8e0ffff700 1 --2- 172.21.15.42:0/777746559 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e100a6210 0x7f8e100a6640 secure :-1 s=THROTTLE_DONE pgs=1602 cs=0 l=1 rx=0x7f8e08010380 tx=0x7f8e0800ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:16.078 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:16.064+0000 7f8e0d7fa700 1 -- 172.21.15.42:0/777746559 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e04001870 con 0x7f8e100a6210 2019-12-05T06:08:16.078 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:16.064+0000 7f8e0d7fa700 1 -- 172.21.15.42:0/777746559 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8dfc0060c0 con 0x7f8e100a6210 2019-12-05T06:08:16.078 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:16.064+0000 7f8e16663700 1 -- 172.21.15.42:0/777746559 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e10131610 con 0x7f8e100a6210 2019-12-05T06:08:16.079 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:16.064+0000 7f8e16663700 1 -- 172.21.15.42:0/777746559 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e10131970 con 0x7f8e100a6210 2019-12-05T06:08:16.079 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:16.064+0000 7f8e0ffff700 1 --2- 172.21.15.42:0/777746559 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e100a6210 0x7f8e100a6640 secure :-1 s=THROTTLE_DONE pgs=1602 cs=0 l=1 rx=0x7f8e08010380 tx=0x7f8e0800ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:16.079 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:16.068+0000 7f8e0d7fa700 1 -- 172.21.15.42:0/777746559 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f8dfc006390 con 0x7f8e100a6210 2019-12-05T06:08:16.079 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:16.068+0000 7f8e0ffff700 1 --2- 172.21.15.42:0/777746559 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e100a6210 0x7f8e100a6640 secure :-1 s=THROTTLE_DONE pgs=1602 cs=0 l=1 rx=0x7f8e08010380 tx=0x7f8e0800ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:16.079 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:16.068+0000 7f8e0d7fa700 1 --2- 172.21.15.42:0/777746559 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8df003b280 0x7f8df003d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:08:16.079 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:16.068+0000 7f8e0d7fa700 1 -- 172.21.15.42:0/777746559 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9901+0+0 (secure 0 0 0) 0x7f8dfc04aa70 con 0x7f8e100a6210 2019-12-05T06:08:16.079 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:16.068+0000 7f8e14c00700 1 --2- 172.21.15.42:0/777746559 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8df003b280 0x7f8df003d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:16.080 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:16.068+0000 7f8e14c00700 1 --2- 172.21.15.42:0/777746559 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8df003b280 0x7f8df003d790 crc :-1 s=READY pgs=890 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:08:16.080 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:16.072+0000 7f8e16663700 1 -- 172.21.15.42:0/777746559 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e1005d110 con 0x7f8e100a6210 2019-12-05T06:08:16.086 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:16.080+0000 7f8e0ffff700 1 --2- 172.21.15.42:0/777746559 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e100a6210 0x7f8e100a6640 secure :-1 s=THROTTLE_DONE pgs=1602 cs=0 l=1 rx=0x7f8e08010380 tx=0x7f8e0800ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:16.086 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:16.080+0000 7f8e0d7fa700 1 -- 172.21.15.42:0/777746559 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f8dfc031af0 con 0x7f8e100a6210 2019-12-05T06:08:16.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:16.284+0000 7f8e16663700 1 -- 172.21.15.42:0/777746559 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8e100a90e0 con 0x7f8e100a6210 2019-12-05T06:08:16.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:16.284+0000 7f8e0ffff700 1 --2- 172.21.15.42:0/777746559 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e100a6210 0x7f8e100a6640 secure :-1 s=THROTTLE_DONE pgs=1602 cs=0 l=1 rx=0x7f8e08010380 tx=0x7f8e0800ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:16.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:16.284+0000 7f8e0d7fa700 1 -- 172.21.15.42:0/777746559 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 257 v257) v1 ==== 95+0+1775 (secure 0 0 0) 0x7f8dfc031d50 con 0x7f8e100a6210 2019-12-05T06:08:16.290 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 257 2019-12-05T06:08:16.291 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:08:16.291 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":257,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":257,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:08:14.898258+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":94,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6929},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6929":{"gid":6929,"name":"a","rank":0,"incarnation":254,"state":"up:active","state_seq":19,"addr":"172.21.15.42:6835/2636699468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2636699468},{"type":"v1","addr":"172.21.15.42:6835","nonce":2636699468}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:08:16.292 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:16.284+0000 7f8e16663700 1 -- 172.21.15.42:0/777746559 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8df003b280 msgr2=0x7f8df003d790 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:16.292 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:16.284+0000 7f8e16663700 1 --2- 172.21.15.42:0/777746559 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8df003b280 0x7f8df003d790 crc :-1 s=READY pgs=890 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:16.292 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:16.284+0000 7f8e16663700 1 -- 172.21.15.42:0/777746559 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e100a6210 msgr2=0x7f8e100a6640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:16.292 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:16.284+0000 7f8e16663700 1 --2- 172.21.15.42:0/777746559 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e100a6210 0x7f8e100a6640 secure :-1 s=READY pgs=1602 cs=0 l=1 rx=0x7f8e08010380 tx=0x7f8e0800ff70).stop 2019-12-05T06:08:16.293 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:16.284+0000 7f8e16663700 1 -- 172.21.15.42:0/777746559 shutdown_connections 2019-12-05T06:08:16.293 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:16.284+0000 7f8e16663700 1 --2- 172.21.15.42:0/777746559 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8df003b280 0x7f8df003d790 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:16.293 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:16.284+0000 7f8e16663700 1 --2- 172.21.15.42:0/777746559 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e100a6210 0x7f8e100a6640 unknown :-1 s=CLOSED pgs=1602 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:16.293 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:16.288+0000 7f8e16663700 1 -- 172.21.15.42:0/777746559 shutdown_connections 2019-12-05T06:08:16.293 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:16.288+0000 7f8e16663700 1 -- 172.21.15.42:0/777746559 wait complete. 2019-12-05T06:08:16.293 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:16.288+0000 7f8e16663700 1 -- 172.21.15.42:0/777746559 >> 172.21.15.42:0/777746559 conn(0x7f8e100a6e40 msgr2=0x7f8e1014f240 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:08:16.311 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a has gained rank=0, replacing gid=9573 2019-12-05T06:08:16.311 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 1 secs before reviving mds.b 2019-12-05T06:08:17.312 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.b 2019-12-05T06:08:17.312 INFO:tasks.ceph.mds.b:Restarting daemon 2019-12-05T06:08:17.312 INFO:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2019-12-05T06:08:17.317 INFO:tasks.ceph.mds.b:Started 2019-12-05T06:08:17.317 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:08:17.378 INFO:tasks.ceph.mds.b.smithi040.stdout:starting mds.b at 2019-12-05T06:08:17.422 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:17.412+0000 7fc5c338a700 1 Processor -- start 2019-12-05T06:08:17.422 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:17.412+0000 7fc5c338a700 1 -- start start 2019-12-05T06:08:17.422 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:17.412+0000 7fc5c338a700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc5bc08df80 0x7fc5bc09f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:08:17.423 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:17.412+0000 7fc5c338a700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5bc09fc60 con 0x7fc5bc0a7740 2019-12-05T06:08:17.423 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:17.412+0000 7fc5c338a700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5bc0a0710 con 0x7fc5bc09f830 2019-12-05T06:08:17.423 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:17.412+0000 7fc5c338a700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5bc0a0b00 con 0x7fc5bc08df80 2019-12-05T06:08:17.423 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:17.412+0000 7fc5c2128700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fc5bc09f830 0x7fc5bc14efe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:49034/0 (socket says 172.21.15.42:49034) 2019-12-05T06:08:17.423 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:17.412+0000 7fc5c2128700 1 -- 172.21.15.42:0/2128379422 learned_addr learned my addr 172.21.15.42:0/2128379422 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:08:17.423 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:17.412+0000 7fc5c1126700 1 --2- 172.21.15.42:0/2128379422 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc5bc08df80 0x7fc5bc09f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:17.423 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:17.412+0000 7fc5c0925700 1 -- 172.21.15.42:0/2128379422 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 878570284 0 0) 0x7fc5bc0a0710 con 0x7fc5bc09f830 2019-12-05T06:08:17.424 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:17.412+0000 7fc5c0925700 1 -- 172.21.15.42:0/2128379422 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5a80030b0 con 0x7fc5bc09f830 2019-12-05T06:08:17.424 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:17.412+0000 7fc5c0925700 1 -- 172.21.15.42:0/2128379422 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3587748105 0 0) 0x7fc5bc09fc60 con 0x7fc5bc0a7740 2019-12-05T06:08:17.424 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:17.412+0000 7fc5c1126700 1 -- 172.21.15.42:0/2128379422 >> v1:172.21.15.42:6789/0 conn(0x7fc5bc09f830 legacy=0x7fc5bc14efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:17.424 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:17.412+0000 7fc5c1126700 1 -- 172.21.15.42:0/2128379422 >> v1:172.21.15.40:6789/0 conn(0x7fc5bc0a7740 legacy=0x7fc5bc07e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:17.424 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:17.412+0000 7fc5c1126700 1 -- 172.21.15.42:0/2128379422 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5bc07ef20 con 0x7fc5bc08df80 2019-12-05T06:08:17.424 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:17.412+0000 7fc5c1126700 1 --2- 172.21.15.42:0/2128379422 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc5bc08df80 0x7fc5bc09f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fc5ac002f30 tx=0x7fc5ac00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:17.425 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:17.412+0000 7fc5c1126700 1 --2- 172.21.15.42:0/2128379422 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc5bc08df80 0x7fc5bc09f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fc5ac002f30 tx=0x7fc5ac00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:17.425 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:17.412+0000 7fc5c1126700 1 --2- 172.21.15.42:0/2128379422 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc5bc08df80 0x7fc5bc09f2e0 secure :-1 s=READY pgs=464 cs=0 l=1 rx=0x7fc5ac002f30 tx=0x7fc5ac00fbf0).ready entity=mon.2 client_cookie=89043775439b3ee8 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:08:17.425 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:17.416+0000 7fc5c1126700 1 --2- 172.21.15.42:0/2128379422 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc5bc08df80 0x7fc5bc09f2e0 secure :-1 s=THROTTLE_DONE pgs=464 cs=0 l=1 rx=0x7fc5ac002f30 tx=0x7fc5ac00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:17.425 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:17.416+0000 7fc5c0925700 1 -- 172.21.15.42:0/2128379422 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc5ac0112b0 con 0x7fc5bc08df80 2019-12-05T06:08:17.425 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:17.416+0000 7fc5c1126700 1 --2- 172.21.15.42:0/2128379422 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc5bc08df80 0x7fc5bc09f2e0 secure :-1 s=THROTTLE_DONE pgs=464 cs=0 l=1 rx=0x7fc5ac002f30 tx=0x7fc5ac00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:17.426 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:17.416+0000 7fc5c0925700 1 -- 172.21.15.42:0/2128379422 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5ac008bc0 con 0x7fc5bc08df80 2019-12-05T06:08:17.426 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:17.416+0000 7fc5c1126700 1 --2- 172.21.15.42:0/2128379422 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc5bc08df80 0x7fc5bc09f2e0 secure :-1 s=THROTTLE_DONE pgs=464 cs=0 l=1 rx=0x7fc5ac002f30 tx=0x7fc5ac00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:17.426 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:17.416+0000 7fc5c338a700 1 -- 172.21.15.42:0/2128379422 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc5bc08df80 msgr2=0x7fc5bc09f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:17.426 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:17.416+0000 7fc5c0925700 1 -- 172.21.15.42:0/2128379422 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc5ac011890 con 0x7fc5bc08df80 2019-12-05T06:08:17.426 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:17.416+0000 7fc5c338a700 1 --2- 172.21.15.42:0/2128379422 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc5bc08df80 0x7fc5bc09f2e0 secure :-1 s=READY pgs=464 cs=0 l=1 rx=0x7fc5ac002f30 tx=0x7fc5ac00fbf0).stop 2019-12-05T06:08:17.426 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:17.416+0000 7fc5c338a700 1 -- 172.21.15.42:0/2128379422 shutdown_connections 2019-12-05T06:08:17.427 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:17.416+0000 7fc5c338a700 1 --2- 172.21.15.42:0/2128379422 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc5bc08df80 0x7fc5bc09f2e0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:17.427 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:17.416+0000 7fc5c338a700 1 -- 172.21.15.42:0/2128379422 shutdown_connections 2019-12-05T06:08:17.427 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:17.416+0000 7fc5c338a700 1 -- 172.21.15.42:0/2128379422 wait complete. 2019-12-05T06:08:17.427 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:17.416+0000 7fc5c338a700 1 -- 172.21.15.42:0/2128379422 >> 172.21.15.42:0/2128379422 conn(0x7fc5bc0aabe0 msgr2=0x7fc5bc0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:08:17.432 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:17.420+0000 7fc5c338a700 1 Processor -- start 2019-12-05T06:08:17.432 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:17.420+0000 7fc5c338a700 1 -- start start 2019-12-05T06:08:17.432 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:17.420+0000 7fc5c338a700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc5bc116360 0x7fc5bc142c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:08:17.432 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:17.420+0000 7fc5c338a700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc5bc128f20 0x7fc5bc0a3a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:08:17.432 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:17.420+0000 7fc5c338a700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5bc128780 con 0x7fc5bc129350 2019-12-05T06:08:17.433 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:17.420+0000 7fc5c338a700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5bc116790 con 0x7fc5bc116360 2019-12-05T06:08:17.433 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:17.420+0000 7fc5c338a700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5bc0a2170 con 0x7fc5bc128f20 2019-12-05T06:08:17.433 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:17.420+0000 7fc5c1126700 1 -- >> v2:172.21.15.42:3300/0 conn(0x7fc5bc128f20 msgr2=0x7fc5bc0a3a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2019-12-05T06:08:17.433 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:17.420+0000 7fc5c1126700 1 -- >> v2:172.21.15.42:3300/0 conn(0x7fc5bc128f20 msgr2=0x7fc5bc0a3a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2019-12-05T06:08:17.433 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:17.420+0000 7fc5c1126700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc5bc128f20 0x7fc5bc0a3a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner read peer banner failed r=-1 ((1) Operation not permitted) 2019-12-05T06:08:17.433 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:17.420+0000 7fc5c1126700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc5bc128f20 0x7fc5bc0a3a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rx=0 tx=0)._fault waiting 0.200000 2019-12-05T06:08:17.433 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:17.420+0000 7fc5c2128700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fc5bc129350 0x7fc5bc08eb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:49040/0 (socket says 172.21.15.42:49040) 2019-12-05T06:08:17.434 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:17.420+0000 7fc5c2128700 1 -- 172.21.15.42:0/3011583711 learned_addr learned my addr 172.21.15.42:0/3011583711 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:08:17.434 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:17.420+0000 7fc5c1927700 1 --2- 172.21.15.42:0/3011583711 >> v2:172.21.15.40:3300/0 conn(0x7fc5bc116360 0x7fc5bc142c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:17.434 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:17.420+0000 7fc5b27fc700 1 -- 172.21.15.42:0/3011583711 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 112124415 0 0) 0x7fc5bc128780 con 0x7fc5bc129350 2019-12-05T06:08:17.434 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:17.420+0000 7fc5b27fc700 1 -- 172.21.15.42:0/3011583711 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5a40030b0 con 0x7fc5bc129350 2019-12-05T06:08:17.434 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:17.420+0000 7fc5c1927700 1 -- 172.21.15.42:0/3011583711 >> v2:172.21.15.42:3300/0 conn(0x7fc5bc128f20 msgr2=0x7fc5bc0a3a90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2019-12-05T06:08:17.434 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:17.420+0000 7fc5c1927700 1 --2- 172.21.15.42:0/3011583711 >> v2:172.21.15.42:3300/0 conn(0x7fc5bc128f20 0x7fc5bc0a3a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:17.434 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:17.420+0000 7fc5c1927700 1 -- 172.21.15.42:0/3011583711 >> v1:172.21.15.42:6789/0 conn(0x7fc5bc129350 legacy=0x7fc5bc08eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:17.435 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:17.420+0000 7fc5c1927700 1 -- 172.21.15.42:0/3011583711 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5bc143160 con 0x7fc5bc116360 2019-12-05T06:08:17.435 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:17.420+0000 7fc5c1927700 1 --2- 172.21.15.42:0/3011583711 >> v2:172.21.15.40:3300/0 conn(0x7fc5bc116360 0x7fc5bc142c10 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fc5bc0a3710 tx=0x7fc5b80108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:17.435 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:17.420+0000 7fc5c1927700 1 --2- 172.21.15.42:0/3011583711 >> v2:172.21.15.40:3300/0 conn(0x7fc5bc116360 0x7fc5bc142c10 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fc5bc0a3710 tx=0x7fc5b80108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:17.435 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:17.420+0000 7fc5c1927700 1 --2- 172.21.15.42:0/3011583711 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc5bc116360 0x7fc5bc142c10 secure :-1 s=READY pgs=1605 cs=0 l=1 rx=0x7fc5bc0a3710 tx=0x7fc5b80108f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:08:17.435 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:17.420+0000 7fc5c1927700 1 --2- 172.21.15.42:0/3011583711 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc5bc116360 0x7fc5bc142c10 secure :-1 s=THROTTLE_DONE pgs=1605 cs=0 l=1 rx=0x7fc5bc0a3710 tx=0x7fc5b80108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:17.436 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:17.420+0000 7fc5b27fc700 1 -- 172.21.15.42:0/3011583711 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc5b800cb90 con 0x7fc5bc116360 2019-12-05T06:08:17.436 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:17.420+0000 7fc5c1927700 1 --2- 172.21.15.42:0/3011583711 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc5bc116360 0x7fc5bc142c10 secure :-1 s=THROTTLE_DONE pgs=1605 cs=0 l=1 rx=0x7fc5bc0a3710 tx=0x7fc5b80108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:17.436 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:17.420+0000 7fc5b27fc700 1 -- 172.21.15.42:0/3011583711 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5b800c7b0 con 0x7fc5bc116360 2019-12-05T06:08:17.436 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:17.420+0000 7fc5c1927700 1 --2- 172.21.15.42:0/3011583711 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc5bc116360 0x7fc5bc142c10 secure :-1 s=THROTTLE_DONE pgs=1605 cs=0 l=1 rx=0x7fc5bc0a3710 tx=0x7fc5b80108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:17.437 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:17.420+0000 7fc5c338a700 1 -- 172.21.15.42:0/3011583711 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5bc0a3130 con 0x7fc5bc116360 2019-12-05T06:08:17.437 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:17.420+0000 7fc5b27fc700 1 -- 172.21.15.42:0/3011583711 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc5b80110b0 con 0x7fc5bc116360 2019-12-05T06:08:17.437 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:17.420+0000 7fc5c338a700 1 -- 172.21.15.42:0/3011583711 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5bc0a3fe0 con 0x7fc5bc116360 2019-12-05T06:08:17.437 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:17.424+0000 7fc5c1927700 1 --2- 172.21.15.42:0/3011583711 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc5bc116360 0x7fc5bc142c10 secure :-1 s=THROTTLE_DONE pgs=1605 cs=0 l=1 rx=0x7fc5bc0a3710 tx=0x7fc5b80108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:17.437 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:17.424+0000 7fc5b27fc700 1 -- 172.21.15.42:0/3011583711 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fc5b8011390 con 0x7fc5bc116360 2019-12-05T06:08:17.438 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:17.424+0000 7fc5c1927700 1 --2- 172.21.15.42:0/3011583711 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc5bc116360 0x7fc5bc142c10 secure :-1 s=THROTTLE_DONE pgs=1605 cs=0 l=1 rx=0x7fc5bc0a3710 tx=0x7fc5b80108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:17.438 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:17.424+0000 7fc5b27fc700 1 --2- 172.21.15.42:0/3011583711 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc5a403b2a0 0x7fc5a403d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:08:17.438 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:17.424+0000 7fc5b27fc700 1 -- 172.21.15.42:0/3011583711 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9901+0+0 (secure 0 0 0) 0x7fc5b804b260 con 0x7fc5bc116360 2019-12-05T06:08:17.438 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:17.424+0000 7fc5c1126700 1 --2- 172.21.15.42:0/3011583711 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc5a403b2a0 0x7fc5a403d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:17.438 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:17.424+0000 7fc5c1126700 1 --2- 172.21.15.42:0/3011583711 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc5a403b2a0 0x7fc5a403d7b0 crc :-1 s=READY pgs=891 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:08:17.438 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:17.428+0000 7fc5c338a700 1 -- 172.21.15.42:0/3011583711 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5bc05d110 con 0x7fc5bc116360 2019-12-05T06:08:17.441 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:17.436+0000 7fc5c1927700 1 --2- 172.21.15.42:0/3011583711 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc5bc116360 0x7fc5bc142c10 secure :-1 s=THROTTLE_DONE pgs=1605 cs=0 l=1 rx=0x7fc5bc0a3710 tx=0x7fc5b80108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:17.442 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:17.436+0000 7fc5b27fc700 1 -- 172.21.15.42:0/3011583711 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fc5b80322c0 con 0x7fc5bc116360 2019-12-05T06:08:17.637 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:17.632+0000 7fc5c338a700 1 -- 172.21.15.42:0/3011583711 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc5bc05b8c0 con 0x7fc5bc116360 2019-12-05T06:08:17.638 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:17.632+0000 7fc5c1927700 1 --2- 172.21.15.42:0/3011583711 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc5bc116360 0x7fc5bc142c10 secure :-1 s=THROTTLE_DONE pgs=1605 cs=0 l=1 rx=0x7fc5bc0a3710 tx=0x7fc5b80108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:17.638 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:17.632+0000 7fc5b27fc700 1 -- 172.21.15.42:0/3011583711 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 257 v257) v1 ==== 95+0+1775 (secure 0 0 0) 0x7fc5b8032520 con 0x7fc5bc116360 2019-12-05T06:08:17.639 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 257 2019-12-05T06:08:17.639 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:08:17.639 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":257,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":257,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:08:14.898258+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":94,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6929},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6929":{"gid":6929,"name":"a","rank":0,"incarnation":254,"state":"up:active","state_seq":19,"addr":"172.21.15.42:6835/2636699468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2636699468},{"type":"v1","addr":"172.21.15.42:6835","nonce":2636699468}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:08:17.641 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:17.636+0000 7fc5c338a700 1 -- 172.21.15.42:0/3011583711 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc5a403b2a0 msgr2=0x7fc5a403d7b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:17.641 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:17.636+0000 7fc5c338a700 1 --2- 172.21.15.42:0/3011583711 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc5a403b2a0 0x7fc5a403d7b0 crc :-1 s=READY pgs=891 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:17.641 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:17.636+0000 7fc5c338a700 1 -- 172.21.15.42:0/3011583711 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc5bc116360 msgr2=0x7fc5bc142c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:17.642 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:17.636+0000 7fc5c338a700 1 --2- 172.21.15.42:0/3011583711 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc5bc116360 0x7fc5bc142c10 secure :-1 s=READY pgs=1605 cs=0 l=1 rx=0x7fc5bc0a3710 tx=0x7fc5b80108f0).stop 2019-12-05T06:08:17.642 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:17.636+0000 7fc5c338a700 1 -- 172.21.15.42:0/3011583711 shutdown_connections 2019-12-05T06:08:17.642 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:17.636+0000 7fc5c338a700 1 --2- 172.21.15.42:0/3011583711 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc5a403b2a0 0x7fc5a403d7b0 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:17.642 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:17.636+0000 7fc5c338a700 1 --2- 172.21.15.42:0/3011583711 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc5bc116360 0x7fc5bc142c10 unknown :-1 s=CLOSED pgs=1605 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:17.642 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:17.636+0000 7fc5c338a700 1 --2- 172.21.15.42:0/3011583711 >> v2:172.21.15.42:3300/0 conn(0x7fc5bc128f20 0x7fc5bc0a3a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:17.642 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:17.636+0000 7fc5c338a700 1 -- 172.21.15.42:0/3011583711 shutdown_connections 2019-12-05T06:08:17.642 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:17.636+0000 7fc5c338a700 1 -- 172.21.15.42:0/3011583711 wait complete. 2019-12-05T06:08:17.643 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:17.636+0000 7fc5c338a700 1 -- 172.21.15.42:0/3011583711 >> 172.21.15.42:0/3011583711 conn(0x7fc5bc07e450 msgr2=0x7fc5bc09f380 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:08:17.652 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2019-12-05T06:08:19.652 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:08:19.787 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:19.748+0000 7f0596223700 1 Processor -- start 2019-12-05T06:08:19.788 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:19.748+0000 7f0596223700 1 -- start start 2019-12-05T06:08:19.788 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:19.748+0000 7f0596223700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f05900a7740 0x7f059008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:08:19.789 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:19.752+0000 7f0596223700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05900a0710 con 0x7f059009fae0 2019-12-05T06:08:19.789 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:19.752+0000 7f0596223700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05900a9ba0 con 0x7f059007e720 2019-12-05T06:08:19.789 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:19.752+0000 7f0596223700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05900a0b00 con 0x7f05900a7740 2019-12-05T06:08:19.789 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:19.752+0000 7f058ffff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f05900a7740 0x7f059008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:19.789 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:19.752+0000 7f0587fff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f059007e720 0x7f059009f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:49060/0 (socket says 172.21.15.42:49060) 2019-12-05T06:08:19.790 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:19.752+0000 7f058ffff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f05900a7740 0x7f059008df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:49020/0 (socket says 172.21.15.42:49020) 2019-12-05T06:08:19.790 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:19.752+0000 7f0587fff700 1 -- 172.21.15.42:0/1356554901 learned_addr learned my addr 172.21.15.42:0/1356554901 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:08:19.790 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:19.752+0000 7f058ffff700 1 -- 172.21.15.42:0/1356554901 >> v1:172.21.15.42:6789/0 conn(0x7f059007e720 legacy=0x7f059009f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:19.790 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:19.752+0000 7f0594fc1700 1 -- 172.21.15.42:0/1356554901 >> v1:172.21.15.40:6789/0 conn(0x7f059009fae0 legacy=0x7f059014efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2019-12-05T06:08:19.790 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:19.752+0000 7f0594fc1700 1 -- 172.21.15.42:0/1356554901 >> v1:172.21.15.40:6789/0 conn(0x7f059009fae0 legacy=0x7f059014efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2019-12-05T06:08:19.791 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:19.752+0000 7f0594fc1700 1 --1- 172.21.15.42:0/1356554901 >> v1:172.21.15.40:6789/0 conn(0x7f059009fae0 0x7f059014efe0 :-1 s=OPENED pgs=1606 cs=1 l=1).handle_message read tag failed 2019-12-05T06:08:19.791 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:19.752+0000 7f058ffff700 1 -- 172.21.15.42:0/1356554901 >> v1:172.21.15.40:6789/0 conn(0x7f059009fae0 legacy=0x7f059014efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:19.791 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:19.752+0000 7f0594fc1700 1 --1- 172.21.15.42:0/1356554901 >> v1:172.21.15.40:6789/0 conn(0x7f059009fae0 0x7f059014efe0 :-1 s=OPENED pgs=1606 cs=1 l=1).fault on lossy channel, failing 2019-12-05T06:08:19.792 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:19.752+0000 7f058ffff700 1 -- 172.21.15.42:0/1356554901 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f059007eb50 con 0x7f05900a7740 2019-12-05T06:08:19.792 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:19.752+0000 7f058ffff700 1 --2- 172.21.15.42:0/1356554901 >> v2:172.21.15.42:3300/0 conn(0x7f05900a7740 0x7f059008df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f0580002f30 tx=0x7f058000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:19.792 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:19.752+0000 7f058ffff700 1 --2- 172.21.15.42:0/1356554901 >> v2:172.21.15.42:3300/0 conn(0x7f05900a7740 0x7f059008df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f0580002f30 tx=0x7f058000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:19.792 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:19.752+0000 7f058ffff700 1 --2- 172.21.15.42:0/1356554901 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f05900a7740 0x7f059008df80 secure :-1 s=READY pgs=2115 cs=0 l=1 rx=0x7f0580002f30 tx=0x7f058000fbf0).ready entity=mon.1 client_cookie=7ff13187e812b4cb server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:08:19.792 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:19.752+0000 7f058ffff700 1 --2- 172.21.15.42:0/1356554901 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f05900a7740 0x7f059008df80 secure :-1 s=THROTTLE_DONE pgs=2115 cs=0 l=1 rx=0x7f0580002f30 tx=0x7f058000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:19.792 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:19.752+0000 7f058f7fe700 1 -- 172.21.15.42:0/1356554901 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0580012060 con 0x7f05900a7740 2019-12-05T06:08:19.792 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:19.752+0000 7f058ffff700 1 --2- 172.21.15.42:0/1356554901 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f05900a7740 0x7f059008df80 secure :-1 s=THROTTLE_DONE pgs=2115 cs=0 l=1 rx=0x7f0580002f30 tx=0x7f058000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:19.793 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:19.752+0000 7f058ffff700 1 --2- 172.21.15.42:0/1356554901 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f05900a7740 0x7f059008df80 secure :-1 s=THROTTLE_DONE pgs=2115 cs=0 l=1 rx=0x7f0580002f30 tx=0x7f058000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:19.793 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:19.752+0000 7f058f7fe700 1 -- 172.21.15.42:0/1356554901 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05800110c0 con 0x7f05900a7740 2019-12-05T06:08:19.793 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:19.752+0000 7f058f7fe700 1 -- 172.21.15.42:0/1356554901 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0580008da0 con 0x7f05900a7740 2019-12-05T06:08:19.793 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:19.752+0000 7f0596223700 1 -- 172.21.15.42:0/1356554901 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f05900a7740 msgr2=0x7f059008df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:19.793 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:19.752+0000 7f0596223700 1 --2- 172.21.15.42:0/1356554901 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f05900a7740 0x7f059008df80 secure :-1 s=READY pgs=2115 cs=0 l=1 rx=0x7f0580002f30 tx=0x7f058000fbf0).stop 2019-12-05T06:08:19.793 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:19.752+0000 7f0596223700 1 -- 172.21.15.42:0/1356554901 shutdown_connections 2019-12-05T06:08:19.794 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:19.752+0000 7f0596223700 1 --2- 172.21.15.42:0/1356554901 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f05900a7740 0x7f059008df80 unknown :-1 s=CLOSED pgs=2115 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:19.794 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:19.752+0000 7f0596223700 1 -- 172.21.15.42:0/1356554901 shutdown_connections 2019-12-05T06:08:19.794 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:19.752+0000 7f0596223700 1 -- 172.21.15.42:0/1356554901 wait complete. 2019-12-05T06:08:19.794 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:19.752+0000 7f0596223700 1 -- 172.21.15.42:0/1356554901 >> 172.21.15.42:0/1356554901 conn(0x7f05900aabe0 msgr2=0x7f05900a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:08:19.795 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:19.756+0000 7f0596223700 1 Processor -- start 2019-12-05T06:08:19.795 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:19.756+0000 7f0596223700 1 -- start start 2019-12-05T06:08:19.796 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:19.756+0000 7f0596223700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0590059700 0x7f0590078b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:08:19.796 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:19.760+0000 7f0596223700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0590122fb0 con 0x7f05900a6c50 2019-12-05T06:08:19.796 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:19.760+0000 7f0596223700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05900a7080 con 0x7f0590122b80 2019-12-05T06:08:19.797 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:19.760+0000 7f0596223700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05900a5800 con 0x7f0590059700 2019-12-05T06:08:19.797 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:19.760+0000 7f058ffff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f0590122b80 0x7f05900654b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:49064/0 (socket says 172.21.15.42:49064) 2019-12-05T06:08:19.797 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:19.760+0000 7f058ffff700 1 -- 172.21.15.42:0/589136480 learned_addr learned my addr 172.21.15.42:0/589136480 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:08:19.797 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:19.760+0000 7f0594fc1700 1 --2- 172.21.15.42:0/589136480 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0590059700 0x7f0590078b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:19.799 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:19.760+0000 7f058d7fa700 1 -- 172.21.15.42:0/589136480 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1594926033 0 0) 0x7f05900a7080 con 0x7f0590122b80 2019-12-05T06:08:19.799 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:19.760+0000 7f058d7fa700 1 -- 172.21.15.42:0/589136480 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05740030b0 con 0x7f0590122b80 2019-12-05T06:08:19.799 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:19.760+0000 7f058d7fa700 1 -- 172.21.15.42:0/589136480 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2804310365 0 0) 0x7f0590122fb0 con 0x7f05900a6c50 2019-12-05T06:08:19.799 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:19.760+0000 7f058d7fa700 1 -- 172.21.15.42:0/589136480 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05900a7080 con 0x7f05900a6c50 2019-12-05T06:08:19.799 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:19.760+0000 7f058d7fa700 1 -- 172.21.15.42:0/589136480 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 3143328956 0 0) 0x7f05740030b0 con 0x7f0590122b80 2019-12-05T06:08:19.800 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:19.760+0000 7f058d7fa700 1 -- 172.21.15.42:0/589136480 --> v1:172.21.15.42:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0590122fb0 con 0x7f0590122b80 2019-12-05T06:08:19.800 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:19.760+0000 7f058d7fa700 1 -- 172.21.15.42:0/589136480 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7f0580008790 con 0x7f0590122b80 2019-12-05T06:08:19.802 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:19.760+0000 7f0594fc1700 1 -- 172.21.15.42:0/589136480 >> v1:172.21.15.42:6789/0 conn(0x7f0590122b80 legacy=0x7f05900654b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:19.802 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:19.760+0000 7f0594fc1700 1 -- 172.21.15.42:0/589136480 >> v1:172.21.15.40:6789/0 conn(0x7f05900a6c50 legacy=0x7f05900752f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:19.802 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:19.760+0000 7f0594fc1700 1 -- 172.21.15.42:0/589136480 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f059007b080 con 0x7f0590059700 2019-12-05T06:08:19.803 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:19.760+0000 7f0594fc1700 1 --2- 172.21.15.42:0/589136480 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0590059700 0x7f0590078b10 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f0588001210 tx=0x7f05800082d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:19.803 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:19.760+0000 7f0594fc1700 1 --2- 172.21.15.42:0/589136480 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0590059700 0x7f0590078b10 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f0588001210 tx=0x7f05800082d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:19.803 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:19.760+0000 7f0594fc1700 1 --2- 172.21.15.42:0/589136480 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0590059700 0x7f0590078b10 secure :-1 s=READY pgs=465 cs=0 l=1 rx=0x7f0588001210 tx=0x7f05800082d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:08:19.803 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:19.760+0000 7f0594fc1700 1 --2- 172.21.15.42:0/589136480 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0590059700 0x7f0590078b10 secure :-1 s=THROTTLE_DONE pgs=465 cs=0 l=1 rx=0x7f0588001210 tx=0x7f05800082d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:19.804 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:19.760+0000 7f058d7fa700 1 -- 172.21.15.42:0/589136480 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0588005c90 con 0x7f0590059700 2019-12-05T06:08:19.804 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:19.760+0000 7f0594fc1700 1 --2- 172.21.15.42:0/589136480 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0590059700 0x7f0590078b10 secure :-1 s=THROTTLE_DONE pgs=465 cs=0 l=1 rx=0x7f0588001210 tx=0x7f05800082d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:19.804 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:19.760+0000 7f0596223700 1 -- 172.21.15.42:0/589136480 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05900a5190 con 0x7f0590059700 2019-12-05T06:08:19.804 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:19.760+0000 7f058d7fa700 1 -- 172.21.15.42:0/589136480 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05880058b0 con 0x7f0590059700 2019-12-05T06:08:19.805 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:19.760+0000 7f0594fc1700 1 --2- 172.21.15.42:0/589136480 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0590059700 0x7f0590078b10 secure :-1 s=THROTTLE_DONE pgs=465 cs=0 l=1 rx=0x7f0588001210 tx=0x7f05800082d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:19.805 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:19.760+0000 7f058d7fa700 1 -- 172.21.15.42:0/589136480 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f05880064a0 con 0x7f0590059700 2019-12-05T06:08:19.805 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:19.760+0000 7f0596223700 1 -- 172.21.15.42:0/589136480 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0590059b30 con 0x7f0590059700 2019-12-05T06:08:19.805 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:19.760+0000 7f0594fc1700 1 --2- 172.21.15.42:0/589136480 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0590059700 0x7f0590078b10 secure :-1 s=THROTTLE_DONE pgs=465 cs=0 l=1 rx=0x7f0588001210 tx=0x7f05800082d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:19.806 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:19.764+0000 7f058d7fa700 1 -- 172.21.15.42:0/589136480 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f05880067c0 con 0x7f0590059700 2019-12-05T06:08:19.814 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:19.764+0000 7f0594fc1700 1 --2- 172.21.15.42:0/589136480 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0590059700 0x7f0590078b10 secure :-1 s=THROTTLE_DONE pgs=465 cs=0 l=1 rx=0x7f0588001210 tx=0x7f05800082d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:19.814 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:19.764+0000 7f058d7fa700 1 --2- 172.21.15.42:0/589136480 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f057403b820 0x7f057403dd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:08:19.814 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:19.764+0000 7f058d7fa700 1 -- 172.21.15.42:0/589136480 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9901+0+0 (secure 0 0 0) 0x7f058804acb0 con 0x7f0590059700 2019-12-05T06:08:19.815 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:19.764+0000 7f058ffff700 1 --2- 172.21.15.42:0/589136480 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f057403b820 0x7f057403dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:19.815 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:19.764+0000 7f058ffff700 1 --2- 172.21.15.42:0/589136480 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f057403b820 0x7f057403dd30 crc :-1 s=READY pgs=892 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:08:19.815 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:19.764+0000 7f0596223700 1 -- 172.21.15.42:0/589136480 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f059005d110 con 0x7f0590059700 2019-12-05T06:08:19.815 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:19.772+0000 7f0594fc1700 1 --2- 172.21.15.42:0/589136480 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0590059700 0x7f0590078b10 secure :-1 s=THROTTLE_DONE pgs=465 cs=0 l=1 rx=0x7f0588001210 tx=0x7f05800082d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:19.816 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:19.772+0000 7f058d7fa700 1 -- 172.21.15.42:0/589136480 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f0588031d30 con 0x7f0590059700 2019-12-05T06:08:19.951 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:19.940+0000 7f0596223700 1 -- 172.21.15.42:0/589136480 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f059008e570 con 0x7f0590059700 2019-12-05T06:08:19.951 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:19.944+0000 7f0594fc1700 1 --2- 172.21.15.42:0/589136480 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0590059700 0x7f0590078b10 secure :-1 s=THROTTLE_DONE pgs=465 cs=0 l=1 rx=0x7f0588001210 tx=0x7f05800082d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:19.951 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:19.944+0000 7f058d7fa700 1 -- 172.21.15.42:0/589136480 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 257 v257) v1 ==== 95+0+1775 (secure 0 0 0) 0x7f0588031f90 con 0x7f0590059700 2019-12-05T06:08:19.951 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 257 2019-12-05T06:08:19.951 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:08:19.952 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":257,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":257,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:08:14.898258+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":94,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6929},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6929":{"gid":6929,"name":"a","rank":0,"incarnation":254,"state":"up:active","state_seq":19,"addr":"172.21.15.42:6835/2636699468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2636699468},{"type":"v1","addr":"172.21.15.42:6835","nonce":2636699468}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:08:19.954 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:19.948+0000 7f0596223700 1 -- 172.21.15.42:0/589136480 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f057403b820 msgr2=0x7f057403dd30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:19.954 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:19.948+0000 7f0596223700 1 --2- 172.21.15.42:0/589136480 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f057403b820 0x7f057403dd30 crc :-1 s=READY pgs=892 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:19.954 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:19.948+0000 7f0596223700 1 -- 172.21.15.42:0/589136480 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0590059700 msgr2=0x7f0590078b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:19.954 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:19.948+0000 7f0596223700 1 --2- 172.21.15.42:0/589136480 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0590059700 0x7f0590078b10 secure :-1 s=READY pgs=465 cs=0 l=1 rx=0x7f0588001210 tx=0x7f05800082d0).stop 2019-12-05T06:08:19.955 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:19.948+0000 7f0596223700 1 -- 172.21.15.42:0/589136480 shutdown_connections 2019-12-05T06:08:19.955 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:19.948+0000 7f0596223700 1 --2- 172.21.15.42:0/589136480 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f057403b820 0x7f057403dd30 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:19.955 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:19.948+0000 7f0596223700 1 --2- 172.21.15.42:0/589136480 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0590059700 0x7f0590078b10 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:19.955 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:19.948+0000 7f0596223700 1 -- 172.21.15.42:0/589136480 shutdown_connections 2019-12-05T06:08:19.955 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:19.948+0000 7f0596223700 1 -- 172.21.15.42:0/589136480 wait complete. 2019-12-05T06:08:19.955 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:19.948+0000 7f0596223700 1 -- 172.21.15.42:0/589136480 >> 172.21.15.42:0/589136480 conn(0x7f059007ee20 msgr2=0x7f059011d570 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:08:19.964 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2019-12-05T06:08:21.971 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:08:22.083 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:22.072+0000 7f3f8136b700 1 Processor -- start 2019-12-05T06:08:22.083 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:22.072+0000 7f3f8136b700 1 -- start start 2019-12-05T06:08:22.083 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:22.072+0000 7f3f8136b700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f3f7c09fae0 0x7f3f7c14fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:08:22.083 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:22.072+0000 7f3f8136b700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f7c0a0710 con 0x7f3f7c0a7740 2019-12-05T06:08:22.083 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:22.072+0000 7f3f8136b700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f7c07ef20 con 0x7f3f7c08df80 2019-12-05T06:08:22.083 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:22.072+0000 7f3f8136b700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f7c0a0b00 con 0x7f3f7c09fae0 2019-12-05T06:08:22.084 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:22.072+0000 7f3f7affd700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f3f7c08df80 0x7f3f7c09f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:49076/0 (socket says 172.21.15.42:49076) 2019-12-05T06:08:22.084 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:22.072+0000 7f3f7affd700 1 -- 172.21.15.42:0/2297974517 learned_addr learned my addr 172.21.15.42:0/2297974517 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:08:22.084 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:22.072+0000 7f3f7bfff700 1 --2- 172.21.15.42:0/2297974517 >> v2:172.21.15.40:3300/0 conn(0x7f3f7c09fae0 0x7f3f7c14fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:22.084 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:22.072+0000 7f3f7a7fc700 1 -- 172.21.15.42:0/2297974517 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 780165263 0 0) 0x7f3f7c07ef20 con 0x7f3f7c08df80 2019-12-05T06:08:22.084 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:22.072+0000 7f3f7a7fc700 1 -- 172.21.15.42:0/2297974517 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f680030b0 con 0x7f3f7c08df80 2019-12-05T06:08:22.084 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:22.072+0000 7f3f7a7fc700 1 -- 172.21.15.42:0/2297974517 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3889675650 0 0) 0x7f3f680030b0 con 0x7f3f7c08df80 2019-12-05T06:08:22.085 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:22.072+0000 7f3f7a7fc700 1 -- 172.21.15.42:0/2297974517 >> v2:172.21.15.40:3300/0 conn(0x7f3f7c09fae0 msgr2=0x7f3f7c14fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:08:22.085 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:22.072+0000 7f3f7a7fc700 1 --2- 172.21.15.42:0/2297974517 >> v2:172.21.15.40:3300/0 conn(0x7f3f7c09fae0 0x7f3f7c14fff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:08:22.085 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:22.072+0000 7f3f7a7fc700 1 -- 172.21.15.42:0/2297974517 >> v1:172.21.15.40:6789/0 conn(0x7f3f7c0a7740 legacy=0x7f3f7c07e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:22.085 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:22.072+0000 7f3f7a7fc700 1 -- 172.21.15.42:0/2297974517 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f7c0a9ba0 con 0x7f3f7c08df80 2019-12-05T06:08:22.085 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:22.072+0000 7f3f7a7fc700 1 -- 172.21.15.42:0/2297974517 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7f3f64001750 con 0x7f3f7c08df80 2019-12-05T06:08:22.085 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:22.072+0000 7f3f7a7fc700 1 -- 172.21.15.42:0/2297974517 >> v1:172.21.15.42:6789/0 conn(0x7f3f7c08df80 legacy=0x7f3f7c09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:22.085 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:22.072+0000 7f3f7a7fc700 1 --2- 172.21.15.42:0/2297974517 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3f680036e0 0x7f3f68003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:08:22.085 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:22.072+0000 7f3f7a7fc700 1 --2- 172.21.15.42:0/2297974517 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3f68004420 0x7f3f68004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:08:22.086 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:22.072+0000 7f3f7a7fc700 1 --2- 172.21.15.42:0/2297974517 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f68004da0 0x7f3f6800b670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:08:22.086 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:22.072+0000 7f3f7a7fc700 1 -- 172.21.15.42:0/2297974517 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f7c0a0b00 con 0x7f3f68004da0 2019-12-05T06:08:22.086 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:22.072+0000 7f3f7affd700 1 --2- 172.21.15.42:0/2297974517 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3f68004420 0x7f3f68004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:22.086 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:22.072+0000 7f3f7a7fc700 1 -- 172.21.15.42:0/2297974517 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f6804fc60 con 0x7f3f68004420 2019-12-05T06:08:22.086 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:22.072+0000 7f3f7a7fc700 1 -- 172.21.15.42:0/2297974517 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f6804fea0 con 0x7f3f680036e0 2019-12-05T06:08:22.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:22.072+0000 7f3f7b7fe700 1 --2- 172.21.15.42:0/2297974517 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3f680036e0 0x7f3f68003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:22.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:22.072+0000 7f3f7bfff700 1 --2- 172.21.15.42:0/2297974517 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f68004da0 0x7f3f6800b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:22.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:22.072+0000 7f3f7affd700 1 -- 172.21.15.42:0/2297974517 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3f680036e0 msgr2=0x7f3f68003b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:08:22.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:22.072+0000 7f3f7affd700 1 --2- 172.21.15.42:0/2297974517 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3f680036e0 0x7f3f68003b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:08:22.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:22.072+0000 7f3f7affd700 1 -- 172.21.15.42:0/2297974517 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f68004da0 msgr2=0x7f3f6800b670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:08:22.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:22.072+0000 7f3f7affd700 1 --2- 172.21.15.42:0/2297974517 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f68004da0 0x7f3f6800b670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:08:22.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:22.072+0000 7f3f7affd700 1 -- 172.21.15.42:0/2297974517 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f680501a0 con 0x7f3f68004420 2019-12-05T06:08:22.088 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:22.072+0000 7f3f7bfff700 1 --2- 172.21.15.42:0/2297974517 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f68004da0 0x7f3f6800b670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:08:22.088 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:22.072+0000 7f3f7bfff700 1 -- 172.21.15.42:0/2297974517 reap_dead start 2019-12-05T06:08:22.088 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:22.072+0000 7f3f7affd700 1 --2- 172.21.15.42:0/2297974517 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3f68004420 0x7f3f68004850 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f3f64001950 tx=0x7f3f6400c240).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:22.088 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:22.072+0000 7f3f7affd700 1 --2- 172.21.15.42:0/2297974517 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3f68004420 0x7f3f68004850 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f3f64001950 tx=0x7f3f6400c240).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:22.088 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:22.072+0000 7f3f7affd700 1 --2- 172.21.15.42:0/2297974517 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3f68004420 0x7f3f68004850 secure :-1 s=READY pgs=2118 cs=0 l=1 rx=0x7f3f64001950 tx=0x7f3f6400c240).ready entity=mon.1 client_cookie=12e39ef7f029098 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:08:22.089 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:22.072+0000 7f3f7affd700 1 --2- 172.21.15.42:0/2297974517 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3f68004420 0x7f3f68004850 secure :-1 s=THROTTLE_DONE pgs=2118 cs=0 l=1 rx=0x7f3f64001950 tx=0x7f3f6400c240).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:22.089 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:22.072+0000 7f3f7a7fc700 1 -- 172.21.15.42:0/2297974517 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3f64010350 con 0x7f3f68004420 2019-12-05T06:08:22.089 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:22.072+0000 7f3f7affd700 1 --2- 172.21.15.42:0/2297974517 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3f68004420 0x7f3f68004850 secure :-1 s=THROTTLE_DONE pgs=2118 cs=0 l=1 rx=0x7f3f64001950 tx=0x7f3f6400c240).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:22.089 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:22.072+0000 7f3f7a7fc700 1 -- 172.21.15.42:0/2297974517 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f6400ed20 con 0x7f3f68004420 2019-12-05T06:08:22.089 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:22.072+0000 7f3f7affd700 1 --2- 172.21.15.42:0/2297974517 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3f68004420 0x7f3f68004850 secure :-1 s=THROTTLE_DONE pgs=2118 cs=0 l=1 rx=0x7f3f64001950 tx=0x7f3f6400c240).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:22.090 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:22.072+0000 7f3f7a7fc700 1 -- 172.21.15.42:0/2297974517 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3f64010930 con 0x7f3f68004420 2019-12-05T06:08:22.090 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:22.072+0000 7f3f8136b700 1 -- 172.21.15.42:0/2297974517 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3f68004420 msgr2=0x7f3f68004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:22.090 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:22.072+0000 7f3f8136b700 1 --2- 172.21.15.42:0/2297974517 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3f68004420 0x7f3f68004850 secure :-1 s=READY pgs=2118 cs=0 l=1 rx=0x7f3f64001950 tx=0x7f3f6400c240).stop 2019-12-05T06:08:22.090 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:22.076+0000 7f3f8136b700 1 -- 172.21.15.42:0/2297974517 shutdown_connections 2019-12-05T06:08:22.090 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:22.076+0000 7f3f8136b700 1 --2- 172.21.15.42:0/2297974517 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3f68004420 0x7f3f68004850 unknown :-1 s=CLOSED pgs=2118 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:22.090 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:22.076+0000 7f3f8136b700 1 -- 172.21.15.42:0/2297974517 shutdown_connections 2019-12-05T06:08:22.091 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:22.076+0000 7f3f8136b700 1 -- 172.21.15.42:0/2297974517 wait complete. 2019-12-05T06:08:22.091 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:22.076+0000 7f3f8136b700 1 -- 172.21.15.42:0/2297974517 >> 172.21.15.42:0/2297974517 conn(0x7f3f7c0aabe0 msgr2=0x7f3f7c0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:08:22.091 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:22.076+0000 7f3f8136b700 1 Processor -- start 2019-12-05T06:08:22.091 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:22.076+0000 7f3f8136b700 1 -- start start 2019-12-05T06:08:22.096 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:22.080+0000 7f3f8136b700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f3f7c11a2d0 0x7f3f7c11bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:08:22.096 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:22.080+0000 7f3f8136b700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f7c0a69d0 con 0x7f3f7c12fd80 2019-12-05T06:08:22.096 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:22.080+0000 7f3f8136b700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f7c0a6c00 con 0x7f3f7c11b7f0 2019-12-05T06:08:22.096 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:22.080+0000 7f3f8136b700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f7c11b600 con 0x7f3f7c11a2d0 2019-12-05T06:08:22.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:22.080+0000 7f3f7affd700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f3f7c11b7f0 0x7f3f7c12f580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:49088/0 (socket says 172.21.15.42:49088) 2019-12-05T06:08:22.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:22.080+0000 7f3f7affd700 1 -- 172.21.15.42:0/2103848988 learned_addr learned my addr 172.21.15.42:0/2103848988 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:08:22.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:22.080+0000 7f3f7b7fe700 1 --2- 172.21.15.42:0/2103848988 >> v2:172.21.15.40:3300/0 conn(0x7f3f7c11a2d0 0x7f3f7c11bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:22.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:22.080+0000 7f3f73fff700 1 -- 172.21.15.42:0/2103848988 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 639539933 0 0) 0x7f3f7c0a6c00 con 0x7f3f7c11b7f0 2019-12-05T06:08:22.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:22.080+0000 7f3f73fff700 1 -- 172.21.15.42:0/2103848988 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f600030b0 con 0x7f3f7c11b7f0 2019-12-05T06:08:22.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:22.080+0000 7f3f73fff700 1 -- 172.21.15.42:0/2103848988 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1333499308 0 0) 0x7f3f7c0a69d0 con 0x7f3f7c12fd80 2019-12-05T06:08:22.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:22.080+0000 7f3f73fff700 1 -- 172.21.15.42:0/2103848988 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f7c0a6c00 con 0x7f3f7c12fd80 2019-12-05T06:08:22.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:22.080+0000 7f3f7b7fe700 1 -- 172.21.15.42:0/2103848988 >> v1:172.21.15.42:6789/0 conn(0x7f3f7c11b7f0 legacy=0x7f3f7c12f580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:22.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:22.080+0000 7f3f73fff700 1 -- 172.21.15.42:0/2103848988 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 838633966 0 0) 0x7f3f600030b0 con 0x7f3f7c11b7f0 2019-12-05T06:08:22.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:22.080+0000 7f3f7b7fe700 1 -- 172.21.15.42:0/2103848988 >> v1:172.21.15.40:6789/0 conn(0x7f3f7c12fd80 legacy=0x7f3f7c0a61d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:22.100 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:22.080+0000 7f3f7b7fe700 1 -- 172.21.15.42:0/2103848988 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f7c139a00 con 0x7f3f7c11a2d0 2019-12-05T06:08:22.100 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:22.080+0000 7f3f7b7fe700 1 --2- 172.21.15.42:0/2103848988 >> v2:172.21.15.40:3300/0 conn(0x7f3f7c11a2d0 0x7f3f7c11bfc0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f3f6400b9f0 tx=0x7f3f64010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:22.100 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:22.080+0000 7f3f7b7fe700 1 --2- 172.21.15.42:0/2103848988 >> v2:172.21.15.40:3300/0 conn(0x7f3f7c11a2d0 0x7f3f7c11bfc0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f3f6400b9f0 tx=0x7f3f64010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:22.100 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:22.080+0000 7f3f7b7fe700 1 --2- 172.21.15.42:0/2103848988 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f7c11a2d0 0x7f3f7c11bfc0 secure :-1 s=READY pgs=1610 cs=0 l=1 rx=0x7f3f6400b9f0 tx=0x7f3f64010b50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:08:22.100 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:22.080+0000 7f3f7b7fe700 1 --2- 172.21.15.42:0/2103848988 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f7c11a2d0 0x7f3f7c11bfc0 secure :-1 s=THROTTLE_DONE pgs=1610 cs=0 l=1 rx=0x7f3f6400b9f0 tx=0x7f3f64010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:22.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:22.080+0000 7f3f73fff700 1 -- 172.21.15.42:0/2103848988 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3f6400ed20 con 0x7f3f7c11a2d0 2019-12-05T06:08:22.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:22.080+0000 7f3f7b7fe700 1 --2- 172.21.15.42:0/2103848988 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f7c11a2d0 0x7f3f7c11bfc0 secure :-1 s=THROTTLE_DONE pgs=1610 cs=0 l=1 rx=0x7f3f6400b9f0 tx=0x7f3f64010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:22.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:22.080+0000 7f3f7b7fe700 1 --2- 172.21.15.42:0/2103848988 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f7c11a2d0 0x7f3f7c11bfc0 secure :-1 s=THROTTLE_DONE pgs=1610 cs=0 l=1 rx=0x7f3f6400b9f0 tx=0x7f3f64010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:22.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:22.080+0000 7f3f73fff700 1 -- 172.21.15.42:0/2103848988 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f64010550 con 0x7f3f7c11a2d0 2019-12-05T06:08:22.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:22.080+0000 7f3f73fff700 1 -- 172.21.15.42:0/2103848988 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3f74012510 con 0x7f3f7c11a2d0 2019-12-05T06:08:22.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:22.080+0000 7f3f8136b700 1 -- 172.21.15.42:0/2103848988 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f7c1379e0 con 0x7f3f7c11a2d0 2019-12-05T06:08:22.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:22.080+0000 7f3f8136b700 1 -- 172.21.15.42:0/2103848988 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f7c137eb0 con 0x7f3f7c11a2d0 2019-12-05T06:08:22.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:22.080+0000 7f3f7b7fe700 1 --2- 172.21.15.42:0/2103848988 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f7c11a2d0 0x7f3f7c11bfc0 secure :-1 s=THROTTLE_DONE pgs=1610 cs=0 l=1 rx=0x7f3f6400b9f0 tx=0x7f3f64010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:22.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:22.080+0000 7f3f73fff700 1 -- 172.21.15.42:0/2103848988 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f3f74012870 con 0x7f3f7c11a2d0 2019-12-05T06:08:22.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:22.080+0000 7f3f7b7fe700 1 --2- 172.21.15.42:0/2103848988 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f7c11a2d0 0x7f3f7c11bfc0 secure :-1 s=THROTTLE_DONE pgs=1610 cs=0 l=1 rx=0x7f3f6400b9f0 tx=0x7f3f64010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:22.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:22.080+0000 7f3f73fff700 1 --2- 172.21.15.42:0/2103848988 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3f6003b370 0x7f3f6003d880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:08:22.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:22.084+0000 7f3f73fff700 1 -- 172.21.15.42:0/2103848988 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9901+0+0 (secure 0 0 0) 0x7f3f7404ae10 con 0x7f3f7c11a2d0 2019-12-05T06:08:22.103 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:22.084+0000 7f3f7affd700 1 --2- 172.21.15.42:0/2103848988 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3f6003b370 0x7f3f6003d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:22.103 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:22.084+0000 7f3f7affd700 1 --2- 172.21.15.42:0/2103848988 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3f6003b370 0x7f3f6003d880 crc :-1 s=READY pgs=894 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:08:22.103 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:22.088+0000 7f3f8136b700 1 -- 172.21.15.42:0/2103848988 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f7c05d110 con 0x7f3f7c11a2d0 2019-12-05T06:08:22.103 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:22.096+0000 7f3f7b7fe700 1 --2- 172.21.15.42:0/2103848988 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f7c11a2d0 0x7f3f7c11bfc0 secure :-1 s=THROTTLE_DONE pgs=1610 cs=0 l=1 rx=0x7f3f6400b9f0 tx=0x7f3f64010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:22.104 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:22.096+0000 7f3f73fff700 1 -- 172.21.15.42:0/2103848988 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f3f74031e90 con 0x7f3f7c11a2d0 2019-12-05T06:08:22.271 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:22.264+0000 7f3f8136b700 1 -- 172.21.15.42:0/2103848988 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3f7c05b8c0 con 0x7f3f7c11a2d0 2019-12-05T06:08:22.271 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:22.264+0000 7f3f7b7fe700 1 -- 172.21.15.42:0/2103848988 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f7c11a2d0 msgr2=0x7f3f7c11bfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2019-12-05T06:08:22.271 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:22.264+0000 7f3f7b7fe700 1 -- 172.21.15.42:0/2103848988 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f7c11a2d0 msgr2=0x7f3f7c11bfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2019-12-05T06:08:22.271 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:22.264+0000 7f3f7b7fe700 1 --2- 172.21.15.42:0/2103848988 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f7c11a2d0 0x7f3f7c11bfc0 secure :-1 s=READY pgs=1610 cs=0 l=1 rx=0x7f3f6400b9f0 tx=0x7f3f64010b50).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-12-05T06:08:22.271 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:22.264+0000 7f3f7b7fe700 1 --2- 172.21.15.42:0/2103848988 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f7c11a2d0 0x7f3f7c11bfc0 secure :-1 s=READY pgs=1610 cs=0 l=1 rx=0x7f3f6400b9f0 tx=0x7f3f64010b50).stop 2019-12-05T06:08:22.272 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:22.264+0000 7f3f73fff700 1 -- 172.21.15.42:0/2103848988 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f7c11a2d0 msgr2=0x7f3f7c11bfc0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2019-12-05T06:08:22.272 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:22.264+0000 7f3f73fff700 1 --2- 172.21.15.42:0/2103848988 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f7c11a2d0 0x7f3f7c11bfc0 unknown :-1 s=CLOSED pgs=1610 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:22.272 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:22.264+0000 7f3f73fff700 1 --2- 172.21.15.42:0/2103848988 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3f60043f00 0x7f3f60046340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:08:22.272 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:22.264+0000 7f3f73fff700 1 --2- 172.21.15.42:0/2103848988 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f60046a00 0x7f3f60048ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:08:22.272 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:22.264+0000 7f3f73fff700 1 --2- 172.21.15.42:0/2103848988 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3f600496a0 0x7f3f6004bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:08:22.272 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:22.264+0000 7f3f73fff700 1 -- 172.21.15.42:0/2103848988 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f6004c3d0 con 0x7f3f60046a00 2019-12-05T06:08:22.273 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:22.264+0000 7f3f73fff700 1 -- 172.21.15.42:0/2103848988 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f6004c5c0 con 0x7f3f600496a0 2019-12-05T06:08:22.273 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:22.264+0000 7f3f73fff700 1 -- 172.21.15.42:0/2103848988 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f6004c800 con 0x7f3f60043f00 2019-12-05T06:08:22.273 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:22.264+0000 7f3f7b7fe700 1 --2- 172.21.15.42:0/2103848988 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3f600496a0 0x7f3f6004bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:22.273 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:22.264+0000 7f3f7bfff700 1 --2- 172.21.15.42:0/2103848988 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f60046a00 0x7f3f60048ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:22.273 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:22.264+0000 7f3f7b7fe700 1 --2- 172.21.15.42:0/2103848988 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3f60043f00 0x7f3f60046340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:22.275 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:22.264+0000 7f3f7bfff700 1 -- 172.21.15.42:0/2103848988 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3f60043f00 msgr2=0x7f3f60046340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:22.275 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:22.264+0000 7f3f7bfff700 1 --2- 172.21.15.42:0/2103848988 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3f60043f00 0x7f3f60046340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:22.275 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:22.264+0000 7f3f7bfff700 1 -- 172.21.15.42:0/2103848988 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3f600496a0 msgr2=0x7f3f6004bbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:22.275 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:22.264+0000 7f3f7bfff700 1 --2- 172.21.15.42:0/2103848988 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3f600496a0 0x7f3f6004bbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:22.276 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:22.264+0000 7f3f7bfff700 1 -- 172.21.15.42:0/2103848988 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f3f6004caa0 con 0x7f3f60046a00 2019-12-05T06:08:22.276 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:22.264+0000 7f3f7bfff700 1 -- 172.21.15.42:0/2103848988 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3f6c0104d0 con 0x7f3f60046a00 2019-12-05T06:08:22.276 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:22.264+0000 7f3f7b7fe700 1 --2- 172.21.15.42:0/2103848988 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3f600496a0 0x7f3f6004bbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:08:22.276 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:22.264+0000 7f3f7bfff700 1 --2- 172.21.15.42:0/2103848988 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f60046a00 0x7f3f60048ec0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f3f6c002f30 tx=0x7f3f6c010a60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:22.276 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:22.268+0000 7f3f7bfff700 1 -- 172.21.15.42:0/2103848988 reap_dead start 2019-12-05T06:08:22.276 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:22.268+0000 7f3f7bfff700 1 --2- 172.21.15.42:0/2103848988 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f60046a00 0x7f3f60048ec0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f3f6c002f30 tx=0x7f3f6c010a60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:22.277 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:22.268+0000 7f3f7bfff700 1 --2- 172.21.15.42:0/2103848988 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f60046a00 0x7f3f60048ec0 secure :-1 s=READY pgs=1611 cs=0 l=1 rx=0x7f3f6c002f30 tx=0x7f3f6c010a60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:08:22.277 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:22.268+0000 7f3f7bfff700 1 --2- 172.21.15.42:0/2103848988 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f60046a00 0x7f3f60048ec0 secure :-1 s=THROTTLE_DONE pgs=1611 cs=0 l=1 rx=0x7f3f6c002f30 tx=0x7f3f6c010a60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:22.277 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:22.268+0000 7f3f73fff700 1 -- 172.21.15.42:0/2103848988 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3f6c0104d0 con 0x7f3f60046a00 2019-12-05T06:08:22.277 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:22.268+0000 7f3f7bfff700 1 --2- 172.21.15.42:0/2103848988 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f60046a00 0x7f3f60048ec0 secure :-1 s=THROTTLE_DONE pgs=1611 cs=0 l=1 rx=0x7f3f6c002f30 tx=0x7f3f6c010a60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:22.277 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:22.268+0000 7f3f73fff700 1 -- 172.21.15.42:0/2103848988 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f6c005540 con 0x7f3f60046a00 2019-12-05T06:08:22.277 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:22.268+0000 7f3f7bfff700 1 --2- 172.21.15.42:0/2103848988 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f60046a00 0x7f3f60048ec0 secure :-1 s=THROTTLE_DONE pgs=1611 cs=0 l=1 rx=0x7f3f6c002f30 tx=0x7f3f6c010a60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:22.277 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:22.268+0000 7f3f7bfff700 1 --2- 172.21.15.42:0/2103848988 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f60046a00 0x7f3f60048ec0 secure :-1 s=THROTTLE_DONE pgs=1611 cs=0 l=1 rx=0x7f3f6c002f30 tx=0x7f3f6c010a60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:22.278 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:22.268+0000 7f3f73fff700 1 -- 172.21.15.42:0/2103848988 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f3f6c0059c0 con 0x7f3f60046a00 2019-12-05T06:08:22.278 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:22.268+0000 7f3f73fff700 1 -- 172.21.15.42:0/2103848988 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 258 v258) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f3f6c04ac40 con 0x7f3f60046a00 2019-12-05T06:08:22.279 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 258 2019-12-05T06:08:22.279 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:08:22.279 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":258,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6977,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/3355679996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3355679996},{"type":"v1","addr":"172.21.15.40:6833","nonce":3355679996}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":258}],"filesystems":[{"mdsmap":{"epoch":257,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:08:14.898258+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":94,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6929},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6929":{"gid":6929,"name":"a","rank":0,"incarnation":254,"state":"up:active","state_seq":19,"addr":"172.21.15.42:6835/2636699468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2636699468},{"type":"v1","addr":"172.21.15.42:6835","nonce":2636699468}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:08:22.282 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:22.276+0000 7f3f8136b700 1 -- 172.21.15.42:0/2103848988 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3f6003b370 msgr2=0x7f3f6003d880 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:22.282 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:22.276+0000 7f3f8136b700 1 --2- 172.21.15.42:0/2103848988 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3f6003b370 0x7f3f6003d880 crc :-1 s=READY pgs=894 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:22.282 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:22.276+0000 7f3f8136b700 1 -- 172.21.15.42:0/2103848988 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f60046a00 msgr2=0x7f3f60048ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:22.283 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:22.276+0000 7f3f8136b700 1 --2- 172.21.15.42:0/2103848988 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f60046a00 0x7f3f60048ec0 secure :-1 s=READY pgs=1611 cs=0 l=1 rx=0x7f3f6c002f30 tx=0x7f3f6c010a60).stop 2019-12-05T06:08:22.283 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:22.276+0000 7f3f8136b700 1 -- 172.21.15.42:0/2103848988 shutdown_connections 2019-12-05T06:08:22.283 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:22.276+0000 7f3f8136b700 1 --2- 172.21.15.42:0/2103848988 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f7c11a2d0 0x7f3f7c11bfc0 unknown :-1 s=CLOSED pgs=1610 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:22.283 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:22.276+0000 7f3f8136b700 1 --2- 172.21.15.42:0/2103848988 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f60046a00 0x7f3f60048ec0 unknown :-1 s=CLOSED pgs=1611 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:22.283 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:22.276+0000 7f3f8136b700 1 --2- 172.21.15.42:0/2103848988 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3f6003b370 0x7f3f6003d880 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:22.283 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:22.276+0000 7f3f8136b700 1 -- 172.21.15.42:0/2103848988 shutdown_connections 2019-12-05T06:08:22.284 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:22.276+0000 7f3f8136b700 1 -- 172.21.15.42:0/2103848988 wait complete. 2019-12-05T06:08:22.284 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:08:22.276+0000 7f3f8136b700 1 -- 172.21.15.42:0/2103848988 >> 172.21.15.42:0/2103848988 conn(0x7f3f7c141b80 msgr2=0x7f3f7c11d550 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:08:22.295 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b reported in up:standby state 2019-12-05T06:08:22.295 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 88 secs before thrashing 2019-12-05T06:08:44.016 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:08:44.021 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:09:14.069 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:09:14.080 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:09:44.122 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:09:44.126 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:09:50.296 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:09:50.400 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.387+0000 7fabbe366700 1 Processor -- start 2019-12-05T06:09:50.400 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.387+0000 7fabbe366700 1 -- start start 2019-12-05T06:09:50.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.387+0000 7fabbe366700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fabb808df80 0x7fabb809f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:09:50.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.387+0000 7fabbe366700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabb809fc60 con 0x7fabb80a7740 2019-12-05T06:09:50.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.387+0000 7fabbe366700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabb80a0710 con 0x7fabb809f830 2019-12-05T06:09:50.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.387+0000 7fabbe366700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fabb80a0b00 con 0x7fabb808df80 2019-12-05T06:09:50.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.391+0000 7fabb7fff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fabb808df80 0x7fabb809f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:09:50.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.391+0000 7fabbd104700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fabb809f830 0x7fabb814efe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:49462/0 (socket says 172.21.15.42:49462) 2019-12-05T06:09:50.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.391+0000 7fabbd104700 1 -- 172.21.15.42:0/685791971 learned_addr learned my addr 172.21.15.42:0/685791971 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:09:50.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.391+0000 7fabb7fff700 1 -- 172.21.15.42:0/685791971 >> v1:172.21.15.42:6789/0 conn(0x7fabb809f830 legacy=0x7fabb814efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:09:50.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.391+0000 7fabb7fff700 1 -- 172.21.15.42:0/685791971 >> v1:172.21.15.40:6789/0 conn(0x7fabb80a7740 legacy=0x7fabb807e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:09:50.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.391+0000 7fabb7fff700 1 -- 172.21.15.42:0/685791971 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabb807ef20 con 0x7fabb808df80 2019-12-05T06:09:50.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.391+0000 7fabb7fff700 1 --2- 172.21.15.42:0/685791971 >> v2:172.21.15.42:3300/0 conn(0x7fabb808df80 0x7fabb809f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7faba4008790 tx=0x7faba400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:09:50.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.391+0000 7fabb7fff700 1 --2- 172.21.15.42:0/685791971 >> v2:172.21.15.42:3300/0 conn(0x7fabb808df80 0x7fabb809f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7faba4008790 tx=0x7faba400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:09:50.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.391+0000 7fabb7fff700 1 --2- 172.21.15.42:0/685791971 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fabb808df80 0x7fabb809f2e0 secure :-1 s=READY pgs=2126 cs=0 l=1 rx=0x7faba4008790 tx=0x7faba400fbf0).ready entity=mon.1 client_cookie=a9d46b204a3b12a4 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:09:50.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.391+0000 7fabb7fff700 1 --2- 172.21.15.42:0/685791971 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fabb808df80 0x7fabb809f2e0 secure :-1 s=THROTTLE_DONE pgs=2126 cs=0 l=1 rx=0x7faba4008790 tx=0x7faba400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:09:50.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.391+0000 7fabb77fe700 1 -- 172.21.15.42:0/685791971 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faba4011270 con 0x7fabb808df80 2019-12-05T06:09:50.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.391+0000 7fabb7fff700 1 --2- 172.21.15.42:0/685791971 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fabb808df80 0x7fabb809f2e0 secure :-1 s=THROTTLE_DONE pgs=2126 cs=0 l=1 rx=0x7faba4008790 tx=0x7faba400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:09:50.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.391+0000 7fabb7fff700 1 --2- 172.21.15.42:0/685791971 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fabb808df80 0x7fabb809f2e0 secure :-1 s=THROTTLE_DONE pgs=2126 cs=0 l=1 rx=0x7faba4008790 tx=0x7faba400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:09:50.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.391+0000 7fabb77fe700 1 -- 172.21.15.42:0/685791971 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faba4008bc0 con 0x7fabb808df80 2019-12-05T06:09:50.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.391+0000 7fabb77fe700 1 -- 172.21.15.42:0/685791971 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faba4011850 con 0x7fabb808df80 2019-12-05T06:09:50.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.391+0000 7fabbe366700 1 -- 172.21.15.42:0/685791971 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fabb808df80 msgr2=0x7fabb809f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:09:50.404 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.391+0000 7fabbe366700 1 --2- 172.21.15.42:0/685791971 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fabb808df80 0x7fabb809f2e0 secure :-1 s=READY pgs=2126 cs=0 l=1 rx=0x7faba4008790 tx=0x7faba400fbf0).stop 2019-12-05T06:09:50.404 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.391+0000 7fabbe366700 1 -- 172.21.15.42:0/685791971 shutdown_connections 2019-12-05T06:09:50.404 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.391+0000 7fabbe366700 1 --2- 172.21.15.42:0/685791971 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fabb808df80 0x7fabb809f2e0 unknown :-1 s=CLOSED pgs=2126 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:09:50.404 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.391+0000 7fabbe366700 1 -- 172.21.15.42:0/685791971 shutdown_connections 2019-12-05T06:09:50.404 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.391+0000 7fabbe366700 1 -- 172.21.15.42:0/685791971 wait complete. 2019-12-05T06:09:50.404 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.391+0000 7fabbe366700 1 -- 172.21.15.42:0/685791971 >> 172.21.15.42:0/685791971 conn(0x7fabb80aabe0 msgr2=0x7fabb80a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:09:50.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.391+0000 7fabbe366700 1 Processor -- start 2019-12-05T06:09:50.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.391+0000 7fabbe366700 1 -- start start 2019-12-05T06:09:50.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.395+0000 7fabbe366700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fabb80a4080 0x7fabb805fe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:09:50.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.395+0000 7fabbe366700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fabb80a34f0 0x7fabb812ada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:09:50.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.395+0000 7fabbe366700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabb80a3a20 con 0x7fabb81240b0 2019-12-05T06:09:50.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.395+0000 7fabbe366700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fabb80604f0 con 0x7fabb80a34f0 2019-12-05T06:09:50.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.395+0000 7fabbe366700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fabb8060790 con 0x7fabb80a4080 2019-12-05T06:09:50.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.395+0000 7fabbc903700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fabb80a4080 0x7fabb805fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:09:50.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.395+0000 7fabb7fff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fabb81240b0 0x7fabb80a2cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:49466/0 (socket says 172.21.15.42:49466) 2019-12-05T06:09:50.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.395+0000 7fabb7fff700 1 -- 172.21.15.42:0/3543710219 learned_addr learned my addr 172.21.15.42:0/3543710219 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:09:50.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.395+0000 7fabbd104700 1 --2- 172.21.15.42:0/3543710219 >> v2:172.21.15.40:3300/0 conn(0x7fabb80a34f0 0x7fabb812ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:09:50.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.395+0000 7fabbc903700 1 -- 172.21.15.42:0/3543710219 >> v2:172.21.15.40:3300/0 conn(0x7fabb80a34f0 msgr2=0x7fabb812ada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:09:50.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.395+0000 7fabbc903700 1 --2- 172.21.15.42:0/3543710219 >> v2:172.21.15.40:3300/0 conn(0x7fabb80a34f0 0x7fabb812ada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:09:50.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.395+0000 7fabbc903700 1 -- 172.21.15.42:0/3543710219 >> v1:172.21.15.42:6789/0 conn(0x7fabb81240b0 legacy=0x7fabb80a2cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:09:50.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.395+0000 7fabb57fa700 1 -- 172.21.15.42:0/3543710219 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1732214955 0 0) 0x7fabb80a3a20 con 0x7fabb81240b0 2019-12-05T06:09:50.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.395+0000 7fabbc903700 1 -- 172.21.15.42:0/3543710219 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabb8144c20 con 0x7fabb80a4080 2019-12-05T06:09:50.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.395+0000 7fabbc903700 1 --2- 172.21.15.42:0/3543710219 >> v2:172.21.15.42:3300/0 conn(0x7fabb80a4080 0x7fabb805fe20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fabb80a2590 tx=0x7fabac0108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:09:50.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.395+0000 7fabbc903700 1 --2- 172.21.15.42:0/3543710219 >> v2:172.21.15.42:3300/0 conn(0x7fabb80a4080 0x7fabb805fe20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fabb80a2590 tx=0x7fabac0108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:09:50.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.395+0000 7fabbc903700 1 --2- 172.21.15.42:0/3543710219 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fabb80a4080 0x7fabb805fe20 secure :-1 s=READY pgs=2128 cs=0 l=1 rx=0x7fabb80a2590 tx=0x7fabac0108f0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:09:50.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.395+0000 7fabbc903700 1 --2- 172.21.15.42:0/3543710219 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fabb80a4080 0x7fabb805fe20 secure :-1 s=THROTTLE_DONE pgs=2128 cs=0 l=1 rx=0x7fabb80a2590 tx=0x7fabac0108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:09:50.411 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.395+0000 7fabb57fa700 1 -- 172.21.15.42:0/3543710219 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fabac00cb90 con 0x7fabb80a4080 2019-12-05T06:09:50.411 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.395+0000 7fabbc903700 1 --2- 172.21.15.42:0/3543710219 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fabb80a4080 0x7fabb805fe20 secure :-1 s=THROTTLE_DONE pgs=2128 cs=0 l=1 rx=0x7fabb80a2590 tx=0x7fabac0108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:09:50.411 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.395+0000 7fabbc903700 1 --2- 172.21.15.42:0/3543710219 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fabb80a4080 0x7fabb805fe20 secure :-1 s=THROTTLE_DONE pgs=2128 cs=0 l=1 rx=0x7fabb80a2590 tx=0x7fabac0108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:09:50.411 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.395+0000 7fabb57fa700 1 -- 172.21.15.42:0/3543710219 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabac00c7b0 con 0x7fabb80a4080 2019-12-05T06:09:50.411 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.395+0000 7fabb57fa700 1 -- 172.21.15.42:0/3543710219 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fabac011070 con 0x7fabb80a4080 2019-12-05T06:09:50.412 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.395+0000 7fabbe366700 1 -- 172.21.15.42:0/3543710219 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabb81244e0 con 0x7fabb80a4080 2019-12-05T06:09:50.412 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.395+0000 7fabbe366700 1 -- 172.21.15.42:0/3543710219 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabb812b2f0 con 0x7fabb80a4080 2019-12-05T06:09:50.412 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.395+0000 7fabbc903700 1 --2- 172.21.15.42:0/3543710219 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fabb80a4080 0x7fabb805fe20 secure :-1 s=THROTTLE_DONE pgs=2128 cs=0 l=1 rx=0x7fabb80a2590 tx=0x7fabac0108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:09:50.412 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.395+0000 7fabb57fa700 1 -- 172.21.15.42:0/3543710219 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fabac011380 con 0x7fabb80a4080 2019-12-05T06:09:50.412 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.395+0000 7fabbc903700 1 --2- 172.21.15.42:0/3543710219 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fabb80a4080 0x7fabb805fe20 secure :-1 s=THROTTLE_DONE pgs=2128 cs=0 l=1 rx=0x7fabb80a2590 tx=0x7fabac0108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:09:50.412 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.395+0000 7fabb57fa700 1 --2- 172.21.15.42:0/3543710219 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fab9803ac10 0x7fab9803d120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:09:50.412 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.395+0000 7fabb57fa700 1 -- 172.21.15.42:0/3543710219 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9901+0+0 (secure 0 0 0) 0x7fabac04b1b0 con 0x7fabb80a4080 2019-12-05T06:09:50.413 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.395+0000 7fabb7fff700 1 --2- 172.21.15.42:0/3543710219 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fab9803ac10 0x7fab9803d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:09:50.413 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.399+0000 7fabb7fff700 1 --2- 172.21.15.42:0/3543710219 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fab9803ac10 0x7fab9803d120 crc :-1 s=READY pgs=895 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:09:50.413 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.403+0000 7fabbe366700 1 -- 172.21.15.42:0/3543710219 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabb805d110 con 0x7fabb80a4080 2019-12-05T06:09:50.414 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.407+0000 7fabbc903700 1 --2- 172.21.15.42:0/3543710219 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fabb80a4080 0x7fabb805fe20 secure :-1 s=THROTTLE_DONE pgs=2128 cs=0 l=1 rx=0x7fabb80a2590 tx=0x7fabac0108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:09:50.414 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.407+0000 7fabb57fa700 1 -- 172.21.15.42:0/3543710219 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fabac032230 con 0x7fabb80a4080 2019-12-05T06:09:50.589 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.583+0000 7fabbe366700 1 -- 172.21.15.42:0/3543710219 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fabb805b8c0 con 0x7fabb80a4080 2019-12-05T06:09:50.590 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.583+0000 7fabbc903700 1 --2- 172.21.15.42:0/3543710219 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fabb80a4080 0x7fabb805fe20 secure :-1 s=THROTTLE_DONE pgs=2128 cs=0 l=1 rx=0x7fabb80a2590 tx=0x7fabac0108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:09:50.590 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.583+0000 7fabb57fa700 1 -- 172.21.15.42:0/3543710219 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 258 v258) v1 ==== 95+0+2112 (secure 0 0 0) 0x7fabac032490 con 0x7fabb80a4080 2019-12-05T06:09:50.591 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 258 2019-12-05T06:09:50.591 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:09:50.591 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":258,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6977,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/3355679996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3355679996},{"type":"v1","addr":"172.21.15.40:6833","nonce":3355679996}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":258}],"filesystems":[{"mdsmap":{"epoch":257,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:08:14.898258+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":94,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6929},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6929":{"gid":6929,"name":"a","rank":0,"incarnation":254,"state":"up:active","state_seq":19,"addr":"172.21.15.42:6835/2636699468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2636699468},{"type":"v1","addr":"172.21.15.42:6835","nonce":2636699468}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:09:50.594 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.587+0000 7fabbe366700 1 -- 172.21.15.42:0/3543710219 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fab9803ac10 msgr2=0x7fab9803d120 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:09:50.594 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.587+0000 7fabbe366700 1 --2- 172.21.15.42:0/3543710219 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fab9803ac10 0x7fab9803d120 crc :-1 s=READY pgs=895 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:09:50.594 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.587+0000 7fabbe366700 1 -- 172.21.15.42:0/3543710219 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fabb80a4080 msgr2=0x7fabb805fe20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:09:50.594 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.587+0000 7fabbe366700 1 --2- 172.21.15.42:0/3543710219 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fabb80a4080 0x7fabb805fe20 secure :-1 s=READY pgs=2128 cs=0 l=1 rx=0x7fabb80a2590 tx=0x7fabac0108f0).stop 2019-12-05T06:09:50.594 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.587+0000 7fabbe366700 1 -- 172.21.15.42:0/3543710219 shutdown_connections 2019-12-05T06:09:50.595 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.587+0000 7fabbe366700 1 --2- 172.21.15.42:0/3543710219 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fab9803ac10 0x7fab9803d120 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:09:50.595 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.587+0000 7fabbe366700 1 --2- 172.21.15.42:0/3543710219 >> v2:172.21.15.40:3300/0 conn(0x7fabb80a34f0 0x7fabb812ada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:09:50.595 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.587+0000 7fabbe366700 1 --2- 172.21.15.42:0/3543710219 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fabb80a4080 0x7fabb805fe20 unknown :-1 s=CLOSED pgs=2128 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:09:50.595 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.587+0000 7fabbe366700 1 -- 172.21.15.42:0/3543710219 shutdown_connections 2019-12-05T06:09:50.595 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.587+0000 7fabbe366700 1 -- 172.21.15.42:0/3543710219 wait complete. 2019-12-05T06:09:50.595 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.587+0000 7fabbe366700 1 -- 172.21.15.42:0/3543710219 >> 172.21.15.42:0/3543710219 conn(0x7fabb805a190 msgr2=0x7fabb81418d0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:09:50.604 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.a (rank=0) 2019-12-05T06:09:50.605 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2019-12-05T06:09:50.605 INFO:teuthology.orchestra.run:waiting for 300 2019-12-05T06:09:50.614 INFO:tasks.ceph.mds.a:Stopped 2019-12-05T06:09:50.614 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:09:50.786 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.771+0000 7f4a4942b700 1 Processor -- start 2019-12-05T06:09:50.787 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.771+0000 7f4a4942b700 1 -- start start 2019-12-05T06:09:50.787 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.771+0000 7f4a4942b700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f4a4409fae0 0x7f4a4414fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:09:50.787 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.771+0000 7f4a4942b700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a440a0710 con 0x7f4a440a7740 2019-12-05T06:09:50.787 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.771+0000 7f4a4942b700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a4407ef20 con 0x7f4a4408df80 2019-12-05T06:09:50.787 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.771+0000 7f4a4942b700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a440a0b00 con 0x7f4a4409fae0 2019-12-05T06:09:50.787 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.771+0000 7f4a42ffd700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f4a4408df80 0x7f4a4409f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:49476/0 (socket says 172.21.15.42:49476) 2019-12-05T06:09:50.788 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.771+0000 7f4a42ffd700 1 -- 172.21.15.42:0/463338796 learned_addr learned my addr 172.21.15.42:0/463338796 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:09:50.788 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.771+0000 7f4a43fff700 1 --2- 172.21.15.42:0/463338796 >> v2:172.21.15.40:3300/0 conn(0x7f4a4409fae0 0x7f4a4414fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:09:50.788 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.771+0000 7f4a427fc700 1 -- 172.21.15.42:0/463338796 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3682983903 0 0) 0x7f4a4407ef20 con 0x7f4a4408df80 2019-12-05T06:09:50.789 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.771+0000 7f4a427fc700 1 -- 172.21.15.42:0/463338796 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a240030b0 con 0x7f4a4408df80 2019-12-05T06:09:50.789 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.771+0000 7f4a427fc700 1 -- 172.21.15.42:0/463338796 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2312997384 0 0) 0x7f4a440a0710 con 0x7f4a440a7740 2019-12-05T06:09:50.789 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.771+0000 7f4a427fc700 1 -- 172.21.15.42:0/463338796 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a4407ef20 con 0x7f4a440a7740 2019-12-05T06:09:50.789 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.771+0000 7f4a427fc700 1 -- 172.21.15.42:0/463338796 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2922036543 0 0) 0x7f4a240030b0 con 0x7f4a4408df80 2019-12-05T06:09:50.789 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.771+0000 7f4a427fc700 1 -- 172.21.15.42:0/463338796 >> v2:172.21.15.40:3300/0 conn(0x7f4a4409fae0 msgr2=0x7f4a4414fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:09:50.789 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.771+0000 7f4a427fc700 1 --2- 172.21.15.42:0/463338796 >> v2:172.21.15.40:3300/0 conn(0x7f4a4409fae0 0x7f4a4414fff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:09:50.789 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.771+0000 7f4a427fc700 1 -- 172.21.15.42:0/463338796 >> v1:172.21.15.40:6789/0 conn(0x7f4a440a7740 legacy=0x7f4a4407e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:09:50.790 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.771+0000 7f4a427fc700 1 -- 172.21.15.42:0/463338796 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a440a9ba0 con 0x7f4a4408df80 2019-12-05T06:09:50.790 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.771+0000 7f4a427fc700 1 -- 172.21.15.42:0/463338796 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7f4a34001ad0 con 0x7f4a4408df80 2019-12-05T06:09:50.790 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.771+0000 7f4a427fc700 1 -- 172.21.15.42:0/463338796 >> v1:172.21.15.42:6789/0 conn(0x7f4a4408df80 legacy=0x7f4a4409f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:09:50.790 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.775+0000 7f4a427fc700 1 --2- 172.21.15.42:0/463338796 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4a24003960 0x7f4a24003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:09:50.790 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.775+0000 7f4a427fc700 1 --2- 172.21.15.42:0/463338796 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4a24004470 0x7f4a24004940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:09:50.790 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.775+0000 7f4a427fc700 1 --2- 172.21.15.42:0/463338796 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4a24004fa0 0x7f4a2404f700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:09:50.791 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.775+0000 7f4a427fc700 1 -- 172.21.15.42:0/463338796 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a440a0b00 con 0x7f4a24004470 2019-12-05T06:09:50.791 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.775+0000 7f4a427fc700 1 -- 172.21.15.42:0/463338796 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a2404fec0 con 0x7f4a24004fa0 2019-12-05T06:09:50.791 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.775+0000 7f4a427fc700 1 -- 172.21.15.42:0/463338796 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a24050100 con 0x7f4a24003960 2019-12-05T06:09:50.791 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.775+0000 7f4a42ffd700 1 --2- 172.21.15.42:0/463338796 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4a24004470 0x7f4a24004940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:09:50.791 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.775+0000 7f4a437fe700 1 --2- 172.21.15.42:0/463338796 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4a24003960 0x7f4a24003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:09:50.792 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.775+0000 7f4a43fff700 1 --2- 172.21.15.42:0/463338796 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4a24004fa0 0x7f4a2404f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:09:50.792 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.775+0000 7f4a437fe700 1 -- 172.21.15.42:0/463338796 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4a24004fa0 msgr2=0x7f4a2404f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:09:50.792 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.775+0000 7f4a437fe700 1 --2- 172.21.15.42:0/463338796 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4a24004fa0 0x7f4a2404f700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:09:50.792 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.775+0000 7f4a437fe700 1 -- 172.21.15.42:0/463338796 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4a24004470 msgr2=0x7f4a24004940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:09:50.792 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.775+0000 7f4a437fe700 1 --2- 172.21.15.42:0/463338796 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4a24004470 0x7f4a24004940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:09:50.792 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.775+0000 7f4a437fe700 1 -- 172.21.15.42:0/463338796 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a24050400 con 0x7f4a24003960 2019-12-05T06:09:50.792 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.775+0000 7f4a43fff700 1 -- 172.21.15.42:0/463338796 reap_dead start 2019-12-05T06:09:50.793 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.775+0000 7f4a437fe700 1 --2- 172.21.15.42:0/463338796 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4a24003960 0x7f4a24003db0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f4a2c000fe0 tx=0x7f4a2c00be70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:09:50.793 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.775+0000 7f4a437fe700 1 --2- 172.21.15.42:0/463338796 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4a24003960 0x7f4a24003db0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f4a2c000fe0 tx=0x7f4a2c00be70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:09:50.793 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.775+0000 7f4a437fe700 1 --2- 172.21.15.42:0/463338796 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4a24003960 0x7f4a24003db0 secure :-1 s=READY pgs=469 cs=0 l=1 rx=0x7f4a2c000fe0 tx=0x7f4a2c00be70).ready entity=mon.2 client_cookie=81d24be6a3a334d6 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:09:50.793 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.775+0000 7f4a437fe700 1 --2- 172.21.15.42:0/463338796 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4a24003960 0x7f4a24003db0 secure :-1 s=THROTTLE_DONE pgs=469 cs=0 l=1 rx=0x7f4a2c000fe0 tx=0x7f4a2c00be70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:09:50.793 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.775+0000 7f4a427fc700 1 -- 172.21.15.42:0/463338796 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4a2c010be0 con 0x7f4a24003960 2019-12-05T06:09:50.793 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.775+0000 7f4a437fe700 1 --2- 172.21.15.42:0/463338796 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4a24003960 0x7f4a24003db0 secure :-1 s=THROTTLE_DONE pgs=469 cs=0 l=1 rx=0x7f4a2c000fe0 tx=0x7f4a2c00be70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:09:50.793 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.775+0000 7f4a437fe700 1 --2- 172.21.15.42:0/463338796 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4a24003960 0x7f4a24003db0 secure :-1 s=THROTTLE_DONE pgs=469 cs=0 l=1 rx=0x7f4a2c000fe0 tx=0x7f4a2c00be70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:09:50.794 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.775+0000 7f4a427fc700 1 -- 172.21.15.42:0/463338796 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a2c010800 con 0x7f4a24003960 2019-12-05T06:09:50.794 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.775+0000 7f4a427fc700 1 -- 172.21.15.42:0/463338796 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4a2c00c5b0 con 0x7f4a24003960 2019-12-05T06:09:50.794 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.775+0000 7f4a4942b700 1 -- 172.21.15.42:0/463338796 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4a24003960 msgr2=0x7f4a24003db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:09:50.794 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.775+0000 7f4a4942b700 1 --2- 172.21.15.42:0/463338796 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4a24003960 0x7f4a24003db0 secure :-1 s=READY pgs=469 cs=0 l=1 rx=0x7f4a2c000fe0 tx=0x7f4a2c00be70).stop 2019-12-05T06:09:50.794 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.775+0000 7f4a4942b700 1 -- 172.21.15.42:0/463338796 shutdown_connections 2019-12-05T06:09:50.794 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.775+0000 7f4a4942b700 1 --2- 172.21.15.42:0/463338796 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4a24003960 0x7f4a24003db0 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:09:50.794 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.775+0000 7f4a4942b700 1 -- 172.21.15.42:0/463338796 shutdown_connections 2019-12-05T06:09:50.794 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.775+0000 7f4a4942b700 1 -- 172.21.15.42:0/463338796 wait complete. 2019-12-05T06:09:50.795 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.775+0000 7f4a4942b700 1 -- 172.21.15.42:0/463338796 >> 172.21.15.42:0/463338796 conn(0x7f4a440aabe0 msgr2=0x7f4a440a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:09:50.795 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.779+0000 7f4a4942b700 1 Processor -- start 2019-12-05T06:09:50.795 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.779+0000 7f4a4942b700 1 -- start start 2019-12-05T06:09:50.799 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.779+0000 7f4a4942b700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4a4411a6c0 0x7f4a44126480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:09:50.799 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.779+0000 7f4a4942b700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f4a440732f0 0x7f4a44073720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:09:50.799 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.783+0000 7f4a4942b700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a44126da0 con 0x7f4a441243b0 2019-12-05T06:09:50.799 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.783+0000 7f4a4942b700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a4411bbf0 con 0x7f4a440732f0 2019-12-05T06:09:50.799 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.783+0000 7f4a4942b700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a4411be90 con 0x7f4a4411a6c0 2019-12-05T06:09:50.799 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.783+0000 7f4a43fff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f4a440732f0 0x7f4a44073720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:09:50.800 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.783+0000 7f4a43fff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f4a440732f0 0x7f4a44073720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:49452/0 (socket says 172.21.15.42:49452) 2019-12-05T06:09:50.800 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.783+0000 7f4a43fff700 1 -- 172.21.15.42:0/3706187695 learned_addr learned my addr 172.21.15.42:0/3706187695 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:09:50.800 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.783+0000 7f4a42ffd700 1 --2- 172.21.15.42:0/3706187695 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4a4411a6c0 0x7f4a44126480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:09:50.800 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.783+0000 7f4a43fff700 1 -- 172.21.15.42:0/3706187695 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4a4411a6c0 msgr2=0x7f4a44126480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:09:50.800 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.783+0000 7f4a43fff700 1 --2- 172.21.15.42:0/3706187695 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4a4411a6c0 0x7f4a44126480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:09:50.800 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.783+0000 7f4a43fff700 1 -- 172.21.15.42:0/3706187695 >> v1:172.21.15.40:6789/0 conn(0x7f4a441243b0 legacy=0x7f4a44125c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:09:50.801 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.783+0000 7f4a3bfff700 1 -- 172.21.15.42:0/3706187695 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 502475304 0 0) 0x7f4a44126da0 con 0x7f4a441243b0 2019-12-05T06:09:50.801 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.783+0000 7f4a43fff700 1 -- 172.21.15.42:0/3706187695 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a4405ae30 con 0x7f4a440732f0 2019-12-05T06:09:50.801 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.783+0000 7f4a43fff700 1 --2- 172.21.15.42:0/3706187695 >> v2:172.21.15.42:3300/0 conn(0x7f4a440732f0 0x7f4a44073720 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f4a34001490 tx=0x7f4a3c010930).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:09:50.801 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.783+0000 7f4a43fff700 1 --2- 172.21.15.42:0/3706187695 >> v2:172.21.15.42:3300/0 conn(0x7f4a440732f0 0x7f4a44073720 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f4a34001490 tx=0x7f4a3c010930).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:09:50.802 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.783+0000 7f4a43fff700 1 --2- 172.21.15.42:0/3706187695 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4a440732f0 0x7f4a44073720 secure :-1 s=READY pgs=2130 cs=0 l=1 rx=0x7f4a34001490 tx=0x7f4a3c010930).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:09:50.802 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.783+0000 7f4a43fff700 1 --2- 172.21.15.42:0/3706187695 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4a440732f0 0x7f4a44073720 secure :-1 s=THROTTLE_DONE pgs=2130 cs=0 l=1 rx=0x7f4a34001490 tx=0x7f4a3c010930).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:09:50.802 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.783+0000 7f4a3bfff700 1 -- 172.21.15.42:0/3706187695 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4a3c005840 con 0x7f4a440732f0 2019-12-05T06:09:50.802 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.783+0000 7f4a43fff700 1 --2- 172.21.15.42:0/3706187695 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4a440732f0 0x7f4a44073720 secure :-1 s=THROTTLE_DONE pgs=2130 cs=0 l=1 rx=0x7f4a34001490 tx=0x7f4a3c010930).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:09:50.802 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.783+0000 7f4a43fff700 1 --2- 172.21.15.42:0/3706187695 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4a440732f0 0x7f4a44073720 secure :-1 s=THROTTLE_DONE pgs=2130 cs=0 l=1 rx=0x7f4a34001490 tx=0x7f4a3c010930).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:09:50.802 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.783+0000 7f4a3bfff700 1 -- 172.21.15.42:0/3706187695 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a3c007160 con 0x7f4a440732f0 2019-12-05T06:09:50.802 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.783+0000 7f4a4942b700 1 -- 172.21.15.42:0/3706187695 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a4411b790 con 0x7f4a440732f0 2019-12-05T06:09:50.803 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.783+0000 7f4a3bfff700 1 -- 172.21.15.42:0/3706187695 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4a3c0082f0 con 0x7f4a440732f0 2019-12-05T06:09:50.803 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.783+0000 7f4a4942b700 1 -- 172.21.15.42:0/3706187695 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a44073e90 con 0x7f4a440732f0 2019-12-05T06:09:50.803 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.783+0000 7f4a43fff700 1 --2- 172.21.15.42:0/3706187695 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4a440732f0 0x7f4a44073720 secure :-1 s=THROTTLE_DONE pgs=2130 cs=0 l=1 rx=0x7f4a34001490 tx=0x7f4a3c010930).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:09:50.803 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.783+0000 7f4a3bfff700 1 -- 172.21.15.42:0/3706187695 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f4a3c008540 con 0x7f4a440732f0 2019-12-05T06:09:50.803 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.783+0000 7f4a43fff700 1 --2- 172.21.15.42:0/3706187695 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4a440732f0 0x7f4a44073720 secure :-1 s=THROTTLE_DONE pgs=2130 cs=0 l=1 rx=0x7f4a34001490 tx=0x7f4a3c010930).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:09:50.803 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.787+0000 7f4a3bfff700 1 --2- 172.21.15.42:0/3706187695 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4a1c03abc0 0x7f4a1c03d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:09:50.803 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.787+0000 7f4a3bfff700 1 -- 172.21.15.42:0/3706187695 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9901+0+0 (secure 0 0 0) 0x7f4a3c04ace0 con 0x7f4a440732f0 2019-12-05T06:09:50.804 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.787+0000 7f4a437fe700 1 --2- 172.21.15.42:0/3706187695 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4a1c03abc0 0x7f4a1c03d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:09:50.804 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.787+0000 7f4a437fe700 1 --2- 172.21.15.42:0/3706187695 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4a1c03abc0 0x7f4a1c03d0d0 crc :-1 s=READY pgs=896 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:09:50.804 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.787+0000 7f4a4942b700 1 -- 172.21.15.42:0/3706187695 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a4411c050 con 0x7f4a440732f0 2019-12-05T06:09:50.805 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.795+0000 7f4a43fff700 1 --2- 172.21.15.42:0/3706187695 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4a440732f0 0x7f4a44073720 secure :-1 s=THROTTLE_DONE pgs=2130 cs=0 l=1 rx=0x7f4a34001490 tx=0x7f4a3c010930).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:09:50.805 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.795+0000 7f4a3bfff700 1 -- 172.21.15.42:0/3706187695 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f4a3c031c80 con 0x7f4a440732f0 2019-12-05T06:09:50.916 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~0s 2019-12-05T06:09:50.964 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.959+0000 7f4a4942b700 1 -- 172.21.15.42:0/3706187695 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4a4405d110 con 0x7f4a440732f0 2019-12-05T06:09:50.964 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.959+0000 7f4a43fff700 1 --2- 172.21.15.42:0/3706187695 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4a440732f0 0x7f4a44073720 secure :-1 s=THROTTLE_DONE pgs=2130 cs=0 l=1 rx=0x7f4a34001490 tx=0x7f4a3c010930).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:09:50.965 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.959+0000 7f4a3bfff700 1 -- 172.21.15.42:0/3706187695 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 258 v258) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f4a3c031ee0 con 0x7f4a440732f0 2019-12-05T06:09:50.965 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 258 2019-12-05T06:09:50.965 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:09:50.965 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":258,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6977,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/3355679996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3355679996},{"type":"v1","addr":"172.21.15.40:6833","nonce":3355679996}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":258}],"filesystems":[{"mdsmap":{"epoch":257,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:08:14.898258+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":94,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6929},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6929":{"gid":6929,"name":"a","rank":0,"incarnation":254,"state":"up:active","state_seq":19,"addr":"172.21.15.42:6835/2636699468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2636699468},{"type":"v1","addr":"172.21.15.42:6835","nonce":2636699468}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:09:50.967 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.959+0000 7f4a4942b700 1 -- 172.21.15.42:0/3706187695 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4a1c03abc0 msgr2=0x7f4a1c03d0d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:09:50.967 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.959+0000 7f4a4942b700 1 --2- 172.21.15.42:0/3706187695 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4a1c03abc0 0x7f4a1c03d0d0 crc :-1 s=READY pgs=896 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:09:50.968 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.959+0000 7f4a4942b700 1 -- 172.21.15.42:0/3706187695 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4a440732f0 msgr2=0x7f4a44073720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:09:50.968 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.959+0000 7f4a4942b700 1 --2- 172.21.15.42:0/3706187695 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4a440732f0 0x7f4a44073720 secure :-1 s=READY pgs=2130 cs=0 l=1 rx=0x7f4a34001490 tx=0x7f4a3c010930).stop 2019-12-05T06:09:50.968 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.959+0000 7f4a4942b700 1 -- 172.21.15.42:0/3706187695 shutdown_connections 2019-12-05T06:09:50.969 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.959+0000 7f4a4942b700 1 --2- 172.21.15.42:0/3706187695 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4a1c03abc0 0x7f4a1c03d0d0 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:09:50.970 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.959+0000 7f4a4942b700 1 --2- 172.21.15.42:0/3706187695 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4a440732f0 0x7f4a44073720 unknown :-1 s=CLOSED pgs=2130 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:09:50.970 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.959+0000 7f4a4942b700 1 --2- 172.21.15.42:0/3706187695 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4a4411a6c0 0x7f4a44126480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:09:50.970 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.959+0000 7f4a4942b700 1 -- 172.21.15.42:0/3706187695 shutdown_connections 2019-12-05T06:09:50.970 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.959+0000 7f4a4942b700 1 -- 172.21.15.42:0/3706187695 wait complete. 2019-12-05T06:09:50.970 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:50.959+0000 7f4a4942b700 1 -- 172.21.15.42:0/3706187695 >> 172.21.15.42:0/3706187695 conn(0x7f4a4408fde0 msgr2=0x7f4a4408eb70 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:09:50.977 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:09:52.978 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:09:53.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:53.075+0000 7f4621f70700 1 Processor -- start 2019-12-05T06:09:53.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:53.075+0000 7f4621f70700 1 -- start start 2019-12-05T06:09:53.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:53.075+0000 7f4621f70700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f461c0a7740 0x7f461c08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:09:53.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:53.075+0000 7f4621f70700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f461c09fae0 0x7f461c07eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:09:53.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:53.075+0000 7f4621f70700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f461c0a0710 con 0x7f461c07e720 2019-12-05T06:09:53.088 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:53.075+0000 7f4621f70700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f461c0a0b00 con 0x7f461c09fae0 2019-12-05T06:09:53.088 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:53.075+0000 7f4621f70700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f461c08e5b0 con 0x7f461c0a7740 2019-12-05T06:09:53.088 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:53.075+0000 7f461bfff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f461c0a7740 0x7f461c08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:09:53.088 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:53.075+0000 7f461bfff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f461c0a7740 0x7f461c08df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:49466/0 (socket says 172.21.15.42:49466) 2019-12-05T06:09:53.088 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:53.075+0000 7f461bfff700 1 -- 172.21.15.42:0/3777462900 learned_addr learned my addr 172.21.15.42:0/3777462900 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:09:53.088 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:53.075+0000 7f4620d0e700 1 --2- 172.21.15.42:0/3777462900 >> v2:172.21.15.40:3300/0 conn(0x7f461c09fae0 0x7f461c07eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:09:53.088 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:53.079+0000 7f461bfff700 1 -- 172.21.15.42:0/3777462900 >> v2:172.21.15.40:3300/0 conn(0x7f461c09fae0 msgr2=0x7f461c07eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:09:53.089 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:53.079+0000 7f461bfff700 1 --2- 172.21.15.42:0/3777462900 >> v2:172.21.15.40:3300/0 conn(0x7f461c09fae0 0x7f461c07eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:09:53.089 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:53.079+0000 7f461bfff700 1 -- 172.21.15.42:0/3777462900 >> v1:172.21.15.40:6789/0 conn(0x7f461c07e720 legacy=0x7f461c09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:09:53.089 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:53.079+0000 7f461bfff700 1 -- 172.21.15.42:0/3777462900 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f461c07f0a0 con 0x7f461c0a7740 2019-12-05T06:09:53.089 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:53.079+0000 7f461bfff700 1 --2- 172.21.15.42:0/3777462900 >> v2:172.21.15.42:3300/0 conn(0x7f461c0a7740 0x7f461c08df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f460c002f30 tx=0x7f460c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:09:53.089 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:53.079+0000 7f461bfff700 1 --2- 172.21.15.42:0/3777462900 >> v2:172.21.15.42:3300/0 conn(0x7f461c0a7740 0x7f461c08df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f460c002f30 tx=0x7f460c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:09:53.089 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:53.079+0000 7f461bfff700 1 --2- 172.21.15.42:0/3777462900 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f461c0a7740 0x7f461c08df80 secure :-1 s=READY pgs=2131 cs=0 l=1 rx=0x7f460c002f30 tx=0x7f460c00fd30).ready entity=mon.1 client_cookie=96e2f2b83fd0f926 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:09:53.089 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:53.079+0000 7f461bfff700 1 --2- 172.21.15.42:0/3777462900 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f461c0a7740 0x7f461c08df80 secure :-1 s=THROTTLE_DONE pgs=2131 cs=0 l=1 rx=0x7f460c002f30 tx=0x7f460c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:09:53.090 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:53.079+0000 7f461bfff700 1 --2- 172.21.15.42:0/3777462900 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f461c0a7740 0x7f461c08df80 secure :-1 s=THROTTLE_DONE pgs=2131 cs=0 l=1 rx=0x7f460c002f30 tx=0x7f460c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:09:53.090 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:53.079+0000 7f461affd700 1 -- 172.21.15.42:0/3777462900 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f460c008c90 con 0x7f461c0a7740 2019-12-05T06:09:53.090 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:53.079+0000 7f461bfff700 1 --2- 172.21.15.42:0/3777462900 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f461c0a7740 0x7f461c08df80 secure :-1 s=THROTTLE_DONE pgs=2131 cs=0 l=1 rx=0x7f460c002f30 tx=0x7f460c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:09:53.090 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:53.079+0000 7f461affd700 1 -- 172.21.15.42:0/3777462900 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f460c0088b0 con 0x7f461c0a7740 2019-12-05T06:09:53.090 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:53.079+0000 7f461affd700 1 -- 172.21.15.42:0/3777462900 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f460c011ad0 con 0x7f461c0a7740 2019-12-05T06:09:53.091 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:53.079+0000 7f4621f70700 1 -- 172.21.15.42:0/3777462900 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f461c0a7740 msgr2=0x7f461c08df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:09:53.091 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:53.079+0000 7f4621f70700 1 --2- 172.21.15.42:0/3777462900 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f461c0a7740 0x7f461c08df80 secure :-1 s=READY pgs=2131 cs=0 l=1 rx=0x7f460c002f30 tx=0x7f460c00fd30).stop 2019-12-05T06:09:53.091 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:53.079+0000 7f4621f70700 1 -- 172.21.15.42:0/3777462900 shutdown_connections 2019-12-05T06:09:53.091 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:53.079+0000 7f4621f70700 1 --2- 172.21.15.42:0/3777462900 >> v2:172.21.15.40:3300/0 conn(0x7f461c09fae0 0x7f461c07eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:09:53.091 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:53.079+0000 7f4621f70700 1 --2- 172.21.15.42:0/3777462900 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f461c0a7740 0x7f461c08df80 unknown :-1 s=CLOSED pgs=2131 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:09:53.091 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:53.079+0000 7f4621f70700 1 -- 172.21.15.42:0/3777462900 shutdown_connections 2019-12-05T06:09:53.092 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:53.079+0000 7f4621f70700 1 -- 172.21.15.42:0/3777462900 wait complete. 2019-12-05T06:09:53.092 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:53.079+0000 7f4621f70700 1 -- 172.21.15.42:0/3777462900 >> 172.21.15.42:0/3777462900 conn(0x7f461c0aabe0 msgr2=0x7f461c0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:09:53.094 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:53.083+0000 7f4621f70700 1 Processor -- start 2019-12-05T06:09:53.095 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:53.083+0000 7f4621f70700 1 -- start start 2019-12-05T06:09:53.095 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:53.083+0000 7f4621f70700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f461c118840 0x7f461c07bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:09:53.095 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:53.083+0000 7f4621f70700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f461c07aff0 0x7f461c07b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:09:53.095 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:53.083+0000 7f4621f70700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f461c07d240 con 0x7f461c130510 2019-12-05T06:09:53.095 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:53.083+0000 7f4621f70700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f461c07d670 con 0x7f461c07aff0 2019-12-05T06:09:53.095 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:53.083+0000 7f4621f70700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f461c07d910 con 0x7f461c118840 2019-12-05T06:09:53.096 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:53.083+0000 7f461b7fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f461c07aff0 0x7f461c07b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:09:53.096 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:53.083+0000 7f4620d0e700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f461c130510 0x7f461c130940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:49512/0 (socket says 172.21.15.42:49512) 2019-12-05T06:09:53.096 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:53.083+0000 7f4620d0e700 1 -- 172.21.15.42:0/2792168448 learned_addr learned my addr 172.21.15.42:0/2792168448 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:09:53.096 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:53.083+0000 7f461bfff700 1 --2- 172.21.15.42:0/2792168448 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f461c118840 0x7f461c07bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:09:53.096 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:53.083+0000 7f461b7fe700 1 -- 172.21.15.42:0/2792168448 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f461c118840 msgr2=0x7f461c07bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:09:53.096 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:53.083+0000 7f461b7fe700 1 --2- 172.21.15.42:0/2792168448 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f461c118840 0x7f461c07bf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:09:53.096 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:53.083+0000 7f461b7fe700 1 -- 172.21.15.42:0/2792168448 >> v1:172.21.15.42:6789/0 conn(0x7f461c130510 legacy=0x7f461c130940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:09:53.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:53.083+0000 7f461b7fe700 1 -- 172.21.15.42:0/2792168448 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f461c0a7040 con 0x7f461c07aff0 2019-12-05T06:09:53.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:53.083+0000 7f4618ff9700 1 -- 172.21.15.42:0/2792168448 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3984103002 0 0) 0x7f461c07d240 con 0x7f461c130510 2019-12-05T06:09:53.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:53.083+0000 7f461b7fe700 1 --2- 172.21.15.42:0/2792168448 >> v2:172.21.15.42:3300/0 conn(0x7f461c07aff0 0x7f461c07b420 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f461c07d830 tx=0x7f460400c320).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:09:53.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:53.083+0000 7f461b7fe700 1 --2- 172.21.15.42:0/2792168448 >> v2:172.21.15.42:3300/0 conn(0x7f461c07aff0 0x7f461c07b420 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f461c07d830 tx=0x7f460400c320).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:09:53.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:53.083+0000 7f461b7fe700 1 --2- 172.21.15.42:0/2792168448 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f461c07aff0 0x7f461c07b420 secure :-1 s=READY pgs=2133 cs=0 l=1 rx=0x7f461c07d830 tx=0x7f460400c320).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:09:53.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:53.083+0000 7f461b7fe700 1 --2- 172.21.15.42:0/2792168448 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f461c07aff0 0x7f461c07b420 secure :-1 s=THROTTLE_DONE pgs=2133 cs=0 l=1 rx=0x7f461c07d830 tx=0x7f460400c320).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:09:53.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:53.083+0000 7f4618ff9700 1 -- 172.21.15.42:0/2792168448 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f460400ca80 con 0x7f461c07aff0 2019-12-05T06:09:53.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:53.083+0000 7f461b7fe700 1 --2- 172.21.15.42:0/2792168448 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f461c07aff0 0x7f461c07b420 secure :-1 s=THROTTLE_DONE pgs=2133 cs=0 l=1 rx=0x7f461c07d830 tx=0x7f460400c320).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:09:53.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:53.083+0000 7f461b7fe700 1 --2- 172.21.15.42:0/2792168448 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f461c07aff0 0x7f461c07b420 secure :-1 s=THROTTLE_DONE pgs=2133 cs=0 l=1 rx=0x7f461c07d830 tx=0x7f460400c320).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:09:53.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:53.083+0000 7f4618ff9700 1 -- 172.21.15.42:0/2792168448 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f460400c6a0 con 0x7f461c07aff0 2019-12-05T06:09:53.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:53.083+0000 7f4621f70700 1 -- 172.21.15.42:0/2792168448 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f461c1377e0 con 0x7f461c07aff0 2019-12-05T06:09:53.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:53.083+0000 7f4618ff9700 1 -- 172.21.15.42:0/2792168448 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f46040110e0 con 0x7f461c07aff0 2019-12-05T06:09:53.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:53.083+0000 7f4621f70700 1 -- 172.21.15.42:0/2792168448 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f461c137d90 con 0x7f461c07aff0 2019-12-05T06:09:53.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:53.087+0000 7f461b7fe700 1 --2- 172.21.15.42:0/2792168448 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f461c07aff0 0x7f461c07b420 secure :-1 s=THROTTLE_DONE pgs=2133 cs=0 l=1 rx=0x7f461c07d830 tx=0x7f460400c320).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:09:53.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:53.087+0000 7f4618ff9700 1 -- 172.21.15.42:0/2792168448 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f46040113b0 con 0x7f461c07aff0 2019-12-05T06:09:53.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:53.087+0000 7f461b7fe700 1 --2- 172.21.15.42:0/2792168448 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f461c07aff0 0x7f461c07b420 secure :-1 s=THROTTLE_DONE pgs=2133 cs=0 l=1 rx=0x7f461c07d830 tx=0x7f460400c320).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:09:53.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:53.087+0000 7f4618ff9700 1 --2- 172.21.15.42:0/2792168448 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f45f403ace0 0x7f45f403d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:09:53.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:53.087+0000 7f4618ff9700 1 -- 172.21.15.42:0/2792168448 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9901+0+0 (secure 0 0 0) 0x7f460404b150 con 0x7f461c07aff0 2019-12-05T06:09:53.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:53.087+0000 7f461bfff700 1 --2- 172.21.15.42:0/2792168448 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f45f403ace0 0x7f45f403d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:09:53.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:53.087+0000 7f461bfff700 1 --2- 172.21.15.42:0/2792168448 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f45f403ace0 0x7f45f403d1f0 crc :-1 s=READY pgs=897 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:09:53.100 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:53.091+0000 7f4621f70700 1 -- 172.21.15.42:0/2792168448 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f461c07d470 con 0x7f461c07aff0 2019-12-05T06:09:53.108 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:53.103+0000 7f461b7fe700 1 --2- 172.21.15.42:0/2792168448 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f461c07aff0 0x7f461c07b420 secure :-1 s=THROTTLE_DONE pgs=2133 cs=0 l=1 rx=0x7f461c07d830 tx=0x7f460400c320).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:09:53.108 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:53.103+0000 7f4618ff9700 1 -- 172.21.15.42:0/2792168448 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f4604032190 con 0x7f461c07aff0 2019-12-05T06:09:53.277 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:53.271+0000 7f4621f70700 1 -- 172.21.15.42:0/2792168448 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f461c05d110 con 0x7f461c07aff0 2019-12-05T06:09:53.278 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:53.271+0000 7f461b7fe700 1 --2- 172.21.15.42:0/2792168448 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f461c07aff0 0x7f461c07b420 secure :-1 s=THROTTLE_DONE pgs=2133 cs=0 l=1 rx=0x7f461c07d830 tx=0x7f460400c320).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:09:53.278 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:53.271+0000 7f4618ff9700 1 -- 172.21.15.42:0/2792168448 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 258 v258) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f46040323f0 con 0x7f461c07aff0 2019-12-05T06:09:53.278 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 258 2019-12-05T06:09:53.279 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:09:53.279 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":258,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6977,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/3355679996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3355679996},{"type":"v1","addr":"172.21.15.40:6833","nonce":3355679996}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":258}],"filesystems":[{"mdsmap":{"epoch":257,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:08:14.898258+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":94,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6929},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6929":{"gid":6929,"name":"a","rank":0,"incarnation":254,"state":"up:active","state_seq":19,"addr":"172.21.15.42:6835/2636699468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2636699468},{"type":"v1","addr":"172.21.15.42:6835","nonce":2636699468}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:09:53.281 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:53.275+0000 7f4621f70700 1 -- 172.21.15.42:0/2792168448 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f45f403ace0 msgr2=0x7f45f403d1f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:09:53.281 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:53.275+0000 7f4621f70700 1 --2- 172.21.15.42:0/2792168448 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f45f403ace0 0x7f45f403d1f0 crc :-1 s=READY pgs=897 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:09:53.281 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:53.275+0000 7f4621f70700 1 -- 172.21.15.42:0/2792168448 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f461c07aff0 msgr2=0x7f461c07b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:09:53.281 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:53.275+0000 7f4621f70700 1 --2- 172.21.15.42:0/2792168448 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f461c07aff0 0x7f461c07b420 secure :-1 s=READY pgs=2133 cs=0 l=1 rx=0x7f461c07d830 tx=0x7f460400c320).stop 2019-12-05T06:09:53.281 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:53.275+0000 7f4621f70700 1 -- 172.21.15.42:0/2792168448 shutdown_connections 2019-12-05T06:09:53.281 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:53.275+0000 7f4621f70700 1 --2- 172.21.15.42:0/2792168448 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f45f403ace0 0x7f45f403d1f0 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:09:53.282 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:53.275+0000 7f4621f70700 1 --2- 172.21.15.42:0/2792168448 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f461c118840 0x7f461c07bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:09:53.282 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:53.275+0000 7f4621f70700 1 --2- 172.21.15.42:0/2792168448 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f461c07aff0 0x7f461c07b420 unknown :-1 s=CLOSED pgs=2133 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:09:53.282 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:53.275+0000 7f4621f70700 1 -- 172.21.15.42:0/2792168448 shutdown_connections 2019-12-05T06:09:53.282 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:53.275+0000 7f4621f70700 1 -- 172.21.15.42:0/2792168448 wait complete. 2019-12-05T06:09:53.282 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:53.275+0000 7f4621f70700 1 -- 172.21.15.42:0/2792168448 >> 172.21.15.42:0/2792168448 conn(0x7f461c08fda0 msgr2=0x7f461c09f620 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:09:53.291 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:09:55.292 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:09:55.395 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:55.383+0000 7fca30551700 1 Processor -- start 2019-12-05T06:09:55.396 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:55.383+0000 7fca30551700 1 -- start start 2019-12-05T06:09:55.396 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:55.387+0000 7fca30551700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fca2808df80 0x7fca2809f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:09:55.396 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:55.387+0000 7fca30551700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fca2809f830 0x7fca2814efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:09:55.396 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:55.387+0000 7fca30551700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca2809fc60 con 0x7fca280a7740 2019-12-05T06:09:55.396 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:55.387+0000 7fca30551700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca280a0b00 con 0x7fca2808df80 2019-12-05T06:09:55.396 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:55.387+0000 7fca30551700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca2808e5b0 con 0x7fca2809f830 2019-12-05T06:09:55.397 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:55.387+0000 7fca2e2ed700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fca2808df80 0x7fca2809f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:09:55.397 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:55.387+0000 7fca2e2ed700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fca2808df80 0x7fca2809f2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:49488/0 (socket says 172.21.15.42:49488) 2019-12-05T06:09:55.397 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:55.387+0000 7fca2e2ed700 1 -- 172.21.15.42:0/459089864 learned_addr learned my addr 172.21.15.42:0/459089864 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:09:55.397 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:55.387+0000 7fca2f2ef700 1 --2- 172.21.15.42:0/459089864 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fca2809f830 0x7fca2814efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:09:55.398 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:55.387+0000 7fca2e2ed700 1 -- 172.21.15.42:0/459089864 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fca2809f830 msgr2=0x7fca2814efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:09:55.398 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:55.387+0000 7fca2e2ed700 1 --2- 172.21.15.42:0/459089864 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fca2809f830 0x7fca2814efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:09:55.398 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:55.387+0000 7fca2e2ed700 1 -- 172.21.15.42:0/459089864 >> v1:172.21.15.40:6789/0 conn(0x7fca280a7740 legacy=0x7fca2807e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:09:55.398 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:55.387+0000 7fca2e2ed700 1 -- 172.21.15.42:0/459089864 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca280a0710 con 0x7fca2808df80 2019-12-05T06:09:55.398 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:55.387+0000 7fca2e2ed700 1 --2- 172.21.15.42:0/459089864 >> v2:172.21.15.42:3300/0 conn(0x7fca2808df80 0x7fca2809f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fca1c008b10 tx=0x7fca1c0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:09:55.398 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:55.387+0000 7fca2e2ed700 1 --2- 172.21.15.42:0/459089864 >> v2:172.21.15.42:3300/0 conn(0x7fca2808df80 0x7fca2809f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fca1c008b10 tx=0x7fca1c0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:09:55.398 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:55.387+0000 7fca2e2ed700 1 --2- 172.21.15.42:0/459089864 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fca2808df80 0x7fca2809f2e0 secure :-1 s=READY pgs=2134 cs=0 l=1 rx=0x7fca1c008b10 tx=0x7fca1c0100b0).ready entity=mon.1 client_cookie=da50b314e6faf37f server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:09:55.399 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:55.387+0000 7fca2e2ed700 1 --2- 172.21.15.42:0/459089864 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fca2808df80 0x7fca2809f2e0 secure :-1 s=THROTTLE_DONE pgs=2134 cs=0 l=1 rx=0x7fca1c008b10 tx=0x7fca1c0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:09:55.399 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:55.387+0000 7fca2e2ed700 1 --2- 172.21.15.42:0/459089864 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fca2808df80 0x7fca2809f2e0 secure :-1 s=THROTTLE_DONE pgs=2134 cs=0 l=1 rx=0x7fca1c008b10 tx=0x7fca1c0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:09:55.399 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:55.387+0000 7fca2daec700 1 -- 172.21.15.42:0/459089864 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fca1c010da0 con 0x7fca2808df80 2019-12-05T06:09:55.399 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:55.387+0000 7fca2e2ed700 1 --2- 172.21.15.42:0/459089864 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fca2808df80 0x7fca2809f2e0 secure :-1 s=THROTTLE_DONE pgs=2134 cs=0 l=1 rx=0x7fca1c008b10 tx=0x7fca1c0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:09:55.399 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:55.387+0000 7fca2daec700 1 -- 172.21.15.42:0/459089864 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca1c0109c0 con 0x7fca2808df80 2019-12-05T06:09:55.399 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:55.387+0000 7fca2daec700 1 -- 172.21.15.42:0/459089864 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fca1c0114a0 con 0x7fca2808df80 2019-12-05T06:09:55.399 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:55.387+0000 7fca30551700 1 -- 172.21.15.42:0/459089864 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fca2808df80 msgr2=0x7fca2809f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:09:55.400 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:55.387+0000 7fca30551700 1 --2- 172.21.15.42:0/459089864 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fca2808df80 0x7fca2809f2e0 secure :-1 s=READY pgs=2134 cs=0 l=1 rx=0x7fca1c008b10 tx=0x7fca1c0100b0).stop 2019-12-05T06:09:55.400 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:55.387+0000 7fca30551700 1 -- 172.21.15.42:0/459089864 shutdown_connections 2019-12-05T06:09:55.400 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:55.387+0000 7fca30551700 1 --2- 172.21.15.42:0/459089864 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fca2809f830 0x7fca2814efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:09:55.400 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:55.387+0000 7fca30551700 1 --2- 172.21.15.42:0/459089864 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fca2808df80 0x7fca2809f2e0 unknown :-1 s=CLOSED pgs=2134 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:09:55.400 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:55.387+0000 7fca30551700 1 -- 172.21.15.42:0/459089864 shutdown_connections 2019-12-05T06:09:55.400 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:55.387+0000 7fca30551700 1 -- 172.21.15.42:0/459089864 wait complete. 2019-12-05T06:09:55.400 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:55.387+0000 7fca30551700 1 -- 172.21.15.42:0/459089864 >> 172.21.15.42:0/459089864 conn(0x7fca280aabe0 msgr2=0x7fca280a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:09:55.404 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:55.391+0000 7fca30551700 1 Processor -- start 2019-12-05T06:09:55.404 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:55.391+0000 7fca30551700 1 -- start start 2019-12-05T06:09:55.404 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:55.391+0000 7fca30551700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fca280a20a0 0x7fca280a2ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:09:55.404 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:55.391+0000 7fca30551700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fca2812cd80 0x7fca2812d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:09:55.404 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:55.391+0000 7fca30551700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fca2807a360 0x7fca2807a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:09:55.404 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:55.391+0000 7fca30551700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca2807ace0 con 0x7fca280a20a0 2019-12-05T06:09:55.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:55.391+0000 7fca30551700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca2812c840 con 0x7fca2807a360 2019-12-05T06:09:55.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:55.391+0000 7fca30551700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca2812cae0 con 0x7fca2812cd80 2019-12-05T06:09:55.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:55.395+0000 7fca2f2ef700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fca2807a360 0x7fca2807a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:09:55.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:55.395+0000 7fca2e2ed700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fca2812cd80 0x7fca2812d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:09:55.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:55.395+0000 7fca2eaee700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fca280a20a0 0x7fca280a2ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:09:55.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:55.395+0000 7fca2f2ef700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fca2807a360 0x7fca2807a790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:49496/0 (socket says 172.21.15.42:49496) 2019-12-05T06:09:55.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:55.395+0000 7fca2f2ef700 1 -- 172.21.15.42:0/2823848085 learned_addr learned my addr 172.21.15.42:0/2823848085 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:09:55.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:55.395+0000 7fca2f2ef700 1 -- 172.21.15.42:0/2823848085 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fca2812cd80 msgr2=0x7fca2812d1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:09:55.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:55.395+0000 7fca2f2ef700 1 --2- 172.21.15.42:0/2823848085 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fca2812cd80 0x7fca2812d1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:09:55.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:55.395+0000 7fca2f2ef700 1 -- 172.21.15.42:0/2823848085 >> v2:172.21.15.40:3300/0 conn(0x7fca280a20a0 msgr2=0x7fca280a2ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:09:55.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:55.395+0000 7fca2f2ef700 1 --2- 172.21.15.42:0/2823848085 >> v2:172.21.15.40:3300/0 conn(0x7fca280a20a0 0x7fca280a2ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:09:55.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:55.395+0000 7fca2f2ef700 1 -- 172.21.15.42:0/2823848085 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca280a24d0 con 0x7fca2807a360 2019-12-05T06:09:55.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:55.395+0000 7fca2eaee700 1 --2- 172.21.15.42:0/2823848085 >> v2:172.21.15.40:3300/0 conn(0x7fca280a20a0 0x7fca280a2ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:09:55.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:55.395+0000 7fca2f2ef700 1 --2- 172.21.15.42:0/2823848085 >> v2:172.21.15.42:3300/0 conn(0x7fca2807a360 0x7fca2807a790 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fca280a3db0 tx=0x7fca20011b20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:09:55.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:55.395+0000 7fca2f2ef700 1 --2- 172.21.15.42:0/2823848085 >> v2:172.21.15.42:3300/0 conn(0x7fca2807a360 0x7fca2807a790 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fca280a3db0 tx=0x7fca20011b20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:09:55.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:55.395+0000 7fca2f2ef700 1 --2- 172.21.15.42:0/2823848085 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fca2807a360 0x7fca2807a790 secure :-1 s=READY pgs=2135 cs=0 l=1 rx=0x7fca280a3db0 tx=0x7fca20011b20).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:09:55.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:55.395+0000 7fca2f2ef700 1 --2- 172.21.15.42:0/2823848085 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fca2807a360 0x7fca2807a790 secure :-1 s=THROTTLE_DONE pgs=2135 cs=0 l=1 rx=0x7fca280a3db0 tx=0x7fca20011b20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:09:55.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:55.395+0000 7fca2f2ef700 1 --2- 172.21.15.42:0/2823848085 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fca2807a360 0x7fca2807a790 secure :-1 s=THROTTLE_DONE pgs=2135 cs=0 l=1 rx=0x7fca280a3db0 tx=0x7fca20011b20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:09:55.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:55.395+0000 7fca2f2ef700 1 --2- 172.21.15.42:0/2823848085 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fca2807a360 0x7fca2807a790 secure :-1 s=THROTTLE_DONE pgs=2135 cs=0 l=1 rx=0x7fca280a3db0 tx=0x7fca20011b20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:09:55.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:55.395+0000 7fca1b7fe700 1 -- 172.21.15.42:0/2823848085 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fca20005930 con 0x7fca2807a360 2019-12-05T06:09:55.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:55.395+0000 7fca1b7fe700 1 -- 172.21.15.42:0/2823848085 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca20007160 con 0x7fca2807a360 2019-12-05T06:09:55.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:55.395+0000 7fca30551700 1 -- 172.21.15.42:0/2823848085 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca2807afa0 con 0x7fca2807a360 2019-12-05T06:09:55.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:55.395+0000 7fca1b7fe700 1 -- 172.21.15.42:0/2823848085 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fca200131a0 con 0x7fca2807a360 2019-12-05T06:09:55.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:55.395+0000 7fca30551700 1 -- 172.21.15.42:0/2823848085 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca280a9770 con 0x7fca2807a360 2019-12-05T06:09:55.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:55.395+0000 7fca2f2ef700 1 --2- 172.21.15.42:0/2823848085 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fca2807a360 0x7fca2807a790 secure :-1 s=THROTTLE_DONE pgs=2135 cs=0 l=1 rx=0x7fca280a3db0 tx=0x7fca20011b20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:09:55.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:55.395+0000 7fca1b7fe700 1 -- 172.21.15.42:0/2823848085 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fca200134b0 con 0x7fca2807a360 2019-12-05T06:09:55.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:55.395+0000 7fca2f2ef700 1 --2- 172.21.15.42:0/2823848085 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fca2807a360 0x7fca2807a790 secure :-1 s=THROTTLE_DONE pgs=2135 cs=0 l=1 rx=0x7fca280a3db0 tx=0x7fca20011b20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:09:55.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:55.395+0000 7fca1b7fe700 1 --2- 172.21.15.42:0/2823848085 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fca1003ad70 0x7fca1003d280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:09:55.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:55.395+0000 7fca1b7fe700 1 -- 172.21.15.42:0/2823848085 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9901+0+0 (secure 0 0 0) 0x7fca2004bf30 con 0x7fca2807a360 2019-12-05T06:09:55.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:55.395+0000 7fca2eaee700 1 --2- 172.21.15.42:0/2823848085 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fca1003ad70 0x7fca1003d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:09:55.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:55.395+0000 7fca2eaee700 1 --2- 172.21.15.42:0/2823848085 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fca1003ad70 0x7fca1003d280 crc :-1 s=READY pgs=898 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:09:55.416 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:55.399+0000 7fca30551700 1 -- 172.21.15.42:0/2823848085 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca280a35e0 con 0x7fca2807a360 2019-12-05T06:09:55.416 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:55.403+0000 7fca2f2ef700 1 --2- 172.21.15.42:0/2823848085 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fca2807a360 0x7fca2807a790 secure :-1 s=THROTTLE_DONE pgs=2135 cs=0 l=1 rx=0x7fca280a3db0 tx=0x7fca20011b20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:09:55.416 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:55.403+0000 7fca1b7fe700 1 -- 172.21.15.42:0/2823848085 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fca20032fb0 con 0x7fca2807a360 2019-12-05T06:09:55.578 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:55.567+0000 7fca30551700 1 -- 172.21.15.42:0/2823848085 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fca2805d110 con 0x7fca2807a360 2019-12-05T06:09:55.579 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:55.567+0000 7fca2f2ef700 1 --2- 172.21.15.42:0/2823848085 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fca2807a360 0x7fca2807a790 secure :-1 s=THROTTLE_DONE pgs=2135 cs=0 l=1 rx=0x7fca280a3db0 tx=0x7fca20011b20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:09:55.579 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:55.567+0000 7fca1b7fe700 1 -- 172.21.15.42:0/2823848085 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 258 v258) v1 ==== 95+0+2112 (secure 0 0 0) 0x7fca20033210 con 0x7fca2807a360 2019-12-05T06:09:55.579 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 258 2019-12-05T06:09:55.579 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:55.567+0000 7fca30551700 1 -- 172.21.15.42:0/2823848085 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fca1003ad70 msgr2=0x7fca1003d280 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:09:55.579 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:55.567+0000 7fca30551700 1 --2- 172.21.15.42:0/2823848085 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fca1003ad70 0x7fca1003d280 crc :-1 s=READY pgs=898 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:09:55.579 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:55.567+0000 7fca30551700 1 -- 172.21.15.42:0/2823848085 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fca2807a360 msgr2=0x7fca2807a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:09:55.580 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:55.567+0000 7fca30551700 1 --2- 172.21.15.42:0/2823848085 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fca2807a360 0x7fca2807a790 secure :-1 s=READY pgs=2135 cs=0 l=1 rx=0x7fca280a3db0 tx=0x7fca20011b20).stop 2019-12-05T06:09:55.580 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:55.571+0000 7fca30551700 1 -- 172.21.15.42:0/2823848085 shutdown_connections 2019-12-05T06:09:55.580 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:55.571+0000 7fca30551700 1 --2- 172.21.15.42:0/2823848085 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fca1003ad70 0x7fca1003d280 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:09:55.580 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:55.571+0000 7fca30551700 1 --2- 172.21.15.42:0/2823848085 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fca2807a360 0x7fca2807a790 unknown :-1 s=CLOSED pgs=2135 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:09:55.580 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:55.571+0000 7fca30551700 1 --2- 172.21.15.42:0/2823848085 >> v2:172.21.15.40:3300/0 conn(0x7fca280a20a0 0x7fca280a2ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:09:55.580 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:55.571+0000 7fca30551700 1 --2- 172.21.15.42:0/2823848085 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fca2812cd80 0x7fca2812d1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:09:55.580 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:55.571+0000 7fca30551700 1 -- 172.21.15.42:0/2823848085 shutdown_connections 2019-12-05T06:09:55.581 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:55.571+0000 7fca30551700 1 -- 172.21.15.42:0/2823848085 wait complete. 2019-12-05T06:09:55.581 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:55.571+0000 7fca30551700 1 -- 172.21.15.42:0/2823848085 >> 172.21.15.42:0/2823848085 conn(0x7fca280a7740 msgr2=0x7fca28059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:09:55.581 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:09:55.581 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":258,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6977,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/3355679996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3355679996},{"type":"v1","addr":"172.21.15.40:6833","nonce":3355679996}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":258}],"filesystems":[{"mdsmap":{"epoch":257,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:08:14.898258+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":94,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6929},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6929":{"gid":6929,"name":"a","rank":0,"incarnation":254,"state":"up:active","state_seq":19,"addr":"172.21.15.42:6835/2636699468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2636699468},{"type":"v1","addr":"172.21.15.42:6835","nonce":2636699468}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:09:55.586 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:09:57.340 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a has been restored 2019-12-05T06:09:57.589 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:09:57.697 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:57.683+0000 7f1000661700 1 Processor -- start 2019-12-05T06:09:57.697 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:57.683+0000 7f1000661700 1 -- start start 2019-12-05T06:09:57.697 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:57.687+0000 7f1000661700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0ff80a7730 0x7f0ff808df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:09:57.697 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:57.687+0000 7f1000661700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0ff807e710 0x7f0ff807eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:09:57.697 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:57.687+0000 7f1000661700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ff80a0700 con 0x7f0ff809f2d0 2019-12-05T06:09:57.697 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:57.687+0000 7f1000661700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ff80a0af0 con 0x7f0ff807e710 2019-12-05T06:09:57.698 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:57.687+0000 7f1000661700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ff808e5a0 con 0x7f0ff80a7730 2019-12-05T06:09:57.698 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:57.687+0000 7f0ffe3fd700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0ff807e710 0x7f0ff807eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:09:57.698 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:57.687+0000 7f0ffe3fd700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0ff807e710 0x7f0ff807eb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:49502/0 (socket says 172.21.15.42:49502) 2019-12-05T06:09:57.698 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:57.687+0000 7f0ffe3fd700 1 -- 172.21.15.42:0/2569375381 learned_addr learned my addr 172.21.15.42:0/2569375381 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:09:57.698 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:57.687+0000 7f0ffebfe700 1 --2- 172.21.15.42:0/2569375381 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0ff80a7730 0x7f0ff808df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:09:57.698 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:57.687+0000 7f0ffe3fd700 1 -- 172.21.15.42:0/2569375381 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0ff80a7730 msgr2=0x7f0ff808df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:09:57.698 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:57.687+0000 7f0ffe3fd700 1 --2- 172.21.15.42:0/2569375381 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0ff80a7730 0x7f0ff808df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:09:57.699 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:57.687+0000 7f0ffe3fd700 1 -- 172.21.15.42:0/2569375381 >> v1:172.21.15.40:6789/0 conn(0x7f0ff809f2d0 legacy=0x7f0ff809f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:09:57.699 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:57.687+0000 7f0ffe3fd700 1 -- 172.21.15.42:0/2569375381 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ff807f090 con 0x7f0ff807e710 2019-12-05T06:09:57.699 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:57.687+0000 7f0ffe3fd700 1 --2- 172.21.15.42:0/2569375381 >> v2:172.21.15.42:3300/0 conn(0x7f0ff807e710 0x7f0ff807eb40 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f0ff4008790 tx=0x7f0ff400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:09:57.699 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:57.687+0000 7f0ffe3fd700 1 --2- 172.21.15.42:0/2569375381 >> v2:172.21.15.42:3300/0 conn(0x7f0ff807e710 0x7f0ff807eb40 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f0ff4008790 tx=0x7f0ff400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:09:57.700 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:57.687+0000 7f0ffe3fd700 1 --2- 172.21.15.42:0/2569375381 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0ff807e710 0x7f0ff807eb40 secure :-1 s=READY pgs=2136 cs=0 l=1 rx=0x7f0ff4008790 tx=0x7f0ff400fd30).ready entity=mon.1 client_cookie=6b04615dad18b38b server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:09:57.700 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:57.687+0000 7f0ffe3fd700 1 --2- 172.21.15.42:0/2569375381 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0ff807e710 0x7f0ff807eb40 secure :-1 s=THROTTLE_DONE pgs=2136 cs=0 l=1 rx=0x7f0ff4008790 tx=0x7f0ff400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:09:57.700 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:57.687+0000 7f0ffdbfc700 1 -- 172.21.15.42:0/2569375381 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0ff40113b0 con 0x7f0ff807e710 2019-12-05T06:09:57.700 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:57.687+0000 7f0ffe3fd700 1 --2- 172.21.15.42:0/2569375381 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0ff807e710 0x7f0ff807eb40 secure :-1 s=THROTTLE_DONE pgs=2136 cs=0 l=1 rx=0x7f0ff4008790 tx=0x7f0ff400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:09:57.700 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:57.687+0000 7f0ffe3fd700 1 --2- 172.21.15.42:0/2569375381 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0ff807e710 0x7f0ff807eb40 secure :-1 s=THROTTLE_DONE pgs=2136 cs=0 l=1 rx=0x7f0ff4008790 tx=0x7f0ff400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:09:57.700 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:57.687+0000 7f0ffdbfc700 1 -- 172.21.15.42:0/2569375381 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ff4008bb0 con 0x7f0ff807e710 2019-12-05T06:09:57.701 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:57.687+0000 7f0ffdbfc700 1 -- 172.21.15.42:0/2569375381 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0ff4011990 con 0x7f0ff807e710 2019-12-05T06:09:57.701 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:57.687+0000 7f1000661700 1 -- 172.21.15.42:0/2569375381 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0ff807e710 msgr2=0x7f0ff807eb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:09:57.701 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:57.687+0000 7f1000661700 1 --2- 172.21.15.42:0/2569375381 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0ff807e710 0x7f0ff807eb40 secure :-1 s=READY pgs=2136 cs=0 l=1 rx=0x7f0ff4008790 tx=0x7f0ff400fd30).stop 2019-12-05T06:09:57.701 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:57.687+0000 7f1000661700 1 -- 172.21.15.42:0/2569375381 shutdown_connections 2019-12-05T06:09:57.701 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:57.687+0000 7f1000661700 1 --2- 172.21.15.42:0/2569375381 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0ff80a7730 0x7f0ff808df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:09:57.702 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:57.687+0000 7f1000661700 1 --2- 172.21.15.42:0/2569375381 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0ff807e710 0x7f0ff807eb40 unknown :-1 s=CLOSED pgs=2136 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:09:57.702 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:57.687+0000 7f1000661700 1 -- 172.21.15.42:0/2569375381 shutdown_connections 2019-12-05T06:09:57.702 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:57.687+0000 7f1000661700 1 -- 172.21.15.42:0/2569375381 wait complete. 2019-12-05T06:09:57.702 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:57.687+0000 7f1000661700 1 -- 172.21.15.42:0/2569375381 >> 172.21.15.42:0/2569375381 conn(0x7f0ff80aabd0 msgr2=0x7f0ff80a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:09:57.702 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:57.691+0000 7f1000661700 1 Processor -- start 2019-12-05T06:09:57.702 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:57.691+0000 7f1000661700 1 -- start start 2019-12-05T06:09:57.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:57.691+0000 7f1000661700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0ff80a6da0 0x7f0ff812a190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:09:57.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:57.691+0000 7f1000661700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f0ff81135e0 0x7f0ff81315e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:09:57.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:57.691+0000 7f1000661700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ff814ed00 con 0x7f0ff8065970 2019-12-05T06:09:57.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:57.691+0000 7f1000661700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ff80a6860 con 0x7f0ff81135e0 2019-12-05T06:09:57.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:57.691+0000 7f1000661700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ff80a6b00 con 0x7f0ff80a6da0 2019-12-05T06:09:57.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:57.691+0000 7f0ffebfe700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0ff80a6da0 0x7f0ff812a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:09:57.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:57.691+0000 7f0fff3ff700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f0ff81135e0 0x7f0ff81315e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:09:57.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:57.691+0000 7f0ffebfe700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0ff80a6da0 0x7f0ff812a190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:49976/0 (socket says 172.21.15.42:49976) 2019-12-05T06:09:57.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:57.691+0000 7f0ffe3fd700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f0ff8065970 0x7f0ff8123da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:58986/0 (socket says 172.21.15.42:58986) 2019-12-05T06:09:57.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:57.691+0000 7f0ffebfe700 1 -- 172.21.15.42:0/572114176 learned_addr learned my addr 172.21.15.42:0/572114176 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:09:57.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:57.691+0000 7f0ffebfe700 1 -- 172.21.15.42:0/572114176 >> v2:172.21.15.40:3300/0 conn(0x7f0ff81135e0 msgr2=0x7f0ff81315e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:09:57.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:57.691+0000 7f0ffebfe700 1 --2- 172.21.15.42:0/572114176 >> v2:172.21.15.40:3300/0 conn(0x7f0ff81135e0 0x7f0ff81315e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:09:57.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:57.691+0000 7f0ffebfe700 1 -- 172.21.15.42:0/572114176 >> v1:172.21.15.40:6789/0 conn(0x7f0ff8065970 legacy=0x7f0ff8123da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:09:57.708 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:57.691+0000 7f0ffebfe700 1 -- 172.21.15.42:0/572114176 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ff81245a0 con 0x7f0ff80a6da0 2019-12-05T06:09:57.708 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:57.691+0000 7f0fff3ff700 1 --2- 172.21.15.42:0/572114176 >> v2:172.21.15.40:3300/0 conn(0x7f0ff81135e0 0x7f0ff81315e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:09:57.708 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:57.695+0000 7f0ffebfe700 1 --2- 172.21.15.42:0/572114176 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0ff80a6da0 0x7f0ff812a190 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f0ff4008370 tx=0x7f0ff400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:09:57.708 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:57.695+0000 7f0ffebfe700 1 --2- 172.21.15.42:0/572114176 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0ff80a6da0 0x7f0ff812a190 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f0ff4008370 tx=0x7f0ff400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:09:57.708 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:57.695+0000 7f0ffebfe700 1 --2- 172.21.15.42:0/572114176 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0ff80a6da0 0x7f0ff812a190 secure :-1 s=READY pgs=470 cs=0 l=1 rx=0x7f0ff4008370 tx=0x7f0ff400fd30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:09:57.708 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:57.695+0000 7f0ffebfe700 1 --2- 172.21.15.42:0/572114176 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0ff80a6da0 0x7f0ff812a190 secure :-1 s=THROTTLE_DONE pgs=470 cs=0 l=1 rx=0x7f0ff4008370 tx=0x7f0ff400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:09:57.708 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:57.695+0000 7f0ffebfe700 1 --2- 172.21.15.42:0/572114176 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0ff80a6da0 0x7f0ff812a190 secure :-1 s=THROTTLE_DONE pgs=470 cs=0 l=1 rx=0x7f0ff4008370 tx=0x7f0ff400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:09:57.709 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:57.695+0000 7f0ffebfe700 1 --2- 172.21.15.42:0/572114176 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0ff80a6da0 0x7f0ff812a190 secure :-1 s=THROTTLE_DONE pgs=470 cs=0 l=1 rx=0x7f0ff4008370 tx=0x7f0ff400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:09:57.709 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:57.695+0000 7f0feb7fe700 1 -- 172.21.15.42:0/572114176 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0fec005840 con 0x7f0ff80a6da0 2019-12-05T06:09:57.709 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:57.695+0000 7f0feb7fe700 1 -- 172.21.15.42:0/572114176 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fec007160 con 0x7f0ff80a6da0 2019-12-05T06:09:57.709 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:57.695+0000 7f1000661700 1 -- 172.21.15.42:0/572114176 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ff812a6e0 con 0x7f0ff80a6da0 2019-12-05T06:09:57.709 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:57.695+0000 7f0feb7fe700 1 -- 172.21.15.42:0/572114176 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0fec0060c0 con 0x7f0ff80a6da0 2019-12-05T06:09:57.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:57.695+0000 7f1000661700 1 -- 172.21.15.42:0/572114176 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ff8144ff0 con 0x7f0ff80a6da0 2019-12-05T06:09:57.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:57.695+0000 7f0ffebfe700 1 --2- 172.21.15.42:0/572114176 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0ff80a6da0 0x7f0ff812a190 secure :-1 s=THROTTLE_DONE pgs=470 cs=0 l=1 rx=0x7f0ff4008370 tx=0x7f0ff400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:09:57.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:57.695+0000 7f0feb7fe700 1 -- 172.21.15.42:0/572114176 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f0fec007a20 con 0x7f0ff80a6da0 2019-12-05T06:09:57.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:57.695+0000 7f0ffebfe700 1 --2- 172.21.15.42:0/572114176 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0ff80a6da0 0x7f0ff812a190 secure :-1 s=THROTTLE_DONE pgs=470 cs=0 l=1 rx=0x7f0ff4008370 tx=0x7f0ff400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:09:57.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:57.695+0000 7f0feb7fe700 1 --2- 172.21.15.42:0/572114176 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0fe003ae10 0x7f0fe003d320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:09:57.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:57.695+0000 7f0feb7fe700 1 -- 172.21.15.42:0/572114176 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9901+0+0 (secure 0 0 0) 0x7f0fec04a9a0 con 0x7f0ff80a6da0 2019-12-05T06:09:57.711 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:57.695+0000 7f0ffe3fd700 1 --2- 172.21.15.42:0/572114176 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0fe003ae10 0x7f0fe003d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:09:57.711 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:57.699+0000 7f0ffe3fd700 1 --2- 172.21.15.42:0/572114176 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0fe003ae10 0x7f0fe003d320 crc :-1 s=READY pgs=899 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:09:57.711 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:57.703+0000 7f1000661700 1 -- 172.21.15.42:0/572114176 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ff805d100 con 0x7f0ff80a6da0 2019-12-05T06:09:57.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:57.711+0000 7f0ffebfe700 1 --2- 172.21.15.42:0/572114176 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0ff80a6da0 0x7f0ff812a190 secure :-1 s=THROTTLE_DONE pgs=470 cs=0 l=1 rx=0x7f0ff4008370 tx=0x7f0ff400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:09:57.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:57.711+0000 7f0feb7fe700 1 -- 172.21.15.42:0/572114176 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f0fec0319e0 con 0x7f0ff80a6da0 2019-12-05T06:09:57.889 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:57.883+0000 7f1000661700 1 -- 172.21.15.42:0/572114176 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0ff805b8b0 con 0x7f0ff80a6da0 2019-12-05T06:09:57.890 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:57.883+0000 7f0ffebfe700 1 --2- 172.21.15.42:0/572114176 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0ff80a6da0 0x7f0ff812a190 secure :-1 s=THROTTLE_DONE pgs=470 cs=0 l=1 rx=0x7f0ff4008370 tx=0x7f0ff400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:09:57.890 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:57.883+0000 7f0feb7fe700 1 -- 172.21.15.42:0/572114176 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 258 v258) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f0fec007d30 con 0x7f0ff80a6da0 2019-12-05T06:09:57.892 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 258 2019-12-05T06:09:57.892 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:09:57.892 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":258,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6977,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/3355679996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3355679996},{"type":"v1","addr":"172.21.15.40:6833","nonce":3355679996}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":258}],"filesystems":[{"mdsmap":{"epoch":257,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:08:14.898258+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":94,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6929},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6929":{"gid":6929,"name":"a","rank":0,"incarnation":254,"state":"up:active","state_seq":19,"addr":"172.21.15.42:6835/2636699468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2636699468},{"type":"v1","addr":"172.21.15.42:6835","nonce":2636699468}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:09:57.895 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:57.887+0000 7f1000661700 1 -- 172.21.15.42:0/572114176 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0fe003ae10 msgr2=0x7f0fe003d320 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:09:57.895 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:57.887+0000 7f1000661700 1 --2- 172.21.15.42:0/572114176 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0fe003ae10 0x7f0fe003d320 crc :-1 s=READY pgs=899 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:09:57.895 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:57.887+0000 7f1000661700 1 -- 172.21.15.42:0/572114176 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0ff80a6da0 msgr2=0x7f0ff812a190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:09:57.895 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:57.887+0000 7f1000661700 1 --2- 172.21.15.42:0/572114176 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0ff80a6da0 0x7f0ff812a190 secure :-1 s=READY pgs=470 cs=0 l=1 rx=0x7f0ff4008370 tx=0x7f0ff400fd30).stop 2019-12-05T06:09:57.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:57.887+0000 7f1000661700 1 -- 172.21.15.42:0/572114176 shutdown_connections 2019-12-05T06:09:57.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:57.887+0000 7f1000661700 1 --2- 172.21.15.42:0/572114176 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0fe003ae10 0x7f0fe003d320 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:09:57.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:57.887+0000 7f1000661700 1 --2- 172.21.15.42:0/572114176 >> v2:172.21.15.40:3300/0 conn(0x7f0ff81135e0 0x7f0ff81315e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:09:57.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:57.887+0000 7f1000661700 1 --2- 172.21.15.42:0/572114176 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0ff80a6da0 0x7f0ff812a190 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:09:57.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:57.887+0000 7f1000661700 1 -- 172.21.15.42:0/572114176 shutdown_connections 2019-12-05T06:09:57.897 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:57.887+0000 7f1000661700 1 -- 172.21.15.42:0/572114176 wait complete. 2019-12-05T06:09:57.897 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:09:57.887+0000 7f1000661700 1 -- 172.21.15.42:0/572114176 >> 172.21.15.42:0/572114176 conn(0x7f0ff80596f0 msgr2=0x7f0ff8059d10 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:09:57.905 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:09:59.906 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:10:00.018 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:00.003+0000 7f26c9b14700 1 Processor -- start 2019-12-05T06:10:00.018 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:00.007+0000 7f26c9b14700 1 -- start start 2019-12-05T06:10:00.019 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:00.007+0000 7f26c9b14700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f26c40a7730 0x7f26c408df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:10:00.019 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:00.007+0000 7f26c9b14700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f26c407e710 0x7f26c407eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:10:00.019 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:00.007+0000 7f26c9b14700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f26c409f2d0 0x7f26c409f700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:10:00.019 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:00.007+0000 7f26c9b14700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26c40a0af0 con 0x7f26c407e710 2019-12-05T06:10:00.019 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:00.007+0000 7f26c9b14700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26c408e5a0 con 0x7f26c409f2d0 2019-12-05T06:10:00.019 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:00.007+0000 7f26c9b14700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26c408e840 con 0x7f26c40a7730 2019-12-05T06:10:00.020 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:00.007+0000 7f26c88b2700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f26c409f2d0 0x7f26c409f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:10:00.020 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:00.007+0000 7f26c88b2700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f26c409f2d0 0x7f26c409f700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:49520/0 (socket says 172.21.15.42:49520) 2019-12-05T06:10:00.020 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:00.007+0000 7f26c88b2700 1 -- 172.21.15.42:0/839780470 learned_addr learned my addr 172.21.15.42:0/839780470 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:10:00.020 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:00.007+0000 7f26c37fe700 1 --2- 172.21.15.42:0/839780470 >> v2:172.21.15.40:3300/0 conn(0x7f26c407e710 0x7f26c407eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:10:00.020 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:00.007+0000 7f26c3fff700 1 --2- 172.21.15.42:0/839780470 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f26c40a7730 0x7f26c408df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:10:00.020 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:00.007+0000 7f26c88b2700 1 -- 172.21.15.42:0/839780470 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f26c40a7730 msgr2=0x7f26c408df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:10:00.020 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:00.007+0000 7f26c88b2700 1 --2- 172.21.15.42:0/839780470 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f26c40a7730 0x7f26c408df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:10:00.021 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:00.007+0000 7f26c88b2700 1 -- 172.21.15.42:0/839780470 >> v2:172.21.15.40:3300/0 conn(0x7f26c407e710 msgr2=0x7f26c407eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:10:00.021 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:00.007+0000 7f26c88b2700 1 --2- 172.21.15.42:0/839780470 >> v2:172.21.15.40:3300/0 conn(0x7f26c407e710 0x7f26c407eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:10:00.021 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:00.007+0000 7f26c88b2700 1 -- 172.21.15.42:0/839780470 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26c407f090 con 0x7f26c409f2d0 2019-12-05T06:10:00.021 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:00.007+0000 7f26c88b2700 1 --2- 172.21.15.42:0/839780470 >> v2:172.21.15.42:3300/0 conn(0x7f26c409f2d0 0x7f26c409f700 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f26c408e760 tx=0x7f26b400fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:00.021 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:00.007+0000 7f26c88b2700 1 --2- 172.21.15.42:0/839780470 >> v2:172.21.15.42:3300/0 conn(0x7f26c409f2d0 0x7f26c409f700 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f26c408e760 tx=0x7f26b400fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:00.021 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:00.007+0000 7f26c88b2700 1 --2- 172.21.15.42:0/839780470 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f26c409f2d0 0x7f26c409f700 secure :-1 s=READY pgs=2137 cs=0 l=1 rx=0x7f26c408e760 tx=0x7f26b400fdb0).ready entity=mon.1 client_cookie=15d054aee98e5f9d server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:10:00.021 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:00.007+0000 7f26c88b2700 1 --2- 172.21.15.42:0/839780470 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f26c409f2d0 0x7f26c409f700 secure :-1 s=THROTTLE_DONE pgs=2137 cs=0 l=1 rx=0x7f26c408e760 tx=0x7f26b400fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:00.022 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:00.007+0000 7f26c88b2700 1 --2- 172.21.15.42:0/839780470 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f26c409f2d0 0x7f26c409f700 secure :-1 s=THROTTLE_DONE pgs=2137 cs=0 l=1 rx=0x7f26c408e760 tx=0x7f26b400fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:00.022 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:00.007+0000 7f26c2ffd700 1 -- 172.21.15.42:0/839780470 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f26b4008c90 con 0x7f26c409f2d0 2019-12-05T06:10:00.022 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:00.007+0000 7f26c88b2700 1 --2- 172.21.15.42:0/839780470 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f26c409f2d0 0x7f26c409f700 secure :-1 s=THROTTLE_DONE pgs=2137 cs=0 l=1 rx=0x7f26c408e760 tx=0x7f26b400fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:00.022 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:00.007+0000 7f26c2ffd700 1 -- 172.21.15.42:0/839780470 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26b40088b0 con 0x7f26c409f2d0 2019-12-05T06:10:00.022 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:00.007+0000 7f26c2ffd700 1 -- 172.21.15.42:0/839780470 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f26b4011b50 con 0x7f26c409f2d0 2019-12-05T06:10:00.022 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:00.007+0000 7f26c9b14700 1 -- 172.21.15.42:0/839780470 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f26c409f2d0 msgr2=0x7f26c409f700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:10:00.022 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:00.011+0000 7f26c9b14700 1 --2- 172.21.15.42:0/839780470 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f26c409f2d0 0x7f26c409f700 secure :-1 s=READY pgs=2137 cs=0 l=1 rx=0x7f26c408e760 tx=0x7f26b400fdb0).stop 2019-12-05T06:10:00.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:00.011+0000 7f26c9b14700 1 -- 172.21.15.42:0/839780470 shutdown_connections 2019-12-05T06:10:00.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:00.011+0000 7f26c9b14700 1 --2- 172.21.15.42:0/839780470 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f26c409f2d0 0x7f26c409f700 unknown :-1 s=CLOSED pgs=2137 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:10:00.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:00.011+0000 7f26c9b14700 1 --2- 172.21.15.42:0/839780470 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f26c40a7730 0x7f26c408df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:10:00.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:00.011+0000 7f26c9b14700 1 --2- 172.21.15.42:0/839780470 >> v2:172.21.15.40:3300/0 conn(0x7f26c407e710 0x7f26c407eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:10:00.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:00.011+0000 7f26c9b14700 1 -- 172.21.15.42:0/839780470 shutdown_connections 2019-12-05T06:10:00.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:00.011+0000 7f26c9b14700 1 -- 172.21.15.42:0/839780470 wait complete. 2019-12-05T06:10:00.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:00.011+0000 7f26c9b14700 1 -- 172.21.15.42:0/839780470 >> 172.21.15.42:0/839780470 conn(0x7f26c40aabd0 msgr2=0x7f26c40a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:10:00.027 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:00.015+0000 7f26c9b14700 1 Processor -- start 2019-12-05T06:10:00.027 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:00.015+0000 7f26c9b14700 1 -- start start 2019-12-05T06:10:00.027 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:00.015+0000 7f26c9b14700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f26c411b120 0x7f26c411a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:10:00.028 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:00.015+0000 7f26c9b14700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26c411ab40 con 0x7f26c4059c00 2019-12-05T06:10:00.028 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:00.015+0000 7f26c9b14700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26c411f740 con 0x7f26c4121610 2019-12-05T06:10:00.028 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:00.015+0000 7f26c9b14700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26c4120890 con 0x7f26c411b120 2019-12-05T06:10:00.028 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:00.015+0000 7f26c3fff700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f26c411b120 0x7f26c411a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:10:00.028 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:00.015+0000 7f26c37fe700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f26c4121610 0x7f26c408eb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:49564/0 (socket says 172.21.15.42:49564) 2019-12-05T06:10:00.028 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:00.015+0000 7f26c37fe700 1 -- 172.21.15.42:0/2209700384 learned_addr learned my addr 172.21.15.42:0/2209700384 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:10:00.029 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:00.015+0000 7f26c0ff9700 1 -- 172.21.15.42:0/2209700384 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2069401988 0 0) 0x7f26c411f740 con 0x7f26c4121610 2019-12-05T06:10:00.029 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:00.015+0000 7f26c0ff9700 1 -- 172.21.15.42:0/2209700384 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26a80030b0 con 0x7f26c4121610 2019-12-05T06:10:00.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:00.015+0000 7f26c3fff700 1 -- 172.21.15.42:0/2209700384 >> v1:172.21.15.42:6789/0 conn(0x7f26c4121610 legacy=0x7f26c408eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:10:00.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:00.015+0000 7f26c3fff700 1 -- 172.21.15.42:0/2209700384 >> v1:172.21.15.40:6789/0 conn(0x7f26c4059c00 legacy=0x7f26c408f360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:10:00.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:00.015+0000 7f26c3fff700 1 -- 172.21.15.42:0/2209700384 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26c411fa50 con 0x7f26c411b120 2019-12-05T06:10:00.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:00.015+0000 7f26c3fff700 1 --2- 172.21.15.42:0/2209700384 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f26c411b120 0x7f26c411a5f0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f26bc0032b0 tx=0x7f26bc010d60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:00.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:00.015+0000 7f26c3fff700 1 --2- 172.21.15.42:0/2209700384 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f26c411b120 0x7f26c411a5f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f26bc0032b0 tx=0x7f26bc010d60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:00.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:00.015+0000 7f26c3fff700 1 --2- 172.21.15.42:0/2209700384 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f26c411b120 0x7f26c411a5f0 secure :-1 s=READY pgs=471 cs=0 l=1 rx=0x7f26bc0032b0 tx=0x7f26bc010d60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:10:00.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:00.019+0000 7f26c3fff700 1 --2- 172.21.15.42:0/2209700384 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f26c411b120 0x7f26c411a5f0 secure :-1 s=THROTTLE_DONE pgs=471 cs=0 l=1 rx=0x7f26bc0032b0 tx=0x7f26bc010d60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:00.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:00.019+0000 7f26c0ff9700 1 -- 172.21.15.42:0/2209700384 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f26bc005ca0 con 0x7f26c411b120 2019-12-05T06:10:00.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:00.019+0000 7f26c3fff700 1 --2- 172.21.15.42:0/2209700384 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f26c411b120 0x7f26c411a5f0 secure :-1 s=THROTTLE_DONE pgs=471 cs=0 l=1 rx=0x7f26bc0032b0 tx=0x7f26bc010d60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:00.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:00.019+0000 7f26c3fff700 1 --2- 172.21.15.42:0/2209700384 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f26c411b120 0x7f26c411a5f0 secure :-1 s=THROTTLE_DONE pgs=471 cs=0 l=1 rx=0x7f26bc0032b0 tx=0x7f26bc010d60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:00.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:00.019+0000 7f26c0ff9700 1 -- 172.21.15.42:0/2209700384 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26bc0058c0 con 0x7f26c411b120 2019-12-05T06:10:00.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:00.019+0000 7f26c9b14700 1 -- 172.21.15.42:0/2209700384 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26c411e9f0 con 0x7f26c411b120 2019-12-05T06:10:00.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:00.019+0000 7f26c0ff9700 1 -- 172.21.15.42:0/2209700384 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f26bc012c70 con 0x7f26c411b120 2019-12-05T06:10:00.032 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:00.019+0000 7f26c9b14700 1 -- 172.21.15.42:0/2209700384 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26c405a030 con 0x7f26c411b120 2019-12-05T06:10:00.032 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:00.019+0000 7f26c3fff700 1 --2- 172.21.15.42:0/2209700384 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f26c411b120 0x7f26c411a5f0 secure :-1 s=THROTTLE_DONE pgs=471 cs=0 l=1 rx=0x7f26bc0032b0 tx=0x7f26bc010d60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:00.032 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:00.019+0000 7f26c3fff700 1 --2- 172.21.15.42:0/2209700384 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f26c411b120 0x7f26c411a5f0 secure :-1 s=THROTTLE_DONE pgs=471 cs=0 l=1 rx=0x7f26bc0032b0 tx=0x7f26bc010d60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:00.032 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:00.019+0000 7f26c0ff9700 1 -- 172.21.15.42:0/2209700384 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f26bc00c3f0 con 0x7f26c411b120 2019-12-05T06:10:00.032 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:00.019+0000 7f26c0ff9700 1 --2- 172.21.15.42:0/2209700384 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f26a803b260 0x7f26a803d770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:10:00.032 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:00.019+0000 7f26c0ff9700 1 -- 172.21.15.42:0/2209700384 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9901+0+0 (secure 0 0 0) 0x7f26bc04bbc0 con 0x7f26c411b120 2019-12-05T06:10:00.032 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:00.019+0000 7f26c37fe700 1 --2- 172.21.15.42:0/2209700384 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f26a803b260 0x7f26a803d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:10:00.033 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:00.019+0000 7f26c37fe700 1 --2- 172.21.15.42:0/2209700384 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f26a803b260 0x7f26a803d770 crc :-1 s=READY pgs=900 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:10:00.033 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:00.023+0000 7f26c9b14700 1 -- 172.21.15.42:0/2209700384 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26c405d100 con 0x7f26c411b120 2019-12-05T06:10:00.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:00.031+0000 7f26c3fff700 1 --2- 172.21.15.42:0/2209700384 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f26c411b120 0x7f26c411a5f0 secure :-1 s=THROTTLE_DONE pgs=471 cs=0 l=1 rx=0x7f26bc0032b0 tx=0x7f26bc010d60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:00.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:00.031+0000 7f26c0ff9700 1 -- 172.21.15.42:0/2209700384 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f26bc032ba0 con 0x7f26c411b120 2019-12-05T06:10:00.206 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:00.199+0000 7f26c9b14700 1 -- 172.21.15.42:0/2209700384 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f26c405b8b0 con 0x7f26c411b120 2019-12-05T06:10:00.207 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:00.199+0000 7f26c3fff700 1 --2- 172.21.15.42:0/2209700384 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f26c411b120 0x7f26c411a5f0 secure :-1 s=THROTTLE_DONE pgs=471 cs=0 l=1 rx=0x7f26bc0032b0 tx=0x7f26bc010d60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:00.207 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:00.199+0000 7f26c0ff9700 1 -- 172.21.15.42:0/2209700384 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 258 v258) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f26bc032e00 con 0x7f26c411b120 2019-12-05T06:10:00.209 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:10:00.210 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":258,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6977,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/3355679996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3355679996},{"type":"v1","addr":"172.21.15.40:6833","nonce":3355679996}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":258}],"filesystems":[{"mdsmap":{"epoch":257,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:08:14.898258+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":94,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6929},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6929":{"gid":6929,"name":"a","rank":0,"incarnation":254,"state":"up:active","state_seq":19,"addr":"172.21.15.42:6835/2636699468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2636699468},{"type":"v1","addr":"172.21.15.42:6835","nonce":2636699468}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:10:00.210 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 258 2019-12-05T06:10:00.212 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:00.203+0000 7f26c9b14700 1 -- 172.21.15.42:0/2209700384 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f26a803b260 msgr2=0x7f26a803d770 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:10:00.212 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:00.203+0000 7f26c9b14700 1 --2- 172.21.15.42:0/2209700384 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f26a803b260 0x7f26a803d770 crc :-1 s=READY pgs=900 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:10:00.212 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:00.203+0000 7f26c9b14700 1 -- 172.21.15.42:0/2209700384 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f26c411b120 msgr2=0x7f26c411a5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:10:00.212 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:00.203+0000 7f26c9b14700 1 --2- 172.21.15.42:0/2209700384 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f26c411b120 0x7f26c411a5f0 secure :-1 s=READY pgs=471 cs=0 l=1 rx=0x7f26bc0032b0 tx=0x7f26bc010d60).stop 2019-12-05T06:10:00.212 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:00.207+0000 7f26c9b14700 1 -- 172.21.15.42:0/2209700384 shutdown_connections 2019-12-05T06:10:00.213 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:00.207+0000 7f26c9b14700 1 --2- 172.21.15.42:0/2209700384 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f26a803b260 0x7f26a803d770 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:10:00.213 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:00.207+0000 7f26c9b14700 1 --2- 172.21.15.42:0/2209700384 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f26c411b120 0x7f26c411a5f0 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:10:00.213 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:00.207+0000 7f26c9b14700 1 -- 172.21.15.42:0/2209700384 shutdown_connections 2019-12-05T06:10:00.213 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:00.207+0000 7f26c9b14700 1 -- 172.21.15.42:0/2209700384 wait complete. 2019-12-05T06:10:00.213 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:00.207+0000 7f26c9b14700 1 -- 172.21.15.42:0/2209700384 >> 172.21.15.42:0/2209700384 conn(0x7f26c407e760 msgr2=0x7f26c409f370 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:10:00.221 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:10:02.223 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:10:02.335 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:02.323+0000 7fac537d7700 1 Processor -- start 2019-12-05T06:10:02.335 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:02.323+0000 7fac537d7700 1 -- start start 2019-12-05T06:10:02.336 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:02.323+0000 7fac537d7700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fac4c08df70 0x7fac4c09f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:10:02.336 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:02.323+0000 7fac537d7700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fac4c09f820 0x7fac4c14efd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:10:02.336 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:02.323+0000 7fac537d7700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac4c09fc50 con 0x7fac4c0a7730 2019-12-05T06:10:02.336 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:02.323+0000 7fac537d7700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac4c0a0af0 con 0x7fac4c08df70 2019-12-05T06:10:02.336 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:02.323+0000 7fac537d7700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac4c08e5a0 con 0x7fac4c09f820 2019-12-05T06:10:02.336 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:02.327+0000 7fac51573700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fac4c08df70 0x7fac4c09f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:10:02.337 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:02.327+0000 7fac51573700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fac4c08df70 0x7fac4c09f2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:49534/0 (socket says 172.21.15.42:49534) 2019-12-05T06:10:02.337 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:02.327+0000 7fac51573700 1 -- 172.21.15.42:0/286511146 learned_addr learned my addr 172.21.15.42:0/286511146 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:10:02.337 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:02.327+0000 7fac52575700 1 --2- 172.21.15.42:0/286511146 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fac4c09f820 0x7fac4c14efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:10:02.337 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:02.327+0000 7fac51573700 1 -- 172.21.15.42:0/286511146 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fac4c09f820 msgr2=0x7fac4c14efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:10:02.337 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:02.327+0000 7fac51573700 1 --2- 172.21.15.42:0/286511146 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fac4c09f820 0x7fac4c14efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:10:02.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:02.327+0000 7fac51573700 1 -- 172.21.15.42:0/286511146 >> v1:172.21.15.40:6789/0 conn(0x7fac4c0a7730 legacy=0x7fac4c07e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:10:02.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:02.327+0000 7fac51573700 1 -- 172.21.15.42:0/286511146 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac4c0a0700 con 0x7fac4c08df70 2019-12-05T06:10:02.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:02.327+0000 7fac51573700 1 --2- 172.21.15.42:0/286511146 >> v2:172.21.15.42:3300/0 conn(0x7fac4c08df70 0x7fac4c09f2d0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fac3c008790 tx=0x7fac3c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:02.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:02.327+0000 7fac51573700 1 --2- 172.21.15.42:0/286511146 >> v2:172.21.15.42:3300/0 conn(0x7fac4c08df70 0x7fac4c09f2d0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fac3c008790 tx=0x7fac3c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:02.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:02.327+0000 7fac51573700 1 --2- 172.21.15.42:0/286511146 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fac4c08df70 0x7fac4c09f2d0 secure :-1 s=READY pgs=2139 cs=0 l=1 rx=0x7fac3c008790 tx=0x7fac3c00fd30).ready entity=mon.1 client_cookie=bd30208ad093586e server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:10:02.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:02.327+0000 7fac51573700 1 --2- 172.21.15.42:0/286511146 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fac4c08df70 0x7fac4c09f2d0 secure :-1 s=THROTTLE_DONE pgs=2139 cs=0 l=1 rx=0x7fac3c008790 tx=0x7fac3c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:02.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:02.327+0000 7fac50d72700 1 -- 172.21.15.42:0/286511146 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fac3c008c90 con 0x7fac4c08df70 2019-12-05T06:10:02.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:02.327+0000 7fac51573700 1 --2- 172.21.15.42:0/286511146 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fac4c08df70 0x7fac4c09f2d0 secure :-1 s=THROTTLE_DONE pgs=2139 cs=0 l=1 rx=0x7fac3c008790 tx=0x7fac3c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:02.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:02.327+0000 7fac51573700 1 --2- 172.21.15.42:0/286511146 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fac4c08df70 0x7fac4c09f2d0 secure :-1 s=THROTTLE_DONE pgs=2139 cs=0 l=1 rx=0x7fac3c008790 tx=0x7fac3c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:02.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:02.327+0000 7fac50d72700 1 -- 172.21.15.42:0/286511146 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac3c0088b0 con 0x7fac4c08df70 2019-12-05T06:10:02.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:02.327+0000 7fac50d72700 1 -- 172.21.15.42:0/286511146 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fac3c011b10 con 0x7fac4c08df70 2019-12-05T06:10:02.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:02.327+0000 7fac537d7700 1 -- 172.21.15.42:0/286511146 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fac4c08df70 msgr2=0x7fac4c09f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:10:02.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:02.327+0000 7fac537d7700 1 --2- 172.21.15.42:0/286511146 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fac4c08df70 0x7fac4c09f2d0 secure :-1 s=READY pgs=2139 cs=0 l=1 rx=0x7fac3c008790 tx=0x7fac3c00fd30).stop 2019-12-05T06:10:02.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:02.327+0000 7fac537d7700 1 -- 172.21.15.42:0/286511146 shutdown_connections 2019-12-05T06:10:02.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:02.327+0000 7fac537d7700 1 --2- 172.21.15.42:0/286511146 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fac4c09f820 0x7fac4c14efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:10:02.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:02.327+0000 7fac537d7700 1 --2- 172.21.15.42:0/286511146 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fac4c08df70 0x7fac4c09f2d0 unknown :-1 s=CLOSED pgs=2139 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:10:02.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:02.327+0000 7fac537d7700 1 -- 172.21.15.42:0/286511146 shutdown_connections 2019-12-05T06:10:02.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:02.327+0000 7fac537d7700 1 -- 172.21.15.42:0/286511146 wait complete. 2019-12-05T06:10:02.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:02.327+0000 7fac537d7700 1 -- 172.21.15.42:0/286511146 >> 172.21.15.42:0/286511146 conn(0x7fac4c0aabd0 msgr2=0x7fac4c0a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:10:02.361 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:02.331+0000 7fac537d7700 1 Processor -- start 2019-12-05T06:10:02.361 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:02.331+0000 7fac537d7700 1 -- start start 2019-12-05T06:10:02.362 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:02.331+0000 7fac537d7700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fac4c0a6970 0x7fac4c144190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:10:02.364 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:02.331+0000 7fac537d7700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac4c11ba60 con 0x7fac4c1446e0 2019-12-05T06:10:02.365 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:02.331+0000 7fac537d7700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac4c136bf0 con 0x7fac4c112450 2019-12-05T06:10:02.365 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:02.331+0000 7fac537d7700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac4c0a6190 con 0x7fac4c0a6970 2019-12-05T06:10:02.366 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:02.331+0000 7fac51d74700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fac4c112450 0x7fac4c12a1c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:49574/0 (socket says 172.21.15.42:49574) 2019-12-05T06:10:02.366 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:02.331+0000 7fac51d74700 1 -- 172.21.15.42:0/890222833 learned_addr learned my addr 172.21.15.42:0/890222833 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:10:02.366 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:02.331+0000 7fac51573700 1 --2- 172.21.15.42:0/890222833 >> v2:172.21.15.40:3300/0 conn(0x7fac4c0a6970 0x7fac4c144190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:10:02.366 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:02.335+0000 7fac427fc700 1 -- 172.21.15.42:0/890222833 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 120752160 0 0) 0x7fac4c136bf0 con 0x7fac4c112450 2019-12-05T06:10:02.367 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:02.335+0000 7fac427fc700 1 -- 172.21.15.42:0/890222833 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac340030b0 con 0x7fac4c112450 2019-12-05T06:10:02.367 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:02.335+0000 7fac51573700 1 -- 172.21.15.42:0/890222833 >> v1:172.21.15.42:6789/0 conn(0x7fac4c112450 legacy=0x7fac4c12a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:10:02.367 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:02.335+0000 7fac427fc700 1 -- 172.21.15.42:0/890222833 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 4227816726 0 0) 0x7fac340030b0 con 0x7fac4c112450 2019-12-05T06:10:02.368 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:02.335+0000 7fac51573700 1 -- 172.21.15.42:0/890222833 >> v1:172.21.15.40:6789/0 conn(0x7fac4c1446e0 legacy=0x7fac4c11b260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:10:02.368 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:02.335+0000 7fac51573700 1 -- 172.21.15.42:0/890222833 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac4c137e30 con 0x7fac4c0a6970 2019-12-05T06:10:02.369 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:02.335+0000 7fac51573700 1 --2- 172.21.15.42:0/890222833 >> v2:172.21.15.40:3300/0 conn(0x7fac4c0a6970 0x7fac4c144190 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fac48001c60 tx=0x7fac48001ca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:02.369 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:02.335+0000 7fac51573700 1 --2- 172.21.15.42:0/890222833 >> v2:172.21.15.40:3300/0 conn(0x7fac4c0a6970 0x7fac4c144190 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fac48001c60 tx=0x7fac48001ca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:02.370 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:02.335+0000 7fac51573700 1 --2- 172.21.15.42:0/890222833 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fac4c0a6970 0x7fac4c144190 secure :-1 s=READY pgs=1632 cs=0 l=1 rx=0x7fac48001c60 tx=0x7fac48001ca0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:10:02.370 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:02.335+0000 7fac51573700 1 --2- 172.21.15.42:0/890222833 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fac4c0a6970 0x7fac4c144190 secure :-1 s=THROTTLE_DONE pgs=1632 cs=0 l=1 rx=0x7fac48001c60 tx=0x7fac48001ca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:02.370 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:02.335+0000 7fac427fc700 1 -- 172.21.15.42:0/890222833 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fac48001a60 con 0x7fac4c0a6970 2019-12-05T06:10:02.370 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:02.335+0000 7fac51573700 1 --2- 172.21.15.42:0/890222833 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fac4c0a6970 0x7fac4c144190 secure :-1 s=THROTTLE_DONE pgs=1632 cs=0 l=1 rx=0x7fac48001c60 tx=0x7fac48001ca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:02.371 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:02.335+0000 7fac51573700 1 --2- 172.21.15.42:0/890222833 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fac4c0a6970 0x7fac4c144190 secure :-1 s=THROTTLE_DONE pgs=1632 cs=0 l=1 rx=0x7fac48001c60 tx=0x7fac48001ca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:02.371 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:02.335+0000 7fac427fc700 1 -- 172.21.15.42:0/890222833 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac48001870 con 0x7fac4c0a6970 2019-12-05T06:10:02.371 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:02.335+0000 7fac427fc700 1 -- 172.21.15.42:0/890222833 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fac3c00f660 con 0x7fac4c0a6970 2019-12-05T06:10:02.372 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:02.335+0000 7fac537d7700 1 -- 172.21.15.42:0/890222833 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac4c135be0 con 0x7fac4c0a6970 2019-12-05T06:10:02.372 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:02.335+0000 7fac537d7700 1 -- 172.21.15.42:0/890222833 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac4c136160 con 0x7fac4c0a6970 2019-12-05T06:10:02.372 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:02.335+0000 7fac51573700 1 --2- 172.21.15.42:0/890222833 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fac4c0a6970 0x7fac4c144190 secure :-1 s=THROTTLE_DONE pgs=1632 cs=0 l=1 rx=0x7fac48001c60 tx=0x7fac48001ca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:02.372 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:02.335+0000 7fac427fc700 1 -- 172.21.15.42:0/890222833 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fac3c005540 con 0x7fac4c0a6970 2019-12-05T06:10:02.373 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:02.335+0000 7fac51573700 1 --2- 172.21.15.42:0/890222833 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fac4c0a6970 0x7fac4c144190 secure :-1 s=THROTTLE_DONE pgs=1632 cs=0 l=1 rx=0x7fac48001c60 tx=0x7fac48001ca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:02.373 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:02.339+0000 7fac427fc700 1 --2- 172.21.15.42:0/890222833 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fac3403f620 0x7fac34041b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:10:02.373 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:02.339+0000 7fac427fc700 1 -- 172.21.15.42:0/890222833 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9901+0+0 (secure 0 0 0) 0x7fac3c04b0a0 con 0x7fac4c0a6970 2019-12-05T06:10:02.373 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:02.339+0000 7fac51d74700 1 --2- 172.21.15.42:0/890222833 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fac3403f620 0x7fac34041b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:10:02.374 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:02.339+0000 7fac51d74700 1 --2- 172.21.15.42:0/890222833 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fac3403f620 0x7fac34041b30 crc :-1 s=READY pgs=901 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:10:02.374 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:02.339+0000 7fac537d7700 1 -- 172.21.15.42:0/890222833 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac4c05d100 con 0x7fac4c0a6970 2019-12-05T06:10:02.375 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:02.351+0000 7fac51573700 1 --2- 172.21.15.42:0/890222833 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fac4c0a6970 0x7fac4c144190 secure :-1 s=THROTTLE_DONE pgs=1632 cs=0 l=1 rx=0x7fac48001c60 tx=0x7fac48001ca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:02.375 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:02.351+0000 7fac427fc700 1 -- 172.21.15.42:0/890222833 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fac3c032360 con 0x7fac4c0a6970 2019-12-05T06:10:02.525 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:02.519+0000 7fac537d7700 1 -- 172.21.15.42:0/890222833 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fac4c05b8b0 con 0x7fac4c0a6970 2019-12-05T06:10:02.526 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:02.519+0000 7fac51573700 1 --2- 172.21.15.42:0/890222833 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fac4c0a6970 0x7fac4c144190 secure :-1 s=THROTTLE_DONE pgs=1632 cs=0 l=1 rx=0x7fac48001c60 tx=0x7fac48001ca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:02.526 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:02.519+0000 7fac427fc700 1 -- 172.21.15.42:0/890222833 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 258 v258) v1 ==== 95+0+2112 (secure 0 0 0) 0x7fac3c007160 con 0x7fac4c0a6970 2019-12-05T06:10:02.528 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:10:02.528 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":258,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6977,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/3355679996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3355679996},{"type":"v1","addr":"172.21.15.40:6833","nonce":3355679996}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":258}],"filesystems":[{"mdsmap":{"epoch":257,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:08:14.898258+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":94,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6929},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6929":{"gid":6929,"name":"a","rank":0,"incarnation":254,"state":"up:active","state_seq":19,"addr":"172.21.15.42:6835/2636699468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2636699468},{"type":"v1","addr":"172.21.15.42:6835","nonce":2636699468}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:10:02.529 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 258 2019-12-05T06:10:02.531 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:02.523+0000 7fac537d7700 1 -- 172.21.15.42:0/890222833 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fac3403f620 msgr2=0x7fac34041b30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:10:02.531 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:02.523+0000 7fac537d7700 1 --2- 172.21.15.42:0/890222833 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fac3403f620 0x7fac34041b30 crc :-1 s=READY pgs=901 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:10:02.531 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:02.523+0000 7fac537d7700 1 -- 172.21.15.42:0/890222833 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fac4c0a6970 msgr2=0x7fac4c144190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:10:02.531 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:02.523+0000 7fac537d7700 1 --2- 172.21.15.42:0/890222833 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fac4c0a6970 0x7fac4c144190 secure :-1 s=READY pgs=1632 cs=0 l=1 rx=0x7fac48001c60 tx=0x7fac48001ca0).stop 2019-12-05T06:10:02.532 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:02.523+0000 7fac537d7700 1 -- 172.21.15.42:0/890222833 shutdown_connections 2019-12-05T06:10:02.532 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:02.523+0000 7fac537d7700 1 --2- 172.21.15.42:0/890222833 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fac3403f620 0x7fac34041b30 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:10:02.532 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:02.523+0000 7fac537d7700 1 --2- 172.21.15.42:0/890222833 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fac4c0a6970 0x7fac4c144190 unknown :-1 s=CLOSED pgs=1632 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:10:02.533 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:02.523+0000 7fac537d7700 1 -- 172.21.15.42:0/890222833 shutdown_connections 2019-12-05T06:10:02.533 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:02.523+0000 7fac537d7700 1 -- 172.21.15.42:0/890222833 wait complete. 2019-12-05T06:10:02.533 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:02.523+0000 7fac537d7700 1 -- 172.21.15.42:0/890222833 >> 172.21.15.42:0/890222833 conn(0x7fac4c07df50 msgr2=0x7fac4c11be30 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:10:02.540 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:10:04.542 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:10:04.645 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:04.627+0000 7fb35f88e700 1 Processor -- start 2019-12-05T06:10:04.645 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:04.627+0000 7fb35f88e700 1 -- start start 2019-12-05T06:10:04.646 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:04.627+0000 7fb35f88e700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb35808df80 0x7fb35809f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:10:04.646 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:04.627+0000 7fb35f88e700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb35809fc60 con 0x7fb3580a7740 2019-12-05T06:10:04.646 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:04.627+0000 7fb35f88e700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3580a0710 con 0x7fb35809f830 2019-12-05T06:10:04.646 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:04.627+0000 7fb35f88e700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3580a0b00 con 0x7fb35808df80 2019-12-05T06:10:04.646 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:04.627+0000 7fb35e62c700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fb35809f830 0x7fb35814efe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:49588/0 (socket says 172.21.15.42:49588) 2019-12-05T06:10:04.646 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:04.627+0000 7fb35e62c700 1 -- 172.21.15.42:0/4178869785 learned_addr learned my addr 172.21.15.42:0/4178869785 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:10:04.647 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:04.627+0000 7fb35d62a700 1 --2- 172.21.15.42:0/4178869785 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb35808df80 0x7fb35809f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:10:04.647 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:04.631+0000 7fb35ce29700 1 -- 172.21.15.42:0/4178869785 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3125777324 0 0) 0x7fb35809fc60 con 0x7fb3580a7740 2019-12-05T06:10:04.647 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:04.631+0000 7fb35ce29700 1 -- 172.21.15.42:0/4178869785 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3440030b0 con 0x7fb3580a7740 2019-12-05T06:10:04.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:04.631+0000 7fb35ce29700 1 -- 172.21.15.42:0/4178869785 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1762615739 0 0) 0x7fb3580a0710 con 0x7fb35809f830 2019-12-05T06:10:04.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:04.631+0000 7fb35ce29700 1 -- 172.21.15.42:0/4178869785 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb35809fc60 con 0x7fb35809f830 2019-12-05T06:10:04.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:04.631+0000 7fb35d62a700 1 -- 172.21.15.42:0/4178869785 >> v1:172.21.15.42:6789/0 conn(0x7fb35809f830 legacy=0x7fb35814efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:10:04.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:04.631+0000 7fb35d62a700 1 -- 172.21.15.42:0/4178869785 >> v1:172.21.15.40:6789/0 conn(0x7fb3580a7740 legacy=0x7fb35807e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:10:04.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:04.631+0000 7fb35d62a700 1 -- 172.21.15.42:0/4178869785 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb35807ef20 con 0x7fb35808df80 2019-12-05T06:10:04.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:04.631+0000 7fb35d62a700 1 --2- 172.21.15.42:0/4178869785 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb35808df80 0x7fb35809f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fb348002f30 tx=0x7fb34800fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:04.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:04.631+0000 7fb35d62a700 1 --2- 172.21.15.42:0/4178869785 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb35808df80 0x7fb35809f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fb348002f30 tx=0x7fb34800fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:04.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:04.631+0000 7fb35d62a700 1 --2- 172.21.15.42:0/4178869785 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb35808df80 0x7fb35809f2e0 secure :-1 s=READY pgs=472 cs=0 l=1 rx=0x7fb348002f30 tx=0x7fb34800fbf0).ready entity=mon.2 client_cookie=32733156878c3d8d server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:10:04.650 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:04.631+0000 7fb35d62a700 1 --2- 172.21.15.42:0/4178869785 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb35808df80 0x7fb35809f2e0 secure :-1 s=THROTTLE_DONE pgs=472 cs=0 l=1 rx=0x7fb348002f30 tx=0x7fb34800fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:04.650 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:04.631+0000 7fb35ce29700 1 -- 172.21.15.42:0/4178869785 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb3480112b0 con 0x7fb35808df80 2019-12-05T06:10:04.650 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:04.631+0000 7fb35d62a700 1 --2- 172.21.15.42:0/4178869785 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb35808df80 0x7fb35809f2e0 secure :-1 s=THROTTLE_DONE pgs=472 cs=0 l=1 rx=0x7fb348002f30 tx=0x7fb34800fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:04.650 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:04.631+0000 7fb35ce29700 1 -- 172.21.15.42:0/4178869785 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb348008bc0 con 0x7fb35808df80 2019-12-05T06:10:04.650 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:04.631+0000 7fb35d62a700 1 --2- 172.21.15.42:0/4178869785 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb35808df80 0x7fb35809f2e0 secure :-1 s=THROTTLE_DONE pgs=472 cs=0 l=1 rx=0x7fb348002f30 tx=0x7fb34800fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:04.650 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:04.631+0000 7fb35ce29700 1 -- 172.21.15.42:0/4178869785 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb348011890 con 0x7fb35808df80 2019-12-05T06:10:04.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:04.631+0000 7fb35f88e700 1 -- 172.21.15.42:0/4178869785 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb35808df80 msgr2=0x7fb35809f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:10:04.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:04.631+0000 7fb35f88e700 1 --2- 172.21.15.42:0/4178869785 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb35808df80 0x7fb35809f2e0 secure :-1 s=READY pgs=472 cs=0 l=1 rx=0x7fb348002f30 tx=0x7fb34800fbf0).stop 2019-12-05T06:10:04.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:04.631+0000 7fb35f88e700 1 -- 172.21.15.42:0/4178869785 shutdown_connections 2019-12-05T06:10:04.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:04.631+0000 7fb35f88e700 1 --2- 172.21.15.42:0/4178869785 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb35808df80 0x7fb35809f2e0 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:10:04.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:04.631+0000 7fb35f88e700 1 -- 172.21.15.42:0/4178869785 shutdown_connections 2019-12-05T06:10:04.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:04.631+0000 7fb35f88e700 1 -- 172.21.15.42:0/4178869785 wait complete. 2019-12-05T06:10:04.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:04.631+0000 7fb35f88e700 1 -- 172.21.15.42:0/4178869785 >> 172.21.15.42:0/4178869785 conn(0x7fb3580aabe0 msgr2=0x7fb3580a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:10:04.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:04.635+0000 7fb35f88e700 1 Processor -- start 2019-12-05T06:10:04.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:04.635+0000 7fb35f88e700 1 -- start start 2019-12-05T06:10:04.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:04.635+0000 7fb35f88e700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb3580a4b80 0x7fb3581448b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:10:04.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:04.635+0000 7fb35f88e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb358129300 0x7fb35807b370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:10:04.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:04.635+0000 7fb35f88e700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb35807b8c0 con 0x7fb358064400 2019-12-05T06:10:04.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:04.635+0000 7fb35f88e700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3581454b0 con 0x7fb358129300 2019-12-05T06:10:04.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:04.635+0000 7fb35f88e700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb358144f70 con 0x7fb3580a4b80 2019-12-05T06:10:04.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:04.635+0000 7fb35d62a700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fb358064400 0x7fb358128b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:49592/0 (socket says 172.21.15.42:49592) 2019-12-05T06:10:04.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:04.635+0000 7fb35d62a700 1 -- 172.21.15.42:0/2708345515 learned_addr learned my addr 172.21.15.42:0/2708345515 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:10:04.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:04.635+0000 7fb35e62c700 1 --2- 172.21.15.42:0/2708345515 >> v2:172.21.15.42:3300/0 conn(0x7fb358129300 0x7fb35807b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:10:04.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:04.635+0000 7fb35de2b700 1 --2- 172.21.15.42:0/2708345515 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb3580a4b80 0x7fb3581448b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:10:04.654 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:04.635+0000 7fb34e7fc700 1 -- 172.21.15.42:0/2708345515 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 927061459 0 0) 0x7fb35807b8c0 con 0x7fb358064400 2019-12-05T06:10:04.654 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:04.635+0000 7fb34e7fc700 1 -- 172.21.15.42:0/2708345515 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3400030b0 con 0x7fb358064400 2019-12-05T06:10:04.654 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:04.635+0000 7fb35e62c700 1 -- 172.21.15.42:0/2708345515 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb3580a4b80 msgr2=0x7fb3581448b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:10:04.654 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:04.635+0000 7fb35e62c700 1 --2- 172.21.15.42:0/2708345515 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb3580a4b80 0x7fb3581448b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:10:04.654 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:04.635+0000 7fb35e62c700 1 -- 172.21.15.42:0/2708345515 >> v1:172.21.15.42:6789/0 conn(0x7fb358064400 legacy=0x7fb358128b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:10:04.655 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:04.635+0000 7fb35e62c700 1 -- 172.21.15.42:0/2708345515 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb35807baf0 con 0x7fb358129300 2019-12-05T06:10:04.655 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:04.635+0000 7fb35de2b700 1 --2- 172.21.15.42:0/2708345515 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb3580a4b80 0x7fb3581448b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:10:04.655 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:04.635+0000 7fb35e62c700 1 --2- 172.21.15.42:0/2708345515 >> v2:172.21.15.42:3300/0 conn(0x7fb358129300 0x7fb35807b370 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fb358063380 tx=0x7fb350010810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:04.655 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:04.635+0000 7fb35e62c700 1 --2- 172.21.15.42:0/2708345515 >> v2:172.21.15.42:3300/0 conn(0x7fb358129300 0x7fb35807b370 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fb358063380 tx=0x7fb350010810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:04.655 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:04.635+0000 7fb35e62c700 1 --2- 172.21.15.42:0/2708345515 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb358129300 0x7fb35807b370 secure :-1 s=READY pgs=2143 cs=0 l=1 rx=0x7fb358063380 tx=0x7fb350010810).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:10:04.655 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:04.639+0000 7fb35e62c700 1 --2- 172.21.15.42:0/2708345515 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb358129300 0x7fb35807b370 secure :-1 s=THROTTLE_DONE pgs=2143 cs=0 l=1 rx=0x7fb358063380 tx=0x7fb350010810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:04.655 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:04.639+0000 7fb34e7fc700 1 -- 172.21.15.42:0/2708345515 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb350005b40 con 0x7fb358129300 2019-12-05T06:10:04.656 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:04.639+0000 7fb35e62c700 1 --2- 172.21.15.42:0/2708345515 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb358129300 0x7fb35807b370 secure :-1 s=THROTTLE_DONE pgs=2143 cs=0 l=1 rx=0x7fb358063380 tx=0x7fb350010810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:04.656 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:04.639+0000 7fb35e62c700 1 --2- 172.21.15.42:0/2708345515 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb358129300 0x7fb35807b370 secure :-1 s=THROTTLE_DONE pgs=2143 cs=0 l=1 rx=0x7fb358063380 tx=0x7fb350010810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:04.656 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:04.639+0000 7fb34e7fc700 1 -- 172.21.15.42:0/2708345515 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb350005760 con 0x7fb358129300 2019-12-05T06:10:04.656 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:04.639+0000 7fb35f88e700 1 -- 172.21.15.42:0/2708345515 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb35807bdc0 con 0x7fb358129300 2019-12-05T06:10:04.656 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:04.639+0000 7fb34e7fc700 1 -- 172.21.15.42:0/2708345515 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb3500062b0 con 0x7fb358129300 2019-12-05T06:10:04.656 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:04.639+0000 7fb35f88e700 1 -- 172.21.15.42:0/2708345515 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb358130150 con 0x7fb358129300 2019-12-05T06:10:04.656 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:04.639+0000 7fb35e62c700 1 --2- 172.21.15.42:0/2708345515 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb358129300 0x7fb35807b370 secure :-1 s=THROTTLE_DONE pgs=2143 cs=0 l=1 rx=0x7fb358063380 tx=0x7fb350010810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:04.658 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:04.639+0000 7fb34e7fc700 1 -- 172.21.15.42:0/2708345515 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fb350006580 con 0x7fb358129300 2019-12-05T06:10:04.659 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:04.639+0000 7fb35e62c700 1 --2- 172.21.15.42:0/2708345515 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb358129300 0x7fb35807b370 secure :-1 s=THROTTLE_DONE pgs=2143 cs=0 l=1 rx=0x7fb358063380 tx=0x7fb350010810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:04.659 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:04.639+0000 7fb34e7fc700 1 --2- 172.21.15.42:0/2708345515 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb34003b2d0 0x7fb34003d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:10:04.659 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:04.639+0000 7fb34e7fc700 1 -- 172.21.15.42:0/2708345515 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9901+0+0 (secure 0 0 0) 0x7fb35004aed0 con 0x7fb358129300 2019-12-05T06:10:04.660 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:04.639+0000 7fb35de2b700 1 --2- 172.21.15.42:0/2708345515 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb34003b2d0 0x7fb34003d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:10:04.660 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:04.639+0000 7fb35de2b700 1 --2- 172.21.15.42:0/2708345515 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb34003b2d0 0x7fb34003d7e0 crc :-1 s=READY pgs=902 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:10:04.660 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:04.643+0000 7fb35f88e700 1 -- 172.21.15.42:0/2708345515 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb358064830 con 0x7fb358129300 2019-12-05T06:10:04.660 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:04.651+0000 7fb35e62c700 1 --2- 172.21.15.42:0/2708345515 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb358129300 0x7fb35807b370 secure :-1 s=THROTTLE_DONE pgs=2143 cs=0 l=1 rx=0x7fb358063380 tx=0x7fb350010810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:04.660 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:04.651+0000 7fb34e7fc700 1 -- 172.21.15.42:0/2708345515 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fb350031f50 con 0x7fb358129300 2019-12-05T06:10:04.828 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:04.819+0000 7fb35f88e700 1 -- 172.21.15.42:0/2708345515 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb35805d110 con 0x7fb358129300 2019-12-05T06:10:04.830 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:04.823+0000 7fb35e62c700 1 --2- 172.21.15.42:0/2708345515 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb358129300 0x7fb35807b370 secure :-1 s=THROTTLE_DONE pgs=2143 cs=0 l=1 rx=0x7fb358063380 tx=0x7fb350010810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:04.830 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:04.823+0000 7fb34e7fc700 1 -- 172.21.15.42:0/2708345515 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 258 v258) v1 ==== 95+0+2112 (secure 0 0 0) 0x7fb3500321b0 con 0x7fb358129300 2019-12-05T06:10:04.830 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 258 2019-12-05T06:10:04.831 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:10:04.831 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":258,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6977,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/3355679996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3355679996},{"type":"v1","addr":"172.21.15.40:6833","nonce":3355679996}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":258}],"filesystems":[{"mdsmap":{"epoch":257,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:08:14.898258+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":94,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6929},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6929":{"gid":6929,"name":"a","rank":0,"incarnation":254,"state":"up:active","state_seq":19,"addr":"172.21.15.42:6835/2636699468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2636699468},{"type":"v1","addr":"172.21.15.42:6835","nonce":2636699468}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:10:04.833 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:04.823+0000 7fb35f88e700 1 -- 172.21.15.42:0/2708345515 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb34003b2d0 msgr2=0x7fb34003d7e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:10:04.833 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:04.823+0000 7fb35f88e700 1 --2- 172.21.15.42:0/2708345515 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb34003b2d0 0x7fb34003d7e0 crc :-1 s=READY pgs=902 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:10:04.833 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:04.823+0000 7fb35f88e700 1 -- 172.21.15.42:0/2708345515 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb358129300 msgr2=0x7fb35807b370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:10:04.833 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:04.823+0000 7fb35f88e700 1 --2- 172.21.15.42:0/2708345515 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb358129300 0x7fb35807b370 secure :-1 s=READY pgs=2143 cs=0 l=1 rx=0x7fb358063380 tx=0x7fb350010810).stop 2019-12-05T06:10:04.834 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:04.827+0000 7fb35f88e700 1 -- 172.21.15.42:0/2708345515 shutdown_connections 2019-12-05T06:10:04.834 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:04.827+0000 7fb35f88e700 1 --2- 172.21.15.42:0/2708345515 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb34003b2d0 0x7fb34003d7e0 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:10:04.834 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:04.827+0000 7fb35f88e700 1 --2- 172.21.15.42:0/2708345515 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb358129300 0x7fb35807b370 unknown :-1 s=CLOSED pgs=2143 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:10:04.834 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:04.827+0000 7fb35f88e700 1 --2- 172.21.15.42:0/2708345515 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb3580a4b80 0x7fb3581448b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:10:04.834 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:04.827+0000 7fb35f88e700 1 -- 172.21.15.42:0/2708345515 shutdown_connections 2019-12-05T06:10:04.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:04.827+0000 7fb35f88e700 1 -- 172.21.15.42:0/2708345515 wait complete. 2019-12-05T06:10:04.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:04.827+0000 7fb35f88e700 1 -- 172.21.15.42:0/2708345515 >> 172.21.15.42:0/2708345515 conn(0x7fb35809f040 msgr2=0x7fb35807e540 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:10:04.846 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:10:06.847 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:10:06.952 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:06.943+0000 7f5faa10e700 1 Processor -- start 2019-12-05T06:10:06.953 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:06.943+0000 7f5faa10e700 1 -- start start 2019-12-05T06:10:06.953 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:06.943+0000 7f5faa10e700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5fa40a7740 0x7f5fa408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:10:06.953 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:06.943+0000 7f5faa10e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5fa407e720 0x7f5fa407eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:10:06.953 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:06.943+0000 7f5faa10e700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fa40a0710 con 0x7f5fa409f2e0 2019-12-05T06:10:06.953 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:06.943+0000 7f5faa10e700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5fa40a0b00 con 0x7f5fa40a7740 2019-12-05T06:10:06.953 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:06.943+0000 7f5faa10e700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5fa408e5b0 con 0x7f5fa407e720 2019-12-05T06:10:06.953 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:06.943+0000 7f5fa37fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5fa407e720 0x7f5fa407eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:10:06.954 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:06.943+0000 7f5fa37fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5fa407e720 0x7f5fa407eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:49564/0 (socket says 172.21.15.42:49564) 2019-12-05T06:10:06.954 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:06.943+0000 7f5fa37fe700 1 -- 172.21.15.42:0/2902343539 learned_addr learned my addr 172.21.15.42:0/2902343539 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:10:06.954 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:06.943+0000 7f5fa3fff700 1 --2- 172.21.15.42:0/2902343539 >> v2:172.21.15.40:3300/0 conn(0x7f5fa40a7740 0x7f5fa408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:10:06.954 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:06.943+0000 7f5fa37fe700 1 -- 172.21.15.42:0/2902343539 >> v2:172.21.15.40:3300/0 conn(0x7f5fa40a7740 msgr2=0x7f5fa408df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:10:06.954 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:06.943+0000 7f5fa37fe700 1 --2- 172.21.15.42:0/2902343539 >> v2:172.21.15.40:3300/0 conn(0x7f5fa40a7740 0x7f5fa408df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:10:06.954 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:06.943+0000 7f5fa37fe700 1 -- 172.21.15.42:0/2902343539 >> v1:172.21.15.42:6789/0 conn(0x7f5fa409f2e0 legacy=0x7f5fa409f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:10:06.955 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:06.943+0000 7f5fa37fe700 1 -- 172.21.15.42:0/2902343539 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fa407f0a0 con 0x7f5fa407e720 2019-12-05T06:10:06.955 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:06.943+0000 7f5fa3fff700 1 --2- 172.21.15.42:0/2902343539 >> v2:172.21.15.40:3300/0 conn(0x7f5fa40a7740 0x7f5fa408df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:10:06.955 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:06.943+0000 7f5fa2ffd700 1 -- 172.21.15.42:0/2902343539 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2144286408 0 0) 0x7f5fa40a0710 con 0x7f5fa409f2e0 2019-12-05T06:10:06.955 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:06.943+0000 7f5fa37fe700 1 --2- 172.21.15.42:0/2902343539 >> v2:172.21.15.42:3300/0 conn(0x7f5fa407e720 0x7f5fa407eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f5f8c002f30 tx=0x7f5f8c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:06.955 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:06.943+0000 7f5fa37fe700 1 --2- 172.21.15.42:0/2902343539 >> v2:172.21.15.42:3300/0 conn(0x7f5fa407e720 0x7f5fa407eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f5f8c002f30 tx=0x7f5f8c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:06.955 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:06.943+0000 7f5fa37fe700 1 --2- 172.21.15.42:0/2902343539 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5fa407e720 0x7f5fa407eb50 secure :-1 s=READY pgs=2145 cs=0 l=1 rx=0x7f5f8c002f30 tx=0x7f5f8c00fd30).ready entity=mon.1 client_cookie=31fd1645f58ef8cf server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:10:06.956 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:06.943+0000 7f5fa37fe700 1 --2- 172.21.15.42:0/2902343539 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5fa407e720 0x7f5fa407eb50 secure :-1 s=THROTTLE_DONE pgs=2145 cs=0 l=1 rx=0x7f5f8c002f30 tx=0x7f5f8c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:06.956 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:06.943+0000 7f5fa2ffd700 1 -- 172.21.15.42:0/2902343539 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5f8c011370 con 0x7f5fa407e720 2019-12-05T06:10:06.956 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:06.943+0000 7f5fa37fe700 1 --2- 172.21.15.42:0/2902343539 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5fa407e720 0x7f5fa407eb50 secure :-1 s=THROTTLE_DONE pgs=2145 cs=0 l=1 rx=0x7f5f8c002f30 tx=0x7f5f8c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:06.957 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:06.943+0000 7f5fa37fe700 1 --2- 172.21.15.42:0/2902343539 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5fa407e720 0x7f5fa407eb50 secure :-1 s=THROTTLE_DONE pgs=2145 cs=0 l=1 rx=0x7f5f8c002f30 tx=0x7f5f8c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:06.957 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:06.943+0000 7f5fa2ffd700 1 -- 172.21.15.42:0/2902343539 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f8c008bb0 con 0x7f5fa407e720 2019-12-05T06:10:06.957 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:06.943+0000 7f5fa2ffd700 1 -- 172.21.15.42:0/2902343539 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5f8c011950 con 0x7f5fa407e720 2019-12-05T06:10:06.957 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:06.943+0000 7f5faa10e700 1 -- 172.21.15.42:0/2902343539 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5fa407e720 msgr2=0x7f5fa407eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:10:06.957 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:06.943+0000 7f5faa10e700 1 --2- 172.21.15.42:0/2902343539 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5fa407e720 0x7f5fa407eb50 secure :-1 s=READY pgs=2145 cs=0 l=1 rx=0x7f5f8c002f30 tx=0x7f5f8c00fd30).stop 2019-12-05T06:10:06.957 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:06.943+0000 7f5faa10e700 1 -- 172.21.15.42:0/2902343539 shutdown_connections 2019-12-05T06:10:06.957 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:06.943+0000 7f5faa10e700 1 --2- 172.21.15.42:0/2902343539 >> v2:172.21.15.40:3300/0 conn(0x7f5fa40a7740 0x7f5fa408df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:10:06.958 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:06.943+0000 7f5faa10e700 1 --2- 172.21.15.42:0/2902343539 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5fa407e720 0x7f5fa407eb50 unknown :-1 s=CLOSED pgs=2145 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:10:06.958 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:06.943+0000 7f5faa10e700 1 -- 172.21.15.42:0/2902343539 shutdown_connections 2019-12-05T06:10:06.958 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:06.943+0000 7f5faa10e700 1 -- 172.21.15.42:0/2902343539 wait complete. 2019-12-05T06:10:06.958 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:06.943+0000 7f5faa10e700 1 -- 172.21.15.42:0/2902343539 >> 172.21.15.42:0/2902343539 conn(0x7f5fa40aabe0 msgr2=0x7f5fa40a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:10:06.963 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:06.947+0000 7f5faa10e700 1 Processor -- start 2019-12-05T06:10:06.963 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:06.947+0000 7f5faa10e700 1 -- start start 2019-12-05T06:10:06.963 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:06.947+0000 7f5faa10e700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5fa411c970 0x7f5fa4136fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:10:06.963 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:06.947+0000 7f5faa10e700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fa40a4330 con 0x7f5fa40a3ca0 2019-12-05T06:10:06.964 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:06.947+0000 7f5faa10e700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fa412e830 con 0x7f5fa412b590 2019-12-05T06:10:06.964 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:06.947+0000 7f5faa10e700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5fa412ebe0 con 0x7f5fa411c970 2019-12-05T06:10:06.964 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:06.947+0000 7f5fa37fe700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f5fa412b590 0x7f5fa4133790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:49610/0 (socket says 172.21.15.42:49610) 2019-12-05T06:10:06.964 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:06.947+0000 7f5fa37fe700 1 -- 172.21.15.42:0/1277987794 learned_addr learned my addr 172.21.15.42:0/1277987794 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:10:06.964 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:06.947+0000 7f5fa8eac700 1 --2- 172.21.15.42:0/1277987794 >> v2:172.21.15.40:3300/0 conn(0x7f5fa411c970 0x7f5fa4136fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:10:06.964 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:06.947+0000 7f5fa0ff9700 1 -- 172.21.15.42:0/1277987794 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1322389294 0 0) 0x7f5fa412e830 con 0x7f5fa412b590 2019-12-05T06:10:06.965 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:06.947+0000 7f5fa0ff9700 1 -- 172.21.15.42:0/1277987794 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f880030b0 con 0x7f5fa412b590 2019-12-05T06:10:06.965 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:06.951+0000 7f5fa0ff9700 1 -- 172.21.15.42:0/1277987794 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 856489032 0 0) 0x7f5f880030b0 con 0x7f5fa412b590 2019-12-05T06:10:06.965 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:06.951+0000 7f5fa0ff9700 1 -- 172.21.15.42:0/1277987794 --> v1:172.21.15.42:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5fa412e830 con 0x7f5fa412b590 2019-12-05T06:10:06.965 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:06.951+0000 7f5fa0ff9700 1 -- 172.21.15.42:0/1277987794 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7f5f8c011370 con 0x7f5fa412b590 2019-12-05T06:10:06.965 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:06.951+0000 7f5fa0ff9700 1 -- 172.21.15.42:0/1277987794 >> v2:172.21.15.40:3300/0 conn(0x7f5fa411c970 msgr2=0x7f5fa4136fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:10:06.965 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:06.951+0000 7f5fa0ff9700 1 --2- 172.21.15.42:0/1277987794 >> v2:172.21.15.40:3300/0 conn(0x7f5fa411c970 0x7f5fa4136fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:10:06.965 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:06.951+0000 7f5fa0ff9700 1 -- 172.21.15.42:0/1277987794 >> v1:172.21.15.42:6789/0 conn(0x7f5fa412b590 legacy=0x7f5fa4133790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:10:06.966 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:06.951+0000 7f5fa0ff9700 1 -- 172.21.15.42:0/1277987794 >> v1:172.21.15.40:6789/0 conn(0x7f5fa40a3ca0 legacy=0x7f5fa412ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:10:06.966 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:06.951+0000 7f5fa0ff9700 1 --2- 172.21.15.42:0/1277987794 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5f880038a0 0x7f5f88003cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:10:06.966 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:06.951+0000 7f5fa0ff9700 1 --2- 172.21.15.42:0/1277987794 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5f88004420 0x7f5f88004850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:10:06.966 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:06.951+0000 7f5fa0ff9700 1 --2- 172.21.15.42:0/1277987794 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5f88004e40 0x7f5f8800fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:10:06.966 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:06.951+0000 7f5fa0ff9700 1 -- 172.21.15.42:0/1277987794 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fa412ebe0 con 0x7f5f88004e40 2019-12-05T06:10:06.967 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:06.951+0000 7f5fa0ff9700 1 -- 172.21.15.42:0/1277987794 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f88010090 con 0x7f5f88004420 2019-12-05T06:10:06.967 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:06.951+0000 7f5fa0ff9700 1 -- 172.21.15.42:0/1277987794 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f880102d0 con 0x7f5f880038a0 2019-12-05T06:10:06.967 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:06.951+0000 7f5fa37fe700 1 --2- 172.21.15.42:0/1277987794 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5f88004420 0x7f5f88004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:10:06.967 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:06.951+0000 7f5fa8eac700 1 --2- 172.21.15.42:0/1277987794 >> v2:172.21.15.40:3300/0 conn(0x7f5fa411c970 0x7f5fa4136fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:10:06.967 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:06.951+0000 7f5fa3fff700 1 --2- 172.21.15.42:0/1277987794 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5f880038a0 0x7f5f88003cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:10:06.967 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:06.951+0000 7f5fa8eac700 1 --2- 172.21.15.42:0/1277987794 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5f88004e40 0x7f5f8800fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:10:06.968 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:06.951+0000 7f5fa37fe700 1 -- 172.21.15.42:0/1277987794 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5f880038a0 msgr2=0x7f5f88003cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:10:06.968 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:06.951+0000 7f5fa37fe700 1 --2- 172.21.15.42:0/1277987794 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5f880038a0 0x7f5f88003cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:10:06.970 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:06.951+0000 7f5fa37fe700 1 -- 172.21.15.42:0/1277987794 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5f88004e40 msgr2=0x7f5f8800fa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:10:06.971 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:06.951+0000 7f5fa37fe700 1 --2- 172.21.15.42:0/1277987794 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5f88004e40 0x7f5f8800fa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:10:06.971 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:06.951+0000 7f5fa37fe700 1 -- 172.21.15.42:0/1277987794 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fa411adc0 con 0x7f5f88004420 2019-12-05T06:10:06.971 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:06.951+0000 7f5faa10e700 1 -- 172.21.15.42:0/1277987794 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fa411cda0 con 0x7f5f88004420 2019-12-05T06:10:06.971 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:06.951+0000 7f5fa37fe700 1 --2- 172.21.15.42:0/1277987794 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5f88004420 0x7f5f88004850 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f5f8c0081d0 tx=0x7f5f8c00f800).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:06.972 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:06.951+0000 7f5fa8eac700 1 --2- 172.21.15.42:0/1277987794 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5f88004e40 0x7f5f8800fa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).send_auth_request state changed! 2019-12-05T06:10:06.972 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:06.951+0000 7f5fa8eac700 1 -- 172.21.15.42:0/1277987794 reap_dead start 2019-12-05T06:10:06.972 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:06.951+0000 7f5faa10e700 1 -- 172.21.15.42:0/1277987794 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fa40a6fa0 con 0x7f5f88004420 2019-12-05T06:10:06.973 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:06.951+0000 7f5fa37fe700 1 --2- 172.21.15.42:0/1277987794 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5f88004420 0x7f5f88004850 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f5f8c0081d0 tx=0x7f5f8c00f800).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:06.973 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:06.951+0000 7f5fa37fe700 1 --2- 172.21.15.42:0/1277987794 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5f88004420 0x7f5f88004850 secure :-1 s=READY pgs=2147 cs=0 l=1 rx=0x7f5f8c0081d0 tx=0x7f5f8c00f800).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:10:06.973 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:06.951+0000 7f5fa37fe700 1 --2- 172.21.15.42:0/1277987794 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5f88004420 0x7f5f88004850 secure :-1 s=THROTTLE_DONE pgs=2147 cs=0 l=1 rx=0x7f5f8c0081d0 tx=0x7f5f8c00f800).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:06.973 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:06.951+0000 7f5fa37fe700 1 --2- 172.21.15.42:0/1277987794 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5f88004420 0x7f5f88004850 secure :-1 s=THROTTLE_DONE pgs=2147 cs=0 l=1 rx=0x7f5f8c0081d0 tx=0x7f5f8c00f800).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:06.973 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:06.951+0000 7f5fa0ff9700 1 -- 172.21.15.42:0/1277987794 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5f8c005540 con 0x7f5f88004420 2019-12-05T06:10:06.973 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:06.951+0000 7f5fa37fe700 1 --2- 172.21.15.42:0/1277987794 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5f88004420 0x7f5f88004850 secure :-1 s=THROTTLE_DONE pgs=2147 cs=0 l=1 rx=0x7f5f8c0081d0 tx=0x7f5f8c00f800).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:06.973 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:06.951+0000 7f5fa0ff9700 1 -- 172.21.15.42:0/1277987794 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f8c007150 con 0x7f5f88004420 2019-12-05T06:10:06.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:06.951+0000 7f5fa0ff9700 1 -- 172.21.15.42:0/1277987794 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5f8c005b20 con 0x7f5f88004420 2019-12-05T06:10:06.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:06.951+0000 7f5fa37fe700 1 --2- 172.21.15.42:0/1277987794 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5f88004420 0x7f5f88004850 secure :-1 s=THROTTLE_DONE pgs=2147 cs=0 l=1 rx=0x7f5f8c0081d0 tx=0x7f5f8c00f800).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:06.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:06.951+0000 7f5fa0ff9700 1 -- 172.21.15.42:0/1277987794 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f5f8c00a4f0 con 0x7f5f88004420 2019-12-05T06:10:06.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:06.951+0000 7f5fa37fe700 1 --2- 172.21.15.42:0/1277987794 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5f88004420 0x7f5f88004850 secure :-1 s=THROTTLE_DONE pgs=2147 cs=0 l=1 rx=0x7f5f8c0081d0 tx=0x7f5f8c00f800).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:06.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:06.951+0000 7f5fa0ff9700 1 --2- 172.21.15.42:0/1277987794 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5f88042de0 0x7f5f880452f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:10:06.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:06.951+0000 7f5fa0ff9700 1 -- 172.21.15.42:0/1277987794 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9901+0+0 (secure 0 0 0) 0x7f5f8c04c180 con 0x7f5f88004420 2019-12-05T06:10:06.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:06.951+0000 7f5fa3fff700 1 --2- 172.21.15.42:0/1277987794 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5f88042de0 0x7f5f880452f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:10:06.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:06.955+0000 7f5fa3fff700 1 --2- 172.21.15.42:0/1277987794 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5f88042de0 0x7f5f880452f0 crc :-1 s=READY pgs=903 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:10:06.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:06.959+0000 7f5faa10e700 1 -- 172.21.15.42:0/1277987794 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f7c003200 con 0x7f5f88004420 2019-12-05T06:10:06.976 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:06.963+0000 7f5fa37fe700 1 --2- 172.21.15.42:0/1277987794 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5f88004420 0x7f5f88004850 secure :-1 s=THROTTLE_DONE pgs=2147 cs=0 l=1 rx=0x7f5f8c0081d0 tx=0x7f5f8c00f800).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:06.976 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:06.963+0000 7f5fa0ff9700 1 -- 172.21.15.42:0/1277987794 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f5f8c033220 con 0x7f5f88004420 2019-12-05T06:10:07.142 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:07.135+0000 7f5faa10e700 1 -- 172.21.15.42:0/1277987794 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5f7c003630 con 0x7f5f88004420 2019-12-05T06:10:07.143 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:07.135+0000 7f5fa37fe700 1 --2- 172.21.15.42:0/1277987794 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5f88004420 0x7f5f88004850 secure :-1 s=THROTTLE_DONE pgs=2147 cs=0 l=1 rx=0x7f5f8c0081d0 tx=0x7f5f8c00f800).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:07.143 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:07.135+0000 7f5fa0ff9700 1 -- 172.21.15.42:0/1277987794 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 258 v258) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f5f8c033480 con 0x7f5f88004420 2019-12-05T06:10:07.145 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:10:07.145 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":258,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6977,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/3355679996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3355679996},{"type":"v1","addr":"172.21.15.40:6833","nonce":3355679996}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":258}],"filesystems":[{"mdsmap":{"epoch":257,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:08:14.898258+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":94,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6929},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6929":{"gid":6929,"name":"a","rank":0,"incarnation":254,"state":"up:active","state_seq":19,"addr":"172.21.15.42:6835/2636699468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2636699468},{"type":"v1","addr":"172.21.15.42:6835","nonce":2636699468}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:10:07.146 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 258 2019-12-05T06:10:07.148 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:07.139+0000 7f5faa10e700 1 -- 172.21.15.42:0/1277987794 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5f88042de0 msgr2=0x7f5f880452f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:10:07.148 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:07.139+0000 7f5faa10e700 1 --2- 172.21.15.42:0/1277987794 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5f88042de0 0x7f5f880452f0 crc :-1 s=READY pgs=903 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:10:07.149 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:07.139+0000 7f5faa10e700 1 -- 172.21.15.42:0/1277987794 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5f88004420 msgr2=0x7f5f88004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:10:07.149 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:07.139+0000 7f5faa10e700 1 --2- 172.21.15.42:0/1277987794 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5f88004420 0x7f5f88004850 secure :-1 s=READY pgs=2147 cs=0 l=1 rx=0x7f5f8c0081d0 tx=0x7f5f8c00f800).stop 2019-12-05T06:10:07.149 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:07.139+0000 7f5faa10e700 1 -- 172.21.15.42:0/1277987794 shutdown_connections 2019-12-05T06:10:07.149 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:07.139+0000 7f5faa10e700 1 --2- 172.21.15.42:0/1277987794 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5f88042de0 0x7f5f880452f0 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:10:07.149 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:07.139+0000 7f5faa10e700 1 --2- 172.21.15.42:0/1277987794 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5f88004420 0x7f5f88004850 unknown :-1 s=CLOSED pgs=2147 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:10:07.149 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:07.143+0000 7f5faa10e700 1 -- 172.21.15.42:0/1277987794 shutdown_connections 2019-12-05T06:10:07.149 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:07.143+0000 7f5faa10e700 1 -- 172.21.15.42:0/1277987794 wait complete. 2019-12-05T06:10:07.150 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:07.143+0000 7f5faa10e700 1 -- 172.21.15.42:0/1277987794 >> 172.21.15.42:0/1277987794 conn(0x7f5fa407de90 msgr2=0x7f5fa407e6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:10:07.158 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:10:09.159 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:10:09.266 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:09.255+0000 7fe23a46c700 1 Processor -- start 2019-12-05T06:10:09.267 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:09.255+0000 7fe23a46c700 1 -- start start 2019-12-05T06:10:09.267 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:09.255+0000 7fe23a46c700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe2340a7730 0x7fe23408df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:10:09.267 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:09.255+0000 7fe23a46c700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe23407e710 0x7fe23407eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:10:09.267 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:09.255+0000 7fe23a46c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe23409f2d0 0x7fe23409f700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:10:09.268 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:09.255+0000 7fe23a46c700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2340a0af0 con 0x7fe23407e710 2019-12-05T06:10:09.269 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:09.255+0000 7fe23a46c700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe23408e5a0 con 0x7fe23409f2d0 2019-12-05T06:10:09.269 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:09.255+0000 7fe23a46c700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe23408e840 con 0x7fe2340a7730 2019-12-05T06:10:09.269 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:09.255+0000 7fe23920a700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe23409f2d0 0x7fe23409f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:10:09.270 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:09.255+0000 7fe23920a700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe23409f2d0 0x7fe23409f700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:49588/0 (socket says 172.21.15.42:49588) 2019-12-05T06:10:09.271 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:09.255+0000 7fe23920a700 1 -- 172.21.15.42:0/2459426061 learned_addr learned my addr 172.21.15.42:0/2459426061 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:10:09.271 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:09.255+0000 7fe233fff700 1 --2- 172.21.15.42:0/2459426061 >> v2:172.21.15.40:3300/0 conn(0x7fe23407e710 0x7fe23407eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:10:09.271 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:09.255+0000 7fe238a09700 1 --2- 172.21.15.42:0/2459426061 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe2340a7730 0x7fe23408df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:10:09.272 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:09.255+0000 7fe23920a700 1 -- 172.21.15.42:0/2459426061 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe2340a7730 msgr2=0x7fe23408df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:10:09.272 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:09.255+0000 7fe23920a700 1 --2- 172.21.15.42:0/2459426061 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe2340a7730 0x7fe23408df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:10:09.273 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:09.255+0000 7fe23920a700 1 -- 172.21.15.42:0/2459426061 >> v2:172.21.15.40:3300/0 conn(0x7fe23407e710 msgr2=0x7fe23407eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:10:09.273 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:09.255+0000 7fe23920a700 1 --2- 172.21.15.42:0/2459426061 >> v2:172.21.15.40:3300/0 conn(0x7fe23407e710 0x7fe23407eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:10:09.273 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:09.259+0000 7fe23920a700 1 -- 172.21.15.42:0/2459426061 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe23407f090 con 0x7fe23409f2d0 2019-12-05T06:10:09.273 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:09.259+0000 7fe23920a700 1 --2- 172.21.15.42:0/2459426061 >> v2:172.21.15.42:3300/0 conn(0x7fe23409f2d0 0x7fe23409f700 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fe23408e760 tx=0x7fe22c00fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:09.273 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:09.259+0000 7fe23920a700 1 --2- 172.21.15.42:0/2459426061 >> v2:172.21.15.42:3300/0 conn(0x7fe23409f2d0 0x7fe23409f700 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fe23408e760 tx=0x7fe22c00fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:09.274 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:09.259+0000 7fe23920a700 1 --2- 172.21.15.42:0/2459426061 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe23409f2d0 0x7fe23409f700 secure :-1 s=READY pgs=2148 cs=0 l=1 rx=0x7fe23408e760 tx=0x7fe22c00fdb0).ready entity=mon.1 client_cookie=bcb7613c63c6dfc1 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:10:09.274 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:09.259+0000 7fe23920a700 1 --2- 172.21.15.42:0/2459426061 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe23409f2d0 0x7fe23409f700 secure :-1 s=THROTTLE_DONE pgs=2148 cs=0 l=1 rx=0x7fe23408e760 tx=0x7fe22c00fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:09.274 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:09.259+0000 7fe2337fe700 1 -- 172.21.15.42:0/2459426061 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe22c008c90 con 0x7fe23409f2d0 2019-12-05T06:10:09.274 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:09.259+0000 7fe23920a700 1 --2- 172.21.15.42:0/2459426061 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe23409f2d0 0x7fe23409f700 secure :-1 s=THROTTLE_DONE pgs=2148 cs=0 l=1 rx=0x7fe23408e760 tx=0x7fe22c00fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:09.275 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:09.259+0000 7fe23920a700 1 --2- 172.21.15.42:0/2459426061 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe23409f2d0 0x7fe23409f700 secure :-1 s=THROTTLE_DONE pgs=2148 cs=0 l=1 rx=0x7fe23408e760 tx=0x7fe22c00fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:09.275 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:09.259+0000 7fe2337fe700 1 -- 172.21.15.42:0/2459426061 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe22c0088b0 con 0x7fe23409f2d0 2019-12-05T06:10:09.275 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:09.259+0000 7fe2337fe700 1 -- 172.21.15.42:0/2459426061 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe22c011b50 con 0x7fe23409f2d0 2019-12-05T06:10:09.275 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:09.259+0000 7fe23a46c700 1 -- 172.21.15.42:0/2459426061 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe23409f2d0 msgr2=0x7fe23409f700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:10:09.276 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:09.259+0000 7fe23a46c700 1 --2- 172.21.15.42:0/2459426061 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe23409f2d0 0x7fe23409f700 secure :-1 s=READY pgs=2148 cs=0 l=1 rx=0x7fe23408e760 tx=0x7fe22c00fdb0).stop 2019-12-05T06:10:09.277 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:09.259+0000 7fe23a46c700 1 -- 172.21.15.42:0/2459426061 shutdown_connections 2019-12-05T06:10:09.277 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:09.259+0000 7fe23a46c700 1 --2- 172.21.15.42:0/2459426061 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe23409f2d0 0x7fe23409f700 unknown :-1 s=CLOSED pgs=2148 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:10:09.277 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:09.259+0000 7fe23a46c700 1 --2- 172.21.15.42:0/2459426061 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe2340a7730 0x7fe23408df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:10:09.277 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:09.259+0000 7fe23a46c700 1 --2- 172.21.15.42:0/2459426061 >> v2:172.21.15.40:3300/0 conn(0x7fe23407e710 0x7fe23407eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:10:09.277 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:09.259+0000 7fe23a46c700 1 -- 172.21.15.42:0/2459426061 shutdown_connections 2019-12-05T06:10:09.278 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:09.259+0000 7fe23a46c700 1 -- 172.21.15.42:0/2459426061 wait complete. 2019-12-05T06:10:09.278 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:09.259+0000 7fe23a46c700 1 -- 172.21.15.42:0/2459426061 >> 172.21.15.42:0/2459426061 conn(0x7fe2340aabd0 msgr2=0x7fe2340a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:10:09.287 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:09.263+0000 7fe23a46c700 1 Processor -- start 2019-12-05T06:10:09.287 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:09.263+0000 7fe23a46c700 1 -- start start 2019-12-05T06:10:09.287 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:09.263+0000 7fe23a46c700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe234059f40 0x7fe23411f280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:10:09.288 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:09.263+0000 7fe23a46c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe23412d7a0 0x7fe23411b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:10:09.288 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:09.263+0000 7fe23a46c700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe23407d730 0x7fe23407db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:10:09.288 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:09.263+0000 7fe23a46c700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe23412cd20 con 0x7fe234059f40 2019-12-05T06:10:09.288 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:09.263+0000 7fe23a46c700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe23412cfc0 con 0x7fe23412d7a0 2019-12-05T06:10:09.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:09.263+0000 7fe23a46c700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe23412d260 con 0x7fe23407d730 2019-12-05T06:10:09.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:09.263+0000 7fe233fff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe23412d7a0 0x7fe23411b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:10:09.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:09.263+0000 7fe233fff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe23412d7a0 0x7fe23411b7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:49590/0 (socket says 172.21.15.42:49590) 2019-12-05T06:10:09.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:09.263+0000 7fe233fff700 1 -- 172.21.15.42:0/3561154811 learned_addr learned my addr 172.21.15.42:0/3561154811 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:10:09.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:09.263+0000 7fe23920a700 1 --2- 172.21.15.42:0/3561154811 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe23407d730 0x7fe23407db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:10:09.291 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:09.263+0000 7fe238a09700 1 --2- 172.21.15.42:0/3561154811 >> v2:172.21.15.40:3300/0 conn(0x7fe234059f40 0x7fe23411f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:10:09.292 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:09.263+0000 7fe233fff700 1 -- 172.21.15.42:0/3561154811 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe23407d730 msgr2=0x7fe23407db60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:10:09.292 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:09.263+0000 7fe233fff700 1 --2- 172.21.15.42:0/3561154811 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe23407d730 0x7fe23407db60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:10:09.292 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:09.263+0000 7fe233fff700 1 -- 172.21.15.42:0/3561154811 >> v2:172.21.15.40:3300/0 conn(0x7fe234059f40 msgr2=0x7fe23411f280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:10:09.292 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:09.263+0000 7fe233fff700 1 --2- 172.21.15.42:0/3561154811 >> v2:172.21.15.40:3300/0 conn(0x7fe234059f40 0x7fe23411f280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:10:09.293 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:09.263+0000 7fe233fff700 1 -- 172.21.15.42:0/3561154811 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe23412dbd0 con 0x7fe23412d7a0 2019-12-05T06:10:09.294 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:09.263+0000 7fe23920a700 1 --2- 172.21.15.42:0/3561154811 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe23407d730 0x7fe23407db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:10:09.294 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:09.263+0000 7fe233fff700 1 --2- 172.21.15.42:0/3561154811 >> v2:172.21.15.42:3300/0 conn(0x7fe23412d7a0 0x7fe23411b7b0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fe23412d180 tx=0x7fe224010f60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:09.294 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:09.263+0000 7fe233fff700 1 --2- 172.21.15.42:0/3561154811 >> v2:172.21.15.42:3300/0 conn(0x7fe23412d7a0 0x7fe23411b7b0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fe23412d180 tx=0x7fe224010f60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:09.294 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:09.263+0000 7fe233fff700 1 --2- 172.21.15.42:0/3561154811 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe23412d7a0 0x7fe23411b7b0 secure :-1 s=READY pgs=2149 cs=0 l=1 rx=0x7fe23412d180 tx=0x7fe224010f60).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:10:09.295 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:09.263+0000 7fe233fff700 1 --2- 172.21.15.42:0/3561154811 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe23412d7a0 0x7fe23411b7b0 secure :-1 s=THROTTLE_DONE pgs=2149 cs=0 l=1 rx=0x7fe23412d180 tx=0x7fe224010f60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:09.295 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:09.263+0000 7fe233fff700 1 --2- 172.21.15.42:0/3561154811 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe23412d7a0 0x7fe23411b7b0 secure :-1 s=THROTTLE_DONE pgs=2149 cs=0 l=1 rx=0x7fe23412d180 tx=0x7fe224010f60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:09.295 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:09.263+0000 7fe233fff700 1 --2- 172.21.15.42:0/3561154811 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe23412d7a0 0x7fe23411b7b0 secure :-1 s=THROTTLE_DONE pgs=2149 cs=0 l=1 rx=0x7fe23412d180 tx=0x7fe224010f60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:09.295 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:09.263+0000 7fe2317fa700 1 -- 172.21.15.42:0/3561154811 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe22400e860 con 0x7fe23412d7a0 2019-12-05T06:10:09.296 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:09.263+0000 7fe2317fa700 1 -- 172.21.15.42:0/3561154811 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe224010160 con 0x7fe23412d7a0 2019-12-05T06:10:09.296 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:09.263+0000 7fe23a46c700 1 -- 172.21.15.42:0/3561154811 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe23407e0b0 con 0x7fe23412d7a0 2019-12-05T06:10:09.296 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:09.263+0000 7fe2317fa700 1 -- 172.21.15.42:0/3561154811 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe22400c3f0 con 0x7fe23412d7a0 2019-12-05T06:10:09.296 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:09.263+0000 7fe23a46c700 1 -- 172.21.15.42:0/3561154811 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe23411bd00 con 0x7fe23412d7a0 2019-12-05T06:10:09.296 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:09.267+0000 7fe233fff700 1 --2- 172.21.15.42:0/3561154811 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe23412d7a0 0x7fe23411b7b0 secure :-1 s=THROTTLE_DONE pgs=2149 cs=0 l=1 rx=0x7fe23412d180 tx=0x7fe224010f60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:09.297 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:09.267+0000 7fe2317fa700 1 -- 172.21.15.42:0/3561154811 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fe2240058c0 con 0x7fe23412d7a0 2019-12-05T06:10:09.297 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:09.267+0000 7fe233fff700 1 --2- 172.21.15.42:0/3561154811 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe23412d7a0 0x7fe23411b7b0 secure :-1 s=THROTTLE_DONE pgs=2149 cs=0 l=1 rx=0x7fe23412d180 tx=0x7fe224010f60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:09.297 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:09.267+0000 7fe2317fa700 1 --2- 172.21.15.42:0/3561154811 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe20c03ae60 0x7fe20c03d370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:10:09.298 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:09.267+0000 7fe2317fa700 1 -- 172.21.15.42:0/3561154811 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9901+0+0 (secure 0 0 0) 0x7fe22404afe0 con 0x7fe23412d7a0 2019-12-05T06:10:09.298 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:09.267+0000 7fe238a09700 1 --2- 172.21.15.42:0/3561154811 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe20c03ae60 0x7fe20c03d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:10:09.298 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:09.267+0000 7fe238a09700 1 --2- 172.21.15.42:0/3561154811 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe20c03ae60 0x7fe20c03d370 crc :-1 s=READY pgs=904 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:10:09.298 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:09.271+0000 7fe23a46c700 1 -- 172.21.15.42:0/3561154811 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe23405d100 con 0x7fe23412d7a0 2019-12-05T06:10:09.299 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:09.279+0000 7fe233fff700 1 --2- 172.21.15.42:0/3561154811 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe23412d7a0 0x7fe23411b7b0 secure :-1 s=THROTTLE_DONE pgs=2149 cs=0 l=1 rx=0x7fe23412d180 tx=0x7fe224010f60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:09.299 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:09.279+0000 7fe2317fa700 1 -- 172.21.15.42:0/3561154811 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fe224032000 con 0x7fe23412d7a0 2019-12-05T06:10:09.445 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:09.439+0000 7fe23a46c700 1 -- 172.21.15.42:0/3561154811 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe23405b8b0 con 0x7fe23412d7a0 2019-12-05T06:10:09.446 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:09.439+0000 7fe233fff700 1 --2- 172.21.15.42:0/3561154811 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe23412d7a0 0x7fe23411b7b0 secure :-1 s=THROTTLE_DONE pgs=2149 cs=0 l=1 rx=0x7fe23412d180 tx=0x7fe224010f60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:09.446 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:09.439+0000 7fe2317fa700 1 -- 172.21.15.42:0/3561154811 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 258 v258) v1 ==== 95+0+2112 (secure 0 0 0) 0x7fe22402f0d0 con 0x7fe23412d7a0 2019-12-05T06:10:09.446 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 258 2019-12-05T06:10:09.446 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:10:09.446 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":258,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6977,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/3355679996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3355679996},{"type":"v1","addr":"172.21.15.40:6833","nonce":3355679996}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":258}],"filesystems":[{"mdsmap":{"epoch":257,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:08:14.898258+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":94,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6929},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6929":{"gid":6929,"name":"a","rank":0,"incarnation":254,"state":"up:active","state_seq":19,"addr":"172.21.15.42:6835/2636699468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2636699468},{"type":"v1","addr":"172.21.15.42:6835","nonce":2636699468}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:10:09.449 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:09.443+0000 7fe23a46c700 1 -- 172.21.15.42:0/3561154811 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe20c03ae60 msgr2=0x7fe20c03d370 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:10:09.449 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:09.443+0000 7fe23a46c700 1 --2- 172.21.15.42:0/3561154811 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe20c03ae60 0x7fe20c03d370 crc :-1 s=READY pgs=904 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:10:09.449 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:09.443+0000 7fe23a46c700 1 -- 172.21.15.42:0/3561154811 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe23412d7a0 msgr2=0x7fe23411b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:10:09.449 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:09.443+0000 7fe23a46c700 1 --2- 172.21.15.42:0/3561154811 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe23412d7a0 0x7fe23411b7b0 secure :-1 s=READY pgs=2149 cs=0 l=1 rx=0x7fe23412d180 tx=0x7fe224010f60).stop 2019-12-05T06:10:09.449 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:09.443+0000 7fe23a46c700 1 -- 172.21.15.42:0/3561154811 shutdown_connections 2019-12-05T06:10:09.449 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:09.443+0000 7fe23a46c700 1 --2- 172.21.15.42:0/3561154811 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe20c03ae60 0x7fe20c03d370 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:10:09.450 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:09.443+0000 7fe23a46c700 1 --2- 172.21.15.42:0/3561154811 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe23407d730 0x7fe23407db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:10:09.450 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:09.443+0000 7fe23a46c700 1 --2- 172.21.15.42:0/3561154811 >> v2:172.21.15.40:3300/0 conn(0x7fe234059f40 0x7fe23411f280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:10:09.450 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:09.443+0000 7fe23a46c700 1 --2- 172.21.15.42:0/3561154811 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe23412d7a0 0x7fe23411b7b0 unknown :-1 s=CLOSED pgs=2149 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:10:09.450 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:09.443+0000 7fe23a46c700 1 -- 172.21.15.42:0/3561154811 shutdown_connections 2019-12-05T06:10:09.451 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:09.443+0000 7fe23a46c700 1 -- 172.21.15.42:0/3561154811 wait complete. 2019-12-05T06:10:09.451 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:09.443+0000 7fe23a46c700 1 -- 172.21.15.42:0/3561154811 >> 172.21.15.42:0/3561154811 conn(0x7fe23407e760 msgr2=0x7fe23409f370 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:10:09.459 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:10:11.461 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:10:11.566 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.555+0000 7f7085056700 1 Processor -- start 2019-12-05T06:10:11.566 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.555+0000 7f7085056700 1 -- start start 2019-12-05T06:10:11.567 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.555+0000 7f7085056700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f70800a7740 0x7f708008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:10:11.567 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.555+0000 7f7085056700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f708009fae0 0x7f708007eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:10:11.567 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.555+0000 7f7085056700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70800a0710 con 0x7f708007e720 2019-12-05T06:10:11.567 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.555+0000 7f7085056700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70800a0b00 con 0x7f70800a7740 2019-12-05T06:10:11.567 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.555+0000 7f7085056700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f708008e5b0 con 0x7f708009fae0 2019-12-05T06:10:11.567 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.555+0000 7f7076d9d700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f708007e720 0x7f708009f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:49638/0 (socket says 172.21.15.42:49638) 2019-12-05T06:10:11.568 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.555+0000 7f707f59e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f70800a7740 0x7f708008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:10:11.568 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.555+0000 7f7076d9d700 1 -- 172.21.15.42:0/432801083 learned_addr learned my addr 172.21.15.42:0/432801083 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:10:11.568 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.555+0000 7f7084855700 1 --2- 172.21.15.42:0/432801083 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f708009fae0 0x7f708007eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:10:11.568 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.555+0000 7f707ed9d700 1 -- 172.21.15.42:0/432801083 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 163419184 0 0) 0x7f70800a0710 con 0x7f708007e720 2019-12-05T06:10:11.568 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.555+0000 7f707ed9d700 1 -- 172.21.15.42:0/432801083 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f706c0030b0 con 0x7f708007e720 2019-12-05T06:10:11.568 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.555+0000 7f7084855700 1 -- 172.21.15.42:0/432801083 >> v2:172.21.15.42:3300/0 conn(0x7f70800a7740 msgr2=0x7f708008df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:10:11.568 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.555+0000 7f7084855700 1 --2- 172.21.15.42:0/432801083 >> v2:172.21.15.42:3300/0 conn(0x7f70800a7740 0x7f708008df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:10:11.569 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.555+0000 7f7084855700 1 -- 172.21.15.42:0/432801083 >> v1:172.21.15.42:6789/0 conn(0x7f708007e720 legacy=0x7f708009f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:10:11.569 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.555+0000 7f7084855700 1 -- 172.21.15.42:0/432801083 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f708007f0a0 con 0x7f708009fae0 2019-12-05T06:10:11.569 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.555+0000 7f707f59e700 1 --2- 172.21.15.42:0/432801083 >> v2:172.21.15.42:3300/0 conn(0x7f70800a7740 0x7f708008df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:10:11.569 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.555+0000 7f7084855700 1 --2- 172.21.15.42:0/432801083 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f708009fae0 0x7f708007eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f7078008b10 tx=0x7f707800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:11.569 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.559+0000 7f7084855700 1 --2- 172.21.15.42:0/432801083 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f708009fae0 0x7f708007eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f7078008b10 tx=0x7f707800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:11.569 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.559+0000 7f7084855700 1 --2- 172.21.15.42:0/432801083 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f708009fae0 0x7f708007eb50 secure :-1 s=READY pgs=473 cs=0 l=1 rx=0x7f7078008b10 tx=0x7f707800fff0).ready entity=mon.2 client_cookie=7db1917732c5fc38 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:10:11.570 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.559+0000 7f7084855700 1 --2- 172.21.15.42:0/432801083 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f708009fae0 0x7f708007eb50 secure :-1 s=THROTTLE_DONE pgs=473 cs=0 l=1 rx=0x7f7078008b10 tx=0x7f707800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:11.570 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.559+0000 7f707ed9d700 1 -- 172.21.15.42:0/432801083 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7078010900 con 0x7f708009fae0 2019-12-05T06:10:11.570 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.559+0000 7f7084855700 1 --2- 172.21.15.42:0/432801083 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f708009fae0 0x7f708007eb50 secure :-1 s=THROTTLE_DONE pgs=473 cs=0 l=1 rx=0x7f7078008b10 tx=0x7f707800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:11.570 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.559+0000 7f707ed9d700 1 -- 172.21.15.42:0/432801083 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7078008b50 con 0x7f708009fae0 2019-12-05T06:10:11.570 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.559+0000 7f7084855700 1 --2- 172.21.15.42:0/432801083 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f708009fae0 0x7f708007eb50 secure :-1 s=THROTTLE_DONE pgs=473 cs=0 l=1 rx=0x7f7078008b10 tx=0x7f707800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:11.570 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.559+0000 7f707ed9d700 1 -- 172.21.15.42:0/432801083 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f70780110c0 con 0x7f708009fae0 2019-12-05T06:10:11.570 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.559+0000 7f7085056700 1 -- 172.21.15.42:0/432801083 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f708009fae0 msgr2=0x7f708007eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:10:11.571 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.559+0000 7f7085056700 1 --2- 172.21.15.42:0/432801083 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f708009fae0 0x7f708007eb50 secure :-1 s=READY pgs=473 cs=0 l=1 rx=0x7f7078008b10 tx=0x7f707800fff0).stop 2019-12-05T06:10:11.571 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.559+0000 7f7085056700 1 -- 172.21.15.42:0/432801083 shutdown_connections 2019-12-05T06:10:11.571 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.559+0000 7f7085056700 1 --2- 172.21.15.42:0/432801083 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f708009fae0 0x7f708007eb50 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:10:11.571 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.559+0000 7f7085056700 1 --2- 172.21.15.42:0/432801083 >> v2:172.21.15.42:3300/0 conn(0x7f70800a7740 0x7f708008df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:10:11.571 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.559+0000 7f7085056700 1 -- 172.21.15.42:0/432801083 shutdown_connections 2019-12-05T06:10:11.571 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.559+0000 7f7085056700 1 -- 172.21.15.42:0/432801083 wait complete. 2019-12-05T06:10:11.571 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.559+0000 7f7085056700 1 -- 172.21.15.42:0/432801083 >> 172.21.15.42:0/432801083 conn(0x7f70800aabe0 msgr2=0x7f70800a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:10:11.576 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.559+0000 7f7085056700 1 Processor -- start 2019-12-05T06:10:11.576 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.559+0000 7f7085056700 1 -- start start 2019-12-05T06:10:11.576 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.563+0000 7f7085056700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7080118390 0x7f708007bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:10:11.577 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.563+0000 7f7085056700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f708007d890 con 0x7f708013b7b0 2019-12-05T06:10:11.577 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.563+0000 7f7085056700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f708013c3e0 con 0x7f708007d460 2019-12-05T06:10:11.577 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.563+0000 7f7085056700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7080119cd0 con 0x7f7080118390 2019-12-05T06:10:11.577 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.563+0000 7f7076d9d700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f708007d460 0x7f708011b530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:49644/0 (socket says 172.21.15.42:49644) 2019-12-05T06:10:11.577 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.563+0000 7f7084855700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f708013b7b0 0x7f708013bbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:59086/0 (socket says 172.21.15.42:59086) 2019-12-05T06:10:11.577 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.563+0000 7f707f59e700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7080118390 0x7f708007bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:10:11.578 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.563+0000 7f7076d9d700 1 -- 172.21.15.42:0/2599574262 learned_addr learned my addr 172.21.15.42:0/2599574262 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:10:11.578 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.563+0000 7f707cd99700 1 -- 172.21.15.42:0/2599574262 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1489159588 0 0) 0x7f708007d890 con 0x7f708013b7b0 2019-12-05T06:10:11.578 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.563+0000 7f707cd99700 1 -- 172.21.15.42:0/2599574262 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70640030b0 con 0x7f708013b7b0 2019-12-05T06:10:11.578 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.563+0000 7f707cd99700 1 -- 172.21.15.42:0/2599574262 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3760638614 0 0) 0x7f708013c3e0 con 0x7f708007d460 2019-12-05T06:10:11.578 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.563+0000 7f707cd99700 1 -- 172.21.15.42:0/2599574262 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f708007d890 con 0x7f708007d460 2019-12-05T06:10:11.578 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.563+0000 7f707f59e700 1 -- 172.21.15.42:0/2599574262 >> v1:172.21.15.42:6789/0 conn(0x7f708007d460 legacy=0x7f708011b530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:10:11.578 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.563+0000 7f707f59e700 1 -- 172.21.15.42:0/2599574262 >> v1:172.21.15.40:6789/0 conn(0x7f708013b7b0 legacy=0x7f708013bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:10:11.579 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.563+0000 7f707f59e700 1 -- 172.21.15.42:0/2599574262 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7080119830 con 0x7f7080118390 2019-12-05T06:10:11.579 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.563+0000 7f707f59e700 1 --2- 172.21.15.42:0/2599574262 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7080118390 0x7f708007bdc0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f7070002f30 tx=0x7f7070010bd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:11.579 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.563+0000 7f707f59e700 1 --2- 172.21.15.42:0/2599574262 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7080118390 0x7f708007bdc0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f7070002f30 tx=0x7f7070010bd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:11.579 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.563+0000 7f707f59e700 1 --2- 172.21.15.42:0/2599574262 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7080118390 0x7f708007bdc0 secure :-1 s=READY pgs=474 cs=0 l=1 rx=0x7f7070002f30 tx=0x7f7070010bd0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:10:11.579 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.563+0000 7f707f59e700 1 --2- 172.21.15.42:0/2599574262 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7080118390 0x7f708007bdc0 secure :-1 s=THROTTLE_DONE pgs=474 cs=0 l=1 rx=0x7f7070002f30 tx=0x7f7070010bd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:11.579 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.563+0000 7f707cd99700 1 -- 172.21.15.42:0/2599574262 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f707000d0c0 con 0x7f7080118390 2019-12-05T06:10:11.580 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.563+0000 7f707f59e700 1 --2- 172.21.15.42:0/2599574262 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7080118390 0x7f708007bdc0 secure :-1 s=THROTTLE_DONE pgs=474 cs=0 l=1 rx=0x7f7070002f30 tx=0x7f7070010bd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:11.580 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.563+0000 7f7085056700 1 -- 172.21.15.42:0/2599574262 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7080141b70 con 0x7f7080118390 2019-12-05T06:10:11.580 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.563+0000 7f707cd99700 1 -- 172.21.15.42:0/2599574262 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7070007150 con 0x7f7080118390 2019-12-05T06:10:11.580 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.563+0000 7f707f59e700 1 --2- 172.21.15.42:0/2599574262 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7080118390 0x7f708007bdc0 secure :-1 s=THROTTLE_DONE pgs=474 cs=0 l=1 rx=0x7f7070002f30 tx=0x7f7070010bd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:11.580 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.563+0000 7f707cd99700 1 -- 172.21.15.42:0/2599574262 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f70700129c0 con 0x7f7080118390 2019-12-05T06:10:11.580 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.563+0000 7f7085056700 1 -- 172.21.15.42:0/2599574262 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70801444a0 con 0x7f7080118390 2019-12-05T06:10:11.581 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.567+0000 7f707f59e700 1 --2- 172.21.15.42:0/2599574262 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7080118390 0x7f708007bdc0 secure :-1 s=THROTTLE_DONE pgs=474 cs=0 l=1 rx=0x7f7070002f30 tx=0x7f7070010bd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:11.581 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.567+0000 7f707cd99700 1 -- 172.21.15.42:0/2599574262 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f7070012ce0 con 0x7f7080118390 2019-12-05T06:10:11.581 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.567+0000 7f707f59e700 1 --2- 172.21.15.42:0/2599574262 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7080118390 0x7f708007bdc0 secure :-1 s=THROTTLE_DONE pgs=474 cs=0 l=1 rx=0x7f7070002f30 tx=0x7f7070010bd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:11.581 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.567+0000 7f707cd99700 1 --2- 172.21.15.42:0/2599574262 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f706403b450 0x7f706403d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:10:11.581 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.567+0000 7f707cd99700 1 -- 172.21.15.42:0/2599574262 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9987+0+0 (secure 0 0 0) 0x7f707004b210 con 0x7f7080118390 2019-12-05T06:10:11.581 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.567+0000 7f7076d9d700 1 --2- 172.21.15.42:0/2599574262 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f706403b450 0x7f706403d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:10:11.582 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.567+0000 7f7076d9d700 1 --2- 172.21.15.42:0/2599574262 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f706403b450 0x7f706403d960 crc :-1 s=READY pgs=905 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:10:11.582 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.571+0000 7f7085056700 1 -- 172.21.15.42:0/2599574262 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f708005d110 con 0x7f7080118390 2019-12-05T06:10:11.585 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.579+0000 7f707f59e700 1 --2- 172.21.15.42:0/2599574262 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7080118390 0x7f708007bdc0 secure :-1 s=THROTTLE_DONE pgs=474 cs=0 l=1 rx=0x7f7070002f30 tx=0x7f7070010bd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:11.586 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.579+0000 7f707cd99700 1 -- 172.21.15.42:0/2599574262 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f70700324d0 con 0x7f7080118390 2019-12-05T06:10:11.753 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.747+0000 7f7085056700 1 -- 172.21.15.42:0/2599574262 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f708005b8c0 con 0x7f7080118390 2019-12-05T06:10:11.754 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.747+0000 7f707f59e700 1 --2- 172.21.15.42:0/2599574262 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7080118390 0x7f708007bdc0 secure :-1 s=THROTTLE_DONE pgs=474 cs=0 l=1 rx=0x7f7070002f30 tx=0x7f7070010bd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:11.755 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.747+0000 7f707cd99700 1 -- 172.21.15.42:0/2599574262 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 259 v259) v1 ==== 95+0+1774 (secure 0 0 0) 0x7f7070033080 con 0x7f7080118390 2019-12-05T06:10:11.759 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 259 2019-12-05T06:10:11.760 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:10:11.760 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":259,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":259,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:10:09.529943+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":95,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6977},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6977":{"gid":6977,"name":"b","rank":0,"incarnation":259,"state":"up:replay","state_seq":2,"addr":"172.21.15.40:6833/3355679996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3355679996},{"type":"v1","addr":"172.21.15.40:6833","nonce":3355679996}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:10:11.761 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.751+0000 7f7085056700 1 -- 172.21.15.42:0/2599574262 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f706403b450 msgr2=0x7f706403d960 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:10:11.761 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.751+0000 7f7085056700 1 --2- 172.21.15.42:0/2599574262 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f706403b450 0x7f706403d960 crc :-1 s=READY pgs=905 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:10:11.761 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.751+0000 7f7085056700 1 -- 172.21.15.42:0/2599574262 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7080118390 msgr2=0x7f708007bdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:10:11.761 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.751+0000 7f7085056700 1 --2- 172.21.15.42:0/2599574262 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7080118390 0x7f708007bdc0 secure :-1 s=READY pgs=474 cs=0 l=1 rx=0x7f7070002f30 tx=0x7f7070010bd0).stop 2019-12-05T06:10:11.764 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.751+0000 7f7085056700 1 -- 172.21.15.42:0/2599574262 shutdown_connections 2019-12-05T06:10:11.764 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.751+0000 7f7085056700 1 --2- 172.21.15.42:0/2599574262 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f706403b450 0x7f706403d960 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:10:11.764 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.751+0000 7f7085056700 1 --2- 172.21.15.42:0/2599574262 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7080118390 0x7f708007bdc0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:10:11.764 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.751+0000 7f7085056700 1 -- 172.21.15.42:0/2599574262 shutdown_connections 2019-12-05T06:10:11.764 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.751+0000 7f7085056700 1 -- 172.21.15.42:0/2599574262 wait complete. 2019-12-05T06:10:11.765 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.751+0000 7f7085056700 1 -- 172.21.15.42:0/2599574262 >> 172.21.15.42:0/2599574262 conn(0x7f708007ec60 msgr2=0x7f7080059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:10:11.770 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a down, removed from mdsmap 2019-12-05T06:10:11.771 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2019-12-05T06:10:11.771 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:10:11.879 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.851+0000 7f808b086700 1 Processor -- start 2019-12-05T06:10:11.879 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.851+0000 7f808b086700 1 -- start start 2019-12-05T06:10:11.880 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.855+0000 7f808b086700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f80840a7740 0x7f808408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:10:11.880 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.855+0000 7f808b086700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f808409fae0 0x7f808407eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:10:11.880 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.855+0000 7f808b086700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80840a0710 con 0x7f808407e720 2019-12-05T06:10:11.880 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.855+0000 7f808b086700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80840a0b00 con 0x7f80840a7740 2019-12-05T06:10:11.880 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.855+0000 7f808b086700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f808408e5b0 con 0x7f808409fae0 2019-12-05T06:10:11.880 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.855+0000 7f8089623700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f80840a7740 0x7f808408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:10:11.881 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.855+0000 7f8089623700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f80840a7740 0x7f808408df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:49612/0 (socket says 172.21.15.42:49612) 2019-12-05T06:10:11.881 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.855+0000 7f8089623700 1 -- 172.21.15.42:0/1162195769 learned_addr learned my addr 172.21.15.42:0/1162195769 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:10:11.881 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.855+0000 7f8089e24700 1 --2- 172.21.15.42:0/1162195769 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f808409fae0 0x7f808407eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:10:11.881 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.855+0000 7f8089623700 1 -- 172.21.15.42:0/1162195769 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f808409fae0 msgr2=0x7f808407eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:10:11.881 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.855+0000 7f8089623700 1 --2- 172.21.15.42:0/1162195769 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f808409fae0 0x7f808407eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:10:11.881 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.855+0000 7f8089623700 1 -- 172.21.15.42:0/1162195769 >> v1:172.21.15.42:6789/0 conn(0x7f808407e720 legacy=0x7f808409f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:10:11.882 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.855+0000 7f8089623700 1 -- 172.21.15.42:0/1162195769 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f808407f0a0 con 0x7f80840a7740 2019-12-05T06:10:11.882 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.855+0000 7f8089623700 1 --2- 172.21.15.42:0/1162195769 >> v2:172.21.15.42:3300/0 conn(0x7f80840a7740 0x7f808408df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f8074008790 tx=0x7f807400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:11.882 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.855+0000 7f8089623700 1 --2- 172.21.15.42:0/1162195769 >> v2:172.21.15.42:3300/0 conn(0x7f80840a7740 0x7f808408df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f8074008790 tx=0x7f807400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:11.882 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.855+0000 7f8089623700 1 --2- 172.21.15.42:0/1162195769 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f80840a7740 0x7f808408df80 secure :-1 s=READY pgs=2153 cs=0 l=1 rx=0x7f8074008790 tx=0x7f807400fd30).ready entity=mon.1 client_cookie=41030b7059bce7a8 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:10:11.882 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.855+0000 7f8089623700 1 --2- 172.21.15.42:0/1162195769 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f80840a7740 0x7f808408df80 secure :-1 s=THROTTLE_DONE pgs=2153 cs=0 l=1 rx=0x7f8074008790 tx=0x7f807400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:11.883 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.855+0000 7f8089623700 1 --2- 172.21.15.42:0/1162195769 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f80840a7740 0x7f808408df80 secure :-1 s=THROTTLE_DONE pgs=2153 cs=0 l=1 rx=0x7f8074008790 tx=0x7f807400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:11.883 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.855+0000 7f8089623700 1 --2- 172.21.15.42:0/1162195769 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f80840a7740 0x7f808408df80 secure :-1 s=THROTTLE_DONE pgs=2153 cs=0 l=1 rx=0x7f8074008790 tx=0x7f807400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:11.883 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.855+0000 7f807bfff700 1 -- 172.21.15.42:0/1162195769 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8074008c90 con 0x7f80840a7740 2019-12-05T06:10:11.883 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.855+0000 7f807bfff700 1 -- 172.21.15.42:0/1162195769 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80740088b0 con 0x7f80840a7740 2019-12-05T06:10:11.883 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.855+0000 7f807bfff700 1 -- 172.21.15.42:0/1162195769 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8074011b10 con 0x7f80840a7740 2019-12-05T06:10:11.883 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.855+0000 7f808b086700 1 -- 172.21.15.42:0/1162195769 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f80840a7740 msgr2=0x7f808408df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:10:11.884 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.855+0000 7f808b086700 1 --2- 172.21.15.42:0/1162195769 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f80840a7740 0x7f808408df80 secure :-1 s=READY pgs=2153 cs=0 l=1 rx=0x7f8074008790 tx=0x7f807400fd30).stop 2019-12-05T06:10:11.884 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.855+0000 7f808b086700 1 -- 172.21.15.42:0/1162195769 shutdown_connections 2019-12-05T06:10:11.884 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.855+0000 7f808b086700 1 --2- 172.21.15.42:0/1162195769 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f808409fae0 0x7f808407eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:10:11.884 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.855+0000 7f808b086700 1 --2- 172.21.15.42:0/1162195769 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f80840a7740 0x7f808408df80 unknown :-1 s=CLOSED pgs=2153 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:10:11.884 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.855+0000 7f808b086700 1 -- 172.21.15.42:0/1162195769 shutdown_connections 2019-12-05T06:10:11.884 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.855+0000 7f808b086700 1 -- 172.21.15.42:0/1162195769 wait complete. 2019-12-05T06:10:11.885 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.855+0000 7f808b086700 1 -- 172.21.15.42:0/1162195769 >> 172.21.15.42:0/1162195769 conn(0x7f80840aabe0 msgr2=0x7f80840a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:10:11.885 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.859+0000 7f808b086700 1 Processor -- start 2019-12-05T06:10:11.885 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.859+0000 7f808b086700 1 -- start start 2019-12-05T06:10:11.885 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.859+0000 7f808b086700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f80840a41e0 0x7f8084128100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:10:11.885 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.859+0000 7f808b086700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8084144b50 0x7f808411b530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:10:11.885 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.859+0000 7f808b086700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8084128650 con 0x7f8084077400 2019-12-05T06:10:11.886 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.859+0000 7f808b086700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8084076b10 con 0x7f80840a41e0 2019-12-05T06:10:11.886 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.859+0000 7f808b086700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8084077900 con 0x7f8084144b50 2019-12-05T06:10:11.887 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.859+0000 7f8088e22700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f8084077400 0x7f8084079480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:49658/0 (socket says 172.21.15.42:49658) 2019-12-05T06:10:11.887 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.859+0000 7f8089e24700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8084144b50 0x7f808411b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:10:11.887 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.859+0000 7f8088e22700 1 -- 172.21.15.42:0/4150862961 learned_addr learned my addr 172.21.15.42:0/4150862961 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:10:11.887 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.859+0000 7f8089623700 1 --2- 172.21.15.42:0/4150862961 >> v2:172.21.15.40:3300/0 conn(0x7f80840a41e0 0x7f8084128100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:10:11.887 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.863+0000 7f8079ffb700 1 -- 172.21.15.42:0/4150862961 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3361915550 0 0) 0x7f8084128650 con 0x7f8084077400 2019-12-05T06:10:11.887 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.863+0000 7f8079ffb700 1 -- 172.21.15.42:0/4150862961 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80640030b0 con 0x7f8084077400 2019-12-05T06:10:11.888 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.863+0000 7f8079ffb700 1 -- 172.21.15.42:0/4150862961 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 677327967 0 0) 0x7f80640030b0 con 0x7f8084077400 2019-12-05T06:10:11.888 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.863+0000 7f8089e24700 1 -- 172.21.15.42:0/4150862961 >> v2:172.21.15.40:3300/0 conn(0x7f80840a41e0 msgr2=0x7f8084128100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:10:11.888 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.863+0000 7f8089e24700 1 --2- 172.21.15.42:0/4150862961 >> v2:172.21.15.40:3300/0 conn(0x7f80840a41e0 0x7f8084128100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:10:11.888 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.863+0000 7f8089e24700 1 -- 172.21.15.42:0/4150862961 >> v1:172.21.15.42:6789/0 conn(0x7f8084077400 legacy=0x7f8084079480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:10:11.888 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.863+0000 7f8089e24700 1 -- 172.21.15.42:0/4150862961 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f808411ba80 con 0x7f8084144b50 2019-12-05T06:10:11.888 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.863+0000 7f8089e24700 1 --2- 172.21.15.42:0/4150862961 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8084144b50 0x7f808411b530 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f8084145310 tx=0x7f8080002060).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:11.889 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.863+0000 7f8089e24700 1 --2- 172.21.15.42:0/4150862961 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8084144b50 0x7f808411b530 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f8084145310 tx=0x7f8080002060).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:11.889 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.863+0000 7f8089e24700 1 --2- 172.21.15.42:0/4150862961 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8084144b50 0x7f808411b530 secure :-1 s=READY pgs=475 cs=0 l=1 rx=0x7f8084145310 tx=0x7f8080002060).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:10:11.889 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.863+0000 7f8089e24700 1 --2- 172.21.15.42:0/4150862961 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8084144b50 0x7f808411b530 secure :-1 s=THROTTLE_DONE pgs=475 cs=0 l=1 rx=0x7f8084145310 tx=0x7f8080002060).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:11.889 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.863+0000 7f8079ffb700 1 -- 172.21.15.42:0/4150862961 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8080001de0 con 0x7f8084144b50 2019-12-05T06:10:11.889 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.863+0000 7f8089e24700 1 --2- 172.21.15.42:0/4150862961 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8084144b50 0x7f808411b530 secure :-1 s=THROTTLE_DONE pgs=475 cs=0 l=1 rx=0x7f8084145310 tx=0x7f8080002060).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:11.889 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.863+0000 7f8089e24700 1 --2- 172.21.15.42:0/4150862961 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8084144b50 0x7f808411b530 secure :-1 s=THROTTLE_DONE pgs=475 cs=0 l=1 rx=0x7f8084145310 tx=0x7f8080002060).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:11.889 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.863+0000 7f8079ffb700 1 -- 172.21.15.42:0/4150862961 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8080001bf0 con 0x7f8084144b50 2019-12-05T06:10:11.890 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.863+0000 7f808b086700 1 -- 172.21.15.42:0/4150862961 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8084077ba0 con 0x7f8084144b50 2019-12-05T06:10:11.890 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.863+0000 7f8079ffb700 1 -- 172.21.15.42:0/4150862961 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f807c001850 con 0x7f8084144b50 2019-12-05T06:10:11.890 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.863+0000 7f808b086700 1 -- 172.21.15.42:0/4150862961 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f808412f550 con 0x7f8084144b50 2019-12-05T06:10:11.890 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.863+0000 7f8089e24700 1 --2- 172.21.15.42:0/4150862961 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8084144b50 0x7f808411b530 secure :-1 s=THROTTLE_DONE pgs=475 cs=0 l=1 rx=0x7f8084145310 tx=0x7f8080002060).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:11.890 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.863+0000 7f8079ffb700 1 -- 172.21.15.42:0/4150862961 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f807c001b20 con 0x7f8084144b50 2019-12-05T06:10:11.891 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.863+0000 7f8089e24700 1 --2- 172.21.15.42:0/4150862961 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8084144b50 0x7f808411b530 secure :-1 s=THROTTLE_DONE pgs=475 cs=0 l=1 rx=0x7f8084145310 tx=0x7f8080002060).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:11.891 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.863+0000 7f8079ffb700 1 --2- 172.21.15.42:0/4150862961 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f806403b260 0x7f806403d770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:10:11.891 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.863+0000 7f8079ffb700 1 -- 172.21.15.42:0/4150862961 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9987+0+0 (secure 0 0 0) 0x7f807c04a7e0 con 0x7f8084144b50 2019-12-05T06:10:11.891 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.863+0000 7f8089623700 1 --2- 172.21.15.42:0/4150862961 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f806403b260 0x7f806403d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:10:11.891 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.863+0000 7f8089623700 1 --2- 172.21.15.42:0/4150862961 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f806403b260 0x7f806403d770 crc :-1 s=READY pgs=907 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:10:11.891 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.867+0000 7f808b086700 1 -- 172.21.15.42:0/4150862961 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f808405d110 con 0x7f8084144b50 2019-12-05T06:10:11.892 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.875+0000 7f8089e24700 1 --2- 172.21.15.42:0/4150862961 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8084144b50 0x7f808411b530 secure :-1 s=THROTTLE_DONE pgs=475 cs=0 l=1 rx=0x7f8084145310 tx=0x7f8080002060).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:11.892 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:11.875+0000 7f8079ffb700 1 -- 172.21.15.42:0/4150862961 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f807c031840 con 0x7f8084144b50 2019-12-05T06:10:12.044 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:12.039+0000 7f808b086700 1 -- 172.21.15.42:0/4150862961 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f808405b8c0 con 0x7f8084144b50 2019-12-05T06:10:12.045 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:12.039+0000 7f8089e24700 1 --2- 172.21.15.42:0/4150862961 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8084144b50 0x7f808411b530 secure :-1 s=THROTTLE_DONE pgs=475 cs=0 l=1 rx=0x7f8084145310 tx=0x7f8080002060).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:12.046 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:12.039+0000 7f8079ffb700 1 -- 172.21.15.42:0/4150862961 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 259 v259) v1 ==== 95+0+1774 (secure 0 0 0) 0x7f807c007190 con 0x7f8084144b50 2019-12-05T06:10:12.046 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 259 2019-12-05T06:10:12.046 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:10:12.046 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":259,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":259,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:10:09.529943+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":95,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6977},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6977":{"gid":6977,"name":"b","rank":0,"incarnation":259,"state":"up:replay","state_seq":2,"addr":"172.21.15.40:6833/3355679996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3355679996},{"type":"v1","addr":"172.21.15.40:6833","nonce":3355679996}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:10:12.048 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:12.043+0000 7f808b086700 1 -- 172.21.15.42:0/4150862961 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f806403b260 msgr2=0x7f806403d770 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:10:12.048 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:12.043+0000 7f808b086700 1 --2- 172.21.15.42:0/4150862961 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f806403b260 0x7f806403d770 crc :-1 s=READY pgs=907 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:10:12.048 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:12.043+0000 7f808b086700 1 -- 172.21.15.42:0/4150862961 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8084144b50 msgr2=0x7f808411b530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:10:12.048 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:12.043+0000 7f808b086700 1 --2- 172.21.15.42:0/4150862961 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8084144b50 0x7f808411b530 secure :-1 s=READY pgs=475 cs=0 l=1 rx=0x7f8084145310 tx=0x7f8080002060).stop 2019-12-05T06:10:12.049 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:12.043+0000 7f808b086700 1 -- 172.21.15.42:0/4150862961 shutdown_connections 2019-12-05T06:10:12.049 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:12.043+0000 7f808b086700 1 --2- 172.21.15.42:0/4150862961 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f806403b260 0x7f806403d770 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:10:12.049 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:12.043+0000 7f808b086700 1 --2- 172.21.15.42:0/4150862961 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8084144b50 0x7f808411b530 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:10:12.049 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:12.043+0000 7f808b086700 1 --2- 172.21.15.42:0/4150862961 >> v2:172.21.15.40:3300/0 conn(0x7f80840a41e0 0x7f8084128100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:10:12.049 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:12.043+0000 7f808b086700 1 -- 172.21.15.42:0/4150862961 shutdown_connections 2019-12-05T06:10:12.050 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:12.043+0000 7f808b086700 1 -- 172.21.15.42:0/4150862961 wait complete. 2019-12-05T06:10:12.050 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:12.043+0000 7f808b086700 1 -- 172.21.15.42:0/4150862961 >> 172.21.15.42:0/4150862961 conn(0x7f808409f040 msgr2=0x7f808409f580 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:10:12.066 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 259, "feature_flags": { "enable_multiple": false, "ever_enabled_multiple": false }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2019-12-05T05:03:35.604738+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 259, "ever_allowed_features": 2, "explicitly_allowed_features": 2, "failed": [], "flags": 18, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_6977": { "addr": "172.21.15.40:6833/3355679996", "addrs": { "addrvec": [ { "addr": "172.21.15.40:6832", "nonce": 3355679996, "type": "v2" }, { "addr": "172.21.15.40:6833", "nonce": 3355679996, "type": "v1" } ] }, "export_targets": [], "features": 4540138292836696063, "flags": 0, "gid": 6977, "incarnation": 259, "name": "b", "rank": 0, "state": "up:replay", "state_seq": 2 } }, "last_failure": 0, "last_failure_osd_epoch": 95, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 2, "min_compat_client": "0 (unknown)", "modified": "2019-12-05T06:10:09.529943+0000", "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6977 } } } ], "standbys": [] } 2019-12-05T06:10:14.067 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:10:14.170 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:10:14.172 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:14.159+0000 7f997cfcb700 1 Processor -- start 2019-12-05T06:10:14.172 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:14.159+0000 7f997cfcb700 1 -- start start 2019-12-05T06:10:14.172 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:14.159+0000 7f997cfcb700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f99780a7740 0x7f997808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:10:14.172 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:14.159+0000 7f997cfcb700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f997807e720 0x7f997807eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:10:14.172 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:14.159+0000 7f997cfcb700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99780a0710 con 0x7f997809f2e0 2019-12-05T06:10:14.172 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:14.159+0000 7f997cfcb700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99780a0b00 con 0x7f99780a7740 2019-12-05T06:10:14.173 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:14.159+0000 7f997cfcb700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f997808e5b0 con 0x7f997807e720 2019-12-05T06:10:14.173 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:14.159+0000 7f9976d9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f99780a7740 0x7f997808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:10:14.173 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:14.163+0000 7f9976d9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f99780a7740 0x7f997808df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:49632/0 (socket says 172.21.15.42:49632) 2019-12-05T06:10:14.173 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:14.163+0000 7f9976d9d700 1 -- 172.21.15.42:0/2209754499 learned_addr learned my addr 172.21.15.42:0/2209754499 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:10:14.173 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:14.163+0000 7f997659c700 1 --2- 172.21.15.42:0/2209754499 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f997807e720 0x7f997807eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:10:14.174 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:14.163+0000 7f9976d9d700 1 -- 172.21.15.42:0/2209754499 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f997807e720 msgr2=0x7f997807eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:10:14.174 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:14.163+0000 7f9976d9d700 1 --2- 172.21.15.42:0/2209754499 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f997807e720 0x7f997807eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:10:14.174 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:14.163+0000 7f9976d9d700 1 -- 172.21.15.42:0/2209754499 >> v1:172.21.15.42:6789/0 conn(0x7f997809f2e0 legacy=0x7f997809f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:10:14.174 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:14.163+0000 7f9976d9d700 1 -- 172.21.15.42:0/2209754499 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f997807f0a0 con 0x7f99780a7740 2019-12-05T06:10:14.174 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:14.163+0000 7f9975d9b700 1 -- 172.21.15.42:0/2209754499 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1667980132 0 0) 0x7f99780a0710 con 0x7f997809f2e0 2019-12-05T06:10:14.174 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:14.163+0000 7f9976d9d700 1 --2- 172.21.15.42:0/2209754499 >> v2:172.21.15.42:3300/0 conn(0x7f99780a7740 0x7f997808df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f9968008b10 tx=0x7f99680100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:14.174 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:14.163+0000 7f9976d9d700 1 --2- 172.21.15.42:0/2209754499 >> v2:172.21.15.42:3300/0 conn(0x7f99780a7740 0x7f997808df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f9968008b10 tx=0x7f99680100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:14.175 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:14.163+0000 7f9976d9d700 1 --2- 172.21.15.42:0/2209754499 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f99780a7740 0x7f997808df80 secure :-1 s=READY pgs=2156 cs=0 l=1 rx=0x7f9968008b10 tx=0x7f99680100b0).ready entity=mon.1 client_cookie=1b308303af1148b3 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:10:14.175 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:14.163+0000 7f9976d9d700 1 --2- 172.21.15.42:0/2209754499 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f99780a7740 0x7f997808df80 secure :-1 s=THROTTLE_DONE pgs=2156 cs=0 l=1 rx=0x7f9968008b10 tx=0x7f99680100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:14.175 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:14.163+0000 7f9976d9d700 1 --2- 172.21.15.42:0/2209754499 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f99780a7740 0x7f997808df80 secure :-1 s=THROTTLE_DONE pgs=2156 cs=0 l=1 rx=0x7f9968008b10 tx=0x7f99680100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:14.175 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:14.163+0000 7f9975d9b700 1 -- 172.21.15.42:0/2209754499 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f996800a160 con 0x7f99780a7740 2019-12-05T06:10:14.175 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:14.163+0000 7f9976d9d700 1 --2- 172.21.15.42:0/2209754499 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f99780a7740 0x7f997808df80 secure :-1 s=THROTTLE_DONE pgs=2156 cs=0 l=1 rx=0x7f9968008b10 tx=0x7f99680100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:14.175 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:14.163+0000 7f9975d9b700 1 -- 172.21.15.42:0/2209754499 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9968011a30 con 0x7f99780a7740 2019-12-05T06:10:14.175 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:14.163+0000 7f9975d9b700 1 -- 172.21.15.42:0/2209754499 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f996800a550 con 0x7f99780a7740 2019-12-05T06:10:14.176 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:14.163+0000 7f997cfcb700 1 -- 172.21.15.42:0/2209754499 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f99780a7740 msgr2=0x7f997808df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:10:14.176 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:14.163+0000 7f997cfcb700 1 --2- 172.21.15.42:0/2209754499 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f99780a7740 0x7f997808df80 secure :-1 s=READY pgs=2156 cs=0 l=1 rx=0x7f9968008b10 tx=0x7f99680100b0).stop 2019-12-05T06:10:14.176 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:14.163+0000 7f997cfcb700 1 -- 172.21.15.42:0/2209754499 shutdown_connections 2019-12-05T06:10:14.176 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:14.163+0000 7f997cfcb700 1 --2- 172.21.15.42:0/2209754499 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f99780a7740 0x7f997808df80 unknown :-1 s=CLOSED pgs=2156 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:10:14.176 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:14.163+0000 7f997cfcb700 1 --2- 172.21.15.42:0/2209754499 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f997807e720 0x7f997807eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:10:14.176 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:14.163+0000 7f997cfcb700 1 -- 172.21.15.42:0/2209754499 shutdown_connections 2019-12-05T06:10:14.176 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:14.163+0000 7f997cfcb700 1 -- 172.21.15.42:0/2209754499 wait complete. 2019-12-05T06:10:14.177 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:14.163+0000 7f997cfcb700 1 -- 172.21.15.42:0/2209754499 >> 172.21.15.42:0/2209754499 conn(0x7f99780aabe0 msgr2=0x7f99780a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:10:14.180 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:14.167+0000 7f997cfcb700 1 Processor -- start 2019-12-05T06:10:14.180 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:14.167+0000 7f997cfcb700 1 -- start start 2019-12-05T06:10:14.180 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:14.167+0000 7f997cfcb700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f99781215e0 0x7f997811af70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:10:14.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:14.167+0000 7f997cfcb700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f9978123f10 0x7f9978122930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:10:14.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:14.167+0000 7f997cfcb700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f9978122e80 0x7f997808eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:10:14.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:14.167+0000 7f997cfcb700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99781246b0 con 0x7f9978122e80 2019-12-05T06:10:14.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:14.167+0000 7f997cfcb700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f997811a790 con 0x7f9978123f10 2019-12-05T06:10:14.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:14.167+0000 7f997cfcb700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f997811aa30 con 0x7f99781215e0 2019-12-05T06:10:14.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:14.167+0000 7f997659c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f9978123f10 0x7f9978122930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:10:14.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:14.167+0000 7f997659c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f9978123f10 0x7f9978122930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:49640/0 (socket says 172.21.15.42:49640) 2019-12-05T06:10:14.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:14.167+0000 7f997659c700 1 -- 172.21.15.42:0/1883688233 learned_addr learned my addr 172.21.15.42:0/1883688233 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:10:14.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:14.167+0000 7f9976d9d700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f99781215e0 0x7f997811af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:10:14.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:14.167+0000 7f997759e700 1 --2- 172.21.15.42:0/1883688233 >> v2:172.21.15.40:3300/0 conn(0x7f9978122e80 0x7f997808eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:10:14.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:14.167+0000 7f997659c700 1 -- 172.21.15.42:0/1883688233 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f99781215e0 msgr2=0x7f997811af70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:10:14.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:14.167+0000 7f997659c700 1 --2- 172.21.15.42:0/1883688233 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f99781215e0 0x7f997811af70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:10:14.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:14.167+0000 7f997659c700 1 -- 172.21.15.42:0/1883688233 >> v2:172.21.15.40:3300/0 conn(0x7f9978122e80 msgr2=0x7f997808eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:10:14.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:14.167+0000 7f997659c700 1 --2- 172.21.15.42:0/1883688233 >> v2:172.21.15.40:3300/0 conn(0x7f9978122e80 0x7f997808eb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:10:14.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:14.167+0000 7f997659c700 1 -- 172.21.15.42:0/1883688233 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9978124340 con 0x7f9978123f10 2019-12-05T06:10:14.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:14.167+0000 7f997659c700 1 --2- 172.21.15.42:0/1883688233 >> v2:172.21.15.42:3300/0 conn(0x7f9978123f10 0x7f9978122930 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f997811abf0 tx=0x7f996c010cc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:14.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:14.167+0000 7f997659c700 1 --2- 172.21.15.42:0/1883688233 >> v2:172.21.15.42:3300/0 conn(0x7f9978123f10 0x7f9978122930 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f997811abf0 tx=0x7f996c010cc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:14.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:14.167+0000 7f997659c700 1 --2- 172.21.15.42:0/1883688233 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9978123f10 0x7f9978122930 secure :-1 s=READY pgs=2157 cs=0 l=1 rx=0x7f997811abf0 tx=0x7f996c010cc0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:10:14.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:14.167+0000 7f997659c700 1 --2- 172.21.15.42:0/1883688233 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9978123f10 0x7f9978122930 secure :-1 s=THROTTLE_DONE pgs=2157 cs=0 l=1 rx=0x7f997811abf0 tx=0x7f996c010cc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:14.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:14.167+0000 7f997659c700 1 --2- 172.21.15.42:0/1883688233 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9978123f10 0x7f9978122930 secure :-1 s=THROTTLE_DONE pgs=2157 cs=0 l=1 rx=0x7f997811abf0 tx=0x7f996c010cc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:14.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:14.167+0000 7f997659c700 1 --2- 172.21.15.42:0/1883688233 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9978123f10 0x7f9978122930 secure :-1 s=THROTTLE_DONE pgs=2157 cs=0 l=1 rx=0x7f997811abf0 tx=0x7f996c010cc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:14.184 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:14.167+0000 7f99677fe700 1 -- 172.21.15.42:0/1883688233 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f996c005930 con 0x7f9978123f10 2019-12-05T06:10:14.184 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:14.167+0000 7f99677fe700 1 -- 172.21.15.42:0/1883688233 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f996c007160 con 0x7f9978123f10 2019-12-05T06:10:14.184 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:14.167+0000 7f997cfcb700 1 -- 172.21.15.42:0/1883688233 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f997808f0c0 con 0x7f9978123f10 2019-12-05T06:10:14.184 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:14.167+0000 7f99677fe700 1 -- 172.21.15.42:0/1883688233 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f996c012300 con 0x7f9978123f10 2019-12-05T06:10:14.187 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:14.171+0000 7f997cfcb700 1 -- 172.21.15.42:0/1883688233 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f997808f640 con 0x7f9978123f10 2019-12-05T06:10:14.187 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:14.171+0000 7f997659c700 1 --2- 172.21.15.42:0/1883688233 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9978123f10 0x7f9978122930 secure :-1 s=THROTTLE_DONE pgs=2157 cs=0 l=1 rx=0x7f997811abf0 tx=0x7f996c010cc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:14.188 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:14.171+0000 7f99677fe700 1 -- 172.21.15.42:0/1883688233 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f996c012610 con 0x7f9978123f10 2019-12-05T06:10:14.188 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:14.171+0000 7f997659c700 1 --2- 172.21.15.42:0/1883688233 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9978123f10 0x7f9978122930 secure :-1 s=THROTTLE_DONE pgs=2157 cs=0 l=1 rx=0x7f997811abf0 tx=0x7f996c010cc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:14.188 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:14.171+0000 7f99677fe700 1 --2- 172.21.15.42:0/1883688233 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f995c03ae90 0x7f995c03d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:10:14.188 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:14.171+0000 7f99677fe700 1 -- 172.21.15.42:0/1883688233 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9987+0+0 (secure 0 0 0) 0x7f996c04b430 con 0x7f9978123f10 2019-12-05T06:10:14.188 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:14.171+0000 7f9976d9d700 1 --2- 172.21.15.42:0/1883688233 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f995c03ae90 0x7f995c03d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:10:14.188 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:14.171+0000 7f9976d9d700 1 --2- 172.21.15.42:0/1883688233 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f995c03ae90 0x7f995c03d3a0 crc :-1 s=READY pgs=909 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:10:14.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:14.175+0000 7f997cfcb700 1 -- 172.21.15.42:0/1883688233 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f997805d110 con 0x7f9978123f10 2019-12-05T06:10:14.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:14.183+0000 7f997659c700 1 --2- 172.21.15.42:0/1883688233 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9978123f10 0x7f9978122930 secure :-1 s=THROTTLE_DONE pgs=2157 cs=0 l=1 rx=0x7f997811abf0 tx=0x7f996c010cc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:14.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:14.183+0000 7f99677fe700 1 -- 172.21.15.42:0/1883688233 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f996c0324b0 con 0x7f9978123f10 2019-12-05T06:10:14.190 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:10:14.358 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:14.351+0000 7f997cfcb700 1 -- 172.21.15.42:0/1883688233 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f997805b8c0 con 0x7f9978123f10 2019-12-05T06:10:14.359 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:14.351+0000 7f997659c700 1 --2- 172.21.15.42:0/1883688233 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9978123f10 0x7f9978122930 secure :-1 s=THROTTLE_DONE pgs=2157 cs=0 l=1 rx=0x7f997811abf0 tx=0x7f996c010cc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:14.359 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:14.351+0000 7f99677fe700 1 -- 172.21.15.42:0/1883688233 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 261 v261) v1 ==== 95+0+1775 (secure 0 0 0) 0x7f996c032710 con 0x7f9978123f10 2019-12-05T06:10:14.360 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 261 2019-12-05T06:10:14.360 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:10:14.360 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":261,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":261,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:10:13.923084+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":95,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6977},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6977":{"gid":6977,"name":"b","rank":0,"incarnation":259,"state":"up:rejoin","state_seq":31,"addr":"172.21.15.40:6833/3355679996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3355679996},{"type":"v1","addr":"172.21.15.40:6833","nonce":3355679996}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:10:14.362 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:14.355+0000 7f997cfcb700 1 -- 172.21.15.42:0/1883688233 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f995c03ae90 msgr2=0x7f995c03d3a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:10:14.362 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:14.355+0000 7f997cfcb700 1 --2- 172.21.15.42:0/1883688233 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f995c03ae90 0x7f995c03d3a0 crc :-1 s=READY pgs=909 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:10:14.362 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:14.355+0000 7f997cfcb700 1 -- 172.21.15.42:0/1883688233 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9978123f10 msgr2=0x7f9978122930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:10:14.362 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:14.355+0000 7f997cfcb700 1 --2- 172.21.15.42:0/1883688233 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9978123f10 0x7f9978122930 secure :-1 s=READY pgs=2157 cs=0 l=1 rx=0x7f997811abf0 tx=0x7f996c010cc0).stop 2019-12-05T06:10:14.362 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:14.355+0000 7f997cfcb700 1 -- 172.21.15.42:0/1883688233 shutdown_connections 2019-12-05T06:10:14.363 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:14.355+0000 7f997cfcb700 1 --2- 172.21.15.42:0/1883688233 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f995c03ae90 0x7f995c03d3a0 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:10:14.363 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:14.355+0000 7f997cfcb700 1 --2- 172.21.15.42:0/1883688233 >> v2:172.21.15.40:3300/0 conn(0x7f9978122e80 0x7f997808eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:10:14.363 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:14.355+0000 7f997cfcb700 1 --2- 172.21.15.42:0/1883688233 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f99781215e0 0x7f997811af70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:10:14.363 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:14.355+0000 7f997cfcb700 1 --2- 172.21.15.42:0/1883688233 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9978123f10 0x7f9978122930 unknown :-1 s=CLOSED pgs=2157 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:10:14.363 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:14.355+0000 7f997cfcb700 1 -- 172.21.15.42:0/1883688233 shutdown_connections 2019-12-05T06:10:14.363 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:14.355+0000 7f997cfcb700 1 -- 172.21.15.42:0/1883688233 wait complete. 2019-12-05T06:10:14.363 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:14.355+0000 7f997cfcb700 1 -- 172.21.15.42:0/1883688233 >> 172.21.15.42:0/1883688233 conn(0x7f99780a7740 msgr2=0x7f997809f6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:10:14.372 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2019-12-05T06:10:16.373 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:10:16.483 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:16.471+0000 7f6062daa700 1 Processor -- start 2019-12-05T06:10:16.484 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:16.471+0000 7f6062daa700 1 -- start start 2019-12-05T06:10:16.484 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:16.471+0000 7f6062daa700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f605c08df80 0x7f605c09f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:10:16.484 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:16.471+0000 7f6062daa700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f605c09f830 0x7f605c14efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:10:16.484 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:16.471+0000 7f6062daa700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f605c09fc60 con 0x7f605c0a7740 2019-12-05T06:10:16.484 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:16.471+0000 7f6062daa700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f605c0a0b00 con 0x7f605c09f830 2019-12-05T06:10:16.485 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:16.471+0000 7f6062daa700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f605c08e5b0 con 0x7f605c08df80 2019-12-05T06:10:16.485 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:16.471+0000 7f6061347700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f605c0a7740 0x7f605c07e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:49696/0 (socket says 172.21.15.42:49696) 2019-12-05T06:10:16.485 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:16.471+0000 7f6061347700 1 -- 172.21.15.42:0/2647564729 learned_addr learned my addr 172.21.15.42:0/2647564729 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:10:16.485 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:16.471+0000 7f6061b48700 1 --2- 172.21.15.42:0/2647564729 >> v2:172.21.15.42:3300/0 conn(0x7f605c09f830 0x7f605c14efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:10:16.485 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:16.471+0000 7f6060b46700 1 --2- 172.21.15.42:0/2647564729 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f605c08df80 0x7f605c09f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:10:16.485 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:16.471+0000 7f6061b48700 1 -- 172.21.15.42:0/2647564729 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f605c08df80 msgr2=0x7f605c09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:10:16.485 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:16.471+0000 7f6061b48700 1 --2- 172.21.15.42:0/2647564729 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f605c08df80 0x7f605c09f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:10:16.486 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:16.471+0000 7f6061b48700 1 -- 172.21.15.42:0/2647564729 >> v1:172.21.15.42:6789/0 conn(0x7f605c0a7740 legacy=0x7f605c07e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:10:16.486 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:16.471+0000 7f6061b48700 1 -- 172.21.15.42:0/2647564729 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f605c0a0710 con 0x7f605c09f830 2019-12-05T06:10:16.486 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:16.471+0000 7f6061b48700 1 --2- 172.21.15.42:0/2647564729 >> v2:172.21.15.42:3300/0 conn(0x7f605c09f830 0x7f605c14efe0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f6054008b10 tx=0x7f605400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:16.486 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:16.471+0000 7f6061b48700 1 --2- 172.21.15.42:0/2647564729 >> v2:172.21.15.42:3300/0 conn(0x7f605c09f830 0x7f605c14efe0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f6054008b10 tx=0x7f605400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:16.486 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:16.471+0000 7f6061b48700 1 --2- 172.21.15.42:0/2647564729 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f605c09f830 0x7f605c14efe0 secure :-1 s=READY pgs=2159 cs=0 l=1 rx=0x7f6054008b10 tx=0x7f605400fff0).ready entity=mon.1 client_cookie=2fa59d9105817316 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:10:16.486 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:16.475+0000 7f6061b48700 1 --2- 172.21.15.42:0/2647564729 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f605c09f830 0x7f605c14efe0 secure :-1 s=THROTTLE_DONE pgs=2159 cs=0 l=1 rx=0x7f6054008b10 tx=0x7f605400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:16.486 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:16.475+0000 7f6061b48700 1 --2- 172.21.15.42:0/2647564729 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f605c09f830 0x7f605c14efe0 secure :-1 s=THROTTLE_DONE pgs=2159 cs=0 l=1 rx=0x7f6054008b10 tx=0x7f605400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:16.487 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:16.475+0000 7f6053fff700 1 -- 172.21.15.42:0/2647564729 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6054010900 con 0x7f605c09f830 2019-12-05T06:10:16.487 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:16.475+0000 7f6061b48700 1 --2- 172.21.15.42:0/2647564729 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f605c09f830 0x7f605c14efe0 secure :-1 s=THROTTLE_DONE pgs=2159 cs=0 l=1 rx=0x7f6054008b10 tx=0x7f605400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:16.487 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:16.475+0000 7f6053fff700 1 -- 172.21.15.42:0/2647564729 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6054008b50 con 0x7f605c09f830 2019-12-05T06:10:16.487 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:16.475+0000 7f6053fff700 1 -- 172.21.15.42:0/2647564729 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f60540110c0 con 0x7f605c09f830 2019-12-05T06:10:16.487 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:16.475+0000 7f6062daa700 1 -- 172.21.15.42:0/2647564729 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f605c09f830 msgr2=0x7f605c14efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:10:16.487 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:16.475+0000 7f6062daa700 1 --2- 172.21.15.42:0/2647564729 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f605c09f830 0x7f605c14efe0 secure :-1 s=READY pgs=2159 cs=0 l=1 rx=0x7f6054008b10 tx=0x7f605400fff0).stop 2019-12-05T06:10:16.487 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:16.475+0000 7f6062daa700 1 -- 172.21.15.42:0/2647564729 shutdown_connections 2019-12-05T06:10:16.488 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:16.475+0000 7f6062daa700 1 --2- 172.21.15.42:0/2647564729 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f605c09f830 0x7f605c14efe0 unknown :-1 s=CLOSED pgs=2159 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:10:16.488 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:16.475+0000 7f6062daa700 1 --2- 172.21.15.42:0/2647564729 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f605c08df80 0x7f605c09f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:10:16.488 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:16.475+0000 7f6062daa700 1 -- 172.21.15.42:0/2647564729 shutdown_connections 2019-12-05T06:10:16.488 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:16.475+0000 7f6062daa700 1 -- 172.21.15.42:0/2647564729 wait complete. 2019-12-05T06:10:16.488 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:16.475+0000 7f6062daa700 1 -- 172.21.15.42:0/2647564729 >> 172.21.15.42:0/2647564729 conn(0x7f605c0aabe0 msgr2=0x7f605c0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:10:16.492 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:16.479+0000 7f6062daa700 1 Processor -- start 2019-12-05T06:10:16.493 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:16.479+0000 7f6062daa700 1 -- start start 2019-12-05T06:10:16.493 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:16.479+0000 7f6062daa700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f605c144b50 0x7f605c0a38b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:10:16.493 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:16.479+0000 7f6062daa700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f605c0a3e00 0x7f605c1150b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:10:16.493 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:16.479+0000 7f6062daa700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f605c115600 con 0x7f605c1244c0 2019-12-05T06:10:16.493 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:16.479+0000 7f6062daa700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f605c0a4230 con 0x7f605c0a3e00 2019-12-05T06:10:16.493 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:16.479+0000 7f6062daa700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f605c118220 con 0x7f605c144b50 2019-12-05T06:10:16.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:16.479+0000 7f6060b46700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f605c144b50 0x7f605c0a38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:10:16.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:16.479+0000 7f6060b46700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f605c144b50 0x7f605c0a38b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:49666/0 (socket says 172.21.15.42:49666) 2019-12-05T06:10:16.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:16.479+0000 7f6060b46700 1 -- 172.21.15.42:0/1764017948 learned_addr learned my addr 172.21.15.42:0/1764017948 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:10:16.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:16.479+0000 7f6061b48700 1 --2- 172.21.15.42:0/1764017948 >> v2:172.21.15.40:3300/0 conn(0x7f605c0a3e00 0x7f605c1150b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:10:16.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:16.479+0000 7f6060b46700 1 -- 172.21.15.42:0/1764017948 >> v2:172.21.15.40:3300/0 conn(0x7f605c0a3e00 msgr2=0x7f605c1150b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:10:16.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:16.479+0000 7f6060b46700 1 --2- 172.21.15.42:0/1764017948 >> v2:172.21.15.40:3300/0 conn(0x7f605c0a3e00 0x7f605c1150b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:10:16.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:16.479+0000 7f6060b46700 1 -- 172.21.15.42:0/1764017948 >> v1:172.21.15.40:6789/0 conn(0x7f605c1244c0 legacy=0x7f605c1281f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:10:16.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:16.479+0000 7f6060b46700 1 -- 172.21.15.42:0/1764017948 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f605c115830 con 0x7f605c144b50 2019-12-05T06:10:16.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:16.479+0000 7f6051ffb700 1 -- 172.21.15.42:0/1764017948 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4272897684 0 0) 0x7f605c115600 con 0x7f605c1244c0 2019-12-05T06:10:16.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:16.479+0000 7f6060b46700 1 --2- 172.21.15.42:0/1764017948 >> v2:172.21.15.42:3300/0 conn(0x7f605c144b50 0x7f605c0a38b0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f604c003070 tx=0x7f604c010cc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:16.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:16.479+0000 7f6060b46700 1 --2- 172.21.15.42:0/1764017948 >> v2:172.21.15.42:3300/0 conn(0x7f605c144b50 0x7f605c0a38b0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f604c003070 tx=0x7f604c010cc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:16.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:16.479+0000 7f6060b46700 1 --2- 172.21.15.42:0/1764017948 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f605c144b50 0x7f605c0a38b0 secure :-1 s=READY pgs=2160 cs=0 l=1 rx=0x7f604c003070 tx=0x7f604c010cc0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:10:16.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:16.479+0000 7f6060b46700 1 --2- 172.21.15.42:0/1764017948 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f605c144b50 0x7f605c0a38b0 secure :-1 s=THROTTLE_DONE pgs=2160 cs=0 l=1 rx=0x7f604c003070 tx=0x7f604c010cc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:16.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:16.479+0000 7f6060b46700 1 --2- 172.21.15.42:0/1764017948 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f605c144b50 0x7f605c0a38b0 secure :-1 s=THROTTLE_DONE pgs=2160 cs=0 l=1 rx=0x7f604c003070 tx=0x7f604c010cc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:16.496 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:16.479+0000 7f6051ffb700 1 -- 172.21.15.42:0/1764017948 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f604c005980 con 0x7f605c144b50 2019-12-05T06:10:16.496 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:16.479+0000 7f6060b46700 1 --2- 172.21.15.42:0/1764017948 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f605c144b50 0x7f605c0a38b0 secure :-1 s=THROTTLE_DONE pgs=2160 cs=0 l=1 rx=0x7f604c003070 tx=0x7f604c010cc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:16.496 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:16.479+0000 7f6051ffb700 1 -- 172.21.15.42:0/1764017948 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f604c007160 con 0x7f605c144b50 2019-12-05T06:10:16.496 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:16.479+0000 7f6062daa700 1 -- 172.21.15.42:0/1764017948 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f605c12eed0 con 0x7f605c144b50 2019-12-05T06:10:16.496 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:16.479+0000 7f6051ffb700 1 -- 172.21.15.42:0/1764017948 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f604c012340 con 0x7f605c144b50 2019-12-05T06:10:16.496 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:16.479+0000 7f6062daa700 1 -- 172.21.15.42:0/1764017948 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f605c12f3a0 con 0x7f605c144b50 2019-12-05T06:10:16.497 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:16.483+0000 7f6060b46700 1 --2- 172.21.15.42:0/1764017948 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f605c144b50 0x7f605c0a38b0 secure :-1 s=THROTTLE_DONE pgs=2160 cs=0 l=1 rx=0x7f604c003070 tx=0x7f604c010cc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:16.497 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:16.483+0000 7f6051ffb700 1 -- 172.21.15.42:0/1764017948 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f604c012650 con 0x7f605c144b50 2019-12-05T06:10:16.497 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:16.483+0000 7f6060b46700 1 --2- 172.21.15.42:0/1764017948 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f605c144b50 0x7f605c0a38b0 secure :-1 s=THROTTLE_DONE pgs=2160 cs=0 l=1 rx=0x7f604c003070 tx=0x7f604c010cc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:16.497 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:16.483+0000 7f6051ffb700 1 --2- 172.21.15.42:0/1764017948 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f603c03acf0 0x7f603c03d200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:10:16.497 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:16.483+0000 7f6051ffb700 1 -- 172.21.15.42:0/1764017948 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9987+0+0 (secure 0 0 0) 0x7f604c04b3f0 con 0x7f605c144b50 2019-12-05T06:10:16.497 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:16.483+0000 7f6061347700 1 --2- 172.21.15.42:0/1764017948 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f603c03acf0 0x7f603c03d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:10:16.497 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:16.483+0000 7f6061347700 1 --2- 172.21.15.42:0/1764017948 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f603c03acf0 0x7f603c03d200 crc :-1 s=READY pgs=910 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:10:16.498 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:16.487+0000 7f6062daa700 1 -- 172.21.15.42:0/1764017948 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f605c05d110 con 0x7f605c144b50 2019-12-05T06:10:16.498 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:16.491+0000 7f6060b46700 1 --2- 172.21.15.42:0/1764017948 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f605c144b50 0x7f605c0a38b0 secure :-1 s=THROTTLE_DONE pgs=2160 cs=0 l=1 rx=0x7f604c003070 tx=0x7f604c010cc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:16.499 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:16.491+0000 7f6051ffb700 1 -- 172.21.15.42:0/1764017948 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f604c032470 con 0x7f605c144b50 2019-12-05T06:10:16.662 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:16.655+0000 7f6062daa700 1 -- 172.21.15.42:0/1764017948 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f605c05b8c0 con 0x7f605c144b50 2019-12-05T06:10:16.663 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:16.655+0000 7f6060b46700 1 --2- 172.21.15.42:0/1764017948 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f605c144b50 0x7f605c0a38b0 secure :-1 s=THROTTLE_DONE pgs=2160 cs=0 l=1 rx=0x7f604c003070 tx=0x7f604c010cc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:16.663 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:16.655+0000 7f6051ffb700 1 -- 172.21.15.42:0/1764017948 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 262 v262) v1 ==== 95+0+1775 (secure 0 0 0) 0x7f604c0326d0 con 0x7f605c144b50 2019-12-05T06:10:16.663 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 262 2019-12-05T06:10:16.663 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:10:16.663 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":262,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":262,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:10:14.933627+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":95,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6977},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6977":{"gid":6977,"name":"b","rank":0,"incarnation":259,"state":"up:active","state_seq":32,"addr":"172.21.15.40:6833/3355679996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3355679996},{"type":"v1","addr":"172.21.15.40:6833","nonce":3355679996}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:10:16.666 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:16.659+0000 7f6062daa700 1 -- 172.21.15.42:0/1764017948 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f603c03acf0 msgr2=0x7f603c03d200 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:10:16.666 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:16.659+0000 7f6062daa700 1 --2- 172.21.15.42:0/1764017948 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f603c03acf0 0x7f603c03d200 crc :-1 s=READY pgs=910 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:10:16.666 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:16.659+0000 7f6062daa700 1 -- 172.21.15.42:0/1764017948 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f605c144b50 msgr2=0x7f605c0a38b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:10:16.666 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:16.659+0000 7f6062daa700 1 --2- 172.21.15.42:0/1764017948 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f605c144b50 0x7f605c0a38b0 secure :-1 s=READY pgs=2160 cs=0 l=1 rx=0x7f604c003070 tx=0x7f604c010cc0).stop 2019-12-05T06:10:16.666 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:16.659+0000 7f6062daa700 1 -- 172.21.15.42:0/1764017948 shutdown_connections 2019-12-05T06:10:16.667 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:16.659+0000 7f6062daa700 1 --2- 172.21.15.42:0/1764017948 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f603c03acf0 0x7f603c03d200 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:10:16.667 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:16.659+0000 7f6062daa700 1 --2- 172.21.15.42:0/1764017948 >> v2:172.21.15.40:3300/0 conn(0x7f605c0a3e00 0x7f605c1150b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:10:16.667 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:16.659+0000 7f6062daa700 1 --2- 172.21.15.42:0/1764017948 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f605c144b50 0x7f605c0a38b0 unknown :-1 s=CLOSED pgs=2160 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:10:16.667 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:16.659+0000 7f6062daa700 1 -- 172.21.15.42:0/1764017948 shutdown_connections 2019-12-05T06:10:16.667 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:16.659+0000 7f6062daa700 1 -- 172.21.15.42:0/1764017948 wait complete. 2019-12-05T06:10:16.667 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:16.659+0000 7f6062daa700 1 -- 172.21.15.42:0/1764017948 >> 172.21.15.42:0/1764017948 conn(0x7f605c0a82a0 msgr2=0x7f605c115e80 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:10:16.681 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b has gained rank=0, replacing gid=6929 2019-12-05T06:10:16.682 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 4 secs before reviving mds.a 2019-12-05T06:10:20.682 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.a 2019-12-05T06:10:20.683 INFO:tasks.ceph.mds.a:Restarting daemon 2019-12-05T06:10:20.683 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2019-12-05T06:10:20.687 INFO:tasks.ceph.mds.a:Started 2019-12-05T06:10:20.687 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:10:20.745 INFO:tasks.ceph.mds.a.smithi042.stdout:starting mds.a at 2019-12-05T06:10:20.785 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:20.775+0000 7f2b20ad3700 1 Processor -- start 2019-12-05T06:10:20.786 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:20.775+0000 7f2b20ad3700 1 -- start start 2019-12-05T06:10:20.786 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:20.775+0000 7f2b20ad3700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2b1c0a7740 0x7f2b1c08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:10:20.786 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:20.775+0000 7f2b20ad3700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2b1c07e720 0x7f2b1c07eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:10:20.786 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:20.775+0000 7f2b20ad3700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2b1c09f2e0 0x7f2b1c09f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:10:20.786 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:20.775+0000 7f2b20ad3700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b1c0a0b00 con 0x7f2b1c09f2e0 2019-12-05T06:10:20.787 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:20.775+0000 7f2b20ad3700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b1c08e5b0 con 0x7f2b1c0a7740 2019-12-05T06:10:20.787 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:20.775+0000 7f2b20ad3700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b1c08e850 con 0x7f2b1c07e720 2019-12-05T06:10:20.787 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:20.775+0000 7f2b1ad9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2b1c0a7740 0x7f2b1c08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:10:20.787 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:20.775+0000 7f2b1ad9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2b1c0a7740 0x7f2b1c08df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:49704/0 (socket says 172.21.15.42:49704) 2019-12-05T06:10:20.787 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:20.775+0000 7f2b1ad9d700 1 -- 172.21.15.42:0/1802948531 learned_addr learned my addr 172.21.15.42:0/1802948531 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:10:20.787 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:20.775+0000 7f2b1b59e700 1 --2- 172.21.15.42:0/1802948531 >> v2:172.21.15.40:3300/0 conn(0x7f2b1c09f2e0 0x7f2b1c09f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:10:20.788 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:20.775+0000 7f2b1a59c700 1 --2- 172.21.15.42:0/1802948531 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2b1c07e720 0x7f2b1c07eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:10:20.788 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:20.775+0000 7f2b1ad9d700 1 -- 172.21.15.42:0/1802948531 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2b1c07e720 msgr2=0x7f2b1c07eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:10:20.788 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:20.775+0000 7f2b1ad9d700 1 --2- 172.21.15.42:0/1802948531 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2b1c07e720 0x7f2b1c07eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:10:20.788 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:20.775+0000 7f2b1ad9d700 1 -- 172.21.15.42:0/1802948531 >> v2:172.21.15.40:3300/0 conn(0x7f2b1c09f2e0 msgr2=0x7f2b1c09f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:10:20.788 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:20.775+0000 7f2b1ad9d700 1 --2- 172.21.15.42:0/1802948531 >> v2:172.21.15.40:3300/0 conn(0x7f2b1c09f2e0 0x7f2b1c09f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:10:20.788 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:20.775+0000 7f2b1ad9d700 1 -- 172.21.15.42:0/1802948531 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b1c07f0a0 con 0x7f2b1c0a7740 2019-12-05T06:10:20.789 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:20.775+0000 7f2b1a59c700 1 --2- 172.21.15.42:0/1802948531 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2b1c07e720 0x7f2b1c07eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:10:20.789 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:20.775+0000 7f2b1ad9d700 1 --2- 172.21.15.42:0/1802948531 >> v2:172.21.15.42:3300/0 conn(0x7f2b1c0a7740 0x7f2b1c08df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f2b1c08e770 tx=0x7f2b0c00fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:20.789 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:20.775+0000 7f2b1ad9d700 1 --2- 172.21.15.42:0/1802948531 >> v2:172.21.15.42:3300/0 conn(0x7f2b1c0a7740 0x7f2b1c08df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f2b1c08e770 tx=0x7f2b0c00fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:20.789 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:20.775+0000 7f2b1ad9d700 1 --2- 172.21.15.42:0/1802948531 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2b1c0a7740 0x7f2b1c08df80 secure :-1 s=READY pgs=2165 cs=0 l=1 rx=0x7f2b1c08e770 tx=0x7f2b0c00fdb0).ready entity=mon.1 client_cookie=85870fd0362456a2 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:10:20.789 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:20.775+0000 7f2b1ad9d700 1 --2- 172.21.15.42:0/1802948531 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2b1c0a7740 0x7f2b1c08df80 secure :-1 s=THROTTLE_DONE pgs=2165 cs=0 l=1 rx=0x7f2b1c08e770 tx=0x7f2b0c00fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:20.789 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:20.775+0000 7f2b19d9b700 1 -- 172.21.15.42:0/1802948531 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2b0c008c90 con 0x7f2b1c0a7740 2019-12-05T06:10:20.790 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:20.775+0000 7f2b1ad9d700 1 --2- 172.21.15.42:0/1802948531 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2b1c0a7740 0x7f2b1c08df80 secure :-1 s=THROTTLE_DONE pgs=2165 cs=0 l=1 rx=0x7f2b1c08e770 tx=0x7f2b0c00fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:20.790 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:20.775+0000 7f2b1ad9d700 1 --2- 172.21.15.42:0/1802948531 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2b1c0a7740 0x7f2b1c08df80 secure :-1 s=THROTTLE_DONE pgs=2165 cs=0 l=1 rx=0x7f2b1c08e770 tx=0x7f2b0c00fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:20.790 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:20.775+0000 7f2b19d9b700 1 -- 172.21.15.42:0/1802948531 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b0c0088b0 con 0x7f2b1c0a7740 2019-12-05T06:10:20.790 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:20.775+0000 7f2b19d9b700 1 -- 172.21.15.42:0/1802948531 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2b0c011b50 con 0x7f2b1c0a7740 2019-12-05T06:10:20.790 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:20.775+0000 7f2b20ad3700 1 -- 172.21.15.42:0/1802948531 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2b1c0a7740 msgr2=0x7f2b1c08df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:10:20.791 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:20.775+0000 7f2b20ad3700 1 --2- 172.21.15.42:0/1802948531 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2b1c0a7740 0x7f2b1c08df80 secure :-1 s=READY pgs=2165 cs=0 l=1 rx=0x7f2b1c08e770 tx=0x7f2b0c00fdb0).stop 2019-12-05T06:10:20.791 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:20.779+0000 7f2b20ad3700 1 -- 172.21.15.42:0/1802948531 shutdown_connections 2019-12-05T06:10:20.791 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:20.779+0000 7f2b20ad3700 1 --2- 172.21.15.42:0/1802948531 >> v2:172.21.15.40:3300/0 conn(0x7f2b1c09f2e0 0x7f2b1c09f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:10:20.792 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:20.779+0000 7f2b20ad3700 1 --2- 172.21.15.42:0/1802948531 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2b1c0a7740 0x7f2b1c08df80 unknown :-1 s=CLOSED pgs=2165 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:10:20.793 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:20.779+0000 7f2b20ad3700 1 --2- 172.21.15.42:0/1802948531 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2b1c07e720 0x7f2b1c07eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:10:20.793 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:20.779+0000 7f2b20ad3700 1 -- 172.21.15.42:0/1802948531 shutdown_connections 2019-12-05T06:10:20.794 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:20.779+0000 7f2b20ad3700 1 -- 172.21.15.42:0/1802948531 wait complete. 2019-12-05T06:10:20.794 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:20.779+0000 7f2b20ad3700 1 -- 172.21.15.42:0/1802948531 >> 172.21.15.42:0/1802948531 conn(0x7f2b1c0aabe0 msgr2=0x7f2b1c0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:10:20.798 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:20.779+0000 7f2b20ad3700 1 Processor -- start 2019-12-05T06:10:20.799 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:20.779+0000 7f2b20ad3700 1 -- start start 2019-12-05T06:10:20.799 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:20.783+0000 7f2b20ad3700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2b1c07cc00 0x7f2b1c07d030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:10:20.799 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:20.783+0000 7f2b20ad3700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2b1c060010 0x7f2b1c060440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:10:20.799 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:20.783+0000 7f2b20ad3700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b1c144b70 con 0x7f2b1c0a3a60 2019-12-05T06:10:20.799 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:20.783+0000 7f2b20ad3700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b1c144e80 con 0x7f2b1c07cc00 2019-12-05T06:10:20.800 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:20.783+0000 7f2b20ad3700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b1c145120 con 0x7f2b1c060010 2019-12-05T06:10:20.800 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:20.783+0000 7f2b1b59e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2b1c060010 0x7f2b1c060440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:10:20.800 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:20.783+0000 7f2b1ad9d700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f2b1c0a3a60 0x7f2b1c1441c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:49750/0 (socket says 172.21.15.42:49750) 2019-12-05T06:10:20.800 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:20.783+0000 7f2b1ad9d700 1 -- 172.21.15.42:0/3352814122 learned_addr learned my addr 172.21.15.42:0/3352814122 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:10:20.800 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:20.783+0000 7f2b1a59c700 1 --2- 172.21.15.42:0/3352814122 >> v2:172.21.15.40:3300/0 conn(0x7f2b1c07cc00 0x7f2b1c07d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:10:20.800 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:20.783+0000 7f2b1b59e700 1 -- 172.21.15.42:0/3352814122 >> v2:172.21.15.40:3300/0 conn(0x7f2b1c07cc00 msgr2=0x7f2b1c07d030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:10:20.801 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:20.783+0000 7f2b1b59e700 1 --2- 172.21.15.42:0/3352814122 >> v2:172.21.15.40:3300/0 conn(0x7f2b1c07cc00 0x7f2b1c07d030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:10:20.801 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:20.783+0000 7f2b1b59e700 1 -- 172.21.15.42:0/3352814122 >> v1:172.21.15.42:6789/0 conn(0x7f2b1c0a3a60 legacy=0x7f2b1c1441c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:10:20.801 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:20.783+0000 7f2b0b7fe700 1 -- 172.21.15.42:0/3352814122 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2685679267 0 0) 0x7f2b1c144b70 con 0x7f2b1c0a3a60 2019-12-05T06:10:20.801 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:20.783+0000 7f2b1b59e700 1 -- 172.21.15.42:0/3352814122 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b1c060990 con 0x7f2b1c060010 2019-12-05T06:10:20.801 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:20.783+0000 7f2b1b59e700 1 --2- 172.21.15.42:0/3352814122 >> v2:172.21.15.42:3300/0 conn(0x7f2b1c060010 0x7f2b1c060440 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f2b140032b0 tx=0x7f2b14010d00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:20.801 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:20.783+0000 7f2b1b59e700 1 --2- 172.21.15.42:0/3352814122 >> v2:172.21.15.42:3300/0 conn(0x7f2b1c060010 0x7f2b1c060440 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f2b140032b0 tx=0x7f2b14010d00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:20.801 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:20.783+0000 7f2b1b59e700 1 --2- 172.21.15.42:0/3352814122 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2b1c060010 0x7f2b1c060440 secure :-1 s=READY pgs=2167 cs=0 l=1 rx=0x7f2b140032b0 tx=0x7f2b14010d00).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:10:20.802 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:20.783+0000 7f2b1b59e700 1 --2- 172.21.15.42:0/3352814122 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2b1c060010 0x7f2b1c060440 secure :-1 s=THROTTLE_DONE pgs=2167 cs=0 l=1 rx=0x7f2b140032b0 tx=0x7f2b14010d00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:20.802 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:20.783+0000 7f2b0b7fe700 1 -- 172.21.15.42:0/3352814122 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2b14005d40 con 0x7f2b1c060010 2019-12-05T06:10:20.802 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:20.783+0000 7f2b1b59e700 1 --2- 172.21.15.42:0/3352814122 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2b1c060010 0x7f2b1c060440 secure :-1 s=THROTTLE_DONE pgs=2167 cs=0 l=1 rx=0x7f2b140032b0 tx=0x7f2b14010d00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:20.802 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:20.783+0000 7f2b1b59e700 1 --2- 172.21.15.42:0/3352814122 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2b1c060010 0x7f2b1c060440 secure :-1 s=THROTTLE_DONE pgs=2167 cs=0 l=1 rx=0x7f2b140032b0 tx=0x7f2b14010d00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:20.802 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:20.783+0000 7f2b0b7fe700 1 -- 172.21.15.42:0/3352814122 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b14005960 con 0x7f2b1c060010 2019-12-05T06:10:20.802 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:20.783+0000 7f2b20ad3700 1 -- 172.21.15.42:0/3352814122 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b1c1389e0 con 0x7f2b1c060010 2019-12-05T06:10:20.802 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:20.783+0000 7f2b0b7fe700 1 -- 172.21.15.42:0/3352814122 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2b14012c10 con 0x7f2b1c060010 2019-12-05T06:10:20.808 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:20.783+0000 7f2b20ad3700 1 -- 172.21.15.42:0/3352814122 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b1c138ee0 con 0x7f2b1c060010 2019-12-05T06:10:20.811 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:20.783+0000 7f2b1b59e700 1 --2- 172.21.15.42:0/3352814122 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2b1c060010 0x7f2b1c060440 secure :-1 s=THROTTLE_DONE pgs=2167 cs=0 l=1 rx=0x7f2b140032b0 tx=0x7f2b14010d00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:20.811 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:20.783+0000 7f2b0b7fe700 1 -- 172.21.15.42:0/3352814122 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f2b140093b0 con 0x7f2b1c060010 2019-12-05T06:10:20.811 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:20.783+0000 7f2b1b59e700 1 --2- 172.21.15.42:0/3352814122 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2b1c060010 0x7f2b1c060440 secure :-1 s=THROTTLE_DONE pgs=2167 cs=0 l=1 rx=0x7f2b140032b0 tx=0x7f2b14010d00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:20.812 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:20.783+0000 7f2b0b7fe700 1 --2- 172.21.15.42:0/3352814122 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2b0003ac30 0x7f2b0003d140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:10:20.812 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:20.783+0000 7f2b0b7fe700 1 -- 172.21.15.42:0/3352814122 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9987+0+0 (secure 0 0 0) 0x7f2b1404bd20 con 0x7f2b1c060010 2019-12-05T06:10:20.813 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:20.787+0000 7f2b1ad9d700 1 --2- 172.21.15.42:0/3352814122 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2b0003ac30 0x7f2b0003d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:10:20.814 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:20.787+0000 7f2b1ad9d700 1 --2- 172.21.15.42:0/3352814122 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2b0003ac30 0x7f2b0003d140 crc :-1 s=READY pgs=911 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:10:20.814 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:20.791+0000 7f2b20ad3700 1 -- 172.21.15.42:0/3352814122 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b1c05d110 con 0x7f2b1c060010 2019-12-05T06:10:20.815 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:20.795+0000 7f2b1b59e700 1 --2- 172.21.15.42:0/3352814122 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2b1c060010 0x7f2b1c060440 secure :-1 s=THROTTLE_DONE pgs=2167 cs=0 l=1 rx=0x7f2b140032b0 tx=0x7f2b14010d00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:20.815 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:20.795+0000 7f2b0b7fe700 1 -- 172.21.15.42:0/3352814122 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f2b14032da0 con 0x7f2b1c060010 2019-12-05T06:10:20.980 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:20.975+0000 7f2b20ad3700 1 -- 172.21.15.42:0/3352814122 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2b1c0a8be0 con 0x7f2b1c060010 2019-12-05T06:10:20.981 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:20.975+0000 7f2b1b59e700 1 --2- 172.21.15.42:0/3352814122 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2b1c060010 0x7f2b1c060440 secure :-1 s=THROTTLE_DONE pgs=2167 cs=0 l=1 rx=0x7f2b140032b0 tx=0x7f2b14010d00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:20.981 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:20.975+0000 7f2b0b7fe700 1 -- 172.21.15.42:0/3352814122 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 262 v262) v1 ==== 95+0+1775 (secure 0 0 0) 0x7f2b14033000 con 0x7f2b1c060010 2019-12-05T06:10:20.981 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 262 2019-12-05T06:10:20.982 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:10:20.982 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":262,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":262,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:10:14.933627+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":95,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6977},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6977":{"gid":6977,"name":"b","rank":0,"incarnation":259,"state":"up:active","state_seq":32,"addr":"172.21.15.40:6833/3355679996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3355679996},{"type":"v1","addr":"172.21.15.40:6833","nonce":3355679996}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:10:20.985 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:20.979+0000 7f2b20ad3700 1 -- 172.21.15.42:0/3352814122 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2b0003ac30 msgr2=0x7f2b0003d140 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:10:20.985 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:20.979+0000 7f2b20ad3700 1 --2- 172.21.15.42:0/3352814122 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2b0003ac30 0x7f2b0003d140 crc :-1 s=READY pgs=911 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:10:20.985 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:20.979+0000 7f2b20ad3700 1 -- 172.21.15.42:0/3352814122 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2b1c060010 msgr2=0x7f2b1c060440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:10:20.985 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:20.979+0000 7f2b20ad3700 1 --2- 172.21.15.42:0/3352814122 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2b1c060010 0x7f2b1c060440 secure :-1 s=READY pgs=2167 cs=0 l=1 rx=0x7f2b140032b0 tx=0x7f2b14010d00).stop 2019-12-05T06:10:20.985 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:20.979+0000 7f2b20ad3700 1 -- 172.21.15.42:0/3352814122 shutdown_connections 2019-12-05T06:10:20.986 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:20.979+0000 7f2b20ad3700 1 --2- 172.21.15.42:0/3352814122 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2b0003ac30 0x7f2b0003d140 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:10:20.986 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:20.979+0000 7f2b20ad3700 1 --2- 172.21.15.42:0/3352814122 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2b1c060010 0x7f2b1c060440 unknown :-1 s=CLOSED pgs=2167 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:10:20.986 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:20.979+0000 7f2b20ad3700 1 --2- 172.21.15.42:0/3352814122 >> v2:172.21.15.40:3300/0 conn(0x7f2b1c07cc00 0x7f2b1c07d030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:10:20.987 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:20.979+0000 7f2b20ad3700 1 -- 172.21.15.42:0/3352814122 shutdown_connections 2019-12-05T06:10:20.987 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:20.979+0000 7f2b20ad3700 1 -- 172.21.15.42:0/3352814122 wait complete. 2019-12-05T06:10:20.987 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:20.979+0000 7f2b20ad3700 1 -- 172.21.15.42:0/3352814122 >> 172.21.15.42:0/3352814122 conn(0x7f2b1c07e4f0 msgr2=0x7f2b1c11c950 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:10:20.996 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2019-12-05T06:10:22.998 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:10:23.117 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:23.103+0000 7f38042c5700 1 Processor -- start 2019-12-05T06:10:23.118 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:23.103+0000 7f38042c5700 1 -- start start 2019-12-05T06:10:23.118 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:23.103+0000 7f38042c5700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f37fc0a7740 0x7f37fc08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:10:23.118 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:23.103+0000 7f38042c5700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f37fc07e720 0x7f37fc07eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:10:23.118 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:23.103+0000 7f38042c5700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37fc0a0710 con 0x7f37fc09f2e0 2019-12-05T06:10:23.118 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:23.103+0000 7f38042c5700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37fc0a0b00 con 0x7f37fc07e720 2019-12-05T06:10:23.119 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:23.103+0000 7f38042c5700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37fc08e5b0 con 0x7f37fc0a7740 2019-12-05T06:10:23.119 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:23.107+0000 7f3802061700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f37fc07e720 0x7f37fc07eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:10:23.119 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:23.107+0000 7f3802061700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f37fc07e720 0x7f37fc07eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:49730/0 (socket says 172.21.15.42:49730) 2019-12-05T06:10:23.119 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:23.107+0000 7f3802061700 1 -- 172.21.15.42:0/2478127545 learned_addr learned my addr 172.21.15.42:0/2478127545 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:10:23.119 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:23.107+0000 7f3802862700 1 --2- 172.21.15.42:0/2478127545 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f37fc0a7740 0x7f37fc08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:10:23.119 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:23.107+0000 7f3802061700 1 -- 172.21.15.42:0/2478127545 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f37fc0a7740 msgr2=0x7f37fc08df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:10:23.122 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:23.107+0000 7f3802061700 1 --2- 172.21.15.42:0/2478127545 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f37fc0a7740 0x7f37fc08df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:10:23.122 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:23.107+0000 7f3802061700 1 -- 172.21.15.42:0/2478127545 >> v1:172.21.15.40:6789/0 conn(0x7f37fc09f2e0 legacy=0x7f37fc09f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:10:23.123 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:23.107+0000 7f3802061700 1 -- 172.21.15.42:0/2478127545 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37fc07f0a0 con 0x7f37fc07e720 2019-12-05T06:10:23.123 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:23.107+0000 7f3802061700 1 --2- 172.21.15.42:0/2478127545 >> v2:172.21.15.42:3300/0 conn(0x7f37fc07e720 0x7f37fc07eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f37ec008790 tx=0x7f37ec00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:23.123 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:23.107+0000 7f3802061700 1 --2- 172.21.15.42:0/2478127545 >> v2:172.21.15.42:3300/0 conn(0x7f37fc07e720 0x7f37fc07eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f37ec008790 tx=0x7f37ec00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:23.123 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:23.107+0000 7f3802061700 1 --2- 172.21.15.42:0/2478127545 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f37fc07e720 0x7f37fc07eb50 secure :-1 s=READY pgs=2168 cs=0 l=1 rx=0x7f37ec008790 tx=0x7f37ec00fd30).ready entity=mon.1 client_cookie=9936cc73d478de3c server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:10:23.123 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:23.107+0000 7f3802061700 1 --2- 172.21.15.42:0/2478127545 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f37fc07e720 0x7f37fc07eb50 secure :-1 s=THROTTLE_DONE pgs=2168 cs=0 l=1 rx=0x7f37ec008790 tx=0x7f37ec00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:23.123 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:23.107+0000 7f3801860700 1 -- 172.21.15.42:0/2478127545 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f37ec0113b0 con 0x7f37fc07e720 2019-12-05T06:10:23.124 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:23.107+0000 7f3802061700 1 --2- 172.21.15.42:0/2478127545 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f37fc07e720 0x7f37fc07eb50 secure :-1 s=THROTTLE_DONE pgs=2168 cs=0 l=1 rx=0x7f37ec008790 tx=0x7f37ec00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:23.124 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:23.107+0000 7f3802061700 1 --2- 172.21.15.42:0/2478127545 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f37fc07e720 0x7f37fc07eb50 secure :-1 s=THROTTLE_DONE pgs=2168 cs=0 l=1 rx=0x7f37ec008790 tx=0x7f37ec00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:23.124 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:23.107+0000 7f3801860700 1 -- 172.21.15.42:0/2478127545 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37ec008bb0 con 0x7f37fc07e720 2019-12-05T06:10:23.124 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:23.107+0000 7f3801860700 1 -- 172.21.15.42:0/2478127545 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f37ec011990 con 0x7f37fc07e720 2019-12-05T06:10:23.124 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:23.107+0000 7f38042c5700 1 -- 172.21.15.42:0/2478127545 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f37fc07e720 msgr2=0x7f37fc07eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:10:23.124 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:23.107+0000 7f38042c5700 1 --2- 172.21.15.42:0/2478127545 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f37fc07e720 0x7f37fc07eb50 secure :-1 s=READY pgs=2168 cs=0 l=1 rx=0x7f37ec008790 tx=0x7f37ec00fd30).stop 2019-12-05T06:10:23.125 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:23.107+0000 7f38042c5700 1 -- 172.21.15.42:0/2478127545 shutdown_connections 2019-12-05T06:10:23.125 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:23.107+0000 7f38042c5700 1 --2- 172.21.15.42:0/2478127545 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f37fc0a7740 0x7f37fc08df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:10:23.125 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:23.107+0000 7f38042c5700 1 --2- 172.21.15.42:0/2478127545 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f37fc07e720 0x7f37fc07eb50 unknown :-1 s=CLOSED pgs=2168 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:10:23.125 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:23.107+0000 7f38042c5700 1 -- 172.21.15.42:0/2478127545 shutdown_connections 2019-12-05T06:10:23.125 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:23.107+0000 7f38042c5700 1 -- 172.21.15.42:0/2478127545 wait complete. 2019-12-05T06:10:23.126 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:23.107+0000 7f38042c5700 1 -- 172.21.15.42:0/2478127545 >> 172.21.15.42:0/2478127545 conn(0x7f37fc0aabe0 msgr2=0x7f37fc0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:10:23.126 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:23.111+0000 7f38042c5700 1 Processor -- start 2019-12-05T06:10:23.126 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:23.111+0000 7f38042c5700 1 -- start start 2019-12-05T06:10:23.129 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:23.111+0000 7f38042c5700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f37fc124350 0x7f37fc131b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:10:23.130 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:23.111+0000 7f38042c5700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f37fc1320c0 0x7f37fc11f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:10:23.130 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:23.111+0000 7f38042c5700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f37fc11fd40 0x7f37fc08eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:10:23.130 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:23.111+0000 7f38042c5700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37fc1324f0 con 0x7f37fc11fd40 2019-12-05T06:10:23.130 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:23.111+0000 7f38042c5700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37fc12e070 con 0x7f37fc1320c0 2019-12-05T06:10:23.130 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:23.111+0000 7f38042c5700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37fc131630 con 0x7f37fc124350 2019-12-05T06:10:23.130 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:23.111+0000 7f3802061700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f37fc1320c0 0x7f37fc11f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:10:23.131 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:23.111+0000 7f3802061700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f37fc1320c0 0x7f37fc11f7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:49736/0 (socket says 172.21.15.42:49736) 2019-12-05T06:10:23.131 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:23.111+0000 7f3802061700 1 -- 172.21.15.42:0/1746539296 learned_addr learned my addr 172.21.15.42:0/1746539296 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:10:23.131 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:23.111+0000 7f3803063700 1 --2- 172.21.15.42:0/1746539296 >> v2:172.21.15.40:3300/0 conn(0x7f37fc11fd40 0x7f37fc08eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:10:23.131 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:23.111+0000 7f3802862700 1 --2- 172.21.15.42:0/1746539296 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f37fc124350 0x7f37fc131b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:10:23.131 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:23.111+0000 7f3802061700 1 -- 172.21.15.42:0/1746539296 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f37fc124350 msgr2=0x7f37fc131b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:10:23.131 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:23.111+0000 7f3802061700 1 --2- 172.21.15.42:0/1746539296 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f37fc124350 0x7f37fc131b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:10:23.132 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:23.111+0000 7f3802061700 1 -- 172.21.15.42:0/1746539296 >> v2:172.21.15.40:3300/0 conn(0x7f37fc11fd40 msgr2=0x7f37fc08eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:10:23.141 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:23.111+0000 7f3802061700 1 --2- 172.21.15.42:0/1746539296 >> v2:172.21.15.40:3300/0 conn(0x7f37fc11fd40 0x7f37fc08eb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:10:23.141 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:23.111+0000 7f3802061700 1 -- 172.21.15.42:0/1746539296 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37fc120170 con 0x7f37fc1320c0 2019-12-05T06:10:23.141 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:23.111+0000 7f3802061700 1 --2- 172.21.15.42:0/1746539296 >> v2:172.21.15.42:3300/0 conn(0x7f37fc1320c0 0x7f37fc11f7f0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f37fc12e230 tx=0x7f37ec008c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:23.142 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:23.111+0000 7f3802061700 1 --2- 172.21.15.42:0/1746539296 >> v2:172.21.15.42:3300/0 conn(0x7f37fc1320c0 0x7f37fc11f7f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f37fc12e230 tx=0x7f37ec008c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:23.142 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:23.111+0000 7f3802061700 1 --2- 172.21.15.42:0/1746539296 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f37fc1320c0 0x7f37fc11f7f0 secure :-1 s=READY pgs=2169 cs=0 l=1 rx=0x7f37fc12e230 tx=0x7f37ec008c30).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:10:23.142 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:23.115+0000 7f3802061700 1 --2- 172.21.15.42:0/1746539296 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f37fc1320c0 0x7f37fc11f7f0 secure :-1 s=THROTTLE_DONE pgs=2169 cs=0 l=1 rx=0x7f37fc12e230 tx=0x7f37ec008c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:23.142 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:23.115+0000 7f3802061700 1 --2- 172.21.15.42:0/1746539296 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f37fc1320c0 0x7f37fc11f7f0 secure :-1 s=THROTTLE_DONE pgs=2169 cs=0 l=1 rx=0x7f37fc12e230 tx=0x7f37ec008c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:23.142 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:23.115+0000 7f3802061700 1 --2- 172.21.15.42:0/1746539296 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f37fc1320c0 0x7f37fc11f7f0 secure :-1 s=THROTTLE_DONE pgs=2169 cs=0 l=1 rx=0x7f37fc12e230 tx=0x7f37ec008c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:23.143 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:23.115+0000 7f37f37fe700 1 -- 172.21.15.42:0/1746539296 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f37ec007150 con 0x7f37fc1320c0 2019-12-05T06:10:23.143 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:23.115+0000 7f37f37fe700 1 -- 172.21.15.42:0/1746539296 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37ec0115b0 con 0x7f37fc1320c0 2019-12-05T06:10:23.143 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:23.115+0000 7f38042c5700 1 -- 172.21.15.42:0/1746539296 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37fc0a6dc0 con 0x7f37fc1320c0 2019-12-05T06:10:23.143 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:23.115+0000 7f37f37fe700 1 -- 172.21.15.42:0/1746539296 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f37ec005730 con 0x7f37fc1320c0 2019-12-05T06:10:23.143 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:23.115+0000 7f38042c5700 1 -- 172.21.15.42:0/1746539296 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37fc12ee60 con 0x7f37fc1320c0 2019-12-05T06:10:23.143 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:23.115+0000 7f3802061700 1 --2- 172.21.15.42:0/1746539296 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f37fc1320c0 0x7f37fc11f7f0 secure :-1 s=THROTTLE_DONE pgs=2169 cs=0 l=1 rx=0x7f37fc12e230 tx=0x7f37ec008c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:23.143 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:23.115+0000 7f37f37fe700 1 -- 172.21.15.42:0/1746539296 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f37ec005a40 con 0x7f37fc1320c0 2019-12-05T06:10:23.144 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:23.115+0000 7f3802061700 1 --2- 172.21.15.42:0/1746539296 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f37fc1320c0 0x7f37fc11f7f0 secure :-1 s=THROTTLE_DONE pgs=2169 cs=0 l=1 rx=0x7f37fc12e230 tx=0x7f37ec008c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:23.144 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:23.115+0000 7f37f37fe700 1 --2- 172.21.15.42:0/1746539296 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f37e403ae90 0x7f37e403d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:10:23.144 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:23.115+0000 7f37f37fe700 1 -- 172.21.15.42:0/1746539296 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9987+0+0 (secure 0 0 0) 0x7f37ec04b520 con 0x7f37fc1320c0 2019-12-05T06:10:23.144 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:23.115+0000 7f3802862700 1 --2- 172.21.15.42:0/1746539296 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f37e403ae90 0x7f37e403d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:10:23.144 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:23.115+0000 7f3802862700 1 --2- 172.21.15.42:0/1746539296 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f37e403ae90 0x7f37e403d3a0 crc :-1 s=READY pgs=912 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:10:23.144 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:23.119+0000 7f38042c5700 1 -- 172.21.15.42:0/1746539296 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37fc05d110 con 0x7f37fc1320c0 2019-12-05T06:10:23.145 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:23.123+0000 7f3802061700 1 --2- 172.21.15.42:0/1746539296 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f37fc1320c0 0x7f37fc11f7f0 secure :-1 s=THROTTLE_DONE pgs=2169 cs=0 l=1 rx=0x7f37fc12e230 tx=0x7f37ec008c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:23.145 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:23.123+0000 7f37f37fe700 1 -- 172.21.15.42:0/1746539296 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f37ec005d50 con 0x7f37fc1320c0 2019-12-05T06:10:23.299 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:23.291+0000 7f38042c5700 1 -- 172.21.15.42:0/1746539296 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f37fc05b8c0 con 0x7f37fc1320c0 2019-12-05T06:10:23.300 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:23.295+0000 7f3802061700 1 --2- 172.21.15.42:0/1746539296 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f37fc1320c0 0x7f37fc11f7f0 secure :-1 s=THROTTLE_DONE pgs=2169 cs=0 l=1 rx=0x7f37fc12e230 tx=0x7f37ec008c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:23.300 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:23.295+0000 7f37f37fe700 1 -- 172.21.15.42:0/1746539296 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 262 v262) v1 ==== 95+0+1775 (secure 0 0 0) 0x7f37ec0301f0 con 0x7f37fc1320c0 2019-12-05T06:10:23.301 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 262 2019-12-05T06:10:23.301 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:10:23.302 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":262,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":262,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:10:14.933627+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":95,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6977},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6977":{"gid":6977,"name":"b","rank":0,"incarnation":259,"state":"up:active","state_seq":32,"addr":"172.21.15.40:6833/3355679996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3355679996},{"type":"v1","addr":"172.21.15.40:6833","nonce":3355679996}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:10:23.303 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:23.295+0000 7f38042c5700 1 -- 172.21.15.42:0/1746539296 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f37e403ae90 msgr2=0x7f37e403d3a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:10:23.303 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:23.295+0000 7f38042c5700 1 --2- 172.21.15.42:0/1746539296 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f37e403ae90 0x7f37e403d3a0 crc :-1 s=READY pgs=912 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:10:23.303 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:23.295+0000 7f38042c5700 1 -- 172.21.15.42:0/1746539296 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f37fc1320c0 msgr2=0x7f37fc11f7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:10:23.304 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:23.295+0000 7f38042c5700 1 --2- 172.21.15.42:0/1746539296 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f37fc1320c0 0x7f37fc11f7f0 secure :-1 s=READY pgs=2169 cs=0 l=1 rx=0x7f37fc12e230 tx=0x7f37ec008c30).stop 2019-12-05T06:10:23.304 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:23.295+0000 7f38042c5700 1 -- 172.21.15.42:0/1746539296 shutdown_connections 2019-12-05T06:10:23.304 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:23.295+0000 7f38042c5700 1 --2- 172.21.15.42:0/1746539296 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f37e403ae90 0x7f37e403d3a0 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:10:23.304 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:23.295+0000 7f38042c5700 1 --2- 172.21.15.42:0/1746539296 >> v2:172.21.15.40:3300/0 conn(0x7f37fc11fd40 0x7f37fc08eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:10:23.304 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:23.295+0000 7f38042c5700 1 --2- 172.21.15.42:0/1746539296 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f37fc124350 0x7f37fc131b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:10:23.305 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:23.295+0000 7f38042c5700 1 --2- 172.21.15.42:0/1746539296 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f37fc1320c0 0x7f37fc11f7f0 unknown :-1 s=CLOSED pgs=2169 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:10:23.305 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:23.295+0000 7f38042c5700 1 -- 172.21.15.42:0/1746539296 shutdown_connections 2019-12-05T06:10:23.305 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:23.295+0000 7f38042c5700 1 -- 172.21.15.42:0/1746539296 wait complete. 2019-12-05T06:10:23.305 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:23.295+0000 7f38042c5700 1 -- 172.21.15.42:0/1746539296 >> 172.21.15.42:0/1746539296 conn(0x7f37fc0a9870 msgr2=0x7f37fc11c610 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:10:23.312 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2019-12-05T06:10:25.313 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:10:25.430 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:25.415+0000 7f0dda729700 1 Processor -- start 2019-12-05T06:10:25.431 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:25.415+0000 7f0dda729700 1 -- start start 2019-12-05T06:10:25.431 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:25.415+0000 7f0dda729700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0dd408df80 0x7f0dd409f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:10:25.431 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:25.415+0000 7f0dda729700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0dd409fc60 con 0x7f0dd409f830 2019-12-05T06:10:25.431 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:25.415+0000 7f0dda729700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0dd40a0710 con 0x7f0dd40a7740 2019-12-05T06:10:25.431 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:25.415+0000 7f0dda729700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0dd40a0b00 con 0x7f0dd408df80 2019-12-05T06:10:25.431 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:25.415+0000 7f0dd3fff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0dd408df80 0x7f0dd409f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:10:25.431 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:25.415+0000 7f0dd3fff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0dd408df80 0x7f0dd409f2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:49756/0 (socket says 172.21.15.42:49756) 2019-12-05T06:10:25.432 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:25.415+0000 7f0dd3fff700 1 -- 172.21.15.42:0/3639779273 learned_addr learned my addr 172.21.15.42:0/3639779273 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:10:25.432 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:25.419+0000 7f0dd3fff700 1 -- 172.21.15.42:0/3639779273 >> v1:172.21.15.42:6789/0 conn(0x7f0dd40a7740 legacy=0x7f0dd407e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:10:25.432 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:25.419+0000 7f0dd37fe700 1 -- 172.21.15.42:0/3639779273 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1081345244 0 0) 0x7f0dd40a0710 con 0x7f0dd40a7740 2019-12-05T06:10:25.432 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:25.419+0000 7f0dd3fff700 1 -- 172.21.15.42:0/3639779273 >> v1:172.21.15.40:6789/0 conn(0x7f0dd409f830 legacy=0x7f0dd414efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:10:25.432 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:25.419+0000 7f0dd3fff700 1 -- 172.21.15.42:0/3639779273 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dd407ef20 con 0x7f0dd408df80 2019-12-05T06:10:25.432 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:25.419+0000 7f0dd3fff700 1 --2- 172.21.15.42:0/3639779273 >> v2:172.21.15.42:3300/0 conn(0x7f0dd408df80 0x7f0dd409f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f0dc0008790 tx=0x7f0dc000fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:25.432 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:25.419+0000 7f0dd3fff700 1 --2- 172.21.15.42:0/3639779273 >> v2:172.21.15.42:3300/0 conn(0x7f0dd408df80 0x7f0dd409f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f0dc0008790 tx=0x7f0dc000fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:25.433 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:25.419+0000 7f0dd3fff700 1 --2- 172.21.15.42:0/3639779273 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0dd408df80 0x7f0dd409f2e0 secure :-1 s=READY pgs=2171 cs=0 l=1 rx=0x7f0dc0008790 tx=0x7f0dc000fcb0).ready entity=mon.1 client_cookie=dfcadb767ef489cc server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:10:25.433 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:25.419+0000 7f0dd3fff700 1 --2- 172.21.15.42:0/3639779273 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0dd408df80 0x7f0dd409f2e0 secure :-1 s=THROTTLE_DONE pgs=2171 cs=0 l=1 rx=0x7f0dc0008790 tx=0x7f0dc000fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:25.433 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:25.419+0000 7f0dd37fe700 1 -- 172.21.15.42:0/3639779273 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0dc0011520 con 0x7f0dd408df80 2019-12-05T06:10:25.434 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:25.419+0000 7f0dd3fff700 1 --2- 172.21.15.42:0/3639779273 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0dd408df80 0x7f0dd409f2e0 secure :-1 s=THROTTLE_DONE pgs=2171 cs=0 l=1 rx=0x7f0dc0008790 tx=0x7f0dc000fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:25.434 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:25.419+0000 7f0dd3fff700 1 --2- 172.21.15.42:0/3639779273 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0dd408df80 0x7f0dd409f2e0 secure :-1 s=THROTTLE_DONE pgs=2171 cs=0 l=1 rx=0x7f0dc0008790 tx=0x7f0dc000fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:25.434 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:25.419+0000 7f0dd37fe700 1 -- 172.21.15.42:0/3639779273 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0dc0008ca0 con 0x7f0dd408df80 2019-12-05T06:10:25.435 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:25.419+0000 7f0dd37fe700 1 -- 172.21.15.42:0/3639779273 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0dc0011b00 con 0x7f0dd408df80 2019-12-05T06:10:25.435 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:25.419+0000 7f0dda729700 1 -- 172.21.15.42:0/3639779273 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0dd408df80 msgr2=0x7f0dd409f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:10:25.435 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:25.419+0000 7f0dda729700 1 --2- 172.21.15.42:0/3639779273 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0dd408df80 0x7f0dd409f2e0 secure :-1 s=READY pgs=2171 cs=0 l=1 rx=0x7f0dc0008790 tx=0x7f0dc000fcb0).stop 2019-12-05T06:10:25.435 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:25.419+0000 7f0dda729700 1 -- 172.21.15.42:0/3639779273 shutdown_connections 2019-12-05T06:10:25.435 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:25.419+0000 7f0dda729700 1 --2- 172.21.15.42:0/3639779273 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0dd408df80 0x7f0dd409f2e0 unknown :-1 s=CLOSED pgs=2171 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:10:25.437 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:25.419+0000 7f0dda729700 1 -- 172.21.15.42:0/3639779273 shutdown_connections 2019-12-05T06:10:25.437 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:25.419+0000 7f0dda729700 1 -- 172.21.15.42:0/3639779273 wait complete. 2019-12-05T06:10:25.437 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:25.419+0000 7f0dda729700 1 -- 172.21.15.42:0/3639779273 >> 172.21.15.42:0/3639779273 conn(0x7f0dd40aabe0 msgr2=0x7f0dd40a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:10:25.438 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:25.419+0000 7f0dda729700 1 Processor -- start 2019-12-05T06:10:25.438 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:25.419+0000 7f0dda729700 1 -- start start 2019-12-05T06:10:25.438 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:25.423+0000 7f0dda729700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0dd411dcc0 0x7f0dd40a66c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:10:25.438 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:25.423+0000 7f0dda729700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f0dd411a470 0x7f0dd411a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:10:25.438 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:25.423+0000 7f0dda729700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0dd411c2f0 con 0x7f0dd406e2a0 2019-12-05T06:10:25.438 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:25.423+0000 7f0dda729700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0dd40a6ce0 con 0x7f0dd411a470 2019-12-05T06:10:25.438 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:25.423+0000 7f0dda729700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0dd40a6f80 con 0x7f0dd411dcc0 2019-12-05T06:10:25.439 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:25.423+0000 7f0dd8cc6700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0dd411dcc0 0x7f0dd40a66c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:10:25.439 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:25.423+0000 7f0dd8cc6700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0dd411dcc0 0x7f0dd40a66c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:49760/0 (socket says 172.21.15.42:49760) 2019-12-05T06:10:25.439 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:25.423+0000 7f0dd8cc6700 1 -- 172.21.15.42:0/1163953079 learned_addr learned my addr 172.21.15.42:0/1163953079 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:10:25.439 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:25.423+0000 7f0dd3fff700 1 --2- 172.21.15.42:0/1163953079 >> v2:172.21.15.40:3300/0 conn(0x7f0dd411a470 0x7f0dd411a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:10:25.439 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:25.423+0000 7f0dd8cc6700 1 -- 172.21.15.42:0/1163953079 >> v2:172.21.15.40:3300/0 conn(0x7f0dd411a470 msgr2=0x7f0dd411a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:10:25.439 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:25.423+0000 7f0dd8cc6700 1 --2- 172.21.15.42:0/1163953079 >> v2:172.21.15.40:3300/0 conn(0x7f0dd411a470 0x7f0dd411a8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:10:25.439 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:25.423+0000 7f0dd8cc6700 1 -- 172.21.15.42:0/1163953079 >> v1:172.21.15.40:6789/0 conn(0x7f0dd406e2a0 legacy=0x7f0dd406e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:10:25.440 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:25.423+0000 7f0dd8cc6700 1 -- 172.21.15.42:0/1163953079 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dd411adf0 con 0x7f0dd411dcc0 2019-12-05T06:10:25.440 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:25.423+0000 7f0dd8cc6700 1 --2- 172.21.15.42:0/1163953079 >> v2:172.21.15.42:3300/0 conn(0x7f0dd411dcc0 0x7f0dd40a66c0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f0dc8001460 tx=0x7f0dc80109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:25.440 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:25.423+0000 7f0dd8cc6700 1 --2- 172.21.15.42:0/1163953079 >> v2:172.21.15.42:3300/0 conn(0x7f0dd411dcc0 0x7f0dd40a66c0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f0dc8001460 tx=0x7f0dc80109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:25.440 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:25.423+0000 7f0dd8cc6700 1 --2- 172.21.15.42:0/1163953079 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0dd411dcc0 0x7f0dd40a66c0 secure :-1 s=READY pgs=2172 cs=0 l=1 rx=0x7f0dc8001460 tx=0x7f0dc80109a0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:10:25.440 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:25.423+0000 7f0dd8cc6700 1 --2- 172.21.15.42:0/1163953079 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0dd411dcc0 0x7f0dd40a66c0 secure :-1 s=THROTTLE_DONE pgs=2172 cs=0 l=1 rx=0x7f0dc8001460 tx=0x7f0dc80109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:25.440 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:25.423+0000 7f0dd8cc6700 1 --2- 172.21.15.42:0/1163953079 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0dd411dcc0 0x7f0dd40a66c0 secure :-1 s=THROTTLE_DONE pgs=2172 cs=0 l=1 rx=0x7f0dc8001460 tx=0x7f0dc80109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:25.440 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:25.423+0000 7f0dd8cc6700 1 --2- 172.21.15.42:0/1163953079 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0dd411dcc0 0x7f0dd40a66c0 secure :-1 s=THROTTLE_DONE pgs=2172 cs=0 l=1 rx=0x7f0dc8001460 tx=0x7f0dc80109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:25.441 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:25.423+0000 7f0dd17fa700 1 -- 172.21.15.42:0/1163953079 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0dc800ca80 con 0x7f0dd411dcc0 2019-12-05T06:10:25.441 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:25.423+0000 7f0dd17fa700 1 -- 172.21.15.42:0/1163953079 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0dc800c6a0 con 0x7f0dd411dcc0 2019-12-05T06:10:25.441 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:25.423+0000 7f0dda729700 1 -- 172.21.15.42:0/1163953079 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0dd411b800 con 0x7f0dd411dcc0 2019-12-05T06:10:25.441 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:25.423+0000 7f0dd17fa700 1 -- 172.21.15.42:0/1163953079 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0dc8011120 con 0x7f0dd411dcc0 2019-12-05T06:10:25.441 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:25.423+0000 7f0dda729700 1 -- 172.21.15.42:0/1163953079 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0dd406f070 con 0x7f0dd411dcc0 2019-12-05T06:10:25.443 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:25.423+0000 7f0dd8cc6700 1 --2- 172.21.15.42:0/1163953079 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0dd411dcc0 0x7f0dd40a66c0 secure :-1 s=THROTTLE_DONE pgs=2172 cs=0 l=1 rx=0x7f0dc8001460 tx=0x7f0dc80109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:25.443 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:25.423+0000 7f0dd17fa700 1 -- 172.21.15.42:0/1163953079 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f0dc8011430 con 0x7f0dd411dcc0 2019-12-05T06:10:25.443 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:25.423+0000 7f0dd8cc6700 1 --2- 172.21.15.42:0/1163953079 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0dd411dcc0 0x7f0dd40a66c0 secure :-1 s=THROTTLE_DONE pgs=2172 cs=0 l=1 rx=0x7f0dc8001460 tx=0x7f0dc80109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:25.444 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:25.427+0000 7f0dd17fa700 1 --2- 172.21.15.42:0/1163953079 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0db403ae10 0x7f0db403d320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:10:25.444 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:25.427+0000 7f0dd17fa700 1 -- 172.21.15.42:0/1163953079 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9987+0+0 (secure 0 0 0) 0x7f0dc804b1c0 con 0x7f0dd411dcc0 2019-12-05T06:10:25.444 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:25.427+0000 7f0dd3fff700 1 --2- 172.21.15.42:0/1163953079 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0db403ae10 0x7f0db403d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:10:25.444 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:25.427+0000 7f0dd3fff700 1 --2- 172.21.15.42:0/1163953079 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0db403ae10 0x7f0db403d320 crc :-1 s=READY pgs=913 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:10:25.444 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:25.431+0000 7f0dda729700 1 -- 172.21.15.42:0/1163953079 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0dd405d110 con 0x7f0dd411dcc0 2019-12-05T06:10:25.445 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:25.435+0000 7f0dd8cc6700 1 --2- 172.21.15.42:0/1163953079 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0dd411dcc0 0x7f0dd40a66c0 secure :-1 s=THROTTLE_DONE pgs=2172 cs=0 l=1 rx=0x7f0dc8001460 tx=0x7f0dc80109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:25.445 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:25.439+0000 7f0dd17fa700 1 -- 172.21.15.42:0/1163953079 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f0dc8032240 con 0x7f0dd411dcc0 2019-12-05T06:10:25.619 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:25.611+0000 7f0dda729700 1 -- 172.21.15.42:0/1163953079 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0dd405b8c0 con 0x7f0dd411dcc0 2019-12-05T06:10:25.619 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:25.611+0000 7f0dd8cc6700 1 --2- 172.21.15.42:0/1163953079 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0dd411dcc0 0x7f0dd40a66c0 secure :-1 s=THROTTLE_DONE pgs=2172 cs=0 l=1 rx=0x7f0dc8001460 tx=0x7f0dc80109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:25.620 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:25.611+0000 7f0dd17fa700 1 -- 172.21.15.42:0/1163953079 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 263 v263) v1 ==== 95+0+2113 (secure 0 0 0) 0x7f0dc80324a0 con 0x7f0dd411dcc0 2019-12-05T06:10:25.620 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 263 2019-12-05T06:10:25.620 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:10:25.620 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":263,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":10000,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/3923117701","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3923117701},{"type":"v1","addr":"172.21.15.42:6835","nonce":3923117701}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":263}],"filesystems":[{"mdsmap":{"epoch":262,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:10:14.933627+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":95,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6977},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6977":{"gid":6977,"name":"b","rank":0,"incarnation":259,"state":"up:active","state_seq":32,"addr":"172.21.15.40:6833/3355679996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3355679996},{"type":"v1","addr":"172.21.15.40:6833","nonce":3355679996}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:10:25.622 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:25.615+0000 7f0dda729700 1 -- 172.21.15.42:0/1163953079 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0db403ae10 msgr2=0x7f0db403d320 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:10:25.623 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:25.615+0000 7f0dda729700 1 --2- 172.21.15.42:0/1163953079 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0db403ae10 0x7f0db403d320 crc :-1 s=READY pgs=913 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:10:25.623 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:25.615+0000 7f0dda729700 1 -- 172.21.15.42:0/1163953079 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0dd411dcc0 msgr2=0x7f0dd40a66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:10:25.623 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:25.615+0000 7f0dda729700 1 --2- 172.21.15.42:0/1163953079 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0dd411dcc0 0x7f0dd40a66c0 secure :-1 s=READY pgs=2172 cs=0 l=1 rx=0x7f0dc8001460 tx=0x7f0dc80109a0).stop 2019-12-05T06:10:25.623 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:25.615+0000 7f0dda729700 1 -- 172.21.15.42:0/1163953079 shutdown_connections 2019-12-05T06:10:25.623 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:25.615+0000 7f0dda729700 1 --2- 172.21.15.42:0/1163953079 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0db403ae10 0x7f0db403d320 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:10:25.624 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:25.615+0000 7f0dda729700 1 --2- 172.21.15.42:0/1163953079 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0dd411dcc0 0x7f0dd40a66c0 unknown :-1 s=CLOSED pgs=2172 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:10:25.625 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:25.615+0000 7f0dda729700 1 --2- 172.21.15.42:0/1163953079 >> v2:172.21.15.40:3300/0 conn(0x7f0dd411a470 0x7f0dd411a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:10:25.626 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:25.615+0000 7f0dda729700 1 -- 172.21.15.42:0/1163953079 shutdown_connections 2019-12-05T06:10:25.626 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:25.615+0000 7f0dda729700 1 -- 172.21.15.42:0/1163953079 wait complete. 2019-12-05T06:10:25.626 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:10:25.615+0000 7f0dda729700 1 -- 172.21.15.42:0/1163953079 >> 172.21.15.42:0/1163953079 conn(0x7f0dd408fde0 msgr2=0x7f0dd407e4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:10:25.633 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a reported in up:standby state 2019-12-05T06:10:25.633 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 49 secs before thrashing 2019-12-05T06:10:44.226 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:10:44.230 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:11:14.280 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:11:14.284 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:11:14.634 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:11:14.768 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:14.758+0000 7f53fda80700 1 Processor -- start 2019-12-05T06:11:14.768 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:14.758+0000 7f53fda80700 1 -- start start 2019-12-05T06:11:14.768 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:14.762+0000 7f53fda80700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f53f80a7740 0x7f53f808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:11:14.769 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:14.762+0000 7f53fda80700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f53f809fae0 0x7f53f807eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:11:14.769 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:14.762+0000 7f53fda80700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53f80a0710 con 0x7f53f807e720 2019-12-05T06:11:14.769 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:14.762+0000 7f53fda80700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53f80a0b00 con 0x7f53f809fae0 2019-12-05T06:11:14.769 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:14.762+0000 7f53fda80700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53f808e5b0 con 0x7f53f80a7740 2019-12-05T06:11:14.769 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:14.762+0000 7f53effff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f53f807e720 0x7f53f809f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:49952/0 (socket says 172.21.15.42:49952) 2019-12-05T06:11:14.769 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:14.762+0000 7f53effff700 1 -- 172.21.15.42:0/606975564 learned_addr learned my addr 172.21.15.42:0/606975564 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:11:14.769 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:14.762+0000 7f53fc81e700 1 --2- 172.21.15.42:0/606975564 >> v2:172.21.15.40:3300/0 conn(0x7f53f809fae0 0x7f53f807eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:11:14.770 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:14.762+0000 7f53f7fff700 1 --2- 172.21.15.42:0/606975564 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f53f80a7740 0x7f53f808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:11:14.770 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:14.762+0000 7f53f77fe700 1 -- 172.21.15.42:0/606975564 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1123611173 0 0) 0x7f53f80a0710 con 0x7f53f807e720 2019-12-05T06:11:14.770 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:14.762+0000 7f53f77fe700 1 -- 172.21.15.42:0/606975564 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53e40030b0 con 0x7f53f807e720 2019-12-05T06:11:14.770 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:14.762+0000 7f53f7fff700 1 -- 172.21.15.42:0/606975564 >> v2:172.21.15.40:3300/0 conn(0x7f53f809fae0 msgr2=0x7f53f807eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:11:14.770 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:14.762+0000 7f53f7fff700 1 --2- 172.21.15.42:0/606975564 >> v2:172.21.15.40:3300/0 conn(0x7f53f809fae0 0x7f53f807eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:11:14.770 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:14.762+0000 7f53f7fff700 1 -- 172.21.15.42:0/606975564 >> v1:172.21.15.42:6789/0 conn(0x7f53f807e720 legacy=0x7f53f809f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:14.770 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:14.762+0000 7f53f7fff700 1 -- 172.21.15.42:0/606975564 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53f807f0a0 con 0x7f53f80a7740 2019-12-05T06:11:14.771 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:14.762+0000 7f53fc81e700 1 --2- 172.21.15.42:0/606975564 >> v2:172.21.15.40:3300/0 conn(0x7f53f809fae0 0x7f53f807eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:11:14.771 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:14.762+0000 7f53f7fff700 1 --2- 172.21.15.42:0/606975564 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f53f80a7740 0x7f53f808df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f53e8008790 tx=0x7f53e800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:14.771 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:14.762+0000 7f53f7fff700 1 --2- 172.21.15.42:0/606975564 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f53f80a7740 0x7f53f808df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f53e8008790 tx=0x7f53e800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:14.771 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:14.762+0000 7f53f7fff700 1 --2- 172.21.15.42:0/606975564 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f53f80a7740 0x7f53f808df80 secure :-1 s=READY pgs=478 cs=0 l=1 rx=0x7f53e8008790 tx=0x7f53e800fc70).ready entity=mon.2 client_cookie=15fba1d69e8de3b4 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:11:14.771 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:14.762+0000 7f53f7fff700 1 --2- 172.21.15.42:0/606975564 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f53f80a7740 0x7f53f808df80 secure :-1 s=THROTTLE_DONE pgs=478 cs=0 l=1 rx=0x7f53e8008790 tx=0x7f53e800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:14.771 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:14.762+0000 7f53f77fe700 1 -- 172.21.15.42:0/606975564 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f53e8008bb0 con 0x7f53f80a7740 2019-12-05T06:11:14.771 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:14.762+0000 7f53f7fff700 1 --2- 172.21.15.42:0/606975564 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f53f80a7740 0x7f53f808df80 secure :-1 s=THROTTLE_DONE pgs=478 cs=0 l=1 rx=0x7f53e8008790 tx=0x7f53e800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:14.772 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:14.762+0000 7f53f77fe700 1 -- 172.21.15.42:0/606975564 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53e80087d0 con 0x7f53f80a7740 2019-12-05T06:11:14.772 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:14.762+0000 7f53f7fff700 1 --2- 172.21.15.42:0/606975564 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f53f80a7740 0x7f53f808df80 secure :-1 s=THROTTLE_DONE pgs=478 cs=0 l=1 rx=0x7f53e8008790 tx=0x7f53e800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:14.772 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:14.762+0000 7f53f77fe700 1 -- 172.21.15.42:0/606975564 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f53e8011860 con 0x7f53f80a7740 2019-12-05T06:11:14.772 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:14.762+0000 7f53fda80700 1 -- 172.21.15.42:0/606975564 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f53f80a7740 msgr2=0x7f53f808df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:14.775 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:14.762+0000 7f53fda80700 1 --2- 172.21.15.42:0/606975564 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f53f80a7740 0x7f53f808df80 secure :-1 s=READY pgs=478 cs=0 l=1 rx=0x7f53e8008790 tx=0x7f53e800fc70).stop 2019-12-05T06:11:14.775 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:14.762+0000 7f53fda80700 1 -- 172.21.15.42:0/606975564 shutdown_connections 2019-12-05T06:11:14.775 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:14.762+0000 7f53fda80700 1 --2- 172.21.15.42:0/606975564 >> v2:172.21.15.40:3300/0 conn(0x7f53f809fae0 0x7f53f807eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:11:14.775 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:14.762+0000 7f53fda80700 1 --2- 172.21.15.42:0/606975564 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f53f80a7740 0x7f53f808df80 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:11:14.775 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:14.762+0000 7f53fda80700 1 -- 172.21.15.42:0/606975564 shutdown_connections 2019-12-05T06:11:14.775 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:14.762+0000 7f53fda80700 1 -- 172.21.15.42:0/606975564 wait complete. 2019-12-05T06:11:14.776 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:14.762+0000 7f53fda80700 1 -- 172.21.15.42:0/606975564 >> 172.21.15.42:0/606975564 conn(0x7f53f80aabe0 msgr2=0x7f53f80a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:11:14.782 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:14.766+0000 7f53fda80700 1 Processor -- start 2019-12-05T06:11:14.782 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:14.766+0000 7f53fda80700 1 -- start start 2019-12-05T06:11:14.783 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:14.766+0000 7f53fda80700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f53f811eeb0 0x7f53f80a4bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:11:14.783 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:14.766+0000 7f53fda80700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f53f811b6a0 0x7f53f811a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:11:14.783 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:14.766+0000 7f53fda80700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53f811afb0 con 0x7f53f80a68d0 2019-12-05T06:11:14.783 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:14.766+0000 7f53fda80700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53f811c000 con 0x7f53f811b6a0 2019-12-05T06:11:14.783 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:14.766+0000 7f53fda80700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53f811c2a0 con 0x7f53f811eeb0 2019-12-05T06:11:14.783 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:14.766+0000 7f53f7fff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f53f811eeb0 0x7f53f80a4bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:11:14.783 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:14.766+0000 7f53f7fff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f53f811eeb0 0x7f53f80a4bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:49918/0 (socket says 172.21.15.42:49918) 2019-12-05T06:11:14.784 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:14.766+0000 7f53f7fff700 1 -- 172.21.15.42:0/2339729332 learned_addr learned my addr 172.21.15.42:0/2339729332 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:11:14.784 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:14.766+0000 7f53effff700 1 --2- 172.21.15.42:0/2339729332 >> v2:172.21.15.40:3300/0 conn(0x7f53f811b6a0 0x7f53f811a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:11:14.784 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:14.766+0000 7f53f57fa700 1 -- 172.21.15.42:0/2339729332 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2361067683 0 0) 0x7f53f811afb0 con 0x7f53f80a68d0 2019-12-05T06:11:14.784 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:14.766+0000 7f53f57fa700 1 -- 172.21.15.42:0/2339729332 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53dc0030b0 con 0x7f53f80a68d0 2019-12-05T06:11:14.784 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:14.766+0000 7f53f7fff700 1 -- 172.21.15.42:0/2339729332 >> v2:172.21.15.40:3300/0 conn(0x7f53f811b6a0 msgr2=0x7f53f811a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:14.785 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:14.766+0000 7f53f7fff700 1 --2- 172.21.15.42:0/2339729332 >> v2:172.21.15.40:3300/0 conn(0x7f53f811b6a0 0x7f53f811a260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:11:14.785 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:14.766+0000 7f53f7fff700 1 -- 172.21.15.42:0/2339729332 >> v1:172.21.15.40:6789/0 conn(0x7f53f80a68d0 legacy=0x7f53f811a7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:14.785 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:14.766+0000 7f53f7fff700 1 -- 172.21.15.42:0/2339729332 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53f811bbf0 con 0x7f53f811eeb0 2019-12-05T06:11:14.785 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:14.770+0000 7f53f7fff700 1 --2- 172.21.15.42:0/2339729332 >> v2:172.21.15.42:3300/0 conn(0x7f53f811eeb0 0x7f53f80a4bd0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f53e8008790 tx=0x7f53e80118a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:14.786 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:14.770+0000 7f53f7fff700 1 --2- 172.21.15.42:0/2339729332 >> v2:172.21.15.42:3300/0 conn(0x7f53f811eeb0 0x7f53f80a4bd0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f53e8008790 tx=0x7f53e80118a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:14.786 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:14.770+0000 7f53f7fff700 1 --2- 172.21.15.42:0/2339729332 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f53f811eeb0 0x7f53f80a4bd0 secure :-1 s=READY pgs=2176 cs=0 l=1 rx=0x7f53e8008790 tx=0x7f53e80118a0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:11:14.786 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:14.770+0000 7f53f7fff700 1 --2- 172.21.15.42:0/2339729332 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f53f811eeb0 0x7f53f80a4bd0 secure :-1 s=THROTTLE_DONE pgs=2176 cs=0 l=1 rx=0x7f53e8008790 tx=0x7f53e80118a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:14.786 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:14.770+0000 7f53f57fa700 1 -- 172.21.15.42:0/2339729332 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f53e800f280 con 0x7f53f811eeb0 2019-12-05T06:11:14.787 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:14.770+0000 7f53f7fff700 1 --2- 172.21.15.42:0/2339729332 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f53f811eeb0 0x7f53f80a4bd0 secure :-1 s=THROTTLE_DONE pgs=2176 cs=0 l=1 rx=0x7f53e8008790 tx=0x7f53e80118a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:14.789 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:14.770+0000 7f53f7fff700 1 --2- 172.21.15.42:0/2339729332 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f53f811eeb0 0x7f53f80a4bd0 secure :-1 s=THROTTLE_DONE pgs=2176 cs=0 l=1 rx=0x7f53e8008790 tx=0x7f53e80118a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:14.789 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:14.770+0000 7f53f57fa700 1 -- 172.21.15.42:0/2339729332 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53e8008db0 con 0x7f53f811eeb0 2019-12-05T06:11:14.789 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:14.770+0000 7f53fda80700 1 -- 172.21.15.42:0/2339729332 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53f80a5120 con 0x7f53f811eeb0 2019-12-05T06:11:14.790 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:14.770+0000 7f53f57fa700 1 -- 172.21.15.42:0/2339729332 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f53e8005540 con 0x7f53f811eeb0 2019-12-05T06:11:14.791 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:14.770+0000 7f53fda80700 1 -- 172.21.15.42:0/2339729332 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53f808f0d0 con 0x7f53f811eeb0 2019-12-05T06:11:14.791 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:14.770+0000 7f53f7fff700 1 --2- 172.21.15.42:0/2339729332 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f53f811eeb0 0x7f53f80a4bd0 secure :-1 s=THROTTLE_DONE pgs=2176 cs=0 l=1 rx=0x7f53e8008790 tx=0x7f53e80118a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:14.791 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:14.770+0000 7f53f57fa700 1 -- 172.21.15.42:0/2339729332 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f53e80057e0 con 0x7f53f811eeb0 2019-12-05T06:11:14.791 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:14.770+0000 7f53f7fff700 1 --2- 172.21.15.42:0/2339729332 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f53f811eeb0 0x7f53f80a4bd0 secure :-1 s=THROTTLE_DONE pgs=2176 cs=0 l=1 rx=0x7f53e8008790 tx=0x7f53e80118a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:14.791 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:14.770+0000 7f53f57fa700 1 --2- 172.21.15.42:0/2339729332 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f53dc03b260 0x7f53dc03d770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:11:14.791 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:14.770+0000 7f53f57fa700 1 -- 172.21.15.42:0/2339729332 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9987+0+0 (secure 0 0 0) 0x7f53e804b3c0 con 0x7f53f811eeb0 2019-12-05T06:11:14.792 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:14.770+0000 7f53effff700 1 --2- 172.21.15.42:0/2339729332 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f53dc03b260 0x7f53dc03d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:11:14.792 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:14.774+0000 7f53effff700 1 --2- 172.21.15.42:0/2339729332 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f53dc03b260 0x7f53dc03d770 crc :-1 s=READY pgs=917 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:11:14.792 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:14.774+0000 7f53fda80700 1 -- 172.21.15.42:0/2339729332 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53f805d110 con 0x7f53f811eeb0 2019-12-05T06:11:14.793 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:14.782+0000 7f53f7fff700 1 --2- 172.21.15.42:0/2339729332 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f53f811eeb0 0x7f53f80a4bd0 secure :-1 s=THROTTLE_DONE pgs=2176 cs=0 l=1 rx=0x7f53e8008790 tx=0x7f53e80118a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:14.793 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:14.782+0000 7f53f57fa700 1 -- 172.21.15.42:0/2339729332 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f53e8032420 con 0x7f53f811eeb0 2019-12-05T06:11:14.970 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:14.966+0000 7f53fda80700 1 -- 172.21.15.42:0/2339729332 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f53f8142010 con 0x7f53f811eeb0 2019-12-05T06:11:14.970 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:14.966+0000 7f53f7fff700 1 --2- 172.21.15.42:0/2339729332 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f53f811eeb0 0x7f53f80a4bd0 secure :-1 s=THROTTLE_DONE pgs=2176 cs=0 l=1 rx=0x7f53e8008790 tx=0x7f53e80118a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:14.970 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:14.966+0000 7f53f57fa700 1 -- 172.21.15.42:0/2339729332 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 263 v263) v1 ==== 95+0+2113 (secure 0 0 0) 0x7f53e8032680 con 0x7f53f811eeb0 2019-12-05T06:11:14.970 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 263 2019-12-05T06:11:14.971 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:11:14.971 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":263,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":10000,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/3923117701","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3923117701},{"type":"v1","addr":"172.21.15.42:6835","nonce":3923117701}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":263}],"filesystems":[{"mdsmap":{"epoch":262,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:10:14.933627+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":95,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6977},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6977":{"gid":6977,"name":"b","rank":0,"incarnation":259,"state":"up:active","state_seq":32,"addr":"172.21.15.40:6833/3355679996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3355679996},{"type":"v1","addr":"172.21.15.40:6833","nonce":3355679996}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:11:14.973 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:14.970+0000 7f53fda80700 1 -- 172.21.15.42:0/2339729332 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f53dc03b260 msgr2=0x7f53dc03d770 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:14.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:14.970+0000 7f53fda80700 1 --2- 172.21.15.42:0/2339729332 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f53dc03b260 0x7f53dc03d770 crc :-1 s=READY pgs=917 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:11:14.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:14.970+0000 7f53fda80700 1 -- 172.21.15.42:0/2339729332 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f53f811eeb0 msgr2=0x7f53f80a4bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:14.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:14.970+0000 7f53fda80700 1 --2- 172.21.15.42:0/2339729332 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f53f811eeb0 0x7f53f80a4bd0 secure :-1 s=READY pgs=2176 cs=0 l=1 rx=0x7f53e8008790 tx=0x7f53e80118a0).stop 2019-12-05T06:11:14.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:14.970+0000 7f53fda80700 1 -- 172.21.15.42:0/2339729332 shutdown_connections 2019-12-05T06:11:14.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:14.970+0000 7f53fda80700 1 --2- 172.21.15.42:0/2339729332 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f53dc03b260 0x7f53dc03d770 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:11:14.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:14.970+0000 7f53fda80700 1 --2- 172.21.15.42:0/2339729332 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f53f811eeb0 0x7f53f80a4bd0 unknown :-1 s=CLOSED pgs=2176 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:11:14.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:14.970+0000 7f53fda80700 1 --2- 172.21.15.42:0/2339729332 >> v2:172.21.15.40:3300/0 conn(0x7f53f811b6a0 0x7f53f811a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:11:14.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:14.970+0000 7f53fda80700 1 -- 172.21.15.42:0/2339729332 shutdown_connections 2019-12-05T06:11:14.976 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:14.970+0000 7f53fda80700 1 -- 172.21.15.42:0/2339729332 wait complete. 2019-12-05T06:11:14.976 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:14.970+0000 7f53fda80700 1 -- 172.21.15.42:0/2339729332 >> 172.21.15.42:0/2339729332 conn(0x7f53f807e4f0 msgr2=0x7f53f811d820 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:11:14.983 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.b (rank=0) 2019-12-05T06:11:14.983 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2019-12-05T06:11:14.984 INFO:teuthology.orchestra.run:waiting for 300 2019-12-05T06:11:14.997 INFO:tasks.ceph.mds.b:Stopped 2019-12-05T06:11:14.998 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:11:15.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:15.090+0000 7f48a8750700 1 Processor -- start 2019-12-05T06:11:15.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:15.090+0000 7f48a8750700 1 -- start start 2019-12-05T06:11:15.100 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:15.090+0000 7f48a8750700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f48a00a7740 0x7f48a008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:11:15.100 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:15.090+0000 7f48a8750700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f48a009fae0 0x7f48a007eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:11:15.100 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:15.090+0000 7f48a8750700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48a00a0710 con 0x7f48a007e720 2019-12-05T06:11:15.100 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:15.090+0000 7f48a8750700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48a00a0b00 con 0x7f48a00a7740 2019-12-05T06:11:15.100 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:15.090+0000 7f48a8750700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48a008e5b0 con 0x7f48a009fae0 2019-12-05T06:11:15.100 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:15.090+0000 7f48a6ced700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f48a00a7740 0x7f48a008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:11:15.100 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:15.090+0000 7f48a6ced700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f48a00a7740 0x7f48a008df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:49932/0 (socket says 172.21.15.42:49932) 2019-12-05T06:11:15.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:15.090+0000 7f48a6ced700 1 -- 172.21.15.42:0/4158247082 learned_addr learned my addr 172.21.15.42:0/4158247082 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:11:15.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:15.090+0000 7f48a74ee700 1 --2- 172.21.15.42:0/4158247082 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f48a009fae0 0x7f48a007eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:11:15.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:15.090+0000 7f48a6ced700 1 -- 172.21.15.42:0/4158247082 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f48a009fae0 msgr2=0x7f48a007eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:11:15.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:15.090+0000 7f48a6ced700 1 --2- 172.21.15.42:0/4158247082 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f48a009fae0 0x7f48a007eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:11:15.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:15.090+0000 7f48a6ced700 1 -- 172.21.15.42:0/4158247082 >> v1:172.21.15.40:6789/0 conn(0x7f48a007e720 legacy=0x7f48a009f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:15.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:15.090+0000 7f48a6ced700 1 -- 172.21.15.42:0/4158247082 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48a007f0a0 con 0x7f48a00a7740 2019-12-05T06:11:15.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:15.090+0000 7f48a6ced700 1 --2- 172.21.15.42:0/4158247082 >> v2:172.21.15.42:3300/0 conn(0x7f48a00a7740 0x7f48a008df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f4894008b10 tx=0x7f48940100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:15.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:15.094+0000 7f48a6ced700 1 --2- 172.21.15.42:0/4158247082 >> v2:172.21.15.42:3300/0 conn(0x7f48a00a7740 0x7f48a008df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f4894008b10 tx=0x7f48940100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:15.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:15.094+0000 7f48a6ced700 1 --2- 172.21.15.42:0/4158247082 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f48a00a7740 0x7f48a008df80 secure :-1 s=READY pgs=2177 cs=0 l=1 rx=0x7f4894008b10 tx=0x7f48940100b0).ready entity=mon.1 client_cookie=e4aa7443d2c7e2b3 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:11:15.104 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:15.094+0000 7f48a6ced700 1 --2- 172.21.15.42:0/4158247082 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f48a00a7740 0x7f48a008df80 secure :-1 s=THROTTLE_DONE pgs=2177 cs=0 l=1 rx=0x7f4894008b10 tx=0x7f48940100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:15.104 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:15.094+0000 7f48a5ceb700 1 -- 172.21.15.42:0/4158247082 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4894010da0 con 0x7f48a00a7740 2019-12-05T06:11:15.104 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:15.094+0000 7f48a6ced700 1 --2- 172.21.15.42:0/4158247082 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f48a00a7740 0x7f48a008df80 secure :-1 s=THROTTLE_DONE pgs=2177 cs=0 l=1 rx=0x7f4894008b10 tx=0x7f48940100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:15.104 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:15.094+0000 7f48a6ced700 1 --2- 172.21.15.42:0/4158247082 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f48a00a7740 0x7f48a008df80 secure :-1 s=THROTTLE_DONE pgs=2177 cs=0 l=1 rx=0x7f4894008b10 tx=0x7f48940100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:15.104 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:15.094+0000 7f48a5ceb700 1 -- 172.21.15.42:0/4158247082 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48940109c0 con 0x7f48a00a7740 2019-12-05T06:11:15.104 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:15.094+0000 7f48a5ceb700 1 -- 172.21.15.42:0/4158247082 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f48940114a0 con 0x7f48a00a7740 2019-12-05T06:11:15.105 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:15.094+0000 7f48a8750700 1 -- 172.21.15.42:0/4158247082 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f48a00a7740 msgr2=0x7f48a008df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:15.105 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:15.094+0000 7f48a8750700 1 --2- 172.21.15.42:0/4158247082 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f48a00a7740 0x7f48a008df80 secure :-1 s=READY pgs=2177 cs=0 l=1 rx=0x7f4894008b10 tx=0x7f48940100b0).stop 2019-12-05T06:11:15.105 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:15.094+0000 7f48a8750700 1 -- 172.21.15.42:0/4158247082 shutdown_connections 2019-12-05T06:11:15.105 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:15.094+0000 7f48a8750700 1 --2- 172.21.15.42:0/4158247082 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f48a009fae0 0x7f48a007eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:11:15.105 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:15.094+0000 7f48a8750700 1 --2- 172.21.15.42:0/4158247082 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f48a00a7740 0x7f48a008df80 unknown :-1 s=CLOSED pgs=2177 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:11:15.105 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:15.094+0000 7f48a8750700 1 -- 172.21.15.42:0/4158247082 shutdown_connections 2019-12-05T06:11:15.108 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:15.094+0000 7f48a8750700 1 -- 172.21.15.42:0/4158247082 wait complete. 2019-12-05T06:11:15.108 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:15.094+0000 7f48a8750700 1 -- 172.21.15.42:0/4158247082 >> 172.21.15.42:0/4158247082 conn(0x7f48a00aabe0 msgr2=0x7f48a00a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:11:15.116 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:15.094+0000 7f48a8750700 1 Processor -- start 2019-12-05T06:11:15.116 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:15.098+0000 7f48a8750700 1 -- start start 2019-12-05T06:11:15.116 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:15.098+0000 7f48a8750700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f48a01168f0 0x7f48a0128380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:11:15.119 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:15.098+0000 7f48a8750700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48a01170a0 con 0x7f48a01288d0 2019-12-05T06:11:15.119 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:15.098+0000 7f48a8750700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48a0128d00 con 0x7f48a0114540 2019-12-05T06:11:15.119 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:15.098+0000 7f48a8750700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48a0118bb0 con 0x7f48a01168f0 2019-12-05T06:11:15.119 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:15.098+0000 7f48a6ced700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f48a0114540 0x7f48a01190b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:49976/0 (socket says 172.21.15.42:49976) 2019-12-05T06:11:15.119 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:15.098+0000 7f48a6ced700 1 -- 172.21.15.42:0/2397287920 learned_addr learned my addr 172.21.15.42:0/2397287920 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:11:15.120 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:15.098+0000 7f48a64ec700 1 --2- 172.21.15.42:0/2397287920 >> v2:172.21.15.40:3300/0 conn(0x7f48a01168f0 0x7f48a0128380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:11:15.120 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:15.098+0000 7f489f7fe700 1 -- 172.21.15.42:0/2397287920 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3676581267 0 0) 0x7f48a0128d00 con 0x7f48a0114540 2019-12-05T06:11:15.120 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:15.098+0000 7f489f7fe700 1 -- 172.21.15.42:0/2397287920 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48880030b0 con 0x7f48a0114540 2019-12-05T06:11:15.120 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:15.098+0000 7f489f7fe700 1 -- 172.21.15.42:0/2397287920 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 2543019060 0 0) 0x7f48880030b0 con 0x7f48a0114540 2019-12-05T06:11:15.120 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:15.098+0000 7f489f7fe700 1 -- 172.21.15.42:0/2397287920 --> v1:172.21.15.42:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f48a0128d00 con 0x7f48a0114540 2019-12-05T06:11:15.121 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:15.098+0000 7f489f7fe700 1 -- 172.21.15.42:0/2397287920 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7f48940087f0 con 0x7f48a0114540 2019-12-05T06:11:15.121 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:15.098+0000 7f48a64ec700 1 -- 172.21.15.42:0/2397287920 >> v1:172.21.15.42:6789/0 conn(0x7f48a0114540 legacy=0x7f48a01190b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:15.121 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:15.098+0000 7f48a64ec700 1 -- 172.21.15.42:0/2397287920 >> v1:172.21.15.40:6789/0 conn(0x7f48a01288d0 legacy=0x7f48a012f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:15.121 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:15.098+0000 7f48a64ec700 1 -- 172.21.15.42:0/2397287920 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48a0127eb0 con 0x7f48a01168f0 2019-12-05T06:11:15.121 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:15.098+0000 7f48a64ec700 1 --2- 172.21.15.42:0/2397287920 >> v2:172.21.15.40:3300/0 conn(0x7f48a01168f0 0x7f48a0128380 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f488c001430 tx=0x7f4898002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:15.121 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:15.098+0000 7f48a64ec700 1 --2- 172.21.15.42:0/2397287920 >> v2:172.21.15.40:3300/0 conn(0x7f48a01168f0 0x7f48a0128380 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f488c001430 tx=0x7f4898002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:15.122 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:15.098+0000 7f48a64ec700 1 --2- 172.21.15.42:0/2397287920 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f48a01168f0 0x7f48a0128380 secure :-1 s=READY pgs=1652 cs=0 l=1 rx=0x7f488c001430 tx=0x7f4898002ff0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:11:15.122 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:15.098+0000 7f48a64ec700 1 --2- 172.21.15.42:0/2397287920 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f48a01168f0 0x7f48a0128380 secure :-1 s=THROTTLE_DONE pgs=1652 cs=0 l=1 rx=0x7f488c001430 tx=0x7f4898002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:15.122 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:15.098+0000 7f489f7fe700 1 -- 172.21.15.42:0/2397287920 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f488c00c960 con 0x7f48a01168f0 2019-12-05T06:11:15.122 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:15.098+0000 7f48a64ec700 1 --2- 172.21.15.42:0/2397287920 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f48a01168f0 0x7f48a0128380 secure :-1 s=THROTTLE_DONE pgs=1652 cs=0 l=1 rx=0x7f488c001430 tx=0x7f4898002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:15.122 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:15.098+0000 7f489f7fe700 1 -- 172.21.15.42:0/2397287920 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f488c00c580 con 0x7f48a01168f0 2019-12-05T06:11:15.123 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:15.098+0000 7f48a64ec700 1 --2- 172.21.15.42:0/2397287920 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f48a01168f0 0x7f48a0128380 secure :-1 s=THROTTLE_DONE pgs=1652 cs=0 l=1 rx=0x7f488c001430 tx=0x7f4898002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:15.123 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:15.098+0000 7f48a8750700 1 -- 172.21.15.42:0/2397287920 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48a01198b0 con 0x7f48a01168f0 2019-12-05T06:11:15.123 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:15.098+0000 7f489f7fe700 1 -- 172.21.15.42:0/2397287920 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f488c010d00 con 0x7f48a01168f0 2019-12-05T06:11:15.123 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:15.098+0000 7f48a8750700 1 -- 172.21.15.42:0/2397287920 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48a0116d20 con 0x7f48a01168f0 2019-12-05T06:11:15.123 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:15.102+0000 7f48a64ec700 1 --2- 172.21.15.42:0/2397287920 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f48a01168f0 0x7f48a0128380 secure :-1 s=THROTTLE_DONE pgs=1652 cs=0 l=1 rx=0x7f488c001430 tx=0x7f4898002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:15.123 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:15.102+0000 7f489f7fe700 1 -- 172.21.15.42:0/2397287920 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f488c007430 con 0x7f48a01168f0 2019-12-05T06:11:15.123 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:15.102+0000 7f48a64ec700 1 --2- 172.21.15.42:0/2397287920 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f48a01168f0 0x7f48a0128380 secure :-1 s=THROTTLE_DONE pgs=1652 cs=0 l=1 rx=0x7f488c001430 tx=0x7f4898002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:15.124 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:15.102+0000 7f489f7fe700 1 --2- 172.21.15.42:0/2397287920 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f488803b5d0 0x7f488803dae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:11:15.124 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:15.102+0000 7f489f7fe700 1 -- 172.21.15.42:0/2397287920 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9987+0+0 (secure 0 0 0) 0x7f488c04af20 con 0x7f48a01168f0 2019-12-05T06:11:15.125 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:15.102+0000 7f48a6ced700 1 --2- 172.21.15.42:0/2397287920 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f488803b5d0 0x7f488803dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:11:15.125 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:15.102+0000 7f48a6ced700 1 --2- 172.21.15.42:0/2397287920 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f488803b5d0 0x7f488803dae0 crc :-1 s=READY pgs=918 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:11:15.125 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:15.106+0000 7f48a8750700 1 -- 172.21.15.42:0/2397287920 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48a005d110 con 0x7f48a01168f0 2019-12-05T06:11:15.126 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:15.114+0000 7f48a64ec700 1 --2- 172.21.15.42:0/2397287920 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f48a01168f0 0x7f48a0128380 secure :-1 s=THROTTLE_DONE pgs=1652 cs=0 l=1 rx=0x7f488c001430 tx=0x7f4898002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:15.126 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:15.114+0000 7f489f7fe700 1 -- 172.21.15.42:0/2397287920 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f488c031f20 con 0x7f48a01168f0 2019-12-05T06:11:15.217 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~0s 2019-12-05T06:11:15.285 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:15.282+0000 7f48a8750700 1 -- 172.21.15.42:0/2397287920 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f48a005b8c0 con 0x7f48a01168f0 2019-12-05T06:11:15.286 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:15.282+0000 7f48a64ec700 1 --2- 172.21.15.42:0/2397287920 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f48a01168f0 0x7f48a0128380 secure :-1 s=THROTTLE_DONE pgs=1652 cs=0 l=1 rx=0x7f488c001430 tx=0x7f4898002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:15.287 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:15.282+0000 7f489f7fe700 1 -- 172.21.15.42:0/2397287920 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 263 v263) v1 ==== 95+0+2113 (secure 0 0 0) 0x7f488c032180 con 0x7f48a01168f0 2019-12-05T06:11:15.287 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:11:15.288 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":263,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":10000,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/3923117701","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3923117701},{"type":"v1","addr":"172.21.15.42:6835","nonce":3923117701}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":263}],"filesystems":[{"mdsmap":{"epoch":262,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:10:14.933627+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":95,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6977},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6977":{"gid":6977,"name":"b","rank":0,"incarnation":259,"state":"up:active","state_seq":32,"addr":"172.21.15.40:6833/3355679996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3355679996},{"type":"v1","addr":"172.21.15.40:6833","nonce":3355679996}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:11:15.288 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 263 2019-12-05T06:11:15.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:15.286+0000 7f48a8750700 1 -- 172.21.15.42:0/2397287920 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f488803b5d0 msgr2=0x7f488803dae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:15.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:15.286+0000 7f48a8750700 1 --2- 172.21.15.42:0/2397287920 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f488803b5d0 0x7f488803dae0 crc :-1 s=READY pgs=918 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:11:15.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:15.286+0000 7f48a8750700 1 -- 172.21.15.42:0/2397287920 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f48a01168f0 msgr2=0x7f48a0128380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:15.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:15.286+0000 7f48a8750700 1 --2- 172.21.15.42:0/2397287920 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f48a01168f0 0x7f48a0128380 secure :-1 s=READY pgs=1652 cs=0 l=1 rx=0x7f488c001430 tx=0x7f4898002ff0).stop 2019-12-05T06:11:15.291 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:15.286+0000 7f48a8750700 1 -- 172.21.15.42:0/2397287920 shutdown_connections 2019-12-05T06:11:15.291 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:15.286+0000 7f48a8750700 1 --2- 172.21.15.42:0/2397287920 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f488803b5d0 0x7f488803dae0 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:11:15.291 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:15.286+0000 7f48a8750700 1 --2- 172.21.15.42:0/2397287920 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f48a01168f0 0x7f48a0128380 unknown :-1 s=CLOSED pgs=1652 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:11:15.291 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:15.286+0000 7f48a8750700 1 -- 172.21.15.42:0/2397287920 shutdown_connections 2019-12-05T06:11:15.291 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:15.286+0000 7f48a8750700 1 -- 172.21.15.42:0/2397287920 wait complete. 2019-12-05T06:11:15.292 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:15.286+0000 7f48a8750700 1 -- 172.21.15.42:0/2397287920 >> 172.21.15.42:0/2397287920 conn(0x7f48a00aabe0 msgr2=0x7f48a009f4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:11:15.300 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:11:17.302 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:11:17.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:17.398+0000 7f50245d6700 1 Processor -- start 2019-12-05T06:11:17.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:17.398+0000 7f50245d6700 1 -- start start 2019-12-05T06:11:17.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:17.402+0000 7f50245d6700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f501c0a7740 0x7f501c08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:11:17.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:17.402+0000 7f50245d6700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f501c0a0710 con 0x7f501c07e720 2019-12-05T06:11:17.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:17.402+0000 7f50245d6700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f501c0a9ba0 con 0x7f501c09fae0 2019-12-05T06:11:17.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:17.402+0000 7f50245d6700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f501c0a0b00 con 0x7f501c0a7740 2019-12-05T06:11:17.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:17.402+0000 7f5023374700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f501c09fae0 0x7f501c14efe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:49994/0 (socket says 172.21.15.42:49994) 2019-12-05T06:11:17.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:17.402+0000 7f5023374700 1 -- 172.21.15.42:0/3906380135 learned_addr learned my addr 172.21.15.42:0/3906380135 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:11:17.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:17.402+0000 7f5022b73700 1 --2- 172.21.15.42:0/3906380135 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f501c0a7740 0x7f501c08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:11:17.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:17.402+0000 7f5021b71700 1 -- 172.21.15.42:0/3906380135 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2428407330 0 0) 0x7f501c0a9ba0 con 0x7f501c09fae0 2019-12-05T06:11:17.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:17.402+0000 7f5021b71700 1 -- 172.21.15.42:0/3906380135 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50080030b0 con 0x7f501c09fae0 2019-12-05T06:11:17.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:17.402+0000 7f5021b71700 1 -- 172.21.15.42:0/3906380135 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1166516291 0 0) 0x7f501c0a0710 con 0x7f501c07e720 2019-12-05T06:11:17.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:17.402+0000 7f5021b71700 1 -- 172.21.15.42:0/3906380135 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f501c0a9ba0 con 0x7f501c07e720 2019-12-05T06:11:17.411 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:17.402+0000 7f5022b73700 1 -- 172.21.15.42:0/3906380135 >> v1:172.21.15.42:6789/0 conn(0x7f501c09fae0 legacy=0x7f501c14efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:17.411 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:17.402+0000 7f5022b73700 1 -- 172.21.15.42:0/3906380135 >> v1:172.21.15.40:6789/0 conn(0x7f501c07e720 legacy=0x7f501c09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:17.411 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:17.402+0000 7f5022b73700 1 -- 172.21.15.42:0/3906380135 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f501c07eb50 con 0x7f501c0a7740 2019-12-05T06:11:17.411 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:17.402+0000 7f5022b73700 1 --2- 172.21.15.42:0/3906380135 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f501c0a7740 0x7f501c08df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f5018002f30 tx=0x7f501800fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:17.411 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:17.402+0000 7f5022b73700 1 --2- 172.21.15.42:0/3906380135 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f501c0a7740 0x7f501c08df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f5018002f30 tx=0x7f501800fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:17.411 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:17.402+0000 7f5022b73700 1 --2- 172.21.15.42:0/3906380135 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f501c0a7740 0x7f501c08df80 secure :-1 s=READY pgs=479 cs=0 l=1 rx=0x7f5018002f30 tx=0x7f501800fbf0).ready entity=mon.2 client_cookie=f7fb8cd167c9b990 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:11:17.411 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:17.402+0000 7f5022b73700 1 --2- 172.21.15.42:0/3906380135 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f501c0a7740 0x7f501c08df80 secure :-1 s=THROTTLE_DONE pgs=479 cs=0 l=1 rx=0x7f5018002f30 tx=0x7f501800fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:17.412 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:17.402+0000 7f5021b71700 1 -- 172.21.15.42:0/3906380135 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f50180112b0 con 0x7f501c0a7740 2019-12-05T06:11:17.412 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:17.402+0000 7f5022b73700 1 --2- 172.21.15.42:0/3906380135 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f501c0a7740 0x7f501c08df80 secure :-1 s=THROTTLE_DONE pgs=479 cs=0 l=1 rx=0x7f5018002f30 tx=0x7f501800fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:17.412 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:17.402+0000 7f5022b73700 1 --2- 172.21.15.42:0/3906380135 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f501c0a7740 0x7f501c08df80 secure :-1 s=THROTTLE_DONE pgs=479 cs=0 l=1 rx=0x7f5018002f30 tx=0x7f501800fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:17.412 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:17.402+0000 7f5021b71700 1 -- 172.21.15.42:0/3906380135 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5018008bc0 con 0x7f501c0a7740 2019-12-05T06:11:17.412 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:17.402+0000 7f5021b71700 1 -- 172.21.15.42:0/3906380135 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5018011890 con 0x7f501c0a7740 2019-12-05T06:11:17.412 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:17.402+0000 7f50245d6700 1 -- 172.21.15.42:0/3906380135 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f501c0a7740 msgr2=0x7f501c08df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:17.412 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:17.402+0000 7f50245d6700 1 --2- 172.21.15.42:0/3906380135 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f501c0a7740 0x7f501c08df80 secure :-1 s=READY pgs=479 cs=0 l=1 rx=0x7f5018002f30 tx=0x7f501800fbf0).stop 2019-12-05T06:11:17.413 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:17.406+0000 7f50245d6700 1 -- 172.21.15.42:0/3906380135 shutdown_connections 2019-12-05T06:11:17.413 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:17.406+0000 7f50245d6700 1 --2- 172.21.15.42:0/3906380135 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f501c0a7740 0x7f501c08df80 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:11:17.413 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:17.406+0000 7f50245d6700 1 -- 172.21.15.42:0/3906380135 shutdown_connections 2019-12-05T06:11:17.413 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:17.406+0000 7f50245d6700 1 -- 172.21.15.42:0/3906380135 wait complete. 2019-12-05T06:11:17.413 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:17.406+0000 7f50245d6700 1 -- 172.21.15.42:0/3906380135 >> 172.21.15.42:0/3906380135 conn(0x7f501c0aabe0 msgr2=0x7f501c0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:11:17.416 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:17.406+0000 7f50245d6700 1 Processor -- start 2019-12-05T06:11:17.416 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:17.406+0000 7f50245d6700 1 -- start start 2019-12-05T06:11:17.417 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:17.406+0000 7f50245d6700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f501c116990 0x7f501c1277b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:11:17.417 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:17.406+0000 7f50245d6700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f501c127d00 0x7f501c07b880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:11:17.417 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:17.406+0000 7f50245d6700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f501c143110 con 0x7f501c07bdd0 2019-12-05T06:11:17.417 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:17.406+0000 7f50245d6700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f501c142970 con 0x7f501c127d00 2019-12-05T06:11:17.417 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:17.406+0000 7f50245d6700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f501c142bd0 con 0x7f501c116990 2019-12-05T06:11:17.418 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:17.406+0000 7f5022b73700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f501c116990 0x7f501c1277b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:11:17.418 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:17.406+0000 7f5022b73700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f501c116990 0x7f501c1277b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:49958/0 (socket says 172.21.15.42:49958) 2019-12-05T06:11:17.418 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:17.406+0000 7f5022b73700 1 -- 172.21.15.42:0/1026733015 learned_addr learned my addr 172.21.15.42:0/1026733015 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:11:17.418 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:17.406+0000 7f5022372700 1 --2- 172.21.15.42:0/1026733015 >> v2:172.21.15.40:3300/0 conn(0x7f501c127d00 0x7f501c07b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:11:17.418 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:17.406+0000 7f5022b73700 1 -- 172.21.15.42:0/1026733015 >> v2:172.21.15.40:3300/0 conn(0x7f501c127d00 msgr2=0x7f501c07b880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:17.418 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:17.406+0000 7f5022b73700 1 --2- 172.21.15.42:0/1026733015 >> v2:172.21.15.40:3300/0 conn(0x7f501c127d00 0x7f501c07b880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:11:17.419 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:17.410+0000 7f5022b73700 1 -- 172.21.15.42:0/1026733015 >> v1:172.21.15.40:6789/0 conn(0x7f501c07bdd0 legacy=0x7f501c144220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:17.419 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:17.410+0000 7f5022b73700 1 -- 172.21.15.42:0/1026733015 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f501c144a20 con 0x7f501c116990 2019-12-05T06:11:17.419 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:17.410+0000 7f5022b73700 1 --2- 172.21.15.42:0/1026733015 >> v2:172.21.15.42:3300/0 conn(0x7f501c116990 0x7f501c1277b0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f501c141710 tx=0x7f5018007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:17.419 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:17.410+0000 7f5022b73700 1 --2- 172.21.15.42:0/1026733015 >> v2:172.21.15.42:3300/0 conn(0x7f501c116990 0x7f501c1277b0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f501c141710 tx=0x7f5018007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:17.419 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:17.410+0000 7f5022b73700 1 --2- 172.21.15.42:0/1026733015 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f501c116990 0x7f501c1277b0 secure :-1 s=READY pgs=2180 cs=0 l=1 rx=0x7f501c141710 tx=0x7f5018007710).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:11:17.420 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:17.410+0000 7f5022b73700 1 --2- 172.21.15.42:0/1026733015 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f501c116990 0x7f501c1277b0 secure :-1 s=THROTTLE_DONE pgs=2180 cs=0 l=1 rx=0x7f501c141710 tx=0x7f5018007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:17.420 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:17.410+0000 7f5022b73700 1 --2- 172.21.15.42:0/1026733015 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f501c116990 0x7f501c1277b0 secure :-1 s=THROTTLE_DONE pgs=2180 cs=0 l=1 rx=0x7f501c141710 tx=0x7f5018007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:17.420 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:17.410+0000 7f50137fe700 1 -- 172.21.15.42:0/1026733015 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5018007340 con 0x7f501c116990 2019-12-05T06:11:17.420 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:17.410+0000 7f5022b73700 1 --2- 172.21.15.42:0/1026733015 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f501c116990 0x7f501c1277b0 secure :-1 s=THROTTLE_DONE pgs=2180 cs=0 l=1 rx=0x7f501c141710 tx=0x7f5018007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:17.420 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:17.410+0000 7f50137fe700 1 -- 172.21.15.42:0/1026733015 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50180114b0 con 0x7f501c116990 2019-12-05T06:11:17.420 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:17.410+0000 7f50245d6700 1 -- 172.21.15.42:0/1026733015 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f501c12ec00 con 0x7f501c116990 2019-12-05T06:11:17.420 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:17.410+0000 7f50137fe700 1 -- 172.21.15.42:0/1026733015 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f501800f360 con 0x7f501c116990 2019-12-05T06:11:17.421 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:17.410+0000 7f50245d6700 1 -- 172.21.15.42:0/1026733015 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f501c12f1b0 con 0x7f501c116990 2019-12-05T06:11:17.421 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:17.410+0000 7f5022b73700 1 --2- 172.21.15.42:0/1026733015 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f501c116990 0x7f501c1277b0 secure :-1 s=THROTTLE_DONE pgs=2180 cs=0 l=1 rx=0x7f501c141710 tx=0x7f5018007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:17.421 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:17.410+0000 7f50137fe700 1 -- 172.21.15.42:0/1026733015 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f5018005540 con 0x7f501c116990 2019-12-05T06:11:17.421 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:17.410+0000 7f5022b73700 1 --2- 172.21.15.42:0/1026733015 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f501c116990 0x7f501c1277b0 secure :-1 s=THROTTLE_DONE pgs=2180 cs=0 l=1 rx=0x7f501c141710 tx=0x7f5018007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:17.421 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:17.410+0000 7f50137fe700 1 --2- 172.21.15.42:0/1026733015 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f500403acc0 0x7f500403d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:11:17.421 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:17.410+0000 7f50137fe700 1 -- 172.21.15.42:0/1026733015 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9987+0+0 (secure 0 0 0) 0x7f501804b6d0 con 0x7f501c116990 2019-12-05T06:11:17.422 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:17.410+0000 7f5022372700 1 --2- 172.21.15.42:0/1026733015 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f500403acc0 0x7f500403d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:11:17.422 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:17.414+0000 7f5022372700 1 --2- 172.21.15.42:0/1026733015 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f500403acc0 0x7f500403d1d0 crc :-1 s=READY pgs=919 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:11:17.422 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:17.418+0000 7f50245d6700 1 -- 172.21.15.42:0/1026733015 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f501c05d110 con 0x7f501c116990 2019-12-05T06:11:17.427 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:17.422+0000 7f5022b73700 1 --2- 172.21.15.42:0/1026733015 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f501c116990 0x7f501c1277b0 secure :-1 s=THROTTLE_DONE pgs=2180 cs=0 l=1 rx=0x7f501c141710 tx=0x7f5018007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:17.427 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:17.422+0000 7f50137fe700 1 -- 172.21.15.42:0/1026733015 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f5018005850 con 0x7f501c116990 2019-12-05T06:11:17.601 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:17.598+0000 7f50245d6700 1 -- 172.21.15.42:0/1026733015 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f501c05b8c0 con 0x7f501c116990 2019-12-05T06:11:17.601 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:17.598+0000 7f5022b73700 1 --2- 172.21.15.42:0/1026733015 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f501c116990 0x7f501c1277b0 secure :-1 s=THROTTLE_DONE pgs=2180 cs=0 l=1 rx=0x7f501c141710 tx=0x7f5018007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:17.602 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:17.598+0000 7f50137fe700 1 -- 172.21.15.42:0/1026733015 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 263 v263) v1 ==== 95+0+2113 (secure 0 0 0) 0x7f5018012c00 con 0x7f501c116990 2019-12-05T06:11:17.604 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 263 2019-12-05T06:11:17.604 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:11:17.604 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":263,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":10000,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/3923117701","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3923117701},{"type":"v1","addr":"172.21.15.42:6835","nonce":3923117701}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":263}],"filesystems":[{"mdsmap":{"epoch":262,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:10:14.933627+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":95,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6977},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6977":{"gid":6977,"name":"b","rank":0,"incarnation":259,"state":"up:active","state_seq":32,"addr":"172.21.15.40:6833/3355679996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3355679996},{"type":"v1","addr":"172.21.15.40:6833","nonce":3355679996}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:11:17.608 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:17.602+0000 7f50245d6700 1 -- 172.21.15.42:0/1026733015 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f500403acc0 msgr2=0x7f500403d1d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:17.608 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:17.602+0000 7f50245d6700 1 --2- 172.21.15.42:0/1026733015 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f500403acc0 0x7f500403d1d0 crc :-1 s=READY pgs=919 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:11:17.609 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:17.602+0000 7f50245d6700 1 -- 172.21.15.42:0/1026733015 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f501c116990 msgr2=0x7f501c1277b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:17.609 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:17.602+0000 7f50245d6700 1 --2- 172.21.15.42:0/1026733015 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f501c116990 0x7f501c1277b0 secure :-1 s=READY pgs=2180 cs=0 l=1 rx=0x7f501c141710 tx=0x7f5018007710).stop 2019-12-05T06:11:17.609 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:17.606+0000 7f50245d6700 1 -- 172.21.15.42:0/1026733015 shutdown_connections 2019-12-05T06:11:17.609 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:17.606+0000 7f50245d6700 1 --2- 172.21.15.42:0/1026733015 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f500403acc0 0x7f500403d1d0 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:11:17.610 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:17.606+0000 7f50245d6700 1 --2- 172.21.15.42:0/1026733015 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f501c116990 0x7f501c1277b0 unknown :-1 s=CLOSED pgs=2180 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:11:17.610 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:17.606+0000 7f50245d6700 1 --2- 172.21.15.42:0/1026733015 >> v2:172.21.15.40:3300/0 conn(0x7f501c127d00 0x7f501c07b880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:11:17.610 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:17.606+0000 7f50245d6700 1 -- 172.21.15.42:0/1026733015 shutdown_connections 2019-12-05T06:11:17.610 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:17.606+0000 7f50245d6700 1 -- 172.21.15.42:0/1026733015 wait complete. 2019-12-05T06:11:17.610 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:17.606+0000 7f50245d6700 1 -- 172.21.15.42:0/1026733015 >> 172.21.15.42:0/1026733015 conn(0x7f501c0a7740 msgr2=0x7f501c07d9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:11:17.621 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:11:19.622 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:11:19.732 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:19.722+0000 7f1007764700 1 Processor -- start 2019-12-05T06:11:19.732 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:19.722+0000 7f1007764700 1 -- start start 2019-12-05T06:11:19.732 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:19.726+0000 7f1007764700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f10000a7730 0x7f100008df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:11:19.732 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:19.726+0000 7f1007764700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f100009fad0 0x7f100007eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:11:19.732 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:19.726+0000 7f1007764700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10000a0700 con 0x7f100007e710 2019-12-05T06:11:19.733 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:19.726+0000 7f1007764700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10000a0af0 con 0x7f100009fad0 2019-12-05T06:11:19.733 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:19.726+0000 7f1007764700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f100008e5a0 con 0x7f10000a7730 2019-12-05T06:11:19.733 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:19.726+0000 7f1005500700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f100007e710 0x7f100009f2d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:50010/0 (socket says 172.21.15.42:50010) 2019-12-05T06:11:19.733 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:19.726+0000 7f1005500700 1 -- 172.21.15.42:0/4282294796 learned_addr learned my addr 172.21.15.42:0/4282294796 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:11:19.733 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:19.726+0000 7f1005d01700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f10000a7730 0x7f100008df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:11:19.734 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:19.726+0000 7f1006502700 1 --2- 172.21.15.42:0/4282294796 >> v2:172.21.15.40:3300/0 conn(0x7f100009fad0 0x7f100007eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:11:19.734 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:19.726+0000 7f1004cff700 1 -- 172.21.15.42:0/4282294796 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3946905487 0 0) 0x7f10000a0700 con 0x7f100007e710 2019-12-05T06:11:19.734 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:19.726+0000 7f1004cff700 1 -- 172.21.15.42:0/4282294796 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0fec0030b0 con 0x7f100007e710 2019-12-05T06:11:19.734 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:19.726+0000 7f1005d01700 1 -- 172.21.15.42:0/4282294796 >> v2:172.21.15.40:3300/0 conn(0x7f100009fad0 msgr2=0x7f100007eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:11:19.734 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:19.726+0000 7f1005d01700 1 --2- 172.21.15.42:0/4282294796 >> v2:172.21.15.40:3300/0 conn(0x7f100009fad0 0x7f100007eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:11:19.734 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:19.726+0000 7f1005d01700 1 -- 172.21.15.42:0/4282294796 >> v1:172.21.15.42:6789/0 conn(0x7f100007e710 legacy=0x7f100009f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:19.734 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:19.726+0000 7f1005d01700 1 -- 172.21.15.42:0/4282294796 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f100007f090 con 0x7f10000a7730 2019-12-05T06:11:19.735 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:19.726+0000 7f1005d01700 1 --2- 172.21.15.42:0/4282294796 >> v2:172.21.15.42:3300/0 conn(0x7f10000a7730 0x7f100008df70 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f0ffc002f30 tx=0x7f0ffc00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:19.735 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:19.726+0000 7f1005d01700 1 --2- 172.21.15.42:0/4282294796 >> v2:172.21.15.42:3300/0 conn(0x7f10000a7730 0x7f100008df70 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f0ffc002f30 tx=0x7f0ffc00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:19.735 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:19.726+0000 7f1005d01700 1 --2- 172.21.15.42:0/4282294796 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f10000a7730 0x7f100008df70 secure :-1 s=READY pgs=2182 cs=0 l=1 rx=0x7f0ffc002f30 tx=0x7f0ffc00fc70).ready entity=mon.1 client_cookie=60fb0fc481a5c806 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:11:19.735 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:19.726+0000 7f1005d01700 1 --2- 172.21.15.42:0/4282294796 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f10000a7730 0x7f100008df70 secure :-1 s=THROTTLE_DONE pgs=2182 cs=0 l=1 rx=0x7f0ffc002f30 tx=0x7f0ffc00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:19.735 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:19.726+0000 7f1004cff700 1 -- 172.21.15.42:0/4282294796 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0ffc008bb0 con 0x7f10000a7730 2019-12-05T06:11:19.736 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:19.726+0000 7f1005d01700 1 --2- 172.21.15.42:0/4282294796 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f10000a7730 0x7f100008df70 secure :-1 s=THROTTLE_DONE pgs=2182 cs=0 l=1 rx=0x7f0ffc002f30 tx=0x7f0ffc00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:19.736 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:19.726+0000 7f1005d01700 1 --2- 172.21.15.42:0/4282294796 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f10000a7730 0x7f100008df70 secure :-1 s=THROTTLE_DONE pgs=2182 cs=0 l=1 rx=0x7f0ffc002f30 tx=0x7f0ffc00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:19.736 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:19.726+0000 7f1004cff700 1 -- 172.21.15.42:0/4282294796 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ffc0087d0 con 0x7f10000a7730 2019-12-05T06:11:19.736 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:19.726+0000 7f1004cff700 1 -- 172.21.15.42:0/4282294796 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0ffc011820 con 0x7f10000a7730 2019-12-05T06:11:19.736 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:19.726+0000 7f1007764700 1 -- 172.21.15.42:0/4282294796 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f10000a7730 msgr2=0x7f100008df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:19.737 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:19.726+0000 7f1007764700 1 --2- 172.21.15.42:0/4282294796 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f10000a7730 0x7f100008df70 secure :-1 s=READY pgs=2182 cs=0 l=1 rx=0x7f0ffc002f30 tx=0x7f0ffc00fc70).stop 2019-12-05T06:11:19.737 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:19.726+0000 7f1007764700 1 -- 172.21.15.42:0/4282294796 shutdown_connections 2019-12-05T06:11:19.737 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:19.726+0000 7f1007764700 1 --2- 172.21.15.42:0/4282294796 >> v2:172.21.15.40:3300/0 conn(0x7f100009fad0 0x7f100007eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:11:19.737 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:19.726+0000 7f1007764700 1 --2- 172.21.15.42:0/4282294796 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f10000a7730 0x7f100008df70 unknown :-1 s=CLOSED pgs=2182 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:11:19.737 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:19.726+0000 7f1007764700 1 -- 172.21.15.42:0/4282294796 shutdown_connections 2019-12-05T06:11:19.737 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:19.726+0000 7f1007764700 1 -- 172.21.15.42:0/4282294796 wait complete. 2019-12-05T06:11:19.737 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:19.726+0000 7f1007764700 1 -- 172.21.15.42:0/4282294796 >> 172.21.15.42:0/4282294796 conn(0x7f10000aabd0 msgr2=0x7f10000a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:11:19.751 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:19.730+0000 7f1007764700 1 Processor -- start 2019-12-05T06:11:19.751 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:19.730+0000 7f1007764700 1 -- start start 2019-12-05T06:11:19.751 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:19.730+0000 7f1007764700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f1000119c90 0x7f1000144800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:11:19.751 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:19.730+0000 7f1007764700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f100011a850 0x7f1000117dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:11:19.751 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:19.730+0000 7f1007764700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1000144d50 con 0x7f1000118310 2019-12-05T06:11:19.752 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:19.730+0000 7f1007764700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f100011b120 con 0x7f1000119c90 2019-12-05T06:11:19.752 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:19.730+0000 7f1007764700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1000117b20 con 0x7f100011a850 2019-12-05T06:11:19.752 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:19.730+0000 7f1005d01700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f1000119c90 0x7f1000144800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:11:19.752 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:19.730+0000 7f1005d01700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f1000119c90 0x7f1000144800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:49978/0 (socket says 172.21.15.42:49978) 2019-12-05T06:11:19.752 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:19.730+0000 7f1005d01700 1 -- 172.21.15.42:0/2240411278 learned_addr learned my addr 172.21.15.42:0/2240411278 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:11:19.752 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:19.730+0000 7f1005500700 1 --2- 172.21.15.42:0/2240411278 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f100011a850 0x7f1000117dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:11:19.752 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:19.730+0000 7f1005d01700 1 -- 172.21.15.42:0/2240411278 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f100011a850 msgr2=0x7f1000117dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:19.753 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:19.730+0000 7f1005d01700 1 --2- 172.21.15.42:0/2240411278 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f100011a850 0x7f1000117dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:11:19.753 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:19.730+0000 7f1005d01700 1 -- 172.21.15.42:0/2240411278 >> v1:172.21.15.40:6789/0 conn(0x7f1000118310 legacy=0x7f100008eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:19.753 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:19.730+0000 7f1005d01700 1 -- 172.21.15.42:0/2240411278 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f100011b4c0 con 0x7f1000119c90 2019-12-05T06:11:19.753 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:19.730+0000 7f1005d01700 1 --2- 172.21.15.42:0/2240411278 >> v2:172.21.15.42:3300/0 conn(0x7f1000119c90 0x7f1000144800 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f1000144f80 tx=0x7f0ffc011de0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:19.753 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:19.730+0000 7f1005d01700 1 --2- 172.21.15.42:0/2240411278 >> v2:172.21.15.42:3300/0 conn(0x7f1000119c90 0x7f1000144800 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f1000144f80 tx=0x7f0ffc011de0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:19.753 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:19.730+0000 7f1005d01700 1 --2- 172.21.15.42:0/2240411278 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1000119c90 0x7f1000144800 secure :-1 s=READY pgs=2183 cs=0 l=1 rx=0x7f1000144f80 tx=0x7f0ffc011de0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:11:19.753 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:19.730+0000 7f1005d01700 1 --2- 172.21.15.42:0/2240411278 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1000119c90 0x7f1000144800 secure :-1 s=THROTTLE_DONE pgs=2183 cs=0 l=1 rx=0x7f1000144f80 tx=0x7f0ffc011de0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:19.754 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:19.730+0000 7f1005d01700 1 --2- 172.21.15.42:0/2240411278 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1000119c90 0x7f1000144800 secure :-1 s=THROTTLE_DONE pgs=2183 cs=0 l=1 rx=0x7f1000144f80 tx=0x7f0ffc011de0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:19.754 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:19.730+0000 7f0ff67fc700 1 -- 172.21.15.42:0/2240411278 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0ffc00f280 con 0x7f1000119c90 2019-12-05T06:11:19.754 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:19.730+0000 7f1005d01700 1 --2- 172.21.15.42:0/2240411278 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1000119c90 0x7f1000144800 secure :-1 s=THROTTLE_DONE pgs=2183 cs=0 l=1 rx=0x7f1000144f80 tx=0x7f0ffc011de0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:19.754 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:19.730+0000 7f0ff67fc700 1 -- 172.21.15.42:0/2240411278 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ffc008db0 con 0x7f1000119c90 2019-12-05T06:11:19.754 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:19.730+0000 7f1007764700 1 -- 172.21.15.42:0/2240411278 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f100011ac80 con 0x7f1000119c90 2019-12-05T06:11:19.755 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:19.730+0000 7f0ff67fc700 1 -- 172.21.15.42:0/2240411278 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0ffc005540 con 0x7f1000119c90 2019-12-05T06:11:19.755 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:19.734+0000 7f1007764700 1 -- 172.21.15.42:0/2240411278 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f100008f560 con 0x7f1000119c90 2019-12-05T06:11:19.755 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:19.734+0000 7f1005d01700 1 --2- 172.21.15.42:0/2240411278 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1000119c90 0x7f1000144800 secure :-1 s=THROTTLE_DONE pgs=2183 cs=0 l=1 rx=0x7f1000144f80 tx=0x7f0ffc011de0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:19.755 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:19.734+0000 7f0ff67fc700 1 -- 172.21.15.42:0/2240411278 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f0ffc0130b0 con 0x7f1000119c90 2019-12-05T06:11:19.755 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:19.734+0000 7f1005d01700 1 --2- 172.21.15.42:0/2240411278 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1000119c90 0x7f1000144800 secure :-1 s=THROTTLE_DONE pgs=2183 cs=0 l=1 rx=0x7f1000144f80 tx=0x7f0ffc011de0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:19.755 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:19.734+0000 7f0ff67fc700 1 --2- 172.21.15.42:0/2240411278 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0fe803ae30 0x7f0fe803d340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:11:19.755 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:19.734+0000 7f0ff67fc700 1 -- 172.21.15.42:0/2240411278 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9987+0+0 (secure 0 0 0) 0x7f0ffc04b5b0 con 0x7f1000119c90 2019-12-05T06:11:19.756 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:19.734+0000 7f1005500700 1 --2- 172.21.15.42:0/2240411278 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0fe803ae30 0x7f0fe803d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:11:19.756 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:19.734+0000 7f1005500700 1 --2- 172.21.15.42:0/2240411278 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0fe803ae30 0x7f0fe803d340 crc :-1 s=READY pgs=920 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:11:19.756 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:19.738+0000 7f1007764700 1 -- 172.21.15.42:0/2240411278 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f100005d100 con 0x7f1000119c90 2019-12-05T06:11:19.756 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:19.746+0000 7f1005d01700 1 --2- 172.21.15.42:0/2240411278 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1000119c90 0x7f1000144800 secure :-1 s=THROTTLE_DONE pgs=2183 cs=0 l=1 rx=0x7f1000144f80 tx=0x7f0ffc011de0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:19.757 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:19.746+0000 7f0ff67fc700 1 -- 172.21.15.42:0/2240411278 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f0ffc033440 con 0x7f1000119c90 2019-12-05T06:11:19.908 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:19.906+0000 7f1007764700 1 -- 172.21.15.42:0/2240411278 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f100005b8b0 con 0x7f1000119c90 2019-12-05T06:11:19.909 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:19.906+0000 7f1005d01700 1 --2- 172.21.15.42:0/2240411278 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1000119c90 0x7f1000144800 secure :-1 s=THROTTLE_DONE pgs=2183 cs=0 l=1 rx=0x7f1000144f80 tx=0x7f0ffc011de0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:19.909 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:19.906+0000 7f0ff67fc700 1 -- 172.21.15.42:0/2240411278 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 263 v263) v1 ==== 95+0+2113 (secure 0 0 0) 0x7f0ffc0323b0 con 0x7f1000119c90 2019-12-05T06:11:19.910 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 263 2019-12-05T06:11:19.910 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:11:19.910 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":263,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":10000,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/3923117701","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3923117701},{"type":"v1","addr":"172.21.15.42:6835","nonce":3923117701}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":263}],"filesystems":[{"mdsmap":{"epoch":262,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:10:14.933627+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":95,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6977},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6977":{"gid":6977,"name":"b","rank":0,"incarnation":259,"state":"up:active","state_seq":32,"addr":"172.21.15.40:6833/3355679996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3355679996},{"type":"v1","addr":"172.21.15.40:6833","nonce":3355679996}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:11:19.911 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:19.906+0000 7f1007764700 1 -- 172.21.15.42:0/2240411278 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0fe803ae30 msgr2=0x7f0fe803d340 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:19.911 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:19.906+0000 7f1007764700 1 --2- 172.21.15.42:0/2240411278 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0fe803ae30 0x7f0fe803d340 crc :-1 s=READY pgs=920 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:11:19.911 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:19.906+0000 7f1007764700 1 -- 172.21.15.42:0/2240411278 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1000119c90 msgr2=0x7f1000144800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:19.911 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:19.906+0000 7f1007764700 1 --2- 172.21.15.42:0/2240411278 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1000119c90 0x7f1000144800 secure :-1 s=READY pgs=2183 cs=0 l=1 rx=0x7f1000144f80 tx=0x7f0ffc011de0).stop 2019-12-05T06:11:19.912 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:19.906+0000 7f1007764700 1 -- 172.21.15.42:0/2240411278 shutdown_connections 2019-12-05T06:11:19.913 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:19.906+0000 7f1007764700 1 --2- 172.21.15.42:0/2240411278 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0fe803ae30 0x7f0fe803d340 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:11:19.913 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:19.906+0000 7f1007764700 1 --2- 172.21.15.42:0/2240411278 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1000119c90 0x7f1000144800 unknown :-1 s=CLOSED pgs=2183 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:11:19.913 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:19.906+0000 7f1007764700 1 --2- 172.21.15.42:0/2240411278 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f100011a850 0x7f1000117dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:11:19.913 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:19.910+0000 7f1007764700 1 -- 172.21.15.42:0/2240411278 shutdown_connections 2019-12-05T06:11:19.913 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:19.910+0000 7f1007764700 1 -- 172.21.15.42:0/2240411278 wait complete. 2019-12-05T06:11:19.913 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:19.910+0000 7f1007764700 1 -- 172.21.15.42:0/2240411278 >> 172.21.15.42:0/2240411278 conn(0x7f10000aabd0 msgr2=0x7f100011c530 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:11:19.922 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:11:21.631 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b has been restored 2019-12-05T06:11:21.923 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:11:22.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:22.022+0000 7f6d4df4c700 1 Processor -- start 2019-12-05T06:11:22.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:22.022+0000 7f6d4df4c700 1 -- start start 2019-12-05T06:11:22.032 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:22.022+0000 7f6d4df4c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6d480a7730 0x7f6d4808df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:11:22.032 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:22.022+0000 7f6d4df4c700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d480a0700 con 0x7f6d4809fad0 2019-12-05T06:11:22.032 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:22.022+0000 7f6d4df4c700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d480a9b90 con 0x7f6d4807e710 2019-12-05T06:11:22.033 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:22.022+0000 7f6d4df4c700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d480a0af0 con 0x7f6d480a7730 2019-12-05T06:11:22.033 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:22.026+0000 7f6d47fff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6d480a7730 0x7f6d4808df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:11:22.033 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:22.026+0000 7f6d47fff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6d480a7730 0x7f6d4808df70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:49992/0 (socket says 172.21.15.42:49992) 2019-12-05T06:11:22.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:22.026+0000 7f6d477fe700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f6d4807e710 0x7f6d4809f2d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:50028/0 (socket says 172.21.15.42:50028) 2019-12-05T06:11:22.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:22.026+0000 7f6d47fff700 1 -- 172.21.15.42:0/1081832790 learned_addr learned my addr 172.21.15.42:0/1081832790 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:11:22.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:22.026+0000 7f6d47fff700 1 -- 172.21.15.42:0/1081832790 >> v1:172.21.15.42:6789/0 conn(0x7f6d4807e710 legacy=0x7f6d4809f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:22.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:22.026+0000 7f6d47fff700 1 -- 172.21.15.42:0/1081832790 >> v1:172.21.15.40:6789/0 conn(0x7f6d4809fad0 legacy=0x7f6d4814efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:22.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:22.026+0000 7f6d47fff700 1 -- 172.21.15.42:0/1081832790 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d4807eb40 con 0x7f6d480a7730 2019-12-05T06:11:22.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:22.026+0000 7f6d47fff700 1 --2- 172.21.15.42:0/1081832790 >> v2:172.21.15.42:3300/0 conn(0x7f6d480a7730 0x7f6d4808df70 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f6d38008790 tx=0x7f6d3800fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:22.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:22.026+0000 7f6d47fff700 1 --2- 172.21.15.42:0/1081832790 >> v2:172.21.15.42:3300/0 conn(0x7f6d480a7730 0x7f6d4808df70 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f6d38008790 tx=0x7f6d3800fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:22.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:22.026+0000 7f6d47fff700 1 --2- 172.21.15.42:0/1081832790 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6d480a7730 0x7f6d4808df70 secure :-1 s=READY pgs=2188 cs=0 l=1 rx=0x7f6d38008790 tx=0x7f6d3800fcb0).ready entity=mon.1 client_cookie=de1f3aca3020541 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:11:22.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:22.026+0000 7f6d47fff700 1 --2- 172.21.15.42:0/1081832790 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6d480a7730 0x7f6d4808df70 secure :-1 s=THROTTLE_DONE pgs=2188 cs=0 l=1 rx=0x7f6d38008790 tx=0x7f6d3800fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:22.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:22.026+0000 7f6d46ffd700 1 -- 172.21.15.42:0/1081832790 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6d38011520 con 0x7f6d480a7730 2019-12-05T06:11:22.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:22.026+0000 7f6d47fff700 1 --2- 172.21.15.42:0/1081832790 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6d480a7730 0x7f6d4808df70 secure :-1 s=THROTTLE_DONE pgs=2188 cs=0 l=1 rx=0x7f6d38008790 tx=0x7f6d3800fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:22.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:22.026+0000 7f6d47fff700 1 --2- 172.21.15.42:0/1081832790 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6d480a7730 0x7f6d4808df70 secure :-1 s=THROTTLE_DONE pgs=2188 cs=0 l=1 rx=0x7f6d38008790 tx=0x7f6d3800fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:22.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:22.026+0000 7f6d46ffd700 1 -- 172.21.15.42:0/1081832790 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d38008ca0 con 0x7f6d480a7730 2019-12-05T06:11:22.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:22.026+0000 7f6d46ffd700 1 -- 172.21.15.42:0/1081832790 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6d38011b00 con 0x7f6d480a7730 2019-12-05T06:11:22.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:22.026+0000 7f6d4df4c700 1 -- 172.21.15.42:0/1081832790 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6d480a7730 msgr2=0x7f6d4808df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:22.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:22.026+0000 7f6d4df4c700 1 --2- 172.21.15.42:0/1081832790 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6d480a7730 0x7f6d4808df70 secure :-1 s=READY pgs=2188 cs=0 l=1 rx=0x7f6d38008790 tx=0x7f6d3800fcb0).stop 2019-12-05T06:11:22.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:22.026+0000 7f6d4df4c700 1 -- 172.21.15.42:0/1081832790 shutdown_connections 2019-12-05T06:11:22.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:22.026+0000 7f6d4df4c700 1 --2- 172.21.15.42:0/1081832790 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6d480a7730 0x7f6d4808df70 unknown :-1 s=CLOSED pgs=2188 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:11:22.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:22.026+0000 7f6d4df4c700 1 -- 172.21.15.42:0/1081832790 shutdown_connections 2019-12-05T06:11:22.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:22.026+0000 7f6d4df4c700 1 -- 172.21.15.42:0/1081832790 wait complete. 2019-12-05T06:11:22.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:22.026+0000 7f6d4df4c700 1 -- 172.21.15.42:0/1081832790 >> 172.21.15.42:0/1081832790 conn(0x7f6d480aabd0 msgr2=0x7f6d480a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:11:22.042 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:22.030+0000 7f6d4df4c700 1 Processor -- start 2019-12-05T06:11:22.042 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:22.030+0000 7f6d4df4c700 1 -- start start 2019-12-05T06:11:22.042 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:22.030+0000 7f6d4df4c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6d481476d0 0x7f6d481450a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:11:22.042 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:22.030+0000 7f6d4df4c700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d481266d0 con 0x7f6d48148450 2019-12-05T06:11:22.042 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:22.030+0000 7f6d4df4c700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d4812ed10 con 0x7f6d48118ed0 2019-12-05T06:11:22.042 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:22.030+0000 7f6d4df4c700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d481427b0 con 0x7f6d481476d0 2019-12-05T06:11:22.043 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:22.030+0000 7f6d477fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6d481476d0 0x7f6d481450a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:11:22.043 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:22.030+0000 7f6d47fff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f6d48118ed0 0x7f6d48142b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:50034/0 (socket says 172.21.15.42:50034) 2019-12-05T06:11:22.043 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:22.030+0000 7f6d47fff700 1 -- 172.21.15.42:0/2824949859 learned_addr learned my addr 172.21.15.42:0/2824949859 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:11:22.044 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:22.030+0000 7f6d477fe700 1 --2- 172.21.15.42:0/2824949859 >> v2:172.21.15.42:3300/0 conn(0x7f6d481476d0 0x7f6d481450a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:49998/0 (socket says 172.21.15.42:49998) 2019-12-05T06:11:22.044 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:22.030+0000 7f6d44ff9700 1 -- 172.21.15.42:0/2824949859 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 758820198 0 0) 0x7f6d481266d0 con 0x7f6d48148450 2019-12-05T06:11:22.044 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:22.030+0000 7f6d477fe700 1 -- 172.21.15.42:0/2824949859 >> v1:172.21.15.42:6789/0 conn(0x7f6d48118ed0 legacy=0x7f6d48142b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:22.044 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:22.030+0000 7f6d477fe700 1 -- 172.21.15.42:0/2824949859 >> v1:172.21.15.40:6789/0 conn(0x7f6d48148450 legacy=0x7f6d48125ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:22.045 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:22.030+0000 7f6d477fe700 1 -- 172.21.15.42:0/2824949859 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d4812ff50 con 0x7f6d481476d0 2019-12-05T06:11:22.045 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:22.030+0000 7f6d477fe700 1 --2- 172.21.15.42:0/2824949859 >> v2:172.21.15.42:3300/0 conn(0x7f6d481476d0 0x7f6d481450a0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f6d481473d0 tx=0x7f6d3800fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:22.045 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:22.034+0000 7f6d477fe700 1 --2- 172.21.15.42:0/2824949859 >> v2:172.21.15.42:3300/0 conn(0x7f6d481476d0 0x7f6d481450a0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f6d481473d0 tx=0x7f6d3800fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:22.045 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:22.034+0000 7f6d477fe700 1 --2- 172.21.15.42:0/2824949859 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6d481476d0 0x7f6d481450a0 secure :-1 s=READY pgs=2190 cs=0 l=1 rx=0x7f6d481473d0 tx=0x7f6d3800fcb0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:11:22.046 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:22.034+0000 7f6d477fe700 1 --2- 172.21.15.42:0/2824949859 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6d481476d0 0x7f6d481450a0 secure :-1 s=THROTTLE_DONE pgs=2190 cs=0 l=1 rx=0x7f6d481473d0 tx=0x7f6d3800fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:22.046 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:22.034+0000 7f6d477fe700 1 --2- 172.21.15.42:0/2824949859 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6d481476d0 0x7f6d481450a0 secure :-1 s=THROTTLE_DONE pgs=2190 cs=0 l=1 rx=0x7f6d481473d0 tx=0x7f6d3800fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:22.046 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:22.034+0000 7f6d477fe700 1 --2- 172.21.15.42:0/2824949859 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6d481476d0 0x7f6d481450a0 secure :-1 s=THROTTLE_DONE pgs=2190 cs=0 l=1 rx=0x7f6d481473d0 tx=0x7f6d3800fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:22.047 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:22.034+0000 7f6d44ff9700 1 -- 172.21.15.42:0/2824949859 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6d3c006850 con 0x7f6d481476d0 2019-12-05T06:11:22.047 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:22.034+0000 7f6d44ff9700 1 -- 172.21.15.42:0/2824949859 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d3c006470 con 0x7f6d481476d0 2019-12-05T06:11:22.047 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:22.034+0000 7f6d44ff9700 1 -- 172.21.15.42:0/2824949859 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6d3c0110e0 con 0x7f6d481476d0 2019-12-05T06:11:22.047 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:22.034+0000 7f6d4df4c700 1 -- 172.21.15.42:0/2824949859 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d4812dd00 con 0x7f6d481476d0 2019-12-05T06:11:22.047 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:22.034+0000 7f6d4df4c700 1 -- 172.21.15.42:0/2824949859 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d4812e060 con 0x7f6d481476d0 2019-12-05T06:11:22.047 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:22.034+0000 7f6d477fe700 1 --2- 172.21.15.42:0/2824949859 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6d481476d0 0x7f6d481450a0 secure :-1 s=THROTTLE_DONE pgs=2190 cs=0 l=1 rx=0x7f6d481473d0 tx=0x7f6d3800fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:22.047 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:22.034+0000 7f6d44ff9700 1 -- 172.21.15.42:0/2824949859 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f6d3c0113b0 con 0x7f6d481476d0 2019-12-05T06:11:22.048 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:22.034+0000 7f6d477fe700 1 --2- 172.21.15.42:0/2824949859 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6d481476d0 0x7f6d481450a0 secure :-1 s=THROTTLE_DONE pgs=2190 cs=0 l=1 rx=0x7f6d481473d0 tx=0x7f6d3800fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:22.048 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:22.034+0000 7f6d44ff9700 1 --2- 172.21.15.42:0/2824949859 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6d2803ac80 0x7f6d2803d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:11:22.048 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:22.034+0000 7f6d44ff9700 1 -- 172.21.15.42:0/2824949859 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9987+0+0 (secure 0 0 0) 0x7f6d3c04af40 con 0x7f6d481476d0 2019-12-05T06:11:22.048 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:22.034+0000 7f6d47fff700 1 --2- 172.21.15.42:0/2824949859 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6d2803ac80 0x7f6d2803d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:11:22.048 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:22.038+0000 7f6d47fff700 1 --2- 172.21.15.42:0/2824949859 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6d2803ac80 0x7f6d2803d190 crc :-1 s=READY pgs=921 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:11:22.049 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:22.038+0000 7f6d4df4c700 1 -- 172.21.15.42:0/2824949859 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d4805d100 con 0x7f6d481476d0 2019-12-05T06:11:22.049 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:22.046+0000 7f6d477fe700 1 --2- 172.21.15.42:0/2824949859 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6d481476d0 0x7f6d481450a0 secure :-1 s=THROTTLE_DONE pgs=2190 cs=0 l=1 rx=0x7f6d481473d0 tx=0x7f6d3800fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:22.049 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:22.046+0000 7f6d44ff9700 1 -- 172.21.15.42:0/2824949859 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f6d3c031fa0 con 0x7f6d481476d0 2019-12-05T06:11:22.209 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:22.206+0000 7f6d4df4c700 1 -- 172.21.15.42:0/2824949859 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6d4805b8b0 con 0x7f6d481476d0 2019-12-05T06:11:22.210 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:22.206+0000 7f6d477fe700 1 --2- 172.21.15.42:0/2824949859 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6d481476d0 0x7f6d481450a0 secure :-1 s=THROTTLE_DONE pgs=2190 cs=0 l=1 rx=0x7f6d481473d0 tx=0x7f6d3800fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:22.210 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:22.206+0000 7f6d44ff9700 1 -- 172.21.15.42:0/2824949859 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 263 v263) v1 ==== 95+0+2113 (secure 0 0 0) 0x7f6d3c032200 con 0x7f6d481476d0 2019-12-05T06:11:22.211 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 263 2019-12-05T06:11:22.211 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:11:22.211 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":263,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":10000,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/3923117701","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3923117701},{"type":"v1","addr":"172.21.15.42:6835","nonce":3923117701}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":263}],"filesystems":[{"mdsmap":{"epoch":262,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:10:14.933627+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":95,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6977},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6977":{"gid":6977,"name":"b","rank":0,"incarnation":259,"state":"up:active","state_seq":32,"addr":"172.21.15.40:6833/3355679996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3355679996},{"type":"v1","addr":"172.21.15.40:6833","nonce":3355679996}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:11:22.214 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:22.210+0000 7f6d4df4c700 1 -- 172.21.15.42:0/2824949859 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6d2803ac80 msgr2=0x7f6d2803d190 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:22.214 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:22.210+0000 7f6d4df4c700 1 --2- 172.21.15.42:0/2824949859 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6d2803ac80 0x7f6d2803d190 crc :-1 s=READY pgs=921 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:11:22.215 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:22.210+0000 7f6d4df4c700 1 -- 172.21.15.42:0/2824949859 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6d481476d0 msgr2=0x7f6d481450a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:22.215 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:22.210+0000 7f6d4df4c700 1 --2- 172.21.15.42:0/2824949859 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6d481476d0 0x7f6d481450a0 secure :-1 s=READY pgs=2190 cs=0 l=1 rx=0x7f6d481473d0 tx=0x7f6d3800fcb0).stop 2019-12-05T06:11:22.215 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:22.210+0000 7f6d4df4c700 1 -- 172.21.15.42:0/2824949859 shutdown_connections 2019-12-05T06:11:22.215 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:22.210+0000 7f6d4df4c700 1 --2- 172.21.15.42:0/2824949859 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6d2803ac80 0x7f6d2803d190 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:11:22.215 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:22.210+0000 7f6d4df4c700 1 --2- 172.21.15.42:0/2824949859 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6d481476d0 0x7f6d481450a0 unknown :-1 s=CLOSED pgs=2190 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:11:22.215 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:22.210+0000 7f6d4df4c700 1 -- 172.21.15.42:0/2824949859 shutdown_connections 2019-12-05T06:11:22.215 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:22.210+0000 7f6d4df4c700 1 -- 172.21.15.42:0/2824949859 wait complete. 2019-12-05T06:11:22.216 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:22.210+0000 7f6d4df4c700 1 -- 172.21.15.42:0/2824949859 >> 172.21.15.42:0/2824949859 conn(0x7f6d4808fd90 msgr2=0x7f6d480596f0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:11:22.229 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:11:24.230 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:11:24.334 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:24.326+0000 7f5abfe1f700 1 Processor -- start 2019-12-05T06:11:24.334 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:24.326+0000 7f5abfe1f700 1 -- start start 2019-12-05T06:11:24.334 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:24.326+0000 7f5abfe1f700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5ab809fae0 0x7f5ab814fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:11:24.334 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:24.326+0000 7f5abfe1f700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ab80a0710 con 0x7f5ab808df80 2019-12-05T06:11:24.334 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:24.326+0000 7f5abfe1f700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ab807ef20 con 0x7f5ab80a7740 2019-12-05T06:11:24.335 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:24.326+0000 7f5abfe1f700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ab80a0b00 con 0x7f5ab809fae0 2019-12-05T06:11:24.335 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:24.326+0000 7f5abebbd700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5ab809fae0 0x7f5ab814fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:11:24.335 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:24.326+0000 7f5abebbd700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5ab809fae0 0x7f5ab814fff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:50010/0 (socket says 172.21.15.42:50010) 2019-12-05T06:11:24.335 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:24.326+0000 7f5abebbd700 1 -- 172.21.15.42:0/4291018562 learned_addr learned my addr 172.21.15.42:0/4291018562 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:11:24.335 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:24.326+0000 7f5abebbd700 1 -- 172.21.15.42:0/4291018562 >> v1:172.21.15.42:6789/0 conn(0x7f5ab80a7740 legacy=0x7f5ab807e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:24.335 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:24.326+0000 7f5abebbd700 1 -- 172.21.15.42:0/4291018562 >> v1:172.21.15.40:6789/0 conn(0x7f5ab808df80 legacy=0x7f5ab809f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:24.335 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:24.326+0000 7f5abebbd700 1 -- 172.21.15.42:0/4291018562 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ab80a9ba0 con 0x7f5ab809fae0 2019-12-05T06:11:24.336 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:24.326+0000 7f5abebbd700 1 --2- 172.21.15.42:0/4291018562 >> v2:172.21.15.42:3300/0 conn(0x7f5ab809fae0 0x7f5ab814fff0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f5ab0008790 tx=0x7f5ab000fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:24.336 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:24.326+0000 7f5abebbd700 1 --2- 172.21.15.42:0/4291018562 >> v2:172.21.15.42:3300/0 conn(0x7f5ab809fae0 0x7f5ab814fff0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f5ab0008790 tx=0x7f5ab000fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:24.336 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:24.326+0000 7f5abebbd700 1 --2- 172.21.15.42:0/4291018562 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5ab809fae0 0x7f5ab814fff0 secure :-1 s=READY pgs=2193 cs=0 l=1 rx=0x7f5ab0008790 tx=0x7f5ab000fcb0).ready entity=mon.1 client_cookie=26f5e06114e52a4d server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:11:24.336 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:24.326+0000 7f5abebbd700 1 --2- 172.21.15.42:0/4291018562 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5ab809fae0 0x7f5ab814fff0 secure :-1 s=THROTTLE_DONE pgs=2193 cs=0 l=1 rx=0x7f5ab0008790 tx=0x7f5ab000fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:24.336 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:24.326+0000 7f5abebbd700 1 --2- 172.21.15.42:0/4291018562 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5ab809fae0 0x7f5ab814fff0 secure :-1 s=THROTTLE_DONE pgs=2193 cs=0 l=1 rx=0x7f5ab0008790 tx=0x7f5ab000fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:24.337 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:24.326+0000 7f5abd3ba700 1 -- 172.21.15.42:0/4291018562 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5ab0008c90 con 0x7f5ab809fae0 2019-12-05T06:11:24.337 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:24.326+0000 7f5abebbd700 1 --2- 172.21.15.42:0/4291018562 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5ab809fae0 0x7f5ab814fff0 secure :-1 s=THROTTLE_DONE pgs=2193 cs=0 l=1 rx=0x7f5ab0008790 tx=0x7f5ab000fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:24.337 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:24.326+0000 7f5abd3ba700 1 -- 172.21.15.42:0/4291018562 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ab00088b0 con 0x7f5ab809fae0 2019-12-05T06:11:24.337 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:24.326+0000 7f5abd3ba700 1 -- 172.21.15.42:0/4291018562 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5ab0011b80 con 0x7f5ab809fae0 2019-12-05T06:11:24.337 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:24.326+0000 7f5abfe1f700 1 -- 172.21.15.42:0/4291018562 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5ab809fae0 msgr2=0x7f5ab814fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:24.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:24.326+0000 7f5abfe1f700 1 --2- 172.21.15.42:0/4291018562 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5ab809fae0 0x7f5ab814fff0 secure :-1 s=READY pgs=2193 cs=0 l=1 rx=0x7f5ab0008790 tx=0x7f5ab000fcb0).stop 2019-12-05T06:11:24.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:24.330+0000 7f5abfe1f700 1 -- 172.21.15.42:0/4291018562 shutdown_connections 2019-12-05T06:11:24.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:24.330+0000 7f5abfe1f700 1 --2- 172.21.15.42:0/4291018562 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5ab809fae0 0x7f5ab814fff0 unknown :-1 s=CLOSED pgs=2193 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:11:24.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:24.330+0000 7f5abfe1f700 1 -- 172.21.15.42:0/4291018562 shutdown_connections 2019-12-05T06:11:24.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:24.330+0000 7f5abfe1f700 1 -- 172.21.15.42:0/4291018562 wait complete. 2019-12-05T06:11:24.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:24.330+0000 7f5abfe1f700 1 -- 172.21.15.42:0/4291018562 >> 172.21.15.42:0/4291018562 conn(0x7f5ab80aabe0 msgr2=0x7f5ab80a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:11:24.343 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:24.330+0000 7f5abfe1f700 1 Processor -- start 2019-12-05T06:11:24.344 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:24.330+0000 7f5abfe1f700 1 -- start start 2019-12-05T06:11:24.344 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:24.330+0000 7f5abfe1f700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5ab81207c0 0x7f5ab8120bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:11:24.344 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:24.330+0000 7f5abfe1f700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ab80a31e0 con 0x7f5ab81437f0 2019-12-05T06:11:24.344 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:24.330+0000 7f5abfe1f700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ab8121140 con 0x7f5ab80a5f10 2019-12-05T06:11:24.344 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:24.330+0000 7f5abfe1f700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ab8143c20 con 0x7f5ab81207c0 2019-12-05T06:11:24.344 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:24.330+0000 7f5abe3bc700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f5ab80a5f10 0x7f5ab80a38a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:50050/0 (socket says 172.21.15.42:50050) 2019-12-05T06:11:24.345 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:24.330+0000 7f5abebbd700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5ab81207c0 0x7f5ab8120bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:11:24.345 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:24.330+0000 7f5abe3bc700 1 -- 172.21.15.42:0/1612327144 learned_addr learned my addr 172.21.15.42:0/1612327144 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:11:24.345 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:24.334+0000 7f5aaaffd700 1 -- 172.21.15.42:0/1612327144 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3643519901 0 0) 0x7f5ab8121140 con 0x7f5ab80a5f10 2019-12-05T06:11:24.345 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:24.334+0000 7f5abebbd700 1 -- 172.21.15.42:0/1612327144 >> v1:172.21.15.42:6789/0 conn(0x7f5ab80a5f10 legacy=0x7f5ab80a38a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:24.346 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:24.334+0000 7f5abebbd700 1 -- 172.21.15.42:0/1612327144 >> v1:172.21.15.40:6789/0 conn(0x7f5ab81437f0 legacy=0x7f5ab80a29e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:24.346 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:24.334+0000 7f5abebbd700 1 -- 172.21.15.42:0/1612327144 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ab812dfd0 con 0x7f5ab81207c0 2019-12-05T06:11:24.346 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:24.334+0000 7f5abebbd700 1 --2- 172.21.15.42:0/1612327144 >> v2:172.21.15.40:3300/0 conn(0x7f5ab81207c0 0x7f5ab8120bf0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f5ab0008790 tx=0x7f5ab0008b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:24.346 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:24.334+0000 7f5abebbd700 1 --2- 172.21.15.42:0/1612327144 >> v2:172.21.15.40:3300/0 conn(0x7f5ab81207c0 0x7f5ab8120bf0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f5ab0008790 tx=0x7f5ab0008b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:24.346 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:24.334+0000 7f5abebbd700 1 --2- 172.21.15.42:0/1612327144 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ab81207c0 0x7f5ab8120bf0 secure :-1 s=READY pgs=1661 cs=0 l=1 rx=0x7f5ab0008790 tx=0x7f5ab0008b60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:11:24.346 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:24.334+0000 7f5abebbd700 1 --2- 172.21.15.42:0/1612327144 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ab81207c0 0x7f5ab8120bf0 secure :-1 s=THROTTLE_DONE pgs=1661 cs=0 l=1 rx=0x7f5ab0008790 tx=0x7f5ab0008b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:24.347 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:24.334+0000 7f5abebbd700 1 --2- 172.21.15.42:0/1612327144 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ab81207c0 0x7f5ab8120bf0 secure :-1 s=THROTTLE_DONE pgs=1661 cs=0 l=1 rx=0x7f5ab0008790 tx=0x7f5ab0008b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:24.347 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:24.334+0000 7f5aaaffd700 1 -- 172.21.15.42:0/1612327144 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5ab000f470 con 0x7f5ab81207c0 2019-12-05T06:11:24.347 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:24.334+0000 7f5abebbd700 1 --2- 172.21.15.42:0/1612327144 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ab81207c0 0x7f5ab8120bf0 secure :-1 s=THROTTLE_DONE pgs=1661 cs=0 l=1 rx=0x7f5ab0008790 tx=0x7f5ab0008b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:24.347 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:24.334+0000 7f5aaaffd700 1 -- 172.21.15.42:0/1612327144 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ab00117a0 con 0x7f5ab81207c0 2019-12-05T06:11:24.347 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:24.334+0000 7f5abfe1f700 1 -- 172.21.15.42:0/1612327144 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ab812bfb0 con 0x7f5ab81207c0 2019-12-05T06:11:24.347 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:24.334+0000 7f5aaaffd700 1 -- 172.21.15.42:0/1612327144 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5ab0005730 con 0x7f5ab81207c0 2019-12-05T06:11:24.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:24.334+0000 7f5abfe1f700 1 -- 172.21.15.42:0/1612327144 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ab812c500 con 0x7f5ab81207c0 2019-12-05T06:11:24.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:24.334+0000 7f5abebbd700 1 --2- 172.21.15.42:0/1612327144 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ab81207c0 0x7f5ab8120bf0 secure :-1 s=THROTTLE_DONE pgs=1661 cs=0 l=1 rx=0x7f5ab0008790 tx=0x7f5ab0008b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:24.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:24.334+0000 7f5aaaffd700 1 -- 172.21.15.42:0/1612327144 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f5ab0005a40 con 0x7f5ab81207c0 2019-12-05T06:11:24.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:24.334+0000 7f5abebbd700 1 --2- 172.21.15.42:0/1612327144 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ab81207c0 0x7f5ab8120bf0 secure :-1 s=THROTTLE_DONE pgs=1661 cs=0 l=1 rx=0x7f5ab0008790 tx=0x7f5ab0008b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:24.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:24.334+0000 7f5aaaffd700 1 --2- 172.21.15.42:0/1612327144 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5aa003aca0 0x7f5aa003d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:11:24.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:24.334+0000 7f5aaaffd700 1 -- 172.21.15.42:0/1612327144 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9987+0+0 (secure 0 0 0) 0x7f5ab004b970 con 0x7f5ab81207c0 2019-12-05T06:11:24.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:24.334+0000 7f5abe3bc700 1 --2- 172.21.15.42:0/1612327144 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5aa003aca0 0x7f5aa003d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:11:24.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:24.338+0000 7f5abe3bc700 1 --2- 172.21.15.42:0/1612327144 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5aa003aca0 0x7f5aa003d1b0 crc :-1 s=READY pgs=922 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:11:24.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:24.342+0000 7f5abfe1f700 1 -- 172.21.15.42:0/1612327144 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ab805d110 con 0x7f5ab81207c0 2019-12-05T06:11:24.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:24.354+0000 7f5abebbd700 1 --2- 172.21.15.42:0/1612327144 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ab81207c0 0x7f5ab8120bf0 secure :-1 s=THROTTLE_DONE pgs=1661 cs=0 l=1 rx=0x7f5ab0008790 tx=0x7f5ab0008b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:24.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:24.354+0000 7f5aaaffd700 1 -- 172.21.15.42:0/1612327144 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f5ab0008de0 con 0x7f5ab81207c0 2019-12-05T06:11:24.527 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:24.522+0000 7f5abfe1f700 1 -- 172.21.15.42:0/1612327144 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5ab805b8c0 con 0x7f5ab81207c0 2019-12-05T06:11:24.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:24.526+0000 7f5abebbd700 1 --2- 172.21.15.42:0/1612327144 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ab81207c0 0x7f5ab8120bf0 secure :-1 s=THROTTLE_DONE pgs=1661 cs=0 l=1 rx=0x7f5ab0008790 tx=0x7f5ab0008b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:24.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:24.526+0000 7f5aaaffd700 1 -- 172.21.15.42:0/1612327144 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 263 v263) v1 ==== 95+0+2113 (secure 0 0 0) 0x7f5ab0033820 con 0x7f5ab81207c0 2019-12-05T06:11:24.530 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 263 2019-12-05T06:11:24.530 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:11:24.530 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":263,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":10000,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/3923117701","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3923117701},{"type":"v1","addr":"172.21.15.42:6835","nonce":3923117701}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":263}],"filesystems":[{"mdsmap":{"epoch":262,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:10:14.933627+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":95,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6977},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6977":{"gid":6977,"name":"b","rank":0,"incarnation":259,"state":"up:active","state_seq":32,"addr":"172.21.15.40:6833/3355679996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3355679996},{"type":"v1","addr":"172.21.15.40:6833","nonce":3355679996}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:11:24.532 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:24.530+0000 7f5abfe1f700 1 -- 172.21.15.42:0/1612327144 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5aa003aca0 msgr2=0x7f5aa003d1b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:24.533 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:24.530+0000 7f5abfe1f700 1 --2- 172.21.15.42:0/1612327144 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5aa003aca0 0x7f5aa003d1b0 crc :-1 s=READY pgs=922 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:11:24.533 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:24.530+0000 7f5abfe1f700 1 -- 172.21.15.42:0/1612327144 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ab81207c0 msgr2=0x7f5ab8120bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:24.533 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:24.530+0000 7f5abfe1f700 1 --2- 172.21.15.42:0/1612327144 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ab81207c0 0x7f5ab8120bf0 secure :-1 s=READY pgs=1661 cs=0 l=1 rx=0x7f5ab0008790 tx=0x7f5ab0008b60).stop 2019-12-05T06:11:24.533 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:24.530+0000 7f5abfe1f700 1 -- 172.21.15.42:0/1612327144 shutdown_connections 2019-12-05T06:11:24.534 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:24.530+0000 7f5abfe1f700 1 --2- 172.21.15.42:0/1612327144 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5aa003aca0 0x7f5aa003d1b0 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:11:24.535 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:24.530+0000 7f5abfe1f700 1 --2- 172.21.15.42:0/1612327144 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ab81207c0 0x7f5ab8120bf0 unknown :-1 s=CLOSED pgs=1661 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:11:24.535 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:24.530+0000 7f5abfe1f700 1 -- 172.21.15.42:0/1612327144 shutdown_connections 2019-12-05T06:11:24.535 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:24.530+0000 7f5abfe1f700 1 -- 172.21.15.42:0/1612327144 wait complete. 2019-12-05T06:11:24.536 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:24.530+0000 7f5abfe1f700 1 -- 172.21.15.42:0/1612327144 >> 172.21.15.42:0/1612327144 conn(0x7f5ab80a82a0 msgr2=0x7f5ab807db50 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:11:24.543 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:11:26.544 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:11:26.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:26.642+0000 7f9d932eb700 1 Processor -- start 2019-12-05T06:11:26.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:26.642+0000 7f9d932eb700 1 -- start start 2019-12-05T06:11:26.654 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:26.642+0000 7f9d932eb700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f9d8c0a7740 0x7f9d8c08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:11:26.654 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:26.642+0000 7f9d932eb700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9d8c07e720 0x7f9d8c07eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:11:26.654 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:26.642+0000 7f9d932eb700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d8c0a0710 con 0x7f9d8c09f2e0 2019-12-05T06:11:26.654 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:26.642+0000 7f9d932eb700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d8c0a0b00 con 0x7f9d8c0a7740 2019-12-05T06:11:26.655 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:26.642+0000 7f9d932eb700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d8c08e5b0 con 0x7f9d8c07e720 2019-12-05T06:11:26.655 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:26.642+0000 7f9d91888700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f9d8c0a7740 0x7f9d8c08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:11:26.655 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:26.642+0000 7f9d92089700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f9d8c09f2e0 0x7f9d8c09f710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:50072/0 (socket says 172.21.15.42:50072) 2019-12-05T06:11:26.655 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:26.642+0000 7f9d91888700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f9d8c0a7740 0x7f9d8c08df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:50030/0 (socket says 172.21.15.42:50030) 2019-12-05T06:11:26.656 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:26.642+0000 7f9d92089700 1 -- 172.21.15.42:0/3322126271 learned_addr learned my addr 172.21.15.42:0/3322126271 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:11:26.656 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:26.642+0000 7f9d91087700 1 --2- 172.21.15.42:0/3322126271 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9d8c07e720 0x7f9d8c07eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:11:26.656 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:26.646+0000 7f9d90886700 1 -- 172.21.15.42:0/3322126271 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3034052851 0 0) 0x7f9d8c0a0710 con 0x7f9d8c09f2e0 2019-12-05T06:11:26.656 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:26.646+0000 7f9d90886700 1 -- 172.21.15.42:0/3322126271 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d780030b0 con 0x7f9d8c09f2e0 2019-12-05T06:11:26.656 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:26.646+0000 7f9d91087700 1 -- 172.21.15.42:0/3322126271 >> v2:172.21.15.42:3300/0 conn(0x7f9d8c0a7740 msgr2=0x7f9d8c08df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:11:26.657 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:26.646+0000 7f9d91087700 1 --2- 172.21.15.42:0/3322126271 >> v2:172.21.15.42:3300/0 conn(0x7f9d8c0a7740 0x7f9d8c08df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:11:26.657 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:26.646+0000 7f9d91087700 1 -- 172.21.15.42:0/3322126271 >> v1:172.21.15.42:6789/0 conn(0x7f9d8c09f2e0 legacy=0x7f9d8c09f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:26.657 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:26.646+0000 7f9d91087700 1 -- 172.21.15.42:0/3322126271 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d8c07f0a0 con 0x7f9d8c07e720 2019-12-05T06:11:26.657 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:26.646+0000 7f9d91087700 1 --2- 172.21.15.42:0/3322126271 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9d8c07e720 0x7f9d8c07eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f9d7c008790 tx=0x7f9d7c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:26.658 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:26.646+0000 7f9d91087700 1 --2- 172.21.15.42:0/3322126271 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9d8c07e720 0x7f9d8c07eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f9d7c008790 tx=0x7f9d7c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:26.658 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:26.646+0000 7f9d91087700 1 --2- 172.21.15.42:0/3322126271 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9d8c07e720 0x7f9d8c07eb50 secure :-1 s=READY pgs=480 cs=0 l=1 rx=0x7f9d7c008790 tx=0x7f9d7c00fc70).ready entity=mon.2 client_cookie=e0fc0069fd12574 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:11:26.658 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:26.646+0000 7f9d91087700 1 --2- 172.21.15.42:0/3322126271 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9d8c07e720 0x7f9d8c07eb50 secure :-1 s=THROTTLE_DONE pgs=480 cs=0 l=1 rx=0x7f9d7c008790 tx=0x7f9d7c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:26.659 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:26.646+0000 7f9d90886700 1 -- 172.21.15.42:0/3322126271 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9d7c0112f0 con 0x7f9d8c07e720 2019-12-05T06:11:26.659 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:26.646+0000 7f9d91087700 1 --2- 172.21.15.42:0/3322126271 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9d8c07e720 0x7f9d8c07eb50 secure :-1 s=THROTTLE_DONE pgs=480 cs=0 l=1 rx=0x7f9d7c008790 tx=0x7f9d7c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:26.659 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:26.646+0000 7f9d90886700 1 -- 172.21.15.42:0/3322126271 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d7c008ad0 con 0x7f9d8c07e720 2019-12-05T06:11:26.659 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:26.646+0000 7f9d91087700 1 --2- 172.21.15.42:0/3322126271 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9d8c07e720 0x7f9d8c07eb50 secure :-1 s=THROTTLE_DONE pgs=480 cs=0 l=1 rx=0x7f9d7c008790 tx=0x7f9d7c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:26.660 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:26.646+0000 7f9d90886700 1 -- 172.21.15.42:0/3322126271 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9d7c0118d0 con 0x7f9d8c07e720 2019-12-05T06:11:26.660 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:26.646+0000 7f9d932eb700 1 -- 172.21.15.42:0/3322126271 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9d8c07e720 msgr2=0x7f9d8c07eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:26.660 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:26.646+0000 7f9d932eb700 1 --2- 172.21.15.42:0/3322126271 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9d8c07e720 0x7f9d8c07eb50 secure :-1 s=READY pgs=480 cs=0 l=1 rx=0x7f9d7c008790 tx=0x7f9d7c00fc70).stop 2019-12-05T06:11:26.660 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:26.646+0000 7f9d932eb700 1 -- 172.21.15.42:0/3322126271 shutdown_connections 2019-12-05T06:11:26.661 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:26.646+0000 7f9d932eb700 1 --2- 172.21.15.42:0/3322126271 >> v2:172.21.15.42:3300/0 conn(0x7f9d8c0a7740 0x7f9d8c08df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:11:26.661 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:26.646+0000 7f9d932eb700 1 --2- 172.21.15.42:0/3322126271 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9d8c07e720 0x7f9d8c07eb50 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:11:26.661 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:26.646+0000 7f9d932eb700 1 -- 172.21.15.42:0/3322126271 shutdown_connections 2019-12-05T06:11:26.661 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:26.646+0000 7f9d932eb700 1 -- 172.21.15.42:0/3322126271 wait complete. 2019-12-05T06:11:26.661 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:26.646+0000 7f9d932eb700 1 -- 172.21.15.42:0/3322126271 >> 172.21.15.42:0/3322126271 conn(0x7f9d8c0aabe0 msgr2=0x7f9d8c0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:11:26.667 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:26.650+0000 7f9d932eb700 1 Processor -- start 2019-12-05T06:11:26.668 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:26.650+0000 7f9d932eb700 1 -- start start 2019-12-05T06:11:26.668 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:26.650+0000 7f9d932eb700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f9d8c133710 0x7f9d8c124260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:11:26.668 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:26.654+0000 7f9d932eb700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9d8c116cd0 0x7f9d8c118170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:11:26.668 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:26.654+0000 7f9d932eb700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d8c1247b0 con 0x7f9d8c11b3c0 2019-12-05T06:11:26.669 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:26.654+0000 7f9d932eb700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d8c11ae80 con 0x7f9d8c133710 2019-12-05T06:11:26.669 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:26.654+0000 7f9d932eb700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d8c11b120 con 0x7f9d8c116cd0 2019-12-05T06:11:26.670 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:26.654+0000 7f9d91087700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f9d8c11b3c0 0x7f9d8c1164d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:50076/0 (socket says 172.21.15.42:50076) 2019-12-05T06:11:26.670 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:26.654+0000 7f9d91087700 1 -- 172.21.15.42:0/2324225891 learned_addr learned my addr 172.21.15.42:0/2324225891 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:11:26.670 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:26.654+0000 7f9d91888700 1 --2- 172.21.15.42:0/2324225891 >> v2:172.21.15.40:3300/0 conn(0x7f9d8c133710 0x7f9d8c124260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:11:26.670 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:26.654+0000 7f9d92089700 1 --2- 172.21.15.42:0/2324225891 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9d8c116cd0 0x7f9d8c118170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:11:26.671 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:26.654+0000 7f9d827fc700 1 -- 172.21.15.42:0/2324225891 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1450048795 0 0) 0x7f9d8c1247b0 con 0x7f9d8c11b3c0 2019-12-05T06:11:26.671 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:26.654+0000 7f9d827fc700 1 -- 172.21.15.42:0/2324225891 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d740030b0 con 0x7f9d8c11b3c0 2019-12-05T06:11:26.671 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:26.654+0000 7f9d91888700 1 -- 172.21.15.42:0/2324225891 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9d8c116cd0 msgr2=0x7f9d8c118170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:26.671 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:26.654+0000 7f9d91888700 1 --2- 172.21.15.42:0/2324225891 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9d8c116cd0 0x7f9d8c118170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:11:26.671 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:26.654+0000 7f9d91888700 1 -- 172.21.15.42:0/2324225891 >> v1:172.21.15.42:6789/0 conn(0x7f9d8c11b3c0 legacy=0x7f9d8c1164d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:26.672 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:26.654+0000 7f9d91888700 1 -- 172.21.15.42:0/2324225891 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d8c11b7f0 con 0x7f9d8c133710 2019-12-05T06:11:26.672 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:26.654+0000 7f9d91888700 1 --2- 172.21.15.42:0/2324225891 >> v2:172.21.15.40:3300/0 conn(0x7f9d8c133710 0x7f9d8c124260 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f9d8c11b040 tx=0x7f9d88010a10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:26.672 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:26.654+0000 7f9d91888700 1 --2- 172.21.15.42:0/2324225891 >> v2:172.21.15.40:3300/0 conn(0x7f9d8c133710 0x7f9d8c124260 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f9d8c11b040 tx=0x7f9d88010a10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:26.672 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:26.654+0000 7f9d91888700 1 --2- 172.21.15.42:0/2324225891 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9d8c133710 0x7f9d8c124260 secure :-1 s=READY pgs=1662 cs=0 l=1 rx=0x7f9d8c11b040 tx=0x7f9d88010a10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:11:26.673 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:26.654+0000 7f9d91888700 1 --2- 172.21.15.42:0/2324225891 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9d8c133710 0x7f9d8c124260 secure :-1 s=THROTTLE_DONE pgs=1662 cs=0 l=1 rx=0x7f9d8c11b040 tx=0x7f9d88010a10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:26.673 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:26.654+0000 7f9d827fc700 1 -- 172.21.15.42:0/2324225891 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9d88005920 con 0x7f9d8c133710 2019-12-05T06:11:26.674 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:26.654+0000 7f9d91888700 1 --2- 172.21.15.42:0/2324225891 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9d8c133710 0x7f9d8c124260 secure :-1 s=THROTTLE_DONE pgs=1662 cs=0 l=1 rx=0x7f9d8c11b040 tx=0x7f9d88010a10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:26.674 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:26.654+0000 7f9d827fc700 1 -- 172.21.15.42:0/2324225891 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d88005540 con 0x7f9d8c133710 2019-12-05T06:11:26.674 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:26.654+0000 7f9d91888700 1 --2- 172.21.15.42:0/2324225891 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9d8c133710 0x7f9d8c124260 secure :-1 s=THROTTLE_DONE pgs=1662 cs=0 l=1 rx=0x7f9d8c11b040 tx=0x7f9d88010a10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:26.674 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:26.654+0000 7f9d932eb700 1 -- 172.21.15.42:0/2324225891 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d8c1186c0 con 0x7f9d8c133710 2019-12-05T06:11:26.675 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:26.654+0000 7f9d827fc700 1 -- 172.21.15.42:0/2324225891 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9d88012500 con 0x7f9d8c133710 2019-12-05T06:11:26.675 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:26.654+0000 7f9d932eb700 1 -- 172.21.15.42:0/2324225891 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d8c118bf0 con 0x7f9d8c133710 2019-12-05T06:11:26.675 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:26.654+0000 7f9d91888700 1 --2- 172.21.15.42:0/2324225891 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9d8c133710 0x7f9d8c124260 secure :-1 s=THROTTLE_DONE pgs=1662 cs=0 l=1 rx=0x7f9d8c11b040 tx=0x7f9d88010a10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:26.675 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:26.658+0000 7f9d827fc700 1 -- 172.21.15.42:0/2324225891 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f9d880127e0 con 0x7f9d8c133710 2019-12-05T06:11:26.676 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:26.658+0000 7f9d91888700 1 --2- 172.21.15.42:0/2324225891 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9d8c133710 0x7f9d8c124260 secure :-1 s=THROTTLE_DONE pgs=1662 cs=0 l=1 rx=0x7f9d8c11b040 tx=0x7f9d88010a10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:26.676 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:26.658+0000 7f9d932eb700 1 -- 172.21.15.42:0/2324225891 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d8c05d110 con 0x7f9d8c133710 2019-12-05T06:11:26.676 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:26.658+0000 7f9d827fc700 1 --2- 172.21.15.42:0/2324225891 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9d7403b280 0x7f9d7403d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:11:26.676 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:26.658+0000 7f9d827fc700 1 -- 172.21.15.42:0/2324225891 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9987+0+0 (secure 0 0 0) 0x7f9d8804aad0 con 0x7f9d8c133710 2019-12-05T06:11:26.676 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:26.658+0000 7f9d91087700 1 --2- 172.21.15.42:0/2324225891 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9d7403b280 0x7f9d7403d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:11:26.677 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:26.658+0000 7f9d91087700 1 --2- 172.21.15.42:0/2324225891 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9d7403b280 0x7f9d7403d790 crc :-1 s=READY pgs=923 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:11:26.678 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:26.666+0000 7f9d91888700 1 --2- 172.21.15.42:0/2324225891 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9d8c133710 0x7f9d8c124260 secure :-1 s=THROTTLE_DONE pgs=1662 cs=0 l=1 rx=0x7f9d8c11b040 tx=0x7f9d88010a10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:26.678 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:26.666+0000 7f9d827fc700 1 -- 172.21.15.42:0/2324225891 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f9d88031b30 con 0x7f9d8c133710 2019-12-05T06:11:26.834 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:26.830+0000 7f9d932eb700 1 -- 172.21.15.42:0/2324225891 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9d8c11ac60 con 0x7f9d8c133710 2019-12-05T06:11:26.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:26.830+0000 7f9d91888700 1 --2- 172.21.15.42:0/2324225891 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9d8c133710 0x7f9d8c124260 secure :-1 s=THROTTLE_DONE pgs=1662 cs=0 l=1 rx=0x7f9d8c11b040 tx=0x7f9d88010a10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:26.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:26.830+0000 7f9d827fc700 1 -- 172.21.15.42:0/2324225891 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 263 v263) v1 ==== 95+0+2113 (secure 0 0 0) 0x7f9d88031d90 con 0x7f9d8c133710 2019-12-05T06:11:26.838 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:11:26.838 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":263,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":10000,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/3923117701","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3923117701},{"type":"v1","addr":"172.21.15.42:6835","nonce":3923117701}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":263}],"filesystems":[{"mdsmap":{"epoch":262,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:10:14.933627+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":95,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6977},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6977":{"gid":6977,"name":"b","rank":0,"incarnation":259,"state":"up:active","state_seq":32,"addr":"172.21.15.40:6833/3355679996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3355679996},{"type":"v1","addr":"172.21.15.40:6833","nonce":3355679996}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:11:26.838 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 263 2019-12-05T06:11:26.840 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:26.838+0000 7f9d932eb700 1 -- 172.21.15.42:0/2324225891 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9d7403b280 msgr2=0x7f9d7403d790 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:26.841 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:26.838+0000 7f9d932eb700 1 --2- 172.21.15.42:0/2324225891 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9d7403b280 0x7f9d7403d790 crc :-1 s=READY pgs=923 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:11:26.841 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:26.838+0000 7f9d932eb700 1 -- 172.21.15.42:0/2324225891 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9d8c133710 msgr2=0x7f9d8c124260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:26.841 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:26.838+0000 7f9d932eb700 1 --2- 172.21.15.42:0/2324225891 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9d8c133710 0x7f9d8c124260 secure :-1 s=READY pgs=1662 cs=0 l=1 rx=0x7f9d8c11b040 tx=0x7f9d88010a10).stop 2019-12-05T06:11:26.841 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:26.838+0000 7f9d932eb700 1 -- 172.21.15.42:0/2324225891 shutdown_connections 2019-12-05T06:11:26.841 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:26.838+0000 7f9d932eb700 1 --2- 172.21.15.42:0/2324225891 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9d7403b280 0x7f9d7403d790 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:11:26.842 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:26.838+0000 7f9d932eb700 1 --2- 172.21.15.42:0/2324225891 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9d8c116cd0 0x7f9d8c118170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:11:26.842 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:26.838+0000 7f9d932eb700 1 --2- 172.21.15.42:0/2324225891 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9d8c133710 0x7f9d8c124260 unknown :-1 s=CLOSED pgs=1662 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:11:26.842 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:26.838+0000 7f9d932eb700 1 -- 172.21.15.42:0/2324225891 shutdown_connections 2019-12-05T06:11:26.842 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:26.838+0000 7f9d932eb700 1 -- 172.21.15.42:0/2324225891 wait complete. 2019-12-05T06:11:26.842 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:26.838+0000 7f9d932eb700 1 -- 172.21.15.42:0/2324225891 >> 172.21.15.42:0/2324225891 conn(0x7f9d8c09fa10 msgr2=0x7f9d8c11d870 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:11:26.856 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:11:28.857 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:11:28.967 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:28.958+0000 7fc2d5da4700 1 Processor -- start 2019-12-05T06:11:28.967 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:28.958+0000 7fc2d5da4700 1 -- start start 2019-12-05T06:11:28.967 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:28.958+0000 7fc2d5da4700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc2d00a7740 0x7fc2d008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:11:28.968 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:28.958+0000 7fc2d5da4700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc2d009fae0 0x7fc2d007eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:11:28.968 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:28.958+0000 7fc2d5da4700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2d00a0710 con 0x7fc2d007e720 2019-12-05T06:11:28.968 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:28.958+0000 7fc2d5da4700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2d00a0b00 con 0x7fc2d00a7740 2019-12-05T06:11:28.968 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:28.958+0000 7fc2d5da4700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2d008e5b0 con 0x7fc2d009fae0 2019-12-05T06:11:28.968 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:28.958+0000 7fc2d4b42700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc2d009fae0 0x7fc2d007eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:11:28.968 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:28.958+0000 7fc2cffff700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc2d00a7740 0x7fc2d008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:11:28.968 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:28.958+0000 7fc2d4b42700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc2d009fae0 0x7fc2d007eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:50512/0 (socket says 172.21.15.42:50512) 2019-12-05T06:11:28.969 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:28.958+0000 7fc2d4b42700 1 -- 172.21.15.42:0/2370913037 learned_addr learned my addr 172.21.15.42:0/2370913037 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:11:28.969 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:28.958+0000 7fc2cf7fe700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fc2d007e720 0x7fc2d009f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:59524/0 (socket says 172.21.15.42:59524) 2019-12-05T06:11:28.969 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:28.958+0000 7fc2cffff700 1 -- 172.21.15.42:0/2370913037 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc2d009fae0 msgr2=0x7fc2d007eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:11:28.969 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:28.958+0000 7fc2cffff700 1 --2- 172.21.15.42:0/2370913037 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc2d009fae0 0x7fc2d007eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:11:28.969 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:28.958+0000 7fc2cffff700 1 -- 172.21.15.42:0/2370913037 >> v1:172.21.15.40:6789/0 conn(0x7fc2d007e720 legacy=0x7fc2d009f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:28.969 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:28.958+0000 7fc2cffff700 1 -- 172.21.15.42:0/2370913037 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2d007f0a0 con 0x7fc2d00a7740 2019-12-05T06:11:28.969 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:28.958+0000 7fc2ceffd700 1 -- 172.21.15.42:0/2370913037 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3125313979 0 0) 0x7fc2d00a0710 con 0x7fc2d007e720 2019-12-05T06:11:28.970 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:28.958+0000 7fc2cffff700 1 --2- 172.21.15.42:0/2370913037 >> v2:172.21.15.40:3300/0 conn(0x7fc2d00a7740 0x7fc2d008df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fc2c0008b10 tx=0x7fc2c000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:28.970 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:28.962+0000 7fc2cffff700 1 --2- 172.21.15.42:0/2370913037 >> v2:172.21.15.40:3300/0 conn(0x7fc2d00a7740 0x7fc2d008df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fc2c0008b10 tx=0x7fc2c000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:28.970 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:28.962+0000 7fc2cffff700 1 --2- 172.21.15.42:0/2370913037 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc2d00a7740 0x7fc2d008df80 secure :-1 s=READY pgs=1664 cs=0 l=1 rx=0x7fc2c0008b10 tx=0x7fc2c000fff0).ready entity=mon.0 client_cookie=5a01186d13d2396d server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:11:28.970 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:28.962+0000 7fc2cffff700 1 --2- 172.21.15.42:0/2370913037 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc2d00a7740 0x7fc2d008df80 secure :-1 s=THROTTLE_DONE pgs=1664 cs=0 l=1 rx=0x7fc2c0008b10 tx=0x7fc2c000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:28.970 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:28.962+0000 7fc2ceffd700 1 -- 172.21.15.42:0/2370913037 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc2c0010900 con 0x7fc2d00a7740 2019-12-05T06:11:28.970 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:28.962+0000 7fc2cffff700 1 --2- 172.21.15.42:0/2370913037 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc2d00a7740 0x7fc2d008df80 secure :-1 s=THROTTLE_DONE pgs=1664 cs=0 l=1 rx=0x7fc2c0008b10 tx=0x7fc2c000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:28.971 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:28.962+0000 7fc2cffff700 1 --2- 172.21.15.42:0/2370913037 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc2d00a7740 0x7fc2d008df80 secure :-1 s=THROTTLE_DONE pgs=1664 cs=0 l=1 rx=0x7fc2c0008b10 tx=0x7fc2c000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:28.971 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:28.962+0000 7fc2ceffd700 1 -- 172.21.15.42:0/2370913037 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2c0008b50 con 0x7fc2d00a7740 2019-12-05T06:11:28.971 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:28.962+0000 7fc2ceffd700 1 -- 172.21.15.42:0/2370913037 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc2c00110c0 con 0x7fc2d00a7740 2019-12-05T06:11:28.971 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:28.962+0000 7fc2d5da4700 1 -- 172.21.15.42:0/2370913037 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc2d00a7740 msgr2=0x7fc2d008df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:28.971 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:28.962+0000 7fc2d5da4700 1 --2- 172.21.15.42:0/2370913037 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc2d00a7740 0x7fc2d008df80 secure :-1 s=READY pgs=1664 cs=0 l=1 rx=0x7fc2c0008b10 tx=0x7fc2c000fff0).stop 2019-12-05T06:11:28.971 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:28.962+0000 7fc2d5da4700 1 -- 172.21.15.42:0/2370913037 shutdown_connections 2019-12-05T06:11:28.971 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:28.962+0000 7fc2d5da4700 1 --2- 172.21.15.42:0/2370913037 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc2d009fae0 0x7fc2d007eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:11:28.971 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:28.962+0000 7fc2d5da4700 1 --2- 172.21.15.42:0/2370913037 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc2d00a7740 0x7fc2d008df80 unknown :-1 s=CLOSED pgs=1664 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:11:28.972 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:28.962+0000 7fc2d5da4700 1 -- 172.21.15.42:0/2370913037 shutdown_connections 2019-12-05T06:11:28.972 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:28.962+0000 7fc2d5da4700 1 -- 172.21.15.42:0/2370913037 wait complete. 2019-12-05T06:11:28.972 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:28.962+0000 7fc2d5da4700 1 -- 172.21.15.42:0/2370913037 >> 172.21.15.42:0/2370913037 conn(0x7fc2d00aabe0 msgr2=0x7fc2d00a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:11:28.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:28.966+0000 7fc2d5da4700 1 Processor -- start 2019-12-05T06:11:28.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:28.966+0000 7fc2d5da4700 1 -- start start 2019-12-05T06:11:28.976 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:28.966+0000 7fc2d5da4700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc2d011b040 0x7fc2d007dac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:11:28.976 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:28.966+0000 7fc2d5da4700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc2d0134ca0 0x7fc2d01370e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:11:28.976 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:28.966+0000 7fc2d5da4700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2d011a6b0 con 0x7fc2d00a6c70 2019-12-05T06:11:28.976 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:28.966+0000 7fc2d5da4700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2d00a5e30 con 0x7fc2d011b040 2019-12-05T06:11:28.976 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:28.966+0000 7fc2d5da4700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2d00a60d0 con 0x7fc2d0134ca0 2019-12-05T06:11:28.976 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:28.966+0000 7fc2d4b42700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc2d0134ca0 0x7fc2d01370e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:11:28.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:28.966+0000 7fc2cffff700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc2d011b040 0x7fc2d007dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:11:28.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:28.966+0000 7fc2d4b42700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc2d0134ca0 0x7fc2d01370e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:50520/0 (socket says 172.21.15.42:50520) 2019-12-05T06:11:28.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:28.966+0000 7fc2cf7fe700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fc2d00a6c70 0x7fc2d0119eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:59534/0 (socket says 172.21.15.42:59534) 2019-12-05T06:11:28.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:28.966+0000 7fc2d4b42700 1 -- 172.21.15.42:0/3522550135 learned_addr learned my addr 172.21.15.42:0/3522550135 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:11:28.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:28.966+0000 7fc2d4b42700 1 -- 172.21.15.42:0/3522550135 >> v2:172.21.15.40:3300/0 conn(0x7fc2d011b040 msgr2=0x7fc2d007dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:28.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:28.966+0000 7fc2d4b42700 1 --2- 172.21.15.42:0/3522550135 >> v2:172.21.15.40:3300/0 conn(0x7fc2d011b040 0x7fc2d007dac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:11:28.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:28.966+0000 7fc2d4b42700 1 -- 172.21.15.42:0/3522550135 >> v1:172.21.15.40:6789/0 conn(0x7fc2d00a6c70 legacy=0x7fc2d0119eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:28.978 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:28.966+0000 7fc2d4b42700 1 -- 172.21.15.42:0/3522550135 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2d00a4a50 con 0x7fc2d0134ca0 2019-12-05T06:11:28.978 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:28.966+0000 7fc2d4b42700 1 --2- 172.21.15.42:0/3522550135 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc2d0134ca0 0x7fc2d01370e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fc2c8002f30 tx=0x7fc2c8012c50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:28.978 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:28.970+0000 7fc2d4b42700 1 --2- 172.21.15.42:0/3522550135 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc2d0134ca0 0x7fc2d01370e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fc2c8002f30 tx=0x7fc2c8012c50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:28.978 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:28.970+0000 7fc2d4b42700 1 --2- 172.21.15.42:0/3522550135 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc2d0134ca0 0x7fc2d01370e0 secure :-1 s=READY pgs=481 cs=0 l=1 rx=0x7fc2c8002f30 tx=0x7fc2c8012c50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:11:28.978 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:28.970+0000 7fc2d4b42700 1 --2- 172.21.15.42:0/3522550135 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc2d0134ca0 0x7fc2d01370e0 secure :-1 s=THROTTLE_DONE pgs=481 cs=0 l=1 rx=0x7fc2c8002f30 tx=0x7fc2c8012c50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:28.978 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:28.970+0000 7fc2ccff9700 1 -- 172.21.15.42:0/3522550135 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc2c8005930 con 0x7fc2d0134ca0 2019-12-05T06:11:28.978 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:28.970+0000 7fc2d4b42700 1 --2- 172.21.15.42:0/3522550135 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc2d0134ca0 0x7fc2d01370e0 secure :-1 s=THROTTLE_DONE pgs=481 cs=0 l=1 rx=0x7fc2c8002f30 tx=0x7fc2c8012c50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:28.979 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:28.970+0000 7fc2d4b42700 1 --2- 172.21.15.42:0/3522550135 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc2d0134ca0 0x7fc2d01370e0 secure :-1 s=THROTTLE_DONE pgs=481 cs=0 l=1 rx=0x7fc2c8002f30 tx=0x7fc2c8012c50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:28.979 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:28.970+0000 7fc2ccff9700 1 -- 172.21.15.42:0/3522550135 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2c8007160 con 0x7fc2d0134ca0 2019-12-05T06:11:28.979 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:28.970+0000 7fc2ccff9700 1 -- 172.21.15.42:0/3522550135 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc2c80088b0 con 0x7fc2d0134ca0 2019-12-05T06:11:28.979 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:28.970+0000 7fc2d5da4700 1 -- 172.21.15.42:0/3522550135 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2d00a6610 con 0x7fc2d0134ca0 2019-12-05T06:11:28.979 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:28.970+0000 7fc2d5da4700 1 -- 172.21.15.42:0/3522550135 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2d00a5a40 con 0x7fc2d0134ca0 2019-12-05T06:11:28.979 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:28.970+0000 7fc2d4b42700 1 --2- 172.21.15.42:0/3522550135 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc2d0134ca0 0x7fc2d01370e0 secure :-1 s=THROTTLE_DONE pgs=481 cs=0 l=1 rx=0x7fc2c8002f30 tx=0x7fc2c8012c50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:28.979 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:28.970+0000 7fc2ccff9700 1 -- 172.21.15.42:0/3522550135 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fc2c8008bc0 con 0x7fc2d0134ca0 2019-12-05T06:11:28.980 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:28.970+0000 7fc2d4b42700 1 --2- 172.21.15.42:0/3522550135 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc2d0134ca0 0x7fc2d01370e0 secure :-1 s=THROTTLE_DONE pgs=481 cs=0 l=1 rx=0x7fc2c8002f30 tx=0x7fc2c8012c50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:28.980 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:28.970+0000 7fc2ccff9700 1 --2- 172.21.15.42:0/3522550135 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc2b403ae00 0x7fc2b403d310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:11:28.980 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:28.970+0000 7fc2ccff9700 1 -- 172.21.15.42:0/3522550135 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9987+0+0 (secure 0 0 0) 0x7fc2c804d300 con 0x7fc2d0134ca0 2019-12-05T06:11:28.980 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:28.970+0000 7fc2cffff700 1 --2- 172.21.15.42:0/3522550135 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc2b403ae00 0x7fc2b403d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:11:28.980 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:28.970+0000 7fc2cffff700 1 --2- 172.21.15.42:0/3522550135 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc2b403ae00 0x7fc2b403d310 crc :-1 s=READY pgs=924 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:11:28.981 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:28.974+0000 7fc2d5da4700 1 -- 172.21.15.42:0/3522550135 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2d005d110 con 0x7fc2d0134ca0 2019-12-05T06:11:28.986 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:28.982+0000 7fc2d4b42700 1 --2- 172.21.15.42:0/3522550135 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc2d0134ca0 0x7fc2d01370e0 secure :-1 s=THROTTLE_DONE pgs=481 cs=0 l=1 rx=0x7fc2c8002f30 tx=0x7fc2c8012c50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:28.986 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:28.982+0000 7fc2ccff9700 1 -- 172.21.15.42:0/3522550135 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fc2c8034320 con 0x7fc2d0134ca0 2019-12-05T06:11:29.154 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:29.150+0000 7fc2d5da4700 1 -- 172.21.15.42:0/3522550135 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc2d005b8c0 con 0x7fc2d0134ca0 2019-12-05T06:11:29.155 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:29.150+0000 7fc2d4b42700 1 --2- 172.21.15.42:0/3522550135 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc2d0134ca0 0x7fc2d01370e0 secure :-1 s=THROTTLE_DONE pgs=481 cs=0 l=1 rx=0x7fc2c8002f30 tx=0x7fc2c8012c50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:29.155 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:29.150+0000 7fc2ccff9700 1 -- 172.21.15.42:0/3522550135 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 263 v263) v1 ==== 95+0+2113 (secure 0 0 0) 0x7fc2c8034580 con 0x7fc2d0134ca0 2019-12-05T06:11:29.157 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 263 2019-12-05T06:11:29.157 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:11:29.157 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":263,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":10000,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/3923117701","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3923117701},{"type":"v1","addr":"172.21.15.42:6835","nonce":3923117701}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":263}],"filesystems":[{"mdsmap":{"epoch":262,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:10:14.933627+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":95,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6977},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6977":{"gid":6977,"name":"b","rank":0,"incarnation":259,"state":"up:active","state_seq":32,"addr":"172.21.15.40:6833/3355679996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3355679996},{"type":"v1","addr":"172.21.15.40:6833","nonce":3355679996}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:11:29.160 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:29.154+0000 7fc2d5da4700 1 -- 172.21.15.42:0/3522550135 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc2b403ae00 msgr2=0x7fc2b403d310 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:29.160 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:29.154+0000 7fc2d5da4700 1 --2- 172.21.15.42:0/3522550135 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc2b403ae00 0x7fc2b403d310 crc :-1 s=READY pgs=924 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:11:29.160 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:29.154+0000 7fc2d5da4700 1 -- 172.21.15.42:0/3522550135 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc2d0134ca0 msgr2=0x7fc2d01370e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:29.161 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:29.154+0000 7fc2d5da4700 1 --2- 172.21.15.42:0/3522550135 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc2d0134ca0 0x7fc2d01370e0 secure :-1 s=READY pgs=481 cs=0 l=1 rx=0x7fc2c8002f30 tx=0x7fc2c8012c50).stop 2019-12-05T06:11:29.161 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:29.154+0000 7fc2d5da4700 1 -- 172.21.15.42:0/3522550135 shutdown_connections 2019-12-05T06:11:29.161 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:29.154+0000 7fc2d5da4700 1 --2- 172.21.15.42:0/3522550135 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc2b403ae00 0x7fc2b403d310 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:11:29.161 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:29.154+0000 7fc2d5da4700 1 --2- 172.21.15.42:0/3522550135 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc2d0134ca0 0x7fc2d01370e0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:11:29.161 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:29.154+0000 7fc2d5da4700 1 --2- 172.21.15.42:0/3522550135 >> v2:172.21.15.40:3300/0 conn(0x7fc2d011b040 0x7fc2d007dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:11:29.161 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:29.158+0000 7fc2d5da4700 1 -- 172.21.15.42:0/3522550135 shutdown_connections 2019-12-05T06:11:29.161 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:29.158+0000 7fc2d5da4700 1 -- 172.21.15.42:0/3522550135 wait complete. 2019-12-05T06:11:29.162 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:29.158+0000 7fc2d5da4700 1 -- 172.21.15.42:0/3522550135 >> 172.21.15.42:0/3522550135 conn(0x7fc2d007e4f0 msgr2=0x7fc2d0059ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:11:29.170 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:11:31.171 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:11:31.262 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.254+0000 7fdef2b6b700 1 Processor -- start 2019-12-05T06:11:31.263 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.254+0000 7fdef2b6b700 1 -- start start 2019-12-05T06:11:31.263 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.254+0000 7fdef2b6b700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fdeec0a7740 0x7fdeec08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:11:31.263 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.254+0000 7fdef2b6b700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fdeec07e720 0x7fdeec07eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:11:31.263 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.254+0000 7fdef2b6b700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdeec09f2e0 0x7fdeec09f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:11:31.263 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.254+0000 7fdef2b6b700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdeec0a0b00 con 0x7fdeec07e720 2019-12-05T06:11:31.263 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.254+0000 7fdef2b6b700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdeec08e5b0 con 0x7fdeec0a7740 2019-12-05T06:11:31.263 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.254+0000 7fdef2b6b700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdeec08e850 con 0x7fdeec09f2e0 2019-12-05T06:11:31.264 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.254+0000 7fdef1108700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fdeec0a7740 0x7fdeec08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:11:31.264 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.254+0000 7fdef1108700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fdeec0a7740 0x7fdeec08df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:50074/0 (socket says 172.21.15.42:50074) 2019-12-05T06:11:31.264 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.254+0000 7fdef1108700 1 -- 172.21.15.42:0/4001688768 learned_addr learned my addr 172.21.15.42:0/4001688768 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:11:31.264 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.258+0000 7fdef1108700 1 -- 172.21.15.42:0/4001688768 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdeec09f2e0 msgr2=0x7fdeec09f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:11:31.264 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.258+0000 7fdef1108700 1 --2- 172.21.15.42:0/4001688768 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdeec09f2e0 0x7fdeec09f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:11:31.264 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.258+0000 7fdef1108700 1 -- 172.21.15.42:0/4001688768 >> v2:172.21.15.40:3300/0 conn(0x7fdeec07e720 msgr2=0x7fdeec07eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:11:31.265 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.258+0000 7fdef1108700 1 --2- 172.21.15.42:0/4001688768 >> v2:172.21.15.40:3300/0 conn(0x7fdeec07e720 0x7fdeec07eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:11:31.265 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.258+0000 7fdef1108700 1 -- 172.21.15.42:0/4001688768 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdeec07f0a0 con 0x7fdeec0a7740 2019-12-05T06:11:31.265 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.258+0000 7fdef1108700 1 --2- 172.21.15.42:0/4001688768 >> v2:172.21.15.42:3300/0 conn(0x7fdeec0a7740 0x7fdeec08df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fdeec0a0f50 tx=0x7fdee800fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:31.265 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.258+0000 7fdef1108700 1 --2- 172.21.15.42:0/4001688768 >> v2:172.21.15.42:3300/0 conn(0x7fdeec0a7740 0x7fdeec08df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fdeec0a0f50 tx=0x7fdee800fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:31.265 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.258+0000 7fdef1108700 1 --2- 172.21.15.42:0/4001688768 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdeec0a7740 0x7fdeec08df80 secure :-1 s=READY pgs=2198 cs=0 l=1 rx=0x7fdeec0a0f50 tx=0x7fdee800fdb0).ready entity=mon.1 client_cookie=c4f8f121535cf112 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:11:31.265 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.258+0000 7fdef1108700 1 --2- 172.21.15.42:0/4001688768 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdeec0a7740 0x7fdeec08df80 secure :-1 s=THROTTLE_DONE pgs=2198 cs=0 l=1 rx=0x7fdeec0a0f50 tx=0x7fdee800fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:31.265 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.258+0000 7fdee3fff700 1 -- 172.21.15.42:0/4001688768 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdee8008c90 con 0x7fdeec0a7740 2019-12-05T06:11:31.266 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.258+0000 7fdef1108700 1 --2- 172.21.15.42:0/4001688768 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdeec0a7740 0x7fdeec08df80 secure :-1 s=THROTTLE_DONE pgs=2198 cs=0 l=1 rx=0x7fdeec0a0f50 tx=0x7fdee800fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:31.266 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.258+0000 7fdef1108700 1 --2- 172.21.15.42:0/4001688768 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdeec0a7740 0x7fdeec08df80 secure :-1 s=THROTTLE_DONE pgs=2198 cs=0 l=1 rx=0x7fdeec0a0f50 tx=0x7fdee800fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:31.266 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.258+0000 7fdee3fff700 1 -- 172.21.15.42:0/4001688768 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdee80088b0 con 0x7fdeec0a7740 2019-12-05T06:11:31.266 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.258+0000 7fdee3fff700 1 -- 172.21.15.42:0/4001688768 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdee8011b50 con 0x7fdeec0a7740 2019-12-05T06:11:31.266 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.258+0000 7fdef2b6b700 1 -- 172.21.15.42:0/4001688768 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdeec0a7740 msgr2=0x7fdeec08df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:31.268 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.258+0000 7fdef2b6b700 1 --2- 172.21.15.42:0/4001688768 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdeec0a7740 0x7fdeec08df80 secure :-1 s=READY pgs=2198 cs=0 l=1 rx=0x7fdeec0a0f50 tx=0x7fdee800fdb0).stop 2019-12-05T06:11:31.268 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.258+0000 7fdef2b6b700 1 -- 172.21.15.42:0/4001688768 shutdown_connections 2019-12-05T06:11:31.268 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.258+0000 7fdef2b6b700 1 --2- 172.21.15.42:0/4001688768 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdeec09f2e0 0x7fdeec09f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:11:31.268 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.258+0000 7fdef2b6b700 1 --2- 172.21.15.42:0/4001688768 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdeec0a7740 0x7fdeec08df80 unknown :-1 s=CLOSED pgs=2198 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:11:31.268 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.258+0000 7fdef2b6b700 1 --2- 172.21.15.42:0/4001688768 >> v2:172.21.15.40:3300/0 conn(0x7fdeec07e720 0x7fdeec07eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:11:31.268 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.258+0000 7fdef2b6b700 1 -- 172.21.15.42:0/4001688768 shutdown_connections 2019-12-05T06:11:31.269 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.258+0000 7fdef2b6b700 1 -- 172.21.15.42:0/4001688768 wait complete. 2019-12-05T06:11:31.269 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.258+0000 7fdef2b6b700 1 -- 172.21.15.42:0/4001688768 >> 172.21.15.42:0/4001688768 conn(0x7fdeec0aabe0 msgr2=0x7fdeec0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:11:31.273 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.258+0000 7fdef2b6b700 1 Processor -- start 2019-12-05T06:11:31.273 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.258+0000 7fdef2b6b700 1 -- start start 2019-12-05T06:11:31.273 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.262+0000 7fdef2b6b700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fdeec11b420 0x7fdeec131c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:11:31.273 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.262+0000 7fdef2b6b700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdeec08eb70 con 0x7fdeec1321c0 2019-12-05T06:11:31.274 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.262+0000 7fdef2b6b700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdeec131690 con 0x7fdeec11a5f0 2019-12-05T06:11:31.274 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.262+0000 7fdef2b6b700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdeec1319d0 con 0x7fdeec11b420 2019-12-05T06:11:31.274 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.262+0000 7fdef1108700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fdeec11a5f0 0x7fdeec0a4cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:50118/0 (socket says 172.21.15.42:50118) 2019-12-05T06:11:31.274 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.262+0000 7fdef1108700 1 -- 172.21.15.42:0/3917102514 learned_addr learned my addr 172.21.15.42:0/3917102514 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:11:31.274 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.262+0000 7fdef0907700 1 --2- 172.21.15.42:0/3917102514 >> v2:172.21.15.40:3300/0 conn(0x7fdeec11b420 0x7fdeec131c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:11:31.274 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.262+0000 7fdee1ffb700 1 -- 172.21.15.42:0/3917102514 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3157117301 0 0) 0x7fdeec131690 con 0x7fdeec11a5f0 2019-12-05T06:11:31.275 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.262+0000 7fdee1ffb700 1 -- 172.21.15.42:0/3917102514 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdecc0030b0 con 0x7fdeec11a5f0 2019-12-05T06:11:31.275 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.262+0000 7fdee1ffb700 1 -- 172.21.15.42:0/3917102514 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 1022841015 0 0) 0x7fdecc0030b0 con 0x7fdeec11a5f0 2019-12-05T06:11:31.275 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.262+0000 7fdee1ffb700 1 -- 172.21.15.42:0/3917102514 --> v1:172.21.15.42:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdeec131690 con 0x7fdeec11a5f0 2019-12-05T06:11:31.275 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.262+0000 7fdee1ffb700 1 -- 172.21.15.42:0/3917102514 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7fdee8008470 con 0x7fdeec11a5f0 2019-12-05T06:11:31.275 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.262+0000 7fdef0907700 1 -- 172.21.15.42:0/3917102514 >> v1:172.21.15.42:6789/0 conn(0x7fdeec11a5f0 legacy=0x7fdeec0a4cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:31.275 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.262+0000 7fdef0907700 1 -- 172.21.15.42:0/3917102514 >> v1:172.21.15.40:6789/0 conn(0x7fdeec1321c0 legacy=0x7fdeec138ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:31.275 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.262+0000 7fdef0907700 1 -- 172.21.15.42:0/3917102514 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdeec0a56b0 con 0x7fdeec11b420 2019-12-05T06:11:31.276 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.262+0000 7fdef0907700 1 --2- 172.21.15.42:0/3917102514 >> v2:172.21.15.40:3300/0 conn(0x7fdeec11b420 0x7fdeec131c70 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fdeec0aacf0 tx=0x7fdee80082d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:31.276 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.262+0000 7fdef0907700 1 --2- 172.21.15.42:0/3917102514 >> v2:172.21.15.40:3300/0 conn(0x7fdeec11b420 0x7fdeec131c70 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fdeec0aacf0 tx=0x7fdee80082d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:31.276 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.262+0000 7fdef0907700 1 --2- 172.21.15.42:0/3917102514 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdeec11b420 0x7fdeec131c70 secure :-1 s=READY pgs=1667 cs=0 l=1 rx=0x7fdeec0aacf0 tx=0x7fdee80082d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:11:31.276 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.262+0000 7fdef0907700 1 --2- 172.21.15.42:0/3917102514 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdeec11b420 0x7fdeec131c70 secure :-1 s=THROTTLE_DONE pgs=1667 cs=0 l=1 rx=0x7fdeec0aacf0 tx=0x7fdee80082d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:31.276 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.262+0000 7fdee1ffb700 1 -- 172.21.15.42:0/3917102514 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdedc005940 con 0x7fdeec11b420 2019-12-05T06:11:31.276 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.262+0000 7fdef2b6b700 1 -- 172.21.15.42:0/3917102514 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdeec07b6b0 con 0x7fdeec11b420 2019-12-05T06:11:31.276 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.262+0000 7fdef0907700 1 --2- 172.21.15.42:0/3917102514 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdeec11b420 0x7fdeec131c70 secure :-1 s=THROTTLE_DONE pgs=1667 cs=0 l=1 rx=0x7fdeec0aacf0 tx=0x7fdee80082d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:31.277 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.262+0000 7fdee1ffb700 1 -- 172.21.15.42:0/3917102514 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdedc005560 con 0x7fdeec11b420 2019-12-05T06:11:31.278 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.262+0000 7fdef0907700 1 --2- 172.21.15.42:0/3917102514 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdeec11b420 0x7fdeec131c70 secure :-1 s=THROTTLE_DONE pgs=1667 cs=0 l=1 rx=0x7fdeec0aacf0 tx=0x7fdee80082d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:31.278 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.262+0000 7fdef2b6b700 1 -- 172.21.15.42:0/3917102514 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdeec13a2f0 con 0x7fdeec11b420 2019-12-05T06:11:31.278 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.262+0000 7fdee1ffb700 1 -- 172.21.15.42:0/3917102514 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdedc010f90 con 0x7fdeec11b420 2019-12-05T06:11:31.279 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.262+0000 7fdef0907700 1 --2- 172.21.15.42:0/3917102514 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdeec11b420 0x7fdeec131c70 secure :-1 s=THROTTLE_DONE pgs=1667 cs=0 l=1 rx=0x7fdeec0aacf0 tx=0x7fdee80082d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:31.279 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.262+0000 7fdee1ffb700 1 -- 172.21.15.42:0/3917102514 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fdedc0112b0 con 0x7fdeec11b420 2019-12-05T06:11:31.279 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.262+0000 7fdef0907700 1 --2- 172.21.15.42:0/3917102514 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdeec11b420 0x7fdeec131c70 secure :-1 s=THROTTLE_DONE pgs=1667 cs=0 l=1 rx=0x7fdeec0aacf0 tx=0x7fdee80082d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:31.279 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.262+0000 7fdee1ffb700 1 --2- 172.21.15.42:0/3917102514 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fdecc03b620 0x7fdecc03db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:11:31.279 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.262+0000 7fdee1ffb700 1 -- 172.21.15.42:0/3917102514 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 10073+0+0 (secure 0 0 0) 0x7fdedc04a9f0 con 0x7fdeec11b420 2019-12-05T06:11:31.279 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.266+0000 7fdef1108700 1 --2- 172.21.15.42:0/3917102514 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fdecc03b620 0x7fdecc03db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:11:31.279 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.266+0000 7fdef1108700 1 --2- 172.21.15.42:0/3917102514 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fdecc03b620 0x7fdecc03db30 crc :-1 s=READY pgs=925 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:11:31.280 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.270+0000 7fdef2b6b700 1 -- 172.21.15.42:0/3917102514 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdeec05d110 con 0x7fdeec11b420 2019-12-05T06:11:31.283 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.278+0000 7fdef0907700 1 --2- 172.21.15.42:0/3917102514 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdeec11b420 0x7fdeec131c70 secure :-1 s=THROTTLE_DONE pgs=1667 cs=0 l=1 rx=0x7fdeec0aacf0 tx=0x7fdee80082d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:31.283 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.278+0000 7fdee1ffb700 1 -- 172.21.15.42:0/3917102514 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fdedc031a70 con 0x7fdeec11b420 2019-12-05T06:11:31.448 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.442+0000 7fdef2b6b700 1 -- 172.21.15.42:0/3917102514 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdeec05b8c0 con 0x7fdeec11b420 2019-12-05T06:11:31.449 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.446+0000 7fdef0907700 1 --2- 172.21.15.42:0/3917102514 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdeec11b420 0x7fdeec131c70 secure :-1 s=THROTTLE_DONE pgs=1667 cs=0 l=1 rx=0x7fdeec0aacf0 tx=0x7fdee80082d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:31.449 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.446+0000 7fdee1ffb700 1 -- 172.21.15.42:0/3917102514 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 264 v264) v1 ==== 95+0+1777 (secure 0 0 0) 0x7fdedc031cd0 con 0x7fdeec11b420 2019-12-05T06:11:31.449 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 264 2019-12-05T06:11:31.450 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:11:31.450 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":264,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":264,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:11:29.571626+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":96,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10000},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10000":{"gid":10000,"name":"a","rank":0,"incarnation":264,"state":"up:replay","state_seq":2,"addr":"172.21.15.42:6835/3923117701","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3923117701},{"type":"v1","addr":"172.21.15.42:6835","nonce":3923117701}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:11:31.451 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.446+0000 7fdef2b6b700 1 -- 172.21.15.42:0/3917102514 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fdecc03b620 msgr2=0x7fdecc03db30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:31.451 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.446+0000 7fdef2b6b700 1 --2- 172.21.15.42:0/3917102514 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fdecc03b620 0x7fdecc03db30 crc :-1 s=READY pgs=925 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:11:31.452 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.446+0000 7fdef2b6b700 1 -- 172.21.15.42:0/3917102514 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdeec11b420 msgr2=0x7fdeec131c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:31.452 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.446+0000 7fdef2b6b700 1 --2- 172.21.15.42:0/3917102514 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdeec11b420 0x7fdeec131c70 secure :-1 s=READY pgs=1667 cs=0 l=1 rx=0x7fdeec0aacf0 tx=0x7fdee80082d0).stop 2019-12-05T06:11:31.452 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.446+0000 7fdef2b6b700 1 -- 172.21.15.42:0/3917102514 shutdown_connections 2019-12-05T06:11:31.452 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.446+0000 7fdef2b6b700 1 --2- 172.21.15.42:0/3917102514 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fdecc03b620 0x7fdecc03db30 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:11:31.452 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.446+0000 7fdef2b6b700 1 --2- 172.21.15.42:0/3917102514 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdeec11b420 0x7fdeec131c70 unknown :-1 s=CLOSED pgs=1667 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:11:31.453 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.446+0000 7fdef2b6b700 1 -- 172.21.15.42:0/3917102514 shutdown_connections 2019-12-05T06:11:31.453 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.446+0000 7fdef2b6b700 1 -- 172.21.15.42:0/3917102514 wait complete. 2019-12-05T06:11:31.453 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.446+0000 7fdef2b6b700 1 -- 172.21.15.42:0/3917102514 >> 172.21.15.42:0/3917102514 conn(0x7fdeec07eb80 msgr2=0x7fdeec059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:11:31.459 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b down, removed from mdsmap 2019-12-05T06:11:31.459 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2019-12-05T06:11:31.460 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:11:31.562 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.554+0000 7f02ee11c700 1 Processor -- start 2019-12-05T06:11:31.562 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.554+0000 7f02ee11c700 1 -- start start 2019-12-05T06:11:31.563 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.554+0000 7f02ee11c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f02e809fad0 0x7f02e814ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:11:31.563 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.554+0000 7f02ee11c700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02e80a0700 con 0x7f02e80a7730 2019-12-05T06:11:31.563 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.554+0000 7f02ee11c700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02e807ef10 con 0x7f02e808df70 2019-12-05T06:11:31.563 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.554+0000 7f02ee11c700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02e80a0af0 con 0x7f02e809fad0 2019-12-05T06:11:31.563 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.554+0000 7f02eceba700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f02e809fad0 0x7f02e814ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:11:31.563 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.554+0000 7f02eceba700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f02e809fad0 0x7f02e814ffe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:50092/0 (socket says 172.21.15.42:50092) 2019-12-05T06:11:31.564 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.554+0000 7f02eceba700 1 -- 172.21.15.42:0/1627580947 learned_addr learned my addr 172.21.15.42:0/1627580947 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:11:31.564 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.554+0000 7f02eceba700 1 -- 172.21.15.42:0/1627580947 >> v1:172.21.15.42:6789/0 conn(0x7f02e808df70 legacy=0x7f02e809f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:31.564 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.554+0000 7f02eceba700 1 -- 172.21.15.42:0/1627580947 >> v1:172.21.15.40:6789/0 conn(0x7f02e80a7730 legacy=0x7f02e807e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:31.564 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.554+0000 7f02eceba700 1 -- 172.21.15.42:0/1627580947 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02e80a9b90 con 0x7f02e809fad0 2019-12-05T06:11:31.564 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.554+0000 7f02eceba700 1 --2- 172.21.15.42:0/1627580947 >> v2:172.21.15.42:3300/0 conn(0x7f02e809fad0 0x7f02e814ffe0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f02d8008790 tx=0x7f02d800fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:31.564 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.554+0000 7f02eceba700 1 --2- 172.21.15.42:0/1627580947 >> v2:172.21.15.42:3300/0 conn(0x7f02e809fad0 0x7f02e814ffe0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f02d8008790 tx=0x7f02d800fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:31.564 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.554+0000 7f02eceba700 1 --2- 172.21.15.42:0/1627580947 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f02e809fad0 0x7f02e814ffe0 secure :-1 s=READY pgs=2201 cs=0 l=1 rx=0x7f02d8008790 tx=0x7f02d800fcb0).ready entity=mon.1 client_cookie=211ebb29cca8fbc4 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:11:31.565 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.558+0000 7f02eceba700 1 --2- 172.21.15.42:0/1627580947 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f02e809fad0 0x7f02e814ffe0 secure :-1 s=THROTTLE_DONE pgs=2201 cs=0 l=1 rx=0x7f02d8008790 tx=0x7f02d800fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:31.565 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.558+0000 7f02eceba700 1 --2- 172.21.15.42:0/1627580947 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f02e809fad0 0x7f02e814ffe0 secure :-1 s=THROTTLE_DONE pgs=2201 cs=0 l=1 rx=0x7f02d8008790 tx=0x7f02d800fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:31.565 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.558+0000 7f02e6ffd700 1 -- 172.21.15.42:0/1627580947 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f02d8008c90 con 0x7f02e809fad0 2019-12-05T06:11:31.565 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.558+0000 7f02eceba700 1 --2- 172.21.15.42:0/1627580947 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f02e809fad0 0x7f02e814ffe0 secure :-1 s=THROTTLE_DONE pgs=2201 cs=0 l=1 rx=0x7f02d8008790 tx=0x7f02d800fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:31.565 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.558+0000 7f02e6ffd700 1 -- 172.21.15.42:0/1627580947 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02d80088b0 con 0x7f02e809fad0 2019-12-05T06:11:31.565 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.558+0000 7f02e6ffd700 1 -- 172.21.15.42:0/1627580947 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f02d8011b80 con 0x7f02e809fad0 2019-12-05T06:11:31.565 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.558+0000 7f02ee11c700 1 -- 172.21.15.42:0/1627580947 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f02e809fad0 msgr2=0x7f02e814ffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:31.566 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.558+0000 7f02ee11c700 1 --2- 172.21.15.42:0/1627580947 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f02e809fad0 0x7f02e814ffe0 secure :-1 s=READY pgs=2201 cs=0 l=1 rx=0x7f02d8008790 tx=0x7f02d800fcb0).stop 2019-12-05T06:11:31.566 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.558+0000 7f02ee11c700 1 -- 172.21.15.42:0/1627580947 shutdown_connections 2019-12-05T06:11:31.566 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.558+0000 7f02ee11c700 1 --2- 172.21.15.42:0/1627580947 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f02e809fad0 0x7f02e814ffe0 unknown :-1 s=CLOSED pgs=2201 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:11:31.566 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.558+0000 7f02ee11c700 1 -- 172.21.15.42:0/1627580947 shutdown_connections 2019-12-05T06:11:31.566 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.558+0000 7f02ee11c700 1 -- 172.21.15.42:0/1627580947 wait complete. 2019-12-05T06:11:31.566 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.558+0000 7f02ee11c700 1 -- 172.21.15.42:0/1627580947 >> 172.21.15.42:0/1627580947 conn(0x7f02e80aabd0 msgr2=0x7f02e80a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:11:31.566 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.558+0000 7f02ee11c700 1 Processor -- start 2019-12-05T06:11:31.567 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.558+0000 7f02ee11c700 1 -- start start 2019-12-05T06:11:31.570 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.562+0000 7f02ee11c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f02e807b0e0 0x7f02e8145100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:11:31.570 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.562+0000 7f02ee11c700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02e8150220 con 0x7f02e807e820 2019-12-05T06:11:31.570 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.562+0000 7f02ee11c700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02e80a5f70 con 0x7f02e80a6630 2019-12-05T06:11:31.571 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.562+0000 7f02ee11c700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02e80a61a0 con 0x7f02e807b0e0 2019-12-05T06:11:31.571 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.562+0000 7f02eceba700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f02e807b0e0 0x7f02e8145100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:11:31.572 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.562+0000 7f02e77fe700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f02e80a6630 0x7f02e807a8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:50134/0 (socket says 172.21.15.42:50134) 2019-12-05T06:11:31.572 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.562+0000 7f02e77fe700 1 -- 172.21.15.42:0/728278041 learned_addr learned my addr 172.21.15.42:0/728278041 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:11:31.572 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.562+0000 7f02eceba700 1 -- 172.21.15.42:0/728278041 >> v1:172.21.15.42:6789/0 conn(0x7f02e80a6630 legacy=0x7f02e807a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:31.572 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.562+0000 7f02eceba700 1 -- 172.21.15.42:0/728278041 >> v1:172.21.15.40:6789/0 conn(0x7f02e807e820 legacy=0x7f02e80a3850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:31.572 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.562+0000 7f02eceba700 1 -- 172.21.15.42:0/728278041 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02e81311f0 con 0x7f02e807b0e0 2019-12-05T06:11:31.572 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.562+0000 7f02eceba700 1 --2- 172.21.15.42:0/728278041 >> v2:172.21.15.42:3300/0 conn(0x7f02e807b0e0 0x7f02e8145100 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f02d8008790 tx=0x7f02d8008b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:31.572 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.562+0000 7f02eceba700 1 --2- 172.21.15.42:0/728278041 >> v2:172.21.15.42:3300/0 conn(0x7f02e807b0e0 0x7f02e8145100 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f02d8008790 tx=0x7f02d8008b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:31.573 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.562+0000 7f02eceba700 1 --2- 172.21.15.42:0/728278041 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f02e807b0e0 0x7f02e8145100 secure :-1 s=READY pgs=2203 cs=0 l=1 rx=0x7f02d8008790 tx=0x7f02d8008b60).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:11:31.573 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.562+0000 7f02eceba700 1 --2- 172.21.15.42:0/728278041 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f02e807b0e0 0x7f02e8145100 secure :-1 s=THROTTLE_DONE pgs=2203 cs=0 l=1 rx=0x7f02d8008790 tx=0x7f02d8008b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:31.573 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.562+0000 7f02e4ff9700 1 -- 172.21.15.42:0/728278041 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f02d800f470 con 0x7f02e807b0e0 2019-12-05T06:11:31.573 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.562+0000 7f02eceba700 1 --2- 172.21.15.42:0/728278041 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f02e807b0e0 0x7f02e8145100 secure :-1 s=THROTTLE_DONE pgs=2203 cs=0 l=1 rx=0x7f02d8008790 tx=0x7f02d8008b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:31.573 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.562+0000 7f02eceba700 1 --2- 172.21.15.42:0/728278041 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f02e807b0e0 0x7f02e8145100 secure :-1 s=THROTTLE_DONE pgs=2203 cs=0 l=1 rx=0x7f02d8008790 tx=0x7f02d8008b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:31.573 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.562+0000 7f02e4ff9700 1 -- 172.21.15.42:0/728278041 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02d80117a0 con 0x7f02e807b0e0 2019-12-05T06:11:31.573 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.562+0000 7f02ee11c700 1 -- 172.21.15.42:0/728278041 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02e812f1d0 con 0x7f02e807b0e0 2019-12-05T06:11:31.574 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.562+0000 7f02e4ff9700 1 -- 172.21.15.42:0/728278041 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f02d8005730 con 0x7f02e807b0e0 2019-12-05T06:11:31.574 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.562+0000 7f02ee11c700 1 -- 172.21.15.42:0/728278041 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02e812f720 con 0x7f02e807b0e0 2019-12-05T06:11:31.574 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.562+0000 7f02eceba700 1 --2- 172.21.15.42:0/728278041 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f02e807b0e0 0x7f02e8145100 secure :-1 s=THROTTLE_DONE pgs=2203 cs=0 l=1 rx=0x7f02d8008790 tx=0x7f02d8008b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:31.574 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.562+0000 7f02e4ff9700 1 -- 172.21.15.42:0/728278041 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f02d8005a40 con 0x7f02e807b0e0 2019-12-05T06:11:31.574 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.562+0000 7f02eceba700 1 --2- 172.21.15.42:0/728278041 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f02e807b0e0 0x7f02e8145100 secure :-1 s=THROTTLE_DONE pgs=2203 cs=0 l=1 rx=0x7f02d8008790 tx=0x7f02d8008b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:31.574 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.562+0000 7f02e4ff9700 1 --2- 172.21.15.42:0/728278041 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f02cc03ad00 0x7f02cc03d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:11:31.574 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.562+0000 7f02e4ff9700 1 -- 172.21.15.42:0/728278041 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 10073+0+0 (secure 0 0 0) 0x7f02d804b9a0 con 0x7f02e807b0e0 2019-12-05T06:11:31.575 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.562+0000 7f02e7fff700 1 --2- 172.21.15.42:0/728278041 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f02cc03ad00 0x7f02cc03d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:11:31.575 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.562+0000 7f02e7fff700 1 --2- 172.21.15.42:0/728278041 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f02cc03ad00 0x7f02cc03d210 crc :-1 s=READY pgs=926 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:11:31.575 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.570+0000 7f02ee11c700 1 -- 172.21.15.42:0/728278041 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02e805d100 con 0x7f02e807b0e0 2019-12-05T06:11:31.578 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.574+0000 7f02eceba700 1 --2- 172.21.15.42:0/728278041 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f02e807b0e0 0x7f02e8145100 secure :-1 s=THROTTLE_DONE pgs=2203 cs=0 l=1 rx=0x7f02d8008790 tx=0x7f02d8008b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:31.578 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.574+0000 7f02e4ff9700 1 -- 172.21.15.42:0/728278041 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f02d8008de0 con 0x7f02e807b0e0 2019-12-05T06:11:31.731 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.726+0000 7f02ee11c700 1 -- 172.21.15.42:0/728278041 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f02e805b8b0 con 0x7f02e807b0e0 2019-12-05T06:11:31.732 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.730+0000 7f02eceba700 1 --2- 172.21.15.42:0/728278041 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f02e807b0e0 0x7f02e8145100 secure :-1 s=THROTTLE_DONE pgs=2203 cs=0 l=1 rx=0x7f02d8008790 tx=0x7f02d8008b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:31.732 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.730+0000 7f02e4ff9700 1 -- 172.21.15.42:0/728278041 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 264 v264) v1 ==== 95+0+1777 (secure 0 0 0) 0x7f02d8034bd0 con 0x7f02e807b0e0 2019-12-05T06:11:31.733 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 264 2019-12-05T06:11:31.733 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:11:31.733 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":264,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":264,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:11:29.571626+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":96,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10000},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10000":{"gid":10000,"name":"a","rank":0,"incarnation":264,"state":"up:replay","state_seq":2,"addr":"172.21.15.42:6835/3923117701","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3923117701},{"type":"v1","addr":"172.21.15.42:6835","nonce":3923117701}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:11:31.735 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.730+0000 7f02ee11c700 1 -- 172.21.15.42:0/728278041 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f02cc03ad00 msgr2=0x7f02cc03d210 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:31.735 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.730+0000 7f02ee11c700 1 --2- 172.21.15.42:0/728278041 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f02cc03ad00 0x7f02cc03d210 crc :-1 s=READY pgs=926 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:11:31.735 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.730+0000 7f02ee11c700 1 -- 172.21.15.42:0/728278041 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f02e807b0e0 msgr2=0x7f02e8145100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:31.735 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.730+0000 7f02ee11c700 1 --2- 172.21.15.42:0/728278041 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f02e807b0e0 0x7f02e8145100 secure :-1 s=READY pgs=2203 cs=0 l=1 rx=0x7f02d8008790 tx=0x7f02d8008b60).stop 2019-12-05T06:11:31.736 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.730+0000 7f02ee11c700 1 -- 172.21.15.42:0/728278041 shutdown_connections 2019-12-05T06:11:31.736 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.730+0000 7f02ee11c700 1 --2- 172.21.15.42:0/728278041 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f02cc03ad00 0x7f02cc03d210 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:11:31.737 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.730+0000 7f02ee11c700 1 --2- 172.21.15.42:0/728278041 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f02e807b0e0 0x7f02e8145100 unknown :-1 s=CLOSED pgs=2203 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:11:31.737 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.730+0000 7f02ee11c700 1 -- 172.21.15.42:0/728278041 shutdown_connections 2019-12-05T06:11:31.737 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.730+0000 7f02ee11c700 1 -- 172.21.15.42:0/728278041 wait complete. 2019-12-05T06:11:31.737 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:31.730+0000 7f02ee11c700 1 -- 172.21.15.42:0/728278041 >> 172.21.15.42:0/728278041 conn(0x7f02e808f4d0 msgr2=0x7f02e80596f0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:11:31.745 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 264, "feature_flags": { "enable_multiple": false, "ever_enabled_multiple": false }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2019-12-05T05:03:35.604738+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 264, "ever_allowed_features": 2, "explicitly_allowed_features": 2, "failed": [], "flags": 18, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_10000": { "addr": "172.21.15.42:6835/3923117701", "addrs": { "addrvec": [ { "addr": "172.21.15.42:6834", "nonce": 3923117701, "type": "v2" }, { "addr": "172.21.15.42:6835", "nonce": 3923117701, "type": "v1" } ] }, "export_targets": [], "features": 4540138292836696063, "flags": 0, "gid": 10000, "incarnation": 264, "name": "a", "rank": 0, "state": "up:replay", "state_seq": 2 } }, "last_failure": 0, "last_failure_osd_epoch": 96, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 2, "min_compat_client": "0 (unknown)", "modified": "2019-12-05T06:11:29.571626+0000", "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 10000 } } } ], "standbys": [] } 2019-12-05T06:11:33.746 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:11:33.852 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:33.842+0000 7fa54499c700 1 Processor -- start 2019-12-05T06:11:33.852 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:33.842+0000 7fa54499c700 1 -- start start 2019-12-05T06:11:33.852 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:33.842+0000 7fa54499c700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa54008df70 0x7fa54009f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:11:33.852 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:33.842+0000 7fa54499c700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa54009f820 0x7fa54014efd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:11:33.853 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:33.842+0000 7fa54499c700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa54009fc50 con 0x7fa5400a7730 2019-12-05T06:11:33.853 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:33.842+0000 7fa54499c700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5400a0af0 con 0x7fa54008df70 2019-12-05T06:11:33.853 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:33.842+0000 7fa54499c700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa54008e5a0 con 0x7fa54009f820 2019-12-05T06:11:33.853 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:33.842+0000 7fa53ed9d700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fa5400a7730 0x7fa54007e710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:50146/0 (socket says 172.21.15.42:50146) 2019-12-05T06:11:33.854 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:33.842+0000 7fa53ed9d700 1 -- 172.21.15.42:0/1453659132 learned_addr learned my addr 172.21.15.42:0/1453659132 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:11:33.854 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:33.842+0000 7fa53e59c700 1 --2- 172.21.15.42:0/1453659132 >> v2:172.21.15.40:3300/0 conn(0x7fa54008df70 0x7fa54009f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:11:33.855 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:33.842+0000 7fa53f59e700 1 --2- 172.21.15.42:0/1453659132 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa54009f820 0x7fa54014efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:11:33.855 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:33.842+0000 7fa53dd9b700 1 -- 172.21.15.42:0/1453659132 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 990194583 0 0) 0x7fa54009fc50 con 0x7fa5400a7730 2019-12-05T06:11:33.855 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:33.842+0000 7fa53dd9b700 1 -- 172.21.15.42:0/1453659132 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5280030b0 con 0x7fa5400a7730 2019-12-05T06:11:33.855 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:33.846+0000 7fa53dd9b700 1 -- 172.21.15.42:0/1453659132 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3634748644 0 0) 0x7fa5280030b0 con 0x7fa5400a7730 2019-12-05T06:11:33.855 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:33.846+0000 7fa53dd9b700 1 -- 172.21.15.42:0/1453659132 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa54009f820 msgr2=0x7fa54014efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:11:33.855 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:33.846+0000 7fa53dd9b700 1 --2- 172.21.15.42:0/1453659132 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa54009f820 0x7fa54014efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:11:33.856 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:33.846+0000 7fa53dd9b700 1 -- 172.21.15.42:0/1453659132 >> v2:172.21.15.40:3300/0 conn(0x7fa54008df70 msgr2=0x7fa54009f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:11:33.856 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:33.846+0000 7fa53dd9b700 1 --2- 172.21.15.42:0/1453659132 >> v2:172.21.15.40:3300/0 conn(0x7fa54008df70 0x7fa54009f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:11:33.856 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:33.846+0000 7fa53dd9b700 1 -- 172.21.15.42:0/1453659132 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5400a0700 con 0x7fa5400a7730 2019-12-05T06:11:33.856 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:33.846+0000 7fa53dd9b700 1 -- 172.21.15.42:0/1453659132 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7fa530001ad0 con 0x7fa5400a7730 2019-12-05T06:11:33.856 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:33.846+0000 7fa53dd9b700 1 -- 172.21.15.42:0/1453659132 >> v1:172.21.15.42:6789/0 conn(0x7fa5400a7730 legacy=0x7fa54007e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:33.856 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:33.846+0000 7fa53dd9b700 1 --2- 172.21.15.42:0/1453659132 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa5280036e0 0x7fa528003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:11:33.856 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:33.846+0000 7fa53dd9b700 1 --2- 172.21.15.42:0/1453659132 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa528004420 0x7fa528004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:11:33.857 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:33.846+0000 7fa53dd9b700 1 --2- 172.21.15.42:0/1453659132 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa528004da0 0x7fa52804f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:11:33.857 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:33.846+0000 7fa53dd9b700 1 -- 172.21.15.42:0/1453659132 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5400a0af0 con 0x7fa5280036e0 2019-12-05T06:11:33.857 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:33.846+0000 7fa53dd9b700 1 -- 172.21.15.42:0/1453659132 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa54008e5a0 con 0x7fa528004420 2019-12-05T06:11:33.857 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:33.846+0000 7fa53dd9b700 1 -- 172.21.15.42:0/1453659132 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa528005250 con 0x7fa528004da0 2019-12-05T06:11:33.857 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:33.846+0000 7fa53e59c700 1 --2- 172.21.15.42:0/1453659132 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa528004420 0x7fa528004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:11:33.857 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:33.846+0000 7fa53ed9d700 1 --2- 172.21.15.42:0/1453659132 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa5280036e0 0x7fa528003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:11:33.857 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:33.846+0000 7fa53f59e700 1 --2- 172.21.15.42:0/1453659132 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa528004da0 0x7fa52804f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:11:33.858 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:33.846+0000 7fa53e59c700 1 -- 172.21.15.42:0/1453659132 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa528004da0 msgr2=0x7fa52804f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:11:33.858 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:33.846+0000 7fa53e59c700 1 --2- 172.21.15.42:0/1453659132 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa528004da0 0x7fa52804f680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:11:33.858 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:33.846+0000 7fa53e59c700 1 -- 172.21.15.42:0/1453659132 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa5280036e0 msgr2=0x7fa528003b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:11:33.858 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:33.846+0000 7fa53e59c700 1 --2- 172.21.15.42:0/1453659132 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa5280036e0 0x7fa528003b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:11:33.858 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:33.846+0000 7fa53e59c700 1 -- 172.21.15.42:0/1453659132 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa52804fd10 con 0x7fa528004420 2019-12-05T06:11:33.858 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:33.846+0000 7fa53f59e700 1 --2- 172.21.15.42:0/1453659132 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa528004da0 0x7fa52804f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:11:33.859 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:33.846+0000 7fa53e59c700 1 --2- 172.21.15.42:0/1453659132 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa528004420 0x7fa528004850 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fa534002f30 tx=0x7fa53400ffe0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:33.859 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:33.846+0000 7fa53e59c700 1 --2- 172.21.15.42:0/1453659132 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa528004420 0x7fa528004850 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fa534002f30 tx=0x7fa53400ffe0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:33.859 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:33.846+0000 7fa53e59c700 1 --2- 172.21.15.42:0/1453659132 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa528004420 0x7fa528004850 secure :-1 s=READY pgs=2205 cs=0 l=1 rx=0x7fa534002f30 tx=0x7fa53400ffe0).ready entity=mon.1 client_cookie=91de079bc8156843 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:11:33.859 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:33.846+0000 7fa53e59c700 1 --2- 172.21.15.42:0/1453659132 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa528004420 0x7fa528004850 secure :-1 s=THROTTLE_DONE pgs=2205 cs=0 l=1 rx=0x7fa534002f30 tx=0x7fa53400ffe0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:33.859 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:33.846+0000 7fa53dd9b700 1 -- 172.21.15.42:0/1453659132 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa534010d10 con 0x7fa528004420 2019-12-05T06:11:33.859 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:33.846+0000 7fa53e59c700 1 --2- 172.21.15.42:0/1453659132 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa528004420 0x7fa528004850 secure :-1 s=THROTTLE_DONE pgs=2205 cs=0 l=1 rx=0x7fa534002f30 tx=0x7fa53400ffe0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:33.859 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:33.846+0000 7fa53dd9b700 1 -- 172.21.15.42:0/1453659132 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa534010930 con 0x7fa528004420 2019-12-05T06:11:33.860 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:33.846+0000 7fa53e59c700 1 --2- 172.21.15.42:0/1453659132 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa528004420 0x7fa528004850 secure :-1 s=THROTTLE_DONE pgs=2205 cs=0 l=1 rx=0x7fa534002f30 tx=0x7fa53400ffe0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:33.861 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:33.846+0000 7fa53dd9b700 1 -- 172.21.15.42:0/1453659132 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa534008bd0 con 0x7fa528004420 2019-12-05T06:11:33.861 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:33.846+0000 7fa54499c700 1 -- 172.21.15.42:0/1453659132 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa528004420 msgr2=0x7fa528004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:33.861 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:33.846+0000 7fa54499c700 1 --2- 172.21.15.42:0/1453659132 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa528004420 0x7fa528004850 secure :-1 s=READY pgs=2205 cs=0 l=1 rx=0x7fa534002f30 tx=0x7fa53400ffe0).stop 2019-12-05T06:11:33.861 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:33.846+0000 7fa53f59e700 1 -- 172.21.15.42:0/1453659132 reap_dead start 2019-12-05T06:11:33.861 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:33.846+0000 7fa54499c700 1 -- 172.21.15.42:0/1453659132 shutdown_connections 2019-12-05T06:11:33.861 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:33.846+0000 7fa54499c700 1 -- 172.21.15.42:0/1453659132 shutdown_connections 2019-12-05T06:11:33.862 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:33.846+0000 7fa54499c700 1 -- 172.21.15.42:0/1453659132 wait complete. 2019-12-05T06:11:33.862 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:33.846+0000 7fa54499c700 1 -- 172.21.15.42:0/1453659132 >> 172.21.15.42:0/1453659132 conn(0x7fa5400aabd0 msgr2=0x7fa5400a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:11:33.867 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:33.850+0000 7fa54499c700 1 Processor -- start 2019-12-05T06:11:33.867 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:33.850+0000 7fa54499c700 1 -- start start 2019-12-05T06:11:33.867 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:33.850+0000 7fa54499c700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa540126ae0 0x7fa5400a68c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:11:33.867 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:33.850+0000 7fa54499c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fa5400a6e10 0x7fa54011aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:11:33.867 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:33.850+0000 7fa54499c700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa54011b400 0x7fa540133870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:11:33.867 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:33.850+0000 7fa54499c700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa54011b830 con 0x7fa54011b400 2019-12-05T06:11:33.868 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:33.850+0000 7fa54499c700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5400a4e60 con 0x7fa5400a6e10 2019-12-05T06:11:33.868 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:33.850+0000 7fa54499c700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5400a5100 con 0x7fa540126ae0 2019-12-05T06:11:33.868 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:33.850+0000 7fa53e59c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fa5400a6e10 0x7fa54011aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:11:33.868 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:33.850+0000 7fa53e59c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fa5400a6e10 0x7fa54011aeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:50122/0 (socket says 172.21.15.42:50122) 2019-12-05T06:11:33.868 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:33.850+0000 7fa53e59c700 1 -- 172.21.15.42:0/564746116 learned_addr learned my addr 172.21.15.42:0/564746116 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:11:33.868 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:33.850+0000 7fa53ed9d700 1 --2- 172.21.15.42:0/564746116 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa540126ae0 0x7fa5400a68c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:11:33.869 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:33.850+0000 7fa53f59e700 1 --2- 172.21.15.42:0/564746116 >> v2:172.21.15.40:3300/0 conn(0x7fa54011b400 0x7fa540133870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:11:33.869 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:33.850+0000 7fa53e59c700 1 -- 172.21.15.42:0/564746116 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa540126ae0 msgr2=0x7fa5400a68c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:33.869 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:33.850+0000 7fa53e59c700 1 --2- 172.21.15.42:0/564746116 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa540126ae0 0x7fa5400a68c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:11:33.869 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:33.850+0000 7fa53e59c700 1 -- 172.21.15.42:0/564746116 >> v2:172.21.15.40:3300/0 conn(0x7fa54011b400 msgr2=0x7fa540133870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:33.869 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:33.850+0000 7fa53e59c700 1 --2- 172.21.15.42:0/564746116 >> v2:172.21.15.40:3300/0 conn(0x7fa54011b400 0x7fa540133870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:11:33.869 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:33.850+0000 7fa53e59c700 1 -- 172.21.15.42:0/564746116 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5400a5640 con 0x7fa5400a6e10 2019-12-05T06:11:33.869 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:33.850+0000 7fa53e59c700 1 --2- 172.21.15.42:0/564746116 >> v2:172.21.15.42:3300/0 conn(0x7fa5400a6e10 0x7fa54011aeb0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fa5400a5020 tx=0x7fa534010f30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:33.870 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:33.850+0000 7fa53e59c700 1 --2- 172.21.15.42:0/564746116 >> v2:172.21.15.42:3300/0 conn(0x7fa5400a6e10 0x7fa54011aeb0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fa5400a5020 tx=0x7fa534010f30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:33.870 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:33.850+0000 7fa53e59c700 1 --2- 172.21.15.42:0/564746116 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa5400a6e10 0x7fa54011aeb0 secure :-1 s=READY pgs=2206 cs=0 l=1 rx=0x7fa5400a5020 tx=0x7fa534010f30).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:11:33.870 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:33.850+0000 7fa53e59c700 1 --2- 172.21.15.42:0/564746116 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa5400a6e10 0x7fa54011aeb0 secure :-1 s=THROTTLE_DONE pgs=2206 cs=0 l=1 rx=0x7fa5400a5020 tx=0x7fa534010f30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:33.870 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:33.850+0000 7fa52f7fe700 1 -- 172.21.15.42:0/564746116 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa53400f680 con 0x7fa5400a6e10 2019-12-05T06:11:33.870 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:33.850+0000 7fa53e59c700 1 --2- 172.21.15.42:0/564746116 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa5400a6e10 0x7fa54011aeb0 secure :-1 s=THROTTLE_DONE pgs=2206 cs=0 l=1 rx=0x7fa5400a5020 tx=0x7fa534010f30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:33.870 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:33.850+0000 7fa52f7fe700 1 -- 172.21.15.42:0/564746116 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5340087f0 con 0x7fa5400a6e10 2019-12-05T06:11:33.871 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:33.850+0000 7fa53e59c700 1 --2- 172.21.15.42:0/564746116 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa5400a6e10 0x7fa54011aeb0 secure :-1 s=THROTTLE_DONE pgs=2206 cs=0 l=1 rx=0x7fa5400a5020 tx=0x7fa534010f30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:33.871 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:33.850+0000 7fa54499c700 1 -- 172.21.15.42:0/564746116 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa540126f10 con 0x7fa5400a6e10 2019-12-05T06:11:33.872 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:33.850+0000 7fa52f7fe700 1 -- 172.21.15.42:0/564746116 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa534005730 con 0x7fa5400a6e10 2019-12-05T06:11:33.872 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:33.850+0000 7fa54499c700 1 -- 172.21.15.42:0/564746116 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa540141e80 con 0x7fa5400a6e10 2019-12-05T06:11:33.872 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:33.850+0000 7fa53e59c700 1 --2- 172.21.15.42:0/564746116 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa5400a6e10 0x7fa54011aeb0 secure :-1 s=THROTTLE_DONE pgs=2206 cs=0 l=1 rx=0x7fa5400a5020 tx=0x7fa534010f30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:33.872 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:33.854+0000 7fa52f7fe700 1 -- 172.21.15.42:0/564746116 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fa534005930 con 0x7fa5400a6e10 2019-12-05T06:11:33.872 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:33.854+0000 7fa53e59c700 1 --2- 172.21.15.42:0/564746116 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa5400a6e10 0x7fa54011aeb0 secure :-1 s=THROTTLE_DONE pgs=2206 cs=0 l=1 rx=0x7fa5400a5020 tx=0x7fa534010f30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:33.873 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:33.854+0000 7fa52f7fe700 1 --2- 172.21.15.42:0/564746116 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa52403cf00 0x7fa52403f410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:11:33.873 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:33.854+0000 7fa52f7fe700 1 -- 172.21.15.42:0/564746116 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 10073+0+0 (secure 0 0 0) 0x7fa53404af30 con 0x7fa5400a6e10 2019-12-05T06:11:33.873 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:33.854+0000 7fa53ed9d700 1 --2- 172.21.15.42:0/564746116 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa52403cf00 0x7fa52403f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:11:33.873 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:33.854+0000 7fa53ed9d700 1 --2- 172.21.15.42:0/564746116 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa52403cf00 0x7fa52403f410 crc :-1 s=READY pgs=927 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:11:33.874 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:33.858+0000 7fa54499c700 1 -- 172.21.15.42:0/564746116 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa54005d100 con 0x7fa5400a6e10 2019-12-05T06:11:33.875 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:33.862+0000 7fa53e59c700 1 --2- 172.21.15.42:0/564746116 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa5400a6e10 0x7fa54011aeb0 secure :-1 s=THROTTLE_DONE pgs=2206 cs=0 l=1 rx=0x7fa5400a5020 tx=0x7fa534010f30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:33.875 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:33.862+0000 7fa52f7fe700 1 -- 172.21.15.42:0/564746116 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fa534031f90 con 0x7fa5400a6e10 2019-12-05T06:11:34.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:34.026+0000 7fa54499c700 1 -- 172.21.15.42:0/564746116 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa54005b8b0 con 0x7fa5400a6e10 2019-12-05T06:11:34.033 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:34.026+0000 7fa53e59c700 1 --2- 172.21.15.42:0/564746116 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa5400a6e10 0x7fa54011aeb0 secure :-1 s=THROTTLE_DONE pgs=2206 cs=0 l=1 rx=0x7fa5400a5020 tx=0x7fa534010f30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:34.033 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:34.026+0000 7fa52f7fe700 1 -- 172.21.15.42:0/564746116 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 266 v266) v1 ==== 95+0+1778 (secure 0 0 0) 0x7fa5340321f0 con 0x7fa5400a6e10 2019-12-05T06:11:34.033 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 266 2019-12-05T06:11:34.033 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:11:34.033 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":266,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":266,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:11:33.944189+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":96,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10000},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10000":{"gid":10000,"name":"a","rank":0,"incarnation":264,"state":"up:rejoin","state_seq":20,"addr":"172.21.15.42:6835/3923117701","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3923117701},{"type":"v1","addr":"172.21.15.42:6835","nonce":3923117701}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:11:34.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:34.030+0000 7fa54499c700 1 -- 172.21.15.42:0/564746116 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa52403cf00 msgr2=0x7fa52403f410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:34.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:34.030+0000 7fa54499c700 1 --2- 172.21.15.42:0/564746116 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa52403cf00 0x7fa52403f410 crc :-1 s=READY pgs=927 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:11:34.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:34.030+0000 7fa54499c700 1 -- 172.21.15.42:0/564746116 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa5400a6e10 msgr2=0x7fa54011aeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:34.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:34.030+0000 7fa54499c700 1 --2- 172.21.15.42:0/564746116 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa5400a6e10 0x7fa54011aeb0 secure :-1 s=READY pgs=2206 cs=0 l=1 rx=0x7fa5400a5020 tx=0x7fa534010f30).stop 2019-12-05T06:11:34.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:34.030+0000 7fa54499c700 1 -- 172.21.15.42:0/564746116 shutdown_connections 2019-12-05T06:11:34.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:34.030+0000 7fa54499c700 1 --2- 172.21.15.42:0/564746116 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa52403cf00 0x7fa52403f410 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:11:34.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:34.030+0000 7fa54499c700 1 --2- 172.21.15.42:0/564746116 >> v2:172.21.15.40:3300/0 conn(0x7fa54011b400 0x7fa540133870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:11:34.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:34.030+0000 7fa54499c700 1 --2- 172.21.15.42:0/564746116 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa540126ae0 0x7fa5400a68c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:11:34.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:34.030+0000 7fa54499c700 1 --2- 172.21.15.42:0/564746116 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa5400a6e10 0x7fa54011aeb0 unknown :-1 s=CLOSED pgs=2206 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:11:34.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:34.030+0000 7fa54499c700 1 -- 172.21.15.42:0/564746116 shutdown_connections 2019-12-05T06:11:34.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:34.030+0000 7fa54499c700 1 -- 172.21.15.42:0/564746116 wait complete. 2019-12-05T06:11:34.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:34.030+0000 7fa54499c700 1 -- 172.21.15.42:0/564746116 >> 172.21.15.42:0/564746116 conn(0x7fa5400a80a0 msgr2=0x7fa54011c3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:11:34.044 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2019-12-05T06:11:36.045 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:11:36.149 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.142+0000 7f4eb2898700 1 Processor -- start 2019-12-05T06:11:36.150 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.142+0000 7f4eb2898700 1 -- start start 2019-12-05T06:11:36.150 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.142+0000 7f4eb2898700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f4eac0a7740 0x7f4eac08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:11:36.150 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.142+0000 7f4eb2898700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4eac09fae0 0x7f4eac07eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:11:36.150 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.142+0000 7f4eb2898700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4eac0a0710 con 0x7f4eac07e720 2019-12-05T06:11:36.150 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.142+0000 7f4eb2898700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4eac0a0b00 con 0x7f4eac0a7740 2019-12-05T06:11:36.150 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.142+0000 7f4eb2898700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4eac08e5b0 con 0x7f4eac09fae0 2019-12-05T06:11:36.151 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.142+0000 7f4eabfff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f4eac07e720 0x7f4eac09f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:50172/0 (socket says 172.21.15.42:50172) 2019-12-05T06:11:36.151 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.142+0000 7f4eabfff700 1 -- 172.21.15.42:0/4100806055 learned_addr learned my addr 172.21.15.42:0/4100806055 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:11:36.151 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.142+0000 7f4eb1636700 1 --2- 172.21.15.42:0/4100806055 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4eac09fae0 0x7f4eac07eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:11:36.151 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.142+0000 7f4eb0e35700 1 --2- 172.21.15.42:0/4100806055 >> v2:172.21.15.40:3300/0 conn(0x7f4eac0a7740 0x7f4eac08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:11:36.151 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.142+0000 7f4eab7fe700 1 -- 172.21.15.42:0/4100806055 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2020824434 0 0) 0x7f4eac0a0710 con 0x7f4eac07e720 2019-12-05T06:11:36.151 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.142+0000 7f4eab7fe700 1 -- 172.21.15.42:0/4100806055 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e9c0030b0 con 0x7f4eac07e720 2019-12-05T06:11:36.151 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.142+0000 7f4eb0e35700 1 -- 172.21.15.42:0/4100806055 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4eac09fae0 msgr2=0x7f4eac07eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:11:36.152 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.142+0000 7f4eb0e35700 1 --2- 172.21.15.42:0/4100806055 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4eac09fae0 0x7f4eac07eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:11:36.152 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.142+0000 7f4eb0e35700 1 -- 172.21.15.42:0/4100806055 >> v1:172.21.15.42:6789/0 conn(0x7f4eac07e720 legacy=0x7f4eac09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:36.152 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.142+0000 7f4eb0e35700 1 -- 172.21.15.42:0/4100806055 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4eac07f0a0 con 0x7f4eac0a7740 2019-12-05T06:11:36.152 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.142+0000 7f4eb0e35700 1 --2- 172.21.15.42:0/4100806055 >> v2:172.21.15.40:3300/0 conn(0x7f4eac0a7740 0x7f4eac08df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f4e98001990 tx=0x7f4e980014a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:36.152 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.142+0000 7f4eb1636700 1 --2- 172.21.15.42:0/4100806055 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4eac09fae0 0x7f4eac07eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:11:36.152 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.142+0000 7f4eb0e35700 1 --2- 172.21.15.42:0/4100806055 >> v2:172.21.15.40:3300/0 conn(0x7f4eac0a7740 0x7f4eac08df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f4e98001990 tx=0x7f4e980014a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:36.152 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.142+0000 7f4eb0e35700 1 --2- 172.21.15.42:0/4100806055 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4eac0a7740 0x7f4eac08df80 secure :-1 s=READY pgs=1670 cs=0 l=1 rx=0x7f4e98001990 tx=0x7f4e980014a0).ready entity=mon.0 client_cookie=ebd1142727c38c4c server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:11:36.153 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.142+0000 7f4eb0e35700 1 --2- 172.21.15.42:0/4100806055 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4eac0a7740 0x7f4eac08df80 secure :-1 s=THROTTLE_DONE pgs=1670 cs=0 l=1 rx=0x7f4e98001990 tx=0x7f4e980014a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:36.153 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.142+0000 7f4eb0e35700 1 --2- 172.21.15.42:0/4100806055 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4eac0a7740 0x7f4eac08df80 secure :-1 s=THROTTLE_DONE pgs=1670 cs=0 l=1 rx=0x7f4e98001990 tx=0x7f4e980014a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:36.153 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.142+0000 7f4eab7fe700 1 -- 172.21.15.42:0/4100806055 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4e98001750 con 0x7f4eac0a7740 2019-12-05T06:11:36.153 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.142+0000 7f4eb0e35700 1 --2- 172.21.15.42:0/4100806055 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4eac0a7740 0x7f4eac08df80 secure :-1 s=THROTTLE_DONE pgs=1670 cs=0 l=1 rx=0x7f4e98001990 tx=0x7f4e980014a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:36.154 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.142+0000 7f4eab7fe700 1 -- 172.21.15.42:0/4100806055 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e98001560 con 0x7f4eac0a7740 2019-12-05T06:11:36.154 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.146+0000 7f4eab7fe700 1 -- 172.21.15.42:0/4100806055 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4ea0008b70 con 0x7f4eac0a7740 2019-12-05T06:11:36.154 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.146+0000 7f4eb2898700 1 -- 172.21.15.42:0/4100806055 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4eac0a7740 msgr2=0x7f4eac08df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:36.154 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.146+0000 7f4eb2898700 1 --2- 172.21.15.42:0/4100806055 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4eac0a7740 0x7f4eac08df80 secure :-1 s=READY pgs=1670 cs=0 l=1 rx=0x7f4e98001990 tx=0x7f4e980014a0).stop 2019-12-05T06:11:36.154 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.146+0000 7f4eb2898700 1 -- 172.21.15.42:0/4100806055 shutdown_connections 2019-12-05T06:11:36.154 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.146+0000 7f4eb2898700 1 --2- 172.21.15.42:0/4100806055 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4eac09fae0 0x7f4eac07eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:11:36.155 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.146+0000 7f4eb2898700 1 --2- 172.21.15.42:0/4100806055 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4eac0a7740 0x7f4eac08df80 unknown :-1 s=CLOSED pgs=1670 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:11:36.155 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.146+0000 7f4eb2898700 1 -- 172.21.15.42:0/4100806055 shutdown_connections 2019-12-05T06:11:36.155 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.146+0000 7f4eb2898700 1 -- 172.21.15.42:0/4100806055 wait complete. 2019-12-05T06:11:36.155 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.146+0000 7f4eb2898700 1 -- 172.21.15.42:0/4100806055 >> 172.21.15.42:0/4100806055 conn(0x7f4eac0aabe0 msgr2=0x7f4eac0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:11:36.161 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.146+0000 7f4eb2898700 1 Processor -- start 2019-12-05T06:11:36.161 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.146+0000 7f4eb2898700 1 -- start start 2019-12-05T06:11:36.161 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.146+0000 7f4eb2898700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f4eac119760 0x7f4eac059c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:11:36.161 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.146+0000 7f4eb2898700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f4eac11bb00 0x7f4eac08f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:11:36.161 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.146+0000 7f4eb2898700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4eac05a160 con 0x7f4eac11b6d0 2019-12-05T06:11:36.162 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.146+0000 7f4eb2898700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4eac144f60 con 0x7f4eac119760 2019-12-05T06:11:36.162 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.146+0000 7f4eb2898700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4eac145200 con 0x7f4eac11bb00 2019-12-05T06:11:36.162 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.150+0000 7f4eb1636700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f4eac11bb00 0x7f4eac08f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:11:36.162 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.150+0000 7f4eb1636700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f4eac11bb00 0x7f4eac08f370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:50142/0 (socket says 172.21.15.42:50142) 2019-12-05T06:11:36.162 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.150+0000 7f4eb1636700 1 -- 172.21.15.42:0/2535753383 learned_addr learned my addr 172.21.15.42:0/2535753383 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:11:36.162 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.150+0000 7f4eb0e35700 1 --2- 172.21.15.42:0/2535753383 >> v2:172.21.15.40:3300/0 conn(0x7f4eac119760 0x7f4eac059c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:11:36.162 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.150+0000 7f4eb1636700 1 -- 172.21.15.42:0/2535753383 >> v2:172.21.15.40:3300/0 conn(0x7f4eac119760 msgr2=0x7f4eac059c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:36.163 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.150+0000 7f4eb1636700 1 --2- 172.21.15.42:0/2535753383 >> v2:172.21.15.40:3300/0 conn(0x7f4eac119760 0x7f4eac059c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:11:36.163 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.150+0000 7f4eb1636700 1 -- 172.21.15.42:0/2535753383 >> v1:172.21.15.40:6789/0 conn(0x7f4eac11b6d0 legacy=0x7f4eac08eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:36.163 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.150+0000 7f4eb1636700 1 -- 172.21.15.42:0/2535753383 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4eac119b90 con 0x7f4eac11bb00 2019-12-05T06:11:36.163 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.150+0000 7f4eb1636700 1 --2- 172.21.15.42:0/2535753383 >> v2:172.21.15.42:3300/0 conn(0x7f4eac11bb00 0x7f4eac08f370 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f4eac11c3a0 tx=0x7f4ea4010f00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:36.163 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.150+0000 7f4eb1636700 1 --2- 172.21.15.42:0/2535753383 >> v2:172.21.15.42:3300/0 conn(0x7f4eac11bb00 0x7f4eac08f370 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f4eac11c3a0 tx=0x7f4ea4010f00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:36.163 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.150+0000 7f4eb1636700 1 --2- 172.21.15.42:0/2535753383 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4eac11bb00 0x7f4eac08f370 secure :-1 s=READY pgs=2208 cs=0 l=1 rx=0x7f4eac11c3a0 tx=0x7f4ea4010f00).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:11:36.163 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.150+0000 7f4eb1636700 1 --2- 172.21.15.42:0/2535753383 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4eac11bb00 0x7f4eac08f370 secure :-1 s=THROTTLE_DONE pgs=2208 cs=0 l=1 rx=0x7f4eac11c3a0 tx=0x7f4ea4010f00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:36.164 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.150+0000 7f4ea97fa700 1 -- 172.21.15.42:0/2535753383 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4ea4005d40 con 0x7f4eac11bb00 2019-12-05T06:11:36.164 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.150+0000 7f4eb1636700 1 --2- 172.21.15.42:0/2535753383 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4eac11bb00 0x7f4eac08f370 secure :-1 s=THROTTLE_DONE pgs=2208 cs=0 l=1 rx=0x7f4eac11c3a0 tx=0x7f4ea4010f00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:36.164 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.150+0000 7f4ea97fa700 1 -- 172.21.15.42:0/2535753383 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ea4005960 con 0x7f4eac11bb00 2019-12-05T06:11:36.164 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.150+0000 7f4eb1636700 1 --2- 172.21.15.42:0/2535753383 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4eac11bb00 0x7f4eac08f370 secure :-1 s=THROTTLE_DONE pgs=2208 cs=0 l=1 rx=0x7f4eac11c3a0 tx=0x7f4ea4010f00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:36.165 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.150+0000 7f4eb2898700 1 -- 172.21.15.42:0/2535753383 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4eac08f8c0 con 0x7f4eac11bb00 2019-12-05T06:11:36.165 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.150+0000 7f4ea97fa700 1 -- 172.21.15.42:0/2535753383 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4ea4011c00 con 0x7f4eac11bb00 2019-12-05T06:11:36.166 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.150+0000 7f4eb2898700 1 -- 172.21.15.42:0/2535753383 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4eac11a2f0 con 0x7f4eac11bb00 2019-12-05T06:11:36.166 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.150+0000 7f4eb1636700 1 --2- 172.21.15.42:0/2535753383 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4eac11bb00 0x7f4eac08f370 secure :-1 s=THROTTLE_DONE pgs=2208 cs=0 l=1 rx=0x7f4eac11c3a0 tx=0x7f4ea4010f00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:36.166 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.150+0000 7f4ea97fa700 1 -- 172.21.15.42:0/2535753383 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f4ea4007180 con 0x7f4eac11bb00 2019-12-05T06:11:36.167 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.150+0000 7f4eb1636700 1 --2- 172.21.15.42:0/2535753383 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4eac11bb00 0x7f4eac08f370 secure :-1 s=THROTTLE_DONE pgs=2208 cs=0 l=1 rx=0x7f4eac11c3a0 tx=0x7f4ea4010f00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:36.167 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.150+0000 7f4ea97fa700 1 --2- 172.21.15.42:0/2535753383 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4e8c03ae20 0x7f4e8c03d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:11:36.167 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.150+0000 7f4ea97fa700 1 -- 172.21.15.42:0/2535753383 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 10073+0+0 (secure 0 0 0) 0x7f4ea404afa0 con 0x7f4eac11bb00 2019-12-05T06:11:36.167 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.150+0000 7f4eb0e35700 1 --2- 172.21.15.42:0/2535753383 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4e8c03ae20 0x7f4e8c03d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:11:36.168 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.150+0000 7f4eb0e35700 1 --2- 172.21.15.42:0/2535753383 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4e8c03ae20 0x7f4e8c03d330 crc :-1 s=READY pgs=928 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:11:36.168 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.158+0000 7f4eb2898700 1 -- 172.21.15.42:0/2535753383 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4eac05d110 con 0x7f4eac11bb00 2019-12-05T06:11:36.169 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.162+0000 7f4eb1636700 1 --2- 172.21.15.42:0/2535753383 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4eac11bb00 0x7f4eac08f370 secure :-1 s=THROTTLE_DONE pgs=2208 cs=0 l=1 rx=0x7f4eac11c3a0 tx=0x7f4ea4010f00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:36.169 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.162+0000 7f4ea97fa700 1 -- 172.21.15.42:0/2535753383 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f4ea4031fa0 con 0x7f4eac11bb00 2019-12-05T06:11:36.368 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.366+0000 7f4eb2898700 1 -- 172.21.15.42:0/2535753383 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4eac05b8c0 con 0x7f4eac11bb00 2019-12-05T06:11:36.369 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.366+0000 7f4eb1636700 1 --2- 172.21.15.42:0/2535753383 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4eac11bb00 0x7f4eac08f370 secure :-1 s=THROTTLE_DONE pgs=2208 cs=0 l=1 rx=0x7f4eac11c3a0 tx=0x7f4ea4010f00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:36.369 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.366+0000 7f4ea97fa700 1 -- 172.21.15.42:0/2535753383 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 267 v267) v1 ==== 95+0+1778 (secure 0 0 0) 0x7f4ea4032200 con 0x7f4eac11bb00 2019-12-05T06:11:36.369 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 267 2019-12-05T06:11:36.370 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:11:36.370 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":267,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":267,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:11:34.953298+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":96,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10000},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10000":{"gid":10000,"name":"a","rank":0,"incarnation":264,"state":"up:active","state_seq":21,"addr":"172.21.15.42:6835/3923117701","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3923117701},{"type":"v1","addr":"172.21.15.42:6835","nonce":3923117701}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:11:36.372 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.366+0000 7f4eb2898700 1 -- 172.21.15.42:0/2535753383 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4e8c03ae20 msgr2=0x7f4e8c03d330 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:36.372 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.366+0000 7f4eb2898700 1 --2- 172.21.15.42:0/2535753383 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4e8c03ae20 0x7f4e8c03d330 crc :-1 s=READY pgs=928 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:11:36.373 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.366+0000 7f4eb2898700 1 -- 172.21.15.42:0/2535753383 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4eac11bb00 msgr2=0x7f4eac08f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:36.373 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.366+0000 7f4eb2898700 1 --2- 172.21.15.42:0/2535753383 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4eac11bb00 0x7f4eac08f370 secure :-1 s=READY pgs=2208 cs=0 l=1 rx=0x7f4eac11c3a0 tx=0x7f4ea4010f00).stop 2019-12-05T06:11:36.373 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.366+0000 7f4eb2898700 1 -- 172.21.15.42:0/2535753383 shutdown_connections 2019-12-05T06:11:36.373 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.366+0000 7f4eb2898700 1 --2- 172.21.15.42:0/2535753383 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4e8c03ae20 0x7f4e8c03d330 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:11:36.373 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.366+0000 7f4eb2898700 1 --2- 172.21.15.42:0/2535753383 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4eac11bb00 0x7f4eac08f370 unknown :-1 s=CLOSED pgs=2208 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:11:36.373 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.366+0000 7f4eb2898700 1 --2- 172.21.15.42:0/2535753383 >> v2:172.21.15.40:3300/0 conn(0x7f4eac119760 0x7f4eac059c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:11:36.373 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.370+0000 7f4eb2898700 1 -- 172.21.15.42:0/2535753383 shutdown_connections 2019-12-05T06:11:36.374 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.370+0000 7f4eb2898700 1 -- 172.21.15.42:0/2535753383 wait complete. 2019-12-05T06:11:36.374 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.370+0000 7f4eb2898700 1 -- 172.21.15.42:0/2535753383 >> 172.21.15.42:0/2535753383 conn(0x7f4eac0a6bb0 msgr2=0x7f4eac11c470 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:11:36.385 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a has gained rank=0, replacing gid=6977 2019-12-05T06:11:36.386 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 0 secs before reviving mds.b 2019-12-05T06:11:36.386 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.b 2019-12-05T06:11:36.386 INFO:tasks.ceph.mds.b:Restarting daemon 2019-12-05T06:11:36.394 INFO:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2019-12-05T06:11:36.399 INFO:tasks.ceph.mds.b:Started 2019-12-05T06:11:36.399 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:11:36.460 INFO:tasks.ceph.mds.b.smithi040.stdout:starting mds.b at 2019-12-05T06:11:36.508 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.498+0000 7f590935c700 1 Processor -- start 2019-12-05T06:11:36.509 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.498+0000 7f590935c700 1 -- start start 2019-12-05T06:11:36.509 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.502+0000 7f590935c700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f59040a7740 0x7f590408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:11:36.509 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.502+0000 7f590935c700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59040a0710 con 0x7f590409fae0 2019-12-05T06:11:36.509 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.502+0000 7f590935c700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59040a9ba0 con 0x7f590407e720 2019-12-05T06:11:36.509 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.502+0000 7f590935c700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59040a0b00 con 0x7f59040a7740 2019-12-05T06:11:36.510 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.502+0000 7f5902ffd700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f590407e720 0x7f590409f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:50188/0 (socket says 172.21.15.42:50188) 2019-12-05T06:11:36.510 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.502+0000 7f5902ffd700 1 -- 172.21.15.42:0/2952659117 learned_addr learned my addr 172.21.15.42:0/2952659117 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:11:36.510 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.502+0000 7f59037fe700 1 --2- 172.21.15.42:0/2952659117 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f59040a7740 0x7f590408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:11:36.510 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.502+0000 7f59027fc700 1 -- 172.21.15.42:0/2952659117 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4288562145 0 0) 0x7f59040a9ba0 con 0x7f590407e720 2019-12-05T06:11:36.510 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.502+0000 7f59027fc700 1 -- 172.21.15.42:0/2952659117 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58ec0030b0 con 0x7f590407e720 2019-12-05T06:11:36.510 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.502+0000 7f59027fc700 1 -- 172.21.15.42:0/2952659117 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1790428663 0 0) 0x7f59040a0710 con 0x7f590409fae0 2019-12-05T06:11:36.511 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.502+0000 7f59027fc700 1 -- 172.21.15.42:0/2952659117 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59040a9ba0 con 0x7f590409fae0 2019-12-05T06:11:36.511 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.502+0000 7f59037fe700 1 -- 172.21.15.42:0/2952659117 >> v1:172.21.15.42:6789/0 conn(0x7f590407e720 legacy=0x7f590409f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:36.511 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.502+0000 7f59037fe700 1 -- 172.21.15.42:0/2952659117 >> v1:172.21.15.40:6789/0 conn(0x7f590409fae0 legacy=0x7f590414efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:36.511 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.502+0000 7f59037fe700 1 -- 172.21.15.42:0/2952659117 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f590407eb50 con 0x7f59040a7740 2019-12-05T06:11:36.511 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.502+0000 7f59037fe700 1 --2- 172.21.15.42:0/2952659117 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f59040a7740 0x7f590408df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f58fc0032b0 tx=0x7f58fc00ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:36.511 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.502+0000 7f59037fe700 1 --2- 172.21.15.42:0/2952659117 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f59040a7740 0x7f590408df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f58fc0032b0 tx=0x7f58fc00ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:36.511 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.502+0000 7f59037fe700 1 --2- 172.21.15.42:0/2952659117 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f59040a7740 0x7f590408df80 secure :-1 s=READY pgs=483 cs=0 l=1 rx=0x7f58fc0032b0 tx=0x7f58fc00ff70).ready entity=mon.2 client_cookie=e0fc91334752caa8 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:11:36.512 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.502+0000 7f59037fe700 1 --2- 172.21.15.42:0/2952659117 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f59040a7740 0x7f590408df80 secure :-1 s=THROTTLE_DONE pgs=483 cs=0 l=1 rx=0x7f58fc0032b0 tx=0x7f58fc00ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:36.512 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.502+0000 7f59027fc700 1 -- 172.21.15.42:0/2952659117 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f58fc011a10 con 0x7f59040a7740 2019-12-05T06:11:36.512 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.502+0000 7f59037fe700 1 --2- 172.21.15.42:0/2952659117 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f59040a7740 0x7f590408df80 secure :-1 s=THROTTLE_DONE pgs=483 cs=0 l=1 rx=0x7f58fc0032b0 tx=0x7f58fc00ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:36.512 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.502+0000 7f59027fc700 1 -- 172.21.15.42:0/2952659117 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58fc011630 con 0x7f59040a7740 2019-12-05T06:11:36.512 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.502+0000 7f59037fe700 1 --2- 172.21.15.42:0/2952659117 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f59040a7740 0x7f590408df80 secure :-1 s=THROTTLE_DONE pgs=483 cs=0 l=1 rx=0x7f58fc0032b0 tx=0x7f58fc00ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:36.512 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.502+0000 7f590935c700 1 -- 172.21.15.42:0/2952659117 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f59040a7740 msgr2=0x7f590408df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:36.512 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.502+0000 7f59027fc700 1 -- 172.21.15.42:0/2952659117 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f58fc00a160 con 0x7f59040a7740 2019-12-05T06:11:36.513 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.502+0000 7f590935c700 1 --2- 172.21.15.42:0/2952659117 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f59040a7740 0x7f590408df80 secure :-1 s=READY pgs=483 cs=0 l=1 rx=0x7f58fc0032b0 tx=0x7f58fc00ff70).stop 2019-12-05T06:11:36.513 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.502+0000 7f590935c700 1 -- 172.21.15.42:0/2952659117 shutdown_connections 2019-12-05T06:11:36.513 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.502+0000 7f590935c700 1 --2- 172.21.15.42:0/2952659117 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f59040a7740 0x7f590408df80 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:11:36.513 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.502+0000 7f590935c700 1 -- 172.21.15.42:0/2952659117 shutdown_connections 2019-12-05T06:11:36.513 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.502+0000 7f590935c700 1 -- 172.21.15.42:0/2952659117 wait complete. 2019-12-05T06:11:36.513 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.502+0000 7f590935c700 1 -- 172.21.15.42:0/2952659117 >> 172.21.15.42:0/2952659117 conn(0x7f59040aabe0 msgr2=0x7f59040a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:11:36.521 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.506+0000 7f590935c700 1 Processor -- start 2019-12-05T06:11:36.521 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.506+0000 7f590935c700 1 -- start start 2019-12-05T06:11:36.521 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.506+0000 7f590935c700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f590407b260 0x7f5904143590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:11:36.522 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.506+0000 7f590935c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f590407d030 0x7f5904126a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:11:36.525 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.506+0000 7f590935c700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5904120d40 con 0x7f5904126f70 2019-12-05T06:11:36.539 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.506+0000 7f590935c700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5904143ae0 con 0x7f590407d030 2019-12-05T06:11:36.539 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.506+0000 7f590935c700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5904143d70 con 0x7f590407b260 2019-12-05T06:11:36.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.506+0000 7f5902ffd700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f590407d030 0x7f5904126a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:11:36.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.506+0000 7f5902ffd700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f590407d030 0x7f5904126a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:50156/0 (socket says 172.21.15.42:50156) 2019-12-05T06:11:36.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.506+0000 7f5902ffd700 1 -- 172.21.15.42:0/1931136227 learned_addr learned my addr 172.21.15.42:0/1931136227 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:11:36.543 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.506+0000 7f5903fff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f5904126f70 0x7f5904120540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:50196/0 (socket says 172.21.15.42:50196) 2019-12-05T06:11:36.543 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.506+0000 7f59037fe700 1 --2- 172.21.15.42:0/1931136227 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f590407b260 0x7f5904143590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:11:36.543 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.510+0000 7f5902ffd700 1 -- 172.21.15.42:0/1931136227 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f590407b260 msgr2=0x7f5904143590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:36.543 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.510+0000 7f5902ffd700 1 --2- 172.21.15.42:0/1931136227 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f590407b260 0x7f5904143590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:11:36.543 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.510+0000 7f5902ffd700 1 -- 172.21.15.42:0/1931136227 >> v1:172.21.15.42:6789/0 conn(0x7f5904126f70 legacy=0x7f5904120540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:36.544 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.510+0000 7f5902ffd700 1 -- 172.21.15.42:0/1931136227 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5904144490 con 0x7f590407d030 2019-12-05T06:11:36.548 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.510+0000 7f5902ffd700 1 --2- 172.21.15.42:0/1931136227 >> v2:172.21.15.42:3300/0 conn(0x7f590407d030 0x7f5904126a20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f58f40043a0 tx=0x7f58f4010a00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:36.549 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.510+0000 7f5902ffd700 1 --2- 172.21.15.42:0/1931136227 >> v2:172.21.15.42:3300/0 conn(0x7f590407d030 0x7f5904126a20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f58f40043a0 tx=0x7f58f4010a00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:36.550 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.510+0000 7f5902ffd700 1 --2- 172.21.15.42:0/1931136227 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f590407d030 0x7f5904126a20 secure :-1 s=READY pgs=2214 cs=0 l=1 rx=0x7f58f40043a0 tx=0x7f58f4010a00).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:11:36.550 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.510+0000 7f5902ffd700 1 --2- 172.21.15.42:0/1931136227 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f590407d030 0x7f5904126a20 secure :-1 s=THROTTLE_DONE pgs=2214 cs=0 l=1 rx=0x7f58f40043a0 tx=0x7f58f4010a00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:36.552 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.510+0000 7f5902ffd700 1 --2- 172.21.15.42:0/1931136227 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f590407d030 0x7f5904126a20 secure :-1 s=THROTTLE_DONE pgs=2214 cs=0 l=1 rx=0x7f58f40043a0 tx=0x7f58f4010a00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:36.552 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.510+0000 7f5902ffd700 1 --2- 172.21.15.42:0/1931136227 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f590407d030 0x7f5904126a20 secure :-1 s=THROTTLE_DONE pgs=2214 cs=0 l=1 rx=0x7f58f40043a0 tx=0x7f58f4010a00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:36.554 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.510+0000 7f58ebfff700 1 -- 172.21.15.42:0/1931136227 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f58f400cab0 con 0x7f590407d030 2019-12-05T06:11:36.554 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.510+0000 7f58ebfff700 1 -- 172.21.15.42:0/1931136227 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58f4010250 con 0x7f590407d030 2019-12-05T06:11:36.554 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.510+0000 7f590935c700 1 -- 172.21.15.42:0/1931136227 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5904144010 con 0x7f590407d030 2019-12-05T06:11:36.554 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.510+0000 7f590935c700 1 -- 172.21.15.42:0/1931136227 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f590412de70 con 0x7f590407d030 2019-12-05T06:11:36.555 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.510+0000 7f58ebfff700 1 -- 172.21.15.42:0/1931136227 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f58f400d2b0 con 0x7f590407d030 2019-12-05T06:11:36.555 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.510+0000 7f5902ffd700 1 --2- 172.21.15.42:0/1931136227 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f590407d030 0x7f5904126a20 secure :-1 s=THROTTLE_DONE pgs=2214 cs=0 l=1 rx=0x7f58f40043a0 tx=0x7f58f4010a00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:36.555 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.510+0000 7f58ebfff700 1 -- 172.21.15.42:0/1931136227 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f58f400d4b0 con 0x7f590407d030 2019-12-05T06:11:36.555 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.510+0000 7f5902ffd700 1 --2- 172.21.15.42:0/1931136227 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f590407d030 0x7f5904126a20 secure :-1 s=THROTTLE_DONE pgs=2214 cs=0 l=1 rx=0x7f58f40043a0 tx=0x7f58f4010a00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:36.555 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.510+0000 7f58ebfff700 1 --2- 172.21.15.42:0/1931136227 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f58e403ad40 0x7f58e403d250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:11:36.556 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.510+0000 7f58ebfff700 1 -- 172.21.15.42:0/1931136227 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 10073+0+0 (secure 0 0 0) 0x7f58f404aab0 con 0x7f590407d030 2019-12-05T06:11:36.556 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.510+0000 7f59037fe700 1 --2- 172.21.15.42:0/1931136227 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f58e403ad40 0x7f58e403d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:11:36.556 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.510+0000 7f59037fe700 1 --2- 172.21.15.42:0/1931136227 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f58e403ad40 0x7f58e403d250 crc :-1 s=READY pgs=929 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:11:36.556 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.514+0000 7f590935c700 1 -- 172.21.15.42:0/1931136227 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58dc003200 con 0x7f590407d030 2019-12-05T06:11:36.560 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.518+0000 7f5902ffd700 1 --2- 172.21.15.42:0/1931136227 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f590407d030 0x7f5904126a20 secure :-1 s=THROTTLE_DONE pgs=2214 cs=0 l=1 rx=0x7f58f40043a0 tx=0x7f58f4010a00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:36.560 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.518+0000 7f58ebfff700 1 -- 172.21.15.42:0/1931136227 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f5904144490 con 0x7f590407d030 2019-12-05T06:11:36.711 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.706+0000 7f590935c700 1 -- 172.21.15.42:0/1931136227 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f58dc003630 con 0x7f590407d030 2019-12-05T06:11:36.712 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.710+0000 7f5902ffd700 1 --2- 172.21.15.42:0/1931136227 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f590407d030 0x7f5904126a20 secure :-1 s=THROTTLE_DONE pgs=2214 cs=0 l=1 rx=0x7f58f40043a0 tx=0x7f58f4010a00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:36.712 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.710+0000 7f58ebfff700 1 -- 172.21.15.42:0/1931136227 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 267 v267) v1 ==== 95+0+1778 (secure 0 0 0) 0x7f58f4031b70 con 0x7f590407d030 2019-12-05T06:11:36.713 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 267 2019-12-05T06:11:36.713 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:11:36.713 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":267,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":267,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:11:34.953298+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":96,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10000},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10000":{"gid":10000,"name":"a","rank":0,"incarnation":264,"state":"up:active","state_seq":21,"addr":"172.21.15.42:6835/3923117701","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3923117701},{"type":"v1","addr":"172.21.15.42:6835","nonce":3923117701}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:11:36.716 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.710+0000 7f590935c700 1 -- 172.21.15.42:0/1931136227 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f58e403ad40 msgr2=0x7f58e403d250 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:36.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.710+0000 7f590935c700 1 --2- 172.21.15.42:0/1931136227 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f58e403ad40 0x7f58e403d250 crc :-1 s=READY pgs=929 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:11:36.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.710+0000 7f590935c700 1 -- 172.21.15.42:0/1931136227 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f590407d030 msgr2=0x7f5904126a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:36.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.710+0000 7f590935c700 1 --2- 172.21.15.42:0/1931136227 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f590407d030 0x7f5904126a20 secure :-1 s=READY pgs=2214 cs=0 l=1 rx=0x7f58f40043a0 tx=0x7f58f4010a00).stop 2019-12-05T06:11:36.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.710+0000 7f590935c700 1 -- 172.21.15.42:0/1931136227 shutdown_connections 2019-12-05T06:11:36.718 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.710+0000 7f590935c700 1 --2- 172.21.15.42:0/1931136227 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f58e403ad40 0x7f58e403d250 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:11:36.718 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.710+0000 7f590935c700 1 --2- 172.21.15.42:0/1931136227 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f590407b260 0x7f5904143590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:11:36.725 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.710+0000 7f590935c700 1 --2- 172.21.15.42:0/1931136227 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f590407d030 0x7f5904126a20 unknown :-1 s=CLOSED pgs=2214 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:11:36.725 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.710+0000 7f590935c700 1 -- 172.21.15.42:0/1931136227 shutdown_connections 2019-12-05T06:11:36.725 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.710+0000 7f590935c700 1 -- 172.21.15.42:0/1931136227 wait complete. 2019-12-05T06:11:36.725 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:36.710+0000 7f590935c700 1 -- 172.21.15.42:0/1931136227 >> 172.21.15.42:0/1931136227 conn(0x7f59040a7740 msgr2=0x7f590414e840 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:11:36.727 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2019-12-05T06:11:38.727 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:11:38.837 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:38.826+0000 7f6e22fd0700 1 Processor -- start 2019-12-05T06:11:38.837 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:38.826+0000 7f6e22fd0700 1 -- start start 2019-12-05T06:11:38.837 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:38.830+0000 7f6e22fd0700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6e1c08df80 0x7f6e1c09f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:11:38.838 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:38.830+0000 7f6e22fd0700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6e1c09f830 0x7f6e1c14efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:11:38.838 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:38.830+0000 7f6e22fd0700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e1c09fc60 con 0x7f6e1c0a7740 2019-12-05T06:11:38.838 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:38.830+0000 7f6e22fd0700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e1c0a0b00 con 0x7f6e1c09f830 2019-12-05T06:11:38.838 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:38.830+0000 7f6e22fd0700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e1c08e5b0 con 0x7f6e1c08df80 2019-12-05T06:11:38.838 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:38.830+0000 7f6e21d6e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6e1c09f830 0x7f6e1c14efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:11:38.838 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:38.830+0000 7f6e2156d700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f6e1c0a7740 0x7f6e1c07e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:50212/0 (socket says 172.21.15.42:50212) 2019-12-05T06:11:38.838 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:38.830+0000 7f6e2156d700 1 -- 172.21.15.42:0/2217941733 learned_addr learned my addr 172.21.15.42:0/2217941733 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:11:38.839 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:38.830+0000 7f6e20d6c700 1 --2- 172.21.15.42:0/2217941733 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6e1c08df80 0x7f6e1c09f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:11:38.839 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:38.830+0000 7f6e21d6e700 1 -- 172.21.15.42:0/2217941733 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6e1c08df80 msgr2=0x7f6e1c09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:11:38.839 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:38.830+0000 7f6e21d6e700 1 --2- 172.21.15.42:0/2217941733 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6e1c08df80 0x7f6e1c09f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:11:38.839 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:38.830+0000 7f6e21d6e700 1 -- 172.21.15.42:0/2217941733 >> v1:172.21.15.42:6789/0 conn(0x7f6e1c0a7740 legacy=0x7f6e1c07e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:38.839 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:38.830+0000 7f6e21d6e700 1 -- 172.21.15.42:0/2217941733 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e1c0a0710 con 0x7f6e1c09f830 2019-12-05T06:11:38.839 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:38.830+0000 7f6e21d6e700 1 --2- 172.21.15.42:0/2217941733 >> v2:172.21.15.42:3300/0 conn(0x7f6e1c09f830 0x7f6e1c14efe0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f6e14008b10 tx=0x7f6e1400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:38.839 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:38.830+0000 7f6e21d6e700 1 --2- 172.21.15.42:0/2217941733 >> v2:172.21.15.42:3300/0 conn(0x7f6e1c09f830 0x7f6e1c14efe0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f6e14008b10 tx=0x7f6e1400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:38.840 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:38.830+0000 7f6e21d6e700 1 --2- 172.21.15.42:0/2217941733 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6e1c09f830 0x7f6e1c14efe0 secure :-1 s=READY pgs=2216 cs=0 l=1 rx=0x7f6e14008b10 tx=0x7f6e1400fff0).ready entity=mon.1 client_cookie=e626a1a9cbe835b5 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:11:38.840 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:38.830+0000 7f6e21d6e700 1 --2- 172.21.15.42:0/2217941733 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6e1c09f830 0x7f6e1c14efe0 secure :-1 s=THROTTLE_DONE pgs=2216 cs=0 l=1 rx=0x7f6e14008b10 tx=0x7f6e1400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:38.840 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:38.830+0000 7f6e21d6e700 1 --2- 172.21.15.42:0/2217941733 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6e1c09f830 0x7f6e1c14efe0 secure :-1 s=THROTTLE_DONE pgs=2216 cs=0 l=1 rx=0x7f6e14008b10 tx=0x7f6e1400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:38.840 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:38.830+0000 7f6e0ffff700 1 -- 172.21.15.42:0/2217941733 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6e14010900 con 0x7f6e1c09f830 2019-12-05T06:11:38.840 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:38.830+0000 7f6e21d6e700 1 --2- 172.21.15.42:0/2217941733 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6e1c09f830 0x7f6e1c14efe0 secure :-1 s=THROTTLE_DONE pgs=2216 cs=0 l=1 rx=0x7f6e14008b10 tx=0x7f6e1400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:38.840 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:38.830+0000 7f6e0ffff700 1 -- 172.21.15.42:0/2217941733 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e14008b50 con 0x7f6e1c09f830 2019-12-05T06:11:38.840 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:38.830+0000 7f6e0ffff700 1 -- 172.21.15.42:0/2217941733 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6e140110c0 con 0x7f6e1c09f830 2019-12-05T06:11:38.841 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:38.830+0000 7f6e22fd0700 1 -- 172.21.15.42:0/2217941733 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6e1c09f830 msgr2=0x7f6e1c14efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:38.841 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:38.830+0000 7f6e22fd0700 1 --2- 172.21.15.42:0/2217941733 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6e1c09f830 0x7f6e1c14efe0 secure :-1 s=READY pgs=2216 cs=0 l=1 rx=0x7f6e14008b10 tx=0x7f6e1400fff0).stop 2019-12-05T06:11:38.841 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:38.830+0000 7f6e22fd0700 1 -- 172.21.15.42:0/2217941733 shutdown_connections 2019-12-05T06:11:38.841 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:38.830+0000 7f6e22fd0700 1 --2- 172.21.15.42:0/2217941733 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6e1c09f830 0x7f6e1c14efe0 unknown :-1 s=CLOSED pgs=2216 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:11:38.841 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:38.830+0000 7f6e22fd0700 1 --2- 172.21.15.42:0/2217941733 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6e1c08df80 0x7f6e1c09f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:11:38.842 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:38.830+0000 7f6e22fd0700 1 -- 172.21.15.42:0/2217941733 shutdown_connections 2019-12-05T06:11:38.842 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:38.830+0000 7f6e22fd0700 1 -- 172.21.15.42:0/2217941733 wait complete. 2019-12-05T06:11:38.842 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:38.830+0000 7f6e22fd0700 1 -- 172.21.15.42:0/2217941733 >> 172.21.15.42:0/2217941733 conn(0x7f6e1c0aabe0 msgr2=0x7f6e1c0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:11:38.842 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:38.834+0000 7f6e22fd0700 1 Processor -- start 2019-12-05T06:11:38.842 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:38.834+0000 7f6e22fd0700 1 -- start start 2019-12-05T06:11:38.845 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:38.834+0000 7f6e22fd0700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f6e1c121a70 0x7f6e1c1185a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:11:38.845 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:38.834+0000 7f6e22fd0700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e1c117ba0 con 0x7f6e1c117040 2019-12-05T06:11:38.846 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:38.834+0000 7f6e22fd0700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e1c122460 con 0x7f6e1c122030 2019-12-05T06:11:38.846 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:38.834+0000 7f6e22fd0700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e1c11a490 con 0x7f6e1c121a70 2019-12-05T06:11:38.846 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:38.834+0000 7f6e20d6c700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f6e1c122030 0x7f6e1c0a6b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:50222/0 (socket says 172.21.15.42:50222) 2019-12-05T06:11:38.847 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:38.834+0000 7f6e20d6c700 1 -- 172.21.15.42:0/2662114002 learned_addr learned my addr 172.21.15.42:0/2662114002 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:11:38.847 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:38.834+0000 7f6e2156d700 1 --2- 172.21.15.42:0/2662114002 >> v2:172.21.15.40:3300/0 conn(0x7f6e1c121a70 0x7f6e1c1185a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:11:38.847 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:38.834+0000 7f6e0dffb700 1 -- 172.21.15.42:0/2662114002 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4253551155 0 0) 0x7f6e1c122460 con 0x7f6e1c122030 2019-12-05T06:11:38.847 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:38.834+0000 7f6e0dffb700 1 -- 172.21.15.42:0/2662114002 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6dfc0030b0 con 0x7f6e1c122030 2019-12-05T06:11:38.847 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:38.834+0000 7f6e0dffb700 1 -- 172.21.15.42:0/2662114002 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 1983053078 0 0) 0x7f6dfc0030b0 con 0x7f6e1c122030 2019-12-05T06:11:38.848 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:38.834+0000 7f6e2156d700 1 -- 172.21.15.42:0/2662114002 >> v1:172.21.15.42:6789/0 conn(0x7f6e1c122030 legacy=0x7f6e1c0a6b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:38.848 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:38.834+0000 7f6e2156d700 1 -- 172.21.15.42:0/2662114002 >> v1:172.21.15.40:6789/0 conn(0x7f6e1c117040 legacy=0x7f6e1c139c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:38.848 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:38.834+0000 7f6e2156d700 1 -- 172.21.15.42:0/2662114002 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e1c117470 con 0x7f6e1c121a70 2019-12-05T06:11:38.848 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:38.834+0000 7f6e2156d700 1 --2- 172.21.15.42:0/2662114002 >> v2:172.21.15.40:3300/0 conn(0x7f6e1c121a70 0x7f6e1c1185a0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f6e10001690 tx=0x7f6e10002f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:38.849 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:38.834+0000 7f6e2156d700 1 --2- 172.21.15.42:0/2662114002 >> v2:172.21.15.40:3300/0 conn(0x7f6e1c121a70 0x7f6e1c1185a0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f6e10001690 tx=0x7f6e10002f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:38.849 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:38.834+0000 7f6e2156d700 1 --2- 172.21.15.42:0/2662114002 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6e1c121a70 0x7f6e1c1185a0 secure :-1 s=READY pgs=1674 cs=0 l=1 rx=0x7f6e10001690 tx=0x7f6e10002f70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:11:38.849 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:38.834+0000 7f6e2156d700 1 --2- 172.21.15.42:0/2662114002 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6e1c121a70 0x7f6e1c1185a0 secure :-1 s=THROTTLE_DONE pgs=1674 cs=0 l=1 rx=0x7f6e10001690 tx=0x7f6e10002f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:38.849 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:38.834+0000 7f6e0dffb700 1 -- 172.21.15.42:0/2662114002 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6e10001490 con 0x7f6e1c121a70 2019-12-05T06:11:38.849 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:38.834+0000 7f6e2156d700 1 --2- 172.21.15.42:0/2662114002 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6e1c121a70 0x7f6e1c1185a0 secure :-1 s=THROTTLE_DONE pgs=1674 cs=0 l=1 rx=0x7f6e10001690 tx=0x7f6e10002f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:38.849 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:38.834+0000 7f6e2156d700 1 --2- 172.21.15.42:0/2662114002 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6e1c121a70 0x7f6e1c1185a0 secure :-1 s=THROTTLE_DONE pgs=1674 cs=0 l=1 rx=0x7f6e10001690 tx=0x7f6e10002f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:38.850 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:38.834+0000 7f6e0dffb700 1 -- 172.21.15.42:0/2662114002 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e100012a0 con 0x7f6e1c121a70 2019-12-05T06:11:38.850 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:38.838+0000 7f6e22fd0700 1 -- 172.21.15.42:0/2662114002 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e1c117740 con 0x7f6e1c121a70 2019-12-05T06:11:38.850 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:38.838+0000 7f6e0dffb700 1 -- 172.21.15.42:0/2662114002 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6e18011730 con 0x7f6e1c121a70 2019-12-05T06:11:38.850 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:38.838+0000 7f6e22fd0700 1 -- 172.21.15.42:0/2662114002 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e1c13b420 con 0x7f6e1c121a70 2019-12-05T06:11:38.850 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:38.838+0000 7f6e2156d700 1 --2- 172.21.15.42:0/2662114002 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6e1c121a70 0x7f6e1c1185a0 secure :-1 s=THROTTLE_DONE pgs=1674 cs=0 l=1 rx=0x7f6e10001690 tx=0x7f6e10002f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:38.850 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:38.838+0000 7f6e0dffb700 1 -- 172.21.15.42:0/2662114002 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f6e18011a00 con 0x7f6e1c121a70 2019-12-05T06:11:38.850 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:38.838+0000 7f6e2156d700 1 --2- 172.21.15.42:0/2662114002 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6e1c121a70 0x7f6e1c1185a0 secure :-1 s=THROTTLE_DONE pgs=1674 cs=0 l=1 rx=0x7f6e10001690 tx=0x7f6e10002f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:38.850 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:38.838+0000 7f6e0dffb700 1 --2- 172.21.15.42:0/2662114002 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6dfc03b210 0x7f6dfc03d720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:11:38.851 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:38.838+0000 7f6e0dffb700 1 -- 172.21.15.42:0/2662114002 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 10073+0+0 (secure 0 0 0) 0x7f6e1804ad60 con 0x7f6e1c121a70 2019-12-05T06:11:38.851 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:38.838+0000 7f6e20d6c700 1 --2- 172.21.15.42:0/2662114002 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6dfc03b210 0x7f6dfc03d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:11:38.851 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:38.838+0000 7f6e20d6c700 1 --2- 172.21.15.42:0/2662114002 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6dfc03b210 0x7f6dfc03d720 crc :-1 s=READY pgs=930 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:11:38.851 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:38.842+0000 7f6e22fd0700 1 -- 172.21.15.42:0/2662114002 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e1c05d110 con 0x7f6e1c121a70 2019-12-05T06:11:38.856 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:38.854+0000 7f6e2156d700 1 --2- 172.21.15.42:0/2662114002 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6e1c121a70 0x7f6e1c1185a0 secure :-1 s=THROTTLE_DONE pgs=1674 cs=0 l=1 rx=0x7f6e10001690 tx=0x7f6e10002f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:38.856 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:38.854+0000 7f6e0dffb700 1 -- 172.21.15.42:0/2662114002 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f6e18011de0 con 0x7f6e1c121a70 2019-12-05T06:11:39.026 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:39.022+0000 7f6e22fd0700 1 -- 172.21.15.42:0/2662114002 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6e1c05b8c0 con 0x7f6e1c121a70 2019-12-05T06:11:39.027 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:39.022+0000 7f6e2156d700 1 --2- 172.21.15.42:0/2662114002 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6e1c121a70 0x7f6e1c1185a0 secure :-1 s=THROTTLE_DONE pgs=1674 cs=0 l=1 rx=0x7f6e10001690 tx=0x7f6e10002f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:39.027 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:39.022+0000 7f6e0dffb700 1 -- 172.21.15.42:0/2662114002 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 267 v267) v1 ==== 95+0+1778 (secure 0 0 0) 0x7f6e18031de0 con 0x7f6e1c121a70 2019-12-05T06:11:39.027 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 267 2019-12-05T06:11:39.027 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:11:39.027 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":267,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":267,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:11:34.953298+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":96,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10000},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10000":{"gid":10000,"name":"a","rank":0,"incarnation":264,"state":"up:active","state_seq":21,"addr":"172.21.15.42:6835/3923117701","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3923117701},{"type":"v1","addr":"172.21.15.42:6835","nonce":3923117701}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:11:39.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:39.026+0000 7f6e22fd0700 1 -- 172.21.15.42:0/2662114002 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6dfc03b210 msgr2=0x7f6dfc03d720 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:39.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:39.026+0000 7f6e22fd0700 1 --2- 172.21.15.42:0/2662114002 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6dfc03b210 0x7f6dfc03d720 crc :-1 s=READY pgs=930 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:11:39.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:39.026+0000 7f6e22fd0700 1 -- 172.21.15.42:0/2662114002 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6e1c121a70 msgr2=0x7f6e1c1185a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:39.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:39.026+0000 7f6e22fd0700 1 --2- 172.21.15.42:0/2662114002 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6e1c121a70 0x7f6e1c1185a0 secure :-1 s=READY pgs=1674 cs=0 l=1 rx=0x7f6e10001690 tx=0x7f6e10002f70).stop 2019-12-05T06:11:39.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:39.026+0000 7f6e22fd0700 1 -- 172.21.15.42:0/2662114002 shutdown_connections 2019-12-05T06:11:39.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:39.026+0000 7f6e22fd0700 1 --2- 172.21.15.42:0/2662114002 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6dfc03b210 0x7f6dfc03d720 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:11:39.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:39.026+0000 7f6e22fd0700 1 --2- 172.21.15.42:0/2662114002 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6e1c121a70 0x7f6e1c1185a0 unknown :-1 s=CLOSED pgs=1674 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:11:39.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:39.026+0000 7f6e22fd0700 1 -- 172.21.15.42:0/2662114002 shutdown_connections 2019-12-05T06:11:39.032 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:39.026+0000 7f6e22fd0700 1 -- 172.21.15.42:0/2662114002 wait complete. 2019-12-05T06:11:39.032 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:39.026+0000 7f6e22fd0700 1 -- 172.21.15.42:0/2662114002 >> 172.21.15.42:0/2662114002 conn(0x7f6e1c141360 msgr2=0x7f6e1c0a97c0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:11:39.042 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2019-12-05T06:11:41.043 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:11:41.155 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:41.142+0000 7fe5b90d4700 1 Processor -- start 2019-12-05T06:11:41.156 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:41.142+0000 7fe5b90d4700 1 -- start start 2019-12-05T06:11:41.156 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:41.142+0000 7fe5b90d4700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe5b40a7740 0x7fe5b408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:11:41.156 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:41.142+0000 7fe5b90d4700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe5b407e720 0x7fe5b407eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:11:41.156 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:41.142+0000 7fe5b90d4700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5b40a0710 con 0x7fe5b409f2e0 2019-12-05T06:11:41.156 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:41.142+0000 7fe5b90d4700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5b40a0b00 con 0x7fe5b40a7740 2019-12-05T06:11:41.156 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:41.142+0000 7fe5b90d4700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5b408e5b0 con 0x7fe5b407e720 2019-12-05T06:11:41.157 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:41.142+0000 7fe5b359e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe5b40a7740 0x7fe5b408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:11:41.157 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:41.142+0000 7fe5b2d9d700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe5b407e720 0x7fe5b407eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:11:41.157 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:41.142+0000 7fe5b359e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe5b40a7740 0x7fe5b408df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:50200/0 (socket says 172.21.15.42:50200) 2019-12-05T06:11:41.157 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:41.142+0000 7fe5b2d9d700 1 --2- 172.21.15.42:0/237013598 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe5b407e720 0x7fe5b407eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:50668/0 (socket says 172.21.15.42:50668) 2019-12-05T06:11:41.158 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:41.142+0000 7fe5b359e700 1 -- 172.21.15.42:0/237013598 learned_addr learned my addr 172.21.15.42:0/237013598 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:11:41.158 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:41.142+0000 7fe5b2d9d700 1 -- 172.21.15.42:0/237013598 >> v2:172.21.15.42:3300/0 conn(0x7fe5b40a7740 msgr2=0x7fe5b408df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:11:41.158 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:41.142+0000 7fe5b2d9d700 1 --2- 172.21.15.42:0/237013598 >> v2:172.21.15.42:3300/0 conn(0x7fe5b40a7740 0x7fe5b408df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:11:41.158 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:41.142+0000 7fe5b2d9d700 1 -- 172.21.15.42:0/237013598 >> v1:172.21.15.40:6789/0 conn(0x7fe5b409f2e0 legacy=0x7fe5b409f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:41.158 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:41.142+0000 7fe5b2d9d700 1 -- 172.21.15.42:0/237013598 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5b407f0a0 con 0x7fe5b407e720 2019-12-05T06:11:41.158 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:41.142+0000 7fe5b359e700 1 --2- 172.21.15.42:0/237013598 >> v2:172.21.15.42:3300/0 conn(0x7fe5b40a7740 0x7fe5b408df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:11:41.159 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:41.142+0000 7fe5b2d9d700 1 --2- 172.21.15.42:0/237013598 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe5b407e720 0x7fe5b407eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fe59c008790 tx=0x7fe59c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:41.159 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:41.142+0000 7fe5b2d9d700 1 --2- 172.21.15.42:0/237013598 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe5b407e720 0x7fe5b407eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fe59c008790 tx=0x7fe59c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:41.159 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:41.142+0000 7fe5b2d9d700 1 --2- 172.21.15.42:0/237013598 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe5b407e720 0x7fe5b407eb50 secure :-1 s=READY pgs=484 cs=0 l=1 rx=0x7fe59c008790 tx=0x7fe59c00fc70).ready entity=mon.2 client_cookie=12e3496a44450d50 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:11:41.159 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:41.142+0000 7fe5b2d9d700 1 --2- 172.21.15.42:0/237013598 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe5b407e720 0x7fe5b407eb50 secure :-1 s=THROTTLE_DONE pgs=484 cs=0 l=1 rx=0x7fe59c008790 tx=0x7fe59c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:41.159 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:41.142+0000 7fe5b259c700 1 -- 172.21.15.42:0/237013598 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe59c0112f0 con 0x7fe5b407e720 2019-12-05T06:11:41.159 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:41.142+0000 7fe5b2d9d700 1 --2- 172.21.15.42:0/237013598 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe5b407e720 0x7fe5b407eb50 secure :-1 s=THROTTLE_DONE pgs=484 cs=0 l=1 rx=0x7fe59c008790 tx=0x7fe59c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:41.159 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:41.142+0000 7fe5b2d9d700 1 --2- 172.21.15.42:0/237013598 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe5b407e720 0x7fe5b407eb50 secure :-1 s=THROTTLE_DONE pgs=484 cs=0 l=1 rx=0x7fe59c008790 tx=0x7fe59c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:41.160 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:41.142+0000 7fe5b259c700 1 -- 172.21.15.42:0/237013598 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe59c008ad0 con 0x7fe5b407e720 2019-12-05T06:11:41.160 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:41.142+0000 7fe5b259c700 1 -- 172.21.15.42:0/237013598 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe59c0118d0 con 0x7fe5b407e720 2019-12-05T06:11:41.160 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:41.142+0000 7fe5b90d4700 1 -- 172.21.15.42:0/237013598 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe5b407e720 msgr2=0x7fe5b407eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:41.160 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:41.142+0000 7fe5b90d4700 1 --2- 172.21.15.42:0/237013598 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe5b407e720 0x7fe5b407eb50 secure :-1 s=READY pgs=484 cs=0 l=1 rx=0x7fe59c008790 tx=0x7fe59c00fc70).stop 2019-12-05T06:11:41.160 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:41.142+0000 7fe5b90d4700 1 -- 172.21.15.42:0/237013598 shutdown_connections 2019-12-05T06:11:41.160 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:41.142+0000 7fe5b90d4700 1 --2- 172.21.15.42:0/237013598 >> v2:172.21.15.42:3300/0 conn(0x7fe5b40a7740 0x7fe5b408df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:11:41.160 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:41.146+0000 7fe5b90d4700 1 --2- 172.21.15.42:0/237013598 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe5b407e720 0x7fe5b407eb50 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:11:41.161 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:41.146+0000 7fe5b90d4700 1 -- 172.21.15.42:0/237013598 shutdown_connections 2019-12-05T06:11:41.161 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:41.146+0000 7fe5b90d4700 1 -- 172.21.15.42:0/237013598 wait complete. 2019-12-05T06:11:41.161 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:41.146+0000 7fe5b90d4700 1 -- 172.21.15.42:0/237013598 >> 172.21.15.42:0/237013598 conn(0x7fe5b40aabe0 msgr2=0x7fe5b40a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:11:41.161 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:41.146+0000 7fe5b90d4700 1 Processor -- start 2019-12-05T06:11:41.161 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:41.146+0000 7fe5b90d4700 1 -- start start 2019-12-05T06:11:41.161 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:41.146+0000 7fe5b90d4700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe5b4116480 0x7fe5b411bd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:11:41.161 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:41.146+0000 7fe5b90d4700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe5b411a310 0x7fe5b411f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:11:41.162 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:41.146+0000 7fe5b90d4700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5b411a740 con 0x7fe5b41225f0 2019-12-05T06:11:41.162 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:41.146+0000 7fe5b90d4700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5b41168b0 con 0x7fe5b4116480 2019-12-05T06:11:41.162 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:41.146+0000 7fe5b90d4700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5b4116b40 con 0x7fe5b411a310 2019-12-05T06:11:41.162 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:41.146+0000 7fe5b2d9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe5b4116480 0x7fe5b411bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:11:41.162 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:41.146+0000 7fe5b359e700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fe5b41225f0 0x7fe5b411ecf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:50244/0 (socket says 172.21.15.42:50244) 2019-12-05T06:11:41.162 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:41.146+0000 7fe5b2d9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe5b4116480 0x7fe5b411bd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:50208/0 (socket says 172.21.15.42:50208) 2019-12-05T06:11:41.162 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:41.146+0000 7fe5b359e700 1 -- 172.21.15.42:0/699626050 learned_addr learned my addr 172.21.15.42:0/699626050 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:11:41.163 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:41.146+0000 7fe5b88d3700 1 --2- 172.21.15.42:0/699626050 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe5b411a310 0x7fe5b411f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:11:41.163 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:41.146+0000 7fe5b2d9d700 1 -- 172.21.15.42:0/699626050 >> v2:172.21.15.42:3300/0 conn(0x7fe5b4116480 msgr2=0x7fe5b411bd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2019-12-05T06:11:41.163 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:41.146+0000 7fe5b2d9d700 1 -- 172.21.15.42:0/699626050 >> v2:172.21.15.42:3300/0 conn(0x7fe5b4116480 msgr2=0x7fe5b411bd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2019-12-05T06:11:41.163 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:41.146+0000 7fe5b2d9d700 1 --2- 172.21.15.42:0/699626050 >> v2:172.21.15.42:3300/0 conn(0x7fe5b4116480 0x7fe5b411bd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-12-05T06:11:41.163 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:41.146+0000 7fe5b2d9d700 1 --2- 172.21.15.42:0/699626050 >> v2:172.21.15.42:3300/0 conn(0x7fe5b4116480 0x7fe5b411bd30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rx=0 tx=0)._fault waiting 0.200000 2019-12-05T06:11:41.163 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:41.146+0000 7fe5abfff700 1 -- 172.21.15.42:0/699626050 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3600999435 0 0) 0x7fe5b411a740 con 0x7fe5b41225f0 2019-12-05T06:11:41.164 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:41.146+0000 7fe5abfff700 1 -- 172.21.15.42:0/699626050 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe58c0030b0 con 0x7fe5b41225f0 2019-12-05T06:11:41.164 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:41.150+0000 7fe5b88d3700 1 -- 172.21.15.42:0/699626050 >> v2:172.21.15.42:3300/0 conn(0x7fe5b4116480 msgr2=0x7fe5b411bd30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2019-12-05T06:11:41.164 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:41.150+0000 7fe5b88d3700 1 --2- 172.21.15.42:0/699626050 >> v2:172.21.15.42:3300/0 conn(0x7fe5b4116480 0x7fe5b411bd30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:11:41.164 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:41.150+0000 7fe5b88d3700 1 -- 172.21.15.42:0/699626050 >> v1:172.21.15.42:6789/0 conn(0x7fe5b41225f0 legacy=0x7fe5b411ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:41.166 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:41.150+0000 7fe5b88d3700 1 -- 172.21.15.42:0/699626050 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5b411fa40 con 0x7fe5b411a310 2019-12-05T06:11:41.166 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:41.150+0000 7fe5b88d3700 1 --2- 172.21.15.42:0/699626050 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe5b411a310 0x7fe5b411f4f0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fe5a4001210 tx=0x7fe5a4010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:41.166 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:41.150+0000 7fe5b88d3700 1 --2- 172.21.15.42:0/699626050 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe5b411a310 0x7fe5b411f4f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fe5a4001210 tx=0x7fe5a4010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:41.166 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:41.150+0000 7fe5b88d3700 1 --2- 172.21.15.42:0/699626050 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe5b411a310 0x7fe5b411f4f0 secure :-1 s=READY pgs=485 cs=0 l=1 rx=0x7fe5a4001210 tx=0x7fe5a4010b40).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:11:41.166 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:41.150+0000 7fe5b88d3700 1 --2- 172.21.15.42:0/699626050 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe5b411a310 0x7fe5b411f4f0 secure :-1 s=THROTTLE_DONE pgs=485 cs=0 l=1 rx=0x7fe5a4001210 tx=0x7fe5a4010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:41.167 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:41.150+0000 7fe5abfff700 1 -- 172.21.15.42:0/699626050 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe5a4006dc0 con 0x7fe5b411a310 2019-12-05T06:11:41.167 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:41.150+0000 7fe5b88d3700 1 --2- 172.21.15.42:0/699626050 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe5b411a310 0x7fe5b411f4f0 secure :-1 s=THROTTLE_DONE pgs=485 cs=0 l=1 rx=0x7fe5a4001210 tx=0x7fe5a4010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:41.167 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:41.150+0000 7fe5abfff700 1 -- 172.21.15.42:0/699626050 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5a40069e0 con 0x7fe5b411a310 2019-12-05T06:11:41.167 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:41.150+0000 7fe5b88d3700 1 --2- 172.21.15.42:0/699626050 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe5b411a310 0x7fe5b411f4f0 secure :-1 s=THROTTLE_DONE pgs=485 cs=0 l=1 rx=0x7fe5a4001210 tx=0x7fe5a4010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:41.167 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:41.150+0000 7fe5b90d4700 1 -- 172.21.15.42:0/699626050 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5b40a6f80 con 0x7fe5b411a310 2019-12-05T06:11:41.168 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:41.150+0000 7fe5abfff700 1 -- 172.21.15.42:0/699626050 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe5a4002620 con 0x7fe5b411a310 2019-12-05T06:11:41.168 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:41.150+0000 7fe5b90d4700 1 -- 172.21.15.42:0/699626050 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5b4077300 con 0x7fe5b411a310 2019-12-05T06:11:41.168 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:41.150+0000 7fe5b88d3700 1 --2- 172.21.15.42:0/699626050 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe5b411a310 0x7fe5b411f4f0 secure :-1 s=THROTTLE_DONE pgs=485 cs=0 l=1 rx=0x7fe5a4001210 tx=0x7fe5a4010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:41.168 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:41.150+0000 7fe5abfff700 1 -- 172.21.15.42:0/699626050 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fe5a4002940 con 0x7fe5b411a310 2019-12-05T06:11:41.168 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:41.150+0000 7fe5b88d3700 1 --2- 172.21.15.42:0/699626050 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe5b411a310 0x7fe5b411f4f0 secure :-1 s=THROTTLE_DONE pgs=485 cs=0 l=1 rx=0x7fe5a4001210 tx=0x7fe5a4010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:41.168 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:41.150+0000 7fe5abfff700 1 --2- 172.21.15.42:0/699626050 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe58c03b270 0x7fe58c03d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:11:41.169 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:41.150+0000 7fe5abfff700 1 -- 172.21.15.42:0/699626050 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 10073+0+0 (secure 0 0 0) 0x7fe5a404b510 con 0x7fe5b411a310 2019-12-05T06:11:41.169 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:41.150+0000 7fe5b359e700 1 --2- 172.21.15.42:0/699626050 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe58c03b270 0x7fe58c03d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:11:41.169 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:41.150+0000 7fe5b359e700 1 --2- 172.21.15.42:0/699626050 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe58c03b270 0x7fe58c03d780 crc :-1 s=READY pgs=932 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:11:41.170 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:41.158+0000 7fe5b90d4700 1 -- 172.21.15.42:0/699626050 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5b405d110 con 0x7fe5b411a310 2019-12-05T06:11:41.171 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:41.166+0000 7fe5b88d3700 1 --2- 172.21.15.42:0/699626050 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe5b411a310 0x7fe5b411f4f0 secure :-1 s=THROTTLE_DONE pgs=485 cs=0 l=1 rx=0x7fe5a4001210 tx=0x7fe5a4010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:41.171 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:41.166+0000 7fe5abfff700 1 -- 172.21.15.42:0/699626050 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fe5a4032570 con 0x7fe5b411a310 2019-12-05T06:11:41.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:41.334+0000 7fe5b90d4700 1 -- 172.21.15.42:0/699626050 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe5b40a71f0 con 0x7fe5b411a310 2019-12-05T06:11:41.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:41.334+0000 7fe5b88d3700 1 --2- 172.21.15.42:0/699626050 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe5b411a310 0x7fe5b411f4f0 secure :-1 s=THROTTLE_DONE pgs=485 cs=0 l=1 rx=0x7fe5a4001210 tx=0x7fe5a4010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:41.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:41.334+0000 7fe5abfff700 1 -- 172.21.15.42:0/699626050 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 268 v268) v1 ==== 95+0+2115 (secure 0 0 0) 0x7fe5a40327d0 con 0x7fe5b411a310 2019-12-05T06:11:41.341 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:11:41.342 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":268,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7094,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/3785523496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3785523496},{"type":"v1","addr":"172.21.15.40:6833","nonce":3785523496}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":268}],"filesystems":[{"mdsmap":{"epoch":267,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:11:34.953298+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":96,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10000},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10000":{"gid":10000,"name":"a","rank":0,"incarnation":264,"state":"up:active","state_seq":21,"addr":"172.21.15.42:6835/3923117701","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3923117701},{"type":"v1","addr":"172.21.15.42:6835","nonce":3923117701}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:11:41.342 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 268 2019-12-05T06:11:41.344 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:41.338+0000 7fe5b90d4700 1 -- 172.21.15.42:0/699626050 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe58c03b270 msgr2=0x7fe58c03d780 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:41.345 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:41.338+0000 7fe5b90d4700 1 --2- 172.21.15.42:0/699626050 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe58c03b270 0x7fe58c03d780 crc :-1 s=READY pgs=932 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:11:41.345 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:41.338+0000 7fe5b90d4700 1 -- 172.21.15.42:0/699626050 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe5b411a310 msgr2=0x7fe5b411f4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:41.345 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:41.338+0000 7fe5b90d4700 1 --2- 172.21.15.42:0/699626050 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe5b411a310 0x7fe5b411f4f0 secure :-1 s=READY pgs=485 cs=0 l=1 rx=0x7fe5a4001210 tx=0x7fe5a4010b40).stop 2019-12-05T06:11:41.345 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:41.342+0000 7fe5b90d4700 1 -- 172.21.15.42:0/699626050 shutdown_connections 2019-12-05T06:11:41.346 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:41.342+0000 7fe5b90d4700 1 --2- 172.21.15.42:0/699626050 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe58c03b270 0x7fe58c03d780 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:11:41.346 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:41.342+0000 7fe5b90d4700 1 --2- 172.21.15.42:0/699626050 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe5b411a310 0x7fe5b411f4f0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:11:41.346 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:41.342+0000 7fe5b90d4700 1 --2- 172.21.15.42:0/699626050 >> v2:172.21.15.42:3300/0 conn(0x7fe5b4116480 0x7fe5b411bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:11:41.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:41.342+0000 7fe5b90d4700 1 -- 172.21.15.42:0/699626050 shutdown_connections 2019-12-05T06:11:41.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:41.342+0000 7fe5b90d4700 1 -- 172.21.15.42:0/699626050 wait complete. 2019-12-05T06:11:41.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:41.342+0000 7fe5b90d4700 1 -- 172.21.15.42:0/699626050 >> 172.21.15.42:0/699626050 conn(0x7fe5b40aabe0 msgr2=0x7fe5b40a96f0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:11:41.354 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b reported in up:standby state 2019-12-05T06:11:41.354 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 13 secs before thrashing 2019-12-05T06:11:44.339 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:11:44.343 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:11:54.355 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:11:54.471 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.458+0000 7f9ebc5b5700 1 Processor -- start 2019-12-05T06:11:54.471 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.458+0000 7f9ebc5b5700 1 -- start start 2019-12-05T06:11:54.471 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.458+0000 7f9ebc5b5700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9eb40a7740 0x7f9eb408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:11:54.471 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.458+0000 7f9ebc5b5700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f9eb407e720 0x7f9eb407eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:11:54.472 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.458+0000 7f9ebc5b5700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9eb40a0710 con 0x7f9eb409f2e0 2019-12-05T06:11:54.472 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.458+0000 7f9ebc5b5700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9eb40a0b00 con 0x7f9eb407e720 2019-12-05T06:11:54.472 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.458+0000 7f9ebc5b5700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9eb408e5b0 con 0x7f9eb40a7740 2019-12-05T06:11:54.472 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.458+0000 7f9ebab52700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9eb40a7740 0x7f9eb408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:11:54.472 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.458+0000 7f9eba351700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f9eb407e720 0x7f9eb407eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:11:54.472 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.458+0000 7f9ebab52700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9eb40a7740 0x7f9eb408df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:50714/0 (socket says 172.21.15.42:50714) 2019-12-05T06:11:54.472 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.458+0000 7f9ebab52700 1 -- 172.21.15.42:0/2980812125 learned_addr learned my addr 172.21.15.42:0/2980812125 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:11:54.473 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.458+0000 7f9ebab52700 1 -- 172.21.15.42:0/2980812125 >> v2:172.21.15.40:3300/0 conn(0x7f9eb407e720 msgr2=0x7f9eb407eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:11:54.473 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.458+0000 7f9ebab52700 1 --2- 172.21.15.42:0/2980812125 >> v2:172.21.15.40:3300/0 conn(0x7f9eb407e720 0x7f9eb407eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:11:54.473 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.458+0000 7f9ebab52700 1 -- 172.21.15.42:0/2980812125 >> v1:172.21.15.40:6789/0 conn(0x7f9eb409f2e0 legacy=0x7f9eb409f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:54.473 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.458+0000 7f9ebab52700 1 -- 172.21.15.42:0/2980812125 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9eb407f0a0 con 0x7f9eb40a7740 2019-12-05T06:11:54.473 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.458+0000 7f9ebab52700 1 --2- 172.21.15.42:0/2980812125 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9eb40a7740 0x7f9eb408df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f9eb0008790 tx=0x7f9eb000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:54.473 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.458+0000 7f9ebab52700 1 --2- 172.21.15.42:0/2980812125 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9eb40a7740 0x7f9eb408df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f9eb0008790 tx=0x7f9eb000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:54.473 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.458+0000 7f9ebab52700 1 --2- 172.21.15.42:0/2980812125 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9eb40a7740 0x7f9eb408df80 secure :-1 s=READY pgs=486 cs=0 l=1 rx=0x7f9eb0008790 tx=0x7f9eb000fd30).ready entity=mon.2 client_cookie=256f54aa4a8220d6 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:11:54.474 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.458+0000 7f9ebab52700 1 --2- 172.21.15.42:0/2980812125 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9eb40a7740 0x7f9eb408df80 secure :-1 s=THROTTLE_DONE pgs=486 cs=0 l=1 rx=0x7f9eb0008790 tx=0x7f9eb000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:54.474 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.458+0000 7f9eb9b50700 1 -- 172.21.15.42:0/2980812125 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9eb00113b0 con 0x7f9eb40a7740 2019-12-05T06:11:54.474 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.458+0000 7f9ebab52700 1 --2- 172.21.15.42:0/2980812125 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9eb40a7740 0x7f9eb408df80 secure :-1 s=THROTTLE_DONE pgs=486 cs=0 l=1 rx=0x7f9eb0008790 tx=0x7f9eb000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:54.474 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.458+0000 7f9eb9b50700 1 -- 172.21.15.42:0/2980812125 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9eb0008bb0 con 0x7f9eb40a7740 2019-12-05T06:11:54.474 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.458+0000 7f9ebab52700 1 --2- 172.21.15.42:0/2980812125 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9eb40a7740 0x7f9eb408df80 secure :-1 s=THROTTLE_DONE pgs=486 cs=0 l=1 rx=0x7f9eb0008790 tx=0x7f9eb000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:54.474 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.458+0000 7f9eb9b50700 1 -- 172.21.15.42:0/2980812125 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9eb0011990 con 0x7f9eb40a7740 2019-12-05T06:11:54.474 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.458+0000 7f9ebc5b5700 1 -- 172.21.15.42:0/2980812125 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9eb40a7740 msgr2=0x7f9eb408df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:54.475 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.458+0000 7f9ebc5b5700 1 --2- 172.21.15.42:0/2980812125 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9eb40a7740 0x7f9eb408df80 secure :-1 s=READY pgs=486 cs=0 l=1 rx=0x7f9eb0008790 tx=0x7f9eb000fd30).stop 2019-12-05T06:11:54.475 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.462+0000 7f9ebc5b5700 1 -- 172.21.15.42:0/2980812125 shutdown_connections 2019-12-05T06:11:54.475 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.462+0000 7f9ebc5b5700 1 --2- 172.21.15.42:0/2980812125 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9eb40a7740 0x7f9eb408df80 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:11:54.475 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.462+0000 7f9ebc5b5700 1 --2- 172.21.15.42:0/2980812125 >> v2:172.21.15.40:3300/0 conn(0x7f9eb407e720 0x7f9eb407eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:11:54.475 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.462+0000 7f9ebc5b5700 1 -- 172.21.15.42:0/2980812125 shutdown_connections 2019-12-05T06:11:54.475 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.462+0000 7f9ebc5b5700 1 -- 172.21.15.42:0/2980812125 wait complete. 2019-12-05T06:11:54.476 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.462+0000 7f9ebc5b5700 1 -- 172.21.15.42:0/2980812125 >> 172.21.15.42:0/2980812125 conn(0x7f9eb40aabe0 msgr2=0x7f9eb40a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:11:54.476 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.462+0000 7f9ebc5b5700 1 Processor -- start 2019-12-05T06:11:54.476 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.462+0000 7f9ebc5b5700 1 -- start start 2019-12-05T06:11:54.476 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.462+0000 7f9ebc5b5700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f9eb40a5990 0x7f9eb411c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:11:54.476 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.462+0000 7f9ebc5b5700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9eb407b750 con 0x7f9eb41317a0 2019-12-05T06:11:54.476 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.462+0000 7f9ebc5b5700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9eb40a4a00 con 0x7f9eb4124b20 2019-12-05T06:11:54.477 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.462+0000 7f9ebc5b5700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9eb40a5fe0 con 0x7f9eb40a5990 2019-12-05T06:11:54.477 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.466+0000 7f9eba351700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f9eb40a5990 0x7f9eb411c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:11:54.477 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.466+0000 7f9eba351700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f9eb40a5990 0x7f9eb411c290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:50256/0 (socket says 172.21.15.42:50256) 2019-12-05T06:11:54.477 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.466+0000 7f9eba351700 1 -- 172.21.15.42:0/1293377902 learned_addr learned my addr 172.21.15.42:0/1293377902 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:11:54.477 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.466+0000 7f9eba351700 1 -- 172.21.15.42:0/1293377902 >> v1:172.21.15.42:6789/0 conn(0x7f9eb4124b20 legacy=0x7f9eb407af50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:54.477 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.466+0000 7f9eba351700 1 -- 172.21.15.42:0/1293377902 >> v1:172.21.15.40:6789/0 conn(0x7f9eb41317a0 legacy=0x7f9eb4131bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:54.477 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.466+0000 7f9eab7fe700 1 -- 172.21.15.42:0/1293377902 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2517977977 0 0) 0x7f9eb40a4a00 con 0x7f9eb4124b20 2019-12-05T06:11:54.478 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.466+0000 7f9eba351700 1 -- 172.21.15.42:0/1293377902 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9eb413bc20 con 0x7f9eb40a5990 2019-12-05T06:11:54.478 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.466+0000 7f9eba351700 1 --2- 172.21.15.42:0/1293377902 >> v2:172.21.15.42:3300/0 conn(0x7f9eb40a5990 0x7f9eb411c290 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f9ea4003070 tx=0x7f9ea4010bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:54.478 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.466+0000 7f9eba351700 1 --2- 172.21.15.42:0/1293377902 >> v2:172.21.15.42:3300/0 conn(0x7f9eb40a5990 0x7f9eb411c290 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f9ea4003070 tx=0x7f9ea4010bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:54.478 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.466+0000 7f9eba351700 1 --2- 172.21.15.42:0/1293377902 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9eb40a5990 0x7f9eb411c290 secure :-1 s=READY pgs=2223 cs=0 l=1 rx=0x7f9ea4003070 tx=0x7f9ea4010bc0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:11:54.478 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.466+0000 7f9eba351700 1 --2- 172.21.15.42:0/1293377902 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9eb40a5990 0x7f9eb411c290 secure :-1 s=THROTTLE_DONE pgs=2223 cs=0 l=1 rx=0x7f9ea4003070 tx=0x7f9ea4010bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:54.478 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.466+0000 7f9eab7fe700 1 -- 172.21.15.42:0/1293377902 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9ea4005930 con 0x7f9eb40a5990 2019-12-05T06:11:54.478 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.466+0000 7f9ebc5b5700 1 -- 172.21.15.42:0/1293377902 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9eb4139c00 con 0x7f9eb40a5990 2019-12-05T06:11:54.479 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.466+0000 7f9ebc5b5700 1 -- 172.21.15.42:0/1293377902 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9eb4139e70 con 0x7f9eb40a5990 2019-12-05T06:11:54.479 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.466+0000 7f9eba351700 1 --2- 172.21.15.42:0/1293377902 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9eb40a5990 0x7f9eb411c290 secure :-1 s=THROTTLE_DONE pgs=2223 cs=0 l=1 rx=0x7f9ea4003070 tx=0x7f9ea4010bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:54.479 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.466+0000 7f9eba351700 1 --2- 172.21.15.42:0/1293377902 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9eb40a5990 0x7f9eb411c290 secure :-1 s=THROTTLE_DONE pgs=2223 cs=0 l=1 rx=0x7f9ea4003070 tx=0x7f9ea4010bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:54.479 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.466+0000 7f9eab7fe700 1 -- 172.21.15.42:0/1293377902 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ea4007160 con 0x7f9eb40a5990 2019-12-05T06:11:54.479 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.466+0000 7f9eab7fe700 1 -- 172.21.15.42:0/1293377902 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9ea4012240 con 0x7f9eb40a5990 2019-12-05T06:11:54.479 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.466+0000 7f9eba351700 1 --2- 172.21.15.42:0/1293377902 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9eb40a5990 0x7f9eb411c290 secure :-1 s=THROTTLE_DONE pgs=2223 cs=0 l=1 rx=0x7f9ea4003070 tx=0x7f9ea4010bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:54.479 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.466+0000 7f9eab7fe700 1 -- 172.21.15.42:0/1293377902 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f9ea4012560 con 0x7f9eb40a5990 2019-12-05T06:11:54.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.466+0000 7f9eba351700 1 --2- 172.21.15.42:0/1293377902 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9eb40a5990 0x7f9eb411c290 secure :-1 s=THROTTLE_DONE pgs=2223 cs=0 l=1 rx=0x7f9ea4003070 tx=0x7f9ea4010bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:54.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.466+0000 7f9eab7fe700 1 --2- 172.21.15.42:0/1293377902 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9e9c03abc0 0x7f9e9c03d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:11:54.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.466+0000 7f9eab7fe700 1 -- 172.21.15.42:0/1293377902 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 10073+0+0 (secure 0 0 0) 0x7f9ea404b380 con 0x7f9eb40a5990 2019-12-05T06:11:54.482 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.466+0000 7f9ebab52700 1 --2- 172.21.15.42:0/1293377902 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9e9c03abc0 0x7f9e9c03d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:11:54.482 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.470+0000 7f9ebab52700 1 --2- 172.21.15.42:0/1293377902 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9e9c03abc0 0x7f9e9c03d0d0 crc :-1 s=READY pgs=934 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:11:54.482 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.474+0000 7f9ebc5b5700 1 -- 172.21.15.42:0/1293377902 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e90003200 con 0x7f9eb40a5990 2019-12-05T06:11:54.483 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.478+0000 7f9eba351700 1 --2- 172.21.15.42:0/1293377902 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9eb40a5990 0x7f9eb411c290 secure :-1 s=THROTTLE_DONE pgs=2223 cs=0 l=1 rx=0x7f9ea4003070 tx=0x7f9ea4010bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:54.483 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.478+0000 7f9eab7fe700 1 -- 172.21.15.42:0/1293377902 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f9ea40323e0 con 0x7f9eb40a5990 2019-12-05T06:11:54.679 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.674+0000 7f9ebc5b5700 1 -- 172.21.15.42:0/1293377902 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9e90003630 con 0x7f9eb40a5990 2019-12-05T06:11:54.680 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.674+0000 7f9eba351700 1 --2- 172.21.15.42:0/1293377902 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9eb40a5990 0x7f9eb411c290 secure :-1 s=THROTTLE_DONE pgs=2223 cs=0 l=1 rx=0x7f9ea4003070 tx=0x7f9ea4010bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:54.680 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.674+0000 7f9eab7fe700 1 -- 172.21.15.42:0/1293377902 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 268 v268) v1 ==== 95+0+2115 (secure 0 0 0) 0x7f9ea4032640 con 0x7f9eb40a5990 2019-12-05T06:11:54.681 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 268 2019-12-05T06:11:54.681 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:11:54.681 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":268,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7094,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/3785523496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3785523496},{"type":"v1","addr":"172.21.15.40:6833","nonce":3785523496}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":268}],"filesystems":[{"mdsmap":{"epoch":267,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:11:34.953298+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":96,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10000},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10000":{"gid":10000,"name":"a","rank":0,"incarnation":264,"state":"up:active","state_seq":21,"addr":"172.21.15.42:6835/3923117701","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3923117701},{"type":"v1","addr":"172.21.15.42:6835","nonce":3923117701}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:11:54.683 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.678+0000 7f9ebc5b5700 1 -- 172.21.15.42:0/1293377902 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9e9c03abc0 msgr2=0x7f9e9c03d0d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:54.683 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.678+0000 7f9ebc5b5700 1 --2- 172.21.15.42:0/1293377902 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9e9c03abc0 0x7f9e9c03d0d0 crc :-1 s=READY pgs=934 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:11:54.683 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.678+0000 7f9ebc5b5700 1 -- 172.21.15.42:0/1293377902 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9eb40a5990 msgr2=0x7f9eb411c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:54.683 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.678+0000 7f9ebc5b5700 1 --2- 172.21.15.42:0/1293377902 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9eb40a5990 0x7f9eb411c290 secure :-1 s=READY pgs=2223 cs=0 l=1 rx=0x7f9ea4003070 tx=0x7f9ea4010bc0).stop 2019-12-05T06:11:54.683 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.678+0000 7f9ebc5b5700 1 -- 172.21.15.42:0/1293377902 shutdown_connections 2019-12-05T06:11:54.683 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.678+0000 7f9ebc5b5700 1 --2- 172.21.15.42:0/1293377902 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9e9c03abc0 0x7f9e9c03d0d0 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:11:54.684 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.678+0000 7f9ebc5b5700 1 --2- 172.21.15.42:0/1293377902 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9eb40a5990 0x7f9eb411c290 unknown :-1 s=CLOSED pgs=2223 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:11:54.684 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.678+0000 7f9ebc5b5700 1 -- 172.21.15.42:0/1293377902 shutdown_connections 2019-12-05T06:11:54.684 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.678+0000 7f9ebc5b5700 1 -- 172.21.15.42:0/1293377902 wait complete. 2019-12-05T06:11:54.684 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.678+0000 7f9ebc5b5700 1 -- 172.21.15.42:0/1293377902 >> 172.21.15.42:0/1293377902 conn(0x7f9eb411cb80 msgr2=0x7f9eb40a9790 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:11:54.693 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.a (rank=0) 2019-12-05T06:11:54.693 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2019-12-05T06:11:54.694 INFO:teuthology.orchestra.run:waiting for 300 2019-12-05T06:11:54.704 INFO:tasks.ceph.mds.a:Stopped 2019-12-05T06:11:54.704 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:11:54.867 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.858+0000 7fb6b8552700 1 Processor -- start 2019-12-05T06:11:54.867 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.858+0000 7fb6b8552700 1 -- start start 2019-12-05T06:11:54.868 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.858+0000 7fb6b8552700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb6b008df80 0x7fb6b009f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:11:54.868 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.858+0000 7fb6b8552700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb6b009f830 0x7fb6b014efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:11:54.868 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.858+0000 7fb6b8552700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6b009fc60 con 0x7fb6b00a7740 2019-12-05T06:11:54.868 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.858+0000 7fb6b8552700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6b00a0b00 con 0x7fb6b009f830 2019-12-05T06:11:54.868 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.858+0000 7fb6b8552700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6b008e5b0 con 0x7fb6b008df80 2019-12-05T06:11:54.869 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.858+0000 7fb6b62ee700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb6b008df80 0x7fb6b009f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:11:54.869 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.858+0000 7fb6b62ee700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb6b008df80 0x7fb6b009f2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:50266/0 (socket says 172.21.15.42:50266) 2019-12-05T06:11:54.869 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.858+0000 7fb6b62ee700 1 -- 172.21.15.42:0/3366159079 learned_addr learned my addr 172.21.15.42:0/3366159079 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:11:54.869 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.858+0000 7fb6b72f0700 1 --2- 172.21.15.42:0/3366159079 >> v2:172.21.15.40:3300/0 conn(0x7fb6b009f830 0x7fb6b014efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:11:54.869 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.862+0000 7fb6b62ee700 1 -- 172.21.15.42:0/3366159079 >> v2:172.21.15.40:3300/0 conn(0x7fb6b009f830 msgr2=0x7fb6b014efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:11:54.870 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.862+0000 7fb6b62ee700 1 --2- 172.21.15.42:0/3366159079 >> v2:172.21.15.40:3300/0 conn(0x7fb6b009f830 0x7fb6b014efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:11:54.870 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.862+0000 7fb6b62ee700 1 -- 172.21.15.42:0/3366159079 >> v1:172.21.15.40:6789/0 conn(0x7fb6b00a7740 legacy=0x7fb6b007e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:54.870 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.862+0000 7fb6b62ee700 1 -- 172.21.15.42:0/3366159079 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6b00a0710 con 0x7fb6b008df80 2019-12-05T06:11:54.870 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.862+0000 7fb6b62ee700 1 --2- 172.21.15.42:0/3366159079 >> v2:172.21.15.42:3300/0 conn(0x7fb6b008df80 0x7fb6b009f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fb6a0002f30 tx=0x7fb6a000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:54.870 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.862+0000 7fb6b62ee700 1 --2- 172.21.15.42:0/3366159079 >> v2:172.21.15.42:3300/0 conn(0x7fb6b008df80 0x7fb6b009f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fb6a0002f30 tx=0x7fb6a000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:54.870 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.862+0000 7fb6b62ee700 1 --2- 172.21.15.42:0/3366159079 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb6b008df80 0x7fb6b009f2e0 secure :-1 s=READY pgs=2224 cs=0 l=1 rx=0x7fb6a0002f30 tx=0x7fb6a000fd30).ready entity=mon.1 client_cookie=aa07ed0db49b2a00 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:11:54.871 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.862+0000 7fb6b62ee700 1 --2- 172.21.15.42:0/3366159079 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb6b008df80 0x7fb6b009f2e0 secure :-1 s=THROTTLE_DONE pgs=2224 cs=0 l=1 rx=0x7fb6a0002f30 tx=0x7fb6a000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:54.871 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.862+0000 7fb6b5aed700 1 -- 172.21.15.42:0/3366159079 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb6a0008c90 con 0x7fb6b008df80 2019-12-05T06:11:54.871 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.862+0000 7fb6b62ee700 1 --2- 172.21.15.42:0/3366159079 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb6b008df80 0x7fb6b009f2e0 secure :-1 s=THROTTLE_DONE pgs=2224 cs=0 l=1 rx=0x7fb6a0002f30 tx=0x7fb6a000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:54.871 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.862+0000 7fb6b62ee700 1 --2- 172.21.15.42:0/3366159079 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb6b008df80 0x7fb6b009f2e0 secure :-1 s=THROTTLE_DONE pgs=2224 cs=0 l=1 rx=0x7fb6a0002f30 tx=0x7fb6a000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:54.871 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.862+0000 7fb6b5aed700 1 -- 172.21.15.42:0/3366159079 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6a00088b0 con 0x7fb6b008df80 2019-12-05T06:11:54.871 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.862+0000 7fb6b5aed700 1 -- 172.21.15.42:0/3366159079 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb6a0011ad0 con 0x7fb6b008df80 2019-12-05T06:11:54.872 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.862+0000 7fb6b8552700 1 -- 172.21.15.42:0/3366159079 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb6b008df80 msgr2=0x7fb6b009f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:54.872 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.862+0000 7fb6b8552700 1 --2- 172.21.15.42:0/3366159079 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb6b008df80 0x7fb6b009f2e0 secure :-1 s=READY pgs=2224 cs=0 l=1 rx=0x7fb6a0002f30 tx=0x7fb6a000fd30).stop 2019-12-05T06:11:54.872 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.862+0000 7fb6b8552700 1 -- 172.21.15.42:0/3366159079 shutdown_connections 2019-12-05T06:11:54.872 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.862+0000 7fb6b8552700 1 --2- 172.21.15.42:0/3366159079 >> v2:172.21.15.40:3300/0 conn(0x7fb6b009f830 0x7fb6b014efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:11:54.872 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.862+0000 7fb6b8552700 1 --2- 172.21.15.42:0/3366159079 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb6b008df80 0x7fb6b009f2e0 unknown :-1 s=CLOSED pgs=2224 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:11:54.872 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.862+0000 7fb6b8552700 1 -- 172.21.15.42:0/3366159079 shutdown_connections 2019-12-05T06:11:54.873 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.862+0000 7fb6b8552700 1 -- 172.21.15.42:0/3366159079 wait complete. 2019-12-05T06:11:54.873 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.862+0000 7fb6b8552700 1 -- 172.21.15.42:0/3366159079 >> 172.21.15.42:0/3366159079 conn(0x7fb6b00aabe0 msgr2=0x7fb6b00a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:11:54.876 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.866+0000 7fb6b8552700 1 Processor -- start 2019-12-05T06:11:54.876 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.866+0000 7fb6b8552700 1 -- start start 2019-12-05T06:11:54.876 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.866+0000 7fb6b8552700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb6b0116500 0x7fb6b011b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:11:54.876 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.866+0000 7fb6b8552700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6b011be40 con 0x7fb6b011aac0 2019-12-05T06:11:54.876 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.866+0000 7fb6b8552700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6b0118ec0 con 0x7fb6b011a690 2019-12-05T06:11:54.877 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.866+0000 7fb6b8552700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6b007c510 con 0x7fb6b0116500 2019-12-05T06:11:54.877 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.866+0000 7fb6b62ee700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fb6b011a690 0x7fb6b008eb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:50312/0 (socket says 172.21.15.42:50312) 2019-12-05T06:11:54.877 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.866+0000 7fb6b6aef700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb6b0116500 0x7fb6b011b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:11:54.877 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.866+0000 7fb6b62ee700 1 -- 172.21.15.42:0/2487417543 learned_addr learned my addr 172.21.15.42:0/2487417543 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:11:54.877 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.866+0000 7fb6b6aef700 1 -- 172.21.15.42:0/2487417543 >> v1:172.21.15.42:6789/0 conn(0x7fb6b011a690 legacy=0x7fb6b008eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:54.877 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.866+0000 7fb6b6aef700 1 -- 172.21.15.42:0/2487417543 >> v1:172.21.15.40:6789/0 conn(0x7fb6b011aac0 legacy=0x7fb6b008f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:54.877 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.866+0000 7fb6a77fe700 1 -- 172.21.15.42:0/2487417543 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 963561259 0 0) 0x7fb6b011be40 con 0x7fb6b011aac0 2019-12-05T06:11:54.878 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.866+0000 7fb6b6aef700 1 -- 172.21.15.42:0/2487417543 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6b007ccf0 con 0x7fb6b0116500 2019-12-05T06:11:54.878 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.866+0000 7fb6b6aef700 1 --2- 172.21.15.42:0/2487417543 >> v2:172.21.15.42:3300/0 conn(0x7fb6b0116500 0x7fb6b011b8f0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fb6ac001430 tx=0x7fb6ac00c320).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:54.878 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.866+0000 7fb6b6aef700 1 --2- 172.21.15.42:0/2487417543 >> v2:172.21.15.42:3300/0 conn(0x7fb6b0116500 0x7fb6b011b8f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fb6ac001430 tx=0x7fb6ac00c320).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:54.878 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.866+0000 7fb6b6aef700 1 --2- 172.21.15.42:0/2487417543 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb6b0116500 0x7fb6b011b8f0 secure :-1 s=READY pgs=2226 cs=0 l=1 rx=0x7fb6ac001430 tx=0x7fb6ac00c320).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:11:54.878 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.866+0000 7fb6b6aef700 1 --2- 172.21.15.42:0/2487417543 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb6b0116500 0x7fb6b011b8f0 secure :-1 s=THROTTLE_DONE pgs=2226 cs=0 l=1 rx=0x7fb6ac001430 tx=0x7fb6ac00c320).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:54.878 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.866+0000 7fb6a77fe700 1 -- 172.21.15.42:0/2487417543 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb6ac00cb70 con 0x7fb6b0116500 2019-12-05T06:11:54.878 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.866+0000 7fb6b6aef700 1 --2- 172.21.15.42:0/2487417543 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb6b0116500 0x7fb6b011b8f0 secure :-1 s=THROTTLE_DONE pgs=2226 cs=0 l=1 rx=0x7fb6ac001430 tx=0x7fb6ac00c320).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:54.879 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.866+0000 7fb6a77fe700 1 -- 172.21.15.42:0/2487417543 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6ac00c790 con 0x7fb6b0116500 2019-12-05T06:11:54.879 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.866+0000 7fb6b6aef700 1 --2- 172.21.15.42:0/2487417543 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb6b0116500 0x7fb6b011b8f0 secure :-1 s=THROTTLE_DONE pgs=2226 cs=0 l=1 rx=0x7fb6ac001430 tx=0x7fb6ac00c320).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:54.879 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.866+0000 7fb6b8552700 1 -- 172.21.15.42:0/2487417543 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6b012e6a0 con 0x7fb6b0116500 2019-12-05T06:11:54.879 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.866+0000 7fb6a77fe700 1 -- 172.21.15.42:0/2487417543 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb6ac011070 con 0x7fb6b0116500 2019-12-05T06:11:54.879 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.866+0000 7fb6b8552700 1 -- 172.21.15.42:0/2487417543 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6b012ebd0 con 0x7fb6b0116500 2019-12-05T06:11:54.879 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.866+0000 7fb6b6aef700 1 --2- 172.21.15.42:0/2487417543 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb6b0116500 0x7fb6b011b8f0 secure :-1 s=THROTTLE_DONE pgs=2226 cs=0 l=1 rx=0x7fb6ac001430 tx=0x7fb6ac00c320).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:54.879 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.870+0000 7fb6a77fe700 1 -- 172.21.15.42:0/2487417543 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fb6ac0113d0 con 0x7fb6b0116500 2019-12-05T06:11:54.880 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.870+0000 7fb6b6aef700 1 --2- 172.21.15.42:0/2487417543 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb6b0116500 0x7fb6b011b8f0 secure :-1 s=THROTTLE_DONE pgs=2226 cs=0 l=1 rx=0x7fb6ac001430 tx=0x7fb6ac00c320).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:54.880 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.870+0000 7fb6a77fe700 1 --2- 172.21.15.42:0/2487417543 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb69803ab90 0x7fb69803d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:11:54.880 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.870+0000 7fb6a77fe700 1 -- 172.21.15.42:0/2487417543 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 10073+0+0 (secure 0 0 0) 0x7fb6ac04b2f0 con 0x7fb6b0116500 2019-12-05T06:11:54.880 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.870+0000 7fb6b62ee700 1 --2- 172.21.15.42:0/2487417543 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb69803ab90 0x7fb69803d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:11:54.880 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.870+0000 7fb6b62ee700 1 --2- 172.21.15.42:0/2487417543 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb69803ab90 0x7fb69803d0a0 crc :-1 s=READY pgs=935 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:11:54.881 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.874+0000 7fb6b8552700 1 -- 172.21.15.42:0/2487417543 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6b005d110 con 0x7fb6b0116500 2019-12-05T06:11:54.882 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.878+0000 7fb6b6aef700 1 --2- 172.21.15.42:0/2487417543 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb6b0116500 0x7fb6b011b8f0 secure :-1 s=THROTTLE_DONE pgs=2226 cs=0 l=1 rx=0x7fb6ac001430 tx=0x7fb6ac00c320).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:54.882 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:54.878+0000 7fb6a77fe700 1 -- 172.21.15.42:0/2487417543 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fb6ac032350 con 0x7fb6b0116500 2019-12-05T06:11:55.045 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:55.042+0000 7fb6b8552700 1 -- 172.21.15.42:0/2487417543 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb6b005b8c0 con 0x7fb6b0116500 2019-12-05T06:11:55.046 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:55.042+0000 7fb6b6aef700 1 --2- 172.21.15.42:0/2487417543 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb6b0116500 0x7fb6b011b8f0 secure :-1 s=THROTTLE_DONE pgs=2226 cs=0 l=1 rx=0x7fb6ac001430 tx=0x7fb6ac00c320).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:55.047 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:55.042+0000 7fb6a77fe700 1 -- 172.21.15.42:0/2487417543 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 268 v268) v1 ==== 95+0+2115 (secure 0 0 0) 0x7fb6ac0325b0 con 0x7fb6b0116500 2019-12-05T06:11:55.047 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 268 2019-12-05T06:11:55.047 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:11:55.047 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":268,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7094,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/3785523496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3785523496},{"type":"v1","addr":"172.21.15.40:6833","nonce":3785523496}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":268}],"filesystems":[{"mdsmap":{"epoch":267,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:11:34.953298+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":96,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10000},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10000":{"gid":10000,"name":"a","rank":0,"incarnation":264,"state":"up:active","state_seq":21,"addr":"172.21.15.42:6835/3923117701","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3923117701},{"type":"v1","addr":"172.21.15.42:6835","nonce":3923117701}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:11:55.049 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:55.046+0000 7fb6b8552700 1 -- 172.21.15.42:0/2487417543 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb69803ab90 msgr2=0x7fb69803d0a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:55.049 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:55.046+0000 7fb6b8552700 1 --2- 172.21.15.42:0/2487417543 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb69803ab90 0x7fb69803d0a0 crc :-1 s=READY pgs=935 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:11:55.049 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:55.046+0000 7fb6b8552700 1 -- 172.21.15.42:0/2487417543 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb6b0116500 msgr2=0x7fb6b011b8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:55.050 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:55.046+0000 7fb6b8552700 1 --2- 172.21.15.42:0/2487417543 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb6b0116500 0x7fb6b011b8f0 secure :-1 s=READY pgs=2226 cs=0 l=1 rx=0x7fb6ac001430 tx=0x7fb6ac00c320).stop 2019-12-05T06:11:55.050 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:55.046+0000 7fb6b8552700 1 -- 172.21.15.42:0/2487417543 shutdown_connections 2019-12-05T06:11:55.050 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:55.046+0000 7fb6b8552700 1 --2- 172.21.15.42:0/2487417543 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb69803ab90 0x7fb69803d0a0 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:11:55.050 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:55.046+0000 7fb6b8552700 1 --2- 172.21.15.42:0/2487417543 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb6b0116500 0x7fb6b011b8f0 unknown :-1 s=CLOSED pgs=2226 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:11:55.050 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:55.046+0000 7fb6b8552700 1 -- 172.21.15.42:0/2487417543 shutdown_connections 2019-12-05T06:11:55.050 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:55.046+0000 7fb6b8552700 1 -- 172.21.15.42:0/2487417543 wait complete. 2019-12-05T06:11:55.051 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:55.046+0000 7fb6b8552700 1 -- 172.21.15.42:0/2487417543 >> 172.21.15.42:0/2487417543 conn(0x7fb6b00a97e0 msgr2=0x7fb6b0059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:11:55.059 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:11:57.060 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:11:57.169 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:57.158+0000 7f75f963f700 1 Processor -- start 2019-12-05T06:11:57.170 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:57.158+0000 7f75f963f700 1 -- start start 2019-12-05T06:11:57.170 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:57.158+0000 7f75f963f700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f75f409fae0 0x7f75f414fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:11:57.170 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:57.158+0000 7f75f963f700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75f40a0710 con 0x7f75f408df80 2019-12-05T06:11:57.170 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:57.158+0000 7f75f963f700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75f407ef20 con 0x7f75f40a7740 2019-12-05T06:11:57.170 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:57.158+0000 7f75f963f700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75f40a0b00 con 0x7f75f409fae0 2019-12-05T06:11:57.170 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:57.158+0000 7f75f37fe700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f75f40a7740 0x7f75f407e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:50326/0 (socket says 172.21.15.42:50326) 2019-12-05T06:11:57.171 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:57.158+0000 7f75f37fe700 1 -- 172.21.15.42:0/1054726085 learned_addr learned my addr 172.21.15.42:0/1054726085 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:11:57.171 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:57.158+0000 7f75f3fff700 1 --2- 172.21.15.42:0/1054726085 >> v2:172.21.15.40:3300/0 conn(0x7f75f409fae0 0x7f75f414fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:11:57.171 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:57.162+0000 7f75f2ffd700 1 -- 172.21.15.42:0/1054726085 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2444513902 0 0) 0x7f75f407ef20 con 0x7f75f40a7740 2019-12-05T06:11:57.171 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:57.162+0000 7f75f2ffd700 1 -- 172.21.15.42:0/1054726085 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75e00030b0 con 0x7f75f40a7740 2019-12-05T06:11:57.171 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:57.162+0000 7f75f3fff700 1 -- 172.21.15.42:0/1054726085 >> v1:172.21.15.42:6789/0 conn(0x7f75f40a7740 legacy=0x7f75f407e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:57.171 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:57.162+0000 7f75f3fff700 1 -- 172.21.15.42:0/1054726085 >> v1:172.21.15.40:6789/0 conn(0x7f75f408df80 legacy=0x7f75f409f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:57.171 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:57.162+0000 7f75f3fff700 1 -- 172.21.15.42:0/1054726085 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75f40a9ba0 con 0x7f75f409fae0 2019-12-05T06:11:57.172 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:57.162+0000 7f75f3fff700 1 --2- 172.21.15.42:0/1054726085 >> v2:172.21.15.40:3300/0 conn(0x7f75f409fae0 0x7f75f414fff0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f75ec008b10 tx=0x7f75ec00ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:57.172 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:57.162+0000 7f75f3fff700 1 --2- 172.21.15.42:0/1054726085 >> v2:172.21.15.40:3300/0 conn(0x7f75f409fae0 0x7f75f414fff0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f75ec008b10 tx=0x7f75ec00ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:57.172 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:57.162+0000 7f75f3fff700 1 --2- 172.21.15.42:0/1054726085 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f75f409fae0 0x7f75f414fff0 secure :-1 s=READY pgs=1685 cs=0 l=1 rx=0x7f75ec008b10 tx=0x7f75ec00ff70).ready entity=mon.0 client_cookie=57d7546e1474127f server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:11:57.172 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:57.162+0000 7f75f3fff700 1 --2- 172.21.15.42:0/1054726085 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f75f409fae0 0x7f75f414fff0 secure :-1 s=THROTTLE_DONE pgs=1685 cs=0 l=1 rx=0x7f75ec008b10 tx=0x7f75ec00ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:57.172 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:57.162+0000 7f75f2ffd700 1 -- 172.21.15.42:0/1054726085 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f75ec010970 con 0x7f75f409fae0 2019-12-05T06:11:57.172 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:57.162+0000 7f75f3fff700 1 --2- 172.21.15.42:0/1054726085 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f75f409fae0 0x7f75f414fff0 secure :-1 s=THROTTLE_DONE pgs=1685 cs=0 l=1 rx=0x7f75ec008b10 tx=0x7f75ec00ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:57.172 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:57.162+0000 7f75f2ffd700 1 -- 172.21.15.42:0/1054726085 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75ec008b50 con 0x7f75f409fae0 2019-12-05T06:11:57.173 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:57.162+0000 7f75f3fff700 1 --2- 172.21.15.42:0/1054726085 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f75f409fae0 0x7f75f414fff0 secure :-1 s=THROTTLE_DONE pgs=1685 cs=0 l=1 rx=0x7f75ec008b10 tx=0x7f75ec00ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:57.173 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:57.162+0000 7f75f2ffd700 1 -- 172.21.15.42:0/1054726085 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f75ec0110c0 con 0x7f75f409fae0 2019-12-05T06:11:57.173 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:57.162+0000 7f75f963f700 1 -- 172.21.15.42:0/1054726085 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f75f409fae0 msgr2=0x7f75f414fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:57.173 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:57.162+0000 7f75f963f700 1 --2- 172.21.15.42:0/1054726085 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f75f409fae0 0x7f75f414fff0 secure :-1 s=READY pgs=1685 cs=0 l=1 rx=0x7f75ec008b10 tx=0x7f75ec00ff70).stop 2019-12-05T06:11:57.173 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:57.162+0000 7f75f963f700 1 -- 172.21.15.42:0/1054726085 shutdown_connections 2019-12-05T06:11:57.173 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:57.162+0000 7f75f963f700 1 --2- 172.21.15.42:0/1054726085 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f75f409fae0 0x7f75f414fff0 unknown :-1 s=CLOSED pgs=1685 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:11:57.173 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:57.162+0000 7f75f963f700 1 -- 172.21.15.42:0/1054726085 shutdown_connections 2019-12-05T06:11:57.174 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:57.162+0000 7f75f963f700 1 -- 172.21.15.42:0/1054726085 wait complete. 2019-12-05T06:11:57.174 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:57.162+0000 7f75f963f700 1 -- 172.21.15.42:0/1054726085 >> 172.21.15.42:0/1054726085 conn(0x7f75f40aabe0 msgr2=0x7f75f40a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:11:57.174 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:57.166+0000 7f75f963f700 1 Processor -- start 2019-12-05T06:11:57.174 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:57.166+0000 7f75f963f700 1 -- start start 2019-12-05T06:11:57.177 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:57.166+0000 7f75f963f700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f75f4059f50 0x7f75f4145860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:11:57.177 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:57.166+0000 7f75f963f700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75f411c260 con 0x7f75f412a630 2019-12-05T06:11:57.177 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:57.166+0000 7f75f963f700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75f412aa60 con 0x7f75f412a200 2019-12-05T06:11:57.177 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:57.166+0000 7f75f963f700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75f40a5510 con 0x7f75f4059f50 2019-12-05T06:11:57.178 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:57.166+0000 7f75eaffd700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f75f412a200 0x7f75f411b5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:50332/0 (socket says 172.21.15.42:50332) 2019-12-05T06:11:57.178 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:57.166+0000 7f75eaffd700 1 -- 172.21.15.42:0/1514956289 learned_addr learned my addr 172.21.15.42:0/1514956289 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:11:57.178 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:57.166+0000 7f75f37fe700 1 --2- 172.21.15.42:0/1514956289 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f75f4059f50 0x7f75f4145860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:11:57.178 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:57.170+0000 7f75f0ff9700 1 -- 172.21.15.42:0/1514956289 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3320508022 0 0) 0x7f75f412aa60 con 0x7f75f412a200 2019-12-05T06:11:57.178 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:57.170+0000 7f75f0ff9700 1 -- 172.21.15.42:0/1514956289 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75d80030b0 con 0x7f75f412a200 2019-12-05T06:11:57.178 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:57.170+0000 7f75f0ff9700 1 -- 172.21.15.42:0/1514956289 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 795319501 0 0) 0x7f75f411c260 con 0x7f75f412a630 2019-12-05T06:11:57.179 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:57.170+0000 7f75f0ff9700 1 -- 172.21.15.42:0/1514956289 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75f412aa60 con 0x7f75f412a630 2019-12-05T06:11:57.179 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:57.170+0000 7f75f37fe700 1 -- 172.21.15.42:0/1514956289 >> v1:172.21.15.42:6789/0 conn(0x7f75f412a200 legacy=0x7f75f411b5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:57.179 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:57.170+0000 7f75f37fe700 1 -- 172.21.15.42:0/1514956289 >> v1:172.21.15.40:6789/0 conn(0x7f75f412a630 legacy=0x7f75f4137b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:57.179 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:57.170+0000 7f75f37fe700 1 -- 172.21.15.42:0/1514956289 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75f4145db0 con 0x7f75f4059f50 2019-12-05T06:11:57.179 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:57.170+0000 7f75f37fe700 1 --2- 172.21.15.42:0/1514956289 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f75f4059f50 0x7f75f4145860 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f75e4000b80 tx=0x7f75e4010810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:57.179 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:57.170+0000 7f75f37fe700 1 --2- 172.21.15.42:0/1514956289 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f75f4059f50 0x7f75f4145860 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f75e4000b80 tx=0x7f75e4010810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:57.179 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:57.170+0000 7f75f37fe700 1 --2- 172.21.15.42:0/1514956289 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f75f4059f50 0x7f75f4145860 secure :-1 s=READY pgs=487 cs=0 l=1 rx=0x7f75e4000b80 tx=0x7f75e4010810).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:11:57.179 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:57.170+0000 7f75f37fe700 1 --2- 172.21.15.42:0/1514956289 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f75f4059f50 0x7f75f4145860 secure :-1 s=THROTTLE_DONE pgs=487 cs=0 l=1 rx=0x7f75e4000b80 tx=0x7f75e4010810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:57.180 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:57.170+0000 7f75f0ff9700 1 -- 172.21.15.42:0/1514956289 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f75e4005c30 con 0x7f75f4059f50 2019-12-05T06:11:57.180 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:57.170+0000 7f75f37fe700 1 --2- 172.21.15.42:0/1514956289 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f75f4059f50 0x7f75f4145860 secure :-1 s=THROTTLE_DONE pgs=487 cs=0 l=1 rx=0x7f75e4000b80 tx=0x7f75e4010810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:57.180 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:57.170+0000 7f75f0ff9700 1 -- 172.21.15.42:0/1514956289 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75e4005850 con 0x7f75f4059f50 2019-12-05T06:11:57.180 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:57.170+0000 7f75f37fe700 1 --2- 172.21.15.42:0/1514956289 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f75f4059f50 0x7f75f4145860 secure :-1 s=THROTTLE_DONE pgs=487 cs=0 l=1 rx=0x7f75e4000b80 tx=0x7f75e4010810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:57.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:57.170+0000 7f75f0ff9700 1 -- 172.21.15.42:0/1514956289 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f75e40064a0 con 0x7f75f4059f50 2019-12-05T06:11:57.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:57.170+0000 7f75f963f700 1 -- 172.21.15.42:0/1514956289 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75f40a6320 con 0x7f75f4059f50 2019-12-05T06:11:57.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:57.170+0000 7f75f963f700 1 -- 172.21.15.42:0/1514956289 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75f411bda0 con 0x7f75f4059f50 2019-12-05T06:11:57.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:57.170+0000 7f75f37fe700 1 --2- 172.21.15.42:0/1514956289 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f75f4059f50 0x7f75f4145860 secure :-1 s=THROTTLE_DONE pgs=487 cs=0 l=1 rx=0x7f75e4000b80 tx=0x7f75e4010810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:57.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:57.174+0000 7f75f0ff9700 1 -- 172.21.15.42:0/1514956289 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f75e4006780 con 0x7f75f4059f50 2019-12-05T06:11:57.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:57.174+0000 7f75f37fe700 1 --2- 172.21.15.42:0/1514956289 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f75f4059f50 0x7f75f4145860 secure :-1 s=THROTTLE_DONE pgs=487 cs=0 l=1 rx=0x7f75e4000b80 tx=0x7f75e4010810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:57.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:57.174+0000 7f75f0ff9700 1 --2- 172.21.15.42:0/1514956289 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f75d803b470 0x7f75d803d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:11:57.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:57.174+0000 7f75f0ff9700 1 -- 172.21.15.42:0/1514956289 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 10073+0+0 (secure 0 0 0) 0x7f75e404aee0 con 0x7f75f4059f50 2019-12-05T06:11:57.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:57.174+0000 7f75eaffd700 1 --2- 172.21.15.42:0/1514956289 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f75d803b470 0x7f75d803d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:11:57.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:57.174+0000 7f75eaffd700 1 --2- 172.21.15.42:0/1514956289 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f75d803b470 0x7f75d803d980 crc :-1 s=READY pgs=936 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:11:57.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:57.174+0000 7f75f963f700 1 -- 172.21.15.42:0/1514956289 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75f405d110 con 0x7f75f4059f50 2019-12-05T06:11:57.186 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:57.182+0000 7f75f37fe700 1 --2- 172.21.15.42:0/1514956289 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f75f4059f50 0x7f75f4145860 secure :-1 s=THROTTLE_DONE pgs=487 cs=0 l=1 rx=0x7f75e4000b80 tx=0x7f75e4010810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:57.186 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:57.182+0000 7f75f0ff9700 1 -- 172.21.15.42:0/1514956289 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f75e4031f60 con 0x7f75f4059f50 2019-12-05T06:11:57.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:57.354+0000 7f75f963f700 1 -- 172.21.15.42:0/1514956289 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f75f405b8c0 con 0x7f75f4059f50 2019-12-05T06:11:57.358 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:57.354+0000 7f75f37fe700 1 --2- 172.21.15.42:0/1514956289 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f75f4059f50 0x7f75f4145860 secure :-1 s=THROTTLE_DONE pgs=487 cs=0 l=1 rx=0x7f75e4000b80 tx=0x7f75e4010810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:57.359 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:57.354+0000 7f75f0ff9700 1 -- 172.21.15.42:0/1514956289 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 268 v268) v1 ==== 95+0+2115 (secure 0 0 0) 0x7f75e40321c0 con 0x7f75f4059f50 2019-12-05T06:11:57.361 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 268 2019-12-05T06:11:57.361 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:11:57.361 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":268,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7094,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/3785523496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3785523496},{"type":"v1","addr":"172.21.15.40:6833","nonce":3785523496}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":268}],"filesystems":[{"mdsmap":{"epoch":267,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:11:34.953298+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":96,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10000},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10000":{"gid":10000,"name":"a","rank":0,"incarnation":264,"state":"up:active","state_seq":21,"addr":"172.21.15.42:6835/3923117701","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3923117701},{"type":"v1","addr":"172.21.15.42:6835","nonce":3923117701}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:11:57.363 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:57.358+0000 7f75f963f700 1 -- 172.21.15.42:0/1514956289 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f75d803b470 msgr2=0x7f75d803d980 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:57.363 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:57.358+0000 7f75f963f700 1 --2- 172.21.15.42:0/1514956289 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f75d803b470 0x7f75d803d980 crc :-1 s=READY pgs=936 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:11:57.363 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:57.358+0000 7f75f963f700 1 -- 172.21.15.42:0/1514956289 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f75f4059f50 msgr2=0x7f75f4145860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:57.363 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:57.358+0000 7f75f963f700 1 --2- 172.21.15.42:0/1514956289 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f75f4059f50 0x7f75f4145860 secure :-1 s=READY pgs=487 cs=0 l=1 rx=0x7f75e4000b80 tx=0x7f75e4010810).stop 2019-12-05T06:11:57.364 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:57.358+0000 7f75f963f700 1 -- 172.21.15.42:0/1514956289 shutdown_connections 2019-12-05T06:11:57.364 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:57.358+0000 7f75f963f700 1 --2- 172.21.15.42:0/1514956289 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f75d803b470 0x7f75d803d980 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:11:57.364 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:57.358+0000 7f75f963f700 1 --2- 172.21.15.42:0/1514956289 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f75f4059f50 0x7f75f4145860 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:11:57.364 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:57.358+0000 7f75f963f700 1 -- 172.21.15.42:0/1514956289 shutdown_connections 2019-12-05T06:11:57.365 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:57.358+0000 7f75f963f700 1 -- 172.21.15.42:0/1514956289 wait complete. 2019-12-05T06:11:57.365 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:57.358+0000 7f75f963f700 1 -- 172.21.15.42:0/1514956289 >> 172.21.15.42:0/1514956289 conn(0x7f75f40aabe0 msgr2=0x7f75f411cbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:11:57.374 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:11:59.377 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:11:59.486 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:59.478+0000 7f9d941a3700 1 Processor -- start 2019-12-05T06:11:59.487 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:59.478+0000 7f9d941a3700 1 -- start start 2019-12-05T06:11:59.487 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:59.478+0000 7f9d941a3700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f9d8c0a7740 0x7f9d8c08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:11:59.487 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:59.478+0000 7f9d941a3700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f9d8c07e720 0x7f9d8c07eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:11:59.488 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:59.478+0000 7f9d941a3700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d8c0a0710 con 0x7f9d8c09f2e0 2019-12-05T06:11:59.488 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:59.478+0000 7f9d941a3700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d8c0a0b00 con 0x7f9d8c07e720 2019-12-05T06:11:59.488 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:59.478+0000 7f9d941a3700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d8c08e5b0 con 0x7f9d8c0a7740 2019-12-05T06:11:59.488 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:59.478+0000 7f9d92740700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f9d8c0a7740 0x7f9d8c08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:11:59.488 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:59.478+0000 7f9d92740700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f9d8c0a7740 0x7f9d8c08df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:50308/0 (socket says 172.21.15.42:50308) 2019-12-05T06:11:59.488 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:59.478+0000 7f9d92740700 1 -- 172.21.15.42:0/3212547688 learned_addr learned my addr 172.21.15.42:0/3212547688 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:11:59.489 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:59.478+0000 7f9d91f3f700 1 --2- 172.21.15.42:0/3212547688 >> v2:172.21.15.40:3300/0 conn(0x7f9d8c07e720 0x7f9d8c07eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:11:59.489 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:59.478+0000 7f9d92740700 1 -- 172.21.15.42:0/3212547688 >> v2:172.21.15.40:3300/0 conn(0x7f9d8c07e720 msgr2=0x7f9d8c07eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:11:59.489 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:59.478+0000 7f9d92740700 1 --2- 172.21.15.42:0/3212547688 >> v2:172.21.15.40:3300/0 conn(0x7f9d8c07e720 0x7f9d8c07eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:11:59.489 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:59.478+0000 7f9d92740700 1 -- 172.21.15.42:0/3212547688 >> v1:172.21.15.40:6789/0 conn(0x7f9d8c09f2e0 legacy=0x7f9d8c09f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:59.489 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:59.478+0000 7f9d92740700 1 -- 172.21.15.42:0/3212547688 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d8c07f0a0 con 0x7f9d8c0a7740 2019-12-05T06:11:59.489 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:59.478+0000 7f9d91f3f700 1 --2- 172.21.15.42:0/3212547688 >> v2:172.21.15.40:3300/0 conn(0x7f9d8c07e720 0x7f9d8c07eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:11:59.489 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:59.478+0000 7f9d92740700 1 --2- 172.21.15.42:0/3212547688 >> v2:172.21.15.42:3300/0 conn(0x7f9d8c0a7740 0x7f9d8c08df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f9d88002f30 tx=0x7f9d8800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:59.490 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:59.482+0000 7f9d92740700 1 --2- 172.21.15.42:0/3212547688 >> v2:172.21.15.42:3300/0 conn(0x7f9d8c0a7740 0x7f9d8c08df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f9d88002f30 tx=0x7f9d8800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:59.490 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:59.482+0000 7f9d92740700 1 --2- 172.21.15.42:0/3212547688 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9d8c0a7740 0x7f9d8c08df80 secure :-1 s=READY pgs=2229 cs=0 l=1 rx=0x7f9d88002f30 tx=0x7f9d8800fd30).ready entity=mon.1 client_cookie=990e4baf8edf17c5 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:11:59.490 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:59.482+0000 7f9d92740700 1 --2- 172.21.15.42:0/3212547688 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9d8c0a7740 0x7f9d8c08df80 secure :-1 s=THROTTLE_DONE pgs=2229 cs=0 l=1 rx=0x7f9d88002f30 tx=0x7f9d8800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:59.490 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:59.482+0000 7f9d92740700 1 --2- 172.21.15.42:0/3212547688 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9d8c0a7740 0x7f9d8c08df80 secure :-1 s=THROTTLE_DONE pgs=2229 cs=0 l=1 rx=0x7f9d88002f30 tx=0x7f9d8800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:59.490 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:59.482+0000 7f9d9173e700 1 -- 172.21.15.42:0/3212547688 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9d88011370 con 0x7f9d8c0a7740 2019-12-05T06:11:59.490 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:59.482+0000 7f9d92740700 1 --2- 172.21.15.42:0/3212547688 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9d8c0a7740 0x7f9d8c08df80 secure :-1 s=THROTTLE_DONE pgs=2229 cs=0 l=1 rx=0x7f9d88002f30 tx=0x7f9d8800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:59.491 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:59.482+0000 7f9d9173e700 1 -- 172.21.15.42:0/3212547688 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d88008bb0 con 0x7f9d8c0a7740 2019-12-05T06:11:59.491 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:59.482+0000 7f9d9173e700 1 -- 172.21.15.42:0/3212547688 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9d88011950 con 0x7f9d8c0a7740 2019-12-05T06:11:59.491 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:59.482+0000 7f9d941a3700 1 -- 172.21.15.42:0/3212547688 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9d8c0a7740 msgr2=0x7f9d8c08df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:59.491 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:59.482+0000 7f9d941a3700 1 --2- 172.21.15.42:0/3212547688 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9d8c0a7740 0x7f9d8c08df80 secure :-1 s=READY pgs=2229 cs=0 l=1 rx=0x7f9d88002f30 tx=0x7f9d8800fd30).stop 2019-12-05T06:11:59.491 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:59.482+0000 7f9d941a3700 1 -- 172.21.15.42:0/3212547688 shutdown_connections 2019-12-05T06:11:59.491 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:59.482+0000 7f9d941a3700 1 --2- 172.21.15.42:0/3212547688 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9d8c0a7740 0x7f9d8c08df80 unknown :-1 s=CLOSED pgs=2229 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:11:59.491 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:59.482+0000 7f9d941a3700 1 --2- 172.21.15.42:0/3212547688 >> v2:172.21.15.40:3300/0 conn(0x7f9d8c07e720 0x7f9d8c07eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:11:59.492 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:59.482+0000 7f9d941a3700 1 -- 172.21.15.42:0/3212547688 shutdown_connections 2019-12-05T06:11:59.492 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:59.482+0000 7f9d941a3700 1 -- 172.21.15.42:0/3212547688 wait complete. 2019-12-05T06:11:59.492 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:59.482+0000 7f9d941a3700 1 -- 172.21.15.42:0/3212547688 >> 172.21.15.42:0/3212547688 conn(0x7f9d8c0aabe0 msgr2=0x7f9d8c0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:11:59.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:59.482+0000 7f9d941a3700 1 Processor -- start 2019-12-05T06:11:59.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:59.482+0000 7f9d941a3700 1 -- start start 2019-12-05T06:11:59.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:59.486+0000 7f9d941a3700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f9d8c0a6690 0x7f9d8c064960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:11:59.496 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:59.486+0000 7f9d941a3700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f9d8c0634d0 0x7f9d8c0a5290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:11:59.496 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:59.486+0000 7f9d941a3700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d8c0a5c10 con 0x7f9d8c0a57e0 2019-12-05T06:11:59.496 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:59.486+0000 7f9d941a3700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d8c064eb0 con 0x7f9d8c0634d0 2019-12-05T06:11:59.496 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:59.486+0000 7f9d941a3700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d8c0a6bb0 con 0x7f9d8c0a6690 2019-12-05T06:11:59.496 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:59.486+0000 7f9d92740700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f9d8c0a6690 0x7f9d8c064960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:11:59.496 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:59.486+0000 7f9d92740700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f9d8c0a6690 0x7f9d8c064960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:50314/0 (socket says 172.21.15.42:50314) 2019-12-05T06:11:59.500 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:59.486+0000 7f9d92740700 1 -- 172.21.15.42:0/1221728848 learned_addr learned my addr 172.21.15.42:0/1221728848 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:11:59.500 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:59.486+0000 7f9d91f3f700 1 --2- 172.21.15.42:0/1221728848 >> v2:172.21.15.40:3300/0 conn(0x7f9d8c0634d0 0x7f9d8c0a5290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:11:59.500 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:59.486+0000 7f9d92740700 1 -- 172.21.15.42:0/1221728848 >> v2:172.21.15.40:3300/0 conn(0x7f9d8c0634d0 msgr2=0x7f9d8c0a5290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:59.500 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:59.486+0000 7f9d92740700 1 --2- 172.21.15.42:0/1221728848 >> v2:172.21.15.40:3300/0 conn(0x7f9d8c0634d0 0x7f9d8c0a5290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:11:59.500 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:59.486+0000 7f9d92740700 1 -- 172.21.15.42:0/1221728848 >> v1:172.21.15.40:6789/0 conn(0x7f9d8c0a57e0 legacy=0x7f9d8c08eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:59.501 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:59.486+0000 7f9d92740700 1 -- 172.21.15.42:0/1221728848 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d8c08f350 con 0x7f9d8c0a6690 2019-12-05T06:11:59.501 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:59.486+0000 7f9d92740700 1 --2- 172.21.15.42:0/1221728848 >> v2:172.21.15.42:3300/0 conn(0x7f9d8c0a6690 0x7f9d8c064960 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f9d88002f30 tx=0x7f9d88008bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:59.501 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:59.486+0000 7f9d92740700 1 --2- 172.21.15.42:0/1221728848 >> v2:172.21.15.42:3300/0 conn(0x7f9d8c0a6690 0x7f9d8c064960 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f9d88002f30 tx=0x7f9d88008bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:59.501 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:59.486+0000 7f9d92740700 1 --2- 172.21.15.42:0/1221728848 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9d8c0a6690 0x7f9d8c064960 secure :-1 s=READY pgs=2230 cs=0 l=1 rx=0x7f9d88002f30 tx=0x7f9d88008bb0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:11:59.501 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:59.486+0000 7f9d92740700 1 --2- 172.21.15.42:0/1221728848 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9d8c0a6690 0x7f9d8c064960 secure :-1 s=THROTTLE_DONE pgs=2230 cs=0 l=1 rx=0x7f9d88002f30 tx=0x7f9d88008bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:59.501 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:59.486+0000 7f9d82ffd700 1 -- 172.21.15.42:0/1221728848 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9d88007150 con 0x7f9d8c0a6690 2019-12-05T06:11:59.502 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:59.486+0000 7f9d92740700 1 --2- 172.21.15.42:0/1221728848 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9d8c0a6690 0x7f9d8c064960 secure :-1 s=THROTTLE_DONE pgs=2230 cs=0 l=1 rx=0x7f9d88002f30 tx=0x7f9d88008bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:59.502 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:59.486+0000 7f9d92740700 1 --2- 172.21.15.42:0/1221728848 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9d8c0a6690 0x7f9d8c064960 secure :-1 s=THROTTLE_DONE pgs=2230 cs=0 l=1 rx=0x7f9d88002f30 tx=0x7f9d88008bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:59.502 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:59.486+0000 7f9d82ffd700 1 -- 172.21.15.42:0/1221728848 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d88011570 con 0x7f9d8c0a6690 2019-12-05T06:11:59.502 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:59.486+0000 7f9d941a3700 1 -- 172.21.15.42:0/1221728848 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d8c08f620 con 0x7f9d8c0a6690 2019-12-05T06:11:59.502 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:59.486+0000 7f9d82ffd700 1 -- 172.21.15.42:0/1221728848 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9d88005730 con 0x7f9d8c0a6690 2019-12-05T06:11:59.502 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:59.486+0000 7f9d941a3700 1 -- 172.21.15.42:0/1221728848 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d8c1318a0 con 0x7f9d8c0a6690 2019-12-05T06:11:59.503 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:59.486+0000 7f9d92740700 1 --2- 172.21.15.42:0/1221728848 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9d8c0a6690 0x7f9d8c064960 secure :-1 s=THROTTLE_DONE pgs=2230 cs=0 l=1 rx=0x7f9d88002f30 tx=0x7f9d88008bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:59.503 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:59.490+0000 7f9d82ffd700 1 -- 172.21.15.42:0/1221728848 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f9d88005a40 con 0x7f9d8c0a6690 2019-12-05T06:11:59.503 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:59.490+0000 7f9d92740700 1 --2- 172.21.15.42:0/1221728848 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9d8c0a6690 0x7f9d8c064960 secure :-1 s=THROTTLE_DONE pgs=2230 cs=0 l=1 rx=0x7f9d88002f30 tx=0x7f9d88008bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:59.503 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:59.490+0000 7f9d82ffd700 1 --2- 172.21.15.42:0/1221728848 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9d7403ae00 0x7f9d7403d310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:11:59.503 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:59.490+0000 7f9d82ffd700 1 -- 172.21.15.42:0/1221728848 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 10073+0+0 (secure 0 0 0) 0x7f9d8804b4f0 con 0x7f9d8c0a6690 2019-12-05T06:11:59.504 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:59.490+0000 7f9d91f3f700 1 --2- 172.21.15.42:0/1221728848 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9d7403ae00 0x7f9d7403d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:11:59.504 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:59.490+0000 7f9d91f3f700 1 --2- 172.21.15.42:0/1221728848 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9d7403ae00 0x7f9d7403d310 crc :-1 s=READY pgs=937 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:11:59.505 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:59.494+0000 7f9d941a3700 1 -- 172.21.15.42:0/1221728848 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d8c05d110 con 0x7f9d8c0a6690 2019-12-05T06:11:59.505 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:59.502+0000 7f9d92740700 1 --2- 172.21.15.42:0/1221728848 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9d8c0a6690 0x7f9d8c064960 secure :-1 s=THROTTLE_DONE pgs=2230 cs=0 l=1 rx=0x7f9d88002f30 tx=0x7f9d88008bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:59.506 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:59.502+0000 7f9d82ffd700 1 -- 172.21.15.42:0/1221728848 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f9d88005d50 con 0x7f9d8c0a6690 2019-12-05T06:11:59.679 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:59.674+0000 7f9d941a3700 1 -- 172.21.15.42:0/1221728848 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9d8c05b8c0 con 0x7f9d8c0a6690 2019-12-05T06:11:59.680 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:59.674+0000 7f9d92740700 1 --2- 172.21.15.42:0/1221728848 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9d8c0a6690 0x7f9d8c064960 secure :-1 s=THROTTLE_DONE pgs=2230 cs=0 l=1 rx=0x7f9d88002f30 tx=0x7f9d88008bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:59.680 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:59.674+0000 7f9d82ffd700 1 -- 172.21.15.42:0/1221728848 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 268 v268) v1 ==== 95+0+2115 (secure 0 0 0) 0x7f9d880301f0 con 0x7f9d8c0a6690 2019-12-05T06:11:59.680 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 268 2019-12-05T06:11:59.680 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:11:59.681 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":268,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7094,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/3785523496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3785523496},{"type":"v1","addr":"172.21.15.40:6833","nonce":3785523496}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":268}],"filesystems":[{"mdsmap":{"epoch":267,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:11:34.953298+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":96,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10000},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10000":{"gid":10000,"name":"a","rank":0,"incarnation":264,"state":"up:active","state_seq":21,"addr":"172.21.15.42:6835/3923117701","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3923117701},{"type":"v1","addr":"172.21.15.42:6835","nonce":3923117701}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:11:59.682 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:59.678+0000 7f9d941a3700 1 -- 172.21.15.42:0/1221728848 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9d7403ae00 msgr2=0x7f9d7403d310 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:59.682 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:59.678+0000 7f9d941a3700 1 --2- 172.21.15.42:0/1221728848 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9d7403ae00 0x7f9d7403d310 crc :-1 s=READY pgs=937 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:11:59.683 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:59.678+0000 7f9d941a3700 1 -- 172.21.15.42:0/1221728848 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9d8c0a6690 msgr2=0x7f9d8c064960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:59.683 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:59.678+0000 7f9d941a3700 1 --2- 172.21.15.42:0/1221728848 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9d8c0a6690 0x7f9d8c064960 secure :-1 s=READY pgs=2230 cs=0 l=1 rx=0x7f9d88002f30 tx=0x7f9d88008bb0).stop 2019-12-05T06:11:59.683 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:59.678+0000 7f9d941a3700 1 -- 172.21.15.42:0/1221728848 shutdown_connections 2019-12-05T06:11:59.683 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:59.678+0000 7f9d941a3700 1 --2- 172.21.15.42:0/1221728848 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9d7403ae00 0x7f9d7403d310 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:11:59.683 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:59.678+0000 7f9d941a3700 1 --2- 172.21.15.42:0/1221728848 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9d8c0a6690 0x7f9d8c064960 unknown :-1 s=CLOSED pgs=2230 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:11:59.683 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:59.678+0000 7f9d941a3700 1 --2- 172.21.15.42:0/1221728848 >> v2:172.21.15.40:3300/0 conn(0x7f9d8c0634d0 0x7f9d8c0a5290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:11:59.684 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:59.678+0000 7f9d941a3700 1 -- 172.21.15.42:0/1221728848 shutdown_connections 2019-12-05T06:11:59.684 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:59.678+0000 7f9d941a3700 1 -- 172.21.15.42:0/1221728848 wait complete. 2019-12-05T06:11:59.684 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:11:59.678+0000 7f9d941a3700 1 -- 172.21.15.42:0/1221728848 >> 172.21.15.42:0/1221728848 conn(0x7f9d8c0a89d0 msgr2=0x7f9d8c059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:11:59.692 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:12:01.693 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:12:01.805 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:01.794+0000 7f3cc8fea700 1 Processor -- start 2019-12-05T06:12:01.806 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:01.794+0000 7f3cc8fea700 1 -- start start 2019-12-05T06:12:01.806 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:01.798+0000 7f3cc8fea700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f3cc408df80 0x7f3cc409f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:12:01.806 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:01.798+0000 7f3cc8fea700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3cc409f830 0x7f3cc414efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:12:01.806 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:01.798+0000 7f3cc8fea700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cc409fc60 con 0x7f3cc40a7740 2019-12-05T06:12:01.806 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:01.798+0000 7f3cc8fea700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3cc40a0b00 con 0x7f3cc408df80 2019-12-05T06:12:01.807 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:01.798+0000 7f3cc8fea700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3cc408e5b0 con 0x7f3cc409f830 2019-12-05T06:12:01.807 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:01.798+0000 7f3cc359e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3cc409f830 0x7f3cc414efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:12:01.807 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:01.798+0000 7f3cc2d9d700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f3cc40a7740 0x7f3cc407e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:50362/0 (socket says 172.21.15.42:50362) 2019-12-05T06:12:01.807 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:01.798+0000 7f3cc2d9d700 1 -- 172.21.15.42:0/3865018775 learned_addr learned my addr 172.21.15.42:0/3865018775 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:12:01.808 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:01.798+0000 7f3cc259c700 1 --2- 172.21.15.42:0/3865018775 >> v2:172.21.15.40:3300/0 conn(0x7f3cc408df80 0x7f3cc409f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:12:01.808 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:01.798+0000 7f3cc359e700 1 -- 172.21.15.42:0/3865018775 >> v2:172.21.15.40:3300/0 conn(0x7f3cc408df80 msgr2=0x7f3cc409f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:12:01.808 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:01.798+0000 7f3cc359e700 1 --2- 172.21.15.42:0/3865018775 >> v2:172.21.15.40:3300/0 conn(0x7f3cc408df80 0x7f3cc409f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:12:01.808 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:01.798+0000 7f3cc359e700 1 -- 172.21.15.42:0/3865018775 >> v1:172.21.15.42:6789/0 conn(0x7f3cc40a7740 legacy=0x7f3cc407e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:01.808 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:01.798+0000 7f3cc1d9b700 1 -- 172.21.15.42:0/3865018775 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3067052795 0 0) 0x7f3cc409fc60 con 0x7f3cc40a7740 2019-12-05T06:12:01.808 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:01.798+0000 7f3cc359e700 1 -- 172.21.15.42:0/3865018775 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cc40a0710 con 0x7f3cc409f830 2019-12-05T06:12:01.808 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:01.798+0000 7f3cc359e700 1 --2- 172.21.15.42:0/3865018775 >> v2:172.21.15.42:3300/0 conn(0x7f3cc409f830 0x7f3cc414efe0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f3cbc0032b0 tx=0x7f3cbc00fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:01.809 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:01.798+0000 7f3cc359e700 1 --2- 172.21.15.42:0/3865018775 >> v2:172.21.15.42:3300/0 conn(0x7f3cc409f830 0x7f3cc414efe0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f3cbc0032b0 tx=0x7f3cbc00fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:01.809 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:01.798+0000 7f3cc359e700 1 --2- 172.21.15.42:0/3865018775 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3cc409f830 0x7f3cc414efe0 secure :-1 s=READY pgs=2232 cs=0 l=1 rx=0x7f3cbc0032b0 tx=0x7f3cbc00fff0).ready entity=mon.1 client_cookie=62466601ee8dedfb server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:12:01.809 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:01.798+0000 7f3cc359e700 1 --2- 172.21.15.42:0/3865018775 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3cc409f830 0x7f3cc414efe0 secure :-1 s=THROTTLE_DONE pgs=2232 cs=0 l=1 rx=0x7f3cbc0032b0 tx=0x7f3cbc00fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:01.809 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:01.798+0000 7f3cc1d9b700 1 -- 172.21.15.42:0/3865018775 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3cbc0108c0 con 0x7f3cc409f830 2019-12-05T06:12:01.809 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:01.798+0000 7f3cc359e700 1 --2- 172.21.15.42:0/3865018775 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3cc409f830 0x7f3cc414efe0 secure :-1 s=THROTTLE_DONE pgs=2232 cs=0 l=1 rx=0x7f3cbc0032b0 tx=0x7f3cbc00fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:01.809 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:01.798+0000 7f3cc1d9b700 1 -- 172.21.15.42:0/3865018775 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cbc008b50 con 0x7f3cc409f830 2019-12-05T06:12:01.809 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:01.798+0000 7f3cc359e700 1 --2- 172.21.15.42:0/3865018775 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3cc409f830 0x7f3cc414efe0 secure :-1 s=THROTTLE_DONE pgs=2232 cs=0 l=1 rx=0x7f3cbc0032b0 tx=0x7f3cbc00fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:01.810 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:01.798+0000 7f3cc8fea700 1 -- 172.21.15.42:0/3865018775 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3cc409f830 msgr2=0x7f3cc414efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:01.810 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:01.798+0000 7f3cc1d9b700 1 -- 172.21.15.42:0/3865018775 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3cbc0110c0 con 0x7f3cc409f830 2019-12-05T06:12:01.810 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:01.798+0000 7f3cc8fea700 1 --2- 172.21.15.42:0/3865018775 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3cc409f830 0x7f3cc414efe0 secure :-1 s=READY pgs=2232 cs=0 l=1 rx=0x7f3cbc0032b0 tx=0x7f3cbc00fff0).stop 2019-12-05T06:12:01.810 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:01.802+0000 7f3cc8fea700 1 -- 172.21.15.42:0/3865018775 shutdown_connections 2019-12-05T06:12:01.810 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:01.802+0000 7f3cc8fea700 1 --2- 172.21.15.42:0/3865018775 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3cc409f830 0x7f3cc414efe0 unknown :-1 s=CLOSED pgs=2232 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:01.810 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:01.802+0000 7f3cc8fea700 1 --2- 172.21.15.42:0/3865018775 >> v2:172.21.15.40:3300/0 conn(0x7f3cc408df80 0x7f3cc409f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:12:01.811 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:01.802+0000 7f3cc8fea700 1 -- 172.21.15.42:0/3865018775 shutdown_connections 2019-12-05T06:12:01.811 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:01.802+0000 7f3cc8fea700 1 -- 172.21.15.42:0/3865018775 wait complete. 2019-12-05T06:12:01.811 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:01.802+0000 7f3cc8fea700 1 -- 172.21.15.42:0/3865018775 >> 172.21.15.42:0/3865018775 conn(0x7f3cc40aabe0 msgr2=0x7f3cc40a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:12:01.814 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:01.802+0000 7f3cc8fea700 1 Processor -- start 2019-12-05T06:12:01.815 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:01.802+0000 7f3cc8fea700 1 -- start start 2019-12-05T06:12:01.815 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:01.806+0000 7f3cc8fea700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3cc40a19f0 0x7f3cc4123010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:12:01.815 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:01.806+0000 7f3cc8fea700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f3cc4126f30 0x7f3cc412da20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:12:01.815 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:01.806+0000 7f3cc8fea700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cc407a8a0 con 0x7f3cc407aea0 2019-12-05T06:12:01.815 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:01.806+0000 7f3cc8fea700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3cc407b2d0 con 0x7f3cc4126f30 2019-12-05T06:12:01.816 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:01.806+0000 7f3cc8fea700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3cc4126750 con 0x7f3cc40a19f0 2019-12-05T06:12:01.816 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:01.806+0000 7f3cc2d9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3cc40a19f0 0x7f3cc4123010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:12:01.816 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:01.806+0000 7f3cc2d9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3cc40a19f0 0x7f3cc4123010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:50330/0 (socket says 172.21.15.42:50330) 2019-12-05T06:12:01.816 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:01.806+0000 7f3cc2d9d700 1 -- 172.21.15.42:0/3097695426 learned_addr learned my addr 172.21.15.42:0/3097695426 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:12:01.816 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:01.806+0000 7f3cc259c700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f3cc407aea0 0x7f3cc414e870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:50372/0 (socket says 172.21.15.42:50372) 2019-12-05T06:12:01.816 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:01.806+0000 7f3cc359e700 1 --2- 172.21.15.42:0/3097695426 >> v2:172.21.15.40:3300/0 conn(0x7f3cc4126f30 0x7f3cc412da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:12:01.817 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:01.806+0000 7f3cc2d9d700 1 -- 172.21.15.42:0/3097695426 >> v2:172.21.15.40:3300/0 conn(0x7f3cc4126f30 msgr2=0x7f3cc412da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:01.817 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:01.806+0000 7f3cc2d9d700 1 --2- 172.21.15.42:0/3097695426 >> v2:172.21.15.40:3300/0 conn(0x7f3cc4126f30 0x7f3cc412da20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:01.817 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:01.806+0000 7f3cc2d9d700 1 -- 172.21.15.42:0/3097695426 >> v1:172.21.15.42:6789/0 conn(0x7f3cc407aea0 legacy=0x7f3cc414e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:01.818 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:01.806+0000 7f3cc2d9d700 1 -- 172.21.15.42:0/3097695426 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cc40a4380 con 0x7f3cc40a19f0 2019-12-05T06:12:01.818 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:01.806+0000 7f3cc2d9d700 1 --2- 172.21.15.42:0/3097695426 >> v2:172.21.15.42:3300/0 conn(0x7f3cc40a19f0 0x7f3cc4123010 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f3cac000b80 tx=0x7f3cac010940).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:01.818 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:01.806+0000 7f3cc2d9d700 1 --2- 172.21.15.42:0/3097695426 >> v2:172.21.15.42:3300/0 conn(0x7f3cc40a19f0 0x7f3cc4123010 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f3cac000b80 tx=0x7f3cac010940).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:01.819 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:01.806+0000 7f3cc2d9d700 1 --2- 172.21.15.42:0/3097695426 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3cc40a19f0 0x7f3cc4123010 secure :-1 s=READY pgs=2234 cs=0 l=1 rx=0x7f3cac000b80 tx=0x7f3cac010940).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:12:01.819 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:01.806+0000 7f3cc2d9d700 1 --2- 172.21.15.42:0/3097695426 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3cc40a19f0 0x7f3cc4123010 secure :-1 s=THROTTLE_DONE pgs=2234 cs=0 l=1 rx=0x7f3cac000b80 tx=0x7f3cac010940).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:01.819 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:01.806+0000 7f3cc2d9d700 1 --2- 172.21.15.42:0/3097695426 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3cc40a19f0 0x7f3cc4123010 secure :-1 s=THROTTLE_DONE pgs=2234 cs=0 l=1 rx=0x7f3cac000b80 tx=0x7f3cac010940).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:01.819 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:01.806+0000 7f3cc2d9d700 1 --2- 172.21.15.42:0/3097695426 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3cc40a19f0 0x7f3cc4123010 secure :-1 s=THROTTLE_DONE pgs=2234 cs=0 l=1 rx=0x7f3cac000b80 tx=0x7f3cac010940).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:01.820 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:01.806+0000 7f3cbb7fe700 1 -- 172.21.15.42:0/3097695426 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3cac011670 con 0x7f3cc40a19f0 2019-12-05T06:12:01.820 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:01.806+0000 7f3cbb7fe700 1 -- 172.21.15.42:0/3097695426 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cac011290 con 0x7f3cc40a19f0 2019-12-05T06:12:01.820 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:01.806+0000 7f3cbb7fe700 1 -- 172.21.15.42:0/3097695426 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3cac011c50 con 0x7f3cc40a19f0 2019-12-05T06:12:01.820 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:01.806+0000 7f3cc8fea700 1 -- 172.21.15.42:0/3097695426 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cc4127360 con 0x7f3cc40a19f0 2019-12-05T06:12:01.821 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:01.806+0000 7f3cc8fea700 1 -- 172.21.15.42:0/3097695426 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cc412df70 con 0x7f3cc40a19f0 2019-12-05T06:12:01.821 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:01.810+0000 7f3cc2d9d700 1 --2- 172.21.15.42:0/3097695426 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3cc40a19f0 0x7f3cc4123010 secure :-1 s=THROTTLE_DONE pgs=2234 cs=0 l=1 rx=0x7f3cac000b80 tx=0x7f3cac010940).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:01.821 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:01.810+0000 7f3cbb7fe700 1 -- 172.21.15.42:0/3097695426 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f3cac00b330 con 0x7f3cc40a19f0 2019-12-05T06:12:01.821 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:01.810+0000 7f3cc2d9d700 1 --2- 172.21.15.42:0/3097695426 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3cc40a19f0 0x7f3cc4123010 secure :-1 s=THROTTLE_DONE pgs=2234 cs=0 l=1 rx=0x7f3cac000b80 tx=0x7f3cac010940).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:01.822 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:01.810+0000 7f3cbb7fe700 1 --2- 172.21.15.42:0/3097695426 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3ca803adc0 0x7f3ca803d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:12:01.822 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:01.810+0000 7f3cbb7fe700 1 -- 172.21.15.42:0/3097695426 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 10073+0+0 (secure 0 0 0) 0x7f3cac04b280 con 0x7f3cc40a19f0 2019-12-05T06:12:01.822 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:01.810+0000 7f3cc259c700 1 --2- 172.21.15.42:0/3097695426 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3ca803adc0 0x7f3ca803d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:12:01.822 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:01.810+0000 7f3cc259c700 1 --2- 172.21.15.42:0/3097695426 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3ca803adc0 0x7f3ca803d2d0 crc :-1 s=READY pgs=938 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:12:01.823 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:01.814+0000 7f3cc8fea700 1 -- 172.21.15.42:0/3097695426 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cc405d110 con 0x7f3cc40a19f0 2019-12-05T06:12:01.825 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:01.822+0000 7f3cc2d9d700 1 --2- 172.21.15.42:0/3097695426 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3cc40a19f0 0x7f3cc4123010 secure :-1 s=THROTTLE_DONE pgs=2234 cs=0 l=1 rx=0x7f3cac000b80 tx=0x7f3cac010940).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:01.825 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:01.822+0000 7f3cbb7fe700 1 -- 172.21.15.42:0/3097695426 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f3cac032300 con 0x7f3cc40a19f0 2019-12-05T06:12:01.991 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:01.986+0000 7f3cc8fea700 1 -- 172.21.15.42:0/3097695426 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3cc405b8c0 con 0x7f3cc40a19f0 2019-12-05T06:12:01.992 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:01.990+0000 7f3cc2d9d700 1 --2- 172.21.15.42:0/3097695426 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3cc40a19f0 0x7f3cc4123010 secure :-1 s=THROTTLE_DONE pgs=2234 cs=0 l=1 rx=0x7f3cac000b80 tx=0x7f3cac010940).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:01.992 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:01.990+0000 7f3cbb7fe700 1 -- 172.21.15.42:0/3097695426 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 268 v268) v1 ==== 95+0+2115 (secure 0 0 0) 0x7f3cac032560 con 0x7f3cc40a19f0 2019-12-05T06:12:01.992 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 268 2019-12-05T06:12:01.993 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:12:01.993 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":268,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7094,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/3785523496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3785523496},{"type":"v1","addr":"172.21.15.40:6833","nonce":3785523496}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":268}],"filesystems":[{"mdsmap":{"epoch":267,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:11:34.953298+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":96,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10000},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10000":{"gid":10000,"name":"a","rank":0,"incarnation":264,"state":"up:active","state_seq":21,"addr":"172.21.15.42:6835/3923117701","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3923117701},{"type":"v1","addr":"172.21.15.42:6835","nonce":3923117701}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:12:01.994 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:01.990+0000 7f3cc8fea700 1 -- 172.21.15.42:0/3097695426 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3ca803adc0 msgr2=0x7f3ca803d2d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:01.994 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:01.990+0000 7f3cc8fea700 1 --2- 172.21.15.42:0/3097695426 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3ca803adc0 0x7f3ca803d2d0 crc :-1 s=READY pgs=938 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:01.995 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:01.990+0000 7f3cc8fea700 1 -- 172.21.15.42:0/3097695426 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3cc40a19f0 msgr2=0x7f3cc4123010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:01.995 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:01.990+0000 7f3cc8fea700 1 --2- 172.21.15.42:0/3097695426 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3cc40a19f0 0x7f3cc4123010 secure :-1 s=READY pgs=2234 cs=0 l=1 rx=0x7f3cac000b80 tx=0x7f3cac010940).stop 2019-12-05T06:12:01.995 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:01.990+0000 7f3cc8fea700 1 -- 172.21.15.42:0/3097695426 shutdown_connections 2019-12-05T06:12:01.996 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:01.990+0000 7f3cc8fea700 1 --2- 172.21.15.42:0/3097695426 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3ca803adc0 0x7f3ca803d2d0 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:01.996 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:01.990+0000 7f3cc8fea700 1 --2- 172.21.15.42:0/3097695426 >> v2:172.21.15.40:3300/0 conn(0x7f3cc4126f30 0x7f3cc412da20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:01.996 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:01.990+0000 7f3cc8fea700 1 --2- 172.21.15.42:0/3097695426 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3cc40a19f0 0x7f3cc4123010 unknown :-1 s=CLOSED pgs=2234 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:01.996 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:01.990+0000 7f3cc8fea700 1 -- 172.21.15.42:0/3097695426 shutdown_connections 2019-12-05T06:12:01.997 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:01.990+0000 7f3cc8fea700 1 -- 172.21.15.42:0/3097695426 wait complete. 2019-12-05T06:12:01.997 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:01.990+0000 7f3cc8fea700 1 -- 172.21.15.42:0/3097695426 >> 172.21.15.42:0/3097695426 conn(0x7f3cc4079090 msgr2=0x7f3cc407e670 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:12:02.005 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:12:04.005 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:12:04.119 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:04.110+0000 7f450ffd5700 1 Processor -- start 2019-12-05T06:12:04.119 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:04.110+0000 7f450ffd5700 1 -- start start 2019-12-05T06:12:04.119 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:04.110+0000 7f450ffd5700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f450809fae0 0x7f450814fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:12:04.119 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:04.110+0000 7f450ffd5700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45080a0710 con 0x7f45080a7740 2019-12-05T06:12:04.119 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:04.110+0000 7f450ffd5700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f450807ef20 con 0x7f450808df80 2019-12-05T06:12:04.119 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:04.110+0000 7f450ffd5700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45080a0b00 con 0x7f450809fae0 2019-12-05T06:12:04.120 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:04.110+0000 7f450ed73700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f450809fae0 0x7f450814fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:12:04.120 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:04.110+0000 7f450dd71700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f450808df80 0x7f450809f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:50376/0 (socket says 172.21.15.42:50376) 2019-12-05T06:12:04.120 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:04.110+0000 7f450dd71700 1 -- 172.21.15.42:0/2954848616 learned_addr learned my addr 172.21.15.42:0/2954848616 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:12:04.120 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:04.110+0000 7f450d570700 1 -- 172.21.15.42:0/2954848616 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3443546995 0 0) 0x7f450807ef20 con 0x7f450808df80 2019-12-05T06:12:04.120 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:04.110+0000 7f450d570700 1 -- 172.21.15.42:0/2954848616 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44f40030b0 con 0x7f450808df80 2019-12-05T06:12:04.120 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:04.114+0000 7f450ed73700 1 -- 172.21.15.42:0/2954848616 >> v1:172.21.15.42:6789/0 conn(0x7f450808df80 legacy=0x7f450809f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:04.121 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:04.114+0000 7f450ed73700 1 -- 172.21.15.42:0/2954848616 >> v1:172.21.15.40:6789/0 conn(0x7f45080a7740 legacy=0x7f450807e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:04.121 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:04.114+0000 7f450d570700 1 -- 172.21.15.42:0/2954848616 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2678063277 0 0) 0x7f45080a0710 con 0x7f45080a7740 2019-12-05T06:12:04.121 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:04.114+0000 7f450ed73700 1 -- 172.21.15.42:0/2954848616 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45080a9ba0 con 0x7f450809fae0 2019-12-05T06:12:04.121 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:04.114+0000 7f450ed73700 1 --2- 172.21.15.42:0/2954848616 >> v2:172.21.15.42:3300/0 conn(0x7f450809fae0 0x7f450814fff0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f4500008790 tx=0x7f450000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:04.121 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:04.114+0000 7f450ed73700 1 --2- 172.21.15.42:0/2954848616 >> v2:172.21.15.42:3300/0 conn(0x7f450809fae0 0x7f450814fff0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f4500008790 tx=0x7f450000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:04.122 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:04.114+0000 7f450ed73700 1 --2- 172.21.15.42:0/2954848616 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f450809fae0 0x7f450814fff0 secure :-1 s=READY pgs=2236 cs=0 l=1 rx=0x7f4500008790 tx=0x7f450000fbf0).ready entity=mon.1 client_cookie=c7fcaf481cc16a6 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:12:04.122 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:04.114+0000 7f450ed73700 1 --2- 172.21.15.42:0/2954848616 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f450809fae0 0x7f450814fff0 secure :-1 s=THROTTLE_DONE pgs=2236 cs=0 l=1 rx=0x7f4500008790 tx=0x7f450000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:04.123 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:04.114+0000 7f450d570700 1 -- 172.21.15.42:0/2954848616 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4500008bb0 con 0x7f450809fae0 2019-12-05T06:12:04.123 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:04.114+0000 7f450ed73700 1 --2- 172.21.15.42:0/2954848616 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f450809fae0 0x7f450814fff0 secure :-1 s=THROTTLE_DONE pgs=2236 cs=0 l=1 rx=0x7f4500008790 tx=0x7f450000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:04.123 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:04.114+0000 7f450ed73700 1 --2- 172.21.15.42:0/2954848616 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f450809fae0 0x7f450814fff0 secure :-1 s=THROTTLE_DONE pgs=2236 cs=0 l=1 rx=0x7f4500008790 tx=0x7f450000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:04.123 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:04.114+0000 7f450d570700 1 -- 172.21.15.42:0/2954848616 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45000087d0 con 0x7f450809fae0 2019-12-05T06:12:04.123 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:04.114+0000 7f450d570700 1 -- 172.21.15.42:0/2954848616 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f45000118d0 con 0x7f450809fae0 2019-12-05T06:12:04.123 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:04.114+0000 7f450ffd5700 1 -- 172.21.15.42:0/2954848616 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f450809fae0 msgr2=0x7f450814fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:04.124 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:04.114+0000 7f450ffd5700 1 --2- 172.21.15.42:0/2954848616 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f450809fae0 0x7f450814fff0 secure :-1 s=READY pgs=2236 cs=0 l=1 rx=0x7f4500008790 tx=0x7f450000fbf0).stop 2019-12-05T06:12:04.124 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:04.114+0000 7f450ffd5700 1 -- 172.21.15.42:0/2954848616 shutdown_connections 2019-12-05T06:12:04.124 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:04.114+0000 7f450ffd5700 1 --2- 172.21.15.42:0/2954848616 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f450809fae0 0x7f450814fff0 unknown :-1 s=CLOSED pgs=2236 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:04.124 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:04.114+0000 7f450ffd5700 1 -- 172.21.15.42:0/2954848616 shutdown_connections 2019-12-05T06:12:04.124 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:04.114+0000 7f450ffd5700 1 -- 172.21.15.42:0/2954848616 wait complete. 2019-12-05T06:12:04.124 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:04.114+0000 7f450ffd5700 1 -- 172.21.15.42:0/2954848616 >> 172.21.15.42:0/2954848616 conn(0x7f45080aabe0 msgr2=0x7f45080a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:12:04.129 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:04.118+0000 7f450ffd5700 1 Processor -- start 2019-12-05T06:12:04.130 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:04.118+0000 7f450ffd5700 1 -- start start 2019-12-05T06:12:04.130 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:04.118+0000 7f450ffd5700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f450808efb0 0x7f4508065560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:12:04.130 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:04.118+0000 7f450ffd5700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f45081305a0 0x7f4508077cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:12:04.130 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:04.118+0000 7f450ffd5700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4508131420 con 0x7f45081309d0 2019-12-05T06:12:04.130 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:04.118+0000 7f450ffd5700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4508130ee0 con 0x7f45081305a0 2019-12-05T06:12:04.130 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:04.118+0000 7f450ffd5700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4508131180 con 0x7f450808efb0 2019-12-05T06:12:04.131 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:04.118+0000 7f450dd71700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f45081305a0 0x7f4508077cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:12:04.131 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:04.118+0000 7f450dd71700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f45081305a0 0x7f4508077cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.42:38162/0 (socket says 172.21.15.42:38162) 2019-12-05T06:12:04.131 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:04.118+0000 7f450e572700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f450808efb0 0x7f4508065560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:12:04.131 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:04.118+0000 7f450dd71700 1 -- 172.21.15.42:0/4287657436 learned_addr learned my addr 172.21.15.42:0/4287657436 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:12:04.131 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:04.122+0000 7f450e572700 1 -- 172.21.15.42:0/4287657436 >> v2:172.21.15.40:3300/0 conn(0x7f45081305a0 msgr2=0x7f4508077cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:04.131 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:04.122+0000 7f450e572700 1 --2- 172.21.15.42:0/4287657436 >> v2:172.21.15.40:3300/0 conn(0x7f45081305a0 0x7f4508077cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:04.132 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:04.122+0000 7f450e572700 1 -- 172.21.15.42:0/4287657436 >> v1:172.21.15.40:6789/0 conn(0x7f45081309d0 legacy=0x7f4508063840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:04.132 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:04.122+0000 7f450e572700 1 -- 172.21.15.42:0/4287657436 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4508064040 con 0x7f450808efb0 2019-12-05T06:12:04.133 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:04.122+0000 7f450e572700 1 --2- 172.21.15.42:0/4287657436 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f450808efb0 0x7f4508065560 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f4504000b80 tx=0x7f4504010840).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:04.133 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:04.122+0000 7f450e572700 1 --2- 172.21.15.42:0/4287657436 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f450808efb0 0x7f4508065560 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f4504000b80 tx=0x7f4504010840).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:04.133 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:04.122+0000 7f450e572700 1 --2- 172.21.15.42:0/4287657436 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f450808efb0 0x7f4508065560 secure :-1 s=READY pgs=488 cs=0 l=1 rx=0x7f4504000b80 tx=0x7f4504010840).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:12:04.133 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:04.122+0000 7f450e572700 1 --2- 172.21.15.42:0/4287657436 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f450808efb0 0x7f4508065560 secure :-1 s=THROTTLE_DONE pgs=488 cs=0 l=1 rx=0x7f4504000b80 tx=0x7f4504010840).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:04.133 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:04.122+0000 7f44faffd700 1 -- 172.21.15.42:0/4287657436 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4504006a40 con 0x7f450808efb0 2019-12-05T06:12:04.133 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:04.122+0000 7f450e572700 1 --2- 172.21.15.42:0/4287657436 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f450808efb0 0x7f4508065560 secure :-1 s=THROTTLE_DONE pgs=488 cs=0 l=1 rx=0x7f4504000b80 tx=0x7f4504010840).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:04.134 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:04.122+0000 7f450e572700 1 --2- 172.21.15.42:0/4287657436 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f450808efb0 0x7f4508065560 secure :-1 s=THROTTLE_DONE pgs=488 cs=0 l=1 rx=0x7f4504000b80 tx=0x7f4504010840).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:04.134 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:04.122+0000 7f44faffd700 1 -- 172.21.15.42:0/4287657436 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4504006660 con 0x7f450808efb0 2019-12-05T06:12:04.134 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:04.122+0000 7f450ffd5700 1 -- 172.21.15.42:0/4287657436 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4508078240 con 0x7f450808efb0 2019-12-05T06:12:04.134 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:04.122+0000 7f44faffd700 1 -- 172.21.15.42:0/4287657436 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4504011080 con 0x7f450808efb0 2019-12-05T06:12:04.134 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:04.122+0000 7f450ffd5700 1 -- 172.21.15.42:0/4287657436 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45081379f0 con 0x7f450808efb0 2019-12-05T06:12:04.134 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:04.122+0000 7f450e572700 1 --2- 172.21.15.42:0/4287657436 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f450808efb0 0x7f4508065560 secure :-1 s=THROTTLE_DONE pgs=488 cs=0 l=1 rx=0x7f4504000b80 tx=0x7f4504010840).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:04.134 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:04.122+0000 7f44faffd700 1 -- 172.21.15.42:0/4287657436 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f4504011390 con 0x7f450808efb0 2019-12-05T06:12:04.135 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:04.122+0000 7f450e572700 1 --2- 172.21.15.42:0/4287657436 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f450808efb0 0x7f4508065560 secure :-1 s=THROTTLE_DONE pgs=488 cs=0 l=1 rx=0x7f4504000b80 tx=0x7f4504010840).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:04.135 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:04.122+0000 7f44faffd700 1 --2- 172.21.15.42:0/4287657436 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f44f003ae00 0x7f44f003d310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:12:04.135 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:04.122+0000 7f44faffd700 1 -- 172.21.15.42:0/4287657436 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 10073+0+0 (secure 0 0 0) 0x7f450404b150 con 0x7f450808efb0 2019-12-05T06:12:04.135 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:04.126+0000 7f450dd71700 1 --2- 172.21.15.42:0/4287657436 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f44f003ae00 0x7f44f003d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:12:04.135 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:04.126+0000 7f450dd71700 1 --2- 172.21.15.42:0/4287657436 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f44f003ae00 0x7f44f003d310 crc :-1 s=READY pgs=939 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:12:04.137 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:04.126+0000 7f450ffd5700 1 -- 172.21.15.42:0/4287657436 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f450805d110 con 0x7f450808efb0 2019-12-05T06:12:04.140 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:04.138+0000 7f450e572700 1 --2- 172.21.15.42:0/4287657436 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f450808efb0 0x7f4508065560 secure :-1 s=THROTTLE_DONE pgs=488 cs=0 l=1 rx=0x7f4504000b80 tx=0x7f4504010840).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:04.140 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:04.138+0000 7f44faffd700 1 -- 172.21.15.42:0/4287657436 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f4504032190 con 0x7f450808efb0 2019-12-05T06:12:04.310 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:04.306+0000 7f450ffd5700 1 -- 172.21.15.42:0/4287657436 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f450805b8c0 con 0x7f450808efb0 2019-12-05T06:12:04.310 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:04.306+0000 7f450e572700 1 --2- 172.21.15.42:0/4287657436 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f450808efb0 0x7f4508065560 secure :-1 s=THROTTLE_DONE pgs=488 cs=0 l=1 rx=0x7f4504000b80 tx=0x7f4504010840).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:04.311 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:04.306+0000 7f44faffd700 1 -- 172.21.15.42:0/4287657436 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 268 v268) v1 ==== 95+0+2115 (secure 0 0 0) 0x7f45040323f0 con 0x7f450808efb0 2019-12-05T06:12:04.313 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 268 2019-12-05T06:12:04.313 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:12:04.313 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":268,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7094,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/3785523496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3785523496},{"type":"v1","addr":"172.21.15.40:6833","nonce":3785523496}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":268}],"filesystems":[{"mdsmap":{"epoch":267,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:11:34.953298+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":96,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10000},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10000":{"gid":10000,"name":"a","rank":0,"incarnation":264,"state":"up:active","state_seq":21,"addr":"172.21.15.42:6835/3923117701","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3923117701},{"type":"v1","addr":"172.21.15.42:6835","nonce":3923117701}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:12:04.316 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:04.310+0000 7f450ffd5700 1 -- 172.21.15.42:0/4287657436 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f44f003ae00 msgr2=0x7f44f003d310 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:04.316 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:04.310+0000 7f450ffd5700 1 --2- 172.21.15.42:0/4287657436 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f44f003ae00 0x7f44f003d310 crc :-1 s=READY pgs=939 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:04.316 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:04.310+0000 7f450ffd5700 1 -- 172.21.15.42:0/4287657436 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f450808efb0 msgr2=0x7f4508065560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:04.316 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:04.310+0000 7f450ffd5700 1 --2- 172.21.15.42:0/4287657436 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f450808efb0 0x7f4508065560 secure :-1 s=READY pgs=488 cs=0 l=1 rx=0x7f4504000b80 tx=0x7f4504010840).stop 2019-12-05T06:12:04.317 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:04.310+0000 7f450ffd5700 1 -- 172.21.15.42:0/4287657436 shutdown_connections 2019-12-05T06:12:04.318 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:04.310+0000 7f450ffd5700 1 --2- 172.21.15.42:0/4287657436 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f44f003ae00 0x7f44f003d310 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:04.318 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:04.310+0000 7f450ffd5700 1 --2- 172.21.15.42:0/4287657436 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f450808efb0 0x7f4508065560 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:04.318 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:04.310+0000 7f450ffd5700 1 --2- 172.21.15.42:0/4287657436 >> v2:172.21.15.40:3300/0 conn(0x7f45081305a0 0x7f4508077cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:04.318 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:04.314+0000 7f450ffd5700 1 -- 172.21.15.42:0/4287657436 shutdown_connections 2019-12-05T06:12:04.320 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:04.314+0000 7f450ffd5700 1 -- 172.21.15.42:0/4287657436 wait complete. 2019-12-05T06:12:04.320 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:04.314+0000 7f450ffd5700 1 -- 172.21.15.42:0/4287657436 >> 172.21.15.42:0/4287657436 conn(0x7f45080a6fc0 msgr2=0x7f450807e4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:12:04.332 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:12:06.333 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:12:06.443 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:06.434+0000 7f294422e700 1 Processor -- start 2019-12-05T06:12:06.444 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:06.434+0000 7f294422e700 1 -- start start 2019-12-05T06:12:06.444 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:06.434+0000 7f294422e700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f293c0a7740 0x7f293c08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:12:06.444 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:06.434+0000 7f294422e700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f293c0a0710 con 0x7f293c09fae0 2019-12-05T06:12:06.444 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:06.434+0000 7f294422e700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f293c0a9ba0 con 0x7f293c07e720 2019-12-05T06:12:06.444 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:06.434+0000 7f294422e700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f293c0a0b00 con 0x7f293c0a7740 2019-12-05T06:12:06.445 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:06.434+0000 7f2941fca700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f293c07e720 0x7f293c09f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:50396/0 (socket says 172.21.15.42:50396) 2019-12-05T06:12:06.445 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:06.434+0000 7f2941fca700 1 -- 172.21.15.42:0/2643451025 learned_addr learned my addr 172.21.15.42:0/2643451025 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:12:06.445 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:06.434+0000 7f29427cb700 1 --2- 172.21.15.42:0/2643451025 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f293c0a7740 0x7f293c08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:12:06.445 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:06.434+0000 7f29417c9700 1 -- 172.21.15.42:0/2643451025 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1459824631 0 0) 0x7f293c0a9ba0 con 0x7f293c07e720 2019-12-05T06:12:06.445 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:06.434+0000 7f29417c9700 1 -- 172.21.15.42:0/2643451025 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29280030b0 con 0x7f293c07e720 2019-12-05T06:12:06.445 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:06.434+0000 7f29417c9700 1 -- 172.21.15.42:0/2643451025 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3111914777 0 0) 0x7f293c0a0710 con 0x7f293c09fae0 2019-12-05T06:12:06.446 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:06.434+0000 7f29417c9700 1 -- 172.21.15.42:0/2643451025 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f293c0a9ba0 con 0x7f293c09fae0 2019-12-05T06:12:06.446 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:06.434+0000 7f29427cb700 1 -- 172.21.15.42:0/2643451025 >> v1:172.21.15.42:6789/0 conn(0x7f293c07e720 legacy=0x7f293c09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:06.446 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:06.434+0000 7f29427cb700 1 -- 172.21.15.42:0/2643451025 >> v1:172.21.15.40:6789/0 conn(0x7f293c09fae0 legacy=0x7f293c14efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:06.446 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:06.434+0000 7f29427cb700 1 -- 172.21.15.42:0/2643451025 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f293c07eb50 con 0x7f293c0a7740 2019-12-05T06:12:06.447 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:06.434+0000 7f29427cb700 1 --2- 172.21.15.42:0/2643451025 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f293c0a7740 0x7f293c08df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f2938002f30 tx=0x7f293800fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:06.447 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:06.438+0000 7f29427cb700 1 --2- 172.21.15.42:0/2643451025 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f293c0a7740 0x7f293c08df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f2938002f30 tx=0x7f293800fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:06.447 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:06.438+0000 7f29427cb700 1 --2- 172.21.15.42:0/2643451025 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f293c0a7740 0x7f293c08df80 secure :-1 s=READY pgs=489 cs=0 l=1 rx=0x7f2938002f30 tx=0x7f293800fbf0).ready entity=mon.2 client_cookie=9b6ab0ae05428742 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:12:06.447 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:06.438+0000 7f29427cb700 1 --2- 172.21.15.42:0/2643451025 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f293c0a7740 0x7f293c08df80 secure :-1 s=THROTTLE_DONE pgs=489 cs=0 l=1 rx=0x7f2938002f30 tx=0x7f293800fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:06.447 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:06.438+0000 7f29417c9700 1 -- 172.21.15.42:0/2643451025 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f29380112b0 con 0x7f293c0a7740 2019-12-05T06:12:06.447 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:06.438+0000 7f29427cb700 1 --2- 172.21.15.42:0/2643451025 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f293c0a7740 0x7f293c08df80 secure :-1 s=THROTTLE_DONE pgs=489 cs=0 l=1 rx=0x7f2938002f30 tx=0x7f293800fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:06.447 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:06.438+0000 7f29427cb700 1 --2- 172.21.15.42:0/2643451025 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f293c0a7740 0x7f293c08df80 secure :-1 s=THROTTLE_DONE pgs=489 cs=0 l=1 rx=0x7f2938002f30 tx=0x7f293800fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:06.448 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:06.438+0000 7f29417c9700 1 -- 172.21.15.42:0/2643451025 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2938008bc0 con 0x7f293c0a7740 2019-12-05T06:12:06.448 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:06.438+0000 7f29417c9700 1 -- 172.21.15.42:0/2643451025 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2938011890 con 0x7f293c0a7740 2019-12-05T06:12:06.448 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:06.438+0000 7f294422e700 1 -- 172.21.15.42:0/2643451025 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f293c0a7740 msgr2=0x7f293c08df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:06.448 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:06.438+0000 7f294422e700 1 --2- 172.21.15.42:0/2643451025 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f293c0a7740 0x7f293c08df80 secure :-1 s=READY pgs=489 cs=0 l=1 rx=0x7f2938002f30 tx=0x7f293800fbf0).stop 2019-12-05T06:12:06.448 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:06.438+0000 7f294422e700 1 -- 172.21.15.42:0/2643451025 shutdown_connections 2019-12-05T06:12:06.448 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:06.438+0000 7f294422e700 1 --2- 172.21.15.42:0/2643451025 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f293c0a7740 0x7f293c08df80 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:06.448 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:06.438+0000 7f294422e700 1 -- 172.21.15.42:0/2643451025 shutdown_connections 2019-12-05T06:12:06.449 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:06.438+0000 7f294422e700 1 -- 172.21.15.42:0/2643451025 wait complete. 2019-12-05T06:12:06.449 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:06.438+0000 7f294422e700 1 -- 172.21.15.42:0/2643451025 >> 172.21.15.42:0/2643451025 conn(0x7f293c0aabe0 msgr2=0x7f293c0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:12:06.452 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:06.442+0000 7f294422e700 1 Processor -- start 2019-12-05T06:12:06.458 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:06.442+0000 7f294422e700 1 -- start start 2019-12-05T06:12:06.458 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:06.442+0000 7f294422e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f293c123cd0 0x7f293c11be90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:12:06.458 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:06.442+0000 7f294422e700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f293c11aad0 0x7f293c11af00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:12:06.458 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:06.442+0000 7f294422e700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f293c07b8d0 con 0x7f293c07d030 2019-12-05T06:12:06.458 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:06.442+0000 7f294422e700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f293c07dd90 con 0x7f293c123cd0 2019-12-05T06:12:06.459 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:06.442+0000 7f294422e700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f293c07d5b0 con 0x7f293c11aad0 2019-12-05T06:12:06.459 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:06.442+0000 7f29427cb700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f293c123cd0 0x7f293c11be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:12:06.459 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:06.442+0000 7f29427cb700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f293c123cd0 0x7f293c11be90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:50362/0 (socket says 172.21.15.42:50362) 2019-12-05T06:12:06.459 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:06.442+0000 7f29427cb700 1 -- 172.21.15.42:0/1923850315 learned_addr learned my addr 172.21.15.42:0/1923850315 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:12:06.459 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:06.442+0000 7f2942fcc700 1 --2- 172.21.15.42:0/1923850315 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f293c11aad0 0x7f293c11af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:12:06.460 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:06.442+0000 7f29427cb700 1 -- 172.21.15.42:0/1923850315 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f293c11aad0 msgr2=0x7f293c11af00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:06.460 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:06.442+0000 7f29427cb700 1 --2- 172.21.15.42:0/1923850315 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f293c11aad0 0x7f293c11af00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:06.460 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:06.442+0000 7f29427cb700 1 -- 172.21.15.42:0/1923850315 >> v1:172.21.15.42:6789/0 conn(0x7f293c07d030 legacy=0x7f293c07b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:06.460 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:06.442+0000 7f29427cb700 1 -- 172.21.15.42:0/1923850315 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f293c11b450 con 0x7f293c123cd0 2019-12-05T06:12:06.460 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:06.442+0000 7f29427cb700 1 --2- 172.21.15.42:0/1923850315 >> v2:172.21.15.42:3300/0 conn(0x7f293c123cd0 0x7f293c11be90 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f2938002f30 tx=0x7f2938007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:06.461 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:06.442+0000 7f29427cb700 1 --2- 172.21.15.42:0/1923850315 >> v2:172.21.15.42:3300/0 conn(0x7f293c123cd0 0x7f293c11be90 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f2938002f30 tx=0x7f2938007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:06.461 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:06.442+0000 7f29427cb700 1 --2- 172.21.15.42:0/1923850315 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f293c123cd0 0x7f293c11be90 secure :-1 s=READY pgs=2239 cs=0 l=1 rx=0x7f2938002f30 tx=0x7f2938007710).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:12:06.461 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:06.446+0000 7f29427cb700 1 --2- 172.21.15.42:0/1923850315 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f293c123cd0 0x7f293c11be90 secure :-1 s=THROTTLE_DONE pgs=2239 cs=0 l=1 rx=0x7f2938002f30 tx=0x7f2938007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:06.461 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:06.446+0000 7f2932ffd700 1 -- 172.21.15.42:0/1923850315 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2938007340 con 0x7f293c123cd0 2019-12-05T06:12:06.461 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:06.446+0000 7f29427cb700 1 --2- 172.21.15.42:0/1923850315 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f293c123cd0 0x7f293c11be90 secure :-1 s=THROTTLE_DONE pgs=2239 cs=0 l=1 rx=0x7f2938002f30 tx=0x7f2938007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:06.462 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:06.446+0000 7f2932ffd700 1 -- 172.21.15.42:0/1923850315 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29380114b0 con 0x7f293c123cd0 2019-12-05T06:12:06.462 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:06.446+0000 7f29427cb700 1 --2- 172.21.15.42:0/1923850315 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f293c123cd0 0x7f293c11be90 secure :-1 s=THROTTLE_DONE pgs=2239 cs=0 l=1 rx=0x7f2938002f30 tx=0x7f2938007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:06.462 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:06.446+0000 7f294422e700 1 -- 172.21.15.42:0/1923850315 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f293c137af0 con 0x7f293c123cd0 2019-12-05T06:12:06.462 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:06.446+0000 7f2932ffd700 1 -- 172.21.15.42:0/1923850315 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f293800f360 con 0x7f293c123cd0 2019-12-05T06:12:06.462 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:06.446+0000 7f294422e700 1 -- 172.21.15.42:0/1923850315 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f293c138020 con 0x7f293c123cd0 2019-12-05T06:12:06.463 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:06.446+0000 7f29427cb700 1 --2- 172.21.15.42:0/1923850315 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f293c123cd0 0x7f293c11be90 secure :-1 s=THROTTLE_DONE pgs=2239 cs=0 l=1 rx=0x7f2938002f30 tx=0x7f2938007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:06.463 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:06.446+0000 7f2932ffd700 1 -- 172.21.15.42:0/1923850315 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f2938005540 con 0x7f293c123cd0 2019-12-05T06:12:06.463 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:06.446+0000 7f29427cb700 1 --2- 172.21.15.42:0/1923850315 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f293c123cd0 0x7f293c11be90 secure :-1 s=THROTTLE_DONE pgs=2239 cs=0 l=1 rx=0x7f2938002f30 tx=0x7f2938007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:06.463 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:06.446+0000 7f2932ffd700 1 --2- 172.21.15.42:0/1923850315 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f292403ad60 0x7f292403d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:12:06.463 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:06.446+0000 7f2932ffd700 1 -- 172.21.15.42:0/1923850315 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 10073+0+0 (secure 0 0 0) 0x7f293804b6a0 con 0x7f293c123cd0 2019-12-05T06:12:06.463 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:06.446+0000 7f2941fca700 1 --2- 172.21.15.42:0/1923850315 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f292403ad60 0x7f292403d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:12:06.464 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:06.450+0000 7f2941fca700 1 --2- 172.21.15.42:0/1923850315 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f292403ad60 0x7f292403d270 crc :-1 s=READY pgs=940 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:12:06.464 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:06.450+0000 7f294422e700 1 -- 172.21.15.42:0/1923850315 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f293c05d110 con 0x7f293c123cd0 2019-12-05T06:12:06.464 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:06.454+0000 7f29427cb700 1 --2- 172.21.15.42:0/1923850315 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f293c123cd0 0x7f293c11be90 secure :-1 s=THROTTLE_DONE pgs=2239 cs=0 l=1 rx=0x7f2938002f30 tx=0x7f2938007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:06.465 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:06.454+0000 7f2932ffd700 1 -- 172.21.15.42:0/1923850315 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f293800f600 con 0x7f293c123cd0 2019-12-05T06:12:06.629 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:06.626+0000 7f294422e700 1 -- 172.21.15.42:0/1923850315 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f293c05b8c0 con 0x7f293c123cd0 2019-12-05T06:12:06.630 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:06.626+0000 7f29427cb700 1 --2- 172.21.15.42:0/1923850315 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f293c123cd0 0x7f293c11be90 secure :-1 s=THROTTLE_DONE pgs=2239 cs=0 l=1 rx=0x7f2938002f30 tx=0x7f2938007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:06.630 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:06.626+0000 7f2932ffd700 1 -- 172.21.15.42:0/1923850315 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 268 v268) v1 ==== 95+0+2115 (secure 0 0 0) 0x7f2938012c00 con 0x7f293c123cd0 2019-12-05T06:12:06.633 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:12:06.633 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":268,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7094,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/3785523496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3785523496},{"type":"v1","addr":"172.21.15.40:6833","nonce":3785523496}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":268}],"filesystems":[{"mdsmap":{"epoch":267,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:11:34.953298+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":96,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10000},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10000":{"gid":10000,"name":"a","rank":0,"incarnation":264,"state":"up:active","state_seq":21,"addr":"172.21.15.42:6835/3923117701","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3923117701},{"type":"v1","addr":"172.21.15.42:6835","nonce":3923117701}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:12:06.633 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 268 2019-12-05T06:12:06.635 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:06.630+0000 7f294422e700 1 -- 172.21.15.42:0/1923850315 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f292403ad60 msgr2=0x7f292403d270 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:06.635 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:06.630+0000 7f294422e700 1 --2- 172.21.15.42:0/1923850315 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f292403ad60 0x7f292403d270 crc :-1 s=READY pgs=940 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:06.635 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:06.630+0000 7f294422e700 1 -- 172.21.15.42:0/1923850315 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f293c123cd0 msgr2=0x7f293c11be90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:06.636 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:06.630+0000 7f294422e700 1 --2- 172.21.15.42:0/1923850315 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f293c123cd0 0x7f293c11be90 secure :-1 s=READY pgs=2239 cs=0 l=1 rx=0x7f2938002f30 tx=0x7f2938007710).stop 2019-12-05T06:12:06.636 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:06.630+0000 7f294422e700 1 -- 172.21.15.42:0/1923850315 shutdown_connections 2019-12-05T06:12:06.636 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:06.630+0000 7f294422e700 1 --2- 172.21.15.42:0/1923850315 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f292403ad60 0x7f292403d270 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:06.636 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:06.630+0000 7f294422e700 1 --2- 172.21.15.42:0/1923850315 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f293c11aad0 0x7f293c11af00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:06.636 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:06.630+0000 7f294422e700 1 --2- 172.21.15.42:0/1923850315 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f293c123cd0 0x7f293c11be90 unknown :-1 s=CLOSED pgs=2239 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:06.636 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:06.630+0000 7f294422e700 1 -- 172.21.15.42:0/1923850315 shutdown_connections 2019-12-05T06:12:06.637 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:06.630+0000 7f294422e700 1 -- 172.21.15.42:0/1923850315 wait complete. 2019-12-05T06:12:06.637 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:06.630+0000 7f294422e700 1 -- 172.21.15.42:0/1923850315 >> 172.21.15.42:0/1923850315 conn(0x7f293c11d290 msgr2=0x7f293c059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:12:06.650 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:12:08.651 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:12:08.754 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:08.742+0000 7f40c9a46700 1 Processor -- start 2019-12-05T06:12:08.756 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:08.746+0000 7f40c9a46700 1 -- start start 2019-12-05T06:12:08.756 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:08.746+0000 7f40c9a46700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f40c40a7740 0x7f40c408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:12:08.756 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:08.746+0000 7f40c9a46700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f40c407e720 0x7f40c407eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:12:08.756 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:08.746+0000 7f40c9a46700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40c40a0710 con 0x7f40c409f2e0 2019-12-05T06:12:08.757 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:08.746+0000 7f40c9a46700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40c40a0b00 con 0x7f40c407e720 2019-12-05T06:12:08.757 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:08.746+0000 7f40c9a46700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40c408e5b0 con 0x7f40c40a7740 2019-12-05T06:12:08.757 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:08.746+0000 7f40c3fff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f40c409f2e0 0x7f40c409f710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:50414/0 (socket says 172.21.15.42:50414) 2019-12-05T06:12:08.757 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:08.746+0000 7f40c3fff700 1 -- 172.21.15.42:0/3697621198 learned_addr learned my addr 172.21.15.42:0/3697621198 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:12:08.757 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:08.746+0000 7f40c2ffd700 1 --2- 172.21.15.42:0/3697621198 >> v2:172.21.15.40:3300/0 conn(0x7f40c407e720 0x7f40c407eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:12:08.758 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:08.746+0000 7f40c37fe700 1 --2- 172.21.15.42:0/3697621198 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f40c40a7740 0x7f40c408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:12:08.758 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:08.746+0000 7f40c27fc700 1 -- 172.21.15.42:0/3697621198 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1740857332 0 0) 0x7f40c40a0710 con 0x7f40c409f2e0 2019-12-05T06:12:08.758 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:08.746+0000 7f40c27fc700 1 -- 172.21.15.42:0/3697621198 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40ac0030b0 con 0x7f40c409f2e0 2019-12-05T06:12:08.759 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:08.746+0000 7f40c2ffd700 1 -- 172.21.15.42:0/3697621198 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f40c40a7740 msgr2=0x7f40c408df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:12:08.759 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:08.746+0000 7f40c2ffd700 1 --2- 172.21.15.42:0/3697621198 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f40c40a7740 0x7f40c408df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:12:08.759 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:08.746+0000 7f40c2ffd700 1 -- 172.21.15.42:0/3697621198 >> v1:172.21.15.42:6789/0 conn(0x7f40c409f2e0 legacy=0x7f40c409f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:08.759 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:08.746+0000 7f40c2ffd700 1 -- 172.21.15.42:0/3697621198 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40c407f0a0 con 0x7f40c407e720 2019-12-05T06:12:08.759 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:08.746+0000 7f40c2ffd700 1 --2- 172.21.15.42:0/3697621198 >> v2:172.21.15.40:3300/0 conn(0x7f40c407e720 0x7f40c407eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f40b8008790 tx=0x7f40b800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:08.760 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:08.746+0000 7f40c2ffd700 1 --2- 172.21.15.42:0/3697621198 >> v2:172.21.15.40:3300/0 conn(0x7f40c407e720 0x7f40c407eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f40b8008790 tx=0x7f40b800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:08.762 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:08.746+0000 7f40c2ffd700 1 --2- 172.21.15.42:0/3697621198 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f40c407e720 0x7f40c407eb50 secure :-1 s=READY pgs=1692 cs=0 l=1 rx=0x7f40b8008790 tx=0x7f40b800fc70).ready entity=mon.0 client_cookie=794dafc7d169ec7c server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:12:08.762 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:08.746+0000 7f40c2ffd700 1 --2- 172.21.15.42:0/3697621198 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f40c407e720 0x7f40c407eb50 secure :-1 s=THROTTLE_DONE pgs=1692 cs=0 l=1 rx=0x7f40b8008790 tx=0x7f40b800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:08.762 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:08.746+0000 7f40c27fc700 1 -- 172.21.15.42:0/3697621198 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f40b80112f0 con 0x7f40c407e720 2019-12-05T06:12:08.762 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:08.746+0000 7f40c2ffd700 1 --2- 172.21.15.42:0/3697621198 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f40c407e720 0x7f40c407eb50 secure :-1 s=THROTTLE_DONE pgs=1692 cs=0 l=1 rx=0x7f40b8008790 tx=0x7f40b800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:08.762 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:08.746+0000 7f40c27fc700 1 -- 172.21.15.42:0/3697621198 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40b8008ad0 con 0x7f40c407e720 2019-12-05T06:12:08.762 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:08.746+0000 7f40c2ffd700 1 --2- 172.21.15.42:0/3697621198 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f40c407e720 0x7f40c407eb50 secure :-1 s=THROTTLE_DONE pgs=1692 cs=0 l=1 rx=0x7f40b8008790 tx=0x7f40b800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:08.763 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:08.746+0000 7f40c27fc700 1 -- 172.21.15.42:0/3697621198 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f40b80118d0 con 0x7f40c407e720 2019-12-05T06:12:08.763 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:08.746+0000 7f40c9a46700 1 -- 172.21.15.42:0/3697621198 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f40c407e720 msgr2=0x7f40c407eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:08.763 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:08.746+0000 7f40c9a46700 1 --2- 172.21.15.42:0/3697621198 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f40c407e720 0x7f40c407eb50 secure :-1 s=READY pgs=1692 cs=0 l=1 rx=0x7f40b8008790 tx=0x7f40b800fc70).stop 2019-12-05T06:12:08.763 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:08.750+0000 7f40c9a46700 1 -- 172.21.15.42:0/3697621198 shutdown_connections 2019-12-05T06:12:08.763 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:08.750+0000 7f40c9a46700 1 --2- 172.21.15.42:0/3697621198 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f40c40a7740 0x7f40c408df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:12:08.763 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:08.750+0000 7f40c9a46700 1 --2- 172.21.15.42:0/3697621198 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f40c407e720 0x7f40c407eb50 unknown :-1 s=CLOSED pgs=1692 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:08.763 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:08.750+0000 7f40c9a46700 1 -- 172.21.15.42:0/3697621198 shutdown_connections 2019-12-05T06:12:08.764 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:08.750+0000 7f40c9a46700 1 -- 172.21.15.42:0/3697621198 wait complete. 2019-12-05T06:12:08.764 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:08.750+0000 7f40c9a46700 1 -- 172.21.15.42:0/3697621198 >> 172.21.15.42:0/3697621198 conn(0x7f40c40aabe0 msgr2=0x7f40c40a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:12:08.767 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:08.750+0000 7f40c9a46700 1 Processor -- start 2019-12-05T06:12:08.768 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:08.750+0000 7f40c9a46700 1 -- start start 2019-12-05T06:12:08.768 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:08.754+0000 7f40c9a46700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f40c40a6c10 0x7f40c4118f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:12:08.768 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:08.754+0000 7f40c9a46700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40c4146f00 con 0x7f40c40a52b0 2019-12-05T06:12:08.768 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:08.754+0000 7f40c9a46700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40c4079250 con 0x7f40c4146590 2019-12-05T06:12:08.768 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:08.754+0000 7f40c9a46700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40c40a64b0 con 0x7f40c40a6c10 2019-12-05T06:12:08.769 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:08.754+0000 7f40c2ffd700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f40c4146590 0x7f40c4078a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:50416/0 (socket says 172.21.15.42:50416) 2019-12-05T06:12:08.769 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:08.754+0000 7f40c2ffd700 1 -- 172.21.15.42:0/3541131046 learned_addr learned my addr 172.21.15.42:0/3541131046 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:12:08.769 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:08.754+0000 7f40c37fe700 1 --2- 172.21.15.42:0/3541131046 >> v2:172.21.15.42:3300/0 conn(0x7f40c40a6c10 0x7f40c4118f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:12:08.769 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:08.754+0000 7f40abfff700 1 -- 172.21.15.42:0/3541131046 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1692896414 0 0) 0x7f40c4079250 con 0x7f40c4146590 2019-12-05T06:12:08.770 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:08.754+0000 7f40abfff700 1 -- 172.21.15.42:0/3541131046 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40a40030b0 con 0x7f40c4146590 2019-12-05T06:12:08.770 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:08.754+0000 7f40abfff700 1 -- 172.21.15.42:0/3541131046 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2401455209 0 0) 0x7f40c4146f00 con 0x7f40c40a52b0 2019-12-05T06:12:08.770 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:08.754+0000 7f40abfff700 1 -- 172.21.15.42:0/3541131046 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40c4079250 con 0x7f40c40a52b0 2019-12-05T06:12:08.770 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:08.754+0000 7f40c37fe700 1 -- 172.21.15.42:0/3541131046 >> v1:172.21.15.42:6789/0 conn(0x7f40c4146590 legacy=0x7f40c4078a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:08.771 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:08.754+0000 7f40c37fe700 1 -- 172.21.15.42:0/3541131046 >> v1:172.21.15.40:6789/0 conn(0x7f40c40a52b0 legacy=0x7f40c40a56e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:08.772 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:08.754+0000 7f40c37fe700 1 -- 172.21.15.42:0/3541131046 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40c4137720 con 0x7f40c40a6c10 2019-12-05T06:12:08.772 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:08.754+0000 7f40c37fe700 1 --2- 172.21.15.42:0/3541131046 >> v2:172.21.15.42:3300/0 conn(0x7f40c40a6c10 0x7f40c4118f60 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f40b40032b0 tx=0x7f40b4010d30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:08.772 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:08.754+0000 7f40c37fe700 1 --2- 172.21.15.42:0/3541131046 >> v2:172.21.15.42:3300/0 conn(0x7f40c40a6c10 0x7f40c4118f60 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f40b40032b0 tx=0x7f40b4010d30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:08.773 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:08.754+0000 7f40c37fe700 1 --2- 172.21.15.42:0/3541131046 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f40c40a6c10 0x7f40c4118f60 secure :-1 s=READY pgs=2242 cs=0 l=1 rx=0x7f40b40032b0 tx=0x7f40b4010d30).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:12:08.773 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:08.754+0000 7f40c37fe700 1 --2- 172.21.15.42:0/3541131046 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f40c40a6c10 0x7f40c4118f60 secure :-1 s=THROTTLE_DONE pgs=2242 cs=0 l=1 rx=0x7f40b40032b0 tx=0x7f40b4010d30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:08.773 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:08.754+0000 7f40abfff700 1 -- 172.21.15.42:0/3541131046 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f40b4005ca0 con 0x7f40c40a6c10 2019-12-05T06:12:08.773 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:08.754+0000 7f40c37fe700 1 --2- 172.21.15.42:0/3541131046 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f40c40a6c10 0x7f40c4118f60 secure :-1 s=THROTTLE_DONE pgs=2242 cs=0 l=1 rx=0x7f40b40032b0 tx=0x7f40b4010d30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:08.773 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:08.754+0000 7f40c37fe700 1 --2- 172.21.15.42:0/3541131046 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f40c40a6c10 0x7f40c4118f60 secure :-1 s=THROTTLE_DONE pgs=2242 cs=0 l=1 rx=0x7f40b40032b0 tx=0x7f40b4010d30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:08.773 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:08.754+0000 7f40abfff700 1 -- 172.21.15.42:0/3541131046 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40b40058c0 con 0x7f40c40a6c10 2019-12-05T06:12:08.773 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:08.754+0000 7f40c9a46700 1 -- 172.21.15.42:0/3541131046 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40c4135700 con 0x7f40c40a6c10 2019-12-05T06:12:08.774 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:08.754+0000 7f40abfff700 1 -- 172.21.15.42:0/3541131046 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f40b4012c40 con 0x7f40c40a6c10 2019-12-05T06:12:08.774 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:08.754+0000 7f40c9a46700 1 -- 172.21.15.42:0/3541131046 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40c4135cb0 con 0x7f40c40a6c10 2019-12-05T06:12:08.774 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:08.758+0000 7f40c37fe700 1 --2- 172.21.15.42:0/3541131046 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f40c40a6c10 0x7f40c4118f60 secure :-1 s=THROTTLE_DONE pgs=2242 cs=0 l=1 rx=0x7f40b40032b0 tx=0x7f40b4010d30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:08.774 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:08.758+0000 7f40abfff700 1 -- 172.21.15.42:0/3541131046 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f40b400c3f0 con 0x7f40c40a6c10 2019-12-05T06:12:08.774 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:08.758+0000 7f40c37fe700 1 --2- 172.21.15.42:0/3541131046 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f40c40a6c10 0x7f40c4118f60 secure :-1 s=THROTTLE_DONE pgs=2242 cs=0 l=1 rx=0x7f40b40032b0 tx=0x7f40b4010d30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:08.774 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:08.758+0000 7f40abfff700 1 --2- 172.21.15.42:0/3541131046 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f40a403b4a0 0x7f40a403d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:12:08.774 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:08.758+0000 7f40abfff700 1 -- 172.21.15.42:0/3541131046 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 10073+0+0 (secure 0 0 0) 0x7f40b404ba20 con 0x7f40c40a6c10 2019-12-05T06:12:08.775 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:08.758+0000 7f40c2ffd700 1 --2- 172.21.15.42:0/3541131046 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f40a403b4a0 0x7f40a403d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:12:08.775 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:08.758+0000 7f40c2ffd700 1 --2- 172.21.15.42:0/3541131046 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f40a403b4a0 0x7f40a403d9b0 crc :-1 s=READY pgs=941 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:12:08.775 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:08.758+0000 7f40c9a46700 1 -- 172.21.15.42:0/3541131046 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40c405d110 con 0x7f40c40a6c10 2019-12-05T06:12:08.775 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:08.762+0000 7f40c37fe700 1 --2- 172.21.15.42:0/3541131046 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f40c40a6c10 0x7f40c4118f60 secure :-1 s=THROTTLE_DONE pgs=2242 cs=0 l=1 rx=0x7f40b40032b0 tx=0x7f40b4010d30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:08.775 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:08.762+0000 7f40abfff700 1 -- 172.21.15.42:0/3541131046 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f40b406a400 con 0x7f40c40a6c10 2019-12-05T06:12:08.934 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:08.930+0000 7f40c9a46700 1 -- 172.21.15.42:0/3541131046 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f40c405b8c0 con 0x7f40c40a6c10 2019-12-05T06:12:08.935 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:08.930+0000 7f40c37fe700 1 --2- 172.21.15.42:0/3541131046 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f40c40a6c10 0x7f40c4118f60 secure :-1 s=THROTTLE_DONE pgs=2242 cs=0 l=1 rx=0x7f40b40032b0 tx=0x7f40b4010d30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:08.935 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:08.930+0000 7f40abfff700 1 -- 172.21.15.42:0/3541131046 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 268 v268) v1 ==== 95+0+2115 (secure 0 0 0) 0x7f40b4032aa0 con 0x7f40c40a6c10 2019-12-05T06:12:08.936 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 268 2019-12-05T06:12:08.936 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:12:08.936 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":268,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7094,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/3785523496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3785523496},{"type":"v1","addr":"172.21.15.40:6833","nonce":3785523496}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":268}],"filesystems":[{"mdsmap":{"epoch":267,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:11:34.953298+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":96,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10000},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10000":{"gid":10000,"name":"a","rank":0,"incarnation":264,"state":"up:active","state_seq":21,"addr":"172.21.15.42:6835/3923117701","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3923117701},{"type":"v1","addr":"172.21.15.42:6835","nonce":3923117701}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:12:08.938 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:08.934+0000 7f40c9a46700 1 -- 172.21.15.42:0/3541131046 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f40a403b4a0 msgr2=0x7f40a403d9b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:08.939 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:08.934+0000 7f40c9a46700 1 --2- 172.21.15.42:0/3541131046 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f40a403b4a0 0x7f40a403d9b0 crc :-1 s=READY pgs=941 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:08.940 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:08.934+0000 7f40c9a46700 1 -- 172.21.15.42:0/3541131046 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f40c40a6c10 msgr2=0x7f40c4118f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:08.941 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:08.934+0000 7f40c9a46700 1 --2- 172.21.15.42:0/3541131046 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f40c40a6c10 0x7f40c4118f60 secure :-1 s=READY pgs=2242 cs=0 l=1 rx=0x7f40b40032b0 tx=0x7f40b4010d30).stop 2019-12-05T06:12:08.941 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:08.934+0000 7f40c9a46700 1 -- 172.21.15.42:0/3541131046 shutdown_connections 2019-12-05T06:12:08.941 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:08.934+0000 7f40c9a46700 1 --2- 172.21.15.42:0/3541131046 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f40a403b4a0 0x7f40a403d9b0 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:08.941 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:08.934+0000 7f40c9a46700 1 --2- 172.21.15.42:0/3541131046 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f40c40a6c10 0x7f40c4118f60 unknown :-1 s=CLOSED pgs=2242 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:08.942 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:08.934+0000 7f40c9a46700 1 -- 172.21.15.42:0/3541131046 shutdown_connections 2019-12-05T06:12:08.942 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:08.934+0000 7f40c9a46700 1 -- 172.21.15.42:0/3541131046 wait complete. 2019-12-05T06:12:08.942 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:08.934+0000 7f40c9a46700 1 -- 172.21.15.42:0/3541131046 >> 172.21.15.42:0/3541131046 conn(0x7f40c40a9790 msgr2=0x7f40c4059ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:12:08.953 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:12:10.953 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:12:11.057 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.046+0000 7efc91004700 1 Processor -- start 2019-12-05T06:12:11.057 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.046+0000 7efc91004700 1 -- start start 2019-12-05T06:12:11.057 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.046+0000 7efc91004700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efc8c08df80 0x7efc8c09f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:12:11.057 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.046+0000 7efc91004700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7efc8c09f830 0x7efc8c14efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:12:11.058 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.046+0000 7efc91004700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc8c09fc60 con 0x7efc8c0a7740 2019-12-05T06:12:11.058 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.046+0000 7efc91004700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc8c0a0b00 con 0x7efc8c09f830 2019-12-05T06:12:11.058 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.046+0000 7efc91004700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc8c08e5b0 con 0x7efc8c08df80 2019-12-05T06:12:11.058 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.046+0000 7efc8b59e700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7efc8c0a7740 0x7efc8c07e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:50424/0 (socket says 172.21.15.42:50424) 2019-12-05T06:12:11.058 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.046+0000 7efc8b59e700 1 -- 172.21.15.42:0/3419541343 learned_addr learned my addr 172.21.15.42:0/3419541343 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:12:11.058 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.046+0000 7efc90803700 1 --2- 172.21.15.42:0/3419541343 >> v2:172.21.15.42:3300/0 conn(0x7efc8c09f830 0x7efc8c14efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:12:11.059 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.046+0000 7efc83fff700 1 --2- 172.21.15.42:0/3419541343 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efc8c08df80 0x7efc8c09f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:12:11.059 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.050+0000 7efc90803700 1 -- 172.21.15.42:0/3419541343 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efc8c08df80 msgr2=0x7efc8c09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:12:11.059 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.050+0000 7efc90803700 1 --2- 172.21.15.42:0/3419541343 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efc8c08df80 0x7efc8c09f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:12:11.060 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.050+0000 7efc90803700 1 -- 172.21.15.42:0/3419541343 >> v1:172.21.15.42:6789/0 conn(0x7efc8c0a7740 legacy=0x7efc8c07e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:11.060 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.050+0000 7efc90803700 1 -- 172.21.15.42:0/3419541343 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc8c0a0710 con 0x7efc8c09f830 2019-12-05T06:12:11.060 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.050+0000 7efc90803700 1 --2- 172.21.15.42:0/3419541343 >> v2:172.21.15.42:3300/0 conn(0x7efc8c09f830 0x7efc8c14efe0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7efc84008b10 tx=0x7efc8400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:11.060 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.050+0000 7efc90803700 1 --2- 172.21.15.42:0/3419541343 >> v2:172.21.15.42:3300/0 conn(0x7efc8c09f830 0x7efc8c14efe0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7efc84008b10 tx=0x7efc8400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:11.060 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.050+0000 7efc90803700 1 --2- 172.21.15.42:0/3419541343 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efc8c09f830 0x7efc8c14efe0 secure :-1 s=READY pgs=2244 cs=0 l=1 rx=0x7efc84008b10 tx=0x7efc8400fff0).ready entity=mon.1 client_cookie=5af9892c7d54d047 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:12:11.061 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.050+0000 7efc90803700 1 --2- 172.21.15.42:0/3419541343 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efc8c09f830 0x7efc8c14efe0 secure :-1 s=THROTTLE_DONE pgs=2244 cs=0 l=1 rx=0x7efc84008b10 tx=0x7efc8400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:11.061 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.050+0000 7efc8ad9d700 1 -- 172.21.15.42:0/3419541343 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efc84010900 con 0x7efc8c09f830 2019-12-05T06:12:11.061 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.050+0000 7efc90803700 1 --2- 172.21.15.42:0/3419541343 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efc8c09f830 0x7efc8c14efe0 secure :-1 s=THROTTLE_DONE pgs=2244 cs=0 l=1 rx=0x7efc84008b10 tx=0x7efc8400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:11.061 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.050+0000 7efc8ad9d700 1 -- 172.21.15.42:0/3419541343 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc84008b50 con 0x7efc8c09f830 2019-12-05T06:12:11.061 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.050+0000 7efc90803700 1 --2- 172.21.15.42:0/3419541343 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efc8c09f830 0x7efc8c14efe0 secure :-1 s=THROTTLE_DONE pgs=2244 cs=0 l=1 rx=0x7efc84008b10 tx=0x7efc8400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:11.061 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.050+0000 7efc8ad9d700 1 -- 172.21.15.42:0/3419541343 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efc840110c0 con 0x7efc8c09f830 2019-12-05T06:12:11.062 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.050+0000 7efc91004700 1 -- 172.21.15.42:0/3419541343 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efc8c09f830 msgr2=0x7efc8c14efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:11.062 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.050+0000 7efc91004700 1 --2- 172.21.15.42:0/3419541343 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efc8c09f830 0x7efc8c14efe0 secure :-1 s=READY pgs=2244 cs=0 l=1 rx=0x7efc84008b10 tx=0x7efc8400fff0).stop 2019-12-05T06:12:11.062 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.050+0000 7efc91004700 1 -- 172.21.15.42:0/3419541343 shutdown_connections 2019-12-05T06:12:11.063 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.050+0000 7efc91004700 1 --2- 172.21.15.42:0/3419541343 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efc8c09f830 0x7efc8c14efe0 unknown :-1 s=CLOSED pgs=2244 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:11.063 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.050+0000 7efc91004700 1 --2- 172.21.15.42:0/3419541343 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efc8c08df80 0x7efc8c09f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:12:11.063 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.050+0000 7efc91004700 1 -- 172.21.15.42:0/3419541343 shutdown_connections 2019-12-05T06:12:11.064 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.050+0000 7efc91004700 1 -- 172.21.15.42:0/3419541343 wait complete. 2019-12-05T06:12:11.064 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.050+0000 7efc91004700 1 -- 172.21.15.42:0/3419541343 >> 172.21.15.42:0/3419541343 conn(0x7efc8c0aabe0 msgr2=0x7efc8c0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:12:11.064 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.054+0000 7efc91004700 1 Processor -- start 2019-12-05T06:12:11.064 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.054+0000 7efc91004700 1 -- start start 2019-12-05T06:12:11.069 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.054+0000 7efc91004700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efc8c11aa90 0x7efc8c11aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:12:11.070 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.054+0000 7efc91004700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc8c131430 con 0x7efc8c1449b0 2019-12-05T06:12:11.070 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.054+0000 7efc91004700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc8c122970 con 0x7efc8c0722c0 2019-12-05T06:12:11.071 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.054+0000 7efc91004700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc8c11b410 con 0x7efc8c11aa90 2019-12-05T06:12:11.071 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.054+0000 7efc90803700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7efc8c0722c0 0x7efc8c074700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:50434/0 (socket says 172.21.15.42:50434) 2019-12-05T06:12:11.071 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.054+0000 7efc90803700 1 -- 172.21.15.42:0/2463306502 learned_addr learned my addr 172.21.15.42:0/2463306502 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:12:11.072 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.054+0000 7efc83fff700 1 --2- 172.21.15.42:0/2463306502 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efc8c11aa90 0x7efc8c11aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:12:11.072 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.054+0000 7efc88d99700 1 -- 172.21.15.42:0/2463306502 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 44887558 0 0) 0x7efc8c122970 con 0x7efc8c0722c0 2019-12-05T06:12:11.072 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.054+0000 7efc88d99700 1 -- 172.21.15.42:0/2463306502 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc700030b0 con 0x7efc8c0722c0 2019-12-05T06:12:11.072 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.054+0000 7efc88d99700 1 -- 172.21.15.42:0/2463306502 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4112125325 0 0) 0x7efc8c131430 con 0x7efc8c1449b0 2019-12-05T06:12:11.073 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.054+0000 7efc83fff700 1 -- 172.21.15.42:0/2463306502 >> v1:172.21.15.42:6789/0 conn(0x7efc8c0722c0 legacy=0x7efc8c074700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:11.073 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.054+0000 7efc83fff700 1 -- 172.21.15.42:0/2463306502 >> v1:172.21.15.40:6789/0 conn(0x7efc8c1449b0 legacy=0x7efc8c1199d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:11.074 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.054+0000 7efc83fff700 1 -- 172.21.15.42:0/2463306502 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc8c077f30 con 0x7efc8c11aa90 2019-12-05T06:12:11.074 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.054+0000 7efc83fff700 1 --2- 172.21.15.42:0/2463306502 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efc8c11aa90 0x7efc8c11aec0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7efc74002f30 tx=0x7efc74010b20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:11.074 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.054+0000 7efc83fff700 1 --2- 172.21.15.42:0/2463306502 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efc8c11aa90 0x7efc8c11aec0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7efc74002f30 tx=0x7efc74010b20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:11.075 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.054+0000 7efc83fff700 1 --2- 172.21.15.42:0/2463306502 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efc8c11aa90 0x7efc8c11aec0 secure :-1 s=READY pgs=490 cs=0 l=1 rx=0x7efc74002f30 tx=0x7efc74010b20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:12:11.075 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.058+0000 7efc83fff700 1 --2- 172.21.15.42:0/2463306502 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efc8c11aa90 0x7efc8c11aec0 secure :-1 s=THROTTLE_DONE pgs=490 cs=0 l=1 rx=0x7efc74002f30 tx=0x7efc74010b20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:11.076 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.058+0000 7efc88d99700 1 -- 172.21.15.42:0/2463306502 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efc74005540 con 0x7efc8c11aa90 2019-12-05T06:12:11.076 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.058+0000 7efc83fff700 1 --2- 172.21.15.42:0/2463306502 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efc8c11aa90 0x7efc8c11aec0 secure :-1 s=THROTTLE_DONE pgs=490 cs=0 l=1 rx=0x7efc74002f30 tx=0x7efc74010b20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:11.076 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.058+0000 7efc83fff700 1 --2- 172.21.15.42:0/2463306502 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efc8c11aa90 0x7efc8c11aec0 secure :-1 s=THROTTLE_DONE pgs=490 cs=0 l=1 rx=0x7efc74002f30 tx=0x7efc74010b20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:11.077 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.058+0000 7efc88d99700 1 -- 172.21.15.42:0/2463306502 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc74007150 con 0x7efc8c11aa90 2019-12-05T06:12:11.077 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.058+0000 7efc91004700 1 -- 172.21.15.42:0/2463306502 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc8c075f10 con 0x7efc8c11aa90 2019-12-05T06:12:11.078 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.058+0000 7efc88d99700 1 -- 172.21.15.42:0/2463306502 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efc74005b20 con 0x7efc8c11aa90 2019-12-05T06:12:11.078 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.058+0000 7efc91004700 1 -- 172.21.15.42:0/2463306502 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc8c076440 con 0x7efc8c11aa90 2019-12-05T06:12:11.078 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.058+0000 7efc83fff700 1 --2- 172.21.15.42:0/2463306502 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efc8c11aa90 0x7efc8c11aec0 secure :-1 s=THROTTLE_DONE pgs=490 cs=0 l=1 rx=0x7efc74002f30 tx=0x7efc74010b20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:11.079 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.058+0000 7efc88d99700 1 -- 172.21.15.42:0/2463306502 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7efc740123c0 con 0x7efc8c11aa90 2019-12-05T06:12:11.079 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.058+0000 7efc83fff700 1 --2- 172.21.15.42:0/2463306502 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efc8c11aa90 0x7efc8c11aec0 secure :-1 s=THROTTLE_DONE pgs=490 cs=0 l=1 rx=0x7efc74002f30 tx=0x7efc74010b20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:11.079 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.058+0000 7efc88d99700 1 --2- 172.21.15.42:0/2463306502 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7efc7003b1e0 0x7efc7003d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:12:11.079 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.058+0000 7efc88d99700 1 -- 172.21.15.42:0/2463306502 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 10159+0+0 (secure 0 0 0) 0x7efc7404adf0 con 0x7efc8c11aa90 2019-12-05T06:12:11.080 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.058+0000 7efc8b59e700 1 --2- 172.21.15.42:0/2463306502 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7efc7003b1e0 0x7efc7003d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:12:11.080 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.058+0000 7efc8b59e700 1 --2- 172.21.15.42:0/2463306502 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7efc7003b1e0 0x7efc7003d6f0 crc :-1 s=READY pgs=942 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:12:11.081 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.062+0000 7efc91004700 1 -- 172.21.15.42:0/2463306502 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc8c05d110 con 0x7efc8c11aa90 2019-12-05T06:12:11.082 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.070+0000 7efc83fff700 1 --2- 172.21.15.42:0/2463306502 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efc8c11aa90 0x7efc8c11aec0 secure :-1 s=THROTTLE_DONE pgs=490 cs=0 l=1 rx=0x7efc74002f30 tx=0x7efc74010b20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:11.082 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.070+0000 7efc88d99700 1 -- 172.21.15.42:0/2463306502 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7efc74031e70 con 0x7efc8c11aa90 2019-12-05T06:12:11.249 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.238+0000 7efc91004700 1 -- 172.21.15.42:0/2463306502 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efc8c05b8c0 con 0x7efc8c11aa90 2019-12-05T06:12:11.249 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.238+0000 7efc83fff700 1 --2- 172.21.15.42:0/2463306502 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efc8c11aa90 0x7efc8c11aec0 secure :-1 s=THROTTLE_DONE pgs=490 cs=0 l=1 rx=0x7efc74002f30 tx=0x7efc74010b20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:11.249 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.238+0000 7efc88d99700 1 -- 172.21.15.42:0/2463306502 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 269 v269) v1 ==== 95+0+1774 (secure 0 0 0) 0x7efc740320d0 con 0x7efc8c11aa90 2019-12-05T06:12:11.249 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 269 2019-12-05T06:12:11.250 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.238+0000 7efc91004700 1 -- 172.21.15.42:0/2463306502 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7efc7003b1e0 msgr2=0x7efc7003d6f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:11.250 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.238+0000 7efc91004700 1 --2- 172.21.15.42:0/2463306502 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7efc7003b1e0 0x7efc7003d6f0 crc :-1 s=READY pgs=942 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:11.250 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.238+0000 7efc91004700 1 -- 172.21.15.42:0/2463306502 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efc8c11aa90 msgr2=0x7efc8c11aec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:11.250 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.238+0000 7efc91004700 1 --2- 172.21.15.42:0/2463306502 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efc8c11aa90 0x7efc8c11aec0 secure :-1 s=READY pgs=490 cs=0 l=1 rx=0x7efc74002f30 tx=0x7efc74010b20).stop 2019-12-05T06:12:11.250 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.242+0000 7efc91004700 1 -- 172.21.15.42:0/2463306502 shutdown_connections 2019-12-05T06:12:11.250 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.242+0000 7efc91004700 1 --2- 172.21.15.42:0/2463306502 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7efc7003b1e0 0x7efc7003d6f0 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:11.250 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.242+0000 7efc91004700 1 --2- 172.21.15.42:0/2463306502 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efc8c11aa90 0x7efc8c11aec0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:11.251 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.242+0000 7efc91004700 1 -- 172.21.15.42:0/2463306502 shutdown_connections 2019-12-05T06:12:11.251 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.242+0000 7efc91004700 1 -- 172.21.15.42:0/2463306502 wait complete. 2019-12-05T06:12:11.251 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.242+0000 7efc91004700 1 -- 172.21.15.42:0/2463306502 >> 172.21.15.42:0/2463306502 conn(0x7efc8c0a9c20 msgr2=0x7efc8c09f1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:12:11.251 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:12:11.252 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":269,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":269,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:12:09.584141+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":97,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7094},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7094":{"gid":7094,"name":"b","rank":0,"incarnation":269,"state":"up:replay","state_seq":2,"addr":"172.21.15.40:6833/3785523496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3785523496},{"type":"v1","addr":"172.21.15.40:6833","nonce":3785523496}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:12:11.254 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a down, removed from mdsmap 2019-12-05T06:12:11.254 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2019-12-05T06:12:11.254 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:12:11.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.342+0000 7f1b5e82e700 1 Processor -- start 2019-12-05T06:12:11.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.342+0000 7f1b5e82e700 1 -- start start 2019-12-05T06:12:11.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.346+0000 7f1b5e82e700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1b5808df80 0x7f1b5809f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:12:11.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.346+0000 7f1b5e82e700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b5809fc60 con 0x7f1b5809f830 2019-12-05T06:12:11.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.346+0000 7f1b5e82e700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b580a0710 con 0x7f1b580a7740 2019-12-05T06:12:11.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.346+0000 7f1b5e82e700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b580a0b00 con 0x7f1b5808df80 2019-12-05T06:12:11.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.346+0000 7f1b5cdcb700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f1b580a7740 0x7f1b5807e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:50438/0 (socket says 172.21.15.42:50438) 2019-12-05T06:12:11.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.346+0000 7f1b5cdcb700 1 -- 172.21.15.42:0/3703563593 learned_addr learned my addr 172.21.15.42:0/3703563593 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:12:11.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.346+0000 7f1b57fff700 1 --2- 172.21.15.42:0/3703563593 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1b5808df80 0x7f1b5809f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:12:11.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.346+0000 7f1b577fe700 1 -- 172.21.15.42:0/3703563593 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4143627126 0 0) 0x7f1b580a0710 con 0x7f1b580a7740 2019-12-05T06:12:11.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.346+0000 7f1b577fe700 1 -- 172.21.15.42:0/3703563593 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b440030b0 con 0x7f1b580a7740 2019-12-05T06:12:11.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.346+0000 7f1b577fe700 1 -- 172.21.15.42:0/3703563593 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 318346719 0 0) 0x7f1b5809fc60 con 0x7f1b5809f830 2019-12-05T06:12:11.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.346+0000 7f1b577fe700 1 -- 172.21.15.42:0/3703563593 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b580a0710 con 0x7f1b5809f830 2019-12-05T06:12:11.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.346+0000 7f1b57fff700 1 -- 172.21.15.42:0/3703563593 >> v1:172.21.15.42:6789/0 conn(0x7f1b580a7740 legacy=0x7f1b5807e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:11.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.346+0000 7f1b57fff700 1 -- 172.21.15.42:0/3703563593 >> v1:172.21.15.40:6789/0 conn(0x7f1b5809f830 legacy=0x7f1b5814efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:11.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.346+0000 7f1b57fff700 1 -- 172.21.15.42:0/3703563593 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b5807ef20 con 0x7f1b5808df80 2019-12-05T06:12:11.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.346+0000 7f1b57fff700 1 --2- 172.21.15.42:0/3703563593 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1b5808df80 0x7f1b5809f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f1b400032b0 tx=0x7f1b4000ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:11.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.346+0000 7f1b57fff700 1 --2- 172.21.15.42:0/3703563593 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1b5808df80 0x7f1b5809f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f1b400032b0 tx=0x7f1b4000ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:11.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.346+0000 7f1b57fff700 1 --2- 172.21.15.42:0/3703563593 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1b5808df80 0x7f1b5809f2e0 secure :-1 s=READY pgs=491 cs=0 l=1 rx=0x7f1b400032b0 tx=0x7f1b4000ff70).ready entity=mon.2 client_cookie=c32147ce6f608430 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:12:11.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.346+0000 7f1b57fff700 1 --2- 172.21.15.42:0/3703563593 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1b5808df80 0x7f1b5809f2e0 secure :-1 s=THROTTLE_DONE pgs=491 cs=0 l=1 rx=0x7f1b400032b0 tx=0x7f1b4000ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:11.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.346+0000 7f1b577fe700 1 -- 172.21.15.42:0/3703563593 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1b40011a10 con 0x7f1b5808df80 2019-12-05T06:12:11.358 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.346+0000 7f1b57fff700 1 --2- 172.21.15.42:0/3703563593 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1b5808df80 0x7f1b5809f2e0 secure :-1 s=THROTTLE_DONE pgs=491 cs=0 l=1 rx=0x7f1b400032b0 tx=0x7f1b4000ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:11.358 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.346+0000 7f1b57fff700 1 --2- 172.21.15.42:0/3703563593 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1b5808df80 0x7f1b5809f2e0 secure :-1 s=THROTTLE_DONE pgs=491 cs=0 l=1 rx=0x7f1b400032b0 tx=0x7f1b4000ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:11.358 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.346+0000 7f1b577fe700 1 -- 172.21.15.42:0/3703563593 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b40011630 con 0x7f1b5808df80 2019-12-05T06:12:11.358 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.346+0000 7f1b577fe700 1 -- 172.21.15.42:0/3703563593 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1b4000a160 con 0x7f1b5808df80 2019-12-05T06:12:11.358 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.346+0000 7f1b5e82e700 1 -- 172.21.15.42:0/3703563593 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1b5808df80 msgr2=0x7f1b5809f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:11.358 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.346+0000 7f1b5e82e700 1 --2- 172.21.15.42:0/3703563593 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1b5808df80 0x7f1b5809f2e0 secure :-1 s=READY pgs=491 cs=0 l=1 rx=0x7f1b400032b0 tx=0x7f1b4000ff70).stop 2019-12-05T06:12:11.359 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.346+0000 7f1b5e82e700 1 -- 172.21.15.42:0/3703563593 shutdown_connections 2019-12-05T06:12:11.359 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.346+0000 7f1b5e82e700 1 --2- 172.21.15.42:0/3703563593 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1b5808df80 0x7f1b5809f2e0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:11.359 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.346+0000 7f1b5e82e700 1 -- 172.21.15.42:0/3703563593 shutdown_connections 2019-12-05T06:12:11.359 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.346+0000 7f1b5e82e700 1 -- 172.21.15.42:0/3703563593 wait complete. 2019-12-05T06:12:11.359 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.346+0000 7f1b5e82e700 1 -- 172.21.15.42:0/3703563593 >> 172.21.15.42:0/3703563593 conn(0x7f1b580aabe0 msgr2=0x7f1b580a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:12:11.359 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.350+0000 7f1b5e82e700 1 Processor -- start 2019-12-05T06:12:11.359 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.350+0000 7f1b5e82e700 1 -- start start 2019-12-05T06:12:11.363 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.350+0000 7f1b5e82e700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1b581442d0 0x7f1b580a33f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:12:11.363 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.350+0000 7f1b5e82e700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f1b5807b900 0x7f1b5807bd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:12:11.363 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.350+0000 7f1b5e82e700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b580a2090 con 0x7f1b581418e0 2019-12-05T06:12:11.363 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.350+0000 7f1b5e82e700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b5807c280 con 0x7f1b5807b900 2019-12-05T06:12:11.364 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.350+0000 7f1b5e82e700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b58144700 con 0x7f1b581442d0 2019-12-05T06:12:11.364 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.350+0000 7f1b57fff700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1b581442d0 0x7f1b580a33f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:12:11.364 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.350+0000 7f1b57fff700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1b581442d0 0x7f1b580a33f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:50874/0 (socket says 172.21.15.42:50874) 2019-12-05T06:12:11.364 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.350+0000 7f1b57fff700 1 -- 172.21.15.42:0/1915187157 learned_addr learned my addr 172.21.15.42:0/1915187157 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:12:11.365 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.350+0000 7f1b5d5cc700 1 --2- 172.21.15.42:0/1915187157 >> v2:172.21.15.40:3300/0 conn(0x7f1b5807b900 0x7f1b5807bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:12:11.365 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.350+0000 7f1b57fff700 1 -- 172.21.15.42:0/1915187157 >> v2:172.21.15.40:3300/0 conn(0x7f1b5807b900 msgr2=0x7f1b5807bd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:11.365 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.350+0000 7f1b57fff700 1 --2- 172.21.15.42:0/1915187157 >> v2:172.21.15.40:3300/0 conn(0x7f1b5807b900 0x7f1b5807bd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:11.365 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.350+0000 7f1b57fff700 1 -- 172.21.15.42:0/1915187157 >> v1:172.21.15.40:6789/0 conn(0x7f1b581418e0 legacy=0x7f1b580a16e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:11.365 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.350+0000 7f1b57fff700 1 -- 172.21.15.42:0/1915187157 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b580a4450 con 0x7f1b581442d0 2019-12-05T06:12:11.365 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.350+0000 7f1b57fff700 1 --2- 172.21.15.42:0/1915187157 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1b581442d0 0x7f1b580a33f0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f1b400032b0 tx=0x7f1b40011e00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:11.366 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.354+0000 7f1b57fff700 1 --2- 172.21.15.42:0/1915187157 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1b581442d0 0x7f1b580a33f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f1b400032b0 tx=0x7f1b40011e00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:11.366 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.354+0000 7f1b57fff700 1 --2- 172.21.15.42:0/1915187157 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1b581442d0 0x7f1b580a33f0 secure :-1 s=READY pgs=492 cs=0 l=1 rx=0x7f1b400032b0 tx=0x7f1b40011e00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:12:11.366 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.354+0000 7f1b57fff700 1 --2- 172.21.15.42:0/1915187157 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1b581442d0 0x7f1b580a33f0 secure :-1 s=THROTTLE_DONE pgs=492 cs=0 l=1 rx=0x7f1b400032b0 tx=0x7f1b40011e00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:11.366 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.354+0000 7f1b57fff700 1 --2- 172.21.15.42:0/1915187157 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1b581442d0 0x7f1b580a33f0 secure :-1 s=THROTTLE_DONE pgs=492 cs=0 l=1 rx=0x7f1b400032b0 tx=0x7f1b40011e00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:11.366 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.354+0000 7f1b57fff700 1 --2- 172.21.15.42:0/1915187157 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1b581442d0 0x7f1b580a33f0 secure :-1 s=THROTTLE_DONE pgs=492 cs=0 l=1 rx=0x7f1b400032b0 tx=0x7f1b40011e00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:11.366 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.354+0000 7f1b557fa700 1 -- 172.21.15.42:0/1915187157 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1b40005ab0 con 0x7f1b581442d0 2019-12-05T06:12:11.367 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.354+0000 7f1b557fa700 1 -- 172.21.15.42:0/1915187157 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b40011c10 con 0x7f1b581442d0 2019-12-05T06:12:11.367 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.354+0000 7f1b5e82e700 1 -- 172.21.15.42:0/1915187157 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b5812a9b0 con 0x7f1b581442d0 2019-12-05T06:12:11.367 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.354+0000 7f1b557fa700 1 -- 172.21.15.42:0/1915187157 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1b4000f550 con 0x7f1b581442d0 2019-12-05T06:12:11.367 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.354+0000 7f1b5e82e700 1 -- 172.21.15.42:0/1915187157 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b5812aee0 con 0x7f1b581442d0 2019-12-05T06:12:11.367 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.354+0000 7f1b57fff700 1 --2- 172.21.15.42:0/1915187157 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1b581442d0 0x7f1b580a33f0 secure :-1 s=THROTTLE_DONE pgs=492 cs=0 l=1 rx=0x7f1b400032b0 tx=0x7f1b40011e00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:11.367 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.354+0000 7f1b557fa700 1 -- 172.21.15.42:0/1915187157 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f1b4000f860 con 0x7f1b581442d0 2019-12-05T06:12:11.367 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.354+0000 7f1b57fff700 1 --2- 172.21.15.42:0/1915187157 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1b581442d0 0x7f1b580a33f0 secure :-1 s=THROTTLE_DONE pgs=492 cs=0 l=1 rx=0x7f1b400032b0 tx=0x7f1b40011e00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:11.368 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.354+0000 7f1b557fa700 1 --2- 172.21.15.42:0/1915187157 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f1b3c03ad10 0x7f1b3c03d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:12:11.368 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.354+0000 7f1b557fa700 1 -- 172.21.15.42:0/1915187157 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 10159+0+0 (secure 0 0 0) 0x7f1b4004b680 con 0x7f1b581442d0 2019-12-05T06:12:11.368 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.354+0000 7f1b5cdcb700 1 --2- 172.21.15.42:0/1915187157 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f1b3c03ad10 0x7f1b3c03d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:12:11.368 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.354+0000 7f1b5cdcb700 1 --2- 172.21.15.42:0/1915187157 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f1b3c03ad10 0x7f1b3c03d220 crc :-1 s=READY pgs=943 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:12:11.369 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.358+0000 7f1b5e82e700 1 -- 172.21.15.42:0/1915187157 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b5805d110 con 0x7f1b581442d0 2019-12-05T06:12:11.386 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.366+0000 7f1b57fff700 1 --2- 172.21.15.42:0/1915187157 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1b581442d0 0x7f1b580a33f0 secure :-1 s=THROTTLE_DONE pgs=492 cs=0 l=1 rx=0x7f1b400032b0 tx=0x7f1b40011e00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:11.386 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.366+0000 7f1b557fa700 1 -- 172.21.15.42:0/1915187157 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f1b40033510 con 0x7f1b581442d0 2019-12-05T06:12:11.533 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.530+0000 7f1b5e82e700 1 -- 172.21.15.42:0/1915187157 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1b5805b8c0 con 0x7f1b581442d0 2019-12-05T06:12:11.535 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.530+0000 7f1b57fff700 1 --2- 172.21.15.42:0/1915187157 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1b581442d0 0x7f1b580a33f0 secure :-1 s=THROTTLE_DONE pgs=492 cs=0 l=1 rx=0x7f1b400032b0 tx=0x7f1b40011e00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:11.535 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.530+0000 7f1b557fa700 1 -- 172.21.15.42:0/1915187157 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 269 v269) v1 ==== 95+0+1774 (secure 0 0 0) 0x7f1b40033ff0 con 0x7f1b581442d0 2019-12-05T06:12:11.535 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 269 2019-12-05T06:12:11.535 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:12:11.536 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":269,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":269,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:12:09.584141+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":97,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7094},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7094":{"gid":7094,"name":"b","rank":0,"incarnation":269,"state":"up:replay","state_seq":2,"addr":"172.21.15.40:6833/3785523496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3785523496},{"type":"v1","addr":"172.21.15.40:6833","nonce":3785523496}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:12:11.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.534+0000 7f1b5e82e700 1 -- 172.21.15.42:0/1915187157 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f1b3c03ad10 msgr2=0x7f1b3c03d220 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:11.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.534+0000 7f1b5e82e700 1 --2- 172.21.15.42:0/1915187157 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f1b3c03ad10 0x7f1b3c03d220 crc :-1 s=READY pgs=943 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:11.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.534+0000 7f1b5e82e700 1 -- 172.21.15.42:0/1915187157 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1b581442d0 msgr2=0x7f1b580a33f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:11.539 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.534+0000 7f1b5e82e700 1 --2- 172.21.15.42:0/1915187157 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1b581442d0 0x7f1b580a33f0 secure :-1 s=READY pgs=492 cs=0 l=1 rx=0x7f1b400032b0 tx=0x7f1b40011e00).stop 2019-12-05T06:12:11.548 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.534+0000 7f1b5e82e700 1 -- 172.21.15.42:0/1915187157 shutdown_connections 2019-12-05T06:12:11.549 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.534+0000 7f1b5e82e700 1 --2- 172.21.15.42:0/1915187157 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f1b3c03ad10 0x7f1b3c03d220 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:11.549 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.534+0000 7f1b5e82e700 1 --2- 172.21.15.42:0/1915187157 >> v2:172.21.15.40:3300/0 conn(0x7f1b5807b900 0x7f1b5807bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:11.549 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.534+0000 7f1b5e82e700 1 --2- 172.21.15.42:0/1915187157 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1b581442d0 0x7f1b580a33f0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:11.550 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.534+0000 7f1b5e82e700 1 -- 172.21.15.42:0/1915187157 shutdown_connections 2019-12-05T06:12:11.550 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.534+0000 7f1b5e82e700 1 -- 172.21.15.42:0/1915187157 wait complete. 2019-12-05T06:12:11.550 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:11.534+0000 7f1b5e82e700 1 -- 172.21.15.42:0/1915187157 >> 172.21.15.42:0/1915187157 conn(0x7f1b58115f50 msgr2=0x7f1b580a6a60 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:12:11.551 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 269, "feature_flags": { "enable_multiple": false, "ever_enabled_multiple": false }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2019-12-05T05:03:35.604738+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 269, "ever_allowed_features": 2, "explicitly_allowed_features": 2, "failed": [], "flags": 18, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_7094": { "addr": "172.21.15.40:6833/3785523496", "addrs": { "addrvec": [ { "addr": "172.21.15.40:6832", "nonce": 3785523496, "type": "v2" }, { "addr": "172.21.15.40:6833", "nonce": 3785523496, "type": "v1" } ] }, "export_targets": [], "features": 4540138292836696063, "flags": 0, "gid": 7094, "incarnation": 269, "name": "b", "rank": 0, "state": "up:replay", "state_seq": 2 } }, "last_failure": 0, "last_failure_osd_epoch": 97, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 2, "min_compat_client": "0 (unknown)", "modified": "2019-12-05T06:12:09.584141+0000", "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7094 } } } ], "standbys": [] } 2019-12-05T06:12:13.552 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:12:13.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:13.646+0000 7f580a153700 1 Processor -- start 2019-12-05T06:12:13.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:13.646+0000 7f580a153700 1 -- start start 2019-12-05T06:12:13.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:13.646+0000 7f580a153700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f58040a7730 0x7f580408df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:12:13.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:13.646+0000 7f580a153700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f580407e710 0x7f580407eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:12:13.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:13.646+0000 7f580a153700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58040a0700 con 0x7f580409f2d0 2019-12-05T06:12:13.654 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:13.646+0000 7f580a153700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58040a0af0 con 0x7f580407e710 2019-12-05T06:12:13.654 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:13.646+0000 7f580a153700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f580408e5a0 con 0x7f58040a7730 2019-12-05T06:12:13.654 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:13.646+0000 7f58037fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f580407e710 0x7f580407eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:12:13.654 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:13.646+0000 7f58037fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f580407e710 0x7f580407eb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:50416/0 (socket says 172.21.15.42:50416) 2019-12-05T06:12:13.654 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:13.646+0000 7f58037fe700 1 -- 172.21.15.42:0/2008324524 learned_addr learned my addr 172.21.15.42:0/2008324524 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:12:13.654 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:13.646+0000 7f5803fff700 1 --2- 172.21.15.42:0/2008324524 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f58040a7730 0x7f580408df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:12:13.654 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:13.646+0000 7f58037fe700 1 -- 172.21.15.42:0/2008324524 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f58040a7730 msgr2=0x7f580408df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:12:13.655 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:13.646+0000 7f58037fe700 1 --2- 172.21.15.42:0/2008324524 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f58040a7730 0x7f580408df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:12:13.655 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:13.646+0000 7f58037fe700 1 -- 172.21.15.42:0/2008324524 >> v1:172.21.15.40:6789/0 conn(0x7f580409f2d0 legacy=0x7f580409f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:13.655 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:13.646+0000 7f58037fe700 1 -- 172.21.15.42:0/2008324524 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f580407f090 con 0x7f580407e710 2019-12-05T06:12:13.655 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:13.646+0000 7f5803fff700 1 --2- 172.21.15.42:0/2008324524 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f58040a7730 0x7f580408df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:12:13.655 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:13.646+0000 7f58037fe700 1 --2- 172.21.15.42:0/2008324524 >> v2:172.21.15.42:3300/0 conn(0x7f580407e710 0x7f580407eb40 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f57f4008790 tx=0x7f57f400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:13.655 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:13.646+0000 7f58037fe700 1 --2- 172.21.15.42:0/2008324524 >> v2:172.21.15.42:3300/0 conn(0x7f580407e710 0x7f580407eb40 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f57f4008790 tx=0x7f57f400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:13.655 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:13.646+0000 7f58037fe700 1 --2- 172.21.15.42:0/2008324524 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f580407e710 0x7f580407eb40 secure :-1 s=READY pgs=2247 cs=0 l=1 rx=0x7f57f4008790 tx=0x7f57f400fd30).ready entity=mon.1 client_cookie=6213456cb61a43b5 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:12:13.656 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:13.646+0000 7f58037fe700 1 --2- 172.21.15.42:0/2008324524 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f580407e710 0x7f580407eb40 secure :-1 s=THROTTLE_DONE pgs=2247 cs=0 l=1 rx=0x7f57f4008790 tx=0x7f57f400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:13.656 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:13.646+0000 7f5802ffd700 1 -- 172.21.15.42:0/2008324524 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f57f40113b0 con 0x7f580407e710 2019-12-05T06:12:13.656 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:13.646+0000 7f58037fe700 1 --2- 172.21.15.42:0/2008324524 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f580407e710 0x7f580407eb40 secure :-1 s=THROTTLE_DONE pgs=2247 cs=0 l=1 rx=0x7f57f4008790 tx=0x7f57f400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:13.656 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:13.646+0000 7f5802ffd700 1 -- 172.21.15.42:0/2008324524 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57f4008bb0 con 0x7f580407e710 2019-12-05T06:12:13.657 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:13.646+0000 7f58037fe700 1 --2- 172.21.15.42:0/2008324524 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f580407e710 0x7f580407eb40 secure :-1 s=THROTTLE_DONE pgs=2247 cs=0 l=1 rx=0x7f57f4008790 tx=0x7f57f400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:13.657 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:13.646+0000 7f5802ffd700 1 -- 172.21.15.42:0/2008324524 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f57f4011990 con 0x7f580407e710 2019-12-05T06:12:13.657 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:13.646+0000 7f580a153700 1 -- 172.21.15.42:0/2008324524 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f580407e710 msgr2=0x7f580407eb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:13.657 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:13.646+0000 7f580a153700 1 --2- 172.21.15.42:0/2008324524 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f580407e710 0x7f580407eb40 secure :-1 s=READY pgs=2247 cs=0 l=1 rx=0x7f57f4008790 tx=0x7f57f400fd30).stop 2019-12-05T06:12:13.657 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:13.650+0000 7f580a153700 1 -- 172.21.15.42:0/2008324524 shutdown_connections 2019-12-05T06:12:13.657 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:13.650+0000 7f580a153700 1 --2- 172.21.15.42:0/2008324524 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f58040a7730 0x7f580408df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:12:13.658 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:13.650+0000 7f580a153700 1 --2- 172.21.15.42:0/2008324524 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f580407e710 0x7f580407eb40 unknown :-1 s=CLOSED pgs=2247 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:13.658 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:13.650+0000 7f580a153700 1 -- 172.21.15.42:0/2008324524 shutdown_connections 2019-12-05T06:12:13.658 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:13.650+0000 7f580a153700 1 -- 172.21.15.42:0/2008324524 wait complete. 2019-12-05T06:12:13.658 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:13.650+0000 7f580a153700 1 -- 172.21.15.42:0/2008324524 >> 172.21.15.42:0/2008324524 conn(0x7f58040aabd0 msgr2=0x7f58040a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:12:13.661 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:13.650+0000 7f580a153700 1 Processor -- start 2019-12-05T06:12:13.661 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:13.650+0000 7f580a153700 1 -- start start 2019-12-05T06:12:13.661 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:13.650+0000 7f580a153700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f580414e5b0 0x7f58041280f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:12:13.661 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:13.650+0000 7f580a153700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f580407b750 0x7f58041359e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:12:13.662 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:13.650+0000 7f580a153700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58040a6420 con 0x7f5804079b80 2019-12-05T06:12:13.662 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:13.650+0000 7f580a153700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f580407a660 con 0x7f580414e5b0 2019-12-05T06:12:13.662 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:13.650+0000 7f580a153700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58040a6900 con 0x7f580407b750 2019-12-05T06:12:13.662 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:13.650+0000 7f58037fe700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f5804079b80 0x7f580407af50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:50464/0 (socket says 172.21.15.42:50464) 2019-12-05T06:12:13.662 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:13.650+0000 7f58037fe700 1 -- 172.21.15.42:0/1733225270 learned_addr learned my addr 172.21.15.42:0/1733225270 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:12:13.662 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:13.650+0000 7f5803fff700 1 --2- 172.21.15.42:0/1733225270 >> v2:172.21.15.40:3300/0 conn(0x7f580414e5b0 0x7f58041280f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:12:13.662 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:13.650+0000 7f5808ef1700 1 --2- 172.21.15.42:0/1733225270 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f580407b750 0x7f58041359e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:12:13.663 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:13.650+0000 7f5800ff9700 1 -- 172.21.15.42:0/1733225270 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2442862232 0 0) 0x7f58040a6420 con 0x7f5804079b80 2019-12-05T06:12:13.663 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:13.650+0000 7f5800ff9700 1 -- 172.21.15.42:0/1733225270 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57e80030b0 con 0x7f5804079b80 2019-12-05T06:12:13.663 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:13.650+0000 7f5808ef1700 1 -- 172.21.15.42:0/1733225270 >> v2:172.21.15.40:3300/0 conn(0x7f580414e5b0 msgr2=0x7f58041280f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:13.663 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:13.650+0000 7f5808ef1700 1 --2- 172.21.15.42:0/1733225270 >> v2:172.21.15.40:3300/0 conn(0x7f580414e5b0 0x7f58041280f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:13.663 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:13.650+0000 7f5808ef1700 1 -- 172.21.15.42:0/1733225270 >> v1:172.21.15.42:6789/0 conn(0x7f5804079b80 legacy=0x7f580407af50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:13.663 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:13.650+0000 7f5808ef1700 1 -- 172.21.15.42:0/1733225270 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5804115ec0 con 0x7f580407b750 2019-12-05T06:12:13.663 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:13.650+0000 7f5808ef1700 1 --2- 172.21.15.42:0/1733225270 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f580407b750 0x7f58041359e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f57fc001210 tx=0x7f57fc010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:13.664 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:13.650+0000 7f5808ef1700 1 --2- 172.21.15.42:0/1733225270 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f580407b750 0x7f58041359e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f57fc001210 tx=0x7f57fc010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:13.664 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:13.654+0000 7f5808ef1700 1 --2- 172.21.15.42:0/1733225270 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f580407b750 0x7f58041359e0 secure :-1 s=READY pgs=493 cs=0 l=1 rx=0x7f57fc001210 tx=0x7f57fc010b40).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:12:13.664 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:13.654+0000 7f5808ef1700 1 --2- 172.21.15.42:0/1733225270 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f580407b750 0x7f58041359e0 secure :-1 s=THROTTLE_DONE pgs=493 cs=0 l=1 rx=0x7f57fc001210 tx=0x7f57fc010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:13.664 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:13.654+0000 7f5800ff9700 1 -- 172.21.15.42:0/1733225270 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f57fc002240 con 0x7f580407b750 2019-12-05T06:12:13.664 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:13.654+0000 7f5808ef1700 1 --2- 172.21.15.42:0/1733225270 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f580407b750 0x7f58041359e0 secure :-1 s=THROTTLE_DONE pgs=493 cs=0 l=1 rx=0x7f57fc001210 tx=0x7f57fc010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:13.664 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:13.654+0000 7f580a153700 1 -- 172.21.15.42:0/1733225270 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5804128640 con 0x7f580407b750 2019-12-05T06:12:13.664 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:13.654+0000 7f5800ff9700 1 -- 172.21.15.42:0/1733225270 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57fc006ad0 con 0x7f580407b750 2019-12-05T06:12:13.665 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:13.654+0000 7f5808ef1700 1 --2- 172.21.15.42:0/1733225270 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f580407b750 0x7f58041359e0 secure :-1 s=THROTTLE_DONE pgs=493 cs=0 l=1 rx=0x7f57fc001210 tx=0x7f57fc010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:13.665 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:13.654+0000 7f580a153700 1 -- 172.21.15.42:0/1733225270 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5804135f30 con 0x7f580407b750 2019-12-05T06:12:13.665 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:13.654+0000 7f5800ff9700 1 -- 172.21.15.42:0/1733225270 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f57fc002670 con 0x7f580407b750 2019-12-05T06:12:13.665 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:13.654+0000 7f5808ef1700 1 --2- 172.21.15.42:0/1733225270 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f580407b750 0x7f58041359e0 secure :-1 s=THROTTLE_DONE pgs=493 cs=0 l=1 rx=0x7f57fc001210 tx=0x7f57fc010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:13.665 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:13.654+0000 7f5800ff9700 1 -- 172.21.15.42:0/1733225270 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f57fc002870 con 0x7f580407b750 2019-12-05T06:12:13.665 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:13.654+0000 7f5808ef1700 1 --2- 172.21.15.42:0/1733225270 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f580407b750 0x7f58041359e0 secure :-1 s=THROTTLE_DONE pgs=493 cs=0 l=1 rx=0x7f57fc001210 tx=0x7f57fc010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:13.666 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:13.654+0000 7f5800ff9700 1 --2- 172.21.15.42:0/1733225270 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f57e803b220 0x7f57e803d730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:12:13.666 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:13.654+0000 7f5800ff9700 1 -- 172.21.15.42:0/1733225270 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 10159+0+0 (secure 0 0 0) 0x7f57fc04b350 con 0x7f580407b750 2019-12-05T06:12:13.668 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:13.654+0000 7f5803fff700 1 --2- 172.21.15.42:0/1733225270 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f57e803b220 0x7f57e803d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:12:13.669 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:13.654+0000 7f5803fff700 1 --2- 172.21.15.42:0/1733225270 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f57e803b220 0x7f57e803d730 crc :-1 s=READY pgs=944 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:12:13.669 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:13.662+0000 7f580a153700 1 -- 172.21.15.42:0/1733225270 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f580405d100 con 0x7f580407b750 2019-12-05T06:12:13.674 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:13.670+0000 7f5808ef1700 1 --2- 172.21.15.42:0/1733225270 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f580407b750 0x7f58041359e0 secure :-1 s=THROTTLE_DONE pgs=493 cs=0 l=1 rx=0x7f57fc001210 tx=0x7f57fc010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:13.674 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:13.670+0000 7f5800ff9700 1 -- 172.21.15.42:0/1733225270 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f57fc0323b0 con 0x7f580407b750 2019-12-05T06:12:13.843 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:13.838+0000 7f580a153700 1 -- 172.21.15.42:0/1733225270 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f580405b8b0 con 0x7f580407b750 2019-12-05T06:12:13.843 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:13.842+0000 7f5808ef1700 1 --2- 172.21.15.42:0/1733225270 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f580407b750 0x7f58041359e0 secure :-1 s=THROTTLE_DONE pgs=493 cs=0 l=1 rx=0x7f57fc001210 tx=0x7f57fc010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:13.844 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:13.842+0000 7f5800ff9700 1 -- 172.21.15.42:0/1733225270 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 270 v270) v1 ==== 95+0+1778 (secure 0 0 0) 0x7f57fc032610 con 0x7f580407b750 2019-12-05T06:12:13.846 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:12:13.846 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":270,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":270,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:12:12.996235+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":97,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7094},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7094":{"gid":7094,"name":"b","rank":0,"incarnation":269,"state":"up:reconnect","state_seq":10,"addr":"172.21.15.40:6833/3785523496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3785523496},{"type":"v1","addr":"172.21.15.40:6833","nonce":3785523496}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:12:13.846 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 270 2019-12-05T06:12:13.848 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:13.846+0000 7f580a153700 1 -- 172.21.15.42:0/1733225270 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f57e803b220 msgr2=0x7f57e803d730 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:13.849 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:13.846+0000 7f580a153700 1 --2- 172.21.15.42:0/1733225270 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f57e803b220 0x7f57e803d730 crc :-1 s=READY pgs=944 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:13.849 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:13.846+0000 7f580a153700 1 -- 172.21.15.42:0/1733225270 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f580407b750 msgr2=0x7f58041359e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:13.849 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:13.846+0000 7f580a153700 1 --2- 172.21.15.42:0/1733225270 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f580407b750 0x7f58041359e0 secure :-1 s=READY pgs=493 cs=0 l=1 rx=0x7f57fc001210 tx=0x7f57fc010b40).stop 2019-12-05T06:12:13.849 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:13.846+0000 7f580a153700 1 -- 172.21.15.42:0/1733225270 shutdown_connections 2019-12-05T06:12:13.849 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:13.846+0000 7f580a153700 1 --2- 172.21.15.42:0/1733225270 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f57e803b220 0x7f57e803d730 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:13.849 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:13.846+0000 7f580a153700 1 --2- 172.21.15.42:0/1733225270 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f580407b750 0x7f58041359e0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:13.849 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:13.846+0000 7f580a153700 1 --2- 172.21.15.42:0/1733225270 >> v2:172.21.15.40:3300/0 conn(0x7f580414e5b0 0x7f58041280f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:13.850 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:13.846+0000 7f580a153700 1 -- 172.21.15.42:0/1733225270 shutdown_connections 2019-12-05T06:12:13.850 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:13.846+0000 7f580a153700 1 -- 172.21.15.42:0/1733225270 wait complete. 2019-12-05T06:12:13.850 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:13.846+0000 7f580a153700 1 -- 172.21.15.42:0/1733225270 >> 172.21.15.42:0/1733225270 conn(0x7f58040a9a90 msgr2=0x7f58040596f0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:12:13.859 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2019-12-05T06:12:14.388 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:12:14.392 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:12:15.860 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:12:15.972 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:15.962+0000 7fc47754c700 1 Processor -- start 2019-12-05T06:12:15.973 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:15.962+0000 7fc47754c700 1 -- start start 2019-12-05T06:12:15.973 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:15.966+0000 7fc47754c700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc4700a7740 0x7fc47008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:12:15.973 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:15.966+0000 7fc47754c700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc47007e720 0x7fc47007eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:12:15.973 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:15.966+0000 7fc47754c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc47009f2e0 0x7fc47009f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:12:15.973 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:15.966+0000 7fc47754c700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4700a0b00 con 0x7fc4700a7740 2019-12-05T06:12:15.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:15.966+0000 7fc47754c700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc47008e5b0 con 0x7fc47009f2e0 2019-12-05T06:12:15.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:15.966+0000 7fc47754c700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc47008e850 con 0x7fc47007e720 2019-12-05T06:12:15.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:15.966+0000 7fc4762ea700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc47009f2e0 0x7fc47009f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:12:15.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:15.966+0000 7fc4762ea700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc47009f2e0 0x7fc47009f710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:50440/0 (socket says 172.21.15.42:50440) 2019-12-05T06:12:15.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:15.966+0000 7fc4762ea700 1 -- 172.21.15.42:0/974024167 learned_addr learned my addr 172.21.15.42:0/974024167 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:12:15.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:15.966+0000 7fc4752e8700 1 --2- 172.21.15.42:0/974024167 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc47007e720 0x7fc47007eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:12:15.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:15.966+0000 7fc475ae9700 1 --2- 172.21.15.42:0/974024167 >> v2:172.21.15.40:3300/0 conn(0x7fc4700a7740 0x7fc47008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:12:15.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:15.966+0000 7fc4762ea700 1 -- 172.21.15.42:0/974024167 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc47007e720 msgr2=0x7fc47007eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:12:15.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:15.966+0000 7fc4762ea700 1 --2- 172.21.15.42:0/974024167 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc47007e720 0x7fc47007eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:12:15.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:15.966+0000 7fc4762ea700 1 -- 172.21.15.42:0/974024167 >> v2:172.21.15.40:3300/0 conn(0x7fc4700a7740 msgr2=0x7fc47008df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:12:15.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:15.966+0000 7fc4762ea700 1 --2- 172.21.15.42:0/974024167 >> v2:172.21.15.40:3300/0 conn(0x7fc4700a7740 0x7fc47008df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:12:15.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:15.966+0000 7fc4762ea700 1 -- 172.21.15.42:0/974024167 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc47007f0a0 con 0x7fc47009f2e0 2019-12-05T06:12:15.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:15.966+0000 7fc4752e8700 1 --2- 172.21.15.42:0/974024167 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc47007e720 0x7fc47007eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:12:15.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:15.966+0000 7fc475ae9700 1 --2- 172.21.15.42:0/974024167 >> v2:172.21.15.40:3300/0 conn(0x7fc4700a7740 0x7fc47008df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:12:15.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:15.966+0000 7fc4762ea700 1 --2- 172.21.15.42:0/974024167 >> v2:172.21.15.42:3300/0 conn(0x7fc47009f2e0 0x7fc47009f710 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fc47008e770 tx=0x7fc468010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:15.976 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:15.966+0000 7fc4762ea700 1 --2- 172.21.15.42:0/974024167 >> v2:172.21.15.42:3300/0 conn(0x7fc47009f2e0 0x7fc47009f710 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fc47008e770 tx=0x7fc468010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:15.976 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:15.966+0000 7fc4762ea700 1 --2- 172.21.15.42:0/974024167 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc47009f2e0 0x7fc47009f710 secure :-1 s=READY pgs=2249 cs=0 l=1 rx=0x7fc47008e770 tx=0x7fc468010130).ready entity=mon.1 client_cookie=e2e31f0bce8d137 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:12:15.976 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:15.966+0000 7fc4762ea700 1 --2- 172.21.15.42:0/974024167 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc47009f2e0 0x7fc47009f710 secure :-1 s=THROTTLE_DONE pgs=2249 cs=0 l=1 rx=0x7fc47008e770 tx=0x7fc468010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:15.976 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:15.966+0000 7fc4762ea700 1 --2- 172.21.15.42:0/974024167 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc47009f2e0 0x7fc47009f710 secure :-1 s=THROTTLE_DONE pgs=2249 cs=0 l=1 rx=0x7fc47008e770 tx=0x7fc468010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:15.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:15.966+0000 7fc474ae7700 1 -- 172.21.15.42:0/974024167 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc468010de0 con 0x7fc47009f2e0 2019-12-05T06:12:15.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:15.966+0000 7fc4762ea700 1 --2- 172.21.15.42:0/974024167 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc47009f2e0 0x7fc47009f710 secure :-1 s=THROTTLE_DONE pgs=2249 cs=0 l=1 rx=0x7fc47008e770 tx=0x7fc468010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:15.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:15.966+0000 7fc474ae7700 1 -- 172.21.15.42:0/974024167 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc468010a00 con 0x7fc47009f2e0 2019-12-05T06:12:15.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:15.966+0000 7fc474ae7700 1 -- 172.21.15.42:0/974024167 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc4680114a0 con 0x7fc47009f2e0 2019-12-05T06:12:15.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:15.966+0000 7fc47754c700 1 -- 172.21.15.42:0/974024167 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc47009f2e0 msgr2=0x7fc47009f710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:15.978 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:15.966+0000 7fc47754c700 1 --2- 172.21.15.42:0/974024167 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc47009f2e0 0x7fc47009f710 secure :-1 s=READY pgs=2249 cs=0 l=1 rx=0x7fc47008e770 tx=0x7fc468010130).stop 2019-12-05T06:12:15.978 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:15.966+0000 7fc47754c700 1 -- 172.21.15.42:0/974024167 shutdown_connections 2019-12-05T06:12:15.986 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:15.966+0000 7fc47754c700 1 --2- 172.21.15.42:0/974024167 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc47009f2e0 0x7fc47009f710 unknown :-1 s=CLOSED pgs=2249 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:15.986 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:15.966+0000 7fc47754c700 1 --2- 172.21.15.42:0/974024167 >> v2:172.21.15.40:3300/0 conn(0x7fc4700a7740 0x7fc47008df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:12:15.986 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:15.966+0000 7fc47754c700 1 --2- 172.21.15.42:0/974024167 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc47007e720 0x7fc47007eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:12:15.986 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:15.970+0000 7fc47754c700 1 -- 172.21.15.42:0/974024167 shutdown_connections 2019-12-05T06:12:15.986 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:15.970+0000 7fc47754c700 1 -- 172.21.15.42:0/974024167 wait complete. 2019-12-05T06:12:15.987 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:15.970+0000 7fc47754c700 1 -- 172.21.15.42:0/974024167 >> 172.21.15.42:0/974024167 conn(0x7fc4700aabe0 msgr2=0x7fc4700a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:12:16.006 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:15.970+0000 7fc47754c700 1 Processor -- start 2019-12-05T06:12:16.007 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:15.970+0000 7fc47754c700 1 -- start start 2019-12-05T06:12:16.007 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:15.970+0000 7fc47754c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc470116500 0x7fc470143fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:12:16.007 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:15.970+0000 7fc47754c700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc4700a3230 0x7fc47008f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:12:16.007 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:15.970+0000 7fc47754c700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4700a2c60 con 0x7fc4700a1510 2019-12-05T06:12:16.007 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:15.970+0000 7fc47754c700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4700a1c90 con 0x7fc4700a3230 2019-12-05T06:12:16.007 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:15.970+0000 7fc47754c700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4700a2f90 con 0x7fc470116500 2019-12-05T06:12:16.008 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:15.970+0000 7fc475ae9700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc470116500 0x7fc470143fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:12:16.008 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:15.970+0000 7fc475ae9700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc470116500 0x7fc470143fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:50442/0 (socket says 172.21.15.42:50442) 2019-12-05T06:12:16.008 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:15.970+0000 7fc475ae9700 1 -- 172.21.15.42:0/57934060 learned_addr learned my addr 172.21.15.42:0/57934060 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:12:16.009 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:15.970+0000 7fc4762ea700 1 --2- 172.21.15.42:0/57934060 >> v2:172.21.15.40:3300/0 conn(0x7fc4700a3230 0x7fc47008f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:12:16.009 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:15.974+0000 7fc475ae9700 1 -- 172.21.15.42:0/57934060 >> v2:172.21.15.40:3300/0 conn(0x7fc4700a3230 msgr2=0x7fc47008f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:16.009 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:15.974+0000 7fc475ae9700 1 --2- 172.21.15.42:0/57934060 >> v2:172.21.15.40:3300/0 conn(0x7fc4700a3230 0x7fc47008f370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:16.009 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:15.974+0000 7fc475ae9700 1 -- 172.21.15.42:0/57934060 >> v1:172.21.15.40:6789/0 conn(0x7fc4700a1510 legacy=0x7fc47008eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:16.009 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:15.974+0000 7fc475ae9700 1 -- 172.21.15.42:0/57934060 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4700a1940 con 0x7fc470116500 2019-12-05T06:12:16.009 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:15.974+0000 7fc475ae9700 1 --2- 172.21.15.42:0/57934060 >> v2:172.21.15.42:3300/0 conn(0x7fc470116500 0x7fc470143fc0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fc46c002ef0 tx=0x7fc46c010d00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:16.009 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:15.974+0000 7fc475ae9700 1 --2- 172.21.15.42:0/57934060 >> v2:172.21.15.42:3300/0 conn(0x7fc470116500 0x7fc470143fc0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fc46c002ef0 tx=0x7fc46c010d00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:16.010 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:15.974+0000 7fc475ae9700 1 --2- 172.21.15.42:0/57934060 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc470116500 0x7fc470143fc0 secure :-1 s=READY pgs=2250 cs=0 l=1 rx=0x7fc46c002ef0 tx=0x7fc46c010d00).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:12:16.010 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:15.974+0000 7fc475ae9700 1 --2- 172.21.15.42:0/57934060 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc470116500 0x7fc470143fc0 secure :-1 s=THROTTLE_DONE pgs=2250 cs=0 l=1 rx=0x7fc46c002ef0 tx=0x7fc46c010d00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:16.010 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:15.974+0000 7fc4627fc700 1 -- 172.21.15.42:0/57934060 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc46c005a30 con 0x7fc470116500 2019-12-05T06:12:16.010 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:15.974+0000 7fc475ae9700 1 --2- 172.21.15.42:0/57934060 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc470116500 0x7fc470143fc0 secure :-1 s=THROTTLE_DONE pgs=2250 cs=0 l=1 rx=0x7fc46c002ef0 tx=0x7fc46c010d00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:16.010 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:15.974+0000 7fc475ae9700 1 --2- 172.21.15.42:0/57934060 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc470116500 0x7fc470143fc0 secure :-1 s=THROTTLE_DONE pgs=2250 cs=0 l=1 rx=0x7fc46c002ef0 tx=0x7fc46c010d00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:16.010 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:15.974+0000 7fc4627fc700 1 -- 172.21.15.42:0/57934060 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc46c007200 con 0x7fc470116500 2019-12-05T06:12:16.010 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:15.974+0000 7fc47754c700 1 -- 172.21.15.42:0/57934060 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc47008f8c0 con 0x7fc470116500 2019-12-05T06:12:16.011 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:15.974+0000 7fc4627fc700 1 -- 172.21.15.42:0/57934060 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc46c012570 con 0x7fc470116500 2019-12-05T06:12:16.011 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:15.974+0000 7fc47754c700 1 -- 172.21.15.42:0/57934060 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc470144fd0 con 0x7fc470116500 2019-12-05T06:12:16.011 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:15.974+0000 7fc475ae9700 1 --2- 172.21.15.42:0/57934060 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc470116500 0x7fc470143fc0 secure :-1 s=THROTTLE_DONE pgs=2250 cs=0 l=1 rx=0x7fc46c002ef0 tx=0x7fc46c010d00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:16.011 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:15.974+0000 7fc4627fc700 1 -- 172.21.15.42:0/57934060 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fc46c0128d0 con 0x7fc470116500 2019-12-05T06:12:16.011 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:15.974+0000 7fc475ae9700 1 --2- 172.21.15.42:0/57934060 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc470116500 0x7fc470143fc0 secure :-1 s=THROTTLE_DONE pgs=2250 cs=0 l=1 rx=0x7fc46c002ef0 tx=0x7fc46c010d00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:16.011 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:15.974+0000 7fc4627fc700 1 --2- 172.21.15.42:0/57934060 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc45803ae20 0x7fc45803d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:12:16.011 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:15.978+0000 7fc4627fc700 1 -- 172.21.15.42:0/57934060 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 10159+0+0 (secure 0 0 0) 0x7fc46c04b860 con 0x7fc470116500 2019-12-05T06:12:16.011 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:15.978+0000 7fc4752e8700 1 --2- 172.21.15.42:0/57934060 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc45803ae20 0x7fc45803d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:12:16.012 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:15.978+0000 7fc4752e8700 1 --2- 172.21.15.42:0/57934060 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc45803ae20 0x7fc45803d330 crc :-1 s=READY pgs=945 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:12:16.012 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:15.978+0000 7fc47754c700 1 -- 172.21.15.42:0/57934060 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc47005d110 con 0x7fc470116500 2019-12-05T06:12:16.012 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:15.990+0000 7fc475ae9700 1 --2- 172.21.15.42:0/57934060 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc470116500 0x7fc470143fc0 secure :-1 s=THROTTLE_DONE pgs=2250 cs=0 l=1 rx=0x7fc46c002ef0 tx=0x7fc46c010d00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:16.012 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:15.990+0000 7fc4627fc700 1 -- 172.21.15.42:0/57934060 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fc46c0328c0 con 0x7fc470116500 2019-12-05T06:12:16.163 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:16.158+0000 7fc47754c700 1 -- 172.21.15.42:0/57934060 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc470143620 con 0x7fc470116500 2019-12-05T06:12:16.163 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:16.158+0000 7fc475ae9700 1 --2- 172.21.15.42:0/57934060 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc470116500 0x7fc470143fc0 secure :-1 s=THROTTLE_DONE pgs=2250 cs=0 l=1 rx=0x7fc46c002ef0 tx=0x7fc46c010d00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:16.163 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:16.158+0000 7fc4627fc700 1 -- 172.21.15.42:0/57934060 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 272 v272) v1 ==== 95+0+1775 (secure 0 0 0) 0x7fc46c032b20 con 0x7fc470116500 2019-12-05T06:12:16.163 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 272 2019-12-05T06:12:16.164 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:12:16.164 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":272,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":272,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:12:15.021996+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":97,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7094},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7094":{"gid":7094,"name":"b","rank":0,"incarnation":269,"state":"up:active","state_seq":12,"addr":"172.21.15.40:6833/3785523496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3785523496},{"type":"v1","addr":"172.21.15.40:6833","nonce":3785523496}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:12:16.174 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:16.162+0000 7fc47754c700 1 -- 172.21.15.42:0/57934060 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc45803ae20 msgr2=0x7fc45803d330 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:16.174 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:16.162+0000 7fc47754c700 1 --2- 172.21.15.42:0/57934060 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc45803ae20 0x7fc45803d330 crc :-1 s=READY pgs=945 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:16.175 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:16.162+0000 7fc47754c700 1 -- 172.21.15.42:0/57934060 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc470116500 msgr2=0x7fc470143fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:16.175 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:16.162+0000 7fc47754c700 1 --2- 172.21.15.42:0/57934060 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc470116500 0x7fc470143fc0 secure :-1 s=READY pgs=2250 cs=0 l=1 rx=0x7fc46c002ef0 tx=0x7fc46c010d00).stop 2019-12-05T06:12:16.175 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:16.162+0000 7fc47754c700 1 -- 172.21.15.42:0/57934060 shutdown_connections 2019-12-05T06:12:16.175 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:16.162+0000 7fc47754c700 1 --2- 172.21.15.42:0/57934060 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc45803ae20 0x7fc45803d330 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:16.176 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:16.162+0000 7fc47754c700 1 --2- 172.21.15.42:0/57934060 >> v2:172.21.15.40:3300/0 conn(0x7fc4700a3230 0x7fc47008f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:16.176 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:16.162+0000 7fc47754c700 1 --2- 172.21.15.42:0/57934060 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc470116500 0x7fc470143fc0 unknown :-1 s=CLOSED pgs=2250 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:16.176 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:16.162+0000 7fc47754c700 1 -- 172.21.15.42:0/57934060 shutdown_connections 2019-12-05T06:12:16.176 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:16.162+0000 7fc47754c700 1 -- 172.21.15.42:0/57934060 wait complete. 2019-12-05T06:12:16.177 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:16.162+0000 7fc47754c700 1 -- 172.21.15.42:0/57934060 >> 172.21.15.42:0/57934060 conn(0x7fc4700a8410 msgr2=0x7fc4700a9640 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:12:16.178 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b has gained rank=0, replacing gid=10000 2019-12-05T06:12:16.179 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 7 secs before reviving mds.a 2019-12-05T06:12:23.179 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.a 2019-12-05T06:12:23.179 INFO:tasks.ceph.mds.a:Restarting daemon 2019-12-05T06:12:23.179 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2019-12-05T06:12:23.183 INFO:tasks.ceph.mds.a:Started 2019-12-05T06:12:23.184 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:12:23.242 INFO:tasks.ceph.mds.a.smithi042.stdout:starting mds.a at 2019-12-05T06:12:23.280 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:23.270+0000 7fa42e0b4700 1 Processor -- start 2019-12-05T06:12:23.280 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:23.270+0000 7fa42e0b4700 1 -- start start 2019-12-05T06:12:23.281 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:23.274+0000 7fa42e0b4700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa4280a7740 0x7fa42808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:12:23.281 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:23.274+0000 7fa42e0b4700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fa42807e720 0x7fa42807eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:12:23.281 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:23.274+0000 7fa42e0b4700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4280a0710 con 0x7fa42809f2e0 2019-12-05T06:12:23.281 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:23.274+0000 7fa42e0b4700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4280a0b00 con 0x7fa4280a7740 2019-12-05T06:12:23.281 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:23.274+0000 7fa42e0b4700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa42808e5b0 con 0x7fa42807e720 2019-12-05T06:12:23.281 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:23.274+0000 7fa4277fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fa42807e720 0x7fa42807eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:12:23.282 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:23.274+0000 7fa427fff700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa4280a7740 0x7fa42808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:12:23.282 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:23.274+0000 7fa42ce52700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fa42809f2e0 0x7fa42809f710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:59984/0 (socket says 172.21.15.42:59984) 2019-12-05T06:12:23.282 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:23.274+0000 7fa42ce52700 1 -- 172.21.15.42:0/1022874152 learned_addr learned my addr 172.21.15.42:0/1022874152 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:12:23.282 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:23.274+0000 7fa4277fe700 1 -- 172.21.15.42:0/1022874152 >> v2:172.21.15.40:3300/0 conn(0x7fa4280a7740 msgr2=0x7fa42808df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:12:23.283 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:23.274+0000 7fa4277fe700 1 --2- 172.21.15.42:0/1022874152 >> v2:172.21.15.40:3300/0 conn(0x7fa4280a7740 0x7fa42808df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:12:23.283 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:23.274+0000 7fa4277fe700 1 -- 172.21.15.42:0/1022874152 >> v1:172.21.15.40:6789/0 conn(0x7fa42809f2e0 legacy=0x7fa42809f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:23.283 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:23.274+0000 7fa4277fe700 1 -- 172.21.15.42:0/1022874152 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa42807f0a0 con 0x7fa42807e720 2019-12-05T06:12:23.283 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:23.274+0000 7fa4277fe700 1 --2- 172.21.15.42:0/1022874152 >> v2:172.21.15.42:3300/0 conn(0x7fa42807e720 0x7fa42807eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fa418002f30 tx=0x7fa41800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:23.283 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:23.274+0000 7fa4277fe700 1 --2- 172.21.15.42:0/1022874152 >> v2:172.21.15.42:3300/0 conn(0x7fa42807e720 0x7fa42807eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fa418002f30 tx=0x7fa41800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:23.283 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:23.274+0000 7fa4277fe700 1 --2- 172.21.15.42:0/1022874152 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa42807e720 0x7fa42807eb50 secure :-1 s=READY pgs=2254 cs=0 l=1 rx=0x7fa418002f30 tx=0x7fa41800fc70).ready entity=mon.1 client_cookie=91ef12468577bd1d server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:12:23.283 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:23.274+0000 7fa4277fe700 1 --2- 172.21.15.42:0/1022874152 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa42807e720 0x7fa42807eb50 secure :-1 s=THROTTLE_DONE pgs=2254 cs=0 l=1 rx=0x7fa418002f30 tx=0x7fa41800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:23.284 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:23.274+0000 7fa4277fe700 1 --2- 172.21.15.42:0/1022874152 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa42807e720 0x7fa42807eb50 secure :-1 s=THROTTLE_DONE pgs=2254 cs=0 l=1 rx=0x7fa418002f30 tx=0x7fa41800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:23.284 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:23.274+0000 7fa426ffd700 1 -- 172.21.15.42:0/1022874152 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa4180112b0 con 0x7fa42807e720 2019-12-05T06:12:23.284 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:23.274+0000 7fa4277fe700 1 --2- 172.21.15.42:0/1022874152 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa42807e720 0x7fa42807eb50 secure :-1 s=THROTTLE_DONE pgs=2254 cs=0 l=1 rx=0x7fa418002f30 tx=0x7fa41800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:23.284 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:23.274+0000 7fa426ffd700 1 -- 172.21.15.42:0/1022874152 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa418008ad0 con 0x7fa42807e720 2019-12-05T06:12:23.284 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:23.274+0000 7fa426ffd700 1 -- 172.21.15.42:0/1022874152 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa418011890 con 0x7fa42807e720 2019-12-05T06:12:23.285 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:23.274+0000 7fa42e0b4700 1 -- 172.21.15.42:0/1022874152 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa42807e720 msgr2=0x7fa42807eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:23.285 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:23.274+0000 7fa42e0b4700 1 --2- 172.21.15.42:0/1022874152 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa42807e720 0x7fa42807eb50 secure :-1 s=READY pgs=2254 cs=0 l=1 rx=0x7fa418002f30 tx=0x7fa41800fc70).stop 2019-12-05T06:12:23.285 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:23.274+0000 7fa42e0b4700 1 -- 172.21.15.42:0/1022874152 shutdown_connections 2019-12-05T06:12:23.285 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:23.274+0000 7fa42e0b4700 1 --2- 172.21.15.42:0/1022874152 >> v2:172.21.15.40:3300/0 conn(0x7fa4280a7740 0x7fa42808df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:12:23.285 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:23.274+0000 7fa42e0b4700 1 --2- 172.21.15.42:0/1022874152 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa42807e720 0x7fa42807eb50 unknown :-1 s=CLOSED pgs=2254 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:23.286 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:23.274+0000 7fa42e0b4700 1 -- 172.21.15.42:0/1022874152 shutdown_connections 2019-12-05T06:12:23.286 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:23.274+0000 7fa42e0b4700 1 -- 172.21.15.42:0/1022874152 wait complete. 2019-12-05T06:12:23.286 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:23.274+0000 7fa42e0b4700 1 -- 172.21.15.42:0/1022874152 >> 172.21.15.42:0/1022874152 conn(0x7fa4280aabe0 msgr2=0x7fa4280a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:12:23.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:23.278+0000 7fa42e0b4700 1 Processor -- start 2019-12-05T06:12:23.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:23.278+0000 7fa42e0b4700 1 -- start start 2019-12-05T06:12:23.291 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:23.278+0000 7fa42e0b4700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa42808ee90 0x7fa4280a6ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:12:23.291 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:23.278+0000 7fa42e0b4700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fa428125330 0x7fa428125760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:12:23.291 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:23.278+0000 7fa42e0b4700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa428125cb0 con 0x7fa4280a5160 2019-12-05T06:12:23.291 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:23.278+0000 7fa42e0b4700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa428116d90 con 0x7fa42808ee90 2019-12-05T06:12:23.291 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:23.278+0000 7fa42e0b4700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa428117030 con 0x7fa428125330 2019-12-05T06:12:23.291 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:23.278+0000 7fa4277fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fa428125330 0x7fa428125760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:12:23.291 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:23.278+0000 7fa4277fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fa428125330 0x7fa428125760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:50510/0 (socket says 172.21.15.42:50510) 2019-12-05T06:12:23.292 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:23.278+0000 7fa4277fe700 1 -- 172.21.15.42:0/1306651206 learned_addr learned my addr 172.21.15.42:0/1306651206 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:12:23.292 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:23.278+0000 7fa427fff700 1 --2- 172.21.15.42:0/1306651206 >> v2:172.21.15.40:3300/0 conn(0x7fa42808ee90 0x7fa4280a6ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:12:23.292 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:23.278+0000 7fa4277fe700 1 -- 172.21.15.42:0/1306651206 >> v2:172.21.15.40:3300/0 conn(0x7fa42808ee90 msgr2=0x7fa4280a6ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:23.292 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:23.278+0000 7fa4277fe700 1 --2- 172.21.15.42:0/1306651206 >> v2:172.21.15.40:3300/0 conn(0x7fa42808ee90 0x7fa4280a6ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:23.292 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:23.278+0000 7fa4277fe700 1 -- 172.21.15.42:0/1306651206 >> v1:172.21.15.40:6789/0 conn(0x7fa4280a5160 legacy=0x7fa4280a5590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:23.292 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:23.278+0000 7fa4277fe700 1 -- 172.21.15.42:0/1306651206 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4280a7030 con 0x7fa428125330 2019-12-05T06:12:23.295 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:23.278+0000 7fa4277fe700 1 --2- 172.21.15.42:0/1306651206 >> v2:172.21.15.42:3300/0 conn(0x7fa428125330 0x7fa428125760 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fa4281166b0 tx=0x7fa418008d80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:23.295 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:23.278+0000 7fa4277fe700 1 --2- 172.21.15.42:0/1306651206 >> v2:172.21.15.42:3300/0 conn(0x7fa428125330 0x7fa428125760 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fa4281166b0 tx=0x7fa418008d80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:23.296 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:23.278+0000 7fa4277fe700 1 --2- 172.21.15.42:0/1306651206 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa428125330 0x7fa428125760 secure :-1 s=READY pgs=2255 cs=0 l=1 rx=0x7fa4281166b0 tx=0x7fa418008d80).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:12:23.296 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:23.278+0000 7fa4277fe700 1 --2- 172.21.15.42:0/1306651206 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa428125330 0x7fa428125760 secure :-1 s=THROTTLE_DONE pgs=2255 cs=0 l=1 rx=0x7fa4281166b0 tx=0x7fa418008d80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:23.296 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:23.278+0000 7fa4277fe700 1 --2- 172.21.15.42:0/1306651206 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa428125330 0x7fa428125760 secure :-1 s=THROTTLE_DONE pgs=2255 cs=0 l=1 rx=0x7fa4281166b0 tx=0x7fa418008d80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:23.296 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:23.278+0000 7fa424ff9700 1 -- 172.21.15.42:0/1306651206 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa418007150 con 0x7fa428125330 2019-12-05T06:12:23.296 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:23.278+0000 7fa4277fe700 1 --2- 172.21.15.42:0/1306651206 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa428125330 0x7fa428125760 secure :-1 s=THROTTLE_DONE pgs=2255 cs=0 l=1 rx=0x7fa4281166b0 tx=0x7fa418008d80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:23.296 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:23.278+0000 7fa424ff9700 1 -- 172.21.15.42:0/1306651206 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4180114b0 con 0x7fa428125330 2019-12-05T06:12:23.296 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:23.278+0000 7fa42e0b4700 1 -- 172.21.15.42:0/1306651206 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4280a5d90 con 0x7fa428125330 2019-12-05T06:12:23.297 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:23.278+0000 7fa424ff9700 1 -- 172.21.15.42:0/1306651206 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa418005730 con 0x7fa428125330 2019-12-05T06:12:23.297 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:23.278+0000 7fa42e0b4700 1 -- 172.21.15.42:0/1306651206 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa42812c270 con 0x7fa428125330 2019-12-05T06:12:23.297 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:23.282+0000 7fa4277fe700 1 --2- 172.21.15.42:0/1306651206 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa428125330 0x7fa428125760 secure :-1 s=THROTTLE_DONE pgs=2255 cs=0 l=1 rx=0x7fa4281166b0 tx=0x7fa418008d80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:23.299 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:23.282+0000 7fa424ff9700 1 -- 172.21.15.42:0/1306651206 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fa418005a00 con 0x7fa428125330 2019-12-05T06:12:23.300 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:23.282+0000 7fa4277fe700 1 --2- 172.21.15.42:0/1306651206 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa428125330 0x7fa428125760 secure :-1 s=THROTTLE_DONE pgs=2255 cs=0 l=1 rx=0x7fa4281166b0 tx=0x7fa418008d80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:23.300 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:23.282+0000 7fa424ff9700 1 --2- 172.21.15.42:0/1306651206 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa40803ad80 0x7fa40803d290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:12:23.300 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:23.282+0000 7fa424ff9700 1 -- 172.21.15.42:0/1306651206 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 10159+0+0 (secure 0 0 0) 0x7fa41804ba50 con 0x7fa428125330 2019-12-05T06:12:23.300 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:23.282+0000 7fa427fff700 1 --2- 172.21.15.42:0/1306651206 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa40803ad80 0x7fa40803d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:12:23.300 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:23.282+0000 7fa427fff700 1 --2- 172.21.15.42:0/1306651206 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa40803ad80 0x7fa40803d290 crc :-1 s=READY pgs=946 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:12:23.302 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:23.286+0000 7fa42e0b4700 1 -- 172.21.15.42:0/1306651206 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa42805d110 con 0x7fa428125330 2019-12-05T06:12:23.302 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:23.294+0000 7fa4277fe700 1 --2- 172.21.15.42:0/1306651206 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa428125330 0x7fa428125760 secure :-1 s=THROTTLE_DONE pgs=2255 cs=0 l=1 rx=0x7fa4281166b0 tx=0x7fa418008d80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:23.302 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:23.294+0000 7fa424ff9700 1 -- 172.21.15.42:0/1306651206 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fa418032cf0 con 0x7fa428125330 2019-12-05T06:12:23.476 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:23.474+0000 7fa42e0b4700 1 -- 172.21.15.42:0/1306651206 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa4280a6530 con 0x7fa428125330 2019-12-05T06:12:23.476 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:23.474+0000 7fa4277fe700 1 --2- 172.21.15.42:0/1306651206 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa428125330 0x7fa428125760 secure :-1 s=THROTTLE_DONE pgs=2255 cs=0 l=1 rx=0x7fa4281166b0 tx=0x7fa418008d80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:23.477 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:23.474+0000 7fa424ff9700 1 -- 172.21.15.42:0/1306651206 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 272 v272) v1 ==== 95+0+1775 (secure 0 0 0) 0x7fa418011e30 con 0x7fa428125330 2019-12-05T06:12:23.477 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:12:23.477 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":272,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":272,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:12:15.021996+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":97,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7094},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7094":{"gid":7094,"name":"b","rank":0,"incarnation":269,"state":"up:active","state_seq":12,"addr":"172.21.15.40:6833/3785523496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3785523496},{"type":"v1","addr":"172.21.15.40:6833","nonce":3785523496}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:12:23.478 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 272 2019-12-05T06:12:23.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:23.474+0000 7fa42e0b4700 1 -- 172.21.15.42:0/1306651206 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa40803ad80 msgr2=0x7fa40803d290 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:23.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:23.474+0000 7fa42e0b4700 1 --2- 172.21.15.42:0/1306651206 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa40803ad80 0x7fa40803d290 crc :-1 s=READY pgs=946 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:23.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:23.474+0000 7fa42e0b4700 1 -- 172.21.15.42:0/1306651206 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa428125330 msgr2=0x7fa428125760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:23.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:23.474+0000 7fa42e0b4700 1 --2- 172.21.15.42:0/1306651206 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa428125330 0x7fa428125760 secure :-1 s=READY pgs=2255 cs=0 l=1 rx=0x7fa4281166b0 tx=0x7fa418008d80).stop 2019-12-05T06:12:23.482 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:23.474+0000 7fa42e0b4700 1 -- 172.21.15.42:0/1306651206 shutdown_connections 2019-12-05T06:12:23.482 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:23.474+0000 7fa42e0b4700 1 --2- 172.21.15.42:0/1306651206 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa40803ad80 0x7fa40803d290 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:23.482 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:23.474+0000 7fa42e0b4700 1 --2- 172.21.15.42:0/1306651206 >> v2:172.21.15.40:3300/0 conn(0x7fa42808ee90 0x7fa4280a6ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:23.482 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:23.474+0000 7fa42e0b4700 1 --2- 172.21.15.42:0/1306651206 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa428125330 0x7fa428125760 unknown :-1 s=CLOSED pgs=2255 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:23.482 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:23.474+0000 7fa42e0b4700 1 -- 172.21.15.42:0/1306651206 shutdown_connections 2019-12-05T06:12:23.482 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:23.474+0000 7fa42e0b4700 1 -- 172.21.15.42:0/1306651206 wait complete. 2019-12-05T06:12:23.483 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:23.474+0000 7fa42e0b4700 1 -- 172.21.15.42:0/1306651206 >> 172.21.15.42:0/1306651206 conn(0x7fa4280a9850 msgr2=0x7fa42807d9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:12:23.490 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2019-12-05T06:12:25.491 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:12:25.591 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:25.582+0000 7f5da4b63700 1 Processor -- start 2019-12-05T06:12:25.591 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:25.582+0000 7f5da4b63700 1 -- start start 2019-12-05T06:12:25.592 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:25.582+0000 7f5da4b63700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5da009fae0 0x7f5da014fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:12:25.592 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:25.582+0000 7f5da4b63700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5da00a0710 con 0x7f5da00a7740 2019-12-05T06:12:25.592 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:25.582+0000 7f5da4b63700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5da007ef20 con 0x7f5da008df80 2019-12-05T06:12:25.592 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:25.582+0000 7f5da4b63700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5da00a0b00 con 0x7f5da009fae0 2019-12-05T06:12:25.592 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:25.586+0000 7f5d9659c700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f5da008df80 0x7f5da009f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:50566/0 (socket says 172.21.15.42:50566) 2019-12-05T06:12:25.592 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:25.586+0000 7f5d9659c700 1 -- 172.21.15.42:0/1333665287 learned_addr learned my addr 172.21.15.42:0/1333665287 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:12:25.593 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:25.586+0000 7f5d9f59e700 1 --2- 172.21.15.42:0/1333665287 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5da009fae0 0x7f5da014fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:12:25.593 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:25.586+0000 7f5d9e59c700 1 -- 172.21.15.42:0/1333665287 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 472469364 0 0) 0x7f5da007ef20 con 0x7f5da008df80 2019-12-05T06:12:25.593 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:25.586+0000 7f5d9e59c700 1 -- 172.21.15.42:0/1333665287 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d8c0030b0 con 0x7f5da008df80 2019-12-05T06:12:25.593 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:25.586+0000 7f5d9e59c700 1 -- 172.21.15.42:0/1333665287 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2766020999 0 0) 0x7f5da00a0710 con 0x7f5da00a7740 2019-12-05T06:12:25.593 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:25.586+0000 7f5d9f59e700 1 -- 172.21.15.42:0/1333665287 >> v1:172.21.15.42:6789/0 conn(0x7f5da008df80 legacy=0x7f5da009f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:25.593 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:25.586+0000 7f5d9f59e700 1 -- 172.21.15.42:0/1333665287 >> v1:172.21.15.40:6789/0 conn(0x7f5da00a7740 legacy=0x7f5da007e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:25.594 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:25.586+0000 7f5d9f59e700 1 -- 172.21.15.42:0/1333665287 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5da00a9ba0 con 0x7f5da009fae0 2019-12-05T06:12:25.594 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:25.586+0000 7f5d9f59e700 1 --2- 172.21.15.42:0/1333665287 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5da009fae0 0x7f5da014fff0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f5d980032b0 tx=0x7f5d9800ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:25.594 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:25.586+0000 7f5d9f59e700 1 --2- 172.21.15.42:0/1333665287 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5da009fae0 0x7f5da014fff0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f5d980032b0 tx=0x7f5d9800ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:25.594 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:25.586+0000 7f5d9f59e700 1 --2- 172.21.15.42:0/1333665287 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5da009fae0 0x7f5da014fff0 secure :-1 s=READY pgs=496 cs=0 l=1 rx=0x7f5d980032b0 tx=0x7f5d9800ff70).ready entity=mon.2 client_cookie=a2212270d2363983 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:12:25.594 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:25.586+0000 7f5d9f59e700 1 --2- 172.21.15.42:0/1333665287 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5da009fae0 0x7f5da014fff0 secure :-1 s=THROTTLE_DONE pgs=496 cs=0 l=1 rx=0x7f5d980032b0 tx=0x7f5d9800ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:25.594 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:25.586+0000 7f5d9e59c700 1 -- 172.21.15.42:0/1333665287 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5d980109b0 con 0x7f5da009fae0 2019-12-05T06:12:25.594 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:25.586+0000 7f5d9f59e700 1 --2- 172.21.15.42:0/1333665287 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5da009fae0 0x7f5da014fff0 secure :-1 s=THROTTLE_DONE pgs=496 cs=0 l=1 rx=0x7f5d980032b0 tx=0x7f5d9800ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:25.595 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:25.586+0000 7f5d9e59c700 1 -- 172.21.15.42:0/1333665287 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d98008b50 con 0x7f5da009fae0 2019-12-05T06:12:25.595 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:25.586+0000 7f5d9f59e700 1 --2- 172.21.15.42:0/1333665287 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5da009fae0 0x7f5da014fff0 secure :-1 s=THROTTLE_DONE pgs=496 cs=0 l=1 rx=0x7f5d980032b0 tx=0x7f5d9800ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:25.595 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:25.586+0000 7f5da4b63700 1 -- 172.21.15.42:0/1333665287 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5da009fae0 msgr2=0x7f5da014fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:25.595 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:25.586+0000 7f5d9e59c700 1 -- 172.21.15.42:0/1333665287 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5d980110c0 con 0x7f5da009fae0 2019-12-05T06:12:25.595 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:25.586+0000 7f5da4b63700 1 --2- 172.21.15.42:0/1333665287 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5da009fae0 0x7f5da014fff0 secure :-1 s=READY pgs=496 cs=0 l=1 rx=0x7f5d980032b0 tx=0x7f5d9800ff70).stop 2019-12-05T06:12:25.595 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:25.586+0000 7f5da4b63700 1 -- 172.21.15.42:0/1333665287 shutdown_connections 2019-12-05T06:12:25.596 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:25.586+0000 7f5da4b63700 1 --2- 172.21.15.42:0/1333665287 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5da009fae0 0x7f5da014fff0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:25.596 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:25.586+0000 7f5da4b63700 1 -- 172.21.15.42:0/1333665287 shutdown_connections 2019-12-05T06:12:25.596 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:25.586+0000 7f5da4b63700 1 -- 172.21.15.42:0/1333665287 wait complete. 2019-12-05T06:12:25.596 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:25.586+0000 7f5da4b63700 1 -- 172.21.15.42:0/1333665287 >> 172.21.15.42:0/1333665287 conn(0x7f5da00aabe0 msgr2=0x7f5da00a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:12:25.598 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:25.590+0000 7f5da4b63700 1 Processor -- start 2019-12-05T06:12:25.598 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:25.590+0000 7f5da4b63700 1 -- start start 2019-12-05T06:12:25.598 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:25.590+0000 7f5da4b63700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5da0064bf0 0x7f5da0143b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:12:25.599 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:25.590+0000 7f5da4b63700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5da0142990 con 0x7f5da008eeb0 2019-12-05T06:12:25.599 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:25.590+0000 7f5da4b63700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5da0065aa0 con 0x7f5da012df20 2019-12-05T06:12:25.599 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:25.590+0000 7f5da4b63700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5da00646b0 con 0x7f5da0064bf0 2019-12-05T06:12:25.600 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:25.590+0000 7f5d9f59e700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f5da012df20 0x7f5da0130360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:50574/0 (socket says 172.21.15.42:50574) 2019-12-05T06:12:25.600 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:25.590+0000 7f5d9f59e700 1 -- 172.21.15.42:0/2278689051 learned_addr learned my addr 172.21.15.42:0/2278689051 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:12:25.601 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:25.590+0000 7f5d9659c700 1 --2- 172.21.15.42:0/2278689051 >> v2:172.21.15.40:3300/0 conn(0x7f5da0064bf0 0x7f5da0143b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:12:25.601 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:25.590+0000 7f5d9659c700 1 -- 172.21.15.42:0/2278689051 >> v1:172.21.15.42:6789/0 conn(0x7f5da012df20 legacy=0x7f5da0130360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:25.601 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:25.590+0000 7f5d9659c700 1 -- 172.21.15.42:0/2278689051 >> v1:172.21.15.40:6789/0 conn(0x7f5da008eeb0 legacy=0x7f5da007bb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:25.601 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:25.590+0000 7f5d9659c700 1 -- 172.21.15.42:0/2278689051 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5da0142530 con 0x7f5da0064bf0 2019-12-05T06:12:25.601 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:25.590+0000 7f5d9659c700 1 --2- 172.21.15.42:0/2278689051 >> v2:172.21.15.40:3300/0 conn(0x7f5da0064bf0 0x7f5da0143b30 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f5d88000b80 tx=0x7f5d88010810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:25.601 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:25.590+0000 7f5d9659c700 1 --2- 172.21.15.42:0/2278689051 >> v2:172.21.15.40:3300/0 conn(0x7f5da0064bf0 0x7f5da0143b30 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f5d88000b80 tx=0x7f5d88010810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:25.602 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:25.590+0000 7f5d9659c700 1 --2- 172.21.15.42:0/2278689051 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5da0064bf0 0x7f5da0143b30 secure :-1 s=READY pgs=1705 cs=0 l=1 rx=0x7f5d88000b80 tx=0x7f5d88010810).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:12:25.603 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:25.594+0000 7f5d9659c700 1 --2- 172.21.15.42:0/2278689051 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5da0064bf0 0x7f5da0143b30 secure :-1 s=THROTTLE_DONE pgs=1705 cs=0 l=1 rx=0x7f5d88000b80 tx=0x7f5d88010810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:25.604 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:25.594+0000 7f5d97fff700 1 -- 172.21.15.42:0/2278689051 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5d88005c30 con 0x7f5da0064bf0 2019-12-05T06:12:25.604 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:25.594+0000 7f5d9659c700 1 --2- 172.21.15.42:0/2278689051 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5da0064bf0 0x7f5da0143b30 secure :-1 s=THROTTLE_DONE pgs=1705 cs=0 l=1 rx=0x7f5d88000b80 tx=0x7f5d88010810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:25.604 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:25.594+0000 7f5d9659c700 1 --2- 172.21.15.42:0/2278689051 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5da0064bf0 0x7f5da0143b30 secure :-1 s=THROTTLE_DONE pgs=1705 cs=0 l=1 rx=0x7f5d88000b80 tx=0x7f5d88010810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:25.604 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:25.594+0000 7f5d97fff700 1 -- 172.21.15.42:0/2278689051 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d88005850 con 0x7f5da0064bf0 2019-12-05T06:12:25.604 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:25.594+0000 7f5da4b63700 1 -- 172.21.15.42:0/2278689051 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5da0065620 con 0x7f5da0064bf0 2019-12-05T06:12:25.604 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:25.594+0000 7f5d97fff700 1 -- 172.21.15.42:0/2278689051 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5d880064a0 con 0x7f5da0064bf0 2019-12-05T06:12:25.604 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:25.594+0000 7f5da4b63700 1 -- 172.21.15.42:0/2278689051 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5da0144080 con 0x7f5da0064bf0 2019-12-05T06:12:25.605 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:25.598+0000 7f5d9659c700 1 --2- 172.21.15.42:0/2278689051 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5da0064bf0 0x7f5da0143b30 secure :-1 s=THROTTLE_DONE pgs=1705 cs=0 l=1 rx=0x7f5d88000b80 tx=0x7f5d88010810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:25.605 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:25.598+0000 7f5d97fff700 1 -- 172.21.15.42:0/2278689051 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f5d880067b0 con 0x7f5da0064bf0 2019-12-05T06:12:25.605 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:25.598+0000 7f5d9659c700 1 --2- 172.21.15.42:0/2278689051 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5da0064bf0 0x7f5da0143b30 secure :-1 s=THROTTLE_DONE pgs=1705 cs=0 l=1 rx=0x7f5d88000b80 tx=0x7f5d88010810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:25.605 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:25.598+0000 7f5d97fff700 1 --2- 172.21.15.42:0/2278689051 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5d8403ad90 0x7f5d8403d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:12:25.605 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:25.598+0000 7f5d97fff700 1 -- 172.21.15.42:0/2278689051 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 10159+0+0 (secure 0 0 0) 0x7f5d8804b080 con 0x7f5da0064bf0 2019-12-05T06:12:25.605 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:25.598+0000 7f5d9ed9d700 1 --2- 172.21.15.42:0/2278689051 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5d8403ad90 0x7f5d8403d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:12:25.605 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:25.598+0000 7f5da4b63700 1 -- 172.21.15.42:0/2278689051 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5da005d110 con 0x7f5da0064bf0 2019-12-05T06:12:25.606 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:25.598+0000 7f5d9ed9d700 1 --2- 172.21.15.42:0/2278689051 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5d8403ad90 0x7f5d8403d2a0 crc :-1 s=READY pgs=947 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:12:25.609 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:25.606+0000 7f5d9659c700 1 --2- 172.21.15.42:0/2278689051 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5da0064bf0 0x7f5da0143b30 secure :-1 s=THROTTLE_DONE pgs=1705 cs=0 l=1 rx=0x7f5d88000b80 tx=0x7f5d88010810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:25.610 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:25.606+0000 7f5d97fff700 1 -- 172.21.15.42:0/2278689051 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f5d88032100 con 0x7f5da0064bf0 2019-12-05T06:12:25.783 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:25.778+0000 7f5da4b63700 1 -- 172.21.15.42:0/2278689051 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5da005b8c0 con 0x7f5da0064bf0 2019-12-05T06:12:25.784 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:25.778+0000 7f5d9659c700 1 --2- 172.21.15.42:0/2278689051 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5da0064bf0 0x7f5da0143b30 secure :-1 s=THROTTLE_DONE pgs=1705 cs=0 l=1 rx=0x7f5d88000b80 tx=0x7f5d88010810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:25.784 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:25.778+0000 7f5d97fff700 1 -- 172.21.15.42:0/2278689051 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 272 v272) v1 ==== 95+0+1775 (secure 0 0 0) 0x7f5d88032360 con 0x7f5da0064bf0 2019-12-05T06:12:25.784 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 272 2019-12-05T06:12:25.785 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:12:25.785 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":272,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":272,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:12:15.021996+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":97,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7094},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7094":{"gid":7094,"name":"b","rank":0,"incarnation":269,"state":"up:active","state_seq":12,"addr":"172.21.15.40:6833/3785523496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3785523496},{"type":"v1","addr":"172.21.15.40:6833","nonce":3785523496}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:12:25.787 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:25.782+0000 7f5da4b63700 1 -- 172.21.15.42:0/2278689051 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5d8403ad90 msgr2=0x7f5d8403d2a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:25.788 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:25.782+0000 7f5da4b63700 1 --2- 172.21.15.42:0/2278689051 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5d8403ad90 0x7f5d8403d2a0 crc :-1 s=READY pgs=947 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:25.788 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:25.782+0000 7f5da4b63700 1 -- 172.21.15.42:0/2278689051 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5da0064bf0 msgr2=0x7f5da0143b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:25.788 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:25.782+0000 7f5da4b63700 1 --2- 172.21.15.42:0/2278689051 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5da0064bf0 0x7f5da0143b30 secure :-1 s=READY pgs=1705 cs=0 l=1 rx=0x7f5d88000b80 tx=0x7f5d88010810).stop 2019-12-05T06:12:25.788 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:25.782+0000 7f5da4b63700 1 -- 172.21.15.42:0/2278689051 shutdown_connections 2019-12-05T06:12:25.788 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:25.782+0000 7f5da4b63700 1 --2- 172.21.15.42:0/2278689051 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5d8403ad90 0x7f5d8403d2a0 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:25.788 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:25.782+0000 7f5da4b63700 1 --2- 172.21.15.42:0/2278689051 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5da0064bf0 0x7f5da0143b30 unknown :-1 s=CLOSED pgs=1705 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:25.789 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:25.782+0000 7f5da4b63700 1 -- 172.21.15.42:0/2278689051 shutdown_connections 2019-12-05T06:12:25.789 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:25.782+0000 7f5da4b63700 1 -- 172.21.15.42:0/2278689051 wait complete. 2019-12-05T06:12:25.789 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:25.782+0000 7f5da4b63700 1 -- 172.21.15.42:0/2278689051 >> 172.21.15.42:0/2278689051 conn(0x7f5da00a82a0 msgr2=0x7f5da009f350 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:12:25.797 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2019-12-05T06:12:27.799 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:12:27.914 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:27.902+0000 7fe6758ee700 1 Processor -- start 2019-12-05T06:12:27.914 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:27.902+0000 7fe6758ee700 1 -- start start 2019-12-05T06:12:27.914 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:27.906+0000 7fe6758ee700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe6700a7740 0x7fe67008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:12:27.914 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:27.906+0000 7fe6758ee700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe67009fae0 0x7fe67007eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:12:27.914 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:27.906+0000 7fe6758ee700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6700a0710 con 0x7fe67007e720 2019-12-05T06:12:27.914 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:27.906+0000 7fe6758ee700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6700a0b00 con 0x7fe6700a7740 2019-12-05T06:12:27.915 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:27.906+0000 7fe6758ee700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe67008e5b0 con 0x7fe67009fae0 2019-12-05T06:12:27.915 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:27.906+0000 7fe66f7fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe6700a7740 0x7fe67008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:12:27.915 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:27.906+0000 7fe66f7fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe6700a7740 0x7fe67008df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:50556/0 (socket says 172.21.15.42:50556) 2019-12-05T06:12:27.915 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:27.906+0000 7fe66f7fe700 1 -- 172.21.15.42:0/2572569272 learned_addr learned my addr 172.21.15.42:0/2572569272 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:12:27.915 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:27.906+0000 7fe66ffff700 1 --2- 172.21.15.42:0/2572569272 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe67009fae0 0x7fe67007eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:12:27.915 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:27.906+0000 7fe66f7fe700 1 -- 172.21.15.42:0/2572569272 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe67009fae0 msgr2=0x7fe67007eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:12:27.916 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:27.906+0000 7fe66f7fe700 1 --2- 172.21.15.42:0/2572569272 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe67009fae0 0x7fe67007eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:12:27.916 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:27.906+0000 7fe66f7fe700 1 -- 172.21.15.42:0/2572569272 >> v1:172.21.15.40:6789/0 conn(0x7fe67007e720 legacy=0x7fe67009f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:27.916 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:27.906+0000 7fe66f7fe700 1 -- 172.21.15.42:0/2572569272 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe67007f0a0 con 0x7fe6700a7740 2019-12-05T06:12:27.916 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:27.906+0000 7fe66f7fe700 1 --2- 172.21.15.42:0/2572569272 >> v2:172.21.15.42:3300/0 conn(0x7fe6700a7740 0x7fe67008df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fe660008790 tx=0x7fe66000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:27.916 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:27.906+0000 7fe66f7fe700 1 --2- 172.21.15.42:0/2572569272 >> v2:172.21.15.42:3300/0 conn(0x7fe6700a7740 0x7fe67008df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fe660008790 tx=0x7fe66000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:27.916 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:27.906+0000 7fe66f7fe700 1 --2- 172.21.15.42:0/2572569272 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe6700a7740 0x7fe67008df80 secure :-1 s=READY pgs=2258 cs=0 l=1 rx=0x7fe660008790 tx=0x7fe66000fd30).ready entity=mon.1 client_cookie=5db4c91a192c17e1 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:12:27.916 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:27.906+0000 7fe66f7fe700 1 --2- 172.21.15.42:0/2572569272 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe6700a7740 0x7fe67008df80 secure :-1 s=THROTTLE_DONE pgs=2258 cs=0 l=1 rx=0x7fe660008790 tx=0x7fe66000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:27.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:27.906+0000 7fe66e7fc700 1 -- 172.21.15.42:0/2572569272 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe660008c90 con 0x7fe6700a7740 2019-12-05T06:12:27.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:27.906+0000 7fe66f7fe700 1 --2- 172.21.15.42:0/2572569272 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe6700a7740 0x7fe67008df80 secure :-1 s=THROTTLE_DONE pgs=2258 cs=0 l=1 rx=0x7fe660008790 tx=0x7fe66000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:27.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:27.906+0000 7fe66e7fc700 1 -- 172.21.15.42:0/2572569272 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6600088b0 con 0x7fe6700a7740 2019-12-05T06:12:27.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:27.906+0000 7fe66f7fe700 1 --2- 172.21.15.42:0/2572569272 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe6700a7740 0x7fe67008df80 secure :-1 s=THROTTLE_DONE pgs=2258 cs=0 l=1 rx=0x7fe660008790 tx=0x7fe66000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:27.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:27.906+0000 7fe66e7fc700 1 -- 172.21.15.42:0/2572569272 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe660011b10 con 0x7fe6700a7740 2019-12-05T06:12:27.918 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:27.906+0000 7fe6758ee700 1 -- 172.21.15.42:0/2572569272 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe6700a7740 msgr2=0x7fe67008df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:27.918 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:27.906+0000 7fe6758ee700 1 --2- 172.21.15.42:0/2572569272 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe6700a7740 0x7fe67008df80 secure :-1 s=READY pgs=2258 cs=0 l=1 rx=0x7fe660008790 tx=0x7fe66000fd30).stop 2019-12-05T06:12:27.919 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:27.910+0000 7fe6758ee700 1 -- 172.21.15.42:0/2572569272 shutdown_connections 2019-12-05T06:12:27.919 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:27.910+0000 7fe6758ee700 1 --2- 172.21.15.42:0/2572569272 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe67009fae0 0x7fe67007eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:12:27.919 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:27.910+0000 7fe6758ee700 1 --2- 172.21.15.42:0/2572569272 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe6700a7740 0x7fe67008df80 unknown :-1 s=CLOSED pgs=2258 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:27.919 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:27.910+0000 7fe6758ee700 1 -- 172.21.15.42:0/2572569272 shutdown_connections 2019-12-05T06:12:27.919 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:27.910+0000 7fe6758ee700 1 -- 172.21.15.42:0/2572569272 wait complete. 2019-12-05T06:12:27.920 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:27.910+0000 7fe6758ee700 1 -- 172.21.15.42:0/2572569272 >> 172.21.15.42:0/2572569272 conn(0x7fe6700aabe0 msgr2=0x7fe6700a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:12:27.923 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:27.910+0000 7fe6758ee700 1 Processor -- start 2019-12-05T06:12:27.923 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:27.910+0000 7fe6758ee700 1 -- start start 2019-12-05T06:12:27.923 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:27.914+0000 7fe6758ee700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe670142a40 0x7fe670118500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:12:27.923 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:27.914+0000 7fe6758ee700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe67012d2b0 0x7fe670130c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:12:27.923 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:27.914+0000 7fe6758ee700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe670118a50 con 0x7fe67012ce80 2019-12-05T06:12:27.923 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:27.914+0000 7fe6758ee700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe670117210 con 0x7fe670142a40 2019-12-05T06:12:27.924 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:27.914+0000 7fe6758ee700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6701174b0 con 0x7fe67012d2b0 2019-12-05T06:12:27.924 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:27.914+0000 7fe66f7fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe670142a40 0x7fe670118500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:12:27.924 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:27.914+0000 7fe66f7fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe670142a40 0x7fe670118500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:50562/0 (socket says 172.21.15.42:50562) 2019-12-05T06:12:27.924 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:27.914+0000 7fe66f7fe700 1 -- 172.21.15.42:0/2849589541 learned_addr learned my addr 172.21.15.42:0/2849589541 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:12:27.924 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:27.914+0000 7fe66ffff700 1 --2- 172.21.15.42:0/2849589541 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe67012d2b0 0x7fe670130c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:12:27.924 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:27.914+0000 7fe66f7fe700 1 -- 172.21.15.42:0/2849589541 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe67012d2b0 msgr2=0x7fe670130c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:27.924 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:27.914+0000 7fe66f7fe700 1 --2- 172.21.15.42:0/2849589541 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe67012d2b0 0x7fe670130c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:27.925 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:27.914+0000 7fe657fff700 1 -- 172.21.15.42:0/2849589541 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 766857707 0 0) 0x7fe670118a50 con 0x7fe67012ce80 2019-12-05T06:12:27.925 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:27.914+0000 7fe66f7fe700 1 -- 172.21.15.42:0/2849589541 >> v1:172.21.15.40:6789/0 conn(0x7fe67012ce80 legacy=0x7fe670130440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:27.925 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:27.914+0000 7fe66f7fe700 1 -- 172.21.15.42:0/2849589541 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe670118e10 con 0x7fe670142a40 2019-12-05T06:12:27.925 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:27.914+0000 7fe66f7fe700 1 --2- 172.21.15.42:0/2849589541 >> v2:172.21.15.42:3300/0 conn(0x7fe670142a40 0x7fe670118500 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fe67012a9f0 tx=0x7fe66000f720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:27.925 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:27.914+0000 7fe66f7fe700 1 --2- 172.21.15.42:0/2849589541 >> v2:172.21.15.42:3300/0 conn(0x7fe670142a40 0x7fe670118500 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fe67012a9f0 tx=0x7fe66000f720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:27.925 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:27.914+0000 7fe66f7fe700 1 --2- 172.21.15.42:0/2849589541 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe670142a40 0x7fe670118500 secure :-1 s=READY pgs=2259 cs=0 l=1 rx=0x7fe67012a9f0 tx=0x7fe66000f720).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:12:27.925 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:27.914+0000 7fe66f7fe700 1 --2- 172.21.15.42:0/2849589541 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe670142a40 0x7fe670118500 secure :-1 s=THROTTLE_DONE pgs=2259 cs=0 l=1 rx=0x7fe67012a9f0 tx=0x7fe66000f720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:27.926 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:27.914+0000 7fe657fff700 1 -- 172.21.15.42:0/2849589541 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe660007350 con 0x7fe670142a40 2019-12-05T06:12:27.926 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:27.914+0000 7fe66f7fe700 1 --2- 172.21.15.42:0/2849589541 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe670142a40 0x7fe670118500 secure :-1 s=THROTTLE_DONE pgs=2259 cs=0 l=1 rx=0x7fe67012a9f0 tx=0x7fe66000f720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:27.926 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:27.914+0000 7fe66f7fe700 1 --2- 172.21.15.42:0/2849589541 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe670142a40 0x7fe670118500 secure :-1 s=THROTTLE_DONE pgs=2259 cs=0 l=1 rx=0x7fe67012a9f0 tx=0x7fe66000f720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:27.926 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:27.914+0000 7fe657fff700 1 -- 172.21.15.42:0/2849589541 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe660011730 con 0x7fe670142a40 2019-12-05T06:12:27.926 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:27.914+0000 7fe6758ee700 1 -- 172.21.15.42:0/2849589541 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe670131190 con 0x7fe670142a40 2019-12-05T06:12:27.926 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:27.914+0000 7fe657fff700 1 -- 172.21.15.42:0/2849589541 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe660005540 con 0x7fe670142a40 2019-12-05T06:12:27.926 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:27.914+0000 7fe6758ee700 1 -- 172.21.15.42:0/2849589541 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe670117c90 con 0x7fe670142a40 2019-12-05T06:12:27.927 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:27.914+0000 7fe66f7fe700 1 --2- 172.21.15.42:0/2849589541 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe670142a40 0x7fe670118500 secure :-1 s=THROTTLE_DONE pgs=2259 cs=0 l=1 rx=0x7fe67012a9f0 tx=0x7fe66000f720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:27.927 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:27.918+0000 7fe657fff700 1 -- 172.21.15.42:0/2849589541 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fe660005850 con 0x7fe670142a40 2019-12-05T06:12:27.927 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:27.918+0000 7fe66f7fe700 1 --2- 172.21.15.42:0/2849589541 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe670142a40 0x7fe670118500 secure :-1 s=THROTTLE_DONE pgs=2259 cs=0 l=1 rx=0x7fe67012a9f0 tx=0x7fe66000f720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:27.927 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:27.918+0000 7fe657fff700 1 --2- 172.21.15.42:0/2849589541 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe65003ad10 0x7fe65003d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:12:27.927 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:27.918+0000 7fe657fff700 1 -- 172.21.15.42:0/2849589541 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 10159+0+0 (secure 0 0 0) 0x7fe66004da80 con 0x7fe670142a40 2019-12-05T06:12:27.927 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:27.918+0000 7fe66effd700 1 --2- 172.21.15.42:0/2849589541 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe65003ad10 0x7fe65003d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:12:27.927 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:27.918+0000 7fe66effd700 1 --2- 172.21.15.42:0/2849589541 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe65003ad10 0x7fe65003d220 crc :-1 s=READY pgs=949 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:12:27.928 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:27.918+0000 7fe6758ee700 1 -- 172.21.15.42:0/2849589541 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe67005d110 con 0x7fe670142a40 2019-12-05T06:12:27.930 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:27.926+0000 7fe66f7fe700 1 --2- 172.21.15.42:0/2849589541 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe670142a40 0x7fe670118500 secure :-1 s=THROTTLE_DONE pgs=2259 cs=0 l=1 rx=0x7fe67012a9f0 tx=0x7fe66000f720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:27.930 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:27.926+0000 7fe657fff700 1 -- 172.21.15.42:0/2849589541 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fe660035910 con 0x7fe670142a40 2019-12-05T06:12:28.114 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:28.094+0000 7fe6758ee700 1 -- 172.21.15.42:0/2849589541 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe6700aadf0 con 0x7fe670142a40 2019-12-05T06:12:28.114 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:28.094+0000 7fe66f7fe700 1 --2- 172.21.15.42:0/2849589541 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe670142a40 0x7fe670118500 secure :-1 s=THROTTLE_DONE pgs=2259 cs=0 l=1 rx=0x7fe67012a9f0 tx=0x7fe66000f720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:28.114 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:28.094+0000 7fe657fff700 1 -- 172.21.15.42:0/2849589541 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 273 v273) v1 ==== 95+0+2113 (secure 0 0 0) 0x7fe66000dbf0 con 0x7fe670142a40 2019-12-05T06:12:28.114 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 273 2019-12-05T06:12:28.115 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:28.098+0000 7fe6758ee700 1 -- 172.21.15.42:0/2849589541 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe65003ad10 msgr2=0x7fe65003d220 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:28.115 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:28.098+0000 7fe6758ee700 1 --2- 172.21.15.42:0/2849589541 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe65003ad10 0x7fe65003d220 crc :-1 s=READY pgs=949 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:28.115 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:28.098+0000 7fe6758ee700 1 -- 172.21.15.42:0/2849589541 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe670142a40 msgr2=0x7fe670118500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:28.115 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:28.098+0000 7fe6758ee700 1 --2- 172.21.15.42:0/2849589541 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe670142a40 0x7fe670118500 secure :-1 s=READY pgs=2259 cs=0 l=1 rx=0x7fe67012a9f0 tx=0x7fe66000f720).stop 2019-12-05T06:12:28.115 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:28.098+0000 7fe6758ee700 1 -- 172.21.15.42:0/2849589541 shutdown_connections 2019-12-05T06:12:28.115 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:28.098+0000 7fe6758ee700 1 --2- 172.21.15.42:0/2849589541 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe65003ad10 0x7fe65003d220 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:28.115 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:28.098+0000 7fe6758ee700 1 --2- 172.21.15.42:0/2849589541 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe67012d2b0 0x7fe670130c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:28.116 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:28.098+0000 7fe6758ee700 1 --2- 172.21.15.42:0/2849589541 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe670142a40 0x7fe670118500 unknown :-1 s=CLOSED pgs=2259 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:28.116 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:28.098+0000 7fe6758ee700 1 -- 172.21.15.42:0/2849589541 shutdown_connections 2019-12-05T06:12:28.116 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:28.098+0000 7fe6758ee700 1 -- 172.21.15.42:0/2849589541 wait complete. 2019-12-05T06:12:28.116 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:28.098+0000 7fe6758ee700 1 -- 172.21.15.42:0/2849589541 >> 172.21.15.42:0/2849589541 conn(0x7fe6700a82a0 msgr2=0x7fe67009f4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:12:28.116 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:12:28.116 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":273,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":10246,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/4032647860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":4032647860},{"type":"v1","addr":"172.21.15.42:6835","nonce":4032647860}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":273}],"filesystems":[{"mdsmap":{"epoch":272,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:12:15.021996+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":97,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7094},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7094":{"gid":7094,"name":"b","rank":0,"incarnation":269,"state":"up:active","state_seq":12,"addr":"172.21.15.40:6833/3785523496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3785523496},{"type":"v1","addr":"172.21.15.40:6833","nonce":3785523496}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:12:28.117 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a reported in up:standby state 2019-12-05T06:12:28.117 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 30 secs before thrashing 2019-12-05T06:12:44.982 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:12:44.987 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:12:58.118 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:12:58.223 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.213+0000 7f124bc88700 1 Processor -- start 2019-12-05T06:12:58.223 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.213+0000 7f124bc88700 1 -- start start 2019-12-05T06:12:58.223 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.213+0000 7f124bc88700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f12440a7740 0x7f124408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:12:58.223 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.213+0000 7f124bc88700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f124409fae0 0x7f124407eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:12:58.223 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.213+0000 7f124bc88700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12440a0710 con 0x7f124407e720 2019-12-05T06:12:58.223 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.213+0000 7f124bc88700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12440a0b00 con 0x7f12440a7740 2019-12-05T06:12:58.224 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.213+0000 7f124bc88700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f124408e5b0 con 0x7f124409fae0 2019-12-05T06:12:58.224 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.213+0000 7f1249a24700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f124407e720 0x7f124409f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:50726/0 (socket says 172.21.15.42:50726) 2019-12-05T06:12:58.224 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.213+0000 7f1249a24700 1 -- 172.21.15.42:0/36710466 learned_addr learned my addr 172.21.15.42:0/36710466 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:12:58.224 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.213+0000 7f124a225700 1 --2- 172.21.15.42:0/36710466 >> v2:172.21.15.40:3300/0 conn(0x7f12440a7740 0x7f124408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:12:58.224 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.213+0000 7f124aa26700 1 --2- 172.21.15.42:0/36710466 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f124409fae0 0x7f124407eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:12:58.224 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.213+0000 7f1249223700 1 -- 172.21.15.42:0/36710466 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 40959112 0 0) 0x7f12440a0710 con 0x7f124407e720 2019-12-05T06:12:58.224 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.213+0000 7f1249223700 1 -- 172.21.15.42:0/36710466 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12300030b0 con 0x7f124407e720 2019-12-05T06:12:58.225 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.213+0000 7f1249223700 1 -- 172.21.15.42:0/36710466 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2022263993 0 0) 0x7f12300030b0 con 0x7f124407e720 2019-12-05T06:12:58.225 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.213+0000 7f1249223700 1 -- 172.21.15.42:0/36710466 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f124409fae0 msgr2=0x7f124407eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:12:58.225 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.213+0000 7f1249223700 1 --2- 172.21.15.42:0/36710466 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f124409fae0 0x7f124407eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:12:58.225 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.213+0000 7f1249223700 1 -- 172.21.15.42:0/36710466 >> v2:172.21.15.40:3300/0 conn(0x7f12440a7740 msgr2=0x7f124408df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:12:58.225 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.213+0000 7f1249223700 1 --2- 172.21.15.42:0/36710466 >> v2:172.21.15.40:3300/0 conn(0x7f12440a7740 0x7f124408df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:12:58.225 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.213+0000 7f1249223700 1 -- 172.21.15.42:0/36710466 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f124407f0a0 con 0x7f124407e720 2019-12-05T06:12:58.225 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.213+0000 7f1249223700 1 -- 172.21.15.42:0/36710466 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7f1234001750 con 0x7f124407e720 2019-12-05T06:12:58.226 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.213+0000 7f1249223700 1 -- 172.21.15.42:0/36710466 >> v1:172.21.15.42:6789/0 conn(0x7f124407e720 legacy=0x7f124409f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:58.226 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.213+0000 7f1249223700 1 --2- 172.21.15.42:0/36710466 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f12300036e0 0x7f1230003b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:12:58.226 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.213+0000 7f1249223700 1 --2- 172.21.15.42:0/36710466 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1230004420 0x7f1230004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:12:58.226 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.213+0000 7f1249223700 1 --2- 172.21.15.42:0/36710466 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1230004da0 0x7f123004f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:12:58.226 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.213+0000 7f1249223700 1 -- 172.21.15.42:0/36710466 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12440a0b00 con 0x7f12300036e0 2019-12-05T06:12:58.226 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.213+0000 7f1249223700 1 -- 172.21.15.42:0/36710466 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f124408e5b0 con 0x7f1230004420 2019-12-05T06:12:58.226 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.213+0000 7f1249223700 1 -- 172.21.15.42:0/36710466 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12300051d0 con 0x7f1230004da0 2019-12-05T06:12:58.227 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.213+0000 7f1249a24700 1 --2- 172.21.15.42:0/36710466 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1230004420 0x7f1230004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:12:58.227 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.213+0000 7f124a225700 1 --2- 172.21.15.42:0/36710466 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f12300036e0 0x7f1230003b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:12:58.227 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.213+0000 7f124aa26700 1 --2- 172.21.15.42:0/36710466 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1230004da0 0x7f123004f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:12:58.227 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.217+0000 7f1249a24700 1 -- 172.21.15.42:0/36710466 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1230004da0 msgr2=0x7f123004f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:12:58.227 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.217+0000 7f1249a24700 1 --2- 172.21.15.42:0/36710466 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1230004da0 0x7f123004f680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:12:58.227 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.217+0000 7f1249a24700 1 -- 172.21.15.42:0/36710466 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f12300036e0 msgr2=0x7f1230003b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:12:58.227 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.217+0000 7f1249a24700 1 --2- 172.21.15.42:0/36710466 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f12300036e0 0x7f1230003b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:12:58.228 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.217+0000 7f1249a24700 1 -- 172.21.15.42:0/36710466 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f123004fc30 con 0x7f1230004420 2019-12-05T06:12:58.228 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.217+0000 7f1249a24700 1 --2- 172.21.15.42:0/36710466 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1230004420 0x7f1230004850 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f1234001a50 tx=0x7f1234010be0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:58.228 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.217+0000 7f1249a24700 1 --2- 172.21.15.42:0/36710466 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1230004420 0x7f1230004850 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f1234001a50 tx=0x7f1234010be0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:58.228 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.217+0000 7f1249a24700 1 --2- 172.21.15.42:0/36710466 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1230004420 0x7f1230004850 secure :-1 s=READY pgs=2262 cs=0 l=1 rx=0x7f1234001a50 tx=0x7f1234010be0).ready entity=mon.1 client_cookie=8470983e66760f50 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:12:58.228 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.217+0000 7f1249a24700 1 --2- 172.21.15.42:0/36710466 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1230004420 0x7f1230004850 secure :-1 s=THROTTLE_DONE pgs=2262 cs=0 l=1 rx=0x7f1234001a50 tx=0x7f1234010be0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:58.228 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.217+0000 7f1249a24700 1 --2- 172.21.15.42:0/36710466 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1230004420 0x7f1230004850 secure :-1 s=THROTTLE_DONE pgs=2262 cs=0 l=1 rx=0x7f1234001a50 tx=0x7f1234010be0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:58.228 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.217+0000 7f1249223700 1 -- 172.21.15.42:0/36710466 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f123400e0c0 con 0x7f1230004420 2019-12-05T06:12:58.229 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.217+0000 7f1249a24700 1 --2- 172.21.15.42:0/36710466 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1230004420 0x7f1230004850 secure :-1 s=THROTTLE_DONE pgs=2262 cs=0 l=1 rx=0x7f1234001a50 tx=0x7f1234010be0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:58.229 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.217+0000 7f1249223700 1 -- 172.21.15.42:0/36710466 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1234011530 con 0x7f1230004420 2019-12-05T06:12:58.229 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.217+0000 7f1249223700 1 -- 172.21.15.42:0/36710466 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f123400e6a0 con 0x7f1230004420 2019-12-05T06:12:58.229 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.217+0000 7f124bc88700 1 -- 172.21.15.42:0/36710466 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1230004420 msgr2=0x7f1230004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:58.229 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.217+0000 7f124bc88700 1 --2- 172.21.15.42:0/36710466 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1230004420 0x7f1230004850 secure :-1 s=READY pgs=2262 cs=0 l=1 rx=0x7f1234001a50 tx=0x7f1234010be0).stop 2019-12-05T06:12:58.229 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.217+0000 7f124aa26700 1 -- 172.21.15.42:0/36710466 reap_dead start 2019-12-05T06:12:58.229 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.217+0000 7f124bc88700 1 -- 172.21.15.42:0/36710466 shutdown_connections 2019-12-05T06:12:58.230 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.217+0000 7f124bc88700 1 -- 172.21.15.42:0/36710466 shutdown_connections 2019-12-05T06:12:58.230 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.217+0000 7f124bc88700 1 -- 172.21.15.42:0/36710466 wait complete. 2019-12-05T06:12:58.230 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.217+0000 7f124bc88700 1 -- 172.21.15.42:0/36710466 >> 172.21.15.42:0/36710466 conn(0x7f12440aabe0 msgr2=0x7f12440a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:12:58.233 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.221+0000 7f124bc88700 1 Processor -- start 2019-12-05T06:12:58.233 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.221+0000 7f124bc88700 1 -- start start 2019-12-05T06:12:58.233 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.221+0000 7f124bc88700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f12441379b0 0x7f1244139df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:12:58.234 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.221+0000 7f124bc88700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12441246f0 con 0x7f12440a28f0 2019-12-05T06:12:58.234 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.221+0000 7f124bc88700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12440a4840 con 0x7f12440a6ba0 2019-12-05T06:12:58.234 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.221+0000 7f124bc88700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12440a23b0 con 0x7f12441379b0 2019-12-05T06:12:58.234 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.221+0000 7f124a225700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f12440a6ba0 0x7f124408eb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:50736/0 (socket says 172.21.15.42:50736) 2019-12-05T06:12:58.234 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.221+0000 7f124a225700 1 -- 172.21.15.42:0/1528650166 learned_addr learned my addr 172.21.15.42:0/1528650166 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:12:58.234 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.221+0000 7f124aa26700 1 --2- 172.21.15.42:0/1528650166 >> v2:172.21.15.42:3300/0 conn(0x7f12441379b0 0x7f1244139df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:12:58.234 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.221+0000 7f123affd700 1 -- 172.21.15.42:0/1528650166 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3729339014 0 0) 0x7f12440a4840 con 0x7f12440a6ba0 2019-12-05T06:12:58.235 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.221+0000 7f123affd700 1 -- 172.21.15.42:0/1528650166 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f122c0030b0 con 0x7f12440a6ba0 2019-12-05T06:12:58.235 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.221+0000 7f123affd700 1 -- 172.21.15.42:0/1528650166 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1679039247 0 0) 0x7f12441246f0 con 0x7f12440a28f0 2019-12-05T06:12:58.235 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.221+0000 7f123affd700 1 -- 172.21.15.42:0/1528650166 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12440a4840 con 0x7f12440a28f0 2019-12-05T06:12:58.235 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.221+0000 7f124aa26700 1 -- 172.21.15.42:0/1528650166 >> v1:172.21.15.42:6789/0 conn(0x7f12440a6ba0 legacy=0x7f124408eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:58.235 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.221+0000 7f124aa26700 1 -- 172.21.15.42:0/1528650166 >> v1:172.21.15.40:6789/0 conn(0x7f12440a28f0 legacy=0x7f124407ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:58.235 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.221+0000 7f124aa26700 1 -- 172.21.15.42:0/1528650166 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12440a6470 con 0x7f12441379b0 2019-12-05T06:12:58.235 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.221+0000 7f124aa26700 1 --2- 172.21.15.42:0/1528650166 >> v2:172.21.15.42:3300/0 conn(0x7f12441379b0 0x7f1244139df0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f12440a70d0 tx=0x7f123c0109b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:58.236 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.221+0000 7f124aa26700 1 --2- 172.21.15.42:0/1528650166 >> v2:172.21.15.42:3300/0 conn(0x7f12441379b0 0x7f1244139df0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f12440a70d0 tx=0x7f123c0109b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:58.236 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.221+0000 7f124aa26700 1 --2- 172.21.15.42:0/1528650166 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f12441379b0 0x7f1244139df0 secure :-1 s=READY pgs=2264 cs=0 l=1 rx=0x7f12440a70d0 tx=0x7f123c0109b0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:12:58.236 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.225+0000 7f124aa26700 1 --2- 172.21.15.42:0/1528650166 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f12441379b0 0x7f1244139df0 secure :-1 s=THROTTLE_DONE pgs=2264 cs=0 l=1 rx=0x7f12440a70d0 tx=0x7f123c0109b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:58.236 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.225+0000 7f124aa26700 1 --2- 172.21.15.42:0/1528650166 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f12441379b0 0x7f1244139df0 secure :-1 s=THROTTLE_DONE pgs=2264 cs=0 l=1 rx=0x7f12440a70d0 tx=0x7f123c0109b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:58.236 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.225+0000 7f123affd700 1 -- 172.21.15.42:0/1528650166 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f123c007160 con 0x7f12441379b0 2019-12-05T06:12:58.236 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.225+0000 7f124aa26700 1 --2- 172.21.15.42:0/1528650166 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f12441379b0 0x7f1244139df0 secure :-1 s=THROTTLE_DONE pgs=2264 cs=0 l=1 rx=0x7f12440a70d0 tx=0x7f123c0109b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:58.236 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.225+0000 7f123affd700 1 -- 172.21.15.42:0/1528650166 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f123c005bc0 con 0x7f12441379b0 2019-12-05T06:12:58.237 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.225+0000 7f124bc88700 1 -- 172.21.15.42:0/1528650166 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f124407b470 con 0x7f12441379b0 2019-12-05T06:12:58.237 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.225+0000 7f123affd700 1 -- 172.21.15.42:0/1528650166 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f123c0082d0 con 0x7f12441379b0 2019-12-05T06:12:58.237 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.225+0000 7f124bc88700 1 -- 172.21.15.42:0/1528650166 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f124413a340 con 0x7f12441379b0 2019-12-05T06:12:58.237 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.225+0000 7f124aa26700 1 --2- 172.21.15.42:0/1528650166 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f12441379b0 0x7f1244139df0 secure :-1 s=THROTTLE_DONE pgs=2264 cs=0 l=1 rx=0x7f12440a70d0 tx=0x7f123c0109b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:58.237 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.225+0000 7f123affd700 1 -- 172.21.15.42:0/1528650166 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f123c0084d0 con 0x7f12441379b0 2019-12-05T06:12:58.237 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.225+0000 7f124aa26700 1 --2- 172.21.15.42:0/1528650166 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f12441379b0 0x7f1244139df0 secure :-1 s=THROTTLE_DONE pgs=2264 cs=0 l=1 rx=0x7f12440a70d0 tx=0x7f123c0109b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:58.237 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.225+0000 7f123affd700 1 --2- 172.21.15.42:0/1528650166 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f122c03b4a0 0x7f122c03d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:12:58.238 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.225+0000 7f123affd700 1 -- 172.21.15.42:0/1528650166 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 10159+0+0 (secure 0 0 0) 0x7f123c04b290 con 0x7f12441379b0 2019-12-05T06:12:58.238 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.225+0000 7f124a225700 1 --2- 172.21.15.42:0/1528650166 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f122c03b4a0 0x7f122c03d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:12:58.238 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.225+0000 7f124a225700 1 --2- 172.21.15.42:0/1528650166 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f122c03b4a0 0x7f122c03d9b0 crc :-1 s=READY pgs=950 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:12:58.238 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.225+0000 7f124bc88700 1 -- 172.21.15.42:0/1528650166 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f124405d110 con 0x7f12441379b0 2019-12-05T06:12:58.240 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.237+0000 7f124aa26700 1 --2- 172.21.15.42:0/1528650166 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f12441379b0 0x7f1244139df0 secure :-1 s=THROTTLE_DONE pgs=2264 cs=0 l=1 rx=0x7f12440a70d0 tx=0x7f123c0109b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:58.240 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.237+0000 7f123affd700 1 -- 172.21.15.42:0/1528650166 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f123c007740 con 0x7f12441379b0 2019-12-05T06:12:58.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.401+0000 7f124bc88700 1 -- 172.21.15.42:0/1528650166 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f12440a2190 con 0x7f12441379b0 2019-12-05T06:12:58.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.401+0000 7f124aa26700 1 --2- 172.21.15.42:0/1528650166 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f12441379b0 0x7f1244139df0 secure :-1 s=THROTTLE_DONE pgs=2264 cs=0 l=1 rx=0x7f12440a70d0 tx=0x7f123c0109b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:58.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.401+0000 7f123affd700 1 -- 172.21.15.42:0/1528650166 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 273 v273) v1 ==== 95+0+2113 (secure 0 0 0) 0x7f123c032310 con 0x7f12441379b0 2019-12-05T06:12:58.407 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 273 2019-12-05T06:12:58.407 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:12:58.408 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":273,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":10246,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/4032647860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":4032647860},{"type":"v1","addr":"172.21.15.42:6835","nonce":4032647860}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":273}],"filesystems":[{"mdsmap":{"epoch":272,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:12:15.021996+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":97,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7094},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7094":{"gid":7094,"name":"b","rank":0,"incarnation":269,"state":"up:active","state_seq":12,"addr":"172.21.15.40:6833/3785523496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3785523496},{"type":"v1","addr":"172.21.15.40:6833","nonce":3785523496}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:12:58.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.405+0000 7f124bc88700 1 -- 172.21.15.42:0/1528650166 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f122c03b4a0 msgr2=0x7f122c03d9b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:58.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.405+0000 7f124bc88700 1 --2- 172.21.15.42:0/1528650166 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f122c03b4a0 0x7f122c03d9b0 crc :-1 s=READY pgs=950 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:58.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.405+0000 7f124bc88700 1 -- 172.21.15.42:0/1528650166 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f12441379b0 msgr2=0x7f1244139df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:58.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.405+0000 7f124bc88700 1 --2- 172.21.15.42:0/1528650166 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f12441379b0 0x7f1244139df0 secure :-1 s=READY pgs=2264 cs=0 l=1 rx=0x7f12440a70d0 tx=0x7f123c0109b0).stop 2019-12-05T06:12:58.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.405+0000 7f124bc88700 1 -- 172.21.15.42:0/1528650166 shutdown_connections 2019-12-05T06:12:58.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.405+0000 7f124bc88700 1 --2- 172.21.15.42:0/1528650166 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f122c03b4a0 0x7f122c03d9b0 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:58.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.405+0000 7f124bc88700 1 --2- 172.21.15.42:0/1528650166 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f12441379b0 0x7f1244139df0 unknown :-1 s=CLOSED pgs=2264 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:58.411 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.405+0000 7f124bc88700 1 -- 172.21.15.42:0/1528650166 shutdown_connections 2019-12-05T06:12:58.411 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.405+0000 7f124bc88700 1 -- 172.21.15.42:0/1528650166 wait complete. 2019-12-05T06:12:58.411 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.405+0000 7f124bc88700 1 -- 172.21.15.42:0/1528650166 >> 172.21.15.42:0/1528650166 conn(0x7f124411c6a0 msgr2=0x7f12440a0110 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:12:58.419 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.b (rank=0) 2019-12-05T06:12:58.420 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2019-12-05T06:12:58.420 INFO:teuthology.orchestra.run:waiting for 300 2019-12-05T06:12:58.437 INFO:tasks.ceph.mds.b:Stopped 2019-12-05T06:12:58.437 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:12:58.542 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.533+0000 7f3d492d9700 1 Processor -- start 2019-12-05T06:12:58.543 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.533+0000 7f3d492d9700 1 -- start start 2019-12-05T06:12:58.543 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.533+0000 7f3d492d9700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3d440a7740 0x7f3d4408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:12:58.543 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.533+0000 7f3d492d9700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d440a0710 con 0x7f3d4407e720 2019-12-05T06:12:58.543 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.533+0000 7f3d492d9700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d440a9ba0 con 0x7f3d4409fae0 2019-12-05T06:12:58.544 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.533+0000 7f3d492d9700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d440a0b00 con 0x7f3d440a7740 2019-12-05T06:12:58.544 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.533+0000 7f3d43fff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f3d4409fae0 0x7f3d4414efe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:50746/0 (socket says 172.21.15.42:50746) 2019-12-05T06:12:58.544 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.533+0000 7f3d43fff700 1 -- 172.21.15.42:0/3301285936 learned_addr learned my addr 172.21.15.42:0/3301285936 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:12:58.544 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.533+0000 7f3d437fe700 1 --2- 172.21.15.42:0/3301285936 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3d440a7740 0x7f3d4408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:12:58.544 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.533+0000 7f3d427fc700 1 -- 172.21.15.42:0/3301285936 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2742739889 0 0) 0x7f3d440a9ba0 con 0x7f3d4409fae0 2019-12-05T06:12:58.544 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.533+0000 7f3d427fc700 1 -- 172.21.15.42:0/3301285936 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d300030b0 con 0x7f3d4409fae0 2019-12-05T06:12:58.544 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.533+0000 7f3d427fc700 1 -- 172.21.15.42:0/3301285936 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3436506968 0 0) 0x7f3d440a0710 con 0x7f3d4407e720 2019-12-05T06:12:58.545 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.533+0000 7f3d427fc700 1 -- 172.21.15.42:0/3301285936 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d440a9ba0 con 0x7f3d4407e720 2019-12-05T06:12:58.545 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.533+0000 7f3d437fe700 1 -- 172.21.15.42:0/3301285936 >> v1:172.21.15.42:6789/0 conn(0x7f3d4409fae0 legacy=0x7f3d4414efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:58.545 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.533+0000 7f3d437fe700 1 -- 172.21.15.42:0/3301285936 >> v1:172.21.15.40:6789/0 conn(0x7f3d4407e720 legacy=0x7f3d4409f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:58.545 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.533+0000 7f3d437fe700 1 -- 172.21.15.42:0/3301285936 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d4407eb50 con 0x7f3d440a7740 2019-12-05T06:12:58.545 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.533+0000 7f3d437fe700 1 --2- 172.21.15.42:0/3301285936 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3d440a7740 0x7f3d4408df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f3d34002f30 tx=0x7f3d3400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:58.545 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.533+0000 7f3d437fe700 1 --2- 172.21.15.42:0/3301285936 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3d440a7740 0x7f3d4408df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f3d34002f30 tx=0x7f3d3400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:58.545 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.533+0000 7f3d437fe700 1 --2- 172.21.15.42:0/3301285936 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3d440a7740 0x7f3d4408df80 secure :-1 s=READY pgs=498 cs=0 l=1 rx=0x7f3d34002f30 tx=0x7f3d3400fbf0).ready entity=mon.2 client_cookie=ad7827d921224eda server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:12:58.546 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.537+0000 7f3d437fe700 1 --2- 172.21.15.42:0/3301285936 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3d440a7740 0x7f3d4408df80 secure :-1 s=THROTTLE_DONE pgs=498 cs=0 l=1 rx=0x7f3d34002f30 tx=0x7f3d3400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:58.546 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.537+0000 7f3d427fc700 1 -- 172.21.15.42:0/3301285936 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3d340112b0 con 0x7f3d440a7740 2019-12-05T06:12:58.546 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.537+0000 7f3d437fe700 1 --2- 172.21.15.42:0/3301285936 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3d440a7740 0x7f3d4408df80 secure :-1 s=THROTTLE_DONE pgs=498 cs=0 l=1 rx=0x7f3d34002f30 tx=0x7f3d3400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:58.546 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.537+0000 7f3d427fc700 1 -- 172.21.15.42:0/3301285936 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d34008bc0 con 0x7f3d440a7740 2019-12-05T06:12:58.546 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.537+0000 7f3d437fe700 1 --2- 172.21.15.42:0/3301285936 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3d440a7740 0x7f3d4408df80 secure :-1 s=THROTTLE_DONE pgs=498 cs=0 l=1 rx=0x7f3d34002f30 tx=0x7f3d3400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:58.546 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.537+0000 7f3d427fc700 1 -- 172.21.15.42:0/3301285936 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3d34011890 con 0x7f3d440a7740 2019-12-05T06:12:58.547 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.537+0000 7f3d492d9700 1 -- 172.21.15.42:0/3301285936 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3d440a7740 msgr2=0x7f3d4408df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:58.547 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.537+0000 7f3d492d9700 1 --2- 172.21.15.42:0/3301285936 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3d440a7740 0x7f3d4408df80 secure :-1 s=READY pgs=498 cs=0 l=1 rx=0x7f3d34002f30 tx=0x7f3d3400fbf0).stop 2019-12-05T06:12:58.547 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.537+0000 7f3d492d9700 1 -- 172.21.15.42:0/3301285936 shutdown_connections 2019-12-05T06:12:58.547 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.537+0000 7f3d492d9700 1 --2- 172.21.15.42:0/3301285936 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3d440a7740 0x7f3d4408df80 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:58.547 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.537+0000 7f3d492d9700 1 -- 172.21.15.42:0/3301285936 shutdown_connections 2019-12-05T06:12:58.547 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.537+0000 7f3d492d9700 1 -- 172.21.15.42:0/3301285936 wait complete. 2019-12-05T06:12:58.547 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.537+0000 7f3d492d9700 1 -- 172.21.15.42:0/3301285936 >> 172.21.15.42:0/3301285936 conn(0x7f3d440aabe0 msgr2=0x7f3d440a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:12:58.550 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.537+0000 7f3d492d9700 1 Processor -- start 2019-12-05T06:12:58.550 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.541+0000 7f3d492d9700 1 -- start start 2019-12-05T06:12:58.551 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.541+0000 7f3d492d9700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f3d44126100 0x7f3d440a3e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:12:58.551 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.541+0000 7f3d492d9700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3d44144f30 0x7f3d44125680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:12:58.551 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.541+0000 7f3d492d9700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d44145360 con 0x7f3d44142530 2019-12-05T06:12:58.551 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.541+0000 7f3d492d9700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d44126530 con 0x7f3d44126100 2019-12-05T06:12:58.551 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.541+0000 7f3d492d9700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d44125bd0 con 0x7f3d44144f30 2019-12-05T06:12:58.551 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.541+0000 7f3d437fe700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f3d44126100 0x7f3d440a3e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:12:58.552 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.541+0000 7f3d42ffd700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3d44144f30 0x7f3d44125680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:12:58.552 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.541+0000 7f3d437fe700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f3d44126100 0x7f3d440a3e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.42:38530/0 (socket says 172.21.15.42:38530) 2019-12-05T06:12:58.552 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.541+0000 7f3d437fe700 1 -- 172.21.15.42:0/2661863305 learned_addr learned my addr 172.21.15.42:0/2661863305 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:12:58.552 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.541+0000 7f3d437fe700 1 -- 172.21.15.42:0/2661863305 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3d44144f30 msgr2=0x7f3d44125680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:58.552 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.541+0000 7f3d437fe700 1 --2- 172.21.15.42:0/2661863305 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3d44144f30 0x7f3d44125680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:58.552 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.541+0000 7f3d437fe700 1 -- 172.21.15.42:0/2661863305 >> v1:172.21.15.40:6789/0 conn(0x7f3d44142530 legacy=0x7f3d44122900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:58.552 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.541+0000 7f3d3bfff700 1 -- 172.21.15.42:0/2661863305 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3883959115 0 0) 0x7f3d44145360 con 0x7f3d44142530 2019-12-05T06:12:58.553 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.541+0000 7f3d437fe700 1 -- 172.21.15.42:0/2661863305 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d44142960 con 0x7f3d44126100 2019-12-05T06:12:58.553 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.541+0000 7f3d42ffd700 1 --2- 172.21.15.42:0/2661863305 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3d44144f30 0x7f3d44125680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:12:58.553 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.541+0000 7f3d437fe700 1 --2- 172.21.15.42:0/2661863305 >> v2:172.21.15.40:3300/0 conn(0x7f3d44126100 0x7f3d440a3e70 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f3d34002f30 tx=0x7f3d34007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:58.553 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.541+0000 7f3d437fe700 1 --2- 172.21.15.42:0/2661863305 >> v2:172.21.15.40:3300/0 conn(0x7f3d44126100 0x7f3d440a3e70 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f3d34002f30 tx=0x7f3d34007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:58.553 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.541+0000 7f3d437fe700 1 --2- 172.21.15.42:0/2661863305 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3d44126100 0x7f3d440a3e70 secure :-1 s=READY pgs=1717 cs=0 l=1 rx=0x7f3d34002f30 tx=0x7f3d34007710).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:12:58.553 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.541+0000 7f3d437fe700 1 --2- 172.21.15.42:0/2661863305 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3d44126100 0x7f3d440a3e70 secure :-1 s=THROTTLE_DONE pgs=1717 cs=0 l=1 rx=0x7f3d34002f30 tx=0x7f3d34007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:58.554 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.541+0000 7f3d3bfff700 1 -- 172.21.15.42:0/2661863305 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3d34007340 con 0x7f3d44126100 2019-12-05T06:12:58.554 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.541+0000 7f3d437fe700 1 --2- 172.21.15.42:0/2661863305 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3d44126100 0x7f3d440a3e70 secure :-1 s=THROTTLE_DONE pgs=1717 cs=0 l=1 rx=0x7f3d34002f30 tx=0x7f3d34007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:58.554 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.541+0000 7f3d437fe700 1 --2- 172.21.15.42:0/2661863305 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3d44126100 0x7f3d440a3e70 secure :-1 s=THROTTLE_DONE pgs=1717 cs=0 l=1 rx=0x7f3d34002f30 tx=0x7f3d34007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:58.554 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.541+0000 7f3d3bfff700 1 -- 172.21.15.42:0/2661863305 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d340114b0 con 0x7f3d44126100 2019-12-05T06:12:58.554 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.541+0000 7f3d3bfff700 1 -- 172.21.15.42:0/2661863305 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3d3400f360 con 0x7f3d44126100 2019-12-05T06:12:58.554 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.541+0000 7f3d492d9700 1 -- 172.21.15.42:0/2661863305 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d4408eb70 con 0x7f3d44126100 2019-12-05T06:12:58.554 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.541+0000 7f3d492d9700 1 -- 172.21.15.42:0/2661863305 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d4408f070 con 0x7f3d44126100 2019-12-05T06:12:58.555 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.545+0000 7f3d437fe700 1 --2- 172.21.15.42:0/2661863305 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3d44126100 0x7f3d440a3e70 secure :-1 s=THROTTLE_DONE pgs=1717 cs=0 l=1 rx=0x7f3d34002f30 tx=0x7f3d34007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:58.555 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.545+0000 7f3d3bfff700 1 -- 172.21.15.42:0/2661863305 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f3d34005540 con 0x7f3d44126100 2019-12-05T06:12:58.555 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.545+0000 7f3d437fe700 1 --2- 172.21.15.42:0/2661863305 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3d44126100 0x7f3d440a3e70 secure :-1 s=THROTTLE_DONE pgs=1717 cs=0 l=1 rx=0x7f3d34002f30 tx=0x7f3d34007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:58.555 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.545+0000 7f3d3bfff700 1 --2- 172.21.15.42:0/2661863305 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3d2803acd0 0x7f3d2803d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:12:58.555 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.545+0000 7f3d3bfff700 1 -- 172.21.15.42:0/2661863305 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 10159+0+0 (secure 0 0 0) 0x7f3d3404b710 con 0x7f3d44126100 2019-12-05T06:12:58.555 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.545+0000 7f3d42ffd700 1 --2- 172.21.15.42:0/2661863305 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3d2803acd0 0x7f3d2803d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:12:58.555 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.545+0000 7f3d42ffd700 1 --2- 172.21.15.42:0/2661863305 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3d2803acd0 0x7f3d2803d1e0 crc :-1 s=READY pgs=951 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:12:58.556 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.549+0000 7f3d492d9700 1 -- 172.21.15.42:0/2661863305 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d4405d110 con 0x7f3d44126100 2019-12-05T06:12:58.563 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.557+0000 7f3d437fe700 1 --2- 172.21.15.42:0/2661863305 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3d44126100 0x7f3d440a3e70 secure :-1 s=THROTTLE_DONE pgs=1717 cs=0 l=1 rx=0x7f3d34002f30 tx=0x7f3d34007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:58.564 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.557+0000 7f3d3bfff700 1 -- 172.21.15.42:0/2661863305 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f3d3400f600 con 0x7f3d44126100 2019-12-05T06:12:58.737 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.729+0000 7f3d492d9700 1 -- 172.21.15.42:0/2661863305 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3d4405b8c0 con 0x7f3d44126100 2019-12-05T06:12:58.737 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.729+0000 7f3d437fe700 1 --2- 172.21.15.42:0/2661863305 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3d44126100 0x7f3d440a3e70 secure :-1 s=THROTTLE_DONE pgs=1717 cs=0 l=1 rx=0x7f3d34002f30 tx=0x7f3d34007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:58.737 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.729+0000 7f3d3bfff700 1 -- 172.21.15.42:0/2661863305 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 273 v273) v1 ==== 95+0+2113 (secure 0 0 0) 0x7f3d34032930 con 0x7f3d44126100 2019-12-05T06:12:58.738 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 273 2019-12-05T06:12:58.738 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:12:58.738 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":273,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":10246,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/4032647860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":4032647860},{"type":"v1","addr":"172.21.15.42:6835","nonce":4032647860}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":273}],"filesystems":[{"mdsmap":{"epoch":272,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:12:15.021996+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":97,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7094},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7094":{"gid":7094,"name":"b","rank":0,"incarnation":269,"state":"up:active","state_seq":12,"addr":"172.21.15.40:6833/3785523496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3785523496},{"type":"v1","addr":"172.21.15.40:6833","nonce":3785523496}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:12:58.739 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.733+0000 7f3d492d9700 1 -- 172.21.15.42:0/2661863305 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3d2803acd0 msgr2=0x7f3d2803d1e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:58.739 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.733+0000 7f3d492d9700 1 --2- 172.21.15.42:0/2661863305 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3d2803acd0 0x7f3d2803d1e0 crc :-1 s=READY pgs=951 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:58.739 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.733+0000 7f3d492d9700 1 -- 172.21.15.42:0/2661863305 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3d44126100 msgr2=0x7f3d440a3e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:58.739 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.733+0000 7f3d492d9700 1 --2- 172.21.15.42:0/2661863305 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3d44126100 0x7f3d440a3e70 secure :-1 s=READY pgs=1717 cs=0 l=1 rx=0x7f3d34002f30 tx=0x7f3d34007710).stop 2019-12-05T06:12:58.740 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.733+0000 7f3d492d9700 1 -- 172.21.15.42:0/2661863305 shutdown_connections 2019-12-05T06:12:58.740 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.733+0000 7f3d492d9700 1 --2- 172.21.15.42:0/2661863305 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3d2803acd0 0x7f3d2803d1e0 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:58.740 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.733+0000 7f3d492d9700 1 --2- 172.21.15.42:0/2661863305 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3d44126100 0x7f3d440a3e70 unknown :-1 s=CLOSED pgs=1717 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:58.740 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.733+0000 7f3d492d9700 1 --2- 172.21.15.42:0/2661863305 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3d44144f30 0x7f3d44125680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:58.740 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.733+0000 7f3d492d9700 1 -- 172.21.15.42:0/2661863305 shutdown_connections 2019-12-05T06:12:58.740 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.733+0000 7f3d492d9700 1 -- 172.21.15.42:0/2661863305 wait complete. 2019-12-05T06:12:58.740 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:12:58.733+0000 7f3d492d9700 1 -- 172.21.15.42:0/2661863305 >> 172.21.15.42:0/2661863305 conn(0x7f3d440a82a0 msgr2=0x7f3d4409f580 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:12:58.749 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:13:00.750 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:13:00.855 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:00.845+0000 7f4092a93700 1 Processor -- start 2019-12-05T06:13:00.855 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:00.845+0000 7f4092a93700 1 -- start start 2019-12-05T06:13:00.855 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:00.845+0000 7f4092a93700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f408c0a7730 0x7f408c08df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:13:00.855 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:00.845+0000 7f4092a93700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f408c07e710 0x7f408c07eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:13:00.856 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:00.845+0000 7f4092a93700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f408c0a0700 con 0x7f408c09f2d0 2019-12-05T06:13:00.856 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:00.845+0000 7f4092a93700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f408c0a0af0 con 0x7f408c0a7730 2019-12-05T06:13:00.856 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:00.845+0000 7f4092a93700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f408c08e5a0 con 0x7f408c07e710 2019-12-05T06:13:00.856 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:00.845+0000 7f4091030700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f408c0a7730 0x7f408c08df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:13:00.856 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:00.845+0000 7f4091831700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f408c09f2d0 0x7f408c09f700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:50770/0 (socket says 172.21.15.42:50770) 2019-12-05T06:13:00.856 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:00.845+0000 7f4091831700 1 -- 172.21.15.42:0/2522281788 learned_addr learned my addr 172.21.15.42:0/2522281788 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:13:00.856 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:00.845+0000 7f409082f700 1 --2- 172.21.15.42:0/2522281788 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f408c07e710 0x7f408c07eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:13:00.857 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:00.845+0000 7f4091030700 1 -- 172.21.15.42:0/2522281788 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f408c07e710 msgr2=0x7f408c07eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:13:00.857 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:00.845+0000 7f4091030700 1 --2- 172.21.15.42:0/2522281788 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f408c07e710 0x7f408c07eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:13:00.857 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:00.845+0000 7f4091030700 1 -- 172.21.15.42:0/2522281788 >> v1:172.21.15.42:6789/0 conn(0x7f408c09f2d0 legacy=0x7f408c09f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:13:00.857 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:00.845+0000 7f407ffff700 1 -- 172.21.15.42:0/2522281788 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4069845613 0 0) 0x7f408c0a0700 con 0x7f408c09f2d0 2019-12-05T06:13:00.857 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:00.845+0000 7f4091030700 1 -- 172.21.15.42:0/2522281788 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f408c07f090 con 0x7f408c0a7730 2019-12-05T06:13:00.857 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:00.845+0000 7f409082f700 1 --2- 172.21.15.42:0/2522281788 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f408c07e710 0x7f408c07eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:13:00.857 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:00.845+0000 7f4091030700 1 --2- 172.21.15.42:0/2522281788 >> v2:172.21.15.42:3300/0 conn(0x7f408c0a7730 0x7f408c08df70 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f4080008790 tx=0x7f408000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:00.858 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:00.845+0000 7f4091030700 1 --2- 172.21.15.42:0/2522281788 >> v2:172.21.15.42:3300/0 conn(0x7f408c0a7730 0x7f408c08df70 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f4080008790 tx=0x7f408000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:00.858 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:00.845+0000 7f4091030700 1 --2- 172.21.15.42:0/2522281788 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f408c0a7730 0x7f408c08df70 secure :-1 s=READY pgs=2267 cs=0 l=1 rx=0x7f4080008790 tx=0x7f408000fc70).ready entity=mon.1 client_cookie=82e49328cd64a24 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:13:00.858 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:00.845+0000 7f4091030700 1 --2- 172.21.15.42:0/2522281788 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f408c0a7730 0x7f408c08df70 secure :-1 s=THROTTLE_DONE pgs=2267 cs=0 l=1 rx=0x7f4080008790 tx=0x7f408000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:00.858 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:00.845+0000 7f407ffff700 1 -- 172.21.15.42:0/2522281788 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f40800112f0 con 0x7f408c0a7730 2019-12-05T06:13:00.858 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:00.845+0000 7f4091030700 1 --2- 172.21.15.42:0/2522281788 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f408c0a7730 0x7f408c08df70 secure :-1 s=THROTTLE_DONE pgs=2267 cs=0 l=1 rx=0x7f4080008790 tx=0x7f408000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:00.858 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:00.845+0000 7f407ffff700 1 -- 172.21.15.42:0/2522281788 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4080008ad0 con 0x7f408c0a7730 2019-12-05T06:13:00.858 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:00.845+0000 7f4091030700 1 --2- 172.21.15.42:0/2522281788 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f408c0a7730 0x7f408c08df70 secure :-1 s=THROTTLE_DONE pgs=2267 cs=0 l=1 rx=0x7f4080008790 tx=0x7f408000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:00.859 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:00.849+0000 7f4092a93700 1 -- 172.21.15.42:0/2522281788 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f408c0a7730 msgr2=0x7f408c08df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:13:00.859 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:00.849+0000 7f407ffff700 1 -- 172.21.15.42:0/2522281788 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f40800118d0 con 0x7f408c0a7730 2019-12-05T06:13:00.859 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:00.849+0000 7f4092a93700 1 --2- 172.21.15.42:0/2522281788 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f408c0a7730 0x7f408c08df70 secure :-1 s=READY pgs=2267 cs=0 l=1 rx=0x7f4080008790 tx=0x7f408000fc70).stop 2019-12-05T06:13:00.859 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:00.849+0000 7f4092a93700 1 -- 172.21.15.42:0/2522281788 shutdown_connections 2019-12-05T06:13:00.859 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:00.849+0000 7f4092a93700 1 --2- 172.21.15.42:0/2522281788 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f408c0a7730 0x7f408c08df70 unknown :-1 s=CLOSED pgs=2267 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:13:00.859 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:00.849+0000 7f4092a93700 1 --2- 172.21.15.42:0/2522281788 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f408c07e710 0x7f408c07eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:13:00.859 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:00.849+0000 7f4092a93700 1 -- 172.21.15.42:0/2522281788 shutdown_connections 2019-12-05T06:13:00.860 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:00.849+0000 7f4092a93700 1 -- 172.21.15.42:0/2522281788 wait complete. 2019-12-05T06:13:00.860 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:00.849+0000 7f4092a93700 1 -- 172.21.15.42:0/2522281788 >> 172.21.15.42:0/2522281788 conn(0x7f408c0aabd0 msgr2=0x7f408c0a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:13:00.860 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:00.849+0000 7f4092a93700 1 Processor -- start 2019-12-05T06:13:00.860 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:00.849+0000 7f4092a93700 1 -- start start 2019-12-05T06:13:00.863 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:00.849+0000 7f4092a93700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f408c07b9a0 0x7f408c112500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:13:00.863 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:00.849+0000 7f4092a93700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f408c144730 con 0x7f408c117de0 2019-12-05T06:13:00.863 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:00.849+0000 7f4092a93700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f408c142d40 con 0x7f408c142910 2019-12-05T06:13:00.863 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:00.849+0000 7f4092a93700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f408c14ed00 con 0x7f408c07b9a0 2019-12-05T06:13:00.863 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:00.849+0000 7f4091831700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f408c142910 0x7f408c12c220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:50774/0 (socket says 172.21.15.42:50774) 2019-12-05T06:13:00.864 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:00.853+0000 7f4091831700 1 -- 172.21.15.42:0/3670146954 learned_addr learned my addr 172.21.15.42:0/3670146954 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:13:00.864 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:00.853+0000 7f409082f700 1 --2- 172.21.15.42:0/3670146954 >> v2:172.21.15.40:3300/0 conn(0x7f408c07b9a0 0x7f408c112500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:13:00.864 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:00.853+0000 7f407dffb700 1 -- 172.21.15.42:0/3670146954 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1986666943 0 0) 0x7f408c142d40 con 0x7f408c142910 2019-12-05T06:13:00.864 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:00.853+0000 7f407dffb700 1 -- 172.21.15.42:0/3670146954 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f406c0030b0 con 0x7f408c142910 2019-12-05T06:13:00.864 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:00.853+0000 7f409082f700 1 -- 172.21.15.42:0/3670146954 >> v1:172.21.15.42:6789/0 conn(0x7f408c142910 legacy=0x7f408c12c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:13:00.864 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:00.853+0000 7f407dffb700 1 -- 172.21.15.42:0/3670146954 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4255429917 0 0) 0x7f408c144730 con 0x7f408c117de0 2019-12-05T06:13:00.865 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:00.853+0000 7f409082f700 1 -- 172.21.15.42:0/3670146954 >> v1:172.21.15.40:6789/0 conn(0x7f408c117de0 legacy=0x7f408c07b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:13:00.865 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:00.853+0000 7f409082f700 1 -- 172.21.15.42:0/3670146954 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f408c143ae0 con 0x7f408c07b9a0 2019-12-05T06:13:00.865 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:00.853+0000 7f409082f700 1 --2- 172.21.15.42:0/3670146954 >> v2:172.21.15.40:3300/0 conn(0x7f408c07b9a0 0x7f408c112500 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f40880032b0 tx=0x7f4088011020).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:00.865 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:00.853+0000 7f409082f700 1 --2- 172.21.15.42:0/3670146954 >> v2:172.21.15.40:3300/0 conn(0x7f408c07b9a0 0x7f408c112500 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f40880032b0 tx=0x7f4088011020).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:00.865 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:00.853+0000 7f409082f700 1 --2- 172.21.15.42:0/3670146954 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f408c07b9a0 0x7f408c112500 secure :-1 s=READY pgs=1719 cs=0 l=1 rx=0x7f40880032b0 tx=0x7f4088011020).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:13:00.865 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:00.853+0000 7f409082f700 1 --2- 172.21.15.42:0/3670146954 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f408c07b9a0 0x7f408c112500 secure :-1 s=THROTTLE_DONE pgs=1719 cs=0 l=1 rx=0x7f40880032b0 tx=0x7f4088011020).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:00.865 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:00.853+0000 7f407dffb700 1 -- 172.21.15.42:0/3670146954 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4088007160 con 0x7f408c07b9a0 2019-12-05T06:13:00.865 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:00.853+0000 7f409082f700 1 --2- 172.21.15.42:0/3670146954 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f408c07b9a0 0x7f408c112500 secure :-1 s=THROTTLE_DONE pgs=1719 cs=0 l=1 rx=0x7f40880032b0 tx=0x7f4088011020).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:00.866 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:00.853+0000 7f409082f700 1 --2- 172.21.15.42:0/3670146954 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f408c07b9a0 0x7f408c112500 secure :-1 s=THROTTLE_DONE pgs=1719 cs=0 l=1 rx=0x7f40880032b0 tx=0x7f4088011020).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:00.866 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:00.853+0000 7f407dffb700 1 -- 172.21.15.42:0/3670146954 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4088005c10 con 0x7f408c07b9a0 2019-12-05T06:13:00.866 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:00.853+0000 7f4092a93700 1 -- 172.21.15.42:0/3670146954 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f408c12da30 con 0x7f408c07b9a0 2019-12-05T06:13:00.866 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:00.853+0000 7f407dffb700 1 -- 172.21.15.42:0/3670146954 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4088007740 con 0x7f408c07b9a0 2019-12-05T06:13:00.866 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:00.853+0000 7f4092a93700 1 -- 172.21.15.42:0/3670146954 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f408c12dfe0 con 0x7f408c07b9a0 2019-12-05T06:13:00.866 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:00.853+0000 7f409082f700 1 --2- 172.21.15.42:0/3670146954 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f408c07b9a0 0x7f408c112500 secure :-1 s=THROTTLE_DONE pgs=1719 cs=0 l=1 rx=0x7f40880032b0 tx=0x7f4088011020).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:00.866 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:00.853+0000 7f407dffb700 1 -- 172.21.15.42:0/3670146954 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f408800b250 con 0x7f408c07b9a0 2019-12-05T06:13:00.867 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:00.853+0000 7f409082f700 1 --2- 172.21.15.42:0/3670146954 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f408c07b9a0 0x7f408c112500 secure :-1 s=THROTTLE_DONE pgs=1719 cs=0 l=1 rx=0x7f40880032b0 tx=0x7f4088011020).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:00.867 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:00.853+0000 7f407dffb700 1 --2- 172.21.15.42:0/3670146954 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f406c03b1e0 0x7f406c03d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:13:00.867 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:00.857+0000 7f407dffb700 1 -- 172.21.15.42:0/3670146954 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 10159+0+0 (secure 0 0 0) 0x7f408804b310 con 0x7f408c07b9a0 2019-12-05T06:13:00.867 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:00.857+0000 7f4091030700 1 --2- 172.21.15.42:0/3670146954 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f406c03b1e0 0x7f406c03d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:13:00.867 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:00.857+0000 7f4091030700 1 --2- 172.21.15.42:0/3670146954 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f406c03b1e0 0x7f406c03d6f0 crc :-1 s=READY pgs=952 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:13:00.868 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:00.861+0000 7f4092a93700 1 -- 172.21.15.42:0/3670146954 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f408c05d100 con 0x7f408c07b9a0 2019-12-05T06:13:00.876 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:00.869+0000 7f409082f700 1 --2- 172.21.15.42:0/3670146954 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f408c07b9a0 0x7f408c112500 secure :-1 s=THROTTLE_DONE pgs=1719 cs=0 l=1 rx=0x7f40880032b0 tx=0x7f4088011020).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:00.876 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:00.869+0000 7f407dffb700 1 -- 172.21.15.42:0/3670146954 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f4088032390 con 0x7f408c07b9a0 2019-12-05T06:13:01.049 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:01.045+0000 7f4092a93700 1 -- 172.21.15.42:0/3670146954 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f408c05b8b0 con 0x7f408c07b9a0 2019-12-05T06:13:01.051 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:01.045+0000 7f409082f700 1 --2- 172.21.15.42:0/3670146954 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f408c07b9a0 0x7f408c112500 secure :-1 s=THROTTLE_DONE pgs=1719 cs=0 l=1 rx=0x7f40880032b0 tx=0x7f4088011020).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:01.051 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:01.045+0000 7f407dffb700 1 -- 172.21.15.42:0/3670146954 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 273 v273) v1 ==== 95+0+2113 (secure 0 0 0) 0x7f40880325f0 con 0x7f408c07b9a0 2019-12-05T06:13:01.052 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 273 2019-12-05T06:13:01.052 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:13:01.053 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":273,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":10246,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/4032647860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":4032647860},{"type":"v1","addr":"172.21.15.42:6835","nonce":4032647860}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":273}],"filesystems":[{"mdsmap":{"epoch":272,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:12:15.021996+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":97,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7094},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7094":{"gid":7094,"name":"b","rank":0,"incarnation":269,"state":"up:active","state_seq":12,"addr":"172.21.15.40:6833/3785523496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3785523496},{"type":"v1","addr":"172.21.15.40:6833","nonce":3785523496}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:13:01.055 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:01.049+0000 7f4092a93700 1 -- 172.21.15.42:0/3670146954 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f406c03b1e0 msgr2=0x7f406c03d6f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:13:01.055 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:01.049+0000 7f4092a93700 1 --2- 172.21.15.42:0/3670146954 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f406c03b1e0 0x7f406c03d6f0 crc :-1 s=READY pgs=952 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:13:01.055 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:01.049+0000 7f4092a93700 1 -- 172.21.15.42:0/3670146954 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f408c07b9a0 msgr2=0x7f408c112500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:13:01.055 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:01.049+0000 7f4092a93700 1 --2- 172.21.15.42:0/3670146954 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f408c07b9a0 0x7f408c112500 secure :-1 s=READY pgs=1719 cs=0 l=1 rx=0x7f40880032b0 tx=0x7f4088011020).stop 2019-12-05T06:13:01.056 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:01.049+0000 7f4092a93700 1 -- 172.21.15.42:0/3670146954 shutdown_connections 2019-12-05T06:13:01.056 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:01.049+0000 7f4092a93700 1 --2- 172.21.15.42:0/3670146954 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f406c03b1e0 0x7f406c03d6f0 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:13:01.056 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:01.049+0000 7f4092a93700 1 --2- 172.21.15.42:0/3670146954 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f408c07b9a0 0x7f408c112500 unknown :-1 s=CLOSED pgs=1719 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:13:01.056 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:01.049+0000 7f4092a93700 1 -- 172.21.15.42:0/3670146954 shutdown_connections 2019-12-05T06:13:01.056 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:01.049+0000 7f4092a93700 1 -- 172.21.15.42:0/3670146954 wait complete. 2019-12-05T06:13:01.056 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:01.049+0000 7f4092a93700 1 -- 172.21.15.42:0/3670146954 >> 172.21.15.42:0/3670146954 conn(0x7f408c0a7f80 msgr2=0x7f408c0596f0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:13:01.066 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:13:03.066 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:13:03.176 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:03.165+0000 7f4ac8a70700 1 Processor -- start 2019-12-05T06:13:03.176 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:03.165+0000 7f4ac8a70700 1 -- start start 2019-12-05T06:13:03.176 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:03.169+0000 7f4ac8a70700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f4ac40a7740 0x7f4ac408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:13:03.177 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:03.169+0000 7f4ac8a70700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f4ac409fae0 0x7f4ac407eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:13:03.177 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:03.169+0000 7f4ac8a70700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ac40a0710 con 0x7f4ac407e720 2019-12-05T06:13:03.177 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:03.169+0000 7f4ac8a70700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ac40a0b00 con 0x7f4ac409fae0 2019-12-05T06:13:03.177 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:03.169+0000 7f4ac8a70700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ac408e5b0 con 0x7f4ac40a7740 2019-12-05T06:13:03.177 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:03.169+0000 7f4ac2d9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f4ac40a7740 0x7f4ac408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:13:03.177 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:03.169+0000 7f4ac2d9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f4ac40a7740 0x7f4ac408df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:50746/0 (socket says 172.21.15.42:50746) 2019-12-05T06:13:03.178 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:03.169+0000 7f4ac2d9d700 1 -- 172.21.15.42:0/3685775957 learned_addr learned my addr 172.21.15.42:0/3685775957 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:13:03.178 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:03.169+0000 7f4ac359e700 1 --2- 172.21.15.42:0/3685775957 >> v2:172.21.15.40:3300/0 conn(0x7f4ac409fae0 0x7f4ac407eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:13:03.178 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:03.169+0000 7f4ac2d9d700 1 -- 172.21.15.42:0/3685775957 >> v2:172.21.15.40:3300/0 conn(0x7f4ac409fae0 msgr2=0x7f4ac407eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:13:03.178 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:03.169+0000 7f4ac2d9d700 1 --2- 172.21.15.42:0/3685775957 >> v2:172.21.15.40:3300/0 conn(0x7f4ac409fae0 0x7f4ac407eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:13:03.178 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:03.169+0000 7f4ac2d9d700 1 -- 172.21.15.42:0/3685775957 >> v1:172.21.15.40:6789/0 conn(0x7f4ac407e720 legacy=0x7f4ac409f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:13:03.179 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:03.169+0000 7f4ac2d9d700 1 -- 172.21.15.42:0/3685775957 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ac407f0a0 con 0x7f4ac40a7740 2019-12-05T06:13:03.179 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:03.169+0000 7f4ac2d9d700 1 --2- 172.21.15.42:0/3685775957 >> v2:172.21.15.42:3300/0 conn(0x7f4ac40a7740 0x7f4ac408df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f4ab4002f30 tx=0x7f4ab400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:03.179 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:03.169+0000 7f4ac2d9d700 1 --2- 172.21.15.42:0/3685775957 >> v2:172.21.15.42:3300/0 conn(0x7f4ac40a7740 0x7f4ac408df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f4ab4002f30 tx=0x7f4ab400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:03.180 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:03.169+0000 7f4ac2d9d700 1 --2- 172.21.15.42:0/3685775957 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4ac40a7740 0x7f4ac408df80 secure :-1 s=READY pgs=2269 cs=0 l=1 rx=0x7f4ab4002f30 tx=0x7f4ab400fd30).ready entity=mon.1 client_cookie=d7eb4171cc81aeec server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:13:03.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:03.169+0000 7f4ac2d9d700 1 --2- 172.21.15.42:0/3685775957 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4ac40a7740 0x7f4ac408df80 secure :-1 s=THROTTLE_DONE pgs=2269 cs=0 l=1 rx=0x7f4ab4002f30 tx=0x7f4ab400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:03.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:03.169+0000 7f4ac2d9d700 1 --2- 172.21.15.42:0/3685775957 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4ac40a7740 0x7f4ac408df80 secure :-1 s=THROTTLE_DONE pgs=2269 cs=0 l=1 rx=0x7f4ab4002f30 tx=0x7f4ab400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:03.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:03.169+0000 7f4ac1d9b700 1 -- 172.21.15.42:0/3685775957 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4ab4008c90 con 0x7f4ac40a7740 2019-12-05T06:13:03.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:03.169+0000 7f4ac2d9d700 1 --2- 172.21.15.42:0/3685775957 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4ac40a7740 0x7f4ac408df80 secure :-1 s=THROTTLE_DONE pgs=2269 cs=0 l=1 rx=0x7f4ab4002f30 tx=0x7f4ab400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:03.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:03.169+0000 7f4ac1d9b700 1 -- 172.21.15.42:0/3685775957 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ab40088b0 con 0x7f4ac40a7740 2019-12-05T06:13:03.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:03.169+0000 7f4ac1d9b700 1 -- 172.21.15.42:0/3685775957 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4ab4011ad0 con 0x7f4ac40a7740 2019-12-05T06:13:03.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:03.169+0000 7f4ac8a70700 1 -- 172.21.15.42:0/3685775957 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4ac40a7740 msgr2=0x7f4ac408df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:13:03.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:03.169+0000 7f4ac8a70700 1 --2- 172.21.15.42:0/3685775957 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4ac40a7740 0x7f4ac408df80 secure :-1 s=READY pgs=2269 cs=0 l=1 rx=0x7f4ab4002f30 tx=0x7f4ab400fd30).stop 2019-12-05T06:13:03.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:03.169+0000 7f4ac8a70700 1 -- 172.21.15.42:0/3685775957 shutdown_connections 2019-12-05T06:13:03.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:03.169+0000 7f4ac8a70700 1 --2- 172.21.15.42:0/3685775957 >> v2:172.21.15.40:3300/0 conn(0x7f4ac409fae0 0x7f4ac407eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:13:03.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:03.169+0000 7f4ac8a70700 1 --2- 172.21.15.42:0/3685775957 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4ac40a7740 0x7f4ac408df80 unknown :-1 s=CLOSED pgs=2269 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:13:03.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:03.169+0000 7f4ac8a70700 1 -- 172.21.15.42:0/3685775957 shutdown_connections 2019-12-05T06:13:03.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:03.169+0000 7f4ac8a70700 1 -- 172.21.15.42:0/3685775957 wait complete. 2019-12-05T06:13:03.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:03.169+0000 7f4ac8a70700 1 -- 172.21.15.42:0/3685775957 >> 172.21.15.42:0/3685775957 conn(0x7f4ac40aabe0 msgr2=0x7f4ac40a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:13:03.186 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:03.173+0000 7f4ac8a70700 1 Processor -- start 2019-12-05T06:13:03.186 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:03.173+0000 7f4ac8a70700 1 -- start start 2019-12-05T06:13:03.186 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:03.173+0000 7f4ac8a70700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f4ac411c150 0x7f4ac41161c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:13:03.187 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:03.173+0000 7f4ac8a70700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4ac411a800 0x7f4ac407d200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:13:03.187 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:03.173+0000 7f4ac8a70700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ac4116710 con 0x7f4ac411ac30 2019-12-05T06:13:03.187 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:03.173+0000 7f4ac8a70700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ac4133c80 con 0x7f4ac411c150 2019-12-05T06:13:03.187 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:03.173+0000 7f4ac8a70700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ac4133f20 con 0x7f4ac411a800 2019-12-05T06:13:03.187 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:03.173+0000 7f4ac2d9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f4ac411c150 0x7f4ac41161c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:13:03.187 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:03.173+0000 7f4ac2d9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f4ac411c150 0x7f4ac41161c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:50750/0 (socket says 172.21.15.42:50750) 2019-12-05T06:13:03.188 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:03.173+0000 7f4ac2d9d700 1 -- 172.21.15.42:0/2014076304 learned_addr learned my addr 172.21.15.42:0/2014076304 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:13:03.188 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:03.173+0000 7f4ac259c700 1 --2- 172.21.15.42:0/2014076304 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4ac411a800 0x7f4ac407d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:13:03.188 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:03.177+0000 7f4abb7fe700 1 -- 172.21.15.42:0/2014076304 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1464499716 0 0) 0x7f4ac4116710 con 0x7f4ac411ac30 2019-12-05T06:13:03.188 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:03.177+0000 7f4ac2d9d700 1 -- 172.21.15.42:0/2014076304 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4ac411a800 msgr2=0x7f4ac407d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:13:03.188 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:03.177+0000 7f4ac2d9d700 1 --2- 172.21.15.42:0/2014076304 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4ac411a800 0x7f4ac407d200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:13:03.188 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:03.177+0000 7f4ac2d9d700 1 -- 172.21.15.42:0/2014076304 >> v1:172.21.15.40:6789/0 conn(0x7f4ac411ac30 legacy=0x7f4ac408eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:13:03.188 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:03.177+0000 7f4ac2d9d700 1 -- 172.21.15.42:0/2014076304 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ac407b740 con 0x7f4ac411c150 2019-12-05T06:13:03.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:03.177+0000 7f4ac2d9d700 1 --2- 172.21.15.42:0/2014076304 >> v2:172.21.15.42:3300/0 conn(0x7f4ac411c150 0x7f4ac41161c0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f4ac411b9d0 tx=0x7f4ab4008bf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:03.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:03.177+0000 7f4ac2d9d700 1 --2- 172.21.15.42:0/2014076304 >> v2:172.21.15.42:3300/0 conn(0x7f4ac411c150 0x7f4ac41161c0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f4ac411b9d0 tx=0x7f4ab4008bf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:03.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:03.177+0000 7f4ac2d9d700 1 --2- 172.21.15.42:0/2014076304 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4ac411c150 0x7f4ac41161c0 secure :-1 s=READY pgs=2270 cs=0 l=1 rx=0x7f4ac411b9d0 tx=0x7f4ab4008bf0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:13:03.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:03.177+0000 7f4ac2d9d700 1 --2- 172.21.15.42:0/2014076304 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4ac411c150 0x7f4ac41161c0 secure :-1 s=THROTTLE_DONE pgs=2270 cs=0 l=1 rx=0x7f4ac411b9d0 tx=0x7f4ab4008bf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:03.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:03.177+0000 7f4abb7fe700 1 -- 172.21.15.42:0/2014076304 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4ab400f470 con 0x7f4ac411c150 2019-12-05T06:13:03.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:03.177+0000 7f4ac2d9d700 1 --2- 172.21.15.42:0/2014076304 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4ac411c150 0x7f4ac41161c0 secure :-1 s=THROTTLE_DONE pgs=2270 cs=0 l=1 rx=0x7f4ac411b9d0 tx=0x7f4ab4008bf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:03.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:03.177+0000 7f4ac2d9d700 1 --2- 172.21.15.42:0/2014076304 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4ac411c150 0x7f4ac41161c0 secure :-1 s=THROTTLE_DONE pgs=2270 cs=0 l=1 rx=0x7f4ac411b9d0 tx=0x7f4ab4008bf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:03.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:03.177+0000 7f4abb7fe700 1 -- 172.21.15.42:0/2014076304 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ab40116f0 con 0x7f4ac411c150 2019-12-05T06:13:03.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:03.177+0000 7f4ac8a70700 1 -- 172.21.15.42:0/2014076304 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ac4119f90 con 0x7f4ac411c150 2019-12-05T06:13:03.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:03.177+0000 7f4abb7fe700 1 -- 172.21.15.42:0/2014076304 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4ab4005730 con 0x7f4ac411c150 2019-12-05T06:13:03.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:03.177+0000 7f4ac8a70700 1 -- 172.21.15.42:0/2014076304 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ac4134460 con 0x7f4ac411c150 2019-12-05T06:13:03.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:03.177+0000 7f4ac2d9d700 1 --2- 172.21.15.42:0/2014076304 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4ac411c150 0x7f4ac41161c0 secure :-1 s=THROTTLE_DONE pgs=2270 cs=0 l=1 rx=0x7f4ac411b9d0 tx=0x7f4ab4008bf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:03.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:03.177+0000 7f4abb7fe700 1 -- 172.21.15.42:0/2014076304 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f4ab4005a40 con 0x7f4ac411c150 2019-12-05T06:13:03.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:03.177+0000 7f4ac2d9d700 1 --2- 172.21.15.42:0/2014076304 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4ac411c150 0x7f4ac41161c0 secure :-1 s=THROTTLE_DONE pgs=2270 cs=0 l=1 rx=0x7f4ac411b9d0 tx=0x7f4ab4008bf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:03.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:03.177+0000 7f4abb7fe700 1 --2- 172.21.15.42:0/2014076304 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4aa803acf0 0x7f4aa803d200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:13:03.193 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:03.177+0000 7f4abb7fe700 1 -- 172.21.15.42:0/2014076304 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 10159+0+0 (secure 0 0 0) 0x7f4ab404b9b0 con 0x7f4ac411c150 2019-12-05T06:13:03.193 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:03.177+0000 7f4ac259c700 1 --2- 172.21.15.42:0/2014076304 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4aa803acf0 0x7f4aa803d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:13:03.193 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:03.181+0000 7f4ac259c700 1 --2- 172.21.15.42:0/2014076304 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4aa803acf0 0x7f4aa803d200 crc :-1 s=READY pgs=953 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:13:03.193 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:03.181+0000 7f4ac8a70700 1 -- 172.21.15.42:0/2014076304 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ac405d110 con 0x7f4ac411c150 2019-12-05T06:13:03.194 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:03.189+0000 7f4ac2d9d700 1 --2- 172.21.15.42:0/2014076304 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4ac411c150 0x7f4ac41161c0 secure :-1 s=THROTTLE_DONE pgs=2270 cs=0 l=1 rx=0x7f4ac411b9d0 tx=0x7f4ab4008bf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:03.194 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:03.189+0000 7f4abb7fe700 1 -- 172.21.15.42:0/2014076304 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f4ab40076e0 con 0x7f4ac411c150 2019-12-05T06:13:03.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:03.353+0000 7f4ac8a70700 1 -- 172.21.15.42:0/2014076304 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4ac405b8c0 con 0x7f4ac411c150 2019-12-05T06:13:03.358 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:03.353+0000 7f4ac2d9d700 1 --2- 172.21.15.42:0/2014076304 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4ac411c150 0x7f4ac41161c0 secure :-1 s=THROTTLE_DONE pgs=2270 cs=0 l=1 rx=0x7f4ac411b9d0 tx=0x7f4ab4008bf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:03.358 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:03.353+0000 7f4abb7fe700 1 -- 172.21.15.42:0/2014076304 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 273 v273) v1 ==== 95+0+2113 (secure 0 0 0) 0x7f4ab4005d50 con 0x7f4ac411c150 2019-12-05T06:13:03.359 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 273 2019-12-05T06:13:03.359 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:13:03.359 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":273,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":10246,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/4032647860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":4032647860},{"type":"v1","addr":"172.21.15.42:6835","nonce":4032647860}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":273}],"filesystems":[{"mdsmap":{"epoch":272,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:12:15.021996+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":97,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7094},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7094":{"gid":7094,"name":"b","rank":0,"incarnation":269,"state":"up:active","state_seq":12,"addr":"172.21.15.40:6833/3785523496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3785523496},{"type":"v1","addr":"172.21.15.40:6833","nonce":3785523496}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:13:03.361 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:03.357+0000 7f4ac8a70700 1 -- 172.21.15.42:0/2014076304 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4aa803acf0 msgr2=0x7f4aa803d200 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:13:03.361 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:03.357+0000 7f4ac8a70700 1 --2- 172.21.15.42:0/2014076304 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4aa803acf0 0x7f4aa803d200 crc :-1 s=READY pgs=953 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:13:03.361 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:03.357+0000 7f4ac8a70700 1 -- 172.21.15.42:0/2014076304 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4ac411c150 msgr2=0x7f4ac41161c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:13:03.361 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:03.357+0000 7f4ac8a70700 1 --2- 172.21.15.42:0/2014076304 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4ac411c150 0x7f4ac41161c0 secure :-1 s=READY pgs=2270 cs=0 l=1 rx=0x7f4ac411b9d0 tx=0x7f4ab4008bf0).stop 2019-12-05T06:13:03.362 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:03.357+0000 7f4ac8a70700 1 -- 172.21.15.42:0/2014076304 shutdown_connections 2019-12-05T06:13:03.362 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:03.357+0000 7f4ac8a70700 1 --2- 172.21.15.42:0/2014076304 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4aa803acf0 0x7f4aa803d200 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:13:03.362 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:03.357+0000 7f4ac8a70700 1 --2- 172.21.15.42:0/2014076304 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4ac411c150 0x7f4ac41161c0 unknown :-1 s=CLOSED pgs=2270 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:13:03.362 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:03.357+0000 7f4ac8a70700 1 --2- 172.21.15.42:0/2014076304 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4ac411a800 0x7f4ac407d200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:13:03.362 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:03.357+0000 7f4ac8a70700 1 -- 172.21.15.42:0/2014076304 shutdown_connections 2019-12-05T06:13:03.362 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:03.357+0000 7f4ac8a70700 1 -- 172.21.15.42:0/2014076304 wait complete. 2019-12-05T06:13:03.363 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:03.357+0000 7f4ac8a70700 1 -- 172.21.15.42:0/2014076304 >> 172.21.15.42:0/2014076304 conn(0x7f4ac408fda0 msgr2=0x7f4ac409f620 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:13:03.369 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:13:05.371 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:13:05.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:05.469+0000 7f9ab43ea700 1 Processor -- start 2019-12-05T06:13:05.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:05.469+0000 7f9ab43ea700 1 -- start start 2019-12-05T06:13:05.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:05.469+0000 7f9ab43ea700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f9aac09fae0 0x7f9aac14fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:13:05.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:05.469+0000 7f9ab43ea700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9aac0a0710 con 0x7f9aac08df80 2019-12-05T06:13:05.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:05.473+0000 7f9ab43ea700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9aac07ef20 con 0x7f9aac0a7740 2019-12-05T06:13:05.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:05.473+0000 7f9ab43ea700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9aac0a0b00 con 0x7f9aac09fae0 2019-12-05T06:13:05.482 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:05.473+0000 7f9ab2987700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f9aac0a7740 0x7f9aac07e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:50800/0 (socket says 172.21.15.42:50800) 2019-12-05T06:13:05.482 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:05.473+0000 7f9ab2987700 1 -- 172.21.15.42:0/1694335796 learned_addr learned my addr 172.21.15.42:0/1694335796 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:13:05.482 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:05.473+0000 7f9ab3188700 1 --2- 172.21.15.42:0/1694335796 >> v2:172.21.15.40:3300/0 conn(0x7f9aac09fae0 0x7f9aac14fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:13:05.482 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:05.473+0000 7f9ab1985700 1 -- 172.21.15.42:0/1694335796 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1018127458 0 0) 0x7f9aac07ef20 con 0x7f9aac0a7740 2019-12-05T06:13:05.482 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:05.473+0000 7f9ab1985700 1 -- 172.21.15.42:0/1694335796 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a980030b0 con 0x7f9aac0a7740 2019-12-05T06:13:05.483 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:05.473+0000 7f9ab3188700 1 -- 172.21.15.42:0/1694335796 >> v1:172.21.15.42:6789/0 conn(0x7f9aac0a7740 legacy=0x7f9aac07e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:13:05.483 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:05.473+0000 7f9ab1985700 1 -- 172.21.15.42:0/1694335796 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2396556336 0 0) 0x7f9a980030b0 con 0x7f9aac0a7740 2019-12-05T06:13:05.483 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:05.473+0000 7f9ab3188700 1 -- 172.21.15.42:0/1694335796 >> v1:172.21.15.40:6789/0 conn(0x7f9aac08df80 legacy=0x7f9aac09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:13:05.483 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:05.473+0000 7f9ab3188700 1 -- 172.21.15.42:0/1694335796 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9aac0a9ba0 con 0x7f9aac09fae0 2019-12-05T06:13:05.483 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:05.473+0000 7f9ab3188700 1 --2- 172.21.15.42:0/1694335796 >> v2:172.21.15.40:3300/0 conn(0x7f9aac09fae0 0x7f9aac14fff0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f9a9c001110 tx=0x7f9aa80014a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:05.483 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:05.473+0000 7f9ab3188700 1 --2- 172.21.15.42:0/1694335796 >> v2:172.21.15.40:3300/0 conn(0x7f9aac09fae0 0x7f9aac14fff0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f9a9c001110 tx=0x7f9aa80014a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:05.483 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:05.473+0000 7f9ab3188700 1 --2- 172.21.15.42:0/1694335796 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9aac09fae0 0x7f9aac14fff0 secure :-1 s=READY pgs=1723 cs=0 l=1 rx=0x7f9a9c001110 tx=0x7f9aa80014a0).ready entity=mon.0 client_cookie=b51950e54db999cb server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:13:05.484 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:05.473+0000 7f9ab3188700 1 --2- 172.21.15.42:0/1694335796 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9aac09fae0 0x7f9aac14fff0 secure :-1 s=THROTTLE_DONE pgs=1723 cs=0 l=1 rx=0x7f9a9c001110 tx=0x7f9aa80014a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:05.484 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:05.473+0000 7f9ab1985700 1 -- 172.21.15.42:0/1694335796 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9aa8001750 con 0x7f9aac09fae0 2019-12-05T06:13:05.484 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:05.473+0000 7f9ab3188700 1 --2- 172.21.15.42:0/1694335796 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9aac09fae0 0x7f9aac14fff0 secure :-1 s=THROTTLE_DONE pgs=1723 cs=0 l=1 rx=0x7f9a9c001110 tx=0x7f9aa80014a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:05.484 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:05.473+0000 7f9ab3188700 1 --2- 172.21.15.42:0/1694335796 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9aac09fae0 0x7f9aac14fff0 secure :-1 s=THROTTLE_DONE pgs=1723 cs=0 l=1 rx=0x7f9a9c001110 tx=0x7f9aa80014a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:05.485 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:05.473+0000 7f9ab1985700 1 -- 172.21.15.42:0/1694335796 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9aa8001560 con 0x7f9aac09fae0 2019-12-05T06:13:05.485 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:05.473+0000 7f9ab1985700 1 -- 172.21.15.42:0/1694335796 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9aa4010a60 con 0x7f9aac09fae0 2019-12-05T06:13:05.485 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:05.473+0000 7f9ab43ea700 1 -- 172.21.15.42:0/1694335796 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9aac09fae0 msgr2=0x7f9aac14fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:13:05.485 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:05.473+0000 7f9ab43ea700 1 --2- 172.21.15.42:0/1694335796 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9aac09fae0 0x7f9aac14fff0 secure :-1 s=READY pgs=1723 cs=0 l=1 rx=0x7f9a9c001110 tx=0x7f9aa80014a0).stop 2019-12-05T06:13:05.486 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:05.473+0000 7f9ab43ea700 1 -- 172.21.15.42:0/1694335796 shutdown_connections 2019-12-05T06:13:05.486 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:05.473+0000 7f9ab43ea700 1 --2- 172.21.15.42:0/1694335796 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9aac09fae0 0x7f9aac14fff0 unknown :-1 s=CLOSED pgs=1723 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:13:05.486 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:05.473+0000 7f9ab43ea700 1 -- 172.21.15.42:0/1694335796 shutdown_connections 2019-12-05T06:13:05.486 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:05.473+0000 7f9ab43ea700 1 -- 172.21.15.42:0/1694335796 wait complete. 2019-12-05T06:13:05.486 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:05.473+0000 7f9ab43ea700 1 -- 172.21.15.42:0/1694335796 >> 172.21.15.42:0/1694335796 conn(0x7f9aac0aabe0 msgr2=0x7f9aac0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:13:05.489 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:05.477+0000 7f9ab43ea700 1 Processor -- start 2019-12-05T06:13:05.490 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:05.477+0000 7f9ab43ea700 1 -- start start 2019-12-05T06:13:05.490 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:05.481+0000 7f9ab43ea700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9aac119aa0 0x7f9aac131780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:13:05.490 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:05.481+0000 7f9ab43ea700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f9aac12e1c0 0x7f9aac12e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:13:05.490 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:05.481+0000 7f9ab43ea700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9aac12eb40 con 0x7f9aac08f7a0 2019-12-05T06:13:05.490 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:05.481+0000 7f9ab43ea700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9aac0a69c0 con 0x7f9aac12e1c0 2019-12-05T06:13:05.491 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:05.481+0000 7f9ab43ea700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9aac0a4c20 con 0x7f9aac119aa0 2019-12-05T06:13:05.491 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:05.481+0000 7f9ab2186700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f9aac08f7a0 0x7f9aac0a61c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:50806/0 (socket says 172.21.15.42:50806) 2019-12-05T06:13:05.491 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:05.481+0000 7f9ab2186700 1 -- 172.21.15.42:0/3289832095 learned_addr learned my addr 172.21.15.42:0/3289832095 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:13:05.491 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:05.481+0000 7f9ab3188700 1 --2- 172.21.15.42:0/3289832095 >> v2:172.21.15.40:3300/0 conn(0x7f9aac12e1c0 0x7f9aac12e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:13:05.491 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:05.481+0000 7f9ab2987700 1 --2- 172.21.15.42:0/3289832095 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9aac119aa0 0x7f9aac131780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:13:05.492 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:05.481+0000 7f9aa37fe700 1 -- 172.21.15.42:0/3289832095 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3596151571 0 0) 0x7f9aac12eb40 con 0x7f9aac08f7a0 2019-12-05T06:13:05.492 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:05.481+0000 7f9aa37fe700 1 -- 172.21.15.42:0/3289832095 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a940030b0 con 0x7f9aac08f7a0 2019-12-05T06:13:05.492 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:05.481+0000 7f9ab3188700 1 -- 172.21.15.42:0/3289832095 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9aac119aa0 msgr2=0x7f9aac131780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:13:05.492 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:05.481+0000 7f9ab3188700 1 --2- 172.21.15.42:0/3289832095 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9aac119aa0 0x7f9aac131780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:13:05.492 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:05.481+0000 7f9ab3188700 1 -- 172.21.15.42:0/3289832095 >> v1:172.21.15.42:6789/0 conn(0x7f9aac08f7a0 legacy=0x7f9aac0a61c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:13:05.492 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:05.481+0000 7f9aa37fe700 1 -- 172.21.15.42:0/3289832095 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 1791929974 0 0) 0x7f9a940030b0 con 0x7f9aac08f7a0 2019-12-05T06:13:05.493 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:05.481+0000 7f9ab3188700 1 -- 172.21.15.42:0/3289832095 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9aac139be0 con 0x7f9aac12e1c0 2019-12-05T06:13:05.493 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:05.481+0000 7f9ab2987700 1 --2- 172.21.15.42:0/3289832095 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9aac119aa0 0x7f9aac131780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:13:05.493 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:05.481+0000 7f9ab3188700 1 --2- 172.21.15.42:0/3289832095 >> v2:172.21.15.40:3300/0 conn(0x7f9aac12e1c0 0x7f9aac12e5f0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f9a9c001db0 tx=0x7f9a9c0018c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:05.493 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:05.481+0000 7f9ab3188700 1 --2- 172.21.15.42:0/3289832095 >> v2:172.21.15.40:3300/0 conn(0x7f9aac12e1c0 0x7f9aac12e5f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f9a9c001db0 tx=0x7f9a9c0018c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:05.493 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:05.481+0000 7f9ab3188700 1 --2- 172.21.15.42:0/3289832095 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9aac12e1c0 0x7f9aac12e5f0 secure :-1 s=READY pgs=1724 cs=0 l=1 rx=0x7f9a9c001db0 tx=0x7f9a9c0018c0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:13:05.493 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:05.481+0000 7f9ab3188700 1 --2- 172.21.15.42:0/3289832095 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9aac12e1c0 0x7f9aac12e5f0 secure :-1 s=THROTTLE_DONE pgs=1724 cs=0 l=1 rx=0x7f9a9c001db0 tx=0x7f9a9c0018c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:05.493 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:05.481+0000 7f9aa37fe700 1 -- 172.21.15.42:0/3289832095 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9a9c001b70 con 0x7f9aac12e1c0 2019-12-05T06:13:05.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:05.481+0000 7f9ab3188700 1 --2- 172.21.15.42:0/3289832095 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9aac12e1c0 0x7f9aac12e5f0 secure :-1 s=THROTTLE_DONE pgs=1724 cs=0 l=1 rx=0x7f9a9c001db0 tx=0x7f9a9c0018c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:05.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:05.481+0000 7f9ab43ea700 1 -- 172.21.15.42:0/3289832095 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9aac138bd0 con 0x7f9aac12e1c0 2019-12-05T06:13:05.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:05.481+0000 7f9aa37fe700 1 -- 172.21.15.42:0/3289832095 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a9c001980 con 0x7f9aac12e1c0 2019-12-05T06:13:05.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:05.481+0000 7f9ab3188700 1 --2- 172.21.15.42:0/3289832095 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9aac12e1c0 0x7f9aac12e5f0 secure :-1 s=THROTTLE_DONE pgs=1724 cs=0 l=1 rx=0x7f9a9c001db0 tx=0x7f9a9c0018c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:05.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:05.481+0000 7f9aa37fe700 1 -- 172.21.15.42:0/3289832095 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9aa4005ab0 con 0x7f9aac12e1c0 2019-12-05T06:13:05.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:05.481+0000 7f9ab43ea700 1 -- 172.21.15.42:0/3289832095 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9aac1390a0 con 0x7f9aac12e1c0 2019-12-05T06:13:05.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:05.481+0000 7f9ab3188700 1 --2- 172.21.15.42:0/3289832095 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9aac12e1c0 0x7f9aac12e5f0 secure :-1 s=THROTTLE_DONE pgs=1724 cs=0 l=1 rx=0x7f9a9c001db0 tx=0x7f9a9c0018c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:05.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:05.485+0000 7f9aa37fe700 1 -- 172.21.15.42:0/3289832095 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f9aa4007160 con 0x7f9aac12e1c0 2019-12-05T06:13:05.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:05.485+0000 7f9ab3188700 1 --2- 172.21.15.42:0/3289832095 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9aac12e1c0 0x7f9aac12e5f0 secure :-1 s=THROTTLE_DONE pgs=1724 cs=0 l=1 rx=0x7f9a9c001db0 tx=0x7f9a9c0018c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:05.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:05.485+0000 7f9aa37fe700 1 --2- 172.21.15.42:0/3289832095 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9a9403b280 0x7f9a9403d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:13:05.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:05.485+0000 7f9aa37fe700 1 -- 172.21.15.42:0/3289832095 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 10159+0+0 (secure 0 0 0) 0x7f9aa404ab90 con 0x7f9aac12e1c0 2019-12-05T06:13:05.496 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:05.485+0000 7f9ab2987700 1 --2- 172.21.15.42:0/3289832095 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9a9403b280 0x7f9a9403d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:13:05.496 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:05.485+0000 7f9ab2987700 1 --2- 172.21.15.42:0/3289832095 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9a9403b280 0x7f9a9403d790 crc :-1 s=READY pgs=954 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:13:05.496 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:05.485+0000 7f9ab43ea700 1 -- 172.21.15.42:0/3289832095 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a88003200 con 0x7f9aac12e1c0 2019-12-05T06:13:05.500 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:05.497+0000 7f9ab3188700 1 --2- 172.21.15.42:0/3289832095 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9aac12e1c0 0x7f9aac12e5f0 secure :-1 s=THROTTLE_DONE pgs=1724 cs=0 l=1 rx=0x7f9a9c001db0 tx=0x7f9a9c0018c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:05.500 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:05.497+0000 7f9aa37fe700 1 -- 172.21.15.42:0/3289832095 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f9aa4031bf0 con 0x7f9aac12e1c0 2019-12-05T06:13:05.670 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:05.665+0000 7f9ab43ea700 1 -- 172.21.15.42:0/3289832095 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9a88003630 con 0x7f9aac12e1c0 2019-12-05T06:13:05.672 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:05.665+0000 7f9ab3188700 1 --2- 172.21.15.42:0/3289832095 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9aac12e1c0 0x7f9aac12e5f0 secure :-1 s=THROTTLE_DONE pgs=1724 cs=0 l=1 rx=0x7f9a9c001db0 tx=0x7f9a9c0018c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:05.672 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:05.665+0000 7f9aa37fe700 1 -- 172.21.15.42:0/3289832095 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 273 v273) v1 ==== 95+0+2113 (secure 0 0 0) 0x7f9aa4031980 con 0x7f9aac12e1c0 2019-12-05T06:13:05.674 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 273 2019-12-05T06:13:05.674 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:13:05.675 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":273,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":10246,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/4032647860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":4032647860},{"type":"v1","addr":"172.21.15.42:6835","nonce":4032647860}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":273}],"filesystems":[{"mdsmap":{"epoch":272,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:12:15.021996+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":97,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7094},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7094":{"gid":7094,"name":"b","rank":0,"incarnation":269,"state":"up:active","state_seq":12,"addr":"172.21.15.40:6833/3785523496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3785523496},{"type":"v1","addr":"172.21.15.40:6833","nonce":3785523496}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:13:05.677 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:05.673+0000 7f9ab43ea700 1 -- 172.21.15.42:0/3289832095 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9a9403b280 msgr2=0x7f9a9403d790 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:13:05.677 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:05.673+0000 7f9ab43ea700 1 --2- 172.21.15.42:0/3289832095 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9a9403b280 0x7f9a9403d790 crc :-1 s=READY pgs=954 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:13:05.677 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:05.673+0000 7f9ab43ea700 1 -- 172.21.15.42:0/3289832095 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9aac12e1c0 msgr2=0x7f9aac12e5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:13:05.677 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:05.673+0000 7f9ab43ea700 1 --2- 172.21.15.42:0/3289832095 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9aac12e1c0 0x7f9aac12e5f0 secure :-1 s=READY pgs=1724 cs=0 l=1 rx=0x7f9a9c001db0 tx=0x7f9a9c0018c0).stop 2019-12-05T06:13:05.678 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:05.673+0000 7f9ab43ea700 1 -- 172.21.15.42:0/3289832095 shutdown_connections 2019-12-05T06:13:05.678 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:05.673+0000 7f9ab43ea700 1 --2- 172.21.15.42:0/3289832095 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9a9403b280 0x7f9a9403d790 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:13:05.678 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:05.673+0000 7f9ab43ea700 1 --2- 172.21.15.42:0/3289832095 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9aac12e1c0 0x7f9aac12e5f0 unknown :-1 s=CLOSED pgs=1724 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:13:05.678 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:05.673+0000 7f9ab43ea700 1 --2- 172.21.15.42:0/3289832095 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9aac119aa0 0x7f9aac131780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:13:05.678 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:05.673+0000 7f9ab43ea700 1 -- 172.21.15.42:0/3289832095 shutdown_connections 2019-12-05T06:13:05.678 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:05.673+0000 7f9ab43ea700 1 -- 172.21.15.42:0/3289832095 wait complete. 2019-12-05T06:13:05.678 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:05.673+0000 7f9ab43ea700 1 -- 172.21.15.42:0/3289832095 >> 172.21.15.42:0/3289832095 conn(0x7f9aac08fde0 msgr2=0x7f9aac059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:13:05.688 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:13:07.690 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:13:07.797 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:07.785+0000 7fdd1e354700 1 Processor -- start 2019-12-05T06:13:07.797 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:07.785+0000 7fdd1e354700 1 -- start start 2019-12-05T06:13:07.797 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:07.789+0000 7fdd1e354700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdd180a7740 0x7fdd1808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:13:07.797 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:07.789+0000 7fdd1e354700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fdd1807e720 0x7fdd1807eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:13:07.797 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:07.789+0000 7fdd1e354700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fdd1809f2e0 0x7fdd1809f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:13:07.798 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:07.789+0000 7fdd1e354700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd180a0b00 con 0x7fdd1807e720 2019-12-05T06:13:07.798 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:07.789+0000 7fdd1e354700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd1808e5b0 con 0x7fdd1809f2e0 2019-12-05T06:13:07.798 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:07.789+0000 7fdd1e354700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd1808e850 con 0x7fdd180a7740 2019-12-05T06:13:07.798 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:07.789+0000 7fdd1c8f1700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdd180a7740 0x7fdd1808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:13:07.798 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:07.789+0000 7fdd1d0f2700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fdd1809f2e0 0x7fdd1809f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:13:07.798 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:07.789+0000 7fdd17fff700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fdd1807e720 0x7fdd1807eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:13:07.799 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:07.789+0000 7fdd1c8f1700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdd180a7740 0x7fdd1808df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:51242/0 (socket says 172.21.15.42:51242) 2019-12-05T06:13:07.799 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:07.789+0000 7fdd1c8f1700 1 -- 172.21.15.42:0/2646536899 learned_addr learned my addr 172.21.15.42:0/2646536899 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:13:07.799 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:07.789+0000 7fdd1c8f1700 1 -- 172.21.15.42:0/2646536899 >> v2:172.21.15.42:3300/0 conn(0x7fdd1809f2e0 msgr2=0x7fdd1809f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:13:07.799 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:07.789+0000 7fdd1c8f1700 1 --2- 172.21.15.42:0/2646536899 >> v2:172.21.15.42:3300/0 conn(0x7fdd1809f2e0 0x7fdd1809f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:13:07.800 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:07.789+0000 7fdd1c8f1700 1 -- 172.21.15.42:0/2646536899 >> v2:172.21.15.40:3300/0 conn(0x7fdd1807e720 msgr2=0x7fdd1807eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:13:07.800 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:07.789+0000 7fdd1c8f1700 1 --2- 172.21.15.42:0/2646536899 >> v2:172.21.15.40:3300/0 conn(0x7fdd1807e720 0x7fdd1807eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:13:07.800 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:07.789+0000 7fdd1c8f1700 1 -- 172.21.15.42:0/2646536899 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd1807f0a0 con 0x7fdd180a7740 2019-12-05T06:13:07.801 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:07.789+0000 7fdd17fff700 1 --2- 172.21.15.42:0/2646536899 >> v2:172.21.15.40:3300/0 conn(0x7fdd1807e720 0x7fdd1807eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:13:07.801 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:07.789+0000 7fdd1c8f1700 1 --2- 172.21.15.42:0/2646536899 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdd180a7740 0x7fdd1808df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fdd080032b0 tx=0x7fdd08010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:07.801 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:07.789+0000 7fdd1c8f1700 1 --2- 172.21.15.42:0/2646536899 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdd180a7740 0x7fdd1808df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fdd080032b0 tx=0x7fdd08010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:07.801 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:07.789+0000 7fdd1c8f1700 1 --2- 172.21.15.42:0/2646536899 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdd180a7740 0x7fdd1808df80 secure :-1 s=READY pgs=499 cs=0 l=1 rx=0x7fdd080032b0 tx=0x7fdd08010130).ready entity=mon.2 client_cookie=f974f696f6025eaa server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:13:07.801 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:07.789+0000 7fdd1c8f1700 1 --2- 172.21.15.42:0/2646536899 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdd180a7740 0x7fdd1808df80 secure :-1 s=THROTTLE_DONE pgs=499 cs=0 l=1 rx=0x7fdd080032b0 tx=0x7fdd08010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:07.801 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:07.789+0000 7fdd1c8f1700 1 --2- 172.21.15.42:0/2646536899 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdd180a7740 0x7fdd1808df80 secure :-1 s=THROTTLE_DONE pgs=499 cs=0 l=1 rx=0x7fdd080032b0 tx=0x7fdd08010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:07.802 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:07.789+0000 7fdd1c8f1700 1 --2- 172.21.15.42:0/2646536899 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdd180a7740 0x7fdd1808df80 secure :-1 s=THROTTLE_DONE pgs=499 cs=0 l=1 rx=0x7fdd080032b0 tx=0x7fdd08010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:07.803 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:07.789+0000 7fdd177fe700 1 -- 172.21.15.42:0/2646536899 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdd08010de0 con 0x7fdd180a7740 2019-12-05T06:13:07.803 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:07.789+0000 7fdd177fe700 1 -- 172.21.15.42:0/2646536899 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd08010a00 con 0x7fdd180a7740 2019-12-05T06:13:07.803 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:07.789+0000 7fdd177fe700 1 -- 172.21.15.42:0/2646536899 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdd080114a0 con 0x7fdd180a7740 2019-12-05T06:13:07.803 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:07.789+0000 7fdd1e354700 1 -- 172.21.15.42:0/2646536899 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdd180a7740 msgr2=0x7fdd1808df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:13:07.804 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:07.789+0000 7fdd1e354700 1 --2- 172.21.15.42:0/2646536899 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdd180a7740 0x7fdd1808df80 secure :-1 s=READY pgs=499 cs=0 l=1 rx=0x7fdd080032b0 tx=0x7fdd08010130).stop 2019-12-05T06:13:07.804 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:07.793+0000 7fdd1e354700 1 -- 172.21.15.42:0/2646536899 shutdown_connections 2019-12-05T06:13:07.804 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:07.793+0000 7fdd1e354700 1 --2- 172.21.15.42:0/2646536899 >> v2:172.21.15.42:3300/0 conn(0x7fdd1809f2e0 0x7fdd1809f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:13:07.804 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:07.793+0000 7fdd1e354700 1 --2- 172.21.15.42:0/2646536899 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdd180a7740 0x7fdd1808df80 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:13:07.804 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:07.793+0000 7fdd1e354700 1 --2- 172.21.15.42:0/2646536899 >> v2:172.21.15.40:3300/0 conn(0x7fdd1807e720 0x7fdd1807eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:13:07.805 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:07.793+0000 7fdd1e354700 1 -- 172.21.15.42:0/2646536899 shutdown_connections 2019-12-05T06:13:07.805 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:07.793+0000 7fdd1e354700 1 -- 172.21.15.42:0/2646536899 wait complete. 2019-12-05T06:13:07.805 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:07.793+0000 7fdd1e354700 1 -- 172.21.15.42:0/2646536899 >> 172.21.15.42:0/2646536899 conn(0x7fdd180aabe0 msgr2=0x7fdd180a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:13:07.810 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:07.793+0000 7fdd1e354700 1 Processor -- start 2019-12-05T06:13:07.811 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:07.793+0000 7fdd1e354700 1 -- start start 2019-12-05T06:13:07.811 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:07.797+0000 7fdd1e354700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fdd1811bed0 0x7fdd181202e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:13:07.811 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:07.797+0000 7fdd1e354700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdd1811e1d0 0x7fdd1811e600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:13:07.811 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:07.797+0000 7fdd1e354700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd18115e80 con 0x7fdd18119f20 2019-12-05T06:13:07.811 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:07.797+0000 7fdd1e354700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd1811eb50 con 0x7fdd1811bed0 2019-12-05T06:13:07.811 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:07.797+0000 7fdd1e354700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd18120900 con 0x7fdd1811e1d0 2019-12-05T06:13:07.812 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:07.797+0000 7fdd17fff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fdd1811bed0 0x7fdd181202e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:13:07.812 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:07.797+0000 7fdd1d0f2700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdd1811e1d0 0x7fdd1811e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:13:07.813 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:07.797+0000 7fdd17fff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fdd1811bed0 0x7fdd181202e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:50784/0 (socket says 172.21.15.42:50784) 2019-12-05T06:13:07.813 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:07.797+0000 7fdd17fff700 1 -- 172.21.15.42:0/2534576082 learned_addr learned my addr 172.21.15.42:0/2534576082 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:13:07.813 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:07.797+0000 7fdd17fff700 1 -- 172.21.15.42:0/2534576082 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdd1811e1d0 msgr2=0x7fdd1811e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:13:07.813 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:07.797+0000 7fdd17fff700 1 --2- 172.21.15.42:0/2534576082 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdd1811e1d0 0x7fdd1811e600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:13:07.813 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:07.797+0000 7fdd17fff700 1 -- 172.21.15.42:0/2534576082 >> v1:172.21.15.40:6789/0 conn(0x7fdd18119f20 legacy=0x7fdd1807daa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:13:07.814 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:07.797+0000 7fdd17fff700 1 -- 172.21.15.42:0/2534576082 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd18137e50 con 0x7fdd1811bed0 2019-12-05T06:13:07.814 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:07.797+0000 7fdd17fff700 1 --2- 172.21.15.42:0/2534576082 >> v2:172.21.15.42:3300/0 conn(0x7fdd1811bed0 0x7fdd181202e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fdd18120ac0 tx=0x7fdd0c010de0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:07.814 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:07.797+0000 7fdd17fff700 1 --2- 172.21.15.42:0/2534576082 >> v2:172.21.15.42:3300/0 conn(0x7fdd1811bed0 0x7fdd181202e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fdd18120ac0 tx=0x7fdd0c010de0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:07.814 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:07.797+0000 7fdd17fff700 1 --2- 172.21.15.42:0/2534576082 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdd1811bed0 0x7fdd181202e0 secure :-1 s=READY pgs=2273 cs=0 l=1 rx=0x7fdd18120ac0 tx=0x7fdd0c010de0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:13:07.814 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:07.797+0000 7fdd17fff700 1 --2- 172.21.15.42:0/2534576082 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdd1811bed0 0x7fdd181202e0 secure :-1 s=THROTTLE_DONE pgs=2273 cs=0 l=1 rx=0x7fdd18120ac0 tx=0x7fdd0c010de0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:07.815 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:07.797+0000 7fdd157fa700 1 -- 172.21.15.42:0/2534576082 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdd0c005890 con 0x7fdd1811bed0 2019-12-05T06:13:07.815 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:07.797+0000 7fdd17fff700 1 --2- 172.21.15.42:0/2534576082 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdd1811bed0 0x7fdd181202e0 secure :-1 s=THROTTLE_DONE pgs=2273 cs=0 l=1 rx=0x7fdd18120ac0 tx=0x7fdd0c010de0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:07.815 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:07.797+0000 7fdd17fff700 1 --2- 172.21.15.42:0/2534576082 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdd1811bed0 0x7fdd181202e0 secure :-1 s=THROTTLE_DONE pgs=2273 cs=0 l=1 rx=0x7fdd18120ac0 tx=0x7fdd0c010de0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:07.815 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:07.797+0000 7fdd157fa700 1 -- 172.21.15.42:0/2534576082 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd0c007160 con 0x7fdd1811bed0 2019-12-05T06:13:07.816 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:07.797+0000 7fdd1e354700 1 -- 172.21.15.42:0/2534576082 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd180a9790 con 0x7fdd1811bed0 2019-12-05T06:13:07.816 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:07.797+0000 7fdd157fa700 1 -- 172.21.15.42:0/2534576082 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdd0c012420 con 0x7fdd1811bed0 2019-12-05T06:13:07.816 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:07.797+0000 7fdd1e354700 1 -- 172.21.15.42:0/2534576082 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd180a9cc0 con 0x7fdd1811bed0 2019-12-05T06:13:07.816 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:07.797+0000 7fdd17fff700 1 --2- 172.21.15.42:0/2534576082 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdd1811bed0 0x7fdd181202e0 secure :-1 s=THROTTLE_DONE pgs=2273 cs=0 l=1 rx=0x7fdd18120ac0 tx=0x7fdd0c010de0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:07.817 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:07.797+0000 7fdd157fa700 1 -- 172.21.15.42:0/2534576082 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fdd0c010180 con 0x7fdd1811bed0 2019-12-05T06:13:07.817 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:07.797+0000 7fdd17fff700 1 --2- 172.21.15.42:0/2534576082 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdd1811bed0 0x7fdd181202e0 secure :-1 s=THROTTLE_DONE pgs=2273 cs=0 l=1 rx=0x7fdd18120ac0 tx=0x7fdd0c010de0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:07.817 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:07.801+0000 7fdd157fa700 1 --2- 172.21.15.42:0/2534576082 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fdcf803ae40 0x7fdcf803d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:13:07.817 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:07.801+0000 7fdd157fa700 1 -- 172.21.15.42:0/2534576082 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 10159+0+0 (secure 0 0 0) 0x7fdd0c04b030 con 0x7fdd1811bed0 2019-12-05T06:13:07.817 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:07.801+0000 7fdd1c8f1700 1 --2- 172.21.15.42:0/2534576082 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fdcf803ae40 0x7fdcf803d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:13:07.817 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:07.801+0000 7fdd1c8f1700 1 --2- 172.21.15.42:0/2534576082 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fdcf803ae40 0x7fdcf803d350 crc :-1 s=READY pgs=955 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:13:07.818 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:07.801+0000 7fdd1e354700 1 -- 172.21.15.42:0/2534576082 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd1805d110 con 0x7fdd1811bed0 2019-12-05T06:13:07.818 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:07.809+0000 7fdd17fff700 1 --2- 172.21.15.42:0/2534576082 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdd1811bed0 0x7fdd181202e0 secure :-1 s=THROTTLE_DONE pgs=2273 cs=0 l=1 rx=0x7fdd18120ac0 tx=0x7fdd0c010de0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:07.818 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:07.809+0000 7fdd157fa700 1 -- 172.21.15.42:0/2534576082 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fdd0c0320b0 con 0x7fdd1811bed0 2019-12-05T06:13:07.976 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:07.969+0000 7fdd1e354700 1 -- 172.21.15.42:0/2534576082 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdd1805b8c0 con 0x7fdd1811bed0 2019-12-05T06:13:07.976 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:07.973+0000 7fdd17fff700 1 --2- 172.21.15.42:0/2534576082 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdd1811bed0 0x7fdd181202e0 secure :-1 s=THROTTLE_DONE pgs=2273 cs=0 l=1 rx=0x7fdd18120ac0 tx=0x7fdd0c010de0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:07.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:07.973+0000 7fdd157fa700 1 -- 172.21.15.42:0/2534576082 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 273 v273) v1 ==== 95+0+2113 (secure 0 0 0) 0x7fdd0c032310 con 0x7fdd1811bed0 2019-12-05T06:13:07.979 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 273 2019-12-05T06:13:07.979 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:13:07.979 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":273,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":10246,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/4032647860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":4032647860},{"type":"v1","addr":"172.21.15.42:6835","nonce":4032647860}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":273}],"filesystems":[{"mdsmap":{"epoch":272,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:12:15.021996+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":97,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7094},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7094":{"gid":7094,"name":"b","rank":0,"incarnation":269,"state":"up:active","state_seq":12,"addr":"172.21.15.40:6833/3785523496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3785523496},{"type":"v1","addr":"172.21.15.40:6833","nonce":3785523496}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:13:07.983 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:07.977+0000 7fdd1e354700 1 -- 172.21.15.42:0/2534576082 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fdcf803ae40 msgr2=0x7fdcf803d350 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:13:07.983 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:07.977+0000 7fdd1e354700 1 --2- 172.21.15.42:0/2534576082 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fdcf803ae40 0x7fdcf803d350 crc :-1 s=READY pgs=955 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:13:07.983 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:07.977+0000 7fdd1e354700 1 -- 172.21.15.42:0/2534576082 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdd1811bed0 msgr2=0x7fdd181202e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:13:07.984 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:07.977+0000 7fdd1e354700 1 --2- 172.21.15.42:0/2534576082 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdd1811bed0 0x7fdd181202e0 secure :-1 s=READY pgs=2273 cs=0 l=1 rx=0x7fdd18120ac0 tx=0x7fdd0c010de0).stop 2019-12-05T06:13:07.984 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:07.977+0000 7fdd1e354700 1 -- 172.21.15.42:0/2534576082 shutdown_connections 2019-12-05T06:13:07.984 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:07.977+0000 7fdd1e354700 1 --2- 172.21.15.42:0/2534576082 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fdcf803ae40 0x7fdcf803d350 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:13:07.984 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:07.977+0000 7fdd1e354700 1 --2- 172.21.15.42:0/2534576082 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdd1811e1d0 0x7fdd1811e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:13:07.984 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:07.977+0000 7fdd1e354700 1 --2- 172.21.15.42:0/2534576082 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdd1811bed0 0x7fdd181202e0 unknown :-1 s=CLOSED pgs=2273 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:13:07.984 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:07.977+0000 7fdd1e354700 1 -- 172.21.15.42:0/2534576082 shutdown_connections 2019-12-05T06:13:07.984 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:07.977+0000 7fdd1e354700 1 -- 172.21.15.42:0/2534576082 wait complete. 2019-12-05T06:13:07.985 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:07.977+0000 7fdd1e354700 1 -- 172.21.15.42:0/2534576082 >> 172.21.15.42:0/2534576082 conn(0x7fdd180a7740 msgr2=0x7fdd180a8070 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:13:07.995 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:13:09.997 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:13:10.108 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:10.097+0000 7f2ff2669700 1 Processor -- start 2019-12-05T06:13:10.108 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:10.097+0000 7f2ff2669700 1 -- start start 2019-12-05T06:13:10.108 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:10.097+0000 7f2ff2669700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2fec0a7740 0x7f2fec08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:13:10.108 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:10.097+0000 7f2ff2669700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2fec09fae0 0x7f2fec07eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:13:10.108 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:10.097+0000 7f2ff2669700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fec0a0710 con 0x7f2fec07e720 2019-12-05T06:13:10.108 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:10.097+0000 7f2ff2669700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fec0a0b00 con 0x7f2fec0a7740 2019-12-05T06:13:10.109 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:10.097+0000 7f2ff2669700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fec08e5b0 con 0x7f2fec09fae0 2019-12-05T06:13:10.109 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:10.097+0000 7f2ff0c06700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2fec0a7740 0x7f2fec08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:13:10.109 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:10.097+0000 7f2ff0c06700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2fec0a7740 0x7f2fec08df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:50790/0 (socket says 172.21.15.42:50790) 2019-12-05T06:13:10.109 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:10.097+0000 7f2ff0c06700 1 -- 172.21.15.42:0/1090564613 learned_addr learned my addr 172.21.15.42:0/1090564613 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:13:10.109 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:10.101+0000 7f2ff1407700 1 --2- 172.21.15.42:0/1090564613 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2fec09fae0 0x7f2fec07eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:13:10.109 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:10.101+0000 7f2ff0c06700 1 -- 172.21.15.42:0/1090564613 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2fec09fae0 msgr2=0x7f2fec07eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:13:10.109 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:10.101+0000 7f2ff0c06700 1 --2- 172.21.15.42:0/1090564613 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2fec09fae0 0x7f2fec07eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:13:10.110 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:10.101+0000 7f2ff0c06700 1 -- 172.21.15.42:0/1090564613 >> v1:172.21.15.40:6789/0 conn(0x7f2fec07e720 legacy=0x7f2fec09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:13:10.110 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:10.101+0000 7f2ff0c06700 1 -- 172.21.15.42:0/1090564613 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fec07f0a0 con 0x7f2fec0a7740 2019-12-05T06:13:10.110 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:10.101+0000 7f2ff0c06700 1 --2- 172.21.15.42:0/1090564613 >> v2:172.21.15.42:3300/0 conn(0x7f2fec0a7740 0x7f2fec08df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f2fdc008790 tx=0x7f2fdc00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:10.111 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:10.101+0000 7f2ff0c06700 1 --2- 172.21.15.42:0/1090564613 >> v2:172.21.15.42:3300/0 conn(0x7f2fec0a7740 0x7f2fec08df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f2fdc008790 tx=0x7f2fdc00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:10.111 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:10.101+0000 7f2ff0c06700 1 --2- 172.21.15.42:0/1090564613 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2fec0a7740 0x7f2fec08df80 secure :-1 s=READY pgs=2274 cs=0 l=1 rx=0x7f2fdc008790 tx=0x7f2fdc00fd30).ready entity=mon.1 client_cookie=5805752d3c8b888a server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:13:10.112 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:10.101+0000 7f2ff0c06700 1 --2- 172.21.15.42:0/1090564613 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2fec0a7740 0x7f2fec08df80 secure :-1 s=THROTTLE_DONE pgs=2274 cs=0 l=1 rx=0x7f2fdc008790 tx=0x7f2fdc00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:10.112 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:10.101+0000 7f2feb7fe700 1 -- 172.21.15.42:0/1090564613 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2fdc008c90 con 0x7f2fec0a7740 2019-12-05T06:13:10.112 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:10.101+0000 7f2ff0c06700 1 --2- 172.21.15.42:0/1090564613 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2fec0a7740 0x7f2fec08df80 secure :-1 s=THROTTLE_DONE pgs=2274 cs=0 l=1 rx=0x7f2fdc008790 tx=0x7f2fdc00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:10.112 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:10.101+0000 7f2ff0c06700 1 --2- 172.21.15.42:0/1090564613 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2fec0a7740 0x7f2fec08df80 secure :-1 s=THROTTLE_DONE pgs=2274 cs=0 l=1 rx=0x7f2fdc008790 tx=0x7f2fdc00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:10.112 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:10.101+0000 7f2feb7fe700 1 -- 172.21.15.42:0/1090564613 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fdc0088b0 con 0x7f2fec0a7740 2019-12-05T06:13:10.113 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:10.101+0000 7f2feb7fe700 1 -- 172.21.15.42:0/1090564613 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2fdc011b10 con 0x7f2fec0a7740 2019-12-05T06:13:10.113 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:10.101+0000 7f2ff2669700 1 -- 172.21.15.42:0/1090564613 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2fec0a7740 msgr2=0x7f2fec08df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:13:10.113 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:10.101+0000 7f2ff2669700 1 --2- 172.21.15.42:0/1090564613 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2fec0a7740 0x7f2fec08df80 secure :-1 s=READY pgs=2274 cs=0 l=1 rx=0x7f2fdc008790 tx=0x7f2fdc00fd30).stop 2019-12-05T06:13:10.113 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:10.101+0000 7f2ff2669700 1 -- 172.21.15.42:0/1090564613 shutdown_connections 2019-12-05T06:13:10.113 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:10.101+0000 7f2ff2669700 1 --2- 172.21.15.42:0/1090564613 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2fec09fae0 0x7f2fec07eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:13:10.113 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:10.101+0000 7f2ff2669700 1 --2- 172.21.15.42:0/1090564613 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2fec0a7740 0x7f2fec08df80 unknown :-1 s=CLOSED pgs=2274 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:13:10.113 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:10.101+0000 7f2ff2669700 1 -- 172.21.15.42:0/1090564613 shutdown_connections 2019-12-05T06:13:10.114 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:10.101+0000 7f2ff2669700 1 -- 172.21.15.42:0/1090564613 wait complete. 2019-12-05T06:13:10.114 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:10.101+0000 7f2ff2669700 1 -- 172.21.15.42:0/1090564613 >> 172.21.15.42:0/1090564613 conn(0x7f2fec0aabe0 msgr2=0x7f2fec0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:13:10.117 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:10.105+0000 7f2ff2669700 1 Processor -- start 2019-12-05T06:13:10.117 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:10.105+0000 7f2ff2669700 1 -- start start 2019-12-05T06:13:10.117 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:10.105+0000 7f2ff2669700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2fec112a10 0x7f2fec112e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:13:10.117 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:10.105+0000 7f2ff2669700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fec126df0 con 0x7f2fec11a520 2019-12-05T06:13:10.118 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:10.105+0000 7f2ff2669700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fec130260 con 0x7f2fec1135d0 2019-12-05T06:13:10.118 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:10.105+0000 7f2ff2669700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fec115e80 con 0x7f2fec112a10 2019-12-05T06:13:10.118 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:10.105+0000 7f2ff0c06700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f2fec1135d0 0x7f2fec1155e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:50834/0 (socket says 172.21.15.42:50834) 2019-12-05T06:13:10.118 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:10.105+0000 7f2ff0c06700 1 -- 172.21.15.42:0/2104359649 learned_addr learned my addr 172.21.15.42:0/2104359649 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:13:10.118 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:10.105+0000 7f2ff1407700 1 --2- 172.21.15.42:0/2104359649 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2fec112a10 0x7f2fec112e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:13:10.118 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:10.105+0000 7f2fe97fa700 1 -- 172.21.15.42:0/2104359649 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4220424339 0 0) 0x7f2fec130260 con 0x7f2fec1135d0 2019-12-05T06:13:10.119 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:10.105+0000 7f2fe97fa700 1 -- 172.21.15.42:0/2104359649 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2fcc0030b0 con 0x7f2fec1135d0 2019-12-05T06:13:10.119 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:10.105+0000 7f2fe97fa700 1 -- 172.21.15.42:0/2104359649 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1935077390 0 0) 0x7f2fec126df0 con 0x7f2fec11a520 2019-12-05T06:13:10.119 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:10.105+0000 7f2fe97fa700 1 -- 172.21.15.42:0/2104359649 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2fec130260 con 0x7f2fec11a520 2019-12-05T06:13:10.119 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:10.105+0000 7f2ff1407700 1 -- 172.21.15.42:0/2104359649 >> v1:172.21.15.42:6789/0 conn(0x7f2fec1135d0 legacy=0x7f2fec1155e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:13:10.119 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:10.105+0000 7f2ff1407700 1 -- 172.21.15.42:0/2104359649 >> v1:172.21.15.40:6789/0 conn(0x7f2fec11a520 legacy=0x7f2fec127550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:13:10.119 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:10.105+0000 7f2ff1407700 1 -- 172.21.15.42:0/2104359649 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fec1314a0 con 0x7f2fec112a10 2019-12-05T06:13:10.120 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:10.105+0000 7f2ff1407700 1 --2- 172.21.15.42:0/2104359649 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2fec112a10 0x7f2fec112e40 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f2fe40032b0 tx=0x7f2fe4010dc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:10.120 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:10.109+0000 7f2ff1407700 1 --2- 172.21.15.42:0/2104359649 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2fec112a10 0x7f2fec112e40 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f2fe40032b0 tx=0x7f2fe4010dc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:10.120 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:10.109+0000 7f2ff1407700 1 --2- 172.21.15.42:0/2104359649 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2fec112a10 0x7f2fec112e40 secure :-1 s=READY pgs=500 cs=0 l=1 rx=0x7f2fe40032b0 tx=0x7f2fe4010dc0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:13:10.120 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:10.109+0000 7f2ff1407700 1 --2- 172.21.15.42:0/2104359649 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2fec112a10 0x7f2fec112e40 secure :-1 s=THROTTLE_DONE pgs=500 cs=0 l=1 rx=0x7f2fe40032b0 tx=0x7f2fe4010dc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:10.120 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:10.109+0000 7f2fe97fa700 1 -- 172.21.15.42:0/2104359649 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2fe4005ca0 con 0x7f2fec112a10 2019-12-05T06:13:10.120 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:10.109+0000 7f2ff1407700 1 --2- 172.21.15.42:0/2104359649 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2fec112a10 0x7f2fec112e40 secure :-1 s=THROTTLE_DONE pgs=500 cs=0 l=1 rx=0x7f2fe40032b0 tx=0x7f2fe4010dc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:10.121 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:10.109+0000 7f2ff1407700 1 --2- 172.21.15.42:0/2104359649 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2fec112a10 0x7f2fec112e40 secure :-1 s=THROTTLE_DONE pgs=500 cs=0 l=1 rx=0x7f2fe40032b0 tx=0x7f2fe4010dc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:10.122 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:10.109+0000 7f2fe97fa700 1 -- 172.21.15.42:0/2104359649 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fe40058c0 con 0x7f2fec112a10 2019-12-05T06:13:10.122 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:10.109+0000 7f2ff2669700 1 -- 172.21.15.42:0/2104359649 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fec12f250 con 0x7f2fec112a10 2019-12-05T06:13:10.122 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:10.109+0000 7f2fe97fa700 1 -- 172.21.15.42:0/2104359649 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2fe4011be0 con 0x7f2fec112a10 2019-12-05T06:13:10.122 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:10.109+0000 7f2ff2669700 1 -- 172.21.15.42:0/2104359649 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fec12f7d0 con 0x7f2fec112a10 2019-12-05T06:13:10.122 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:10.109+0000 7f2ff1407700 1 --2- 172.21.15.42:0/2104359649 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2fec112a10 0x7f2fec112e40 secure :-1 s=THROTTLE_DONE pgs=500 cs=0 l=1 rx=0x7f2fe40032b0 tx=0x7f2fe4010dc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:10.123 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:10.109+0000 7f2ff1407700 1 --2- 172.21.15.42:0/2104359649 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2fec112a10 0x7f2fec112e40 secure :-1 s=THROTTLE_DONE pgs=500 cs=0 l=1 rx=0x7f2fe40032b0 tx=0x7f2fe4010dc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:10.123 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:10.109+0000 7f2fe97fa700 1 -- 172.21.15.42:0/2104359649 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f2fe4007180 con 0x7f2fec112a10 2019-12-05T06:13:10.123 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:10.109+0000 7f2fe97fa700 1 --2- 172.21.15.42:0/2104359649 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2fcc03b4d0 0x7f2fcc03d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:13:10.123 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:10.109+0000 7f2fe97fa700 1 -- 172.21.15.42:0/2104359649 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 10159+0+0 (secure 0 0 0) 0x7f2fe404aca0 con 0x7f2fec112a10 2019-12-05T06:13:10.123 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:10.109+0000 7f2ff0c06700 1 --2- 172.21.15.42:0/2104359649 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2fcc03b4d0 0x7f2fcc03d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:13:10.123 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:10.109+0000 7f2ff0c06700 1 --2- 172.21.15.42:0/2104359649 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2fcc03b4d0 0x7f2fcc03d9e0 crc :-1 s=READY pgs=956 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:13:10.123 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:10.113+0000 7f2ff2669700 1 -- 172.21.15.42:0/2104359649 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fec05d110 con 0x7f2fec112a10 2019-12-05T06:13:10.126 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:10.121+0000 7f2ff1407700 1 --2- 172.21.15.42:0/2104359649 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2fec112a10 0x7f2fec112e40 secure :-1 s=THROTTLE_DONE pgs=500 cs=0 l=1 rx=0x7f2fe40032b0 tx=0x7f2fe4010dc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:10.126 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:10.121+0000 7f2fe97fa700 1 -- 172.21.15.42:0/2104359649 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f2fe4031ce0 con 0x7f2fec112a10 2019-12-05T06:13:10.309 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:10.293+0000 7f2ff2669700 1 -- 172.21.15.42:0/2104359649 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2fec05b8c0 con 0x7f2fec112a10 2019-12-05T06:13:10.310 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:10.293+0000 7f2ff1407700 1 --2- 172.21.15.42:0/2104359649 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2fec112a10 0x7f2fec112e40 secure :-1 s=THROTTLE_DONE pgs=500 cs=0 l=1 rx=0x7f2fe40032b0 tx=0x7f2fe4010dc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:10.310 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:10.293+0000 7f2fe97fa700 1 -- 172.21.15.42:0/2104359649 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 273 v273) v1 ==== 95+0+2113 (secure 0 0 0) 0x7f2fe4031f40 con 0x7f2fec112a10 2019-12-05T06:13:10.310 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 273 2019-12-05T06:13:10.310 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:10.297+0000 7f2ff2669700 1 -- 172.21.15.42:0/2104359649 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2fcc03b4d0 msgr2=0x7f2fcc03d9e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:13:10.311 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:10.297+0000 7f2ff2669700 1 --2- 172.21.15.42:0/2104359649 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2fcc03b4d0 0x7f2fcc03d9e0 crc :-1 s=READY pgs=956 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:13:10.311 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:10.297+0000 7f2ff2669700 1 -- 172.21.15.42:0/2104359649 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2fec112a10 msgr2=0x7f2fec112e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:13:10.311 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:10.297+0000 7f2ff2669700 1 --2- 172.21.15.42:0/2104359649 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2fec112a10 0x7f2fec112e40 secure :-1 s=READY pgs=500 cs=0 l=1 rx=0x7f2fe40032b0 tx=0x7f2fe4010dc0).stop 2019-12-05T06:13:10.311 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:10.297+0000 7f2ff2669700 1 -- 172.21.15.42:0/2104359649 shutdown_connections 2019-12-05T06:13:10.312 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:10.297+0000 7f2ff2669700 1 --2- 172.21.15.42:0/2104359649 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2fcc03b4d0 0x7f2fcc03d9e0 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:13:10.312 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:10.297+0000 7f2ff2669700 1 --2- 172.21.15.42:0/2104359649 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2fec112a10 0x7f2fec112e40 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:13:10.312 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:10.297+0000 7f2ff2669700 1 -- 172.21.15.42:0/2104359649 shutdown_connections 2019-12-05T06:13:10.312 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:10.297+0000 7f2ff2669700 1 -- 172.21.15.42:0/2104359649 wait complete. 2019-12-05T06:13:10.313 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:10.297+0000 7f2ff2669700 1 -- 172.21.15.42:0/2104359649 >> 172.21.15.42:0/2104359649 conn(0x7f2fec08fda0 msgr2=0x7f2fec09f4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:13:10.313 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:13:10.313 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":273,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":10246,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/4032647860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":4032647860},{"type":"v1","addr":"172.21.15.42:6835","nonce":4032647860}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":273}],"filesystems":[{"mdsmap":{"epoch":272,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:12:15.021996+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":97,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7094},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7094":{"gid":7094,"name":"b","rank":0,"incarnation":269,"state":"up:active","state_seq":12,"addr":"172.21.15.40:6833/3785523496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3785523496},{"type":"v1","addr":"172.21.15.40:6833","nonce":3785523496}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:13:10.316 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:13:12.317 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:13:12.428 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:12.413+0000 7fe66fc4b700 1 Processor -- start 2019-12-05T06:13:12.429 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:12.413+0000 7fe66fc4b700 1 -- start start 2019-12-05T06:13:12.429 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:12.417+0000 7fe66fc4b700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe6680a7730 0x7fe66808df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:13:12.429 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:12.417+0000 7fe66fc4b700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe66809fad0 0x7fe66807eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:13:12.429 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:12.417+0000 7fe66fc4b700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6680a0700 con 0x7fe66807e710 2019-12-05T06:13:12.429 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:12.417+0000 7fe66fc4b700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6680a0af0 con 0x7fe6680a7730 2019-12-05T06:13:12.430 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:12.417+0000 7fe66fc4b700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe66808e5a0 con 0x7fe66809fad0 2019-12-05T06:13:12.430 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:12.417+0000 7fe66d9e7700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fe66807e710 0x7fe66809f2d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:50848/0 (socket says 172.21.15.42:50848) 2019-12-05T06:13:12.430 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:12.417+0000 7fe66d9e7700 1 -- 172.21.15.42:0/283613687 learned_addr learned my addr 172.21.15.42:0/283613687 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:13:12.430 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:12.417+0000 7fe66e1e8700 1 --2- 172.21.15.42:0/283613687 >> v2:172.21.15.40:3300/0 conn(0x7fe6680a7730 0x7fe66808df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:13:12.430 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:12.417+0000 7fe66e9e9700 1 --2- 172.21.15.42:0/283613687 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe66809fad0 0x7fe66807eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:13:12.430 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:12.417+0000 7fe66d1e6700 1 -- 172.21.15.42:0/283613687 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1750276746 0 0) 0x7fe6680a0700 con 0x7fe66807e710 2019-12-05T06:13:12.431 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:12.417+0000 7fe66e1e8700 1 -- 172.21.15.42:0/283613687 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe66809fad0 msgr2=0x7fe66807eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:13:12.431 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:12.417+0000 7fe66e1e8700 1 --2- 172.21.15.42:0/283613687 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe66809fad0 0x7fe66807eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:13:12.431 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:12.417+0000 7fe66e1e8700 1 -- 172.21.15.42:0/283613687 >> v1:172.21.15.42:6789/0 conn(0x7fe66807e710 legacy=0x7fe66809f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:13:12.431 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:12.417+0000 7fe66e1e8700 1 -- 172.21.15.42:0/283613687 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe66807f090 con 0x7fe6680a7730 2019-12-05T06:13:12.431 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:12.417+0000 7fe66e1e8700 1 --2- 172.21.15.42:0/283613687 >> v2:172.21.15.40:3300/0 conn(0x7fe6680a7730 0x7fe66808df70 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fe664008790 tx=0x7fe66400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:12.431 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:12.417+0000 7fe66e1e8700 1 --2- 172.21.15.42:0/283613687 >> v2:172.21.15.40:3300/0 conn(0x7fe6680a7730 0x7fe66808df70 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fe664008790 tx=0x7fe66400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:12.431 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:12.417+0000 7fe66e1e8700 1 --2- 172.21.15.42:0/283613687 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe6680a7730 0x7fe66808df70 secure :-1 s=READY pgs=1728 cs=0 l=1 rx=0x7fe664008790 tx=0x7fe66400fc70).ready entity=mon.0 client_cookie=deb4334d965b337a server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:13:12.432 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:12.417+0000 7fe66e1e8700 1 --2- 172.21.15.42:0/283613687 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe6680a7730 0x7fe66808df70 secure :-1 s=THROTTLE_DONE pgs=1728 cs=0 l=1 rx=0x7fe664008790 tx=0x7fe66400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:12.432 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:12.417+0000 7fe66e1e8700 1 --2- 172.21.15.42:0/283613687 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe6680a7730 0x7fe66808df70 secure :-1 s=THROTTLE_DONE pgs=1728 cs=0 l=1 rx=0x7fe664008790 tx=0x7fe66400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:12.432 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:12.417+0000 7fe66d1e6700 1 -- 172.21.15.42:0/283613687 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe664008bb0 con 0x7fe6680a7730 2019-12-05T06:13:12.432 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:12.417+0000 7fe66e1e8700 1 --2- 172.21.15.42:0/283613687 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe6680a7730 0x7fe66808df70 secure :-1 s=THROTTLE_DONE pgs=1728 cs=0 l=1 rx=0x7fe664008790 tx=0x7fe66400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:12.432 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:12.417+0000 7fe66d1e6700 1 -- 172.21.15.42:0/283613687 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6640087d0 con 0x7fe6680a7730 2019-12-05T06:13:12.433 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:12.417+0000 7fe66d1e6700 1 -- 172.21.15.42:0/283613687 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe664011860 con 0x7fe6680a7730 2019-12-05T06:13:12.433 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:12.417+0000 7fe66fc4b700 1 -- 172.21.15.42:0/283613687 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe6680a7730 msgr2=0x7fe66808df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:13:12.433 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:12.417+0000 7fe66fc4b700 1 --2- 172.21.15.42:0/283613687 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe6680a7730 0x7fe66808df70 secure :-1 s=READY pgs=1728 cs=0 l=1 rx=0x7fe664008790 tx=0x7fe66400fc70).stop 2019-12-05T06:13:12.433 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:12.417+0000 7fe66fc4b700 1 -- 172.21.15.42:0/283613687 shutdown_connections 2019-12-05T06:13:12.433 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:12.417+0000 7fe66fc4b700 1 --2- 172.21.15.42:0/283613687 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe66809fad0 0x7fe66807eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:13:12.434 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:12.417+0000 7fe66fc4b700 1 --2- 172.21.15.42:0/283613687 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe6680a7730 0x7fe66808df70 unknown :-1 s=CLOSED pgs=1728 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:13:12.434 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:12.417+0000 7fe66fc4b700 1 -- 172.21.15.42:0/283613687 shutdown_connections 2019-12-05T06:13:12.434 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:12.417+0000 7fe66fc4b700 1 -- 172.21.15.42:0/283613687 wait complete. 2019-12-05T06:13:12.434 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:12.417+0000 7fe66fc4b700 1 -- 172.21.15.42:0/283613687 >> 172.21.15.42:0/283613687 conn(0x7fe6680aabd0 msgr2=0x7fe6680a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:13:12.434 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:12.421+0000 7fe66fc4b700 1 Processor -- start 2019-12-05T06:13:12.435 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:12.421+0000 7fe66fc4b700 1 -- start start 2019-12-05T06:13:12.435 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:12.425+0000 7fe66fc4b700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe6681211c0 0x7fe668117050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:13:12.435 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:12.425+0000 7fe66fc4b700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe66807df90 con 0x7fe6681232e0 2019-12-05T06:13:12.435 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:12.425+0000 7fe66fc4b700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe668130540 con 0x7fe66811df80 2019-12-05T06:13:12.435 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:12.425+0000 7fe66fc4b700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe668123710 con 0x7fe6681211c0 2019-12-05T06:13:12.435 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:12.425+0000 7fe66e1e8700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe6681211c0 0x7fe668117050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:13:12.435 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:12.425+0000 7fe66e1e8700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe6681211c0 0x7fe668117050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:50816/0 (socket says 172.21.15.42:50816) 2019-12-05T06:13:12.436 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:12.425+0000 7fe66e1e8700 1 -- 172.21.15.42:0/3364096919 learned_addr learned my addr 172.21.15.42:0/3364096919 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:13:12.447 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:12.425+0000 7fe66e1e8700 1 -- 172.21.15.42:0/3364096919 >> v1:172.21.15.42:6789/0 conn(0x7fe66811df80 legacy=0x7fe668137990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:13:12.447 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:12.425+0000 7fe66e1e8700 1 -- 172.21.15.42:0/3364096919 >> v1:172.21.15.40:6789/0 conn(0x7fe6681232e0 legacy=0x7fe66807d790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:13:12.447 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:12.425+0000 7fe66e1e8700 1 -- 172.21.15.42:0/3364096919 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe66811e3b0 con 0x7fe6681211c0 2019-12-05T06:13:12.447 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:12.425+0000 7fe66e1e8700 1 --2- 172.21.15.42:0/3364096919 >> v2:172.21.15.42:3300/0 conn(0x7fe6681211c0 0x7fe668117050 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fe664008790 tx=0x7fe664011960).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:12.447 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:12.425+0000 7fe66e1e8700 1 --2- 172.21.15.42:0/3364096919 >> v2:172.21.15.42:3300/0 conn(0x7fe6681211c0 0x7fe668117050 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fe664008790 tx=0x7fe664011960).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:12.447 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:12.425+0000 7fe66e1e8700 1 --2- 172.21.15.42:0/3364096919 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe6681211c0 0x7fe668117050 secure :-1 s=READY pgs=2278 cs=0 l=1 rx=0x7fe664008790 tx=0x7fe664011960).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:13:12.448 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:12.425+0000 7fe66e1e8700 1 --2- 172.21.15.42:0/3364096919 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe6681211c0 0x7fe668117050 secure :-1 s=THROTTLE_DONE pgs=2278 cs=0 l=1 rx=0x7fe664008790 tx=0x7fe664011960).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:12.448 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:12.425+0000 7fe66e1e8700 1 --2- 172.21.15.42:0/3364096919 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe6681211c0 0x7fe668117050 secure :-1 s=THROTTLE_DONE pgs=2278 cs=0 l=1 rx=0x7fe664008790 tx=0x7fe664011960).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:12.448 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:12.425+0000 7fe65affd700 1 -- 172.21.15.42:0/3364096919 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe66400f280 con 0x7fe6681211c0 2019-12-05T06:13:12.448 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:12.425+0000 7fe66e1e8700 1 --2- 172.21.15.42:0/3364096919 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe6681211c0 0x7fe668117050 secure :-1 s=THROTTLE_DONE pgs=2278 cs=0 l=1 rx=0x7fe664008790 tx=0x7fe664011960).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:12.450 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:12.425+0000 7fe65affd700 1 -- 172.21.15.42:0/3364096919 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe664008db0 con 0x7fe6681211c0 2019-12-05T06:13:12.450 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:12.425+0000 7fe66fc4b700 1 -- 172.21.15.42:0/3364096919 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe66811e5c0 con 0x7fe6681211c0 2019-12-05T06:13:12.450 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:12.425+0000 7fe65affd700 1 -- 172.21.15.42:0/3364096919 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe664005540 con 0x7fe6681211c0 2019-12-05T06:13:12.450 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:12.425+0000 7fe66fc4b700 1 -- 172.21.15.42:0/3364096919 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6681391a0 con 0x7fe6681211c0 2019-12-05T06:13:12.450 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:12.425+0000 7fe66e1e8700 1 --2- 172.21.15.42:0/3364096919 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe6681211c0 0x7fe668117050 secure :-1 s=THROTTLE_DONE pgs=2278 cs=0 l=1 rx=0x7fe664008790 tx=0x7fe664011960).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:12.450 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:12.425+0000 7fe65affd700 1 -- 172.21.15.42:0/3364096919 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fe664005850 con 0x7fe6681211c0 2019-12-05T06:13:12.451 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:12.429+0000 7fe66e1e8700 1 --2- 172.21.15.42:0/3364096919 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe6681211c0 0x7fe668117050 secure :-1 s=THROTTLE_DONE pgs=2278 cs=0 l=1 rx=0x7fe664008790 tx=0x7fe664011960).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:12.451 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:12.429+0000 7fe65affd700 1 --2- 172.21.15.42:0/3364096919 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe65003ad30 0x7fe65003d240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:13:12.451 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:12.429+0000 7fe65affd700 1 -- 172.21.15.42:0/3364096919 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 10159+0+0 (secure 0 0 0) 0x7fe66404b850 con 0x7fe6681211c0 2019-12-05T06:13:12.451 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:12.429+0000 7fe66d9e7700 1 --2- 172.21.15.42:0/3364096919 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe65003ad30 0x7fe65003d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:13:12.451 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:12.429+0000 7fe66d9e7700 1 --2- 172.21.15.42:0/3364096919 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe65003ad30 0x7fe65003d240 crc :-1 s=READY pgs=957 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:13:12.452 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:12.429+0000 7fe66fc4b700 1 -- 172.21.15.42:0/3364096919 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe66805d100 con 0x7fe6681211c0 2019-12-05T06:13:12.452 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:12.433+0000 7fe66e1e8700 1 --2- 172.21.15.42:0/3364096919 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe6681211c0 0x7fe668117050 secure :-1 s=THROTTLE_DONE pgs=2278 cs=0 l=1 rx=0x7fe664008790 tx=0x7fe664011960).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:12.452 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:12.433+0000 7fe65affd700 1 -- 172.21.15.42:0/3364096919 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fe6640328d0 con 0x7fe6681211c0 2019-12-05T06:13:12.606 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:12.601+0000 7fe66fc4b700 1 -- 172.21.15.42:0/3364096919 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe66811e830 con 0x7fe6681211c0 2019-12-05T06:13:12.607 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:12.601+0000 7fe66e1e8700 1 --2- 172.21.15.42:0/3364096919 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe6681211c0 0x7fe668117050 secure :-1 s=THROTTLE_DONE pgs=2278 cs=0 l=1 rx=0x7fe664008790 tx=0x7fe664011960).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:12.607 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:12.601+0000 7fe65affd700 1 -- 172.21.15.42:0/3364096919 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 273 v273) v1 ==== 95+0+2113 (secure 0 0 0) 0x7fe664032b30 con 0x7fe6681211c0 2019-12-05T06:13:12.607 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 273 2019-12-05T06:13:12.608 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:13:12.608 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":273,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":10246,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/4032647860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":4032647860},{"type":"v1","addr":"172.21.15.42:6835","nonce":4032647860}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":273}],"filesystems":[{"mdsmap":{"epoch":272,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:12:15.021996+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":97,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7094},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7094":{"gid":7094,"name":"b","rank":0,"incarnation":269,"state":"up:active","state_seq":12,"addr":"172.21.15.40:6833/3785523496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3785523496},{"type":"v1","addr":"172.21.15.40:6833","nonce":3785523496}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:13:12.610 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:12.605+0000 7fe66fc4b700 1 -- 172.21.15.42:0/3364096919 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe65003ad30 msgr2=0x7fe65003d240 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:13:12.610 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:12.605+0000 7fe66fc4b700 1 --2- 172.21.15.42:0/3364096919 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe65003ad30 0x7fe65003d240 crc :-1 s=READY pgs=957 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:13:12.610 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:12.605+0000 7fe66fc4b700 1 -- 172.21.15.42:0/3364096919 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe6681211c0 msgr2=0x7fe668117050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:13:12.611 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:12.605+0000 7fe66fc4b700 1 --2- 172.21.15.42:0/3364096919 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe6681211c0 0x7fe668117050 secure :-1 s=READY pgs=2278 cs=0 l=1 rx=0x7fe664008790 tx=0x7fe664011960).stop 2019-12-05T06:13:12.611 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:12.605+0000 7fe66fc4b700 1 -- 172.21.15.42:0/3364096919 shutdown_connections 2019-12-05T06:13:12.611 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:12.605+0000 7fe66fc4b700 1 --2- 172.21.15.42:0/3364096919 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe65003ad30 0x7fe65003d240 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:13:12.612 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:12.605+0000 7fe66fc4b700 1 --2- 172.21.15.42:0/3364096919 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe6681211c0 0x7fe668117050 unknown :-1 s=CLOSED pgs=2278 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:13:12.612 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:12.605+0000 7fe66fc4b700 1 -- 172.21.15.42:0/3364096919 shutdown_connections 2019-12-05T06:13:12.612 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:12.605+0000 7fe66fc4b700 1 -- 172.21.15.42:0/3364096919 wait complete. 2019-12-05T06:13:12.612 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:12.605+0000 7fe66fc4b700 1 -- 172.21.15.42:0/3364096919 >> 172.21.15.42:0/3364096919 conn(0x7fe6680a7730 msgr2=0x7fe6680596f0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:13:12.619 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:13:14.620 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:13:14.713 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.697+0000 7f25edc6a700 1 Processor -- start 2019-12-05T06:13:14.713 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.697+0000 7f25edc6a700 1 -- start start 2019-12-05T06:13:14.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.697+0000 7f25edc6a700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f25e80a7740 0x7f25e808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:13:14.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.697+0000 7f25edc6a700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f25e807e720 0x7f25e807eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:13:14.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.697+0000 7f25edc6a700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25e80a0710 con 0x7f25e809f2e0 2019-12-05T06:13:14.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.697+0000 7f25edc6a700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25e80a0b00 con 0x7f25e807e720 2019-12-05T06:13:14.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.697+0000 7f25edc6a700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25e808e5b0 con 0x7f25e80a7740 2019-12-05T06:13:14.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.697+0000 7f25e7fff700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f25e80a7740 0x7f25e808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:13:14.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.697+0000 7f25e77fe700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f25e807e720 0x7f25e807eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:13:14.715 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.697+0000 7f25e7fff700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f25e80a7740 0x7f25e808df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:51304/0 (socket says 172.21.15.42:51304) 2019-12-05T06:13:14.715 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.697+0000 7f25e7fff700 1 -- 172.21.15.42:0/4010052018 learned_addr learned my addr 172.21.15.42:0/4010052018 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:13:14.715 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.701+0000 7f25e7fff700 1 -- 172.21.15.42:0/4010052018 >> v2:172.21.15.40:3300/0 conn(0x7f25e807e720 msgr2=0x7f25e807eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:13:14.715 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.701+0000 7f25e7fff700 1 --2- 172.21.15.42:0/4010052018 >> v2:172.21.15.40:3300/0 conn(0x7f25e807e720 0x7f25e807eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:13:14.715 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.701+0000 7f25e7fff700 1 -- 172.21.15.42:0/4010052018 >> v1:172.21.15.40:6789/0 conn(0x7f25e809f2e0 legacy=0x7f25e809f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:13:14.715 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.701+0000 7f25e7fff700 1 -- 172.21.15.42:0/4010052018 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25e807f0a0 con 0x7f25e80a7740 2019-12-05T06:13:14.716 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.701+0000 7f25e7fff700 1 --2- 172.21.15.42:0/4010052018 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f25e80a7740 0x7f25e808df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f25e0008b10 tx=0x7f25e00100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:14.716 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.701+0000 7f25e7fff700 1 --2- 172.21.15.42:0/4010052018 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f25e80a7740 0x7f25e808df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f25e0008b10 tx=0x7f25e00100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:14.716 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.701+0000 7f25e7fff700 1 --2- 172.21.15.42:0/4010052018 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f25e80a7740 0x7f25e808df80 secure :-1 s=READY pgs=501 cs=0 l=1 rx=0x7f25e0008b10 tx=0x7f25e00100b0).ready entity=mon.2 client_cookie=e4c6f15d6f085a55 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:13:14.716 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.701+0000 7f25e7fff700 1 --2- 172.21.15.42:0/4010052018 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f25e80a7740 0x7f25e808df80 secure :-1 s=THROTTLE_DONE pgs=501 cs=0 l=1 rx=0x7f25e0008b10 tx=0x7f25e00100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:14.716 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.701+0000 7f25e6ffd700 1 -- 172.21.15.42:0/4010052018 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f25e000a160 con 0x7f25e80a7740 2019-12-05T06:13:14.716 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.701+0000 7f25e7fff700 1 --2- 172.21.15.42:0/4010052018 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f25e80a7740 0x7f25e808df80 secure :-1 s=THROTTLE_DONE pgs=501 cs=0 l=1 rx=0x7f25e0008b10 tx=0x7f25e00100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:14.716 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.701+0000 7f25e6ffd700 1 -- 172.21.15.42:0/4010052018 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25e0011a30 con 0x7f25e80a7740 2019-12-05T06:13:14.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.701+0000 7f25e7fff700 1 --2- 172.21.15.42:0/4010052018 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f25e80a7740 0x7f25e808df80 secure :-1 s=THROTTLE_DONE pgs=501 cs=0 l=1 rx=0x7f25e0008b10 tx=0x7f25e00100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:14.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.701+0000 7f25edc6a700 1 -- 172.21.15.42:0/4010052018 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f25e80a7740 msgr2=0x7f25e808df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:13:14.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.701+0000 7f25e6ffd700 1 -- 172.21.15.42:0/4010052018 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f25e000a550 con 0x7f25e80a7740 2019-12-05T06:13:14.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.701+0000 7f25edc6a700 1 --2- 172.21.15.42:0/4010052018 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f25e80a7740 0x7f25e808df80 secure :-1 s=READY pgs=501 cs=0 l=1 rx=0x7f25e0008b10 tx=0x7f25e00100b0).stop 2019-12-05T06:13:14.718 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.701+0000 7f25edc6a700 1 -- 172.21.15.42:0/4010052018 shutdown_connections 2019-12-05T06:13:14.718 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.701+0000 7f25edc6a700 1 --2- 172.21.15.42:0/4010052018 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f25e80a7740 0x7f25e808df80 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:13:14.718 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.701+0000 7f25edc6a700 1 --2- 172.21.15.42:0/4010052018 >> v2:172.21.15.40:3300/0 conn(0x7f25e807e720 0x7f25e807eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:13:14.718 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.701+0000 7f25edc6a700 1 -- 172.21.15.42:0/4010052018 shutdown_connections 2019-12-05T06:13:14.718 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.701+0000 7f25edc6a700 1 -- 172.21.15.42:0/4010052018 wait complete. 2019-12-05T06:13:14.718 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.701+0000 7f25edc6a700 1 -- 172.21.15.42:0/4010052018 >> 172.21.15.42:0/4010052018 conn(0x7f25e80aabe0 msgr2=0x7f25e80a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:13:14.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.701+0000 7f25edc6a700 1 Processor -- start 2019-12-05T06:13:14.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.701+0000 7f25edc6a700 1 -- start start 2019-12-05T06:13:14.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.701+0000 7f25edc6a700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f25e8130850 0x7f25e80a5680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:13:14.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.701+0000 7f25edc6a700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f25e8121840 0x7f25e812d290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:13:14.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.701+0000 7f25edc6a700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f25e812d7e0 0x7f25e811b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:13:14.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.701+0000 7f25edc6a700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25e80a5bd0 con 0x7f25e8130850 2019-12-05T06:13:14.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.705+0000 7f25edc6a700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25e8130d60 con 0x7f25e812d7e0 2019-12-05T06:13:14.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.705+0000 7f25edc6a700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25e8131000 con 0x7f25e8121840 2019-12-05T06:13:14.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.705+0000 7f25eca08700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f25e812d7e0 0x7f25e811b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:13:14.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.705+0000 7f25eca08700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f25e812d7e0 0x7f25e811b7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:50848/0 (socket says 172.21.15.42:50848) 2019-12-05T06:13:14.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.705+0000 7f25eca08700 1 -- 172.21.15.42:0/1649427727 learned_addr learned my addr 172.21.15.42:0/1649427727 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:13:14.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.705+0000 7f25e7fff700 1 --2- 172.21.15.42:0/1649427727 >> v2:172.21.15.40:3300/0 conn(0x7f25e8130850 0x7f25e80a5680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:13:14.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.705+0000 7f25e77fe700 1 --2- 172.21.15.42:0/1649427727 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f25e8121840 0x7f25e812d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:13:14.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.705+0000 7f25eca08700 1 -- 172.21.15.42:0/1649427727 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f25e8121840 msgr2=0x7f25e812d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:13:14.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.705+0000 7f25eca08700 1 --2- 172.21.15.42:0/1649427727 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f25e8121840 0x7f25e812d290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:13:14.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.705+0000 7f25eca08700 1 -- 172.21.15.42:0/1649427727 >> v2:172.21.15.40:3300/0 conn(0x7f25e8130850 msgr2=0x7f25e80a5680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:13:14.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.705+0000 7f25eca08700 1 --2- 172.21.15.42:0/1649427727 >> v2:172.21.15.40:3300/0 conn(0x7f25e8130850 0x7f25e80a5680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:13:14.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.705+0000 7f25eca08700 1 -- 172.21.15.42:0/1649427727 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25e8131540 con 0x7f25e812d7e0 2019-12-05T06:13:14.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.705+0000 7f25e77fe700 1 --2- 172.21.15.42:0/1649427727 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f25e8121840 0x7f25e812d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:13:14.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.705+0000 7f25eca08700 1 --2- 172.21.15.42:0/1649427727 >> v2:172.21.15.42:3300/0 conn(0x7f25e812d7e0 0x7f25e811b7a0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f25e8130f20 tx=0x7f25d8010840).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:14.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.705+0000 7f25eca08700 1 --2- 172.21.15.42:0/1649427727 >> v2:172.21.15.42:3300/0 conn(0x7f25e812d7e0 0x7f25e811b7a0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f25e8130f20 tx=0x7f25d8010840).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:14.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.705+0000 7f25eca08700 1 --2- 172.21.15.42:0/1649427727 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f25e812d7e0 0x7f25e811b7a0 secure :-1 s=READY pgs=2279 cs=0 l=1 rx=0x7f25e8130f20 tx=0x7f25d8010840).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:13:14.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.705+0000 7f25eca08700 1 --2- 172.21.15.42:0/1649427727 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f25e812d7e0 0x7f25e811b7a0 secure :-1 s=THROTTLE_DONE pgs=2279 cs=0 l=1 rx=0x7f25e8130f20 tx=0x7f25d8010840).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:14.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.705+0000 7f25eca08700 1 --2- 172.21.15.42:0/1649427727 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f25e812d7e0 0x7f25e811b7a0 secure :-1 s=THROTTLE_DONE pgs=2279 cs=0 l=1 rx=0x7f25e8130f20 tx=0x7f25d8010840).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:14.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.705+0000 7f25e4ff9700 1 -- 172.21.15.42:0/1649427727 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f25d8006a40 con 0x7f25e812d7e0 2019-12-05T06:13:14.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.705+0000 7f25eca08700 1 --2- 172.21.15.42:0/1649427727 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f25e812d7e0 0x7f25e811b7a0 secure :-1 s=THROTTLE_DONE pgs=2279 cs=0 l=1 rx=0x7f25e8130f20 tx=0x7f25d8010840).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:14.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.705+0000 7f25e4ff9700 1 -- 172.21.15.42:0/1649427727 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25d8006660 con 0x7f25e812d7e0 2019-12-05T06:13:14.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.705+0000 7f25edc6a700 1 -- 172.21.15.42:0/1649427727 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25e811bcf0 con 0x7f25e812d7e0 2019-12-05T06:13:14.724 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.705+0000 7f25e4ff9700 1 -- 172.21.15.42:0/1649427727 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f25d8011040 con 0x7f25e812d7e0 2019-12-05T06:13:14.724 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.705+0000 7f25edc6a700 1 -- 172.21.15.42:0/1649427727 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25e808eb70 con 0x7f25e812d7e0 2019-12-05T06:13:14.724 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.705+0000 7f25eca08700 1 --2- 172.21.15.42:0/1649427727 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f25e812d7e0 0x7f25e811b7a0 secure :-1 s=THROTTLE_DONE pgs=2279 cs=0 l=1 rx=0x7f25e8130f20 tx=0x7f25d8010840).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:14.724 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.705+0000 7f25edc6a700 1 -- 172.21.15.42:0/1649427727 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25e805d110 con 0x7f25e812d7e0 2019-12-05T06:13:14.724 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.705+0000 7f25e4ff9700 1 -- 172.21.15.42:0/1649427727 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f25d8011310 con 0x7f25e812d7e0 2019-12-05T06:13:14.724 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.705+0000 7f25eca08700 1 --2- 172.21.15.42:0/1649427727 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f25e812d7e0 0x7f25e811b7a0 secure :-1 s=THROTTLE_DONE pgs=2279 cs=0 l=1 rx=0x7f25e8130f20 tx=0x7f25d8010840).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:14.724 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.705+0000 7f25e4ff9700 1 --2- 172.21.15.42:0/1649427727 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f25cc03ae60 0x7f25cc03d370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:13:14.725 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.705+0000 7f25e4ff9700 1 -- 172.21.15.42:0/1649427727 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 10245+0+0 (secure 0 0 0) 0x7f25d804aff0 con 0x7f25e812d7e0 2019-12-05T06:13:14.725 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.705+0000 7f25e7fff700 1 --2- 172.21.15.42:0/1649427727 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f25cc03ae60 0x7f25cc03d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:13:14.725 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.709+0000 7f25e7fff700 1 --2- 172.21.15.42:0/1649427727 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f25cc03ae60 0x7f25cc03d370 crc :-1 s=READY pgs=959 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:13:14.726 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.713+0000 7f25eca08700 1 --2- 172.21.15.42:0/1649427727 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f25e812d7e0 0x7f25e811b7a0 secure :-1 s=THROTTLE_DONE pgs=2279 cs=0 l=1 rx=0x7f25e8130f20 tx=0x7f25d8010840).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:14.726 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.713+0000 7f25e4ff9700 1 -- 172.21.15.42:0/1649427727 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f25d8032090 con 0x7f25e812d7e0 2019-12-05T06:13:14.880 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.877+0000 7f25edc6a700 1 -- 172.21.15.42:0/1649427727 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f25e805b8c0 con 0x7f25e812d7e0 2019-12-05T06:13:14.881 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.877+0000 7f25eca08700 1 --2- 172.21.15.42:0/1649427727 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f25e812d7e0 0x7f25e811b7a0 secure :-1 s=THROTTLE_DONE pgs=2279 cs=0 l=1 rx=0x7f25e8130f20 tx=0x7f25d8010840).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:14.881 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.877+0000 7f25e4ff9700 1 -- 172.21.15.42:0/1649427727 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 274 v274) v1 ==== 95+0+1777 (secure 0 0 0) 0x7f25d80322f0 con 0x7f25e812d7e0 2019-12-05T06:13:14.882 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 274 2019-12-05T06:13:14.882 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:13:14.882 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":274,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":274,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:13:14.605063+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":98,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10246},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10246":{"gid":10246,"name":"a","rank":0,"incarnation":274,"state":"up:replay","state_seq":2,"addr":"172.21.15.42:6835/4032647860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":4032647860},{"type":"v1","addr":"172.21.15.42:6835","nonce":4032647860}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:13:14.884 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.881+0000 7f25edc6a700 1 -- 172.21.15.42:0/1649427727 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f25cc03ae60 msgr2=0x7f25cc03d370 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:13:14.884 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.881+0000 7f25edc6a700 1 --2- 172.21.15.42:0/1649427727 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f25cc03ae60 0x7f25cc03d370 crc :-1 s=READY pgs=959 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:13:14.884 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.881+0000 7f25edc6a700 1 -- 172.21.15.42:0/1649427727 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f25e812d7e0 msgr2=0x7f25e811b7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:13:14.884 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.881+0000 7f25edc6a700 1 --2- 172.21.15.42:0/1649427727 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f25e812d7e0 0x7f25e811b7a0 secure :-1 s=READY pgs=2279 cs=0 l=1 rx=0x7f25e8130f20 tx=0x7f25d8010840).stop 2019-12-05T06:13:14.885 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.881+0000 7f25edc6a700 1 -- 172.21.15.42:0/1649427727 shutdown_connections 2019-12-05T06:13:14.885 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.881+0000 7f25edc6a700 1 --2- 172.21.15.42:0/1649427727 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f25cc03ae60 0x7f25cc03d370 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:13:14.885 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.881+0000 7f25edc6a700 1 --2- 172.21.15.42:0/1649427727 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f25e812d7e0 0x7f25e811b7a0 unknown :-1 s=CLOSED pgs=2279 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:13:14.885 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.881+0000 7f25edc6a700 1 --2- 172.21.15.42:0/1649427727 >> v2:172.21.15.40:3300/0 conn(0x7f25e8130850 0x7f25e80a5680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:13:14.885 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.881+0000 7f25edc6a700 1 --2- 172.21.15.42:0/1649427727 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f25e8121840 0x7f25e812d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:13:14.885 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.881+0000 7f25edc6a700 1 -- 172.21.15.42:0/1649427727 shutdown_connections 2019-12-05T06:13:14.885 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.881+0000 7f25edc6a700 1 -- 172.21.15.42:0/1649427727 wait complete. 2019-12-05T06:13:14.886 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.881+0000 7f25edc6a700 1 -- 172.21.15.42:0/1649427727 >> 172.21.15.42:0/1649427727 conn(0x7f25e8066fe0 msgr2=0x7f25e811c610 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:13:14.894 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b down, removed from mdsmap 2019-12-05T06:13:14.894 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2019-12-05T06:13:14.894 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:13:14.984 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.973+0000 7fbe411ff700 1 Processor -- start 2019-12-05T06:13:14.984 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.973+0000 7fbe411ff700 1 -- start start 2019-12-05T06:13:14.984 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.977+0000 7fbe411ff700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fbe3c0a7740 0x7fbe3c08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:13:14.984 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.977+0000 7fbe411ff700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbe3c09fae0 0x7fbe3c07eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:13:14.984 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.977+0000 7fbe411ff700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe3c0a0710 con 0x7fbe3c07e720 2019-12-05T06:13:14.985 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.977+0000 7fbe411ff700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe3c0a0b00 con 0x7fbe3c0a7740 2019-12-05T06:13:14.985 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.977+0000 7fbe411ff700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe3c08e5b0 con 0x7fbe3c09fae0 2019-12-05T06:13:14.985 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.977+0000 7fbe3ad9d700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fbe3c07e720 0x7fbe3c09f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:50892/0 (socket says 172.21.15.42:50892) 2019-12-05T06:13:14.985 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.977+0000 7fbe3ad9d700 1 -- 172.21.15.42:0/1625988298 learned_addr learned my addr 172.21.15.42:0/1625988298 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:13:14.985 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.977+0000 7fbe409fe700 1 --2- 172.21.15.42:0/1625988298 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbe3c09fae0 0x7fbe3c07eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:13:14.985 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.977+0000 7fbe3b59e700 1 --2- 172.21.15.42:0/1625988298 >> v2:172.21.15.40:3300/0 conn(0x7fbe3c0a7740 0x7fbe3c08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:13:14.986 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.977+0000 7fbe3a59c700 1 -- 172.21.15.42:0/1625988298 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2567367641 0 0) 0x7fbe3c0a0710 con 0x7fbe3c07e720 2019-12-05T06:13:14.986 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.977+0000 7fbe3a59c700 1 -- 172.21.15.42:0/1625988298 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe280030b0 con 0x7fbe3c07e720 2019-12-05T06:13:14.986 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.977+0000 7fbe3a59c700 1 -- 172.21.15.42:0/1625988298 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3685266846 0 0) 0x7fbe280030b0 con 0x7fbe3c07e720 2019-12-05T06:13:14.986 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.977+0000 7fbe409fe700 1 -- 172.21.15.42:0/1625988298 >> v2:172.21.15.40:3300/0 conn(0x7fbe3c0a7740 msgr2=0x7fbe3c08df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:13:14.986 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.977+0000 7fbe409fe700 1 --2- 172.21.15.42:0/1625988298 >> v2:172.21.15.40:3300/0 conn(0x7fbe3c0a7740 0x7fbe3c08df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:13:14.986 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.977+0000 7fbe409fe700 1 -- 172.21.15.42:0/1625988298 >> v1:172.21.15.42:6789/0 conn(0x7fbe3c07e720 legacy=0x7fbe3c09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:13:14.986 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.977+0000 7fbe409fe700 1 -- 172.21.15.42:0/1625988298 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe3c07f0a0 con 0x7fbe3c09fae0 2019-12-05T06:13:14.987 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.977+0000 7fbe409fe700 1 --2- 172.21.15.42:0/1625988298 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbe3c09fae0 0x7fbe3c07eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fbe2c008790 tx=0x7fbe240014a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:14.987 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.977+0000 7fbe409fe700 1 --2- 172.21.15.42:0/1625988298 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbe3c09fae0 0x7fbe3c07eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fbe2c008790 tx=0x7fbe240014a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:14.987 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.977+0000 7fbe409fe700 1 --2- 172.21.15.42:0/1625988298 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbe3c09fae0 0x7fbe3c07eb50 secure :-1 s=READY pgs=502 cs=0 l=1 rx=0x7fbe2c008790 tx=0x7fbe240014a0).ready entity=mon.2 client_cookie=b4f967c870366048 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:13:14.987 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.977+0000 7fbe409fe700 1 --2- 172.21.15.42:0/1625988298 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbe3c09fae0 0x7fbe3c07eb50 secure :-1 s=THROTTLE_DONE pgs=502 cs=0 l=1 rx=0x7fbe2c008790 tx=0x7fbe240014a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:14.987 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.977+0000 7fbe3a59c700 1 -- 172.21.15.42:0/1625988298 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbe24001750 con 0x7fbe3c09fae0 2019-12-05T06:13:14.988 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.977+0000 7fbe409fe700 1 --2- 172.21.15.42:0/1625988298 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbe3c09fae0 0x7fbe3c07eb50 secure :-1 s=THROTTLE_DONE pgs=502 cs=0 l=1 rx=0x7fbe2c008790 tx=0x7fbe240014a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:14.988 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.977+0000 7fbe3a59c700 1 -- 172.21.15.42:0/1625988298 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe24001560 con 0x7fbe3c09fae0 2019-12-05T06:13:14.988 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.977+0000 7fbe409fe700 1 --2- 172.21.15.42:0/1625988298 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbe3c09fae0 0x7fbe3c07eb50 secure :-1 s=THROTTLE_DONE pgs=502 cs=0 l=1 rx=0x7fbe2c008790 tx=0x7fbe240014a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:14.988 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.977+0000 7fbe3a59c700 1 -- 172.21.15.42:0/1625988298 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbe2c008da0 con 0x7fbe3c09fae0 2019-12-05T06:13:14.988 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.977+0000 7fbe411ff700 1 -- 172.21.15.42:0/1625988298 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbe3c09fae0 msgr2=0x7fbe3c07eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:13:14.989 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.977+0000 7fbe411ff700 1 --2- 172.21.15.42:0/1625988298 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbe3c09fae0 0x7fbe3c07eb50 secure :-1 s=READY pgs=502 cs=0 l=1 rx=0x7fbe2c008790 tx=0x7fbe240014a0).stop 2019-12-05T06:13:14.989 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.977+0000 7fbe411ff700 1 -- 172.21.15.42:0/1625988298 shutdown_connections 2019-12-05T06:13:14.989 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.977+0000 7fbe411ff700 1 --2- 172.21.15.42:0/1625988298 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbe3c09fae0 0x7fbe3c07eb50 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:13:14.989 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.977+0000 7fbe411ff700 1 --2- 172.21.15.42:0/1625988298 >> v2:172.21.15.40:3300/0 conn(0x7fbe3c0a7740 0x7fbe3c08df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:13:14.989 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.977+0000 7fbe411ff700 1 -- 172.21.15.42:0/1625988298 shutdown_connections 2019-12-05T06:13:14.989 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.977+0000 7fbe411ff700 1 -- 172.21.15.42:0/1625988298 wait complete. 2019-12-05T06:13:14.989 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.977+0000 7fbe411ff700 1 -- 172.21.15.42:0/1625988298 >> 172.21.15.42:0/1625988298 conn(0x7fbe3c0aabe0 msgr2=0x7fbe3c0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:13:14.994 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.981+0000 7fbe411ff700 1 Processor -- start 2019-12-05T06:13:14.995 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.981+0000 7fbe411ff700 1 -- start start 2019-12-05T06:13:14.995 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.981+0000 7fbe411ff700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbe3c070ac0 0x7fbe3c078540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:13:14.995 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.981+0000 7fbe411ff700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe3c116db0 con 0x7fbe3c1221b0 2019-12-05T06:13:14.995 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.981+0000 7fbe411ff700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe3c1225e0 con 0x7fbe3c0a9930 2019-12-05T06:13:14.995 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.981+0000 7fbe411ff700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe3c117700 con 0x7fbe3c070ac0 2019-12-05T06:13:14.995 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.981+0000 7fbe3b59e700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fbe3c0a9930 0x7fbe3c1165b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:50896/0 (socket says 172.21.15.42:50896) 2019-12-05T06:13:14.995 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.981+0000 7fbe3b59e700 1 -- 172.21.15.42:0/3283132563 learned_addr learned my addr 172.21.15.42:0/3283132563 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:13:14.996 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.981+0000 7fbe409fe700 1 --2- 172.21.15.42:0/3283132563 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbe3c070ac0 0x7fbe3c078540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:13:14.996 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.981+0000 7fbe33fff700 1 -- 172.21.15.42:0/3283132563 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3815214324 0 0) 0x7fbe3c1225e0 con 0x7fbe3c0a9930 2019-12-05T06:13:14.996 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.981+0000 7fbe33fff700 1 -- 172.21.15.42:0/3283132563 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe200030b0 con 0x7fbe3c0a9930 2019-12-05T06:13:14.996 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.981+0000 7fbe33fff700 1 -- 172.21.15.42:0/3283132563 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 3117992034 0 0) 0x7fbe200030b0 con 0x7fbe3c0a9930 2019-12-05T06:13:14.996 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.981+0000 7fbe33fff700 1 -- 172.21.15.42:0/3283132563 --> v1:172.21.15.42:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbe3c1225e0 con 0x7fbe3c0a9930 2019-12-05T06:13:14.996 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.981+0000 7fbe33fff700 1 -- 172.21.15.42:0/3283132563 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3267213460 0 0) 0x7fbe24000fa0 con 0x7fbe3c1221b0 2019-12-05T06:13:14.997 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.981+0000 7fbe33fff700 1 -- 172.21.15.42:0/3283132563 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe200030b0 con 0x7fbe3c1221b0 2019-12-05T06:13:14.997 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.981+0000 7fbe33fff700 1 -- 172.21.15.42:0/3283132563 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7fbe34001810 con 0x7fbe3c0a9930 2019-12-05T06:13:14.997 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.981+0000 7fbe33fff700 1 -- 172.21.15.42:0/3283132563 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbe3c070ac0 msgr2=0x7fbe3c078540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:13:14.997 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.981+0000 7fbe33fff700 1 --2- 172.21.15.42:0/3283132563 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbe3c070ac0 0x7fbe3c078540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:13:14.997 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.981+0000 7fbe33fff700 1 -- 172.21.15.42:0/3283132563 >> v1:172.21.15.42:6789/0 conn(0x7fbe3c0a9930 legacy=0x7fbe3c1165b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:13:14.997 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.981+0000 7fbe33fff700 1 -- 172.21.15.42:0/3283132563 >> v1:172.21.15.40:6789/0 conn(0x7fbe3c1221b0 legacy=0x7fbe3c0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:13:14.997 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.981+0000 7fbe33fff700 1 --2- 172.21.15.42:0/3283132563 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbe20003ad0 0x7fbe20004420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:13:14.998 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.981+0000 7fbe33fff700 1 --2- 172.21.15.42:0/3283132563 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbe20003f90 0x7fbe200049d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:13:14.998 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.981+0000 7fbe33fff700 1 --2- 172.21.15.42:0/3283132563 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbe20009660 0x7fbe2000baa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:13:14.998 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.981+0000 7fbe33fff700 1 -- 172.21.15.42:0/3283132563 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe3c117700 con 0x7fbe20009660 2019-12-05T06:13:14.998 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.981+0000 7fbe33fff700 1 -- 172.21.15.42:0/3283132563 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe20005220 con 0x7fbe20003f90 2019-12-05T06:13:14.998 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.981+0000 7fbe33fff700 1 -- 172.21.15.42:0/3283132563 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe2000c020 con 0x7fbe20003ad0 2019-12-05T06:13:14.999 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.981+0000 7fbe409fe700 1 --2- 172.21.15.42:0/3283132563 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbe3c070ac0 0x7fbe3c078540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:13:14.999 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.981+0000 7fbe3ad9d700 1 --2- 172.21.15.42:0/3283132563 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbe20003f90 0x7fbe200049d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:13:14.999 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.981+0000 7fbe409fe700 1 --2- 172.21.15.42:0/3283132563 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbe20009660 0x7fbe2000baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:13:15.000 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.981+0000 7fbe3b59e700 1 --2- 172.21.15.42:0/3283132563 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbe20003ad0 0x7fbe20004420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:13:15.000 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.981+0000 7fbe3ad9d700 1 -- 172.21.15.42:0/3283132563 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbe20003ad0 msgr2=0x7fbe20004420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:13:15.000 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.981+0000 7fbe3ad9d700 1 --2- 172.21.15.42:0/3283132563 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbe20003ad0 0x7fbe20004420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:13:15.000 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.981+0000 7fbe3ad9d700 1 -- 172.21.15.42:0/3283132563 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbe20009660 msgr2=0x7fbe2000baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:13:15.000 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.981+0000 7fbe3ad9d700 1 --2- 172.21.15.42:0/3283132563 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbe20009660 0x7fbe2000baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:13:15.001 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.985+0000 7fbe3ad9d700 1 -- 172.21.15.42:0/3283132563 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe3c0a6a70 con 0x7fbe20003f90 2019-12-05T06:13:15.001 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.985+0000 7fbe411ff700 1 -- 172.21.15.42:0/3283132563 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe3c070ac0 con 0x7fbe20003f90 2019-12-05T06:13:15.001 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.985+0000 7fbe411ff700 1 -- 172.21.15.42:0/3283132563 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe3c070f90 con 0x7fbe20003f90 2019-12-05T06:13:15.001 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.985+0000 7fbe3ad9d700 1 --2- 172.21.15.42:0/3283132563 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbe20003f90 0x7fbe200049d0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fbe240012c0 tx=0x7fbe24010e00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:15.001 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.985+0000 7fbe3ad9d700 1 --2- 172.21.15.42:0/3283132563 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbe20003f90 0x7fbe200049d0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fbe240012c0 tx=0x7fbe24010e00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:15.001 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.985+0000 7fbe3ad9d700 1 --2- 172.21.15.42:0/3283132563 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbe20003f90 0x7fbe200049d0 secure :-1 s=READY pgs=2282 cs=0 l=1 rx=0x7fbe240012c0 tx=0x7fbe24010e00).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:13:15.001 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.985+0000 7fbe3ad9d700 1 --2- 172.21.15.42:0/3283132563 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbe20003f90 0x7fbe200049d0 secure :-1 s=THROTTLE_DONE pgs=2282 cs=0 l=1 rx=0x7fbe240012c0 tx=0x7fbe24010e00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:15.002 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.985+0000 7fbe33fff700 1 -- 172.21.15.42:0/3283132563 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbe2400cb70 con 0x7fbe20003f90 2019-12-05T06:13:15.002 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.985+0000 7fbe3ad9d700 1 --2- 172.21.15.42:0/3283132563 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbe20003f90 0x7fbe200049d0 secure :-1 s=THROTTLE_DONE pgs=2282 cs=0 l=1 rx=0x7fbe240012c0 tx=0x7fbe24010e00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:15.002 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.985+0000 7fbe33fff700 1 -- 172.21.15.42:0/3283132563 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe2400c790 con 0x7fbe20003f90 2019-12-05T06:13:15.002 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.985+0000 7fbe3ad9d700 1 --2- 172.21.15.42:0/3283132563 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbe20003f90 0x7fbe200049d0 secure :-1 s=THROTTLE_DONE pgs=2282 cs=0 l=1 rx=0x7fbe240012c0 tx=0x7fbe24010e00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:15.003 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.985+0000 7fbe33fff700 1 -- 172.21.15.42:0/3283132563 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbe24004620 con 0x7fbe20003f90 2019-12-05T06:13:15.003 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.985+0000 7fbe3ad9d700 1 --2- 172.21.15.42:0/3283132563 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbe20003f90 0x7fbe200049d0 secure :-1 s=THROTTLE_DONE pgs=2282 cs=0 l=1 rx=0x7fbe240012c0 tx=0x7fbe24010e00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:15.003 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.985+0000 7fbe33fff700 1 -- 172.21.15.42:0/3283132563 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fbe24004820 con 0x7fbe20003f90 2019-12-05T06:13:15.003 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.985+0000 7fbe3ad9d700 1 --2- 172.21.15.42:0/3283132563 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbe20003f90 0x7fbe200049d0 secure :-1 s=THROTTLE_DONE pgs=2282 cs=0 l=1 rx=0x7fbe240012c0 tx=0x7fbe24010e00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:15.004 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.985+0000 7fbe33fff700 1 --2- 172.21.15.42:0/3283132563 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbe20043f60 0x7fbe20046470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:13:15.004 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.985+0000 7fbe33fff700 1 -- 172.21.15.42:0/3283132563 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 10245+0+0 (secure 0 0 0) 0x7fbe3c116db0 con 0x7fbe20003f90 2019-12-05T06:13:15.004 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.985+0000 7fbe3b59e700 1 --2- 172.21.15.42:0/3283132563 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbe20043f60 0x7fbe20046470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:13:15.004 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.985+0000 7fbe3b59e700 1 --2- 172.21.15.42:0/3283132563 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbe20043f60 0x7fbe20046470 crc :-1 s=READY pgs=960 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:13:15.004 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.989+0000 7fbe411ff700 1 -- 172.21.15.42:0/3283132563 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe14003200 con 0x7fbe20003f90 2019-12-05T06:13:15.005 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.997+0000 7fbe3ad9d700 1 --2- 172.21.15.42:0/3283132563 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbe20003f90 0x7fbe200049d0 secure :-1 s=THROTTLE_DONE pgs=2282 cs=0 l=1 rx=0x7fbe240012c0 tx=0x7fbe24010e00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:15.005 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:14.997+0000 7fbe33fff700 1 -- 172.21.15.42:0/3283132563 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fbe24032270 con 0x7fbe20003f90 2019-12-05T06:13:15.042 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:13:15.045 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:13:15.162 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:15.157+0000 7fbe411ff700 1 -- 172.21.15.42:0/3283132563 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbe14003630 con 0x7fbe20003f90 2019-12-05T06:13:15.162 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:15.157+0000 7fbe3ad9d700 1 --2- 172.21.15.42:0/3283132563 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbe20003f90 0x7fbe200049d0 secure :-1 s=THROTTLE_DONE pgs=2282 cs=0 l=1 rx=0x7fbe240012c0 tx=0x7fbe24010e00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:15.163 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:15.157+0000 7fbe33fff700 1 -- 172.21.15.42:0/3283132563 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 274 v274) v1 ==== 95+0+1777 (secure 0 0 0) 0x7fbe24010270 con 0x7fbe20003f90 2019-12-05T06:13:15.163 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 274 2019-12-05T06:13:15.164 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:13:15.164 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":274,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":274,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:13:14.605063+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":98,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10246},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10246":{"gid":10246,"name":"a","rank":0,"incarnation":274,"state":"up:replay","state_seq":2,"addr":"172.21.15.42:6835/4032647860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":4032647860},{"type":"v1","addr":"172.21.15.42:6835","nonce":4032647860}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:13:15.165 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:15.161+0000 7fbe411ff700 1 -- 172.21.15.42:0/3283132563 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbe20043f60 msgr2=0x7fbe20046470 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:13:15.166 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:15.161+0000 7fbe411ff700 1 --2- 172.21.15.42:0/3283132563 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbe20043f60 0x7fbe20046470 crc :-1 s=READY pgs=960 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:13:15.166 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:15.161+0000 7fbe409fe700 1 -- 172.21.15.42:0/3283132563 reap_dead start 2019-12-05T06:13:15.166 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:15.161+0000 7fbe411ff700 1 -- 172.21.15.42:0/3283132563 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbe20003f90 msgr2=0x7fbe200049d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:13:15.166 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:15.161+0000 7fbe411ff700 1 --2- 172.21.15.42:0/3283132563 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbe20003f90 0x7fbe200049d0 secure :-1 s=READY pgs=2282 cs=0 l=1 rx=0x7fbe240012c0 tx=0x7fbe24010e00).stop 2019-12-05T06:13:15.166 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:15.161+0000 7fbe411ff700 1 -- 172.21.15.42:0/3283132563 shutdown_connections 2019-12-05T06:13:15.166 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:15.161+0000 7fbe411ff700 1 --2- 172.21.15.42:0/3283132563 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbe20003f90 0x7fbe200049d0 unknown :-1 s=CLOSED pgs=2282 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:13:15.167 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:15.161+0000 7fbe411ff700 1 -- 172.21.15.42:0/3283132563 shutdown_connections 2019-12-05T06:13:15.167 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:15.161+0000 7fbe411ff700 1 -- 172.21.15.42:0/3283132563 wait complete. 2019-12-05T06:13:15.167 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:15.161+0000 7fbe411ff700 1 -- 172.21.15.42:0/3283132563 >> 172.21.15.42:0/3283132563 conn(0x7fbe3c05a120 msgr2=0x7fbe3c07e4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:13:15.175 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 274, "feature_flags": { "enable_multiple": false, "ever_enabled_multiple": false }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2019-12-05T05:03:35.604738+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 274, "ever_allowed_features": 2, "explicitly_allowed_features": 2, "failed": [], "flags": 18, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_10246": { "addr": "172.21.15.42:6835/4032647860", "addrs": { "addrvec": [ { "addr": "172.21.15.42:6834", "nonce": 4032647860, "type": "v2" }, { "addr": "172.21.15.42:6835", "nonce": 4032647860, "type": "v1" } ] }, "export_targets": [], "features": 4540138292836696063, "flags": 0, "gid": 10246, "incarnation": 274, "name": "a", "rank": 0, "state": "up:replay", "state_seq": 2 } }, "last_failure": 0, "last_failure_osd_epoch": 98, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 2, "min_compat_client": "0 (unknown)", "modified": "2019-12-05T06:13:14.605063+0000", "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 10246 } } } ], "standbys": [] } 2019-12-05T06:13:17.176 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:13:17.259 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:17.249+0000 7f212e833700 1 Processor -- start 2019-12-05T06:13:17.260 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:17.249+0000 7f212e833700 1 -- start start 2019-12-05T06:13:17.260 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:17.249+0000 7f212e833700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f212808df70 0x7f212809f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:13:17.260 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:17.249+0000 7f212e833700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f212809f820 0x7f212814efd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:13:17.260 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:17.249+0000 7f212e833700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f212809fc50 con 0x7f21280a7730 2019-12-05T06:13:17.260 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:17.249+0000 7f212e833700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21280a0af0 con 0x7f212808df70 2019-12-05T06:13:17.260 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:17.249+0000 7f212e833700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f212808e5a0 con 0x7f212809f820 2019-12-05T06:13:17.261 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:17.253+0000 7f212d5d1700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f212809f820 0x7f212814efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:13:17.261 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:17.253+0000 7f212d5d1700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f212809f820 0x7f212814efd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:50876/0 (socket says 172.21.15.42:50876) 2019-12-05T06:13:17.261 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:17.253+0000 7f212d5d1700 1 -- 172.21.15.42:0/3947920897 learned_addr learned my addr 172.21.15.42:0/3947920897 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:13:17.261 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:17.253+0000 7f2127fff700 1 --2- 172.21.15.42:0/3947920897 >> v2:172.21.15.40:3300/0 conn(0x7f212808df70 0x7f212809f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:13:17.261 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:17.253+0000 7f212d5d1700 1 -- 172.21.15.42:0/3947920897 >> v2:172.21.15.40:3300/0 conn(0x7f212808df70 msgr2=0x7f212809f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:13:17.261 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:17.253+0000 7f212d5d1700 1 --2- 172.21.15.42:0/3947920897 >> v2:172.21.15.40:3300/0 conn(0x7f212808df70 0x7f212809f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:13:17.261 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:17.253+0000 7f212d5d1700 1 -- 172.21.15.42:0/3947920897 >> v1:172.21.15.40:6789/0 conn(0x7f21280a7730 legacy=0x7f212807e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:13:17.262 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:17.253+0000 7f212d5d1700 1 -- 172.21.15.42:0/3947920897 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21280a0700 con 0x7f212809f820 2019-12-05T06:13:17.262 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:17.253+0000 7f212d5d1700 1 --2- 172.21.15.42:0/3947920897 >> v2:172.21.15.42:3300/0 conn(0x7f212809f820 0x7f212814efd0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f2120002f30 tx=0x7f212000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:17.262 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:17.253+0000 7f212d5d1700 1 --2- 172.21.15.42:0/3947920897 >> v2:172.21.15.42:3300/0 conn(0x7f212809f820 0x7f212814efd0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f2120002f30 tx=0x7f212000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:17.262 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:17.253+0000 7f212d5d1700 1 --2- 172.21.15.42:0/3947920897 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f212809f820 0x7f212814efd0 secure :-1 s=READY pgs=2283 cs=0 l=1 rx=0x7f2120002f30 tx=0x7f212000fd30).ready entity=mon.1 client_cookie=2230675ba6e25951 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:13:17.262 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:17.253+0000 7f212d5d1700 1 --2- 172.21.15.42:0/3947920897 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f212809f820 0x7f212814efd0 secure :-1 s=THROTTLE_DONE pgs=2283 cs=0 l=1 rx=0x7f2120002f30 tx=0x7f212000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:17.262 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:17.253+0000 7f21277fe700 1 -- 172.21.15.42:0/3947920897 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2120008c90 con 0x7f212809f820 2019-12-05T06:13:17.262 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:17.253+0000 7f212d5d1700 1 --2- 172.21.15.42:0/3947920897 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f212809f820 0x7f212814efd0 secure :-1 s=THROTTLE_DONE pgs=2283 cs=0 l=1 rx=0x7f2120002f30 tx=0x7f212000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:17.263 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:17.253+0000 7f212d5d1700 1 --2- 172.21.15.42:0/3947920897 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f212809f820 0x7f212814efd0 secure :-1 s=THROTTLE_DONE pgs=2283 cs=0 l=1 rx=0x7f2120002f30 tx=0x7f212000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:17.263 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:17.253+0000 7f21277fe700 1 -- 172.21.15.42:0/3947920897 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21200088b0 con 0x7f212809f820 2019-12-05T06:13:17.263 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:17.253+0000 7f21277fe700 1 -- 172.21.15.42:0/3947920897 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2120011ad0 con 0x7f212809f820 2019-12-05T06:13:17.263 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:17.253+0000 7f212e833700 1 -- 172.21.15.42:0/3947920897 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f212809f820 msgr2=0x7f212814efd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:13:17.263 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:17.253+0000 7f212e833700 1 --2- 172.21.15.42:0/3947920897 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f212809f820 0x7f212814efd0 secure :-1 s=READY pgs=2283 cs=0 l=1 rx=0x7f2120002f30 tx=0x7f212000fd30).stop 2019-12-05T06:13:17.263 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:17.253+0000 7f212e833700 1 -- 172.21.15.42:0/3947920897 shutdown_connections 2019-12-05T06:13:17.263 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:17.253+0000 7f212e833700 1 --2- 172.21.15.42:0/3947920897 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f212809f820 0x7f212814efd0 unknown :-1 s=CLOSED pgs=2283 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:13:17.264 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:17.253+0000 7f212e833700 1 --2- 172.21.15.42:0/3947920897 >> v2:172.21.15.40:3300/0 conn(0x7f212808df70 0x7f212809f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:13:17.264 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:17.253+0000 7f212e833700 1 -- 172.21.15.42:0/3947920897 shutdown_connections 2019-12-05T06:13:17.264 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:17.253+0000 7f212e833700 1 -- 172.21.15.42:0/3947920897 wait complete. 2019-12-05T06:13:17.264 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:17.253+0000 7f212e833700 1 -- 172.21.15.42:0/3947920897 >> 172.21.15.42:0/3947920897 conn(0x7f21280aabd0 msgr2=0x7f21280a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:13:17.268 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:17.257+0000 7f212e833700 1 Processor -- start 2019-12-05T06:13:17.268 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:17.257+0000 7f212e833700 1 -- start start 2019-12-05T06:13:17.268 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:17.257+0000 7f212e833700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f21281236c0 0x7f21280a6c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:13:17.268 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:17.257+0000 7f212e833700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f212807afd0 0x7f2128131400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:13:17.269 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:17.257+0000 7f212e833700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21280a71c0 con 0x7f212807b400 2019-12-05T06:13:17.269 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:17.257+0000 7f212e833700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21280a4c40 con 0x7f212807afd0 2019-12-05T06:13:17.269 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:17.257+0000 7f212e833700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21280a5f40 con 0x7f21281236c0 2019-12-05T06:13:17.269 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:17.257+0000 7f2127fff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f212807afd0 0x7f2128131400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:13:17.269 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:17.257+0000 7f212d5d1700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f212807b400 0x7f212808eb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:50918/0 (socket says 172.21.15.42:50918) 2019-12-05T06:13:17.269 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:17.257+0000 7f2127fff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f212807afd0 0x7f2128131400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:50880/0 (socket says 172.21.15.42:50880) 2019-12-05T06:13:17.269 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:17.257+0000 7f212d5d1700 1 -- 172.21.15.42:0/2815191550 learned_addr learned my addr 172.21.15.42:0/2815191550 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:13:17.270 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:17.257+0000 7f212cdd0700 1 --2- 172.21.15.42:0/2815191550 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f21281236c0 0x7f21280a6c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:13:17.270 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:17.257+0000 7f2127fff700 1 -- 172.21.15.42:0/2815191550 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f21281236c0 msgr2=0x7f21280a6c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:13:17.270 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:17.257+0000 7f2127fff700 1 --2- 172.21.15.42:0/2815191550 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f21281236c0 0x7f21280a6c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:13:17.270 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:17.257+0000 7f2127fff700 1 -- 172.21.15.42:0/2815191550 >> v1:172.21.15.42:6789/0 conn(0x7f212807b400 legacy=0x7f212808eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:13:17.270 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:17.257+0000 7f21257fa700 1 -- 172.21.15.42:0/2815191550 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2656683809 0 0) 0x7f21280a71c0 con 0x7f212807b400 2019-12-05T06:13:17.270 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:17.257+0000 7f2127fff700 1 -- 172.21.15.42:0/2815191550 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21280a5c30 con 0x7f212807afd0 2019-12-05T06:13:17.270 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:17.257+0000 7f2127fff700 1 --2- 172.21.15.42:0/2815191550 >> v2:172.21.15.42:3300/0 conn(0x7f212807afd0 0x7f2128131400 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f211c002f30 tx=0x7f211c010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:17.271 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:17.261+0000 7f2127fff700 1 --2- 172.21.15.42:0/2815191550 >> v2:172.21.15.42:3300/0 conn(0x7f212807afd0 0x7f2128131400 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f211c002f30 tx=0x7f211c010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:17.271 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:17.261+0000 7f2127fff700 1 --2- 172.21.15.42:0/2815191550 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f212807afd0 0x7f2128131400 secure :-1 s=READY pgs=2285 cs=0 l=1 rx=0x7f211c002f30 tx=0x7f211c010b40).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:13:17.271 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:17.261+0000 7f2127fff700 1 --2- 172.21.15.42:0/2815191550 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f212807afd0 0x7f2128131400 secure :-1 s=THROTTLE_DONE pgs=2285 cs=0 l=1 rx=0x7f211c002f30 tx=0x7f211c010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:17.271 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:17.261+0000 7f21257fa700 1 -- 172.21.15.42:0/2815191550 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f211c0125f0 con 0x7f212807afd0 2019-12-05T06:13:17.271 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:17.261+0000 7f2127fff700 1 --2- 172.21.15.42:0/2815191550 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f212807afd0 0x7f2128131400 secure :-1 s=THROTTLE_DONE pgs=2285 cs=0 l=1 rx=0x7f211c002f30 tx=0x7f211c010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:17.271 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:17.261+0000 7f2127fff700 1 --2- 172.21.15.42:0/2815191550 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f212807afd0 0x7f2128131400 secure :-1 s=THROTTLE_DONE pgs=2285 cs=0 l=1 rx=0x7f211c002f30 tx=0x7f211c010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:17.271 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:17.261+0000 7f21257fa700 1 -- 172.21.15.42:0/2815191550 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f211c007150 con 0x7f212807afd0 2019-12-05T06:13:17.272 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:17.261+0000 7f212e833700 1 -- 172.21.15.42:0/2815191550 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21280a6720 con 0x7f212807afd0 2019-12-05T06:13:17.272 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:17.261+0000 7f21257fa700 1 -- 172.21.15.42:0/2815191550 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f211c012bd0 con 0x7f212807afd0 2019-12-05T06:13:17.272 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:17.261+0000 7f212e833700 1 -- 172.21.15.42:0/2815191550 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2128131950 con 0x7f212807afd0 2019-12-05T06:13:17.272 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:17.261+0000 7f2127fff700 1 --2- 172.21.15.42:0/2815191550 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f212807afd0 0x7f2128131400 secure :-1 s=THROTTLE_DONE pgs=2285 cs=0 l=1 rx=0x7f211c002f30 tx=0x7f211c010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:17.272 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:17.261+0000 7f21257fa700 1 -- 172.21.15.42:0/2815191550 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f211c005540 con 0x7f212807afd0 2019-12-05T06:13:17.272 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:17.261+0000 7f2127fff700 1 --2- 172.21.15.42:0/2815191550 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f212807afd0 0x7f2128131400 secure :-1 s=THROTTLE_DONE pgs=2285 cs=0 l=1 rx=0x7f211c002f30 tx=0x7f211c010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:17.272 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:17.261+0000 7f21257fa700 1 --2- 172.21.15.42:0/2815191550 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f210803abe0 0x7f210803d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:13:17.273 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:17.261+0000 7f21257fa700 1 -- 172.21.15.42:0/2815191550 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 10245+0+0 (secure 0 0 0) 0x7f211c04b140 con 0x7f212807afd0 2019-12-05T06:13:17.273 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:17.261+0000 7f212cdd0700 1 --2- 172.21.15.42:0/2815191550 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f210803abe0 0x7f210803d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:13:17.273 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:17.261+0000 7f212cdd0700 1 --2- 172.21.15.42:0/2815191550 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f210803abe0 0x7f210803d0f0 crc :-1 s=READY pgs=961 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:13:17.274 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:17.265+0000 7f212e833700 1 -- 172.21.15.42:0/2815191550 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f212805d100 con 0x7f212807afd0 2019-12-05T06:13:17.277 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:17.273+0000 7f2127fff700 1 --2- 172.21.15.42:0/2815191550 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f212807afd0 0x7f2128131400 secure :-1 s=THROTTLE_DONE pgs=2285 cs=0 l=1 rx=0x7f211c002f30 tx=0x7f211c010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:17.277 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:17.273+0000 7f21257fa700 1 -- 172.21.15.42:0/2815191550 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f211c032160 con 0x7f212807afd0 2019-12-05T06:13:17.437 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:17.433+0000 7f212e833700 1 -- 172.21.15.42:0/2815191550 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f212805b8b0 con 0x7f212807afd0 2019-12-05T06:13:17.438 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:17.433+0000 7f2127fff700 1 --2- 172.21.15.42:0/2815191550 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f212807afd0 0x7f2128131400 secure :-1 s=THROTTLE_DONE pgs=2285 cs=0 l=1 rx=0x7f211c002f30 tx=0x7f211c010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:17.438 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:17.433+0000 7f21257fa700 1 -- 172.21.15.42:0/2815191550 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 274 v274) v1 ==== 95+0+1777 (secure 0 0 0) 0x7f211c0323c0 con 0x7f212807afd0 2019-12-05T06:13:17.438 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 274 2019-12-05T06:13:17.439 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:13:17.439 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":274,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":274,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:13:14.605063+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":98,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10246},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10246":{"gid":10246,"name":"a","rank":0,"incarnation":274,"state":"up:replay","state_seq":2,"addr":"172.21.15.42:6835/4032647860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":4032647860},{"type":"v1","addr":"172.21.15.42:6835","nonce":4032647860}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:13:17.442 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:17.437+0000 7f212e833700 1 -- 172.21.15.42:0/2815191550 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f210803abe0 msgr2=0x7f210803d0f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:13:17.442 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:17.437+0000 7f212e833700 1 --2- 172.21.15.42:0/2815191550 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f210803abe0 0x7f210803d0f0 crc :-1 s=READY pgs=961 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:13:17.442 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:17.437+0000 7f212e833700 1 -- 172.21.15.42:0/2815191550 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f212807afd0 msgr2=0x7f2128131400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:13:17.443 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:17.437+0000 7f212e833700 1 --2- 172.21.15.42:0/2815191550 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f212807afd0 0x7f2128131400 secure :-1 s=READY pgs=2285 cs=0 l=1 rx=0x7f211c002f30 tx=0x7f211c010b40).stop 2019-12-05T06:13:17.443 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:17.437+0000 7f212e833700 1 -- 172.21.15.42:0/2815191550 shutdown_connections 2019-12-05T06:13:17.443 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:17.437+0000 7f212e833700 1 --2- 172.21.15.42:0/2815191550 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f210803abe0 0x7f210803d0f0 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:13:17.443 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:17.437+0000 7f212e833700 1 --2- 172.21.15.42:0/2815191550 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f21281236c0 0x7f21280a6c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:13:17.444 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:17.437+0000 7f212e833700 1 --2- 172.21.15.42:0/2815191550 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f212807afd0 0x7f2128131400 unknown :-1 s=CLOSED pgs=2285 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:13:17.444 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:17.437+0000 7f212e833700 1 -- 172.21.15.42:0/2815191550 shutdown_connections 2019-12-05T06:13:17.445 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:17.437+0000 7f212e833700 1 -- 172.21.15.42:0/2815191550 wait complete. 2019-12-05T06:13:17.445 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:17.437+0000 7f212e833700 1 -- 172.21.15.42:0/2815191550 >> 172.21.15.42:0/2815191550 conn(0x7f212808fd90 msgr2=0x7f21280596f0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:13:17.449 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2019-12-05T06:13:19.451 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:13:19.560 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:19.549+0000 7f40f95fd700 1 Processor -- start 2019-12-05T06:13:19.560 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:19.549+0000 7f40f95fd700 1 -- start start 2019-12-05T06:13:19.561 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:19.553+0000 7f40f95fd700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f40f40a7730 0x7f40f408df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:13:19.562 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:19.553+0000 7f40f95fd700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f40f409fad0 0x7f40f407eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:13:19.562 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:19.553+0000 7f40f95fd700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40f40a0700 con 0x7f40f407e710 2019-12-05T06:13:19.562 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:19.553+0000 7f40f95fd700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40f40a0af0 con 0x7f40f40a7730 2019-12-05T06:13:19.562 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:19.553+0000 7f40f95fd700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40f408e5a0 con 0x7f40f409fad0 2019-12-05T06:13:19.563 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:19.553+0000 7f40f37fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f40f40a7730 0x7f40f408df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:13:19.563 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:19.553+0000 7f40f37fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f40f40a7730 0x7f40f408df70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:50892/0 (socket says 172.21.15.42:50892) 2019-12-05T06:13:19.563 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:19.553+0000 7f40f37fe700 1 -- 172.21.15.42:0/1158872173 learned_addr learned my addr 172.21.15.42:0/1158872173 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:13:19.563 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:19.553+0000 7f40f3fff700 1 --2- 172.21.15.42:0/1158872173 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f40f409fad0 0x7f40f407eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:13:19.563 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:19.553+0000 7f40f37fe700 1 -- 172.21.15.42:0/1158872173 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f40f409fad0 msgr2=0x7f40f407eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:13:19.563 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:19.553+0000 7f40f37fe700 1 --2- 172.21.15.42:0/1158872173 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f40f409fad0 0x7f40f407eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:13:19.563 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:19.553+0000 7f40f37fe700 1 -- 172.21.15.42:0/1158872173 >> v1:172.21.15.40:6789/0 conn(0x7f40f407e710 legacy=0x7f40f409f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:13:19.564 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:19.553+0000 7f40f37fe700 1 -- 172.21.15.42:0/1158872173 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40f407f090 con 0x7f40f40a7730 2019-12-05T06:13:19.564 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:19.553+0000 7f40f37fe700 1 --2- 172.21.15.42:0/1158872173 >> v2:172.21.15.42:3300/0 conn(0x7f40f40a7730 0x7f40f408df70 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f40e4008790 tx=0x7f40e400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:19.564 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:19.553+0000 7f40f37fe700 1 --2- 172.21.15.42:0/1158872173 >> v2:172.21.15.42:3300/0 conn(0x7f40f40a7730 0x7f40f408df70 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f40e4008790 tx=0x7f40e400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:19.564 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:19.553+0000 7f40f37fe700 1 --2- 172.21.15.42:0/1158872173 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f40f40a7730 0x7f40f408df70 secure :-1 s=READY pgs=2286 cs=0 l=1 rx=0x7f40e4008790 tx=0x7f40e400fd30).ready entity=mon.1 client_cookie=b0236e728bc54676 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:13:19.564 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:19.553+0000 7f40f37fe700 1 --2- 172.21.15.42:0/1158872173 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f40f40a7730 0x7f40f408df70 secure :-1 s=THROTTLE_DONE pgs=2286 cs=0 l=1 rx=0x7f40e4008790 tx=0x7f40e400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:19.566 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:19.553+0000 7f40f37fe700 1 --2- 172.21.15.42:0/1158872173 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f40f40a7730 0x7f40f408df70 secure :-1 s=THROTTLE_DONE pgs=2286 cs=0 l=1 rx=0x7f40e4008790 tx=0x7f40e400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:19.566 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:19.553+0000 7f40f27fc700 1 -- 172.21.15.42:0/1158872173 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f40e4008c90 con 0x7f40f40a7730 2019-12-05T06:13:19.566 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:19.553+0000 7f40f37fe700 1 --2- 172.21.15.42:0/1158872173 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f40f40a7730 0x7f40f408df70 secure :-1 s=THROTTLE_DONE pgs=2286 cs=0 l=1 rx=0x7f40e4008790 tx=0x7f40e400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:19.566 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:19.553+0000 7f40f27fc700 1 -- 172.21.15.42:0/1158872173 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40e40088b0 con 0x7f40f40a7730 2019-12-05T06:13:19.566 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:19.553+0000 7f40f27fc700 1 -- 172.21.15.42:0/1158872173 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f40e4011b10 con 0x7f40f40a7730 2019-12-05T06:13:19.566 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:19.553+0000 7f40f95fd700 1 -- 172.21.15.42:0/1158872173 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f40f40a7730 msgr2=0x7f40f408df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:13:19.567 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:19.553+0000 7f40f95fd700 1 --2- 172.21.15.42:0/1158872173 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f40f40a7730 0x7f40f408df70 secure :-1 s=READY pgs=2286 cs=0 l=1 rx=0x7f40e4008790 tx=0x7f40e400fd30).stop 2019-12-05T06:13:19.567 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:19.553+0000 7f40f95fd700 1 -- 172.21.15.42:0/1158872173 shutdown_connections 2019-12-05T06:13:19.567 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:19.553+0000 7f40f95fd700 1 --2- 172.21.15.42:0/1158872173 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f40f409fad0 0x7f40f407eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:13:19.567 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:19.553+0000 7f40f95fd700 1 --2- 172.21.15.42:0/1158872173 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f40f40a7730 0x7f40f408df70 unknown :-1 s=CLOSED pgs=2286 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:13:19.567 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:19.553+0000 7f40f95fd700 1 -- 172.21.15.42:0/1158872173 shutdown_connections 2019-12-05T06:13:19.567 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:19.553+0000 7f40f95fd700 1 -- 172.21.15.42:0/1158872173 wait complete. 2019-12-05T06:13:19.567 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:19.553+0000 7f40f95fd700 1 -- 172.21.15.42:0/1158872173 >> 172.21.15.42:0/1158872173 conn(0x7f40f40aabd0 msgr2=0x7f40f40a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:13:19.571 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:19.557+0000 7f40f95fd700 1 Processor -- start 2019-12-05T06:13:19.571 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:19.557+0000 7f40f95fd700 1 -- start start 2019-12-05T06:13:19.571 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:19.557+0000 7f40f95fd700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f40f40a4ad0 0x7f40f4144ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:13:19.572 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:19.557+0000 7f40f95fd700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f40f407c780 0x7f40f407bb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:13:19.572 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:19.557+0000 7f40f95fd700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40f407c090 con 0x7f40f411cc00 2019-12-05T06:13:19.572 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:19.557+0000 7f40f95fd700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40f40a55c0 con 0x7f40f40a4ad0 2019-12-05T06:13:19.573 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:19.557+0000 7f40f95fd700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40f40a5860 con 0x7f40f407c780 2019-12-05T06:13:19.573 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:19.557+0000 7f40f2ffd700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f40f407c780 0x7f40f407bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:13:19.573 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:19.557+0000 7f40f2ffd700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f40f407c780 0x7f40f407bb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:50902/0 (socket says 172.21.15.42:50902) 2019-12-05T06:13:19.573 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:19.557+0000 7f40f2ffd700 1 -- 172.21.15.42:0/2852810599 learned_addr learned my addr 172.21.15.42:0/2852810599 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:13:19.573 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:19.557+0000 7f40f37fe700 1 --2- 172.21.15.42:0/2852810599 >> v2:172.21.15.40:3300/0 conn(0x7f40f40a4ad0 0x7f40f4144ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:13:19.573 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:19.557+0000 7f40f2ffd700 1 -- 172.21.15.42:0/2852810599 >> v2:172.21.15.40:3300/0 conn(0x7f40f40a4ad0 msgr2=0x7f40f4144ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:13:19.574 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:19.557+0000 7f40f2ffd700 1 --2- 172.21.15.42:0/2852810599 >> v2:172.21.15.40:3300/0 conn(0x7f40f40a4ad0 0x7f40f4144ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:13:19.574 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:19.557+0000 7f40f2ffd700 1 -- 172.21.15.42:0/2852810599 >> v1:172.21.15.40:6789/0 conn(0x7f40f411cc00 legacy=0x7f40f411b740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:13:19.574 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:19.557+0000 7f40f2ffd700 1 -- 172.21.15.42:0/2852810599 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40f40a5b00 con 0x7f40f407c780 2019-12-05T06:13:19.574 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:19.557+0000 7f40f2ffd700 1 --2- 172.21.15.42:0/2852810599 >> v2:172.21.15.42:3300/0 conn(0x7f40f407c780 0x7f40f407bb40 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f40e8000e90 tx=0x7f40e80109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:19.574 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:19.557+0000 7f40f2ffd700 1 --2- 172.21.15.42:0/2852810599 >> v2:172.21.15.42:3300/0 conn(0x7f40f407c780 0x7f40f407bb40 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f40e8000e90 tx=0x7f40e80109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:19.574 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:19.557+0000 7f40f2ffd700 1 --2- 172.21.15.42:0/2852810599 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f40f407c780 0x7f40f407bb40 secure :-1 s=READY pgs=2287 cs=0 l=1 rx=0x7f40e8000e90 tx=0x7f40e80109a0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:13:19.574 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:19.557+0000 7f40f2ffd700 1 --2- 172.21.15.42:0/2852810599 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f40f407c780 0x7f40f407bb40 secure :-1 s=THROTTLE_DONE pgs=2287 cs=0 l=1 rx=0x7f40e8000e90 tx=0x7f40e80109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:19.575 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:19.557+0000 7f40f2ffd700 1 --2- 172.21.15.42:0/2852810599 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f40f407c780 0x7f40f407bb40 secure :-1 s=THROTTLE_DONE pgs=2287 cs=0 l=1 rx=0x7f40e8000e90 tx=0x7f40e80109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:19.576 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:19.557+0000 7f40dbfff700 1 -- 172.21.15.42:0/2852810599 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f40e800caa0 con 0x7f40f407c780 2019-12-05T06:13:19.576 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:19.557+0000 7f40f2ffd700 1 --2- 172.21.15.42:0/2852810599 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f40f407c780 0x7f40f407bb40 secure :-1 s=THROTTLE_DONE pgs=2287 cs=0 l=1 rx=0x7f40e8000e90 tx=0x7f40e80109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:19.576 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:19.561+0000 7f40dbfff700 1 -- 172.21.15.42:0/2852810599 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40e800c6c0 con 0x7f40f407c780 2019-12-05T06:13:19.577 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:19.561+0000 7f40f95fd700 1 -- 172.21.15.42:0/2852810599 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40f411d030 con 0x7f40f407c780 2019-12-05T06:13:19.577 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:19.561+0000 7f40dbfff700 1 -- 172.21.15.42:0/2852810599 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f40e8012210 con 0x7f40f407c780 2019-12-05T06:13:19.577 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:19.561+0000 7f40f95fd700 1 -- 172.21.15.42:0/2852810599 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40f411bf40 con 0x7f40f407c780 2019-12-05T06:13:19.578 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:19.561+0000 7f40f2ffd700 1 --2- 172.21.15.42:0/2852810599 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f40f407c780 0x7f40f407bb40 secure :-1 s=THROTTLE_DONE pgs=2287 cs=0 l=1 rx=0x7f40e8000e90 tx=0x7f40e80109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:19.578 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:19.561+0000 7f40dbfff700 1 -- 172.21.15.42:0/2852810599 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f40e80124e0 con 0x7f40f407c780 2019-12-05T06:13:19.578 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:19.561+0000 7f40f2ffd700 1 --2- 172.21.15.42:0/2852810599 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f40f407c780 0x7f40f407bb40 secure :-1 s=THROTTLE_DONE pgs=2287 cs=0 l=1 rx=0x7f40e8000e90 tx=0x7f40e80109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:19.578 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:19.561+0000 7f40dbfff700 1 --2- 172.21.15.42:0/2852810599 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f40d403ad50 0x7f40d403d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:13:19.578 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:19.561+0000 7f40dbfff700 1 -- 172.21.15.42:0/2852810599 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 10245+0+0 (secure 0 0 0) 0x7f40e804b2f0 con 0x7f40f407c780 2019-12-05T06:13:19.579 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:19.561+0000 7f40f37fe700 1 --2- 172.21.15.42:0/2852810599 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f40d403ad50 0x7f40d403d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:13:19.579 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:19.561+0000 7f40f37fe700 1 --2- 172.21.15.42:0/2852810599 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f40d403ad50 0x7f40d403d260 crc :-1 s=READY pgs=962 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:13:19.579 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:19.565+0000 7f40f95fd700 1 -- 172.21.15.42:0/2852810599 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40f40a53b0 con 0x7f40f407c780 2019-12-05T06:13:19.580 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:19.573+0000 7f40f2ffd700 1 --2- 172.21.15.42:0/2852810599 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f40f407c780 0x7f40f407bb40 secure :-1 s=THROTTLE_DONE pgs=2287 cs=0 l=1 rx=0x7f40e8000e90 tx=0x7f40e80109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:19.580 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:19.573+0000 7f40dbfff700 1 -- 172.21.15.42:0/2852810599 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f40e8032370 con 0x7f40f407c780 2019-12-05T06:13:19.756 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:19.753+0000 7f40f95fd700 1 -- 172.21.15.42:0/2852810599 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f40f405d100 con 0x7f40f407c780 2019-12-05T06:13:19.757 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:19.753+0000 7f40f2ffd700 1 --2- 172.21.15.42:0/2852810599 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f40f407c780 0x7f40f407bb40 secure :-1 s=THROTTLE_DONE pgs=2287 cs=0 l=1 rx=0x7f40e8000e90 tx=0x7f40e80109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:19.757 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:19.753+0000 7f40dbfff700 1 -- 172.21.15.42:0/2852810599 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 276 v276) v1 ==== 95+0+1778 (secure 0 0 0) 0x7f40e80325d0 con 0x7f40f407c780 2019-12-05T06:13:19.759 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 276 2019-12-05T06:13:19.759 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:13:19.759 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":276,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":276,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:13:18.970651+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":98,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10246},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10246":{"gid":10246,"name":"a","rank":0,"incarnation":274,"state":"up:rejoin","state_seq":16,"addr":"172.21.15.42:6835/4032647860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":4032647860},{"type":"v1","addr":"172.21.15.42:6835","nonce":4032647860}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:13:19.762 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:19.757+0000 7f40f95fd700 1 -- 172.21.15.42:0/2852810599 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f40d403ad50 msgr2=0x7f40d403d260 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:13:19.762 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:19.757+0000 7f40f95fd700 1 --2- 172.21.15.42:0/2852810599 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f40d403ad50 0x7f40d403d260 crc :-1 s=READY pgs=962 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:13:19.763 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:19.757+0000 7f40f95fd700 1 -- 172.21.15.42:0/2852810599 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f40f407c780 msgr2=0x7f40f407bb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:13:19.763 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:19.757+0000 7f40f95fd700 1 --2- 172.21.15.42:0/2852810599 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f40f407c780 0x7f40f407bb40 secure :-1 s=READY pgs=2287 cs=0 l=1 rx=0x7f40e8000e90 tx=0x7f40e80109a0).stop 2019-12-05T06:13:19.763 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:19.757+0000 7f40f95fd700 1 -- 172.21.15.42:0/2852810599 shutdown_connections 2019-12-05T06:13:19.763 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:19.757+0000 7f40f95fd700 1 --2- 172.21.15.42:0/2852810599 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f40d403ad50 0x7f40d403d260 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:13:19.763 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:19.757+0000 7f40f95fd700 1 --2- 172.21.15.42:0/2852810599 >> v2:172.21.15.40:3300/0 conn(0x7f40f40a4ad0 0x7f40f4144ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:13:19.763 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:19.757+0000 7f40f95fd700 1 --2- 172.21.15.42:0/2852810599 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f40f407c780 0x7f40f407bb40 unknown :-1 s=CLOSED pgs=2287 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:13:19.763 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:19.757+0000 7f40f95fd700 1 -- 172.21.15.42:0/2852810599 shutdown_connections 2019-12-05T06:13:19.764 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:19.757+0000 7f40f95fd700 1 -- 172.21.15.42:0/2852810599 wait complete. 2019-12-05T06:13:19.764 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:19.757+0000 7f40f95fd700 1 -- 172.21.15.42:0/2852810599 >> 172.21.15.42:0/2852810599 conn(0x7f40f40a80a0 msgr2=0x7f40f411d8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:13:19.771 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2019-12-05T06:13:21.773 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:13:21.865 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:21.857+0000 7f1af2018700 1 Processor -- start 2019-12-05T06:13:21.866 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:21.857+0000 7f1af2018700 1 -- start start 2019-12-05T06:13:21.866 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:21.857+0000 7f1af2018700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f1aec09fae0 0x7f1aec14fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:13:21.866 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:21.857+0000 7f1af2018700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1aec0a0710 con 0x7f1aec08df80 2019-12-05T06:13:21.866 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:21.857+0000 7f1af2018700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1aec07ef20 con 0x7f1aec0a7740 2019-12-05T06:13:21.866 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:21.857+0000 7f1af2018700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1aec0a0b00 con 0x7f1aec09fae0 2019-12-05T06:13:21.867 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:21.857+0000 7f1aebfff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f1aec0a7740 0x7f1aec07e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:50948/0 (socket says 172.21.15.42:50948) 2019-12-05T06:13:21.867 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:21.857+0000 7f1aebfff700 1 -- 172.21.15.42:0/4181910972 learned_addr learned my addr 172.21.15.42:0/4181910972 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:13:21.867 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:21.857+0000 7f1af0db6700 1 --2- 172.21.15.42:0/4181910972 >> v2:172.21.15.42:3300/0 conn(0x7f1aec09fae0 0x7f1aec14fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:13:21.867 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:21.857+0000 7f1af0db6700 1 -- 172.21.15.42:0/4181910972 >> v1:172.21.15.42:6789/0 conn(0x7f1aec0a7740 legacy=0x7f1aec07e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:13:21.867 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:21.857+0000 7f1af0db6700 1 -- 172.21.15.42:0/4181910972 >> v1:172.21.15.40:6789/0 conn(0x7f1aec08df80 legacy=0x7f1aec09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:13:21.867 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:21.857+0000 7f1aeaffd700 1 -- 172.21.15.42:0/4181910972 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2891548881 0 0) 0x7f1aec07ef20 con 0x7f1aec0a7740 2019-12-05T06:13:21.867 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:21.857+0000 7f1af0db6700 1 -- 172.21.15.42:0/4181910972 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1aec0a9ba0 con 0x7f1aec09fae0 2019-12-05T06:13:21.868 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:21.857+0000 7f1af0db6700 1 --2- 172.21.15.42:0/4181910972 >> v2:172.21.15.42:3300/0 conn(0x7f1aec09fae0 0x7f1aec14fff0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f1ae4008790 tx=0x7f1ae400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:21.868 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:21.857+0000 7f1af0db6700 1 --2- 172.21.15.42:0/4181910972 >> v2:172.21.15.42:3300/0 conn(0x7f1aec09fae0 0x7f1aec14fff0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f1ae4008790 tx=0x7f1ae400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:21.868 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:21.857+0000 7f1af0db6700 1 --2- 172.21.15.42:0/4181910972 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1aec09fae0 0x7f1aec14fff0 secure :-1 s=READY pgs=2289 cs=0 l=1 rx=0x7f1ae4008790 tx=0x7f1ae400fbf0).ready entity=mon.1 client_cookie=e883f2ecbb76fe80 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:13:21.869 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:21.861+0000 7f1af0db6700 1 --2- 172.21.15.42:0/4181910972 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1aec09fae0 0x7f1aec14fff0 secure :-1 s=THROTTLE_DONE pgs=2289 cs=0 l=1 rx=0x7f1ae4008790 tx=0x7f1ae400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:21.869 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:21.861+0000 7f1aeaffd700 1 -- 172.21.15.42:0/4181910972 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1ae4008bb0 con 0x7f1aec09fae0 2019-12-05T06:13:21.869 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:21.861+0000 7f1af0db6700 1 --2- 172.21.15.42:0/4181910972 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1aec09fae0 0x7f1aec14fff0 secure :-1 s=THROTTLE_DONE pgs=2289 cs=0 l=1 rx=0x7f1ae4008790 tx=0x7f1ae400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:21.869 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:21.861+0000 7f1af0db6700 1 --2- 172.21.15.42:0/4181910972 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1aec09fae0 0x7f1aec14fff0 secure :-1 s=THROTTLE_DONE pgs=2289 cs=0 l=1 rx=0x7f1ae4008790 tx=0x7f1ae400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:21.869 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:21.861+0000 7f1aeaffd700 1 -- 172.21.15.42:0/4181910972 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ae40087d0 con 0x7f1aec09fae0 2019-12-05T06:13:21.869 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:21.861+0000 7f1aeaffd700 1 -- 172.21.15.42:0/4181910972 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1ae40118d0 con 0x7f1aec09fae0 2019-12-05T06:13:21.870 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:21.861+0000 7f1af2018700 1 -- 172.21.15.42:0/4181910972 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1aec09fae0 msgr2=0x7f1aec14fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:13:21.870 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:21.861+0000 7f1af2018700 1 --2- 172.21.15.42:0/4181910972 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1aec09fae0 0x7f1aec14fff0 secure :-1 s=READY pgs=2289 cs=0 l=1 rx=0x7f1ae4008790 tx=0x7f1ae400fbf0).stop 2019-12-05T06:13:21.870 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:21.861+0000 7f1af2018700 1 -- 172.21.15.42:0/4181910972 shutdown_connections 2019-12-05T06:13:21.870 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:21.861+0000 7f1af2018700 1 --2- 172.21.15.42:0/4181910972 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1aec09fae0 0x7f1aec14fff0 unknown :-1 s=CLOSED pgs=2289 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:13:21.870 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:21.861+0000 7f1af2018700 1 -- 172.21.15.42:0/4181910972 shutdown_connections 2019-12-05T06:13:21.871 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:21.861+0000 7f1af2018700 1 -- 172.21.15.42:0/4181910972 wait complete. 2019-12-05T06:13:21.871 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:21.861+0000 7f1af2018700 1 -- 172.21.15.42:0/4181910972 >> 172.21.15.42:0/4181910972 conn(0x7f1aec0aabe0 msgr2=0x7f1aec0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:13:21.881 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:21.861+0000 7f1af2018700 1 Processor -- start 2019-12-05T06:13:21.882 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:21.861+0000 7f1af2018700 1 -- start start 2019-12-05T06:13:21.882 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:21.865+0000 7f1af2018700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f1aec07d130 0x7f1aec07d560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:13:21.882 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:21.865+0000 7f1af2018700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f1aec117b90 0x7f1aec117fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:13:21.883 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:21.865+0000 7f1af2018700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1aec123ac0 con 0x7f1aec118cd0 2019-12-05T06:13:21.883 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:21.865+0000 7f1af2018700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1aec124040 con 0x7f1aec117b90 2019-12-05T06:13:21.883 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:21.865+0000 7f1af2018700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1aec123da0 con 0x7f1aec07d130 2019-12-05T06:13:21.883 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:21.865+0000 7f1aeb7fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f1aec07d130 0x7f1aec07d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:13:21.884 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:21.865+0000 7f1aeb7fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f1aec07d130 0x7f1aec07d560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:50918/0 (socket says 172.21.15.42:50918) 2019-12-05T06:13:21.884 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:21.865+0000 7f1aeb7fe700 1 -- 172.21.15.42:0/2265433647 learned_addr learned my addr 172.21.15.42:0/2265433647 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:13:21.884 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:21.865+0000 7f1af0db6700 1 --2- 172.21.15.42:0/2265433647 >> v2:172.21.15.40:3300/0 conn(0x7f1aec117b90 0x7f1aec117fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:13:21.884 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:21.865+0000 7f1aeb7fe700 1 -- 172.21.15.42:0/2265433647 >> v2:172.21.15.40:3300/0 conn(0x7f1aec117b90 msgr2=0x7f1aec117fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:13:21.885 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:21.865+0000 7f1aeb7fe700 1 --2- 172.21.15.42:0/2265433647 >> v2:172.21.15.40:3300/0 conn(0x7f1aec117b90 0x7f1aec117fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:13:21.885 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:21.865+0000 7f1aeb7fe700 1 -- 172.21.15.42:0/2265433647 >> v1:172.21.15.40:6789/0 conn(0x7f1aec118cd0 legacy=0x7f1aec1232c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:13:21.885 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:21.865+0000 7f1aeb7fe700 1 -- 172.21.15.42:0/2265433647 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1aec116c30 con 0x7f1aec07d130 2019-12-05T06:13:21.885 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:21.865+0000 7f1aeb7fe700 1 --2- 172.21.15.42:0/2265433647 >> v2:172.21.15.42:3300/0 conn(0x7f1aec07d130 0x7f1aec07d560 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f1ad4000b80 tx=0x7f1ad400c320).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:21.885 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:21.865+0000 7f1aeb7fe700 1 --2- 172.21.15.42:0/2265433647 >> v2:172.21.15.42:3300/0 conn(0x7f1aec07d130 0x7f1aec07d560 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f1ad4000b80 tx=0x7f1ad400c320).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:21.885 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:21.865+0000 7f1aeb7fe700 1 --2- 172.21.15.42:0/2265433647 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1aec07d130 0x7f1aec07d560 secure :-1 s=READY pgs=2290 cs=0 l=1 rx=0x7f1ad4000b80 tx=0x7f1ad400c320).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:13:21.885 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:21.865+0000 7f1aeb7fe700 1 --2- 172.21.15.42:0/2265433647 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1aec07d130 0x7f1aec07d560 secure :-1 s=THROTTLE_DONE pgs=2290 cs=0 l=1 rx=0x7f1ad4000b80 tx=0x7f1ad400c320).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:21.885 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:21.865+0000 7f1aeb7fe700 1 --2- 172.21.15.42:0/2265433647 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1aec07d130 0x7f1aec07d560 secure :-1 s=THROTTLE_DONE pgs=2290 cs=0 l=1 rx=0x7f1ad4000b80 tx=0x7f1ad400c320).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:21.886 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:21.865+0000 7f1ae8ff9700 1 -- 172.21.15.42:0/2265433647 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1ad400ca80 con 0x7f1aec07d130 2019-12-05T06:13:21.886 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:21.865+0000 7f1aeb7fe700 1 --2- 172.21.15.42:0/2265433647 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1aec07d130 0x7f1aec07d560 secure :-1 s=THROTTLE_DONE pgs=2290 cs=0 l=1 rx=0x7f1ad4000b80 tx=0x7f1ad400c320).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:21.886 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:21.865+0000 7f1ae8ff9700 1 -- 172.21.15.42:0/2265433647 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ad400c6a0 con 0x7f1aec07d130 2019-12-05T06:13:21.886 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:21.865+0000 7f1af2018700 1 -- 172.21.15.42:0/2265433647 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1aec1166d0 con 0x7f1aec07d130 2019-12-05T06:13:21.886 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:21.865+0000 7f1ae8ff9700 1 -- 172.21.15.42:0/2265433647 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1ad4011120 con 0x7f1aec07d130 2019-12-05T06:13:21.886 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:21.865+0000 7f1af2018700 1 -- 172.21.15.42:0/2265433647 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1aec1186b0 con 0x7f1aec07d130 2019-12-05T06:13:21.886 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:21.865+0000 7f1aeb7fe700 1 --2- 172.21.15.42:0/2265433647 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1aec07d130 0x7f1aec07d560 secure :-1 s=THROTTLE_DONE pgs=2290 cs=0 l=1 rx=0x7f1ad4000b80 tx=0x7f1ad400c320).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:21.887 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:21.865+0000 7f1ae8ff9700 1 -- 172.21.15.42:0/2265433647 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f1ad4011430 con 0x7f1aec07d130 2019-12-05T06:13:21.887 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:21.865+0000 7f1aeb7fe700 1 --2- 172.21.15.42:0/2265433647 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1aec07d130 0x7f1aec07d560 secure :-1 s=THROTTLE_DONE pgs=2290 cs=0 l=1 rx=0x7f1ad4000b80 tx=0x7f1ad400c320).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:21.887 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:21.865+0000 7f1ae8ff9700 1 --2- 172.21.15.42:0/2265433647 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f1ad003aca0 0x7f1ad003d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:13:21.887 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:21.865+0000 7f1ae8ff9700 1 -- 172.21.15.42:0/2265433647 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 10245+0+0 (secure 0 0 0) 0x7f1ad404b2f0 con 0x7f1aec07d130 2019-12-05T06:13:21.887 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:21.869+0000 7f1aebfff700 1 --2- 172.21.15.42:0/2265433647 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f1ad003aca0 0x7f1ad003d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:13:21.887 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:21.869+0000 7f1aebfff700 1 --2- 172.21.15.42:0/2265433647 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f1ad003aca0 0x7f1ad003d1b0 crc :-1 s=READY pgs=963 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:13:21.887 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:21.873+0000 7f1af2018700 1 -- 172.21.15.42:0/2265433647 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1aec05d110 con 0x7f1aec07d130 2019-12-05T06:13:21.888 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:21.877+0000 7f1aeb7fe700 1 --2- 172.21.15.42:0/2265433647 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1aec07d130 0x7f1aec07d560 secure :-1 s=THROTTLE_DONE pgs=2290 cs=0 l=1 rx=0x7f1ad4000b80 tx=0x7f1ad400c320).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:21.888 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:21.877+0000 7f1ae8ff9700 1 -- 172.21.15.42:0/2265433647 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f1ad4032310 con 0x7f1aec07d130 2019-12-05T06:13:22.046 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:22.041+0000 7f1af2018700 1 -- 172.21.15.42:0/2265433647 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1aec05b8c0 con 0x7f1aec07d130 2019-12-05T06:13:22.047 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:22.041+0000 7f1aeb7fe700 1 --2- 172.21.15.42:0/2265433647 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1aec07d130 0x7f1aec07d560 secure :-1 s=THROTTLE_DONE pgs=2290 cs=0 l=1 rx=0x7f1ad4000b80 tx=0x7f1ad400c320).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:22.047 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:22.041+0000 7f1ae8ff9700 1 -- 172.21.15.42:0/2265433647 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 277 v277) v1 ==== 95+0+1778 (secure 0 0 0) 0x7f1ad4032570 con 0x7f1aec07d130 2019-12-05T06:13:22.048 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 277 2019-12-05T06:13:22.048 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:13:22.048 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":277,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":277,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:13:19.980391+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":98,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10246},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10246":{"gid":10246,"name":"a","rank":0,"incarnation":274,"state":"up:active","state_seq":17,"addr":"172.21.15.42:6835/4032647860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":4032647860},{"type":"v1","addr":"172.21.15.42:6835","nonce":4032647860}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:13:22.050 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:22.045+0000 7f1af2018700 1 -- 172.21.15.42:0/2265433647 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f1ad003aca0 msgr2=0x7f1ad003d1b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:13:22.050 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:22.045+0000 7f1af2018700 1 --2- 172.21.15.42:0/2265433647 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f1ad003aca0 0x7f1ad003d1b0 crc :-1 s=READY pgs=963 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:13:22.050 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:22.045+0000 7f1af2018700 1 -- 172.21.15.42:0/2265433647 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1aec07d130 msgr2=0x7f1aec07d560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:13:22.051 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:22.045+0000 7f1af2018700 1 --2- 172.21.15.42:0/2265433647 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1aec07d130 0x7f1aec07d560 secure :-1 s=READY pgs=2290 cs=0 l=1 rx=0x7f1ad4000b80 tx=0x7f1ad400c320).stop 2019-12-05T06:13:22.051 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:22.045+0000 7f1af2018700 1 -- 172.21.15.42:0/2265433647 shutdown_connections 2019-12-05T06:13:22.051 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:22.045+0000 7f1af2018700 1 --2- 172.21.15.42:0/2265433647 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f1ad003aca0 0x7f1ad003d1b0 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:13:22.051 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:22.045+0000 7f1af2018700 1 --2- 172.21.15.42:0/2265433647 >> v2:172.21.15.40:3300/0 conn(0x7f1aec117b90 0x7f1aec117fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:13:22.051 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:22.045+0000 7f1af2018700 1 --2- 172.21.15.42:0/2265433647 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1aec07d130 0x7f1aec07d560 unknown :-1 s=CLOSED pgs=2290 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:13:22.051 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:22.045+0000 7f1af2018700 1 -- 172.21.15.42:0/2265433647 shutdown_connections 2019-12-05T06:13:22.051 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:22.045+0000 7f1af2018700 1 -- 172.21.15.42:0/2265433647 wait complete. 2019-12-05T06:13:22.052 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:22.045+0000 7f1af2018700 1 -- 172.21.15.42:0/2265433647 >> 172.21.15.42:0/2265433647 conn(0x7f1aec0a8420 msgr2=0x7f1aec11c810 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:13:22.061 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a has gained rank=0, replacing gid=7094 2019-12-05T06:13:22.062 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 5 secs before reviving mds.b 2019-12-05T06:13:27.063 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.b 2019-12-05T06:13:27.063 INFO:tasks.ceph.mds.b:Restarting daemon 2019-12-05T06:13:27.063 INFO:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2019-12-05T06:13:27.067 INFO:tasks.ceph.mds.b:Started 2019-12-05T06:13:27.068 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:13:27.129 INFO:tasks.ceph.mds.b.smithi040.stdout:starting mds.b at 2019-12-05T06:13:27.173 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:27.161+0000 7f6b8c2d7700 1 Processor -- start 2019-12-05T06:13:27.173 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:27.161+0000 7f6b8c2d7700 1 -- start start 2019-12-05T06:13:27.173 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:27.161+0000 7f6b8c2d7700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6b840a7730 0x7f6b8408df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:13:27.173 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:27.161+0000 7f6b8c2d7700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b840a0700 con 0x7f6b8407e710 2019-12-05T06:13:27.174 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:27.161+0000 7f6b8c2d7700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b840a9b90 con 0x7f6b8409fad0 2019-12-05T06:13:27.174 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:27.161+0000 7f6b8c2d7700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b840a0af0 con 0x7f6b840a7730 2019-12-05T06:13:27.174 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:27.165+0000 7f6b8b075700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f6b8409fad0 0x7f6b8414efd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:50994/0 (socket says 172.21.15.42:50994) 2019-12-05T06:13:27.174 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:27.165+0000 7f6b8b075700 1 -- 172.21.15.42:0/478641490 learned_addr learned my addr 172.21.15.42:0/478641490 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:13:27.174 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:27.165+0000 7f6b8a874700 1 --2- 172.21.15.42:0/478641490 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6b840a7730 0x7f6b8408df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:13:27.174 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:27.165+0000 7f6b89872700 1 -- 172.21.15.42:0/478641490 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2664328874 0 0) 0x7f6b840a9b90 con 0x7f6b8409fad0 2019-12-05T06:13:27.174 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:27.165+0000 7f6b89872700 1 -- 172.21.15.42:0/478641490 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b6c0030b0 con 0x7f6b8409fad0 2019-12-05T06:13:27.175 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:27.165+0000 7f6b89872700 1 -- 172.21.15.42:0/478641490 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4047842973 0 0) 0x7f6b840a0700 con 0x7f6b8407e710 2019-12-05T06:13:27.175 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:27.165+0000 7f6b89872700 1 -- 172.21.15.42:0/478641490 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b840a9b90 con 0x7f6b8407e710 2019-12-05T06:13:27.175 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:27.165+0000 7f6b8a874700 1 -- 172.21.15.42:0/478641490 >> v1:172.21.15.42:6789/0 conn(0x7f6b8409fad0 legacy=0x7f6b8414efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:13:27.175 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:27.165+0000 7f6b8a874700 1 -- 172.21.15.42:0/478641490 >> v1:172.21.15.40:6789/0 conn(0x7f6b8407e710 legacy=0x7f6b8409f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:13:27.175 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:27.165+0000 7f6b8a874700 1 -- 172.21.15.42:0/478641490 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b8407eb40 con 0x7f6b840a7730 2019-12-05T06:13:27.176 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:27.165+0000 7f6b8a874700 1 --2- 172.21.15.42:0/478641490 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6b840a7730 0x7f6b8408df70 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f6b74002f30 tx=0x7f6b7400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:27.176 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:27.165+0000 7f6b8a874700 1 --2- 172.21.15.42:0/478641490 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6b840a7730 0x7f6b8408df70 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f6b74002f30 tx=0x7f6b7400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:27.176 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:27.165+0000 7f6b8a874700 1 --2- 172.21.15.42:0/478641490 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6b840a7730 0x7f6b8408df70 secure :-1 s=READY pgs=505 cs=0 l=1 rx=0x7f6b74002f30 tx=0x7f6b7400fbf0).ready entity=mon.2 client_cookie=9aba85682c3173f7 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:13:27.176 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:27.165+0000 7f6b8a874700 1 --2- 172.21.15.42:0/478641490 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6b840a7730 0x7f6b8408df70 secure :-1 s=THROTTLE_DONE pgs=505 cs=0 l=1 rx=0x7f6b74002f30 tx=0x7f6b7400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:27.176 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:27.165+0000 7f6b8a874700 1 --2- 172.21.15.42:0/478641490 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6b840a7730 0x7f6b8408df70 secure :-1 s=THROTTLE_DONE pgs=505 cs=0 l=1 rx=0x7f6b74002f30 tx=0x7f6b7400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:27.176 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:27.165+0000 7f6b89872700 1 -- 172.21.15.42:0/478641490 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6b740112b0 con 0x7f6b840a7730 2019-12-05T06:13:27.176 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:27.165+0000 7f6b8a874700 1 --2- 172.21.15.42:0/478641490 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6b840a7730 0x7f6b8408df70 secure :-1 s=THROTTLE_DONE pgs=505 cs=0 l=1 rx=0x7f6b74002f30 tx=0x7f6b7400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:27.184 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:27.165+0000 7f6b89872700 1 -- 172.21.15.42:0/478641490 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b74008bc0 con 0x7f6b840a7730 2019-12-05T06:13:27.185 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:27.165+0000 7f6b89872700 1 -- 172.21.15.42:0/478641490 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6b74011890 con 0x7f6b840a7730 2019-12-05T06:13:27.185 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:27.165+0000 7f6b8c2d7700 1 -- 172.21.15.42:0/478641490 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6b840a7730 msgr2=0x7f6b8408df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:13:27.185 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:27.165+0000 7f6b8c2d7700 1 --2- 172.21.15.42:0/478641490 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6b840a7730 0x7f6b8408df70 secure :-1 s=READY pgs=505 cs=0 l=1 rx=0x7f6b74002f30 tx=0x7f6b7400fbf0).stop 2019-12-05T06:13:27.185 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:27.165+0000 7f6b8c2d7700 1 -- 172.21.15.42:0/478641490 shutdown_connections 2019-12-05T06:13:27.185 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:27.165+0000 7f6b8c2d7700 1 --2- 172.21.15.42:0/478641490 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6b840a7730 0x7f6b8408df70 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:13:27.185 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:27.165+0000 7f6b8c2d7700 1 -- 172.21.15.42:0/478641490 shutdown_connections 2019-12-05T06:13:27.185 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:27.165+0000 7f6b8c2d7700 1 -- 172.21.15.42:0/478641490 wait complete. 2019-12-05T06:13:27.186 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:27.165+0000 7f6b8c2d7700 1 -- 172.21.15.42:0/478641490 >> 172.21.15.42:0/478641490 conn(0x7f6b840aabd0 msgr2=0x7f6b840a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:13:27.186 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:27.169+0000 7f6b8c2d7700 1 Processor -- start 2019-12-05T06:13:27.186 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:27.169+0000 7f6b8c2d7700 1 -- start start 2019-12-05T06:13:27.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:27.169+0000 7f6b8c2d7700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6b840a6b80 0x7f6b840a1a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:13:27.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:27.169+0000 7f6b8c2d7700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b841209d0 con 0x7f6b840a2eb0 2019-12-05T06:13:27.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:27.169+0000 7f6b8c2d7700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b840a39d0 con 0x7f6b840a1f90 2019-12-05T06:13:27.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:27.169+0000 7f6b8c2d7700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b8407c230 con 0x7f6b840a6b80 2019-12-05T06:13:27.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:27.169+0000 7f6b8a874700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6b840a6b80 0x7f6b840a1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:13:27.193 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:27.169+0000 7f6b8a874700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6b840a6b80 0x7f6b840a1a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:50962/0 (socket says 172.21.15.42:50962) 2019-12-05T06:13:27.193 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:27.169+0000 7f6b8a874700 1 -- 172.21.15.42:0/22488560 learned_addr learned my addr 172.21.15.42:0/22488560 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:13:27.193 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:27.169+0000 7f6b837fe700 1 -- 172.21.15.42:0/22488560 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1609200306 0 0) 0x7f6b841209d0 con 0x7f6b840a2eb0 2019-12-05T06:13:27.193 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:27.169+0000 7f6b837fe700 1 -- 172.21.15.42:0/22488560 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b640030b0 con 0x7f6b840a2eb0 2019-12-05T06:13:27.193 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:27.169+0000 7f6b8a874700 1 -- 172.21.15.42:0/22488560 >> v1:172.21.15.42:6789/0 conn(0x7f6b840a1f90 legacy=0x7f6b8407ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:13:27.193 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:27.169+0000 7f6b837fe700 1 -- 172.21.15.42:0/22488560 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3193460490 0 0) 0x7f6b840a39d0 con 0x7f6b840a1f90 2019-12-05T06:13:27.193 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:27.169+0000 7f6b8a874700 1 -- 172.21.15.42:0/22488560 >> v1:172.21.15.40:6789/0 conn(0x7f6b840a2eb0 legacy=0x7f6b841201d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:13:27.194 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:27.169+0000 7f6b8a874700 1 -- 172.21.15.42:0/22488560 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b8412dc10 con 0x7f6b840a6b80 2019-12-05T06:13:27.194 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:27.169+0000 7f6b8a874700 1 --2- 172.21.15.42:0/22488560 >> v2:172.21.15.42:3300/0 conn(0x7f6b840a6b80 0x7f6b840a1a40 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f6b74002f30 tx=0x7f6b740086a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:27.194 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:27.169+0000 7f6b8a874700 1 --2- 172.21.15.42:0/22488560 >> v2:172.21.15.42:3300/0 conn(0x7f6b840a6b80 0x7f6b840a1a40 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f6b74002f30 tx=0x7f6b740086a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:27.195 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:27.169+0000 7f6b8a874700 1 --2- 172.21.15.42:0/22488560 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6b840a6b80 0x7f6b840a1a40 secure :-1 s=READY pgs=2293 cs=0 l=1 rx=0x7f6b74002f30 tx=0x7f6b740086a0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:13:27.196 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:27.173+0000 7f6b8a874700 1 --2- 172.21.15.42:0/22488560 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6b840a6b80 0x7f6b840a1a40 secure :-1 s=THROTTLE_DONE pgs=2293 cs=0 l=1 rx=0x7f6b74002f30 tx=0x7f6b740086a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:27.196 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:27.173+0000 7f6b837fe700 1 -- 172.21.15.42:0/22488560 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6b74007340 con 0x7f6b840a6b80 2019-12-05T06:13:27.196 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:27.173+0000 7f6b8a874700 1 --2- 172.21.15.42:0/22488560 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6b840a6b80 0x7f6b840a1a40 secure :-1 s=THROTTLE_DONE pgs=2293 cs=0 l=1 rx=0x7f6b74002f30 tx=0x7f6b740086a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:27.197 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:27.173+0000 7f6b8a874700 1 --2- 172.21.15.42:0/22488560 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6b840a6b80 0x7f6b840a1a40 secure :-1 s=THROTTLE_DONE pgs=2293 cs=0 l=1 rx=0x7f6b74002f30 tx=0x7f6b740086a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:27.197 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:27.173+0000 7f6b837fe700 1 -- 172.21.15.42:0/22488560 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b740114b0 con 0x7f6b840a6b80 2019-12-05T06:13:27.197 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:27.173+0000 7f6b8c2d7700 1 -- 172.21.15.42:0/22488560 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b8412bbf0 con 0x7f6b840a6b80 2019-12-05T06:13:27.197 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:27.173+0000 7f6b837fe700 1 -- 172.21.15.42:0/22488560 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6b7400f360 con 0x7f6b840a6b80 2019-12-05T06:13:27.197 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:27.173+0000 7f6b8c2d7700 1 -- 172.21.15.42:0/22488560 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b8412c170 con 0x7f6b840a6b80 2019-12-05T06:13:27.197 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:27.173+0000 7f6b8a874700 1 --2- 172.21.15.42:0/22488560 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6b840a6b80 0x7f6b840a1a40 secure :-1 s=THROTTLE_DONE pgs=2293 cs=0 l=1 rx=0x7f6b74002f30 tx=0x7f6b740086a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:27.197 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:27.173+0000 7f6b837fe700 1 -- 172.21.15.42:0/22488560 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f6b74005540 con 0x7f6b840a6b80 2019-12-05T06:13:27.198 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:27.173+0000 7f6b8a874700 1 --2- 172.21.15.42:0/22488560 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6b840a6b80 0x7f6b840a1a40 secure :-1 s=THROTTLE_DONE pgs=2293 cs=0 l=1 rx=0x7f6b74002f30 tx=0x7f6b740086a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:27.198 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:27.173+0000 7f6b837fe700 1 --2- 172.21.15.42:0/22488560 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6b6403b1e0 0x7f6b6403d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:13:27.198 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:27.173+0000 7f6b837fe700 1 -- 172.21.15.42:0/22488560 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 10245+0+0 (secure 0 0 0) 0x7f6b7404b4e0 con 0x7f6b840a6b80 2019-12-05T06:13:27.198 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:27.173+0000 7f6b8a073700 1 --2- 172.21.15.42:0/22488560 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6b6403b1e0 0x7f6b6403d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:13:27.198 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:27.173+0000 7f6b8a073700 1 --2- 172.21.15.42:0/22488560 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6b6403b1e0 0x7f6b6403d6f0 crc :-1 s=READY pgs=965 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:13:27.198 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:27.177+0000 7f6b8c2d7700 1 -- 172.21.15.42:0/22488560 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b8405d100 con 0x7f6b840a6b80 2019-12-05T06:13:27.198 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:27.185+0000 7f6b8a874700 1 --2- 172.21.15.42:0/22488560 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6b840a6b80 0x7f6b840a1a40 secure :-1 s=THROTTLE_DONE pgs=2293 cs=0 l=1 rx=0x7f6b74002f30 tx=0x7f6b740086a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:27.199 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:27.185+0000 7f6b837fe700 1 -- 172.21.15.42:0/22488560 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f6b7400f600 con 0x7f6b840a6b80 2019-12-05T06:13:27.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:27.349+0000 7f6b8c2d7700 1 -- 172.21.15.42:0/22488560 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6b8405b8b0 con 0x7f6b840a6b80 2019-12-05T06:13:27.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:27.349+0000 7f6b8a874700 1 --2- 172.21.15.42:0/22488560 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6b840a6b80 0x7f6b840a1a40 secure :-1 s=THROTTLE_DONE pgs=2293 cs=0 l=1 rx=0x7f6b74002f30 tx=0x7f6b740086a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:27.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:27.349+0000 7f6b837fe700 1 -- 172.21.15.42:0/22488560 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 277 v277) v1 ==== 95+0+1778 (secure 0 0 0) 0x7f6b74033390 con 0x7f6b840a6b80 2019-12-05T06:13:27.355 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 277 2019-12-05T06:13:27.355 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:13:27.355 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":277,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":277,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:13:19.980391+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":98,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10246},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10246":{"gid":10246,"name":"a","rank":0,"incarnation":274,"state":"up:active","state_seq":17,"addr":"172.21.15.42:6835/4032647860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":4032647860},{"type":"v1","addr":"172.21.15.42:6835","nonce":4032647860}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:13:27.358 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:27.353+0000 7f6b8c2d7700 1 -- 172.21.15.42:0/22488560 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6b6403b1e0 msgr2=0x7f6b6403d6f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:13:27.358 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:27.353+0000 7f6b8c2d7700 1 --2- 172.21.15.42:0/22488560 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6b6403b1e0 0x7f6b6403d6f0 crc :-1 s=READY pgs=965 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:13:27.358 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:27.353+0000 7f6b8c2d7700 1 -- 172.21.15.42:0/22488560 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6b840a6b80 msgr2=0x7f6b840a1a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:13:27.358 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:27.353+0000 7f6b8c2d7700 1 --2- 172.21.15.42:0/22488560 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6b840a6b80 0x7f6b840a1a40 secure :-1 s=READY pgs=2293 cs=0 l=1 rx=0x7f6b74002f30 tx=0x7f6b740086a0).stop 2019-12-05T06:13:27.359 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:27.353+0000 7f6b8c2d7700 1 -- 172.21.15.42:0/22488560 shutdown_connections 2019-12-05T06:13:27.359 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:27.353+0000 7f6b8c2d7700 1 --2- 172.21.15.42:0/22488560 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6b6403b1e0 0x7f6b6403d6f0 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:13:27.359 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:27.353+0000 7f6b8c2d7700 1 --2- 172.21.15.42:0/22488560 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6b840a6b80 0x7f6b840a1a40 unknown :-1 s=CLOSED pgs=2293 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:13:27.362 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:27.353+0000 7f6b8c2d7700 1 -- 172.21.15.42:0/22488560 shutdown_connections 2019-12-05T06:13:27.362 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:27.353+0000 7f6b8c2d7700 1 -- 172.21.15.42:0/22488560 wait complete. 2019-12-05T06:13:27.362 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:27.353+0000 7f6b8c2d7700 1 -- 172.21.15.42:0/22488560 >> 172.21.15.42:0/22488560 conn(0x7f6b840a7f80 msgr2=0x7f6b841415f0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:13:27.366 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2019-12-05T06:13:29.367 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:13:29.478 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:29.469+0000 7fa73c337700 1 Processor -- start 2019-12-05T06:13:29.478 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:29.469+0000 7fa73c337700 1 -- start start 2019-12-05T06:13:29.478 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:29.469+0000 7fa73c337700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fa7340a7730 0x7fa73408df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:13:29.478 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:29.469+0000 7fa73c337700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa73407e710 0x7fa73407eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:13:29.478 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:29.469+0000 7fa73c337700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7340a0700 con 0x7fa73409f2d0 2019-12-05T06:13:29.479 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:29.469+0000 7fa73c337700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7340a0af0 con 0x7fa73407e710 2019-12-05T06:13:29.479 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:29.469+0000 7fa73c337700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa73408e5a0 con 0x7fa7340a7730 2019-12-05T06:13:29.479 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:29.469+0000 7fa73a8d4700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fa7340a7730 0x7fa73408df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:13:29.479 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:29.469+0000 7fa73a8d4700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fa7340a7730 0x7fa73408df70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:50974/0 (socket says 172.21.15.42:50974) 2019-12-05T06:13:29.479 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:29.469+0000 7fa73a8d4700 1 -- 172.21.15.42:0/3990903428 learned_addr learned my addr 172.21.15.42:0/3990903428 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:13:29.479 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:29.469+0000 7fa73a0d3700 1 --2- 172.21.15.42:0/3990903428 >> v2:172.21.15.40:3300/0 conn(0x7fa73407e710 0x7fa73407eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:13:29.479 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:29.469+0000 7fa73a8d4700 1 -- 172.21.15.42:0/3990903428 >> v2:172.21.15.40:3300/0 conn(0x7fa73407e710 msgr2=0x7fa73407eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:13:29.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:29.469+0000 7fa73a8d4700 1 --2- 172.21.15.42:0/3990903428 >> v2:172.21.15.40:3300/0 conn(0x7fa73407e710 0x7fa73407eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:13:29.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:29.469+0000 7fa73a8d4700 1 -- 172.21.15.42:0/3990903428 >> v1:172.21.15.40:6789/0 conn(0x7fa73409f2d0 legacy=0x7fa73409f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:13:29.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:29.469+0000 7fa73a8d4700 1 -- 172.21.15.42:0/3990903428 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa73407f090 con 0x7fa7340a7730 2019-12-05T06:13:29.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:29.469+0000 7fa73a8d4700 1 --2- 172.21.15.42:0/3990903428 >> v2:172.21.15.42:3300/0 conn(0x7fa7340a7730 0x7fa73408df70 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fa730002f30 tx=0x7fa73000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:29.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:29.469+0000 7fa73a8d4700 1 --2- 172.21.15.42:0/3990903428 >> v2:172.21.15.42:3300/0 conn(0x7fa7340a7730 0x7fa73408df70 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fa730002f30 tx=0x7fa73000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:29.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:29.469+0000 7fa73a8d4700 1 --2- 172.21.15.42:0/3990903428 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa7340a7730 0x7fa73408df70 secure :-1 s=READY pgs=2294 cs=0 l=1 rx=0x7fa730002f30 tx=0x7fa73000fd30).ready entity=mon.1 client_cookie=68fdc84bb6dc3c43 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:13:29.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:29.473+0000 7fa73a8d4700 1 --2- 172.21.15.42:0/3990903428 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa7340a7730 0x7fa73408df70 secure :-1 s=THROTTLE_DONE pgs=2294 cs=0 l=1 rx=0x7fa730002f30 tx=0x7fa73000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:29.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:29.473+0000 7fa7398d2700 1 -- 172.21.15.42:0/3990903428 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa730011370 con 0x7fa7340a7730 2019-12-05T06:13:29.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:29.473+0000 7fa73a8d4700 1 --2- 172.21.15.42:0/3990903428 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa7340a7730 0x7fa73408df70 secure :-1 s=THROTTLE_DONE pgs=2294 cs=0 l=1 rx=0x7fa730002f30 tx=0x7fa73000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:29.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:29.473+0000 7fa73a8d4700 1 --2- 172.21.15.42:0/3990903428 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa7340a7730 0x7fa73408df70 secure :-1 s=THROTTLE_DONE pgs=2294 cs=0 l=1 rx=0x7fa730002f30 tx=0x7fa73000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:29.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:29.473+0000 7fa7398d2700 1 -- 172.21.15.42:0/3990903428 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa730008bb0 con 0x7fa7340a7730 2019-12-05T06:13:29.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:29.473+0000 7fa7398d2700 1 -- 172.21.15.42:0/3990903428 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa730011950 con 0x7fa7340a7730 2019-12-05T06:13:29.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:29.473+0000 7fa73c337700 1 -- 172.21.15.42:0/3990903428 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa7340a7730 msgr2=0x7fa73408df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:13:29.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:29.473+0000 7fa73c337700 1 --2- 172.21.15.42:0/3990903428 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa7340a7730 0x7fa73408df70 secure :-1 s=READY pgs=2294 cs=0 l=1 rx=0x7fa730002f30 tx=0x7fa73000fd30).stop 2019-12-05T06:13:29.482 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:29.473+0000 7fa73c337700 1 -- 172.21.15.42:0/3990903428 shutdown_connections 2019-12-05T06:13:29.482 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:29.473+0000 7fa73c337700 1 --2- 172.21.15.42:0/3990903428 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa7340a7730 0x7fa73408df70 unknown :-1 s=CLOSED pgs=2294 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:13:29.482 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:29.473+0000 7fa73c337700 1 --2- 172.21.15.42:0/3990903428 >> v2:172.21.15.40:3300/0 conn(0x7fa73407e710 0x7fa73407eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:13:29.482 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:29.473+0000 7fa73c337700 1 -- 172.21.15.42:0/3990903428 shutdown_connections 2019-12-05T06:13:29.482 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:29.473+0000 7fa73c337700 1 -- 172.21.15.42:0/3990903428 wait complete. 2019-12-05T06:13:29.482 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:29.473+0000 7fa73c337700 1 -- 172.21.15.42:0/3990903428 >> 172.21.15.42:0/3990903428 conn(0x7fa7340aabd0 msgr2=0x7fa7340a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:13:29.486 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:29.473+0000 7fa73c337700 1 Processor -- start 2019-12-05T06:13:29.487 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:29.473+0000 7fa73c337700 1 -- start start 2019-12-05T06:13:29.487 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:29.477+0000 7fa73c337700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fa73405f7b0 0x7fa734144bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:13:29.487 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:29.477+0000 7fa73c337700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa73408ee80 0x7fa734143740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:13:29.487 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:29.477+0000 7fa73c337700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa734143c90 0x7fa734124d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:13:29.487 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:29.477+0000 7fa73c337700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa73405fcc0 con 0x7fa734143c90 2019-12-05T06:13:29.487 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:29.477+0000 7fa73c337700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa73405ff60 con 0x7fa73405f7b0 2019-12-05T06:13:29.487 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:29.477+0000 7fa73c337700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7341440c0 con 0x7fa73408ee80 2019-12-05T06:13:29.488 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:29.477+0000 7fa73a8d4700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fa73405f7b0 0x7fa734144bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:13:29.488 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:29.477+0000 7fa73a8d4700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fa73405f7b0 0x7fa734144bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:50980/0 (socket says 172.21.15.42:50980) 2019-12-05T06:13:29.488 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:29.477+0000 7fa73a8d4700 1 -- 172.21.15.42:0/1016870960 learned_addr learned my addr 172.21.15.42:0/1016870960 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:13:29.488 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:29.477+0000 7fa73a0d3700 1 --2- 172.21.15.42:0/1016870960 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa73408ee80 0x7fa734143740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:13:29.488 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:29.477+0000 7fa73b0d5700 1 --2- 172.21.15.42:0/1016870960 >> v2:172.21.15.40:3300/0 conn(0x7fa734143c90 0x7fa734124d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:13:29.489 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:29.477+0000 7fa73a8d4700 1 -- 172.21.15.42:0/1016870960 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa73408ee80 msgr2=0x7fa734143740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:13:29.489 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:29.477+0000 7fa73a8d4700 1 --2- 172.21.15.42:0/1016870960 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa73408ee80 0x7fa734143740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:13:29.489 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:29.477+0000 7fa73a8d4700 1 -- 172.21.15.42:0/1016870960 >> v2:172.21.15.40:3300/0 conn(0x7fa734143c90 msgr2=0x7fa734124d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:13:29.489 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:29.477+0000 7fa73a8d4700 1 --2- 172.21.15.42:0/1016870960 >> v2:172.21.15.40:3300/0 conn(0x7fa734143c90 0x7fa734124d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:13:29.489 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:29.477+0000 7fa73a8d4700 1 -- 172.21.15.42:0/1016870960 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7341252c0 con 0x7fa73405f7b0 2019-12-05T06:13:29.489 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:29.477+0000 7fa73a8d4700 1 --2- 172.21.15.42:0/1016870960 >> v2:172.21.15.42:3300/0 conn(0x7fa73405f7b0 0x7fa734144bb0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fa730002f30 tx=0x7fa730008c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:29.489 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:29.477+0000 7fa73a8d4700 1 --2- 172.21.15.42:0/1016870960 >> v2:172.21.15.42:3300/0 conn(0x7fa73405f7b0 0x7fa734144bb0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fa730002f30 tx=0x7fa730008c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:29.490 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:29.477+0000 7fa73a8d4700 1 --2- 172.21.15.42:0/1016870960 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa73405f7b0 0x7fa734144bb0 secure :-1 s=READY pgs=2295 cs=0 l=1 rx=0x7fa730002f30 tx=0x7fa730008c30).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:13:29.490 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:29.477+0000 7fa73a8d4700 1 --2- 172.21.15.42:0/1016870960 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa73405f7b0 0x7fa734144bb0 secure :-1 s=THROTTLE_DONE pgs=2295 cs=0 l=1 rx=0x7fa730002f30 tx=0x7fa730008c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:29.490 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:29.477+0000 7fa72b7fe700 1 -- 172.21.15.42:0/1016870960 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa730007150 con 0x7fa73405f7b0 2019-12-05T06:13:29.490 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:29.477+0000 7fa73a8d4700 1 --2- 172.21.15.42:0/1016870960 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa73405f7b0 0x7fa734144bb0 secure :-1 s=THROTTLE_DONE pgs=2295 cs=0 l=1 rx=0x7fa730002f30 tx=0x7fa730008c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:29.490 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:29.477+0000 7fa73a8d4700 1 --2- 172.21.15.42:0/1016870960 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa73405f7b0 0x7fa734144bb0 secure :-1 s=THROTTLE_DONE pgs=2295 cs=0 l=1 rx=0x7fa730002f30 tx=0x7fa730008c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:29.490 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:29.477+0000 7fa72b7fe700 1 -- 172.21.15.42:0/1016870960 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa730011570 con 0x7fa73405f7b0 2019-12-05T06:13:29.490 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:29.477+0000 7fa73c337700 1 -- 172.21.15.42:0/1016870960 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa734125590 con 0x7fa73405f7b0 2019-12-05T06:13:29.491 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:29.477+0000 7fa72b7fe700 1 -- 172.21.15.42:0/1016870960 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa730005730 con 0x7fa73405f7b0 2019-12-05T06:13:29.491 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:29.477+0000 7fa73c337700 1 -- 172.21.15.42:0/1016870960 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7341281b0 con 0x7fa73405f7b0 2019-12-05T06:13:29.491 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:29.477+0000 7fa73a8d4700 1 --2- 172.21.15.42:0/1016870960 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa73405f7b0 0x7fa734144bb0 secure :-1 s=THROTTLE_DONE pgs=2295 cs=0 l=1 rx=0x7fa730002f30 tx=0x7fa730008c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:29.491 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:29.477+0000 7fa72b7fe700 1 -- 172.21.15.42:0/1016870960 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fa730005a40 con 0x7fa73405f7b0 2019-12-05T06:13:29.491 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:29.477+0000 7fa73a8d4700 1 --2- 172.21.15.42:0/1016870960 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa73405f7b0 0x7fa734144bb0 secure :-1 s=THROTTLE_DONE pgs=2295 cs=0 l=1 rx=0x7fa730002f30 tx=0x7fa730008c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:29.491 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:29.481+0000 7fa72b7fe700 1 --2- 172.21.15.42:0/1016870960 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa71c03ae60 0x7fa71c03d370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:13:29.491 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:29.481+0000 7fa72b7fe700 1 -- 172.21.15.42:0/1016870960 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 10245+0+0 (secure 0 0 0) 0x7fa73004b390 con 0x7fa73405f7b0 2019-12-05T06:13:29.492 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:29.481+0000 7fa73a0d3700 1 --2- 172.21.15.42:0/1016870960 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa71c03ae60 0x7fa71c03d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:13:29.492 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:29.481+0000 7fa73a0d3700 1 --2- 172.21.15.42:0/1016870960 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa71c03ae60 0x7fa71c03d370 crc :-1 s=READY pgs=966 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:13:29.492 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:29.485+0000 7fa73c337700 1 -- 172.21.15.42:0/1016870960 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa73405d100 con 0x7fa73405f7b0 2019-12-05T06:13:29.496 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:29.493+0000 7fa73a8d4700 1 --2- 172.21.15.42:0/1016870960 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa73405f7b0 0x7fa734144bb0 secure :-1 s=THROTTLE_DONE pgs=2295 cs=0 l=1 rx=0x7fa730002f30 tx=0x7fa730008c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:29.496 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:29.493+0000 7fa72b7fe700 1 -- 172.21.15.42:0/1016870960 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fa730033220 con 0x7fa73405f7b0 2019-12-05T06:13:29.672 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:29.669+0000 7fa73c337700 1 -- 172.21.15.42:0/1016870960 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa73405b8b0 con 0x7fa73405f7b0 2019-12-05T06:13:29.673 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:29.669+0000 7fa73a8d4700 1 --2- 172.21.15.42:0/1016870960 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa73405f7b0 0x7fa734144bb0 secure :-1 s=THROTTLE_DONE pgs=2295 cs=0 l=1 rx=0x7fa730002f30 tx=0x7fa730008c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:29.673 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:29.669+0000 7fa72b7fe700 1 -- 172.21.15.42:0/1016870960 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 277 v277) v1 ==== 95+0+1778 (secure 0 0 0) 0x7fa730032930 con 0x7fa73405f7b0 2019-12-05T06:13:29.673 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 277 2019-12-05T06:13:29.673 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:13:29.674 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":277,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":277,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:13:19.980391+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":98,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10246},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10246":{"gid":10246,"name":"a","rank":0,"incarnation":274,"state":"up:active","state_seq":17,"addr":"172.21.15.42:6835/4032647860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":4032647860},{"type":"v1","addr":"172.21.15.42:6835","nonce":4032647860}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:13:29.676 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:29.669+0000 7fa73c337700 1 -- 172.21.15.42:0/1016870960 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa71c03ae60 msgr2=0x7fa71c03d370 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:13:29.676 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:29.669+0000 7fa73c337700 1 --2- 172.21.15.42:0/1016870960 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa71c03ae60 0x7fa71c03d370 crc :-1 s=READY pgs=966 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:13:29.676 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:29.669+0000 7fa73c337700 1 -- 172.21.15.42:0/1016870960 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa73405f7b0 msgr2=0x7fa734144bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:13:29.676 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:29.669+0000 7fa73c337700 1 --2- 172.21.15.42:0/1016870960 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa73405f7b0 0x7fa734144bb0 secure :-1 s=READY pgs=2295 cs=0 l=1 rx=0x7fa730002f30 tx=0x7fa730008c30).stop 2019-12-05T06:13:29.676 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:29.669+0000 7fa73c337700 1 -- 172.21.15.42:0/1016870960 shutdown_connections 2019-12-05T06:13:29.676 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:29.669+0000 7fa73c337700 1 --2- 172.21.15.42:0/1016870960 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa71c03ae60 0x7fa71c03d370 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:13:29.677 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:29.669+0000 7fa73c337700 1 --2- 172.21.15.42:0/1016870960 >> v2:172.21.15.40:3300/0 conn(0x7fa734143c90 0x7fa734124d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:13:29.677 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:29.669+0000 7fa73c337700 1 --2- 172.21.15.42:0/1016870960 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa73405f7b0 0x7fa734144bb0 unknown :-1 s=CLOSED pgs=2295 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:13:29.677 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:29.669+0000 7fa73c337700 1 --2- 172.21.15.42:0/1016870960 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa73408ee80 0x7fa734143740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:13:29.677 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:29.669+0000 7fa73c337700 1 -- 172.21.15.42:0/1016870960 shutdown_connections 2019-12-05T06:13:29.677 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:29.669+0000 7fa73c337700 1 -- 172.21.15.42:0/1016870960 wait complete. 2019-12-05T06:13:29.677 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:29.669+0000 7fa73c337700 1 -- 172.21.15.42:0/1016870960 >> 172.21.15.42:0/1016870960 conn(0x7fa7340a8400 msgr2=0x7fa7340a9760 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:13:29.690 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2019-12-05T06:13:31.692 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:13:31.797 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:31.789+0000 7fc32b813700 1 Processor -- start 2019-12-05T06:13:31.797 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:31.789+0000 7fc32b813700 1 -- start start 2019-12-05T06:13:31.798 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:31.789+0000 7fc32b813700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc3240a7740 0x7fc32408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:13:31.798 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:31.789+0000 7fc32b813700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc32409fae0 0x7fc32407eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:13:31.798 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:31.789+0000 7fc32b813700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3240a0710 con 0x7fc32407e720 2019-12-05T06:13:31.798 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:31.789+0000 7fc32b813700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3240a0b00 con 0x7fc32409fae0 2019-12-05T06:13:31.798 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:31.789+0000 7fc32b813700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc32408e5b0 con 0x7fc3240a7740 2019-12-05T06:13:31.798 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:31.789+0000 7fc32a5b1700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc32409fae0 0x7fc32407eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:13:31.799 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:31.789+0000 7fc32a5b1700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc32409fae0 0x7fc32407eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:50992/0 (socket says 172.21.15.42:50992) 2019-12-05T06:13:31.799 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:31.789+0000 7fc32a5b1700 1 -- 172.21.15.42:0/801411172 learned_addr learned my addr 172.21.15.42:0/801411172 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:13:31.799 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:31.789+0000 7fc329db0700 1 --2- 172.21.15.42:0/801411172 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc3240a7740 0x7fc32408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:13:31.799 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:31.789+0000 7fc32a5b1700 1 -- 172.21.15.42:0/801411172 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc3240a7740 msgr2=0x7fc32408df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:13:31.799 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:31.789+0000 7fc32a5b1700 1 --2- 172.21.15.42:0/801411172 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc3240a7740 0x7fc32408df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:13:31.799 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:31.789+0000 7fc32a5b1700 1 -- 172.21.15.42:0/801411172 >> v1:172.21.15.40:6789/0 conn(0x7fc32407e720 legacy=0x7fc32409f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:13:31.799 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:31.789+0000 7fc32a5b1700 1 -- 172.21.15.42:0/801411172 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc32407f0a0 con 0x7fc32409fae0 2019-12-05T06:13:31.800 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:31.789+0000 7fc32a5b1700 1 --2- 172.21.15.42:0/801411172 >> v2:172.21.15.42:3300/0 conn(0x7fc32409fae0 0x7fc32407eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fc314008790 tx=0x7fc31400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:31.800 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:31.789+0000 7fc32a5b1700 1 --2- 172.21.15.42:0/801411172 >> v2:172.21.15.42:3300/0 conn(0x7fc32409fae0 0x7fc32407eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fc314008790 tx=0x7fc31400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:31.800 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:31.789+0000 7fc32a5b1700 1 --2- 172.21.15.42:0/801411172 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc32409fae0 0x7fc32407eb50 secure :-1 s=READY pgs=2296 cs=0 l=1 rx=0x7fc314008790 tx=0x7fc31400fd30).ready entity=mon.1 client_cookie=f840550dae2290d4 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:13:31.800 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:31.793+0000 7fc32a5b1700 1 --2- 172.21.15.42:0/801411172 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc32409fae0 0x7fc32407eb50 secure :-1 s=THROTTLE_DONE pgs=2296 cs=0 l=1 rx=0x7fc314008790 tx=0x7fc31400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:31.801 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:31.793+0000 7fc32a5b1700 1 --2- 172.21.15.42:0/801411172 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc32409fae0 0x7fc32407eb50 secure :-1 s=THROTTLE_DONE pgs=2296 cs=0 l=1 rx=0x7fc314008790 tx=0x7fc31400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:31.801 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:31.793+0000 7fc328dae700 1 -- 172.21.15.42:0/801411172 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc314008c90 con 0x7fc32409fae0 2019-12-05T06:13:31.801 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:31.793+0000 7fc32a5b1700 1 --2- 172.21.15.42:0/801411172 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc32409fae0 0x7fc32407eb50 secure :-1 s=THROTTLE_DONE pgs=2296 cs=0 l=1 rx=0x7fc314008790 tx=0x7fc31400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:31.801 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:31.793+0000 7fc328dae700 1 -- 172.21.15.42:0/801411172 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3140088b0 con 0x7fc32409fae0 2019-12-05T06:13:31.801 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:31.793+0000 7fc328dae700 1 -- 172.21.15.42:0/801411172 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc314011b10 con 0x7fc32409fae0 2019-12-05T06:13:31.802 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:31.793+0000 7fc32b813700 1 -- 172.21.15.42:0/801411172 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc32409fae0 msgr2=0x7fc32407eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:13:31.802 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:31.793+0000 7fc32b813700 1 --2- 172.21.15.42:0/801411172 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc32409fae0 0x7fc32407eb50 secure :-1 s=READY pgs=2296 cs=0 l=1 rx=0x7fc314008790 tx=0x7fc31400fd30).stop 2019-12-05T06:13:31.802 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:31.793+0000 7fc32b813700 1 -- 172.21.15.42:0/801411172 shutdown_connections 2019-12-05T06:13:31.802 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:31.793+0000 7fc32b813700 1 --2- 172.21.15.42:0/801411172 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc32409fae0 0x7fc32407eb50 unknown :-1 s=CLOSED pgs=2296 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:13:31.802 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:31.793+0000 7fc32b813700 1 --2- 172.21.15.42:0/801411172 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc3240a7740 0x7fc32408df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:13:31.802 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:31.793+0000 7fc32b813700 1 -- 172.21.15.42:0/801411172 shutdown_connections 2019-12-05T06:13:31.802 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:31.793+0000 7fc32b813700 1 -- 172.21.15.42:0/801411172 wait complete. 2019-12-05T06:13:31.803 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:31.793+0000 7fc32b813700 1 -- 172.21.15.42:0/801411172 >> 172.21.15.42:0/801411172 conn(0x7fc3240aabe0 msgr2=0x7fc3240a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:13:31.812 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:31.793+0000 7fc32b813700 1 Processor -- start 2019-12-05T06:13:31.812 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:31.793+0000 7fc32b813700 1 -- start start 2019-12-05T06:13:31.812 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:31.793+0000 7fc32b813700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc3240a6a70 0x7fc32411a800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:13:31.812 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:31.793+0000 7fc32b813700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc32411bca0 0x7fc3240a4840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:13:31.813 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:31.793+0000 7fc32b813700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3240a4d90 con 0x7fc324116500 2019-12-05T06:13:31.813 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:31.793+0000 7fc32b813700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3240a6f80 con 0x7fc32411bca0 2019-12-05T06:13:31.813 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:31.793+0000 7fc32b813700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc32411a560 con 0x7fc3240a6a70 2019-12-05T06:13:31.813 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:31.797+0000 7fc32a5b1700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc32411bca0 0x7fc3240a4840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:13:31.813 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:31.797+0000 7fc329db0700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc3240a6a70 0x7fc32411a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:13:31.813 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:31.797+0000 7fc32a5b1700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc32411bca0 0x7fc3240a4840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:50998/0 (socket says 172.21.15.42:50998) 2019-12-05T06:13:31.813 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:31.797+0000 7fc32a5b1700 1 -- 172.21.15.42:0/2783412099 learned_addr learned my addr 172.21.15.42:0/2783412099 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:13:31.814 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:31.797+0000 7fc32a5b1700 1 -- 172.21.15.42:0/2783412099 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc3240a6a70 msgr2=0x7fc32411a800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:13:31.814 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:31.797+0000 7fc32a5b1700 1 --2- 172.21.15.42:0/2783412099 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc3240a6a70 0x7fc32411a800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:13:31.814 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:31.797+0000 7fc3227fc700 1 -- 172.21.15.42:0/2783412099 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1488304977 0 0) 0x7fc3240a4d90 con 0x7fc324116500 2019-12-05T06:13:31.814 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:31.797+0000 7fc32a5b1700 1 -- 172.21.15.42:0/2783412099 >> v1:172.21.15.40:6789/0 conn(0x7fc324116500 legacy=0x7fc32408eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:13:31.814 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:31.797+0000 7fc32a5b1700 1 -- 172.21.15.42:0/2783412099 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc32411c0d0 con 0x7fc32411bca0 2019-12-05T06:13:31.814 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:31.797+0000 7fc32a5b1700 1 --2- 172.21.15.42:0/2783412099 >> v2:172.21.15.42:3300/0 conn(0x7fc32411bca0 0x7fc3240a4840 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fc314008790 tx=0x7fc314008b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:31.815 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:31.797+0000 7fc32a5b1700 1 --2- 172.21.15.42:0/2783412099 >> v2:172.21.15.42:3300/0 conn(0x7fc32411bca0 0x7fc3240a4840 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fc314008790 tx=0x7fc314008b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:31.815 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:31.797+0000 7fc32a5b1700 1 --2- 172.21.15.42:0/2783412099 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc32411bca0 0x7fc3240a4840 secure :-1 s=READY pgs=2297 cs=0 l=1 rx=0x7fc314008790 tx=0x7fc314008b60).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:13:31.815 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:31.797+0000 7fc32a5b1700 1 --2- 172.21.15.42:0/2783412099 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc32411bca0 0x7fc3240a4840 secure :-1 s=THROTTLE_DONE pgs=2297 cs=0 l=1 rx=0x7fc314008790 tx=0x7fc314008b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:31.815 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:31.797+0000 7fc3227fc700 1 -- 172.21.15.42:0/2783412099 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc31400f680 con 0x7fc32411bca0 2019-12-05T06:13:31.815 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:31.797+0000 7fc32a5b1700 1 --2- 172.21.15.42:0/2783412099 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc32411bca0 0x7fc3240a4840 secure :-1 s=THROTTLE_DONE pgs=2297 cs=0 l=1 rx=0x7fc314008790 tx=0x7fc314008b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:31.815 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:31.797+0000 7fc32a5b1700 1 --2- 172.21.15.42:0/2783412099 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc32411bca0 0x7fc3240a4840 secure :-1 s=THROTTLE_DONE pgs=2297 cs=0 l=1 rx=0x7fc314008790 tx=0x7fc314008b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:31.815 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:31.797+0000 7fc3227fc700 1 -- 172.21.15.42:0/2783412099 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc314011730 con 0x7fc32411bca0 2019-12-05T06:13:31.816 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:31.797+0000 7fc3227fc700 1 -- 172.21.15.42:0/2783412099 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc314005920 con 0x7fc32411bca0 2019-12-05T06:13:31.816 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:31.797+0000 7fc32b813700 1 -- 172.21.15.42:0/2783412099 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc32408f370 con 0x7fc32411bca0 2019-12-05T06:13:31.816 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:31.797+0000 7fc32b813700 1 -- 172.21.15.42:0/2783412099 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc32408f8a0 con 0x7fc32411bca0 2019-12-05T06:13:31.816 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:31.797+0000 7fc32a5b1700 1 --2- 172.21.15.42:0/2783412099 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc32411bca0 0x7fc3240a4840 secure :-1 s=THROTTLE_DONE pgs=2297 cs=0 l=1 rx=0x7fc314008790 tx=0x7fc314008b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:31.817 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:31.797+0000 7fc3227fc700 1 -- 172.21.15.42:0/2783412099 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fc314005c30 con 0x7fc32411bca0 2019-12-05T06:13:31.817 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:31.797+0000 7fc32a5b1700 1 --2- 172.21.15.42:0/2783412099 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc32411bca0 0x7fc3240a4840 secure :-1 s=THROTTLE_DONE pgs=2297 cs=0 l=1 rx=0x7fc314008790 tx=0x7fc314008b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:31.817 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:31.797+0000 7fc3227fc700 1 --2- 172.21.15.42:0/2783412099 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc30403ac50 0x7fc30403d160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:13:31.817 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:31.797+0000 7fc3227fc700 1 -- 172.21.15.42:0/2783412099 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 10245+0+0 (secure 0 0 0) 0x7fc31404b6a0 con 0x7fc32411bca0 2019-12-05T06:13:31.817 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:31.797+0000 7fc329db0700 1 --2- 172.21.15.42:0/2783412099 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc30403ac50 0x7fc30403d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:13:31.817 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:31.797+0000 7fc329db0700 1 --2- 172.21.15.42:0/2783412099 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc30403ac50 0x7fc30403d160 crc :-1 s=READY pgs=968 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:13:31.817 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:31.805+0000 7fc32b813700 1 -- 172.21.15.42:0/2783412099 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc32405d110 con 0x7fc32411bca0 2019-12-05T06:13:31.820 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:31.817+0000 7fc32a5b1700 1 --2- 172.21.15.42:0/2783412099 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc32411bca0 0x7fc3240a4840 secure :-1 s=THROTTLE_DONE pgs=2297 cs=0 l=1 rx=0x7fc314008790 tx=0x7fc314008b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:31.820 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:31.817+0000 7fc3227fc700 1 -- 172.21.15.42:0/2783412099 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fc314032720 con 0x7fc32411bca0 2019-12-05T06:13:31.993 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:31.989+0000 7fc32b813700 1 -- 172.21.15.42:0/2783412099 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc32405b8c0 con 0x7fc32411bca0 2019-12-05T06:13:31.994 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:31.989+0000 7fc32a5b1700 1 --2- 172.21.15.42:0/2783412099 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc32411bca0 0x7fc3240a4840 secure :-1 s=THROTTLE_DONE pgs=2297 cs=0 l=1 rx=0x7fc314008790 tx=0x7fc314008b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:31.994 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:31.989+0000 7fc3227fc700 1 -- 172.21.15.42:0/2783412099 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 278 v278) v1 ==== 95+0+2115 (secure 0 0 0) 0x7fc314030340 con 0x7fc32411bca0 2019-12-05T06:13:31.995 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 278 2019-12-05T06:13:31.995 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:13:31.995 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":278,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7217,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/2667648531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2667648531},{"type":"v1","addr":"172.21.15.40:6833","nonce":2667648531}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":278}],"filesystems":[{"mdsmap":{"epoch":277,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:13:19.980391+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":98,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10246},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10246":{"gid":10246,"name":"a","rank":0,"incarnation":274,"state":"up:active","state_seq":17,"addr":"172.21.15.42:6835/4032647860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":4032647860},{"type":"v1","addr":"172.21.15.42:6835","nonce":4032647860}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:13:31.998 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:31.993+0000 7fc32b813700 1 -- 172.21.15.42:0/2783412099 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc30403ac50 msgr2=0x7fc30403d160 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:13:31.998 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:31.993+0000 7fc32b813700 1 --2- 172.21.15.42:0/2783412099 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc30403ac50 0x7fc30403d160 crc :-1 s=READY pgs=968 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:13:31.998 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:31.993+0000 7fc32b813700 1 -- 172.21.15.42:0/2783412099 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc32411bca0 msgr2=0x7fc3240a4840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:13:31.999 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:31.993+0000 7fc32b813700 1 --2- 172.21.15.42:0/2783412099 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc32411bca0 0x7fc3240a4840 secure :-1 s=READY pgs=2297 cs=0 l=1 rx=0x7fc314008790 tx=0x7fc314008b60).stop 2019-12-05T06:13:31.999 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:31.993+0000 7fc32b813700 1 -- 172.21.15.42:0/2783412099 shutdown_connections 2019-12-05T06:13:31.999 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:31.993+0000 7fc32b813700 1 --2- 172.21.15.42:0/2783412099 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc30403ac50 0x7fc30403d160 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:13:31.999 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:31.993+0000 7fc32b813700 1 --2- 172.21.15.42:0/2783412099 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc32411bca0 0x7fc3240a4840 unknown :-1 s=CLOSED pgs=2297 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:13:32.000 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:31.993+0000 7fc32b813700 1 --2- 172.21.15.42:0/2783412099 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc3240a6a70 0x7fc32411a800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:13:32.000 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:31.993+0000 7fc32b813700 1 -- 172.21.15.42:0/2783412099 shutdown_connections 2019-12-05T06:13:32.000 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:31.993+0000 7fc32b813700 1 -- 172.21.15.42:0/2783412099 wait complete. 2019-12-05T06:13:32.000 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:13:31.993+0000 7fc32b813700 1 -- 172.21.15.42:0/2783412099 >> 172.21.15.42:0/2783412099 conn(0x7fc3240a8230 msgr2=0x7fc32411c4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:13:32.018 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b reported in up:standby state 2019-12-05T06:13:32.018 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 48 secs before thrashing 2019-12-05T06:13:45.085 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:13:45.090 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:14:15.148 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:14:15.153 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:14:20.019 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:14:20.119 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.109+0000 7faf44e53700 1 Processor -- start 2019-12-05T06:14:20.124 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.109+0000 7faf44e53700 1 -- start start 2019-12-05T06:14:20.124 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.109+0000 7faf44e53700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7faf400a7740 0x7faf4008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:14:20.125 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.109+0000 7faf44e53700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7faf4007e720 0x7faf4007eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:14:20.125 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.109+0000 7faf44e53700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf400a0710 con 0x7faf4009f2e0 2019-12-05T06:14:20.125 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.109+0000 7faf44e53700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf400a0b00 con 0x7faf400a7740 2019-12-05T06:14:20.125 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.109+0000 7faf44e53700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf4008e5b0 con 0x7faf4007e720 2019-12-05T06:14:20.125 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.109+0000 7faf3ed9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7faf400a7740 0x7faf4008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:20.125 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.109+0000 7faf3ed9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7faf400a7740 0x7faf4008df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:51178/0 (socket says 172.21.15.42:51178) 2019-12-05T06:14:20.125 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.109+0000 7faf3ed9d700 1 -- 172.21.15.42:0/516166772 learned_addr learned my addr 172.21.15.42:0/516166772 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:14:20.126 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.109+0000 7faf3e59c700 1 --2- 172.21.15.42:0/516166772 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7faf4007e720 0x7faf4007eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:20.126 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.109+0000 7faf3ed9d700 1 -- 172.21.15.42:0/516166772 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7faf4007e720 msgr2=0x7faf4007eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:14:20.126 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.109+0000 7faf3ed9d700 1 --2- 172.21.15.42:0/516166772 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7faf4007e720 0x7faf4007eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:14:20.126 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.109+0000 7faf3ed9d700 1 -- 172.21.15.42:0/516166772 >> v1:172.21.15.40:6789/0 conn(0x7faf4009f2e0 legacy=0x7faf4009f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:20.126 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.109+0000 7faf3ed9d700 1 -- 172.21.15.42:0/516166772 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf4007f0a0 con 0x7faf400a7740 2019-12-05T06:14:20.126 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.113+0000 7faf3ed9d700 1 --2- 172.21.15.42:0/516166772 >> v2:172.21.15.42:3300/0 conn(0x7faf400a7740 0x7faf4008df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7faf30008790 tx=0x7faf3000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:20.127 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.113+0000 7faf3ed9d700 1 --2- 172.21.15.42:0/516166772 >> v2:172.21.15.42:3300/0 conn(0x7faf400a7740 0x7faf4008df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7faf30008790 tx=0x7faf3000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:20.127 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.113+0000 7faf3ed9d700 1 --2- 172.21.15.42:0/516166772 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf400a7740 0x7faf4008df80 secure :-1 s=READY pgs=2303 cs=0 l=1 rx=0x7faf30008790 tx=0x7faf3000fd30).ready entity=mon.1 client_cookie=82bb94d547645636 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:14:20.127 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.113+0000 7faf3ed9d700 1 --2- 172.21.15.42:0/516166772 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf400a7740 0x7faf4008df80 secure :-1 s=THROTTLE_DONE pgs=2303 cs=0 l=1 rx=0x7faf30008790 tx=0x7faf3000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:20.127 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.113+0000 7faf3ed9d700 1 --2- 172.21.15.42:0/516166772 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf400a7740 0x7faf4008df80 secure :-1 s=THROTTLE_DONE pgs=2303 cs=0 l=1 rx=0x7faf30008790 tx=0x7faf3000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:20.127 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.113+0000 7faf3dd9b700 1 -- 172.21.15.42:0/516166772 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faf300113b0 con 0x7faf400a7740 2019-12-05T06:14:20.127 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.113+0000 7faf3ed9d700 1 --2- 172.21.15.42:0/516166772 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf400a7740 0x7faf4008df80 secure :-1 s=THROTTLE_DONE pgs=2303 cs=0 l=1 rx=0x7faf30008790 tx=0x7faf3000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:20.127 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.113+0000 7faf3dd9b700 1 -- 172.21.15.42:0/516166772 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf30008bb0 con 0x7faf400a7740 2019-12-05T06:14:20.128 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.113+0000 7faf3dd9b700 1 -- 172.21.15.42:0/516166772 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faf30011990 con 0x7faf400a7740 2019-12-05T06:14:20.129 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.113+0000 7faf44e53700 1 -- 172.21.15.42:0/516166772 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf400a7740 msgr2=0x7faf4008df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:20.129 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.113+0000 7faf44e53700 1 --2- 172.21.15.42:0/516166772 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf400a7740 0x7faf4008df80 secure :-1 s=READY pgs=2303 cs=0 l=1 rx=0x7faf30008790 tx=0x7faf3000fd30).stop 2019-12-05T06:14:20.129 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.113+0000 7faf44e53700 1 -- 172.21.15.42:0/516166772 shutdown_connections 2019-12-05T06:14:20.130 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.113+0000 7faf44e53700 1 --2- 172.21.15.42:0/516166772 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf400a7740 0x7faf4008df80 unknown :-1 s=CLOSED pgs=2303 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:20.130 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.113+0000 7faf44e53700 1 --2- 172.21.15.42:0/516166772 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7faf4007e720 0x7faf4007eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:14:20.131 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.113+0000 7faf44e53700 1 -- 172.21.15.42:0/516166772 shutdown_connections 2019-12-05T06:14:20.131 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.113+0000 7faf44e53700 1 -- 172.21.15.42:0/516166772 wait complete. 2019-12-05T06:14:20.131 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.113+0000 7faf44e53700 1 -- 172.21.15.42:0/516166772 >> 172.21.15.42:0/516166772 conn(0x7faf400aabe0 msgr2=0x7faf400a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:14:20.135 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.117+0000 7faf44e53700 1 Processor -- start 2019-12-05T06:14:20.135 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.117+0000 7faf44e53700 1 -- start start 2019-12-05T06:14:20.135 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.117+0000 7faf44e53700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7faf40121490 0x7faf40118930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:14:20.135 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.117+0000 7faf44e53700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7faf40117620 0x7faf40126710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:14:20.135 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.117+0000 7faf44e53700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf40119ea0 con 0x7faf40123c00 2019-12-05T06:14:20.136 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.117+0000 7faf44e53700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf40117bb0 con 0x7faf40121490 2019-12-05T06:14:20.136 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.117+0000 7faf44e53700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf40117e50 con 0x7faf40117620 2019-12-05T06:14:20.136 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.117+0000 7faf3ed9d700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7faf40123c00 0x7faf40120c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:51222/0 (socket says 172.21.15.42:51222) 2019-12-05T06:14:20.136 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.117+0000 7faf3ed9d700 1 -- 172.21.15.42:0/1041434544 learned_addr learned my addr 172.21.15.42:0/1041434544 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:14:20.136 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.117+0000 7faf3e59c700 1 --2- 172.21.15.42:0/1041434544 >> v2:172.21.15.42:3300/0 conn(0x7faf40121490 0x7faf40118930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:20.136 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.117+0000 7faf3f59e700 1 --2- 172.21.15.42:0/1041434544 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7faf40117620 0x7faf40126710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:20.136 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.117+0000 7faf3e59c700 1 -- 172.21.15.42:0/1041434544 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7faf40117620 msgr2=0x7faf40126710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:20.137 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.117+0000 7faf3e59c700 1 --2- 172.21.15.42:0/1041434544 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7faf40117620 0x7faf40126710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:20.137 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.117+0000 7faf2f7fe700 1 -- 172.21.15.42:0/1041434544 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3918378439 0 0) 0x7faf40119ea0 con 0x7faf40123c00 2019-12-05T06:14:20.137 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.117+0000 7faf3e59c700 1 -- 172.21.15.42:0/1041434544 >> v1:172.21.15.42:6789/0 conn(0x7faf40123c00 legacy=0x7faf40120c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:20.137 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.117+0000 7faf3e59c700 1 -- 172.21.15.42:0/1041434544 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf40126c60 con 0x7faf40121490 2019-12-05T06:14:20.137 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.117+0000 7faf3e59c700 1 --2- 172.21.15.42:0/1041434544 >> v2:172.21.15.42:3300/0 conn(0x7faf40121490 0x7faf40118930 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7faf40119c10 tx=0x7faf34010b20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:20.137 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.117+0000 7faf3e59c700 1 --2- 172.21.15.42:0/1041434544 >> v2:172.21.15.42:3300/0 conn(0x7faf40121490 0x7faf40118930 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7faf40119c10 tx=0x7faf34010b20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:20.137 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.117+0000 7faf3e59c700 1 --2- 172.21.15.42:0/1041434544 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf40121490 0x7faf40118930 secure :-1 s=READY pgs=2305 cs=0 l=1 rx=0x7faf40119c10 tx=0x7faf34010b20).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:14:20.138 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.117+0000 7faf3e59c700 1 --2- 172.21.15.42:0/1041434544 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf40121490 0x7faf40118930 secure :-1 s=THROTTLE_DONE pgs=2305 cs=0 l=1 rx=0x7faf40119c10 tx=0x7faf34010b20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:20.138 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.117+0000 7faf2f7fe700 1 -- 172.21.15.42:0/1041434544 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faf34012970 con 0x7faf40121490 2019-12-05T06:14:20.138 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.117+0000 7faf3e59c700 1 --2- 172.21.15.42:0/1041434544 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf40121490 0x7faf40118930 secure :-1 s=THROTTLE_DONE pgs=2305 cs=0 l=1 rx=0x7faf40119c10 tx=0x7faf34010b20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:20.138 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.117+0000 7faf2f7fe700 1 -- 172.21.15.42:0/1041434544 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf34012590 con 0x7faf40121490 2019-12-05T06:14:20.139 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.117+0000 7faf44e53700 1 -- 172.21.15.42:0/1041434544 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf40126ed0 con 0x7faf40121490 2019-12-05T06:14:20.139 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.117+0000 7faf3e59c700 1 --2- 172.21.15.42:0/1041434544 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf40121490 0x7faf40118930 secure :-1 s=THROTTLE_DONE pgs=2305 cs=0 l=1 rx=0x7faf40119c10 tx=0x7faf34010b20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:20.139 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.117+0000 7faf2f7fe700 1 -- 172.21.15.42:0/1041434544 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faf34005730 con 0x7faf40121490 2019-12-05T06:14:20.139 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.121+0000 7faf44e53700 1 -- 172.21.15.42:0/1041434544 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf40127450 con 0x7faf40121490 2019-12-05T06:14:20.139 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.121+0000 7faf3e59c700 1 --2- 172.21.15.42:0/1041434544 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf40121490 0x7faf40118930 secure :-1 s=THROTTLE_DONE pgs=2305 cs=0 l=1 rx=0x7faf40119c10 tx=0x7faf34010b20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:20.140 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.121+0000 7faf2f7fe700 1 -- 172.21.15.42:0/1041434544 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7faf34007180 con 0x7faf40121490 2019-12-05T06:14:20.140 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.121+0000 7faf3e59c700 1 --2- 172.21.15.42:0/1041434544 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf40121490 0x7faf40118930 secure :-1 s=THROTTLE_DONE pgs=2305 cs=0 l=1 rx=0x7faf40119c10 tx=0x7faf34010b20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:20.140 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.121+0000 7faf2f7fe700 1 --2- 172.21.15.42:0/1041434544 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7faf2403ac50 0x7faf2403d160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:14:20.141 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.121+0000 7faf2f7fe700 1 -- 172.21.15.42:0/1041434544 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 10245+0+0 (secure 0 0 0) 0x7faf3404af90 con 0x7faf40121490 2019-12-05T06:14:20.141 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.121+0000 7faf3ed9d700 1 --2- 172.21.15.42:0/1041434544 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7faf2403ac50 0x7faf2403d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:20.141 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.121+0000 7faf3ed9d700 1 --2- 172.21.15.42:0/1041434544 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7faf2403ac50 0x7faf2403d160 crc :-1 s=READY pgs=971 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:14:20.141 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.125+0000 7faf44e53700 1 -- 172.21.15.42:0/1041434544 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf4005d110 con 0x7faf40121490 2019-12-05T06:14:20.142 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.133+0000 7faf3e59c700 1 --2- 172.21.15.42:0/1041434544 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf40121490 0x7faf40118930 secure :-1 s=THROTTLE_DONE pgs=2305 cs=0 l=1 rx=0x7faf40119c10 tx=0x7faf34010b20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:20.142 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.133+0000 7faf2f7fe700 1 -- 172.21.15.42:0/1041434544 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7faf34032250 con 0x7faf40121490 2019-12-05T06:14:20.309 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.305+0000 7faf44e53700 1 -- 172.21.15.42:0/1041434544 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faf4005b8c0 con 0x7faf40121490 2019-12-05T06:14:20.310 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.305+0000 7faf3e59c700 1 --2- 172.21.15.42:0/1041434544 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf40121490 0x7faf40118930 secure :-1 s=THROTTLE_DONE pgs=2305 cs=0 l=1 rx=0x7faf40119c10 tx=0x7faf34010b20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:20.311 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.305+0000 7faf2f7fe700 1 -- 172.21.15.42:0/1041434544 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 278 v278) v1 ==== 95+0+2115 (secure 0 0 0) 0x7faf34032e00 con 0x7faf40121490 2019-12-05T06:14:20.311 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 278 2019-12-05T06:14:20.311 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:14:20.311 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":278,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7217,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/2667648531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2667648531},{"type":"v1","addr":"172.21.15.40:6833","nonce":2667648531}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":278}],"filesystems":[{"mdsmap":{"epoch":277,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:13:19.980391+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":98,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10246},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10246":{"gid":10246,"name":"a","rank":0,"incarnation":274,"state":"up:active","state_seq":17,"addr":"172.21.15.42:6835/4032647860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":4032647860},{"type":"v1","addr":"172.21.15.42:6835","nonce":4032647860}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:14:20.318 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.309+0000 7faf44e53700 1 -- 172.21.15.42:0/1041434544 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7faf2403ac50 msgr2=0x7faf2403d160 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:20.318 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.309+0000 7faf44e53700 1 --2- 172.21.15.42:0/1041434544 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7faf2403ac50 0x7faf2403d160 crc :-1 s=READY pgs=971 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:20.318 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.309+0000 7faf44e53700 1 -- 172.21.15.42:0/1041434544 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf40121490 msgr2=0x7faf40118930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:20.318 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.309+0000 7faf44e53700 1 --2- 172.21.15.42:0/1041434544 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf40121490 0x7faf40118930 secure :-1 s=READY pgs=2305 cs=0 l=1 rx=0x7faf40119c10 tx=0x7faf34010b20).stop 2019-12-05T06:14:20.319 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.309+0000 7faf44e53700 1 -- 172.21.15.42:0/1041434544 shutdown_connections 2019-12-05T06:14:20.319 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.309+0000 7faf44e53700 1 --2- 172.21.15.42:0/1041434544 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7faf2403ac50 0x7faf2403d160 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:20.319 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.309+0000 7faf44e53700 1 --2- 172.21.15.42:0/1041434544 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7faf40117620 0x7faf40126710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:20.319 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.309+0000 7faf44e53700 1 --2- 172.21.15.42:0/1041434544 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf40121490 0x7faf40118930 unknown :-1 s=CLOSED pgs=2305 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:20.319 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.309+0000 7faf44e53700 1 -- 172.21.15.42:0/1041434544 shutdown_connections 2019-12-05T06:14:20.319 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.309+0000 7faf44e53700 1 -- 172.21.15.42:0/1041434544 wait complete. 2019-12-05T06:14:20.320 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.309+0000 7faf44e53700 1 -- 172.21.15.42:0/1041434544 >> 172.21.15.42:0/1041434544 conn(0x7faf400a8940 msgr2=0x7faf4009f2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:14:20.325 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.a (rank=0) 2019-12-05T06:14:20.325 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2019-12-05T06:14:20.325 INFO:teuthology.orchestra.run:waiting for 300 2019-12-05T06:14:20.342 INFO:tasks.ceph.mds.a:Stopped 2019-12-05T06:14:20.342 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:14:20.511 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.501+0000 7f9ee8084700 1 Processor -- start 2019-12-05T06:14:20.512 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.501+0000 7f9ee8084700 1 -- start start 2019-12-05T06:14:20.512 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.501+0000 7f9ee8084700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9ee00a7740 0x7f9ee008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:14:20.512 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.501+0000 7f9ee8084700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ee00a0710 con 0x7f9ee009fae0 2019-12-05T06:14:20.512 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.501+0000 7f9ee8084700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ee00a9ba0 con 0x7f9ee007e720 2019-12-05T06:14:20.512 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.501+0000 7f9ee8084700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ee00a0b00 con 0x7f9ee00a7740 2019-12-05T06:14:20.512 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.501+0000 7f9ee5e20700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f9ee007e720 0x7f9ee009f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:51232/0 (socket says 172.21.15.42:51232) 2019-12-05T06:14:20.513 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.501+0000 7f9ee5e20700 1 -- 172.21.15.42:0/3783911784 learned_addr learned my addr 172.21.15.42:0/3783911784 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:14:20.513 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.501+0000 7f9ee6621700 1 --2- 172.21.15.42:0/3783911784 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9ee00a7740 0x7f9ee008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:20.513 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.501+0000 7f9ee6621700 1 -- 172.21.15.42:0/3783911784 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9ee00a7740 msgr2=0x7f9ee008df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2019-12-05T06:14:20.513 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.501+0000 7f9ee6621700 1 -- 172.21.15.42:0/3783911784 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9ee00a7740 msgr2=0x7f9ee008df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2019-12-05T06:14:20.513 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.501+0000 7f9ee6621700 1 --2- 172.21.15.42:0/3783911784 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9ee00a7740 0x7f9ee008df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-12-05T06:14:20.520 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.501+0000 7f9ee6621700 1 --2- 172.21.15.42:0/3783911784 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9ee00a7740 0x7f9ee008df80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rx=0 tx=0)._fault waiting 0.200000 2019-12-05T06:14:20.520 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.501+0000 7f9ee561f700 1 -- 172.21.15.42:0/3783911784 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3243182616 0 0) 0x7f9ee00a9ba0 con 0x7f9ee007e720 2019-12-05T06:14:20.521 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.501+0000 7f9ee561f700 1 -- 172.21.15.42:0/3783911784 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ec80030b0 con 0x7f9ee007e720 2019-12-05T06:14:20.521 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.501+0000 7f9ee561f700 1 -- 172.21.15.42:0/3783911784 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 823489609 0 0) 0x7f9ee00a0710 con 0x7f9ee009fae0 2019-12-05T06:14:20.521 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.501+0000 7f9ee561f700 1 -- 172.21.15.42:0/3783911784 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ee00a9ba0 con 0x7f9ee009fae0 2019-12-05T06:14:20.521 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.501+0000 7f9ee561f700 1 -- 172.21.15.42:0/3783911784 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 11141737 0 0) 0x7f9ec80030b0 con 0x7f9ee007e720 2019-12-05T06:14:20.521 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.501+0000 7f9ee561f700 1 -- 172.21.15.42:0/3783911784 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9ee00a7740 msgr2=0x7f9ee008df80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2019-12-05T06:14:20.521 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.501+0000 7f9ee561f700 1 --2- 172.21.15.42:0/3783911784 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9ee00a7740 0x7f9ee008df80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:14:20.521 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.501+0000 7f9ee561f700 1 -- 172.21.15.42:0/3783911784 >> v1:172.21.15.40:6789/0 conn(0x7f9ee009fae0 legacy=0x7f9ee014efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:20.522 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.501+0000 7f9ee561f700 1 -- 172.21.15.42:0/3783911784 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ee007eb50 con 0x7f9ee007e720 2019-12-05T06:14:20.522 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.501+0000 7f9ee561f700 1 -- 172.21.15.42:0/3783911784 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7f9ed0001ad0 con 0x7f9ee007e720 2019-12-05T06:14:20.522 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.501+0000 7f9ee561f700 1 -- 172.21.15.42:0/3783911784 >> v1:172.21.15.42:6789/0 conn(0x7f9ee007e720 legacy=0x7f9ee009f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:20.522 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.501+0000 7f9ee561f700 1 --2- 172.21.15.42:0/3783911784 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9ec8003940 0x7f9ec8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:14:20.522 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.501+0000 7f9ee561f700 1 --2- 172.21.15.42:0/3783911784 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9ec8004420 0x7f9ec8004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:14:20.522 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.501+0000 7f9ee6621700 1 --2- 172.21.15.42:0/3783911784 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9ec8003940 0x7f9ec8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:20.523 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.501+0000 7f9ee561f700 1 --2- 172.21.15.42:0/3783911784 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9ec8004e10 0x7f9ec804f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:14:20.523 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.501+0000 7f9ee561f700 1 -- 172.21.15.42:0/3783911784 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ee00a0b00 con 0x7f9ec8004e10 2019-12-05T06:14:20.523 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.501+0000 7f9ee561f700 1 -- 172.21.15.42:0/3783911784 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ec804fbd0 con 0x7f9ec8003940 2019-12-05T06:14:20.523 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.501+0000 7f9ee561f700 1 -- 172.21.15.42:0/3783911784 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ec804fdc0 con 0x7f9ec8004420 2019-12-05T06:14:20.523 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.505+0000 7f9ee5e20700 1 --2- 172.21.15.42:0/3783911784 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9ec8004420 0x7f9ec8004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:20.523 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.505+0000 7f9ee6e22700 1 --2- 172.21.15.42:0/3783911784 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9ec8004e10 0x7f9ec804f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:20.523 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.505+0000 7f9ee6621700 1 -- 172.21.15.42:0/3783911784 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9ec8004420 msgr2=0x7f9ec8004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:14:20.524 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.505+0000 7f9ee6621700 1 --2- 172.21.15.42:0/3783911784 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9ec8004420 0x7f9ec8004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:14:20.524 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.505+0000 7f9ee6621700 1 -- 172.21.15.42:0/3783911784 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9ec8004e10 msgr2=0x7f9ec804f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:14:20.524 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.505+0000 7f9ee6621700 1 --2- 172.21.15.42:0/3783911784 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9ec8004e10 0x7f9ec804f680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:14:20.524 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.505+0000 7f9ee6621700 1 -- 172.21.15.42:0/3783911784 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ec80500c0 con 0x7f9ec8003940 2019-12-05T06:14:20.524 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.505+0000 7f9ee6e22700 1 --2- 172.21.15.42:0/3783911784 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9ec8004e10 0x7f9ec804f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:14:20.524 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.505+0000 7f9ee6621700 1 --2- 172.21.15.42:0/3783911784 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9ec8003940 0x7f9ec8003d90 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f9edc002f30 tx=0x7f9edc00ff30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:20.525 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.505+0000 7f9ee6621700 1 --2- 172.21.15.42:0/3783911784 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9ec8003940 0x7f9ec8003d90 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f9edc002f30 tx=0x7f9edc00ff30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:20.525 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.505+0000 7f9ee6621700 1 --2- 172.21.15.42:0/3783911784 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9ec8003940 0x7f9ec8003d90 secure :-1 s=READY pgs=2307 cs=0 l=1 rx=0x7f9edc002f30 tx=0x7f9edc00ff30).ready entity=mon.1 client_cookie=452d1d1a14f2125 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:14:20.525 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.505+0000 7f9ee6621700 1 --2- 172.21.15.42:0/3783911784 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9ec8003940 0x7f9ec8003d90 secure :-1 s=THROTTLE_DONE pgs=2307 cs=0 l=1 rx=0x7f9edc002f30 tx=0x7f9edc00ff30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:20.525 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.505+0000 7f9ee6621700 1 --2- 172.21.15.42:0/3783911784 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9ec8003940 0x7f9ec8003d90 secure :-1 s=THROTTLE_DONE pgs=2307 cs=0 l=1 rx=0x7f9edc002f30 tx=0x7f9edc00ff30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:20.525 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.505+0000 7f9ee561f700 1 -- 172.21.15.42:0/3783911784 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9edc010d00 con 0x7f9ec8003940 2019-12-05T06:14:20.525 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.505+0000 7f9ee6621700 1 --2- 172.21.15.42:0/3783911784 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9ec8003940 0x7f9ec8003d90 secure :-1 s=THROTTLE_DONE pgs=2307 cs=0 l=1 rx=0x7f9edc002f30 tx=0x7f9edc00ff30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:20.526 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.505+0000 7f9ee561f700 1 -- 172.21.15.42:0/3783911784 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9edc010920 con 0x7f9ec8003940 2019-12-05T06:14:20.526 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.505+0000 7f9ee561f700 1 -- 172.21.15.42:0/3783911784 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9edc008b90 con 0x7f9ec8003940 2019-12-05T06:14:20.526 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.505+0000 7f9ee8084700 1 -- 172.21.15.42:0/3783911784 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9ec8003940 msgr2=0x7f9ec8003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:20.526 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.505+0000 7f9ee8084700 1 --2- 172.21.15.42:0/3783911784 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9ec8003940 0x7f9ec8003d90 secure :-1 s=READY pgs=2307 cs=0 l=1 rx=0x7f9edc002f30 tx=0x7f9edc00ff30).stop 2019-12-05T06:14:20.526 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.505+0000 7f9ee6e22700 1 -- 172.21.15.42:0/3783911784 reap_dead start 2019-12-05T06:14:20.527 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.505+0000 7f9ee8084700 1 -- 172.21.15.42:0/3783911784 shutdown_connections 2019-12-05T06:14:20.527 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.505+0000 7f9ee8084700 1 -- 172.21.15.42:0/3783911784 shutdown_connections 2019-12-05T06:14:20.527 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.505+0000 7f9ee8084700 1 -- 172.21.15.42:0/3783911784 wait complete. 2019-12-05T06:14:20.527 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.505+0000 7f9ee8084700 1 -- 172.21.15.42:0/3783911784 >> 172.21.15.42:0/3783911784 conn(0x7f9ee00aabe0 msgr2=0x7f9ee00a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:14:20.527 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.505+0000 7f9ee8084700 1 Processor -- start 2019-12-05T06:14:20.528 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.505+0000 7f9ee8084700 1 -- start start 2019-12-05T06:14:20.531 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.509+0000 7f9ee8084700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f9ee00a63f0 0x7f9ee012aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:14:20.531 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.509+0000 7f9ee8084700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9ee012afe0 0x7f9ee012b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:14:20.531 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.509+0000 7f9ee8084700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ee00a5900 con 0x7f9ee011d560 2019-12-05T06:14:20.531 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.509+0000 7f9ee8084700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ee011a630 con 0x7f9ee00a63f0 2019-12-05T06:14:20.531 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.509+0000 7f9ee8084700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ee011a8d0 con 0x7f9ee012afe0 2019-12-05T06:14:20.532 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.509+0000 7f9ee5e20700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f9ee00a63f0 0x7f9ee012aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:20.532 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.509+0000 7f9ee5e20700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f9ee00a63f0 0x7f9ee012aa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:51208/0 (socket says 172.21.15.42:51208) 2019-12-05T06:14:20.532 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.509+0000 7f9ee5e20700 1 -- 172.21.15.42:0/848780953 learned_addr learned my addr 172.21.15.42:0/848780953 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:14:20.532 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.509+0000 7f9ee6e22700 1 --2- 172.21.15.42:0/848780953 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9ee012afe0 0x7f9ee012b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:20.532 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.509+0000 7f9ee5e20700 1 -- 172.21.15.42:0/848780953 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9ee012afe0 msgr2=0x7f9ee012b410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:20.532 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.509+0000 7f9ee5e20700 1 --2- 172.21.15.42:0/848780953 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9ee012afe0 0x7f9ee012b410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:20.532 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.509+0000 7f9ee5e20700 1 -- 172.21.15.42:0/848780953 >> v1:172.21.15.40:6789/0 conn(0x7f9ee011d560 legacy=0x7f9ee00a5100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:20.533 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.509+0000 7f9ee5e20700 1 -- 172.21.15.42:0/848780953 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ee00a4840 con 0x7f9ee00a63f0 2019-12-05T06:14:20.533 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.509+0000 7f9ee5e20700 1 --2- 172.21.15.42:0/848780953 >> v2:172.21.15.42:3300/0 conn(0x7f9ee00a63f0 0x7f9ee012aa90 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f9ee011a7f0 tx=0x7f9ed0010ec0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:20.533 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.509+0000 7f9ee5e20700 1 --2- 172.21.15.42:0/848780953 >> v2:172.21.15.42:3300/0 conn(0x7f9ee00a63f0 0x7f9ee012aa90 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f9ee011a7f0 tx=0x7f9ed0010ec0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:20.533 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.509+0000 7f9ee5e20700 1 --2- 172.21.15.42:0/848780953 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9ee00a63f0 0x7f9ee012aa90 secure :-1 s=READY pgs=2308 cs=0 l=1 rx=0x7f9ee011a7f0 tx=0x7f9ed0010ec0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:14:20.533 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.509+0000 7f9ee5e20700 1 --2- 172.21.15.42:0/848780953 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9ee00a63f0 0x7f9ee012aa90 secure :-1 s=THROTTLE_DONE pgs=2308 cs=0 l=1 rx=0x7f9ee011a7f0 tx=0x7f9ed0010ec0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:20.533 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.509+0000 7f9ee5e20700 1 --2- 172.21.15.42:0/848780953 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9ee00a63f0 0x7f9ee012aa90 secure :-1 s=THROTTLE_DONE pgs=2308 cs=0 l=1 rx=0x7f9ee011a7f0 tx=0x7f9ed0010ec0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:20.533 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.509+0000 7f9ee5e20700 1 --2- 172.21.15.42:0/848780953 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9ee00a63f0 0x7f9ee012aa90 secure :-1 s=THROTTLE_DONE pgs=2308 cs=0 l=1 rx=0x7f9ee011a7f0 tx=0x7f9ed0010ec0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:20.534 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.509+0000 7f9ed6ffd700 1 -- 172.21.15.42:0/848780953 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9ed0011ca0 con 0x7f9ee00a63f0 2019-12-05T06:14:20.534 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.509+0000 7f9ed6ffd700 1 -- 172.21.15.42:0/848780953 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ed00118c0 con 0x7f9ee00a63f0 2019-12-05T06:14:20.534 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.509+0000 7f9ee8084700 1 -- 172.21.15.42:0/848780953 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ee011b0b0 con 0x7f9ee00a63f0 2019-12-05T06:14:20.534 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.509+0000 7f9ed6ffd700 1 -- 172.21.15.42:0/848780953 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9ed0004b60 con 0x7f9ee00a63f0 2019-12-05T06:14:20.534 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.509+0000 7f9ee8084700 1 -- 172.21.15.42:0/848780953 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ee00a6820 con 0x7f9ee00a63f0 2019-12-05T06:14:20.534 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.509+0000 7f9ee5e20700 1 --2- 172.21.15.42:0/848780953 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9ee00a63f0 0x7f9ee012aa90 secure :-1 s=THROTTLE_DONE pgs=2308 cs=0 l=1 rx=0x7f9ee011a7f0 tx=0x7f9ed0010ec0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:20.534 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.509+0000 7f9ed6ffd700 1 -- 172.21.15.42:0/848780953 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f9ed000f070 con 0x7f9ee00a63f0 2019-12-05T06:14:20.535 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.509+0000 7f9ee5e20700 1 --2- 172.21.15.42:0/848780953 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9ee00a63f0 0x7f9ee012aa90 secure :-1 s=THROTTLE_DONE pgs=2308 cs=0 l=1 rx=0x7f9ee011a7f0 tx=0x7f9ed0010ec0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:20.535 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.513+0000 7f9ed6ffd700 1 --2- 172.21.15.42:0/848780953 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9ec003ad10 0x7f9ec003d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:14:20.535 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.513+0000 7f9ed6ffd700 1 -- 172.21.15.42:0/848780953 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 10245+0+0 (secure 0 0 0) 0x7f9ed004b1c0 con 0x7f9ee00a63f0 2019-12-05T06:14:20.535 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.513+0000 7f9ee6621700 1 --2- 172.21.15.42:0/848780953 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9ec003ad10 0x7f9ec003d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:20.535 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.513+0000 7f9ee6621700 1 --2- 172.21.15.42:0/848780953 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9ec003ad10 0x7f9ec003d220 crc :-1 s=READY pgs=972 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:14:20.535 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.517+0000 7f9ee8084700 1 -- 172.21.15.42:0/848780953 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ee005d110 con 0x7f9ee00a63f0 2019-12-05T06:14:20.535 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.525+0000 7f9ee5e20700 1 --2- 172.21.15.42:0/848780953 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9ee00a63f0 0x7f9ee012aa90 secure :-1 s=THROTTLE_DONE pgs=2308 cs=0 l=1 rx=0x7f9ee011a7f0 tx=0x7f9ed0010ec0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:20.536 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.525+0000 7f9ed6ffd700 1 -- 172.21.15.42:0/848780953 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f9ed0032240 con 0x7f9ee00a63f0 2019-12-05T06:14:20.691 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.685+0000 7f9ee8084700 1 -- 172.21.15.42:0/848780953 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9ee005b8c0 con 0x7f9ee00a63f0 2019-12-05T06:14:20.692 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.689+0000 7f9ee5e20700 1 --2- 172.21.15.42:0/848780953 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9ee00a63f0 0x7f9ee012aa90 secure :-1 s=THROTTLE_DONE pgs=2308 cs=0 l=1 rx=0x7f9ee011a7f0 tx=0x7f9ed0010ec0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:20.692 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.689+0000 7f9ed6ffd700 1 -- 172.21.15.42:0/848780953 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 278 v278) v1 ==== 95+0+2115 (secure 0 0 0) 0x7f9ed00324a0 con 0x7f9ee00a63f0 2019-12-05T06:14:20.694 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:14:20.694 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":278,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7217,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/2667648531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2667648531},{"type":"v1","addr":"172.21.15.40:6833","nonce":2667648531}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":278}],"filesystems":[{"mdsmap":{"epoch":277,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:13:19.980391+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":98,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10246},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10246":{"gid":10246,"name":"a","rank":0,"incarnation":274,"state":"up:active","state_seq":17,"addr":"172.21.15.42:6835/4032647860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":4032647860},{"type":"v1","addr":"172.21.15.42:6835","nonce":4032647860}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:14:20.695 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 278 2019-12-05T06:14:20.697 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.693+0000 7f9ee8084700 1 -- 172.21.15.42:0/848780953 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9ec003ad10 msgr2=0x7f9ec003d220 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:20.697 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.693+0000 7f9ee8084700 1 --2- 172.21.15.42:0/848780953 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9ec003ad10 0x7f9ec003d220 crc :-1 s=READY pgs=972 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:20.703 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.693+0000 7f9ee8084700 1 -- 172.21.15.42:0/848780953 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9ee00a63f0 msgr2=0x7f9ee012aa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:20.704 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.693+0000 7f9ee8084700 1 --2- 172.21.15.42:0/848780953 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9ee00a63f0 0x7f9ee012aa90 secure :-1 s=READY pgs=2308 cs=0 l=1 rx=0x7f9ee011a7f0 tx=0x7f9ed0010ec0).stop 2019-12-05T06:14:20.704 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.693+0000 7f9ee8084700 1 -- 172.21.15.42:0/848780953 shutdown_connections 2019-12-05T06:14:20.704 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.693+0000 7f9ee8084700 1 --2- 172.21.15.42:0/848780953 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9ec003ad10 0x7f9ec003d220 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:20.704 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.693+0000 7f9ee8084700 1 --2- 172.21.15.42:0/848780953 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9ee012afe0 0x7f9ee012b410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:20.704 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.693+0000 7f9ee8084700 1 --2- 172.21.15.42:0/848780953 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9ee00a63f0 0x7f9ee012aa90 unknown :-1 s=CLOSED pgs=2308 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:20.704 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.693+0000 7f9ee8084700 1 -- 172.21.15.42:0/848780953 shutdown_connections 2019-12-05T06:14:20.704 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.693+0000 7f9ee8084700 1 -- 172.21.15.42:0/848780953 wait complete. 2019-12-05T06:14:20.705 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:20.693+0000 7f9ee8084700 1 -- 172.21.15.42:0/848780953 >> 172.21.15.42:0/848780953 conn(0x7f9ee00aabe0 msgr2=0x7f9ee011b460 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:14:20.707 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:14:22.707 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:14:22.815 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:22.805+0000 7fb86d73c700 1 Processor -- start 2019-12-05T06:14:22.815 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:22.805+0000 7fb86d73c700 1 -- start start 2019-12-05T06:14:22.815 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:22.805+0000 7fb86d73c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb86808df80 0x7fb86809f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:14:22.815 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:22.805+0000 7fb86d73c700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb86809f830 0x7fb86814efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:14:22.815 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:22.805+0000 7fb86d73c700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb86809fc60 con 0x7fb8680a7740 2019-12-05T06:14:22.816 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:22.805+0000 7fb86d73c700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8680a0b00 con 0x7fb86808df80 2019-12-05T06:14:22.816 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:22.805+0000 7fb86d73c700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb86808e5b0 con 0x7fb86809f830 2019-12-05T06:14:22.816 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:22.805+0000 7fb866ffd700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb86808df80 0x7fb86809f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:22.816 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:22.805+0000 7fb866ffd700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb86808df80 0x7fb86809f2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:51224/0 (socket says 172.21.15.42:51224) 2019-12-05T06:14:22.816 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:22.805+0000 7fb866ffd700 1 -- 172.21.15.42:0/2300086501 learned_addr learned my addr 172.21.15.42:0/2300086501 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:14:22.816 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:22.805+0000 7fb8677fe700 1 --1- 172.21.15.42:0/2300086501 >> v1:172.21.15.42:6789/0 conn(0x7fb8680a7740 0x7fb86807e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:51260/0 (socket says 172.21.15.42:51260) 2019-12-05T06:14:22.816 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:22.805+0000 7fb867fff700 1 --2- 172.21.15.42:0/2300086501 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb86809f830 0x7fb86814efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:22.817 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:22.805+0000 7fb866ffd700 1 -- 172.21.15.42:0/2300086501 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb86809f830 msgr2=0x7fb86814efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:14:22.817 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:22.805+0000 7fb866ffd700 1 --2- 172.21.15.42:0/2300086501 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb86809f830 0x7fb86814efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:14:22.817 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:22.805+0000 7fb866ffd700 1 -- 172.21.15.42:0/2300086501 >> v1:172.21.15.42:6789/0 conn(0x7fb8680a7740 legacy=0x7fb86807e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:22.817 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:22.805+0000 7fb866ffd700 1 -- 172.21.15.42:0/2300086501 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8680a0710 con 0x7fb86808df80 2019-12-05T06:14:22.817 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:22.805+0000 7fb8667fc700 1 -- 172.21.15.42:0/2300086501 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4001090568 0 0) 0x7fb86809fc60 con 0x7fb8680a7740 2019-12-05T06:14:22.817 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:22.805+0000 7fb866ffd700 1 --2- 172.21.15.42:0/2300086501 >> v2:172.21.15.42:3300/0 conn(0x7fb86808df80 0x7fb86809f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fb850008b10 tx=0x7fb8500100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:22.817 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:22.805+0000 7fb866ffd700 1 --2- 172.21.15.42:0/2300086501 >> v2:172.21.15.42:3300/0 conn(0x7fb86808df80 0x7fb86809f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fb850008b10 tx=0x7fb8500100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:22.818 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:22.805+0000 7fb866ffd700 1 --2- 172.21.15.42:0/2300086501 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb86808df80 0x7fb86809f2e0 secure :-1 s=READY pgs=2313 cs=0 l=1 rx=0x7fb850008b10 tx=0x7fb8500100b0).ready entity=mon.1 client_cookie=6d0bcff490d675a server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:14:22.818 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:22.805+0000 7fb866ffd700 1 --2- 172.21.15.42:0/2300086501 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb86808df80 0x7fb86809f2e0 secure :-1 s=THROTTLE_DONE pgs=2313 cs=0 l=1 rx=0x7fb850008b10 tx=0x7fb8500100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:22.818 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:22.805+0000 7fb8667fc700 1 -- 172.21.15.42:0/2300086501 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb850010da0 con 0x7fb86808df80 2019-12-05T06:14:22.818 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:22.809+0000 7fb866ffd700 1 --2- 172.21.15.42:0/2300086501 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb86808df80 0x7fb86809f2e0 secure :-1 s=THROTTLE_DONE pgs=2313 cs=0 l=1 rx=0x7fb850008b10 tx=0x7fb8500100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:22.818 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:22.809+0000 7fb866ffd700 1 --2- 172.21.15.42:0/2300086501 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb86808df80 0x7fb86809f2e0 secure :-1 s=THROTTLE_DONE pgs=2313 cs=0 l=1 rx=0x7fb850008b10 tx=0x7fb8500100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:22.818 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:22.809+0000 7fb8667fc700 1 -- 172.21.15.42:0/2300086501 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8500109c0 con 0x7fb86808df80 2019-12-05T06:14:22.818 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:22.809+0000 7fb8667fc700 1 -- 172.21.15.42:0/2300086501 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb8500114a0 con 0x7fb86808df80 2019-12-05T06:14:22.819 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:22.809+0000 7fb86d73c700 1 -- 172.21.15.42:0/2300086501 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb86808df80 msgr2=0x7fb86809f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:22.819 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:22.809+0000 7fb86d73c700 1 --2- 172.21.15.42:0/2300086501 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb86808df80 0x7fb86809f2e0 secure :-1 s=READY pgs=2313 cs=0 l=1 rx=0x7fb850008b10 tx=0x7fb8500100b0).stop 2019-12-05T06:14:22.819 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:22.809+0000 7fb86d73c700 1 -- 172.21.15.42:0/2300086501 shutdown_connections 2019-12-05T06:14:22.819 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:22.809+0000 7fb86d73c700 1 --2- 172.21.15.42:0/2300086501 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb86809f830 0x7fb86814efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:14:22.819 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:22.809+0000 7fb86d73c700 1 --2- 172.21.15.42:0/2300086501 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb86808df80 0x7fb86809f2e0 unknown :-1 s=CLOSED pgs=2313 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:22.819 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:22.809+0000 7fb86d73c700 1 -- 172.21.15.42:0/2300086501 shutdown_connections 2019-12-05T06:14:22.819 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:22.809+0000 7fb86d73c700 1 -- 172.21.15.42:0/2300086501 wait complete. 2019-12-05T06:14:22.820 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:22.809+0000 7fb86d73c700 1 -- 172.21.15.42:0/2300086501 >> 172.21.15.42:0/2300086501 conn(0x7fb8680aabe0 msgr2=0x7fb8680a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:14:22.820 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:22.809+0000 7fb86d73c700 1 Processor -- start 2019-12-05T06:14:22.820 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:22.809+0000 7fb86d73c700 1 -- start start 2019-12-05T06:14:22.823 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:22.809+0000 7fb86d73c700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb8680a6380 0x7fb8680a4b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:14:22.824 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:22.809+0000 7fb86d73c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb86812ea70 0x7fb8680742d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:14:22.824 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:22.809+0000 7fb86d73c700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8680a5770 con 0x7fb86811d6b0 2019-12-05T06:14:22.824 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:22.809+0000 7fb86d73c700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8680a5230 con 0x7fb86812ea70 2019-12-05T06:14:22.824 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:22.809+0000 7fb86d73c700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8680a54d0 con 0x7fb8680a6380 2019-12-05T06:14:22.824 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:22.813+0000 7fb867fff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb86812ea70 0x7fb8680742d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:22.825 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:22.813+0000 7fb867fff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb86812ea70 0x7fb8680742d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:51230/0 (socket says 172.21.15.42:51230) 2019-12-05T06:14:22.825 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:22.813+0000 7fb867fff700 1 -- 172.21.15.42:0/3526448028 learned_addr learned my addr 172.21.15.42:0/3526448028 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:14:22.825 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:22.813+0000 7fb8677fe700 1 --2- 172.21.15.42:0/3526448028 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb8680a6380 0x7fb8680a4b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:22.825 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:22.813+0000 7fb8677fe700 1 -- 172.21.15.42:0/3526448028 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb8680a6380 msgr2=0x7fb8680a4b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2019-12-05T06:14:22.825 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:22.813+0000 7fb8677fe700 1 -- 172.21.15.42:0/3526448028 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb8680a6380 msgr2=0x7fb8680a4b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2019-12-05T06:14:22.825 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:22.813+0000 7fb8677fe700 1 --2- 172.21.15.42:0/3526448028 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb8680a6380 0x7fb8680a4b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-12-05T06:14:22.825 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:22.813+0000 7fb8677fe700 1 --2- 172.21.15.42:0/3526448028 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb8680a6380 0x7fb8680a4b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rx=0 tx=0)._fault waiting 0.200000 2019-12-05T06:14:22.826 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:22.813+0000 7fb867fff700 1 -- 172.21.15.42:0/3526448028 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb8680a6380 msgr2=0x7fb8680a4b20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2019-12-05T06:14:22.826 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:22.813+0000 7fb867fff700 1 --2- 172.21.15.42:0/3526448028 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb8680a6380 0x7fb8680a4b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:22.826 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:22.813+0000 7fb867fff700 1 -- 172.21.15.42:0/3526448028 >> v1:172.21.15.40:6789/0 conn(0x7fb86811d6b0 legacy=0x7fb86812e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:22.826 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:22.813+0000 7fb85ffff700 1 -- 172.21.15.42:0/3526448028 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3869980291 0 0) 0x7fb8500087f0 con 0x7fb86811d6b0 2019-12-05T06:14:22.826 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:22.813+0000 7fb867fff700 1 -- 172.21.15.42:0/3526448028 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb86813ee50 con 0x7fb86812ea70 2019-12-05T06:14:22.826 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:22.813+0000 7fb867fff700 1 --2- 172.21.15.42:0/3526448028 >> v2:172.21.15.42:3300/0 conn(0x7fb86812ea70 0x7fb8680742d0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fb8680a53f0 tx=0x7fb860010990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:22.826 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:22.813+0000 7fb867fff700 1 --2- 172.21.15.42:0/3526448028 >> v2:172.21.15.42:3300/0 conn(0x7fb86812ea70 0x7fb8680742d0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fb8680a53f0 tx=0x7fb860010990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:22.827 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:22.813+0000 7fb867fff700 1 --2- 172.21.15.42:0/3526448028 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb86812ea70 0x7fb8680742d0 secure :-1 s=READY pgs=2314 cs=0 l=1 rx=0x7fb8680a53f0 tx=0x7fb860010990).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:14:22.827 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:22.813+0000 7fb867fff700 1 --2- 172.21.15.42:0/3526448028 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb86812ea70 0x7fb8680742d0 secure :-1 s=THROTTLE_DONE pgs=2314 cs=0 l=1 rx=0x7fb8680a53f0 tx=0x7fb860010990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:22.827 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:22.813+0000 7fb85ffff700 1 -- 172.21.15.42:0/3526448028 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb860005930 con 0x7fb86812ea70 2019-12-05T06:14:22.827 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:22.813+0000 7fb867fff700 1 --2- 172.21.15.42:0/3526448028 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb86812ea70 0x7fb8680742d0 secure :-1 s=THROTTLE_DONE pgs=2314 cs=0 l=1 rx=0x7fb8680a53f0 tx=0x7fb860010990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:22.827 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:22.813+0000 7fb85ffff700 1 -- 172.21.15.42:0/3526448028 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb860007200 con 0x7fb86812ea70 2019-12-05T06:14:22.827 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:22.813+0000 7fb867fff700 1 --2- 172.21.15.42:0/3526448028 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb86812ea70 0x7fb8680742d0 secure :-1 s=THROTTLE_DONE pgs=2314 cs=0 l=1 rx=0x7fb8680a53f0 tx=0x7fb860010990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:22.827 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:22.813+0000 7fb86d73c700 1 -- 172.21.15.42:0/3526448028 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb86811dae0 con 0x7fb86812ea70 2019-12-05T06:14:22.827 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:22.813+0000 7fb85ffff700 1 -- 172.21.15.42:0/3526448028 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb8600060c0 con 0x7fb86812ea70 2019-12-05T06:14:22.828 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:22.813+0000 7fb86d73c700 1 -- 172.21.15.42:0/3526448028 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb868074a40 con 0x7fb86812ea70 2019-12-05T06:14:22.828 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:22.813+0000 7fb867fff700 1 --2- 172.21.15.42:0/3526448028 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb86812ea70 0x7fb8680742d0 secure :-1 s=THROTTLE_DONE pgs=2314 cs=0 l=1 rx=0x7fb8680a53f0 tx=0x7fb860010990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:22.828 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:22.813+0000 7fb85ffff700 1 -- 172.21.15.42:0/3526448028 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fb860006360 con 0x7fb86812ea70 2019-12-05T06:14:22.828 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:22.813+0000 7fb867fff700 1 --2- 172.21.15.42:0/3526448028 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb86812ea70 0x7fb8680742d0 secure :-1 s=THROTTLE_DONE pgs=2314 cs=0 l=1 rx=0x7fb8680a53f0 tx=0x7fb860010990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:22.828 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:22.813+0000 7fb85ffff700 1 --2- 172.21.15.42:0/3526448028 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb84003ac40 0x7fb84003d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:14:22.828 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:22.817+0000 7fb85ffff700 1 -- 172.21.15.42:0/3526448028 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 10245+0+0 (secure 0 0 0) 0x7fb86004ad50 con 0x7fb86812ea70 2019-12-05T06:14:22.828 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:22.817+0000 7fb8677fe700 1 --2- 172.21.15.42:0/3526448028 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb84003ac40 0x7fb84003d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:22.829 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:22.817+0000 7fb8677fe700 1 --2- 172.21.15.42:0/3526448028 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb84003ac40 0x7fb84003d150 crc :-1 s=READY pgs=973 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:14:22.829 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:22.821+0000 7fb86d73c700 1 -- 172.21.15.42:0/3526448028 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb86805d110 con 0x7fb86812ea70 2019-12-05T06:14:22.833 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:22.829+0000 7fb867fff700 1 --2- 172.21.15.42:0/3526448028 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb86812ea70 0x7fb8680742d0 secure :-1 s=THROTTLE_DONE pgs=2314 cs=0 l=1 rx=0x7fb8680a53f0 tx=0x7fb860010990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:22.833 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:22.829+0000 7fb85ffff700 1 -- 172.21.15.42:0/3526448028 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fb860031d50 con 0x7fb86812ea70 2019-12-05T06:14:23.014 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:23.009+0000 7fb86d73c700 1 -- 172.21.15.42:0/3526448028 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb86805b8c0 con 0x7fb86812ea70 2019-12-05T06:14:23.015 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:23.009+0000 7fb867fff700 1 --2- 172.21.15.42:0/3526448028 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb86812ea70 0x7fb8680742d0 secure :-1 s=THROTTLE_DONE pgs=2314 cs=0 l=1 rx=0x7fb8680a53f0 tx=0x7fb860010990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:23.015 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:23.009+0000 7fb85ffff700 1 -- 172.21.15.42:0/3526448028 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 278 v278) v1 ==== 95+0+2115 (secure 0 0 0) 0x7fb860031fb0 con 0x7fb86812ea70 2019-12-05T06:14:23.015 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 278 2019-12-05T06:14:23.015 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:14:23.015 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":278,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7217,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/2667648531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2667648531},{"type":"v1","addr":"172.21.15.40:6833","nonce":2667648531}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":278}],"filesystems":[{"mdsmap":{"epoch":277,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:13:19.980391+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":98,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10246},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10246":{"gid":10246,"name":"a","rank":0,"incarnation":274,"state":"up:active","state_seq":17,"addr":"172.21.15.42:6835/4032647860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":4032647860},{"type":"v1","addr":"172.21.15.42:6835","nonce":4032647860}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:14:23.019 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:23.013+0000 7fb86d73c700 1 -- 172.21.15.42:0/3526448028 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb84003ac40 msgr2=0x7fb84003d150 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:23.019 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:23.013+0000 7fb86d73c700 1 --2- 172.21.15.42:0/3526448028 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb84003ac40 0x7fb84003d150 crc :-1 s=READY pgs=973 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:23.019 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:23.013+0000 7fb86d73c700 1 -- 172.21.15.42:0/3526448028 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb86812ea70 msgr2=0x7fb8680742d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:23.020 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:23.013+0000 7fb86d73c700 1 --2- 172.21.15.42:0/3526448028 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb86812ea70 0x7fb8680742d0 secure :-1 s=READY pgs=2314 cs=0 l=1 rx=0x7fb8680a53f0 tx=0x7fb860010990).stop 2019-12-05T06:14:23.020 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:23.013+0000 7fb86d73c700 1 -- 172.21.15.42:0/3526448028 shutdown_connections 2019-12-05T06:14:23.020 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:23.013+0000 7fb86d73c700 1 --2- 172.21.15.42:0/3526448028 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb84003ac40 0x7fb84003d150 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:23.020 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:23.013+0000 7fb86d73c700 1 --2- 172.21.15.42:0/3526448028 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb86812ea70 0x7fb8680742d0 unknown :-1 s=CLOSED pgs=2314 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:23.020 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:23.013+0000 7fb86d73c700 1 --2- 172.21.15.42:0/3526448028 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb8680a6380 0x7fb8680a4b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:23.021 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:23.013+0000 7fb86d73c700 1 -- 172.21.15.42:0/3526448028 shutdown_connections 2019-12-05T06:14:23.021 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:23.013+0000 7fb86d73c700 1 -- 172.21.15.42:0/3526448028 wait complete. 2019-12-05T06:14:23.021 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:23.013+0000 7fb86d73c700 1 -- 172.21.15.42:0/3526448028 >> 172.21.15.42:0/3526448028 conn(0x7fb8680a9840 msgr2=0x7fb86807e870 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:14:23.031 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:14:25.033 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:14:25.138 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:25.129+0000 7fb0e0667700 1 Processor -- start 2019-12-05T06:14:25.139 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:25.129+0000 7fb0e0667700 1 -- start start 2019-12-05T06:14:25.139 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:25.129+0000 7fb0e0667700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb0d80a7740 0x7fb0d808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:14:25.139 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:25.129+0000 7fb0e0667700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0d80a0710 con 0x7fb0d809fae0 2019-12-05T06:14:25.140 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:25.129+0000 7fb0e0667700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0d80a9ba0 con 0x7fb0d807e720 2019-12-05T06:14:25.140 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:25.129+0000 7fb0e0667700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0d80a0b00 con 0x7fb0d80a7740 2019-12-05T06:14:25.140 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:25.129+0000 7fb0de403700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fb0d807e720 0x7fb0d809f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:51278/0 (socket says 172.21.15.42:51278) 2019-12-05T06:14:25.140 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:25.129+0000 7fb0de403700 1 -- 172.21.15.42:0/2880701969 learned_addr learned my addr 172.21.15.42:0/2880701969 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:14:25.140 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:25.129+0000 7fb0dec04700 1 --2- 172.21.15.42:0/2880701969 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb0d80a7740 0x7fb0d808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:25.141 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:25.129+0000 7fb0ddc02700 1 -- 172.21.15.42:0/2880701969 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1366042603 0 0) 0x7fb0d80a9ba0 con 0x7fb0d807e720 2019-12-05T06:14:25.141 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:25.129+0000 7fb0ddc02700 1 -- 172.21.15.42:0/2880701969 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0c00030b0 con 0x7fb0d807e720 2019-12-05T06:14:25.141 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:25.129+0000 7fb0ddc02700 1 -- 172.21.15.42:0/2880701969 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 865632654 0 0) 0x7fb0d80a0710 con 0x7fb0d809fae0 2019-12-05T06:14:25.141 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:25.129+0000 7fb0ddc02700 1 -- 172.21.15.42:0/2880701969 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0d80a9ba0 con 0x7fb0d809fae0 2019-12-05T06:14:25.141 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:25.129+0000 7fb0dec04700 1 -- 172.21.15.42:0/2880701969 >> v1:172.21.15.42:6789/0 conn(0x7fb0d807e720 legacy=0x7fb0d809f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:25.141 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:25.133+0000 7fb0dec04700 1 -- 172.21.15.42:0/2880701969 >> v1:172.21.15.40:6789/0 conn(0x7fb0d809fae0 legacy=0x7fb0d814efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:25.142 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:25.133+0000 7fb0dec04700 1 -- 172.21.15.42:0/2880701969 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0d807eb50 con 0x7fb0d80a7740 2019-12-05T06:14:25.142 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:25.133+0000 7fb0dec04700 1 --2- 172.21.15.42:0/2880701969 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb0d80a7740 0x7fb0d808df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fb0c8002f30 tx=0x7fb0c800fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:25.142 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:25.133+0000 7fb0dec04700 1 --2- 172.21.15.42:0/2880701969 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb0d80a7740 0x7fb0d808df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fb0c8002f30 tx=0x7fb0c800fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:25.142 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:25.133+0000 7fb0dec04700 1 --2- 172.21.15.42:0/2880701969 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb0d80a7740 0x7fb0d808df80 secure :-1 s=READY pgs=506 cs=0 l=1 rx=0x7fb0c8002f30 tx=0x7fb0c800fbf0).ready entity=mon.2 client_cookie=efe053b2a1b05149 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:14:25.142 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:25.133+0000 7fb0dec04700 1 --2- 172.21.15.42:0/2880701969 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb0d80a7740 0x7fb0d808df80 secure :-1 s=THROTTLE_DONE pgs=506 cs=0 l=1 rx=0x7fb0c8002f30 tx=0x7fb0c800fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:25.142 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:25.133+0000 7fb0ddc02700 1 -- 172.21.15.42:0/2880701969 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb0c80112b0 con 0x7fb0d80a7740 2019-12-05T06:14:25.142 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:25.133+0000 7fb0dec04700 1 --2- 172.21.15.42:0/2880701969 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb0d80a7740 0x7fb0d808df80 secure :-1 s=THROTTLE_DONE pgs=506 cs=0 l=1 rx=0x7fb0c8002f30 tx=0x7fb0c800fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:25.143 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:25.133+0000 7fb0ddc02700 1 -- 172.21.15.42:0/2880701969 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0c8008bc0 con 0x7fb0d80a7740 2019-12-05T06:14:25.143 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:25.133+0000 7fb0dec04700 1 --2- 172.21.15.42:0/2880701969 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb0d80a7740 0x7fb0d808df80 secure :-1 s=THROTTLE_DONE pgs=506 cs=0 l=1 rx=0x7fb0c8002f30 tx=0x7fb0c800fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:25.143 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:25.133+0000 7fb0ddc02700 1 -- 172.21.15.42:0/2880701969 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb0c8011890 con 0x7fb0d80a7740 2019-12-05T06:14:25.143 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:25.133+0000 7fb0e0667700 1 -- 172.21.15.42:0/2880701969 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb0d80a7740 msgr2=0x7fb0d808df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:25.143 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:25.133+0000 7fb0e0667700 1 --2- 172.21.15.42:0/2880701969 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb0d80a7740 0x7fb0d808df80 secure :-1 s=READY pgs=506 cs=0 l=1 rx=0x7fb0c8002f30 tx=0x7fb0c800fbf0).stop 2019-12-05T06:14:25.143 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:25.133+0000 7fb0e0667700 1 -- 172.21.15.42:0/2880701969 shutdown_connections 2019-12-05T06:14:25.144 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:25.133+0000 7fb0e0667700 1 --2- 172.21.15.42:0/2880701969 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb0d80a7740 0x7fb0d808df80 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:25.144 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:25.133+0000 7fb0e0667700 1 -- 172.21.15.42:0/2880701969 shutdown_connections 2019-12-05T06:14:25.144 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:25.133+0000 7fb0e0667700 1 -- 172.21.15.42:0/2880701969 wait complete. 2019-12-05T06:14:25.144 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:25.133+0000 7fb0e0667700 1 -- 172.21.15.42:0/2880701969 >> 172.21.15.42:0/2880701969 conn(0x7fb0d80aabe0 msgr2=0x7fb0d80a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:14:25.150 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:25.133+0000 7fb0e0667700 1 Processor -- start 2019-12-05T06:14:25.151 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:25.133+0000 7fb0e0667700 1 -- start start 2019-12-05T06:14:25.151 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:25.137+0000 7fb0e0667700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb0d807cfb0 0x7fb0d814a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:14:25.151 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:25.137+0000 7fb0e0667700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb0d814aa40 0x7fb0d808f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:14:25.151 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:25.137+0000 7fb0e0667700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0d80a5f50 con 0x7fb0d80a5b20 2019-12-05T06:14:25.151 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:25.137+0000 7fb0e0667700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0d814ae70 con 0x7fb0d807cfb0 2019-12-05T06:14:25.151 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:25.137+0000 7fb0e0667700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0d814a6a0 con 0x7fb0d814aa40 2019-12-05T06:14:25.152 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:25.137+0000 7fb0df405700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fb0d80a5b20 0x7fb0d81184a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:51284/0 (socket says 172.21.15.42:51284) 2019-12-05T06:14:25.152 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:25.137+0000 7fb0df405700 1 -- 172.21.15.42:0/2410449626 learned_addr learned my addr 172.21.15.42:0/2410449626 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:14:25.152 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:25.137+0000 7fb0de403700 1 --2- 172.21.15.42:0/2410449626 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb0d814aa40 0x7fb0d808f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:25.152 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:25.137+0000 7fb0dec04700 1 --2- 172.21.15.42:0/2410449626 >> v2:172.21.15.40:3300/0 conn(0x7fb0d807cfb0 0x7fb0d814a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:25.152 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:25.137+0000 7fb0cf7fe700 1 -- 172.21.15.42:0/2410449626 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3783391187 0 0) 0x7fb0d80a5f50 con 0x7fb0d80a5b20 2019-12-05T06:14:25.153 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:25.137+0000 7fb0cf7fe700 1 -- 172.21.15.42:0/2410449626 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0b80030b0 con 0x7fb0d80a5b20 2019-12-05T06:14:25.153 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:25.137+0000 7fb0cf7fe700 1 -- 172.21.15.42:0/2410449626 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 797353884 0 0) 0x7fb0b80030b0 con 0x7fb0d80a5b20 2019-12-05T06:14:25.153 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:25.137+0000 7fb0dec04700 1 -- 172.21.15.42:0/2410449626 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb0d814aa40 msgr2=0x7fb0d808f5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:25.153 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:25.137+0000 7fb0dec04700 1 --2- 172.21.15.42:0/2410449626 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb0d814aa40 0x7fb0d808f5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:25.154 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:25.137+0000 7fb0dec04700 1 -- 172.21.15.42:0/2410449626 >> v1:172.21.15.42:6789/0 conn(0x7fb0d80a5b20 legacy=0x7fb0d81184a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:25.154 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:25.137+0000 7fb0dec04700 1 -- 172.21.15.42:0/2410449626 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0d8118ca0 con 0x7fb0d807cfb0 2019-12-05T06:14:25.154 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:25.137+0000 7fb0de403700 1 --2- 172.21.15.42:0/2410449626 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb0d814aa40 0x7fb0d808f5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:14:25.154 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:25.137+0000 7fb0dec04700 1 --2- 172.21.15.42:0/2410449626 >> v2:172.21.15.40:3300/0 conn(0x7fb0d807cfb0 0x7fb0d814a070 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fb0d0001b70 tx=0x7fb0d0001bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:25.154 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:25.137+0000 7fb0dec04700 1 --2- 172.21.15.42:0/2410449626 >> v2:172.21.15.40:3300/0 conn(0x7fb0d807cfb0 0x7fb0d814a070 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fb0d0001b70 tx=0x7fb0d0001bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:25.154 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:25.137+0000 7fb0dec04700 1 --2- 172.21.15.42:0/2410449626 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb0d807cfb0 0x7fb0d814a070 secure :-1 s=READY pgs=1760 cs=0 l=1 rx=0x7fb0d0001b70 tx=0x7fb0d0001bb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:14:25.155 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:25.137+0000 7fb0dec04700 1 --2- 172.21.15.42:0/2410449626 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb0d807cfb0 0x7fb0d814a070 secure :-1 s=THROTTLE_DONE pgs=1760 cs=0 l=1 rx=0x7fb0d0001b70 tx=0x7fb0d0001bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:25.155 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:25.137+0000 7fb0cf7fe700 1 -- 172.21.15.42:0/2410449626 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb0d0001930 con 0x7fb0d807cfb0 2019-12-05T06:14:25.155 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:25.137+0000 7fb0dec04700 1 --2- 172.21.15.42:0/2410449626 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb0d807cfb0 0x7fb0d814a070 secure :-1 s=THROTTLE_DONE pgs=1760 cs=0 l=1 rx=0x7fb0d0001b70 tx=0x7fb0d0001bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:25.155 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:25.137+0000 7fb0dec04700 1 --2- 172.21.15.42:0/2410449626 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb0d807cfb0 0x7fb0d814a070 secure :-1 s=THROTTLE_DONE pgs=1760 cs=0 l=1 rx=0x7fb0d0001b70 tx=0x7fb0d0001bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:25.155 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:25.137+0000 7fb0cf7fe700 1 -- 172.21.15.42:0/2410449626 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0d0001740 con 0x7fb0d807cfb0 2019-12-05T06:14:25.155 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:25.137+0000 7fb0cf7fe700 1 -- 172.21.15.42:0/2410449626 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb0c8011b70 con 0x7fb0d807cfb0 2019-12-05T06:14:25.156 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:25.137+0000 7fb0e0667700 1 -- 172.21.15.42:0/2410449626 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0d8118f70 con 0x7fb0d807cfb0 2019-12-05T06:14:25.156 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:25.137+0000 7fb0e0667700 1 -- 172.21.15.42:0/2410449626 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0d8130340 con 0x7fb0d807cfb0 2019-12-05T06:14:25.156 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:25.137+0000 7fb0dec04700 1 --2- 172.21.15.42:0/2410449626 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb0d807cfb0 0x7fb0d814a070 secure :-1 s=THROTTLE_DONE pgs=1760 cs=0 l=1 rx=0x7fb0d0001b70 tx=0x7fb0d0001bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:25.156 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:25.141+0000 7fb0cf7fe700 1 -- 172.21.15.42:0/2410449626 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fb0c800f360 con 0x7fb0d807cfb0 2019-12-05T06:14:25.156 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:25.141+0000 7fb0dec04700 1 --2- 172.21.15.42:0/2410449626 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb0d807cfb0 0x7fb0d814a070 secure :-1 s=THROTTLE_DONE pgs=1760 cs=0 l=1 rx=0x7fb0d0001b70 tx=0x7fb0d0001bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:25.156 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:25.141+0000 7fb0cf7fe700 1 --2- 172.21.15.42:0/2410449626 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb0b803b220 0x7fb0b803d730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:14:25.156 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:25.141+0000 7fb0cf7fe700 1 -- 172.21.15.42:0/2410449626 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 10245+0+0 (secure 0 0 0) 0x7fb0c804afc0 con 0x7fb0d807cfb0 2019-12-05T06:14:25.157 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:25.141+0000 7fb0de403700 1 --2- 172.21.15.42:0/2410449626 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb0b803b220 0x7fb0b803d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:25.157 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:25.141+0000 7fb0de403700 1 --2- 172.21.15.42:0/2410449626 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb0b803b220 0x7fb0b803d730 crc :-1 s=READY pgs=974 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:14:25.157 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:25.145+0000 7fb0e0667700 1 -- 172.21.15.42:0/2410449626 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0d805d110 con 0x7fb0d807cfb0 2019-12-05T06:14:25.160 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:25.153+0000 7fb0dec04700 1 --2- 172.21.15.42:0/2410449626 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb0d807cfb0 0x7fb0d814a070 secure :-1 s=THROTTLE_DONE pgs=1760 cs=0 l=1 rx=0x7fb0d0001b70 tx=0x7fb0d0001bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:25.160 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:25.153+0000 7fb0cf7fe700 1 -- 172.21.15.42:0/2410449626 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fb0c8032eb0 con 0x7fb0d807cfb0 2019-12-05T06:14:25.331 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:25.325+0000 7fb0e0667700 1 -- 172.21.15.42:0/2410449626 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb0d805b8c0 con 0x7fb0d807cfb0 2019-12-05T06:14:25.332 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:25.329+0000 7fb0dec04700 1 --2- 172.21.15.42:0/2410449626 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb0d807cfb0 0x7fb0d814a070 secure :-1 s=THROTTLE_DONE pgs=1760 cs=0 l=1 rx=0x7fb0d0001b70 tx=0x7fb0d0001bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:25.332 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:25.329+0000 7fb0cf7fe700 1 -- 172.21.15.42:0/2410449626 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 278 v278) v1 ==== 95+0+2115 (secure 0 0 0) 0x7fb0c800ebe0 con 0x7fb0d807cfb0 2019-12-05T06:14:25.335 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:14:25.335 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":278,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7217,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/2667648531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2667648531},{"type":"v1","addr":"172.21.15.40:6833","nonce":2667648531}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":278}],"filesystems":[{"mdsmap":{"epoch":277,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:13:19.980391+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":98,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10246},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10246":{"gid":10246,"name":"a","rank":0,"incarnation":274,"state":"up:active","state_seq":17,"addr":"172.21.15.42:6835/4032647860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":4032647860},{"type":"v1","addr":"172.21.15.42:6835","nonce":4032647860}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:14:25.335 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 278 2019-12-05T06:14:25.337 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:25.333+0000 7fb0e0667700 1 -- 172.21.15.42:0/2410449626 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb0b803b220 msgr2=0x7fb0b803d730 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:25.337 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:25.333+0000 7fb0e0667700 1 --2- 172.21.15.42:0/2410449626 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb0b803b220 0x7fb0b803d730 crc :-1 s=READY pgs=974 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:25.337 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:25.333+0000 7fb0e0667700 1 -- 172.21.15.42:0/2410449626 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb0d807cfb0 msgr2=0x7fb0d814a070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:25.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:25.333+0000 7fb0e0667700 1 --2- 172.21.15.42:0/2410449626 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb0d807cfb0 0x7fb0d814a070 secure :-1 s=READY pgs=1760 cs=0 l=1 rx=0x7fb0d0001b70 tx=0x7fb0d0001bb0).stop 2019-12-05T06:14:25.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:25.333+0000 7fb0e0667700 1 -- 172.21.15.42:0/2410449626 shutdown_connections 2019-12-05T06:14:25.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:25.333+0000 7fb0e0667700 1 --2- 172.21.15.42:0/2410449626 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb0b803b220 0x7fb0b803d730 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:25.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:25.333+0000 7fb0e0667700 1 --2- 172.21.15.42:0/2410449626 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb0d807cfb0 0x7fb0d814a070 unknown :-1 s=CLOSED pgs=1760 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:25.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:25.333+0000 7fb0e0667700 1 --2- 172.21.15.42:0/2410449626 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb0d814aa40 0x7fb0d808f5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:25.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:25.333+0000 7fb0e0667700 1 -- 172.21.15.42:0/2410449626 shutdown_connections 2019-12-05T06:14:25.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:25.333+0000 7fb0e0667700 1 -- 172.21.15.42:0/2410449626 wait complete. 2019-12-05T06:14:25.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:25.333+0000 7fb0e0667700 1 -- 172.21.15.42:0/2410449626 >> 172.21.15.42:0/2410449626 conn(0x7fb0d80a7740 msgr2=0x7fb0d8059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:14:25.348 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:14:27.349 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:14:27.464 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:27.453+0000 7f2d329a0700 1 Processor -- start 2019-12-05T06:14:27.464 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:27.453+0000 7f2d329a0700 1 -- start start 2019-12-05T06:14:27.464 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:27.457+0000 7f2d329a0700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2d2c0a7730 0x7f2d2c08df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:14:27.464 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:27.457+0000 7f2d329a0700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2d2c07e710 0x7f2d2c07eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:14:27.465 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:27.457+0000 7f2d329a0700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d2c0a0700 con 0x7f2d2c09f2d0 2019-12-05T06:14:27.465 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:27.457+0000 7f2d329a0700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d2c0a0af0 con 0x7f2d2c0a7730 2019-12-05T06:14:27.465 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:27.457+0000 7f2d329a0700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d2c08e5a0 con 0x7f2d2c07e710 2019-12-05T06:14:27.465 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:27.457+0000 7f2d30f3d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2d2c0a7730 0x7f2d2c08df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:27.465 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:27.457+0000 7f2d30f3d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2d2c0a7730 0x7f2d2c08df70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:51254/0 (socket says 172.21.15.42:51254) 2019-12-05T06:14:27.466 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:27.457+0000 7f2d30f3d700 1 -- 172.21.15.42:0/4101905799 learned_addr learned my addr 172.21.15.42:0/4101905799 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:14:27.466 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:27.457+0000 7f2d2bfff700 1 --2- 172.21.15.42:0/4101905799 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2d2c07e710 0x7f2d2c07eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:27.466 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:27.457+0000 7f2d2b7fe700 1 -- 172.21.15.42:0/4101905799 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2326344747 0 0) 0x7f2d2c0a0700 con 0x7f2d2c09f2d0 2019-12-05T06:14:27.466 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:27.457+0000 7f2d30f3d700 1 -- 172.21.15.42:0/4101905799 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2d2c07e710 msgr2=0x7f2d2c07eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:14:27.466 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:27.457+0000 7f2d30f3d700 1 --2- 172.21.15.42:0/4101905799 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2d2c07e710 0x7f2d2c07eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:14:27.466 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:27.457+0000 7f2d30f3d700 1 -- 172.21.15.42:0/4101905799 >> v1:172.21.15.40:6789/0 conn(0x7f2d2c09f2d0 legacy=0x7f2d2c09f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:27.467 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:27.457+0000 7f2d30f3d700 1 -- 172.21.15.42:0/4101905799 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d2c07f090 con 0x7f2d2c0a7730 2019-12-05T06:14:27.467 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:27.457+0000 7f2d30f3d700 1 --2- 172.21.15.42:0/4101905799 >> v2:172.21.15.42:3300/0 conn(0x7f2d2c0a7730 0x7f2d2c08df70 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f2d1c008790 tx=0x7f2d1c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:27.467 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:27.457+0000 7f2d30f3d700 1 --2- 172.21.15.42:0/4101905799 >> v2:172.21.15.42:3300/0 conn(0x7f2d2c0a7730 0x7f2d2c08df70 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f2d1c008790 tx=0x7f2d1c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:27.467 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:27.457+0000 7f2d30f3d700 1 --2- 172.21.15.42:0/4101905799 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2d2c0a7730 0x7f2d2c08df70 secure :-1 s=READY pgs=2317 cs=0 l=1 rx=0x7f2d1c008790 tx=0x7f2d1c00fd30).ready entity=mon.1 client_cookie=c2508ac4624dc076 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:14:27.467 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:27.457+0000 7f2d30f3d700 1 --2- 172.21.15.42:0/4101905799 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2d2c0a7730 0x7f2d2c08df70 secure :-1 s=THROTTLE_DONE pgs=2317 cs=0 l=1 rx=0x7f2d1c008790 tx=0x7f2d1c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:27.468 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:27.457+0000 7f2d2b7fe700 1 -- 172.21.15.42:0/4101905799 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2d1c0113b0 con 0x7f2d2c0a7730 2019-12-05T06:14:27.468 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:27.457+0000 7f2d30f3d700 1 --2- 172.21.15.42:0/4101905799 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2d2c0a7730 0x7f2d2c08df70 secure :-1 s=THROTTLE_DONE pgs=2317 cs=0 l=1 rx=0x7f2d1c008790 tx=0x7f2d1c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:27.468 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:27.457+0000 7f2d30f3d700 1 --2- 172.21.15.42:0/4101905799 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2d2c0a7730 0x7f2d2c08df70 secure :-1 s=THROTTLE_DONE pgs=2317 cs=0 l=1 rx=0x7f2d1c008790 tx=0x7f2d1c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:27.468 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:27.457+0000 7f2d2b7fe700 1 -- 172.21.15.42:0/4101905799 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d1c008bb0 con 0x7f2d2c0a7730 2019-12-05T06:14:27.469 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:27.457+0000 7f2d2b7fe700 1 -- 172.21.15.42:0/4101905799 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2d1c011990 con 0x7f2d2c0a7730 2019-12-05T06:14:27.469 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:27.457+0000 7f2d329a0700 1 -- 172.21.15.42:0/4101905799 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2d2c0a7730 msgr2=0x7f2d2c08df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:27.469 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:27.457+0000 7f2d329a0700 1 --2- 172.21.15.42:0/4101905799 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2d2c0a7730 0x7f2d2c08df70 secure :-1 s=READY pgs=2317 cs=0 l=1 rx=0x7f2d1c008790 tx=0x7f2d1c00fd30).stop 2019-12-05T06:14:27.469 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:27.457+0000 7f2d329a0700 1 -- 172.21.15.42:0/4101905799 shutdown_connections 2019-12-05T06:14:27.469 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:27.457+0000 7f2d329a0700 1 --2- 172.21.15.42:0/4101905799 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2d2c0a7730 0x7f2d2c08df70 unknown :-1 s=CLOSED pgs=2317 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:27.470 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:27.457+0000 7f2d329a0700 1 --2- 172.21.15.42:0/4101905799 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2d2c07e710 0x7f2d2c07eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:14:27.470 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:27.457+0000 7f2d329a0700 1 -- 172.21.15.42:0/4101905799 shutdown_connections 2019-12-05T06:14:27.470 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:27.457+0000 7f2d329a0700 1 -- 172.21.15.42:0/4101905799 wait complete. 2019-12-05T06:14:27.471 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:27.457+0000 7f2d329a0700 1 -- 172.21.15.42:0/4101905799 >> 172.21.15.42:0/4101905799 conn(0x7f2d2c0aabd0 msgr2=0x7f2d2c0a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:14:27.474 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:27.461+0000 7f2d329a0700 1 Processor -- start 2019-12-05T06:14:27.474 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:27.461+0000 7f2d329a0700 1 -- start start 2019-12-05T06:14:27.474 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:27.461+0000 7f2d329a0700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2d2c120360 0x7f2d2c130250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:14:27.475 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:27.461+0000 7f2d329a0700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d2c1307a0 con 0x7f2d2c124a10 2019-12-05T06:14:27.475 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:27.461+0000 7f2d329a0700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d2c1396c0 con 0x7f2d2c11ec70 2019-12-05T06:14:27.475 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:27.461+0000 7f2d329a0700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d2c11fe20 con 0x7f2d2c120360 2019-12-05T06:14:27.475 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:27.461+0000 7f2d2bfff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2d2c120360 0x7f2d2c130250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:27.475 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:27.461+0000 7f2d3173e700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f2d2c11ec70 0x7f2d2c11f0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:51298/0 (socket says 172.21.15.42:51298) 2019-12-05T06:14:27.475 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:27.461+0000 7f2d3173e700 1 -- 172.21.15.42:0/1144588107 learned_addr learned my addr 172.21.15.42:0/1144588107 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:14:27.475 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:27.461+0000 7f2d2bfff700 1 -- 172.21.15.42:0/1144588107 >> v1:172.21.15.42:6789/0 conn(0x7f2d2c11ec70 legacy=0x7f2d2c11f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:27.476 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:27.461+0000 7f2d2bfff700 1 -- 172.21.15.42:0/1144588107 >> v1:172.21.15.40:6789/0 conn(0x7f2d2c124a10 legacy=0x7f2d2c07da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:27.476 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:27.461+0000 7f2d297fa700 1 -- 172.21.15.42:0/1144588107 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 436311135 0 0) 0x7f2d2c1396c0 con 0x7f2d2c11ec70 2019-12-05T06:14:27.476 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:27.461+0000 7f2d2bfff700 1 -- 172.21.15.42:0/1144588107 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d2c08eb60 con 0x7f2d2c120360 2019-12-05T06:14:27.476 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:27.461+0000 7f2d2bfff700 1 --2- 172.21.15.42:0/1144588107 >> v2:172.21.15.42:3300/0 conn(0x7f2d2c120360 0x7f2d2c130250 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f2d18002f30 tx=0x7f2d18010af0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:27.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:27.461+0000 7f2d2bfff700 1 --2- 172.21.15.42:0/1144588107 >> v2:172.21.15.42:3300/0 conn(0x7f2d2c120360 0x7f2d2c130250 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f2d18002f30 tx=0x7f2d18010af0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:27.482 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:27.461+0000 7f2d2bfff700 1 --2- 172.21.15.42:0/1144588107 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2d2c120360 0x7f2d2c130250 secure :-1 s=READY pgs=2319 cs=0 l=1 rx=0x7f2d18002f30 tx=0x7f2d18010af0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:14:27.482 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:27.461+0000 7f2d2bfff700 1 --2- 172.21.15.42:0/1144588107 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2d2c120360 0x7f2d2c130250 secure :-1 s=THROTTLE_DONE pgs=2319 cs=0 l=1 rx=0x7f2d18002f30 tx=0x7f2d18010af0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:27.482 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:27.461+0000 7f2d297fa700 1 -- 172.21.15.42:0/1144588107 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2d180125a0 con 0x7f2d2c120360 2019-12-05T06:14:27.482 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:27.461+0000 7f2d2bfff700 1 --2- 172.21.15.42:0/1144588107 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2d2c120360 0x7f2d2c130250 secure :-1 s=THROTTLE_DONE pgs=2319 cs=0 l=1 rx=0x7f2d18002f30 tx=0x7f2d18010af0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:27.482 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:27.461+0000 7f2d297fa700 1 -- 172.21.15.42:0/1144588107 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d18007150 con 0x7f2d2c120360 2019-12-05T06:14:27.482 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:27.461+0000 7f2d2bfff700 1 --2- 172.21.15.42:0/1144588107 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2d2c120360 0x7f2d2c130250 secure :-1 s=THROTTLE_DONE pgs=2319 cs=0 l=1 rx=0x7f2d18002f30 tx=0x7f2d18010af0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:27.483 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:27.461+0000 7f2d329a0700 1 -- 172.21.15.42:0/1144588107 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d2c08ee50 con 0x7f2d2c120360 2019-12-05T06:14:27.483 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:27.461+0000 7f2d297fa700 1 -- 172.21.15.42:0/1144588107 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2d18012b80 con 0x7f2d2c120360 2019-12-05T06:14:27.483 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:27.465+0000 7f2d329a0700 1 -- 172.21.15.42:0/1144588107 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d2c08f350 con 0x7f2d2c120360 2019-12-05T06:14:27.483 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:27.465+0000 7f2d2bfff700 1 --2- 172.21.15.42:0/1144588107 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2d2c120360 0x7f2d2c130250 secure :-1 s=THROTTLE_DONE pgs=2319 cs=0 l=1 rx=0x7f2d18002f30 tx=0x7f2d18010af0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:27.484 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:27.465+0000 7f2d297fa700 1 -- 172.21.15.42:0/1144588107 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f2d18005540 con 0x7f2d2c120360 2019-12-05T06:14:27.484 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:27.465+0000 7f2d2bfff700 1 --2- 172.21.15.42:0/1144588107 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2d2c120360 0x7f2d2c130250 secure :-1 s=THROTTLE_DONE pgs=2319 cs=0 l=1 rx=0x7f2d18002f30 tx=0x7f2d18010af0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:27.484 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:27.465+0000 7f2d297fa700 1 --2- 172.21.15.42:0/1144588107 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2d0803acb0 0x7f2d0803d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:14:27.484 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:27.465+0000 7f2d297fa700 1 -- 172.21.15.42:0/1144588107 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 10245+0+0 (secure 0 0 0) 0x7f2d1804b0a0 con 0x7f2d2c120360 2019-12-05T06:14:27.484 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:27.465+0000 7f2d30f3d700 1 --2- 172.21.15.42:0/1144588107 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2d0803acb0 0x7f2d0803d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:27.485 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:27.465+0000 7f2d30f3d700 1 --2- 172.21.15.42:0/1144588107 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2d0803acb0 0x7f2d0803d1c0 crc :-1 s=READY pgs=975 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:14:27.485 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:27.469+0000 7f2d329a0700 1 -- 172.21.15.42:0/1144588107 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d2c05d100 con 0x7f2d2c120360 2019-12-05T06:14:27.485 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:27.473+0000 7f2d2bfff700 1 --2- 172.21.15.42:0/1144588107 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2d2c120360 0x7f2d2c130250 secure :-1 s=THROTTLE_DONE pgs=2319 cs=0 l=1 rx=0x7f2d18002f30 tx=0x7f2d18010af0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:27.485 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:27.473+0000 7f2d297fa700 1 -- 172.21.15.42:0/1144588107 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f2d18032120 con 0x7f2d2c120360 2019-12-05T06:14:27.645 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:27.641+0000 7f2d329a0700 1 -- 172.21.15.42:0/1144588107 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2d2c05b8b0 con 0x7f2d2c120360 2019-12-05T06:14:27.646 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:27.641+0000 7f2d2bfff700 1 --2- 172.21.15.42:0/1144588107 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2d2c120360 0x7f2d2c130250 secure :-1 s=THROTTLE_DONE pgs=2319 cs=0 l=1 rx=0x7f2d18002f30 tx=0x7f2d18010af0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:27.646 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:27.641+0000 7f2d297fa700 1 -- 172.21.15.42:0/1144588107 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 278 v278) v1 ==== 95+0+2115 (secure 0 0 0) 0x7f2d18032380 con 0x7f2d2c120360 2019-12-05T06:14:27.647 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 278 2019-12-05T06:14:27.647 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:14:27.647 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":278,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7217,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/2667648531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2667648531},{"type":"v1","addr":"172.21.15.40:6833","nonce":2667648531}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":278}],"filesystems":[{"mdsmap":{"epoch":277,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:13:19.980391+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":98,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10246},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10246":{"gid":10246,"name":"a","rank":0,"incarnation":274,"state":"up:active","state_seq":17,"addr":"172.21.15.42:6835/4032647860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":4032647860},{"type":"v1","addr":"172.21.15.42:6835","nonce":4032647860}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:14:27.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:27.645+0000 7f2d329a0700 1 -- 172.21.15.42:0/1144588107 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2d0803acb0 msgr2=0x7f2d0803d1c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:27.650 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:27.645+0000 7f2d329a0700 1 --2- 172.21.15.42:0/1144588107 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2d0803acb0 0x7f2d0803d1c0 crc :-1 s=READY pgs=975 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:27.650 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:27.645+0000 7f2d329a0700 1 -- 172.21.15.42:0/1144588107 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2d2c120360 msgr2=0x7f2d2c130250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:27.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:27.645+0000 7f2d329a0700 1 --2- 172.21.15.42:0/1144588107 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2d2c120360 0x7f2d2c130250 secure :-1 s=READY pgs=2319 cs=0 l=1 rx=0x7f2d18002f30 tx=0x7f2d18010af0).stop 2019-12-05T06:14:27.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:27.645+0000 7f2d329a0700 1 -- 172.21.15.42:0/1144588107 shutdown_connections 2019-12-05T06:14:27.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:27.645+0000 7f2d329a0700 1 --2- 172.21.15.42:0/1144588107 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2d0803acb0 0x7f2d0803d1c0 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:27.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:27.645+0000 7f2d329a0700 1 --2- 172.21.15.42:0/1144588107 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2d2c120360 0x7f2d2c130250 unknown :-1 s=CLOSED pgs=2319 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:27.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:27.645+0000 7f2d329a0700 1 -- 172.21.15.42:0/1144588107 shutdown_connections 2019-12-05T06:14:27.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:27.645+0000 7f2d329a0700 1 -- 172.21.15.42:0/1144588107 wait complete. 2019-12-05T06:14:27.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:27.645+0000 7f2d329a0700 1 -- 172.21.15.42:0/1144588107 >> 172.21.15.42:0/1144588107 conn(0x7f2d2c0aabd0 msgr2=0x7f2d2c11c460 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:14:27.662 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:14:29.663 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:14:29.768 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:29.757+0000 7f181d690700 1 Processor -- start 2019-12-05T06:14:29.768 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:29.757+0000 7f181d690700 1 -- start start 2019-12-05T06:14:29.768 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:29.757+0000 7f181d690700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f18180a7740 0x7f181808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:14:29.769 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:29.757+0000 7f181d690700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18180a0710 con 0x7f181807e720 2019-12-05T06:14:29.769 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:29.757+0000 7f181d690700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18180a9ba0 con 0x7f181809fae0 2019-12-05T06:14:29.769 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:29.757+0000 7f181d690700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18180a0b00 con 0x7f18180a7740 2019-12-05T06:14:29.770 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:29.757+0000 7f1817fff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f181809fae0 0x7f181814efe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:51310/0 (socket says 172.21.15.42:51310) 2019-12-05T06:14:29.770 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:29.757+0000 7f1817fff700 1 -- 172.21.15.42:0/2198143322 learned_addr learned my addr 172.21.15.42:0/2198143322 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:14:29.771 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:29.757+0000 7f18177fe700 1 --2- 172.21.15.42:0/2198143322 >> v2:172.21.15.40:3300/0 conn(0x7f18180a7740 0x7f181808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:29.771 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:29.757+0000 7f18167fc700 1 -- 172.21.15.42:0/2198143322 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2784310063 0 0) 0x7f18180a9ba0 con 0x7f181809fae0 2019-12-05T06:14:29.771 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:29.757+0000 7f18167fc700 1 -- 172.21.15.42:0/2198143322 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18000030b0 con 0x7f181809fae0 2019-12-05T06:14:29.771 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:29.757+0000 7f18167fc700 1 -- 172.21.15.42:0/2198143322 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 747350185 0 0) 0x7f18000030b0 con 0x7f181809fae0 2019-12-05T06:14:29.771 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:29.757+0000 7f18167fc700 1 -- 172.21.15.42:0/2198143322 >> v2:172.21.15.40:3300/0 conn(0x7f18180a7740 msgr2=0x7f181808df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:14:29.772 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:29.757+0000 7f18167fc700 1 --2- 172.21.15.42:0/2198143322 >> v2:172.21.15.40:3300/0 conn(0x7f18180a7740 0x7f181808df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:14:29.772 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:29.757+0000 7f18167fc700 1 -- 172.21.15.42:0/2198143322 >> v1:172.21.15.40:6789/0 conn(0x7f181807e720 legacy=0x7f181809f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:29.772 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:29.757+0000 7f18167fc700 1 -- 172.21.15.42:0/2198143322 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f181807eb50 con 0x7f181809fae0 2019-12-05T06:14:29.772 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:29.757+0000 7f18167fc700 1 -- 172.21.15.42:0/2198143322 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7f1810001ad0 con 0x7f181809fae0 2019-12-05T06:14:29.772 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:29.757+0000 7f18167fc700 1 -- 172.21.15.42:0/2198143322 >> v1:172.21.15.42:6789/0 conn(0x7f181809fae0 legacy=0x7f181814efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:29.772 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:29.757+0000 7f18167fc700 1 --2- 172.21.15.42:0/2198143322 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f18000036e0 0x7f1800003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:14:29.773 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:29.757+0000 7f18167fc700 1 --2- 172.21.15.42:0/2198143322 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1800004420 0x7f1800004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:14:29.773 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:29.757+0000 7f18167fc700 1 --2- 172.21.15.42:0/2198143322 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1800004df0 0x7f180000b670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:14:29.773 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:29.757+0000 7f18167fc700 1 -- 172.21.15.42:0/2198143322 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18180a0b00 con 0x7f18000036e0 2019-12-05T06:14:29.773 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:29.757+0000 7f18167fc700 1 -- 172.21.15.42:0/2198143322 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f180004fce0 con 0x7f1800004420 2019-12-05T06:14:29.773 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:29.757+0000 7f18167fc700 1 -- 172.21.15.42:0/2198143322 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f180004ff20 con 0x7f1800004df0 2019-12-05T06:14:29.773 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:29.757+0000 7f1816ffd700 1 --2- 172.21.15.42:0/2198143322 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1800004420 0x7f1800004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:29.773 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:29.761+0000 7f18177fe700 1 --2- 172.21.15.42:0/2198143322 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f18000036e0 0x7f1800003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:29.774 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:29.761+0000 7f1817fff700 1 --2- 172.21.15.42:0/2198143322 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1800004df0 0x7f180000b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:29.774 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:29.761+0000 7f1816ffd700 1 -- 172.21.15.42:0/2198143322 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1800004df0 msgr2=0x7f180000b670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:14:29.774 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:29.761+0000 7f1816ffd700 1 --2- 172.21.15.42:0/2198143322 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1800004df0 0x7f180000b670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:14:29.774 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:29.761+0000 7f1816ffd700 1 -- 172.21.15.42:0/2198143322 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f18000036e0 msgr2=0x7f1800003b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:14:29.774 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:29.761+0000 7f1816ffd700 1 --2- 172.21.15.42:0/2198143322 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f18000036e0 0x7f1800003b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:14:29.774 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:29.761+0000 7f1816ffd700 1 -- 172.21.15.42:0/2198143322 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1800050220 con 0x7f1800004420 2019-12-05T06:14:29.775 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:29.761+0000 7f18177fe700 1 --2- 172.21.15.42:0/2198143322 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f18000036e0 0x7f1800003b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:14:29.775 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:29.761+0000 7f1817fff700 1 -- 172.21.15.42:0/2198143322 reap_dead start 2019-12-05T06:14:29.776 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:29.761+0000 7f1816ffd700 1 --2- 172.21.15.42:0/2198143322 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1800004420 0x7f1800004850 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f180c001000 tx=0x7f180c00fcd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:29.776 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:29.761+0000 7f1816ffd700 1 --2- 172.21.15.42:0/2198143322 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1800004420 0x7f1800004850 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f180c001000 tx=0x7f180c00fcd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:29.776 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:29.761+0000 7f1816ffd700 1 --2- 172.21.15.42:0/2198143322 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1800004420 0x7f1800004850 secure :-1 s=READY pgs=2321 cs=0 l=1 rx=0x7f180c001000 tx=0x7f180c00fcd0).ready entity=mon.1 client_cookie=a3ec3b354497f1a7 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:14:29.776 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:29.761+0000 7f1816ffd700 1 --2- 172.21.15.42:0/2198143322 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1800004420 0x7f1800004850 secure :-1 s=THROTTLE_DONE pgs=2321 cs=0 l=1 rx=0x7f180c001000 tx=0x7f180c00fcd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:29.776 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:29.761+0000 7f18167fc700 1 -- 172.21.15.42:0/2198143322 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f180c003540 con 0x7f1800004420 2019-12-05T06:14:29.777 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:29.761+0000 7f1816ffd700 1 --2- 172.21.15.42:0/2198143322 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1800004420 0x7f1800004850 secure :-1 s=THROTTLE_DONE pgs=2321 cs=0 l=1 rx=0x7f180c001000 tx=0x7f180c00fcd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:29.777 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:29.761+0000 7f18167fc700 1 -- 172.21.15.42:0/2198143322 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f180c003160 con 0x7f1800004420 2019-12-05T06:14:29.777 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:29.761+0000 7f1816ffd700 1 --2- 172.21.15.42:0/2198143322 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1800004420 0x7f1800004850 secure :-1 s=THROTTLE_DONE pgs=2321 cs=0 l=1 rx=0x7f180c001000 tx=0x7f180c00fcd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:29.777 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:29.761+0000 7f18167fc700 1 -- 172.21.15.42:0/2198143322 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f180c003b20 con 0x7f1800004420 2019-12-05T06:14:29.777 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:29.761+0000 7f181d690700 1 -- 172.21.15.42:0/2198143322 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1800004420 msgr2=0x7f1800004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:29.777 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:29.761+0000 7f181d690700 1 --2- 172.21.15.42:0/2198143322 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1800004420 0x7f1800004850 secure :-1 s=READY pgs=2321 cs=0 l=1 rx=0x7f180c001000 tx=0x7f180c00fcd0).stop 2019-12-05T06:14:29.777 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:29.761+0000 7f181d690700 1 -- 172.21.15.42:0/2198143322 shutdown_connections 2019-12-05T06:14:29.778 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:29.761+0000 7f181d690700 1 --2- 172.21.15.42:0/2198143322 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1800004420 0x7f1800004850 unknown :-1 s=CLOSED pgs=2321 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:29.778 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:29.761+0000 7f181d690700 1 -- 172.21.15.42:0/2198143322 shutdown_connections 2019-12-05T06:14:29.778 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:29.761+0000 7f181d690700 1 -- 172.21.15.42:0/2198143322 wait complete. 2019-12-05T06:14:29.778 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:29.761+0000 7f181d690700 1 -- 172.21.15.42:0/2198143322 >> 172.21.15.42:0/2198143322 conn(0x7f18180aabe0 msgr2=0x7f18180a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:14:29.782 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:29.765+0000 7f181d690700 1 Processor -- start 2019-12-05T06:14:29.783 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:29.765+0000 7f181d690700 1 -- start start 2019-12-05T06:14:29.783 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:29.765+0000 7f181d690700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f181811bc00 0x7f1818123dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:14:29.783 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:29.765+0000 7f181d690700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f1818134110 0x7f1818077300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:14:29.783 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:29.765+0000 7f181d690700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f181811a9d0 con 0x7f1818124320 2019-12-05T06:14:29.783 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:29.765+0000 7f181d690700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f181811a490 con 0x7f1818134110 2019-12-05T06:14:29.783 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:29.765+0000 7f181d690700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f181811a730 con 0x7f181811bc00 2019-12-05T06:14:29.783 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:29.765+0000 7f18177fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f181811bc00 0x7f1818123dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:29.784 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:29.765+0000 7f18177fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f181811bc00 0x7f1818123dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:51280/0 (socket says 172.21.15.42:51280) 2019-12-05T06:14:29.784 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:29.765+0000 7f18177fe700 1 -- 172.21.15.42:0/1062182891 learned_addr learned my addr 172.21.15.42:0/1062182891 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:14:29.784 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:29.765+0000 7f1817fff700 1 --2- 172.21.15.42:0/1062182891 >> v2:172.21.15.40:3300/0 conn(0x7f1818134110 0x7f1818077300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:29.784 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:29.765+0000 7f18177fe700 1 -- 172.21.15.42:0/1062182891 >> v2:172.21.15.40:3300/0 conn(0x7f1818134110 msgr2=0x7f1818077300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:29.784 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:29.765+0000 7f18177fe700 1 --2- 172.21.15.42:0/1062182891 >> v2:172.21.15.40:3300/0 conn(0x7f1818134110 0x7f1818077300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:29.784 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:29.765+0000 7f18177fe700 1 -- 172.21.15.42:0/1062182891 >> v1:172.21.15.42:6789/0 conn(0x7f1818124320 legacy=0x7f1818133910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:29.785 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:29.765+0000 7f18177fe700 1 -- 172.21.15.42:0/1062182891 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f181805ae30 con 0x7f181811bc00 2019-12-05T06:14:29.785 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:29.765+0000 7f18177fe700 1 --2- 172.21.15.42:0/1062182891 >> v2:172.21.15.42:3300/0 conn(0x7f181811bc00 0x7f1818123dd0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f18180a4600 tx=0x7f1808010a40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:29.785 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:29.765+0000 7f18177fe700 1 --2- 172.21.15.42:0/1062182891 >> v2:172.21.15.42:3300/0 conn(0x7f181811bc00 0x7f1818123dd0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f18180a4600 tx=0x7f1808010a40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:29.785 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:29.765+0000 7f18177fe700 1 --2- 172.21.15.42:0/1062182891 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f181811bc00 0x7f1818123dd0 secure :-1 s=READY pgs=2323 cs=0 l=1 rx=0x7f18180a4600 tx=0x7f1808010a40).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:14:29.785 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:29.765+0000 7f18177fe700 1 --2- 172.21.15.42:0/1062182891 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f181811bc00 0x7f1818123dd0 secure :-1 s=THROTTLE_DONE pgs=2323 cs=0 l=1 rx=0x7f18180a4600 tx=0x7f1808010a40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:29.785 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:29.765+0000 7f18177fe700 1 --2- 172.21.15.42:0/1062182891 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f181811bc00 0x7f1818123dd0 secure :-1 s=THROTTLE_DONE pgs=2323 cs=0 l=1 rx=0x7f18180a4600 tx=0x7f1808010a40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:29.785 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:29.765+0000 7f17fffff700 1 -- 172.21.15.42:0/1062182891 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1808005540 con 0x7f181811bc00 2019-12-05T06:14:29.786 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:29.765+0000 7f18177fe700 1 --2- 172.21.15.42:0/1062182891 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f181811bc00 0x7f1818123dd0 secure :-1 s=THROTTLE_DONE pgs=2323 cs=0 l=1 rx=0x7f18180a4600 tx=0x7f1808010a40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:29.786 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:29.765+0000 7f17fffff700 1 -- 172.21.15.42:0/1062182891 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1808007150 con 0x7f181811bc00 2019-12-05T06:14:29.786 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:29.765+0000 7f181d690700 1 -- 172.21.15.42:0/1062182891 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f181811c6f0 con 0x7f181811bc00 2019-12-05T06:14:29.786 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:29.765+0000 7f17fffff700 1 -- 172.21.15.42:0/1062182891 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1808005b60 con 0x7f181811bc00 2019-12-05T06:14:29.786 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:29.765+0000 7f181d690700 1 -- 172.21.15.42:0/1062182891 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f181811ce50 con 0x7f181811bc00 2019-12-05T06:14:29.786 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:29.769+0000 7f18177fe700 1 --2- 172.21.15.42:0/1062182891 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f181811bc00 0x7f1818123dd0 secure :-1 s=THROTTLE_DONE pgs=2323 cs=0 l=1 rx=0x7f18180a4600 tx=0x7f1808010a40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:29.786 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:29.769+0000 7f17fffff700 1 -- 172.21.15.42:0/1062182891 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f180800d4f0 con 0x7f181811bc00 2019-12-05T06:14:29.786 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:29.769+0000 7f18177fe700 1 --2- 172.21.15.42:0/1062182891 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f181811bc00 0x7f1818123dd0 secure :-1 s=THROTTLE_DONE pgs=2323 cs=0 l=1 rx=0x7f18180a4600 tx=0x7f1808010a40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:29.787 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:29.769+0000 7f17fffff700 1 --2- 172.21.15.42:0/1062182891 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f17f803ae10 0x7f17f803d320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:14:29.787 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:29.769+0000 7f17fffff700 1 -- 172.21.15.42:0/1062182891 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 10245+0+0 (secure 0 0 0) 0x7f180804aca0 con 0x7f181811bc00 2019-12-05T06:14:29.787 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:29.769+0000 7f1816ffd700 1 --2- 172.21.15.42:0/1062182891 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f17f803ae10 0x7f17f803d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:29.787 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:29.769+0000 7f1816ffd700 1 --2- 172.21.15.42:0/1062182891 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f17f803ae10 0x7f17f803d320 crc :-1 s=READY pgs=976 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:14:29.787 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:29.769+0000 7f181d690700 1 -- 172.21.15.42:0/1062182891 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f181805d110 con 0x7f181811bc00 2019-12-05T06:14:29.787 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:29.777+0000 7f18177fe700 1 --2- 172.21.15.42:0/1062182891 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f181811bc00 0x7f1818123dd0 secure :-1 s=THROTTLE_DONE pgs=2323 cs=0 l=1 rx=0x7f18180a4600 tx=0x7f1808010a40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:29.787 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:29.777+0000 7f17fffff700 1 -- 172.21.15.42:0/1062182891 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f1808031d20 con 0x7f181811bc00 2019-12-05T06:14:29.952 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:29.945+0000 7f181d690700 1 -- 172.21.15.42:0/1062182891 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f181805b8c0 con 0x7f181811bc00 2019-12-05T06:14:29.952 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:29.949+0000 7f18177fe700 1 --2- 172.21.15.42:0/1062182891 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f181811bc00 0x7f1818123dd0 secure :-1 s=THROTTLE_DONE pgs=2323 cs=0 l=1 rx=0x7f18180a4600 tx=0x7f1808010a40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:29.952 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:29.949+0000 7f17fffff700 1 -- 172.21.15.42:0/1062182891 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 278 v278) v1 ==== 95+0+2115 (secure 0 0 0) 0x7f1808031f80 con 0x7f181811bc00 2019-12-05T06:14:29.954 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 278 2019-12-05T06:14:29.955 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:14:29.955 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":278,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7217,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/2667648531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2667648531},{"type":"v1","addr":"172.21.15.40:6833","nonce":2667648531}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":278}],"filesystems":[{"mdsmap":{"epoch":277,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:13:19.980391+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":98,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10246},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10246":{"gid":10246,"name":"a","rank":0,"incarnation":274,"state":"up:active","state_seq":17,"addr":"172.21.15.42:6835/4032647860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":4032647860},{"type":"v1","addr":"172.21.15.42:6835","nonce":4032647860}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:14:29.957 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:29.953+0000 7f181d690700 1 -- 172.21.15.42:0/1062182891 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f17f803ae10 msgr2=0x7f17f803d320 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:29.957 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:29.953+0000 7f181d690700 1 --2- 172.21.15.42:0/1062182891 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f17f803ae10 0x7f17f803d320 crc :-1 s=READY pgs=976 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:29.958 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:29.953+0000 7f181d690700 1 -- 172.21.15.42:0/1062182891 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f181811bc00 msgr2=0x7f1818123dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:29.958 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:29.953+0000 7f181d690700 1 --2- 172.21.15.42:0/1062182891 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f181811bc00 0x7f1818123dd0 secure :-1 s=READY pgs=2323 cs=0 l=1 rx=0x7f18180a4600 tx=0x7f1808010a40).stop 2019-12-05T06:14:29.958 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:29.953+0000 7f181d690700 1 -- 172.21.15.42:0/1062182891 shutdown_connections 2019-12-05T06:14:29.958 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:29.953+0000 7f181d690700 1 --2- 172.21.15.42:0/1062182891 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f17f803ae10 0x7f17f803d320 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:29.958 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:29.953+0000 7f181d690700 1 --2- 172.21.15.42:0/1062182891 >> v2:172.21.15.40:3300/0 conn(0x7f1818134110 0x7f1818077300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:29.959 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:29.953+0000 7f181d690700 1 --2- 172.21.15.42:0/1062182891 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f181811bc00 0x7f1818123dd0 unknown :-1 s=CLOSED pgs=2323 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:29.959 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:29.953+0000 7f181d690700 1 -- 172.21.15.42:0/1062182891 shutdown_connections 2019-12-05T06:14:29.959 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:29.953+0000 7f181d690700 1 -- 172.21.15.42:0/1062182891 wait complete. 2019-12-05T06:14:29.959 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:29.953+0000 7f181d690700 1 -- 172.21.15.42:0/1062182891 >> 172.21.15.42:0/1062182891 conn(0x7f18180a89d0 msgr2=0x7f18180a9790 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:14:29.967 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:14:31.968 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:14:32.083 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:32.061+0000 7f3abd9a2700 1 Processor -- start 2019-12-05T06:14:32.085 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:32.061+0000 7f3abd9a2700 1 -- start start 2019-12-05T06:14:32.086 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:32.065+0000 7f3abd9a2700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3ab80a7740 0x7f3ab808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:14:32.086 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:32.065+0000 7f3abd9a2700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3ab807e720 0x7f3ab807eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:14:32.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:32.065+0000 7f3abd9a2700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ab80a0710 con 0x7f3ab809f2e0 2019-12-05T06:14:32.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:32.065+0000 7f3abd9a2700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ab80a0b00 con 0x7f3ab80a7740 2019-12-05T06:14:32.088 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:32.065+0000 7f3abd9a2700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ab808e5b0 con 0x7f3ab807e720 2019-12-05T06:14:32.089 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:32.065+0000 7f3ab77fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3ab80a7740 0x7f3ab808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:32.089 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:32.065+0000 7f3ab77fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3ab80a7740 0x7f3ab808df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:51288/0 (socket says 172.21.15.42:51288) 2019-12-05T06:14:32.089 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:32.065+0000 7f3ab77fe700 1 -- 172.21.15.42:0/2400026709 learned_addr learned my addr 172.21.15.42:0/2400026709 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:14:32.089 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:32.065+0000 7f3ab6ffd700 1 --2- 172.21.15.42:0/2400026709 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3ab807e720 0x7f3ab807eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:32.089 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:32.065+0000 7f3ab77fe700 1 -- 172.21.15.42:0/2400026709 >> v2:172.21.15.42:3300/0 conn(0x7f3ab80a7740 msgr2=0x7f3ab808df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2019-12-05T06:14:32.090 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:32.065+0000 7f3ab77fe700 1 -- 172.21.15.42:0/2400026709 >> v2:172.21.15.42:3300/0 conn(0x7f3ab80a7740 msgr2=0x7f3ab808df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2019-12-05T06:14:32.090 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:32.065+0000 7f3ab77fe700 1 --2- 172.21.15.42:0/2400026709 >> v2:172.21.15.42:3300/0 conn(0x7f3ab80a7740 0x7f3ab808df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-12-05T06:14:32.090 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:32.065+0000 7f3ab77fe700 1 --2- 172.21.15.42:0/2400026709 >> v2:172.21.15.42:3300/0 conn(0x7f3ab80a7740 0x7f3ab808df80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rx=0 tx=0)._fault waiting 0.200000 2019-12-05T06:14:32.090 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:32.065+0000 7f3ab67fc700 1 -- 172.21.15.42:0/2400026709 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2882164939 0 0) 0x7f3ab80a0710 con 0x7f3ab809f2e0 2019-12-05T06:14:32.090 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:32.065+0000 7f3ab6ffd700 1 -- 172.21.15.42:0/2400026709 >> v2:172.21.15.42:3300/0 conn(0x7f3ab80a7740 msgr2=0x7f3ab808df80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2019-12-05T06:14:32.091 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:32.065+0000 7f3ab6ffd700 1 --2- 172.21.15.42:0/2400026709 >> v2:172.21.15.42:3300/0 conn(0x7f3ab80a7740 0x7f3ab808df80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:14:32.091 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:32.065+0000 7f3ab6ffd700 1 -- 172.21.15.42:0/2400026709 >> v1:172.21.15.40:6789/0 conn(0x7f3ab809f2e0 legacy=0x7f3ab809f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:32.091 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:32.065+0000 7f3ab6ffd700 1 -- 172.21.15.42:0/2400026709 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ab807f0a0 con 0x7f3ab807e720 2019-12-05T06:14:32.091 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:32.065+0000 7f3ab6ffd700 1 --2- 172.21.15.42:0/2400026709 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3ab807e720 0x7f3ab807eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f3aac008790 tx=0x7f3aac00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:32.091 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:32.065+0000 7f3ab6ffd700 1 --2- 172.21.15.42:0/2400026709 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3ab807e720 0x7f3ab807eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f3aac008790 tx=0x7f3aac00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:32.091 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:32.065+0000 7f3ab6ffd700 1 --2- 172.21.15.42:0/2400026709 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3ab807e720 0x7f3ab807eb50 secure :-1 s=READY pgs=507 cs=0 l=1 rx=0x7f3aac008790 tx=0x7f3aac00fc70).ready entity=mon.2 client_cookie=a0ec7cdb88431c35 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:14:32.092 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:32.065+0000 7f3ab6ffd700 1 --2- 172.21.15.42:0/2400026709 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3ab807e720 0x7f3ab807eb50 secure :-1 s=THROTTLE_DONE pgs=507 cs=0 l=1 rx=0x7f3aac008790 tx=0x7f3aac00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:32.092 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:32.065+0000 7f3ab6ffd700 1 --2- 172.21.15.42:0/2400026709 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3ab807e720 0x7f3ab807eb50 secure :-1 s=THROTTLE_DONE pgs=507 cs=0 l=1 rx=0x7f3aac008790 tx=0x7f3aac00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:32.092 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:32.065+0000 7f3ab67fc700 1 -- 172.21.15.42:0/2400026709 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3aac0112f0 con 0x7f3ab807e720 2019-12-05T06:14:32.092 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:32.065+0000 7f3ab6ffd700 1 --2- 172.21.15.42:0/2400026709 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3ab807e720 0x7f3ab807eb50 secure :-1 s=THROTTLE_DONE pgs=507 cs=0 l=1 rx=0x7f3aac008790 tx=0x7f3aac00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:32.092 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:32.065+0000 7f3ab67fc700 1 -- 172.21.15.42:0/2400026709 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3aac008ad0 con 0x7f3ab807e720 2019-12-05T06:14:32.092 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:32.065+0000 7f3ab67fc700 1 -- 172.21.15.42:0/2400026709 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3aac0118d0 con 0x7f3ab807e720 2019-12-05T06:14:32.093 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:32.065+0000 7f3abd9a2700 1 -- 172.21.15.42:0/2400026709 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3ab807e720 msgr2=0x7f3ab807eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:32.093 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:32.065+0000 7f3abd9a2700 1 --2- 172.21.15.42:0/2400026709 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3ab807e720 0x7f3ab807eb50 secure :-1 s=READY pgs=507 cs=0 l=1 rx=0x7f3aac008790 tx=0x7f3aac00fc70).stop 2019-12-05T06:14:32.093 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:32.069+0000 7f3abd9a2700 1 -- 172.21.15.42:0/2400026709 shutdown_connections 2019-12-05T06:14:32.093 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:32.069+0000 7f3abd9a2700 1 --2- 172.21.15.42:0/2400026709 >> v2:172.21.15.42:3300/0 conn(0x7f3ab80a7740 0x7f3ab808df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:14:32.093 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:32.069+0000 7f3abd9a2700 1 --2- 172.21.15.42:0/2400026709 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3ab807e720 0x7f3ab807eb50 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:32.093 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:32.069+0000 7f3abd9a2700 1 -- 172.21.15.42:0/2400026709 shutdown_connections 2019-12-05T06:14:32.093 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:32.069+0000 7f3abd9a2700 1 -- 172.21.15.42:0/2400026709 wait complete. 2019-12-05T06:14:32.094 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:32.069+0000 7f3abd9a2700 1 -- 172.21.15.42:0/2400026709 >> 172.21.15.42:0/2400026709 conn(0x7f3ab80aabe0 msgr2=0x7f3ab80a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:14:32.094 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:32.069+0000 7f3abd9a2700 1 Processor -- start 2019-12-05T06:14:32.094 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:32.069+0000 7f3abd9a2700 1 -- start start 2019-12-05T06:14:32.094 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:32.073+0000 7f3abd9a2700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3ab8059f50 0x7f3ab80a3c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:14:32.094 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:32.073+0000 7f3abd9a2700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f3ab8121320 0x7f3ab8137720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:14:32.094 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:32.073+0000 7f3abd9a2700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ab8121940 con 0x7f3ab8120ef0 2019-12-05T06:14:32.095 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:32.073+0000 7f3abd9a2700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ab81305a0 con 0x7f3ab8121320 2019-12-05T06:14:32.095 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:32.073+0000 7f3abd9a2700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ab812cd10 con 0x7f3ab8059f50 2019-12-05T06:14:32.095 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:32.073+0000 7f3ab6ffd700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f3ab8120ef0 0x7f3ab8130840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:51336/0 (socket says 172.21.15.42:51336) 2019-12-05T06:14:32.095 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:32.073+0000 7f3ab6ffd700 1 -- 172.21.15.42:0/3047218555 learned_addr learned my addr 172.21.15.42:0/3047218555 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:14:32.095 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:32.073+0000 7f3ab77fe700 1 --2- 172.21.15.42:0/3047218555 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3ab8059f50 0x7f3ab80a3c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:32.095 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:32.073+0000 7f3ab7fff700 1 --2- 172.21.15.42:0/3047218555 >> v2:172.21.15.40:3300/0 conn(0x7f3ab8121320 0x7f3ab8137720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:32.095 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:32.073+0000 7f3a9ffff700 1 -- 172.21.15.42:0/3047218555 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1582258135 0 0) 0x7f3ab8121940 con 0x7f3ab8120ef0 2019-12-05T06:14:32.096 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:32.073+0000 7f3a9ffff700 1 -- 172.21.15.42:0/3047218555 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a980030b0 con 0x7f3ab8120ef0 2019-12-05T06:14:32.096 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:32.073+0000 7f3ab77fe700 1 -- 172.21.15.42:0/3047218555 >> v2:172.21.15.40:3300/0 conn(0x7f3ab8121320 msgr2=0x7f3ab8137720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:32.096 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:32.073+0000 7f3ab77fe700 1 --2- 172.21.15.42:0/3047218555 >> v2:172.21.15.40:3300/0 conn(0x7f3ab8121320 0x7f3ab8137720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:32.096 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:32.073+0000 7f3ab77fe700 1 -- 172.21.15.42:0/3047218555 >> v1:172.21.15.42:6789/0 conn(0x7f3ab8120ef0 legacy=0x7f3ab8130840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:32.096 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:32.073+0000 7f3ab77fe700 1 -- 172.21.15.42:0/3047218555 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ab8131040 con 0x7f3ab8059f50 2019-12-05T06:14:32.096 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:32.073+0000 7f3ab7fff700 1 --2- 172.21.15.42:0/3047218555 >> v2:172.21.15.40:3300/0 conn(0x7f3ab8121320 0x7f3ab8137720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:14:32.096 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:32.073+0000 7f3ab77fe700 1 --2- 172.21.15.42:0/3047218555 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3ab8059f50 0x7f3ab80a3c20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f3aa80032b0 tx=0x7f3aa8010dc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:32.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:32.073+0000 7f3ab77fe700 1 --2- 172.21.15.42:0/3047218555 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3ab8059f50 0x7f3ab80a3c20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f3aa80032b0 tx=0x7f3aa8010dc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:32.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:32.073+0000 7f3ab77fe700 1 --2- 172.21.15.42:0/3047218555 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3ab8059f50 0x7f3ab80a3c20 secure :-1 s=READY pgs=508 cs=0 l=1 rx=0x7f3aa80032b0 tx=0x7f3aa8010dc0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:14:32.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:32.073+0000 7f3ab77fe700 1 --2- 172.21.15.42:0/3047218555 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3ab8059f50 0x7f3ab80a3c20 secure :-1 s=THROTTLE_DONE pgs=508 cs=0 l=1 rx=0x7f3aa80032b0 tx=0x7f3aa8010dc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:32.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:32.073+0000 7f3a9ffff700 1 -- 172.21.15.42:0/3047218555 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3aa8007350 con 0x7f3ab8059f50 2019-12-05T06:14:32.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:32.073+0000 7f3ab77fe700 1 --2- 172.21.15.42:0/3047218555 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3ab8059f50 0x7f3ab80a3c20 secure :-1 s=THROTTLE_DONE pgs=508 cs=0 l=1 rx=0x7f3aa80032b0 tx=0x7f3aa8010dc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:32.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:32.073+0000 7f3a9ffff700 1 -- 172.21.15.42:0/3047218555 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3aa8005cb0 con 0x7f3ab8059f50 2019-12-05T06:14:32.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:32.073+0000 7f3ab77fe700 1 --2- 172.21.15.42:0/3047218555 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3ab8059f50 0x7f3ab80a3c20 secure :-1 s=THROTTLE_DONE pgs=508 cs=0 l=1 rx=0x7f3aa80032b0 tx=0x7f3aa8010dc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:32.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:32.073+0000 7f3abd9a2700 1 -- 172.21.15.42:0/3047218555 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ab811f1f0 con 0x7f3ab8059f50 2019-12-05T06:14:32.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:32.073+0000 7f3a9ffff700 1 -- 172.21.15.42:0/3047218555 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3aa8012670 con 0x7f3ab8059f50 2019-12-05T06:14:32.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:32.073+0000 7f3abd9a2700 1 -- 172.21.15.42:0/3047218555 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ab80a4310 con 0x7f3ab8059f50 2019-12-05T06:14:32.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:32.073+0000 7f3ab77fe700 1 --2- 172.21.15.42:0/3047218555 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3ab8059f50 0x7f3ab80a3c20 secure :-1 s=THROTTLE_DONE pgs=508 cs=0 l=1 rx=0x7f3aa80032b0 tx=0x7f3aa8010dc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:32.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:32.077+0000 7f3a9ffff700 1 -- 172.21.15.42:0/3047218555 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f3aa8012950 con 0x7f3ab8059f50 2019-12-05T06:14:32.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:32.077+0000 7f3ab77fe700 1 --2- 172.21.15.42:0/3047218555 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3ab8059f50 0x7f3ab80a3c20 secure :-1 s=THROTTLE_DONE pgs=508 cs=0 l=1 rx=0x7f3aa80032b0 tx=0x7f3aa8010dc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:32.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:32.077+0000 7f3a9ffff700 1 --2- 172.21.15.42:0/3047218555 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3a9803b200 0x7f3a9803d710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:14:32.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:32.077+0000 7f3a9ffff700 1 -- 172.21.15.42:0/3047218555 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 10245+0+0 (secure 0 0 0) 0x7f3aa804b5d0 con 0x7f3ab8059f50 2019-12-05T06:14:32.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:32.077+0000 7f3ab6ffd700 1 --2- 172.21.15.42:0/3047218555 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3a9803b200 0x7f3a9803d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:32.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:32.077+0000 7f3ab6ffd700 1 --2- 172.21.15.42:0/3047218555 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3a9803b200 0x7f3a9803d710 crc :-1 s=READY pgs=977 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:14:32.100 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:32.077+0000 7f3abd9a2700 1 -- 172.21.15.42:0/3047218555 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ab805d110 con 0x7f3ab8059f50 2019-12-05T06:14:32.100 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:32.089+0000 7f3ab77fe700 1 --2- 172.21.15.42:0/3047218555 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3ab8059f50 0x7f3ab80a3c20 secure :-1 s=THROTTLE_DONE pgs=508 cs=0 l=1 rx=0x7f3aa80032b0 tx=0x7f3aa8010dc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:32.100 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:32.089+0000 7f3a9ffff700 1 -- 172.21.15.42:0/3047218555 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f3aa8032590 con 0x7f3ab8059f50 2019-12-05T06:14:32.264 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:32.261+0000 7f3abd9a2700 1 -- 172.21.15.42:0/3047218555 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3ab805b8c0 con 0x7f3ab8059f50 2019-12-05T06:14:32.265 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:32.261+0000 7f3ab77fe700 1 --2- 172.21.15.42:0/3047218555 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3ab8059f50 0x7f3ab80a3c20 secure :-1 s=THROTTLE_DONE pgs=508 cs=0 l=1 rx=0x7f3aa80032b0 tx=0x7f3aa8010dc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:32.265 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:32.261+0000 7f3a9ffff700 1 -- 172.21.15.42:0/3047218555 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 278 v278) v1 ==== 95+0+2115 (secure 0 0 0) 0x7f3aa80327f0 con 0x7f3ab8059f50 2019-12-05T06:14:32.267 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 278 2019-12-05T06:14:32.267 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:14:32.267 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":278,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7217,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/2667648531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2667648531},{"type":"v1","addr":"172.21.15.40:6833","nonce":2667648531}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":278}],"filesystems":[{"mdsmap":{"epoch":277,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:13:19.980391+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":98,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10246},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10246":{"gid":10246,"name":"a","rank":0,"incarnation":274,"state":"up:active","state_seq":17,"addr":"172.21.15.42:6835/4032647860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":4032647860},{"type":"v1","addr":"172.21.15.42:6835","nonce":4032647860}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:14:32.271 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:32.265+0000 7f3abd9a2700 1 -- 172.21.15.42:0/3047218555 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3a9803b200 msgr2=0x7f3a9803d710 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:32.272 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:32.265+0000 7f3abd9a2700 1 --2- 172.21.15.42:0/3047218555 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3a9803b200 0x7f3a9803d710 crc :-1 s=READY pgs=977 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:32.272 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:32.265+0000 7f3abd9a2700 1 -- 172.21.15.42:0/3047218555 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3ab8059f50 msgr2=0x7f3ab80a3c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:32.272 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:32.265+0000 7f3abd9a2700 1 --2- 172.21.15.42:0/3047218555 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3ab8059f50 0x7f3ab80a3c20 secure :-1 s=READY pgs=508 cs=0 l=1 rx=0x7f3aa80032b0 tx=0x7f3aa8010dc0).stop 2019-12-05T06:14:32.272 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:32.265+0000 7f3abd9a2700 1 -- 172.21.15.42:0/3047218555 shutdown_connections 2019-12-05T06:14:32.273 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:32.265+0000 7f3abd9a2700 1 --2- 172.21.15.42:0/3047218555 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3a9803b200 0x7f3a9803d710 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:32.273 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:32.265+0000 7f3abd9a2700 1 --2- 172.21.15.42:0/3047218555 >> v2:172.21.15.40:3300/0 conn(0x7f3ab8121320 0x7f3ab8137720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:32.273 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:32.265+0000 7f3abd9a2700 1 --2- 172.21.15.42:0/3047218555 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3ab8059f50 0x7f3ab80a3c20 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:32.273 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:32.265+0000 7f3abd9a2700 1 -- 172.21.15.42:0/3047218555 shutdown_connections 2019-12-05T06:14:32.274 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:32.265+0000 7f3abd9a2700 1 -- 172.21.15.42:0/3047218555 wait complete. 2019-12-05T06:14:32.274 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:32.265+0000 7f3abd9a2700 1 -- 172.21.15.42:0/3047218555 >> 172.21.15.42:0/3047218555 conn(0x7f3ab809f2f0 msgr2=0x7f3ab811c3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:14:32.281 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:14:34.283 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:14:34.398 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:34.389+0000 7f32dfa0b700 1 Processor -- start 2019-12-05T06:14:34.398 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:34.389+0000 7f32dfa0b700 1 -- start start 2019-12-05T06:14:34.398 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:34.389+0000 7f32dfa0b700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f32d80a7740 0x7f32d808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:14:34.399 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:34.389+0000 7f32dfa0b700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f32d807e720 0x7f32d807eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:14:34.399 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:34.389+0000 7f32dfa0b700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32d80a0710 con 0x7f32d809f2e0 2019-12-05T06:14:34.399 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:34.389+0000 7f32dfa0b700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32d80a0b00 con 0x7f32d807e720 2019-12-05T06:14:34.399 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:34.389+0000 7f32dfa0b700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32d808e5b0 con 0x7f32d80a7740 2019-12-05T06:14:34.399 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:34.389+0000 7f32dd7a7700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f32d807e720 0x7f32d807eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:34.399 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:34.389+0000 7f32dd7a7700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f32d807e720 0x7f32d807eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:51308/0 (socket says 172.21.15.42:51308) 2019-12-05T06:14:34.399 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:34.389+0000 7f32dd7a7700 1 -- 172.21.15.42:0/2452284726 learned_addr learned my addr 172.21.15.42:0/2452284726 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:14:34.400 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:34.389+0000 7f32ddfa8700 1 --2- 172.21.15.42:0/2452284726 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f32d80a7740 0x7f32d808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:34.400 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:34.389+0000 7f32dd7a7700 1 -- 172.21.15.42:0/2452284726 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f32d80a7740 msgr2=0x7f32d808df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:14:34.400 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:34.389+0000 7f32dd7a7700 1 --2- 172.21.15.42:0/2452284726 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f32d80a7740 0x7f32d808df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:14:34.400 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:34.389+0000 7f32dd7a7700 1 -- 172.21.15.42:0/2452284726 >> v1:172.21.15.42:6789/0 conn(0x7f32d809f2e0 legacy=0x7f32d809f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:34.400 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:34.389+0000 7f32dd7a7700 1 -- 172.21.15.42:0/2452284726 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32d807f0a0 con 0x7f32d807e720 2019-12-05T06:14:34.400 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:34.389+0000 7f32dcfa6700 1 -- 172.21.15.42:0/2452284726 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3861989624 0 0) 0x7f32d80a0710 con 0x7f32d809f2e0 2019-12-05T06:14:34.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:34.389+0000 7f32dd7a7700 1 --2- 172.21.15.42:0/2452284726 >> v2:172.21.15.42:3300/0 conn(0x7f32d807e720 0x7f32d807eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f32c8008790 tx=0x7f32c800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:34.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:34.389+0000 7f32dd7a7700 1 --2- 172.21.15.42:0/2452284726 >> v2:172.21.15.42:3300/0 conn(0x7f32d807e720 0x7f32d807eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f32c8008790 tx=0x7f32c800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:34.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:34.389+0000 7f32dd7a7700 1 --2- 172.21.15.42:0/2452284726 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f32d807e720 0x7f32d807eb50 secure :-1 s=READY pgs=2326 cs=0 l=1 rx=0x7f32c8008790 tx=0x7f32c800fd30).ready entity=mon.1 client_cookie=aaa09fb7842bcdc1 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:14:34.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:34.389+0000 7f32dd7a7700 1 --2- 172.21.15.42:0/2452284726 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f32d807e720 0x7f32d807eb50 secure :-1 s=THROTTLE_DONE pgs=2326 cs=0 l=1 rx=0x7f32c8008790 tx=0x7f32c800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:34.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:34.389+0000 7f32dcfa6700 1 -- 172.21.15.42:0/2452284726 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f32c80113b0 con 0x7f32d807e720 2019-12-05T06:14:34.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:34.389+0000 7f32dd7a7700 1 --2- 172.21.15.42:0/2452284726 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f32d807e720 0x7f32d807eb50 secure :-1 s=THROTTLE_DONE pgs=2326 cs=0 l=1 rx=0x7f32c8008790 tx=0x7f32c800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:34.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:34.389+0000 7f32dd7a7700 1 --2- 172.21.15.42:0/2452284726 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f32d807e720 0x7f32d807eb50 secure :-1 s=THROTTLE_DONE pgs=2326 cs=0 l=1 rx=0x7f32c8008790 tx=0x7f32c800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:34.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:34.389+0000 7f32dcfa6700 1 -- 172.21.15.42:0/2452284726 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32c8008bb0 con 0x7f32d807e720 2019-12-05T06:14:34.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:34.389+0000 7f32dcfa6700 1 -- 172.21.15.42:0/2452284726 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f32c8011990 con 0x7f32d807e720 2019-12-05T06:14:34.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:34.389+0000 7f32dfa0b700 1 -- 172.21.15.42:0/2452284726 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f32d807e720 msgr2=0x7f32d807eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:34.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:34.393+0000 7f32dfa0b700 1 --2- 172.21.15.42:0/2452284726 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f32d807e720 0x7f32d807eb50 secure :-1 s=READY pgs=2326 cs=0 l=1 rx=0x7f32c8008790 tx=0x7f32c800fd30).stop 2019-12-05T06:14:34.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:34.393+0000 7f32dfa0b700 1 -- 172.21.15.42:0/2452284726 shutdown_connections 2019-12-05T06:14:34.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:34.393+0000 7f32dfa0b700 1 --2- 172.21.15.42:0/2452284726 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f32d80a7740 0x7f32d808df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:14:34.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:34.393+0000 7f32dfa0b700 1 --2- 172.21.15.42:0/2452284726 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f32d807e720 0x7f32d807eb50 unknown :-1 s=CLOSED pgs=2326 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:34.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:34.393+0000 7f32dfa0b700 1 -- 172.21.15.42:0/2452284726 shutdown_connections 2019-12-05T06:14:34.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:34.393+0000 7f32dfa0b700 1 -- 172.21.15.42:0/2452284726 wait complete. 2019-12-05T06:14:34.404 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:34.393+0000 7f32dfa0b700 1 -- 172.21.15.42:0/2452284726 >> 172.21.15.42:0/2452284726 conn(0x7f32d80aabe0 msgr2=0x7f32d80a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:14:34.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:34.393+0000 7f32dfa0b700 1 Processor -- start 2019-12-05T06:14:34.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:34.393+0000 7f32dfa0b700 1 -- start start 2019-12-05T06:14:34.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:34.397+0000 7f32dfa0b700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f32d8124200 0x7f32d811be70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:14:34.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:34.397+0000 7f32dfa0b700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f32d80a36d0 0x7f32d807d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:14:34.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:34.397+0000 7f32dfa0b700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f32d80a20e0 0x7f32d80a2510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:14:34.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:34.397+0000 7f32dfa0b700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32d81300b0 con 0x7f32d8124200 2019-12-05T06:14:34.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:34.397+0000 7f32dfa0b700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32d80a3ec0 con 0x7f32d80a36d0 2019-12-05T06:14:34.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:34.397+0000 7f32dfa0b700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32d80a4160 con 0x7f32d80a20e0 2019-12-05T06:14:34.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:34.397+0000 7f32dd7a7700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f32d80a36d0 0x7f32d807d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:34.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:34.397+0000 7f32dd7a7700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f32d80a36d0 0x7f32d807d920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:51310/0 (socket says 172.21.15.42:51310) 2019-12-05T06:14:34.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:34.397+0000 7f32dd7a7700 1 -- 172.21.15.42:0/773817865 learned_addr learned my addr 172.21.15.42:0/773817865 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:14:34.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:34.397+0000 7f32ddfa8700 1 --2- 172.21.15.42:0/773817865 >> v2:172.21.15.40:3300/0 conn(0x7f32d8124200 0x7f32d811be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:34.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:34.397+0000 7f32de7a9700 1 --2- 172.21.15.42:0/773817865 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f32d80a20e0 0x7f32d80a2510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:34.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:34.397+0000 7f32dd7a7700 1 -- 172.21.15.42:0/773817865 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f32d80a20e0 msgr2=0x7f32d80a2510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:34.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:34.397+0000 7f32dd7a7700 1 --2- 172.21.15.42:0/773817865 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f32d80a20e0 0x7f32d80a2510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:34.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:34.397+0000 7f32dd7a7700 1 -- 172.21.15.42:0/773817865 >> v2:172.21.15.40:3300/0 conn(0x7f32d8124200 msgr2=0x7f32d811be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:34.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:34.397+0000 7f32dd7a7700 1 --2- 172.21.15.42:0/773817865 >> v2:172.21.15.40:3300/0 conn(0x7f32d8124200 0x7f32d811be70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:34.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:34.397+0000 7f32dd7a7700 1 -- 172.21.15.42:0/773817865 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32d80a2a60 con 0x7f32d80a36d0 2019-12-05T06:14:34.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:34.397+0000 7f32dd7a7700 1 --2- 172.21.15.42:0/773817865 >> v2:172.21.15.42:3300/0 conn(0x7f32d80a36d0 0x7f32d807d920 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f32d8125060 tx=0x7f32c8008c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:34.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:34.397+0000 7f32dd7a7700 1 --2- 172.21.15.42:0/773817865 >> v2:172.21.15.42:3300/0 conn(0x7f32d80a36d0 0x7f32d807d920 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f32d8125060 tx=0x7f32c8008c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:34.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:34.397+0000 7f32dd7a7700 1 --2- 172.21.15.42:0/773817865 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f32d80a36d0 0x7f32d807d920 secure :-1 s=READY pgs=2327 cs=0 l=1 rx=0x7f32d8125060 tx=0x7f32c8008c30).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:14:34.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:34.397+0000 7f32dd7a7700 1 --2- 172.21.15.42:0/773817865 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f32d80a36d0 0x7f32d807d920 secure :-1 s=THROTTLE_DONE pgs=2327 cs=0 l=1 rx=0x7f32d8125060 tx=0x7f32c8008c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:34.411 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:34.397+0000 7f32dd7a7700 1 --2- 172.21.15.42:0/773817865 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f32d80a36d0 0x7f32d807d920 secure :-1 s=THROTTLE_DONE pgs=2327 cs=0 l=1 rx=0x7f32d8125060 tx=0x7f32c8008c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:34.411 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:34.397+0000 7f32ce7fc700 1 -- 172.21.15.42:0/773817865 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f32c8007150 con 0x7f32d80a36d0 2019-12-05T06:14:34.411 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:34.397+0000 7f32dd7a7700 1 --2- 172.21.15.42:0/773817865 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f32d80a36d0 0x7f32d807d920 secure :-1 s=THROTTLE_DONE pgs=2327 cs=0 l=1 rx=0x7f32d8125060 tx=0x7f32c8008c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:34.411 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:34.397+0000 7f32ce7fc700 1 -- 172.21.15.42:0/773817865 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32c80115b0 con 0x7f32d80a36d0 2019-12-05T06:14:34.411 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:34.397+0000 7f32dfa0b700 1 -- 172.21.15.42:0/773817865 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32d80a3b00 con 0x7f32d80a36d0 2019-12-05T06:14:34.411 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:34.397+0000 7f32ce7fc700 1 -- 172.21.15.42:0/773817865 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f32c8005730 con 0x7f32d80a36d0 2019-12-05T06:14:34.411 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:34.397+0000 7f32dfa0b700 1 -- 172.21.15.42:0/773817865 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32d807de70 con 0x7f32d80a36d0 2019-12-05T06:14:34.412 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:34.401+0000 7f32dd7a7700 1 --2- 172.21.15.42:0/773817865 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f32d80a36d0 0x7f32d807d920 secure :-1 s=THROTTLE_DONE pgs=2327 cs=0 l=1 rx=0x7f32d8125060 tx=0x7f32c8008c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:34.412 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:34.401+0000 7f32ce7fc700 1 -- 172.21.15.42:0/773817865 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f32c8005930 con 0x7f32d80a36d0 2019-12-05T06:14:34.412 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:34.401+0000 7f32dd7a7700 1 --2- 172.21.15.42:0/773817865 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f32d80a36d0 0x7f32d807d920 secure :-1 s=THROTTLE_DONE pgs=2327 cs=0 l=1 rx=0x7f32d8125060 tx=0x7f32c8008c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:34.412 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:34.401+0000 7f32ce7fc700 1 --2- 172.21.15.42:0/773817865 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f32c003ad40 0x7f32c003d250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:14:34.412 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:34.401+0000 7f32ce7fc700 1 -- 172.21.15.42:0/773817865 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 10245+0+0 (secure 0 0 0) 0x7f32c804b4e0 con 0x7f32d80a36d0 2019-12-05T06:14:34.412 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:34.401+0000 7f32ddfa8700 1 --2- 172.21.15.42:0/773817865 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f32c003ad40 0x7f32c003d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:34.412 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:34.401+0000 7f32ddfa8700 1 --2- 172.21.15.42:0/773817865 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f32c003ad40 0x7f32c003d250 crc :-1 s=READY pgs=978 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:14:34.414 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:34.405+0000 7f32dfa0b700 1 -- 172.21.15.42:0/773817865 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32d805d110 con 0x7f32d80a36d0 2019-12-05T06:14:34.417 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:34.413+0000 7f32dd7a7700 1 --2- 172.21.15.42:0/773817865 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f32d80a36d0 0x7f32d807d920 secure :-1 s=THROTTLE_DONE pgs=2327 cs=0 l=1 rx=0x7f32d8125060 tx=0x7f32c8008c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:34.417 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:34.413+0000 7f32ce7fc700 1 -- 172.21.15.42:0/773817865 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f32c8032560 con 0x7f32d80a36d0 2019-12-05T06:14:34.587 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:34.581+0000 7f32dfa0b700 1 -- 172.21.15.42:0/773817865 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f32d805b8c0 con 0x7f32d80a36d0 2019-12-05T06:14:34.588 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:34.585+0000 7f32dd7a7700 1 --2- 172.21.15.42:0/773817865 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f32d80a36d0 0x7f32d807d920 secure :-1 s=THROTTLE_DONE pgs=2327 cs=0 l=1 rx=0x7f32d8125060 tx=0x7f32c8008c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:34.588 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:34.585+0000 7f32ce7fc700 1 -- 172.21.15.42:0/773817865 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 278 v278) v1 ==== 95+0+2115 (secure 0 0 0) 0x7f32c80327c0 con 0x7f32d80a36d0 2019-12-05T06:14:34.588 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 278 2019-12-05T06:14:34.589 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:14:34.589 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":278,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7217,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/2667648531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2667648531},{"type":"v1","addr":"172.21.15.40:6833","nonce":2667648531}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":278}],"filesystems":[{"mdsmap":{"epoch":277,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:13:19.980391+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":98,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10246},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10246":{"gid":10246,"name":"a","rank":0,"incarnation":274,"state":"up:active","state_seq":17,"addr":"172.21.15.42:6835/4032647860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":4032647860},{"type":"v1","addr":"172.21.15.42:6835","nonce":4032647860}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:14:34.592 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:34.585+0000 7f32dfa0b700 1 -- 172.21.15.42:0/773817865 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f32c003ad40 msgr2=0x7f32c003d250 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:34.593 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:34.585+0000 7f32dfa0b700 1 --2- 172.21.15.42:0/773817865 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f32c003ad40 0x7f32c003d250 crc :-1 s=READY pgs=978 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:34.593 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:34.585+0000 7f32dfa0b700 1 -- 172.21.15.42:0/773817865 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f32d80a36d0 msgr2=0x7f32d807d920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:34.593 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:34.585+0000 7f32dfa0b700 1 --2- 172.21.15.42:0/773817865 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f32d80a36d0 0x7f32d807d920 secure :-1 s=READY pgs=2327 cs=0 l=1 rx=0x7f32d8125060 tx=0x7f32c8008c30).stop 2019-12-05T06:14:34.593 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:34.585+0000 7f32dfa0b700 1 -- 172.21.15.42:0/773817865 shutdown_connections 2019-12-05T06:14:34.593 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:34.585+0000 7f32dfa0b700 1 --2- 172.21.15.42:0/773817865 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f32c003ad40 0x7f32c003d250 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:34.594 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:34.585+0000 7f32dfa0b700 1 --2- 172.21.15.42:0/773817865 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f32d80a20e0 0x7f32d80a2510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:34.594 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:34.585+0000 7f32dfa0b700 1 --2- 172.21.15.42:0/773817865 >> v2:172.21.15.40:3300/0 conn(0x7f32d8124200 0x7f32d811be70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:34.594 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:34.585+0000 7f32dfa0b700 1 --2- 172.21.15.42:0/773817865 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f32d80a36d0 0x7f32d807d920 unknown :-1 s=CLOSED pgs=2327 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:34.594 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:34.585+0000 7f32dfa0b700 1 -- 172.21.15.42:0/773817865 shutdown_connections 2019-12-05T06:14:34.594 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:34.585+0000 7f32dfa0b700 1 -- 172.21.15.42:0/773817865 wait complete. 2019-12-05T06:14:34.594 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:34.585+0000 7f32dfa0b700 1 -- 172.21.15.42:0/773817865 >> 172.21.15.42:0/773817865 conn(0x7f32d80a6bb0 msgr2=0x7f32d807e5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:14:34.606 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:14:36.607 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:14:36.715 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:36.705+0000 7fc879f91700 1 Processor -- start 2019-12-05T06:14:36.715 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:36.705+0000 7fc879f91700 1 -- start start 2019-12-05T06:14:36.716 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:36.705+0000 7fc879f91700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc8740a7730 0x7fc87408df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:14:36.716 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:36.705+0000 7fc879f91700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc87409fad0 0x7fc87407eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:14:36.716 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:36.705+0000 7fc879f91700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8740a0700 con 0x7fc87407e710 2019-12-05T06:14:36.716 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:36.705+0000 7fc879f91700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8740a0af0 con 0x7fc8740a7730 2019-12-05T06:14:36.716 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:36.705+0000 7fc879f91700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc87408e5a0 con 0x7fc87409fad0 2019-12-05T06:14:36.716 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:36.705+0000 7fc86b7fe700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fc87407e710 0x7fc87409f2d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:51356/0 (socket says 172.21.15.42:51356) 2019-12-05T06:14:36.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:36.705+0000 7fc86b7fe700 1 -- 172.21.15.42:0/2435582516 learned_addr learned my addr 172.21.15.42:0/2435582516 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:14:36.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:36.705+0000 7fc873fff700 1 --2- 172.21.15.42:0/2435582516 >> v2:172.21.15.40:3300/0 conn(0x7fc8740a7730 0x7fc87408df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:36.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:36.705+0000 7fc878d2f700 1 --2- 172.21.15.42:0/2435582516 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc87409fad0 0x7fc87407eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:36.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:36.705+0000 7fc8737fe700 1 -- 172.21.15.42:0/2435582516 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1293851483 0 0) 0x7fc8740a0700 con 0x7fc87407e710 2019-12-05T06:14:36.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:36.705+0000 7fc8737fe700 1 -- 172.21.15.42:0/2435582516 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8600030b0 con 0x7fc87407e710 2019-12-05T06:14:36.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:36.709+0000 7fc878d2f700 1 -- 172.21.15.42:0/2435582516 >> v2:172.21.15.40:3300/0 conn(0x7fc8740a7730 msgr2=0x7fc87408df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:14:36.718 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:36.709+0000 7fc878d2f700 1 --2- 172.21.15.42:0/2435582516 >> v2:172.21.15.40:3300/0 conn(0x7fc8740a7730 0x7fc87408df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:14:36.718 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:36.709+0000 7fc878d2f700 1 -- 172.21.15.42:0/2435582516 >> v1:172.21.15.42:6789/0 conn(0x7fc87407e710 legacy=0x7fc87409f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:36.718 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:36.709+0000 7fc878d2f700 1 -- 172.21.15.42:0/2435582516 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc87407f090 con 0x7fc87409fad0 2019-12-05T06:14:36.718 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:36.709+0000 7fc878d2f700 1 --2- 172.21.15.42:0/2435582516 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc87409fad0 0x7fc87407eb40 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fc86c008b10 tx=0x7fc86c00fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:36.718 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:36.709+0000 7fc878d2f700 1 --2- 172.21.15.42:0/2435582516 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc87409fad0 0x7fc87407eb40 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fc86c008b10 tx=0x7fc86c00fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:36.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:36.709+0000 7fc878d2f700 1 --2- 172.21.15.42:0/2435582516 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc87409fad0 0x7fc87407eb40 secure :-1 s=READY pgs=509 cs=0 l=1 rx=0x7fc86c008b10 tx=0x7fc86c00fff0).ready entity=mon.2 client_cookie=8573a1ff3f54a524 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:14:36.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:36.709+0000 7fc878d2f700 1 --2- 172.21.15.42:0/2435582516 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc87409fad0 0x7fc87407eb40 secure :-1 s=THROTTLE_DONE pgs=509 cs=0 l=1 rx=0x7fc86c008b10 tx=0x7fc86c00fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:36.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:36.709+0000 7fc8737fe700 1 -- 172.21.15.42:0/2435582516 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc86c010900 con 0x7fc87409fad0 2019-12-05T06:14:36.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:36.709+0000 7fc878d2f700 1 --2- 172.21.15.42:0/2435582516 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc87409fad0 0x7fc87407eb40 secure :-1 s=THROTTLE_DONE pgs=509 cs=0 l=1 rx=0x7fc86c008b10 tx=0x7fc86c00fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:36.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:36.709+0000 7fc8737fe700 1 -- 172.21.15.42:0/2435582516 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc86c008b50 con 0x7fc87409fad0 2019-12-05T06:14:36.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:36.709+0000 7fc878d2f700 1 --2- 172.21.15.42:0/2435582516 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc87409fad0 0x7fc87407eb40 secure :-1 s=THROTTLE_DONE pgs=509 cs=0 l=1 rx=0x7fc86c008b10 tx=0x7fc86c00fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:36.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:36.709+0000 7fc8737fe700 1 -- 172.21.15.42:0/2435582516 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc86c0110c0 con 0x7fc87409fad0 2019-12-05T06:14:36.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:36.709+0000 7fc879f91700 1 -- 172.21.15.42:0/2435582516 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc87409fad0 msgr2=0x7fc87407eb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:36.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:36.709+0000 7fc879f91700 1 --2- 172.21.15.42:0/2435582516 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc87409fad0 0x7fc87407eb40 secure :-1 s=READY pgs=509 cs=0 l=1 rx=0x7fc86c008b10 tx=0x7fc86c00fff0).stop 2019-12-05T06:14:36.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:36.709+0000 7fc879f91700 1 -- 172.21.15.42:0/2435582516 shutdown_connections 2019-12-05T06:14:36.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:36.709+0000 7fc879f91700 1 --2- 172.21.15.42:0/2435582516 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc87409fad0 0x7fc87407eb40 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:36.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:36.709+0000 7fc879f91700 1 --2- 172.21.15.42:0/2435582516 >> v2:172.21.15.40:3300/0 conn(0x7fc8740a7730 0x7fc87408df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:14:36.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:36.709+0000 7fc879f91700 1 -- 172.21.15.42:0/2435582516 shutdown_connections 2019-12-05T06:14:36.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:36.709+0000 7fc879f91700 1 -- 172.21.15.42:0/2435582516 wait complete. 2019-12-05T06:14:36.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:36.709+0000 7fc879f91700 1 -- 172.21.15.42:0/2435582516 >> 172.21.15.42:0/2435582516 conn(0x7fc8740aabd0 msgr2=0x7fc8740a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:14:36.724 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:36.713+0000 7fc879f91700 1 Processor -- start 2019-12-05T06:14:36.724 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:36.713+0000 7fc879f91700 1 -- start start 2019-12-05T06:14:36.724 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:36.713+0000 7fc879f91700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc8741164f0 0x7fc87407ae70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:14:36.725 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:36.713+0000 7fc879f91700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc874133760 0x7fc874133b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:14:36.725 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:36.713+0000 7fc879f91700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc87407b3c0 con 0x7fc8740a6de0 2019-12-05T06:14:36.725 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:36.713+0000 7fc879f91700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8740a6500 con 0x7fc8741164f0 2019-12-05T06:14:36.725 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:36.713+0000 7fc879f91700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8741340e0 con 0x7fc874133760 2019-12-05T06:14:36.725 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:36.713+0000 7fc86b7fe700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc8741164f0 0x7fc87407ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:36.725 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:36.713+0000 7fc86b7fe700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc8741164f0 0x7fc87407ae70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.42:39142/0 (socket says 172.21.15.42:39142) 2019-12-05T06:14:36.725 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:36.713+0000 7fc86b7fe700 1 -- 172.21.15.42:0/3056107885 learned_addr learned my addr 172.21.15.42:0/3056107885 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:14:36.726 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:36.713+0000 7fc878d2f700 1 --2- 172.21.15.42:0/3056107885 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc874133760 0x7fc874133b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:36.726 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:36.713+0000 7fc878d2f700 1 -- 172.21.15.42:0/3056107885 >> v2:172.21.15.40:3300/0 conn(0x7fc8741164f0 msgr2=0x7fc87407ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:36.726 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:36.713+0000 7fc878d2f700 1 --2- 172.21.15.42:0/3056107885 >> v2:172.21.15.40:3300/0 conn(0x7fc8741164f0 0x7fc87407ae70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:36.726 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:36.713+0000 7fc878d2f700 1 -- 172.21.15.42:0/3056107885 >> v1:172.21.15.40:6789/0 conn(0x7fc8740a6de0 legacy=0x7fc8740a5d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:36.726 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:36.713+0000 7fc878d2f700 1 -- 172.21.15.42:0/3056107885 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc87413ba40 con 0x7fc874133760 2019-12-05T06:14:36.726 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:36.713+0000 7fc878d2f700 1 --2- 172.21.15.42:0/3056107885 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc874133760 0x7fc874133b90 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fc86c008b10 tx=0x7fc86c010900).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:36.727 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:36.713+0000 7fc878d2f700 1 --2- 172.21.15.42:0/3056107885 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc874133760 0x7fc874133b90 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fc86c008b10 tx=0x7fc86c010900).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:36.727 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:36.713+0000 7fc878d2f700 1 --2- 172.21.15.42:0/3056107885 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc874133760 0x7fc874133b90 secure :-1 s=READY pgs=510 cs=0 l=1 rx=0x7fc86c008b10 tx=0x7fc86c010900).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:14:36.727 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:36.713+0000 7fc878d2f700 1 --2- 172.21.15.42:0/3056107885 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc874133760 0x7fc874133b90 secure :-1 s=THROTTLE_DONE pgs=510 cs=0 l=1 rx=0x7fc86c008b10 tx=0x7fc86c010900).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:36.727 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:36.713+0000 7fc8717fa700 1 -- 172.21.15.42:0/3056107885 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc86c005ab0 con 0x7fc874133760 2019-12-05T06:14:36.727 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:36.713+0000 7fc878d2f700 1 --2- 172.21.15.42:0/3056107885 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc874133760 0x7fc874133b90 secure :-1 s=THROTTLE_DONE pgs=510 cs=0 l=1 rx=0x7fc86c008b10 tx=0x7fc86c010900).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:36.727 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:36.713+0000 7fc879f91700 1 -- 172.21.15.42:0/3056107885 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc87413aa30 con 0x7fc874133760 2019-12-05T06:14:36.727 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:36.713+0000 7fc8717fa700 1 -- 172.21.15.42:0/3056107885 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc86c010b00 con 0x7fc874133760 2019-12-05T06:14:36.728 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:36.713+0000 7fc878d2f700 1 --2- 172.21.15.42:0/3056107885 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc874133760 0x7fc874133b90 secure :-1 s=THROTTLE_DONE pgs=510 cs=0 l=1 rx=0x7fc86c008b10 tx=0x7fc86c010900).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:36.728 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:36.713+0000 7fc8717fa700 1 -- 172.21.15.42:0/3056107885 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc86c00f470 con 0x7fc874133760 2019-12-05T06:14:36.728 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:36.713+0000 7fc879f91700 1 -- 172.21.15.42:0/3056107885 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc87413af30 con 0x7fc874133760 2019-12-05T06:14:36.728 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:36.717+0000 7fc878d2f700 1 --2- 172.21.15.42:0/3056107885 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc874133760 0x7fc874133b90 secure :-1 s=THROTTLE_DONE pgs=510 cs=0 l=1 rx=0x7fc86c008b10 tx=0x7fc86c010900).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:36.728 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:36.717+0000 7fc878d2f700 1 --2- 172.21.15.42:0/3056107885 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc874133760 0x7fc874133b90 secure :-1 s=THROTTLE_DONE pgs=510 cs=0 l=1 rx=0x7fc86c008b10 tx=0x7fc86c010900).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:36.728 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:36.717+0000 7fc8717fa700 1 -- 172.21.15.42:0/3056107885 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fc86c00f750 con 0x7fc874133760 2019-12-05T06:14:36.728 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:36.717+0000 7fc8717fa700 1 --2- 172.21.15.42:0/3056107885 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc85803ad60 0x7fc85803d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:14:36.729 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:36.717+0000 7fc8717fa700 1 -- 172.21.15.42:0/3056107885 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 10331+0+0 (secure 0 0 0) 0x7fc86c08f1f0 con 0x7fc874133760 2019-12-05T06:14:36.729 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:36.717+0000 7fc873fff700 1 --2- 172.21.15.42:0/3056107885 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc85803ad60 0x7fc85803d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:36.729 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:36.717+0000 7fc873fff700 1 --2- 172.21.15.42:0/3056107885 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc85803ad60 0x7fc85803d270 crc :-1 s=READY pgs=979 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:14:36.729 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:36.721+0000 7fc879f91700 1 -- 172.21.15.42:0/3056107885 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc87405d100 con 0x7fc874133760 2019-12-05T06:14:36.733 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:36.729+0000 7fc878d2f700 1 --2- 172.21.15.42:0/3056107885 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc874133760 0x7fc874133b90 secure :-1 s=THROTTLE_DONE pgs=510 cs=0 l=1 rx=0x7fc86c008b10 tx=0x7fc86c010900).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:36.733 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:36.729+0000 7fc8717fa700 1 -- 172.21.15.42:0/3056107885 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fc86c077020 con 0x7fc874133760 2019-12-05T06:14:36.905 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:36.897+0000 7fc879f91700 1 -- 172.21.15.42:0/3056107885 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc87405b8b0 con 0x7fc874133760 2019-12-05T06:14:36.905 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:36.901+0000 7fc878d2f700 1 --2- 172.21.15.42:0/3056107885 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc874133760 0x7fc874133b90 secure :-1 s=THROTTLE_DONE pgs=510 cs=0 l=1 rx=0x7fc86c008b10 tx=0x7fc86c010900).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:36.905 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:36.901+0000 7fc8717fa700 1 -- 172.21.15.42:0/3056107885 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 279 v279) v1 ==== 95+0+1774 (secure 0 0 0) 0x7fc86c076730 con 0x7fc874133760 2019-12-05T06:14:36.906 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 279 2019-12-05T06:14:36.906 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:14:36.906 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":279,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":279,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:14:34.629090+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":99,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7217},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7217":{"gid":7217,"name":"b","rank":0,"incarnation":279,"state":"up:replay","state_seq":2,"addr":"172.21.15.40:6833/2667648531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2667648531},{"type":"v1","addr":"172.21.15.40:6833","nonce":2667648531}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:14:36.907 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:36.901+0000 7fc879f91700 1 -- 172.21.15.42:0/3056107885 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc85803ad60 msgr2=0x7fc85803d270 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:36.908 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:36.901+0000 7fc879f91700 1 --2- 172.21.15.42:0/3056107885 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc85803ad60 0x7fc85803d270 crc :-1 s=READY pgs=979 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:36.908 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:36.901+0000 7fc879f91700 1 -- 172.21.15.42:0/3056107885 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc874133760 msgr2=0x7fc874133b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:36.908 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:36.901+0000 7fc879f91700 1 --2- 172.21.15.42:0/3056107885 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc874133760 0x7fc874133b90 secure :-1 s=READY pgs=510 cs=0 l=1 rx=0x7fc86c008b10 tx=0x7fc86c010900).stop 2019-12-05T06:14:36.908 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:36.901+0000 7fc879f91700 1 -- 172.21.15.42:0/3056107885 shutdown_connections 2019-12-05T06:14:36.908 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:36.901+0000 7fc879f91700 1 --2- 172.21.15.42:0/3056107885 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc85803ad60 0x7fc85803d270 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:36.909 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:36.901+0000 7fc879f91700 1 --2- 172.21.15.42:0/3056107885 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc874133760 0x7fc874133b90 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:36.909 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:36.901+0000 7fc879f91700 1 --2- 172.21.15.42:0/3056107885 >> v2:172.21.15.40:3300/0 conn(0x7fc8741164f0 0x7fc87407ae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:36.909 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:36.901+0000 7fc879f91700 1 -- 172.21.15.42:0/3056107885 shutdown_connections 2019-12-05T06:14:36.909 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:36.901+0000 7fc879f91700 1 -- 172.21.15.42:0/3056107885 wait complete. 2019-12-05T06:14:36.910 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:36.901+0000 7fc879f91700 1 -- 172.21.15.42:0/3056107885 >> 172.21.15.42:0/3056107885 conn(0x7fc87411d330 msgr2=0x7fc8740596f0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:14:36.917 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a down, removed from mdsmap 2019-12-05T06:14:36.918 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2019-12-05T06:14:36.918 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:14:37.016 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:37.005+0000 7f986024f700 1 Processor -- start 2019-12-05T06:14:37.017 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:37.005+0000 7f986024f700 1 -- start start 2019-12-05T06:14:37.017 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:37.009+0000 7f986024f700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f98580a7740 0x7f985808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:14:37.017 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:37.009+0000 7f986024f700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f985809fae0 0x7f985807eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:14:37.018 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:37.009+0000 7f986024f700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98580a0710 con 0x7f985807e720 2019-12-05T06:14:37.018 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:37.009+0000 7f986024f700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98580a0b00 con 0x7f98580a7740 2019-12-05T06:14:37.018 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:37.009+0000 7f986024f700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f985808e5b0 con 0x7f985809fae0 2019-12-05T06:14:37.021 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:37.009+0000 7f985e7ec700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f98580a7740 0x7f985808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:37.021 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:37.009+0000 7f985dfeb700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f985807e720 0x7f985809f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:51370/0 (socket says 172.21.15.42:51370) 2019-12-05T06:14:37.022 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:37.009+0000 7f985e7ec700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f98580a7740 0x7f985808df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:51332/0 (socket says 172.21.15.42:51332) 2019-12-05T06:14:37.022 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:37.009+0000 7f985dfeb700 1 -- 172.21.15.42:0/3204522718 learned_addr learned my addr 172.21.15.42:0/3204522718 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:14:37.022 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:37.009+0000 7f985efed700 1 --2- 172.21.15.42:0/3204522718 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f985809fae0 0x7f985807eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:37.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:37.009+0000 7f985e7ec700 1 -- 172.21.15.42:0/3204522718 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f985809fae0 msgr2=0x7f985807eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:14:37.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:37.009+0000 7f985e7ec700 1 --2- 172.21.15.42:0/3204522718 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f985809fae0 0x7f985807eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:14:37.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:37.009+0000 7f985e7ec700 1 -- 172.21.15.42:0/3204522718 >> v1:172.21.15.42:6789/0 conn(0x7f985807e720 legacy=0x7f985809f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:37.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:37.009+0000 7f985e7ec700 1 -- 172.21.15.42:0/3204522718 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f985807f0a0 con 0x7f98580a7740 2019-12-05T06:14:37.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:37.009+0000 7f985e7ec700 1 --2- 172.21.15.42:0/3204522718 >> v2:172.21.15.42:3300/0 conn(0x7f98580a7740 0x7f985808df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f9854008790 tx=0x7f985400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:37.024 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:37.009+0000 7f985e7ec700 1 --2- 172.21.15.42:0/3204522718 >> v2:172.21.15.42:3300/0 conn(0x7f98580a7740 0x7f985808df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f9854008790 tx=0x7f985400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:37.024 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:37.009+0000 7f985e7ec700 1 --2- 172.21.15.42:0/3204522718 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f98580a7740 0x7f985808df80 secure :-1 s=READY pgs=2330 cs=0 l=1 rx=0x7f9854008790 tx=0x7f985400fc70).ready entity=mon.1 client_cookie=e702ee3046bf28f8 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:14:37.024 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:37.009+0000 7f985e7ec700 1 --2- 172.21.15.42:0/3204522718 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f98580a7740 0x7f985808df80 secure :-1 s=THROTTLE_DONE pgs=2330 cs=0 l=1 rx=0x7f9854008790 tx=0x7f985400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:37.025 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:37.009+0000 7f985d7ea700 1 -- 172.21.15.42:0/3204522718 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9854008bb0 con 0x7f98580a7740 2019-12-05T06:14:37.025 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:37.009+0000 7f985e7ec700 1 --2- 172.21.15.42:0/3204522718 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f98580a7740 0x7f985808df80 secure :-1 s=THROTTLE_DONE pgs=2330 cs=0 l=1 rx=0x7f9854008790 tx=0x7f985400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:37.025 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:37.009+0000 7f985e7ec700 1 --2- 172.21.15.42:0/3204522718 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f98580a7740 0x7f985808df80 secure :-1 s=THROTTLE_DONE pgs=2330 cs=0 l=1 rx=0x7f9854008790 tx=0x7f985400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:37.025 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:37.009+0000 7f985d7ea700 1 -- 172.21.15.42:0/3204522718 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98540087d0 con 0x7f98580a7740 2019-12-05T06:14:37.026 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:37.009+0000 7f985d7ea700 1 -- 172.21.15.42:0/3204522718 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9854011860 con 0x7f98580a7740 2019-12-05T06:14:37.026 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:37.009+0000 7f986024f700 1 -- 172.21.15.42:0/3204522718 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f98580a7740 msgr2=0x7f985808df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:37.026 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:37.009+0000 7f986024f700 1 --2- 172.21.15.42:0/3204522718 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f98580a7740 0x7f985808df80 secure :-1 s=READY pgs=2330 cs=0 l=1 rx=0x7f9854008790 tx=0x7f985400fc70).stop 2019-12-05T06:14:37.026 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:37.009+0000 7f986024f700 1 -- 172.21.15.42:0/3204522718 shutdown_connections 2019-12-05T06:14:37.026 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:37.009+0000 7f986024f700 1 --2- 172.21.15.42:0/3204522718 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f985809fae0 0x7f985807eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:14:37.027 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:37.009+0000 7f986024f700 1 --2- 172.21.15.42:0/3204522718 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f98580a7740 0x7f985808df80 unknown :-1 s=CLOSED pgs=2330 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:37.027 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:37.009+0000 7f986024f700 1 -- 172.21.15.42:0/3204522718 shutdown_connections 2019-12-05T06:14:37.027 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:37.009+0000 7f986024f700 1 -- 172.21.15.42:0/3204522718 wait complete. 2019-12-05T06:14:37.027 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:37.009+0000 7f986024f700 1 -- 172.21.15.42:0/3204522718 >> 172.21.15.42:0/3204522718 conn(0x7f98580aabe0 msgr2=0x7f98580a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:14:37.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:37.013+0000 7f986024f700 1 Processor -- start 2019-12-05T06:14:37.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:37.013+0000 7f986024f700 1 -- start start 2019-12-05T06:14:37.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:37.013+0000 7f986024f700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f98580a6a70 0x7f985812aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:14:37.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:37.013+0000 7f986024f700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f985812af50 0x7f98581273d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:14:37.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:37.013+0000 7f986024f700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9858126d80 con 0x7f9858127920 2019-12-05T06:14:37.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:37.013+0000 7f986024f700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f985812a4c0 con 0x7f985812af50 2019-12-05T06:14:37.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:37.013+0000 7f986024f700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f985812a760 con 0x7f98580a6a70 2019-12-05T06:14:37.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:37.013+0000 7f985e7ec700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f98580a6a70 0x7f985812aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:37.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:37.013+0000 7f985e7ec700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f98580a6a70 0x7f985812aa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:51806/0 (socket says 172.21.15.42:51806) 2019-12-05T06:14:37.039 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:37.013+0000 7f985e7ec700 1 -- 172.21.15.42:0/3181741359 learned_addr learned my addr 172.21.15.42:0/3181741359 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:14:37.039 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:37.013+0000 7f985efed700 1 --1- 172.21.15.42:0/3181741359 >> v1:172.21.15.40:6789/0 conn(0x7f9858127920 0x7f9858065040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:60816/0 (socket says 172.21.15.42:60816) 2019-12-05T06:14:37.040 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:37.013+0000 7f985dfeb700 1 --2- 172.21.15.42:0/3181741359 >> v2:172.21.15.40:3300/0 conn(0x7f985812af50 0x7f98581273d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:37.040 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:37.017+0000 7f984affd700 1 -- 172.21.15.42:0/3181741359 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3103409782 0 0) 0x7f9858126d80 con 0x7f9858127920 2019-12-05T06:14:37.040 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:37.017+0000 7f985e7ec700 1 -- 172.21.15.42:0/3181741359 >> v2:172.21.15.40:3300/0 conn(0x7f985812af50 msgr2=0x7f98581273d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:37.040 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:37.017+0000 7f985e7ec700 1 --2- 172.21.15.42:0/3181741359 >> v2:172.21.15.40:3300/0 conn(0x7f985812af50 0x7f98581273d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:37.041 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:37.017+0000 7f985e7ec700 1 -- 172.21.15.42:0/3181741359 >> v1:172.21.15.40:6789/0 conn(0x7f9858127920 legacy=0x7f9858065040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:37.041 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:37.017+0000 7f985e7ec700 1 -- 172.21.15.42:0/3181741359 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9858065840 con 0x7f98580a6a70 2019-12-05T06:14:37.041 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:37.017+0000 7f985e7ec700 1 --2- 172.21.15.42:0/3181741359 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f98580a6a70 0x7f985812aa00 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f9854008790 tx=0x7f9854008a80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:37.041 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:37.017+0000 7f985e7ec700 1 --2- 172.21.15.42:0/3181741359 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f98580a6a70 0x7f985812aa00 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f9854008790 tx=0x7f9854008a80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:37.042 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:37.017+0000 7f985e7ec700 1 --2- 172.21.15.42:0/3181741359 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f98580a6a70 0x7f985812aa00 secure :-1 s=READY pgs=511 cs=0 l=1 rx=0x7f9854008790 tx=0x7f9854008a80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:14:37.043 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:37.017+0000 7f985e7ec700 1 --2- 172.21.15.42:0/3181741359 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f98580a6a70 0x7f985812aa00 secure :-1 s=THROTTLE_DONE pgs=511 cs=0 l=1 rx=0x7f9854008790 tx=0x7f9854008a80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:37.044 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:37.017+0000 7f984affd700 1 -- 172.21.15.42:0/3181741359 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f985400f490 con 0x7f98580a6a70 2019-12-05T06:14:37.044 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:37.017+0000 7f985e7ec700 1 --2- 172.21.15.42:0/3181741359 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f98580a6a70 0x7f985812aa00 secure :-1 s=THROTTLE_DONE pgs=511 cs=0 l=1 rx=0x7f9854008790 tx=0x7f9854008a80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:37.044 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:37.017+0000 7f984affd700 1 -- 172.21.15.42:0/3181741359 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9854008db0 con 0x7f98580a6a70 2019-12-05T06:14:37.044 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:37.017+0000 7f985e7ec700 1 --2- 172.21.15.42:0/3181741359 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f98580a6a70 0x7f985812aa00 secure :-1 s=THROTTLE_DONE pgs=511 cs=0 l=1 rx=0x7f9854008790 tx=0x7f9854008a80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:37.045 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:37.017+0000 7f986024f700 1 -- 172.21.15.42:0/3181741359 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9858131790 con 0x7f98580a6a70 2019-12-05T06:14:37.045 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:37.017+0000 7f984affd700 1 -- 172.21.15.42:0/3181741359 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9854005920 con 0x7f98580a6a70 2019-12-05T06:14:37.045 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:37.017+0000 7f986024f700 1 -- 172.21.15.42:0/3181741359 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9858131b20 con 0x7f98580a6a70 2019-12-05T06:14:37.046 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:37.017+0000 7f985e7ec700 1 --2- 172.21.15.42:0/3181741359 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f98580a6a70 0x7f985812aa00 secure :-1 s=THROTTLE_DONE pgs=511 cs=0 l=1 rx=0x7f9854008790 tx=0x7f9854008a80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:37.046 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:37.017+0000 7f985e7ec700 1 --2- 172.21.15.42:0/3181741359 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f98580a6a70 0x7f985812aa00 secure :-1 s=THROTTLE_DONE pgs=511 cs=0 l=1 rx=0x7f9854008790 tx=0x7f9854008a80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:37.046 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:37.017+0000 7f984affd700 1 -- 172.21.15.42:0/3181741359 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f9854005bc0 con 0x7f98580a6a70 2019-12-05T06:14:37.046 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:37.017+0000 7f984affd700 1 --2- 172.21.15.42:0/3181741359 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f984003ac90 0x7f984003d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:14:37.047 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:37.017+0000 7f984affd700 1 -- 172.21.15.42:0/3181741359 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 10331+0+0 (secure 0 0 0) 0x7f985404b990 con 0x7f98580a6a70 2019-12-05T06:14:37.047 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:37.017+0000 7f985dfeb700 1 --2- 172.21.15.42:0/3181741359 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f984003ac90 0x7f984003d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:37.047 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:37.017+0000 7f985dfeb700 1 --2- 172.21.15.42:0/3181741359 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f984003ac90 0x7f984003d1a0 crc :-1 s=READY pgs=980 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:14:37.047 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:37.021+0000 7f986024f700 1 -- 172.21.15.42:0/3181741359 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f985805d110 con 0x7f98580a6a70 2019-12-05T06:14:37.048 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:37.029+0000 7f985e7ec700 1 --2- 172.21.15.42:0/3181741359 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f98580a6a70 0x7f985812aa00 secure :-1 s=THROTTLE_DONE pgs=511 cs=0 l=1 rx=0x7f9854008790 tx=0x7f9854008a80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:37.049 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:37.029+0000 7f984affd700 1 -- 172.21.15.42:0/3181741359 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f9854030610 con 0x7f98580a6a70 2019-12-05T06:14:37.204 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:37.197+0000 7f986024f700 1 -- 172.21.15.42:0/3181741359 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f985805b8c0 con 0x7f98580a6a70 2019-12-05T06:14:37.204 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:37.197+0000 7f985e7ec700 1 --2- 172.21.15.42:0/3181741359 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f98580a6a70 0x7f985812aa00 secure :-1 s=THROTTLE_DONE pgs=511 cs=0 l=1 rx=0x7f9854008790 tx=0x7f9854008a80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:37.204 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:37.197+0000 7f984affd700 1 -- 172.21.15.42:0/3181741359 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 279 v279) v1 ==== 95+0+1774 (secure 0 0 0) 0x7f9854007160 con 0x7f98580a6a70 2019-12-05T06:14:37.204 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 279 2019-12-05T06:14:37.204 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:14:37.205 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":279,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":279,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:14:34.629090+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":99,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7217},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7217":{"gid":7217,"name":"b","rank":0,"incarnation":279,"state":"up:replay","state_seq":2,"addr":"172.21.15.40:6833/2667648531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2667648531},{"type":"v1","addr":"172.21.15.40:6833","nonce":2667648531}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:14:37.208 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:37.201+0000 7f986024f700 1 -- 172.21.15.42:0/3181741359 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f984003ac90 msgr2=0x7f984003d1a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:37.208 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:37.201+0000 7f986024f700 1 --2- 172.21.15.42:0/3181741359 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f984003ac90 0x7f984003d1a0 crc :-1 s=READY pgs=980 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:37.208 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:37.201+0000 7f986024f700 1 -- 172.21.15.42:0/3181741359 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f98580a6a70 msgr2=0x7f985812aa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:37.209 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:37.201+0000 7f986024f700 1 --2- 172.21.15.42:0/3181741359 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f98580a6a70 0x7f985812aa00 secure :-1 s=READY pgs=511 cs=0 l=1 rx=0x7f9854008790 tx=0x7f9854008a80).stop 2019-12-05T06:14:37.209 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:37.201+0000 7f986024f700 1 -- 172.21.15.42:0/3181741359 shutdown_connections 2019-12-05T06:14:37.209 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:37.201+0000 7f986024f700 1 --2- 172.21.15.42:0/3181741359 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f984003ac90 0x7f984003d1a0 unknown :-1 s=CLOSED pgs=980 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:37.209 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:37.201+0000 7f986024f700 1 --2- 172.21.15.42:0/3181741359 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f98580a6a70 0x7f985812aa00 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:37.209 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:37.201+0000 7f986024f700 1 --2- 172.21.15.42:0/3181741359 >> v2:172.21.15.40:3300/0 conn(0x7f985812af50 0x7f98581273d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:37.209 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:37.201+0000 7f986024f700 1 -- 172.21.15.42:0/3181741359 shutdown_connections 2019-12-05T06:14:37.210 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:37.201+0000 7f986024f700 1 -- 172.21.15.42:0/3181741359 wait complete. 2019-12-05T06:14:37.210 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:37.201+0000 7f986024f700 1 -- 172.21.15.42:0/3181741359 >> 172.21.15.42:0/3181741359 conn(0x7f98580a9a40 msgr2=0x7f9858059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:14:37.228 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 279, "feature_flags": { "enable_multiple": false, "ever_enabled_multiple": false }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2019-12-05T05:03:35.604738+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 279, "ever_allowed_features": 2, "explicitly_allowed_features": 2, "failed": [], "flags": 18, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_7217": { "addr": "172.21.15.40:6833/2667648531", "addrs": { "addrvec": [ { "addr": "172.21.15.40:6832", "nonce": 2667648531, "type": "v2" }, { "addr": "172.21.15.40:6833", "nonce": 2667648531, "type": "v1" } ] }, "export_targets": [], "features": 4540138292836696063, "flags": 0, "gid": 7217, "incarnation": 279, "name": "b", "rank": 0, "state": "up:replay", "state_seq": 2 } }, "last_failure": 0, "last_failure_osd_epoch": 99, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 2, "min_compat_client": "0 (unknown)", "modified": "2019-12-05T06:14:34.629090+0000", "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7217 } } } ], "standbys": [] } 2019-12-05T06:14:39.229 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:14:39.333 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:39.320+0000 7f7615a5f700 1 Processor -- start 2019-12-05T06:14:39.334 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:39.320+0000 7f7615a5f700 1 -- start start 2019-12-05T06:14:39.334 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:39.324+0000 7f7615a5f700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f761008df80 0x7f761009f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:14:39.334 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:39.324+0000 7f7615a5f700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f761009f830 0x7f761014efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:14:39.334 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:39.324+0000 7f7615a5f700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f761009fc60 con 0x7f76100a7740 2019-12-05T06:14:39.334 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:39.324+0000 7f7615a5f700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76100a0b00 con 0x7f761008df80 2019-12-05T06:14:39.335 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:39.324+0000 7f7615a5f700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f761008e5b0 con 0x7f761009f830 2019-12-05T06:14:39.335 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:39.324+0000 7f760effd700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f761008df80 0x7f761009f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:39.335 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:39.324+0000 7f760f7fe700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f76100a7740 0x7f761007e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:51386/0 (socket says 172.21.15.42:51386) 2019-12-05T06:14:39.335 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:39.324+0000 7f760f7fe700 1 -- 172.21.15.42:0/3112090158 learned_addr learned my addr 172.21.15.42:0/3112090158 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:14:39.335 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:39.324+0000 7f760ffff700 1 --2- 172.21.15.42:0/3112090158 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f761009f830 0x7f761014efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:39.335 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:39.324+0000 7f760effd700 1 -- 172.21.15.42:0/3112090158 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f761009f830 msgr2=0x7f761014efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:14:39.335 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:39.324+0000 7f760effd700 1 --2- 172.21.15.42:0/3112090158 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f761009f830 0x7f761014efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:14:39.336 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:39.324+0000 7f760effd700 1 -- 172.21.15.42:0/3112090158 >> v1:172.21.15.42:6789/0 conn(0x7f76100a7740 legacy=0x7f761007e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:39.336 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:39.324+0000 7f760effd700 1 -- 172.21.15.42:0/3112090158 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76100a0710 con 0x7f761008df80 2019-12-05T06:14:39.336 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:39.324+0000 7f760e7fc700 1 -- 172.21.15.42:0/3112090158 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 755489526 0 0) 0x7f761009fc60 con 0x7f76100a7740 2019-12-05T06:14:39.336 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:39.324+0000 7f760effd700 1 --2- 172.21.15.42:0/3112090158 >> v2:172.21.15.42:3300/0 conn(0x7f761008df80 0x7f761009f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f75f8008790 tx=0x7f75f800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:39.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:39.324+0000 7f760effd700 1 --2- 172.21.15.42:0/3112090158 >> v2:172.21.15.42:3300/0 conn(0x7f761008df80 0x7f761009f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f75f8008790 tx=0x7f75f800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:39.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:39.324+0000 7f760effd700 1 --2- 172.21.15.42:0/3112090158 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f761008df80 0x7f761009f2e0 secure :-1 s=READY pgs=2332 cs=0 l=1 rx=0x7f75f8008790 tx=0x7f75f800fc70).ready entity=mon.1 client_cookie=bf858f10b4d5698 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:14:39.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:39.324+0000 7f760effd700 1 --2- 172.21.15.42:0/3112090158 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f761008df80 0x7f761009f2e0 secure :-1 s=THROTTLE_DONE pgs=2332 cs=0 l=1 rx=0x7f75f8008790 tx=0x7f75f800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:39.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:39.324+0000 7f760e7fc700 1 -- 172.21.15.42:0/3112090158 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f75f8008bb0 con 0x7f761008df80 2019-12-05T06:14:39.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:39.324+0000 7f760effd700 1 --2- 172.21.15.42:0/3112090158 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f761008df80 0x7f761009f2e0 secure :-1 s=THROTTLE_DONE pgs=2332 cs=0 l=1 rx=0x7f75f8008790 tx=0x7f75f800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:39.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:39.324+0000 7f760effd700 1 --2- 172.21.15.42:0/3112090158 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f761008df80 0x7f761009f2e0 secure :-1 s=THROTTLE_DONE pgs=2332 cs=0 l=1 rx=0x7f75f8008790 tx=0x7f75f800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:39.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:39.324+0000 7f760e7fc700 1 -- 172.21.15.42:0/3112090158 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75f80087d0 con 0x7f761008df80 2019-12-05T06:14:39.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:39.324+0000 7f760e7fc700 1 -- 172.21.15.42:0/3112090158 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f75f8011860 con 0x7f761008df80 2019-12-05T06:14:39.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:39.324+0000 7f7615a5f700 1 -- 172.21.15.42:0/3112090158 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f761008df80 msgr2=0x7f761009f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:39.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:39.324+0000 7f7615a5f700 1 --2- 172.21.15.42:0/3112090158 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f761008df80 0x7f761009f2e0 secure :-1 s=READY pgs=2332 cs=0 l=1 rx=0x7f75f8008790 tx=0x7f75f800fc70).stop 2019-12-05T06:14:39.341 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:39.328+0000 7f7615a5f700 1 -- 172.21.15.42:0/3112090158 shutdown_connections 2019-12-05T06:14:39.341 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:39.328+0000 7f7615a5f700 1 --2- 172.21.15.42:0/3112090158 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f761009f830 0x7f761014efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:14:39.341 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:39.328+0000 7f7615a5f700 1 --2- 172.21.15.42:0/3112090158 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f761008df80 0x7f761009f2e0 unknown :-1 s=CLOSED pgs=2332 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:39.341 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:39.328+0000 7f7615a5f700 1 -- 172.21.15.42:0/3112090158 shutdown_connections 2019-12-05T06:14:39.341 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:39.328+0000 7f7615a5f700 1 -- 172.21.15.42:0/3112090158 wait complete. 2019-12-05T06:14:39.341 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:39.328+0000 7f7615a5f700 1 -- 172.21.15.42:0/3112090158 >> 172.21.15.42:0/3112090158 conn(0x7f76100aabe0 msgr2=0x7f76100a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:14:39.345 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:39.328+0000 7f7615a5f700 1 Processor -- start 2019-12-05T06:14:39.345 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:39.328+0000 7f7615a5f700 1 -- start start 2019-12-05T06:14:39.346 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:39.332+0000 7f7615a5f700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f76100a6d00 0x7f761007d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:14:39.346 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:39.332+0000 7f7615a5f700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f761012d250 0x7f7610130be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:14:39.346 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:39.332+0000 7f7615a5f700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76100a7130 con 0x7f761012ce20 2019-12-05T06:14:39.346 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:39.332+0000 7f7615a5f700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76100a4a30 con 0x7f761012d250 2019-12-05T06:14:39.346 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:39.332+0000 7f7615a5f700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76100a5640 con 0x7f76100a6d00 2019-12-05T06:14:39.347 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:39.332+0000 7f760ffff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f761012d250 0x7f7610130be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:39.347 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:39.332+0000 7f760ffff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f761012d250 0x7f7610130be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:51358/0 (socket says 172.21.15.42:51358) 2019-12-05T06:14:39.347 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:39.332+0000 7f760ffff700 1 -- 172.21.15.42:0/1005705241 learned_addr learned my addr 172.21.15.42:0/1005705241 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:14:39.347 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:39.332+0000 7f760f7fe700 1 --2- 172.21.15.42:0/1005705241 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f76100a6d00 0x7f761007d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:39.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:39.332+0000 7f760ffff700 1 -- 172.21.15.42:0/1005705241 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f76100a6d00 msgr2=0x7f761007d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:39.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:39.332+0000 7f760ffff700 1 --2- 172.21.15.42:0/1005705241 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f76100a6d00 0x7f761007d430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:39.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:39.332+0000 7f760ffff700 1 -- 172.21.15.42:0/1005705241 >> v1:172.21.15.40:6789/0 conn(0x7f761012ce20 legacy=0x7f76101303e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:39.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:39.332+0000 7f760ffff700 1 -- 172.21.15.42:0/1005705241 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f761007d980 con 0x7f761012d250 2019-12-05T06:14:39.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:39.332+0000 7f760ffff700 1 --2- 172.21.15.42:0/1005705241 >> v2:172.21.15.42:3300/0 conn(0x7f761012d250 0x7f7610130be0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f76080032b0 tx=0x7f7608010d10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:39.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:39.332+0000 7f760ffff700 1 --2- 172.21.15.42:0/1005705241 >> v2:172.21.15.42:3300/0 conn(0x7f761012d250 0x7f7610130be0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f76080032b0 tx=0x7f7608010d10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:39.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:39.332+0000 7f760ffff700 1 --2- 172.21.15.42:0/1005705241 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f761012d250 0x7f7610130be0 secure :-1 s=READY pgs=2333 cs=0 l=1 rx=0x7f76080032b0 tx=0x7f7608010d10).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:14:39.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:39.332+0000 7f760ffff700 1 --2- 172.21.15.42:0/1005705241 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f761012d250 0x7f7610130be0 secure :-1 s=THROTTLE_DONE pgs=2333 cs=0 l=1 rx=0x7f76080032b0 tx=0x7f7608010d10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:39.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:39.332+0000 7f760ffff700 1 --2- 172.21.15.42:0/1005705241 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f761012d250 0x7f7610130be0 secure :-1 s=THROTTLE_DONE pgs=2333 cs=0 l=1 rx=0x7f76080032b0 tx=0x7f7608010d10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:39.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:39.332+0000 7f7607fff700 1 -- 172.21.15.42:0/1005705241 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7608007350 con 0x7f761012d250 2019-12-05T06:14:39.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:39.332+0000 7f760ffff700 1 --2- 172.21.15.42:0/1005705241 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f761012d250 0x7f7610130be0 secure :-1 s=THROTTLE_DONE pgs=2333 cs=0 l=1 rx=0x7f76080032b0 tx=0x7f7608010d10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:39.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:39.332+0000 7f7607fff700 1 -- 172.21.15.42:0/1005705241 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7608005cb0 con 0x7f761012d250 2019-12-05T06:14:39.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:39.332+0000 7f7615a5f700 1 -- 172.21.15.42:0/1005705241 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7610131130 con 0x7f761012d250 2019-12-05T06:14:39.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:39.332+0000 7f7607fff700 1 -- 172.21.15.42:0/1005705241 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f76080123d0 con 0x7f761012d250 2019-12-05T06:14:39.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:39.332+0000 7f7615a5f700 1 -- 172.21.15.42:0/1005705241 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f761012d680 con 0x7f761012d250 2019-12-05T06:14:39.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:39.336+0000 7f760ffff700 1 --2- 172.21.15.42:0/1005705241 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f761012d250 0x7f7610130be0 secure :-1 s=THROTTLE_DONE pgs=2333 cs=0 l=1 rx=0x7f76080032b0 tx=0x7f7608010d10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:39.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:39.336+0000 7f7607fff700 1 -- 172.21.15.42:0/1005705241 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f76080126a0 con 0x7f761012d250 2019-12-05T06:14:39.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:39.336+0000 7f760ffff700 1 --2- 172.21.15.42:0/1005705241 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f761012d250 0x7f7610130be0 secure :-1 s=THROTTLE_DONE pgs=2333 cs=0 l=1 rx=0x7f76080032b0 tx=0x7f7608010d10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:39.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:39.336+0000 7f7607fff700 1 --2- 172.21.15.42:0/1005705241 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f75f403ae70 0x7f75f403d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:14:39.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:39.336+0000 7f7607fff700 1 -- 172.21.15.42:0/1005705241 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 10331+0+0 (secure 0 0 0) 0x7f760804b860 con 0x7f761012d250 2019-12-05T06:14:39.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:39.336+0000 7f760f7fe700 1 --2- 172.21.15.42:0/1005705241 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f75f403ae70 0x7f75f403d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:39.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:39.336+0000 7f760f7fe700 1 --2- 172.21.15.42:0/1005705241 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f75f403ae70 0x7f75f403d380 crc :-1 s=READY pgs=981 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:14:39.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:39.336+0000 7f7615a5f700 1 -- 172.21.15.42:0/1005705241 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f761005d110 con 0x7f761012d250 2019-12-05T06:14:39.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:39.340+0000 7f760ffff700 1 --2- 172.21.15.42:0/1005705241 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f761012d250 0x7f7610130be0 secure :-1 s=THROTTLE_DONE pgs=2333 cs=0 l=1 rx=0x7f76080032b0 tx=0x7f7608010d10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:39.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:39.340+0000 7f7607fff700 1 -- 172.21.15.42:0/1005705241 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f76080328e0 con 0x7f761012d250 2019-12-05T06:14:39.509 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:39.504+0000 7f7615a5f700 1 -- 172.21.15.42:0/1005705241 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f761005b8c0 con 0x7f761012d250 2019-12-05T06:14:39.515 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:39.504+0000 7f760ffff700 1 --2- 172.21.15.42:0/1005705241 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f761012d250 0x7f7610130be0 secure :-1 s=THROTTLE_DONE pgs=2333 cs=0 l=1 rx=0x7f76080032b0 tx=0x7f7608010d10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:39.515 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:39.504+0000 7f7607fff700 1 -- 172.21.15.42:0/1005705241 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 281 v281) v1 ==== 95+0+1775 (secure 0 0 0) 0x7f7608032b40 con 0x7f761012d250 2019-12-05T06:14:39.516 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 281 2019-12-05T06:14:39.516 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:14:39.516 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":281,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":281,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:14:38.995921+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":99,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7217},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7217":{"gid":7217,"name":"b","rank":0,"incarnation":279,"state":"up:rejoin","state_seq":20,"addr":"172.21.15.40:6833/2667648531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2667648531},{"type":"v1","addr":"172.21.15.40:6833","nonce":2667648531}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:14:39.517 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:39.508+0000 7f7615a5f700 1 -- 172.21.15.42:0/1005705241 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f75f403ae70 msgr2=0x7f75f403d380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:39.517 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:39.508+0000 7f7615a5f700 1 --2- 172.21.15.42:0/1005705241 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f75f403ae70 0x7f75f403d380 crc :-1 s=READY pgs=981 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:39.518 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:39.508+0000 7f7615a5f700 1 -- 172.21.15.42:0/1005705241 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f761012d250 msgr2=0x7f7610130be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:39.518 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:39.508+0000 7f7615a5f700 1 --2- 172.21.15.42:0/1005705241 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f761012d250 0x7f7610130be0 secure :-1 s=READY pgs=2333 cs=0 l=1 rx=0x7f76080032b0 tx=0x7f7608010d10).stop 2019-12-05T06:14:39.518 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:39.508+0000 7f7615a5f700 1 -- 172.21.15.42:0/1005705241 shutdown_connections 2019-12-05T06:14:39.518 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:39.508+0000 7f7615a5f700 1 --2- 172.21.15.42:0/1005705241 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f75f403ae70 0x7f75f403d380 unknown :-1 s=CLOSED pgs=981 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:39.518 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:39.508+0000 7f7615a5f700 1 --2- 172.21.15.42:0/1005705241 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f761012d250 0x7f7610130be0 unknown :-1 s=CLOSED pgs=2333 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:39.518 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:39.508+0000 7f7615a5f700 1 --2- 172.21.15.42:0/1005705241 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f76100a6d00 0x7f761007d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:39.518 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:39.508+0000 7f7615a5f700 1 -- 172.21.15.42:0/1005705241 shutdown_connections 2019-12-05T06:14:39.519 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:39.508+0000 7f7615a5f700 1 -- 172.21.15.42:0/1005705241 wait complete. 2019-12-05T06:14:39.519 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:39.508+0000 7f7615a5f700 1 -- 172.21.15.42:0/1005705241 >> 172.21.15.42:0/1005705241 conn(0x7f761008fde0 msgr2=0x7f7610059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:14:39.522 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2019-12-05T06:14:41.524 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:14:41.627 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:41.616+0000 7fbf4bf05700 1 Processor -- start 2019-12-05T06:14:41.628 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:41.616+0000 7fbf4bf05700 1 -- start start 2019-12-05T06:14:41.628 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:41.616+0000 7fbf4bf05700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fbf4408df80 0x7fbf4409f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:14:41.628 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:41.616+0000 7fbf4bf05700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbf4409f830 0x7fbf4414efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:14:41.628 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:41.616+0000 7fbf4bf05700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf4409fc60 con 0x7fbf440a7740 2019-12-05T06:14:41.628 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:41.616+0000 7fbf4bf05700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf440a0b00 con 0x7fbf4408df80 2019-12-05T06:14:41.629 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:41.616+0000 7fbf4bf05700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf4408e5b0 con 0x7fbf4409f830 2019-12-05T06:14:41.629 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:41.616+0000 7fbf4a4a2700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fbf440a7740 0x7fbf4407e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:51410/0 (socket says 172.21.15.42:51410) 2019-12-05T06:14:41.629 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:41.616+0000 7fbf4a4a2700 1 -- 172.21.15.42:0/1420193288 learned_addr learned my addr 172.21.15.42:0/1420193288 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:14:41.629 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:41.616+0000 7fbf49ca1700 1 --2- 172.21.15.42:0/1420193288 >> v2:172.21.15.40:3300/0 conn(0x7fbf4408df80 0x7fbf4409f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:41.629 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:41.616+0000 7fbf4aca3700 1 --2- 172.21.15.42:0/1420193288 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbf4409f830 0x7fbf4414efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:41.629 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:41.616+0000 7fbf494a0700 1 -- 172.21.15.42:0/1420193288 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 680917173 0 0) 0x7fbf4409fc60 con 0x7fbf440a7740 2019-12-05T06:14:41.630 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:41.616+0000 7fbf494a0700 1 -- 172.21.15.42:0/1420193288 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf300030b0 con 0x7fbf440a7740 2019-12-05T06:14:41.630 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:41.616+0000 7fbf494a0700 1 -- 172.21.15.42:0/1420193288 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1081460077 0 0) 0x7fbf300030b0 con 0x7fbf440a7740 2019-12-05T06:14:41.630 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:41.616+0000 7fbf494a0700 1 -- 172.21.15.42:0/1420193288 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbf4409f830 msgr2=0x7fbf4414efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:14:41.630 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:41.616+0000 7fbf494a0700 1 --2- 172.21.15.42:0/1420193288 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbf4409f830 0x7fbf4414efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:14:41.630 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:41.616+0000 7fbf494a0700 1 -- 172.21.15.42:0/1420193288 >> v2:172.21.15.40:3300/0 conn(0x7fbf4408df80 msgr2=0x7fbf4409f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:14:41.630 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:41.616+0000 7fbf494a0700 1 --2- 172.21.15.42:0/1420193288 >> v2:172.21.15.40:3300/0 conn(0x7fbf4408df80 0x7fbf4409f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:14:41.630 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:41.616+0000 7fbf494a0700 1 -- 172.21.15.42:0/1420193288 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf440a0710 con 0x7fbf440a7740 2019-12-05T06:14:41.631 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:41.616+0000 7fbf494a0700 1 -- 172.21.15.42:0/1420193288 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7fbf40001750 con 0x7fbf440a7740 2019-12-05T06:14:41.631 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:41.616+0000 7fbf494a0700 1 -- 172.21.15.42:0/1420193288 >> v1:172.21.15.42:6789/0 conn(0x7fbf440a7740 legacy=0x7fbf4407e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:41.631 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:41.616+0000 7fbf494a0700 1 --2- 172.21.15.42:0/1420193288 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbf300036e0 0x7fbf30003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:14:41.631 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:41.616+0000 7fbf494a0700 1 --2- 172.21.15.42:0/1420193288 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbf30004420 0x7fbf30004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:14:41.631 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:41.616+0000 7fbf494a0700 1 --2- 172.21.15.42:0/1420193288 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbf30004da0 0x7fbf3004f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:14:41.631 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:41.616+0000 7fbf494a0700 1 -- 172.21.15.42:0/1420193288 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf440a0b00 con 0x7fbf300036e0 2019-12-05T06:14:41.631 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:41.616+0000 7fbf494a0700 1 -- 172.21.15.42:0/1420193288 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf4408e5b0 con 0x7fbf30004da0 2019-12-05T06:14:41.632 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:41.616+0000 7fbf494a0700 1 -- 172.21.15.42:0/1420193288 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf30005250 con 0x7fbf30004420 2019-12-05T06:14:41.632 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:41.620+0000 7fbf4aca3700 1 --2- 172.21.15.42:0/1420193288 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbf30004da0 0x7fbf3004f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:41.632 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:41.620+0000 7fbf49ca1700 1 --2- 172.21.15.42:0/1420193288 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbf30004420 0x7fbf30004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:41.632 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:41.620+0000 7fbf4a4a2700 1 --2- 172.21.15.42:0/1420193288 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbf300036e0 0x7fbf30003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:41.632 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:41.620+0000 7fbf4aca3700 1 -- 172.21.15.42:0/1420193288 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbf30004420 msgr2=0x7fbf30004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:14:41.632 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:41.620+0000 7fbf4aca3700 1 --2- 172.21.15.42:0/1420193288 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbf30004420 0x7fbf30004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:14:41.632 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:41.620+0000 7fbf4aca3700 1 -- 172.21.15.42:0/1420193288 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbf300036e0 msgr2=0x7fbf30003b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:14:41.633 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:41.620+0000 7fbf4aca3700 1 --2- 172.21.15.42:0/1420193288 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbf300036e0 0x7fbf30003b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:14:41.633 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:41.620+0000 7fbf4aca3700 1 -- 172.21.15.42:0/1420193288 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf3004fd10 con 0x7fbf30004da0 2019-12-05T06:14:41.633 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:41.620+0000 7fbf4aca3700 1 --2- 172.21.15.42:0/1420193288 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbf30004da0 0x7fbf3004f680 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fbf3c002f30 tx=0x7fbf3c00ffa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:41.633 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:41.620+0000 7fbf4aca3700 1 --2- 172.21.15.42:0/1420193288 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbf30004da0 0x7fbf3004f680 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fbf3c002f30 tx=0x7fbf3c00ffa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:41.633 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:41.620+0000 7fbf4aca3700 1 --2- 172.21.15.42:0/1420193288 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbf30004da0 0x7fbf3004f680 secure :-1 s=READY pgs=2335 cs=0 l=1 rx=0x7fbf3c002f30 tx=0x7fbf3c00ffa0).ready entity=mon.1 client_cookie=46c7605a7bf178dc server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:14:41.633 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:41.620+0000 7fbf4aca3700 1 --2- 172.21.15.42:0/1420193288 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbf30004da0 0x7fbf3004f680 secure :-1 s=THROTTLE_DONE pgs=2335 cs=0 l=1 rx=0x7fbf3c002f30 tx=0x7fbf3c00ffa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:41.633 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:41.620+0000 7fbf494a0700 1 -- 172.21.15.42:0/1420193288 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbf3c010cd0 con 0x7fbf30004da0 2019-12-05T06:14:41.634 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:41.620+0000 7fbf4aca3700 1 --2- 172.21.15.42:0/1420193288 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbf30004da0 0x7fbf3004f680 secure :-1 s=THROTTLE_DONE pgs=2335 cs=0 l=1 rx=0x7fbf3c002f30 tx=0x7fbf3c00ffa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:41.634 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:41.620+0000 7fbf4aca3700 1 --2- 172.21.15.42:0/1420193288 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbf30004da0 0x7fbf3004f680 secure :-1 s=THROTTLE_DONE pgs=2335 cs=0 l=1 rx=0x7fbf3c002f30 tx=0x7fbf3c00ffa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:41.634 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:41.620+0000 7fbf494a0700 1 -- 172.21.15.42:0/1420193288 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf3c0108f0 con 0x7fbf30004da0 2019-12-05T06:14:41.634 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:41.620+0000 7fbf494a0700 1 -- 172.21.15.42:0/1420193288 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbf3c008bb0 con 0x7fbf30004da0 2019-12-05T06:14:41.634 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:41.620+0000 7fbf4bf05700 1 -- 172.21.15.42:0/1420193288 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbf30004da0 msgr2=0x7fbf3004f680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:41.634 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:41.620+0000 7fbf4bf05700 1 --2- 172.21.15.42:0/1420193288 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbf30004da0 0x7fbf3004f680 secure :-1 s=READY pgs=2335 cs=0 l=1 rx=0x7fbf3c002f30 tx=0x7fbf3c00ffa0).stop 2019-12-05T06:14:41.634 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:41.620+0000 7fbf4aca3700 1 -- 172.21.15.42:0/1420193288 reap_dead start 2019-12-05T06:14:41.635 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:41.620+0000 7fbf4bf05700 1 -- 172.21.15.42:0/1420193288 shutdown_connections 2019-12-05T06:14:41.635 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:41.620+0000 7fbf4bf05700 1 -- 172.21.15.42:0/1420193288 shutdown_connections 2019-12-05T06:14:41.635 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:41.620+0000 7fbf4bf05700 1 -- 172.21.15.42:0/1420193288 wait complete. 2019-12-05T06:14:41.635 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:41.620+0000 7fbf4bf05700 1 -- 172.21.15.42:0/1420193288 >> 172.21.15.42:0/1420193288 conn(0x7fbf440aabe0 msgr2=0x7fbf440a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:14:41.641 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:41.624+0000 7fbf4bf05700 1 Processor -- start 2019-12-05T06:14:41.642 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:41.624+0000 7fbf4bf05700 1 -- start start 2019-12-05T06:14:41.642 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:41.624+0000 7fbf4bf05700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbf4411b010 0x7fbf44065930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:14:41.642 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:41.624+0000 7fbf4bf05700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fbf4407c5b0 0x7fbf4408eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:14:41.643 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:41.624+0000 7fbf4bf05700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf4407d100 con 0x7fbf4407bc00 2019-12-05T06:14:41.643 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:41.624+0000 7fbf4bf05700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf440656d0 con 0x7fbf4407c5b0 2019-12-05T06:14:41.643 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:41.624+0000 7fbf4bf05700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf4411a5d0 con 0x7fbf4411b010 2019-12-05T06:14:41.644 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:41.624+0000 7fbf4a4a2700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbf4411b010 0x7fbf44065930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:41.644 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:41.624+0000 7fbf49ca1700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fbf4407c5b0 0x7fbf4408eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:41.645 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:41.624+0000 7fbf49ca1700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fbf4407c5b0 0x7fbf4408eb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.42:39202/0 (socket says 172.21.15.42:39202) 2019-12-05T06:14:41.645 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:41.624+0000 7fbf4a4a2700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbf4411b010 0x7fbf44065930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:51850/0 (socket says 172.21.15.42:51850) 2019-12-05T06:14:41.645 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:41.624+0000 7fbf49ca1700 1 -- 172.21.15.42:0/1389003322 learned_addr learned my addr 172.21.15.42:0/1389003322 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:14:41.645 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:41.628+0000 7fbf49ca1700 1 -- 172.21.15.42:0/1389003322 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbf4411b010 msgr2=0x7fbf44065930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:41.645 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:41.628+0000 7fbf49ca1700 1 --2- 172.21.15.42:0/1389003322 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbf4411b010 0x7fbf44065930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:41.646 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:41.628+0000 7fbf49ca1700 1 -- 172.21.15.42:0/1389003322 >> v1:172.21.15.40:6789/0 conn(0x7fbf4407bc00 legacy=0x7fbf44122400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:41.646 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:41.628+0000 7fbf49ca1700 1 -- 172.21.15.42:0/1389003322 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf4408f0c0 con 0x7fbf4407c5b0 2019-12-05T06:14:41.646 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:41.628+0000 7fbf49ca1700 1 --2- 172.21.15.42:0/1389003322 >> v2:172.21.15.40:3300/0 conn(0x7fbf4407c5b0 0x7fbf4408eb70 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fbf4407d5d0 tx=0x7fbf38010e10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:41.646 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:41.628+0000 7fbf49ca1700 1 --2- 172.21.15.42:0/1389003322 >> v2:172.21.15.40:3300/0 conn(0x7fbf4407c5b0 0x7fbf4408eb70 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fbf4407d5d0 tx=0x7fbf38010e10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:41.646 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:41.628+0000 7fbf49ca1700 1 --2- 172.21.15.42:0/1389003322 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbf4407c5b0 0x7fbf4408eb70 secure :-1 s=READY pgs=1769 cs=0 l=1 rx=0x7fbf4407d5d0 tx=0x7fbf38010e10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:14:41.646 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:41.628+0000 7fbf49ca1700 1 --2- 172.21.15.42:0/1389003322 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbf4407c5b0 0x7fbf4408eb70 secure :-1 s=THROTTLE_DONE pgs=1769 cs=0 l=1 rx=0x7fbf4407d5d0 tx=0x7fbf38010e10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:41.647 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:41.628+0000 7fbf36ffd700 1 -- 172.21.15.42:0/1389003322 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbf38005ca0 con 0x7fbf4407c5b0 2019-12-05T06:14:41.647 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:41.628+0000 7fbf49ca1700 1 --2- 172.21.15.42:0/1389003322 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbf4407c5b0 0x7fbf4408eb70 secure :-1 s=THROTTLE_DONE pgs=1769 cs=0 l=1 rx=0x7fbf4407d5d0 tx=0x7fbf38010e10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:41.647 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:41.628+0000 7fbf49ca1700 1 --2- 172.21.15.42:0/1389003322 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbf4407c5b0 0x7fbf4408eb70 secure :-1 s=THROTTLE_DONE pgs=1769 cs=0 l=1 rx=0x7fbf4407d5d0 tx=0x7fbf38010e10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:41.647 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:41.628+0000 7fbf36ffd700 1 -- 172.21.15.42:0/1389003322 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf380058c0 con 0x7fbf4407c5b0 2019-12-05T06:14:41.647 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:41.628+0000 7fbf4bf05700 1 -- 172.21.15.42:0/1389003322 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf4407cac0 con 0x7fbf4407c5b0 2019-12-05T06:14:41.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:41.628+0000 7fbf36ffd700 1 -- 172.21.15.42:0/1389003322 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbf38007530 con 0x7fbf4407c5b0 2019-12-05T06:14:41.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:41.628+0000 7fbf4bf05700 1 -- 172.21.15.42:0/1389003322 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf4407c030 con 0x7fbf4407c5b0 2019-12-05T06:14:41.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:41.628+0000 7fbf49ca1700 1 --2- 172.21.15.42:0/1389003322 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbf4407c5b0 0x7fbf4408eb70 secure :-1 s=THROTTLE_DONE pgs=1769 cs=0 l=1 rx=0x7fbf4407d5d0 tx=0x7fbf38010e10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:41.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:41.628+0000 7fbf49ca1700 1 --2- 172.21.15.42:0/1389003322 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbf4407c5b0 0x7fbf4408eb70 secure :-1 s=THROTTLE_DONE pgs=1769 cs=0 l=1 rx=0x7fbf4407d5d0 tx=0x7fbf38010e10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:41.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:41.628+0000 7fbf36ffd700 1 -- 172.21.15.42:0/1389003322 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fbf38012790 con 0x7fbf4407c5b0 2019-12-05T06:14:41.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:41.628+0000 7fbf36ffd700 1 --2- 172.21.15.42:0/1389003322 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbf2c03acc0 0x7fbf2c03d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:14:41.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:41.628+0000 7fbf36ffd700 1 -- 172.21.15.42:0/1389003322 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 10331+0+0 (secure 0 0 0) 0x7fbf3804b9b0 con 0x7fbf4407c5b0 2019-12-05T06:14:41.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:41.628+0000 7fbf4a4a2700 1 --2- 172.21.15.42:0/1389003322 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbf2c03acc0 0x7fbf2c03d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:41.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:41.628+0000 7fbf4a4a2700 1 --2- 172.21.15.42:0/1389003322 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbf2c03acc0 0x7fbf2c03d1d0 crc :-1 s=READY pgs=982 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:14:41.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:41.632+0000 7fbf4bf05700 1 -- 172.21.15.42:0/1389003322 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf4405d110 con 0x7fbf4407c5b0 2019-12-05T06:14:41.650 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:41.640+0000 7fbf49ca1700 1 --2- 172.21.15.42:0/1389003322 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbf4407c5b0 0x7fbf4408eb70 secure :-1 s=THROTTLE_DONE pgs=1769 cs=0 l=1 rx=0x7fbf4407d5d0 tx=0x7fbf38010e10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:41.650 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:41.640+0000 7fbf36ffd700 1 -- 172.21.15.42:0/1389003322 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fbf38032970 con 0x7fbf4407c5b0 2019-12-05T06:14:41.816 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:41.812+0000 7fbf4bf05700 1 -- 172.21.15.42:0/1389003322 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbf4405b8c0 con 0x7fbf4407c5b0 2019-12-05T06:14:41.817 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:41.812+0000 7fbf49ca1700 1 --2- 172.21.15.42:0/1389003322 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbf4407c5b0 0x7fbf4408eb70 secure :-1 s=THROTTLE_DONE pgs=1769 cs=0 l=1 rx=0x7fbf4407d5d0 tx=0x7fbf38010e10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:41.817 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:41.812+0000 7fbf36ffd700 1 -- 172.21.15.42:0/1389003322 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 282 v282) v1 ==== 95+0+1775 (secure 0 0 0) 0x7fbf38032bd0 con 0x7fbf4407c5b0 2019-12-05T06:14:41.819 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 282 2019-12-05T06:14:41.819 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:14:41.819 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":282,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":282,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:14:40.006630+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":99,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7217},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7217":{"gid":7217,"name":"b","rank":0,"incarnation":279,"state":"up:active","state_seq":21,"addr":"172.21.15.40:6833/2667648531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2667648531},{"type":"v1","addr":"172.21.15.40:6833","nonce":2667648531}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:14:41.822 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:41.816+0000 7fbf4bf05700 1 -- 172.21.15.42:0/1389003322 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbf2c03acc0 msgr2=0x7fbf2c03d1d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:41.822 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:41.816+0000 7fbf4bf05700 1 --2- 172.21.15.42:0/1389003322 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbf2c03acc0 0x7fbf2c03d1d0 crc :-1 s=READY pgs=982 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:41.822 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:41.816+0000 7fbf4bf05700 1 -- 172.21.15.42:0/1389003322 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbf4407c5b0 msgr2=0x7fbf4408eb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:41.822 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:41.816+0000 7fbf4bf05700 1 --2- 172.21.15.42:0/1389003322 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbf4407c5b0 0x7fbf4408eb70 secure :-1 s=READY pgs=1769 cs=0 l=1 rx=0x7fbf4407d5d0 tx=0x7fbf38010e10).stop 2019-12-05T06:14:41.822 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:41.816+0000 7fbf4bf05700 1 -- 172.21.15.42:0/1389003322 shutdown_connections 2019-12-05T06:14:41.823 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:41.816+0000 7fbf4bf05700 1 --2- 172.21.15.42:0/1389003322 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbf2c03acc0 0x7fbf2c03d1d0 unknown :-1 s=CLOSED pgs=982 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:41.823 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:41.816+0000 7fbf4bf05700 1 --2- 172.21.15.42:0/1389003322 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbf4411b010 0x7fbf44065930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:41.823 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:41.816+0000 7fbf4bf05700 1 --2- 172.21.15.42:0/1389003322 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbf4407c5b0 0x7fbf4408eb70 unknown :-1 s=CLOSED pgs=1769 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:41.823 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:41.816+0000 7fbf4bf05700 1 -- 172.21.15.42:0/1389003322 shutdown_connections 2019-12-05T06:14:41.823 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:41.816+0000 7fbf4bf05700 1 -- 172.21.15.42:0/1389003322 wait complete. 2019-12-05T06:14:41.824 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:41.816+0000 7fbf4bf05700 1 -- 172.21.15.42:0/1389003322 >> 172.21.15.42:0/1389003322 conn(0x7fbf440a9950 msgr2=0x7fbf4407e910 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:14:41.833 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b has gained rank=0, replacing gid=10246 2019-12-05T06:14:41.833 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 2 secs before reviving mds.a 2019-12-05T06:14:43.834 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.a 2019-12-05T06:14:43.834 INFO:tasks.ceph.mds.a:Restarting daemon 2019-12-05T06:14:43.834 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2019-12-05T06:14:43.838 INFO:tasks.ceph.mds.a:Started 2019-12-05T06:14:43.838 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:14:43.896 INFO:tasks.ceph.mds.a.smithi042.stdout:starting mds.a at 2019-12-05T06:14:43.943 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:43.932+0000 7f3cbb2a4700 1 Processor -- start 2019-12-05T06:14:43.944 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:43.932+0000 7f3cbb2a4700 1 -- start start 2019-12-05T06:14:43.944 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:43.932+0000 7f3cbb2a4700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3cb40a7730 0x7f3cb408df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:14:43.944 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:43.932+0000 7f3cbb2a4700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f3cb407e710 0x7f3cb407eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:14:43.944 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:43.932+0000 7f3cbb2a4700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3cb409f2d0 0x7f3cb409f700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:14:43.944 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:43.932+0000 7f3cbb2a4700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3cb40a0af0 con 0x7f3cb407e710 2019-12-05T06:14:43.944 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:43.932+0000 7f3cbb2a4700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3cb408e5a0 con 0x7f3cb409f2d0 2019-12-05T06:14:43.944 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:43.932+0000 7f3cbb2a4700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cb408e840 con 0x7f3cb40a7730 2019-12-05T06:14:43.945 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:43.932+0000 7f3cba042700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3cb409f2d0 0x7f3cb409f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:43.945 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:43.932+0000 7f3cba042700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3cb409f2d0 0x7f3cb409f700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:51412/0 (socket says 172.21.15.42:51412) 2019-12-05T06:14:43.945 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:43.932+0000 7f3cba042700 1 -- 172.21.15.42:0/2729993641 learned_addr learned my addr 172.21.15.42:0/2729993641 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:14:43.945 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:43.932+0000 7f3cb9841700 1 --2- 172.21.15.42:0/2729993641 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3cb40a7730 0x7f3cb408df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:43.945 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:43.932+0000 7f3cb9040700 1 --2- 172.21.15.42:0/2729993641 >> v2:172.21.15.40:3300/0 conn(0x7f3cb407e710 0x7f3cb407eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:43.945 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:43.932+0000 7f3cba042700 1 -- 172.21.15.42:0/2729993641 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3cb40a7730 msgr2=0x7f3cb408df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:14:43.945 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:43.932+0000 7f3cba042700 1 --2- 172.21.15.42:0/2729993641 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3cb40a7730 0x7f3cb408df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:14:43.946 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:43.932+0000 7f3cba042700 1 -- 172.21.15.42:0/2729993641 >> v2:172.21.15.40:3300/0 conn(0x7f3cb407e710 msgr2=0x7f3cb407eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:14:43.946 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:43.932+0000 7f3cba042700 1 --2- 172.21.15.42:0/2729993641 >> v2:172.21.15.40:3300/0 conn(0x7f3cb407e710 0x7f3cb407eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:14:43.946 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:43.932+0000 7f3cba042700 1 -- 172.21.15.42:0/2729993641 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cb407f090 con 0x7f3cb409f2d0 2019-12-05T06:14:43.946 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:43.932+0000 7f3cba042700 1 --2- 172.21.15.42:0/2729993641 >> v2:172.21.15.42:3300/0 conn(0x7f3cb409f2d0 0x7f3cb409f700 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f3cb408e760 tx=0x7f3cac010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:43.946 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:43.932+0000 7f3cba042700 1 --2- 172.21.15.42:0/2729993641 >> v2:172.21.15.42:3300/0 conn(0x7f3cb409f2d0 0x7f3cb409f700 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f3cb408e760 tx=0x7f3cac010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:43.946 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:43.932+0000 7f3cba042700 1 --2- 172.21.15.42:0/2729993641 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3cb409f2d0 0x7f3cb409f700 secure :-1 s=READY pgs=2338 cs=0 l=1 rx=0x7f3cb408e760 tx=0x7f3cac010130).ready entity=mon.1 client_cookie=b2b787d393914225 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:14:43.946 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:43.932+0000 7f3cba042700 1 --2- 172.21.15.42:0/2729993641 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3cb409f2d0 0x7f3cb409f700 secure :-1 s=THROTTLE_DONE pgs=2338 cs=0 l=1 rx=0x7f3cb408e760 tx=0x7f3cac010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:43.947 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:43.932+0000 7f3cb883f700 1 -- 172.21.15.42:0/2729993641 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3cac010de0 con 0x7f3cb409f2d0 2019-12-05T06:14:43.947 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:43.932+0000 7f3cba042700 1 --2- 172.21.15.42:0/2729993641 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3cb409f2d0 0x7f3cb409f700 secure :-1 s=THROTTLE_DONE pgs=2338 cs=0 l=1 rx=0x7f3cb408e760 tx=0x7f3cac010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:43.947 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:43.932+0000 7f3cba042700 1 --2- 172.21.15.42:0/2729993641 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3cb409f2d0 0x7f3cb409f700 secure :-1 s=THROTTLE_DONE pgs=2338 cs=0 l=1 rx=0x7f3cb408e760 tx=0x7f3cac010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:43.947 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:43.932+0000 7f3cb883f700 1 -- 172.21.15.42:0/2729993641 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cac010a00 con 0x7f3cb409f2d0 2019-12-05T06:14:43.947 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:43.932+0000 7f3cb883f700 1 -- 172.21.15.42:0/2729993641 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3cac0114a0 con 0x7f3cb409f2d0 2019-12-05T06:14:43.947 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:43.932+0000 7f3cbb2a4700 1 -- 172.21.15.42:0/2729993641 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3cb409f2d0 msgr2=0x7f3cb409f700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:43.947 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:43.932+0000 7f3cbb2a4700 1 --2- 172.21.15.42:0/2729993641 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3cb409f2d0 0x7f3cb409f700 secure :-1 s=READY pgs=2338 cs=0 l=1 rx=0x7f3cb408e760 tx=0x7f3cac010130).stop 2019-12-05T06:14:43.948 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:43.936+0000 7f3cbb2a4700 1 -- 172.21.15.42:0/2729993641 shutdown_connections 2019-12-05T06:14:43.948 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:43.936+0000 7f3cbb2a4700 1 --2- 172.21.15.42:0/2729993641 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3cb409f2d0 0x7f3cb409f700 unknown :-1 s=CLOSED pgs=2338 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:43.948 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:43.936+0000 7f3cbb2a4700 1 --2- 172.21.15.42:0/2729993641 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3cb40a7730 0x7f3cb408df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:14:43.948 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:43.936+0000 7f3cbb2a4700 1 --2- 172.21.15.42:0/2729993641 >> v2:172.21.15.40:3300/0 conn(0x7f3cb407e710 0x7f3cb407eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:14:43.948 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:43.936+0000 7f3cbb2a4700 1 -- 172.21.15.42:0/2729993641 shutdown_connections 2019-12-05T06:14:43.948 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:43.936+0000 7f3cbb2a4700 1 -- 172.21.15.42:0/2729993641 wait complete. 2019-12-05T06:14:43.949 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:43.936+0000 7f3cbb2a4700 1 -- 172.21.15.42:0/2729993641 >> 172.21.15.42:0/2729993641 conn(0x7f3cb40aabd0 msgr2=0x7f3cb40a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:14:43.959 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:43.936+0000 7f3cbb2a4700 1 Processor -- start 2019-12-05T06:14:43.959 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:43.936+0000 7f3cbb2a4700 1 -- start start 2019-12-05T06:14:43.959 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:43.940+0000 7f3cbb2a4700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3cb411eb70 0x7f3cb4059da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:14:43.960 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:43.940+0000 7f3cbb2a4700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cb4121ae0 con 0x7f3cb4133450 2019-12-05T06:14:43.960 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:43.940+0000 7f3cbb2a4700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cb411bcd0 con 0x7f3cb4120150 2019-12-05T06:14:43.960 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:43.940+0000 7f3cbb2a4700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cb411efa0 con 0x7f3cb411eb70 2019-12-05T06:14:43.960 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:43.940+0000 7f3cb9841700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f3cb4120150 0x7f3cb411b4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:51452/0 (socket says 172.21.15.42:51452) 2019-12-05T06:14:43.960 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:43.940+0000 7f3cb9841700 1 -- 172.21.15.42:0/878234091 learned_addr learned my addr 172.21.15.42:0/878234091 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:14:43.960 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:43.940+0000 7f3cba042700 1 --2- 172.21.15.42:0/878234091 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3cb411eb70 0x7f3cb4059da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:43.960 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:43.940+0000 7f3caa7fc700 1 -- 172.21.15.42:0/878234091 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3832566916 0 0) 0x7f3cb4121ae0 con 0x7f3cb4133450 2019-12-05T06:14:43.961 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:43.940+0000 7f3caa7fc700 1 -- 172.21.15.42:0/878234091 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c9c0030b0 con 0x7f3cb4133450 2019-12-05T06:14:43.961 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:43.940+0000 7f3caa7fc700 1 -- 172.21.15.42:0/878234091 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2311894863 0 0) 0x7f3cb411bcd0 con 0x7f3cb4120150 2019-12-05T06:14:43.961 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:43.940+0000 7f3caa7fc700 1 -- 172.21.15.42:0/878234091 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3cb4121ae0 con 0x7f3cb4120150 2019-12-05T06:14:43.961 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:43.940+0000 7f3cba042700 1 -- 172.21.15.42:0/878234091 >> v1:172.21.15.42:6789/0 conn(0x7f3cb4120150 legacy=0x7f3cb411b4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:43.961 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:43.940+0000 7f3cba042700 1 -- 172.21.15.42:0/878234091 >> v1:172.21.15.40:6789/0 conn(0x7f3cb4133450 legacy=0x7f3cb40752e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:43.961 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:43.940+0000 7f3cba042700 1 -- 172.21.15.42:0/878234091 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cb411bf00 con 0x7f3cb411eb70 2019-12-05T06:14:43.962 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:43.940+0000 7f3cba042700 1 --2- 172.21.15.42:0/878234091 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3cb411eb70 0x7f3cb4059da0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f3cb408e760 tx=0x7f3cac010c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:43.962 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:43.940+0000 7f3cba042700 1 --2- 172.21.15.42:0/878234091 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3cb411eb70 0x7f3cb4059da0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f3cb408e760 tx=0x7f3cac010c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:43.962 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:43.940+0000 7f3cba042700 1 --2- 172.21.15.42:0/878234091 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3cb411eb70 0x7f3cb4059da0 secure :-1 s=READY pgs=514 cs=0 l=1 rx=0x7f3cb408e760 tx=0x7f3cac010c30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:14:43.962 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:43.940+0000 7f3cba042700 1 --2- 172.21.15.42:0/878234091 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3cb411eb70 0x7f3cb4059da0 secure :-1 s=THROTTLE_DONE pgs=514 cs=0 l=1 rx=0x7f3cb408e760 tx=0x7f3cac010c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:43.963 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:43.940+0000 7f3caa7fc700 1 -- 172.21.15.42:0/878234091 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3cac011b80 con 0x7f3cb411eb70 2019-12-05T06:14:43.963 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:43.940+0000 7f3cbb2a4700 1 -- 172.21.15.42:0/878234091 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cb4120ef0 con 0x7f3cb411eb70 2019-12-05T06:14:43.963 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:43.940+0000 7f3cba042700 1 --2- 172.21.15.42:0/878234091 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3cb411eb70 0x7f3cb4059da0 secure :-1 s=THROTTLE_DONE pgs=514 cs=0 l=1 rx=0x7f3cb408e760 tx=0x7f3cac010c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:43.963 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:43.940+0000 7f3caa7fc700 1 -- 172.21.15.42:0/878234091 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cac0110c0 con 0x7f3cb411eb70 2019-12-05T06:14:43.963 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:43.940+0000 7f3cba042700 1 --2- 172.21.15.42:0/878234091 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3cb411eb70 0x7f3cb4059da0 secure :-1 s=THROTTLE_DONE pgs=514 cs=0 l=1 rx=0x7f3cb408e760 tx=0x7f3cac010c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:43.963 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:43.940+0000 7f3cbb2a4700 1 -- 172.21.15.42:0/878234091 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cb41213c0 con 0x7f3cb411eb70 2019-12-05T06:14:43.963 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:43.940+0000 7f3caa7fc700 1 -- 172.21.15.42:0/878234091 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3cac005ab0 con 0x7f3cb411eb70 2019-12-05T06:14:43.964 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:43.940+0000 7f3cba042700 1 --2- 172.21.15.42:0/878234091 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3cb411eb70 0x7f3cb4059da0 secure :-1 s=THROTTLE_DONE pgs=514 cs=0 l=1 rx=0x7f3cb408e760 tx=0x7f3cac010c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:43.964 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:43.944+0000 7f3caa7fc700 1 -- 172.21.15.42:0/878234091 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f3cac00f280 con 0x7f3cb411eb70 2019-12-05T06:14:43.964 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:43.944+0000 7f3cba042700 1 --2- 172.21.15.42:0/878234091 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3cb411eb70 0x7f3cb4059da0 secure :-1 s=THROTTLE_DONE pgs=514 cs=0 l=1 rx=0x7f3cb408e760 tx=0x7f3cac010c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:43.964 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:43.944+0000 7f3caa7fc700 1 --2- 172.21.15.42:0/878234091 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3c9c03b460 0x7f3c9c03d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:14:43.964 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:43.944+0000 7f3caa7fc700 1 -- 172.21.15.42:0/878234091 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 10331+0+0 (secure 0 0 0) 0x7f3cac08f510 con 0x7f3cb411eb70 2019-12-05T06:14:43.964 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:43.944+0000 7f3cb9841700 1 --2- 172.21.15.42:0/878234091 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3c9c03b460 0x7f3c9c03d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:43.965 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:43.944+0000 7f3cb9841700 1 --2- 172.21.15.42:0/878234091 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3c9c03b460 0x7f3c9c03d970 crc :-1 s=READY pgs=984 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:14:43.965 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:43.944+0000 7f3cbb2a4700 1 -- 172.21.15.42:0/878234091 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cb405d100 con 0x7f3cb411eb70 2019-12-05T06:14:43.965 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:43.952+0000 7f3cba042700 1 --2- 172.21.15.42:0/878234091 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3cb411eb70 0x7f3cb4059da0 secure :-1 s=THROTTLE_DONE pgs=514 cs=0 l=1 rx=0x7f3cb408e760 tx=0x7f3cac010c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:43.966 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:43.952+0000 7f3caa7fc700 1 -- 172.21.15.42:0/878234091 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f3cac005d00 con 0x7f3cb411eb70 2019-12-05T06:14:44.137 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:44.132+0000 7f3cbb2a4700 1 -- 172.21.15.42:0/878234091 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3cb405b8b0 con 0x7f3cb411eb70 2019-12-05T06:14:44.138 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:44.132+0000 7f3cba042700 1 --2- 172.21.15.42:0/878234091 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3cb411eb70 0x7f3cb4059da0 secure :-1 s=THROTTLE_DONE pgs=514 cs=0 l=1 rx=0x7f3cb408e760 tx=0x7f3cac010c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:44.138 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:44.132+0000 7f3caa7fc700 1 -- 172.21.15.42:0/878234091 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 282 v282) v1 ==== 95+0+1775 (secure 0 0 0) 0x7f3cac00f590 con 0x7f3cb411eb70 2019-12-05T06:14:44.140 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 282 2019-12-05T06:14:44.140 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:14:44.140 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":282,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":282,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:14:40.006630+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":99,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7217},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7217":{"gid":7217,"name":"b","rank":0,"incarnation":279,"state":"up:active","state_seq":21,"addr":"172.21.15.40:6833/2667648531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2667648531},{"type":"v1","addr":"172.21.15.40:6833","nonce":2667648531}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:14:44.143 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:44.136+0000 7f3cbb2a4700 1 -- 172.21.15.42:0/878234091 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3c9c03b460 msgr2=0x7f3c9c03d970 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:44.143 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:44.136+0000 7f3cbb2a4700 1 --2- 172.21.15.42:0/878234091 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3c9c03b460 0x7f3c9c03d970 crc :-1 s=READY pgs=984 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:44.143 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:44.136+0000 7f3cbb2a4700 1 -- 172.21.15.42:0/878234091 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3cb411eb70 msgr2=0x7f3cb4059da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:44.143 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:44.136+0000 7f3cbb2a4700 1 --2- 172.21.15.42:0/878234091 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3cb411eb70 0x7f3cb4059da0 secure :-1 s=READY pgs=514 cs=0 l=1 rx=0x7f3cb408e760 tx=0x7f3cac010c30).stop 2019-12-05T06:14:44.143 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:44.136+0000 7f3cbb2a4700 1 -- 172.21.15.42:0/878234091 shutdown_connections 2019-12-05T06:14:44.143 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:44.136+0000 7f3cbb2a4700 1 --2- 172.21.15.42:0/878234091 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3c9c03b460 0x7f3c9c03d970 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:44.143 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:44.136+0000 7f3cbb2a4700 1 --2- 172.21.15.42:0/878234091 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3cb411eb70 0x7f3cb4059da0 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:44.144 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:44.136+0000 7f3cbb2a4700 1 -- 172.21.15.42:0/878234091 shutdown_connections 2019-12-05T06:14:44.144 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:44.136+0000 7f3cbb2a4700 1 -- 172.21.15.42:0/878234091 wait complete. 2019-12-05T06:14:44.144 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:44.136+0000 7f3cbb2a4700 1 -- 172.21.15.42:0/878234091 >> 172.21.15.42:0/878234091 conn(0x7f3cb40a7f80 msgr2=0x7f3cb411c530 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:14:44.153 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2019-12-05T06:14:45.205 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:14:45.209 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:14:46.154 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:14:46.259 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:46.248+0000 7f732b392700 1 Processor -- start 2019-12-05T06:14:46.260 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:46.248+0000 7f732b392700 1 -- start start 2019-12-05T06:14:46.260 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:46.248+0000 7f732b392700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f73240a7740 0x7f732408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:14:46.260 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:46.248+0000 7f732b392700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f732409fae0 0x7f732407eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:14:46.261 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:46.248+0000 7f732b392700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73240a0710 con 0x7f732407e720 2019-12-05T06:14:46.261 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:46.248+0000 7f732b392700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73240a0b00 con 0x7f732409fae0 2019-12-05T06:14:46.261 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:46.248+0000 7f732b392700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f732408e5b0 con 0x7f73240a7740 2019-12-05T06:14:46.261 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:46.248+0000 7f732a130700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f732409fae0 0x7f732407eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:46.261 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:46.248+0000 7f732a130700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f732409fae0 0x7f732407eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:51436/0 (socket says 172.21.15.42:51436) 2019-12-05T06:14:46.262 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:46.248+0000 7f732a130700 1 -- 172.21.15.42:0/4193367625 learned_addr learned my addr 172.21.15.42:0/4193367625 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:14:46.262 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:46.248+0000 7f732992f700 1 --2- 172.21.15.42:0/4193367625 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f73240a7740 0x7f732408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:46.262 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:46.252+0000 7f732892d700 1 -- 172.21.15.42:0/4193367625 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 179494560 0 0) 0x7f73240a0710 con 0x7f732407e720 2019-12-05T06:14:46.262 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:46.252+0000 7f732a130700 1 -- 172.21.15.42:0/4193367625 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f73240a7740 msgr2=0x7f732408df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:14:46.262 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:46.252+0000 7f732a130700 1 --2- 172.21.15.42:0/4193367625 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f73240a7740 0x7f732408df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:14:46.262 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:46.252+0000 7f732a130700 1 -- 172.21.15.42:0/4193367625 >> v1:172.21.15.40:6789/0 conn(0x7f732407e720 legacy=0x7f732409f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:46.262 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:46.252+0000 7f732a130700 1 -- 172.21.15.42:0/4193367625 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f732407f0a0 con 0x7f732409fae0 2019-12-05T06:14:46.263 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:46.252+0000 7f732a130700 1 --2- 172.21.15.42:0/4193367625 >> v2:172.21.15.42:3300/0 conn(0x7f732409fae0 0x7f732407eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f731c008b10 tx=0x7f731c0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:46.263 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:46.252+0000 7f732a130700 1 --2- 172.21.15.42:0/4193367625 >> v2:172.21.15.42:3300/0 conn(0x7f732409fae0 0x7f732407eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f731c008b10 tx=0x7f731c0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:46.263 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:46.252+0000 7f732a130700 1 --2- 172.21.15.42:0/4193367625 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f732409fae0 0x7f732407eb50 secure :-1 s=READY pgs=2340 cs=0 l=1 rx=0x7f731c008b10 tx=0x7f731c0100b0).ready entity=mon.1 client_cookie=b2d77894f4ffdeb5 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:14:46.263 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:46.252+0000 7f732a130700 1 --2- 172.21.15.42:0/4193367625 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f732409fae0 0x7f732407eb50 secure :-1 s=THROTTLE_DONE pgs=2340 cs=0 l=1 rx=0x7f731c008b10 tx=0x7f731c0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:46.263 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:46.252+0000 7f732892d700 1 -- 172.21.15.42:0/4193367625 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f731c010da0 con 0x7f732409fae0 2019-12-05T06:14:46.263 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:46.252+0000 7f732a130700 1 --2- 172.21.15.42:0/4193367625 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f732409fae0 0x7f732407eb50 secure :-1 s=THROTTLE_DONE pgs=2340 cs=0 l=1 rx=0x7f731c008b10 tx=0x7f731c0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:46.263 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:46.252+0000 7f732892d700 1 -- 172.21.15.42:0/4193367625 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f731c0109c0 con 0x7f732409fae0 2019-12-05T06:14:46.264 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:46.252+0000 7f732a130700 1 --2- 172.21.15.42:0/4193367625 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f732409fae0 0x7f732407eb50 secure :-1 s=THROTTLE_DONE pgs=2340 cs=0 l=1 rx=0x7f731c008b10 tx=0x7f731c0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:46.264 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:46.252+0000 7f732892d700 1 -- 172.21.15.42:0/4193367625 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f731c0114a0 con 0x7f732409fae0 2019-12-05T06:14:46.264 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:46.252+0000 7f732b392700 1 -- 172.21.15.42:0/4193367625 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f732409fae0 msgr2=0x7f732407eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:46.264 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:46.252+0000 7f732b392700 1 --2- 172.21.15.42:0/4193367625 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f732409fae0 0x7f732407eb50 secure :-1 s=READY pgs=2340 cs=0 l=1 rx=0x7f731c008b10 tx=0x7f731c0100b0).stop 2019-12-05T06:14:46.264 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:46.252+0000 7f732b392700 1 -- 172.21.15.42:0/4193367625 shutdown_connections 2019-12-05T06:14:46.264 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:46.252+0000 7f732b392700 1 --2- 172.21.15.42:0/4193367625 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f732409fae0 0x7f732407eb50 unknown :-1 s=CLOSED pgs=2340 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:46.264 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:46.252+0000 7f732b392700 1 --2- 172.21.15.42:0/4193367625 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f73240a7740 0x7f732408df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:14:46.265 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:46.252+0000 7f732b392700 1 -- 172.21.15.42:0/4193367625 shutdown_connections 2019-12-05T06:14:46.265 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:46.252+0000 7f732b392700 1 -- 172.21.15.42:0/4193367625 wait complete. 2019-12-05T06:14:46.265 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:46.252+0000 7f732b392700 1 -- 172.21.15.42:0/4193367625 >> 172.21.15.42:0/4193367625 conn(0x7f73240aabe0 msgr2=0x7f73240a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:14:46.269 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:46.256+0000 7f732b392700 1 Processor -- start 2019-12-05T06:14:46.269 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:46.256+0000 7f732b392700 1 -- start start 2019-12-05T06:14:46.269 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:46.256+0000 7f732b392700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f73240a5b10 0x7f7324133440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:14:46.269 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:46.256+0000 7f732b392700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f732411b9a0 con 0x7f732411e460 2019-12-05T06:14:46.269 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:46.256+0000 7f732b392700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f732411e890 con 0x7f732411a800 2019-12-05T06:14:46.270 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:46.256+0000 7f732b392700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f732411e1c0 con 0x7f73240a5b10 2019-12-05T06:14:46.270 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:46.256+0000 7f732912e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f73240a5b10 0x7f7324133440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:46.270 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:46.256+0000 7f732992f700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f732411a800 0x7f732411b1a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:51476/0 (socket says 172.21.15.42:51476) 2019-12-05T06:14:46.270 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:46.256+0000 7f732992f700 1 -- 172.21.15.42:0/699013863 learned_addr learned my addr 172.21.15.42:0/699013863 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:14:46.271 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:46.256+0000 7f732912e700 1 -- 172.21.15.42:0/699013863 >> v1:172.21.15.42:6789/0 conn(0x7f732411a800 legacy=0x7f732411b1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:46.271 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:46.256+0000 7f732912e700 1 -- 172.21.15.42:0/699013863 >> v1:172.21.15.40:6789/0 conn(0x7f732411e460 legacy=0x7f732411fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:46.271 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:46.256+0000 7f731a7fc700 1 -- 172.21.15.42:0/699013863 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2482140357 0 0) 0x7f732411e890 con 0x7f732411a800 2019-12-05T06:14:46.271 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:46.256+0000 7f732912e700 1 -- 172.21.15.42:0/699013863 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7324120440 con 0x7f73240a5b10 2019-12-05T06:14:46.272 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:46.256+0000 7f732912e700 1 --2- 172.21.15.42:0/699013863 >> v2:172.21.15.42:3300/0 conn(0x7f73240a5b10 0x7f7324133440 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f731c0086f0 tx=0x7f731c0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:46.272 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:46.260+0000 7f732912e700 1 --2- 172.21.15.42:0/699013863 >> v2:172.21.15.42:3300/0 conn(0x7f73240a5b10 0x7f7324133440 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f731c0086f0 tx=0x7f731c0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:46.272 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:46.260+0000 7f732912e700 1 --2- 172.21.15.42:0/699013863 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f73240a5b10 0x7f7324133440 secure :-1 s=READY pgs=2342 cs=0 l=1 rx=0x7f731c0086f0 tx=0x7f731c0100b0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:14:46.272 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:46.260+0000 7f732912e700 1 --2- 172.21.15.42:0/699013863 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f73240a5b10 0x7f7324133440 secure :-1 s=THROTTLE_DONE pgs=2342 cs=0 l=1 rx=0x7f731c0086f0 tx=0x7f731c0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:46.272 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:46.260+0000 7f731a7fc700 1 -- 172.21.15.42:0/699013863 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7314006b30 con 0x7f73240a5b10 2019-12-05T06:14:46.272 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:46.260+0000 7f732912e700 1 --2- 172.21.15.42:0/699013863 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f73240a5b10 0x7f7324133440 secure :-1 s=THROTTLE_DONE pgs=2342 cs=0 l=1 rx=0x7f731c0086f0 tx=0x7f731c0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:46.272 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:46.260+0000 7f731a7fc700 1 -- 172.21.15.42:0/699013863 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7314006750 con 0x7f73240a5b10 2019-12-05T06:14:46.273 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:46.260+0000 7f732912e700 1 --2- 172.21.15.42:0/699013863 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f73240a5b10 0x7f7324133440 secure :-1 s=THROTTLE_DONE pgs=2342 cs=0 l=1 rx=0x7f731c0086f0 tx=0x7f731c0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:46.273 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:46.260+0000 7f731a7fc700 1 -- 172.21.15.42:0/699013863 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7314010ef0 con 0x7f73240a5b10 2019-12-05T06:14:46.273 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:46.260+0000 7f732b392700 1 -- 172.21.15.42:0/699013863 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7324120730 con 0x7f73240a5b10 2019-12-05T06:14:46.273 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:46.260+0000 7f732b392700 1 -- 172.21.15.42:0/699013863 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f732407b4e0 con 0x7f73240a5b10 2019-12-05T06:14:46.273 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:46.260+0000 7f732912e700 1 --2- 172.21.15.42:0/699013863 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f73240a5b10 0x7f7324133440 secure :-1 s=THROTTLE_DONE pgs=2342 cs=0 l=1 rx=0x7f731c0086f0 tx=0x7f731c0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:46.273 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:46.260+0000 7f731a7fc700 1 -- 172.21.15.42:0/699013863 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f7314011210 con 0x7f73240a5b10 2019-12-05T06:14:46.273 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:46.260+0000 7f732912e700 1 --2- 172.21.15.42:0/699013863 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f73240a5b10 0x7f7324133440 secure :-1 s=THROTTLE_DONE pgs=2342 cs=0 l=1 rx=0x7f731c0086f0 tx=0x7f731c0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:46.274 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:46.260+0000 7f731a7fc700 1 --2- 172.21.15.42:0/699013863 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f730c03abc0 0x7f730c03d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:14:46.274 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:46.260+0000 7f731a7fc700 1 -- 172.21.15.42:0/699013863 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 10331+0+0 (secure 0 0 0) 0x7f731404b100 con 0x7f73240a5b10 2019-12-05T06:14:46.274 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:46.260+0000 7f732992f700 1 --2- 172.21.15.42:0/699013863 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f730c03abc0 0x7f730c03d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:46.274 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:46.260+0000 7f732992f700 1 --2- 172.21.15.42:0/699013863 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f730c03abc0 0x7f730c03d0d0 crc :-1 s=READY pgs=985 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:14:46.274 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:46.264+0000 7f732b392700 1 -- 172.21.15.42:0/699013863 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f732405d110 con 0x7f73240a5b10 2019-12-05T06:14:46.276 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:46.272+0000 7f732912e700 1 --2- 172.21.15.42:0/699013863 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f73240a5b10 0x7f7324133440 secure :-1 s=THROTTLE_DONE pgs=2342 cs=0 l=1 rx=0x7f731c0086f0 tx=0x7f731c0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:46.277 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:46.272+0000 7f731a7fc700 1 -- 172.21.15.42:0/699013863 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f7314032180 con 0x7f73240a5b10 2019-12-05T06:14:46.485 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:46.480+0000 7f732b392700 1 -- 172.21.15.42:0/699013863 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f732405b8c0 con 0x7f73240a5b10 2019-12-05T06:14:46.486 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:46.480+0000 7f732912e700 1 --2- 172.21.15.42:0/699013863 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f73240a5b10 0x7f7324133440 secure :-1 s=THROTTLE_DONE pgs=2342 cs=0 l=1 rx=0x7f731c0086f0 tx=0x7f731c0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:46.486 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:46.480+0000 7f731a7fc700 1 -- 172.21.15.42:0/699013863 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 282 v282) v1 ==== 95+0+1775 (secure 0 0 0) 0x7f73140323e0 con 0x7f73240a5b10 2019-12-05T06:14:46.486 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 282 2019-12-05T06:14:46.487 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:14:46.487 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":282,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":282,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:14:40.006630+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":99,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7217},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7217":{"gid":7217,"name":"b","rank":0,"incarnation":279,"state":"up:active","state_seq":21,"addr":"172.21.15.40:6833/2667648531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2667648531},{"type":"v1","addr":"172.21.15.40:6833","nonce":2667648531}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:14:46.490 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:46.484+0000 7f732b392700 1 -- 172.21.15.42:0/699013863 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f730c03abc0 msgr2=0x7f730c03d0d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:46.490 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:46.484+0000 7f732b392700 1 --2- 172.21.15.42:0/699013863 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f730c03abc0 0x7f730c03d0d0 crc :-1 s=READY pgs=985 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:46.490 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:46.484+0000 7f732b392700 1 -- 172.21.15.42:0/699013863 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f73240a5b10 msgr2=0x7f7324133440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:46.491 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:46.484+0000 7f732b392700 1 --2- 172.21.15.42:0/699013863 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f73240a5b10 0x7f7324133440 secure :-1 s=READY pgs=2342 cs=0 l=1 rx=0x7f731c0086f0 tx=0x7f731c0100b0).stop 2019-12-05T06:14:46.491 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:46.484+0000 7f732b392700 1 -- 172.21.15.42:0/699013863 shutdown_connections 2019-12-05T06:14:46.491 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:46.484+0000 7f732b392700 1 --2- 172.21.15.42:0/699013863 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f730c03abc0 0x7f730c03d0d0 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:46.491 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:46.484+0000 7f732b392700 1 --2- 172.21.15.42:0/699013863 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f73240a5b10 0x7f7324133440 unknown :-1 s=CLOSED pgs=2342 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:46.491 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:46.484+0000 7f732b392700 1 -- 172.21.15.42:0/699013863 shutdown_connections 2019-12-05T06:14:46.492 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:46.484+0000 7f732b392700 1 -- 172.21.15.42:0/699013863 wait complete. 2019-12-05T06:14:46.492 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:46.484+0000 7f732b392700 1 -- 172.21.15.42:0/699013863 >> 172.21.15.42:0/699013863 conn(0x7f73240a7f90 msgr2=0x7f732411c4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:14:46.499 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2019-12-05T06:14:48.501 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:14:48.603 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:48.592+0000 7f1d70cbf700 1 Processor -- start 2019-12-05T06:14:48.603 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:48.592+0000 7f1d70cbf700 1 -- start start 2019-12-05T06:14:48.603 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:48.592+0000 7f1d70cbf700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f1d6c0a7740 0x7f1d6c08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:14:48.603 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:48.592+0000 7f1d70cbf700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f1d6c07e720 0x7f1d6c07eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:14:48.603 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:48.592+0000 7f1d70cbf700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1d6c09f2e0 0x7f1d6c09f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:14:48.604 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:48.592+0000 7f1d70cbf700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d6c0a0b00 con 0x7f1d6c07e720 2019-12-05T06:14:48.604 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:48.592+0000 7f1d70cbf700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d6c08e5b0 con 0x7f1d6c0a7740 2019-12-05T06:14:48.604 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:48.592+0000 7f1d70cbf700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d6c08e850 con 0x7f1d6c09f2e0 2019-12-05T06:14:48.604 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:48.592+0000 7f1d6ad9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f1d6c0a7740 0x7f1d6c08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:48.604 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:48.592+0000 7f1d6ad9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f1d6c0a7740 0x7f1d6c08df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:51446/0 (socket says 172.21.15.42:51446) 2019-12-05T06:14:48.604 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:48.592+0000 7f1d6ad9d700 1 -- 172.21.15.42:0/1035675511 learned_addr learned my addr 172.21.15.42:0/1035675511 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:14:48.604 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:48.592+0000 7f1d6259c700 1 --2- 172.21.15.42:0/1035675511 >> v2:172.21.15.40:3300/0 conn(0x7f1d6c07e720 0x7f1d6c07eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:48.605 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:48.592+0000 7f1d6b59e700 1 --2- 172.21.15.42:0/1035675511 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1d6c09f2e0 0x7f1d6c09f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:48.605 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:48.596+0000 7f1d6ad9d700 1 -- 172.21.15.42:0/1035675511 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1d6c09f2e0 msgr2=0x7f1d6c09f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:14:48.605 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:48.596+0000 7f1d6ad9d700 1 --2- 172.21.15.42:0/1035675511 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1d6c09f2e0 0x7f1d6c09f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:14:48.605 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:48.596+0000 7f1d6ad9d700 1 -- 172.21.15.42:0/1035675511 >> v2:172.21.15.40:3300/0 conn(0x7f1d6c07e720 msgr2=0x7f1d6c07eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:14:48.605 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:48.596+0000 7f1d6ad9d700 1 --2- 172.21.15.42:0/1035675511 >> v2:172.21.15.40:3300/0 conn(0x7f1d6c07e720 0x7f1d6c07eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:14:48.605 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:48.596+0000 7f1d6ad9d700 1 -- 172.21.15.42:0/1035675511 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d6c07f0a0 con 0x7f1d6c0a7740 2019-12-05T06:14:48.605 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:48.596+0000 7f1d6259c700 1 --2- 172.21.15.42:0/1035675511 >> v2:172.21.15.40:3300/0 conn(0x7f1d6c07e720 0x7f1d6c07eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:14:48.606 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:48.596+0000 7f1d6ad9d700 1 --2- 172.21.15.42:0/1035675511 >> v2:172.21.15.42:3300/0 conn(0x7f1d6c0a7740 0x7f1d6c08df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f1d6c0a0f50 tx=0x7f1d5c00fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:48.606 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:48.596+0000 7f1d6ad9d700 1 --2- 172.21.15.42:0/1035675511 >> v2:172.21.15.42:3300/0 conn(0x7f1d6c0a7740 0x7f1d6c08df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f1d6c0a0f50 tx=0x7f1d5c00fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:48.606 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:48.596+0000 7f1d6ad9d700 1 --2- 172.21.15.42:0/1035675511 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1d6c0a7740 0x7f1d6c08df80 secure :-1 s=READY pgs=2343 cs=0 l=1 rx=0x7f1d6c0a0f50 tx=0x7f1d5c00fdb0).ready entity=mon.1 client_cookie=585f6f9fe1b6857f server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:14:48.606 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:48.596+0000 7f1d6ad9d700 1 --2- 172.21.15.42:0/1035675511 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1d6c0a7740 0x7f1d6c08df80 secure :-1 s=THROTTLE_DONE pgs=2343 cs=0 l=1 rx=0x7f1d6c0a0f50 tx=0x7f1d5c00fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:48.606 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:48.596+0000 7f1d6ad9d700 1 --2- 172.21.15.42:0/1035675511 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1d6c0a7740 0x7f1d6c08df80 secure :-1 s=THROTTLE_DONE pgs=2343 cs=0 l=1 rx=0x7f1d6c0a0f50 tx=0x7f1d5c00fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:48.606 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:48.596+0000 7f1d6a59c700 1 -- 172.21.15.42:0/1035675511 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1d5c008c90 con 0x7f1d6c0a7740 2019-12-05T06:14:48.607 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:48.596+0000 7f1d6ad9d700 1 --2- 172.21.15.42:0/1035675511 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1d6c0a7740 0x7f1d6c08df80 secure :-1 s=THROTTLE_DONE pgs=2343 cs=0 l=1 rx=0x7f1d6c0a0f50 tx=0x7f1d5c00fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:48.607 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:48.596+0000 7f1d6a59c700 1 -- 172.21.15.42:0/1035675511 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d5c0088b0 con 0x7f1d6c0a7740 2019-12-05T06:14:48.607 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:48.596+0000 7f1d6a59c700 1 -- 172.21.15.42:0/1035675511 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1d5c011b50 con 0x7f1d6c0a7740 2019-12-05T06:14:48.607 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:48.596+0000 7f1d70cbf700 1 -- 172.21.15.42:0/1035675511 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1d6c0a7740 msgr2=0x7f1d6c08df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:48.607 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:48.596+0000 7f1d70cbf700 1 --2- 172.21.15.42:0/1035675511 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1d6c0a7740 0x7f1d6c08df80 secure :-1 s=READY pgs=2343 cs=0 l=1 rx=0x7f1d6c0a0f50 tx=0x7f1d5c00fdb0).stop 2019-12-05T06:14:48.607 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:48.596+0000 7f1d70cbf700 1 -- 172.21.15.42:0/1035675511 shutdown_connections 2019-12-05T06:14:48.607 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:48.596+0000 7f1d70cbf700 1 --2- 172.21.15.42:0/1035675511 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1d6c09f2e0 0x7f1d6c09f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:14:48.608 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:48.596+0000 7f1d70cbf700 1 --2- 172.21.15.42:0/1035675511 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1d6c0a7740 0x7f1d6c08df80 unknown :-1 s=CLOSED pgs=2343 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:48.608 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:48.596+0000 7f1d70cbf700 1 --2- 172.21.15.42:0/1035675511 >> v2:172.21.15.40:3300/0 conn(0x7f1d6c07e720 0x7f1d6c07eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:14:48.608 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:48.596+0000 7f1d70cbf700 1 -- 172.21.15.42:0/1035675511 shutdown_connections 2019-12-05T06:14:48.608 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:48.596+0000 7f1d70cbf700 1 -- 172.21.15.42:0/1035675511 wait complete. 2019-12-05T06:14:48.608 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:48.596+0000 7f1d70cbf700 1 -- 172.21.15.42:0/1035675511 >> 172.21.15.42:0/1035675511 conn(0x7f1d6c0aabe0 msgr2=0x7f1d6c0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:14:48.611 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:48.600+0000 7f1d70cbf700 1 Processor -- start 2019-12-05T06:14:48.611 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:48.600+0000 7f1d70cbf700 1 -- start start 2019-12-05T06:14:48.612 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:48.600+0000 7f1d70cbf700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f1d6c134170 0x7f1d6c078310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:14:48.612 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:48.600+0000 7f1d70cbf700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d6c064e40 con 0x7f1d6c133540 2019-12-05T06:14:48.612 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:48.600+0000 7f1d70cbf700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d6c0a4c60 con 0x7f1d6c0a5a40 2019-12-05T06:14:48.612 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:48.600+0000 7f1d70cbf700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d6c123d60 con 0x7f1d6c134170 2019-12-05T06:14:48.612 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:48.600+0000 7f1d6ad9d700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f1d6c0a5a40 0x7f1d6c142530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:51490/0 (socket says 172.21.15.42:51490) 2019-12-05T06:14:48.613 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:48.600+0000 7f1d6ad9d700 1 -- 172.21.15.42:0/1288889996 learned_addr learned my addr 172.21.15.42:0/1288889996 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:14:48.613 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:48.600+0000 7f1d6b59e700 1 --2- 172.21.15.42:0/1288889996 >> v2:172.21.15.40:3300/0 conn(0x7f1d6c134170 0x7f1d6c078310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:48.613 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:48.600+0000 7f1d63fff700 1 -- 172.21.15.42:0/1288889996 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 27037342 0 0) 0x7f1d6c0a4c60 con 0x7f1d6c0a5a40 2019-12-05T06:14:48.613 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:48.600+0000 7f1d63fff700 1 -- 172.21.15.42:0/1288889996 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d500030b0 con 0x7f1d6c0a5a40 2019-12-05T06:14:48.613 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:48.600+0000 7f1d6b59e700 1 -- 172.21.15.42:0/1288889996 >> v1:172.21.15.42:6789/0 conn(0x7f1d6c0a5a40 legacy=0x7f1d6c142530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:48.613 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:48.600+0000 7f1d6b59e700 1 -- 172.21.15.42:0/1288889996 >> v1:172.21.15.40:6789/0 conn(0x7f1d6c133540 legacy=0x7f1d6c133970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:48.613 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:48.600+0000 7f1d63fff700 1 -- 172.21.15.42:0/1288889996 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4091930770 0 0) 0x7f1d6c064e40 con 0x7f1d6c133540 2019-12-05T06:14:48.614 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:48.600+0000 7f1d6b59e700 1 -- 172.21.15.42:0/1288889996 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d6c0a5370 con 0x7f1d6c134170 2019-12-05T06:14:48.614 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:48.600+0000 7f1d6b59e700 1 --2- 172.21.15.42:0/1288889996 >> v2:172.21.15.40:3300/0 conn(0x7f1d6c134170 0x7f1d6c078310 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f1d640032b0 tx=0x7f1d64010dc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:48.614 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:48.600+0000 7f1d6b59e700 1 --2- 172.21.15.42:0/1288889996 >> v2:172.21.15.40:3300/0 conn(0x7f1d6c134170 0x7f1d6c078310 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f1d640032b0 tx=0x7f1d64010dc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:48.614 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:48.600+0000 7f1d6b59e700 1 --2- 172.21.15.42:0/1288889996 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1d6c134170 0x7f1d6c078310 secure :-1 s=READY pgs=1776 cs=0 l=1 rx=0x7f1d640032b0 tx=0x7f1d64010dc0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:14:48.614 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:48.600+0000 7f1d6b59e700 1 --2- 172.21.15.42:0/1288889996 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1d6c134170 0x7f1d6c078310 secure :-1 s=THROTTLE_DONE pgs=1776 cs=0 l=1 rx=0x7f1d640032b0 tx=0x7f1d64010dc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:48.614 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:48.600+0000 7f1d63fff700 1 -- 172.21.15.42:0/1288889996 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1d64005ab0 con 0x7f1d6c134170 2019-12-05T06:14:48.614 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:48.600+0000 7f1d6b59e700 1 --2- 172.21.15.42:0/1288889996 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1d6c134170 0x7f1d6c078310 secure :-1 s=THROTTLE_DONE pgs=1776 cs=0 l=1 rx=0x7f1d640032b0 tx=0x7f1d64010dc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:48.615 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:48.600+0000 7f1d63fff700 1 -- 172.21.15.42:0/1288889996 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d6400e160 con 0x7f1d6c134170 2019-12-05T06:14:48.615 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:48.600+0000 7f1d6b59e700 1 --2- 172.21.15.42:0/1288889996 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1d6c134170 0x7f1d6c078310 secure :-1 s=THROTTLE_DONE pgs=1776 cs=0 l=1 rx=0x7f1d640032b0 tx=0x7f1d64010dc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:48.615 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:48.600+0000 7f1d63fff700 1 -- 172.21.15.42:0/1288889996 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1d64012ab0 con 0x7f1d6c134170 2019-12-05T06:14:48.615 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:48.600+0000 7f1d70cbf700 1 -- 172.21.15.42:0/1288889996 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d6c0a6f50 con 0x7f1d6c134170 2019-12-05T06:14:48.615 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:48.600+0000 7f1d70cbf700 1 -- 172.21.15.42:0/1288889996 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d6c078860 con 0x7f1d6c134170 2019-12-05T06:14:48.615 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:48.604+0000 7f1d6b59e700 1 --2- 172.21.15.42:0/1288889996 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1d6c134170 0x7f1d6c078310 secure :-1 s=THROTTLE_DONE pgs=1776 cs=0 l=1 rx=0x7f1d640032b0 tx=0x7f1d64010dc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:48.615 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:48.604+0000 7f1d63fff700 1 -- 172.21.15.42:0/1288889996 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f1d64007180 con 0x7f1d6c134170 2019-12-05T06:14:48.616 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:48.604+0000 7f1d6b59e700 1 --2- 172.21.15.42:0/1288889996 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1d6c134170 0x7f1d6c078310 secure :-1 s=THROTTLE_DONE pgs=1776 cs=0 l=1 rx=0x7f1d640032b0 tx=0x7f1d64010dc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:48.616 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:48.604+0000 7f1d63fff700 1 --2- 172.21.15.42:0/1288889996 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f1d5003b1c0 0x7f1d5003d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:14:48.616 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:48.604+0000 7f1d63fff700 1 -- 172.21.15.42:0/1288889996 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 10331+0+0 (secure 0 0 0) 0x7f1d6404add0 con 0x7f1d6c134170 2019-12-05T06:14:48.616 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:48.604+0000 7f1d6ad9d700 1 --2- 172.21.15.42:0/1288889996 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f1d5003b1c0 0x7f1d5003d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:48.616 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:48.604+0000 7f1d6ad9d700 1 --2- 172.21.15.42:0/1288889996 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f1d5003b1c0 0x7f1d5003d6d0 crc :-1 s=READY pgs=986 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:14:48.617 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:48.608+0000 7f1d70cbf700 1 -- 172.21.15.42:0/1288889996 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d6c05d110 con 0x7f1d6c134170 2019-12-05T06:14:48.622 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:48.616+0000 7f1d6b59e700 1 --2- 172.21.15.42:0/1288889996 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1d6c134170 0x7f1d6c078310 secure :-1 s=THROTTLE_DONE pgs=1776 cs=0 l=1 rx=0x7f1d640032b0 tx=0x7f1d64010dc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:48.622 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:48.616+0000 7f1d63fff700 1 -- 172.21.15.42:0/1288889996 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f1d64012d50 con 0x7f1d6c134170 2019-12-05T06:14:48.792 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:48.788+0000 7f1d70cbf700 1 -- 172.21.15.42:0/1288889996 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1d6c05b8c0 con 0x7f1d6c134170 2019-12-05T06:14:48.793 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:48.788+0000 7f1d6b59e700 1 --2- 172.21.15.42:0/1288889996 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1d6c134170 0x7f1d6c078310 secure :-1 s=THROTTLE_DONE pgs=1776 cs=0 l=1 rx=0x7f1d640032b0 tx=0x7f1d64010dc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:48.793 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:48.788+0000 7f1d63fff700 1 -- 172.21.15.42:0/1288889996 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 283 v283) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f1d64010190 con 0x7f1d6c134170 2019-12-05T06:14:48.795 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 283 2019-12-05T06:14:48.795 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:14:48.796 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":283,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7295,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/3228549071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3228549071},{"type":"v1","addr":"172.21.15.42:6835","nonce":3228549071}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":283}],"filesystems":[{"mdsmap":{"epoch":282,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:14:40.006630+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":99,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7217},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7217":{"gid":7217,"name":"b","rank":0,"incarnation":279,"state":"up:active","state_seq":21,"addr":"172.21.15.40:6833/2667648531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2667648531},{"type":"v1","addr":"172.21.15.40:6833","nonce":2667648531}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:14:48.798 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:48.792+0000 7f1d70cbf700 1 -- 172.21.15.42:0/1288889996 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f1d5003b1c0 msgr2=0x7f1d5003d6d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:48.798 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:48.792+0000 7f1d70cbf700 1 --2- 172.21.15.42:0/1288889996 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f1d5003b1c0 0x7f1d5003d6d0 crc :-1 s=READY pgs=986 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:48.799 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:48.792+0000 7f1d70cbf700 1 -- 172.21.15.42:0/1288889996 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1d6c134170 msgr2=0x7f1d6c078310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:48.799 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:48.792+0000 7f1d70cbf700 1 --2- 172.21.15.42:0/1288889996 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1d6c134170 0x7f1d6c078310 secure :-1 s=READY pgs=1776 cs=0 l=1 rx=0x7f1d640032b0 tx=0x7f1d64010dc0).stop 2019-12-05T06:14:48.799 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:48.792+0000 7f1d70cbf700 1 -- 172.21.15.42:0/1288889996 shutdown_connections 2019-12-05T06:14:48.799 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:48.792+0000 7f1d70cbf700 1 --2- 172.21.15.42:0/1288889996 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f1d5003b1c0 0x7f1d5003d6d0 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:48.799 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:48.792+0000 7f1d70cbf700 1 --2- 172.21.15.42:0/1288889996 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1d6c134170 0x7f1d6c078310 unknown :-1 s=CLOSED pgs=1776 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:48.799 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:48.792+0000 7f1d70cbf700 1 -- 172.21.15.42:0/1288889996 shutdown_connections 2019-12-05T06:14:48.800 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:48.792+0000 7f1d70cbf700 1 -- 172.21.15.42:0/1288889996 wait complete. 2019-12-05T06:14:48.800 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:14:48.792+0000 7f1d70cbf700 1 -- 172.21.15.42:0/1288889996 >> 172.21.15.42:0/1288889996 conn(0x7f1d6c07e4f0 msgr2=0x7f1d6c09f510 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:14:48.808 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a reported in up:standby state 2019-12-05T06:14:48.808 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 117 secs before thrashing 2019-12-05T06:15:15.255 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:15:15.258 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:15:45.311 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:15:45.315 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:16:15.359 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:16:15.364 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:16:45.407 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:16:45.411 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:16:45.809 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:16:45.914 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:45.899+0000 7fcc44d8b700 1 Processor -- start 2019-12-05T06:16:45.914 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:45.899+0000 7fcc44d8b700 1 -- start start 2019-12-05T06:16:45.914 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:45.899+0000 7fcc44d8b700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fcc4008df80 0x7fcc4009f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:16:45.915 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:45.899+0000 7fcc44d8b700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fcc4009f830 0x7fcc4014efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:16:45.915 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:45.899+0000 7fcc44d8b700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc4009fc60 con 0x7fcc400a7740 2019-12-05T06:16:45.915 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:45.899+0000 7fcc44d8b700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc400a0b00 con 0x7fcc4009f830 2019-12-05T06:16:45.915 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:45.899+0000 7fcc44d8b700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc4008e5b0 con 0x7fcc4008df80 2019-12-05T06:16:45.915 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:45.899+0000 7fcc3e59c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fcc4008df80 0x7fcc4009f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:16:45.916 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:45.899+0000 7fcc3e59c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fcc4008df80 0x7fcc4009f2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:51882/0 (socket says 172.21.15.42:51882) 2019-12-05T06:16:45.916 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:45.899+0000 7fcc3e59c700 1 -- 172.21.15.42:0/1452836891 learned_addr learned my addr 172.21.15.42:0/1452836891 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:16:45.916 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:45.899+0000 7fcc3f59e700 1 --2- 172.21.15.42:0/1452836891 >> v2:172.21.15.40:3300/0 conn(0x7fcc4009f830 0x7fcc4014efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:16:45.916 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:45.899+0000 7fcc3e59c700 1 -- 172.21.15.42:0/1452836891 >> v2:172.21.15.40:3300/0 conn(0x7fcc4009f830 msgr2=0x7fcc4014efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:16:45.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:45.899+0000 7fcc3e59c700 1 --2- 172.21.15.42:0/1452836891 >> v2:172.21.15.40:3300/0 conn(0x7fcc4009f830 0x7fcc4014efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:16:45.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:45.899+0000 7fcc3e59c700 1 -- 172.21.15.42:0/1452836891 >> v1:172.21.15.40:6789/0 conn(0x7fcc400a7740 legacy=0x7fcc4007e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:45.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:45.899+0000 7fcc3e59c700 1 -- 172.21.15.42:0/1452836891 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc400a0710 con 0x7fcc4008df80 2019-12-05T06:16:45.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:45.899+0000 7fcc3e59c700 1 --2- 172.21.15.42:0/1452836891 >> v2:172.21.15.42:3300/0 conn(0x7fcc4008df80 0x7fcc4009f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fcc34002f30 tx=0x7fcc3400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:45.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:45.903+0000 7fcc3e59c700 1 --2- 172.21.15.42:0/1452836891 >> v2:172.21.15.42:3300/0 conn(0x7fcc4008df80 0x7fcc4009f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fcc34002f30 tx=0x7fcc3400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:45.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:45.903+0000 7fcc3e59c700 1 --2- 172.21.15.42:0/1452836891 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcc4008df80 0x7fcc4009f2e0 secure :-1 s=READY pgs=2349 cs=0 l=1 rx=0x7fcc34002f30 tx=0x7fcc3400fd30).ready entity=mon.1 client_cookie=745a01e3b0fdf780 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:16:45.918 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:45.903+0000 7fcc3e59c700 1 --2- 172.21.15.42:0/1452836891 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcc4008df80 0x7fcc4009f2e0 secure :-1 s=THROTTLE_DONE pgs=2349 cs=0 l=1 rx=0x7fcc34002f30 tx=0x7fcc3400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:45.918 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:45.903+0000 7fcc3e59c700 1 --2- 172.21.15.42:0/1452836891 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcc4008df80 0x7fcc4009f2e0 secure :-1 s=THROTTLE_DONE pgs=2349 cs=0 l=1 rx=0x7fcc34002f30 tx=0x7fcc3400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:45.918 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:45.903+0000 7fcc3dd9b700 1 -- 172.21.15.42:0/1452836891 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcc34008c90 con 0x7fcc4008df80 2019-12-05T06:16:45.918 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:45.903+0000 7fcc3e59c700 1 --2- 172.21.15.42:0/1452836891 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcc4008df80 0x7fcc4009f2e0 secure :-1 s=THROTTLE_DONE pgs=2349 cs=0 l=1 rx=0x7fcc34002f30 tx=0x7fcc3400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:45.918 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:45.903+0000 7fcc3dd9b700 1 -- 172.21.15.42:0/1452836891 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc340088b0 con 0x7fcc4008df80 2019-12-05T06:16:45.918 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:45.903+0000 7fcc3dd9b700 1 -- 172.21.15.42:0/1452836891 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcc34011ad0 con 0x7fcc4008df80 2019-12-05T06:16:45.919 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:45.903+0000 7fcc44d8b700 1 -- 172.21.15.42:0/1452836891 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcc4008df80 msgr2=0x7fcc4009f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:45.919 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:45.903+0000 7fcc44d8b700 1 --2- 172.21.15.42:0/1452836891 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcc4008df80 0x7fcc4009f2e0 secure :-1 s=READY pgs=2349 cs=0 l=1 rx=0x7fcc34002f30 tx=0x7fcc3400fd30).stop 2019-12-05T06:16:45.919 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:45.903+0000 7fcc44d8b700 1 -- 172.21.15.42:0/1452836891 shutdown_connections 2019-12-05T06:16:45.919 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:45.903+0000 7fcc44d8b700 1 --2- 172.21.15.42:0/1452836891 >> v2:172.21.15.40:3300/0 conn(0x7fcc4009f830 0x7fcc4014efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:16:45.919 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:45.903+0000 7fcc44d8b700 1 --2- 172.21.15.42:0/1452836891 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcc4008df80 0x7fcc4009f2e0 unknown :-1 s=CLOSED pgs=2349 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:16:45.919 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:45.903+0000 7fcc44d8b700 1 -- 172.21.15.42:0/1452836891 shutdown_connections 2019-12-05T06:16:45.920 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:45.903+0000 7fcc44d8b700 1 -- 172.21.15.42:0/1452836891 wait complete. 2019-12-05T06:16:45.920 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:45.903+0000 7fcc44d8b700 1 -- 172.21.15.42:0/1452836891 >> 172.21.15.42:0/1452836891 conn(0x7fcc400aabe0 msgr2=0x7fcc400a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:16:45.920 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:45.903+0000 7fcc44d8b700 1 Processor -- start 2019-12-05T06:16:45.921 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:45.903+0000 7fcc44d8b700 1 -- start start 2019-12-05T06:16:45.921 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:45.907+0000 7fcc44d8b700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fcc4008eeb0 0x7fcc401485d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:16:45.921 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:45.907+0000 7fcc44d8b700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc4014dfd0 con 0x7fcc4012a0a0 2019-12-05T06:16:45.921 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:45.907+0000 7fcc44d8b700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc40116db0 con 0x7fcc4014dba0 2019-12-05T06:16:45.921 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:45.907+0000 7fcc44d8b700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc4014e2b0 con 0x7fcc4008eeb0 2019-12-05T06:16:45.921 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:45.907+0000 7fcc3e59c700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fcc4014dba0 0x7fcc401165b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:51926/0 (socket says 172.21.15.42:51926) 2019-12-05T06:16:45.922 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:45.907+0000 7fcc3e59c700 1 -- 172.21.15.42:0/1261117158 learned_addr learned my addr 172.21.15.42:0/1261117158 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:16:45.922 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:45.907+0000 7fcc3ed9d700 1 --2- 172.21.15.42:0/1261117158 >> v2:172.21.15.42:3300/0 conn(0x7fcc4008eeb0 0x7fcc401485d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:16:45.922 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:45.907+0000 7fcc2f7fe700 1 -- 172.21.15.42:0/1261117158 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 518102658 0 0) 0x7fcc40116db0 con 0x7fcc4014dba0 2019-12-05T06:16:45.922 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:45.907+0000 7fcc2f7fe700 1 -- 172.21.15.42:0/1261117158 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc240030b0 con 0x7fcc4014dba0 2019-12-05T06:16:45.922 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:45.907+0000 7fcc2f7fe700 1 -- 172.21.15.42:0/1261117158 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 279536853 0 0) 0x7fcc30001ec0 con 0x7fcc4012a0a0 2019-12-05T06:16:45.922 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:45.907+0000 7fcc2f7fe700 1 -- 172.21.15.42:0/1261117158 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc40116db0 con 0x7fcc4012a0a0 2019-12-05T06:16:45.922 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:45.907+0000 7fcc2f7fe700 1 -- 172.21.15.42:0/1261117158 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 173986025 0 0) 0x7fcc240030b0 con 0x7fcc4014dba0 2019-12-05T06:16:45.923 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:45.907+0000 7fcc2f7fe700 1 -- 172.21.15.42:0/1261117158 --> v1:172.21.15.42:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcc30001ec0 con 0x7fcc4014dba0 2019-12-05T06:16:45.923 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:45.907+0000 7fcc2f7fe700 1 -- 172.21.15.42:0/1261117158 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7fcc34011de0 con 0x7fcc4014dba0 2019-12-05T06:16:45.923 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:45.907+0000 7fcc2f7fe700 1 -- 172.21.15.42:0/1261117158 >> v2:172.21.15.42:3300/0 conn(0x7fcc4008eeb0 msgr2=0x7fcc401485d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:45.923 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:45.907+0000 7fcc2f7fe700 1 --2- 172.21.15.42:0/1261117158 >> v2:172.21.15.42:3300/0 conn(0x7fcc4008eeb0 0x7fcc401485d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:16:45.923 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:45.907+0000 7fcc2f7fe700 1 -- 172.21.15.42:0/1261117158 >> v1:172.21.15.42:6789/0 conn(0x7fcc4014dba0 legacy=0x7fcc401165b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:45.923 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:45.907+0000 7fcc2f7fe700 1 -- 172.21.15.42:0/1261117158 >> v1:172.21.15.40:6789/0 conn(0x7fcc4012a0a0 legacy=0x7fcc4012a4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:45.923 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:45.907+0000 7fcc2f7fe700 1 --2- 172.21.15.42:0/1261117158 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcc24003ab0 0x7fcc24004420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:16:45.924 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:45.907+0000 7fcc2f7fe700 1 --2- 172.21.15.42:0/1261117158 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc24004970 0x7fcc24004da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:16:45.924 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:45.907+0000 7fcc2f7fe700 1 --2- 172.21.15.42:0/1261117158 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcc24009660 0x7fcc2400baa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:16:45.924 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:45.907+0000 7fcc2f7fe700 1 -- 172.21.15.42:0/1261117158 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc4014e2b0 con 0x7fcc24004970 2019-12-05T06:16:45.924 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:45.907+0000 7fcc2f7fe700 1 -- 172.21.15.42:0/1261117158 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc2400bff0 con 0x7fcc24009660 2019-12-05T06:16:45.924 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:45.907+0000 7fcc2f7fe700 1 -- 172.21.15.42:0/1261117158 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc2400c1b0 con 0x7fcc24003ab0 2019-12-05T06:16:45.924 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:45.907+0000 7fcc3ed9d700 1 --2- 172.21.15.42:0/1261117158 >> v2:172.21.15.42:3300/0 conn(0x7fcc4008eeb0 0x7fcc401485d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:16:45.925 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:45.907+0000 7fcc3f59e700 1 --2- 172.21.15.42:0/1261117158 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcc24009660 0x7fcc2400baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:16:45.925 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:45.907+0000 7fcc3e59c700 1 --2- 172.21.15.42:0/1261117158 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc24004970 0x7fcc24004da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:16:45.928 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:45.907+0000 7fcc3ed9d700 1 --2- 172.21.15.42:0/1261117158 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcc24003ab0 0x7fcc24004420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:16:45.928 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:45.907+0000 7fcc3f59e700 1 -- 172.21.15.42:0/1261117158 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcc24003ab0 msgr2=0x7fcc24004420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:45.928 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:45.907+0000 7fcc3f59e700 1 --2- 172.21.15.42:0/1261117158 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcc24003ab0 0x7fcc24004420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:16:45.928 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:45.907+0000 7fcc3f59e700 1 -- 172.21.15.42:0/1261117158 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc24004970 msgr2=0x7fcc24004da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:45.928 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:45.907+0000 7fcc3f59e700 1 --2- 172.21.15.42:0/1261117158 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc24004970 0x7fcc24004da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:16:45.929 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:45.907+0000 7fcc3f59e700 1 -- 172.21.15.42:0/1261117158 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc4014e7f0 con 0x7fcc24009660 2019-12-05T06:16:45.929 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:45.907+0000 7fcc3ed9d700 1 --2- 172.21.15.42:0/1261117158 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcc24003ab0 0x7fcc24004420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).send_auth_request state changed! 2019-12-05T06:16:45.929 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:45.907+0000 7fcc3e59c700 1 --2- 172.21.15.42:0/1261117158 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc24004970 0x7fcc24004da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:16:45.929 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:45.907+0000 7fcc3f59e700 1 --2- 172.21.15.42:0/1261117158 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcc24009660 0x7fcc2400baa0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fcc300079d0 tx=0x7fcc30010d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:45.929 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:45.907+0000 7fcc44d8b700 1 -- 172.21.15.42:0/1261117158 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc40132380 con 0x7fcc24009660 2019-12-05T06:16:45.929 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:45.907+0000 7fcc3f59e700 1 -- 172.21.15.42:0/1261117158 reap_dead start 2019-12-05T06:16:45.930 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:45.907+0000 7fcc3f59e700 1 --2- 172.21.15.42:0/1261117158 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcc24009660 0x7fcc2400baa0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fcc300079d0 tx=0x7fcc30010d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:45.930 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:45.907+0000 7fcc3f59e700 1 --2- 172.21.15.42:0/1261117158 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcc24009660 0x7fcc2400baa0 secure :-1 s=READY pgs=2351 cs=0 l=1 rx=0x7fcc300079d0 tx=0x7fcc30010d50).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:16:45.930 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:45.907+0000 7fcc44d8b700 1 -- 172.21.15.42:0/1261117158 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc4008eeb0 con 0x7fcc24009660 2019-12-05T06:16:45.930 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:45.907+0000 7fcc3f59e700 1 --2- 172.21.15.42:0/1261117158 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcc24009660 0x7fcc2400baa0 secure :-1 s=THROTTLE_DONE pgs=2351 cs=0 l=1 rx=0x7fcc300079d0 tx=0x7fcc30010d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:45.930 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:45.907+0000 7fcc2f7fe700 1 -- 172.21.15.42:0/1261117158 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcc30012b70 con 0x7fcc24009660 2019-12-05T06:16:45.930 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:45.907+0000 7fcc3f59e700 1 --2- 172.21.15.42:0/1261117158 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcc24009660 0x7fcc2400baa0 secure :-1 s=THROTTLE_DONE pgs=2351 cs=0 l=1 rx=0x7fcc300079d0 tx=0x7fcc30010d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:45.930 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:45.907+0000 7fcc3f59e700 1 --2- 172.21.15.42:0/1261117158 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcc24009660 0x7fcc2400baa0 secure :-1 s=THROTTLE_DONE pgs=2351 cs=0 l=1 rx=0x7fcc300079d0 tx=0x7fcc30010d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:45.931 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:45.907+0000 7fcc2f7fe700 1 -- 172.21.15.42:0/1261117158 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc30012790 con 0x7fcc24009660 2019-12-05T06:16:45.931 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:45.907+0000 7fcc2f7fe700 1 -- 172.21.15.42:0/1261117158 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcc30005730 con 0x7fcc24009660 2019-12-05T06:16:45.931 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:45.911+0000 7fcc3f59e700 1 --2- 172.21.15.42:0/1261117158 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcc24009660 0x7fcc2400baa0 secure :-1 s=THROTTLE_DONE pgs=2351 cs=0 l=1 rx=0x7fcc300079d0 tx=0x7fcc30010d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:45.931 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:45.911+0000 7fcc2f7fe700 1 -- 172.21.15.42:0/1261117158 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fcc30005930 con 0x7fcc24009660 2019-12-05T06:16:45.931 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:45.911+0000 7fcc3f59e700 1 --2- 172.21.15.42:0/1261117158 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcc24009660 0x7fcc2400baa0 secure :-1 s=THROTTLE_DONE pgs=2351 cs=0 l=1 rx=0x7fcc300079d0 tx=0x7fcc30010d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:45.931 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:45.911+0000 7fcc2f7fe700 1 --2- 172.21.15.42:0/1261117158 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fcc2403ece0 0x7fcc240411f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:16:45.932 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:45.911+0000 7fcc2f7fe700 1 -- 172.21.15.42:0/1261117158 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 10331+0+0 (secure 0 0 0) 0x7fcc4014dfd0 con 0x7fcc24009660 2019-12-05T06:16:45.932 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:45.911+0000 7fcc3ed9d700 1 --2- 172.21.15.42:0/1261117158 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fcc2403ece0 0x7fcc240411f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:16:45.932 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:45.911+0000 7fcc3ed9d700 1 --2- 172.21.15.42:0/1261117158 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fcc2403ece0 0x7fcc240411f0 crc :-1 s=READY pgs=990 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:16:45.932 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:45.915+0000 7fcc44d8b700 1 -- 172.21.15.42:0/1261117158 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc18003200 con 0x7fcc24009660 2019-12-05T06:16:45.932 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:45.923+0000 7fcc3f59e700 1 --2- 172.21.15.42:0/1261117158 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcc24009660 0x7fcc2400baa0 secure :-1 s=THROTTLE_DONE pgs=2351 cs=0 l=1 rx=0x7fcc300079d0 tx=0x7fcc30010d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:45.933 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:45.923+0000 7fcc2f7fe700 1 -- 172.21.15.42:0/1261117158 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fcc30032350 con 0x7fcc24009660 2019-12-05T06:16:46.091 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:46.083+0000 7fcc44d8b700 1 -- 172.21.15.42:0/1261117158 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcc18003630 con 0x7fcc24009660 2019-12-05T06:16:46.091 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:46.083+0000 7fcc3f59e700 1 --2- 172.21.15.42:0/1261117158 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcc24009660 0x7fcc2400baa0 secure :-1 s=THROTTLE_DONE pgs=2351 cs=0 l=1 rx=0x7fcc300079d0 tx=0x7fcc30010d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:46.092 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:46.083+0000 7fcc2f7fe700 1 -- 172.21.15.42:0/1261117158 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 283 v283) v1 ==== 95+0+2112 (secure 0 0 0) 0x7fcc300325b0 con 0x7fcc24009660 2019-12-05T06:16:46.092 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 283 2019-12-05T06:16:46.093 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:16:46.093 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":283,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7295,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/3228549071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3228549071},{"type":"v1","addr":"172.21.15.42:6835","nonce":3228549071}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":283}],"filesystems":[{"mdsmap":{"epoch":282,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:14:40.006630+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":99,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7217},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7217":{"gid":7217,"name":"b","rank":0,"incarnation":279,"state":"up:active","state_seq":21,"addr":"172.21.15.40:6833/2667648531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2667648531},{"type":"v1","addr":"172.21.15.40:6833","nonce":2667648531}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:16:46.094 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:46.087+0000 7fcc44d8b700 1 -- 172.21.15.42:0/1261117158 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fcc2403ece0 msgr2=0x7fcc240411f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:46.094 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:46.087+0000 7fcc44d8b700 1 --2- 172.21.15.42:0/1261117158 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fcc2403ece0 0x7fcc240411f0 crc :-1 s=READY pgs=990 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:16:46.095 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:46.087+0000 7fcc44d8b700 1 -- 172.21.15.42:0/1261117158 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcc24009660 msgr2=0x7fcc2400baa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:46.095 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:46.087+0000 7fcc44d8b700 1 --2- 172.21.15.42:0/1261117158 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcc24009660 0x7fcc2400baa0 secure :-1 s=READY pgs=2351 cs=0 l=1 rx=0x7fcc300079d0 tx=0x7fcc30010d50).stop 2019-12-05T06:16:46.095 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:46.087+0000 7fcc44d8b700 1 -- 172.21.15.42:0/1261117158 shutdown_connections 2019-12-05T06:16:46.095 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:46.087+0000 7fcc44d8b700 1 --2- 172.21.15.42:0/1261117158 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fcc2403ece0 0x7fcc240411f0 unknown :-1 s=CLOSED pgs=990 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:16:46.095 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:46.087+0000 7fcc44d8b700 1 --2- 172.21.15.42:0/1261117158 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcc24009660 0x7fcc2400baa0 unknown :-1 s=CLOSED pgs=2351 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:16:46.095 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:46.087+0000 7fcc44d8b700 1 -- 172.21.15.42:0/1261117158 shutdown_connections 2019-12-05T06:16:46.095 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:46.087+0000 7fcc44d8b700 1 -- 172.21.15.42:0/1261117158 wait complete. 2019-12-05T06:16:46.096 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:46.087+0000 7fcc44d8b700 1 -- 172.21.15.42:0/1261117158 >> 172.21.15.42:0/1261117158 conn(0x7fcc400aabe0 msgr2=0x7fcc400a9770 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:16:46.104 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.b (rank=0) 2019-12-05T06:16:46.104 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2019-12-05T06:16:46.104 INFO:teuthology.orchestra.run:waiting for 300 2019-12-05T06:16:46.121 INFO:tasks.ceph.mds.b:Stopped 2019-12-05T06:16:46.121 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:16:46.214 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:46.203+0000 7f819656b700 1 Processor -- start 2019-12-05T06:16:46.214 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:46.203+0000 7f819656b700 1 -- start start 2019-12-05T06:16:46.214 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:46.203+0000 7f819656b700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f81900a7740 0x7f819008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:16:46.215 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:46.203+0000 7f819656b700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f819009fae0 0x7f819007eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:16:46.215 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:46.203+0000 7f819656b700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81900a0710 con 0x7f819007e720 2019-12-05T06:16:46.215 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:46.203+0000 7f819656b700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81900a0b00 con 0x7f81900a7740 2019-12-05T06:16:46.215 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:46.203+0000 7f819656b700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f819008e5b0 con 0x7f819009fae0 2019-12-05T06:16:46.215 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:46.203+0000 7f8194b08700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f81900a7740 0x7f819008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:16:46.215 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:46.203+0000 7f818ffff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f819007e720 0x7f819009f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:51944/0 (socket says 172.21.15.42:51944) 2019-12-05T06:16:46.216 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:46.203+0000 7f8194b08700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f81900a7740 0x7f819008df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:51904/0 (socket says 172.21.15.42:51904) 2019-12-05T06:16:46.216 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:46.203+0000 7f818ffff700 1 -- 172.21.15.42:0/142552089 learned_addr learned my addr 172.21.15.42:0/142552089 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:16:46.216 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:46.203+0000 7f8195309700 1 --2- 172.21.15.42:0/142552089 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f819009fae0 0x7f819007eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:16:46.216 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:46.203+0000 7f8194b08700 1 -- 172.21.15.42:0/142552089 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f819009fae0 msgr2=0x7f819007eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:16:46.216 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:46.203+0000 7f8194b08700 1 --2- 172.21.15.42:0/142552089 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f819009fae0 0x7f819007eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:16:46.217 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:46.203+0000 7f8194b08700 1 -- 172.21.15.42:0/142552089 >> v1:172.21.15.42:6789/0 conn(0x7f819007e720 legacy=0x7f819009f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:46.217 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:46.203+0000 7f818f7fe700 1 -- 172.21.15.42:0/142552089 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2340283877 0 0) 0x7f81900a0710 con 0x7f819007e720 2019-12-05T06:16:46.217 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:46.203+0000 7f8194b08700 1 -- 172.21.15.42:0/142552089 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f819007f0a0 con 0x7f81900a7740 2019-12-05T06:16:46.217 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:46.203+0000 7f8194b08700 1 --2- 172.21.15.42:0/142552089 >> v2:172.21.15.42:3300/0 conn(0x7f81900a7740 0x7f819008df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f8184008790 tx=0x7f818400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:46.217 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:46.203+0000 7f8194b08700 1 --2- 172.21.15.42:0/142552089 >> v2:172.21.15.42:3300/0 conn(0x7f81900a7740 0x7f819008df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f8184008790 tx=0x7f818400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:46.217 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:46.203+0000 7f8194b08700 1 --2- 172.21.15.42:0/142552089 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f81900a7740 0x7f819008df80 secure :-1 s=READY pgs=2353 cs=0 l=1 rx=0x7f8184008790 tx=0x7f818400fc70).ready entity=mon.1 client_cookie=fb0da6f10e636851 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:16:46.217 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:46.203+0000 7f8194b08700 1 --2- 172.21.15.42:0/142552089 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f81900a7740 0x7f819008df80 secure :-1 s=THROTTLE_DONE pgs=2353 cs=0 l=1 rx=0x7f8184008790 tx=0x7f818400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:46.218 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:46.203+0000 7f818f7fe700 1 -- 172.21.15.42:0/142552089 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8184008bb0 con 0x7f81900a7740 2019-12-05T06:16:46.218 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:46.203+0000 7f8194b08700 1 --2- 172.21.15.42:0/142552089 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f81900a7740 0x7f819008df80 secure :-1 s=THROTTLE_DONE pgs=2353 cs=0 l=1 rx=0x7f8184008790 tx=0x7f818400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:46.218 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:46.203+0000 7f8194b08700 1 --2- 172.21.15.42:0/142552089 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f81900a7740 0x7f819008df80 secure :-1 s=THROTTLE_DONE pgs=2353 cs=0 l=1 rx=0x7f8184008790 tx=0x7f818400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:46.218 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:46.203+0000 7f818f7fe700 1 -- 172.21.15.42:0/142552089 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81840087d0 con 0x7f81900a7740 2019-12-05T06:16:46.218 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:46.203+0000 7f818f7fe700 1 -- 172.21.15.42:0/142552089 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8184011860 con 0x7f81900a7740 2019-12-05T06:16:46.218 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:46.207+0000 7f819656b700 1 -- 172.21.15.42:0/142552089 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f81900a7740 msgr2=0x7f819008df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:46.218 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:46.207+0000 7f819656b700 1 --2- 172.21.15.42:0/142552089 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f81900a7740 0x7f819008df80 secure :-1 s=READY pgs=2353 cs=0 l=1 rx=0x7f8184008790 tx=0x7f818400fc70).stop 2019-12-05T06:16:46.219 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:46.207+0000 7f819656b700 1 -- 172.21.15.42:0/142552089 shutdown_connections 2019-12-05T06:16:46.219 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:46.207+0000 7f819656b700 1 --2- 172.21.15.42:0/142552089 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f819009fae0 0x7f819007eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:16:46.219 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:46.207+0000 7f819656b700 1 --2- 172.21.15.42:0/142552089 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f81900a7740 0x7f819008df80 unknown :-1 s=CLOSED pgs=2353 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:16:46.219 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:46.207+0000 7f819656b700 1 -- 172.21.15.42:0/142552089 shutdown_connections 2019-12-05T06:16:46.219 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:46.207+0000 7f819656b700 1 -- 172.21.15.42:0/142552089 wait complete. 2019-12-05T06:16:46.219 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:46.207+0000 7f819656b700 1 -- 172.21.15.42:0/142552089 >> 172.21.15.42:0/142552089 conn(0x7f81900aabe0 msgr2=0x7f81900a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:16:46.222 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:46.207+0000 7f819656b700 1 Processor -- start 2019-12-05T06:16:46.222 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:46.211+0000 7f819656b700 1 -- start start 2019-12-05T06:16:46.223 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:46.211+0000 7f819656b700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8190118240 0x7f81901127b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:16:46.223 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:46.211+0000 7f819656b700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f81901388c0 0x7f819013ad00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:16:46.223 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:46.211+0000 7f819656b700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81900a5ed0 con 0x7f81900a61d0 2019-12-05T06:16:46.223 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:46.211+0000 7f819656b700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81900a4ed0 con 0x7f8190118240 2019-12-05T06:16:46.223 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:46.211+0000 7f819656b700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81900a5170 con 0x7f81901388c0 2019-12-05T06:16:46.223 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:46.211+0000 7f8195309700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f81901388c0 0x7f819013ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:16:46.223 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:46.211+0000 7f8195309700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f81901388c0 0x7f819013ad00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:51914/0 (socket says 172.21.15.42:51914) 2019-12-05T06:16:46.224 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:46.211+0000 7f8195309700 1 -- 172.21.15.42:0/822088374 learned_addr learned my addr 172.21.15.42:0/822088374 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:16:46.224 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:46.211+0000 7f8194b08700 1 --2- 172.21.15.42:0/822088374 >> v2:172.21.15.40:3300/0 conn(0x7f8190118240 0x7f81901127b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:16:46.224 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:46.211+0000 7f8195309700 1 -- 172.21.15.42:0/822088374 >> v2:172.21.15.40:3300/0 conn(0x7f8190118240 msgr2=0x7f81901127b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:46.224 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:46.211+0000 7f8195309700 1 --2- 172.21.15.42:0/822088374 >> v2:172.21.15.40:3300/0 conn(0x7f8190118240 0x7f81901127b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:16:46.224 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:46.211+0000 7f8195309700 1 -- 172.21.15.42:0/822088374 >> v1:172.21.15.42:6789/0 conn(0x7f81900a61d0 legacy=0x7f81900a6600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:46.225 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:46.211+0000 7f8195309700 1 -- 172.21.15.42:0/822088374 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8190112d00 con 0x7f81901388c0 2019-12-05T06:16:46.225 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:46.211+0000 7f8194b08700 1 --2- 172.21.15.42:0/822088374 >> v2:172.21.15.40:3300/0 conn(0x7f8190118240 0x7f81901127b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:16:46.225 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:46.211+0000 7f8195309700 1 --2- 172.21.15.42:0/822088374 >> v2:172.21.15.42:3300/0 conn(0x7f81901388c0 0x7f819013ad00 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f81880032b0 tx=0x7f8188013050).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:46.225 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:46.211+0000 7f8195309700 1 --2- 172.21.15.42:0/822088374 >> v2:172.21.15.42:3300/0 conn(0x7f81901388c0 0x7f819013ad00 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f81880032b0 tx=0x7f8188013050).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:46.225 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:46.211+0000 7f8195309700 1 --2- 172.21.15.42:0/822088374 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f81901388c0 0x7f819013ad00 secure :-1 s=READY pgs=2355 cs=0 l=1 rx=0x7f81880032b0 tx=0x7f8188013050).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:16:46.225 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:46.211+0000 7f8195309700 1 --2- 172.21.15.42:0/822088374 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f81901388c0 0x7f819013ad00 secure :-1 s=THROTTLE_DONE pgs=2355 cs=0 l=1 rx=0x7f81880032b0 tx=0x7f8188013050).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:46.226 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:46.211+0000 7f8195309700 1 --2- 172.21.15.42:0/822088374 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f81901388c0 0x7f819013ad00 secure :-1 s=THROTTLE_DONE pgs=2355 cs=0 l=1 rx=0x7f81880032b0 tx=0x7f8188013050).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:46.226 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:46.211+0000 7f818d7fa700 1 -- 172.21.15.42:0/822088374 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f81880058c0 con 0x7f81901388c0 2019-12-05T06:16:46.226 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:46.211+0000 7f8195309700 1 --2- 172.21.15.42:0/822088374 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f81901388c0 0x7f819013ad00 secure :-1 s=THROTTLE_DONE pgs=2355 cs=0 l=1 rx=0x7f81880032b0 tx=0x7f8188013050).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:46.226 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:46.211+0000 7f818d7fa700 1 -- 172.21.15.42:0/822088374 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8188013a90 con 0x7f81901388c0 2019-12-05T06:16:46.227 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:46.211+0000 7f818d7fa700 1 -- 172.21.15.42:0/822088374 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f818800e600 con 0x7f81901388c0 2019-12-05T06:16:46.227 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:46.211+0000 7f819656b700 1 -- 172.21.15.42:0/822088374 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f819011b990 con 0x7f81901388c0 2019-12-05T06:16:46.227 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:46.211+0000 7f819656b700 1 -- 172.21.15.42:0/822088374 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8190119c30 con 0x7f81901388c0 2019-12-05T06:16:46.227 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:46.215+0000 7f8195309700 1 --2- 172.21.15.42:0/822088374 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f81901388c0 0x7f819013ad00 secure :-1 s=THROTTLE_DONE pgs=2355 cs=0 l=1 rx=0x7f81880032b0 tx=0x7f8188013050).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:46.227 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:46.215+0000 7f8195309700 1 --2- 172.21.15.42:0/822088374 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f81901388c0 0x7f819013ad00 secure :-1 s=THROTTLE_DONE pgs=2355 cs=0 l=1 rx=0x7f81880032b0 tx=0x7f8188013050).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:46.227 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:46.215+0000 7f818d7fa700 1 -- 172.21.15.42:0/822088374 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f818800e910 con 0x7f81901388c0 2019-12-05T06:16:46.227 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:46.215+0000 7f818d7fa700 1 --2- 172.21.15.42:0/822088374 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f817003ae10 0x7f817003d320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:16:46.228 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:46.215+0000 7f818d7fa700 1 -- 172.21.15.42:0/822088374 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 10331+0+0 (secure 0 0 0) 0x7f818804d140 con 0x7f81901388c0 2019-12-05T06:16:46.228 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:46.215+0000 7f8194b08700 1 --2- 172.21.15.42:0/822088374 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f817003ae10 0x7f817003d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:16:46.228 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:46.215+0000 7f8194b08700 1 --2- 172.21.15.42:0/822088374 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f817003ae10 0x7f817003d320 crc :-1 s=READY pgs=991 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:16:46.229 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:46.219+0000 7f819656b700 1 -- 172.21.15.42:0/822088374 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f819005d110 con 0x7f81901388c0 2019-12-05T06:16:46.229 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:46.223+0000 7f8195309700 1 --2- 172.21.15.42:0/822088374 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f81901388c0 0x7f819013ad00 secure :-1 s=THROTTLE_DONE pgs=2355 cs=0 l=1 rx=0x7f81880032b0 tx=0x7f8188013050).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:46.229 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:46.223+0000 7f818d7fa700 1 -- 172.21.15.42:0/822088374 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f8188034180 con 0x7f81901388c0 2019-12-05T06:16:46.393 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:46.383+0000 7f819656b700 1 -- 172.21.15.42:0/822088374 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f819005b8c0 con 0x7f81901388c0 2019-12-05T06:16:46.393 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:46.387+0000 7f8195309700 1 --2- 172.21.15.42:0/822088374 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f81901388c0 0x7f819013ad00 secure :-1 s=THROTTLE_DONE pgs=2355 cs=0 l=1 rx=0x7f81880032b0 tx=0x7f8188013050).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:46.393 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:46.387+0000 7f818d7fa700 1 -- 172.21.15.42:0/822088374 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 283 v283) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f81880343e0 con 0x7f81901388c0 2019-12-05T06:16:46.394 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 283 2019-12-05T06:16:46.394 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:16:46.394 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":283,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7295,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/3228549071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3228549071},{"type":"v1","addr":"172.21.15.42:6835","nonce":3228549071}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":283}],"filesystems":[{"mdsmap":{"epoch":282,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:14:40.006630+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":99,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7217},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7217":{"gid":7217,"name":"b","rank":0,"incarnation":279,"state":"up:active","state_seq":21,"addr":"172.21.15.40:6833/2667648531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2667648531},{"type":"v1","addr":"172.21.15.40:6833","nonce":2667648531}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:16:46.395 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:46.387+0000 7f819656b700 1 -- 172.21.15.42:0/822088374 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f817003ae10 msgr2=0x7f817003d320 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:46.396 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:46.387+0000 7f819656b700 1 --2- 172.21.15.42:0/822088374 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f817003ae10 0x7f817003d320 crc :-1 s=READY pgs=991 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:16:46.396 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:46.387+0000 7f819656b700 1 -- 172.21.15.42:0/822088374 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f81901388c0 msgr2=0x7f819013ad00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:46.396 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:46.387+0000 7f819656b700 1 --2- 172.21.15.42:0/822088374 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f81901388c0 0x7f819013ad00 secure :-1 s=READY pgs=2355 cs=0 l=1 rx=0x7f81880032b0 tx=0x7f8188013050).stop 2019-12-05T06:16:46.396 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:46.387+0000 7f819656b700 1 -- 172.21.15.42:0/822088374 shutdown_connections 2019-12-05T06:16:46.397 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:46.387+0000 7f819656b700 1 --2- 172.21.15.42:0/822088374 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f817003ae10 0x7f817003d320 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:16:46.397 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:46.387+0000 7f819656b700 1 --2- 172.21.15.42:0/822088374 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f81901388c0 0x7f819013ad00 unknown :-1 s=CLOSED pgs=2355 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:16:46.397 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:46.387+0000 7f819656b700 1 --2- 172.21.15.42:0/822088374 >> v2:172.21.15.40:3300/0 conn(0x7f8190118240 0x7f81901127b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:16:46.397 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:46.387+0000 7f819656b700 1 -- 172.21.15.42:0/822088374 shutdown_connections 2019-12-05T06:16:46.397 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:46.387+0000 7f819656b700 1 -- 172.21.15.42:0/822088374 wait complete. 2019-12-05T06:16:46.397 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:46.387+0000 7f819656b700 1 -- 172.21.15.42:0/822088374 >> 172.21.15.42:0/822088374 conn(0x7f81900aabe0 msgr2=0x7f8190059ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:16:46.404 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:16:48.405 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:16:48.509 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:48.499+0000 7f7a7e386700 1 Processor -- start 2019-12-05T06:16:48.510 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:48.499+0000 7f7a7e386700 1 -- start start 2019-12-05T06:16:48.510 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:48.499+0000 7f7a7e386700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f7a7808df80 0x7f7a7809f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:16:48.510 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:48.499+0000 7f7a7e386700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a7809fc60 con 0x7f7a780a7740 2019-12-05T06:16:48.510 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:48.499+0000 7f7a7e386700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a780a0710 con 0x7f7a7809f830 2019-12-05T06:16:48.510 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:48.499+0000 7f7a7e386700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a780a0b00 con 0x7f7a7808df80 2019-12-05T06:16:48.510 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:48.499+0000 7f7a77fff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f7a7808df80 0x7f7a7809f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:16:48.510 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:48.499+0000 7f7a7c923700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f7a780a7740 0x7f7a7807e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:33170/0 (socket says 172.21.15.42:33170) 2019-12-05T06:16:48.511 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:48.499+0000 7f7a7c923700 1 -- 172.21.15.42:0/775707312 learned_addr learned my addr 172.21.15.42:0/775707312 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:16:48.511 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:48.499+0000 7f7a7d124700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f7a7809f830 0x7f7a7814efe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:51966/0 (socket says 172.21.15.42:51966) 2019-12-05T06:16:48.511 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:48.499+0000 7f7a777fe700 1 -- 172.21.15.42:0/775707312 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3658043875 0 0) 0x7f7a7809fc60 con 0x7f7a780a7740 2019-12-05T06:16:48.511 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:48.499+0000 7f7a77fff700 1 -- 172.21.15.42:0/775707312 >> v1:172.21.15.42:6789/0 conn(0x7f7a7809f830 legacy=0x7f7a7814efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:48.511 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:48.499+0000 7f7a77fff700 1 -- 172.21.15.42:0/775707312 >> v1:172.21.15.40:6789/0 conn(0x7f7a780a7740 legacy=0x7f7a7807e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:48.511 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:48.499+0000 7f7a77fff700 1 -- 172.21.15.42:0/775707312 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a7807ef20 con 0x7f7a7808df80 2019-12-05T06:16:48.512 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:48.499+0000 7f7a77fff700 1 --2- 172.21.15.42:0/775707312 >> v2:172.21.15.42:3300/0 conn(0x7f7a7808df80 0x7f7a7809f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f7a60008b10 tx=0x7f7a6000ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:48.512 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:48.499+0000 7f7a77fff700 1 --2- 172.21.15.42:0/775707312 >> v2:172.21.15.42:3300/0 conn(0x7f7a7808df80 0x7f7a7809f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f7a60008b10 tx=0x7f7a6000ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:48.512 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:48.499+0000 7f7a77fff700 1 --2- 172.21.15.42:0/775707312 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7a7808df80 0x7f7a7809f2e0 secure :-1 s=READY pgs=2357 cs=0 l=1 rx=0x7f7a60008b10 tx=0x7f7a6000ff70).ready entity=mon.1 client_cookie=142d8dac24aa4e11 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:16:48.512 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:48.499+0000 7f7a77fff700 1 --2- 172.21.15.42:0/775707312 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7a7808df80 0x7f7a7809f2e0 secure :-1 s=THROTTLE_DONE pgs=2357 cs=0 l=1 rx=0x7f7a60008b10 tx=0x7f7a6000ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:48.512 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:48.499+0000 7f7a777fe700 1 -- 172.21.15.42:0/775707312 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7a600119d0 con 0x7f7a7808df80 2019-12-05T06:16:48.513 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:48.499+0000 7f7a77fff700 1 --2- 172.21.15.42:0/775707312 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7a7808df80 0x7f7a7809f2e0 secure :-1 s=THROTTLE_DONE pgs=2357 cs=0 l=1 rx=0x7f7a60008b10 tx=0x7f7a6000ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:48.513 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:48.499+0000 7f7a77fff700 1 --2- 172.21.15.42:0/775707312 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7a7808df80 0x7f7a7809f2e0 secure :-1 s=THROTTLE_DONE pgs=2357 cs=0 l=1 rx=0x7f7a60008b10 tx=0x7f7a6000ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:48.513 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:48.499+0000 7f7a777fe700 1 -- 172.21.15.42:0/775707312 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a600115f0 con 0x7f7a7808df80 2019-12-05T06:16:48.513 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:48.499+0000 7f7a777fe700 1 -- 172.21.15.42:0/775707312 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7a6000a160 con 0x7f7a7808df80 2019-12-05T06:16:48.513 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:48.499+0000 7f7a7e386700 1 -- 172.21.15.42:0/775707312 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7a7808df80 msgr2=0x7f7a7809f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:48.513 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:48.499+0000 7f7a7e386700 1 --2- 172.21.15.42:0/775707312 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7a7808df80 0x7f7a7809f2e0 secure :-1 s=READY pgs=2357 cs=0 l=1 rx=0x7f7a60008b10 tx=0x7f7a6000ff70).stop 2019-12-05T06:16:48.514 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:48.503+0000 7f7a7e386700 1 -- 172.21.15.42:0/775707312 shutdown_connections 2019-12-05T06:16:48.514 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:48.503+0000 7f7a7e386700 1 --2- 172.21.15.42:0/775707312 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7a7808df80 0x7f7a7809f2e0 unknown :-1 s=CLOSED pgs=2357 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:16:48.514 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:48.503+0000 7f7a7e386700 1 -- 172.21.15.42:0/775707312 shutdown_connections 2019-12-05T06:16:48.514 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:48.503+0000 7f7a7e386700 1 -- 172.21.15.42:0/775707312 wait complete. 2019-12-05T06:16:48.514 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:48.503+0000 7f7a7e386700 1 -- 172.21.15.42:0/775707312 >> 172.21.15.42:0/775707312 conn(0x7f7a780aabe0 msgr2=0x7f7a780a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:16:48.516 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:48.503+0000 7f7a7e386700 1 Processor -- start 2019-12-05T06:16:48.516 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:48.503+0000 7f7a7e386700 1 -- start start 2019-12-05T06:16:48.516 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:48.507+0000 7f7a7e386700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7a78116500 0x7f7a780a4a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:16:48.517 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:48.507+0000 7f7a7e386700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7a7807ae00 0x7f7a7807b230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:16:48.517 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:48.507+0000 7f7a7e386700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f7a781335a0 0x7f7a781339d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:16:48.517 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:48.507+0000 7f7a7e386700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a7807cc60 con 0x7f7a78116500 2019-12-05T06:16:48.517 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:48.507+0000 7f7a7e386700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a7807cf00 con 0x7f7a781335a0 2019-12-05T06:16:48.517 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:48.507+0000 7f7a7e386700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a7807d1a0 con 0x7f7a7807ae00 2019-12-05T06:16:48.518 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:48.507+0000 7f7a7d124700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f7a781335a0 0x7f7a781339d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:16:48.518 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:48.507+0000 7f7a7c923700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7a78116500 0x7f7a780a4a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:16:48.518 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:48.507+0000 7f7a77fff700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7a7807ae00 0x7f7a7807b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:16:48.519 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:48.507+0000 7f7a7d124700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f7a781335a0 0x7f7a781339d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:51932/0 (socket says 172.21.15.42:51932) 2019-12-05T06:16:48.519 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:48.507+0000 7f7a7d124700 1 -- 172.21.15.42:0/2460988778 learned_addr learned my addr 172.21.15.42:0/2460988778 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:16:48.519 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:48.507+0000 7f7a7d124700 1 -- 172.21.15.42:0/2460988778 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7a7807ae00 msgr2=0x7f7a7807b230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:48.519 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:48.507+0000 7f7a7d124700 1 --2- 172.21.15.42:0/2460988778 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7a7807ae00 0x7f7a7807b230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:16:48.519 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:48.507+0000 7f7a7d124700 1 -- 172.21.15.42:0/2460988778 >> v2:172.21.15.40:3300/0 conn(0x7f7a78116500 msgr2=0x7f7a780a4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:48.519 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:48.507+0000 7f7a7d124700 1 --2- 172.21.15.42:0/2460988778 >> v2:172.21.15.40:3300/0 conn(0x7f7a78116500 0x7f7a780a4a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:16:48.520 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:48.507+0000 7f7a7d124700 1 -- 172.21.15.42:0/2460988778 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a780a4fa0 con 0x7f7a781335a0 2019-12-05T06:16:48.520 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:48.507+0000 7f7a7d124700 1 --2- 172.21.15.42:0/2460988778 >> v2:172.21.15.42:3300/0 conn(0x7f7a781335a0 0x7f7a781339d0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f7a7807ce20 tx=0x7f7a70010850).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:48.520 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:48.507+0000 7f7a7d124700 1 -- 172.21.15.42:0/2460988778 >> v2:172.21.15.42:3300/0 conn(0x7f7a781335a0 msgr2=0x7f7a781339d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2019-12-05T06:16:48.520 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:48.507+0000 7f7a7d124700 1 -- 172.21.15.42:0/2460988778 >> v2:172.21.15.42:3300/0 conn(0x7f7a781335a0 msgr2=0x7f7a781339d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2019-12-05T06:16:48.520 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:48.507+0000 7f7a7d124700 1 --2- 172.21.15.42:0/2460988778 >> v2:172.21.15.42:3300/0 conn(0x7f7a781335a0 0x7f7a781339d0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f7a7807ce20 tx=0x7f7a70010850).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-12-05T06:16:48.520 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:48.507+0000 7f7a7d124700 1 --2- 172.21.15.42:0/2460988778 >> v2:172.21.15.42:3300/0 conn(0x7f7a781335a0 0x7f7a781339d0 secure :-1 s=START_CONNECT pgs=0 cs=0 l=1 rx=0x7f7a7807ce20 tx=0x7f7a70010850)._fault waiting 0.200000 2019-12-05T06:16:48.716 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:48.711+0000 7f7a7d124700 1 --2- 172.21.15.42:0/2460988778 >> v2:172.21.15.42:3300/0 conn(0x7f7a781335a0 0x7f7a781339d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:16:48.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:48.711+0000 7f7a7d124700 0 --2- 172.21.15.42:0/2460988778 >> v2:172.21.15.42:3300/0 conn(0x7f7a781335a0 0x7f7a781339d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2019-12-05T06:16:48.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:48.711+0000 7f7a7d124700 1 --2- 172.21.15.42:0/2460988778 >> v2:172.21.15.42:3300/0 conn(0x7f7a781335a0 0x7f7a781339d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:16:48.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:48.711+0000 7f7a757fa700 1 -- 172.21.15.42:0/2460988778 >> v2:172.21.15.42:3300/0 conn(0x7f7a781335a0 msgr2=0x7f7a781339d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2019-12-05T06:16:48.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:48.711+0000 7f7a757fa700 1 --2- 172.21.15.42:0/2460988778 >> v2:172.21.15.42:3300/0 conn(0x7f7a781335a0 0x7f7a781339d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:16:48.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:48.711+0000 7f7a757fa700 1 -- 172.21.15.42:0/2460988778 --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a5c00aed0 con 0x7f7a5c006fa0 2019-12-05T06:16:48.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:48.711+0000 7f7a757fa700 1 -- 172.21.15.42:0/2460988778 --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a5c00c1a0 con 0x7f7a5c003090 2019-12-05T06:16:48.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:48.711+0000 7f7a757fa700 1 -- 172.21.15.42:0/2460988778 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2036874352 0 0) 0x7f7a5c00c1a0 con 0x7f7a5c003090 2019-12-05T06:16:48.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:48.711+0000 7f7a757fa700 1 -- 172.21.15.42:0/2460988778 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a5c00b1e0 con 0x7f7a5c003090 2019-12-05T06:16:48.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:48.711+0000 7f7a757fa700 1 -- 172.21.15.42:0/2460988778 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3052282660 0 0) 0x7f7a5c00aed0 con 0x7f7a5c006fa0 2019-12-05T06:16:48.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:48.711+0000 7f7a757fa700 1 -- 172.21.15.42:0/2460988778 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a5c00c1a0 con 0x7f7a5c006fa0 2019-12-05T06:16:48.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:48.711+0000 7f7a757fa700 1 -- 172.21.15.42:0/2460988778 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 1765255810 0 0) 0x7f7a5c00b1e0 con 0x7f7a5c003090 2019-12-05T06:16:48.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:48.711+0000 7f7a757fa700 1 -- 172.21.15.42:0/2460988778 --> v1:172.21.15.42:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7a5c00aed0 con 0x7f7a5c003090 2019-12-05T06:16:48.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:48.711+0000 7f7a757fa700 1 -- 172.21.15.42:0/2460988778 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7f7a68004620 con 0x7f7a5c003090 2019-12-05T06:16:48.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:48.711+0000 7f7a757fa700 1 -- 172.21.15.42:0/2460988778 >> v1:172.21.15.42:6789/0 conn(0x7f7a5c003090 legacy=0x7f7a5c005560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:48.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:48.711+0000 7f7a757fa700 1 -- 172.21.15.42:0/2460988778 >> v1:172.21.15.40:6789/0 conn(0x7f7a5c006fa0 legacy=0x7f7a5c0094b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:48.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:48.711+0000 7f7a757fa700 1 --2- 172.21.15.42:0/2460988778 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7a5c00c3d0 0x7f7a5c00c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:16:48.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:48.711+0000 7f7a757fa700 1 --2- 172.21.15.42:0/2460988778 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7a5c00cd50 0x7f7a5c011820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:16:48.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:48.711+0000 7f7a7d124700 1 -- 172.21.15.42:0/2460988778 reap_dead start 2019-12-05T06:16:48.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:48.711+0000 7f7a757fa700 1 --2- 172.21.15.42:0/2460988778 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7a5c006fa0 0x7f7a5c0094b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:16:48.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:48.711+0000 7f7a757fa700 1 -- 172.21.15.42:0/2460988778 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a5c009a00 con 0x7f7a5c006fa0 2019-12-05T06:16:48.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:48.711+0000 7f7a757fa700 1 -- 172.21.15.42:0/2460988778 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a5c00be70 con 0x7f7a5c00cd50 2019-12-05T06:16:48.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:48.711+0000 7f7a757fa700 1 -- 172.21.15.42:0/2460988778 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a5c009d60 con 0x7f7a5c00c3d0 2019-12-05T06:16:48.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:48.711+0000 7f7a77fff700 1 --2- 172.21.15.42:0/2460988778 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7a5c00cd50 0x7f7a5c011820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:16:48.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:48.711+0000 7f7a7c923700 1 --2- 172.21.15.42:0/2460988778 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7a5c00c3d0 0x7f7a5c00c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:16:48.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:48.711+0000 7f7a7d124700 1 --2- 172.21.15.42:0/2460988778 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7a5c006fa0 0x7f7a5c0094b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:16:48.724 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:48.711+0000 7f7a77fff700 1 -- 172.21.15.42:0/2460988778 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7a5c00c3d0 msgr2=0x7f7a5c00c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:48.724 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:48.711+0000 7f7a77fff700 1 --2- 172.21.15.42:0/2460988778 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7a5c00c3d0 0x7f7a5c00c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:16:48.724 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:48.711+0000 7f7a77fff700 1 -- 172.21.15.42:0/2460988778 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7a5c006fa0 msgr2=0x7f7a5c0094b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:48.724 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:48.711+0000 7f7a77fff700 1 --2- 172.21.15.42:0/2460988778 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7a5c006fa0 0x7f7a5c0094b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:16:48.724 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:48.711+0000 7f7a77fff700 1 -- 172.21.15.42:0/2460988778 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a5c00d530 con 0x7f7a5c00cd50 2019-12-05T06:16:48.724 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:48.711+0000 7f7a7e386700 1 -- 172.21.15.42:0/2460988778 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a7807c830 con 0x7f7a5c00cd50 2019-12-05T06:16:48.724 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:48.711+0000 7f7a77fff700 1 --2- 172.21.15.42:0/2460988778 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7a5c00cd50 0x7f7a5c011820 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f7a60008f60 tx=0x7f7a6000a4e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:48.725 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:48.715+0000 7f7a7e386700 1 -- 172.21.15.42:0/2460988778 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a78116720 con 0x7f7a5c00cd50 2019-12-05T06:16:48.725 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:48.715+0000 7f7a77fff700 1 --2- 172.21.15.42:0/2460988778 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7a5c00cd50 0x7f7a5c011820 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f7a60008f60 tx=0x7f7a6000a4e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:48.725 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:48.715+0000 7f7a77fff700 1 --2- 172.21.15.42:0/2460988778 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7a5c00cd50 0x7f7a5c011820 secure :-1 s=READY pgs=2359 cs=0 l=1 rx=0x7f7a60008f60 tx=0x7f7a6000a4e0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:16:48.725 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:48.715+0000 7f7a77fff700 1 --2- 172.21.15.42:0/2460988778 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7a5c00cd50 0x7f7a5c011820 secure :-1 s=THROTTLE_DONE pgs=2359 cs=0 l=1 rx=0x7f7a60008f60 tx=0x7f7a6000a4e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:48.725 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:48.715+0000 7f7a757fa700 1 -- 172.21.15.42:0/2460988778 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7a6000ad10 con 0x7f7a5c00cd50 2019-12-05T06:16:48.725 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:48.715+0000 7f7a77fff700 1 --2- 172.21.15.42:0/2460988778 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7a5c00cd50 0x7f7a5c011820 secure :-1 s=THROTTLE_DONE pgs=2359 cs=0 l=1 rx=0x7f7a60008f60 tx=0x7f7a6000a4e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:48.725 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:48.715+0000 7f7a77fff700 1 --2- 172.21.15.42:0/2460988778 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7a5c00cd50 0x7f7a5c011820 secure :-1 s=THROTTLE_DONE pgs=2359 cs=0 l=1 rx=0x7f7a60008f60 tx=0x7f7a6000a4e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:48.726 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:48.715+0000 7f7a757fa700 1 -- 172.21.15.42:0/2460988778 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a6000a930 con 0x7f7a5c00cd50 2019-12-05T06:16:48.726 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:48.715+0000 7f7a757fa700 1 -- 172.21.15.42:0/2460988778 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7a60005ca0 con 0x7f7a5c00cd50 2019-12-05T06:16:48.727 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:48.715+0000 7f7a77fff700 1 --2- 172.21.15.42:0/2460988778 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7a5c00cd50 0x7f7a5c011820 secure :-1 s=THROTTLE_DONE pgs=2359 cs=0 l=1 rx=0x7f7a60008f60 tx=0x7f7a6000a4e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:48.727 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:48.715+0000 7f7a757fa700 1 -- 172.21.15.42:0/2460988778 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f7a60007150 con 0x7f7a5c00cd50 2019-12-05T06:16:48.727 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:48.715+0000 7f7a77fff700 1 --2- 172.21.15.42:0/2460988778 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7a5c00cd50 0x7f7a5c011820 secure :-1 s=THROTTLE_DONE pgs=2359 cs=0 l=1 rx=0x7f7a60008f60 tx=0x7f7a6000a4e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:48.727 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:48.715+0000 7f7a757fa700 1 --2- 172.21.15.42:0/2460988778 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f7a5c044a40 0x7f7a5c046f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:16:48.727 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:48.715+0000 7f7a757fa700 1 -- 172.21.15.42:0/2460988778 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 10331+0+0 (secure 0 0 0) 0x7f7a6004c200 con 0x7f7a5c00cd50 2019-12-05T06:16:48.727 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:48.715+0000 7f7a7c923700 1 --2- 172.21.15.42:0/2460988778 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f7a5c044a40 0x7f7a5c046f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:16:48.728 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:48.715+0000 7f7a7c923700 1 --2- 172.21.15.42:0/2460988778 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f7a5c044a40 0x7f7a5c046f50 crc :-1 s=READY pgs=992 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:16:48.769 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:48.763+0000 7f7a7e386700 1 -- 172.21.15.42:0/2460988778 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a50003200 con 0x7f7a5c00cd50 2019-12-05T06:16:48.778 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:48.771+0000 7f7a77fff700 1 --2- 172.21.15.42:0/2460988778 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7a5c00cd50 0x7f7a5c011820 secure :-1 s=THROTTLE_DONE pgs=2359 cs=0 l=1 rx=0x7f7a60008f60 tx=0x7f7a6000a4e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:48.778 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:48.771+0000 7f7a757fa700 1 -- 172.21.15.42:0/2460988778 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f7a60033240 con 0x7f7a5c00cd50 2019-12-05T06:16:48.948 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:48.939+0000 7f7a7e386700 1 -- 172.21.15.42:0/2460988778 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7a50003630 con 0x7f7a5c00cd50 2019-12-05T06:16:48.948 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:48.943+0000 7f7a77fff700 1 --2- 172.21.15.42:0/2460988778 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7a5c00cd50 0x7f7a5c011820 secure :-1 s=THROTTLE_DONE pgs=2359 cs=0 l=1 rx=0x7f7a60008f60 tx=0x7f7a6000a4e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:48.949 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:48.943+0000 7f7a757fa700 1 -- 172.21.15.42:0/2460988778 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 283 v283) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f7a600334a0 con 0x7f7a5c00cd50 2019-12-05T06:16:48.949 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 283 2019-12-05T06:16:48.949 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:16:48.949 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":283,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7295,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/3228549071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3228549071},{"type":"v1","addr":"172.21.15.42:6835","nonce":3228549071}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":283}],"filesystems":[{"mdsmap":{"epoch":282,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:14:40.006630+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":99,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7217},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7217":{"gid":7217,"name":"b","rank":0,"incarnation":279,"state":"up:active","state_seq":21,"addr":"172.21.15.40:6833/2667648531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2667648531},{"type":"v1","addr":"172.21.15.40:6833","nonce":2667648531}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:16:48.951 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:48.943+0000 7f7a7e386700 1 -- 172.21.15.42:0/2460988778 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f7a5c044a40 msgr2=0x7f7a5c046f50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:48.951 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:48.943+0000 7f7a7e386700 1 --2- 172.21.15.42:0/2460988778 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f7a5c044a40 0x7f7a5c046f50 crc :-1 s=READY pgs=992 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:16:48.951 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:48.943+0000 7f7a7e386700 1 -- 172.21.15.42:0/2460988778 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7a5c00cd50 msgr2=0x7f7a5c011820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:48.952 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:48.943+0000 7f7a7e386700 1 --2- 172.21.15.42:0/2460988778 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7a5c00cd50 0x7f7a5c011820 secure :-1 s=READY pgs=2359 cs=0 l=1 rx=0x7f7a60008f60 tx=0x7f7a6000a4e0).stop 2019-12-05T06:16:48.952 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:48.943+0000 7f7a7e386700 1 -- 172.21.15.42:0/2460988778 shutdown_connections 2019-12-05T06:16:48.952 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:48.943+0000 7f7a7e386700 1 --2- 172.21.15.42:0/2460988778 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f7a5c044a40 0x7f7a5c046f50 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:16:48.952 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:48.943+0000 7f7a7e386700 1 --2- 172.21.15.42:0/2460988778 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7a5c00cd50 0x7f7a5c011820 unknown :-1 s=CLOSED pgs=2359 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:16:48.952 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:48.943+0000 7f7a7e386700 1 --2- 172.21.15.42:0/2460988778 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7a5c006fa0 0x7f7a5c0094b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:16:48.952 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:48.943+0000 7f7a7e386700 1 --2- 172.21.15.42:0/2460988778 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7a5c00c3d0 0x7f7a5c00c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:16:48.952 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:48.943+0000 7f7a7e386700 1 -- 172.21.15.42:0/2460988778 shutdown_connections 2019-12-05T06:16:48.953 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:48.943+0000 7f7a7e386700 1 -- 172.21.15.42:0/2460988778 wait complete. 2019-12-05T06:16:48.953 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:48.943+0000 7f7a7e386700 1 -- 172.21.15.42:0/2460988778 >> 172.21.15.42:0/2460988778 conn(0x7f7a780a8240 msgr2=0x7f7a78059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:16:48.960 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:16:50.962 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:16:51.072 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:51.059+0000 7f460c412700 1 Processor -- start 2019-12-05T06:16:51.072 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:51.059+0000 7f460c412700 1 -- start start 2019-12-05T06:16:51.073 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:51.063+0000 7f460c412700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f460408df80 0x7f460409f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:16:51.073 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:51.063+0000 7f460c412700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f460409f830 0x7f460414efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:16:51.073 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:51.063+0000 7f460c412700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f460409fc60 con 0x7f46040a7740 2019-12-05T06:16:51.073 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:51.063+0000 7f460c412700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46040a0b00 con 0x7f460409f830 2019-12-05T06:16:51.074 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:51.063+0000 7f460c412700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f460408e5b0 con 0x7f460408df80 2019-12-05T06:16:51.074 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:51.063+0000 7f460a1ae700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f460408df80 0x7f460409f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:16:51.074 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:51.063+0000 7f460a1ae700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f460408df80 0x7f460409f2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:51954/0 (socket says 172.21.15.42:51954) 2019-12-05T06:16:51.074 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:51.063+0000 7f460a1ae700 1 -- 172.21.15.42:0/2373501512 learned_addr learned my addr 172.21.15.42:0/2373501512 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:16:51.075 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:51.063+0000 7f460b1b0700 1 --2- 172.21.15.42:0/2373501512 >> v2:172.21.15.40:3300/0 conn(0x7f460409f830 0x7f460414efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:16:51.075 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:51.063+0000 7f460a1ae700 1 -- 172.21.15.42:0/2373501512 >> v2:172.21.15.40:3300/0 conn(0x7f460409f830 msgr2=0x7f460414efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:16:51.076 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:51.063+0000 7f460a1ae700 1 --2- 172.21.15.42:0/2373501512 >> v2:172.21.15.40:3300/0 conn(0x7f460409f830 0x7f460414efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:16:51.076 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:51.063+0000 7f460a1ae700 1 -- 172.21.15.42:0/2373501512 >> v1:172.21.15.40:6789/0 conn(0x7f46040a7740 legacy=0x7f460407e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:51.076 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:51.063+0000 7f460a1ae700 1 -- 172.21.15.42:0/2373501512 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46040a0710 con 0x7f460408df80 2019-12-05T06:16:51.076 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:51.063+0000 7f460b1b0700 1 --2- 172.21.15.42:0/2373501512 >> v2:172.21.15.40:3300/0 conn(0x7f460409f830 0x7f460414efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:16:51.077 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:51.063+0000 7f460a1ae700 1 --2- 172.21.15.42:0/2373501512 >> v2:172.21.15.42:3300/0 conn(0x7f460408df80 0x7f460409f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f4600002f30 tx=0x7f460000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:51.077 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:51.063+0000 7f460a1ae700 1 --2- 172.21.15.42:0/2373501512 >> v2:172.21.15.42:3300/0 conn(0x7f460408df80 0x7f460409f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f4600002f30 tx=0x7f460000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:51.077 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:51.063+0000 7f460a1ae700 1 --2- 172.21.15.42:0/2373501512 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f460408df80 0x7f460409f2e0 secure :-1 s=READY pgs=2360 cs=0 l=1 rx=0x7f4600002f30 tx=0x7f460000fd30).ready entity=mon.1 client_cookie=ec2397ec790bb438 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:16:51.077 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:51.063+0000 7f460a1ae700 1 --2- 172.21.15.42:0/2373501512 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f460408df80 0x7f460409f2e0 secure :-1 s=THROTTLE_DONE pgs=2360 cs=0 l=1 rx=0x7f4600002f30 tx=0x7f460000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:51.077 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:51.063+0000 7f46099ad700 1 -- 172.21.15.42:0/2373501512 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4600008c90 con 0x7f460408df80 2019-12-05T06:16:51.077 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:51.063+0000 7f460a1ae700 1 --2- 172.21.15.42:0/2373501512 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f460408df80 0x7f460409f2e0 secure :-1 s=THROTTLE_DONE pgs=2360 cs=0 l=1 rx=0x7f4600002f30 tx=0x7f460000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:51.077 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:51.063+0000 7f46099ad700 1 -- 172.21.15.42:0/2373501512 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46000088b0 con 0x7f460408df80 2019-12-05T06:16:51.078 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:51.063+0000 7f460a1ae700 1 --2- 172.21.15.42:0/2373501512 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f460408df80 0x7f460409f2e0 secure :-1 s=THROTTLE_DONE pgs=2360 cs=0 l=1 rx=0x7f4600002f30 tx=0x7f460000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:51.078 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:51.063+0000 7f46099ad700 1 -- 172.21.15.42:0/2373501512 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4600011ad0 con 0x7f460408df80 2019-12-05T06:16:51.078 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:51.063+0000 7f460c412700 1 -- 172.21.15.42:0/2373501512 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f460408df80 msgr2=0x7f460409f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:51.078 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:51.063+0000 7f460c412700 1 --2- 172.21.15.42:0/2373501512 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f460408df80 0x7f460409f2e0 secure :-1 s=READY pgs=2360 cs=0 l=1 rx=0x7f4600002f30 tx=0x7f460000fd30).stop 2019-12-05T06:16:51.078 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:51.063+0000 7f460c412700 1 -- 172.21.15.42:0/2373501512 shutdown_connections 2019-12-05T06:16:51.078 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:51.063+0000 7f460c412700 1 --2- 172.21.15.42:0/2373501512 >> v2:172.21.15.40:3300/0 conn(0x7f460409f830 0x7f460414efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:16:51.078 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:51.063+0000 7f460c412700 1 --2- 172.21.15.42:0/2373501512 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f460408df80 0x7f460409f2e0 unknown :-1 s=CLOSED pgs=2360 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:16:51.079 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:51.063+0000 7f460c412700 1 -- 172.21.15.42:0/2373501512 shutdown_connections 2019-12-05T06:16:51.079 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:51.063+0000 7f460c412700 1 -- 172.21.15.42:0/2373501512 wait complete. 2019-12-05T06:16:51.079 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:51.063+0000 7f460c412700 1 -- 172.21.15.42:0/2373501512 >> 172.21.15.42:0/2373501512 conn(0x7f46040aabe0 msgr2=0x7f46040a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:16:51.082 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:51.067+0000 7f460c412700 1 Processor -- start 2019-12-05T06:16:51.082 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:51.067+0000 7f460c412700 1 -- start start 2019-12-05T06:16:51.082 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:51.071+0000 7f460c412700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f46040639d0 0x7f46041266b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:16:51.082 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:51.071+0000 7f460c412700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46040650c0 con 0x7f460408eeb0 2019-12-05T06:16:51.083 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:51.071+0000 7f460c412700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f460414ed10 con 0x7f4604126c00 2019-12-05T06:16:51.083 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:51.071+0000 7f460c412700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46040653d0 con 0x7f46040639d0 2019-12-05T06:16:51.083 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:51.071+0000 7f460b1b0700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f4604126c00 0x7f46041310c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:52000/0 (socket says 172.21.15.42:52000) 2019-12-05T06:16:51.083 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:51.071+0000 7f460b1b0700 1 -- 172.21.15.42:0/416499901 learned_addr learned my addr 172.21.15.42:0/416499901 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:16:51.083 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:51.071+0000 7f460a1ae700 1 --2- 172.21.15.42:0/416499901 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f46040639d0 0x7f46041266b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:16:51.083 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:51.071+0000 7f45f77fe700 1 -- 172.21.15.42:0/416499901 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1095676855 0 0) 0x7f460414ed10 con 0x7f4604126c00 2019-12-05T06:16:51.084 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:51.071+0000 7f45f77fe700 1 -- 172.21.15.42:0/416499901 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45ec0030b0 con 0x7f4604126c00 2019-12-05T06:16:51.084 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:51.071+0000 7f45f77fe700 1 -- 172.21.15.42:0/416499901 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3488156458 0 0) 0x7f46040650c0 con 0x7f460408eeb0 2019-12-05T06:16:51.084 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:51.071+0000 7f45f77fe700 1 -- 172.21.15.42:0/416499901 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f460414ed10 con 0x7f460408eeb0 2019-12-05T06:16:51.084 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:51.071+0000 7f460a1ae700 1 -- 172.21.15.42:0/416499901 >> v1:172.21.15.42:6789/0 conn(0x7f4604126c00 legacy=0x7f46041310c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:51.084 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:51.071+0000 7f460a1ae700 1 -- 172.21.15.42:0/416499901 >> v1:172.21.15.40:6789/0 conn(0x7f460408eeb0 legacy=0x7f4604144d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:51.084 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:51.071+0000 7f460a1ae700 1 -- 172.21.15.42:0/416499901 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4604127030 con 0x7f46040639d0 2019-12-05T06:16:51.084 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:51.071+0000 7f460a1ae700 1 --2- 172.21.15.42:0/416499901 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f46040639d0 0x7f46041266b0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f4604064de0 tx=0x7f4600008db0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:51.085 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:51.071+0000 7f460a1ae700 1 --2- 172.21.15.42:0/416499901 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f46040639d0 0x7f46041266b0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f4604064de0 tx=0x7f4600008db0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:51.085 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:51.071+0000 7f460a1ae700 1 --2- 172.21.15.42:0/416499901 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f46040639d0 0x7f46041266b0 secure :-1 s=READY pgs=524 cs=0 l=1 rx=0x7f4604064de0 tx=0x7f4600008db0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:16:51.085 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:51.071+0000 7f460a1ae700 1 --2- 172.21.15.42:0/416499901 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f46040639d0 0x7f46041266b0 secure :-1 s=THROTTLE_DONE pgs=524 cs=0 l=1 rx=0x7f4604064de0 tx=0x7f4600008db0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:51.085 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:51.071+0000 7f45f77fe700 1 -- 172.21.15.42:0/416499901 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f460000f680 con 0x7f46040639d0 2019-12-05T06:16:51.085 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:51.071+0000 7f460a1ae700 1 --2- 172.21.15.42:0/416499901 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f46040639d0 0x7f46041266b0 secure :-1 s=THROTTLE_DONE pgs=524 cs=0 l=1 rx=0x7f4604064de0 tx=0x7f4600008db0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:51.086 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:51.071+0000 7f460a1ae700 1 --2- 172.21.15.42:0/416499901 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f46040639d0 0x7f46041266b0 secure :-1 s=THROTTLE_DONE pgs=524 cs=0 l=1 rx=0x7f4604064de0 tx=0x7f4600008db0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:51.086 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:51.071+0000 7f45f77fe700 1 -- 172.21.15.42:0/416499901 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46000116f0 con 0x7f46040639d0 2019-12-05T06:16:51.086 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:51.071+0000 7f45f77fe700 1 -- 172.21.15.42:0/416499901 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4600005920 con 0x7f46040639d0 2019-12-05T06:16:51.086 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:51.071+0000 7f460c412700 1 -- 172.21.15.42:0/416499901 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4604063e00 con 0x7f46040639d0 2019-12-05T06:16:51.086 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:51.071+0000 7f460c412700 1 -- 172.21.15.42:0/416499901 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46041328d0 con 0x7f46040639d0 2019-12-05T06:16:51.086 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:51.071+0000 7f460a1ae700 1 --2- 172.21.15.42:0/416499901 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f46040639d0 0x7f46041266b0 secure :-1 s=THROTTLE_DONE pgs=524 cs=0 l=1 rx=0x7f4604064de0 tx=0x7f4600008db0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:51.086 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:51.071+0000 7f45f77fe700 1 -- 172.21.15.42:0/416499901 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f4600005b70 con 0x7f46040639d0 2019-12-05T06:16:51.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:51.075+0000 7f460a1ae700 1 --2- 172.21.15.42:0/416499901 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f46040639d0 0x7f46041266b0 secure :-1 s=THROTTLE_DONE pgs=524 cs=0 l=1 rx=0x7f4604064de0 tx=0x7f4600008db0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:51.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:51.075+0000 7f45f77fe700 1 --2- 172.21.15.42:0/416499901 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f45ec03b490 0x7f45ec03d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:16:51.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:51.075+0000 7f45f77fe700 1 -- 172.21.15.42:0/416499901 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 10331+0+0 (secure 0 0 0) 0x7f460004ba60 con 0x7f46040639d0 2019-12-05T06:16:51.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:51.075+0000 7f460a9af700 1 --2- 172.21.15.42:0/416499901 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f45ec03b490 0x7f45ec03d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:16:51.088 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:51.075+0000 7f460a9af700 1 --2- 172.21.15.42:0/416499901 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f45ec03b490 0x7f45ec03d9a0 crc :-1 s=READY pgs=993 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:16:51.089 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:51.075+0000 7f460c412700 1 -- 172.21.15.42:0/416499901 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f460405d110 con 0x7f46040639d0 2019-12-05T06:16:51.092 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:51.083+0000 7f460a1ae700 1 --2- 172.21.15.42:0/416499901 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f46040639d0 0x7f46041266b0 secure :-1 s=THROTTLE_DONE pgs=524 cs=0 l=1 rx=0x7f4604064de0 tx=0x7f4600008db0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:51.092 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:51.083+0000 7f45f77fe700 1 -- 172.21.15.42:0/416499901 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f4600030730 con 0x7f46040639d0 2019-12-05T06:16:51.263 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:51.255+0000 7f460c412700 1 -- 172.21.15.42:0/416499901 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f460405b8c0 con 0x7f46040639d0 2019-12-05T06:16:51.264 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:51.259+0000 7f460a1ae700 1 --2- 172.21.15.42:0/416499901 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f46040639d0 0x7f46041266b0 secure :-1 s=THROTTLE_DONE pgs=524 cs=0 l=1 rx=0x7f4604064de0 tx=0x7f4600008db0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:51.264 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:51.259+0000 7f45f77fe700 1 -- 172.21.15.42:0/416499901 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 283 v283) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f46000338f0 con 0x7f46040639d0 2019-12-05T06:16:51.266 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 283 2019-12-05T06:16:51.266 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:16:51.267 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":283,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7295,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/3228549071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3228549071},{"type":"v1","addr":"172.21.15.42:6835","nonce":3228549071}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":283}],"filesystems":[{"mdsmap":{"epoch":282,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:14:40.006630+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":99,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7217},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7217":{"gid":7217,"name":"b","rank":0,"incarnation":279,"state":"up:active","state_seq":21,"addr":"172.21.15.40:6833/2667648531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2667648531},{"type":"v1","addr":"172.21.15.40:6833","nonce":2667648531}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:16:51.270 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:51.263+0000 7f460c412700 1 -- 172.21.15.42:0/416499901 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f45ec03b490 msgr2=0x7f45ec03d9a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:51.270 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:51.263+0000 7f460c412700 1 --2- 172.21.15.42:0/416499901 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f45ec03b490 0x7f45ec03d9a0 crc :-1 s=READY pgs=993 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:16:51.271 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:51.263+0000 7f460c412700 1 -- 172.21.15.42:0/416499901 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f46040639d0 msgr2=0x7f46041266b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:51.271 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:51.263+0000 7f460c412700 1 --2- 172.21.15.42:0/416499901 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f46040639d0 0x7f46041266b0 secure :-1 s=READY pgs=524 cs=0 l=1 rx=0x7f4604064de0 tx=0x7f4600008db0).stop 2019-12-05T06:16:51.271 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:51.263+0000 7f460c412700 1 -- 172.21.15.42:0/416499901 shutdown_connections 2019-12-05T06:16:51.271 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:51.263+0000 7f460c412700 1 --2- 172.21.15.42:0/416499901 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f45ec03b490 0x7f45ec03d9a0 unknown :-1 s=CLOSED pgs=993 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:16:51.272 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:51.263+0000 7f460c412700 1 --2- 172.21.15.42:0/416499901 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f46040639d0 0x7f46041266b0 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:16:51.272 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:51.263+0000 7f460c412700 1 -- 172.21.15.42:0/416499901 shutdown_connections 2019-12-05T06:16:51.272 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:51.263+0000 7f460c412700 1 -- 172.21.15.42:0/416499901 wait complete. 2019-12-05T06:16:51.272 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:51.263+0000 7f460c412700 1 -- 172.21.15.42:0/416499901 >> 172.21.15.42:0/416499901 conn(0x7f46040a6f30 msgr2=0x7f460409f380 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:16:51.282 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:16:53.284 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:16:53.390 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:53.375+0000 7f81bdbab700 1 Processor -- start 2019-12-05T06:16:53.390 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:53.375+0000 7f81bdbab700 1 -- start start 2019-12-05T06:16:53.390 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:53.379+0000 7f81bdbab700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f81b808df80 0x7f81b809f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:16:53.390 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:53.379+0000 7f81bdbab700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f81b809f830 0x7f81b814efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:16:53.390 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:53.379+0000 7f81bdbab700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81b809fc60 con 0x7f81b80a7740 2019-12-05T06:16:53.391 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:53.379+0000 7f81bdbab700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81b80a0b00 con 0x7f81b808df80 2019-12-05T06:16:53.391 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:53.379+0000 7f81bdbab700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81b808e5b0 con 0x7f81b809f830 2019-12-05T06:16:53.391 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:53.379+0000 7f81b7fff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f81b80a7740 0x7f81b807e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:52010/0 (socket says 172.21.15.42:52010) 2019-12-05T06:16:53.391 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:53.379+0000 7f81b7fff700 1 -- 172.21.15.42:0/738660696 learned_addr learned my addr 172.21.15.42:0/738660696 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:16:53.391 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:53.379+0000 7f81bc949700 1 --2- 172.21.15.42:0/738660696 >> v2:172.21.15.42:3300/0 conn(0x7f81b809f830 0x7f81b814efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:16:53.391 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:53.379+0000 7f81b77fe700 1 --2- 172.21.15.42:0/738660696 >> v2:172.21.15.40:3300/0 conn(0x7f81b808df80 0x7f81b809f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:16:53.392 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:53.379+0000 7f81b6ffd700 1 -- 172.21.15.42:0/738660696 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2051818065 0 0) 0x7f81b809fc60 con 0x7f81b80a7740 2019-12-05T06:16:53.392 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:53.379+0000 7f81b6ffd700 1 -- 172.21.15.42:0/738660696 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81a00030b0 con 0x7f81b80a7740 2019-12-05T06:16:53.392 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:53.379+0000 7f81b6ffd700 1 -- 172.21.15.42:0/738660696 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4115014843 0 0) 0x7f81a00030b0 con 0x7f81b80a7740 2019-12-05T06:16:53.392 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:53.379+0000 7f81bc949700 1 -- 172.21.15.42:0/738660696 >> v2:172.21.15.40:3300/0 conn(0x7f81b808df80 msgr2=0x7f81b809f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:16:53.392 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:53.379+0000 7f81bc949700 1 --2- 172.21.15.42:0/738660696 >> v2:172.21.15.40:3300/0 conn(0x7f81b808df80 0x7f81b809f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:16:53.392 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:53.379+0000 7f81bc949700 1 -- 172.21.15.42:0/738660696 >> v1:172.21.15.42:6789/0 conn(0x7f81b80a7740 legacy=0x7f81b807e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:53.392 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:53.379+0000 7f81bc949700 1 -- 172.21.15.42:0/738660696 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81b80a0710 con 0x7f81b809f830 2019-12-05T06:16:53.393 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:53.379+0000 7f81b77fe700 1 --2- 172.21.15.42:0/738660696 >> v2:172.21.15.40:3300/0 conn(0x7f81b808df80 0x7f81b809f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:16:53.393 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:53.379+0000 7f81bc949700 1 --2- 172.21.15.42:0/738660696 >> v2:172.21.15.42:3300/0 conn(0x7f81b809f830 0x7f81b814efe0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f81a8001d10 tx=0x7f81a8001d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:53.393 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:53.379+0000 7f81bc949700 1 --2- 172.21.15.42:0/738660696 >> v2:172.21.15.42:3300/0 conn(0x7f81b809f830 0x7f81b814efe0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f81a8001d10 tx=0x7f81a8001d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:53.393 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:53.379+0000 7f81bc949700 1 --2- 172.21.15.42:0/738660696 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f81b809f830 0x7f81b814efe0 secure :-1 s=READY pgs=2363 cs=0 l=1 rx=0x7f81a8001d10 tx=0x7f81a8001d50).ready entity=mon.1 client_cookie=6cdb130f1215e73d server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:16:53.394 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:53.379+0000 7f81bc949700 1 --2- 172.21.15.42:0/738660696 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f81b809f830 0x7f81b814efe0 secure :-1 s=THROTTLE_DONE pgs=2363 cs=0 l=1 rx=0x7f81a8001d10 tx=0x7f81a8001d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:53.394 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:53.379+0000 7f81b6ffd700 1 -- 172.21.15.42:0/738660696 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f81a8001ad0 con 0x7f81b809f830 2019-12-05T06:16:53.394 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:53.379+0000 7f81bc949700 1 --2- 172.21.15.42:0/738660696 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f81b809f830 0x7f81b814efe0 secure :-1 s=THROTTLE_DONE pgs=2363 cs=0 l=1 rx=0x7f81a8001d10 tx=0x7f81a8001d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:53.394 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:53.379+0000 7f81b6ffd700 1 -- 172.21.15.42:0/738660696 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81a80018e0 con 0x7f81b809f830 2019-12-05T06:16:53.394 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:53.379+0000 7f81bc949700 1 --2- 172.21.15.42:0/738660696 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f81b809f830 0x7f81b814efe0 secure :-1 s=THROTTLE_DONE pgs=2363 cs=0 l=1 rx=0x7f81a8001d10 tx=0x7f81a8001d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:53.394 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:53.379+0000 7f81b6ffd700 1 -- 172.21.15.42:0/738660696 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f81b0011480 con 0x7f81b809f830 2019-12-05T06:16:53.395 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:53.379+0000 7f81bdbab700 1 -- 172.21.15.42:0/738660696 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f81b809f830 msgr2=0x7f81b814efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:53.395 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:53.379+0000 7f81bdbab700 1 --2- 172.21.15.42:0/738660696 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f81b809f830 0x7f81b814efe0 secure :-1 s=READY pgs=2363 cs=0 l=1 rx=0x7f81a8001d10 tx=0x7f81a8001d50).stop 2019-12-05T06:16:53.395 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:53.379+0000 7f81bdbab700 1 -- 172.21.15.42:0/738660696 shutdown_connections 2019-12-05T06:16:53.395 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:53.379+0000 7f81bdbab700 1 --2- 172.21.15.42:0/738660696 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f81b809f830 0x7f81b814efe0 unknown :-1 s=CLOSED pgs=2363 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:16:53.395 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:53.379+0000 7f81bdbab700 1 --2- 172.21.15.42:0/738660696 >> v2:172.21.15.40:3300/0 conn(0x7f81b808df80 0x7f81b809f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:16:53.395 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:53.379+0000 7f81bdbab700 1 -- 172.21.15.42:0/738660696 shutdown_connections 2019-12-05T06:16:53.396 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:53.383+0000 7f81bdbab700 1 -- 172.21.15.42:0/738660696 wait complete. 2019-12-05T06:16:53.396 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:53.383+0000 7f81bdbab700 1 -- 172.21.15.42:0/738660696 >> 172.21.15.42:0/738660696 conn(0x7f81b80aabe0 msgr2=0x7f81b80a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:16:53.396 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:53.383+0000 7f81bdbab700 1 Processor -- start 2019-12-05T06:16:53.396 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:53.383+0000 7f81bdbab700 1 -- start start 2019-12-05T06:16:53.400 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:53.383+0000 7f81bdbab700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f81b808f4c0 0x7f81b81302e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:16:53.400 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:53.383+0000 7f81bdbab700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81b8130830 con 0x7f81b811af90 2019-12-05T06:16:53.400 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:53.383+0000 7f81bdbab700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81b808f8f0 con 0x7f81b80a6110 2019-12-05T06:16:53.400 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:53.383+0000 7f81bdbab700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81b80a6540 con 0x7f81b808f4c0 2019-12-05T06:16:53.400 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:53.383+0000 7f81b77fe700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f81b80a6110 0x7f81b8135720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:52018/0 (socket says 172.21.15.42:52018) 2019-12-05T06:16:53.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:53.383+0000 7f81b77fe700 1 -- 172.21.15.42:0/1417888024 learned_addr learned my addr 172.21.15.42:0/1417888024 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:16:53.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:53.383+0000 7f81bc949700 1 --2- 172.21.15.42:0/1417888024 >> v2:172.21.15.42:3300/0 conn(0x7f81b808f4c0 0x7f81b81302e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:16:53.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:53.383+0000 7f81bc949700 1 -- 172.21.15.42:0/1417888024 >> v1:172.21.15.42:6789/0 conn(0x7f81b80a6110 legacy=0x7f81b8135720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:53.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:53.387+0000 7f81bc949700 1 -- 172.21.15.42:0/1417888024 >> v1:172.21.15.40:6789/0 conn(0x7f81b811af90 legacy=0x7f81b811a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:53.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:53.383+0000 7f81b4ff9700 1 -- 172.21.15.42:0/1417888024 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 857189234 0 0) 0x7f81b8130830 con 0x7f81b811af90 2019-12-05T06:16:53.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:53.387+0000 7f81bc949700 1 -- 172.21.15.42:0/1417888024 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81b80a6a00 con 0x7f81b808f4c0 2019-12-05T06:16:53.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:53.387+0000 7f81bc949700 1 --2- 172.21.15.42:0/1417888024 >> v2:172.21.15.42:3300/0 conn(0x7f81b808f4c0 0x7f81b81302e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f81b8121f70 tx=0x7f81b0011580).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:53.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:53.387+0000 7f81bc949700 1 --2- 172.21.15.42:0/1417888024 >> v2:172.21.15.42:3300/0 conn(0x7f81b808f4c0 0x7f81b81302e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f81b8121f70 tx=0x7f81b0011580).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:53.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:53.387+0000 7f81bc949700 1 --2- 172.21.15.42:0/1417888024 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f81b808f4c0 0x7f81b81302e0 secure :-1 s=READY pgs=2365 cs=0 l=1 rx=0x7f81b8121f70 tx=0x7f81b0011580).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:16:53.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:53.387+0000 7f81bc949700 1 --2- 172.21.15.42:0/1417888024 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f81b808f4c0 0x7f81b81302e0 secure :-1 s=THROTTLE_DONE pgs=2365 cs=0 l=1 rx=0x7f81b8121f70 tx=0x7f81b0011580).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:53.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:53.387+0000 7f81b4ff9700 1 -- 172.21.15.42:0/1417888024 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f81b0007350 con 0x7f81b808f4c0 2019-12-05T06:16:53.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:53.387+0000 7f81bc949700 1 --2- 172.21.15.42:0/1417888024 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f81b808f4c0 0x7f81b81302e0 secure :-1 s=THROTTLE_DONE pgs=2365 cs=0 l=1 rx=0x7f81b8121f70 tx=0x7f81b0011580).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:53.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:53.387+0000 7f81bc949700 1 --2- 172.21.15.42:0/1417888024 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f81b808f4c0 0x7f81b81302e0 secure :-1 s=THROTTLE_DONE pgs=2365 cs=0 l=1 rx=0x7f81b8121f70 tx=0x7f81b0011580).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:53.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:53.387+0000 7f81b4ff9700 1 -- 172.21.15.42:0/1417888024 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81b0008b60 con 0x7f81b808f4c0 2019-12-05T06:16:53.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:53.387+0000 7f81bdbab700 1 -- 172.21.15.42:0/1417888024 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81b81161c0 con 0x7f81b808f4c0 2019-12-05T06:16:53.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:53.387+0000 7f81b4ff9700 1 -- 172.21.15.42:0/1417888024 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f81b0011b00 con 0x7f81b808f4c0 2019-12-05T06:16:53.404 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:53.387+0000 7f81bdbab700 1 -- 172.21.15.42:0/1417888024 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81b8116710 con 0x7f81b808f4c0 2019-12-05T06:16:53.404 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:53.387+0000 7f81bc949700 1 --2- 172.21.15.42:0/1417888024 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f81b808f4c0 0x7f81b81302e0 secure :-1 s=THROTTLE_DONE pgs=2365 cs=0 l=1 rx=0x7f81b8121f70 tx=0x7f81b0011580).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:53.404 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:53.387+0000 7f81b4ff9700 1 -- 172.21.15.42:0/1417888024 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f81b0005540 con 0x7f81b808f4c0 2019-12-05T06:16:53.404 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:53.387+0000 7f81bc949700 1 --2- 172.21.15.42:0/1417888024 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f81b808f4c0 0x7f81b81302e0 secure :-1 s=THROTTLE_DONE pgs=2365 cs=0 l=1 rx=0x7f81b8121f70 tx=0x7f81b0011580).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:53.404 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:53.387+0000 7f81b4ff9700 1 --2- 172.21.15.42:0/1417888024 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f819803ac30 0x7f819803d140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:16:53.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:53.387+0000 7f81b4ff9700 1 -- 172.21.15.42:0/1417888024 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 10331+0+0 (secure 0 0 0) 0x7f81b004b090 con 0x7f81b808f4c0 2019-12-05T06:16:53.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:53.387+0000 7f81b7fff700 1 --2- 172.21.15.42:0/1417888024 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f819803ac30 0x7f819803d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:16:53.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:53.387+0000 7f81b7fff700 1 --2- 172.21.15.42:0/1417888024 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f819803ac30 0x7f819803d140 crc :-1 s=READY pgs=994 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:16:53.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:53.391+0000 7f81bdbab700 1 -- 172.21.15.42:0/1417888024 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81b805d110 con 0x7f81b808f4c0 2019-12-05T06:16:53.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:53.395+0000 7f81bc949700 1 --2- 172.21.15.42:0/1417888024 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f81b808f4c0 0x7f81b81302e0 secure :-1 s=THROTTLE_DONE pgs=2365 cs=0 l=1 rx=0x7f81b8121f70 tx=0x7f81b0011580).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:53.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:53.395+0000 7f81b4ff9700 1 -- 172.21.15.42:0/1417888024 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f81b0007f50 con 0x7f81b808f4c0 2019-12-05T06:16:53.570 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:53.563+0000 7f81bdbab700 1 -- 172.21.15.42:0/1417888024 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f81b805b8c0 con 0x7f81b808f4c0 2019-12-05T06:16:53.570 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:53.563+0000 7f81bc949700 1 --2- 172.21.15.42:0/1417888024 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f81b808f4c0 0x7f81b81302e0 secure :-1 s=THROTTLE_DONE pgs=2365 cs=0 l=1 rx=0x7f81b8121f70 tx=0x7f81b0011580).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:53.570 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:53.563+0000 7f81b4ff9700 1 -- 172.21.15.42:0/1417888024 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 283 v283) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f81b0032370 con 0x7f81b808f4c0 2019-12-05T06:16:53.572 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 283 2019-12-05T06:16:53.572 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:16:53.572 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":283,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7295,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/3228549071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3228549071},{"type":"v1","addr":"172.21.15.42:6835","nonce":3228549071}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":283}],"filesystems":[{"mdsmap":{"epoch":282,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:14:40.006630+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":99,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7217},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7217":{"gid":7217,"name":"b","rank":0,"incarnation":279,"state":"up:active","state_seq":21,"addr":"172.21.15.40:6833/2667648531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2667648531},{"type":"v1","addr":"172.21.15.40:6833","nonce":2667648531}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:16:53.575 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:53.567+0000 7f81bdbab700 1 -- 172.21.15.42:0/1417888024 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f819803ac30 msgr2=0x7f819803d140 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:53.575 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:53.567+0000 7f81bdbab700 1 --2- 172.21.15.42:0/1417888024 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f819803ac30 0x7f819803d140 crc :-1 s=READY pgs=994 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:16:53.575 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:53.567+0000 7f81bdbab700 1 -- 172.21.15.42:0/1417888024 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f81b808f4c0 msgr2=0x7f81b81302e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:53.575 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:53.567+0000 7f81bdbab700 1 --2- 172.21.15.42:0/1417888024 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f81b808f4c0 0x7f81b81302e0 secure :-1 s=READY pgs=2365 cs=0 l=1 rx=0x7f81b8121f70 tx=0x7f81b0011580).stop 2019-12-05T06:16:53.575 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:53.567+0000 7f81bdbab700 1 -- 172.21.15.42:0/1417888024 shutdown_connections 2019-12-05T06:16:53.575 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:53.567+0000 7f81bdbab700 1 --2- 172.21.15.42:0/1417888024 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f819803ac30 0x7f819803d140 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:16:53.576 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:53.567+0000 7f81bdbab700 1 --2- 172.21.15.42:0/1417888024 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f81b808f4c0 0x7f81b81302e0 unknown :-1 s=CLOSED pgs=2365 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:16:53.576 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:53.567+0000 7f81bdbab700 1 -- 172.21.15.42:0/1417888024 shutdown_connections 2019-12-05T06:16:53.576 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:53.567+0000 7f81bdbab700 1 -- 172.21.15.42:0/1417888024 wait complete. 2019-12-05T06:16:53.576 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:53.567+0000 7f81bdbab700 1 -- 172.21.15.42:0/1417888024 >> 172.21.15.42:0/1417888024 conn(0x7f81b808fde0 msgr2=0x7f81b80a9790 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:16:53.585 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:16:55.586 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:16:55.689 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:55.675+0000 7f09b7498700 1 Processor -- start 2019-12-05T06:16:55.689 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:55.679+0000 7f09b7498700 1 -- start start 2019-12-05T06:16:55.689 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:55.679+0000 7f09b7498700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f09b00a7740 0x7f09b008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:16:55.689 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:55.679+0000 7f09b7498700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f09b009fae0 0x7f09b007eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:16:55.689 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:55.679+0000 7f09b7498700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09b00a0710 con 0x7f09b007e720 2019-12-05T06:16:55.690 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:55.679+0000 7f09b7498700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09b00a0b00 con 0x7f09b00a7740 2019-12-05T06:16:55.690 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:55.679+0000 7f09b7498700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09b008e5b0 con 0x7f09b009fae0 2019-12-05T06:16:55.690 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:55.679+0000 7f09b5234700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f09b007e720 0x7f09b009f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:52026/0 (socket says 172.21.15.42:52026) 2019-12-05T06:16:55.690 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:55.679+0000 7f09b5234700 1 -- 172.21.15.42:0/2512313402 learned_addr learned my addr 172.21.15.42:0/2512313402 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:16:55.690 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:55.679+0000 7f09b5a35700 1 --2- 172.21.15.42:0/2512313402 >> v2:172.21.15.40:3300/0 conn(0x7f09b00a7740 0x7f09b008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:16:55.690 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:55.679+0000 7f09b6236700 1 --2- 172.21.15.42:0/2512313402 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f09b009fae0 0x7f09b007eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:16:55.690 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:55.679+0000 7f09b4a33700 1 -- 172.21.15.42:0/2512313402 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 248942143 0 0) 0x7f09b00a0710 con 0x7f09b007e720 2019-12-05T06:16:55.691 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:55.679+0000 7f09b4a33700 1 -- 172.21.15.42:0/2512313402 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09980030b0 con 0x7f09b007e720 2019-12-05T06:16:55.691 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:55.679+0000 7f09b5a35700 1 -- 172.21.15.42:0/2512313402 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f09b009fae0 msgr2=0x7f09b007eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:16:55.691 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:55.679+0000 7f09b5a35700 1 --2- 172.21.15.42:0/2512313402 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f09b009fae0 0x7f09b007eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:16:55.691 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:55.679+0000 7f09b5a35700 1 -- 172.21.15.42:0/2512313402 >> v1:172.21.15.42:6789/0 conn(0x7f09b007e720 legacy=0x7f09b009f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:55.691 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:55.679+0000 7f09b5a35700 1 -- 172.21.15.42:0/2512313402 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09b007f0a0 con 0x7f09b00a7740 2019-12-05T06:16:55.691 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:55.679+0000 7f09b5a35700 1 --2- 172.21.15.42:0/2512313402 >> v2:172.21.15.40:3300/0 conn(0x7f09b00a7740 0x7f09b008df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f09a0008b10 tx=0x7f09a000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:55.692 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:55.679+0000 7f09b5a35700 1 --2- 172.21.15.42:0/2512313402 >> v2:172.21.15.40:3300/0 conn(0x7f09b00a7740 0x7f09b008df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f09a0008b10 tx=0x7f09a000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:55.692 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:55.679+0000 7f09b5a35700 1 --2- 172.21.15.42:0/2512313402 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f09b00a7740 0x7f09b008df80 secure :-1 s=READY pgs=1797 cs=0 l=1 rx=0x7f09a0008b10 tx=0x7f09a000fff0).ready entity=mon.0 client_cookie=c6122c758e045620 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:16:55.692 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:55.679+0000 7f09b5a35700 1 --2- 172.21.15.42:0/2512313402 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f09b00a7740 0x7f09b008df80 secure :-1 s=THROTTLE_DONE pgs=1797 cs=0 l=1 rx=0x7f09a0008b10 tx=0x7f09a000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:55.692 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:55.679+0000 7f09b4a33700 1 -- 172.21.15.42:0/2512313402 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f09a0010900 con 0x7f09b00a7740 2019-12-05T06:16:55.692 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:55.679+0000 7f09b5a35700 1 --2- 172.21.15.42:0/2512313402 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f09b00a7740 0x7f09b008df80 secure :-1 s=THROTTLE_DONE pgs=1797 cs=0 l=1 rx=0x7f09a0008b10 tx=0x7f09a000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:55.692 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:55.679+0000 7f09b5a35700 1 --2- 172.21.15.42:0/2512313402 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f09b00a7740 0x7f09b008df80 secure :-1 s=THROTTLE_DONE pgs=1797 cs=0 l=1 rx=0x7f09a0008b10 tx=0x7f09a000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:55.692 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:55.679+0000 7f09b4a33700 1 -- 172.21.15.42:0/2512313402 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09a0008b50 con 0x7f09b00a7740 2019-12-05T06:16:55.693 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:55.679+0000 7f09b4a33700 1 -- 172.21.15.42:0/2512313402 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f09a00110c0 con 0x7f09b00a7740 2019-12-05T06:16:55.694 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:55.679+0000 7f09b7498700 1 -- 172.21.15.42:0/2512313402 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f09b00a7740 msgr2=0x7f09b008df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:55.694 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:55.679+0000 7f09b7498700 1 --2- 172.21.15.42:0/2512313402 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f09b00a7740 0x7f09b008df80 secure :-1 s=READY pgs=1797 cs=0 l=1 rx=0x7f09a0008b10 tx=0x7f09a000fff0).stop 2019-12-05T06:16:55.694 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:55.679+0000 7f09b7498700 1 -- 172.21.15.42:0/2512313402 shutdown_connections 2019-12-05T06:16:55.695 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:55.679+0000 7f09b7498700 1 --2- 172.21.15.42:0/2512313402 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f09b009fae0 0x7f09b007eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:16:55.696 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:55.679+0000 7f09b7498700 1 --2- 172.21.15.42:0/2512313402 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f09b00a7740 0x7f09b008df80 unknown :-1 s=CLOSED pgs=1797 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:16:55.696 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:55.683+0000 7f09b7498700 1 -- 172.21.15.42:0/2512313402 shutdown_connections 2019-12-05T06:16:55.696 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:55.683+0000 7f09b7498700 1 -- 172.21.15.42:0/2512313402 wait complete. 2019-12-05T06:16:55.696 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:55.683+0000 7f09b7498700 1 -- 172.21.15.42:0/2512313402 >> 172.21.15.42:0/2512313402 conn(0x7f09b00aabe0 msgr2=0x7f09b00a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:16:55.702 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:55.683+0000 7f09b7498700 1 Processor -- start 2019-12-05T06:16:55.702 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:55.683+0000 7f09b7498700 1 -- start start 2019-12-05T06:16:55.702 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:55.683+0000 7f09b7498700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f09b0124d00 0x7f09b0122670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:16:55.702 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:55.683+0000 7f09b7498700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09b00a5300 con 0x7f09b0133470 2019-12-05T06:16:55.702 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:55.683+0000 7f09b7498700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09b007bf70 con 0x7f09b0077310 2019-12-05T06:16:55.703 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:55.683+0000 7f09b7498700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09b0122df0 con 0x7f09b0124d00 2019-12-05T06:16:55.703 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:55.683+0000 7f09b6236700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f09b0077310 0x7f09b0079750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:52032/0 (socket says 172.21.15.42:52032) 2019-12-05T06:16:55.703 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:55.683+0000 7f09b6236700 1 -- 172.21.15.42:0/4283447177 learned_addr learned my addr 172.21.15.42:0/4283447177 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:16:55.703 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:55.683+0000 7f09b5234700 1 --2- 172.21.15.42:0/4283447177 >> v2:172.21.15.40:3300/0 conn(0x7f09b0124d00 0x7f09b0122670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:16:55.703 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:55.687+0000 7f09ae7fc700 1 -- 172.21.15.42:0/4283447177 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4228053372 0 0) 0x7f09b007bf70 con 0x7f09b0077310 2019-12-05T06:16:55.704 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:55.687+0000 7f09ae7fc700 1 -- 172.21.15.42:0/4283447177 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09900030b0 con 0x7f09b0077310 2019-12-05T06:16:55.704 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:55.687+0000 7f09ae7fc700 1 -- 172.21.15.42:0/4283447177 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4049770676 0 0) 0x7f09b00a5300 con 0x7f09b0133470 2019-12-05T06:16:55.704 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:55.687+0000 7f09ae7fc700 1 -- 172.21.15.42:0/4283447177 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09b007bf70 con 0x7f09b0133470 2019-12-05T06:16:55.704 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:55.687+0000 7f09b5234700 1 -- 172.21.15.42:0/4283447177 >> v1:172.21.15.42:6789/0 conn(0x7f09b0077310 legacy=0x7f09b0079750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:55.704 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:55.687+0000 7f09b5234700 1 -- 172.21.15.42:0/4283447177 >> v1:172.21.15.40:6789/0 conn(0x7f09b0133470 legacy=0x7f09b01239b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:55.705 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:55.687+0000 7f09b5234700 1 -- 172.21.15.42:0/4283447177 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09b00a4b90 con 0x7f09b0124d00 2019-12-05T06:16:55.705 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:55.687+0000 7f09b5234700 1 --2- 172.21.15.42:0/4283447177 >> v2:172.21.15.40:3300/0 conn(0x7f09b0124d00 0x7f09b0122670 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f09a40043a0 tx=0x7f09a4010860).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:55.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:55.687+0000 7f09b5234700 1 --2- 172.21.15.42:0/4283447177 >> v2:172.21.15.40:3300/0 conn(0x7f09b0124d00 0x7f09b0122670 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f09a40043a0 tx=0x7f09a4010860).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:55.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:55.687+0000 7f09b5234700 1 --2- 172.21.15.42:0/4283447177 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f09b0124d00 0x7f09b0122670 secure :-1 s=READY pgs=1799 cs=0 l=1 rx=0x7f09a40043a0 tx=0x7f09a4010860).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:16:55.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:55.687+0000 7f09b5234700 1 --2- 172.21.15.42:0/4283447177 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f09b0124d00 0x7f09b0122670 secure :-1 s=THROTTLE_DONE pgs=1799 cs=0 l=1 rx=0x7f09a40043a0 tx=0x7f09a4010860).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:55.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:55.687+0000 7f09ae7fc700 1 -- 172.21.15.42:0/4283447177 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f09a4005c30 con 0x7f09b0124d00 2019-12-05T06:16:55.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:55.687+0000 7f09b5234700 1 --2- 172.21.15.42:0/4283447177 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f09b0124d00 0x7f09b0122670 secure :-1 s=THROTTLE_DONE pgs=1799 cs=0 l=1 rx=0x7f09a40043a0 tx=0x7f09a4010860).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:55.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:55.687+0000 7f09b5234700 1 --2- 172.21.15.42:0/4283447177 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f09b0124d00 0x7f09b0122670 secure :-1 s=THROTTLE_DONE pgs=1799 cs=0 l=1 rx=0x7f09a40043a0 tx=0x7f09a4010860).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:55.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:55.687+0000 7f09ae7fc700 1 -- 172.21.15.42:0/4283447177 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09a4005850 con 0x7f09b0124d00 2019-12-05T06:16:55.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:55.687+0000 7f09b7498700 1 -- 172.21.15.42:0/4283447177 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09b007af60 con 0x7f09b0124d00 2019-12-05T06:16:55.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:55.687+0000 7f09ae7fc700 1 -- 172.21.15.42:0/4283447177 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f09a40064a0 con 0x7f09b0124d00 2019-12-05T06:16:55.708 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:55.687+0000 7f09b7498700 1 -- 172.21.15.42:0/4283447177 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09b007b490 con 0x7f09b0124d00 2019-12-05T06:16:55.708 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:55.687+0000 7f09b5234700 1 --2- 172.21.15.42:0/4283447177 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f09b0124d00 0x7f09b0122670 secure :-1 s=THROTTLE_DONE pgs=1799 cs=0 l=1 rx=0x7f09a40043a0 tx=0x7f09a4010860).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:55.708 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:55.691+0000 7f09ae7fc700 1 -- 172.21.15.42:0/4283447177 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f09a40066a0 con 0x7f09b0124d00 2019-12-05T06:16:55.708 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:55.691+0000 7f09b5234700 1 --2- 172.21.15.42:0/4283447177 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f09b0124d00 0x7f09b0122670 secure :-1 s=THROTTLE_DONE pgs=1799 cs=0 l=1 rx=0x7f09a40043a0 tx=0x7f09a4010860).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:55.708 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:55.691+0000 7f09ae7fc700 1 --2- 172.21.15.42:0/4283447177 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f099003b4a0 0x7f099003d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:16:55.709 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:55.691+0000 7f09ae7fc700 1 -- 172.21.15.42:0/4283447177 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 10331+0+0 (secure 0 0 0) 0x7f09a404aff0 con 0x7f09b0124d00 2019-12-05T06:16:55.709 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:55.691+0000 7f09b5a35700 1 --2- 172.21.15.42:0/4283447177 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f099003b4a0 0x7f099003d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:16:55.709 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:55.691+0000 7f09b5a35700 1 --2- 172.21.15.42:0/4283447177 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f099003b4a0 0x7f099003d9b0 crc :-1 s=READY pgs=995 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:16:55.709 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:55.695+0000 7f09b7498700 1 -- 172.21.15.42:0/4283447177 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09b005d110 con 0x7f09b0124d00 2019-12-05T06:16:55.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:55.703+0000 7f09b5234700 1 --2- 172.21.15.42:0/4283447177 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f09b0124d00 0x7f09b0122670 secure :-1 s=THROTTLE_DONE pgs=1799 cs=0 l=1 rx=0x7f09a40043a0 tx=0x7f09a4010860).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:55.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:55.703+0000 7f09ae7fc700 1 -- 172.21.15.42:0/4283447177 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f09b00a4b90 con 0x7f09b0124d00 2019-12-05T06:16:55.880 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:55.871+0000 7f09b7498700 1 -- 172.21.15.42:0/4283447177 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f09b005b8c0 con 0x7f09b0124d00 2019-12-05T06:16:55.881 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:55.875+0000 7f09b5234700 1 --2- 172.21.15.42:0/4283447177 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f09b0124d00 0x7f09b0122670 secure :-1 s=THROTTLE_DONE pgs=1799 cs=0 l=1 rx=0x7f09a40043a0 tx=0x7f09a4010860).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:55.881 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:55.875+0000 7f09ae7fc700 1 -- 172.21.15.42:0/4283447177 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 283 v283) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f09a4032070 con 0x7f09b0124d00 2019-12-05T06:16:55.883 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 283 2019-12-05T06:16:55.883 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:16:55.883 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":283,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7295,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/3228549071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3228549071},{"type":"v1","addr":"172.21.15.42:6835","nonce":3228549071}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":283}],"filesystems":[{"mdsmap":{"epoch":282,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:14:40.006630+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":99,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7217},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7217":{"gid":7217,"name":"b","rank":0,"incarnation":279,"state":"up:active","state_seq":21,"addr":"172.21.15.40:6833/2667648531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2667648531},{"type":"v1","addr":"172.21.15.40:6833","nonce":2667648531}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:16:55.885 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:55.879+0000 7f09b7498700 1 -- 172.21.15.42:0/4283447177 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f099003b4a0 msgr2=0x7f099003d9b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:55.885 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:55.879+0000 7f09b7498700 1 --2- 172.21.15.42:0/4283447177 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f099003b4a0 0x7f099003d9b0 crc :-1 s=READY pgs=995 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:16:55.886 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:55.879+0000 7f09b7498700 1 -- 172.21.15.42:0/4283447177 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f09b0124d00 msgr2=0x7f09b0122670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:55.886 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:55.879+0000 7f09b7498700 1 --2- 172.21.15.42:0/4283447177 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f09b0124d00 0x7f09b0122670 secure :-1 s=READY pgs=1799 cs=0 l=1 rx=0x7f09a40043a0 tx=0x7f09a4010860).stop 2019-12-05T06:16:55.886 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:55.879+0000 7f09b7498700 1 -- 172.21.15.42:0/4283447177 shutdown_connections 2019-12-05T06:16:55.886 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:55.879+0000 7f09b7498700 1 --2- 172.21.15.42:0/4283447177 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f099003b4a0 0x7f099003d9b0 unknown :-1 s=CLOSED pgs=995 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:16:55.888 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:55.879+0000 7f09b7498700 1 --2- 172.21.15.42:0/4283447177 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f09b0124d00 0x7f09b0122670 unknown :-1 s=CLOSED pgs=1799 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:16:55.888 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:55.879+0000 7f09b7498700 1 -- 172.21.15.42:0/4283447177 shutdown_connections 2019-12-05T06:16:55.888 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:55.879+0000 7f09b7498700 1 -- 172.21.15.42:0/4283447177 wait complete. 2019-12-05T06:16:55.888 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:55.879+0000 7f09b7498700 1 -- 172.21.15.42:0/4283447177 >> 172.21.15.42:0/4283447177 conn(0x7f09b00a7740 msgr2=0x7f09b00a9640 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:16:55.896 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:16:57.896 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:16:58.002 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:57.991+0000 7fef06d63700 1 Processor -- start 2019-12-05T06:16:58.003 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:57.991+0000 7fef06d63700 1 -- start start 2019-12-05T06:16:58.003 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:57.991+0000 7fef06d63700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fef000a7740 0x7fef0008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:16:58.003 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:57.991+0000 7fef06d63700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fef0007e720 0x7fef0007eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:16:58.003 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:57.991+0000 7fef06d63700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef000a0710 con 0x7fef0009f2e0 2019-12-05T06:16:58.003 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:57.991+0000 7fef06d63700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef000a0b00 con 0x7fef0007e720 2019-12-05T06:16:58.004 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:57.991+0000 7fef06d63700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef0008e5b0 con 0x7fef000a7740 2019-12-05T06:16:58.004 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:57.991+0000 7fef05300700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fef000a7740 0x7fef0008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:16:58.004 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:57.991+0000 7fef05300700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fef000a7740 0x7fef0008df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:52006/0 (socket says 172.21.15.42:52006) 2019-12-05T06:16:58.004 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:57.991+0000 7fef05300700 1 -- 172.21.15.42:0/3063909070 learned_addr learned my addr 172.21.15.42:0/3063909070 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:16:58.004 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:57.991+0000 7fef04aff700 1 --2- 172.21.15.42:0/3063909070 >> v2:172.21.15.40:3300/0 conn(0x7fef0007e720 0x7fef0007eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:16:58.004 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:57.991+0000 7fef05300700 1 -- 172.21.15.42:0/3063909070 >> v2:172.21.15.40:3300/0 conn(0x7fef0007e720 msgr2=0x7fef0007eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:16:58.004 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:57.991+0000 7fef05300700 1 --2- 172.21.15.42:0/3063909070 >> v2:172.21.15.40:3300/0 conn(0x7fef0007e720 0x7fef0007eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:16:58.005 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:57.995+0000 7fef05300700 1 -- 172.21.15.42:0/3063909070 >> v1:172.21.15.42:6789/0 conn(0x7fef0009f2e0 legacy=0x7fef0009f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:58.005 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:57.995+0000 7fef05300700 1 -- 172.21.15.42:0/3063909070 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef0007f0a0 con 0x7fef000a7740 2019-12-05T06:16:58.005 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:57.995+0000 7fef05300700 1 --2- 172.21.15.42:0/3063909070 >> v2:172.21.15.42:3300/0 conn(0x7fef000a7740 0x7fef0008df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7feefc002f30 tx=0x7feefc00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:58.005 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:57.995+0000 7fef05300700 1 --2- 172.21.15.42:0/3063909070 >> v2:172.21.15.42:3300/0 conn(0x7fef000a7740 0x7fef0008df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7feefc002f30 tx=0x7feefc00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:58.005 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:57.995+0000 7fef05300700 1 --2- 172.21.15.42:0/3063909070 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fef000a7740 0x7fef0008df80 secure :-1 s=READY pgs=2369 cs=0 l=1 rx=0x7feefc002f30 tx=0x7feefc00fd30).ready entity=mon.1 client_cookie=b288cab71b490238 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:16:58.005 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:57.995+0000 7fef05300700 1 --2- 172.21.15.42:0/3063909070 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fef000a7740 0x7fef0008df80 secure :-1 s=THROTTLE_DONE pgs=2369 cs=0 l=1 rx=0x7feefc002f30 tx=0x7feefc00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:58.005 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:57.995+0000 7feef7fff700 1 -- 172.21.15.42:0/3063909070 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7feefc011370 con 0x7fef000a7740 2019-12-05T06:16:58.006 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:57.995+0000 7fef05300700 1 --2- 172.21.15.42:0/3063909070 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fef000a7740 0x7fef0008df80 secure :-1 s=THROTTLE_DONE pgs=2369 cs=0 l=1 rx=0x7feefc002f30 tx=0x7feefc00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:58.006 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:57.995+0000 7feef7fff700 1 -- 172.21.15.42:0/3063909070 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feefc008bb0 con 0x7fef000a7740 2019-12-05T06:16:58.006 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:57.995+0000 7fef05300700 1 --2- 172.21.15.42:0/3063909070 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fef000a7740 0x7fef0008df80 secure :-1 s=THROTTLE_DONE pgs=2369 cs=0 l=1 rx=0x7feefc002f30 tx=0x7feefc00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:58.006 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:57.995+0000 7feef7fff700 1 -- 172.21.15.42:0/3063909070 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7feefc011950 con 0x7fef000a7740 2019-12-05T06:16:58.006 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:57.995+0000 7fef06d63700 1 -- 172.21.15.42:0/3063909070 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fef000a7740 msgr2=0x7fef0008df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:58.006 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:57.995+0000 7fef06d63700 1 --2- 172.21.15.42:0/3063909070 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fef000a7740 0x7fef0008df80 secure :-1 s=READY pgs=2369 cs=0 l=1 rx=0x7feefc002f30 tx=0x7feefc00fd30).stop 2019-12-05T06:16:58.006 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:57.995+0000 7fef06d63700 1 -- 172.21.15.42:0/3063909070 shutdown_connections 2019-12-05T06:16:58.007 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:57.995+0000 7fef06d63700 1 --2- 172.21.15.42:0/3063909070 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fef000a7740 0x7fef0008df80 unknown :-1 s=CLOSED pgs=2369 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:16:58.007 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:57.995+0000 7fef06d63700 1 --2- 172.21.15.42:0/3063909070 >> v2:172.21.15.40:3300/0 conn(0x7fef0007e720 0x7fef0007eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:16:58.007 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:57.995+0000 7fef06d63700 1 -- 172.21.15.42:0/3063909070 shutdown_connections 2019-12-05T06:16:58.007 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:57.995+0000 7fef06d63700 1 -- 172.21.15.42:0/3063909070 wait complete. 2019-12-05T06:16:58.007 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:57.995+0000 7fef06d63700 1 -- 172.21.15.42:0/3063909070 >> 172.21.15.42:0/3063909070 conn(0x7fef000aabe0 msgr2=0x7fef000a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:16:58.010 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:57.999+0000 7fef06d63700 1 Processor -- start 2019-12-05T06:16:58.011 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:57.999+0000 7fef06d63700 1 -- start start 2019-12-05T06:16:58.011 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:57.999+0000 7fef06d63700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fef00124b50 0x7fef000a9910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:16:58.011 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:57.999+0000 7fef06d63700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fef0007c410 0x7fef00078340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:16:58.011 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:57.999+0000 7fef06d63700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef000a3c60 con 0x7fef0007bfe0 2019-12-05T06:16:58.011 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:57.999+0000 7fef06d63700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef0011ade0 con 0x7fef00124b50 2019-12-05T06:16:58.011 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:57.999+0000 7fef06d63700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef0007bd40 con 0x7fef0007c410 2019-12-05T06:16:58.012 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:57.999+0000 7fef05300700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fef00124b50 0x7fef000a9910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:16:58.012 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:57.999+0000 7fef05300700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fef00124b50 0x7fef000a9910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.42:39826/0 (socket says 172.21.15.42:39826) 2019-12-05T06:16:58.012 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:57.999+0000 7fef05300700 1 -- 172.21.15.42:0/819722012 learned_addr learned my addr 172.21.15.42:0/819722012 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:16:58.012 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:57.999+0000 7fef05b01700 1 --2- 172.21.15.42:0/819722012 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fef0007c410 0x7fef00078340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:16:58.012 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:57.999+0000 7fef05300700 1 -- 172.21.15.42:0/819722012 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fef0007c410 msgr2=0x7fef00078340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:58.012 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:57.999+0000 7fef05300700 1 --2- 172.21.15.42:0/819722012 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fef0007c410 0x7fef00078340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:16:58.013 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:57.999+0000 7fef05300700 1 -- 172.21.15.42:0/819722012 >> v1:172.21.15.40:6789/0 conn(0x7fef0007bfe0 legacy=0x7fef0011a4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:58.013 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:57.999+0000 7feef5ffb700 1 -- 172.21.15.42:0/819722012 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3951787562 0 0) 0x7fef000a3c60 con 0x7fef0007bfe0 2019-12-05T06:16:58.013 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:57.999+0000 7fef05300700 1 -- 172.21.15.42:0/819722012 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef00078890 con 0x7fef00124b50 2019-12-05T06:16:58.013 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:57.999+0000 7fef05300700 1 --2- 172.21.15.42:0/819722012 >> v2:172.21.15.40:3300/0 conn(0x7fef00124b50 0x7fef000a9910 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fef0011b0b0 tx=0x7feefc008bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:58.013 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:57.999+0000 7fef05300700 1 --2- 172.21.15.42:0/819722012 >> v2:172.21.15.40:3300/0 conn(0x7fef00124b50 0x7fef000a9910 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fef0011b0b0 tx=0x7feefc008bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:58.013 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:57.999+0000 7fef05300700 1 --2- 172.21.15.42:0/819722012 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fef00124b50 0x7fef000a9910 secure :-1 s=READY pgs=1801 cs=0 l=1 rx=0x7fef0011b0b0 tx=0x7feefc008bb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:16:58.014 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:58.003+0000 7fef05300700 1 --2- 172.21.15.42:0/819722012 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fef00124b50 0x7fef000a9910 secure :-1 s=THROTTLE_DONE pgs=1801 cs=0 l=1 rx=0x7fef0011b0b0 tx=0x7feefc008bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:58.014 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:58.003+0000 7feef5ffb700 1 -- 172.21.15.42:0/819722012 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7feefc007340 con 0x7fef00124b50 2019-12-05T06:16:58.014 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:58.003+0000 7fef05300700 1 --2- 172.21.15.42:0/819722012 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fef00124b50 0x7fef000a9910 secure :-1 s=THROTTLE_DONE pgs=1801 cs=0 l=1 rx=0x7fef0011b0b0 tx=0x7feefc008bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:58.014 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:58.003+0000 7fef05300700 1 --2- 172.21.15.42:0/819722012 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fef00124b50 0x7fef000a9910 secure :-1 s=THROTTLE_DONE pgs=1801 cs=0 l=1 rx=0x7fef0011b0b0 tx=0x7feefc008bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:58.014 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:58.003+0000 7feef5ffb700 1 -- 172.21.15.42:0/819722012 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feefc011570 con 0x7fef00124b50 2019-12-05T06:16:58.014 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:58.003+0000 7feef5ffb700 1 -- 172.21.15.42:0/819722012 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7feefc005540 con 0x7fef00124b50 2019-12-05T06:16:58.014 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:58.003+0000 7fef06d63700 1 -- 172.21.15.42:0/819722012 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef00078b60 con 0x7fef00124b50 2019-12-05T06:16:58.015 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:58.003+0000 7fef06d63700 1 -- 172.21.15.42:0/819722012 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef0012f780 con 0x7fef00124b50 2019-12-05T06:16:58.015 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:58.003+0000 7fef05300700 1 --2- 172.21.15.42:0/819722012 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fef00124b50 0x7fef000a9910 secure :-1 s=THROTTLE_DONE pgs=1801 cs=0 l=1 rx=0x7fef0011b0b0 tx=0x7feefc008bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:58.015 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:58.003+0000 7feef5ffb700 1 -- 172.21.15.42:0/819722012 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7feefc005850 con 0x7fef00124b50 2019-12-05T06:16:58.015 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:58.003+0000 7fef05300700 1 --2- 172.21.15.42:0/819722012 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fef00124b50 0x7fef000a9910 secure :-1 s=THROTTLE_DONE pgs=1801 cs=0 l=1 rx=0x7fef0011b0b0 tx=0x7feefc008bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:58.015 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:58.003+0000 7feef5ffb700 1 --2- 172.21.15.42:0/819722012 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7feee003ad10 0x7feee003d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:16:58.016 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:58.003+0000 7feef5ffb700 1 -- 172.21.15.42:0/819722012 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 10331+0+0 (secure 0 0 0) 0x7feefc04b3f0 con 0x7fef00124b50 2019-12-05T06:16:58.016 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:58.003+0000 7fef04aff700 1 --2- 172.21.15.42:0/819722012 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7feee003ad10 0x7feee003d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:16:58.016 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:58.003+0000 7fef04aff700 1 --2- 172.21.15.42:0/819722012 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7feee003ad10 0x7feee003d220 crc :-1 s=READY pgs=996 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:16:58.016 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:58.007+0000 7fef06d63700 1 -- 172.21.15.42:0/819722012 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef0005d110 con 0x7fef00124b50 2019-12-05T06:16:58.022 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:58.015+0000 7fef05300700 1 --2- 172.21.15.42:0/819722012 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fef00124b50 0x7fef000a9910 secure :-1 s=THROTTLE_DONE pgs=1801 cs=0 l=1 rx=0x7fef0011b0b0 tx=0x7feefc008bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:58.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:58.015+0000 7feef5ffb700 1 -- 172.21.15.42:0/819722012 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7feefc032470 con 0x7fef00124b50 2019-12-05T06:16:58.193 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:58.187+0000 7fef06d63700 1 -- 172.21.15.42:0/819722012 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fef0005b8c0 con 0x7fef00124b50 2019-12-05T06:16:58.193 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:58.187+0000 7fef05300700 1 --2- 172.21.15.42:0/819722012 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fef00124b50 0x7fef000a9910 secure :-1 s=THROTTLE_DONE pgs=1801 cs=0 l=1 rx=0x7fef0011b0b0 tx=0x7feefc008bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:58.194 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:58.187+0000 7feef5ffb700 1 -- 172.21.15.42:0/819722012 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 283 v283) v1 ==== 95+0+2112 (secure 0 0 0) 0x7feefc0326d0 con 0x7fef00124b50 2019-12-05T06:16:58.196 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 283 2019-12-05T06:16:58.196 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:16:58.196 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":283,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7295,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/3228549071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3228549071},{"type":"v1","addr":"172.21.15.42:6835","nonce":3228549071}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":283}],"filesystems":[{"mdsmap":{"epoch":282,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:14:40.006630+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":99,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7217},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7217":{"gid":7217,"name":"b","rank":0,"incarnation":279,"state":"up:active","state_seq":21,"addr":"172.21.15.40:6833/2667648531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2667648531},{"type":"v1","addr":"172.21.15.40:6833","nonce":2667648531}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:16:58.198 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:58.191+0000 7fef06d63700 1 -- 172.21.15.42:0/819722012 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7feee003ad10 msgr2=0x7feee003d220 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:58.198 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:58.191+0000 7fef06d63700 1 --2- 172.21.15.42:0/819722012 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7feee003ad10 0x7feee003d220 crc :-1 s=READY pgs=996 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:16:58.199 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:58.191+0000 7fef06d63700 1 -- 172.21.15.42:0/819722012 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fef00124b50 msgr2=0x7fef000a9910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:58.199 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:58.191+0000 7fef06d63700 1 --2- 172.21.15.42:0/819722012 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fef00124b50 0x7fef000a9910 secure :-1 s=READY pgs=1801 cs=0 l=1 rx=0x7fef0011b0b0 tx=0x7feefc008bb0).stop 2019-12-05T06:16:58.199 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:58.191+0000 7fef06d63700 1 -- 172.21.15.42:0/819722012 shutdown_connections 2019-12-05T06:16:58.199 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:58.191+0000 7fef06d63700 1 --2- 172.21.15.42:0/819722012 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7feee003ad10 0x7feee003d220 unknown :-1 s=CLOSED pgs=996 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:16:58.199 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:58.191+0000 7fef06d63700 1 --2- 172.21.15.42:0/819722012 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fef0007c410 0x7fef00078340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:16:58.199 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:58.191+0000 7fef06d63700 1 --2- 172.21.15.42:0/819722012 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fef00124b50 0x7fef000a9910 unknown :-1 s=CLOSED pgs=1801 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:16:58.200 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:58.191+0000 7fef06d63700 1 -- 172.21.15.42:0/819722012 shutdown_connections 2019-12-05T06:16:58.200 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:58.191+0000 7fef06d63700 1 -- 172.21.15.42:0/819722012 wait complete. 2019-12-05T06:16:58.200 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:16:58.191+0000 7fef06d63700 1 -- 172.21.15.42:0/819722012 >> 172.21.15.42:0/819722012 conn(0x7fef000a82a0 msgr2=0x7fef0009f3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:16:58.208 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:17:00.209 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:17:00.309 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.291+0000 7f469f9ba700 1 Processor -- start 2019-12-05T06:17:00.309 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.291+0000 7f469f9ba700 1 -- start start 2019-12-05T06:17:00.309 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.291+0000 7f469f9ba700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f46980a7730 0x7f469808df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:17:00.309 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.291+0000 7f469f9ba700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f469807e710 0x7f469807eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:17:00.309 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.291+0000 7f469f9ba700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46980a0700 con 0x7f469809f2d0 2019-12-05T06:17:00.310 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.291+0000 7f469f9ba700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46980a0af0 con 0x7f46980a7730 2019-12-05T06:17:00.310 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.291+0000 7f469f9ba700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f469808e5a0 con 0x7f469807e710 2019-12-05T06:17:00.310 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.291+0000 7f469d756700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f469807e710 0x7f469807eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:17:00.310 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.291+0000 7f469d756700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f469807e710 0x7f469807eb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:52038/0 (socket says 172.21.15.42:52038) 2019-12-05T06:17:00.310 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.291+0000 7f469d756700 1 -- 172.21.15.42:0/3457141220 learned_addr learned my addr 172.21.15.42:0/3457141220 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:17:00.310 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.291+0000 7f469df57700 1 --2- 172.21.15.42:0/3457141220 >> v2:172.21.15.40:3300/0 conn(0x7f46980a7730 0x7f469808df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:17:00.311 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.291+0000 7f469d756700 1 -- 172.21.15.42:0/3457141220 >> v2:172.21.15.40:3300/0 conn(0x7f46980a7730 msgr2=0x7f469808df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:17:00.311 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.291+0000 7f469d756700 1 --2- 172.21.15.42:0/3457141220 >> v2:172.21.15.40:3300/0 conn(0x7f46980a7730 0x7f469808df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:17:00.311 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.291+0000 7f469d756700 1 -- 172.21.15.42:0/3457141220 >> v1:172.21.15.40:6789/0 conn(0x7f469809f2d0 legacy=0x7f469809f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:17:00.311 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.291+0000 7f469d756700 1 -- 172.21.15.42:0/3457141220 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f469807f090 con 0x7f469807e710 2019-12-05T06:17:00.312 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.291+0000 7f469d756700 1 --2- 172.21.15.42:0/3457141220 >> v2:172.21.15.42:3300/0 conn(0x7f469807e710 0x7f469807eb40 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f468c0032b0 tx=0x7f468c0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:00.312 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.291+0000 7f469d756700 1 --2- 172.21.15.42:0/3457141220 >> v2:172.21.15.42:3300/0 conn(0x7f469807e710 0x7f469807eb40 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f468c0032b0 tx=0x7f468c0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:00.312 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.295+0000 7f469d756700 1 --2- 172.21.15.42:0/3457141220 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f469807e710 0x7f469807eb40 secure :-1 s=READY pgs=2370 cs=0 l=1 rx=0x7f468c0032b0 tx=0x7f468c0100b0).ready entity=mon.1 client_cookie=b2f057d0b66c1a27 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:17:00.312 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.295+0000 7f469d756700 1 --2- 172.21.15.42:0/3457141220 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f469807e710 0x7f469807eb40 secure :-1 s=THROTTLE_DONE pgs=2370 cs=0 l=1 rx=0x7f468c0032b0 tx=0x7f468c0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:00.312 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.295+0000 7f469cf55700 1 -- 172.21.15.42:0/3457141220 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f468c011dd0 con 0x7f469807e710 2019-12-05T06:17:00.313 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.295+0000 7f469d756700 1 --2- 172.21.15.42:0/3457141220 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f469807e710 0x7f469807eb40 secure :-1 s=THROTTLE_DONE pgs=2370 cs=0 l=1 rx=0x7f468c0032b0 tx=0x7f468c0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:00.313 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.295+0000 7f469d756700 1 --2- 172.21.15.42:0/3457141220 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f469807e710 0x7f469807eb40 secure :-1 s=THROTTLE_DONE pgs=2370 cs=0 l=1 rx=0x7f468c0032b0 tx=0x7f468c0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:00.313 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.295+0000 7f469cf55700 1 -- 172.21.15.42:0/3457141220 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f468c0119f0 con 0x7f469807e710 2019-12-05T06:17:00.313 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.295+0000 7f469cf55700 1 -- 172.21.15.42:0/3457141220 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f468c00a540 con 0x7f469807e710 2019-12-05T06:17:00.313 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.295+0000 7f469f9ba700 1 -- 172.21.15.42:0/3457141220 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f469807e710 msgr2=0x7f469807eb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:17:00.313 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.295+0000 7f469f9ba700 1 --2- 172.21.15.42:0/3457141220 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f469807e710 0x7f469807eb40 secure :-1 s=READY pgs=2370 cs=0 l=1 rx=0x7f468c0032b0 tx=0x7f468c0100b0).stop 2019-12-05T06:17:00.314 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.295+0000 7f469f9ba700 1 -- 172.21.15.42:0/3457141220 shutdown_connections 2019-12-05T06:17:00.315 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.295+0000 7f469f9ba700 1 --2- 172.21.15.42:0/3457141220 >> v2:172.21.15.40:3300/0 conn(0x7f46980a7730 0x7f469808df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:17:00.315 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.295+0000 7f469f9ba700 1 --2- 172.21.15.42:0/3457141220 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f469807e710 0x7f469807eb40 unknown :-1 s=CLOSED pgs=2370 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:17:00.315 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.295+0000 7f469f9ba700 1 -- 172.21.15.42:0/3457141220 shutdown_connections 2019-12-05T06:17:00.315 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.295+0000 7f469f9ba700 1 -- 172.21.15.42:0/3457141220 wait complete. 2019-12-05T06:17:00.315 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.295+0000 7f469f9ba700 1 -- 172.21.15.42:0/3457141220 >> 172.21.15.42:0/3457141220 conn(0x7f46980aabd0 msgr2=0x7f46980a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:17:00.315 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.295+0000 7f469f9ba700 1 Processor -- start 2019-12-05T06:17:00.316 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.295+0000 7f469f9ba700 1 -- start start 2019-12-05T06:17:00.316 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.299+0000 7f469f9ba700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f469811b170 0x7f46980a9780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:17:00.316 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.299+0000 7f469f9ba700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f46980752e0 0x7f4698077720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:17:00.316 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.299+0000 7f469f9ba700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46980a9cd0 con 0x7f46981169f0 2019-12-05T06:17:00.316 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.299+0000 7f469f9ba700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4698117fe0 con 0x7f46980752e0 2019-12-05T06:17:00.317 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.299+0000 7f469f9ba700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4698118280 con 0x7f469811b170 2019-12-05T06:17:00.317 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.299+0000 7f469df57700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f469811b170 0x7f46980a9780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:17:00.317 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.299+0000 7f469df57700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f469811b170 0x7f46980a9780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:52044/0 (socket says 172.21.15.42:52044) 2019-12-05T06:17:00.318 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.299+0000 7f469df57700 1 -- 172.21.15.42:0/263266456 learned_addr learned my addr 172.21.15.42:0/263266456 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:17:00.318 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.299+0000 7f469e758700 1 --2- 172.21.15.42:0/263266456 >> v2:172.21.15.40:3300/0 conn(0x7f46980752e0 0x7f4698077720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:17:00.318 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.299+0000 7f469df57700 1 -- 172.21.15.42:0/263266456 >> v2:172.21.15.40:3300/0 conn(0x7f46980752e0 msgr2=0x7f4698077720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:17:00.318 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.299+0000 7f469df57700 1 --2- 172.21.15.42:0/263266456 >> v2:172.21.15.40:3300/0 conn(0x7f46980752e0 0x7f4698077720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:17:00.318 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.299+0000 7f469df57700 1 -- 172.21.15.42:0/263266456 >> v1:172.21.15.40:6789/0 conn(0x7f46981169f0 legacy=0x7f4698116e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:17:00.319 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.299+0000 7f469df57700 1 -- 172.21.15.42:0/263266456 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f469811b5a0 con 0x7f469811b170 2019-12-05T06:17:00.319 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.299+0000 7f469e758700 1 --2- 172.21.15.42:0/263266456 >> v2:172.21.15.40:3300/0 conn(0x7f46980752e0 0x7f4698077720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:17:00.319 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.299+0000 7f469df57700 1 --2- 172.21.15.42:0/263266456 >> v2:172.21.15.42:3300/0 conn(0x7f469811b170 0x7f46980a9780 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f46980aaf80 tx=0x7f46900109b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:00.319 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.299+0000 7f469df57700 1 --2- 172.21.15.42:0/263266456 >> v2:172.21.15.42:3300/0 conn(0x7f469811b170 0x7f46980a9780 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f46980aaf80 tx=0x7f46900109b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:00.319 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.299+0000 7f469df57700 1 --2- 172.21.15.42:0/263266456 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f469811b170 0x7f46980a9780 secure :-1 s=READY pgs=2371 cs=0 l=1 rx=0x7f46980aaf80 tx=0x7f46900109b0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:17:00.319 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.299+0000 7f469df57700 1 --2- 172.21.15.42:0/263266456 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f469811b170 0x7f46980a9780 secure :-1 s=THROTTLE_DONE pgs=2371 cs=0 l=1 rx=0x7f46980aaf80 tx=0x7f46900109b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:00.319 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.299+0000 7f469df57700 1 --2- 172.21.15.42:0/263266456 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f469811b170 0x7f46980a9780 secure :-1 s=THROTTLE_DONE pgs=2371 cs=0 l=1 rx=0x7f46980aaf80 tx=0x7f46900109b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:00.320 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.299+0000 7f46967fc700 1 -- 172.21.15.42:0/263266456 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4690009bb0 con 0x7f469811b170 2019-12-05T06:17:00.320 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.299+0000 7f469df57700 1 --2- 172.21.15.42:0/263266456 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f469811b170 0x7f46980a9780 secure :-1 s=THROTTLE_DONE pgs=2371 cs=0 l=1 rx=0x7f46980aaf80 tx=0x7f46900109b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:00.320 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.299+0000 7f46967fc700 1 -- 172.21.15.42:0/263266456 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4690007d30 con 0x7f469811b170 2019-12-05T06:17:00.320 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.299+0000 7f469f9ba700 1 -- 172.21.15.42:0/263266456 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4698141f50 con 0x7f469811b170 2019-12-05T06:17:00.320 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.299+0000 7f46967fc700 1 -- 172.21.15.42:0/263266456 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4690009390 con 0x7f469811b170 2019-12-05T06:17:00.320 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.299+0000 7f469f9ba700 1 -- 172.21.15.42:0/263266456 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4698117ce0 con 0x7f469811b170 2019-12-05T06:17:00.320 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.299+0000 7f469df57700 1 --2- 172.21.15.42:0/263266456 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f469811b170 0x7f46980a9780 secure :-1 s=THROTTLE_DONE pgs=2371 cs=0 l=1 rx=0x7f46980aaf80 tx=0x7f46900109b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:00.321 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.299+0000 7f46967fc700 1 -- 172.21.15.42:0/263266456 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f4690007170 con 0x7f469811b170 2019-12-05T06:17:00.321 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.299+0000 7f469df57700 1 --2- 172.21.15.42:0/263266456 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f469811b170 0x7f46980a9780 secure :-1 s=THROTTLE_DONE pgs=2371 cs=0 l=1 rx=0x7f46980aaf80 tx=0x7f46900109b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:00.321 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.299+0000 7f46967fc700 1 --2- 172.21.15.42:0/263266456 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f467803ae10 0x7f467803d320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:17:00.321 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.299+0000 7f46967fc700 1 -- 172.21.15.42:0/263266456 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 10417+0+0 (secure 0 0 0) 0x7f469004abb0 con 0x7f469811b170 2019-12-05T06:17:00.321 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.299+0000 7f469d756700 1 --2- 172.21.15.42:0/263266456 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f467803ae10 0x7f467803d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:17:00.322 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.299+0000 7f469d756700 1 --2- 172.21.15.42:0/263266456 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f467803ae10 0x7f467803d320 crc :-1 s=READY pgs=997 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:17:00.322 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.307+0000 7f469f9ba700 1 -- 172.21.15.42:0/263266456 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f469805d100 con 0x7f469811b170 2019-12-05T06:17:00.323 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.315+0000 7f469df57700 1 --2- 172.21.15.42:0/263266456 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f469811b170 0x7f46980a9780 secure :-1 s=THROTTLE_DONE pgs=2371 cs=0 l=1 rx=0x7f46980aaf80 tx=0x7f46900109b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:00.323 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.315+0000 7f46967fc700 1 -- 172.21.15.42:0/263266456 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f46900096a0 con 0x7f469811b170 2019-12-05T06:17:00.483 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.475+0000 7f469f9ba700 1 -- 172.21.15.42:0/263266456 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f469805b8b0 con 0x7f469811b170 2019-12-05T06:17:00.484 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.475+0000 7f469df57700 1 --2- 172.21.15.42:0/263266456 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f469811b170 0x7f46980a9780 secure :-1 s=THROTTLE_DONE pgs=2371 cs=0 l=1 rx=0x7f46980aaf80 tx=0x7f46900109b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:00.484 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.475+0000 7f46967fc700 1 -- 172.21.15.42:0/263266456 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 284 v284) v1 ==== 95+0+1775 (secure 0 0 0) 0x7f4690031e90 con 0x7f469811b170 2019-12-05T06:17:00.484 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 284 2019-12-05T06:17:00.484 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:17:00.484 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":284,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":284,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:16:59.699379+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":100,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7295},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7295":{"gid":7295,"name":"a","rank":0,"incarnation":284,"state":"up:replay","state_seq":2,"addr":"172.21.15.42:6835/3228549071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3228549071},{"type":"v1","addr":"172.21.15.42:6835","nonce":3228549071}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:17:00.487 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.479+0000 7f469f9ba700 1 -- 172.21.15.42:0/263266456 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f467803ae10 msgr2=0x7f467803d320 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:17:00.487 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.479+0000 7f469f9ba700 1 --2- 172.21.15.42:0/263266456 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f467803ae10 0x7f467803d320 crc :-1 s=READY pgs=997 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:17:00.487 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.479+0000 7f469f9ba700 1 -- 172.21.15.42:0/263266456 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f469811b170 msgr2=0x7f46980a9780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:17:00.487 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.479+0000 7f469f9ba700 1 --2- 172.21.15.42:0/263266456 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f469811b170 0x7f46980a9780 secure :-1 s=READY pgs=2371 cs=0 l=1 rx=0x7f46980aaf80 tx=0x7f46900109b0).stop 2019-12-05T06:17:00.487 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.479+0000 7f469f9ba700 1 -- 172.21.15.42:0/263266456 shutdown_connections 2019-12-05T06:17:00.488 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.479+0000 7f469f9ba700 1 --2- 172.21.15.42:0/263266456 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f467803ae10 0x7f467803d320 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:17:00.488 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.479+0000 7f469f9ba700 1 --2- 172.21.15.42:0/263266456 >> v2:172.21.15.40:3300/0 conn(0x7f46980752e0 0x7f4698077720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:17:00.488 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.479+0000 7f469f9ba700 1 --2- 172.21.15.42:0/263266456 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f469811b170 0x7f46980a9780 unknown :-1 s=CLOSED pgs=2371 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:17:00.488 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.479+0000 7f469f9ba700 1 -- 172.21.15.42:0/263266456 shutdown_connections 2019-12-05T06:17:00.488 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.479+0000 7f469f9ba700 1 -- 172.21.15.42:0/263266456 wait complete. 2019-12-05T06:17:00.488 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.479+0000 7f469f9ba700 1 -- 172.21.15.42:0/263266456 >> 172.21.15.42:0/263266456 conn(0x7f46980a8290 msgr2=0x7f469811d1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:17:00.501 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b down, removed from mdsmap 2019-12-05T06:17:00.501 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2019-12-05T06:17:00.502 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:17:00.606 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.591+0000 7fe82be06700 1 Processor -- start 2019-12-05T06:17:00.607 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.591+0000 7fe82be06700 1 -- start start 2019-12-05T06:17:00.608 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.595+0000 7fe82be06700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe8240a7730 0x7fe82408df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:17:00.608 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.595+0000 7fe82be06700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe82409fad0 0x7fe82407eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:17:00.608 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.595+0000 7fe82be06700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8240a0700 con 0x7fe82407e710 2019-12-05T06:17:00.608 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.595+0000 7fe82be06700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8240a0af0 con 0x7fe82409fad0 2019-12-05T06:17:00.608 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.595+0000 7fe82be06700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe82408e5a0 con 0x7fe8240a7730 2019-12-05T06:17:00.608 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.595+0000 7fe82a3a3700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe8240a7730 0x7fe82408df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:17:00.609 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.595+0000 7fe82a3a3700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe8240a7730 0x7fe82408df70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:52056/0 (socket says 172.21.15.42:52056) 2019-12-05T06:17:00.609 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.595+0000 7fe82a3a3700 1 -- 172.21.15.42:0/1990415606 learned_addr learned my addr 172.21.15.42:0/1990415606 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:17:00.609 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.595+0000 7fe82aba4700 1 --2- 172.21.15.42:0/1990415606 >> v2:172.21.15.40:3300/0 conn(0x7fe82409fad0 0x7fe82407eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:17:00.609 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.595+0000 7fe82a3a3700 1 -- 172.21.15.42:0/1990415606 >> v2:172.21.15.40:3300/0 conn(0x7fe82409fad0 msgr2=0x7fe82407eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:17:00.609 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.595+0000 7fe82a3a3700 1 --2- 172.21.15.42:0/1990415606 >> v2:172.21.15.40:3300/0 conn(0x7fe82409fad0 0x7fe82407eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:17:00.610 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.595+0000 7fe82a3a3700 1 -- 172.21.15.42:0/1990415606 >> v1:172.21.15.40:6789/0 conn(0x7fe82407e710 legacy=0x7fe82409f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:17:00.610 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.595+0000 7fe82a3a3700 1 -- 172.21.15.42:0/1990415606 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe82407f090 con 0x7fe8240a7730 2019-12-05T06:17:00.610 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.595+0000 7fe82a3a3700 1 --2- 172.21.15.42:0/1990415606 >> v2:172.21.15.42:3300/0 conn(0x7fe8240a7730 0x7fe82408df70 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fe820002f30 tx=0x7fe82000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:00.610 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.595+0000 7fe82a3a3700 1 --2- 172.21.15.42:0/1990415606 >> v2:172.21.15.42:3300/0 conn(0x7fe8240a7730 0x7fe82408df70 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fe820002f30 tx=0x7fe82000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:00.610 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.595+0000 7fe82a3a3700 1 --2- 172.21.15.42:0/1990415606 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe8240a7730 0x7fe82408df70 secure :-1 s=READY pgs=2373 cs=0 l=1 rx=0x7fe820002f30 tx=0x7fe82000fd30).ready entity=mon.1 client_cookie=8271dac4f7de5d4b server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:17:00.610 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.595+0000 7fe82a3a3700 1 --2- 172.21.15.42:0/1990415606 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe8240a7730 0x7fe82408df70 secure :-1 s=THROTTLE_DONE pgs=2373 cs=0 l=1 rx=0x7fe820002f30 tx=0x7fe82000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:00.611 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.595+0000 7fe8293a1700 1 -- 172.21.15.42:0/1990415606 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe820008c90 con 0x7fe8240a7730 2019-12-05T06:17:00.611 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.595+0000 7fe82a3a3700 1 --2- 172.21.15.42:0/1990415606 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe8240a7730 0x7fe82408df70 secure :-1 s=THROTTLE_DONE pgs=2373 cs=0 l=1 rx=0x7fe820002f30 tx=0x7fe82000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:00.611 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.595+0000 7fe8293a1700 1 -- 172.21.15.42:0/1990415606 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8200088b0 con 0x7fe8240a7730 2019-12-05T06:17:00.611 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.595+0000 7fe82a3a3700 1 --2- 172.21.15.42:0/1990415606 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe8240a7730 0x7fe82408df70 secure :-1 s=THROTTLE_DONE pgs=2373 cs=0 l=1 rx=0x7fe820002f30 tx=0x7fe82000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:00.611 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.595+0000 7fe8293a1700 1 -- 172.21.15.42:0/1990415606 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe820011ad0 con 0x7fe8240a7730 2019-12-05T06:17:00.611 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.595+0000 7fe82be06700 1 -- 172.21.15.42:0/1990415606 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe8240a7730 msgr2=0x7fe82408df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:17:00.611 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.595+0000 7fe82be06700 1 --2- 172.21.15.42:0/1990415606 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe8240a7730 0x7fe82408df70 secure :-1 s=READY pgs=2373 cs=0 l=1 rx=0x7fe820002f30 tx=0x7fe82000fd30).stop 2019-12-05T06:17:00.612 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.595+0000 7fe82be06700 1 -- 172.21.15.42:0/1990415606 shutdown_connections 2019-12-05T06:17:00.612 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.595+0000 7fe82be06700 1 --2- 172.21.15.42:0/1990415606 >> v2:172.21.15.40:3300/0 conn(0x7fe82409fad0 0x7fe82407eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:17:00.612 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.595+0000 7fe82be06700 1 --2- 172.21.15.42:0/1990415606 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe8240a7730 0x7fe82408df70 unknown :-1 s=CLOSED pgs=2373 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:17:00.612 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.595+0000 7fe82be06700 1 -- 172.21.15.42:0/1990415606 shutdown_connections 2019-12-05T06:17:00.612 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.595+0000 7fe82be06700 1 -- 172.21.15.42:0/1990415606 wait complete. 2019-12-05T06:17:00.613 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.595+0000 7fe82be06700 1 -- 172.21.15.42:0/1990415606 >> 172.21.15.42:0/1990415606 conn(0x7fe8240aabd0 msgr2=0x7fe8240a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:17:00.613 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.595+0000 7fe82be06700 1 Processor -- start 2019-12-05T06:17:00.613 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.595+0000 7fe82be06700 1 -- start start 2019-12-05T06:17:00.613 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.599+0000 7fe82be06700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe8241156b0 0x7fe82411c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:17:00.613 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.599+0000 7fe82be06700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe824118aa0 0x7fe824118ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:17:00.613 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.599+0000 7fe82be06700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8241283a0 con 0x7fe824127f70 2019-12-05T06:17:00.613 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.599+0000 7fe82be06700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe82411bf00 con 0x7fe8241156b0 2019-12-05T06:17:00.614 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.599+0000 7fe82be06700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe82411c1a0 con 0x7fe824118aa0 2019-12-05T06:17:00.614 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.599+0000 7fe82a3a3700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe8241156b0 0x7fe82411c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:17:00.614 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.599+0000 7fe82a3a3700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe8241156b0 0x7fe82411c6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:52060/0 (socket says 172.21.15.42:52060) 2019-12-05T06:17:00.614 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.599+0000 7fe82a3a3700 1 -- 172.21.15.42:0/2162164592 learned_addr learned my addr 172.21.15.42:0/2162164592 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:17:00.614 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.599+0000 7fe82aba4700 1 --2- 172.21.15.42:0/2162164592 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe824118aa0 0x7fe824118ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:17:00.614 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.599+0000 7fe82a3a3700 1 -- 172.21.15.42:0/2162164592 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe824118aa0 msgr2=0x7fe824118ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:17:00.614 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.599+0000 7fe82a3a3700 1 --2- 172.21.15.42:0/2162164592 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe824118aa0 0x7fe824118ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:17:00.615 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.599+0000 7fe82a3a3700 1 -- 172.21.15.42:0/2162164592 >> v1:172.21.15.42:6789/0 conn(0x7fe824127f70 legacy=0x7fe82408f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:17:00.615 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.599+0000 7fe82a3a3700 1 -- 172.21.15.42:0/2162164592 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8241303d0 con 0x7fe8241156b0 2019-12-05T06:17:00.615 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.599+0000 7fe82a3a3700 1 --2- 172.21.15.42:0/2162164592 >> v2:172.21.15.42:3300/0 conn(0x7fe8241156b0 0x7fe82411c6e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fe82411c0c0 tx=0x7fe820008ba0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:00.615 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.599+0000 7fe82a3a3700 1 --2- 172.21.15.42:0/2162164592 >> v2:172.21.15.42:3300/0 conn(0x7fe8241156b0 0x7fe82411c6e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fe82411c0c0 tx=0x7fe820008ba0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:00.615 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.599+0000 7fe82a3a3700 1 --2- 172.21.15.42:0/2162164592 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe8241156b0 0x7fe82411c6e0 secure :-1 s=READY pgs=2375 cs=0 l=1 rx=0x7fe82411c0c0 tx=0x7fe820008ba0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:17:00.615 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.599+0000 7fe82a3a3700 1 --2- 172.21.15.42:0/2162164592 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe8241156b0 0x7fe82411c6e0 secure :-1 s=THROTTLE_DONE pgs=2375 cs=0 l=1 rx=0x7fe82411c0c0 tx=0x7fe820008ba0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:00.616 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.599+0000 7fe81affd700 1 -- 172.21.15.42:0/2162164592 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe82000f680 con 0x7fe8241156b0 2019-12-05T06:17:00.616 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.599+0000 7fe82a3a3700 1 --2- 172.21.15.42:0/2162164592 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe8241156b0 0x7fe82411c6e0 secure :-1 s=THROTTLE_DONE pgs=2375 cs=0 l=1 rx=0x7fe82411c0c0 tx=0x7fe820008ba0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:00.616 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.599+0000 7fe81affd700 1 -- 172.21.15.42:0/2162164592 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8200116f0 con 0x7fe8241156b0 2019-12-05T06:17:00.616 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.599+0000 7fe82a3a3700 1 --2- 172.21.15.42:0/2162164592 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe8241156b0 0x7fe82411c6e0 secure :-1 s=THROTTLE_DONE pgs=2375 cs=0 l=1 rx=0x7fe82411c0c0 tx=0x7fe820008ba0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:00.616 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.599+0000 7fe82be06700 1 -- 172.21.15.42:0/2162164592 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe82412f3c0 con 0x7fe8241156b0 2019-12-05T06:17:00.617 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.599+0000 7fe81affd700 1 -- 172.21.15.42:0/2162164592 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe820005920 con 0x7fe8241156b0 2019-12-05T06:17:00.617 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.599+0000 7fe82be06700 1 -- 172.21.15.42:0/2162164592 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe82412f7a0 con 0x7fe8241156b0 2019-12-05T06:17:00.617 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.599+0000 7fe82a3a3700 1 --2- 172.21.15.42:0/2162164592 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe8241156b0 0x7fe82411c6e0 secure :-1 s=THROTTLE_DONE pgs=2375 cs=0 l=1 rx=0x7fe82411c0c0 tx=0x7fe820008ba0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:00.617 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.599+0000 7fe81affd700 1 -- 172.21.15.42:0/2162164592 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fe820005c40 con 0x7fe8241156b0 2019-12-05T06:17:00.617 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.599+0000 7fe82a3a3700 1 --2- 172.21.15.42:0/2162164592 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe8241156b0 0x7fe82411c6e0 secure :-1 s=THROTTLE_DONE pgs=2375 cs=0 l=1 rx=0x7fe82411c0c0 tx=0x7fe820008ba0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:00.622 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.599+0000 7fe81affd700 1 --2- 172.21.15.42:0/2162164592 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe80c03ad80 0x7fe80c03d290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:17:00.622 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.599+0000 7fe81affd700 1 -- 172.21.15.42:0/2162164592 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 10417+0+0 (secure 0 0 0) 0x7fe82004bbd0 con 0x7fe8241156b0 2019-12-05T06:17:00.622 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.599+0000 7fe829ba2700 1 --2- 172.21.15.42:0/2162164592 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe80c03ad80 0x7fe80c03d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:17:00.623 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.599+0000 7fe829ba2700 1 --2- 172.21.15.42:0/2162164592 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe80c03ad80 0x7fe80c03d290 crc :-1 s=READY pgs=998 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:17:00.623 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.607+0000 7fe82be06700 1 -- 172.21.15.42:0/2162164592 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe82405d100 con 0x7fe8241156b0 2019-12-05T06:17:00.624 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.611+0000 7fe82a3a3700 1 --2- 172.21.15.42:0/2162164592 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe8241156b0 0x7fe82411c6e0 secure :-1 s=THROTTLE_DONE pgs=2375 cs=0 l=1 rx=0x7fe82411c0c0 tx=0x7fe820008ba0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:00.624 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.611+0000 7fe81affd700 1 -- 172.21.15.42:0/2162164592 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fe820032c30 con 0x7fe8241156b0 2019-12-05T06:17:00.783 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.775+0000 7fe82be06700 1 -- 172.21.15.42:0/2162164592 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe82405b8b0 con 0x7fe8241156b0 2019-12-05T06:17:00.783 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.775+0000 7fe82a3a3700 1 --2- 172.21.15.42:0/2162164592 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe8241156b0 0x7fe82411c6e0 secure :-1 s=THROTTLE_DONE pgs=2375 cs=0 l=1 rx=0x7fe82411c0c0 tx=0x7fe820008ba0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:00.783 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.775+0000 7fe81affd700 1 -- 172.21.15.42:0/2162164592 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 284 v284) v1 ==== 95+0+1775 (secure 0 0 0) 0x7fe820034de0 con 0x7fe8241156b0 2019-12-05T06:17:00.784 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:17:00.784 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":284,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":284,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:16:59.699379+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":100,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7295},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7295":{"gid":7295,"name":"a","rank":0,"incarnation":284,"state":"up:replay","state_seq":2,"addr":"172.21.15.42:6835/3228549071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3228549071},{"type":"v1","addr":"172.21.15.42:6835","nonce":3228549071}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:17:00.784 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 284 2019-12-05T06:17:00.786 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.779+0000 7fe82be06700 1 -- 172.21.15.42:0/2162164592 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe80c03ad80 msgr2=0x7fe80c03d290 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:17:00.786 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.779+0000 7fe82be06700 1 --2- 172.21.15.42:0/2162164592 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe80c03ad80 0x7fe80c03d290 crc :-1 s=READY pgs=998 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:17:00.786 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.779+0000 7fe82be06700 1 -- 172.21.15.42:0/2162164592 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe8241156b0 msgr2=0x7fe82411c6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:17:00.787 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.779+0000 7fe82be06700 1 --2- 172.21.15.42:0/2162164592 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe8241156b0 0x7fe82411c6e0 secure :-1 s=READY pgs=2375 cs=0 l=1 rx=0x7fe82411c0c0 tx=0x7fe820008ba0).stop 2019-12-05T06:17:00.787 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.779+0000 7fe82be06700 1 -- 172.21.15.42:0/2162164592 shutdown_connections 2019-12-05T06:17:00.787 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.779+0000 7fe82be06700 1 --2- 172.21.15.42:0/2162164592 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe80c03ad80 0x7fe80c03d290 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:17:00.787 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.779+0000 7fe82be06700 1 --2- 172.21.15.42:0/2162164592 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe824118aa0 0x7fe824118ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:17:00.787 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.779+0000 7fe82be06700 1 --2- 172.21.15.42:0/2162164592 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe8241156b0 0x7fe82411c6e0 unknown :-1 s=CLOSED pgs=2375 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:17:00.787 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.779+0000 7fe82be06700 1 -- 172.21.15.42:0/2162164592 shutdown_connections 2019-12-05T06:17:00.787 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.779+0000 7fe82be06700 1 -- 172.21.15.42:0/2162164592 wait complete. 2019-12-05T06:17:00.788 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:00.779+0000 7fe82be06700 1 -- 172.21.15.42:0/2162164592 >> 172.21.15.42:0/2162164592 conn(0x7fe8240a9c90 msgr2=0x7fe82409f4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:17:00.795 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 284, "feature_flags": { "enable_multiple": false, "ever_enabled_multiple": false }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2019-12-05T05:03:35.604738+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 284, "ever_allowed_features": 2, "explicitly_allowed_features": 2, "failed": [], "flags": 18, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_7295": { "addr": "172.21.15.42:6835/3228549071", "addrs": { "addrvec": [ { "addr": "172.21.15.42:6834", "nonce": 3228549071, "type": "v2" }, { "addr": "172.21.15.42:6835", "nonce": 3228549071, "type": "v1" } ] }, "export_targets": [], "features": 4540138292836696063, "flags": 0, "gid": 7295, "incarnation": 284, "name": "a", "rank": 0, "state": "up:replay", "state_seq": 2 } }, "last_failure": 0, "last_failure_osd_epoch": 100, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 2, "min_compat_client": "0 (unknown)", "modified": "2019-12-05T06:16:59.699379+0000", "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7295 } } } ], "standbys": [] } 2019-12-05T06:17:02.797 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:17:02.898 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:02.887+0000 7f233b655700 1 Processor -- start 2019-12-05T06:17:02.898 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:02.887+0000 7f233b655700 1 -- start start 2019-12-05T06:17:02.898 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:02.887+0000 7f233b655700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f233408df80 0x7f233409f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:17:02.899 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:02.887+0000 7f233b655700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f233409fc60 con 0x7f233409f830 2019-12-05T06:17:02.899 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:02.887+0000 7f233b655700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23340a0710 con 0x7f23340a7740 2019-12-05T06:17:02.899 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:02.887+0000 7f233b655700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23340a0b00 con 0x7f233408df80 2019-12-05T06:17:02.899 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:02.887+0000 7f23393f1700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f233408df80 0x7f233409f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:17:02.899 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:02.887+0000 7f2339bf2700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f23340a7740 0x7f233407e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:52112/0 (socket says 172.21.15.42:52112) 2019-12-05T06:17:02.899 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:02.887+0000 7f2339bf2700 1 -- 172.21.15.42:0/2740754859 learned_addr learned my addr 172.21.15.42:0/2740754859 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:17:02.900 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:02.887+0000 7f23393f1700 1 -- 172.21.15.42:0/2740754859 >> v1:172.21.15.42:6789/0 conn(0x7f23340a7740 legacy=0x7f233407e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:17:02.900 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:02.887+0000 7f23393f1700 1 -- 172.21.15.42:0/2740754859 >> v1:172.21.15.40:6789/0 conn(0x7f233409f830 legacy=0x7f233414efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:17:02.900 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:02.887+0000 7f23393f1700 1 -- 172.21.15.42:0/2740754859 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f233407ef20 con 0x7f233408df80 2019-12-05T06:17:02.900 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:02.891+0000 7f23393f1700 1 --2- 172.21.15.42:0/2740754859 >> v2:172.21.15.42:3300/0 conn(0x7f233408df80 0x7f233409f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f2328008b10 tx=0x7f232800ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:02.900 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:02.891+0000 7f23393f1700 1 --2- 172.21.15.42:0/2740754859 >> v2:172.21.15.42:3300/0 conn(0x7f233408df80 0x7f233409f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f2328008b10 tx=0x7f232800ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:02.900 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:02.891+0000 7f23393f1700 1 --2- 172.21.15.42:0/2740754859 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f233408df80 0x7f233409f2e0 secure :-1 s=READY pgs=2377 cs=0 l=1 rx=0x7f2328008b10 tx=0x7f232800ff70).ready entity=mon.1 client_cookie=6acc965d1016c3ad server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:17:02.901 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:02.891+0000 7f23393f1700 1 --2- 172.21.15.42:0/2740754859 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f233408df80 0x7f233409f2e0 secure :-1 s=THROTTLE_DONE pgs=2377 cs=0 l=1 rx=0x7f2328008b10 tx=0x7f232800ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:02.901 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:02.891+0000 7f2338bf0700 1 -- 172.21.15.42:0/2740754859 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f23280119d0 con 0x7f233408df80 2019-12-05T06:17:02.902 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:02.891+0000 7f23393f1700 1 --2- 172.21.15.42:0/2740754859 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f233408df80 0x7f233409f2e0 secure :-1 s=THROTTLE_DONE pgs=2377 cs=0 l=1 rx=0x7f2328008b10 tx=0x7f232800ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:02.902 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:02.891+0000 7f2338bf0700 1 -- 172.21.15.42:0/2740754859 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23280115f0 con 0x7f233408df80 2019-12-05T06:17:02.902 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:02.891+0000 7f23393f1700 1 --2- 172.21.15.42:0/2740754859 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f233408df80 0x7f233409f2e0 secure :-1 s=THROTTLE_DONE pgs=2377 cs=0 l=1 rx=0x7f2328008b10 tx=0x7f232800ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:02.902 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:02.891+0000 7f2338bf0700 1 -- 172.21.15.42:0/2740754859 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f232800a160 con 0x7f233408df80 2019-12-05T06:17:02.902 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:02.891+0000 7f233b655700 1 -- 172.21.15.42:0/2740754859 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f233408df80 msgr2=0x7f233409f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:17:02.902 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:02.891+0000 7f233b655700 1 --2- 172.21.15.42:0/2740754859 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f233408df80 0x7f233409f2e0 secure :-1 s=READY pgs=2377 cs=0 l=1 rx=0x7f2328008b10 tx=0x7f232800ff70).stop 2019-12-05T06:17:02.903 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:02.891+0000 7f233b655700 1 -- 172.21.15.42:0/2740754859 shutdown_connections 2019-12-05T06:17:02.903 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:02.891+0000 7f233b655700 1 --2- 172.21.15.42:0/2740754859 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f233408df80 0x7f233409f2e0 unknown :-1 s=CLOSED pgs=2377 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:17:02.903 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:02.891+0000 7f233b655700 1 -- 172.21.15.42:0/2740754859 shutdown_connections 2019-12-05T06:17:02.904 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:02.891+0000 7f233b655700 1 -- 172.21.15.42:0/2740754859 wait complete. 2019-12-05T06:17:02.904 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:02.891+0000 7f233b655700 1 -- 172.21.15.42:0/2740754859 >> 172.21.15.42:0/2740754859 conn(0x7f23340aabe0 msgr2=0x7f23340a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:17:02.908 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:02.895+0000 7f233b655700 1 Processor -- start 2019-12-05T06:17:02.908 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:02.895+0000 7f233b655700 1 -- start start 2019-12-05T06:17:02.908 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:02.895+0000 7f233b655700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2334116970 0x7f2334118940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:17:02.908 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:02.895+0000 7f233b655700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2334114490 0x7f2334077d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:17:02.908 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:02.895+0000 7f233b655700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23340a6850 con 0x7f23340782d0 2019-12-05T06:17:02.909 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:02.895+0000 7f233b655700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2334116740 con 0x7f2334116970 2019-12-05T06:17:02.909 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:02.895+0000 7f233b655700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2334116e40 con 0x7f2334114490 2019-12-05T06:17:02.909 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:02.895+0000 7f23393f1700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2334114490 0x7f2334077d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:17:02.909 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:02.895+0000 7f23393f1700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2334114490 0x7f2334077d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:52082/0 (socket says 172.21.15.42:52082) 2019-12-05T06:17:02.909 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:02.895+0000 7f23393f1700 1 -- 172.21.15.42:0/1984358041 learned_addr learned my addr 172.21.15.42:0/1984358041 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:17:02.909 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:02.895+0000 7f233a3f3700 1 --1- 172.21.15.42:0/1984358041 >> v1:172.21.15.42:6789/0 conn(0x7f23340782d0 0x7f23340a6050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:52122/0 (socket says 172.21.15.42:52122) 2019-12-05T06:17:02.910 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:02.895+0000 7f2339bf2700 1 --2- 172.21.15.42:0/1984358041 >> v2:172.21.15.40:3300/0 conn(0x7f2334116970 0x7f2334118940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:17:02.910 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:02.895+0000 7f23393f1700 1 -- 172.21.15.42:0/1984358041 >> v2:172.21.15.40:3300/0 conn(0x7f2334116970 msgr2=0x7f2334118940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:17:02.910 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:02.895+0000 7f23267fc700 1 -- 172.21.15.42:0/1984358041 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1942287823 0 0) 0x7f23340a6850 con 0x7f23340782d0 2019-12-05T06:17:02.910 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:02.895+0000 7f23393f1700 1 --2- 172.21.15.42:0/1984358041 >> v2:172.21.15.40:3300/0 conn(0x7f2334116970 0x7f2334118940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:17:02.910 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:02.895+0000 7f23393f1700 1 -- 172.21.15.42:0/1984358041 >> v1:172.21.15.42:6789/0 conn(0x7f23340782d0 legacy=0x7f23340a6050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:17:02.910 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:02.895+0000 7f23393f1700 1 -- 172.21.15.42:0/1984358041 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23340a6a80 con 0x7f2334114490 2019-12-05T06:17:02.910 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:02.895+0000 7f23393f1700 1 --2- 172.21.15.42:0/1984358041 >> v2:172.21.15.42:3300/0 conn(0x7f2334114490 0x7f2334077d80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f2328008b10 tx=0x7f232800a2e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:02.911 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:02.895+0000 7f23393f1700 1 --2- 172.21.15.42:0/1984358041 >> v2:172.21.15.42:3300/0 conn(0x7f2334114490 0x7f2334077d80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f2328008b10 tx=0x7f232800a2e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:02.911 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:02.895+0000 7f23393f1700 1 --2- 172.21.15.42:0/1984358041 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2334114490 0x7f2334077d80 secure :-1 s=READY pgs=2379 cs=0 l=1 rx=0x7f2328008b10 tx=0x7f232800a2e0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:17:02.911 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:02.895+0000 7f23393f1700 1 --2- 172.21.15.42:0/1984358041 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2334114490 0x7f2334077d80 secure :-1 s=THROTTLE_DONE pgs=2379 cs=0 l=1 rx=0x7f2328008b10 tx=0x7f232800a2e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:02.911 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:02.895+0000 7f23267fc700 1 -- 172.21.15.42:0/1984358041 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f232800f800 con 0x7f2334114490 2019-12-05T06:17:02.912 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:02.895+0000 7f23393f1700 1 --2- 172.21.15.42:0/1984358041 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2334114490 0x7f2334077d80 secure :-1 s=THROTTLE_DONE pgs=2379 cs=0 l=1 rx=0x7f2328008b10 tx=0x7f232800a2e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:02.912 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:02.895+0000 7f23393f1700 1 --2- 172.21.15.42:0/1984358041 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2334114490 0x7f2334077d80 secure :-1 s=THROTTLE_DONE pgs=2379 cs=0 l=1 rx=0x7f2328008b10 tx=0x7f232800a2e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:02.912 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:02.895+0000 7f23267fc700 1 -- 172.21.15.42:0/1984358041 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2328011bd0 con 0x7f2334114490 2019-12-05T06:17:02.912 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:02.895+0000 7f23267fc700 1 -- 172.21.15.42:0/1984358041 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2328005ca0 con 0x7f2334114490 2019-12-05T06:17:02.912 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:02.895+0000 7f233b655700 1 -- 172.21.15.42:0/1984358041 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f233412cd20 con 0x7f2334114490 2019-12-05T06:17:02.912 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:02.895+0000 7f233b655700 1 -- 172.21.15.42:0/1984358041 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f233412d2d0 con 0x7f2334114490 2019-12-05T06:17:02.912 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:02.899+0000 7f23393f1700 1 --2- 172.21.15.42:0/1984358041 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2334114490 0x7f2334077d80 secure :-1 s=THROTTLE_DONE pgs=2379 cs=0 l=1 rx=0x7f2328008b10 tx=0x7f232800a2e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:02.913 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:02.899+0000 7f23267fc700 1 -- 172.21.15.42:0/1984358041 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f2328007150 con 0x7f2334114490 2019-12-05T06:17:02.913 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:02.899+0000 7f23393f1700 1 --2- 172.21.15.42:0/1984358041 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2334114490 0x7f2334077d80 secure :-1 s=THROTTLE_DONE pgs=2379 cs=0 l=1 rx=0x7f2328008b10 tx=0x7f232800a2e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:02.913 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:02.899+0000 7f23267fc700 1 --2- 172.21.15.42:0/1984358041 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f231c03ac80 0x7f231c03d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:17:02.913 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:02.899+0000 7f23267fc700 1 -- 172.21.15.42:0/1984358041 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 10417+0+0 (secure 0 0 0) 0x7f232804b7a0 con 0x7f2334114490 2019-12-05T06:17:02.914 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:02.899+0000 7f2339bf2700 1 --2- 172.21.15.42:0/1984358041 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f231c03ac80 0x7f231c03d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:17:02.914 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:02.899+0000 7f2339bf2700 1 --2- 172.21.15.42:0/1984358041 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f231c03ac80 0x7f231c03d190 crc :-1 s=READY pgs=999 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:17:02.915 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:02.903+0000 7f233b655700 1 -- 172.21.15.42:0/1984358041 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f233405d110 con 0x7f2334114490 2019-12-05T06:17:02.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:02.911+0000 7f23393f1700 1 --2- 172.21.15.42:0/1984358041 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2334114490 0x7f2334077d80 secure :-1 s=THROTTLE_DONE pgs=2379 cs=0 l=1 rx=0x7f2328008b10 tx=0x7f232800a2e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:02.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:02.911+0000 7f23267fc700 1 -- 172.21.15.42:0/1984358041 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f2328032a80 con 0x7f2334114490 2019-12-05T06:17:03.079 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:03.071+0000 7f233b655700 1 -- 172.21.15.42:0/1984358041 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f233405b8c0 con 0x7f2334114490 2019-12-05T06:17:03.080 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:03.071+0000 7f23393f1700 1 --2- 172.21.15.42:0/1984358041 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2334114490 0x7f2334077d80 secure :-1 s=THROTTLE_DONE pgs=2379 cs=0 l=1 rx=0x7f2328008b10 tx=0x7f232800a2e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:03.080 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:03.071+0000 7f23267fc700 1 -- 172.21.15.42:0/1984358041 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 285 v285) v1 ==== 95+0+1779 (secure 0 0 0) 0x7f2328030440 con 0x7f2334114490 2019-12-05T06:17:03.080 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 285 2019-12-05T06:17:03.081 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:17:03.081 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":285,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":285,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:17:02.742260+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":100,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7295},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7295":{"gid":7295,"name":"a","rank":0,"incarnation":284,"state":"up:reconnect","state_seq":36,"addr":"172.21.15.42:6835/3228549071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3228549071},{"type":"v1","addr":"172.21.15.42:6835","nonce":3228549071}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:17:03.082 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:03.075+0000 7f233b655700 1 -- 172.21.15.42:0/1984358041 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f231c03ac80 msgr2=0x7f231c03d190 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:17:03.083 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:03.075+0000 7f233b655700 1 --2- 172.21.15.42:0/1984358041 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f231c03ac80 0x7f231c03d190 crc :-1 s=READY pgs=999 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:17:03.083 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:03.075+0000 7f233b655700 1 -- 172.21.15.42:0/1984358041 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2334114490 msgr2=0x7f2334077d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:17:03.083 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:03.075+0000 7f233b655700 1 --2- 172.21.15.42:0/1984358041 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2334114490 0x7f2334077d80 secure :-1 s=READY pgs=2379 cs=0 l=1 rx=0x7f2328008b10 tx=0x7f232800a2e0).stop 2019-12-05T06:17:03.083 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:03.075+0000 7f233b655700 1 -- 172.21.15.42:0/1984358041 shutdown_connections 2019-12-05T06:17:03.084 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:03.075+0000 7f233b655700 1 --2- 172.21.15.42:0/1984358041 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f231c03ac80 0x7f231c03d190 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:17:03.084 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:03.075+0000 7f233b655700 1 --2- 172.21.15.42:0/1984358041 >> v2:172.21.15.40:3300/0 conn(0x7f2334116970 0x7f2334118940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:17:03.085 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:03.075+0000 7f233b655700 1 --2- 172.21.15.42:0/1984358041 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2334114490 0x7f2334077d80 unknown :-1 s=CLOSED pgs=2379 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:17:03.085 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:03.075+0000 7f233b655700 1 -- 172.21.15.42:0/1984358041 shutdown_connections 2019-12-05T06:17:03.085 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:03.075+0000 7f233b655700 1 -- 172.21.15.42:0/1984358041 wait complete. 2019-12-05T06:17:03.085 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:03.075+0000 7f233b655700 1 -- 172.21.15.42:0/1984358041 >> 172.21.15.42:0/1984358041 conn(0x7f23340a6fd0 msgr2=0x7f2334111db0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:17:03.092 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2019-12-05T06:17:05.092 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:17:05.200 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:05.187+0000 7f780344e700 1 Processor -- start 2019-12-05T06:17:05.200 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:05.187+0000 7f780344e700 1 -- start start 2019-12-05T06:17:05.200 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:05.187+0000 7f780344e700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f77fc0a7730 0x7f77fc08df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:17:05.200 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:05.187+0000 7f780344e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f77fc09fad0 0x7f77fc07eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:17:05.200 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:05.187+0000 7f780344e700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77fc0a0700 con 0x7f77fc07e710 2019-12-05T06:17:05.201 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:05.187+0000 7f780344e700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77fc0a0af0 con 0x7f77fc0a7730 2019-12-05T06:17:05.201 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:05.187+0000 7f780344e700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77fc08e5a0 con 0x7f77fc09fad0 2019-12-05T06:17:05.201 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:05.187+0000 7f78021ec700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f77fc09fad0 0x7f77fc07eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:17:05.201 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:05.187+0000 7f78021ec700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f77fc09fad0 0x7f77fc07eb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:52100/0 (socket says 172.21.15.42:52100) 2019-12-05T06:17:05.201 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:05.187+0000 7f78021ec700 1 -- 172.21.15.42:0/2614995741 learned_addr learned my addr 172.21.15.42:0/2614995741 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:17:05.201 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:05.187+0000 7f78019eb700 1 --2- 172.21.15.42:0/2614995741 >> v2:172.21.15.40:3300/0 conn(0x7f77fc0a7730 0x7f77fc08df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:17:05.201 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:05.187+0000 7f78021ec700 1 -- 172.21.15.42:0/2614995741 >> v2:172.21.15.40:3300/0 conn(0x7f77fc0a7730 msgr2=0x7f77fc08df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:17:05.202 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:05.187+0000 7f78021ec700 1 --2- 172.21.15.42:0/2614995741 >> v2:172.21.15.40:3300/0 conn(0x7f77fc0a7730 0x7f77fc08df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:17:05.202 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:05.187+0000 7f78009e9700 1 -- 172.21.15.42:0/2614995741 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3642405345 0 0) 0x7f77fc0a0700 con 0x7f77fc07e710 2019-12-05T06:17:05.202 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:05.187+0000 7f78021ec700 1 -- 172.21.15.42:0/2614995741 >> v1:172.21.15.40:6789/0 conn(0x7f77fc07e710 legacy=0x7f77fc09f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:17:05.202 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:05.187+0000 7f78021ec700 1 -- 172.21.15.42:0/2614995741 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77fc07f090 con 0x7f77fc09fad0 2019-12-05T06:17:05.202 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:05.187+0000 7f78021ec700 1 --2- 172.21.15.42:0/2614995741 >> v2:172.21.15.42:3300/0 conn(0x7f77fc09fad0 0x7f77fc07eb40 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f77f40032b0 tx=0x7f77f40100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:05.202 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:05.187+0000 7f78021ec700 1 --2- 172.21.15.42:0/2614995741 >> v2:172.21.15.42:3300/0 conn(0x7f77fc09fad0 0x7f77fc07eb40 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f77f40032b0 tx=0x7f77f40100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:05.202 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:05.187+0000 7f78021ec700 1 --2- 172.21.15.42:0/2614995741 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f77fc09fad0 0x7f77fc07eb40 secure :-1 s=READY pgs=2380 cs=0 l=1 rx=0x7f77f40032b0 tx=0x7f77f40100b0).ready entity=mon.1 client_cookie=299e39487f0d5672 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:17:05.203 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:05.187+0000 7f78021ec700 1 --2- 172.21.15.42:0/2614995741 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f77fc09fad0 0x7f77fc07eb40 secure :-1 s=THROTTLE_DONE pgs=2380 cs=0 l=1 rx=0x7f77f40032b0 tx=0x7f77f40100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:05.203 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:05.187+0000 7f78009e9700 1 -- 172.21.15.42:0/2614995741 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f77f4010d60 con 0x7f77fc09fad0 2019-12-05T06:17:05.203 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:05.187+0000 7f78021ec700 1 --2- 172.21.15.42:0/2614995741 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f77fc09fad0 0x7f77fc07eb40 secure :-1 s=THROTTLE_DONE pgs=2380 cs=0 l=1 rx=0x7f77f40032b0 tx=0x7f77f40100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:05.203 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:05.187+0000 7f78021ec700 1 --2- 172.21.15.42:0/2614995741 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f77fc09fad0 0x7f77fc07eb40 secure :-1 s=THROTTLE_DONE pgs=2380 cs=0 l=1 rx=0x7f77f40032b0 tx=0x7f77f40100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:05.203 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:05.187+0000 7f78009e9700 1 -- 172.21.15.42:0/2614995741 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77f4010980 con 0x7f77fc09fad0 2019-12-05T06:17:05.203 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:05.187+0000 7f78009e9700 1 -- 172.21.15.42:0/2614995741 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f77f40114a0 con 0x7f77fc09fad0 2019-12-05T06:17:05.203 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:05.187+0000 7f780344e700 1 -- 172.21.15.42:0/2614995741 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f77fc09fad0 msgr2=0x7f77fc07eb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:17:05.203 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:05.187+0000 7f780344e700 1 --2- 172.21.15.42:0/2614995741 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f77fc09fad0 0x7f77fc07eb40 secure :-1 s=READY pgs=2380 cs=0 l=1 rx=0x7f77f40032b0 tx=0x7f77f40100b0).stop 2019-12-05T06:17:05.204 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:05.191+0000 7f780344e700 1 -- 172.21.15.42:0/2614995741 shutdown_connections 2019-12-05T06:17:05.204 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:05.191+0000 7f780344e700 1 --2- 172.21.15.42:0/2614995741 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f77fc09fad0 0x7f77fc07eb40 unknown :-1 s=CLOSED pgs=2380 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:17:05.204 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:05.191+0000 7f780344e700 1 --2- 172.21.15.42:0/2614995741 >> v2:172.21.15.40:3300/0 conn(0x7f77fc0a7730 0x7f77fc08df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:17:05.204 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:05.191+0000 7f780344e700 1 -- 172.21.15.42:0/2614995741 shutdown_connections 2019-12-05T06:17:05.204 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:05.191+0000 7f780344e700 1 -- 172.21.15.42:0/2614995741 wait complete. 2019-12-05T06:17:05.204 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:05.191+0000 7f780344e700 1 -- 172.21.15.42:0/2614995741 >> 172.21.15.42:0/2614995741 conn(0x7f77fc0aabd0 msgr2=0x7f77fc0a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:17:05.204 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:05.191+0000 7f780344e700 1 Processor -- start 2019-12-05T06:17:05.205 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:05.191+0000 7f780344e700 1 -- start start 2019-12-05T06:17:05.205 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:05.191+0000 7f780344e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f77fc11b500 0x7f77fc122010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:17:05.205 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:05.191+0000 7f780344e700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f77fc11bde0 0x7f77fc11ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:17:05.205 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:05.191+0000 7f780344e700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77fc11c210 con 0x7f77fc116a50 2019-12-05T06:17:05.205 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:05.191+0000 7f780344e700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77fc1165f0 con 0x7f77fc11bde0 2019-12-05T06:17:05.205 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:05.191+0000 7f780344e700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77fc117280 con 0x7f77fc11b500 2019-12-05T06:17:05.205 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:05.191+0000 7f78019eb700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f77fc11b500 0x7f77fc122010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:17:05.206 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:05.195+0000 7f78021ec700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f77fc116a50 0x7f77fc119f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:52144/0 (socket says 172.21.15.42:52144) 2019-12-05T06:17:05.206 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:05.195+0000 7f78021ec700 1 -- 172.21.15.42:0/3112747487 learned_addr learned my addr 172.21.15.42:0/3112747487 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:17:05.206 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:05.195+0000 7f78011ea700 1 --2- 172.21.15.42:0/3112747487 >> v2:172.21.15.40:3300/0 conn(0x7f77fc11bde0 0x7f77fc11ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:17:05.208 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:05.195+0000 7f78019eb700 1 -- 172.21.15.42:0/3112747487 >> v2:172.21.15.40:3300/0 conn(0x7f77fc11bde0 msgr2=0x7f77fc11ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:17:05.208 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:05.195+0000 7f77ee7fc700 1 -- 172.21.15.42:0/3112747487 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1789132462 0 0) 0x7f77fc11c210 con 0x7f77fc116a50 2019-12-05T06:17:05.208 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:05.195+0000 7f78019eb700 1 --2- 172.21.15.42:0/3112747487 >> v2:172.21.15.40:3300/0 conn(0x7f77fc11bde0 0x7f77fc11ed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:17:05.208 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:05.195+0000 7f78019eb700 1 -- 172.21.15.42:0/3112747487 >> v1:172.21.15.42:6789/0 conn(0x7f77fc116a50 legacy=0x7f77fc119f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:17:05.208 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:05.195+0000 7f78019eb700 1 -- 172.21.15.42:0/3112747487 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77fc116e80 con 0x7f77fc11b500 2019-12-05T06:17:05.209 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:05.195+0000 7f78019eb700 1 --2- 172.21.15.42:0/3112747487 >> v2:172.21.15.42:3300/0 conn(0x7f77fc11b500 0x7f77fc122010 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f77f0002f30 tx=0x7f77f00109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:05.209 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:05.195+0000 7f78019eb700 1 --2- 172.21.15.42:0/3112747487 >> v2:172.21.15.42:3300/0 conn(0x7f77fc11b500 0x7f77fc122010 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f77f0002f30 tx=0x7f77f00109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:05.209 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:05.195+0000 7f78019eb700 1 --2- 172.21.15.42:0/3112747487 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f77fc11b500 0x7f77fc122010 secure :-1 s=READY pgs=2382 cs=0 l=1 rx=0x7f77f0002f30 tx=0x7f77f00109e0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:17:05.210 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:05.195+0000 7f78019eb700 1 --2- 172.21.15.42:0/3112747487 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f77fc11b500 0x7f77fc122010 secure :-1 s=THROTTLE_DONE pgs=2382 cs=0 l=1 rx=0x7f77f0002f30 tx=0x7f77f00109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:05.210 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:05.195+0000 7f78019eb700 1 --2- 172.21.15.42:0/3112747487 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f77fc11b500 0x7f77fc122010 secure :-1 s=THROTTLE_DONE pgs=2382 cs=0 l=1 rx=0x7f77f0002f30 tx=0x7f77f00109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:05.210 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:05.195+0000 7f77ee7fc700 1 -- 172.21.15.42:0/3112747487 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f77f0005920 con 0x7f77fc11b500 2019-12-05T06:17:05.210 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:05.195+0000 7f78019eb700 1 --2- 172.21.15.42:0/3112747487 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f77fc11b500 0x7f77fc122010 secure :-1 s=THROTTLE_DONE pgs=2382 cs=0 l=1 rx=0x7f77f0002f30 tx=0x7f77f00109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:05.210 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:05.195+0000 7f77ee7fc700 1 -- 172.21.15.42:0/3112747487 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77f0005540 con 0x7f77fc11b500 2019-12-05T06:17:05.210 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:05.195+0000 7f780344e700 1 -- 172.21.15.42:0/3112747487 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77fc11a780 con 0x7f77fc11b500 2019-12-05T06:17:05.210 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:05.195+0000 7f77ee7fc700 1 -- 172.21.15.42:0/3112747487 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f77f0012490 con 0x7f77fc11b500 2019-12-05T06:17:05.211 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:05.195+0000 7f780344e700 1 -- 172.21.15.42:0/3112747487 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77fc11ad30 con 0x7f77fc11b500 2019-12-05T06:17:05.211 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:05.195+0000 7f78019eb700 1 --2- 172.21.15.42:0/3112747487 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f77fc11b500 0x7f77fc122010 secure :-1 s=THROTTLE_DONE pgs=2382 cs=0 l=1 rx=0x7f77f0002f30 tx=0x7f77f00109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:05.211 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:05.195+0000 7f77ee7fc700 1 -- 172.21.15.42:0/3112747487 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f77f0007180 con 0x7f77fc11b500 2019-12-05T06:17:05.211 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:05.195+0000 7f78019eb700 1 --2- 172.21.15.42:0/3112747487 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f77fc11b500 0x7f77fc122010 secure :-1 s=THROTTLE_DONE pgs=2382 cs=0 l=1 rx=0x7f77f0002f30 tx=0x7f77f00109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:05.211 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:05.195+0000 7f77ee7fc700 1 --2- 172.21.15.42:0/3112747487 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f77e403ac60 0x7f77e403d170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:17:05.211 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:05.195+0000 7f77ee7fc700 1 -- 172.21.15.42:0/3112747487 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 10417+0+0 (secure 0 0 0) 0x7f77f004ace0 con 0x7f77fc11b500 2019-12-05T06:17:05.211 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:05.199+0000 7f78011ea700 1 --2- 172.21.15.42:0/3112747487 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f77e403ac60 0x7f77e403d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:17:05.212 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:05.199+0000 7f78011ea700 1 --2- 172.21.15.42:0/3112747487 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f77e403ac60 0x7f77e403d170 crc :-1 s=READY pgs=1000 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:17:05.212 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:05.203+0000 7f780344e700 1 -- 172.21.15.42:0/3112747487 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77fc11b930 con 0x7f77fc11b500 2019-12-05T06:17:05.214 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:05.207+0000 7f78019eb700 1 --2- 172.21.15.42:0/3112747487 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f77fc11b500 0x7f77fc122010 secure :-1 s=THROTTLE_DONE pgs=2382 cs=0 l=1 rx=0x7f77f0002f30 tx=0x7f77f00109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:05.215 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:05.207+0000 7f77ee7fc700 1 -- 172.21.15.42:0/3112747487 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f77f0031d60 con 0x7f77fc11b500 2019-12-05T06:17:05.383 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:05.375+0000 7f780344e700 1 -- 172.21.15.42:0/3112747487 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f77fc05d100 con 0x7f77fc11b500 2019-12-05T06:17:05.384 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:05.379+0000 7f78019eb700 1 --2- 172.21.15.42:0/3112747487 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f77fc11b500 0x7f77fc122010 secure :-1 s=THROTTLE_DONE pgs=2382 cs=0 l=1 rx=0x7f77f0002f30 tx=0x7f77f00109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:05.384 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:05.379+0000 7f77ee7fc700 1 -- 172.21.15.42:0/3112747487 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 287 v287) v1 ==== 95+0+1776 (secure 0 0 0) 0x7f77f0031fc0 con 0x7f77fc11b500 2019-12-05T06:17:05.385 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 287 2019-12-05T06:17:05.385 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:17:05.385 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":287,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":287,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:17:04.753930+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":100,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7295},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7295":{"gid":7295,"name":"a","rank":0,"incarnation":284,"state":"up:active","state_seq":38,"addr":"172.21.15.42:6835/3228549071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3228549071},{"type":"v1","addr":"172.21.15.42:6835","nonce":3228549071}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:17:05.387 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:05.379+0000 7f780344e700 1 -- 172.21.15.42:0/3112747487 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f77e403ac60 msgr2=0x7f77e403d170 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:17:05.387 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:05.379+0000 7f780344e700 1 --2- 172.21.15.42:0/3112747487 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f77e403ac60 0x7f77e403d170 crc :-1 s=READY pgs=1000 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:17:05.388 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:05.379+0000 7f780344e700 1 -- 172.21.15.42:0/3112747487 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f77fc11b500 msgr2=0x7f77fc122010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:17:05.388 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:05.379+0000 7f780344e700 1 --2- 172.21.15.42:0/3112747487 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f77fc11b500 0x7f77fc122010 secure :-1 s=READY pgs=2382 cs=0 l=1 rx=0x7f77f0002f30 tx=0x7f77f00109e0).stop 2019-12-05T06:17:05.389 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:05.379+0000 7f780344e700 1 -- 172.21.15.42:0/3112747487 shutdown_connections 2019-12-05T06:17:05.389 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:05.379+0000 7f780344e700 1 --2- 172.21.15.42:0/3112747487 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f77e403ac60 0x7f77e403d170 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:17:05.389 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:05.379+0000 7f780344e700 1 --2- 172.21.15.42:0/3112747487 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f77fc11b500 0x7f77fc122010 unknown :-1 s=CLOSED pgs=2382 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:17:05.389 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:05.379+0000 7f780344e700 1 --2- 172.21.15.42:0/3112747487 >> v2:172.21.15.40:3300/0 conn(0x7f77fc11bde0 0x7f77fc11ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:17:05.389 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:05.383+0000 7f780344e700 1 -- 172.21.15.42:0/3112747487 shutdown_connections 2019-12-05T06:17:05.390 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:05.383+0000 7f780344e700 1 -- 172.21.15.42:0/3112747487 wait complete. 2019-12-05T06:17:05.390 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:05.383+0000 7f780344e700 1 -- 172.21.15.42:0/3112747487 >> 172.21.15.42:0/3112747487 conn(0x7f77fc07e5c0 msgr2=0x7f77fc08eb40 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:17:05.400 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a has gained rank=0, replacing gid=7217 2019-12-05T06:17:05.400 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 4 secs before reviving mds.b 2019-12-05T06:17:09.401 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.b 2019-12-05T06:17:09.401 INFO:tasks.ceph.mds.b:Restarting daemon 2019-12-05T06:17:09.401 INFO:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2019-12-05T06:17:09.405 INFO:tasks.ceph.mds.b:Started 2019-12-05T06:17:09.405 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:17:09.466 INFO:tasks.ceph.mds.b.smithi040.stdout:starting mds.b at 2019-12-05T06:17:09.515 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:09.495+0000 7f815d26a700 1 Processor -- start 2019-12-05T06:17:09.516 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:09.495+0000 7f815d26a700 1 -- start start 2019-12-05T06:17:09.516 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:09.499+0000 7f815d26a700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f815808df80 0x7f815809f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:17:09.516 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:09.499+0000 7f815d26a700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f815809fc60 con 0x7f81580a7740 2019-12-05T06:17:09.516 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:09.499+0000 7f815d26a700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81580a0710 con 0x7f815809f830 2019-12-05T06:17:09.516 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:09.499+0000 7f815d26a700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81580a0b00 con 0x7f815808df80 2019-12-05T06:17:09.517 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:09.499+0000 7f8157fff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f815809f830 0x7f815814efe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:52166/0 (socket says 172.21.15.42:52166) 2019-12-05T06:17:09.517 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:09.499+0000 7f8157fff700 1 -- 172.21.15.42:0/1895390104 learned_addr learned my addr 172.21.15.42:0/1895390104 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:17:09.517 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:09.499+0000 7f814ffff700 1 --2- 172.21.15.42:0/1895390104 >> v2:172.21.15.40:3300/0 conn(0x7f815808df80 0x7f815809f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:17:09.517 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:09.499+0000 7f8156ffd700 1 -- 172.21.15.42:0/1895390104 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2233841917 0 0) 0x7f81580a0710 con 0x7f815809f830 2019-12-05T06:17:09.517 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:09.499+0000 7f8156ffd700 1 -- 172.21.15.42:0/1895390104 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81440030b0 con 0x7f815809f830 2019-12-05T06:17:09.517 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:09.499+0000 7f8156ffd700 1 -- 172.21.15.42:0/1895390104 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3032882621 0 0) 0x7f815809fc60 con 0x7f81580a7740 2019-12-05T06:17:09.517 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:09.499+0000 7f8156ffd700 1 -- 172.21.15.42:0/1895390104 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81580a0710 con 0x7f81580a7740 2019-12-05T06:17:09.518 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:09.499+0000 7f814ffff700 1 -- 172.21.15.42:0/1895390104 >> v1:172.21.15.42:6789/0 conn(0x7f815809f830 legacy=0x7f815814efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:17:09.518 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:09.499+0000 7f814ffff700 1 -- 172.21.15.42:0/1895390104 >> v1:172.21.15.40:6789/0 conn(0x7f81580a7740 legacy=0x7f815807e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:17:09.518 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:09.499+0000 7f814ffff700 1 -- 172.21.15.42:0/1895390104 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f815807ef20 con 0x7f815808df80 2019-12-05T06:17:09.518 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:09.499+0000 7f814ffff700 1 --2- 172.21.15.42:0/1895390104 >> v2:172.21.15.40:3300/0 conn(0x7f815808df80 0x7f815809f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f8140008790 tx=0x7f814000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:09.518 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:09.499+0000 7f814ffff700 1 --2- 172.21.15.42:0/1895390104 >> v2:172.21.15.40:3300/0 conn(0x7f815808df80 0x7f815809f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f8140008790 tx=0x7f814000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:09.518 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:09.499+0000 7f814ffff700 1 --2- 172.21.15.42:0/1895390104 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f815808df80 0x7f815809f2e0 secure :-1 s=READY pgs=1811 cs=0 l=1 rx=0x7f8140008790 tx=0x7f814000fbf0).ready entity=mon.0 client_cookie=da088438622291f8 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:17:09.518 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:09.499+0000 7f814ffff700 1 --2- 172.21.15.42:0/1895390104 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f815808df80 0x7f815809f2e0 secure :-1 s=THROTTLE_DONE pgs=1811 cs=0 l=1 rx=0x7f8140008790 tx=0x7f814000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:09.519 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:09.499+0000 7f8156ffd700 1 -- 172.21.15.42:0/1895390104 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8140011270 con 0x7f815808df80 2019-12-05T06:17:09.519 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:09.499+0000 7f814ffff700 1 --2- 172.21.15.42:0/1895390104 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f815808df80 0x7f815809f2e0 secure :-1 s=THROTTLE_DONE pgs=1811 cs=0 l=1 rx=0x7f8140008790 tx=0x7f814000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:09.519 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:09.499+0000 7f814ffff700 1 --2- 172.21.15.42:0/1895390104 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f815808df80 0x7f815809f2e0 secure :-1 s=THROTTLE_DONE pgs=1811 cs=0 l=1 rx=0x7f8140008790 tx=0x7f814000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:09.519 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:09.499+0000 7f8156ffd700 1 -- 172.21.15.42:0/1895390104 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8140008bc0 con 0x7f815808df80 2019-12-05T06:17:09.519 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:09.499+0000 7f8156ffd700 1 -- 172.21.15.42:0/1895390104 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8140011850 con 0x7f815808df80 2019-12-05T06:17:09.519 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:09.499+0000 7f815d26a700 1 -- 172.21.15.42:0/1895390104 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f815808df80 msgr2=0x7f815809f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:17:09.519 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:09.499+0000 7f815d26a700 1 --2- 172.21.15.42:0/1895390104 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f815808df80 0x7f815809f2e0 secure :-1 s=READY pgs=1811 cs=0 l=1 rx=0x7f8140008790 tx=0x7f814000fbf0).stop 2019-12-05T06:17:09.520 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:09.503+0000 7f815d26a700 1 -- 172.21.15.42:0/1895390104 shutdown_connections 2019-12-05T06:17:09.520 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:09.503+0000 7f815d26a700 1 --2- 172.21.15.42:0/1895390104 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f815808df80 0x7f815809f2e0 unknown :-1 s=CLOSED pgs=1811 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:17:09.520 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:09.503+0000 7f815d26a700 1 -- 172.21.15.42:0/1895390104 shutdown_connections 2019-12-05T06:17:09.520 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:09.503+0000 7f815d26a700 1 -- 172.21.15.42:0/1895390104 wait complete. 2019-12-05T06:17:09.520 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:09.503+0000 7f815d26a700 1 -- 172.21.15.42:0/1895390104 >> 172.21.15.42:0/1895390104 conn(0x7f81580aabe0 msgr2=0x7f81580a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:17:09.520 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:09.503+0000 7f815d26a700 1 Processor -- start 2019-12-05T06:17:09.521 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:09.503+0000 7f815d26a700 1 -- start start 2019-12-05T06:17:09.521 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:09.507+0000 7f815d26a700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8158142cd0 0x7f8158144860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:17:09.521 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:09.507+0000 7f815d26a700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f81580a41a0 0x7f8158121ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:17:09.521 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:09.507+0000 7f815d26a700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8158122440 con 0x7f815814a1c0 2019-12-05T06:17:09.521 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:09.507+0000 7f815d26a700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8158142530 con 0x7f8158142cd0 2019-12-05T06:17:09.521 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:09.507+0000 7f815d26a700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81580a39c0 con 0x7f81580a41a0 2019-12-05T06:17:09.521 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:09.507+0000 7f814ffff700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f81580a41a0 0x7f8158121ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:17:09.522 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:09.507+0000 7f814ffff700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f81580a41a0 0x7f8158121ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:52598/0 (socket says 172.21.15.42:52598) 2019-12-05T06:17:09.522 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:09.507+0000 7f81577fe700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8158142cd0 0x7f8158144860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:17:09.522 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:09.507+0000 7f814ffff700 1 -- 172.21.15.42:0/1946777668 learned_addr learned my addr 172.21.15.42:0/1946777668 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:17:09.522 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:09.507+0000 7f814ffff700 1 -- 172.21.15.42:0/1946777668 >> v2:172.21.15.40:3300/0 conn(0x7f8158142cd0 msgr2=0x7f8158144860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:17:09.522 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:09.507+0000 7f814ffff700 1 --2- 172.21.15.42:0/1946777668 >> v2:172.21.15.40:3300/0 conn(0x7f8158142cd0 0x7f8158144860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:17:09.522 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:09.507+0000 7f814ffff700 1 -- 172.21.15.42:0/1946777668 >> v1:172.21.15.40:6789/0 conn(0x7f815814a1c0 legacy=0x7f81581254b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:17:09.522 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:09.507+0000 7f814ffff700 1 -- 172.21.15.42:0/1946777668 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81580a3590 con 0x7f81580a41a0 2019-12-05T06:17:09.523 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:09.507+0000 7f814ffff700 1 --2- 172.21.15.42:0/1946777668 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f81580a41a0 0x7f8158121ef0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f8140008790 tx=0x7f8140008c70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:09.523 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:09.507+0000 7f814ffff700 1 --2- 172.21.15.42:0/1946777668 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f81580a41a0 0x7f8158121ef0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f8140008790 tx=0x7f8140008c70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:09.523 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:09.507+0000 7f814ffff700 1 --2- 172.21.15.42:0/1946777668 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f81580a41a0 0x7f8158121ef0 secure :-1 s=READY pgs=526 cs=0 l=1 rx=0x7f8140008790 tx=0x7f8140008c70).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:17:09.523 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:09.507+0000 7f814ffff700 1 --2- 172.21.15.42:0/1946777668 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f81580a41a0 0x7f8158121ef0 secure :-1 s=THROTTLE_DONE pgs=526 cs=0 l=1 rx=0x7f8140008790 tx=0x7f8140008c70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:09.523 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:09.507+0000 7f8154ff9700 1 -- 172.21.15.42:0/1946777668 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8140007150 con 0x7f81580a41a0 2019-12-05T06:17:09.523 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:09.507+0000 7f815d26a700 1 -- 172.21.15.42:0/1946777668 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f815814a5f0 con 0x7f81580a41a0 2019-12-05T06:17:09.523 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:09.507+0000 7f814ffff700 1 --2- 172.21.15.42:0/1946777668 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f81580a41a0 0x7f8158121ef0 secure :-1 s=THROTTLE_DONE pgs=526 cs=0 l=1 rx=0x7f8140008790 tx=0x7f8140008c70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:09.524 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:09.507+0000 7f814ffff700 1 --2- 172.21.15.42:0/1946777668 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f81580a41a0 0x7f8158121ef0 secure :-1 s=THROTTLE_DONE pgs=526 cs=0 l=1 rx=0x7f8140008790 tx=0x7f8140008c70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:09.524 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:09.507+0000 7f8154ff9700 1 -- 172.21.15.42:0/1946777668 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8140011470 con 0x7f81580a41a0 2019-12-05T06:17:09.524 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:09.507+0000 7f8154ff9700 1 -- 172.21.15.42:0/1946777668 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f814000f550 con 0x7f81580a41a0 2019-12-05T06:17:09.524 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:09.507+0000 7f815d26a700 1 -- 172.21.15.42:0/1946777668 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8158125cb0 con 0x7f81580a41a0 2019-12-05T06:17:09.525 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:09.511+0000 7f814ffff700 1 --2- 172.21.15.42:0/1946777668 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f81580a41a0 0x7f8158121ef0 secure :-1 s=THROTTLE_DONE pgs=526 cs=0 l=1 rx=0x7f8140008790 tx=0x7f8140008c70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:09.526 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:09.511+0000 7f8154ff9700 1 -- 172.21.15.42:0/1946777668 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f8140005540 con 0x7f81580a41a0 2019-12-05T06:17:09.526 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:09.511+0000 7f814ffff700 1 --2- 172.21.15.42:0/1946777668 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f81580a41a0 0x7f8158121ef0 secure :-1 s=THROTTLE_DONE pgs=526 cs=0 l=1 rx=0x7f8140008790 tx=0x7f8140008c70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:09.526 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:09.511+0000 7f8154ff9700 1 --2- 172.21.15.42:0/1946777668 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f813c03aca0 0x7f813c03d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:17:09.526 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:09.511+0000 7f8154ff9700 1 -- 172.21.15.42:0/1946777668 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 10417+0+0 (secure 0 0 0) 0x7f814004bb10 con 0x7f81580a41a0 2019-12-05T06:17:09.526 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:09.511+0000 7f81577fe700 1 --2- 172.21.15.42:0/1946777668 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f813c03aca0 0x7f813c03d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:17:09.526 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:09.511+0000 7f81577fe700 1 --2- 172.21.15.42:0/1946777668 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f813c03aca0 0x7f813c03d1b0 crc :-1 s=READY pgs=1001 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:17:09.527 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:09.511+0000 7f815d26a700 1 -- 172.21.15.42:0/1946777668 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f815805d110 con 0x7f81580a41a0 2019-12-05T06:17:09.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:09.523+0000 7f814ffff700 1 --2- 172.21.15.42:0/1946777668 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f81580a41a0 0x7f8158121ef0 secure :-1 s=THROTTLE_DONE pgs=526 cs=0 l=1 rx=0x7f8140008790 tx=0x7f8140008c70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:09.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:09.523+0000 7f8154ff9700 1 -- 172.21.15.42:0/1946777668 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f8140032b70 con 0x7f81580a41a0 2019-12-05T06:17:09.701 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:09.695+0000 7f815d26a700 1 -- 172.21.15.42:0/1946777668 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f815805b8c0 con 0x7f81580a41a0 2019-12-05T06:17:09.702 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:09.695+0000 7f814ffff700 1 --2- 172.21.15.42:0/1946777668 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f81580a41a0 0x7f8158121ef0 secure :-1 s=THROTTLE_DONE pgs=526 cs=0 l=1 rx=0x7f8140008790 tx=0x7f8140008c70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:09.702 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:09.695+0000 7f8154ff9700 1 -- 172.21.15.42:0/1946777668 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 287 v287) v1 ==== 95+0+1776 (secure 0 0 0) 0x7f8140030840 con 0x7f81580a41a0 2019-12-05T06:17:09.702 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 287 2019-12-05T06:17:09.703 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:17:09.703 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":287,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":287,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:17:04.753930+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":100,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7295},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7295":{"gid":7295,"name":"a","rank":0,"incarnation":284,"state":"up:active","state_seq":38,"addr":"172.21.15.42:6835/3228549071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3228549071},{"type":"v1","addr":"172.21.15.42:6835","nonce":3228549071}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:17:09.705 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:09.699+0000 7f815d26a700 1 -- 172.21.15.42:0/1946777668 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f813c03aca0 msgr2=0x7f813c03d1b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:17:09.705 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:09.699+0000 7f815d26a700 1 --2- 172.21.15.42:0/1946777668 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f813c03aca0 0x7f813c03d1b0 crc :-1 s=READY pgs=1001 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:17:09.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:09.699+0000 7f815d26a700 1 -- 172.21.15.42:0/1946777668 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f81580a41a0 msgr2=0x7f8158121ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:17:09.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:09.699+0000 7f815d26a700 1 --2- 172.21.15.42:0/1946777668 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f81580a41a0 0x7f8158121ef0 secure :-1 s=READY pgs=526 cs=0 l=1 rx=0x7f8140008790 tx=0x7f8140008c70).stop 2019-12-05T06:17:09.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:09.699+0000 7f815d26a700 1 -- 172.21.15.42:0/1946777668 shutdown_connections 2019-12-05T06:17:09.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:09.699+0000 7f815d26a700 1 --2- 172.21.15.42:0/1946777668 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f813c03aca0 0x7f813c03d1b0 unknown :-1 s=CLOSED pgs=1001 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:17:09.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:09.699+0000 7f815d26a700 1 --2- 172.21.15.42:0/1946777668 >> v2:172.21.15.40:3300/0 conn(0x7f8158142cd0 0x7f8158144860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:17:09.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:09.699+0000 7f815d26a700 1 --2- 172.21.15.42:0/1946777668 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f81580a41a0 0x7f8158121ef0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:17:09.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:09.699+0000 7f815d26a700 1 -- 172.21.15.42:0/1946777668 shutdown_connections 2019-12-05T06:17:09.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:09.699+0000 7f815d26a700 1 -- 172.21.15.42:0/1946777668 wait complete. 2019-12-05T06:17:09.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:09.699+0000 7f815d26a700 1 -- 172.21.15.42:0/1946777668 >> 172.21.15.42:0/1946777668 conn(0x7f81580a82a0 msgr2=0x7f8158059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:17:09.715 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2019-12-05T06:17:11.716 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:17:11.826 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:11.815+0000 7f2596161700 1 Processor -- start 2019-12-05T06:17:11.826 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:11.815+0000 7f2596161700 1 -- start start 2019-12-05T06:17:11.826 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:11.815+0000 7f2596161700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f25900a7740 0x7f259008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:17:11.826 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:11.815+0000 7f2596161700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f259007e720 0x7f259007eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:17:11.827 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:11.815+0000 7f2596161700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25900a0710 con 0x7f259009f2e0 2019-12-05T06:17:11.827 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:11.815+0000 7f2596161700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25900a0b00 con 0x7f25900a7740 2019-12-05T06:17:11.827 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:11.815+0000 7f2596161700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f259008e5b0 con 0x7f259007e720 2019-12-05T06:17:11.827 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:11.815+0000 7f258f7fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f259007e720 0x7f259007eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:17:11.827 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:11.815+0000 7f258f7fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f259007e720 0x7f259007eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:52148/0 (socket says 172.21.15.42:52148) 2019-12-05T06:17:11.827 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:11.815+0000 7f258f7fe700 1 -- 172.21.15.42:0/2582446985 learned_addr learned my addr 172.21.15.42:0/2582446985 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:17:11.827 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:11.815+0000 7f258ffff700 1 --2- 172.21.15.42:0/2582446985 >> v2:172.21.15.40:3300/0 conn(0x7f25900a7740 0x7f259008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:17:11.828 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:11.815+0000 7f258f7fe700 1 -- 172.21.15.42:0/2582446985 >> v2:172.21.15.40:3300/0 conn(0x7f25900a7740 msgr2=0x7f259008df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:17:11.828 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:11.815+0000 7f258f7fe700 1 --2- 172.21.15.42:0/2582446985 >> v2:172.21.15.40:3300/0 conn(0x7f25900a7740 0x7f259008df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:17:11.828 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:11.815+0000 7f258f7fe700 1 -- 172.21.15.42:0/2582446985 >> v1:172.21.15.42:6789/0 conn(0x7f259009f2e0 legacy=0x7f259009f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:17:11.828 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:11.815+0000 7f258f7fe700 1 -- 172.21.15.42:0/2582446985 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f259007f0a0 con 0x7f259007e720 2019-12-05T06:17:11.828 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:11.815+0000 7f258f7fe700 1 --2- 172.21.15.42:0/2582446985 >> v2:172.21.15.42:3300/0 conn(0x7f259007e720 0x7f259007eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f2578002f30 tx=0x7f257800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:11.828 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:11.815+0000 7f258f7fe700 1 --2- 172.21.15.42:0/2582446985 >> v2:172.21.15.42:3300/0 conn(0x7f259007e720 0x7f259007eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f2578002f30 tx=0x7f257800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:11.829 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:11.815+0000 7f258f7fe700 1 --2- 172.21.15.42:0/2582446985 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f259007e720 0x7f259007eb50 secure :-1 s=READY pgs=2386 cs=0 l=1 rx=0x7f2578002f30 tx=0x7f257800fd30).ready entity=mon.1 client_cookie=28875056f53a6b05 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:17:11.829 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:11.815+0000 7f258f7fe700 1 --2- 172.21.15.42:0/2582446985 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f259007e720 0x7f259007eb50 secure :-1 s=THROTTLE_DONE pgs=2386 cs=0 l=1 rx=0x7f2578002f30 tx=0x7f257800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:11.829 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:11.815+0000 7f258f7fe700 1 --2- 172.21.15.42:0/2582446985 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f259007e720 0x7f259007eb50 secure :-1 s=THROTTLE_DONE pgs=2386 cs=0 l=1 rx=0x7f2578002f30 tx=0x7f257800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:11.829 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:11.815+0000 7f258effd700 1 -- 172.21.15.42:0/2582446985 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2578011370 con 0x7f259007e720 2019-12-05T06:17:11.829 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:11.815+0000 7f258f7fe700 1 --2- 172.21.15.42:0/2582446985 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f259007e720 0x7f259007eb50 secure :-1 s=THROTTLE_DONE pgs=2386 cs=0 l=1 rx=0x7f2578002f30 tx=0x7f257800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:11.829 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:11.815+0000 7f258effd700 1 -- 172.21.15.42:0/2582446985 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2578008bb0 con 0x7f259007e720 2019-12-05T06:17:11.829 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:11.819+0000 7f258effd700 1 -- 172.21.15.42:0/2582446985 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2578011950 con 0x7f259007e720 2019-12-05T06:17:11.830 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:11.819+0000 7f2596161700 1 -- 172.21.15.42:0/2582446985 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f259007e720 msgr2=0x7f259007eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:17:11.830 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:11.819+0000 7f2596161700 1 --2- 172.21.15.42:0/2582446985 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f259007e720 0x7f259007eb50 secure :-1 s=READY pgs=2386 cs=0 l=1 rx=0x7f2578002f30 tx=0x7f257800fd30).stop 2019-12-05T06:17:11.830 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:11.819+0000 7f2596161700 1 -- 172.21.15.42:0/2582446985 shutdown_connections 2019-12-05T06:17:11.830 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:11.819+0000 7f2596161700 1 --2- 172.21.15.42:0/2582446985 >> v2:172.21.15.40:3300/0 conn(0x7f25900a7740 0x7f259008df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:17:11.830 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:11.819+0000 7f2596161700 1 --2- 172.21.15.42:0/2582446985 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f259007e720 0x7f259007eb50 unknown :-1 s=CLOSED pgs=2386 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:17:11.830 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:11.819+0000 7f2596161700 1 -- 172.21.15.42:0/2582446985 shutdown_connections 2019-12-05T06:17:11.830 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:11.819+0000 7f2596161700 1 -- 172.21.15.42:0/2582446985 wait complete. 2019-12-05T06:17:11.831 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:11.819+0000 7f2596161700 1 -- 172.21.15.42:0/2582446985 >> 172.21.15.42:0/2582446985 conn(0x7f25900aabe0 msgr2=0x7f25900a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:17:11.834 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:11.819+0000 7f2596161700 1 Processor -- start 2019-12-05T06:17:11.834 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:11.819+0000 7f2596161700 1 -- start start 2019-12-05T06:17:11.834 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:11.823+0000 7f2596161700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2590116500 0x7f259007d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:17:11.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:11.823+0000 7f2596161700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2590129530 0x7f2590129960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:17:11.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:11.823+0000 7f2596161700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f259011b360 con 0x7f259011b8e0 2019-12-05T06:17:11.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:11.823+0000 7f2596161700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f259007c750 con 0x7f2590129530 2019-12-05T06:17:11.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:11.823+0000 7f2596161700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f259007c9f0 con 0x7f2590116500 2019-12-05T06:17:11.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:11.823+0000 7f258ffff700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2590116500 0x7f259007d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:17:11.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:11.823+0000 7f2594eff700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2590129530 0x7f2590129960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:17:11.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:11.823+0000 7f258f7fe700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f259011b8e0 0x7f259011bd10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:33400/0 (socket says 172.21.15.42:33400) 2019-12-05T06:17:11.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:11.823+0000 7f258ffff700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2590116500 0x7f259007d540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:52620/0 (socket says 172.21.15.42:52620) 2019-12-05T06:17:11.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:11.823+0000 7f258f7fe700 1 -- 172.21.15.42:0/4049512443 learned_addr learned my addr 172.21.15.42:0/4049512443 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:17:11.837 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:11.823+0000 7f258cff9700 1 -- 172.21.15.42:0/4049512443 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 86711535 0 0) 0x7f259011b360 con 0x7f259011b8e0 2019-12-05T06:17:11.837 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:11.823+0000 7f258cff9700 1 -- 172.21.15.42:0/4049512443 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25680030b0 con 0x7f259011b8e0 2019-12-05T06:17:11.837 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:11.823+0000 7f2594eff700 1 -- 172.21.15.42:0/4049512443 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2590116500 msgr2=0x7f259007d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:17:11.837 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:11.823+0000 7f2594eff700 1 --2- 172.21.15.42:0/4049512443 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2590116500 0x7f259007d540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:17:11.837 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:11.823+0000 7f2594eff700 1 -- 172.21.15.42:0/4049512443 >> v1:172.21.15.40:6789/0 conn(0x7f259011b8e0 legacy=0x7f259011bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:17:11.837 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:11.823+0000 7f2594eff700 1 -- 172.21.15.42:0/4049512443 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f259007c3a0 con 0x7f2590129530 2019-12-05T06:17:11.837 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:11.823+0000 7f258ffff700 1 --2- 172.21.15.42:0/4049512443 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2590116500 0x7f259007d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:17:11.838 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:11.823+0000 7f2594eff700 1 --2- 172.21.15.42:0/4049512443 >> v2:172.21.15.40:3300/0 conn(0x7f2590129530 0x7f2590129960 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f259007c910 tx=0x7f25880107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:11.838 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:11.823+0000 7f2594eff700 1 --2- 172.21.15.42:0/4049512443 >> v2:172.21.15.40:3300/0 conn(0x7f2590129530 0x7f2590129960 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f259007c910 tx=0x7f25880107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:11.838 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:11.823+0000 7f2594eff700 1 --2- 172.21.15.42:0/4049512443 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2590129530 0x7f2590129960 secure :-1 s=READY pgs=1814 cs=0 l=1 rx=0x7f259007c910 tx=0x7f25880107c0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:17:11.838 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:11.823+0000 7f2594eff700 1 --2- 172.21.15.42:0/4049512443 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2590129530 0x7f2590129960 secure :-1 s=THROTTLE_DONE pgs=1814 cs=0 l=1 rx=0x7f259007c910 tx=0x7f25880107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:11.838 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:11.823+0000 7f2594eff700 1 --2- 172.21.15.42:0/4049512443 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2590129530 0x7f2590129960 secure :-1 s=THROTTLE_DONE pgs=1814 cs=0 l=1 rx=0x7f259007c910 tx=0x7f25880107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:11.838 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:11.823+0000 7f258cff9700 1 -- 172.21.15.42:0/4049512443 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2588006a40 con 0x7f2590129530 2019-12-05T06:17:11.838 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:11.823+0000 7f2594eff700 1 --2- 172.21.15.42:0/4049512443 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2590129530 0x7f2590129960 secure :-1 s=THROTTLE_DONE pgs=1814 cs=0 l=1 rx=0x7f259007c910 tx=0x7f25880107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:11.839 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:11.823+0000 7f258cff9700 1 -- 172.21.15.42:0/4049512443 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2588006660 con 0x7f2590129530 2019-12-05T06:17:11.839 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:11.823+0000 7f258cff9700 1 -- 172.21.15.42:0/4049512443 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2588010fc0 con 0x7f2590129530 2019-12-05T06:17:11.839 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:11.823+0000 7f2596161700 1 -- 172.21.15.42:0/4049512443 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2590129eb0 con 0x7f2590129530 2019-12-05T06:17:11.839 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:11.823+0000 7f2596161700 1 -- 172.21.15.42:0/4049512443 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25901307a0 con 0x7f2590129530 2019-12-05T06:17:11.839 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:11.827+0000 7f2594eff700 1 --2- 172.21.15.42:0/4049512443 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2590129530 0x7f2590129960 secure :-1 s=THROTTLE_DONE pgs=1814 cs=0 l=1 rx=0x7f259007c910 tx=0x7f25880107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:11.839 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:11.827+0000 7f258cff9700 1 -- 172.21.15.42:0/4049512443 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f25880112d0 con 0x7f2590129530 2019-12-05T06:17:11.839 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:11.827+0000 7f2594eff700 1 --2- 172.21.15.42:0/4049512443 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2590129530 0x7f2590129960 secure :-1 s=THROTTLE_DONE pgs=1814 cs=0 l=1 rx=0x7f259007c910 tx=0x7f25880107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:11.840 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:11.827+0000 7f258cff9700 1 --2- 172.21.15.42:0/4049512443 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f256803b250 0x7f256803d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:17:11.840 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:11.827+0000 7f258cff9700 1 -- 172.21.15.42:0/4049512443 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 10417+0+0 (secure 0 0 0) 0x7f258804b050 con 0x7f2590129530 2019-12-05T06:17:11.840 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:11.827+0000 7f258ffff700 1 --2- 172.21.15.42:0/4049512443 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f256803b250 0x7f256803d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:17:11.840 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:11.827+0000 7f258ffff700 1 --2- 172.21.15.42:0/4049512443 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f256803b250 0x7f256803d760 crc :-1 s=READY pgs=1002 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:17:11.840 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:11.831+0000 7f2596161700 1 -- 172.21.15.42:0/4049512443 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f259005d110 con 0x7f2590129530 2019-12-05T06:17:11.846 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:11.839+0000 7f2594eff700 1 --2- 172.21.15.42:0/4049512443 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2590129530 0x7f2590129960 secure :-1 s=THROTTLE_DONE pgs=1814 cs=0 l=1 rx=0x7f259007c910 tx=0x7f25880107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:11.846 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:11.839+0000 7f258cff9700 1 -- 172.21.15.42:0/4049512443 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f2588032090 con 0x7f2590129530 2019-12-05T06:17:12.029 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:12.023+0000 7f2596161700 1 -- 172.21.15.42:0/4049512443 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f259005b8c0 con 0x7f2590129530 2019-12-05T06:17:12.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:12.023+0000 7f2594eff700 1 --2- 172.21.15.42:0/4049512443 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2590129530 0x7f2590129960 secure :-1 s=THROTTLE_DONE pgs=1814 cs=0 l=1 rx=0x7f259007c910 tx=0x7f25880107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:12.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:12.023+0000 7f258cff9700 1 -- 172.21.15.42:0/4049512443 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 287 v287) v1 ==== 95+0+1776 (secure 0 0 0) 0x7f25880322f0 con 0x7f2590129530 2019-12-05T06:17:12.032 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:17:12.032 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":287,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":287,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:17:04.753930+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":100,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7295},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7295":{"gid":7295,"name":"a","rank":0,"incarnation":284,"state":"up:active","state_seq":38,"addr":"172.21.15.42:6835/3228549071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3228549071},{"type":"v1","addr":"172.21.15.42:6835","nonce":3228549071}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:17:12.033 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 287 2019-12-05T06:17:12.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:12.027+0000 7f2596161700 1 -- 172.21.15.42:0/4049512443 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f256803b250 msgr2=0x7f256803d760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:17:12.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:12.027+0000 7f2596161700 1 --2- 172.21.15.42:0/4049512443 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f256803b250 0x7f256803d760 crc :-1 s=READY pgs=1002 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:17:12.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:12.027+0000 7f2596161700 1 -- 172.21.15.42:0/4049512443 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2590129530 msgr2=0x7f2590129960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:17:12.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:12.027+0000 7f2596161700 1 --2- 172.21.15.42:0/4049512443 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2590129530 0x7f2590129960 secure :-1 s=READY pgs=1814 cs=0 l=1 rx=0x7f259007c910 tx=0x7f25880107c0).stop 2019-12-05T06:17:12.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:12.027+0000 7f2596161700 1 -- 172.21.15.42:0/4049512443 shutdown_connections 2019-12-05T06:17:12.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:12.027+0000 7f2596161700 1 --2- 172.21.15.42:0/4049512443 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f256803b250 0x7f256803d760 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:17:12.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:12.027+0000 7f2596161700 1 --2- 172.21.15.42:0/4049512443 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2590129530 0x7f2590129960 unknown :-1 s=CLOSED pgs=1814 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:17:12.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:12.027+0000 7f2596161700 1 --2- 172.21.15.42:0/4049512443 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2590116500 0x7f259007d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:17:12.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:12.027+0000 7f2596161700 1 -- 172.21.15.42:0/4049512443 shutdown_connections 2019-12-05T06:17:12.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:12.027+0000 7f2596161700 1 -- 172.21.15.42:0/4049512443 wait complete. 2019-12-05T06:17:12.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:12.027+0000 7f2596161700 1 -- 172.21.15.42:0/4049512443 >> 172.21.15.42:0/4049512443 conn(0x7f25900a7f90 msgr2=0x7f2590059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:17:12.045 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2019-12-05T06:17:14.046 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:17:14.151 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:14.135+0000 7fca78bfa700 1 Processor -- start 2019-12-05T06:17:14.152 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:14.135+0000 7fca78bfa700 1 -- start start 2019-12-05T06:17:14.152 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:14.139+0000 7fca78bfa700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fca7408df80 0x7fca7409f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:17:14.152 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:14.139+0000 7fca78bfa700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fca7409f830 0x7fca7414efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:17:14.153 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:14.139+0000 7fca78bfa700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca7409fc60 con 0x7fca740a7740 2019-12-05T06:17:14.153 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:14.139+0000 7fca78bfa700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca740a0b00 con 0x7fca7409f830 2019-12-05T06:17:14.153 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:14.139+0000 7fca78bfa700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca7408e5b0 con 0x7fca7408df80 2019-12-05T06:17:14.153 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:14.139+0000 7fca7259c700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fca7408df80 0x7fca7409f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:17:14.154 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:14.139+0000 7fca7359e700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fca7409f830 0x7fca7414efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:17:14.154 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:14.139+0000 7fca7259c700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fca7408df80 0x7fca7409f2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:52640/0 (socket says 172.21.15.42:52640) 2019-12-05T06:17:14.154 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:14.139+0000 7fca7259c700 1 -- 172.21.15.42:0/1993398895 learned_addr learned my addr 172.21.15.42:0/1993398895 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:17:14.154 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:14.139+0000 7fca7359e700 1 -- 172.21.15.42:0/1993398895 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fca7408df80 msgr2=0x7fca7409f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:17:14.154 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:14.139+0000 7fca7359e700 1 --2- 172.21.15.42:0/1993398895 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fca7408df80 0x7fca7409f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:17:14.155 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:14.139+0000 7fca7359e700 1 -- 172.21.15.42:0/1993398895 >> v1:172.21.15.40:6789/0 conn(0x7fca740a7740 legacy=0x7fca7407e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:17:14.155 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:14.139+0000 7fca7359e700 1 -- 172.21.15.42:0/1993398895 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca740a0710 con 0x7fca7409f830 2019-12-05T06:17:14.155 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:14.139+0000 7fca7359e700 1 --2- 172.21.15.42:0/1993398895 >> v2:172.21.15.40:3300/0 conn(0x7fca7409f830 0x7fca7414efe0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fca6c008b10 tx=0x7fca6c00fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:14.155 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:14.139+0000 7fca7359e700 1 --2- 172.21.15.42:0/1993398895 >> v2:172.21.15.40:3300/0 conn(0x7fca7409f830 0x7fca7414efe0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fca6c008b10 tx=0x7fca6c00fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:14.156 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:14.139+0000 7fca7359e700 1 --2- 172.21.15.42:0/1993398895 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fca7409f830 0x7fca7414efe0 secure :-1 s=READY pgs=1816 cs=0 l=1 rx=0x7fca6c008b10 tx=0x7fca6c00fff0).ready entity=mon.0 client_cookie=8d2890308f84dbdb server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:17:14.156 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:14.139+0000 7fca7359e700 1 --2- 172.21.15.42:0/1993398895 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fca7409f830 0x7fca7414efe0 secure :-1 s=THROTTLE_DONE pgs=1816 cs=0 l=1 rx=0x7fca6c008b10 tx=0x7fca6c00fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:14.169 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:14.139+0000 7fca71d9b700 1 -- 172.21.15.42:0/1993398895 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fca6c010900 con 0x7fca7409f830 2019-12-05T06:17:14.170 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:14.139+0000 7fca7359e700 1 --2- 172.21.15.42:0/1993398895 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fca7409f830 0x7fca7414efe0 secure :-1 s=THROTTLE_DONE pgs=1816 cs=0 l=1 rx=0x7fca6c008b10 tx=0x7fca6c00fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:14.170 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:14.139+0000 7fca71d9b700 1 -- 172.21.15.42:0/1993398895 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca6c008b50 con 0x7fca7409f830 2019-12-05T06:17:14.170 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:14.139+0000 7fca7359e700 1 --2- 172.21.15.42:0/1993398895 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fca7409f830 0x7fca7414efe0 secure :-1 s=THROTTLE_DONE pgs=1816 cs=0 l=1 rx=0x7fca6c008b10 tx=0x7fca6c00fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:14.170 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:14.139+0000 7fca71d9b700 1 -- 172.21.15.42:0/1993398895 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fca6c0110c0 con 0x7fca7409f830 2019-12-05T06:17:14.170 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:14.139+0000 7fca78bfa700 1 -- 172.21.15.42:0/1993398895 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fca7409f830 msgr2=0x7fca7414efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:17:14.170 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:14.139+0000 7fca78bfa700 1 --2- 172.21.15.42:0/1993398895 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fca7409f830 0x7fca7414efe0 secure :-1 s=READY pgs=1816 cs=0 l=1 rx=0x7fca6c008b10 tx=0x7fca6c00fff0).stop 2019-12-05T06:17:14.171 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:14.139+0000 7fca78bfa700 1 -- 172.21.15.42:0/1993398895 shutdown_connections 2019-12-05T06:17:14.171 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:14.139+0000 7fca78bfa700 1 --2- 172.21.15.42:0/1993398895 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fca7409f830 0x7fca7414efe0 unknown :-1 s=CLOSED pgs=1816 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:17:14.171 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:14.139+0000 7fca78bfa700 1 --2- 172.21.15.42:0/1993398895 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fca7408df80 0x7fca7409f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:17:14.171 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:14.139+0000 7fca78bfa700 1 -- 172.21.15.42:0/1993398895 shutdown_connections 2019-12-05T06:17:14.171 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:14.139+0000 7fca78bfa700 1 -- 172.21.15.42:0/1993398895 wait complete. 2019-12-05T06:17:14.175 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:14.139+0000 7fca78bfa700 1 -- 172.21.15.42:0/1993398895 >> 172.21.15.42:0/1993398895 conn(0x7fca740aabe0 msgr2=0x7fca740a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:17:14.176 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:14.143+0000 7fca78bfa700 1 Processor -- start 2019-12-05T06:17:14.176 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:14.143+0000 7fca78bfa700 1 -- start start 2019-12-05T06:17:14.176 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:14.143+0000 7fca78bfa700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fca74122ad0 0x7fca74122250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:17:14.176 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:14.143+0000 7fca78bfa700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fca740702c0 0x7fca740706f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:17:14.176 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:14.143+0000 7fca78bfa700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca741166a0 con 0x7fca7411ac90 2019-12-05T06:17:14.176 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:14.143+0000 7fca78bfa700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca7411a800 con 0x7fca74122ad0 2019-12-05T06:17:14.177 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:14.143+0000 7fca78bfa700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca7411b4c0 con 0x7fca740702c0 2019-12-05T06:17:14.177 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:14.143+0000 7fca72d9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fca74122ad0 0x7fca74122250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:17:14.177 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:14.143+0000 7fca7259c700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fca7411ac90 0x7fca7411bca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:52218/0 (socket says 172.21.15.42:52218) 2019-12-05T06:17:14.177 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:14.143+0000 7fca7259c700 1 -- 172.21.15.42:0/2261047413 learned_addr learned my addr 172.21.15.42:0/2261047413 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:17:14.177 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:14.143+0000 7fca7359e700 1 --2- 172.21.15.42:0/2261047413 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fca740702c0 0x7fca740706f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:17:14.178 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:14.143+0000 7fca6b7fe700 1 -- 172.21.15.42:0/2261047413 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1424236042 0 0) 0x7fca741166a0 con 0x7fca7411ac90 2019-12-05T06:17:14.178 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:14.143+0000 7fca72d9d700 1 -- 172.21.15.42:0/2261047413 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fca740702c0 msgr2=0x7fca740706f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:17:14.179 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:14.143+0000 7fca72d9d700 1 --2- 172.21.15.42:0/2261047413 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fca740702c0 0x7fca740706f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:17:14.179 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:14.143+0000 7fca72d9d700 1 -- 172.21.15.42:0/2261047413 >> v1:172.21.15.42:6789/0 conn(0x7fca7411ac90 legacy=0x7fca7411bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:17:14.188 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:14.143+0000 7fca72d9d700 1 -- 172.21.15.42:0/2261047413 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca7411b0c0 con 0x7fca74122ad0 2019-12-05T06:17:14.188 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:14.143+0000 7fca72d9d700 1 --2- 172.21.15.42:0/2261047413 >> v2:172.21.15.42:3300/0 conn(0x7fca74122ad0 0x7fca74122250 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fca5c000e90 tx=0x7fca5c0108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:14.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:14.143+0000 7fca72d9d700 1 --2- 172.21.15.42:0/2261047413 >> v2:172.21.15.42:3300/0 conn(0x7fca74122ad0 0x7fca74122250 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fca5c000e90 tx=0x7fca5c0108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:14.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:14.143+0000 7fca72d9d700 1 --2- 172.21.15.42:0/2261047413 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fca74122ad0 0x7fca74122250 secure :-1 s=READY pgs=2388 cs=0 l=1 rx=0x7fca5c000e90 tx=0x7fca5c0108f0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:17:14.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:14.143+0000 7fca72d9d700 1 --2- 172.21.15.42:0/2261047413 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fca74122ad0 0x7fca74122250 secure :-1 s=THROTTLE_DONE pgs=2388 cs=0 l=1 rx=0x7fca5c000e90 tx=0x7fca5c0108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:14.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:14.143+0000 7fca6b7fe700 1 -- 172.21.15.42:0/2261047413 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fca5c00ca80 con 0x7fca74122ad0 2019-12-05T06:17:14.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:14.143+0000 7fca72d9d700 1 --2- 172.21.15.42:0/2261047413 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fca74122ad0 0x7fca74122250 secure :-1 s=THROTTLE_DONE pgs=2388 cs=0 l=1 rx=0x7fca5c000e90 tx=0x7fca5c0108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:14.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:14.143+0000 7fca78bfa700 1 -- 172.21.15.42:0/2261047413 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca74070c40 con 0x7fca74122ad0 2019-12-05T06:17:14.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:14.143+0000 7fca6b7fe700 1 -- 172.21.15.42:0/2261047413 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca5c00c6a0 con 0x7fca74122ad0 2019-12-05T06:17:14.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:14.143+0000 7fca72d9d700 1 --2- 172.21.15.42:0/2261047413 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fca74122ad0 0x7fca74122250 secure :-1 s=THROTTLE_DONE pgs=2388 cs=0 l=1 rx=0x7fca5c000e90 tx=0x7fca5c0108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:14.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:14.143+0000 7fca6b7fe700 1 -- 172.21.15.42:0/2261047413 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fca5c0110f0 con 0x7fca74122ad0 2019-12-05T06:17:14.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:14.143+0000 7fca78bfa700 1 -- 172.21.15.42:0/2261047413 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca74077530 con 0x7fca74122ad0 2019-12-05T06:17:14.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:14.147+0000 7fca72d9d700 1 --2- 172.21.15.42:0/2261047413 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fca74122ad0 0x7fca74122250 secure :-1 s=THROTTLE_DONE pgs=2388 cs=0 l=1 rx=0x7fca5c000e90 tx=0x7fca5c0108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:14.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:14.147+0000 7fca6b7fe700 1 -- 172.21.15.42:0/2261047413 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fca5c0113d0 con 0x7fca74122ad0 2019-12-05T06:17:14.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:14.147+0000 7fca72d9d700 1 --2- 172.21.15.42:0/2261047413 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fca74122ad0 0x7fca74122250 secure :-1 s=THROTTLE_DONE pgs=2388 cs=0 l=1 rx=0x7fca5c000e90 tx=0x7fca5c0108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:14.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:14.147+0000 7fca6b7fe700 1 --2- 172.21.15.42:0/2261047413 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fca5803acc0 0x7fca5803d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:17:14.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:14.147+0000 7fca6b7fe700 1 -- 172.21.15.42:0/2261047413 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 10417+0+0 (secure 0 0 0) 0x7fca5c04b140 con 0x7fca74122ad0 2019-12-05T06:17:14.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:14.147+0000 7fca7259c700 1 --2- 172.21.15.42:0/2261047413 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fca5803acc0 0x7fca5803d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:17:14.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:14.147+0000 7fca7259c700 1 --2- 172.21.15.42:0/2261047413 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fca5803acc0 0x7fca5803d1d0 crc :-1 s=READY pgs=1004 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:17:14.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:14.151+0000 7fca78bfa700 1 -- 172.21.15.42:0/2261047413 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca7405d110 con 0x7fca74122ad0 2019-12-05T06:17:14.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:14.159+0000 7fca72d9d700 1 --2- 172.21.15.42:0/2261047413 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fca74122ad0 0x7fca74122250 secure :-1 s=THROTTLE_DONE pgs=2388 cs=0 l=1 rx=0x7fca5c000e90 tx=0x7fca5c0108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:14.193 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:14.159+0000 7fca6b7fe700 1 -- 172.21.15.42:0/2261047413 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fca5c0321a0 con 0x7fca74122ad0 2019-12-05T06:17:14.333 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:14.327+0000 7fca78bfa700 1 -- 172.21.15.42:0/2261047413 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fca7405b8c0 con 0x7fca74122ad0 2019-12-05T06:17:14.334 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:14.327+0000 7fca72d9d700 1 --2- 172.21.15.42:0/2261047413 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fca74122ad0 0x7fca74122250 secure :-1 s=THROTTLE_DONE pgs=2388 cs=0 l=1 rx=0x7fca5c000e90 tx=0x7fca5c0108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:14.335 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:14.327+0000 7fca6b7fe700 1 -- 172.21.15.42:0/2261047413 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 288 v288) v1 ==== 95+0+2114 (secure 0 0 0) 0x7fca5c032400 con 0x7fca74122ad0 2019-12-05T06:17:14.335 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 288 2019-12-05T06:17:14.335 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:17:14.335 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":288,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":10476,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/1473620421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1473620421},{"type":"v1","addr":"172.21.15.40:6833","nonce":1473620421}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":288}],"filesystems":[{"mdsmap":{"epoch":287,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:17:04.753930+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":100,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7295},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7295":{"gid":7295,"name":"a","rank":0,"incarnation":284,"state":"up:active","state_seq":38,"addr":"172.21.15.42:6835/3228549071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3228549071},{"type":"v1","addr":"172.21.15.42:6835","nonce":3228549071}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:17:14.337 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:14.331+0000 7fca78bfa700 1 -- 172.21.15.42:0/2261047413 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fca5803acc0 msgr2=0x7fca5803d1d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:17:14.337 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:14.331+0000 7fca78bfa700 1 --2- 172.21.15.42:0/2261047413 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fca5803acc0 0x7fca5803d1d0 crc :-1 s=READY pgs=1004 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:17:14.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:14.331+0000 7fca78bfa700 1 -- 172.21.15.42:0/2261047413 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fca74122ad0 msgr2=0x7fca74122250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:17:14.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:14.331+0000 7fca78bfa700 1 --2- 172.21.15.42:0/2261047413 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fca74122ad0 0x7fca74122250 secure :-1 s=READY pgs=2388 cs=0 l=1 rx=0x7fca5c000e90 tx=0x7fca5c0108f0).stop 2019-12-05T06:17:14.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:14.331+0000 7fca78bfa700 1 -- 172.21.15.42:0/2261047413 shutdown_connections 2019-12-05T06:17:14.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:14.331+0000 7fca78bfa700 1 --2- 172.21.15.42:0/2261047413 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fca5803acc0 0x7fca5803d1d0 unknown :-1 s=CLOSED pgs=1004 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:17:14.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:14.331+0000 7fca78bfa700 1 --2- 172.21.15.42:0/2261047413 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fca740702c0 0x7fca740706f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:17:14.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:14.331+0000 7fca78bfa700 1 --2- 172.21.15.42:0/2261047413 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fca74122ad0 0x7fca74122250 unknown :-1 s=CLOSED pgs=2388 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:17:14.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:14.331+0000 7fca78bfa700 1 -- 172.21.15.42:0/2261047413 shutdown_connections 2019-12-05T06:17:14.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:14.331+0000 7fca78bfa700 1 -- 172.21.15.42:0/2261047413 wait complete. 2019-12-05T06:17:14.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:17:14.331+0000 7fca78bfa700 1 -- 172.21.15.42:0/2261047413 >> 172.21.15.42:0/2261047413 conn(0x7fca740a9c20 msgr2=0x7fca7409f1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:17:14.347 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b reported in up:standby state 2019-12-05T06:17:14.347 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 98 secs before thrashing 2019-12-05T06:17:15.474 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:17:15.478 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:17:45.541 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:17:45.544 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:18:15.588 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:18:15.593 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:18:45.666 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:18:45.675 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:18:52.349 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:18:52.464 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:52.450+0000 7fd903be4700 1 Processor -- start 2019-12-05T06:18:52.464 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:52.450+0000 7fd903be4700 1 -- start start 2019-12-05T06:18:52.464 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:52.450+0000 7fd903be4700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd8fc08df80 0x7fd8fc09f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:18:52.464 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:52.450+0000 7fd903be4700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8fc09fc60 con 0x7fd8fc0a7740 2019-12-05T06:18:52.465 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:52.450+0000 7fd903be4700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8fc0a0710 con 0x7fd8fc09f830 2019-12-05T06:18:52.465 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:52.450+0000 7fd903be4700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8fc0a0b00 con 0x7fd8fc08df80 2019-12-05T06:18:52.465 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:52.454+0000 7fd902982700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fd8fc09f830 0x7fd8fc14efe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:52632/0 (socket says 172.21.15.42:52632) 2019-12-05T06:18:52.465 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:52.454+0000 7fd902982700 1 -- 172.21.15.42:0/1542185420 learned_addr learned my addr 172.21.15.42:0/1542185420 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:18:52.465 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:52.454+0000 7fd902181700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fd8fc0a7740 0x7fd8fc07e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:33836/0 (socket says 172.21.15.42:33836) 2019-12-05T06:18:52.465 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:52.454+0000 7fd901980700 1 --2- 172.21.15.42:0/1542185420 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd8fc08df80 0x7fd8fc09f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:18:52.465 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:52.454+0000 7fd90117f700 1 -- 172.21.15.42:0/1542185420 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3429848326 0 0) 0x7fd8fc0a0710 con 0x7fd8fc09f830 2019-12-05T06:18:52.466 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:52.454+0000 7fd90117f700 1 -- 172.21.15.42:0/1542185420 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8e40030b0 con 0x7fd8fc09f830 2019-12-05T06:18:52.466 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:52.454+0000 7fd90117f700 1 -- 172.21.15.42:0/1542185420 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 605439625 0 0) 0x7fd8fc09fc60 con 0x7fd8fc0a7740 2019-12-05T06:18:52.466 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:52.454+0000 7fd90117f700 1 -- 172.21.15.42:0/1542185420 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8fc0a0710 con 0x7fd8fc0a7740 2019-12-05T06:18:52.466 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:52.454+0000 7fd901980700 1 -- 172.21.15.42:0/1542185420 >> v1:172.21.15.42:6789/0 conn(0x7fd8fc09f830 legacy=0x7fd8fc14efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:18:52.466 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:52.454+0000 7fd901980700 1 -- 172.21.15.42:0/1542185420 >> v1:172.21.15.40:6789/0 conn(0x7fd8fc0a7740 legacy=0x7fd8fc07e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:18:52.466 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:52.454+0000 7fd901980700 1 -- 172.21.15.42:0/1542185420 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8fc07ef20 con 0x7fd8fc08df80 2019-12-05T06:18:52.466 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:52.454+0000 7fd901980700 1 --2- 172.21.15.42:0/1542185420 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd8fc08df80 0x7fd8fc09f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fd8f0002f30 tx=0x7fd8f000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:18:52.467 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:52.454+0000 7fd901980700 1 --2- 172.21.15.42:0/1542185420 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd8fc08df80 0x7fd8fc09f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fd8f0002f30 tx=0x7fd8f000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:18:52.467 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:52.454+0000 7fd901980700 1 --2- 172.21.15.42:0/1542185420 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd8fc08df80 0x7fd8fc09f2e0 secure :-1 s=READY pgs=530 cs=0 l=1 rx=0x7fd8f0002f30 tx=0x7fd8f000fbf0).ready entity=mon.2 client_cookie=4baeefad653864e2 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:18:52.467 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:52.454+0000 7fd901980700 1 --2- 172.21.15.42:0/1542185420 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd8fc08df80 0x7fd8fc09f2e0 secure :-1 s=THROTTLE_DONE pgs=530 cs=0 l=1 rx=0x7fd8f0002f30 tx=0x7fd8f000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:18:52.468 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:52.454+0000 7fd90117f700 1 -- 172.21.15.42:0/1542185420 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd8f00112b0 con 0x7fd8fc08df80 2019-12-05T06:18:52.468 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:52.454+0000 7fd901980700 1 --2- 172.21.15.42:0/1542185420 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd8fc08df80 0x7fd8fc09f2e0 secure :-1 s=THROTTLE_DONE pgs=530 cs=0 l=1 rx=0x7fd8f0002f30 tx=0x7fd8f000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:18:52.468 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:52.454+0000 7fd90117f700 1 -- 172.21.15.42:0/1542185420 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8f0008bc0 con 0x7fd8fc08df80 2019-12-05T06:18:52.468 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:52.454+0000 7fd901980700 1 --2- 172.21.15.42:0/1542185420 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd8fc08df80 0x7fd8fc09f2e0 secure :-1 s=THROTTLE_DONE pgs=530 cs=0 l=1 rx=0x7fd8f0002f30 tx=0x7fd8f000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:18:52.468 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:52.454+0000 7fd90117f700 1 -- 172.21.15.42:0/1542185420 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd8f0011890 con 0x7fd8fc08df80 2019-12-05T06:18:52.469 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:52.454+0000 7fd903be4700 1 -- 172.21.15.42:0/1542185420 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd8fc08df80 msgr2=0x7fd8fc09f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:18:52.469 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:52.454+0000 7fd903be4700 1 --2- 172.21.15.42:0/1542185420 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd8fc08df80 0x7fd8fc09f2e0 secure :-1 s=READY pgs=530 cs=0 l=1 rx=0x7fd8f0002f30 tx=0x7fd8f000fbf0).stop 2019-12-05T06:18:52.469 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:52.454+0000 7fd903be4700 1 -- 172.21.15.42:0/1542185420 shutdown_connections 2019-12-05T06:18:52.469 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:52.454+0000 7fd903be4700 1 --2- 172.21.15.42:0/1542185420 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd8fc08df80 0x7fd8fc09f2e0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:18:52.469 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:52.454+0000 7fd903be4700 1 -- 172.21.15.42:0/1542185420 shutdown_connections 2019-12-05T06:18:52.469 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:52.454+0000 7fd903be4700 1 -- 172.21.15.42:0/1542185420 wait complete. 2019-12-05T06:18:52.469 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:52.454+0000 7fd903be4700 1 -- 172.21.15.42:0/1542185420 >> 172.21.15.42:0/1542185420 conn(0x7fd8fc0aabe0 msgr2=0x7fd8fc0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:18:52.490 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:52.458+0000 7fd903be4700 1 Processor -- start 2019-12-05T06:18:53.116 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:52.458+0000 7fd903be4700 1 -- start start 2019-12-05T06:18:53.116 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:52.458+0000 7fd903be4700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd8fc116500 0x7fd8fc143580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:18:53.117 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:52.458+0000 7fd903be4700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd8fc0a6270 0x7fd8fc133850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:18:53.117 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:52.458+0000 7fd903be4700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8fc133da0 con 0x7fd8fc0a6ca0 2019-12-05T06:18:53.117 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:52.458+0000 7fd903be4700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8fc142760 con 0x7fd8fc0a6270 2019-12-05T06:18:53.117 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:52.458+0000 7fd903be4700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8fc143cc0 con 0x7fd8fc116500 2019-12-05T06:18:53.117 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:52.458+0000 7fd902982700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd8fc0a6270 0x7fd8fc133850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:18:53.118 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:52.458+0000 7fd902181700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd8fc116500 0x7fd8fc143580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:18:53.118 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:52.458+0000 7fd902982700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd8fc0a6270 0x7fd8fc133850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:52600/0 (socket says 172.21.15.42:52600) 2019-12-05T06:18:53.118 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:52.458+0000 7fd902982700 1 -- 172.21.15.42:0/3863789834 learned_addr learned my addr 172.21.15.42:0/3863789834 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:18:53.118 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:52.458+0000 7fd901980700 1 --1- 172.21.15.42:0/3863789834 >> v1:172.21.15.40:6789/0 conn(0x7fd8fc0a6ca0 0x7fd8fc0a5a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:33844/0 (socket says 172.21.15.42:33844) 2019-12-05T06:18:53.118 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:52.462+0000 7fd902982700 1 -- 172.21.15.42:0/3863789834 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd8fc116500 msgr2=0x7fd8fc143580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:18:53.121 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:52.462+0000 7fd902982700 1 --2- 172.21.15.42:0/3863789834 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd8fc116500 0x7fd8fc143580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:18:53.122 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:52.462+0000 7fd902982700 1 -- 172.21.15.42:0/3863789834 >> v1:172.21.15.40:6789/0 conn(0x7fd8fc0a6ca0 legacy=0x7fd8fc0a5a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:18:53.122 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:52.462+0000 7fd902982700 1 -- 172.21.15.42:0/3863789834 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8fc133fd0 con 0x7fd8fc0a6270 2019-12-05T06:18:53.122 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:52.462+0000 7fd902982700 1 --2- 172.21.15.42:0/3863789834 >> v2:172.21.15.42:3300/0 conn(0x7fd8fc0a6270 0x7fd8fc133850 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fd8fc143e80 tx=0x7fd8f4010860).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:18:53.122 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:52.462+0000 7fd902982700 1 --2- 172.21.15.42:0/3863789834 >> v2:172.21.15.42:3300/0 conn(0x7fd8fc0a6270 0x7fd8fc133850 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fd8fc143e80 tx=0x7fd8f4010860).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:18:53.122 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:52.462+0000 7fd902982700 1 --2- 172.21.15.42:0/3863789834 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd8fc0a6270 0x7fd8fc133850 secure :-1 s=READY pgs=2391 cs=0 l=1 rx=0x7fd8fc143e80 tx=0x7fd8f4010860).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:18:53.122 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:52.462+0000 7fd902982700 1 --2- 172.21.15.42:0/3863789834 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd8fc0a6270 0x7fd8fc133850 secure :-1 s=THROTTLE_DONE pgs=2391 cs=0 l=1 rx=0x7fd8fc143e80 tx=0x7fd8f4010860).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:18:53.123 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:52.462+0000 7fd902982700 1 --2- 172.21.15.42:0/3863789834 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd8fc0a6270 0x7fd8fc133850 secure :-1 s=THROTTLE_DONE pgs=2391 cs=0 l=1 rx=0x7fd8fc143e80 tx=0x7fd8f4010860).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:18:53.123 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:52.462+0000 7fd902982700 1 --2- 172.21.15.42:0/3863789834 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd8fc0a6270 0x7fd8fc133850 secure :-1 s=THROTTLE_DONE pgs=2391 cs=0 l=1 rx=0x7fd8fc143e80 tx=0x7fd8f4010860).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:18:53.123 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:52.462+0000 7fd8faffd700 1 -- 172.21.15.42:0/3863789834 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd8f4005b40 con 0x7fd8fc0a6270 2019-12-05T06:18:53.123 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:52.462+0000 7fd8faffd700 1 -- 172.21.15.42:0/3863789834 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8f4005760 con 0x7fd8fc0a6270 2019-12-05T06:18:53.123 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:52.462+0000 7fd903be4700 1 -- 172.21.15.42:0/3863789834 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8fc134240 con 0x7fd8fc0a6270 2019-12-05T06:18:53.124 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:52.462+0000 7fd8faffd700 1 -- 172.21.15.42:0/3863789834 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd8f40062b0 con 0x7fd8fc0a6270 2019-12-05T06:18:53.124 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:52.462+0000 7fd903be4700 1 -- 172.21.15.42:0/3863789834 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8fc0762f0 con 0x7fd8fc0a6270 2019-12-05T06:18:53.124 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:52.462+0000 7fd902982700 1 --2- 172.21.15.42:0/3863789834 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd8fc0a6270 0x7fd8fc133850 secure :-1 s=THROTTLE_DONE pgs=2391 cs=0 l=1 rx=0x7fd8fc143e80 tx=0x7fd8f4010860).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:18:53.124 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:52.462+0000 7fd8faffd700 1 -- 172.21.15.42:0/3863789834 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fd8f40065c0 con 0x7fd8fc0a6270 2019-12-05T06:18:53.124 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:52.462+0000 7fd902982700 1 --2- 172.21.15.42:0/3863789834 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd8fc0a6270 0x7fd8fc133850 secure :-1 s=THROTTLE_DONE pgs=2391 cs=0 l=1 rx=0x7fd8fc143e80 tx=0x7fd8f4010860).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:18:53.125 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:52.462+0000 7fd8faffd700 1 --2- 172.21.15.42:0/3863789834 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd8dc03ada0 0x7fd8dc03d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:18:53.125 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:52.462+0000 7fd8faffd700 1 -- 172.21.15.42:0/3863789834 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 10417+0+0 (secure 0 0 0) 0x7fd8f404af70 con 0x7fd8fc0a6270 2019-12-05T06:18:53.125 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:52.462+0000 7fd902181700 1 --2- 172.21.15.42:0/3863789834 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd8dc03ada0 0x7fd8dc03d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:18:53.125 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:52.462+0000 7fd902181700 1 --2- 172.21.15.42:0/3863789834 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd8dc03ada0 0x7fd8dc03d2b0 crc :-1 s=READY pgs=1008 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:18:53.125 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:52.466+0000 7fd903be4700 1 -- 172.21.15.42:0/3863789834 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8fc05d110 con 0x7fd8fc0a6270 2019-12-05T06:18:53.126 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:52.474+0000 7fd902982700 1 --2- 172.21.15.42:0/3863789834 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd8fc0a6270 0x7fd8fc133850 secure :-1 s=THROTTLE_DONE pgs=2391 cs=0 l=1 rx=0x7fd8fc143e80 tx=0x7fd8f4010860).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:18:53.126 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:52.474+0000 7fd8faffd700 1 -- 172.21.15.42:0/3863789834 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fd8f4031ff0 con 0x7fd8fc0a6270 2019-12-05T06:18:53.128 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:52.662+0000 7fd903be4700 1 -- 172.21.15.42:0/3863789834 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd8fc05b8c0 con 0x7fd8fc0a6270 2019-12-05T06:18:53.129 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:52.662+0000 7fd902982700 1 --2- 172.21.15.42:0/3863789834 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd8fc0a6270 0x7fd8fc133850 secure :-1 s=THROTTLE_DONE pgs=2391 cs=0 l=1 rx=0x7fd8fc143e80 tx=0x7fd8f4010860).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:18:53.129 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:52.662+0000 7fd8faffd700 1 -- 172.21.15.42:0/3863789834 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 288 v288) v1 ==== 95+0+2114 (secure 0 0 0) 0x7fd8f4032250 con 0x7fd8fc0a6270 2019-12-05T06:18:53.129 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 288 2019-12-05T06:18:53.129 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:52.666+0000 7fd903be4700 1 -- 172.21.15.42:0/3863789834 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd8dc03ada0 msgr2=0x7fd8dc03d2b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:18:53.129 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:52.666+0000 7fd903be4700 1 --2- 172.21.15.42:0/3863789834 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd8dc03ada0 0x7fd8dc03d2b0 crc :-1 s=READY pgs=1008 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:18:53.129 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:52.666+0000 7fd903be4700 1 -- 172.21.15.42:0/3863789834 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd8fc0a6270 msgr2=0x7fd8fc133850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:18:53.129 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:52.666+0000 7fd903be4700 1 --2- 172.21.15.42:0/3863789834 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd8fc0a6270 0x7fd8fc133850 secure :-1 s=READY pgs=2391 cs=0 l=1 rx=0x7fd8fc143e80 tx=0x7fd8f4010860).stop 2019-12-05T06:18:53.130 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:52.666+0000 7fd903be4700 1 -- 172.21.15.42:0/3863789834 shutdown_connections 2019-12-05T06:18:53.130 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:52.666+0000 7fd903be4700 1 --2- 172.21.15.42:0/3863789834 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd8dc03ada0 0x7fd8dc03d2b0 unknown :-1 s=CLOSED pgs=1008 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:18:53.130 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:52.666+0000 7fd903be4700 1 --2- 172.21.15.42:0/3863789834 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd8fc0a6270 0x7fd8fc133850 unknown :-1 s=CLOSED pgs=2391 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:18:53.130 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:52.666+0000 7fd903be4700 1 --2- 172.21.15.42:0/3863789834 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd8fc116500 0x7fd8fc143580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:18:53.130 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:52.666+0000 7fd903be4700 1 -- 172.21.15.42:0/3863789834 shutdown_connections 2019-12-05T06:18:53.130 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:52.666+0000 7fd903be4700 1 -- 172.21.15.42:0/3863789834 wait complete. 2019-12-05T06:18:53.130 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:52.666+0000 7fd903be4700 1 -- 172.21.15.42:0/3863789834 >> 172.21.15.42:0/3863789834 conn(0x7fd8fc07e9a0 msgr2=0x7fd8fc059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:18:53.131 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:18:53.131 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":288,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":10476,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/1473620421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1473620421},{"type":"v1","addr":"172.21.15.40:6833","nonce":1473620421}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":288}],"filesystems":[{"mdsmap":{"epoch":287,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:17:04.753930+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":100,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7295},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7295":{"gid":7295,"name":"a","rank":0,"incarnation":284,"state":"up:active","state_seq":38,"addr":"172.21.15.42:6835/3228549071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3228549071},{"type":"v1","addr":"172.21.15.42:6835","nonce":3228549071}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:18:53.132 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.a (rank=0) 2019-12-05T06:18:53.132 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2019-12-05T06:18:53.132 INFO:teuthology.orchestra.run:waiting for 300 2019-12-05T06:18:53.154 INFO:tasks.ceph.mds.a:Stopped 2019-12-05T06:18:53.155 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:18:53.341 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:53.298+0000 7f038ed57700 1 Processor -- start 2019-12-05T06:18:53.341 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:53.298+0000 7f038ed57700 1 -- start start 2019-12-05T06:18:53.341 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:53.298+0000 7f038ed57700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f03880a7740 0x7f038808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:18:53.341 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:53.298+0000 7f038ed57700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03880a0710 con 0x7f038807e720 2019-12-05T06:18:53.341 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:53.298+0000 7f038ed57700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03880a9ba0 con 0x7f038809fae0 2019-12-05T06:18:53.342 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:53.298+0000 7f038ed57700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03880a0b00 con 0x7f03880a7740 2019-12-05T06:18:53.342 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:53.298+0000 7f038d2f4700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f03880a7740 0x7f038808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:18:53.342 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:53.298+0000 7f038daf5700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f038809fae0 0x7f038814efe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:52646/0 (socket says 172.21.15.42:52646) 2019-12-05T06:18:53.342 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:53.298+0000 7f038daf5700 1 -- 172.21.15.42:0/2992617592 learned_addr learned my addr 172.21.15.42:0/2992617592 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:18:53.342 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:53.298+0000 7f0387fff700 1 -- 172.21.15.42:0/2992617592 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4245928991 0 0) 0x7f03880a9ba0 con 0x7f038809fae0 2019-12-05T06:18:53.342 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:53.298+0000 7f0387fff700 1 -- 172.21.15.42:0/2992617592 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03700030b0 con 0x7f038809fae0 2019-12-05T06:18:53.342 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:53.298+0000 7f038d2f4700 1 -- 172.21.15.42:0/2992617592 >> v1:172.21.15.42:6789/0 conn(0x7f038809fae0 legacy=0x7f038814efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:18:53.343 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:53.298+0000 7f038d2f4700 1 -- 172.21.15.42:0/2992617592 >> v1:172.21.15.40:6789/0 conn(0x7f038807e720 legacy=0x7f038809f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:18:53.343 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:53.298+0000 7f0387fff700 1 -- 172.21.15.42:0/2992617592 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4131678306 0 0) 0x7f03880a0710 con 0x7f038807e720 2019-12-05T06:18:53.343 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:53.298+0000 7f038d2f4700 1 -- 172.21.15.42:0/2992617592 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f038807eb50 con 0x7f03880a7740 2019-12-05T06:18:53.343 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:53.298+0000 7f038d2f4700 1 --2- 172.21.15.42:0/2992617592 >> v2:172.21.15.42:3300/0 conn(0x7f03880a7740 0x7f038808df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f0380008790 tx=0x7f038000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:18:53.343 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:53.298+0000 7f038d2f4700 1 --2- 172.21.15.42:0/2992617592 >> v2:172.21.15.42:3300/0 conn(0x7f03880a7740 0x7f038808df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f0380008790 tx=0x7f038000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:18:53.343 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:53.298+0000 7f038d2f4700 1 --2- 172.21.15.42:0/2992617592 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f03880a7740 0x7f038808df80 secure :-1 s=READY pgs=2393 cs=0 l=1 rx=0x7f0380008790 tx=0x7f038000fbf0).ready entity=mon.1 client_cookie=edd5e72db524e1bf server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:18:53.343 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:53.298+0000 7f038d2f4700 1 --2- 172.21.15.42:0/2992617592 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f03880a7740 0x7f038808df80 secure :-1 s=THROTTLE_DONE pgs=2393 cs=0 l=1 rx=0x7f0380008790 tx=0x7f038000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:18:53.344 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:53.298+0000 7f0387fff700 1 -- 172.21.15.42:0/2992617592 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0380011270 con 0x7f03880a7740 2019-12-05T06:18:53.344 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:53.298+0000 7f038d2f4700 1 --2- 172.21.15.42:0/2992617592 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f03880a7740 0x7f038808df80 secure :-1 s=THROTTLE_DONE pgs=2393 cs=0 l=1 rx=0x7f0380008790 tx=0x7f038000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:18:53.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:53.298+0000 7f0387fff700 1 -- 172.21.15.42:0/2992617592 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0380008bc0 con 0x7f03880a7740 2019-12-05T06:18:53.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:53.298+0000 7f038d2f4700 1 --2- 172.21.15.42:0/2992617592 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f03880a7740 0x7f038808df80 secure :-1 s=THROTTLE_DONE pgs=2393 cs=0 l=1 rx=0x7f0380008790 tx=0x7f038000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:18:53.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:53.298+0000 7f0387fff700 1 -- 172.21.15.42:0/2992617592 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0380011850 con 0x7f03880a7740 2019-12-05T06:18:53.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:53.298+0000 7f038ed57700 1 -- 172.21.15.42:0/2992617592 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f03880a7740 msgr2=0x7f038808df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:18:53.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:53.298+0000 7f038ed57700 1 --2- 172.21.15.42:0/2992617592 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f03880a7740 0x7f038808df80 secure :-1 s=READY pgs=2393 cs=0 l=1 rx=0x7f0380008790 tx=0x7f038000fbf0).stop 2019-12-05T06:18:53.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:53.302+0000 7f038ed57700 1 -- 172.21.15.42:0/2992617592 shutdown_connections 2019-12-05T06:18:53.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:53.302+0000 7f038ed57700 1 --2- 172.21.15.42:0/2992617592 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f03880a7740 0x7f038808df80 unknown :-1 s=CLOSED pgs=2393 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:18:53.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:53.302+0000 7f038ed57700 1 -- 172.21.15.42:0/2992617592 shutdown_connections 2019-12-05T06:18:53.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:53.302+0000 7f038ed57700 1 -- 172.21.15.42:0/2992617592 wait complete. 2019-12-05T06:18:53.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:53.302+0000 7f038ed57700 1 -- 172.21.15.42:0/2992617592 >> 172.21.15.42:0/2992617592 conn(0x7f03880aabe0 msgr2=0x7f03880a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:18:53.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:53.302+0000 7f038ed57700 1 Processor -- start 2019-12-05T06:18:53.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:53.302+0000 7f038ed57700 1 -- start start 2019-12-05T06:18:53.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:53.306+0000 7f038ed57700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f038807d9f0 0x7f038811b270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:18:53.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:53.306+0000 7f038ed57700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0388123a60 0x7f038811d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:18:53.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:53.306+0000 7f038ed57700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f038811deb0 con 0x7f0388123e90 2019-12-05T06:18:53.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:53.306+0000 7f038ed57700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f038811b930 con 0x7f0388123a60 2019-12-05T06:18:53.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:53.306+0000 7f038ed57700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f038811bbd0 con 0x7f038807d9f0 2019-12-05T06:18:53.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:53.306+0000 7f038caf3700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0388123a60 0x7f038811d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:18:53.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:53.306+0000 7f038caf3700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0388123a60 0x7f038811d960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:52614/0 (socket says 172.21.15.42:52614) 2019-12-05T06:18:53.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:53.306+0000 7f038caf3700 1 -- 172.21.15.42:0/1165801292 learned_addr learned my addr 172.21.15.42:0/1165801292 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:18:53.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:53.306+0000 7f038d2f4700 1 --2- 172.21.15.42:0/1165801292 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f038807d9f0 0x7f038811b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:18:53.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:53.306+0000 7f038caf3700 1 -- 172.21.15.42:0/1165801292 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f038807d9f0 msgr2=0x7f038811b270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:18:53.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:53.306+0000 7f038caf3700 1 --2- 172.21.15.42:0/1165801292 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f038807d9f0 0x7f038811b270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:18:53.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:53.306+0000 7f038caf3700 1 -- 172.21.15.42:0/1165801292 >> v1:172.21.15.42:6789/0 conn(0x7f0388123e90 legacy=0x7f03881300b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:18:53.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:53.306+0000 7f038caf3700 1 -- 172.21.15.42:0/1165801292 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03881308b0 con 0x7f0388123a60 2019-12-05T06:18:53.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:53.306+0000 7f038d2f4700 1 --2- 172.21.15.42:0/1165801292 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f038807d9f0 0x7f038811b270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:18:53.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:53.306+0000 7f038caf3700 1 --2- 172.21.15.42:0/1165801292 >> v2:172.21.15.42:3300/0 conn(0x7f0388123a60 0x7f038811d960 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f037c0014b0 tx=0x7f037c0109d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:18:53.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:53.306+0000 7f038caf3700 1 --2- 172.21.15.42:0/1165801292 >> v2:172.21.15.42:3300/0 conn(0x7f0388123a60 0x7f038811d960 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f037c0014b0 tx=0x7f037c0109d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:18:53.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:53.306+0000 7f038caf3700 1 --2- 172.21.15.42:0/1165801292 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0388123a60 0x7f038811d960 secure :-1 s=READY pgs=2395 cs=0 l=1 rx=0x7f037c0014b0 tx=0x7f037c0109d0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:18:53.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:53.306+0000 7f038caf3700 1 --2- 172.21.15.42:0/1165801292 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0388123a60 0x7f038811d960 secure :-1 s=THROTTLE_DONE pgs=2395 cs=0 l=1 rx=0x7f037c0014b0 tx=0x7f037c0109d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:18:53.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:53.306+0000 7f038caf3700 1 --2- 172.21.15.42:0/1165801292 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0388123a60 0x7f038811d960 secure :-1 s=THROTTLE_DONE pgs=2395 cs=0 l=1 rx=0x7f037c0014b0 tx=0x7f037c0109d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:18:53.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:53.306+0000 7f038caf3700 1 --2- 172.21.15.42:0/1165801292 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0388123a60 0x7f038811d960 secure :-1 s=THROTTLE_DONE pgs=2395 cs=0 l=1 rx=0x7f037c0014b0 tx=0x7f037c0109d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:18:53.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:53.306+0000 7f0385ffb700 1 -- 172.21.15.42:0/1165801292 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f037c00caa0 con 0x7f0388123a60 2019-12-05T06:18:53.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:53.306+0000 7f0385ffb700 1 -- 172.21.15.42:0/1165801292 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f037c00c6c0 con 0x7f0388123a60 2019-12-05T06:18:53.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:53.306+0000 7f038ed57700 1 -- 172.21.15.42:0/1165801292 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0388130b80 con 0x7f0388123a60 2019-12-05T06:18:53.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:53.306+0000 7f0385ffb700 1 -- 172.21.15.42:0/1165801292 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f037c011190 con 0x7f0388123a60 2019-12-05T06:18:53.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:53.306+0000 7f038ed57700 1 -- 172.21.15.42:0/1165801292 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0388137720 con 0x7f0388123a60 2019-12-05T06:18:53.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:53.306+0000 7f038caf3700 1 --2- 172.21.15.42:0/1165801292 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0388123a60 0x7f038811d960 secure :-1 s=THROTTLE_DONE pgs=2395 cs=0 l=1 rx=0x7f037c0014b0 tx=0x7f037c0109d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:18:53.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:53.310+0000 7f0385ffb700 1 -- 172.21.15.42:0/1165801292 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f037c007430 con 0x7f0388123a60 2019-12-05T06:18:53.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:53.310+0000 7f038caf3700 1 --2- 172.21.15.42:0/1165801292 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0388123a60 0x7f038811d960 secure :-1 s=THROTTLE_DONE pgs=2395 cs=0 l=1 rx=0x7f037c0014b0 tx=0x7f037c0109d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:18:53.358 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:53.310+0000 7f0385ffb700 1 --2- 172.21.15.42:0/1165801292 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f036803ae60 0x7f036803d370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:18:53.358 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:53.310+0000 7f0385ffb700 1 -- 172.21.15.42:0/1165801292 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 10417+0+0 (secure 0 0 0) 0x7f037c04b1f0 con 0x7f0388123a60 2019-12-05T06:18:53.358 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:53.310+0000 7f038d2f4700 1 --2- 172.21.15.42:0/1165801292 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f036803ae60 0x7f036803d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:18:53.358 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:53.310+0000 7f038d2f4700 1 --2- 172.21.15.42:0/1165801292 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f036803ae60 0x7f036803d370 crc :-1 s=READY pgs=1009 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:18:53.358 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:53.314+0000 7f038ed57700 1 -- 172.21.15.42:0/1165801292 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f038811e0e0 con 0x7f0388123a60 2019-12-05T06:18:53.358 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:53.322+0000 7f038caf3700 1 --2- 172.21.15.42:0/1165801292 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0388123a60 0x7f038811d960 secure :-1 s=THROTTLE_DONE pgs=2395 cs=0 l=1 rx=0x7f037c0014b0 tx=0x7f037c0109d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:18:53.359 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:53.322+0000 7f0385ffb700 1 -- 172.21.15.42:0/1165801292 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f0388130b80 con 0x7f0388123a60 2019-12-05T06:18:53.499 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:53.490+0000 7f038ed57700 1 -- 172.21.15.42:0/1165801292 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f038805d110 con 0x7f0388123a60 2019-12-05T06:18:53.499 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:53.490+0000 7f038caf3700 1 --2- 172.21.15.42:0/1165801292 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0388123a60 0x7f038811d960 secure :-1 s=THROTTLE_DONE pgs=2395 cs=0 l=1 rx=0x7f037c0014b0 tx=0x7f037c0109d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:18:53.500 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:53.490+0000 7f0385ffb700 1 -- 172.21.15.42:0/1165801292 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 288 v288) v1 ==== 95+0+2114 (secure 0 0 0) 0x7f037c032210 con 0x7f0388123a60 2019-12-05T06:18:53.500 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 288 2019-12-05T06:18:53.500 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:18:53.500 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":288,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":10476,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/1473620421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1473620421},{"type":"v1","addr":"172.21.15.40:6833","nonce":1473620421}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":288}],"filesystems":[{"mdsmap":{"epoch":287,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:17:04.753930+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":100,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7295},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7295":{"gid":7295,"name":"a","rank":0,"incarnation":284,"state":"up:active","state_seq":38,"addr":"172.21.15.42:6835/3228549071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3228549071},{"type":"v1","addr":"172.21.15.42:6835","nonce":3228549071}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:18:53.509 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:53.494+0000 7f038ed57700 1 -- 172.21.15.42:0/1165801292 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f036803ae60 msgr2=0x7f036803d370 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:18:53.509 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:53.494+0000 7f038ed57700 1 --2- 172.21.15.42:0/1165801292 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f036803ae60 0x7f036803d370 crc :-1 s=READY pgs=1009 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:18:53.509 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:53.494+0000 7f038ed57700 1 -- 172.21.15.42:0/1165801292 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0388123a60 msgr2=0x7f038811d960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:18:53.509 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:53.494+0000 7f038ed57700 1 --2- 172.21.15.42:0/1165801292 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0388123a60 0x7f038811d960 secure :-1 s=READY pgs=2395 cs=0 l=1 rx=0x7f037c0014b0 tx=0x7f037c0109d0).stop 2019-12-05T06:18:53.509 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:53.494+0000 7f038ed57700 1 -- 172.21.15.42:0/1165801292 shutdown_connections 2019-12-05T06:18:53.509 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:53.494+0000 7f038ed57700 1 --2- 172.21.15.42:0/1165801292 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f036803ae60 0x7f036803d370 unknown :-1 s=CLOSED pgs=1009 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:18:53.510 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:53.494+0000 7f038ed57700 1 --2- 172.21.15.42:0/1165801292 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f038807d9f0 0x7f038811b270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:18:53.510 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:53.494+0000 7f038ed57700 1 --2- 172.21.15.42:0/1165801292 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0388123a60 0x7f038811d960 unknown :-1 s=CLOSED pgs=2395 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:18:53.510 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:53.494+0000 7f038ed57700 1 -- 172.21.15.42:0/1165801292 shutdown_connections 2019-12-05T06:18:53.510 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:53.494+0000 7f038ed57700 1 -- 172.21.15.42:0/1165801292 wait complete. 2019-12-05T06:18:53.510 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:53.494+0000 7f038ed57700 1 -- 172.21.15.42:0/1165801292 >> 172.21.15.42:0/1165801292 conn(0x7f038807ee20 msgr2=0x7f038809f240 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:18:53.523 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:18:55.524 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:18:55.632 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:55.622+0000 7fece0e1e700 1 Processor -- start 2019-12-05T06:18:55.632 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:55.622+0000 7fece0e1e700 1 -- start start 2019-12-05T06:18:55.633 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:55.622+0000 7fece0e1e700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fecdc0a7740 0x7fecdc08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:18:55.633 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:55.622+0000 7fece0e1e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fecdc09fae0 0x7fecdc07eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:18:55.633 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:55.622+0000 7fece0e1e700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecdc0a0710 con 0x7fecdc07e720 2019-12-05T06:18:55.633 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:55.622+0000 7fece0e1e700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fecdc0a0b00 con 0x7fecdc0a7740 2019-12-05T06:18:55.634 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:55.622+0000 7fece0e1e700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fecdc08e5b0 con 0x7fecdc09fae0 2019-12-05T06:18:55.635 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:55.622+0000 7fecdb59e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fecdc09fae0 0x7fecdc07eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:18:55.635 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:55.622+0000 7fecdb59e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fecdc09fae0 0x7fecdc07eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:52630/0 (socket says 172.21.15.42:52630) 2019-12-05T06:18:55.635 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:55.622+0000 7fecdb59e700 1 -- 172.21.15.42:0/2334462131 learned_addr learned my addr 172.21.15.42:0/2334462131 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:18:55.635 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:55.622+0000 7fecdad9d700 1 --2- 172.21.15.42:0/2334462131 >> v2:172.21.15.40:3300/0 conn(0x7fecdc0a7740 0x7fecdc08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:18:55.636 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:55.622+0000 7fecdb59e700 1 -- 172.21.15.42:0/2334462131 >> v2:172.21.15.40:3300/0 conn(0x7fecdc0a7740 msgr2=0x7fecdc08df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:18:55.636 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:55.622+0000 7fecdb59e700 1 --2- 172.21.15.42:0/2334462131 >> v2:172.21.15.40:3300/0 conn(0x7fecdc0a7740 0x7fecdc08df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:18:55.636 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:55.622+0000 7fecdb59e700 1 -- 172.21.15.42:0/2334462131 >> v1:172.21.15.40:6789/0 conn(0x7fecdc07e720 legacy=0x7fecdc09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:18:55.636 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:55.622+0000 7fecdb59e700 1 -- 172.21.15.42:0/2334462131 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecdc07f0a0 con 0x7fecdc09fae0 2019-12-05T06:18:55.637 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:55.622+0000 7fecdb59e700 1 --2- 172.21.15.42:0/2334462131 >> v2:172.21.15.42:3300/0 conn(0x7fecdc09fae0 0x7fecdc07eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fecd40032b0 tx=0x7fecd40100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:18:55.637 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:55.622+0000 7fecdb59e700 1 --2- 172.21.15.42:0/2334462131 >> v2:172.21.15.42:3300/0 conn(0x7fecdc09fae0 0x7fecdc07eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fecd40032b0 tx=0x7fecd40100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:18:55.637 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:55.622+0000 7fecdb59e700 1 --2- 172.21.15.42:0/2334462131 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fecdc09fae0 0x7fecdc07eb50 secure :-1 s=READY pgs=2396 cs=0 l=1 rx=0x7fecd40032b0 tx=0x7fecd40100b0).ready entity=mon.1 client_cookie=2f06f62427f845f1 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:18:55.637 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:55.622+0000 7fecdb59e700 1 --2- 172.21.15.42:0/2334462131 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fecdc09fae0 0x7fecdc07eb50 secure :-1 s=THROTTLE_DONE pgs=2396 cs=0 l=1 rx=0x7fecd40032b0 tx=0x7fecd40100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:18:55.637 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:55.622+0000 7fecd9d9b700 1 -- 172.21.15.42:0/2334462131 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fecd4010d60 con 0x7fecdc09fae0 2019-12-05T06:18:55.637 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:55.622+0000 7fecdb59e700 1 --2- 172.21.15.42:0/2334462131 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fecdc09fae0 0x7fecdc07eb50 secure :-1 s=THROTTLE_DONE pgs=2396 cs=0 l=1 rx=0x7fecd40032b0 tx=0x7fecd40100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:18:55.638 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:55.622+0000 7fecd9d9b700 1 -- 172.21.15.42:0/2334462131 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecd4010980 con 0x7fecdc09fae0 2019-12-05T06:18:55.638 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:55.622+0000 7fecdb59e700 1 --2- 172.21.15.42:0/2334462131 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fecdc09fae0 0x7fecdc07eb50 secure :-1 s=THROTTLE_DONE pgs=2396 cs=0 l=1 rx=0x7fecd40032b0 tx=0x7fecd40100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:18:55.638 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:55.622+0000 7fece0e1e700 1 -- 172.21.15.42:0/2334462131 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fecdc09fae0 msgr2=0x7fecdc07eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:18:55.638 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:55.622+0000 7fecd9d9b700 1 -- 172.21.15.42:0/2334462131 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fecd40114a0 con 0x7fecdc09fae0 2019-12-05T06:18:55.638 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:55.622+0000 7fece0e1e700 1 --2- 172.21.15.42:0/2334462131 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fecdc09fae0 0x7fecdc07eb50 secure :-1 s=READY pgs=2396 cs=0 l=1 rx=0x7fecd40032b0 tx=0x7fecd40100b0).stop 2019-12-05T06:18:55.639 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:55.622+0000 7fece0e1e700 1 -- 172.21.15.42:0/2334462131 shutdown_connections 2019-12-05T06:18:55.639 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:55.622+0000 7fece0e1e700 1 --2- 172.21.15.42:0/2334462131 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fecdc09fae0 0x7fecdc07eb50 unknown :-1 s=CLOSED pgs=2396 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:18:55.639 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:55.622+0000 7fece0e1e700 1 --2- 172.21.15.42:0/2334462131 >> v2:172.21.15.40:3300/0 conn(0x7fecdc0a7740 0x7fecdc08df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:18:55.639 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:55.622+0000 7fece0e1e700 1 -- 172.21.15.42:0/2334462131 shutdown_connections 2019-12-05T06:18:55.639 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:55.622+0000 7fece0e1e700 1 -- 172.21.15.42:0/2334462131 wait complete. 2019-12-05T06:18:55.639 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:55.622+0000 7fece0e1e700 1 -- 172.21.15.42:0/2334462131 >> 172.21.15.42:0/2334462131 conn(0x7fecdc0aabe0 msgr2=0x7fecdc0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:18:55.640 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:55.630+0000 7fece0e1e700 1 Processor -- start 2019-12-05T06:18:55.640 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:55.630+0000 7fece0e1e700 1 -- start start 2019-12-05T06:18:55.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:55.630+0000 7fece0e1e700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fecdc139170 0x7fecdc13b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:18:55.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:55.630+0000 7fece0e1e700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecdc07bd50 con 0x7fecdc11ac40 2019-12-05T06:18:55.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:55.630+0000 7fece0e1e700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecdc13cb10 con 0x7fecdc11bf20 2019-12-05T06:18:55.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:55.630+0000 7fece0e1e700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fecdc07bf80 con 0x7fecdc139170 2019-12-05T06:18:55.654 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:55.630+0000 7fecdad9d700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fecdc11bf20 0x7fecdc07cba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:52670/0 (socket says 172.21.15.42:52670) 2019-12-05T06:18:55.654 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:55.630+0000 7fecdad9d700 1 -- 172.21.15.42:0/2866247115 learned_addr learned my addr 172.21.15.42:0/2866247115 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:18:55.654 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:55.630+0000 7fecdb59e700 1 --2- 172.21.15.42:0/2866247115 >> v2:172.21.15.40:3300/0 conn(0x7fecdc139170 0x7fecdc13b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:18:55.654 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:55.630+0000 7feccb7fe700 1 -- 172.21.15.42:0/2866247115 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2152399355 0 0) 0x7fecdc13cb10 con 0x7fecdc11bf20 2019-12-05T06:18:55.655 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:55.630+0000 7feccb7fe700 1 -- 172.21.15.42:0/2866247115 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fecc00030b0 con 0x7fecdc11bf20 2019-12-05T06:18:55.655 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:55.630+0000 7feccb7fe700 1 -- 172.21.15.42:0/2866247115 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3402197587 0 0) 0x7fecdc07bd50 con 0x7fecdc11ac40 2019-12-05T06:18:55.655 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:55.630+0000 7feccb7fe700 1 -- 172.21.15.42:0/2866247115 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fecdc13cb10 con 0x7fecdc11ac40 2019-12-05T06:18:55.656 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:55.630+0000 7feccb7fe700 1 -- 172.21.15.42:0/2866247115 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 3862537711 0 0) 0x7fecc00030b0 con 0x7fecdc11bf20 2019-12-05T06:18:55.656 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:55.630+0000 7feccb7fe700 1 -- 172.21.15.42:0/2866247115 --> v1:172.21.15.42:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fecdc07bd50 con 0x7fecdc11bf20 2019-12-05T06:18:55.656 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:55.630+0000 7feccb7fe700 1 -- 172.21.15.42:0/2866247115 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7feccc002580 con 0x7fecdc11bf20 2019-12-05T06:18:55.656 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:55.630+0000 7feccb7fe700 1 -- 172.21.15.42:0/2866247115 >> v2:172.21.15.40:3300/0 conn(0x7fecdc139170 msgr2=0x7fecdc13b5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:18:55.656 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:55.630+0000 7feccb7fe700 1 --2- 172.21.15.42:0/2866247115 >> v2:172.21.15.40:3300/0 conn(0x7fecdc139170 0x7fecdc13b5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:18:55.656 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:55.630+0000 7feccb7fe700 1 -- 172.21.15.42:0/2866247115 >> v1:172.21.15.42:6789/0 conn(0x7fecdc11bf20 legacy=0x7fecdc07cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:18:55.656 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:55.630+0000 7feccb7fe700 1 -- 172.21.15.42:0/2866247115 >> v1:172.21.15.40:6789/0 conn(0x7fecdc11ac40 legacy=0x7fecdc11b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:18:55.657 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:55.630+0000 7feccb7fe700 1 --2- 172.21.15.42:0/2866247115 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fecc0003ab0 0x7fecc0004420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:18:55.657 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:55.630+0000 7feccb7fe700 1 --2- 172.21.15.42:0/2866247115 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fecc0004970 0x7fecc0004da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:18:55.657 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:55.630+0000 7feccb7fe700 1 --2- 172.21.15.42:0/2866247115 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fecc0009660 0x7fecc000baa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:18:55.657 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:55.630+0000 7feccb7fe700 1 -- 172.21.15.42:0/2866247115 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecdc07bf80 con 0x7fecc0009660 2019-12-05T06:18:55.657 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:55.630+0000 7feccb7fe700 1 -- 172.21.15.42:0/2866247115 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecc000bff0 con 0x7fecc0003ab0 2019-12-05T06:18:55.657 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:55.630+0000 7feccb7fe700 1 -- 172.21.15.42:0/2866247115 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fecc000c1b0 con 0x7fecc0004970 2019-12-05T06:18:55.657 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:55.630+0000 7fecdb59e700 1 --2- 172.21.15.42:0/2866247115 >> v2:172.21.15.40:3300/0 conn(0x7fecdc139170 0x7fecdc13b5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:18:55.658 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:55.630+0000 7fecdad9d700 1 --2- 172.21.15.42:0/2866247115 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fecc0003ab0 0x7fecc0004420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:18:55.658 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:55.630+0000 7fecdb59e700 1 --2- 172.21.15.42:0/2866247115 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fecc0009660 0x7fecc000baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:18:55.658 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:55.630+0000 7fecda59c700 1 --2- 172.21.15.42:0/2866247115 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fecc0004970 0x7fecc0004da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:18:55.658 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:55.630+0000 7fecdad9d700 1 -- 172.21.15.42:0/2866247115 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fecc0004970 msgr2=0x7fecc0004da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:18:55.658 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:55.630+0000 7fecdad9d700 1 --2- 172.21.15.42:0/2866247115 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fecc0004970 0x7fecc0004da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:18:55.658 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:55.630+0000 7fecdad9d700 1 -- 172.21.15.42:0/2866247115 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fecc0009660 msgr2=0x7fecc000baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:18:55.658 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:55.630+0000 7fecdad9d700 1 --2- 172.21.15.42:0/2866247115 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fecc0009660 0x7fecc000baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:18:55.658 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:55.630+0000 7fecdb59e700 1 -- 172.21.15.42:0/2866247115 reap_dead start 2019-12-05T06:18:55.659 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:55.630+0000 7fecdad9d700 1 -- 172.21.15.42:0/2866247115 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecdc13dd50 con 0x7fecc0003ab0 2019-12-05T06:18:55.659 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:55.630+0000 7fece0e1e700 1 -- 172.21.15.42:0/2866247115 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecdc11bf20 con 0x7fecc0003ab0 2019-12-05T06:18:55.659 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:55.630+0000 7fecdad9d700 1 --2- 172.21.15.42:0/2866247115 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fecc0003ab0 0x7fecc0004420 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7feccc0078c0 tx=0x7feccc0136c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:18:55.659 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:55.630+0000 7fece0e1e700 1 -- 172.21.15.42:0/2866247115 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fecdc07cba0 con 0x7fecc0003ab0 2019-12-05T06:18:55.659 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:55.630+0000 7fecdad9d700 1 --2- 172.21.15.42:0/2866247115 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fecc0003ab0 0x7fecc0004420 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7feccc0078c0 tx=0x7feccc0136c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:18:55.659 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:55.630+0000 7fecdad9d700 1 --2- 172.21.15.42:0/2866247115 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fecc0003ab0 0x7fecc0004420 secure :-1 s=READY pgs=2398 cs=0 l=1 rx=0x7feccc0078c0 tx=0x7feccc0136c0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:18:55.659 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:55.630+0000 7fecdad9d700 1 --2- 172.21.15.42:0/2866247115 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fecc0003ab0 0x7fecc0004420 secure :-1 s=THROTTLE_DONE pgs=2398 cs=0 l=1 rx=0x7feccc0078c0 tx=0x7feccc0136c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:18:55.660 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:55.630+0000 7feccb7fe700 1 -- 172.21.15.42:0/2866247115 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7feccc00ec70 con 0x7fecc0003ab0 2019-12-05T06:18:55.660 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:55.630+0000 7fecdad9d700 1 --2- 172.21.15.42:0/2866247115 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fecc0003ab0 0x7fecc0004420 secure :-1 s=THROTTLE_DONE pgs=2398 cs=0 l=1 rx=0x7feccc0078c0 tx=0x7feccc0136c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:18:55.660 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:55.630+0000 7fecdad9d700 1 --2- 172.21.15.42:0/2866247115 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fecc0003ab0 0x7fecc0004420 secure :-1 s=THROTTLE_DONE pgs=2398 cs=0 l=1 rx=0x7feccc0078c0 tx=0x7feccc0136c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:18:55.660 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:55.630+0000 7feccb7fe700 1 -- 172.21.15.42:0/2866247115 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feccc00cd80 con 0x7fecc0003ab0 2019-12-05T06:18:55.660 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:55.630+0000 7feccb7fe700 1 -- 172.21.15.42:0/2866247115 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7feccc00e540 con 0x7fecc0003ab0 2019-12-05T06:18:55.660 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:55.630+0000 7fecdad9d700 1 --2- 172.21.15.42:0/2866247115 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fecc0003ab0 0x7fecc0004420 secure :-1 s=THROTTLE_DONE pgs=2398 cs=0 l=1 rx=0x7feccc0078c0 tx=0x7feccc0136c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:18:55.660 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:55.630+0000 7feccb7fe700 1 -- 172.21.15.42:0/2866247115 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7feccc010180 con 0x7fecc0003ab0 2019-12-05T06:18:55.661 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:55.630+0000 7fecdad9d700 1 --2- 172.21.15.42:0/2866247115 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fecc0003ab0 0x7fecc0004420 secure :-1 s=THROTTLE_DONE pgs=2398 cs=0 l=1 rx=0x7feccc0078c0 tx=0x7feccc0136c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:18:55.661 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:55.630+0000 7feccb7fe700 1 --2- 172.21.15.42:0/2866247115 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fecc003ec10 0x7fecc0041120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:18:55.661 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:55.630+0000 7feccb7fe700 1 -- 172.21.15.42:0/2866247115 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 10417+0+0 (secure 0 0 0) 0x7feccc04d8f0 con 0x7fecc0003ab0 2019-12-05T06:18:55.661 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:55.630+0000 7fecda59c700 1 --2- 172.21.15.42:0/2866247115 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fecc003ec10 0x7fecc0041120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:18:55.661 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:55.630+0000 7fecda59c700 1 --2- 172.21.15.42:0/2866247115 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fecc003ec10 0x7fecc0041120 crc :-1 s=READY pgs=1010 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:18:55.661 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:55.634+0000 7fece0e1e700 1 -- 172.21.15.42:0/2866247115 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fecb4003200 con 0x7fecc0003ab0 2019-12-05T06:18:55.661 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:55.646+0000 7fecdad9d700 1 --2- 172.21.15.42:0/2866247115 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fecc0003ab0 0x7fecc0004420 secure :-1 s=THROTTLE_DONE pgs=2398 cs=0 l=1 rx=0x7feccc0078c0 tx=0x7feccc0136c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:18:55.662 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:55.646+0000 7feccb7fe700 1 -- 172.21.15.42:0/2866247115 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7feccc007190 con 0x7fecc0003ab0 2019-12-05T06:18:55.813 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:55.806+0000 7fece0e1e700 1 -- 172.21.15.42:0/2866247115 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fecb4003630 con 0x7fecc0003ab0 2019-12-05T06:18:55.813 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:55.806+0000 7fecdad9d700 1 --2- 172.21.15.42:0/2866247115 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fecc0003ab0 0x7fecc0004420 secure :-1 s=THROTTLE_DONE pgs=2398 cs=0 l=1 rx=0x7feccc0078c0 tx=0x7feccc0136c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:18:55.814 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:55.806+0000 7feccb7fe700 1 -- 172.21.15.42:0/2866247115 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 288 v288) v1 ==== 95+0+2114 (secure 0 0 0) 0x7feccc034990 con 0x7fecc0003ab0 2019-12-05T06:18:55.814 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 288 2019-12-05T06:18:55.814 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:18:55.814 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":288,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":10476,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/1473620421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1473620421},{"type":"v1","addr":"172.21.15.40:6833","nonce":1473620421}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":288}],"filesystems":[{"mdsmap":{"epoch":287,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:17:04.753930+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":100,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7295},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7295":{"gid":7295,"name":"a","rank":0,"incarnation":284,"state":"up:active","state_seq":38,"addr":"172.21.15.42:6835/3228549071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3228549071},{"type":"v1","addr":"172.21.15.42:6835","nonce":3228549071}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:18:55.821 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:55.806+0000 7fece0e1e700 1 -- 172.21.15.42:0/2866247115 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fecc003ec10 msgr2=0x7fecc0041120 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:18:55.822 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:55.806+0000 7fece0e1e700 1 --2- 172.21.15.42:0/2866247115 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fecc003ec10 0x7fecc0041120 crc :-1 s=READY pgs=1010 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:18:55.822 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:55.806+0000 7fece0e1e700 1 -- 172.21.15.42:0/2866247115 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fecc0003ab0 msgr2=0x7fecc0004420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:18:55.822 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:55.806+0000 7fece0e1e700 1 --2- 172.21.15.42:0/2866247115 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fecc0003ab0 0x7fecc0004420 secure :-1 s=READY pgs=2398 cs=0 l=1 rx=0x7feccc0078c0 tx=0x7feccc0136c0).stop 2019-12-05T06:18:55.822 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:55.810+0000 7fece0e1e700 1 -- 172.21.15.42:0/2866247115 shutdown_connections 2019-12-05T06:18:55.822 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:55.810+0000 7fece0e1e700 1 --2- 172.21.15.42:0/2866247115 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fecc003ec10 0x7fecc0041120 unknown :-1 s=CLOSED pgs=1010 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:18:55.822 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:55.810+0000 7fece0e1e700 1 --2- 172.21.15.42:0/2866247115 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fecc0003ab0 0x7fecc0004420 unknown :-1 s=CLOSED pgs=2398 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:18:55.822 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:55.810+0000 7fece0e1e700 1 -- 172.21.15.42:0/2866247115 shutdown_connections 2019-12-05T06:18:55.823 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:55.810+0000 7fece0e1e700 1 -- 172.21.15.42:0/2866247115 wait complete. 2019-12-05T06:18:55.823 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:55.810+0000 7fece0e1e700 1 -- 172.21.15.42:0/2866247115 >> 172.21.15.42:0/2866247115 conn(0x7fecdc0a9790 msgr2=0x7fecdc11c520 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:18:55.826 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:18:57.828 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:18:57.943 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:57.934+0000 7fdecaaaf700 1 Processor -- start 2019-12-05T06:18:57.943 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:57.934+0000 7fdecaaaf700 1 -- start start 2019-12-05T06:18:57.943 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:57.934+0000 7fdecaaaf700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fdec408df80 0x7fdec409f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:18:57.944 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:57.934+0000 7fdecaaaf700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdec409fc60 con 0x7fdec40a7740 2019-12-05T06:18:57.944 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:57.934+0000 7fdecaaaf700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdec40a0710 con 0x7fdec409f830 2019-12-05T06:18:57.944 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:57.934+0000 7fdecaaaf700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdec40a0b00 con 0x7fdec408df80 2019-12-05T06:18:57.944 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:57.934+0000 7fdec884b700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fdec408df80 0x7fdec409f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:18:57.944 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:57.934+0000 7fdec884b700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fdec408df80 0x7fdec409f2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:52650/0 (socket says 172.21.15.42:52650) 2019-12-05T06:18:57.944 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:57.934+0000 7fdec884b700 1 -- 172.21.15.42:0/1964072796 learned_addr learned my addr 172.21.15.42:0/1964072796 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:18:57.944 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:57.934+0000 7fdec984d700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fdec409f830 0x7fdec414efe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:52690/0 (socket says 172.21.15.42:52690) 2019-12-05T06:18:57.945 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:57.938+0000 7fdec884b700 1 -- 172.21.15.42:0/1964072796 >> v1:172.21.15.42:6789/0 conn(0x7fdec409f830 legacy=0x7fdec414efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:18:57.945 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:57.938+0000 7fdeb7fff700 1 -- 172.21.15.42:0/1964072796 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1404086605 0 0) 0x7fdec409fc60 con 0x7fdec40a7740 2019-12-05T06:18:57.945 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:57.938+0000 7fdec884b700 1 -- 172.21.15.42:0/1964072796 >> v1:172.21.15.40:6789/0 conn(0x7fdec40a7740 legacy=0x7fdec407e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:18:57.946 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:57.938+0000 7fdec884b700 1 -- 172.21.15.42:0/1964072796 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdec407ef20 con 0x7fdec408df80 2019-12-05T06:18:57.946 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:57.938+0000 7fdec884b700 1 --2- 172.21.15.42:0/1964072796 >> v2:172.21.15.42:3300/0 conn(0x7fdec408df80 0x7fdec409f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fdeb8008b10 tx=0x7fdebc001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:18:57.946 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:57.938+0000 7fdec884b700 1 --2- 172.21.15.42:0/1964072796 >> v2:172.21.15.42:3300/0 conn(0x7fdec408df80 0x7fdec409f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fdeb8008b10 tx=0x7fdebc001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:18:57.946 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:57.938+0000 7fdec884b700 1 --2- 172.21.15.42:0/1964072796 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdec408df80 0x7fdec409f2e0 secure :-1 s=READY pgs=2400 cs=0 l=1 rx=0x7fdeb8008b10 tx=0x7fdebc001150).ready entity=mon.1 client_cookie=a20baf1ad22fe3 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:18:57.946 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:57.938+0000 7fdec884b700 1 --2- 172.21.15.42:0/1964072796 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdec408df80 0x7fdec409f2e0 secure :-1 s=THROTTLE_DONE pgs=2400 cs=0 l=1 rx=0x7fdeb8008b10 tx=0x7fdebc001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:18:57.946 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:57.938+0000 7fdeb7fff700 1 -- 172.21.15.42:0/1964072796 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdeb8010d10 con 0x7fdec408df80 2019-12-05T06:18:57.947 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:57.938+0000 7fdec884b700 1 --2- 172.21.15.42:0/1964072796 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdec408df80 0x7fdec409f2e0 secure :-1 s=THROTTLE_DONE pgs=2400 cs=0 l=1 rx=0x7fdeb8008b10 tx=0x7fdebc001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:18:57.947 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:57.938+0000 7fdec884b700 1 --2- 172.21.15.42:0/1964072796 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdec408df80 0x7fdec409f2e0 secure :-1 s=THROTTLE_DONE pgs=2400 cs=0 l=1 rx=0x7fdeb8008b10 tx=0x7fdebc001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:18:57.947 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:57.938+0000 7fdeb7fff700 1 -- 172.21.15.42:0/1964072796 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdeb8010930 con 0x7fdec408df80 2019-12-05T06:18:57.948 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:57.938+0000 7fdeb7fff700 1 -- 172.21.15.42:0/1964072796 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdeb80114a0 con 0x7fdec408df80 2019-12-05T06:18:57.948 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:57.938+0000 7fdecaaaf700 1 -- 172.21.15.42:0/1964072796 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdec408df80 msgr2=0x7fdec409f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:18:57.948 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:57.938+0000 7fdecaaaf700 1 --2- 172.21.15.42:0/1964072796 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdec408df80 0x7fdec409f2e0 secure :-1 s=READY pgs=2400 cs=0 l=1 rx=0x7fdeb8008b10 tx=0x7fdebc001150).stop 2019-12-05T06:18:57.949 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:57.938+0000 7fdecaaaf700 1 -- 172.21.15.42:0/1964072796 shutdown_connections 2019-12-05T06:18:57.949 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:57.938+0000 7fdecaaaf700 1 --2- 172.21.15.42:0/1964072796 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdec408df80 0x7fdec409f2e0 unknown :-1 s=CLOSED pgs=2400 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:18:57.949 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:57.938+0000 7fdecaaaf700 1 -- 172.21.15.42:0/1964072796 shutdown_connections 2019-12-05T06:18:57.949 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:57.938+0000 7fdecaaaf700 1 -- 172.21.15.42:0/1964072796 wait complete. 2019-12-05T06:18:57.949 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:57.938+0000 7fdecaaaf700 1 -- 172.21.15.42:0/1964072796 >> 172.21.15.42:0/1964072796 conn(0x7fdec40aabe0 msgr2=0x7fdec40a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:18:57.952 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:57.942+0000 7fdecaaaf700 1 Processor -- start 2019-12-05T06:18:57.953 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:57.942+0000 7fdecaaaf700 1 -- start start 2019-12-05T06:18:57.953 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:57.942+0000 7fdecaaaf700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fdec407c350 0x7fdec405ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:18:57.953 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:57.942+0000 7fdecaaaf700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fdec40a9ad0 0x7fdec4125cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:18:57.953 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:57.942+0000 7fdecaaaf700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdec4144940 con 0x7fdec411ee80 2019-12-05T06:18:57.953 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:57.942+0000 7fdecaaaf700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdec4125240 con 0x7fdec40a9ad0 2019-12-05T06:18:57.953 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:57.942+0000 7fdecaaaf700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdec41254e0 con 0x7fdec407c350 2019-12-05T06:18:57.953 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:57.942+0000 7fdec984d700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fdec411ee80 0x7fdec411f2b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:52696/0 (socket says 172.21.15.42:52696) 2019-12-05T06:18:57.954 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:57.942+0000 7fdec984d700 1 -- 172.21.15.42:0/649501784 learned_addr learned my addr 172.21.15.42:0/649501784 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:18:57.954 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:57.942+0000 7fdec904c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fdec407c350 0x7fdec405ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:18:57.954 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:57.942+0000 7fdec884b700 1 --2- 172.21.15.42:0/649501784 >> v2:172.21.15.40:3300/0 conn(0x7fdec40a9ad0 0x7fdec4125cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:18:57.954 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:57.942+0000 7fdeb5ffb700 1 -- 172.21.15.42:0/649501784 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2281780010 0 0) 0x7fdec4144940 con 0x7fdec411ee80 2019-12-05T06:18:57.954 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:57.942+0000 7fdeb5ffb700 1 -- 172.21.15.42:0/649501784 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdea40030b0 con 0x7fdec411ee80 2019-12-05T06:18:57.954 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:57.942+0000 7fdec904c700 1 -- 172.21.15.42:0/649501784 >> v2:172.21.15.40:3300/0 conn(0x7fdec40a9ad0 msgr2=0x7fdec4125cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:18:57.954 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:57.942+0000 7fdec904c700 1 --2- 172.21.15.42:0/649501784 >> v2:172.21.15.40:3300/0 conn(0x7fdec40a9ad0 0x7fdec4125cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:18:57.955 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:57.942+0000 7fdec904c700 1 -- 172.21.15.42:0/649501784 >> v1:172.21.15.42:6789/0 conn(0x7fdec411ee80 legacy=0x7fdec411f2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:18:57.955 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:57.942+0000 7fdec904c700 1 -- 172.21.15.42:0/649501784 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdec4145420 con 0x7fdec407c350 2019-12-05T06:18:57.955 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:57.946+0000 7fdec904c700 1 --2- 172.21.15.42:0/649501784 >> v2:172.21.15.42:3300/0 conn(0x7fdec407c350 0x7fdec405ed80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fdec0000b80 tx=0x7fdec0010810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:18:57.955 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:57.946+0000 7fdec904c700 1 --2- 172.21.15.42:0/649501784 >> v2:172.21.15.42:3300/0 conn(0x7fdec407c350 0x7fdec405ed80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fdec0000b80 tx=0x7fdec0010810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:18:57.955 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:57.946+0000 7fdec904c700 1 --2- 172.21.15.42:0/649501784 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdec407c350 0x7fdec405ed80 secure :-1 s=READY pgs=2402 cs=0 l=1 rx=0x7fdec0000b80 tx=0x7fdec0010810).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:18:57.955 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:57.946+0000 7fdec904c700 1 --2- 172.21.15.42:0/649501784 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdec407c350 0x7fdec405ed80 secure :-1 s=THROTTLE_DONE pgs=2402 cs=0 l=1 rx=0x7fdec0000b80 tx=0x7fdec0010810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:18:57.956 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:57.946+0000 7fdec904c700 1 --2- 172.21.15.42:0/649501784 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdec407c350 0x7fdec405ed80 secure :-1 s=THROTTLE_DONE pgs=2402 cs=0 l=1 rx=0x7fdec0000b80 tx=0x7fdec0010810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:18:57.956 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:57.946+0000 7fdeb5ffb700 1 -- 172.21.15.42:0/649501784 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdec0006a40 con 0x7fdec407c350 2019-12-05T06:18:57.956 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:57.946+0000 7fdec904c700 1 --2- 172.21.15.42:0/649501784 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdec407c350 0x7fdec405ed80 secure :-1 s=THROTTLE_DONE pgs=2402 cs=0 l=1 rx=0x7fdec0000b80 tx=0x7fdec0010810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:18:57.956 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:57.946+0000 7fdeb5ffb700 1 -- 172.21.15.42:0/649501784 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdec0006660 con 0x7fdec407c350 2019-12-05T06:18:57.956 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:57.946+0000 7fdeb5ffb700 1 -- 172.21.15.42:0/649501784 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdec0011050 con 0x7fdec407c350 2019-12-05T06:18:57.957 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:57.946+0000 7fdecaaaf700 1 -- 172.21.15.42:0/649501784 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdec412fc50 con 0x7fdec407c350 2019-12-05T06:18:57.957 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:57.946+0000 7fdecaaaf700 1 -- 172.21.15.42:0/649501784 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdec412ffb0 con 0x7fdec407c350 2019-12-05T06:18:57.957 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:57.946+0000 7fdec904c700 1 --2- 172.21.15.42:0/649501784 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdec407c350 0x7fdec405ed80 secure :-1 s=THROTTLE_DONE pgs=2402 cs=0 l=1 rx=0x7fdec0000b80 tx=0x7fdec0010810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:18:57.957 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:57.946+0000 7fdec904c700 1 --2- 172.21.15.42:0/649501784 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdec407c350 0x7fdec405ed80 secure :-1 s=THROTTLE_DONE pgs=2402 cs=0 l=1 rx=0x7fdec0000b80 tx=0x7fdec0010810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:18:57.957 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:57.946+0000 7fdeb5ffb700 1 -- 172.21.15.42:0/649501784 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fdec0011360 con 0x7fdec407c350 2019-12-05T06:18:57.957 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:57.946+0000 7fdeb5ffb700 1 --2- 172.21.15.42:0/649501784 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fdea403b2d0 0x7fdea403d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:18:57.958 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:57.946+0000 7fdeb5ffb700 1 -- 172.21.15.42:0/649501784 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 10417+0+0 (secure 0 0 0) 0x7fdec004b100 con 0x7fdec407c350 2019-12-05T06:18:57.958 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:57.946+0000 7fdec884b700 1 --2- 172.21.15.42:0/649501784 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fdea403b2d0 0x7fdea403d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:18:57.958 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:57.946+0000 7fdec884b700 1 --2- 172.21.15.42:0/649501784 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fdea403b2d0 0x7fdea403d7e0 crc :-1 s=READY pgs=1011 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:18:57.959 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:57.950+0000 7fdecaaaf700 1 -- 172.21.15.42:0/649501784 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdec405d110 con 0x7fdec407c350 2019-12-05T06:18:57.961 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:57.958+0000 7fdec904c700 1 --2- 172.21.15.42:0/649501784 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdec407c350 0x7fdec405ed80 secure :-1 s=THROTTLE_DONE pgs=2402 cs=0 l=1 rx=0x7fdec0000b80 tx=0x7fdec0010810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:18:57.961 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:57.958+0000 7fdeb5ffb700 1 -- 172.21.15.42:0/649501784 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fdec0032140 con 0x7fdec407c350 2019-12-05T06:18:58.133 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:58.126+0000 7fdecaaaf700 1 -- 172.21.15.42:0/649501784 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdec405b8c0 con 0x7fdec407c350 2019-12-05T06:18:58.133 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:58.126+0000 7fdec904c700 1 --2- 172.21.15.42:0/649501784 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdec407c350 0x7fdec405ed80 secure :-1 s=THROTTLE_DONE pgs=2402 cs=0 l=1 rx=0x7fdec0000b80 tx=0x7fdec0010810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:18:58.133 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:58.126+0000 7fdeb5ffb700 1 -- 172.21.15.42:0/649501784 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 288 v288) v1 ==== 95+0+2114 (secure 0 0 0) 0x7fdec00323a0 con 0x7fdec407c350 2019-12-05T06:18:58.133 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 288 2019-12-05T06:18:58.134 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:18:58.134 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":288,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":10476,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/1473620421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1473620421},{"type":"v1","addr":"172.21.15.40:6833","nonce":1473620421}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":288}],"filesystems":[{"mdsmap":{"epoch":287,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:17:04.753930+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":100,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7295},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7295":{"gid":7295,"name":"a","rank":0,"incarnation":284,"state":"up:active","state_seq":38,"addr":"172.21.15.42:6835/3228549071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3228549071},{"type":"v1","addr":"172.21.15.42:6835","nonce":3228549071}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:18:58.135 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:58.130+0000 7fdecaaaf700 1 -- 172.21.15.42:0/649501784 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fdea403b2d0 msgr2=0x7fdea403d7e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:18:58.135 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:58.130+0000 7fdecaaaf700 1 --2- 172.21.15.42:0/649501784 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fdea403b2d0 0x7fdea403d7e0 crc :-1 s=READY pgs=1011 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:18:58.135 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:58.130+0000 7fdecaaaf700 1 -- 172.21.15.42:0/649501784 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdec407c350 msgr2=0x7fdec405ed80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:18:58.135 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:58.130+0000 7fdecaaaf700 1 --2- 172.21.15.42:0/649501784 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdec407c350 0x7fdec405ed80 secure :-1 s=READY pgs=2402 cs=0 l=1 rx=0x7fdec0000b80 tx=0x7fdec0010810).stop 2019-12-05T06:18:58.135 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:58.130+0000 7fdecaaaf700 1 -- 172.21.15.42:0/649501784 shutdown_connections 2019-12-05T06:18:58.136 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:58.130+0000 7fdecaaaf700 1 --2- 172.21.15.42:0/649501784 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fdea403b2d0 0x7fdea403d7e0 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:18:58.136 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:58.130+0000 7fdecaaaf700 1 --2- 172.21.15.42:0/649501784 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdec407c350 0x7fdec405ed80 unknown :-1 s=CLOSED pgs=2402 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:18:58.136 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:58.130+0000 7fdecaaaf700 1 --2- 172.21.15.42:0/649501784 >> v2:172.21.15.40:3300/0 conn(0x7fdec40a9ad0 0x7fdec4125cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:18:58.136 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:58.130+0000 7fdecaaaf700 1 -- 172.21.15.42:0/649501784 shutdown_connections 2019-12-05T06:18:58.136 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:58.130+0000 7fdecaaaf700 1 -- 172.21.15.42:0/649501784 wait complete. 2019-12-05T06:18:58.136 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:18:58.130+0000 7fdecaaaf700 1 -- 172.21.15.42:0/649501784 >> 172.21.15.42:0/649501784 conn(0x7fdec40a7f90 msgr2=0x7fdec40a6a60 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:18:58.146 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:19:00.147 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:19:00.251 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:00.242+0000 7fb10e15a700 1 Processor -- start 2019-12-05T06:19:00.251 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:00.242+0000 7fb10e15a700 1 -- start start 2019-12-05T06:19:00.251 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:00.242+0000 7fb10e15a700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb1080a7740 0x7fb10808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:19:00.252 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:00.242+0000 7fb10e15a700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb10807e720 0x7fb10807eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:19:00.252 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:00.242+0000 7fb10e15a700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1080a0710 con 0x7fb10809f2e0 2019-12-05T06:19:00.252 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:00.242+0000 7fb10e15a700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1080a0b00 con 0x7fb10807e720 2019-12-05T06:19:00.252 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:00.242+0000 7fb10e15a700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb10808e5b0 con 0x7fb1080a7740 2019-12-05T06:19:00.252 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:00.242+0000 7fb1077fe700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb10807e720 0x7fb10807eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:00.253 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:00.242+0000 7fb107fff700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb1080a7740 0x7fb10808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:00.253 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:00.242+0000 7fb1077fe700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb10807e720 0x7fb10807eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.42:40480/0 (socket says 172.21.15.42:40480) 2019-12-05T06:19:00.253 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:00.242+0000 7fb1077fe700 1 -- 172.21.15.42:0/534166558 learned_addr learned my addr 172.21.15.42:0/534166558 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:19:00.253 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:00.246+0000 7fb1077fe700 1 -- 172.21.15.42:0/534166558 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb1080a7740 msgr2=0x7fb10808df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:19:00.253 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:00.246+0000 7fb1077fe700 1 --2- 172.21.15.42:0/534166558 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb1080a7740 0x7fb10808df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:19:00.253 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:00.246+0000 7fb1077fe700 1 -- 172.21.15.42:0/534166558 >> v1:172.21.15.40:6789/0 conn(0x7fb10809f2e0 legacy=0x7fb10809f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:00.253 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:00.246+0000 7fb1077fe700 1 -- 172.21.15.42:0/534166558 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb10807f0a0 con 0x7fb10807e720 2019-12-05T06:19:00.254 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:00.246+0000 7fb1077fe700 1 --2- 172.21.15.42:0/534166558 >> v2:172.21.15.40:3300/0 conn(0x7fb10807e720 0x7fb10807eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fb0fc008790 tx=0x7fb0fc00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:00.254 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:00.246+0000 7fb1077fe700 1 --2- 172.21.15.42:0/534166558 >> v2:172.21.15.40:3300/0 conn(0x7fb10807e720 0x7fb10807eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fb0fc008790 tx=0x7fb0fc00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:00.254 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:00.246+0000 7fb1077fe700 1 --2- 172.21.15.42:0/534166558 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb10807e720 0x7fb10807eb50 secure :-1 s=READY pgs=1829 cs=0 l=1 rx=0x7fb0fc008790 tx=0x7fb0fc00fd30).ready entity=mon.0 client_cookie=18147199ec0afede server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:19:00.254 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:00.246+0000 7fb1077fe700 1 --2- 172.21.15.42:0/534166558 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb10807e720 0x7fb10807eb50 secure :-1 s=THROTTLE_DONE pgs=1829 cs=0 l=1 rx=0x7fb0fc008790 tx=0x7fb0fc00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:00.254 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:00.246+0000 7fb106ffd700 1 -- 172.21.15.42:0/534166558 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb0fc0113b0 con 0x7fb10807e720 2019-12-05T06:19:00.254 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:00.246+0000 7fb1077fe700 1 --2- 172.21.15.42:0/534166558 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb10807e720 0x7fb10807eb50 secure :-1 s=THROTTLE_DONE pgs=1829 cs=0 l=1 rx=0x7fb0fc008790 tx=0x7fb0fc00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:00.254 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:00.246+0000 7fb1077fe700 1 --2- 172.21.15.42:0/534166558 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb10807e720 0x7fb10807eb50 secure :-1 s=THROTTLE_DONE pgs=1829 cs=0 l=1 rx=0x7fb0fc008790 tx=0x7fb0fc00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:00.255 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:00.246+0000 7fb106ffd700 1 -- 172.21.15.42:0/534166558 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0fc008bb0 con 0x7fb10807e720 2019-12-05T06:19:00.255 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:00.246+0000 7fb106ffd700 1 -- 172.21.15.42:0/534166558 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb0fc011990 con 0x7fb10807e720 2019-12-05T06:19:00.255 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:00.246+0000 7fb10e15a700 1 -- 172.21.15.42:0/534166558 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb10807e720 msgr2=0x7fb10807eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:00.255 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:00.246+0000 7fb10e15a700 1 --2- 172.21.15.42:0/534166558 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb10807e720 0x7fb10807eb50 secure :-1 s=READY pgs=1829 cs=0 l=1 rx=0x7fb0fc008790 tx=0x7fb0fc00fd30).stop 2019-12-05T06:19:00.255 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:00.246+0000 7fb10e15a700 1 -- 172.21.15.42:0/534166558 shutdown_connections 2019-12-05T06:19:00.255 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:00.246+0000 7fb10e15a700 1 --2- 172.21.15.42:0/534166558 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb1080a7740 0x7fb10808df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:19:00.256 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:00.246+0000 7fb10e15a700 1 --2- 172.21.15.42:0/534166558 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb10807e720 0x7fb10807eb50 unknown :-1 s=CLOSED pgs=1829 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:00.256 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:00.246+0000 7fb10e15a700 1 -- 172.21.15.42:0/534166558 shutdown_connections 2019-12-05T06:19:00.256 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:00.246+0000 7fb10e15a700 1 -- 172.21.15.42:0/534166558 wait complete. 2019-12-05T06:19:00.256 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:00.246+0000 7fb10e15a700 1 -- 172.21.15.42:0/534166558 >> 172.21.15.42:0/534166558 conn(0x7fb1080aabe0 msgr2=0x7fb1080a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:19:00.261 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:00.250+0000 7fb10e15a700 1 Processor -- start 2019-12-05T06:19:00.261 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:00.250+0000 7fb10e15a700 1 -- start start 2019-12-05T06:19:00.261 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:00.250+0000 7fb10e15a700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb108123370 0x7fb108131da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:19:00.261 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:00.250+0000 7fb10e15a700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb1080a5120 0x7fb108138960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:19:00.261 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:00.250+0000 7fb10e15a700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb10811bdd0 con 0x7fb1080a4cf0 2019-12-05T06:19:00.262 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:00.250+0000 7fb10e15a700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1081237a0 con 0x7fb108123370 2019-12-05T06:19:00.262 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:00.250+0000 7fb10e15a700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1080a5710 con 0x7fb1080a5120 2019-12-05T06:19:00.262 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:00.250+0000 7fb1077fe700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fb1080a4cf0 0x7fb10811b5d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:52708/0 (socket says 172.21.15.42:52708) 2019-12-05T06:19:00.262 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:00.250+0000 7fb1077fe700 1 -- 172.21.15.42:0/1256122017 learned_addr learned my addr 172.21.15.42:0/1256122017 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:19:00.263 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:00.250+0000 7fb107fff700 1 --2- 172.21.15.42:0/1256122017 >> v2:172.21.15.40:3300/0 conn(0x7fb108123370 0x7fb108131da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:00.263 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:00.250+0000 7fb10cef8700 1 --2- 172.21.15.42:0/1256122017 >> v2:172.21.15.42:3300/0 conn(0x7fb1080a5120 0x7fb108138960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:00.263 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:00.250+0000 7fb104ff9700 1 -- 172.21.15.42:0/1256122017 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3776612416 0 0) 0x7fb10811bdd0 con 0x7fb1080a4cf0 2019-12-05T06:19:00.263 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:00.250+0000 7fb104ff9700 1 -- 172.21.15.42:0/1256122017 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0e80030b0 con 0x7fb1080a4cf0 2019-12-05T06:19:00.263 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:00.250+0000 7fb107fff700 1 -- 172.21.15.42:0/1256122017 >> v2:172.21.15.42:3300/0 conn(0x7fb1080a5120 msgr2=0x7fb108138960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:00.264 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:00.250+0000 7fb107fff700 1 --2- 172.21.15.42:0/1256122017 >> v2:172.21.15.42:3300/0 conn(0x7fb1080a5120 0x7fb108138960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:00.264 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:00.250+0000 7fb107fff700 1 -- 172.21.15.42:0/1256122017 >> v1:172.21.15.42:6789/0 conn(0x7fb1080a4cf0 legacy=0x7fb10811b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:00.264 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:00.254+0000 7fb107fff700 1 -- 172.21.15.42:0/1256122017 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1081322f0 con 0x7fb108123370 2019-12-05T06:19:00.264 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:00.254+0000 7fb107fff700 1 --2- 172.21.15.42:0/1256122017 >> v2:172.21.15.40:3300/0 conn(0x7fb108123370 0x7fb108131da0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fb0f8002f30 tx=0x7fb0f80109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:00.265 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:00.254+0000 7fb107fff700 1 --2- 172.21.15.42:0/1256122017 >> v2:172.21.15.40:3300/0 conn(0x7fb108123370 0x7fb108131da0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fb0f8002f30 tx=0x7fb0f80109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:00.265 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:00.254+0000 7fb107fff700 1 --2- 172.21.15.42:0/1256122017 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb108123370 0x7fb108131da0 secure :-1 s=READY pgs=1830 cs=0 l=1 rx=0x7fb0f8002f30 tx=0x7fb0f80109e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:19:00.265 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:00.254+0000 7fb107fff700 1 --2- 172.21.15.42:0/1256122017 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb108123370 0x7fb108131da0 secure :-1 s=THROTTLE_DONE pgs=1830 cs=0 l=1 rx=0x7fb0f8002f30 tx=0x7fb0f80109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:00.265 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:00.254+0000 7fb104ff9700 1 -- 172.21.15.42:0/1256122017 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb0f80059c0 con 0x7fb108123370 2019-12-05T06:19:00.265 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:00.254+0000 7fb107fff700 1 --2- 172.21.15.42:0/1256122017 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb108123370 0x7fb108131da0 secure :-1 s=THROTTLE_DONE pgs=1830 cs=0 l=1 rx=0x7fb0f8002f30 tx=0x7fb0f80109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:00.266 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:00.254+0000 7fb107fff700 1 --2- 172.21.15.42:0/1256122017 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb108123370 0x7fb108131da0 secure :-1 s=THROTTLE_DONE pgs=1830 cs=0 l=1 rx=0x7fb0f8002f30 tx=0x7fb0f80109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:00.266 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:00.254+0000 7fb104ff9700 1 -- 172.21.15.42:0/1256122017 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0f80055e0 con 0x7fb108123370 2019-12-05T06:19:00.267 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:00.254+0000 7fb104ff9700 1 -- 172.21.15.42:0/1256122017 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb0f8012510 con 0x7fb108123370 2019-12-05T06:19:00.267 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:00.254+0000 7fb10e15a700 1 -- 172.21.15.42:0/1256122017 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb108121930 con 0x7fb108123370 2019-12-05T06:19:00.267 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:00.254+0000 7fb10e15a700 1 -- 172.21.15.42:0/1256122017 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb108131510 con 0x7fb108123370 2019-12-05T06:19:00.267 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:00.254+0000 7fb107fff700 1 --2- 172.21.15.42:0/1256122017 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb108123370 0x7fb108131da0 secure :-1 s=THROTTLE_DONE pgs=1830 cs=0 l=1 rx=0x7fb0f8002f30 tx=0x7fb0f80109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:00.267 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:00.254+0000 7fb104ff9700 1 -- 172.21.15.42:0/1256122017 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fb0f8012820 con 0x7fb108123370 2019-12-05T06:19:00.267 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:00.254+0000 7fb107fff700 1 --2- 172.21.15.42:0/1256122017 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb108123370 0x7fb108131da0 secure :-1 s=THROTTLE_DONE pgs=1830 cs=0 l=1 rx=0x7fb0f8002f30 tx=0x7fb0f80109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:00.268 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:00.254+0000 7fb104ff9700 1 --2- 172.21.15.42:0/1256122017 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb0e803b2a0 0x7fb0e803d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:19:00.268 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:00.254+0000 7fb104ff9700 1 -- 172.21.15.42:0/1256122017 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 10417+0+0 (secure 0 0 0) 0x7fb0f804ac20 con 0x7fb108123370 2019-12-05T06:19:00.268 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:00.254+0000 7fb1077fe700 1 --2- 172.21.15.42:0/1256122017 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb0e803b2a0 0x7fb0e803d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:00.268 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:00.254+0000 7fb1077fe700 1 --2- 172.21.15.42:0/1256122017 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb0e803b2a0 0x7fb0e803d7b0 crc :-1 s=READY pgs=1012 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:19:00.269 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:00.258+0000 7fb10e15a700 1 -- 172.21.15.42:0/1256122017 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb10805d110 con 0x7fb108123370 2019-12-05T06:19:00.272 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:00.270+0000 7fb107fff700 1 --2- 172.21.15.42:0/1256122017 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb108123370 0x7fb108131da0 secure :-1 s=THROTTLE_DONE pgs=1830 cs=0 l=1 rx=0x7fb0f8002f30 tx=0x7fb0f80109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:00.272 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:00.270+0000 7fb104ff9700 1 -- 172.21.15.42:0/1256122017 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fb0f8031ca0 con 0x7fb108123370 2019-12-05T06:19:00.441 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:00.438+0000 7fb10e15a700 1 -- 172.21.15.42:0/1256122017 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb10805b8c0 con 0x7fb108123370 2019-12-05T06:19:00.442 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:00.438+0000 7fb107fff700 1 --2- 172.21.15.42:0/1256122017 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb108123370 0x7fb108131da0 secure :-1 s=THROTTLE_DONE pgs=1830 cs=0 l=1 rx=0x7fb0f8002f30 tx=0x7fb0f80109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:00.442 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:00.438+0000 7fb104ff9700 1 -- 172.21.15.42:0/1256122017 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 288 v288) v1 ==== 95+0+2114 (secure 0 0 0) 0x7fb0f8031f00 con 0x7fb108123370 2019-12-05T06:19:00.444 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 288 2019-12-05T06:19:00.444 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:19:00.444 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":288,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":10476,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/1473620421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1473620421},{"type":"v1","addr":"172.21.15.40:6833","nonce":1473620421}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":288}],"filesystems":[{"mdsmap":{"epoch":287,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:17:04.753930+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":100,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7295},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7295":{"gid":7295,"name":"a","rank":0,"incarnation":284,"state":"up:active","state_seq":38,"addr":"172.21.15.42:6835/3228549071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3228549071},{"type":"v1","addr":"172.21.15.42:6835","nonce":3228549071}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:19:00.447 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:00.442+0000 7fb10e15a700 1 -- 172.21.15.42:0/1256122017 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb0e803b2a0 msgr2=0x7fb0e803d7b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:00.447 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:00.442+0000 7fb10e15a700 1 --2- 172.21.15.42:0/1256122017 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb0e803b2a0 0x7fb0e803d7b0 crc :-1 s=READY pgs=1012 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:00.447 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:00.442+0000 7fb10e15a700 1 -- 172.21.15.42:0/1256122017 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb108123370 msgr2=0x7fb108131da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:00.447 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:00.442+0000 7fb10e15a700 1 --2- 172.21.15.42:0/1256122017 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb108123370 0x7fb108131da0 secure :-1 s=READY pgs=1830 cs=0 l=1 rx=0x7fb0f8002f30 tx=0x7fb0f80109e0).stop 2019-12-05T06:19:00.447 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:00.442+0000 7fb10e15a700 1 -- 172.21.15.42:0/1256122017 shutdown_connections 2019-12-05T06:19:00.447 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:00.442+0000 7fb10e15a700 1 --2- 172.21.15.42:0/1256122017 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb0e803b2a0 0x7fb0e803d7b0 unknown :-1 s=CLOSED pgs=1012 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:00.448 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:00.442+0000 7fb10e15a700 1 --2- 172.21.15.42:0/1256122017 >> v2:172.21.15.42:3300/0 conn(0x7fb1080a5120 0x7fb108138960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:00.448 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:00.442+0000 7fb10e15a700 1 --2- 172.21.15.42:0/1256122017 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb108123370 0x7fb108131da0 unknown :-1 s=CLOSED pgs=1830 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:00.448 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:00.442+0000 7fb10e15a700 1 -- 172.21.15.42:0/1256122017 shutdown_connections 2019-12-05T06:19:00.455 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:00.442+0000 7fb10e15a700 1 -- 172.21.15.42:0/1256122017 wait complete. 2019-12-05T06:19:00.455 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:00.442+0000 7fb10e15a700 1 -- 172.21.15.42:0/1256122017 >> 172.21.15.42:0/1256122017 conn(0x7fb1080a9790 msgr2=0x7fb10811c610 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:19:00.456 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:19:02.457 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:19:02.561 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:02.550+0000 7fa71704e700 1 Processor -- start 2019-12-05T06:19:02.561 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:02.550+0000 7fa71704e700 1 -- start start 2019-12-05T06:19:02.561 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:02.554+0000 7fa71704e700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa7100a7740 0x7fa71008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:19:02.562 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:02.554+0000 7fa71704e700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7100a0710 con 0x7fa71009fae0 2019-12-05T06:19:02.562 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:02.554+0000 7fa71704e700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7100a9ba0 con 0x7fa71007e720 2019-12-05T06:19:02.562 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:02.554+0000 7fa71704e700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7100a0b00 con 0x7fa7100a7740 2019-12-05T06:19:02.562 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:02.554+0000 7fa714dea700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fa71007e720 0x7fa71009f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:52716/0 (socket says 172.21.15.42:52716) 2019-12-05T06:19:02.562 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:02.554+0000 7fa714dea700 1 -- 172.21.15.42:0/1360388142 learned_addr learned my addr 172.21.15.42:0/1360388142 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:19:02.562 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:02.554+0000 7fa7155eb700 1 --2- 172.21.15.42:0/1360388142 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa7100a7740 0x7fa71008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:02.562 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:02.554+0000 7fa707fff700 1 -- 172.21.15.42:0/1360388142 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 633800950 0 0) 0x7fa7100a9ba0 con 0x7fa71007e720 2019-12-05T06:19:02.563 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:02.554+0000 7fa707fff700 1 -- 172.21.15.42:0/1360388142 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6fc0030b0 con 0x7fa71007e720 2019-12-05T06:19:02.563 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:02.554+0000 7fa707fff700 1 -- 172.21.15.42:0/1360388142 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1277651436 0 0) 0x7fa7100a0710 con 0x7fa71009fae0 2019-12-05T06:19:02.563 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:02.554+0000 7fa707fff700 1 -- 172.21.15.42:0/1360388142 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7100a9ba0 con 0x7fa71009fae0 2019-12-05T06:19:02.563 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:02.554+0000 7fa7155eb700 1 -- 172.21.15.42:0/1360388142 >> v1:172.21.15.42:6789/0 conn(0x7fa71007e720 legacy=0x7fa71009f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:02.563 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:02.554+0000 7fa7155eb700 1 -- 172.21.15.42:0/1360388142 >> v1:172.21.15.40:6789/0 conn(0x7fa71009fae0 legacy=0x7fa71014efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:02.563 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:02.554+0000 7fa7155eb700 1 -- 172.21.15.42:0/1360388142 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa71007eb50 con 0x7fa7100a7740 2019-12-05T06:19:02.563 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:02.554+0000 7fa7155eb700 1 --2- 172.21.15.42:0/1360388142 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa7100a7740 0x7fa71008df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fa70c0032b0 tx=0x7fa70c00ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:02.564 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:02.554+0000 7fa7155eb700 1 --2- 172.21.15.42:0/1360388142 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa7100a7740 0x7fa71008df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fa70c0032b0 tx=0x7fa70c00ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:02.564 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:02.554+0000 7fa7155eb700 1 --2- 172.21.15.42:0/1360388142 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa7100a7740 0x7fa71008df80 secure :-1 s=READY pgs=531 cs=0 l=1 rx=0x7fa70c0032b0 tx=0x7fa70c00ff70).ready entity=mon.2 client_cookie=da9bc33d65f25901 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:19:02.564 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:02.554+0000 7fa7155eb700 1 --2- 172.21.15.42:0/1360388142 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa7100a7740 0x7fa71008df80 secure :-1 s=THROTTLE_DONE pgs=531 cs=0 l=1 rx=0x7fa70c0032b0 tx=0x7fa70c00ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:02.564 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:02.554+0000 7fa707fff700 1 -- 172.21.15.42:0/1360388142 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa70c011a10 con 0x7fa7100a7740 2019-12-05T06:19:02.564 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:02.554+0000 7fa7155eb700 1 --2- 172.21.15.42:0/1360388142 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa7100a7740 0x7fa71008df80 secure :-1 s=THROTTLE_DONE pgs=531 cs=0 l=1 rx=0x7fa70c0032b0 tx=0x7fa70c00ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:02.564 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:02.554+0000 7fa7155eb700 1 --2- 172.21.15.42:0/1360388142 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa7100a7740 0x7fa71008df80 secure :-1 s=THROTTLE_DONE pgs=531 cs=0 l=1 rx=0x7fa70c0032b0 tx=0x7fa70c00ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:02.564 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:02.554+0000 7fa707fff700 1 -- 172.21.15.42:0/1360388142 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa70c011630 con 0x7fa7100a7740 2019-12-05T06:19:02.565 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:02.554+0000 7fa707fff700 1 -- 172.21.15.42:0/1360388142 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa70c00a160 con 0x7fa7100a7740 2019-12-05T06:19:02.565 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:02.554+0000 7fa71704e700 1 -- 172.21.15.42:0/1360388142 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa7100a7740 msgr2=0x7fa71008df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:02.565 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:02.554+0000 7fa71704e700 1 --2- 172.21.15.42:0/1360388142 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa7100a7740 0x7fa71008df80 secure :-1 s=READY pgs=531 cs=0 l=1 rx=0x7fa70c0032b0 tx=0x7fa70c00ff70).stop 2019-12-05T06:19:02.565 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:02.558+0000 7fa71704e700 1 -- 172.21.15.42:0/1360388142 shutdown_connections 2019-12-05T06:19:02.565 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:02.558+0000 7fa71704e700 1 --2- 172.21.15.42:0/1360388142 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa7100a7740 0x7fa71008df80 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:02.565 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:02.558+0000 7fa71704e700 1 -- 172.21.15.42:0/1360388142 shutdown_connections 2019-12-05T06:19:02.565 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:02.558+0000 7fa71704e700 1 -- 172.21.15.42:0/1360388142 wait complete. 2019-12-05T06:19:02.566 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:02.558+0000 7fa71704e700 1 -- 172.21.15.42:0/1360388142 >> 172.21.15.42:0/1360388142 conn(0x7fa7100aabe0 msgr2=0x7fa7100a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:19:02.568 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:02.558+0000 7fa71704e700 1 Processor -- start 2019-12-05T06:19:02.568 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:02.558+0000 7fa71704e700 1 -- start start 2019-12-05T06:19:02.568 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:02.562+0000 7fa71704e700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa71007cfb0 0x7fa710125860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:19:02.568 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:02.562+0000 7fa71704e700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa7100a5480 0x7fa7100a6600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:19:02.568 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:02.562+0000 7fa71704e700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7100a6b50 con 0x7fa710142ff0 2019-12-05T06:19:02.568 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:02.562+0000 7fa71704e700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa710145280 con 0x7fa71007cfb0 2019-12-05T06:19:02.569 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:02.562+0000 7fa71704e700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7100a5e20 con 0x7fa7100a5480 2019-12-05T06:19:02.569 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:02.562+0000 7fa715dec700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa7100a5480 0x7fa7100a6600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:02.569 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:02.562+0000 7fa715dec700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa7100a5480 0x7fa7100a6600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:53152/0 (socket says 172.21.15.42:53152) 2019-12-05T06:19:02.569 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:02.562+0000 7fa715dec700 1 -- 172.21.15.42:0/2221687960 learned_addr learned my addr 172.21.15.42:0/2221687960 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:19:02.569 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:02.562+0000 7fa7155eb700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa71007cfb0 0x7fa710125860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:02.569 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:02.562+0000 7fa715dec700 1 -- 172.21.15.42:0/2221687960 >> v2:172.21.15.40:3300/0 conn(0x7fa71007cfb0 msgr2=0x7fa710125860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:02.570 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:02.562+0000 7fa715dec700 1 --2- 172.21.15.42:0/2221687960 >> v2:172.21.15.40:3300/0 conn(0x7fa71007cfb0 0x7fa710125860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:02.570 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:02.562+0000 7fa715dec700 1 -- 172.21.15.42:0/2221687960 >> v1:172.21.15.40:6789/0 conn(0x7fa710142ff0 legacy=0x7fa710144a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:02.570 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:02.562+0000 7fa715dec700 1 -- 172.21.15.42:0/2221687960 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7100a58b0 con 0x7fa7100a5480 2019-12-05T06:19:02.570 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:02.562+0000 7fa715dec700 1 --2- 172.21.15.42:0/2221687960 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa7100a5480 0x7fa7100a6600 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fa708000e90 tx=0x7fa7080107a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:02.570 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:02.562+0000 7fa715dec700 1 --2- 172.21.15.42:0/2221687960 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa7100a5480 0x7fa7100a6600 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fa708000e90 tx=0x7fa7080107a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:02.570 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:02.562+0000 7fa715dec700 1 --2- 172.21.15.42:0/2221687960 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa7100a5480 0x7fa7100a6600 secure :-1 s=READY pgs=532 cs=0 l=1 rx=0x7fa708000e90 tx=0x7fa7080107a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:19:02.570 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:02.562+0000 7fa715dec700 1 --2- 172.21.15.42:0/2221687960 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa7100a5480 0x7fa7100a6600 secure :-1 s=THROTTLE_DONE pgs=532 cs=0 l=1 rx=0x7fa708000e90 tx=0x7fa7080107a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:02.571 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:02.562+0000 7fa715dec700 1 --2- 172.21.15.42:0/2221687960 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa7100a5480 0x7fa7100a6600 secure :-1 s=THROTTLE_DONE pgs=532 cs=0 l=1 rx=0x7fa708000e90 tx=0x7fa7080107a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:02.571 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:02.562+0000 7fa715dec700 1 --2- 172.21.15.42:0/2221687960 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa7100a5480 0x7fa7100a6600 secure :-1 s=THROTTLE_DONE pgs=532 cs=0 l=1 rx=0x7fa708000e90 tx=0x7fa7080107a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:02.571 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:02.562+0000 7fa705ffb700 1 -- 172.21.15.42:0/2221687960 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa708006a40 con 0x7fa7100a5480 2019-12-05T06:19:02.571 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:02.562+0000 7fa705ffb700 1 -- 172.21.15.42:0/2221687960 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa708006660 con 0x7fa7100a5480 2019-12-05T06:19:02.571 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:02.562+0000 7fa71704e700 1 -- 172.21.15.42:0/2221687960 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa71012ccb0 con 0x7fa7100a5480 2019-12-05T06:19:02.571 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:02.562+0000 7fa705ffb700 1 -- 172.21.15.42:0/2221687960 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa708007640 con 0x7fa7100a5480 2019-12-05T06:19:02.571 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:02.562+0000 7fa71704e700 1 -- 172.21.15.42:0/2221687960 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa71012d1e0 con 0x7fa7100a5480 2019-12-05T06:19:02.572 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:02.562+0000 7fa715dec700 1 --2- 172.21.15.42:0/2221687960 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa7100a5480 0x7fa7100a6600 secure :-1 s=THROTTLE_DONE pgs=532 cs=0 l=1 rx=0x7fa708000e90 tx=0x7fa7080107a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:02.572 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:02.566+0000 7fa705ffb700 1 -- 172.21.15.42:0/2221687960 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fa708007950 con 0x7fa7100a5480 2019-12-05T06:19:02.572 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:02.566+0000 7fa715dec700 1 --2- 172.21.15.42:0/2221687960 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa7100a5480 0x7fa7100a6600 secure :-1 s=THROTTLE_DONE pgs=532 cs=0 l=1 rx=0x7fa708000e90 tx=0x7fa7080107a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:02.573 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:02.566+0000 7fa705ffb700 1 --2- 172.21.15.42:0/2221687960 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa6f003ad50 0x7fa6f003d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:19:02.573 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:02.566+0000 7fa705ffb700 1 -- 172.21.15.42:0/2221687960 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 10417+0+0 (secure 0 0 0) 0x7fa70804aef0 con 0x7fa7100a5480 2019-12-05T06:19:02.573 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:02.566+0000 7fa7155eb700 1 --2- 172.21.15.42:0/2221687960 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa6f003ad50 0x7fa6f003d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:02.573 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:02.566+0000 7fa7155eb700 1 --2- 172.21.15.42:0/2221687960 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa6f003ad50 0x7fa6f003d260 crc :-1 s=READY pgs=1013 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:19:02.573 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:02.566+0000 7fa71704e700 1 -- 172.21.15.42:0/2221687960 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa71005d110 con 0x7fa7100a5480 2019-12-05T06:19:02.579 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:02.574+0000 7fa715dec700 1 --2- 172.21.15.42:0/2221687960 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa7100a5480 0x7fa7100a6600 secure :-1 s=THROTTLE_DONE pgs=532 cs=0 l=1 rx=0x7fa708000e90 tx=0x7fa7080107a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:02.579 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:02.574+0000 7fa705ffb700 1 -- 172.21.15.42:0/2221687960 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fa708031f70 con 0x7fa7100a5480 2019-12-05T06:19:02.747 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:02.742+0000 7fa71704e700 1 -- 172.21.15.42:0/2221687960 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa71005b8c0 con 0x7fa7100a5480 2019-12-05T06:19:02.748 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:02.746+0000 7fa715dec700 1 --2- 172.21.15.42:0/2221687960 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa7100a5480 0x7fa7100a6600 secure :-1 s=THROTTLE_DONE pgs=532 cs=0 l=1 rx=0x7fa708000e90 tx=0x7fa7080107a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:02.748 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:02.746+0000 7fa705ffb700 1 -- 172.21.15.42:0/2221687960 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 288 v288) v1 ==== 95+0+2114 (secure 0 0 0) 0x7fa7080321d0 con 0x7fa7100a5480 2019-12-05T06:19:02.750 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:19:02.750 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":288,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":10476,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/1473620421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1473620421},{"type":"v1","addr":"172.21.15.40:6833","nonce":1473620421}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":288}],"filesystems":[{"mdsmap":{"epoch":287,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:17:04.753930+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":100,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7295},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7295":{"gid":7295,"name":"a","rank":0,"incarnation":284,"state":"up:active","state_seq":38,"addr":"172.21.15.42:6835/3228549071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3228549071},{"type":"v1","addr":"172.21.15.42:6835","nonce":3228549071}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:19:02.750 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 288 2019-12-05T06:19:02.760 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:02.750+0000 7fa71704e700 1 -- 172.21.15.42:0/2221687960 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa6f003ad50 msgr2=0x7fa6f003d260 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:02.761 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:02.750+0000 7fa71704e700 1 --2- 172.21.15.42:0/2221687960 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa6f003ad50 0x7fa6f003d260 crc :-1 s=READY pgs=1013 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:02.761 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:02.750+0000 7fa71704e700 1 -- 172.21.15.42:0/2221687960 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa7100a5480 msgr2=0x7fa7100a6600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:02.761 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:02.750+0000 7fa71704e700 1 --2- 172.21.15.42:0/2221687960 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa7100a5480 0x7fa7100a6600 secure :-1 s=READY pgs=532 cs=0 l=1 rx=0x7fa708000e90 tx=0x7fa7080107a0).stop 2019-12-05T06:19:02.761 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:02.750+0000 7fa71704e700 1 -- 172.21.15.42:0/2221687960 shutdown_connections 2019-12-05T06:19:02.761 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:02.750+0000 7fa71704e700 1 --2- 172.21.15.42:0/2221687960 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa6f003ad50 0x7fa6f003d260 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:02.761 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:02.750+0000 7fa71704e700 1 --2- 172.21.15.42:0/2221687960 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa7100a5480 0x7fa7100a6600 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:02.761 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:02.750+0000 7fa71704e700 1 --2- 172.21.15.42:0/2221687960 >> v2:172.21.15.40:3300/0 conn(0x7fa71007cfb0 0x7fa710125860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:02.762 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:02.750+0000 7fa71704e700 1 -- 172.21.15.42:0/2221687960 shutdown_connections 2019-12-05T06:19:02.762 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:02.750+0000 7fa71704e700 1 -- 172.21.15.42:0/2221687960 wait complete. 2019-12-05T06:19:02.762 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:02.750+0000 7fa71704e700 1 -- 172.21.15.42:0/2221687960 >> 172.21.15.42:0/2221687960 conn(0x7fa7100a7740 msgr2=0x7fa710059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:19:02.763 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:19:04.765 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:19:04.876 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:04.866+0000 7f600c9c5700 1 Processor -- start 2019-12-05T06:19:04.876 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:04.866+0000 7f600c9c5700 1 -- start start 2019-12-05T06:19:04.877 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:04.870+0000 7f600c9c5700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f60080a7730 0x7f600808df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:19:04.877 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:04.870+0000 7f600c9c5700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f600807e710 0x7f600807eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:19:04.877 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:04.870+0000 7f600c9c5700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f600809f2d0 0x7f600809f700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:19:04.877 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:04.870+0000 7f600c9c5700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60080a0af0 con 0x7f600809f2d0 2019-12-05T06:19:04.877 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:04.870+0000 7f600c9c5700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f600808e5a0 con 0x7f60080a7730 2019-12-05T06:19:04.877 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:04.870+0000 7f600c9c5700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f600808e840 con 0x7f600807e710 2019-12-05T06:19:04.878 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:04.870+0000 7f6006d9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f60080a7730 0x7f600808df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:04.878 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:04.870+0000 7f6006d9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f60080a7730 0x7f600808df70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:52692/0 (socket says 172.21.15.42:52692) 2019-12-05T06:19:04.878 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:04.870+0000 7f6006d9d700 1 -- 172.21.15.42:0/3226000326 learned_addr learned my addr 172.21.15.42:0/3226000326 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:19:04.878 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:04.870+0000 7f600659c700 1 --2- 172.21.15.42:0/3226000326 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f600807e710 0x7f600807eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:04.879 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:04.870+0000 7f600759e700 1 --2- 172.21.15.42:0/3226000326 >> v2:172.21.15.40:3300/0 conn(0x7f600809f2d0 0x7f600809f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:04.879 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:04.870+0000 7f6006d9d700 1 -- 172.21.15.42:0/3226000326 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f600807e710 msgr2=0x7f600807eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:19:04.879 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:04.870+0000 7f6006d9d700 1 --2- 172.21.15.42:0/3226000326 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f600807e710 0x7f600807eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:19:04.879 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:04.870+0000 7f6006d9d700 1 -- 172.21.15.42:0/3226000326 >> v2:172.21.15.40:3300/0 conn(0x7f600809f2d0 msgr2=0x7f600809f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:19:04.879 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:04.870+0000 7f6006d9d700 1 --2- 172.21.15.42:0/3226000326 >> v2:172.21.15.40:3300/0 conn(0x7f600809f2d0 0x7f600809f700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:19:04.879 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:04.870+0000 7f6006d9d700 1 -- 172.21.15.42:0/3226000326 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f600807f090 con 0x7f60080a7730 2019-12-05T06:19:04.880 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:04.870+0000 7f600759e700 1 --2- 172.21.15.42:0/3226000326 >> v2:172.21.15.40:3300/0 conn(0x7f600809f2d0 0x7f600809f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:19:04.880 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:04.870+0000 7f6006d9d700 1 --2- 172.21.15.42:0/3226000326 >> v2:172.21.15.42:3300/0 conn(0x7f60080a7730 0x7f600808df70 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f600808e760 tx=0x7f5ff800fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:04.880 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:04.870+0000 7f6006d9d700 1 --2- 172.21.15.42:0/3226000326 >> v2:172.21.15.42:3300/0 conn(0x7f60080a7730 0x7f600808df70 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f600808e760 tx=0x7f5ff800fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:04.880 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:04.870+0000 7f6006d9d700 1 --2- 172.21.15.42:0/3226000326 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f60080a7730 0x7f600808df70 secure :-1 s=READY pgs=2408 cs=0 l=1 rx=0x7f600808e760 tx=0x7f5ff800fdb0).ready entity=mon.1 client_cookie=818989e9535668c4 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:19:04.880 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:04.870+0000 7f6006d9d700 1 --2- 172.21.15.42:0/3226000326 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f60080a7730 0x7f600808df70 secure :-1 s=THROTTLE_DONE pgs=2408 cs=0 l=1 rx=0x7f600808e760 tx=0x7f5ff800fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:04.881 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:04.870+0000 7f6006d9d700 1 --2- 172.21.15.42:0/3226000326 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f60080a7730 0x7f600808df70 secure :-1 s=THROTTLE_DONE pgs=2408 cs=0 l=1 rx=0x7f600808e760 tx=0x7f5ff800fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:04.881 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:04.870+0000 7f6005d9b700 1 -- 172.21.15.42:0/3226000326 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5ff8008c90 con 0x7f60080a7730 2019-12-05T06:19:04.881 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:04.870+0000 7f6006d9d700 1 --2- 172.21.15.42:0/3226000326 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f60080a7730 0x7f600808df70 secure :-1 s=THROTTLE_DONE pgs=2408 cs=0 l=1 rx=0x7f600808e760 tx=0x7f5ff800fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:04.881 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:04.870+0000 7f6005d9b700 1 -- 172.21.15.42:0/3226000326 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ff80088b0 con 0x7f60080a7730 2019-12-05T06:19:04.881 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:04.870+0000 7f6005d9b700 1 -- 172.21.15.42:0/3226000326 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5ff8011b50 con 0x7f60080a7730 2019-12-05T06:19:04.881 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:04.870+0000 7f600c9c5700 1 -- 172.21.15.42:0/3226000326 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f60080a7730 msgr2=0x7f600808df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:04.882 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:04.870+0000 7f600c9c5700 1 --2- 172.21.15.42:0/3226000326 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f60080a7730 0x7f600808df70 secure :-1 s=READY pgs=2408 cs=0 l=1 rx=0x7f600808e760 tx=0x7f5ff800fdb0).stop 2019-12-05T06:19:04.882 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:04.870+0000 7f600c9c5700 1 -- 172.21.15.42:0/3226000326 shutdown_connections 2019-12-05T06:19:04.882 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:04.870+0000 7f600c9c5700 1 --2- 172.21.15.42:0/3226000326 >> v2:172.21.15.40:3300/0 conn(0x7f600809f2d0 0x7f600809f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:19:04.882 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:04.870+0000 7f600c9c5700 1 --2- 172.21.15.42:0/3226000326 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f60080a7730 0x7f600808df70 unknown :-1 s=CLOSED pgs=2408 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:04.883 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:04.870+0000 7f600c9c5700 1 --2- 172.21.15.42:0/3226000326 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f600807e710 0x7f600807eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:19:04.883 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:04.870+0000 7f600c9c5700 1 -- 172.21.15.42:0/3226000326 shutdown_connections 2019-12-05T06:19:04.883 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:04.870+0000 7f600c9c5700 1 -- 172.21.15.42:0/3226000326 wait complete. 2019-12-05T06:19:04.883 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:04.870+0000 7f600c9c5700 1 -- 172.21.15.42:0/3226000326 >> 172.21.15.42:0/3226000326 conn(0x7f60080aabd0 msgr2=0x7f60080a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:19:04.887 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:04.874+0000 7f600c9c5700 1 Processor -- start 2019-12-05T06:19:04.887 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:04.874+0000 7f600c9c5700 1 -- start start 2019-12-05T06:19:04.888 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:04.874+0000 7f600c9c5700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f60081245d0 0x7f6008144e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:19:04.888 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:04.874+0000 7f600c9c5700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6008135710 0x7f6008137b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:19:04.888 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:04.874+0000 7f600c9c5700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f600811ea10 con 0x7f600807db40 2019-12-05T06:19:04.888 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:04.874+0000 7f600c9c5700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6008142d20 con 0x7f6008135710 2019-12-05T06:19:04.889 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:04.874+0000 7f600c9c5700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60081434c0 con 0x7f60081245d0 2019-12-05T06:19:04.889 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:04.874+0000 7f600759e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6008135710 0x7f6008137b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:04.889 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:04.874+0000 7f6006d9d700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f60081245d0 0x7f6008144e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:04.890 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:04.874+0000 7f600759e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6008135710 0x7f6008137b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:52702/0 (socket says 172.21.15.42:52702) 2019-12-05T06:19:04.890 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:04.874+0000 7f600759e700 1 -- 172.21.15.42:0/3663802393 learned_addr learned my addr 172.21.15.42:0/3663802393 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:19:04.890 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:04.878+0000 7f5fff7fe700 1 -- 172.21.15.42:0/3663802393 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1896061086 0 0) 0x7f600811ea10 con 0x7f600807db40 2019-12-05T06:19:04.890 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:04.878+0000 7f5fff7fe700 1 -- 172.21.15.42:0/3663802393 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5fec0030b0 con 0x7f600807db40 2019-12-05T06:19:04.890 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:04.878+0000 7f600759e700 1 -- 172.21.15.42:0/3663802393 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f60081245d0 msgr2=0x7f6008144e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:04.890 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:04.878+0000 7f600759e700 1 --2- 172.21.15.42:0/3663802393 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f60081245d0 0x7f6008144e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:04.890 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:04.878+0000 7f600759e700 1 -- 172.21.15.42:0/3663802393 >> v1:172.21.15.40:6789/0 conn(0x7f600807db40 legacy=0x7f6008142520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:04.891 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:04.878+0000 7f600759e700 1 -- 172.21.15.42:0/3663802393 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6008143a00 con 0x7f6008135710 2019-12-05T06:19:04.891 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:04.878+0000 7f6006d9d700 1 --2- 172.21.15.42:0/3663802393 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f60081245d0 0x7f6008144e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:19:04.891 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:04.878+0000 7f600759e700 1 --2- 172.21.15.42:0/3663802393 >> v2:172.21.15.42:3300/0 conn(0x7f6008135710 0x7f6008137b50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f6008143680 tx=0x7f6000010f00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:04.891 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:04.878+0000 7f600759e700 1 --2- 172.21.15.42:0/3663802393 >> v2:172.21.15.42:3300/0 conn(0x7f6008135710 0x7f6008137b50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f6008143680 tx=0x7f6000010f00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:04.891 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:04.878+0000 7f600759e700 1 --2- 172.21.15.42:0/3663802393 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6008135710 0x7f6008137b50 secure :-1 s=READY pgs=2409 cs=0 l=1 rx=0x7f6008143680 tx=0x7f6000010f00).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:19:04.891 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:04.878+0000 7f600759e700 1 --2- 172.21.15.42:0/3663802393 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6008135710 0x7f6008137b50 secure :-1 s=THROTTLE_DONE pgs=2409 cs=0 l=1 rx=0x7f6008143680 tx=0x7f6000010f00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:04.891 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:04.878+0000 7f5fff7fe700 1 -- 172.21.15.42:0/3663802393 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f60000058c0 con 0x7f6008135710 2019-12-05T06:19:04.892 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:04.878+0000 7f600759e700 1 --2- 172.21.15.42:0/3663802393 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6008135710 0x7f6008137b50 secure :-1 s=THROTTLE_DONE pgs=2409 cs=0 l=1 rx=0x7f6008143680 tx=0x7f6000010f00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:04.892 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:04.878+0000 7f5fff7fe700 1 -- 172.21.15.42:0/3663802393 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f600000e160 con 0x7f6008135710 2019-12-05T06:19:04.892 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:04.878+0000 7f600c9c5700 1 -- 172.21.15.42:0/3663802393 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f600807df70 con 0x7f6008135710 2019-12-05T06:19:04.892 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:04.878+0000 7f600759e700 1 --2- 172.21.15.42:0/3663802393 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6008135710 0x7f6008137b50 secure :-1 s=THROTTLE_DONE pgs=2409 cs=0 l=1 rx=0x7f6008143680 tx=0x7f6000010f00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:04.892 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:04.878+0000 7f5fff7fe700 1 -- 172.21.15.42:0/3663802393 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f600000c850 con 0x7f6008135710 2019-12-05T06:19:04.892 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:04.878+0000 7f600c9c5700 1 -- 172.21.15.42:0/3663802393 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f600808eb60 con 0x7f6008135710 2019-12-05T06:19:04.892 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:04.878+0000 7f600759e700 1 --2- 172.21.15.42:0/3663802393 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6008135710 0x7f6008137b50 secure :-1 s=THROTTLE_DONE pgs=2409 cs=0 l=1 rx=0x7f6008143680 tx=0x7f6000010f00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:04.893 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:04.878+0000 7f5fff7fe700 1 -- 172.21.15.42:0/3663802393 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f600000caf0 con 0x7f6008135710 2019-12-05T06:19:04.893 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:04.878+0000 7f600759e700 1 --2- 172.21.15.42:0/3663802393 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6008135710 0x7f6008137b50 secure :-1 s=THROTTLE_DONE pgs=2409 cs=0 l=1 rx=0x7f6008143680 tx=0x7f6000010f00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:04.893 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:04.878+0000 7f5fff7fe700 1 --2- 172.21.15.42:0/3663802393 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5fec03b230 0x7f5fec03d740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:19:04.893 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:04.878+0000 7f5fff7fe700 1 -- 172.21.15.42:0/3663802393 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 10417+0+0 (secure 0 0 0) 0x7f600004b040 con 0x7f6008135710 2019-12-05T06:19:04.893 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:04.878+0000 7f6006d9d700 1 --2- 172.21.15.42:0/3663802393 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5fec03b230 0x7f5fec03d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:04.893 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:04.882+0000 7f6006d9d700 1 --2- 172.21.15.42:0/3663802393 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5fec03b230 0x7f5fec03d740 crc :-1 s=READY pgs=1014 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:19:04.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:04.882+0000 7f600c9c5700 1 -- 172.21.15.42:0/3663802393 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6008144bf0 con 0x7f6008135710 2019-12-05T06:19:04.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:04.890+0000 7f600759e700 1 --2- 172.21.15.42:0/3663802393 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6008135710 0x7f6008137b50 secure :-1 s=THROTTLE_DONE pgs=2409 cs=0 l=1 rx=0x7f6008143680 tx=0x7f6000010f00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:04.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:04.890+0000 7f5fff7fe700 1 -- 172.21.15.42:0/3663802393 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f6000032040 con 0x7f6008135710 2019-12-05T06:19:05.064 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:05.062+0000 7f600c9c5700 1 -- 172.21.15.42:0/3663802393 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f600805d100 con 0x7f6008135710 2019-12-05T06:19:05.065 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:05.062+0000 7f600759e700 1 --2- 172.21.15.42:0/3663802393 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6008135710 0x7f6008137b50 secure :-1 s=THROTTLE_DONE pgs=2409 cs=0 l=1 rx=0x7f6008143680 tx=0x7f6000010f00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:05.065 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:05.062+0000 7f5fff7fe700 1 -- 172.21.15.42:0/3663802393 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 288 v288) v1 ==== 95+0+2114 (secure 0 0 0) 0x7f60000322a0 con 0x7f6008135710 2019-12-05T06:19:05.065 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 288 2019-12-05T06:19:05.066 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:19:05.066 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":288,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":10476,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/1473620421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1473620421},{"type":"v1","addr":"172.21.15.40:6833","nonce":1473620421}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":288}],"filesystems":[{"mdsmap":{"epoch":287,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:17:04.753930+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":100,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7295},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7295":{"gid":7295,"name":"a","rank":0,"incarnation":284,"state":"up:active","state_seq":38,"addr":"172.21.15.42:6835/3228549071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3228549071},{"type":"v1","addr":"172.21.15.42:6835","nonce":3228549071}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:19:05.068 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:05.062+0000 7f600c9c5700 1 -- 172.21.15.42:0/3663802393 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5fec03b230 msgr2=0x7f5fec03d740 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:05.068 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:05.062+0000 7f600c9c5700 1 --2- 172.21.15.42:0/3663802393 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5fec03b230 0x7f5fec03d740 crc :-1 s=READY pgs=1014 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:05.068 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:05.062+0000 7f600c9c5700 1 -- 172.21.15.42:0/3663802393 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6008135710 msgr2=0x7f6008137b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:05.068 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:05.062+0000 7f600c9c5700 1 --2- 172.21.15.42:0/3663802393 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6008135710 0x7f6008137b50 secure :-1 s=READY pgs=2409 cs=0 l=1 rx=0x7f6008143680 tx=0x7f6000010f00).stop 2019-12-05T06:19:05.069 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:05.062+0000 7f600c9c5700 1 -- 172.21.15.42:0/3663802393 shutdown_connections 2019-12-05T06:19:05.069 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:05.062+0000 7f600c9c5700 1 --2- 172.21.15.42:0/3663802393 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5fec03b230 0x7f5fec03d740 unknown :-1 s=CLOSED pgs=1014 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:05.069 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:05.062+0000 7f600c9c5700 1 --2- 172.21.15.42:0/3663802393 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6008135710 0x7f6008137b50 unknown :-1 s=CLOSED pgs=2409 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:05.069 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:05.062+0000 7f600c9c5700 1 --2- 172.21.15.42:0/3663802393 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f60081245d0 0x7f6008144e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:05.069 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:05.062+0000 7f600c9c5700 1 -- 172.21.15.42:0/3663802393 shutdown_connections 2019-12-05T06:19:05.069 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:05.062+0000 7f600c9c5700 1 -- 172.21.15.42:0/3663802393 wait complete. 2019-12-05T06:19:05.070 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:05.062+0000 7f600c9c5700 1 -- 172.21.15.42:0/3663802393 >> 172.21.15.42:0/3663802393 conn(0x7f60080a7730 msgr2=0x7f600811c3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:19:05.077 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:19:07.080 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:19:07.188 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:07.178+0000 7fb079bae700 1 Processor -- start 2019-12-05T06:19:07.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:07.178+0000 7fb079bae700 1 -- start start 2019-12-05T06:19:07.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:07.182+0000 7fb079bae700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb0740a7740 0x7fb07408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:19:07.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:07.182+0000 7fb079bae700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0740a0710 con 0x7fb07409fae0 2019-12-05T06:19:07.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:07.182+0000 7fb079bae700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0740a9ba0 con 0x7fb07407e720 2019-12-05T06:19:07.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:07.182+0000 7fb079bae700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0740a0b00 con 0x7fb0740a7740 2019-12-05T06:19:07.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:07.182+0000 7fb0737fe700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fb07407e720 0x7fb07409f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:52754/0 (socket says 172.21.15.42:52754) 2019-12-05T06:19:07.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:07.182+0000 7fb0737fe700 1 -- 172.21.15.42:0/119126030 learned_addr learned my addr 172.21.15.42:0/119126030 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:19:07.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:07.182+0000 7fb073fff700 1 --2- 172.21.15.42:0/119126030 >> v2:172.21.15.40:3300/0 conn(0x7fb0740a7740 0x7fb07408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:07.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:07.182+0000 7fb072ffd700 1 -- 172.21.15.42:0/119126030 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1889972528 0 0) 0x7fb0740a9ba0 con 0x7fb07407e720 2019-12-05T06:19:07.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:07.182+0000 7fb072ffd700 1 -- 172.21.15.42:0/119126030 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb05c0030b0 con 0x7fb07407e720 2019-12-05T06:19:07.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:07.182+0000 7fb073fff700 1 -- 172.21.15.42:0/119126030 >> v1:172.21.15.42:6789/0 conn(0x7fb07407e720 legacy=0x7fb07409f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:07.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:07.182+0000 7fb073fff700 1 -- 172.21.15.42:0/119126030 >> v1:172.21.15.40:6789/0 conn(0x7fb07409fae0 legacy=0x7fb07414efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:19:07.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:07.182+0000 7fb073fff700 1 -- 172.21.15.42:0/119126030 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb07407eb50 con 0x7fb0740a7740 2019-12-05T06:19:07.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:07.182+0000 7fb073fff700 1 --2- 172.21.15.42:0/119126030 >> v2:172.21.15.40:3300/0 conn(0x7fb0740a7740 0x7fb07408df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fb064008b10 tx=0x7fb06400ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:07.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:07.182+0000 7fb073fff700 1 --2- 172.21.15.42:0/119126030 >> v2:172.21.15.40:3300/0 conn(0x7fb0740a7740 0x7fb07408df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fb064008b10 tx=0x7fb06400ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:07.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:07.182+0000 7fb073fff700 1 --2- 172.21.15.42:0/119126030 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb0740a7740 0x7fb07408df80 secure :-1 s=READY pgs=1836 cs=0 l=1 rx=0x7fb064008b10 tx=0x7fb06400ff70).ready entity=mon.0 client_cookie=40d7c43b27ca332a server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:19:07.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:07.182+0000 7fb073fff700 1 --2- 172.21.15.42:0/119126030 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb0740a7740 0x7fb07408df80 secure :-1 s=THROTTLE_DONE pgs=1836 cs=0 l=1 rx=0x7fb064008b10 tx=0x7fb06400ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:07.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:07.182+0000 7fb073fff700 1 --2- 172.21.15.42:0/119126030 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb0740a7740 0x7fb07408df80 secure :-1 s=THROTTLE_DONE pgs=1836 cs=0 l=1 rx=0x7fb064008b10 tx=0x7fb06400ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:07.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:07.182+0000 7fb072ffd700 1 -- 172.21.15.42:0/119126030 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb0640119d0 con 0x7fb0740a7740 2019-12-05T06:19:07.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:07.182+0000 7fb073fff700 1 --2- 172.21.15.42:0/119126030 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb0740a7740 0x7fb07408df80 secure :-1 s=THROTTLE_DONE pgs=1836 cs=0 l=1 rx=0x7fb064008b10 tx=0x7fb06400ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:07.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:07.182+0000 7fb072ffd700 1 -- 172.21.15.42:0/119126030 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0640115f0 con 0x7fb0740a7740 2019-12-05T06:19:07.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:07.182+0000 7fb072ffd700 1 -- 172.21.15.42:0/119126030 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb06400a160 con 0x7fb0740a7740 2019-12-05T06:19:07.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:07.182+0000 7fb079bae700 1 -- 172.21.15.42:0/119126030 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb0740a7740 msgr2=0x7fb07408df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:07.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:07.182+0000 7fb079bae700 1 --2- 172.21.15.42:0/119126030 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb0740a7740 0x7fb07408df80 secure :-1 s=READY pgs=1836 cs=0 l=1 rx=0x7fb064008b10 tx=0x7fb06400ff70).stop 2019-12-05T06:19:07.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:07.182+0000 7fb079bae700 1 -- 172.21.15.42:0/119126030 shutdown_connections 2019-12-05T06:19:07.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:07.182+0000 7fb079bae700 1 --2- 172.21.15.42:0/119126030 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb0740a7740 0x7fb07408df80 unknown :-1 s=CLOSED pgs=1836 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:07.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:07.186+0000 7fb079bae700 1 -- 172.21.15.42:0/119126030 shutdown_connections 2019-12-05T06:19:07.193 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:07.186+0000 7fb079bae700 1 -- 172.21.15.42:0/119126030 wait complete. 2019-12-05T06:19:07.193 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:07.186+0000 7fb079bae700 1 -- 172.21.15.42:0/119126030 >> 172.21.15.42:0/119126030 conn(0x7fb0740aabe0 msgr2=0x7fb0740a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:19:07.195 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:07.186+0000 7fb079bae700 1 Processor -- start 2019-12-05T06:19:07.195 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:07.186+0000 7fb079bae700 1 -- start start 2019-12-05T06:19:07.195 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:07.190+0000 7fb079bae700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb0740a4170 0x7fb074145080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:19:07.196 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:07.190+0000 7fb079bae700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb074142740 0x7fb074121f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:19:07.196 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:07.190+0000 7fb079bae700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb074143840 con 0x7fb07407d5c0 2019-12-05T06:19:07.196 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:07.190+0000 7fb079bae700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb074144b80 con 0x7fb074142740 2019-12-05T06:19:07.196 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:07.190+0000 7fb079bae700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb074122490 con 0x7fb0740a4170 2019-12-05T06:19:07.202 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:07.190+0000 7fb073fff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb0740a4170 0x7fb074145080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:07.203 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:07.190+0000 7fb073fff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb0740a4170 0x7fb074145080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:52720/0 (socket says 172.21.15.42:52720) 2019-12-05T06:19:07.203 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:07.190+0000 7fb073fff700 1 -- 172.21.15.42:0/1554589136 learned_addr learned my addr 172.21.15.42:0/1554589136 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:19:07.203 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:07.190+0000 7fb07894c700 1 --1- 172.21.15.42:0/1554589136 >> v1:172.21.15.42:6789/0 conn(0x7fb07407d5c0 0x7fb07407d9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:52762/0 (socket says 172.21.15.42:52762) 2019-12-05T06:19:07.203 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:07.190+0000 7fb0737fe700 1 --2- 172.21.15.42:0/1554589136 >> v2:172.21.15.40:3300/0 conn(0x7fb074142740 0x7fb074121f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:07.203 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:07.190+0000 7fb073fff700 1 -- 172.21.15.42:0/1554589136 >> v2:172.21.15.40:3300/0 conn(0x7fb074142740 msgr2=0x7fb074121f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:07.203 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:07.190+0000 7fb073fff700 1 --2- 172.21.15.42:0/1554589136 >> v2:172.21.15.40:3300/0 conn(0x7fb074142740 0x7fb074121f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:07.203 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:07.190+0000 7fb073fff700 1 -- 172.21.15.42:0/1554589136 >> v1:172.21.15.42:6789/0 conn(0x7fb07407d5c0 legacy=0x7fb07407d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:07.204 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:07.190+0000 7fb073fff700 1 -- 172.21.15.42:0/1554589136 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0741433c0 con 0x7fb0740a4170 2019-12-05T06:19:07.204 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:07.190+0000 7fb073fff700 1 --2- 172.21.15.42:0/1554589136 >> v2:172.21.15.42:3300/0 conn(0x7fb0740a4170 0x7fb074145080 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fb074150760 tx=0x7fb06400f300).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:07.204 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:07.190+0000 7fb073fff700 1 --2- 172.21.15.42:0/1554589136 >> v2:172.21.15.42:3300/0 conn(0x7fb0740a4170 0x7fb074145080 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fb074150760 tx=0x7fb06400f300).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:07.204 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:07.190+0000 7fb073fff700 1 --2- 172.21.15.42:0/1554589136 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb0740a4170 0x7fb074145080 secure :-1 s=READY pgs=2412 cs=0 l=1 rx=0x7fb074150760 tx=0x7fb06400f300).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:19:07.204 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:07.190+0000 7fb073fff700 1 --2- 172.21.15.42:0/1554589136 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb0740a4170 0x7fb074145080 secure :-1 s=THROTTLE_DONE pgs=2412 cs=0 l=1 rx=0x7fb074150760 tx=0x7fb06400f300).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:07.204 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:07.190+0000 7fb073fff700 1 --2- 172.21.15.42:0/1554589136 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb0740a4170 0x7fb074145080 secure :-1 s=THROTTLE_DONE pgs=2412 cs=0 l=1 rx=0x7fb074150760 tx=0x7fb06400f300).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:07.204 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:07.190+0000 7fb073fff700 1 --2- 172.21.15.42:0/1554589136 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb0740a4170 0x7fb074145080 secure :-1 s=THROTTLE_DONE pgs=2412 cs=0 l=1 rx=0x7fb074150760 tx=0x7fb06400f300).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:07.205 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:07.190+0000 7fb070ff9700 1 -- 172.21.15.42:0/1554589136 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb06400f900 con 0x7fb0740a4170 2019-12-05T06:19:07.205 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:07.190+0000 7fb070ff9700 1 -- 172.21.15.42:0/1554589136 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb064011bd0 con 0x7fb0740a4170 2019-12-05T06:19:07.205 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:07.190+0000 7fb079bae700 1 -- 172.21.15.42:0/1554589136 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb07412c950 con 0x7fb0740a4170 2019-12-05T06:19:07.205 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:07.190+0000 7fb070ff9700 1 -- 172.21.15.42:0/1554589136 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb064005ca0 con 0x7fb0740a4170 2019-12-05T06:19:07.205 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:07.190+0000 7fb079bae700 1 -- 172.21.15.42:0/1554589136 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb07412ce80 con 0x7fb0740a4170 2019-12-05T06:19:07.205 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:07.190+0000 7fb073fff700 1 --2- 172.21.15.42:0/1554589136 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb0740a4170 0x7fb074145080 secure :-1 s=THROTTLE_DONE pgs=2412 cs=0 l=1 rx=0x7fb074150760 tx=0x7fb06400f300).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:07.205 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:07.190+0000 7fb070ff9700 1 -- 172.21.15.42:0/1554589136 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fb064007150 con 0x7fb0740a4170 2019-12-05T06:19:07.206 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:07.190+0000 7fb073fff700 1 --2- 172.21.15.42:0/1554589136 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb0740a4170 0x7fb074145080 secure :-1 s=THROTTLE_DONE pgs=2412 cs=0 l=1 rx=0x7fb074150760 tx=0x7fb06400f300).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:07.207 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:07.194+0000 7fb070ff9700 1 --2- 172.21.15.42:0/1554589136 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb05403ae60 0x7fb05403d370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:19:07.207 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:07.194+0000 7fb070ff9700 1 -- 172.21.15.42:0/1554589136 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 10417+0+0 (secure 0 0 0) 0x7fb06404b520 con 0x7fb0740a4170 2019-12-05T06:19:07.207 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:07.194+0000 7fb0737fe700 1 --2- 172.21.15.42:0/1554589136 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb05403ae60 0x7fb05403d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:07.207 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:07.194+0000 7fb0737fe700 1 --2- 172.21.15.42:0/1554589136 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb05403ae60 0x7fb05403d370 crc :-1 s=READY pgs=1015 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:19:07.207 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:07.194+0000 7fb079bae700 1 -- 172.21.15.42:0/1554589136 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb07405d110 con 0x7fb0740a4170 2019-12-05T06:19:07.207 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:07.202+0000 7fb073fff700 1 --2- 172.21.15.42:0/1554589136 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb0740a4170 0x7fb074145080 secure :-1 s=THROTTLE_DONE pgs=2412 cs=0 l=1 rx=0x7fb074150760 tx=0x7fb06400f300).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:07.207 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:07.202+0000 7fb070ff9700 1 -- 172.21.15.42:0/1554589136 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fb064032800 con 0x7fb0740a4170 2019-12-05T06:19:07.371 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:07.362+0000 7fb079bae700 1 -- 172.21.15.42:0/1554589136 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb07405b8c0 con 0x7fb0740a4170 2019-12-05T06:19:07.371 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:07.362+0000 7fb073fff700 1 --2- 172.21.15.42:0/1554589136 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb0740a4170 0x7fb074145080 secure :-1 s=THROTTLE_DONE pgs=2412 cs=0 l=1 rx=0x7fb074150760 tx=0x7fb06400f300).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:07.371 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:07.362+0000 7fb070ff9700 1 -- 172.21.15.42:0/1554589136 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 288 v288) v1 ==== 95+0+2114 (secure 0 0 0) 0x7fb06400dcc0 con 0x7fb0740a4170 2019-12-05T06:19:07.371 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 288 2019-12-05T06:19:07.371 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:07.366+0000 7fb079bae700 1 -- 172.21.15.42:0/1554589136 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb05403ae60 msgr2=0x7fb05403d370 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:07.372 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:07.366+0000 7fb079bae700 1 --2- 172.21.15.42:0/1554589136 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb05403ae60 0x7fb05403d370 crc :-1 s=READY pgs=1015 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:07.372 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:07.366+0000 7fb079bae700 1 -- 172.21.15.42:0/1554589136 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb0740a4170 msgr2=0x7fb074145080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:07.372 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:07.366+0000 7fb079bae700 1 --2- 172.21.15.42:0/1554589136 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb0740a4170 0x7fb074145080 secure :-1 s=READY pgs=2412 cs=0 l=1 rx=0x7fb074150760 tx=0x7fb06400f300).stop 2019-12-05T06:19:07.372 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:07.366+0000 7fb079bae700 1 -- 172.21.15.42:0/1554589136 shutdown_connections 2019-12-05T06:19:07.372 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:07.366+0000 7fb079bae700 1 --2- 172.21.15.42:0/1554589136 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb05403ae60 0x7fb05403d370 unknown :-1 s=CLOSED pgs=1015 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:07.372 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:07.366+0000 7fb079bae700 1 --2- 172.21.15.42:0/1554589136 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb0740a4170 0x7fb074145080 unknown :-1 s=CLOSED pgs=2412 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:07.372 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:07.366+0000 7fb079bae700 1 --2- 172.21.15.42:0/1554589136 >> v2:172.21.15.40:3300/0 conn(0x7fb074142740 0x7fb074121f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:07.373 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:07.366+0000 7fb079bae700 1 -- 172.21.15.42:0/1554589136 shutdown_connections 2019-12-05T06:19:07.373 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:07.366+0000 7fb079bae700 1 -- 172.21.15.42:0/1554589136 wait complete. 2019-12-05T06:19:07.373 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:07.366+0000 7fb079bae700 1 -- 172.21.15.42:0/1554589136 >> 172.21.15.42:0/1554589136 conn(0x7fb07407ee20 msgr2=0x7fb074059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:19:07.373 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:19:07.373 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":288,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":10476,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/1473620421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1473620421},{"type":"v1","addr":"172.21.15.40:6833","nonce":1473620421}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":288}],"filesystems":[{"mdsmap":{"epoch":287,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:17:04.753930+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":100,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7295},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7295":{"gid":7295,"name":"a","rank":0,"incarnation":284,"state":"up:active","state_seq":38,"addr":"172.21.15.42:6835/3228549071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3228549071},{"type":"v1","addr":"172.21.15.42:6835","nonce":3228549071}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:19:07.382 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:19:09.383 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:19:09.488 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:09.478+0000 7f34ecd93700 1 Processor -- start 2019-12-05T06:19:09.489 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:09.478+0000 7f34ecd93700 1 -- start start 2019-12-05T06:19:09.489 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:09.482+0000 7f34ecd93700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f34e80a7740 0x7f34e808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:19:09.489 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:09.482+0000 7f34ecd93700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f34e807e720 0x7f34e807eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:19:09.489 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:09.482+0000 7f34ecd93700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34e80a0710 con 0x7f34e809f2e0 2019-12-05T06:19:09.489 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:09.482+0000 7f34ecd93700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34e80a0b00 con 0x7f34e80a7740 2019-12-05T06:19:09.490 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:09.482+0000 7f34ecd93700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34e808e5b0 con 0x7f34e807e720 2019-12-05T06:19:09.490 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:09.482+0000 7f34e6d9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f34e80a7740 0x7f34e808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:09.490 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:09.482+0000 7f34e6d9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f34e80a7740 0x7f34e808df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:52728/0 (socket says 172.21.15.42:52728) 2019-12-05T06:19:09.490 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:09.482+0000 7f34e759e700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f34e809f2e0 0x7f34e809f710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:52770/0 (socket says 172.21.15.42:52770) 2019-12-05T06:19:09.490 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:09.482+0000 7f34e6d9d700 1 -- 172.21.15.42:0/596371259 learned_addr learned my addr 172.21.15.42:0/596371259 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:19:09.490 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:09.482+0000 7f34e659c700 1 --2- 172.21.15.42:0/596371259 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f34e807e720 0x7f34e807eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:09.491 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:09.482+0000 7f34e6d9d700 1 -- 172.21.15.42:0/596371259 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f34e807e720 msgr2=0x7f34e807eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:19:09.491 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:09.482+0000 7f34e6d9d700 1 --2- 172.21.15.42:0/596371259 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f34e807e720 0x7f34e807eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:19:09.491 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:09.482+0000 7f34e6d9d700 1 -- 172.21.15.42:0/596371259 >> v1:172.21.15.42:6789/0 conn(0x7f34e809f2e0 legacy=0x7f34e809f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:09.491 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:09.482+0000 7f34e5d9b700 1 -- 172.21.15.42:0/596371259 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1314428129 0 0) 0x7f34e80a0710 con 0x7f34e809f2e0 2019-12-05T06:19:09.491 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:09.482+0000 7f34e6d9d700 1 -- 172.21.15.42:0/596371259 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34e807f0a0 con 0x7f34e80a7740 2019-12-05T06:19:09.491 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:09.482+0000 7f34e6d9d700 1 --2- 172.21.15.42:0/596371259 >> v2:172.21.15.42:3300/0 conn(0x7f34e80a7740 0x7f34e808df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f34d8008b10 tx=0x7f34d80100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:09.492 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:09.482+0000 7f34e6d9d700 1 --2- 172.21.15.42:0/596371259 >> v2:172.21.15.42:3300/0 conn(0x7f34e80a7740 0x7f34e808df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f34d8008b10 tx=0x7f34d80100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:09.492 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:09.482+0000 7f34e6d9d700 1 --2- 172.21.15.42:0/596371259 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f34e80a7740 0x7f34e808df80 secure :-1 s=READY pgs=2414 cs=0 l=1 rx=0x7f34d8008b10 tx=0x7f34d80100b0).ready entity=mon.1 client_cookie=9fdec858b6ede442 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:19:09.492 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:09.482+0000 7f34e6d9d700 1 --2- 172.21.15.42:0/596371259 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f34e80a7740 0x7f34e808df80 secure :-1 s=THROTTLE_DONE pgs=2414 cs=0 l=1 rx=0x7f34d8008b10 tx=0x7f34d80100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:09.492 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:09.482+0000 7f34e5d9b700 1 -- 172.21.15.42:0/596371259 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f34d800a160 con 0x7f34e80a7740 2019-12-05T06:19:09.493 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:09.482+0000 7f34e6d9d700 1 --2- 172.21.15.42:0/596371259 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f34e80a7740 0x7f34e808df80 secure :-1 s=THROTTLE_DONE pgs=2414 cs=0 l=1 rx=0x7f34d8008b10 tx=0x7f34d80100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:09.493 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:09.482+0000 7f34e6d9d700 1 --2- 172.21.15.42:0/596371259 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f34e80a7740 0x7f34e808df80 secure :-1 s=THROTTLE_DONE pgs=2414 cs=0 l=1 rx=0x7f34d8008b10 tx=0x7f34d80100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:09.493 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:09.482+0000 7f34e5d9b700 1 -- 172.21.15.42:0/596371259 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34d8011a30 con 0x7f34e80a7740 2019-12-05T06:19:09.493 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:09.482+0000 7f34e5d9b700 1 -- 172.21.15.42:0/596371259 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f34d800a550 con 0x7f34e80a7740 2019-12-05T06:19:09.493 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:09.482+0000 7f34ecd93700 1 -- 172.21.15.42:0/596371259 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f34e80a7740 msgr2=0x7f34e808df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:09.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:09.482+0000 7f34ecd93700 1 --2- 172.21.15.42:0/596371259 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f34e80a7740 0x7f34e808df80 secure :-1 s=READY pgs=2414 cs=0 l=1 rx=0x7f34d8008b10 tx=0x7f34d80100b0).stop 2019-12-05T06:19:09.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:09.482+0000 7f34ecd93700 1 -- 172.21.15.42:0/596371259 shutdown_connections 2019-12-05T06:19:09.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:09.482+0000 7f34ecd93700 1 --2- 172.21.15.42:0/596371259 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f34e80a7740 0x7f34e808df80 unknown :-1 s=CLOSED pgs=2414 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:09.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:09.482+0000 7f34ecd93700 1 --2- 172.21.15.42:0/596371259 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f34e807e720 0x7f34e807eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:19:09.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:09.486+0000 7f34ecd93700 1 -- 172.21.15.42:0/596371259 shutdown_connections 2019-12-05T06:19:09.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:09.486+0000 7f34ecd93700 1 -- 172.21.15.42:0/596371259 wait complete. 2019-12-05T06:19:09.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:09.486+0000 7f34ecd93700 1 -- 172.21.15.42:0/596371259 >> 172.21.15.42:0/596371259 conn(0x7f34e80aabe0 msgr2=0x7f34e80a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:19:09.499 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:09.486+0000 7f34ecd93700 1 Processor -- start 2019-12-05T06:19:09.499 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:09.486+0000 7f34ecd93700 1 -- start start 2019-12-05T06:19:09.500 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:09.490+0000 7f34ecd93700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f34e81215e0 0x7f34e811af70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:19:09.500 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:09.490+0000 7f34ecd93700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f34e8123f10 0x7f34e8122930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:19:09.500 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:09.490+0000 7f34ecd93700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34e8124340 con 0x7f34e8122e80 2019-12-05T06:19:09.500 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:09.490+0000 7f34ecd93700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34e81246b0 con 0x7f34e81215e0 2019-12-05T06:19:09.500 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:09.490+0000 7f34ecd93700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34e811a790 con 0x7f34e8123f10 2019-12-05T06:19:09.500 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:09.490+0000 7f34e759e700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f34e8122e80 0x7f34e808eb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:52772/0 (socket says 172.21.15.42:52772) 2019-12-05T06:19:09.501 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:09.490+0000 7f34e759e700 1 -- 172.21.15.42:0/3754873853 learned_addr learned my addr 172.21.15.42:0/3754873853 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:19:09.501 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:09.490+0000 7f34e659c700 1 --2- 172.21.15.42:0/3754873853 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f34e8123f10 0x7f34e8122930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:09.501 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:09.490+0000 7f34e6d9d700 1 --2- 172.21.15.42:0/3754873853 >> v2:172.21.15.40:3300/0 conn(0x7f34e81215e0 0x7f34e811af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:09.501 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:09.490+0000 7f34e759e700 1 -- 172.21.15.42:0/3754873853 >> v1:172.21.15.42:6789/0 conn(0x7f34e8122e80 legacy=0x7f34e808eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2019-12-05T06:19:09.501 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:09.490+0000 7f34e759e700 1 -- 172.21.15.42:0/3754873853 >> v1:172.21.15.42:6789/0 conn(0x7f34e8122e80 legacy=0x7f34e808eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2019-12-05T06:19:09.501 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:09.490+0000 7f34e759e700 1 --1- 172.21.15.42:0/3754873853 >> v1:172.21.15.42:6789/0 conn(0x7f34e8122e80 0x7f34e808eb70 :-1 s=OPENED pgs=2415 cs=1 l=1).handle_message read tag failed 2019-12-05T06:19:09.502 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:09.490+0000 7f34e759e700 1 --1- 172.21.15.42:0/3754873853 >> v1:172.21.15.42:6789/0 conn(0x7f34e8122e80 0x7f34e808eb70 :-1 s=OPENED pgs=2415 cs=1 l=1).fault on lossy channel, failing 2019-12-05T06:19:09.502 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:09.490+0000 7f34e6d9d700 1 -- 172.21.15.42:0/3754873853 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f34e8123f10 msgr2=0x7f34e8122930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:09.502 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:09.490+0000 7f34e6d9d700 1 --2- 172.21.15.42:0/3754873853 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f34e8123f10 0x7f34e8122930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:09.502 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:09.490+0000 7f34e6d9d700 1 -- 172.21.15.42:0/3754873853 >> v1:172.21.15.42:6789/0 conn(0x7f34e8122e80 legacy=0x7f34e808eb70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2019-12-05T06:19:09.502 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:09.490+0000 7f34e6d9d700 1 -- 172.21.15.42:0/3754873853 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34e808f370 con 0x7f34e81215e0 2019-12-05T06:19:09.502 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:09.490+0000 7f34e659c700 1 --2- 172.21.15.42:0/3754873853 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f34e8123f10 0x7f34e8122930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:19:09.503 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:09.490+0000 7f34e6d9d700 1 --2- 172.21.15.42:0/3754873853 >> v2:172.21.15.40:3300/0 conn(0x7f34e81215e0 0x7f34e811af70 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f34e8124590 tx=0x7f34d8011ce0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:09.503 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:09.490+0000 7f34e6d9d700 1 --2- 172.21.15.42:0/3754873853 >> v2:172.21.15.40:3300/0 conn(0x7f34e81215e0 0x7f34e811af70 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f34e8124590 tx=0x7f34d8011ce0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:09.503 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:09.490+0000 7f34e6d9d700 1 --2- 172.21.15.42:0/3754873853 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f34e81215e0 0x7f34e811af70 secure :-1 s=READY pgs=1837 cs=0 l=1 rx=0x7f34e8124590 tx=0x7f34d8011ce0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:19:09.503 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:09.490+0000 7f34e6d9d700 1 --2- 172.21.15.42:0/3754873853 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f34e81215e0 0x7f34e811af70 secure :-1 s=THROTTLE_DONE pgs=1837 cs=0 l=1 rx=0x7f34e8124590 tx=0x7f34d8011ce0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:09.504 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:09.490+0000 7f34df7fe700 1 -- 172.21.15.42:0/3754873853 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f34d800ad70 con 0x7f34e81215e0 2019-12-05T06:19:09.504 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:09.490+0000 7f34e6d9d700 1 --2- 172.21.15.42:0/3754873853 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f34e81215e0 0x7f34e811af70 secure :-1 s=THROTTLE_DONE pgs=1837 cs=0 l=1 rx=0x7f34e8124590 tx=0x7f34d8011ce0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:09.504 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:09.490+0000 7f34e6d9d700 1 --2- 172.21.15.42:0/3754873853 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f34e81215e0 0x7f34e811af70 secure :-1 s=THROTTLE_DONE pgs=1837 cs=0 l=1 rx=0x7f34e8124590 tx=0x7f34d8011ce0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:09.504 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:09.490+0000 7f34df7fe700 1 -- 172.21.15.42:0/3754873853 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34d8011e10 con 0x7f34e81215e0 2019-12-05T06:19:09.505 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:09.490+0000 7f34ecd93700 1 -- 172.21.15.42:0/3754873853 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34e808f640 con 0x7f34e81215e0 2019-12-05T06:19:09.505 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:09.490+0000 7f34df7fe700 1 -- 172.21.15.42:0/3754873853 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f34d8005ca0 con 0x7f34e81215e0 2019-12-05T06:19:09.505 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:09.490+0000 7f34ecd93700 1 -- 172.21.15.42:0/3754873853 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34e807a330 con 0x7f34e81215e0 2019-12-05T06:19:09.505 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:09.490+0000 7f34e6d9d700 1 --2- 172.21.15.42:0/3754873853 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f34e81215e0 0x7f34e811af70 secure :-1 s=THROTTLE_DONE pgs=1837 cs=0 l=1 rx=0x7f34e8124590 tx=0x7f34d8011ce0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:09.505 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:09.494+0000 7f34df7fe700 1 -- 172.21.15.42:0/3754873853 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f34d8007150 con 0x7f34e81215e0 2019-12-05T06:19:09.505 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:09.494+0000 7f34e6d9d700 1 --2- 172.21.15.42:0/3754873853 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f34e81215e0 0x7f34e811af70 secure :-1 s=THROTTLE_DONE pgs=1837 cs=0 l=1 rx=0x7f34e8124590 tx=0x7f34d8011ce0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:09.506 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:09.494+0000 7f34df7fe700 1 --2- 172.21.15.42:0/3754873853 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f34cc03ade0 0x7f34cc03d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:19:09.506 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:09.494+0000 7f34df7fe700 1 -- 172.21.15.42:0/3754873853 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 10417+0+0 (secure 0 0 0) 0x7f34d804bbd0 con 0x7f34e81215e0 2019-12-05T06:19:09.506 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:09.494+0000 7f34e659c700 1 --2- 172.21.15.42:0/3754873853 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f34cc03ade0 0x7f34cc03d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:09.506 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:09.494+0000 7f34e659c700 1 --2- 172.21.15.42:0/3754873853 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f34cc03ade0 0x7f34cc03d2f0 crc :-1 s=READY pgs=1016 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:19:09.507 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:09.494+0000 7f34ecd93700 1 -- 172.21.15.42:0/3754873853 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34e805d110 con 0x7f34e81215e0 2019-12-05T06:19:09.508 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:09.506+0000 7f34e6d9d700 1 --2- 172.21.15.42:0/3754873853 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f34e81215e0 0x7f34e811af70 secure :-1 s=THROTTLE_DONE pgs=1837 cs=0 l=1 rx=0x7f34e8124590 tx=0x7f34d8011ce0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:09.508 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:09.506+0000 7f34df7fe700 1 -- 172.21.15.42:0/3754873853 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f34d8032c50 con 0x7f34e81215e0 2019-12-05T06:19:09.678 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:09.674+0000 7f34ecd93700 1 -- 172.21.15.42:0/3754873853 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f34e805b8c0 con 0x7f34e81215e0 2019-12-05T06:19:09.679 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:09.674+0000 7f34e6d9d700 1 --2- 172.21.15.42:0/3754873853 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f34e81215e0 0x7f34e811af70 secure :-1 s=THROTTLE_DONE pgs=1837 cs=0 l=1 rx=0x7f34e8124590 tx=0x7f34d8011ce0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:09.679 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:09.674+0000 7f34df7fe700 1 -- 172.21.15.42:0/3754873853 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 288 v288) v1 ==== 95+0+2114 (secure 0 0 0) 0x7f34d8032eb0 con 0x7f34e81215e0 2019-12-05T06:19:09.681 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:19:09.682 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":288,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":10476,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/1473620421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1473620421},{"type":"v1","addr":"172.21.15.40:6833","nonce":1473620421}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":288}],"filesystems":[{"mdsmap":{"epoch":287,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:17:04.753930+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":100,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7295},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7295":{"gid":7295,"name":"a","rank":0,"incarnation":284,"state":"up:active","state_seq":38,"addr":"172.21.15.42:6835/3228549071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3228549071},{"type":"v1","addr":"172.21.15.42:6835","nonce":3228549071}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:19:09.682 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 288 2019-12-05T06:19:09.684 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:09.678+0000 7f34ecd93700 1 -- 172.21.15.42:0/3754873853 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f34cc03ade0 msgr2=0x7f34cc03d2f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:09.684 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:09.678+0000 7f34ecd93700 1 --2- 172.21.15.42:0/3754873853 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f34cc03ade0 0x7f34cc03d2f0 crc :-1 s=READY pgs=1016 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:09.685 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:09.678+0000 7f34ecd93700 1 -- 172.21.15.42:0/3754873853 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f34e81215e0 msgr2=0x7f34e811af70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:09.685 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:09.678+0000 7f34ecd93700 1 --2- 172.21.15.42:0/3754873853 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f34e81215e0 0x7f34e811af70 secure :-1 s=READY pgs=1837 cs=0 l=1 rx=0x7f34e8124590 tx=0x7f34d8011ce0).stop 2019-12-05T06:19:09.685 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:09.678+0000 7f34ecd93700 1 -- 172.21.15.42:0/3754873853 shutdown_connections 2019-12-05T06:19:09.685 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:09.678+0000 7f34ecd93700 1 --2- 172.21.15.42:0/3754873853 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f34cc03ade0 0x7f34cc03d2f0 unknown :-1 s=CLOSED pgs=1016 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:09.685 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:09.678+0000 7f34ecd93700 1 --2- 172.21.15.42:0/3754873853 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f34e81215e0 0x7f34e811af70 unknown :-1 s=CLOSED pgs=1837 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:09.686 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:09.678+0000 7f34ecd93700 1 --2- 172.21.15.42:0/3754873853 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f34e8123f10 0x7f34e8122930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:09.686 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:09.678+0000 7f34ecd93700 1 -- 172.21.15.42:0/3754873853 shutdown_connections 2019-12-05T06:19:09.686 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:09.678+0000 7f34ecd93700 1 -- 172.21.15.42:0/3754873853 wait complete. 2019-12-05T06:19:09.686 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:09.678+0000 7f34ecd93700 1 -- 172.21.15.42:0/3754873853 >> 172.21.15.42:0/3754873853 conn(0x7f34e80a7740 msgr2=0x7f34e809f6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:19:09.697 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:19:11.698 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:19:11.859 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:11.850+0000 7f018c9ab700 1 Processor -- start 2019-12-05T06:19:11.859 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:11.850+0000 7f018c9ab700 1 -- start start 2019-12-05T06:19:11.859 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:11.850+0000 7f018c9ab700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f018809fae0 0x7f018814fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:19:11.859 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:11.850+0000 7f018c9ab700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01880a0710 con 0x7f018808df80 2019-12-05T06:19:11.860 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:11.850+0000 7f018c9ab700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f018807ef20 con 0x7f01880a7740 2019-12-05T06:19:11.860 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:11.850+0000 7f018c9ab700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01880a0b00 con 0x7f018809fae0 2019-12-05T06:19:11.860 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:11.850+0000 7f0186d9d700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f01880a7740 0x7f018807e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:52780/0 (socket says 172.21.15.42:52780) 2019-12-05T06:19:11.860 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:11.850+0000 7f0186d9d700 1 -- 172.21.15.42:0/2626742677 learned_addr learned my addr 172.21.15.42:0/2626742677 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:19:11.860 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:11.850+0000 7f018759e700 1 --2- 172.21.15.42:0/2626742677 >> v2:172.21.15.42:3300/0 conn(0x7f018809fae0 0x7f018814fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:11.860 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:11.850+0000 7f0185d9b700 1 -- 172.21.15.42:0/2626742677 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1397017309 0 0) 0x7f018807ef20 con 0x7f01880a7740 2019-12-05T06:19:11.860 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:11.850+0000 7f0185d9b700 1 -- 172.21.15.42:0/2626742677 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01700030b0 con 0x7f01880a7740 2019-12-05T06:19:11.861 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:11.850+0000 7f0185d9b700 1 -- 172.21.15.42:0/2626742677 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3229192631 0 0) 0x7f01880a0710 con 0x7f018808df80 2019-12-05T06:19:11.861 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:11.850+0000 7f0185d9b700 1 -- 172.21.15.42:0/2626742677 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f018807ef20 con 0x7f018808df80 2019-12-05T06:19:11.861 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:11.850+0000 7f018759e700 1 -- 172.21.15.42:0/2626742677 >> v1:172.21.15.42:6789/0 conn(0x7f01880a7740 legacy=0x7f018807e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:11.861 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:11.850+0000 7f018759e700 1 -- 172.21.15.42:0/2626742677 >> v1:172.21.15.40:6789/0 conn(0x7f018808df80 legacy=0x7f018809f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:11.861 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:11.850+0000 7f018759e700 1 -- 172.21.15.42:0/2626742677 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01880a9ba0 con 0x7f018809fae0 2019-12-05T06:19:11.861 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:11.850+0000 7f018759e700 1 --2- 172.21.15.42:0/2626742677 >> v2:172.21.15.42:3300/0 conn(0x7f018809fae0 0x7f018814fff0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f0180008b10 tx=0x7f018000ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:11.861 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:11.850+0000 7f018759e700 1 --2- 172.21.15.42:0/2626742677 >> v2:172.21.15.42:3300/0 conn(0x7f018809fae0 0x7f018814fff0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f0180008b10 tx=0x7f018000ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:11.862 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:11.850+0000 7f018759e700 1 --2- 172.21.15.42:0/2626742677 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f018809fae0 0x7f018814fff0 secure :-1 s=READY pgs=2417 cs=0 l=1 rx=0x7f0180008b10 tx=0x7f018000ff70).ready entity=mon.1 client_cookie=9bf6537ea1f08b3c server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:19:11.862 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:11.850+0000 7f018759e700 1 --2- 172.21.15.42:0/2626742677 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f018809fae0 0x7f018814fff0 secure :-1 s=THROTTLE_DONE pgs=2417 cs=0 l=1 rx=0x7f0180008b10 tx=0x7f018000ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:11.862 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:11.850+0000 7f0185d9b700 1 -- 172.21.15.42:0/2626742677 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0180010970 con 0x7f018809fae0 2019-12-05T06:19:11.862 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:11.850+0000 7f018759e700 1 --2- 172.21.15.42:0/2626742677 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f018809fae0 0x7f018814fff0 secure :-1 s=THROTTLE_DONE pgs=2417 cs=0 l=1 rx=0x7f0180008b10 tx=0x7f018000ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:11.862 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:11.850+0000 7f018759e700 1 --2- 172.21.15.42:0/2626742677 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f018809fae0 0x7f018814fff0 secure :-1 s=THROTTLE_DONE pgs=2417 cs=0 l=1 rx=0x7f0180008b10 tx=0x7f018000ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:11.862 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:11.850+0000 7f0185d9b700 1 -- 172.21.15.42:0/2626742677 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0180008b50 con 0x7f018809fae0 2019-12-05T06:19:11.862 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:11.850+0000 7f0185d9b700 1 -- 172.21.15.42:0/2626742677 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f01800110c0 con 0x7f018809fae0 2019-12-05T06:19:11.863 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:11.850+0000 7f018c9ab700 1 -- 172.21.15.42:0/2626742677 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f018809fae0 msgr2=0x7f018814fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:11.863 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:11.850+0000 7f018c9ab700 1 --2- 172.21.15.42:0/2626742677 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f018809fae0 0x7f018814fff0 secure :-1 s=READY pgs=2417 cs=0 l=1 rx=0x7f0180008b10 tx=0x7f018000ff70).stop 2019-12-05T06:19:11.863 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:11.854+0000 7f018c9ab700 1 -- 172.21.15.42:0/2626742677 shutdown_connections 2019-12-05T06:19:11.863 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:11.854+0000 7f018c9ab700 1 --2- 172.21.15.42:0/2626742677 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f018809fae0 0x7f018814fff0 unknown :-1 s=CLOSED pgs=2417 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:11.863 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:11.854+0000 7f018c9ab700 1 -- 172.21.15.42:0/2626742677 shutdown_connections 2019-12-05T06:19:11.863 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:11.854+0000 7f018c9ab700 1 -- 172.21.15.42:0/2626742677 wait complete. 2019-12-05T06:19:11.863 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:11.854+0000 7f018c9ab700 1 -- 172.21.15.42:0/2626742677 >> 172.21.15.42:0/2626742677 conn(0x7f01880aabe0 msgr2=0x7f01880a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:19:11.868 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:11.854+0000 7f018c9ab700 1 Processor -- start 2019-12-05T06:19:11.868 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:11.854+0000 7f018c9ab700 1 -- start start 2019-12-05T06:19:11.868 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:11.858+0000 7f018c9ab700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f01880a3960 0x7f0188079aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:19:11.869 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:11.858+0000 7f018c9ab700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f018807b4d0 0x7f0188144860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:19:11.869 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:11.858+0000 7f018c9ab700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01881451e0 con 0x7f0188144db0 2019-12-05T06:19:11.869 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:11.858+0000 7f018c9ab700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f018807a720 con 0x7f01880a3960 2019-12-05T06:19:11.869 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:11.858+0000 7f018c9ab700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f018807a1e0 con 0x7f018807b4d0 2019-12-05T06:19:11.869 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:11.858+0000 7f018659c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f018807b4d0 0x7f0188144860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:11.870 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:11.858+0000 7f018659c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f018807b4d0 0x7f0188144860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:52750/0 (socket says 172.21.15.42:52750) 2019-12-05T06:19:11.870 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:11.858+0000 7f018659c700 1 -- 172.21.15.42:0/3625691006 learned_addr learned my addr 172.21.15.42:0/3625691006 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:19:11.870 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:11.858+0000 7f0186d9d700 1 --2- 172.21.15.42:0/3625691006 >> v2:172.21.15.40:3300/0 conn(0x7f01880a3960 0x7f0188079aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:11.870 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:11.858+0000 7f018659c700 1 -- 172.21.15.42:0/3625691006 >> v2:172.21.15.40:3300/0 conn(0x7f01880a3960 msgr2=0x7f0188079aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:11.870 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:11.858+0000 7f018659c700 1 --2- 172.21.15.42:0/3625691006 >> v2:172.21.15.40:3300/0 conn(0x7f01880a3960 0x7f0188079aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:11.870 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:11.858+0000 7f018659c700 1 -- 172.21.15.42:0/3625691006 >> v1:172.21.15.42:6789/0 conn(0x7f0188144db0 legacy=0x7f01881223c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:11.870 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:11.858+0000 7f018659c700 1 -- 172.21.15.42:0/3625691006 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0188122bc0 con 0x7f018807b4d0 2019-12-05T06:19:11.871 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:11.858+0000 7f01777fe700 1 -- 172.21.15.42:0/3625691006 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1736459158 0 0) 0x7f01881451e0 con 0x7f0188144db0 2019-12-05T06:19:11.871 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:11.858+0000 7f018659c700 1 --2- 172.21.15.42:0/3625691006 >> v2:172.21.15.42:3300/0 conn(0x7f018807b4d0 0x7f0188144860 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f017c000e90 tx=0x7f017c012bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:11.872 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:11.858+0000 7f018659c700 1 --2- 172.21.15.42:0/3625691006 >> v2:172.21.15.42:3300/0 conn(0x7f018807b4d0 0x7f0188144860 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f017c000e90 tx=0x7f017c012bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:11.872 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:11.858+0000 7f018659c700 1 --2- 172.21.15.42:0/3625691006 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f018807b4d0 0x7f0188144860 secure :-1 s=READY pgs=2419 cs=0 l=1 rx=0x7f017c000e90 tx=0x7f017c012bb0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:19:11.872 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:11.858+0000 7f018659c700 1 --2- 172.21.15.42:0/3625691006 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f018807b4d0 0x7f0188144860 secure :-1 s=THROTTLE_DONE pgs=2419 cs=0 l=1 rx=0x7f017c000e90 tx=0x7f017c012bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:11.872 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:11.858+0000 7f018659c700 1 --2- 172.21.15.42:0/3625691006 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f018807b4d0 0x7f0188144860 secure :-1 s=THROTTLE_DONE pgs=2419 cs=0 l=1 rx=0x7f017c000e90 tx=0x7f017c012bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:11.872 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:11.858+0000 7f01777fe700 1 -- 172.21.15.42:0/3625691006 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f017c00caa0 con 0x7f018807b4d0 2019-12-05T06:19:11.872 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:11.858+0000 7f018659c700 1 --2- 172.21.15.42:0/3625691006 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f018807b4d0 0x7f0188144860 secure :-1 s=THROTTLE_DONE pgs=2419 cs=0 l=1 rx=0x7f017c000e90 tx=0x7f017c012bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:11.872 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:11.858+0000 7f01777fe700 1 -- 172.21.15.42:0/3625691006 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f017c00c6c0 con 0x7f018807b4d0 2019-12-05T06:19:11.874 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:11.858+0000 7f018c9ab700 1 -- 172.21.15.42:0/3625691006 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0188122e90 con 0x7f018807b4d0 2019-12-05T06:19:11.874 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:11.858+0000 7f01777fe700 1 -- 172.21.15.42:0/3625691006 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f017c00d6a0 con 0x7f018807b4d0 2019-12-05T06:19:11.874 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:11.858+0000 7f018c9ab700 1 -- 172.21.15.42:0/3625691006 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f018808eb70 con 0x7f018807b4d0 2019-12-05T06:19:11.875 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:11.862+0000 7f018659c700 1 --2- 172.21.15.42:0/3625691006 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f018807b4d0 0x7f0188144860 secure :-1 s=THROTTLE_DONE pgs=2419 cs=0 l=1 rx=0x7f017c000e90 tx=0x7f017c012bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:11.875 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:11.862+0000 7f01777fe700 1 -- 172.21.15.42:0/3625691006 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f017c00d9b0 con 0x7f018807b4d0 2019-12-05T06:19:11.875 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:11.862+0000 7f018659c700 1 --2- 172.21.15.42:0/3625691006 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f018807b4d0 0x7f0188144860 secure :-1 s=THROTTLE_DONE pgs=2419 cs=0 l=1 rx=0x7f017c000e90 tx=0x7f017c012bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:11.875 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:11.862+0000 7f01777fe700 1 --2- 172.21.15.42:0/3625691006 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f016c03ac80 0x7f016c03d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:19:11.875 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:11.862+0000 7f01777fe700 1 -- 172.21.15.42:0/3625691006 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 10503+0+0 (secure 0 0 0) 0x7f017c04d670 con 0x7f018807b4d0 2019-12-05T06:19:11.875 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:11.862+0000 7f0186d9d700 1 --2- 172.21.15.42:0/3625691006 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f016c03ac80 0x7f016c03d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:11.875 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:11.862+0000 7f0186d9d700 1 --2- 172.21.15.42:0/3625691006 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f016c03ac80 0x7f016c03d190 crc :-1 s=READY pgs=1017 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:19:11.876 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:11.862+0000 7f018c9ab700 1 -- 172.21.15.42:0/3625691006 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f018805d110 con 0x7f018807b4d0 2019-12-05T06:19:11.876 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:11.874+0000 7f018659c700 1 --2- 172.21.15.42:0/3625691006 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f018807b4d0 0x7f0188144860 secure :-1 s=THROTTLE_DONE pgs=2419 cs=0 l=1 rx=0x7f017c000e90 tx=0x7f017c012bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:11.876 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:11.874+0000 7f01777fe700 1 -- 172.21.15.42:0/3625691006 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f017c015060 con 0x7f018807b4d0 2019-12-05T06:19:12.043 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:12.038+0000 7f018c9ab700 1 -- 172.21.15.42:0/3625691006 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f018805b8c0 con 0x7f018807b4d0 2019-12-05T06:19:12.044 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:12.042+0000 7f018659c700 1 --2- 172.21.15.42:0/3625691006 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f018807b4d0 0x7f0188144860 secure :-1 s=THROTTLE_DONE pgs=2419 cs=0 l=1 rx=0x7f017c000e90 tx=0x7f017c012bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:12.044 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:12.042+0000 7f01777fe700 1 -- 172.21.15.42:0/3625691006 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 289 v289) v1 ==== 95+0+1778 (secure 0 0 0) 0x7f017c0346f0 con 0x7f018807b4d0 2019-12-05T06:19:12.046 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 289 2019-12-05T06:19:12.046 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:19:12.047 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":289,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":289,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:19:09.751427+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":101,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10476},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10476":{"gid":10476,"name":"b","rank":0,"incarnation":289,"state":"up:replay","state_seq":2,"addr":"172.21.15.40:6833/1473620421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1473620421},{"type":"v1","addr":"172.21.15.40:6833","nonce":1473620421}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:19:12.050 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:12.046+0000 7f018c9ab700 1 -- 172.21.15.42:0/3625691006 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f016c03ac80 msgr2=0x7f016c03d190 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:12.050 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:12.046+0000 7f018c9ab700 1 --2- 172.21.15.42:0/3625691006 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f016c03ac80 0x7f016c03d190 crc :-1 s=READY pgs=1017 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:12.050 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:12.046+0000 7f018c9ab700 1 -- 172.21.15.42:0/3625691006 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f018807b4d0 msgr2=0x7f0188144860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:12.051 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:12.046+0000 7f018c9ab700 1 --2- 172.21.15.42:0/3625691006 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f018807b4d0 0x7f0188144860 secure :-1 s=READY pgs=2419 cs=0 l=1 rx=0x7f017c000e90 tx=0x7f017c012bb0).stop 2019-12-05T06:19:12.051 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:12.046+0000 7f018c9ab700 1 -- 172.21.15.42:0/3625691006 shutdown_connections 2019-12-05T06:19:12.051 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:12.046+0000 7f018c9ab700 1 --2- 172.21.15.42:0/3625691006 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f016c03ac80 0x7f016c03d190 unknown :-1 s=CLOSED pgs=1017 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:12.051 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:12.046+0000 7f018c9ab700 1 --2- 172.21.15.42:0/3625691006 >> v2:172.21.15.40:3300/0 conn(0x7f01880a3960 0x7f0188079aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:12.051 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:12.046+0000 7f018c9ab700 1 --2- 172.21.15.42:0/3625691006 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f018807b4d0 0x7f0188144860 unknown :-1 s=CLOSED pgs=2419 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:12.051 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:12.046+0000 7f018c9ab700 1 -- 172.21.15.42:0/3625691006 shutdown_connections 2019-12-05T06:19:12.051 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:12.046+0000 7f018c9ab700 1 -- 172.21.15.42:0/3625691006 wait complete. 2019-12-05T06:19:12.052 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:12.046+0000 7f018c9ab700 1 -- 172.21.15.42:0/3625691006 >> 172.21.15.42:0/3625691006 conn(0x7f01880a82a0 msgr2=0x7f01880a6a60 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:19:12.064 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a down, removed from mdsmap 2019-12-05T06:19:12.064 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2019-12-05T06:19:12.064 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:19:12.171 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:12.154+0000 7f0e2c176700 1 Processor -- start 2019-12-05T06:19:12.172 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:12.154+0000 7f0e2c176700 1 -- start start 2019-12-05T06:19:12.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:12.154+0000 7f0e2c176700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0e240a7740 0x7f0e2408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:19:12.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:12.154+0000 7f0e2c176700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f0e2407e720 0x7f0e2407eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:19:12.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:12.154+0000 7f0e2c176700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e240a0710 con 0x7f0e2409f2e0 2019-12-05T06:19:12.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:12.154+0000 7f0e2c176700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e240a0b00 con 0x7f0e2407e720 2019-12-05T06:19:12.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:12.154+0000 7f0e2c176700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e2408e5b0 con 0x7f0e240a7740 2019-12-05T06:19:12.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:12.154+0000 7f0e2a713700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0e240a7740 0x7f0e2408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:12.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:12.154+0000 7f0e29f12700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f0e2407e720 0x7f0e2407eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:12.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:12.154+0000 7f0e2af14700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f0e2409f2e0 0x7f0e2409f710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:34006/0 (socket says 172.21.15.42:34006) 2019-12-05T06:19:12.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:12.154+0000 7f0e2af14700 1 -- 172.21.15.42:0/3489438776 learned_addr learned my addr 172.21.15.42:0/3489438776 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:19:12.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:12.154+0000 7f0e2a713700 1 -- 172.21.15.42:0/3489438776 >> v2:172.21.15.40:3300/0 conn(0x7f0e2407e720 msgr2=0x7f0e2407eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:19:12.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:12.154+0000 7f0e2a713700 1 --2- 172.21.15.42:0/3489438776 >> v2:172.21.15.40:3300/0 conn(0x7f0e2407e720 0x7f0e2407eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:19:12.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:12.154+0000 7f0e2a713700 1 -- 172.21.15.42:0/3489438776 >> v1:172.21.15.40:6789/0 conn(0x7f0e2409f2e0 legacy=0x7f0e2409f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:12.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:12.154+0000 7f0e2a713700 1 -- 172.21.15.42:0/3489438776 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e2407f0a0 con 0x7f0e240a7740 2019-12-05T06:19:12.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:12.154+0000 7f0e29f12700 1 --2- 172.21.15.42:0/3489438776 >> v2:172.21.15.40:3300/0 conn(0x7f0e2407e720 0x7f0e2407eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:19:12.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:12.154+0000 7f0e2a713700 1 --2- 172.21.15.42:0/3489438776 >> v2:172.21.15.42:3300/0 conn(0x7f0e240a7740 0x7f0e2408df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f0e20002f30 tx=0x7f0e2000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:12.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:12.154+0000 7f0e2a713700 1 --2- 172.21.15.42:0/3489438776 >> v2:172.21.15.42:3300/0 conn(0x7f0e240a7740 0x7f0e2408df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f0e20002f30 tx=0x7f0e2000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:12.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:12.154+0000 7f0e2a713700 1 --2- 172.21.15.42:0/3489438776 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0e240a7740 0x7f0e2408df80 secure :-1 s=READY pgs=2420 cs=0 l=1 rx=0x7f0e20002f30 tx=0x7f0e2000fc70).ready entity=mon.1 client_cookie=cb6e460a3bbabe0c server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:19:12.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:12.158+0000 7f0e2a713700 1 --2- 172.21.15.42:0/3489438776 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0e240a7740 0x7f0e2408df80 secure :-1 s=THROTTLE_DONE pgs=2420 cs=0 l=1 rx=0x7f0e20002f30 tx=0x7f0e2000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:12.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:12.158+0000 7f0e29711700 1 -- 172.21.15.42:0/3489438776 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0e200112b0 con 0x7f0e240a7740 2019-12-05T06:19:12.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:12.158+0000 7f0e2a713700 1 --2- 172.21.15.42:0/3489438776 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0e240a7740 0x7f0e2408df80 secure :-1 s=THROTTLE_DONE pgs=2420 cs=0 l=1 rx=0x7f0e20002f30 tx=0x7f0e2000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:12.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:12.158+0000 7f0e29711700 1 -- 172.21.15.42:0/3489438776 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e20008ad0 con 0x7f0e240a7740 2019-12-05T06:19:12.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:12.158+0000 7f0e2a713700 1 --2- 172.21.15.42:0/3489438776 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0e240a7740 0x7f0e2408df80 secure :-1 s=THROTTLE_DONE pgs=2420 cs=0 l=1 rx=0x7f0e20002f30 tx=0x7f0e2000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:12.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:12.158+0000 7f0e29711700 1 -- 172.21.15.42:0/3489438776 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0e20011890 con 0x7f0e240a7740 2019-12-05T06:19:12.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:12.158+0000 7f0e2c176700 1 -- 172.21.15.42:0/3489438776 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0e240a7740 msgr2=0x7f0e2408df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:12.206 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:12.158+0000 7f0e2c176700 1 --2- 172.21.15.42:0/3489438776 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0e240a7740 0x7f0e2408df80 secure :-1 s=READY pgs=2420 cs=0 l=1 rx=0x7f0e20002f30 tx=0x7f0e2000fc70).stop 2019-12-05T06:19:12.207 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:12.158+0000 7f0e2c176700 1 -- 172.21.15.42:0/3489438776 shutdown_connections 2019-12-05T06:19:12.207 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:12.158+0000 7f0e2c176700 1 --2- 172.21.15.42:0/3489438776 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0e240a7740 0x7f0e2408df80 unknown :-1 s=CLOSED pgs=2420 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:12.207 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:12.158+0000 7f0e2c176700 1 --2- 172.21.15.42:0/3489438776 >> v2:172.21.15.40:3300/0 conn(0x7f0e2407e720 0x7f0e2407eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:19:12.207 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:12.158+0000 7f0e2c176700 1 -- 172.21.15.42:0/3489438776 shutdown_connections 2019-12-05T06:19:12.207 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:12.158+0000 7f0e2c176700 1 -- 172.21.15.42:0/3489438776 wait complete. 2019-12-05T06:19:12.208 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:12.158+0000 7f0e2c176700 1 -- 172.21.15.42:0/3489438776 >> 172.21.15.42:0/3489438776 conn(0x7f0e240aabe0 msgr2=0x7f0e240a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:19:12.208 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:12.158+0000 7f0e2c176700 1 Processor -- start 2019-12-05T06:19:12.208 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:12.158+0000 7f0e2c176700 1 -- start start 2019-12-05T06:19:12.208 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:12.158+0000 7f0e2c176700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0e24116500 0x7f0e2411a690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:19:12.208 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:12.158+0000 7f0e2c176700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f0e2407c680 0x7f0e2413ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:19:12.208 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:12.158+0000 7f0e2c176700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e2411bbf0 con 0x7f0e2411b7c0 2019-12-05T06:19:12.209 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:12.158+0000 7f0e2c176700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e2407b660 con 0x7f0e2407c680 2019-12-05T06:19:12.209 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:12.158+0000 7f0e2c176700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e2411bf60 con 0x7f0e24116500 2019-12-05T06:19:12.209 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:12.158+0000 7f0e29f12700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f0e2411b7c0 0x7f0e2407ae60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:34008/0 (socket says 172.21.15.42:34008) 2019-12-05T06:19:12.209 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:12.158+0000 7f0e29f12700 1 -- 172.21.15.42:0/708852417 learned_addr learned my addr 172.21.15.42:0/708852417 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:19:12.209 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:12.158+0000 7f0e2af14700 1 --2- 172.21.15.42:0/708852417 >> v2:172.21.15.40:3300/0 conn(0x7f0e2407c680 0x7f0e2413ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:12.209 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:12.158+0000 7f0e2a713700 1 --2- 172.21.15.42:0/708852417 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0e24116500 0x7f0e2411a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:12.209 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:12.158+0000 7f0e16ffd700 1 -- 172.21.15.42:0/708852417 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1166645967 0 0) 0x7f0e2411bbf0 con 0x7f0e2411b7c0 2019-12-05T06:19:12.210 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:12.162+0000 7f0e16ffd700 1 -- 172.21.15.42:0/708852417 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e0c0030b0 con 0x7f0e2411b7c0 2019-12-05T06:19:12.210 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:12.162+0000 7f0e16ffd700 1 -- 172.21.15.42:0/708852417 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 546544364 0 0) 0x7f0e0c0030b0 con 0x7f0e2411b7c0 2019-12-05T06:19:12.210 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:12.162+0000 7f0e16ffd700 1 -- 172.21.15.42:0/708852417 --> v1:172.21.15.40:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0e2411bbf0 con 0x7f0e2411b7c0 2019-12-05T06:19:12.210 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:12.162+0000 7f0e16ffd700 1 -- 172.21.15.42:0/708852417 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7f0e18002500 con 0x7f0e2411b7c0 2019-12-05T06:19:12.210 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:12.162+0000 7f0e2af14700 1 -- 172.21.15.42:0/708852417 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0e24116500 msgr2=0x7f0e2411a690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:12.210 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:12.162+0000 7f0e2af14700 1 --2- 172.21.15.42:0/708852417 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0e24116500 0x7f0e2411a690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:12.210 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:12.162+0000 7f0e2af14700 1 -- 172.21.15.42:0/708852417 >> v1:172.21.15.40:6789/0 conn(0x7f0e2411b7c0 legacy=0x7f0e2407ae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:12.211 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:12.162+0000 7f0e2af14700 1 -- 172.21.15.42:0/708852417 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e24118ec0 con 0x7f0e2407c680 2019-12-05T06:19:12.211 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:12.162+0000 7f0e2af14700 1 --2- 172.21.15.42:0/708852417 >> v2:172.21.15.40:3300/0 conn(0x7f0e2407c680 0x7f0e2413ac40 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f0e2411c120 tx=0x7f0e18007a50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:12.211 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:12.162+0000 7f0e2af14700 1 --2- 172.21.15.42:0/708852417 >> v2:172.21.15.40:3300/0 conn(0x7f0e2407c680 0x7f0e2413ac40 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f0e2411c120 tx=0x7f0e18007a50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:12.211 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:12.162+0000 7f0e2af14700 1 --2- 172.21.15.42:0/708852417 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0e2407c680 0x7f0e2413ac40 secure :-1 s=READY pgs=1841 cs=0 l=1 rx=0x7f0e2411c120 tx=0x7f0e18007a50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:19:12.211 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:12.162+0000 7f0e2af14700 1 --2- 172.21.15.42:0/708852417 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0e2407c680 0x7f0e2413ac40 secure :-1 s=THROTTLE_DONE pgs=1841 cs=0 l=1 rx=0x7f0e2411c120 tx=0x7f0e18007a50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:12.211 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:12.162+0000 7f0e16ffd700 1 -- 172.21.15.42:0/708852417 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0e1c013060 con 0x7f0e2407c680 2019-12-05T06:19:12.211 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:12.162+0000 7f0e2af14700 1 --2- 172.21.15.42:0/708852417 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0e2407c680 0x7f0e2413ac40 secure :-1 s=THROTTLE_DONE pgs=1841 cs=0 l=1 rx=0x7f0e2411c120 tx=0x7f0e18007a50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:12.212 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:12.162+0000 7f0e2af14700 1 --2- 172.21.15.42:0/708852417 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0e2407c680 0x7f0e2413ac40 secure :-1 s=THROTTLE_DONE pgs=1841 cs=0 l=1 rx=0x7f0e2411c120 tx=0x7f0e18007a50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:12.212 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:12.162+0000 7f0e16ffd700 1 -- 172.21.15.42:0/708852417 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e1c0047b0 con 0x7f0e2407c680 2019-12-05T06:19:12.212 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:12.162+0000 7f0e2c176700 1 -- 172.21.15.42:0/708852417 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e2407cab0 con 0x7f0e2407c680 2019-12-05T06:19:12.212 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:12.162+0000 7f0e16ffd700 1 -- 172.21.15.42:0/708852417 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0e1c0122c0 con 0x7f0e2407c680 2019-12-05T06:19:12.218 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:12.162+0000 7f0e2c176700 1 -- 172.21.15.42:0/708852417 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e2413b190 con 0x7f0e2407c680 2019-12-05T06:19:12.218 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:12.162+0000 7f0e2af14700 1 --2- 172.21.15.42:0/708852417 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0e2407c680 0x7f0e2413ac40 secure :-1 s=THROTTLE_DONE pgs=1841 cs=0 l=1 rx=0x7f0e2411c120 tx=0x7f0e18007a50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:12.219 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:12.162+0000 7f0e16ffd700 1 -- 172.21.15.42:0/708852417 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f0e1c012590 con 0x7f0e2407c680 2019-12-05T06:19:12.219 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:12.162+0000 7f0e2af14700 1 --2- 172.21.15.42:0/708852417 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0e2407c680 0x7f0e2413ac40 secure :-1 s=THROTTLE_DONE pgs=1841 cs=0 l=1 rx=0x7f0e2411c120 tx=0x7f0e18007a50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:12.219 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:12.166+0000 7f0e16ffd700 1 --2- 172.21.15.42:0/708852417 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0e0c03b6c0 0x7f0e0c03dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:19:12.219 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:12.166+0000 7f0e16ffd700 1 -- 172.21.15.42:0/708852417 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 10503+0+0 (secure 0 0 0) 0x7f0e1c04ace0 con 0x7f0e2407c680 2019-12-05T06:19:12.219 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:12.166+0000 7f0e2a713700 1 --2- 172.21.15.42:0/708852417 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0e0c03b6c0 0x7f0e0c03dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:12.220 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:12.166+0000 7f0e2a713700 1 --2- 172.21.15.42:0/708852417 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0e0c03b6c0 0x7f0e0c03dbd0 crc :-1 s=READY pgs=1018 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:19:12.220 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:12.170+0000 7f0e2c176700 1 -- 172.21.15.42:0/708852417 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e2405d110 con 0x7f0e2407c680 2019-12-05T06:19:12.221 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:12.178+0000 7f0e2af14700 1 --2- 172.21.15.42:0/708852417 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0e2407c680 0x7f0e2413ac40 secure :-1 s=THROTTLE_DONE pgs=1841 cs=0 l=1 rx=0x7f0e2411c120 tx=0x7f0e18007a50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:12.221 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:12.178+0000 7f0e16ffd700 1 -- 172.21.15.42:0/708852417 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f0e1c031d00 con 0x7f0e2407c680 2019-12-05T06:19:12.347 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:12.342+0000 7f0e2c176700 1 -- 172.21.15.42:0/708852417 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0e2405b8c0 con 0x7f0e2407c680 2019-12-05T06:19:12.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:12.346+0000 7f0e2af14700 1 --2- 172.21.15.42:0/708852417 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0e2407c680 0x7f0e2413ac40 secure :-1 s=THROTTLE_DONE pgs=1841 cs=0 l=1 rx=0x7f0e2411c120 tx=0x7f0e18007a50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:12.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:12.346+0000 7f0e16ffd700 1 -- 172.21.15.42:0/708852417 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 289 v289) v1 ==== 95+0+1778 (secure 0 0 0) 0x7f0e1c031f60 con 0x7f0e2407c680 2019-12-05T06:19:12.351 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 289 2019-12-05T06:19:12.351 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:19:12.351 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":289,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":289,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:19:09.751427+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":101,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10476},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10476":{"gid":10476,"name":"b","rank":0,"incarnation":289,"state":"up:replay","state_seq":2,"addr":"172.21.15.40:6833/1473620421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1473620421},{"type":"v1","addr":"172.21.15.40:6833","nonce":1473620421}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:19:12.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:12.350+0000 7f0e2c176700 1 -- 172.21.15.42:0/708852417 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0e0c03b6c0 msgr2=0x7f0e0c03dbd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:12.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:12.350+0000 7f0e2c176700 1 --2- 172.21.15.42:0/708852417 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0e0c03b6c0 0x7f0e0c03dbd0 crc :-1 s=READY pgs=1018 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:12.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:12.350+0000 7f0e2c176700 1 -- 172.21.15.42:0/708852417 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0e2407c680 msgr2=0x7f0e2413ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:12.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:12.350+0000 7f0e2c176700 1 --2- 172.21.15.42:0/708852417 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0e2407c680 0x7f0e2413ac40 secure :-1 s=READY pgs=1841 cs=0 l=1 rx=0x7f0e2411c120 tx=0x7f0e18007a50).stop 2019-12-05T06:19:12.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:12.350+0000 7f0e2c176700 1 -- 172.21.15.42:0/708852417 shutdown_connections 2019-12-05T06:19:12.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:12.350+0000 7f0e2c176700 1 --2- 172.21.15.42:0/708852417 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0e0c03b6c0 0x7f0e0c03dbd0 unknown :-1 s=CLOSED pgs=1018 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:12.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:12.350+0000 7f0e2c176700 1 --2- 172.21.15.42:0/708852417 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0e2407c680 0x7f0e2413ac40 unknown :-1 s=CLOSED pgs=1841 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:12.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:12.350+0000 7f0e2c176700 1 --2- 172.21.15.42:0/708852417 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0e24116500 0x7f0e2411a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:12.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:12.350+0000 7f0e2c176700 1 -- 172.21.15.42:0/708852417 shutdown_connections 2019-12-05T06:19:12.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:12.350+0000 7f0e2c176700 1 -- 172.21.15.42:0/708852417 wait complete. 2019-12-05T06:19:12.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:12.350+0000 7f0e2c176700 1 -- 172.21.15.42:0/708852417 >> 172.21.15.42:0/708852417 conn(0x7f0e240a7f90 msgr2=0x7f0e24059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:19:12.365 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 289, "feature_flags": { "enable_multiple": false, "ever_enabled_multiple": false }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2019-12-05T05:03:35.604738+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 289, "ever_allowed_features": 2, "explicitly_allowed_features": 2, "failed": [], "flags": 18, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_10476": { "addr": "172.21.15.40:6833/1473620421", "addrs": { "addrvec": [ { "addr": "172.21.15.40:6832", "nonce": 1473620421, "type": "v2" }, { "addr": "172.21.15.40:6833", "nonce": 1473620421, "type": "v1" } ] }, "export_targets": [], "features": 4540138292836696063, "flags": 0, "gid": 10476, "incarnation": 289, "name": "b", "rank": 0, "state": "up:replay", "state_seq": 2 } }, "last_failure": 0, "last_failure_osd_epoch": 101, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 2, "min_compat_client": "0 (unknown)", "modified": "2019-12-05T06:19:09.751427+0000", "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 10476 } } } ], "standbys": [] } 2019-12-05T06:19:14.367 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:19:14.476 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:14.466+0000 7f4a23183700 1 Processor -- start 2019-12-05T06:19:14.477 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:14.466+0000 7f4a23183700 1 -- start start 2019-12-05T06:19:14.477 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:14.466+0000 7f4a23183700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f4a1c0a7740 0x7f4a1c08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:19:14.477 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:14.466+0000 7f4a23183700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4a1c07e720 0x7f4a1c07eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:19:14.477 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:14.470+0000 7f4a23183700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a1c0a0710 con 0x7f4a1c09f2e0 2019-12-05T06:19:14.477 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:14.470+0000 7f4a23183700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a1c0a0b00 con 0x7f4a1c0a7740 2019-12-05T06:19:14.477 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:14.470+0000 7f4a23183700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a1c08e5b0 con 0x7f4a1c07e720 2019-12-05T06:19:14.478 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:14.470+0000 7f4a21720700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f4a1c0a7740 0x7f4a1c08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:14.478 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:14.470+0000 7f4a21720700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f4a1c0a7740 0x7f4a1c08df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:52778/0 (socket says 172.21.15.42:52778) 2019-12-05T06:19:14.478 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:14.470+0000 7f4a21f21700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f4a1c09f2e0 0x7f4a1c09f710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:52812/0 (socket says 172.21.15.42:52812) 2019-12-05T06:19:14.478 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:14.470+0000 7f4a21720700 1 -- 172.21.15.42:0/3184051449 learned_addr learned my addr 172.21.15.42:0/3184051449 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:19:14.478 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:14.470+0000 7f4a20f1f700 1 --2- 172.21.15.42:0/3184051449 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4a1c07e720 0x7f4a1c07eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:14.478 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:14.470+0000 7f4a20f1f700 1 -- 172.21.15.42:0/3184051449 >> v2:172.21.15.42:3300/0 conn(0x7f4a1c0a7740 msgr2=0x7f4a1c08df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:19:14.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:14.470+0000 7f4a20f1f700 1 --2- 172.21.15.42:0/3184051449 >> v2:172.21.15.42:3300/0 conn(0x7f4a1c0a7740 0x7f4a1c08df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:19:14.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:14.470+0000 7f4a20f1f700 1 -- 172.21.15.42:0/3184051449 >> v1:172.21.15.42:6789/0 conn(0x7f4a1c09f2e0 legacy=0x7f4a1c09f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:14.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:14.470+0000 7f4a20f1f700 1 -- 172.21.15.42:0/3184051449 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a1c07f0a0 con 0x7f4a1c07e720 2019-12-05T06:19:14.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:14.470+0000 7f4a20f1f700 1 --2- 172.21.15.42:0/3184051449 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4a1c07e720 0x7f4a1c07eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f4a0c008790 tx=0x7f4a0c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:14.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:14.470+0000 7f4a20f1f700 1 --2- 172.21.15.42:0/3184051449 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4a1c07e720 0x7f4a1c07eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f4a0c008790 tx=0x7f4a0c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:14.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:14.470+0000 7f4a20f1f700 1 --2- 172.21.15.42:0/3184051449 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4a1c07e720 0x7f4a1c07eb50 secure :-1 s=READY pgs=533 cs=0 l=1 rx=0x7f4a0c008790 tx=0x7f4a0c00fc70).ready entity=mon.2 client_cookie=499293a945ef7bea server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:19:14.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:14.470+0000 7f4a20f1f700 1 --2- 172.21.15.42:0/3184051449 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4a1c07e720 0x7f4a1c07eb50 secure :-1 s=THROTTLE_DONE pgs=533 cs=0 l=1 rx=0x7f4a0c008790 tx=0x7f4a0c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:14.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:14.470+0000 7f4a13fff700 1 -- 172.21.15.42:0/3184051449 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4a0c0112f0 con 0x7f4a1c07e720 2019-12-05T06:19:14.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:14.470+0000 7f4a20f1f700 1 --2- 172.21.15.42:0/3184051449 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4a1c07e720 0x7f4a1c07eb50 secure :-1 s=THROTTLE_DONE pgs=533 cs=0 l=1 rx=0x7f4a0c008790 tx=0x7f4a0c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:14.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:14.470+0000 7f4a13fff700 1 -- 172.21.15.42:0/3184051449 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a0c008ad0 con 0x7f4a1c07e720 2019-12-05T06:19:14.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:14.470+0000 7f4a20f1f700 1 --2- 172.21.15.42:0/3184051449 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4a1c07e720 0x7f4a1c07eb50 secure :-1 s=THROTTLE_DONE pgs=533 cs=0 l=1 rx=0x7f4a0c008790 tx=0x7f4a0c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:14.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:14.470+0000 7f4a13fff700 1 -- 172.21.15.42:0/3184051449 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4a0c0118d0 con 0x7f4a1c07e720 2019-12-05T06:19:14.482 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:14.470+0000 7f4a23183700 1 -- 172.21.15.42:0/3184051449 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4a1c07e720 msgr2=0x7f4a1c07eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:14.482 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:14.470+0000 7f4a23183700 1 --2- 172.21.15.42:0/3184051449 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4a1c07e720 0x7f4a1c07eb50 secure :-1 s=READY pgs=533 cs=0 l=1 rx=0x7f4a0c008790 tx=0x7f4a0c00fc70).stop 2019-12-05T06:19:14.482 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:14.470+0000 7f4a23183700 1 -- 172.21.15.42:0/3184051449 shutdown_connections 2019-12-05T06:19:14.482 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:14.470+0000 7f4a23183700 1 --2- 172.21.15.42:0/3184051449 >> v2:172.21.15.42:3300/0 conn(0x7f4a1c0a7740 0x7f4a1c08df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:19:14.483 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:14.470+0000 7f4a23183700 1 --2- 172.21.15.42:0/3184051449 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4a1c07e720 0x7f4a1c07eb50 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:14.483 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:14.470+0000 7f4a23183700 1 -- 172.21.15.42:0/3184051449 shutdown_connections 2019-12-05T06:19:14.483 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:14.470+0000 7f4a23183700 1 -- 172.21.15.42:0/3184051449 wait complete. 2019-12-05T06:19:14.483 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:14.470+0000 7f4a23183700 1 -- 172.21.15.42:0/3184051449 >> 172.21.15.42:0/3184051449 conn(0x7f4a1c0aabe0 msgr2=0x7f4a1c0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:19:14.487 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:14.474+0000 7f4a23183700 1 Processor -- start 2019-12-05T06:19:14.487 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:14.474+0000 7f4a23183700 1 -- start start 2019-12-05T06:19:14.487 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:14.478+0000 7f4a23183700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f4a1c11a7c0 0x7f4a1c118680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:19:14.487 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:14.478+0000 7f4a23183700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a1c118bd0 con 0x7f4a1c11f8d0 2019-12-05T06:19:14.488 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:14.478+0000 7f4a23183700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a1c11abf0 con 0x7f4a1c1192c0 2019-12-05T06:19:14.488 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:14.478+0000 7f4a23183700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a1c11af60 con 0x7f4a1c11a7c0 2019-12-05T06:19:14.488 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:14.478+0000 7f4a21720700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f4a1c1192c0 0x7f4a1c07afe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:52818/0 (socket says 172.21.15.42:52818) 2019-12-05T06:19:14.488 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:14.478+0000 7f4a21720700 1 -- 172.21.15.42:0/3737462173 learned_addr learned my addr 172.21.15.42:0/3737462173 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:19:14.488 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:14.478+0000 7f4a20f1f700 1 --2- 172.21.15.42:0/3737462173 >> v2:172.21.15.40:3300/0 conn(0x7f4a1c11a7c0 0x7f4a1c118680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:14.488 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:14.478+0000 7f4a11ffb700 1 -- 172.21.15.42:0/3737462173 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3465617382 0 0) 0x7f4a1c11abf0 con 0x7f4a1c1192c0 2019-12-05T06:19:14.489 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:14.478+0000 7f4a11ffb700 1 -- 172.21.15.42:0/3737462173 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49fc0030b0 con 0x7f4a1c1192c0 2019-12-05T06:19:14.489 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:14.478+0000 7f4a20f1f700 1 -- 172.21.15.42:0/3737462173 >> v1:172.21.15.42:6789/0 conn(0x7f4a1c1192c0 legacy=0x7f4a1c07afe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:14.489 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:14.478+0000 7f4a20f1f700 1 -- 172.21.15.42:0/3737462173 >> v1:172.21.15.40:6789/0 conn(0x7f4a1c11f8d0 legacy=0x7f4a1c11fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:14.490 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:14.478+0000 7f4a11ffb700 1 -- 172.21.15.42:0/3737462173 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 134037361 0 0) 0x7f4a1c118bd0 con 0x7f4a1c11f8d0 2019-12-05T06:19:14.491 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:14.478+0000 7f4a20f1f700 1 -- 172.21.15.42:0/3737462173 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a1c13a960 con 0x7f4a1c11a7c0 2019-12-05T06:19:14.491 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:14.478+0000 7f4a20f1f700 1 --2- 172.21.15.42:0/3737462173 >> v2:172.21.15.40:3300/0 conn(0x7f4a1c11a7c0 0x7f4a1c118680 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f4a0c008790 tx=0x7f4a0c011ea0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:14.491 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:14.478+0000 7f4a20f1f700 1 --2- 172.21.15.42:0/3737462173 >> v2:172.21.15.40:3300/0 conn(0x7f4a1c11a7c0 0x7f4a1c118680 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f4a0c008790 tx=0x7f4a0c011ea0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:14.491 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:14.478+0000 7f4a20f1f700 1 --2- 172.21.15.42:0/3737462173 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4a1c11a7c0 0x7f4a1c118680 secure :-1 s=READY pgs=1844 cs=0 l=1 rx=0x7f4a0c008790 tx=0x7f4a0c011ea0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:19:14.491 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:14.478+0000 7f4a20f1f700 1 --2- 172.21.15.42:0/3737462173 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4a1c11a7c0 0x7f4a1c118680 secure :-1 s=THROTTLE_DONE pgs=1844 cs=0 l=1 rx=0x7f4a0c008790 tx=0x7f4a0c011ea0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:14.491 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:14.478+0000 7f4a20f1f700 1 --2- 172.21.15.42:0/3737462173 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4a1c11a7c0 0x7f4a1c118680 secure :-1 s=THROTTLE_DONE pgs=1844 cs=0 l=1 rx=0x7f4a0c008790 tx=0x7f4a0c011ea0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:14.492 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:14.478+0000 7f4a11ffb700 1 -- 172.21.15.42:0/3737462173 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4a0c00f360 con 0x7f4a1c11a7c0 2019-12-05T06:19:14.492 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:14.478+0000 7f4a20f1f700 1 --2- 172.21.15.42:0/3737462173 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4a1c11a7c0 0x7f4a1c118680 secure :-1 s=THROTTLE_DONE pgs=1844 cs=0 l=1 rx=0x7f4a0c008790 tx=0x7f4a0c011ea0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:14.492 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:14.478+0000 7f4a11ffb700 1 -- 172.21.15.42:0/3737462173 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a0c0114f0 con 0x7f4a1c11a7c0 2019-12-05T06:19:14.492 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:14.478+0000 7f4a23183700 1 -- 172.21.15.42:0/3737462173 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a1c138940 con 0x7f4a1c11a7c0 2019-12-05T06:19:14.493 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:14.478+0000 7f4a11ffb700 1 -- 172.21.15.42:0/3737462173 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4a0c007340 con 0x7f4a1c11a7c0 2019-12-05T06:19:14.493 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:14.478+0000 7f4a23183700 1 -- 172.21.15.42:0/3737462173 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a1c138e40 con 0x7f4a1c11a7c0 2019-12-05T06:19:14.493 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:14.478+0000 7f4a20f1f700 1 --2- 172.21.15.42:0/3737462173 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4a1c11a7c0 0x7f4a1c118680 secure :-1 s=THROTTLE_DONE pgs=1844 cs=0 l=1 rx=0x7f4a0c008790 tx=0x7f4a0c011ea0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:14.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:14.482+0000 7f4a11ffb700 1 -- 172.21.15.42:0/3737462173 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f4a0c005540 con 0x7f4a1c11a7c0 2019-12-05T06:19:14.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:14.482+0000 7f4a20f1f700 1 --2- 172.21.15.42:0/3737462173 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4a1c11a7c0 0x7f4a1c118680 secure :-1 s=THROTTLE_DONE pgs=1844 cs=0 l=1 rx=0x7f4a0c008790 tx=0x7f4a0c011ea0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:14.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:14.482+0000 7f4a11ffb700 1 --2- 172.21.15.42:0/3737462173 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f49fc03b200 0x7f49fc03d710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:19:14.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:14.482+0000 7f4a11ffb700 1 -- 172.21.15.42:0/3737462173 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 10503+0+0 (secure 0 0 0) 0x7f4a0c08f8d0 con 0x7f4a1c11a7c0 2019-12-05T06:19:14.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:14.482+0000 7f4a21720700 1 --2- 172.21.15.42:0/3737462173 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f49fc03b200 0x7f49fc03d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:14.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:14.482+0000 7f4a21720700 1 --2- 172.21.15.42:0/3737462173 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f49fc03b200 0x7f49fc03d710 crc :-1 s=READY pgs=1019 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:19:14.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:14.482+0000 7f4a23183700 1 -- 172.21.15.42:0/3737462173 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a1c05d110 con 0x7f4a1c11a7c0 2019-12-05T06:19:14.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:14.490+0000 7f4a20f1f700 1 --2- 172.21.15.42:0/3737462173 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4a1c11a7c0 0x7f4a1c118680 secure :-1 s=THROTTLE_DONE pgs=1844 cs=0 l=1 rx=0x7f4a0c008790 tx=0x7f4a0c011ea0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:14.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:14.490+0000 7f4a11ffb700 1 -- 172.21.15.42:0/3737462173 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f4a0c0745e0 con 0x7f4a1c11a7c0 2019-12-05T06:19:14.662 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:14.658+0000 7f4a23183700 1 -- 172.21.15.42:0/3737462173 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4a1c05b8c0 con 0x7f4a1c11a7c0 2019-12-05T06:19:14.663 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:14.658+0000 7f4a20f1f700 1 --2- 172.21.15.42:0/3737462173 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4a1c11a7c0 0x7f4a1c118680 secure :-1 s=THROTTLE_DONE pgs=1844 cs=0 l=1 rx=0x7f4a0c008790 tx=0x7f4a0c011ea0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:14.664 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:14.658+0000 7f4a11ffb700 1 -- 172.21.15.42:0/3737462173 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 291 v291) v1 ==== 95+0+1779 (secure 0 0 0) 0x7f4a0c077800 con 0x7f4a1c11a7c0 2019-12-05T06:19:14.664 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 291 2019-12-05T06:19:14.664 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:19:14.664 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":291,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":291,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:19:14.097350+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":101,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10476},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10476":{"gid":10476,"name":"b","rank":0,"incarnation":289,"state":"up:rejoin","state_seq":33,"addr":"172.21.15.40:6833/1473620421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1473620421},{"type":"v1","addr":"172.21.15.40:6833","nonce":1473620421}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:19:14.666 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:14.662+0000 7f4a23183700 1 -- 172.21.15.42:0/3737462173 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f49fc03b200 msgr2=0x7f49fc03d710 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:14.666 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:14.662+0000 7f4a23183700 1 --2- 172.21.15.42:0/3737462173 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f49fc03b200 0x7f49fc03d710 crc :-1 s=READY pgs=1019 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:14.667 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:14.662+0000 7f4a23183700 1 -- 172.21.15.42:0/3737462173 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4a1c11a7c0 msgr2=0x7f4a1c118680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:14.667 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:14.662+0000 7f4a23183700 1 --2- 172.21.15.42:0/3737462173 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4a1c11a7c0 0x7f4a1c118680 secure :-1 s=READY pgs=1844 cs=0 l=1 rx=0x7f4a0c008790 tx=0x7f4a0c011ea0).stop 2019-12-05T06:19:14.667 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:14.662+0000 7f4a23183700 1 -- 172.21.15.42:0/3737462173 shutdown_connections 2019-12-05T06:19:14.667 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:14.662+0000 7f4a23183700 1 --2- 172.21.15.42:0/3737462173 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f49fc03b200 0x7f49fc03d710 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:14.668 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:14.662+0000 7f4a23183700 1 --2- 172.21.15.42:0/3737462173 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4a1c11a7c0 0x7f4a1c118680 unknown :-1 s=CLOSED pgs=1844 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:14.668 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:14.662+0000 7f4a23183700 1 -- 172.21.15.42:0/3737462173 shutdown_connections 2019-12-05T06:19:14.668 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:14.662+0000 7f4a23183700 1 -- 172.21.15.42:0/3737462173 wait complete. 2019-12-05T06:19:14.668 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:14.662+0000 7f4a23183700 1 -- 172.21.15.42:0/3737462173 >> 172.21.15.42:0/3737462173 conn(0x7f4a1c066fe0 msgr2=0x7f4a1c11c540 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:19:14.677 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2019-12-05T06:19:15.730 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:19:15.737 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:19:16.678 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:19:16.782 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:16.774+0000 7f5a3c227700 1 Processor -- start 2019-12-05T06:19:16.783 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:16.774+0000 7f5a3c227700 1 -- start start 2019-12-05T06:19:16.783 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:16.774+0000 7f5a3c227700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5a3408df70 0x7f5a3409f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:19:16.783 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:16.774+0000 7f5a3c227700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5a3409f820 0x7f5a3414efd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:19:16.783 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:16.774+0000 7f5a3c227700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a3409fc50 con 0x7f5a340a7730 2019-12-05T06:19:16.784 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:16.774+0000 7f5a3c227700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a340a0af0 con 0x7f5a3409f820 2019-12-05T06:19:16.784 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:16.774+0000 7f5a3c227700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a3408e5a0 con 0x7f5a3408df70 2019-12-05T06:19:16.784 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:16.774+0000 7f5a39fc3700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5a3408df70 0x7f5a3409f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:16.784 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:16.774+0000 7f5a39fc3700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5a3408df70 0x7f5a3409f2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:52796/0 (socket says 172.21.15.42:52796) 2019-12-05T06:19:16.785 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:16.774+0000 7f5a3a7c4700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f5a340a7730 0x7f5a3407e710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:52832/0 (socket says 172.21.15.42:52832) 2019-12-05T06:19:16.785 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:16.774+0000 7f5a39fc3700 1 -- 172.21.15.42:0/564899707 learned_addr learned my addr 172.21.15.42:0/564899707 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:19:16.785 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:16.774+0000 7f5a3afc5700 1 --2- 172.21.15.42:0/564899707 >> v2:172.21.15.40:3300/0 conn(0x7f5a3409f820 0x7f5a3414efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:16.785 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:16.774+0000 7f5a39fc3700 1 -- 172.21.15.42:0/564899707 >> v2:172.21.15.40:3300/0 conn(0x7f5a3409f820 msgr2=0x7f5a3414efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:19:16.785 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:16.774+0000 7f5a39fc3700 1 --2- 172.21.15.42:0/564899707 >> v2:172.21.15.40:3300/0 conn(0x7f5a3409f820 0x7f5a3414efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:19:16.785 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:16.774+0000 7f5a39fc3700 1 -- 172.21.15.42:0/564899707 >> v1:172.21.15.42:6789/0 conn(0x7f5a340a7730 legacy=0x7f5a3407e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:16.786 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:16.774+0000 7f5a397c2700 1 -- 172.21.15.42:0/564899707 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 313370745 0 0) 0x7f5a3409fc50 con 0x7f5a340a7730 2019-12-05T06:19:16.786 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:16.774+0000 7f5a39fc3700 1 -- 172.21.15.42:0/564899707 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a340a0700 con 0x7f5a3408df70 2019-12-05T06:19:16.786 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:16.774+0000 7f5a39fc3700 1 --2- 172.21.15.42:0/564899707 >> v2:172.21.15.42:3300/0 conn(0x7f5a3408df70 0x7f5a3409f2d0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f5a280032b0 tx=0x7f5a280100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:16.786 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:16.774+0000 7f5a39fc3700 1 --2- 172.21.15.42:0/564899707 >> v2:172.21.15.42:3300/0 conn(0x7f5a3408df70 0x7f5a3409f2d0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f5a280032b0 tx=0x7f5a280100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:16.786 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:16.774+0000 7f5a39fc3700 1 --2- 172.21.15.42:0/564899707 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5a3408df70 0x7f5a3409f2d0 secure :-1 s=READY pgs=2428 cs=0 l=1 rx=0x7f5a280032b0 tx=0x7f5a280100b0).ready entity=mon.1 client_cookie=9d5c6e7749418200 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:19:16.786 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:16.778+0000 7f5a39fc3700 1 --2- 172.21.15.42:0/564899707 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5a3408df70 0x7f5a3409f2d0 secure :-1 s=THROTTLE_DONE pgs=2428 cs=0 l=1 rx=0x7f5a280032b0 tx=0x7f5a280100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:16.786 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:16.778+0000 7f5a397c2700 1 -- 172.21.15.42:0/564899707 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5a28010d60 con 0x7f5a3408df70 2019-12-05T06:19:16.787 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:16.778+0000 7f5a39fc3700 1 --2- 172.21.15.42:0/564899707 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5a3408df70 0x7f5a3409f2d0 secure :-1 s=THROTTLE_DONE pgs=2428 cs=0 l=1 rx=0x7f5a280032b0 tx=0x7f5a280100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:16.787 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:16.778+0000 7f5a39fc3700 1 --2- 172.21.15.42:0/564899707 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5a3408df70 0x7f5a3409f2d0 secure :-1 s=THROTTLE_DONE pgs=2428 cs=0 l=1 rx=0x7f5a280032b0 tx=0x7f5a280100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:16.787 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:16.778+0000 7f5a397c2700 1 -- 172.21.15.42:0/564899707 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a28010980 con 0x7f5a3408df70 2019-12-05T06:19:16.787 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:16.778+0000 7f5a397c2700 1 -- 172.21.15.42:0/564899707 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5a280114a0 con 0x7f5a3408df70 2019-12-05T06:19:16.787 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:16.778+0000 7f5a3c227700 1 -- 172.21.15.42:0/564899707 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5a3408df70 msgr2=0x7f5a3409f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:16.787 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:16.778+0000 7f5a3c227700 1 --2- 172.21.15.42:0/564899707 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5a3408df70 0x7f5a3409f2d0 secure :-1 s=READY pgs=2428 cs=0 l=1 rx=0x7f5a280032b0 tx=0x7f5a280100b0).stop 2019-12-05T06:19:16.787 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:16.778+0000 7f5a3c227700 1 -- 172.21.15.42:0/564899707 shutdown_connections 2019-12-05T06:19:16.788 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:16.778+0000 7f5a3c227700 1 --2- 172.21.15.42:0/564899707 >> v2:172.21.15.40:3300/0 conn(0x7f5a3409f820 0x7f5a3414efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:19:16.788 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:16.778+0000 7f5a3c227700 1 --2- 172.21.15.42:0/564899707 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5a3408df70 0x7f5a3409f2d0 unknown :-1 s=CLOSED pgs=2428 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:16.788 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:16.778+0000 7f5a3c227700 1 -- 172.21.15.42:0/564899707 shutdown_connections 2019-12-05T06:19:16.788 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:16.778+0000 7f5a3c227700 1 -- 172.21.15.42:0/564899707 wait complete. 2019-12-05T06:19:16.788 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:16.778+0000 7f5a3c227700 1 -- 172.21.15.42:0/564899707 >> 172.21.15.42:0/564899707 conn(0x7f5a340aabd0 msgr2=0x7f5a340a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:19:16.791 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:16.778+0000 7f5a3c227700 1 Processor -- start 2019-12-05T06:19:16.792 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:16.778+0000 7f5a3c227700 1 -- start start 2019-12-05T06:19:16.792 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:16.782+0000 7f5a3c227700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5a3407bc00 0x7f5a3407c030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:19:16.792 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:16.782+0000 7f5a3c227700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a34133690 con 0x7f5a34118720 2019-12-05T06:19:16.792 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:16.782+0000 7f5a3c227700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a3411b7f0 con 0x7f5a341164f0 2019-12-05T06:19:16.792 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:16.782+0000 7f5a3c227700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a34118bf0 con 0x7f5a3407bc00 2019-12-05T06:19:16.792 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:16.782+0000 7f5a39fc3700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f5a341164f0 0x7f5a3411aff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:52842/0 (socket says 172.21.15.42:52842) 2019-12-05T06:19:16.793 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:16.782+0000 7f5a39fc3700 1 -- 172.21.15.42:0/3300983788 learned_addr learned my addr 172.21.15.42:0/3300983788 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:19:16.793 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:16.782+0000 7f5a3afc5700 1 --2- 172.21.15.42:0/3300983788 >> v2:172.21.15.40:3300/0 conn(0x7f5a3407bc00 0x7f5a3407c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:16.793 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:16.782+0000 7f5a26ffd700 1 -- 172.21.15.42:0/3300983788 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3333909037 0 0) 0x7f5a3411b7f0 con 0x7f5a341164f0 2019-12-05T06:19:16.793 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:16.782+0000 7f5a26ffd700 1 -- 172.21.15.42:0/3300983788 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a1c0030b0 con 0x7f5a341164f0 2019-12-05T06:19:16.793 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:16.782+0000 7f5a26ffd700 1 -- 172.21.15.42:0/3300983788 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2336221011 0 0) 0x7f5a34133690 con 0x7f5a34118720 2019-12-05T06:19:16.793 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:16.782+0000 7f5a26ffd700 1 -- 172.21.15.42:0/3300983788 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a3411b7f0 con 0x7f5a34118720 2019-12-05T06:19:16.793 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:16.782+0000 7f5a26ffd700 1 -- 172.21.15.42:0/3300983788 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 3401394073 0 0) 0x7f5a1c0030b0 con 0x7f5a341164f0 2019-12-05T06:19:16.794 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:16.782+0000 7f5a3afc5700 1 -- 172.21.15.42:0/3300983788 >> v1:172.21.15.42:6789/0 conn(0x7f5a341164f0 legacy=0x7f5a3411aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:16.794 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:16.782+0000 7f5a3afc5700 1 -- 172.21.15.42:0/3300983788 >> v1:172.21.15.40:6789/0 conn(0x7f5a34118720 legacy=0x7f5a34133dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:16.794 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:16.782+0000 7f5a3afc5700 1 -- 172.21.15.42:0/3300983788 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a3413db10 con 0x7f5a3407bc00 2019-12-05T06:19:16.794 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:16.782+0000 7f5a3afc5700 1 --2- 172.21.15.42:0/3300983788 >> v2:172.21.15.40:3300/0 conn(0x7f5a3407bc00 0x7f5a3407c030 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f5a34124e70 tx=0x7f5a28008650).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:16.794 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:16.782+0000 7f5a3afc5700 1 --2- 172.21.15.42:0/3300983788 >> v2:172.21.15.40:3300/0 conn(0x7f5a3407bc00 0x7f5a3407c030 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f5a34124e70 tx=0x7f5a28008650).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:16.794 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:16.782+0000 7f5a3afc5700 1 --2- 172.21.15.42:0/3300983788 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5a3407bc00 0x7f5a3407c030 secure :-1 s=READY pgs=1846 cs=0 l=1 rx=0x7f5a34124e70 tx=0x7f5a28008650).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:19:16.794 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:16.782+0000 7f5a3afc5700 1 --2- 172.21.15.42:0/3300983788 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5a3407bc00 0x7f5a3407c030 secure :-1 s=THROTTLE_DONE pgs=1846 cs=0 l=1 rx=0x7f5a34124e70 tx=0x7f5a28008650).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:16.795 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:16.782+0000 7f5a26ffd700 1 -- 172.21.15.42:0/3300983788 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5a28010980 con 0x7f5a3407bc00 2019-12-05T06:19:16.795 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:16.782+0000 7f5a3afc5700 1 --2- 172.21.15.42:0/3300983788 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5a3407bc00 0x7f5a3407c030 secure :-1 s=THROTTLE_DONE pgs=1846 cs=0 l=1 rx=0x7f5a34124e70 tx=0x7f5a28008650).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:16.795 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:16.782+0000 7f5a3c227700 1 -- 172.21.15.42:0/3300983788 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a3413baf0 con 0x7f5a3407bc00 2019-12-05T06:19:16.796 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:16.782+0000 7f5a26ffd700 1 -- 172.21.15.42:0/3300983788 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a280110c0 con 0x7f5a3407bc00 2019-12-05T06:19:16.796 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:16.782+0000 7f5a3afc5700 1 --2- 172.21.15.42:0/3300983788 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5a3407bc00 0x7f5a3407c030 secure :-1 s=THROTTLE_DONE pgs=1846 cs=0 l=1 rx=0x7f5a34124e70 tx=0x7f5a28008650).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:16.805 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:16.782+0000 7f5a26ffd700 1 -- 172.21.15.42:0/3300983788 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5a2c005b20 con 0x7f5a3407bc00 2019-12-05T06:19:16.805 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:16.782+0000 7f5a3c227700 1 -- 172.21.15.42:0/3300983788 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a3413c040 con 0x7f5a3407bc00 2019-12-05T06:19:16.805 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:16.786+0000 7f5a3afc5700 1 --2- 172.21.15.42:0/3300983788 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5a3407bc00 0x7f5a3407c030 secure :-1 s=THROTTLE_DONE pgs=1846 cs=0 l=1 rx=0x7f5a34124e70 tx=0x7f5a28008650).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:16.806 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:16.786+0000 7f5a26ffd700 1 -- 172.21.15.42:0/3300983788 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f5a2c007ae0 con 0x7f5a3407bc00 2019-12-05T06:19:16.806 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:16.786+0000 7f5a3afc5700 1 --2- 172.21.15.42:0/3300983788 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5a3407bc00 0x7f5a3407c030 secure :-1 s=THROTTLE_DONE pgs=1846 cs=0 l=1 rx=0x7f5a34124e70 tx=0x7f5a28008650).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:16.806 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:16.786+0000 7f5a26ffd700 1 --2- 172.21.15.42:0/3300983788 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5a1c03b400 0x7f5a1c03d910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:19:16.806 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:16.786+0000 7f5a26ffd700 1 -- 172.21.15.42:0/3300983788 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 10503+0+0 (secure 0 0 0) 0x7f5a2c011dc0 con 0x7f5a3407bc00 2019-12-05T06:19:16.807 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:16.786+0000 7f5a3a7c4700 1 --2- 172.21.15.42:0/3300983788 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5a1c03b400 0x7f5a1c03d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:16.807 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:16.786+0000 7f5a3a7c4700 1 --2- 172.21.15.42:0/3300983788 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5a1c03b400 0x7f5a1c03d910 crc :-1 s=READY pgs=1020 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:19:16.808 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:16.790+0000 7f5a3c227700 1 -- 172.21.15.42:0/3300983788 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a3405d100 con 0x7f5a3407bc00 2019-12-05T06:19:16.808 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:16.802+0000 7f5a3afc5700 1 --2- 172.21.15.42:0/3300983788 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5a3407bc00 0x7f5a3407c030 secure :-1 s=THROTTLE_DONE pgs=1846 cs=0 l=1 rx=0x7f5a34124e70 tx=0x7f5a28008650).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:16.808 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:16.802+0000 7f5a26ffd700 1 -- 172.21.15.42:0/3300983788 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f5a2c0316f0 con 0x7f5a3407bc00 2019-12-05T06:19:16.982 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:16.974+0000 7f5a3c227700 1 -- 172.21.15.42:0/3300983788 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5a3405b8b0 con 0x7f5a3407bc00 2019-12-05T06:19:16.982 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:16.978+0000 7f5a3afc5700 1 --2- 172.21.15.42:0/3300983788 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5a3407bc00 0x7f5a3407c030 secure :-1 s=THROTTLE_DONE pgs=1846 cs=0 l=1 rx=0x7f5a34124e70 tx=0x7f5a28008650).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:16.983 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:16.978+0000 7f5a26ffd700 1 -- 172.21.15.42:0/3300983788 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 292 v292) v1 ==== 95+0+1779 (secure 0 0 0) 0x7f5a2c031950 con 0x7f5a3407bc00 2019-12-05T06:19:16.983 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 292 2019-12-05T06:19:16.983 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:19:16.984 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":292,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":292,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:19:15.110045+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":101,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10476},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10476":{"gid":10476,"name":"b","rank":0,"incarnation":289,"state":"up:active","state_seq":34,"addr":"172.21.15.40:6833/1473620421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1473620421},{"type":"v1","addr":"172.21.15.40:6833","nonce":1473620421}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:19:16.986 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:16.978+0000 7f5a3c227700 1 -- 172.21.15.42:0/3300983788 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5a1c03b400 msgr2=0x7f5a1c03d910 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:16.986 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:16.978+0000 7f5a3c227700 1 --2- 172.21.15.42:0/3300983788 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5a1c03b400 0x7f5a1c03d910 crc :-1 s=READY pgs=1020 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:16.986 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:16.978+0000 7f5a3c227700 1 -- 172.21.15.42:0/3300983788 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5a3407bc00 msgr2=0x7f5a3407c030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:16.986 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:16.978+0000 7f5a3c227700 1 --2- 172.21.15.42:0/3300983788 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5a3407bc00 0x7f5a3407c030 secure :-1 s=READY pgs=1846 cs=0 l=1 rx=0x7f5a34124e70 tx=0x7f5a28008650).stop 2019-12-05T06:19:16.987 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:16.978+0000 7f5a3c227700 1 -- 172.21.15.42:0/3300983788 shutdown_connections 2019-12-05T06:19:16.988 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:16.978+0000 7f5a3c227700 1 --2- 172.21.15.42:0/3300983788 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5a1c03b400 0x7f5a1c03d910 unknown :-1 s=CLOSED pgs=1020 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:16.988 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:16.978+0000 7f5a3c227700 1 --2- 172.21.15.42:0/3300983788 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5a3407bc00 0x7f5a3407c030 unknown :-1 s=CLOSED pgs=1846 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:16.988 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:16.978+0000 7f5a3c227700 1 -- 172.21.15.42:0/3300983788 shutdown_connections 2019-12-05T06:19:16.988 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:16.978+0000 7f5a3c227700 1 -- 172.21.15.42:0/3300983788 wait complete. 2019-12-05T06:19:16.989 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:16.978+0000 7f5a3c227700 1 -- 172.21.15.42:0/3300983788 >> 172.21.15.42:0/3300983788 conn(0x7f5a3411c760 msgr2=0x7f5a3407e4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:19:16.994 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b has gained rank=0, replacing gid=7295 2019-12-05T06:19:16.994 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 1 secs before reviving mds.a 2019-12-05T06:19:17.995 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.a 2019-12-05T06:19:17.995 INFO:tasks.ceph.mds.a:Restarting daemon 2019-12-05T06:19:17.995 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2019-12-05T06:19:17.999 INFO:tasks.ceph.mds.a:Started 2019-12-05T06:19:17.999 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:19:18.059 INFO:tasks.ceph.mds.a.smithi042.stdout:starting mds.a at 2019-12-05T06:19:18.096 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:18.086+0000 7f65f1abd700 1 Processor -- start 2019-12-05T06:19:18.096 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:18.086+0000 7f65f1abd700 1 -- start start 2019-12-05T06:19:18.096 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:18.086+0000 7f65f1abd700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f65ec0a7740 0x7f65ec08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:19:18.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:18.086+0000 7f65f1abd700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f65ec07e720 0x7f65ec07eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:19:18.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:18.086+0000 7f65f1abd700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65ec0a0710 con 0x7f65ec09f2e0 2019-12-05T06:19:18.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:18.086+0000 7f65f1abd700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65ec0a0b00 con 0x7f65ec0a7740 2019-12-05T06:19:18.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:18.086+0000 7f65f1abd700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65ec08e5b0 con 0x7f65ec07e720 2019-12-05T06:19:18.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:18.086+0000 7f65ebfff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f65ec0a7740 0x7f65ec08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:18.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:18.086+0000 7f65ebfff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f65ec0a7740 0x7f65ec08df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:52826/0 (socket says 172.21.15.42:52826) 2019-12-05T06:19:18.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:18.086+0000 7f65ebfff700 1 -- 172.21.15.42:0/2515429041 learned_addr learned my addr 172.21.15.42:0/2515429041 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:19:18.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:18.086+0000 7f65eb7fe700 1 --2- 172.21.15.42:0/2515429041 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f65ec07e720 0x7f65ec07eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:18.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:18.086+0000 7f65ebfff700 1 -- 172.21.15.42:0/2515429041 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f65ec07e720 msgr2=0x7f65ec07eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:19:18.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:18.086+0000 7f65ebfff700 1 --2- 172.21.15.42:0/2515429041 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f65ec07e720 0x7f65ec07eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:19:18.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:18.086+0000 7f65ebfff700 1 -- 172.21.15.42:0/2515429041 >> v1:172.21.15.42:6789/0 conn(0x7f65ec09f2e0 legacy=0x7f65ec09f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:18.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:18.086+0000 7f65ebfff700 1 -- 172.21.15.42:0/2515429041 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65ec07f0a0 con 0x7f65ec0a7740 2019-12-05T06:19:18.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:18.086+0000 7f65ebfff700 1 --2- 172.21.15.42:0/2515429041 >> v2:172.21.15.42:3300/0 conn(0x7f65ec0a7740 0x7f65ec08df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f65e4008b10 tx=0x7f65e40100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:18.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:18.086+0000 7f65ebfff700 1 --2- 172.21.15.42:0/2515429041 >> v2:172.21.15.42:3300/0 conn(0x7f65ec0a7740 0x7f65ec08df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f65e4008b10 tx=0x7f65e40100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:18.100 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:18.086+0000 7f65ebfff700 1 --2- 172.21.15.42:0/2515429041 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f65ec0a7740 0x7f65ec08df80 secure :-1 s=READY pgs=2434 cs=0 l=1 rx=0x7f65e4008b10 tx=0x7f65e40100b0).ready entity=mon.1 client_cookie=c97b2631c227bdf7 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:19:18.100 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:18.086+0000 7f65ebfff700 1 --2- 172.21.15.42:0/2515429041 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f65ec0a7740 0x7f65ec08df80 secure :-1 s=THROTTLE_DONE pgs=2434 cs=0 l=1 rx=0x7f65e4008b10 tx=0x7f65e40100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:18.100 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:18.086+0000 7f65ebfff700 1 --2- 172.21.15.42:0/2515429041 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f65ec0a7740 0x7f65ec08df80 secure :-1 s=THROTTLE_DONE pgs=2434 cs=0 l=1 rx=0x7f65e4008b10 tx=0x7f65e40100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:18.100 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:18.086+0000 7f65eaffd700 1 -- 172.21.15.42:0/2515429041 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f65e400a160 con 0x7f65ec0a7740 2019-12-05T06:19:18.100 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:18.086+0000 7f65ebfff700 1 --2- 172.21.15.42:0/2515429041 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f65ec0a7740 0x7f65ec08df80 secure :-1 s=THROTTLE_DONE pgs=2434 cs=0 l=1 rx=0x7f65e4008b10 tx=0x7f65e40100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:18.100 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:18.086+0000 7f65eaffd700 1 -- 172.21.15.42:0/2515429041 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65e4011a30 con 0x7f65ec0a7740 2019-12-05T06:19:18.100 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:18.086+0000 7f65eaffd700 1 -- 172.21.15.42:0/2515429041 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f65e400a550 con 0x7f65ec0a7740 2019-12-05T06:19:18.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:18.086+0000 7f65f1abd700 1 -- 172.21.15.42:0/2515429041 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f65ec0a7740 msgr2=0x7f65ec08df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:18.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:18.086+0000 7f65f1abd700 1 --2- 172.21.15.42:0/2515429041 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f65ec0a7740 0x7f65ec08df80 secure :-1 s=READY pgs=2434 cs=0 l=1 rx=0x7f65e4008b10 tx=0x7f65e40100b0).stop 2019-12-05T06:19:18.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:18.086+0000 7f65f1abd700 1 -- 172.21.15.42:0/2515429041 shutdown_connections 2019-12-05T06:19:18.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:18.086+0000 7f65f1abd700 1 --2- 172.21.15.42:0/2515429041 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f65ec0a7740 0x7f65ec08df80 unknown :-1 s=CLOSED pgs=2434 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:18.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:18.086+0000 7f65f1abd700 1 --2- 172.21.15.42:0/2515429041 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f65ec07e720 0x7f65ec07eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:19:18.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:18.086+0000 7f65f1abd700 1 -- 172.21.15.42:0/2515429041 shutdown_connections 2019-12-05T06:19:18.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:18.086+0000 7f65f1abd700 1 -- 172.21.15.42:0/2515429041 wait complete. 2019-12-05T06:19:18.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:18.086+0000 7f65f1abd700 1 -- 172.21.15.42:0/2515429041 >> 172.21.15.42:0/2515429041 conn(0x7f65ec0aabe0 msgr2=0x7f65ec0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:19:18.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:18.090+0000 7f65f1abd700 1 Processor -- start 2019-12-05T06:19:18.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:18.090+0000 7f65f1abd700 1 -- start start 2019-12-05T06:19:18.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:18.090+0000 7f65f1abd700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f65ec11a660 0x7f65ec11da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:19:18.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:18.090+0000 7f65f1abd700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f65ec12d8c0 0x7f65ec12dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:19:18.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:18.090+0000 7f65f1abd700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65ec11e2f0 con 0x7f65ec11f680 2019-12-05T06:19:18.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:18.090+0000 7f65f1abd700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65ec11f140 con 0x7f65ec11a660 2019-12-05T06:19:18.103 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:18.090+0000 7f65f1abd700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65ec11f3e0 con 0x7f65ec12d8c0 2019-12-05T06:19:18.103 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:18.090+0000 7f65ebfff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f65ec11a660 0x7f65ec11da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:18.103 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:18.090+0000 7f65ebfff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f65ec11a660 0x7f65ec11da50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:52834/0 (socket says 172.21.15.42:52834) 2019-12-05T06:19:18.103 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:18.090+0000 7f65ebfff700 1 -- 172.21.15.42:0/4044374692 learned_addr learned my addr 172.21.15.42:0/4044374692 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:19:18.103 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:18.090+0000 7f65f085b700 1 --2- 172.21.15.42:0/4044374692 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f65ec12d8c0 0x7f65ec12dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:18.103 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:18.090+0000 7f65ebfff700 1 -- 172.21.15.42:0/4044374692 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f65ec12d8c0 msgr2=0x7f65ec12dcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:18.103 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:18.090+0000 7f65ebfff700 1 --2- 172.21.15.42:0/4044374692 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f65ec12d8c0 0x7f65ec12dcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:18.104 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:18.090+0000 7f65ebfff700 1 -- 172.21.15.42:0/4044374692 >> v1:172.21.15.40:6789/0 conn(0x7f65ec11f680 legacy=0x7f65ec11fab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:18.104 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:18.090+0000 7f65ebfff700 1 -- 172.21.15.42:0/4044374692 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65ec11dfa0 con 0x7f65ec11a660 2019-12-05T06:19:18.104 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:18.090+0000 7f65e8ff9700 1 -- 172.21.15.42:0/4044374692 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 245403692 0 0) 0x7f65ec11e2f0 con 0x7f65ec11f680 2019-12-05T06:19:18.104 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:18.090+0000 7f65f085b700 1 --2- 172.21.15.42:0/4044374692 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f65ec12d8c0 0x7f65ec12dcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:19:18.104 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:18.090+0000 7f65ebfff700 1 --2- 172.21.15.42:0/4044374692 >> v2:172.21.15.42:3300/0 conn(0x7f65ec11a660 0x7f65ec11da50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f65ec11e520 tx=0x7f65e400ab40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:18.104 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:18.090+0000 7f65ebfff700 1 --2- 172.21.15.42:0/4044374692 >> v2:172.21.15.42:3300/0 conn(0x7f65ec11a660 0x7f65ec11da50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f65ec11e520 tx=0x7f65e400ab40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:18.104 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:18.090+0000 7f65ebfff700 1 --2- 172.21.15.42:0/4044374692 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f65ec11a660 0x7f65ec11da50 secure :-1 s=READY pgs=2435 cs=0 l=1 rx=0x7f65ec11e520 tx=0x7f65e400ab40).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:19:18.105 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:18.090+0000 7f65ebfff700 1 --2- 172.21.15.42:0/4044374692 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f65ec11a660 0x7f65ec11da50 secure :-1 s=THROTTLE_DONE pgs=2435 cs=0 l=1 rx=0x7f65ec11e520 tx=0x7f65e400ab40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:18.105 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:18.090+0000 7f65e8ff9700 1 -- 172.21.15.42:0/4044374692 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f65e4005ab0 con 0x7f65ec11a660 2019-12-05T06:19:18.105 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:18.090+0000 7f65ebfff700 1 --2- 172.21.15.42:0/4044374692 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f65ec11a660 0x7f65ec11da50 secure :-1 s=THROTTLE_DONE pgs=2435 cs=0 l=1 rx=0x7f65ec11e520 tx=0x7f65e400ab40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:18.105 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:18.090+0000 7f65ebfff700 1 --2- 172.21.15.42:0/4044374692 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f65ec11a660 0x7f65ec11da50 secure :-1 s=THROTTLE_DONE pgs=2435 cs=0 l=1 rx=0x7f65ec11e520 tx=0x7f65e400ab40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:18.108 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:18.090+0000 7f65e8ff9700 1 -- 172.21.15.42:0/4044374692 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65e4011e10 con 0x7f65ec11a660 2019-12-05T06:19:18.108 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:18.090+0000 7f65e8ff9700 1 -- 172.21.15.42:0/4044374692 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f65e4007340 con 0x7f65ec11a660 2019-12-05T06:19:18.108 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:18.090+0000 7f65f1abd700 1 -- 172.21.15.42:0/4044374692 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65ec0a5e50 con 0x7f65ec11a660 2019-12-05T06:19:18.108 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:18.090+0000 7f65f1abd700 1 -- 172.21.15.42:0/4044374692 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65ec12e460 con 0x7f65ec11a660 2019-12-05T06:19:18.108 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:18.090+0000 7f65ebfff700 1 --2- 172.21.15.42:0/4044374692 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f65ec11a660 0x7f65ec11da50 secure :-1 s=THROTTLE_DONE pgs=2435 cs=0 l=1 rx=0x7f65ec11e520 tx=0x7f65e400ab40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:18.110 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:18.094+0000 7f65e8ff9700 1 -- 172.21.15.42:0/4044374692 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f65e400f360 con 0x7f65ec11a660 2019-12-05T06:19:18.110 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:18.094+0000 7f65ebfff700 1 --2- 172.21.15.42:0/4044374692 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f65ec11a660 0x7f65ec11da50 secure :-1 s=THROTTLE_DONE pgs=2435 cs=0 l=1 rx=0x7f65ec11e520 tx=0x7f65e400ab40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:18.110 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:18.094+0000 7f65f1abd700 1 -- 172.21.15.42:0/4044374692 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65ec05d110 con 0x7f65ec11a660 2019-12-05T06:19:18.110 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:18.094+0000 7f65e8ff9700 1 --2- 172.21.15.42:0/4044374692 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f65d003ac80 0x7f65d003d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:19:18.110 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:18.094+0000 7f65e8ff9700 1 -- 172.21.15.42:0/4044374692 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 10503+0+0 (secure 0 0 0) 0x7f65e404bd80 con 0x7f65ec11a660 2019-12-05T06:19:18.111 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:18.094+0000 7f65eb7fe700 1 --2- 172.21.15.42:0/4044374692 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f65d003ac80 0x7f65d003d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:18.111 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:18.094+0000 7f65eb7fe700 1 --2- 172.21.15.42:0/4044374692 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f65d003ac80 0x7f65d003d190 crc :-1 s=READY pgs=1021 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:19:18.111 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:18.098+0000 7f65ebfff700 1 --2- 172.21.15.42:0/4044374692 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f65ec11a660 0x7f65ec11da50 secure :-1 s=THROTTLE_DONE pgs=2435 cs=0 l=1 rx=0x7f65ec11e520 tx=0x7f65e400ab40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:18.111 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:18.098+0000 7f65e8ff9700 1 -- 172.21.15.42:0/4044374692 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f65e4033060 con 0x7f65ec11a660 2019-12-05T06:19:18.266 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:18.262+0000 7f65f1abd700 1 -- 172.21.15.42:0/4044374692 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f65ec05b8c0 con 0x7f65ec11a660 2019-12-05T06:19:18.267 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:18.262+0000 7f65ebfff700 1 --2- 172.21.15.42:0/4044374692 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f65ec11a660 0x7f65ec11da50 secure :-1 s=THROTTLE_DONE pgs=2435 cs=0 l=1 rx=0x7f65ec11e520 tx=0x7f65e400ab40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:18.267 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:18.262+0000 7f65e8ff9700 1 -- 172.21.15.42:0/4044374692 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 292 v292) v1 ==== 95+0+1779 (secure 0 0 0) 0x7f65e4007540 con 0x7f65ec11a660 2019-12-05T06:19:18.267 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 292 2019-12-05T06:19:18.268 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:19:18.268 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":292,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":292,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:19:15.110045+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":101,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10476},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10476":{"gid":10476,"name":"b","rank":0,"incarnation":289,"state":"up:active","state_seq":34,"addr":"172.21.15.40:6833/1473620421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1473620421},{"type":"v1","addr":"172.21.15.40:6833","nonce":1473620421}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:19:18.270 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:18.266+0000 7f65f1abd700 1 -- 172.21.15.42:0/4044374692 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f65d003ac80 msgr2=0x7f65d003d190 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:18.270 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:18.266+0000 7f65f1abd700 1 --2- 172.21.15.42:0/4044374692 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f65d003ac80 0x7f65d003d190 crc :-1 s=READY pgs=1021 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:18.270 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:18.266+0000 7f65f1abd700 1 -- 172.21.15.42:0/4044374692 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f65ec11a660 msgr2=0x7f65ec11da50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:18.271 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:18.266+0000 7f65f1abd700 1 --2- 172.21.15.42:0/4044374692 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f65ec11a660 0x7f65ec11da50 secure :-1 s=READY pgs=2435 cs=0 l=1 rx=0x7f65ec11e520 tx=0x7f65e400ab40).stop 2019-12-05T06:19:18.271 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:18.266+0000 7f65f1abd700 1 -- 172.21.15.42:0/4044374692 shutdown_connections 2019-12-05T06:19:18.271 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:18.266+0000 7f65f1abd700 1 --2- 172.21.15.42:0/4044374692 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f65d003ac80 0x7f65d003d190 unknown :-1 s=CLOSED pgs=1021 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:18.271 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:18.266+0000 7f65f1abd700 1 --2- 172.21.15.42:0/4044374692 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f65ec12d8c0 0x7f65ec12dcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:18.271 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:18.266+0000 7f65f1abd700 1 --2- 172.21.15.42:0/4044374692 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f65ec11a660 0x7f65ec11da50 unknown :-1 s=CLOSED pgs=2435 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:18.271 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:18.266+0000 7f65f1abd700 1 -- 172.21.15.42:0/4044374692 shutdown_connections 2019-12-05T06:19:18.271 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:18.266+0000 7f65f1abd700 1 -- 172.21.15.42:0/4044374692 wait complete. 2019-12-05T06:19:18.272 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:18.266+0000 7f65f1abd700 1 -- 172.21.15.42:0/4044374692 >> 172.21.15.42:0/4044374692 conn(0x7f65ec09f9c0 msgr2=0x7f65ec11b320 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:19:18.280 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2019-12-05T06:19:20.281 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:19:20.387 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:20.374+0000 7f65712b0700 1 Processor -- start 2019-12-05T06:19:20.387 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:20.374+0000 7f65712b0700 1 -- start start 2019-12-05T06:19:20.387 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:20.378+0000 7f65712b0700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f656c0a7740 0x7f656c08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:19:20.388 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:20.378+0000 7f65712b0700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f656c0a0710 con 0x7f656c07e720 2019-12-05T06:19:20.388 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:20.378+0000 7f65712b0700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f656c0a9ba0 con 0x7f656c09fae0 2019-12-05T06:19:20.388 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:20.378+0000 7f65712b0700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f656c0a0b00 con 0x7f656c0a7740 2019-12-05T06:19:20.388 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:20.378+0000 7f656bfff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f656c09fae0 0x7f656c14efe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:52886/0 (socket says 172.21.15.42:52886) 2019-12-05T06:19:20.389 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:20.378+0000 7f656bfff700 1 -- 172.21.15.42:0/2898953339 learned_addr learned my addr 172.21.15.42:0/2898953339 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:19:20.389 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:20.378+0000 7f656b7fe700 1 --2- 172.21.15.42:0/2898953339 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f656c0a7740 0x7f656c08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:20.389 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:20.378+0000 7f656affd700 1 -- 172.21.15.42:0/2898953339 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3028399387 0 0) 0x7f656c0a9ba0 con 0x7f656c09fae0 2019-12-05T06:19:20.389 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:20.378+0000 7f656affd700 1 -- 172.21.15.42:0/2898953339 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65580030b0 con 0x7f656c09fae0 2019-12-05T06:19:20.390 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:20.378+0000 7f656affd700 1 -- 172.21.15.42:0/2898953339 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1140334954 0 0) 0x7f656c0a0710 con 0x7f656c07e720 2019-12-05T06:19:20.390 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:20.378+0000 7f656affd700 1 -- 172.21.15.42:0/2898953339 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f656c0a9ba0 con 0x7f656c07e720 2019-12-05T06:19:20.390 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:20.378+0000 7f656b7fe700 1 -- 172.21.15.42:0/2898953339 >> v1:172.21.15.42:6789/0 conn(0x7f656c09fae0 legacy=0x7f656c14efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:20.390 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:20.378+0000 7f656affd700 1 -- 172.21.15.42:0/2898953339 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 783509573 0 0) 0x7f65580030b0 con 0x7f656c09fae0 2019-12-05T06:19:20.391 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:20.378+0000 7f656b7fe700 1 -- 172.21.15.42:0/2898953339 >> v1:172.21.15.40:6789/0 conn(0x7f656c07e720 legacy=0x7f656c09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:20.391 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:20.378+0000 7f656b7fe700 1 -- 172.21.15.42:0/2898953339 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f656c07eb50 con 0x7f656c0a7740 2019-12-05T06:19:20.391 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:20.378+0000 7f656b7fe700 1 --2- 172.21.15.42:0/2898953339 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f656c0a7740 0x7f656c08df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f65540019b0 tx=0x7f6554001480).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:20.393 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:20.378+0000 7f656b7fe700 1 --2- 172.21.15.42:0/2898953339 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f656c0a7740 0x7f656c08df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f65540019b0 tx=0x7f6554001480).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:20.394 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:20.378+0000 7f656b7fe700 1 --2- 172.21.15.42:0/2898953339 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f656c0a7740 0x7f656c08df80 secure :-1 s=READY pgs=535 cs=0 l=1 rx=0x7f65540019b0 tx=0x7f6554001480).ready entity=mon.2 client_cookie=be1dd4ddb88ff260 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:19:20.394 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:20.378+0000 7f656b7fe700 1 --2- 172.21.15.42:0/2898953339 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f656c0a7740 0x7f656c08df80 secure :-1 s=THROTTLE_DONE pgs=535 cs=0 l=1 rx=0x7f65540019b0 tx=0x7f6554001480).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:20.394 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:20.378+0000 7f656affd700 1 -- 172.21.15.42:0/2898953339 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6554001730 con 0x7f656c0a7740 2019-12-05T06:19:20.394 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:20.378+0000 7f656b7fe700 1 --2- 172.21.15.42:0/2898953339 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f656c0a7740 0x7f656c08df80 secure :-1 s=THROTTLE_DONE pgs=535 cs=0 l=1 rx=0x7f65540019b0 tx=0x7f6554001480).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:20.395 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:20.378+0000 7f656affd700 1 -- 172.21.15.42:0/2898953339 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6554001540 con 0x7f656c0a7740 2019-12-05T06:19:20.395 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:20.378+0000 7f656b7fe700 1 --2- 172.21.15.42:0/2898953339 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f656c0a7740 0x7f656c08df80 secure :-1 s=THROTTLE_DONE pgs=535 cs=0 l=1 rx=0x7f65540019b0 tx=0x7f6554001480).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:20.395 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:20.378+0000 7f656affd700 1 -- 172.21.15.42:0/2898953339 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6564001ad0 con 0x7f656c0a7740 2019-12-05T06:19:20.395 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:20.378+0000 7f65712b0700 1 -- 172.21.15.42:0/2898953339 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f656c0a7740 msgr2=0x7f656c08df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:20.396 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:20.378+0000 7f65712b0700 1 --2- 172.21.15.42:0/2898953339 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f656c0a7740 0x7f656c08df80 secure :-1 s=READY pgs=535 cs=0 l=1 rx=0x7f65540019b0 tx=0x7f6554001480).stop 2019-12-05T06:19:20.396 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:20.382+0000 7f65712b0700 1 -- 172.21.15.42:0/2898953339 shutdown_connections 2019-12-05T06:19:20.396 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:20.382+0000 7f65712b0700 1 --2- 172.21.15.42:0/2898953339 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f656c0a7740 0x7f656c08df80 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:20.396 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:20.382+0000 7f65712b0700 1 -- 172.21.15.42:0/2898953339 shutdown_connections 2019-12-05T06:19:20.397 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:20.382+0000 7f65712b0700 1 -- 172.21.15.42:0/2898953339 wait complete. 2019-12-05T06:19:20.397 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:20.382+0000 7f65712b0700 1 -- 172.21.15.42:0/2898953339 >> 172.21.15.42:0/2898953339 conn(0x7f656c0aabe0 msgr2=0x7f656c0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:19:20.397 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:20.382+0000 7f65712b0700 1 Processor -- start 2019-12-05T06:19:20.398 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:20.382+0000 7f65712b0700 1 -- start start 2019-12-05T06:19:20.404 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:20.386+0000 7f65712b0700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f656c143290 0x7f656c07c960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:19:20.404 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:20.386+0000 7f65712b0700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f656c12ce60 0x7f656c07ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:19:20.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:20.386+0000 7f65712b0700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f656c143810 con 0x7f656c142af0 2019-12-05T06:19:20.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:20.386+0000 7f65712b0700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f656c05ee90 con 0x7f656c12ce60 2019-12-05T06:19:20.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:20.386+0000 7f65712b0700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f656c142530 con 0x7f656c143290 2019-12-05T06:19:20.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:20.386+0000 7f656b7fe700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f656c142af0 0x7f656c05f130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:52888/0 (socket says 172.21.15.42:52888) 2019-12-05T06:19:20.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:20.386+0000 7f656b7fe700 1 -- 172.21.15.42:0/164160143 learned_addr learned my addr 172.21.15.42:0/164160143 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:19:20.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:20.386+0000 7f6562ffd700 1 --2- 172.21.15.42:0/164160143 >> v2:172.21.15.42:3300/0 conn(0x7f656c143290 0x7f656c07c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:20.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:20.386+0000 7f656bfff700 1 --2- 172.21.15.42:0/164160143 >> v2:172.21.15.40:3300/0 conn(0x7f656c12ce60 0x7f656c07ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:20.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:20.386+0000 7f6568ff9700 1 -- 172.21.15.42:0/164160143 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 700223702 0 0) 0x7f656c143810 con 0x7f656c142af0 2019-12-05T06:19:20.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:20.386+0000 7f6568ff9700 1 -- 172.21.15.42:0/164160143 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65500030b0 con 0x7f656c142af0 2019-12-05T06:19:20.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:20.386+0000 7f6562ffd700 1 -- 172.21.15.42:0/164160143 >> v2:172.21.15.40:3300/0 conn(0x7f656c12ce60 msgr2=0x7f656c07ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:20.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:20.386+0000 7f6562ffd700 1 --2- 172.21.15.42:0/164160143 >> v2:172.21.15.40:3300/0 conn(0x7f656c12ce60 0x7f656c07ceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:20.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:20.386+0000 7f6562ffd700 1 -- 172.21.15.42:0/164160143 >> v1:172.21.15.42:6789/0 conn(0x7f656c142af0 legacy=0x7f656c05f130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:20.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:20.386+0000 7f6562ffd700 1 -- 172.21.15.42:0/164160143 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f656c144330 con 0x7f656c143290 2019-12-05T06:19:20.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:20.386+0000 7f6562ffd700 1 --2- 172.21.15.42:0/164160143 >> v2:172.21.15.42:3300/0 conn(0x7f656c143290 0x7f656c07c960 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f6554000b80 tx=0x7f65540108c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:20.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:20.386+0000 7f6562ffd700 1 --2- 172.21.15.42:0/164160143 >> v2:172.21.15.42:3300/0 conn(0x7f656c143290 0x7f656c07c960 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f6554000b80 tx=0x7f65540108c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:20.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:20.386+0000 7f6562ffd700 1 --2- 172.21.15.42:0/164160143 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f656c143290 0x7f656c07c960 secure :-1 s=READY pgs=2438 cs=0 l=1 rx=0x7f6554000b80 tx=0x7f65540108c0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:19:20.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:20.386+0000 7f6562ffd700 1 --2- 172.21.15.42:0/164160143 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f656c143290 0x7f656c07c960 secure :-1 s=THROTTLE_DONE pgs=2438 cs=0 l=1 rx=0x7f6554000b80 tx=0x7f65540108c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:20.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:20.386+0000 7f6562ffd700 1 --2- 172.21.15.42:0/164160143 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f656c143290 0x7f656c07c960 secure :-1 s=THROTTLE_DONE pgs=2438 cs=0 l=1 rx=0x7f6554000b80 tx=0x7f65540108c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:20.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:20.386+0000 7f6568ff9700 1 -- 172.21.15.42:0/164160143 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f65540116e0 con 0x7f656c143290 2019-12-05T06:19:20.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:20.386+0000 7f6562ffd700 1 --2- 172.21.15.42:0/164160143 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f656c143290 0x7f656c07c960 secure :-1 s=THROTTLE_DONE pgs=2438 cs=0 l=1 rx=0x7f6554000b80 tx=0x7f65540108c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:20.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:20.386+0000 7f6568ff9700 1 -- 172.21.15.42:0/164160143 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6554011300 con 0x7f656c143290 2019-12-05T06:19:20.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:20.386+0000 7f65712b0700 1 -- 172.21.15.42:0/164160143 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f656c11a040 con 0x7f656c143290 2019-12-05T06:19:20.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:20.386+0000 7f6568ff9700 1 -- 172.21.15.42:0/164160143 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6554011cc0 con 0x7f656c143290 2019-12-05T06:19:20.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:20.386+0000 7f65712b0700 1 -- 172.21.15.42:0/164160143 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f656c11a590 con 0x7f656c143290 2019-12-05T06:19:20.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:20.386+0000 7f6562ffd700 1 --2- 172.21.15.42:0/164160143 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f656c143290 0x7f656c07c960 secure :-1 s=THROTTLE_DONE pgs=2438 cs=0 l=1 rx=0x7f6554000b80 tx=0x7f65540108c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:20.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:20.386+0000 7f6568ff9700 1 -- 172.21.15.42:0/164160143 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f655400f2a0 con 0x7f656c143290 2019-12-05T06:19:20.411 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:20.386+0000 7f6562ffd700 1 --2- 172.21.15.42:0/164160143 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f656c143290 0x7f656c07c960 secure :-1 s=THROTTLE_DONE pgs=2438 cs=0 l=1 rx=0x7f6554000b80 tx=0x7f65540108c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:20.411 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:20.390+0000 7f6568ff9700 1 --2- 172.21.15.42:0/164160143 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f655003b280 0x7f655003d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:19:20.411 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:20.390+0000 7f6568ff9700 1 -- 172.21.15.42:0/164160143 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 10503+0+0 (secure 0 0 0) 0x7f655404ae50 con 0x7f656c143290 2019-12-05T06:19:20.411 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:20.390+0000 7f656b7fe700 1 --2- 172.21.15.42:0/164160143 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f655003b280 0x7f655003d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:20.411 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:20.390+0000 7f656b7fe700 1 --2- 172.21.15.42:0/164160143 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f655003b280 0x7f655003d790 crc :-1 s=READY pgs=1023 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:19:20.411 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:20.390+0000 7f65712b0700 1 -- 172.21.15.42:0/164160143 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f656c05d110 con 0x7f656c143290 2019-12-05T06:19:20.411 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:20.398+0000 7f6562ffd700 1 --2- 172.21.15.42:0/164160143 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f656c143290 0x7f656c07c960 secure :-1 s=THROTTLE_DONE pgs=2438 cs=0 l=1 rx=0x7f6554000b80 tx=0x7f65540108c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:20.412 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:20.398+0000 7f6568ff9700 1 -- 172.21.15.42:0/164160143 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f6554031ed0 con 0x7f656c143290 2019-12-05T06:19:20.582 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:20.578+0000 7f65712b0700 1 -- 172.21.15.42:0/164160143 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f656c05b8c0 con 0x7f656c143290 2019-12-05T06:19:20.583 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:20.578+0000 7f6562ffd700 1 --2- 172.21.15.42:0/164160143 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f656c143290 0x7f656c07c960 secure :-1 s=THROTTLE_DONE pgs=2438 cs=0 l=1 rx=0x7f6554000b80 tx=0x7f65540108c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:20.583 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:20.578+0000 7f6568ff9700 1 -- 172.21.15.42:0/164160143 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 292 v292) v1 ==== 95+0+1779 (secure 0 0 0) 0x7f6554032130 con 0x7f656c143290 2019-12-05T06:19:20.583 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 292 2019-12-05T06:19:20.584 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:19:20.584 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":292,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":292,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:19:15.110045+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":101,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10476},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10476":{"gid":10476,"name":"b","rank":0,"incarnation":289,"state":"up:active","state_seq":34,"addr":"172.21.15.40:6833/1473620421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1473620421},{"type":"v1","addr":"172.21.15.40:6833","nonce":1473620421}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:19:20.586 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:20.582+0000 7f65712b0700 1 -- 172.21.15.42:0/164160143 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f655003b280 msgr2=0x7f655003d790 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:20.586 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:20.582+0000 7f65712b0700 1 --2- 172.21.15.42:0/164160143 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f655003b280 0x7f655003d790 crc :-1 s=READY pgs=1023 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:20.586 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:20.582+0000 7f65712b0700 1 -- 172.21.15.42:0/164160143 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f656c143290 msgr2=0x7f656c07c960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:20.586 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:20.582+0000 7f65712b0700 1 --2- 172.21.15.42:0/164160143 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f656c143290 0x7f656c07c960 secure :-1 s=READY pgs=2438 cs=0 l=1 rx=0x7f6554000b80 tx=0x7f65540108c0).stop 2019-12-05T06:19:20.587 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:20.582+0000 7f65712b0700 1 -- 172.21.15.42:0/164160143 shutdown_connections 2019-12-05T06:19:20.587 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:20.582+0000 7f65712b0700 1 --2- 172.21.15.42:0/164160143 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f655003b280 0x7f655003d790 unknown :-1 s=CLOSED pgs=1023 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:20.587 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:20.582+0000 7f65712b0700 1 --2- 172.21.15.42:0/164160143 >> v2:172.21.15.40:3300/0 conn(0x7f656c12ce60 0x7f656c07ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:20.587 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:20.582+0000 7f65712b0700 1 --2- 172.21.15.42:0/164160143 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f656c143290 0x7f656c07c960 unknown :-1 s=CLOSED pgs=2438 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:20.587 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:20.582+0000 7f65712b0700 1 -- 172.21.15.42:0/164160143 shutdown_connections 2019-12-05T06:19:20.587 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:20.582+0000 7f65712b0700 1 -- 172.21.15.42:0/164160143 wait complete. 2019-12-05T06:19:20.588 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:20.582+0000 7f65712b0700 1 -- 172.21.15.42:0/164160143 >> 172.21.15.42:0/164160143 conn(0x7f656c09f1e0 msgr2=0x7f656c0a7f90 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:19:20.596 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2019-12-05T06:19:22.599 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:19:22.708 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:22.698+0000 7f12fd00c700 1 Processor -- start 2019-12-05T06:19:22.708 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:22.698+0000 7f12fd00c700 1 -- start start 2019-12-05T06:19:22.708 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:22.698+0000 7f12fd00c700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f12f80a7740 0x7f12f808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:19:22.708 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:22.702+0000 7f12fd00c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f12f807e720 0x7f12f807eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:19:22.708 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:22.702+0000 7f12fd00c700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12f80a0710 con 0x7f12f809f2e0 2019-12-05T06:19:22.709 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:22.702+0000 7f12fd00c700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12f80a0b00 con 0x7f12f807e720 2019-12-05T06:19:22.709 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:22.702+0000 7f12fd00c700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12f808e5b0 con 0x7f12f80a7740 2019-12-05T06:19:22.709 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:22.702+0000 7f12f6d9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f12f807e720 0x7f12f807eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:22.709 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:22.702+0000 7f12f6d9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f12f807e720 0x7f12f807eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:52866/0 (socket says 172.21.15.42:52866) 2019-12-05T06:19:22.709 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:22.702+0000 7f12f6d9d700 1 -- 172.21.15.42:0/4258017235 learned_addr learned my addr 172.21.15.42:0/4258017235 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:19:22.709 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:22.702+0000 7f12f759e700 1 --2- 172.21.15.42:0/4258017235 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f12f80a7740 0x7f12f808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:22.711 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:22.702+0000 7f12f6d9d700 1 -- 172.21.15.42:0/4258017235 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f12f80a7740 msgr2=0x7f12f808df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:19:22.711 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:22.702+0000 7f12f6d9d700 1 --2- 172.21.15.42:0/4258017235 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f12f80a7740 0x7f12f808df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:19:22.711 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:22.702+0000 7f12f6d9d700 1 -- 172.21.15.42:0/4258017235 >> v1:172.21.15.40:6789/0 conn(0x7f12f809f2e0 legacy=0x7f12f809f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:22.712 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:22.702+0000 7f12f6d9d700 1 -- 172.21.15.42:0/4258017235 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12f807f0a0 con 0x7f12f807e720 2019-12-05T06:19:22.713 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:22.702+0000 7f12f6d9d700 1 --2- 172.21.15.42:0/4258017235 >> v2:172.21.15.42:3300/0 conn(0x7f12f807e720 0x7f12f807eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f12e8008b10 tx=0x7f12e80100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:22.713 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:22.702+0000 7f12f6d9d700 1 --2- 172.21.15.42:0/4258017235 >> v2:172.21.15.42:3300/0 conn(0x7f12f807e720 0x7f12f807eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f12e8008b10 tx=0x7f12e80100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:22.713 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:22.702+0000 7f12f6d9d700 1 --2- 172.21.15.42:0/4258017235 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f12f807e720 0x7f12f807eb50 secure :-1 s=READY pgs=2439 cs=0 l=1 rx=0x7f12e8008b10 tx=0x7f12e80100b0).ready entity=mon.1 client_cookie=cafcc13d03f57356 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:19:22.713 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:22.702+0000 7f12f6d9d700 1 --2- 172.21.15.42:0/4258017235 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f12f807e720 0x7f12f807eb50 secure :-1 s=THROTTLE_DONE pgs=2439 cs=0 l=1 rx=0x7f12e8008b10 tx=0x7f12e80100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:22.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:22.702+0000 7f12f6d9d700 1 --2- 172.21.15.42:0/4258017235 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f12f807e720 0x7f12f807eb50 secure :-1 s=THROTTLE_DONE pgs=2439 cs=0 l=1 rx=0x7f12e8008b10 tx=0x7f12e80100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:22.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:22.702+0000 7f12f6d9d700 1 --2- 172.21.15.42:0/4258017235 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f12f807e720 0x7f12f807eb50 secure :-1 s=THROTTLE_DONE pgs=2439 cs=0 l=1 rx=0x7f12e8008b10 tx=0x7f12e80100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:22.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:22.702+0000 7f12f659c700 1 -- 172.21.15.42:0/4258017235 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f12e800a160 con 0x7f12f807e720 2019-12-05T06:19:22.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:22.702+0000 7f12f659c700 1 -- 172.21.15.42:0/4258017235 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12e8011a30 con 0x7f12f807e720 2019-12-05T06:19:22.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:22.702+0000 7f12f659c700 1 -- 172.21.15.42:0/4258017235 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f12e800a550 con 0x7f12f807e720 2019-12-05T06:19:22.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:22.702+0000 7f12fd00c700 1 -- 172.21.15.42:0/4258017235 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f12f807e720 msgr2=0x7f12f807eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:22.715 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:22.702+0000 7f12fd00c700 1 --2- 172.21.15.42:0/4258017235 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f12f807e720 0x7f12f807eb50 secure :-1 s=READY pgs=2439 cs=0 l=1 rx=0x7f12e8008b10 tx=0x7f12e80100b0).stop 2019-12-05T06:19:22.715 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:22.702+0000 7f12fd00c700 1 -- 172.21.15.42:0/4258017235 shutdown_connections 2019-12-05T06:19:22.715 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:22.702+0000 7f12fd00c700 1 --2- 172.21.15.42:0/4258017235 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f12f80a7740 0x7f12f808df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:19:22.715 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:22.702+0000 7f12fd00c700 1 --2- 172.21.15.42:0/4258017235 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f12f807e720 0x7f12f807eb50 unknown :-1 s=CLOSED pgs=2439 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:22.715 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:22.702+0000 7f12fd00c700 1 -- 172.21.15.42:0/4258017235 shutdown_connections 2019-12-05T06:19:22.715 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:22.702+0000 7f12fd00c700 1 -- 172.21.15.42:0/4258017235 wait complete. 2019-12-05T06:19:22.715 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:22.702+0000 7f12fd00c700 1 -- 172.21.15.42:0/4258017235 >> 172.21.15.42:0/4258017235 conn(0x7f12f80aabe0 msgr2=0x7f12f80a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:19:22.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:22.706+0000 7f12fd00c700 1 Processor -- start 2019-12-05T06:19:22.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:22.706+0000 7f12fd00c700 1 -- start start 2019-12-05T06:19:22.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:22.706+0000 7f12fd00c700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f12f8124510 0x7f12f8130c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:19:22.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:22.706+0000 7f12fd00c700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f12f8135930 0x7f12f8137d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:19:22.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:22.706+0000 7f12fd00c700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12f811f0d0 con 0x7f12f812cf30 2019-12-05T06:19:22.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:22.706+0000 7f12fd00c700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12f81304f0 con 0x7f12f8124510 2019-12-05T06:19:22.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:22.706+0000 7f12fd00c700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12f8130750 con 0x7f12f8135930 2019-12-05T06:19:22.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:22.706+0000 7f12f759e700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f12f812cf30 0x7f12f807d180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:52908/0 (socket says 172.21.15.42:52908) 2019-12-05T06:19:22.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:22.706+0000 7f12f759e700 1 -- 172.21.15.42:0/1068080097 learned_addr learned my addr 172.21.15.42:0/1068080097 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:19:22.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:22.706+0000 7f12f6d9d700 1 --2- 172.21.15.42:0/1068080097 >> v2:172.21.15.40:3300/0 conn(0x7f12f8124510 0x7f12f8130c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:22.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:22.706+0000 7f12fc80b700 1 --2- 172.21.15.42:0/1068080097 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f12f8135930 0x7f12f8137d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:22.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:22.706+0000 7f12effff700 1 -- 172.21.15.42:0/1068080097 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2218222401 0 0) 0x7f12f811f0d0 con 0x7f12f812cf30 2019-12-05T06:19:22.724 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:22.706+0000 7f12effff700 1 -- 172.21.15.42:0/1068080097 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12dc0030b0 con 0x7f12f812cf30 2019-12-05T06:19:22.724 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:22.706+0000 7f12f6d9d700 1 -- 172.21.15.42:0/1068080097 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f12f8135930 msgr2=0x7f12f8137d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:22.724 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:22.706+0000 7f12f6d9d700 1 --2- 172.21.15.42:0/1068080097 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f12f8135930 0x7f12f8137d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:22.725 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:22.706+0000 7f12f6d9d700 1 -- 172.21.15.42:0/1068080097 >> v1:172.21.15.42:6789/0 conn(0x7f12f812cf30 legacy=0x7f12f807d180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:22.725 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:22.706+0000 7f12effff700 1 -- 172.21.15.42:0/1068080097 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 554143819 0 0) 0x7f12dc0030b0 con 0x7f12f812cf30 2019-12-05T06:19:22.725 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:22.706+0000 7f12f6d9d700 1 -- 172.21.15.42:0/1068080097 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12f8124940 con 0x7f12f8124510 2019-12-05T06:19:22.726 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:22.706+0000 7f12fc80b700 1 --2- 172.21.15.42:0/1068080097 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f12f8135930 0x7f12f8137d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:19:22.726 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:22.706+0000 7f12f6d9d700 1 --2- 172.21.15.42:0/1068080097 >> v2:172.21.15.40:3300/0 conn(0x7f12f8124510 0x7f12f8130c90 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f12e00016d0 tx=0x7f12e0002f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:22.726 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:22.710+0000 7f12f6d9d700 1 --2- 172.21.15.42:0/1068080097 >> v2:172.21.15.40:3300/0 conn(0x7f12f8124510 0x7f12f8130c90 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f12e00016d0 tx=0x7f12e0002f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:22.726 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:22.710+0000 7f12f6d9d700 1 --2- 172.21.15.42:0/1068080097 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f12f8124510 0x7f12f8130c90 secure :-1 s=READY pgs=1852 cs=0 l=1 rx=0x7f12e00016d0 tx=0x7f12e0002f70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:19:22.726 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:22.710+0000 7f12f6d9d700 1 --2- 172.21.15.42:0/1068080097 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f12f8124510 0x7f12f8130c90 secure :-1 s=THROTTLE_DONE pgs=1852 cs=0 l=1 rx=0x7f12e00016d0 tx=0x7f12e0002f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:22.726 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:22.710+0000 7f12effff700 1 -- 172.21.15.42:0/1068080097 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f12e0001490 con 0x7f12f8124510 2019-12-05T06:19:22.726 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:22.710+0000 7f12f6d9d700 1 --2- 172.21.15.42:0/1068080097 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f12f8124510 0x7f12f8130c90 secure :-1 s=THROTTLE_DONE pgs=1852 cs=0 l=1 rx=0x7f12e00016d0 tx=0x7f12e0002f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:22.727 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:22.710+0000 7f12fd00c700 1 -- 172.21.15.42:0/1068080097 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12f81311e0 con 0x7f12f8124510 2019-12-05T06:19:22.727 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:22.710+0000 7f12effff700 1 -- 172.21.15.42:0/1068080097 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12e00012a0 con 0x7f12f8124510 2019-12-05T06:19:22.727 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:22.710+0000 7f12f6d9d700 1 --2- 172.21.15.42:0/1068080097 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f12f8124510 0x7f12f8130c90 secure :-1 s=THROTTLE_DONE pgs=1852 cs=0 l=1 rx=0x7f12e00016d0 tx=0x7f12e0002f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:22.727 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:22.710+0000 7f12effff700 1 -- 172.21.15.42:0/1068080097 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f12e80058c0 con 0x7f12f8124510 2019-12-05T06:19:22.727 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:22.710+0000 7f12fd00c700 1 -- 172.21.15.42:0/1068080097 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12f812dca0 con 0x7f12f8124510 2019-12-05T06:19:22.727 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:22.710+0000 7f12f6d9d700 1 --2- 172.21.15.42:0/1068080097 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f12f8124510 0x7f12f8130c90 secure :-1 s=THROTTLE_DONE pgs=1852 cs=0 l=1 rx=0x7f12e00016d0 tx=0x7f12e0002f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:22.728 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:22.710+0000 7f12effff700 1 -- 172.21.15.42:0/1068080097 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f12e8005b60 con 0x7f12f8124510 2019-12-05T06:19:22.728 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:22.710+0000 7f12f6d9d700 1 --2- 172.21.15.42:0/1068080097 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f12f8124510 0x7f12f8130c90 secure :-1 s=THROTTLE_DONE pgs=1852 cs=0 l=1 rx=0x7f12e00016d0 tx=0x7f12e0002f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:22.728 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:22.710+0000 7f12effff700 1 --2- 172.21.15.42:0/1068080097 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f12dc03b230 0x7f12dc03d740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:19:22.728 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:22.710+0000 7f12effff700 1 -- 172.21.15.42:0/1068080097 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 10503+0+0 (secure 0 0 0) 0x7f12e804b6e0 con 0x7f12f8124510 2019-12-05T06:19:22.728 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:22.710+0000 7f12f759e700 1 --2- 172.21.15.42:0/1068080097 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f12dc03b230 0x7f12dc03d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:22.728 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:22.710+0000 7f12f759e700 1 --2- 172.21.15.42:0/1068080097 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f12dc03b230 0x7f12dc03d740 crc :-1 s=READY pgs=1025 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:19:22.728 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:22.714+0000 7f12fd00c700 1 -- 172.21.15.42:0/1068080097 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12f805d110 con 0x7f12f8124510 2019-12-05T06:19:22.729 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:22.726+0000 7f12f6d9d700 1 --2- 172.21.15.42:0/1068080097 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f12f8124510 0x7f12f8130c90 secure :-1 s=THROTTLE_DONE pgs=1852 cs=0 l=1 rx=0x7f12e00016d0 tx=0x7f12e0002f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:22.729 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:22.726+0000 7f12effff700 1 -- 172.21.15.42:0/1068080097 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f12e8032740 con 0x7f12f8124510 2019-12-05T06:19:22.897 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:22.894+0000 7f12fd00c700 1 -- 172.21.15.42:0/1068080097 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f12f80a71b0 con 0x7f12f8124510 2019-12-05T06:19:22.898 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:22.894+0000 7f12f6d9d700 1 --2- 172.21.15.42:0/1068080097 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f12f8124510 0x7f12f8130c90 secure :-1 s=THROTTLE_DONE pgs=1852 cs=0 l=1 rx=0x7f12e00016d0 tx=0x7f12e0002f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:22.898 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:22.894+0000 7f12effff700 1 -- 172.21.15.42:0/1068080097 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 293 v293) v1 ==== 95+0+2117 (secure 0 0 0) 0x7f12e80329a0 con 0x7f12f8124510 2019-12-05T06:19:22.900 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 293 2019-12-05T06:19:22.902 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:19:22.902 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":293,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":10711,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/2638662429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2638662429},{"type":"v1","addr":"172.21.15.42:6835","nonce":2638662429}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":293}],"filesystems":[{"mdsmap":{"epoch":292,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:19:15.110045+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":101,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10476},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10476":{"gid":10476,"name":"b","rank":0,"incarnation":289,"state":"up:active","state_seq":34,"addr":"172.21.15.40:6833/1473620421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1473620421},{"type":"v1","addr":"172.21.15.40:6833","nonce":1473620421}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:19:22.904 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:22.898+0000 7f12fd00c700 1 -- 172.21.15.42:0/1068080097 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f12dc03b230 msgr2=0x7f12dc03d740 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:22.904 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:22.898+0000 7f12fd00c700 1 --2- 172.21.15.42:0/1068080097 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f12dc03b230 0x7f12dc03d740 crc :-1 s=READY pgs=1025 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:22.904 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:22.898+0000 7f12fd00c700 1 -- 172.21.15.42:0/1068080097 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f12f8124510 msgr2=0x7f12f8130c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:22.904 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:22.898+0000 7f12fd00c700 1 --2- 172.21.15.42:0/1068080097 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f12f8124510 0x7f12f8130c90 secure :-1 s=READY pgs=1852 cs=0 l=1 rx=0x7f12e00016d0 tx=0x7f12e0002f70).stop 2019-12-05T06:19:22.905 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:22.898+0000 7f12fd00c700 1 -- 172.21.15.42:0/1068080097 shutdown_connections 2019-12-05T06:19:22.905 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:22.902+0000 7f12fd00c700 1 --2- 172.21.15.42:0/1068080097 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f12dc03b230 0x7f12dc03d740 unknown :-1 s=CLOSED pgs=1025 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:22.905 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:22.902+0000 7f12fd00c700 1 --2- 172.21.15.42:0/1068080097 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f12f8135930 0x7f12f8137d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:22.905 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:22.902+0000 7f12fd00c700 1 --2- 172.21.15.42:0/1068080097 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f12f8124510 0x7f12f8130c90 unknown :-1 s=CLOSED pgs=1852 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:22.905 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:22.902+0000 7f12fd00c700 1 -- 172.21.15.42:0/1068080097 shutdown_connections 2019-12-05T06:19:22.905 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:22.902+0000 7f12fd00c700 1 -- 172.21.15.42:0/1068080097 wait complete. 2019-12-05T06:19:22.905 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:22.902+0000 7f12fd00c700 1 -- 172.21.15.42:0/1068080097 >> 172.21.15.42:0/1068080097 conn(0x7f12f80a9870 msgr2=0x7f12f811c610 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:19:22.917 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a reported in up:standby state 2019-12-05T06:19:22.917 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 7 secs before thrashing 2019-12-05T06:19:29.918 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:19:30.016 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.006+0000 7f317f76a700 1 Processor -- start 2019-12-05T06:19:30.016 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.006+0000 7f317f76a700 1 -- start start 2019-12-05T06:19:30.016 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.010+0000 7f317f76a700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f31780a7740 0x7f317808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:19:30.017 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.010+0000 7f317f76a700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f317809fae0 0x7f317807eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:19:30.017 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.010+0000 7f317f76a700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31780a0710 con 0x7f317807e720 2019-12-05T06:19:30.017 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.010+0000 7f317f76a700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31780a0b00 con 0x7f31780a7740 2019-12-05T06:19:30.017 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.010+0000 7f317f76a700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f317808e5b0 con 0x7f317809fae0 2019-12-05T06:19:30.018 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.010+0000 7f317d506700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f317807e720 0x7f317809f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:52952/0 (socket says 172.21.15.42:52952) 2019-12-05T06:19:30.018 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.010+0000 7f317d506700 1 -- 172.21.15.42:0/1241422012 learned_addr learned my addr 172.21.15.42:0/1241422012 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:19:30.018 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.010+0000 7f317dd07700 1 --2- 172.21.15.42:0/1241422012 >> v2:172.21.15.40:3300/0 conn(0x7f31780a7740 0x7f317808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:30.018 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.010+0000 7f317e508700 1 --2- 172.21.15.42:0/1241422012 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f317809fae0 0x7f317807eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:30.018 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.010+0000 7f317cd05700 1 -- 172.21.15.42:0/1241422012 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4119812482 0 0) 0x7f31780a0710 con 0x7f317807e720 2019-12-05T06:19:30.018 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.010+0000 7f317cd05700 1 -- 172.21.15.42:0/1241422012 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31600030b0 con 0x7f317807e720 2019-12-05T06:19:30.018 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.010+0000 7f317e508700 1 -- 172.21.15.42:0/1241422012 >> v2:172.21.15.40:3300/0 conn(0x7f31780a7740 msgr2=0x7f317808df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:19:30.019 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.010+0000 7f317e508700 1 --2- 172.21.15.42:0/1241422012 >> v2:172.21.15.40:3300/0 conn(0x7f31780a7740 0x7f317808df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:19:30.019 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.010+0000 7f317e508700 1 -- 172.21.15.42:0/1241422012 >> v1:172.21.15.42:6789/0 conn(0x7f317807e720 legacy=0x7f317809f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:30.019 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.010+0000 7f317e508700 1 -- 172.21.15.42:0/1241422012 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f317807f0a0 con 0x7f317809fae0 2019-12-05T06:19:30.019 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.010+0000 7f317e508700 1 --2- 172.21.15.42:0/1241422012 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f317809fae0 0x7f317807eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f3168008790 tx=0x7f316800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:30.019 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.010+0000 7f317e508700 1 --2- 172.21.15.42:0/1241422012 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f317809fae0 0x7f317807eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f3168008790 tx=0x7f316800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:30.019 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.010+0000 7f317e508700 1 --2- 172.21.15.42:0/1241422012 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f317809fae0 0x7f317807eb50 secure :-1 s=READY pgs=536 cs=0 l=1 rx=0x7f3168008790 tx=0x7f316800fc70).ready entity=mon.2 client_cookie=9be427d9d05a7e77 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:19:30.019 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.010+0000 7f317e508700 1 --2- 172.21.15.42:0/1241422012 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f317809fae0 0x7f317807eb50 secure :-1 s=THROTTLE_DONE pgs=536 cs=0 l=1 rx=0x7f3168008790 tx=0x7f316800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:30.020 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.010+0000 7f317cd05700 1 -- 172.21.15.42:0/1241422012 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3168008bb0 con 0x7f317809fae0 2019-12-05T06:19:30.020 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.010+0000 7f317e508700 1 --2- 172.21.15.42:0/1241422012 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f317809fae0 0x7f317807eb50 secure :-1 s=THROTTLE_DONE pgs=536 cs=0 l=1 rx=0x7f3168008790 tx=0x7f316800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:30.020 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.010+0000 7f317cd05700 1 -- 172.21.15.42:0/1241422012 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31680087d0 con 0x7f317809fae0 2019-12-05T06:19:30.020 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.010+0000 7f317e508700 1 --2- 172.21.15.42:0/1241422012 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f317809fae0 0x7f317807eb50 secure :-1 s=THROTTLE_DONE pgs=536 cs=0 l=1 rx=0x7f3168008790 tx=0x7f316800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:30.020 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.010+0000 7f317cd05700 1 -- 172.21.15.42:0/1241422012 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3168011860 con 0x7f317809fae0 2019-12-05T06:19:30.020 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.010+0000 7f317f76a700 1 -- 172.21.15.42:0/1241422012 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f317809fae0 msgr2=0x7f317807eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:30.020 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.010+0000 7f317f76a700 1 --2- 172.21.15.42:0/1241422012 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f317809fae0 0x7f317807eb50 secure :-1 s=READY pgs=536 cs=0 l=1 rx=0x7f3168008790 tx=0x7f316800fc70).stop 2019-12-05T06:19:30.021 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.010+0000 7f317f76a700 1 -- 172.21.15.42:0/1241422012 shutdown_connections 2019-12-05T06:19:30.021 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.010+0000 7f317f76a700 1 --2- 172.21.15.42:0/1241422012 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f317809fae0 0x7f317807eb50 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:30.021 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.010+0000 7f317f76a700 1 --2- 172.21.15.42:0/1241422012 >> v2:172.21.15.40:3300/0 conn(0x7f31780a7740 0x7f317808df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:19:30.021 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.010+0000 7f317f76a700 1 -- 172.21.15.42:0/1241422012 shutdown_connections 2019-12-05T06:19:30.021 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.010+0000 7f317f76a700 1 -- 172.21.15.42:0/1241422012 wait complete. 2019-12-05T06:19:30.021 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.010+0000 7f317f76a700 1 -- 172.21.15.42:0/1241422012 >> 172.21.15.42:0/1241422012 conn(0x7f31780aabe0 msgr2=0x7f31780a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:19:30.027 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.014+0000 7f317f76a700 1 Processor -- start 2019-12-05T06:19:30.027 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.014+0000 7f317f76a700 1 -- start start 2019-12-05T06:19:30.028 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.014+0000 7f317f76a700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f31780a66a0 0x7f31781388a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:19:30.028 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.014+0000 7f317f76a700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31781230b0 con 0x7f317811bab0 2019-12-05T06:19:30.028 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.014+0000 7f317f76a700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3178130c40 con 0x7f31780a6270 2019-12-05T06:19:30.028 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.014+0000 7f317f76a700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3178122640 con 0x7f31780a66a0 2019-12-05T06:19:30.028 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.014+0000 7f317d506700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f31780a6270 0x7f317807aa30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:52958/0 (socket says 172.21.15.42:52958) 2019-12-05T06:19:30.028 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.014+0000 7f317d506700 1 -- 172.21.15.42:0/2245425793 learned_addr learned my addr 172.21.15.42:0/2245425793 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:19:30.028 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.014+0000 7f317e508700 1 --2- 172.21.15.42:0/2245425793 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f31780a66a0 0x7f31781388a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:30.029 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.014+0000 7f31767fc700 1 -- 172.21.15.42:0/2245425793 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3902282306 0 0) 0x7f3178130c40 con 0x7f31780a6270 2019-12-05T06:19:30.029 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.014+0000 7f31767fc700 1 -- 172.21.15.42:0/2245425793 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31580030b0 con 0x7f31780a6270 2019-12-05T06:19:30.029 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.014+0000 7f31767fc700 1 -- 172.21.15.42:0/2245425793 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 445670165 0 0) 0x7f31781230b0 con 0x7f317811bab0 2019-12-05T06:19:30.029 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.014+0000 7f31767fc700 1 -- 172.21.15.42:0/2245425793 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3178130c40 con 0x7f317811bab0 2019-12-05T06:19:30.029 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.014+0000 7f31767fc700 1 -- 172.21.15.42:0/2245425793 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 2566015852 0 0) 0x7f31580030b0 con 0x7f31780a6270 2019-12-05T06:19:30.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.014+0000 7f31767fc700 1 -- 172.21.15.42:0/2245425793 --> v1:172.21.15.42:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f31781230b0 con 0x7f31780a6270 2019-12-05T06:19:30.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.014+0000 7f31767fc700 1 -- 172.21.15.42:0/2245425793 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7f316c001990 con 0x7f31780a6270 2019-12-05T06:19:30.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.014+0000 7f31767fc700 1 -- 172.21.15.42:0/2245425793 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f31780a66a0 msgr2=0x7f31781388a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:30.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.014+0000 7f31767fc700 1 --2- 172.21.15.42:0/2245425793 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f31780a66a0 0x7f31781388a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:30.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.014+0000 7f31767fc700 1 -- 172.21.15.42:0/2245425793 >> v1:172.21.15.42:6789/0 conn(0x7f31780a6270 legacy=0x7f317807aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:30.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.014+0000 7f31767fc700 1 -- 172.21.15.42:0/2245425793 >> v1:172.21.15.40:6789/0 conn(0x7f317811bab0 legacy=0x7f3178130440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:30.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.014+0000 7f31767fc700 1 --2- 172.21.15.42:0/2245425793 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3158003ab0 0x7f3158004420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:19:30.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.014+0000 7f31767fc700 1 --2- 172.21.15.42:0/2245425793 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3158004970 0x7f3158004da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:19:30.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.014+0000 7f31767fc700 1 --2- 172.21.15.42:0/2245425793 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3158009660 0x7f315800baa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:19:30.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.014+0000 7f31767fc700 1 -- 172.21.15.42:0/2245425793 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3178122640 con 0x7f3158003ab0 2019-12-05T06:19:30.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.014+0000 7f31767fc700 1 -- 172.21.15.42:0/2245425793 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31580040e0 con 0x7f3158009660 2019-12-05T06:19:30.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.014+0000 7f31767fc700 1 -- 172.21.15.42:0/2245425793 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f315800bff0 con 0x7f3158004970 2019-12-05T06:19:30.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.014+0000 7f317e508700 1 --2- 172.21.15.42:0/2245425793 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3158009660 0x7f315800baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:30.032 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.014+0000 7f317dd07700 1 --2- 172.21.15.42:0/2245425793 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3158003ab0 0x7f3158004420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:30.032 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.014+0000 7f317d506700 1 --2- 172.21.15.42:0/2245425793 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3158004970 0x7f3158004da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:30.032 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.014+0000 7f317e508700 1 -- 172.21.15.42:0/2245425793 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3158004970 msgr2=0x7f3158004da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:30.032 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.014+0000 7f317e508700 1 --2- 172.21.15.42:0/2245425793 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3158004970 0x7f3158004da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:30.032 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.014+0000 7f317e508700 1 -- 172.21.15.42:0/2245425793 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3158003ab0 msgr2=0x7f3158004420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:30.032 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.014+0000 7f317e508700 1 --2- 172.21.15.42:0/2245425793 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3158003ab0 0x7f3158004420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:30.032 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.014+0000 7f317e508700 1 -- 172.21.15.42:0/2245425793 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31780a6ad0 con 0x7f3158009660 2019-12-05T06:19:30.033 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.014+0000 7f317f76a700 1 -- 172.21.15.42:0/2245425793 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f317807b230 con 0x7f3158009660 2019-12-05T06:19:30.033 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.014+0000 7f317e508700 1 --2- 172.21.15.42:0/2245425793 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3158009660 0x7f315800baa0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f3168008790 tx=0x7f3168008b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:30.033 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.014+0000 7f317d506700 1 --2- 172.21.15.42:0/2245425793 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3158004970 0x7f3158004da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:19:30.033 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.014+0000 7f317f76a700 1 -- 172.21.15.42:0/2245425793 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31780a66a0 con 0x7f3158009660 2019-12-05T06:19:30.033 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.018+0000 7f317e508700 1 --2- 172.21.15.42:0/2245425793 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3158009660 0x7f315800baa0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f3168008790 tx=0x7f3168008b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:30.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.018+0000 7f317e508700 1 --2- 172.21.15.42:0/2245425793 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3158009660 0x7f315800baa0 secure :-1 s=READY pgs=2443 cs=0 l=1 rx=0x7f3168008790 tx=0x7f3168008b00).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:19:30.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.018+0000 7f317e508700 1 --2- 172.21.15.42:0/2245425793 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3158009660 0x7f315800baa0 secure :-1 s=THROTTLE_DONE pgs=2443 cs=0 l=1 rx=0x7f3168008790 tx=0x7f3168008b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:30.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.018+0000 7f317e508700 1 --2- 172.21.15.42:0/2245425793 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3158009660 0x7f315800baa0 secure :-1 s=THROTTLE_DONE pgs=2443 cs=0 l=1 rx=0x7f3168008790 tx=0x7f3168008b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:30.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.018+0000 7f31767fc700 1 -- 172.21.15.42:0/2245425793 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3168005540 con 0x7f3158009660 2019-12-05T06:19:30.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.018+0000 7f317e508700 1 --2- 172.21.15.42:0/2245425793 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3158009660 0x7f315800baa0 secure :-1 s=THROTTLE_DONE pgs=2443 cs=0 l=1 rx=0x7f3168008790 tx=0x7f3168008b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:30.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.018+0000 7f31767fc700 1 -- 172.21.15.42:0/2245425793 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3168008db0 con 0x7f3158009660 2019-12-05T06:19:30.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.018+0000 7f31767fc700 1 -- 172.21.15.42:0/2245425793 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3168005b20 con 0x7f3158009660 2019-12-05T06:19:30.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.018+0000 7f317e508700 1 --2- 172.21.15.42:0/2245425793 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3158009660 0x7f315800baa0 secure :-1 s=THROTTLE_DONE pgs=2443 cs=0 l=1 rx=0x7f3168008790 tx=0x7f3168008b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:30.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.018+0000 7f31767fc700 1 -- 172.21.15.42:0/2245425793 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f3168013400 con 0x7f3158009660 2019-12-05T06:19:30.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.018+0000 7f317e508700 1 --2- 172.21.15.42:0/2245425793 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3158009660 0x7f315800baa0 secure :-1 s=THROTTLE_DONE pgs=2443 cs=0 l=1 rx=0x7f3168008790 tx=0x7f3168008b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:30.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.018+0000 7f31767fc700 1 --2- 172.21.15.42:0/2245425793 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3158043db0 0x7f31580462c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:19:30.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.018+0000 7f31767fc700 1 -- 172.21.15.42:0/2245425793 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 10503+0+0 (secure 0 0 0) 0x7f316804b540 con 0x7f3158009660 2019-12-05T06:19:30.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.018+0000 7f317dd07700 1 --2- 172.21.15.42:0/2245425793 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3158043db0 0x7f31580462c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:30.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.018+0000 7f317dd07700 1 --2- 172.21.15.42:0/2245425793 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3158043db0 0x7f31580462c0 crc :-1 s=READY pgs=1026 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:19:30.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.022+0000 7f317f76a700 1 -- 172.21.15.42:0/2245425793 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3154003200 con 0x7f3158009660 2019-12-05T06:19:30.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.030+0000 7f317e508700 1 --2- 172.21.15.42:0/2245425793 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3158009660 0x7f315800baa0 secure :-1 s=THROTTLE_DONE pgs=2443 cs=0 l=1 rx=0x7f3168008790 tx=0x7f3168008b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:30.039 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.030+0000 7f31767fc700 1 -- 172.21.15.42:0/2245425793 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f3168013d50 con 0x7f3158009660 2019-12-05T06:19:30.195 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.190+0000 7f317f76a700 1 -- 172.21.15.42:0/2245425793 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3154003630 con 0x7f3158009660 2019-12-05T06:19:30.197 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.194+0000 7f317e508700 1 --2- 172.21.15.42:0/2245425793 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3158009660 0x7f315800baa0 secure :-1 s=THROTTLE_DONE pgs=2443 cs=0 l=1 rx=0x7f3168008790 tx=0x7f3168008b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:30.197 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.194+0000 7f31767fc700 1 -- 172.21.15.42:0/2245425793 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 293 v293) v1 ==== 95+0+2117 (secure 0 0 0) 0x7f3168033100 con 0x7f3158009660 2019-12-05T06:19:30.197 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 293 2019-12-05T06:19:30.197 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:19:30.198 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":293,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":10711,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/2638662429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2638662429},{"type":"v1","addr":"172.21.15.42:6835","nonce":2638662429}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":293}],"filesystems":[{"mdsmap":{"epoch":292,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:19:15.110045+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":101,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10476},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10476":{"gid":10476,"name":"b","rank":0,"incarnation":289,"state":"up:active","state_seq":34,"addr":"172.21.15.40:6833/1473620421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1473620421},{"type":"v1","addr":"172.21.15.40:6833","nonce":1473620421}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:19:30.200 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.194+0000 7f317f76a700 1 -- 172.21.15.42:0/2245425793 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3158043db0 msgr2=0x7f31580462c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:30.200 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.194+0000 7f317f76a700 1 --2- 172.21.15.42:0/2245425793 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3158043db0 0x7f31580462c0 crc :-1 s=READY pgs=1026 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:30.200 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.194+0000 7f317f76a700 1 -- 172.21.15.42:0/2245425793 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3158009660 msgr2=0x7f315800baa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:30.200 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.194+0000 7f317f76a700 1 --2- 172.21.15.42:0/2245425793 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3158009660 0x7f315800baa0 secure :-1 s=READY pgs=2443 cs=0 l=1 rx=0x7f3168008790 tx=0x7f3168008b00).stop 2019-12-05T06:19:30.200 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.194+0000 7f317e508700 1 -- 172.21.15.42:0/2245425793 reap_dead start 2019-12-05T06:19:30.201 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.194+0000 7f317e508700 1 -- 172.21.15.42:0/2245425793 reap_dead start 2019-12-05T06:19:30.201 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.194+0000 7f317f76a700 1 -- 172.21.15.42:0/2245425793 shutdown_connections 2019-12-05T06:19:30.201 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.194+0000 7f317f76a700 1 -- 172.21.15.42:0/2245425793 shutdown_connections 2019-12-05T06:19:30.201 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.194+0000 7f317f76a700 1 -- 172.21.15.42:0/2245425793 wait complete. 2019-12-05T06:19:30.201 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.194+0000 7f317f76a700 1 -- 172.21.15.42:0/2245425793 >> 172.21.15.42:0/2245425793 conn(0x7f317811d340 msgr2=0x7f3178059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:19:30.208 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.b (rank=0) 2019-12-05T06:19:30.208 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2019-12-05T06:19:30.209 INFO:teuthology.orchestra.run:waiting for 300 2019-12-05T06:19:30.223 INFO:tasks.ceph.mds.b:Stopped 2019-12-05T06:19:30.223 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:19:30.326 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.314+0000 7ff565ae7700 1 Processor -- start 2019-12-05T06:19:30.327 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.314+0000 7ff565ae7700 1 -- start start 2019-12-05T06:19:30.327 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.318+0000 7ff565ae7700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff5600a7740 0x7ff56008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:19:30.327 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.318+0000 7ff565ae7700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff56009fae0 0x7ff56007eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:19:30.328 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.318+0000 7ff565ae7700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5600a0710 con 0x7ff56007e720 2019-12-05T06:19:30.328 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.318+0000 7ff565ae7700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5600a0b00 con 0x7ff5600a7740 2019-12-05T06:19:30.328 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.318+0000 7ff565ae7700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff56008e5b0 con 0x7ff56009fae0 2019-12-05T06:19:30.328 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.318+0000 7ff55ffff700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff5600a7740 0x7ff56008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:30.328 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.318+0000 7ff55f7fe700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7ff56007e720 0x7ff56009f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:52974/0 (socket says 172.21.15.42:52974) 2019-12-05T06:19:30.328 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.318+0000 7ff55f7fe700 1 -- 172.21.15.42:0/1062537695 learned_addr learned my addr 172.21.15.42:0/1062537695 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:19:30.329 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.318+0000 7ff564885700 1 --2- 172.21.15.42:0/1062537695 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff56009fae0 0x7ff56007eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:30.329 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.318+0000 7ff55effd700 1 -- 172.21.15.42:0/1062537695 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2692486361 0 0) 0x7ff5600a0710 con 0x7ff56007e720 2019-12-05T06:19:30.329 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.318+0000 7ff55effd700 1 -- 172.21.15.42:0/1062537695 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5480030b0 con 0x7ff56007e720 2019-12-05T06:19:30.329 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.318+0000 7ff55ffff700 1 -- 172.21.15.42:0/1062537695 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff56009fae0 msgr2=0x7ff56007eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:19:30.329 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.318+0000 7ff55ffff700 1 --2- 172.21.15.42:0/1062537695 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff56009fae0 0x7ff56007eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:19:30.329 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.318+0000 7ff55ffff700 1 -- 172.21.15.42:0/1062537695 >> v1:172.21.15.42:6789/0 conn(0x7ff56007e720 legacy=0x7ff56009f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:30.329 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.318+0000 7ff55ffff700 1 -- 172.21.15.42:0/1062537695 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff56007f0a0 con 0x7ff5600a7740 2019-12-05T06:19:30.330 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.318+0000 7ff564885700 1 --2- 172.21.15.42:0/1062537695 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff56009fae0 0x7ff56007eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:19:30.330 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.318+0000 7ff55ffff700 1 --2- 172.21.15.42:0/1062537695 >> v2:172.21.15.40:3300/0 conn(0x7ff5600a7740 0x7ff56008df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7ff550008790 tx=0x7ff55000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:30.330 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.318+0000 7ff55ffff700 1 --2- 172.21.15.42:0/1062537695 >> v2:172.21.15.40:3300/0 conn(0x7ff5600a7740 0x7ff56008df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7ff550008790 tx=0x7ff55000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:30.330 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.318+0000 7ff55ffff700 1 --2- 172.21.15.42:0/1062537695 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff5600a7740 0x7ff56008df80 secure :-1 s=READY pgs=1854 cs=0 l=1 rx=0x7ff550008790 tx=0x7ff55000fc70).ready entity=mon.0 client_cookie=3e298be43a99209d server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:19:30.330 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.318+0000 7ff55ffff700 1 --2- 172.21.15.42:0/1062537695 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff5600a7740 0x7ff56008df80 secure :-1 s=THROTTLE_DONE pgs=1854 cs=0 l=1 rx=0x7ff550008790 tx=0x7ff55000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:30.330 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.318+0000 7ff55effd700 1 -- 172.21.15.42:0/1062537695 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff550008bb0 con 0x7ff5600a7740 2019-12-05T06:19:30.330 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.318+0000 7ff55ffff700 1 --2- 172.21.15.42:0/1062537695 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff5600a7740 0x7ff56008df80 secure :-1 s=THROTTLE_DONE pgs=1854 cs=0 l=1 rx=0x7ff550008790 tx=0x7ff55000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:30.332 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.318+0000 7ff55ffff700 1 --2- 172.21.15.42:0/1062537695 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff5600a7740 0x7ff56008df80 secure :-1 s=THROTTLE_DONE pgs=1854 cs=0 l=1 rx=0x7ff550008790 tx=0x7ff55000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:30.332 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.318+0000 7ff55effd700 1 -- 172.21.15.42:0/1062537695 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5500087d0 con 0x7ff5600a7740 2019-12-05T06:19:30.333 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.318+0000 7ff55effd700 1 -- 172.21.15.42:0/1062537695 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff550011860 con 0x7ff5600a7740 2019-12-05T06:19:30.333 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.318+0000 7ff565ae7700 1 -- 172.21.15.42:0/1062537695 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff5600a7740 msgr2=0x7ff56008df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:30.334 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.318+0000 7ff565ae7700 1 --2- 172.21.15.42:0/1062537695 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff5600a7740 0x7ff56008df80 secure :-1 s=READY pgs=1854 cs=0 l=1 rx=0x7ff550008790 tx=0x7ff55000fc70).stop 2019-12-05T06:19:30.334 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.322+0000 7ff565ae7700 1 -- 172.21.15.42:0/1062537695 shutdown_connections 2019-12-05T06:19:30.334 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.322+0000 7ff565ae7700 1 --2- 172.21.15.42:0/1062537695 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff56009fae0 0x7ff56007eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:19:30.334 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.322+0000 7ff565ae7700 1 --2- 172.21.15.42:0/1062537695 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff5600a7740 0x7ff56008df80 unknown :-1 s=CLOSED pgs=1854 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:30.334 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.322+0000 7ff565ae7700 1 -- 172.21.15.42:0/1062537695 shutdown_connections 2019-12-05T06:19:30.335 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.322+0000 7ff565ae7700 1 -- 172.21.15.42:0/1062537695 wait complete. 2019-12-05T06:19:30.335 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.322+0000 7ff565ae7700 1 -- 172.21.15.42:0/1062537695 >> 172.21.15.42:0/1062537695 conn(0x7ff5600aabe0 msgr2=0x7ff5600a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:19:30.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.322+0000 7ff565ae7700 1 Processor -- start 2019-12-05T06:19:30.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.322+0000 7ff565ae7700 1 -- start start 2019-12-05T06:19:30.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.326+0000 7ff565ae7700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff56008f090 0x7ff5600722c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:19:30.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.326+0000 7ff565ae7700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5601194b0 con 0x7ff56008f4c0 2019-12-05T06:19:30.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.326+0000 7ff565ae7700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff560141fa0 con 0x7ff5601188c0 2019-12-05T06:19:30.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.326+0000 7ff565ae7700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff560117710 con 0x7ff56008f090 2019-12-05T06:19:30.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.326+0000 7ff55f7fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff56008f090 0x7ff5600722c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:30.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.326+0000 7ff55f7fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff56008f090 0x7ff5600722c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:52942/0 (socket says 172.21.15.42:52942) 2019-12-05T06:19:30.341 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.326+0000 7ff55ffff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7ff5601188c0 0x7ff5601161c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:52978/0 (socket says 172.21.15.42:52978) 2019-12-05T06:19:30.341 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.326+0000 7ff55f7fe700 1 -- 172.21.15.42:0/1233503982 learned_addr learned my addr 172.21.15.42:0/1233503982 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:19:30.341 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.326+0000 7ff55f7fe700 1 -- 172.21.15.42:0/1233503982 >> v1:172.21.15.42:6789/0 conn(0x7ff5601188c0 legacy=0x7ff5601161c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:30.341 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.326+0000 7ff55cff9700 1 -- 172.21.15.42:0/1233503982 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 215234082 0 0) 0x7ff560141fa0 con 0x7ff5601188c0 2019-12-05T06:19:30.341 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.326+0000 7ff55f7fe700 1 -- 172.21.15.42:0/1233503982 >> v1:172.21.15.40:6789/0 conn(0x7ff56008f4c0 legacy=0x7ff560074820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:30.344 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.326+0000 7ff55f7fe700 1 -- 172.21.15.42:0/1233503982 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff560118cf0 con 0x7ff56008f090 2019-12-05T06:19:30.344 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.326+0000 7ff55f7fe700 1 --2- 172.21.15.42:0/1233503982 >> v2:172.21.15.42:3300/0 conn(0x7ff56008f090 0x7ff5600722c0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7ff56007ee80 tx=0x7ff55400c430).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:30.345 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.326+0000 7ff55f7fe700 1 --2- 172.21.15.42:0/1233503982 >> v2:172.21.15.42:3300/0 conn(0x7ff56008f090 0x7ff5600722c0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7ff56007ee80 tx=0x7ff55400c430).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:30.345 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.326+0000 7ff55f7fe700 1 --2- 172.21.15.42:0/1233503982 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff56008f090 0x7ff5600722c0 secure :-1 s=READY pgs=2446 cs=0 l=1 rx=0x7ff56007ee80 tx=0x7ff55400c430).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:19:30.345 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.326+0000 7ff55f7fe700 1 --2- 172.21.15.42:0/1233503982 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff56008f090 0x7ff5600722c0 secure :-1 s=THROTTLE_DONE pgs=2446 cs=0 l=1 rx=0x7ff56007ee80 tx=0x7ff55400c430).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:30.346 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.326+0000 7ff55cff9700 1 -- 172.21.15.42:0/1233503982 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff55400ca90 con 0x7ff56008f090 2019-12-05T06:19:30.346 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.326+0000 7ff55f7fe700 1 --2- 172.21.15.42:0/1233503982 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff56008f090 0x7ff5600722c0 secure :-1 s=THROTTLE_DONE pgs=2446 cs=0 l=1 rx=0x7ff56007ee80 tx=0x7ff55400c430).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:30.346 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.326+0000 7ff55f7fe700 1 --2- 172.21.15.42:0/1233503982 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff56008f090 0x7ff5600722c0 secure :-1 s=THROTTLE_DONE pgs=2446 cs=0 l=1 rx=0x7ff56007ee80 tx=0x7ff55400c430).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:30.346 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.326+0000 7ff55cff9700 1 -- 172.21.15.42:0/1233503982 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff554010250 con 0x7ff56008f090 2019-12-05T06:19:30.347 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.326+0000 7ff565ae7700 1 -- 172.21.15.42:0/1233503982 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff560119e20 con 0x7ff56008f090 2019-12-05T06:19:30.347 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.326+0000 7ff55cff9700 1 -- 172.21.15.42:0/1233503982 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff55400d2b0 con 0x7ff56008f090 2019-12-05T06:19:30.347 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.326+0000 7ff565ae7700 1 -- 172.21.15.42:0/1233503982 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff560117010 con 0x7ff56008f090 2019-12-05T06:19:30.347 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.326+0000 7ff55f7fe700 1 --2- 172.21.15.42:0/1233503982 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff56008f090 0x7ff5600722c0 secure :-1 s=THROTTLE_DONE pgs=2446 cs=0 l=1 rx=0x7ff56007ee80 tx=0x7ff55400c430).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:30.347 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.330+0000 7ff55cff9700 1 -- 172.21.15.42:0/1233503982 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7ff55400d610 con 0x7ff56008f090 2019-12-05T06:19:30.347 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.330+0000 7ff55f7fe700 1 --2- 172.21.15.42:0/1233503982 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff56008f090 0x7ff5600722c0 secure :-1 s=THROTTLE_DONE pgs=2446 cs=0 l=1 rx=0x7ff56007ee80 tx=0x7ff55400c430).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:30.347 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.330+0000 7ff55cff9700 1 --2- 172.21.15.42:0/1233503982 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff54003ac80 0x7ff54003d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:19:30.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.330+0000 7ff55cff9700 1 -- 172.21.15.42:0/1233503982 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 10503+0+0 (secure 0 0 0) 0x7ff55404ab50 con 0x7ff56008f090 2019-12-05T06:19:30.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.330+0000 7ff55ffff700 1 --2- 172.21.15.42:0/1233503982 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff54003ac80 0x7ff54003d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:30.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.330+0000 7ff55ffff700 1 --2- 172.21.15.42:0/1233503982 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff54003ac80 0x7ff54003d190 crc :-1 s=READY pgs=1027 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:19:30.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.330+0000 7ff565ae7700 1 -- 172.21.15.42:0/1233503982 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff56005d110 con 0x7ff56008f090 2019-12-05T06:19:30.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.338+0000 7ff55f7fe700 1 --2- 172.21.15.42:0/1233503982 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff56008f090 0x7ff5600722c0 secure :-1 s=THROTTLE_DONE pgs=2446 cs=0 l=1 rx=0x7ff56007ee80 tx=0x7ff55400c430).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:30.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.338+0000 7ff55cff9700 1 -- 172.21.15.42:0/1233503982 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7ff554031b90 con 0x7ff56008f090 2019-12-05T06:19:30.511 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.506+0000 7ff565ae7700 1 -- 172.21.15.42:0/1233503982 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff5600a9ed0 con 0x7ff56008f090 2019-12-05T06:19:30.512 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.510+0000 7ff55f7fe700 1 --2- 172.21.15.42:0/1233503982 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff56008f090 0x7ff5600722c0 secure :-1 s=THROTTLE_DONE pgs=2446 cs=0 l=1 rx=0x7ff56007ee80 tx=0x7ff55400c430).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:30.512 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.510+0000 7ff55cff9700 1 -- 172.21.15.42:0/1233503982 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 293 v293) v1 ==== 95+0+2117 (secure 0 0 0) 0x7ff554031df0 con 0x7ff56008f090 2019-12-05T06:19:30.514 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 293 2019-12-05T06:19:30.514 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:19:30.514 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":293,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":10711,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/2638662429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2638662429},{"type":"v1","addr":"172.21.15.42:6835","nonce":2638662429}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":293}],"filesystems":[{"mdsmap":{"epoch":292,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:19:15.110045+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":101,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10476},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10476":{"gid":10476,"name":"b","rank":0,"incarnation":289,"state":"up:active","state_seq":34,"addr":"172.21.15.40:6833/1473620421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1473620421},{"type":"v1","addr":"172.21.15.40:6833","nonce":1473620421}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:19:30.518 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.514+0000 7ff565ae7700 1 -- 172.21.15.42:0/1233503982 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff54003ac80 msgr2=0x7ff54003d190 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:30.519 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.514+0000 7ff565ae7700 1 --2- 172.21.15.42:0/1233503982 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff54003ac80 0x7ff54003d190 crc :-1 s=READY pgs=1027 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:30.519 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.514+0000 7ff565ae7700 1 -- 172.21.15.42:0/1233503982 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff56008f090 msgr2=0x7ff5600722c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:30.519 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.514+0000 7ff565ae7700 1 --2- 172.21.15.42:0/1233503982 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff56008f090 0x7ff5600722c0 secure :-1 s=READY pgs=2446 cs=0 l=1 rx=0x7ff56007ee80 tx=0x7ff55400c430).stop 2019-12-05T06:19:30.519 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.514+0000 7ff565ae7700 1 -- 172.21.15.42:0/1233503982 shutdown_connections 2019-12-05T06:19:30.519 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.514+0000 7ff565ae7700 1 --2- 172.21.15.42:0/1233503982 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff54003ac80 0x7ff54003d190 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:30.519 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.514+0000 7ff565ae7700 1 --2- 172.21.15.42:0/1233503982 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff56008f090 0x7ff5600722c0 unknown :-1 s=CLOSED pgs=2446 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:30.519 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.514+0000 7ff565ae7700 1 -- 172.21.15.42:0/1233503982 shutdown_connections 2019-12-05T06:19:30.520 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.514+0000 7ff565ae7700 1 -- 172.21.15.42:0/1233503982 wait complete. 2019-12-05T06:19:30.520 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:30.514+0000 7ff565ae7700 1 -- 172.21.15.42:0/1233503982 >> 172.21.15.42:0/1233503982 conn(0x7ff56008fda0 msgr2=0x7ff5600a9640 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:19:30.531 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:19:32.532 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:19:32.646 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:32.638+0000 7f26ee4a0700 1 Processor -- start 2019-12-05T06:19:32.646 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:32.638+0000 7f26ee4a0700 1 -- start start 2019-12-05T06:19:32.646 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:32.638+0000 7f26ee4a0700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f26e80a7740 0x7f26e808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:19:32.646 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:32.638+0000 7f26ee4a0700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f26e807e720 0x7f26e807eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:19:32.646 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:32.638+0000 7f26ee4a0700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f26e809f2e0 0x7f26e809f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:19:32.647 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:32.638+0000 7f26ee4a0700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26e80a0b00 con 0x7f26e807e720 2019-12-05T06:19:32.647 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:32.638+0000 7f26ee4a0700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26e808e5b0 con 0x7f26e809f2e0 2019-12-05T06:19:32.647 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:32.638+0000 7f26ee4a0700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26e808e850 con 0x7f26e80a7740 2019-12-05T06:19:32.647 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:32.638+0000 7f26ed23e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f26e809f2e0 0x7f26e809f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:32.647 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:32.638+0000 7f26ed23e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f26e809f2e0 0x7f26e809f710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:52958/0 (socket says 172.21.15.42:52958) 2019-12-05T06:19:32.647 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:32.638+0000 7f26ed23e700 1 -- 172.21.15.42:0/818595634 learned_addr learned my addr 172.21.15.42:0/818595634 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:19:32.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:32.638+0000 7f26e7fff700 1 --2- 172.21.15.42:0/818595634 >> v2:172.21.15.40:3300/0 conn(0x7f26e807e720 0x7f26e807eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:32.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:32.638+0000 7f26eca3d700 1 --2- 172.21.15.42:0/818595634 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f26e80a7740 0x7f26e808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:32.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:32.638+0000 7f26ed23e700 1 -- 172.21.15.42:0/818595634 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f26e80a7740 msgr2=0x7f26e808df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:19:32.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:32.638+0000 7f26ed23e700 1 --2- 172.21.15.42:0/818595634 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f26e80a7740 0x7f26e808df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:19:32.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:32.638+0000 7f26ed23e700 1 -- 172.21.15.42:0/818595634 >> v2:172.21.15.40:3300/0 conn(0x7f26e807e720 msgr2=0x7f26e807eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:19:32.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:32.638+0000 7f26ed23e700 1 --2- 172.21.15.42:0/818595634 >> v2:172.21.15.40:3300/0 conn(0x7f26e807e720 0x7f26e807eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:19:32.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:32.638+0000 7f26ed23e700 1 -- 172.21.15.42:0/818595634 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26e807f0a0 con 0x7f26e809f2e0 2019-12-05T06:19:32.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:32.638+0000 7f26ed23e700 1 --2- 172.21.15.42:0/818595634 >> v2:172.21.15.42:3300/0 conn(0x7f26e809f2e0 0x7f26e809f710 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f26e808e770 tx=0x7f26d800fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:32.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:32.638+0000 7f26ed23e700 1 --2- 172.21.15.42:0/818595634 >> v2:172.21.15.42:3300/0 conn(0x7f26e809f2e0 0x7f26e809f710 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f26e808e770 tx=0x7f26d800fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:32.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:32.638+0000 7f26ed23e700 1 --2- 172.21.15.42:0/818595634 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f26e809f2e0 0x7f26e809f710 secure :-1 s=READY pgs=2447 cs=0 l=1 rx=0x7f26e808e770 tx=0x7f26d800fdb0).ready entity=mon.1 client_cookie=bafda37157f7204e server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:19:32.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:32.638+0000 7f26ed23e700 1 --2- 172.21.15.42:0/818595634 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f26e809f2e0 0x7f26e809f710 secure :-1 s=THROTTLE_DONE pgs=2447 cs=0 l=1 rx=0x7f26e808e770 tx=0x7f26d800fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:32.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:32.638+0000 7f26ed23e700 1 --2- 172.21.15.42:0/818595634 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f26e809f2e0 0x7f26e809f710 secure :-1 s=THROTTLE_DONE pgs=2447 cs=0 l=1 rx=0x7f26e808e770 tx=0x7f26d800fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:32.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:32.638+0000 7f26e77fe700 1 -- 172.21.15.42:0/818595634 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f26d8008c90 con 0x7f26e809f2e0 2019-12-05T06:19:32.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:32.638+0000 7f26ed23e700 1 --2- 172.21.15.42:0/818595634 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f26e809f2e0 0x7f26e809f710 secure :-1 s=THROTTLE_DONE pgs=2447 cs=0 l=1 rx=0x7f26e808e770 tx=0x7f26d800fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:32.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:32.638+0000 7f26e77fe700 1 -- 172.21.15.42:0/818595634 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26d80088b0 con 0x7f26e809f2e0 2019-12-05T06:19:32.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:32.638+0000 7f26e77fe700 1 -- 172.21.15.42:0/818595634 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f26d8011b50 con 0x7f26e809f2e0 2019-12-05T06:19:32.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:32.638+0000 7f26ee4a0700 1 -- 172.21.15.42:0/818595634 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f26e809f2e0 msgr2=0x7f26e809f710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:32.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:32.638+0000 7f26ee4a0700 1 --2- 172.21.15.42:0/818595634 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f26e809f2e0 0x7f26e809f710 secure :-1 s=READY pgs=2447 cs=0 l=1 rx=0x7f26e808e770 tx=0x7f26d800fdb0).stop 2019-12-05T06:19:32.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:32.638+0000 7f26ee4a0700 1 -- 172.21.15.42:0/818595634 shutdown_connections 2019-12-05T06:19:32.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:32.638+0000 7f26ee4a0700 1 --2- 172.21.15.42:0/818595634 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f26e809f2e0 0x7f26e809f710 unknown :-1 s=CLOSED pgs=2447 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:32.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:32.638+0000 7f26ee4a0700 1 --2- 172.21.15.42:0/818595634 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f26e80a7740 0x7f26e808df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:19:32.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:32.638+0000 7f26ee4a0700 1 --2- 172.21.15.42:0/818595634 >> v2:172.21.15.40:3300/0 conn(0x7f26e807e720 0x7f26e807eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:19:32.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:32.638+0000 7f26ee4a0700 1 -- 172.21.15.42:0/818595634 shutdown_connections 2019-12-05T06:19:32.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:32.642+0000 7f26ee4a0700 1 -- 172.21.15.42:0/818595634 wait complete. 2019-12-05T06:19:32.654 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:32.642+0000 7f26ee4a0700 1 -- 172.21.15.42:0/818595634 >> 172.21.15.42:0/818595634 conn(0x7f26e80aabe0 msgr2=0x7f26e80a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:19:32.654 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:32.642+0000 7f26ee4a0700 1 Processor -- start 2019-12-05T06:19:32.654 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:32.642+0000 7f26ee4a0700 1 -- start start 2019-12-05T06:19:32.657 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:32.642+0000 7f26ee4a0700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f26e8116c40 0x7f26e811be50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:19:32.657 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:32.642+0000 7f26ee4a0700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f26e808eb70 0x7f26e811ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:19:32.657 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:32.642+0000 7f26ee4a0700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26e808efa0 con 0x7f26e80a6f30 2019-12-05T06:19:32.657 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:32.642+0000 7f26ee4a0700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26e808f280 con 0x7f26e808eb70 2019-12-05T06:19:32.657 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:32.642+0000 7f26ee4a0700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26e811b910 con 0x7f26e8116c40 2019-12-05T06:19:32.657 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:32.642+0000 7f26e7fff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f26e8116c40 0x7f26e811be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:32.657 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:32.642+0000 7f26e7fff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f26e8116c40 0x7f26e811be50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:52962/0 (socket says 172.21.15.42:52962) 2019-12-05T06:19:32.658 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:32.642+0000 7f26e7fff700 1 -- 172.21.15.42:0/2482857918 learned_addr learned my addr 172.21.15.42:0/2482857918 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:19:32.658 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:32.642+0000 7f26ed23e700 1 --2- 172.21.15.42:0/2482857918 >> v2:172.21.15.40:3300/0 conn(0x7f26e808eb70 0x7f26e811ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:32.658 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:32.642+0000 7f26e7fff700 1 -- 172.21.15.42:0/2482857918 >> v2:172.21.15.40:3300/0 conn(0x7f26e808eb70 msgr2=0x7f26e811ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:32.658 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:32.642+0000 7f26e7fff700 1 --2- 172.21.15.42:0/2482857918 >> v2:172.21.15.40:3300/0 conn(0x7f26e808eb70 0x7f26e811ed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:32.658 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:32.642+0000 7f26e7fff700 1 -- 172.21.15.42:0/2482857918 >> v1:172.21.15.42:6789/0 conn(0x7f26e80a6f30 legacy=0x7f26e811a0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:32.658 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:32.642+0000 7f26e7fff700 1 -- 172.21.15.42:0/2482857918 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26e811f260 con 0x7f26e8116c40 2019-12-05T06:19:32.658 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:32.642+0000 7f26e57fa700 1 -- 172.21.15.42:0/2482857918 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3821752887 0 0) 0x7f26d0001ec0 con 0x7f26e80a6f30 2019-12-05T06:19:32.659 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:32.642+0000 7f26e7fff700 1 --2- 172.21.15.42:0/2482857918 >> v2:172.21.15.42:3300/0 conn(0x7f26e8116c40 0x7f26e811be50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f26e00033f0 tx=0x7f26e0010f40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:32.659 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:32.642+0000 7f26e7fff700 1 --2- 172.21.15.42:0/2482857918 >> v2:172.21.15.42:3300/0 conn(0x7f26e8116c40 0x7f26e811be50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f26e00033f0 tx=0x7f26e0010f40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:32.659 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:32.642+0000 7f26e7fff700 1 --2- 172.21.15.42:0/2482857918 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f26e8116c40 0x7f26e811be50 secure :-1 s=READY pgs=2449 cs=0 l=1 rx=0x7f26e00033f0 tx=0x7f26e0010f40).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:19:32.659 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:32.642+0000 7f26e7fff700 1 --2- 172.21.15.42:0/2482857918 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f26e8116c40 0x7f26e811be50 secure :-1 s=THROTTLE_DONE pgs=2449 cs=0 l=1 rx=0x7f26e00033f0 tx=0x7f26e0010f40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:32.659 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:32.646+0000 7f26e57fa700 1 -- 172.21.15.42:0/2482857918 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f26e00058c0 con 0x7f26e8116c40 2019-12-05T06:19:32.659 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:32.646+0000 7f26ee4a0700 1 -- 172.21.15.42:0/2482857918 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26e81162c0 con 0x7f26e8116c40 2019-12-05T06:19:32.659 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:32.646+0000 7f26e7fff700 1 --2- 172.21.15.42:0/2482857918 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f26e8116c40 0x7f26e811be50 secure :-1 s=THROTTLE_DONE pgs=2449 cs=0 l=1 rx=0x7f26e00033f0 tx=0x7f26e0010f40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:32.660 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:32.646+0000 7f26e57fa700 1 -- 172.21.15.42:0/2482857918 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26e0010200 con 0x7f26e8116c40 2019-12-05T06:19:32.660 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:32.646+0000 7f26e7fff700 1 --2- 172.21.15.42:0/2482857918 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f26e8116c40 0x7f26e811be50 secure :-1 s=THROTTLE_DONE pgs=2449 cs=0 l=1 rx=0x7f26e00033f0 tx=0x7f26e0010f40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:32.660 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:32.646+0000 7f26e57fa700 1 -- 172.21.15.42:0/2482857918 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f26e000e820 con 0x7f26e8116c40 2019-12-05T06:19:32.660 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:32.646+0000 7f26ee4a0700 1 -- 172.21.15.42:0/2482857918 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26e811a8f0 con 0x7f26e8116c40 2019-12-05T06:19:32.660 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:32.646+0000 7f26e7fff700 1 --2- 172.21.15.42:0/2482857918 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f26e8116c40 0x7f26e811be50 secure :-1 s=THROTTLE_DONE pgs=2449 cs=0 l=1 rx=0x7f26e00033f0 tx=0x7f26e0010f40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:32.660 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:32.646+0000 7f26e57fa700 1 -- 172.21.15.42:0/2482857918 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f26e000eac0 con 0x7f26e8116c40 2019-12-05T06:19:32.660 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:32.646+0000 7f26e7fff700 1 --2- 172.21.15.42:0/2482857918 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f26e8116c40 0x7f26e811be50 secure :-1 s=THROTTLE_DONE pgs=2449 cs=0 l=1 rx=0x7f26e00033f0 tx=0x7f26e0010f40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:32.661 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:32.646+0000 7f26e57fa700 1 --2- 172.21.15.42:0/2482857918 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f26c003ad60 0x7f26c003d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:19:32.661 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:32.646+0000 7f26e57fa700 1 -- 172.21.15.42:0/2482857918 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 10503+0+0 (secure 0 0 0) 0x7f26e004b0b0 con 0x7f26e8116c40 2019-12-05T06:19:32.661 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:32.646+0000 7f26eca3d700 1 --2- 172.21.15.42:0/2482857918 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f26c003ad60 0x7f26c003d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:32.661 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:32.646+0000 7f26eca3d700 1 --2- 172.21.15.42:0/2482857918 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f26c003ad60 0x7f26c003d270 crc :-1 s=READY pgs=1028 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:19:32.661 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:32.654+0000 7f26ee4a0700 1 -- 172.21.15.42:0/2482857918 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26e811af00 con 0x7f26e8116c40 2019-12-05T06:19:32.665 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:32.662+0000 7f26e7fff700 1 --2- 172.21.15.42:0/2482857918 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f26e8116c40 0x7f26e811be50 secure :-1 s=THROTTLE_DONE pgs=2449 cs=0 l=1 rx=0x7f26e00033f0 tx=0x7f26e0010f40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:32.665 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:32.662+0000 7f26e57fa700 1 -- 172.21.15.42:0/2482857918 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f26e811f260 con 0x7f26e8116c40 2019-12-05T06:19:32.832 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:32.830+0000 7f26ee4a0700 1 -- 172.21.15.42:0/2482857918 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f26e805d110 con 0x7f26e8116c40 2019-12-05T06:19:32.833 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:32.830+0000 7f26e7fff700 1 --2- 172.21.15.42:0/2482857918 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f26e8116c40 0x7f26e811be50 secure :-1 s=THROTTLE_DONE pgs=2449 cs=0 l=1 rx=0x7f26e00033f0 tx=0x7f26e0010f40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:32.833 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:32.830+0000 7f26e57fa700 1 -- 172.21.15.42:0/2482857918 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 293 v293) v1 ==== 95+0+2117 (secure 0 0 0) 0x7f26e000edd0 con 0x7f26e8116c40 2019-12-05T06:19:32.835 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 293 2019-12-05T06:19:32.835 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:19:32.835 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":293,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":10711,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/2638662429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2638662429},{"type":"v1","addr":"172.21.15.42:6835","nonce":2638662429}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":293}],"filesystems":[{"mdsmap":{"epoch":292,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:19:15.110045+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":101,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10476},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10476":{"gid":10476,"name":"b","rank":0,"incarnation":289,"state":"up:active","state_seq":34,"addr":"172.21.15.40:6833/1473620421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1473620421},{"type":"v1","addr":"172.21.15.40:6833","nonce":1473620421}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:19:32.839 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:32.834+0000 7f26ee4a0700 1 -- 172.21.15.42:0/2482857918 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f26c003ad60 msgr2=0x7f26c003d270 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:32.839 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:32.834+0000 7f26ee4a0700 1 --2- 172.21.15.42:0/2482857918 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f26c003ad60 0x7f26c003d270 crc :-1 s=READY pgs=1028 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:32.840 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:32.834+0000 7f26ee4a0700 1 -- 172.21.15.42:0/2482857918 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f26e8116c40 msgr2=0x7f26e811be50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:32.840 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:32.834+0000 7f26ee4a0700 1 --2- 172.21.15.42:0/2482857918 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f26e8116c40 0x7f26e811be50 secure :-1 s=READY pgs=2449 cs=0 l=1 rx=0x7f26e00033f0 tx=0x7f26e0010f40).stop 2019-12-05T06:19:32.840 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:32.834+0000 7f26ee4a0700 1 -- 172.21.15.42:0/2482857918 shutdown_connections 2019-12-05T06:19:32.840 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:32.834+0000 7f26ee4a0700 1 --2- 172.21.15.42:0/2482857918 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f26c003ad60 0x7f26c003d270 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:32.840 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:32.834+0000 7f26ee4a0700 1 --2- 172.21.15.42:0/2482857918 >> v2:172.21.15.40:3300/0 conn(0x7f26e808eb70 0x7f26e811ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:32.840 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:32.834+0000 7f26ee4a0700 1 --2- 172.21.15.42:0/2482857918 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f26e8116c40 0x7f26e811be50 unknown :-1 s=CLOSED pgs=2449 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:32.840 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:32.834+0000 7f26ee4a0700 1 -- 172.21.15.42:0/2482857918 shutdown_connections 2019-12-05T06:19:32.841 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:32.834+0000 7f26ee4a0700 1 -- 172.21.15.42:0/2482857918 wait complete. 2019-12-05T06:19:32.841 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:32.834+0000 7f26ee4a0700 1 -- 172.21.15.42:0/2482857918 >> 172.21.15.42:0/2482857918 conn(0x7f26e809f380 msgr2=0x7f26e811d6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:19:32.852 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:19:34.857 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:19:34.963 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:34.954+0000 7fcc94857700 1 Processor -- start 2019-12-05T06:19:34.963 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:34.954+0000 7fcc94857700 1 -- start start 2019-12-05T06:19:34.963 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:34.954+0000 7fcc94857700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcc900a7740 0x7fcc9008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:19:34.963 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:34.954+0000 7fcc94857700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fcc9009fae0 0x7fcc9007eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:19:34.963 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:34.954+0000 7fcc94857700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc900a0710 con 0x7fcc9007e720 2019-12-05T06:19:34.964 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:34.954+0000 7fcc94857700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc900a0b00 con 0x7fcc9009fae0 2019-12-05T06:19:34.964 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:34.954+0000 7fcc94857700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc9008e5b0 con 0x7fcc900a7740 2019-12-05T06:19:34.964 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:34.954+0000 7fcc8f59e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fcc9009fae0 0x7fcc9007eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:34.964 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:34.954+0000 7fcc8659c700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fcc9007e720 0x7fcc9009f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:53010/0 (socket says 172.21.15.42:53010) 2019-12-05T06:19:34.964 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:34.954+0000 7fcc8f59e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fcc9009fae0 0x7fcc9007eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:52974/0 (socket says 172.21.15.42:52974) 2019-12-05T06:19:34.964 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:34.954+0000 7fcc8659c700 1 -- 172.21.15.42:0/1971540709 learned_addr learned my addr 172.21.15.42:0/1971540709 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:19:34.964 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:34.954+0000 7fcc8ed9d700 1 --2- 172.21.15.42:0/1971540709 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcc900a7740 0x7fcc9008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:34.965 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:34.958+0000 7fcc8f59e700 1 -- 172.21.15.42:0/1971540709 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcc900a7740 msgr2=0x7fcc9008df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:19:34.965 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:34.958+0000 7fcc8f59e700 1 --2- 172.21.15.42:0/1971540709 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcc900a7740 0x7fcc9008df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:19:34.965 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:34.958+0000 7fcc8f59e700 1 -- 172.21.15.42:0/1971540709 >> v1:172.21.15.42:6789/0 conn(0x7fcc9007e720 legacy=0x7fcc9009f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:34.965 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:34.958+0000 7fcc8f59e700 1 -- 172.21.15.42:0/1971540709 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc9007f0a0 con 0x7fcc9009fae0 2019-12-05T06:19:34.965 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:34.958+0000 7fcc8f59e700 1 --2- 172.21.15.42:0/1971540709 >> v2:172.21.15.42:3300/0 conn(0x7fcc9009fae0 0x7fcc9007eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fcc88008b10 tx=0x7fcc8800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:34.965 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:34.958+0000 7fcc8f59e700 1 --2- 172.21.15.42:0/1971540709 >> v2:172.21.15.42:3300/0 conn(0x7fcc9009fae0 0x7fcc9007eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fcc88008b10 tx=0x7fcc8800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:34.965 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:34.958+0000 7fcc8f59e700 1 --2- 172.21.15.42:0/1971540709 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcc9009fae0 0x7fcc9007eb50 secure :-1 s=READY pgs=2451 cs=0 l=1 rx=0x7fcc88008b10 tx=0x7fcc8800fff0).ready entity=mon.1 client_cookie=3d669bff417d98a3 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:19:34.966 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:34.958+0000 7fcc8f59e700 1 --2- 172.21.15.42:0/1971540709 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcc9009fae0 0x7fcc9007eb50 secure :-1 s=THROTTLE_DONE pgs=2451 cs=0 l=1 rx=0x7fcc88008b10 tx=0x7fcc8800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:34.966 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:34.958+0000 7fcc8e59c700 1 -- 172.21.15.42:0/1971540709 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcc88010900 con 0x7fcc9009fae0 2019-12-05T06:19:34.968 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:34.958+0000 7fcc8f59e700 1 --2- 172.21.15.42:0/1971540709 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcc9009fae0 0x7fcc9007eb50 secure :-1 s=THROTTLE_DONE pgs=2451 cs=0 l=1 rx=0x7fcc88008b10 tx=0x7fcc8800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:34.968 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:34.958+0000 7fcc8e59c700 1 -- 172.21.15.42:0/1971540709 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc88008b50 con 0x7fcc9009fae0 2019-12-05T06:19:34.968 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:34.958+0000 7fcc8f59e700 1 --2- 172.21.15.42:0/1971540709 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcc9009fae0 0x7fcc9007eb50 secure :-1 s=THROTTLE_DONE pgs=2451 cs=0 l=1 rx=0x7fcc88008b10 tx=0x7fcc8800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:34.969 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:34.958+0000 7fcc8e59c700 1 -- 172.21.15.42:0/1971540709 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcc880110c0 con 0x7fcc9009fae0 2019-12-05T06:19:34.969 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:34.958+0000 7fcc94857700 1 -- 172.21.15.42:0/1971540709 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcc9009fae0 msgr2=0x7fcc9007eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:34.969 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:34.958+0000 7fcc94857700 1 --2- 172.21.15.42:0/1971540709 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcc9009fae0 0x7fcc9007eb50 secure :-1 s=READY pgs=2451 cs=0 l=1 rx=0x7fcc88008b10 tx=0x7fcc8800fff0).stop 2019-12-05T06:19:34.969 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:34.958+0000 7fcc94857700 1 -- 172.21.15.42:0/1971540709 shutdown_connections 2019-12-05T06:19:34.969 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:34.958+0000 7fcc94857700 1 --2- 172.21.15.42:0/1971540709 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcc9009fae0 0x7fcc9007eb50 unknown :-1 s=CLOSED pgs=2451 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:34.969 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:34.958+0000 7fcc94857700 1 --2- 172.21.15.42:0/1971540709 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcc900a7740 0x7fcc9008df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:19:34.969 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:34.958+0000 7fcc94857700 1 -- 172.21.15.42:0/1971540709 shutdown_connections 2019-12-05T06:19:34.970 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:34.958+0000 7fcc94857700 1 -- 172.21.15.42:0/1971540709 wait complete. 2019-12-05T06:19:34.970 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:34.958+0000 7fcc94857700 1 -- 172.21.15.42:0/1971540709 >> 172.21.15.42:0/1971540709 conn(0x7fcc900aabe0 msgr2=0x7fcc900a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:19:34.973 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:34.962+0000 7fcc94857700 1 Processor -- start 2019-12-05T06:19:34.973 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:34.962+0000 7fcc94857700 1 -- start start 2019-12-05T06:19:34.973 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:34.962+0000 7fcc94857700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fcc9008eb70 0x7fcc901267e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:19:34.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:34.962+0000 7fcc94857700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fcc90126d30 0x7fcc90127160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:19:34.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:34.962+0000 7fcc94857700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc9011c490 con 0x7fcc90116500 2019-12-05T06:19:34.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:34.962+0000 7fcc94857700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc9011bf50 con 0x7fcc90126d30 2019-12-05T06:19:34.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:34.962+0000 7fcc94857700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc9011c1f0 con 0x7fcc9008eb70 2019-12-05T06:19:34.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:34.962+0000 7fcc8659c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fcc9008eb70 0x7fcc901267e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:34.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:34.962+0000 7fcc8659c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fcc9008eb70 0x7fcc901267e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:52978/0 (socket says 172.21.15.42:52978) 2019-12-05T06:19:34.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:34.962+0000 7fcc8659c700 1 -- 172.21.15.42:0/2337570361 learned_addr learned my addr 172.21.15.42:0/2337570361 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:19:34.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:34.962+0000 7fcc8f59e700 1 --2- 172.21.15.42:0/2337570361 >> v2:172.21.15.40:3300/0 conn(0x7fcc90126d30 0x7fcc90127160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:34.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:34.962+0000 7fcc8659c700 1 -- 172.21.15.42:0/2337570361 >> v2:172.21.15.40:3300/0 conn(0x7fcc90126d30 msgr2=0x7fcc90127160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:34.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:34.962+0000 7fcc8659c700 1 --2- 172.21.15.42:0/2337570361 >> v2:172.21.15.40:3300/0 conn(0x7fcc90126d30 0x7fcc90127160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:34.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:34.962+0000 7fcc8659c700 1 -- 172.21.15.42:0/2337570361 >> v1:172.21.15.40:6789/0 conn(0x7fcc90116500 legacy=0x7fcc9011a690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:34.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:34.962+0000 7fcc8659c700 1 -- 172.21.15.42:0/2337570361 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc9011ae90 con 0x7fcc9008eb70 2019-12-05T06:19:34.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:34.962+0000 7fcc8659c700 1 --2- 172.21.15.42:0/2337570361 >> v2:172.21.15.42:3300/0 conn(0x7fcc9008eb70 0x7fcc901267e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fcc78004370 tx=0x7fcc78010a00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:34.976 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:34.962+0000 7fcc8659c700 1 --2- 172.21.15.42:0/2337570361 >> v2:172.21.15.42:3300/0 conn(0x7fcc9008eb70 0x7fcc901267e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fcc78004370 tx=0x7fcc78010a00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:34.976 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:34.962+0000 7fcc8659c700 1 --2- 172.21.15.42:0/2337570361 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcc9008eb70 0x7fcc901267e0 secure :-1 s=READY pgs=2452 cs=0 l=1 rx=0x7fcc78004370 tx=0x7fcc78010a00).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:19:34.976 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:34.962+0000 7fcc8659c700 1 --2- 172.21.15.42:0/2337570361 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcc9008eb70 0x7fcc901267e0 secure :-1 s=THROTTLE_DONE pgs=2452 cs=0 l=1 rx=0x7fcc78004370 tx=0x7fcc78010a00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:34.976 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:34.962+0000 7fcc87fff700 1 -- 172.21.15.42:0/2337570361 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcc7800c8b0 con 0x7fcc9008eb70 2019-12-05T06:19:34.976 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:34.962+0000 7fcc8659c700 1 --2- 172.21.15.42:0/2337570361 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcc9008eb70 0x7fcc901267e0 secure :-1 s=THROTTLE_DONE pgs=2452 cs=0 l=1 rx=0x7fcc78004370 tx=0x7fcc78010a00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:34.976 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:34.962+0000 7fcc8659c700 1 --2- 172.21.15.42:0/2337570361 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcc9008eb70 0x7fcc901267e0 secure :-1 s=THROTTLE_DONE pgs=2452 cs=0 l=1 rx=0x7fcc78004370 tx=0x7fcc78010a00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:34.976 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:34.962+0000 7fcc87fff700 1 -- 172.21.15.42:0/2337570361 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc7800c4d0 con 0x7fcc9008eb70 2019-12-05T06:19:34.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:34.962+0000 7fcc94857700 1 -- 172.21.15.42:0/2337570361 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc9008f8b0 con 0x7fcc9008eb70 2019-12-05T06:19:34.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:34.962+0000 7fcc87fff700 1 -- 172.21.15.42:0/2337570361 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcc78004430 con 0x7fcc9008eb70 2019-12-05T06:19:34.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:34.962+0000 7fcc94857700 1 -- 172.21.15.42:0/2337570361 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc9007bb70 con 0x7fcc9008eb70 2019-12-05T06:19:34.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:34.962+0000 7fcc8659c700 1 --2- 172.21.15.42:0/2337570361 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcc9008eb70 0x7fcc901267e0 secure :-1 s=THROTTLE_DONE pgs=2452 cs=0 l=1 rx=0x7fcc78004370 tx=0x7fcc78010a00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:34.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:34.966+0000 7fcc8659c700 1 --2- 172.21.15.42:0/2337570361 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcc9008eb70 0x7fcc901267e0 secure :-1 s=THROTTLE_DONE pgs=2452 cs=0 l=1 rx=0x7fcc78004370 tx=0x7fcc78010a00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:34.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:34.966+0000 7fcc87fff700 1 -- 172.21.15.42:0/2337570361 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fcc78004700 con 0x7fcc9008eb70 2019-12-05T06:19:34.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:34.966+0000 7fcc87fff700 1 --2- 172.21.15.42:0/2337570361 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fcc7403ad20 0x7fcc7403d230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:19:34.978 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:34.966+0000 7fcc87fff700 1 -- 172.21.15.42:0/2337570361 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 10503+0+0 (secure 0 0 0) 0x7fcc7804ac50 con 0x7fcc9008eb70 2019-12-05T06:19:34.978 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:34.966+0000 7fcc8ed9d700 1 --2- 172.21.15.42:0/2337570361 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fcc7403ad20 0x7fcc7403d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:34.978 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:34.966+0000 7fcc8ed9d700 1 --2- 172.21.15.42:0/2337570361 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fcc7403ad20 0x7fcc7403d230 crc :-1 s=READY pgs=1029 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:19:34.984 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:34.970+0000 7fcc94857700 1 -- 172.21.15.42:0/2337570361 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc9005d110 con 0x7fcc9008eb70 2019-12-05T06:19:34.985 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:34.978+0000 7fcc8659c700 1 --2- 172.21.15.42:0/2337570361 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcc9008eb70 0x7fcc901267e0 secure :-1 s=THROTTLE_DONE pgs=2452 cs=0 l=1 rx=0x7fcc78004370 tx=0x7fcc78010a00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:34.985 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:34.978+0000 7fcc87fff700 1 -- 172.21.15.42:0/2337570361 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fcc78031c90 con 0x7fcc9008eb70 2019-12-05T06:19:35.156 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:35.154+0000 7fcc94857700 1 -- 172.21.15.42:0/2337570361 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcc9005b8c0 con 0x7fcc9008eb70 2019-12-05T06:19:35.157 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:35.154+0000 7fcc8659c700 1 --2- 172.21.15.42:0/2337570361 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcc9008eb70 0x7fcc901267e0 secure :-1 s=THROTTLE_DONE pgs=2452 cs=0 l=1 rx=0x7fcc78004370 tx=0x7fcc78010a00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:35.157 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:35.154+0000 7fcc87fff700 1 -- 172.21.15.42:0/2337570361 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 293 v293) v1 ==== 95+0+2117 (secure 0 0 0) 0x7fcc78031ef0 con 0x7fcc9008eb70 2019-12-05T06:19:35.158 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 293 2019-12-05T06:19:35.158 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:19:35.158 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":293,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":10711,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/2638662429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2638662429},{"type":"v1","addr":"172.21.15.42:6835","nonce":2638662429}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":293}],"filesystems":[{"mdsmap":{"epoch":292,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:19:15.110045+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":101,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10476},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10476":{"gid":10476,"name":"b","rank":0,"incarnation":289,"state":"up:active","state_seq":34,"addr":"172.21.15.40:6833/1473620421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1473620421},{"type":"v1","addr":"172.21.15.40:6833","nonce":1473620421}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:19:35.160 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:35.154+0000 7fcc94857700 1 -- 172.21.15.42:0/2337570361 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fcc7403ad20 msgr2=0x7fcc7403d230 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:35.160 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:35.154+0000 7fcc94857700 1 --2- 172.21.15.42:0/2337570361 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fcc7403ad20 0x7fcc7403d230 crc :-1 s=READY pgs=1029 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:35.160 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:35.154+0000 7fcc94857700 1 -- 172.21.15.42:0/2337570361 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcc9008eb70 msgr2=0x7fcc901267e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:35.161 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:35.154+0000 7fcc94857700 1 --2- 172.21.15.42:0/2337570361 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcc9008eb70 0x7fcc901267e0 secure :-1 s=READY pgs=2452 cs=0 l=1 rx=0x7fcc78004370 tx=0x7fcc78010a00).stop 2019-12-05T06:19:35.161 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:35.154+0000 7fcc94857700 1 -- 172.21.15.42:0/2337570361 shutdown_connections 2019-12-05T06:19:35.161 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:35.154+0000 7fcc94857700 1 --2- 172.21.15.42:0/2337570361 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fcc7403ad20 0x7fcc7403d230 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:35.161 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:35.154+0000 7fcc94857700 1 --2- 172.21.15.42:0/2337570361 >> v2:172.21.15.40:3300/0 conn(0x7fcc90126d30 0x7fcc90127160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:35.161 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:35.154+0000 7fcc94857700 1 --2- 172.21.15.42:0/2337570361 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcc9008eb70 0x7fcc901267e0 unknown :-1 s=CLOSED pgs=2452 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:35.161 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:35.158+0000 7fcc94857700 1 -- 172.21.15.42:0/2337570361 shutdown_connections 2019-12-05T06:19:35.161 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:35.158+0000 7fcc94857700 1 -- 172.21.15.42:0/2337570361 wait complete. 2019-12-05T06:19:35.162 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:35.158+0000 7fcc94857700 1 -- 172.21.15.42:0/2337570361 >> 172.21.15.42:0/2337570361 conn(0x7fcc900a9800 msgr2=0x7fcc9011c770 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:19:35.171 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:19:37.172 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:19:37.280 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:37.270+0000 7f1dc8613700 1 Processor -- start 2019-12-05T06:19:37.280 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:37.270+0000 7f1dc8613700 1 -- start start 2019-12-05T06:19:37.280 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:37.270+0000 7f1dc8613700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f1dc00a7740 0x7f1dc008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:19:37.280 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:37.270+0000 7f1dc8613700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1dc007e720 0x7f1dc007eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:19:37.280 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:37.270+0000 7f1dc8613700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f1dc009f2e0 0x7f1dc009f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:19:37.280 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:37.270+0000 7f1dc8613700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1dc00a0b00 con 0x7f1dc009f2e0 2019-12-05T06:19:37.281 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:37.270+0000 7f1dc8613700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1dc008e5b0 con 0x7f1dc00a7740 2019-12-05T06:19:37.281 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:37.270+0000 7f1dc8613700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1dc008e850 con 0x7f1dc007e720 2019-12-05T06:19:37.281 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:37.270+0000 7f1dc6bb0700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f1dc00a7740 0x7f1dc008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:37.281 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:37.270+0000 7f1dc6bb0700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f1dc00a7740 0x7f1dc008df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:52988/0 (socket says 172.21.15.42:52988) 2019-12-05T06:19:37.281 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:37.270+0000 7f1dc6bb0700 1 -- 172.21.15.42:0/652736637 learned_addr learned my addr 172.21.15.42:0/652736637 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:19:37.281 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:37.270+0000 7f1dc63af700 1 --2- 172.21.15.42:0/652736637 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1dc007e720 0x7f1dc007eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:37.281 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:37.270+0000 7f1dc73b1700 1 --2- 172.21.15.42:0/652736637 >> v2:172.21.15.40:3300/0 conn(0x7f1dc009f2e0 0x7f1dc009f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:37.282 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:37.274+0000 7f1dc63af700 1 -- 172.21.15.42:0/652736637 >> v2:172.21.15.42:3300/0 conn(0x7f1dc00a7740 msgr2=0x7f1dc008df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:19:37.282 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:37.274+0000 7f1dc63af700 1 --2- 172.21.15.42:0/652736637 >> v2:172.21.15.42:3300/0 conn(0x7f1dc00a7740 0x7f1dc008df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:19:37.282 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:37.274+0000 7f1dc63af700 1 -- 172.21.15.42:0/652736637 >> v2:172.21.15.40:3300/0 conn(0x7f1dc009f2e0 msgr2=0x7f1dc009f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:19:37.282 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:37.274+0000 7f1dc63af700 1 --2- 172.21.15.42:0/652736637 >> v2:172.21.15.40:3300/0 conn(0x7f1dc009f2e0 0x7f1dc009f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:19:37.282 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:37.274+0000 7f1dc63af700 1 -- 172.21.15.42:0/652736637 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dc007f0a0 con 0x7f1dc007e720 2019-12-05T06:19:37.282 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:37.274+0000 7f1dc63af700 1 --2- 172.21.15.42:0/652736637 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1dc007e720 0x7f1dc007eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f1db0002f30 tx=0x7f1db000fcf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:37.282 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:37.274+0000 7f1dc63af700 1 --2- 172.21.15.42:0/652736637 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1dc007e720 0x7f1dc007eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f1db0002f30 tx=0x7f1db000fcf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:37.283 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:37.274+0000 7f1dc63af700 1 --2- 172.21.15.42:0/652736637 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1dc007e720 0x7f1dc007eb50 secure :-1 s=READY pgs=537 cs=0 l=1 rx=0x7f1db0002f30 tx=0x7f1db000fcf0).ready entity=mon.2 client_cookie=3de65fb34791472e server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:19:37.283 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:37.274+0000 7f1dc63af700 1 --2- 172.21.15.42:0/652736637 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1dc007e720 0x7f1dc007eb50 secure :-1 s=THROTTLE_DONE pgs=537 cs=0 l=1 rx=0x7f1db0002f30 tx=0x7f1db000fcf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:37.283 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:37.274+0000 7f1dc63af700 1 --2- 172.21.15.42:0/652736637 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1dc007e720 0x7f1dc007eb50 secure :-1 s=THROTTLE_DONE pgs=537 cs=0 l=1 rx=0x7f1db0002f30 tx=0x7f1db000fcf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:37.283 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:37.274+0000 7f1dc63af700 1 --2- 172.21.15.42:0/652736637 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1dc007e720 0x7f1dc007eb50 secure :-1 s=THROTTLE_DONE pgs=537 cs=0 l=1 rx=0x7f1db0002f30 tx=0x7f1db000fcf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:37.283 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:37.274+0000 7f1dc5bae700 1 -- 172.21.15.42:0/652736637 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1db0008bb0 con 0x7f1dc007e720 2019-12-05T06:19:37.283 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:37.274+0000 7f1dc5bae700 1 -- 172.21.15.42:0/652736637 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1db00087d0 con 0x7f1dc007e720 2019-12-05T06:19:37.283 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:37.274+0000 7f1dc5bae700 1 -- 172.21.15.42:0/652736637 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1db00118a0 con 0x7f1dc007e720 2019-12-05T06:19:37.284 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:37.274+0000 7f1dc8613700 1 -- 172.21.15.42:0/652736637 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1dc007e720 msgr2=0x7f1dc007eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:37.284 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:37.274+0000 7f1dc8613700 1 --2- 172.21.15.42:0/652736637 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1dc007e720 0x7f1dc007eb50 secure :-1 s=READY pgs=537 cs=0 l=1 rx=0x7f1db0002f30 tx=0x7f1db000fcf0).stop 2019-12-05T06:19:37.284 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:37.274+0000 7f1dc8613700 1 -- 172.21.15.42:0/652736637 shutdown_connections 2019-12-05T06:19:37.284 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:37.274+0000 7f1dc8613700 1 --2- 172.21.15.42:0/652736637 >> v2:172.21.15.40:3300/0 conn(0x7f1dc009f2e0 0x7f1dc009f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:19:37.284 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:37.274+0000 7f1dc8613700 1 --2- 172.21.15.42:0/652736637 >> v2:172.21.15.42:3300/0 conn(0x7f1dc00a7740 0x7f1dc008df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:19:37.284 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:37.274+0000 7f1dc8613700 1 --2- 172.21.15.42:0/652736637 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1dc007e720 0x7f1dc007eb50 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:37.285 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:37.274+0000 7f1dc8613700 1 -- 172.21.15.42:0/652736637 shutdown_connections 2019-12-05T06:19:37.285 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:37.274+0000 7f1dc8613700 1 -- 172.21.15.42:0/652736637 wait complete. 2019-12-05T06:19:37.285 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:37.274+0000 7f1dc8613700 1 -- 172.21.15.42:0/652736637 >> 172.21.15.42:0/652736637 conn(0x7f1dc00aabe0 msgr2=0x7f1dc00a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:19:37.285 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:37.278+0000 7f1dc8613700 1 Processor -- start 2019-12-05T06:19:37.295 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:37.278+0000 7f1dc8613700 1 -- start start 2019-12-05T06:19:37.295 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:37.278+0000 7f1dc8613700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1dc0120e20 0x7f1dc011b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:19:37.295 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:37.278+0000 7f1dc8613700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f1dc0116500 0x7f1dc011bd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:19:37.295 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:37.278+0000 7f1dc8613700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f1dc01217d0 0x7f1dc01301d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:19:37.295 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:37.278+0000 7f1dc8613700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1dc008faa0 con 0x7f1dc0116500 2019-12-05T06:19:37.296 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:37.278+0000 7f1dc8613700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1dc01214b0 con 0x7f1dc01217d0 2019-12-05T06:19:37.296 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:37.278+0000 7f1dc8613700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1dc008f560 con 0x7f1dc0120e20 2019-12-05T06:19:37.296 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:37.278+0000 7f1dc73b1700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f1dc01217d0 0x7f1dc01301d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:37.296 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:37.278+0000 7f1dc73b1700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f1dc01217d0 0x7f1dc01301d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:52996/0 (socket says 172.21.15.42:52996) 2019-12-05T06:19:37.296 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:37.278+0000 7f1dc73b1700 1 -- 172.21.15.42:0/1126414336 learned_addr learned my addr 172.21.15.42:0/1126414336 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:19:37.296 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:37.278+0000 7f1dc63af700 1 --2- 172.21.15.42:0/1126414336 >> v2:172.21.15.40:3300/0 conn(0x7f1dc0116500 0x7f1dc011bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:37.296 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:37.278+0000 7f1dc6bb0700 1 --2- 172.21.15.42:0/1126414336 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1dc0120e20 0x7f1dc011b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:37.297 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:37.282+0000 7f1dc73b1700 1 -- 172.21.15.42:0/1126414336 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1dc0120e20 msgr2=0x7f1dc011b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:37.297 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:37.282+0000 7f1dc73b1700 1 --2- 172.21.15.42:0/1126414336 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1dc0120e20 0x7f1dc011b7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:37.297 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:37.282+0000 7f1dc73b1700 1 -- 172.21.15.42:0/1126414336 >> v2:172.21.15.40:3300/0 conn(0x7f1dc0116500 msgr2=0x7f1dc011bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:37.297 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:37.282+0000 7f1dc73b1700 1 --2- 172.21.15.42:0/1126414336 >> v2:172.21.15.40:3300/0 conn(0x7f1dc0116500 0x7f1dc011bd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:37.297 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:37.282+0000 7f1dc73b1700 1 -- 172.21.15.42:0/1126414336 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dc011c270 con 0x7f1dc01217d0 2019-12-05T06:19:37.297 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:37.282+0000 7f1dc73b1700 1 --2- 172.21.15.42:0/1126414336 >> v2:172.21.15.42:3300/0 conn(0x7f1dc01217d0 0x7f1dc01301d0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f1db80032b0 tx=0x7f1db8010d80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:37.297 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:37.282+0000 7f1dc73b1700 1 --2- 172.21.15.42:0/1126414336 >> v2:172.21.15.42:3300/0 conn(0x7f1dc01217d0 0x7f1dc01301d0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f1db80032b0 tx=0x7f1db8010d80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:37.298 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:37.282+0000 7f1dc73b1700 1 --2- 172.21.15.42:0/1126414336 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1dc01217d0 0x7f1dc01301d0 secure :-1 s=READY pgs=2453 cs=0 l=1 rx=0x7f1db80032b0 tx=0x7f1db8010d80).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:19:37.298 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:37.282+0000 7f1dc73b1700 1 --2- 172.21.15.42:0/1126414336 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1dc01217d0 0x7f1dc01301d0 secure :-1 s=THROTTLE_DONE pgs=2453 cs=0 l=1 rx=0x7f1db80032b0 tx=0x7f1db8010d80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:37.298 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:37.282+0000 7f1db77fe700 1 -- 172.21.15.42:0/1126414336 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1db8012ab0 con 0x7f1dc01217d0 2019-12-05T06:19:37.298 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:37.282+0000 7f1dc73b1700 1 --2- 172.21.15.42:0/1126414336 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1dc01217d0 0x7f1dc01301d0 secure :-1 s=THROTTLE_DONE pgs=2453 cs=0 l=1 rx=0x7f1db80032b0 tx=0x7f1db8010d80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:37.298 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:37.282+0000 7f1dc8613700 1 -- 172.21.15.42:0/1126414336 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1dc0130720 con 0x7f1dc01217d0 2019-12-05T06:19:37.298 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:37.282+0000 7f1db77fe700 1 -- 172.21.15.42:0/1126414336 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1db80126d0 con 0x7f1dc01217d0 2019-12-05T06:19:37.298 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:37.282+0000 7f1dc73b1700 1 --2- 172.21.15.42:0/1126414336 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1dc01217d0 0x7f1dc01301d0 secure :-1 s=THROTTLE_DONE pgs=2453 cs=0 l=1 rx=0x7f1db80032b0 tx=0x7f1db8010d80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:37.299 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:37.282+0000 7f1db77fe700 1 -- 172.21.15.42:0/1126414336 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1db800e760 con 0x7f1dc01217d0 2019-12-05T06:19:37.299 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:37.282+0000 7f1dc8613700 1 -- 172.21.15.42:0/1126414336 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1dc0130c50 con 0x7f1dc01217d0 2019-12-05T06:19:37.299 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:37.282+0000 7f1dc73b1700 1 --2- 172.21.15.42:0/1126414336 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1dc01217d0 0x7f1dc01301d0 secure :-1 s=THROTTLE_DONE pgs=2453 cs=0 l=1 rx=0x7f1db80032b0 tx=0x7f1db8010d80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:37.299 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:37.282+0000 7f1db77fe700 1 -- 172.21.15.42:0/1126414336 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f1db800ea40 con 0x7f1dc01217d0 2019-12-05T06:19:37.299 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:37.282+0000 7f1dc73b1700 1 --2- 172.21.15.42:0/1126414336 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1dc01217d0 0x7f1dc01301d0 secure :-1 s=THROTTLE_DONE pgs=2453 cs=0 l=1 rx=0x7f1db80032b0 tx=0x7f1db8010d80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:37.299 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:37.282+0000 7f1db77fe700 1 --2- 172.21.15.42:0/1126414336 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f1da803acf0 0x7f1da803d200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:19:37.299 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:37.282+0000 7f1db77fe700 1 -- 172.21.15.42:0/1126414336 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 10503+0+0 (secure 0 0 0) 0x7f1db804b9f0 con 0x7f1dc01217d0 2019-12-05T06:19:37.300 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:37.282+0000 7f1dc6bb0700 1 --2- 172.21.15.42:0/1126414336 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f1da803acf0 0x7f1da803d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:37.300 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:37.286+0000 7f1dc6bb0700 1 --2- 172.21.15.42:0/1126414336 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f1da803acf0 0x7f1da803d200 crc :-1 s=READY pgs=1030 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:19:37.300 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:37.286+0000 7f1dc8613700 1 -- 172.21.15.42:0/1126414336 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1dc005d110 con 0x7f1dc01217d0 2019-12-05T06:19:37.300 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:37.290+0000 7f1dc73b1700 1 --2- 172.21.15.42:0/1126414336 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1dc01217d0 0x7f1dc01301d0 secure :-1 s=THROTTLE_DONE pgs=2453 cs=0 l=1 rx=0x7f1db80032b0 tx=0x7f1db8010d80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:37.300 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:37.290+0000 7f1db77fe700 1 -- 172.21.15.42:0/1126414336 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f1db8032990 con 0x7f1dc01217d0 2019-12-05T06:19:37.458 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:37.454+0000 7f1dc8613700 1 -- 172.21.15.42:0/1126414336 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1dc005b8c0 con 0x7f1dc01217d0 2019-12-05T06:19:37.459 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:37.454+0000 7f1dc73b1700 1 --2- 172.21.15.42:0/1126414336 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1dc01217d0 0x7f1dc01301d0 secure :-1 s=THROTTLE_DONE pgs=2453 cs=0 l=1 rx=0x7f1db80032b0 tx=0x7f1db8010d80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:37.459 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:37.454+0000 7f1db77fe700 1 -- 172.21.15.42:0/1126414336 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 293 v293) v1 ==== 95+0+2117 (secure 0 0 0) 0x7f1db8032bf0 con 0x7f1dc01217d0 2019-12-05T06:19:37.459 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 293 2019-12-05T06:19:37.459 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:19:37.459 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":293,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":10711,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/2638662429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2638662429},{"type":"v1","addr":"172.21.15.42:6835","nonce":2638662429}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":293}],"filesystems":[{"mdsmap":{"epoch":292,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:19:15.110045+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":101,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10476},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10476":{"gid":10476,"name":"b","rank":0,"incarnation":289,"state":"up:active","state_seq":34,"addr":"172.21.15.40:6833/1473620421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1473620421},{"type":"v1","addr":"172.21.15.40:6833","nonce":1473620421}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:19:37.462 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:37.458+0000 7f1dc8613700 1 -- 172.21.15.42:0/1126414336 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f1da803acf0 msgr2=0x7f1da803d200 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:37.462 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:37.458+0000 7f1dc8613700 1 --2- 172.21.15.42:0/1126414336 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f1da803acf0 0x7f1da803d200 crc :-1 s=READY pgs=1030 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:37.462 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:37.458+0000 7f1dc8613700 1 -- 172.21.15.42:0/1126414336 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1dc01217d0 msgr2=0x7f1dc01301d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:37.462 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:37.458+0000 7f1dc8613700 1 --2- 172.21.15.42:0/1126414336 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1dc01217d0 0x7f1dc01301d0 secure :-1 s=READY pgs=2453 cs=0 l=1 rx=0x7f1db80032b0 tx=0x7f1db8010d80).stop 2019-12-05T06:19:37.462 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:37.458+0000 7f1dc8613700 1 -- 172.21.15.42:0/1126414336 shutdown_connections 2019-12-05T06:19:37.462 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:37.458+0000 7f1dc8613700 1 --2- 172.21.15.42:0/1126414336 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f1da803acf0 0x7f1da803d200 unknown :-1 s=CLOSED pgs=1030 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:37.462 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:37.458+0000 7f1dc8613700 1 --2- 172.21.15.42:0/1126414336 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1dc01217d0 0x7f1dc01301d0 unknown :-1 s=CLOSED pgs=2453 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:37.463 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:37.458+0000 7f1dc8613700 1 --2- 172.21.15.42:0/1126414336 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1dc0120e20 0x7f1dc011b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:37.463 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:37.458+0000 7f1dc8613700 1 --2- 172.21.15.42:0/1126414336 >> v2:172.21.15.40:3300/0 conn(0x7f1dc0116500 0x7f1dc011bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:37.463 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:37.458+0000 7f1dc8613700 1 -- 172.21.15.42:0/1126414336 shutdown_connections 2019-12-05T06:19:37.463 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:37.458+0000 7f1dc8613700 1 -- 172.21.15.42:0/1126414336 wait complete. 2019-12-05T06:19:37.463 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:37.458+0000 7f1dc8613700 1 -- 172.21.15.42:0/1126414336 >> 172.21.15.42:0/1126414336 conn(0x7f1dc007e680 msgr2=0x7f1dc009f220 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:19:37.471 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:19:39.472 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:19:39.582 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:39.570+0000 7f4422bbc700 1 Processor -- start 2019-12-05T06:19:39.583 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:39.574+0000 7f4422bbc700 1 -- start start 2019-12-05T06:19:39.583 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:39.574+0000 7f4422bbc700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f441c0a7740 0x7f441c08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:19:39.583 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:39.574+0000 7f4422bbc700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f441c0a0710 con 0x7f441c07e720 2019-12-05T06:19:39.583 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:39.574+0000 7f4422bbc700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f441c0a9ba0 con 0x7f441c09fae0 2019-12-05T06:19:39.584 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:39.574+0000 7f4422bbc700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f441c0a0b00 con 0x7f441c0a7740 2019-12-05T06:19:39.596 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:39.574+0000 7f442195a700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f441c09fae0 0x7f441c14efe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:53040/0 (socket says 172.21.15.42:53040) 2019-12-05T06:19:39.596 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:39.574+0000 7f442195a700 1 -- 172.21.15.42:0/3293332452 learned_addr learned my addr 172.21.15.42:0/3293332452 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:19:39.598 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:39.574+0000 7f4421159700 1 --2- 172.21.15.42:0/3293332452 >> v2:172.21.15.40:3300/0 conn(0x7f441c0a7740 0x7f441c08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:39.598 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:39.574+0000 7f4413fff700 1 -- 172.21.15.42:0/3293332452 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2831857810 0 0) 0x7f441c0a9ba0 con 0x7f441c09fae0 2019-12-05T06:19:39.598 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:39.574+0000 7f4413fff700 1 -- 172.21.15.42:0/3293332452 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44080030b0 con 0x7f441c09fae0 2019-12-05T06:19:39.599 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:39.574+0000 7f4413fff700 1 -- 172.21.15.42:0/3293332452 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3601142350 0 0) 0x7f44080030b0 con 0x7f441c09fae0 2019-12-05T06:19:39.606 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:39.574+0000 7f4421159700 1 -- 172.21.15.42:0/3293332452 >> v1:172.21.15.42:6789/0 conn(0x7f441c09fae0 legacy=0x7f441c14efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:39.606 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:39.574+0000 7f4421159700 1 -- 172.21.15.42:0/3293332452 >> v1:172.21.15.40:6789/0 conn(0x7f441c07e720 legacy=0x7f441c09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:39.606 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:39.574+0000 7f4421159700 1 -- 172.21.15.42:0/3293332452 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f441c07eb50 con 0x7f441c0a7740 2019-12-05T06:19:39.606 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:39.574+0000 7f4421159700 1 --2- 172.21.15.42:0/3293332452 >> v2:172.21.15.40:3300/0 conn(0x7f441c0a7740 0x7f441c08df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f4418008790 tx=0x7f4414001d10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:39.606 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:39.574+0000 7f4421159700 1 --2- 172.21.15.42:0/3293332452 >> v2:172.21.15.40:3300/0 conn(0x7f441c0a7740 0x7f441c08df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f4418008790 tx=0x7f4414001d10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:39.607 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:39.574+0000 7f4421159700 1 --2- 172.21.15.42:0/3293332452 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f441c0a7740 0x7f441c08df80 secure :-1 s=READY pgs=1858 cs=0 l=1 rx=0x7f4418008790 tx=0x7f4414001d10).ready entity=mon.0 client_cookie=1053f98db0f9ddcd server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:19:39.607 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:39.574+0000 7f4421159700 1 --2- 172.21.15.42:0/3293332452 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f441c0a7740 0x7f441c08df80 secure :-1 s=THROTTLE_DONE pgs=1858 cs=0 l=1 rx=0x7f4418008790 tx=0x7f4414001d10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:39.607 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:39.574+0000 7f4413fff700 1 -- 172.21.15.42:0/3293332452 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4414001ad0 con 0x7f441c0a7740 2019-12-05T06:19:39.607 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:39.574+0000 7f4421159700 1 --2- 172.21.15.42:0/3293332452 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f441c0a7740 0x7f441c08df80 secure :-1 s=THROTTLE_DONE pgs=1858 cs=0 l=1 rx=0x7f4418008790 tx=0x7f4414001d10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:39.607 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:39.574+0000 7f4421159700 1 --2- 172.21.15.42:0/3293332452 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f441c0a7740 0x7f441c08df80 secure :-1 s=THROTTLE_DONE pgs=1858 cs=0 l=1 rx=0x7f4418008790 tx=0x7f4414001d10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:39.607 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:39.574+0000 7f4413fff700 1 -- 172.21.15.42:0/3293332452 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44140018e0 con 0x7f441c0a7740 2019-12-05T06:19:39.608 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:39.574+0000 7f4413fff700 1 -- 172.21.15.42:0/3293332452 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4418011360 con 0x7f441c0a7740 2019-12-05T06:19:39.608 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:39.574+0000 7f4422bbc700 1 -- 172.21.15.42:0/3293332452 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f441c0a7740 msgr2=0x7f441c08df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:39.608 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:39.578+0000 7f4422bbc700 1 --2- 172.21.15.42:0/3293332452 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f441c0a7740 0x7f441c08df80 secure :-1 s=READY pgs=1858 cs=0 l=1 rx=0x7f4418008790 tx=0x7f4414001d10).stop 2019-12-05T06:19:39.608 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:39.578+0000 7f4422bbc700 1 -- 172.21.15.42:0/3293332452 shutdown_connections 2019-12-05T06:19:39.608 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:39.578+0000 7f4422bbc700 1 --2- 172.21.15.42:0/3293332452 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f441c0a7740 0x7f441c08df80 unknown :-1 s=CLOSED pgs=1858 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:39.608 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:39.578+0000 7f4422bbc700 1 -- 172.21.15.42:0/3293332452 shutdown_connections 2019-12-05T06:19:39.608 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:39.578+0000 7f4422bbc700 1 -- 172.21.15.42:0/3293332452 wait complete. 2019-12-05T06:19:39.609 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:39.578+0000 7f4422bbc700 1 -- 172.21.15.42:0/3293332452 >> 172.21.15.42:0/3293332452 conn(0x7f441c0aabe0 msgr2=0x7f441c0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:19:39.613 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:39.578+0000 7f4422bbc700 1 Processor -- start 2019-12-05T06:19:39.613 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:39.582+0000 7f4422bbc700 1 -- start start 2019-12-05T06:19:39.613 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:39.582+0000 7f4422bbc700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f441c11df60 0x7f441c145790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:19:39.613 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:39.582+0000 7f4422bbc700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f441c145ce0 0x7f441c134530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:19:39.613 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:39.582+0000 7f4422bbc700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f441c12b0c0 con 0x7f441c0a56b0 2019-12-05T06:19:39.614 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:39.582+0000 7f4422bbc700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f441c147870 con 0x7f441c11df60 2019-12-05T06:19:39.614 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:39.582+0000 7f4422bbc700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f441c147b10 con 0x7f441c145ce0 2019-12-05T06:19:39.614 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:39.582+0000 7f4421159700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f441c0a56b0 0x7f441c148050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:34252/0 (socket says 172.21.15.42:34252) 2019-12-05T06:19:39.614 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:39.582+0000 7f4420958700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f441c11df60 0x7f441c145790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:39.614 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:39.582+0000 7f4421159700 1 -- 172.21.15.42:0/3361635059 learned_addr learned my addr 172.21.15.42:0/3361635059 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:19:39.614 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:39.582+0000 7f442195a700 1 --2- 172.21.15.42:0/3361635059 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f441c145ce0 0x7f441c134530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:39.614 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:39.582+0000 7f4411ffb700 1 -- 172.21.15.42:0/3361635059 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3455970065 0 0) 0x7f441c12b0c0 con 0x7f441c0a56b0 2019-12-05T06:19:39.615 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:39.582+0000 7f4411ffb700 1 -- 172.21.15.42:0/3361635059 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43fc0030b0 con 0x7f441c0a56b0 2019-12-05T06:19:39.615 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:39.582+0000 7f4420958700 1 -- 172.21.15.42:0/3361635059 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f441c145ce0 msgr2=0x7f441c134530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:39.615 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:39.582+0000 7f4420958700 1 --2- 172.21.15.42:0/3361635059 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f441c145ce0 0x7f441c134530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:39.615 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:39.582+0000 7f4420958700 1 -- 172.21.15.42:0/3361635059 >> v1:172.21.15.40:6789/0 conn(0x7f441c0a56b0 legacy=0x7f441c148050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:39.615 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:39.582+0000 7f4420958700 1 -- 172.21.15.42:0/3361635059 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f441c11d1a0 con 0x7f441c11df60 2019-12-05T06:19:39.615 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:39.582+0000 7f4420958700 1 --2- 172.21.15.42:0/3361635059 >> v2:172.21.15.42:3300/0 conn(0x7f441c11df60 0x7f441c145790 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f440c000b80 tx=0x7f440c0108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:39.615 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:39.582+0000 7f4420958700 1 --2- 172.21.15.42:0/3361635059 >> v2:172.21.15.42:3300/0 conn(0x7f441c11df60 0x7f441c145790 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f440c000b80 tx=0x7f440c0108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:39.616 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:39.582+0000 7f4420958700 1 --2- 172.21.15.42:0/3361635059 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f441c11df60 0x7f441c145790 secure :-1 s=READY pgs=2455 cs=0 l=1 rx=0x7f440c000b80 tx=0x7f440c0108f0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:19:39.616 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:39.582+0000 7f4420958700 1 --2- 172.21.15.42:0/3361635059 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f441c11df60 0x7f441c145790 secure :-1 s=THROTTLE_DONE pgs=2455 cs=0 l=1 rx=0x7f440c000b80 tx=0x7f440c0108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:39.616 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:39.582+0000 7f4411ffb700 1 -- 172.21.15.42:0/3361635059 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f440c00ca80 con 0x7f441c11df60 2019-12-05T06:19:39.616 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:39.582+0000 7f4420958700 1 --2- 172.21.15.42:0/3361635059 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f441c11df60 0x7f441c145790 secure :-1 s=THROTTLE_DONE pgs=2455 cs=0 l=1 rx=0x7f440c000b80 tx=0x7f440c0108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:39.616 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:39.582+0000 7f4411ffb700 1 -- 172.21.15.42:0/3361635059 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f440c00c6a0 con 0x7f441c11df60 2019-12-05T06:19:39.616 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:39.582+0000 7f4420958700 1 --2- 172.21.15.42:0/3361635059 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f441c11df60 0x7f441c145790 secure :-1 s=THROTTLE_DONE pgs=2455 cs=0 l=1 rx=0x7f440c000b80 tx=0x7f440c0108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:39.616 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:39.582+0000 7f4422bbc700 1 -- 172.21.15.42:0/3361635059 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f441c0a6530 con 0x7f441c11df60 2019-12-05T06:19:39.617 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:39.582+0000 7f4411ffb700 1 -- 172.21.15.42:0/3361635059 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f440c0110b0 con 0x7f441c11df60 2019-12-05T06:19:39.617 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:39.582+0000 7f4422bbc700 1 -- 172.21.15.42:0/3361635059 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f441c1474f0 con 0x7f441c11df60 2019-12-05T06:19:39.617 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:39.586+0000 7f4420958700 1 --2- 172.21.15.42:0/3361635059 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f441c11df60 0x7f441c145790 secure :-1 s=THROTTLE_DONE pgs=2455 cs=0 l=1 rx=0x7f440c000b80 tx=0x7f440c0108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:39.617 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:39.586+0000 7f4411ffb700 1 -- 172.21.15.42:0/3361635059 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f440c0113d0 con 0x7f441c11df60 2019-12-05T06:19:39.617 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:39.586+0000 7f4420958700 1 --2- 172.21.15.42:0/3361635059 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f441c11df60 0x7f441c145790 secure :-1 s=THROTTLE_DONE pgs=2455 cs=0 l=1 rx=0x7f440c000b80 tx=0x7f440c0108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:39.617 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:39.586+0000 7f4411ffb700 1 --2- 172.21.15.42:0/3361635059 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f43fc03b250 0x7f43fc03d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:19:39.617 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:39.586+0000 7f4411ffb700 1 -- 172.21.15.42:0/3361635059 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 10503+0+0 (secure 0 0 0) 0x7f440c04b190 con 0x7f441c11df60 2019-12-05T06:19:39.618 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:39.586+0000 7f4421159700 1 --2- 172.21.15.42:0/3361635059 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f43fc03b250 0x7f43fc03d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:39.618 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:39.586+0000 7f4421159700 1 --2- 172.21.15.42:0/3361635059 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f43fc03b250 0x7f43fc03d760 crc :-1 s=READY pgs=1031 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:19:39.618 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:39.586+0000 7f4422bbc700 1 -- 172.21.15.42:0/3361635059 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f441c05d110 con 0x7f441c11df60 2019-12-05T06:19:39.618 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:39.594+0000 7f4420958700 1 --2- 172.21.15.42:0/3361635059 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f441c11df60 0x7f441c145790 secure :-1 s=THROTTLE_DONE pgs=2455 cs=0 l=1 rx=0x7f440c000b80 tx=0x7f440c0108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:39.618 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:39.594+0000 7f4411ffb700 1 -- 172.21.15.42:0/3361635059 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f440c032190 con 0x7f441c11df60 2019-12-05T06:19:39.762 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:39.758+0000 7f4422bbc700 1 -- 172.21.15.42:0/3361635059 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f441c07e180 con 0x7f441c11df60 2019-12-05T06:19:39.769 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:39.766+0000 7f4420958700 1 --2- 172.21.15.42:0/3361635059 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f441c11df60 0x7f441c145790 secure :-1 s=THROTTLE_DONE pgs=2455 cs=0 l=1 rx=0x7f440c000b80 tx=0x7f440c0108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:39.769 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:39.766+0000 7f4411ffb700 1 -- 172.21.15.42:0/3361635059 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 293 v293) v1 ==== 95+0+2117 (secure 0 0 0) 0x7f440c0323f0 con 0x7f441c11df60 2019-12-05T06:19:39.770 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:19:39.770 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":293,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":10711,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/2638662429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2638662429},{"type":"v1","addr":"172.21.15.42:6835","nonce":2638662429}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":293}],"filesystems":[{"mdsmap":{"epoch":292,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:19:15.110045+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":101,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10476},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10476":{"gid":10476,"name":"b","rank":0,"incarnation":289,"state":"up:active","state_seq":34,"addr":"172.21.15.40:6833/1473620421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1473620421},{"type":"v1","addr":"172.21.15.40:6833","nonce":1473620421}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:19:39.770 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 293 2019-12-05T06:19:39.772 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:39.766+0000 7f4422bbc700 1 -- 172.21.15.42:0/3361635059 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f43fc03b250 msgr2=0x7f43fc03d760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:39.772 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:39.766+0000 7f4422bbc700 1 --2- 172.21.15.42:0/3361635059 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f43fc03b250 0x7f43fc03d760 crc :-1 s=READY pgs=1031 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:39.773 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:39.766+0000 7f4422bbc700 1 -- 172.21.15.42:0/3361635059 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f441c11df60 msgr2=0x7f441c145790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:39.773 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:39.766+0000 7f4422bbc700 1 --2- 172.21.15.42:0/3361635059 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f441c11df60 0x7f441c145790 secure :-1 s=READY pgs=2455 cs=0 l=1 rx=0x7f440c000b80 tx=0x7f440c0108f0).stop 2019-12-05T06:19:39.773 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:39.770+0000 7f4422bbc700 1 -- 172.21.15.42:0/3361635059 shutdown_connections 2019-12-05T06:19:39.773 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:39.770+0000 7f4422bbc700 1 --2- 172.21.15.42:0/3361635059 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f43fc03b250 0x7f43fc03d760 unknown :-1 s=CLOSED pgs=1031 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:39.773 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:39.770+0000 7f4422bbc700 1 --2- 172.21.15.42:0/3361635059 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f441c145ce0 0x7f441c134530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:39.773 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:39.770+0000 7f4422bbc700 1 --2- 172.21.15.42:0/3361635059 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f441c11df60 0x7f441c145790 unknown :-1 s=CLOSED pgs=2455 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:39.773 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:39.770+0000 7f4422bbc700 1 -- 172.21.15.42:0/3361635059 shutdown_connections 2019-12-05T06:19:39.774 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:39.770+0000 7f4422bbc700 1 -- 172.21.15.42:0/3361635059 wait complete. 2019-12-05T06:19:39.774 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:39.770+0000 7f4422bbc700 1 -- 172.21.15.42:0/3361635059 >> 172.21.15.42:0/3361635059 conn(0x7f441c059ef0 msgr2=0x7f441c11bfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:19:39.787 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:19:41.790 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:19:41.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:41.886+0000 7f8693538700 1 Processor -- start 2019-12-05T06:19:41.897 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:41.886+0000 7f8693538700 1 -- start start 2019-12-05T06:19:41.897 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:41.890+0000 7f8693538700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f868c0a7730 0x7f868c08df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:19:41.897 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:41.890+0000 7f8693538700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f868c09fad0 0x7f868c07eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:19:41.897 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:41.890+0000 7f8693538700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f868c0a0700 con 0x7f868c07e710 2019-12-05T06:19:41.897 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:41.890+0000 7f8693538700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f868c0a0af0 con 0x7f868c09fad0 2019-12-05T06:19:41.898 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:41.890+0000 7f8693538700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f868c08e5a0 con 0x7f868c0a7730 2019-12-05T06:19:41.898 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:41.890+0000 7f8691ad5700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f868c0a7730 0x7f868c08df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:41.898 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:41.890+0000 7f86922d6700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f868c09fad0 0x7f868c07eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:41.898 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:41.890+0000 7f8691ad5700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f868c0a7730 0x7f868c08df70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:53480/0 (socket says 172.21.15.42:53480) 2019-12-05T06:19:41.899 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:41.890+0000 7f86912d4700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f868c07e710 0x7f868c09f2d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:34262/0 (socket says 172.21.15.42:34262) 2019-12-05T06:19:41.899 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:41.890+0000 7f8691ad5700 1 -- 172.21.15.42:0/1368455170 learned_addr learned my addr 172.21.15.42:0/1368455170 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:19:41.899 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:41.890+0000 7f8690ad3700 1 -- 172.21.15.42:0/1368455170 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1257607690 0 0) 0x7f868c0a0700 con 0x7f868c07e710 2019-12-05T06:19:41.899 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:41.890+0000 7f8690ad3700 1 -- 172.21.15.42:0/1368455170 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86780030b0 con 0x7f868c07e710 2019-12-05T06:19:41.899 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:41.890+0000 7f8691ad5700 1 -- 172.21.15.42:0/1368455170 >> v2:172.21.15.40:3300/0 conn(0x7f868c09fad0 msgr2=0x7f868c07eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:19:41.899 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:41.890+0000 7f8691ad5700 1 --2- 172.21.15.42:0/1368455170 >> v2:172.21.15.40:3300/0 conn(0x7f868c09fad0 0x7f868c07eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:19:41.899 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:41.890+0000 7f8691ad5700 1 -- 172.21.15.42:0/1368455170 >> v1:172.21.15.40:6789/0 conn(0x7f868c07e710 legacy=0x7f868c09f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:41.899 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:41.890+0000 7f8691ad5700 1 -- 172.21.15.42:0/1368455170 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f868c07f090 con 0x7f868c0a7730 2019-12-05T06:19:41.900 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:41.890+0000 7f86922d6700 1 --2- 172.21.15.42:0/1368455170 >> v2:172.21.15.40:3300/0 conn(0x7f868c09fad0 0x7f868c07eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:19:41.900 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:41.890+0000 7f8691ad5700 1 --2- 172.21.15.42:0/1368455170 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f868c0a7730 0x7f868c08df70 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f8680008790 tx=0x7f868000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:41.900 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:41.890+0000 7f8691ad5700 1 --2- 172.21.15.42:0/1368455170 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f868c0a7730 0x7f868c08df70 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f8680008790 tx=0x7f868000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:41.900 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:41.890+0000 7f8691ad5700 1 --2- 172.21.15.42:0/1368455170 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f868c0a7730 0x7f868c08df70 secure :-1 s=READY pgs=538 cs=0 l=1 rx=0x7f8680008790 tx=0x7f868000fd30).ready entity=mon.2 client_cookie=89dd9c1f40f556eb server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:19:41.900 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:41.890+0000 7f8691ad5700 1 --2- 172.21.15.42:0/1368455170 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f868c0a7730 0x7f868c08df70 secure :-1 s=THROTTLE_DONE pgs=538 cs=0 l=1 rx=0x7f8680008790 tx=0x7f868000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:41.900 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:41.890+0000 7f8691ad5700 1 --2- 172.21.15.42:0/1368455170 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f868c0a7730 0x7f868c08df70 secure :-1 s=THROTTLE_DONE pgs=538 cs=0 l=1 rx=0x7f8680008790 tx=0x7f868000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:41.900 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:41.890+0000 7f8690ad3700 1 -- 172.21.15.42:0/1368455170 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8680008c90 con 0x7f868c0a7730 2019-12-05T06:19:41.901 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:41.890+0000 7f8691ad5700 1 --2- 172.21.15.42:0/1368455170 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f868c0a7730 0x7f868c08df70 secure :-1 s=THROTTLE_DONE pgs=538 cs=0 l=1 rx=0x7f8680008790 tx=0x7f868000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:41.901 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:41.890+0000 7f8690ad3700 1 -- 172.21.15.42:0/1368455170 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86800088b0 con 0x7f868c0a7730 2019-12-05T06:19:41.901 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:41.890+0000 7f8690ad3700 1 -- 172.21.15.42:0/1368455170 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8680011b10 con 0x7f868c0a7730 2019-12-05T06:19:41.901 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:41.890+0000 7f8693538700 1 -- 172.21.15.42:0/1368455170 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f868c0a7730 msgr2=0x7f868c08df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:41.901 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:41.890+0000 7f8693538700 1 --2- 172.21.15.42:0/1368455170 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f868c0a7730 0x7f868c08df70 secure :-1 s=READY pgs=538 cs=0 l=1 rx=0x7f8680008790 tx=0x7f868000fd30).stop 2019-12-05T06:19:41.901 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:41.890+0000 7f8693538700 1 -- 172.21.15.42:0/1368455170 shutdown_connections 2019-12-05T06:19:41.901 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:41.890+0000 7f8693538700 1 --2- 172.21.15.42:0/1368455170 >> v2:172.21.15.40:3300/0 conn(0x7f868c09fad0 0x7f868c07eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:19:41.902 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:41.890+0000 7f8693538700 1 --2- 172.21.15.42:0/1368455170 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f868c0a7730 0x7f868c08df70 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:41.902 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:41.890+0000 7f8693538700 1 -- 172.21.15.42:0/1368455170 shutdown_connections 2019-12-05T06:19:41.902 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:41.890+0000 7f8693538700 1 -- 172.21.15.42:0/1368455170 wait complete. 2019-12-05T06:19:41.902 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:41.890+0000 7f8693538700 1 -- 172.21.15.42:0/1368455170 >> 172.21.15.42:0/1368455170 conn(0x7f868c0aabd0 msgr2=0x7f868c0a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:19:41.905 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:41.894+0000 7f8693538700 1 Processor -- start 2019-12-05T06:19:41.906 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:41.894+0000 7f8693538700 1 -- start start 2019-12-05T06:19:41.907 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:41.894+0000 7f8693538700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f868c122d40 0x7f868c07d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:19:41.908 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:41.894+0000 7f8693538700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f868c07df40 0x7f868c1304b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:19:41.909 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:41.894+0000 7f8693538700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f868c1247f0 con 0x7f868c130a00 2019-12-05T06:19:41.909 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:41.894+0000 7f8693538700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f868c1216a0 con 0x7f868c07df40 2019-12-05T06:19:41.910 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:41.894+0000 7f8693538700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f868c121940 con 0x7f868c122d40 2019-12-05T06:19:41.910 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:41.894+0000 7f8691ad5700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f868c122d40 0x7f868c07d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:41.910 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:41.894+0000 7f86922d6700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f868c130a00 0x7f868c118f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:53060/0 (socket says 172.21.15.42:53060) 2019-12-05T06:19:41.910 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:41.894+0000 7f86922d6700 1 -- 172.21.15.42:0/2798663259 learned_addr learned my addr 172.21.15.42:0/2798663259 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:19:41.911 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:41.894+0000 7f86912d4700 1 --2- 172.21.15.42:0/2798663259 >> v2:172.21.15.40:3300/0 conn(0x7f868c07df40 0x7f868c1304b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:41.911 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:41.894+0000 7f8691ad5700 1 -- 172.21.15.42:0/2798663259 >> v2:172.21.15.40:3300/0 conn(0x7f868c07df40 msgr2=0x7f868c1304b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:41.911 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:41.894+0000 7f8691ad5700 1 --2- 172.21.15.42:0/2798663259 >> v2:172.21.15.40:3300/0 conn(0x7f868c07df40 0x7f868c1304b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:41.911 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:41.894+0000 7f8691ad5700 1 -- 172.21.15.42:0/2798663259 >> v1:172.21.15.42:6789/0 conn(0x7f868c130a00 legacy=0x7f868c118f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:41.911 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:41.894+0000 7f867e7fc700 1 -- 172.21.15.42:0/2798663259 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2811967106 0 0) 0x7f868c1247f0 con 0x7f868c130a00 2019-12-05T06:19:41.911 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:41.894+0000 7f8691ad5700 1 -- 172.21.15.42:0/2798663259 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f868c130e30 con 0x7f868c122d40 2019-12-05T06:19:41.911 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:41.894+0000 7f8691ad5700 1 --2- 172.21.15.42:0/2798663259 >> v2:172.21.15.42:3300/0 conn(0x7f868c122d40 0x7f868c07d9f0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f8680008790 tx=0x7f8680008930).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:41.912 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:41.894+0000 7f8691ad5700 1 --2- 172.21.15.42:0/2798663259 >> v2:172.21.15.42:3300/0 conn(0x7f868c122d40 0x7f868c07d9f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f8680008790 tx=0x7f8680008930).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:41.912 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:41.894+0000 7f8691ad5700 1 --2- 172.21.15.42:0/2798663259 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f868c122d40 0x7f868c07d9f0 secure :-1 s=READY pgs=2457 cs=0 l=1 rx=0x7f8680008790 tx=0x7f8680008930).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:19:41.912 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:41.898+0000 7f8691ad5700 1 --2- 172.21.15.42:0/2798663259 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f868c122d40 0x7f868c07d9f0 secure :-1 s=THROTTLE_DONE pgs=2457 cs=0 l=1 rx=0x7f8680008790 tx=0x7f8680008930).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:41.912 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:41.898+0000 7f867e7fc700 1 -- 172.21.15.42:0/2798663259 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f868000f280 con 0x7f868c122d40 2019-12-05T06:19:41.912 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:41.898+0000 7f8691ad5700 1 --2- 172.21.15.42:0/2798663259 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f868c122d40 0x7f868c07d9f0 secure :-1 s=THROTTLE_DONE pgs=2457 cs=0 l=1 rx=0x7f8680008790 tx=0x7f8680008930).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:41.912 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:41.898+0000 7f8693538700 1 -- 172.21.15.42:0/2798663259 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f868c121270 con 0x7f868c122d40 2019-12-05T06:19:41.912 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:41.898+0000 7f867e7fc700 1 -- 172.21.15.42:0/2798663259 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8680011730 con 0x7f868c122d40 2019-12-05T06:19:41.913 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:41.898+0000 7f8691ad5700 1 --2- 172.21.15.42:0/2798663259 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f868c122d40 0x7f868c07d9f0 secure :-1 s=THROTTLE_DONE pgs=2457 cs=0 l=1 rx=0x7f8680008790 tx=0x7f8680008930).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:41.913 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:41.898+0000 7f867e7fc700 1 -- 172.21.15.42:0/2798663259 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8680005540 con 0x7f868c122d40 2019-12-05T06:19:41.913 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:41.898+0000 7f8693538700 1 -- 172.21.15.42:0/2798663259 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f868c1221f0 con 0x7f868c122d40 2019-12-05T06:19:41.913 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:41.898+0000 7f8691ad5700 1 --2- 172.21.15.42:0/2798663259 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f868c122d40 0x7f868c07d9f0 secure :-1 s=THROTTLE_DONE pgs=2457 cs=0 l=1 rx=0x7f8680008790 tx=0x7f8680008930).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:41.913 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:41.898+0000 7f867e7fc700 1 -- 172.21.15.42:0/2798663259 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f8680005740 con 0x7f868c122d40 2019-12-05T06:19:41.913 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:41.898+0000 7f8691ad5700 1 --2- 172.21.15.42:0/2798663259 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f868c122d40 0x7f868c07d9f0 secure :-1 s=THROTTLE_DONE pgs=2457 cs=0 l=1 rx=0x7f8680008790 tx=0x7f8680008930).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:41.913 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:41.898+0000 7f867e7fc700 1 --2- 172.21.15.42:0/2798663259 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f867403ac60 0x7f867403d170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:19:41.914 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:41.898+0000 7f867e7fc700 1 -- 172.21.15.42:0/2798663259 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 10503+0+0 (secure 0 0 0) 0x7f868004b2c0 con 0x7f868c122d40 2019-12-05T06:19:41.914 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:41.898+0000 7f86912d4700 1 --2- 172.21.15.42:0/2798663259 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f867403ac60 0x7f867403d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:41.914 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:41.898+0000 7f86912d4700 1 --2- 172.21.15.42:0/2798663259 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f867403ac60 0x7f867403d170 crc :-1 s=READY pgs=1032 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:19:41.914 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:41.902+0000 7f8693538700 1 -- 172.21.15.42:0/2798663259 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f868c05d100 con 0x7f868c122d40 2019-12-05T06:19:41.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:41.914+0000 7f8691ad5700 1 --2- 172.21.15.42:0/2798663259 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f868c122d40 0x7f868c07d9f0 secure :-1 s=THROTTLE_DONE pgs=2457 cs=0 l=1 rx=0x7f8680008790 tx=0x7f8680008930).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:41.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:41.914+0000 7f867e7fc700 1 -- 172.21.15.42:0/2798663259 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f8680032320 con 0x7f868c122d40 2019-12-05T06:19:42.088 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:42.086+0000 7f8693538700 1 -- 172.21.15.42:0/2798663259 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f868c05b8b0 con 0x7f868c122d40 2019-12-05T06:19:42.089 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:42.086+0000 7f8691ad5700 1 --2- 172.21.15.42:0/2798663259 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f868c122d40 0x7f868c07d9f0 secure :-1 s=THROTTLE_DONE pgs=2457 cs=0 l=1 rx=0x7f8680008790 tx=0x7f8680008930).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:42.089 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:42.086+0000 7f867e7fc700 1 -- 172.21.15.42:0/2798663259 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 293 v293) v1 ==== 95+0+2117 (secure 0 0 0) 0x7f8680032580 con 0x7f868c122d40 2019-12-05T06:19:42.089 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 293 2019-12-05T06:19:42.089 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:19:42.089 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":293,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":10711,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/2638662429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2638662429},{"type":"v1","addr":"172.21.15.42:6835","nonce":2638662429}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":293}],"filesystems":[{"mdsmap":{"epoch":292,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:19:15.110045+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":101,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10476},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10476":{"gid":10476,"name":"b","rank":0,"incarnation":289,"state":"up:active","state_seq":34,"addr":"172.21.15.40:6833/1473620421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1473620421},{"type":"v1","addr":"172.21.15.40:6833","nonce":1473620421}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:19:42.092 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:42.086+0000 7f8693538700 1 -- 172.21.15.42:0/2798663259 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f867403ac60 msgr2=0x7f867403d170 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:42.092 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:42.086+0000 7f8693538700 1 --2- 172.21.15.42:0/2798663259 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f867403ac60 0x7f867403d170 crc :-1 s=READY pgs=1032 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:42.092 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:42.086+0000 7f8693538700 1 -- 172.21.15.42:0/2798663259 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f868c122d40 msgr2=0x7f868c07d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:42.092 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:42.086+0000 7f8693538700 1 --2- 172.21.15.42:0/2798663259 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f868c122d40 0x7f868c07d9f0 secure :-1 s=READY pgs=2457 cs=0 l=1 rx=0x7f8680008790 tx=0x7f8680008930).stop 2019-12-05T06:19:42.092 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:42.086+0000 7f8693538700 1 -- 172.21.15.42:0/2798663259 shutdown_connections 2019-12-05T06:19:42.093 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:42.086+0000 7f8693538700 1 --2- 172.21.15.42:0/2798663259 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f867403ac60 0x7f867403d170 unknown :-1 s=CLOSED pgs=1032 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:42.093 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:42.086+0000 7f8693538700 1 --2- 172.21.15.42:0/2798663259 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f868c122d40 0x7f868c07d9f0 unknown :-1 s=CLOSED pgs=2457 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:42.093 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:42.086+0000 7f8693538700 1 --2- 172.21.15.42:0/2798663259 >> v2:172.21.15.40:3300/0 conn(0x7f868c07df40 0x7f868c1304b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:42.093 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:42.086+0000 7f8693538700 1 -- 172.21.15.42:0/2798663259 shutdown_connections 2019-12-05T06:19:42.093 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:42.086+0000 7f8693538700 1 -- 172.21.15.42:0/2798663259 wait complete. 2019-12-05T06:19:42.093 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:42.086+0000 7f8693538700 1 -- 172.21.15.42:0/2798663259 >> 172.21.15.42:0/2798663259 conn(0x7f868c0aabd0 msgr2=0x7f868c0a9780 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:19:42.102 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:19:44.103 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:19:44.208 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:44.198+0000 7f7987e45700 1 Processor -- start 2019-12-05T06:19:44.208 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:44.198+0000 7f7987e45700 1 -- start start 2019-12-05T06:19:44.209 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:44.202+0000 7f7987e45700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f79800a7740 0x7f798008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:19:44.209 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:44.202+0000 7f7987e45700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f798007e720 0x7f798007eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:19:44.209 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:44.202+0000 7f7987e45700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79800a0710 con 0x7f798009f2e0 2019-12-05T06:19:44.209 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:44.202+0000 7f7987e45700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79800a0b00 con 0x7f798007e720 2019-12-05T06:19:44.210 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:44.202+0000 7f7987e45700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f798008e5b0 con 0x7f79800a7740 2019-12-05T06:19:44.210 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:44.202+0000 7f7985be1700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f798007e720 0x7f798007eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:44.210 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:44.202+0000 7f7986be3700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f798009f2e0 0x7f798009f710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:53070/0 (socket says 172.21.15.42:53070) 2019-12-05T06:19:44.210 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:44.202+0000 7f7985be1700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f798007e720 0x7f798007eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:53032/0 (socket says 172.21.15.42:53032) 2019-12-05T06:19:44.210 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:44.202+0000 7f7986be3700 1 -- 172.21.15.42:0/41845448 learned_addr learned my addr 172.21.15.42:0/41845448 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:19:44.211 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:44.202+0000 7f79863e2700 1 --2- 172.21.15.42:0/41845448 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f79800a7740 0x7f798008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:44.211 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:44.202+0000 7f7985be1700 1 -- 172.21.15.42:0/41845448 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f79800a7740 msgr2=0x7f798008df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:19:44.211 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:44.202+0000 7f7985be1700 1 --2- 172.21.15.42:0/41845448 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f79800a7740 0x7f798008df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:19:44.211 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:44.202+0000 7f7985be1700 1 -- 172.21.15.42:0/41845448 >> v1:172.21.15.42:6789/0 conn(0x7f798009f2e0 legacy=0x7f798009f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:44.211 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:44.202+0000 7f79853e0700 1 -- 172.21.15.42:0/41845448 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2074083359 0 0) 0x7f79800a0710 con 0x7f798009f2e0 2019-12-05T06:19:44.211 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:44.202+0000 7f7985be1700 1 -- 172.21.15.42:0/41845448 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f798007f0a0 con 0x7f798007e720 2019-12-05T06:19:44.211 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:44.202+0000 7f7985be1700 1 --2- 172.21.15.42:0/41845448 >> v2:172.21.15.42:3300/0 conn(0x7f798007e720 0x7f798007eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f7970008790 tx=0x7f797000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:44.212 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:44.202+0000 7f7985be1700 1 --2- 172.21.15.42:0/41845448 >> v2:172.21.15.42:3300/0 conn(0x7f798007e720 0x7f798007eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f7970008790 tx=0x7f797000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:44.212 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:44.202+0000 7f7985be1700 1 --2- 172.21.15.42:0/41845448 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f798007e720 0x7f798007eb50 secure :-1 s=READY pgs=2459 cs=0 l=1 rx=0x7f7970008790 tx=0x7f797000fc70).ready entity=mon.1 client_cookie=44af615419816b55 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:19:44.212 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:44.202+0000 7f7985be1700 1 --2- 172.21.15.42:0/41845448 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f798007e720 0x7f798007eb50 secure :-1 s=THROTTLE_DONE pgs=2459 cs=0 l=1 rx=0x7f7970008790 tx=0x7f797000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:44.212 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:44.202+0000 7f7985be1700 1 --2- 172.21.15.42:0/41845448 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f798007e720 0x7f798007eb50 secure :-1 s=THROTTLE_DONE pgs=2459 cs=0 l=1 rx=0x7f7970008790 tx=0x7f797000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:44.212 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:44.202+0000 7f79853e0700 1 -- 172.21.15.42:0/41845448 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f79700112f0 con 0x7f798007e720 2019-12-05T06:19:44.212 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:44.202+0000 7f7985be1700 1 --2- 172.21.15.42:0/41845448 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f798007e720 0x7f798007eb50 secure :-1 s=THROTTLE_DONE pgs=2459 cs=0 l=1 rx=0x7f7970008790 tx=0x7f797000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:44.213 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:44.202+0000 7f79853e0700 1 -- 172.21.15.42:0/41845448 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7970008ad0 con 0x7f798007e720 2019-12-05T06:19:44.213 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:44.202+0000 7f79853e0700 1 -- 172.21.15.42:0/41845448 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f79700118d0 con 0x7f798007e720 2019-12-05T06:19:44.213 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:44.202+0000 7f7987e45700 1 -- 172.21.15.42:0/41845448 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f798007e720 msgr2=0x7f798007eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:44.213 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:44.202+0000 7f7987e45700 1 --2- 172.21.15.42:0/41845448 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f798007e720 0x7f798007eb50 secure :-1 s=READY pgs=2459 cs=0 l=1 rx=0x7f7970008790 tx=0x7f797000fc70).stop 2019-12-05T06:19:44.213 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:44.202+0000 7f7987e45700 1 -- 172.21.15.42:0/41845448 shutdown_connections 2019-12-05T06:19:44.213 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:44.202+0000 7f7987e45700 1 --2- 172.21.15.42:0/41845448 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f79800a7740 0x7f798008df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:19:44.214 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:44.202+0000 7f7987e45700 1 --2- 172.21.15.42:0/41845448 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f798007e720 0x7f798007eb50 unknown :-1 s=CLOSED pgs=2459 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:44.214 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:44.202+0000 7f7987e45700 1 -- 172.21.15.42:0/41845448 shutdown_connections 2019-12-05T06:19:44.214 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:44.202+0000 7f7987e45700 1 -- 172.21.15.42:0/41845448 wait complete. 2019-12-05T06:19:44.214 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:44.202+0000 7f7987e45700 1 -- 172.21.15.42:0/41845448 >> 172.21.15.42:0/41845448 conn(0x7f79800aabe0 msgr2=0x7f79800a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:19:44.220 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:44.206+0000 7f7987e45700 1 Processor -- start 2019-12-05T06:19:44.220 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:44.206+0000 7f7987e45700 1 -- start start 2019-12-05T06:19:44.220 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:44.206+0000 7f7987e45700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f7980121470 0x7f7980144c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:19:44.220 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:44.206+0000 7f7987e45700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7980145180 0x7f79801231f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:19:44.221 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:44.206+0000 7f7987e45700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7980131290 con 0x7f7980123740 2019-12-05T06:19:44.221 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:44.206+0000 7f7987e45700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79801218a0 con 0x7f7980121470 2019-12-05T06:19:44.221 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:44.206+0000 7f7987e45700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7980144990 con 0x7f7980145180 2019-12-05T06:19:44.221 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:44.206+0000 7f79863e2700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f7980121470 0x7f7980144c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:44.222 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:44.206+0000 7f7986be3700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f7980123740 0x7f798008eb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:53078/0 (socket says 172.21.15.42:53078) 2019-12-05T06:19:44.222 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:44.206+0000 7f7986be3700 1 -- 172.21.15.42:0/993920272 learned_addr learned my addr 172.21.15.42:0/993920272 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:19:44.222 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:44.206+0000 7f7985be1700 1 --2- 172.21.15.42:0/993920272 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7980145180 0x7f79801231f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:44.222 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:44.206+0000 7f79863e2700 1 -- 172.21.15.42:0/993920272 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7980145180 msgr2=0x7f79801231f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:44.222 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:44.206+0000 7f79863e2700 1 --2- 172.21.15.42:0/993920272 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7980145180 0x7f79801231f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:44.222 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:44.206+0000 7f7976ffd700 1 -- 172.21.15.42:0/993920272 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3811971643 0 0) 0x7f7980131290 con 0x7f7980123740 2019-12-05T06:19:44.222 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:44.206+0000 7f79863e2700 1 -- 172.21.15.42:0/993920272 >> v1:172.21.15.42:6789/0 conn(0x7f7980123740 legacy=0x7f798008eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:44.223 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:44.206+0000 7f79863e2700 1 -- 172.21.15.42:0/993920272 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7980120990 con 0x7f7980121470 2019-12-05T06:19:44.223 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:44.206+0000 7f79863e2700 1 --2- 172.21.15.42:0/993920272 >> v2:172.21.15.42:3300/0 conn(0x7f7980121470 0x7f7980144c30 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f797c002f30 tx=0x7f797c0109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:44.223 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:44.206+0000 7f79863e2700 1 --2- 172.21.15.42:0/993920272 >> v2:172.21.15.42:3300/0 conn(0x7f7980121470 0x7f7980144c30 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f797c002f30 tx=0x7f797c0109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:44.223 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:44.206+0000 7f79863e2700 1 --2- 172.21.15.42:0/993920272 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7980121470 0x7f7980144c30 secure :-1 s=READY pgs=2461 cs=0 l=1 rx=0x7f797c002f30 tx=0x7f797c0109e0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:19:44.223 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:44.206+0000 7f79863e2700 1 --2- 172.21.15.42:0/993920272 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7980121470 0x7f7980144c30 secure :-1 s=THROTTLE_DONE pgs=2461 cs=0 l=1 rx=0x7f797c002f30 tx=0x7f797c0109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:44.223 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:44.206+0000 7f7976ffd700 1 -- 172.21.15.42:0/993920272 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f797c005920 con 0x7f7980121470 2019-12-05T06:19:44.223 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:44.206+0000 7f79863e2700 1 --2- 172.21.15.42:0/993920272 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7980121470 0x7f7980144c30 secure :-1 s=THROTTLE_DONE pgs=2461 cs=0 l=1 rx=0x7f797c002f30 tx=0x7f797c0109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:44.224 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:44.206+0000 7f79863e2700 1 --2- 172.21.15.42:0/993920272 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7980121470 0x7f7980144c30 secure :-1 s=THROTTLE_DONE pgs=2461 cs=0 l=1 rx=0x7f797c002f30 tx=0x7f797c0109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:44.224 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:44.206+0000 7f7976ffd700 1 -- 172.21.15.42:0/993920272 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f797c005540 con 0x7f7980121470 2019-12-05T06:19:44.224 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:44.206+0000 7f7987e45700 1 -- 172.21.15.42:0/993920272 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7980123b70 con 0x7f7980121470 2019-12-05T06:19:44.224 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:44.206+0000 7f7976ffd700 1 -- 172.21.15.42:0/993920272 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f797c012490 con 0x7f7980121470 2019-12-05T06:19:44.224 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:44.206+0000 7f7987e45700 1 -- 172.21.15.42:0/993920272 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f798008f370 con 0x7f7980121470 2019-12-05T06:19:44.224 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:44.210+0000 7f79863e2700 1 --2- 172.21.15.42:0/993920272 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7980121470 0x7f7980144c30 secure :-1 s=THROTTLE_DONE pgs=2461 cs=0 l=1 rx=0x7f797c002f30 tx=0x7f797c0109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:44.224 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:44.210+0000 7f7976ffd700 1 -- 172.21.15.42:0/993920272 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f797c007180 con 0x7f7980121470 2019-12-05T06:19:44.225 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:44.210+0000 7f79863e2700 1 --2- 172.21.15.42:0/993920272 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7980121470 0x7f7980144c30 secure :-1 s=THROTTLE_DONE pgs=2461 cs=0 l=1 rx=0x7f797c002f30 tx=0x7f797c0109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:44.225 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:44.210+0000 7f7976ffd700 1 --2- 172.21.15.42:0/993920272 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f796803ab80 0x7f796803d090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:19:44.225 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:44.210+0000 7f7976ffd700 1 -- 172.21.15.42:0/993920272 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 10503+0+0 (secure 0 0 0) 0x7f797c04ad90 con 0x7f7980121470 2019-12-05T06:19:44.225 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:44.210+0000 7f7985be1700 1 --2- 172.21.15.42:0/993920272 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f796803ab80 0x7f796803d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:44.225 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:44.210+0000 7f7985be1700 1 --2- 172.21.15.42:0/993920272 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f796803ab80 0x7f796803d090 crc :-1 s=READY pgs=1033 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:19:44.225 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:44.214+0000 7f7987e45700 1 -- 172.21.15.42:0/993920272 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f798005d110 con 0x7f7980121470 2019-12-05T06:19:44.226 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:44.218+0000 7f79863e2700 1 --2- 172.21.15.42:0/993920272 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7980121470 0x7f7980144c30 secure :-1 s=THROTTLE_DONE pgs=2461 cs=0 l=1 rx=0x7f797c002f30 tx=0x7f797c0109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:44.226 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:44.218+0000 7f7976ffd700 1 -- 172.21.15.42:0/993920272 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f797c031df0 con 0x7f7980121470 2019-12-05T06:19:44.390 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:44.386+0000 7f7987e45700 1 -- 172.21.15.42:0/993920272 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f798005b8c0 con 0x7f7980121470 2019-12-05T06:19:44.391 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:44.386+0000 7f79863e2700 1 --2- 172.21.15.42:0/993920272 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7980121470 0x7f7980144c30 secure :-1 s=THROTTLE_DONE pgs=2461 cs=0 l=1 rx=0x7f797c002f30 tx=0x7f797c0109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:44.391 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:44.386+0000 7f7976ffd700 1 -- 172.21.15.42:0/993920272 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 293 v293) v1 ==== 95+0+2117 (secure 0 0 0) 0x7f797c032050 con 0x7f7980121470 2019-12-05T06:19:44.391 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 293 2019-12-05T06:19:44.391 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:19:44.392 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":293,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":10711,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/2638662429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2638662429},{"type":"v1","addr":"172.21.15.42:6835","nonce":2638662429}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":293}],"filesystems":[{"mdsmap":{"epoch":292,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:19:15.110045+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":101,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10476},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10476":{"gid":10476,"name":"b","rank":0,"incarnation":289,"state":"up:active","state_seq":34,"addr":"172.21.15.40:6833/1473620421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1473620421},{"type":"v1","addr":"172.21.15.40:6833","nonce":1473620421}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:19:44.394 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:44.390+0000 7f7987e45700 1 -- 172.21.15.42:0/993920272 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f796803ab80 msgr2=0x7f796803d090 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:44.394 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:44.390+0000 7f7987e45700 1 --2- 172.21.15.42:0/993920272 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f796803ab80 0x7f796803d090 crc :-1 s=READY pgs=1033 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:44.394 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:44.390+0000 7f7987e45700 1 -- 172.21.15.42:0/993920272 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7980121470 msgr2=0x7f7980144c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:44.394 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:44.390+0000 7f7987e45700 1 --2- 172.21.15.42:0/993920272 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7980121470 0x7f7980144c30 secure :-1 s=READY pgs=2461 cs=0 l=1 rx=0x7f797c002f30 tx=0x7f797c0109e0).stop 2019-12-05T06:19:44.394 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:44.390+0000 7f7987e45700 1 -- 172.21.15.42:0/993920272 shutdown_connections 2019-12-05T06:19:44.394 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:44.390+0000 7f7987e45700 1 --2- 172.21.15.42:0/993920272 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f796803ab80 0x7f796803d090 unknown :-1 s=CLOSED pgs=1033 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:44.394 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:44.390+0000 7f7987e45700 1 --2- 172.21.15.42:0/993920272 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7980121470 0x7f7980144c30 unknown :-1 s=CLOSED pgs=2461 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:44.395 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:44.390+0000 7f7987e45700 1 --2- 172.21.15.42:0/993920272 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7980145180 0x7f79801231f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:44.395 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:44.390+0000 7f7987e45700 1 -- 172.21.15.42:0/993920272 shutdown_connections 2019-12-05T06:19:44.395 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:44.390+0000 7f7987e45700 1 -- 172.21.15.42:0/993920272 wait complete. 2019-12-05T06:19:44.395 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:44.390+0000 7f7987e45700 1 -- 172.21.15.42:0/993920272 >> 172.21.15.42:0/993920272 conn(0x7f79800a6bb0 msgr2=0x7f798011c470 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:19:44.403 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:19:45.802 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:19:45.805 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:19:46.405 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:19:46.524 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:46.514+0000 7f506861d700 1 Processor -- start 2019-12-05T06:19:46.524 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:46.514+0000 7f506861d700 1 -- start start 2019-12-05T06:19:46.524 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:46.514+0000 7f506861d700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f506009fae0 0x7f506014fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:19:46.525 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:46.514+0000 7f506861d700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50600a0710 con 0x7f50600a7740 2019-12-05T06:19:46.525 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:46.514+0000 7f506861d700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f506007ef20 con 0x7f506008df80 2019-12-05T06:19:46.525 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:46.514+0000 7f506861d700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50600a0b00 con 0x7f506009fae0 2019-12-05T06:19:46.525 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:46.514+0000 7f50663b9700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f506008df80 0x7f506009f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:53084/0 (socket says 172.21.15.42:53084) 2019-12-05T06:19:46.525 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:46.514+0000 7f50663b9700 1 -- 172.21.15.42:0/934258479 learned_addr learned my addr 172.21.15.42:0/934258479 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:19:46.525 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:46.514+0000 7f50673bb700 1 --2- 172.21.15.42:0/934258479 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f506009fae0 0x7f506014fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:46.525 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:46.514+0000 7f5065bb8700 1 -- 172.21.15.42:0/934258479 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1177969991 0 0) 0x7f506007ef20 con 0x7f506008df80 2019-12-05T06:19:46.526 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:46.514+0000 7f5065bb8700 1 -- 172.21.15.42:0/934258479 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f504c0030b0 con 0x7f506008df80 2019-12-05T06:19:46.526 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:46.514+0000 7f5065bb8700 1 -- 172.21.15.42:0/934258479 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3759169747 0 0) 0x7f50600a0710 con 0x7f50600a7740 2019-12-05T06:19:46.526 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:46.514+0000 7f5065bb8700 1 -- 172.21.15.42:0/934258479 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f506007ef20 con 0x7f50600a7740 2019-12-05T06:19:46.526 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:46.514+0000 7f5065bb8700 1 -- 172.21.15.42:0/934258479 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3353866323 0 0) 0x7f504c0030b0 con 0x7f506008df80 2019-12-05T06:19:46.526 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:46.514+0000 7f5065bb8700 1 -- 172.21.15.42:0/934258479 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f506009fae0 msgr2=0x7f506014fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:19:46.526 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:46.514+0000 7f5065bb8700 1 --2- 172.21.15.42:0/934258479 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f506009fae0 0x7f506014fff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:19:46.526 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:46.514+0000 7f5065bb8700 1 -- 172.21.15.42:0/934258479 >> v1:172.21.15.40:6789/0 conn(0x7f50600a7740 legacy=0x7f506007e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:46.527 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:46.514+0000 7f5065bb8700 1 -- 172.21.15.42:0/934258479 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50600a9ba0 con 0x7f506008df80 2019-12-05T06:19:46.527 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:46.514+0000 7f5065bb8700 1 -- 172.21.15.42:0/934258479 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7f5050001750 con 0x7f506008df80 2019-12-05T06:19:46.527 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:46.514+0000 7f50673bb700 1 --2- 172.21.15.42:0/934258479 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f506009fae0 0x7f506014fff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:19:46.527 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:46.514+0000 7f5065bb8700 1 -- 172.21.15.42:0/934258479 >> v1:172.21.15.42:6789/0 conn(0x7f506008df80 legacy=0x7f506009f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:46.527 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:46.514+0000 7f5065bb8700 1 --2- 172.21.15.42:0/934258479 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f504c003940 0x7f504c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:19:46.527 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:46.514+0000 7f5065bb8700 1 --2- 172.21.15.42:0/934258479 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f504c004420 0x7f504c004870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:19:46.527 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:46.514+0000 7f5065bb8700 1 --2- 172.21.15.42:0/934258479 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f504c004e30 0x7f504c04f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:19:46.528 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:46.514+0000 7f5065bb8700 1 -- 172.21.15.42:0/934258479 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50600a0b00 con 0x7f504c003940 2019-12-05T06:19:46.528 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:46.514+0000 7f5065bb8700 1 -- 172.21.15.42:0/934258479 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f504c04fc60 con 0x7f504c004e30 2019-12-05T06:19:46.528 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:46.514+0000 7f5065bb8700 1 -- 172.21.15.42:0/934258479 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f504c04fea0 con 0x7f504c004420 2019-12-05T06:19:46.528 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:46.514+0000 7f5066bba700 1 --2- 172.21.15.42:0/934258479 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f504c003940 0x7f504c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:46.528 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:46.514+0000 7f50673bb700 1 --2- 172.21.15.42:0/934258479 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f504c004e30 0x7f504c04f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:46.528 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:46.514+0000 7f50663b9700 1 --2- 172.21.15.42:0/934258479 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f504c004420 0x7f504c004870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:46.528 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:46.518+0000 7f50673bb700 1 -- 172.21.15.42:0/934258479 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f504c004420 msgr2=0x7f504c004870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:19:46.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:46.518+0000 7f50673bb700 1 --2- 172.21.15.42:0/934258479 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f504c004420 0x7f504c004870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:19:46.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:46.518+0000 7f50673bb700 1 -- 172.21.15.42:0/934258479 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f504c003940 msgr2=0x7f504c003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:19:46.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:46.518+0000 7f50673bb700 1 --2- 172.21.15.42:0/934258479 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f504c003940 0x7f504c003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:19:46.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:46.518+0000 7f50673bb700 1 -- 172.21.15.42:0/934258479 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f504c0501a0 con 0x7f504c004e30 2019-12-05T06:19:46.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:46.518+0000 7f5066bba700 1 --2- 172.21.15.42:0/934258479 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f504c003940 0x7f504c003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:19:46.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:46.518+0000 7f50673bb700 1 --2- 172.21.15.42:0/934258479 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f504c004e30 0x7f504c04f680 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f504c0053f0 tx=0x7f505800be30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:46.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:46.518+0000 7f50673bb700 1 --2- 172.21.15.42:0/934258479 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f504c004e30 0x7f504c04f680 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f504c0053f0 tx=0x7f505800be30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:46.530 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:46.518+0000 7f50673bb700 1 --2- 172.21.15.42:0/934258479 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f504c004e30 0x7f504c04f680 secure :-1 s=READY pgs=2463 cs=0 l=1 rx=0x7f504c0053f0 tx=0x7f505800be30).ready entity=mon.1 client_cookie=1359c01d8540f687 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:19:46.530 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:46.518+0000 7f50673bb700 1 --2- 172.21.15.42:0/934258479 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f504c004e30 0x7f504c04f680 secure :-1 s=THROTTLE_DONE pgs=2463 cs=0 l=1 rx=0x7f504c0053f0 tx=0x7f505800be30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:46.530 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:46.518+0000 7f5065bb8700 1 -- 172.21.15.42:0/934258479 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f505800e7f0 con 0x7f504c004e30 2019-12-05T06:19:46.530 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:46.518+0000 7f50673bb700 1 --2- 172.21.15.42:0/934258479 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f504c004e30 0x7f504c04f680 secure :-1 s=THROTTLE_DONE pgs=2463 cs=0 l=1 rx=0x7f504c0053f0 tx=0x7f505800be30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:46.530 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:46.518+0000 7f50673bb700 1 --2- 172.21.15.42:0/934258479 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f504c004e30 0x7f504c04f680 secure :-1 s=THROTTLE_DONE pgs=2463 cs=0 l=1 rx=0x7f504c0053f0 tx=0x7f505800be30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:46.530 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:46.518+0000 7f5065bb8700 1 -- 172.21.15.42:0/934258479 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5058010200 con 0x7f504c004e30 2019-12-05T06:19:46.530 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:46.518+0000 7f5065bb8700 1 -- 172.21.15.42:0/934258479 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f505800edf0 con 0x7f504c004e30 2019-12-05T06:19:46.531 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:46.518+0000 7f506861d700 1 -- 172.21.15.42:0/934258479 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f504c004e30 msgr2=0x7f504c04f680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:46.531 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:46.518+0000 7f506861d700 1 --2- 172.21.15.42:0/934258479 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f504c004e30 0x7f504c04f680 secure :-1 s=READY pgs=2463 cs=0 l=1 rx=0x7f504c0053f0 tx=0x7f505800be30).stop 2019-12-05T06:19:46.531 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:46.518+0000 7f50673bb700 1 -- 172.21.15.42:0/934258479 reap_dead start 2019-12-05T06:19:46.531 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:46.518+0000 7f506861d700 1 -- 172.21.15.42:0/934258479 shutdown_connections 2019-12-05T06:19:46.531 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:46.518+0000 7f506861d700 1 -- 172.21.15.42:0/934258479 shutdown_connections 2019-12-05T06:19:46.531 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:46.518+0000 7f506861d700 1 -- 172.21.15.42:0/934258479 wait complete. 2019-12-05T06:19:46.531 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:46.518+0000 7f506861d700 1 -- 172.21.15.42:0/934258479 >> 172.21.15.42:0/934258479 conn(0x7f50600aabe0 msgr2=0x7f50600a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:19:46.535 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:46.522+0000 7f506861d700 1 Processor -- start 2019-12-05T06:19:46.535 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:46.522+0000 7f506861d700 1 -- start start 2019-12-05T06:19:46.535 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:46.522+0000 7f506861d700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f50600a5d20 0x7f50600a27a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:19:46.535 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:46.522+0000 7f506861d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f50601446f0 0x7f506007c020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:19:46.535 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:46.522+0000 7f506861d700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5060144e00 con 0x7f506007bbf0 2019-12-05T06:19:46.535 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:46.522+0000 7f506861d700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50600a2e00 con 0x7f50601446f0 2019-12-05T06:19:46.535 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:46.522+0000 7f506861d700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50600a30a0 con 0x7f50600a5d20 2019-12-05T06:19:46.536 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:46.522+0000 7f50673bb700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f50601446f0 0x7f506007c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:46.536 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:46.522+0000 7f50673bb700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f50601446f0 0x7f506007c020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:53058/0 (socket says 172.21.15.42:53058) 2019-12-05T06:19:46.536 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:46.522+0000 7f50673bb700 1 -- 172.21.15.42:0/2658131187 learned_addr learned my addr 172.21.15.42:0/2658131187 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:19:46.536 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:46.522+0000 7f5066bba700 1 --2- 172.21.15.42:0/2658131187 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f50600a5d20 0x7f50600a27a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:46.536 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:46.522+0000 7f50673bb700 1 -- 172.21.15.42:0/2658131187 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f50600a5d20 msgr2=0x7f50600a27a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:46.536 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:46.522+0000 7f50673bb700 1 --2- 172.21.15.42:0/2658131187 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f50600a5d20 0x7f50600a27a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:46.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:46.522+0000 7f50673bb700 1 -- 172.21.15.42:0/2658131187 >> v1:172.21.15.42:6789/0 conn(0x7f506007bbf0 legacy=0x7f5060143ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:46.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:46.522+0000 7f50673bb700 1 -- 172.21.15.42:0/2658131187 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50600a4840 con 0x7f50601446f0 2019-12-05T06:19:46.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:46.522+0000 7f50673bb700 1 --2- 172.21.15.42:0/2658131187 >> v2:172.21.15.42:3300/0 conn(0x7f50601446f0 0x7f506007c020 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f50600a14d0 tx=0x7f5058010200).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:46.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:46.522+0000 7f50673bb700 1 --2- 172.21.15.42:0/2658131187 >> v2:172.21.15.42:3300/0 conn(0x7f50601446f0 0x7f506007c020 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f50600a14d0 tx=0x7f5058010200).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:46.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:46.522+0000 7f50673bb700 1 --2- 172.21.15.42:0/2658131187 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f50601446f0 0x7f506007c020 secure :-1 s=READY pgs=2465 cs=0 l=1 rx=0x7f50600a14d0 tx=0x7f5058010200).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:19:46.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:46.522+0000 7f50673bb700 1 --2- 172.21.15.42:0/2658131187 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f50601446f0 0x7f506007c020 secure :-1 s=THROTTLE_DONE pgs=2465 cs=0 l=1 rx=0x7f50600a14d0 tx=0x7f5058010200).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:46.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:46.522+0000 7f50577fe700 1 -- 172.21.15.42:0/2658131187 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f505800edd0 con 0x7f50601446f0 2019-12-05T06:19:46.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:46.522+0000 7f50673bb700 1 --2- 172.21.15.42:0/2658131187 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f50601446f0 0x7f506007c020 secure :-1 s=THROTTLE_DONE pgs=2465 cs=0 l=1 rx=0x7f50600a14d0 tx=0x7f5058010200).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:46.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:46.522+0000 7f50673bb700 1 --2- 172.21.15.42:0/2658131187 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f50601446f0 0x7f506007c020 secure :-1 s=THROTTLE_DONE pgs=2465 cs=0 l=1 rx=0x7f50600a14d0 tx=0x7f5058010200).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:46.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:46.522+0000 7f50577fe700 1 -- 172.21.15.42:0/2658131187 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f505800e9f0 con 0x7f50601446f0 2019-12-05T06:19:46.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:46.522+0000 7f506861d700 1 -- 172.21.15.42:0/2658131187 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5060134740 con 0x7f50601446f0 2019-12-05T06:19:46.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:46.522+0000 7f50577fe700 1 -- 172.21.15.42:0/2658131187 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f50580110c0 con 0x7f50601446f0 2019-12-05T06:19:46.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:46.522+0000 7f506861d700 1 -- 172.21.15.42:0/2658131187 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5060134c40 con 0x7f50601446f0 2019-12-05T06:19:46.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:46.526+0000 7f50673bb700 1 --2- 172.21.15.42:0/2658131187 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f50601446f0 0x7f506007c020 secure :-1 s=THROTTLE_DONE pgs=2465 cs=0 l=1 rx=0x7f50600a14d0 tx=0x7f5058010200).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:46.539 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:46.526+0000 7f50577fe700 1 -- 172.21.15.42:0/2658131187 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f50580112c0 con 0x7f50601446f0 2019-12-05T06:19:46.539 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:46.526+0000 7f50673bb700 1 --2- 172.21.15.42:0/2658131187 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f50601446f0 0x7f506007c020 secure :-1 s=THROTTLE_DONE pgs=2465 cs=0 l=1 rx=0x7f50600a14d0 tx=0x7f5058010200).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:46.539 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:46.526+0000 7f50577fe700 1 --2- 172.21.15.42:0/2658131187 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f504803ad20 0x7f504803d230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:19:46.539 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:46.526+0000 7f50577fe700 1 -- 172.21.15.42:0/2658131187 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 10503+0+0 (secure 0 0 0) 0x7f505c00b730 con 0x7f50601446f0 2019-12-05T06:19:46.539 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:46.526+0000 7f5066bba700 1 --2- 172.21.15.42:0/2658131187 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f504803ad20 0x7f504803d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:46.539 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:46.526+0000 7f5066bba700 1 --2- 172.21.15.42:0/2658131187 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f504803ad20 0x7f504803d230 crc :-1 s=READY pgs=1034 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:19:46.539 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:46.526+0000 7f506861d700 1 -- 172.21.15.42:0/2658131187 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f506005d110 con 0x7f50601446f0 2019-12-05T06:19:46.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:46.534+0000 7f50673bb700 1 --2- 172.21.15.42:0/2658131187 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f50601446f0 0x7f506007c020 secure :-1 s=THROTTLE_DONE pgs=2465 cs=0 l=1 rx=0x7f50600a14d0 tx=0x7f5058010200).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:46.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:46.534+0000 7f50577fe700 1 -- 172.21.15.42:0/2658131187 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f5058032140 con 0x7f50601446f0 2019-12-05T06:19:46.704 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:46.702+0000 7f506861d700 1 -- 172.21.15.42:0/2658131187 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5060145110 con 0x7f50601446f0 2019-12-05T06:19:46.705 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:46.702+0000 7f50673bb700 1 --2- 172.21.15.42:0/2658131187 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f50601446f0 0x7f506007c020 secure :-1 s=THROTTLE_DONE pgs=2465 cs=0 l=1 rx=0x7f50600a14d0 tx=0x7f5058010200).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:46.705 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:46.702+0000 7f50577fe700 1 -- 172.21.15.42:0/2658131187 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 293 v293) v1 ==== 95+0+2117 (secure 0 0 0) 0x7f5058007160 con 0x7f50601446f0 2019-12-05T06:19:46.707 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:19:46.707 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":293,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":10711,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/2638662429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2638662429},{"type":"v1","addr":"172.21.15.42:6835","nonce":2638662429}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":293}],"filesystems":[{"mdsmap":{"epoch":292,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:19:15.110045+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":101,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10476},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10476":{"gid":10476,"name":"b","rank":0,"incarnation":289,"state":"up:active","state_seq":34,"addr":"172.21.15.40:6833/1473620421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1473620421},{"type":"v1","addr":"172.21.15.40:6833","nonce":1473620421}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:19:46.707 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 293 2019-12-05T06:19:46.709 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:46.706+0000 7f506861d700 1 -- 172.21.15.42:0/2658131187 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f504803ad20 msgr2=0x7f504803d230 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:46.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:46.706+0000 7f506861d700 1 --2- 172.21.15.42:0/2658131187 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f504803ad20 0x7f504803d230 crc :-1 s=READY pgs=1034 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:46.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:46.706+0000 7f506861d700 1 -- 172.21.15.42:0/2658131187 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f50601446f0 msgr2=0x7f506007c020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:46.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:46.706+0000 7f506861d700 1 --2- 172.21.15.42:0/2658131187 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f50601446f0 0x7f506007c020 secure :-1 s=READY pgs=2465 cs=0 l=1 rx=0x7f50600a14d0 tx=0x7f5058010200).stop 2019-12-05T06:19:46.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:46.706+0000 7f506861d700 1 -- 172.21.15.42:0/2658131187 shutdown_connections 2019-12-05T06:19:46.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:46.706+0000 7f506861d700 1 --2- 172.21.15.42:0/2658131187 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f504803ad20 0x7f504803d230 unknown :-1 s=CLOSED pgs=1034 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:46.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:46.706+0000 7f506861d700 1 --2- 172.21.15.42:0/2658131187 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f50601446f0 0x7f506007c020 unknown :-1 s=CLOSED pgs=2465 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:46.711 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:46.706+0000 7f506861d700 1 --2- 172.21.15.42:0/2658131187 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f50600a5d20 0x7f50600a27a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:46.711 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:46.706+0000 7f506861d700 1 -- 172.21.15.42:0/2658131187 shutdown_connections 2019-12-05T06:19:46.711 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:46.706+0000 7f506861d700 1 -- 172.21.15.42:0/2658131187 wait complete. 2019-12-05T06:19:46.711 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:46.706+0000 7f506861d700 1 -- 172.21.15.42:0/2658131187 >> 172.21.15.42:0/2658131187 conn(0x7f506008fde0 msgr2=0x7f5060118f50 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:19:46.720 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:19:48.721 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:19:48.823 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:48.814+0000 7fa7c239c700 1 Processor -- start 2019-12-05T06:19:48.823 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:48.814+0000 7fa7c239c700 1 -- start start 2019-12-05T06:19:48.823 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:48.814+0000 7fa7c239c700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa7bc09fae0 0x7fa7bc14fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:19:48.823 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:48.814+0000 7fa7c239c700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7bc0a0710 con 0x7fa7bc0a7740 2019-12-05T06:19:48.823 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:48.814+0000 7fa7c239c700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7bc07ef20 con 0x7fa7bc08df80 2019-12-05T06:19:48.824 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:48.814+0000 7fa7c239c700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7bc0a0b00 con 0x7fa7bc09fae0 2019-12-05T06:19:48.824 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:48.818+0000 7fa7bbfff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fa7bc08df80 0x7fa7bc09f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:53104/0 (socket says 172.21.15.42:53104) 2019-12-05T06:19:48.824 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:48.818+0000 7fa7bbfff700 1 -- 172.21.15.42:0/397109649 learned_addr learned my addr 172.21.15.42:0/397109649 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:19:48.824 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:48.818+0000 7fa7c113a700 1 --2- 172.21.15.42:0/397109649 >> v2:172.21.15.40:3300/0 conn(0x7fa7bc09fae0 0x7fa7bc14fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:48.824 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:48.818+0000 7fa7bb7fe700 1 -- 172.21.15.42:0/397109649 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3598897353 0 0) 0x7fa7bc07ef20 con 0x7fa7bc08df80 2019-12-05T06:19:48.825 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:48.818+0000 7fa7bb7fe700 1 -- 172.21.15.42:0/397109649 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7a80030b0 con 0x7fa7bc08df80 2019-12-05T06:19:48.825 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:48.818+0000 7fa7c113a700 1 -- 172.21.15.42:0/397109649 >> v1:172.21.15.42:6789/0 conn(0x7fa7bc08df80 legacy=0x7fa7bc09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:48.825 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:48.818+0000 7fa7c113a700 1 -- 172.21.15.42:0/397109649 >> v1:172.21.15.40:6789/0 conn(0x7fa7bc0a7740 legacy=0x7fa7bc07e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:48.825 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:48.818+0000 7fa7c113a700 1 -- 172.21.15.42:0/397109649 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7bc0a9ba0 con 0x7fa7bc09fae0 2019-12-05T06:19:48.825 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:48.818+0000 7fa7c113a700 1 --2- 172.21.15.42:0/397109649 >> v2:172.21.15.40:3300/0 conn(0x7fa7bc09fae0 0x7fa7bc14fff0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fa7b4008790 tx=0x7fa7b400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:48.826 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:48.818+0000 7fa7c113a700 1 --2- 172.21.15.42:0/397109649 >> v2:172.21.15.40:3300/0 conn(0x7fa7bc09fae0 0x7fa7bc14fff0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fa7b4008790 tx=0x7fa7b400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:48.826 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:48.818+0000 7fa7c113a700 1 --2- 172.21.15.42:0/397109649 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7bc09fae0 0x7fa7bc14fff0 secure :-1 s=READY pgs=1863 cs=0 l=1 rx=0x7fa7b4008790 tx=0x7fa7b400fbf0).ready entity=mon.0 client_cookie=554c06da0f7ce71b server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:19:48.826 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:48.818+0000 7fa7c113a700 1 --2- 172.21.15.42:0/397109649 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7bc09fae0 0x7fa7bc14fff0 secure :-1 s=THROTTLE_DONE pgs=1863 cs=0 l=1 rx=0x7fa7b4008790 tx=0x7fa7b400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:48.826 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:48.818+0000 7fa7bb7fe700 1 -- 172.21.15.42:0/397109649 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa7b4008bb0 con 0x7fa7bc09fae0 2019-12-05T06:19:48.826 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:48.818+0000 7fa7c113a700 1 --2- 172.21.15.42:0/397109649 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7bc09fae0 0x7fa7bc14fff0 secure :-1 s=THROTTLE_DONE pgs=1863 cs=0 l=1 rx=0x7fa7b4008790 tx=0x7fa7b400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:48.826 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:48.818+0000 7fa7c113a700 1 --2- 172.21.15.42:0/397109649 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7bc09fae0 0x7fa7bc14fff0 secure :-1 s=THROTTLE_DONE pgs=1863 cs=0 l=1 rx=0x7fa7b4008790 tx=0x7fa7b400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:48.826 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:48.818+0000 7fa7bb7fe700 1 -- 172.21.15.42:0/397109649 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7b40087d0 con 0x7fa7bc09fae0 2019-12-05T06:19:48.827 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:48.818+0000 7fa7bb7fe700 1 -- 172.21.15.42:0/397109649 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa7b40118d0 con 0x7fa7bc09fae0 2019-12-05T06:19:48.827 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:48.818+0000 7fa7c239c700 1 -- 172.21.15.42:0/397109649 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7bc09fae0 msgr2=0x7fa7bc14fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:48.827 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:48.818+0000 7fa7c239c700 1 --2- 172.21.15.42:0/397109649 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7bc09fae0 0x7fa7bc14fff0 secure :-1 s=READY pgs=1863 cs=0 l=1 rx=0x7fa7b4008790 tx=0x7fa7b400fbf0).stop 2019-12-05T06:19:48.827 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:48.818+0000 7fa7c239c700 1 -- 172.21.15.42:0/397109649 shutdown_connections 2019-12-05T06:19:48.827 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:48.818+0000 7fa7c239c700 1 --2- 172.21.15.42:0/397109649 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7bc09fae0 0x7fa7bc14fff0 unknown :-1 s=CLOSED pgs=1863 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:48.828 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:48.818+0000 7fa7c239c700 1 -- 172.21.15.42:0/397109649 shutdown_connections 2019-12-05T06:19:48.828 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:48.818+0000 7fa7c239c700 1 -- 172.21.15.42:0/397109649 wait complete. 2019-12-05T06:19:48.828 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:48.818+0000 7fa7c239c700 1 -- 172.21.15.42:0/397109649 >> 172.21.15.42:0/397109649 conn(0x7fa7bc0aabe0 msgr2=0x7fa7bc0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:19:48.832 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:48.818+0000 7fa7c239c700 1 Processor -- start 2019-12-05T06:19:48.832 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:48.818+0000 7fa7c239c700 1 -- start start 2019-12-05T06:19:48.833 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:48.822+0000 7fa7c239c700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa7bc0636b0 0x7fa7bc07b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:19:48.833 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:48.822+0000 7fa7c239c700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa7bc064910 0x7fa7bc122580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:19:48.833 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:48.822+0000 7fa7c239c700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7bc122ad0 con 0x7fa7bc0a3960 2019-12-05T06:19:48.833 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:48.822+0000 7fa7c239c700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7bc064d40 con 0x7fa7bc064910 2019-12-05T06:19:48.833 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:48.822+0000 7fa7c239c700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7bc064fb0 con 0x7fa7bc0636b0 2019-12-05T06:19:48.833 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:48.822+0000 7fa7c0939700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fa7bc0a3960 0x7fa7bc079110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:53110/0 (socket says 172.21.15.42:53110) 2019-12-05T06:19:48.834 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:48.822+0000 7fa7c0939700 1 -- 172.21.15.42:0/3696833190 learned_addr learned my addr 172.21.15.42:0/3696833190 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:19:48.834 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:48.822+0000 7fa7bbfff700 1 --2- 172.21.15.42:0/3696833190 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa7bc0636b0 0x7fa7bc07b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:48.834 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:48.822+0000 7fa7c113a700 1 --2- 172.21.15.42:0/3696833190 >> v2:172.21.15.40:3300/0 conn(0x7fa7bc064910 0x7fa7bc122580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:48.834 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:48.822+0000 7fa7b97fa700 1 -- 172.21.15.42:0/3696833190 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 219709569 0 0) 0x7fa7bc122ad0 con 0x7fa7bc0a3960 2019-12-05T06:19:48.834 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:48.822+0000 7fa7b97fa700 1 -- 172.21.15.42:0/3696833190 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7a00030b0 con 0x7fa7bc0a3960 2019-12-05T06:19:48.834 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:48.822+0000 7fa7bbfff700 1 -- 172.21.15.42:0/3696833190 >> v2:172.21.15.40:3300/0 conn(0x7fa7bc064910 msgr2=0x7fa7bc122580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:48.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:48.822+0000 7fa7bbfff700 1 --2- 172.21.15.42:0/3696833190 >> v2:172.21.15.40:3300/0 conn(0x7fa7bc064910 0x7fa7bc122580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:48.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:48.822+0000 7fa7bbfff700 1 -- 172.21.15.42:0/3696833190 >> v1:172.21.15.42:6789/0 conn(0x7fa7bc0a3960 legacy=0x7fa7bc079110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:48.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:48.822+0000 7fa7bbfff700 1 -- 172.21.15.42:0/3696833190 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7bc122d00 con 0x7fa7bc0636b0 2019-12-05T06:19:48.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:48.822+0000 7fa7bbfff700 1 --2- 172.21.15.42:0/3696833190 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa7bc0636b0 0x7fa7bc07b3c0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fa7a4000b80 tx=0x7fa7a4010890).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:48.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:48.822+0000 7fa7bbfff700 1 --2- 172.21.15.42:0/3696833190 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa7bc0636b0 0x7fa7bc07b3c0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fa7a4000b80 tx=0x7fa7a4010890).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:48.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:48.822+0000 7fa7bbfff700 1 --2- 172.21.15.42:0/3696833190 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa7bc0636b0 0x7fa7bc07b3c0 secure :-1 s=READY pgs=539 cs=0 l=1 rx=0x7fa7a4000b80 tx=0x7fa7a4010890).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:19:48.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:48.822+0000 7fa7bbfff700 1 --2- 172.21.15.42:0/3696833190 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa7bc0636b0 0x7fa7bc07b3c0 secure :-1 s=THROTTLE_DONE pgs=539 cs=0 l=1 rx=0x7fa7a4000b80 tx=0x7fa7a4010890).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:48.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:48.822+0000 7fa7b97fa700 1 -- 172.21.15.42:0/3696833190 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa7a4005b40 con 0x7fa7bc0636b0 2019-12-05T06:19:48.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:48.822+0000 7fa7bbfff700 1 --2- 172.21.15.42:0/3696833190 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa7bc0636b0 0x7fa7bc07b3c0 secure :-1 s=THROTTLE_DONE pgs=539 cs=0 l=1 rx=0x7fa7a4000b80 tx=0x7fa7a4010890).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:48.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:48.822+0000 7fa7b97fa700 1 -- 172.21.15.42:0/3696833190 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7a4005760 con 0x7fa7bc0636b0 2019-12-05T06:19:48.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:48.822+0000 7fa7bbfff700 1 --2- 172.21.15.42:0/3696833190 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa7bc0636b0 0x7fa7bc07b3c0 secure :-1 s=THROTTLE_DONE pgs=539 cs=0 l=1 rx=0x7fa7a4000b80 tx=0x7fa7a4010890).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:48.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:48.822+0000 7fa7c239c700 1 -- 172.21.15.42:0/3696833190 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7bc122fd0 con 0x7fa7bc0636b0 2019-12-05T06:19:48.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:48.822+0000 7fa7b97fa700 1 -- 172.21.15.42:0/3696833190 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa7a40062b0 con 0x7fa7bc0636b0 2019-12-05T06:19:48.837 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:48.822+0000 7fa7c239c700 1 -- 172.21.15.42:0/3696833190 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7bc130770 con 0x7fa7bc0636b0 2019-12-05T06:19:48.837 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:48.822+0000 7fa7bbfff700 1 --2- 172.21.15.42:0/3696833190 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa7bc0636b0 0x7fa7bc07b3c0 secure :-1 s=THROTTLE_DONE pgs=539 cs=0 l=1 rx=0x7fa7a4000b80 tx=0x7fa7a4010890).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:48.837 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:48.826+0000 7fa7b97fa700 1 -- 172.21.15.42:0/3696833190 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fa7a4006550 con 0x7fa7bc0636b0 2019-12-05T06:19:48.837 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:48.826+0000 7fa7bbfff700 1 --2- 172.21.15.42:0/3696833190 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa7bc0636b0 0x7fa7bc07b3c0 secure :-1 s=THROTTLE_DONE pgs=539 cs=0 l=1 rx=0x7fa7a4000b80 tx=0x7fa7a4010890).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:48.837 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:48.826+0000 7fa7b97fa700 1 --2- 172.21.15.42:0/3696833190 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa7a003b280 0x7fa7a003d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:19:48.837 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:48.826+0000 7fa7b97fa700 1 -- 172.21.15.42:0/3696833190 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 10503+0+0 (secure 0 0 0) 0x7fa7a404b020 con 0x7fa7bc0636b0 2019-12-05T06:19:48.838 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:48.826+0000 7fa7c0939700 1 --2- 172.21.15.42:0/3696833190 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa7a003b280 0x7fa7a003d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:48.838 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:48.826+0000 7fa7c0939700 1 --2- 172.21.15.42:0/3696833190 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa7a003b280 0x7fa7a003d790 crc :-1 s=READY pgs=1035 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:19:48.839 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:48.830+0000 7fa7c239c700 1 -- 172.21.15.42:0/3696833190 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7bc05d110 con 0x7fa7bc0636b0 2019-12-05T06:19:48.845 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:48.842+0000 7fa7bbfff700 1 --2- 172.21.15.42:0/3696833190 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa7bc0636b0 0x7fa7bc07b3c0 secure :-1 s=THROTTLE_DONE pgs=539 cs=0 l=1 rx=0x7fa7a4000b80 tx=0x7fa7a4010890).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:48.845 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:48.842+0000 7fa7b97fa700 1 -- 172.21.15.42:0/3696833190 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fa7a4032080 con 0x7fa7bc0636b0 2019-12-05T06:19:49.015 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:49.010+0000 7fa7c239c700 1 -- 172.21.15.42:0/3696833190 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa7bc05b8c0 con 0x7fa7bc0636b0 2019-12-05T06:19:49.016 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:49.010+0000 7fa7bbfff700 1 --2- 172.21.15.42:0/3696833190 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa7bc0636b0 0x7fa7bc07b3c0 secure :-1 s=THROTTLE_DONE pgs=539 cs=0 l=1 rx=0x7fa7a4000b80 tx=0x7fa7a4010890).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:49.016 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:49.014+0000 7fa7b97fa700 1 -- 172.21.15.42:0/3696833190 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 293 v293) v1 ==== 95+0+2117 (secure 0 0 0) 0x7fa7a40322e0 con 0x7fa7bc0636b0 2019-12-05T06:19:49.018 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 293 2019-12-05T06:19:49.018 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:19:49.018 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":293,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":10711,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/2638662429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2638662429},{"type":"v1","addr":"172.21.15.42:6835","nonce":2638662429}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":293}],"filesystems":[{"mdsmap":{"epoch":292,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:19:15.110045+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":101,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10476},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10476":{"gid":10476,"name":"b","rank":0,"incarnation":289,"state":"up:active","state_seq":34,"addr":"172.21.15.40:6833/1473620421","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1473620421},{"type":"v1","addr":"172.21.15.40:6833","nonce":1473620421}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:19:49.021 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:49.018+0000 7fa7c239c700 1 -- 172.21.15.42:0/3696833190 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa7a003b280 msgr2=0x7fa7a003d790 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:49.021 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:49.018+0000 7fa7c239c700 1 --2- 172.21.15.42:0/3696833190 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa7a003b280 0x7fa7a003d790 crc :-1 s=READY pgs=1035 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:49.022 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:49.018+0000 7fa7c239c700 1 -- 172.21.15.42:0/3696833190 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa7bc0636b0 msgr2=0x7fa7bc07b3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:49.022 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:49.018+0000 7fa7c239c700 1 --2- 172.21.15.42:0/3696833190 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa7bc0636b0 0x7fa7bc07b3c0 secure :-1 s=READY pgs=539 cs=0 l=1 rx=0x7fa7a4000b80 tx=0x7fa7a4010890).stop 2019-12-05T06:19:49.022 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:49.018+0000 7fa7c239c700 1 -- 172.21.15.42:0/3696833190 shutdown_connections 2019-12-05T06:19:49.022 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:49.018+0000 7fa7c239c700 1 --2- 172.21.15.42:0/3696833190 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa7a003b280 0x7fa7a003d790 unknown :-1 s=CLOSED pgs=1035 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:49.022 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:49.018+0000 7fa7c239c700 1 --2- 172.21.15.42:0/3696833190 >> v2:172.21.15.40:3300/0 conn(0x7fa7bc064910 0x7fa7bc122580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:49.022 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:49.018+0000 7fa7c239c700 1 --2- 172.21.15.42:0/3696833190 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa7bc0636b0 0x7fa7bc07b3c0 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:49.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:49.018+0000 7fa7c239c700 1 -- 172.21.15.42:0/3696833190 shutdown_connections 2019-12-05T06:19:49.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:49.018+0000 7fa7c239c700 1 -- 172.21.15.42:0/3696833190 wait complete. 2019-12-05T06:19:49.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:49.018+0000 7fa7c239c700 1 -- 172.21.15.42:0/3696833190 >> 172.21.15.42:0/3696833190 conn(0x7fa7bc07bcd0 msgr2=0x7fa7bc0a6a60 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:19:49.031 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:19:51.036 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:19:51.168 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.150+0000 7f0339166700 1 Processor -- start 2019-12-05T06:19:51.168 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.154+0000 7f0339166700 1 -- start start 2019-12-05T06:19:51.177 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.154+0000 7f0339166700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f03340a7740 0x7f033408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:19:51.177 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.154+0000 7f0339166700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f033409fae0 0x7f033407eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:19:51.177 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.154+0000 7f0339166700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03340a0710 con 0x7f033407e720 2019-12-05T06:19:51.177 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.154+0000 7f0339166700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03340a0b00 con 0x7f033409fae0 2019-12-05T06:19:51.177 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.154+0000 7f0339166700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f033408e5b0 con 0x7f03340a7740 2019-12-05T06:19:51.177 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.154+0000 7f033359e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f03340a7740 0x7f033408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:51.178 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.154+0000 7f033359e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f03340a7740 0x7f033408df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:53094/0 (socket says 172.21.15.42:53094) 2019-12-05T06:19:51.178 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.154+0000 7f033359e700 1 -- 172.21.15.42:0/1022475555 learned_addr learned my addr 172.21.15.42:0/1022475555 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:19:51.178 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.154+0000 7f0338965700 1 --2- 172.21.15.42:0/1022475555 >> v2:172.21.15.40:3300/0 conn(0x7f033409fae0 0x7f033407eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:51.178 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.154+0000 7f033359e700 1 -- 172.21.15.42:0/1022475555 >> v2:172.21.15.40:3300/0 conn(0x7f033409fae0 msgr2=0x7f033407eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:19:51.178 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.154+0000 7f033359e700 1 --2- 172.21.15.42:0/1022475555 >> v2:172.21.15.40:3300/0 conn(0x7f033409fae0 0x7f033407eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:19:51.178 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.154+0000 7f033359e700 1 -- 172.21.15.42:0/1022475555 >> v1:172.21.15.40:6789/0 conn(0x7f033407e720 legacy=0x7f033409f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:19:51.178 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.154+0000 7f033359e700 1 -- 172.21.15.42:0/1022475555 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f033407f0a0 con 0x7f03340a7740 2019-12-05T06:19:51.179 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.154+0000 7f033359e700 1 --2- 172.21.15.42:0/1022475555 >> v2:172.21.15.42:3300/0 conn(0x7f03340a7740 0x7f033408df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f0324002f30 tx=0x7f032400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:51.179 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.154+0000 7f033359e700 1 --2- 172.21.15.42:0/1022475555 >> v2:172.21.15.42:3300/0 conn(0x7f03340a7740 0x7f033408df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f0324002f30 tx=0x7f032400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:51.179 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.154+0000 7f033359e700 1 --2- 172.21.15.42:0/1022475555 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f03340a7740 0x7f033408df80 secure :-1 s=READY pgs=2469 cs=0 l=1 rx=0x7f0324002f30 tx=0x7f032400fd30).ready entity=mon.1 client_cookie=983907b92f728fcf server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:19:51.179 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.154+0000 7f033359e700 1 --2- 172.21.15.42:0/1022475555 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f03340a7740 0x7f033408df80 secure :-1 s=THROTTLE_DONE pgs=2469 cs=0 l=1 rx=0x7f0324002f30 tx=0x7f032400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:51.179 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.154+0000 7f033359e700 1 --2- 172.21.15.42:0/1022475555 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f03340a7740 0x7f033408df80 secure :-1 s=THROTTLE_DONE pgs=2469 cs=0 l=1 rx=0x7f0324002f30 tx=0x7f032400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:51.179 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.154+0000 7f033259c700 1 -- 172.21.15.42:0/1022475555 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0324008c90 con 0x7f03340a7740 2019-12-05T06:19:51.179 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.154+0000 7f033359e700 1 --2- 172.21.15.42:0/1022475555 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f03340a7740 0x7f033408df80 secure :-1 s=THROTTLE_DONE pgs=2469 cs=0 l=1 rx=0x7f0324002f30 tx=0x7f032400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:51.180 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.154+0000 7f033259c700 1 -- 172.21.15.42:0/1022475555 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03240088b0 con 0x7f03340a7740 2019-12-05T06:19:51.180 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.154+0000 7f033259c700 1 -- 172.21.15.42:0/1022475555 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0324011ad0 con 0x7f03340a7740 2019-12-05T06:19:51.180 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.154+0000 7f0339166700 1 -- 172.21.15.42:0/1022475555 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f03340a7740 msgr2=0x7f033408df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:51.180 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.154+0000 7f0339166700 1 --2- 172.21.15.42:0/1022475555 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f03340a7740 0x7f033408df80 secure :-1 s=READY pgs=2469 cs=0 l=1 rx=0x7f0324002f30 tx=0x7f032400fd30).stop 2019-12-05T06:19:51.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.154+0000 7f0339166700 1 -- 172.21.15.42:0/1022475555 shutdown_connections 2019-12-05T06:19:51.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.154+0000 7f0339166700 1 --2- 172.21.15.42:0/1022475555 >> v2:172.21.15.40:3300/0 conn(0x7f033409fae0 0x7f033407eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:19:51.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.154+0000 7f0339166700 1 --2- 172.21.15.42:0/1022475555 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f03340a7740 0x7f033408df80 unknown :-1 s=CLOSED pgs=2469 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:51.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.154+0000 7f0339166700 1 -- 172.21.15.42:0/1022475555 shutdown_connections 2019-12-05T06:19:51.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.154+0000 7f0339166700 1 -- 172.21.15.42:0/1022475555 wait complete. 2019-12-05T06:19:51.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.154+0000 7f0339166700 1 -- 172.21.15.42:0/1022475555 >> 172.21.15.42:0/1022475555 conn(0x7f03340aabe0 msgr2=0x7f03340a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:19:51.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.158+0000 7f0339166700 1 Processor -- start 2019-12-05T06:19:51.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.158+0000 7f0339166700 1 -- start start 2019-12-05T06:19:51.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.158+0000 7f0339166700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f03340a1d20 0x7f03340a3eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:19:51.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.158+0000 7f0339166700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0334078b40 con 0x7f033407bec0 2019-12-05T06:19:51.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.158+0000 7f0339166700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0334078d70 con 0x7f033407a1c0 2019-12-05T06:19:51.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.158+0000 7f0339166700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03340a4400 con 0x7f03340a1d20 2019-12-05T06:19:51.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.158+0000 7f0332d9d700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f033407a1c0 0x7f033407b6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:53140/0 (socket says 172.21.15.42:53140) 2019-12-05T06:19:51.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.158+0000 7f0332d9d700 1 -- 172.21.15.42:0/846739503 learned_addr learned my addr 172.21.15.42:0/846739503 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:19:51.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.158+0000 7f033359e700 1 --2- 172.21.15.42:0/846739503 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f03340a1d20 0x7f03340a3eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:51.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.158+0000 7f031bfff700 1 -- 172.21.15.42:0/846739503 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2420850180 0 0) 0x7f0334078d70 con 0x7f033407a1c0 2019-12-05T06:19:51.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.158+0000 7f031bfff700 1 -- 172.21.15.42:0/846739503 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03140030b0 con 0x7f033407a1c0 2019-12-05T06:19:51.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.158+0000 7f031bfff700 1 -- 172.21.15.42:0/846739503 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 1443463954 0 0) 0x7f03140030b0 con 0x7f033407a1c0 2019-12-05T06:19:51.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.158+0000 7f031bfff700 1 -- 172.21.15.42:0/846739503 --> v1:172.21.15.42:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0334078d70 con 0x7f033407a1c0 2019-12-05T06:19:51.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.158+0000 7f031bfff700 1 -- 172.21.15.42:0/846739503 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7f0328001a00 con 0x7f033407a1c0 2019-12-05T06:19:51.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.158+0000 7f031bfff700 1 -- 172.21.15.42:0/846739503 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f03340a1d20 msgr2=0x7f03340a3eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:51.184 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.158+0000 7f031bfff700 1 --2- 172.21.15.42:0/846739503 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f03340a1d20 0x7f03340a3eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:51.184 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.158+0000 7f031bfff700 1 -- 172.21.15.42:0/846739503 >> v1:172.21.15.42:6789/0 conn(0x7f033407a1c0 legacy=0x7f033407b6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:51.184 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.158+0000 7f031bfff700 1 -- 172.21.15.42:0/846739503 >> v1:172.21.15.40:6789/0 conn(0x7f033407bec0 legacy=0x7f0334078340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:51.184 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.158+0000 7f031bfff700 1 --2- 172.21.15.42:0/846739503 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f03140038a0 0x7f0314003cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:19:51.184 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.158+0000 7f031bfff700 1 --2- 172.21.15.42:0/846739503 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0314004420 0x7f0314004850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:19:51.184 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.158+0000 7f031bfff700 1 --2- 172.21.15.42:0/846739503 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0314004da0 0x7f031400d880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:19:51.184 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.158+0000 7f031bfff700 1 -- 172.21.15.42:0/846739503 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03340a4400 con 0x7f0314004da0 2019-12-05T06:19:51.185 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.158+0000 7f031bfff700 1 -- 172.21.15.42:0/846739503 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0314009660 con 0x7f03140038a0 2019-12-05T06:19:51.185 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.158+0000 7f031bfff700 1 -- 172.21.15.42:0/846739503 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03140098a0 con 0x7f0314004420 2019-12-05T06:19:51.185 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.158+0000 7f033359e700 1 --2- 172.21.15.42:0/846739503 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f03140038a0 0x7f0314003cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:51.185 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.158+0000 7f0338965700 1 --2- 172.21.15.42:0/846739503 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0314004da0 0x7f031400d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:51.185 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.158+0000 7f0332d9d700 1 --2- 172.21.15.42:0/846739503 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0314004420 0x7f0314004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:51.185 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.158+0000 7f033359e700 1 -- 172.21.15.42:0/846739503 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0314004420 msgr2=0x7f0314004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:51.185 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.158+0000 7f033359e700 1 --2- 172.21.15.42:0/846739503 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0314004420 0x7f0314004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:51.186 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.158+0000 7f033359e700 1 -- 172.21.15.42:0/846739503 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0314004da0 msgr2=0x7f031400d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:51.186 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.158+0000 7f033359e700 1 --2- 172.21.15.42:0/846739503 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0314004da0 0x7f031400d880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:51.186 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.158+0000 7f033359e700 1 -- 172.21.15.42:0/846739503 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0334078fa0 con 0x7f03140038a0 2019-12-05T06:19:51.186 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.158+0000 7f0339166700 1 -- 172.21.15.42:0/846739503 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03340a1d20 con 0x7f03140038a0 2019-12-05T06:19:51.186 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.158+0000 7f033359e700 1 --2- 172.21.15.42:0/846739503 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f03140038a0 0x7f0314003cd0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f0324002f30 tx=0x7f0324008eb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:51.186 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.158+0000 7f0339166700 1 -- 172.21.15.42:0/846739503 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03340a3eb0 con 0x7f03140038a0 2019-12-05T06:19:51.186 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.158+0000 7f033359e700 1 --2- 172.21.15.42:0/846739503 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f03140038a0 0x7f0314003cd0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f0324002f30 tx=0x7f0324008eb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:51.187 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.158+0000 7f033359e700 1 --2- 172.21.15.42:0/846739503 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f03140038a0 0x7f0314003cd0 secure :-1 s=READY pgs=2471 cs=0 l=1 rx=0x7f0324002f30 tx=0x7f0324008eb0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:19:51.187 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.158+0000 7f033359e700 1 --2- 172.21.15.42:0/846739503 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f03140038a0 0x7f0314003cd0 secure :-1 s=THROTTLE_DONE pgs=2471 cs=0 l=1 rx=0x7f0324002f30 tx=0x7f0324008eb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:51.187 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.158+0000 7f031bfff700 1 -- 172.21.15.42:0/846739503 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0324007350 con 0x7f03140038a0 2019-12-05T06:19:51.187 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.158+0000 7f033359e700 1 --2- 172.21.15.42:0/846739503 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f03140038a0 0x7f0314003cd0 secure :-1 s=THROTTLE_DONE pgs=2471 cs=0 l=1 rx=0x7f0324002f30 tx=0x7f0324008eb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:51.187 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.158+0000 7f033359e700 1 --2- 172.21.15.42:0/846739503 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f03140038a0 0x7f0314003cd0 secure :-1 s=THROTTLE_DONE pgs=2471 cs=0 l=1 rx=0x7f0324002f30 tx=0x7f0324008eb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:51.187 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.158+0000 7f031bfff700 1 -- 172.21.15.42:0/846739503 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03240116f0 con 0x7f03140038a0 2019-12-05T06:19:51.187 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.158+0000 7f031bfff700 1 -- 172.21.15.42:0/846739503 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0324005540 con 0x7f03140038a0 2019-12-05T06:19:51.188 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.162+0000 7f033359e700 1 --2- 172.21.15.42:0/846739503 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f03140038a0 0x7f0314003cd0 secure :-1 s=THROTTLE_DONE pgs=2471 cs=0 l=1 rx=0x7f0324002f30 tx=0x7f0324008eb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:51.188 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.162+0000 7f031bfff700 1 -- 172.21.15.42:0/846739503 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f0324005740 con 0x7f03140038a0 2019-12-05T06:19:51.188 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.162+0000 7f033359e700 1 --2- 172.21.15.42:0/846739503 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f03140038a0 0x7f0314003cd0 secure :-1 s=THROTTLE_DONE pgs=2471 cs=0 l=1 rx=0x7f0324002f30 tx=0x7f0324008eb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:51.188 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.162+0000 7f031bfff700 1 --2- 172.21.15.42:0/846739503 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f03140480a0 0x7f031404a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:19:51.188 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.162+0000 7f031bfff700 1 -- 172.21.15.42:0/846739503 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 10589+0+0 (secure 0 0 0) 0x7f032404bee0 con 0x7f03140038a0 2019-12-05T06:19:51.188 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.162+0000 7f0332d9d700 1 --2- 172.21.15.42:0/846739503 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f03140480a0 0x7f031404a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:51.188 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.162+0000 7f0332d9d700 1 --2- 172.21.15.42:0/846739503 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f03140480a0 0x7f031404a5b0 crc :-1 s=READY pgs=1036 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:19:51.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.166+0000 7f0339166700 1 -- 172.21.15.42:0/846739503 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f030c003200 con 0x7f03140038a0 2019-12-05T06:19:51.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.174+0000 7f033359e700 1 --2- 172.21.15.42:0/846739503 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f03140038a0 0x7f0314003cd0 secure :-1 s=THROTTLE_DONE pgs=2471 cs=0 l=1 rx=0x7f0324002f30 tx=0x7f0324008eb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:51.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.174+0000 7f031bfff700 1 -- 172.21.15.42:0/846739503 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f0324068400 con 0x7f03140038a0 2019-12-05T06:19:51.332 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.326+0000 7f0339166700 1 -- 172.21.15.42:0/846739503 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f030c003630 con 0x7f03140038a0 2019-12-05T06:19:51.332 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.330+0000 7f033359e700 1 --2- 172.21.15.42:0/846739503 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f03140038a0 0x7f0314003cd0 secure :-1 s=THROTTLE_DONE pgs=2471 cs=0 l=1 rx=0x7f0324002f30 tx=0x7f0324008eb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:51.332 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.330+0000 7f031bfff700 1 -- 172.21.15.42:0/846739503 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 294 v294) v1 ==== 95+0+1778 (secure 0 0 0) 0x7f0324032f40 con 0x7f03140038a0 2019-12-05T06:19:51.333 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 294 2019-12-05T06:19:51.333 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:19:51.333 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":294,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":294,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:19:49.764458+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":102,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10711},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10711":{"gid":10711,"name":"a","rank":0,"incarnation":294,"state":"up:replay","state_seq":2,"addr":"172.21.15.42:6835/2638662429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2638662429},{"type":"v1","addr":"172.21.15.42:6835","nonce":2638662429}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:19:51.337 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.330+0000 7f0339166700 1 -- 172.21.15.42:0/846739503 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f03140480a0 msgr2=0x7f031404a5b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:51.337 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.330+0000 7f0339166700 1 --2- 172.21.15.42:0/846739503 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f03140480a0 0x7f031404a5b0 crc :-1 s=READY pgs=1036 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:51.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.330+0000 7f0339166700 1 -- 172.21.15.42:0/846739503 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f03140038a0 msgr2=0x7f0314003cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:51.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.330+0000 7f0338965700 1 -- 172.21.15.42:0/846739503 reap_dead start 2019-12-05T06:19:51.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.330+0000 7f0339166700 1 --2- 172.21.15.42:0/846739503 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f03140038a0 0x7f0314003cd0 secure :-1 s=READY pgs=2471 cs=0 l=1 rx=0x7f0324002f30 tx=0x7f0324008eb0).stop 2019-12-05T06:19:51.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.330+0000 7f0339166700 1 -- 172.21.15.42:0/846739503 shutdown_connections 2019-12-05T06:19:51.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.330+0000 7f0339166700 1 --2- 172.21.15.42:0/846739503 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f03140038a0 0x7f0314003cd0 unknown :-1 s=CLOSED pgs=2471 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:51.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.330+0000 7f0339166700 1 -- 172.21.15.42:0/846739503 shutdown_connections 2019-12-05T06:19:51.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.330+0000 7f0339166700 1 -- 172.21.15.42:0/846739503 wait complete. 2019-12-05T06:19:51.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.330+0000 7f0339166700 1 -- 172.21.15.42:0/846739503 >> 172.21.15.42:0/846739503 conn(0x7f03340aabe0 msgr2=0x7f03340a9640 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:19:51.353 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b down, removed from mdsmap 2019-12-05T06:19:51.353 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2019-12-05T06:19:51.353 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:19:51.473 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.462+0000 7f4000cff700 1 Processor -- start 2019-12-05T06:19:51.473 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.462+0000 7f4000cff700 1 -- start start 2019-12-05T06:19:51.474 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.462+0000 7f4000cff700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3ffc0a7740 0x7f3ffc08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:19:51.474 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.462+0000 7f4000cff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3ffc07e720 0x7f3ffc07eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:19:51.474 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.462+0000 7f4000cff700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ffc0a0710 con 0x7f3ffc09f2e0 2019-12-05T06:19:51.474 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.462+0000 7f4000cff700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ffc0a0b00 con 0x7f3ffc07e720 2019-12-05T06:19:51.474 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.462+0000 7f4000cff700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ffc08e5b0 con 0x7f3ffc0a7740 2019-12-05T06:19:51.474 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.462+0000 7f3ffa59c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3ffc07e720 0x7f3ffc07eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:51.475 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.462+0000 7f3ffa59c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3ffc07e720 0x7f3ffc07eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:53116/0 (socket says 172.21.15.42:53116) 2019-12-05T06:19:51.475 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.462+0000 7f3ffa59c700 1 -- 172.21.15.42:0/3698760333 learned_addr learned my addr 172.21.15.42:0/3698760333 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:19:51.475 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.466+0000 7f3ffad9d700 1 --2- 172.21.15.42:0/3698760333 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3ffc0a7740 0x7f3ffc08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:51.475 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.466+0000 7f3ffa59c700 1 -- 172.21.15.42:0/3698760333 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3ffc0a7740 msgr2=0x7f3ffc08df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:19:51.475 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.466+0000 7f3ffa59c700 1 --2- 172.21.15.42:0/3698760333 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3ffc0a7740 0x7f3ffc08df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:19:51.475 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.466+0000 7f3ffa59c700 1 -- 172.21.15.42:0/3698760333 >> v1:172.21.15.40:6789/0 conn(0x7f3ffc09f2e0 legacy=0x7f3ffc09f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:19:51.475 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.466+0000 7f3ffa59c700 1 -- 172.21.15.42:0/3698760333 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ffc07f0a0 con 0x7f3ffc07e720 2019-12-05T06:19:51.476 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.466+0000 7f3ffa59c700 1 --2- 172.21.15.42:0/3698760333 >> v2:172.21.15.42:3300/0 conn(0x7f3ffc07e720 0x7f3ffc07eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f3ff0008790 tx=0x7f3ff000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:51.476 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.466+0000 7f3ffa59c700 1 --2- 172.21.15.42:0/3698760333 >> v2:172.21.15.42:3300/0 conn(0x7f3ffc07e720 0x7f3ffc07eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f3ff0008790 tx=0x7f3ff000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:51.476 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.466+0000 7f3ffa59c700 1 --2- 172.21.15.42:0/3698760333 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3ffc07e720 0x7f3ffc07eb50 secure :-1 s=READY pgs=2472 cs=0 l=1 rx=0x7f3ff0008790 tx=0x7f3ff000fd30).ready entity=mon.1 client_cookie=135478a9d02a349f server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:19:51.476 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.466+0000 7f3ffa59c700 1 --2- 172.21.15.42:0/3698760333 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3ffc07e720 0x7f3ffc07eb50 secure :-1 s=THROTTLE_DONE pgs=2472 cs=0 l=1 rx=0x7f3ff0008790 tx=0x7f3ff000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:51.476 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.466+0000 7f3ff9d9b700 1 -- 172.21.15.42:0/3698760333 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3ff00113b0 con 0x7f3ffc07e720 2019-12-05T06:19:51.476 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.466+0000 7f3ffa59c700 1 --2- 172.21.15.42:0/3698760333 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3ffc07e720 0x7f3ffc07eb50 secure :-1 s=THROTTLE_DONE pgs=2472 cs=0 l=1 rx=0x7f3ff0008790 tx=0x7f3ff000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:51.476 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.466+0000 7f3ffa59c700 1 --2- 172.21.15.42:0/3698760333 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3ffc07e720 0x7f3ffc07eb50 secure :-1 s=THROTTLE_DONE pgs=2472 cs=0 l=1 rx=0x7f3ff0008790 tx=0x7f3ff000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:51.477 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.466+0000 7f3ff9d9b700 1 -- 172.21.15.42:0/3698760333 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ff0008bb0 con 0x7f3ffc07e720 2019-12-05T06:19:51.477 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.466+0000 7f3ff9d9b700 1 -- 172.21.15.42:0/3698760333 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3ff0011990 con 0x7f3ffc07e720 2019-12-05T06:19:51.477 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.466+0000 7f4000cff700 1 -- 172.21.15.42:0/3698760333 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3ffc07e720 msgr2=0x7f3ffc07eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:51.477 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.466+0000 7f4000cff700 1 --2- 172.21.15.42:0/3698760333 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3ffc07e720 0x7f3ffc07eb50 secure :-1 s=READY pgs=2472 cs=0 l=1 rx=0x7f3ff0008790 tx=0x7f3ff000fd30).stop 2019-12-05T06:19:51.477 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.466+0000 7f4000cff700 1 -- 172.21.15.42:0/3698760333 shutdown_connections 2019-12-05T06:19:51.477 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.466+0000 7f4000cff700 1 --2- 172.21.15.42:0/3698760333 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3ffc0a7740 0x7f3ffc08df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:19:51.477 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.466+0000 7f4000cff700 1 --2- 172.21.15.42:0/3698760333 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3ffc07e720 0x7f3ffc07eb50 unknown :-1 s=CLOSED pgs=2472 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:51.478 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.466+0000 7f4000cff700 1 -- 172.21.15.42:0/3698760333 shutdown_connections 2019-12-05T06:19:51.478 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.466+0000 7f4000cff700 1 -- 172.21.15.42:0/3698760333 wait complete. 2019-12-05T06:19:51.478 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.466+0000 7f4000cff700 1 -- 172.21.15.42:0/3698760333 >> 172.21.15.42:0/3698760333 conn(0x7f3ffc0aabe0 msgr2=0x7f3ffc0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:19:51.478 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.466+0000 7f4000cff700 1 Processor -- start 2019-12-05T06:19:51.478 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.466+0000 7f4000cff700 1 -- start start 2019-12-05T06:19:51.478 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.466+0000 7f4000cff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3ffc14e630 0x7f3ffc14ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:19:51.478 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.466+0000 7f4000cff700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ffc143790 con 0x7f3ffc0a3760 2019-12-05T06:19:51.479 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.466+0000 7f4000cff700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ffc1316c0 con 0x7f3ffc07c7d0 2019-12-05T06:19:51.479 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.466+0000 7f4000cff700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ffc0a2bd0 con 0x7f3ffc14e630 2019-12-05T06:19:51.479 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.470+0000 7f3ffad9d700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f3ffc07c7d0 0x7f3ffc0a1fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:53158/0 (socket says 172.21.15.42:53158) 2019-12-05T06:19:51.479 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.470+0000 7f3ffad9d700 1 -- 172.21.15.42:0/1872299574 learned_addr learned my addr 172.21.15.42:0/1872299574 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:19:51.479 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.470+0000 7f3ffa59c700 1 --2- 172.21.15.42:0/1872299574 >> v2:172.21.15.42:3300/0 conn(0x7f3ffc14e630 0x7f3ffc14ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:51.479 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.470+0000 7f3feb7fe700 1 -- 172.21.15.42:0/1872299574 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1430078301 0 0) 0x7f3ffc1316c0 con 0x7f3ffc07c7d0 2019-12-05T06:19:51.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.470+0000 7f3feb7fe700 1 -- 172.21.15.42:0/1872299574 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3fe00030b0 con 0x7f3ffc07c7d0 2019-12-05T06:19:51.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.470+0000 7f3ffa59c700 1 -- 172.21.15.42:0/1872299574 >> v1:172.21.15.42:6789/0 conn(0x7f3ffc07c7d0 legacy=0x7f3ffc0a1fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:51.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.470+0000 7f3ffa59c700 1 -- 172.21.15.42:0/1872299574 >> v1:172.21.15.40:6789/0 conn(0x7f3ffc0a3760 legacy=0x7f3ffc0a3b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:51.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.470+0000 7f3ffa59c700 1 -- 172.21.15.42:0/1872299574 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ffc132900 con 0x7f3ffc14e630 2019-12-05T06:19:51.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.470+0000 7f3ffa59c700 1 --2- 172.21.15.42:0/1872299574 >> v2:172.21.15.42:3300/0 conn(0x7f3ffc14e630 0x7f3ffc14ea60 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f3ffc0a33e0 tx=0x7f3ff0011f80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:51.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.470+0000 7f3ffa59c700 1 --2- 172.21.15.42:0/1872299574 >> v2:172.21.15.42:3300/0 conn(0x7f3ffc14e630 0x7f3ffc14ea60 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f3ffc0a33e0 tx=0x7f3ff0011f80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:51.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.470+0000 7f3ffa59c700 1 --2- 172.21.15.42:0/1872299574 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3ffc14e630 0x7f3ffc14ea60 secure :-1 s=READY pgs=2474 cs=0 l=1 rx=0x7f3ffc0a33e0 tx=0x7f3ff0011f80).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:19:51.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.470+0000 7f3ffa59c700 1 --2- 172.21.15.42:0/1872299574 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3ffc14e630 0x7f3ffc14ea60 secure :-1 s=THROTTLE_DONE pgs=2474 cs=0 l=1 rx=0x7f3ffc0a33e0 tx=0x7f3ff0011f80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:51.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.470+0000 7f3feb7fe700 1 -- 172.21.15.42:0/1872299574 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3ff0007150 con 0x7f3ffc14e630 2019-12-05T06:19:51.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.470+0000 7f4000cff700 1 -- 172.21.15.42:0/1872299574 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ffc1306b0 con 0x7f3ffc14e630 2019-12-05T06:19:51.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.470+0000 7f3ffa59c700 1 --2- 172.21.15.42:0/1872299574 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3ffc14e630 0x7f3ffc14ea60 secure :-1 s=THROTTLE_DONE pgs=2474 cs=0 l=1 rx=0x7f3ffc0a33e0 tx=0x7f3ff0011f80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:51.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.470+0000 7f3ffa59c700 1 --2- 172.21.15.42:0/1872299574 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3ffc14e630 0x7f3ffc14ea60 secure :-1 s=THROTTLE_DONE pgs=2474 cs=0 l=1 rx=0x7f3ffc0a33e0 tx=0x7f3ff0011f80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:51.482 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.470+0000 7f4000cff700 1 -- 172.21.15.42:0/1872299574 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ffc130c60 con 0x7f3ffc14e630 2019-12-05T06:19:51.482 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.470+0000 7f3feb7fe700 1 -- 172.21.15.42:0/1872299574 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ff00115b0 con 0x7f3ffc14e630 2019-12-05T06:19:51.482 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.470+0000 7f3feb7fe700 1 -- 172.21.15.42:0/1872299574 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3ff0005730 con 0x7f3ffc14e630 2019-12-05T06:19:51.482 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.470+0000 7f3ffa59c700 1 --2- 172.21.15.42:0/1872299574 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3ffc14e630 0x7f3ffc14ea60 secure :-1 s=THROTTLE_DONE pgs=2474 cs=0 l=1 rx=0x7f3ffc0a33e0 tx=0x7f3ff0011f80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:51.483 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.470+0000 7f3feb7fe700 1 -- 172.21.15.42:0/1872299574 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f3ff0005a40 con 0x7f3ffc14e630 2019-12-05T06:19:51.483 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.470+0000 7f3ffa59c700 1 --2- 172.21.15.42:0/1872299574 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3ffc14e630 0x7f3ffc14ea60 secure :-1 s=THROTTLE_DONE pgs=2474 cs=0 l=1 rx=0x7f3ffc0a33e0 tx=0x7f3ff0011f80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:51.483 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.470+0000 7f3feb7fe700 1 --2- 172.21.15.42:0/1872299574 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3fe003b270 0x7f3fe003d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:19:51.484 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.470+0000 7f3feb7fe700 1 -- 172.21.15.42:0/1872299574 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 10589+0+0 (secure 0 0 0) 0x7f3ff004b470 con 0x7f3ffc14e630 2019-12-05T06:19:51.484 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.470+0000 7f3ffad9d700 1 --2- 172.21.15.42:0/1872299574 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3fe003b270 0x7f3fe003d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:51.484 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.470+0000 7f3ffad9d700 1 --2- 172.21.15.42:0/1872299574 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3fe003b270 0x7f3fe003d780 crc :-1 s=READY pgs=1037 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:19:51.484 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.478+0000 7f4000cff700 1 -- 172.21.15.42:0/1872299574 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fd4003200 con 0x7f3ffc14e630 2019-12-05T06:19:51.486 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.482+0000 7f3ffa59c700 1 --2- 172.21.15.42:0/1872299574 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3ffc14e630 0x7f3ffc14ea60 secure :-1 s=THROTTLE_DONE pgs=2474 cs=0 l=1 rx=0x7f3ffc0a33e0 tx=0x7f3ff0011f80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:51.486 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.482+0000 7f3feb7fe700 1 -- 172.21.15.42:0/1872299574 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f3ff0005d50 con 0x7f3ffc14e630 2019-12-05T06:19:51.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.646+0000 7f4000cff700 1 -- 172.21.15.42:0/1872299574 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3fd4003630 con 0x7f3ffc14e630 2019-12-05T06:19:51.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.646+0000 7f3ffa59c700 1 --2- 172.21.15.42:0/1872299574 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3ffc14e630 0x7f3ffc14ea60 secure :-1 s=THROTTLE_DONE pgs=2474 cs=0 l=1 rx=0x7f3ffc0a33e0 tx=0x7f3ff0011f80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:51.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.646+0000 7f3feb7fe700 1 -- 172.21.15.42:0/1872299574 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 294 v294) v1 ==== 95+0+1778 (secure 0 0 0) 0x7f3ff00301a0 con 0x7f3ffc14e630 2019-12-05T06:19:51.649 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 294 2019-12-05T06:19:51.650 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:19:51.650 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":294,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":294,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:19:49.764458+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":102,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10711},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10711":{"gid":10711,"name":"a","rank":0,"incarnation":294,"state":"up:replay","state_seq":2,"addr":"172.21.15.42:6835/2638662429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2638662429},{"type":"v1","addr":"172.21.15.42:6835","nonce":2638662429}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:19:51.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.646+0000 7f4000cff700 1 -- 172.21.15.42:0/1872299574 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3fe003b270 msgr2=0x7f3fe003d780 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:51.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.646+0000 7f4000cff700 1 --2- 172.21.15.42:0/1872299574 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3fe003b270 0x7f3fe003d780 crc :-1 s=READY pgs=1037 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:51.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.646+0000 7f4000cff700 1 -- 172.21.15.42:0/1872299574 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3ffc14e630 msgr2=0x7f3ffc14ea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:51.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.646+0000 7f4000cff700 1 --2- 172.21.15.42:0/1872299574 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3ffc14e630 0x7f3ffc14ea60 secure :-1 s=READY pgs=2474 cs=0 l=1 rx=0x7f3ffc0a33e0 tx=0x7f3ff0011f80).stop 2019-12-05T06:19:51.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.646+0000 7f4000cff700 1 -- 172.21.15.42:0/1872299574 shutdown_connections 2019-12-05T06:19:51.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.646+0000 7f4000cff700 1 --2- 172.21.15.42:0/1872299574 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3fe003b270 0x7f3fe003d780 unknown :-1 s=CLOSED pgs=1037 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:51.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.646+0000 7f4000cff700 1 --2- 172.21.15.42:0/1872299574 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3ffc14e630 0x7f3ffc14ea60 unknown :-1 s=CLOSED pgs=2474 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:51.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.646+0000 7f4000cff700 1 -- 172.21.15.42:0/1872299574 shutdown_connections 2019-12-05T06:19:51.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.646+0000 7f4000cff700 1 -- 172.21.15.42:0/1872299574 wait complete. 2019-12-05T06:19:51.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:51.646+0000 7f4000cff700 1 -- 172.21.15.42:0/1872299574 >> 172.21.15.42:0/1872299574 conn(0x7f3ffc0a8400 msgr2=0x7f3ffc0a4840 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:19:51.662 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 294, "feature_flags": { "enable_multiple": false, "ever_enabled_multiple": false }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2019-12-05T05:03:35.604738+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 294, "ever_allowed_features": 2, "explicitly_allowed_features": 2, "failed": [], "flags": 18, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_10711": { "addr": "172.21.15.42:6835/2638662429", "addrs": { "addrvec": [ { "addr": "172.21.15.42:6834", "nonce": 2638662429, "type": "v2" }, { "addr": "172.21.15.42:6835", "nonce": 2638662429, "type": "v1" } ] }, "export_targets": [], "features": 4540138292836696063, "flags": 0, "gid": 10711, "incarnation": 294, "name": "a", "rank": 0, "state": "up:replay", "state_seq": 2 } }, "last_failure": 0, "last_failure_osd_epoch": 102, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 2, "min_compat_client": "0 (unknown)", "modified": "2019-12-05T06:19:49.764458+0000", "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 10711 } } } ], "standbys": [] } 2019-12-05T06:19:53.663 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:19:53.768 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:53.758+0000 7f80411db700 1 Processor -- start 2019-12-05T06:19:53.769 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:53.758+0000 7f80411db700 1 -- start start 2019-12-05T06:19:53.769 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:53.762+0000 7f80411db700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f803c08df80 0x7f803c09f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:19:53.769 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:53.762+0000 7f80411db700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f803c09fc60 con 0x7f803c0a7740 2019-12-05T06:19:53.769 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:53.762+0000 7f80411db700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f803c0a0710 con 0x7f803c09f830 2019-12-05T06:19:53.769 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:53.762+0000 7f80411db700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f803c0a0b00 con 0x7f803c08df80 2019-12-05T06:19:53.769 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:53.762+0000 7f803ad9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f803c08df80 0x7f803c09f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:53.770 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:53.762+0000 7f803ad9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f803c08df80 0x7f803c09f2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:53132/0 (socket says 172.21.15.42:53132) 2019-12-05T06:19:53.770 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:53.762+0000 7f80409da700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f803c09f830 0x7f803c14efe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:53172/0 (socket says 172.21.15.42:53172) 2019-12-05T06:19:53.770 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:53.762+0000 7f803b59e700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f803c0a7740 0x7f803c07e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:34376/0 (socket says 172.21.15.42:34376) 2019-12-05T06:19:53.770 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:53.762+0000 7f803ad9d700 1 -- 172.21.15.42:0/1960156514 learned_addr learned my addr 172.21.15.42:0/1960156514 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:19:53.770 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:53.762+0000 7f803ad9d700 1 -- 172.21.15.42:0/1960156514 >> v1:172.21.15.42:6789/0 conn(0x7f803c09f830 legacy=0x7f803c14efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:53.770 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:53.762+0000 7f803ad9d700 1 -- 172.21.15.42:0/1960156514 >> v1:172.21.15.40:6789/0 conn(0x7f803c0a7740 legacy=0x7f803c07e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:53.770 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:53.762+0000 7f803a59c700 1 -- 172.21.15.42:0/1960156514 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2388895357 0 0) 0x7f803c09fc60 con 0x7f803c0a7740 2019-12-05T06:19:53.771 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:53.762+0000 7f803ad9d700 1 -- 172.21.15.42:0/1960156514 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f803c07ef20 con 0x7f803c08df80 2019-12-05T06:19:53.771 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:53.762+0000 7f803ad9d700 1 --2- 172.21.15.42:0/1960156514 >> v2:172.21.15.42:3300/0 conn(0x7f803c08df80 0x7f803c09f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f8030008790 tx=0x7f803000fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:53.771 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:53.762+0000 7f803ad9d700 1 --2- 172.21.15.42:0/1960156514 >> v2:172.21.15.42:3300/0 conn(0x7f803c08df80 0x7f803c09f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f8030008790 tx=0x7f803000fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:53.771 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:53.762+0000 7f803ad9d700 1 --2- 172.21.15.42:0/1960156514 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f803c08df80 0x7f803c09f2e0 secure :-1 s=READY pgs=2476 cs=0 l=1 rx=0x7f8030008790 tx=0x7f803000fcb0).ready entity=mon.1 client_cookie=80181bf85d4a4c9f server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:19:53.771 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:53.762+0000 7f803ad9d700 1 --2- 172.21.15.42:0/1960156514 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f803c08df80 0x7f803c09f2e0 secure :-1 s=THROTTLE_DONE pgs=2476 cs=0 l=1 rx=0x7f8030008790 tx=0x7f803000fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:53.771 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:53.762+0000 7f803a59c700 1 -- 172.21.15.42:0/1960156514 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8030011520 con 0x7f803c08df80 2019-12-05T06:19:53.771 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:53.762+0000 7f803ad9d700 1 --2- 172.21.15.42:0/1960156514 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f803c08df80 0x7f803c09f2e0 secure :-1 s=THROTTLE_DONE pgs=2476 cs=0 l=1 rx=0x7f8030008790 tx=0x7f803000fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:53.772 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:53.762+0000 7f803ad9d700 1 --2- 172.21.15.42:0/1960156514 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f803c08df80 0x7f803c09f2e0 secure :-1 s=THROTTLE_DONE pgs=2476 cs=0 l=1 rx=0x7f8030008790 tx=0x7f803000fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:53.772 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:53.762+0000 7f803a59c700 1 -- 172.21.15.42:0/1960156514 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8030008ca0 con 0x7f803c08df80 2019-12-05T06:19:53.772 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:53.762+0000 7f803a59c700 1 -- 172.21.15.42:0/1960156514 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8030011b00 con 0x7f803c08df80 2019-12-05T06:19:53.772 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:53.762+0000 7f80411db700 1 -- 172.21.15.42:0/1960156514 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f803c08df80 msgr2=0x7f803c09f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:53.772 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:53.762+0000 7f80411db700 1 --2- 172.21.15.42:0/1960156514 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f803c08df80 0x7f803c09f2e0 secure :-1 s=READY pgs=2476 cs=0 l=1 rx=0x7f8030008790 tx=0x7f803000fcb0).stop 2019-12-05T06:19:53.772 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:53.762+0000 7f80411db700 1 -- 172.21.15.42:0/1960156514 shutdown_connections 2019-12-05T06:19:53.772 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:53.762+0000 7f80411db700 1 --2- 172.21.15.42:0/1960156514 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f803c08df80 0x7f803c09f2e0 unknown :-1 s=CLOSED pgs=2476 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:53.772 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:53.762+0000 7f80411db700 1 -- 172.21.15.42:0/1960156514 shutdown_connections 2019-12-05T06:19:53.773 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:53.762+0000 7f80411db700 1 -- 172.21.15.42:0/1960156514 wait complete. 2019-12-05T06:19:53.773 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:53.762+0000 7f80411db700 1 -- 172.21.15.42:0/1960156514 >> 172.21.15.42:0/1960156514 conn(0x7f803c0aabe0 msgr2=0x7f803c0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:19:53.776 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:53.766+0000 7f80411db700 1 Processor -- start 2019-12-05T06:19:53.777 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:53.766+0000 7f80411db700 1 -- start start 2019-12-05T06:19:53.777 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:53.770+0000 7f80411db700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f803c07c8a0 0x7f803c121f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:19:53.777 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:53.770+0000 7f80411db700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f803c0a41d0 0x7f803c14f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:19:53.777 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:53.770+0000 7f80411db700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f803c14f620 0x7f803c143080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:19:53.777 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:53.770+0000 7f80411db700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f803c122740 con 0x7f803c07c8a0 2019-12-05T06:19:53.777 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:53.770+0000 7f80411db700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f803c1229e0 con 0x7f803c14f620 2019-12-05T06:19:53.777 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:53.770+0000 7f80411db700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f803c122c80 con 0x7f803c0a41d0 2019-12-05T06:19:53.778 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:53.770+0000 7f80409da700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f803c14f620 0x7f803c143080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:53.778 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:53.770+0000 7f80409da700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f803c14f620 0x7f803c143080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:53140/0 (socket says 172.21.15.42:53140) 2019-12-05T06:19:53.778 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:53.770+0000 7f80409da700 1 -- 172.21.15.42:0/3007565189 learned_addr learned my addr 172.21.15.42:0/3007565189 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:19:53.778 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:53.770+0000 7f803b59e700 1 --2- 172.21.15.42:0/3007565189 >> v2:172.21.15.40:3300/0 conn(0x7f803c07c8a0 0x7f803c121f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:53.778 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:53.770+0000 7f803ad9d700 1 --2- 172.21.15.42:0/3007565189 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f803c0a41d0 0x7f803c14f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:53.778 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:53.770+0000 7f80409da700 1 -- 172.21.15.42:0/3007565189 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f803c0a41d0 msgr2=0x7f803c14f0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:53.778 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:53.770+0000 7f80409da700 1 --2- 172.21.15.42:0/3007565189 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f803c0a41d0 0x7f803c14f0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:53.779 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:53.770+0000 7f80409da700 1 -- 172.21.15.42:0/3007565189 >> v2:172.21.15.40:3300/0 conn(0x7f803c07c8a0 msgr2=0x7f803c121f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:53.780 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:53.770+0000 7f80409da700 1 --2- 172.21.15.42:0/3007565189 >> v2:172.21.15.40:3300/0 conn(0x7f803c07c8a0 0x7f803c121f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:53.780 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:53.770+0000 7f80409da700 1 -- 172.21.15.42:0/3007565189 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f803c122460 con 0x7f803c14f620 2019-12-05T06:19:53.780 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:53.770+0000 7f80409da700 1 --2- 172.21.15.42:0/3007565189 >> v2:172.21.15.42:3300/0 conn(0x7f803c14f620 0x7f803c143080 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f803c122ba0 tx=0x7f8034010bd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:53.781 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:53.770+0000 7f80409da700 1 --2- 172.21.15.42:0/3007565189 >> v2:172.21.15.42:3300/0 conn(0x7f803c14f620 0x7f803c143080 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f803c122ba0 tx=0x7f8034010bd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:53.781 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:53.770+0000 7f80409da700 1 --2- 172.21.15.42:0/3007565189 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f803c14f620 0x7f803c143080 secure :-1 s=READY pgs=2477 cs=0 l=1 rx=0x7f803c122ba0 tx=0x7f8034010bd0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:19:53.781 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:53.770+0000 7f80409da700 1 --2- 172.21.15.42:0/3007565189 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f803c14f620 0x7f803c143080 secure :-1 s=THROTTLE_DONE pgs=2477 cs=0 l=1 rx=0x7f803c122ba0 tx=0x7f8034010bd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:53.782 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:53.770+0000 7f80409da700 1 --2- 172.21.15.42:0/3007565189 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f803c14f620 0x7f803c143080 secure :-1 s=THROTTLE_DONE pgs=2477 cs=0 l=1 rx=0x7f803c122ba0 tx=0x7f8034010bd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:53.782 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:53.770+0000 7f8023fff700 1 -- 172.21.15.42:0/3007565189 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8034004ad0 con 0x7f803c14f620 2019-12-05T06:19:53.782 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:53.770+0000 7f80409da700 1 --2- 172.21.15.42:0/3007565189 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f803c14f620 0x7f803c143080 secure :-1 s=THROTTLE_DONE pgs=2477 cs=0 l=1 rx=0x7f803c122ba0 tx=0x7f8034010bd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:53.782 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:53.770+0000 7f8023fff700 1 -- 172.21.15.42:0/3007565189 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80340046f0 con 0x7f803c14f620 2019-12-05T06:19:53.782 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:53.770+0000 7f80411db700 1 -- 172.21.15.42:0/3007565189 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f803c1435d0 con 0x7f803c14f620 2019-12-05T06:19:53.783 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:53.770+0000 7f8023fff700 1 -- 172.21.15.42:0/3007565189 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8034012740 con 0x7f803c14f620 2019-12-05T06:19:53.783 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:53.770+0000 7f80411db700 1 -- 172.21.15.42:0/3007565189 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f803c1254d0 con 0x7f803c14f620 2019-12-05T06:19:53.783 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:53.770+0000 7f80409da700 1 --2- 172.21.15.42:0/3007565189 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f803c14f620 0x7f803c143080 secure :-1 s=THROTTLE_DONE pgs=2477 cs=0 l=1 rx=0x7f803c122ba0 tx=0x7f8034010bd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:53.783 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:53.774+0000 7f8023fff700 1 -- 172.21.15.42:0/3007565189 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f8034012a10 con 0x7f803c14f620 2019-12-05T06:19:53.783 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:53.774+0000 7f80409da700 1 --2- 172.21.15.42:0/3007565189 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f803c14f620 0x7f803c143080 secure :-1 s=THROTTLE_DONE pgs=2477 cs=0 l=1 rx=0x7f803c122ba0 tx=0x7f8034010bd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:53.783 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:53.774+0000 7f8023fff700 1 --2- 172.21.15.42:0/3007565189 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f801c03add0 0x7f801c03d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:19:53.783 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:53.774+0000 7f8023fff700 1 -- 172.21.15.42:0/3007565189 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 10589+0+0 (secure 0 0 0) 0x7f803404b3c0 con 0x7f803c14f620 2019-12-05T06:19:53.784 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:53.774+0000 7f803b59e700 1 --2- 172.21.15.42:0/3007565189 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f801c03add0 0x7f801c03d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:53.784 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:53.774+0000 7f803b59e700 1 --2- 172.21.15.42:0/3007565189 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f801c03add0 0x7f801c03d2e0 crc :-1 s=READY pgs=1038 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:19:53.784 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:53.774+0000 7f80411db700 1 -- 172.21.15.42:0/3007565189 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f803c14fa50 con 0x7f803c14f620 2019-12-05T06:19:53.788 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:53.786+0000 7f80409da700 1 --2- 172.21.15.42:0/3007565189 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f803c14f620 0x7f803c143080 secure :-1 s=THROTTLE_DONE pgs=2477 cs=0 l=1 rx=0x7f803c122ba0 tx=0x7f8034010bd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:53.788 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:53.786+0000 7f8023fff700 1 -- 172.21.15.42:0/3007565189 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f80340326a0 con 0x7f803c14f620 2019-12-05T06:19:53.957 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:53.954+0000 7f80411db700 1 -- 172.21.15.42:0/3007565189 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f803c05d110 con 0x7f803c14f620 2019-12-05T06:19:53.958 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:53.954+0000 7f80409da700 1 --2- 172.21.15.42:0/3007565189 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f803c14f620 0x7f803c143080 secure :-1 s=THROTTLE_DONE pgs=2477 cs=0 l=1 rx=0x7f803c122ba0 tx=0x7f8034010bd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:53.958 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:53.954+0000 7f8023fff700 1 -- 172.21.15.42:0/3007565189 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 295 v295) v1 ==== 95+0+1782 (secure 0 0 0) 0x7f8034012d20 con 0x7f803c14f620 2019-12-05T06:19:53.960 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:19:53.960 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":295,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":295,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:19:53.094961+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":102,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10711},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10711":{"gid":10711,"name":"a","rank":0,"incarnation":294,"state":"up:reconnect","state_seq":10,"addr":"172.21.15.42:6835/2638662429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2638662429},{"type":"v1","addr":"172.21.15.42:6835","nonce":2638662429}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:19:53.961 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 295 2019-12-05T06:19:53.963 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:53.958+0000 7f80411db700 1 -- 172.21.15.42:0/3007565189 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f801c03add0 msgr2=0x7f801c03d2e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:53.963 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:53.958+0000 7f80411db700 1 --2- 172.21.15.42:0/3007565189 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f801c03add0 0x7f801c03d2e0 crc :-1 s=READY pgs=1038 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:53.963 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:53.958+0000 7f80411db700 1 -- 172.21.15.42:0/3007565189 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f803c14f620 msgr2=0x7f803c143080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:53.963 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:53.958+0000 7f80411db700 1 --2- 172.21.15.42:0/3007565189 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f803c14f620 0x7f803c143080 secure :-1 s=READY pgs=2477 cs=0 l=1 rx=0x7f803c122ba0 tx=0x7f8034010bd0).stop 2019-12-05T06:19:53.963 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:53.958+0000 7f80411db700 1 -- 172.21.15.42:0/3007565189 shutdown_connections 2019-12-05T06:19:53.963 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:53.958+0000 7f80411db700 1 --2- 172.21.15.42:0/3007565189 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f801c03add0 0x7f801c03d2e0 unknown :-1 s=CLOSED pgs=1038 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:53.964 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:53.958+0000 7f80411db700 1 --2- 172.21.15.42:0/3007565189 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f803c14f620 0x7f803c143080 unknown :-1 s=CLOSED pgs=2477 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:53.964 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:53.958+0000 7f80411db700 1 --2- 172.21.15.42:0/3007565189 >> v2:172.21.15.40:3300/0 conn(0x7f803c07c8a0 0x7f803c121f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:53.964 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:53.958+0000 7f80411db700 1 --2- 172.21.15.42:0/3007565189 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f803c0a41d0 0x7f803c14f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:53.964 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:53.958+0000 7f80411db700 1 -- 172.21.15.42:0/3007565189 shutdown_connections 2019-12-05T06:19:53.964 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:53.958+0000 7f80411db700 1 -- 172.21.15.42:0/3007565189 wait complete. 2019-12-05T06:19:53.964 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:53.958+0000 7f80411db700 1 -- 172.21.15.42:0/3007565189 >> 172.21.15.42:0/3007565189 conn(0x7f803c08fde0 msgr2=0x7f803c14ff20 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:19:53.973 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2019-12-05T06:19:55.975 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:19:56.080 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:56.070+0000 7fd37f93d700 1 Processor -- start 2019-12-05T06:19:56.080 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:56.070+0000 7fd37f93d700 1 -- start start 2019-12-05T06:19:56.080 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:56.070+0000 7fd37f93d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd37808df80 0x7fd37809f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:19:56.080 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:56.070+0000 7fd37f93d700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd37809fc60 con 0x7fd3780a7740 2019-12-05T06:19:56.081 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:56.070+0000 7fd37f93d700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3780a0710 con 0x7fd37809f830 2019-12-05T06:19:56.081 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:56.070+0000 7fd37f93d700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3780a0b00 con 0x7fd37808df80 2019-12-05T06:19:56.081 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:56.070+0000 7fd37d6d9700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd37808df80 0x7fd37809f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:56.081 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:56.070+0000 7fd37d6d9700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd37808df80 0x7fd37809f2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:53152/0 (socket says 172.21.15.42:53152) 2019-12-05T06:19:56.081 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:56.070+0000 7fd37d6d9700 1 -- 172.21.15.42:0/3556354411 learned_addr learned my addr 172.21.15.42:0/3556354411 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:19:56.082 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:56.070+0000 7fd37ced8700 1 -- 172.21.15.42:0/3556354411 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 341629032 0 0) 0x7fd37809fc60 con 0x7fd3780a7740 2019-12-05T06:19:56.082 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:56.070+0000 7fd37d6d9700 1 -- 172.21.15.42:0/3556354411 >> v1:172.21.15.42:6789/0 conn(0x7fd37809f830 legacy=0x7fd37814efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:56.082 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:56.070+0000 7fd37d6d9700 1 -- 172.21.15.42:0/3556354411 >> v1:172.21.15.40:6789/0 conn(0x7fd3780a7740 legacy=0x7fd37807e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:56.082 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:56.070+0000 7fd37d6d9700 1 -- 172.21.15.42:0/3556354411 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd37807ef20 con 0x7fd37808df80 2019-12-05T06:19:56.082 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:56.070+0000 7fd37d6d9700 1 --2- 172.21.15.42:0/3556354411 >> v2:172.21.15.42:3300/0 conn(0x7fd37808df80 0x7fd37809f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fd368008790 tx=0x7fd36800fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:56.082 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:56.070+0000 7fd37d6d9700 1 --2- 172.21.15.42:0/3556354411 >> v2:172.21.15.42:3300/0 conn(0x7fd37808df80 0x7fd37809f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fd368008790 tx=0x7fd36800fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:56.082 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:56.070+0000 7fd37d6d9700 1 --2- 172.21.15.42:0/3556354411 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd37808df80 0x7fd37809f2e0 secure :-1 s=READY pgs=2479 cs=0 l=1 rx=0x7fd368008790 tx=0x7fd36800fcb0).ready entity=mon.1 client_cookie=47d6df62386d089d server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:19:56.083 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:56.070+0000 7fd37d6d9700 1 --2- 172.21.15.42:0/3556354411 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd37808df80 0x7fd37809f2e0 secure :-1 s=THROTTLE_DONE pgs=2479 cs=0 l=1 rx=0x7fd368008790 tx=0x7fd36800fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:56.083 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:56.070+0000 7fd37d6d9700 1 --2- 172.21.15.42:0/3556354411 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd37808df80 0x7fd37809f2e0 secure :-1 s=THROTTLE_DONE pgs=2479 cs=0 l=1 rx=0x7fd368008790 tx=0x7fd36800fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:56.083 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:56.070+0000 7fd37ced8700 1 -- 172.21.15.42:0/3556354411 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd368011520 con 0x7fd37808df80 2019-12-05T06:19:56.083 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:56.070+0000 7fd37d6d9700 1 --2- 172.21.15.42:0/3556354411 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd37808df80 0x7fd37809f2e0 secure :-1 s=THROTTLE_DONE pgs=2479 cs=0 l=1 rx=0x7fd368008790 tx=0x7fd36800fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:56.083 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:56.070+0000 7fd37ced8700 1 -- 172.21.15.42:0/3556354411 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd368008ca0 con 0x7fd37808df80 2019-12-05T06:19:56.083 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:56.070+0000 7fd37ced8700 1 -- 172.21.15.42:0/3556354411 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd368011b00 con 0x7fd37808df80 2019-12-05T06:19:56.084 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:56.070+0000 7fd37f93d700 1 -- 172.21.15.42:0/3556354411 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd37808df80 msgr2=0x7fd37809f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:56.084 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:56.070+0000 7fd37f93d700 1 --2- 172.21.15.42:0/3556354411 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd37808df80 0x7fd37809f2e0 secure :-1 s=READY pgs=2479 cs=0 l=1 rx=0x7fd368008790 tx=0x7fd36800fcb0).stop 2019-12-05T06:19:56.084 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:56.074+0000 7fd37f93d700 1 -- 172.21.15.42:0/3556354411 shutdown_connections 2019-12-05T06:19:56.084 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:56.074+0000 7fd37f93d700 1 --2- 172.21.15.42:0/3556354411 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd37808df80 0x7fd37809f2e0 unknown :-1 s=CLOSED pgs=2479 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:56.084 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:56.074+0000 7fd37f93d700 1 -- 172.21.15.42:0/3556354411 shutdown_connections 2019-12-05T06:19:56.085 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:56.074+0000 7fd37f93d700 1 -- 172.21.15.42:0/3556354411 wait complete. 2019-12-05T06:19:56.085 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:56.074+0000 7fd37f93d700 1 -- 172.21.15.42:0/3556354411 >> 172.21.15.42:0/3556354411 conn(0x7fd3780aabe0 msgr2=0x7fd3780a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:19:56.085 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:56.074+0000 7fd37f93d700 1 Processor -- start 2019-12-05T06:19:56.085 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:56.074+0000 7fd37f93d700 1 -- start start 2019-12-05T06:19:56.085 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:56.074+0000 7fd37f93d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd3780a1510 0x7fd3780a1940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:19:56.086 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:56.074+0000 7fd37f93d700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd378122ec0 0x7fd3781232f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:19:56.086 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:56.074+0000 7fd37f93d700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd37807deb0 con 0x7fd3780a25b0 2019-12-05T06:19:56.086 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:56.074+0000 7fd37f93d700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3780a3de0 con 0x7fd3780a1510 2019-12-05T06:19:56.086 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:56.074+0000 7fd37f93d700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3780a4080 con 0x7fd378122ec0 2019-12-05T06:19:56.086 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:56.074+0000 7fd37d6d9700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd3780a1510 0x7fd3780a1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:56.086 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:56.074+0000 7fd37d6d9700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd3780a1510 0x7fd3780a1940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:53158/0 (socket says 172.21.15.42:53158) 2019-12-05T06:19:56.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:56.074+0000 7fd37d6d9700 1 -- 172.21.15.42:0/588722693 learned_addr learned my addr 172.21.15.42:0/588722693 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:19:56.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:56.074+0000 7fd37e6db700 1 --2- 172.21.15.42:0/588722693 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd378122ec0 0x7fd3781232f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:56.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:56.074+0000 7fd37d6d9700 1 -- 172.21.15.42:0/588722693 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd378122ec0 msgr2=0x7fd3781232f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:56.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:56.074+0000 7fd37d6d9700 1 --2- 172.21.15.42:0/588722693 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd378122ec0 0x7fd3781232f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:56.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:56.074+0000 7fd37d6d9700 1 -- 172.21.15.42:0/588722693 >> v1:172.21.15.42:6789/0 conn(0x7fd3780a25b0 legacy=0x7fd37811a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:56.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:56.074+0000 7fd37d6d9700 1 -- 172.21.15.42:0/588722693 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3780a4320 con 0x7fd3780a1510 2019-12-05T06:19:56.088 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:56.074+0000 7fd37d6d9700 1 --2- 172.21.15.42:0/588722693 >> v2:172.21.15.42:3300/0 conn(0x7fd3780a1510 0x7fd3780a1940 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fd37811c510 tx=0x7fd368011640).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:56.088 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:56.074+0000 7fd37d6d9700 1 --2- 172.21.15.42:0/588722693 >> v2:172.21.15.42:3300/0 conn(0x7fd3780a1510 0x7fd3780a1940 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fd37811c510 tx=0x7fd368011640).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:56.088 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:56.074+0000 7fd37d6d9700 1 --2- 172.21.15.42:0/588722693 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd3780a1510 0x7fd3780a1940 secure :-1 s=READY pgs=2481 cs=0 l=1 rx=0x7fd37811c510 tx=0x7fd368011640).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:19:56.088 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:56.074+0000 7fd37d6d9700 1 --2- 172.21.15.42:0/588722693 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd3780a1510 0x7fd3780a1940 secure :-1 s=THROTTLE_DONE pgs=2481 cs=0 l=1 rx=0x7fd37811c510 tx=0x7fd368011640).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:56.088 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:56.074+0000 7fd37d6d9700 1 --2- 172.21.15.42:0/588722693 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd3780a1510 0x7fd3780a1940 secure :-1 s=THROTTLE_DONE pgs=2481 cs=0 l=1 rx=0x7fd37811c510 tx=0x7fd368011640).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:56.088 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:56.074+0000 7fd36e7fc700 1 -- 172.21.15.42:0/588722693 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd368007150 con 0x7fd3780a1510 2019-12-05T06:19:56.089 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:56.074+0000 7fd37d6d9700 1 --2- 172.21.15.42:0/588722693 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd3780a1510 0x7fd3780a1940 secure :-1 s=THROTTLE_DONE pgs=2481 cs=0 l=1 rx=0x7fd37811c510 tx=0x7fd368011640).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:56.089 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:56.074+0000 7fd36e7fc700 1 -- 172.21.15.42:0/588722693 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd368011720 con 0x7fd3780a1510 2019-12-05T06:19:56.089 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:56.074+0000 7fd37f93d700 1 -- 172.21.15.42:0/588722693 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd378123840 con 0x7fd3780a1510 2019-12-05T06:19:56.089 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:56.074+0000 7fd36e7fc700 1 -- 172.21.15.42:0/588722693 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd368005730 con 0x7fd3780a1510 2019-12-05T06:19:56.089 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:56.074+0000 7fd37f93d700 1 -- 172.21.15.42:0/588722693 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd37812a280 con 0x7fd3780a1510 2019-12-05T06:19:56.091 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:56.078+0000 7fd37d6d9700 1 --2- 172.21.15.42:0/588722693 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd3780a1510 0x7fd3780a1940 secure :-1 s=THROTTLE_DONE pgs=2481 cs=0 l=1 rx=0x7fd37811c510 tx=0x7fd368011640).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:56.091 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:56.078+0000 7fd36e7fc700 1 -- 172.21.15.42:0/588722693 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fd368005a40 con 0x7fd3780a1510 2019-12-05T06:19:56.092 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:56.078+0000 7fd37d6d9700 1 --2- 172.21.15.42:0/588722693 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd3780a1510 0x7fd3780a1940 secure :-1 s=THROTTLE_DONE pgs=2481 cs=0 l=1 rx=0x7fd37811c510 tx=0x7fd368011640).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:56.092 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:56.078+0000 7fd36e7fc700 1 --2- 172.21.15.42:0/588722693 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd36003ad60 0x7fd36003d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:19:56.092 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:56.078+0000 7fd36e7fc700 1 -- 172.21.15.42:0/588722693 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 10589+0+0 (secure 0 0 0) 0x7fd36804b810 con 0x7fd3780a1510 2019-12-05T06:19:56.093 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:56.078+0000 7fd37deda700 1 --2- 172.21.15.42:0/588722693 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd36003ad60 0x7fd36003d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:56.093 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:56.078+0000 7fd37deda700 1 --2- 172.21.15.42:0/588722693 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd36003ad60 0x7fd36003d270 crc :-1 s=READY pgs=1039 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:19:56.093 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:56.082+0000 7fd37f93d700 1 -- 172.21.15.42:0/588722693 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd37805d110 con 0x7fd3780a1510 2019-12-05T06:19:56.093 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:56.090+0000 7fd37d6d9700 1 --2- 172.21.15.42:0/588722693 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd3780a1510 0x7fd3780a1940 secure :-1 s=THROTTLE_DONE pgs=2481 cs=0 l=1 rx=0x7fd37811c510 tx=0x7fd368011640).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:56.094 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:56.090+0000 7fd36e7fc700 1 -- 172.21.15.42:0/588722693 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fd368032890 con 0x7fd3780a1510 2019-12-05T06:19:56.273 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:56.270+0000 7fd37f93d700 1 -- 172.21.15.42:0/588722693 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd37805b8c0 con 0x7fd3780a1510 2019-12-05T06:19:56.274 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:56.270+0000 7fd37d6d9700 1 --2- 172.21.15.42:0/588722693 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd3780a1510 0x7fd3780a1940 secure :-1 s=THROTTLE_DONE pgs=2481 cs=0 l=1 rx=0x7fd37811c510 tx=0x7fd368011640).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:56.274 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:56.270+0000 7fd36e7fc700 1 -- 172.21.15.42:0/588722693 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 297 v297) v1 ==== 95+0+1779 (secure 0 0 0) 0x7fd368005d50 con 0x7fd3780a1510 2019-12-05T06:19:56.275 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 297 2019-12-05T06:19:56.275 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:19:56.275 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":297,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":297,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:19:55.115618+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":102,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10711},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10711":{"gid":10711,"name":"a","rank":0,"incarnation":294,"state":"up:active","state_seq":12,"addr":"172.21.15.42:6835/2638662429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2638662429},{"type":"v1","addr":"172.21.15.42:6835","nonce":2638662429}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:19:56.277 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:56.274+0000 7fd37f93d700 1 -- 172.21.15.42:0/588722693 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd36003ad60 msgr2=0x7fd36003d270 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:56.277 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:56.274+0000 7fd37f93d700 1 --2- 172.21.15.42:0/588722693 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd36003ad60 0x7fd36003d270 crc :-1 s=READY pgs=1039 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:56.278 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:56.274+0000 7fd37f93d700 1 -- 172.21.15.42:0/588722693 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd3780a1510 msgr2=0x7fd3780a1940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:56.278 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:56.274+0000 7fd37f93d700 1 --2- 172.21.15.42:0/588722693 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd3780a1510 0x7fd3780a1940 secure :-1 s=READY pgs=2481 cs=0 l=1 rx=0x7fd37811c510 tx=0x7fd368011640).stop 2019-12-05T06:19:56.278 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:56.274+0000 7fd37f93d700 1 -- 172.21.15.42:0/588722693 shutdown_connections 2019-12-05T06:19:56.278 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:56.274+0000 7fd37f93d700 1 --2- 172.21.15.42:0/588722693 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd36003ad60 0x7fd36003d270 unknown :-1 s=CLOSED pgs=1039 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:56.278 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:56.274+0000 7fd37f93d700 1 --2- 172.21.15.42:0/588722693 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd378122ec0 0x7fd3781232f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:56.278 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:56.274+0000 7fd37f93d700 1 --2- 172.21.15.42:0/588722693 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd3780a1510 0x7fd3780a1940 unknown :-1 s=CLOSED pgs=2481 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:56.279 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:56.274+0000 7fd37f93d700 1 -- 172.21.15.42:0/588722693 shutdown_connections 2019-12-05T06:19:56.279 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:56.274+0000 7fd37f93d700 1 -- 172.21.15.42:0/588722693 wait complete. 2019-12-05T06:19:56.279 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:56.274+0000 7fd37f93d700 1 -- 172.21.15.42:0/588722693 >> 172.21.15.42:0/588722693 conn(0x7fd3780a7f90 msgr2=0x7fd37808eb70 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:19:56.287 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a has gained rank=0, replacing gid=10476 2019-12-05T06:19:56.288 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 1 secs before reviving mds.b 2019-12-05T06:19:57.289 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.b 2019-12-05T06:19:57.289 INFO:tasks.ceph.mds.b:Restarting daemon 2019-12-05T06:19:57.289 INFO:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2019-12-05T06:19:57.294 INFO:tasks.ceph.mds.b:Started 2019-12-05T06:19:57.294 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:19:57.355 INFO:tasks.ceph.mds.b.smithi040.stdout:starting mds.b at 2019-12-05T06:19:57.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:57.394+0000 7fd2c1256700 1 Processor -- start 2019-12-05T06:19:57.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:57.394+0000 7fd2c1256700 1 -- start start 2019-12-05T06:19:57.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:57.394+0000 7fd2c1256700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd2bc0a7740 0x7fd2bc08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:19:57.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:57.394+0000 7fd2c1256700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd2bc07e720 0x7fd2bc07eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:19:57.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:57.394+0000 7fd2c1256700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2bc0a0710 con 0x7fd2bc09f2e0 2019-12-05T06:19:57.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:57.394+0000 7fd2c1256700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2bc0a0b00 con 0x7fd2bc0a7740 2019-12-05T06:19:57.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:57.394+0000 7fd2c1256700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2bc08e5b0 con 0x7fd2bc07e720 2019-12-05T06:19:57.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:57.394+0000 7fd2bad9d700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd2bc07e720 0x7fd2bc07eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:57.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:57.394+0000 7fd2bb59e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd2bc0a7740 0x7fd2bc08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:57.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:57.394+0000 7fd2c0a55700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fd2bc09f2e0 0x7fd2bc09f710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:53212/0 (socket says 172.21.15.42:53212) 2019-12-05T06:19:57.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:57.394+0000 7fd2bad9d700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd2bc07e720 0x7fd2bc07eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:53636/0 (socket says 172.21.15.42:53636) 2019-12-05T06:19:57.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:57.394+0000 7fd2c0a55700 1 -- 172.21.15.42:0/1136685230 learned_addr learned my addr 172.21.15.42:0/1136685230 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:19:57.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:57.394+0000 7fd2bad9d700 1 -- 172.21.15.42:0/1136685230 >> v2:172.21.15.42:3300/0 conn(0x7fd2bc0a7740 msgr2=0x7fd2bc08df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:19:57.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:57.394+0000 7fd2bad9d700 1 --2- 172.21.15.42:0/1136685230 >> v2:172.21.15.42:3300/0 conn(0x7fd2bc0a7740 0x7fd2bc08df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:19:57.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:57.394+0000 7fd2bad9d700 1 -- 172.21.15.42:0/1136685230 >> v1:172.21.15.42:6789/0 conn(0x7fd2bc09f2e0 legacy=0x7fd2bc09f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:57.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:57.394+0000 7fd2bad9d700 1 -- 172.21.15.42:0/1136685230 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2bc07f0a0 con 0x7fd2bc07e720 2019-12-05T06:19:57.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:57.398+0000 7fd2bad9d700 1 --2- 172.21.15.42:0/1136685230 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd2bc07e720 0x7fd2bc07eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fd2ac008b10 tx=0x7fd2ac00fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:57.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:57.398+0000 7fd2bad9d700 1 --2- 172.21.15.42:0/1136685230 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd2bc07e720 0x7fd2bc07eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fd2ac008b10 tx=0x7fd2ac00fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:57.411 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:57.398+0000 7fd2bad9d700 1 --2- 172.21.15.42:0/1136685230 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd2bc07e720 0x7fd2bc07eb50 secure :-1 s=READY pgs=540 cs=0 l=1 rx=0x7fd2ac008b10 tx=0x7fd2ac00fff0).ready entity=mon.2 client_cookie=47f3f6cea3d2120 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:19:57.411 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:57.398+0000 7fd2bad9d700 1 --2- 172.21.15.42:0/1136685230 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd2bc07e720 0x7fd2bc07eb50 secure :-1 s=THROTTLE_DONE pgs=540 cs=0 l=1 rx=0x7fd2ac008b10 tx=0x7fd2ac00fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:57.411 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:57.398+0000 7fd2ba59c700 1 -- 172.21.15.42:0/1136685230 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd2ac011a50 con 0x7fd2bc07e720 2019-12-05T06:19:57.411 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:57.398+0000 7fd2bad9d700 1 --2- 172.21.15.42:0/1136685230 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd2bc07e720 0x7fd2bc07eb50 secure :-1 s=THROTTLE_DONE pgs=540 cs=0 l=1 rx=0x7fd2ac008b10 tx=0x7fd2ac00fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:57.411 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:57.398+0000 7fd2ba59c700 1 -- 172.21.15.42:0/1136685230 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2ac011670 con 0x7fd2bc07e720 2019-12-05T06:19:57.411 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:57.398+0000 7fd2bad9d700 1 --2- 172.21.15.42:0/1136685230 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd2bc07e720 0x7fd2bc07eb50 secure :-1 s=THROTTLE_DONE pgs=540 cs=0 l=1 rx=0x7fd2ac008b10 tx=0x7fd2ac00fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:57.411 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:57.398+0000 7fd2c1256700 1 -- 172.21.15.42:0/1136685230 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd2bc07e720 msgr2=0x7fd2bc07eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:57.412 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:57.398+0000 7fd2ba59c700 1 -- 172.21.15.42:0/1136685230 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd2ac00a350 con 0x7fd2bc07e720 2019-12-05T06:19:57.413 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:57.398+0000 7fd2c1256700 1 --2- 172.21.15.42:0/1136685230 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd2bc07e720 0x7fd2bc07eb50 secure :-1 s=READY pgs=540 cs=0 l=1 rx=0x7fd2ac008b10 tx=0x7fd2ac00fff0).stop 2019-12-05T06:19:57.413 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:57.398+0000 7fd2c1256700 1 -- 172.21.15.42:0/1136685230 shutdown_connections 2019-12-05T06:19:57.413 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:57.398+0000 7fd2c1256700 1 --2- 172.21.15.42:0/1136685230 >> v2:172.21.15.42:3300/0 conn(0x7fd2bc0a7740 0x7fd2bc08df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:19:57.413 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:57.398+0000 7fd2c1256700 1 --2- 172.21.15.42:0/1136685230 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd2bc07e720 0x7fd2bc07eb50 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:57.413 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:57.398+0000 7fd2c1256700 1 -- 172.21.15.42:0/1136685230 shutdown_connections 2019-12-05T06:19:57.413 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:57.398+0000 7fd2c1256700 1 -- 172.21.15.42:0/1136685230 wait complete. 2019-12-05T06:19:57.413 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:57.398+0000 7fd2c1256700 1 -- 172.21.15.42:0/1136685230 >> 172.21.15.42:0/1136685230 conn(0x7fd2bc0aabe0 msgr2=0x7fd2bc0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:19:57.414 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:57.398+0000 7fd2c1256700 1 Processor -- start 2019-12-05T06:19:57.414 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:57.398+0000 7fd2c1256700 1 -- start start 2019-12-05T06:19:57.414 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:57.402+0000 7fd2c1256700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd2bc1444e0 0x7fd2bc144910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:19:57.414 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:57.402+0000 7fd2c1256700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2bc136750 con 0x7fd2bc12ad30 2019-12-05T06:19:57.414 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:57.402+0000 7fd2c1256700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2bc137990 con 0x7fd2bc12ba30 2019-12-05T06:19:57.414 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:57.402+0000 7fd2c1256700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2bc143d40 con 0x7fd2bc1444e0 2019-12-05T06:19:57.414 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:57.402+0000 7fd2bad9d700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fd2bc12ba30 0x7fd2bc142d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:53214/0 (socket says 172.21.15.42:53214) 2019-12-05T06:19:57.415 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:57.402+0000 7fd2bad9d700 1 -- 172.21.15.42:0/4063967275 learned_addr learned my addr 172.21.15.42:0/4063967275 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:19:57.415 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:57.402+0000 7fd2c0a55700 1 --2- 172.21.15.42:0/4063967275 >> v2:172.21.15.42:3300/0 conn(0x7fd2bc1444e0 0x7fd2bc144910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:57.415 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:57.402+0000 7fd2b3fff700 1 -- 172.21.15.42:0/4063967275 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2966234972 0 0) 0x7fd2bc136750 con 0x7fd2bc12ad30 2019-12-05T06:19:57.415 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:57.402+0000 7fd2b3fff700 1 -- 172.21.15.42:0/4063967275 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2a00030b0 con 0x7fd2bc12ad30 2019-12-05T06:19:57.415 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:57.402+0000 7fd2c0a55700 1 -- 172.21.15.42:0/4063967275 >> v1:172.21.15.42:6789/0 conn(0x7fd2bc12ba30 legacy=0x7fd2bc142d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:57.415 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:57.402+0000 7fd2c0a55700 1 -- 172.21.15.42:0/4063967275 >> v1:172.21.15.40:6789/0 conn(0x7fd2bc12ad30 legacy=0x7fd2bc118020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:57.415 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:57.402+0000 7fd2b3fff700 1 -- 172.21.15.42:0/4063967275 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3569600375 0 0) 0x7fd2bc137990 con 0x7fd2bc12ba30 2019-12-05T06:19:57.416 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:57.402+0000 7fd2c0a55700 1 -- 172.21.15.42:0/4063967275 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2bc138bd0 con 0x7fd2bc1444e0 2019-12-05T06:19:57.416 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:57.402+0000 7fd2c0a55700 1 --2- 172.21.15.42:0/4063967275 >> v2:172.21.15.42:3300/0 conn(0x7fd2bc1444e0 0x7fd2bc144910 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fd2b40043a0 tx=0x7fd2b40107d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:57.416 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:57.402+0000 7fd2c0a55700 1 --2- 172.21.15.42:0/4063967275 >> v2:172.21.15.42:3300/0 conn(0x7fd2bc1444e0 0x7fd2bc144910 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fd2b40043a0 tx=0x7fd2b40107d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:57.416 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:57.402+0000 7fd2c0a55700 1 --2- 172.21.15.42:0/4063967275 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd2bc1444e0 0x7fd2bc144910 secure :-1 s=READY pgs=2486 cs=0 l=1 rx=0x7fd2b40043a0 tx=0x7fd2b40107d0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:19:57.416 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:57.402+0000 7fd2c0a55700 1 --2- 172.21.15.42:0/4063967275 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd2bc1444e0 0x7fd2bc144910 secure :-1 s=THROTTLE_DONE pgs=2486 cs=0 l=1 rx=0x7fd2b40043a0 tx=0x7fd2b40107d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:57.416 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:57.402+0000 7fd2b3fff700 1 -- 172.21.15.42:0/4063967275 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd2b40115f0 con 0x7fd2bc1444e0 2019-12-05T06:19:57.416 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:57.402+0000 7fd2c0a55700 1 --2- 172.21.15.42:0/4063967275 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd2bc1444e0 0x7fd2bc144910 secure :-1 s=THROTTLE_DONE pgs=2486 cs=0 l=1 rx=0x7fd2b40043a0 tx=0x7fd2b40107d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:57.417 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:57.402+0000 7fd2b3fff700 1 -- 172.21.15.42:0/4063967275 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2b4011210 con 0x7fd2bc1444e0 2019-12-05T06:19:57.417 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:57.402+0000 7fd2c0a55700 1 --2- 172.21.15.42:0/4063967275 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd2bc1444e0 0x7fd2bc144910 secure :-1 s=THROTTLE_DONE pgs=2486 cs=0 l=1 rx=0x7fd2b40043a0 tx=0x7fd2b40107d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:57.417 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:57.402+0000 7fd2c1256700 1 -- 172.21.15.42:0/4063967275 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2bc136980 con 0x7fd2bc1444e0 2019-12-05T06:19:57.417 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:57.402+0000 7fd2b3fff700 1 -- 172.21.15.42:0/4063967275 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd2b4011bd0 con 0x7fd2bc1444e0 2019-12-05T06:19:57.417 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:57.402+0000 7fd2c1256700 1 -- 172.21.15.42:0/4063967275 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2bc136e50 con 0x7fd2bc1444e0 2019-12-05T06:19:57.418 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:57.402+0000 7fd2c0a55700 1 --2- 172.21.15.42:0/4063967275 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd2bc1444e0 0x7fd2bc144910 secure :-1 s=THROTTLE_DONE pgs=2486 cs=0 l=1 rx=0x7fd2b40043a0 tx=0x7fd2b40107d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:57.419 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:57.406+0000 7fd2b3fff700 1 -- 172.21.15.42:0/4063967275 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fd2b400b330 con 0x7fd2bc1444e0 2019-12-05T06:19:57.419 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:57.406+0000 7fd2c0a55700 1 --2- 172.21.15.42:0/4063967275 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd2bc1444e0 0x7fd2bc144910 secure :-1 s=THROTTLE_DONE pgs=2486 cs=0 l=1 rx=0x7fd2b40043a0 tx=0x7fd2b40107d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:57.419 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:57.406+0000 7fd2b3fff700 1 --2- 172.21.15.42:0/4063967275 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd2a003b1c0 0x7fd2a003d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:19:57.419 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:57.406+0000 7fd2b3fff700 1 -- 172.21.15.42:0/4063967275 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 10589+0+0 (secure 0 0 0) 0x7fd2b404b090 con 0x7fd2bc1444e0 2019-12-05T06:19:57.419 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:57.406+0000 7fd2bb59e700 1 --2- 172.21.15.42:0/4063967275 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd2a003b1c0 0x7fd2a003d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:57.419 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:57.406+0000 7fd2bb59e700 1 --2- 172.21.15.42:0/4063967275 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd2a003b1c0 0x7fd2a003d6d0 crc :-1 s=READY pgs=1040 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:19:57.419 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:57.410+0000 7fd2c1256700 1 -- 172.21.15.42:0/4063967275 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2bc05d110 con 0x7fd2bc1444e0 2019-12-05T06:19:57.422 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:57.418+0000 7fd2c0a55700 1 --2- 172.21.15.42:0/4063967275 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd2bc1444e0 0x7fd2bc144910 secure :-1 s=THROTTLE_DONE pgs=2486 cs=0 l=1 rx=0x7fd2b40043a0 tx=0x7fd2b40107d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:57.422 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:57.418+0000 7fd2b3fff700 1 -- 172.21.15.42:0/4063967275 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fd2b40320f0 con 0x7fd2bc1444e0 2019-12-05T06:19:57.590 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:57.586+0000 7fd2c1256700 1 -- 172.21.15.42:0/4063967275 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd2bc05b8c0 con 0x7fd2bc1444e0 2019-12-05T06:19:57.592 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:57.586+0000 7fd2c0a55700 1 --2- 172.21.15.42:0/4063967275 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd2bc1444e0 0x7fd2bc144910 secure :-1 s=THROTTLE_DONE pgs=2486 cs=0 l=1 rx=0x7fd2b40043a0 tx=0x7fd2b40107d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:57.592 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:57.586+0000 7fd2b3fff700 1 -- 172.21.15.42:0/4063967275 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 297 v297) v1 ==== 95+0+1779 (secure 0 0 0) 0x7fd2b4032350 con 0x7fd2bc1444e0 2019-12-05T06:19:57.592 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 297 2019-12-05T06:19:57.593 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:19:57.593 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":297,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":297,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:19:55.115618+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":102,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10711},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10711":{"gid":10711,"name":"a","rank":0,"incarnation":294,"state":"up:active","state_seq":12,"addr":"172.21.15.42:6835/2638662429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2638662429},{"type":"v1","addr":"172.21.15.42:6835","nonce":2638662429}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:19:57.594 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:57.590+0000 7fd2c1256700 1 -- 172.21.15.42:0/4063967275 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd2a003b1c0 msgr2=0x7fd2a003d6d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:57.594 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:57.590+0000 7fd2c1256700 1 --2- 172.21.15.42:0/4063967275 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd2a003b1c0 0x7fd2a003d6d0 crc :-1 s=READY pgs=1040 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:57.595 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:57.590+0000 7fd2c1256700 1 -- 172.21.15.42:0/4063967275 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd2bc1444e0 msgr2=0x7fd2bc144910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:57.595 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:57.590+0000 7fd2c1256700 1 --2- 172.21.15.42:0/4063967275 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd2bc1444e0 0x7fd2bc144910 secure :-1 s=READY pgs=2486 cs=0 l=1 rx=0x7fd2b40043a0 tx=0x7fd2b40107d0).stop 2019-12-05T06:19:57.595 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:57.590+0000 7fd2c1256700 1 -- 172.21.15.42:0/4063967275 shutdown_connections 2019-12-05T06:19:57.595 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:57.590+0000 7fd2c1256700 1 --2- 172.21.15.42:0/4063967275 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd2a003b1c0 0x7fd2a003d6d0 unknown :-1 s=CLOSED pgs=1040 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:57.595 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:57.590+0000 7fd2c1256700 1 --2- 172.21.15.42:0/4063967275 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd2bc1444e0 0x7fd2bc144910 unknown :-1 s=CLOSED pgs=2486 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:57.596 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:57.590+0000 7fd2c1256700 1 -- 172.21.15.42:0/4063967275 shutdown_connections 2019-12-05T06:19:57.596 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:57.590+0000 7fd2c1256700 1 -- 172.21.15.42:0/4063967275 wait complete. 2019-12-05T06:19:57.596 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:57.590+0000 7fd2c1256700 1 -- 172.21.15.42:0/4063967275 >> 172.21.15.42:0/4063967275 conn(0x7fd2bc0a82a0 msgr2=0x7fd2bc0a98e0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:19:57.604 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2019-12-05T06:19:59.605 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:19:59.733 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:59.714+0000 7fbc9eae7700 1 Processor -- start 2019-12-05T06:19:59.733 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:59.714+0000 7fbc9eae7700 1 -- start start 2019-12-05T06:19:59.733 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:59.718+0000 7fbc9eae7700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fbc9809fae0 0x7fbc9814fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:19:59.734 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:59.718+0000 7fbc9eae7700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc980a0710 con 0x7fbc980a7740 2019-12-05T06:19:59.734 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:59.718+0000 7fbc9eae7700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc9807ef20 con 0x7fbc9808df80 2019-12-05T06:19:59.734 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:59.718+0000 7fbc9eae7700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc980a0b00 con 0x7fbc9809fae0 2019-12-05T06:19:59.734 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:59.718+0000 7fbc9c883700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fbc9808df80 0x7fbc9809f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:53228/0 (socket says 172.21.15.42:53228) 2019-12-05T06:19:59.734 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:59.718+0000 7fbc9c883700 1 -- 172.21.15.42:0/1181636585 learned_addr learned my addr 172.21.15.42:0/1181636585 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:19:59.734 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:59.718+0000 7fbc9d885700 1 --2- 172.21.15.42:0/1181636585 >> v2:172.21.15.40:3300/0 conn(0x7fbc9809fae0 0x7fbc9814fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:59.734 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:59.718+0000 7fbc8ffff700 1 -- 172.21.15.42:0/1181636585 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2252965841 0 0) 0x7fbc9807ef20 con 0x7fbc9808df80 2019-12-05T06:19:59.735 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:59.718+0000 7fbc8ffff700 1 -- 172.21.15.42:0/1181636585 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc840030b0 con 0x7fbc9808df80 2019-12-05T06:19:59.735 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:59.718+0000 7fbc8ffff700 1 -- 172.21.15.42:0/1181636585 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2063122819 0 0) 0x7fbc840030b0 con 0x7fbc9808df80 2019-12-05T06:19:59.735 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:59.718+0000 7fbc8ffff700 1 -- 172.21.15.42:0/1181636585 >> v2:172.21.15.40:3300/0 conn(0x7fbc9809fae0 msgr2=0x7fbc9814fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:19:59.735 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:59.718+0000 7fbc8ffff700 1 --2- 172.21.15.42:0/1181636585 >> v2:172.21.15.40:3300/0 conn(0x7fbc9809fae0 0x7fbc9814fff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:19:59.735 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:59.718+0000 7fbc8ffff700 1 -- 172.21.15.42:0/1181636585 >> v1:172.21.15.40:6789/0 conn(0x7fbc980a7740 legacy=0x7fbc9807e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:59.735 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:59.718+0000 7fbc8ffff700 1 -- 172.21.15.42:0/1181636585 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc980a9ba0 con 0x7fbc9808df80 2019-12-05T06:19:59.735 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:59.718+0000 7fbc8ffff700 1 -- 172.21.15.42:0/1181636585 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7fbc88001750 con 0x7fbc9808df80 2019-12-05T06:19:59.736 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:59.718+0000 7fbc8ffff700 1 -- 172.21.15.42:0/1181636585 >> v1:172.21.15.42:6789/0 conn(0x7fbc9808df80 legacy=0x7fbc9809f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:59.736 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:59.718+0000 7fbc8ffff700 1 --2- 172.21.15.42:0/1181636585 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbc840036e0 0x7fbc84003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:19:59.736 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:59.718+0000 7fbc8ffff700 1 --2- 172.21.15.42:0/1181636585 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbc84004420 0x7fbc84004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:19:59.736 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:59.718+0000 7fbc8ffff700 1 --2- 172.21.15.42:0/1181636585 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbc84004da0 0x7fbc8400b670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:19:59.736 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:59.718+0000 7fbc8ffff700 1 -- 172.21.15.42:0/1181636585 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc980a0b00 con 0x7fbc840036e0 2019-12-05T06:19:59.736 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:59.718+0000 7fbc8ffff700 1 -- 172.21.15.42:0/1181636585 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc8404fc60 con 0x7fbc84004da0 2019-12-05T06:19:59.736 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:59.718+0000 7fbc8ffff700 1 -- 172.21.15.42:0/1181636585 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc8404fea0 con 0x7fbc84004420 2019-12-05T06:19:59.737 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:59.718+0000 7fbc9d885700 1 --2- 172.21.15.42:0/1181636585 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbc84004da0 0x7fbc8400b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:59.737 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:59.718+0000 7fbc9d084700 1 --2- 172.21.15.42:0/1181636585 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbc840036e0 0x7fbc84003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:59.737 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:59.718+0000 7fbc9c883700 1 --2- 172.21.15.42:0/1181636585 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbc84004420 0x7fbc84004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:59.737 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:59.718+0000 7fbc9d885700 1 -- 172.21.15.42:0/1181636585 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbc84004420 msgr2=0x7fbc84004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:19:59.737 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:59.718+0000 7fbc9d885700 1 --2- 172.21.15.42:0/1181636585 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbc84004420 0x7fbc84004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:19:59.737 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:59.718+0000 7fbc9d885700 1 -- 172.21.15.42:0/1181636585 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbc840036e0 msgr2=0x7fbc84003b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:19:59.737 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:59.718+0000 7fbc9d885700 1 --2- 172.21.15.42:0/1181636585 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbc840036e0 0x7fbc84003b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:19:59.738 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:59.718+0000 7fbc9d885700 1 -- 172.21.15.42:0/1181636585 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc840501a0 con 0x7fbc84004da0 2019-12-05T06:19:59.738 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:59.718+0000 7fbc9c883700 1 --2- 172.21.15.42:0/1181636585 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbc84004420 0x7fbc84004850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:19:59.738 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:59.718+0000 7fbc9d885700 1 --2- 172.21.15.42:0/1181636585 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbc84004da0 0x7fbc8400b670 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fbc900032b0 tx=0x7fbc90010320).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:59.738 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:59.718+0000 7fbc9d885700 1 -- 172.21.15.42:0/1181636585 reap_dead start 2019-12-05T06:19:59.738 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:59.718+0000 7fbc9d885700 1 --2- 172.21.15.42:0/1181636585 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbc84004da0 0x7fbc8400b670 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fbc900032b0 tx=0x7fbc90010320).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:59.738 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:59.718+0000 7fbc9d885700 1 --2- 172.21.15.42:0/1181636585 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbc84004da0 0x7fbc8400b670 secure :-1 s=READY pgs=2488 cs=0 l=1 rx=0x7fbc900032b0 tx=0x7fbc90010320).ready entity=mon.1 client_cookie=b16a213617aefec8 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:19:59.738 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:59.718+0000 7fbc9d885700 1 --2- 172.21.15.42:0/1181636585 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbc84004da0 0x7fbc8400b670 secure :-1 s=THROTTLE_DONE pgs=2488 cs=0 l=1 rx=0x7fbc900032b0 tx=0x7fbc90010320).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:59.739 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:59.718+0000 7fbc8ffff700 1 -- 172.21.15.42:0/1181636585 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbc90008d40 con 0x7fbc84004da0 2019-12-05T06:19:59.739 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:59.718+0000 7fbc9d885700 1 --2- 172.21.15.42:0/1181636585 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbc84004da0 0x7fbc8400b670 secure :-1 s=THROTTLE_DONE pgs=2488 cs=0 l=1 rx=0x7fbc900032b0 tx=0x7fbc90010320).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:59.739 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:59.718+0000 7fbc8ffff700 1 -- 172.21.15.42:0/1181636585 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc90010c70 con 0x7fbc84004da0 2019-12-05T06:19:59.739 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:59.718+0000 7fbc9d885700 1 --2- 172.21.15.42:0/1181636585 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbc84004da0 0x7fbc8400b670 secure :-1 s=THROTTLE_DONE pgs=2488 cs=0 l=1 rx=0x7fbc900032b0 tx=0x7fbc90010320).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:59.739 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:59.718+0000 7fbc9eae7700 1 -- 172.21.15.42:0/1181636585 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbc84004da0 msgr2=0x7fbc8400b670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:59.739 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:59.718+0000 7fbc8ffff700 1 -- 172.21.15.42:0/1181636585 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbc90005ca0 con 0x7fbc84004da0 2019-12-05T06:19:59.739 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:59.718+0000 7fbc9eae7700 1 --2- 172.21.15.42:0/1181636585 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbc84004da0 0x7fbc8400b670 secure :-1 s=READY pgs=2488 cs=0 l=1 rx=0x7fbc900032b0 tx=0x7fbc90010320).stop 2019-12-05T06:19:59.740 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:59.718+0000 7fbc9eae7700 1 -- 172.21.15.42:0/1181636585 shutdown_connections 2019-12-05T06:19:59.740 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:59.718+0000 7fbc9eae7700 1 --2- 172.21.15.42:0/1181636585 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbc84004da0 0x7fbc8400b670 unknown :-1 s=CLOSED pgs=2488 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:59.742 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:59.718+0000 7fbc9eae7700 1 -- 172.21.15.42:0/1181636585 shutdown_connections 2019-12-05T06:19:59.742 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:59.718+0000 7fbc9eae7700 1 -- 172.21.15.42:0/1181636585 wait complete. 2019-12-05T06:19:59.742 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:59.718+0000 7fbc9eae7700 1 -- 172.21.15.42:0/1181636585 >> 172.21.15.42:0/1181636585 conn(0x7fbc980aabe0 msgr2=0x7fbc980a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:19:59.742 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:59.722+0000 7fbc9eae7700 1 Processor -- start 2019-12-05T06:19:59.743 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:59.722+0000 7fbc9eae7700 1 -- start start 2019-12-05T06:19:59.743 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:59.722+0000 7fbc9eae7700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fbc981247f0 0x7fbc9807b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:19:59.743 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:59.722+0000 7fbc9eae7700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fbc9807aa70 0x7fbc9807aea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:19:59.743 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:59.722+0000 7fbc9eae7700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc9807b3f0 con 0x7fbc9807cdd0 2019-12-05T06:19:59.743 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:59.722+0000 7fbc9eae7700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc980a6d90 con 0x7fbc981247f0 2019-12-05T06:19:59.744 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:59.722+0000 7fbc9eae7700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc980a7030 con 0x7fbc9807aa70 2019-12-05T06:19:59.744 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:59.726+0000 7fbc9c883700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fbc9807aa70 0x7fbc9807aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:59.744 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:59.726+0000 7fbc9c883700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fbc9807aa70 0x7fbc9807aea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:53202/0 (socket says 172.21.15.42:53202) 2019-12-05T06:19:59.744 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:59.726+0000 7fbc9c883700 1 -- 172.21.15.42:0/2656708967 learned_addr learned my addr 172.21.15.42:0/2656708967 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:19:59.745 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:59.726+0000 7fbc9d084700 1 --2- 172.21.15.42:0/2656708967 >> v2:172.21.15.40:3300/0 conn(0x7fbc981247f0 0x7fbc9807b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:59.745 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:59.726+0000 7fbc9c883700 1 -- 172.21.15.42:0/2656708967 >> v2:172.21.15.40:3300/0 conn(0x7fbc981247f0 msgr2=0x7fbc9807b7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:59.745 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:59.726+0000 7fbc9c883700 1 --2- 172.21.15.42:0/2656708967 >> v2:172.21.15.40:3300/0 conn(0x7fbc981247f0 0x7fbc9807b7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:59.745 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:59.726+0000 7fbc9c883700 1 -- 172.21.15.42:0/2656708967 >> v1:172.21.15.40:6789/0 conn(0x7fbc9807cdd0 legacy=0x7fbc9807d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:59.745 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:59.726+0000 7fbc9c883700 1 -- 172.21.15.42:0/2656708967 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc9807bd10 con 0x7fbc9807aa70 2019-12-05T06:19:59.745 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:59.726+0000 7fbc9c883700 1 --2- 172.21.15.42:0/2656708967 >> v2:172.21.15.42:3300/0 conn(0x7fbc9807aa70 0x7fbc9807aea0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fbc98123f20 tx=0x7fbc88010a10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:59.745 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:59.726+0000 7fbc9c883700 1 --2- 172.21.15.42:0/2656708967 >> v2:172.21.15.42:3300/0 conn(0x7fbc9807aa70 0x7fbc9807aea0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fbc98123f20 tx=0x7fbc88010a10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:59.746 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:59.726+0000 7fbc9c883700 1 --2- 172.21.15.42:0/2656708967 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbc9807aa70 0x7fbc9807aea0 secure :-1 s=READY pgs=2489 cs=0 l=1 rx=0x7fbc98123f20 tx=0x7fbc88010a10).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:19:59.746 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:59.726+0000 7fbc9c883700 1 --2- 172.21.15.42:0/2656708967 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbc9807aa70 0x7fbc9807aea0 secure :-1 s=THROTTLE_DONE pgs=2489 cs=0 l=1 rx=0x7fbc98123f20 tx=0x7fbc88010a10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:59.746 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:59.726+0000 7fbc9c883700 1 --2- 172.21.15.42:0/2656708967 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbc9807aa70 0x7fbc9807aea0 secure :-1 s=THROTTLE_DONE pgs=2489 cs=0 l=1 rx=0x7fbc98123f20 tx=0x7fbc88010a10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:59.746 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:59.726+0000 7fbc8dffb700 1 -- 172.21.15.42:0/2656708967 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbc8800c940 con 0x7fbc9807aa70 2019-12-05T06:19:59.746 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:59.726+0000 7fbc9c883700 1 --2- 172.21.15.42:0/2656708967 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbc9807aa70 0x7fbc9807aea0 secure :-1 s=THROTTLE_DONE pgs=2489 cs=0 l=1 rx=0x7fbc98123f20 tx=0x7fbc88010a10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:59.746 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:59.726+0000 7fbc8dffb700 1 -- 172.21.15.42:0/2656708967 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc88010250 con 0x7fbc9807aa70 2019-12-05T06:19:59.746 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:59.726+0000 7fbc9eae7700 1 -- 172.21.15.42:0/2656708967 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc981302f0 con 0x7fbc9807aa70 2019-12-05T06:19:59.747 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:59.726+0000 7fbc8dffb700 1 -- 172.21.15.42:0/2656708967 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbc8800d0c0 con 0x7fbc9807aa70 2019-12-05T06:19:59.747 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:59.726+0000 7fbc9eae7700 1 -- 172.21.15.42:0/2656708967 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc981378e0 con 0x7fbc9807aa70 2019-12-05T06:19:59.747 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:59.726+0000 7fbc9c883700 1 --2- 172.21.15.42:0/2656708967 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbc9807aa70 0x7fbc9807aea0 secure :-1 s=THROTTLE_DONE pgs=2489 cs=0 l=1 rx=0x7fbc98123f20 tx=0x7fbc88010a10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:59.747 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:59.726+0000 7fbc8dffb700 1 -- 172.21.15.42:0/2656708967 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fbc88005440 con 0x7fbc9807aa70 2019-12-05T06:19:59.747 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:59.726+0000 7fbc9c883700 1 --2- 172.21.15.42:0/2656708967 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbc9807aa70 0x7fbc9807aea0 secure :-1 s=THROTTLE_DONE pgs=2489 cs=0 l=1 rx=0x7fbc98123f20 tx=0x7fbc88010a10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:59.747 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:59.726+0000 7fbc8dffb700 1 --2- 172.21.15.42:0/2656708967 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbc7803ac70 0x7fbc7803d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:19:59.748 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:59.726+0000 7fbc8dffb700 1 -- 172.21.15.42:0/2656708967 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 10589+0+0 (secure 0 0 0) 0x7fbc8804abc0 con 0x7fbc9807aa70 2019-12-05T06:19:59.748 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:59.726+0000 7fbc9d084700 1 --2- 172.21.15.42:0/2656708967 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbc7803ac70 0x7fbc7803d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:59.748 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:59.726+0000 7fbc9d084700 1 --2- 172.21.15.42:0/2656708967 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbc7803ac70 0x7fbc7803d180 crc :-1 s=READY pgs=1041 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:19:59.749 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:59.730+0000 7fbc9eae7700 1 -- 172.21.15.42:0/2656708967 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc9805d110 con 0x7fbc9807aa70 2019-12-05T06:19:59.749 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:59.738+0000 7fbc9c883700 1 --2- 172.21.15.42:0/2656708967 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbc9807aa70 0x7fbc9807aea0 secure :-1 s=THROTTLE_DONE pgs=2489 cs=0 l=1 rx=0x7fbc98123f20 tx=0x7fbc88010a10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:59.749 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:59.738+0000 7fbc8dffb700 1 -- 172.21.15.42:0/2656708967 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fbc9807bd10 con 0x7fbc9807aa70 2019-12-05T06:19:59.904 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:59.902+0000 7fbc9eae7700 1 -- 172.21.15.42:0/2656708967 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbc9805b8c0 con 0x7fbc9807aa70 2019-12-05T06:19:59.905 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:59.902+0000 7fbc9c883700 1 --2- 172.21.15.42:0/2656708967 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbc9807aa70 0x7fbc9807aea0 secure :-1 s=THROTTLE_DONE pgs=2489 cs=0 l=1 rx=0x7fbc98123f20 tx=0x7fbc88010a10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:59.905 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:59.902+0000 7fbc8dffb700 1 -- 172.21.15.42:0/2656708967 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 297 v297) v1 ==== 95+0+1779 (secure 0 0 0) 0x7fbc88031c40 con 0x7fbc9807aa70 2019-12-05T06:19:59.906 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 297 2019-12-05T06:19:59.906 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:19:59.906 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":297,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":297,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:19:55.115618+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":102,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10711},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10711":{"gid":10711,"name":"a","rank":0,"incarnation":294,"state":"up:active","state_seq":12,"addr":"172.21.15.42:6835/2638662429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2638662429},{"type":"v1","addr":"172.21.15.42:6835","nonce":2638662429}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:19:59.907 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:59.902+0000 7fbc9eae7700 1 -- 172.21.15.42:0/2656708967 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbc7803ac70 msgr2=0x7fbc7803d180 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:59.907 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:59.902+0000 7fbc9eae7700 1 --2- 172.21.15.42:0/2656708967 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbc7803ac70 0x7fbc7803d180 crc :-1 s=READY pgs=1041 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:59.908 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:59.902+0000 7fbc9eae7700 1 -- 172.21.15.42:0/2656708967 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbc9807aa70 msgr2=0x7fbc9807aea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:59.908 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:59.902+0000 7fbc9eae7700 1 --2- 172.21.15.42:0/2656708967 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbc9807aa70 0x7fbc9807aea0 secure :-1 s=READY pgs=2489 cs=0 l=1 rx=0x7fbc98123f20 tx=0x7fbc88010a10).stop 2019-12-05T06:19:59.909 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:59.902+0000 7fbc9eae7700 1 -- 172.21.15.42:0/2656708967 shutdown_connections 2019-12-05T06:19:59.909 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:59.902+0000 7fbc9eae7700 1 --2- 172.21.15.42:0/2656708967 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbc7803ac70 0x7fbc7803d180 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:59.909 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:59.902+0000 7fbc9eae7700 1 --2- 172.21.15.42:0/2656708967 >> v2:172.21.15.40:3300/0 conn(0x7fbc981247f0 0x7fbc9807b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:59.909 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:59.906+0000 7fbc9eae7700 1 --2- 172.21.15.42:0/2656708967 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbc9807aa70 0x7fbc9807aea0 unknown :-1 s=CLOSED pgs=2489 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:59.909 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:59.906+0000 7fbc9eae7700 1 -- 172.21.15.42:0/2656708967 shutdown_connections 2019-12-05T06:19:59.909 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:59.906+0000 7fbc9eae7700 1 -- 172.21.15.42:0/2656708967 wait complete. 2019-12-05T06:19:59.910 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:19:59.906+0000 7fbc9eae7700 1 -- 172.21.15.42:0/2656708967 >> 172.21.15.42:0/2656708967 conn(0x7fbc9811cd50 msgr2=0x7fbc9808ef90 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:19:59.918 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2019-12-05T06:20:01.918 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:20:02.022 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:20:02.014+0000 7fb939330700 1 Processor -- start 2019-12-05T06:20:02.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:20:02.014+0000 7fb939330700 1 -- start start 2019-12-05T06:20:02.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:20:02.014+0000 7fb939330700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb9340a7730 0x7fb93408df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:20:02.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:20:02.014+0000 7fb939330700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb93409fad0 0x7fb93407eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:20:02.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:20:02.014+0000 7fb939330700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9340a0700 con 0x7fb93407e710 2019-12-05T06:20:02.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:20:02.014+0000 7fb939330700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9340a0af0 con 0x7fb93409fad0 2019-12-05T06:20:02.024 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:20:02.014+0000 7fb939330700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb93408e5a0 con 0x7fb9340a7730 2019-12-05T06:20:02.024 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:20:02.014+0000 7fb9337fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb9340a7730 0x7fb93408df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:20:02.024 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:20:02.014+0000 7fb9337fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb9340a7730 0x7fb93408df70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:53216/0 (socket says 172.21.15.42:53216) 2019-12-05T06:20:02.024 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:20:02.014+0000 7fb9337fe700 1 -- 172.21.15.42:0/4097236287 learned_addr learned my addr 172.21.15.42:0/4097236287 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:20:02.025 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:20:02.014+0000 7fb933fff700 1 --2- 172.21.15.42:0/4097236287 >> v2:172.21.15.40:3300/0 conn(0x7fb93409fad0 0x7fb93407eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:20:02.025 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:20:02.014+0000 7fb9337fe700 1 -- 172.21.15.42:0/4097236287 >> v2:172.21.15.40:3300/0 conn(0x7fb93409fad0 msgr2=0x7fb93407eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:20:02.025 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:20:02.014+0000 7fb9337fe700 1 --2- 172.21.15.42:0/4097236287 >> v2:172.21.15.40:3300/0 conn(0x7fb93409fad0 0x7fb93407eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:20:02.025 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:20:02.014+0000 7fb9337fe700 1 -- 172.21.15.42:0/4097236287 >> v1:172.21.15.40:6789/0 conn(0x7fb93407e710 legacy=0x7fb93409f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:20:02.025 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:20:02.014+0000 7fb9337fe700 1 -- 172.21.15.42:0/4097236287 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb93407f090 con 0x7fb9340a7730 2019-12-05T06:20:02.025 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:20:02.014+0000 7fb9337fe700 1 --2- 172.21.15.42:0/4097236287 >> v2:172.21.15.42:3300/0 conn(0x7fb9340a7730 0x7fb93408df70 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fb924002f30 tx=0x7fb92400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:02.026 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:20:02.018+0000 7fb9337fe700 1 --2- 172.21.15.42:0/4097236287 >> v2:172.21.15.42:3300/0 conn(0x7fb9340a7730 0x7fb93408df70 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fb924002f30 tx=0x7fb92400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:02.026 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:20:02.018+0000 7fb9337fe700 1 --2- 172.21.15.42:0/4097236287 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb9340a7730 0x7fb93408df70 secure :-1 s=READY pgs=2490 cs=0 l=1 rx=0x7fb924002f30 tx=0x7fb92400fd30).ready entity=mon.1 client_cookie=c28db12ebea35afe server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:20:02.026 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:20:02.018+0000 7fb9337fe700 1 --2- 172.21.15.42:0/4097236287 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb9340a7730 0x7fb93408df70 secure :-1 s=THROTTLE_DONE pgs=2490 cs=0 l=1 rx=0x7fb924002f30 tx=0x7fb92400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:02.026 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:20:02.018+0000 7fb9337fe700 1 --2- 172.21.15.42:0/4097236287 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb9340a7730 0x7fb93408df70 secure :-1 s=THROTTLE_DONE pgs=2490 cs=0 l=1 rx=0x7fb924002f30 tx=0x7fb92400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:02.026 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:20:02.018+0000 7fb9337fe700 1 --2- 172.21.15.42:0/4097236287 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb9340a7730 0x7fb93408df70 secure :-1 s=THROTTLE_DONE pgs=2490 cs=0 l=1 rx=0x7fb924002f30 tx=0x7fb92400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:02.027 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:20:02.018+0000 7fb932ffd700 1 -- 172.21.15.42:0/4097236287 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb924008c90 con 0x7fb9340a7730 2019-12-05T06:20:02.027 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:20:02.018+0000 7fb932ffd700 1 -- 172.21.15.42:0/4097236287 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9240088b0 con 0x7fb9340a7730 2019-12-05T06:20:02.027 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:20:02.018+0000 7fb932ffd700 1 -- 172.21.15.42:0/4097236287 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb924011ad0 con 0x7fb9340a7730 2019-12-05T06:20:02.027 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:20:02.018+0000 7fb939330700 1 -- 172.21.15.42:0/4097236287 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb9340a7730 msgr2=0x7fb93408df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:20:02.027 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:20:02.018+0000 7fb939330700 1 --2- 172.21.15.42:0/4097236287 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb9340a7730 0x7fb93408df70 secure :-1 s=READY pgs=2490 cs=0 l=1 rx=0x7fb924002f30 tx=0x7fb92400fd30).stop 2019-12-05T06:20:02.028 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:20:02.018+0000 7fb939330700 1 -- 172.21.15.42:0/4097236287 shutdown_connections 2019-12-05T06:20:02.028 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:20:02.018+0000 7fb939330700 1 --2- 172.21.15.42:0/4097236287 >> v2:172.21.15.40:3300/0 conn(0x7fb93409fad0 0x7fb93407eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:20:02.028 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:20:02.018+0000 7fb939330700 1 --2- 172.21.15.42:0/4097236287 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb9340a7730 0x7fb93408df70 unknown :-1 s=CLOSED pgs=2490 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:20:02.028 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:20:02.018+0000 7fb939330700 1 -- 172.21.15.42:0/4097236287 shutdown_connections 2019-12-05T06:20:02.029 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:20:02.018+0000 7fb939330700 1 -- 172.21.15.42:0/4097236287 wait complete. 2019-12-05T06:20:02.029 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:20:02.018+0000 7fb939330700 1 -- 172.21.15.42:0/4097236287 >> 172.21.15.42:0/4097236287 conn(0x7fb9340aabd0 msgr2=0x7fb9340a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:20:02.033 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:20:02.022+0000 7fb939330700 1 Processor -- start 2019-12-05T06:20:02.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:20:02.022+0000 7fb939330700 1 -- start start 2019-12-05T06:20:02.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:20:02.022+0000 7fb939330700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb93414e930 0x7fb934142520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:20:02.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:20:02.022+0000 7fb939330700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb934143760 con 0x7fb934123aa0 2019-12-05T06:20:02.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:20:02.022+0000 7fb939330700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb934130430 con 0x7fb934143ed0 2019-12-05T06:20:02.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:20:02.022+0000 7fb939330700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb934142ca0 con 0x7fb93414e930 2019-12-05T06:20:02.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:20:02.022+0000 7fb9337fe700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fb934143ed0 0x7fb93414e130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:53264/0 (socket says 172.21.15.42:53264) 2019-12-05T06:20:02.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:20:02.022+0000 7fb9337fe700 1 -- 172.21.15.42:0/269740936 learned_addr learned my addr 172.21.15.42:0/269740936 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:20:02.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:20:02.022+0000 7fb92affd700 1 --2- 172.21.15.42:0/269740936 >> v2:172.21.15.40:3300/0 conn(0x7fb93414e930 0x7fb934142520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:20:02.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:20:02.022+0000 7fb930ff9700 1 -- 172.21.15.42:0/269740936 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2119698040 0 0) 0x7fb934130430 con 0x7fb934143ed0 2019-12-05T06:20:02.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:20:02.022+0000 7fb930ff9700 1 -- 172.21.15.42:0/269740936 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9180030b0 con 0x7fb934143ed0 2019-12-05T06:20:02.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:20:02.022+0000 7fb92affd700 1 -- 172.21.15.42:0/269740936 >> v1:172.21.15.42:6789/0 conn(0x7fb934143ed0 legacy=0x7fb93414e130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:20:02.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:20:02.022+0000 7fb92affd700 1 -- 172.21.15.42:0/269740936 >> v1:172.21.15.40:6789/0 conn(0x7fb934123aa0 legacy=0x7fb934123ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:20:02.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:20:02.022+0000 7fb930ff9700 1 -- 172.21.15.42:0/269740936 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3034576199 0 0) 0x7fb934143760 con 0x7fb934123aa0 2019-12-05T06:20:02.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:20:02.022+0000 7fb92affd700 1 -- 172.21.15.42:0/269740936 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb934131670 con 0x7fb93414e930 2019-12-05T06:20:02.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:20:02.022+0000 7fb92affd700 1 --2- 172.21.15.42:0/269740936 >> v2:172.21.15.40:3300/0 conn(0x7fb93414e930 0x7fb934142520 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fb91c001430 tx=0x7fb91c0108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:02.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:20:02.022+0000 7fb92affd700 1 --2- 172.21.15.42:0/269740936 >> v2:172.21.15.40:3300/0 conn(0x7fb93414e930 0x7fb934142520 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fb91c001430 tx=0x7fb91c0108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:02.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:20:02.022+0000 7fb92affd700 1 --2- 172.21.15.42:0/269740936 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb93414e930 0x7fb934142520 secure :-1 s=READY pgs=1878 cs=0 l=1 rx=0x7fb91c001430 tx=0x7fb91c0108f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:20:02.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:20:02.022+0000 7fb92affd700 1 --2- 172.21.15.42:0/269740936 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb93414e930 0x7fb934142520 secure :-1 s=THROTTLE_DONE pgs=1878 cs=0 l=1 rx=0x7fb91c001430 tx=0x7fb91c0108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:02.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:20:02.026+0000 7fb930ff9700 1 -- 172.21.15.42:0/269740936 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb91c00cb60 con 0x7fb93414e930 2019-12-05T06:20:02.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:20:02.026+0000 7fb92affd700 1 --2- 172.21.15.42:0/269740936 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb93414e930 0x7fb934142520 secure :-1 s=THROTTLE_DONE pgs=1878 cs=0 l=1 rx=0x7fb91c001430 tx=0x7fb91c0108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:02.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:20:02.026+0000 7fb92affd700 1 --2- 172.21.15.42:0/269740936 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb93414e930 0x7fb934142520 secure :-1 s=THROTTLE_DONE pgs=1878 cs=0 l=1 rx=0x7fb91c001430 tx=0x7fb91c0108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:02.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:20:02.026+0000 7fb930ff9700 1 -- 172.21.15.42:0/269740936 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb91c00c780 con 0x7fb93414e930 2019-12-05T06:20:02.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:20:02.026+0000 7fb939330700 1 -- 172.21.15.42:0/269740936 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb93412f420 con 0x7fb93414e930 2019-12-05T06:20:02.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:20:02.026+0000 7fb930ff9700 1 -- 172.21.15.42:0/269740936 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb91c011070 con 0x7fb93414e930 2019-12-05T06:20:02.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:20:02.026+0000 7fb939330700 1 -- 172.21.15.42:0/269740936 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb93412f9d0 con 0x7fb93414e930 2019-12-05T06:20:02.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:20:02.026+0000 7fb92affd700 1 --2- 172.21.15.42:0/269740936 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb93414e930 0x7fb934142520 secure :-1 s=THROTTLE_DONE pgs=1878 cs=0 l=1 rx=0x7fb91c001430 tx=0x7fb91c0108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:02.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:20:02.026+0000 7fb930ff9700 1 -- 172.21.15.42:0/269740936 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fb91c011380 con 0x7fb93414e930 2019-12-05T06:20:02.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:20:02.026+0000 7fb92affd700 1 --2- 172.21.15.42:0/269740936 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb93414e930 0x7fb934142520 secure :-1 s=THROTTLE_DONE pgs=1878 cs=0 l=1 rx=0x7fb91c001430 tx=0x7fb91c0108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:02.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:20:02.026+0000 7fb930ff9700 1 --2- 172.21.15.42:0/269740936 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb91803b1e0 0x7fb91803d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:20:02.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:20:02.026+0000 7fb930ff9700 1 -- 172.21.15.42:0/269740936 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 10589+0+0 (secure 0 0 0) 0x7fb91c04b1a0 con 0x7fb93414e930 2019-12-05T06:20:02.039 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:20:02.026+0000 7fb9337fe700 1 --2- 172.21.15.42:0/269740936 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb91803b1e0 0x7fb91803d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:20:02.039 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:20:02.026+0000 7fb9337fe700 1 --2- 172.21.15.42:0/269740936 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb91803b1e0 0x7fb91803d6f0 crc :-1 s=READY pgs=1043 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:20:02.039 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:20:02.030+0000 7fb939330700 1 -- 172.21.15.42:0/269740936 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb93405d100 con 0x7fb93414e930 2019-12-05T06:20:02.040 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:20:02.038+0000 7fb92affd700 1 --2- 172.21.15.42:0/269740936 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb93414e930 0x7fb934142520 secure :-1 s=THROTTLE_DONE pgs=1878 cs=0 l=1 rx=0x7fb91c001430 tx=0x7fb91c0108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:02.040 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:20:02.038+0000 7fb930ff9700 1 -- 172.21.15.42:0/269740936 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fb91c032220 con 0x7fb93414e930 2019-12-05T06:20:02.232 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:20:02.226+0000 7fb939330700 1 -- 172.21.15.42:0/269740936 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb93405b8b0 con 0x7fb93414e930 2019-12-05T06:20:02.232 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:20:02.230+0000 7fb92affd700 1 --2- 172.21.15.42:0/269740936 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb93414e930 0x7fb934142520 secure :-1 s=THROTTLE_DONE pgs=1878 cs=0 l=1 rx=0x7fb91c001430 tx=0x7fb91c0108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:02.233 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:20:02.230+0000 7fb930ff9700 1 -- 172.21.15.42:0/269740936 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 298 v298) v1 ==== 95+0+2114 (secure 0 0 0) 0x7fb91c032480 con 0x7fb93414e930 2019-12-05T06:20:02.235 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 298 2019-12-05T06:20:02.235 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:20:02.235 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":298,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":10710,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/526579407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":526579407},{"type":"v1","addr":"172.21.15.40:6833","nonce":526579407}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":298}],"filesystems":[{"mdsmap":{"epoch":297,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:19:55.115618+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":102,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10711},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10711":{"gid":10711,"name":"a","rank":0,"incarnation":294,"state":"up:active","state_seq":12,"addr":"172.21.15.42:6835/2638662429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2638662429},{"type":"v1","addr":"172.21.15.42:6835","nonce":2638662429}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:20:02.244 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:20:02.234+0000 7fb939330700 1 -- 172.21.15.42:0/269740936 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb91803b1e0 msgr2=0x7fb91803d6f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:20:02.245 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:20:02.234+0000 7fb939330700 1 --2- 172.21.15.42:0/269740936 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb91803b1e0 0x7fb91803d6f0 crc :-1 s=READY pgs=1043 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:20:02.245 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:20:02.234+0000 7fb939330700 1 -- 172.21.15.42:0/269740936 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb93414e930 msgr2=0x7fb934142520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:20:02.245 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:20:02.234+0000 7fb939330700 1 --2- 172.21.15.42:0/269740936 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb93414e930 0x7fb934142520 secure :-1 s=READY pgs=1878 cs=0 l=1 rx=0x7fb91c001430 tx=0x7fb91c0108f0).stop 2019-12-05T06:20:02.245 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:20:02.234+0000 7fb939330700 1 -- 172.21.15.42:0/269740936 shutdown_connections 2019-12-05T06:20:02.245 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:20:02.234+0000 7fb939330700 1 --2- 172.21.15.42:0/269740936 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb91803b1e0 0x7fb91803d6f0 unknown :-1 s=CLOSED pgs=1043 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:20:02.245 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:20:02.234+0000 7fb939330700 1 --2- 172.21.15.42:0/269740936 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb93414e930 0x7fb934142520 unknown :-1 s=CLOSED pgs=1878 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:20:02.246 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:20:02.234+0000 7fb939330700 1 -- 172.21.15.42:0/269740936 shutdown_connections 2019-12-05T06:20:02.246 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:20:02.234+0000 7fb939330700 1 -- 172.21.15.42:0/269740936 wait complete. 2019-12-05T06:20:02.246 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:20:02.234+0000 7fb939330700 1 -- 172.21.15.42:0/269740936 >> 172.21.15.42:0/269740936 conn(0x7fb9340a89c0 msgr2=0x7fb93409f610 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:20:02.253 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b reported in up:standby state 2019-12-05T06:20:02.253 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 78 secs before thrashing 2019-12-05T06:20:15.843 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:20:15.846 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:20:45.892 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:20:45.895 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:21:15.949 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:21:15.955 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:21:15.996 INFO:tasks.ceph.mds.a.smithi042.stderr:2019-12-05T06:21:15.981+0000 7fc559cc8700 -1 received signal: Hangup from killall ceph-mds -1 (PID: 41686) UID: 0 2019-12-05T06:21:20.261 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:21:20.363 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.353+0000 7f6b078c9700 1 Processor -- start 2019-12-05T06:21:20.363 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.353+0000 7f6b078c9700 1 -- start start 2019-12-05T06:21:20.363 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.357+0000 7f6b078c9700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6b0008df80 0x7f6b0009f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:21:20.363 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.357+0000 7f6b078c9700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f6b0009f830 0x7f6b0014efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:21:20.364 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.357+0000 7f6b078c9700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b0009fc60 con 0x7f6b000a7740 2019-12-05T06:21:20.364 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.357+0000 7f6b078c9700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b000a0b00 con 0x7f6b0009f830 2019-12-05T06:21:20.364 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.357+0000 7f6b078c9700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b0008e5b0 con 0x7f6b0008df80 2019-12-05T06:21:20.364 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.357+0000 7f6b05665700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6b0008df80 0x7f6b0009f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:20.364 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.357+0000 7f6b05665700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6b0008df80 0x7f6b0009f2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:53584/0 (socket says 172.21.15.42:53584) 2019-12-05T06:21:20.364 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.357+0000 7f6b05665700 1 -- 172.21.15.42:0/1071598814 learned_addr learned my addr 172.21.15.42:0/1071598814 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:21:20.364 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.357+0000 7f6b06667700 1 --2- 172.21.15.42:0/1071598814 >> v2:172.21.15.40:3300/0 conn(0x7f6b0009f830 0x7f6b0014efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:20.365 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.357+0000 7f6b05665700 1 -- 172.21.15.42:0/1071598814 >> v2:172.21.15.40:3300/0 conn(0x7f6b0009f830 msgr2=0x7f6b0014efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:21:20.365 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.357+0000 7f6b05665700 1 --2- 172.21.15.42:0/1071598814 >> v2:172.21.15.40:3300/0 conn(0x7f6b0009f830 0x7f6b0014efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:21:20.365 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.357+0000 7f6b05665700 1 -- 172.21.15.42:0/1071598814 >> v1:172.21.15.40:6789/0 conn(0x7f6b000a7740 legacy=0x7f6b0007e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:20.365 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.357+0000 7f6b05665700 1 -- 172.21.15.42:0/1071598814 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b000a0710 con 0x7f6b0008df80 2019-12-05T06:21:20.365 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.357+0000 7f6b05665700 1 --2- 172.21.15.42:0/1071598814 >> v2:172.21.15.42:3300/0 conn(0x7f6b0008df80 0x7f6b0009f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f6af0002f30 tx=0x7f6af000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:20.365 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.357+0000 7f6b05665700 1 --2- 172.21.15.42:0/1071598814 >> v2:172.21.15.42:3300/0 conn(0x7f6b0008df80 0x7f6b0009f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f6af0002f30 tx=0x7f6af000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:20.365 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.357+0000 7f6b05665700 1 --2- 172.21.15.42:0/1071598814 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6b0008df80 0x7f6b0009f2e0 secure :-1 s=READY pgs=2499 cs=0 l=1 rx=0x7f6af0002f30 tx=0x7f6af000fd30).ready entity=mon.1 client_cookie=d258e9484d16f0b6 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:21:20.366 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.357+0000 7f6b05665700 1 --2- 172.21.15.42:0/1071598814 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6b0008df80 0x7f6b0009f2e0 secure :-1 s=THROTTLE_DONE pgs=2499 cs=0 l=1 rx=0x7f6af0002f30 tx=0x7f6af000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:20.366 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.357+0000 7f6b04e64700 1 -- 172.21.15.42:0/1071598814 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6af0008c90 con 0x7f6b0008df80 2019-12-05T06:21:20.366 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.357+0000 7f6b05665700 1 --2- 172.21.15.42:0/1071598814 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6b0008df80 0x7f6b0009f2e0 secure :-1 s=THROTTLE_DONE pgs=2499 cs=0 l=1 rx=0x7f6af0002f30 tx=0x7f6af000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:20.366 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.357+0000 7f6b04e64700 1 -- 172.21.15.42:0/1071598814 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6af00088b0 con 0x7f6b0008df80 2019-12-05T06:21:20.366 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.357+0000 7f6b05665700 1 --2- 172.21.15.42:0/1071598814 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6b0008df80 0x7f6b0009f2e0 secure :-1 s=THROTTLE_DONE pgs=2499 cs=0 l=1 rx=0x7f6af0002f30 tx=0x7f6af000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:20.366 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.357+0000 7f6b04e64700 1 -- 172.21.15.42:0/1071598814 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6af0011ad0 con 0x7f6b0008df80 2019-12-05T06:21:20.366 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.357+0000 7f6b078c9700 1 -- 172.21.15.42:0/1071598814 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6b0008df80 msgr2=0x7f6b0009f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:20.367 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.357+0000 7f6b078c9700 1 --2- 172.21.15.42:0/1071598814 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6b0008df80 0x7f6b0009f2e0 secure :-1 s=READY pgs=2499 cs=0 l=1 rx=0x7f6af0002f30 tx=0x7f6af000fd30).stop 2019-12-05T06:21:20.367 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.357+0000 7f6b078c9700 1 -- 172.21.15.42:0/1071598814 shutdown_connections 2019-12-05T06:21:20.367 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.357+0000 7f6b078c9700 1 --2- 172.21.15.42:0/1071598814 >> v2:172.21.15.40:3300/0 conn(0x7f6b0009f830 0x7f6b0014efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:21:20.367 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.357+0000 7f6b078c9700 1 --2- 172.21.15.42:0/1071598814 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6b0008df80 0x7f6b0009f2e0 unknown :-1 s=CLOSED pgs=2499 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:20.367 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.357+0000 7f6b078c9700 1 -- 172.21.15.42:0/1071598814 shutdown_connections 2019-12-05T06:21:20.367 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.357+0000 7f6b078c9700 1 -- 172.21.15.42:0/1071598814 wait complete. 2019-12-05T06:21:20.367 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.357+0000 7f6b078c9700 1 -- 172.21.15.42:0/1071598814 >> 172.21.15.42:0/1071598814 conn(0x7f6b000aabe0 msgr2=0x7f6b000a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:21:20.382 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.361+0000 7f6b078c9700 1 Processor -- start 2019-12-05T06:21:20.382 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.361+0000 7f6b078c9700 1 -- start start 2019-12-05T06:21:20.383 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.361+0000 7f6b078c9700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6b0007d8a0 0x7f6b000a5100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:21:20.383 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.361+0000 7f6b078c9700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6b00119b70 0x7f6b00119fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:21:20.383 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.361+0000 7f6b078c9700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b000a5650 con 0x7f6b000a1510 2019-12-05T06:21:20.383 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.361+0000 7f6b078c9700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b000a5990 con 0x7f6b00119b70 2019-12-05T06:21:20.383 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.361+0000 7f6b078c9700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b001198d0 con 0x7f6b0007d8a0 2019-12-05T06:21:20.383 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.361+0000 7f6b06667700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6b00119b70 0x7f6b00119fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:20.384 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.361+0000 7f6b06667700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6b00119b70 0x7f6b00119fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:53592/0 (socket says 172.21.15.42:53592) 2019-12-05T06:21:20.384 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.361+0000 7f6b06667700 1 -- 172.21.15.42:0/1299388331 learned_addr learned my addr 172.21.15.42:0/1299388331 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:21:20.384 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.361+0000 7f6b05665700 1 --2- 172.21.15.42:0/1299388331 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6b0007d8a0 0x7f6b000a5100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:20.384 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.361+0000 7f6b06667700 1 -- 172.21.15.42:0/1299388331 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6b0007d8a0 msgr2=0x7f6b000a5100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:20.384 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.361+0000 7f6b06667700 1 --2- 172.21.15.42:0/1299388331 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6b0007d8a0 0x7f6b000a5100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:20.384 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.361+0000 7f6b06667700 1 -- 172.21.15.42:0/1299388331 >> v1:172.21.15.42:6789/0 conn(0x7f6b000a1510 legacy=0x7f6b0012a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:20.384 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.361+0000 7f6b06667700 1 -- 172.21.15.42:0/1299388331 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b000a5c30 con 0x7f6b00119b70 2019-12-05T06:21:20.385 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.361+0000 7f6b06667700 1 --2- 172.21.15.42:0/1299388331 >> v2:172.21.15.42:3300/0 conn(0x7f6b00119b70 0x7f6b00119fa0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f6b000a5b50 tx=0x7f6af8010c40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:20.385 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.361+0000 7f6b06667700 1 --2- 172.21.15.42:0/1299388331 >> v2:172.21.15.42:3300/0 conn(0x7f6b00119b70 0x7f6b00119fa0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f6b000a5b50 tx=0x7f6af8010c40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:20.385 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.361+0000 7f6b06667700 1 --2- 172.21.15.42:0/1299388331 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6b00119b70 0x7f6b00119fa0 secure :-1 s=READY pgs=2501 cs=0 l=1 rx=0x7f6b000a5b50 tx=0x7f6af8010c40).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:21:20.385 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.361+0000 7f6b06667700 1 --2- 172.21.15.42:0/1299388331 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6b00119b70 0x7f6b00119fa0 secure :-1 s=THROTTLE_DONE pgs=2501 cs=0 l=1 rx=0x7f6b000a5b50 tx=0x7f6af8010c40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:20.385 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.361+0000 7f6af67fc700 1 -- 172.21.15.42:0/1299388331 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6af8007350 con 0x7f6b00119b70 2019-12-05T06:21:20.385 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.361+0000 7f6b06667700 1 --2- 172.21.15.42:0/1299388331 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6b00119b70 0x7f6b00119fa0 secure :-1 s=THROTTLE_DONE pgs=2501 cs=0 l=1 rx=0x7f6b000a5b50 tx=0x7f6af8010c40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:20.385 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.361+0000 7f6b06667700 1 --2- 172.21.15.42:0/1299388331 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6b00119b70 0x7f6b00119fa0 secure :-1 s=THROTTLE_DONE pgs=2501 cs=0 l=1 rx=0x7f6b000a5b50 tx=0x7f6af8010c40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:20.386 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.361+0000 7f6af67fc700 1 -- 172.21.15.42:0/1299388331 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6af8005cb0 con 0x7f6b00119b70 2019-12-05T06:21:20.386 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.361+0000 7f6b078c9700 1 -- 172.21.15.42:0/1299388331 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b00129a50 con 0x7f6b00119b70 2019-12-05T06:21:20.386 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.361+0000 7f6af67fc700 1 -- 172.21.15.42:0/1299388331 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6af80124b0 con 0x7f6b00119b70 2019-12-05T06:21:20.386 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.361+0000 7f6b078c9700 1 -- 172.21.15.42:0/1299388331 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b00131040 con 0x7f6b00119b70 2019-12-05T06:21:20.386 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.361+0000 7f6b06667700 1 --2- 172.21.15.42:0/1299388331 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6b00119b70 0x7f6b00119fa0 secure :-1 s=THROTTLE_DONE pgs=2501 cs=0 l=1 rx=0x7f6b000a5b50 tx=0x7f6af8010c40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:20.386 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.361+0000 7f6af67fc700 1 -- 172.21.15.42:0/1299388331 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f6af8012780 con 0x7f6b00119b70 2019-12-05T06:21:20.386 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.365+0000 7f6b06667700 1 --2- 172.21.15.42:0/1299388331 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6b00119b70 0x7f6b00119fa0 secure :-1 s=THROTTLE_DONE pgs=2501 cs=0 l=1 rx=0x7f6b000a5b50 tx=0x7f6af8010c40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:20.387 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.365+0000 7f6af67fc700 1 --2- 172.21.15.42:0/1299388331 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6ae803ad50 0x7f6ae803d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:21:20.387 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.365+0000 7f6af67fc700 1 -- 172.21.15.42:0/1299388331 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 10589+0+0 (secure 0 0 0) 0x7f6af804b390 con 0x7f6b00119b70 2019-12-05T06:21:20.387 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.365+0000 7f6b05e66700 1 --2- 172.21.15.42:0/1299388331 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6ae803ad50 0x7f6ae803d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:20.387 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.365+0000 7f6b05e66700 1 --2- 172.21.15.42:0/1299388331 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6ae803ad50 0x7f6ae803d260 crc :-1 s=READY pgs=1045 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:21:20.387 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.369+0000 7f6b078c9700 1 -- 172.21.15.42:0/1299388331 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b0005d110 con 0x7f6b00119b70 2019-12-05T06:21:20.387 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.377+0000 7f6b06667700 1 --2- 172.21.15.42:0/1299388331 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6b00119b70 0x7f6b00119fa0 secure :-1 s=THROTTLE_DONE pgs=2501 cs=0 l=1 rx=0x7f6b000a5b50 tx=0x7f6af8010c40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:20.387 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.377+0000 7f6af67fc700 1 -- 172.21.15.42:0/1299388331 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f6af8032410 con 0x7f6b00119b70 2019-12-05T06:21:20.618 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.613+0000 7f6b078c9700 1 -- 172.21.15.42:0/1299388331 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6b0005b8c0 con 0x7f6b00119b70 2019-12-05T06:21:20.620 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.613+0000 7f6b06667700 1 --2- 172.21.15.42:0/1299388331 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6b00119b70 0x7f6b00119fa0 secure :-1 s=THROTTLE_DONE pgs=2501 cs=0 l=1 rx=0x7f6b000a5b50 tx=0x7f6af8010c40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:20.620 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.613+0000 7f6af67fc700 1 -- 172.21.15.42:0/1299388331 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 298 v298) v1 ==== 95+0+2114 (secure 0 0 0) 0x7f6af80321c0 con 0x7f6b00119b70 2019-12-05T06:21:20.620 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 298 2019-12-05T06:21:20.621 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:21:20.621 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":298,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":10710,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/526579407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":526579407},{"type":"v1","addr":"172.21.15.40:6833","nonce":526579407}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":298}],"filesystems":[{"mdsmap":{"epoch":297,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:19:55.115618+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":102,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10711},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10711":{"gid":10711,"name":"a","rank":0,"incarnation":294,"state":"up:active","state_seq":12,"addr":"172.21.15.42:6835/2638662429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2638662429},{"type":"v1","addr":"172.21.15.42:6835","nonce":2638662429}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:21:20.622 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.617+0000 7f6b078c9700 1 -- 172.21.15.42:0/1299388331 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6ae803ad50 msgr2=0x7f6ae803d260 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:20.622 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.617+0000 7f6b078c9700 1 --2- 172.21.15.42:0/1299388331 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6ae803ad50 0x7f6ae803d260 crc :-1 s=READY pgs=1045 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:20.622 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.617+0000 7f6b078c9700 1 -- 172.21.15.42:0/1299388331 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6b00119b70 msgr2=0x7f6b00119fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:20.623 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.617+0000 7f6b078c9700 1 --2- 172.21.15.42:0/1299388331 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6b00119b70 0x7f6b00119fa0 secure :-1 s=READY pgs=2501 cs=0 l=1 rx=0x7f6b000a5b50 tx=0x7f6af8010c40).stop 2019-12-05T06:21:20.623 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.617+0000 7f6b078c9700 1 -- 172.21.15.42:0/1299388331 shutdown_connections 2019-12-05T06:21:20.623 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.617+0000 7f6b078c9700 1 --2- 172.21.15.42:0/1299388331 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6ae803ad50 0x7f6ae803d260 unknown :-1 s=CLOSED pgs=1045 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:20.623 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.617+0000 7f6b078c9700 1 --2- 172.21.15.42:0/1299388331 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6b00119b70 0x7f6b00119fa0 unknown :-1 s=CLOSED pgs=2501 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:20.623 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.617+0000 7f6b078c9700 1 --2- 172.21.15.42:0/1299388331 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6b0007d8a0 0x7f6b000a5100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:20.624 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.617+0000 7f6b078c9700 1 -- 172.21.15.42:0/1299388331 shutdown_connections 2019-12-05T06:21:20.624 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.617+0000 7f6b078c9700 1 -- 172.21.15.42:0/1299388331 wait complete. 2019-12-05T06:21:20.624 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.617+0000 7f6b078c9700 1 -- 172.21.15.42:0/1299388331 >> 172.21.15.42:0/1299388331 conn(0x7f6b000a9850 msgr2=0x7f6b0007cd60 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:21:20.632 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.a (rank=0) 2019-12-05T06:21:20.632 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2019-12-05T06:21:20.633 INFO:teuthology.orchestra.run:waiting for 300 2019-12-05T06:21:20.643 INFO:tasks.ceph.mds.a:Stopped 2019-12-05T06:21:20.643 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:21:20.815 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.805+0000 7f74b853b700 1 Processor -- start 2019-12-05T06:21:20.816 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.805+0000 7f74b853b700 1 -- start start 2019-12-05T06:21:20.816 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.805+0000 7f74b853b700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f74b00a7740 0x7f74b008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:21:20.816 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.805+0000 7f74b853b700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74b00a0710 con 0x7f74b009fae0 2019-12-05T06:21:20.816 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.805+0000 7f74b853b700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74b00a9ba0 con 0x7f74b007e720 2019-12-05T06:21:20.817 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.805+0000 7f74b853b700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74b00a0b00 con 0x7f74b00a7740 2019-12-05T06:21:20.817 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.805+0000 7f74b62d7700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f74b007e720 0x7f74b009f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:53650/0 (socket says 172.21.15.42:53650) 2019-12-05T06:21:20.817 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.805+0000 7f74b62d7700 1 -- 172.21.15.42:0/1401354439 learned_addr learned my addr 172.21.15.42:0/1401354439 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:21:20.817 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.805+0000 7f74b6ad8700 1 --2- 172.21.15.42:0/1401354439 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f74b00a7740 0x7f74b008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:20.818 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.805+0000 7f74b5ad6700 1 -- 172.21.15.42:0/1401354439 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3734349388 0 0) 0x7f74b00a9ba0 con 0x7f74b007e720 2019-12-05T06:21:20.818 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.805+0000 7f74b5ad6700 1 -- 172.21.15.42:0/1401354439 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74980030b0 con 0x7f74b007e720 2019-12-05T06:21:20.818 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.809+0000 7f74b6ad8700 1 -- 172.21.15.42:0/1401354439 >> v1:172.21.15.42:6789/0 conn(0x7f74b007e720 legacy=0x7f74b009f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:20.818 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.809+0000 7f74b6ad8700 1 -- 172.21.15.42:0/1401354439 >> v1:172.21.15.40:6789/0 conn(0x7f74b009fae0 legacy=0x7f74b014efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:20.819 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.809+0000 7f74b5ad6700 1 -- 172.21.15.42:0/1401354439 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2210136017 0 0) 0x7f74b00a0710 con 0x7f74b009fae0 2019-12-05T06:21:20.819 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.809+0000 7f74b6ad8700 1 -- 172.21.15.42:0/1401354439 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74b007eb50 con 0x7f74b00a7740 2019-12-05T06:21:20.819 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.809+0000 7f74b6ad8700 1 --2- 172.21.15.42:0/1401354439 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f74b00a7740 0x7f74b008df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f74a80032b0 tx=0x7f74a800ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:20.819 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.809+0000 7f74b6ad8700 1 --2- 172.21.15.42:0/1401354439 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f74b00a7740 0x7f74b008df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f74a80032b0 tx=0x7f74a800ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:20.819 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.809+0000 7f74b6ad8700 1 --2- 172.21.15.42:0/1401354439 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f74b00a7740 0x7f74b008df80 secure :-1 s=READY pgs=544 cs=0 l=1 rx=0x7f74a80032b0 tx=0x7f74a800ff70).ready entity=mon.2 client_cookie=c2110539ce9b46ac server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:21:20.820 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.809+0000 7f74b6ad8700 1 --2- 172.21.15.42:0/1401354439 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f74b00a7740 0x7f74b008df80 secure :-1 s=THROTTLE_DONE pgs=544 cs=0 l=1 rx=0x7f74a80032b0 tx=0x7f74a800ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:20.820 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.809+0000 7f74b5ad6700 1 -- 172.21.15.42:0/1401354439 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f74a8011a10 con 0x7f74b00a7740 2019-12-05T06:21:20.820 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.809+0000 7f74b6ad8700 1 --2- 172.21.15.42:0/1401354439 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f74b00a7740 0x7f74b008df80 secure :-1 s=THROTTLE_DONE pgs=544 cs=0 l=1 rx=0x7f74a80032b0 tx=0x7f74a800ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:20.820 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.809+0000 7f74b6ad8700 1 --2- 172.21.15.42:0/1401354439 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f74b00a7740 0x7f74b008df80 secure :-1 s=THROTTLE_DONE pgs=544 cs=0 l=1 rx=0x7f74a80032b0 tx=0x7f74a800ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:20.820 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.809+0000 7f74b5ad6700 1 -- 172.21.15.42:0/1401354439 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74a8011630 con 0x7f74b00a7740 2019-12-05T06:21:20.821 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.809+0000 7f74b5ad6700 1 -- 172.21.15.42:0/1401354439 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f74a800a160 con 0x7f74b00a7740 2019-12-05T06:21:20.821 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.809+0000 7f74b853b700 1 -- 172.21.15.42:0/1401354439 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f74b00a7740 msgr2=0x7f74b008df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:20.821 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.809+0000 7f74b853b700 1 --2- 172.21.15.42:0/1401354439 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f74b00a7740 0x7f74b008df80 secure :-1 s=READY pgs=544 cs=0 l=1 rx=0x7f74a80032b0 tx=0x7f74a800ff70).stop 2019-12-05T06:21:20.821 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.809+0000 7f74b853b700 1 -- 172.21.15.42:0/1401354439 shutdown_connections 2019-12-05T06:21:20.821 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.809+0000 7f74b853b700 1 --2- 172.21.15.42:0/1401354439 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f74b00a7740 0x7f74b008df80 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:20.821 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.809+0000 7f74b853b700 1 -- 172.21.15.42:0/1401354439 shutdown_connections 2019-12-05T06:21:20.822 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.809+0000 7f74b853b700 1 -- 172.21.15.42:0/1401354439 wait complete. 2019-12-05T06:21:20.822 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.809+0000 7f74b853b700 1 -- 172.21.15.42:0/1401354439 >> 172.21.15.42:0/1401354439 conn(0x7f74b00aabe0 msgr2=0x7f74b00a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:21:20.822 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.809+0000 7f74b853b700 1 Processor -- start 2019-12-05T06:21:20.822 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.809+0000 7f74b853b700 1 -- start start 2019-12-05T06:21:20.827 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.813+0000 7f74b853b700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f74b00a3e30 0x7f74b0142c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:21:20.827 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.813+0000 7f74b853b700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74b007ae20 con 0x7f74b014e690 2019-12-05T06:21:20.827 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.813+0000 7f74b853b700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74b0144d40 con 0x7f74b01431c0 2019-12-05T06:21:20.828 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.813+0000 7f74b853b700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74b0144560 con 0x7f74b00a3e30 2019-12-05T06:21:20.828 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.813+0000 7f74b62d7700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f74b01431c0 0x7f74b007a620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:53654/0 (socket says 172.21.15.42:53654) 2019-12-05T06:21:20.828 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.813+0000 7f74b62d7700 1 -- 172.21.15.42:0/1839111673 learned_addr learned my addr 172.21.15.42:0/1839111673 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:21:20.828 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.813+0000 7f74b6ad8700 1 --2- 172.21.15.42:0/1839111673 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f74b00a3e30 0x7f74b0142c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:20.828 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.813+0000 7f74af7fe700 1 -- 172.21.15.42:0/1839111673 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1414746770 0 0) 0x7f74b0144d40 con 0x7f74b01431c0 2019-12-05T06:21:20.828 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.813+0000 7f74af7fe700 1 -- 172.21.15.42:0/1839111673 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74900030b0 con 0x7f74b01431c0 2019-12-05T06:21:20.828 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.813+0000 7f74af7fe700 1 -- 172.21.15.42:0/1839111673 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2707234000 0 0) 0x7f74b007ae20 con 0x7f74b014e690 2019-12-05T06:21:20.829 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.813+0000 7f74af7fe700 1 -- 172.21.15.42:0/1839111673 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74b0144d40 con 0x7f74b014e690 2019-12-05T06:21:20.829 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.813+0000 7f74af7fe700 1 -- 172.21.15.42:0/1839111673 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 4238857451 0 0) 0x7f74900030b0 con 0x7f74b01431c0 2019-12-05T06:21:20.829 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.813+0000 7f74af7fe700 1 -- 172.21.15.42:0/1839111673 --> v1:172.21.15.42:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f74b007ae20 con 0x7f74b01431c0 2019-12-05T06:21:20.829 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.813+0000 7f74af7fe700 1 -- 172.21.15.42:0/1839111673 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7f74a4001930 con 0x7f74b01431c0 2019-12-05T06:21:20.829 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.813+0000 7f74af7fe700 1 -- 172.21.15.42:0/1839111673 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f74b00a3e30 msgr2=0x7f74b0142c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:20.829 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.813+0000 7f74af7fe700 1 --2- 172.21.15.42:0/1839111673 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f74b00a3e30 0x7f74b0142c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:20.829 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.813+0000 7f74af7fe700 1 -- 172.21.15.42:0/1839111673 >> v1:172.21.15.42:6789/0 conn(0x7f74b01431c0 legacy=0x7f74b007a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:20.830 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.813+0000 7f74af7fe700 1 -- 172.21.15.42:0/1839111673 >> v1:172.21.15.40:6789/0 conn(0x7f74b014e690 legacy=0x7f74b014eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:20.830 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.813+0000 7f74af7fe700 1 --2- 172.21.15.42:0/1839111673 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7490003ab0 0x7f7490004420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:21:20.830 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.813+0000 7f74af7fe700 1 --2- 172.21.15.42:0/1839111673 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7490003ee0 0x7f7490004970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:21:20.830 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.813+0000 7f74af7fe700 1 --2- 172.21.15.42:0/1839111673 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7490004f60 0x7f749000b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:21:20.830 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.813+0000 7f74af7fe700 1 -- 172.21.15.42:0/1839111673 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74b0144560 con 0x7f7490004f60 2019-12-05T06:21:20.830 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.813+0000 7f74af7fe700 1 -- 172.21.15.42:0/1839111673 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f749000bd10 con 0x7f7490003ee0 2019-12-05T06:21:20.830 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.813+0000 7f74af7fe700 1 -- 172.21.15.42:0/1839111673 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f749000bf50 con 0x7f7490003ab0 2019-12-05T06:21:20.831 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.813+0000 7f74b62d7700 1 --2- 172.21.15.42:0/1839111673 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7490003ee0 0x7f7490004970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:20.831 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.813+0000 7f74b6ad8700 1 --2- 172.21.15.42:0/1839111673 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7490003ab0 0x7f7490004420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:20.831 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.813+0000 7f74b72d9700 1 --2- 172.21.15.42:0/1839111673 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7490004f60 0x7f749000b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:20.831 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.813+0000 7f74b62d7700 1 -- 172.21.15.42:0/1839111673 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7490003ab0 msgr2=0x7f7490004420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:20.831 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.813+0000 7f74b62d7700 1 --2- 172.21.15.42:0/1839111673 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7490003ab0 0x7f7490004420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:20.831 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.813+0000 7f74b62d7700 1 -- 172.21.15.42:0/1839111673 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7490004f60 msgr2=0x7f749000b670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:20.832 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.813+0000 7f74b62d7700 1 --2- 172.21.15.42:0/1839111673 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7490004f60 0x7f749000b670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:20.832 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.813+0000 7f74b62d7700 1 -- 172.21.15.42:0/1839111673 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74b014f2c0 con 0x7f7490003ee0 2019-12-05T06:21:20.832 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.813+0000 7f74b853b700 1 -- 172.21.15.42:0/1839111673 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74b00a3e30 con 0x7f7490003ee0 2019-12-05T06:21:20.832 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.813+0000 7f74b62d7700 1 --2- 172.21.15.42:0/1839111673 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7490003ee0 0x7f7490004970 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f74a4001700 tx=0x7f74a4011540).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:20.833 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.813+0000 7f74b853b700 1 -- 172.21.15.42:0/1839111673 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74b0142c70 con 0x7f7490003ee0 2019-12-05T06:21:20.833 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.813+0000 7f74b62d7700 1 --2- 172.21.15.42:0/1839111673 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7490003ee0 0x7f7490004970 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f74a4001700 tx=0x7f74a4011540).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:20.833 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.813+0000 7f74b62d7700 1 --2- 172.21.15.42:0/1839111673 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7490003ee0 0x7f7490004970 secure :-1 s=READY pgs=2504 cs=0 l=1 rx=0x7f74a4001700 tx=0x7f74a4011540).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:21:20.833 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.813+0000 7f74b62d7700 1 --2- 172.21.15.42:0/1839111673 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7490003ee0 0x7f7490004970 secure :-1 s=THROTTLE_DONE pgs=2504 cs=0 l=1 rx=0x7f74a4001700 tx=0x7f74a4011540).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:20.834 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.813+0000 7f74b62d7700 1 --2- 172.21.15.42:0/1839111673 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7490003ee0 0x7f7490004970 secure :-1 s=THROTTLE_DONE pgs=2504 cs=0 l=1 rx=0x7f74a4001700 tx=0x7f74a4011540).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:20.834 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.813+0000 7f74af7fe700 1 -- 172.21.15.42:0/1839111673 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f74a400b070 con 0x7f7490003ee0 2019-12-05T06:21:20.834 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.813+0000 7f74b62d7700 1 --2- 172.21.15.42:0/1839111673 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7490003ee0 0x7f7490004970 secure :-1 s=THROTTLE_DONE pgs=2504 cs=0 l=1 rx=0x7f74a4001700 tx=0x7f74a4011540).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:20.834 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.813+0000 7f74af7fe700 1 -- 172.21.15.42:0/1839111673 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74a4011b50 con 0x7f7490003ee0 2019-12-05T06:21:20.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.813+0000 7f74af7fe700 1 -- 172.21.15.42:0/1839111673 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f74a400a7a0 con 0x7f7490003ee0 2019-12-05T06:21:20.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.813+0000 7f74b62d7700 1 --2- 172.21.15.42:0/1839111673 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7490003ee0 0x7f7490004970 secure :-1 s=THROTTLE_DONE pgs=2504 cs=0 l=1 rx=0x7f74a4001700 tx=0x7f74a4011540).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:20.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.813+0000 7f74af7fe700 1 -- 172.21.15.42:0/1839111673 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f74a400a9a0 con 0x7f7490003ee0 2019-12-05T06:21:20.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.817+0000 7f74b62d7700 1 --2- 172.21.15.42:0/1839111673 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7490003ee0 0x7f7490004970 secure :-1 s=THROTTLE_DONE pgs=2504 cs=0 l=1 rx=0x7f74a4001700 tx=0x7f74a4011540).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:20.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.817+0000 7f74af7fe700 1 --2- 172.21.15.42:0/1839111673 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f7490043e30 0x7f7490046340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:21:20.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.817+0000 7f74af7fe700 1 -- 172.21.15.42:0/1839111673 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 10589+0+0 (secure 0 0 0) 0x7f74a404b360 con 0x7f7490003ee0 2019-12-05T06:21:20.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.817+0000 7f74b6ad8700 1 --2- 172.21.15.42:0/1839111673 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f7490043e30 0x7f7490046340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:20.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.817+0000 7f74b6ad8700 1 --2- 172.21.15.42:0/1839111673 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f7490043e30 0x7f7490046340 crc :-1 s=READY pgs=1046 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:21:20.837 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.821+0000 7f74b853b700 1 -- 172.21.15.42:0/1839111673 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f748c003200 con 0x7f7490003ee0 2019-12-05T06:21:20.838 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.829+0000 7f74b62d7700 1 --2- 172.21.15.42:0/1839111673 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7490003ee0 0x7f7490004970 secure :-1 s=THROTTLE_DONE pgs=2504 cs=0 l=1 rx=0x7f74a4001700 tx=0x7f74a4011540).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:20.838 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.829+0000 7f74af7fe700 1 -- 172.21.15.42:0/1839111673 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f74a4032400 con 0x7f7490003ee0 2019-12-05T06:21:20.996 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.989+0000 7f74b853b700 1 -- 172.21.15.42:0/1839111673 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f748c003630 con 0x7f7490003ee0 2019-12-05T06:21:20.997 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.993+0000 7f74b62d7700 1 --2- 172.21.15.42:0/1839111673 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7490003ee0 0x7f7490004970 secure :-1 s=THROTTLE_DONE pgs=2504 cs=0 l=1 rx=0x7f74a4001700 tx=0x7f74a4011540).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:20.997 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.993+0000 7f74af7fe700 1 -- 172.21.15.42:0/1839111673 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 298 v298) v1 ==== 95+0+2114 (secure 0 0 0) 0x7f74a400a190 con 0x7f7490003ee0 2019-12-05T06:21:20.997 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 298 2019-12-05T06:21:20.997 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:21:20.997 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":298,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":10710,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/526579407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":526579407},{"type":"v1","addr":"172.21.15.40:6833","nonce":526579407}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":298}],"filesystems":[{"mdsmap":{"epoch":297,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:19:55.115618+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":102,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10711},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10711":{"gid":10711,"name":"a","rank":0,"incarnation":294,"state":"up:active","state_seq":12,"addr":"172.21.15.42:6835/2638662429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2638662429},{"type":"v1","addr":"172.21.15.42:6835","nonce":2638662429}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:21:21.005 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.993+0000 7f74b853b700 1 -- 172.21.15.42:0/1839111673 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f7490043e30 msgr2=0x7f7490046340 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:21.005 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.993+0000 7f74b853b700 1 --2- 172.21.15.42:0/1839111673 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f7490043e30 0x7f7490046340 crc :-1 s=READY pgs=1046 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:21.005 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.993+0000 7f74b853b700 1 -- 172.21.15.42:0/1839111673 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7490003ee0 msgr2=0x7f7490004970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:21.006 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.993+0000 7f74b72d9700 1 -- 172.21.15.42:0/1839111673 reap_dead start 2019-12-05T06:21:21.006 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.993+0000 7f74b853b700 1 --2- 172.21.15.42:0/1839111673 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7490003ee0 0x7f7490004970 secure :-1 s=READY pgs=2504 cs=0 l=1 rx=0x7f74a4001700 tx=0x7f74a4011540).stop 2019-12-05T06:21:21.006 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.993+0000 7f74b853b700 1 -- 172.21.15.42:0/1839111673 shutdown_connections 2019-12-05T06:21:21.006 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.993+0000 7f74b853b700 1 --2- 172.21.15.42:0/1839111673 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7490003ee0 0x7f7490004970 unknown :-1 s=CLOSED pgs=2504 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:21.006 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.993+0000 7f74b853b700 1 -- 172.21.15.42:0/1839111673 shutdown_connections 2019-12-05T06:21:21.006 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.993+0000 7f74b853b700 1 -- 172.21.15.42:0/1839111673 wait complete. 2019-12-05T06:21:21.006 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:20.993+0000 7f74b853b700 1 -- 172.21.15.42:0/1839111673 >> 172.21.15.42:0/1839111673 conn(0x7f74b007b940 msgr2=0x7f74b00a6a60 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:21:21.009 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:21:23.010 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:21:23.109 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:23.101+0000 7f7c45875700 1 Processor -- start 2019-12-05T06:21:23.110 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:23.101+0000 7f7c45875700 1 -- start start 2019-12-05T06:21:23.110 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:23.101+0000 7f7c45875700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7c400a7730 0x7f7c4008df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:21:23.110 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:23.101+0000 7f7c45875700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f7c4007e710 0x7f7c4007eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:21:23.110 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:23.101+0000 7f7c45875700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7c4009f2d0 0x7f7c4009f700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:21:23.110 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:23.101+0000 7f7c45875700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c400a0af0 con 0x7f7c4009f2d0 2019-12-05T06:21:23.110 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:23.101+0000 7f7c45875700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c4008e5a0 con 0x7f7c4007e710 2019-12-05T06:21:23.110 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:23.101+0000 7f7c45875700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c4008e840 con 0x7f7c400a7730 2019-12-05T06:21:23.111 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:23.101+0000 7f7c3effd700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f7c4007e710 0x7f7c4007eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:23.111 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:23.101+0000 7f7c3effd700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f7c4007e710 0x7f7c4007eb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:53648/0 (socket says 172.21.15.42:53648) 2019-12-05T06:21:23.111 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:23.101+0000 7f7c3effd700 1 -- 172.21.15.42:0/3883300760 learned_addr learned my addr 172.21.15.42:0/3883300760 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:21:23.111 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:23.101+0000 7f7c3f7fe700 1 --2- 172.21.15.42:0/3883300760 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7c400a7730 0x7f7c4008df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:23.111 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:23.101+0000 7f7c3ffff700 1 --2- 172.21.15.42:0/3883300760 >> v2:172.21.15.40:3300/0 conn(0x7f7c4009f2d0 0x7f7c4009f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:23.111 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:23.101+0000 7f7c3effd700 1 -- 172.21.15.42:0/3883300760 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7c400a7730 msgr2=0x7f7c4008df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:21:23.111 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:23.101+0000 7f7c3effd700 1 --2- 172.21.15.42:0/3883300760 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7c400a7730 0x7f7c4008df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:21:23.112 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:23.101+0000 7f7c3effd700 1 -- 172.21.15.42:0/3883300760 >> v2:172.21.15.40:3300/0 conn(0x7f7c4009f2d0 msgr2=0x7f7c4009f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:21:23.112 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:23.101+0000 7f7c3effd700 1 --2- 172.21.15.42:0/3883300760 >> v2:172.21.15.40:3300/0 conn(0x7f7c4009f2d0 0x7f7c4009f700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:21:23.112 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:23.101+0000 7f7c3effd700 1 -- 172.21.15.42:0/3883300760 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c4007f090 con 0x7f7c4007e710 2019-12-05T06:21:23.112 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:23.101+0000 7f7c3effd700 1 --2- 172.21.15.42:0/3883300760 >> v2:172.21.15.42:3300/0 conn(0x7f7c4007e710 0x7f7c4007eb40 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f7c4008e760 tx=0x7f7c2800fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:23.112 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:23.101+0000 7f7c3effd700 1 --2- 172.21.15.42:0/3883300760 >> v2:172.21.15.42:3300/0 conn(0x7f7c4007e710 0x7f7c4007eb40 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f7c4008e760 tx=0x7f7c2800fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:23.112 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:23.101+0000 7f7c3effd700 1 --2- 172.21.15.42:0/3883300760 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7c4007e710 0x7f7c4007eb40 secure :-1 s=READY pgs=2505 cs=0 l=1 rx=0x7f7c4008e760 tx=0x7f7c2800fdb0).ready entity=mon.1 client_cookie=aca7f86cdcb054cd server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:21:23.113 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:23.101+0000 7f7c3effd700 1 --2- 172.21.15.42:0/3883300760 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7c4007e710 0x7f7c4007eb40 secure :-1 s=THROTTLE_DONE pgs=2505 cs=0 l=1 rx=0x7f7c4008e760 tx=0x7f7c2800fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:23.113 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:23.101+0000 7f7c3effd700 1 --2- 172.21.15.42:0/3883300760 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7c4007e710 0x7f7c4007eb40 secure :-1 s=THROTTLE_DONE pgs=2505 cs=0 l=1 rx=0x7f7c4008e760 tx=0x7f7c2800fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:23.113 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:23.101+0000 7f7c3e7fc700 1 -- 172.21.15.42:0/3883300760 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7c28008c90 con 0x7f7c4007e710 2019-12-05T06:21:23.113 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:23.101+0000 7f7c3effd700 1 --2- 172.21.15.42:0/3883300760 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7c4007e710 0x7f7c4007eb40 secure :-1 s=THROTTLE_DONE pgs=2505 cs=0 l=1 rx=0x7f7c4008e760 tx=0x7f7c2800fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:23.113 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:23.101+0000 7f7c3e7fc700 1 -- 172.21.15.42:0/3883300760 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c280088b0 con 0x7f7c4007e710 2019-12-05T06:21:23.113 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:23.101+0000 7f7c3e7fc700 1 -- 172.21.15.42:0/3883300760 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7c28011b50 con 0x7f7c4007e710 2019-12-05T06:21:23.113 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:23.101+0000 7f7c45875700 1 -- 172.21.15.42:0/3883300760 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7c4007e710 msgr2=0x7f7c4007eb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:23.113 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:23.101+0000 7f7c45875700 1 --2- 172.21.15.42:0/3883300760 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7c4007e710 0x7f7c4007eb40 secure :-1 s=READY pgs=2505 cs=0 l=1 rx=0x7f7c4008e760 tx=0x7f7c2800fdb0).stop 2019-12-05T06:21:23.114 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:23.105+0000 7f7c45875700 1 -- 172.21.15.42:0/3883300760 shutdown_connections 2019-12-05T06:21:23.114 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:23.105+0000 7f7c45875700 1 --2- 172.21.15.42:0/3883300760 >> v2:172.21.15.40:3300/0 conn(0x7f7c4009f2d0 0x7f7c4009f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:21:23.114 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:23.105+0000 7f7c45875700 1 --2- 172.21.15.42:0/3883300760 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7c400a7730 0x7f7c4008df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:21:23.114 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:23.105+0000 7f7c45875700 1 --2- 172.21.15.42:0/3883300760 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7c4007e710 0x7f7c4007eb40 unknown :-1 s=CLOSED pgs=2505 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:23.114 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:23.105+0000 7f7c45875700 1 -- 172.21.15.42:0/3883300760 shutdown_connections 2019-12-05T06:21:23.114 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:23.105+0000 7f7c45875700 1 -- 172.21.15.42:0/3883300760 wait complete. 2019-12-05T06:21:23.114 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:23.105+0000 7f7c45875700 1 -- 172.21.15.42:0/3883300760 >> 172.21.15.42:0/3883300760 conn(0x7f7c400aabd0 msgr2=0x7f7c400a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:21:23.118 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:23.105+0000 7f7c45875700 1 Processor -- start 2019-12-05T06:21:23.118 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:23.105+0000 7f7c45875700 1 -- start start 2019-12-05T06:21:23.119 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:23.105+0000 7f7c45875700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f7c400a9ac0 0x7f7c4012d100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:21:23.119 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:23.105+0000 7f7c45875700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7c401443e0 0x7f7c40142be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:21:23.119 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:23.105+0000 7f7c45875700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c40144810 con 0x7f7c4007b7a0 2019-12-05T06:21:23.119 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:23.105+0000 7f7c45875700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c401426a0 con 0x7f7c401443e0 2019-12-05T06:21:23.119 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:23.105+0000 7f7c45875700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c40142940 con 0x7f7c400a9ac0 2019-12-05T06:21:23.119 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:23.105+0000 7f7c3f7fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f7c400a9ac0 0x7f7c4012d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:23.119 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:23.105+0000 7f7c3f7fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f7c400a9ac0 0x7f7c4012d100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:53652/0 (socket says 172.21.15.42:53652) 2019-12-05T06:21:23.120 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:23.105+0000 7f7c3f7fe700 1 -- 172.21.15.42:0/1970522651 learned_addr learned my addr 172.21.15.42:0/1970522651 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:21:23.120 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:23.105+0000 7f7c3effd700 1 --2- 172.21.15.42:0/1970522651 >> v2:172.21.15.40:3300/0 conn(0x7f7c401443e0 0x7f7c40142be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:23.120 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:23.109+0000 7f7c3f7fe700 1 -- 172.21.15.42:0/1970522651 >> v2:172.21.15.40:3300/0 conn(0x7f7c401443e0 msgr2=0x7f7c40142be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:23.120 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:23.109+0000 7f7c3f7fe700 1 --2- 172.21.15.42:0/1970522651 >> v2:172.21.15.40:3300/0 conn(0x7f7c401443e0 0x7f7c40142be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:23.120 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:23.109+0000 7f7c3f7fe700 1 -- 172.21.15.42:0/1970522651 >> v1:172.21.15.42:6789/0 conn(0x7f7c4007b7a0 legacy=0x7f7c4007bbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:23.120 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:23.109+0000 7f7c3f7fe700 1 -- 172.21.15.42:0/1970522651 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c40143130 con 0x7f7c400a9ac0 2019-12-05T06:21:23.120 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:23.109+0000 7f7c3effd700 1 --2- 172.21.15.42:0/1970522651 >> v2:172.21.15.40:3300/0 conn(0x7f7c401443e0 0x7f7c40142be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:21:23.121 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:23.109+0000 7f7c3f7fe700 1 --2- 172.21.15.42:0/1970522651 >> v2:172.21.15.42:3300/0 conn(0x7f7c400a9ac0 0x7f7c4012d100 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f7c30003070 tx=0x7f7c30012d70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:23.121 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:23.109+0000 7f7c3f7fe700 1 --2- 172.21.15.42:0/1970522651 >> v2:172.21.15.42:3300/0 conn(0x7f7c400a9ac0 0x7f7c4012d100 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f7c30003070 tx=0x7f7c30012d70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:23.121 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:23.109+0000 7f7c3f7fe700 1 --2- 172.21.15.42:0/1970522651 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7c400a9ac0 0x7f7c4012d100 secure :-1 s=READY pgs=2507 cs=0 l=1 rx=0x7f7c30003070 tx=0x7f7c30012d70).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:21:23.121 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:23.109+0000 7f7c3f7fe700 1 --2- 172.21.15.42:0/1970522651 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7c400a9ac0 0x7f7c4012d100 secure :-1 s=THROTTLE_DONE pgs=2507 cs=0 l=1 rx=0x7f7c30003070 tx=0x7f7c30012d70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:23.121 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:23.109+0000 7f7c37fff700 1 -- 172.21.15.42:0/1970522651 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7c300147e0 con 0x7f7c400a9ac0 2019-12-05T06:21:23.121 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:23.109+0000 7f7c3f7fe700 1 --2- 172.21.15.42:0/1970522651 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7c400a9ac0 0x7f7c4012d100 secure :-1 s=THROTTLE_DONE pgs=2507 cs=0 l=1 rx=0x7f7c30003070 tx=0x7f7c30012d70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:23.121 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:23.109+0000 7f7c3f7fe700 1 --2- 172.21.15.42:0/1970522651 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7c400a9ac0 0x7f7c4012d100 secure :-1 s=THROTTLE_DONE pgs=2507 cs=0 l=1 rx=0x7f7c30003070 tx=0x7f7c30012d70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:23.122 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:23.109+0000 7f7c37fff700 1 -- 172.21.15.42:0/1970522651 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c30007150 con 0x7f7c400a9ac0 2019-12-05T06:21:23.122 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:23.109+0000 7f7c37fff700 1 -- 172.21.15.42:0/1970522651 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7c30014dc0 con 0x7f7c400a9ac0 2019-12-05T06:21:23.122 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:23.109+0000 7f7c45875700 1 -- 172.21.15.42:0/1970522651 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c40144f40 con 0x7f7c400a9ac0 2019-12-05T06:21:23.122 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:23.109+0000 7f7c45875700 1 -- 172.21.15.42:0/1970522651 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c40118420 con 0x7f7c400a9ac0 2019-12-05T06:21:23.122 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:23.109+0000 7f7c3f7fe700 1 --2- 172.21.15.42:0/1970522651 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7c400a9ac0 0x7f7c4012d100 secure :-1 s=THROTTLE_DONE pgs=2507 cs=0 l=1 rx=0x7f7c30003070 tx=0x7f7c30012d70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:23.122 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:23.109+0000 7f7c37fff700 1 -- 172.21.15.42:0/1970522651 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f7c300055d0 con 0x7f7c400a9ac0 2019-12-05T06:21:23.122 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:23.109+0000 7f7c3f7fe700 1 --2- 172.21.15.42:0/1970522651 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7c400a9ac0 0x7f7c4012d100 secure :-1 s=THROTTLE_DONE pgs=2507 cs=0 l=1 rx=0x7f7c30003070 tx=0x7f7c30012d70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:23.123 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:23.109+0000 7f7c37fff700 1 --2- 172.21.15.42:0/1970522651 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f7c1803ae40 0x7f7c1803d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:21:23.123 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:23.109+0000 7f7c37fff700 1 -- 172.21.15.42:0/1970522651 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 10589+0+0 (secure 0 0 0) 0x7f7c3004cfd0 con 0x7f7c400a9ac0 2019-12-05T06:21:23.123 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:23.109+0000 7f7c3effd700 1 --2- 172.21.15.42:0/1970522651 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f7c1803ae40 0x7f7c1803d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:23.123 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:23.109+0000 7f7c3effd700 1 --2- 172.21.15.42:0/1970522651 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f7c1803ae40 0x7f7c1803d350 crc :-1 s=READY pgs=1047 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:21:23.124 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:23.117+0000 7f7c45875700 1 -- 172.21.15.42:0/1970522651 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c401453f0 con 0x7f7c400a9ac0 2019-12-05T06:21:23.133 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:23.125+0000 7f7c3f7fe700 1 --2- 172.21.15.42:0/1970522651 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7c400a9ac0 0x7f7c4012d100 secure :-1 s=THROTTLE_DONE pgs=2507 cs=0 l=1 rx=0x7f7c30003070 tx=0x7f7c30012d70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:23.134 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:23.125+0000 7f7c37fff700 1 -- 172.21.15.42:0/1970522651 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f7c30034050 con 0x7f7c400a9ac0 2019-12-05T06:21:23.294 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:23.289+0000 7f7c45875700 1 -- 172.21.15.42:0/1970522651 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7c4005d100 con 0x7f7c400a9ac0 2019-12-05T06:21:23.294 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:23.289+0000 7f7c3f7fe700 1 --2- 172.21.15.42:0/1970522651 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7c400a9ac0 0x7f7c4012d100 secure :-1 s=THROTTLE_DONE pgs=2507 cs=0 l=1 rx=0x7f7c30003070 tx=0x7f7c30012d70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:23.295 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:23.289+0000 7f7c37fff700 1 -- 172.21.15.42:0/1970522651 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 298 v298) v1 ==== 95+0+2114 (secure 0 0 0) 0x7f7c300342b0 con 0x7f7c400a9ac0 2019-12-05T06:21:23.295 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 298 2019-12-05T06:21:23.295 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:21:23.295 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":298,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":10710,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/526579407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":526579407},{"type":"v1","addr":"172.21.15.40:6833","nonce":526579407}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":298}],"filesystems":[{"mdsmap":{"epoch":297,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:19:55.115618+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":102,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10711},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10711":{"gid":10711,"name":"a","rank":0,"incarnation":294,"state":"up:active","state_seq":12,"addr":"172.21.15.42:6835/2638662429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2638662429},{"type":"v1","addr":"172.21.15.42:6835","nonce":2638662429}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:21:23.297 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:23.293+0000 7f7c45875700 1 -- 172.21.15.42:0/1970522651 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f7c1803ae40 msgr2=0x7f7c1803d350 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:23.298 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:23.293+0000 7f7c45875700 1 --2- 172.21.15.42:0/1970522651 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f7c1803ae40 0x7f7c1803d350 crc :-1 s=READY pgs=1047 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:23.298 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:23.293+0000 7f7c45875700 1 -- 172.21.15.42:0/1970522651 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7c400a9ac0 msgr2=0x7f7c4012d100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:23.298 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:23.293+0000 7f7c45875700 1 --2- 172.21.15.42:0/1970522651 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7c400a9ac0 0x7f7c4012d100 secure :-1 s=READY pgs=2507 cs=0 l=1 rx=0x7f7c30003070 tx=0x7f7c30012d70).stop 2019-12-05T06:21:23.298 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:23.293+0000 7f7c45875700 1 -- 172.21.15.42:0/1970522651 shutdown_connections 2019-12-05T06:21:23.298 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:23.293+0000 7f7c45875700 1 --2- 172.21.15.42:0/1970522651 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f7c1803ae40 0x7f7c1803d350 unknown :-1 s=CLOSED pgs=1047 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:23.298 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:23.293+0000 7f7c45875700 1 --2- 172.21.15.42:0/1970522651 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7c400a9ac0 0x7f7c4012d100 unknown :-1 s=CLOSED pgs=2507 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:23.299 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:23.293+0000 7f7c45875700 1 --2- 172.21.15.42:0/1970522651 >> v2:172.21.15.40:3300/0 conn(0x7f7c401443e0 0x7f7c40142be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:23.299 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:23.293+0000 7f7c45875700 1 -- 172.21.15.42:0/1970522651 shutdown_connections 2019-12-05T06:21:23.299 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:23.293+0000 7f7c45875700 1 -- 172.21.15.42:0/1970522651 wait complete. 2019-12-05T06:21:23.299 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:23.293+0000 7f7c45875700 1 -- 172.21.15.42:0/1970522651 >> 172.21.15.42:0/1970522651 conn(0x7f7c400a8a60 msgr2=0x7f7c40059ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:21:23.307 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:21:25.309 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:21:25.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:25.393+0000 7f947f9b3700 1 Processor -- start 2019-12-05T06:21:25.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:25.393+0000 7f947f9b3700 1 -- start start 2019-12-05T06:21:25.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:25.393+0000 7f947f9b3700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f94780a7740 0x7f947808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:21:25.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:25.393+0000 7f947f9b3700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f947809fae0 0x7f947807eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:21:25.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:25.393+0000 7f947f9b3700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94780a0710 con 0x7f947807e720 2019-12-05T06:21:25.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:25.393+0000 7f947f9b3700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94780a0b00 con 0x7f94780a7740 2019-12-05T06:21:25.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:25.393+0000 7f947f9b3700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f947808e5b0 con 0x7f947809fae0 2019-12-05T06:21:25.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:25.393+0000 7f947df50700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f94780a7740 0x7f947808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:25.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:25.393+0000 7f947df50700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f94780a7740 0x7f947808df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:53662/0 (socket says 172.21.15.42:53662) 2019-12-05T06:21:25.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:25.393+0000 7f947df50700 1 -- 172.21.15.42:0/4013180219 learned_addr learned my addr 172.21.15.42:0/4013180219 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:21:25.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:25.393+0000 7f947df50700 1 -- 172.21.15.42:0/4013180219 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f947809fae0 msgr2=0x7f947807eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:21:25.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:25.393+0000 7f947e751700 1 --2- 172.21.15.42:0/4013180219 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f947809fae0 0x7f947807eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:25.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:25.393+0000 7f947df50700 1 --2- 172.21.15.42:0/4013180219 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f947809fae0 0x7f947807eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:21:25.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:25.393+0000 7f947df50700 1 -- 172.21.15.42:0/4013180219 >> v1:172.21.15.40:6789/0 conn(0x7f947807e720 legacy=0x7f947809f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:21:25.404 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:25.393+0000 7f947df50700 1 -- 172.21.15.42:0/4013180219 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f947807f0a0 con 0x7f94780a7740 2019-12-05T06:21:25.404 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:25.393+0000 7f947e751700 1 --2- 172.21.15.42:0/4013180219 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f947809fae0 0x7f947807eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).send_auth_request state changed! 2019-12-05T06:21:25.404 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:25.393+0000 7f947df50700 1 --2- 172.21.15.42:0/4013180219 >> v2:172.21.15.42:3300/0 conn(0x7f94780a7740 0x7f947808df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f9470008b10 tx=0x7f94700100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:25.404 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:25.393+0000 7f947df50700 1 --2- 172.21.15.42:0/4013180219 >> v2:172.21.15.42:3300/0 conn(0x7f94780a7740 0x7f947808df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f9470008b10 tx=0x7f94700100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:25.404 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:25.393+0000 7f947df50700 1 --2- 172.21.15.42:0/4013180219 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f94780a7740 0x7f947808df80 secure :-1 s=READY pgs=2508 cs=0 l=1 rx=0x7f9470008b10 tx=0x7f94700100b0).ready entity=mon.1 client_cookie=f8dacb6a655a1daf server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:21:25.404 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:25.393+0000 7f947df50700 1 --2- 172.21.15.42:0/4013180219 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f94780a7740 0x7f947808df80 secure :-1 s=THROTTLE_DONE pgs=2508 cs=0 l=1 rx=0x7f9470008b10 tx=0x7f94700100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:25.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:25.393+0000 7f947cf4e700 1 -- 172.21.15.42:0/4013180219 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9470010da0 con 0x7f94780a7740 2019-12-05T06:21:25.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:25.393+0000 7f947df50700 1 --2- 172.21.15.42:0/4013180219 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f94780a7740 0x7f947808df80 secure :-1 s=THROTTLE_DONE pgs=2508 cs=0 l=1 rx=0x7f9470008b10 tx=0x7f94700100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:25.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:25.393+0000 7f947df50700 1 --2- 172.21.15.42:0/4013180219 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f94780a7740 0x7f947808df80 secure :-1 s=THROTTLE_DONE pgs=2508 cs=0 l=1 rx=0x7f9470008b10 tx=0x7f94700100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:25.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:25.393+0000 7f947cf4e700 1 -- 172.21.15.42:0/4013180219 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94700109c0 con 0x7f94780a7740 2019-12-05T06:21:25.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:25.393+0000 7f947cf4e700 1 -- 172.21.15.42:0/4013180219 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f94700114a0 con 0x7f94780a7740 2019-12-05T06:21:25.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:25.393+0000 7f947f9b3700 1 -- 172.21.15.42:0/4013180219 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f94780a7740 msgr2=0x7f947808df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:25.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:25.393+0000 7f947f9b3700 1 --2- 172.21.15.42:0/4013180219 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f94780a7740 0x7f947808df80 secure :-1 s=READY pgs=2508 cs=0 l=1 rx=0x7f9470008b10 tx=0x7f94700100b0).stop 2019-12-05T06:21:25.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:25.397+0000 7f947f9b3700 1 -- 172.21.15.42:0/4013180219 shutdown_connections 2019-12-05T06:21:25.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:25.397+0000 7f947f9b3700 1 --2- 172.21.15.42:0/4013180219 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f947809fae0 0x7f947807eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:21:25.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:25.397+0000 7f947f9b3700 1 --2- 172.21.15.42:0/4013180219 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f94780a7740 0x7f947808df80 unknown :-1 s=CLOSED pgs=2508 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:25.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:25.397+0000 7f947f9b3700 1 -- 172.21.15.42:0/4013180219 shutdown_connections 2019-12-05T06:21:25.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:25.397+0000 7f947f9b3700 1 -- 172.21.15.42:0/4013180219 wait complete. 2019-12-05T06:21:25.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:25.397+0000 7f947f9b3700 1 -- 172.21.15.42:0/4013180219 >> 172.21.15.42:0/4013180219 conn(0x7f94780aabe0 msgr2=0x7f94780a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:21:25.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:25.397+0000 7f947f9b3700 1 Processor -- start 2019-12-05T06:21:25.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:25.397+0000 7f947f9b3700 1 -- start start 2019-12-05T06:21:25.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:25.397+0000 7f947f9b3700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f9478121300 0x7f9478117820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:21:25.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:25.397+0000 7f947f9b3700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f9478117d70 0x7f94780742d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:21:25.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:25.397+0000 7f947f9b3700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9478121a40 con 0x7f947811e3c0 2019-12-05T06:21:25.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:25.397+0000 7f947f9b3700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9478118900 con 0x7f9478121300 2019-12-05T06:21:25.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:25.397+0000 7f947f9b3700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9478118ba0 con 0x7f9478117d70 2019-12-05T06:21:25.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:25.397+0000 7f947e751700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f9478117d70 0x7f94780742d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:25.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:25.397+0000 7f947df50700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f947811e3c0 0x7f94781419e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:53706/0 (socket says 172.21.15.42:53706) 2019-12-05T06:21:25.411 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:25.397+0000 7f947df50700 1 -- 172.21.15.42:0/2298127855 learned_addr learned my addr 172.21.15.42:0/2298127855 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:21:25.411 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:25.397+0000 7f947d74f700 1 --2- 172.21.15.42:0/2298127855 >> v2:172.21.15.40:3300/0 conn(0x7f9478121300 0x7f9478117820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:25.411 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:25.397+0000 7f947e751700 1 -- 172.21.15.42:0/2298127855 >> v2:172.21.15.40:3300/0 conn(0x7f9478121300 msgr2=0x7f9478117820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:25.411 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:25.397+0000 7f946e7fc700 1 -- 172.21.15.42:0/2298127855 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2695665696 0 0) 0x7f9478121a40 con 0x7f947811e3c0 2019-12-05T06:21:25.411 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:25.397+0000 7f947e751700 1 --2- 172.21.15.42:0/2298127855 >> v2:172.21.15.40:3300/0 conn(0x7f9478121300 0x7f9478117820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:25.411 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:25.397+0000 7f947e751700 1 -- 172.21.15.42:0/2298127855 >> v1:172.21.15.42:6789/0 conn(0x7f947811e3c0 legacy=0x7f94781419e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:25.412 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:25.397+0000 7f947e751700 1 -- 172.21.15.42:0/2298127855 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9478121730 con 0x7f9478117d70 2019-12-05T06:21:25.418 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:25.397+0000 7f947e751700 1 --2- 172.21.15.42:0/2298127855 >> v2:172.21.15.42:3300/0 conn(0x7f9478117d70 0x7f94780742d0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f9468002f30 tx=0x7f94680107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:25.418 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:25.397+0000 7f947e751700 1 --2- 172.21.15.42:0/2298127855 >> v2:172.21.15.42:3300/0 conn(0x7f9478117d70 0x7f94780742d0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f9468002f30 tx=0x7f94680107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:25.418 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:25.397+0000 7f947e751700 1 --2- 172.21.15.42:0/2298127855 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9478117d70 0x7f94780742d0 secure :-1 s=READY pgs=2510 cs=0 l=1 rx=0x7f9468002f30 tx=0x7f94680107c0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:21:25.418 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:25.401+0000 7f947e751700 1 --2- 172.21.15.42:0/2298127855 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9478117d70 0x7f94780742d0 secure :-1 s=THROTTLE_DONE pgs=2510 cs=0 l=1 rx=0x7f9468002f30 tx=0x7f94680107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:25.418 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:25.401+0000 7f946e7fc700 1 -- 172.21.15.42:0/2298127855 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9468005b20 con 0x7f9478117d70 2019-12-05T06:21:25.419 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:25.401+0000 7f947e751700 1 --2- 172.21.15.42:0/2298127855 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9478117d70 0x7f94780742d0 secure :-1 s=THROTTLE_DONE pgs=2510 cs=0 l=1 rx=0x7f9468002f30 tx=0x7f94680107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:25.419 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:25.401+0000 7f947e751700 1 --2- 172.21.15.42:0/2298127855 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9478117d70 0x7f94780742d0 secure :-1 s=THROTTLE_DONE pgs=2510 cs=0 l=1 rx=0x7f9468002f30 tx=0x7f94680107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:25.419 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:25.401+0000 7f946e7fc700 1 -- 172.21.15.42:0/2298127855 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9468007200 con 0x7f9478117d70 2019-12-05T06:21:25.419 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:25.401+0000 7f947f9b3700 1 -- 172.21.15.42:0/2298127855 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9478118e40 con 0x7f9478117d70 2019-12-05T06:21:25.419 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:25.401+0000 7f946e7fc700 1 -- 172.21.15.42:0/2298127855 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f94680062b0 con 0x7f9478117d70 2019-12-05T06:21:25.419 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:25.401+0000 7f947f9b3700 1 -- 172.21.15.42:0/2298127855 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94781181a0 con 0x7f9478117d70 2019-12-05T06:21:25.420 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:25.401+0000 7f947e751700 1 --2- 172.21.15.42:0/2298127855 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9478117d70 0x7f94780742d0 secure :-1 s=THROTTLE_DONE pgs=2510 cs=0 l=1 rx=0x7f9468002f30 tx=0x7f94680107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:25.420 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:25.401+0000 7f946e7fc700 1 -- 172.21.15.42:0/2298127855 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f9468007930 con 0x7f9478117d70 2019-12-05T06:21:25.420 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:25.401+0000 7f947e751700 1 --2- 172.21.15.42:0/2298127855 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9478117d70 0x7f94780742d0 secure :-1 s=THROTTLE_DONE pgs=2510 cs=0 l=1 rx=0x7f9468002f30 tx=0x7f94680107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:25.420 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:25.401+0000 7f946e7fc700 1 --2- 172.21.15.42:0/2298127855 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f945803ad00 0x7f945803d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:21:25.420 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:25.401+0000 7f946e7fc700 1 -- 172.21.15.42:0/2298127855 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 10589+0+0 (secure 0 0 0) 0x7f946804acf0 con 0x7f9478117d70 2019-12-05T06:21:25.420 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:25.401+0000 7f947df50700 1 --2- 172.21.15.42:0/2298127855 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f945803ad00 0x7f945803d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:25.420 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:25.401+0000 7f947df50700 1 --2- 172.21.15.42:0/2298127855 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f945803ad00 0x7f945803d210 crc :-1 s=READY pgs=1048 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:21:25.421 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:25.409+0000 7f947f9b3700 1 -- 172.21.15.42:0/2298127855 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f947805d110 con 0x7f9478117d70 2019-12-05T06:21:25.422 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:25.413+0000 7f947e751700 1 --2- 172.21.15.42:0/2298127855 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9478117d70 0x7f94780742d0 secure :-1 s=THROTTLE_DONE pgs=2510 cs=0 l=1 rx=0x7f9468002f30 tx=0x7f94680107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:25.422 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:25.413+0000 7f946e7fc700 1 -- 172.21.15.42:0/2298127855 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f9468031d10 con 0x7f9478117d70 2019-12-05T06:21:25.576 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:25.573+0000 7f947f9b3700 1 -- 172.21.15.42:0/2298127855 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f947805b8c0 con 0x7f9478117d70 2019-12-05T06:21:25.577 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:25.573+0000 7f947e751700 1 --2- 172.21.15.42:0/2298127855 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9478117d70 0x7f94780742d0 secure :-1 s=THROTTLE_DONE pgs=2510 cs=0 l=1 rx=0x7f9468002f30 tx=0x7f94680107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:25.577 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:25.573+0000 7f946e7fc700 1 -- 172.21.15.42:0/2298127855 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 298 v298) v1 ==== 95+0+2114 (secure 0 0 0) 0x7f9468031f70 con 0x7f9478117d70 2019-12-05T06:21:25.578 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 298 2019-12-05T06:21:25.578 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:21:25.578 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":298,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":10710,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/526579407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":526579407},{"type":"v1","addr":"172.21.15.40:6833","nonce":526579407}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":298}],"filesystems":[{"mdsmap":{"epoch":297,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:19:55.115618+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":102,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10711},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10711":{"gid":10711,"name":"a","rank":0,"incarnation":294,"state":"up:active","state_seq":12,"addr":"172.21.15.42:6835/2638662429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2638662429},{"type":"v1","addr":"172.21.15.42:6835","nonce":2638662429}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:21:25.579 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:25.573+0000 7f947f9b3700 1 -- 172.21.15.42:0/2298127855 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f945803ad00 msgr2=0x7f945803d210 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:25.580 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:25.573+0000 7f947f9b3700 1 --2- 172.21.15.42:0/2298127855 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f945803ad00 0x7f945803d210 crc :-1 s=READY pgs=1048 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:25.580 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:25.573+0000 7f947f9b3700 1 -- 172.21.15.42:0/2298127855 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9478117d70 msgr2=0x7f94780742d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:25.580 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:25.573+0000 7f947f9b3700 1 --2- 172.21.15.42:0/2298127855 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9478117d70 0x7f94780742d0 secure :-1 s=READY pgs=2510 cs=0 l=1 rx=0x7f9468002f30 tx=0x7f94680107c0).stop 2019-12-05T06:21:25.580 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:25.573+0000 7f947f9b3700 1 -- 172.21.15.42:0/2298127855 shutdown_connections 2019-12-05T06:21:25.580 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:25.573+0000 7f947f9b3700 1 --2- 172.21.15.42:0/2298127855 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f945803ad00 0x7f945803d210 unknown :-1 s=CLOSED pgs=1048 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:25.580 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:25.573+0000 7f947f9b3700 1 --2- 172.21.15.42:0/2298127855 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9478117d70 0x7f94780742d0 unknown :-1 s=CLOSED pgs=2510 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:25.580 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:25.573+0000 7f947f9b3700 1 --2- 172.21.15.42:0/2298127855 >> v2:172.21.15.40:3300/0 conn(0x7f9478121300 0x7f9478117820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:25.581 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:25.573+0000 7f947f9b3700 1 -- 172.21.15.42:0/2298127855 shutdown_connections 2019-12-05T06:21:25.581 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:25.573+0000 7f947f9b3700 1 -- 172.21.15.42:0/2298127855 wait complete. 2019-12-05T06:21:25.581 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:25.573+0000 7f947f9b3700 1 -- 172.21.15.42:0/2298127855 >> 172.21.15.42:0/2298127855 conn(0x7f94780a9840 msgr2=0x7f947811c750 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:21:25.588 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:21:27.589 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:21:27.685 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:27.677+0000 7f2c57cd9700 1 Processor -- start 2019-12-05T06:21:27.686 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:27.677+0000 7f2c57cd9700 1 -- start start 2019-12-05T06:21:27.686 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:27.677+0000 7f2c57cd9700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2c5008df70 0x7f2c5009f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:21:27.686 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:27.677+0000 7f2c57cd9700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c5009fc50 con 0x7f2c5009f820 2019-12-05T06:21:27.686 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:27.677+0000 7f2c57cd9700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c500a0700 con 0x7f2c500a7730 2019-12-05T06:21:27.686 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:27.677+0000 7f2c57cd9700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c500a0af0 con 0x7f2c5008df70 2019-12-05T06:21:27.687 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:27.677+0000 7f2c56276700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f2c500a7730 0x7f2c5007e710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:53716/0 (socket says 172.21.15.42:53716) 2019-12-05T06:21:27.687 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:27.677+0000 7f2c56276700 1 -- 172.21.15.42:0/32345916 learned_addr learned my addr 172.21.15.42:0/32345916 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:21:27.687 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:27.677+0000 7f2c55a75700 1 --2- 172.21.15.42:0/32345916 >> v2:172.21.15.42:3300/0 conn(0x7f2c5008df70 0x7f2c5009f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:27.687 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:27.677+0000 7f2c55274700 1 -- 172.21.15.42:0/32345916 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2893155758 0 0) 0x7f2c500a0700 con 0x7f2c500a7730 2019-12-05T06:21:27.687 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:27.677+0000 7f2c55274700 1 -- 172.21.15.42:0/32345916 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c3c0030b0 con 0x7f2c500a7730 2019-12-05T06:21:27.687 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:27.677+0000 7f2c55a75700 1 -- 172.21.15.42:0/32345916 >> v1:172.21.15.42:6789/0 conn(0x7f2c500a7730 legacy=0x7f2c5007e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:27.687 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:27.677+0000 7f2c55a75700 1 -- 172.21.15.42:0/32345916 >> v1:172.21.15.40:6789/0 conn(0x7f2c5009f820 legacy=0x7f2c5014efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:27.688 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:27.677+0000 7f2c55a75700 1 -- 172.21.15.42:0/32345916 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c5007ef10 con 0x7f2c5008df70 2019-12-05T06:21:27.688 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:27.677+0000 7f2c55a75700 1 --2- 172.21.15.42:0/32345916 >> v2:172.21.15.42:3300/0 conn(0x7f2c5008df70 0x7f2c5009f2d0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f2c40008790 tx=0x7f2c4000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:27.688 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:27.677+0000 7f2c55a75700 1 --2- 172.21.15.42:0/32345916 >> v2:172.21.15.42:3300/0 conn(0x7f2c5008df70 0x7f2c5009f2d0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f2c40008790 tx=0x7f2c4000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:27.688 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:27.677+0000 7f2c55a75700 1 --2- 172.21.15.42:0/32345916 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2c5008df70 0x7f2c5009f2d0 secure :-1 s=READY pgs=2512 cs=0 l=1 rx=0x7f2c40008790 tx=0x7f2c4000fbf0).ready entity=mon.1 client_cookie=18cfab30f0a1ea44 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:21:27.688 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:27.681+0000 7f2c55a75700 1 --2- 172.21.15.42:0/32345916 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2c5008df70 0x7f2c5009f2d0 secure :-1 s=THROTTLE_DONE pgs=2512 cs=0 l=1 rx=0x7f2c40008790 tx=0x7f2c4000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:27.688 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:27.681+0000 7f2c55274700 1 -- 172.21.15.42:0/32345916 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2c40011270 con 0x7f2c5008df70 2019-12-05T06:21:27.688 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:27.681+0000 7f2c55a75700 1 --2- 172.21.15.42:0/32345916 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2c5008df70 0x7f2c5009f2d0 secure :-1 s=THROTTLE_DONE pgs=2512 cs=0 l=1 rx=0x7f2c40008790 tx=0x7f2c4000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:27.689 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:27.681+0000 7f2c55274700 1 -- 172.21.15.42:0/32345916 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c40008bc0 con 0x7f2c5008df70 2019-12-05T06:21:27.689 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:27.681+0000 7f2c55a75700 1 --2- 172.21.15.42:0/32345916 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2c5008df70 0x7f2c5009f2d0 secure :-1 s=THROTTLE_DONE pgs=2512 cs=0 l=1 rx=0x7f2c40008790 tx=0x7f2c4000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:27.689 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:27.681+0000 7f2c55274700 1 -- 172.21.15.42:0/32345916 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2c40011850 con 0x7f2c5008df70 2019-12-05T06:21:27.689 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:27.681+0000 7f2c57cd9700 1 -- 172.21.15.42:0/32345916 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2c5008df70 msgr2=0x7f2c5009f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:27.689 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:27.681+0000 7f2c57cd9700 1 --2- 172.21.15.42:0/32345916 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2c5008df70 0x7f2c5009f2d0 secure :-1 s=READY pgs=2512 cs=0 l=1 rx=0x7f2c40008790 tx=0x7f2c4000fbf0).stop 2019-12-05T06:21:27.689 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:27.681+0000 7f2c57cd9700 1 -- 172.21.15.42:0/32345916 shutdown_connections 2019-12-05T06:21:27.689 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:27.681+0000 7f2c57cd9700 1 --2- 172.21.15.42:0/32345916 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2c5008df70 0x7f2c5009f2d0 unknown :-1 s=CLOSED pgs=2512 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:27.690 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:27.681+0000 7f2c57cd9700 1 -- 172.21.15.42:0/32345916 shutdown_connections 2019-12-05T06:21:27.690 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:27.681+0000 7f2c57cd9700 1 -- 172.21.15.42:0/32345916 wait complete. 2019-12-05T06:21:27.690 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:27.681+0000 7f2c57cd9700 1 -- 172.21.15.42:0/32345916 >> 172.21.15.42:0/32345916 conn(0x7f2c500aabd0 msgr2=0x7f2c500a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:21:27.694 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:27.681+0000 7f2c57cd9700 1 Processor -- start 2019-12-05T06:21:27.694 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:27.681+0000 7f2c57cd9700 1 -- start start 2019-12-05T06:21:27.695 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:27.681+0000 7f2c57cd9700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2c501431b0 0x7f2c500a33c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:21:27.695 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:27.681+0000 7f2c57cd9700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c500a4250 con 0x7f2c501189d0 2019-12-05T06:21:27.695 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:27.681+0000 7f2c57cd9700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c501437c0 con 0x7f2c500a29b0 2019-12-05T06:21:27.695 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:27.681+0000 7f2c57cd9700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c50142a10 con 0x7f2c501431b0 2019-12-05T06:21:27.695 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:27.681+0000 7f2c56276700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f2c500a29b0 0x7f2c50115ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:53722/0 (socket says 172.21.15.42:53722) 2019-12-05T06:21:27.695 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:27.681+0000 7f2c56276700 1 -- 172.21.15.42:0/983556737 learned_addr learned my addr 172.21.15.42:0/983556737 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:21:27.695 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:27.685+0000 7f2c56a77700 1 --2- 172.21.15.42:0/983556737 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2c501431b0 0x7f2c500a33c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:27.696 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:27.685+0000 7f2c46ffd700 1 -- 172.21.15.42:0/983556737 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2217797213 0 0) 0x7f2c501437c0 con 0x7f2c500a29b0 2019-12-05T06:21:27.696 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:27.685+0000 7f2c46ffd700 1 -- 172.21.15.42:0/983556737 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c380030b0 con 0x7f2c500a29b0 2019-12-05T06:21:27.696 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:27.685+0000 7f2c46ffd700 1 -- 172.21.15.42:0/983556737 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 2181892298 0 0) 0x7f2c380030b0 con 0x7f2c500a29b0 2019-12-05T06:21:27.696 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:27.685+0000 7f2c46ffd700 1 -- 172.21.15.42:0/983556737 --> v1:172.21.15.42:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2c501437c0 con 0x7f2c500a29b0 2019-12-05T06:21:27.697 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:27.685+0000 7f2c46ffd700 1 -- 172.21.15.42:0/983556737 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7f2c4c001990 con 0x7f2c500a29b0 2019-12-05T06:21:27.697 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:27.685+0000 7f2c46ffd700 1 -- 172.21.15.42:0/983556737 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2c501431b0 msgr2=0x7f2c500a33c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:27.697 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:27.685+0000 7f2c46ffd700 1 --2- 172.21.15.42:0/983556737 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2c501431b0 0x7f2c500a33c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:27.697 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:27.685+0000 7f2c46ffd700 1 -- 172.21.15.42:0/983556737 >> v1:172.21.15.42:6789/0 conn(0x7f2c500a29b0 legacy=0x7f2c50115ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:27.697 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:27.685+0000 7f2c46ffd700 1 -- 172.21.15.42:0/983556737 >> v1:172.21.15.40:6789/0 conn(0x7f2c501189d0 legacy=0x7f2c501167f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:27.698 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:27.685+0000 7f2c46ffd700 1 --2- 172.21.15.42:0/983556737 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2c380038a0 0x7f2c38003cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:21:27.698 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:27.685+0000 7f2c46ffd700 1 --2- 172.21.15.42:0/983556737 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2c38004420 0x7f2c38004850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:21:27.698 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:27.685+0000 7f2c46ffd700 1 --2- 172.21.15.42:0/983556737 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2c38004da0 0x7f2c3800b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:21:27.698 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:27.685+0000 7f2c46ffd700 1 -- 172.21.15.42:0/983556737 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c50142a10 con 0x7f2c38004da0 2019-12-05T06:21:27.698 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:27.685+0000 7f2c46ffd700 1 -- 172.21.15.42:0/983556737 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c3800bbc0 con 0x7f2c38004420 2019-12-05T06:21:27.698 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:27.685+0000 7f2c46ffd700 1 -- 172.21.15.42:0/983556737 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c3800bd80 con 0x7f2c380038a0 2019-12-05T06:21:27.698 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:27.685+0000 7f2c55a75700 1 --2- 172.21.15.42:0/983556737 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2c38004420 0x7f2c38004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:27.699 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:27.685+0000 7f2c56276700 1 --2- 172.21.15.42:0/983556737 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2c380038a0 0x7f2c38003cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:27.699 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:27.685+0000 7f2c56a77700 1 --2- 172.21.15.42:0/983556737 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2c38004da0 0x7f2c3800b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:27.699 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:27.685+0000 7f2c55a75700 1 -- 172.21.15.42:0/983556737 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2c380038a0 msgr2=0x7f2c38003cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:27.699 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:27.685+0000 7f2c55a75700 1 --2- 172.21.15.42:0/983556737 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2c380038a0 0x7f2c38003cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:27.699 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:27.685+0000 7f2c55a75700 1 -- 172.21.15.42:0/983556737 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2c38004da0 msgr2=0x7f2c3800b670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:27.699 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:27.685+0000 7f2c55a75700 1 --2- 172.21.15.42:0/983556737 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2c38004da0 0x7f2c3800b670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:27.699 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:27.685+0000 7f2c55a75700 1 -- 172.21.15.42:0/983556737 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c501439f0 con 0x7f2c38004420 2019-12-05T06:21:27.700 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:27.685+0000 7f2c57cd9700 1 -- 172.21.15.42:0/983556737 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c501431b0 con 0x7f2c38004420 2019-12-05T06:21:27.700 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:27.685+0000 7f2c56276700 1 --2- 172.21.15.42:0/983556737 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2c380038a0 0x7f2c38003cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:21:27.700 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:27.685+0000 7f2c55a75700 1 --2- 172.21.15.42:0/983556737 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2c38004420 0x7f2c38004850 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f2c40008370 tx=0x7f2c40011b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:27.700 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:27.685+0000 7f2c57cd9700 1 -- 172.21.15.42:0/983556737 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c500a33c0 con 0x7f2c38004420 2019-12-05T06:21:27.700 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:27.685+0000 7f2c55a75700 1 --2- 172.21.15.42:0/983556737 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2c38004420 0x7f2c38004850 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f2c40008370 tx=0x7f2c40011b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:27.700 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:27.685+0000 7f2c55a75700 1 --2- 172.21.15.42:0/983556737 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2c38004420 0x7f2c38004850 secure :-1 s=READY pgs=2514 cs=0 l=1 rx=0x7f2c40008370 tx=0x7f2c40011b30).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:21:27.700 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:27.685+0000 7f2c55a75700 1 --2- 172.21.15.42:0/983556737 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2c38004420 0x7f2c38004850 secure :-1 s=THROTTLE_DONE pgs=2514 cs=0 l=1 rx=0x7f2c40008370 tx=0x7f2c40011b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:27.701 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:27.685+0000 7f2c46ffd700 1 -- 172.21.15.42:0/983556737 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2c40007360 con 0x7f2c38004420 2019-12-05T06:21:27.701 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:27.685+0000 7f2c55a75700 1 --2- 172.21.15.42:0/983556737 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2c38004420 0x7f2c38004850 secure :-1 s=THROTTLE_DONE pgs=2514 cs=0 l=1 rx=0x7f2c40008370 tx=0x7f2c40011b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:27.701 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:27.685+0000 7f2c55a75700 1 --2- 172.21.15.42:0/983556737 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2c38004420 0x7f2c38004850 secure :-1 s=THROTTLE_DONE pgs=2514 cs=0 l=1 rx=0x7f2c40008370 tx=0x7f2c40011b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:27.701 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:27.685+0000 7f2c46ffd700 1 -- 172.21.15.42:0/983556737 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c40011470 con 0x7f2c38004420 2019-12-05T06:21:27.701 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:27.685+0000 7f2c46ffd700 1 -- 172.21.15.42:0/983556737 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2c40005540 con 0x7f2c38004420 2019-12-05T06:21:27.701 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:27.685+0000 7f2c55a75700 1 --2- 172.21.15.42:0/983556737 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2c38004420 0x7f2c38004850 secure :-1 s=THROTTLE_DONE pgs=2514 cs=0 l=1 rx=0x7f2c40008370 tx=0x7f2c40011b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:27.701 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:27.685+0000 7f2c46ffd700 1 -- 172.21.15.42:0/983556737 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f2c40005740 con 0x7f2c38004420 2019-12-05T06:21:27.702 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:27.685+0000 7f2c55a75700 1 --2- 172.21.15.42:0/983556737 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2c38004420 0x7f2c38004850 secure :-1 s=THROTTLE_DONE pgs=2514 cs=0 l=1 rx=0x7f2c40008370 tx=0x7f2c40011b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:27.702 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:27.685+0000 7f2c46ffd700 1 --2- 172.21.15.42:0/983556737 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2c38043b90 0x7f2c380460a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:21:27.702 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:27.689+0000 7f2c46ffd700 1 -- 172.21.15.42:0/983556737 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 10589+0+0 (secure 0 0 0) 0x7f2c4004d4e0 con 0x7f2c38004420 2019-12-05T06:21:27.702 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:27.689+0000 7f2c56276700 1 --2- 172.21.15.42:0/983556737 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2c38043b90 0x7f2c380460a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:27.702 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:27.689+0000 7f2c56276700 1 --2- 172.21.15.42:0/983556737 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2c38043b90 0x7f2c380460a0 crc :-1 s=READY pgs=1049 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:21:27.703 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:27.693+0000 7f2c57cd9700 1 -- 172.21.15.42:0/983556737 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c2c003200 con 0x7f2c38004420 2019-12-05T06:21:27.703 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:27.697+0000 7f2c55a75700 1 --2- 172.21.15.42:0/983556737 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2c38004420 0x7f2c38004850 secure :-1 s=THROTTLE_DONE pgs=2514 cs=0 l=1 rx=0x7f2c40008370 tx=0x7f2c40011b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:27.703 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:27.697+0000 7f2c46ffd700 1 -- 172.21.15.42:0/983556737 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f2c40034540 con 0x7f2c38004420 2019-12-05T06:21:27.859 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:27.853+0000 7f2c57cd9700 1 -- 172.21.15.42:0/983556737 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2c2c003630 con 0x7f2c38004420 2019-12-05T06:21:27.860 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:27.853+0000 7f2c55a75700 1 --2- 172.21.15.42:0/983556737 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2c38004420 0x7f2c38004850 secure :-1 s=THROTTLE_DONE pgs=2514 cs=0 l=1 rx=0x7f2c40008370 tx=0x7f2c40011b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:27.860 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:27.853+0000 7f2c46ffd700 1 -- 172.21.15.42:0/983556737 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 298 v298) v1 ==== 95+0+2114 (secure 0 0 0) 0x7f2c400347a0 con 0x7f2c38004420 2019-12-05T06:21:27.860 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 298 2019-12-05T06:21:27.860 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:21:27.860 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":298,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":10710,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/526579407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":526579407},{"type":"v1","addr":"172.21.15.40:6833","nonce":526579407}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":298}],"filesystems":[{"mdsmap":{"epoch":297,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:19:55.115618+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":102,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10711},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10711":{"gid":10711,"name":"a","rank":0,"incarnation":294,"state":"up:active","state_seq":12,"addr":"172.21.15.42:6835/2638662429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2638662429},{"type":"v1","addr":"172.21.15.42:6835","nonce":2638662429}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:21:27.863 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:27.857+0000 7f2c57cd9700 1 -- 172.21.15.42:0/983556737 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2c38043b90 msgr2=0x7f2c380460a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:27.863 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:27.857+0000 7f2c57cd9700 1 --2- 172.21.15.42:0/983556737 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2c38043b90 0x7f2c380460a0 crc :-1 s=READY pgs=1049 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:27.864 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:27.857+0000 7f2c57cd9700 1 -- 172.21.15.42:0/983556737 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2c38004420 msgr2=0x7f2c38004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:27.864 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:27.857+0000 7f2c57cd9700 1 --2- 172.21.15.42:0/983556737 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2c38004420 0x7f2c38004850 secure :-1 s=READY pgs=2514 cs=0 l=1 rx=0x7f2c40008370 tx=0x7f2c40011b30).stop 2019-12-05T06:21:27.864 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:27.857+0000 7f2c56a77700 1 -- 172.21.15.42:0/983556737 reap_dead start 2019-12-05T06:21:27.864 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:27.857+0000 7f2c57cd9700 1 -- 172.21.15.42:0/983556737 shutdown_connections 2019-12-05T06:21:27.864 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:27.857+0000 7f2c57cd9700 1 -- 172.21.15.42:0/983556737 shutdown_connections 2019-12-05T06:21:27.864 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:27.857+0000 7f2c57cd9700 1 -- 172.21.15.42:0/983556737 wait complete. 2019-12-05T06:21:27.864 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:27.857+0000 7f2c57cd9700 1 -- 172.21.15.42:0/983556737 >> 172.21.15.42:0/983556737 conn(0x7f2c5008fdd0 msgr2=0x7f2c5007e5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:21:27.876 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:21:29.877 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:21:29.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:29.965+0000 7fd34471f700 1 Processor -- start 2019-12-05T06:21:29.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:29.965+0000 7fd34471f700 1 -- start start 2019-12-05T06:21:29.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:29.965+0000 7fd34471f700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd33c08df80 0x7fd33c09f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:21:29.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:29.965+0000 7fd34471f700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd33c09f830 0x7fd33c14efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:21:29.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:29.965+0000 7fd34471f700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd33c09fc60 con 0x7fd33c0a7740 2019-12-05T06:21:29.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:29.965+0000 7fd34471f700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd33c0a0b00 con 0x7fd33c08df80 2019-12-05T06:21:29.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:29.965+0000 7fd34471f700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd33c08e5b0 con 0x7fd33c09f830 2019-12-05T06:21:29.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:29.965+0000 7fd3424bb700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd33c08df80 0x7fd33c09f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:29.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:29.965+0000 7fd3424bb700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd33c08df80 0x7fd33c09f2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:53700/0 (socket says 172.21.15.42:53700) 2019-12-05T06:21:29.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:29.965+0000 7fd3424bb700 1 -- 172.21.15.42:0/2445485568 learned_addr learned my addr 172.21.15.42:0/2445485568 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:21:29.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:29.965+0000 7fd3434bd700 1 --2- 172.21.15.42:0/2445485568 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd33c09f830 0x7fd33c14efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:29.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:29.965+0000 7fd3424bb700 1 -- 172.21.15.42:0/2445485568 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd33c09f830 msgr2=0x7fd33c14efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:21:29.976 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:29.965+0000 7fd3424bb700 1 --2- 172.21.15.42:0/2445485568 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd33c09f830 0x7fd33c14efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:21:29.976 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:29.965+0000 7fd3424bb700 1 -- 172.21.15.42:0/2445485568 >> v1:172.21.15.42:6789/0 conn(0x7fd33c0a7740 legacy=0x7fd33c07e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:29.976 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:29.965+0000 7fd3424bb700 1 -- 172.21.15.42:0/2445485568 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd33c0a0710 con 0x7fd33c08df80 2019-12-05T06:21:29.976 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:29.965+0000 7fd3424bb700 1 --2- 172.21.15.42:0/2445485568 >> v2:172.21.15.42:3300/0 conn(0x7fd33c08df80 0x7fd33c09f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fd32c008790 tx=0x7fd32c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:29.976 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:29.965+0000 7fd3424bb700 1 --2- 172.21.15.42:0/2445485568 >> v2:172.21.15.42:3300/0 conn(0x7fd33c08df80 0x7fd33c09f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fd32c008790 tx=0x7fd32c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:29.976 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:29.965+0000 7fd3424bb700 1 --2- 172.21.15.42:0/2445485568 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd33c08df80 0x7fd33c09f2e0 secure :-1 s=READY pgs=2516 cs=0 l=1 rx=0x7fd32c008790 tx=0x7fd32c00fd30).ready entity=mon.1 client_cookie=80e608caf901807 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:21:29.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:29.965+0000 7fd3424bb700 1 --2- 172.21.15.42:0/2445485568 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd33c08df80 0x7fd33c09f2e0 secure :-1 s=THROTTLE_DONE pgs=2516 cs=0 l=1 rx=0x7fd32c008790 tx=0x7fd32c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:29.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:29.965+0000 7fd341cba700 1 -- 172.21.15.42:0/2445485568 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd32c008c90 con 0x7fd33c08df80 2019-12-05T06:21:29.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:29.965+0000 7fd3424bb700 1 --2- 172.21.15.42:0/2445485568 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd33c08df80 0x7fd33c09f2e0 secure :-1 s=THROTTLE_DONE pgs=2516 cs=0 l=1 rx=0x7fd32c008790 tx=0x7fd32c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:29.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:29.965+0000 7fd341cba700 1 -- 172.21.15.42:0/2445485568 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd32c0088b0 con 0x7fd33c08df80 2019-12-05T06:21:29.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:29.965+0000 7fd3424bb700 1 --2- 172.21.15.42:0/2445485568 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd33c08df80 0x7fd33c09f2e0 secure :-1 s=THROTTLE_DONE pgs=2516 cs=0 l=1 rx=0x7fd32c008790 tx=0x7fd32c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:29.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:29.965+0000 7fd341cba700 1 -- 172.21.15.42:0/2445485568 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd32c011b10 con 0x7fd33c08df80 2019-12-05T06:21:29.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:29.965+0000 7fd34471f700 1 -- 172.21.15.42:0/2445485568 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd33c08df80 msgr2=0x7fd33c09f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:29.978 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:29.965+0000 7fd34471f700 1 --2- 172.21.15.42:0/2445485568 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd33c08df80 0x7fd33c09f2e0 secure :-1 s=READY pgs=2516 cs=0 l=1 rx=0x7fd32c008790 tx=0x7fd32c00fd30).stop 2019-12-05T06:21:29.978 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:29.969+0000 7fd34471f700 1 -- 172.21.15.42:0/2445485568 shutdown_connections 2019-12-05T06:21:29.978 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:29.969+0000 7fd34471f700 1 --2- 172.21.15.42:0/2445485568 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd33c09f830 0x7fd33c14efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:21:29.978 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:29.969+0000 7fd34471f700 1 --2- 172.21.15.42:0/2445485568 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd33c08df80 0x7fd33c09f2e0 unknown :-1 s=CLOSED pgs=2516 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:29.978 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:29.969+0000 7fd34471f700 1 -- 172.21.15.42:0/2445485568 shutdown_connections 2019-12-05T06:21:29.978 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:29.969+0000 7fd34471f700 1 -- 172.21.15.42:0/2445485568 wait complete. 2019-12-05T06:21:29.978 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:29.969+0000 7fd34471f700 1 -- 172.21.15.42:0/2445485568 >> 172.21.15.42:0/2445485568 conn(0x7fd33c0aabe0 msgr2=0x7fd33c0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:21:29.978 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:29.969+0000 7fd34471f700 1 Processor -- start 2019-12-05T06:21:29.979 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:29.969+0000 7fd34471f700 1 -- start start 2019-12-05T06:21:29.981 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:29.969+0000 7fd34471f700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd33c0a1ff0 0x7fd33c1122c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:21:29.982 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:29.969+0000 7fd34471f700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd33c1132f0 0x7fd33c12efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:21:29.982 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:29.969+0000 7fd34471f700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd33c0a2a80 con 0x7fd33c0a6bb0 2019-12-05T06:21:29.982 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:29.969+0000 7fd34471f700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd33c0a2540 con 0x7fd33c0a1ff0 2019-12-05T06:21:29.982 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:29.969+0000 7fd34471f700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd33c0a27e0 con 0x7fd33c1132f0 2019-12-05T06:21:29.982 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:29.969+0000 7fd3434bd700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd33c1132f0 0x7fd33c12efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:29.982 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:29.969+0000 7fd3434bd700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd33c1132f0 0x7fd33c12efc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:53704/0 (socket says 172.21.15.42:53704) 2019-12-05T06:21:29.983 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:29.969+0000 7fd3434bd700 1 -- 172.21.15.42:0/2192007844 learned_addr learned my addr 172.21.15.42:0/2192007844 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:21:29.983 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:29.969+0000 7fd342cbc700 1 --2- 172.21.15.42:0/2192007844 >> v2:172.21.15.40:3300/0 conn(0x7fd33c0a1ff0 0x7fd33c1122c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:29.983 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:29.969+0000 7fd3434bd700 1 -- 172.21.15.42:0/2192007844 >> v2:172.21.15.40:3300/0 conn(0x7fd33c0a1ff0 msgr2=0x7fd33c1122c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:29.983 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:29.969+0000 7fd3434bd700 1 --2- 172.21.15.42:0/2192007844 >> v2:172.21.15.40:3300/0 conn(0x7fd33c0a1ff0 0x7fd33c1122c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:29.983 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:29.969+0000 7fd3434bd700 1 -- 172.21.15.42:0/2192007844 >> v1:172.21.15.42:6789/0 conn(0x7fd33c0a6bb0 legacy=0x7fd33c112af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:29.983 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:29.969+0000 7fd3434bd700 1 -- 172.21.15.42:0/2192007844 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd33c0a6fe0 con 0x7fd33c1132f0 2019-12-05T06:21:29.983 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:29.969+0000 7fd3434bd700 1 --2- 172.21.15.42:0/2192007844 >> v2:172.21.15.42:3300/0 conn(0x7fd33c1132f0 0x7fd33c12efc0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fd3340032b0 tx=0x7fd334010c40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:29.984 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:29.969+0000 7fd3434bd700 1 --2- 172.21.15.42:0/2192007844 >> v2:172.21.15.42:3300/0 conn(0x7fd33c1132f0 0x7fd33c12efc0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fd3340032b0 tx=0x7fd334010c40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:29.984 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:29.969+0000 7fd3434bd700 1 --2- 172.21.15.42:0/2192007844 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd33c1132f0 0x7fd33c12efc0 secure :-1 s=READY pgs=2518 cs=0 l=1 rx=0x7fd3340032b0 tx=0x7fd334010c40).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:21:29.984 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:29.969+0000 7fd3434bd700 1 --2- 172.21.15.42:0/2192007844 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd33c1132f0 0x7fd33c12efc0 secure :-1 s=THROTTLE_DONE pgs=2518 cs=0 l=1 rx=0x7fd3340032b0 tx=0x7fd334010c40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:29.984 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:29.969+0000 7fd3434bd700 1 --2- 172.21.15.42:0/2192007844 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd33c1132f0 0x7fd33c12efc0 secure :-1 s=THROTTLE_DONE pgs=2518 cs=0 l=1 rx=0x7fd3340032b0 tx=0x7fd334010c40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:29.984 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:29.969+0000 7fd3337fe700 1 -- 172.21.15.42:0/2192007844 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd334007350 con 0x7fd33c1132f0 2019-12-05T06:21:29.984 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:29.969+0000 7fd3434bd700 1 --2- 172.21.15.42:0/2192007844 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd33c1132f0 0x7fd33c12efc0 secure :-1 s=THROTTLE_DONE pgs=2518 cs=0 l=1 rx=0x7fd3340032b0 tx=0x7fd334010c40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:29.984 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:29.969+0000 7fd3337fe700 1 -- 172.21.15.42:0/2192007844 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd334005cb0 con 0x7fd33c1132f0 2019-12-05T06:21:29.985 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:29.973+0000 7fd3337fe700 1 -- 172.21.15.42:0/2192007844 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd3340124f0 con 0x7fd33c1132f0 2019-12-05T06:21:29.985 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:29.973+0000 7fd34471f700 1 -- 172.21.15.42:0/2192007844 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd33c14eba0 con 0x7fd33c1132f0 2019-12-05T06:21:29.985 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:29.973+0000 7fd34471f700 1 -- 172.21.15.42:0/2192007844 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd33c12f6b0 con 0x7fd33c1132f0 2019-12-05T06:21:29.985 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:29.973+0000 7fd3434bd700 1 --2- 172.21.15.42:0/2192007844 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd33c1132f0 0x7fd33c12efc0 secure :-1 s=THROTTLE_DONE pgs=2518 cs=0 l=1 rx=0x7fd3340032b0 tx=0x7fd334010c40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:29.985 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:29.973+0000 7fd3337fe700 1 -- 172.21.15.42:0/2192007844 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fd334012800 con 0x7fd33c1132f0 2019-12-05T06:21:29.985 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:29.973+0000 7fd3434bd700 1 --2- 172.21.15.42:0/2192007844 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd33c1132f0 0x7fd33c12efc0 secure :-1 s=THROTTLE_DONE pgs=2518 cs=0 l=1 rx=0x7fd3340032b0 tx=0x7fd334010c40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:29.985 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:29.973+0000 7fd3337fe700 1 --2- 172.21.15.42:0/2192007844 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd32403ae70 0x7fd32403d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:21:29.985 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:29.973+0000 7fd3337fe700 1 -- 172.21.15.42:0/2192007844 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 10589+0+0 (secure 0 0 0) 0x7fd33404b360 con 0x7fd33c1132f0 2019-12-05T06:21:29.986 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:29.973+0000 7fd342cbc700 1 --2- 172.21.15.42:0/2192007844 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd32403ae70 0x7fd32403d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:29.986 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:29.973+0000 7fd342cbc700 1 --2- 172.21.15.42:0/2192007844 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd32403ae70 0x7fd32403d380 crc :-1 s=READY pgs=1050 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:21:29.986 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:29.981+0000 7fd34471f700 1 -- 172.21.15.42:0/2192007844 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd33c05d110 con 0x7fd33c1132f0 2019-12-05T06:21:29.991 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:29.985+0000 7fd3434bd700 1 --2- 172.21.15.42:0/2192007844 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd33c1132f0 0x7fd33c12efc0 secure :-1 s=THROTTLE_DONE pgs=2518 cs=0 l=1 rx=0x7fd3340032b0 tx=0x7fd334010c40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:29.992 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:29.985+0000 7fd3337fe700 1 -- 172.21.15.42:0/2192007844 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fd3340323e0 con 0x7fd33c1132f0 2019-12-05T06:21:30.151 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:30.145+0000 7fd34471f700 1 -- 172.21.15.42:0/2192007844 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd33c0a43d0 con 0x7fd33c1132f0 2019-12-05T06:21:30.152 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:30.145+0000 7fd3434bd700 1 --2- 172.21.15.42:0/2192007844 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd33c1132f0 0x7fd33c12efc0 secure :-1 s=THROTTLE_DONE pgs=2518 cs=0 l=1 rx=0x7fd3340032b0 tx=0x7fd334010c40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:30.152 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:30.145+0000 7fd3337fe700 1 -- 172.21.15.42:0/2192007844 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 298 v298) v1 ==== 95+0+2114 (secure 0 0 0) 0x7fd334032640 con 0x7fd33c1132f0 2019-12-05T06:21:30.153 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 298 2019-12-05T06:21:30.153 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:21:30.153 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":298,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":10710,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/526579407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":526579407},{"type":"v1","addr":"172.21.15.40:6833","nonce":526579407}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":298}],"filesystems":[{"mdsmap":{"epoch":297,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:19:55.115618+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":102,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10711},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10711":{"gid":10711,"name":"a","rank":0,"incarnation":294,"state":"up:active","state_seq":12,"addr":"172.21.15.42:6835/2638662429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2638662429},{"type":"v1","addr":"172.21.15.42:6835","nonce":2638662429}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:21:30.155 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:30.149+0000 7fd34471f700 1 -- 172.21.15.42:0/2192007844 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd32403ae70 msgr2=0x7fd32403d380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:30.155 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:30.149+0000 7fd34471f700 1 --2- 172.21.15.42:0/2192007844 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd32403ae70 0x7fd32403d380 crc :-1 s=READY pgs=1050 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:30.155 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:30.149+0000 7fd34471f700 1 -- 172.21.15.42:0/2192007844 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd33c1132f0 msgr2=0x7fd33c12efc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:30.155 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:30.149+0000 7fd34471f700 1 --2- 172.21.15.42:0/2192007844 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd33c1132f0 0x7fd33c12efc0 secure :-1 s=READY pgs=2518 cs=0 l=1 rx=0x7fd3340032b0 tx=0x7fd334010c40).stop 2019-12-05T06:21:30.156 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:30.149+0000 7fd34471f700 1 -- 172.21.15.42:0/2192007844 shutdown_connections 2019-12-05T06:21:30.156 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:30.149+0000 7fd34471f700 1 --2- 172.21.15.42:0/2192007844 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd32403ae70 0x7fd32403d380 unknown :-1 s=CLOSED pgs=1050 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:30.156 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:30.149+0000 7fd34471f700 1 --2- 172.21.15.42:0/2192007844 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd33c1132f0 0x7fd33c12efc0 unknown :-1 s=CLOSED pgs=2518 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:30.157 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:30.149+0000 7fd34471f700 1 --2- 172.21.15.42:0/2192007844 >> v2:172.21.15.40:3300/0 conn(0x7fd33c0a1ff0 0x7fd33c1122c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:30.157 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:30.149+0000 7fd34471f700 1 -- 172.21.15.42:0/2192007844 shutdown_connections 2019-12-05T06:21:30.158 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:30.149+0000 7fd34471f700 1 -- 172.21.15.42:0/2192007844 wait complete. 2019-12-05T06:21:30.158 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:30.149+0000 7fd34471f700 1 -- 172.21.15.42:0/2192007844 >> 172.21.15.42:0/2192007844 conn(0x7fd33c0a9770 msgr2=0x7fd33c07cf70 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:21:30.164 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:21:32.165 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:21:32.263 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:32.253+0000 7f6dbc8b2700 1 Processor -- start 2019-12-05T06:21:32.263 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:32.253+0000 7f6dbc8b2700 1 -- start start 2019-12-05T06:21:32.264 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:32.253+0000 7f6dbc8b2700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f6db80a7740 0x7f6db808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:21:32.264 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:32.253+0000 7f6dbc8b2700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6db80a0710 con 0x7f6db807e720 2019-12-05T06:21:32.264 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:32.253+0000 7f6dbc8b2700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6db80a9ba0 con 0x7f6db809fae0 2019-12-05T06:21:32.264 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:32.253+0000 7f6dbc8b2700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6db80a0b00 con 0x7f6db80a7740 2019-12-05T06:21:32.264 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:32.253+0000 7f6db759e700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f6db809fae0 0x7f6db814efe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:53756/0 (socket says 172.21.15.42:53756) 2019-12-05T06:21:32.264 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:32.253+0000 7f6db759e700 1 -- 172.21.15.42:0/3634350439 learned_addr learned my addr 172.21.15.42:0/3634350439 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:21:32.264 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:32.253+0000 7f6db6d9d700 1 --2- 172.21.15.42:0/3634350439 >> v2:172.21.15.40:3300/0 conn(0x7f6db80a7740 0x7f6db808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:32.265 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:32.253+0000 7f6db5d9b700 1 -- 172.21.15.42:0/3634350439 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1388800174 0 0) 0x7f6db80a9ba0 con 0x7f6db809fae0 2019-12-05T06:21:32.265 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:32.253+0000 7f6db5d9b700 1 -- 172.21.15.42:0/3634350439 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6da00030b0 con 0x7f6db809fae0 2019-12-05T06:21:32.265 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:32.257+0000 7f6db5d9b700 1 -- 172.21.15.42:0/3634350439 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2760320179 0 0) 0x7f6da00030b0 con 0x7f6db809fae0 2019-12-05T06:21:32.265 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:32.257+0000 7f6db5d9b700 1 -- 172.21.15.42:0/3634350439 >> v2:172.21.15.40:3300/0 conn(0x7f6db80a7740 msgr2=0x7f6db808df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:21:32.266 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:32.257+0000 7f6db5d9b700 1 --2- 172.21.15.42:0/3634350439 >> v2:172.21.15.40:3300/0 conn(0x7f6db80a7740 0x7f6db808df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:21:32.266 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:32.257+0000 7f6db5d9b700 1 -- 172.21.15.42:0/3634350439 >> v1:172.21.15.40:6789/0 conn(0x7f6db807e720 legacy=0x7f6db809f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:32.266 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:32.257+0000 7f6db5d9b700 1 -- 172.21.15.42:0/3634350439 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6db807eb50 con 0x7f6db809fae0 2019-12-05T06:21:32.266 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:32.257+0000 7f6db5d9b700 1 -- 172.21.15.42:0/3634350439 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7f6db0001ad0 con 0x7f6db809fae0 2019-12-05T06:21:32.266 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:32.257+0000 7f6db5d9b700 1 -- 172.21.15.42:0/3634350439 >> v1:172.21.15.42:6789/0 conn(0x7f6db809fae0 legacy=0x7f6db814efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:32.267 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:32.257+0000 7f6db5d9b700 1 --2- 172.21.15.42:0/3634350439 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6da00036e0 0x7f6da0003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:21:32.267 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:32.257+0000 7f6db5d9b700 1 --2- 172.21.15.42:0/3634350439 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6da0004420 0x7f6da0004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:21:32.267 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:32.257+0000 7f6db5d9b700 1 --2- 172.21.15.42:0/3634350439 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6da0004df0 0x7f6da000b670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:21:32.267 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:32.257+0000 7f6db5d9b700 1 -- 172.21.15.42:0/3634350439 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6db80a0b00 con 0x7f6da0004df0 2019-12-05T06:21:32.268 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:32.257+0000 7f6db6d9d700 1 --2- 172.21.15.42:0/3634350439 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6da00036e0 0x7f6da0003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:32.268 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:32.257+0000 7f6db5d9b700 1 -- 172.21.15.42:0/3634350439 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6da004fce0 con 0x7f6da00036e0 2019-12-05T06:21:32.268 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:32.257+0000 7f6db659c700 1 --2- 172.21.15.42:0/3634350439 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6da0004420 0x7f6da0004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:32.268 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:32.257+0000 7f6db5d9b700 1 -- 172.21.15.42:0/3634350439 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6da004ff20 con 0x7f6da0004420 2019-12-05T06:21:32.269 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:32.257+0000 7f6db759e700 1 --2- 172.21.15.42:0/3634350439 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6da0004df0 0x7f6da000b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:32.269 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:32.257+0000 7f6db6d9d700 1 -- 172.21.15.42:0/3634350439 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6da0004420 msgr2=0x7f6da0004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:21:32.269 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:32.257+0000 7f6db6d9d700 1 --2- 172.21.15.42:0/3634350439 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6da0004420 0x7f6da0004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:21:32.269 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:32.257+0000 7f6db6d9d700 1 -- 172.21.15.42:0/3634350439 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6da0004df0 msgr2=0x7f6da000b670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:21:32.269 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:32.257+0000 7f6db6d9d700 1 --2- 172.21.15.42:0/3634350439 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6da0004df0 0x7f6da000b670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:21:32.269 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:32.257+0000 7f6db759e700 1 -- 172.21.15.42:0/3634350439 reap_dead start 2019-12-05T06:21:32.270 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:32.257+0000 7f6db6d9d700 1 -- 172.21.15.42:0/3634350439 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6da0050220 con 0x7f6da00036e0 2019-12-05T06:21:32.270 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:32.257+0000 7f6db6d9d700 1 --2- 172.21.15.42:0/3634350439 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6da00036e0 0x7f6da0003b50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f6da8002f30 tx=0x7f6da800ffe0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:32.270 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:32.257+0000 7f6db6d9d700 1 --2- 172.21.15.42:0/3634350439 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6da00036e0 0x7f6da0003b50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f6da8002f30 tx=0x7f6da800ffe0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:32.270 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:32.257+0000 7f6db6d9d700 1 --2- 172.21.15.42:0/3634350439 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6da00036e0 0x7f6da0003b50 secure :-1 s=READY pgs=2521 cs=0 l=1 rx=0x7f6da8002f30 tx=0x7f6da800ffe0).ready entity=mon.1 client_cookie=44cc789504d0673d server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:21:32.270 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:32.257+0000 7f6db6d9d700 1 --2- 172.21.15.42:0/3634350439 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6da00036e0 0x7f6da0003b50 secure :-1 s=THROTTLE_DONE pgs=2521 cs=0 l=1 rx=0x7f6da8002f30 tx=0x7f6da800ffe0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:32.270 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:32.257+0000 7f6db5d9b700 1 -- 172.21.15.42:0/3634350439 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6da8010d10 con 0x7f6da00036e0 2019-12-05T06:21:32.270 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:32.257+0000 7f6db6d9d700 1 --2- 172.21.15.42:0/3634350439 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6da00036e0 0x7f6da0003b50 secure :-1 s=THROTTLE_DONE pgs=2521 cs=0 l=1 rx=0x7f6da8002f30 tx=0x7f6da800ffe0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:32.271 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:32.257+0000 7f6db5d9b700 1 -- 172.21.15.42:0/3634350439 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6da8010930 con 0x7f6da00036e0 2019-12-05T06:21:32.271 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:32.257+0000 7f6db6d9d700 1 --2- 172.21.15.42:0/3634350439 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6da00036e0 0x7f6da0003b50 secure :-1 s=THROTTLE_DONE pgs=2521 cs=0 l=1 rx=0x7f6da8002f30 tx=0x7f6da800ffe0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:32.271 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:32.257+0000 7f6db5d9b700 1 -- 172.21.15.42:0/3634350439 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6da8008bb0 con 0x7f6da00036e0 2019-12-05T06:21:32.271 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:32.257+0000 7f6dbc8b2700 1 -- 172.21.15.42:0/3634350439 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6da00036e0 msgr2=0x7f6da0003b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:32.271 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:32.257+0000 7f6dbc8b2700 1 --2- 172.21.15.42:0/3634350439 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6da00036e0 0x7f6da0003b50 secure :-1 s=READY pgs=2521 cs=0 l=1 rx=0x7f6da8002f30 tx=0x7f6da800ffe0).stop 2019-12-05T06:21:32.271 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:32.257+0000 7f6dbc8b2700 1 -- 172.21.15.42:0/3634350439 shutdown_connections 2019-12-05T06:21:32.271 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:32.257+0000 7f6dbc8b2700 1 --2- 172.21.15.42:0/3634350439 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6da00036e0 0x7f6da0003b50 unknown :-1 s=CLOSED pgs=2521 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:32.272 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:32.257+0000 7f6dbc8b2700 1 -- 172.21.15.42:0/3634350439 shutdown_connections 2019-12-05T06:21:32.272 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:32.257+0000 7f6dbc8b2700 1 -- 172.21.15.42:0/3634350439 wait complete. 2019-12-05T06:21:32.272 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:32.257+0000 7f6dbc8b2700 1 -- 172.21.15.42:0/3634350439 >> 172.21.15.42:0/3634350439 conn(0x7f6db80aabe0 msgr2=0x7f6db80a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:21:32.275 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:32.261+0000 7f6dbc8b2700 1 Processor -- start 2019-12-05T06:21:32.276 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:32.261+0000 7f6dbc8b2700 1 -- start start 2019-12-05T06:21:32.276 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:32.261+0000 7f6dbc8b2700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6db8124360 0x7f6db8131020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:21:32.276 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:32.261+0000 7f6dbc8b2700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6db81304c0 con 0x7f6db812cf00 2019-12-05T06:21:32.276 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:32.261+0000 7f6dbc8b2700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6db812d330 con 0x7f6db80a5bd0 2019-12-05T06:21:32.276 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:32.261+0000 7f6dbc8b2700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6db812d560 con 0x7f6db8124360 2019-12-05T06:21:32.277 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:32.261+0000 7f6db6d9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6db8124360 0x7f6db8131020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:32.277 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:32.261+0000 7f6db759e700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f6db80a5bd0 0x7f6db80a6000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:53766/0 (socket says 172.21.15.42:53766) 2019-12-05T06:21:32.277 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:32.261+0000 7f6db759e700 1 -- 172.21.15.42:0/302834287 learned_addr learned my addr 172.21.15.42:0/302834287 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:21:32.277 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:32.261+0000 7f6da77fe700 1 -- 172.21.15.42:0/302834287 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4149299247 0 0) 0x7f6db812d330 con 0x7f6db80a5bd0 2019-12-05T06:21:32.277 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:32.261+0000 7f6da77fe700 1 -- 172.21.15.42:0/302834287 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d9c0030b0 con 0x7f6db80a5bd0 2019-12-05T06:21:32.277 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:32.261+0000 7f6db6d9d700 1 -- 172.21.15.42:0/302834287 >> v1:172.21.15.42:6789/0 conn(0x7f6db80a5bd0 legacy=0x7f6db80a6000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:32.278 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:32.261+0000 7f6db6d9d700 1 -- 172.21.15.42:0/302834287 >> v1:172.21.15.40:6789/0 conn(0x7f6db812cf00 legacy=0x7f6db8117bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:32.278 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:32.261+0000 7f6db6d9d700 1 -- 172.21.15.42:0/302834287 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6db813a940 con 0x7f6db8124360 2019-12-05T06:21:32.278 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:32.261+0000 7f6db6d9d700 1 --2- 172.21.15.42:0/302834287 >> v2:172.21.15.42:3300/0 conn(0x7f6db8124360 0x7f6db8131020 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f6da8002f30 tx=0x7f6da8008ea0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:32.278 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:32.261+0000 7f6db6d9d700 1 --2- 172.21.15.42:0/302834287 >> v2:172.21.15.42:3300/0 conn(0x7f6db8124360 0x7f6db8131020 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f6da8002f30 tx=0x7f6da8008ea0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:32.278 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:32.261+0000 7f6db6d9d700 1 --2- 172.21.15.42:0/302834287 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6db8124360 0x7f6db8131020 secure :-1 s=READY pgs=2523 cs=0 l=1 rx=0x7f6da8002f30 tx=0x7f6da8008ea0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:21:32.279 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:32.261+0000 7f6db6d9d700 1 --2- 172.21.15.42:0/302834287 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6db8124360 0x7f6db8131020 secure :-1 s=THROTTLE_DONE pgs=2523 cs=0 l=1 rx=0x7f6da8002f30 tx=0x7f6da8008ea0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:32.279 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:32.261+0000 7f6db6d9d700 1 --2- 172.21.15.42:0/302834287 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6db8124360 0x7f6db8131020 secure :-1 s=THROTTLE_DONE pgs=2523 cs=0 l=1 rx=0x7f6da8002f30 tx=0x7f6da8008ea0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:32.279 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:32.261+0000 7f6db6d9d700 1 --2- 172.21.15.42:0/302834287 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6db8124360 0x7f6db8131020 secure :-1 s=THROTTLE_DONE pgs=2523 cs=0 l=1 rx=0x7f6da8002f30 tx=0x7f6da8008ea0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:32.279 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:32.261+0000 7f6da77fe700 1 -- 172.21.15.42:0/302834287 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6da8010d30 con 0x7f6db8124360 2019-12-05T06:21:32.280 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:32.261+0000 7f6da77fe700 1 -- 172.21.15.42:0/302834287 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6da80087d0 con 0x7f6db8124360 2019-12-05T06:21:32.280 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:32.261+0000 7f6dbc8b2700 1 -- 172.21.15.42:0/302834287 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6db8118500 con 0x7f6db8124360 2019-12-05T06:21:32.280 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:32.261+0000 7f6da77fe700 1 -- 172.21.15.42:0/302834287 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6da800f490 con 0x7f6db8124360 2019-12-05T06:21:32.280 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:32.261+0000 7f6dbc8b2700 1 -- 172.21.15.42:0/302834287 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6db8118a50 con 0x7f6db8124360 2019-12-05T06:21:32.281 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:32.261+0000 7f6db6d9d700 1 --2- 172.21.15.42:0/302834287 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6db8124360 0x7f6db8131020 secure :-1 s=THROTTLE_DONE pgs=2523 cs=0 l=1 rx=0x7f6da8002f30 tx=0x7f6da8008ea0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:32.281 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:32.265+0000 7f6da77fe700 1 -- 172.21.15.42:0/302834287 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f6da800f760 con 0x7f6db8124360 2019-12-05T06:21:32.281 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:32.265+0000 7f6db6d9d700 1 --2- 172.21.15.42:0/302834287 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6db8124360 0x7f6db8131020 secure :-1 s=THROTTLE_DONE pgs=2523 cs=0 l=1 rx=0x7f6da8002f30 tx=0x7f6da8008ea0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:32.281 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:32.265+0000 7f6da77fe700 1 --2- 172.21.15.42:0/302834287 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6d9c03b220 0x7f6d9c03d730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:21:32.281 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:32.265+0000 7f6da77fe700 1 -- 172.21.15.42:0/302834287 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 10589+0+0 (secure 0 0 0) 0x7f6da804d2f0 con 0x7f6db8124360 2019-12-05T06:21:32.282 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:32.265+0000 7f6db659c700 1 --2- 172.21.15.42:0/302834287 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6d9c03b220 0x7f6d9c03d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:32.282 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:32.265+0000 7f6db659c700 1 --2- 172.21.15.42:0/302834287 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6d9c03b220 0x7f6d9c03d730 crc :-1 s=READY pgs=1051 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:21:32.282 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:32.269+0000 7f6dbc8b2700 1 -- 172.21.15.42:0/302834287 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6db805d110 con 0x7f6db8124360 2019-12-05T06:21:32.282 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:32.273+0000 7f6db6d9d700 1 --2- 172.21.15.42:0/302834287 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6db8124360 0x7f6db8131020 secure :-1 s=THROTTLE_DONE pgs=2523 cs=0 l=1 rx=0x7f6da8002f30 tx=0x7f6da8008ea0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:32.283 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:32.273+0000 7f6da77fe700 1 -- 172.21.15.42:0/302834287 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f6da8035180 con 0x7f6db8124360 2019-12-05T06:21:32.436 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:32.433+0000 7f6dbc8b2700 1 -- 172.21.15.42:0/302834287 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6db805b8c0 con 0x7f6db8124360 2019-12-05T06:21:32.437 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:32.433+0000 7f6db6d9d700 1 --2- 172.21.15.42:0/302834287 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6db8124360 0x7f6db8131020 secure :-1 s=THROTTLE_DONE pgs=2523 cs=0 l=1 rx=0x7f6da8002f30 tx=0x7f6da8008ea0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:32.438 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:32.433+0000 7f6da77fe700 1 -- 172.21.15.42:0/302834287 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 298 v298) v1 ==== 95+0+2114 (secure 0 0 0) 0x7f6da8017ba0 con 0x7f6db8124360 2019-12-05T06:21:32.438 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 298 2019-12-05T06:21:32.438 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:21:32.438 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":298,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":10710,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/526579407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":526579407},{"type":"v1","addr":"172.21.15.40:6833","nonce":526579407}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":298}],"filesystems":[{"mdsmap":{"epoch":297,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:19:55.115618+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":102,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10711},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10711":{"gid":10711,"name":"a","rank":0,"incarnation":294,"state":"up:active","state_seq":12,"addr":"172.21.15.42:6835/2638662429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2638662429},{"type":"v1","addr":"172.21.15.42:6835","nonce":2638662429}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:21:32.441 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:32.433+0000 7f6dbc8b2700 1 -- 172.21.15.42:0/302834287 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6d9c03b220 msgr2=0x7f6d9c03d730 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:32.441 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:32.433+0000 7f6dbc8b2700 1 --2- 172.21.15.42:0/302834287 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6d9c03b220 0x7f6d9c03d730 crc :-1 s=READY pgs=1051 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:32.441 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:32.433+0000 7f6dbc8b2700 1 -- 172.21.15.42:0/302834287 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6db8124360 msgr2=0x7f6db8131020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:32.442 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:32.433+0000 7f6dbc8b2700 1 --2- 172.21.15.42:0/302834287 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6db8124360 0x7f6db8131020 secure :-1 s=READY pgs=2523 cs=0 l=1 rx=0x7f6da8002f30 tx=0x7f6da8008ea0).stop 2019-12-05T06:21:32.442 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:32.433+0000 7f6dbc8b2700 1 -- 172.21.15.42:0/302834287 shutdown_connections 2019-12-05T06:21:32.442 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:32.433+0000 7f6dbc8b2700 1 --2- 172.21.15.42:0/302834287 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6d9c03b220 0x7f6d9c03d730 unknown :-1 s=CLOSED pgs=1051 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:32.442 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:32.433+0000 7f6dbc8b2700 1 --2- 172.21.15.42:0/302834287 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6db8124360 0x7f6db8131020 unknown :-1 s=CLOSED pgs=2523 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:32.443 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:32.433+0000 7f6dbc8b2700 1 -- 172.21.15.42:0/302834287 shutdown_connections 2019-12-05T06:21:32.443 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:32.433+0000 7f6dbc8b2700 1 -- 172.21.15.42:0/302834287 wait complete. 2019-12-05T06:21:32.443 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:32.433+0000 7f6dbc8b2700 1 -- 172.21.15.42:0/302834287 >> 172.21.15.42:0/302834287 conn(0x7f6db80a82a0 msgr2=0x7f6db8111db0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:21:32.449 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:21:34.450 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:21:34.549 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:34.537+0000 7fbf26045700 1 Processor -- start 2019-12-05T06:21:34.551 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:34.537+0000 7fbf26045700 1 -- start start 2019-12-05T06:21:34.552 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:34.537+0000 7fbf26045700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fbf2008df80 0x7fbf2009f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:21:34.552 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:34.537+0000 7fbf26045700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbf2009f830 0x7fbf2014efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:21:34.552 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:34.537+0000 7fbf26045700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf2009fc60 con 0x7fbf200a7740 2019-12-05T06:21:34.552 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:34.537+0000 7fbf26045700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf200a0b00 con 0x7fbf2008df80 2019-12-05T06:21:34.552 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:34.537+0000 7fbf26045700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf2008e5b0 con 0x7fbf2009f830 2019-12-05T06:21:34.553 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:34.537+0000 7fbf1ffff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fbf200a7740 0x7fbf2007e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:53774/0 (socket says 172.21.15.42:53774) 2019-12-05T06:21:34.553 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:34.537+0000 7fbf1ffff700 1 -- 172.21.15.42:0/731836936 learned_addr learned my addr 172.21.15.42:0/731836936 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:21:34.553 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:34.537+0000 7fbf17fff700 1 --2- 172.21.15.42:0/731836936 >> v2:172.21.15.40:3300/0 conn(0x7fbf2008df80 0x7fbf2009f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:34.553 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:34.537+0000 7fbf24de3700 1 --2- 172.21.15.42:0/731836936 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbf2009f830 0x7fbf2014efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:34.553 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:34.541+0000 7fbf1f7fe700 1 -- 172.21.15.42:0/731836936 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1431288975 0 0) 0x7fbf2009fc60 con 0x7fbf200a7740 2019-12-05T06:21:34.554 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:34.541+0000 7fbf1f7fe700 1 -- 172.21.15.42:0/731836936 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf0c0030b0 con 0x7fbf200a7740 2019-12-05T06:21:34.554 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:34.541+0000 7fbf1f7fe700 1 -- 172.21.15.42:0/731836936 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4021343646 0 0) 0x7fbf0c0030b0 con 0x7fbf200a7740 2019-12-05T06:21:34.554 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:34.541+0000 7fbf1f7fe700 1 -- 172.21.15.42:0/731836936 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbf2009f830 msgr2=0x7fbf2014efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:21:34.554 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:34.541+0000 7fbf1f7fe700 1 --2- 172.21.15.42:0/731836936 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbf2009f830 0x7fbf2014efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:21:34.554 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:34.541+0000 7fbf1f7fe700 1 -- 172.21.15.42:0/731836936 >> v2:172.21.15.40:3300/0 conn(0x7fbf2008df80 msgr2=0x7fbf2009f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:21:34.554 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:34.541+0000 7fbf1f7fe700 1 --2- 172.21.15.42:0/731836936 >> v2:172.21.15.40:3300/0 conn(0x7fbf2008df80 0x7fbf2009f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:21:34.555 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:34.541+0000 7fbf1f7fe700 1 -- 172.21.15.42:0/731836936 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf200a0710 con 0x7fbf200a7740 2019-12-05T06:21:34.555 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:34.541+0000 7fbf1f7fe700 1 -- 172.21.15.42:0/731836936 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7fbf10001750 con 0x7fbf200a7740 2019-12-05T06:21:34.555 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:34.541+0000 7fbf1f7fe700 1 -- 172.21.15.42:0/731836936 >> v1:172.21.15.42:6789/0 conn(0x7fbf200a7740 legacy=0x7fbf2007e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:34.555 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:34.541+0000 7fbf1f7fe700 1 --2- 172.21.15.42:0/731836936 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbf0c0036e0 0x7fbf0c003b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:21:34.555 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:34.541+0000 7fbf1f7fe700 1 --2- 172.21.15.42:0/731836936 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbf0c004420 0x7fbf0c004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:21:34.555 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:34.541+0000 7fbf1f7fe700 1 --2- 172.21.15.42:0/731836936 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbf0c004da0 0x7fbf0c04f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:21:34.555 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:34.541+0000 7fbf1f7fe700 1 -- 172.21.15.42:0/731836936 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf200a0b00 con 0x7fbf0c0036e0 2019-12-05T06:21:34.556 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:34.541+0000 7fbf1f7fe700 1 -- 172.21.15.42:0/731836936 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf2008e5b0 con 0x7fbf0c004da0 2019-12-05T06:21:34.556 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:34.541+0000 7fbf1f7fe700 1 -- 172.21.15.42:0/731836936 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf0c004160 con 0x7fbf0c004420 2019-12-05T06:21:34.556 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:34.541+0000 7fbf24de3700 1 --2- 172.21.15.42:0/731836936 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbf0c004da0 0x7fbf0c04f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:34.556 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:34.541+0000 7fbf17fff700 1 --2- 172.21.15.42:0/731836936 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbf0c004420 0x7fbf0c004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:34.556 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:34.541+0000 7fbf1ffff700 1 --2- 172.21.15.42:0/731836936 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbf0c0036e0 0x7fbf0c003b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:34.557 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:34.541+0000 7fbf24de3700 1 -- 172.21.15.42:0/731836936 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbf0c004420 msgr2=0x7fbf0c004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:21:34.557 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:34.541+0000 7fbf24de3700 1 --2- 172.21.15.42:0/731836936 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbf0c004420 0x7fbf0c004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:21:34.557 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:34.541+0000 7fbf24de3700 1 -- 172.21.15.42:0/731836936 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbf0c0036e0 msgr2=0x7fbf0c003b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:21:34.557 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:34.541+0000 7fbf24de3700 1 --2- 172.21.15.42:0/731836936 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbf0c0036e0 0x7fbf0c003b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:21:34.557 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:34.541+0000 7fbf24de3700 1 -- 172.21.15.42:0/731836936 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf0c04fc90 con 0x7fbf0c004da0 2019-12-05T06:21:34.557 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:34.541+0000 7fbf24de3700 1 --2- 172.21.15.42:0/731836936 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbf0c004da0 0x7fbf0c04f680 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fbf180032b0 tx=0x7fbf18010320).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:34.558 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:34.541+0000 7fbf24de3700 1 --2- 172.21.15.42:0/731836936 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbf0c004da0 0x7fbf0c04f680 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fbf180032b0 tx=0x7fbf18010320).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:34.558 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:34.541+0000 7fbf24de3700 1 --2- 172.21.15.42:0/731836936 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbf0c004da0 0x7fbf0c04f680 secure :-1 s=READY pgs=2525 cs=0 l=1 rx=0x7fbf180032b0 tx=0x7fbf18010320).ready entity=mon.1 client_cookie=f54f9e9a27307447 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:21:34.558 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:34.541+0000 7fbf24de3700 1 --2- 172.21.15.42:0/731836936 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbf0c004da0 0x7fbf0c04f680 secure :-1 s=THROTTLE_DONE pgs=2525 cs=0 l=1 rx=0x7fbf180032b0 tx=0x7fbf18010320).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:34.558 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:34.541+0000 7fbf24de3700 1 --2- 172.21.15.42:0/731836936 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbf0c004da0 0x7fbf0c04f680 secure :-1 s=THROTTLE_DONE pgs=2525 cs=0 l=1 rx=0x7fbf180032b0 tx=0x7fbf18010320).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:34.559 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:34.541+0000 7fbf1f7fe700 1 -- 172.21.15.42:0/731836936 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbf18008d40 con 0x7fbf0c004da0 2019-12-05T06:21:34.559 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:34.541+0000 7fbf24de3700 1 --2- 172.21.15.42:0/731836936 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbf0c004da0 0x7fbf0c04f680 secure :-1 s=THROTTLE_DONE pgs=2525 cs=0 l=1 rx=0x7fbf180032b0 tx=0x7fbf18010320).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:34.559 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:34.541+0000 7fbf1f7fe700 1 -- 172.21.15.42:0/731836936 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf18010c70 con 0x7fbf0c004da0 2019-12-05T06:21:34.560 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:34.541+0000 7fbf1f7fe700 1 -- 172.21.15.42:0/731836936 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbf18005ca0 con 0x7fbf0c004da0 2019-12-05T06:21:34.560 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:34.541+0000 7fbf26045700 1 -- 172.21.15.42:0/731836936 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbf0c004da0 msgr2=0x7fbf0c04f680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:34.560 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:34.541+0000 7fbf26045700 1 --2- 172.21.15.42:0/731836936 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbf0c004da0 0x7fbf0c04f680 secure :-1 s=READY pgs=2525 cs=0 l=1 rx=0x7fbf180032b0 tx=0x7fbf18010320).stop 2019-12-05T06:21:34.560 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:34.541+0000 7fbf24de3700 1 -- 172.21.15.42:0/731836936 reap_dead start 2019-12-05T06:21:34.560 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:34.541+0000 7fbf26045700 1 -- 172.21.15.42:0/731836936 shutdown_connections 2019-12-05T06:21:34.560 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:34.541+0000 7fbf26045700 1 -- 172.21.15.42:0/731836936 shutdown_connections 2019-12-05T06:21:34.561 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:34.541+0000 7fbf26045700 1 -- 172.21.15.42:0/731836936 wait complete. 2019-12-05T06:21:34.562 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:34.541+0000 7fbf26045700 1 -- 172.21.15.42:0/731836936 >> 172.21.15.42:0/731836936 conn(0x7fbf200aabe0 msgr2=0x7fbf200a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:21:34.562 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:34.545+0000 7fbf26045700 1 Processor -- start 2019-12-05T06:21:34.562 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:34.545+0000 7fbf26045700 1 -- start start 2019-12-05T06:21:34.565 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:34.545+0000 7fbf26045700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbf200a97e0 0x7fbf20123320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:21:34.565 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:34.545+0000 7fbf26045700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fbf20121990 0x7fbf20121dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:21:34.566 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:34.545+0000 7fbf26045700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fbf2007d920 0x7fbf2007dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:21:34.566 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:34.545+0000 7fbf26045700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf200a9c10 con 0x7fbf2007d920 2019-12-05T06:21:34.566 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:34.545+0000 7fbf26045700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf2011a170 con 0x7fbf20121990 2019-12-05T06:21:34.566 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:34.545+0000 7fbf26045700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf2011a410 con 0x7fbf200a97e0 2019-12-05T06:21:34.566 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:34.545+0000 7fbf17fff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fbf20121990 0x7fbf20121dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:34.566 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:34.545+0000 7fbf17fff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fbf20121990 0x7fbf20121dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:53750/0 (socket says 172.21.15.42:53750) 2019-12-05T06:21:34.566 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:34.545+0000 7fbf17fff700 1 -- 172.21.15.42:0/3240225050 learned_addr learned my addr 172.21.15.42:0/3240225050 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:21:34.567 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:34.545+0000 7fbf1ffff700 1 --2- 172.21.15.42:0/3240225050 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbf200a97e0 0x7fbf20123320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:34.567 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:34.545+0000 7fbf24de3700 1 --2- 172.21.15.42:0/3240225050 >> v2:172.21.15.40:3300/0 conn(0x7fbf2007d920 0x7fbf2007dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:34.567 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:34.549+0000 7fbf17fff700 1 -- 172.21.15.42:0/3240225050 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbf200a97e0 msgr2=0x7fbf20123320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:34.567 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:34.549+0000 7fbf17fff700 1 --2- 172.21.15.42:0/3240225050 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbf200a97e0 0x7fbf20123320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:34.567 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:34.549+0000 7fbf17fff700 1 -- 172.21.15.42:0/3240225050 >> v2:172.21.15.40:3300/0 conn(0x7fbf2007d920 msgr2=0x7fbf2007dd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:34.567 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:34.549+0000 7fbf17fff700 1 --2- 172.21.15.42:0/3240225050 >> v2:172.21.15.40:3300/0 conn(0x7fbf2007d920 0x7fbf2007dd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:34.567 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:34.549+0000 7fbf17fff700 1 -- 172.21.15.42:0/3240225050 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf2007e2a0 con 0x7fbf20121990 2019-12-05T06:21:34.568 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:34.549+0000 7fbf17fff700 1 --2- 172.21.15.42:0/3240225050 >> v2:172.21.15.42:3300/0 conn(0x7fbf20121990 0x7fbf20121dc0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fbf20118410 tx=0x7fbf08010b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:34.568 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:34.549+0000 7fbf17fff700 1 --2- 172.21.15.42:0/3240225050 >> v2:172.21.15.42:3300/0 conn(0x7fbf20121990 0x7fbf20121dc0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fbf20118410 tx=0x7fbf08010b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:34.568 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:34.549+0000 7fbf17fff700 1 --2- 172.21.15.42:0/3240225050 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbf20121990 0x7fbf20121dc0 secure :-1 s=READY pgs=2526 cs=0 l=1 rx=0x7fbf20118410 tx=0x7fbf08010b30).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:21:34.568 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:34.549+0000 7fbf17fff700 1 --2- 172.21.15.42:0/3240225050 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbf20121990 0x7fbf20121dc0 secure :-1 s=THROTTLE_DONE pgs=2526 cs=0 l=1 rx=0x7fbf20118410 tx=0x7fbf08010b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:34.568 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:34.549+0000 7fbf1d7fa700 1 -- 172.21.15.42:0/3240225050 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbf08005930 con 0x7fbf20121990 2019-12-05T06:21:34.568 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:34.549+0000 7fbf17fff700 1 --2- 172.21.15.42:0/3240225050 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbf20121990 0x7fbf20121dc0 secure :-1 s=THROTTLE_DONE pgs=2526 cs=0 l=1 rx=0x7fbf20118410 tx=0x7fbf08010b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:34.568 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:34.549+0000 7fbf1d7fa700 1 -- 172.21.15.42:0/3240225050 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf08007160 con 0x7fbf20121990 2019-12-05T06:21:34.569 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:34.549+0000 7fbf17fff700 1 --2- 172.21.15.42:0/3240225050 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbf20121990 0x7fbf20121dc0 secure :-1 s=THROTTLE_DONE pgs=2526 cs=0 l=1 rx=0x7fbf20118410 tx=0x7fbf08010b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:34.569 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:34.549+0000 7fbf26045700 1 -- 172.21.15.42:0/3240225050 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf2011b430 con 0x7fbf20121990 2019-12-05T06:21:34.569 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:34.549+0000 7fbf1d7fa700 1 -- 172.21.15.42:0/3240225050 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbf080081c0 con 0x7fbf20121990 2019-12-05T06:21:34.569 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:34.549+0000 7fbf26045700 1 -- 172.21.15.42:0/3240225050 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf20123870 con 0x7fbf20121990 2019-12-05T06:21:34.569 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:34.549+0000 7fbf17fff700 1 --2- 172.21.15.42:0/3240225050 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbf20121990 0x7fbf20121dc0 secure :-1 s=THROTTLE_DONE pgs=2526 cs=0 l=1 rx=0x7fbf20118410 tx=0x7fbf08010b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:34.569 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:34.549+0000 7fbf1d7fa700 1 -- 172.21.15.42:0/3240225050 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fbf080084a0 con 0x7fbf20121990 2019-12-05T06:21:34.569 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:34.549+0000 7fbf17fff700 1 --2- 172.21.15.42:0/3240225050 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbf20121990 0x7fbf20121dc0 secure :-1 s=THROTTLE_DONE pgs=2526 cs=0 l=1 rx=0x7fbf20118410 tx=0x7fbf08010b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:34.570 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:34.549+0000 7fbf1d7fa700 1 --2- 172.21.15.42:0/3240225050 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbf0403ad20 0x7fbf0403d230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:21:34.570 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:34.549+0000 7fbf1d7fa700 1 -- 172.21.15.42:0/3240225050 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 10589+0+0 (secure 0 0 0) 0x7fbf0804b670 con 0x7fbf20121990 2019-12-05T06:21:34.570 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:34.549+0000 7fbf1ffff700 1 --2- 172.21.15.42:0/3240225050 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbf0403ad20 0x7fbf0403d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:34.570 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:34.549+0000 7fbf1ffff700 1 --2- 172.21.15.42:0/3240225050 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbf0403ad20 0x7fbf0403d230 crc :-1 s=READY pgs=1052 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:21:34.570 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:34.553+0000 7fbf26045700 1 -- 172.21.15.42:0/3240225050 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf2005d110 con 0x7fbf20121990 2019-12-05T06:21:34.570 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:34.557+0000 7fbf17fff700 1 --2- 172.21.15.42:0/3240225050 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbf20121990 0x7fbf20121dc0 secure :-1 s=THROTTLE_DONE pgs=2526 cs=0 l=1 rx=0x7fbf20118410 tx=0x7fbf08010b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:34.571 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:34.557+0000 7fbf1d7fa700 1 -- 172.21.15.42:0/3240225050 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fbf08032670 con 0x7fbf20121990 2019-12-05T06:21:34.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:34.717+0000 7fbf26045700 1 -- 172.21.15.42:0/3240225050 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbf2005b8c0 con 0x7fbf20121990 2019-12-05T06:21:34.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:34.717+0000 7fbf17fff700 1 --2- 172.21.15.42:0/3240225050 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbf20121990 0x7fbf20121dc0 secure :-1 s=THROTTLE_DONE pgs=2526 cs=0 l=1 rx=0x7fbf20118410 tx=0x7fbf08010b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:34.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:34.717+0000 7fbf1d7fa700 1 -- 172.21.15.42:0/3240225050 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 298 v298) v1 ==== 95+0+2114 (secure 0 0 0) 0x7fbf080328d0 con 0x7fbf20121990 2019-12-05T06:21:34.723 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 298 2019-12-05T06:21:34.723 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:21:34.723 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":298,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":10710,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/526579407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":526579407},{"type":"v1","addr":"172.21.15.40:6833","nonce":526579407}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":298}],"filesystems":[{"mdsmap":{"epoch":297,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:19:55.115618+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":102,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10711},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10711":{"gid":10711,"name":"a","rank":0,"incarnation":294,"state":"up:active","state_seq":12,"addr":"172.21.15.42:6835/2638662429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2638662429},{"type":"v1","addr":"172.21.15.42:6835","nonce":2638662429}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:21:34.725 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:34.721+0000 7fbf26045700 1 -- 172.21.15.42:0/3240225050 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbf0403ad20 msgr2=0x7fbf0403d230 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:34.725 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:34.721+0000 7fbf26045700 1 --2- 172.21.15.42:0/3240225050 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbf0403ad20 0x7fbf0403d230 crc :-1 s=READY pgs=1052 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:34.725 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:34.721+0000 7fbf26045700 1 -- 172.21.15.42:0/3240225050 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbf20121990 msgr2=0x7fbf20121dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:34.725 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:34.721+0000 7fbf26045700 1 --2- 172.21.15.42:0/3240225050 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbf20121990 0x7fbf20121dc0 secure :-1 s=READY pgs=2526 cs=0 l=1 rx=0x7fbf20118410 tx=0x7fbf08010b30).stop 2019-12-05T06:21:34.725 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:34.721+0000 7fbf26045700 1 -- 172.21.15.42:0/3240225050 shutdown_connections 2019-12-05T06:21:34.726 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:34.721+0000 7fbf26045700 1 --2- 172.21.15.42:0/3240225050 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbf0403ad20 0x7fbf0403d230 unknown :-1 s=CLOSED pgs=1052 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:34.726 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:34.721+0000 7fbf26045700 1 --2- 172.21.15.42:0/3240225050 >> v2:172.21.15.40:3300/0 conn(0x7fbf2007d920 0x7fbf2007dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:34.727 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:34.721+0000 7fbf26045700 1 --2- 172.21.15.42:0/3240225050 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbf200a97e0 0x7fbf20123320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:34.727 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:34.721+0000 7fbf26045700 1 --2- 172.21.15.42:0/3240225050 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbf20121990 0x7fbf20121dc0 unknown :-1 s=CLOSED pgs=2526 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:34.727 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:34.721+0000 7fbf26045700 1 -- 172.21.15.42:0/3240225050 shutdown_connections 2019-12-05T06:21:34.727 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:34.721+0000 7fbf26045700 1 -- 172.21.15.42:0/3240225050 wait complete. 2019-12-05T06:21:34.727 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:34.721+0000 7fbf26045700 1 -- 172.21.15.42:0/3240225050 >> 172.21.15.42:0/3240225050 conn(0x7fbf20059ef0 msgr2=0x7fbf200a7f90 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:21:34.734 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:21:36.735 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:21:36.834 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:36.821+0000 7face8fa1700 1 Processor -- start 2019-12-05T06:21:36.834 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:36.821+0000 7face8fa1700 1 -- start start 2019-12-05T06:21:36.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:36.821+0000 7face8fa1700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7face408df80 0x7face409f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:21:36.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:36.821+0000 7face8fa1700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7face409f830 0x7face414efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:21:36.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:36.821+0000 7face8fa1700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7face409fc60 con 0x7face40a7740 2019-12-05T06:21:36.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:36.821+0000 7face8fa1700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7face40a0b00 con 0x7face408df80 2019-12-05T06:21:36.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:36.821+0000 7face8fa1700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7face408e5b0 con 0x7face409f830 2019-12-05T06:21:36.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:36.821+0000 7face359e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7face409f830 0x7face414efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:36.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:36.821+0000 7face359e700 1 -- >> v2:172.21.15.42:3300/0 conn(0x7face409f830 msgr2=0x7face414efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2019-12-05T06:21:36.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:36.821+0000 7face359e700 1 -- >> v2:172.21.15.42:3300/0 conn(0x7face409f830 msgr2=0x7face414efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2019-12-05T06:21:36.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:36.821+0000 7face359e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7face409f830 0x7face414efe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-12-05T06:21:36.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:36.821+0000 7face259c700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7face408df80 0x7face409f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:36.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:36.821+0000 7face359e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7face409f830 0x7face414efe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rx=0 tx=0)._fault waiting 0.200000 2019-12-05T06:21:36.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:36.821+0000 7face259c700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7face408df80 0x7face409f2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.42:41574/0 (socket says 172.21.15.42:41574) 2019-12-05T06:21:36.837 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:36.821+0000 7face259c700 1 -- 172.21.15.42:0/3613750243 learned_addr learned my addr 172.21.15.42:0/3613750243 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:21:36.837 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:36.821+0000 7face2d9d700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7face40a7740 0x7face407e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:35002/0 (socket says 172.21.15.42:35002) 2019-12-05T06:21:36.837 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:36.825+0000 7face1d9b700 1 -- 172.21.15.42:0/3613750243 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 909655274 0 0) 0x7face409fc60 con 0x7face40a7740 2019-12-05T06:21:36.837 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:36.825+0000 7face1d9b700 1 -- 172.21.15.42:0/3613750243 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7facd00030b0 con 0x7face40a7740 2019-12-05T06:21:36.837 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:36.825+0000 7face259c700 1 -- 172.21.15.42:0/3613750243 >> v2:172.21.15.42:3300/0 conn(0x7face409f830 msgr2=0x7face414efe0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2019-12-05T06:21:36.837 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:36.825+0000 7face259c700 1 --2- 172.21.15.42:0/3613750243 >> v2:172.21.15.42:3300/0 conn(0x7face409f830 0x7face414efe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:21:36.838 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:36.825+0000 7face259c700 1 -- 172.21.15.42:0/3613750243 >> v1:172.21.15.40:6789/0 conn(0x7face40a7740 legacy=0x7face407e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:36.838 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:36.825+0000 7face259c700 1 -- 172.21.15.42:0/3613750243 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7face40a0710 con 0x7face408df80 2019-12-05T06:21:36.838 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:36.825+0000 7face259c700 1 --2- 172.21.15.42:0/3613750243 >> v2:172.21.15.40:3300/0 conn(0x7face408df80 0x7face409f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7faccc002f30 tx=0x7faccc00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:36.838 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:36.825+0000 7face259c700 1 --2- 172.21.15.42:0/3613750243 >> v2:172.21.15.40:3300/0 conn(0x7face408df80 0x7face409f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7faccc002f30 tx=0x7faccc00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:36.838 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:36.825+0000 7face259c700 1 --2- 172.21.15.42:0/3613750243 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7face408df80 0x7face409f2e0 secure :-1 s=READY pgs=1894 cs=0 l=1 rx=0x7faccc002f30 tx=0x7faccc00fd30).ready entity=mon.0 client_cookie=8290f954ed80a165 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:21:36.838 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:36.825+0000 7face259c700 1 --2- 172.21.15.42:0/3613750243 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7face408df80 0x7face409f2e0 secure :-1 s=THROTTLE_DONE pgs=1894 cs=0 l=1 rx=0x7faccc002f30 tx=0x7faccc00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:36.838 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:36.825+0000 7face1d9b700 1 -- 172.21.15.42:0/3613750243 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faccc008c90 con 0x7face408df80 2019-12-05T06:21:36.839 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:36.825+0000 7face259c700 1 --2- 172.21.15.42:0/3613750243 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7face408df80 0x7face409f2e0 secure :-1 s=THROTTLE_DONE pgs=1894 cs=0 l=1 rx=0x7faccc002f30 tx=0x7faccc00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:36.839 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:36.825+0000 7face259c700 1 --2- 172.21.15.42:0/3613750243 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7face408df80 0x7face409f2e0 secure :-1 s=THROTTLE_DONE pgs=1894 cs=0 l=1 rx=0x7faccc002f30 tx=0x7faccc00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:36.839 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:36.825+0000 7face1d9b700 1 -- 172.21.15.42:0/3613750243 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faccc0088b0 con 0x7face408df80 2019-12-05T06:21:36.839 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:36.825+0000 7face1d9b700 1 -- 172.21.15.42:0/3613750243 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faccc011ad0 con 0x7face408df80 2019-12-05T06:21:36.839 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:36.825+0000 7face8fa1700 1 -- 172.21.15.42:0/3613750243 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7face408df80 msgr2=0x7face409f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:36.839 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:36.825+0000 7face8fa1700 1 --2- 172.21.15.42:0/3613750243 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7face408df80 0x7face409f2e0 secure :-1 s=READY pgs=1894 cs=0 l=1 rx=0x7faccc002f30 tx=0x7faccc00fd30).stop 2019-12-05T06:21:36.840 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:36.829+0000 7face8fa1700 1 -- 172.21.15.42:0/3613750243 shutdown_connections 2019-12-05T06:21:36.840 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:36.829+0000 7face8fa1700 1 --2- 172.21.15.42:0/3613750243 >> v2:172.21.15.42:3300/0 conn(0x7face409f830 0x7face414efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:21:36.840 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:36.829+0000 7face8fa1700 1 --2- 172.21.15.42:0/3613750243 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7face408df80 0x7face409f2e0 unknown :-1 s=CLOSED pgs=1894 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:36.840 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:36.829+0000 7face8fa1700 1 -- 172.21.15.42:0/3613750243 shutdown_connections 2019-12-05T06:21:36.840 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:36.829+0000 7face8fa1700 1 -- 172.21.15.42:0/3613750243 wait complete. 2019-12-05T06:21:36.841 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:36.829+0000 7face8fa1700 1 -- 172.21.15.42:0/3613750243 >> 172.21.15.42:0/3613750243 conn(0x7face40aabe0 msgr2=0x7face40a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:21:36.844 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:36.829+0000 7face8fa1700 1 Processor -- start 2019-12-05T06:21:36.845 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:36.829+0000 7face8fa1700 1 -- start start 2019-12-05T06:21:36.845 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:36.829+0000 7face8fa1700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7face40a9ab0 0x7face40a36a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:21:36.845 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:36.829+0000 7face8fa1700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7face4061330 0x7face41446a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:21:36.845 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:36.833+0000 7face8fa1700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7face405f970 con 0x7face4144bf0 2019-12-05T06:21:36.845 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:36.833+0000 7face8fa1700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7face405f230 con 0x7face40a9ab0 2019-12-05T06:21:36.845 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:36.833+0000 7face8fa1700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7face414ed10 con 0x7face4061330 2019-12-05T06:21:36.846 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:36.833+0000 7face359e700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7face4144bf0 0x7face407ab20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:53804/0 (socket says 172.21.15.42:53804) 2019-12-05T06:21:36.846 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:36.833+0000 7face359e700 1 -- 172.21.15.42:0/918218169 learned_addr learned my addr 172.21.15.42:0/918218169 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:21:36.846 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:36.833+0000 7face259c700 1 --2- 172.21.15.42:0/918218169 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7face4061330 0x7face41446a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:36.846 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:36.833+0000 7face2d9d700 1 --2- 172.21.15.42:0/918218169 >> v2:172.21.15.40:3300/0 conn(0x7face40a9ab0 0x7face40a36a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:36.846 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:36.833+0000 7facdb7fe700 1 -- 172.21.15.42:0/918218169 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2766558435 0 0) 0x7face405f970 con 0x7face4144bf0 2019-12-05T06:21:36.846 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:36.833+0000 7face2d9d700 1 -- 172.21.15.42:0/918218169 >> v2:172.21.15.40:3300/0 conn(0x7face40a9ab0 msgr2=0x7face40a36a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2019-12-05T06:21:36.847 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:36.833+0000 7face2d9d700 1 -- 172.21.15.42:0/918218169 >> v2:172.21.15.40:3300/0 conn(0x7face40a9ab0 msgr2=0x7face40a36a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2019-12-05T06:21:36.847 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:36.833+0000 7face2d9d700 1 --2- 172.21.15.42:0/918218169 >> v2:172.21.15.40:3300/0 conn(0x7face40a9ab0 0x7face40a36a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-12-05T06:21:36.847 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:36.833+0000 7facdb7fe700 1 -- 172.21.15.42:0/918218169 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7facc80030b0 con 0x7face4144bf0 2019-12-05T06:21:36.847 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:36.833+0000 7face2d9d700 1 --2- 172.21.15.42:0/918218169 >> v2:172.21.15.40:3300/0 conn(0x7face40a9ab0 0x7face40a36a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rx=0 tx=0)._fault waiting 0.200000 2019-12-05T06:21:36.847 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:36.833+0000 7face259c700 1 -- 172.21.15.42:0/918218169 >> v2:172.21.15.40:3300/0 conn(0x7face40a9ab0 msgr2=0x7face40a36a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2019-12-05T06:21:36.848 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:36.833+0000 7face259c700 1 --2- 172.21.15.42:0/918218169 >> v2:172.21.15.40:3300/0 conn(0x7face40a9ab0 0x7face40a36a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:36.848 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:36.833+0000 7face259c700 1 -- 172.21.15.42:0/918218169 >> v1:172.21.15.42:6789/0 conn(0x7face4144bf0 legacy=0x7face407ab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:36.848 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:36.833+0000 7face259c700 1 -- 172.21.15.42:0/918218169 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7face40a3bf0 con 0x7face4061330 2019-12-05T06:21:36.848 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:36.833+0000 7face259c700 1 --2- 172.21.15.42:0/918218169 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7face4061330 0x7face41446a0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7faccc002f30 tx=0x7faccc0118e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:36.848 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:36.833+0000 7face259c700 1 --2- 172.21.15.42:0/918218169 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7face4061330 0x7face41446a0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7faccc002f30 tx=0x7faccc0118e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:36.848 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:36.833+0000 7face259c700 1 --2- 172.21.15.42:0/918218169 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7face4061330 0x7face41446a0 secure :-1 s=READY pgs=547 cs=0 l=1 rx=0x7faccc002f30 tx=0x7faccc0118e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:21:36.848 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:36.833+0000 7face259c700 1 --2- 172.21.15.42:0/918218169 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7face4061330 0x7face41446a0 secure :-1 s=THROTTLE_DONE pgs=547 cs=0 l=1 rx=0x7faccc002f30 tx=0x7faccc0118e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:36.849 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:36.833+0000 7facdb7fe700 1 -- 172.21.15.42:0/918218169 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faccc00f490 con 0x7face4061330 2019-12-05T06:21:36.849 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:36.833+0000 7face259c700 1 --2- 172.21.15.42:0/918218169 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7face4061330 0x7face41446a0 secure :-1 s=THROTTLE_DONE pgs=547 cs=0 l=1 rx=0x7faccc002f30 tx=0x7faccc0118e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:36.849 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:36.833+0000 7facdb7fe700 1 -- 172.21.15.42:0/918218169 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faccc0116f0 con 0x7face4061330 2019-12-05T06:21:36.849 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:36.833+0000 7face259c700 1 --2- 172.21.15.42:0/918218169 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7face4061330 0x7face41446a0 secure :-1 s=THROTTLE_DONE pgs=547 cs=0 l=1 rx=0x7faccc002f30 tx=0x7faccc0118e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:36.849 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:36.833+0000 7face8fa1700 1 -- 172.21.15.42:0/918218169 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7face405ed80 con 0x7face4061330 2019-12-05T06:21:36.850 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:36.833+0000 7facdb7fe700 1 -- 172.21.15.42:0/918218169 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faccc005730 con 0x7face4061330 2019-12-05T06:21:36.850 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:36.833+0000 7face8fa1700 1 -- 172.21.15.42:0/918218169 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7face4145020 con 0x7face4061330 2019-12-05T06:21:36.850 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:36.833+0000 7face259c700 1 --2- 172.21.15.42:0/918218169 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7face4061330 0x7face41446a0 secure :-1 s=THROTTLE_DONE pgs=547 cs=0 l=1 rx=0x7faccc002f30 tx=0x7faccc0118e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:36.850 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:36.833+0000 7facdb7fe700 1 -- 172.21.15.42:0/918218169 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7faccc0059d0 con 0x7face4061330 2019-12-05T06:21:36.850 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:36.833+0000 7face259c700 1 --2- 172.21.15.42:0/918218169 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7face4061330 0x7face41446a0 secure :-1 s=THROTTLE_DONE pgs=547 cs=0 l=1 rx=0x7faccc002f30 tx=0x7faccc0118e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:36.850 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:36.833+0000 7facdb7fe700 1 --2- 172.21.15.42:0/918218169 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7facc803b2a0 0x7facc803d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:21:36.850 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:36.833+0000 7facdb7fe700 1 -- 172.21.15.42:0/918218169 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 10675+0+0 (secure 0 0 0) 0x7faccc04ba10 con 0x7face4061330 2019-12-05T06:21:36.851 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:36.833+0000 7face2d9d700 1 --2- 172.21.15.42:0/918218169 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7facc803b2a0 0x7facc803d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:36.851 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:36.837+0000 7face2d9d700 1 --2- 172.21.15.42:0/918218169 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7facc803b2a0 0x7facc803d7b0 crc :-1 s=READY pgs=1053 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:21:36.851 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:36.837+0000 7face8fa1700 1 -- 172.21.15.42:0/918218169 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7face414eed0 con 0x7face4061330 2019-12-05T06:21:36.852 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:36.841+0000 7face259c700 1 --2- 172.21.15.42:0/918218169 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7face4061330 0x7face41446a0 secure :-1 s=THROTTLE_DONE pgs=547 cs=0 l=1 rx=0x7faccc002f30 tx=0x7faccc0118e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:36.852 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:36.841+0000 7facdb7fe700 1 -- 172.21.15.42:0/918218169 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7faccc032cd0 con 0x7face4061330 2019-12-05T06:21:37.009 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:37.005+0000 7face8fa1700 1 -- 172.21.15.42:0/918218169 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7face405b8c0 con 0x7face4061330 2019-12-05T06:21:37.010 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:37.005+0000 7face259c700 1 --2- 172.21.15.42:0/918218169 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7face4061330 0x7face41446a0 secure :-1 s=THROTTLE_DONE pgs=547 cs=0 l=1 rx=0x7faccc002f30 tx=0x7faccc0118e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:37.010 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:37.005+0000 7facdb7fe700 1 -- 172.21.15.42:0/918218169 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 299 v299) v1 ==== 95+0+1775 (secure 0 0 0) 0x7faccc033880 con 0x7face4061330 2019-12-05T06:21:37.012 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 299 2019-12-05T06:21:37.012 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:21:37.012 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":299,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":299,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:21:34.796620+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":103,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10710},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10710":{"gid":10710,"name":"b","rank":0,"incarnation":299,"state":"up:replay","state_seq":2,"addr":"172.21.15.40:6833/526579407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":526579407},{"type":"v1","addr":"172.21.15.40:6833","nonce":526579407}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:21:37.015 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:37.009+0000 7face8fa1700 1 -- 172.21.15.42:0/918218169 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7facc803b2a0 msgr2=0x7facc803d7b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:37.015 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:37.009+0000 7face8fa1700 1 --2- 172.21.15.42:0/918218169 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7facc803b2a0 0x7facc803d7b0 crc :-1 s=READY pgs=1053 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:37.015 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:37.009+0000 7face8fa1700 1 -- 172.21.15.42:0/918218169 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7face4061330 msgr2=0x7face41446a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:37.015 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:37.009+0000 7face8fa1700 1 --2- 172.21.15.42:0/918218169 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7face4061330 0x7face41446a0 secure :-1 s=READY pgs=547 cs=0 l=1 rx=0x7faccc002f30 tx=0x7faccc0118e0).stop 2019-12-05T06:21:37.016 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:37.009+0000 7face8fa1700 1 -- 172.21.15.42:0/918218169 shutdown_connections 2019-12-05T06:21:37.016 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:37.009+0000 7face8fa1700 1 --2- 172.21.15.42:0/918218169 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7facc803b2a0 0x7facc803d7b0 unknown :-1 s=CLOSED pgs=1053 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:37.016 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:37.009+0000 7face8fa1700 1 --2- 172.21.15.42:0/918218169 >> v2:172.21.15.40:3300/0 conn(0x7face40a9ab0 0x7face40a36a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:37.016 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:37.009+0000 7face8fa1700 1 --2- 172.21.15.42:0/918218169 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7face4061330 0x7face41446a0 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:37.016 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:37.009+0000 7face8fa1700 1 -- 172.21.15.42:0/918218169 shutdown_connections 2019-12-05T06:21:37.016 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:37.009+0000 7face8fa1700 1 -- 172.21.15.42:0/918218169 wait complete. 2019-12-05T06:21:37.017 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:37.009+0000 7face8fa1700 1 -- 172.21.15.42:0/918218169 >> 172.21.15.42:0/918218169 conn(0x7face408eb70 msgr2=0x7face407c590 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:21:37.024 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a down, removed from mdsmap 2019-12-05T06:21:37.024 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2019-12-05T06:21:37.024 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:21:37.113 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:37.105+0000 7f7fcc086700 1 Processor -- start 2019-12-05T06:21:37.114 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:37.105+0000 7f7fcc086700 1 -- start start 2019-12-05T06:21:37.114 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:37.105+0000 7f7fcc086700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f7fc40a7740 0x7f7fc408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:21:37.114 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:37.105+0000 7f7fcc086700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7fc407e720 0x7f7fc407eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:21:37.114 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:37.105+0000 7f7fcc086700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fc40a0710 con 0x7f7fc409f2e0 2019-12-05T06:21:37.114 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:37.105+0000 7f7fcc086700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7fc40a0b00 con 0x7f7fc407e720 2019-12-05T06:21:37.115 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:37.105+0000 7f7fcc086700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7fc408e5b0 con 0x7f7fc40a7740 2019-12-05T06:21:37.115 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:37.105+0000 7f7fca623700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f7fc40a7740 0x7f7fc408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:37.115 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:37.105+0000 7f7fca623700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f7fc40a7740 0x7f7fc408df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:53770/0 (socket says 172.21.15.42:53770) 2019-12-05T06:21:37.115 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:37.105+0000 7f7fca623700 1 -- 172.21.15.42:0/1576179843 learned_addr learned my addr 172.21.15.42:0/1576179843 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:21:37.115 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:37.105+0000 7f7fc9e22700 1 --2- 172.21.15.42:0/1576179843 >> v2:172.21.15.40:3300/0 conn(0x7f7fc407e720 0x7f7fc407eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:37.115 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:37.105+0000 7f7fca623700 1 -- 172.21.15.42:0/1576179843 >> v2:172.21.15.40:3300/0 conn(0x7f7fc407e720 msgr2=0x7f7fc407eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:21:37.116 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:37.105+0000 7f7fca623700 1 --2- 172.21.15.42:0/1576179843 >> v2:172.21.15.40:3300/0 conn(0x7f7fc407e720 0x7f7fc407eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:21:37.116 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:37.105+0000 7f7fca623700 1 -- 172.21.15.42:0/1576179843 >> v1:172.21.15.40:6789/0 conn(0x7f7fc409f2e0 legacy=0x7f7fc409f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:37.117 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:37.105+0000 7f7fca623700 1 -- 172.21.15.42:0/1576179843 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fc407f0a0 con 0x7f7fc40a7740 2019-12-05T06:21:37.117 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:37.105+0000 7f7fca623700 1 --2- 172.21.15.42:0/1576179843 >> v2:172.21.15.42:3300/0 conn(0x7f7fc40a7740 0x7f7fc408df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f7fbc0032b0 tx=0x7f7fbc0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:37.117 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:37.105+0000 7f7fca623700 1 --2- 172.21.15.42:0/1576179843 >> v2:172.21.15.42:3300/0 conn(0x7f7fc40a7740 0x7f7fc408df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f7fbc0032b0 tx=0x7f7fbc0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:37.117 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:37.105+0000 7f7fca623700 1 --2- 172.21.15.42:0/1576179843 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7fc40a7740 0x7f7fc408df80 secure :-1 s=READY pgs=2528 cs=0 l=1 rx=0x7f7fbc0032b0 tx=0x7f7fbc0100b0).ready entity=mon.1 client_cookie=d108b51e61365d39 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:21:37.117 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:37.105+0000 7f7fca623700 1 --2- 172.21.15.42:0/1576179843 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7fc40a7740 0x7f7fc408df80 secure :-1 s=THROTTLE_DONE pgs=2528 cs=0 l=1 rx=0x7f7fbc0032b0 tx=0x7f7fbc0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:37.117 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:37.105+0000 7f7fc9621700 1 -- 172.21.15.42:0/1576179843 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7fbc011dd0 con 0x7f7fc40a7740 2019-12-05T06:21:37.118 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:37.105+0000 7f7fca623700 1 --2- 172.21.15.42:0/1576179843 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7fc40a7740 0x7f7fc408df80 secure :-1 s=THROTTLE_DONE pgs=2528 cs=0 l=1 rx=0x7f7fbc0032b0 tx=0x7f7fbc0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:37.118 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:37.105+0000 7f7fca623700 1 --2- 172.21.15.42:0/1576179843 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7fc40a7740 0x7f7fc408df80 secure :-1 s=THROTTLE_DONE pgs=2528 cs=0 l=1 rx=0x7f7fbc0032b0 tx=0x7f7fbc0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:37.118 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:37.105+0000 7f7fc9621700 1 -- 172.21.15.42:0/1576179843 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fbc0119f0 con 0x7f7fc40a7740 2019-12-05T06:21:37.118 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:37.105+0000 7f7fc9621700 1 -- 172.21.15.42:0/1576179843 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7fbc00a540 con 0x7f7fc40a7740 2019-12-05T06:21:37.118 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:37.105+0000 7f7fcc086700 1 -- 172.21.15.42:0/1576179843 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7fc40a7740 msgr2=0x7f7fc408df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:37.118 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:37.105+0000 7f7fcc086700 1 --2- 172.21.15.42:0/1576179843 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7fc40a7740 0x7f7fc408df80 secure :-1 s=READY pgs=2528 cs=0 l=1 rx=0x7f7fbc0032b0 tx=0x7f7fbc0100b0).stop 2019-12-05T06:21:37.118 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:37.105+0000 7f7fcc086700 1 -- 172.21.15.42:0/1576179843 shutdown_connections 2019-12-05T06:21:37.119 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:37.105+0000 7f7fcc086700 1 --2- 172.21.15.42:0/1576179843 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7fc40a7740 0x7f7fc408df80 unknown :-1 s=CLOSED pgs=2528 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:37.119 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:37.105+0000 7f7fcc086700 1 --2- 172.21.15.42:0/1576179843 >> v2:172.21.15.40:3300/0 conn(0x7f7fc407e720 0x7f7fc407eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:21:37.119 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:37.105+0000 7f7fcc086700 1 -- 172.21.15.42:0/1576179843 shutdown_connections 2019-12-05T06:21:37.119 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:37.105+0000 7f7fcc086700 1 -- 172.21.15.42:0/1576179843 wait complete. 2019-12-05T06:21:37.119 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:37.105+0000 7f7fcc086700 1 -- 172.21.15.42:0/1576179843 >> 172.21.15.42:0/1576179843 conn(0x7f7fc40aabe0 msgr2=0x7f7fc40a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:21:37.119 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:37.109+0000 7f7fcc086700 1 Processor -- start 2019-12-05T06:21:37.120 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:37.109+0000 7f7fcc086700 1 -- start start 2019-12-05T06:21:37.122 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:37.109+0000 7f7fcc086700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7fc4132680 0x7f7fc4132ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:21:37.123 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:37.109+0000 7f7fcc086700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7fc40702c0 0x7f7fc40706f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:21:37.123 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:37.109+0000 7f7fcc086700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fc4059cd0 con 0x7f7fc40598a0 2019-12-05T06:21:37.123 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:37.109+0000 7f7fcc086700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7fc4063930 con 0x7f7fc4132680 2019-12-05T06:21:37.123 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:37.109+0000 7f7fcc086700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fc4063bd0 con 0x7f7fc40702c0 2019-12-05T06:21:37.123 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:37.109+0000 7f7fcae24700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7fc40702c0 0x7f7fc40706f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:37.123 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:37.109+0000 7f7fc9e22700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7fc4132680 0x7f7fc4132ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:37.123 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:37.109+0000 7f7fcae24700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7fc40702c0 0x7f7fc40706f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:54246/0 (socket says 172.21.15.42:54246) 2019-12-05T06:21:37.124 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:37.109+0000 7f7fcae24700 1 -- 172.21.15.42:0/4059787670 learned_addr learned my addr 172.21.15.42:0/4059787670 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:21:37.124 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:37.109+0000 7f7fcae24700 1 -- 172.21.15.42:0/4059787670 >> v2:172.21.15.40:3300/0 conn(0x7f7fc4132680 msgr2=0x7f7fc4132ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:37.124 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:37.109+0000 7f7fcae24700 1 --2- 172.21.15.42:0/4059787670 >> v2:172.21.15.40:3300/0 conn(0x7f7fc4132680 0x7f7fc4132ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:37.124 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:37.109+0000 7f7fcae24700 1 -- 172.21.15.42:0/4059787670 >> v1:172.21.15.40:6789/0 conn(0x7f7fc40598a0 legacy=0x7f7fc40655c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:37.124 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:37.109+0000 7f7fb6ffd700 1 -- 172.21.15.42:0/4059787670 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1621777635 0 0) 0x7f7fc4059cd0 con 0x7f7fc40598a0 2019-12-05T06:21:37.124 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:37.109+0000 7f7fcae24700 1 -- 172.21.15.42:0/4059787670 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fc4063620 con 0x7f7fc40702c0 2019-12-05T06:21:37.124 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:37.109+0000 7f7fc9e22700 1 --2- 172.21.15.42:0/4059787670 >> v2:172.21.15.40:3300/0 conn(0x7f7fc4132680 0x7f7fc4132ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:21:37.125 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:37.109+0000 7f7fcae24700 1 --2- 172.21.15.42:0/4059787670 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7fc40702c0 0x7f7fc40706f0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f7fb8000e90 tx=0x7f7fb80107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:37.125 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:37.109+0000 7f7fcae24700 1 --2- 172.21.15.42:0/4059787670 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7fc40702c0 0x7f7fc40706f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f7fb8000e90 tx=0x7f7fb80107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:37.125 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:37.109+0000 7f7fcae24700 1 --2- 172.21.15.42:0/4059787670 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7fc40702c0 0x7f7fc40706f0 secure :-1 s=READY pgs=548 cs=0 l=1 rx=0x7f7fb8000e90 tx=0x7f7fb80107c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:21:37.125 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:37.109+0000 7f7fcae24700 1 --2- 172.21.15.42:0/4059787670 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7fc40702c0 0x7f7fc40706f0 secure :-1 s=THROTTLE_DONE pgs=548 cs=0 l=1 rx=0x7f7fb8000e90 tx=0x7f7fb80107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:37.125 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:37.109+0000 7f7fb6ffd700 1 -- 172.21.15.42:0/4059787670 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7fb8006a40 con 0x7f7fc40702c0 2019-12-05T06:21:37.125 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:37.109+0000 7f7fcae24700 1 --2- 172.21.15.42:0/4059787670 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7fc40702c0 0x7f7fc40706f0 secure :-1 s=THROTTLE_DONE pgs=548 cs=0 l=1 rx=0x7f7fb8000e90 tx=0x7f7fb80107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:37.126 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:37.109+0000 7f7fcae24700 1 --2- 172.21.15.42:0/4059787670 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7fc40702c0 0x7f7fc40706f0 secure :-1 s=THROTTLE_DONE pgs=548 cs=0 l=1 rx=0x7f7fb8000e90 tx=0x7f7fb80107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:37.126 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:37.109+0000 7f7fb6ffd700 1 -- 172.21.15.42:0/4059787670 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fb8006660 con 0x7f7fc40702c0 2019-12-05T06:21:37.126 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:37.109+0000 7f7fb6ffd700 1 -- 172.21.15.42:0/4059787670 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7fb8011000 con 0x7f7fc40702c0 2019-12-05T06:21:37.130 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:37.109+0000 7f7fcc086700 1 -- 172.21.15.42:0/4059787670 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fc4133000 con 0x7f7fc40702c0 2019-12-05T06:21:37.130 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:37.109+0000 7f7fcc086700 1 -- 172.21.15.42:0/4059787670 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fc4070c40 con 0x7f7fc40702c0 2019-12-05T06:21:37.130 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:37.113+0000 7f7fcae24700 1 --2- 172.21.15.42:0/4059787670 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7fc40702c0 0x7f7fc40706f0 secure :-1 s=THROTTLE_DONE pgs=548 cs=0 l=1 rx=0x7f7fb8000e90 tx=0x7f7fb80107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:37.131 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:37.113+0000 7f7fb6ffd700 1 -- 172.21.15.42:0/4059787670 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f7fb8011310 con 0x7f7fc40702c0 2019-12-05T06:21:37.131 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:37.113+0000 7f7fcae24700 1 --2- 172.21.15.42:0/4059787670 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7fc40702c0 0x7f7fc40706f0 secure :-1 s=THROTTLE_DONE pgs=548 cs=0 l=1 rx=0x7f7fb8000e90 tx=0x7f7fb80107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:37.131 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:37.113+0000 7f7fb6ffd700 1 --2- 172.21.15.42:0/4059787670 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f7fac03acc0 0x7f7fac03d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:21:37.131 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:37.113+0000 7f7fb6ffd700 1 -- 172.21.15.42:0/4059787670 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 10675+0+0 (secure 0 0 0) 0x7f7fb804b0c0 con 0x7f7fc40702c0 2019-12-05T06:21:37.131 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:37.113+0000 7f7fca623700 1 --2- 172.21.15.42:0/4059787670 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f7fac03acc0 0x7f7fac03d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:37.131 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:37.113+0000 7f7fca623700 1 --2- 172.21.15.42:0/4059787670 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f7fac03acc0 0x7f7fac03d1d0 crc :-1 s=READY pgs=1054 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:21:37.132 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:37.121+0000 7f7fcc086700 1 -- 172.21.15.42:0/4059787670 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fc405d110 con 0x7f7fc40702c0 2019-12-05T06:21:37.132 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:37.129+0000 7f7fcae24700 1 --2- 172.21.15.42:0/4059787670 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7fc40702c0 0x7f7fc40706f0 secure :-1 s=THROTTLE_DONE pgs=548 cs=0 l=1 rx=0x7f7fb8000e90 tx=0x7f7fb80107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:37.133 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:37.129+0000 7f7fb6ffd700 1 -- 172.21.15.42:0/4059787670 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f7fb80320e0 con 0x7f7fc40702c0 2019-12-05T06:21:37.293 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:37.289+0000 7f7fcc086700 1 -- 172.21.15.42:0/4059787670 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7fc405b8c0 con 0x7f7fc40702c0 2019-12-05T06:21:37.295 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:37.289+0000 7f7fcae24700 1 --2- 172.21.15.42:0/4059787670 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7fc40702c0 0x7f7fc40706f0 secure :-1 s=THROTTLE_DONE pgs=548 cs=0 l=1 rx=0x7f7fb8000e90 tx=0x7f7fb80107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:37.295 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:37.289+0000 7f7fb6ffd700 1 -- 172.21.15.42:0/4059787670 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 299 v299) v1 ==== 95+0+1775 (secure 0 0 0) 0x7f7fb8032340 con 0x7f7fc40702c0 2019-12-05T06:21:37.295 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 299 2019-12-05T06:21:37.295 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:21:37.295 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":299,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":299,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:21:34.796620+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":103,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10710},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10710":{"gid":10710,"name":"b","rank":0,"incarnation":299,"state":"up:replay","state_seq":2,"addr":"172.21.15.40:6833/526579407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":526579407},{"type":"v1","addr":"172.21.15.40:6833","nonce":526579407}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:21:37.297 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:37.293+0000 7f7fcc086700 1 -- 172.21.15.42:0/4059787670 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f7fac03acc0 msgr2=0x7f7fac03d1d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:37.298 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:37.293+0000 7f7fcc086700 1 --2- 172.21.15.42:0/4059787670 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f7fac03acc0 0x7f7fac03d1d0 crc :-1 s=READY pgs=1054 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:37.298 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:37.293+0000 7f7fcc086700 1 -- 172.21.15.42:0/4059787670 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7fc40702c0 msgr2=0x7f7fc40706f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:37.298 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:37.293+0000 7f7fcc086700 1 --2- 172.21.15.42:0/4059787670 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7fc40702c0 0x7f7fc40706f0 secure :-1 s=READY pgs=548 cs=0 l=1 rx=0x7f7fb8000e90 tx=0x7f7fb80107c0).stop 2019-12-05T06:21:37.298 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:37.293+0000 7f7fcc086700 1 -- 172.21.15.42:0/4059787670 shutdown_connections 2019-12-05T06:21:37.298 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:37.293+0000 7f7fcc086700 1 --2- 172.21.15.42:0/4059787670 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f7fac03acc0 0x7f7fac03d1d0 unknown :-1 s=CLOSED pgs=1054 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:37.299 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:37.293+0000 7f7fcc086700 1 --2- 172.21.15.42:0/4059787670 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7fc40702c0 0x7f7fc40706f0 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:37.299 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:37.293+0000 7f7fcc086700 1 --2- 172.21.15.42:0/4059787670 >> v2:172.21.15.40:3300/0 conn(0x7f7fc4132680 0x7f7fc4132ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:37.299 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:37.293+0000 7f7fcc086700 1 -- 172.21.15.42:0/4059787670 shutdown_connections 2019-12-05T06:21:37.299 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:37.293+0000 7f7fcc086700 1 -- 172.21.15.42:0/4059787670 wait complete. 2019-12-05T06:21:37.299 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:37.293+0000 7f7fcc086700 1 -- 172.21.15.42:0/4059787670 >> 172.21.15.42:0/4059787670 conn(0x7f7fc40a9c40 msgr2=0x7f7fc40a7f90 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:21:37.307 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 299, "feature_flags": { "enable_multiple": false, "ever_enabled_multiple": false }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2019-12-05T05:03:35.604738+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 299, "ever_allowed_features": 2, "explicitly_allowed_features": 2, "failed": [], "flags": 18, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_10710": { "addr": "172.21.15.40:6833/526579407", "addrs": { "addrvec": [ { "addr": "172.21.15.40:6832", "nonce": 526579407, "type": "v2" }, { "addr": "172.21.15.40:6833", "nonce": 526579407, "type": "v1" } ] }, "export_targets": [], "features": 4540138292836696063, "flags": 0, "gid": 10710, "incarnation": 299, "name": "b", "rank": 0, "state": "up:replay", "state_seq": 2 } }, "last_failure": 0, "last_failure_osd_epoch": 103, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 2, "min_compat_client": "0 (unknown)", "modified": "2019-12-05T06:21:34.796620+0000", "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 10710 } } } ], "standbys": [] } 2019-12-05T06:21:39.308 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:21:39.413 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:39.397+0000 7fa9a1cd8700 1 Processor -- start 2019-12-05T06:21:39.413 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:39.397+0000 7fa9a1cd8700 1 -- start start 2019-12-05T06:21:39.413 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:39.401+0000 7fa9a1cd8700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa99c0a7730 0x7fa99c08df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:21:39.414 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:39.401+0000 7fa9a1cd8700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa99c0a0700 con 0x7fa99c07e710 2019-12-05T06:21:39.414 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:39.401+0000 7fa9a1cd8700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa99c0a9b90 con 0x7fa99c09fad0 2019-12-05T06:21:39.414 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:39.401+0000 7fa9a1cd8700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa99c0a0af0 con 0x7fa99c0a7730 2019-12-05T06:21:39.414 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:39.401+0000 7fa9a0a76700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fa99c09fad0 0x7fa99c14efd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:53830/0 (socket says 172.21.15.42:53830) 2019-12-05T06:21:39.414 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:39.401+0000 7fa9a0a76700 1 -- 172.21.15.42:0/1088421722 learned_addr learned my addr 172.21.15.42:0/1088421722 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:21:39.414 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:39.401+0000 7fa99bfff700 1 --2- 172.21.15.42:0/1088421722 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa99c0a7730 0x7fa99c08df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:39.414 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:39.401+0000 7fa99affd700 1 -- 172.21.15.42:0/1088421722 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 221209674 0 0) 0x7fa99c0a9b90 con 0x7fa99c09fad0 2019-12-05T06:21:39.415 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:39.401+0000 7fa99affd700 1 -- 172.21.15.42:0/1088421722 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9880030b0 con 0x7fa99c09fad0 2019-12-05T06:21:39.415 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:39.401+0000 7fa99affd700 1 -- 172.21.15.42:0/1088421722 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 393556691 0 0) 0x7fa9880030b0 con 0x7fa99c09fad0 2019-12-05T06:21:39.415 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:39.401+0000 7fa99affd700 1 -- 172.21.15.42:0/1088421722 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa99c0a7730 msgr2=0x7fa99c08df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:21:39.416 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:39.401+0000 7fa99affd700 1 --2- 172.21.15.42:0/1088421722 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa99c0a7730 0x7fa99c08df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:21:39.416 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:39.401+0000 7fa99affd700 1 -- 172.21.15.42:0/1088421722 >> v1:172.21.15.40:6789/0 conn(0x7fa99c07e710 legacy=0x7fa99c09f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:39.416 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:39.401+0000 7fa99affd700 1 -- 172.21.15.42:0/1088421722 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa99c07eb40 con 0x7fa99c09fad0 2019-12-05T06:21:39.416 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:39.401+0000 7fa99affd700 1 -- 172.21.15.42:0/1088421722 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7fa98c001ad0 con 0x7fa99c09fad0 2019-12-05T06:21:39.416 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:39.401+0000 7fa99affd700 1 -- 172.21.15.42:0/1088421722 >> v1:172.21.15.42:6789/0 conn(0x7fa99c09fad0 legacy=0x7fa99c14efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:39.416 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:39.401+0000 7fa99affd700 1 --2- 172.21.15.42:0/1088421722 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa988003710 0x7fa988003b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:21:39.417 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:39.401+0000 7fa99affd700 1 --2- 172.21.15.42:0/1088421722 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa988004420 0x7fa988004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:21:39.417 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:39.401+0000 7fa99affd700 1 --2- 172.21.15.42:0/1088421722 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa988004da0 0x7fa98800b670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:21:39.417 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:39.401+0000 7fa99affd700 1 -- 172.21.15.42:0/1088421722 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa99c0a0af0 con 0x7fa988003710 2019-12-05T06:21:39.417 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:39.401+0000 7fa99affd700 1 -- 172.21.15.42:0/1088421722 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9880051d0 con 0x7fa988004420 2019-12-05T06:21:39.417 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:39.401+0000 7fa99affd700 1 -- 172.21.15.42:0/1088421722 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa98804fbd0 con 0x7fa988004da0 2019-12-05T06:21:39.418 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:39.401+0000 7fa99b7fe700 1 --2- 172.21.15.42:0/1088421722 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa988004420 0x7fa988004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:39.418 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:39.401+0000 7fa99bfff700 1 --2- 172.21.15.42:0/1088421722 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa988003710 0x7fa988003b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:39.419 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:39.401+0000 7fa9a0a76700 1 --2- 172.21.15.42:0/1088421722 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa988004da0 0x7fa98800b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:39.419 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:39.401+0000 7fa99b7fe700 1 -- 172.21.15.42:0/1088421722 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa988004da0 msgr2=0x7fa98800b670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:21:39.419 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:39.401+0000 7fa99b7fe700 1 --2- 172.21.15.42:0/1088421722 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa988004da0 0x7fa98800b670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:21:39.419 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:39.401+0000 7fa99b7fe700 1 -- 172.21.15.42:0/1088421722 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa988003710 msgr2=0x7fa988003b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:21:39.419 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:39.401+0000 7fa99b7fe700 1 --2- 172.21.15.42:0/1088421722 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa988003710 0x7fa988003b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:21:39.419 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:39.401+0000 7fa99b7fe700 1 -- 172.21.15.42:0/1088421722 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa98804fed0 con 0x7fa988004420 2019-12-05T06:21:39.420 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:39.401+0000 7fa9a0a76700 1 --2- 172.21.15.42:0/1088421722 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa988004da0 0x7fa98800b670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:21:39.420 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:39.401+0000 7fa99b7fe700 1 --2- 172.21.15.42:0/1088421722 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa988004420 0x7fa988004850 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fa984000fe0 tx=0x7fa98400fd80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:39.420 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:39.401+0000 7fa99b7fe700 1 --2- 172.21.15.42:0/1088421722 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa988004420 0x7fa988004850 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fa984000fe0 tx=0x7fa98400fd80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:39.420 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:39.401+0000 7fa99b7fe700 1 --2- 172.21.15.42:0/1088421722 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa988004420 0x7fa988004850 secure :-1 s=READY pgs=2530 cs=0 l=1 rx=0x7fa984000fe0 tx=0x7fa98400fd80).ready entity=mon.1 client_cookie=110d1ad12283068e server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:21:39.420 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:39.401+0000 7fa99b7fe700 1 --2- 172.21.15.42:0/1088421722 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa988004420 0x7fa988004850 secure :-1 s=THROTTLE_DONE pgs=2530 cs=0 l=1 rx=0x7fa984000fe0 tx=0x7fa98400fd80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:39.421 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:39.401+0000 7fa99affd700 1 -- 172.21.15.42:0/1088421722 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa984003540 con 0x7fa988004420 2019-12-05T06:21:39.421 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:39.401+0000 7fa99b7fe700 1 --2- 172.21.15.42:0/1088421722 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa988004420 0x7fa988004850 secure :-1 s=THROTTLE_DONE pgs=2530 cs=0 l=1 rx=0x7fa984000fe0 tx=0x7fa98400fd80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:39.421 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:39.401+0000 7fa99b7fe700 1 --2- 172.21.15.42:0/1088421722 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa988004420 0x7fa988004850 secure :-1 s=THROTTLE_DONE pgs=2530 cs=0 l=1 rx=0x7fa984000fe0 tx=0x7fa98400fd80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:39.421 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:39.401+0000 7fa99affd700 1 -- 172.21.15.42:0/1088421722 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa984003160 con 0x7fa988004420 2019-12-05T06:21:39.421 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:39.401+0000 7fa99affd700 1 -- 172.21.15.42:0/1088421722 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa984003b20 con 0x7fa988004420 2019-12-05T06:21:39.422 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:39.401+0000 7fa9a1cd8700 1 -- 172.21.15.42:0/1088421722 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa988004420 msgr2=0x7fa988004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:39.422 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:39.401+0000 7fa9a1cd8700 1 --2- 172.21.15.42:0/1088421722 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa988004420 0x7fa988004850 secure :-1 s=READY pgs=2530 cs=0 l=1 rx=0x7fa984000fe0 tx=0x7fa98400fd80).stop 2019-12-05T06:21:39.422 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:39.401+0000 7fa9a0a76700 1 -- 172.21.15.42:0/1088421722 reap_dead start 2019-12-05T06:21:39.422 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:39.401+0000 7fa9a1cd8700 1 -- 172.21.15.42:0/1088421722 shutdown_connections 2019-12-05T06:21:39.422 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:39.401+0000 7fa9a1cd8700 1 -- 172.21.15.42:0/1088421722 shutdown_connections 2019-12-05T06:21:39.422 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:39.401+0000 7fa9a1cd8700 1 -- 172.21.15.42:0/1088421722 wait complete. 2019-12-05T06:21:39.422 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:39.401+0000 7fa9a1cd8700 1 -- 172.21.15.42:0/1088421722 >> 172.21.15.42:0/1088421722 conn(0x7fa99c0aabd0 msgr2=0x7fa99c0a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:21:39.423 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:39.405+0000 7fa9a1cd8700 1 Processor -- start 2019-12-05T06:21:39.423 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:39.405+0000 7fa9a1cd8700 1 -- start start 2019-12-05T06:21:39.423 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:39.405+0000 7fa9a1cd8700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fa99c08f670 0x7fa99c11b500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:21:39.423 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:39.405+0000 7fa9a1cd8700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa99c11ba50 0x7fa99c147890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:21:39.423 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:39.405+0000 7fa9a1cd8700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa99c11b1b0 con 0x7fa99c147de0 2019-12-05T06:21:39.423 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:39.405+0000 7fa9a1cd8700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa99c12f2c0 con 0x7fa99c08f670 2019-12-05T06:21:39.424 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:39.405+0000 7fa9a1cd8700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa99c0a6db0 con 0x7fa99c11ba50 2019-12-05T06:21:39.424 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:39.405+0000 7fa99bfff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fa99c08f670 0x7fa99c11b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:39.424 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:39.405+0000 7fa99bfff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fa99c08f670 0x7fa99c11b500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:53800/0 (socket says 172.21.15.42:53800) 2019-12-05T06:21:39.424 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:39.405+0000 7fa99bfff700 1 -- 172.21.15.42:0/2534626776 learned_addr learned my addr 172.21.15.42:0/2534626776 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:21:39.424 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:39.405+0000 7fa99b7fe700 1 --2- 172.21.15.42:0/2534626776 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa99c11ba50 0x7fa99c147890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:39.424 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:39.405+0000 7fa998ff9700 1 -- 172.21.15.42:0/2534626776 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2505237801 0 0) 0x7fa99c11b1b0 con 0x7fa99c147de0 2019-12-05T06:21:39.425 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:39.405+0000 7fa998ff9700 1 -- 172.21.15.42:0/2534626776 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9800030b0 con 0x7fa99c147de0 2019-12-05T06:21:39.425 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:39.405+0000 7fa99bfff700 1 -- 172.21.15.42:0/2534626776 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa99c11ba50 msgr2=0x7fa99c147890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:39.425 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:39.405+0000 7fa99bfff700 1 --2- 172.21.15.42:0/2534626776 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa99c11ba50 0x7fa99c147890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:39.425 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:39.405+0000 7fa99bfff700 1 -- 172.21.15.42:0/2534626776 >> v1:172.21.15.40:6789/0 conn(0x7fa99c147de0 legacy=0x7fa99c12f790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:39.425 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:39.405+0000 7fa99bfff700 1 -- 172.21.15.42:0/2534626776 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa99c12ff90 con 0x7fa99c08f670 2019-12-05T06:21:39.425 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:39.405+0000 7fa99bfff700 1 --2- 172.21.15.42:0/2534626776 >> v2:172.21.15.42:3300/0 conn(0x7fa99c08f670 0x7fa99c11b500 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fa99c12f480 tx=0x7fa994010c60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:39.426 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:39.405+0000 7fa99bfff700 1 --2- 172.21.15.42:0/2534626776 >> v2:172.21.15.42:3300/0 conn(0x7fa99c08f670 0x7fa99c11b500 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fa99c12f480 tx=0x7fa994010c60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:39.426 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:39.405+0000 7fa99bfff700 1 --2- 172.21.15.42:0/2534626776 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa99c08f670 0x7fa99c11b500 secure :-1 s=READY pgs=2531 cs=0 l=1 rx=0x7fa99c12f480 tx=0x7fa994010c60).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:21:39.426 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:39.405+0000 7fa99bfff700 1 --2- 172.21.15.42:0/2534626776 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa99c08f670 0x7fa99c11b500 secure :-1 s=THROTTLE_DONE pgs=2531 cs=0 l=1 rx=0x7fa99c12f480 tx=0x7fa994010c60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:39.426 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:39.405+0000 7fa998ff9700 1 -- 172.21.15.42:0/2534626776 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa994005930 con 0x7fa99c08f670 2019-12-05T06:21:39.426 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:39.409+0000 7fa99bfff700 1 --2- 172.21.15.42:0/2534626776 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa99c08f670 0x7fa99c11b500 secure :-1 s=THROTTLE_DONE pgs=2531 cs=0 l=1 rx=0x7fa99c12f480 tx=0x7fa994010c60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:39.427 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:39.409+0000 7fa998ff9700 1 -- 172.21.15.42:0/2534626776 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa994007160 con 0x7fa99c08f670 2019-12-05T06:21:39.427 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:39.409+0000 7fa99bfff700 1 --2- 172.21.15.42:0/2534626776 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa99c08f670 0x7fa99c11b500 secure :-1 s=THROTTLE_DONE pgs=2531 cs=0 l=1 rx=0x7fa99c12f480 tx=0x7fa994010c60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:39.427 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:39.409+0000 7fa998ff9700 1 -- 172.21.15.42:0/2534626776 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa9940122a0 con 0x7fa99c08f670 2019-12-05T06:21:39.427 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:39.409+0000 7fa9a1cd8700 1 -- 172.21.15.42:0/2534626776 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa99c136710 con 0x7fa99c08f670 2019-12-05T06:21:39.427 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:39.409+0000 7fa9a1cd8700 1 -- 172.21.15.42:0/2534626776 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa99c136c40 con 0x7fa99c08f670 2019-12-05T06:21:39.430 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:39.409+0000 7fa99bfff700 1 --2- 172.21.15.42:0/2534626776 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa99c08f670 0x7fa99c11b500 secure :-1 s=THROTTLE_DONE pgs=2531 cs=0 l=1 rx=0x7fa99c12f480 tx=0x7fa994010c60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:39.430 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:39.409+0000 7fa998ff9700 1 -- 172.21.15.42:0/2534626776 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fa9940125c0 con 0x7fa99c08f670 2019-12-05T06:21:39.430 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:39.409+0000 7fa99bfff700 1 --2- 172.21.15.42:0/2534626776 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa99c08f670 0x7fa99c11b500 secure :-1 s=THROTTLE_DONE pgs=2531 cs=0 l=1 rx=0x7fa99c12f480 tx=0x7fa994010c60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:39.430 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:39.409+0000 7fa998ff9700 1 --2- 172.21.15.42:0/2534626776 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa98003b280 0x7fa98003d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:21:39.431 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:39.409+0000 7fa998ff9700 1 -- 172.21.15.42:0/2534626776 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 10675+0+0 (secure 0 0 0) 0x7fa99404b860 con 0x7fa99c08f670 2019-12-05T06:21:39.431 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:39.409+0000 7fa99b7fe700 1 --2- 172.21.15.42:0/2534626776 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa98003b280 0x7fa98003d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:39.431 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:39.409+0000 7fa99b7fe700 1 --2- 172.21.15.42:0/2534626776 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa98003b280 0x7fa98003d790 crc :-1 s=READY pgs=1055 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:21:39.431 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:39.413+0000 7fa9a1cd8700 1 -- 172.21.15.42:0/2534626776 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa99c05d100 con 0x7fa99c08f670 2019-12-05T06:21:39.431 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:39.421+0000 7fa99bfff700 1 --2- 172.21.15.42:0/2534626776 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa99c08f670 0x7fa99c11b500 secure :-1 s=THROTTLE_DONE pgs=2531 cs=0 l=1 rx=0x7fa99c12f480 tx=0x7fa994010c60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:39.431 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:39.421+0000 7fa998ff9700 1 -- 172.21.15.42:0/2534626776 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fa9940328c0 con 0x7fa99c08f670 2019-12-05T06:21:39.587 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:39.581+0000 7fa9a1cd8700 1 -- 172.21.15.42:0/2534626776 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa99c05b8b0 con 0x7fa99c08f670 2019-12-05T06:21:39.588 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:39.581+0000 7fa99bfff700 1 --2- 172.21.15.42:0/2534626776 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa99c08f670 0x7fa99c11b500 secure :-1 s=THROTTLE_DONE pgs=2531 cs=0 l=1 rx=0x7fa99c12f480 tx=0x7fa994010c60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:39.588 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:39.581+0000 7fa998ff9700 1 -- 172.21.15.42:0/2534626776 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 300 v300) v1 ==== 95+0+1779 (secure 0 0 0) 0x7fa994032b20 con 0x7fa99c08f670 2019-12-05T06:21:39.588 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 300 2019-12-05T06:21:39.588 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:21:39.588 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":300,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":300,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:21:39.074846+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":103,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10710},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10710":{"gid":10710,"name":"b","rank":0,"incarnation":299,"state":"up:reconnect","state_seq":27,"addr":"172.21.15.40:6833/526579407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":526579407},{"type":"v1","addr":"172.21.15.40:6833","nonce":526579407}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:21:39.591 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:39.585+0000 7fa9a1cd8700 1 -- 172.21.15.42:0/2534626776 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa98003b280 msgr2=0x7fa98003d790 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:39.591 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:39.585+0000 7fa9a1cd8700 1 --2- 172.21.15.42:0/2534626776 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa98003b280 0x7fa98003d790 crc :-1 s=READY pgs=1055 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:39.592 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:39.585+0000 7fa9a1cd8700 1 -- 172.21.15.42:0/2534626776 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa99c08f670 msgr2=0x7fa99c11b500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:39.592 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:39.585+0000 7fa9a1cd8700 1 --2- 172.21.15.42:0/2534626776 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa99c08f670 0x7fa99c11b500 secure :-1 s=READY pgs=2531 cs=0 l=1 rx=0x7fa99c12f480 tx=0x7fa994010c60).stop 2019-12-05T06:21:39.592 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:39.585+0000 7fa9a1cd8700 1 -- 172.21.15.42:0/2534626776 shutdown_connections 2019-12-05T06:21:39.592 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:39.585+0000 7fa9a1cd8700 1 --2- 172.21.15.42:0/2534626776 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa98003b280 0x7fa98003d790 unknown :-1 s=CLOSED pgs=1055 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:39.592 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:39.585+0000 7fa9a1cd8700 1 --2- 172.21.15.42:0/2534626776 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa99c08f670 0x7fa99c11b500 unknown :-1 s=CLOSED pgs=2531 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:39.593 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:39.585+0000 7fa9a1cd8700 1 --2- 172.21.15.42:0/2534626776 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa99c11ba50 0x7fa99c147890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:39.593 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:39.585+0000 7fa9a1cd8700 1 -- 172.21.15.42:0/2534626776 shutdown_connections 2019-12-05T06:21:39.593 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:39.585+0000 7fa9a1cd8700 1 -- 172.21.15.42:0/2534626776 wait complete. 2019-12-05T06:21:39.593 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:39.585+0000 7fa9a1cd8700 1 -- 172.21.15.42:0/2534626776 >> 172.21.15.42:0/2534626776 conn(0x7fa99c0a0f40 msgr2=0x7fa99c141350 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:21:39.604 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2019-12-05T06:21:41.605 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:21:41.700 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:41.689+0000 7fe32415c700 1 Processor -- start 2019-12-05T06:21:41.701 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:41.689+0000 7fe32415c700 1 -- start start 2019-12-05T06:21:41.701 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:41.693+0000 7fe32415c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe31c0a7740 0x7fe31c08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:21:41.701 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:41.693+0000 7fe32415c700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe31c07e720 0x7fe31c07eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:21:41.701 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:41.693+0000 7fe32415c700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe31c09f2e0 0x7fe31c09f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:21:41.701 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:41.693+0000 7fe32415c700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe31c0a0b00 con 0x7fe31c07e720 2019-12-05T06:21:41.701 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:41.693+0000 7fe32415c700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe31c08e5b0 con 0x7fe31c0a7740 2019-12-05T06:21:41.702 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:41.693+0000 7fe32415c700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe31c08e850 con 0x7fe31c09f2e0 2019-12-05T06:21:41.702 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:41.693+0000 7fe3226f9700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe31c0a7740 0x7fe31c08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:41.702 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:41.693+0000 7fe3226f9700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe31c0a7740 0x7fe31c08df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:53824/0 (socket says 172.21.15.42:53824) 2019-12-05T06:21:41.702 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:41.693+0000 7fe3226f9700 1 -- 172.21.15.42:0/520486429 learned_addr learned my addr 172.21.15.42:0/520486429 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:21:41.702 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:41.693+0000 7fe321ef8700 1 --2- 172.21.15.42:0/520486429 >> v2:172.21.15.40:3300/0 conn(0x7fe31c07e720 0x7fe31c07eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:41.703 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:41.693+0000 7fe322efa700 1 --2- 172.21.15.42:0/520486429 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe31c09f2e0 0x7fe31c09f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:41.703 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:41.693+0000 7fe3226f9700 1 -- 172.21.15.42:0/520486429 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe31c09f2e0 msgr2=0x7fe31c09f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:21:41.703 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:41.693+0000 7fe3226f9700 1 --2- 172.21.15.42:0/520486429 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe31c09f2e0 0x7fe31c09f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:21:41.703 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:41.693+0000 7fe3226f9700 1 -- 172.21.15.42:0/520486429 >> v2:172.21.15.40:3300/0 conn(0x7fe31c07e720 msgr2=0x7fe31c07eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:21:41.703 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:41.693+0000 7fe3226f9700 1 --2- 172.21.15.42:0/520486429 >> v2:172.21.15.40:3300/0 conn(0x7fe31c07e720 0x7fe31c07eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:21:41.703 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:41.693+0000 7fe3226f9700 1 -- 172.21.15.42:0/520486429 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe31c07f0a0 con 0x7fe31c0a7740 2019-12-05T06:21:41.704 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:41.693+0000 7fe3226f9700 1 --2- 172.21.15.42:0/520486429 >> v2:172.21.15.42:3300/0 conn(0x7fe31c0a7740 0x7fe31c08df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fe31c0a0f50 tx=0x7fe31800fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:41.704 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:41.693+0000 7fe3226f9700 1 --2- 172.21.15.42:0/520486429 >> v2:172.21.15.42:3300/0 conn(0x7fe31c0a7740 0x7fe31c08df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fe31c0a0f50 tx=0x7fe31800fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:41.704 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:41.693+0000 7fe3226f9700 1 --2- 172.21.15.42:0/520486429 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe31c0a7740 0x7fe31c08df80 secure :-1 s=READY pgs=2533 cs=0 l=1 rx=0x7fe31c0a0f50 tx=0x7fe31800fdb0).ready entity=mon.1 client_cookie=f094348161b93ad8 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:21:41.704 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:41.693+0000 7fe3226f9700 1 --2- 172.21.15.42:0/520486429 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe31c0a7740 0x7fe31c08df80 secure :-1 s=THROTTLE_DONE pgs=2533 cs=0 l=1 rx=0x7fe31c0a0f50 tx=0x7fe31800fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:41.704 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:41.693+0000 7fe3216f7700 1 -- 172.21.15.42:0/520486429 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe318008c90 con 0x7fe31c0a7740 2019-12-05T06:21:41.704 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:41.693+0000 7fe3226f9700 1 --2- 172.21.15.42:0/520486429 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe31c0a7740 0x7fe31c08df80 secure :-1 s=THROTTLE_DONE pgs=2533 cs=0 l=1 rx=0x7fe31c0a0f50 tx=0x7fe31800fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:41.705 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:41.693+0000 7fe3216f7700 1 -- 172.21.15.42:0/520486429 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3180088b0 con 0x7fe31c0a7740 2019-12-05T06:21:41.705 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:41.693+0000 7fe3226f9700 1 --2- 172.21.15.42:0/520486429 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe31c0a7740 0x7fe31c08df80 secure :-1 s=THROTTLE_DONE pgs=2533 cs=0 l=1 rx=0x7fe31c0a0f50 tx=0x7fe31800fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:41.705 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:41.693+0000 7fe3216f7700 1 -- 172.21.15.42:0/520486429 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe318011b50 con 0x7fe31c0a7740 2019-12-05T06:21:41.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:41.693+0000 7fe32415c700 1 -- 172.21.15.42:0/520486429 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe31c0a7740 msgr2=0x7fe31c08df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:41.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:41.693+0000 7fe32415c700 1 --2- 172.21.15.42:0/520486429 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe31c0a7740 0x7fe31c08df80 secure :-1 s=READY pgs=2533 cs=0 l=1 rx=0x7fe31c0a0f50 tx=0x7fe31800fdb0).stop 2019-12-05T06:21:41.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:41.693+0000 7fe32415c700 1 -- 172.21.15.42:0/520486429 shutdown_connections 2019-12-05T06:21:41.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:41.693+0000 7fe32415c700 1 --2- 172.21.15.42:0/520486429 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe31c09f2e0 0x7fe31c09f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:21:41.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:41.693+0000 7fe32415c700 1 --2- 172.21.15.42:0/520486429 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe31c0a7740 0x7fe31c08df80 unknown :-1 s=CLOSED pgs=2533 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:41.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:41.693+0000 7fe32415c700 1 --2- 172.21.15.42:0/520486429 >> v2:172.21.15.40:3300/0 conn(0x7fe31c07e720 0x7fe31c07eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:21:41.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:41.693+0000 7fe32415c700 1 -- 172.21.15.42:0/520486429 shutdown_connections 2019-12-05T06:21:41.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:41.693+0000 7fe32415c700 1 -- 172.21.15.42:0/520486429 wait complete. 2019-12-05T06:21:41.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:41.693+0000 7fe32415c700 1 -- 172.21.15.42:0/520486429 >> 172.21.15.42:0/520486429 conn(0x7fe31c0aabe0 msgr2=0x7fe31c0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:21:41.711 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:41.697+0000 7fe32415c700 1 Processor -- start 2019-12-05T06:21:41.711 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:41.697+0000 7fe32415c700 1 -- start start 2019-12-05T06:21:41.712 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:41.697+0000 7fe32415c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe31c0a3a60 0x7fe31c131570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:21:41.712 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:41.697+0000 7fe32415c700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe31c07c8e0 0x7fe31c12e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:21:41.712 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:41.697+0000 7fe32415c700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe31c1323c0 con 0x7fe31c07cd10 2019-12-05T06:21:41.712 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:41.697+0000 7fe32415c700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe31c131e80 con 0x7fe31c07c8e0 2019-12-05T06:21:41.712 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:41.697+0000 7fe32415c700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe31c132120 con 0x7fe31c0a3a60 2019-12-05T06:21:41.712 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:41.697+0000 7fe3226f9700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe31c0a3a60 0x7fe31c131570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:41.713 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:41.697+0000 7fe322efa700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fe31c07cd10 0x7fe31c1191a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:53872/0 (socket says 172.21.15.42:53872) 2019-12-05T06:21:41.713 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:41.697+0000 7fe322efa700 1 -- 172.21.15.42:0/2800775621 learned_addr learned my addr 172.21.15.42:0/2800775621 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:21:41.713 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:41.697+0000 7fe321ef8700 1 --2- 172.21.15.42:0/2800775621 >> v2:172.21.15.40:3300/0 conn(0x7fe31c07c8e0 0x7fe31c12e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:41.713 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:41.697+0000 7fe3226f9700 1 -- 172.21.15.42:0/2800775621 >> v2:172.21.15.40:3300/0 conn(0x7fe31c07c8e0 msgr2=0x7fe31c12e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:41.713 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:41.697+0000 7fe3226f9700 1 --2- 172.21.15.42:0/2800775621 >> v2:172.21.15.40:3300/0 conn(0x7fe31c07c8e0 0x7fe31c12e050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:41.713 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:41.697+0000 7fe3226f9700 1 -- 172.21.15.42:0/2800775621 >> v1:172.21.15.42:6789/0 conn(0x7fe31c07cd10 legacy=0x7fe31c1191a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:41.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:41.697+0000 7fe312ffd700 1 -- 172.21.15.42:0/2800775621 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3837460847 0 0) 0x7fe31c1323c0 con 0x7fe31c07cd10 2019-12-05T06:21:41.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:41.697+0000 7fe3226f9700 1 -- 172.21.15.42:0/2800775621 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe31c145520 con 0x7fe31c0a3a60 2019-12-05T06:21:41.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:41.697+0000 7fe3226f9700 1 --2- 172.21.15.42:0/2800775621 >> v2:172.21.15.42:3300/0 conn(0x7fe31c0a3a60 0x7fe31c131570 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fe31c0a0f50 tx=0x7fe318008b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:41.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:41.697+0000 7fe3226f9700 1 --2- 172.21.15.42:0/2800775621 >> v2:172.21.15.42:3300/0 conn(0x7fe31c0a3a60 0x7fe31c131570 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fe31c0a0f50 tx=0x7fe318008b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:41.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:41.697+0000 7fe3226f9700 1 --2- 172.21.15.42:0/2800775621 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe31c0a3a60 0x7fe31c131570 secure :-1 s=READY pgs=2535 cs=0 l=1 rx=0x7fe31c0a0f50 tx=0x7fe318008b40).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:21:41.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:41.697+0000 7fe3226f9700 1 --2- 172.21.15.42:0/2800775621 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe31c0a3a60 0x7fe31c131570 secure :-1 s=THROTTLE_DONE pgs=2535 cs=0 l=1 rx=0x7fe31c0a0f50 tx=0x7fe318008b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:41.715 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:41.697+0000 7fe312ffd700 1 -- 172.21.15.42:0/2800775621 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe31800f490 con 0x7fe31c0a3a60 2019-12-05T06:21:41.715 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:41.697+0000 7fe3226f9700 1 --2- 172.21.15.42:0/2800775621 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe31c0a3a60 0x7fe31c131570 secure :-1 s=THROTTLE_DONE pgs=2535 cs=0 l=1 rx=0x7fe31c0a0f50 tx=0x7fe318008b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:41.715 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:41.697+0000 7fe312ffd700 1 -- 172.21.15.42:0/2800775621 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe318011770 con 0x7fe31c0a3a60 2019-12-05T06:21:41.715 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:41.697+0000 7fe32415c700 1 -- 172.21.15.42:0/2800775621 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe31c131ac0 con 0x7fe31c0a3a60 2019-12-05T06:21:41.715 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:41.697+0000 7fe3226f9700 1 --2- 172.21.15.42:0/2800775621 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe31c0a3a60 0x7fe31c131570 secure :-1 s=THROTTLE_DONE pgs=2535 cs=0 l=1 rx=0x7fe31c0a0f50 tx=0x7fe318008b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:41.716 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:41.697+0000 7fe312ffd700 1 -- 172.21.15.42:0/2800775621 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe318005730 con 0x7fe31c0a3a60 2019-12-05T06:21:41.716 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:41.697+0000 7fe32415c700 1 -- 172.21.15.42:0/2800775621 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe31c12e740 con 0x7fe31c0a3a60 2019-12-05T06:21:41.716 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:41.701+0000 7fe3226f9700 1 --2- 172.21.15.42:0/2800775621 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe31c0a3a60 0x7fe31c131570 secure :-1 s=THROTTLE_DONE pgs=2535 cs=0 l=1 rx=0x7fe31c0a0f50 tx=0x7fe318008b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:41.716 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:41.701+0000 7fe312ffd700 1 -- 172.21.15.42:0/2800775621 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fe318005a10 con 0x7fe31c0a3a60 2019-12-05T06:21:41.716 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:41.701+0000 7fe3226f9700 1 --2- 172.21.15.42:0/2800775621 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe31c0a3a60 0x7fe31c131570 secure :-1 s=THROTTLE_DONE pgs=2535 cs=0 l=1 rx=0x7fe31c0a0f50 tx=0x7fe318008b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:41.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:41.701+0000 7fe312ffd700 1 --2- 172.21.15.42:0/2800775621 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe30403ac30 0x7fe30403d140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:21:41.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:41.701+0000 7fe312ffd700 1 -- 172.21.15.42:0/2800775621 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 10675+0+0 (secure 0 0 0) 0x7fe31808f340 con 0x7fe31c0a3a60 2019-12-05T06:21:41.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:41.701+0000 7fe321ef8700 1 --2- 172.21.15.42:0/2800775621 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe30403ac30 0x7fe30403d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:41.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:41.701+0000 7fe321ef8700 1 --2- 172.21.15.42:0/2800775621 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe30403ac30 0x7fe30403d140 crc :-1 s=READY pgs=1056 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:21:41.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:41.705+0000 7fe32415c700 1 -- 172.21.15.42:0/2800775621 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe31c05d110 con 0x7fe31c0a3a60 2019-12-05T06:21:41.718 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:41.713+0000 7fe3226f9700 1 --2- 172.21.15.42:0/2800775621 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe31c0a3a60 0x7fe31c131570 secure :-1 s=THROTTLE_DONE pgs=2535 cs=0 l=1 rx=0x7fe31c0a0f50 tx=0x7fe318008b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:41.718 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:41.713+0000 7fe312ffd700 1 -- 172.21.15.42:0/2800775621 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fe318005d20 con 0x7fe31c0a3a60 2019-12-05T06:21:41.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:41.889+0000 7fe32415c700 1 -- 172.21.15.42:0/2800775621 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe31c05b8c0 con 0x7fe31c0a3a60 2019-12-05T06:21:41.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:41.889+0000 7fe3226f9700 1 --2- 172.21.15.42:0/2800775621 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe31c0a3a60 0x7fe31c131570 secure :-1 s=THROTTLE_DONE pgs=2535 cs=0 l=1 rx=0x7fe31c0a0f50 tx=0x7fe318008b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:41.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:41.889+0000 7fe312ffd700 1 -- 172.21.15.42:0/2800775621 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 302 v302) v1 ==== 95+0+1776 (secure 0 0 0) 0x7fe3180771d0 con 0x7fe31c0a3a60 2019-12-05T06:21:41.896 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 302 2019-12-05T06:21:41.896 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:21:41.897 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":302,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":302,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:21:41.099460+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":103,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10710},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10710":{"gid":10710,"name":"b","rank":0,"incarnation":299,"state":"up:active","state_seq":29,"addr":"172.21.15.40:6833/526579407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":526579407},{"type":"v1","addr":"172.21.15.40:6833","nonce":526579407}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:21:41.899 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:41.893+0000 7fe32415c700 1 -- 172.21.15.42:0/2800775621 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe30403ac30 msgr2=0x7fe30403d140 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:41.899 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:41.893+0000 7fe32415c700 1 --2- 172.21.15.42:0/2800775621 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe30403ac30 0x7fe30403d140 crc :-1 s=READY pgs=1056 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:41.900 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:41.893+0000 7fe32415c700 1 -- 172.21.15.42:0/2800775621 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe31c0a3a60 msgr2=0x7fe31c131570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:41.900 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:41.893+0000 7fe32415c700 1 --2- 172.21.15.42:0/2800775621 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe31c0a3a60 0x7fe31c131570 secure :-1 s=READY pgs=2535 cs=0 l=1 rx=0x7fe31c0a0f50 tx=0x7fe318008b40).stop 2019-12-05T06:21:41.900 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:41.893+0000 7fe32415c700 1 -- 172.21.15.42:0/2800775621 shutdown_connections 2019-12-05T06:21:41.900 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:41.893+0000 7fe32415c700 1 --2- 172.21.15.42:0/2800775621 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe30403ac30 0x7fe30403d140 unknown :-1 s=CLOSED pgs=1056 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:41.900 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:41.893+0000 7fe32415c700 1 --2- 172.21.15.42:0/2800775621 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe31c0a3a60 0x7fe31c131570 unknown :-1 s=CLOSED pgs=2535 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:41.901 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:41.893+0000 7fe32415c700 1 --2- 172.21.15.42:0/2800775621 >> v2:172.21.15.40:3300/0 conn(0x7fe31c07c8e0 0x7fe31c12e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:41.901 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:41.893+0000 7fe32415c700 1 -- 172.21.15.42:0/2800775621 shutdown_connections 2019-12-05T06:21:41.901 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:41.893+0000 7fe32415c700 1 -- 172.21.15.42:0/2800775621 wait complete. 2019-12-05T06:21:41.901 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:41.893+0000 7fe32415c700 1 -- 172.21.15.42:0/2800775621 >> 172.21.15.42:0/2800775621 conn(0x7fe31c0a8a80 msgr2=0x7fe31c11a940 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:21:41.912 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b has gained rank=0, replacing gid=10711 2019-12-05T06:21:41.913 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 7 secs before reviving mds.a 2019-12-05T06:21:48.914 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.a 2019-12-05T06:21:48.914 INFO:tasks.ceph.mds.a:Restarting daemon 2019-12-05T06:21:48.914 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2019-12-05T06:21:48.917 INFO:tasks.ceph.mds.a:Started 2019-12-05T06:21:48.918 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:21:48.965 INFO:tasks.ceph.mds.a.smithi042.stdout:starting mds.a at 2019-12-05T06:21:49.005 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:48.997+0000 7ff756104700 1 Processor -- start 2019-12-05T06:21:49.005 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:48.997+0000 7ff756104700 1 -- start start 2019-12-05T06:21:49.005 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:48.997+0000 7ff756104700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff75008df70 0x7ff75009f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:21:49.006 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:48.997+0000 7ff756104700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff75009fc50 con 0x7ff75009f820 2019-12-05T06:21:49.006 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:48.997+0000 7ff756104700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7500a0700 con 0x7ff7500a7730 2019-12-05T06:21:49.006 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:48.997+0000 7ff756104700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7500a0af0 con 0x7ff75008df70 2019-12-05T06:21:49.006 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:48.997+0000 7ff74f7fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff75008df70 0x7ff75009f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:49.006 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:48.997+0000 7ff74f7fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff75008df70 0x7ff75009f2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:53888/0 (socket says 172.21.15.42:53888) 2019-12-05T06:21:49.006 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:48.997+0000 7ff74f7fe700 1 -- 172.21.15.42:0/2584327035 learned_addr learned my addr 172.21.15.42:0/2584327035 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:21:49.006 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:48.997+0000 7ff74f7fe700 1 -- 172.21.15.42:0/2584327035 >> v1:172.21.15.42:6789/0 conn(0x7ff7500a7730 legacy=0x7ff75007e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:49.007 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:48.997+0000 7ff74f7fe700 1 -- 172.21.15.42:0/2584327035 >> v1:172.21.15.40:6789/0 conn(0x7ff75009f820 legacy=0x7ff75014efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:49.007 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:48.997+0000 7ff74f7fe700 1 -- 172.21.15.42:0/2584327035 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff75007ef10 con 0x7ff75008df70 2019-12-05T06:21:49.007 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:48.997+0000 7ff74f7fe700 1 --2- 172.21.15.42:0/2584327035 >> v2:172.21.15.42:3300/0 conn(0x7ff75008df70 0x7ff75009f2d0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7ff744008790 tx=0x7ff74400fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:49.007 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:48.997+0000 7ff74f7fe700 1 --2- 172.21.15.42:0/2584327035 >> v2:172.21.15.42:3300/0 conn(0x7ff75008df70 0x7ff75009f2d0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7ff744008790 tx=0x7ff74400fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:49.007 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:48.997+0000 7ff74f7fe700 1 --2- 172.21.15.42:0/2584327035 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff75008df70 0x7ff75009f2d0 secure :-1 s=READY pgs=2540 cs=0 l=1 rx=0x7ff744008790 tx=0x7ff74400fcb0).ready entity=mon.1 client_cookie=2a9c87296a69fd81 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:21:49.007 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:48.997+0000 7ff74f7fe700 1 --2- 172.21.15.42:0/2584327035 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff75008df70 0x7ff75009f2d0 secure :-1 s=THROTTLE_DONE pgs=2540 cs=0 l=1 rx=0x7ff744008790 tx=0x7ff74400fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:49.007 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:48.997+0000 7ff74f7fe700 1 --2- 172.21.15.42:0/2584327035 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff75008df70 0x7ff75009f2d0 secure :-1 s=THROTTLE_DONE pgs=2540 cs=0 l=1 rx=0x7ff744008790 tx=0x7ff74400fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:49.008 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:48.997+0000 7ff74effd700 1 -- 172.21.15.42:0/2584327035 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff744011520 con 0x7ff75008df70 2019-12-05T06:21:49.008 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:48.997+0000 7ff74f7fe700 1 --2- 172.21.15.42:0/2584327035 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff75008df70 0x7ff75009f2d0 secure :-1 s=THROTTLE_DONE pgs=2540 cs=0 l=1 rx=0x7ff744008790 tx=0x7ff74400fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:49.008 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:48.997+0000 7ff74effd700 1 -- 172.21.15.42:0/2584327035 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff744008ca0 con 0x7ff75008df70 2019-12-05T06:21:49.008 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:48.997+0000 7ff74effd700 1 -- 172.21.15.42:0/2584327035 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff744011b00 con 0x7ff75008df70 2019-12-05T06:21:49.008 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:48.997+0000 7ff756104700 1 -- 172.21.15.42:0/2584327035 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff75008df70 msgr2=0x7ff75009f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:49.008 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:48.997+0000 7ff756104700 1 --2- 172.21.15.42:0/2584327035 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff75008df70 0x7ff75009f2d0 secure :-1 s=READY pgs=2540 cs=0 l=1 rx=0x7ff744008790 tx=0x7ff74400fcb0).stop 2019-12-05T06:21:49.009 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:49.001+0000 7ff756104700 1 -- 172.21.15.42:0/2584327035 shutdown_connections 2019-12-05T06:21:49.009 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:49.001+0000 7ff756104700 1 --2- 172.21.15.42:0/2584327035 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff75008df70 0x7ff75009f2d0 unknown :-1 s=CLOSED pgs=2540 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:49.009 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:49.001+0000 7ff756104700 1 -- 172.21.15.42:0/2584327035 shutdown_connections 2019-12-05T06:21:49.009 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:49.001+0000 7ff756104700 1 -- 172.21.15.42:0/2584327035 wait complete. 2019-12-05T06:21:49.009 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:49.001+0000 7ff756104700 1 -- 172.21.15.42:0/2584327035 >> 172.21.15.42:0/2584327035 conn(0x7ff7500aabd0 msgr2=0x7ff7500a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:21:49.013 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:49.001+0000 7ff756104700 1 Processor -- start 2019-12-05T06:21:49.013 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:49.001+0000 7ff756104700 1 -- start start 2019-12-05T06:21:49.013 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:49.001+0000 7ff756104700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff750125f80 0x7ff750129000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:21:49.014 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:49.001+0000 7ff756104700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff7501263b0 0x7ff750129550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:21:49.014 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:49.001+0000 7ff756104700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7501479f0 con 0x7ff750145850 2019-12-05T06:21:49.014 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:49.001+0000 7ff756104700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7501257a0 con 0x7ff7501263b0 2019-12-05T06:21:49.014 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:49.001+0000 7ff756104700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff750125a40 con 0x7ff750125f80 2019-12-05T06:21:49.015 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:49.001+0000 7ff754ea2700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7ff750145850 0x7ff750145c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:53934/0 (socket says 172.21.15.42:53934) 2019-12-05T06:21:49.015 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:49.001+0000 7ff754ea2700 1 -- 172.21.15.42:0/1394302426 learned_addr learned my addr 172.21.15.42:0/1394302426 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:21:49.015 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:49.001+0000 7ff74f7fe700 1 --2- 172.21.15.42:0/1394302426 >> v2:172.21.15.40:3300/0 conn(0x7ff7501263b0 0x7ff750129550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:49.015 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:49.001+0000 7ff74ffff700 1 --2- 172.21.15.42:0/1394302426 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff750125f80 0x7ff750129000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:49.015 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:49.001+0000 7ff74cff9700 1 -- 172.21.15.42:0/1394302426 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1278465667 0 0) 0x7ff7501479f0 con 0x7ff750145850 2019-12-05T06:21:49.015 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:49.001+0000 7ff74cff9700 1 -- 172.21.15.42:0/1394302426 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7300030b0 con 0x7ff750145850 2019-12-05T06:21:49.016 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:49.001+0000 7ff74cff9700 1 -- 172.21.15.42:0/1394302426 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 154003402 0 0) 0x7ff7300030b0 con 0x7ff750145850 2019-12-05T06:21:49.016 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:49.001+0000 7ff74cff9700 1 -- 172.21.15.42:0/1394302426 --> v1:172.21.15.42:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff7501479f0 con 0x7ff750145850 2019-12-05T06:21:49.016 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:49.001+0000 7ff74cff9700 1 -- 172.21.15.42:0/1394302426 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7ff748001ca0 con 0x7ff750145850 2019-12-05T06:21:49.016 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:49.001+0000 7ff74cff9700 1 -- 172.21.15.42:0/1394302426 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff750125f80 msgr2=0x7ff750129000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:49.016 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:49.001+0000 7ff74cff9700 1 --2- 172.21.15.42:0/1394302426 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff750125f80 0x7ff750129000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:49.016 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:49.001+0000 7ff74cff9700 1 -- 172.21.15.42:0/1394302426 >> v2:172.21.15.40:3300/0 conn(0x7ff7501263b0 msgr2=0x7ff750129550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:49.016 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:49.001+0000 7ff74cff9700 1 --2- 172.21.15.42:0/1394302426 >> v2:172.21.15.40:3300/0 conn(0x7ff7501263b0 0x7ff750129550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:49.017 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:49.001+0000 7ff74cff9700 1 -- 172.21.15.42:0/1394302426 >> v1:172.21.15.42:6789/0 conn(0x7ff750145850 legacy=0x7ff750145c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:49.017 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:49.001+0000 7ff74cff9700 1 --2- 172.21.15.42:0/1394302426 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff7300038a0 0x7ff730003cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:21:49.017 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:49.001+0000 7ff74cff9700 1 --2- 172.21.15.42:0/1394302426 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff730004420 0x7ff730004850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:21:49.017 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:49.001+0000 7ff74cff9700 1 --2- 172.21.15.42:0/1394302426 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff730004da0 0x7ff73000b5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:21:49.017 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:49.001+0000 7ff74cff9700 1 -- 172.21.15.42:0/1394302426 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7501257a0 con 0x7ff730004da0 2019-12-05T06:21:49.017 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:49.001+0000 7ff74cff9700 1 -- 172.21.15.42:0/1394302426 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff750125a40 con 0x7ff730004420 2019-12-05T06:21:49.017 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:49.001+0000 7ff74cff9700 1 -- 172.21.15.42:0/1394302426 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff73000bb30 con 0x7ff7300038a0 2019-12-05T06:21:49.018 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:49.001+0000 7ff74f7fe700 1 --2- 172.21.15.42:0/1394302426 >> v2:172.21.15.40:3300/0 conn(0x7ff7501263b0 0x7ff750129550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:21:49.018 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:49.001+0000 7ff74ffff700 1 --2- 172.21.15.42:0/1394302426 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff7300038a0 0x7ff730003cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:49.018 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:49.001+0000 7ff754ea2700 1 --2- 172.21.15.42:0/1394302426 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff730004da0 0x7ff73000b5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:49.018 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:49.001+0000 7ff74f7fe700 1 --2- 172.21.15.42:0/1394302426 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff730004420 0x7ff730004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:49.018 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:49.005+0000 7ff74ffff700 1 -- 172.21.15.42:0/1394302426 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff730004420 msgr2=0x7ff730004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:49.018 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:49.005+0000 7ff74ffff700 1 --2- 172.21.15.42:0/1394302426 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff730004420 0x7ff730004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:49.018 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:49.005+0000 7ff74ffff700 1 -- 172.21.15.42:0/1394302426 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff730004da0 msgr2=0x7ff73000b5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:49.019 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:49.005+0000 7ff74ffff700 1 --2- 172.21.15.42:0/1394302426 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff730004da0 0x7ff73000b5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:49.019 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:49.005+0000 7ff74ffff700 1 -- 172.21.15.42:0/1394302426 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff750129aa0 con 0x7ff7300038a0 2019-12-05T06:21:49.019 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:49.005+0000 7ff756104700 1 -- 172.21.15.42:0/1394302426 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff750148790 con 0x7ff7300038a0 2019-12-05T06:21:49.019 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:49.005+0000 7ff74ffff700 1 --2- 172.21.15.42:0/1394302426 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff7300038a0 0x7ff730003cd0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7ff740000b80 tx=0x7ff740010bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:49.019 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:49.005+0000 7ff756104700 1 -- 172.21.15.42:0/1394302426 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff750146480 con 0x7ff7300038a0 2019-12-05T06:21:49.019 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:49.005+0000 7ff74ffff700 1 -- 172.21.15.42:0/1394302426 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff7300038a0 msgr2=0x7ff730003cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2019-12-05T06:21:49.020 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:49.005+0000 7ff74ffff700 1 -- 172.21.15.42:0/1394302426 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff7300038a0 msgr2=0x7ff730003cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2019-12-05T06:21:49.020 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:49.005+0000 7ff74ffff700 1 --2- 172.21.15.42:0/1394302426 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff7300038a0 0x7ff730003cd0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7ff740000b80 tx=0x7ff740010bb0).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-12-05T06:21:49.020 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:49.005+0000 7ff74ffff700 1 --2- 172.21.15.42:0/1394302426 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff7300038a0 0x7ff730003cd0 secure :-1 s=START_CONNECT pgs=0 cs=0 l=1 rx=0x7ff740000b80 tx=0x7ff740010bb0)._fault waiting 0.200000 2019-12-05T06:21:49.020 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:49.013+0000 7ff756104700 1 -- 172.21.15.42:0/1394302426 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff75005d100 con 0x7ff7300038a0 2019-12-05T06:21:49.221 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:49.209+0000 7ff74ffff700 1 --2- 172.21.15.42:0/1394302426 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff7300038a0 0x7ff730003cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:49.221 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:49.209+0000 7ff74ffff700 0 --2- 172.21.15.42:0/1394302426 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff7300038a0 0x7ff730003cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2019-12-05T06:21:49.221 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:49.209+0000 7ff74ffff700 1 --2- 172.21.15.42:0/1394302426 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff7300038a0 0x7ff730003cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:49.221 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:49.209+0000 7ff754ea2700 1 -- 172.21.15.42:0/1394302426 reap_dead start 2019-12-05T06:21:49.221 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:49.209+0000 7ff74cff9700 1 -- 172.21.15.42:0/1394302426 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff7300038a0 msgr2=0x7ff730003cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2019-12-05T06:21:49.222 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:49.209+0000 7ff74cff9700 1 --2- 172.21.15.42:0/1394302426 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff7300038a0 0x7ff730003cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:49.222 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:49.209+0000 7ff74cff9700 1 --2- 172.21.15.42:0/1394302426 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff730004420 0x7ff730004850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:21:49.222 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:49.209+0000 7ff74cff9700 1 --2- 172.21.15.42:0/1394302426 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff7300095d0 0x7ff73000bde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:21:49.222 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:49.209+0000 7ff74cff9700 1 --2- 172.21.15.42:0/1394302426 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff73000c330 0x7ff73000e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:21:49.222 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:49.209+0000 7ff74cff9700 1 -- 172.21.15.42:0/1394302426 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff73000eda0 con 0x7ff730004420 2019-12-05T06:21:49.222 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:49.209+0000 7ff74cff9700 1 -- 172.21.15.42:0/1394302426 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff73000ef90 con 0x7ff73000c330 2019-12-05T06:21:49.222 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:49.209+0000 7ff74cff9700 1 -- 172.21.15.42:0/1394302426 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff73000f1d0 con 0x7ff7300095d0 2019-12-05T06:21:49.223 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:49.209+0000 7ff754ea2700 1 --2- 172.21.15.42:0/1394302426 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff73000c330 0x7ff73000e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:49.223 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:49.209+0000 7ff74ffff700 1 --2- 172.21.15.42:0/1394302426 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff730004420 0x7ff730004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:49.223 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:49.209+0000 7ff74f7fe700 1 --2- 172.21.15.42:0/1394302426 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff7300095d0 0x7ff73000bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:49.223 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:49.209+0000 7ff754ea2700 1 -- 172.21.15.42:0/1394302426 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff7300095d0 msgr2=0x7ff73000bde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:49.223 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:49.209+0000 7ff754ea2700 1 --2- 172.21.15.42:0/1394302426 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff7300095d0 0x7ff73000bde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:49.223 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:49.209+0000 7ff754ea2700 1 -- 172.21.15.42:0/1394302426 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff730004420 msgr2=0x7ff730004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:49.224 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:49.209+0000 7ff754ea2700 1 --2- 172.21.15.42:0/1394302426 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff730004420 0x7ff730004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:49.224 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:49.209+0000 7ff754ea2700 1 -- 172.21.15.42:0/1394302426 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+,osdmap=0}) v3 -- 0x7ff730003460 con 0x7ff73000c330 2019-12-05T06:21:49.224 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:49.209+0000 7ff754ea2700 1 -- 172.21.15.42:0/1394302426 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7480114e0 con 0x7ff73000c330 2019-12-05T06:21:49.224 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:49.209+0000 7ff754ea2700 1 --2- 172.21.15.42:0/1394302426 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff73000c330 0x7ff73000e770 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7ff7480019b0 tx=0x7ff748011a70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:49.224 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:49.209+0000 7ff754ea2700 1 --2- 172.21.15.42:0/1394302426 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff73000c330 0x7ff73000e770 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7ff7480019b0 tx=0x7ff748011a70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:49.224 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:49.209+0000 7ff754ea2700 1 --2- 172.21.15.42:0/1394302426 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff73000c330 0x7ff73000e770 secure :-1 s=READY pgs=2542 cs=0 l=1 rx=0x7ff7480019b0 tx=0x7ff748011a70).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:21:49.225 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:49.209+0000 7ff754ea2700 1 --2- 172.21.15.42:0/1394302426 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff73000c330 0x7ff73000e770 secure :-1 s=THROTTLE_DONE pgs=2542 cs=0 l=1 rx=0x7ff7480019b0 tx=0x7ff748011a70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:49.225 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:49.209+0000 7ff74cff9700 1 -- 172.21.15.42:0/1394302426 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff7480114e0 con 0x7ff73000c330 2019-12-05T06:21:49.225 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:49.209+0000 7ff754ea2700 1 --2- 172.21.15.42:0/1394302426 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff73000c330 0x7ff73000e770 secure :-1 s=THROTTLE_DONE pgs=2542 cs=0 l=1 rx=0x7ff7480019b0 tx=0x7ff748011a70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:49.225 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:49.209+0000 7ff74cff9700 1 -- 172.21.15.42:0/1394302426 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7480056d0 con 0x7ff73000c330 2019-12-05T06:21:49.225 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:49.209+0000 7ff754ea2700 1 --2- 172.21.15.42:0/1394302426 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff73000c330 0x7ff73000e770 secure :-1 s=THROTTLE_DONE pgs=2542 cs=0 l=1 rx=0x7ff7480019b0 tx=0x7ff748011a70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:49.225 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:49.213+0000 7ff74cff9700 1 -- 172.21.15.42:0/1394302426 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7ff748005ab0 con 0x7ff73000c330 2019-12-05T06:21:49.226 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:49.213+0000 7ff754ea2700 1 --2- 172.21.15.42:0/1394302426 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff73000c330 0x7ff73000e770 secure :-1 s=THROTTLE_DONE pgs=2542 cs=0 l=1 rx=0x7ff7480019b0 tx=0x7ff748011a70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:49.226 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:49.213+0000 7ff74cff9700 1 --2- 172.21.15.42:0/1394302426 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff7300446b0 0x7ff730046bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:21:49.226 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:49.213+0000 7ff74cff9700 1 -- 172.21.15.42:0/1394302426 <== mon.1 v2:172.21.15.42:3300/0 4 ==== osd_map(103..103 src has 1..103) v4 ==== 10675+0+0 (secure 0 0 0) 0x7ff74804b3f0 con 0x7ff73000c330 2019-12-05T06:21:49.226 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:49.213+0000 7ff74ffff700 1 --2- 172.21.15.42:0/1394302426 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff7300446b0 0x7ff730046bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:49.226 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:49.213+0000 7ff74ffff700 1 --2- 172.21.15.42:0/1394302426 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff7300446b0 0x7ff730046bc0 crc :-1 s=READY pgs=1057 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:21:49.227 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:49.217+0000 7ff754ea2700 1 --2- 172.21.15.42:0/1394302426 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff73000c330 0x7ff73000e770 secure :-1 s=THROTTLE_DONE pgs=2542 cs=0 l=1 rx=0x7ff7480019b0 tx=0x7ff748011a70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:49.227 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:49.217+0000 7ff74cff9700 1 -- 172.21.15.42:0/1394302426 <== mon.1 v2:172.21.15.42:3300/0 5 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7ff748032470 con 0x7ff73000c330 2019-12-05T06:21:49.412 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:49.409+0000 7ff756104700 1 -- 172.21.15.42:0/1394302426 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff75005b8b0 con 0x7ff73000c330 2019-12-05T06:21:49.413 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:49.409+0000 7ff754ea2700 1 --2- 172.21.15.42:0/1394302426 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff73000c330 0x7ff73000e770 secure :-1 s=THROTTLE_DONE pgs=2542 cs=0 l=1 rx=0x7ff7480019b0 tx=0x7ff748011a70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:49.413 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:49.409+0000 7ff74cff9700 1 -- 172.21.15.42:0/1394302426 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 303 v303) v1 ==== 95+0+2114 (secure 0 0 0) 0x7ff7480326d0 con 0x7ff73000c330 2019-12-05T06:21:49.413 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 303 2019-12-05T06:21:49.414 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:21:49.414 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":303,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":10993,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6835/3964426552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3964426552},{"type":"v1","addr":"172.21.15.42:6835","nonce":3964426552}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":303}],"filesystems":[{"mdsmap":{"epoch":302,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:21:41.099460+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":103,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10710},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10710":{"gid":10710,"name":"b","rank":0,"incarnation":299,"state":"up:active","state_seq":29,"addr":"172.21.15.40:6833/526579407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":526579407},{"type":"v1","addr":"172.21.15.40:6833","nonce":526579407}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:21:49.416 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:49.409+0000 7ff756104700 1 -- 172.21.15.42:0/1394302426 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff7300446b0 msgr2=0x7ff730046bc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:49.416 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:49.409+0000 7ff756104700 1 --2- 172.21.15.42:0/1394302426 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff7300446b0 0x7ff730046bc0 crc :-1 s=READY pgs=1057 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:49.416 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:49.409+0000 7ff756104700 1 -- 172.21.15.42:0/1394302426 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff73000c330 msgr2=0x7ff73000e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:49.416 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:49.409+0000 7ff756104700 1 --2- 172.21.15.42:0/1394302426 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff73000c330 0x7ff73000e770 secure :-1 s=READY pgs=2542 cs=0 l=1 rx=0x7ff7480019b0 tx=0x7ff748011a70).stop 2019-12-05T06:21:49.416 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:49.409+0000 7ff756104700 1 -- 172.21.15.42:0/1394302426 shutdown_connections 2019-12-05T06:21:49.416 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:49.409+0000 7ff756104700 1 --2- 172.21.15.42:0/1394302426 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff7300446b0 0x7ff730046bc0 unknown :-1 s=CLOSED pgs=1057 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:49.417 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:49.409+0000 7ff756104700 1 --2- 172.21.15.42:0/1394302426 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff73000c330 0x7ff73000e770 unknown :-1 s=CLOSED pgs=2542 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:49.417 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:49.409+0000 7ff756104700 1 --2- 172.21.15.42:0/1394302426 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff7300095d0 0x7ff73000bde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:49.417 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:49.409+0000 7ff756104700 1 --2- 172.21.15.42:0/1394302426 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff730004420 0x7ff730004850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:49.417 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:49.409+0000 7ff756104700 1 -- 172.21.15.42:0/1394302426 shutdown_connections 2019-12-05T06:21:49.418 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:49.409+0000 7ff756104700 1 -- 172.21.15.42:0/1394302426 wait complete. 2019-12-05T06:21:49.418 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:21:49.413+0000 7ff756104700 1 -- 172.21.15.42:0/1394302426 >> 172.21.15.42:0/1394302426 conn(0x7ff75005a180 msgr2=0x7ff75007e4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:21:49.426 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a reported in up:standby state 2019-12-05T06:21:49.426 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 99 secs before thrashing 2019-12-05T06:23:13.272 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:23:13.277 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:23:28.427 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:23:28.534 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.520+0000 7f3844aea700 1 Processor -- start 2019-12-05T06:23:28.535 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.520+0000 7f3844aea700 1 -- start start 2019-12-05T06:23:28.535 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.524+0000 7f3844aea700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f38400a7740 0x7f384008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:23:28.535 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.524+0000 7f3844aea700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f384009fae0 0x7f384007eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:23:28.535 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.524+0000 7f3844aea700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38400a0710 con 0x7f384007e720 2019-12-05T06:23:28.535 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.524+0000 7f3844aea700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38400a0b00 con 0x7f384009fae0 2019-12-05T06:23:28.535 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.524+0000 7f3844aea700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f384008e5b0 con 0x7f38400a7740 2019-12-05T06:23:28.536 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.524+0000 7f383e59c700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f384007e720 0x7f384009f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:54348/0 (socket says 172.21.15.42:54348) 2019-12-05T06:23:28.536 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.524+0000 7f383e59c700 1 -- 172.21.15.42:0/452389584 learned_addr learned my addr 172.21.15.42:0/452389584 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:23:28.536 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.524+0000 7f383f59e700 1 --2- 172.21.15.42:0/452389584 >> v2:172.21.15.40:3300/0 conn(0x7f384009fae0 0x7f384007eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:28.536 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.524+0000 7f383ed9d700 1 --2- 172.21.15.42:0/452389584 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f38400a7740 0x7f384008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:28.536 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.524+0000 7f383dd9b700 1 -- 172.21.15.42:0/452389584 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1574863455 0 0) 0x7f38400a0710 con 0x7f384007e720 2019-12-05T06:23:28.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.524+0000 7f383dd9b700 1 -- 172.21.15.42:0/452389584 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38280030b0 con 0x7f384007e720 2019-12-05T06:23:28.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.524+0000 7f383ed9d700 1 -- 172.21.15.42:0/452389584 >> v2:172.21.15.40:3300/0 conn(0x7f384009fae0 msgr2=0x7f384007eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:23:28.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.524+0000 7f383ed9d700 1 --2- 172.21.15.42:0/452389584 >> v2:172.21.15.40:3300/0 conn(0x7f384009fae0 0x7f384007eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:23:28.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.524+0000 7f383ed9d700 1 -- 172.21.15.42:0/452389584 >> v1:172.21.15.42:6789/0 conn(0x7f384007e720 legacy=0x7f384009f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:28.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.524+0000 7f383ed9d700 1 -- 172.21.15.42:0/452389584 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f384007f0a0 con 0x7f38400a7740 2019-12-05T06:23:28.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.524+0000 7f383ed9d700 1 --2- 172.21.15.42:0/452389584 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f38400a7740 0x7f384008df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f3830008790 tx=0x7f383000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:28.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.524+0000 7f383ed9d700 1 --2- 172.21.15.42:0/452389584 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f38400a7740 0x7f384008df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f3830008790 tx=0x7f383000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:28.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.524+0000 7f383ed9d700 1 --2- 172.21.15.42:0/452389584 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f38400a7740 0x7f384008df80 secure :-1 s=READY pgs=550 cs=0 l=1 rx=0x7f3830008790 tx=0x7f383000fc70).ready entity=mon.2 client_cookie=b32eece4cecbeaf server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:23:28.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.524+0000 7f383ed9d700 1 -- 172.21.15.42:0/452389584 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f38400a7740 msgr2=0x7f384008df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2019-12-05T06:23:28.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.524+0000 7f383ed9d700 1 -- 172.21.15.42:0/452389584 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f38400a7740 msgr2=0x7f384008df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2019-12-05T06:23:28.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.524+0000 7f383ed9d700 1 --2- 172.21.15.42:0/452389584 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f38400a7740 0x7f384008df80 secure :-1 s=READY pgs=550 cs=0 l=1 rx=0x7f3830008790 tx=0x7f383000fc70).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-12-05T06:23:28.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.524+0000 7f383ed9d700 1 --2- 172.21.15.42:0/452389584 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f38400a7740 0x7f384008df80 secure :-1 s=READY pgs=550 cs=0 l=1 rx=0x7f3830008790 tx=0x7f383000fc70).stop 2019-12-05T06:23:28.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.524+0000 7f383dd9b700 1 -- 172.21.15.42:0/452389584 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f38400a7740 msgr2=0x7f384008df80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2019-12-05T06:23:28.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.524+0000 7f383dd9b700 1 --2- 172.21.15.42:0/452389584 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f38400a7740 0x7f384008df80 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:28.539 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.524+0000 7f383dd9b700 1 --2- 172.21.15.42:0/452389584 >> v2:172.21.15.42:3300/0 conn(0x7f3828003420 0x7f3828003880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:23:28.539 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.524+0000 7f383dd9b700 1 -- 172.21.15.42:0/452389584 --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38400a0710 con 0x7f3828003ee0 2019-12-05T06:23:28.539 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.524+0000 7f383dd9b700 1 -- 172.21.15.42:0/452389584 --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3828004e10 con 0x7f3828003420 2019-12-05T06:23:28.539 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.524+0000 7f383ed9d700 1 --2- 172.21.15.42:0/452389584 >> v2:172.21.15.42:3300/0 conn(0x7f3828003420 0x7f3828003880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:28.539 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.524+0000 7f383ed9d700 1 -- 172.21.15.42:0/452389584 >> v1:172.21.15.40:6789/0 conn(0x7f3828003ee0 legacy=0x7f3828004440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:28.539 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.528+0000 7f383ed9d700 1 -- 172.21.15.42:0/452389584 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38280050c0 con 0x7f3828003420 2019-12-05T06:23:28.539 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.528+0000 7f383ed9d700 1 --2- 172.21.15.42:0/452389584 >> v2:172.21.15.42:3300/0 conn(0x7f3828003420 0x7f3828003880 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f3830008790 tx=0x7f383000f500).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:28.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.528+0000 7f383ed9d700 1 --2- 172.21.15.42:0/452389584 >> v2:172.21.15.42:3300/0 conn(0x7f3828003420 0x7f3828003880 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f3830008790 tx=0x7f383000f500).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:28.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.528+0000 7f383ed9d700 1 --2- 172.21.15.42:0/452389584 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3828003420 0x7f3828003880 secure :-1 s=READY pgs=2547 cs=0 l=1 rx=0x7f3830008790 tx=0x7f383000f500).ready entity=mon.1 client_cookie=d0712812283a3180 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:23:28.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.528+0000 7f383ed9d700 1 --2- 172.21.15.42:0/452389584 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3828003420 0x7f3828003880 secure :-1 s=THROTTLE_DONE pgs=2547 cs=0 l=1 rx=0x7f3830008790 tx=0x7f383000f500).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:28.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.528+0000 7f383dd9b700 1 -- 172.21.15.42:0/452389584 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3830008ca0 con 0x7f3828003420 2019-12-05T06:23:28.541 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.528+0000 7f383ed9d700 1 --2- 172.21.15.42:0/452389584 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3828003420 0x7f3828003880 secure :-1 s=THROTTLE_DONE pgs=2547 cs=0 l=1 rx=0x7f3830008790 tx=0x7f383000f500).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:28.541 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.528+0000 7f383dd9b700 1 -- 172.21.15.42:0/452389584 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38300088c0 con 0x7f3828003420 2019-12-05T06:23:28.541 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.528+0000 7f383ed9d700 1 --2- 172.21.15.42:0/452389584 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3828003420 0x7f3828003880 secure :-1 s=THROTTLE_DONE pgs=2547 cs=0 l=1 rx=0x7f3830008790 tx=0x7f383000f500).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:28.541 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.528+0000 7f383dd9b700 1 -- 172.21.15.42:0/452389584 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3830011a50 con 0x7f3828003420 2019-12-05T06:23:28.541 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.528+0000 7f3844aea700 1 -- 172.21.15.42:0/452389584 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3828003420 msgr2=0x7f3828003880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:28.541 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.528+0000 7f3844aea700 1 --2- 172.21.15.42:0/452389584 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3828003420 0x7f3828003880 secure :-1 s=READY pgs=2547 cs=0 l=1 rx=0x7f3830008790 tx=0x7f383000f500).stop 2019-12-05T06:23:28.541 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.528+0000 7f383f59e700 1 -- 172.21.15.42:0/452389584 reap_dead start 2019-12-05T06:23:28.542 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.528+0000 7f3844aea700 1 -- 172.21.15.42:0/452389584 shutdown_connections 2019-12-05T06:23:28.542 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.528+0000 7f3844aea700 1 --2- 172.21.15.42:0/452389584 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3828003420 0x7f3828003880 unknown :-1 s=CLOSED pgs=2547 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:28.542 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.528+0000 7f3844aea700 1 -- 172.21.15.42:0/452389584 shutdown_connections 2019-12-05T06:23:28.542 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.528+0000 7f3844aea700 1 -- 172.21.15.42:0/452389584 wait complete. 2019-12-05T06:23:28.542 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.528+0000 7f3844aea700 1 -- 172.21.15.42:0/452389584 >> 172.21.15.42:0/452389584 conn(0x7f38400aabe0 msgr2=0x7f38400a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:23:28.545 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.532+0000 7f3844aea700 1 Processor -- start 2019-12-05T06:23:28.545 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.532+0000 7f3844aea700 1 -- start start 2019-12-05T06:23:28.545 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.532+0000 7f3844aea700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f384012e2a0 0x7f3840118fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:23:28.546 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.532+0000 7f3844aea700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f384012e6d0 con 0x7f3840119db0 2019-12-05T06:23:28.546 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.532+0000 7f3844aea700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f384011af80 con 0x7f38400a1b30 2019-12-05T06:23:28.546 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.532+0000 7f3844aea700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38401187f0 con 0x7f384012e2a0 2019-12-05T06:23:28.546 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.532+0000 7f383ed9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f384012e2a0 0x7f3840118fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:28.546 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.532+0000 7f383e59c700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f38400a1b30 0x7f3840131860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:54360/0 (socket says 172.21.15.42:54360) 2019-12-05T06:23:28.546 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.532+0000 7f383e59c700 1 -- 172.21.15.42:0/3908299621 learned_addr learned my addr 172.21.15.42:0/3908299621 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:23:28.547 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.532+0000 7f383ed9d700 1 -- 172.21.15.42:0/3908299621 >> v1:172.21.15.42:6789/0 conn(0x7f38400a1b30 legacy=0x7f3840131860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:28.547 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.532+0000 7f382f7fe700 1 -- 172.21.15.42:0/3908299621 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3095985190 0 0) 0x7f384011af80 con 0x7f38400a1b30 2019-12-05T06:23:28.547 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.532+0000 7f383ed9d700 1 -- 172.21.15.42:0/3908299621 >> v1:172.21.15.40:6789/0 conn(0x7f3840119db0 legacy=0x7f3840132060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:28.547 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.532+0000 7f383ed9d700 1 -- 172.21.15.42:0/3908299621 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38400a1f60 con 0x7f384012e2a0 2019-12-05T06:23:28.547 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.532+0000 7f383ed9d700 1 --2- 172.21.15.42:0/3908299621 >> v2:172.21.15.42:3300/0 conn(0x7f384012e2a0 0x7f3840118fd0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f38380011d0 tx=0x7f3838007e50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:28.547 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.532+0000 7f383ed9d700 1 --2- 172.21.15.42:0/3908299621 >> v2:172.21.15.42:3300/0 conn(0x7f384012e2a0 0x7f3840118fd0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f38380011d0 tx=0x7f3838007e50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:28.547 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.532+0000 7f383ed9d700 1 --2- 172.21.15.42:0/3908299621 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f384012e2a0 0x7f3840118fd0 secure :-1 s=READY pgs=2549 cs=0 l=1 rx=0x7f38380011d0 tx=0x7f3838007e50).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:23:28.548 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.532+0000 7f383ed9d700 1 --2- 172.21.15.42:0/3908299621 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f384012e2a0 0x7f3840118fd0 secure :-1 s=THROTTLE_DONE pgs=2549 cs=0 l=1 rx=0x7f38380011d0 tx=0x7f3838007e50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:28.548 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.532+0000 7f382f7fe700 1 -- 172.21.15.42:0/3908299621 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3830005540 con 0x7f384012e2a0 2019-12-05T06:23:28.548 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.532+0000 7f383ed9d700 1 --2- 172.21.15.42:0/3908299621 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f384012e2a0 0x7f3840118fd0 secure :-1 s=THROTTLE_DONE pgs=2549 cs=0 l=1 rx=0x7f38380011d0 tx=0x7f3838007e50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:28.548 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.532+0000 7f383ed9d700 1 --2- 172.21.15.42:0/3908299621 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f384012e2a0 0x7f3840118fd0 secure :-1 s=THROTTLE_DONE pgs=2549 cs=0 l=1 rx=0x7f38380011d0 tx=0x7f3838007e50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:28.548 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.532+0000 7f382f7fe700 1 -- 172.21.15.42:0/3908299621 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3830011670 con 0x7f384012e2a0 2019-12-05T06:23:28.548 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.532+0000 7f3844aea700 1 -- 172.21.15.42:0/3908299621 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f384011a1e0 con 0x7f384012e2a0 2019-12-05T06:23:28.548 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.532+0000 7f382f7fe700 1 -- 172.21.15.42:0/3908299621 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3830005b20 con 0x7f384012e2a0 2019-12-05T06:23:28.549 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.532+0000 7f3844aea700 1 -- 172.21.15.42:0/3908299621 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f384013f150 con 0x7f384012e2a0 2019-12-05T06:23:28.549 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.536+0000 7f383ed9d700 1 --2- 172.21.15.42:0/3908299621 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f384012e2a0 0x7f3840118fd0 secure :-1 s=THROTTLE_DONE pgs=2549 cs=0 l=1 rx=0x7f38380011d0 tx=0x7f3838007e50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:28.549 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.536+0000 7f382f7fe700 1 -- 172.21.15.42:0/3908299621 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f38300144a0 con 0x7f384012e2a0 2019-12-05T06:23:28.549 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.536+0000 7f383ed9d700 1 --2- 172.21.15.42:0/3908299621 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f384012e2a0 0x7f3840118fd0 secure :-1 s=THROTTLE_DONE pgs=2549 cs=0 l=1 rx=0x7f38380011d0 tx=0x7f3838007e50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:28.549 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.536+0000 7f382f7fe700 1 --2- 172.21.15.42:0/3908299621 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f382403abb0 0x7f382403d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:23:28.549 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.536+0000 7f382f7fe700 1 -- 172.21.15.42:0/3908299621 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 10675+0+0 (secure 0 0 0) 0x7f383004c420 con 0x7f384012e2a0 2019-12-05T06:23:28.549 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.536+0000 7f383e59c700 1 --2- 172.21.15.42:0/3908299621 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f382403abb0 0x7f382403d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:28.550 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.536+0000 7f3844aea700 1 -- 172.21.15.42:0/3908299621 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f384005d110 con 0x7f384012e2a0 2019-12-05T06:23:28.550 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.536+0000 7f383e59c700 1 --2- 172.21.15.42:0/3908299621 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f382403abb0 0x7f382403d0c0 crc :-1 s=READY pgs=1060 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:23:28.550 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.540+0000 7f383ed9d700 1 --2- 172.21.15.42:0/3908299621 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f384012e2a0 0x7f3840118fd0 secure :-1 s=THROTTLE_DONE pgs=2549 cs=0 l=1 rx=0x7f38380011d0 tx=0x7f3838007e50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:28.551 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.540+0000 7f382f7fe700 1 -- 172.21.15.42:0/3908299621 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f3830005dc0 con 0x7f384012e2a0 2019-12-05T06:23:28.747 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.740+0000 7f3844aea700 1 -- 172.21.15.42:0/3908299621 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f38400a04f0 con 0x7f384012e2a0 2019-12-05T06:23:28.747 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.740+0000 7f383ed9d700 1 --2- 172.21.15.42:0/3908299621 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f384012e2a0 0x7f3840118fd0 secure :-1 s=THROTTLE_DONE pgs=2549 cs=0 l=1 rx=0x7f38380011d0 tx=0x7f3838007e50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:28.748 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.740+0000 7f382f7fe700 1 -- 172.21.15.42:0/3908299621 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 303 v303) v1 ==== 95+0+2114 (secure 0 0 0) 0x7f3830014cd0 con 0x7f384012e2a0 2019-12-05T06:23:28.748 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 303 2019-12-05T06:23:28.748 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:23:28.749 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":303,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":10993,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6835/3964426552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3964426552},{"type":"v1","addr":"172.21.15.42:6835","nonce":3964426552}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":303}],"filesystems":[{"mdsmap":{"epoch":302,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:21:41.099460+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":103,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10710},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10710":{"gid":10710,"name":"b","rank":0,"incarnation":299,"state":"up:active","state_seq":29,"addr":"172.21.15.40:6833/526579407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":526579407},{"type":"v1","addr":"172.21.15.40:6833","nonce":526579407}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:23:28.751 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.744+0000 7f3844aea700 1 -- 172.21.15.42:0/3908299621 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f382403abb0 msgr2=0x7f382403d0c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:28.751 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.744+0000 7f3844aea700 1 --2- 172.21.15.42:0/3908299621 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f382403abb0 0x7f382403d0c0 crc :-1 s=READY pgs=1060 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:28.751 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.744+0000 7f3844aea700 1 -- 172.21.15.42:0/3908299621 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f384012e2a0 msgr2=0x7f3840118fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:28.751 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.744+0000 7f3844aea700 1 --2- 172.21.15.42:0/3908299621 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f384012e2a0 0x7f3840118fd0 secure :-1 s=READY pgs=2549 cs=0 l=1 rx=0x7f38380011d0 tx=0x7f3838007e50).stop 2019-12-05T06:23:28.751 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.744+0000 7f3844aea700 1 -- 172.21.15.42:0/3908299621 shutdown_connections 2019-12-05T06:23:28.751 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.744+0000 7f3844aea700 1 --2- 172.21.15.42:0/3908299621 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f382403abb0 0x7f382403d0c0 unknown :-1 s=CLOSED pgs=1060 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:28.752 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.744+0000 7f3844aea700 1 --2- 172.21.15.42:0/3908299621 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f384012e2a0 0x7f3840118fd0 unknown :-1 s=CLOSED pgs=2549 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:28.752 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.744+0000 7f3844aea700 1 -- 172.21.15.42:0/3908299621 shutdown_connections 2019-12-05T06:23:28.752 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.744+0000 7f3844aea700 1 -- 172.21.15.42:0/3908299621 wait complete. 2019-12-05T06:23:28.752 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.744+0000 7f3844aea700 1 -- 172.21.15.42:0/3908299621 >> 172.21.15.42:0/3908299621 conn(0x7f384008fde0 msgr2=0x7f3840141360 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:23:28.763 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.b (rank=0) 2019-12-05T06:23:28.764 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2019-12-05T06:23:28.764 INFO:teuthology.orchestra.run:waiting for 300 2019-12-05T06:23:28.776 INFO:tasks.ceph.mds.b:Stopped 2019-12-05T06:23:28.777 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:23:28.881 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.868+0000 7f4128dd3700 1 Processor -- start 2019-12-05T06:23:28.881 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.872+0000 7f4128dd3700 1 -- start start 2019-12-05T06:23:28.882 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.872+0000 7f4128dd3700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f41240a7740 0x7f412408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:23:28.882 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.872+0000 7f4128dd3700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f412409fae0 0x7f412407eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:23:28.882 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.872+0000 7f4128dd3700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41240a0710 con 0x7f412407e720 2019-12-05T06:23:28.882 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.872+0000 7f4128dd3700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41240a0b00 con 0x7f41240a7740 2019-12-05T06:23:28.882 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.872+0000 7f4128dd3700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f412408e5b0 con 0x7f412409fae0 2019-12-05T06:23:28.882 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.872+0000 7f412359e700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f412409fae0 0x7f412407eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:28.882 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.872+0000 7f4122d9d700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f41240a7740 0x7f412408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:28.883 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.872+0000 7f412259c700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f412407e720 0x7f412409f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:35582/0 (socket says 172.21.15.42:35582) 2019-12-05T06:23:28.883 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.872+0000 7f412259c700 1 -- 172.21.15.42:0/1482251630 learned_addr learned my addr 172.21.15.42:0/1482251630 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:23:28.883 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.872+0000 7f412359e700 1 -- 172.21.15.42:0/1482251630 >> v2:172.21.15.40:3300/0 conn(0x7f41240a7740 msgr2=0x7f412408df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:23:28.883 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.872+0000 7f412359e700 1 --2- 172.21.15.42:0/1482251630 >> v2:172.21.15.40:3300/0 conn(0x7f41240a7740 0x7f412408df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:23:28.884 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.872+0000 7f412359e700 1 -- 172.21.15.42:0/1482251630 >> v1:172.21.15.40:6789/0 conn(0x7f412407e720 legacy=0x7f412409f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:28.884 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.872+0000 7f4121d9b700 1 -- 172.21.15.42:0/1482251630 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2104220002 0 0) 0x7f41240a0710 con 0x7f412407e720 2019-12-05T06:23:28.884 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.872+0000 7f412359e700 1 -- 172.21.15.42:0/1482251630 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f412407f0a0 con 0x7f412409fae0 2019-12-05T06:23:28.884 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.872+0000 7f4122d9d700 1 --2- 172.21.15.42:0/1482251630 >> v2:172.21.15.40:3300/0 conn(0x7f41240a7740 0x7f412408df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:23:28.884 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.872+0000 7f412359e700 1 --2- 172.21.15.42:0/1482251630 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f412409fae0 0x7f412407eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f411c008b10 tx=0x7f411c00fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:28.884 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.872+0000 7f412359e700 1 --2- 172.21.15.42:0/1482251630 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f412409fae0 0x7f412407eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f411c008b10 tx=0x7f411c00fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:28.884 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.872+0000 7f412359e700 1 --2- 172.21.15.42:0/1482251630 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f412409fae0 0x7f412407eb50 secure :-1 s=READY pgs=551 cs=0 l=1 rx=0x7f411c008b10 tx=0x7f411c00fff0).ready entity=mon.2 client_cookie=e6d60dacc2e2f847 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:23:28.885 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.872+0000 7f412359e700 1 --2- 172.21.15.42:0/1482251630 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f412409fae0 0x7f412407eb50 secure :-1 s=THROTTLE_DONE pgs=551 cs=0 l=1 rx=0x7f411c008b10 tx=0x7f411c00fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:28.885 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.872+0000 7f4121d9b700 1 -- 172.21.15.42:0/1482251630 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f411c010900 con 0x7f412409fae0 2019-12-05T06:23:28.885 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.872+0000 7f412359e700 1 --2- 172.21.15.42:0/1482251630 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f412409fae0 0x7f412407eb50 secure :-1 s=THROTTLE_DONE pgs=551 cs=0 l=1 rx=0x7f411c008b10 tx=0x7f411c00fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:28.885 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.872+0000 7f412359e700 1 --2- 172.21.15.42:0/1482251630 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f412409fae0 0x7f412407eb50 secure :-1 s=THROTTLE_DONE pgs=551 cs=0 l=1 rx=0x7f411c008b10 tx=0x7f411c00fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:28.885 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.872+0000 7f4121d9b700 1 -- 172.21.15.42:0/1482251630 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f411c008b50 con 0x7f412409fae0 2019-12-05T06:23:28.885 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.872+0000 7f4121d9b700 1 -- 172.21.15.42:0/1482251630 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f411c0110c0 con 0x7f412409fae0 2019-12-05T06:23:28.885 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.872+0000 7f4128dd3700 1 -- 172.21.15.42:0/1482251630 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f412409fae0 msgr2=0x7f412407eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:28.886 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.872+0000 7f4128dd3700 1 --2- 172.21.15.42:0/1482251630 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f412409fae0 0x7f412407eb50 secure :-1 s=READY pgs=551 cs=0 l=1 rx=0x7f411c008b10 tx=0x7f411c00fff0).stop 2019-12-05T06:23:28.886 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.876+0000 7f4128dd3700 1 -- 172.21.15.42:0/1482251630 shutdown_connections 2019-12-05T06:23:28.886 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.876+0000 7f4128dd3700 1 --2- 172.21.15.42:0/1482251630 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f412409fae0 0x7f412407eb50 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:28.886 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.876+0000 7f4128dd3700 1 --2- 172.21.15.42:0/1482251630 >> v2:172.21.15.40:3300/0 conn(0x7f41240a7740 0x7f412408df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:23:28.886 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.876+0000 7f4128dd3700 1 -- 172.21.15.42:0/1482251630 shutdown_connections 2019-12-05T06:23:28.886 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.876+0000 7f4128dd3700 1 -- 172.21.15.42:0/1482251630 wait complete. 2019-12-05T06:23:28.886 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.876+0000 7f4128dd3700 1 -- 172.21.15.42:0/1482251630 >> 172.21.15.42:0/1482251630 conn(0x7f41240aabe0 msgr2=0x7f41240a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:23:28.890 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.876+0000 7f4128dd3700 1 Processor -- start 2019-12-05T06:23:28.890 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.876+0000 7f4128dd3700 1 -- start start 2019-12-05T06:23:28.890 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.876+0000 7f4128dd3700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f41240a6a40 0x7f41240a59f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:23:28.890 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.876+0000 7f4128dd3700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f41240a2490 0x7f412413b960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:23:28.890 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.876+0000 7f4128dd3700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4124124a80 con 0x7f41240a2060 2019-12-05T06:23:28.890 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.876+0000 7f4128dd3700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4124131580 con 0x7f41240a6a40 2019-12-05T06:23:28.891 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.876+0000 7f4128dd3700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4124131820 con 0x7f41240a2490 2019-12-05T06:23:28.891 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.880+0000 7f4122d9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f41240a6a40 0x7f41240a59f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:28.891 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.880+0000 7f4122d9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f41240a6a40 0x7f41240a59f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:54342/0 (socket says 172.21.15.42:54342) 2019-12-05T06:23:28.891 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.880+0000 7f412259c700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f41240a2060 0x7f4124131ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:35586/0 (socket says 172.21.15.42:35586) 2019-12-05T06:23:28.891 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.880+0000 7f4122d9d700 1 -- 172.21.15.42:0/2562908388 learned_addr learned my addr 172.21.15.42:0/2562908388 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:23:28.891 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.880+0000 7f412359e700 1 --2- 172.21.15.42:0/2562908388 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f41240a2490 0x7f412413b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:28.891 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.880+0000 7f411b7fe700 1 -- 172.21.15.42:0/2562908388 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2472887848 0 0) 0x7f4124124a80 con 0x7f41240a2060 2019-12-05T06:23:28.892 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.880+0000 7f4122d9d700 1 -- 172.21.15.42:0/2562908388 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f41240a2490 msgr2=0x7f412413b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:28.892 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.880+0000 7f4122d9d700 1 --2- 172.21.15.42:0/2562908388 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f41240a2490 0x7f412413b960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:28.892 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.880+0000 7f4122d9d700 1 -- 172.21.15.42:0/2562908388 >> v1:172.21.15.40:6789/0 conn(0x7f41240a2060 legacy=0x7f4124131ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:28.892 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.880+0000 7f4122d9d700 1 -- 172.21.15.42:0/2562908388 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41240a29e0 con 0x7f41240a6a40 2019-12-05T06:23:28.892 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.880+0000 7f412359e700 1 --2- 172.21.15.42:0/2562908388 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f41240a2490 0x7f412413b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:23:28.892 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.880+0000 7f4122d9d700 1 --2- 172.21.15.42:0/2562908388 >> v2:172.21.15.42:3300/0 conn(0x7f41240a6a40 0x7f41240a59f0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f4124131740 tx=0x7f4114010d60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:28.892 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.880+0000 7f4122d9d700 1 --2- 172.21.15.42:0/2562908388 >> v2:172.21.15.42:3300/0 conn(0x7f41240a6a40 0x7f41240a59f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f4124131740 tx=0x7f4114010d60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:28.893 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.880+0000 7f4122d9d700 1 --2- 172.21.15.42:0/2562908388 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f41240a6a40 0x7f41240a59f0 secure :-1 s=READY pgs=2550 cs=0 l=1 rx=0x7f4124131740 tx=0x7f4114010d60).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:23:28.893 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.880+0000 7f4122d9d700 1 --2- 172.21.15.42:0/2562908388 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f41240a6a40 0x7f41240a59f0 secure :-1 s=THROTTLE_DONE pgs=2550 cs=0 l=1 rx=0x7f4124131740 tx=0x7f4114010d60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:28.893 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.880+0000 7f411b7fe700 1 -- 172.21.15.42:0/2562908388 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f41140126a0 con 0x7f41240a6a40 2019-12-05T06:23:28.893 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.880+0000 7f4128dd3700 1 -- 172.21.15.42:0/2562908388 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41240a50f0 con 0x7f41240a6a40 2019-12-05T06:23:28.893 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.880+0000 7f4122d9d700 1 --2- 172.21.15.42:0/2562908388 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f41240a6a40 0x7f41240a59f0 secure :-1 s=THROTTLE_DONE pgs=2550 cs=0 l=1 rx=0x7f4124131740 tx=0x7f4114010d60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:28.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.880+0000 7f4122d9d700 1 --2- 172.21.15.42:0/2562908388 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f41240a6a40 0x7f41240a59f0 secure :-1 s=THROTTLE_DONE pgs=2550 cs=0 l=1 rx=0x7f4124131740 tx=0x7f4114010d60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:28.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.880+0000 7f411b7fe700 1 -- 172.21.15.42:0/2562908388 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4114007150 con 0x7f41240a6a40 2019-12-05T06:23:28.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.880+0000 7f411b7fe700 1 -- 172.21.15.42:0/2562908388 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4114012cc0 con 0x7f41240a6a40 2019-12-05T06:23:28.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.880+0000 7f4128dd3700 1 -- 172.21.15.42:0/2562908388 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41241322c0 con 0x7f41240a6a40 2019-12-05T06:23:28.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.880+0000 7f4122d9d700 1 --2- 172.21.15.42:0/2562908388 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f41240a6a40 0x7f41240a59f0 secure :-1 s=THROTTLE_DONE pgs=2550 cs=0 l=1 rx=0x7f4124131740 tx=0x7f4114010d60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:28.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.880+0000 7f411b7fe700 1 -- 172.21.15.42:0/2562908388 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f4114005540 con 0x7f41240a6a40 2019-12-05T06:23:28.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.880+0000 7f4122d9d700 1 --2- 172.21.15.42:0/2562908388 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f41240a6a40 0x7f41240a59f0 secure :-1 s=THROTTLE_DONE pgs=2550 cs=0 l=1 rx=0x7f4124131740 tx=0x7f4114010d60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:28.895 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.884+0000 7f411b7fe700 1 --2- 172.21.15.42:0/2562908388 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f410803ac80 0x7f410803d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:23:28.895 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.884+0000 7f411b7fe700 1 -- 172.21.15.42:0/2562908388 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 10675+0+0 (secure 0 0 0) 0x7f411404b930 con 0x7f41240a6a40 2019-12-05T06:23:28.895 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.884+0000 7f412259c700 1 --2- 172.21.15.42:0/2562908388 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f410803ac80 0x7f410803d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:28.895 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.884+0000 7f412259c700 1 --2- 172.21.15.42:0/2562908388 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f410803ac80 0x7f410803d190 crc :-1 s=READY pgs=1061 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:23:28.895 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.884+0000 7f4128dd3700 1 -- 172.21.15.42:0/2562908388 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f412405d110 con 0x7f41240a6a40 2019-12-05T06:23:28.898 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.892+0000 7f4122d9d700 1 --2- 172.21.15.42:0/2562908388 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f41240a6a40 0x7f41240a59f0 secure :-1 s=THROTTLE_DONE pgs=2550 cs=0 l=1 rx=0x7f4124131740 tx=0x7f4114010d60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:28.899 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:28.892+0000 7f411b7fe700 1 -- 172.21.15.42:0/2562908388 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f4114032bf0 con 0x7f41240a6a40 2019-12-05T06:23:29.064 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:29.056+0000 7f4128dd3700 1 -- 172.21.15.42:0/2562908388 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f412405b8c0 con 0x7f41240a6a40 2019-12-05T06:23:29.065 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:29.060+0000 7f4122d9d700 1 --2- 172.21.15.42:0/2562908388 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f41240a6a40 0x7f41240a59f0 secure :-1 s=THROTTLE_DONE pgs=2550 cs=0 l=1 rx=0x7f4124131740 tx=0x7f4114010d60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:29.065 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:29.060+0000 7f411b7fe700 1 -- 172.21.15.42:0/2562908388 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 303 v303) v1 ==== 95+0+2114 (secure 0 0 0) 0x7f41140337a0 con 0x7f41240a6a40 2019-12-05T06:23:29.065 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 303 2019-12-05T06:23:29.065 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:23:29.066 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":303,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":10993,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6835/3964426552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3964426552},{"type":"v1","addr":"172.21.15.42:6835","nonce":3964426552}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":303}],"filesystems":[{"mdsmap":{"epoch":302,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:21:41.099460+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":103,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10710},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10710":{"gid":10710,"name":"b","rank":0,"incarnation":299,"state":"up:active","state_seq":29,"addr":"172.21.15.40:6833/526579407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":526579407},{"type":"v1","addr":"172.21.15.40:6833","nonce":526579407}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:23:29.068 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:29.060+0000 7f4128dd3700 1 -- 172.21.15.42:0/2562908388 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f410803ac80 msgr2=0x7f410803d190 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:29.068 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:29.060+0000 7f4128dd3700 1 --2- 172.21.15.42:0/2562908388 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f410803ac80 0x7f410803d190 crc :-1 s=READY pgs=1061 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:29.068 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:29.060+0000 7f4128dd3700 1 -- 172.21.15.42:0/2562908388 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f41240a6a40 msgr2=0x7f41240a59f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:29.068 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:29.060+0000 7f4128dd3700 1 --2- 172.21.15.42:0/2562908388 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f41240a6a40 0x7f41240a59f0 secure :-1 s=READY pgs=2550 cs=0 l=1 rx=0x7f4124131740 tx=0x7f4114010d60).stop 2019-12-05T06:23:29.068 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:29.060+0000 7f4128dd3700 1 -- 172.21.15.42:0/2562908388 shutdown_connections 2019-12-05T06:23:29.068 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:29.060+0000 7f4128dd3700 1 --2- 172.21.15.42:0/2562908388 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f410803ac80 0x7f410803d190 unknown :-1 s=CLOSED pgs=1061 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:29.069 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:29.060+0000 7f4128dd3700 1 --2- 172.21.15.42:0/2562908388 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f41240a2490 0x7f412413b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:29.069 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:29.060+0000 7f4128dd3700 1 --2- 172.21.15.42:0/2562908388 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f41240a6a40 0x7f41240a59f0 unknown :-1 s=CLOSED pgs=2550 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:29.069 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:29.060+0000 7f4128dd3700 1 -- 172.21.15.42:0/2562908388 shutdown_connections 2019-12-05T06:23:29.069 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:29.060+0000 7f4128dd3700 1 -- 172.21.15.42:0/2562908388 wait complete. 2019-12-05T06:23:29.069 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:29.060+0000 7f4128dd3700 1 -- 172.21.15.42:0/2562908388 >> 172.21.15.42:0/2562908388 conn(0x7f41240a8120 msgr2=0x7f4124059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:23:29.078 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:23:31.079 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:23:31.188 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:31.176+0000 7f312876b700 1 Processor -- start 2019-12-05T06:23:31.188 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:31.176+0000 7f312876b700 1 -- start start 2019-12-05T06:23:31.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:31.180+0000 7f312876b700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f312008df80 0x7f312009f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:23:31.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:31.180+0000 7f312876b700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f312009fc60 con 0x7f31200a7740 2019-12-05T06:23:31.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:31.180+0000 7f312876b700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31200a0710 con 0x7f312009f830 2019-12-05T06:23:31.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:31.180+0000 7f312876b700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31200a0b00 con 0x7f312008df80 2019-12-05T06:23:31.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:31.180+0000 7f3127509700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f312009f830 0x7f312014efe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:54396/0 (socket says 172.21.15.42:54396) 2019-12-05T06:23:31.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:31.180+0000 7f3127509700 1 -- 172.21.15.42:0/3281373648 learned_addr learned my addr 172.21.15.42:0/3281373648 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:23:31.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:31.180+0000 7f3126507700 1 --2- 172.21.15.42:0/3281373648 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f312008df80 0x7f312009f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:31.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:31.180+0000 7f3125d06700 1 -- 172.21.15.42:0/3281373648 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2925031814 0 0) 0x7f31200a0710 con 0x7f312009f830 2019-12-05T06:23:31.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:31.180+0000 7f3125d06700 1 -- 172.21.15.42:0/3281373648 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f310c0030b0 con 0x7f312009f830 2019-12-05T06:23:31.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:31.180+0000 7f3125d06700 1 -- 172.21.15.42:0/3281373648 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2290248008 0 0) 0x7f312009fc60 con 0x7f31200a7740 2019-12-05T06:23:31.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:31.180+0000 7f3126507700 1 -- 172.21.15.42:0/3281373648 >> v1:172.21.15.42:6789/0 conn(0x7f312009f830 legacy=0x7f312014efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:31.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:31.180+0000 7f3126507700 1 -- 172.21.15.42:0/3281373648 >> v1:172.21.15.40:6789/0 conn(0x7f31200a7740 legacy=0x7f312007e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:31.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:31.180+0000 7f3126507700 1 -- 172.21.15.42:0/3281373648 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f312007ef20 con 0x7f312008df80 2019-12-05T06:23:31.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:31.180+0000 7f3126507700 1 --2- 172.21.15.42:0/3281373648 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f312008df80 0x7f312009f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f3110002f30 tx=0x7f311000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:31.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:31.180+0000 7f3126507700 1 --2- 172.21.15.42:0/3281373648 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f312008df80 0x7f312009f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f3110002f30 tx=0x7f311000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:31.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:31.180+0000 7f3126507700 1 --2- 172.21.15.42:0/3281373648 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f312008df80 0x7f312009f2e0 secure :-1 s=READY pgs=552 cs=0 l=1 rx=0x7f3110002f30 tx=0x7f311000fbf0).ready entity=mon.2 client_cookie=29f81f9b19e62227 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:23:31.193 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:31.180+0000 7f3126507700 1 --2- 172.21.15.42:0/3281373648 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f312008df80 0x7f312009f2e0 secure :-1 s=THROTTLE_DONE pgs=552 cs=0 l=1 rx=0x7f3110002f30 tx=0x7f311000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:31.193 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:31.180+0000 7f3125d06700 1 -- 172.21.15.42:0/3281373648 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f31100112b0 con 0x7f312008df80 2019-12-05T06:23:31.193 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:31.180+0000 7f3126507700 1 --2- 172.21.15.42:0/3281373648 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f312008df80 0x7f312009f2e0 secure :-1 s=THROTTLE_DONE pgs=552 cs=0 l=1 rx=0x7f3110002f30 tx=0x7f311000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:31.193 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:31.180+0000 7f3126507700 1 --2- 172.21.15.42:0/3281373648 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f312008df80 0x7f312009f2e0 secure :-1 s=THROTTLE_DONE pgs=552 cs=0 l=1 rx=0x7f3110002f30 tx=0x7f311000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:31.193 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:31.180+0000 7f3125d06700 1 -- 172.21.15.42:0/3281373648 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3110008bc0 con 0x7f312008df80 2019-12-05T06:23:31.194 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:31.180+0000 7f3125d06700 1 -- 172.21.15.42:0/3281373648 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3110011890 con 0x7f312008df80 2019-12-05T06:23:31.194 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:31.180+0000 7f312876b700 1 -- 172.21.15.42:0/3281373648 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f312008df80 msgr2=0x7f312009f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:31.194 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:31.180+0000 7f312876b700 1 --2- 172.21.15.42:0/3281373648 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f312008df80 0x7f312009f2e0 secure :-1 s=READY pgs=552 cs=0 l=1 rx=0x7f3110002f30 tx=0x7f311000fbf0).stop 2019-12-05T06:23:31.194 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:31.180+0000 7f312876b700 1 -- 172.21.15.42:0/3281373648 shutdown_connections 2019-12-05T06:23:31.195 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:31.180+0000 7f312876b700 1 --2- 172.21.15.42:0/3281373648 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f312008df80 0x7f312009f2e0 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:31.195 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:31.180+0000 7f312876b700 1 -- 172.21.15.42:0/3281373648 shutdown_connections 2019-12-05T06:23:31.195 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:31.180+0000 7f312876b700 1 -- 172.21.15.42:0/3281373648 wait complete. 2019-12-05T06:23:31.195 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:31.180+0000 7f312876b700 1 -- 172.21.15.42:0/3281373648 >> 172.21.15.42:0/3281373648 conn(0x7f31200aabe0 msgr2=0x7f31200a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:23:31.200 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:31.184+0000 7f312876b700 1 Processor -- start 2019-12-05T06:23:31.200 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:31.184+0000 7f312876b700 1 -- start start 2019-12-05T06:23:31.201 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:31.184+0000 7f312876b700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f31201451d0 0x7f312007b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:23:31.201 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:31.184+0000 7f312876b700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3120116500 0x7f31200a1880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:23:31.201 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:31.184+0000 7f312876b700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f312007be50 0x7f31200a1dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:23:31.201 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:31.184+0000 7f312876b700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31200a3210 con 0x7f31201451d0 2019-12-05T06:23:31.202 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:31.184+0000 7f312876b700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31200a3470 con 0x7f312007be50 2019-12-05T06:23:31.202 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:31.184+0000 7f312876b700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31200a3710 con 0x7f3120116500 2019-12-05T06:23:31.202 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:31.184+0000 7f3127509700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f312007be50 0x7f31200a1dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:31.202 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:31.184+0000 7f3127509700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f312007be50 0x7f31200a1dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:54364/0 (socket says 172.21.15.42:54364) 2019-12-05T06:23:31.202 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:31.184+0000 7f3126d08700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f31201451d0 0x7f312007b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:31.202 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:31.184+0000 7f3126507700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3120116500 0x7f31200a1880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:31.202 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:31.184+0000 7f3127509700 1 -- 172.21.15.42:0/1110056330 learned_addr learned my addr 172.21.15.42:0/1110056330 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:23:31.203 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:31.184+0000 7f3127509700 1 -- 172.21.15.42:0/1110056330 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3120116500 msgr2=0x7f31200a1880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:31.203 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:31.184+0000 7f3127509700 1 --2- 172.21.15.42:0/1110056330 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3120116500 0x7f31200a1880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:31.203 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:31.184+0000 7f3127509700 1 -- 172.21.15.42:0/1110056330 >> v2:172.21.15.40:3300/0 conn(0x7f31201451d0 msgr2=0x7f312007b900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:31.203 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:31.184+0000 7f3127509700 1 --2- 172.21.15.42:0/1110056330 >> v2:172.21.15.40:3300/0 conn(0x7f31201451d0 0x7f312007b900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:31.203 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:31.184+0000 7f3127509700 1 -- 172.21.15.42:0/1110056330 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31200a15a0 con 0x7f312007be50 2019-12-05T06:23:31.203 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:31.184+0000 7f3127509700 1 --2- 172.21.15.42:0/1110056330 >> v2:172.21.15.42:3300/0 conn(0x7f312007be50 0x7f31200a1dd0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f31200a38d0 tx=0x7f3118010c10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:31.204 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:31.188+0000 7f3127509700 1 --2- 172.21.15.42:0/1110056330 >> v2:172.21.15.42:3300/0 conn(0x7f312007be50 0x7f31200a1dd0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f31200a38d0 tx=0x7f3118010c10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:31.204 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:31.188+0000 7f3127509700 1 --2- 172.21.15.42:0/1110056330 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f312007be50 0x7f31200a1dd0 secure :-1 s=READY pgs=2552 cs=0 l=1 rx=0x7f31200a38d0 tx=0x7f3118010c10).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:23:31.204 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:31.188+0000 7f3127509700 1 --2- 172.21.15.42:0/1110056330 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f312007be50 0x7f31200a1dd0 secure :-1 s=THROTTLE_DONE pgs=2552 cs=0 l=1 rx=0x7f31200a38d0 tx=0x7f3118010c10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:31.204 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:31.188+0000 7f31177fe700 1 -- 172.21.15.42:0/1110056330 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3118005bc0 con 0x7f312007be50 2019-12-05T06:23:31.204 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:31.188+0000 7f3127509700 1 --2- 172.21.15.42:0/1110056330 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f312007be50 0x7f31200a1dd0 secure :-1 s=THROTTLE_DONE pgs=2552 cs=0 l=1 rx=0x7f31200a38d0 tx=0x7f3118010c10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:31.204 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:31.188+0000 7f3127509700 1 --2- 172.21.15.42:0/1110056330 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f312007be50 0x7f31200a1dd0 secure :-1 s=THROTTLE_DONE pgs=2552 cs=0 l=1 rx=0x7f31200a38d0 tx=0x7f3118010c10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:31.205 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:31.188+0000 7f31177fe700 1 -- 172.21.15.42:0/1110056330 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31180057e0 con 0x7f312007be50 2019-12-05T06:23:31.205 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:31.188+0000 7f312876b700 1 -- 172.21.15.42:0/1110056330 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31200a3c50 con 0x7f312007be50 2019-12-05T06:23:31.205 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:31.188+0000 7f31177fe700 1 -- 172.21.15.42:0/1110056330 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3118012800 con 0x7f312007be50 2019-12-05T06:23:31.205 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:31.188+0000 7f312876b700 1 -- 172.21.15.42:0/1110056330 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31200a4280 con 0x7f312007be50 2019-12-05T06:23:31.205 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:31.188+0000 7f3127509700 1 --2- 172.21.15.42:0/1110056330 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f312007be50 0x7f31200a1dd0 secure :-1 s=THROTTLE_DONE pgs=2552 cs=0 l=1 rx=0x7f31200a38d0 tx=0x7f3118010c10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:31.206 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:31.188+0000 7f31177fe700 1 -- 172.21.15.42:0/1110056330 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f3118012a00 con 0x7f312007be50 2019-12-05T06:23:31.206 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:31.188+0000 7f3127509700 1 --2- 172.21.15.42:0/1110056330 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f312007be50 0x7f31200a1dd0 secure :-1 s=THROTTLE_DONE pgs=2552 cs=0 l=1 rx=0x7f31200a38d0 tx=0x7f3118010c10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:31.206 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:31.188+0000 7f31177fe700 1 --2- 172.21.15.42:0/1110056330 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f310803ae90 0x7f310803d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:23:31.206 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:31.188+0000 7f31177fe700 1 -- 172.21.15.42:0/1110056330 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 10675+0+0 (secure 0 0 0) 0x7f311804b360 con 0x7f312007be50 2019-12-05T06:23:31.207 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:31.188+0000 7f3126d08700 1 --2- 172.21.15.42:0/1110056330 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f310803ae90 0x7f310803d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:31.207 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:31.188+0000 7f3126d08700 1 --2- 172.21.15.42:0/1110056330 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f310803ae90 0x7f310803d3a0 crc :-1 s=READY pgs=1062 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:23:31.207 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:31.192+0000 7f312876b700 1 -- 172.21.15.42:0/1110056330 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f312005d110 con 0x7f312007be50 2019-12-05T06:23:31.208 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:31.200+0000 7f3127509700 1 --2- 172.21.15.42:0/1110056330 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f312007be50 0x7f31200a1dd0 secure :-1 s=THROTTLE_DONE pgs=2552 cs=0 l=1 rx=0x7f31200a38d0 tx=0x7f3118010c10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:31.208 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:31.200+0000 7f31177fe700 1 -- 172.21.15.42:0/1110056330 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f3118032380 con 0x7f312007be50 2019-12-05T06:23:31.378 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:31.372+0000 7f312876b700 1 -- 172.21.15.42:0/1110056330 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f312005b8c0 con 0x7f312007be50 2019-12-05T06:23:31.379 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:31.372+0000 7f3127509700 1 --2- 172.21.15.42:0/1110056330 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f312007be50 0x7f31200a1dd0 secure :-1 s=THROTTLE_DONE pgs=2552 cs=0 l=1 rx=0x7f31200a38d0 tx=0x7f3118010c10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:31.379 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:31.372+0000 7f31177fe700 1 -- 172.21.15.42:0/1110056330 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 303 v303) v1 ==== 95+0+2114 (secure 0 0 0) 0x7f31180325e0 con 0x7f312007be50 2019-12-05T06:23:31.379 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 303 2019-12-05T06:23:31.379 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:23:31.379 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":303,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":10993,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6835/3964426552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3964426552},{"type":"v1","addr":"172.21.15.42:6835","nonce":3964426552}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":303}],"filesystems":[{"mdsmap":{"epoch":302,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:21:41.099460+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":103,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10710},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10710":{"gid":10710,"name":"b","rank":0,"incarnation":299,"state":"up:active","state_seq":29,"addr":"172.21.15.40:6833/526579407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":526579407},{"type":"v1","addr":"172.21.15.40:6833","nonce":526579407}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:23:31.381 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:31.376+0000 7f312876b700 1 -- 172.21.15.42:0/1110056330 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f310803ae90 msgr2=0x7f310803d3a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:31.382 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:31.376+0000 7f312876b700 1 --2- 172.21.15.42:0/1110056330 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f310803ae90 0x7f310803d3a0 crc :-1 s=READY pgs=1062 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:31.382 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:31.376+0000 7f312876b700 1 -- 172.21.15.42:0/1110056330 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f312007be50 msgr2=0x7f31200a1dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:31.382 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:31.376+0000 7f312876b700 1 --2- 172.21.15.42:0/1110056330 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f312007be50 0x7f31200a1dd0 secure :-1 s=READY pgs=2552 cs=0 l=1 rx=0x7f31200a38d0 tx=0x7f3118010c10).stop 2019-12-05T06:23:31.382 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:31.376+0000 7f312876b700 1 -- 172.21.15.42:0/1110056330 shutdown_connections 2019-12-05T06:23:31.382 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:31.376+0000 7f312876b700 1 --2- 172.21.15.42:0/1110056330 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f310803ae90 0x7f310803d3a0 unknown :-1 s=CLOSED pgs=1062 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:31.382 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:31.376+0000 7f312876b700 1 --2- 172.21.15.42:0/1110056330 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f312007be50 0x7f31200a1dd0 unknown :-1 s=CLOSED pgs=2552 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:31.382 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:31.376+0000 7f312876b700 1 --2- 172.21.15.42:0/1110056330 >> v2:172.21.15.40:3300/0 conn(0x7f31201451d0 0x7f312007b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:31.383 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:31.376+0000 7f312876b700 1 --2- 172.21.15.42:0/1110056330 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3120116500 0x7f31200a1880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:31.383 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:31.376+0000 7f312876b700 1 -- 172.21.15.42:0/1110056330 shutdown_connections 2019-12-05T06:23:31.383 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:31.376+0000 7f312876b700 1 -- 172.21.15.42:0/1110056330 wait complete. 2019-12-05T06:23:31.383 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:31.376+0000 7f312876b700 1 -- 172.21.15.42:0/1110056330 >> 172.21.15.42:0/1110056330 conn(0x7f31200aabe0 msgr2=0x7f31200a7f90 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:23:31.391 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:23:33.392 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:23:33.501 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:33.488+0000 7f5a9ac67700 1 Processor -- start 2019-12-05T06:23:33.501 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:33.488+0000 7f5a9ac67700 1 -- start start 2019-12-05T06:23:33.501 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:33.492+0000 7f5a9ac67700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5a9408df80 0x7f5a9409f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:23:33.501 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:33.492+0000 7f5a9ac67700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a9409fc60 con 0x7f5a940a7740 2019-12-05T06:23:33.502 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:33.492+0000 7f5a9ac67700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a940a0710 con 0x7f5a9409f830 2019-12-05T06:23:33.503 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:33.492+0000 7f5a9ac67700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a940a0b00 con 0x7f5a9408df80 2019-12-05T06:23:33.503 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:33.492+0000 7f5a99a05700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f5a9409f830 0x7f5a9414efe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:54412/0 (socket says 172.21.15.42:54412) 2019-12-05T06:23:33.503 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:33.492+0000 7f5a99a05700 1 -- 172.21.15.42:0/1481705682 learned_addr learned my addr 172.21.15.42:0/1481705682 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:23:33.504 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:33.492+0000 7f5a98a03700 1 --2- 172.21.15.42:0/1481705682 >> v2:172.21.15.40:3300/0 conn(0x7f5a9408df80 0x7f5a9409f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:33.504 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:33.492+0000 7f5a8bfff700 1 -- 172.21.15.42:0/1481705682 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3084563087 0 0) 0x7f5a940a0710 con 0x7f5a9409f830 2019-12-05T06:23:33.504 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:33.492+0000 7f5a8bfff700 1 -- 172.21.15.42:0/1481705682 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a800030b0 con 0x7f5a9409f830 2019-12-05T06:23:33.504 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:33.492+0000 7f5a8bfff700 1 -- 172.21.15.42:0/1481705682 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1441024675 0 0) 0x7f5a9409fc60 con 0x7f5a940a7740 2019-12-05T06:23:33.504 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:33.492+0000 7f5a8bfff700 1 -- 172.21.15.42:0/1481705682 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a940a0710 con 0x7f5a940a7740 2019-12-05T06:23:33.504 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:33.492+0000 7f5a98a03700 1 -- 172.21.15.42:0/1481705682 >> v1:172.21.15.42:6789/0 conn(0x7f5a9409f830 legacy=0x7f5a9414efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:33.505 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:33.492+0000 7f5a98a03700 1 -- 172.21.15.42:0/1481705682 >> v1:172.21.15.40:6789/0 conn(0x7f5a940a7740 legacy=0x7f5a9407e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:33.505 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:33.492+0000 7f5a98a03700 1 -- 172.21.15.42:0/1481705682 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a9407ef20 con 0x7f5a9408df80 2019-12-05T06:23:33.505 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:33.492+0000 7f5a98a03700 1 --2- 172.21.15.42:0/1481705682 >> v2:172.21.15.40:3300/0 conn(0x7f5a9408df80 0x7f5a9409f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f5a84008790 tx=0x7f5a8400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:33.505 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:33.492+0000 7f5a98a03700 1 --2- 172.21.15.42:0/1481705682 >> v2:172.21.15.40:3300/0 conn(0x7f5a9408df80 0x7f5a9409f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f5a84008790 tx=0x7f5a8400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:33.505 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:33.492+0000 7f5a98a03700 1 --2- 172.21.15.42:0/1481705682 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5a9408df80 0x7f5a9409f2e0 secure :-1 s=READY pgs=1913 cs=0 l=1 rx=0x7f5a84008790 tx=0x7f5a8400fbf0).ready entity=mon.0 client_cookie=2a3832def6478011 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:23:33.505 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:33.492+0000 7f5a98a03700 1 --2- 172.21.15.42:0/1481705682 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5a9408df80 0x7f5a9409f2e0 secure :-1 s=THROTTLE_DONE pgs=1913 cs=0 l=1 rx=0x7f5a84008790 tx=0x7f5a8400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:33.505 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:33.492+0000 7f5a8bfff700 1 -- 172.21.15.42:0/1481705682 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5a84011270 con 0x7f5a9408df80 2019-12-05T06:23:33.506 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:33.492+0000 7f5a98a03700 1 --2- 172.21.15.42:0/1481705682 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5a9408df80 0x7f5a9409f2e0 secure :-1 s=THROTTLE_DONE pgs=1913 cs=0 l=1 rx=0x7f5a84008790 tx=0x7f5a8400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:33.506 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:33.492+0000 7f5a98a03700 1 --2- 172.21.15.42:0/1481705682 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5a9408df80 0x7f5a9409f2e0 secure :-1 s=THROTTLE_DONE pgs=1913 cs=0 l=1 rx=0x7f5a84008790 tx=0x7f5a8400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:33.506 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:33.492+0000 7f5a8bfff700 1 -- 172.21.15.42:0/1481705682 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a84008bc0 con 0x7f5a9408df80 2019-12-05T06:23:33.506 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:33.492+0000 7f5a8bfff700 1 -- 172.21.15.42:0/1481705682 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5a84011850 con 0x7f5a9408df80 2019-12-05T06:23:33.506 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:33.492+0000 7f5a9ac67700 1 -- 172.21.15.42:0/1481705682 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5a9408df80 msgr2=0x7f5a9409f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:33.506 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:33.492+0000 7f5a9ac67700 1 --2- 172.21.15.42:0/1481705682 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5a9408df80 0x7f5a9409f2e0 secure :-1 s=READY pgs=1913 cs=0 l=1 rx=0x7f5a84008790 tx=0x7f5a8400fbf0).stop 2019-12-05T06:23:33.506 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:33.492+0000 7f5a9ac67700 1 -- 172.21.15.42:0/1481705682 shutdown_connections 2019-12-05T06:23:33.507 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:33.496+0000 7f5a9ac67700 1 --2- 172.21.15.42:0/1481705682 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5a9408df80 0x7f5a9409f2e0 unknown :-1 s=CLOSED pgs=1913 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:33.507 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:33.496+0000 7f5a9ac67700 1 -- 172.21.15.42:0/1481705682 shutdown_connections 2019-12-05T06:23:33.507 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:33.496+0000 7f5a9ac67700 1 -- 172.21.15.42:0/1481705682 wait complete. 2019-12-05T06:23:33.507 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:33.496+0000 7f5a9ac67700 1 -- 172.21.15.42:0/1481705682 >> 172.21.15.42:0/1481705682 conn(0x7f5a940aabe0 msgr2=0x7f5a940a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:23:33.513 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:33.496+0000 7f5a9ac67700 1 Processor -- start 2019-12-05T06:23:33.513 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:33.496+0000 7f5a9ac67700 1 -- start start 2019-12-05T06:23:33.513 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:33.500+0000 7f5a9ac67700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5a941120f0 0x7f5a94129f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:23:33.514 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:33.500+0000 7f5a9ac67700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a940a6980 con 0x7f5a940a4840 2019-12-05T06:23:33.514 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:33.500+0000 7f5a9ac67700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a94118370 con 0x7f5a94119d70 2019-12-05T06:23:33.514 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:33.500+0000 7f5a9ac67700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a9411b0c0 con 0x7f5a941120f0 2019-12-05T06:23:33.514 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:33.500+0000 7f5a99204700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f5a94119d70 0x7f5a940a6180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:54416/0 (socket says 172.21.15.42:54416) 2019-12-05T06:23:33.514 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:33.500+0000 7f5a99204700 1 -- 172.21.15.42:0/1433116630 learned_addr learned my addr 172.21.15.42:0/1433116630 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:23:33.514 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:33.500+0000 7f5a99a05700 1 --2- 172.21.15.42:0/1433116630 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5a941120f0 0x7f5a94129f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:33.514 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:33.500+0000 7f5a89ffb700 1 -- 172.21.15.42:0/1433116630 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2598690587 0 0) 0x7f5a94118370 con 0x7f5a94119d70 2019-12-05T06:23:33.515 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:33.500+0000 7f5a89ffb700 1 -- 172.21.15.42:0/1433116630 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a740030b0 con 0x7f5a94119d70 2019-12-05T06:23:33.515 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:33.500+0000 7f5a89ffb700 1 -- 172.21.15.42:0/1433116630 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2636046102 0 0) 0x7f5a940a6980 con 0x7f5a940a4840 2019-12-05T06:23:33.515 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:33.500+0000 7f5a89ffb700 1 -- 172.21.15.42:0/1433116630 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a94118370 con 0x7f5a940a4840 2019-12-05T06:23:33.515 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:33.500+0000 7f5a99a05700 1 -- 172.21.15.42:0/1433116630 >> v1:172.21.15.42:6789/0 conn(0x7f5a94119d70 legacy=0x7f5a940a6180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:33.515 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:33.500+0000 7f5a99a05700 1 -- 172.21.15.42:0/1433116630 >> v1:172.21.15.40:6789/0 conn(0x7f5a940a4840 legacy=0x7f5a94129730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:33.515 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:33.500+0000 7f5a99a05700 1 -- 172.21.15.42:0/1433116630 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a940a9770 con 0x7f5a941120f0 2019-12-05T06:23:33.515 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:33.500+0000 7f5a99a05700 1 --2- 172.21.15.42:0/1433116630 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5a941120f0 0x7f5a94129f30 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f5a8c004720 tx=0x7f5a8c010b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:33.516 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:33.500+0000 7f5a99a05700 1 --2- 172.21.15.42:0/1433116630 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5a941120f0 0x7f5a94129f30 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f5a8c004720 tx=0x7f5a8c010b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:33.517 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:33.500+0000 7f5a99a05700 1 --2- 172.21.15.42:0/1433116630 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5a941120f0 0x7f5a94129f30 secure :-1 s=READY pgs=553 cs=0 l=1 rx=0x7f5a8c004720 tx=0x7f5a8c010b30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:23:33.517 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:33.500+0000 7f5a99a05700 1 --2- 172.21.15.42:0/1433116630 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5a941120f0 0x7f5a94129f30 secure :-1 s=THROTTLE_DONE pgs=553 cs=0 l=1 rx=0x7f5a8c004720 tx=0x7f5a8c010b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:33.517 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:33.500+0000 7f5a89ffb700 1 -- 172.21.15.42:0/1433116630 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5a8c0121f0 con 0x7f5a941120f0 2019-12-05T06:23:33.517 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:33.500+0000 7f5a99a05700 1 --2- 172.21.15.42:0/1433116630 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5a941120f0 0x7f5a94129f30 secure :-1 s=THROTTLE_DONE pgs=553 cs=0 l=1 rx=0x7f5a8c004720 tx=0x7f5a8c010b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:33.517 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:33.500+0000 7f5a9ac67700 1 -- 172.21.15.42:0/1433116630 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a940a99e0 con 0x7f5a941120f0 2019-12-05T06:23:33.518 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:33.500+0000 7f5a99a05700 1 --2- 172.21.15.42:0/1433116630 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5a941120f0 0x7f5a94129f30 secure :-1 s=THROTTLE_DONE pgs=553 cs=0 l=1 rx=0x7f5a8c004720 tx=0x7f5a8c010b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:33.518 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:33.500+0000 7f5a89ffb700 1 -- 172.21.15.42:0/1433116630 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a8c005bd0 con 0x7f5a941120f0 2019-12-05T06:23:33.518 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:33.500+0000 7f5a89ffb700 1 -- 172.21.15.42:0/1433116630 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5a8c0127d0 con 0x7f5a941120f0 2019-12-05T06:23:33.518 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:33.500+0000 7f5a9ac67700 1 -- 172.21.15.42:0/1433116630 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a94134fd0 con 0x7f5a941120f0 2019-12-05T06:23:33.518 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:33.500+0000 7f5a99a05700 1 --2- 172.21.15.42:0/1433116630 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5a941120f0 0x7f5a94129f30 secure :-1 s=THROTTLE_DONE pgs=553 cs=0 l=1 rx=0x7f5a8c004720 tx=0x7f5a8c010b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:33.518 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:33.504+0000 7f5a89ffb700 1 -- 172.21.15.42:0/1433116630 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f5a8c0129d0 con 0x7f5a941120f0 2019-12-05T06:23:33.518 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:33.504+0000 7f5a99a05700 1 --2- 172.21.15.42:0/1433116630 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5a941120f0 0x7f5a94129f30 secure :-1 s=THROTTLE_DONE pgs=553 cs=0 l=1 rx=0x7f5a8c004720 tx=0x7f5a8c010b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:33.519 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:33.504+0000 7f5a89ffb700 1 --2- 172.21.15.42:0/1433116630 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5a7403b470 0x7f5a7403d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:23:33.519 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:33.504+0000 7f5a89ffb700 1 -- 172.21.15.42:0/1433116630 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 10675+0+0 (secure 0 0 0) 0x7f5a8c04b240 con 0x7f5a941120f0 2019-12-05T06:23:33.519 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:33.504+0000 7f5a99204700 1 --2- 172.21.15.42:0/1433116630 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5a7403b470 0x7f5a7403d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:33.519 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:33.504+0000 7f5a99204700 1 --2- 172.21.15.42:0/1433116630 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5a7403b470 0x7f5a7403d980 crc :-1 s=READY pgs=1063 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:23:33.519 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:33.504+0000 7f5a9ac67700 1 -- 172.21.15.42:0/1433116630 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a9405d110 con 0x7f5a941120f0 2019-12-05T06:23:33.520 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:33.512+0000 7f5a99a05700 1 --2- 172.21.15.42:0/1433116630 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5a941120f0 0x7f5a94129f30 secure :-1 s=THROTTLE_DONE pgs=553 cs=0 l=1 rx=0x7f5a8c004720 tx=0x7f5a8c010b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:33.520 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:33.512+0000 7f5a89ffb700 1 -- 172.21.15.42:0/1433116630 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f5a8c013060 con 0x7f5a941120f0 2019-12-05T06:23:33.694 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:33.688+0000 7f5a9ac67700 1 -- 172.21.15.42:0/1433116630 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5a9405b8c0 con 0x7f5a941120f0 2019-12-05T06:23:33.695 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:33.688+0000 7f5a99a05700 1 --2- 172.21.15.42:0/1433116630 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5a941120f0 0x7f5a94129f30 secure :-1 s=THROTTLE_DONE pgs=553 cs=0 l=1 rx=0x7f5a8c004720 tx=0x7f5a8c010b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:33.696 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:33.688+0000 7f5a89ffb700 1 -- 172.21.15.42:0/1433116630 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 303 v303) v1 ==== 95+0+2114 (secure 0 0 0) 0x7f5a8c0322c0 con 0x7f5a941120f0 2019-12-05T06:23:33.698 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:23:33.698 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":303,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":10993,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6835/3964426552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3964426552},{"type":"v1","addr":"172.21.15.42:6835","nonce":3964426552}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":303}],"filesystems":[{"mdsmap":{"epoch":302,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:21:41.099460+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":103,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10710},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10710":{"gid":10710,"name":"b","rank":0,"incarnation":299,"state":"up:active","state_seq":29,"addr":"172.21.15.40:6833/526579407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":526579407},{"type":"v1","addr":"172.21.15.40:6833","nonce":526579407}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:23:33.698 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 303 2019-12-05T06:23:33.701 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:33.696+0000 7f5a9ac67700 1 -- 172.21.15.42:0/1433116630 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5a7403b470 msgr2=0x7f5a7403d980 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:33.701 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:33.696+0000 7f5a9ac67700 1 --2- 172.21.15.42:0/1433116630 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5a7403b470 0x7f5a7403d980 crc :-1 s=READY pgs=1063 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:33.701 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:33.696+0000 7f5a9ac67700 1 -- 172.21.15.42:0/1433116630 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5a941120f0 msgr2=0x7f5a94129f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:33.701 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:33.696+0000 7f5a9ac67700 1 --2- 172.21.15.42:0/1433116630 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5a941120f0 0x7f5a94129f30 secure :-1 s=READY pgs=553 cs=0 l=1 rx=0x7f5a8c004720 tx=0x7f5a8c010b30).stop 2019-12-05T06:23:33.701 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:33.696+0000 7f5a9ac67700 1 -- 172.21.15.42:0/1433116630 shutdown_connections 2019-12-05T06:23:33.701 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:33.696+0000 7f5a9ac67700 1 --2- 172.21.15.42:0/1433116630 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5a7403b470 0x7f5a7403d980 unknown :-1 s=CLOSED pgs=1063 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:33.702 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:33.696+0000 7f5a9ac67700 1 --2- 172.21.15.42:0/1433116630 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5a941120f0 0x7f5a94129f30 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:33.702 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:33.696+0000 7f5a9ac67700 1 -- 172.21.15.42:0/1433116630 shutdown_connections 2019-12-05T06:23:33.702 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:33.696+0000 7f5a9ac67700 1 -- 172.21.15.42:0/1433116630 wait complete. 2019-12-05T06:23:33.702 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:33.696+0000 7f5a9ac67700 1 -- 172.21.15.42:0/1433116630 >> 172.21.15.42:0/1433116630 conn(0x7f5a940a7740 msgr2=0x7f5a9411bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:23:33.711 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:23:35.712 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:23:35.823 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:35.812+0000 7f48c8fd2700 1 Processor -- start 2019-12-05T06:23:35.823 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:35.812+0000 7f48c8fd2700 1 -- start start 2019-12-05T06:23:35.823 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:35.812+0000 7f48c8fd2700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f48c40a7740 0x7f48c408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:23:35.824 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:35.812+0000 7f48c8fd2700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f48c407e720 0x7f48c407eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:23:35.824 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:35.812+0000 7f48c8fd2700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48c40a0710 con 0x7f48c409f2e0 2019-12-05T06:23:35.824 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:35.812+0000 7f48c8fd2700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48c40a0b00 con 0x7f48c407e720 2019-12-05T06:23:35.824 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:35.812+0000 7f48c8fd2700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48c408e5b0 con 0x7f48c40a7740 2019-12-05T06:23:35.824 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:35.812+0000 7f48c359e700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f48c409f2e0 0x7f48c409f710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:54428/0 (socket says 172.21.15.42:54428) 2019-12-05T06:23:35.824 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:35.812+0000 7f48c359e700 1 -- 172.21.15.42:0/2894973215 learned_addr learned my addr 172.21.15.42:0/2894973215 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:23:35.824 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:35.816+0000 7f48c2d9d700 1 --2- 172.21.15.42:0/2894973215 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f48c40a7740 0x7f48c408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:35.825 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:35.816+0000 7f48c259c700 1 --2- 172.21.15.42:0/2894973215 >> v2:172.21.15.40:3300/0 conn(0x7f48c407e720 0x7f48c407eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:35.825 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:35.816+0000 7f48c1d9b700 1 -- 172.21.15.42:0/2894973215 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2816330613 0 0) 0x7f48c40a0710 con 0x7f48c409f2e0 2019-12-05T06:23:35.825 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:35.816+0000 7f48c1d9b700 1 -- 172.21.15.42:0/2894973215 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48b00030b0 con 0x7f48c409f2e0 2019-12-05T06:23:35.825 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:35.816+0000 7f48c2d9d700 1 -- 172.21.15.42:0/2894973215 >> v2:172.21.15.40:3300/0 conn(0x7f48c407e720 msgr2=0x7f48c407eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:23:35.825 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:35.816+0000 7f48c2d9d700 1 --2- 172.21.15.42:0/2894973215 >> v2:172.21.15.40:3300/0 conn(0x7f48c407e720 0x7f48c407eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:23:35.825 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:35.816+0000 7f48c2d9d700 1 -- 172.21.15.42:0/2894973215 >> v1:172.21.15.42:6789/0 conn(0x7f48c409f2e0 legacy=0x7f48c409f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:35.825 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:35.816+0000 7f48c2d9d700 1 -- 172.21.15.42:0/2894973215 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48c407f0a0 con 0x7f48c40a7740 2019-12-05T06:23:35.826 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:35.816+0000 7f48c259c700 1 --2- 172.21.15.42:0/2894973215 >> v2:172.21.15.40:3300/0 conn(0x7f48c407e720 0x7f48c407eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:23:35.826 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:35.816+0000 7f48c2d9d700 1 --2- 172.21.15.42:0/2894973215 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f48c40a7740 0x7f48c408df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f48bc008b10 tx=0x7f48bc00fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:35.826 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:35.816+0000 7f48c2d9d700 1 --2- 172.21.15.42:0/2894973215 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f48c40a7740 0x7f48c408df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f48bc008b10 tx=0x7f48bc00fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:35.826 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:35.816+0000 7f48c2d9d700 1 --2- 172.21.15.42:0/2894973215 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f48c40a7740 0x7f48c408df80 secure :-1 s=READY pgs=554 cs=0 l=1 rx=0x7f48bc008b10 tx=0x7f48bc00fff0).ready entity=mon.2 client_cookie=e0b3358ea8391f50 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:23:35.826 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:35.816+0000 7f48c2d9d700 1 --2- 172.21.15.42:0/2894973215 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f48c40a7740 0x7f48c408df80 secure :-1 s=THROTTLE_DONE pgs=554 cs=0 l=1 rx=0x7f48bc008b10 tx=0x7f48bc00fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:35.826 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:35.816+0000 7f48c1d9b700 1 -- 172.21.15.42:0/2894973215 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f48bc011a50 con 0x7f48c40a7740 2019-12-05T06:23:35.826 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:35.816+0000 7f48c2d9d700 1 --2- 172.21.15.42:0/2894973215 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f48c40a7740 0x7f48c408df80 secure :-1 s=THROTTLE_DONE pgs=554 cs=0 l=1 rx=0x7f48bc008b10 tx=0x7f48bc00fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:35.827 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:35.816+0000 7f48c2d9d700 1 --2- 172.21.15.42:0/2894973215 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f48c40a7740 0x7f48c408df80 secure :-1 s=THROTTLE_DONE pgs=554 cs=0 l=1 rx=0x7f48bc008b10 tx=0x7f48bc00fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:35.827 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:35.816+0000 7f48c1d9b700 1 -- 172.21.15.42:0/2894973215 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48bc011670 con 0x7f48c40a7740 2019-12-05T06:23:35.827 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:35.816+0000 7f48c1d9b700 1 -- 172.21.15.42:0/2894973215 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f48bc00a350 con 0x7f48c40a7740 2019-12-05T06:23:35.827 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:35.816+0000 7f48c8fd2700 1 -- 172.21.15.42:0/2894973215 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f48c40a7740 msgr2=0x7f48c408df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:35.827 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:35.816+0000 7f48c8fd2700 1 --2- 172.21.15.42:0/2894973215 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f48c40a7740 0x7f48c408df80 secure :-1 s=READY pgs=554 cs=0 l=1 rx=0x7f48bc008b10 tx=0x7f48bc00fff0).stop 2019-12-05T06:23:35.827 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:35.816+0000 7f48c8fd2700 1 -- 172.21.15.42:0/2894973215 shutdown_connections 2019-12-05T06:23:35.827 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:35.816+0000 7f48c8fd2700 1 --2- 172.21.15.42:0/2894973215 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f48c40a7740 0x7f48c408df80 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:35.827 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:35.816+0000 7f48c8fd2700 1 --2- 172.21.15.42:0/2894973215 >> v2:172.21.15.40:3300/0 conn(0x7f48c407e720 0x7f48c407eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:23:35.828 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:35.816+0000 7f48c8fd2700 1 -- 172.21.15.42:0/2894973215 shutdown_connections 2019-12-05T06:23:35.828 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:35.816+0000 7f48c8fd2700 1 -- 172.21.15.42:0/2894973215 wait complete. 2019-12-05T06:23:35.828 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:35.816+0000 7f48c8fd2700 1 -- 172.21.15.42:0/2894973215 >> 172.21.15.42:0/2894973215 conn(0x7f48c40aabe0 msgr2=0x7f48c40a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:23:35.831 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:35.820+0000 7f48c8fd2700 1 Processor -- start 2019-12-05T06:23:35.831 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:35.820+0000 7f48c8fd2700 1 -- start start 2019-12-05T06:23:35.831 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:35.820+0000 7f48c8fd2700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f48c40a1630 0x7f48c4144180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:23:35.831 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:35.820+0000 7f48c8fd2700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f48c40a1a60 0x7f48c412caf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:23:35.831 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:35.820+0000 7f48c8fd2700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48c41446d0 con 0x7f48c412d040 2019-12-05T06:23:35.831 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:35.820+0000 7f48c8fd2700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48c4143520 con 0x7f48c40a1630 2019-12-05T06:23:35.831 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:35.820+0000 7f48c8fd2700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48c41436e0 con 0x7f48c40a1a60 2019-12-05T06:23:35.832 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:35.820+0000 7f48c259c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f48c40a1a60 0x7f48c412caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:35.832 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:35.820+0000 7f48c259c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f48c40a1a60 0x7f48c412caf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:54396/0 (socket says 172.21.15.42:54396) 2019-12-05T06:23:35.832 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:35.820+0000 7f48c259c700 1 -- 172.21.15.42:0/1286880088 learned_addr learned my addr 172.21.15.42:0/1286880088 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:23:35.832 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:35.820+0000 7f48c2d9d700 1 --2- 172.21.15.42:0/1286880088 >> v2:172.21.15.40:3300/0 conn(0x7f48c40a1630 0x7f48c4144180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:35.832 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:35.820+0000 7f48c259c700 1 -- 172.21.15.42:0/1286880088 >> v2:172.21.15.40:3300/0 conn(0x7f48c40a1630 msgr2=0x7f48c4144180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:35.832 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:35.820+0000 7f48c259c700 1 --2- 172.21.15.42:0/1286880088 >> v2:172.21.15.40:3300/0 conn(0x7f48c40a1630 0x7f48c4144180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:35.832 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:35.820+0000 7f48c259c700 1 -- 172.21.15.42:0/1286880088 >> v1:172.21.15.42:6789/0 conn(0x7f48c412d040 legacy=0x7f48c4142d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:35.833 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:35.820+0000 7f48c259c700 1 -- 172.21.15.42:0/1286880088 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48c41438d0 con 0x7f48c40a1a60 2019-12-05T06:23:35.833 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:35.820+0000 7f48c259c700 1 --2- 172.21.15.42:0/1286880088 >> v2:172.21.15.42:3300/0 conn(0x7f48c40a1a60 0x7f48c412caf0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f48ac002ef0 tx=0x7f48ac010db0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:35.833 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:35.820+0000 7f48c259c700 1 --2- 172.21.15.42:0/1286880088 >> v2:172.21.15.42:3300/0 conn(0x7f48c40a1a60 0x7f48c412caf0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f48ac002ef0 tx=0x7f48ac010db0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:35.833 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:35.820+0000 7f48c259c700 1 --2- 172.21.15.42:0/1286880088 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f48c40a1a60 0x7f48c412caf0 secure :-1 s=READY pgs=2557 cs=0 l=1 rx=0x7f48ac002ef0 tx=0x7f48ac010db0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:23:35.833 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:35.820+0000 7f48c259c700 1 --2- 172.21.15.42:0/1286880088 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f48c40a1a60 0x7f48c412caf0 secure :-1 s=THROTTLE_DONE pgs=2557 cs=0 l=1 rx=0x7f48ac002ef0 tx=0x7f48ac010db0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:35.833 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:35.820+0000 7f48bb7fe700 1 -- 172.21.15.42:0/1286880088 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f48ac0058e0 con 0x7f48c40a1a60 2019-12-05T06:23:35.834 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:35.820+0000 7f48c259c700 1 --2- 172.21.15.42:0/1286880088 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f48c40a1a60 0x7f48c412caf0 secure :-1 s=THROTTLE_DONE pgs=2557 cs=0 l=1 rx=0x7f48ac002ef0 tx=0x7f48ac010db0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:35.834 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:35.820+0000 7f48c259c700 1 --2- 172.21.15.42:0/1286880088 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f48c40a1a60 0x7f48c412caf0 secure :-1 s=THROTTLE_DONE pgs=2557 cs=0 l=1 rx=0x7f48ac002ef0 tx=0x7f48ac010db0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:35.834 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:35.820+0000 7f48bb7fe700 1 -- 172.21.15.42:0/1286880088 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48ac007160 con 0x7f48c40a1a60 2019-12-05T06:23:35.834 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:35.820+0000 7f48c8fd2700 1 -- 172.21.15.42:0/1286880088 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48c4137500 con 0x7f48c40a1a60 2019-12-05T06:23:35.834 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:35.820+0000 7f48bb7fe700 1 -- 172.21.15.42:0/1286880088 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f48ac012430 con 0x7f48c40a1a60 2019-12-05T06:23:35.834 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:35.820+0000 7f48c8fd2700 1 -- 172.21.15.42:0/1286880088 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48c4137ab0 con 0x7f48c40a1a60 2019-12-05T06:23:35.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:35.824+0000 7f48c259c700 1 --2- 172.21.15.42:0/1286880088 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f48c40a1a60 0x7f48c412caf0 secure :-1 s=THROTTLE_DONE pgs=2557 cs=0 l=1 rx=0x7f48ac002ef0 tx=0x7f48ac010db0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:35.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:35.824+0000 7f48bb7fe700 1 -- 172.21.15.42:0/1286880088 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f48ac010180 con 0x7f48c40a1a60 2019-12-05T06:23:35.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:35.824+0000 7f48c259c700 1 --2- 172.21.15.42:0/1286880088 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f48c40a1a60 0x7f48c412caf0 secure :-1 s=THROTTLE_DONE pgs=2557 cs=0 l=1 rx=0x7f48ac002ef0 tx=0x7f48ac010db0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:35.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:35.824+0000 7f48bb7fe700 1 --2- 172.21.15.42:0/1286880088 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f48a803ae40 0x7f48a803d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:23:35.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:35.824+0000 7f48bb7fe700 1 -- 172.21.15.42:0/1286880088 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 10675+0+0 (secure 0 0 0) 0x7f48ac04b060 con 0x7f48c40a1a60 2019-12-05T06:23:35.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:35.824+0000 7f48c2d9d700 1 --2- 172.21.15.42:0/1286880088 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f48a803ae40 0x7f48a803d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:35.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:35.824+0000 7f48c2d9d700 1 --2- 172.21.15.42:0/1286880088 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f48a803ae40 0x7f48a803d350 crc :-1 s=READY pgs=1064 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:23:35.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:35.828+0000 7f48c8fd2700 1 -- 172.21.15.42:0/1286880088 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48c405d110 con 0x7f48c40a1a60 2019-12-05T06:23:35.841 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:35.836+0000 7f48c259c700 1 --2- 172.21.15.42:0/1286880088 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f48c40a1a60 0x7f48c412caf0 secure :-1 s=THROTTLE_DONE pgs=2557 cs=0 l=1 rx=0x7f48ac002ef0 tx=0x7f48ac010db0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:35.842 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:35.836+0000 7f48bb7fe700 1 -- 172.21.15.42:0/1286880088 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f48ac032340 con 0x7f48c40a1a60 2019-12-05T06:23:36.017 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:36.012+0000 7f48c8fd2700 1 -- 172.21.15.42:0/1286880088 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f48c405b8c0 con 0x7f48c40a1a60 2019-12-05T06:23:36.018 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:36.012+0000 7f48c259c700 1 --2- 172.21.15.42:0/1286880088 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f48c40a1a60 0x7f48c412caf0 secure :-1 s=THROTTLE_DONE pgs=2557 cs=0 l=1 rx=0x7f48ac002ef0 tx=0x7f48ac010db0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:36.019 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:36.012+0000 7f48bb7fe700 1 -- 172.21.15.42:0/1286880088 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 303 v303) v1 ==== 95+0+2114 (secure 0 0 0) 0x7f48ac032ef0 con 0x7f48c40a1a60 2019-12-05T06:23:36.019 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 303 2019-12-05T06:23:36.019 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:23:36.019 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":303,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":10993,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6835/3964426552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3964426552},{"type":"v1","addr":"172.21.15.42:6835","nonce":3964426552}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":303}],"filesystems":[{"mdsmap":{"epoch":302,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:21:41.099460+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":103,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10710},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10710":{"gid":10710,"name":"b","rank":0,"incarnation":299,"state":"up:active","state_seq":29,"addr":"172.21.15.40:6833/526579407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":526579407},{"type":"v1","addr":"172.21.15.40:6833","nonce":526579407}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:23:36.021 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:36.016+0000 7f48c8fd2700 1 -- 172.21.15.42:0/1286880088 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f48a803ae40 msgr2=0x7f48a803d350 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:36.021 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:36.016+0000 7f48c8fd2700 1 --2- 172.21.15.42:0/1286880088 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f48a803ae40 0x7f48a803d350 crc :-1 s=READY pgs=1064 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:36.022 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:36.016+0000 7f48c8fd2700 1 -- 172.21.15.42:0/1286880088 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f48c40a1a60 msgr2=0x7f48c412caf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:36.022 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:36.016+0000 7f48c8fd2700 1 --2- 172.21.15.42:0/1286880088 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f48c40a1a60 0x7f48c412caf0 secure :-1 s=READY pgs=2557 cs=0 l=1 rx=0x7f48ac002ef0 tx=0x7f48ac010db0).stop 2019-12-05T06:23:36.022 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:36.016+0000 7f48c8fd2700 1 -- 172.21.15.42:0/1286880088 shutdown_connections 2019-12-05T06:23:36.022 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:36.016+0000 7f48c8fd2700 1 --2- 172.21.15.42:0/1286880088 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f48a803ae40 0x7f48a803d350 unknown :-1 s=CLOSED pgs=1064 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:36.022 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:36.016+0000 7f48c8fd2700 1 --2- 172.21.15.42:0/1286880088 >> v2:172.21.15.40:3300/0 conn(0x7f48c40a1630 0x7f48c4144180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:36.022 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:36.016+0000 7f48c8fd2700 1 --2- 172.21.15.42:0/1286880088 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f48c40a1a60 0x7f48c412caf0 unknown :-1 s=CLOSED pgs=2557 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:36.022 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:36.016+0000 7f48c8fd2700 1 -- 172.21.15.42:0/1286880088 shutdown_connections 2019-12-05T06:23:36.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:36.016+0000 7f48c8fd2700 1 -- 172.21.15.42:0/1286880088 wait complete. 2019-12-05T06:23:36.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:36.016+0000 7f48c8fd2700 1 -- 172.21.15.42:0/1286880088 >> 172.21.15.42:0/1286880088 conn(0x7f48c407e5d0 msgr2=0x7f48c40a9790 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:23:36.031 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:23:38.033 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:23:38.143 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:38.132+0000 7f9f211eb700 1 Processor -- start 2019-12-05T06:23:38.143 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:38.132+0000 7f9f211eb700 1 -- start start 2019-12-05T06:23:38.143 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:38.132+0000 7f9f211eb700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f9f1c0a7740 0x7f9f1c08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:23:38.143 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:38.132+0000 7f9f211eb700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f9f1c09fae0 0x7f9f1c07eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:23:38.143 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:38.132+0000 7f9f211eb700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f1c0a0710 con 0x7f9f1c07e720 2019-12-05T06:23:38.144 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:38.132+0000 7f9f211eb700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f1c0a0b00 con 0x7f9f1c0a7740 2019-12-05T06:23:38.144 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:38.132+0000 7f9f211eb700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f1c08e5b0 con 0x7f9f1c09fae0 2019-12-05T06:23:38.144 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:38.132+0000 7f9f209ea700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f9f1c09fae0 0x7f9f1c07eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:38.144 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:38.132+0000 7f9f1ad9d700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f9f1c07e720 0x7f9f1c09f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:54440/0 (socket says 172.21.15.42:54440) 2019-12-05T06:23:38.144 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:38.132+0000 7f9f1ad9d700 1 -- 172.21.15.42:0/252281256 learned_addr learned my addr 172.21.15.42:0/252281256 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:23:38.144 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:38.132+0000 7f9f1b59e700 1 --2- 172.21.15.42:0/252281256 >> v2:172.21.15.40:3300/0 conn(0x7f9f1c0a7740 0x7f9f1c08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:38.144 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:38.132+0000 7f9f209ea700 1 -- 172.21.15.42:0/252281256 >> v2:172.21.15.40:3300/0 conn(0x7f9f1c0a7740 msgr2=0x7f9f1c08df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:23:38.145 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:38.132+0000 7f9f209ea700 1 --2- 172.21.15.42:0/252281256 >> v2:172.21.15.40:3300/0 conn(0x7f9f1c0a7740 0x7f9f1c08df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:23:38.145 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:38.132+0000 7f9f209ea700 1 -- 172.21.15.42:0/252281256 >> v1:172.21.15.42:6789/0 conn(0x7f9f1c07e720 legacy=0x7f9f1c09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:38.145 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:38.132+0000 7f9f1a59c700 1 -- 172.21.15.42:0/252281256 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2611113265 0 0) 0x7f9f1c0a0710 con 0x7f9f1c07e720 2019-12-05T06:23:38.145 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:38.132+0000 7f9f209ea700 1 -- 172.21.15.42:0/252281256 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f1c07f0a0 con 0x7f9f1c09fae0 2019-12-05T06:23:38.145 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:38.132+0000 7f9f209ea700 1 --2- 172.21.15.42:0/252281256 >> v2:172.21.15.42:3300/0 conn(0x7f9f1c09fae0 0x7f9f1c07eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f9f140032b0 tx=0x7f9f1400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:38.145 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:38.136+0000 7f9f209ea700 1 --2- 172.21.15.42:0/252281256 >> v2:172.21.15.42:3300/0 conn(0x7f9f1c09fae0 0x7f9f1c07eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f9f140032b0 tx=0x7f9f1400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:38.146 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:38.136+0000 7f9f209ea700 1 --2- 172.21.15.42:0/252281256 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9f1c09fae0 0x7f9f1c07eb50 secure :-1 s=READY pgs=2559 cs=0 l=1 rx=0x7f9f140032b0 tx=0x7f9f1400fff0).ready entity=mon.1 client_cookie=c344f7eb3cd043b7 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:23:38.146 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:38.136+0000 7f9f209ea700 1 --2- 172.21.15.42:0/252281256 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9f1c09fae0 0x7f9f1c07eb50 secure :-1 s=THROTTLE_DONE pgs=2559 cs=0 l=1 rx=0x7f9f140032b0 tx=0x7f9f1400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:38.146 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:38.136+0000 7f9f1a59c700 1 -- 172.21.15.42:0/252281256 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9f140108c0 con 0x7f9f1c09fae0 2019-12-05T06:23:38.146 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:38.136+0000 7f9f209ea700 1 --2- 172.21.15.42:0/252281256 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9f1c09fae0 0x7f9f1c07eb50 secure :-1 s=THROTTLE_DONE pgs=2559 cs=0 l=1 rx=0x7f9f140032b0 tx=0x7f9f1400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:38.146 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:38.136+0000 7f9f209ea700 1 --2- 172.21.15.42:0/252281256 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9f1c09fae0 0x7f9f1c07eb50 secure :-1 s=THROTTLE_DONE pgs=2559 cs=0 l=1 rx=0x7f9f140032b0 tx=0x7f9f1400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:38.147 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:38.136+0000 7f9f1a59c700 1 -- 172.21.15.42:0/252281256 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f14008b50 con 0x7f9f1c09fae0 2019-12-05T06:23:38.147 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:38.136+0000 7f9f1a59c700 1 -- 172.21.15.42:0/252281256 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9f140110c0 con 0x7f9f1c09fae0 2019-12-05T06:23:38.147 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:38.136+0000 7f9f211eb700 1 -- 172.21.15.42:0/252281256 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9f1c09fae0 msgr2=0x7f9f1c07eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:38.147 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:38.136+0000 7f9f211eb700 1 --2- 172.21.15.42:0/252281256 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9f1c09fae0 0x7f9f1c07eb50 secure :-1 s=READY pgs=2559 cs=0 l=1 rx=0x7f9f140032b0 tx=0x7f9f1400fff0).stop 2019-12-05T06:23:38.147 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:38.136+0000 7f9f211eb700 1 -- 172.21.15.42:0/252281256 shutdown_connections 2019-12-05T06:23:38.147 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:38.136+0000 7f9f211eb700 1 --2- 172.21.15.42:0/252281256 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9f1c09fae0 0x7f9f1c07eb50 unknown :-1 s=CLOSED pgs=2559 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:38.147 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:38.136+0000 7f9f211eb700 1 --2- 172.21.15.42:0/252281256 >> v2:172.21.15.40:3300/0 conn(0x7f9f1c0a7740 0x7f9f1c08df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:23:38.148 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:38.136+0000 7f9f211eb700 1 -- 172.21.15.42:0/252281256 shutdown_connections 2019-12-05T06:23:38.148 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:38.136+0000 7f9f211eb700 1 -- 172.21.15.42:0/252281256 wait complete. 2019-12-05T06:23:38.148 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:38.136+0000 7f9f211eb700 1 -- 172.21.15.42:0/252281256 >> 172.21.15.42:0/252281256 conn(0x7f9f1c0aabe0 msgr2=0x7f9f1c0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:23:38.151 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:38.140+0000 7f9f211eb700 1 Processor -- start 2019-12-05T06:23:38.151 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:38.140+0000 7f9f211eb700 1 -- start start 2019-12-05T06:23:38.151 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:38.140+0000 7f9f211eb700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f9f1c0a1510 0x7f9f1c11bc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:23:38.151 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:38.140+0000 7f9f211eb700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f9f1c0a2900 0x7f9f1c0a2d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:23:38.151 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:38.140+0000 7f9f211eb700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f1c11c150 con 0x7f9f1c0a5d00 2019-12-05T06:23:38.152 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:38.140+0000 7f9f211eb700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f1c0a1a80 con 0x7f9f1c0a2900 2019-12-05T06:23:38.152 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:38.140+0000 7f9f211eb700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f1c0a1c40 con 0x7f9f1c0a1510 2019-12-05T06:23:38.152 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:38.140+0000 7f9f1ad9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f9f1c0a1510 0x7f9f1c11bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:38.152 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:38.140+0000 7f9f1ad9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f9f1c0a1510 0x7f9f1c11bc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:54410/0 (socket says 172.21.15.42:54410) 2019-12-05T06:23:38.152 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:38.140+0000 7f9f1ad9d700 1 -- 172.21.15.42:0/4130040042 learned_addr learned my addr 172.21.15.42:0/4130040042 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:23:38.152 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:38.140+0000 7f9f1b59e700 1 --1- 172.21.15.42:0/4130040042 >> v1:172.21.15.42:6789/0 conn(0x7f9f1c0a5d00 0x7f9f1c0a3570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:54446/0 (socket says 172.21.15.42:54446) 2019-12-05T06:23:38.152 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:38.140+0000 7f9f209ea700 1 --2- 172.21.15.42:0/4130040042 >> v2:172.21.15.40:3300/0 conn(0x7f9f1c0a2900 0x7f9f1c0a2d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:38.153 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:38.140+0000 7f9f1ad9d700 1 -- 172.21.15.42:0/4130040042 >> v2:172.21.15.40:3300/0 conn(0x7f9f1c0a2900 msgr2=0x7f9f1c0a2d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:38.153 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:38.140+0000 7f9f1ad9d700 1 --2- 172.21.15.42:0/4130040042 >> v2:172.21.15.40:3300/0 conn(0x7f9f1c0a2900 0x7f9f1c0a2d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:38.153 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:38.140+0000 7f9f1ad9d700 1 -- 172.21.15.42:0/4130040042 >> v1:172.21.15.42:6789/0 conn(0x7f9f1c0a5d00 legacy=0x7f9f1c0a3570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:38.153 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:38.140+0000 7f9f1ad9d700 1 -- 172.21.15.42:0/4130040042 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f1c132a40 con 0x7f9f1c0a1510 2019-12-05T06:23:38.153 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:38.140+0000 7f9f1ad9d700 1 --2- 172.21.15.42:0/4130040042 >> v2:172.21.15.42:3300/0 conn(0x7f9f1c0a1510 0x7f9f1c11bc00 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f9f100043a0 tx=0x7f9f10010940).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:38.153 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:38.140+0000 7f9f1ad9d700 1 --2- 172.21.15.42:0/4130040042 >> v2:172.21.15.42:3300/0 conn(0x7f9f1c0a1510 0x7f9f1c11bc00 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f9f100043a0 tx=0x7f9f10010940).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:38.153 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:38.140+0000 7f9f1ad9d700 1 --2- 172.21.15.42:0/4130040042 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9f1c0a1510 0x7f9f1c11bc00 secure :-1 s=READY pgs=2561 cs=0 l=1 rx=0x7f9f100043a0 tx=0x7f9f10010940).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:23:38.154 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:38.140+0000 7f9f1ad9d700 1 --2- 172.21.15.42:0/4130040042 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9f1c0a1510 0x7f9f1c11bc00 secure :-1 s=THROTTLE_DONE pgs=2561 cs=0 l=1 rx=0x7f9f100043a0 tx=0x7f9f10010940).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:38.154 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:38.140+0000 7f9f1ad9d700 1 --2- 172.21.15.42:0/4130040042 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9f1c0a1510 0x7f9f1c11bc00 secure :-1 s=THROTTLE_DONE pgs=2561 cs=0 l=1 rx=0x7f9f100043a0 tx=0x7f9f10010940).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:38.154 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:38.140+0000 7f9f03fff700 1 -- 172.21.15.42:0/4130040042 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9f10011670 con 0x7f9f1c0a1510 2019-12-05T06:23:38.154 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:38.140+0000 7f9f1ad9d700 1 --2- 172.21.15.42:0/4130040042 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9f1c0a1510 0x7f9f1c11bc00 secure :-1 s=THROTTLE_DONE pgs=2561 cs=0 l=1 rx=0x7f9f100043a0 tx=0x7f9f10010940).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:38.154 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:38.140+0000 7f9f03fff700 1 -- 172.21.15.42:0/4130040042 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f10011290 con 0x7f9f1c0a1510 2019-12-05T06:23:38.154 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:38.140+0000 7f9f211eb700 1 -- 172.21.15.42:0/4130040042 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f1c12b680 con 0x7f9f1c0a1510 2019-12-05T06:23:38.154 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:38.140+0000 7f9f03fff700 1 -- 172.21.15.42:0/4130040042 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9f10011c90 con 0x7f9f1c0a1510 2019-12-05T06:23:38.155 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:38.140+0000 7f9f211eb700 1 -- 172.21.15.42:0/4130040042 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f1c12bb80 con 0x7f9f1c0a1510 2019-12-05T06:23:38.155 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:38.144+0000 7f9f1ad9d700 1 --2- 172.21.15.42:0/4130040042 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9f1c0a1510 0x7f9f1c11bc00 secure :-1 s=THROTTLE_DONE pgs=2561 cs=0 l=1 rx=0x7f9f100043a0 tx=0x7f9f10010940).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:38.155 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:38.144+0000 7f9f03fff700 1 -- 172.21.15.42:0/4130040042 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f9f1000b330 con 0x7f9f1c0a1510 2019-12-05T06:23:38.155 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:38.144+0000 7f9f1ad9d700 1 --2- 172.21.15.42:0/4130040042 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9f1c0a1510 0x7f9f1c11bc00 secure :-1 s=THROTTLE_DONE pgs=2561 cs=0 l=1 rx=0x7f9f100043a0 tx=0x7f9f10010940).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:38.155 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:38.144+0000 7f9f03fff700 1 --2- 172.21.15.42:0/4130040042 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9efc03adb0 0x7f9efc03d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:23:38.155 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:38.144+0000 7f9f03fff700 1 -- 172.21.15.42:0/4130040042 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 10675+0+0 (secure 0 0 0) 0x7f9f1004b210 con 0x7f9f1c0a1510 2019-12-05T06:23:38.155 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:38.144+0000 7f9f1b59e700 1 --2- 172.21.15.42:0/4130040042 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9efc03adb0 0x7f9efc03d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:38.156 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:38.144+0000 7f9f1b59e700 1 --2- 172.21.15.42:0/4130040042 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9efc03adb0 0x7f9efc03d2c0 crc :-1 s=READY pgs=1065 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:23:38.156 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:38.148+0000 7f9f211eb700 1 -- 172.21.15.42:0/4130040042 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f1c05d110 con 0x7f9f1c0a1510 2019-12-05T06:23:38.160 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:38.156+0000 7f9f1ad9d700 1 --2- 172.21.15.42:0/4130040042 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9f1c0a1510 0x7f9f1c11bc00 secure :-1 s=THROTTLE_DONE pgs=2561 cs=0 l=1 rx=0x7f9f100043a0 tx=0x7f9f10010940).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:38.160 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:38.156+0000 7f9f03fff700 1 -- 172.21.15.42:0/4130040042 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f9f10032290 con 0x7f9f1c0a1510 2019-12-05T06:23:38.334 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:38.328+0000 7f9f211eb700 1 -- 172.21.15.42:0/4130040042 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9f1c05b8c0 con 0x7f9f1c0a1510 2019-12-05T06:23:38.335 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:38.328+0000 7f9f1ad9d700 1 --2- 172.21.15.42:0/4130040042 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9f1c0a1510 0x7f9f1c11bc00 secure :-1 s=THROTTLE_DONE pgs=2561 cs=0 l=1 rx=0x7f9f100043a0 tx=0x7f9f10010940).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:38.335 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:38.328+0000 7f9f03fff700 1 -- 172.21.15.42:0/4130040042 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 303 v303) v1 ==== 95+0+2114 (secure 0 0 0) 0x7f9f100324f0 con 0x7f9f1c0a1510 2019-12-05T06:23:38.335 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:23:38.335 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":303,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":10993,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6835/3964426552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3964426552},{"type":"v1","addr":"172.21.15.42:6835","nonce":3964426552}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":303}],"filesystems":[{"mdsmap":{"epoch":302,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:21:41.099460+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":103,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10710},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10710":{"gid":10710,"name":"b","rank":0,"incarnation":299,"state":"up:active","state_seq":29,"addr":"172.21.15.40:6833/526579407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":526579407},{"type":"v1","addr":"172.21.15.40:6833","nonce":526579407}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:23:38.336 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 303 2019-12-05T06:23:38.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:38.332+0000 7f9f211eb700 1 -- 172.21.15.42:0/4130040042 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9efc03adb0 msgr2=0x7f9efc03d2c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:38.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:38.332+0000 7f9f211eb700 1 --2- 172.21.15.42:0/4130040042 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9efc03adb0 0x7f9efc03d2c0 crc :-1 s=READY pgs=1065 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:38.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:38.332+0000 7f9f211eb700 1 -- 172.21.15.42:0/4130040042 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9f1c0a1510 msgr2=0x7f9f1c11bc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:38.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:38.332+0000 7f9f211eb700 1 --2- 172.21.15.42:0/4130040042 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9f1c0a1510 0x7f9f1c11bc00 secure :-1 s=READY pgs=2561 cs=0 l=1 rx=0x7f9f100043a0 tx=0x7f9f10010940).stop 2019-12-05T06:23:38.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:38.332+0000 7f9f211eb700 1 -- 172.21.15.42:0/4130040042 shutdown_connections 2019-12-05T06:23:38.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:38.332+0000 7f9f211eb700 1 --2- 172.21.15.42:0/4130040042 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9efc03adb0 0x7f9efc03d2c0 unknown :-1 s=CLOSED pgs=1065 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:38.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:38.332+0000 7f9f211eb700 1 --2- 172.21.15.42:0/4130040042 >> v2:172.21.15.40:3300/0 conn(0x7f9f1c0a2900 0x7f9f1c0a2d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:38.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:38.332+0000 7f9f211eb700 1 --2- 172.21.15.42:0/4130040042 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9f1c0a1510 0x7f9f1c11bc00 unknown :-1 s=CLOSED pgs=2561 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:38.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:38.332+0000 7f9f211eb700 1 -- 172.21.15.42:0/4130040042 shutdown_connections 2019-12-05T06:23:38.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:38.332+0000 7f9f211eb700 1 -- 172.21.15.42:0/4130040042 wait complete. 2019-12-05T06:23:38.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:38.332+0000 7f9f211eb700 1 -- 172.21.15.42:0/4130040042 >> 172.21.15.42:0/4130040042 conn(0x7f9f1c0aabe0 msgr2=0x7f9f1c11c470 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:23:38.348 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:23:40.349 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:23:40.461 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:40.448+0000 7f9142aba700 1 Processor -- start 2019-12-05T06:23:40.463 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:40.448+0000 7f9142aba700 1 -- start start 2019-12-05T06:23:40.463 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:40.448+0000 7f9142aba700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f913c09fae0 0x7f913c14fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:23:40.464 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:40.448+0000 7f9142aba700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f913c0a0710 con 0x7f913c08df80 2019-12-05T06:23:40.464 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:40.448+0000 7f9142aba700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f913c07ef20 con 0x7f913c0a7740 2019-12-05T06:23:40.469 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:40.448+0000 7f9142aba700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f913c0a0b00 con 0x7f913c09fae0 2019-12-05T06:23:40.469 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:40.448+0000 7f9141057700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f913c0a7740 0x7f913c07e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:54454/0 (socket says 172.21.15.42:54454) 2019-12-05T06:23:40.470 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:40.448+0000 7f9141057700 1 -- 172.21.15.42:0/3527377743 learned_addr learned my addr 172.21.15.42:0/3527377743 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:23:40.470 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:40.452+0000 7f9141858700 1 --2- 172.21.15.42:0/3527377743 >> v2:172.21.15.40:3300/0 conn(0x7f913c09fae0 0x7f913c14fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:40.470 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:40.452+0000 7f9133fff700 1 -- 172.21.15.42:0/3527377743 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1102703157 0 0) 0x7f913c07ef20 con 0x7f913c0a7740 2019-12-05T06:23:40.470 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:40.452+0000 7f9133fff700 1 -- 172.21.15.42:0/3527377743 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91280030b0 con 0x7f913c0a7740 2019-12-05T06:23:40.471 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:40.452+0000 7f9141858700 1 -- 172.21.15.42:0/3527377743 >> v1:172.21.15.42:6789/0 conn(0x7f913c0a7740 legacy=0x7f913c07e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:40.471 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:40.452+0000 7f9141858700 1 -- 172.21.15.42:0/3527377743 >> v1:172.21.15.40:6789/0 conn(0x7f913c08df80 legacy=0x7f913c09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:40.471 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:40.452+0000 7f9133fff700 1 -- 172.21.15.42:0/3527377743 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2676502821 0 0) 0x7f913c0a0710 con 0x7f913c08df80 2019-12-05T06:23:40.471 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:40.452+0000 7f9141858700 1 -- 172.21.15.42:0/3527377743 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f913c0a9ba0 con 0x7f913c09fae0 2019-12-05T06:23:40.472 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:40.452+0000 7f9141858700 1 --2- 172.21.15.42:0/3527377743 >> v2:172.21.15.40:3300/0 conn(0x7f913c09fae0 0x7f913c14fff0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f9134008b10 tx=0x7f913400ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:40.472 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:40.452+0000 7f9141858700 1 --2- 172.21.15.42:0/3527377743 >> v2:172.21.15.40:3300/0 conn(0x7f913c09fae0 0x7f913c14fff0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f9134008b10 tx=0x7f913400ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:40.472 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:40.452+0000 7f9141858700 1 --2- 172.21.15.42:0/3527377743 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f913c09fae0 0x7f913c14fff0 secure :-1 s=READY pgs=1916 cs=0 l=1 rx=0x7f9134008b10 tx=0x7f913400ff70).ready entity=mon.0 client_cookie=295eb9e1ee810f10 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:23:40.473 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:40.452+0000 7f9141858700 1 --2- 172.21.15.42:0/3527377743 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f913c09fae0 0x7f913c14fff0 secure :-1 s=THROTTLE_DONE pgs=1916 cs=0 l=1 rx=0x7f9134008b10 tx=0x7f913400ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:40.473 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:40.452+0000 7f9133fff700 1 -- 172.21.15.42:0/3527377743 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9134010970 con 0x7f913c09fae0 2019-12-05T06:23:40.473 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:40.452+0000 7f9141858700 1 --2- 172.21.15.42:0/3527377743 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f913c09fae0 0x7f913c14fff0 secure :-1 s=THROTTLE_DONE pgs=1916 cs=0 l=1 rx=0x7f9134008b10 tx=0x7f913400ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:40.473 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:40.452+0000 7f9133fff700 1 -- 172.21.15.42:0/3527377743 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9134008b50 con 0x7f913c09fae0 2019-12-05T06:23:40.474 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:40.452+0000 7f9141858700 1 --2- 172.21.15.42:0/3527377743 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f913c09fae0 0x7f913c14fff0 secure :-1 s=THROTTLE_DONE pgs=1916 cs=0 l=1 rx=0x7f9134008b10 tx=0x7f913400ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:40.474 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:40.452+0000 7f9133fff700 1 -- 172.21.15.42:0/3527377743 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f91340110c0 con 0x7f913c09fae0 2019-12-05T06:23:40.474 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:40.452+0000 7f9142aba700 1 -- 172.21.15.42:0/3527377743 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f913c09fae0 msgr2=0x7f913c14fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:40.474 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:40.452+0000 7f9142aba700 1 --2- 172.21.15.42:0/3527377743 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f913c09fae0 0x7f913c14fff0 secure :-1 s=READY pgs=1916 cs=0 l=1 rx=0x7f9134008b10 tx=0x7f913400ff70).stop 2019-12-05T06:23:40.475 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:40.452+0000 7f9142aba700 1 -- 172.21.15.42:0/3527377743 shutdown_connections 2019-12-05T06:23:40.475 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:40.452+0000 7f9142aba700 1 --2- 172.21.15.42:0/3527377743 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f913c09fae0 0x7f913c14fff0 unknown :-1 s=CLOSED pgs=1916 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:40.475 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:40.452+0000 7f9142aba700 1 -- 172.21.15.42:0/3527377743 shutdown_connections 2019-12-05T06:23:40.475 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:40.452+0000 7f9142aba700 1 -- 172.21.15.42:0/3527377743 wait complete. 2019-12-05T06:23:40.475 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:40.452+0000 7f9142aba700 1 -- 172.21.15.42:0/3527377743 >> 172.21.15.42:0/3527377743 conn(0x7f913c0aabe0 msgr2=0x7f913c0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:23:40.476 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:40.456+0000 7f9142aba700 1 Processor -- start 2019-12-05T06:23:40.476 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:40.456+0000 7f9142aba700 1 -- start start 2019-12-05T06:23:40.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:40.456+0000 7f9142aba700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f913c08eeb0 0x7f913c0a3c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:23:40.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:40.456+0000 7f9142aba700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f913c12ac70 0x7f913c07c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:23:40.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:40.456+0000 7f9142aba700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f913c145840 con 0x7f913c12b0a0 2019-12-05T06:23:40.482 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:40.456+0000 7f9142aba700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f913c117d30 con 0x7f913c12ac70 2019-12-05T06:23:40.482 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:40.456+0000 7f9142aba700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f913c117fd0 con 0x7f913c08eeb0 2019-12-05T06:23:40.482 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:40.456+0000 7f9140856700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f913c12ac70 0x7f913c07c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:40.484 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:40.456+0000 7f9141858700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f913c12b0a0 0x7f913c12e230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:54464/0 (socket says 172.21.15.42:54464) 2019-12-05T06:23:40.485 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:40.456+0000 7f9141858700 1 -- 172.21.15.42:0/911885132 learned_addr learned my addr 172.21.15.42:0/911885132 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:23:40.485 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:40.456+0000 7f9141057700 1 --2- 172.21.15.42:0/911885132 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f913c08eeb0 0x7f913c0a3c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:40.485 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:40.456+0000 7f9140856700 1 -- 172.21.15.42:0/911885132 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f913c08eeb0 msgr2=0x7f913c0a3c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:40.485 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:40.456+0000 7f9140856700 1 --2- 172.21.15.42:0/911885132 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f913c08eeb0 0x7f913c0a3c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:40.485 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:40.456+0000 7f9131ffb700 1 -- 172.21.15.42:0/911885132 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1552536354 0 0) 0x7f913c145840 con 0x7f913c12b0a0 2019-12-05T06:23:40.485 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:40.456+0000 7f9140856700 1 -- 172.21.15.42:0/911885132 >> v1:172.21.15.42:6789/0 conn(0x7f913c12b0a0 legacy=0x7f913c12e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:40.486 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:40.456+0000 7f9140856700 1 -- 172.21.15.42:0/911885132 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f913c119da0 con 0x7f913c12ac70 2019-12-05T06:23:40.486 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:40.456+0000 7f9140856700 1 --2- 172.21.15.42:0/911885132 >> v2:172.21.15.42:3300/0 conn(0x7f913c12ac70 0x7f913c07c910 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f913c0a4640 tx=0x7f912c010870).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:40.486 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:40.456+0000 7f9140856700 1 --2- 172.21.15.42:0/911885132 >> v2:172.21.15.42:3300/0 conn(0x7f913c12ac70 0x7f913c07c910 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f913c0a4640 tx=0x7f912c010870).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:40.486 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:40.456+0000 7f9140856700 1 --2- 172.21.15.42:0/911885132 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f913c12ac70 0x7f913c07c910 secure :-1 s=READY pgs=2564 cs=0 l=1 rx=0x7f913c0a4640 tx=0x7f912c010870).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:23:40.486 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:40.456+0000 7f9140856700 1 --2- 172.21.15.42:0/911885132 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f913c12ac70 0x7f913c07c910 secure :-1 s=THROTTLE_DONE pgs=2564 cs=0 l=1 rx=0x7f913c0a4640 tx=0x7f912c010870).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:40.486 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:40.456+0000 7f9140856700 1 --2- 172.21.15.42:0/911885132 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f913c12ac70 0x7f913c07c910 secure :-1 s=THROTTLE_DONE pgs=2564 cs=0 l=1 rx=0x7f913c0a4640 tx=0x7f912c010870).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:40.486 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:40.456+0000 7f9131ffb700 1 -- 172.21.15.42:0/911885132 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f912c006a40 con 0x7f913c12ac70 2019-12-05T06:23:40.487 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:40.456+0000 7f9140856700 1 --2- 172.21.15.42:0/911885132 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f913c12ac70 0x7f913c07c910 secure :-1 s=THROTTLE_DONE pgs=2564 cs=0 l=1 rx=0x7f913c0a4640 tx=0x7f912c010870).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:40.487 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:40.456+0000 7f9131ffb700 1 -- 172.21.15.42:0/911885132 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f912c006660 con 0x7f913c12ac70 2019-12-05T06:23:40.487 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:40.456+0000 7f9142aba700 1 -- 172.21.15.42:0/911885132 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f913c118510 con 0x7f913c12ac70 2019-12-05T06:23:40.487 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:40.456+0000 7f9131ffb700 1 -- 172.21.15.42:0/911885132 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f912c011070 con 0x7f913c12ac70 2019-12-05T06:23:40.487 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:40.460+0000 7f9142aba700 1 -- 172.21.15.42:0/911885132 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f913c07ce60 con 0x7f913c12ac70 2019-12-05T06:23:40.487 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:40.460+0000 7f9140856700 1 --2- 172.21.15.42:0/911885132 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f913c12ac70 0x7f913c07c910 secure :-1 s=THROTTLE_DONE pgs=2564 cs=0 l=1 rx=0x7f913c0a4640 tx=0x7f912c010870).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:40.487 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:40.460+0000 7f9131ffb700 1 -- 172.21.15.42:0/911885132 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f912c011380 con 0x7f913c12ac70 2019-12-05T06:23:40.488 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:40.460+0000 7f9140856700 1 --2- 172.21.15.42:0/911885132 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f913c12ac70 0x7f913c07c910 secure :-1 s=THROTTLE_DONE pgs=2564 cs=0 l=1 rx=0x7f913c0a4640 tx=0x7f912c010870).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:40.488 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:40.460+0000 7f9131ffb700 1 --2- 172.21.15.42:0/911885132 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f911c03ada0 0x7f911c03d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:23:40.488 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:40.460+0000 7f9131ffb700 1 -- 172.21.15.42:0/911885132 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 10675+0+0 (secure 0 0 0) 0x7f912c04b040 con 0x7f913c12ac70 2019-12-05T06:23:40.488 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:40.460+0000 7f9141057700 1 --2- 172.21.15.42:0/911885132 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f911c03ada0 0x7f911c03d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:40.488 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:40.460+0000 7f9141057700 1 --2- 172.21.15.42:0/911885132 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f911c03ada0 0x7f911c03d2b0 crc :-1 s=READY pgs=1066 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:23:40.488 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:40.464+0000 7f9142aba700 1 -- 172.21.15.42:0/911885132 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f913c05d110 con 0x7f913c12ac70 2019-12-05T06:23:40.489 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:40.472+0000 7f9140856700 1 --2- 172.21.15.42:0/911885132 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f913c12ac70 0x7f913c07c910 secure :-1 s=THROTTLE_DONE pgs=2564 cs=0 l=1 rx=0x7f913c0a4640 tx=0x7f912c010870).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:40.489 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:40.472+0000 7f9131ffb700 1 -- 172.21.15.42:0/911885132 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f912c0320c0 con 0x7f913c12ac70 2019-12-05T06:23:40.647 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:40.640+0000 7f9142aba700 1 -- 172.21.15.42:0/911885132 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f913c05b8c0 con 0x7f913c12ac70 2019-12-05T06:23:40.650 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:40.644+0000 7f9140856700 1 -- 172.21.15.42:0/911885132 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f913c12ac70 msgr2=0x7f913c07c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2019-12-05T06:23:40.650 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:40.644+0000 7f9140856700 1 -- 172.21.15.42:0/911885132 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f913c12ac70 msgr2=0x7f913c07c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2019-12-05T06:23:40.650 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:40.644+0000 7f9140856700 1 --2- 172.21.15.42:0/911885132 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f913c12ac70 0x7f913c07c910 secure :-1 s=READY pgs=2564 cs=0 l=1 rx=0x7f913c0a4640 tx=0x7f912c010870).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-12-05T06:23:40.650 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:40.644+0000 7f9140856700 1 --2- 172.21.15.42:0/911885132 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f913c12ac70 0x7f913c07c910 secure :-1 s=READY pgs=2564 cs=0 l=1 rx=0x7f913c0a4640 tx=0x7f912c010870).stop 2019-12-05T06:23:40.650 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:40.644+0000 7f9131ffb700 1 -- 172.21.15.42:0/911885132 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f913c12ac70 msgr2=0x7f913c07c910 unknown :-1 s=STATE_CLOSED l=1).mark_down 2019-12-05T06:23:40.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:40.644+0000 7f9131ffb700 1 --2- 172.21.15.42:0/911885132 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f913c12ac70 0x7f913c07c910 unknown :-1 s=CLOSED pgs=2564 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:40.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:40.644+0000 7f9131ffb700 1 --2- 172.21.15.42:0/911885132 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f911c043ff0 0x7f911c046430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:23:40.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:40.644+0000 7f9131ffb700 1 --2- 172.21.15.42:0/911885132 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f911c046b40 0x7f911c048fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:23:40.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:40.644+0000 7f9131ffb700 1 --2- 172.21.15.42:0/911885132 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f911c049560 0x7f911c04ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:23:40.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:40.644+0000 7f9131ffb700 1 -- 172.21.15.42:0/911885132 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f911c04c1d0 con 0x7f911c043ff0 2019-12-05T06:23:40.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:40.644+0000 7f9131ffb700 1 -- 172.21.15.42:0/911885132 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f911c04c3c0 con 0x7f911c049560 2019-12-05T06:23:40.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:40.644+0000 7f9131ffb700 1 -- 172.21.15.42:0/911885132 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f911c050a10 con 0x7f911c046b40 2019-12-05T06:23:40.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:40.644+0000 7f9141057700 1 --2- 172.21.15.42:0/911885132 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f911c049560 0x7f911c04ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:40.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:40.644+0000 7f9141858700 1 --2- 172.21.15.42:0/911885132 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f911c046b40 0x7f911c048fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:40.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:40.644+0000 7f9140856700 1 --2- 172.21.15.42:0/911885132 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f911c043ff0 0x7f911c046430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:40.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:40.644+0000 7f9141057700 1 -- 172.21.15.42:0/911885132 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f911c046b40 msgr2=0x7f911c048fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:40.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:40.644+0000 7f9141057700 1 --2- 172.21.15.42:0/911885132 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f911c046b40 0x7f911c048fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:40.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:40.644+0000 7f9141057700 1 -- 172.21.15.42:0/911885132 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f911c043ff0 msgr2=0x7f911c046430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:40.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:40.644+0000 7f9141057700 1 --2- 172.21.15.42:0/911885132 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f911c043ff0 0x7f911c046430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:40.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:40.644+0000 7f9141057700 1 -- 172.21.15.42:0/911885132 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f911c050c50 con 0x7f911c049560 2019-12-05T06:23:40.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:40.644+0000 7f9141057700 1 -- 172.21.15.42:0/911885132 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9138011f40 con 0x7f911c049560 2019-12-05T06:23:40.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:40.644+0000 7f9141057700 1 --2- 172.21.15.42:0/911885132 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f911c049560 0x7f911c04ba40 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f9138000b80 tx=0x7f9138012530).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:40.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:40.644+0000 7f9141057700 1 --2- 172.21.15.42:0/911885132 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f911c049560 0x7f911c04ba40 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f9138000b80 tx=0x7f9138012530).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:40.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:40.644+0000 7f9141057700 1 --2- 172.21.15.42:0/911885132 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f911c049560 0x7f911c04ba40 secure :-1 s=READY pgs=2565 cs=0 l=1 rx=0x7f9138000b80 tx=0x7f9138012530).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:23:40.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:40.644+0000 7f9141057700 1 --2- 172.21.15.42:0/911885132 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f911c049560 0x7f911c04ba40 secure :-1 s=THROTTLE_DONE pgs=2565 cs=0 l=1 rx=0x7f9138000b80 tx=0x7f9138012530).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:40.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:40.644+0000 7f9141057700 1 --2- 172.21.15.42:0/911885132 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f911c049560 0x7f911c04ba40 secure :-1 s=THROTTLE_DONE pgs=2565 cs=0 l=1 rx=0x7f9138000b80 tx=0x7f9138012530).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:40.654 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:40.644+0000 7f9131ffb700 1 -- 172.21.15.42:0/911885132 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9138011f40 con 0x7f911c049560 2019-12-05T06:23:40.654 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:40.644+0000 7f9131ffb700 1 -- 172.21.15.42:0/911885132 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f913800c5b0 con 0x7f911c049560 2019-12-05T06:23:40.655 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:40.644+0000 7f9141057700 1 --2- 172.21.15.42:0/911885132 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f911c049560 0x7f911c04ba40 secure :-1 s=THROTTLE_DONE pgs=2565 cs=0 l=1 rx=0x7f9138000b80 tx=0x7f9138012530).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:40.655 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:40.644+0000 7f9131ffb700 1 -- 172.21.15.42:0/911885132 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f913800c990 con 0x7f911c049560 2019-12-05T06:23:40.655 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:40.644+0000 7f9141057700 1 --2- 172.21.15.42:0/911885132 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f911c049560 0x7f911c04ba40 secure :-1 s=THROTTLE_DONE pgs=2565 cs=0 l=1 rx=0x7f9138000b80 tx=0x7f9138012530).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:40.655 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:40.644+0000 7f9131ffb700 1 -- 172.21.15.42:0/911885132 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 303 v303) v1 ==== 95+0+2114 (secure 0 0 0) 0x7f913804bb30 con 0x7f911c049560 2019-12-05T06:23:40.655 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 303 2019-12-05T06:23:40.655 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:23:40.656 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":303,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":10993,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6835/3964426552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3964426552},{"type":"v1","addr":"172.21.15.42:6835","nonce":3964426552}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":303}],"filesystems":[{"mdsmap":{"epoch":302,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:21:41.099460+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":103,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10710},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10710":{"gid":10710,"name":"b","rank":0,"incarnation":299,"state":"up:active","state_seq":29,"addr":"172.21.15.40:6833/526579407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":526579407},{"type":"v1","addr":"172.21.15.40:6833","nonce":526579407}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:23:40.659 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:40.652+0000 7f9142aba700 1 -- 172.21.15.42:0/911885132 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f911c03ada0 msgr2=0x7f911c03d2b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:40.659 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:40.652+0000 7f9142aba700 1 --2- 172.21.15.42:0/911885132 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f911c03ada0 0x7f911c03d2b0 crc :-1 s=READY pgs=1066 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:40.659 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:40.652+0000 7f9142aba700 1 -- 172.21.15.42:0/911885132 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f911c049560 msgr2=0x7f911c04ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:40.659 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:40.652+0000 7f9142aba700 1 --2- 172.21.15.42:0/911885132 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f911c049560 0x7f911c04ba40 secure :-1 s=READY pgs=2565 cs=0 l=1 rx=0x7f9138000b80 tx=0x7f9138012530).stop 2019-12-05T06:23:40.659 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:40.652+0000 7f9141858700 1 -- 172.21.15.42:0/911885132 reap_dead start 2019-12-05T06:23:40.659 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:40.652+0000 7f9142aba700 1 -- 172.21.15.42:0/911885132 shutdown_connections 2019-12-05T06:23:40.659 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:40.652+0000 7f9142aba700 1 --2- 172.21.15.42:0/911885132 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f913c12ac70 0x7f913c07c910 unknown :-1 s=CLOSED pgs=2564 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:40.660 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:40.652+0000 7f9142aba700 1 -- 172.21.15.42:0/911885132 shutdown_connections 2019-12-05T06:23:40.660 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:40.652+0000 7f9142aba700 1 -- 172.21.15.42:0/911885132 wait complete. 2019-12-05T06:23:40.660 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:40.652+0000 7f9142aba700 1 -- 172.21.15.42:0/911885132 >> 172.21.15.42:0/911885132 conn(0x7f913c0a82a0 msgr2=0x7f913c07e720 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:23:40.672 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:23:42.674 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:23:42.789 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:42.768+0000 7f736b754700 1 Processor -- start 2019-12-05T06:23:42.790 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:42.768+0000 7f736b754700 1 -- start start 2019-12-05T06:23:42.790 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:42.768+0000 7f736b754700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f73640a7740 0x7f736408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:23:42.790 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:42.768+0000 7f736b754700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f736409fae0 0x7f736407eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:23:42.790 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:42.768+0000 7f736b754700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73640a0710 con 0x7f736407e720 2019-12-05T06:23:42.791 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:42.768+0000 7f736b754700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73640a0b00 con 0x7f73640a7740 2019-12-05T06:23:42.791 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:42.768+0000 7f736b754700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f736408e5b0 con 0x7f736409fae0 2019-12-05T06:23:42.792 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:42.768+0000 7f736a4f2700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f736409fae0 0x7f736407eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:42.792 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:42.768+0000 7f736a4f2700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f736409fae0 0x7f736407eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:54442/0 (socket says 172.21.15.42:54442) 2019-12-05T06:23:42.792 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:42.768+0000 7f736a4f2700 1 -- 172.21.15.42:0/3753169475 learned_addr learned my addr 172.21.15.42:0/3753169475 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:23:42.793 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:42.768+0000 7f7369cf1700 1 --2- 172.21.15.42:0/3753169475 >> v2:172.21.15.40:3300/0 conn(0x7f73640a7740 0x7f736408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:42.793 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:42.768+0000 7f736a4f2700 1 -- 172.21.15.42:0/3753169475 >> v2:172.21.15.40:3300/0 conn(0x7f73640a7740 msgr2=0x7f736408df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:23:42.793 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:42.768+0000 7f736a4f2700 1 --2- 172.21.15.42:0/3753169475 >> v2:172.21.15.40:3300/0 conn(0x7f73640a7740 0x7f736408df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:23:42.793 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:42.768+0000 7f736a4f2700 1 -- 172.21.15.42:0/3753169475 >> v1:172.21.15.40:6789/0 conn(0x7f736407e720 legacy=0x7f736409f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:42.794 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:42.768+0000 7f736a4f2700 1 -- 172.21.15.42:0/3753169475 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f736407f0a0 con 0x7f736409fae0 2019-12-05T06:23:42.794 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:42.768+0000 7f736a4f2700 1 --2- 172.21.15.42:0/3753169475 >> v2:172.21.15.42:3300/0 conn(0x7f736409fae0 0x7f736407eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f735c002f30 tx=0x7f735c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:42.794 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:42.768+0000 7f736a4f2700 1 --2- 172.21.15.42:0/3753169475 >> v2:172.21.15.42:3300/0 conn(0x7f736409fae0 0x7f736407eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f735c002f30 tx=0x7f735c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:42.794 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:42.768+0000 7f736a4f2700 1 --2- 172.21.15.42:0/3753169475 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f736409fae0 0x7f736407eb50 secure :-1 s=READY pgs=2566 cs=0 l=1 rx=0x7f735c002f30 tx=0x7f735c00fd30).ready entity=mon.1 client_cookie=4ea70ce637e4c3d server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:23:42.795 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:42.772+0000 7f736a4f2700 1 --2- 172.21.15.42:0/3753169475 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f736409fae0 0x7f736407eb50 secure :-1 s=THROTTLE_DONE pgs=2566 cs=0 l=1 rx=0x7f735c002f30 tx=0x7f735c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:42.795 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:42.772+0000 7f7368cef700 1 -- 172.21.15.42:0/3753169475 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f735c008c90 con 0x7f736409fae0 2019-12-05T06:23:42.795 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:42.772+0000 7f736a4f2700 1 --2- 172.21.15.42:0/3753169475 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f736409fae0 0x7f736407eb50 secure :-1 s=THROTTLE_DONE pgs=2566 cs=0 l=1 rx=0x7f735c002f30 tx=0x7f735c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:42.816 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:42.772+0000 7f7368cef700 1 -- 172.21.15.42:0/3753169475 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f735c0088b0 con 0x7f736409fae0 2019-12-05T06:23:42.816 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:42.772+0000 7f736a4f2700 1 --2- 172.21.15.42:0/3753169475 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f736409fae0 0x7f736407eb50 secure :-1 s=THROTTLE_DONE pgs=2566 cs=0 l=1 rx=0x7f735c002f30 tx=0x7f735c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:42.816 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:42.772+0000 7f736b754700 1 -- 172.21.15.42:0/3753169475 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f736409fae0 msgr2=0x7f736407eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:42.816 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:42.772+0000 7f7368cef700 1 -- 172.21.15.42:0/3753169475 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f735c011ad0 con 0x7f736409fae0 2019-12-05T06:23:42.817 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:42.772+0000 7f736b754700 1 --2- 172.21.15.42:0/3753169475 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f736409fae0 0x7f736407eb50 secure :-1 s=READY pgs=2566 cs=0 l=1 rx=0x7f735c002f30 tx=0x7f735c00fd30).stop 2019-12-05T06:23:42.817 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:42.772+0000 7f736b754700 1 -- 172.21.15.42:0/3753169475 shutdown_connections 2019-12-05T06:23:42.817 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:42.772+0000 7f736b754700 1 --2- 172.21.15.42:0/3753169475 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f736409fae0 0x7f736407eb50 unknown :-1 s=CLOSED pgs=2566 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:42.817 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:42.772+0000 7f736b754700 1 --2- 172.21.15.42:0/3753169475 >> v2:172.21.15.40:3300/0 conn(0x7f73640a7740 0x7f736408df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:23:42.817 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:42.772+0000 7f736b754700 1 -- 172.21.15.42:0/3753169475 shutdown_connections 2019-12-05T06:23:42.817 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:42.772+0000 7f736b754700 1 -- 172.21.15.42:0/3753169475 wait complete. 2019-12-05T06:23:42.817 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:42.772+0000 7f736b754700 1 -- 172.21.15.42:0/3753169475 >> 172.21.15.42:0/3753169475 conn(0x7f73640aabe0 msgr2=0x7f73640a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:23:42.818 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:42.772+0000 7f736b754700 1 Processor -- start 2019-12-05T06:23:42.818 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:42.772+0000 7f736b754700 1 -- start start 2019-12-05T06:23:42.818 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:42.776+0000 7f736b754700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f736407b9b0 0x7f7364078360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:23:42.818 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:42.776+0000 7f736b754700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f73640788b0 0x7f7364118470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:23:42.818 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:42.776+0000 7f736b754700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73640a68f0 con 0x7f73641189c0 2019-12-05T06:23:42.818 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:42.776+0000 7f736b754700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f736411cbc0 con 0x7f736407b9b0 2019-12-05T06:23:42.818 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:42.776+0000 7f736b754700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f736411ce60 con 0x7f73640788b0 2019-12-05T06:23:42.819 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:42.776+0000 7f73694f0700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f73640788b0 0x7f7364118470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:42.819 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:42.776+0000 7f73694f0700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f73640788b0 0x7f7364118470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:54446/0 (socket says 172.21.15.42:54446) 2019-12-05T06:23:42.819 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:42.776+0000 7f73694f0700 1 -- 172.21.15.42:0/3445342374 learned_addr learned my addr 172.21.15.42:0/3445342374 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:23:42.819 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:42.776+0000 7f7369cf1700 1 --2- 172.21.15.42:0/3445342374 >> v2:172.21.15.40:3300/0 conn(0x7f736407b9b0 0x7f7364078360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:42.819 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:42.776+0000 7f73694f0700 1 -- 172.21.15.42:0/3445342374 >> v2:172.21.15.40:3300/0 conn(0x7f736407b9b0 msgr2=0x7f7364078360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:42.819 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:42.776+0000 7f73694f0700 1 --2- 172.21.15.42:0/3445342374 >> v2:172.21.15.40:3300/0 conn(0x7f736407b9b0 0x7f7364078360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:42.819 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:42.776+0000 7f73694f0700 1 -- 172.21.15.42:0/3445342374 >> v1:172.21.15.40:6789/0 conn(0x7f73641189c0 legacy=0x7f736411a0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:42.820 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:42.776+0000 7f73694f0700 1 -- 172.21.15.42:0/3445342374 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f736411c720 con 0x7f73640788b0 2019-12-05T06:23:42.820 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:42.776+0000 7f73694f0700 1 --2- 172.21.15.42:0/3445342374 >> v2:172.21.15.42:3300/0 conn(0x7f73640788b0 0x7f7364118470 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f7360000e90 tx=0x7f73600109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:42.820 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:42.776+0000 7f73694f0700 1 --2- 172.21.15.42:0/3445342374 >> v2:172.21.15.42:3300/0 conn(0x7f73640788b0 0x7f7364118470 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f7360000e90 tx=0x7f73600109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:42.820 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:42.776+0000 7f73694f0700 1 --2- 172.21.15.42:0/3445342374 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f73640788b0 0x7f7364118470 secure :-1 s=READY pgs=2567 cs=0 l=1 rx=0x7f7360000e90 tx=0x7f73600109a0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:23:42.823 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:42.776+0000 7f73694f0700 1 --2- 172.21.15.42:0/3445342374 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f73640788b0 0x7f7364118470 secure :-1 s=THROTTLE_DONE pgs=2567 cs=0 l=1 rx=0x7f7360000e90 tx=0x7f73600109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:42.823 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:42.776+0000 7f73694f0700 1 --2- 172.21.15.42:0/3445342374 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f73640788b0 0x7f7364118470 secure :-1 s=THROTTLE_DONE pgs=2567 cs=0 l=1 rx=0x7f7360000e90 tx=0x7f73600109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:42.823 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:42.776+0000 7f735a7fc700 1 -- 172.21.15.42:0/3445342374 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f736000caa0 con 0x7f73640788b0 2019-12-05T06:23:42.823 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:42.776+0000 7f73694f0700 1 --2- 172.21.15.42:0/3445342374 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f73640788b0 0x7f7364118470 secure :-1 s=THROTTLE_DONE pgs=2567 cs=0 l=1 rx=0x7f7360000e90 tx=0x7f73600109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:42.824 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:42.776+0000 7f735a7fc700 1 -- 172.21.15.42:0/3445342374 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f736000c6c0 con 0x7f73640788b0 2019-12-05T06:23:42.824 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:42.776+0000 7f735a7fc700 1 -- 172.21.15.42:0/3445342374 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7360012210 con 0x7f73640788b0 2019-12-05T06:23:42.824 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:42.776+0000 7f736b754700 1 -- 172.21.15.42:0/3445342374 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f736411b530 con 0x7f73640788b0 2019-12-05T06:23:42.824 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:42.776+0000 7f736b754700 1 -- 172.21.15.42:0/3445342374 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f736411a8f0 con 0x7f73640788b0 2019-12-05T06:23:42.824 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:42.776+0000 7f73694f0700 1 --2- 172.21.15.42:0/3445342374 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f73640788b0 0x7f7364118470 secure :-1 s=THROTTLE_DONE pgs=2567 cs=0 l=1 rx=0x7f7360000e90 tx=0x7f73600109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:42.825 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:42.780+0000 7f735a7fc700 1 -- 172.21.15.42:0/3445342374 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f73600124e0 con 0x7f73640788b0 2019-12-05T06:23:42.825 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:42.780+0000 7f73694f0700 1 --2- 172.21.15.42:0/3445342374 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f73640788b0 0x7f7364118470 secure :-1 s=THROTTLE_DONE pgs=2567 cs=0 l=1 rx=0x7f7360000e90 tx=0x7f73600109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:42.825 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:42.780+0000 7f735a7fc700 1 --2- 172.21.15.42:0/3445342374 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f734403adb0 0x7f734403d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:23:42.825 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:42.780+0000 7f735a7fc700 1 -- 172.21.15.42:0/3445342374 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 10675+0+0 (secure 0 0 0) 0x7f736004b3a0 con 0x7f73640788b0 2019-12-05T06:23:42.826 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:42.780+0000 7f7369cf1700 1 --2- 172.21.15.42:0/3445342374 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f734403adb0 0x7f734403d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:42.826 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:42.780+0000 7f7369cf1700 1 --2- 172.21.15.42:0/3445342374 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f734403adb0 0x7f734403d2c0 crc :-1 s=READY pgs=1067 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:23:42.827 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:42.784+0000 7f736b754700 1 -- 172.21.15.42:0/3445342374 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7364118df0 con 0x7f73640788b0 2019-12-05T06:23:42.827 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:42.792+0000 7f73694f0700 1 --2- 172.21.15.42:0/3445342374 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f73640788b0 0x7f7364118470 secure :-1 s=THROTTLE_DONE pgs=2567 cs=0 l=1 rx=0x7f7360000e90 tx=0x7f73600109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:42.827 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:42.792+0000 7f735a7fc700 1 -- 172.21.15.42:0/3445342374 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f7360032420 con 0x7f73640788b0 2019-12-05T06:23:42.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:42.968+0000 7f736b754700 1 -- 172.21.15.42:0/3445342374 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f736405d110 con 0x7f73640788b0 2019-12-05T06:23:42.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:42.968+0000 7f73694f0700 1 --2- 172.21.15.42:0/3445342374 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f73640788b0 0x7f7364118470 secure :-1 s=THROTTLE_DONE pgs=2567 cs=0 l=1 rx=0x7f7360000e90 tx=0x7f73600109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:42.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:42.968+0000 7f735a7fc700 1 -- 172.21.15.42:0/3445342374 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 303 v303) v1 ==== 95+0+2114 (secure 0 0 0) 0x7f7360032680 con 0x7f73640788b0 2019-12-05T06:23:42.977 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:23:42.978 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":303,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":10993,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6835/3964426552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3964426552},{"type":"v1","addr":"172.21.15.42:6835","nonce":3964426552}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":303}],"filesystems":[{"mdsmap":{"epoch":302,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:21:41.099460+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":103,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10710},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10710":{"gid":10710,"name":"b","rank":0,"incarnation":299,"state":"up:active","state_seq":29,"addr":"172.21.15.40:6833/526579407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":526579407},{"type":"v1","addr":"172.21.15.40:6833","nonce":526579407}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:23:42.978 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 303 2019-12-05T06:23:42.980 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:42.972+0000 7f736b754700 1 -- 172.21.15.42:0/3445342374 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f734403adb0 msgr2=0x7f734403d2c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:42.980 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:42.972+0000 7f736b754700 1 --2- 172.21.15.42:0/3445342374 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f734403adb0 0x7f734403d2c0 crc :-1 s=READY pgs=1067 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:42.981 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:42.972+0000 7f736b754700 1 -- 172.21.15.42:0/3445342374 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f73640788b0 msgr2=0x7f7364118470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:42.981 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:42.972+0000 7f736b754700 1 --2- 172.21.15.42:0/3445342374 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f73640788b0 0x7f7364118470 secure :-1 s=READY pgs=2567 cs=0 l=1 rx=0x7f7360000e90 tx=0x7f73600109a0).stop 2019-12-05T06:23:42.981 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:42.972+0000 7f736b754700 1 -- 172.21.15.42:0/3445342374 shutdown_connections 2019-12-05T06:23:42.981 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:42.972+0000 7f736b754700 1 --2- 172.21.15.42:0/3445342374 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f734403adb0 0x7f734403d2c0 unknown :-1 s=CLOSED pgs=1067 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:42.981 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:42.972+0000 7f736b754700 1 --2- 172.21.15.42:0/3445342374 >> v2:172.21.15.40:3300/0 conn(0x7f736407b9b0 0x7f7364078360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:42.981 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:42.972+0000 7f736b754700 1 --2- 172.21.15.42:0/3445342374 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f73640788b0 0x7f7364118470 unknown :-1 s=CLOSED pgs=2567 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:42.982 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:42.972+0000 7f736b754700 1 -- 172.21.15.42:0/3445342374 shutdown_connections 2019-12-05T06:23:42.982 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:42.972+0000 7f736b754700 1 -- 172.21.15.42:0/3445342374 wait complete. 2019-12-05T06:23:42.982 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:42.972+0000 7f736b754700 1 -- 172.21.15.42:0/3445342374 >> 172.21.15.42:0/3445342374 conn(0x7f73640aabe0 msgr2=0x7f73640a7f90 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:23:42.998 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:23:43.343 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:23:43.347 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:23:44.999 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:23:45.109 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.096+0000 7f37b7b3a700 1 Processor -- start 2019-12-05T06:23:45.110 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.096+0000 7f37b7b3a700 1 -- start start 2019-12-05T06:23:45.110 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.100+0000 7f37b7b3a700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f37b008df80 0x7f37b009f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:23:45.110 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.100+0000 7f37b7b3a700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f37b009f830 0x7f37b014efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:23:45.111 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.100+0000 7f37b7b3a700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37b009fc60 con 0x7f37b00a7740 2019-12-05T06:23:45.111 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.100+0000 7f37b7b3a700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37b00a0b00 con 0x7f37b008df80 2019-12-05T06:23:45.111 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.100+0000 7f37b7b3a700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37b008e5b0 con 0x7f37b009f830 2019-12-05T06:23:45.111 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.100+0000 7f37b68d8700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f37b009f830 0x7f37b014efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:45.112 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.100+0000 7f37b58d6700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f37b008df80 0x7f37b009f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:45.112 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.100+0000 7f37b58d6700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f37b008df80 0x7f37b009f2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.42:42278/0 (socket says 172.21.15.42:42278) 2019-12-05T06:23:45.112 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.100+0000 7f37b68d8700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f37b009f830 0x7f37b014efe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:54932/0 (socket says 172.21.15.42:54932) 2019-12-05T06:23:45.112 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.100+0000 7f37b58d6700 1 -- 172.21.15.42:0/1336878775 learned_addr learned my addr 172.21.15.42:0/1336878775 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:23:45.113 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.100+0000 7f37b60d7700 1 --1- 172.21.15.42:0/1336878775 >> v1:172.21.15.40:6789/0 conn(0x7f37b00a7740 0x7f37b007e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:35710/0 (socket says 172.21.15.42:35710) 2019-12-05T06:23:45.113 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.100+0000 7f37b58d6700 1 -- 172.21.15.42:0/1336878775 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f37b009f830 msgr2=0x7f37b014efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:23:45.113 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.100+0000 7f37b58d6700 1 --2- 172.21.15.42:0/1336878775 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f37b009f830 0x7f37b014efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:23:45.114 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.100+0000 7f37b58d6700 1 -- 172.21.15.42:0/1336878775 >> v1:172.21.15.40:6789/0 conn(0x7f37b00a7740 legacy=0x7f37b007e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:45.114 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.100+0000 7f37b50d5700 1 -- 172.21.15.42:0/1336878775 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 630005502 0 0) 0x7f37b009fc60 con 0x7f37b00a7740 2019-12-05T06:23:45.114 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.100+0000 7f37b58d6700 1 -- 172.21.15.42:0/1336878775 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37b00a0710 con 0x7f37b008df80 2019-12-05T06:23:45.114 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.100+0000 7f37b58d6700 1 --2- 172.21.15.42:0/1336878775 >> v2:172.21.15.40:3300/0 conn(0x7f37b008df80 0x7f37b009f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f37a4008790 tx=0x7f37a400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:45.115 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.100+0000 7f37b58d6700 1 --2- 172.21.15.42:0/1336878775 >> v2:172.21.15.40:3300/0 conn(0x7f37b008df80 0x7f37b009f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f37a4008790 tx=0x7f37a400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:45.115 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.100+0000 7f37b58d6700 1 --2- 172.21.15.42:0/1336878775 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f37b008df80 0x7f37b009f2e0 secure :-1 s=READY pgs=1921 cs=0 l=1 rx=0x7f37a4008790 tx=0x7f37a400fd30).ready entity=mon.0 client_cookie=79343fe245cd238b server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:23:45.116 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.100+0000 7f37b58d6700 1 --2- 172.21.15.42:0/1336878775 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f37b008df80 0x7f37b009f2e0 secure :-1 s=THROTTLE_DONE pgs=1921 cs=0 l=1 rx=0x7f37a4008790 tx=0x7f37a400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:45.116 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.100+0000 7f37b50d5700 1 -- 172.21.15.42:0/1336878775 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f37a4008c90 con 0x7f37b008df80 2019-12-05T06:23:45.116 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.100+0000 7f37b58d6700 1 --2- 172.21.15.42:0/1336878775 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f37b008df80 0x7f37b009f2e0 secure :-1 s=THROTTLE_DONE pgs=1921 cs=0 l=1 rx=0x7f37a4008790 tx=0x7f37a400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:45.116 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.100+0000 7f37b50d5700 1 -- 172.21.15.42:0/1336878775 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37a40088b0 con 0x7f37b008df80 2019-12-05T06:23:45.117 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.100+0000 7f37b58d6700 1 --2- 172.21.15.42:0/1336878775 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f37b008df80 0x7f37b009f2e0 secure :-1 s=THROTTLE_DONE pgs=1921 cs=0 l=1 rx=0x7f37a4008790 tx=0x7f37a400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:45.117 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.100+0000 7f37b50d5700 1 -- 172.21.15.42:0/1336878775 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f37a4011b10 con 0x7f37b008df80 2019-12-05T06:23:45.117 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.100+0000 7f37b7b3a700 1 -- 172.21.15.42:0/1336878775 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f37b008df80 msgr2=0x7f37b009f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:45.117 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.100+0000 7f37b7b3a700 1 --2- 172.21.15.42:0/1336878775 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f37b008df80 0x7f37b009f2e0 secure :-1 s=READY pgs=1921 cs=0 l=1 rx=0x7f37a4008790 tx=0x7f37a400fd30).stop 2019-12-05T06:23:45.118 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.100+0000 7f37b7b3a700 1 -- 172.21.15.42:0/1336878775 shutdown_connections 2019-12-05T06:23:45.118 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.100+0000 7f37b7b3a700 1 --2- 172.21.15.42:0/1336878775 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f37b009f830 0x7f37b014efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:23:45.118 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.100+0000 7f37b7b3a700 1 --2- 172.21.15.42:0/1336878775 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f37b008df80 0x7f37b009f2e0 unknown :-1 s=CLOSED pgs=1921 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:45.118 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.100+0000 7f37b7b3a700 1 -- 172.21.15.42:0/1336878775 shutdown_connections 2019-12-05T06:23:45.119 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.100+0000 7f37b7b3a700 1 -- 172.21.15.42:0/1336878775 wait complete. 2019-12-05T06:23:45.119 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.100+0000 7f37b7b3a700 1 -- 172.21.15.42:0/1336878775 >> 172.21.15.42:0/1336878775 conn(0x7f37b00aabe0 msgr2=0x7f37b00a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:23:45.124 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.104+0000 7f37b7b3a700 1 Processor -- start 2019-12-05T06:23:45.125 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.104+0000 7f37b7b3a700 1 -- start start 2019-12-05T06:23:45.125 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.108+0000 7f37b7b3a700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f37b0116530 0x7f37b01337e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:23:45.125 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.108+0000 7f37b7b3a700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f37b011c010 0x7f37b00a6a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:23:45.125 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.108+0000 7f37b7b3a700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37b0120b30 con 0x7f37b008eb70 2019-12-05T06:23:45.126 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.108+0000 7f37b7b3a700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37b011a990 con 0x7f37b0116530 2019-12-05T06:23:45.126 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.108+0000 7f37b7b3a700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37b011ac30 con 0x7f37b011c010 2019-12-05T06:23:45.127 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.108+0000 7f37b58d6700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f37b011c010 0x7f37b00a6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:45.127 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.108+0000 7f37b58d6700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f37b011c010 0x7f37b00a6a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:54470/0 (socket says 172.21.15.42:54470) 2019-12-05T06:23:45.127 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.108+0000 7f37b58d6700 1 -- 172.21.15.42:0/542798766 learned_addr learned my addr 172.21.15.42:0/542798766 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:23:45.127 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.108+0000 7f37b60d7700 1 --2- 172.21.15.42:0/542798766 >> v2:172.21.15.40:3300/0 conn(0x7f37b0116530 0x7f37b01337e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:45.127 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.108+0000 7f37aeffd700 1 -- 172.21.15.42:0/542798766 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1166079093 0 0) 0x7f37b0120b30 con 0x7f37b008eb70 2019-12-05T06:23:45.128 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.108+0000 7f37aeffd700 1 -- 172.21.15.42:0/542798766 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37900030b0 con 0x7f37b008eb70 2019-12-05T06:23:45.128 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.108+0000 7f37b58d6700 1 -- 172.21.15.42:0/542798766 >> v2:172.21.15.40:3300/0 conn(0x7f37b0116530 msgr2=0x7f37b01337e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:45.128 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.108+0000 7f37b58d6700 1 --2- 172.21.15.42:0/542798766 >> v2:172.21.15.40:3300/0 conn(0x7f37b0116530 0x7f37b01337e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:45.128 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.108+0000 7f37b58d6700 1 -- 172.21.15.42:0/542798766 >> v1:172.21.15.40:6789/0 conn(0x7f37b008eb70 legacy=0x7f37b008efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:45.129 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.108+0000 7f37b58d6700 1 -- 172.21.15.42:0/542798766 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37b011a490 con 0x7f37b011c010 2019-12-05T06:23:45.129 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.108+0000 7f37b58d6700 1 --2- 172.21.15.42:0/542798766 >> v2:172.21.15.42:3300/0 conn(0x7f37b011c010 0x7f37b00a6a90 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f37a4008790 tx=0x7f37a4011920).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:45.129 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.108+0000 7f37b58d6700 1 --2- 172.21.15.42:0/542798766 >> v2:172.21.15.42:3300/0 conn(0x7f37b011c010 0x7f37b00a6a90 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f37a4008790 tx=0x7f37a4011920).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:45.129 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.108+0000 7f37b58d6700 1 --2- 172.21.15.42:0/542798766 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f37b011c010 0x7f37b00a6a90 secure :-1 s=READY pgs=2568 cs=0 l=1 rx=0x7f37a4008790 tx=0x7f37a4011920).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:23:45.129 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.108+0000 7f37b58d6700 1 --2- 172.21.15.42:0/542798766 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f37b011c010 0x7f37b00a6a90 secure :-1 s=THROTTLE_DONE pgs=2568 cs=0 l=1 rx=0x7f37a4008790 tx=0x7f37a4011920).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:45.130 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.108+0000 7f37aeffd700 1 -- 172.21.15.42:0/542798766 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f37a400f680 con 0x7f37b011c010 2019-12-05T06:23:45.130 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.108+0000 7f37b58d6700 1 --2- 172.21.15.42:0/542798766 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f37b011c010 0x7f37b00a6a90 secure :-1 s=THROTTLE_DONE pgs=2568 cs=0 l=1 rx=0x7f37a4008790 tx=0x7f37a4011920).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:45.130 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.108+0000 7f37aeffd700 1 -- 172.21.15.42:0/542798766 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37a4011730 con 0x7f37b011c010 2019-12-05T06:23:45.130 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.108+0000 7f37b58d6700 1 --2- 172.21.15.42:0/542798766 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f37b011c010 0x7f37b00a6a90 secure :-1 s=THROTTLE_DONE pgs=2568 cs=0 l=1 rx=0x7f37a4008790 tx=0x7f37a4011920).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:45.131 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.108+0000 7f37b7b3a700 1 -- 172.21.15.42:0/542798766 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37b0133d30 con 0x7f37b011c010 2019-12-05T06:23:45.131 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.108+0000 7f37aeffd700 1 -- 172.21.15.42:0/542798766 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f37a4005920 con 0x7f37b011c010 2019-12-05T06:23:45.145 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.108+0000 7f37b7b3a700 1 -- 172.21.15.42:0/542798766 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37b008f7a0 con 0x7f37b011c010 2019-12-05T06:23:45.146 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.108+0000 7f37b58d6700 1 --2- 172.21.15.42:0/542798766 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f37b011c010 0x7f37b00a6a90 secure :-1 s=THROTTLE_DONE pgs=2568 cs=0 l=1 rx=0x7f37a4008790 tx=0x7f37a4011920).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:45.146 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.112+0000 7f37aeffd700 1 -- 172.21.15.42:0/542798766 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f37a4005bc0 con 0x7f37b011c010 2019-12-05T06:23:45.146 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.112+0000 7f37b58d6700 1 --2- 172.21.15.42:0/542798766 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f37b011c010 0x7f37b00a6a90 secure :-1 s=THROTTLE_DONE pgs=2568 cs=0 l=1 rx=0x7f37a4008790 tx=0x7f37a4011920).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:45.146 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.112+0000 7f37aeffd700 1 --2- 172.21.15.42:0/542798766 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f379003b2b0 0x7f379003d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:23:45.146 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.112+0000 7f37aeffd700 1 -- 172.21.15.42:0/542798766 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 10761+0+0 (secure 0 0 0) 0x7f37a404b810 con 0x7f37b011c010 2019-12-05T06:23:45.146 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.112+0000 7f37b60d7700 1 --2- 172.21.15.42:0/542798766 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f379003b2b0 0x7f379003d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:45.147 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.112+0000 7f37b60d7700 1 --2- 172.21.15.42:0/542798766 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f379003b2b0 0x7f379003d7c0 crc :-1 s=READY pgs=1068 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:23:45.147 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.112+0000 7f37b7b3a700 1 -- 172.21.15.42:0/542798766 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37b005d110 con 0x7f37b011c010 2019-12-05T06:23:45.148 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.120+0000 7f37b58d6700 1 --2- 172.21.15.42:0/542798766 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f37b011c010 0x7f37b00a6a90 secure :-1 s=THROTTLE_DONE pgs=2568 cs=0 l=1 rx=0x7f37a4008790 tx=0x7f37a4011920).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:45.148 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.120+0000 7f37aeffd700 1 -- 172.21.15.42:0/542798766 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f37a4032ad0 con 0x7f37b011c010 2019-12-05T06:23:45.301 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.296+0000 7f37b7b3a700 1 -- 172.21.15.42:0/542798766 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f37b005b8c0 con 0x7f37b011c010 2019-12-05T06:23:45.304 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.296+0000 7f37b58d6700 1 --2- 172.21.15.42:0/542798766 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f37b011c010 0x7f37b00a6a90 secure :-1 s=THROTTLE_DONE pgs=2568 cs=0 l=1 rx=0x7f37a4008790 tx=0x7f37a4011920).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:45.305 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.296+0000 7f37aeffd700 1 -- 172.21.15.42:0/542798766 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 304 v304) v1 ==== 95+0+1778 (secure 0 0 0) 0x7f37a4033680 con 0x7f37b011c010 2019-12-05T06:23:45.305 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 304 2019-12-05T06:23:45.306 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:23:45.306 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":304,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":304,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:23:44.836693+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":104,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10993},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10993":{"gid":10993,"name":"a","rank":0,"incarnation":304,"state":"up:replay","state_seq":1,"addr":"172.21.15.42:6835/3964426552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3964426552},{"type":"v1","addr":"172.21.15.42:6835","nonce":3964426552}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:23:45.307 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.300+0000 7f37b7b3a700 1 -- 172.21.15.42:0/542798766 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f379003b2b0 msgr2=0x7f379003d7c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:45.307 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.300+0000 7f37b7b3a700 1 --2- 172.21.15.42:0/542798766 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f379003b2b0 0x7f379003d7c0 crc :-1 s=READY pgs=1068 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:45.307 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.300+0000 7f37b7b3a700 1 -- 172.21.15.42:0/542798766 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f37b011c010 msgr2=0x7f37b00a6a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:45.308 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.300+0000 7f37b7b3a700 1 --2- 172.21.15.42:0/542798766 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f37b011c010 0x7f37b00a6a90 secure :-1 s=READY pgs=2568 cs=0 l=1 rx=0x7f37a4008790 tx=0x7f37a4011920).stop 2019-12-05T06:23:45.309 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.300+0000 7f37b7b3a700 1 -- 172.21.15.42:0/542798766 shutdown_connections 2019-12-05T06:23:45.309 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.300+0000 7f37b7b3a700 1 --2- 172.21.15.42:0/542798766 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f379003b2b0 0x7f379003d7c0 unknown :-1 s=CLOSED pgs=1068 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:45.309 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.300+0000 7f37b7b3a700 1 --2- 172.21.15.42:0/542798766 >> v2:172.21.15.40:3300/0 conn(0x7f37b0116530 0x7f37b01337e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:45.309 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.300+0000 7f37b7b3a700 1 --2- 172.21.15.42:0/542798766 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f37b011c010 0x7f37b00a6a90 unknown :-1 s=CLOSED pgs=2568 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:45.309 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.300+0000 7f37b7b3a700 1 -- 172.21.15.42:0/542798766 shutdown_connections 2019-12-05T06:23:45.309 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.300+0000 7f37b7b3a700 1 -- 172.21.15.42:0/542798766 wait complete. 2019-12-05T06:23:45.309 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.300+0000 7f37b7b3a700 1 -- 172.21.15.42:0/542798766 >> 172.21.15.42:0/542798766 conn(0x7f37b008fde0 msgr2=0x7f37b0059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:23:45.316 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b down, removed from mdsmap 2019-12-05T06:23:45.317 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2019-12-05T06:23:45.317 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:23:45.421 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.388+0000 7f2bcca5b700 1 Processor -- start 2019-12-05T06:23:45.422 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.388+0000 7f2bcca5b700 1 -- start start 2019-12-05T06:23:45.422 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.388+0000 7f2bcca5b700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2bc80a7740 0x7f2bc808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:23:45.422 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.388+0000 7f2bcca5b700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2bc807e720 0x7f2bc807eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:23:45.422 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.388+0000 7f2bcca5b700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2bc809f2e0 0x7f2bc809f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:23:45.422 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.388+0000 7f2bcca5b700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2bc80a0b00 con 0x7f2bc807e720 2019-12-05T06:23:45.422 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.388+0000 7f2bcca5b700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2bc808e5b0 con 0x7f2bc809f2e0 2019-12-05T06:23:45.423 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.388+0000 7f2bcca5b700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bc808e850 con 0x7f2bc80a7740 2019-12-05T06:23:45.423 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.388+0000 7f2bc759e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2bc809f2e0 0x7f2bc809f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:45.423 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.388+0000 7f2bc759e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2bc809f2e0 0x7f2bc809f710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:54488/0 (socket says 172.21.15.42:54488) 2019-12-05T06:23:45.423 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.388+0000 7f2bc759e700 1 -- 172.21.15.42:0/792581778 learned_addr learned my addr 172.21.15.42:0/792581778 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:23:45.423 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.388+0000 7f2bc659c700 1 --2- 172.21.15.42:0/792581778 >> v2:172.21.15.40:3300/0 conn(0x7f2bc807e720 0x7f2bc807eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:45.423 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.388+0000 7f2bc6d9d700 1 --2- 172.21.15.42:0/792581778 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2bc80a7740 0x7f2bc808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:45.423 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.388+0000 7f2bc759e700 1 -- 172.21.15.42:0/792581778 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2bc80a7740 msgr2=0x7f2bc808df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:23:45.424 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.388+0000 7f2bc759e700 1 --2- 172.21.15.42:0/792581778 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2bc80a7740 0x7f2bc808df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:23:45.424 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.388+0000 7f2bc759e700 1 -- 172.21.15.42:0/792581778 >> v2:172.21.15.40:3300/0 conn(0x7f2bc807e720 msgr2=0x7f2bc807eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:23:45.424 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.388+0000 7f2bc759e700 1 --2- 172.21.15.42:0/792581778 >> v2:172.21.15.40:3300/0 conn(0x7f2bc807e720 0x7f2bc807eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:23:45.424 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.388+0000 7f2bc759e700 1 -- 172.21.15.42:0/792581778 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bc807f0a0 con 0x7f2bc809f2e0 2019-12-05T06:23:45.424 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.388+0000 7f2bc759e700 1 --2- 172.21.15.42:0/792581778 >> v2:172.21.15.42:3300/0 conn(0x7f2bc809f2e0 0x7f2bc809f710 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f2bc808e770 tx=0x7f2bc0010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:45.424 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.388+0000 7f2bc759e700 1 --2- 172.21.15.42:0/792581778 >> v2:172.21.15.42:3300/0 conn(0x7f2bc809f2e0 0x7f2bc809f710 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f2bc808e770 tx=0x7f2bc0010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:45.425 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.388+0000 7f2bc759e700 1 --2- 172.21.15.42:0/792581778 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2bc809f2e0 0x7f2bc809f710 secure :-1 s=READY pgs=2571 cs=0 l=1 rx=0x7f2bc808e770 tx=0x7f2bc0010130).ready entity=mon.1 client_cookie=45580a5718becfe0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:23:45.425 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.392+0000 7f2bc759e700 1 --2- 172.21.15.42:0/792581778 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2bc809f2e0 0x7f2bc809f710 secure :-1 s=THROTTLE_DONE pgs=2571 cs=0 l=1 rx=0x7f2bc808e770 tx=0x7f2bc0010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:45.425 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.392+0000 7f2bc759e700 1 --2- 172.21.15.42:0/792581778 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2bc809f2e0 0x7f2bc809f710 secure :-1 s=THROTTLE_DONE pgs=2571 cs=0 l=1 rx=0x7f2bc808e770 tx=0x7f2bc0010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:45.425 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.392+0000 7f2bc5d9b700 1 -- 172.21.15.42:0/792581778 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2bc0010de0 con 0x7f2bc809f2e0 2019-12-05T06:23:45.425 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.392+0000 7f2bc759e700 1 --2- 172.21.15.42:0/792581778 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2bc809f2e0 0x7f2bc809f710 secure :-1 s=THROTTLE_DONE pgs=2571 cs=0 l=1 rx=0x7f2bc808e770 tx=0x7f2bc0010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:45.425 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.392+0000 7f2bc5d9b700 1 -- 172.21.15.42:0/792581778 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bc0010a00 con 0x7f2bc809f2e0 2019-12-05T06:23:45.427 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.392+0000 7f2bc5d9b700 1 -- 172.21.15.42:0/792581778 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2bc00114a0 con 0x7f2bc809f2e0 2019-12-05T06:23:45.427 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.392+0000 7f2bcca5b700 1 -- 172.21.15.42:0/792581778 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2bc809f2e0 msgr2=0x7f2bc809f710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:45.428 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.392+0000 7f2bcca5b700 1 --2- 172.21.15.42:0/792581778 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2bc809f2e0 0x7f2bc809f710 secure :-1 s=READY pgs=2571 cs=0 l=1 rx=0x7f2bc808e770 tx=0x7f2bc0010130).stop 2019-12-05T06:23:45.428 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.392+0000 7f2bcca5b700 1 -- 172.21.15.42:0/792581778 shutdown_connections 2019-12-05T06:23:45.428 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.392+0000 7f2bcca5b700 1 --2- 172.21.15.42:0/792581778 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2bc809f2e0 0x7f2bc809f710 unknown :-1 s=CLOSED pgs=2571 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:45.429 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.392+0000 7f2bcca5b700 1 --2- 172.21.15.42:0/792581778 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2bc80a7740 0x7f2bc808df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:23:45.429 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.392+0000 7f2bcca5b700 1 --2- 172.21.15.42:0/792581778 >> v2:172.21.15.40:3300/0 conn(0x7f2bc807e720 0x7f2bc807eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:23:45.429 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.392+0000 7f2bcca5b700 1 -- 172.21.15.42:0/792581778 shutdown_connections 2019-12-05T06:23:45.429 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.392+0000 7f2bcca5b700 1 -- 172.21.15.42:0/792581778 wait complete. 2019-12-05T06:23:45.429 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.392+0000 7f2bcca5b700 1 -- 172.21.15.42:0/792581778 >> 172.21.15.42:0/792581778 conn(0x7f2bc80aabe0 msgr2=0x7f2bc80a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:23:45.430 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.392+0000 7f2bcca5b700 1 Processor -- start 2019-12-05T06:23:45.430 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.392+0000 7f2bcca5b700 1 -- start start 2019-12-05T06:23:45.430 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.396+0000 7f2bcca5b700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2bc8121270 0x7f2bc8059c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:23:45.430 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.396+0000 7f2bcca5b700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bc8120bc0 con 0x7f2bc811fe10 2019-12-05T06:23:45.431 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.396+0000 7f2bcca5b700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bc805a160 con 0x7f2bc80722c0 2019-12-05T06:23:45.431 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.396+0000 7f2bcca5b700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bc81450e0 con 0x7f2bc8121270 2019-12-05T06:23:45.431 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.396+0000 7f2bc759e700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f2bc80722c0 0x7f2bc8074700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:54530/0 (socket says 172.21.15.42:54530) 2019-12-05T06:23:45.431 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.396+0000 7f2bc759e700 1 -- 172.21.15.42:0/998121051 learned_addr learned my addr 172.21.15.42:0/998121051 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:23:45.432 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.396+0000 7f2bc6d9d700 1 --2- 172.21.15.42:0/998121051 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2bc8121270 0x7f2bc8059c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:45.432 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.396+0000 7f2bb77fe700 1 -- 172.21.15.42:0/998121051 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4237662558 0 0) 0x7f2bc805a160 con 0x7f2bc80722c0 2019-12-05T06:23:45.433 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.396+0000 7f2bb77fe700 1 -- 172.21.15.42:0/998121051 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2bac0030b0 con 0x7f2bc80722c0 2019-12-05T06:23:45.433 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.396+0000 7f2bb77fe700 1 -- 172.21.15.42:0/998121051 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2391284518 0 0) 0x7f2bc8120bc0 con 0x7f2bc811fe10 2019-12-05T06:23:45.434 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.396+0000 7f2bb77fe700 1 -- 172.21.15.42:0/998121051 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2bc805a160 con 0x7f2bc811fe10 2019-12-05T06:23:45.434 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.396+0000 7f2bc6d9d700 1 -- 172.21.15.42:0/998121051 >> v1:172.21.15.42:6789/0 conn(0x7f2bc80722c0 legacy=0x7f2bc8074700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:45.435 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.396+0000 7f2bc6d9d700 1 -- 172.21.15.42:0/998121051 >> v1:172.21.15.40:6789/0 conn(0x7f2bc811fe10 legacy=0x7f2bc8120240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:45.435 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.396+0000 7f2bc6d9d700 1 -- 172.21.15.42:0/998121051 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bc811eb80 con 0x7f2bc8121270 2019-12-05T06:23:45.436 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.396+0000 7f2bc6d9d700 1 --2- 172.21.15.42:0/998121051 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2bc8121270 0x7f2bc8059c10 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f2bc8145650 tx=0x7f2bb8010960).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:45.436 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.396+0000 7f2bc6d9d700 1 --2- 172.21.15.42:0/998121051 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2bc8121270 0x7f2bc8059c10 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f2bc8145650 tx=0x7f2bb8010960).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:45.436 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.396+0000 7f2bc6d9d700 1 --2- 172.21.15.42:0/998121051 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2bc8121270 0x7f2bc8059c10 secure :-1 s=READY pgs=555 cs=0 l=1 rx=0x7f2bc8145650 tx=0x7f2bb8010960).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:23:45.437 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.396+0000 7f2bc6d9d700 1 --2- 172.21.15.42:0/998121051 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2bc8121270 0x7f2bc8059c10 secure :-1 s=THROTTLE_DONE pgs=555 cs=0 l=1 rx=0x7f2bc8145650 tx=0x7f2bb8010960).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:45.437 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.396+0000 7f2bb77fe700 1 -- 172.21.15.42:0/998121051 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2bb8005930 con 0x7f2bc8121270 2019-12-05T06:23:45.438 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.396+0000 7f2bc6d9d700 1 --2- 172.21.15.42:0/998121051 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2bc8121270 0x7f2bc8059c10 secure :-1 s=THROTTLE_DONE pgs=555 cs=0 l=1 rx=0x7f2bc8145650 tx=0x7f2bb8010960).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:45.438 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.396+0000 7f2bb77fe700 1 -- 172.21.15.42:0/998121051 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bb8007200 con 0x7f2bc8121270 2019-12-05T06:23:45.439 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.396+0000 7f2bc6d9d700 1 --2- 172.21.15.42:0/998121051 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2bc8121270 0x7f2bc8059c10 secure :-1 s=THROTTLE_DONE pgs=555 cs=0 l=1 rx=0x7f2bc8145650 tx=0x7f2bb8010960).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:45.439 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.396+0000 7f2bcca5b700 1 -- 172.21.15.42:0/998121051 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bc8144cc0 con 0x7f2bc8121270 2019-12-05T06:23:45.439 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.396+0000 7f2bb77fe700 1 -- 172.21.15.42:0/998121051 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2bb80060c0 con 0x7f2bc8121270 2019-12-05T06:23:45.440 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.396+0000 7f2bcca5b700 1 -- 172.21.15.42:0/998121051 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bc8075f10 con 0x7f2bc8121270 2019-12-05T06:23:45.440 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.396+0000 7f2bc6d9d700 1 --2- 172.21.15.42:0/998121051 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2bc8121270 0x7f2bc8059c10 secure :-1 s=THROTTLE_DONE pgs=555 cs=0 l=1 rx=0x7f2bc8145650 tx=0x7f2bb8010960).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:45.440 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.396+0000 7f2bb77fe700 1 -- 172.21.15.42:0/998121051 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f2bb80063a0 con 0x7f2bc8121270 2019-12-05T06:23:45.440 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.396+0000 7f2bc6d9d700 1 --2- 172.21.15.42:0/998121051 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2bc8121270 0x7f2bc8059c10 secure :-1 s=THROTTLE_DONE pgs=555 cs=0 l=1 rx=0x7f2bc8145650 tx=0x7f2bb8010960).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:45.441 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.400+0000 7f2bb77fe700 1 --2- 172.21.15.42:0/998121051 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2bac03b470 0x7f2bac03d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:23:45.442 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.400+0000 7f2bb77fe700 1 -- 172.21.15.42:0/998121051 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 10761+0+0 (secure 0 0 0) 0x7f2bb804a9b0 con 0x7f2bc8121270 2019-12-05T06:23:45.442 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.400+0000 7f2bc659c700 1 --2- 172.21.15.42:0/998121051 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2bac03b470 0x7f2bac03d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:45.442 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.400+0000 7f2bc659c700 1 --2- 172.21.15.42:0/998121051 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2bac03b470 0x7f2bac03d980 crc :-1 s=READY pgs=1069 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:23:45.442 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.404+0000 7f2bcca5b700 1 -- 172.21.15.42:0/998121051 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bc805d110 con 0x7f2bc8121270 2019-12-05T06:23:45.443 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.412+0000 7f2bc6d9d700 1 --2- 172.21.15.42:0/998121051 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2bc8121270 0x7f2bc8059c10 secure :-1 s=THROTTLE_DONE pgs=555 cs=0 l=1 rx=0x7f2bc8145650 tx=0x7f2bb8010960).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:45.443 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.412+0000 7f2bb77fe700 1 -- 172.21.15.42:0/998121051 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f2bb8031a30 con 0x7f2bc8121270 2019-12-05T06:23:45.598 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.584+0000 7f2bcca5b700 1 -- 172.21.15.42:0/998121051 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2bc805b8c0 con 0x7f2bc8121270 2019-12-05T06:23:45.598 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.584+0000 7f2bc6d9d700 1 --2- 172.21.15.42:0/998121051 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2bc8121270 0x7f2bc8059c10 secure :-1 s=THROTTLE_DONE pgs=555 cs=0 l=1 rx=0x7f2bc8145650 tx=0x7f2bb8010960).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:45.598 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.584+0000 7f2bb77fe700 1 -- 172.21.15.42:0/998121051 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 304 v304) v1 ==== 95+0+1778 (secure 0 0 0) 0x7f2bb8031c90 con 0x7f2bc8121270 2019-12-05T06:23:45.599 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 304 2019-12-05T06:23:45.599 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.592+0000 7f2bcca5b700 1 -- 172.21.15.42:0/998121051 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2bac03b470 msgr2=0x7f2bac03d980 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:45.599 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.592+0000 7f2bcca5b700 1 --2- 172.21.15.42:0/998121051 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2bac03b470 0x7f2bac03d980 crc :-1 s=READY pgs=1069 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:45.599 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.592+0000 7f2bcca5b700 1 -- 172.21.15.42:0/998121051 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2bc8121270 msgr2=0x7f2bc8059c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:45.599 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.592+0000 7f2bcca5b700 1 --2- 172.21.15.42:0/998121051 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2bc8121270 0x7f2bc8059c10 secure :-1 s=READY pgs=555 cs=0 l=1 rx=0x7f2bc8145650 tx=0x7f2bb8010960).stop 2019-12-05T06:23:45.600 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.592+0000 7f2bcca5b700 1 -- 172.21.15.42:0/998121051 shutdown_connections 2019-12-05T06:23:45.600 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.592+0000 7f2bcca5b700 1 --2- 172.21.15.42:0/998121051 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2bac03b470 0x7f2bac03d980 unknown :-1 s=CLOSED pgs=1069 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:45.600 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.592+0000 7f2bcca5b700 1 --2- 172.21.15.42:0/998121051 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2bc8121270 0x7f2bc8059c10 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:45.600 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.592+0000 7f2bcca5b700 1 -- 172.21.15.42:0/998121051 shutdown_connections 2019-12-05T06:23:45.600 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.592+0000 7f2bcca5b700 1 -- 172.21.15.42:0/998121051 wait complete. 2019-12-05T06:23:45.601 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:45.592+0000 7f2bcca5b700 1 -- 172.21.15.42:0/998121051 >> 172.21.15.42:0/998121051 conn(0x7f2bc80a7f90 msgr2=0x7f2bc811c540 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:23:45.601 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:23:45.602 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":304,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":304,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:23:44.836693+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":104,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10993},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10993":{"gid":10993,"name":"a","rank":0,"incarnation":304,"state":"up:replay","state_seq":1,"addr":"172.21.15.42:6835/3964426552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3964426552},{"type":"v1","addr":"172.21.15.42:6835","nonce":3964426552}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:23:45.607 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 304, "feature_flags": { "enable_multiple": false, "ever_enabled_multiple": false }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2019-12-05T05:03:35.604738+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 304, "ever_allowed_features": 2, "explicitly_allowed_features": 2, "failed": [], "flags": 18, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_10993": { "addr": "172.21.15.42:6835/3964426552", "addrs": { "addrvec": [ { "addr": "172.21.15.42:6834", "nonce": 3964426552, "type": "v2" }, { "addr": "172.21.15.42:6835", "nonce": 3964426552, "type": "v1" } ] }, "export_targets": [], "features": 4540138292836696063, "flags": 0, "gid": 10993, "incarnation": 304, "name": "a", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 104, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 2, "min_compat_client": "0 (unknown)", "modified": "2019-12-05T06:23:44.836693+0000", "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 10993 } } } ], "standbys": [] } 2019-12-05T06:23:47.611 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:23:47.727 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:47.704+0000 7ff2094b2700 1 Processor -- start 2019-12-05T06:23:47.728 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:47.704+0000 7ff2094b2700 1 -- start start 2019-12-05T06:23:47.728 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:47.708+0000 7ff2094b2700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff2040a7740 0x7ff20408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:23:47.728 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:47.708+0000 7ff2094b2700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2040a0710 con 0x7ff20407e720 2019-12-05T06:23:47.728 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:47.708+0000 7ff2094b2700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2040a9ba0 con 0x7ff20409fae0 2019-12-05T06:23:47.728 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:47.708+0000 7ff2094b2700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2040a0b00 con 0x7ff2040a7740 2019-12-05T06:23:47.728 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:47.708+0000 7ff203fff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7ff20409fae0 0x7ff20414efe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:54540/0 (socket says 172.21.15.42:54540) 2019-12-05T06:23:47.729 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:47.708+0000 7ff203fff700 1 -- 172.21.15.42:0/350550326 learned_addr learned my addr 172.21.15.42:0/350550326 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:23:47.729 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:47.708+0000 7ff2037fe700 1 --2- 172.21.15.42:0/350550326 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff2040a7740 0x7ff20408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:47.729 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:47.708+0000 7ff202ffd700 1 -- 172.21.15.42:0/350550326 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1125204168 0 0) 0x7ff2040a9ba0 con 0x7ff20409fae0 2019-12-05T06:23:47.729 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:47.708+0000 7ff202ffd700 1 -- 172.21.15.42:0/350550326 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1f00030b0 con 0x7ff20409fae0 2019-12-05T06:23:47.729 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:47.708+0000 7ff202ffd700 1 -- 172.21.15.42:0/350550326 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2258362617 0 0) 0x7ff2040a0710 con 0x7ff20407e720 2019-12-05T06:23:47.729 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:47.708+0000 7ff202ffd700 1 -- 172.21.15.42:0/350550326 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2040a9ba0 con 0x7ff20407e720 2019-12-05T06:23:47.736 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:47.708+0000 7ff202ffd700 1 -- 172.21.15.42:0/350550326 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1703315169 0 0) 0x7ff1f00030b0 con 0x7ff20409fae0 2019-12-05T06:23:47.744 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:47.708+0000 7ff202ffd700 1 -- 172.21.15.42:0/350550326 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff2040a7740 msgr2=0x7ff20408df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:23:47.745 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:47.708+0000 7ff202ffd700 1 --2- 172.21.15.42:0/350550326 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff2040a7740 0x7ff20408df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:23:47.745 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:47.708+0000 7ff202ffd700 1 -- 172.21.15.42:0/350550326 >> v1:172.21.15.40:6789/0 conn(0x7ff20407e720 legacy=0x7ff20409f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:47.745 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:47.708+0000 7ff202ffd700 1 -- 172.21.15.42:0/350550326 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff20407eb50 con 0x7ff20409fae0 2019-12-05T06:23:47.745 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:47.708+0000 7ff202ffd700 1 -- 172.21.15.42:0/350550326 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7ff1fc001ad0 con 0x7ff20409fae0 2019-12-05T06:23:47.746 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:47.708+0000 7ff202ffd700 1 -- 172.21.15.42:0/350550326 >> v1:172.21.15.42:6789/0 conn(0x7ff20409fae0 legacy=0x7ff20414efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:47.746 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:47.708+0000 7ff202ffd700 1 --2- 172.21.15.42:0/350550326 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff1f0003940 0x7ff1f0003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:23:47.746 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:47.708+0000 7ff202ffd700 1 --2- 172.21.15.42:0/350550326 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff1f0004420 0x7ff1f00048d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:23:47.746 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:47.708+0000 7ff202ffd700 1 --2- 172.21.15.42:0/350550326 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff1f0004fc0 0x7ff1f004f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:23:47.747 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:47.708+0000 7ff202ffd700 1 -- 172.21.15.42:0/350550326 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2040a0b00 con 0x7ff1f0004420 2019-12-05T06:23:47.747 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:47.708+0000 7ff2037fe700 1 --2- 172.21.15.42:0/350550326 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff1f0003940 0x7ff1f0003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:47.748 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:47.708+0000 7ff202ffd700 1 -- 172.21.15.42:0/350550326 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1f004fc00 con 0x7ff1f0003940 2019-12-05T06:23:47.756 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:47.708+0000 7ff202ffd700 1 -- 172.21.15.42:0/350550326 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1f004fe40 con 0x7ff1f0004fc0 2019-12-05T06:23:47.756 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:47.708+0000 7ff1fbfff700 1 --2- 172.21.15.42:0/350550326 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff1f0004420 0x7ff1f00048d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:47.757 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:47.708+0000 7ff2037fe700 1 -- 172.21.15.42:0/350550326 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff1f0004fc0 msgr2=0x7ff1f004f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:23:47.757 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:47.708+0000 7ff2037fe700 1 --2- 172.21.15.42:0/350550326 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff1f0004fc0 0x7ff1f004f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:23:47.757 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:47.708+0000 7ff2037fe700 1 -- 172.21.15.42:0/350550326 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff1f0004420 msgr2=0x7ff1f00048d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:23:47.757 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:47.708+0000 7ff2037fe700 1 --2- 172.21.15.42:0/350550326 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff1f0004420 0x7ff1f00048d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:23:47.757 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:47.708+0000 7ff2037fe700 1 -- 172.21.15.42:0/350550326 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1f0050140 con 0x7ff1f0003940 2019-12-05T06:23:47.757 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:47.708+0000 7ff2037fe700 1 --2- 172.21.15.42:0/350550326 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff1f0003940 0x7ff1f0003d90 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7ff1f4002f30 tx=0x7ff1f400ffe0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:47.758 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:47.708+0000 7ff2037fe700 1 --2- 172.21.15.42:0/350550326 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff1f0003940 0x7ff1f0003d90 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7ff1f4002f30 tx=0x7ff1f400ffe0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:47.758 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:47.708+0000 7ff2037fe700 1 --2- 172.21.15.42:0/350550326 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff1f0003940 0x7ff1f0003d90 secure :-1 s=READY pgs=2574 cs=0 l=1 rx=0x7ff1f4002f30 tx=0x7ff1f400ffe0).ready entity=mon.1 client_cookie=9ac2d9e571d5d721 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:23:47.758 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:47.708+0000 7ff2037fe700 1 --2- 172.21.15.42:0/350550326 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff1f0003940 0x7ff1f0003d90 secure :-1 s=THROTTLE_DONE pgs=2574 cs=0 l=1 rx=0x7ff1f4002f30 tx=0x7ff1f400ffe0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:47.758 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:47.708+0000 7ff202ffd700 1 -- 172.21.15.42:0/350550326 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff1f4010d10 con 0x7ff1f0003940 2019-12-05T06:23:47.758 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:47.708+0000 7ff2037fe700 1 --2- 172.21.15.42:0/350550326 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff1f0003940 0x7ff1f0003d90 secure :-1 s=THROTTLE_DONE pgs=2574 cs=0 l=1 rx=0x7ff1f4002f30 tx=0x7ff1f400ffe0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:47.758 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:47.708+0000 7ff2037fe700 1 --2- 172.21.15.42:0/350550326 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff1f0003940 0x7ff1f0003d90 secure :-1 s=THROTTLE_DONE pgs=2574 cs=0 l=1 rx=0x7ff1f4002f30 tx=0x7ff1f400ffe0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:47.759 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:47.708+0000 7ff202ffd700 1 -- 172.21.15.42:0/350550326 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1f4010930 con 0x7ff1f0003940 2019-12-05T06:23:47.759 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:47.708+0000 7ff202ffd700 1 -- 172.21.15.42:0/350550326 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff1f4008bb0 con 0x7ff1f0003940 2019-12-05T06:23:47.759 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:47.708+0000 7ff2094b2700 1 -- 172.21.15.42:0/350550326 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff1f0003940 msgr2=0x7ff1f0003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:47.759 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:47.708+0000 7ff2094b2700 1 --2- 172.21.15.42:0/350550326 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff1f0003940 0x7ff1f0003d90 secure :-1 s=READY pgs=2574 cs=0 l=1 rx=0x7ff1f4002f30 tx=0x7ff1f400ffe0).stop 2019-12-05T06:23:47.759 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:47.708+0000 7ff203fff700 1 -- 172.21.15.42:0/350550326 reap_dead start 2019-12-05T06:23:47.759 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:47.708+0000 7ff2094b2700 1 -- 172.21.15.42:0/350550326 shutdown_connections 2019-12-05T06:23:47.760 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:47.708+0000 7ff2094b2700 1 -- 172.21.15.42:0/350550326 shutdown_connections 2019-12-05T06:23:47.760 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:47.708+0000 7ff2094b2700 1 -- 172.21.15.42:0/350550326 wait complete. 2019-12-05T06:23:47.760 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:47.708+0000 7ff2094b2700 1 -- 172.21.15.42:0/350550326 >> 172.21.15.42:0/350550326 conn(0x7ff2040aabe0 msgr2=0x7ff2040a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:23:47.760 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:47.712+0000 7ff2094b2700 1 Processor -- start 2019-12-05T06:23:47.760 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:47.712+0000 7ff2094b2700 1 -- start start 2019-12-05T06:23:47.760 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:47.712+0000 7ff2094b2700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff204126be0 0x7ff2040a5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:23:47.761 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:47.712+0000 7ff2094b2700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff204143030 con 0x7ff2040a4840 2019-12-05T06:23:47.761 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:47.712+0000 7ff2094b2700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff204078690 con 0x7ff2040a4c70 2019-12-05T06:23:47.761 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:47.712+0000 7ff2094b2700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff204142890 con 0x7ff204126be0 2019-12-05T06:23:47.761 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:47.712+0000 7ff203fff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7ff2040a4c70 0x7ff204063990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:54550/0 (socket says 172.21.15.42:54550) 2019-12-05T06:23:47.761 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:47.712+0000 7ff203fff700 1 -- 172.21.15.42:0/429356601 learned_addr learned my addr 172.21.15.42:0/429356601 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:23:47.761 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:47.712+0000 7ff2037fe700 1 --2- 172.21.15.42:0/429356601 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff204126be0 0x7ff2040a5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:47.761 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:47.712+0000 7ff200ff9700 1 -- 172.21.15.42:0/429356601 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2403131335 0 0) 0x7ff204078690 con 0x7ff2040a4c70 2019-12-05T06:23:47.762 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:47.712+0000 7ff200ff9700 1 -- 172.21.15.42:0/429356601 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1e80030b0 con 0x7ff2040a4c70 2019-12-05T06:23:47.762 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:47.712+0000 7ff200ff9700 1 -- 172.21.15.42:0/429356601 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 534919795 0 0) 0x7ff204143030 con 0x7ff2040a4840 2019-12-05T06:23:47.762 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:47.712+0000 7ff200ff9700 1 -- 172.21.15.42:0/429356601 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff204078690 con 0x7ff2040a4840 2019-12-05T06:23:47.762 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:47.712+0000 7ff200ff9700 1 -- 172.21.15.42:0/429356601 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 3330938800 0 0) 0x7ff1e80030b0 con 0x7ff2040a4c70 2019-12-05T06:23:47.762 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:47.712+0000 7ff200ff9700 1 -- 172.21.15.42:0/429356601 --> v1:172.21.15.42:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff204143030 con 0x7ff2040a4c70 2019-12-05T06:23:47.762 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:47.712+0000 7ff200ff9700 1 -- 172.21.15.42:0/429356601 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7ff1fc001ad0 con 0x7ff2040a4c70 2019-12-05T06:23:47.763 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:47.712+0000 7ff200ff9700 1 -- 172.21.15.42:0/429356601 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff204126be0 msgr2=0x7ff2040a5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:47.763 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:47.712+0000 7ff200ff9700 1 --2- 172.21.15.42:0/429356601 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff204126be0 0x7ff2040a5ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:47.763 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:47.712+0000 7ff200ff9700 1 -- 172.21.15.42:0/429356601 >> v1:172.21.15.42:6789/0 conn(0x7ff2040a4c70 legacy=0x7ff204063990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:47.763 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:47.712+0000 7ff200ff9700 1 -- 172.21.15.42:0/429356601 >> v1:172.21.15.40:6789/0 conn(0x7ff2040a4840 legacy=0x7ff204063190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:47.763 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:47.712+0000 7ff200ff9700 1 --2- 172.21.15.42:0/429356601 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff1e8003ab0 0x7ff1e8004420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:23:47.763 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:47.712+0000 7ff200ff9700 1 --2- 172.21.15.42:0/429356601 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff1e8003f90 0x7ff1e80049f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:23:47.764 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:47.712+0000 7ff200ff9700 1 --2- 172.21.15.42:0/429356601 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff1e8009660 0x7ff1e800baa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:23:47.764 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:47.712+0000 7ff200ff9700 1 -- 172.21.15.42:0/429356601 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff204142890 con 0x7ff1e8003f90 2019-12-05T06:23:47.764 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:47.712+0000 7ff200ff9700 1 -- 172.21.15.42:0/429356601 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1e800bff0 con 0x7ff1e8009660 2019-12-05T06:23:47.764 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:47.712+0000 7ff200ff9700 1 -- 172.21.15.42:0/429356601 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1e800c1b0 con 0x7ff1e8003ab0 2019-12-05T06:23:47.764 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:47.712+0000 7ff203fff700 1 --2- 172.21.15.42:0/429356601 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff1e8009660 0x7ff1e800baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:47.764 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:47.712+0000 7ff1fbfff700 1 --2- 172.21.15.42:0/429356601 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff1e8003f90 0x7ff1e80049f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:47.765 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:47.712+0000 7ff2037fe700 1 --2- 172.21.15.42:0/429356601 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff1e8003ab0 0x7ff1e8004420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:47.765 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:47.712+0000 7ff203fff700 1 -- 172.21.15.42:0/429356601 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff1e8003ab0 msgr2=0x7ff1e8004420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:47.765 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:47.712+0000 7ff203fff700 1 --2- 172.21.15.42:0/429356601 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff1e8003ab0 0x7ff1e8004420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:47.766 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:47.712+0000 7ff203fff700 1 -- 172.21.15.42:0/429356601 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff1e8003f90 msgr2=0x7ff1e80049f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:47.766 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:47.712+0000 7ff203fff700 1 --2- 172.21.15.42:0/429356601 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff1e8003f90 0x7ff1e80049f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:47.766 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:47.712+0000 7ff203fff700 1 -- 172.21.15.42:0/429356601 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff20405ae30 con 0x7ff1e8009660 2019-12-05T06:23:47.766 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:47.712+0000 7ff2094b2700 1 -- 172.21.15.42:0/429356601 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff204126be0 con 0x7ff1e8009660 2019-12-05T06:23:47.766 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:47.712+0000 7ff2037fe700 1 --2- 172.21.15.42:0/429356601 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff1e8003ab0 0x7ff1e8004420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).send_auth_request state changed! 2019-12-05T06:23:47.767 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:47.712+0000 7ff203fff700 1 --2- 172.21.15.42:0/429356601 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff1e8009660 0x7ff1e800baa0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7ff1fc000f00 tx=0x7ff1fc0113d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:47.767 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:47.712+0000 7ff203fff700 1 --2- 172.21.15.42:0/429356601 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff1e8009660 0x7ff1e800baa0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7ff1fc000f00 tx=0x7ff1fc0113d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:47.767 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:47.712+0000 7ff203fff700 1 --2- 172.21.15.42:0/429356601 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff1e8009660 0x7ff1e800baa0 secure :-1 s=READY pgs=2576 cs=0 l=1 rx=0x7ff1fc000f00 tx=0x7ff1fc0113d0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:23:47.767 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:47.712+0000 7ff2094b2700 1 -- 172.21.15.42:0/429356601 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2040a5ee0 con 0x7ff1e8009660 2019-12-05T06:23:47.768 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:47.712+0000 7ff203fff700 1 --2- 172.21.15.42:0/429356601 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff1e8009660 0x7ff1e800baa0 secure :-1 s=THROTTLE_DONE pgs=2576 cs=0 l=1 rx=0x7ff1fc000f00 tx=0x7ff1fc0113d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:47.768 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:47.712+0000 7ff200ff9700 1 -- 172.21.15.42:0/429356601 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff1fc00a9f0 con 0x7ff1e8009660 2019-12-05T06:23:47.768 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:47.712+0000 7ff203fff700 1 --2- 172.21.15.42:0/429356601 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff1e8009660 0x7ff1e800baa0 secure :-1 s=THROTTLE_DONE pgs=2576 cs=0 l=1 rx=0x7ff1fc000f00 tx=0x7ff1fc0113d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:47.768 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:47.712+0000 7ff203fff700 1 --2- 172.21.15.42:0/429356601 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff1e8009660 0x7ff1e800baa0 secure :-1 s=THROTTLE_DONE pgs=2576 cs=0 l=1 rx=0x7ff1fc000f00 tx=0x7ff1fc0113d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:47.768 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:47.712+0000 7ff200ff9700 1 -- 172.21.15.42:0/429356601 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1fc0015b0 con 0x7ff1e8009660 2019-12-05T06:23:47.769 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:47.712+0000 7ff200ff9700 1 -- 172.21.15.42:0/429356601 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff1fc00b070 con 0x7ff1e8009660 2019-12-05T06:23:47.769 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:47.712+0000 7ff203fff700 1 --2- 172.21.15.42:0/429356601 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff1e8009660 0x7ff1e800baa0 secure :-1 s=THROTTLE_DONE pgs=2576 cs=0 l=1 rx=0x7ff1fc000f00 tx=0x7ff1fc0113d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:47.769 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:47.716+0000 7ff200ff9700 1 -- 172.21.15.42:0/429356601 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7ff1fc00b270 con 0x7ff1e8009660 2019-12-05T06:23:47.772 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:47.716+0000 7ff203fff700 1 --2- 172.21.15.42:0/429356601 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff1e8009660 0x7ff1e800baa0 secure :-1 s=THROTTLE_DONE pgs=2576 cs=0 l=1 rx=0x7ff1fc000f00 tx=0x7ff1fc0113d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:47.772 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:47.716+0000 7ff200ff9700 1 --2- 172.21.15.42:0/429356601 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff1e8043fb0 0x7ff1e80464c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:23:47.772 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:47.716+0000 7ff200ff9700 1 -- 172.21.15.42:0/429356601 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 10761+0+0 (secure 0 0 0) 0x7ff1fc04b050 con 0x7ff1e8009660 2019-12-05T06:23:47.772 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:47.716+0000 7ff2037fe700 1 --2- 172.21.15.42:0/429356601 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff1e8043fb0 0x7ff1e80464c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:47.772 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:47.716+0000 7ff2037fe700 1 --2- 172.21.15.42:0/429356601 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff1e8043fb0 0x7ff1e80464c0 crc :-1 s=READY pgs=1070 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:23:47.773 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:47.720+0000 7ff2094b2700 1 -- 172.21.15.42:0/429356601 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1dc003200 con 0x7ff1e8009660 2019-12-05T06:23:47.773 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:47.724+0000 7ff203fff700 1 --2- 172.21.15.42:0/429356601 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff1e8009660 0x7ff1e800baa0 secure :-1 s=THROTTLE_DONE pgs=2576 cs=0 l=1 rx=0x7ff1fc000f00 tx=0x7ff1fc0113d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:47.773 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:47.724+0000 7ff200ff9700 1 -- 172.21.15.42:0/429356601 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7ff1fc0320d0 con 0x7ff1e8009660 2019-12-05T06:23:47.899 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:47.888+0000 7ff2094b2700 1 -- 172.21.15.42:0/429356601 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff1dc003630 con 0x7ff1e8009660 2019-12-05T06:23:47.899 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:47.888+0000 7ff203fff700 1 --2- 172.21.15.42:0/429356601 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff1e8009660 0x7ff1e800baa0 secure :-1 s=THROTTLE_DONE pgs=2576 cs=0 l=1 rx=0x7ff1fc000f00 tx=0x7ff1fc0113d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:47.899 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:47.888+0000 7ff200ff9700 1 -- 172.21.15.42:0/429356601 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 304 v304) v1 ==== 95+0+1778 (secure 0 0 0) 0x7ff1fc032330 con 0x7ff1e8009660 2019-12-05T06:23:47.900 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 304 2019-12-05T06:23:47.900 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:47.892+0000 7ff2094b2700 1 -- 172.21.15.42:0/429356601 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff1e8043fb0 msgr2=0x7ff1e80464c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:47.900 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:47.892+0000 7ff2094b2700 1 --2- 172.21.15.42:0/429356601 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff1e8043fb0 0x7ff1e80464c0 crc :-1 s=READY pgs=1070 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:47.900 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:47.892+0000 7ff2094b2700 1 -- 172.21.15.42:0/429356601 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff1e8009660 msgr2=0x7ff1e800baa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:47.900 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:47.892+0000 7ff2094b2700 1 --2- 172.21.15.42:0/429356601 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff1e8009660 0x7ff1e800baa0 secure :-1 s=READY pgs=2576 cs=0 l=1 rx=0x7ff1fc000f00 tx=0x7ff1fc0113d0).stop 2019-12-05T06:23:47.900 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:47.892+0000 7ff203fff700 1 -- 172.21.15.42:0/429356601 reap_dead start 2019-12-05T06:23:47.900 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:47.892+0000 7ff203fff700 1 -- 172.21.15.42:0/429356601 reap_dead start 2019-12-05T06:23:47.901 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:47.892+0000 7ff2094b2700 1 -- 172.21.15.42:0/429356601 shutdown_connections 2019-12-05T06:23:47.902 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:47.892+0000 7ff2094b2700 1 -- 172.21.15.42:0/429356601 shutdown_connections 2019-12-05T06:23:47.902 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:47.892+0000 7ff2094b2700 1 -- 172.21.15.42:0/429356601 wait complete. 2019-12-05T06:23:47.902 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:47.892+0000 7ff2094b2700 1 -- 172.21.15.42:0/429356601 >> 172.21.15.42:0/429356601 conn(0x7ff2040a7740 msgr2=0x7ff20409f650 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:23:47.902 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:23:47.902 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":304,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":304,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:23:44.836693+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":104,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10993},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10993":{"gid":10993,"name":"a","rank":0,"incarnation":304,"state":"up:replay","state_seq":1,"addr":"172.21.15.42:6835/3964426552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3964426552},{"type":"v1","addr":"172.21.15.42:6835","nonce":3964426552}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:23:47.908 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2019-12-05T06:23:49.909 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:23:50.020 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:50.008+0000 7f6785370700 1 Processor -- start 2019-12-05T06:23:50.020 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:50.008+0000 7f6785370700 1 -- start start 2019-12-05T06:23:50.020 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:50.008+0000 7f6785370700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f67800a7740 0x7f678008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:23:50.020 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:50.008+0000 7f6785370700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f678007e720 0x7f678007eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:23:50.021 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:50.008+0000 7f6785370700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67800a0710 con 0x7f678009f2e0 2019-12-05T06:23:50.021 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:50.008+0000 7f6785370700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67800a0b00 con 0x7f67800a7740 2019-12-05T06:23:50.021 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:50.008+0000 7f6785370700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f678008e5b0 con 0x7f678007e720 2019-12-05T06:23:50.021 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:50.012+0000 7f677f7fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f67800a7740 0x7f678008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:50.021 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:50.012+0000 7f677ffff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f678009f2e0 0x7f678009f710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:54572/0 (socket says 172.21.15.42:54572) 2019-12-05T06:23:50.021 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:50.012+0000 7f677f7fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f67800a7740 0x7f678008df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:54530/0 (socket says 172.21.15.42:54530) 2019-12-05T06:23:50.022 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:50.012+0000 7f677ffff700 1 -- 172.21.15.42:0/3414816298 learned_addr learned my addr 172.21.15.42:0/3414816298 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:23:50.022 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:50.012+0000 7f677effd700 1 --2- 172.21.15.42:0/3414816298 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f678007e720 0x7f678007eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:50.022 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:50.012+0000 7f677f7fe700 1 -- 172.21.15.42:0/3414816298 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f678007e720 msgr2=0x7f678007eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:23:50.022 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:50.012+0000 7f677f7fe700 1 --2- 172.21.15.42:0/3414816298 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f678007e720 0x7f678007eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:23:50.022 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:50.012+0000 7f677f7fe700 1 -- 172.21.15.42:0/3414816298 >> v1:172.21.15.42:6789/0 conn(0x7f678009f2e0 legacy=0x7f678009f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:50.022 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:50.012+0000 7f677f7fe700 1 -- 172.21.15.42:0/3414816298 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f678007f0a0 con 0x7f67800a7740 2019-12-05T06:23:50.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:50.012+0000 7f677e7fc700 1 -- 172.21.15.42:0/3414816298 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 650891774 0 0) 0x7f67800a0710 con 0x7f678009f2e0 2019-12-05T06:23:50.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:50.012+0000 7f677f7fe700 1 --2- 172.21.15.42:0/3414816298 >> v2:172.21.15.42:3300/0 conn(0x7f67800a7740 0x7f678008df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f6770008b10 tx=0x7f677000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:50.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:50.012+0000 7f677f7fe700 1 --2- 172.21.15.42:0/3414816298 >> v2:172.21.15.42:3300/0 conn(0x7f67800a7740 0x7f678008df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f6770008b10 tx=0x7f677000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:50.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:50.012+0000 7f677f7fe700 1 --2- 172.21.15.42:0/3414816298 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f67800a7740 0x7f678008df80 secure :-1 s=READY pgs=2578 cs=0 l=1 rx=0x7f6770008b10 tx=0x7f677000fff0).ready entity=mon.1 client_cookie=95dadc15cfe5459e server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:23:50.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:50.012+0000 7f677f7fe700 1 --2- 172.21.15.42:0/3414816298 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f67800a7740 0x7f678008df80 secure :-1 s=THROTTLE_DONE pgs=2578 cs=0 l=1 rx=0x7f6770008b10 tx=0x7f677000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:50.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:50.012+0000 7f677e7fc700 1 -- 172.21.15.42:0/3414816298 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6770011a50 con 0x7f67800a7740 2019-12-05T06:23:50.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:50.012+0000 7f677f7fe700 1 --2- 172.21.15.42:0/3414816298 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f67800a7740 0x7f678008df80 secure :-1 s=THROTTLE_DONE pgs=2578 cs=0 l=1 rx=0x7f6770008b10 tx=0x7f677000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:50.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:50.012+0000 7f677e7fc700 1 -- 172.21.15.42:0/3414816298 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6770011670 con 0x7f67800a7740 2019-12-05T06:23:50.024 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:50.012+0000 7f677f7fe700 1 --2- 172.21.15.42:0/3414816298 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f67800a7740 0x7f678008df80 secure :-1 s=THROTTLE_DONE pgs=2578 cs=0 l=1 rx=0x7f6770008b10 tx=0x7f677000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:50.024 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:50.012+0000 7f677e7fc700 1 -- 172.21.15.42:0/3414816298 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f677000a350 con 0x7f67800a7740 2019-12-05T06:23:50.024 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:50.012+0000 7f6785370700 1 -- 172.21.15.42:0/3414816298 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f67800a7740 msgr2=0x7f678008df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:50.024 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:50.012+0000 7f6785370700 1 --2- 172.21.15.42:0/3414816298 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f67800a7740 0x7f678008df80 secure :-1 s=READY pgs=2578 cs=0 l=1 rx=0x7f6770008b10 tx=0x7f677000fff0).stop 2019-12-05T06:23:50.024 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:50.012+0000 7f6785370700 1 -- 172.21.15.42:0/3414816298 shutdown_connections 2019-12-05T06:23:50.024 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:50.012+0000 7f6785370700 1 --2- 172.21.15.42:0/3414816298 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f67800a7740 0x7f678008df80 unknown :-1 s=CLOSED pgs=2578 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:50.024 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:50.012+0000 7f6785370700 1 --2- 172.21.15.42:0/3414816298 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f678007e720 0x7f678007eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:23:50.025 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:50.012+0000 7f6785370700 1 -- 172.21.15.42:0/3414816298 shutdown_connections 2019-12-05T06:23:50.025 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:50.012+0000 7f6785370700 1 -- 172.21.15.42:0/3414816298 wait complete. 2019-12-05T06:23:50.025 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:50.012+0000 7f6785370700 1 -- 172.21.15.42:0/3414816298 >> 172.21.15.42:0/3414816298 conn(0x7f67800aabe0 msgr2=0x7f67800a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:23:50.029 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:50.016+0000 7f6785370700 1 Processor -- start 2019-12-05T06:23:50.029 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:50.016+0000 7f6785370700 1 -- start start 2019-12-05T06:23:50.029 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:50.016+0000 7f6785370700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6780144c50 0x7f678011a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:23:50.029 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:50.016+0000 7f6785370700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f678011b830 con 0x7f678007d850 2019-12-05T06:23:50.029 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:50.016+0000 7f6785370700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f678011ab30 con 0x7f6780127fb0 2019-12-05T06:23:50.029 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:50.016+0000 7f6785370700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6780119d00 con 0x7f6780144c50 2019-12-05T06:23:50.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:50.016+0000 7f677effd700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f6780127fb0 0x7f6780118fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:54576/0 (socket says 172.21.15.42:54576) 2019-12-05T06:23:50.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:50.016+0000 7f677effd700 1 -- 172.21.15.42:0/1617185683 learned_addr learned my addr 172.21.15.42:0/1617185683 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:23:50.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:50.016+0000 7f677f7fe700 1 --2- 172.21.15.42:0/1617185683 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6780144c50 0x7f678011a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:50.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:50.016+0000 7f6777fff700 1 -- 172.21.15.42:0/1617185683 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 218046505 0 0) 0x7f678011ab30 con 0x7f6780127fb0 2019-12-05T06:23:50.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:50.016+0000 7f6777fff700 1 -- 172.21.15.42:0/1617185683 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67640030b0 con 0x7f6780127fb0 2019-12-05T06:23:50.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:50.016+0000 7f6777fff700 1 -- 172.21.15.42:0/1617185683 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2047222491 0 0) 0x7f678011b830 con 0x7f678007d850 2019-12-05T06:23:50.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:50.016+0000 7f6777fff700 1 -- 172.21.15.42:0/1617185683 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f678011ab30 con 0x7f678007d850 2019-12-05T06:23:50.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:50.016+0000 7f677f7fe700 1 -- 172.21.15.42:0/1617185683 >> v1:172.21.15.42:6789/0 conn(0x7f6780127fb0 legacy=0x7f6780118fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:50.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:50.016+0000 7f677f7fe700 1 -- 172.21.15.42:0/1617185683 >> v1:172.21.15.40:6789/0 conn(0x7f678007d850 legacy=0x7f678012f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:50.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:50.016+0000 7f677f7fe700 1 -- 172.21.15.42:0/1617185683 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f678007dc80 con 0x7f6780144c50 2019-12-05T06:23:50.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:50.016+0000 7f677f7fe700 1 --2- 172.21.15.42:0/1617185683 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6780144c50 0x7f678011a5e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f6770008b10 tx=0x7f6770011670).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:50.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:50.016+0000 7f677f7fe700 1 --2- 172.21.15.42:0/1617185683 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6780144c50 0x7f678011a5e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f6770008b10 tx=0x7f6770011670).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:50.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:50.016+0000 7f677f7fe700 1 --2- 172.21.15.42:0/1617185683 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6780144c50 0x7f678011a5e0 secure :-1 s=READY pgs=557 cs=0 l=1 rx=0x7f6770008b10 tx=0x7f6770011670).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:23:50.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:50.016+0000 7f677f7fe700 1 --2- 172.21.15.42:0/1617185683 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6780144c50 0x7f678011a5e0 secure :-1 s=THROTTLE_DONE pgs=557 cs=0 l=1 rx=0x7f6770008b10 tx=0x7f6770011670).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:50.032 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:50.016+0000 7f6777fff700 1 -- 172.21.15.42:0/1617185683 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6770005ab0 con 0x7f6780144c50 2019-12-05T06:23:50.032 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:50.016+0000 7f677f7fe700 1 --2- 172.21.15.42:0/1617185683 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6780144c50 0x7f678011a5e0 secure :-1 s=THROTTLE_DONE pgs=557 cs=0 l=1 rx=0x7f6770008b10 tx=0x7f6770011670).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:50.032 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:50.016+0000 7f677f7fe700 1 --2- 172.21.15.42:0/1617185683 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6780144c50 0x7f678011a5e0 secure :-1 s=THROTTLE_DONE pgs=557 cs=0 l=1 rx=0x7f6770008b10 tx=0x7f6770011670).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:50.032 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:50.016+0000 7f6777fff700 1 -- 172.21.15.42:0/1617185683 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6770011c50 con 0x7f6780144c50 2019-12-05T06:23:50.032 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:50.016+0000 7f6785370700 1 -- 172.21.15.42:0/1617185683 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f678007df50 con 0x7f6780144c50 2019-12-05T06:23:50.032 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:50.016+0000 7f6777fff700 1 -- 172.21.15.42:0/1617185683 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6770007340 con 0x7f6780144c50 2019-12-05T06:23:50.033 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:50.016+0000 7f6785370700 1 -- 172.21.15.42:0/1617185683 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6780130c10 con 0x7f6780144c50 2019-12-05T06:23:50.033 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:50.020+0000 7f677f7fe700 1 --2- 172.21.15.42:0/1617185683 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6780144c50 0x7f678011a5e0 secure :-1 s=THROTTLE_DONE pgs=557 cs=0 l=1 rx=0x7f6770008b10 tx=0x7f6770011670).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:50.033 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:50.020+0000 7f6777fff700 1 -- 172.21.15.42:0/1617185683 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f6770007540 con 0x7f6780144c50 2019-12-05T06:23:50.033 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:50.020+0000 7f677f7fe700 1 --2- 172.21.15.42:0/1617185683 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6780144c50 0x7f678011a5e0 secure :-1 s=THROTTLE_DONE pgs=557 cs=0 l=1 rx=0x7f6770008b10 tx=0x7f6770011670).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:50.033 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:50.020+0000 7f6777fff700 1 --2- 172.21.15.42:0/1617185683 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f676403b490 0x7f676403d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:23:50.033 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:50.020+0000 7f6777fff700 1 -- 172.21.15.42:0/1617185683 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 10761+0+0 (secure 0 0 0) 0x7f677004b6f0 con 0x7f6780144c50 2019-12-05T06:23:50.033 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:50.020+0000 7f677effd700 1 --2- 172.21.15.42:0/1617185683 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f676403b490 0x7f676403d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:50.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:50.020+0000 7f677effd700 1 --2- 172.21.15.42:0/1617185683 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f676403b490 0x7f676403d9a0 crc :-1 s=READY pgs=1071 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:23:50.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:50.024+0000 7f6785370700 1 -- 172.21.15.42:0/1617185683 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f678011ad60 con 0x7f6780144c50 2019-12-05T06:23:50.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:50.032+0000 7f677f7fe700 1 --2- 172.21.15.42:0/1617185683 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6780144c50 0x7f678011a5e0 secure :-1 s=THROTTLE_DONE pgs=557 cs=0 l=1 rx=0x7f6770008b10 tx=0x7f6770011670).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:50.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:50.032+0000 7f6777fff700 1 -- 172.21.15.42:0/1617185683 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f6770033580 con 0x7f6780144c50 2019-12-05T06:23:50.207 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:50.200+0000 7f6785370700 1 -- 172.21.15.42:0/1617185683 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f678005d110 con 0x7f6780144c50 2019-12-05T06:23:50.208 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:50.200+0000 7f677f7fe700 1 --2- 172.21.15.42:0/1617185683 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6780144c50 0x7f678011a5e0 secure :-1 s=THROTTLE_DONE pgs=557 cs=0 l=1 rx=0x7f6770008b10 tx=0x7f6770011670).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:50.208 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:50.200+0000 7f6777fff700 1 -- 172.21.15.42:0/1617185683 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 307 v307) v1 ==== 95+0+1779 (secure 0 0 0) 0x7f677000eb20 con 0x7f6780144c50 2019-12-05T06:23:50.209 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 307 2019-12-05T06:23:50.209 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:23:50.209 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":307,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":307,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:23:50.178163+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":104,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10993},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10993":{"gid":10993,"name":"a","rank":0,"incarnation":304,"state":"up:active","state_seq":33,"addr":"172.21.15.42:6835/3964426552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3964426552},{"type":"v1","addr":"172.21.15.42:6835","nonce":3964426552}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:23:50.211 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:50.200+0000 7f6785370700 1 -- 172.21.15.42:0/1617185683 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f676403b490 msgr2=0x7f676403d9a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:50.211 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:50.200+0000 7f6785370700 1 --2- 172.21.15.42:0/1617185683 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f676403b490 0x7f676403d9a0 crc :-1 s=READY pgs=1071 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:50.211 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:50.200+0000 7f6785370700 1 -- 172.21.15.42:0/1617185683 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6780144c50 msgr2=0x7f678011a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:50.211 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:50.200+0000 7f6785370700 1 --2- 172.21.15.42:0/1617185683 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6780144c50 0x7f678011a5e0 secure :-1 s=READY pgs=557 cs=0 l=1 rx=0x7f6770008b10 tx=0x7f6770011670).stop 2019-12-05T06:23:50.211 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:50.200+0000 7f6785370700 1 -- 172.21.15.42:0/1617185683 shutdown_connections 2019-12-05T06:23:50.211 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:50.200+0000 7f6785370700 1 --2- 172.21.15.42:0/1617185683 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f676403b490 0x7f676403d9a0 unknown :-1 s=CLOSED pgs=1071 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:50.212 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:50.200+0000 7f6785370700 1 --2- 172.21.15.42:0/1617185683 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6780144c50 0x7f678011a5e0 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:50.212 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:50.200+0000 7f6785370700 1 -- 172.21.15.42:0/1617185683 shutdown_connections 2019-12-05T06:23:50.212 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:50.204+0000 7f6785370700 1 -- 172.21.15.42:0/1617185683 wait complete. 2019-12-05T06:23:50.212 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:50.204+0000 7f6785370700 1 -- 172.21.15.42:0/1617185683 >> 172.21.15.42:0/1617185683 conn(0x7f67800a9b50 msgr2=0x7f678009f380 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:23:50.219 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a has gained rank=0, replacing gid=10710 2019-12-05T06:23:50.219 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 4 secs before reviving mds.b 2019-12-05T06:23:54.220 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.b 2019-12-05T06:23:54.220 INFO:tasks.ceph.mds.b:Restarting daemon 2019-12-05T06:23:54.221 INFO:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2019-12-05T06:23:54.224 INFO:tasks.ceph.mds.b:Started 2019-12-05T06:23:54.224 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:23:54.284 INFO:tasks.ceph.mds.b.smithi040.stdout:starting mds.b at 2019-12-05T06:23:54.344 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:54.328+0000 7fd8f0c7b700 1 Processor -- start 2019-12-05T06:23:54.344 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:54.328+0000 7fd8f0c7b700 1 -- start start 2019-12-05T06:23:54.344 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:54.328+0000 7fd8f0c7b700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd8ec0a7740 0x7fd8ec08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:23:54.344 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:54.328+0000 7fd8f0c7b700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8ec0a0710 con 0x7fd8ec09fae0 2019-12-05T06:23:54.345 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:54.328+0000 7fd8f0c7b700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8ec0a9ba0 con 0x7fd8ec07e720 2019-12-05T06:23:54.345 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:54.328+0000 7fd8f0c7b700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8ec0a0b00 con 0x7fd8ec0a7740 2019-12-05T06:23:54.345 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:54.328+0000 7fd8ead9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd8ec0a7740 0x7fd8ec08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:54.345 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:54.328+0000 7fd8ead9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd8ec0a7740 0x7fd8ec08df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:54580/0 (socket says 172.21.15.42:54580) 2019-12-05T06:23:54.345 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:54.328+0000 7fd8ead9d700 1 -- 172.21.15.42:0/3073870371 learned_addr learned my addr 172.21.15.42:0/3073870371 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:23:54.345 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:54.328+0000 7fd8ead9d700 1 -- 172.21.15.42:0/3073870371 >> v1:172.21.15.42:6789/0 conn(0x7fd8ec07e720 legacy=0x7fd8ec09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:54.345 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:54.328+0000 7fd8ead9d700 1 -- 172.21.15.42:0/3073870371 >> v1:172.21.15.40:6789/0 conn(0x7fd8ec09fae0 legacy=0x7fd8ec14efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:54.346 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:54.328+0000 7fd8ead9d700 1 -- 172.21.15.42:0/3073870371 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8ec07eb50 con 0x7fd8ec0a7740 2019-12-05T06:23:54.346 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:54.328+0000 7fd8ead9d700 1 --2- 172.21.15.42:0/3073870371 >> v2:172.21.15.42:3300/0 conn(0x7fd8ec0a7740 0x7fd8ec08df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fd8e4008b10 tx=0x7fd8e4010030).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:54.346 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:54.328+0000 7fd8ead9d700 1 --2- 172.21.15.42:0/3073870371 >> v2:172.21.15.42:3300/0 conn(0x7fd8ec0a7740 0x7fd8ec08df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fd8e4008b10 tx=0x7fd8e4010030).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:54.346 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:54.328+0000 7fd8ead9d700 1 --2- 172.21.15.42:0/3073870371 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd8ec0a7740 0x7fd8ec08df80 secure :-1 s=READY pgs=2582 cs=0 l=1 rx=0x7fd8e4008b10 tx=0x7fd8e4010030).ready entity=mon.1 client_cookie=d9cfa3bc36d1750 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:23:54.346 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:54.328+0000 7fd8ead9d700 1 --2- 172.21.15.42:0/3073870371 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd8ec0a7740 0x7fd8ec08df80 secure :-1 s=THROTTLE_DONE pgs=2582 cs=0 l=1 rx=0x7fd8e4008b10 tx=0x7fd8e4010030).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:54.346 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:54.328+0000 7fd8e9d9b700 1 -- 172.21.15.42:0/3073870371 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd8e400a160 con 0x7fd8ec0a7740 2019-12-05T06:23:54.346 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:54.328+0000 7fd8ead9d700 1 --2- 172.21.15.42:0/3073870371 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd8ec0a7740 0x7fd8ec08df80 secure :-1 s=THROTTLE_DONE pgs=2582 cs=0 l=1 rx=0x7fd8e4008b10 tx=0x7fd8e4010030).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:54.347 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:54.328+0000 7fd8e9d9b700 1 -- 172.21.15.42:0/3073870371 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8e4011aa0 con 0x7fd8ec0a7740 2019-12-05T06:23:54.347 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:54.328+0000 7fd8ead9d700 1 --2- 172.21.15.42:0/3073870371 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd8ec0a7740 0x7fd8ec08df80 secure :-1 s=THROTTLE_DONE pgs=2582 cs=0 l=1 rx=0x7fd8e4008b10 tx=0x7fd8e4010030).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:54.347 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:54.328+0000 7fd8e9d9b700 1 -- 172.21.15.42:0/3073870371 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd8e400a740 con 0x7fd8ec0a7740 2019-12-05T06:23:54.347 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:54.328+0000 7fd8f0c7b700 1 -- 172.21.15.42:0/3073870371 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd8ec0a7740 msgr2=0x7fd8ec08df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:54.347 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:54.328+0000 7fd8f0c7b700 1 --2- 172.21.15.42:0/3073870371 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd8ec0a7740 0x7fd8ec08df80 secure :-1 s=READY pgs=2582 cs=0 l=1 rx=0x7fd8e4008b10 tx=0x7fd8e4010030).stop 2019-12-05T06:23:54.347 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:54.328+0000 7fd8f0c7b700 1 -- 172.21.15.42:0/3073870371 shutdown_connections 2019-12-05T06:23:54.347 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:54.328+0000 7fd8f0c7b700 1 --2- 172.21.15.42:0/3073870371 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd8ec0a7740 0x7fd8ec08df80 unknown :-1 s=CLOSED pgs=2582 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:54.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:54.328+0000 7fd8f0c7b700 1 -- 172.21.15.42:0/3073870371 shutdown_connections 2019-12-05T06:23:54.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:54.328+0000 7fd8f0c7b700 1 -- 172.21.15.42:0/3073870371 wait complete. 2019-12-05T06:23:54.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:54.328+0000 7fd8f0c7b700 1 -- 172.21.15.42:0/3073870371 >> 172.21.15.42:0/3073870371 conn(0x7fd8ec0aabe0 msgr2=0x7fd8ec0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:23:54.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:54.332+0000 7fd8f0c7b700 1 Processor -- start 2019-12-05T06:23:54.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:54.332+0000 7fd8f0c7b700 1 -- start start 2019-12-05T06:23:54.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:54.332+0000 7fd8f0c7b700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd8ec07dff0 0x7fd8ec145040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:23:54.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:54.332+0000 7fd8f0c7b700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8ec143b40 con 0x7fd8ec1199d0 2019-12-05T06:23:54.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:54.332+0000 7fd8f0c7b700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8ec125a00 con 0x7fd8ec142530 2019-12-05T06:23:54.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:54.332+0000 7fd8f0c7b700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8ec144610 con 0x7fd8ec07dff0 2019-12-05T06:23:54.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:54.332+0000 7fd8ea59c700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fd8ec142530 0x7fd8ec07d7f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:54626/0 (socket says 172.21.15.42:54626) 2019-12-05T06:23:54.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:54.332+0000 7fd8ea59c700 1 -- 172.21.15.42:0/2239396030 learned_addr learned my addr 172.21.15.42:0/2239396030 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:23:54.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:54.332+0000 7fd8eb59e700 1 --2- 172.21.15.42:0/2239396030 >> v2:172.21.15.42:3300/0 conn(0x7fd8ec07dff0 0x7fd8ec145040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:54.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:54.336+0000 7fd8eb59e700 1 -- 172.21.15.42:0/2239396030 >> v1:172.21.15.42:6789/0 conn(0x7fd8ec142530 legacy=0x7fd8ec07d7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:54.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:54.336+0000 7fd8eb59e700 1 -- 172.21.15.42:0/2239396030 >> v1:172.21.15.40:6789/0 conn(0x7fd8ec1199d0 legacy=0x7fd8ec143e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:54.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:54.336+0000 7fd8eb59e700 1 -- 172.21.15.42:0/2239396030 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8ec12fe80 con 0x7fd8ec07dff0 2019-12-05T06:23:54.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:54.336+0000 7fd8eb59e700 1 --2- 172.21.15.42:0/2239396030 >> v2:172.21.15.42:3300/0 conn(0x7fd8ec07dff0 0x7fd8ec145040 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fd8ec1433c0 tx=0x7fd8d4010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:54.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:54.336+0000 7fd8eb59e700 1 --2- 172.21.15.42:0/2239396030 >> v2:172.21.15.42:3300/0 conn(0x7fd8ec07dff0 0x7fd8ec145040 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fd8ec1433c0 tx=0x7fd8d4010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:54.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:54.336+0000 7fd8eb59e700 1 --2- 172.21.15.42:0/2239396030 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd8ec07dff0 0x7fd8ec145040 secure :-1 s=READY pgs=2584 cs=0 l=1 rx=0x7fd8ec1433c0 tx=0x7fd8d4010740).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:23:54.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:54.336+0000 7fd8eb59e700 1 --2- 172.21.15.42:0/2239396030 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd8ec07dff0 0x7fd8ec145040 secure :-1 s=THROTTLE_DONE pgs=2584 cs=0 l=1 rx=0x7fd8ec1433c0 tx=0x7fd8d4010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:54.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:54.336+0000 7fd8e37fe700 1 -- 172.21.15.42:0/2239396030 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd8d4006b30 con 0x7fd8ec07dff0 2019-12-05T06:23:54.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:54.336+0000 7fd8eb59e700 1 --2- 172.21.15.42:0/2239396030 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd8ec07dff0 0x7fd8ec145040 secure :-1 s=THROTTLE_DONE pgs=2584 cs=0 l=1 rx=0x7fd8ec1433c0 tx=0x7fd8d4010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:54.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:54.336+0000 7fd8eb59e700 1 --2- 172.21.15.42:0/2239396030 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd8ec07dff0 0x7fd8ec145040 secure :-1 s=THROTTLE_DONE pgs=2584 cs=0 l=1 rx=0x7fd8ec1433c0 tx=0x7fd8d4010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:54.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:54.336+0000 7fd8e37fe700 1 -- 172.21.15.42:0/2239396030 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8d4006750 con 0x7fd8ec07dff0 2019-12-05T06:23:54.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:54.336+0000 7fd8f0c7b700 1 -- 172.21.15.42:0/2239396030 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8ec12de60 con 0x7fd8ec07dff0 2019-12-05T06:23:54.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:54.336+0000 7fd8e37fe700 1 -- 172.21.15.42:0/2239396030 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd8d4010f40 con 0x7fd8ec07dff0 2019-12-05T06:23:54.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:54.336+0000 7fd8f0c7b700 1 -- 172.21.15.42:0/2239396030 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8ec12e3e0 con 0x7fd8ec07dff0 2019-12-05T06:23:54.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:54.336+0000 7fd8eb59e700 1 --2- 172.21.15.42:0/2239396030 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd8ec07dff0 0x7fd8ec145040 secure :-1 s=THROTTLE_DONE pgs=2584 cs=0 l=1 rx=0x7fd8ec1433c0 tx=0x7fd8d4010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:54.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:54.336+0000 7fd8e37fe700 1 -- 172.21.15.42:0/2239396030 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fd8d4011250 con 0x7fd8ec07dff0 2019-12-05T06:23:54.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:54.336+0000 7fd8eb59e700 1 --2- 172.21.15.42:0/2239396030 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd8ec07dff0 0x7fd8ec145040 secure :-1 s=THROTTLE_DONE pgs=2584 cs=0 l=1 rx=0x7fd8ec1433c0 tx=0x7fd8d4010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:54.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:54.336+0000 7fd8e37fe700 1 --2- 172.21.15.42:0/2239396030 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd8d003ac80 0x7fd8d003d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:23:54.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:54.336+0000 7fd8e37fe700 1 -- 172.21.15.42:0/2239396030 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 10761+0+0 (secure 0 0 0) 0x7fd8d404b190 con 0x7fd8ec07dff0 2019-12-05T06:23:54.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:54.336+0000 7fd8ead9d700 1 --2- 172.21.15.42:0/2239396030 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd8d003ac80 0x7fd8d003d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:54.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:54.336+0000 7fd8ead9d700 1 --2- 172.21.15.42:0/2239396030 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd8d003ac80 0x7fd8d003d190 crc :-1 s=READY pgs=1072 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:23:54.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:54.344+0000 7fd8f0c7b700 1 -- 172.21.15.42:0/2239396030 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8ec05d110 con 0x7fd8ec07dff0 2019-12-05T06:23:54.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:54.348+0000 7fd8eb59e700 1 --2- 172.21.15.42:0/2239396030 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd8ec07dff0 0x7fd8ec145040 secure :-1 s=THROTTLE_DONE pgs=2584 cs=0 l=1 rx=0x7fd8ec1433c0 tx=0x7fd8d4010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:54.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:54.348+0000 7fd8e37fe700 1 -- 172.21.15.42:0/2239396030 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fd8d4032210 con 0x7fd8ec07dff0 2019-12-05T06:23:54.522 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:54.512+0000 7fd8f0c7b700 1 -- 172.21.15.42:0/2239396030 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd8ec05b8c0 con 0x7fd8ec07dff0 2019-12-05T06:23:54.522 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:54.512+0000 7fd8eb59e700 1 --2- 172.21.15.42:0/2239396030 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd8ec07dff0 0x7fd8ec145040 secure :-1 s=THROTTLE_DONE pgs=2584 cs=0 l=1 rx=0x7fd8ec1433c0 tx=0x7fd8d4010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:54.522 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:54.512+0000 7fd8e37fe700 1 -- 172.21.15.42:0/2239396030 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 307 v307) v1 ==== 95+0+1779 (secure 0 0 0) 0x7fd8d4032470 con 0x7fd8ec07dff0 2019-12-05T06:23:54.523 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 307 2019-12-05T06:23:54.523 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:54.516+0000 7fd8f0c7b700 1 -- 172.21.15.42:0/2239396030 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd8d003ac80 msgr2=0x7fd8d003d190 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:54.523 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:54.516+0000 7fd8f0c7b700 1 --2- 172.21.15.42:0/2239396030 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd8d003ac80 0x7fd8d003d190 crc :-1 s=READY pgs=1072 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:54.523 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:54.516+0000 7fd8f0c7b700 1 -- 172.21.15.42:0/2239396030 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd8ec07dff0 msgr2=0x7fd8ec145040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:54.524 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:54.516+0000 7fd8f0c7b700 1 --2- 172.21.15.42:0/2239396030 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd8ec07dff0 0x7fd8ec145040 secure :-1 s=READY pgs=2584 cs=0 l=1 rx=0x7fd8ec1433c0 tx=0x7fd8d4010740).stop 2019-12-05T06:23:54.524 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:54.516+0000 7fd8f0c7b700 1 -- 172.21.15.42:0/2239396030 shutdown_connections 2019-12-05T06:23:54.524 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:54.516+0000 7fd8f0c7b700 1 --2- 172.21.15.42:0/2239396030 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd8d003ac80 0x7fd8d003d190 unknown :-1 s=CLOSED pgs=1072 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:54.524 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:54.516+0000 7fd8f0c7b700 1 --2- 172.21.15.42:0/2239396030 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd8ec07dff0 0x7fd8ec145040 unknown :-1 s=CLOSED pgs=2584 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:54.524 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:54.516+0000 7fd8f0c7b700 1 -- 172.21.15.42:0/2239396030 shutdown_connections 2019-12-05T06:23:54.524 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:54.516+0000 7fd8f0c7b700 1 -- 172.21.15.42:0/2239396030 wait complete. 2019-12-05T06:23:54.525 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:54.516+0000 7fd8f0c7b700 1 -- 172.21.15.42:0/2239396030 >> 172.21.15.42:0/2239396030 conn(0x7fd8ec0a7740 msgr2=0x7fd8ec116430 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:23:54.525 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:23:54.525 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":307,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":307,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:23:50.178163+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":104,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10993},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10993":{"gid":10993,"name":"a","rank":0,"incarnation":304,"state":"up:active","state_seq":33,"addr":"172.21.15.42:6835/3964426552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3964426552},{"type":"v1","addr":"172.21.15.42:6835","nonce":3964426552}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:23:54.532 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2019-12-05T06:23:56.535 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:23:57.147 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:57.136+0000 7f4fbed2b700 1 Processor -- start 2019-12-05T06:23:57.147 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:57.136+0000 7f4fbed2b700 1 -- start start 2019-12-05T06:23:57.147 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:57.136+0000 7f4fbed2b700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f4fb808df80 0x7f4fb809f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:23:57.147 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:57.136+0000 7f4fbed2b700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fb809fc60 con 0x7f4fb80a7740 2019-12-05T06:23:57.148 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:57.136+0000 7f4fbed2b700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fb80a0710 con 0x7f4fb809f830 2019-12-05T06:23:57.148 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:57.136+0000 7f4fbed2b700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4fb80a0b00 con 0x7f4fb808df80 2019-12-05T06:23:57.148 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:57.136+0000 7f4fbdac9700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f4fb809f830 0x7f4fb814efe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:54650/0 (socket says 172.21.15.42:54650) 2019-12-05T06:23:57.148 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:57.136+0000 7f4fbdac9700 1 -- 172.21.15.42:0/1147579583 learned_addr learned my addr 172.21.15.42:0/1147579583 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:23:57.148 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:57.136+0000 7f4fbcac7700 1 --2- 172.21.15.42:0/1147579583 >> v2:172.21.15.40:3300/0 conn(0x7f4fb808df80 0x7f4fb809f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:57.148 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:57.136+0000 7f4fabfff700 1 -- 172.21.15.42:0/1147579583 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1666893274 0 0) 0x7f4fb80a0710 con 0x7f4fb809f830 2019-12-05T06:23:57.149 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:57.136+0000 7f4fabfff700 1 -- 172.21.15.42:0/1147579583 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4fa40030b0 con 0x7f4fb809f830 2019-12-05T06:23:57.149 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:57.136+0000 7f4fbcac7700 1 -- 172.21.15.42:0/1147579583 >> v2:172.21.15.40:3300/0 conn(0x7f4fb808df80 msgr2=0x7f4fb809f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2019-12-05T06:23:57.149 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:57.136+0000 7f4fbcac7700 1 -- 172.21.15.42:0/1147579583 >> v2:172.21.15.40:3300/0 conn(0x7f4fb808df80 msgr2=0x7f4fb809f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2019-12-05T06:23:57.149 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:57.136+0000 7f4fbcac7700 1 --2- 172.21.15.42:0/1147579583 >> v2:172.21.15.40:3300/0 conn(0x7f4fb808df80 0x7f4fb809f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-12-05T06:23:57.149 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:57.136+0000 7f4fbcac7700 1 --2- 172.21.15.42:0/1147579583 >> v2:172.21.15.40:3300/0 conn(0x7f4fb808df80 0x7f4fb809f2e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rx=0 tx=0)._fault waiting 0.200000 2019-12-05T06:23:57.149 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:57.136+0000 7f4fabfff700 1 -- 172.21.15.42:0/1147579583 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1991554580 0 0) 0x7f4fa40030b0 con 0x7f4fb809f830 2019-12-05T06:23:57.149 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:57.136+0000 7f4fabfff700 1 -- 172.21.15.42:0/1147579583 >> v2:172.21.15.40:3300/0 conn(0x7f4fb808df80 msgr2=0x7f4fb809f2e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2019-12-05T06:23:57.150 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:57.136+0000 7f4fabfff700 1 --2- 172.21.15.42:0/1147579583 >> v2:172.21.15.40:3300/0 conn(0x7f4fb808df80 0x7f4fb809f2e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:23:57.150 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:57.136+0000 7f4fabfff700 1 -- 172.21.15.42:0/1147579583 >> v1:172.21.15.40:6789/0 conn(0x7f4fb80a7740 legacy=0x7f4fb807e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:57.150 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:57.136+0000 7f4fabfff700 1 -- 172.21.15.42:0/1147579583 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fb807ef20 con 0x7f4fb809f830 2019-12-05T06:23:57.150 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:57.136+0000 7f4fabfff700 1 -- 172.21.15.42:0/1147579583 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7f4fb0001ad0 con 0x7f4fb809f830 2019-12-05T06:23:57.150 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:57.136+0000 7f4fabfff700 1 -- 172.21.15.42:0/1147579583 >> v1:172.21.15.42:6789/0 conn(0x7f4fb809f830 legacy=0x7f4fb814efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:57.150 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:57.136+0000 7f4fabfff700 1 --2- 172.21.15.42:0/1147579583 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4fa40036e0 0x7f4fa4003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:23:57.150 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:57.136+0000 7f4fabfff700 1 --2- 172.21.15.42:0/1147579583 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4fa4004420 0x7f4fa4004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:23:57.151 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:57.136+0000 7f4fabfff700 1 --2- 172.21.15.42:0/1147579583 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4fa4004df0 0x7f4fa400b670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:23:57.151 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:57.136+0000 7f4fabfff700 1 -- 172.21.15.42:0/1147579583 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fb80a0b00 con 0x7f4fa4004df0 2019-12-05T06:23:57.151 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:57.136+0000 7f4fabfff700 1 -- 172.21.15.42:0/1147579583 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fa404fce0 con 0x7f4fa40036e0 2019-12-05T06:23:57.151 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:57.136+0000 7f4fabfff700 1 -- 172.21.15.42:0/1147579583 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fa404ff20 con 0x7f4fa4004420 2019-12-05T06:23:57.151 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:57.136+0000 7f4fbd2c8700 1 --2- 172.21.15.42:0/1147579583 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4fa40036e0 0x7f4fa4003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:57.151 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:57.136+0000 7f4fbcac7700 1 --2- 172.21.15.42:0/1147579583 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4fa4004420 0x7f4fa4004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:57.151 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:57.136+0000 7f4fbdac9700 1 --2- 172.21.15.42:0/1147579583 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4fa4004df0 0x7f4fa400b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:57.152 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:57.140+0000 7f4fbd2c8700 1 -- 172.21.15.42:0/1147579583 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4fa4004420 msgr2=0x7f4fa4004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:23:57.152 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:57.140+0000 7f4fbd2c8700 1 --2- 172.21.15.42:0/1147579583 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4fa4004420 0x7f4fa4004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:23:57.152 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:57.140+0000 7f4fbd2c8700 1 -- 172.21.15.42:0/1147579583 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4fa4004df0 msgr2=0x7f4fa400b670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:23:57.152 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:57.140+0000 7f4fbd2c8700 1 --2- 172.21.15.42:0/1147579583 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4fa4004df0 0x7f4fa400b670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:23:57.152 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:57.140+0000 7f4fbd2c8700 1 -- 172.21.15.42:0/1147579583 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fa4050220 con 0x7f4fa40036e0 2019-12-05T06:23:57.152 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:57.140+0000 7f4fbdac9700 1 --2- 172.21.15.42:0/1147579583 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4fa4004df0 0x7f4fa400b670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:23:57.153 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:57.140+0000 7f4fbdac9700 1 -- 172.21.15.42:0/1147579583 reap_dead start 2019-12-05T06:23:57.153 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:57.140+0000 7f4fbcac7700 1 --2- 172.21.15.42:0/1147579583 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4fa4004420 0x7f4fa4004850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:23:57.153 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:57.140+0000 7f4fbd2c8700 1 --2- 172.21.15.42:0/1147579583 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4fa40036e0 0x7f4fa4003b50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f4fb4001000 tx=0x7f4fb400fcd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:57.153 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:57.140+0000 7f4fbd2c8700 1 --2- 172.21.15.42:0/1147579583 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4fa40036e0 0x7f4fa4003b50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f4fb4001000 tx=0x7f4fb400fcd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:57.153 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:57.140+0000 7f4fbd2c8700 1 --2- 172.21.15.42:0/1147579583 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4fa40036e0 0x7f4fa4003b50 secure :-1 s=READY pgs=2586 cs=0 l=1 rx=0x7f4fb4001000 tx=0x7f4fb400fcd0).ready entity=mon.1 client_cookie=d83eebf734f34a59 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:23:57.153 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:57.140+0000 7f4fbd2c8700 1 --2- 172.21.15.42:0/1147579583 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4fa40036e0 0x7f4fa4003b50 secure :-1 s=THROTTLE_DONE pgs=2586 cs=0 l=1 rx=0x7f4fb4001000 tx=0x7f4fb400fcd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:57.153 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:57.140+0000 7f4fabfff700 1 -- 172.21.15.42:0/1147579583 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4fb4003540 con 0x7f4fa40036e0 2019-12-05T06:23:57.154 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:57.140+0000 7f4fbd2c8700 1 --2- 172.21.15.42:0/1147579583 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4fa40036e0 0x7f4fa4003b50 secure :-1 s=THROTTLE_DONE pgs=2586 cs=0 l=1 rx=0x7f4fb4001000 tx=0x7f4fb400fcd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:57.154 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:57.140+0000 7f4fbd2c8700 1 --2- 172.21.15.42:0/1147579583 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4fa40036e0 0x7f4fa4003b50 secure :-1 s=THROTTLE_DONE pgs=2586 cs=0 l=1 rx=0x7f4fb4001000 tx=0x7f4fb400fcd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:57.154 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:57.140+0000 7f4fabfff700 1 -- 172.21.15.42:0/1147579583 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fb4003160 con 0x7f4fa40036e0 2019-12-05T06:23:57.154 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:57.140+0000 7f4fabfff700 1 -- 172.21.15.42:0/1147579583 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4fb4003b20 con 0x7f4fa40036e0 2019-12-05T06:23:57.154 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:57.140+0000 7f4fbed2b700 1 -- 172.21.15.42:0/1147579583 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4fa40036e0 msgr2=0x7f4fa4003b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:57.154 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:57.140+0000 7f4fbed2b700 1 --2- 172.21.15.42:0/1147579583 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4fa40036e0 0x7f4fa4003b50 secure :-1 s=READY pgs=2586 cs=0 l=1 rx=0x7f4fb4001000 tx=0x7f4fb400fcd0).stop 2019-12-05T06:23:57.154 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:57.140+0000 7f4fbed2b700 1 -- 172.21.15.42:0/1147579583 shutdown_connections 2019-12-05T06:23:57.155 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:57.140+0000 7f4fbed2b700 1 --2- 172.21.15.42:0/1147579583 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4fa40036e0 0x7f4fa4003b50 unknown :-1 s=CLOSED pgs=2586 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:57.155 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:57.140+0000 7f4fbed2b700 1 -- 172.21.15.42:0/1147579583 shutdown_connections 2019-12-05T06:23:57.155 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:57.140+0000 7f4fbed2b700 1 -- 172.21.15.42:0/1147579583 wait complete. 2019-12-05T06:23:57.155 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:57.140+0000 7f4fbed2b700 1 -- 172.21.15.42:0/1147579583 >> 172.21.15.42:0/1147579583 conn(0x7f4fb80aabe0 msgr2=0x7f4fb80a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:23:57.158 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:57.140+0000 7f4fbed2b700 1 Processor -- start 2019-12-05T06:23:57.158 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:57.140+0000 7f4fbed2b700 1 -- start start 2019-12-05T06:23:57.158 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:57.144+0000 7f4fbed2b700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f4fb80a1630 0x7f4fb812ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:23:57.159 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:57.144+0000 7f4fbed2b700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f4fb81249d0 0x7f4fb8124e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:23:57.159 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:57.144+0000 7f4fbed2b700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fb812f3b0 con 0x7f4fb811d890 2019-12-05T06:23:57.159 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:57.144+0000 7f4fbed2b700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4fb812e680 con 0x7f4fb81249d0 2019-12-05T06:23:57.159 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:57.144+0000 7f4fbed2b700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4fb812e920 con 0x7f4fb80a1630 2019-12-05T06:23:57.159 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:57.144+0000 7f4fbcac7700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f4fb80a1630 0x7f4fb812ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:57.159 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:57.144+0000 7f4fbcac7700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f4fb80a1630 0x7f4fb812ee60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:54624/0 (socket says 172.21.15.42:54624) 2019-12-05T06:23:57.159 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:57.144+0000 7f4fbcac7700 1 -- 172.21.15.42:0/50519435 learned_addr learned my addr 172.21.15.42:0/50519435 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:23:57.160 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:57.144+0000 7f4fbdac9700 1 --2- 172.21.15.42:0/50519435 >> v2:172.21.15.40:3300/0 conn(0x7f4fb81249d0 0x7f4fb8124e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:57.160 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:57.144+0000 7f4fbcac7700 1 -- 172.21.15.42:0/50519435 >> v2:172.21.15.40:3300/0 conn(0x7f4fb81249d0 msgr2=0x7f4fb8124e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:57.160 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:57.144+0000 7f4fbcac7700 1 --2- 172.21.15.42:0/50519435 >> v2:172.21.15.40:3300/0 conn(0x7f4fb81249d0 0x7f4fb8124e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:57.160 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:57.144+0000 7f4fbcac7700 1 -- 172.21.15.42:0/50519435 >> v1:172.21.15.40:6789/0 conn(0x7f4fb811d890 legacy=0x7f4fb80a3100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:57.160 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:57.144+0000 7f4fbcac7700 1 -- 172.21.15.42:0/50519435 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fb81452f0 con 0x7f4fb80a1630 2019-12-05T06:23:57.160 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:57.144+0000 7f4fbcac7700 1 --2- 172.21.15.42:0/50519435 >> v2:172.21.15.42:3300/0 conn(0x7f4fb80a1630 0x7f4fb812ee60 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f4fb811ddf0 tx=0x7f4fac0108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:57.161 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:57.144+0000 7f4fbcac7700 1 --2- 172.21.15.42:0/50519435 >> v2:172.21.15.42:3300/0 conn(0x7f4fb80a1630 0x7f4fb812ee60 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f4fb811ddf0 tx=0x7f4fac0108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:57.161 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:57.144+0000 7f4fbcac7700 1 --2- 172.21.15.42:0/50519435 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4fb80a1630 0x7f4fb812ee60 secure :-1 s=READY pgs=2587 cs=0 l=1 rx=0x7f4fb811ddf0 tx=0x7f4fac0108f0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:23:57.161 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:57.144+0000 7f4fbcac7700 1 --2- 172.21.15.42:0/50519435 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4fb80a1630 0x7f4fb812ee60 secure :-1 s=THROTTLE_DONE pgs=2587 cs=0 l=1 rx=0x7f4fb811ddf0 tx=0x7f4fac0108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:57.161 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:57.144+0000 7f4fbcac7700 1 --2- 172.21.15.42:0/50519435 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4fb80a1630 0x7f4fb812ee60 secure :-1 s=THROTTLE_DONE pgs=2587 cs=0 l=1 rx=0x7f4fb811ddf0 tx=0x7f4fac0108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:57.161 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:57.144+0000 7f4fa9ffb700 1 -- 172.21.15.42:0/50519435 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4fac002690 con 0x7f4fb80a1630 2019-12-05T06:23:57.161 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:57.144+0000 7f4fbcac7700 1 --2- 172.21.15.42:0/50519435 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4fb80a1630 0x7f4fb812ee60 secure :-1 s=THROTTLE_DONE pgs=2587 cs=0 l=1 rx=0x7f4fb811ddf0 tx=0x7f4fac0108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:57.161 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:57.144+0000 7f4fa9ffb700 1 -- 172.21.15.42:0/50519435 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fac0022b0 con 0x7f4fb80a1630 2019-12-05T06:23:57.161 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:57.144+0000 7f4fbed2b700 1 -- 172.21.15.42:0/50519435 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fb8125350 con 0x7f4fb80a1630 2019-12-05T06:23:57.162 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:57.144+0000 7f4fa9ffb700 1 -- 172.21.15.42:0/50519435 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4fac002cb0 con 0x7f4fb80a1630 2019-12-05T06:23:57.162 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:57.144+0000 7f4fbed2b700 1 -- 172.21.15.42:0/50519435 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fb81359c0 con 0x7f4fb80a1630 2019-12-05T06:23:57.162 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:57.144+0000 7f4fbcac7700 1 --2- 172.21.15.42:0/50519435 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4fb80a1630 0x7f4fb812ee60 secure :-1 s=THROTTLE_DONE pgs=2587 cs=0 l=1 rx=0x7f4fb811ddf0 tx=0x7f4fac0108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:57.162 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:57.148+0000 7f4fa9ffb700 1 -- 172.21.15.42:0/50519435 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f4fac005540 con 0x7f4fb80a1630 2019-12-05T06:23:57.162 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:57.148+0000 7f4fbcac7700 1 --2- 172.21.15.42:0/50519435 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4fb80a1630 0x7f4fb812ee60 secure :-1 s=THROTTLE_DONE pgs=2587 cs=0 l=1 rx=0x7f4fb811ddf0 tx=0x7f4fac0108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:57.162 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:57.148+0000 7f4fa9ffb700 1 --2- 172.21.15.42:0/50519435 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4f9803ad80 0x7f4f9803d290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:23:57.163 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:57.148+0000 7f4fa9ffb700 1 -- 172.21.15.42:0/50519435 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 10761+0+0 (secure 0 0 0) 0x7f4fac04ad60 con 0x7f4fb80a1630 2019-12-05T06:23:57.163 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:57.148+0000 7f4fbd2c8700 1 --2- 172.21.15.42:0/50519435 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4f9803ad80 0x7f4f9803d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:57.163 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:57.148+0000 7f4fbd2c8700 1 --2- 172.21.15.42:0/50519435 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4f9803ad80 0x7f4f9803d290 crc :-1 s=READY pgs=1073 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:23:57.169 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:57.152+0000 7f4fbed2b700 1 -- 172.21.15.42:0/50519435 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4fb805d110 con 0x7f4fb80a1630 2019-12-05T06:23:57.170 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:57.160+0000 7f4fbcac7700 1 --2- 172.21.15.42:0/50519435 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4fb80a1630 0x7f4fb812ee60 secure :-1 s=THROTTLE_DONE pgs=2587 cs=0 l=1 rx=0x7f4fb811ddf0 tx=0x7f4fac0108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:57.170 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:57.160+0000 7f4fa9ffb700 1 -- 172.21.15.42:0/50519435 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f4fac031de0 con 0x7f4fb80a1630 2019-12-05T06:23:57.330 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:57.324+0000 7f4fbed2b700 1 -- 172.21.15.42:0/50519435 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4fb805b8c0 con 0x7f4fb80a1630 2019-12-05T06:23:57.331 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:57.324+0000 7f4fbcac7700 1 --2- 172.21.15.42:0/50519435 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4fb80a1630 0x7f4fb812ee60 secure :-1 s=THROTTLE_DONE pgs=2587 cs=0 l=1 rx=0x7f4fb811ddf0 tx=0x7f4fac0108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:57.331 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:57.324+0000 7f4fa9ffb700 1 -- 172.21.15.42:0/50519435 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 307 v307) v1 ==== 95+0+1779 (secure 0 0 0) 0x7f4fac032040 con 0x7f4fb80a1630 2019-12-05T06:23:57.331 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 307 2019-12-05T06:23:57.331 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:23:57.331 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":307,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":307,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:23:50.178163+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":104,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10993},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10993":{"gid":10993,"name":"a","rank":0,"incarnation":304,"state":"up:active","state_seq":33,"addr":"172.21.15.42:6835/3964426552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3964426552},{"type":"v1","addr":"172.21.15.42:6835","nonce":3964426552}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:23:57.333 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:57.328+0000 7f4fbed2b700 1 -- 172.21.15.42:0/50519435 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4f9803ad80 msgr2=0x7f4f9803d290 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:57.334 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:57.328+0000 7f4fbed2b700 1 --2- 172.21.15.42:0/50519435 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4f9803ad80 0x7f4f9803d290 crc :-1 s=READY pgs=1073 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:57.334 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:57.328+0000 7f4fbed2b700 1 -- 172.21.15.42:0/50519435 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4fb80a1630 msgr2=0x7f4fb812ee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:57.334 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:57.328+0000 7f4fbed2b700 1 --2- 172.21.15.42:0/50519435 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4fb80a1630 0x7f4fb812ee60 secure :-1 s=READY pgs=2587 cs=0 l=1 rx=0x7f4fb811ddf0 tx=0x7f4fac0108f0).stop 2019-12-05T06:23:57.334 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:57.328+0000 7f4fbed2b700 1 -- 172.21.15.42:0/50519435 shutdown_connections 2019-12-05T06:23:57.334 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:57.328+0000 7f4fbed2b700 1 --2- 172.21.15.42:0/50519435 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4f9803ad80 0x7f4f9803d290 unknown :-1 s=CLOSED pgs=1073 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:57.334 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:57.328+0000 7f4fbed2b700 1 --2- 172.21.15.42:0/50519435 >> v2:172.21.15.40:3300/0 conn(0x7f4fb81249d0 0x7f4fb8124e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:57.334 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:57.328+0000 7f4fbed2b700 1 --2- 172.21.15.42:0/50519435 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4fb80a1630 0x7f4fb812ee60 unknown :-1 s=CLOSED pgs=2587 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:57.335 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:57.328+0000 7f4fbed2b700 1 -- 172.21.15.42:0/50519435 shutdown_connections 2019-12-05T06:23:57.335 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:57.328+0000 7f4fbed2b700 1 -- 172.21.15.42:0/50519435 wait complete. 2019-12-05T06:23:57.335 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:57.328+0000 7f4fbed2b700 1 -- 172.21.15.42:0/50519435 >> 172.21.15.42:0/50519435 conn(0x7f4fb80a7740 msgr2=0x7f4fb807e570 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:23:57.344 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2019-12-05T06:23:59.345 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:23:59.470 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:59.448+0000 7f422b276700 1 Processor -- start 2019-12-05T06:23:59.471 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:59.448+0000 7f422b276700 1 -- start start 2019-12-05T06:23:59.471 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:59.448+0000 7f422b276700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f42240a7740 0x7f422408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:23:59.471 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:59.448+0000 7f422b276700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f422407e720 0x7f422407eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:23:59.471 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:59.448+0000 7f422b276700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42240a0710 con 0x7f422409f2e0 2019-12-05T06:23:59.472 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:59.448+0000 7f422b276700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42240a0b00 con 0x7f422407e720 2019-12-05T06:23:59.472 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:59.448+0000 7f422b276700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f422408e5b0 con 0x7f42240a7740 2019-12-05T06:23:59.472 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:59.448+0000 7f4229813700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f42240a7740 0x7f422408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:59.472 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:59.448+0000 7f4229813700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f42240a7740 0x7f422408df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:54642/0 (socket says 172.21.15.42:54642) 2019-12-05T06:23:59.472 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:59.448+0000 7f4229813700 1 -- 172.21.15.42:0/4211707892 learned_addr learned my addr 172.21.15.42:0/4211707892 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:23:59.472 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:59.448+0000 7f4229012700 1 --2- 172.21.15.42:0/4211707892 >> v2:172.21.15.40:3300/0 conn(0x7f422407e720 0x7f422407eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:59.473 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:59.448+0000 7f4229813700 1 -- 172.21.15.42:0/4211707892 >> v2:172.21.15.40:3300/0 conn(0x7f422407e720 msgr2=0x7f422407eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:23:59.473 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:59.448+0000 7f4229813700 1 --2- 172.21.15.42:0/4211707892 >> v2:172.21.15.40:3300/0 conn(0x7f422407e720 0x7f422407eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:23:59.473 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:59.448+0000 7f4229813700 1 -- 172.21.15.42:0/4211707892 >> v1:172.21.15.42:6789/0 conn(0x7f422409f2e0 legacy=0x7f422409f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:59.473 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:59.448+0000 7f4228811700 1 -- 172.21.15.42:0/4211707892 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2321150950 0 0) 0x7f42240a0710 con 0x7f422409f2e0 2019-12-05T06:23:59.473 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:59.448+0000 7f4229813700 1 -- 172.21.15.42:0/4211707892 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f422407f0a0 con 0x7f42240a7740 2019-12-05T06:23:59.473 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:59.448+0000 7f4229813700 1 --2- 172.21.15.42:0/4211707892 >> v2:172.21.15.42:3300/0 conn(0x7f42240a7740 0x7f422408df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f4220002f30 tx=0x7f422000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:59.473 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:59.452+0000 7f4229813700 1 --2- 172.21.15.42:0/4211707892 >> v2:172.21.15.42:3300/0 conn(0x7f42240a7740 0x7f422408df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f4220002f30 tx=0x7f422000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:59.474 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:59.452+0000 7f4229813700 1 --2- 172.21.15.42:0/4211707892 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f42240a7740 0x7f422408df80 secure :-1 s=READY pgs=2589 cs=0 l=1 rx=0x7f4220002f30 tx=0x7f422000fd30).ready entity=mon.1 client_cookie=a9cf7bd1425878c9 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:23:59.474 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:59.452+0000 7f4229813700 1 --2- 172.21.15.42:0/4211707892 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f42240a7740 0x7f422408df80 secure :-1 s=THROTTLE_DONE pgs=2589 cs=0 l=1 rx=0x7f4220002f30 tx=0x7f422000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:59.474 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:59.452+0000 7f4228811700 1 -- 172.21.15.42:0/4211707892 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4220011370 con 0x7f42240a7740 2019-12-05T06:23:59.474 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:59.452+0000 7f4229813700 1 --2- 172.21.15.42:0/4211707892 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f42240a7740 0x7f422408df80 secure :-1 s=THROTTLE_DONE pgs=2589 cs=0 l=1 rx=0x7f4220002f30 tx=0x7f422000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:59.474 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:59.452+0000 7f4229813700 1 --2- 172.21.15.42:0/4211707892 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f42240a7740 0x7f422408df80 secure :-1 s=THROTTLE_DONE pgs=2589 cs=0 l=1 rx=0x7f4220002f30 tx=0x7f422000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:59.475 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:59.452+0000 7f4228811700 1 -- 172.21.15.42:0/4211707892 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4220008bb0 con 0x7f42240a7740 2019-12-05T06:23:59.475 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:59.452+0000 7f4228811700 1 -- 172.21.15.42:0/4211707892 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4220011950 con 0x7f42240a7740 2019-12-05T06:23:59.475 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:59.452+0000 7f422b276700 1 -- 172.21.15.42:0/4211707892 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f42240a7740 msgr2=0x7f422408df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:59.475 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:59.452+0000 7f422b276700 1 --2- 172.21.15.42:0/4211707892 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f42240a7740 0x7f422408df80 secure :-1 s=READY pgs=2589 cs=0 l=1 rx=0x7f4220002f30 tx=0x7f422000fd30).stop 2019-12-05T06:23:59.475 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:59.452+0000 7f422b276700 1 -- 172.21.15.42:0/4211707892 shutdown_connections 2019-12-05T06:23:59.476 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:59.452+0000 7f422b276700 1 --2- 172.21.15.42:0/4211707892 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f42240a7740 0x7f422408df80 unknown :-1 s=CLOSED pgs=2589 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:59.476 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:59.452+0000 7f422b276700 1 --2- 172.21.15.42:0/4211707892 >> v2:172.21.15.40:3300/0 conn(0x7f422407e720 0x7f422407eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:23:59.476 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:59.452+0000 7f422b276700 1 -- 172.21.15.42:0/4211707892 shutdown_connections 2019-12-05T06:23:59.476 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:59.452+0000 7f422b276700 1 -- 172.21.15.42:0/4211707892 wait complete. 2019-12-05T06:23:59.476 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:59.452+0000 7f422b276700 1 -- 172.21.15.42:0/4211707892 >> 172.21.15.42:0/4211707892 conn(0x7f42240aabe0 msgr2=0x7f42240a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:23:59.476 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:59.452+0000 7f422b276700 1 Processor -- start 2019-12-05T06:23:59.477 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:59.452+0000 7f422b276700 1 -- start start 2019-12-05T06:23:59.477 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:59.456+0000 7f422b276700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4224122ae0 0x7f422411b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:23:59.477 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:59.456+0000 7f422b276700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f4224135660 0x7f4224137aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:23:59.477 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:59.456+0000 7f422b276700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f422411be00 con 0x7f422407dac0 2019-12-05T06:23:59.477 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:59.456+0000 7f422b276700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4224122f10 con 0x7f4224135660 2019-12-05T06:23:59.477 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:59.456+0000 7f422b276700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4224121f70 con 0x7f4224122ae0 2019-12-05T06:23:59.477 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:59.456+0000 7f4229012700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f422407dac0 0x7f4224120ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:54690/0 (socket says 172.21.15.42:54690) 2019-12-05T06:23:59.478 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:59.456+0000 7f4229012700 1 -- 172.21.15.42:0/3133625839 learned_addr learned my addr 172.21.15.42:0/3133625839 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:23:59.478 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:59.456+0000 7f4229813700 1 --2- 172.21.15.42:0/3133625839 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4224122ae0 0x7f422411b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:59.478 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:59.456+0000 7f422a014700 1 --2- 172.21.15.42:0/3133625839 >> v2:172.21.15.40:3300/0 conn(0x7f4224135660 0x7f4224137aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:59.478 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:59.456+0000 7f421a7fc700 1 -- 172.21.15.42:0/3133625839 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2329830349 0 0) 0x7f422411be00 con 0x7f422407dac0 2019-12-05T06:23:59.478 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:59.456+0000 7f421a7fc700 1 -- 172.21.15.42:0/3133625839 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f420c0030b0 con 0x7f422407dac0 2019-12-05T06:23:59.478 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:59.456+0000 7f421a7fc700 1 -- 172.21.15.42:0/3133625839 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 772961469 0 0) 0x7f420c0030b0 con 0x7f422407dac0 2019-12-05T06:23:59.478 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:59.456+0000 7f421a7fc700 1 -- 172.21.15.42:0/3133625839 --> v1:172.21.15.42:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f422411be00 con 0x7f422407dac0 2019-12-05T06:23:59.479 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:59.456+0000 7f4229813700 1 -- 172.21.15.42:0/3133625839 >> v2:172.21.15.40:3300/0 conn(0x7f4224135660 msgr2=0x7f4224137aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:59.479 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:59.456+0000 7f4229813700 1 --2- 172.21.15.42:0/3133625839 >> v2:172.21.15.40:3300/0 conn(0x7f4224135660 0x7f4224137aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:59.479 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:59.456+0000 7f4229813700 1 -- 172.21.15.42:0/3133625839 >> v1:172.21.15.42:6789/0 conn(0x7f422407dac0 legacy=0x7f4224120ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:59.479 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:59.456+0000 7f4229813700 1 -- 172.21.15.42:0/3133625839 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42241216f0 con 0x7f4224122ae0 2019-12-05T06:23:59.479 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:59.456+0000 7f422a014700 1 --2- 172.21.15.42:0/3133625839 >> v2:172.21.15.40:3300/0 conn(0x7f4224135660 0x7f4224137aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:23:59.479 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:59.456+0000 7f4229813700 1 --2- 172.21.15.42:0/3133625839 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4224122ae0 0x7f422411b8b0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f4220002f30 tx=0x7f4214002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:59.479 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:59.456+0000 7f4229813700 1 --2- 172.21.15.42:0/3133625839 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4224122ae0 0x7f422411b8b0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f4220002f30 tx=0x7f4214002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:59.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:59.456+0000 7f4229813700 1 --2- 172.21.15.42:0/3133625839 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4224122ae0 0x7f422411b8b0 secure :-1 s=READY pgs=560 cs=0 l=1 rx=0x7f4220002f30 tx=0x7f4214002ff0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:23:59.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:59.456+0000 7f4229813700 1 --2- 172.21.15.42:0/3133625839 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4224122ae0 0x7f422411b8b0 secure :-1 s=THROTTLE_DONE pgs=560 cs=0 l=1 rx=0x7f4220002f30 tx=0x7f4214002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:59.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:59.456+0000 7f421a7fc700 1 -- 172.21.15.42:0/3133625839 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4220007150 con 0x7f4224122ae0 2019-12-05T06:23:59.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:59.456+0000 7f4229813700 1 --2- 172.21.15.42:0/3133625839 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4224122ae0 0x7f422411b8b0 secure :-1 s=THROTTLE_DONE pgs=560 cs=0 l=1 rx=0x7f4220002f30 tx=0x7f4214002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:59.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:59.456+0000 7f4229813700 1 --2- 172.21.15.42:0/3133625839 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4224122ae0 0x7f422411b8b0 secure :-1 s=THROTTLE_DONE pgs=560 cs=0 l=1 rx=0x7f4220002f30 tx=0x7f4214002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:59.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:59.456+0000 7f421a7fc700 1 -- 172.21.15.42:0/3133625839 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4220011570 con 0x7f4224122ae0 2019-12-05T06:23:59.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:59.456+0000 7f421a7fc700 1 -- 172.21.15.42:0/3133625839 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f422000f550 con 0x7f4224122ae0 2019-12-05T06:23:59.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:59.456+0000 7f422b276700 1 -- 172.21.15.42:0/3133625839 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f422407def0 con 0x7f4224122ae0 2019-12-05T06:23:59.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:59.456+0000 7f422b276700 1 -- 172.21.15.42:0/3133625839 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4224137ff0 con 0x7f4224122ae0 2019-12-05T06:23:59.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:59.456+0000 7f4229813700 1 --2- 172.21.15.42:0/3133625839 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4224122ae0 0x7f422411b8b0 secure :-1 s=THROTTLE_DONE pgs=560 cs=0 l=1 rx=0x7f4220002f30 tx=0x7f4214002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:59.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:59.460+0000 7f421a7fc700 1 -- 172.21.15.42:0/3133625839 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f4220005540 con 0x7f4224122ae0 2019-12-05T06:23:59.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:59.460+0000 7f4229813700 1 --2- 172.21.15.42:0/3133625839 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4224122ae0 0x7f422411b8b0 secure :-1 s=THROTTLE_DONE pgs=560 cs=0 l=1 rx=0x7f4220002f30 tx=0x7f4214002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:59.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:59.460+0000 7f421a7fc700 1 --2- 172.21.15.42:0/3133625839 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f420c03b680 0x7f420c03db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:23:59.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:59.460+0000 7f421a7fc700 1 -- 172.21.15.42:0/3133625839 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 10761+0+0 (secure 0 0 0) 0x7f422008ef20 con 0x7f4224122ae0 2019-12-05T06:23:59.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:59.460+0000 7f4229012700 1 --2- 172.21.15.42:0/3133625839 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f420c03b680 0x7f420c03db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:59.482 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:59.460+0000 7f4229012700 1 --2- 172.21.15.42:0/3133625839 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f420c03b680 0x7f420c03db90 crc :-1 s=READY pgs=1075 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:23:59.482 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:59.464+0000 7f422b276700 1 -- 172.21.15.42:0/3133625839 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f422405d110 con 0x7f4224122ae0 2019-12-05T06:23:59.482 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:59.472+0000 7f4229813700 1 --2- 172.21.15.42:0/3133625839 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4224122ae0 0x7f422411b8b0 secure :-1 s=THROTTLE_DONE pgs=560 cs=0 l=1 rx=0x7f4220002f30 tx=0x7f4214002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:59.483 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:59.472+0000 7f421a7fc700 1 -- 172.21.15.42:0/3133625839 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f4220075f80 con 0x7f4224122ae0 2019-12-05T06:23:59.685 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:59.680+0000 7f422b276700 1 -- 172.21.15.42:0/3133625839 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f422405b8c0 con 0x7f4224122ae0 2019-12-05T06:23:59.782 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:59.680+0000 7f4229813700 1 --2- 172.21.15.42:0/3133625839 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4224122ae0 0x7f422411b8b0 secure :-1 s=THROTTLE_DONE pgs=560 cs=0 l=1 rx=0x7f4220002f30 tx=0x7f4214002ff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:59.782 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:59.680+0000 7f421a7fc700 1 -- 172.21.15.42:0/3133625839 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 308 v308) v1 ==== 95+0+2116 (secure 0 0 0) 0x7f42200761e0 con 0x7f4224122ae0 2019-12-05T06:23:59.783 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 308 2019-12-05T06:23:59.783 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:59.684+0000 7f422b276700 1 -- 172.21.15.42:0/3133625839 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f420c03b680 msgr2=0x7f420c03db90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:59.783 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:59.684+0000 7f422b276700 1 --2- 172.21.15.42:0/3133625839 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f420c03b680 0x7f420c03db90 crc :-1 s=READY pgs=1075 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:59.783 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:59.684+0000 7f422b276700 1 -- 172.21.15.42:0/3133625839 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4224122ae0 msgr2=0x7f422411b8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:59.783 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:59.684+0000 7f422b276700 1 --2- 172.21.15.42:0/3133625839 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4224122ae0 0x7f422411b8b0 secure :-1 s=READY pgs=560 cs=0 l=1 rx=0x7f4220002f30 tx=0x7f4214002ff0).stop 2019-12-05T06:23:59.783 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:59.684+0000 7f422b276700 1 -- 172.21.15.42:0/3133625839 shutdown_connections 2019-12-05T06:23:59.783 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:59.684+0000 7f422b276700 1 --2- 172.21.15.42:0/3133625839 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f420c03b680 0x7f420c03db90 unknown :-1 s=CLOSED pgs=1075 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:59.784 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:59.684+0000 7f422b276700 1 --2- 172.21.15.42:0/3133625839 >> v2:172.21.15.40:3300/0 conn(0x7f4224135660 0x7f4224137aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:59.784 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:59.684+0000 7f422b276700 1 --2- 172.21.15.42:0/3133625839 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4224122ae0 0x7f422411b8b0 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:59.784 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:59.684+0000 7f422b276700 1 -- 172.21.15.42:0/3133625839 shutdown_connections 2019-12-05T06:23:59.784 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:59.684+0000 7f422b276700 1 -- 172.21.15.42:0/3133625839 wait complete. 2019-12-05T06:23:59.784 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:23:59.684+0000 7f422b276700 1 -- 172.21.15.42:0/3133625839 >> 172.21.15.42:0/3133625839 conn(0x7f422407e5d0 msgr2=0x7f422409f380 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:23:59.784 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:23:59.785 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":308,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7568,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/2324657302","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2324657302},{"type":"v1","addr":"172.21.15.40:6833","nonce":2324657302}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":308}],"filesystems":[{"mdsmap":{"epoch":307,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:23:50.178163+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":104,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10993},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10993":{"gid":10993,"name":"a","rank":0,"incarnation":304,"state":"up:active","state_seq":33,"addr":"172.21.15.42:6835/3964426552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3964426552},{"type":"v1","addr":"172.21.15.42:6835","nonce":3964426552}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:23:59.785 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b reported in up:standby state 2019-12-05T06:23:59.785 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 20 secs before thrashing 2019-12-05T06:24:13.385 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:24:13.389 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:24:19.786 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:24:19.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:19.884+0000 7faf66062700 1 Processor -- start 2019-12-05T06:24:19.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:19.884+0000 7faf66062700 1 -- start start 2019-12-05T06:24:19.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:19.884+0000 7faf66062700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7faf600a7740 0x7faf6008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:24:19.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:19.884+0000 7faf66062700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7faf6009fae0 0x7faf6007eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:24:19.895 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:19.884+0000 7faf66062700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf600a0710 con 0x7faf6007e720 2019-12-05T06:24:19.895 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:19.884+0000 7faf66062700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf600a0b00 con 0x7faf6009fae0 2019-12-05T06:24:19.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:19.884+0000 7faf66062700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf6008e5b0 con 0x7faf600a7740 2019-12-05T06:24:19.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:19.884+0000 7faf64e00700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7faf6009fae0 0x7faf6007eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:19.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:19.884+0000 7faf64e00700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7faf6009fae0 0x7faf6007eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:54738/0 (socket says 172.21.15.42:54738) 2019-12-05T06:24:19.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:19.884+0000 7faf64e00700 1 -- 172.21.15.42:0/2562026637 learned_addr learned my addr 172.21.15.42:0/2562026637 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:24:19.897 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:19.884+0000 7faf5ffff700 1 --2- 172.21.15.42:0/2562026637 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7faf600a7740 0x7faf6008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:19.897 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:19.884+0000 7faf64e00700 1 -- 172.21.15.42:0/2562026637 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7faf600a7740 msgr2=0x7faf6008df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:24:19.897 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:19.884+0000 7faf64e00700 1 --2- 172.21.15.42:0/2562026637 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7faf600a7740 0x7faf6008df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:24:19.897 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:19.884+0000 7faf64e00700 1 -- 172.21.15.42:0/2562026637 >> v1:172.21.15.40:6789/0 conn(0x7faf6007e720 legacy=0x7faf6009f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:19.897 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:19.884+0000 7faf64e00700 1 -- 172.21.15.42:0/2562026637 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf6007f0a0 con 0x7faf6009fae0 2019-12-05T06:24:19.898 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:19.884+0000 7faf64e00700 1 --2- 172.21.15.42:0/2562026637 >> v2:172.21.15.42:3300/0 conn(0x7faf6009fae0 0x7faf6007eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7faf58008b10 tx=0x7faf580100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:19.898 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:19.884+0000 7faf64e00700 1 --2- 172.21.15.42:0/2562026637 >> v2:172.21.15.42:3300/0 conn(0x7faf6009fae0 0x7faf6007eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7faf58008b10 tx=0x7faf580100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:19.898 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:19.884+0000 7faf64e00700 1 --2- 172.21.15.42:0/2562026637 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf6009fae0 0x7faf6007eb50 secure :-1 s=READY pgs=2591 cs=0 l=1 rx=0x7faf58008b10 tx=0x7faf580100b0).ready entity=mon.1 client_cookie=ca08c97be5a81192 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:24:19.898 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:19.884+0000 7faf64e00700 1 --2- 172.21.15.42:0/2562026637 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf6009fae0 0x7faf6007eb50 secure :-1 s=THROTTLE_DONE pgs=2591 cs=0 l=1 rx=0x7faf58008b10 tx=0x7faf580100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:19.899 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:19.884+0000 7faf64e00700 1 --2- 172.21.15.42:0/2562026637 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf6009fae0 0x7faf6007eb50 secure :-1 s=THROTTLE_DONE pgs=2591 cs=0 l=1 rx=0x7faf58008b10 tx=0x7faf580100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:19.899 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:19.884+0000 7faf5f7fe700 1 -- 172.21.15.42:0/2562026637 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faf58010da0 con 0x7faf6009fae0 2019-12-05T06:24:19.899 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:19.884+0000 7faf64e00700 1 --2- 172.21.15.42:0/2562026637 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf6009fae0 0x7faf6007eb50 secure :-1 s=THROTTLE_DONE pgs=2591 cs=0 l=1 rx=0x7faf58008b10 tx=0x7faf580100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:19.899 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:19.884+0000 7faf5f7fe700 1 -- 172.21.15.42:0/2562026637 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf580109c0 con 0x7faf6009fae0 2019-12-05T06:24:19.900 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:19.884+0000 7faf5f7fe700 1 -- 172.21.15.42:0/2562026637 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faf580114a0 con 0x7faf6009fae0 2019-12-05T06:24:19.900 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:19.884+0000 7faf66062700 1 -- 172.21.15.42:0/2562026637 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf6009fae0 msgr2=0x7faf6007eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:19.900 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:19.884+0000 7faf66062700 1 --2- 172.21.15.42:0/2562026637 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf6009fae0 0x7faf6007eb50 secure :-1 s=READY pgs=2591 cs=0 l=1 rx=0x7faf58008b10 tx=0x7faf580100b0).stop 2019-12-05T06:24:19.900 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:19.888+0000 7faf66062700 1 -- 172.21.15.42:0/2562026637 shutdown_connections 2019-12-05T06:24:19.901 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:19.888+0000 7faf66062700 1 --2- 172.21.15.42:0/2562026637 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf6009fae0 0x7faf6007eb50 unknown :-1 s=CLOSED pgs=2591 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:19.901 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:19.888+0000 7faf66062700 1 --2- 172.21.15.42:0/2562026637 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7faf600a7740 0x7faf6008df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:24:19.901 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:19.888+0000 7faf66062700 1 -- 172.21.15.42:0/2562026637 shutdown_connections 2019-12-05T06:24:19.901 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:19.888+0000 7faf66062700 1 -- 172.21.15.42:0/2562026637 wait complete. 2019-12-05T06:24:19.902 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:19.888+0000 7faf66062700 1 -- 172.21.15.42:0/2562026637 >> 172.21.15.42:0/2562026637 conn(0x7faf600aabe0 msgr2=0x7faf600a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:24:19.907 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:19.888+0000 7faf66062700 1 Processor -- start 2019-12-05T06:24:19.907 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:19.888+0000 7faf66062700 1 -- start start 2019-12-05T06:24:19.908 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:19.888+0000 7faf66062700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7faf60145080 0x7faf60078a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:24:19.908 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:19.888+0000 7faf66062700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7faf6007a1f0 0x7faf6012e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:24:19.908 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:19.892+0000 7faf66062700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf6007b440 con 0x7faf600a6950 2019-12-05T06:24:19.908 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:19.892+0000 7faf66062700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf6007c800 con 0x7faf6007a1f0 2019-12-05T06:24:19.909 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:19.892+0000 7faf66062700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf6007caa0 con 0x7faf60145080 2019-12-05T06:24:19.909 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:19.892+0000 7faf64e00700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7faf6007a1f0 0x7faf6012e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:19.909 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:19.892+0000 7faf64e00700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7faf6007a1f0 0x7faf6012e2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:54744/0 (socket says 172.21.15.42:54744) 2019-12-05T06:24:19.909 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:19.892+0000 7faf64e00700 1 -- 172.21.15.42:0/2696866820 learned_addr learned my addr 172.21.15.42:0/2696866820 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:24:19.909 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:19.892+0000 7faf5ffff700 1 --2- 172.21.15.42:0/2696866820 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7faf60145080 0x7faf60078a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:19.910 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:19.892+0000 7faf5d7fa700 1 -- 172.21.15.42:0/2696866820 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 241282344 0 0) 0x7faf6007b440 con 0x7faf600a6950 2019-12-05T06:24:19.910 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:19.892+0000 7faf64e00700 1 -- 172.21.15.42:0/2696866820 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7faf60145080 msgr2=0x7faf60078a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:19.910 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:19.892+0000 7faf64e00700 1 --2- 172.21.15.42:0/2696866820 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7faf60145080 0x7faf60078a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:19.910 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:19.892+0000 7faf64e00700 1 -- 172.21.15.42:0/2696866820 >> v1:172.21.15.40:6789/0 conn(0x7faf600a6950 legacy=0x7faf6007ac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:19.911 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:19.892+0000 7faf64e00700 1 -- 172.21.15.42:0/2696866820 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf6007c4c0 con 0x7faf6007a1f0 2019-12-05T06:24:19.911 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:19.892+0000 7faf5ffff700 1 --2- 172.21.15.42:0/2696866820 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7faf60145080 0x7faf60078a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:24:19.911 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:19.892+0000 7faf64e00700 1 --2- 172.21.15.42:0/2696866820 >> v2:172.21.15.42:3300/0 conn(0x7faf6007a1f0 0x7faf6012e2a0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7faf600a7af0 tx=0x7faf58010cf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:19.911 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:19.892+0000 7faf64e00700 1 --2- 172.21.15.42:0/2696866820 >> v2:172.21.15.42:3300/0 conn(0x7faf6007a1f0 0x7faf6012e2a0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7faf600a7af0 tx=0x7faf58010cf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:19.912 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:19.892+0000 7faf64e00700 1 --2- 172.21.15.42:0/2696866820 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf6007a1f0 0x7faf6012e2a0 secure :-1 s=READY pgs=2592 cs=0 l=1 rx=0x7faf600a7af0 tx=0x7faf58010cf0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:24:19.912 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:19.892+0000 7faf64e00700 1 --2- 172.21.15.42:0/2696866820 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf6007a1f0 0x7faf6012e2a0 secure :-1 s=THROTTLE_DONE pgs=2592 cs=0 l=1 rx=0x7faf600a7af0 tx=0x7faf58010cf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:19.912 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:19.892+0000 7faf5d7fa700 1 -- 172.21.15.42:0/2696866820 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faf58011bb0 con 0x7faf6007a1f0 2019-12-05T06:24:19.912 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:19.892+0000 7faf64e00700 1 --2- 172.21.15.42:0/2696866820 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf6007a1f0 0x7faf6012e2a0 secure :-1 s=THROTTLE_DONE pgs=2592 cs=0 l=1 rx=0x7faf600a7af0 tx=0x7faf58010cf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:19.913 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:19.892+0000 7faf64e00700 1 --2- 172.21.15.42:0/2696866820 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf6007a1f0 0x7faf6012e2a0 secure :-1 s=THROTTLE_DONE pgs=2592 cs=0 l=1 rx=0x7faf600a7af0 tx=0x7faf58010cf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:19.913 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:19.892+0000 7faf5d7fa700 1 -- 172.21.15.42:0/2696866820 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf580110c0 con 0x7faf6007a1f0 2019-12-05T06:24:19.913 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:19.892+0000 7faf5d7fa700 1 -- 172.21.15.42:0/2696866820 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faf58005ab0 con 0x7faf6007a1f0 2019-12-05T06:24:19.913 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:19.892+0000 7faf66062700 1 -- 172.21.15.42:0/2696866820 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf6007cd40 con 0x7faf6007a1f0 2019-12-05T06:24:19.914 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:19.892+0000 7faf66062700 1 -- 172.21.15.42:0/2696866820 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf60078f80 con 0x7faf6007a1f0 2019-12-05T06:24:19.914 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:19.892+0000 7faf64e00700 1 --2- 172.21.15.42:0/2696866820 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf6007a1f0 0x7faf6012e2a0 secure :-1 s=THROTTLE_DONE pgs=2592 cs=0 l=1 rx=0x7faf600a7af0 tx=0x7faf58010cf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:19.914 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:19.892+0000 7faf5d7fa700 1 -- 172.21.15.42:0/2696866820 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7faf5800f280 con 0x7faf6007a1f0 2019-12-05T06:24:19.914 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:19.892+0000 7faf64e00700 1 --2- 172.21.15.42:0/2696866820 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf6007a1f0 0x7faf6012e2a0 secure :-1 s=THROTTLE_DONE pgs=2592 cs=0 l=1 rx=0x7faf600a7af0 tx=0x7faf58010cf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:19.915 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:19.896+0000 7faf5d7fa700 1 --2- 172.21.15.42:0/2696866820 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7faf4403ad10 0x7faf4403d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:24:19.915 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:19.896+0000 7faf5d7fa700 1 -- 172.21.15.42:0/2696866820 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 10761+0+0 (secure 0 0 0) 0x7faf5804b520 con 0x7faf6007a1f0 2019-12-05T06:24:19.915 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:19.896+0000 7faf5ffff700 1 --2- 172.21.15.42:0/2696866820 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7faf4403ad10 0x7faf4403d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:19.915 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:19.896+0000 7faf5ffff700 1 --2- 172.21.15.42:0/2696866820 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7faf4403ad10 0x7faf4403d220 crc :-1 s=READY pgs=1076 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:24:19.915 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:19.900+0000 7faf66062700 1 -- 172.21.15.42:0/2696866820 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf6005d110 con 0x7faf6007a1f0 2019-12-05T06:24:19.916 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:19.908+0000 7faf64e00700 1 --2- 172.21.15.42:0/2696866820 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf6007a1f0 0x7faf6012e2a0 secure :-1 s=THROTTLE_DONE pgs=2592 cs=0 l=1 rx=0x7faf600a7af0 tx=0x7faf58010cf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:19.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:19.908+0000 7faf5d7fa700 1 -- 172.21.15.42:0/2696866820 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7faf58005d80 con 0x7faf6007a1f0 2019-12-05T06:24:20.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:20.092+0000 7faf66062700 1 -- 172.21.15.42:0/2696866820 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faf6005b8c0 con 0x7faf6007a1f0 2019-12-05T06:24:20.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:20.092+0000 7faf64e00700 1 --2- 172.21.15.42:0/2696866820 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf6007a1f0 0x7faf6012e2a0 secure :-1 s=THROTTLE_DONE pgs=2592 cs=0 l=1 rx=0x7faf600a7af0 tx=0x7faf58010cf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:20.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:20.092+0000 7faf5d7fa700 1 -- 172.21.15.42:0/2696866820 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 308 v308) v1 ==== 95+0+2116 (secure 0 0 0) 0x7faf580333b0 con 0x7faf6007a1f0 2019-12-05T06:24:20.099 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 308 2019-12-05T06:24:20.100 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:24:20.100 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":308,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7568,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/2324657302","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2324657302},{"type":"v1","addr":"172.21.15.40:6833","nonce":2324657302}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":308}],"filesystems":[{"mdsmap":{"epoch":307,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:23:50.178163+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":104,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10993},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10993":{"gid":10993,"name":"a","rank":0,"incarnation":304,"state":"up:active","state_seq":33,"addr":"172.21.15.42:6835/3964426552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3964426552},{"type":"v1","addr":"172.21.15.42:6835","nonce":3964426552}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:24:20.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:20.096+0000 7faf66062700 1 -- 172.21.15.42:0/2696866820 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7faf4403ad10 msgr2=0x7faf4403d220 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:20.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:20.096+0000 7faf66062700 1 --2- 172.21.15.42:0/2696866820 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7faf4403ad10 0x7faf4403d220 crc :-1 s=READY pgs=1076 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:20.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:20.096+0000 7faf66062700 1 -- 172.21.15.42:0/2696866820 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf6007a1f0 msgr2=0x7faf6012e2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:20.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:20.096+0000 7faf66062700 1 --2- 172.21.15.42:0/2696866820 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf6007a1f0 0x7faf6012e2a0 secure :-1 s=READY pgs=2592 cs=0 l=1 rx=0x7faf600a7af0 tx=0x7faf58010cf0).stop 2019-12-05T06:24:20.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:20.096+0000 7faf66062700 1 -- 172.21.15.42:0/2696866820 shutdown_connections 2019-12-05T06:24:20.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:20.096+0000 7faf66062700 1 --2- 172.21.15.42:0/2696866820 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7faf4403ad10 0x7faf4403d220 unknown :-1 s=CLOSED pgs=1076 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:20.103 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:20.096+0000 7faf66062700 1 --2- 172.21.15.42:0/2696866820 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf6007a1f0 0x7faf6012e2a0 unknown :-1 s=CLOSED pgs=2592 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:20.103 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:20.096+0000 7faf66062700 1 --2- 172.21.15.42:0/2696866820 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7faf60145080 0x7faf60078a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:20.104 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:20.096+0000 7faf66062700 1 -- 172.21.15.42:0/2696866820 shutdown_connections 2019-12-05T06:24:20.104 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:20.096+0000 7faf66062700 1 -- 172.21.15.42:0/2696866820 wait complete. 2019-12-05T06:24:20.104 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:20.096+0000 7faf66062700 1 -- 172.21.15.42:0/2696866820 >> 172.21.15.42:0/2696866820 conn(0x7faf600a4a50 msgr2=0x7faf60059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:24:20.112 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.a (rank=0) 2019-12-05T06:24:20.113 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2019-12-05T06:24:20.113 INFO:teuthology.orchestra.run:waiting for 300 2019-12-05T06:24:20.126 INFO:tasks.ceph.mds.a:Stopped 2019-12-05T06:24:20.127 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:24:20.300 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:20.288+0000 7f93d77f7700 1 Processor -- start 2019-12-05T06:24:20.301 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:20.288+0000 7f93d77f7700 1 -- start start 2019-12-05T06:24:20.301 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:20.288+0000 7f93d77f7700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f93d00a7730 0x7f93d008df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:24:20.302 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:20.288+0000 7f93d77f7700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f93d007e710 0x7f93d007eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:24:20.302 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:20.288+0000 7f93d77f7700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93d00a0700 con 0x7f93d009f2d0 2019-12-05T06:24:20.302 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:20.288+0000 7f93d77f7700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93d00a0af0 con 0x7f93d00a7730 2019-12-05T06:24:20.302 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:20.288+0000 7f93d77f7700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93d008e5a0 con 0x7f93d007e710 2019-12-05T06:24:20.303 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:20.288+0000 7f93d5593700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f93d007e710 0x7f93d007eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:20.303 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:20.288+0000 7f93d5593700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f93d007e710 0x7f93d007eb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:54750/0 (socket says 172.21.15.42:54750) 2019-12-05T06:24:20.303 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:20.288+0000 7f93d5593700 1 -- 172.21.15.42:0/4101205920 learned_addr learned my addr 172.21.15.42:0/4101205920 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:24:20.303 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:20.292+0000 7f93d5d94700 1 --2- 172.21.15.42:0/4101205920 >> v2:172.21.15.40:3300/0 conn(0x7f93d00a7730 0x7f93d008df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:20.304 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:20.292+0000 7f93d5593700 1 -- 172.21.15.42:0/4101205920 >> v2:172.21.15.40:3300/0 conn(0x7f93d00a7730 msgr2=0x7f93d008df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:24:20.304 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:20.292+0000 7f93d5593700 1 --2- 172.21.15.42:0/4101205920 >> v2:172.21.15.40:3300/0 conn(0x7f93d00a7730 0x7f93d008df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:24:20.304 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:20.292+0000 7f93d5593700 1 -- 172.21.15.42:0/4101205920 >> v1:172.21.15.40:6789/0 conn(0x7f93d009f2d0 legacy=0x7f93d009f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:20.304 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:20.292+0000 7f93d5593700 1 -- 172.21.15.42:0/4101205920 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93d007f090 con 0x7f93d007e710 2019-12-05T06:24:20.305 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:20.292+0000 7f93d5593700 1 --2- 172.21.15.42:0/4101205920 >> v2:172.21.15.42:3300/0 conn(0x7f93d007e710 0x7f93d007eb40 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f93c0002f30 tx=0x7f93c000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:20.305 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:20.292+0000 7f93d5593700 1 --2- 172.21.15.42:0/4101205920 >> v2:172.21.15.42:3300/0 conn(0x7f93d007e710 0x7f93d007eb40 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f93c0002f30 tx=0x7f93c000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:20.305 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:20.292+0000 7f93d5593700 1 --2- 172.21.15.42:0/4101205920 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f93d007e710 0x7f93d007eb40 secure :-1 s=READY pgs=2593 cs=0 l=1 rx=0x7f93c0002f30 tx=0x7f93c000fd30).ready entity=mon.1 client_cookie=f9a55a9e472f1f87 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:24:20.305 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:20.292+0000 7f93d5593700 1 --2- 172.21.15.42:0/4101205920 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f93d007e710 0x7f93d007eb40 secure :-1 s=THROTTLE_DONE pgs=2593 cs=0 l=1 rx=0x7f93c0002f30 tx=0x7f93c000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:20.306 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:20.292+0000 7f93d4d92700 1 -- 172.21.15.42:0/4101205920 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f93c0011370 con 0x7f93d007e710 2019-12-05T06:24:20.306 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:20.292+0000 7f93d5593700 1 --2- 172.21.15.42:0/4101205920 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f93d007e710 0x7f93d007eb40 secure :-1 s=THROTTLE_DONE pgs=2593 cs=0 l=1 rx=0x7f93c0002f30 tx=0x7f93c000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:20.306 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:20.292+0000 7f93d4d92700 1 -- 172.21.15.42:0/4101205920 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93c0008bb0 con 0x7f93d007e710 2019-12-05T06:24:20.306 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:20.292+0000 7f93d5593700 1 --2- 172.21.15.42:0/4101205920 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f93d007e710 0x7f93d007eb40 secure :-1 s=THROTTLE_DONE pgs=2593 cs=0 l=1 rx=0x7f93c0002f30 tx=0x7f93c000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:20.307 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:20.292+0000 7f93d4d92700 1 -- 172.21.15.42:0/4101205920 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f93c0011950 con 0x7f93d007e710 2019-12-05T06:24:20.307 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:20.292+0000 7f93d77f7700 1 -- 172.21.15.42:0/4101205920 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f93d007e710 msgr2=0x7f93d007eb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:20.308 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:20.292+0000 7f93d77f7700 1 --2- 172.21.15.42:0/4101205920 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f93d007e710 0x7f93d007eb40 secure :-1 s=READY pgs=2593 cs=0 l=1 rx=0x7f93c0002f30 tx=0x7f93c000fd30).stop 2019-12-05T06:24:20.308 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:20.292+0000 7f93d77f7700 1 -- 172.21.15.42:0/4101205920 shutdown_connections 2019-12-05T06:24:20.308 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:20.292+0000 7f93d77f7700 1 --2- 172.21.15.42:0/4101205920 >> v2:172.21.15.40:3300/0 conn(0x7f93d00a7730 0x7f93d008df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:24:20.308 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:20.292+0000 7f93d77f7700 1 --2- 172.21.15.42:0/4101205920 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f93d007e710 0x7f93d007eb40 unknown :-1 s=CLOSED pgs=2593 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:20.309 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:20.292+0000 7f93d77f7700 1 -- 172.21.15.42:0/4101205920 shutdown_connections 2019-12-05T06:24:20.309 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:20.292+0000 7f93d77f7700 1 -- 172.21.15.42:0/4101205920 wait complete. 2019-12-05T06:24:20.309 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:20.292+0000 7f93d77f7700 1 -- 172.21.15.42:0/4101205920 >> 172.21.15.42:0/4101205920 conn(0x7f93d00aabd0 msgr2=0x7f93d00a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:24:20.315 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:20.296+0000 7f93d77f7700 1 Processor -- start 2019-12-05T06:24:20.315 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:20.296+0000 7f93d77f7700 1 -- start start 2019-12-05T06:24:20.315 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:20.296+0000 7f93d77f7700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f93d00a5240 0x7f93d0118440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:24:20.316 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:20.296+0000 7f93d77f7700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f93d0131500 0x7f93d00a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:24:20.316 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:20.296+0000 7f93d77f7700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93d0118990 con 0x7f93d00a6640 2019-12-05T06:24:20.316 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:20.296+0000 7f93d77f7700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93d00a4a60 con 0x7f93d00a5240 2019-12-05T06:24:20.317 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:20.296+0000 7f93d77f7700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93d00a4d00 con 0x7f93d0131500 2019-12-05T06:24:20.317 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:20.296+0000 7f93d5593700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f93d0131500 0x7f93d00a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:20.318 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:20.296+0000 7f93d5593700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f93d0131500 0x7f93d00a60f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:54758/0 (socket says 172.21.15.42:54758) 2019-12-05T06:24:20.318 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:20.296+0000 7f93d5593700 1 -- 172.21.15.42:0/984690349 learned_addr learned my addr 172.21.15.42:0/984690349 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:24:20.318 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:20.296+0000 7f93d5d94700 1 --2- 172.21.15.42:0/984690349 >> v2:172.21.15.40:3300/0 conn(0x7f93d00a5240 0x7f93d0118440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:20.318 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:20.300+0000 7f93d5593700 1 -- 172.21.15.42:0/984690349 >> v2:172.21.15.40:3300/0 conn(0x7f93d00a5240 msgr2=0x7f93d0118440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:20.318 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:20.300+0000 7f93d5593700 1 --2- 172.21.15.42:0/984690349 >> v2:172.21.15.40:3300/0 conn(0x7f93d00a5240 0x7f93d0118440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:20.319 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:20.300+0000 7f93d5593700 1 -- 172.21.15.42:0/984690349 >> v1:172.21.15.40:6789/0 conn(0x7f93d00a6640 legacy=0x7f93d008eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:20.319 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:20.300+0000 7f93d5593700 1 -- 172.21.15.42:0/984690349 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93d007b510 con 0x7f93d0131500 2019-12-05T06:24:20.319 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:20.300+0000 7f93d5593700 1 --2- 172.21.15.42:0/984690349 >> v2:172.21.15.42:3300/0 conn(0x7f93d0131500 0x7f93d00a60f0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f93c0002f30 tx=0x7f93c0011ee0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:20.319 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:20.300+0000 7f93d5593700 1 --2- 172.21.15.42:0/984690349 >> v2:172.21.15.42:3300/0 conn(0x7f93d0131500 0x7f93d00a60f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f93c0002f30 tx=0x7f93c0011ee0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:20.320 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:20.300+0000 7f93d5593700 1 --2- 172.21.15.42:0/984690349 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f93d0131500 0x7f93d00a60f0 secure :-1 s=READY pgs=2594 cs=0 l=1 rx=0x7f93c0002f30 tx=0x7f93c0011ee0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:24:20.320 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:20.300+0000 7f93d5593700 1 --2- 172.21.15.42:0/984690349 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f93d0131500 0x7f93d00a60f0 secure :-1 s=THROTTLE_DONE pgs=2594 cs=0 l=1 rx=0x7f93c0002f30 tx=0x7f93c0011ee0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:20.320 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:20.300+0000 7f93d5593700 1 --2- 172.21.15.42:0/984690349 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f93d0131500 0x7f93d00a60f0 secure :-1 s=THROTTLE_DONE pgs=2594 cs=0 l=1 rx=0x7f93c0002f30 tx=0x7f93c0011ee0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:20.320 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:20.300+0000 7f93d5593700 1 --2- 172.21.15.42:0/984690349 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f93d0131500 0x7f93d00a60f0 secure :-1 s=THROTTLE_DONE pgs=2594 cs=0 l=1 rx=0x7f93c0002f30 tx=0x7f93c0011ee0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:20.320 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:20.300+0000 7f93c67fc700 1 -- 172.21.15.42:0/984690349 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f93c0007340 con 0x7f93d0131500 2019-12-05T06:24:20.321 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:20.300+0000 7f93c67fc700 1 -- 172.21.15.42:0/984690349 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93c0011570 con 0x7f93d0131500 2019-12-05T06:24:20.321 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:20.300+0000 7f93d77f7700 1 -- 172.21.15.42:0/984690349 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93d0131930 con 0x7f93d0131500 2019-12-05T06:24:20.321 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:20.300+0000 7f93c67fc700 1 -- 172.21.15.42:0/984690349 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f93c0005540 con 0x7f93d0131500 2019-12-05T06:24:20.321 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:20.300+0000 7f93d77f7700 1 -- 172.21.15.42:0/984690349 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93d008f340 con 0x7f93d0131500 2019-12-05T06:24:20.325 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:20.300+0000 7f93d5593700 1 --2- 172.21.15.42:0/984690349 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f93d0131500 0x7f93d00a60f0 secure :-1 s=THROTTLE_DONE pgs=2594 cs=0 l=1 rx=0x7f93c0002f30 tx=0x7f93c0011ee0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:20.325 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:20.300+0000 7f93c67fc700 1 -- 172.21.15.42:0/984690349 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f93c0005810 con 0x7f93d0131500 2019-12-05T06:24:20.325 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:20.300+0000 7f93d5593700 1 --2- 172.21.15.42:0/984690349 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f93d0131500 0x7f93d00a60f0 secure :-1 s=THROTTLE_DONE pgs=2594 cs=0 l=1 rx=0x7f93c0002f30 tx=0x7f93c0011ee0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:20.325 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:20.300+0000 7f93c67fc700 1 --2- 172.21.15.42:0/984690349 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f93b803adb0 0x7f93b803d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:24:20.326 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:20.300+0000 7f93c67fc700 1 -- 172.21.15.42:0/984690349 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 10761+0+0 (secure 0 0 0) 0x7f93c008f3d0 con 0x7f93d0131500 2019-12-05T06:24:20.326 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:20.300+0000 7f93d5d94700 1 --2- 172.21.15.42:0/984690349 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f93b803adb0 0x7f93b803d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:20.326 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:20.300+0000 7f93d5d94700 1 --2- 172.21.15.42:0/984690349 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f93b803adb0 0x7f93b803d2c0 crc :-1 s=READY pgs=1077 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:24:20.326 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:20.304+0000 7f93d77f7700 1 -- 172.21.15.42:0/984690349 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93d008f9b0 con 0x7f93d0131500 2019-12-05T06:24:20.327 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:20.312+0000 7f93d5593700 1 --2- 172.21.15.42:0/984690349 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f93d0131500 0x7f93d00a60f0 secure :-1 s=THROTTLE_DONE pgs=2594 cs=0 l=1 rx=0x7f93c0002f30 tx=0x7f93c0011ee0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:20.328 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:20.312+0000 7f93c67fc700 1 -- 172.21.15.42:0/984690349 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f93c0076450 con 0x7f93d0131500 2019-12-05T06:24:20.432 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~0s 2019-12-05T06:24:20.488 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:20.480+0000 7f93d77f7700 1 -- 172.21.15.42:0/984690349 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f93d005d100 con 0x7f93d0131500 2019-12-05T06:24:20.489 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:20.484+0000 7f93d5593700 1 --2- 172.21.15.42:0/984690349 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f93d0131500 0x7f93d00a60f0 secure :-1 s=THROTTLE_DONE pgs=2594 cs=0 l=1 rx=0x7f93c0002f30 tx=0x7f93c0011ee0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:20.489 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:20.484+0000 7f93c67fc700 1 -- 172.21.15.42:0/984690349 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 308 v308) v1 ==== 95+0+2116 (secure 0 0 0) 0x7f93c00766b0 con 0x7f93d0131500 2019-12-05T06:24:20.491 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 308 2019-12-05T06:24:20.491 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:24:20.492 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":308,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7568,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/2324657302","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2324657302},{"type":"v1","addr":"172.21.15.40:6833","nonce":2324657302}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":308}],"filesystems":[{"mdsmap":{"epoch":307,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:23:50.178163+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":104,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10993},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10993":{"gid":10993,"name":"a","rank":0,"incarnation":304,"state":"up:active","state_seq":33,"addr":"172.21.15.42:6835/3964426552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3964426552},{"type":"v1","addr":"172.21.15.42:6835","nonce":3964426552}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:24:20.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:20.488+0000 7f93d77f7700 1 -- 172.21.15.42:0/984690349 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f93b803adb0 msgr2=0x7f93b803d2c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:20.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:20.488+0000 7f93d77f7700 1 --2- 172.21.15.42:0/984690349 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f93b803adb0 0x7f93b803d2c0 crc :-1 s=READY pgs=1077 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:20.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:20.488+0000 7f93d77f7700 1 -- 172.21.15.42:0/984690349 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f93d0131500 msgr2=0x7f93d00a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:20.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:20.488+0000 7f93d77f7700 1 --2- 172.21.15.42:0/984690349 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f93d0131500 0x7f93d00a60f0 secure :-1 s=READY pgs=2594 cs=0 l=1 rx=0x7f93c0002f30 tx=0x7f93c0011ee0).stop 2019-12-05T06:24:20.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:20.488+0000 7f93d77f7700 1 -- 172.21.15.42:0/984690349 shutdown_connections 2019-12-05T06:24:20.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:20.488+0000 7f93d77f7700 1 --2- 172.21.15.42:0/984690349 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f93b803adb0 0x7f93b803d2c0 unknown :-1 s=CLOSED pgs=1077 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:20.496 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:20.488+0000 7f93d77f7700 1 --2- 172.21.15.42:0/984690349 >> v2:172.21.15.40:3300/0 conn(0x7f93d00a5240 0x7f93d0118440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:20.496 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:20.488+0000 7f93d77f7700 1 --2- 172.21.15.42:0/984690349 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f93d0131500 0x7f93d00a60f0 unknown :-1 s=CLOSED pgs=2594 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:20.496 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:20.488+0000 7f93d77f7700 1 -- 172.21.15.42:0/984690349 shutdown_connections 2019-12-05T06:24:20.496 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:20.488+0000 7f93d77f7700 1 -- 172.21.15.42:0/984690349 wait complete. 2019-12-05T06:24:20.497 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:20.488+0000 7f93d77f7700 1 -- 172.21.15.42:0/984690349 >> 172.21.15.42:0/984690349 conn(0x7f93d00a7f80 msgr2=0x7f93d00596f0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:24:20.505 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:24:22.506 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:24:22.618 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:22.604+0000 7f0d11351700 1 Processor -- start 2019-12-05T06:24:22.618 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:22.604+0000 7f0d11351700 1 -- start start 2019-12-05T06:24:22.619 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:22.604+0000 7f0d11351700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0d0c0a7740 0x7f0d0c08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:24:22.619 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:22.604+0000 7f0d11351700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f0d0c07e720 0x7f0d0c07eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:24:22.619 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:22.604+0000 7f0d11351700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0d0c09f2e0 0x7f0d0c09f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:24:22.619 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:22.604+0000 7f0d11351700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d0c0a0b00 con 0x7f0d0c07e720 2019-12-05T06:24:22.619 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:22.604+0000 7f0d11351700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d0c08e5b0 con 0x7f0d0c0a7740 2019-12-05T06:24:22.619 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:22.604+0000 7f0d11351700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d0c08e850 con 0x7f0d0c09f2e0 2019-12-05T06:24:22.619 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:22.604+0000 7f0d0b7fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0d0c0a7740 0x7f0d0c08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:22.620 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:22.604+0000 7f0d0b7fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0d0c0a7740 0x7f0d0c08df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:54772/0 (socket says 172.21.15.42:54772) 2019-12-05T06:24:22.620 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:22.604+0000 7f0d0b7fe700 1 -- 172.21.15.42:0/3217967762 learned_addr learned my addr 172.21.15.42:0/3217967762 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:24:22.620 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:22.604+0000 7f0d0affd700 1 --2- 172.21.15.42:0/3217967762 >> v2:172.21.15.40:3300/0 conn(0x7f0d0c07e720 0x7f0d0c07eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:22.620 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:22.604+0000 7f0d0bfff700 1 --2- 172.21.15.42:0/3217967762 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0d0c09f2e0 0x7f0d0c09f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:22.620 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:22.604+0000 7f0d0affd700 1 -- 172.21.15.42:0/3217967762 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0d0c09f2e0 msgr2=0x7f0d0c09f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:24:22.620 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:22.604+0000 7f0d0affd700 1 --2- 172.21.15.42:0/3217967762 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0d0c09f2e0 0x7f0d0c09f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:24:22.621 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:22.604+0000 7f0d0affd700 1 -- 172.21.15.42:0/3217967762 >> v2:172.21.15.42:3300/0 conn(0x7f0d0c0a7740 msgr2=0x7f0d0c08df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:24:22.621 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:22.604+0000 7f0d0affd700 1 --2- 172.21.15.42:0/3217967762 >> v2:172.21.15.42:3300/0 conn(0x7f0d0c0a7740 0x7f0d0c08df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:24:22.621 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:22.604+0000 7f0d0affd700 1 -- 172.21.15.42:0/3217967762 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d0c07f0a0 con 0x7f0d0c07e720 2019-12-05T06:24:22.621 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:22.604+0000 7f0d0b7fe700 1 --2- 172.21.15.42:0/3217967762 >> v2:172.21.15.42:3300/0 conn(0x7f0d0c0a7740 0x7f0d0c08df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:24:22.621 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:22.604+0000 7f0d0affd700 1 --2- 172.21.15.42:0/3217967762 >> v2:172.21.15.40:3300/0 conn(0x7f0d0c07e720 0x7f0d0c07eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f0d0c0a0f50 tx=0x7f0cf400fcf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:22.621 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:22.604+0000 7f0d0affd700 1 --2- 172.21.15.42:0/3217967762 >> v2:172.21.15.40:3300/0 conn(0x7f0d0c07e720 0x7f0d0c07eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f0d0c0a0f50 tx=0x7f0cf400fcf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:22.622 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:22.604+0000 7f0d0affd700 1 --2- 172.21.15.42:0/3217967762 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0d0c07e720 0x7f0d0c07eb50 secure :-1 s=READY pgs=1940 cs=0 l=1 rx=0x7f0d0c0a0f50 tx=0x7f0cf400fcf0).ready entity=mon.0 client_cookie=1f57ae04ed653e3f server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:24:22.622 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:22.604+0000 7f0d0affd700 1 --2- 172.21.15.42:0/3217967762 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0d0c07e720 0x7f0d0c07eb50 secure :-1 s=THROTTLE_DONE pgs=1940 cs=0 l=1 rx=0x7f0d0c0a0f50 tx=0x7f0cf400fcf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:22.622 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:22.604+0000 7f0d0a7fc700 1 -- 172.21.15.42:0/3217967762 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0cf4008bb0 con 0x7f0d0c07e720 2019-12-05T06:24:22.622 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:22.604+0000 7f0d0affd700 1 --2- 172.21.15.42:0/3217967762 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0d0c07e720 0x7f0d0c07eb50 secure :-1 s=THROTTLE_DONE pgs=1940 cs=0 l=1 rx=0x7f0d0c0a0f50 tx=0x7f0cf400fcf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:22.622 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:22.604+0000 7f0d0affd700 1 --2- 172.21.15.42:0/3217967762 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0d0c07e720 0x7f0d0c07eb50 secure :-1 s=THROTTLE_DONE pgs=1940 cs=0 l=1 rx=0x7f0d0c0a0f50 tx=0x7f0cf400fcf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:22.622 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:22.604+0000 7f0d0a7fc700 1 -- 172.21.15.42:0/3217967762 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cf40087d0 con 0x7f0d0c07e720 2019-12-05T06:24:22.622 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:22.604+0000 7f0d0a7fc700 1 -- 172.21.15.42:0/3217967762 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0cf40118a0 con 0x7f0d0c07e720 2019-12-05T06:24:22.623 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:22.604+0000 7f0d11351700 1 -- 172.21.15.42:0/3217967762 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0d0c07e720 msgr2=0x7f0d0c07eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:22.623 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:22.604+0000 7f0d11351700 1 --2- 172.21.15.42:0/3217967762 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0d0c07e720 0x7f0d0c07eb50 secure :-1 s=READY pgs=1940 cs=0 l=1 rx=0x7f0d0c0a0f50 tx=0x7f0cf400fcf0).stop 2019-12-05T06:24:22.623 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:22.608+0000 7f0d11351700 1 -- 172.21.15.42:0/3217967762 shutdown_connections 2019-12-05T06:24:22.623 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:22.608+0000 7f0d11351700 1 --2- 172.21.15.42:0/3217967762 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0d0c09f2e0 0x7f0d0c09f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:24:22.623 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:22.608+0000 7f0d11351700 1 --2- 172.21.15.42:0/3217967762 >> v2:172.21.15.42:3300/0 conn(0x7f0d0c0a7740 0x7f0d0c08df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:24:22.623 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:22.608+0000 7f0d11351700 1 --2- 172.21.15.42:0/3217967762 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0d0c07e720 0x7f0d0c07eb50 unknown :-1 s=CLOSED pgs=1940 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:22.623 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:22.608+0000 7f0d11351700 1 -- 172.21.15.42:0/3217967762 shutdown_connections 2019-12-05T06:24:22.623 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:22.608+0000 7f0d11351700 1 -- 172.21.15.42:0/3217967762 wait complete. 2019-12-05T06:24:22.624 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:22.608+0000 7f0d11351700 1 -- 172.21.15.42:0/3217967762 >> 172.21.15.42:0/3217967762 conn(0x7f0d0c0aabe0 msgr2=0x7f0d0c0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:24:22.629 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:22.608+0000 7f0d11351700 1 Processor -- start 2019-12-05T06:24:22.630 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:22.608+0000 7f0d11351700 1 -- start start 2019-12-05T06:24:22.630 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:22.608+0000 7f0d11351700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0d0c116500 0x7f0d0c07c630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:24:22.630 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:22.608+0000 7f0d11351700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0d0c07aca0 0x7f0d0c07b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:24:22.631 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:22.608+0000 7f0d11351700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f0d0c11b570 0x7f0d0c11b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:24:22.631 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:22.608+0000 7f0d11351700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d0c07d170 con 0x7f0d0c11b570 2019-12-05T06:24:22.631 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:22.608+0000 7f0d11351700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d0c07d410 con 0x7f0d0c07aca0 2019-12-05T06:24:22.631 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:22.608+0000 7f0d11351700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d0c07d6b0 con 0x7f0d0c116500 2019-12-05T06:24:22.631 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:22.608+0000 7f0d0affd700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0d0c07aca0 0x7f0d0c07b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:22.632 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:22.608+0000 7f0d0affd700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0d0c07aca0 0x7f0d0c07b0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:54776/0 (socket says 172.21.15.42:54776) 2019-12-05T06:24:22.632 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:22.608+0000 7f0d0affd700 1 -- 172.21.15.42:0/660526160 learned_addr learned my addr 172.21.15.42:0/660526160 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:24:22.632 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:22.608+0000 7f0d0b7fe700 1 --2- 172.21.15.42:0/660526160 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0d0c116500 0x7f0d0c07c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:22.632 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:22.608+0000 7f0d0bfff700 1 --2- 172.21.15.42:0/660526160 >> v2:172.21.15.40:3300/0 conn(0x7f0d0c11b570 0x7f0d0c11b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:22.632 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:22.612+0000 7f0d0b7fe700 1 -- 172.21.15.42:0/660526160 >> v2:172.21.15.42:3300/0 conn(0x7f0d0c07aca0 msgr2=0x7f0d0c07b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:22.632 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:22.612+0000 7f0d0b7fe700 1 --2- 172.21.15.42:0/660526160 >> v2:172.21.15.42:3300/0 conn(0x7f0d0c07aca0 0x7f0d0c07b0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:22.633 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:22.612+0000 7f0d0b7fe700 1 -- 172.21.15.42:0/660526160 >> v2:172.21.15.40:3300/0 conn(0x7f0d0c11b570 msgr2=0x7f0d0c11b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:22.633 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:22.612+0000 7f0d0b7fe700 1 --2- 172.21.15.42:0/660526160 >> v2:172.21.15.40:3300/0 conn(0x7f0d0c11b570 0x7f0d0c11b9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:22.633 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:22.612+0000 7f0d0b7fe700 1 -- 172.21.15.42:0/660526160 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d0c07b940 con 0x7f0d0c116500 2019-12-05T06:24:22.633 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:22.612+0000 7f0d0affd700 1 --2- 172.21.15.42:0/660526160 >> v2:172.21.15.42:3300/0 conn(0x7f0d0c07aca0 0x7f0d0c07b0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:24:22.634 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:22.612+0000 7f0d0b7fe700 1 --2- 172.21.15.42:0/660526160 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0d0c116500 0x7f0d0c07c630 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f0d0c07d330 tx=0x7f0cfc010ce0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:22.634 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:22.612+0000 7f0d0b7fe700 1 --2- 172.21.15.42:0/660526160 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0d0c116500 0x7f0d0c07c630 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f0d0c07d330 tx=0x7f0cfc010ce0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:22.634 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:22.612+0000 7f0d0b7fe700 1 --2- 172.21.15.42:0/660526160 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0d0c116500 0x7f0d0c07c630 secure :-1 s=READY pgs=561 cs=0 l=1 rx=0x7f0d0c07d330 tx=0x7f0cfc010ce0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:24:22.634 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:22.612+0000 7f0d0b7fe700 1 --2- 172.21.15.42:0/660526160 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0d0c116500 0x7f0d0c07c630 secure :-1 s=THROTTLE_DONE pgs=561 cs=0 l=1 rx=0x7f0d0c07d330 tx=0x7f0cfc010ce0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:22.635 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:22.612+0000 7f0d03fff700 1 -- 172.21.15.42:0/660526160 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0cfc0088b0 con 0x7f0d0c116500 2019-12-05T06:24:22.635 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:22.612+0000 7f0d0b7fe700 1 --2- 172.21.15.42:0/660526160 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0d0c116500 0x7f0d0c07c630 secure :-1 s=THROTTLE_DONE pgs=561 cs=0 l=1 rx=0x7f0d0c07d330 tx=0x7f0cfc010ce0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:22.635 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:22.612+0000 7f0d0b7fe700 1 --2- 172.21.15.42:0/660526160 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0d0c116500 0x7f0d0c07c630 secure :-1 s=THROTTLE_DONE pgs=561 cs=0 l=1 rx=0x7f0d0c07d330 tx=0x7f0cfc010ce0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:22.635 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:22.612+0000 7f0d03fff700 1 -- 172.21.15.42:0/660526160 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cfc007160 con 0x7f0d0c116500 2019-12-05T06:24:22.635 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:22.612+0000 7f0d03fff700 1 -- 172.21.15.42:0/660526160 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0cfc005840 con 0x7f0d0c116500 2019-12-05T06:24:22.636 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:22.612+0000 7f0d11351700 1 -- 172.21.15.42:0/660526160 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d0c07cd20 con 0x7f0d0c116500 2019-12-05T06:24:22.636 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:22.612+0000 7f0d11351700 1 -- 172.21.15.42:0/660526160 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d0c08eb70 con 0x7f0d0c116500 2019-12-05T06:24:22.636 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:22.612+0000 7f0d0b7fe700 1 --2- 172.21.15.42:0/660526160 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0d0c116500 0x7f0d0c07c630 secure :-1 s=THROTTLE_DONE pgs=561 cs=0 l=1 rx=0x7f0d0c07d330 tx=0x7f0cfc010ce0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:22.636 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:22.612+0000 7f0d03fff700 1 -- 172.21.15.42:0/660526160 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f0cfc005b50 con 0x7f0d0c116500 2019-12-05T06:24:22.636 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:22.612+0000 7f0d0b7fe700 1 --2- 172.21.15.42:0/660526160 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0d0c116500 0x7f0d0c07c630 secure :-1 s=THROTTLE_DONE pgs=561 cs=0 l=1 rx=0x7f0d0c07d330 tx=0x7f0cfc010ce0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:22.636 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:22.612+0000 7f0d03fff700 1 --2- 172.21.15.42:0/660526160 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0cf003ae90 0x7f0cf003d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:24:22.637 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:22.612+0000 7f0d03fff700 1 -- 172.21.15.42:0/660526160 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 10761+0+0 (secure 0 0 0) 0x7f0cfc04b370 con 0x7f0d0c116500 2019-12-05T06:24:22.637 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:22.612+0000 7f0d0affd700 1 --2- 172.21.15.42:0/660526160 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0cf003ae90 0x7f0cf003d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:22.637 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:22.616+0000 7f0d0affd700 1 --2- 172.21.15.42:0/660526160 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0cf003ae90 0x7f0cf003d3a0 crc :-1 s=READY pgs=1078 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:24:22.637 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:22.620+0000 7f0d11351700 1 -- 172.21.15.42:0/660526160 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d0c05d110 con 0x7f0d0c116500 2019-12-05T06:24:22.638 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:22.628+0000 7f0d0b7fe700 1 --2- 172.21.15.42:0/660526160 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0d0c116500 0x7f0d0c07c630 secure :-1 s=THROTTLE_DONE pgs=561 cs=0 l=1 rx=0x7f0d0c07d330 tx=0x7f0cfc010ce0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:22.638 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:22.628+0000 7f0d03fff700 1 -- 172.21.15.42:0/660526160 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f0cfc0323f0 con 0x7f0d0c116500 2019-12-05T06:24:22.808 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:22.800+0000 7f0d11351700 1 -- 172.21.15.42:0/660526160 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0d0c0a8cb0 con 0x7f0d0c116500 2019-12-05T06:24:22.808 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:22.800+0000 7f0d0b7fe700 1 --2- 172.21.15.42:0/660526160 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0d0c116500 0x7f0d0c07c630 secure :-1 s=THROTTLE_DONE pgs=561 cs=0 l=1 rx=0x7f0d0c07d330 tx=0x7f0cfc010ce0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:22.809 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:22.800+0000 7f0d03fff700 1 -- 172.21.15.42:0/660526160 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 308 v308) v1 ==== 95+0+2116 (secure 0 0 0) 0x7f0cfc032650 con 0x7f0d0c116500 2019-12-05T06:24:22.811 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 308 2019-12-05T06:24:22.811 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:24:22.811 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":308,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7568,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/2324657302","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2324657302},{"type":"v1","addr":"172.21.15.40:6833","nonce":2324657302}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":308}],"filesystems":[{"mdsmap":{"epoch":307,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:23:50.178163+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":104,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10993},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10993":{"gid":10993,"name":"a","rank":0,"incarnation":304,"state":"up:active","state_seq":33,"addr":"172.21.15.42:6835/3964426552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3964426552},{"type":"v1","addr":"172.21.15.42:6835","nonce":3964426552}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:24:22.815 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:22.804+0000 7f0d11351700 1 -- 172.21.15.42:0/660526160 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0cf003ae90 msgr2=0x7f0cf003d3a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:22.815 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:22.804+0000 7f0d11351700 1 --2- 172.21.15.42:0/660526160 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0cf003ae90 0x7f0cf003d3a0 crc :-1 s=READY pgs=1078 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:22.815 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:22.804+0000 7f0d11351700 1 -- 172.21.15.42:0/660526160 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0d0c116500 msgr2=0x7f0d0c07c630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:22.816 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:22.804+0000 7f0d11351700 1 --2- 172.21.15.42:0/660526160 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0d0c116500 0x7f0d0c07c630 secure :-1 s=READY pgs=561 cs=0 l=1 rx=0x7f0d0c07d330 tx=0x7f0cfc010ce0).stop 2019-12-05T06:24:22.816 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:22.804+0000 7f0d11351700 1 -- 172.21.15.42:0/660526160 shutdown_connections 2019-12-05T06:24:22.816 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:22.804+0000 7f0d11351700 1 --2- 172.21.15.42:0/660526160 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0cf003ae90 0x7f0cf003d3a0 unknown :-1 s=CLOSED pgs=1078 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:22.816 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:22.804+0000 7f0d11351700 1 --2- 172.21.15.42:0/660526160 >> v2:172.21.15.40:3300/0 conn(0x7f0d0c11b570 0x7f0d0c11b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:22.816 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:22.804+0000 7f0d11351700 1 --2- 172.21.15.42:0/660526160 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0d0c116500 0x7f0d0c07c630 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:22.816 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:22.804+0000 7f0d11351700 1 --2- 172.21.15.42:0/660526160 >> v2:172.21.15.42:3300/0 conn(0x7f0d0c07aca0 0x7f0d0c07b0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:22.817 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:22.808+0000 7f0d11351700 1 -- 172.21.15.42:0/660526160 shutdown_connections 2019-12-05T06:24:22.817 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:22.808+0000 7f0d11351700 1 -- 172.21.15.42:0/660526160 wait complete. 2019-12-05T06:24:22.817 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:22.808+0000 7f0d11351700 1 -- 172.21.15.42:0/660526160 >> 172.21.15.42:0/660526160 conn(0x7f0d0c07e8a0 msgr2=0x7f0d0c059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:24:22.831 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:24:24.832 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:24:24.942 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:24.928+0000 7f5d38cfe700 1 Processor -- start 2019-12-05T06:24:24.943 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:24.928+0000 7f5d38cfe700 1 -- start start 2019-12-05T06:24:24.943 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:24.928+0000 7f5d38cfe700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5d3408df80 0x7f5d3409f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:24:24.943 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:24.928+0000 7f5d38cfe700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d3409fc60 con 0x7f5d340a7740 2019-12-05T06:24:24.943 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:24.928+0000 7f5d38cfe700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d340a0710 con 0x7f5d3409f830 2019-12-05T06:24:24.944 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:24.928+0000 7f5d38cfe700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d340a0b00 con 0x7f5d3408df80 2019-12-05T06:24:24.944 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:24.932+0000 7f5d3359e700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f5d3409f830 0x7f5d3414efe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:54828/0 (socket says 172.21.15.42:54828) 2019-12-05T06:24:24.944 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:24.932+0000 7f5d3359e700 1 -- 172.21.15.42:0/1820062392 learned_addr learned my addr 172.21.15.42:0/1820062392 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:24:24.944 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:24.932+0000 7f5d3259c700 1 --2- 172.21.15.42:0/1820062392 >> v2:172.21.15.40:3300/0 conn(0x7f5d3408df80 0x7f5d3409f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:24.945 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:24.932+0000 7f5d31d9b700 1 -- 172.21.15.42:0/1820062392 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 254886888 0 0) 0x7f5d340a0710 con 0x7f5d3409f830 2019-12-05T06:24:24.945 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:24.932+0000 7f5d31d9b700 1 -- 172.21.15.42:0/1820062392 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d140030b0 con 0x7f5d3409f830 2019-12-05T06:24:24.945 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:24.932+0000 7f5d31d9b700 1 -- 172.21.15.42:0/1820062392 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2027216098 0 0) 0x7f5d3409fc60 con 0x7f5d340a7740 2019-12-05T06:24:24.945 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:24.932+0000 7f5d31d9b700 1 -- 172.21.15.42:0/1820062392 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d340a0710 con 0x7f5d340a7740 2019-12-05T06:24:24.945 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:24.932+0000 7f5d31d9b700 1 -- 172.21.15.42:0/1820062392 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1428578242 0 0) 0x7f5d140030b0 con 0x7f5d3409f830 2019-12-05T06:24:24.946 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:24.932+0000 7f5d3259c700 1 -- 172.21.15.42:0/1820062392 >> v1:172.21.15.42:6789/0 conn(0x7f5d3409f830 legacy=0x7f5d3414efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:24.946 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:24.932+0000 7f5d3259c700 1 -- 172.21.15.42:0/1820062392 >> v1:172.21.15.40:6789/0 conn(0x7f5d340a7740 legacy=0x7f5d3407e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:24.946 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:24.932+0000 7f5d3259c700 1 -- 172.21.15.42:0/1820062392 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d3407ef20 con 0x7f5d3408df80 2019-12-05T06:24:24.946 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:24.932+0000 7f5d3259c700 1 --2- 172.21.15.42:0/1820062392 >> v2:172.21.15.40:3300/0 conn(0x7f5d3408df80 0x7f5d3409f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f5d1c008b10 tx=0x7f5d2c001950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:24.946 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:24.932+0000 7f5d3259c700 1 --2- 172.21.15.42:0/1820062392 >> v2:172.21.15.40:3300/0 conn(0x7f5d3408df80 0x7f5d3409f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f5d1c008b10 tx=0x7f5d2c001950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:24.946 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:24.932+0000 7f5d3259c700 1 --2- 172.21.15.42:0/1820062392 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5d3408df80 0x7f5d3409f2e0 secure :-1 s=READY pgs=1942 cs=0 l=1 rx=0x7f5d1c008b10 tx=0x7f5d2c001950).ready entity=mon.0 client_cookie=2bf7b6847802888 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:24:24.947 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:24.932+0000 7f5d3259c700 1 --2- 172.21.15.42:0/1820062392 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5d3408df80 0x7f5d3409f2e0 secure :-1 s=THROTTLE_DONE pgs=1942 cs=0 l=1 rx=0x7f5d1c008b10 tx=0x7f5d2c001950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:24.947 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:24.932+0000 7f5d31d9b700 1 -- 172.21.15.42:0/1820062392 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5d2c001750 con 0x7f5d3408df80 2019-12-05T06:24:24.947 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:24.932+0000 7f5d3259c700 1 --2- 172.21.15.42:0/1820062392 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5d3408df80 0x7f5d3409f2e0 secure :-1 s=THROTTLE_DONE pgs=1942 cs=0 l=1 rx=0x7f5d1c008b10 tx=0x7f5d2c001950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:24.947 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:24.932+0000 7f5d3259c700 1 --2- 172.21.15.42:0/1820062392 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5d3408df80 0x7f5d3409f2e0 secure :-1 s=THROTTLE_DONE pgs=1942 cs=0 l=1 rx=0x7f5d1c008b10 tx=0x7f5d2c001950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:24.947 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:24.932+0000 7f5d31d9b700 1 -- 172.21.15.42:0/1820062392 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d2c001560 con 0x7f5d3408df80 2019-12-05T06:24:24.947 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:24.932+0000 7f5d31d9b700 1 -- 172.21.15.42:0/1820062392 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5d1c011ac0 con 0x7f5d3408df80 2019-12-05T06:24:24.947 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:24.932+0000 7f5d38cfe700 1 -- 172.21.15.42:0/1820062392 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5d3408df80 msgr2=0x7f5d3409f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:24.948 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:24.932+0000 7f5d38cfe700 1 --2- 172.21.15.42:0/1820062392 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5d3408df80 0x7f5d3409f2e0 secure :-1 s=READY pgs=1942 cs=0 l=1 rx=0x7f5d1c008b10 tx=0x7f5d2c001950).stop 2019-12-05T06:24:24.948 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:24.932+0000 7f5d38cfe700 1 -- 172.21.15.42:0/1820062392 shutdown_connections 2019-12-05T06:24:24.948 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:24.932+0000 7f5d38cfe700 1 --2- 172.21.15.42:0/1820062392 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5d3408df80 0x7f5d3409f2e0 unknown :-1 s=CLOSED pgs=1942 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:24.948 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:24.932+0000 7f5d38cfe700 1 -- 172.21.15.42:0/1820062392 shutdown_connections 2019-12-05T06:24:24.948 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:24.932+0000 7f5d38cfe700 1 -- 172.21.15.42:0/1820062392 wait complete. 2019-12-05T06:24:24.949 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:24.932+0000 7f5d38cfe700 1 -- 172.21.15.42:0/1820062392 >> 172.21.15.42:0/1820062392 conn(0x7f5d340aabe0 msgr2=0x7f5d340a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:24:24.949 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:24.936+0000 7f5d38cfe700 1 Processor -- start 2019-12-05T06:24:24.949 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:24.936+0000 7f5d38cfe700 1 -- start start 2019-12-05T06:24:24.952 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:24.940+0000 7f5d38cfe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5d3411b7a0 0x7f5d3411bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:24:24.952 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:24.940+0000 7f5d38cfe700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5d34136ff0 0x7f5d34137420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:24:24.952 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:24.940+0000 7f5d38cfe700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d3411a690 con 0x7f5d34118b70 2019-12-05T06:24:24.952 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:24.940+0000 7f5d38cfe700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d3411a9d0 con 0x7f5d3411b7a0 2019-12-05T06:24:24.953 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:24.940+0000 7f5d38cfe700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d3411ac70 con 0x7f5d34136ff0 2019-12-05T06:24:24.953 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:24.940+0000 7f5d3259c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5d3411b7a0 0x7f5d3411bbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:24.953 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:24.940+0000 7f5d3259c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5d3411b7a0 0x7f5d3411bbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:54794/0 (socket says 172.21.15.42:54794) 2019-12-05T06:24:24.953 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:24.940+0000 7f5d3259c700 1 -- 172.21.15.42:0/1263989418 learned_addr learned my addr 172.21.15.42:0/1263989418 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:24:24.953 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:24.940+0000 7f5d3359e700 1 --2- 172.21.15.42:0/1263989418 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5d34136ff0 0x7f5d34137420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:24.953 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:24.940+0000 7f5d3259c700 1 -- 172.21.15.42:0/1263989418 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5d34136ff0 msgr2=0x7f5d34137420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:24.953 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:24.940+0000 7f5d3259c700 1 --2- 172.21.15.42:0/1263989418 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5d34136ff0 0x7f5d34137420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:24.954 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:24.940+0000 7f5d3259c700 1 -- 172.21.15.42:0/1263989418 >> v1:172.21.15.42:6789/0 conn(0x7f5d34118b70 legacy=0x7f5d3407c520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:24.954 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:24.940+0000 7f5d3259c700 1 -- 172.21.15.42:0/1263989418 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d3411c120 con 0x7f5d3411b7a0 2019-12-05T06:24:24.954 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:24.940+0000 7f5d3259c700 1 --2- 172.21.15.42:0/1263989418 >> v2:172.21.15.42:3300/0 conn(0x7f5d3411b7a0 0x7f5d3411bbd0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f5d340a1340 tx=0x7f5d1c0088d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:24.954 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:24.940+0000 7f5d3259c700 1 --2- 172.21.15.42:0/1263989418 >> v2:172.21.15.42:3300/0 conn(0x7f5d3411b7a0 0x7f5d3411bbd0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f5d340a1340 tx=0x7f5d1c0088d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:24.954 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:24.940+0000 7f5d3259c700 1 --2- 172.21.15.42:0/1263989418 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5d3411b7a0 0x7f5d3411bbd0 secure :-1 s=READY pgs=2597 cs=0 l=1 rx=0x7f5d340a1340 tx=0x7f5d1c0088d0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:24:24.954 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:24.940+0000 7f5d3259c700 1 --2- 172.21.15.42:0/1263989418 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5d3411b7a0 0x7f5d3411bbd0 secure :-1 s=THROTTLE_DONE pgs=2597 cs=0 l=1 rx=0x7f5d340a1340 tx=0x7f5d1c0088d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:24.954 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:24.940+0000 7f5d2b7fe700 1 -- 172.21.15.42:0/1263989418 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5d1c00f700 con 0x7f5d3411b7a0 2019-12-05T06:24:24.955 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:24.940+0000 7f5d3259c700 1 --2- 172.21.15.42:0/1263989418 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5d3411b7a0 0x7f5d3411bbd0 secure :-1 s=THROTTLE_DONE pgs=2597 cs=0 l=1 rx=0x7f5d340a1340 tx=0x7f5d1c0088d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:24.955 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:24.940+0000 7f5d3259c700 1 --2- 172.21.15.42:0/1263989418 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5d3411b7a0 0x7f5d3411bbd0 secure :-1 s=THROTTLE_DONE pgs=2597 cs=0 l=1 rx=0x7f5d340a1340 tx=0x7f5d1c0088d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:24.955 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:24.940+0000 7f5d2b7fe700 1 -- 172.21.15.42:0/1263989418 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d1c0116e0 con 0x7f5d3411b7a0 2019-12-05T06:24:24.955 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:24.940+0000 7f5d38cfe700 1 -- 172.21.15.42:0/1263989418 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d34137970 con 0x7f5d3411b7a0 2019-12-05T06:24:24.955 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:24.940+0000 7f5d2b7fe700 1 -- 172.21.15.42:0/1263989418 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5d1c005ab0 con 0x7f5d3411b7a0 2019-12-05T06:24:24.956 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:24.940+0000 7f5d38cfe700 1 -- 172.21.15.42:0/1263989418 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d3413e3b0 con 0x7f5d3411b7a0 2019-12-05T06:24:24.956 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:24.940+0000 7f5d3259c700 1 --2- 172.21.15.42:0/1263989418 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5d3411b7a0 0x7f5d3411bbd0 secure :-1 s=THROTTLE_DONE pgs=2597 cs=0 l=1 rx=0x7f5d340a1340 tx=0x7f5d1c0088d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:24.956 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:24.940+0000 7f5d2b7fe700 1 -- 172.21.15.42:0/1263989418 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f5d1c007150 con 0x7f5d3411b7a0 2019-12-05T06:24:24.956 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:24.940+0000 7f5d3259c700 1 --2- 172.21.15.42:0/1263989418 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5d3411b7a0 0x7f5d3411bbd0 secure :-1 s=THROTTLE_DONE pgs=2597 cs=0 l=1 rx=0x7f5d340a1340 tx=0x7f5d1c0088d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:24.956 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:24.944+0000 7f5d2b7fe700 1 --2- 172.21.15.42:0/1263989418 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5d0c03ad70 0x7f5d0c03d280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:24:24.957 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:24.944+0000 7f5d2b7fe700 1 -- 172.21.15.42:0/1263989418 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 10761+0+0 (secure 0 0 0) 0x7f5d1c04b3f0 con 0x7f5d3411b7a0 2019-12-05T06:24:24.962 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:24.944+0000 7f5d32d9d700 1 --2- 172.21.15.42:0/1263989418 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5d0c03ad70 0x7f5d0c03d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:24.962 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:24.944+0000 7f5d32d9d700 1 --2- 172.21.15.42:0/1263989418 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5d0c03ad70 0x7f5d0c03d280 crc :-1 s=READY pgs=1079 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:24:24.962 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:24.944+0000 7f5d38cfe700 1 -- 172.21.15.42:0/1263989418 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d3405d110 con 0x7f5d3411b7a0 2019-12-05T06:24:24.963 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:24.952+0000 7f5d3259c700 1 --2- 172.21.15.42:0/1263989418 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5d3411b7a0 0x7f5d3411bbd0 secure :-1 s=THROTTLE_DONE pgs=2597 cs=0 l=1 rx=0x7f5d340a1340 tx=0x7f5d1c0088d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:24.963 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:24.952+0000 7f5d2b7fe700 1 -- 172.21.15.42:0/1263989418 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f5d1c032470 con 0x7f5d3411b7a0 2019-12-05T06:24:25.132 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:25.128+0000 7f5d38cfe700 1 -- 172.21.15.42:0/1263989418 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5d3405b8c0 con 0x7f5d3411b7a0 2019-12-05T06:24:25.133 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:25.128+0000 7f5d3259c700 1 --2- 172.21.15.42:0/1263989418 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5d3411b7a0 0x7f5d3411bbd0 secure :-1 s=THROTTLE_DONE pgs=2597 cs=0 l=1 rx=0x7f5d340a1340 tx=0x7f5d1c0088d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:25.133 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:25.128+0000 7f5d2b7fe700 1 -- 172.21.15.42:0/1263989418 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 308 v308) v1 ==== 95+0+2116 (secure 0 0 0) 0x7f5d1c005dc0 con 0x7f5d3411b7a0 2019-12-05T06:24:25.134 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 308 2019-12-05T06:24:25.134 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:24:25.134 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":308,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7568,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/2324657302","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2324657302},{"type":"v1","addr":"172.21.15.40:6833","nonce":2324657302}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":308}],"filesystems":[{"mdsmap":{"epoch":307,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:23:50.178163+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":104,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10993},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10993":{"gid":10993,"name":"a","rank":0,"incarnation":304,"state":"up:active","state_seq":33,"addr":"172.21.15.42:6835/3964426552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3964426552},{"type":"v1","addr":"172.21.15.42:6835","nonce":3964426552}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:24:25.136 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:25.128+0000 7f5d38cfe700 1 -- 172.21.15.42:0/1263989418 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5d0c03ad70 msgr2=0x7f5d0c03d280 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:25.137 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:25.128+0000 7f5d38cfe700 1 --2- 172.21.15.42:0/1263989418 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5d0c03ad70 0x7f5d0c03d280 crc :-1 s=READY pgs=1079 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:25.137 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:25.128+0000 7f5d38cfe700 1 -- 172.21.15.42:0/1263989418 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5d3411b7a0 msgr2=0x7f5d3411bbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:25.137 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:25.128+0000 7f5d38cfe700 1 --2- 172.21.15.42:0/1263989418 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5d3411b7a0 0x7f5d3411bbd0 secure :-1 s=READY pgs=2597 cs=0 l=1 rx=0x7f5d340a1340 tx=0x7f5d1c0088d0).stop 2019-12-05T06:24:25.137 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:25.132+0000 7f5d38cfe700 1 -- 172.21.15.42:0/1263989418 shutdown_connections 2019-12-05T06:24:25.137 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:25.132+0000 7f5d38cfe700 1 --2- 172.21.15.42:0/1263989418 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5d0c03ad70 0x7f5d0c03d280 unknown :-1 s=CLOSED pgs=1079 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:25.137 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:25.132+0000 7f5d38cfe700 1 --2- 172.21.15.42:0/1263989418 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5d34136ff0 0x7f5d34137420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:25.137 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:25.132+0000 7f5d38cfe700 1 --2- 172.21.15.42:0/1263989418 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5d3411b7a0 0x7f5d3411bbd0 unknown :-1 s=CLOSED pgs=2597 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:25.138 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:25.132+0000 7f5d38cfe700 1 -- 172.21.15.42:0/1263989418 shutdown_connections 2019-12-05T06:24:25.138 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:25.132+0000 7f5d38cfe700 1 -- 172.21.15.42:0/1263989418 wait complete. 2019-12-05T06:24:25.138 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:25.132+0000 7f5d38cfe700 1 -- 172.21.15.42:0/1263989418 >> 172.21.15.42:0/1263989418 conn(0x7f5d3408fde0 msgr2=0x7f5d34059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:24:25.146 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:24:26.852 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a has been restored 2019-12-05T06:24:27.148 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:24:27.255 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:27.244+0000 7f9ea2aaa700 1 Processor -- start 2019-12-05T06:24:27.255 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:27.244+0000 7f9ea2aaa700 1 -- start start 2019-12-05T06:24:27.256 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:27.248+0000 7f9ea2aaa700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f9e9c08df80 0x7f9e9c09f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:24:27.256 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:27.248+0000 7f9ea2aaa700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e9c09fc60 con 0x7f9e9c0a7740 2019-12-05T06:24:27.256 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:27.248+0000 7f9ea2aaa700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e9c0a0710 con 0x7f9e9c09f830 2019-12-05T06:24:27.256 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:27.248+0000 7f9ea2aaa700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e9c0a0b00 con 0x7f9e9c08df80 2019-12-05T06:24:27.256 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:27.248+0000 7f9ea0846700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f9e9c08df80 0x7f9e9c09f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:27.256 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:27.248+0000 7f9ea0846700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f9e9c08df80 0x7f9e9c09f2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:54804/0 (socket says 172.21.15.42:54804) 2019-12-05T06:24:27.256 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:27.248+0000 7f9ea0846700 1 -- 172.21.15.42:0/1719102901 learned_addr learned my addr 172.21.15.42:0/1719102901 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:24:27.257 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:27.248+0000 7f9ea0846700 1 -- 172.21.15.42:0/1719102901 >> v1:172.21.15.42:6789/0 conn(0x7f9e9c09f830 legacy=0x7f9e9c14efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:27.257 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:27.248+0000 7f9ea0846700 1 -- 172.21.15.42:0/1719102901 >> v1:172.21.15.40:6789/0 conn(0x7f9e9c0a7740 legacy=0x7f9e9c07e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:27.257 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:27.248+0000 7f9ea0846700 1 -- 172.21.15.42:0/1719102901 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e9c07ef20 con 0x7f9e9c08df80 2019-12-05T06:24:27.257 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:27.248+0000 7f9ea0846700 1 --2- 172.21.15.42:0/1719102901 >> v2:172.21.15.42:3300/0 conn(0x7f9e9c08df80 0x7f9e9c09f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f9e90008790 tx=0x7f9e9000fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:27.257 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:27.248+0000 7f9ea0846700 1 --2- 172.21.15.42:0/1719102901 >> v2:172.21.15.42:3300/0 conn(0x7f9e9c08df80 0x7f9e9c09f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f9e90008790 tx=0x7f9e9000fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:27.257 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:27.248+0000 7f9ea0846700 1 --2- 172.21.15.42:0/1719102901 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9e9c08df80 0x7f9e9c09f2e0 secure :-1 s=READY pgs=2599 cs=0 l=1 rx=0x7f9e90008790 tx=0x7f9e9000fcb0).ready entity=mon.1 client_cookie=bcae89c15af4a52a server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:24:27.257 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:27.248+0000 7f9ea0846700 1 --2- 172.21.15.42:0/1719102901 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9e9c08df80 0x7f9e9c09f2e0 secure :-1 s=THROTTLE_DONE pgs=2599 cs=0 l=1 rx=0x7f9e90008790 tx=0x7f9e9000fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:27.258 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:27.248+0000 7f9e8ffff700 1 -- 172.21.15.42:0/1719102901 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9e90011520 con 0x7f9e9c08df80 2019-12-05T06:24:27.258 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:27.248+0000 7f9ea0846700 1 --2- 172.21.15.42:0/1719102901 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9e9c08df80 0x7f9e9c09f2e0 secure :-1 s=THROTTLE_DONE pgs=2599 cs=0 l=1 rx=0x7f9e90008790 tx=0x7f9e9000fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:27.258 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:27.248+0000 7f9e8ffff700 1 -- 172.21.15.42:0/1719102901 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e90008ca0 con 0x7f9e9c08df80 2019-12-05T06:24:27.258 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:27.248+0000 7f9ea0846700 1 --2- 172.21.15.42:0/1719102901 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9e9c08df80 0x7f9e9c09f2e0 secure :-1 s=THROTTLE_DONE pgs=2599 cs=0 l=1 rx=0x7f9e90008790 tx=0x7f9e9000fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:27.258 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:27.248+0000 7f9e8ffff700 1 -- 172.21.15.42:0/1719102901 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9e90011b00 con 0x7f9e9c08df80 2019-12-05T06:24:27.258 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:27.248+0000 7f9ea2aaa700 1 -- 172.21.15.42:0/1719102901 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9e9c08df80 msgr2=0x7f9e9c09f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:27.258 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:27.248+0000 7f9ea2aaa700 1 --2- 172.21.15.42:0/1719102901 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9e9c08df80 0x7f9e9c09f2e0 secure :-1 s=READY pgs=2599 cs=0 l=1 rx=0x7f9e90008790 tx=0x7f9e9000fcb0).stop 2019-12-05T06:24:27.259 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:27.248+0000 7f9ea2aaa700 1 -- 172.21.15.42:0/1719102901 shutdown_connections 2019-12-05T06:24:27.259 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:27.248+0000 7f9ea2aaa700 1 --2- 172.21.15.42:0/1719102901 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9e9c08df80 0x7f9e9c09f2e0 unknown :-1 s=CLOSED pgs=2599 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:27.259 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:27.248+0000 7f9ea2aaa700 1 -- 172.21.15.42:0/1719102901 shutdown_connections 2019-12-05T06:24:27.259 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:27.248+0000 7f9ea2aaa700 1 -- 172.21.15.42:0/1719102901 wait complete. 2019-12-05T06:24:27.259 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:27.248+0000 7f9ea2aaa700 1 -- 172.21.15.42:0/1719102901 >> 172.21.15.42:0/1719102901 conn(0x7f9e9c0aabe0 msgr2=0x7f9e9c0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:24:27.262 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:27.252+0000 7f9ea2aaa700 1 Processor -- start 2019-12-05T06:24:27.262 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:27.252+0000 7f9ea2aaa700 1 -- start start 2019-12-05T06:24:27.262 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:27.252+0000 7f9ea2aaa700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9e9c08eeb0 0x7f9e9c117740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:24:27.262 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:27.252+0000 7f9ea2aaa700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f9e9c1194b0 0x7f9e9c1198e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:24:27.263 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:27.252+0000 7f9ea2aaa700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f9e9c128e90 0x7f9e9c1292c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:24:27.263 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:27.252+0000 7f9ea2aaa700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e9c118f70 con 0x7f9e9c128e90 2019-12-05T06:24:27.263 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:27.252+0000 7f9ea2aaa700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e9c119210 con 0x7f9e9c1194b0 2019-12-05T06:24:27.263 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:27.252+0000 7f9ea2aaa700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e9c117ec0 con 0x7f9e9c08eeb0 2019-12-05T06:24:27.263 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:27.252+0000 7f9ea0846700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f9e9c1194b0 0x7f9e9c1198e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:27.263 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:27.252+0000 7f9ea0846700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f9e9c1194b0 0x7f9e9c1198e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:54810/0 (socket says 172.21.15.42:54810) 2019-12-05T06:24:27.263 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:27.252+0000 7f9ea0846700 1 -- 172.21.15.42:0/2377874399 learned_addr learned my addr 172.21.15.42:0/2377874399 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:24:27.264 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:27.252+0000 7f9ea1047700 1 --2- 172.21.15.42:0/2377874399 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9e9c08eeb0 0x7f9e9c117740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:27.264 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:27.252+0000 7f9ea1848700 1 --2- 172.21.15.42:0/2377874399 >> v2:172.21.15.40:3300/0 conn(0x7f9e9c128e90 0x7f9e9c1292c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:27.264 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:27.256+0000 7f9ea0846700 1 -- 172.21.15.42:0/2377874399 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9e9c08eeb0 msgr2=0x7f9e9c117740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:27.264 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:27.256+0000 7f9ea0846700 1 --2- 172.21.15.42:0/2377874399 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9e9c08eeb0 0x7f9e9c117740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:27.264 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:27.256+0000 7f9ea0846700 1 -- 172.21.15.42:0/2377874399 >> v2:172.21.15.40:3300/0 conn(0x7f9e9c128e90 msgr2=0x7f9e9c1292c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:27.264 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:27.256+0000 7f9ea0846700 1 --2- 172.21.15.42:0/2377874399 >> v2:172.21.15.40:3300/0 conn(0x7f9e9c128e90 0x7f9e9c1292c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:27.264 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:27.256+0000 7f9ea0846700 1 -- 172.21.15.42:0/2377874399 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e9c0645a0 con 0x7f9e9c1194b0 2019-12-05T06:24:27.265 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:27.256+0000 7f9ea0846700 1 --2- 172.21.15.42:0/2377874399 >> v2:172.21.15.42:3300/0 conn(0x7f9e9c1194b0 0x7f9e9c1198e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f9e90008790 tx=0x7f9e900113b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:27.265 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:27.256+0000 7f9ea0846700 1 --2- 172.21.15.42:0/2377874399 >> v2:172.21.15.42:3300/0 conn(0x7f9e9c1194b0 0x7f9e9c1198e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f9e90008790 tx=0x7f9e900113b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:27.265 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:27.256+0000 7f9ea0846700 1 --2- 172.21.15.42:0/2377874399 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9e9c1194b0 0x7f9e9c1198e0 secure :-1 s=READY pgs=2600 cs=0 l=1 rx=0x7f9e90008790 tx=0x7f9e900113b0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:24:27.265 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:27.256+0000 7f9ea0846700 1 --2- 172.21.15.42:0/2377874399 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9e9c1194b0 0x7f9e9c1198e0 secure :-1 s=THROTTLE_DONE pgs=2600 cs=0 l=1 rx=0x7f9e90008790 tx=0x7f9e900113b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:27.265 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:27.256+0000 7f9ea0846700 1 --2- 172.21.15.42:0/2377874399 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9e9c1194b0 0x7f9e9c1198e0 secure :-1 s=THROTTLE_DONE pgs=2600 cs=0 l=1 rx=0x7f9e90008790 tx=0x7f9e900113b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:27.265 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:27.256+0000 7f9e8dffb700 1 -- 172.21.15.42:0/2377874399 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9e900070b0 con 0x7f9e9c1194b0 2019-12-05T06:24:27.265 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:27.256+0000 7f9ea0846700 1 --2- 172.21.15.42:0/2377874399 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9e9c1194b0 0x7f9e9c1198e0 secure :-1 s=THROTTLE_DONE pgs=2600 cs=0 l=1 rx=0x7f9e90008790 tx=0x7f9e900113b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:27.266 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:27.256+0000 7f9e8dffb700 1 -- 172.21.15.42:0/2377874399 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e90011720 con 0x7f9e9c1194b0 2019-12-05T06:24:27.266 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:27.256+0000 7f9e8dffb700 1 -- 172.21.15.42:0/2377874399 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9e9000a350 con 0x7f9e9c1194b0 2019-12-05T06:24:27.266 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:27.256+0000 7f9ea2aaa700 1 -- 172.21.15.42:0/2377874399 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e9c119e30 con 0x7f9e9c1194b0 2019-12-05T06:24:27.266 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:27.256+0000 7f9ea2aaa700 1 -- 172.21.15.42:0/2377874399 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e9c129810 con 0x7f9e9c1194b0 2019-12-05T06:24:27.266 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:27.256+0000 7f9ea0846700 1 --2- 172.21.15.42:0/2377874399 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9e9c1194b0 0x7f9e9c1198e0 secure :-1 s=THROTTLE_DONE pgs=2600 cs=0 l=1 rx=0x7f9e90008790 tx=0x7f9e900113b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:27.267 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:27.256+0000 7f9e8dffb700 1 -- 172.21.15.42:0/2377874399 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f9e9000a660 con 0x7f9e9c1194b0 2019-12-05T06:24:27.267 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:27.256+0000 7f9ea0846700 1 --2- 172.21.15.42:0/2377874399 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9e9c1194b0 0x7f9e9c1198e0 secure :-1 s=THROTTLE_DONE pgs=2600 cs=0 l=1 rx=0x7f9e90008790 tx=0x7f9e900113b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:27.268 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:27.256+0000 7f9e8dffb700 1 --2- 172.21.15.42:0/2377874399 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9e7c03ae90 0x7f9e7c03d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:24:27.268 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:27.256+0000 7f9e8dffb700 1 -- 172.21.15.42:0/2377874399 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 10761+0+0 (secure 0 0 0) 0x7f9e9004b890 con 0x7f9e9c1194b0 2019-12-05T06:24:27.268 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:27.256+0000 7f9ea1047700 1 --2- 172.21.15.42:0/2377874399 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9e7c03ae90 0x7f9e7c03d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:27.268 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:27.256+0000 7f9ea1047700 1 --2- 172.21.15.42:0/2377874399 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9e7c03ae90 0x7f9e7c03d3a0 crc :-1 s=READY pgs=1080 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:24:27.268 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:27.260+0000 7f9ea2aaa700 1 -- 172.21.15.42:0/2377874399 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e9c05d110 con 0x7f9e9c1194b0 2019-12-05T06:24:27.273 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:27.268+0000 7f9ea0846700 1 --2- 172.21.15.42:0/2377874399 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9e9c1194b0 0x7f9e9c1198e0 secure :-1 s=THROTTLE_DONE pgs=2600 cs=0 l=1 rx=0x7f9e90008790 tx=0x7f9e900113b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:27.273 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:27.268+0000 7f9e8dffb700 1 -- 172.21.15.42:0/2377874399 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f9e90032910 con 0x7f9e9c1194b0 2019-12-05T06:24:27.440 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:27.436+0000 7f9ea2aaa700 1 -- 172.21.15.42:0/2377874399 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9e9c05a4f0 con 0x7f9e9c1194b0 2019-12-05T06:24:27.441 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:27.436+0000 7f9ea0846700 1 --2- 172.21.15.42:0/2377874399 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9e9c1194b0 0x7f9e9c1198e0 secure :-1 s=THROTTLE_DONE pgs=2600 cs=0 l=1 rx=0x7f9e90008790 tx=0x7f9e900113b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:27.441 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:27.436+0000 7f9e8dffb700 1 -- 172.21.15.42:0/2377874399 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 308 v308) v1 ==== 95+0+2116 (secure 0 0 0) 0x7f9e90032b70 con 0x7f9e9c1194b0 2019-12-05T06:24:27.442 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 308 2019-12-05T06:24:27.442 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:24:27.442 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":308,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7568,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/2324657302","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2324657302},{"type":"v1","addr":"172.21.15.40:6833","nonce":2324657302}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":308}],"filesystems":[{"mdsmap":{"epoch":307,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:23:50.178163+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":104,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10993},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10993":{"gid":10993,"name":"a","rank":0,"incarnation":304,"state":"up:active","state_seq":33,"addr":"172.21.15.42:6835/3964426552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3964426552},{"type":"v1","addr":"172.21.15.42:6835","nonce":3964426552}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:24:27.444 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:27.436+0000 7f9ea2aaa700 1 -- 172.21.15.42:0/2377874399 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9e7c03ae90 msgr2=0x7f9e7c03d3a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:27.444 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:27.436+0000 7f9ea2aaa700 1 --2- 172.21.15.42:0/2377874399 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9e7c03ae90 0x7f9e7c03d3a0 crc :-1 s=READY pgs=1080 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:27.444 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:27.436+0000 7f9ea2aaa700 1 -- 172.21.15.42:0/2377874399 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9e9c1194b0 msgr2=0x7f9e9c1198e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:27.444 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:27.436+0000 7f9ea2aaa700 1 --2- 172.21.15.42:0/2377874399 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9e9c1194b0 0x7f9e9c1198e0 secure :-1 s=READY pgs=2600 cs=0 l=1 rx=0x7f9e90008790 tx=0x7f9e900113b0).stop 2019-12-05T06:24:27.444 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:27.436+0000 7f9ea2aaa700 1 -- 172.21.15.42:0/2377874399 shutdown_connections 2019-12-05T06:24:27.445 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:27.436+0000 7f9ea2aaa700 1 --2- 172.21.15.42:0/2377874399 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9e7c03ae90 0x7f9e7c03d3a0 unknown :-1 s=CLOSED pgs=1080 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:27.445 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:27.436+0000 7f9ea2aaa700 1 --2- 172.21.15.42:0/2377874399 >> v2:172.21.15.40:3300/0 conn(0x7f9e9c128e90 0x7f9e9c1292c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:27.445 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:27.436+0000 7f9ea2aaa700 1 --2- 172.21.15.42:0/2377874399 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9e9c08eeb0 0x7f9e9c117740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:27.445 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:27.436+0000 7f9ea2aaa700 1 --2- 172.21.15.42:0/2377874399 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9e9c1194b0 0x7f9e9c1198e0 unknown :-1 s=CLOSED pgs=2600 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:27.445 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:27.440+0000 7f9ea2aaa700 1 -- 172.21.15.42:0/2377874399 shutdown_connections 2019-12-05T06:24:27.445 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:27.440+0000 7f9ea2aaa700 1 -- 172.21.15.42:0/2377874399 wait complete. 2019-12-05T06:24:27.446 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:27.440+0000 7f9ea2aaa700 1 -- 172.21.15.42:0/2377874399 >> 172.21.15.42:0/2377874399 conn(0x7f9e9c09f220 msgr2=0x7f9e9c111db0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:24:27.456 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:24:29.457 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:24:29.571 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:29.560+0000 7f9988d72700 1 Processor -- start 2019-12-05T06:24:29.572 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:29.560+0000 7f9988d72700 1 -- start start 2019-12-05T06:24:29.572 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:29.564+0000 7f9988d72700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f99840a7740 0x7f998408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:24:29.572 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:29.564+0000 7f9988d72700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f998407e720 0x7f998407eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:24:29.572 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:29.564+0000 7f9988d72700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99840a0710 con 0x7f998409f2e0 2019-12-05T06:24:29.573 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:29.564+0000 7f9988d72700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99840a0b00 con 0x7f99840a7740 2019-12-05T06:24:29.573 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:29.564+0000 7f9988d72700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f998408e5b0 con 0x7f998407e720 2019-12-05T06:24:29.573 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:29.564+0000 7f998259c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f998407e720 0x7f998407eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:29.573 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:29.564+0000 7f998259c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f998407e720 0x7f998407eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:54818/0 (socket says 172.21.15.42:54818) 2019-12-05T06:24:29.573 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:29.564+0000 7f998259c700 1 -- 172.21.15.42:0/1743844321 learned_addr learned my addr 172.21.15.42:0/1743844321 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:24:29.573 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:29.564+0000 7f9982d9d700 1 --2- 172.21.15.42:0/1743844321 >> v2:172.21.15.40:3300/0 conn(0x7f99840a7740 0x7f998408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:29.573 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:29.564+0000 7f998259c700 1 -- 172.21.15.42:0/1743844321 >> v2:172.21.15.40:3300/0 conn(0x7f99840a7740 msgr2=0x7f998408df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:24:29.574 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:29.564+0000 7f998259c700 1 --2- 172.21.15.42:0/1743844321 >> v2:172.21.15.40:3300/0 conn(0x7f99840a7740 0x7f998408df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:24:29.574 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:29.564+0000 7f998259c700 1 -- 172.21.15.42:0/1743844321 >> v1:172.21.15.40:6789/0 conn(0x7f998409f2e0 legacy=0x7f998409f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:24:29.574 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:29.564+0000 7f998259c700 1 -- 172.21.15.42:0/1743844321 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f998407f0a0 con 0x7f998407e720 2019-12-05T06:24:29.574 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:29.564+0000 7f998259c700 1 --2- 172.21.15.42:0/1743844321 >> v2:172.21.15.42:3300/0 conn(0x7f998407e720 0x7f998407eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f996c0032b0 tx=0x7f996c0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:29.574 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:29.564+0000 7f998259c700 1 --2- 172.21.15.42:0/1743844321 >> v2:172.21.15.42:3300/0 conn(0x7f998407e720 0x7f998407eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f996c0032b0 tx=0x7f996c0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:29.574 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:29.564+0000 7f998259c700 1 --2- 172.21.15.42:0/1743844321 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f998407e720 0x7f998407eb50 secure :-1 s=READY pgs=2601 cs=0 l=1 rx=0x7f996c0032b0 tx=0x7f996c0100b0).ready entity=mon.1 client_cookie=69e58d6a13b9d382 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:24:29.574 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:29.564+0000 7f998259c700 1 --2- 172.21.15.42:0/1743844321 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f998407e720 0x7f998407eb50 secure :-1 s=THROTTLE_DONE pgs=2601 cs=0 l=1 rx=0x7f996c0032b0 tx=0x7f996c0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:29.575 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:29.564+0000 7f9981d9b700 1 -- 172.21.15.42:0/1743844321 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f996c011dd0 con 0x7f998407e720 2019-12-05T06:24:29.575 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:29.564+0000 7f998259c700 1 --2- 172.21.15.42:0/1743844321 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f998407e720 0x7f998407eb50 secure :-1 s=THROTTLE_DONE pgs=2601 cs=0 l=1 rx=0x7f996c0032b0 tx=0x7f996c0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:29.575 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:29.564+0000 7f998259c700 1 --2- 172.21.15.42:0/1743844321 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f998407e720 0x7f998407eb50 secure :-1 s=THROTTLE_DONE pgs=2601 cs=0 l=1 rx=0x7f996c0032b0 tx=0x7f996c0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:29.575 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:29.564+0000 7f9981d9b700 1 -- 172.21.15.42:0/1743844321 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f996c0119f0 con 0x7f998407e720 2019-12-05T06:24:29.575 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:29.564+0000 7f9981d9b700 1 -- 172.21.15.42:0/1743844321 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f996c00a540 con 0x7f998407e720 2019-12-05T06:24:29.575 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:29.564+0000 7f9988d72700 1 -- 172.21.15.42:0/1743844321 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f998407e720 msgr2=0x7f998407eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:29.575 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:29.564+0000 7f9988d72700 1 --2- 172.21.15.42:0/1743844321 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f998407e720 0x7f998407eb50 secure :-1 s=READY pgs=2601 cs=0 l=1 rx=0x7f996c0032b0 tx=0x7f996c0100b0).stop 2019-12-05T06:24:29.575 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:29.564+0000 7f9988d72700 1 -- 172.21.15.42:0/1743844321 shutdown_connections 2019-12-05T06:24:29.576 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:29.564+0000 7f9988d72700 1 --2- 172.21.15.42:0/1743844321 >> v2:172.21.15.40:3300/0 conn(0x7f99840a7740 0x7f998408df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:24:29.576 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:29.564+0000 7f9988d72700 1 --2- 172.21.15.42:0/1743844321 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f998407e720 0x7f998407eb50 unknown :-1 s=CLOSED pgs=2601 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:29.576 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:29.564+0000 7f9988d72700 1 -- 172.21.15.42:0/1743844321 shutdown_connections 2019-12-05T06:24:29.576 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:29.564+0000 7f9988d72700 1 -- 172.21.15.42:0/1743844321 wait complete. 2019-12-05T06:24:29.576 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:29.564+0000 7f9988d72700 1 -- 172.21.15.42:0/1743844321 >> 172.21.15.42:0/1743844321 conn(0x7f99840aabe0 msgr2=0x7f99840a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:24:29.579 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:29.568+0000 7f9988d72700 1 Processor -- start 2019-12-05T06:24:29.579 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:29.568+0000 7f9988d72700 1 -- start start 2019-12-05T06:24:29.579 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:29.568+0000 7f9988d72700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f9984065af0 0x7f9984063590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:24:29.580 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:29.568+0000 7f9988d72700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f99841237b0 0x7f9984123be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:24:29.580 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:29.568+0000 7f9988d72700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9984127550 con 0x7f998408ee90 2019-12-05T06:24:29.580 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:29.568+0000 7f9988d72700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99840632e0 con 0x7f99841237b0 2019-12-05T06:24:29.580 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:29.568+0000 7f9988d72700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9984063bc0 con 0x7f9984065af0 2019-12-05T06:24:29.580 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:29.568+0000 7f9982d9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f9984065af0 0x7f9984063590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:29.580 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:29.568+0000 7f9982d9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f9984065af0 0x7f9984063590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:54824/0 (socket says 172.21.15.42:54824) 2019-12-05T06:24:29.581 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:29.568+0000 7f9982d9d700 1 -- 172.21.15.42:0/2730694318 learned_addr learned my addr 172.21.15.42:0/2730694318 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:24:29.581 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:29.568+0000 7f998359e700 1 --2- 172.21.15.42:0/2730694318 >> v2:172.21.15.40:3300/0 conn(0x7f99841237b0 0x7f9984123be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:29.581 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:29.568+0000 7f9982d9d700 1 -- 172.21.15.42:0/2730694318 >> v2:172.21.15.40:3300/0 conn(0x7f99841237b0 msgr2=0x7f9984123be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:29.581 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:29.568+0000 7f9982d9d700 1 --2- 172.21.15.42:0/2730694318 >> v2:172.21.15.40:3300/0 conn(0x7f99841237b0 0x7f9984123be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:29.581 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:29.568+0000 7f9982d9d700 1 -- 172.21.15.42:0/2730694318 >> v1:172.21.15.42:6789/0 conn(0x7f998408ee90 legacy=0x7f9984126d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:29.581 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:29.568+0000 7f9982d9d700 1 -- 172.21.15.42:0/2730694318 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9984124130 con 0x7f9984065af0 2019-12-05T06:24:29.581 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:29.568+0000 7f9982d9d700 1 --2- 172.21.15.42:0/2730694318 >> v2:172.21.15.42:3300/0 conn(0x7f9984065af0 0x7f9984063590 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f9984065300 tx=0x7f99740109f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:29.582 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:29.568+0000 7f9982d9d700 1 --2- 172.21.15.42:0/2730694318 >> v2:172.21.15.42:3300/0 conn(0x7f9984065af0 0x7f9984063590 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f9984065300 tx=0x7f99740109f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:29.582 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:29.568+0000 7f9982d9d700 1 --2- 172.21.15.42:0/2730694318 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9984065af0 0x7f9984063590 secure :-1 s=READY pgs=2603 cs=0 l=1 rx=0x7f9984065300 tx=0x7f99740109f0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:24:29.582 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:29.568+0000 7f9982d9d700 1 --2- 172.21.15.42:0/2730694318 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9984065af0 0x7f9984063590 secure :-1 s=THROTTLE_DONE pgs=2603 cs=0 l=1 rx=0x7f9984065300 tx=0x7f99740109f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:29.582 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:29.568+0000 7f997b7fe700 1 -- 172.21.15.42:0/2730694318 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9974005930 con 0x7f9984065af0 2019-12-05T06:24:29.582 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:29.568+0000 7f9982d9d700 1 --2- 172.21.15.42:0/2730694318 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9984065af0 0x7f9984063590 secure :-1 s=THROTTLE_DONE pgs=2603 cs=0 l=1 rx=0x7f9984065300 tx=0x7f99740109f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:29.583 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:29.568+0000 7f9982d9d700 1 --2- 172.21.15.42:0/2730694318 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9984065af0 0x7f9984063590 secure :-1 s=THROTTLE_DONE pgs=2603 cs=0 l=1 rx=0x7f9984065300 tx=0x7f99740109f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:29.583 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:29.568+0000 7f997b7fe700 1 -- 172.21.15.42:0/2730694318 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9974007160 con 0x7f9984065af0 2019-12-05T06:24:29.583 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:29.568+0000 7f9988d72700 1 -- 172.21.15.42:0/2730694318 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9984124400 con 0x7f9984065af0 2019-12-05T06:24:29.583 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:29.568+0000 7f997b7fe700 1 -- 172.21.15.42:0/2730694318 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f99740060c0 con 0x7f9984065af0 2019-12-05T06:24:29.583 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:29.568+0000 7f9988d72700 1 -- 172.21.15.42:0/2730694318 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9984131970 con 0x7f9984065af0 2019-12-05T06:24:29.583 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:29.572+0000 7f9982d9d700 1 --2- 172.21.15.42:0/2730694318 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9984065af0 0x7f9984063590 secure :-1 s=THROTTLE_DONE pgs=2603 cs=0 l=1 rx=0x7f9984065300 tx=0x7f99740109f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:29.583 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:29.572+0000 7f997b7fe700 1 -- 172.21.15.42:0/2730694318 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f9974007a20 con 0x7f9984065af0 2019-12-05T06:24:29.584 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:29.572+0000 7f9982d9d700 1 --2- 172.21.15.42:0/2730694318 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9984065af0 0x7f9984063590 secure :-1 s=THROTTLE_DONE pgs=2603 cs=0 l=1 rx=0x7f9984065300 tx=0x7f99740109f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:29.584 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:29.572+0000 7f997b7fe700 1 --2- 172.21.15.42:0/2730694318 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f996803ae10 0x7f996803d320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:24:29.584 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:29.572+0000 7f997b7fe700 1 -- 172.21.15.42:0/2730694318 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 10761+0+0 (secure 0 0 0) 0x7f997404abf0 con 0x7f9984065af0 2019-12-05T06:24:29.584 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:29.572+0000 7f998259c700 1 --2- 172.21.15.42:0/2730694318 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f996803ae10 0x7f996803d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:29.584 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:29.572+0000 7f998259c700 1 --2- 172.21.15.42:0/2730694318 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f996803ae10 0x7f996803d320 crc :-1 s=READY pgs=1081 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:24:29.585 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:29.576+0000 7f9988d72700 1 -- 172.21.15.42:0/2730694318 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99840aae10 con 0x7f9984065af0 2019-12-05T06:24:29.589 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:29.584+0000 7f9982d9d700 1 --2- 172.21.15.42:0/2730694318 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9984065af0 0x7f9984063590 secure :-1 s=THROTTLE_DONE pgs=2603 cs=0 l=1 rx=0x7f9984065300 tx=0x7f99740109f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:29.590 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:29.584+0000 7f997b7fe700 1 -- 172.21.15.42:0/2730694318 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f9974031c70 con 0x7f9984065af0 2019-12-05T06:24:29.756 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:29.748+0000 7f9988d72700 1 -- 172.21.15.42:0/2730694318 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f998405d110 con 0x7f9984065af0 2019-12-05T06:24:29.757 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:29.752+0000 7f9982d9d700 1 --2- 172.21.15.42:0/2730694318 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9984065af0 0x7f9984063590 secure :-1 s=THROTTLE_DONE pgs=2603 cs=0 l=1 rx=0x7f9984065300 tx=0x7f99740109f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:29.757 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:29.752+0000 7f997b7fe700 1 -- 172.21.15.42:0/2730694318 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 308 v308) v1 ==== 95+0+2116 (secure 0 0 0) 0x7f9974031ed0 con 0x7f9984065af0 2019-12-05T06:24:29.757 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 308 2019-12-05T06:24:29.757 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:24:29.757 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":308,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7568,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/2324657302","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2324657302},{"type":"v1","addr":"172.21.15.40:6833","nonce":2324657302}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":308}],"filesystems":[{"mdsmap":{"epoch":307,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:23:50.178163+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":104,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10993},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10993":{"gid":10993,"name":"a","rank":0,"incarnation":304,"state":"up:active","state_seq":33,"addr":"172.21.15.42:6835/3964426552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3964426552},{"type":"v1","addr":"172.21.15.42:6835","nonce":3964426552}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:24:29.760 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:29.752+0000 7f9988d72700 1 -- 172.21.15.42:0/2730694318 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f996803ae10 msgr2=0x7f996803d320 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:29.760 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:29.752+0000 7f9988d72700 1 --2- 172.21.15.42:0/2730694318 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f996803ae10 0x7f996803d320 crc :-1 s=READY pgs=1081 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:29.760 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:29.752+0000 7f9988d72700 1 -- 172.21.15.42:0/2730694318 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9984065af0 msgr2=0x7f9984063590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:29.760 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:29.752+0000 7f9988d72700 1 --2- 172.21.15.42:0/2730694318 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9984065af0 0x7f9984063590 secure :-1 s=READY pgs=2603 cs=0 l=1 rx=0x7f9984065300 tx=0x7f99740109f0).stop 2019-12-05T06:24:29.761 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:29.752+0000 7f9988d72700 1 -- 172.21.15.42:0/2730694318 shutdown_connections 2019-12-05T06:24:29.761 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:29.752+0000 7f9988d72700 1 --2- 172.21.15.42:0/2730694318 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f996803ae10 0x7f996803d320 unknown :-1 s=CLOSED pgs=1081 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:29.761 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:29.752+0000 7f9988d72700 1 --2- 172.21.15.42:0/2730694318 >> v2:172.21.15.40:3300/0 conn(0x7f99841237b0 0x7f9984123be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:29.761 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:29.752+0000 7f9988d72700 1 --2- 172.21.15.42:0/2730694318 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9984065af0 0x7f9984063590 unknown :-1 s=CLOSED pgs=2603 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:29.762 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:29.752+0000 7f9988d72700 1 -- 172.21.15.42:0/2730694318 shutdown_connections 2019-12-05T06:24:29.762 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:29.752+0000 7f9988d72700 1 -- 172.21.15.42:0/2730694318 wait complete. 2019-12-05T06:24:29.762 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:29.752+0000 7f9988d72700 1 -- 172.21.15.42:0/2730694318 >> 172.21.15.42:0/2730694318 conn(0x7f9984116dd0 msgr2=0x7f9984059c00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:24:29.770 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:24:31.770 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:24:31.884 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:31.860+0000 7f8ef4c90700 1 Processor -- start 2019-12-05T06:24:31.884 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:31.860+0000 7f8ef4c90700 1 -- start start 2019-12-05T06:24:31.884 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:31.864+0000 7f8ef4c90700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8ef008df80 0x7f8ef009f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:24:31.884 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:31.864+0000 7f8ef4c90700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8ef009f830 0x7f8ef014efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:24:31.885 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:31.864+0000 7f8ef4c90700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ef009fc60 con 0x7f8ef00a7740 2019-12-05T06:24:31.885 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:31.864+0000 7f8ef4c90700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ef00a0b00 con 0x7f8ef009f830 2019-12-05T06:24:31.885 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:31.864+0000 7f8ef4c90700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ef008e5b0 con 0x7f8ef008df80 2019-12-05T06:24:31.885 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:31.864+0000 7f8eeed9d700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f8ef00a7740 0x7f8ef007e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:54868/0 (socket says 172.21.15.42:54868) 2019-12-05T06:24:31.885 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:31.864+0000 7f8eeed9d700 1 -- 172.21.15.42:0/1420513543 learned_addr learned my addr 172.21.15.42:0/1420513543 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:24:31.885 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:31.864+0000 7f8eee59c700 1 --2- 172.21.15.42:0/1420513543 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8ef008df80 0x7f8ef009f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:31.885 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:31.864+0000 7f8eef59e700 1 --2- 172.21.15.42:0/1420513543 >> v2:172.21.15.40:3300/0 conn(0x7f8ef009f830 0x7f8ef014efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:31.886 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:31.864+0000 7f8eedd9b700 1 -- 172.21.15.42:0/1420513543 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4070223683 0 0) 0x7f8ef009fc60 con 0x7f8ef00a7740 2019-12-05T06:24:31.886 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:31.864+0000 7f8eedd9b700 1 -- 172.21.15.42:0/1420513543 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ed80030b0 con 0x7f8ef00a7740 2019-12-05T06:24:31.886 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:31.864+0000 7f8eedd9b700 1 -- 172.21.15.42:0/1420513543 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 231601913 0 0) 0x7f8ed80030b0 con 0x7f8ef00a7740 2019-12-05T06:24:31.886 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:31.864+0000 7f8eedd9b700 1 -- 172.21.15.42:0/1420513543 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8ef008df80 msgr2=0x7f8ef009f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:24:31.886 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:31.864+0000 7f8eedd9b700 1 --2- 172.21.15.42:0/1420513543 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8ef008df80 0x7f8ef009f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:24:31.886 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:31.864+0000 7f8eedd9b700 1 -- 172.21.15.42:0/1420513543 >> v2:172.21.15.40:3300/0 conn(0x7f8ef009f830 msgr2=0x7f8ef014efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:24:31.886 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:31.864+0000 7f8eedd9b700 1 --2- 172.21.15.42:0/1420513543 >> v2:172.21.15.40:3300/0 conn(0x7f8ef009f830 0x7f8ef014efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:24:31.887 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:31.864+0000 7f8eedd9b700 1 -- 172.21.15.42:0/1420513543 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ef00a0710 con 0x7f8ef00a7740 2019-12-05T06:24:31.887 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:31.864+0000 7f8eedd9b700 1 -- 172.21.15.42:0/1420513543 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7f8ee0001750 con 0x7f8ef00a7740 2019-12-05T06:24:31.887 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:31.864+0000 7f8eedd9b700 1 -- 172.21.15.42:0/1420513543 >> v1:172.21.15.42:6789/0 conn(0x7f8ef00a7740 legacy=0x7f8ef007e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:31.887 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:31.864+0000 7f8eedd9b700 1 --2- 172.21.15.42:0/1420513543 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8ed80036e0 0x7f8ed8003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:24:31.887 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:31.864+0000 7f8eedd9b700 1 --2- 172.21.15.42:0/1420513543 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8ed8004420 0x7f8ed8004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:24:31.887 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:31.864+0000 7f8eedd9b700 1 --2- 172.21.15.42:0/1420513543 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8ed8004da0 0x7f8ed804f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:24:31.887 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:31.864+0000 7f8eedd9b700 1 -- 172.21.15.42:0/1420513543 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ef00a0b00 con 0x7f8ed8004420 2019-12-05T06:24:31.887 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:31.864+0000 7f8eedd9b700 1 -- 172.21.15.42:0/1420513543 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ef008e5b0 con 0x7f8ed80036e0 2019-12-05T06:24:31.888 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:31.864+0000 7f8eedd9b700 1 -- 172.21.15.42:0/1420513543 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ed8005200 con 0x7f8ed8004da0 2019-12-05T06:24:31.888 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:31.864+0000 7f8eeed9d700 1 --2- 172.21.15.42:0/1420513543 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8ed80036e0 0x7f8ed8003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:31.888 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:31.864+0000 7f8eee59c700 1 --2- 172.21.15.42:0/1420513543 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8ed8004420 0x7f8ed8004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:31.888 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:31.864+0000 7f8eef59e700 1 --2- 172.21.15.42:0/1420513543 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8ed8004da0 0x7f8ed804f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:31.888 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:31.868+0000 7f8eeed9d700 1 -- 172.21.15.42:0/1420513543 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8ed8004da0 msgr2=0x7f8ed804f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:24:31.888 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:31.868+0000 7f8eeed9d700 1 --2- 172.21.15.42:0/1420513543 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8ed8004da0 0x7f8ed804f680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:24:31.889 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:31.868+0000 7f8eeed9d700 1 -- 172.21.15.42:0/1420513543 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8ed8004420 msgr2=0x7f8ed8004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:24:31.889 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:31.868+0000 7f8eeed9d700 1 --2- 172.21.15.42:0/1420513543 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8ed8004420 0x7f8ed8004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:24:31.889 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:31.868+0000 7f8eeed9d700 1 -- 172.21.15.42:0/1420513543 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ed804fcc0 con 0x7f8ed80036e0 2019-12-05T06:24:31.889 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:31.868+0000 7f8eeed9d700 1 --2- 172.21.15.42:0/1420513543 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8ed80036e0 0x7f8ed8003b50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f8ee0001a50 tx=0x7f8ee000c220).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:31.889 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:31.868+0000 7f8eeed9d700 1 --2- 172.21.15.42:0/1420513543 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8ed80036e0 0x7f8ed8003b50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f8ee0001a50 tx=0x7f8ee000c220).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:31.890 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:31.868+0000 7f8eeed9d700 1 --2- 172.21.15.42:0/1420513543 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8ed80036e0 0x7f8ed8003b50 secure :-1 s=READY pgs=2605 cs=0 l=1 rx=0x7f8ee0001a50 tx=0x7f8ee000c220).ready entity=mon.1 client_cookie=72da1d24cff819c0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:24:31.890 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:31.868+0000 7f8eeed9d700 1 --2- 172.21.15.42:0/1420513543 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8ed80036e0 0x7f8ed8003b50 secure :-1 s=THROTTLE_DONE pgs=2605 cs=0 l=1 rx=0x7f8ee0001a50 tx=0x7f8ee000c220).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:31.890 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:31.868+0000 7f8eeed9d700 1 --2- 172.21.15.42:0/1420513543 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8ed80036e0 0x7f8ed8003b50 secure :-1 s=THROTTLE_DONE pgs=2605 cs=0 l=1 rx=0x7f8ee0001a50 tx=0x7f8ee000c220).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:31.890 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:31.868+0000 7f8eedd9b700 1 -- 172.21.15.42:0/1420513543 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8ee0010350 con 0x7f8ed80036e0 2019-12-05T06:24:31.890 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:31.868+0000 7f8eeed9d700 1 --2- 172.21.15.42:0/1420513543 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8ed80036e0 0x7f8ed8003b50 secure :-1 s=THROTTLE_DONE pgs=2605 cs=0 l=1 rx=0x7f8ee0001a50 tx=0x7f8ee000c220).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:31.891 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:31.868+0000 7f8eedd9b700 1 -- 172.21.15.42:0/1420513543 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ee000edf0 con 0x7f8ed80036e0 2019-12-05T06:24:31.891 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:31.868+0000 7f8eedd9b700 1 -- 172.21.15.42:0/1420513543 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8ee0010930 con 0x7f8ed80036e0 2019-12-05T06:24:31.891 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:31.868+0000 7f8ef4c90700 1 -- 172.21.15.42:0/1420513543 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8ed80036e0 msgr2=0x7f8ed8003b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:31.891 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:31.868+0000 7f8ef4c90700 1 --2- 172.21.15.42:0/1420513543 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8ed80036e0 0x7f8ed8003b50 secure :-1 s=READY pgs=2605 cs=0 l=1 rx=0x7f8ee0001a50 tx=0x7f8ee000c220).stop 2019-12-05T06:24:31.891 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:31.868+0000 7f8eef59e700 1 -- 172.21.15.42:0/1420513543 reap_dead start 2019-12-05T06:24:31.891 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:31.868+0000 7f8ef4c90700 1 -- 172.21.15.42:0/1420513543 shutdown_connections 2019-12-05T06:24:31.891 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:31.868+0000 7f8ef4c90700 1 -- 172.21.15.42:0/1420513543 shutdown_connections 2019-12-05T06:24:31.892 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:31.868+0000 7f8ef4c90700 1 -- 172.21.15.42:0/1420513543 wait complete. 2019-12-05T06:24:31.892 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:31.868+0000 7f8ef4c90700 1 -- 172.21.15.42:0/1420513543 >> 172.21.15.42:0/1420513543 conn(0x7f8ef00aabe0 msgr2=0x7f8ef00a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:24:31.892 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:31.872+0000 7f8ef4c90700 1 Processor -- start 2019-12-05T06:24:31.892 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:31.872+0000 7f8ef4c90700 1 -- start start 2019-12-05T06:24:31.892 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:31.872+0000 7f8ef4c90700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8ef007cee0 0x7f8ef011a330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:24:31.892 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:31.872+0000 7f8ef4c90700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f8ef011a880 0x7f8ef0135480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:24:31.892 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:31.872+0000 7f8ef4c90700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ef007d910 con 0x7f8ef011b1e0 2019-12-05T06:24:31.893 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:31.872+0000 7f8ef4c90700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ef007b530 con 0x7f8ef007cee0 2019-12-05T06:24:31.893 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:31.872+0000 7f8ef4c90700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ef0143430 con 0x7f8ef011a880 2019-12-05T06:24:31.893 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:31.872+0000 7f8eef59e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f8ef011a880 0x7f8ef0135480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:31.893 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:31.872+0000 7f8eef59e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f8ef011a880 0x7f8ef0135480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:54846/0 (socket says 172.21.15.42:54846) 2019-12-05T06:24:31.893 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:31.872+0000 7f8eef59e700 1 -- 172.21.15.42:0/3033064980 learned_addr learned my addr 172.21.15.42:0/3033064980 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:24:31.893 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:31.872+0000 7f8eee59c700 1 --2- 172.21.15.42:0/3033064980 >> v2:172.21.15.40:3300/0 conn(0x7f8ef007cee0 0x7f8ef011a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:31.893 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:31.872+0000 7f8eef59e700 1 -- 172.21.15.42:0/3033064980 >> v2:172.21.15.40:3300/0 conn(0x7f8ef007cee0 msgr2=0x7f8ef011a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:31.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:31.872+0000 7f8eef59e700 1 --2- 172.21.15.42:0/3033064980 >> v2:172.21.15.40:3300/0 conn(0x7f8ef007cee0 0x7f8ef011a330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:31.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:31.872+0000 7f8eef59e700 1 -- 172.21.15.42:0/3033064980 >> v1:172.21.15.40:6789/0 conn(0x7f8ef011b1e0 legacy=0x7f8ef007ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:31.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:31.872+0000 7f8eef59e700 1 -- 172.21.15.42:0/3033064980 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ef0143120 con 0x7f8ef011a880 2019-12-05T06:24:31.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:31.872+0000 7f8eee59c700 1 --2- 172.21.15.42:0/3033064980 >> v2:172.21.15.40:3300/0 conn(0x7f8ef007cee0 0x7f8ef011a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:24:31.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:31.872+0000 7f8eef59e700 1 --2- 172.21.15.42:0/3033064980 >> v2:172.21.15.42:3300/0 conn(0x7f8ef011a880 0x7f8ef0135480 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f8ee0001420 tx=0x7f8ee8010cb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:31.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:31.876+0000 7f8eef59e700 1 --2- 172.21.15.42:0/3033064980 >> v2:172.21.15.42:3300/0 conn(0x7f8ef011a880 0x7f8ef0135480 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f8ee0001420 tx=0x7f8ee8010cb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:31.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:31.876+0000 7f8eef59e700 1 --2- 172.21.15.42:0/3033064980 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8ef011a880 0x7f8ef0135480 secure :-1 s=READY pgs=2606 cs=0 l=1 rx=0x7f8ee0001420 tx=0x7f8ee8010cb0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:24:31.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:31.876+0000 7f8eef59e700 1 --2- 172.21.15.42:0/3033064980 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8ef011a880 0x7f8ef0135480 secure :-1 s=THROTTLE_DONE pgs=2606 cs=0 l=1 rx=0x7f8ee0001420 tx=0x7f8ee8010cb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:31.895 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:31.876+0000 7f8eef59e700 1 --2- 172.21.15.42:0/3033064980 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8ef011a880 0x7f8ef0135480 secure :-1 s=THROTTLE_DONE pgs=2606 cs=0 l=1 rx=0x7f8ee0001420 tx=0x7f8ee8010cb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:31.895 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:31.876+0000 7f8edf7fe700 1 -- 172.21.15.42:0/3033064980 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8ee8013060 con 0x7f8ef011a880 2019-12-05T06:24:31.895 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:31.876+0000 7f8eef59e700 1 --2- 172.21.15.42:0/3033064980 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8ef011a880 0x7f8ef0135480 secure :-1 s=THROTTLE_DONE pgs=2606 cs=0 l=1 rx=0x7f8ee0001420 tx=0x7f8ee8010cb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:31.895 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:31.876+0000 7f8edf7fe700 1 -- 172.21.15.42:0/3033064980 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ee80120c0 con 0x7f8ef011a880 2019-12-05T06:24:31.895 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:31.876+0000 7f8ef4c90700 1 -- 172.21.15.42:0/3033064980 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ef011c0f0 con 0x7f8ef011a880 2019-12-05T06:24:31.895 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:31.876+0000 7f8edf7fe700 1 -- 172.21.15.42:0/3033064980 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8ee8008670 con 0x7f8ef011a880 2019-12-05T06:24:31.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:31.876+0000 7f8ef4c90700 1 -- 172.21.15.42:0/3033064980 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ef011acb0 con 0x7f8ef011a880 2019-12-05T06:24:31.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:31.876+0000 7f8eef59e700 1 --2- 172.21.15.42:0/3033064980 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8ef011a880 0x7f8ef0135480 secure :-1 s=THROTTLE_DONE pgs=2606 cs=0 l=1 rx=0x7f8ee0001420 tx=0x7f8ee8010cb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:31.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:31.876+0000 7f8edf7fe700 1 -- 172.21.15.42:0/3033064980 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f8ee8008870 con 0x7f8ef011a880 2019-12-05T06:24:31.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:31.876+0000 7f8eef59e700 1 --2- 172.21.15.42:0/3033064980 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8ef011a880 0x7f8ef0135480 secure :-1 s=THROTTLE_DONE pgs=2606 cs=0 l=1 rx=0x7f8ee0001420 tx=0x7f8ee8010cb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:31.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:31.876+0000 7f8edf7fe700 1 --2- 172.21.15.42:0/3033064980 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8ed403acc0 0x7f8ed403d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:24:31.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:31.876+0000 7f8edf7fe700 1 -- 172.21.15.42:0/3033064980 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 10761+0+0 (secure 0 0 0) 0x7f8ee804aa40 con 0x7f8ef011a880 2019-12-05T06:24:31.897 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:31.876+0000 7f8eeed9d700 1 --2- 172.21.15.42:0/3033064980 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8ed403acc0 0x7f8ed403d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:31.897 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:31.876+0000 7f8ef4c90700 1 -- 172.21.15.42:0/3033064980 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ef0143b10 con 0x7f8ef011a880 2019-12-05T06:24:31.897 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:31.876+0000 7f8eeed9d700 1 --2- 172.21.15.42:0/3033064980 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8ed403acc0 0x7f8ed403d1d0 crc :-1 s=READY pgs=1082 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:24:31.898 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:31.884+0000 7f8eef59e700 1 --2- 172.21.15.42:0/3033064980 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8ef011a880 0x7f8ef0135480 secure :-1 s=THROTTLE_DONE pgs=2606 cs=0 l=1 rx=0x7f8ee0001420 tx=0x7f8ee8010cb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:31.898 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:31.884+0000 7f8edf7fe700 1 -- 172.21.15.42:0/3033064980 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f8ee8031ac0 con 0x7f8ef011a880 2019-12-05T06:24:32.062 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:32.056+0000 7f8ef4c90700 1 -- 172.21.15.42:0/3033064980 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8ef005d110 con 0x7f8ef011a880 2019-12-05T06:24:32.063 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:32.056+0000 7f8eef59e700 1 --2- 172.21.15.42:0/3033064980 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8ef011a880 0x7f8ef0135480 secure :-1 s=THROTTLE_DONE pgs=2606 cs=0 l=1 rx=0x7f8ee0001420 tx=0x7f8ee8010cb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:32.063 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:32.056+0000 7f8edf7fe700 1 -- 172.21.15.42:0/3033064980 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 308 v308) v1 ==== 95+0+2116 (secure 0 0 0) 0x7f8ee8031d20 con 0x7f8ef011a880 2019-12-05T06:24:32.064 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 308 2019-12-05T06:24:32.064 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:24:32.064 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":308,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7568,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/2324657302","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2324657302},{"type":"v1","addr":"172.21.15.40:6833","nonce":2324657302}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":308}],"filesystems":[{"mdsmap":{"epoch":307,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:23:50.178163+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":104,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10993},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10993":{"gid":10993,"name":"a","rank":0,"incarnation":304,"state":"up:active","state_seq":33,"addr":"172.21.15.42:6835/3964426552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3964426552},{"type":"v1","addr":"172.21.15.42:6835","nonce":3964426552}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:24:32.066 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:32.060+0000 7f8ef4c90700 1 -- 172.21.15.42:0/3033064980 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8ed403acc0 msgr2=0x7f8ed403d1d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:32.066 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:32.060+0000 7f8ef4c90700 1 --2- 172.21.15.42:0/3033064980 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8ed403acc0 0x7f8ed403d1d0 crc :-1 s=READY pgs=1082 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:32.066 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:32.060+0000 7f8ef4c90700 1 -- 172.21.15.42:0/3033064980 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8ef011a880 msgr2=0x7f8ef0135480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:32.067 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:32.060+0000 7f8ef4c90700 1 --2- 172.21.15.42:0/3033064980 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8ef011a880 0x7f8ef0135480 secure :-1 s=READY pgs=2606 cs=0 l=1 rx=0x7f8ee0001420 tx=0x7f8ee8010cb0).stop 2019-12-05T06:24:32.067 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:32.060+0000 7f8ef4c90700 1 -- 172.21.15.42:0/3033064980 shutdown_connections 2019-12-05T06:24:32.067 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:32.060+0000 7f8ef4c90700 1 --2- 172.21.15.42:0/3033064980 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8ed403acc0 0x7f8ed403d1d0 unknown :-1 s=CLOSED pgs=1082 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:32.067 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:32.060+0000 7f8ef4c90700 1 --2- 172.21.15.42:0/3033064980 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8ef011a880 0x7f8ef0135480 unknown :-1 s=CLOSED pgs=2606 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:32.067 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:32.060+0000 7f8ef4c90700 1 --2- 172.21.15.42:0/3033064980 >> v2:172.21.15.40:3300/0 conn(0x7f8ef007cee0 0x7f8ef011a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:32.068 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:32.060+0000 7f8ef4c90700 1 -- 172.21.15.42:0/3033064980 shutdown_connections 2019-12-05T06:24:32.068 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:32.060+0000 7f8ef4c90700 1 -- 172.21.15.42:0/3033064980 wait complete. 2019-12-05T06:24:32.068 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:32.060+0000 7f8ef4c90700 1 -- 172.21.15.42:0/3033064980 >> 172.21.15.42:0/3033064980 conn(0x7f8ef00a82a0 msgr2=0x7f8ef0059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:24:32.076 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:24:34.077 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:24:34.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:34.171+0000 7f1542697700 1 Processor -- start 2019-12-05T06:24:34.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:34.171+0000 7f1542697700 1 -- start start 2019-12-05T06:24:34.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:34.171+0000 7f1542697700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f153c09fae0 0x7f153c14fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:24:34.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:34.171+0000 7f1542697700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f153c0a0710 con 0x7f153c0a7740 2019-12-05T06:24:34.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:34.171+0000 7f1542697700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f153c07ef20 con 0x7f153c08df80 2019-12-05T06:24:34.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:34.171+0000 7f1542697700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f153c0a0b00 con 0x7f153c09fae0 2019-12-05T06:24:34.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:34.171+0000 7f1541435700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f153c09fae0 0x7f153c14fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:34.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:34.171+0000 7f1541435700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f153c09fae0 0x7f153c14fff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:54860/0 (socket says 172.21.15.42:54860) 2019-12-05T06:24:34.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:34.171+0000 7f1541435700 1 -- 172.21.15.42:0/1929459100 learned_addr learned my addr 172.21.15.42:0/1929459100 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:24:34.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:34.171+0000 7f1533fff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f153c08df80 0x7f153c09f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:54896/0 (socket says 172.21.15.42:54896) 2019-12-05T06:24:34.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:34.171+0000 7f15337fe700 1 -- 172.21.15.42:0/1929459100 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1639915723 0 0) 0x7f153c0a0710 con 0x7f153c0a7740 2019-12-05T06:24:34.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:34.171+0000 7f1541435700 1 -- 172.21.15.42:0/1929459100 >> v1:172.21.15.42:6789/0 conn(0x7f153c08df80 legacy=0x7f153c09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:34.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:34.171+0000 7f1541435700 1 -- 172.21.15.42:0/1929459100 >> v1:172.21.15.40:6789/0 conn(0x7f153c0a7740 legacy=0x7f153c07e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:34.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:34.171+0000 7f1541435700 1 -- 172.21.15.42:0/1929459100 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f153c0a9ba0 con 0x7f153c09fae0 2019-12-05T06:24:34.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:34.171+0000 7f1541435700 1 --2- 172.21.15.42:0/1929459100 >> v2:172.21.15.42:3300/0 conn(0x7f153c09fae0 0x7f153c14fff0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f1534008b10 tx=0x7f1534010030).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:34.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:34.171+0000 7f1541435700 1 --2- 172.21.15.42:0/1929459100 >> v2:172.21.15.42:3300/0 conn(0x7f153c09fae0 0x7f153c14fff0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f1534008b10 tx=0x7f1534010030).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:34.193 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:34.171+0000 7f1541435700 1 --2- 172.21.15.42:0/1929459100 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f153c09fae0 0x7f153c14fff0 secure :-1 s=READY pgs=2608 cs=0 l=1 rx=0x7f1534008b10 tx=0x7f1534010030).ready entity=mon.1 client_cookie=8a4b78d398a78109 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:24:34.193 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:34.175+0000 7f1541435700 1 --2- 172.21.15.42:0/1929459100 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f153c09fae0 0x7f153c14fff0 secure :-1 s=THROTTLE_DONE pgs=2608 cs=0 l=1 rx=0x7f1534008b10 tx=0x7f1534010030).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:34.193 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:34.175+0000 7f15337fe700 1 -- 172.21.15.42:0/1929459100 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f15340110c0 con 0x7f153c09fae0 2019-12-05T06:24:34.193 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:34.175+0000 7f1541435700 1 --2- 172.21.15.42:0/1929459100 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f153c09fae0 0x7f153c14fff0 secure :-1 s=THROTTLE_DONE pgs=2608 cs=0 l=1 rx=0x7f1534008b10 tx=0x7f1534010030).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:34.193 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:34.175+0000 7f1541435700 1 --2- 172.21.15.42:0/1929459100 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f153c09fae0 0x7f153c14fff0 secure :-1 s=THROTTLE_DONE pgs=2608 cs=0 l=1 rx=0x7f1534008b10 tx=0x7f1534010030).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:34.193 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:34.175+0000 7f15337fe700 1 -- 172.21.15.42:0/1929459100 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1534010a30 con 0x7f153c09fae0 2019-12-05T06:24:34.193 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:34.175+0000 7f15337fe700 1 -- 172.21.15.42:0/1929459100 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f15340114b0 con 0x7f153c09fae0 2019-12-05T06:24:34.194 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:34.175+0000 7f1542697700 1 -- 172.21.15.42:0/1929459100 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f153c09fae0 msgr2=0x7f153c14fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:34.194 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:34.175+0000 7f1542697700 1 --2- 172.21.15.42:0/1929459100 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f153c09fae0 0x7f153c14fff0 secure :-1 s=READY pgs=2608 cs=0 l=1 rx=0x7f1534008b10 tx=0x7f1534010030).stop 2019-12-05T06:24:34.194 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:34.175+0000 7f1542697700 1 -- 172.21.15.42:0/1929459100 shutdown_connections 2019-12-05T06:24:34.194 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:34.175+0000 7f1542697700 1 --2- 172.21.15.42:0/1929459100 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f153c09fae0 0x7f153c14fff0 unknown :-1 s=CLOSED pgs=2608 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:34.194 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:34.175+0000 7f1542697700 1 -- 172.21.15.42:0/1929459100 shutdown_connections 2019-12-05T06:24:34.194 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:34.175+0000 7f1542697700 1 -- 172.21.15.42:0/1929459100 wait complete. 2019-12-05T06:24:34.194 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:34.175+0000 7f1542697700 1 -- 172.21.15.42:0/1929459100 >> 172.21.15.42:0/1929459100 conn(0x7f153c0aabe0 msgr2=0x7f153c0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:24:34.195 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:34.179+0000 7f1542697700 1 Processor -- start 2019-12-05T06:24:34.195 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:34.179+0000 7f1542697700 1 -- start start 2019-12-05T06:24:34.195 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:34.179+0000 7f1542697700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f153c11a8d0 0x7f153c0a48e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:24:34.195 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:34.179+0000 7f1542697700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f153c07c9c0 0x7f153c07cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:24:34.195 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:34.179+0000 7f1542697700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f153c121b10 con 0x7f153c07e830 2019-12-05T06:24:34.195 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:34.179+0000 7f1542697700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f153c0a4f10 con 0x7f153c07c9c0 2019-12-05T06:24:34.196 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:34.179+0000 7f1542697700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f153c0a51b0 con 0x7f153c11a8d0 2019-12-05T06:24:34.196 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:34.179+0000 7f1540c34700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f153c07e830 0x7f153c1231e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:54900/0 (socket says 172.21.15.42:54900) 2019-12-05T06:24:34.196 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:34.179+0000 7f1533fff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f153c11a8d0 0x7f153c0a48e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:34.196 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:34.179+0000 7f1540c34700 1 -- 172.21.15.42:0/333672940 learned_addr learned my addr 172.21.15.42:0/333672940 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:24:34.196 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:34.179+0000 7f1541435700 1 --2- 172.21.15.42:0/333672940 >> v2:172.21.15.40:3300/0 conn(0x7f153c07c9c0 0x7f153c07cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:34.196 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:34.179+0000 7f15317fa700 1 -- 172.21.15.42:0/333672940 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4181044683 0 0) 0x7f153c121b10 con 0x7f153c07e830 2019-12-05T06:24:34.197 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:34.179+0000 7f15317fa700 1 -- 172.21.15.42:0/333672940 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f151c0030b0 con 0x7f153c07e830 2019-12-05T06:24:34.197 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:34.179+0000 7f1533fff700 1 -- 172.21.15.42:0/333672940 >> v2:172.21.15.40:3300/0 conn(0x7f153c07c9c0 msgr2=0x7f153c07cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:34.197 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:34.179+0000 7f1533fff700 1 --2- 172.21.15.42:0/333672940 >> v2:172.21.15.40:3300/0 conn(0x7f153c07c9c0 0x7f153c07cdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:34.197 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:34.179+0000 7f1533fff700 1 -- 172.21.15.42:0/333672940 >> v1:172.21.15.42:6789/0 conn(0x7f153c07e830 legacy=0x7f153c1231e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:34.197 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:34.179+0000 7f1533fff700 1 -- 172.21.15.42:0/333672940 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f153c1239e0 con 0x7f153c11a8d0 2019-12-05T06:24:34.197 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:34.179+0000 7f1541435700 1 --2- 172.21.15.42:0/333672940 >> v2:172.21.15.40:3300/0 conn(0x7f153c07c9c0 0x7f153c07cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:24:34.198 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:34.179+0000 7f1533fff700 1 --2- 172.21.15.42:0/333672940 >> v2:172.21.15.42:3300/0 conn(0x7f153c11a8d0 0x7f153c0a48e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f152c000e90 tx=0x7f152c012a40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:34.198 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:34.179+0000 7f1533fff700 1 --2- 172.21.15.42:0/333672940 >> v2:172.21.15.42:3300/0 conn(0x7f153c11a8d0 0x7f153c0a48e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f152c000e90 tx=0x7f152c012a40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:34.198 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:34.179+0000 7f1533fff700 1 --2- 172.21.15.42:0/333672940 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f153c11a8d0 0x7f153c0a48e0 secure :-1 s=READY pgs=2610 cs=0 l=1 rx=0x7f152c000e90 tx=0x7f152c012a40).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:24:34.198 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:34.183+0000 7f1533fff700 1 --2- 172.21.15.42:0/333672940 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f153c11a8d0 0x7f153c0a48e0 secure :-1 s=THROTTLE_DONE pgs=2610 cs=0 l=1 rx=0x7f152c000e90 tx=0x7f152c012a40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:34.198 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:34.183+0000 7f15317fa700 1 -- 172.21.15.42:0/333672940 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f152c00d0c0 con 0x7f153c11a8d0 2019-12-05T06:24:34.198 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:34.183+0000 7f1533fff700 1 --2- 172.21.15.42:0/333672940 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f153c11a8d0 0x7f153c0a48e0 secure :-1 s=THROTTLE_DONE pgs=2610 cs=0 l=1 rx=0x7f152c000e90 tx=0x7f152c012a40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:34.199 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:34.183+0000 7f1533fff700 1 --2- 172.21.15.42:0/333672940 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f153c11a8d0 0x7f153c0a48e0 secure :-1 s=THROTTLE_DONE pgs=2610 cs=0 l=1 rx=0x7f152c000e90 tx=0x7f152c012a40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:34.199 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:34.183+0000 7f15317fa700 1 -- 172.21.15.42:0/333672940 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f152c013390 con 0x7f153c11a8d0 2019-12-05T06:24:34.199 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:34.183+0000 7f15317fa700 1 -- 172.21.15.42:0/333672940 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f152c00d6a0 con 0x7f153c11a8d0 2019-12-05T06:24:34.199 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:34.183+0000 7f1542697700 1 -- 172.21.15.42:0/333672940 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f153c07d340 con 0x7f153c11a8d0 2019-12-05T06:24:34.199 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:34.183+0000 7f1542697700 1 -- 172.21.15.42:0/333672940 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f153c111db0 con 0x7f153c11a8d0 2019-12-05T06:24:34.201 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:34.183+0000 7f1533fff700 1 --2- 172.21.15.42:0/333672940 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f153c11a8d0 0x7f153c0a48e0 secure :-1 s=THROTTLE_DONE pgs=2610 cs=0 l=1 rx=0x7f152c000e90 tx=0x7f152c012a40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:34.201 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:34.183+0000 7f15317fa700 1 -- 172.21.15.42:0/333672940 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f152c00d9b0 con 0x7f153c11a8d0 2019-12-05T06:24:34.201 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:34.183+0000 7f1533fff700 1 --2- 172.21.15.42:0/333672940 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f153c11a8d0 0x7f153c0a48e0 secure :-1 s=THROTTLE_DONE pgs=2610 cs=0 l=1 rx=0x7f152c000e90 tx=0x7f152c012a40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:34.201 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:34.183+0000 7f15317fa700 1 --2- 172.21.15.42:0/333672940 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f151c03b2d0 0x7f151c03d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:24:34.201 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:34.183+0000 7f15317fa700 1 -- 172.21.15.42:0/333672940 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 10761+0+0 (secure 0 0 0) 0x7f152c04cfa0 con 0x7f153c11a8d0 2019-12-05T06:24:34.201 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:34.183+0000 7f1540c34700 1 --2- 172.21.15.42:0/333672940 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f151c03b2d0 0x7f151c03d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:34.202 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:34.183+0000 7f1540c34700 1 --2- 172.21.15.42:0/333672940 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f151c03b2d0 0x7f151c03d7e0 crc :-1 s=READY pgs=1083 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:24:34.202 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:34.187+0000 7f1542697700 1 -- 172.21.15.42:0/333672940 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f153c05d110 con 0x7f153c11a8d0 2019-12-05T06:24:34.202 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:34.191+0000 7f1533fff700 1 --2- 172.21.15.42:0/333672940 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f153c11a8d0 0x7f153c0a48e0 secure :-1 s=THROTTLE_DONE pgs=2610 cs=0 l=1 rx=0x7f152c000e90 tx=0x7f152c012a40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:34.202 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:34.195+0000 7f15317fa700 1 -- 172.21.15.42:0/333672940 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f152c034020 con 0x7f153c11a8d0 2019-12-05T06:24:34.374 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:34.367+0000 7f1542697700 1 -- 172.21.15.42:0/333672940 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f153c05b8c0 con 0x7f153c11a8d0 2019-12-05T06:24:34.375 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:34.367+0000 7f1533fff700 1 --2- 172.21.15.42:0/333672940 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f153c11a8d0 0x7f153c0a48e0 secure :-1 s=THROTTLE_DONE pgs=2610 cs=0 l=1 rx=0x7f152c000e90 tx=0x7f152c012a40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:34.375 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:34.367+0000 7f15317fa700 1 -- 172.21.15.42:0/333672940 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 308 v308) v1 ==== 95+0+2116 (secure 0 0 0) 0x7f152c034280 con 0x7f153c11a8d0 2019-12-05T06:24:34.377 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 308 2019-12-05T06:24:34.377 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:24:34.377 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":308,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7568,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/2324657302","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2324657302},{"type":"v1","addr":"172.21.15.40:6833","nonce":2324657302}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":308}],"filesystems":[{"mdsmap":{"epoch":307,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:23:50.178163+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":104,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10993},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10993":{"gid":10993,"name":"a","rank":0,"incarnation":304,"state":"up:active","state_seq":33,"addr":"172.21.15.42:6835/3964426552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3964426552},{"type":"v1","addr":"172.21.15.42:6835","nonce":3964426552}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:24:34.380 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:34.371+0000 7f1542697700 1 -- 172.21.15.42:0/333672940 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f151c03b2d0 msgr2=0x7f151c03d7e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:34.380 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:34.371+0000 7f1542697700 1 --2- 172.21.15.42:0/333672940 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f151c03b2d0 0x7f151c03d7e0 crc :-1 s=READY pgs=1083 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:34.380 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:34.371+0000 7f1542697700 1 -- 172.21.15.42:0/333672940 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f153c11a8d0 msgr2=0x7f153c0a48e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:34.381 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:34.371+0000 7f1542697700 1 --2- 172.21.15.42:0/333672940 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f153c11a8d0 0x7f153c0a48e0 secure :-1 s=READY pgs=2610 cs=0 l=1 rx=0x7f152c000e90 tx=0x7f152c012a40).stop 2019-12-05T06:24:34.381 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:34.371+0000 7f1542697700 1 -- 172.21.15.42:0/333672940 shutdown_connections 2019-12-05T06:24:34.381 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:34.371+0000 7f1542697700 1 --2- 172.21.15.42:0/333672940 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f151c03b2d0 0x7f151c03d7e0 unknown :-1 s=CLOSED pgs=1083 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:34.381 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:34.371+0000 7f1542697700 1 --2- 172.21.15.42:0/333672940 >> v2:172.21.15.40:3300/0 conn(0x7f153c07c9c0 0x7f153c07cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:34.381 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:34.371+0000 7f1542697700 1 --2- 172.21.15.42:0/333672940 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f153c11a8d0 0x7f153c0a48e0 unknown :-1 s=CLOSED pgs=2610 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:34.381 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:34.371+0000 7f1542697700 1 -- 172.21.15.42:0/333672940 shutdown_connections 2019-12-05T06:24:34.382 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:34.371+0000 7f1542697700 1 -- 172.21.15.42:0/333672940 wait complete. 2019-12-05T06:24:34.382 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:34.371+0000 7f1542697700 1 -- 172.21.15.42:0/333672940 >> 172.21.15.42:0/333672940 conn(0x7f153c0a7f90 msgr2=0x7f153c059ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:24:34.391 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:24:36.392 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:24:36.499 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.487+0000 7f9e1a8d6700 1 Processor -- start 2019-12-05T06:24:36.499 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.487+0000 7f9e1a8d6700 1 -- start start 2019-12-05T06:24:36.499 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.487+0000 7f9e1a8d6700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f9e140a7730 0x7f9e1408df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:24:36.499 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.487+0000 7f9e1a8d6700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e140a0700 con 0x7f9e1409fad0 2019-12-05T06:24:36.499 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.487+0000 7f9e1a8d6700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e140a9b90 con 0x7f9e1407e710 2019-12-05T06:24:36.499 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.487+0000 7f9e1a8d6700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e140a0af0 con 0x7f9e140a7730 2019-12-05T06:24:36.500 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.487+0000 7f9e13fff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f9e1407e710 0x7f9e1409f2d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:54910/0 (socket says 172.21.15.42:54910) 2019-12-05T06:24:36.500 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.487+0000 7f9e13fff700 1 -- 172.21.15.42:0/4156290275 learned_addr learned my addr 172.21.15.42:0/4156290275 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:24:36.500 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.487+0000 7f9e18e73700 1 --2- 172.21.15.42:0/4156290275 >> v2:172.21.15.40:3300/0 conn(0x7f9e140a7730 0x7f9e1408df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:36.500 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.487+0000 7f9e137fe700 1 -- 172.21.15.42:0/4156290275 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 323362767 0 0) 0x7f9e140a9b90 con 0x7f9e1407e710 2019-12-05T06:24:36.500 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.487+0000 7f9e137fe700 1 -- 172.21.15.42:0/4156290275 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e000030b0 con 0x7f9e1407e710 2019-12-05T06:24:36.500 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.491+0000 7f9e18e73700 1 -- 172.21.15.42:0/4156290275 >> v1:172.21.15.42:6789/0 conn(0x7f9e1407e710 legacy=0x7f9e1409f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:36.501 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.491+0000 7f9e18e73700 1 -- 172.21.15.42:0/4156290275 >> v1:172.21.15.40:6789/0 conn(0x7f9e1409fad0 legacy=0x7f9e1414efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:36.501 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.491+0000 7f9e18e73700 1 -- 172.21.15.42:0/4156290275 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e1407eb40 con 0x7f9e140a7730 2019-12-05T06:24:36.501 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.491+0000 7f9e18e73700 1 --2- 172.21.15.42:0/4156290275 >> v2:172.21.15.40:3300/0 conn(0x7f9e140a7730 0x7f9e1408df70 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f9dfc008790 tx=0x7f9dfc00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:36.501 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.491+0000 7f9e18e73700 1 --2- 172.21.15.42:0/4156290275 >> v2:172.21.15.40:3300/0 conn(0x7f9e140a7730 0x7f9e1408df70 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f9dfc008790 tx=0x7f9dfc00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:36.501 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.491+0000 7f9e18e73700 1 --2- 172.21.15.42:0/4156290275 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9e140a7730 0x7f9e1408df70 secure :-1 s=READY pgs=1948 cs=0 l=1 rx=0x7f9dfc008790 tx=0x7f9dfc00fbf0).ready entity=mon.0 client_cookie=eeaffe058de0050f server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:24:36.501 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.491+0000 7f9e18e73700 1 --2- 172.21.15.42:0/4156290275 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9e140a7730 0x7f9e1408df70 secure :-1 s=THROTTLE_DONE pgs=1948 cs=0 l=1 rx=0x7f9dfc008790 tx=0x7f9dfc00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:36.502 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.491+0000 7f9e137fe700 1 -- 172.21.15.42:0/4156290275 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9dfc011270 con 0x7f9e140a7730 2019-12-05T06:24:36.502 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.491+0000 7f9e18e73700 1 --2- 172.21.15.42:0/4156290275 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9e140a7730 0x7f9e1408df70 secure :-1 s=THROTTLE_DONE pgs=1948 cs=0 l=1 rx=0x7f9dfc008790 tx=0x7f9dfc00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:36.502 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.491+0000 7f9e18e73700 1 --2- 172.21.15.42:0/4156290275 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9e140a7730 0x7f9e1408df70 secure :-1 s=THROTTLE_DONE pgs=1948 cs=0 l=1 rx=0x7f9dfc008790 tx=0x7f9dfc00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:36.502 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.491+0000 7f9e137fe700 1 -- 172.21.15.42:0/4156290275 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9dfc008bc0 con 0x7f9e140a7730 2019-12-05T06:24:36.503 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.491+0000 7f9e137fe700 1 -- 172.21.15.42:0/4156290275 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9dfc011850 con 0x7f9e140a7730 2019-12-05T06:24:36.503 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.491+0000 7f9e1a8d6700 1 -- 172.21.15.42:0/4156290275 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9e140a7730 msgr2=0x7f9e1408df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:36.503 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.491+0000 7f9e1a8d6700 1 --2- 172.21.15.42:0/4156290275 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9e140a7730 0x7f9e1408df70 secure :-1 s=READY pgs=1948 cs=0 l=1 rx=0x7f9dfc008790 tx=0x7f9dfc00fbf0).stop 2019-12-05T06:24:36.503 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.491+0000 7f9e1a8d6700 1 -- 172.21.15.42:0/4156290275 shutdown_connections 2019-12-05T06:24:36.503 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.491+0000 7f9e1a8d6700 1 --2- 172.21.15.42:0/4156290275 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9e140a7730 0x7f9e1408df70 unknown :-1 s=CLOSED pgs=1948 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:36.503 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.491+0000 7f9e1a8d6700 1 -- 172.21.15.42:0/4156290275 shutdown_connections 2019-12-05T06:24:36.504 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.491+0000 7f9e1a8d6700 1 -- 172.21.15.42:0/4156290275 wait complete. 2019-12-05T06:24:36.504 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.491+0000 7f9e1a8d6700 1 -- 172.21.15.42:0/4156290275 >> 172.21.15.42:0/4156290275 conn(0x7f9e140aabd0 msgr2=0x7f9e140a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:24:36.508 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.495+0000 7f9e1a8d6700 1 Processor -- start 2019-12-05T06:24:36.508 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.495+0000 7f9e1a8d6700 1 -- start start 2019-12-05T06:24:36.508 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.495+0000 7f9e1a8d6700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f9e14142ce0 0x7f9e14144720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:24:36.508 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.495+0000 7f9e1a8d6700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9e14122190 0x7f9e141225c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:24:36.508 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.495+0000 7f9e1a8d6700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e14144c70 con 0x7f9e1407b140 2019-12-05T06:24:36.509 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.495+0000 7f9e1a8d6700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e14144f50 con 0x7f9e14142ce0 2019-12-05T06:24:36.509 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.495+0000 7f9e1a8d6700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e141451f0 con 0x7f9e14122190 2019-12-05T06:24:36.509 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.495+0000 7f9e18e73700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f9e14142ce0 0x7f9e14144720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:36.509 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.495+0000 7f9e18e73700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f9e14142ce0 0x7f9e14144720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:54876/0 (socket says 172.21.15.42:54876) 2019-12-05T06:24:36.509 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.495+0000 7f9e18e73700 1 -- 172.21.15.42:0/2909919729 learned_addr learned my addr 172.21.15.42:0/2909919729 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:24:36.509 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.495+0000 7f9e19674700 1 --2- 172.21.15.42:0/2909919729 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9e14122190 0x7f9e141225c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:36.509 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.495+0000 7f9e18e73700 1 -- 172.21.15.42:0/2909919729 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9e14122190 msgr2=0x7f9e141225c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:36.510 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.495+0000 7f9e18e73700 1 --2- 172.21.15.42:0/2909919729 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9e14122190 0x7f9e141225c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:36.510 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.495+0000 7f9e18e73700 1 -- 172.21.15.42:0/2909919729 >> v1:172.21.15.40:6789/0 conn(0x7f9e1407b140 legacy=0x7f9e1407b570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:36.510 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.495+0000 7f9e18e73700 1 -- 172.21.15.42:0/2909919729 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e1414ff00 con 0x7f9e14142ce0 2019-12-05T06:24:36.510 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.495+0000 7f9e18e73700 1 --2- 172.21.15.42:0/2909919729 >> v2:172.21.15.42:3300/0 conn(0x7f9e14142ce0 0x7f9e14144720 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f9e140aaf60 tx=0x7f9dfc008ea0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:36.510 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.495+0000 7f9e18e73700 1 --2- 172.21.15.42:0/2909919729 >> v2:172.21.15.42:3300/0 conn(0x7f9e14142ce0 0x7f9e14144720 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f9e140aaf60 tx=0x7f9dfc008ea0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:36.510 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.495+0000 7f9e18e73700 1 --2- 172.21.15.42:0/2909919729 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9e14142ce0 0x7f9e14144720 secure :-1 s=READY pgs=2612 cs=0 l=1 rx=0x7f9e140aaf60 tx=0x7f9dfc008ea0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:24:36.510 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.495+0000 7f9e18e73700 1 --2- 172.21.15.42:0/2909919729 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9e14142ce0 0x7f9e14144720 secure :-1 s=THROTTLE_DONE pgs=2612 cs=0 l=1 rx=0x7f9e140aaf60 tx=0x7f9dfc008ea0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:36.511 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.495+0000 7f9e117fa700 1 -- 172.21.15.42:0/2909919729 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9dfc007340 con 0x7f9e14142ce0 2019-12-05T06:24:36.511 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.499+0000 7f9e18e73700 1 --2- 172.21.15.42:0/2909919729 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9e14142ce0 0x7f9e14144720 secure :-1 s=THROTTLE_DONE pgs=2612 cs=0 l=1 rx=0x7f9e140aaf60 tx=0x7f9dfc008ea0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:36.511 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.499+0000 7f9e1a8d6700 1 -- 172.21.15.42:0/2909919729 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e14122b10 con 0x7f9e14142ce0 2019-12-05T06:24:36.511 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.499+0000 7f9e117fa700 1 -- 172.21.15.42:0/2909919729 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9dfc011470 con 0x7f9e14142ce0 2019-12-05T06:24:36.511 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.499+0000 7f9e18e73700 1 --2- 172.21.15.42:0/2909919729 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9e14142ce0 0x7f9e14144720 secure :-1 s=THROTTLE_DONE pgs=2612 cs=0 l=1 rx=0x7f9e140aaf60 tx=0x7f9dfc008ea0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:36.511 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.499+0000 7f9e117fa700 1 -- 172.21.15.42:0/2909919729 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9dfc005540 con 0x7f9e14142ce0 2019-12-05T06:24:36.511 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.499+0000 7f9e1a8d6700 1 -- 172.21.15.42:0/2909919729 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e1412cb00 con 0x7f9e14142ce0 2019-12-05T06:24:36.512 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.499+0000 7f9e18e73700 1 --2- 172.21.15.42:0/2909919729 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9e14142ce0 0x7f9e14144720 secure :-1 s=THROTTLE_DONE pgs=2612 cs=0 l=1 rx=0x7f9e140aaf60 tx=0x7f9dfc008ea0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:36.512 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.499+0000 7f9e117fa700 1 -- 172.21.15.42:0/2909919729 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f9dfc005820 con 0x7f9e14142ce0 2019-12-05T06:24:36.512 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.499+0000 7f9e18e73700 1 --2- 172.21.15.42:0/2909919729 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9e14142ce0 0x7f9e14144720 secure :-1 s=THROTTLE_DONE pgs=2612 cs=0 l=1 rx=0x7f9e140aaf60 tx=0x7f9dfc008ea0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:36.512 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.499+0000 7f9e117fa700 1 --2- 172.21.15.42:0/2909919729 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9df803ad20 0x7f9df803d230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:24:36.512 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.499+0000 7f9e117fa700 1 -- 172.21.15.42:0/2909919729 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 10847+0+0 (secure 0 0 0) 0x7f9dfc04bbb0 con 0x7f9e14142ce0 2019-12-05T06:24:36.513 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.499+0000 7f9e13fff700 1 --2- 172.21.15.42:0/2909919729 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9df803ad20 0x7f9df803d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:36.513 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.499+0000 7f9e13fff700 1 --2- 172.21.15.42:0/2909919729 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9df803ad20 0x7f9df803d230 crc :-1 s=READY pgs=1084 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:24:36.513 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.503+0000 7f9e1a8d6700 1 -- 172.21.15.42:0/2909919729 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e14144510 con 0x7f9e14142ce0 2019-12-05T06:24:36.516 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.511+0000 7f9e18e73700 1 --2- 172.21.15.42:0/2909919729 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9e14142ce0 0x7f9e14144720 secure :-1 s=THROTTLE_DONE pgs=2612 cs=0 l=1 rx=0x7f9e140aaf60 tx=0x7f9dfc008ea0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:36.516 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.511+0000 7f9e117fa700 1 -- 172.21.15.42:0/2909919729 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f9dfc030840 con 0x7f9e14142ce0 2019-12-05T06:24:36.691 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.683+0000 7f9e1a8d6700 1 -- 172.21.15.42:0/2909919729 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9e1405d100 con 0x7f9e14142ce0 2019-12-05T06:24:36.691 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.683+0000 7f9e18e73700 1 --2- 172.21.15.42:0/2909919729 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9e14142ce0 0x7f9e14144720 secure :-1 s=THROTTLE_DONE pgs=2612 cs=0 l=1 rx=0x7f9e140aaf60 tx=0x7f9dfc008ea0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:36.691 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.683+0000 7f9e117fa700 1 -- 172.21.15.42:0/2909919729 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 309 v309) v1 ==== 95+0+1775 (secure 0 0 0) 0x7f9dfc005b30 con 0x7f9e14142ce0 2019-12-05T06:24:36.691 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 309 2019-12-05T06:24:36.691 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:24:36.692 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":309,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":309,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:24:34.852514+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":105,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7568},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7568":{"gid":7568,"name":"b","rank":0,"incarnation":309,"state":"up:replay","state_seq":2,"addr":"172.21.15.40:6833/2324657302","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2324657302},{"type":"v1","addr":"172.21.15.40:6833","nonce":2324657302}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:24:36.693 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.687+0000 7f9e1a8d6700 1 -- 172.21.15.42:0/2909919729 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9df803ad20 msgr2=0x7f9df803d230 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:36.693 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.687+0000 7f9e1a8d6700 1 --2- 172.21.15.42:0/2909919729 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9df803ad20 0x7f9df803d230 crc :-1 s=READY pgs=1084 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:36.693 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.687+0000 7f9e1a8d6700 1 -- 172.21.15.42:0/2909919729 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9e14142ce0 msgr2=0x7f9e14144720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:36.693 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.687+0000 7f9e1a8d6700 1 --2- 172.21.15.42:0/2909919729 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9e14142ce0 0x7f9e14144720 secure :-1 s=READY pgs=2612 cs=0 l=1 rx=0x7f9e140aaf60 tx=0x7f9dfc008ea0).stop 2019-12-05T06:24:36.693 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.687+0000 7f9e1a8d6700 1 -- 172.21.15.42:0/2909919729 shutdown_connections 2019-12-05T06:24:36.694 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.687+0000 7f9e1a8d6700 1 --2- 172.21.15.42:0/2909919729 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9df803ad20 0x7f9df803d230 unknown :-1 s=CLOSED pgs=1084 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:36.694 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.687+0000 7f9e1a8d6700 1 --2- 172.21.15.42:0/2909919729 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9e14122190 0x7f9e141225c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:36.694 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.687+0000 7f9e1a8d6700 1 --2- 172.21.15.42:0/2909919729 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9e14142ce0 0x7f9e14144720 unknown :-1 s=CLOSED pgs=2612 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:36.694 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.687+0000 7f9e1a8d6700 1 -- 172.21.15.42:0/2909919729 shutdown_connections 2019-12-05T06:24:36.694 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.687+0000 7f9e1a8d6700 1 -- 172.21.15.42:0/2909919729 wait complete. 2019-12-05T06:24:36.694 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.687+0000 7f9e1a8d6700 1 -- 172.21.15.42:0/2909919729 >> 172.21.15.42:0/2909919729 conn(0x7f9e1407dd30 msgr2=0x7f9e140596f0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:24:36.703 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a down, removed from mdsmap 2019-12-05T06:24:36.703 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2019-12-05T06:24:36.704 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:24:36.806 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.795+0000 7f3134b57700 1 Processor -- start 2019-12-05T06:24:36.806 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.795+0000 7f3134b57700 1 -- start start 2019-12-05T06:24:36.806 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.795+0000 7f3134b57700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f313008df80 0x7f313009f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:24:36.807 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.795+0000 7f3134b57700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f313009fc60 con 0x7f313009f830 2019-12-05T06:24:36.807 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.795+0000 7f3134b57700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31300a0710 con 0x7f31300a7740 2019-12-05T06:24:36.807 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.795+0000 7f3134b57700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31300a0b00 con 0x7f313008df80 2019-12-05T06:24:36.807 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.795+0000 7f312ed9d700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f31300a7740 0x7f313007e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:54922/0 (socket says 172.21.15.42:54922) 2019-12-05T06:24:36.807 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.795+0000 7f312ed9d700 1 -- 172.21.15.42:0/1070922506 learned_addr learned my addr 172.21.15.42:0/1070922506 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:24:36.807 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.795+0000 7f312e59c700 1 --2- 172.21.15.42:0/1070922506 >> v2:172.21.15.40:3300/0 conn(0x7f313008df80 0x7f313009f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:36.808 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.795+0000 7f312dd9b700 1 -- 172.21.15.42:0/1070922506 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3881146474 0 0) 0x7f31300a0710 con 0x7f31300a7740 2019-12-05T06:24:36.808 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.795+0000 7f312dd9b700 1 -- 172.21.15.42:0/1070922506 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f311c0030b0 con 0x7f31300a7740 2019-12-05T06:24:36.808 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.795+0000 7f312e59c700 1 -- 172.21.15.42:0/1070922506 >> v1:172.21.15.42:6789/0 conn(0x7f31300a7740 legacy=0x7f313007e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:36.808 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.795+0000 7f312e59c700 1 -- 172.21.15.42:0/1070922506 >> v1:172.21.15.40:6789/0 conn(0x7f313009f830 legacy=0x7f313014efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:36.808 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.795+0000 7f312e59c700 1 -- 172.21.15.42:0/1070922506 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f313007ef20 con 0x7f313008df80 2019-12-05T06:24:36.808 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.795+0000 7f312e59c700 1 --2- 172.21.15.42:0/1070922506 >> v2:172.21.15.40:3300/0 conn(0x7f313008df80 0x7f313009f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f3120008b10 tx=0x7f312000ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:36.809 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.795+0000 7f312e59c700 1 --2- 172.21.15.42:0/1070922506 >> v2:172.21.15.40:3300/0 conn(0x7f313008df80 0x7f313009f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f3120008b10 tx=0x7f312000ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:36.809 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.795+0000 7f312e59c700 1 --2- 172.21.15.42:0/1070922506 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f313008df80 0x7f313009f2e0 secure :-1 s=READY pgs=1951 cs=0 l=1 rx=0x7f3120008b10 tx=0x7f312000ff70).ready entity=mon.0 client_cookie=1c45c2175e7c7ff2 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:24:36.809 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.795+0000 7f312e59c700 1 --2- 172.21.15.42:0/1070922506 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f313008df80 0x7f313009f2e0 secure :-1 s=THROTTLE_DONE pgs=1951 cs=0 l=1 rx=0x7f3120008b10 tx=0x7f312000ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:36.809 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.795+0000 7f312dd9b700 1 -- 172.21.15.42:0/1070922506 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f31200119d0 con 0x7f313008df80 2019-12-05T06:24:36.809 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.795+0000 7f312e59c700 1 --2- 172.21.15.42:0/1070922506 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f313008df80 0x7f313009f2e0 secure :-1 s=THROTTLE_DONE pgs=1951 cs=0 l=1 rx=0x7f3120008b10 tx=0x7f312000ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:36.809 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.795+0000 7f312e59c700 1 --2- 172.21.15.42:0/1070922506 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f313008df80 0x7f313009f2e0 secure :-1 s=THROTTLE_DONE pgs=1951 cs=0 l=1 rx=0x7f3120008b10 tx=0x7f312000ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:36.809 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.795+0000 7f312dd9b700 1 -- 172.21.15.42:0/1070922506 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31200115f0 con 0x7f313008df80 2019-12-05T06:24:36.810 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.795+0000 7f312dd9b700 1 -- 172.21.15.42:0/1070922506 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f312000a160 con 0x7f313008df80 2019-12-05T06:24:36.810 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.795+0000 7f3134b57700 1 -- 172.21.15.42:0/1070922506 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f313008df80 msgr2=0x7f313009f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:36.810 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.795+0000 7f3134b57700 1 --2- 172.21.15.42:0/1070922506 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f313008df80 0x7f313009f2e0 secure :-1 s=READY pgs=1951 cs=0 l=1 rx=0x7f3120008b10 tx=0x7f312000ff70).stop 2019-12-05T06:24:36.810 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.799+0000 7f3134b57700 1 -- 172.21.15.42:0/1070922506 shutdown_connections 2019-12-05T06:24:36.810 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.799+0000 7f3134b57700 1 --2- 172.21.15.42:0/1070922506 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f313008df80 0x7f313009f2e0 unknown :-1 s=CLOSED pgs=1951 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:36.810 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.799+0000 7f3134b57700 1 -- 172.21.15.42:0/1070922506 shutdown_connections 2019-12-05T06:24:36.810 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.799+0000 7f3134b57700 1 -- 172.21.15.42:0/1070922506 wait complete. 2019-12-05T06:24:36.811 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.799+0000 7f3134b57700 1 -- 172.21.15.42:0/1070922506 >> 172.21.15.42:0/1070922506 conn(0x7f31300aabe0 msgr2=0x7f31300a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:24:36.811 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.799+0000 7f3134b57700 1 Processor -- start 2019-12-05T06:24:36.811 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.799+0000 7f3134b57700 1 -- start start 2019-12-05T06:24:36.814 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.799+0000 7f3134b57700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f3130065540 0x7f3130128b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:24:36.814 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.799+0000 7f3134b57700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3130129080 con 0x7f3130119910 2019-12-05T06:24:36.814 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.799+0000 7f3134b57700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f313007e0a0 con 0x7f3130063840 2019-12-05T06:24:36.814 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.799+0000 7f3134b57700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31301293d0 con 0x7f3130065540 2019-12-05T06:24:36.814 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.799+0000 7f312f59e700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f3130063840 0x7f3130063c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:54928/0 (socket says 172.21.15.42:54928) 2019-12-05T06:24:36.815 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.799+0000 7f312f59e700 1 -- 172.21.15.42:0/1462075081 learned_addr learned my addr 172.21.15.42:0/1462075081 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:24:36.815 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.799+0000 7f312ed9d700 1 --2- 172.21.15.42:0/1462075081 >> v2:172.21.15.40:3300/0 conn(0x7f3130065540 0x7f3130128b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:36.815 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.799+0000 7f31277fe700 1 -- 172.21.15.42:0/1462075081 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2660162166 0 0) 0x7f313007e0a0 con 0x7f3130063840 2019-12-05T06:24:36.815 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.799+0000 7f31277fe700 1 -- 172.21.15.42:0/1462075081 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31140030b0 con 0x7f3130063840 2019-12-05T06:24:36.816 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.799+0000 7f31277fe700 1 -- 172.21.15.42:0/1462075081 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2068204341 0 0) 0x7f3130129080 con 0x7f3130119910 2019-12-05T06:24:36.816 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.799+0000 7f31277fe700 1 -- 172.21.15.42:0/1462075081 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f313007e0a0 con 0x7f3130119910 2019-12-05T06:24:36.816 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.799+0000 7f312ed9d700 1 -- 172.21.15.42:0/1462075081 >> v1:172.21.15.42:6789/0 conn(0x7f3130063840 legacy=0x7f3130063c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:36.816 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.799+0000 7f312ed9d700 1 -- 172.21.15.42:0/1462075081 >> v1:172.21.15.40:6789/0 conn(0x7f3130119910 legacy=0x7f313007d8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:36.816 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.799+0000 7f312ed9d700 1 -- 172.21.15.42:0/1462075081 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3130132fb0 con 0x7f3130065540 2019-12-05T06:24:36.816 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.799+0000 7f312ed9d700 1 --2- 172.21.15.42:0/1462075081 >> v2:172.21.15.40:3300/0 conn(0x7f3130065540 0x7f3130128b30 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f31180043a0 tx=0x7f3118010810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:36.817 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.803+0000 7f312ed9d700 1 --2- 172.21.15.42:0/1462075081 >> v2:172.21.15.40:3300/0 conn(0x7f3130065540 0x7f3130128b30 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f31180043a0 tx=0x7f3118010810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:36.817 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.803+0000 7f312ed9d700 1 --2- 172.21.15.42:0/1462075081 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3130065540 0x7f3130128b30 secure :-1 s=READY pgs=1953 cs=0 l=1 rx=0x7f31180043a0 tx=0x7f3118010810).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:24:36.817 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.803+0000 7f312ed9d700 1 --2- 172.21.15.42:0/1462075081 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3130065540 0x7f3130128b30 secure :-1 s=THROTTLE_DONE pgs=1953 cs=0 l=1 rx=0x7f31180043a0 tx=0x7f3118010810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:36.817 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.803+0000 7f31277fe700 1 -- 172.21.15.42:0/1462075081 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3118005c30 con 0x7f3130065540 2019-12-05T06:24:36.817 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.803+0000 7f312ed9d700 1 --2- 172.21.15.42:0/1462075081 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3130065540 0x7f3130128b30 secure :-1 s=THROTTLE_DONE pgs=1953 cs=0 l=1 rx=0x7f31180043a0 tx=0x7f3118010810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:36.817 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.803+0000 7f31277fe700 1 -- 172.21.15.42:0/1462075081 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3118005850 con 0x7f3130065540 2019-12-05T06:24:36.817 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.803+0000 7f312ed9d700 1 --2- 172.21.15.42:0/1462075081 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3130065540 0x7f3130128b30 secure :-1 s=THROTTLE_DONE pgs=1953 cs=0 l=1 rx=0x7f31180043a0 tx=0x7f3118010810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:36.818 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.803+0000 7f3134b57700 1 -- 172.21.15.42:0/1462075081 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3130130f90 con 0x7f3130065540 2019-12-05T06:24:36.818 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.803+0000 7f31277fe700 1 -- 172.21.15.42:0/1462075081 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f31180064a0 con 0x7f3130065540 2019-12-05T06:24:36.818 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.803+0000 7f3134b57700 1 -- 172.21.15.42:0/1462075081 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31301314c0 con 0x7f3130065540 2019-12-05T06:24:36.818 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.803+0000 7f312ed9d700 1 --2- 172.21.15.42:0/1462075081 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3130065540 0x7f3130128b30 secure :-1 s=THROTTLE_DONE pgs=1953 cs=0 l=1 rx=0x7f31180043a0 tx=0x7f3118010810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:36.818 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.803+0000 7f31277fe700 1 -- 172.21.15.42:0/1462075081 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f3118006780 con 0x7f3130065540 2019-12-05T06:24:36.818 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.803+0000 7f312ed9d700 1 --2- 172.21.15.42:0/1462075081 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3130065540 0x7f3130128b30 secure :-1 s=THROTTLE_DONE pgs=1953 cs=0 l=1 rx=0x7f31180043a0 tx=0x7f3118010810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:36.818 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.803+0000 7f31277fe700 1 --2- 172.21.15.42:0/1462075081 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f311403b390 0x7f311403d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:24:36.819 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.803+0000 7f31277fe700 1 -- 172.21.15.42:0/1462075081 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 10847+0+0 (secure 0 0 0) 0x7f311804afd0 con 0x7f3130065540 2019-12-05T06:24:36.819 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.803+0000 7f312e59c700 1 --2- 172.21.15.42:0/1462075081 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f311403b390 0x7f311403d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:36.819 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.803+0000 7f312e59c700 1 --2- 172.21.15.42:0/1462075081 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f311403b390 0x7f311403d8a0 crc :-1 s=READY pgs=1085 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:24:36.819 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.811+0000 7f3134b57700 1 -- 172.21.15.42:0/1462075081 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f313005d110 con 0x7f3130065540 2019-12-05T06:24:36.827 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.819+0000 7f312ed9d700 1 --2- 172.21.15.42:0/1462075081 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3130065540 0x7f3130128b30 secure :-1 s=THROTTLE_DONE pgs=1953 cs=0 l=1 rx=0x7f31180043a0 tx=0x7f3118010810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:36.827 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.819+0000 7f31277fe700 1 -- 172.21.15.42:0/1462075081 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f3118032050 con 0x7f3130065540 2019-12-05T06:24:37.000 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.995+0000 7f3134b57700 1 -- 172.21.15.42:0/1462075081 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f313005b8c0 con 0x7f3130065540 2019-12-05T06:24:37.001 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.995+0000 7f312ed9d700 1 --2- 172.21.15.42:0/1462075081 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3130065540 0x7f3130128b30 secure :-1 s=THROTTLE_DONE pgs=1953 cs=0 l=1 rx=0x7f31180043a0 tx=0x7f3118010810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:37.001 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.995+0000 7f31277fe700 1 -- 172.21.15.42:0/1462075081 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 309 v309) v1 ==== 95+0+1775 (secure 0 0 0) 0x7f31180322b0 con 0x7f3130065540 2019-12-05T06:24:37.002 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 309 2019-12-05T06:24:37.002 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:24:37.002 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":309,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":309,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:24:34.852514+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":105,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7568},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7568":{"gid":7568,"name":"b","rank":0,"incarnation":309,"state":"up:replay","state_seq":2,"addr":"172.21.15.40:6833/2324657302","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2324657302},{"type":"v1","addr":"172.21.15.40:6833","nonce":2324657302}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:24:37.004 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.995+0000 7f3134b57700 1 -- 172.21.15.42:0/1462075081 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f311403b390 msgr2=0x7f311403d8a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:37.004 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.995+0000 7f3134b57700 1 --2- 172.21.15.42:0/1462075081 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f311403b390 0x7f311403d8a0 crc :-1 s=READY pgs=1085 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:37.004 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.995+0000 7f3134b57700 1 -- 172.21.15.42:0/1462075081 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3130065540 msgr2=0x7f3130128b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:37.004 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.995+0000 7f3134b57700 1 --2- 172.21.15.42:0/1462075081 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3130065540 0x7f3130128b30 secure :-1 s=READY pgs=1953 cs=0 l=1 rx=0x7f31180043a0 tx=0x7f3118010810).stop 2019-12-05T06:24:37.005 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.995+0000 7f3134b57700 1 -- 172.21.15.42:0/1462075081 shutdown_connections 2019-12-05T06:24:37.005 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.995+0000 7f3134b57700 1 --2- 172.21.15.42:0/1462075081 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f311403b390 0x7f311403d8a0 unknown :-1 s=CLOSED pgs=1085 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:37.005 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.995+0000 7f3134b57700 1 --2- 172.21.15.42:0/1462075081 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3130065540 0x7f3130128b30 unknown :-1 s=CLOSED pgs=1953 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:37.005 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.999+0000 7f3134b57700 1 -- 172.21.15.42:0/1462075081 shutdown_connections 2019-12-05T06:24:37.005 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.999+0000 7f3134b57700 1 -- 172.21.15.42:0/1462075081 wait complete. 2019-12-05T06:24:37.005 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:36.999+0000 7f3134b57700 1 -- 172.21.15.42:0/1462075081 >> 172.21.15.42:0/1462075081 conn(0x7f31300a82a0 msgr2=0x7f31300a9770 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:24:37.014 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 309, "feature_flags": { "enable_multiple": false, "ever_enabled_multiple": false }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2019-12-05T05:03:35.604738+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 309, "ever_allowed_features": 2, "explicitly_allowed_features": 2, "failed": [], "flags": 18, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_7568": { "addr": "172.21.15.40:6833/2324657302", "addrs": { "addrvec": [ { "addr": "172.21.15.40:6832", "nonce": 2324657302, "type": "v2" }, { "addr": "172.21.15.40:6833", "nonce": 2324657302, "type": "v1" } ] }, "export_targets": [], "features": 4540138292836696063, "flags": 0, "gid": 7568, "incarnation": 309, "name": "b", "rank": 0, "state": "up:replay", "state_seq": 2 } }, "last_failure": 0, "last_failure_osd_epoch": 105, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 2, "min_compat_client": "0 (unknown)", "modified": "2019-12-05T06:24:34.852514+0000", "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7568 } } } ], "standbys": [] } 2019-12-05T06:24:39.016 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:24:39.125 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:39.111+0000 7fa18c4dc700 1 Processor -- start 2019-12-05T06:24:39.125 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:39.111+0000 7fa18c4dc700 1 -- start start 2019-12-05T06:24:39.126 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:39.111+0000 7fa18c4dc700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa1840a7740 0x7fa18408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:24:39.126 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:39.111+0000 7fa18c4dc700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1840a0710 con 0x7fa18407e720 2019-12-05T06:24:39.126 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:39.111+0000 7fa18c4dc700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1840a9ba0 con 0x7fa18409fae0 2019-12-05T06:24:39.126 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:39.111+0000 7fa18c4dc700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1840a0b00 con 0x7fa1840a7740 2019-12-05T06:24:39.126 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:39.111+0000 7fa18b27a700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fa18409fae0 0x7fa18414efe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:54942/0 (socket says 172.21.15.42:54942) 2019-12-05T06:24:39.126 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:39.111+0000 7fa18b27a700 1 -- 172.21.15.42:0/2374176957 learned_addr learned my addr 172.21.15.42:0/2374176957 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:24:39.126 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:39.111+0000 7fa18aa79700 1 --2- 172.21.15.42:0/2374176957 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa1840a7740 0x7fa18408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:39.127 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:39.111+0000 7fa189a77700 1 -- 172.21.15.42:0/2374176957 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 246625085 0 0) 0x7fa1840a9ba0 con 0x7fa18409fae0 2019-12-05T06:24:39.127 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:39.111+0000 7fa189a77700 1 -- 172.21.15.42:0/2374176957 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1700030b0 con 0x7fa18409fae0 2019-12-05T06:24:39.127 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:39.111+0000 7fa189a77700 1 -- 172.21.15.42:0/2374176957 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1569837805 0 0) 0x7fa1840a0710 con 0x7fa18407e720 2019-12-05T06:24:39.127 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:39.111+0000 7fa189a77700 1 -- 172.21.15.42:0/2374176957 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1840a9ba0 con 0x7fa18407e720 2019-12-05T06:24:39.127 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:39.111+0000 7fa18aa79700 1 -- 172.21.15.42:0/2374176957 >> v1:172.21.15.42:6789/0 conn(0x7fa18409fae0 legacy=0x7fa18414efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:39.127 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:39.111+0000 7fa18aa79700 1 -- 172.21.15.42:0/2374176957 >> v1:172.21.15.40:6789/0 conn(0x7fa18407e720 legacy=0x7fa18409f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:39.127 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:39.111+0000 7fa18aa79700 1 -- 172.21.15.42:0/2374176957 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa18407eb50 con 0x7fa1840a7740 2019-12-05T06:24:39.128 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:39.111+0000 7fa18aa79700 1 --2- 172.21.15.42:0/2374176957 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa1840a7740 0x7fa18408df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fa180002f30 tx=0x7fa18000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:39.128 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:39.111+0000 7fa18aa79700 1 --2- 172.21.15.42:0/2374176957 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa1840a7740 0x7fa18408df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fa180002f30 tx=0x7fa18000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:39.128 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:39.111+0000 7fa18aa79700 1 --2- 172.21.15.42:0/2374176957 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa1840a7740 0x7fa18408df80 secure :-1 s=READY pgs=562 cs=0 l=1 rx=0x7fa180002f30 tx=0x7fa18000fbf0).ready entity=mon.2 client_cookie=b6c44810c5a05094 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:24:39.128 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:39.111+0000 7fa18aa79700 1 --2- 172.21.15.42:0/2374176957 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa1840a7740 0x7fa18408df80 secure :-1 s=THROTTLE_DONE pgs=562 cs=0 l=1 rx=0x7fa180002f30 tx=0x7fa18000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:39.128 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:39.111+0000 7fa18aa79700 1 --2- 172.21.15.42:0/2374176957 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa1840a7740 0x7fa18408df80 secure :-1 s=THROTTLE_DONE pgs=562 cs=0 l=1 rx=0x7fa180002f30 tx=0x7fa18000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:39.128 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:39.111+0000 7fa189a77700 1 -- 172.21.15.42:0/2374176957 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa1800112b0 con 0x7fa1840a7740 2019-12-05T06:24:39.129 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:39.111+0000 7fa18aa79700 1 --2- 172.21.15.42:0/2374176957 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa1840a7740 0x7fa18408df80 secure :-1 s=THROTTLE_DONE pgs=562 cs=0 l=1 rx=0x7fa180002f30 tx=0x7fa18000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:39.129 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:39.115+0000 7fa189a77700 1 -- 172.21.15.42:0/2374176957 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa180008bc0 con 0x7fa1840a7740 2019-12-05T06:24:39.129 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:39.115+0000 7fa189a77700 1 -- 172.21.15.42:0/2374176957 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa180011890 con 0x7fa1840a7740 2019-12-05T06:24:39.129 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:39.115+0000 7fa18c4dc700 1 -- 172.21.15.42:0/2374176957 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa1840a7740 msgr2=0x7fa18408df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:39.129 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:39.115+0000 7fa18c4dc700 1 --2- 172.21.15.42:0/2374176957 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa1840a7740 0x7fa18408df80 secure :-1 s=READY pgs=562 cs=0 l=1 rx=0x7fa180002f30 tx=0x7fa18000fbf0).stop 2019-12-05T06:24:39.129 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:39.115+0000 7fa18c4dc700 1 -- 172.21.15.42:0/2374176957 shutdown_connections 2019-12-05T06:24:39.129 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:39.115+0000 7fa18c4dc700 1 --2- 172.21.15.42:0/2374176957 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa1840a7740 0x7fa18408df80 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:39.129 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:39.115+0000 7fa18c4dc700 1 -- 172.21.15.42:0/2374176957 shutdown_connections 2019-12-05T06:24:39.130 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:39.115+0000 7fa18c4dc700 1 -- 172.21.15.42:0/2374176957 wait complete. 2019-12-05T06:24:39.130 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:39.115+0000 7fa18c4dc700 1 -- 172.21.15.42:0/2374176957 >> 172.21.15.42:0/2374176957 conn(0x7fa1840aabe0 msgr2=0x7fa1840a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:24:39.130 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:39.115+0000 7fa18c4dc700 1 Processor -- start 2019-12-05T06:24:39.130 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:39.115+0000 7fa18c4dc700 1 -- start start 2019-12-05T06:24:39.130 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:39.115+0000 7fa18c4dc700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa1840a28e0 0x7fa1840a3d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:24:39.130 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:39.115+0000 7fa18c4dc700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa18414f600 0x7fa1840a6ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:24:39.130 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:39.115+0000 7fa18c4dc700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fa184120110 0x7fa1841236d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:24:39.131 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:39.115+0000 7fa18c4dc700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1840a2100 con 0x7fa1840a28e0 2019-12-05T06:24:39.131 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:39.115+0000 7fa18c4dc700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1840a23a0 con 0x7fa184120110 2019-12-05T06:24:39.131 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:39.115+0000 7fa18c4dc700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1840a2640 con 0x7fa18414f600 2019-12-05T06:24:39.131 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:39.115+0000 7fa18b27a700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fa184120110 0x7fa1841236d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:39.131 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:39.115+0000 7fa18b27a700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fa184120110 0x7fa1841236d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:54910/0 (socket says 172.21.15.42:54910) 2019-12-05T06:24:39.131 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:39.115+0000 7fa18b27a700 1 -- 172.21.15.42:0/2690305875 learned_addr learned my addr 172.21.15.42:0/2690305875 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:24:39.131 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:39.115+0000 7fa18aa79700 1 --2- 172.21.15.42:0/2690305875 >> v2:172.21.15.40:3300/0 conn(0x7fa1840a28e0 0x7fa1840a3d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:39.132 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:39.115+0000 7fa18a278700 1 --2- 172.21.15.42:0/2690305875 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa18414f600 0x7fa1840a6ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:39.132 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:39.119+0000 7fa18b27a700 1 -- 172.21.15.42:0/2690305875 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa18414f600 msgr2=0x7fa1840a6ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:39.132 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:39.119+0000 7fa18b27a700 1 --2- 172.21.15.42:0/2690305875 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa18414f600 0x7fa1840a6ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:39.132 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:39.119+0000 7fa18b27a700 1 -- 172.21.15.42:0/2690305875 >> v2:172.21.15.40:3300/0 conn(0x7fa1840a28e0 msgr2=0x7fa1840a3d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:39.132 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:39.119+0000 7fa18b27a700 1 --2- 172.21.15.42:0/2690305875 >> v2:172.21.15.40:3300/0 conn(0x7fa1840a28e0 0x7fa1840a3d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:39.132 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:39.119+0000 7fa18b27a700 1 -- 172.21.15.42:0/2690305875 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1840a1dc0 con 0x7fa184120110 2019-12-05T06:24:39.132 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:39.119+0000 7fa18aa79700 1 --2- 172.21.15.42:0/2690305875 >> v2:172.21.15.40:3300/0 conn(0x7fa1840a28e0 0x7fa1840a3d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:24:39.133 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:39.119+0000 7fa18b27a700 1 --2- 172.21.15.42:0/2690305875 >> v2:172.21.15.42:3300/0 conn(0x7fa184120110 0x7fa1841236d0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fa1840a22c0 tx=0x7fa17c0108b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:39.133 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:39.119+0000 7fa18b27a700 1 --2- 172.21.15.42:0/2690305875 >> v2:172.21.15.42:3300/0 conn(0x7fa184120110 0x7fa1841236d0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fa1840a22c0 tx=0x7fa17c0108b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:39.133 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:39.119+0000 7fa18b27a700 1 --2- 172.21.15.42:0/2690305875 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa184120110 0x7fa1841236d0 secure :-1 s=READY pgs=2616 cs=0 l=1 rx=0x7fa1840a22c0 tx=0x7fa17c0108b0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:24:39.133 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:39.119+0000 7fa18b27a700 1 --2- 172.21.15.42:0/2690305875 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa184120110 0x7fa1841236d0 secure :-1 s=THROTTLE_DONE pgs=2616 cs=0 l=1 rx=0x7fa1840a22c0 tx=0x7fa17c0108b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:39.133 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:39.119+0000 7fa18b27a700 1 --2- 172.21.15.42:0/2690305875 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa184120110 0x7fa1841236d0 secure :-1 s=THROTTLE_DONE pgs=2616 cs=0 l=1 rx=0x7fa1840a22c0 tx=0x7fa17c0108b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:39.133 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:39.119+0000 7fa1777fe700 1 -- 172.21.15.42:0/2690305875 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa17c005b40 con 0x7fa184120110 2019-12-05T06:24:39.134 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:39.119+0000 7fa18b27a700 1 --2- 172.21.15.42:0/2690305875 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa184120110 0x7fa1841236d0 secure :-1 s=THROTTLE_DONE pgs=2616 cs=0 l=1 rx=0x7fa1840a22c0 tx=0x7fa17c0108b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:39.134 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:39.119+0000 7fa1777fe700 1 -- 172.21.15.42:0/2690305875 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa17c005760 con 0x7fa184120110 2019-12-05T06:24:39.134 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:39.119+0000 7fa18c4dc700 1 -- 172.21.15.42:0/2690305875 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1840a7030 con 0x7fa184120110 2019-12-05T06:24:39.134 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:39.119+0000 7fa1777fe700 1 -- 172.21.15.42:0/2690305875 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa17c0062b0 con 0x7fa184120110 2019-12-05T06:24:39.134 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:39.119+0000 7fa18c4dc700 1 -- 172.21.15.42:0/2690305875 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa18414fbd0 con 0x7fa184120110 2019-12-05T06:24:39.135 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:39.119+0000 7fa18b27a700 1 --2- 172.21.15.42:0/2690305875 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa184120110 0x7fa1841236d0 secure :-1 s=THROTTLE_DONE pgs=2616 cs=0 l=1 rx=0x7fa1840a22c0 tx=0x7fa17c0108b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:39.135 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:39.119+0000 7fa1777fe700 1 -- 172.21.15.42:0/2690305875 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fa17c006580 con 0x7fa184120110 2019-12-05T06:24:39.136 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:39.119+0000 7fa18b27a700 1 --2- 172.21.15.42:0/2690305875 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa184120110 0x7fa1841236d0 secure :-1 s=THROTTLE_DONE pgs=2616 cs=0 l=1 rx=0x7fa1840a22c0 tx=0x7fa17c0108b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:39.136 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:39.119+0000 7fa1777fe700 1 --2- 172.21.15.42:0/2690305875 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa16c03ad70 0x7fa16c03d280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:24:39.136 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:39.119+0000 7fa1777fe700 1 -- 172.21.15.42:0/2690305875 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 10847+0+0 (secure 0 0 0) 0x7fa17c04aeb0 con 0x7fa184120110 2019-12-05T06:24:39.136 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:39.119+0000 7fa18aa79700 1 --2- 172.21.15.42:0/2690305875 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa16c03ad70 0x7fa16c03d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:39.136 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:39.123+0000 7fa18aa79700 1 --2- 172.21.15.42:0/2690305875 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa16c03ad70 0x7fa16c03d280 crc :-1 s=READY pgs=1086 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:24:39.136 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:39.127+0000 7fa18c4dc700 1 -- 172.21.15.42:0/2690305875 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1840a42e0 con 0x7fa184120110 2019-12-05T06:24:39.140 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:39.135+0000 7fa18b27a700 1 --2- 172.21.15.42:0/2690305875 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa184120110 0x7fa1841236d0 secure :-1 s=THROTTLE_DONE pgs=2616 cs=0 l=1 rx=0x7fa1840a22c0 tx=0x7fa17c0108b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:39.140 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:39.135+0000 7fa1777fe700 1 -- 172.21.15.42:0/2690305875 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fa17c031f30 con 0x7fa184120110 2019-12-05T06:24:39.313 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:39.307+0000 7fa18c4dc700 1 -- 172.21.15.42:0/2690305875 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa18405d110 con 0x7fa184120110 2019-12-05T06:24:39.322 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:39.307+0000 7fa18b27a700 1 --2- 172.21.15.42:0/2690305875 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa184120110 0x7fa1841236d0 secure :-1 s=THROTTLE_DONE pgs=2616 cs=0 l=1 rx=0x7fa1840a22c0 tx=0x7fa17c0108b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:39.322 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:39.307+0000 7fa1777fe700 1 -- 172.21.15.42:0/2690305875 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 311 v311) v1 ==== 95+0+1776 (secure 0 0 0) 0x7fa17c032190 con 0x7fa184120110 2019-12-05T06:24:39.322 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 311 2019-12-05T06:24:39.322 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:39.311+0000 7fa18c4dc700 1 -- 172.21.15.42:0/2690305875 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa16c03ad70 msgr2=0x7fa16c03d280 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:39.323 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:39.311+0000 7fa18c4dc700 1 --2- 172.21.15.42:0/2690305875 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa16c03ad70 0x7fa16c03d280 crc :-1 s=READY pgs=1086 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:39.323 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:39.311+0000 7fa18c4dc700 1 -- 172.21.15.42:0/2690305875 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa184120110 msgr2=0x7fa1841236d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:39.323 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:39.311+0000 7fa18c4dc700 1 --2- 172.21.15.42:0/2690305875 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa184120110 0x7fa1841236d0 secure :-1 s=READY pgs=2616 cs=0 l=1 rx=0x7fa1840a22c0 tx=0x7fa17c0108b0).stop 2019-12-05T06:24:39.323 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:39.311+0000 7fa18c4dc700 1 -- 172.21.15.42:0/2690305875 shutdown_connections 2019-12-05T06:24:39.323 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:39.311+0000 7fa18c4dc700 1 --2- 172.21.15.42:0/2690305875 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa16c03ad70 0x7fa16c03d280 unknown :-1 s=CLOSED pgs=1086 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:39.323 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:39.311+0000 7fa18c4dc700 1 --2- 172.21.15.42:0/2690305875 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa184120110 0x7fa1841236d0 unknown :-1 s=CLOSED pgs=2616 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:39.323 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:39.311+0000 7fa18c4dc700 1 --2- 172.21.15.42:0/2690305875 >> v2:172.21.15.40:3300/0 conn(0x7fa1840a28e0 0x7fa1840a3d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:39.324 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:39.311+0000 7fa18c4dc700 1 --2- 172.21.15.42:0/2690305875 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa18414f600 0x7fa1840a6ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:39.324 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:39.311+0000 7fa18c4dc700 1 -- 172.21.15.42:0/2690305875 shutdown_connections 2019-12-05T06:24:39.324 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:39.311+0000 7fa18c4dc700 1 -- 172.21.15.42:0/2690305875 wait complete. 2019-12-05T06:24:39.324 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:39.311+0000 7fa18c4dc700 1 -- 172.21.15.42:0/2690305875 >> 172.21.15.42:0/2690305875 conn(0x7fa1840aabe0 msgr2=0x7fa18407da70 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:24:39.324 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:24:39.325 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":311,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":311,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:24:38.876156+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":105,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7568},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7568":{"gid":7568,"name":"b","rank":0,"incarnation":309,"state":"up:rejoin","state_seq":13,"addr":"172.21.15.40:6833/2324657302","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2324657302},{"type":"v1","addr":"172.21.15.40:6833","nonce":2324657302}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:24:39.327 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2019-12-05T06:24:41.328 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:24:41.439 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.427+0000 7f2d58532700 1 Processor -- start 2019-12-05T06:24:41.439 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.427+0000 7f2d58532700 1 -- start start 2019-12-05T06:24:41.440 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.427+0000 7f2d58532700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2d5008df80 0x7f2d5009f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:24:41.440 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.427+0000 7f2d58532700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2d5009f830 0x7f2d5014efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:24:41.440 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.427+0000 7f2d58532700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d5009fc60 con 0x7f2d500a7740 2019-12-05T06:24:41.440 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.427+0000 7f2d58532700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d500a0b00 con 0x7f2d5008df80 2019-12-05T06:24:41.440 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.427+0000 7f2d58532700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d5008e5b0 con 0x7f2d5009f830 2019-12-05T06:24:41.441 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.427+0000 7f2d56acf700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f2d500a7740 0x7f2d5007e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:54962/0 (socket says 172.21.15.42:54962) 2019-12-05T06:24:41.441 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.427+0000 7f2d56acf700 1 -- 172.21.15.42:0/3470891457 learned_addr learned my addr 172.21.15.42:0/3470891457 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:24:41.441 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.427+0000 7f2d562ce700 1 --2- 172.21.15.42:0/3470891457 >> v2:172.21.15.40:3300/0 conn(0x7f2d5008df80 0x7f2d5009f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:41.441 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.427+0000 7f2d572d0700 1 --2- 172.21.15.42:0/3470891457 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2d5009f830 0x7f2d5014efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:41.441 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.427+0000 7f2d55acd700 1 -- 172.21.15.42:0/3470891457 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4166328121 0 0) 0x7f2d5009fc60 con 0x7f2d500a7740 2019-12-05T06:24:41.441 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.427+0000 7f2d55acd700 1 -- 172.21.15.42:0/3470891457 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d3c0030b0 con 0x7f2d500a7740 2019-12-05T06:24:41.441 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.427+0000 7f2d562ce700 1 -- 172.21.15.42:0/3470891457 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2d5009f830 msgr2=0x7f2d5014efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:24:41.442 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.427+0000 7f2d562ce700 1 --2- 172.21.15.42:0/3470891457 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2d5009f830 0x7f2d5014efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:24:41.442 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.431+0000 7f2d562ce700 1 -- 172.21.15.42:0/3470891457 >> v1:172.21.15.42:6789/0 conn(0x7f2d500a7740 legacy=0x7f2d5007e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:41.442 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.431+0000 7f2d562ce700 1 -- 172.21.15.42:0/3470891457 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d500a0710 con 0x7f2d5008df80 2019-12-05T06:24:41.442 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.431+0000 7f2d562ce700 1 --2- 172.21.15.42:0/3470891457 >> v2:172.21.15.40:3300/0 conn(0x7f2d5008df80 0x7f2d5009f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f2d40008790 tx=0x7f2d4000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:41.442 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.431+0000 7f2d562ce700 1 --2- 172.21.15.42:0/3470891457 >> v2:172.21.15.40:3300/0 conn(0x7f2d5008df80 0x7f2d5009f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f2d40008790 tx=0x7f2d4000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:41.442 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.431+0000 7f2d562ce700 1 --2- 172.21.15.42:0/3470891457 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2d5008df80 0x7f2d5009f2e0 secure :-1 s=READY pgs=1955 cs=0 l=1 rx=0x7f2d40008790 tx=0x7f2d4000fc70).ready entity=mon.0 client_cookie=dedf8d88520f380f server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:24:41.443 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.431+0000 7f2d562ce700 1 --2- 172.21.15.42:0/3470891457 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2d5008df80 0x7f2d5009f2e0 secure :-1 s=THROTTLE_DONE pgs=1955 cs=0 l=1 rx=0x7f2d40008790 tx=0x7f2d4000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:41.443 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.431+0000 7f2d55acd700 1 -- 172.21.15.42:0/3470891457 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2d40008bb0 con 0x7f2d5008df80 2019-12-05T06:24:41.443 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.431+0000 7f2d562ce700 1 --2- 172.21.15.42:0/3470891457 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2d5008df80 0x7f2d5009f2e0 secure :-1 s=THROTTLE_DONE pgs=1955 cs=0 l=1 rx=0x7f2d40008790 tx=0x7f2d4000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:41.443 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.431+0000 7f2d55acd700 1 -- 172.21.15.42:0/3470891457 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d400087d0 con 0x7f2d5008df80 2019-12-05T06:24:41.443 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.431+0000 7f2d562ce700 1 --2- 172.21.15.42:0/3470891457 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2d5008df80 0x7f2d5009f2e0 secure :-1 s=THROTTLE_DONE pgs=1955 cs=0 l=1 rx=0x7f2d40008790 tx=0x7f2d4000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:41.444 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.431+0000 7f2d55acd700 1 -- 172.21.15.42:0/3470891457 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2d40011860 con 0x7f2d5008df80 2019-12-05T06:24:41.444 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.431+0000 7f2d58532700 1 -- 172.21.15.42:0/3470891457 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2d5008df80 msgr2=0x7f2d5009f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:41.444 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.431+0000 7f2d58532700 1 --2- 172.21.15.42:0/3470891457 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2d5008df80 0x7f2d5009f2e0 secure :-1 s=READY pgs=1955 cs=0 l=1 rx=0x7f2d40008790 tx=0x7f2d4000fc70).stop 2019-12-05T06:24:41.444 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.431+0000 7f2d58532700 1 -- 172.21.15.42:0/3470891457 shutdown_connections 2019-12-05T06:24:41.448 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.431+0000 7f2d58532700 1 --2- 172.21.15.42:0/3470891457 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2d5009f830 0x7f2d5014efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:24:41.449 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.431+0000 7f2d58532700 1 --2- 172.21.15.42:0/3470891457 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2d5008df80 0x7f2d5009f2e0 unknown :-1 s=CLOSED pgs=1955 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:41.449 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.431+0000 7f2d58532700 1 -- 172.21.15.42:0/3470891457 shutdown_connections 2019-12-05T06:24:41.459 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.431+0000 7f2d58532700 1 -- 172.21.15.42:0/3470891457 wait complete. 2019-12-05T06:24:41.459 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.431+0000 7f2d58532700 1 -- 172.21.15.42:0/3470891457 >> 172.21.15.42:0/3470891457 conn(0x7f2d500aabe0 msgr2=0x7f2d500a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:24:41.469 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.435+0000 7f2d58532700 1 Processor -- start 2019-12-05T06:24:41.470 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.435+0000 7f2d58532700 1 -- start start 2019-12-05T06:24:41.471 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.435+0000 7f2d58532700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2d5011a970 0x7f2d501398b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:24:41.471 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.435+0000 7f2d58532700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d5007b5f0 con 0x7f2d501188a0 2019-12-05T06:24:41.471 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.435+0000 7f2d58532700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d5011b2d0 con 0x7f2d50119430 2019-12-05T06:24:41.472 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.435+0000 7f2d58532700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d500a4c00 con 0x7f2d5011a970 2019-12-05T06:24:41.472 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.435+0000 7f2d562ce700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f2d50119430 0x7f2d5007adf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:54970/0 (socket says 172.21.15.42:54970) 2019-12-05T06:24:41.472 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.435+0000 7f2d562ce700 1 -- 172.21.15.42:0/1799278759 learned_addr learned my addr 172.21.15.42:0/1799278759 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:24:41.472 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.435+0000 7f2d572d0700 1 --2- 172.21.15.42:0/1799278759 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2d5011a970 0x7f2d501398b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:41.472 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.435+0000 7f2d477fe700 1 -- 172.21.15.42:0/1799278759 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 42030963 0 0) 0x7f2d5011b2d0 con 0x7f2d50119430 2019-12-05T06:24:41.472 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.435+0000 7f2d477fe700 1 -- 172.21.15.42:0/1799278759 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d380030b0 con 0x7f2d50119430 2019-12-05T06:24:41.473 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.435+0000 7f2d477fe700 1 -- 172.21.15.42:0/1799278759 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2255155857 0 0) 0x7f2d5007b5f0 con 0x7f2d501188a0 2019-12-05T06:24:41.473 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.435+0000 7f2d477fe700 1 -- 172.21.15.42:0/1799278759 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d5011b2d0 con 0x7f2d501188a0 2019-12-05T06:24:41.473 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.435+0000 7f2d572d0700 1 -- 172.21.15.42:0/1799278759 >> v1:172.21.15.42:6789/0 conn(0x7f2d50119430 legacy=0x7f2d5007adf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:41.473 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.435+0000 7f2d572d0700 1 -- 172.21.15.42:0/1799278759 >> v1:172.21.15.40:6789/0 conn(0x7f2d501188a0 legacy=0x7f2d500a4e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:41.473 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.435+0000 7f2d572d0700 1 -- 172.21.15.42:0/1799278759 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d5011a3c0 con 0x7f2d5011a970 2019-12-05T06:24:41.473 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.435+0000 7f2d572d0700 1 --2- 172.21.15.42:0/1799278759 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2d5011a970 0x7f2d501398b0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f2d480032b0 tx=0x7f2d48010c90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:41.474 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.435+0000 7f2d572d0700 1 --2- 172.21.15.42:0/1799278759 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2d5011a970 0x7f2d501398b0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f2d480032b0 tx=0x7f2d48010c90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:41.474 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.435+0000 7f2d572d0700 1 --2- 172.21.15.42:0/1799278759 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2d5011a970 0x7f2d501398b0 secure :-1 s=READY pgs=563 cs=0 l=1 rx=0x7f2d480032b0 tx=0x7f2d48010c90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:24:41.474 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.435+0000 7f2d572d0700 1 --2- 172.21.15.42:0/1799278759 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2d5011a970 0x7f2d501398b0 secure :-1 s=THROTTLE_DONE pgs=563 cs=0 l=1 rx=0x7f2d480032b0 tx=0x7f2d48010c90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:41.474 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.435+0000 7f2d477fe700 1 -- 172.21.15.42:0/1799278759 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2d48007350 con 0x7f2d5011a970 2019-12-05T06:24:41.475 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.435+0000 7f2d572d0700 1 --2- 172.21.15.42:0/1799278759 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2d5011a970 0x7f2d501398b0 secure :-1 s=THROTTLE_DONE pgs=563 cs=0 l=1 rx=0x7f2d480032b0 tx=0x7f2d48010c90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:41.475 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.435+0000 7f2d572d0700 1 --2- 172.21.15.42:0/1799278759 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2d5011a970 0x7f2d501398b0 secure :-1 s=THROTTLE_DONE pgs=563 cs=0 l=1 rx=0x7f2d480032b0 tx=0x7f2d48010c90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:41.475 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.435+0000 7f2d477fe700 1 -- 172.21.15.42:0/1799278759 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d48005cb0 con 0x7f2d5011a970 2019-12-05T06:24:41.475 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.435+0000 7f2d58532700 1 -- 172.21.15.42:0/1799278759 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d50139e00 con 0x7f2d5011a970 2019-12-05T06:24:41.475 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.435+0000 7f2d477fe700 1 -- 172.21.15.42:0/1799278759 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2d48012350 con 0x7f2d5011a970 2019-12-05T06:24:41.475 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.435+0000 7f2d58532700 1 -- 172.21.15.42:0/1799278759 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d5013a380 con 0x7f2d5011a970 2019-12-05T06:24:41.476 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.439+0000 7f2d572d0700 1 --2- 172.21.15.42:0/1799278759 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2d5011a970 0x7f2d501398b0 secure :-1 s=THROTTLE_DONE pgs=563 cs=0 l=1 rx=0x7f2d480032b0 tx=0x7f2d48010c90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:41.476 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.439+0000 7f2d477fe700 1 -- 172.21.15.42:0/1799278759 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f2d48012620 con 0x7f2d5011a970 2019-12-05T06:24:41.476 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.439+0000 7f2d572d0700 1 --2- 172.21.15.42:0/1799278759 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2d5011a970 0x7f2d501398b0 secure :-1 s=THROTTLE_DONE pgs=563 cs=0 l=1 rx=0x7f2d480032b0 tx=0x7f2d48010c90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:41.476 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.439+0000 7f2d477fe700 1 --2- 172.21.15.42:0/1799278759 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2d3803b450 0x7f2d3803d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:24:41.476 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.439+0000 7f2d477fe700 1 -- 172.21.15.42:0/1799278759 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 10847+0+0 (secure 0 0 0) 0x7f2d4804b640 con 0x7f2d5011a970 2019-12-05T06:24:41.476 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.439+0000 7f2d56acf700 1 --2- 172.21.15.42:0/1799278759 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2d3803b450 0x7f2d3803d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:41.476 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.439+0000 7f2d56acf700 1 --2- 172.21.15.42:0/1799278759 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2d3803b450 0x7f2d3803d960 crc :-1 s=READY pgs=1087 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:24:41.477 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.443+0000 7f2d58532700 1 -- 172.21.15.42:0/1799278759 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d5005d110 con 0x7f2d5011a970 2019-12-05T06:24:41.477 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.451+0000 7f2d572d0700 1 --2- 172.21.15.42:0/1799278759 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2d5011a970 0x7f2d501398b0 secure :-1 s=THROTTLE_DONE pgs=563 cs=0 l=1 rx=0x7f2d480032b0 tx=0x7f2d48010c90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:41.477 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.451+0000 7f2d477fe700 1 -- 172.21.15.42:0/1799278759 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f2d480326c0 con 0x7f2d5011a970 2019-12-05T06:24:41.629 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.623+0000 7f2d58532700 1 -- 172.21.15.42:0/1799278759 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2d5005b8c0 con 0x7f2d5011a970 2019-12-05T06:24:41.630 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.623+0000 7f2d572d0700 1 --2- 172.21.15.42:0/1799278759 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2d5011a970 0x7f2d501398b0 secure :-1 s=THROTTLE_DONE pgs=563 cs=0 l=1 rx=0x7f2d480032b0 tx=0x7f2d48010c90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:41.630 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.623+0000 7f2d477fe700 1 -- 172.21.15.42:0/1799278759 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 312 v312) v1 ==== 95+0+1776 (secure 0 0 0) 0x7f2d48032920 con 0x7f2d5011a970 2019-12-05T06:24:41.633 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 312 2019-12-05T06:24:41.633 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:24:41.633 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":312,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":312,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:24:39.885811+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":105,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7568},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7568":{"gid":7568,"name":"b","rank":0,"incarnation":309,"state":"up:active","state_seq":14,"addr":"172.21.15.40:6833/2324657302","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2324657302},{"type":"v1","addr":"172.21.15.40:6833","nonce":2324657302}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:24:41.637 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.631+0000 7f2d58532700 1 -- 172.21.15.42:0/1799278759 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2d3803b450 msgr2=0x7f2d3803d960 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:41.637 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.631+0000 7f2d58532700 1 --2- 172.21.15.42:0/1799278759 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2d3803b450 0x7f2d3803d960 crc :-1 s=READY pgs=1087 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:41.637 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.631+0000 7f2d58532700 1 -- 172.21.15.42:0/1799278759 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2d5011a970 msgr2=0x7f2d501398b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:41.638 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.631+0000 7f2d58532700 1 --2- 172.21.15.42:0/1799278759 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2d5011a970 0x7f2d501398b0 secure :-1 s=READY pgs=563 cs=0 l=1 rx=0x7f2d480032b0 tx=0x7f2d48010c90).stop 2019-12-05T06:24:41.638 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.631+0000 7f2d58532700 1 -- 172.21.15.42:0/1799278759 shutdown_connections 2019-12-05T06:24:41.638 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.631+0000 7f2d58532700 1 --2- 172.21.15.42:0/1799278759 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2d3803b450 0x7f2d3803d960 unknown :-1 s=CLOSED pgs=1087 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:41.638 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.631+0000 7f2d58532700 1 --2- 172.21.15.42:0/1799278759 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2d5011a970 0x7f2d501398b0 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:41.638 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.631+0000 7f2d58532700 1 -- 172.21.15.42:0/1799278759 shutdown_connections 2019-12-05T06:24:41.638 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.631+0000 7f2d58532700 1 -- 172.21.15.42:0/1799278759 wait complete. 2019-12-05T06:24:41.639 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.631+0000 7f2d58532700 1 -- 172.21.15.42:0/1799278759 >> 172.21.15.42:0/1799278759 conn(0x7f2d500aabe0 msgr2=0x7f2d50059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:24:41.650 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b has gained rank=0, replacing gid=10993 2019-12-05T06:24:41.650 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 0 secs before reviving mds.a 2019-12-05T06:24:41.650 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.a 2019-12-05T06:24:41.650 INFO:tasks.ceph.mds.a:Restarting daemon 2019-12-05T06:24:41.651 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2019-12-05T06:24:41.657 INFO:tasks.ceph.mds.a:Started 2019-12-05T06:24:41.657 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:24:41.715 INFO:tasks.ceph.mds.a.smithi042.stdout:starting mds.a at 2019-12-05T06:24:41.755 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.743+0000 7fd75276d700 1 Processor -- start 2019-12-05T06:24:41.756 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.743+0000 7fd75276d700 1 -- start start 2019-12-05T06:24:41.756 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.743+0000 7fd75276d700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd74c0a7740 0x7fd74c08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:24:41.756 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.743+0000 7fd75276d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd74c07e720 0x7fd74c07eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:24:41.756 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.743+0000 7fd75276d700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd74c09f2e0 0x7fd74c09f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:24:41.756 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.743+0000 7fd75276d700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd74c0a0b00 con 0x7fd74c0a7740 2019-12-05T06:24:41.756 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.743+0000 7fd75276d700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd74c08e5b0 con 0x7fd74c07e720 2019-12-05T06:24:41.756 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.743+0000 7fd75276d700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd74c08e850 con 0x7fd74c09f2e0 2019-12-05T06:24:41.757 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.743+0000 7fd74bfff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd74c07e720 0x7fd74c07eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:41.757 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.743+0000 7fd74bfff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd74c07e720 0x7fd74c07eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:54958/0 (socket says 172.21.15.42:54958) 2019-12-05T06:24:41.757 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.743+0000 7fd74bfff700 1 -- 172.21.15.42:0/3875808530 learned_addr learned my addr 172.21.15.42:0/3875808530 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:24:41.757 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.747+0000 7fd75150b700 1 --2- 172.21.15.42:0/3875808530 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd74c09f2e0 0x7fd74c09f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:41.757 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.747+0000 7fd750d0a700 1 --2- 172.21.15.42:0/3875808530 >> v2:172.21.15.40:3300/0 conn(0x7fd74c0a7740 0x7fd74c08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:41.757 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.747+0000 7fd74bfff700 1 -- 172.21.15.42:0/3875808530 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd74c09f2e0 msgr2=0x7fd74c09f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:24:41.757 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.747+0000 7fd74bfff700 1 --2- 172.21.15.42:0/3875808530 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd74c09f2e0 0x7fd74c09f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:24:41.762 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.747+0000 7fd74bfff700 1 -- 172.21.15.42:0/3875808530 >> v2:172.21.15.40:3300/0 conn(0x7fd74c0a7740 msgr2=0x7fd74c08df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:24:41.762 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.747+0000 7fd74bfff700 1 --2- 172.21.15.42:0/3875808530 >> v2:172.21.15.40:3300/0 conn(0x7fd74c0a7740 0x7fd74c08df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:24:41.762 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.747+0000 7fd74bfff700 1 -- 172.21.15.42:0/3875808530 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd74c07f0a0 con 0x7fd74c07e720 2019-12-05T06:24:41.762 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.747+0000 7fd74bfff700 1 --2- 172.21.15.42:0/3875808530 >> v2:172.21.15.42:3300/0 conn(0x7fd74c07e720 0x7fd74c07eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fd74c0a0f50 tx=0x7fd73c00fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:41.762 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.747+0000 7fd74bfff700 1 --2- 172.21.15.42:0/3875808530 >> v2:172.21.15.42:3300/0 conn(0x7fd74c07e720 0x7fd74c07eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fd74c0a0f50 tx=0x7fd73c00fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:41.763 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.747+0000 7fd74bfff700 1 --2- 172.21.15.42:0/3875808530 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd74c07e720 0x7fd74c07eb50 secure :-1 s=READY pgs=2623 cs=0 l=1 rx=0x7fd74c0a0f50 tx=0x7fd73c00fdb0).ready entity=mon.1 client_cookie=2d7baf087052a3e9 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:24:41.763 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.747+0000 7fd74bfff700 1 --2- 172.21.15.42:0/3875808530 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd74c07e720 0x7fd74c07eb50 secure :-1 s=THROTTLE_DONE pgs=2623 cs=0 l=1 rx=0x7fd74c0a0f50 tx=0x7fd73c00fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:41.763 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.747+0000 7fd74bfff700 1 --2- 172.21.15.42:0/3875808530 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd74c07e720 0x7fd74c07eb50 secure :-1 s=THROTTLE_DONE pgs=2623 cs=0 l=1 rx=0x7fd74c0a0f50 tx=0x7fd73c00fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:41.764 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.747+0000 7fd74bfff700 1 --2- 172.21.15.42:0/3875808530 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd74c07e720 0x7fd74c07eb50 secure :-1 s=THROTTLE_DONE pgs=2623 cs=0 l=1 rx=0x7fd74c0a0f50 tx=0x7fd73c00fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:41.764 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.747+0000 7fd74b7fe700 1 -- 172.21.15.42:0/3875808530 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd73c008c90 con 0x7fd74c07e720 2019-12-05T06:24:41.764 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.747+0000 7fd74b7fe700 1 -- 172.21.15.42:0/3875808530 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd73c0088b0 con 0x7fd74c07e720 2019-12-05T06:24:41.764 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.747+0000 7fd74b7fe700 1 -- 172.21.15.42:0/3875808530 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd73c011b50 con 0x7fd74c07e720 2019-12-05T06:24:41.765 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.747+0000 7fd75276d700 1 -- 172.21.15.42:0/3875808530 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd74c07e720 msgr2=0x7fd74c07eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:41.765 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.747+0000 7fd75276d700 1 --2- 172.21.15.42:0/3875808530 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd74c07e720 0x7fd74c07eb50 secure :-1 s=READY pgs=2623 cs=0 l=1 rx=0x7fd74c0a0f50 tx=0x7fd73c00fdb0).stop 2019-12-05T06:24:41.765 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.747+0000 7fd75276d700 1 -- 172.21.15.42:0/3875808530 shutdown_connections 2019-12-05T06:24:41.765 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.747+0000 7fd75276d700 1 --2- 172.21.15.42:0/3875808530 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd74c09f2e0 0x7fd74c09f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:24:41.765 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.747+0000 7fd75276d700 1 --2- 172.21.15.42:0/3875808530 >> v2:172.21.15.40:3300/0 conn(0x7fd74c0a7740 0x7fd74c08df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:24:41.765 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.747+0000 7fd75276d700 1 --2- 172.21.15.42:0/3875808530 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd74c07e720 0x7fd74c07eb50 unknown :-1 s=CLOSED pgs=2623 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:41.766 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.747+0000 7fd75276d700 1 -- 172.21.15.42:0/3875808530 shutdown_connections 2019-12-05T06:24:41.766 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.747+0000 7fd75276d700 1 -- 172.21.15.42:0/3875808530 wait complete. 2019-12-05T06:24:41.766 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.747+0000 7fd75276d700 1 -- 172.21.15.42:0/3875808530 >> 172.21.15.42:0/3875808530 conn(0x7fd74c0aabe0 msgr2=0x7fd74c0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:24:41.769 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.751+0000 7fd75276d700 1 Processor -- start 2019-12-05T06:24:41.770 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.751+0000 7fd75276d700 1 -- start start 2019-12-05T06:24:41.770 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.751+0000 7fd75276d700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd74c065160 0x7fd74c0a2570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:24:41.770 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.751+0000 7fd75276d700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd74c064c20 con 0x7fd74c0a5e50 2019-12-05T06:24:41.770 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.751+0000 7fd75276d700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd74c1303b0 con 0x7fd74c126a40 2019-12-05T06:24:41.770 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.751+0000 7fd75276d700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd74c0a2ba0 con 0x7fd74c065160 2019-12-05T06:24:41.770 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.751+0000 7fd75150b700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fd74c126a40 0x7fd74c126e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:55006/0 (socket says 172.21.15.42:55006) 2019-12-05T06:24:41.771 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.751+0000 7fd75150b700 1 -- 172.21.15.42:0/3266010994 learned_addr learned my addr 172.21.15.42:0/3266010994 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:24:41.771 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.751+0000 7fd750d0a700 1 --2- 172.21.15.42:0/3266010994 >> v2:172.21.15.40:3300/0 conn(0x7fd74c065160 0x7fd74c0a2570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:41.771 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.751+0000 7fd7497fa700 1 -- 172.21.15.42:0/3266010994 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3556353980 0 0) 0x7fd74c1303b0 con 0x7fd74c126a40 2019-12-05T06:24:41.771 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.751+0000 7fd7497fa700 1 -- 172.21.15.42:0/3266010994 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7240030b0 con 0x7fd74c126a40 2019-12-05T06:24:41.771 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.751+0000 7fd750d0a700 1 -- 172.21.15.42:0/3266010994 >> v1:172.21.15.42:6789/0 conn(0x7fd74c126a40 legacy=0x7fd74c126e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:41.771 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.751+0000 7fd7497fa700 1 -- 172.21.15.42:0/3266010994 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2376200443 0 0) 0x7fd74c064c20 con 0x7fd74c0a5e50 2019-12-05T06:24:41.772 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.751+0000 7fd750d0a700 1 -- 172.21.15.42:0/3266010994 >> v1:172.21.15.40:6789/0 conn(0x7fd74c0a5e50 legacy=0x7fd74c064420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:41.772 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.751+0000 7fd750d0a700 1 -- 172.21.15.42:0/3266010994 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd74c065b20 con 0x7fd74c065160 2019-12-05T06:24:41.772 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.751+0000 7fd750d0a700 1 --2- 172.21.15.42:0/3266010994 >> v2:172.21.15.40:3300/0 conn(0x7fd74c065160 0x7fd74c0a2570 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fd744001a50 tx=0x7fd744003370).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:41.772 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.751+0000 7fd750d0a700 1 --2- 172.21.15.42:0/3266010994 >> v2:172.21.15.40:3300/0 conn(0x7fd74c065160 0x7fd74c0a2570 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fd744001a50 tx=0x7fd744003370).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:41.772 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.751+0000 7fd750d0a700 1 --2- 172.21.15.42:0/3266010994 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd74c065160 0x7fd74c0a2570 secure :-1 s=READY pgs=1959 cs=0 l=1 rx=0x7fd744001a50 tx=0x7fd744003370).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:24:41.773 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.751+0000 7fd750d0a700 1 --2- 172.21.15.42:0/3266010994 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd74c065160 0x7fd74c0a2570 secure :-1 s=THROTTLE_DONE pgs=1959 cs=0 l=1 rx=0x7fd744001a50 tx=0x7fd744003370).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:41.773 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.751+0000 7fd7497fa700 1 -- 172.21.15.42:0/3266010994 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd744001810 con 0x7fd74c065160 2019-12-05T06:24:41.773 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.751+0000 7fd750d0a700 1 --2- 172.21.15.42:0/3266010994 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd74c065160 0x7fd74c0a2570 secure :-1 s=THROTTLE_DONE pgs=1959 cs=0 l=1 rx=0x7fd744001a50 tx=0x7fd744003370).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:41.773 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.751+0000 7fd7497fa700 1 -- 172.21.15.42:0/3266010994 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd744001620 con 0x7fd74c065160 2019-12-05T06:24:41.773 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.751+0000 7fd750d0a700 1 --2- 172.21.15.42:0/3266010994 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd74c065160 0x7fd74c0a2570 secure :-1 s=THROTTLE_DONE pgs=1959 cs=0 l=1 rx=0x7fd744001a50 tx=0x7fd744003370).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:41.773 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.751+0000 7fd75276d700 1 -- 172.21.15.42:0/3266010994 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd74c138810 con 0x7fd74c065160 2019-12-05T06:24:41.773 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.755+0000 7fd7497fa700 1 -- 172.21.15.42:0/3266010994 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd734005b20 con 0x7fd74c065160 2019-12-05T06:24:41.774 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.755+0000 7fd75276d700 1 -- 172.21.15.42:0/3266010994 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd74c138d40 con 0x7fd74c065160 2019-12-05T06:24:41.774 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.755+0000 7fd750d0a700 1 --2- 172.21.15.42:0/3266010994 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd74c065160 0x7fd74c0a2570 secure :-1 s=THROTTLE_DONE pgs=1959 cs=0 l=1 rx=0x7fd744001a50 tx=0x7fd744003370).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:41.774 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.755+0000 7fd7497fa700 1 -- 172.21.15.42:0/3266010994 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fd73400a250 con 0x7fd74c065160 2019-12-05T06:24:41.774 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.755+0000 7fd750d0a700 1 --2- 172.21.15.42:0/3266010994 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd74c065160 0x7fd74c0a2570 secure :-1 s=THROTTLE_DONE pgs=1959 cs=0 l=1 rx=0x7fd744001a50 tx=0x7fd744003370).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:41.774 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.755+0000 7fd7497fa700 1 --2- 172.21.15.42:0/3266010994 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd72403b1e0 0x7fd72403d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:24:41.774 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.755+0000 7fd7497fa700 1 -- 172.21.15.42:0/3266010994 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 10847+0+0 (secure 0 0 0) 0x7fd73404a920 con 0x7fd74c065160 2019-12-05T06:24:41.775 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.755+0000 7fd74bfff700 1 --2- 172.21.15.42:0/3266010994 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd72403b1e0 0x7fd72403d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:41.775 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.755+0000 7fd74bfff700 1 --2- 172.21.15.42:0/3266010994 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd72403b1e0 0x7fd72403d6f0 crc :-1 s=READY pgs=1088 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:24:41.775 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.759+0000 7fd75276d700 1 -- 172.21.15.42:0/3266010994 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd74c05d110 con 0x7fd74c065160 2019-12-05T06:24:41.777 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.771+0000 7fd750d0a700 1 --2- 172.21.15.42:0/3266010994 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd74c065160 0x7fd74c0a2570 secure :-1 s=THROTTLE_DONE pgs=1959 cs=0 l=1 rx=0x7fd744001a50 tx=0x7fd744003370).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:41.777 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.771+0000 7fd7497fa700 1 -- 172.21.15.42:0/3266010994 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fd7340319a0 con 0x7fd74c065160 2019-12-05T06:24:41.945 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.939+0000 7fd75276d700 1 -- 172.21.15.42:0/3266010994 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd74c05b8c0 con 0x7fd74c065160 2019-12-05T06:24:41.946 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.939+0000 7fd750d0a700 1 --2- 172.21.15.42:0/3266010994 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd74c065160 0x7fd74c0a2570 secure :-1 s=THROTTLE_DONE pgs=1959 cs=0 l=1 rx=0x7fd744001a50 tx=0x7fd744003370).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:41.946 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.939+0000 7fd7497fa700 1 -- 172.21.15.42:0/3266010994 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 312 v312) v1 ==== 95+0+1776 (secure 0 0 0) 0x7fd734031c00 con 0x7fd74c065160 2019-12-05T06:24:41.948 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 312 2019-12-05T06:24:41.948 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:24:41.949 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":312,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":312,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:24:39.885811+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":105,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7568},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7568":{"gid":7568,"name":"b","rank":0,"incarnation":309,"state":"up:active","state_seq":14,"addr":"172.21.15.40:6833/2324657302","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2324657302},{"type":"v1","addr":"172.21.15.40:6833","nonce":2324657302}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:24:41.951 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.943+0000 7fd75276d700 1 -- 172.21.15.42:0/3266010994 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd72403b1e0 msgr2=0x7fd72403d6f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:41.951 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.943+0000 7fd75276d700 1 --2- 172.21.15.42:0/3266010994 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd72403b1e0 0x7fd72403d6f0 crc :-1 s=READY pgs=1088 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:41.951 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.943+0000 7fd75276d700 1 -- 172.21.15.42:0/3266010994 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd74c065160 msgr2=0x7fd74c0a2570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:41.951 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.943+0000 7fd75276d700 1 --2- 172.21.15.42:0/3266010994 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd74c065160 0x7fd74c0a2570 secure :-1 s=READY pgs=1959 cs=0 l=1 rx=0x7fd744001a50 tx=0x7fd744003370).stop 2019-12-05T06:24:41.952 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.943+0000 7fd75276d700 1 -- 172.21.15.42:0/3266010994 shutdown_connections 2019-12-05T06:24:41.952 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.943+0000 7fd75276d700 1 --2- 172.21.15.42:0/3266010994 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd72403b1e0 0x7fd72403d6f0 unknown :-1 s=CLOSED pgs=1088 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:41.952 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.943+0000 7fd75276d700 1 --2- 172.21.15.42:0/3266010994 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd74c065160 0x7fd74c0a2570 unknown :-1 s=CLOSED pgs=1959 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:41.952 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.943+0000 7fd75276d700 1 -- 172.21.15.42:0/3266010994 shutdown_connections 2019-12-05T06:24:41.952 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.943+0000 7fd75276d700 1 -- 172.21.15.42:0/3266010994 wait complete. 2019-12-05T06:24:41.952 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:41.943+0000 7fd75276d700 1 -- 172.21.15.42:0/3266010994 >> 172.21.15.42:0/3266010994 conn(0x7fd74c0a8a80 msgr2=0x7fd74c09f690 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:24:41.968 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2019-12-05T06:24:43.969 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:24:44.077 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:44.063+0000 7f4282c22700 1 Processor -- start 2019-12-05T06:24:44.077 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:44.063+0000 7f4282c22700 1 -- start start 2019-12-05T06:24:44.077 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:44.067+0000 7f4282c22700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f427c0a7740 0x7f427c08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:24:44.077 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:44.067+0000 7f4282c22700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f427c09fae0 0x7f427c07eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:24:44.078 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:44.067+0000 7f4282c22700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f427c0a0710 con 0x7f427c07e720 2019-12-05T06:24:44.078 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:44.067+0000 7f4282c22700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f427c0a0b00 con 0x7f427c0a7740 2019-12-05T06:24:44.078 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:44.067+0000 7f4282c22700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f427c08e5b0 con 0x7f427c09fae0 2019-12-05T06:24:44.078 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:44.067+0000 7f42819c0700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f427c09fae0 0x7f427c07eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:44.078 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:44.067+0000 7f42819c0700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f427c09fae0 0x7f427c07eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:54974/0 (socket says 172.21.15.42:54974) 2019-12-05T06:24:44.078 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:44.067+0000 7f42819c0700 1 -- 172.21.15.42:0/378926232 learned_addr learned my addr 172.21.15.42:0/378926232 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:24:44.078 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:44.067+0000 7f42811bf700 1 --2- 172.21.15.42:0/378926232 >> v2:172.21.15.40:3300/0 conn(0x7f427c0a7740 0x7f427c08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:44.079 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:44.067+0000 7f42819c0700 1 -- 172.21.15.42:0/378926232 >> v2:172.21.15.40:3300/0 conn(0x7f427c0a7740 msgr2=0x7f427c08df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:24:44.079 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:44.067+0000 7f42819c0700 1 --2- 172.21.15.42:0/378926232 >> v2:172.21.15.40:3300/0 conn(0x7f427c0a7740 0x7f427c08df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:24:44.079 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:44.067+0000 7f42819c0700 1 -- 172.21.15.42:0/378926232 >> v1:172.21.15.42:6789/0 conn(0x7f427c07e720 legacy=0x7f427c09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:44.079 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:44.067+0000 7f42819c0700 1 -- 172.21.15.42:0/378926232 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f427c07f0a0 con 0x7f427c09fae0 2019-12-05T06:24:44.079 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:44.067+0000 7f42811bf700 1 --2- 172.21.15.42:0/378926232 >> v2:172.21.15.40:3300/0 conn(0x7f427c0a7740 0x7f427c08df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:24:44.079 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:44.067+0000 7f42819c0700 1 --2- 172.21.15.42:0/378926232 >> v2:172.21.15.42:3300/0 conn(0x7f427c09fae0 0x7f427c07eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f42740032b0 tx=0x7f42740100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:44.079 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:44.067+0000 7f42819c0700 1 --2- 172.21.15.42:0/378926232 >> v2:172.21.15.42:3300/0 conn(0x7f427c09fae0 0x7f427c07eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f42740032b0 tx=0x7f42740100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:44.080 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:44.067+0000 7f42819c0700 1 --2- 172.21.15.42:0/378926232 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f427c09fae0 0x7f427c07eb50 secure :-1 s=READY pgs=2626 cs=0 l=1 rx=0x7f42740032b0 tx=0x7f42740100b0).ready entity=mon.1 client_cookie=6ee53e69dd3602a3 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:24:44.080 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:44.067+0000 7f42819c0700 1 --2- 172.21.15.42:0/378926232 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f427c09fae0 0x7f427c07eb50 secure :-1 s=THROTTLE_DONE pgs=2626 cs=0 l=1 rx=0x7f42740032b0 tx=0x7f42740100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:44.080 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:44.067+0000 7f4273fff700 1 -- 172.21.15.42:0/378926232 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4274010d60 con 0x7f427c09fae0 2019-12-05T06:24:44.080 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:44.067+0000 7f42819c0700 1 --2- 172.21.15.42:0/378926232 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f427c09fae0 0x7f427c07eb50 secure :-1 s=THROTTLE_DONE pgs=2626 cs=0 l=1 rx=0x7f42740032b0 tx=0x7f42740100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:44.080 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:44.067+0000 7f42819c0700 1 --2- 172.21.15.42:0/378926232 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f427c09fae0 0x7f427c07eb50 secure :-1 s=THROTTLE_DONE pgs=2626 cs=0 l=1 rx=0x7f42740032b0 tx=0x7f42740100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:44.080 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:44.067+0000 7f4273fff700 1 -- 172.21.15.42:0/378926232 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4274010980 con 0x7f427c09fae0 2019-12-05T06:24:44.080 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:44.067+0000 7f4273fff700 1 -- 172.21.15.42:0/378926232 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f42740114a0 con 0x7f427c09fae0 2019-12-05T06:24:44.081 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:44.067+0000 7f4282c22700 1 -- 172.21.15.42:0/378926232 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f427c09fae0 msgr2=0x7f427c07eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:44.081 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:44.067+0000 7f4282c22700 1 --2- 172.21.15.42:0/378926232 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f427c09fae0 0x7f427c07eb50 secure :-1 s=READY pgs=2626 cs=0 l=1 rx=0x7f42740032b0 tx=0x7f42740100b0).stop 2019-12-05T06:24:44.081 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:44.071+0000 7f4282c22700 1 -- 172.21.15.42:0/378926232 shutdown_connections 2019-12-05T06:24:44.081 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:44.071+0000 7f4282c22700 1 --2- 172.21.15.42:0/378926232 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f427c09fae0 0x7f427c07eb50 unknown :-1 s=CLOSED pgs=2626 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:44.082 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:44.071+0000 7f4282c22700 1 --2- 172.21.15.42:0/378926232 >> v2:172.21.15.40:3300/0 conn(0x7f427c0a7740 0x7f427c08df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:24:44.082 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:44.071+0000 7f4282c22700 1 -- 172.21.15.42:0/378926232 shutdown_connections 2019-12-05T06:24:44.082 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:44.071+0000 7f4282c22700 1 -- 172.21.15.42:0/378926232 wait complete. 2019-12-05T06:24:44.082 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:44.071+0000 7f4282c22700 1 -- 172.21.15.42:0/378926232 >> 172.21.15.42:0/378926232 conn(0x7f427c0aabe0 msgr2=0x7f427c0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:24:44.085 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:44.071+0000 7f4282c22700 1 Processor -- start 2019-12-05T06:24:44.085 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:44.071+0000 7f4282c22700 1 -- start start 2019-12-05T06:24:44.085 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:44.075+0000 7f4282c22700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f427c07d0a0 0x7f427c0a3ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:24:44.086 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:44.075+0000 7f4282c22700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f427c07b7a0 0x7f427c07bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:24:44.086 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:44.075+0000 7f4282c22700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f427c07c120 con 0x7f427c142bf0 2019-12-05T06:24:44.086 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:44.075+0000 7f4282c22700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f427c0a4430 con 0x7f427c07d0a0 2019-12-05T06:24:44.086 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:44.075+0000 7f4282c22700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f427c1426b0 con 0x7f427c07b7a0 2019-12-05T06:24:44.086 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:44.075+0000 7f42809be700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f427c142bf0 0x7f427c1437c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:55018/0 (socket says 172.21.15.42:55018) 2019-12-05T06:24:44.086 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:44.075+0000 7f42809be700 1 -- 172.21.15.42:0/2892448711 learned_addr learned my addr 172.21.15.42:0/2892448711 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:24:44.086 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:44.075+0000 7f42811bf700 1 --2- 172.21.15.42:0/2892448711 >> v2:172.21.15.40:3300/0 conn(0x7f427c07d0a0 0x7f427c0a3ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:44.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:44.075+0000 7f42819c0700 1 --2- 172.21.15.42:0/2892448711 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f427c07b7a0 0x7f427c07bbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:44.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:44.075+0000 7f4271ffb700 1 -- 172.21.15.42:0/2892448711 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4119514100 0 0) 0x7f427c07c120 con 0x7f427c142bf0 2019-12-05T06:24:44.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:44.075+0000 7f4271ffb700 1 -- 172.21.15.42:0/2892448711 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f425c0030b0 con 0x7f427c142bf0 2019-12-05T06:24:44.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:44.075+0000 7f4271ffb700 1 -- 172.21.15.42:0/2892448711 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 3854119927 0 0) 0x7f425c0030b0 con 0x7f427c142bf0 2019-12-05T06:24:44.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:44.075+0000 7f42811bf700 1 -- 172.21.15.42:0/2892448711 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f427c07b7a0 msgr2=0x7f427c07bbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:44.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:44.075+0000 7f42811bf700 1 --2- 172.21.15.42:0/2892448711 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f427c07b7a0 0x7f427c07bbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:44.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:44.075+0000 7f42811bf700 1 -- 172.21.15.42:0/2892448711 >> v1:172.21.15.42:6789/0 conn(0x7f427c142bf0 legacy=0x7f427c1437c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:44.088 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:44.075+0000 7f42811bf700 1 -- 172.21.15.42:0/2892448711 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f427c143fc0 con 0x7f427c07d0a0 2019-12-05T06:24:44.088 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:44.075+0000 7f42819c0700 1 --2- 172.21.15.42:0/2892448711 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f427c07b7a0 0x7f427c07bbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:24:44.088 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:44.075+0000 7f42811bf700 1 --2- 172.21.15.42:0/2892448711 >> v2:172.21.15.40:3300/0 conn(0x7f427c07d0a0 0x7f427c0a3ab0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f426c001ca0 tx=0x7f426c0017b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:44.088 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:44.075+0000 7f42811bf700 1 --2- 172.21.15.42:0/2892448711 >> v2:172.21.15.40:3300/0 conn(0x7f427c07d0a0 0x7f427c0a3ab0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f426c001ca0 tx=0x7f426c0017b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:44.088 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:44.075+0000 7f42811bf700 1 --2- 172.21.15.42:0/2892448711 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f427c07d0a0 0x7f427c0a3ab0 secure :-1 s=READY pgs=1960 cs=0 l=1 rx=0x7f426c001ca0 tx=0x7f426c0017b0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:24:44.088 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:44.075+0000 7f42811bf700 1 --2- 172.21.15.42:0/2892448711 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f427c07d0a0 0x7f427c0a3ab0 secure :-1 s=THROTTLE_DONE pgs=1960 cs=0 l=1 rx=0x7f426c001ca0 tx=0x7f426c0017b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:44.088 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:44.075+0000 7f4271ffb700 1 -- 172.21.15.42:0/2892448711 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f426c001a60 con 0x7f427c07d0a0 2019-12-05T06:24:44.089 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:44.075+0000 7f42811bf700 1 --2- 172.21.15.42:0/2892448711 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f427c07d0a0 0x7f427c0a3ab0 secure :-1 s=THROTTLE_DONE pgs=1960 cs=0 l=1 rx=0x7f426c001ca0 tx=0x7f426c0017b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:44.089 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:44.075+0000 7f4271ffb700 1 -- 172.21.15.42:0/2892448711 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f426c001870 con 0x7f427c07d0a0 2019-12-05T06:24:44.089 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:44.075+0000 7f42811bf700 1 --2- 172.21.15.42:0/2892448711 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f427c07d0a0 0x7f427c0a3ab0 secure :-1 s=THROTTLE_DONE pgs=1960 cs=0 l=1 rx=0x7f426c001ca0 tx=0x7f426c0017b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:44.089 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:44.075+0000 7f4282c22700 1 -- 172.21.15.42:0/2892448711 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f427c115ec0 con 0x7f427c07d0a0 2019-12-05T06:24:44.089 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:44.075+0000 7f4271ffb700 1 -- 172.21.15.42:0/2892448711 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4278005b20 con 0x7f427c07d0a0 2019-12-05T06:24:44.089 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:44.075+0000 7f4282c22700 1 -- 172.21.15.42:0/2892448711 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f427c0a4000 con 0x7f427c07d0a0 2019-12-05T06:24:44.089 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:44.079+0000 7f42811bf700 1 --2- 172.21.15.42:0/2892448711 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f427c07d0a0 0x7f427c0a3ab0 secure :-1 s=THROTTLE_DONE pgs=1960 cs=0 l=1 rx=0x7f426c001ca0 tx=0x7f426c0017b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:44.090 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:44.079+0000 7f4271ffb700 1 -- 172.21.15.42:0/2892448711 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f4278007bb0 con 0x7f427c07d0a0 2019-12-05T06:24:44.091 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:44.079+0000 7f42811bf700 1 --2- 172.21.15.42:0/2892448711 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f427c07d0a0 0x7f427c0a3ab0 secure :-1 s=THROTTLE_DONE pgs=1960 cs=0 l=1 rx=0x7f426c001ca0 tx=0x7f426c0017b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:44.091 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:44.079+0000 7f4271ffb700 1 --2- 172.21.15.42:0/2892448711 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f425c03b230 0x7f425c03d740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:24:44.091 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:44.079+0000 7f4271ffb700 1 -- 172.21.15.42:0/2892448711 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 10847+0+0 (secure 0 0 0) 0x7f427804a780 con 0x7f427c07d0a0 2019-12-05T06:24:44.091 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:44.079+0000 7f42809be700 1 --2- 172.21.15.42:0/2892448711 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f425c03b230 0x7f425c03d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:44.092 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:44.079+0000 7f42809be700 1 --2- 172.21.15.42:0/2892448711 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f425c03b230 0x7f425c03d740 crc :-1 s=READY pgs=1089 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:24:44.092 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:44.083+0000 7f4282c22700 1 -- 172.21.15.42:0/2892448711 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f427c05d110 con 0x7f427c07d0a0 2019-12-05T06:24:44.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:44.091+0000 7f42811bf700 1 --2- 172.21.15.42:0/2892448711 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f427c07d0a0 0x7f427c0a3ab0 secure :-1 s=THROTTLE_DONE pgs=1960 cs=0 l=1 rx=0x7f426c001ca0 tx=0x7f426c0017b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:44.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:44.091+0000 7f4271ffb700 1 -- 172.21.15.42:0/2892448711 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f42780317e0 con 0x7f427c07d0a0 2019-12-05T06:24:44.135 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:24:44.139 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:24:44.268 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:44.263+0000 7f4282c22700 1 -- 172.21.15.42:0/2892448711 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f427c05b8c0 con 0x7f427c07d0a0 2019-12-05T06:24:44.273 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:24:44.273 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":312,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":312,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:24:39.885811+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":105,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7568},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7568":{"gid":7568,"name":"b","rank":0,"incarnation":309,"state":"up:active","state_seq":14,"addr":"172.21.15.40:6833/2324657302","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2324657302},{"type":"v1","addr":"172.21.15.40:6833","nonce":2324657302}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:24:44.273 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 312 2019-12-05T06:24:44.281 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:44.263+0000 7f42811bf700 1 --2- 172.21.15.42:0/2892448711 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f427c07d0a0 0x7f427c0a3ab0 secure :-1 s=THROTTLE_DONE pgs=1960 cs=0 l=1 rx=0x7f426c001ca0 tx=0x7f426c0017b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:44.281 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:44.263+0000 7f4271ffb700 1 -- 172.21.15.42:0/2892448711 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 312 v312) v1 ==== 95+0+1776 (secure 0 0 0) 0x7f4278031a40 con 0x7f427c07d0a0 2019-12-05T06:24:44.281 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:44.267+0000 7f4282c22700 1 -- 172.21.15.42:0/2892448711 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f425c03b230 msgr2=0x7f425c03d740 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:44.282 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:44.267+0000 7f4282c22700 1 --2- 172.21.15.42:0/2892448711 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f425c03b230 0x7f425c03d740 crc :-1 s=READY pgs=1089 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:44.282 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:44.267+0000 7f4282c22700 1 -- 172.21.15.42:0/2892448711 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f427c07d0a0 msgr2=0x7f427c0a3ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:44.282 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:44.267+0000 7f4282c22700 1 --2- 172.21.15.42:0/2892448711 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f427c07d0a0 0x7f427c0a3ab0 secure :-1 s=READY pgs=1960 cs=0 l=1 rx=0x7f426c001ca0 tx=0x7f426c0017b0).stop 2019-12-05T06:24:44.282 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:44.267+0000 7f4282c22700 1 -- 172.21.15.42:0/2892448711 shutdown_connections 2019-12-05T06:24:44.282 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:44.267+0000 7f4282c22700 1 --2- 172.21.15.42:0/2892448711 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f425c03b230 0x7f425c03d740 unknown :-1 s=CLOSED pgs=1089 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:44.282 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:44.267+0000 7f4282c22700 1 --2- 172.21.15.42:0/2892448711 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f427c07b7a0 0x7f427c07bbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:44.283 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:44.267+0000 7f4282c22700 1 --2- 172.21.15.42:0/2892448711 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f427c07d0a0 0x7f427c0a3ab0 unknown :-1 s=CLOSED pgs=1960 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:44.283 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:44.267+0000 7f4282c22700 1 -- 172.21.15.42:0/2892448711 shutdown_connections 2019-12-05T06:24:44.283 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:44.267+0000 7f4282c22700 1 -- 172.21.15.42:0/2892448711 wait complete. 2019-12-05T06:24:44.283 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:44.267+0000 7f4282c22700 1 -- 172.21.15.42:0/2892448711 >> 172.21.15.42:0/2892448711 conn(0x7f427c0aabe0 msgr2=0x7f427c116220 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:24:44.299 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2019-12-05T06:24:46.303 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:24:46.414 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:46.399+0000 7fde82aaf700 1 Processor -- start 2019-12-05T06:24:46.414 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:46.399+0000 7fde82aaf700 1 -- start start 2019-12-05T06:24:46.414 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:46.403+0000 7fde82aaf700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fde7c0a7740 0x7fde7c08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:24:46.414 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:46.403+0000 7fde82aaf700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fde7c09fae0 0x7fde7c07eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:24:46.414 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:46.403+0000 7fde82aaf700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde7c0a0710 con 0x7fde7c07e720 2019-12-05T06:24:46.414 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:46.403+0000 7fde82aaf700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde7c0a0b00 con 0x7fde7c09fae0 2019-12-05T06:24:46.415 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:46.403+0000 7fde82aaf700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde7c08e5b0 con 0x7fde7c0a7740 2019-12-05T06:24:46.415 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:46.403+0000 7fde8104c700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fde7c0a7740 0x7fde7c08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:46.415 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:46.403+0000 7fde8084b700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fde7c07e720 0x7fde7c09f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:55040/0 (socket says 172.21.15.42:55040) 2019-12-05T06:24:46.415 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:46.403+0000 7fde8084b700 1 -- 172.21.15.42:0/4080243136 learned_addr learned my addr 172.21.15.42:0/4080243136 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:24:46.415 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:46.403+0000 7fde8184d700 1 --2- 172.21.15.42:0/4080243136 >> v2:172.21.15.42:3300/0 conn(0x7fde7c09fae0 0x7fde7c07eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:46.415 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:46.403+0000 7fde73fff700 1 -- 172.21.15.42:0/4080243136 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1445876662 0 0) 0x7fde7c0a0710 con 0x7fde7c07e720 2019-12-05T06:24:46.415 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:46.403+0000 7fde73fff700 1 -- 172.21.15.42:0/4080243136 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde680030b0 con 0x7fde7c07e720 2019-12-05T06:24:46.416 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:46.403+0000 7fde8104c700 1 -- 172.21.15.42:0/4080243136 >> v2:172.21.15.42:3300/0 conn(0x7fde7c09fae0 msgr2=0x7fde7c07eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:24:46.416 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:46.403+0000 7fde8104c700 1 --2- 172.21.15.42:0/4080243136 >> v2:172.21.15.42:3300/0 conn(0x7fde7c09fae0 0x7fde7c07eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:24:46.416 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:46.403+0000 7fde8104c700 1 -- 172.21.15.42:0/4080243136 >> v1:172.21.15.42:6789/0 conn(0x7fde7c07e720 legacy=0x7fde7c09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:46.416 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:46.403+0000 7fde8104c700 1 -- 172.21.15.42:0/4080243136 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde7c07f0a0 con 0x7fde7c0a7740 2019-12-05T06:24:46.416 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:46.403+0000 7fde8104c700 1 --2- 172.21.15.42:0/4080243136 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fde7c0a7740 0x7fde7c08df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fde78008790 tx=0x7fde7800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:46.416 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:46.403+0000 7fde8104c700 1 --2- 172.21.15.42:0/4080243136 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fde7c0a7740 0x7fde7c08df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fde78008790 tx=0x7fde7800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:46.417 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:46.403+0000 7fde8104c700 1 --2- 172.21.15.42:0/4080243136 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fde7c0a7740 0x7fde7c08df80 secure :-1 s=READY pgs=565 cs=0 l=1 rx=0x7fde78008790 tx=0x7fde7800fc70).ready entity=mon.2 client_cookie=e22e8d5333ca0d22 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:24:46.417 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:46.403+0000 7fde8104c700 1 --2- 172.21.15.42:0/4080243136 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fde7c0a7740 0x7fde7c08df80 secure :-1 s=THROTTLE_DONE pgs=565 cs=0 l=1 rx=0x7fde78008790 tx=0x7fde7800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:46.417 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:46.403+0000 7fde8104c700 1 --2- 172.21.15.42:0/4080243136 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fde7c0a7740 0x7fde7c08df80 secure :-1 s=THROTTLE_DONE pgs=565 cs=0 l=1 rx=0x7fde78008790 tx=0x7fde7800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:46.417 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:46.403+0000 7fde73fff700 1 -- 172.21.15.42:0/4080243136 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fde78008bb0 con 0x7fde7c0a7740 2019-12-05T06:24:46.417 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:46.403+0000 7fde8104c700 1 --2- 172.21.15.42:0/4080243136 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fde7c0a7740 0x7fde7c08df80 secure :-1 s=THROTTLE_DONE pgs=565 cs=0 l=1 rx=0x7fde78008790 tx=0x7fde7800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:46.417 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:46.403+0000 7fde73fff700 1 -- 172.21.15.42:0/4080243136 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde780087d0 con 0x7fde7c0a7740 2019-12-05T06:24:46.417 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:46.403+0000 7fde73fff700 1 -- 172.21.15.42:0/4080243136 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fde78011860 con 0x7fde7c0a7740 2019-12-05T06:24:46.418 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:46.403+0000 7fde82aaf700 1 -- 172.21.15.42:0/4080243136 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fde7c0a7740 msgr2=0x7fde7c08df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:46.418 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:46.403+0000 7fde82aaf700 1 --2- 172.21.15.42:0/4080243136 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fde7c0a7740 0x7fde7c08df80 secure :-1 s=READY pgs=565 cs=0 l=1 rx=0x7fde78008790 tx=0x7fde7800fc70).stop 2019-12-05T06:24:46.418 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:46.407+0000 7fde82aaf700 1 -- 172.21.15.42:0/4080243136 shutdown_connections 2019-12-05T06:24:46.418 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:46.407+0000 7fde82aaf700 1 --2- 172.21.15.42:0/4080243136 >> v2:172.21.15.42:3300/0 conn(0x7fde7c09fae0 0x7fde7c07eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:24:46.418 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:46.407+0000 7fde82aaf700 1 --2- 172.21.15.42:0/4080243136 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fde7c0a7740 0x7fde7c08df80 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:46.418 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:46.407+0000 7fde82aaf700 1 -- 172.21.15.42:0/4080243136 shutdown_connections 2019-12-05T06:24:46.418 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:46.407+0000 7fde82aaf700 1 -- 172.21.15.42:0/4080243136 wait complete. 2019-12-05T06:24:46.419 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:46.407+0000 7fde82aaf700 1 -- 172.21.15.42:0/4080243136 >> 172.21.15.42:0/4080243136 conn(0x7fde7c0aabe0 msgr2=0x7fde7c0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:24:46.422 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:46.407+0000 7fde82aaf700 1 Processor -- start 2019-12-05T06:24:46.422 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:46.407+0000 7fde82aaf700 1 -- start start 2019-12-05T06:24:46.422 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:46.411+0000 7fde82aaf700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fde7c1195c0 0x7fde7c1199f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:24:46.423 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:46.411+0000 7fde82aaf700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fde7c11fcd0 0x7fde7c120100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:24:46.423 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:46.411+0000 7fde82aaf700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde7c11db20 con 0x7fde7c11bf40 2019-12-05T06:24:46.423 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:46.411+0000 7fde82aaf700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde7c130400 con 0x7fde7c1195c0 2019-12-05T06:24:46.423 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:46.411+0000 7fde82aaf700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde7c1306a0 con 0x7fde7c11fcd0 2019-12-05T06:24:46.423 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:46.411+0000 7fde8084b700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fde7c1195c0 0x7fde7c1199f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:46.423 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:46.411+0000 7fde8084b700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fde7c1195c0 0x7fde7c1199f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:55008/0 (socket says 172.21.15.42:55008) 2019-12-05T06:24:46.424 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:46.411+0000 7fde8084b700 1 -- 172.21.15.42:0/2313426042 learned_addr learned my addr 172.21.15.42:0/2313426042 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:24:46.424 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:46.411+0000 7fde8184d700 1 --2- 172.21.15.42:0/2313426042 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fde7c11fcd0 0x7fde7c120100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:46.424 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:46.411+0000 7fde8084b700 1 -- 172.21.15.42:0/2313426042 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fde7c11fcd0 msgr2=0x7fde7c120100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:46.424 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:46.411+0000 7fde8084b700 1 --2- 172.21.15.42:0/2313426042 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fde7c11fcd0 0x7fde7c120100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:46.424 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:46.411+0000 7fde8084b700 1 -- 172.21.15.42:0/2313426042 >> v1:172.21.15.40:6789/0 conn(0x7fde7c11bf40 legacy=0x7fde7c130940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:46.424 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:46.411+0000 7fde71ffb700 1 -- 172.21.15.42:0/2313426042 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3233377545 0 0) 0x7fde7c11db20 con 0x7fde7c11bf40 2019-12-05T06:24:46.424 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:46.411+0000 7fde8084b700 1 -- 172.21.15.42:0/2313426042 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde7c12ccc0 con 0x7fde7c1195c0 2019-12-05T06:24:46.425 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:46.411+0000 7fde8084b700 1 --2- 172.21.15.42:0/2313426042 >> v2:172.21.15.42:3300/0 conn(0x7fde7c1195c0 0x7fde7c1199f0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fde6c0043a0 tx=0x7fde6c010a00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:46.425 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:46.411+0000 7fde8084b700 1 --2- 172.21.15.42:0/2313426042 >> v2:172.21.15.42:3300/0 conn(0x7fde7c1195c0 0x7fde7c1199f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fde6c0043a0 tx=0x7fde6c010a00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:46.425 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:46.411+0000 7fde8084b700 1 --2- 172.21.15.42:0/2313426042 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fde7c1195c0 0x7fde7c1199f0 secure :-1 s=READY pgs=2629 cs=0 l=1 rx=0x7fde6c0043a0 tx=0x7fde6c010a00).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:24:46.425 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:46.411+0000 7fde8084b700 1 --2- 172.21.15.42:0/2313426042 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fde7c1195c0 0x7fde7c1199f0 secure :-1 s=THROTTLE_DONE pgs=2629 cs=0 l=1 rx=0x7fde6c0043a0 tx=0x7fde6c010a00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:46.425 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:46.411+0000 7fde71ffb700 1 -- 172.21.15.42:0/2313426042 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fde6c00c9a0 con 0x7fde7c1195c0 2019-12-05T06:24:46.426 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:46.411+0000 7fde8084b700 1 --2- 172.21.15.42:0/2313426042 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fde7c1195c0 0x7fde7c1199f0 secure :-1 s=THROTTLE_DONE pgs=2629 cs=0 l=1 rx=0x7fde6c0043a0 tx=0x7fde6c010a00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:46.426 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:46.411+0000 7fde82aaf700 1 -- 172.21.15.42:0/2313426042 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde7c08eb70 con 0x7fde7c1195c0 2019-12-05T06:24:46.426 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:46.411+0000 7fde71ffb700 1 -- 172.21.15.42:0/2313426042 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde6c010250 con 0x7fde7c1195c0 2019-12-05T06:24:46.426 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:46.411+0000 7fde8084b700 1 --2- 172.21.15.42:0/2313426042 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fde7c1195c0 0x7fde7c1199f0 secure :-1 s=THROTTLE_DONE pgs=2629 cs=0 l=1 rx=0x7fde6c0043a0 tx=0x7fde6c010a00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:46.426 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:46.411+0000 7fde71ffb700 1 -- 172.21.15.42:0/2313426042 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fde6c00d0c0 con 0x7fde7c1195c0 2019-12-05T06:24:46.426 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:46.411+0000 7fde82aaf700 1 -- 172.21.15.42:0/2313426042 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde7c08f0f0 con 0x7fde7c1195c0 2019-12-05T06:24:46.426 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:46.411+0000 7fde8084b700 1 --2- 172.21.15.42:0/2313426042 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fde7c1195c0 0x7fde7c1199f0 secure :-1 s=THROTTLE_DONE pgs=2629 cs=0 l=1 rx=0x7fde6c0043a0 tx=0x7fde6c010a00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:46.427 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:46.415+0000 7fde71ffb700 1 -- 172.21.15.42:0/2313426042 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fde6c005460 con 0x7fde7c1195c0 2019-12-05T06:24:46.427 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:46.415+0000 7fde8084b700 1 --2- 172.21.15.42:0/2313426042 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fde7c1195c0 0x7fde7c1199f0 secure :-1 s=THROTTLE_DONE pgs=2629 cs=0 l=1 rx=0x7fde6c0043a0 tx=0x7fde6c010a00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:46.427 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:46.415+0000 7fde71ffb700 1 --2- 172.21.15.42:0/2313426042 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fde5c03abc0 0x7fde5c03d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:24:46.427 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:46.415+0000 7fde71ffb700 1 -- 172.21.15.42:0/2313426042 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 10847+0+0 (secure 0 0 0) 0x7fde6c04ac70 con 0x7fde7c1195c0 2019-12-05T06:24:46.427 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:46.415+0000 7fde8104c700 1 --2- 172.21.15.42:0/2313426042 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fde5c03abc0 0x7fde5c03d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:46.428 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:46.415+0000 7fde8104c700 1 --2- 172.21.15.42:0/2313426042 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fde5c03abc0 0x7fde5c03d0d0 crc :-1 s=READY pgs=1091 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:24:46.428 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:46.415+0000 7fde82aaf700 1 -- 172.21.15.42:0/2313426042 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde7c131140 con 0x7fde7c1195c0 2019-12-05T06:24:46.430 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:46.423+0000 7fde8084b700 1 --2- 172.21.15.42:0/2313426042 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fde7c1195c0 0x7fde7c1199f0 secure :-1 s=THROTTLE_DONE pgs=2629 cs=0 l=1 rx=0x7fde6c0043a0 tx=0x7fde6c010a00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:46.431 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:46.423+0000 7fde71ffb700 1 -- 172.21.15.42:0/2313426042 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fde6c031c70 con 0x7fde7c1195c0 2019-12-05T06:24:46.593 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:46.587+0000 7fde82aaf700 1 -- 172.21.15.42:0/2313426042 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fde7c05d110 con 0x7fde7c1195c0 2019-12-05T06:24:46.595 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:46.587+0000 7fde8084b700 1 --2- 172.21.15.42:0/2313426042 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fde7c1195c0 0x7fde7c1199f0 secure :-1 s=THROTTLE_DONE pgs=2629 cs=0 l=1 rx=0x7fde6c0043a0 tx=0x7fde6c010a00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:46.595 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:46.587+0000 7fde71ffb700 1 -- 172.21.15.42:0/2313426042 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 313 v313) v1 ==== 95+0+2114 (secure 0 0 0) 0x7fde6c031ed0 con 0x7fde7c1195c0 2019-12-05T06:24:46.597 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 313 2019-12-05T06:24:46.597 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:24:46.597 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":313,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":11221,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/3427104185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3427104185},{"type":"v1","addr":"172.21.15.42:6835","nonce":3427104185}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":313}],"filesystems":[{"mdsmap":{"epoch":312,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:24:39.885811+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":105,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7568},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7568":{"gid":7568,"name":"b","rank":0,"incarnation":309,"state":"up:active","state_seq":14,"addr":"172.21.15.40:6833/2324657302","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2324657302},{"type":"v1","addr":"172.21.15.40:6833","nonce":2324657302}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:24:46.600 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:46.591+0000 7fde82aaf700 1 -- 172.21.15.42:0/2313426042 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fde5c03abc0 msgr2=0x7fde5c03d0d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:46.600 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:46.591+0000 7fde82aaf700 1 --2- 172.21.15.42:0/2313426042 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fde5c03abc0 0x7fde5c03d0d0 crc :-1 s=READY pgs=1091 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:46.600 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:46.591+0000 7fde82aaf700 1 -- 172.21.15.42:0/2313426042 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fde7c1195c0 msgr2=0x7fde7c1199f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:46.601 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:46.591+0000 7fde82aaf700 1 --2- 172.21.15.42:0/2313426042 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fde7c1195c0 0x7fde7c1199f0 secure :-1 s=READY pgs=2629 cs=0 l=1 rx=0x7fde6c0043a0 tx=0x7fde6c010a00).stop 2019-12-05T06:24:46.601 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:46.591+0000 7fde82aaf700 1 -- 172.21.15.42:0/2313426042 shutdown_connections 2019-12-05T06:24:46.601 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:46.591+0000 7fde82aaf700 1 --2- 172.21.15.42:0/2313426042 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fde5c03abc0 0x7fde5c03d0d0 unknown :-1 s=CLOSED pgs=1091 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:46.601 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:46.591+0000 7fde82aaf700 1 --2- 172.21.15.42:0/2313426042 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fde7c11fcd0 0x7fde7c120100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:46.602 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:46.591+0000 7fde82aaf700 1 --2- 172.21.15.42:0/2313426042 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fde7c1195c0 0x7fde7c1199f0 unknown :-1 s=CLOSED pgs=2629 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:46.602 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:46.591+0000 7fde82aaf700 1 -- 172.21.15.42:0/2313426042 shutdown_connections 2019-12-05T06:24:46.602 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:46.591+0000 7fde82aaf700 1 -- 172.21.15.42:0/2313426042 wait complete. 2019-12-05T06:24:46.602 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:24:46.591+0000 7fde82aaf700 1 -- 172.21.15.42:0/2313426042 >> 172.21.15.42:0/2313426042 conn(0x7fde7c07e8d0 msgr2=0x7fde7c0a7f90 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:24:46.609 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a reported in up:standby state 2019-12-05T06:24:46.609 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 98 secs before thrashing 2019-12-05T06:25:14.364 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:25:14.384 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:25:44.454 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:25:44.459 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:26:14.518 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:26:14.523 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:26:24.610 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:26:24.718 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:24.707+0000 7f36e7cbb700 1 Processor -- start 2019-12-05T06:26:24.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:24.707+0000 7f36e7cbb700 1 -- start start 2019-12-05T06:26:24.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:24.707+0000 7f36e7cbb700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f36e009fad0 0x7f36e014ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:26:24.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:24.707+0000 7f36e7cbb700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36e00a0700 con 0x7f36e008df70 2019-12-05T06:26:24.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:24.707+0000 7f36e7cbb700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36e007ef10 con 0x7f36e00a7730 2019-12-05T06:26:24.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:24.707+0000 7f36e7cbb700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36e00a0af0 con 0x7f36e009fad0 2019-12-05T06:26:24.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:24.707+0000 7f36e6258700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f36e00a7730 0x7f36e007e710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:55144/0 (socket says 172.21.15.42:55144) 2019-12-05T06:26:24.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:24.707+0000 7f36e6258700 1 -- 172.21.15.42:0/4190888693 learned_addr learned my addr 172.21.15.42:0/4190888693 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:26:24.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:24.707+0000 7f36e6a59700 1 --2- 172.21.15.42:0/4190888693 >> v2:172.21.15.40:3300/0 conn(0x7f36e009fad0 0x7f36e014ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:26:24.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:24.707+0000 7f36e5256700 1 -- 172.21.15.42:0/4190888693 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3094817632 0 0) 0x7f36e007ef10 con 0x7f36e00a7730 2019-12-05T06:26:24.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:24.707+0000 7f36e5256700 1 -- 172.21.15.42:0/4190888693 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36cc0030b0 con 0x7f36e00a7730 2019-12-05T06:26:24.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:24.711+0000 7f36e5256700 1 -- 172.21.15.42:0/4190888693 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2368101663 0 0) 0x7f36cc0030b0 con 0x7f36e00a7730 2019-12-05T06:26:24.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:24.711+0000 7f36e6a59700 1 -- 172.21.15.42:0/4190888693 >> v1:172.21.15.42:6789/0 conn(0x7f36e00a7730 legacy=0x7f36e007e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:24.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:24.711+0000 7f36e6a59700 1 -- 172.21.15.42:0/4190888693 >> v1:172.21.15.40:6789/0 conn(0x7f36e008df70 legacy=0x7f36e009f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:24.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:24.711+0000 7f36e6a59700 1 -- 172.21.15.42:0/4190888693 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36e00a9b90 con 0x7f36e009fad0 2019-12-05T06:26:24.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:24.711+0000 7f36e6a59700 1 --2- 172.21.15.42:0/4190888693 >> v2:172.21.15.40:3300/0 conn(0x7f36e009fad0 0x7f36e014ffe0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f36d0001110 tx=0x7f36d0001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:24.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:24.711+0000 7f36e6a59700 1 --2- 172.21.15.42:0/4190888693 >> v2:172.21.15.40:3300/0 conn(0x7f36e009fad0 0x7f36e014ffe0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f36d0001110 tx=0x7f36d0001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:24.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:24.711+0000 7f36e6a59700 1 --2- 172.21.15.42:0/4190888693 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f36e009fad0 0x7f36e014ffe0 secure :-1 s=READY pgs=1964 cs=0 l=1 rx=0x7f36d0001110 tx=0x7f36d0001150).ready entity=mon.0 client_cookie=69bc62a176eedbe3 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:26:24.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:24.711+0000 7f36e6a59700 1 --2- 172.21.15.42:0/4190888693 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f36e009fad0 0x7f36e014ffe0 secure :-1 s=THROTTLE_DONE pgs=1964 cs=0 l=1 rx=0x7f36d0001110 tx=0x7f36d0001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:24.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:24.711+0000 7f36e5256700 1 -- 172.21.15.42:0/4190888693 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f36dc001750 con 0x7f36e009fad0 2019-12-05T06:26:24.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:24.711+0000 7f36e6a59700 1 --2- 172.21.15.42:0/4190888693 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f36e009fad0 0x7f36e014ffe0 secure :-1 s=THROTTLE_DONE pgs=1964 cs=0 l=1 rx=0x7f36d0001110 tx=0x7f36d0001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:24.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:24.711+0000 7f36e6a59700 1 --2- 172.21.15.42:0/4190888693 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f36e009fad0 0x7f36e014ffe0 secure :-1 s=THROTTLE_DONE pgs=1964 cs=0 l=1 rx=0x7f36d0001110 tx=0x7f36d0001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:24.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:24.711+0000 7f36e5256700 1 -- 172.21.15.42:0/4190888693 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36dc001560 con 0x7f36e009fad0 2019-12-05T06:26:24.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:24.711+0000 7f36e5256700 1 -- 172.21.15.42:0/4190888693 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f36d80109e0 con 0x7f36e009fad0 2019-12-05T06:26:24.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:24.711+0000 7f36e7cbb700 1 -- 172.21.15.42:0/4190888693 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f36e009fad0 msgr2=0x7f36e014ffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:24.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:24.711+0000 7f36e7cbb700 1 --2- 172.21.15.42:0/4190888693 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f36e009fad0 0x7f36e014ffe0 secure :-1 s=READY pgs=1964 cs=0 l=1 rx=0x7f36d0001110 tx=0x7f36d0001150).stop 2019-12-05T06:26:24.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:24.711+0000 7f36e7cbb700 1 -- 172.21.15.42:0/4190888693 shutdown_connections 2019-12-05T06:26:24.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:24.711+0000 7f36e7cbb700 1 --2- 172.21.15.42:0/4190888693 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f36e009fad0 0x7f36e014ffe0 unknown :-1 s=CLOSED pgs=1964 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:24.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:24.711+0000 7f36e7cbb700 1 -- 172.21.15.42:0/4190888693 shutdown_connections 2019-12-05T06:26:24.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:24.711+0000 7f36e7cbb700 1 -- 172.21.15.42:0/4190888693 wait complete. 2019-12-05T06:26:24.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:24.711+0000 7f36e7cbb700 1 -- 172.21.15.42:0/4190888693 >> 172.21.15.42:0/4190888693 conn(0x7f36e00aabd0 msgr2=0x7f36e00a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:26:24.726 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:24.715+0000 7f36e7cbb700 1 Processor -- start 2019-12-05T06:26:24.726 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:24.715+0000 7f36e7cbb700 1 -- start start 2019-12-05T06:26:24.726 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:24.715+0000 7f36e7cbb700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f36e0136300 0x7f36e00a52d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:26:24.726 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:24.715+0000 7f36e7cbb700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36e0079b10 con 0x7f36e00a29c0 2019-12-05T06:26:24.727 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:24.715+0000 7f36e7cbb700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36e007ad50 con 0x7f36e0123670 2019-12-05T06:26:24.727 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:24.715+0000 7f36e7cbb700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36e00a5a50 con 0x7f36e0136300 2019-12-05T06:26:24.727 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:24.715+0000 7f36e6258700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f36e0123670 0x7f36e0135b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:55150/0 (socket says 172.21.15.42:55150) 2019-12-05T06:26:24.727 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:24.715+0000 7f36e6258700 1 -- 172.21.15.42:0/2748774793 learned_addr learned my addr 172.21.15.42:0/2748774793 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:26:24.727 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:24.715+0000 7f36e6a59700 1 --2- 172.21.15.42:0/2748774793 >> v2:172.21.15.40:3300/0 conn(0x7f36e0136300 0x7f36e00a52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:26:24.727 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:24.715+0000 7f36d6ffd700 1 -- 172.21.15.42:0/2748774793 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 193244875 0 0) 0x7f36e007ad50 con 0x7f36e0123670 2019-12-05T06:26:24.727 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:24.715+0000 7f36d6ffd700 1 -- 172.21.15.42:0/2748774793 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36c80030b0 con 0x7f36e0123670 2019-12-05T06:26:24.728 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:24.715+0000 7f36e6a59700 1 -- 172.21.15.42:0/2748774793 >> v1:172.21.15.42:6789/0 conn(0x7f36e0123670 legacy=0x7f36e0135b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:24.728 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:24.715+0000 7f36d6ffd700 1 -- 172.21.15.42:0/2748774793 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1357014993 0 0) 0x7f36e0079b10 con 0x7f36e00a29c0 2019-12-05T06:26:24.728 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:24.715+0000 7f36e6a59700 1 -- 172.21.15.42:0/2748774793 >> v1:172.21.15.40:6789/0 conn(0x7f36e00a29c0 legacy=0x7f36e00762f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:24.728 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:24.715+0000 7f36e6a59700 1 -- 172.21.15.42:0/2748774793 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36e007bf90 con 0x7f36e0136300 2019-12-05T06:26:24.728 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:24.715+0000 7f36e6a59700 1 --2- 172.21.15.42:0/2748774793 >> v2:172.21.15.40:3300/0 conn(0x7f36e0136300 0x7f36e00a52d0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f36dc001710 tx=0x7f36dc001260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:24.728 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:24.715+0000 7f36e6a59700 1 --2- 172.21.15.42:0/2748774793 >> v2:172.21.15.40:3300/0 conn(0x7f36e0136300 0x7f36e00a52d0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f36dc001710 tx=0x7f36dc001260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:24.728 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:24.715+0000 7f36e6a59700 1 --2- 172.21.15.42:0/2748774793 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f36e0136300 0x7f36e00a52d0 secure :-1 s=READY pgs=1966 cs=0 l=1 rx=0x7f36dc001710 tx=0x7f36dc001260).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:26:24.729 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:24.715+0000 7f36e6a59700 1 --2- 172.21.15.42:0/2748774793 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f36e0136300 0x7f36e00a52d0 secure :-1 s=THROTTLE_DONE pgs=1966 cs=0 l=1 rx=0x7f36dc001710 tx=0x7f36dc001260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:24.729 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:24.715+0000 7f36d6ffd700 1 -- 172.21.15.42:0/2748774793 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f36dc001510 con 0x7f36e0136300 2019-12-05T06:26:24.729 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:24.715+0000 7f36e7cbb700 1 -- 172.21.15.42:0/2748774793 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36e0079d40 con 0x7f36e0136300 2019-12-05T06:26:24.729 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:24.715+0000 7f36e6a59700 1 --2- 172.21.15.42:0/2748774793 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f36e0136300 0x7f36e00a52d0 secure :-1 s=THROTTLE_DONE pgs=1966 cs=0 l=1 rx=0x7f36dc001710 tx=0x7f36dc001260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:24.729 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:24.715+0000 7f36e6a59700 1 --2- 172.21.15.42:0/2748774793 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f36e0136300 0x7f36e00a52d0 secure :-1 s=THROTTLE_DONE pgs=1966 cs=0 l=1 rx=0x7f36dc001710 tx=0x7f36dc001260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:24.729 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:24.715+0000 7f36d6ffd700 1 -- 172.21.15.42:0/2748774793 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36dc001320 con 0x7f36e0136300 2019-12-05T06:26:24.729 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:24.715+0000 7f36d6ffd700 1 -- 172.21.15.42:0/2748774793 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f36d800f7f0 con 0x7f36e0136300 2019-12-05T06:26:24.730 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:24.715+0000 7f36e7cbb700 1 -- 172.21.15.42:0/2748774793 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36e007a290 con 0x7f36e0136300 2019-12-05T06:26:24.730 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:24.719+0000 7f36e6a59700 1 --2- 172.21.15.42:0/2748774793 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f36e0136300 0x7f36e00a52d0 secure :-1 s=THROTTLE_DONE pgs=1966 cs=0 l=1 rx=0x7f36dc001710 tx=0x7f36dc001260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:24.730 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:24.719+0000 7f36d6ffd700 1 -- 172.21.15.42:0/2748774793 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f36d80058c0 con 0x7f36e0136300 2019-12-05T06:26:24.730 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:24.719+0000 7f36e6a59700 1 --2- 172.21.15.42:0/2748774793 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f36e0136300 0x7f36e00a52d0 secure :-1 s=THROTTLE_DONE pgs=1966 cs=0 l=1 rx=0x7f36dc001710 tx=0x7f36dc001260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:24.730 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:24.719+0000 7f36d6ffd700 1 --2- 172.21.15.42:0/2748774793 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f36c803b220 0x7f36c803d730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:26:24.730 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:24.719+0000 7f36d6ffd700 1 -- 172.21.15.42:0/2748774793 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 10847+0+0 (secure 0 0 0) 0x7f36d804a8f0 con 0x7f36e0136300 2019-12-05T06:26:24.730 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:24.719+0000 7f36e6258700 1 --2- 172.21.15.42:0/2748774793 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f36c803b220 0x7f36c803d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:26:24.731 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:24.719+0000 7f36e6258700 1 --2- 172.21.15.42:0/2748774793 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f36c803b220 0x7f36c803d730 crc :-1 s=READY pgs=1097 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:26:24.731 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:24.723+0000 7f36e7cbb700 1 -- 172.21.15.42:0/2748774793 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36e005d100 con 0x7f36e0136300 2019-12-05T06:26:24.737 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:24.731+0000 7f36e6a59700 1 --2- 172.21.15.42:0/2748774793 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f36e0136300 0x7f36e00a52d0 secure :-1 s=THROTTLE_DONE pgs=1966 cs=0 l=1 rx=0x7f36dc001710 tx=0x7f36dc001260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:24.737 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:24.731+0000 7f36d6ffd700 1 -- 172.21.15.42:0/2748774793 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f36d8031970 con 0x7f36e0136300 2019-12-05T06:26:24.940 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:24.935+0000 7f36e7cbb700 1 -- 172.21.15.42:0/2748774793 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f36e005b8b0 con 0x7f36e0136300 2019-12-05T06:26:24.941 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:24.935+0000 7f36e6a59700 1 --2- 172.21.15.42:0/2748774793 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f36e0136300 0x7f36e00a52d0 secure :-1 s=THROTTLE_DONE pgs=1966 cs=0 l=1 rx=0x7f36dc001710 tx=0x7f36dc001260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:24.941 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:24.935+0000 7f36d6ffd700 1 -- 172.21.15.42:0/2748774793 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 313 v313) v1 ==== 95+0+2114 (secure 0 0 0) 0x7f36d8030de0 con 0x7f36e0136300 2019-12-05T06:26:24.943 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 313 2019-12-05T06:26:24.943 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:26:24.943 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":313,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":11221,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/3427104185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3427104185},{"type":"v1","addr":"172.21.15.42:6835","nonce":3427104185}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":313}],"filesystems":[{"mdsmap":{"epoch":312,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:24:39.885811+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":105,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7568},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7568":{"gid":7568,"name":"b","rank":0,"incarnation":309,"state":"up:active","state_seq":14,"addr":"172.21.15.40:6833/2324657302","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2324657302},{"type":"v1","addr":"172.21.15.40:6833","nonce":2324657302}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:26:24.946 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:24.939+0000 7f36e7cbb700 1 -- 172.21.15.42:0/2748774793 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f36c803b220 msgr2=0x7f36c803d730 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:24.946 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:24.939+0000 7f36e7cbb700 1 --2- 172.21.15.42:0/2748774793 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f36c803b220 0x7f36c803d730 crc :-1 s=READY pgs=1097 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:24.946 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:24.939+0000 7f36e7cbb700 1 -- 172.21.15.42:0/2748774793 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f36e0136300 msgr2=0x7f36e00a52d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:24.946 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:24.939+0000 7f36e7cbb700 1 --2- 172.21.15.42:0/2748774793 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f36e0136300 0x7f36e00a52d0 secure :-1 s=READY pgs=1966 cs=0 l=1 rx=0x7f36dc001710 tx=0x7f36dc001260).stop 2019-12-05T06:26:24.946 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:24.939+0000 7f36e7cbb700 1 -- 172.21.15.42:0/2748774793 shutdown_connections 2019-12-05T06:26:24.947 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:24.939+0000 7f36e7cbb700 1 --2- 172.21.15.42:0/2748774793 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f36c803b220 0x7f36c803d730 unknown :-1 s=CLOSED pgs=1097 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:24.947 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:24.939+0000 7f36e7cbb700 1 --2- 172.21.15.42:0/2748774793 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f36e0136300 0x7f36e00a52d0 unknown :-1 s=CLOSED pgs=1966 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:24.947 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:24.939+0000 7f36e7cbb700 1 -- 172.21.15.42:0/2748774793 shutdown_connections 2019-12-05T06:26:24.947 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:24.939+0000 7f36e7cbb700 1 -- 172.21.15.42:0/2748774793 wait complete. 2019-12-05T06:26:24.947 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:24.939+0000 7f36e7cbb700 1 -- 172.21.15.42:0/2748774793 >> 172.21.15.42:0/2748774793 conn(0x7f36e008fdd0 msgr2=0x7f36e007e910 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:26:24.957 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.b (rank=0) 2019-12-05T06:26:24.958 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2019-12-05T06:26:24.958 INFO:teuthology.orchestra.run:waiting for 300 2019-12-05T06:26:24.973 INFO:tasks.ceph.mds.b:Stopped 2019-12-05T06:26:24.973 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:26:25.077 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:25.067+0000 7f5a52210700 1 Processor -- start 2019-12-05T06:26:25.077 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:25.067+0000 7f5a52210700 1 -- start start 2019-12-05T06:26:25.077 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:25.067+0000 7f5a52210700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5a4c0a7740 0x7f5a4c08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:26:25.077 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:25.067+0000 7f5a52210700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a4c0a0710 con 0x7f5a4c07e720 2019-12-05T06:26:25.078 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:25.067+0000 7f5a52210700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a4c0a9ba0 con 0x7f5a4c09fae0 2019-12-05T06:26:25.078 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:25.067+0000 7f5a52210700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a4c0a0b00 con 0x7f5a4c0a7740 2019-12-05T06:26:25.078 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:25.067+0000 7f5a50fae700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f5a4c09fae0 0x7f5a4c14efe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:55164/0 (socket says 172.21.15.42:55164) 2019-12-05T06:26:25.078 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:25.067+0000 7f5a50fae700 1 -- 172.21.15.42:0/3965206866 learned_addr learned my addr 172.21.15.42:0/3965206866 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:26:25.078 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:25.067+0000 7f5a4bfff700 1 --2- 172.21.15.42:0/3965206866 >> v2:172.21.15.40:3300/0 conn(0x7f5a4c0a7740 0x7f5a4c08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:26:25.078 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:25.067+0000 7f5a4affd700 1 -- 172.21.15.42:0/3965206866 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 664653583 0 0) 0x7f5a4c0a9ba0 con 0x7f5a4c09fae0 2019-12-05T06:26:25.078 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:25.067+0000 7f5a4affd700 1 -- 172.21.15.42:0/3965206866 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a380030b0 con 0x7f5a4c09fae0 2019-12-05T06:26:25.079 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:25.067+0000 7f5a4affd700 1 -- 172.21.15.42:0/3965206866 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2450975823 0 0) 0x7f5a380030b0 con 0x7f5a4c09fae0 2019-12-05T06:26:25.079 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:25.067+0000 7f5a4affd700 1 -- 172.21.15.42:0/3965206866 >> v2:172.21.15.40:3300/0 conn(0x7f5a4c0a7740 msgr2=0x7f5a4c08df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:26:25.079 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:25.067+0000 7f5a4affd700 1 --2- 172.21.15.42:0/3965206866 >> v2:172.21.15.40:3300/0 conn(0x7f5a4c0a7740 0x7f5a4c08df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:26:25.079 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:25.067+0000 7f5a4affd700 1 -- 172.21.15.42:0/3965206866 >> v1:172.21.15.40:6789/0 conn(0x7f5a4c07e720 legacy=0x7f5a4c09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:25.079 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:25.067+0000 7f5a4affd700 1 -- 172.21.15.42:0/3965206866 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a4c07eb50 con 0x7f5a4c09fae0 2019-12-05T06:26:25.079 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:25.067+0000 7f5a4affd700 1 -- 172.21.15.42:0/3965206866 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7f5a3c001750 con 0x7f5a4c09fae0 2019-12-05T06:26:25.079 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:25.067+0000 7f5a4affd700 1 -- 172.21.15.42:0/3965206866 >> v1:172.21.15.42:6789/0 conn(0x7f5a4c09fae0 legacy=0x7f5a4c14efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:25.080 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:25.067+0000 7f5a4affd700 1 --2- 172.21.15.42:0/3965206866 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5a380036e0 0x7f5a38003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:26:25.080 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:25.067+0000 7f5a4affd700 1 --2- 172.21.15.42:0/3965206866 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5a38004420 0x7f5a38004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:26:25.080 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:25.067+0000 7f5a4affd700 1 --2- 172.21.15.42:0/3965206866 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5a38004df0 0x7f5a3800b670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:26:25.080 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:25.067+0000 7f5a4affd700 1 -- 172.21.15.42:0/3965206866 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a4c0a0b00 con 0x7f5a380036e0 2019-12-05T06:26:25.080 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:25.067+0000 7f5a4affd700 1 -- 172.21.15.42:0/3965206866 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a3804fce0 con 0x7f5a38004df0 2019-12-05T06:26:25.080 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:25.067+0000 7f5a4affd700 1 -- 172.21.15.42:0/3965206866 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a3804ff20 con 0x7f5a38004420 2019-12-05T06:26:25.080 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:25.067+0000 7f5a50fae700 1 --2- 172.21.15.42:0/3965206866 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5a38004df0 0x7f5a3800b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:26:25.081 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:25.067+0000 7f5a4bfff700 1 --2- 172.21.15.42:0/3965206866 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5a380036e0 0x7f5a38003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:26:25.081 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:25.067+0000 7f5a4b7fe700 1 --2- 172.21.15.42:0/3965206866 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5a38004420 0x7f5a38004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:26:25.081 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:25.067+0000 7f5a50fae700 1 -- 172.21.15.42:0/3965206866 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5a38004420 msgr2=0x7f5a38004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:26:25.081 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:25.067+0000 7f5a50fae700 1 --2- 172.21.15.42:0/3965206866 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5a38004420 0x7f5a38004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:26:25.081 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:25.067+0000 7f5a50fae700 1 -- 172.21.15.42:0/3965206866 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5a380036e0 msgr2=0x7f5a38003b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:26:25.081 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:25.067+0000 7f5a50fae700 1 --2- 172.21.15.42:0/3965206866 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5a380036e0 0x7f5a38003b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:26:25.081 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:25.067+0000 7f5a50fae700 1 -- 172.21.15.42:0/3965206866 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a38050220 con 0x7f5a38004df0 2019-12-05T06:26:25.082 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:25.067+0000 7f5a50fae700 1 --2- 172.21.15.42:0/3965206866 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5a38004df0 0x7f5a3800b670 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f5a3c001950 tx=0x7f5a3c00c240).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:25.082 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:25.067+0000 7f5a50fae700 1 -- 172.21.15.42:0/3965206866 reap_dead start 2019-12-05T06:26:25.082 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:25.067+0000 7f5a50fae700 1 --2- 172.21.15.42:0/3965206866 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5a38004df0 0x7f5a3800b670 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f5a3c001950 tx=0x7f5a3c00c240).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:25.082 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:25.067+0000 7f5a50fae700 1 --2- 172.21.15.42:0/3965206866 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5a38004df0 0x7f5a3800b670 secure :-1 s=READY pgs=2635 cs=0 l=1 rx=0x7f5a3c001950 tx=0x7f5a3c00c240).ready entity=mon.1 client_cookie=50da35b888924a5d server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:26:25.082 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:25.067+0000 7f5a50fae700 1 --2- 172.21.15.42:0/3965206866 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5a38004df0 0x7f5a3800b670 secure :-1 s=THROTTLE_DONE pgs=2635 cs=0 l=1 rx=0x7f5a3c001950 tx=0x7f5a3c00c240).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:25.082 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:25.067+0000 7f5a50fae700 1 --2- 172.21.15.42:0/3965206866 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5a38004df0 0x7f5a3800b670 secure :-1 s=THROTTLE_DONE pgs=2635 cs=0 l=1 rx=0x7f5a3c001950 tx=0x7f5a3c00c240).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:25.082 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:25.067+0000 7f5a50fae700 1 --2- 172.21.15.42:0/3965206866 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5a38004df0 0x7f5a3800b670 secure :-1 s=THROTTLE_DONE pgs=2635 cs=0 l=1 rx=0x7f5a3c001950 tx=0x7f5a3c00c240).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:25.083 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:25.067+0000 7f5a4affd700 1 -- 172.21.15.42:0/3965206866 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5a3c010350 con 0x7f5a38004df0 2019-12-05T06:26:25.083 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:25.067+0000 7f5a4affd700 1 -- 172.21.15.42:0/3965206866 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a3c00ec80 con 0x7f5a38004df0 2019-12-05T06:26:25.083 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:25.067+0000 7f5a4affd700 1 -- 172.21.15.42:0/3965206866 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5a3c005460 con 0x7f5a38004df0 2019-12-05T06:26:25.083 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:25.067+0000 7f5a52210700 1 -- 172.21.15.42:0/3965206866 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5a38004df0 msgr2=0x7f5a3800b670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:25.083 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:25.067+0000 7f5a52210700 1 --2- 172.21.15.42:0/3965206866 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5a38004df0 0x7f5a3800b670 secure :-1 s=READY pgs=2635 cs=0 l=1 rx=0x7f5a3c001950 tx=0x7f5a3c00c240).stop 2019-12-05T06:26:25.083 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:25.071+0000 7f5a52210700 1 -- 172.21.15.42:0/3965206866 shutdown_connections 2019-12-05T06:26:25.083 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:25.071+0000 7f5a52210700 1 --2- 172.21.15.42:0/3965206866 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5a38004df0 0x7f5a3800b670 unknown :-1 s=CLOSED pgs=2635 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:25.084 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:25.071+0000 7f5a52210700 1 -- 172.21.15.42:0/3965206866 shutdown_connections 2019-12-05T06:26:25.084 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:25.071+0000 7f5a52210700 1 -- 172.21.15.42:0/3965206866 wait complete. 2019-12-05T06:26:25.084 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:25.071+0000 7f5a52210700 1 -- 172.21.15.42:0/3965206866 >> 172.21.15.42:0/3965206866 conn(0x7f5a4c0aabe0 msgr2=0x7f5a4c0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:26:25.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:25.071+0000 7f5a52210700 1 Processor -- start 2019-12-05T06:26:25.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:25.071+0000 7f5a52210700 1 -- start start 2019-12-05T06:26:25.088 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:25.071+0000 7f5a52210700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5a4c11bbe0 0x7f5a4c138b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:26:25.088 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:25.071+0000 7f5a52210700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a4c0a4240 con 0x7f5a4c11fd40 2019-12-05T06:26:25.088 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:25.071+0000 7f5a52210700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a4c11c010 con 0x7f5a4c12c970 2019-12-05T06:26:25.088 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:25.071+0000 7f5a52210700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a4c12cda0 con 0x7f5a4c11bbe0 2019-12-05T06:26:25.088 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:25.071+0000 7f5a4bfff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f5a4c12c970 0x7f5a4c0a3830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:55172/0 (socket says 172.21.15.42:55172) 2019-12-05T06:26:25.088 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:25.071+0000 7f5a4bfff700 1 -- 172.21.15.42:0/996867912 learned_addr learned my addr 172.21.15.42:0/996867912 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:26:25.088 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:25.071+0000 7f5a50fae700 1 --2- 172.21.15.42:0/996867912 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5a4c11bbe0 0x7f5a4c138b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:26:25.089 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:25.071+0000 7f5a48ff9700 1 -- 172.21.15.42:0/996867912 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2182520595 0 0) 0x7f5a4c11c010 con 0x7f5a4c12c970 2019-12-05T06:26:25.089 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:25.071+0000 7f5a48ff9700 1 -- 172.21.15.42:0/996867912 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a300030b0 con 0x7f5a4c12c970 2019-12-05T06:26:25.089 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:25.075+0000 7f5a48ff9700 1 -- 172.21.15.42:0/996867912 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1978426342 0 0) 0x7f5a4c0a4240 con 0x7f5a4c11fd40 2019-12-05T06:26:25.089 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:25.075+0000 7f5a48ff9700 1 -- 172.21.15.42:0/996867912 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a4c11c010 con 0x7f5a4c11fd40 2019-12-05T06:26:25.089 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:25.075+0000 7f5a48ff9700 1 -- 172.21.15.42:0/996867912 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 2373630808 0 0) 0x7f5a300030b0 con 0x7f5a4c12c970 2019-12-05T06:26:25.089 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:25.075+0000 7f5a50fae700 1 -- 172.21.15.42:0/996867912 >> v1:172.21.15.42:6789/0 conn(0x7f5a4c12c970 legacy=0x7f5a4c0a3830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:25.089 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:25.075+0000 7f5a50fae700 1 -- 172.21.15.42:0/996867912 >> v1:172.21.15.40:6789/0 conn(0x7f5a4c11fd40 legacy=0x7f5a4c135370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:25.090 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:25.075+0000 7f5a50fae700 1 -- 172.21.15.42:0/996867912 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a4c0a54f0 con 0x7f5a4c11bbe0 2019-12-05T06:26:25.090 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:25.075+0000 7f5a50fae700 1 --2- 172.21.15.42:0/996867912 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5a4c11bbe0 0x7f5a4c138b90 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f5a34007ca0 tx=0x7f5a44001210).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:25.090 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:25.075+0000 7f5a50fae700 1 --2- 172.21.15.42:0/996867912 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5a4c11bbe0 0x7f5a4c138b90 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f5a34007ca0 tx=0x7f5a44001210).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:25.090 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:25.075+0000 7f5a50fae700 1 --2- 172.21.15.42:0/996867912 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5a4c11bbe0 0x7f5a4c138b90 secure :-1 s=READY pgs=570 cs=0 l=1 rx=0x7f5a34007ca0 tx=0x7f5a44001210).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:26:25.090 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:25.075+0000 7f5a50fae700 1 --2- 172.21.15.42:0/996867912 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5a4c11bbe0 0x7f5a4c138b90 secure :-1 s=THROTTLE_DONE pgs=570 cs=0 l=1 rx=0x7f5a34007ca0 tx=0x7f5a44001210).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:25.090 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:25.075+0000 7f5a48ff9700 1 -- 172.21.15.42:0/996867912 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5a34001490 con 0x7f5a4c11bbe0 2019-12-05T06:26:25.090 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:25.075+0000 7f5a50fae700 1 --2- 172.21.15.42:0/996867912 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5a4c11bbe0 0x7f5a4c138b90 secure :-1 s=THROTTLE_DONE pgs=570 cs=0 l=1 rx=0x7f5a34007ca0 tx=0x7f5a44001210).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:25.091 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:25.075+0000 7f5a48ff9700 1 -- 172.21.15.42:0/996867912 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a340012a0 con 0x7f5a4c11bbe0 2019-12-05T06:26:25.091 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:25.075+0000 7f5a50fae700 1 --2- 172.21.15.42:0/996867912 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5a4c11bbe0 0x7f5a4c138b90 secure :-1 s=THROTTLE_DONE pgs=570 cs=0 l=1 rx=0x7f5a34007ca0 tx=0x7f5a44001210).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:25.091 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:25.075+0000 7f5a52210700 1 -- 172.21.15.42:0/996867912 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a4c1390e0 con 0x7f5a4c11bbe0 2019-12-05T06:26:25.091 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:25.075+0000 7f5a48ff9700 1 -- 172.21.15.42:0/996867912 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5a3c010350 con 0x7f5a4c11bbe0 2019-12-05T06:26:25.091 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:25.075+0000 7f5a52210700 1 -- 172.21.15.42:0/996867912 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a4c1395b0 con 0x7f5a4c11bbe0 2019-12-05T06:26:25.091 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:25.075+0000 7f5a50fae700 1 --2- 172.21.15.42:0/996867912 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5a4c11bbe0 0x7f5a4c138b90 secure :-1 s=THROTTLE_DONE pgs=570 cs=0 l=1 rx=0x7f5a34007ca0 tx=0x7f5a44001210).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:25.091 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:25.075+0000 7f5a48ff9700 1 -- 172.21.15.42:0/996867912 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f5a3c010b10 con 0x7f5a4c11bbe0 2019-12-05T06:26:25.092 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:25.075+0000 7f5a50fae700 1 --2- 172.21.15.42:0/996867912 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5a4c11bbe0 0x7f5a4c138b90 secure :-1 s=THROTTLE_DONE pgs=570 cs=0 l=1 rx=0x7f5a34007ca0 tx=0x7f5a44001210).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:25.092 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:25.075+0000 7f5a48ff9700 1 --2- 172.21.15.42:0/996867912 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5a3003b400 0x7f5a3003d910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:26:25.092 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:25.075+0000 7f5a48ff9700 1 -- 172.21.15.42:0/996867912 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 10847+0+0 (secure 0 0 0) 0x7f5a3c04a930 con 0x7f5a4c11bbe0 2019-12-05T06:26:25.092 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:25.075+0000 7f5a4bfff700 1 --2- 172.21.15.42:0/996867912 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5a3003b400 0x7f5a3003d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:26:25.092 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:25.075+0000 7f5a4bfff700 1 --2- 172.21.15.42:0/996867912 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5a3003b400 0x7f5a3003d910 crc :-1 s=READY pgs=1098 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:26:25.092 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:25.079+0000 7f5a52210700 1 -- 172.21.15.42:0/996867912 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a4c0a4cd0 con 0x7f5a4c11bbe0 2019-12-05T06:26:25.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:25.091+0000 7f5a50fae700 1 --2- 172.21.15.42:0/996867912 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5a4c11bbe0 0x7f5a4c138b90 secure :-1 s=THROTTLE_DONE pgs=570 cs=0 l=1 rx=0x7f5a34007ca0 tx=0x7f5a44001210).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:25.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:25.091+0000 7f5a48ff9700 1 -- 172.21.15.42:0/996867912 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f5a3c0327e0 con 0x7f5a4c11bbe0 2019-12-05T06:26:25.274 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:25.267+0000 7f5a52210700 1 -- 172.21.15.42:0/996867912 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5a4c05d110 con 0x7f5a4c11bbe0 2019-12-05T06:26:25.275 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:25.267+0000 7f5a50fae700 1 --2- 172.21.15.42:0/996867912 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5a4c11bbe0 0x7f5a4c138b90 secure :-1 s=THROTTLE_DONE pgs=570 cs=0 l=1 rx=0x7f5a34007ca0 tx=0x7f5a44001210).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:25.276 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:25.267+0000 7f5a48ff9700 1 -- 172.21.15.42:0/996867912 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 313 v313) v1 ==== 95+0+2114 (secure 0 0 0) 0x7f5a3c0320d0 con 0x7f5a4c11bbe0 2019-12-05T06:26:25.276 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 313 2019-12-05T06:26:25.276 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:26:25.276 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":313,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":11221,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/3427104185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3427104185},{"type":"v1","addr":"172.21.15.42:6835","nonce":3427104185}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":313}],"filesystems":[{"mdsmap":{"epoch":312,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:24:39.885811+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":105,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7568},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7568":{"gid":7568,"name":"b","rank":0,"incarnation":309,"state":"up:active","state_seq":14,"addr":"172.21.15.40:6833/2324657302","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2324657302},{"type":"v1","addr":"172.21.15.40:6833","nonce":2324657302}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:26:25.279 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:25.271+0000 7f5a52210700 1 -- 172.21.15.42:0/996867912 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5a3003b400 msgr2=0x7f5a3003d910 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:25.279 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:25.271+0000 7f5a52210700 1 --2- 172.21.15.42:0/996867912 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5a3003b400 0x7f5a3003d910 crc :-1 s=READY pgs=1098 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:25.279 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:25.271+0000 7f5a52210700 1 -- 172.21.15.42:0/996867912 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5a4c11bbe0 msgr2=0x7f5a4c138b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:25.279 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:25.271+0000 7f5a52210700 1 --2- 172.21.15.42:0/996867912 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5a4c11bbe0 0x7f5a4c138b90 secure :-1 s=READY pgs=570 cs=0 l=1 rx=0x7f5a34007ca0 tx=0x7f5a44001210).stop 2019-12-05T06:26:25.279 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:25.271+0000 7f5a52210700 1 -- 172.21.15.42:0/996867912 shutdown_connections 2019-12-05T06:26:25.280 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:25.271+0000 7f5a52210700 1 --2- 172.21.15.42:0/996867912 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5a3003b400 0x7f5a3003d910 unknown :-1 s=CLOSED pgs=1098 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:25.280 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:25.271+0000 7f5a52210700 1 --2- 172.21.15.42:0/996867912 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5a4c11bbe0 0x7f5a4c138b90 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:25.280 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:25.271+0000 7f5a52210700 1 -- 172.21.15.42:0/996867912 shutdown_connections 2019-12-05T06:26:25.280 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:25.271+0000 7f5a52210700 1 -- 172.21.15.42:0/996867912 wait complete. 2019-12-05T06:26:25.280 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:25.271+0000 7f5a52210700 1 -- 172.21.15.42:0/996867912 >> 172.21.15.42:0/996867912 conn(0x7f5a4c0a7740 msgr2=0x7f5a4c059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:26:25.292 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:26:27.293 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:26:27.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:27.387+0000 7fce2e69c700 1 Processor -- start 2019-12-05T06:26:27.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:27.387+0000 7fce2e69c700 1 -- start start 2019-12-05T06:26:27.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:27.391+0000 7fce2e69c700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fce2808df80 0x7fce2809f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:26:27.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:27.391+0000 7fce2e69c700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fce2809f830 0x7fce2814efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:26:27.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:27.391+0000 7fce2e69c700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce2809fc60 con 0x7fce280a7740 2019-12-05T06:26:27.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:27.391+0000 7fce2e69c700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce280a0b00 con 0x7fce2808df80 2019-12-05T06:26:27.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:27.391+0000 7fce2e69c700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce2808e5b0 con 0x7fce2809f830 2019-12-05T06:26:27.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:27.391+0000 7fce2cc39700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fce280a7740 0x7fce2807e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:55188/0 (socket says 172.21.15.42:55188) 2019-12-05T06:26:27.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:27.391+0000 7fce2cc39700 1 -- 172.21.15.42:0/3403118174 learned_addr learned my addr 172.21.15.42:0/3403118174 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:26:27.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:27.391+0000 7fce27fff700 1 --2- 172.21.15.42:0/3403118174 >> v2:172.21.15.40:3300/0 conn(0x7fce2808df80 0x7fce2809f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:26:27.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:27.391+0000 7fce2d43a700 1 --2- 172.21.15.42:0/3403118174 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fce2809f830 0x7fce2814efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:26:27.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:27.391+0000 7fce27fff700 1 -- 172.21.15.42:0/3403118174 >> v2:172.21.15.40:3300/0 conn(0x7fce2808df80 msgr2=0x7fce2809f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2019-12-05T06:26:27.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:27.391+0000 7fce27fff700 1 -- 172.21.15.42:0/3403118174 >> v2:172.21.15.40:3300/0 conn(0x7fce2808df80 msgr2=0x7fce2809f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2019-12-05T06:26:27.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:27.391+0000 7fce27fff700 1 --2- 172.21.15.42:0/3403118174 >> v2:172.21.15.40:3300/0 conn(0x7fce2808df80 0x7fce2809f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-12-05T06:26:27.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:27.391+0000 7fce27fff700 1 --2- 172.21.15.42:0/3403118174 >> v2:172.21.15.40:3300/0 conn(0x7fce2808df80 0x7fce2809f2e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rx=0 tx=0)._fault waiting 0.200000 2019-12-05T06:26:27.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:27.391+0000 7fce277fe700 1 -- 172.21.15.42:0/3403118174 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2877314154 0 0) 0x7fce2809fc60 con 0x7fce280a7740 2019-12-05T06:26:27.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:27.391+0000 7fce277fe700 1 -- 172.21.15.42:0/3403118174 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce100030b0 con 0x7fce280a7740 2019-12-05T06:26:27.404 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:27.391+0000 7fce2d43a700 1 -- 172.21.15.42:0/3403118174 >> v2:172.21.15.40:3300/0 conn(0x7fce2808df80 msgr2=0x7fce2809f2e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2019-12-05T06:26:27.404 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:27.391+0000 7fce2d43a700 1 --2- 172.21.15.42:0/3403118174 >> v2:172.21.15.40:3300/0 conn(0x7fce2808df80 0x7fce2809f2e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:26:27.404 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:27.391+0000 7fce2d43a700 1 -- 172.21.15.42:0/3403118174 >> v1:172.21.15.42:6789/0 conn(0x7fce280a7740 legacy=0x7fce2807e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:27.404 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:27.391+0000 7fce2d43a700 1 -- 172.21.15.42:0/3403118174 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce280a0710 con 0x7fce2809f830 2019-12-05T06:26:27.404 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:27.391+0000 7fce2d43a700 1 --2- 172.21.15.42:0/3403118174 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fce2809f830 0x7fce2814efe0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fce20008b10 tx=0x7fce2000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:27.404 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:27.391+0000 7fce2d43a700 1 --2- 172.21.15.42:0/3403118174 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fce2809f830 0x7fce2814efe0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fce20008b10 tx=0x7fce2000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:27.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:27.391+0000 7fce2d43a700 1 --2- 172.21.15.42:0/3403118174 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fce2809f830 0x7fce2814efe0 secure :-1 s=READY pgs=571 cs=0 l=1 rx=0x7fce20008b10 tx=0x7fce2000fff0).ready entity=mon.2 client_cookie=ee876ad13ed19a38 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:26:27.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:27.391+0000 7fce2d43a700 1 --2- 172.21.15.42:0/3403118174 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fce2809f830 0x7fce2814efe0 secure :-1 s=THROTTLE_DONE pgs=571 cs=0 l=1 rx=0x7fce20008b10 tx=0x7fce2000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:27.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:27.391+0000 7fce277fe700 1 -- 172.21.15.42:0/3403118174 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fce20010900 con 0x7fce2809f830 2019-12-05T06:26:27.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:27.391+0000 7fce2d43a700 1 --2- 172.21.15.42:0/3403118174 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fce2809f830 0x7fce2814efe0 secure :-1 s=THROTTLE_DONE pgs=571 cs=0 l=1 rx=0x7fce20008b10 tx=0x7fce2000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:27.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:27.391+0000 7fce2d43a700 1 --2- 172.21.15.42:0/3403118174 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fce2809f830 0x7fce2814efe0 secure :-1 s=THROTTLE_DONE pgs=571 cs=0 l=1 rx=0x7fce20008b10 tx=0x7fce2000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:27.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:27.391+0000 7fce277fe700 1 -- 172.21.15.42:0/3403118174 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce20008b50 con 0x7fce2809f830 2019-12-05T06:26:27.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:27.391+0000 7fce277fe700 1 -- 172.21.15.42:0/3403118174 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fce200110c0 con 0x7fce2809f830 2019-12-05T06:26:27.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:27.391+0000 7fce2e69c700 1 -- 172.21.15.42:0/3403118174 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fce2809f830 msgr2=0x7fce2814efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:27.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:27.391+0000 7fce2e69c700 1 --2- 172.21.15.42:0/3403118174 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fce2809f830 0x7fce2814efe0 secure :-1 s=READY pgs=571 cs=0 l=1 rx=0x7fce20008b10 tx=0x7fce2000fff0).stop 2019-12-05T06:26:27.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:27.391+0000 7fce2e69c700 1 -- 172.21.15.42:0/3403118174 shutdown_connections 2019-12-05T06:26:27.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:27.391+0000 7fce2e69c700 1 --2- 172.21.15.42:0/3403118174 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fce2809f830 0x7fce2814efe0 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:27.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:27.391+0000 7fce2e69c700 1 --2- 172.21.15.42:0/3403118174 >> v2:172.21.15.40:3300/0 conn(0x7fce2808df80 0x7fce2809f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:26:27.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:27.391+0000 7fce2e69c700 1 -- 172.21.15.42:0/3403118174 shutdown_connections 2019-12-05T06:26:27.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:27.391+0000 7fce2e69c700 1 -- 172.21.15.42:0/3403118174 wait complete. 2019-12-05T06:26:27.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:27.391+0000 7fce2e69c700 1 -- 172.21.15.42:0/3403118174 >> 172.21.15.42:0/3403118174 conn(0x7fce280aabe0 msgr2=0x7fce280a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:26:27.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:27.395+0000 7fce2e69c700 1 Processor -- start 2019-12-05T06:26:27.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:27.395+0000 7fce2e69c700 1 -- start start 2019-12-05T06:26:27.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:27.395+0000 7fce2e69c700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fce280a6f30 0x7fce28124180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:26:27.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:27.395+0000 7fce2e69c700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fce2807d5d0 0x7fce28137780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:26:27.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:27.395+0000 7fce2e69c700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce281246d0 con 0x7fce2807d1a0 2019-12-05T06:26:27.411 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:27.395+0000 7fce2e69c700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce28131210 con 0x7fce2807d5d0 2019-12-05T06:26:27.411 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:27.395+0000 7fce2e69c700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce280a5b20 con 0x7fce280a6f30 2019-12-05T06:26:27.411 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:27.395+0000 7fce27fff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fce2807d1a0 0x7fce280a4fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:55196/0 (socket says 172.21.15.42:55196) 2019-12-05T06:26:27.411 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:27.395+0000 7fce27fff700 1 -- 172.21.15.42:0/2766389670 learned_addr learned my addr 172.21.15.42:0/2766389670 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:26:27.411 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:27.395+0000 7fce2cc39700 1 --2- 172.21.15.42:0/2766389670 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fce280a6f30 0x7fce28124180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:26:27.411 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:27.395+0000 7fce2d43a700 1 --2- 172.21.15.42:0/2766389670 >> v2:172.21.15.40:3300/0 conn(0x7fce2807d5d0 0x7fce28137780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:26:27.411 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:27.395+0000 7fce257fa700 1 -- 172.21.15.42:0/2766389670 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1753933855 0 0) 0x7fce18000e90 con 0x7fce2807d1a0 2019-12-05T06:26:27.412 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:27.395+0000 7fce257fa700 1 -- 172.21.15.42:0/2766389670 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce080030b0 con 0x7fce2807d1a0 2019-12-05T06:26:27.412 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:27.395+0000 7fce2cc39700 1 -- 172.21.15.42:0/2766389670 >> v2:172.21.15.40:3300/0 conn(0x7fce2807d5d0 msgr2=0x7fce28137780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:27.412 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:27.395+0000 7fce2cc39700 1 --2- 172.21.15.42:0/2766389670 >> v2:172.21.15.40:3300/0 conn(0x7fce2807d5d0 0x7fce28137780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:27.412 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:27.395+0000 7fce2cc39700 1 -- 172.21.15.42:0/2766389670 >> v1:172.21.15.42:6789/0 conn(0x7fce2807d1a0 legacy=0x7fce280a4fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:27.412 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:27.395+0000 7fce2cc39700 1 -- 172.21.15.42:0/2766389670 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce280a57e0 con 0x7fce280a6f30 2019-12-05T06:26:27.412 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:27.395+0000 7fce2d43a700 1 --2- 172.21.15.42:0/2766389670 >> v2:172.21.15.40:3300/0 conn(0x7fce2807d5d0 0x7fce28137780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:26:27.412 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:27.395+0000 7fce2cc39700 1 --2- 172.21.15.42:0/2766389670 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fce280a6f30 0x7fce28124180 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fce1c004370 tx=0x7fce1c0108b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:27.413 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:27.399+0000 7fce2cc39700 1 --2- 172.21.15.42:0/2766389670 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fce280a6f30 0x7fce28124180 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fce1c004370 tx=0x7fce1c0108b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:27.413 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:27.399+0000 7fce2cc39700 1 --2- 172.21.15.42:0/2766389670 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fce280a6f30 0x7fce28124180 secure :-1 s=READY pgs=572 cs=0 l=1 rx=0x7fce1c004370 tx=0x7fce1c0108b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:26:27.413 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:27.399+0000 7fce2cc39700 1 --2- 172.21.15.42:0/2766389670 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fce280a6f30 0x7fce28124180 secure :-1 s=THROTTLE_DONE pgs=572 cs=0 l=1 rx=0x7fce1c004370 tx=0x7fce1c0108b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:27.413 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:27.399+0000 7fce257fa700 1 -- 172.21.15.42:0/2766389670 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fce1c004a20 con 0x7fce280a6f30 2019-12-05T06:26:27.413 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:27.399+0000 7fce2cc39700 1 --2- 172.21.15.42:0/2766389670 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fce280a6f30 0x7fce28124180 secure :-1 s=THROTTLE_DONE pgs=572 cs=0 l=1 rx=0x7fce1c004370 tx=0x7fce1c0108b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:27.413 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:27.399+0000 7fce2e69c700 1 -- 172.21.15.42:0/2766389670 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce28130330 con 0x7fce280a6f30 2019-12-05T06:26:27.413 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:27.399+0000 7fce2cc39700 1 --2- 172.21.15.42:0/2766389670 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fce280a6f30 0x7fce28124180 secure :-1 s=THROTTLE_DONE pgs=572 cs=0 l=1 rx=0x7fce1c004370 tx=0x7fce1c0108b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:27.414 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:27.399+0000 7fce2e69c700 1 -- 172.21.15.42:0/2766389670 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce2808eb70 con 0x7fce280a6f30 2019-12-05T06:26:27.414 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:27.399+0000 7fce257fa700 1 -- 172.21.15.42:0/2766389670 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce1c004640 con 0x7fce280a6f30 2019-12-05T06:26:27.414 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:27.399+0000 7fce257fa700 1 -- 172.21.15.42:0/2766389670 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fce1c005000 con 0x7fce280a6f30 2019-12-05T06:26:27.414 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:27.399+0000 7fce2cc39700 1 --2- 172.21.15.42:0/2766389670 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fce280a6f30 0x7fce28124180 secure :-1 s=THROTTLE_DONE pgs=572 cs=0 l=1 rx=0x7fce1c004370 tx=0x7fce1c0108b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:27.414 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:27.399+0000 7fce257fa700 1 -- 172.21.15.42:0/2766389670 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fce1c008180 con 0x7fce280a6f30 2019-12-05T06:26:27.414 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:27.399+0000 7fce2cc39700 1 --2- 172.21.15.42:0/2766389670 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fce280a6f30 0x7fce28124180 secure :-1 s=THROTTLE_DONE pgs=572 cs=0 l=1 rx=0x7fce1c004370 tx=0x7fce1c0108b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:27.414 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:27.399+0000 7fce257fa700 1 --2- 172.21.15.42:0/2766389670 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fce0803b2d0 0x7fce0803d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:26:27.415 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:27.399+0000 7fce257fa700 1 -- 172.21.15.42:0/2766389670 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 10847+0+0 (secure 0 0 0) 0x7fce1c04ef60 con 0x7fce280a6f30 2019-12-05T06:26:27.415 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:27.399+0000 7fce27fff700 1 --2- 172.21.15.42:0/2766389670 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fce0803b2d0 0x7fce0803d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:26:27.415 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:27.399+0000 7fce27fff700 1 --2- 172.21.15.42:0/2766389670 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fce0803b2d0 0x7fce0803d7e0 crc :-1 s=READY pgs=1099 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:26:27.415 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:27.403+0000 7fce2e69c700 1 -- 172.21.15.42:0/2766389670 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce00003200 con 0x7fce280a6f30 2019-12-05T06:26:27.420 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:27.415+0000 7fce2cc39700 1 --2- 172.21.15.42:0/2766389670 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fce280a6f30 0x7fce28124180 secure :-1 s=THROTTLE_DONE pgs=572 cs=0 l=1 rx=0x7fce1c004370 tx=0x7fce1c0108b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:27.420 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:27.415+0000 7fce257fa700 1 -- 172.21.15.42:0/2766389670 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fce1c036d90 con 0x7fce280a6f30 2019-12-05T06:26:27.621 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:27.615+0000 7fce2e69c700 1 -- 172.21.15.42:0/2766389670 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fce00003630 con 0x7fce280a6f30 2019-12-05T06:26:27.622 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:27.615+0000 7fce2cc39700 1 --2- 172.21.15.42:0/2766389670 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fce280a6f30 0x7fce28124180 secure :-1 s=THROTTLE_DONE pgs=572 cs=0 l=1 rx=0x7fce1c004370 tx=0x7fce1c0108b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:27.622 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:27.615+0000 7fce257fa700 1 -- 172.21.15.42:0/2766389670 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 313 v313) v1 ==== 95+0+2114 (secure 0 0 0) 0x7fce1c0332c0 con 0x7fce280a6f30 2019-12-05T06:26:27.624 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 313 2019-12-05T06:26:27.624 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:26:27.625 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":313,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":11221,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/3427104185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3427104185},{"type":"v1","addr":"172.21.15.42:6835","nonce":3427104185}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":313}],"filesystems":[{"mdsmap":{"epoch":312,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:24:39.885811+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":105,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7568},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7568":{"gid":7568,"name":"b","rank":0,"incarnation":309,"state":"up:active","state_seq":14,"addr":"172.21.15.40:6833/2324657302","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2324657302},{"type":"v1","addr":"172.21.15.40:6833","nonce":2324657302}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:26:27.628 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:27.619+0000 7fce2e69c700 1 -- 172.21.15.42:0/2766389670 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fce0803b2d0 msgr2=0x7fce0803d7e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:27.629 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:27.619+0000 7fce2e69c700 1 --2- 172.21.15.42:0/2766389670 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fce0803b2d0 0x7fce0803d7e0 crc :-1 s=READY pgs=1099 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:27.629 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:27.619+0000 7fce2e69c700 1 -- 172.21.15.42:0/2766389670 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fce280a6f30 msgr2=0x7fce28124180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:27.629 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:27.619+0000 7fce2e69c700 1 --2- 172.21.15.42:0/2766389670 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fce280a6f30 0x7fce28124180 secure :-1 s=READY pgs=572 cs=0 l=1 rx=0x7fce1c004370 tx=0x7fce1c0108b0).stop 2019-12-05T06:26:27.629 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:27.619+0000 7fce2e69c700 1 -- 172.21.15.42:0/2766389670 shutdown_connections 2019-12-05T06:26:27.629 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:27.619+0000 7fce2e69c700 1 --2- 172.21.15.42:0/2766389670 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fce0803b2d0 0x7fce0803d7e0 unknown :-1 s=CLOSED pgs=1099 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:27.629 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:27.619+0000 7fce2e69c700 1 --2- 172.21.15.42:0/2766389670 >> v2:172.21.15.40:3300/0 conn(0x7fce2807d5d0 0x7fce28137780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:27.631 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:27.619+0000 7fce2e69c700 1 --2- 172.21.15.42:0/2766389670 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fce280a6f30 0x7fce28124180 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:27.631 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:27.619+0000 7fce2e69c700 1 -- 172.21.15.42:0/2766389670 shutdown_connections 2019-12-05T06:26:27.632 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:27.619+0000 7fce2e69c700 1 -- 172.21.15.42:0/2766389670 wait complete. 2019-12-05T06:26:27.632 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:27.623+0000 7fce2e69c700 1 -- 172.21.15.42:0/2766389670 >> 172.21.15.42:0/2766389670 conn(0x7fce2811d0f0 msgr2=0x7fce2811d890 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:26:27.639 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:26:29.640 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:26:29.749 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:29.735+0000 7f8055022700 1 Processor -- start 2019-12-05T06:26:29.750 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:29.735+0000 7f8055022700 1 -- start start 2019-12-05T06:26:29.750 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:29.735+0000 7f8055022700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f80500a7740 0x7f805008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:26:29.750 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:29.735+0000 7f8055022700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f805009fae0 0x7f805007eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:26:29.750 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:29.735+0000 7f8055022700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80500a0710 con 0x7f805007e720 2019-12-05T06:26:29.751 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:29.735+0000 7f8055022700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80500a0b00 con 0x7f805009fae0 2019-12-05T06:26:29.754 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:29.735+0000 7f8055022700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f805008e5b0 con 0x7f80500a7740 2019-12-05T06:26:29.754 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:29.735+0000 7f804f59e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f80500a7740 0x7f805008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:26:29.754 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:29.735+0000 7f804f59e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f80500a7740 0x7f805008df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:55166/0 (socket says 172.21.15.42:55166) 2019-12-05T06:26:29.757 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:29.735+0000 7f804f59e700 1 -- 172.21.15.42:0/867443759 learned_addr learned my addr 172.21.15.42:0/867443759 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:26:29.757 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:29.739+0000 7f8054821700 1 --2- 172.21.15.42:0/867443759 >> v2:172.21.15.40:3300/0 conn(0x7f805009fae0 0x7f805007eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:26:29.757 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:29.739+0000 7f804f59e700 1 -- 172.21.15.42:0/867443759 >> v2:172.21.15.40:3300/0 conn(0x7f805009fae0 msgr2=0x7f805007eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:26:29.757 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:29.739+0000 7f804f59e700 1 --2- 172.21.15.42:0/867443759 >> v2:172.21.15.40:3300/0 conn(0x7f805009fae0 0x7f805007eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:26:29.758 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:29.739+0000 7f804f59e700 1 -- 172.21.15.42:0/867443759 >> v1:172.21.15.42:6789/0 conn(0x7f805007e720 legacy=0x7f805009f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:29.758 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:29.739+0000 7f804f59e700 1 -- 172.21.15.42:0/867443759 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f805007f0a0 con 0x7f80500a7740 2019-12-05T06:26:29.758 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:29.739+0000 7f804f59e700 1 --2- 172.21.15.42:0/867443759 >> v2:172.21.15.42:3300/0 conn(0x7f80500a7740 0x7f805008df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f8040002f30 tx=0x7f804000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:29.758 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:29.739+0000 7f804f59e700 1 --2- 172.21.15.42:0/867443759 >> v2:172.21.15.42:3300/0 conn(0x7f80500a7740 0x7f805008df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f8040002f30 tx=0x7f804000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:29.759 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:29.739+0000 7f804f59e700 1 --2- 172.21.15.42:0/867443759 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f80500a7740 0x7f805008df80 secure :-1 s=READY pgs=2640 cs=0 l=1 rx=0x7f8040002f30 tx=0x7f804000fd30).ready entity=mon.1 client_cookie=c5982c83fcc7c554 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:26:29.759 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:29.739+0000 7f804f59e700 1 --2- 172.21.15.42:0/867443759 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f80500a7740 0x7f805008df80 secure :-1 s=THROTTLE_DONE pgs=2640 cs=0 l=1 rx=0x7f8040002f30 tx=0x7f804000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:29.759 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:29.739+0000 7f804f59e700 1 --2- 172.21.15.42:0/867443759 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f80500a7740 0x7f805008df80 secure :-1 s=THROTTLE_DONE pgs=2640 cs=0 l=1 rx=0x7f8040002f30 tx=0x7f804000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:29.759 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:29.739+0000 7f804ed9d700 1 -- 172.21.15.42:0/867443759 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8040008c90 con 0x7f80500a7740 2019-12-05T06:26:29.759 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:29.739+0000 7f804f59e700 1 --2- 172.21.15.42:0/867443759 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f80500a7740 0x7f805008df80 secure :-1 s=THROTTLE_DONE pgs=2640 cs=0 l=1 rx=0x7f8040002f30 tx=0x7f804000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:29.760 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:29.739+0000 7f804ed9d700 1 -- 172.21.15.42:0/867443759 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80400088b0 con 0x7f80500a7740 2019-12-05T06:26:29.760 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:29.739+0000 7f804ed9d700 1 -- 172.21.15.42:0/867443759 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8040011ad0 con 0x7f80500a7740 2019-12-05T06:26:29.760 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:29.739+0000 7f8055022700 1 -- 172.21.15.42:0/867443759 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f80500a7740 msgr2=0x7f805008df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:29.760 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:29.739+0000 7f8055022700 1 --2- 172.21.15.42:0/867443759 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f80500a7740 0x7f805008df80 secure :-1 s=READY pgs=2640 cs=0 l=1 rx=0x7f8040002f30 tx=0x7f804000fd30).stop 2019-12-05T06:26:29.761 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:29.739+0000 7f8055022700 1 -- 172.21.15.42:0/867443759 shutdown_connections 2019-12-05T06:26:29.761 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:29.739+0000 7f8055022700 1 --2- 172.21.15.42:0/867443759 >> v2:172.21.15.40:3300/0 conn(0x7f805009fae0 0x7f805007eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:26:29.761 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:29.739+0000 7f8055022700 1 --2- 172.21.15.42:0/867443759 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f80500a7740 0x7f805008df80 unknown :-1 s=CLOSED pgs=2640 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:29.761 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:29.739+0000 7f8055022700 1 -- 172.21.15.42:0/867443759 shutdown_connections 2019-12-05T06:26:29.762 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:29.739+0000 7f8055022700 1 -- 172.21.15.42:0/867443759 wait complete. 2019-12-05T06:26:29.762 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:29.739+0000 7f8055022700 1 -- 172.21.15.42:0/867443759 >> 172.21.15.42:0/867443759 conn(0x7f80500aabe0 msgr2=0x7f80500a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:26:29.762 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:29.739+0000 7f8055022700 1 Processor -- start 2019-12-05T06:26:29.762 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:29.739+0000 7f8055022700 1 -- start start 2019-12-05T06:26:29.776 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:29.743+0000 7f8055022700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f805011d250 0x7f8050122b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:26:29.776 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:29.743+0000 7f8055022700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8050119cb0 con 0x7f805011ba70 2019-12-05T06:26:29.777 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:29.743+0000 7f8055022700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f805012e410 con 0x7f80500722c0 2019-12-05T06:26:29.777 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:29.743+0000 7f8055022700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f805011c080 con 0x7f805011d250 2019-12-05T06:26:29.777 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:29.743+0000 7f8054821700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f80500722c0 0x7f8050074700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:55214/0 (socket says 172.21.15.42:55214) 2019-12-05T06:26:29.778 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:29.743+0000 7f8054821700 1 -- 172.21.15.42:0/1012825927 learned_addr learned my addr 172.21.15.42:0/1012825927 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:26:29.778 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:29.743+0000 7f8047fff700 1 --2- 172.21.15.42:0/1012825927 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f805011d250 0x7f8050122b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:26:29.778 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:29.743+0000 7f804cd99700 1 -- 172.21.15.42:0/1012825927 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1110025295 0 0) 0x7f805012e410 con 0x7f80500722c0 2019-12-05T06:26:29.778 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:29.743+0000 7f804cd99700 1 -- 172.21.15.42:0/1012825927 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80340030b0 con 0x7f80500722c0 2019-12-05T06:26:29.779 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:29.743+0000 7f804cd99700 1 -- 172.21.15.42:0/1012825927 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 776688472 0 0) 0x7f8050119cb0 con 0x7f805011ba70 2019-12-05T06:26:29.779 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:29.743+0000 7f804cd99700 1 -- 172.21.15.42:0/1012825927 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f805012e410 con 0x7f805011ba70 2019-12-05T06:26:29.779 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:29.743+0000 7f804cd99700 1 -- 172.21.15.42:0/1012825927 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 3907486143 0 0) 0x7f80340030b0 con 0x7f80500722c0 2019-12-05T06:26:29.779 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:29.743+0000 7f804cd99700 1 -- 172.21.15.42:0/1012825927 --> v1:172.21.15.42:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8050119cb0 con 0x7f80500722c0 2019-12-05T06:26:29.780 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:29.743+0000 7f804cd99700 1 -- 172.21.15.42:0/1012825927 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7f8048002840 con 0x7f80500722c0 2019-12-05T06:26:29.780 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:29.743+0000 7f8047fff700 1 -- 172.21.15.42:0/1012825927 >> v1:172.21.15.42:6789/0 conn(0x7f80500722c0 legacy=0x7f8050074700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:29.780 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:29.743+0000 7f8047fff700 1 -- 172.21.15.42:0/1012825927 >> v1:172.21.15.40:6789/0 conn(0x7f805011ba70 legacy=0x7f80501194b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:29.781 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:29.743+0000 7f8047fff700 1 -- 172.21.15.42:0/1012825927 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f805011c320 con 0x7f805011d250 2019-12-05T06:26:29.781 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:29.743+0000 7f8047fff700 1 --2- 172.21.15.42:0/1012825927 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f805011d250 0x7f8050122b20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f805011c890 tx=0x7f804000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:29.781 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:29.743+0000 7f8047fff700 1 --2- 172.21.15.42:0/1012825927 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f805011d250 0x7f8050122b20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f805011c890 tx=0x7f804000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:29.781 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:29.743+0000 7f8047fff700 1 --2- 172.21.15.42:0/1012825927 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f805011d250 0x7f8050122b20 secure :-1 s=READY pgs=573 cs=0 l=1 rx=0x7f805011c890 tx=0x7f804000fd30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:26:29.781 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:29.743+0000 7f8047fff700 1 --2- 172.21.15.42:0/1012825927 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f805011d250 0x7f8050122b20 secure :-1 s=THROTTLE_DONE pgs=573 cs=0 l=1 rx=0x7f805011c890 tx=0x7f804000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:29.781 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:29.743+0000 7f804cd99700 1 -- 172.21.15.42:0/1012825927 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f803800c950 con 0x7f805011d250 2019-12-05T06:26:29.782 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:29.743+0000 7f8047fff700 1 --2- 172.21.15.42:0/1012825927 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f805011d250 0x7f8050122b20 secure :-1 s=THROTTLE_DONE pgs=573 cs=0 l=1 rx=0x7f805011c890 tx=0x7f804000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:29.782 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:29.743+0000 7f8047fff700 1 --2- 172.21.15.42:0/1012825927 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f805011d250 0x7f8050122b20 secure :-1 s=THROTTLE_DONE pgs=573 cs=0 l=1 rx=0x7f805011c890 tx=0x7f804000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:29.782 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:29.743+0000 7f804cd99700 1 -- 172.21.15.42:0/1012825927 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f803800c570 con 0x7f805011d250 2019-12-05T06:26:29.782 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:29.743+0000 7f804cd99700 1 -- 172.21.15.42:0/1012825927 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8038010e40 con 0x7f805011d250 2019-12-05T06:26:29.782 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:29.743+0000 7f8055022700 1 -- 172.21.15.42:0/1012825927 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8050075f10 con 0x7f805011d250 2019-12-05T06:26:29.782 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:29.743+0000 7f8055022700 1 -- 172.21.15.42:0/1012825927 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80500763b0 con 0x7f805011d250 2019-12-05T06:26:29.783 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:29.747+0000 7f8047fff700 1 --2- 172.21.15.42:0/1012825927 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f805011d250 0x7f8050122b20 secure :-1 s=THROTTLE_DONE pgs=573 cs=0 l=1 rx=0x7f805011c890 tx=0x7f804000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:29.783 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:29.747+0000 7f804cd99700 1 -- 172.21.15.42:0/1012825927 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f8038007430 con 0x7f805011d250 2019-12-05T06:26:29.783 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:29.747+0000 7f8047fff700 1 --2- 172.21.15.42:0/1012825927 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f805011d250 0x7f8050122b20 secure :-1 s=THROTTLE_DONE pgs=573 cs=0 l=1 rx=0x7f805011c890 tx=0x7f804000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:29.783 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:29.747+0000 7f804cd99700 1 --2- 172.21.15.42:0/1012825927 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f803403b860 0x7f803403dd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:26:29.783 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:29.747+0000 7f804cd99700 1 -- 172.21.15.42:0/1012825927 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 10847+0+0 (secure 0 0 0) 0x7f803804ae30 con 0x7f805011d250 2019-12-05T06:26:29.783 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:29.747+0000 7f804f59e700 1 --2- 172.21.15.42:0/1012825927 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f803403b860 0x7f803403dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:26:29.783 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:29.747+0000 7f804f59e700 1 --2- 172.21.15.42:0/1012825927 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f803403b860 0x7f803403dd70 crc :-1 s=READY pgs=1100 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:26:29.784 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:29.751+0000 7f8055022700 1 -- 172.21.15.42:0/1012825927 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f805005d110 con 0x7f805011d250 2019-12-05T06:26:29.784 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:29.759+0000 7f8047fff700 1 --2- 172.21.15.42:0/1012825927 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f805011d250 0x7f8050122b20 secure :-1 s=THROTTLE_DONE pgs=573 cs=0 l=1 rx=0x7f805011c890 tx=0x7f804000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:29.784 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:29.759+0000 7f804cd99700 1 -- 172.21.15.42:0/1012825927 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f8038031eb0 con 0x7f805011d250 2019-12-05T06:26:29.999 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:29.991+0000 7f8055022700 1 -- 172.21.15.42:0/1012825927 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f805005b8c0 con 0x7f805011d250 2019-12-05T06:26:30.000 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:29.995+0000 7f8047fff700 1 --2- 172.21.15.42:0/1012825927 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f805011d250 0x7f8050122b20 secure :-1 s=THROTTLE_DONE pgs=573 cs=0 l=1 rx=0x7f805011c890 tx=0x7f804000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:30.000 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:29.995+0000 7f804cd99700 1 -- 172.21.15.42:0/1012825927 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 313 v313) v1 ==== 95+0+2114 (secure 0 0 0) 0x7f8038032110 con 0x7f805011d250 2019-12-05T06:26:30.002 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 313 2019-12-05T06:26:30.002 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:26:30.002 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":313,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":11221,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/3427104185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3427104185},{"type":"v1","addr":"172.21.15.42:6835","nonce":3427104185}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":313}],"filesystems":[{"mdsmap":{"epoch":312,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:24:39.885811+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":105,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7568},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7568":{"gid":7568,"name":"b","rank":0,"incarnation":309,"state":"up:active","state_seq":14,"addr":"172.21.15.40:6833/2324657302","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2324657302},{"type":"v1","addr":"172.21.15.40:6833","nonce":2324657302}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:26:30.005 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:29.999+0000 7f8055022700 1 -- 172.21.15.42:0/1012825927 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f803403b860 msgr2=0x7f803403dd70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:30.005 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:29.999+0000 7f8055022700 1 --2- 172.21.15.42:0/1012825927 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f803403b860 0x7f803403dd70 crc :-1 s=READY pgs=1100 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:30.005 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:29.999+0000 7f8055022700 1 -- 172.21.15.42:0/1012825927 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f805011d250 msgr2=0x7f8050122b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:30.006 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:29.999+0000 7f8055022700 1 --2- 172.21.15.42:0/1012825927 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f805011d250 0x7f8050122b20 secure :-1 s=READY pgs=573 cs=0 l=1 rx=0x7f805011c890 tx=0x7f804000fd30).stop 2019-12-05T06:26:30.006 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:29.999+0000 7f8055022700 1 -- 172.21.15.42:0/1012825927 shutdown_connections 2019-12-05T06:26:30.006 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:29.999+0000 7f8055022700 1 --2- 172.21.15.42:0/1012825927 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f803403b860 0x7f803403dd70 unknown :-1 s=CLOSED pgs=1100 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:30.006 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:29.999+0000 7f8055022700 1 --2- 172.21.15.42:0/1012825927 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f805011d250 0x7f8050122b20 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:30.006 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:29.999+0000 7f8055022700 1 -- 172.21.15.42:0/1012825927 shutdown_connections 2019-12-05T06:26:30.006 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:29.999+0000 7f8055022700 1 -- 172.21.15.42:0/1012825927 wait complete. 2019-12-05T06:26:30.006 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:29.999+0000 7f8055022700 1 -- 172.21.15.42:0/1012825927 >> 172.21.15.42:0/1012825927 conn(0x7f80500aabe0 msgr2=0x7f805011d8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:26:30.016 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:26:32.018 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:26:32.125 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:32.114+0000 7f5e3092c700 1 Processor -- start 2019-12-05T06:26:32.126 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:32.114+0000 7f5e3092c700 1 -- start start 2019-12-05T06:26:32.126 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:32.114+0000 7f5e3092c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5e2c0a7740 0x7f5e2c08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:26:32.126 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:32.114+0000 7f5e3092c700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e2c0a0710 con 0x7f5e2c09fae0 2019-12-05T06:26:32.126 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:32.114+0000 7f5e3092c700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e2c0a9ba0 con 0x7f5e2c07e720 2019-12-05T06:26:32.127 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:32.114+0000 7f5e3092c700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e2c0a0b00 con 0x7f5e2c0a7740 2019-12-05T06:26:32.127 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:32.114+0000 7f5e2ad9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5e2c0a7740 0x7f5e2c08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:26:32.127 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:32.114+0000 7f5e2ad9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5e2c0a7740 0x7f5e2c08df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:55182/0 (socket says 172.21.15.42:55182) 2019-12-05T06:26:32.127 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:32.114+0000 7f5e2ad9d700 1 -- 172.21.15.42:0/2420346947 learned_addr learned my addr 172.21.15.42:0/2420346947 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:26:32.127 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:32.114+0000 7f5e2a59c700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f5e2c07e720 0x7f5e2c09f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:55222/0 (socket says 172.21.15.42:55222) 2019-12-05T06:26:32.127 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:32.114+0000 7f5e2ad9d700 1 -- 172.21.15.42:0/2420346947 >> v1:172.21.15.42:6789/0 conn(0x7f5e2c07e720 legacy=0x7f5e2c09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:32.128 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:32.114+0000 7f5e2ad9d700 1 -- 172.21.15.42:0/2420346947 >> v1:172.21.15.40:6789/0 conn(0x7f5e2c09fae0 legacy=0x7f5e2c14efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:32.128 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:32.114+0000 7f5e2ad9d700 1 -- 172.21.15.42:0/2420346947 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e2c07eb50 con 0x7f5e2c0a7740 2019-12-05T06:26:32.128 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:32.114+0000 7f5e2ad9d700 1 --2- 172.21.15.42:0/2420346947 >> v2:172.21.15.42:3300/0 conn(0x7f5e2c0a7740 0x7f5e2c08df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f5e1c008790 tx=0x7f5e1c00fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:32.128 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:32.114+0000 7f5e2ad9d700 1 --2- 172.21.15.42:0/2420346947 >> v2:172.21.15.42:3300/0 conn(0x7f5e2c0a7740 0x7f5e2c08df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f5e1c008790 tx=0x7f5e1c00fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:32.129 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:32.114+0000 7f5e2ad9d700 1 --2- 172.21.15.42:0/2420346947 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5e2c0a7740 0x7f5e2c08df80 secure :-1 s=READY pgs=2643 cs=0 l=1 rx=0x7f5e1c008790 tx=0x7f5e1c00fcb0).ready entity=mon.1 client_cookie=8e652335ea432d37 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:26:32.129 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:32.114+0000 7f5e2ad9d700 1 --2- 172.21.15.42:0/2420346947 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5e2c0a7740 0x7f5e2c08df80 secure :-1 s=THROTTLE_DONE pgs=2643 cs=0 l=1 rx=0x7f5e1c008790 tx=0x7f5e1c00fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:32.129 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:32.114+0000 7f5e2ad9d700 1 --2- 172.21.15.42:0/2420346947 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5e2c0a7740 0x7f5e2c08df80 secure :-1 s=THROTTLE_DONE pgs=2643 cs=0 l=1 rx=0x7f5e1c008790 tx=0x7f5e1c00fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:32.129 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:32.114+0000 7f5e29d9b700 1 -- 172.21.15.42:0/2420346947 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5e1c011520 con 0x7f5e2c0a7740 2019-12-05T06:26:32.129 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:32.114+0000 7f5e2ad9d700 1 --2- 172.21.15.42:0/2420346947 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5e2c0a7740 0x7f5e2c08df80 secure :-1 s=THROTTLE_DONE pgs=2643 cs=0 l=1 rx=0x7f5e1c008790 tx=0x7f5e1c00fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:32.129 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:32.114+0000 7f5e29d9b700 1 -- 172.21.15.42:0/2420346947 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e1c008ca0 con 0x7f5e2c0a7740 2019-12-05T06:26:32.129 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:32.114+0000 7f5e29d9b700 1 -- 172.21.15.42:0/2420346947 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5e1c011b00 con 0x7f5e2c0a7740 2019-12-05T06:26:32.130 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:32.114+0000 7f5e3092c700 1 -- 172.21.15.42:0/2420346947 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5e2c0a7740 msgr2=0x7f5e2c08df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:32.130 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:32.114+0000 7f5e3092c700 1 --2- 172.21.15.42:0/2420346947 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5e2c0a7740 0x7f5e2c08df80 secure :-1 s=READY pgs=2643 cs=0 l=1 rx=0x7f5e1c008790 tx=0x7f5e1c00fcb0).stop 2019-12-05T06:26:32.131 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:32.118+0000 7f5e3092c700 1 -- 172.21.15.42:0/2420346947 shutdown_connections 2019-12-05T06:26:32.131 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:32.118+0000 7f5e3092c700 1 --2- 172.21.15.42:0/2420346947 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5e2c0a7740 0x7f5e2c08df80 unknown :-1 s=CLOSED pgs=2643 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:32.131 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:32.118+0000 7f5e3092c700 1 -- 172.21.15.42:0/2420346947 shutdown_connections 2019-12-05T06:26:32.131 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:32.118+0000 7f5e3092c700 1 -- 172.21.15.42:0/2420346947 wait complete. 2019-12-05T06:26:32.131 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:32.118+0000 7f5e3092c700 1 -- 172.21.15.42:0/2420346947 >> 172.21.15.42:0/2420346947 conn(0x7f5e2c0aabe0 msgr2=0x7f5e2c0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:26:32.136 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:32.118+0000 7f5e3092c700 1 Processor -- start 2019-12-05T06:26:32.136 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:32.118+0000 7f5e3092c700 1 -- start start 2019-12-05T06:26:32.136 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:32.122+0000 7f5e3092c700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5e2c11c240 0x7f5e2c1306b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:26:32.136 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:32.122+0000 7f5e3092c700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5e2c118cc0 0x7f5e2c117a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:26:32.136 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:32.122+0000 7f5e3092c700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e2c11ba30 con 0x7f5e2c117f50 2019-12-05T06:26:32.136 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:32.122+0000 7f5e3092c700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e2c11bd00 con 0x7f5e2c118cc0 2019-12-05T06:26:32.137 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:32.122+0000 7f5e3092c700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e2c11bfa0 con 0x7f5e2c11c240 2019-12-05T06:26:32.137 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:32.122+0000 7f5e2b59e700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f5e2c117f50 0x7f5e2c07d940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:55228/0 (socket says 172.21.15.42:55228) 2019-12-05T06:26:32.137 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:32.122+0000 7f5e2b59e700 1 -- 172.21.15.42:0/2305166136 learned_addr learned my addr 172.21.15.42:0/2305166136 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:26:32.137 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:32.122+0000 7f5e2ad9d700 1 --2- 172.21.15.42:0/2305166136 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5e2c11c240 0x7f5e2c1306b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:26:32.137 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:32.122+0000 7f5e2a59c700 1 --2- 172.21.15.42:0/2305166136 >> v2:172.21.15.40:3300/0 conn(0x7f5e2c118cc0 0x7f5e2c117a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:26:32.137 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:32.122+0000 7f5e1b7fe700 1 -- 172.21.15.42:0/2305166136 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2142827449 0 0) 0x7f5e2c11ba30 con 0x7f5e2c117f50 2019-12-05T06:26:32.138 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:32.122+0000 7f5e1b7fe700 1 -- 172.21.15.42:0/2305166136 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e100030b0 con 0x7f5e2c117f50 2019-12-05T06:26:32.138 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:32.122+0000 7f5e2a59c700 1 -- 172.21.15.42:0/2305166136 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5e2c11c240 msgr2=0x7f5e2c1306b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:32.138 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:32.122+0000 7f5e2a59c700 1 --2- 172.21.15.42:0/2305166136 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5e2c11c240 0x7f5e2c1306b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:32.138 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:32.122+0000 7f5e2a59c700 1 -- 172.21.15.42:0/2305166136 >> v1:172.21.15.42:6789/0 conn(0x7f5e2c117f50 legacy=0x7f5e2c07d940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:32.138 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:32.122+0000 7f5e2a59c700 1 -- 172.21.15.42:0/2305166136 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e2c07e140 con 0x7f5e2c118cc0 2019-12-05T06:26:32.138 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:32.122+0000 7f5e2a59c700 1 --2- 172.21.15.42:0/2305166136 >> v2:172.21.15.40:3300/0 conn(0x7f5e2c118cc0 0x7f5e2c117a00 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f5e2c11bec0 tx=0x7f5e20010860).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:32.138 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:32.122+0000 7f5e2ad9d700 1 --2- 172.21.15.42:0/2305166136 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5e2c11c240 0x7f5e2c1306b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:26:32.139 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:32.122+0000 7f5e2a59c700 1 --2- 172.21.15.42:0/2305166136 >> v2:172.21.15.40:3300/0 conn(0x7f5e2c118cc0 0x7f5e2c117a00 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f5e2c11bec0 tx=0x7f5e20010860).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:32.139 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:32.122+0000 7f5e2a59c700 1 --2- 172.21.15.42:0/2305166136 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5e2c118cc0 0x7f5e2c117a00 secure :-1 s=READY pgs=1971 cs=0 l=1 rx=0x7f5e2c11bec0 tx=0x7f5e20010860).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:26:32.139 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:32.122+0000 7f5e2a59c700 1 --2- 172.21.15.42:0/2305166136 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5e2c118cc0 0x7f5e2c117a00 secure :-1 s=THROTTLE_DONE pgs=1971 cs=0 l=1 rx=0x7f5e2c11bec0 tx=0x7f5e20010860).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:32.139 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:32.122+0000 7f5e1b7fe700 1 -- 172.21.15.42:0/2305166136 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5e20006850 con 0x7f5e2c118cc0 2019-12-05T06:26:32.139 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:32.122+0000 7f5e2a59c700 1 --2- 172.21.15.42:0/2305166136 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5e2c118cc0 0x7f5e2c117a00 secure :-1 s=THROTTLE_DONE pgs=1971 cs=0 l=1 rx=0x7f5e2c11bec0 tx=0x7f5e20010860).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:32.140 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:32.122+0000 7f5e2a59c700 1 --2- 172.21.15.42:0/2305166136 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5e2c118cc0 0x7f5e2c117a00 secure :-1 s=THROTTLE_DONE pgs=1971 cs=0 l=1 rx=0x7f5e2c11bec0 tx=0x7f5e20010860).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:32.140 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:32.122+0000 7f5e1b7fe700 1 -- 172.21.15.42:0/2305166136 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e20006470 con 0x7f5e2c118cc0 2019-12-05T06:26:32.140 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:32.122+0000 7f5e1b7fe700 1 -- 172.21.15.42:0/2305166136 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5e20011260 con 0x7f5e2c118cc0 2019-12-05T06:26:32.140 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:32.122+0000 7f5e3092c700 1 -- 172.21.15.42:0/2305166136 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e2c130c00 con 0x7f5e2c118cc0 2019-12-05T06:26:32.140 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:32.122+0000 7f5e3092c700 1 -- 172.21.15.42:0/2305166136 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e2c137ca0 con 0x7f5e2c118cc0 2019-12-05T06:26:32.140 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:32.122+0000 7f5e2a59c700 1 --2- 172.21.15.42:0/2305166136 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5e2c118cc0 0x7f5e2c117a00 secure :-1 s=THROTTLE_DONE pgs=1971 cs=0 l=1 rx=0x7f5e2c11bec0 tx=0x7f5e20010860).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:32.140 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:32.122+0000 7f5e1b7fe700 1 -- 172.21.15.42:0/2305166136 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f5e20011580 con 0x7f5e2c118cc0 2019-12-05T06:26:32.141 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:32.122+0000 7f5e2a59c700 1 --2- 172.21.15.42:0/2305166136 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5e2c118cc0 0x7f5e2c117a00 secure :-1 s=THROTTLE_DONE pgs=1971 cs=0 l=1 rx=0x7f5e2c11bec0 tx=0x7f5e20010860).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:32.141 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:32.126+0000 7f5e1b7fe700 1 --2- 172.21.15.42:0/2305166136 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5e1003b1c0 0x7f5e1003d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:26:32.141 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:32.126+0000 7f5e1b7fe700 1 -- 172.21.15.42:0/2305166136 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 10847+0+0 (secure 0 0 0) 0x7f5e2004b020 con 0x7f5e2c118cc0 2019-12-05T06:26:32.141 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:32.126+0000 7f5e2ad9d700 1 --2- 172.21.15.42:0/2305166136 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5e1003b1c0 0x7f5e1003d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:26:32.141 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:32.126+0000 7f5e2ad9d700 1 --2- 172.21.15.42:0/2305166136 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5e1003b1c0 0x7f5e1003d6d0 crc :-1 s=READY pgs=1101 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:26:32.142 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:32.130+0000 7f5e3092c700 1 -- 172.21.15.42:0/2305166136 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e2c05d110 con 0x7f5e2c118cc0 2019-12-05T06:26:32.145 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:32.138+0000 7f5e2a59c700 1 --2- 172.21.15.42:0/2305166136 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5e2c118cc0 0x7f5e2c117a00 secure :-1 s=THROTTLE_DONE pgs=1971 cs=0 l=1 rx=0x7f5e2c11bec0 tx=0x7f5e20010860).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:32.145 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:32.138+0000 7f5e1b7fe700 1 -- 172.21.15.42:0/2305166136 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f5e20032020 con 0x7f5e2c118cc0 2019-12-05T06:26:32.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:32.342+0000 7f5e3092c700 1 -- 172.21.15.42:0/2305166136 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5e2c05b8c0 con 0x7f5e2c118cc0 2019-12-05T06:26:32.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:32.342+0000 7f5e2a59c700 1 --2- 172.21.15.42:0/2305166136 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5e2c118cc0 0x7f5e2c117a00 secure :-1 s=THROTTLE_DONE pgs=1971 cs=0 l=1 rx=0x7f5e2c11bec0 tx=0x7f5e20010860).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:32.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:32.342+0000 7f5e1b7fe700 1 -- 172.21.15.42:0/2305166136 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 313 v313) v1 ==== 95+0+2114 (secure 0 0 0) 0x7f5e20032280 con 0x7f5e2c118cc0 2019-12-05T06:26:32.353 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 313 2019-12-05T06:26:32.353 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:26:32.354 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":313,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":11221,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/3427104185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3427104185},{"type":"v1","addr":"172.21.15.42:6835","nonce":3427104185}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":313}],"filesystems":[{"mdsmap":{"epoch":312,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:24:39.885811+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":105,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7568},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7568":{"gid":7568,"name":"b","rank":0,"incarnation":309,"state":"up:active","state_seq":14,"addr":"172.21.15.40:6833/2324657302","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2324657302},{"type":"v1","addr":"172.21.15.40:6833","nonce":2324657302}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:26:32.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:32.346+0000 7f5e3092c700 1 -- 172.21.15.42:0/2305166136 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5e1003b1c0 msgr2=0x7f5e1003d6d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:32.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:32.346+0000 7f5e3092c700 1 --2- 172.21.15.42:0/2305166136 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5e1003b1c0 0x7f5e1003d6d0 crc :-1 s=READY pgs=1101 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:32.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:32.346+0000 7f5e3092c700 1 -- 172.21.15.42:0/2305166136 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5e2c118cc0 msgr2=0x7f5e2c117a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:32.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:32.346+0000 7f5e3092c700 1 --2- 172.21.15.42:0/2305166136 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5e2c118cc0 0x7f5e2c117a00 secure :-1 s=READY pgs=1971 cs=0 l=1 rx=0x7f5e2c11bec0 tx=0x7f5e20010860).stop 2019-12-05T06:26:32.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:32.346+0000 7f5e3092c700 1 -- 172.21.15.42:0/2305166136 shutdown_connections 2019-12-05T06:26:32.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:32.346+0000 7f5e3092c700 1 --2- 172.21.15.42:0/2305166136 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5e1003b1c0 0x7f5e1003d6d0 unknown :-1 s=CLOSED pgs=1101 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:32.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:32.346+0000 7f5e3092c700 1 --2- 172.21.15.42:0/2305166136 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5e2c11c240 0x7f5e2c1306b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:32.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:32.346+0000 7f5e3092c700 1 --2- 172.21.15.42:0/2305166136 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5e2c118cc0 0x7f5e2c117a00 unknown :-1 s=CLOSED pgs=1971 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:32.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:32.346+0000 7f5e3092c700 1 -- 172.21.15.42:0/2305166136 shutdown_connections 2019-12-05T06:26:32.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:32.346+0000 7f5e3092c700 1 -- 172.21.15.42:0/2305166136 wait complete. 2019-12-05T06:26:32.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:32.346+0000 7f5e3092c700 1 -- 172.21.15.42:0/2305166136 >> 172.21.15.42:0/2305166136 conn(0x7f5e2c08fde0 msgr2=0x7f5e2c059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:26:32.367 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:26:34.367 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:26:34.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:34.466+0000 7f9d407fb700 1 Processor -- start 2019-12-05T06:26:34.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:34.466+0000 7f9d407fb700 1 -- start start 2019-12-05T06:26:34.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:34.466+0000 7f9d407fb700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9d380a7740 0x7f9d3808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:26:34.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:34.466+0000 7f9d407fb700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f9d3807e720 0x7f9d3807eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:26:34.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:34.466+0000 7f9d407fb700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d380a0710 con 0x7f9d3809f2e0 2019-12-05T06:26:34.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:34.466+0000 7f9d407fb700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d380a0b00 con 0x7f9d3807e720 2019-12-05T06:26:34.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:34.466+0000 7f9d407fb700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d3808e5b0 con 0x7f9d380a7740 2019-12-05T06:26:34.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:34.466+0000 7f9d3f599700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f9d3809f2e0 0x7f9d3809f710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:55238/0 (socket says 172.21.15.42:55238) 2019-12-05T06:26:34.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:34.466+0000 7f9d3f599700 1 -- 172.21.15.42:0/2004440026 learned_addr learned my addr 172.21.15.42:0/2004440026 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:26:34.482 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:34.466+0000 7f9d3ed98700 1 --2- 172.21.15.42:0/2004440026 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9d380a7740 0x7f9d3808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:26:34.482 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:34.466+0000 7f9d3e597700 1 --2- 172.21.15.42:0/2004440026 >> v2:172.21.15.40:3300/0 conn(0x7f9d3807e720 0x7f9d3807eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:26:34.482 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:34.470+0000 7f9d3dd96700 1 -- 172.21.15.42:0/2004440026 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1898548018 0 0) 0x7f9d380a0710 con 0x7f9d3809f2e0 2019-12-05T06:26:34.482 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:34.470+0000 7f9d3dd96700 1 -- 172.21.15.42:0/2004440026 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d240030b0 con 0x7f9d3809f2e0 2019-12-05T06:26:34.483 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:34.470+0000 7f9d3ed98700 1 -- 172.21.15.42:0/2004440026 >> v2:172.21.15.40:3300/0 conn(0x7f9d3807e720 msgr2=0x7f9d3807eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:26:34.483 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:34.470+0000 7f9d3ed98700 1 --2- 172.21.15.42:0/2004440026 >> v2:172.21.15.40:3300/0 conn(0x7f9d3807e720 0x7f9d3807eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:26:34.483 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:34.470+0000 7f9d3ed98700 1 -- 172.21.15.42:0/2004440026 >> v1:172.21.15.42:6789/0 conn(0x7f9d3809f2e0 legacy=0x7f9d3809f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:34.483 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:34.470+0000 7f9d3ed98700 1 -- 172.21.15.42:0/2004440026 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d3807f0a0 con 0x7f9d380a7740 2019-12-05T06:26:34.483 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:34.470+0000 7f9d3e597700 1 --2- 172.21.15.42:0/2004440026 >> v2:172.21.15.40:3300/0 conn(0x7f9d3807e720 0x7f9d3807eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:26:34.484 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:34.470+0000 7f9d3ed98700 1 --2- 172.21.15.42:0/2004440026 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9d380a7740 0x7f9d3808df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f9d34008790 tx=0x7f9d3400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:34.484 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:34.470+0000 7f9d3ed98700 1 --2- 172.21.15.42:0/2004440026 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9d380a7740 0x7f9d3808df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f9d34008790 tx=0x7f9d3400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:34.484 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:34.470+0000 7f9d3ed98700 1 --2- 172.21.15.42:0/2004440026 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9d380a7740 0x7f9d3808df80 secure :-1 s=READY pgs=574 cs=0 l=1 rx=0x7f9d34008790 tx=0x7f9d3400fc70).ready entity=mon.2 client_cookie=63f74c9471adcf36 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:26:34.485 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:34.470+0000 7f9d3ed98700 1 --2- 172.21.15.42:0/2004440026 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9d380a7740 0x7f9d3808df80 secure :-1 s=THROTTLE_DONE pgs=574 cs=0 l=1 rx=0x7f9d34008790 tx=0x7f9d3400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:34.485 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:34.470+0000 7f9d3dd96700 1 -- 172.21.15.42:0/2004440026 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9d340112f0 con 0x7f9d380a7740 2019-12-05T06:26:34.486 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:34.470+0000 7f9d3ed98700 1 --2- 172.21.15.42:0/2004440026 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9d380a7740 0x7f9d3808df80 secure :-1 s=THROTTLE_DONE pgs=574 cs=0 l=1 rx=0x7f9d34008790 tx=0x7f9d3400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:34.486 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:34.470+0000 7f9d3dd96700 1 -- 172.21.15.42:0/2004440026 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d34008ad0 con 0x7f9d380a7740 2019-12-05T06:26:34.486 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:34.470+0000 7f9d3ed98700 1 --2- 172.21.15.42:0/2004440026 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9d380a7740 0x7f9d3808df80 secure :-1 s=THROTTLE_DONE pgs=574 cs=0 l=1 rx=0x7f9d34008790 tx=0x7f9d3400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:34.486 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:34.470+0000 7f9d407fb700 1 -- 172.21.15.42:0/2004440026 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9d380a7740 msgr2=0x7f9d3808df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:34.487 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:34.470+0000 7f9d3dd96700 1 -- 172.21.15.42:0/2004440026 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9d340118d0 con 0x7f9d380a7740 2019-12-05T06:26:34.487 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:34.470+0000 7f9d407fb700 1 --2- 172.21.15.42:0/2004440026 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9d380a7740 0x7f9d3808df80 secure :-1 s=READY pgs=574 cs=0 l=1 rx=0x7f9d34008790 tx=0x7f9d3400fc70).stop 2019-12-05T06:26:34.487 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:34.470+0000 7f9d407fb700 1 -- 172.21.15.42:0/2004440026 shutdown_connections 2019-12-05T06:26:34.487 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:34.470+0000 7f9d407fb700 1 --2- 172.21.15.42:0/2004440026 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9d380a7740 0x7f9d3808df80 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:34.487 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:34.470+0000 7f9d407fb700 1 --2- 172.21.15.42:0/2004440026 >> v2:172.21.15.40:3300/0 conn(0x7f9d3807e720 0x7f9d3807eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:26:34.487 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:34.470+0000 7f9d407fb700 1 -- 172.21.15.42:0/2004440026 shutdown_connections 2019-12-05T06:26:34.488 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:34.470+0000 7f9d407fb700 1 -- 172.21.15.42:0/2004440026 wait complete. 2019-12-05T06:26:34.488 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:34.470+0000 7f9d407fb700 1 -- 172.21.15.42:0/2004440026 >> 172.21.15.42:0/2004440026 conn(0x7f9d380aabe0 msgr2=0x7f9d380a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:26:34.492 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:34.474+0000 7f9d407fb700 1 Processor -- start 2019-12-05T06:26:34.492 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:34.474+0000 7f9d407fb700 1 -- start start 2019-12-05T06:26:34.493 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:34.474+0000 7f9d407fb700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f9d3811b5c0 0x7f9d380a3960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:26:34.493 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:34.474+0000 7f9d407fb700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d38118450 con 0x7f9d380722c0 2019-12-05T06:26:34.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:34.474+0000 7f9d407fb700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d38117810 con 0x7f9d38117310 2019-12-05T06:26:34.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:34.474+0000 7f9d407fb700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d38119d00 con 0x7f9d3811b5c0 2019-12-05T06:26:34.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:34.474+0000 7f9d3ed98700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f9d38117310 0x7f9d3811adc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:55240/0 (socket says 172.21.15.42:55240) 2019-12-05T06:26:34.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:34.474+0000 7f9d3ed98700 1 -- 172.21.15.42:0/713546425 learned_addr learned my addr 172.21.15.42:0/713546425 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:26:34.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:34.474+0000 7f9d3e597700 1 --2- 172.21.15.42:0/713546425 >> v2:172.21.15.42:3300/0 conn(0x7f9d3811b5c0 0x7f9d380a3960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:26:34.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:34.474+0000 7f9d3e597700 1 -- 172.21.15.42:0/713546425 >> v1:172.21.15.42:6789/0 conn(0x7f9d38117310 legacy=0x7f9d3811adc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:34.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:34.474+0000 7f9d3e597700 1 -- 172.21.15.42:0/713546425 >> v1:172.21.15.40:6789/0 conn(0x7f9d380722c0 legacy=0x7f9d38074700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:34.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:34.474+0000 7f9d2f7fe700 1 -- 172.21.15.42:0/713546425 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1964274809 0 0) 0x7f9d38117810 con 0x7f9d38117310 2019-12-05T06:26:34.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:34.474+0000 7f9d3e597700 1 -- 172.21.15.42:0/713546425 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d38077f30 con 0x7f9d3811b5c0 2019-12-05T06:26:34.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:34.474+0000 7f9d3e597700 1 --2- 172.21.15.42:0/713546425 >> v2:172.21.15.42:3300/0 conn(0x7f9d3811b5c0 0x7f9d380a3960 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f9d380a4570 tx=0x7f9d28010c50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:34.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:34.474+0000 7f9d3e597700 1 --2- 172.21.15.42:0/713546425 >> v2:172.21.15.42:3300/0 conn(0x7f9d3811b5c0 0x7f9d380a3960 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f9d380a4570 tx=0x7f9d28010c50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:34.496 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:34.474+0000 7f9d3e597700 1 --2- 172.21.15.42:0/713546425 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9d3811b5c0 0x7f9d380a3960 secure :-1 s=READY pgs=2647 cs=0 l=1 rx=0x7f9d380a4570 tx=0x7f9d28010c50).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:26:34.497 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:34.474+0000 7f9d3e597700 1 --2- 172.21.15.42:0/713546425 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9d3811b5c0 0x7f9d380a3960 secure :-1 s=THROTTLE_DONE pgs=2647 cs=0 l=1 rx=0x7f9d380a4570 tx=0x7f9d28010c50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:34.497 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:34.474+0000 7f9d2f7fe700 1 -- 172.21.15.42:0/713546425 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9d28005890 con 0x7f9d3811b5c0 2019-12-05T06:26:34.497 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:34.474+0000 7f9d3e597700 1 --2- 172.21.15.42:0/713546425 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9d3811b5c0 0x7f9d380a3960 secure :-1 s=THROTTLE_DONE pgs=2647 cs=0 l=1 rx=0x7f9d380a4570 tx=0x7f9d28010c50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:34.497 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:34.474+0000 7f9d407fb700 1 -- 172.21.15.42:0/713546425 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d38075f10 con 0x7f9d3811b5c0 2019-12-05T06:26:34.498 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:34.474+0000 7f9d2f7fe700 1 -- 172.21.15.42:0/713546425 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d28007160 con 0x7f9d3811b5c0 2019-12-05T06:26:34.498 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:34.474+0000 7f9d3e597700 1 --2- 172.21.15.42:0/713546425 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9d3811b5c0 0x7f9d380a3960 secure :-1 s=THROTTLE_DONE pgs=2647 cs=0 l=1 rx=0x7f9d380a4570 tx=0x7f9d28010c50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:34.498 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:34.474+0000 7f9d2f7fe700 1 -- 172.21.15.42:0/713546425 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9d280122d0 con 0x7f9d3811b5c0 2019-12-05T06:26:34.498 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:34.474+0000 7f9d407fb700 1 -- 172.21.15.42:0/713546425 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d380764c0 con 0x7f9d3811b5c0 2019-12-05T06:26:34.498 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:34.478+0000 7f9d3e597700 1 --2- 172.21.15.42:0/713546425 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9d3811b5c0 0x7f9d380a3960 secure :-1 s=THROTTLE_DONE pgs=2647 cs=0 l=1 rx=0x7f9d380a4570 tx=0x7f9d28010c50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:34.498 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:34.478+0000 7f9d2f7fe700 1 -- 172.21.15.42:0/713546425 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f9d28012630 con 0x7f9d3811b5c0 2019-12-05T06:26:34.499 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:34.478+0000 7f9d3e597700 1 --2- 172.21.15.42:0/713546425 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9d3811b5c0 0x7f9d380a3960 secure :-1 s=THROTTLE_DONE pgs=2647 cs=0 l=1 rx=0x7f9d380a4570 tx=0x7f9d28010c50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:34.499 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:34.478+0000 7f9d2f7fe700 1 --2- 172.21.15.42:0/713546425 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9d2003ac50 0x7f9d2003d160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:26:34.499 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:34.478+0000 7f9d2f7fe700 1 -- 172.21.15.42:0/713546425 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 10847+0+0 (secure 0 0 0) 0x7f9d2804b700 con 0x7f9d3811b5c0 2019-12-05T06:26:34.499 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:34.478+0000 7f9d3ed98700 1 --2- 172.21.15.42:0/713546425 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9d2003ac50 0x7f9d2003d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:26:34.499 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:34.478+0000 7f9d3ed98700 1 --2- 172.21.15.42:0/713546425 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9d2003ac50 0x7f9d2003d160 crc :-1 s=READY pgs=1102 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:26:34.500 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:34.482+0000 7f9d407fb700 1 -- 172.21.15.42:0/713546425 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d3805d110 con 0x7f9d3811b5c0 2019-12-05T06:26:34.500 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:34.486+0000 7f9d3e597700 1 --2- 172.21.15.42:0/713546425 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9d3811b5c0 0x7f9d380a3960 secure :-1 s=THROTTLE_DONE pgs=2647 cs=0 l=1 rx=0x7f9d380a4570 tx=0x7f9d28010c50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:34.501 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:34.486+0000 7f9d2f7fe700 1 -- 172.21.15.42:0/713546425 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f9d28032780 con 0x7f9d3811b5c0 2019-12-05T06:26:34.689 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:34.682+0000 7f9d407fb700 1 -- 172.21.15.42:0/713546425 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9d3805b8c0 con 0x7f9d3811b5c0 2019-12-05T06:26:34.691 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:34.682+0000 7f9d3e597700 1 --2- 172.21.15.42:0/713546425 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9d3811b5c0 0x7f9d380a3960 secure :-1 s=THROTTLE_DONE pgs=2647 cs=0 l=1 rx=0x7f9d380a4570 tx=0x7f9d28010c50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:34.691 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:34.682+0000 7f9d2f7fe700 1 -- 172.21.15.42:0/713546425 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 313 v313) v1 ==== 95+0+2114 (secure 0 0 0) 0x7f9d280329e0 con 0x7f9d3811b5c0 2019-12-05T06:26:34.691 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 313 2019-12-05T06:26:34.692 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:26:34.692 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":313,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":11221,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/3427104185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3427104185},{"type":"v1","addr":"172.21.15.42:6835","nonce":3427104185}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":313}],"filesystems":[{"mdsmap":{"epoch":312,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:24:39.885811+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":105,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7568},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7568":{"gid":7568,"name":"b","rank":0,"incarnation":309,"state":"up:active","state_seq":14,"addr":"172.21.15.40:6833/2324657302","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2324657302},{"type":"v1","addr":"172.21.15.40:6833","nonce":2324657302}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:26:34.694 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:34.686+0000 7f9d407fb700 1 -- 172.21.15.42:0/713546425 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9d2003ac50 msgr2=0x7f9d2003d160 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:34.695 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:34.686+0000 7f9d407fb700 1 --2- 172.21.15.42:0/713546425 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9d2003ac50 0x7f9d2003d160 crc :-1 s=READY pgs=1102 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:34.695 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:34.686+0000 7f9d407fb700 1 -- 172.21.15.42:0/713546425 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9d3811b5c0 msgr2=0x7f9d380a3960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:34.695 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:34.686+0000 7f9d407fb700 1 --2- 172.21.15.42:0/713546425 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9d3811b5c0 0x7f9d380a3960 secure :-1 s=READY pgs=2647 cs=0 l=1 rx=0x7f9d380a4570 tx=0x7f9d28010c50).stop 2019-12-05T06:26:34.695 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:34.686+0000 7f9d407fb700 1 -- 172.21.15.42:0/713546425 shutdown_connections 2019-12-05T06:26:34.696 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:34.686+0000 7f9d407fb700 1 --2- 172.21.15.42:0/713546425 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9d2003ac50 0x7f9d2003d160 unknown :-1 s=CLOSED pgs=1102 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:34.697 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:34.686+0000 7f9d407fb700 1 --2- 172.21.15.42:0/713546425 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9d3811b5c0 0x7f9d380a3960 unknown :-1 s=CLOSED pgs=2647 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:34.697 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:34.686+0000 7f9d407fb700 1 -- 172.21.15.42:0/713546425 shutdown_connections 2019-12-05T06:26:34.697 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:34.686+0000 7f9d407fb700 1 -- 172.21.15.42:0/713546425 wait complete. 2019-12-05T06:26:34.698 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:34.686+0000 7f9d407fb700 1 -- 172.21.15.42:0/713546425 >> 172.21.15.42:0/713546425 conn(0x7f9d3809f2e0 msgr2=0x7f9d38059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:26:34.703 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:26:36.708 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:26:36.826 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:36.814+0000 7fed16706700 1 Processor -- start 2019-12-05T06:26:36.827 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:36.814+0000 7fed16706700 1 -- start start 2019-12-05T06:26:36.827 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:36.814+0000 7fed16706700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fed100a7740 0x7fed1008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:26:36.827 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:36.814+0000 7fed16706700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fed1009fae0 0x7fed1007eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:26:36.827 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:36.814+0000 7fed16706700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed100a0710 con 0x7fed1007e720 2019-12-05T06:26:36.827 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:36.814+0000 7fed16706700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed100a0b00 con 0x7fed100a7740 2019-12-05T06:26:36.828 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:36.814+0000 7fed16706700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed1008e5b0 con 0x7fed1009fae0 2019-12-05T06:26:36.828 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:36.814+0000 7fed14ca3700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fed100a7740 0x7fed1008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:26:36.828 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:36.814+0000 7fed14ca3700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fed100a7740 0x7fed1008df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:55210/0 (socket says 172.21.15.42:55210) 2019-12-05T06:26:36.828 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:36.814+0000 7fed14ca3700 1 -- 172.21.15.42:0/2888871306 learned_addr learned my addr 172.21.15.42:0/2888871306 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:26:36.828 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:36.814+0000 7fed154a4700 1 --2- 172.21.15.42:0/2888871306 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fed1009fae0 0x7fed1007eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:26:36.828 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:36.818+0000 7fed14ca3700 1 -- 172.21.15.42:0/2888871306 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fed1009fae0 msgr2=0x7fed1007eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:26:36.829 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:36.818+0000 7fed14ca3700 1 --2- 172.21.15.42:0/2888871306 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fed1009fae0 0x7fed1007eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:26:36.829 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:36.818+0000 7fed14ca3700 1 -- 172.21.15.42:0/2888871306 >> v1:172.21.15.40:6789/0 conn(0x7fed1007e720 legacy=0x7fed1009f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:36.829 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:36.818+0000 7fed14ca3700 1 -- 172.21.15.42:0/2888871306 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed1007f0a0 con 0x7fed100a7740 2019-12-05T06:26:36.829 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:36.818+0000 7fed14ca3700 1 --2- 172.21.15.42:0/2888871306 >> v2:172.21.15.42:3300/0 conn(0x7fed100a7740 0x7fed1008df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fed04008790 tx=0x7fed0400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:36.829 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:36.818+0000 7fed14ca3700 1 --2- 172.21.15.42:0/2888871306 >> v2:172.21.15.42:3300/0 conn(0x7fed100a7740 0x7fed1008df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fed04008790 tx=0x7fed0400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:36.830 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:36.818+0000 7fed14ca3700 1 --2- 172.21.15.42:0/2888871306 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fed100a7740 0x7fed1008df80 secure :-1 s=READY pgs=2648 cs=0 l=1 rx=0x7fed04008790 tx=0x7fed0400fd30).ready entity=mon.1 client_cookie=b09fc43ccf9c2fc server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:26:36.830 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:36.818+0000 7fed14ca3700 1 --2- 172.21.15.42:0/2888871306 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fed100a7740 0x7fed1008df80 secure :-1 s=THROTTLE_DONE pgs=2648 cs=0 l=1 rx=0x7fed04008790 tx=0x7fed0400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:36.830 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:36.818+0000 7fed0f7fe700 1 -- 172.21.15.42:0/2888871306 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fed04008c90 con 0x7fed100a7740 2019-12-05T06:26:36.830 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:36.818+0000 7fed14ca3700 1 --2- 172.21.15.42:0/2888871306 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fed100a7740 0x7fed1008df80 secure :-1 s=THROTTLE_DONE pgs=2648 cs=0 l=1 rx=0x7fed04008790 tx=0x7fed0400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:36.830 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:36.818+0000 7fed14ca3700 1 --2- 172.21.15.42:0/2888871306 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fed100a7740 0x7fed1008df80 secure :-1 s=THROTTLE_DONE pgs=2648 cs=0 l=1 rx=0x7fed04008790 tx=0x7fed0400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:36.830 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:36.818+0000 7fed0f7fe700 1 -- 172.21.15.42:0/2888871306 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed040088b0 con 0x7fed100a7740 2019-12-05T06:26:36.830 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:36.818+0000 7fed0f7fe700 1 -- 172.21.15.42:0/2888871306 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fed04011b10 con 0x7fed100a7740 2019-12-05T06:26:36.831 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:36.818+0000 7fed16706700 1 -- 172.21.15.42:0/2888871306 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fed100a7740 msgr2=0x7fed1008df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:36.831 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:36.818+0000 7fed16706700 1 --2- 172.21.15.42:0/2888871306 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fed100a7740 0x7fed1008df80 secure :-1 s=READY pgs=2648 cs=0 l=1 rx=0x7fed04008790 tx=0x7fed0400fd30).stop 2019-12-05T06:26:36.831 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:36.818+0000 7fed16706700 1 -- 172.21.15.42:0/2888871306 shutdown_connections 2019-12-05T06:26:36.831 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:36.818+0000 7fed16706700 1 --2- 172.21.15.42:0/2888871306 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fed1009fae0 0x7fed1007eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:26:36.832 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:36.818+0000 7fed16706700 1 --2- 172.21.15.42:0/2888871306 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fed100a7740 0x7fed1008df80 unknown :-1 s=CLOSED pgs=2648 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:36.832 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:36.818+0000 7fed16706700 1 -- 172.21.15.42:0/2888871306 shutdown_connections 2019-12-05T06:26:36.832 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:36.818+0000 7fed16706700 1 -- 172.21.15.42:0/2888871306 wait complete. 2019-12-05T06:26:36.832 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:36.818+0000 7fed16706700 1 -- 172.21.15.42:0/2888871306 >> 172.21.15.42:0/2888871306 conn(0x7fed100aabe0 msgr2=0x7fed100a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:26:36.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:36.818+0000 7fed16706700 1 Processor -- start 2019-12-05T06:26:36.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:36.818+0000 7fed16706700 1 -- start start 2019-12-05T06:26:36.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:36.818+0000 7fed16706700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fed1011a630 0x7fed101242d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:26:36.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:36.818+0000 7fed16706700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fed1012e2d0 0x7fed1012e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:26:36.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:36.818+0000 7fed16706700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed10132580 con 0x7fed10131890 2019-12-05T06:26:36.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:36.818+0000 7fed16706700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed1011aa60 con 0x7fed1012e2d0 2019-12-05T06:26:36.837 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:36.818+0000 7fed16706700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed10131da0 con 0x7fed1011a630 2019-12-05T06:26:36.837 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:36.818+0000 7fed154a4700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fed1012e2d0 0x7fed1012e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:26:36.837 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:36.818+0000 7fed154a4700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fed1012e2d0 0x7fed1012e700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:55220/0 (socket says 172.21.15.42:55220) 2019-12-05T06:26:36.837 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:36.822+0000 7fed154a4700 1 -- 172.21.15.42:0/2311692599 learned_addr learned my addr 172.21.15.42:0/2311692599 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:26:36.837 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:36.822+0000 7fed14ca3700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fed10131890 0x7fed1007c380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:55254/0 (socket says 172.21.15.42:55254) 2019-12-05T06:26:36.837 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:36.822+0000 7fed0ffff700 1 --2- 172.21.15.42:0/2311692599 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fed1011a630 0x7fed101242d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:26:36.838 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:36.822+0000 7fed154a4700 1 -- 172.21.15.42:0/2311692599 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fed1011a630 msgr2=0x7fed101242d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:36.838 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:36.822+0000 7fed154a4700 1 --2- 172.21.15.42:0/2311692599 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fed1011a630 0x7fed101242d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:36.838 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:36.822+0000 7fed154a4700 1 -- 172.21.15.42:0/2311692599 >> v1:172.21.15.42:6789/0 conn(0x7fed10131890 legacy=0x7fed1007c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:36.838 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:36.822+0000 7fed0d7fa700 1 -- 172.21.15.42:0/2311692599 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1022576461 0 0) 0x7fed10132580 con 0x7fed10131890 2019-12-05T06:26:36.838 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:36.822+0000 7fed154a4700 1 -- 172.21.15.42:0/2311692599 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed100a4b20 con 0x7fed1012e2d0 2019-12-05T06:26:36.839 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:36.822+0000 7fed154a4700 1 --2- 172.21.15.42:0/2311692599 >> v2:172.21.15.42:3300/0 conn(0x7fed1012e2d0 0x7fed1012e700 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fed080032b0 tx=0x7fed08010d00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:36.839 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:36.822+0000 7fed154a4700 1 --2- 172.21.15.42:0/2311692599 >> v2:172.21.15.42:3300/0 conn(0x7fed1012e2d0 0x7fed1012e700 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fed080032b0 tx=0x7fed08010d00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:36.839 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:36.822+0000 7fed154a4700 1 --2- 172.21.15.42:0/2311692599 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fed1012e2d0 0x7fed1012e700 secure :-1 s=READY pgs=2650 cs=0 l=1 rx=0x7fed080032b0 tx=0x7fed08010d00).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:26:36.839 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:36.822+0000 7fed154a4700 1 --2- 172.21.15.42:0/2311692599 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fed1012e2d0 0x7fed1012e700 secure :-1 s=THROTTLE_DONE pgs=2650 cs=0 l=1 rx=0x7fed080032b0 tx=0x7fed08010d00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:36.840 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:36.822+0000 7fed0d7fa700 1 -- 172.21.15.42:0/2311692599 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fed08012a30 con 0x7fed1012e2d0 2019-12-05T06:26:36.840 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:36.822+0000 7fed154a4700 1 --2- 172.21.15.42:0/2311692599 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fed1012e2d0 0x7fed1012e700 secure :-1 s=THROTTLE_DONE pgs=2650 cs=0 l=1 rx=0x7fed080032b0 tx=0x7fed08010d00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:36.840 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:36.822+0000 7fed154a4700 1 --2- 172.21.15.42:0/2311692599 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fed1012e2d0 0x7fed1012e700 secure :-1 s=THROTTLE_DONE pgs=2650 cs=0 l=1 rx=0x7fed080032b0 tx=0x7fed08010d00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:36.841 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:36.822+0000 7fed0d7fa700 1 -- 172.21.15.42:0/2311692599 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed08012650 con 0x7fed1012e2d0 2019-12-05T06:26:36.841 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:36.822+0000 7fed16706700 1 -- 172.21.15.42:0/2311692599 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed10138ce0 con 0x7fed1012e2d0 2019-12-05T06:26:36.842 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:36.822+0000 7fed0d7fa700 1 -- 172.21.15.42:0/2311692599 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fed0800e760 con 0x7fed1012e2d0 2019-12-05T06:26:36.842 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:36.822+0000 7fed16706700 1 -- 172.21.15.42:0/2311692599 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed10139210 con 0x7fed1012e2d0 2019-12-05T06:26:36.842 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:36.822+0000 7fed154a4700 1 --2- 172.21.15.42:0/2311692599 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fed1012e2d0 0x7fed1012e700 secure :-1 s=THROTTLE_DONE pgs=2650 cs=0 l=1 rx=0x7fed080032b0 tx=0x7fed08010d00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:36.842 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:36.822+0000 7fed0d7fa700 1 -- 172.21.15.42:0/2311692599 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fed0800ea70 con 0x7fed1012e2d0 2019-12-05T06:26:36.843 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:36.822+0000 7fed154a4700 1 --2- 172.21.15.42:0/2311692599 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fed1012e2d0 0x7fed1012e700 secure :-1 s=THROTTLE_DONE pgs=2650 cs=0 l=1 rx=0x7fed080032b0 tx=0x7fed08010d00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:36.843 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:36.822+0000 7fed0d7fa700 1 --2- 172.21.15.42:0/2311692599 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fecf003ad10 0x7fecf003d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:26:36.843 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:36.822+0000 7fed0d7fa700 1 -- 172.21.15.42:0/2311692599 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 10847+0+0 (secure 0 0 0) 0x7fed0804b760 con 0x7fed1012e2d0 2019-12-05T06:26:36.843 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:36.822+0000 7fed14ca3700 1 --2- 172.21.15.42:0/2311692599 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fecf003ad10 0x7fecf003d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:26:36.843 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:36.822+0000 7fed14ca3700 1 --2- 172.21.15.42:0/2311692599 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fecf003ad10 0x7fecf003d220 crc :-1 s=READY pgs=1103 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:26:36.844 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:36.830+0000 7fed16706700 1 -- 172.21.15.42:0/2311692599 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed10124820 con 0x7fed1012e2d0 2019-12-05T06:26:36.844 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:36.838+0000 7fed154a4700 1 --2- 172.21.15.42:0/2311692599 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fed1012e2d0 0x7fed1012e700 secure :-1 s=THROTTLE_DONE pgs=2650 cs=0 l=1 rx=0x7fed080032b0 tx=0x7fed08010d00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:36.845 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:36.838+0000 7fed0d7fa700 1 -- 172.21.15.42:0/2311692599 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fed080327e0 con 0x7fed1012e2d0 2019-12-05T06:26:37.032 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:37.026+0000 7fed16706700 1 -- 172.21.15.42:0/2311692599 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fed100aadf0 con 0x7fed1012e2d0 2019-12-05T06:26:37.033 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:37.026+0000 7fed154a4700 1 --2- 172.21.15.42:0/2311692599 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fed1012e2d0 0x7fed1012e700 secure :-1 s=THROTTLE_DONE pgs=2650 cs=0 l=1 rx=0x7fed080032b0 tx=0x7fed08010d00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:37.033 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:37.026+0000 7fed0d7fa700 1 -- 172.21.15.42:0/2311692599 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 313 v313) v1 ==== 95+0+2114 (secure 0 0 0) 0x7fed08032a40 con 0x7fed1012e2d0 2019-12-05T06:26:37.033 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 313 2019-12-05T06:26:37.034 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:26:37.034 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":313,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":11221,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/3427104185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3427104185},{"type":"v1","addr":"172.21.15.42:6835","nonce":3427104185}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":313}],"filesystems":[{"mdsmap":{"epoch":312,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:24:39.885811+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":105,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7568},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7568":{"gid":7568,"name":"b","rank":0,"incarnation":309,"state":"up:active","state_seq":14,"addr":"172.21.15.40:6833/2324657302","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2324657302},{"type":"v1","addr":"172.21.15.40:6833","nonce":2324657302}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:26:37.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:37.026+0000 7fed16706700 1 -- 172.21.15.42:0/2311692599 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fecf003ad10 msgr2=0x7fecf003d220 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:37.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:37.026+0000 7fed16706700 1 --2- 172.21.15.42:0/2311692599 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fecf003ad10 0x7fecf003d220 crc :-1 s=READY pgs=1103 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:37.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:37.026+0000 7fed16706700 1 -- 172.21.15.42:0/2311692599 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fed1012e2d0 msgr2=0x7fed1012e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:37.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:37.026+0000 7fed16706700 1 --2- 172.21.15.42:0/2311692599 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fed1012e2d0 0x7fed1012e700 secure :-1 s=READY pgs=2650 cs=0 l=1 rx=0x7fed080032b0 tx=0x7fed08010d00).stop 2019-12-05T06:26:37.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:37.026+0000 7fed16706700 1 -- 172.21.15.42:0/2311692599 shutdown_connections 2019-12-05T06:26:37.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:37.026+0000 7fed16706700 1 --2- 172.21.15.42:0/2311692599 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fecf003ad10 0x7fecf003d220 unknown :-1 s=CLOSED pgs=1103 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:37.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:37.026+0000 7fed16706700 1 --2- 172.21.15.42:0/2311692599 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fed1012e2d0 0x7fed1012e700 unknown :-1 s=CLOSED pgs=2650 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:37.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:37.026+0000 7fed16706700 1 --2- 172.21.15.42:0/2311692599 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fed1011a630 0x7fed101242d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:37.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:37.026+0000 7fed16706700 1 -- 172.21.15.42:0/2311692599 shutdown_connections 2019-12-05T06:26:37.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:37.026+0000 7fed16706700 1 -- 172.21.15.42:0/2311692599 wait complete. 2019-12-05T06:26:37.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:37.026+0000 7fed16706700 1 -- 172.21.15.42:0/2311692599 >> 172.21.15.42:0/2311692599 conn(0x7fed100a82a0 msgr2=0x7fed1009f4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:26:37.047 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:26:39.048 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:26:39.156 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:39.142+0000 7f0be11de700 1 Processor -- start 2019-12-05T06:26:39.156 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:39.142+0000 7f0be11de700 1 -- start start 2019-12-05T06:26:39.156 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:39.142+0000 7f0be11de700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f0bdc0a7730 0x7f0bdc08df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:26:39.157 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:39.142+0000 7f0be11de700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0bdc07e710 0x7f0bdc07eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:26:39.157 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:39.142+0000 7f0be11de700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bdc0a0700 con 0x7f0bdc09f2d0 2019-12-05T06:26:39.157 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:39.142+0000 7f0be11de700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bdc0a0af0 con 0x7f0bdc0a7730 2019-12-05T06:26:39.157 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:39.142+0000 7f0be11de700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bdc08e5a0 con 0x7f0bdc07e710 2019-12-05T06:26:39.157 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:39.146+0000 7f0be09dd700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f0bdc09f2d0 0x7f0bdc09f700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:55268/0 (socket says 172.21.15.42:55268) 2019-12-05T06:26:39.157 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:39.146+0000 7f0be09dd700 1 -- 172.21.15.42:0/1650982887 learned_addr learned my addr 172.21.15.42:0/1650982887 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:26:39.157 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:39.146+0000 7f0bdb59e700 1 --2- 172.21.15.42:0/1650982887 >> v2:172.21.15.40:3300/0 conn(0x7f0bdc0a7730 0x7f0bdc08df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:26:39.158 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:39.146+0000 7f0bdad9d700 1 --2- 172.21.15.42:0/1650982887 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0bdc07e710 0x7f0bdc07eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:26:39.158 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:39.146+0000 7f0bda59c700 1 -- 172.21.15.42:0/1650982887 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1160740248 0 0) 0x7f0bdc0a0700 con 0x7f0bdc09f2d0 2019-12-05T06:26:39.158 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:39.146+0000 7f0bda59c700 1 -- 172.21.15.42:0/1650982887 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0bc40030b0 con 0x7f0bdc09f2d0 2019-12-05T06:26:39.158 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:39.146+0000 7f0bdad9d700 1 -- 172.21.15.42:0/1650982887 >> v2:172.21.15.40:3300/0 conn(0x7f0bdc0a7730 msgr2=0x7f0bdc08df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:26:39.158 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:39.146+0000 7f0bdad9d700 1 --2- 172.21.15.42:0/1650982887 >> v2:172.21.15.40:3300/0 conn(0x7f0bdc0a7730 0x7f0bdc08df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:26:39.158 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:39.146+0000 7f0bdad9d700 1 -- 172.21.15.42:0/1650982887 >> v1:172.21.15.42:6789/0 conn(0x7f0bdc09f2d0 legacy=0x7f0bdc09f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:39.158 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:39.146+0000 7f0bda59c700 1 -- 172.21.15.42:0/1650982887 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1284372643 0 0) 0x7f0bc40030b0 con 0x7f0bdc09f2d0 2019-12-05T06:26:39.159 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:39.146+0000 7f0bdad9d700 1 -- 172.21.15.42:0/1650982887 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bdc07f090 con 0x7f0bdc07e710 2019-12-05T06:26:39.159 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:39.146+0000 7f0bdad9d700 1 --2- 172.21.15.42:0/1650982887 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0bdc07e710 0x7f0bdc07eb40 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f0bcc008790 tx=0x7f0bd4001d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:39.159 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:39.146+0000 7f0bdad9d700 1 --2- 172.21.15.42:0/1650982887 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0bdc07e710 0x7f0bdc07eb40 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f0bcc008790 tx=0x7f0bd4001d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:39.159 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:39.146+0000 7f0bdad9d700 1 --2- 172.21.15.42:0/1650982887 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0bdc07e710 0x7f0bdc07eb40 secure :-1 s=READY pgs=575 cs=0 l=1 rx=0x7f0bcc008790 tx=0x7f0bd4001d50).ready entity=mon.2 client_cookie=6e777fd094f45d53 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:26:39.159 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:39.146+0000 7f0bdad9d700 1 --2- 172.21.15.42:0/1650982887 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0bdc07e710 0x7f0bdc07eb40 secure :-1 s=THROTTLE_DONE pgs=575 cs=0 l=1 rx=0x7f0bcc008790 tx=0x7f0bd4001d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:39.159 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:39.146+0000 7f0bda59c700 1 -- 172.21.15.42:0/1650982887 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0bd4001ad0 con 0x7f0bdc07e710 2019-12-05T06:26:39.159 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:39.146+0000 7f0bdad9d700 1 --2- 172.21.15.42:0/1650982887 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0bdc07e710 0x7f0bdc07eb40 secure :-1 s=THROTTLE_DONE pgs=575 cs=0 l=1 rx=0x7f0bcc008790 tx=0x7f0bd4001d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:39.160 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:39.146+0000 7f0bdad9d700 1 --2- 172.21.15.42:0/1650982887 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0bdc07e710 0x7f0bdc07eb40 secure :-1 s=THROTTLE_DONE pgs=575 cs=0 l=1 rx=0x7f0bcc008790 tx=0x7f0bd4001d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:39.160 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:39.146+0000 7f0bda59c700 1 -- 172.21.15.42:0/1650982887 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bd40018e0 con 0x7f0bdc07e710 2019-12-05T06:26:39.160 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:39.146+0000 7f0bda59c700 1 -- 172.21.15.42:0/1650982887 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0bcc0113e0 con 0x7f0bdc07e710 2019-12-05T06:26:39.160 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:39.146+0000 7f0be11de700 1 -- 172.21.15.42:0/1650982887 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0bdc07e710 msgr2=0x7f0bdc07eb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:39.160 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:39.146+0000 7f0be11de700 1 --2- 172.21.15.42:0/1650982887 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0bdc07e710 0x7f0bdc07eb40 secure :-1 s=READY pgs=575 cs=0 l=1 rx=0x7f0bcc008790 tx=0x7f0bd4001d50).stop 2019-12-05T06:26:39.160 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:39.146+0000 7f0be11de700 1 -- 172.21.15.42:0/1650982887 shutdown_connections 2019-12-05T06:26:39.160 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:39.146+0000 7f0be11de700 1 --2- 172.21.15.42:0/1650982887 >> v2:172.21.15.40:3300/0 conn(0x7f0bdc0a7730 0x7f0bdc08df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:26:39.161 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:39.146+0000 7f0be11de700 1 --2- 172.21.15.42:0/1650982887 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0bdc07e710 0x7f0bdc07eb40 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:39.161 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:39.146+0000 7f0be11de700 1 -- 172.21.15.42:0/1650982887 shutdown_connections 2019-12-05T06:26:39.161 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:39.146+0000 7f0be11de700 1 -- 172.21.15.42:0/1650982887 wait complete. 2019-12-05T06:26:39.161 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:39.146+0000 7f0be11de700 1 -- 172.21.15.42:0/1650982887 >> 172.21.15.42:0/1650982887 conn(0x7f0bdc0aabd0 msgr2=0x7f0bdc0a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:26:39.164 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:39.150+0000 7f0be11de700 1 Processor -- start 2019-12-05T06:26:39.164 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:39.150+0000 7f0be11de700 1 -- start start 2019-12-05T06:26:39.164 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:39.150+0000 7f0be11de700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0bdc121620 0x7f0bdc121a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:26:39.165 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:39.150+0000 7f0be11de700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bdc116d00 con 0x7f0bdc0a9970 2019-12-05T06:26:39.165 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:39.150+0000 7f0be11de700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bdc134a20 con 0x7f0bdc133df0 2019-12-05T06:26:39.165 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:39.150+0000 7f0be11de700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bdc117040 con 0x7f0bdc121620 2019-12-05T06:26:39.165 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:39.150+0000 7f0bdad9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0bdc121620 0x7f0bdc121a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:26:39.165 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:39.150+0000 7f0bdad9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0bdc121620 0x7f0bdc121a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:55234/0 (socket says 172.21.15.42:55234) 2019-12-05T06:26:39.165 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:39.150+0000 7f0bdad9d700 1 -- 172.21.15.42:0/1485299819 learned_addr learned my addr 172.21.15.42:0/1485299819 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:26:39.165 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:39.150+0000 7f0bdad9d700 1 -- 172.21.15.42:0/1485299819 >> v1:172.21.15.42:6789/0 conn(0x7f0bdc133df0 legacy=0x7f0bdc134220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:39.166 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:39.150+0000 7f0bdad9d700 1 -- 172.21.15.42:0/1485299819 >> v1:172.21.15.40:6789/0 conn(0x7f0bdc0a9970 legacy=0x7f0bdc122470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:39.166 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:39.150+0000 7f0bdad9d700 1 -- 172.21.15.42:0/1485299819 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bdc117580 con 0x7f0bdc121620 2019-12-05T06:26:39.166 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:39.154+0000 7f0bdad9d700 1 --2- 172.21.15.42:0/1485299819 >> v2:172.21.15.42:3300/0 conn(0x7f0bdc121620 0x7f0bdc121a50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f0bcc008790 tx=0x7f0bcc00f490).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:39.166 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:39.154+0000 7f0bdad9d700 1 --2- 172.21.15.42:0/1485299819 >> v2:172.21.15.42:3300/0 conn(0x7f0bdc121620 0x7f0bdc121a50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f0bcc008790 tx=0x7f0bcc00f490).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:39.166 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:39.154+0000 7f0bdad9d700 1 --2- 172.21.15.42:0/1485299819 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0bdc121620 0x7f0bdc121a50 secure :-1 s=READY pgs=2653 cs=0 l=1 rx=0x7f0bcc008790 tx=0x7f0bcc00f490).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:26:39.166 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:39.154+0000 7f0bdad9d700 1 --2- 172.21.15.42:0/1485299819 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0bdc121620 0x7f0bdc121a50 secure :-1 s=THROTTLE_DONE pgs=2653 cs=0 l=1 rx=0x7f0bcc008790 tx=0x7f0bcc00f490).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:39.166 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:39.154+0000 7f0bdad9d700 1 --2- 172.21.15.42:0/1485299819 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0bdc121620 0x7f0bdc121a50 secure :-1 s=THROTTLE_DONE pgs=2653 cs=0 l=1 rx=0x7f0bcc008790 tx=0x7f0bcc00f490).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:39.167 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:39.154+0000 7f0bdad9d700 1 --2- 172.21.15.42:0/1485299819 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0bdc121620 0x7f0bdc121a50 secure :-1 s=THROTTLE_DONE pgs=2653 cs=0 l=1 rx=0x7f0bcc008790 tx=0x7f0bcc00f490).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:39.167 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:39.154+0000 7f0bc3fff700 1 -- 172.21.15.42:0/1485299819 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0bcc007150 con 0x7f0bdc121620 2019-12-05T06:26:39.167 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:39.154+0000 7f0bc3fff700 1 -- 172.21.15.42:0/1485299819 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bcc008cc0 con 0x7f0bdc121620 2019-12-05T06:26:39.167 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:39.154+0000 7f0be11de700 1 -- 172.21.15.42:0/1485299819 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bdc13c140 con 0x7f0bdc121620 2019-12-05T06:26:39.167 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:39.154+0000 7f0bc3fff700 1 -- 172.21.15.42:0/1485299819 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0bcc011af0 con 0x7f0bdc121620 2019-12-05T06:26:39.167 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:39.154+0000 7f0be11de700 1 -- 172.21.15.42:0/1485299819 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bdc13c550 con 0x7f0bdc121620 2019-12-05T06:26:39.167 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:39.154+0000 7f0bdad9d700 1 --2- 172.21.15.42:0/1485299819 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0bdc121620 0x7f0bdc121a50 secure :-1 s=THROTTLE_DONE pgs=2653 cs=0 l=1 rx=0x7f0bcc008790 tx=0x7f0bcc00f490).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:39.168 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:39.154+0000 7f0bc3fff700 1 -- 172.21.15.42:0/1485299819 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f0bcc005540 con 0x7f0bdc121620 2019-12-05T06:26:39.168 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:39.154+0000 7f0bdad9d700 1 --2- 172.21.15.42:0/1485299819 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0bdc121620 0x7f0bdc121a50 secure :-1 s=THROTTLE_DONE pgs=2653 cs=0 l=1 rx=0x7f0bcc008790 tx=0x7f0bcc00f490).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:39.168 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:39.154+0000 7f0bc3fff700 1 --2- 172.21.15.42:0/1485299819 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0bbc03ad70 0x7f0bbc03d280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:26:39.168 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:39.154+0000 7f0bc3fff700 1 -- 172.21.15.42:0/1485299819 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 10847+0+0 (secure 0 0 0) 0x7f0bcc04b590 con 0x7f0bdc121620 2019-12-05T06:26:39.168 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:39.154+0000 7f0bdb59e700 1 --2- 172.21.15.42:0/1485299819 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0bbc03ad70 0x7f0bbc03d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:26:39.168 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:39.154+0000 7f0bdb59e700 1 --2- 172.21.15.42:0/1485299819 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0bbc03ad70 0x7f0bbc03d280 crc :-1 s=READY pgs=1104 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:26:39.169 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:39.158+0000 7f0be11de700 1 -- 172.21.15.42:0/1485299819 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bdc05d100 con 0x7f0bdc121620 2019-12-05T06:26:39.173 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:39.166+0000 7f0bdad9d700 1 --2- 172.21.15.42:0/1485299819 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0bdc121620 0x7f0bdc121a50 secure :-1 s=THROTTLE_DONE pgs=2653 cs=0 l=1 rx=0x7f0bcc008790 tx=0x7f0bcc00f490).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:39.174 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:39.166+0000 7f0bc3fff700 1 -- 172.21.15.42:0/1485299819 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f0bcc012060 con 0x7f0bdc121620 2019-12-05T06:26:39.398 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:39.390+0000 7f0be11de700 1 -- 172.21.15.42:0/1485299819 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0bdc08fa90 con 0x7f0bdc121620 2019-12-05T06:26:39.399 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:39.390+0000 7f0bdad9d700 1 --2- 172.21.15.42:0/1485299819 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0bdc121620 0x7f0bdc121a50 secure :-1 s=THROTTLE_DONE pgs=2653 cs=0 l=1 rx=0x7f0bcc008790 tx=0x7f0bcc00f490).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:39.399 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:39.390+0000 7f0bc3fff700 1 -- 172.21.15.42:0/1485299819 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 313 v313) v1 ==== 95+0+2114 (secure 0 0 0) 0x7f0bcc00cbf0 con 0x7f0bdc121620 2019-12-05T06:26:39.400 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 313 2019-12-05T06:26:39.400 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:26:39.400 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":313,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":11221,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/3427104185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3427104185},{"type":"v1","addr":"172.21.15.42:6835","nonce":3427104185}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":313}],"filesystems":[{"mdsmap":{"epoch":312,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:24:39.885811+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":105,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7568},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7568":{"gid":7568,"name":"b","rank":0,"incarnation":309,"state":"up:active","state_seq":14,"addr":"172.21.15.40:6833/2324657302","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2324657302},{"type":"v1","addr":"172.21.15.40:6833","nonce":2324657302}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:26:39.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:39.394+0000 7f0be11de700 1 -- 172.21.15.42:0/1485299819 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0bbc03ad70 msgr2=0x7f0bbc03d280 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:39.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:39.394+0000 7f0be11de700 1 --2- 172.21.15.42:0/1485299819 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0bbc03ad70 0x7f0bbc03d280 crc :-1 s=READY pgs=1104 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:39.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:39.394+0000 7f0be11de700 1 -- 172.21.15.42:0/1485299819 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0bdc121620 msgr2=0x7f0bdc121a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:39.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:39.394+0000 7f0be11de700 1 --2- 172.21.15.42:0/1485299819 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0bdc121620 0x7f0bdc121a50 secure :-1 s=READY pgs=2653 cs=0 l=1 rx=0x7f0bcc008790 tx=0x7f0bcc00f490).stop 2019-12-05T06:26:39.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:39.394+0000 7f0be11de700 1 -- 172.21.15.42:0/1485299819 shutdown_connections 2019-12-05T06:26:39.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:39.394+0000 7f0be11de700 1 --2- 172.21.15.42:0/1485299819 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0bbc03ad70 0x7f0bbc03d280 unknown :-1 s=CLOSED pgs=1104 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:39.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:39.394+0000 7f0be11de700 1 --2- 172.21.15.42:0/1485299819 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0bdc121620 0x7f0bdc121a50 unknown :-1 s=CLOSED pgs=2653 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:39.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:39.394+0000 7f0be11de700 1 -- 172.21.15.42:0/1485299819 shutdown_connections 2019-12-05T06:26:39.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:39.394+0000 7f0be11de700 1 -- 172.21.15.42:0/1485299819 wait complete. 2019-12-05T06:26:39.404 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:39.394+0000 7f0be11de700 1 -- 172.21.15.42:0/1485299819 >> 172.21.15.42:0/1485299819 conn(0x7f0bdc09f6a0 msgr2=0x7f0bdc0a7f80 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:26:39.413 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:26:41.414 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:26:41.506 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.490+0000 7f0de506d700 1 Processor -- start 2019-12-05T06:26:41.506 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.490+0000 7f0de506d700 1 -- start start 2019-12-05T06:26:41.506 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.490+0000 7f0de506d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0de00a7740 0x7f0de008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:26:41.506 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.490+0000 7f0de506d700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0de00a0710 con 0x7f0de007e720 2019-12-05T06:26:41.507 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.490+0000 7f0de506d700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0de00a9ba0 con 0x7f0de009fae0 2019-12-05T06:26:41.507 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.490+0000 7f0de506d700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0de00a0b00 con 0x7f0de00a7740 2019-12-05T06:26:41.507 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.490+0000 7f0ddf59e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0de00a7740 0x7f0de008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:26:41.507 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.494+0000 7f0ddf59e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0de00a7740 0x7f0de008df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:55256/0 (socket says 172.21.15.42:55256) 2019-12-05T06:26:41.508 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.494+0000 7f0ddf59e700 1 -- 172.21.15.42:0/2471977526 learned_addr learned my addr 172.21.15.42:0/2471977526 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:26:41.508 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.494+0000 7f0ddf59e700 1 -- 172.21.15.42:0/2471977526 >> v1:172.21.15.42:6789/0 conn(0x7f0de009fae0 legacy=0x7f0de014efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:41.508 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.494+0000 7f0ddf59e700 1 -- 172.21.15.42:0/2471977526 >> v1:172.21.15.40:6789/0 conn(0x7f0de007e720 legacy=0x7f0de009f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:41.508 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.494+0000 7f0ddf59e700 1 -- 172.21.15.42:0/2471977526 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0de007eb50 con 0x7f0de00a7740 2019-12-05T06:26:41.509 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.494+0000 7f0ddf59e700 1 --2- 172.21.15.42:0/2471977526 >> v2:172.21.15.42:3300/0 conn(0x7f0de00a7740 0x7f0de008df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f0dd4001040 tx=0x7f0dd80014d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:41.509 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.494+0000 7f0ddf59e700 1 --2- 172.21.15.42:0/2471977526 >> v2:172.21.15.42:3300/0 conn(0x7f0de00a7740 0x7f0de008df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f0dd4001040 tx=0x7f0dd80014d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:41.509 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.494+0000 7f0ddf59e700 1 --2- 172.21.15.42:0/2471977526 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0de00a7740 0x7f0de008df80 secure :-1 s=READY pgs=2655 cs=0 l=1 rx=0x7f0dd4001040 tx=0x7f0dd80014d0).ready entity=mon.1 client_cookie=d7ba6160bbb836d0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:26:41.509 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.494+0000 7f0ddf59e700 1 --2- 172.21.15.42:0/2471977526 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0de00a7740 0x7f0de008df80 secure :-1 s=THROTTLE_DONE pgs=2655 cs=0 l=1 rx=0x7f0dd4001040 tx=0x7f0dd80014d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:41.509 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.494+0000 7f0ddf59e700 1 --2- 172.21.15.42:0/2471977526 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0de00a7740 0x7f0de008df80 secure :-1 s=THROTTLE_DONE pgs=2655 cs=0 l=1 rx=0x7f0dd4001040 tx=0x7f0dd80014d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:41.509 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.494+0000 7f0ddf59e700 1 --2- 172.21.15.42:0/2471977526 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0de00a7740 0x7f0de008df80 secure :-1 s=THROTTLE_DONE pgs=2655 cs=0 l=1 rx=0x7f0dd4001040 tx=0x7f0dd80014d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:41.510 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.494+0000 7f0dde59c700 1 -- 172.21.15.42:0/2471977526 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0dd0012060 con 0x7f0de00a7740 2019-12-05T06:26:41.510 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.494+0000 7f0dde59c700 1 -- 172.21.15.42:0/2471977526 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0dd00110c0 con 0x7f0de00a7740 2019-12-05T06:26:41.510 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.494+0000 7f0dde59c700 1 -- 172.21.15.42:0/2471977526 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0dd00116a0 con 0x7f0de00a7740 2019-12-05T06:26:41.510 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.494+0000 7f0de506d700 1 -- 172.21.15.42:0/2471977526 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0de00a7740 msgr2=0x7f0de008df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:41.510 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.494+0000 7f0de506d700 1 --2- 172.21.15.42:0/2471977526 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0de00a7740 0x7f0de008df80 secure :-1 s=READY pgs=2655 cs=0 l=1 rx=0x7f0dd4001040 tx=0x7f0dd80014d0).stop 2019-12-05T06:26:41.510 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.494+0000 7f0de506d700 1 -- 172.21.15.42:0/2471977526 shutdown_connections 2019-12-05T06:26:41.511 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.494+0000 7f0de506d700 1 --2- 172.21.15.42:0/2471977526 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0de00a7740 0x7f0de008df80 unknown :-1 s=CLOSED pgs=2655 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:41.511 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.494+0000 7f0de506d700 1 -- 172.21.15.42:0/2471977526 shutdown_connections 2019-12-05T06:26:41.511 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.494+0000 7f0de506d700 1 -- 172.21.15.42:0/2471977526 wait complete. 2019-12-05T06:26:41.511 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.494+0000 7f0de506d700 1 -- 172.21.15.42:0/2471977526 >> 172.21.15.42:0/2471977526 conn(0x7f0de00aabe0 msgr2=0x7f0de00a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:26:41.511 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.494+0000 7f0de506d700 1 Processor -- start 2019-12-05T06:26:41.511 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.494+0000 7f0de506d700 1 -- start start 2019-12-05T06:26:41.511 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.498+0000 7f0de506d700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f0de0067320 0x7f0de0142a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:26:41.512 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.498+0000 7f0de506d700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0de0143d00 0x7f0de012a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:26:41.512 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.498+0000 7f0de506d700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0de0144130 con 0x7f0de01290e0 2019-12-05T06:26:41.512 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.498+0000 7f0de506d700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0de0129b20 con 0x7f0de0067320 2019-12-05T06:26:41.512 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.498+0000 7f0de506d700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0de0129dc0 con 0x7f0de0143d00 2019-12-05T06:26:41.512 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.498+0000 7f0de486c700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f0de01290e0 0x7f0de012d0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:55302/0 (socket says 172.21.15.42:55302) 2019-12-05T06:26:41.512 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.498+0000 7f0de486c700 1 -- 172.21.15.42:0/3035761523 learned_addr learned my addr 172.21.15.42:0/3035761523 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:26:41.513 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.498+0000 7f0dded9d700 1 --2- 172.21.15.42:0/3035761523 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0de0143d00 0x7f0de012a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:26:41.513 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.498+0000 7f0ddf59e700 1 --2- 172.21.15.42:0/3035761523 >> v2:172.21.15.40:3300/0 conn(0x7f0de0067320 0x7f0de0142a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:26:41.513 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.498+0000 7f0dc7fff700 1 -- 172.21.15.42:0/3035761523 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3466182622 0 0) 0x7f0de0144130 con 0x7f0de01290e0 2019-12-05T06:26:41.513 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.498+0000 7f0dc7fff700 1 -- 172.21.15.42:0/3035761523 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0dc00030b0 con 0x7f0de01290e0 2019-12-05T06:26:41.513 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.498+0000 7f0dded9d700 1 -- 172.21.15.42:0/3035761523 >> v2:172.21.15.40:3300/0 conn(0x7f0de0067320 msgr2=0x7f0de0142a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:41.513 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.498+0000 7f0dded9d700 1 --2- 172.21.15.42:0/3035761523 >> v2:172.21.15.40:3300/0 conn(0x7f0de0067320 0x7f0de0142a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:41.514 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.498+0000 7f0dded9d700 1 -- 172.21.15.42:0/3035761523 >> v1:172.21.15.42:6789/0 conn(0x7f0de01290e0 legacy=0x7f0de012d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:41.514 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.498+0000 7f0dc7fff700 1 -- 172.21.15.42:0/3035761523 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 3045064599 0 0) 0x7f0dc00030b0 con 0x7f0de01290e0 2019-12-05T06:26:41.514 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.498+0000 7f0dded9d700 1 -- 172.21.15.42:0/3035761523 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0de01180f0 con 0x7f0de0143d00 2019-12-05T06:26:41.514 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.498+0000 7f0ddf59e700 1 --2- 172.21.15.42:0/3035761523 >> v2:172.21.15.40:3300/0 conn(0x7f0de0067320 0x7f0de0142a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:26:41.514 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.498+0000 7f0dded9d700 1 --2- 172.21.15.42:0/3035761523 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0de0143d00 0x7f0de012a5a0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f0de0117fe0 tx=0x7f0dd40109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:41.514 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.498+0000 7f0dded9d700 1 --2- 172.21.15.42:0/3035761523 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0de0143d00 0x7f0de012a5a0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f0de0117fe0 tx=0x7f0dd40109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:41.515 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.498+0000 7f0dded9d700 1 --2- 172.21.15.42:0/3035761523 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0de0143d00 0x7f0de012a5a0 secure :-1 s=READY pgs=576 cs=0 l=1 rx=0x7f0de0117fe0 tx=0x7f0dd40109a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:26:41.517 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.498+0000 7f0dded9d700 1 --2- 172.21.15.42:0/3035761523 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0de0143d00 0x7f0de012a5a0 secure :-1 s=THROTTLE_DONE pgs=576 cs=0 l=1 rx=0x7f0de0117fe0 tx=0x7f0dd40109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:41.517 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.498+0000 7f0dded9d700 1 --2- 172.21.15.42:0/3035761523 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0de0143d00 0x7f0de012a5a0 secure :-1 s=THROTTLE_DONE pgs=576 cs=0 l=1 rx=0x7f0de0117fe0 tx=0x7f0dd40109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:41.517 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.498+0000 7f0dded9d700 1 --2- 172.21.15.42:0/3035761523 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0de0143d00 0x7f0de012a5a0 secure :-1 s=THROTTLE_DONE pgs=576 cs=0 l=1 rx=0x7f0de0117fe0 tx=0x7f0dd40109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:41.517 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.498+0000 7f0dc7fff700 1 -- 172.21.15.42:0/3035761523 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0dd400caa0 con 0x7f0de0143d00 2019-12-05T06:26:41.517 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.498+0000 7f0dc7fff700 1 -- 172.21.15.42:0/3035761523 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0dd400c6c0 con 0x7f0de0143d00 2019-12-05T06:26:41.518 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.498+0000 7f0dc7fff700 1 -- 172.21.15.42:0/3035761523 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0dd4011120 con 0x7f0de0143d00 2019-12-05T06:26:41.519 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.498+0000 7f0de506d700 1 -- 172.21.15.42:0/3035761523 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0de0129510 con 0x7f0de0143d00 2019-12-05T06:26:41.519 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.498+0000 7f0de506d700 1 -- 172.21.15.42:0/3035761523 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0de012d8e0 con 0x7f0de0143d00 2019-12-05T06:26:41.519 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.498+0000 7f0dded9d700 1 --2- 172.21.15.42:0/3035761523 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0de0143d00 0x7f0de012a5a0 secure :-1 s=THROTTLE_DONE pgs=576 cs=0 l=1 rx=0x7f0de0117fe0 tx=0x7f0dd40109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:41.519 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.502+0000 7f0dc7fff700 1 -- 172.21.15.42:0/3035761523 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f0dd4007430 con 0x7f0de0143d00 2019-12-05T06:26:41.519 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.502+0000 7f0dded9d700 1 --2- 172.21.15.42:0/3035761523 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0de0143d00 0x7f0de012a5a0 secure :-1 s=THROTTLE_DONE pgs=576 cs=0 l=1 rx=0x7f0de0117fe0 tx=0x7f0dd40109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:41.520 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.502+0000 7f0dc7fff700 1 --2- 172.21.15.42:0/3035761523 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0dc003b2a0 0x7f0dc003d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:26:41.520 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.502+0000 7f0dc7fff700 1 -- 172.21.15.42:0/3035761523 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 10933+0+0 (secure 0 0 0) 0x7f0dd404b0a0 con 0x7f0de0143d00 2019-12-05T06:26:41.520 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.502+0000 7f0ddf59e700 1 --2- 172.21.15.42:0/3035761523 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0dc003b2a0 0x7f0dc003d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:26:41.520 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.502+0000 7f0ddf59e700 1 --2- 172.21.15.42:0/3035761523 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0dc003b2a0 0x7f0dc003d7b0 crc :-1 s=READY pgs=1105 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:26:41.520 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.506+0000 7f0de506d700 1 -- 172.21.15.42:0/3035761523 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0de005d110 con 0x7f0de0143d00 2019-12-05T06:26:41.523 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.514+0000 7f0dded9d700 1 --2- 172.21.15.42:0/3035761523 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0de0143d00 0x7f0de012a5a0 secure :-1 s=THROTTLE_DONE pgs=576 cs=0 l=1 rx=0x7f0de0117fe0 tx=0x7f0dd40109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:41.523 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.514+0000 7f0dc7fff700 1 -- 172.21.15.42:0/3035761523 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f0dd4032120 con 0x7f0de0143d00 2019-12-05T06:26:41.848 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.842+0000 7f0de506d700 1 -- 172.21.15.42:0/3035761523 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0de005b8c0 con 0x7f0de0143d00 2019-12-05T06:26:41.849 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.842+0000 7f0dded9d700 1 --2- 172.21.15.42:0/3035761523 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0de0143d00 0x7f0de012a5a0 secure :-1 s=THROTTLE_DONE pgs=576 cs=0 l=1 rx=0x7f0de0117fe0 tx=0x7f0dd40109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:41.849 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.842+0000 7f0dc7fff700 1 -- 172.21.15.42:0/3035761523 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 314 v314) v1 ==== 95+0+1778 (secure 0 0 0) 0x7f0dd4032380 con 0x7f0de0143d00 2019-12-05T06:26:41.851 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 314 2019-12-05T06:26:41.851 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:26:41.851 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":314,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":314,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:26:39.891183+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":106,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11221},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11221":{"gid":11221,"name":"a","rank":0,"incarnation":314,"state":"up:replay","state_seq":2,"addr":"172.21.15.42:6835/3427104185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3427104185},{"type":"v1","addr":"172.21.15.42:6835","nonce":3427104185}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:26:41.855 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.846+0000 7f0de506d700 1 -- 172.21.15.42:0/3035761523 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0dc003b2a0 msgr2=0x7f0dc003d7b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:41.855 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.846+0000 7f0de506d700 1 --2- 172.21.15.42:0/3035761523 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0dc003b2a0 0x7f0dc003d7b0 crc :-1 s=READY pgs=1105 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:41.856 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.846+0000 7f0de506d700 1 -- 172.21.15.42:0/3035761523 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0de0143d00 msgr2=0x7f0de012a5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:41.856 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.846+0000 7f0de506d700 1 --2- 172.21.15.42:0/3035761523 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0de0143d00 0x7f0de012a5a0 secure :-1 s=READY pgs=576 cs=0 l=1 rx=0x7f0de0117fe0 tx=0x7f0dd40109a0).stop 2019-12-05T06:26:41.856 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.846+0000 7f0de506d700 1 -- 172.21.15.42:0/3035761523 shutdown_connections 2019-12-05T06:26:41.856 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.846+0000 7f0de506d700 1 --2- 172.21.15.42:0/3035761523 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0dc003b2a0 0x7f0dc003d7b0 unknown :-1 s=CLOSED pgs=1105 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:41.856 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.846+0000 7f0de506d700 1 --2- 172.21.15.42:0/3035761523 >> v2:172.21.15.40:3300/0 conn(0x7f0de0067320 0x7f0de0142a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:41.857 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.846+0000 7f0de506d700 1 --2- 172.21.15.42:0/3035761523 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0de0143d00 0x7f0de012a5a0 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:41.857 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.846+0000 7f0de506d700 1 -- 172.21.15.42:0/3035761523 shutdown_connections 2019-12-05T06:26:41.857 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.846+0000 7f0de506d700 1 -- 172.21.15.42:0/3035761523 wait complete. 2019-12-05T06:26:41.857 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.846+0000 7f0de506d700 1 -- 172.21.15.42:0/3035761523 >> 172.21.15.42:0/3035761523 conn(0x7f0de00a7740 msgr2=0x7f0de0115280 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:26:41.866 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b down, removed from mdsmap 2019-12-05T06:26:41.866 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2019-12-05T06:26:41.867 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:26:41.964 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.950+0000 7f4ba0bd1700 1 Processor -- start 2019-12-05T06:26:41.964 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.950+0000 7f4ba0bd1700 1 -- start start 2019-12-05T06:26:41.965 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.954+0000 7f4ba0bd1700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f4b9c0a7740 0x7f4b9c08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:26:41.965 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.954+0000 7f4ba0bd1700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4b9c07e720 0x7f4b9c07eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:26:41.965 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.954+0000 7f4ba0bd1700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b9c0a0710 con 0x7f4b9c09f2e0 2019-12-05T06:26:41.965 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.954+0000 7f4ba0bd1700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b9c0a0b00 con 0x7f4b9c0a7740 2019-12-05T06:26:41.965 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.954+0000 7f4ba0bd1700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b9c08e5b0 con 0x7f4b9c07e720 2019-12-05T06:26:41.965 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.954+0000 7f4b9b59e700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f4b9c09f2e0 0x7f4b9c09f710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:55312/0 (socket says 172.21.15.42:55312) 2019-12-05T06:26:41.966 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.954+0000 7f4b9b59e700 1 -- 172.21.15.42:0/2473481323 learned_addr learned my addr 172.21.15.42:0/2473481323 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:26:41.966 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.954+0000 7f4b93fff700 1 --2- 172.21.15.42:0/2473481323 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4b9c07e720 0x7f4b9c07eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:26:41.966 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.954+0000 7f4b9ad9d700 1 --2- 172.21.15.42:0/2473481323 >> v2:172.21.15.40:3300/0 conn(0x7f4b9c0a7740 0x7f4b9c08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:26:41.966 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.954+0000 7f4b9a59c700 1 -- 172.21.15.42:0/2473481323 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1035004487 0 0) 0x7f4b9c0a0710 con 0x7f4b9c09f2e0 2019-12-05T06:26:41.966 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.954+0000 7f4b9a59c700 1 -- 172.21.15.42:0/2473481323 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b880030b0 con 0x7f4b9c09f2e0 2019-12-05T06:26:41.966 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.954+0000 7f4b9ad9d700 1 -- 172.21.15.42:0/2473481323 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4b9c07e720 msgr2=0x7f4b9c07eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:26:41.966 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.954+0000 7f4b9ad9d700 1 --2- 172.21.15.42:0/2473481323 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4b9c07e720 0x7f4b9c07eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:26:41.966 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.954+0000 7f4b9ad9d700 1 -- 172.21.15.42:0/2473481323 >> v1:172.21.15.42:6789/0 conn(0x7f4b9c09f2e0 legacy=0x7f4b9c09f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:41.967 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.954+0000 7f4b9ad9d700 1 -- 172.21.15.42:0/2473481323 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b9c07f0a0 con 0x7f4b9c0a7740 2019-12-05T06:26:41.967 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.954+0000 7f4b93fff700 1 --2- 172.21.15.42:0/2473481323 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4b9c07e720 0x7f4b9c07eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:26:41.967 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.954+0000 7f4b9ad9d700 1 --2- 172.21.15.42:0/2473481323 >> v2:172.21.15.40:3300/0 conn(0x7f4b9c0a7740 0x7f4b9c08df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f4b8c008790 tx=0x7f4b8c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:41.967 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.954+0000 7f4b9ad9d700 1 --2- 172.21.15.42:0/2473481323 >> v2:172.21.15.40:3300/0 conn(0x7f4b9c0a7740 0x7f4b9c08df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f4b8c008790 tx=0x7f4b8c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:41.967 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.954+0000 7f4b9ad9d700 1 --2- 172.21.15.42:0/2473481323 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4b9c0a7740 0x7f4b9c08df80 secure :-1 s=READY pgs=1977 cs=0 l=1 rx=0x7f4b8c008790 tx=0x7f4b8c00fc70).ready entity=mon.0 client_cookie=74d42677d50446c5 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:26:41.967 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.954+0000 7f4b9ad9d700 1 --2- 172.21.15.42:0/2473481323 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4b9c0a7740 0x7f4b9c08df80 secure :-1 s=THROTTLE_DONE pgs=1977 cs=0 l=1 rx=0x7f4b8c008790 tx=0x7f4b8c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:41.968 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.954+0000 7f4b9a59c700 1 -- 172.21.15.42:0/2473481323 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4b8c0112f0 con 0x7f4b9c0a7740 2019-12-05T06:26:41.968 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.954+0000 7f4b9ad9d700 1 --2- 172.21.15.42:0/2473481323 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4b9c0a7740 0x7f4b9c08df80 secure :-1 s=THROTTLE_DONE pgs=1977 cs=0 l=1 rx=0x7f4b8c008790 tx=0x7f4b8c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:41.968 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.954+0000 7f4b9ad9d700 1 --2- 172.21.15.42:0/2473481323 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4b9c0a7740 0x7f4b9c08df80 secure :-1 s=THROTTLE_DONE pgs=1977 cs=0 l=1 rx=0x7f4b8c008790 tx=0x7f4b8c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:41.968 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.954+0000 7f4b9a59c700 1 -- 172.21.15.42:0/2473481323 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b8c008ad0 con 0x7f4b9c0a7740 2019-12-05T06:26:41.968 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.954+0000 7f4b9a59c700 1 -- 172.21.15.42:0/2473481323 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4b8c0118d0 con 0x7f4b9c0a7740 2019-12-05T06:26:41.969 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.954+0000 7f4ba0bd1700 1 -- 172.21.15.42:0/2473481323 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4b9c0a7740 msgr2=0x7f4b9c08df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:41.969 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.954+0000 7f4ba0bd1700 1 --2- 172.21.15.42:0/2473481323 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4b9c0a7740 0x7f4b9c08df80 secure :-1 s=READY pgs=1977 cs=0 l=1 rx=0x7f4b8c008790 tx=0x7f4b8c00fc70).stop 2019-12-05T06:26:41.969 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.954+0000 7f4ba0bd1700 1 -- 172.21.15.42:0/2473481323 shutdown_connections 2019-12-05T06:26:41.969 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.954+0000 7f4ba0bd1700 1 --2- 172.21.15.42:0/2473481323 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4b9c0a7740 0x7f4b9c08df80 unknown :-1 s=CLOSED pgs=1977 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:41.969 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.954+0000 7f4ba0bd1700 1 --2- 172.21.15.42:0/2473481323 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4b9c07e720 0x7f4b9c07eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:26:41.969 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.954+0000 7f4ba0bd1700 1 -- 172.21.15.42:0/2473481323 shutdown_connections 2019-12-05T06:26:41.971 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.954+0000 7f4ba0bd1700 1 -- 172.21.15.42:0/2473481323 wait complete. 2019-12-05T06:26:41.971 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.954+0000 7f4ba0bd1700 1 -- 172.21.15.42:0/2473481323 >> 172.21.15.42:0/2473481323 conn(0x7f4b9c0aabe0 msgr2=0x7f4b9c0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:26:41.972 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.954+0000 7f4ba0bd1700 1 Processor -- start 2019-12-05T06:26:41.972 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.958+0000 7f4ba0bd1700 1 -- start start 2019-12-05T06:26:41.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.958+0000 7f4ba0bd1700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f4b9c122a00 0x7f4b9c11a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:26:41.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.958+0000 7f4ba0bd1700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f4b9c12c470 0x7f4b9c142510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:26:41.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.958+0000 7f4ba0bd1700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b9c137ac0 con 0x7f4b9c11ac10 2019-12-05T06:26:41.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.958+0000 7f4ba0bd1700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b9c0a7090 con 0x7f4b9c122a00 2019-12-05T06:26:41.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.958+0000 7f4ba0bd1700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b9c0a6480 con 0x7f4b9c12c470 2019-12-05T06:26:41.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.958+0000 7f4b9b59e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f4b9c12c470 0x7f4b9c142510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:26:41.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.958+0000 7f4b9b59e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f4b9c12c470 0x7f4b9c142510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:55278/0 (socket says 172.21.15.42:55278) 2019-12-05T06:26:41.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.958+0000 7f4b9b59e700 1 -- 172.21.15.42:0/1689137834 learned_addr learned my addr 172.21.15.42:0/1689137834 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:26:41.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.958+0000 7f4b9ad9d700 1 --2- 172.21.15.42:0/1689137834 >> v2:172.21.15.40:3300/0 conn(0x7f4b9c122a00 0x7f4b9c11a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:26:41.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.958+0000 7f4b9b59e700 1 -- 172.21.15.42:0/1689137834 >> v2:172.21.15.40:3300/0 conn(0x7f4b9c122a00 msgr2=0x7f4b9c11a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:41.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.958+0000 7f4b9b59e700 1 --2- 172.21.15.42:0/1689137834 >> v2:172.21.15.40:3300/0 conn(0x7f4b9c122a00 0x7f4b9c11a6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:41.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.958+0000 7f4b9b59e700 1 -- 172.21.15.42:0/1689137834 >> v1:172.21.15.42:6789/0 conn(0x7f4b9c11ac10 legacy=0x7f4b9c0a5360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:41.976 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.958+0000 7f4b9b59e700 1 -- 172.21.15.42:0/1689137834 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b9c0a4e00 con 0x7f4b9c12c470 2019-12-05T06:26:41.976 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.958+0000 7f4b9b59e700 1 --2- 172.21.15.42:0/1689137834 >> v2:172.21.15.42:3300/0 conn(0x7f4b9c12c470 0x7f4b9c142510 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f4b9c0a4ac0 tx=0x7f4b94010b90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:41.976 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.958+0000 7f4b9b59e700 1 --2- 172.21.15.42:0/1689137834 >> v2:172.21.15.42:3300/0 conn(0x7f4b9c12c470 0x7f4b9c142510 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f4b9c0a4ac0 tx=0x7f4b94010b90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:41.976 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.958+0000 7f4b9b59e700 1 --2- 172.21.15.42:0/1689137834 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4b9c12c470 0x7f4b9c142510 secure :-1 s=READY pgs=2659 cs=0 l=1 rx=0x7f4b9c0a4ac0 tx=0x7f4b94010b90).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:26:41.976 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.958+0000 7f4b9b59e700 1 --2- 172.21.15.42:0/1689137834 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4b9c12c470 0x7f4b9c142510 secure :-1 s=THROTTLE_DONE pgs=2659 cs=0 l=1 rx=0x7f4b9c0a4ac0 tx=0x7f4b94010b90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:41.976 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.958+0000 7f4b9b59e700 1 --2- 172.21.15.42:0/1689137834 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4b9c12c470 0x7f4b9c142510 secure :-1 s=THROTTLE_DONE pgs=2659 cs=0 l=1 rx=0x7f4b9c0a4ac0 tx=0x7f4b94010b90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:41.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.958+0000 7f4b937fe700 1 -- 172.21.15.42:0/1689137834 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4b94005bc0 con 0x7f4b9c12c470 2019-12-05T06:26:41.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.958+0000 7f4b9b59e700 1 --2- 172.21.15.42:0/1689137834 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4b9c12c470 0x7f4b9c142510 secure :-1 s=THROTTLE_DONE pgs=2659 cs=0 l=1 rx=0x7f4b9c0a4ac0 tx=0x7f4b94010b90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:41.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.958+0000 7f4b937fe700 1 -- 172.21.15.42:0/1689137834 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b940057e0 con 0x7f4b9c12c470 2019-12-05T06:26:41.978 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.958+0000 7f4ba0bd1700 1 -- 172.21.15.42:0/1689137834 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b9c0a5b60 con 0x7f4b9c12c470 2019-12-05T06:26:41.978 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.958+0000 7f4b937fe700 1 -- 172.21.15.42:0/1689137834 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4b94012870 con 0x7f4b9c12c470 2019-12-05T06:26:41.978 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.958+0000 7f4ba0bd1700 1 -- 172.21.15.42:0/1689137834 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b9c12e8b0 con 0x7f4b9c12c470 2019-12-05T06:26:41.978 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.958+0000 7f4b9b59e700 1 --2- 172.21.15.42:0/1689137834 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4b9c12c470 0x7f4b9c142510 secure :-1 s=THROTTLE_DONE pgs=2659 cs=0 l=1 rx=0x7f4b9c0a4ac0 tx=0x7f4b94010b90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:41.978 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.958+0000 7f4b937fe700 1 -- 172.21.15.42:0/1689137834 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f4b94012a70 con 0x7f4b9c12c470 2019-12-05T06:26:41.978 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.958+0000 7f4b9b59e700 1 --2- 172.21.15.42:0/1689137834 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4b9c12c470 0x7f4b9c142510 secure :-1 s=THROTTLE_DONE pgs=2659 cs=0 l=1 rx=0x7f4b9c0a4ac0 tx=0x7f4b94010b90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:41.979 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.962+0000 7f4b937fe700 1 --2- 172.21.15.42:0/1689137834 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4b8004bd30 0x7f4b8004e240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:26:41.979 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.962+0000 7f4b937fe700 1 -- 172.21.15.42:0/1689137834 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 10933+0+0 (secure 0 0 0) 0x7f4b9404b1f0 con 0x7f4b9c12c470 2019-12-05T06:26:41.979 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.962+0000 7f4b9ad9d700 1 --2- 172.21.15.42:0/1689137834 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4b8004bd30 0x7f4b8004e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:26:41.979 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.962+0000 7f4b9ad9d700 1 --2- 172.21.15.42:0/1689137834 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4b8004bd30 0x7f4b8004e240 crc :-1 s=READY pgs=1106 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:26:41.980 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.966+0000 7f4ba0bd1700 1 -- 172.21.15.42:0/1689137834 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b9c05d110 con 0x7f4b9c12c470 2019-12-05T06:26:41.980 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.970+0000 7f4b9b59e700 1 --2- 172.21.15.42:0/1689137834 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4b9c12c470 0x7f4b9c142510 secure :-1 s=THROTTLE_DONE pgs=2659 cs=0 l=1 rx=0x7f4b9c0a4ac0 tx=0x7f4b94010b90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:41.980 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:41.970+0000 7f4b937fe700 1 -- 172.21.15.42:0/1689137834 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f4b9c0a4e00 con 0x7f4b9c12c470 2019-12-05T06:26:42.212 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:42.202+0000 7f4ba0bd1700 1 -- 172.21.15.42:0/1689137834 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4b9c05b8c0 con 0x7f4b9c12c470 2019-12-05T06:26:42.212 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:42.206+0000 7f4b9b59e700 1 --2- 172.21.15.42:0/1689137834 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4b9c12c470 0x7f4b9c142510 secure :-1 s=THROTTLE_DONE pgs=2659 cs=0 l=1 rx=0x7f4b9c0a4ac0 tx=0x7f4b94010b90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:42.213 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:42.206+0000 7f4b937fe700 1 -- 172.21.15.42:0/1689137834 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 314 v314) v1 ==== 95+0+1778 (secure 0 0 0) 0x7f4b94032270 con 0x7f4b9c12c470 2019-12-05T06:26:42.214 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 314 2019-12-05T06:26:42.215 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:26:42.215 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":314,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":314,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:26:39.891183+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":106,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11221},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11221":{"gid":11221,"name":"a","rank":0,"incarnation":314,"state":"up:replay","state_seq":2,"addr":"172.21.15.42:6835/3427104185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3427104185},{"type":"v1","addr":"172.21.15.42:6835","nonce":3427104185}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:26:42.217 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:42.210+0000 7f4ba0bd1700 1 -- 172.21.15.42:0/1689137834 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4b8004bd30 msgr2=0x7f4b8004e240 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:42.217 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:42.210+0000 7f4ba0bd1700 1 --2- 172.21.15.42:0/1689137834 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4b8004bd30 0x7f4b8004e240 crc :-1 s=READY pgs=1106 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:42.218 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:42.210+0000 7f4ba0bd1700 1 -- 172.21.15.42:0/1689137834 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4b9c12c470 msgr2=0x7f4b9c142510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:42.218 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:42.210+0000 7f4ba0bd1700 1 --2- 172.21.15.42:0/1689137834 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4b9c12c470 0x7f4b9c142510 secure :-1 s=READY pgs=2659 cs=0 l=1 rx=0x7f4b9c0a4ac0 tx=0x7f4b94010b90).stop 2019-12-05T06:26:42.218 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:42.210+0000 7f4ba0bd1700 1 -- 172.21.15.42:0/1689137834 shutdown_connections 2019-12-05T06:26:42.218 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:42.210+0000 7f4ba0bd1700 1 --2- 172.21.15.42:0/1689137834 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4b8004bd30 0x7f4b8004e240 unknown :-1 s=CLOSED pgs=1106 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:42.218 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:42.210+0000 7f4ba0bd1700 1 --2- 172.21.15.42:0/1689137834 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4b9c12c470 0x7f4b9c142510 unknown :-1 s=CLOSED pgs=2659 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:42.218 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:42.210+0000 7f4ba0bd1700 1 --2- 172.21.15.42:0/1689137834 >> v2:172.21.15.40:3300/0 conn(0x7f4b9c122a00 0x7f4b9c11a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:42.218 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:42.210+0000 7f4ba0bd1700 1 -- 172.21.15.42:0/1689137834 shutdown_connections 2019-12-05T06:26:42.219 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:42.210+0000 7f4ba0bd1700 1 -- 172.21.15.42:0/1689137834 wait complete. 2019-12-05T06:26:42.219 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:42.210+0000 7f4ba0bd1700 1 -- 172.21.15.42:0/1689137834 >> 172.21.15.42:0/1689137834 conn(0x7f4b9c0a7f90 msgr2=0x7f4b9c11b690 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:26:42.229 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 314, "feature_flags": { "enable_multiple": false, "ever_enabled_multiple": false }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2019-12-05T05:03:35.604738+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 314, "ever_allowed_features": 2, "explicitly_allowed_features": 2, "failed": [], "flags": 18, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_11221": { "addr": "172.21.15.42:6835/3427104185", "addrs": { "addrvec": [ { "addr": "172.21.15.42:6834", "nonce": 3427104185, "type": "v2" }, { "addr": "172.21.15.42:6835", "nonce": 3427104185, "type": "v1" } ] }, "export_targets": [], "features": 4540138292836696063, "flags": 0, "gid": 11221, "incarnation": 314, "name": "a", "rank": 0, "state": "up:replay", "state_seq": 2 } }, "last_failure": 0, "last_failure_osd_epoch": 106, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 2, "min_compat_client": "0 (unknown)", "modified": "2019-12-05T06:26:39.891183+0000", "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 11221 } } } ], "standbys": [] } 2019-12-05T06:26:44.231 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:26:44.332 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:44.318+0000 7f2d231de700 1 Processor -- start 2019-12-05T06:26:44.332 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:44.318+0000 7f2d231de700 1 -- start start 2019-12-05T06:26:44.332 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:44.318+0000 7f2d231de700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2d1c08df80 0x7f2d1c09f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:26:44.332 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:44.318+0000 7f2d231de700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2d1c09f830 0x7f2d1c14efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:26:44.333 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:44.318+0000 7f2d231de700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d1c09fc60 con 0x7f2d1c0a7740 2019-12-05T06:26:44.333 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:44.318+0000 7f2d231de700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d1c0a0b00 con 0x7f2d1c09f830 2019-12-05T06:26:44.333 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:44.318+0000 7f2d231de700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d1c08e5b0 con 0x7f2d1c08df80 2019-12-05T06:26:44.333 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:44.318+0000 7f2d20f7a700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2d1c08df80 0x7f2d1c09f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:26:44.333 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:44.318+0000 7f2d2177b700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f2d1c0a7740 0x7f2d1c07e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:55324/0 (socket says 172.21.15.42:55324) 2019-12-05T06:26:44.333 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:44.318+0000 7f2d2177b700 1 -- 172.21.15.42:0/2538390601 learned_addr learned my addr 172.21.15.42:0/2538390601 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:26:44.333 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:44.318+0000 7f2d21f7c700 1 --2- 172.21.15.42:0/2538390601 >> v2:172.21.15.40:3300/0 conn(0x7f2d1c09f830 0x7f2d1c14efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:26:44.334 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:44.322+0000 7f2d0ffff700 1 -- 172.21.15.42:0/2538390601 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1688283398 0 0) 0x7f2d1c09fc60 con 0x7f2d1c0a7740 2019-12-05T06:26:44.334 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:44.322+0000 7f2d0ffff700 1 -- 172.21.15.42:0/2538390601 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d080030b0 con 0x7f2d1c0a7740 2019-12-05T06:26:44.334 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:44.322+0000 7f2d20f7a700 1 -- 172.21.15.42:0/2538390601 >> v2:172.21.15.40:3300/0 conn(0x7f2d1c09f830 msgr2=0x7f2d1c14efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:26:44.334 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:44.322+0000 7f2d20f7a700 1 --2- 172.21.15.42:0/2538390601 >> v2:172.21.15.40:3300/0 conn(0x7f2d1c09f830 0x7f2d1c14efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:26:44.334 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:44.322+0000 7f2d20f7a700 1 -- 172.21.15.42:0/2538390601 >> v1:172.21.15.42:6789/0 conn(0x7f2d1c0a7740 legacy=0x7f2d1c07e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:44.334 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:44.322+0000 7f2d20f7a700 1 -- 172.21.15.42:0/2538390601 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d1c0a0710 con 0x7f2d1c08df80 2019-12-05T06:26:44.334 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:44.322+0000 7f2d20f7a700 1 --2- 172.21.15.42:0/2538390601 >> v2:172.21.15.42:3300/0 conn(0x7f2d1c08df80 0x7f2d1c09f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f2d100032b0 tx=0x7f2d1000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:44.335 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:44.322+0000 7f2d20f7a700 1 --2- 172.21.15.42:0/2538390601 >> v2:172.21.15.42:3300/0 conn(0x7f2d1c08df80 0x7f2d1c09f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f2d100032b0 tx=0x7f2d1000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:44.335 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:44.322+0000 7f2d20f7a700 1 --2- 172.21.15.42:0/2538390601 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2d1c08df80 0x7f2d1c09f2e0 secure :-1 s=READY pgs=2661 cs=0 l=1 rx=0x7f2d100032b0 tx=0x7f2d1000fff0).ready entity=mon.1 client_cookie=809aa7a7d0817c60 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:26:44.335 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:44.322+0000 7f2d20f7a700 1 --2- 172.21.15.42:0/2538390601 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2d1c08df80 0x7f2d1c09f2e0 secure :-1 s=THROTTLE_DONE pgs=2661 cs=0 l=1 rx=0x7f2d100032b0 tx=0x7f2d1000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:44.335 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:44.322+0000 7f2d0ffff700 1 -- 172.21.15.42:0/2538390601 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2d100108c0 con 0x7f2d1c08df80 2019-12-05T06:26:44.335 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:44.322+0000 7f2d20f7a700 1 --2- 172.21.15.42:0/2538390601 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2d1c08df80 0x7f2d1c09f2e0 secure :-1 s=THROTTLE_DONE pgs=2661 cs=0 l=1 rx=0x7f2d100032b0 tx=0x7f2d1000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:44.337 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:44.322+0000 7f2d20f7a700 1 --2- 172.21.15.42:0/2538390601 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2d1c08df80 0x7f2d1c09f2e0 secure :-1 s=THROTTLE_DONE pgs=2661 cs=0 l=1 rx=0x7f2d100032b0 tx=0x7f2d1000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:44.337 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:44.322+0000 7f2d0ffff700 1 -- 172.21.15.42:0/2538390601 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d10008b50 con 0x7f2d1c08df80 2019-12-05T06:26:44.337 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:44.322+0000 7f2d0ffff700 1 -- 172.21.15.42:0/2538390601 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2d100110c0 con 0x7f2d1c08df80 2019-12-05T06:26:44.337 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:44.322+0000 7f2d231de700 1 -- 172.21.15.42:0/2538390601 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2d1c08df80 msgr2=0x7f2d1c09f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:44.337 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:44.322+0000 7f2d231de700 1 --2- 172.21.15.42:0/2538390601 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2d1c08df80 0x7f2d1c09f2e0 secure :-1 s=READY pgs=2661 cs=0 l=1 rx=0x7f2d100032b0 tx=0x7f2d1000fff0).stop 2019-12-05T06:26:44.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:44.322+0000 7f2d231de700 1 -- 172.21.15.42:0/2538390601 shutdown_connections 2019-12-05T06:26:44.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:44.322+0000 7f2d231de700 1 --2- 172.21.15.42:0/2538390601 >> v2:172.21.15.40:3300/0 conn(0x7f2d1c09f830 0x7f2d1c14efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:26:44.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:44.322+0000 7f2d231de700 1 --2- 172.21.15.42:0/2538390601 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2d1c08df80 0x7f2d1c09f2e0 unknown :-1 s=CLOSED pgs=2661 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:44.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:44.322+0000 7f2d231de700 1 -- 172.21.15.42:0/2538390601 shutdown_connections 2019-12-05T06:26:44.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:44.322+0000 7f2d231de700 1 -- 172.21.15.42:0/2538390601 wait complete. 2019-12-05T06:26:44.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:44.322+0000 7f2d231de700 1 -- 172.21.15.42:0/2538390601 >> 172.21.15.42:0/2538390601 conn(0x7f2d1c0aabe0 msgr2=0x7f2d1c0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:26:44.341 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:44.326+0000 7f2d231de700 1 Processor -- start 2019-12-05T06:26:44.341 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:44.326+0000 7f2d231de700 1 -- start start 2019-12-05T06:26:44.342 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:44.326+0000 7f2d231de700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2d1c116500 0x7f2d1c11a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:26:44.342 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:44.326+0000 7f2d231de700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2d1c11bcc0 0x7f2d1c11af20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:26:44.342 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:44.326+0000 7f2d231de700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d1c11c4a0 con 0x7f2d1c08eb70 2019-12-05T06:26:44.342 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:44.326+0000 7f2d231de700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d1c11a770 con 0x7f2d1c116500 2019-12-05T06:26:44.342 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:44.326+0000 7f2d231de700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d1c11c200 con 0x7f2d1c11bcc0 2019-12-05T06:26:44.342 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:44.326+0000 7f2d2177b700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2d1c116500 0x7f2d1c11a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:26:44.343 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:44.326+0000 7f2d2177b700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2d1c116500 0x7f2d1c11a9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:55292/0 (socket says 172.21.15.42:55292) 2019-12-05T06:26:44.343 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:44.326+0000 7f2d2177b700 1 -- 172.21.15.42:0/2892605774 learned_addr learned my addr 172.21.15.42:0/2892605774 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:26:44.343 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:44.326+0000 7f2d21f7c700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f2d1c08eb70 0x7f2d1c08efa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:55332/0 (socket says 172.21.15.42:55332) 2019-12-05T06:26:44.343 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:44.326+0000 7f2d20f7a700 1 --2- 172.21.15.42:0/2892605774 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2d1c11bcc0 0x7f2d1c11af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:26:44.343 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:44.326+0000 7f2d2177b700 1 -- 172.21.15.42:0/2892605774 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2d1c11bcc0 msgr2=0x7f2d1c11af20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:44.344 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:44.326+0000 7f2d2177b700 1 --2- 172.21.15.42:0/2892605774 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2d1c11bcc0 0x7f2d1c11af20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:44.344 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:44.326+0000 7f2d2177b700 1 -- 172.21.15.42:0/2892605774 >> v1:172.21.15.42:6789/0 conn(0x7f2d1c08eb70 legacy=0x7f2d1c08efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:44.344 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:44.326+0000 7f2d0dffb700 1 -- 172.21.15.42:0/2892605774 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2771801352 0 0) 0x7f2d14000e90 con 0x7f2d1c08eb70 2019-12-05T06:26:44.345 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:44.326+0000 7f2d2177b700 1 -- 172.21.15.42:0/2892605774 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d1c11b470 con 0x7f2d1c116500 2019-12-05T06:26:44.345 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:44.326+0000 7f2d2177b700 1 --2- 172.21.15.42:0/2892605774 >> v2:172.21.15.42:3300/0 conn(0x7f2d1c116500 0x7f2d1c11a9d0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f2d1c08e500 tx=0x7f2d1800c430).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:44.345 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:44.326+0000 7f2d2177b700 1 --2- 172.21.15.42:0/2892605774 >> v2:172.21.15.42:3300/0 conn(0x7f2d1c116500 0x7f2d1c11a9d0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f2d1c08e500 tx=0x7f2d1800c430).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:44.345 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:44.326+0000 7f2d2177b700 1 --2- 172.21.15.42:0/2892605774 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2d1c116500 0x7f2d1c11a9d0 secure :-1 s=READY pgs=2663 cs=0 l=1 rx=0x7f2d1c08e500 tx=0x7f2d1800c430).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:26:44.345 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:44.326+0000 7f2d2177b700 1 --2- 172.21.15.42:0/2892605774 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2d1c116500 0x7f2d1c11a9d0 secure :-1 s=THROTTLE_DONE pgs=2663 cs=0 l=1 rx=0x7f2d1c08e500 tx=0x7f2d1800c430).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:44.345 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:44.326+0000 7f2d0dffb700 1 -- 172.21.15.42:0/2892605774 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2d1800c9a0 con 0x7f2d1c116500 2019-12-05T06:26:44.345 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:44.326+0000 7f2d2177b700 1 --2- 172.21.15.42:0/2892605774 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2d1c116500 0x7f2d1c11a9d0 secure :-1 s=THROTTLE_DONE pgs=2663 cs=0 l=1 rx=0x7f2d1c08e500 tx=0x7f2d1800c430).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:44.346 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:44.326+0000 7f2d2177b700 1 --2- 172.21.15.42:0/2892605774 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2d1c116500 0x7f2d1c11a9d0 secure :-1 s=THROTTLE_DONE pgs=2663 cs=0 l=1 rx=0x7f2d1c08e500 tx=0x7f2d1800c430).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:44.346 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:44.330+0000 7f2d0dffb700 1 -- 172.21.15.42:0/2892605774 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d18010250 con 0x7f2d1c116500 2019-12-05T06:26:44.346 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:44.330+0000 7f2d231de700 1 -- 172.21.15.42:0/2892605774 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d1c1196c0 con 0x7f2d1c116500 2019-12-05T06:26:44.346 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:44.330+0000 7f2d0dffb700 1 -- 172.21.15.42:0/2892605774 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2d1800d0c0 con 0x7f2d1c116500 2019-12-05T06:26:44.346 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:44.330+0000 7f2d231de700 1 -- 172.21.15.42:0/2892605774 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d1c08f940 con 0x7f2d1c116500 2019-12-05T06:26:44.346 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:44.330+0000 7f2d2177b700 1 --2- 172.21.15.42:0/2892605774 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2d1c116500 0x7f2d1c11a9d0 secure :-1 s=THROTTLE_DONE pgs=2663 cs=0 l=1 rx=0x7f2d1c08e500 tx=0x7f2d1800c430).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:44.346 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:44.330+0000 7f2d0dffb700 1 -- 172.21.15.42:0/2892605774 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f2d18005460 con 0x7f2d1c116500 2019-12-05T06:26:44.347 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:44.330+0000 7f2d2177b700 1 --2- 172.21.15.42:0/2892605774 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2d1c116500 0x7f2d1c11a9d0 secure :-1 s=THROTTLE_DONE pgs=2663 cs=0 l=1 rx=0x7f2d1c08e500 tx=0x7f2d1800c430).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:44.347 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:44.330+0000 7f2d0dffb700 1 --2- 172.21.15.42:0/2892605774 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2cfc03ad20 0x7f2cfc03d230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:26:44.347 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:44.330+0000 7f2d0dffb700 1 -- 172.21.15.42:0/2892605774 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 10933+0+0 (secure 0 0 0) 0x7f2d1804ab40 con 0x7f2d1c116500 2019-12-05T06:26:44.347 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:44.330+0000 7f2d20f7a700 1 --2- 172.21.15.42:0/2892605774 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2cfc03ad20 0x7f2cfc03d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:26:44.347 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:44.330+0000 7f2d20f7a700 1 --2- 172.21.15.42:0/2892605774 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2cfc03ad20 0x7f2cfc03d230 crc :-1 s=READY pgs=1107 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:26:44.347 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:44.334+0000 7f2d231de700 1 -- 172.21.15.42:0/2892605774 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d1c05d110 con 0x7f2d1c116500 2019-12-05T06:26:44.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:44.342+0000 7f2d2177b700 1 --2- 172.21.15.42:0/2892605774 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2d1c116500 0x7f2d1c11a9d0 secure :-1 s=THROTTLE_DONE pgs=2663 cs=0 l=1 rx=0x7f2d1c08e500 tx=0x7f2d1800c430).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:44.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:44.342+0000 7f2d0dffb700 1 -- 172.21.15.42:0/2892605774 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f2d18031bc0 con 0x7f2d1c116500 2019-12-05T06:26:44.515 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:44.506+0000 7f2d231de700 1 -- 172.21.15.42:0/2892605774 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2d1c05b8c0 con 0x7f2d1c116500 2019-12-05T06:26:44.515 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:44.506+0000 7f2d2177b700 1 --2- 172.21.15.42:0/2892605774 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2d1c116500 0x7f2d1c11a9d0 secure :-1 s=THROTTLE_DONE pgs=2663 cs=0 l=1 rx=0x7f2d1c08e500 tx=0x7f2d1800c430).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:44.515 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:44.506+0000 7f2d0dffb700 1 -- 172.21.15.42:0/2892605774 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 315 v315) v1 ==== 95+0+1782 (secure 0 0 0) 0x7f2d18031e20 con 0x7f2d1c116500 2019-12-05T06:26:44.516 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 315 2019-12-05T06:26:44.516 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:26:44.516 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":315,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":315,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:26:44.223326+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":106,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11221},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11221":{"gid":11221,"name":"a","rank":0,"incarnation":314,"state":"up:reconnect","state_seq":32,"addr":"172.21.15.42:6835/3427104185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3427104185},{"type":"v1","addr":"172.21.15.42:6835","nonce":3427104185}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:26:44.518 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:44.510+0000 7f2d231de700 1 -- 172.21.15.42:0/2892605774 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2cfc03ad20 msgr2=0x7f2cfc03d230 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:44.518 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:44.510+0000 7f2d231de700 1 --2- 172.21.15.42:0/2892605774 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2cfc03ad20 0x7f2cfc03d230 crc :-1 s=READY pgs=1107 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:44.518 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:44.510+0000 7f2d231de700 1 -- 172.21.15.42:0/2892605774 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2d1c116500 msgr2=0x7f2d1c11a9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:44.519 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:44.510+0000 7f2d231de700 1 --2- 172.21.15.42:0/2892605774 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2d1c116500 0x7f2d1c11a9d0 secure :-1 s=READY pgs=2663 cs=0 l=1 rx=0x7f2d1c08e500 tx=0x7f2d1800c430).stop 2019-12-05T06:26:44.519 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:44.510+0000 7f2d231de700 1 -- 172.21.15.42:0/2892605774 shutdown_connections 2019-12-05T06:26:44.519 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:44.510+0000 7f2d231de700 1 --2- 172.21.15.42:0/2892605774 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2cfc03ad20 0x7f2cfc03d230 unknown :-1 s=CLOSED pgs=1107 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:44.519 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:44.510+0000 7f2d231de700 1 --2- 172.21.15.42:0/2892605774 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2d1c116500 0x7f2d1c11a9d0 unknown :-1 s=CLOSED pgs=2663 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:44.519 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:44.510+0000 7f2d231de700 1 --2- 172.21.15.42:0/2892605774 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2d1c11bcc0 0x7f2d1c11af20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:44.520 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:44.510+0000 7f2d231de700 1 -- 172.21.15.42:0/2892605774 shutdown_connections 2019-12-05T06:26:44.520 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:44.510+0000 7f2d231de700 1 -- 172.21.15.42:0/2892605774 wait complete. 2019-12-05T06:26:44.520 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:44.510+0000 7f2d231de700 1 -- 172.21.15.42:0/2892605774 >> 172.21.15.42:0/2892605774 conn(0x7f2d1c0a7740 msgr2=0x7f2d1c059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:26:44.527 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2019-12-05T06:26:44.601 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:26:44.613 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:26:46.528 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:26:46.637 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:46.618+0000 7f8e5786a700 1 Processor -- start 2019-12-05T06:26:46.637 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:46.618+0000 7f8e5786a700 1 -- start start 2019-12-05T06:26:46.637 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:46.618+0000 7f8e5786a700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8e500a7740 0x7f8e5008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:26:46.638 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:46.618+0000 7f8e5786a700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8e5009fae0 0x7f8e5007eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:26:46.638 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:46.618+0000 7f8e5786a700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e500a0710 con 0x7f8e5007e720 2019-12-05T06:26:46.638 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:46.618+0000 7f8e5786a700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e500a0b00 con 0x7f8e500a7740 2019-12-05T06:26:46.638 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:46.618+0000 7f8e5786a700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e5008e5b0 con 0x7f8e5009fae0 2019-12-05T06:26:46.638 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:46.622+0000 7f8e56608700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8e5009fae0 0x7f8e5007eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:26:46.639 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:46.622+0000 7f8e55e07700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8e500a7740 0x7f8e5008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:26:46.639 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:46.622+0000 7f8e55606700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f8e5007e720 0x7f8e5009f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:36548/0 (socket says 172.21.15.42:36548) 2019-12-05T06:26:46.639 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:46.622+0000 7f8e55606700 1 -- 172.21.15.42:0/3532733465 learned_addr learned my addr 172.21.15.42:0/3532733465 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:26:46.639 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:46.622+0000 7f8e54e05700 1 -- 172.21.15.42:0/3532733465 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3380813367 0 0) 0x7f8e500a0710 con 0x7f8e5007e720 2019-12-05T06:26:46.639 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:46.622+0000 7f8e54e05700 1 -- 172.21.15.42:0/3532733465 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e3c0030b0 con 0x7f8e5007e720 2019-12-05T06:26:46.639 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:46.622+0000 7f8e56608700 1 -- 172.21.15.42:0/3532733465 >> v2:172.21.15.40:3300/0 conn(0x7f8e500a7740 msgr2=0x7f8e5008df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:26:46.640 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:46.622+0000 7f8e56608700 1 --2- 172.21.15.42:0/3532733465 >> v2:172.21.15.40:3300/0 conn(0x7f8e500a7740 0x7f8e5008df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:26:46.640 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:46.622+0000 7f8e56608700 1 -- 172.21.15.42:0/3532733465 >> v1:172.21.15.40:6789/0 conn(0x7f8e5007e720 legacy=0x7f8e5009f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:46.640 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:46.622+0000 7f8e56608700 1 -- 172.21.15.42:0/3532733465 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e5007f0a0 con 0x7f8e5009fae0 2019-12-05T06:26:46.640 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:46.622+0000 7f8e56608700 1 --2- 172.21.15.42:0/3532733465 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8e5009fae0 0x7f8e5007eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f8e48008b10 tx=0x7f8e4800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:46.640 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:46.622+0000 7f8e56608700 1 --2- 172.21.15.42:0/3532733465 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8e5009fae0 0x7f8e5007eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f8e48008b10 tx=0x7f8e4800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:46.640 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:46.622+0000 7f8e56608700 1 --2- 172.21.15.42:0/3532733465 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8e5009fae0 0x7f8e5007eb50 secure :-1 s=READY pgs=577 cs=0 l=1 rx=0x7f8e48008b10 tx=0x7f8e4800fff0).ready entity=mon.2 client_cookie=341d48008c4fca66 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:26:46.641 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:46.622+0000 7f8e56608700 1 --2- 172.21.15.42:0/3532733465 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8e5009fae0 0x7f8e5007eb50 secure :-1 s=THROTTLE_DONE pgs=577 cs=0 l=1 rx=0x7f8e48008b10 tx=0x7f8e4800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:46.641 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:46.622+0000 7f8e54e05700 1 -- 172.21.15.42:0/3532733465 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8e48010900 con 0x7f8e5009fae0 2019-12-05T06:26:46.641 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:46.622+0000 7f8e56608700 1 --2- 172.21.15.42:0/3532733465 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8e5009fae0 0x7f8e5007eb50 secure :-1 s=THROTTLE_DONE pgs=577 cs=0 l=1 rx=0x7f8e48008b10 tx=0x7f8e4800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:46.641 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:46.622+0000 7f8e56608700 1 --2- 172.21.15.42:0/3532733465 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8e5009fae0 0x7f8e5007eb50 secure :-1 s=THROTTLE_DONE pgs=577 cs=0 l=1 rx=0x7f8e48008b10 tx=0x7f8e4800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:46.641 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:46.622+0000 7f8e54e05700 1 -- 172.21.15.42:0/3532733465 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e48008b50 con 0x7f8e5009fae0 2019-12-05T06:26:46.642 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:46.622+0000 7f8e54e05700 1 -- 172.21.15.42:0/3532733465 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8e480110c0 con 0x7f8e5009fae0 2019-12-05T06:26:46.642 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:46.622+0000 7f8e5786a700 1 -- 172.21.15.42:0/3532733465 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8e5009fae0 msgr2=0x7f8e5007eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:46.642 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:46.622+0000 7f8e5786a700 1 --2- 172.21.15.42:0/3532733465 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8e5009fae0 0x7f8e5007eb50 secure :-1 s=READY pgs=577 cs=0 l=1 rx=0x7f8e48008b10 tx=0x7f8e4800fff0).stop 2019-12-05T06:26:46.642 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:46.622+0000 7f8e5786a700 1 -- 172.21.15.42:0/3532733465 shutdown_connections 2019-12-05T06:26:46.642 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:46.622+0000 7f8e5786a700 1 --2- 172.21.15.42:0/3532733465 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8e5009fae0 0x7f8e5007eb50 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:46.642 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:46.622+0000 7f8e5786a700 1 --2- 172.21.15.42:0/3532733465 >> v2:172.21.15.40:3300/0 conn(0x7f8e500a7740 0x7f8e5008df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:26:46.643 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:46.622+0000 7f8e5786a700 1 -- 172.21.15.42:0/3532733465 shutdown_connections 2019-12-05T06:26:46.643 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:46.622+0000 7f8e5786a700 1 -- 172.21.15.42:0/3532733465 wait complete. 2019-12-05T06:26:46.643 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:46.622+0000 7f8e5786a700 1 -- 172.21.15.42:0/3532733465 >> 172.21.15.42:0/3532733465 conn(0x7f8e500aabe0 msgr2=0x7f8e500a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:26:46.643 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:46.626+0000 7f8e5786a700 1 Processor -- start 2019-12-05T06:26:46.643 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:46.626+0000 7f8e5786a700 1 -- start start 2019-12-05T06:26:46.643 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:46.626+0000 7f8e5786a700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f8e500a6000 0x7f8e50142c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:26:46.644 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:46.626+0000 7f8e5786a700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8e500a4c70 0x7f8e500762f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:26:46.644 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:46.626+0000 7f8e5786a700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e500a50a0 con 0x7f8e500a4840 2019-12-05T06:26:46.644 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:46.626+0000 7f8e5786a700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e500a53b0 con 0x7f8e500a4c70 2019-12-05T06:26:46.644 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:46.626+0000 7f8e5786a700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e501300a0 con 0x7f8e500a6000 2019-12-05T06:26:46.644 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:46.626+0000 7f8e55e07700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f8e500a6000 0x7f8e50142c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:26:46.645 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:46.626+0000 7f8e55606700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f8e500a4840 0x7f8e5012f8a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:55346/0 (socket says 172.21.15.42:55346) 2019-12-05T06:26:46.645 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:46.626+0000 7f8e55606700 1 -- 172.21.15.42:0/3005258001 learned_addr learned my addr 172.21.15.42:0/3005258001 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:26:46.645 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:46.626+0000 7f8e56608700 1 --2- 172.21.15.42:0/3005258001 >> v2:172.21.15.40:3300/0 conn(0x7f8e500a4c70 0x7f8e500762f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:26:46.645 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:46.626+0000 7f8e55e07700 1 -- 172.21.15.42:0/3005258001 >> v2:172.21.15.40:3300/0 conn(0x7f8e500a4c70 msgr2=0x7f8e500762f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:46.645 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:46.626+0000 7f8e55e07700 1 --2- 172.21.15.42:0/3005258001 >> v2:172.21.15.40:3300/0 conn(0x7f8e500a4c70 0x7f8e500762f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:46.645 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:46.626+0000 7f8e55e07700 1 -- 172.21.15.42:0/3005258001 >> v1:172.21.15.42:6789/0 conn(0x7f8e500a4840 legacy=0x7f8e5012f8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:46.646 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:46.626+0000 7f8e55e07700 1 -- 172.21.15.42:0/3005258001 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e50130290 con 0x7f8e500a6000 2019-12-05T06:26:46.646 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:46.626+0000 7f8e55e07700 1 --2- 172.21.15.42:0/3005258001 >> v2:172.21.15.42:3300/0 conn(0x7f8e500a6000 0x7f8e50142c90 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f8e40001150 tx=0x7f8e40001190).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:46.646 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:46.626+0000 7f8e55e07700 1 --2- 172.21.15.42:0/3005258001 >> v2:172.21.15.42:3300/0 conn(0x7f8e500a6000 0x7f8e50142c90 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f8e40001150 tx=0x7f8e40001190).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:46.646 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:46.626+0000 7f8e55e07700 1 --2- 172.21.15.42:0/3005258001 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8e500a6000 0x7f8e50142c90 secure :-1 s=READY pgs=2665 cs=0 l=1 rx=0x7f8e40001150 tx=0x7f8e40001190).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:26:46.646 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:46.626+0000 7f8e55e07700 1 --2- 172.21.15.42:0/3005258001 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8e500a6000 0x7f8e50142c90 secure :-1 s=THROTTLE_DONE pgs=2665 cs=0 l=1 rx=0x7f8e40001150 tx=0x7f8e40001190).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:46.646 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:46.626+0000 7f8e467fc700 1 -- 172.21.15.42:0/3005258001 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8e4c005930 con 0x7f8e500a6000 2019-12-05T06:26:46.647 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:46.626+0000 7f8e55e07700 1 --2- 172.21.15.42:0/3005258001 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8e500a6000 0x7f8e50142c90 secure :-1 s=THROTTLE_DONE pgs=2665 cs=0 l=1 rx=0x7f8e40001150 tx=0x7f8e40001190).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:46.647 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:46.626+0000 7f8e55e07700 1 --2- 172.21.15.42:0/3005258001 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8e500a6000 0x7f8e50142c90 secure :-1 s=THROTTLE_DONE pgs=2665 cs=0 l=1 rx=0x7f8e40001150 tx=0x7f8e40001190).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:46.647 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:46.626+0000 7f8e467fc700 1 -- 172.21.15.42:0/3005258001 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e4c007160 con 0x7f8e500a6000 2019-12-05T06:26:46.647 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:46.626+0000 7f8e5786a700 1 -- 172.21.15.42:0/3005258001 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e50076840 con 0x7f8e500a6000 2019-12-05T06:26:46.647 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:46.626+0000 7f8e467fc700 1 -- 172.21.15.42:0/3005258001 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8e4c0060c0 con 0x7f8e500a6000 2019-12-05T06:26:46.647 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:46.626+0000 7f8e5786a700 1 -- 172.21.15.42:0/3005258001 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e50076d90 con 0x7f8e500a6000 2019-12-05T06:26:46.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:46.626+0000 7f8e55e07700 1 --2- 172.21.15.42:0/3005258001 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8e500a6000 0x7f8e50142c90 secure :-1 s=THROTTLE_DONE pgs=2665 cs=0 l=1 rx=0x7f8e40001150 tx=0x7f8e40001190).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:46.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:46.630+0000 7f8e467fc700 1 -- 172.21.15.42:0/3005258001 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f8e4c0063d0 con 0x7f8e500a6000 2019-12-05T06:26:46.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:46.630+0000 7f8e55e07700 1 --2- 172.21.15.42:0/3005258001 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8e500a6000 0x7f8e50142c90 secure :-1 s=THROTTLE_DONE pgs=2665 cs=0 l=1 rx=0x7f8e40001150 tx=0x7f8e40001190).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:46.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:46.630+0000 7f8e467fc700 1 --2- 172.21.15.42:0/3005258001 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8e3803acf0 0x7f8e3803d200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:26:46.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:46.630+0000 7f8e467fc700 1 -- 172.21.15.42:0/3005258001 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 10933+0+0 (secure 0 0 0) 0x7f8e4c04adc0 con 0x7f8e500a6000 2019-12-05T06:26:46.650 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:46.630+0000 7f8e55606700 1 --2- 172.21.15.42:0/3005258001 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8e3803acf0 0x7f8e3803d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:26:46.650 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:46.630+0000 7f8e55606700 1 --2- 172.21.15.42:0/3005258001 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8e3803acf0 0x7f8e3803d200 crc :-1 s=READY pgs=1108 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:26:46.650 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:46.634+0000 7f8e5786a700 1 -- 172.21.15.42:0/3005258001 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e5005d110 con 0x7f8e500a6000 2019-12-05T06:26:46.650 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:46.638+0000 7f8e55e07700 1 --2- 172.21.15.42:0/3005258001 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8e500a6000 0x7f8e50142c90 secure :-1 s=THROTTLE_DONE pgs=2665 cs=0 l=1 rx=0x7f8e40001150 tx=0x7f8e40001190).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:46.650 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:46.638+0000 7f8e467fc700 1 -- 172.21.15.42:0/3005258001 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f8e4c031e00 con 0x7f8e500a6000 2019-12-05T06:26:46.820 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:46.810+0000 7f8e5786a700 1 -- 172.21.15.42:0/3005258001 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8e500a6950 con 0x7f8e500a6000 2019-12-05T06:26:46.821 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:46.814+0000 7f8e55e07700 1 --2- 172.21.15.42:0/3005258001 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8e500a6000 0x7f8e50142c90 secure :-1 s=THROTTLE_DONE pgs=2665 cs=0 l=1 rx=0x7f8e40001150 tx=0x7f8e40001190).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:46.821 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:46.814+0000 7f8e467fc700 1 -- 172.21.15.42:0/3005258001 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 317 v317) v1 ==== 95+0+1779 (secure 0 0 0) 0x7f8e4c032060 con 0x7f8e500a6000 2019-12-05T06:26:46.821 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 317 2019-12-05T06:26:46.821 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:26:46.822 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":317,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":317,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:26:46.246866+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":106,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11221},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11221":{"gid":11221,"name":"a","rank":0,"incarnation":314,"state":"up:active","state_seq":34,"addr":"172.21.15.42:6835/3427104185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3427104185},{"type":"v1","addr":"172.21.15.42:6835","nonce":3427104185}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:26:46.824 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:46.814+0000 7f8e5786a700 1 -- 172.21.15.42:0/3005258001 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8e3803acf0 msgr2=0x7f8e3803d200 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:46.824 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:46.814+0000 7f8e5786a700 1 --2- 172.21.15.42:0/3005258001 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8e3803acf0 0x7f8e3803d200 crc :-1 s=READY pgs=1108 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:46.824 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:46.814+0000 7f8e5786a700 1 -- 172.21.15.42:0/3005258001 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8e500a6000 msgr2=0x7f8e50142c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:46.824 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:46.814+0000 7f8e5786a700 1 --2- 172.21.15.42:0/3005258001 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8e500a6000 0x7f8e50142c90 secure :-1 s=READY pgs=2665 cs=0 l=1 rx=0x7f8e40001150 tx=0x7f8e40001190).stop 2019-12-05T06:26:46.824 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:46.814+0000 7f8e5786a700 1 -- 172.21.15.42:0/3005258001 shutdown_connections 2019-12-05T06:26:46.824 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:46.814+0000 7f8e5786a700 1 --2- 172.21.15.42:0/3005258001 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8e3803acf0 0x7f8e3803d200 unknown :-1 s=CLOSED pgs=1108 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:46.825 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:46.814+0000 7f8e5786a700 1 --2- 172.21.15.42:0/3005258001 >> v2:172.21.15.40:3300/0 conn(0x7f8e500a4c70 0x7f8e500762f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:46.825 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:46.814+0000 7f8e5786a700 1 --2- 172.21.15.42:0/3005258001 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8e500a6000 0x7f8e50142c90 unknown :-1 s=CLOSED pgs=2665 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:46.825 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:46.814+0000 7f8e5786a700 1 -- 172.21.15.42:0/3005258001 shutdown_connections 2019-12-05T06:26:46.830 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:46.814+0000 7f8e5786a700 1 -- 172.21.15.42:0/3005258001 wait complete. 2019-12-05T06:26:46.830 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:46.814+0000 7f8e5786a700 1 -- 172.21.15.42:0/3005258001 >> 172.21.15.42:0/3005258001 conn(0x7f8e500aabe0 msgr2=0x7f8e50059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:26:46.838 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a has gained rank=0, replacing gid=7568 2019-12-05T06:26:46.838 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 2 secs before reviving mds.b 2019-12-05T06:26:48.839 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.b 2019-12-05T06:26:48.839 INFO:tasks.ceph.mds.b:Restarting daemon 2019-12-05T06:26:48.839 INFO:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2019-12-05T06:26:48.843 INFO:tasks.ceph.mds.b:Started 2019-12-05T06:26:48.843 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:26:48.903 INFO:tasks.ceph.mds.b.smithi040.stdout:starting mds.b at 2019-12-05T06:26:48.971 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:48.938+0000 7f2f5b68d700 1 Processor -- start 2019-12-05T06:26:48.971 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:48.938+0000 7f2f5b68d700 1 -- start start 2019-12-05T06:26:48.972 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:48.942+0000 7f2f5b68d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2f540a7740 0x7f2f5408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:26:48.972 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:48.942+0000 7f2f5b68d700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f540a0710 con 0x7f2f5407e720 2019-12-05T06:26:48.972 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:48.942+0000 7f2f5b68d700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f540a9ba0 con 0x7f2f5409fae0 2019-12-05T06:26:48.988 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:48.942+0000 7f2f5b68d700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f540a0b00 con 0x7f2f540a7740 2019-12-05T06:26:48.988 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:48.942+0000 7f2f59c2a700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2f540a7740 0x7f2f5408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:26:48.989 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:48.942+0000 7f2f59c2a700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2f540a7740 0x7f2f5408df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:55328/0 (socket says 172.21.15.42:55328) 2019-12-05T06:26:48.989 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:48.942+0000 7f2f59c2a700 1 -- 172.21.15.42:0/659799903 learned_addr learned my addr 172.21.15.42:0/659799903 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:26:48.989 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:48.942+0000 7f2f59429700 1 -- 172.21.15.42:0/659799903 >> v1:172.21.15.40:6789/0 conn(0x7f2f5407e720 legacy=0x7f2f5409f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2019-12-05T06:26:48.989 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:48.942+0000 7f2f59429700 1 -- 172.21.15.42:0/659799903 >> v1:172.21.15.40:6789/0 conn(0x7f2f5407e720 legacy=0x7f2f5409f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2019-12-05T06:26:48.990 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:48.942+0000 7f2f59429700 1 --1- 172.21.15.42:0/659799903 >> v1:172.21.15.40:6789/0 conn(0x7f2f5407e720 0x7f2f5409f2e0 :-1 s=OPENED pgs=1984 cs=1 l=1).handle_message read tag failed 2019-12-05T06:26:48.990 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:48.942+0000 7f2f59429700 1 --1- 172.21.15.42:0/659799903 >> v1:172.21.15.40:6789/0 conn(0x7f2f5407e720 0x7f2f5409f2e0 :-1 s=OPENED pgs=1984 cs=1 l=1).fault on lossy channel, failing 2019-12-05T06:26:48.990 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:48.942+0000 7f2f58c28700 1 -- 172.21.15.42:0/659799903 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1269457258 0 0) 0x7f2f540a9ba0 con 0x7f2f5409fae0 2019-12-05T06:26:48.990 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:48.942+0000 7f2f59c2a700 1 -- 172.21.15.42:0/659799903 >> v1:172.21.15.42:6789/0 conn(0x7f2f5409fae0 legacy=0x7f2f5414efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:48.990 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:48.942+0000 7f2f59c2a700 1 -- 172.21.15.42:0/659799903 >> v1:172.21.15.40:6789/0 conn(0x7f2f5407e720 legacy=0x7f2f5409f2e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2019-12-05T06:26:48.991 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:48.942+0000 7f2f59c2a700 1 -- 172.21.15.42:0/659799903 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f5407eb50 con 0x7f2f540a7740 2019-12-05T06:26:48.991 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:48.942+0000 7f2f59c2a700 1 --2- 172.21.15.42:0/659799903 >> v2:172.21.15.42:3300/0 conn(0x7f2f540a7740 0x7f2f5408df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f2f50002f30 tx=0x7f2f5000fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:48.991 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:48.942+0000 7f2f59c2a700 1 --2- 172.21.15.42:0/659799903 >> v2:172.21.15.42:3300/0 conn(0x7f2f540a7740 0x7f2f5408df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f2f50002f30 tx=0x7f2f5000fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:48.991 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:48.942+0000 7f2f59c2a700 1 --2- 172.21.15.42:0/659799903 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2f540a7740 0x7f2f5408df80 secure :-1 s=READY pgs=2668 cs=0 l=1 rx=0x7f2f50002f30 tx=0x7f2f5000fcb0).ready entity=mon.1 client_cookie=e502a0af0affbca1 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:26:48.991 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:48.942+0000 7f2f59c2a700 1 --2- 172.21.15.42:0/659799903 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2f540a7740 0x7f2f5408df80 secure :-1 s=THROTTLE_DONE pgs=2668 cs=0 l=1 rx=0x7f2f50002f30 tx=0x7f2f5000fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:48.991 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:48.942+0000 7f2f58c28700 1 -- 172.21.15.42:0/659799903 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2f500114e0 con 0x7f2f540a7740 2019-12-05T06:26:48.991 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:48.942+0000 7f2f59c2a700 1 --2- 172.21.15.42:0/659799903 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2f540a7740 0x7f2f5408df80 secure :-1 s=THROTTLE_DONE pgs=2668 cs=0 l=1 rx=0x7f2f50002f30 tx=0x7f2f5000fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:48.992 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:48.942+0000 7f2f59c2a700 1 --2- 172.21.15.42:0/659799903 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2f540a7740 0x7f2f5408df80 secure :-1 s=THROTTLE_DONE pgs=2668 cs=0 l=1 rx=0x7f2f50002f30 tx=0x7f2f5000fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:48.992 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:48.942+0000 7f2f58c28700 1 -- 172.21.15.42:0/659799903 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f50008ca0 con 0x7f2f540a7740 2019-12-05T06:26:48.992 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:48.942+0000 7f2f58c28700 1 -- 172.21.15.42:0/659799903 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2f50011ac0 con 0x7f2f540a7740 2019-12-05T06:26:48.992 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:48.942+0000 7f2f5b68d700 1 -- 172.21.15.42:0/659799903 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2f540a7740 msgr2=0x7f2f5408df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:48.992 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:48.942+0000 7f2f5b68d700 1 --2- 172.21.15.42:0/659799903 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2f540a7740 0x7f2f5408df80 secure :-1 s=READY pgs=2668 cs=0 l=1 rx=0x7f2f50002f30 tx=0x7f2f5000fcb0).stop 2019-12-05T06:26:48.992 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:48.942+0000 7f2f5b68d700 1 -- 172.21.15.42:0/659799903 shutdown_connections 2019-12-05T06:26:48.992 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:48.942+0000 7f2f5b68d700 1 --2- 172.21.15.42:0/659799903 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2f540a7740 0x7f2f5408df80 unknown :-1 s=CLOSED pgs=2668 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:48.993 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:48.946+0000 7f2f5b68d700 1 -- 172.21.15.42:0/659799903 shutdown_connections 2019-12-05T06:26:48.993 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:48.946+0000 7f2f5b68d700 1 -- 172.21.15.42:0/659799903 wait complete. 2019-12-05T06:26:48.993 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:48.946+0000 7f2f5b68d700 1 -- 172.21.15.42:0/659799903 >> 172.21.15.42:0/659799903 conn(0x7f2f540aabe0 msgr2=0x7f2f540a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:26:48.993 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:48.946+0000 7f2f5b68d700 1 Processor -- start 2019-12-05T06:26:48.993 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:48.946+0000 7f2f5b68d700 1 -- start start 2019-12-05T06:26:48.993 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:48.946+0000 7f2f5b68d700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2f5407df60 0x7f2f54121e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:26:48.994 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:48.946+0000 7f2f5b68d700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2f54115750 0x7f2f541436a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:26:48.994 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:48.946+0000 7f2f5b68d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2f541223a0 0x7f2f54142530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:26:48.994 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:48.946+0000 7f2f5b68d700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f54125950 con 0x7f2f54115750 2019-12-05T06:26:48.994 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:48.946+0000 7f2f5b68d700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f54125bf0 con 0x7f2f541223a0 2019-12-05T06:26:48.994 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:48.946+0000 7f2f5b68d700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f54125e90 con 0x7f2f5407df60 2019-12-05T06:26:48.994 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:48.946+0000 7f2f5a42b700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2f541223a0 0x7f2f54142530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:26:48.995 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:48.946+0000 7f2f5a42b700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2f541223a0 0x7f2f54142530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:55336/0 (socket says 172.21.15.42:55336) 2019-12-05T06:26:48.995 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:48.946+0000 7f2f5a42b700 1 -- 172.21.15.42:0/4150489922 learned_addr learned my addr 172.21.15.42:0/4150489922 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:26:48.995 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:48.946+0000 7f2f59c2a700 1 --2- 172.21.15.42:0/4150489922 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2f5407df60 0x7f2f54121e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:26:48.995 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:48.946+0000 7f2f59429700 1 --2- 172.21.15.42:0/4150489922 >> v2:172.21.15.40:3300/0 conn(0x7f2f54115750 0x7f2f541436a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:26:48.995 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:48.950+0000 7f2f5a42b700 1 -- 172.21.15.42:0/4150489922 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2f5407df60 msgr2=0x7f2f54121e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:48.995 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:48.950+0000 7f2f5a42b700 1 --2- 172.21.15.42:0/4150489922 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2f5407df60 0x7f2f54121e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:48.996 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:48.950+0000 7f2f5a42b700 1 -- 172.21.15.42:0/4150489922 >> v2:172.21.15.40:3300/0 conn(0x7f2f54115750 msgr2=0x7f2f541436a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:48.996 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:48.950+0000 7f2f5a42b700 1 --2- 172.21.15.42:0/4150489922 >> v2:172.21.15.40:3300/0 conn(0x7f2f54115750 0x7f2f541436a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:48.996 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:48.950+0000 7f2f5a42b700 1 -- 172.21.15.42:0/4150489922 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f54126130 con 0x7f2f541223a0 2019-12-05T06:26:48.996 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:48.950+0000 7f2f5a42b700 1 --2- 172.21.15.42:0/4150489922 >> v2:172.21.15.42:3300/0 conn(0x7f2f541223a0 0x7f2f54142530 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f2f54125db0 tx=0x7f2f4c010bd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:48.996 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:48.950+0000 7f2f5a42b700 1 --2- 172.21.15.42:0/4150489922 >> v2:172.21.15.42:3300/0 conn(0x7f2f541223a0 0x7f2f54142530 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f2f54125db0 tx=0x7f2f4c010bd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:48.996 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:48.950+0000 7f2f5a42b700 1 --2- 172.21.15.42:0/4150489922 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2f541223a0 0x7f2f54142530 secure :-1 s=READY pgs=2669 cs=0 l=1 rx=0x7f2f54125db0 tx=0x7f2f4c010bd0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:26:48.997 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:48.950+0000 7f2f5a42b700 1 --2- 172.21.15.42:0/4150489922 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2f541223a0 0x7f2f54142530 secure :-1 s=THROTTLE_DONE pgs=2669 cs=0 l=1 rx=0x7f2f54125db0 tx=0x7f2f4c010bd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:48.997 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:48.950+0000 7f2f5a42b700 1 --2- 172.21.15.42:0/4150489922 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2f541223a0 0x7f2f54142530 secure :-1 s=THROTTLE_DONE pgs=2669 cs=0 l=1 rx=0x7f2f54125db0 tx=0x7f2f4c010bd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:48.997 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:48.950+0000 7f2f4a7fc700 1 -- 172.21.15.42:0/4150489922 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2f4c006dc0 con 0x7f2f541223a0 2019-12-05T06:26:48.997 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:48.950+0000 7f2f5a42b700 1 --2- 172.21.15.42:0/4150489922 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2f541223a0 0x7f2f54142530 secure :-1 s=THROTTLE_DONE pgs=2669 cs=0 l=1 rx=0x7f2f54125db0 tx=0x7f2f4c010bd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:48.997 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:48.950+0000 7f2f4a7fc700 1 -- 172.21.15.42:0/4150489922 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f4c0069e0 con 0x7f2f541223a0 2019-12-05T06:26:48.997 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:48.950+0000 7f2f5b68d700 1 -- 172.21.15.42:0/4150489922 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f54143bf0 con 0x7f2f541223a0 2019-12-05T06:26:48.997 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:48.950+0000 7f2f4a7fc700 1 -- 172.21.15.42:0/4150489922 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2f4c012630 con 0x7f2f541223a0 2019-12-05T06:26:48.998 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:48.950+0000 7f2f5b68d700 1 -- 172.21.15.42:0/4150489922 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f54125630 con 0x7f2f541223a0 2019-12-05T06:26:48.998 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:48.950+0000 7f2f5a42b700 1 --2- 172.21.15.42:0/4150489922 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2f541223a0 0x7f2f54142530 secure :-1 s=THROTTLE_DONE pgs=2669 cs=0 l=1 rx=0x7f2f54125db0 tx=0x7f2f4c010bd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:48.998 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:48.950+0000 7f2f4a7fc700 1 -- 172.21.15.42:0/4150489922 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f2f4c012940 con 0x7f2f541223a0 2019-12-05T06:26:48.999 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:48.950+0000 7f2f5a42b700 1 --2- 172.21.15.42:0/4150489922 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2f541223a0 0x7f2f54142530 secure :-1 s=THROTTLE_DONE pgs=2669 cs=0 l=1 rx=0x7f2f54125db0 tx=0x7f2f4c010bd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:48.999 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:48.950+0000 7f2f4a7fc700 1 --2- 172.21.15.42:0/4150489922 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2f3c03add0 0x7f2f3c03d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:26:48.999 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:48.950+0000 7f2f4a7fc700 1 -- 172.21.15.42:0/4150489922 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 10933+0+0 (secure 0 0 0) 0x7f2f4c04b5e0 con 0x7f2f541223a0 2019-12-05T06:26:48.999 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:48.950+0000 7f2f59c2a700 1 --2- 172.21.15.42:0/4150489922 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2f3c03add0 0x7f2f3c03d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:26:48.999 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:48.950+0000 7f2f59c2a700 1 --2- 172.21.15.42:0/4150489922 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2f3c03add0 0x7f2f3c03d2e0 crc :-1 s=READY pgs=1109 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:26:48.999 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:48.954+0000 7f2f5b68d700 1 -- 172.21.15.42:0/4150489922 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f5405d110 con 0x7f2f541223a0 2019-12-05T06:26:49.000 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:48.962+0000 7f2f5a42b700 1 --2- 172.21.15.42:0/4150489922 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2f541223a0 0x7f2f54142530 secure :-1 s=THROTTLE_DONE pgs=2669 cs=0 l=1 rx=0x7f2f54125db0 tx=0x7f2f4c010bd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:49.000 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:48.962+0000 7f2f4a7fc700 1 -- 172.21.15.42:0/4150489922 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f2f4c032660 con 0x7f2f541223a0 2019-12-05T06:26:49.131 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:49.122+0000 7f2f5b68d700 1 -- 172.21.15.42:0/4150489922 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2f54115bd0 con 0x7f2f541223a0 2019-12-05T06:26:49.132 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:49.126+0000 7f2f5a42b700 1 --2- 172.21.15.42:0/4150489922 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2f541223a0 0x7f2f54142530 secure :-1 s=THROTTLE_DONE pgs=2669 cs=0 l=1 rx=0x7f2f54125db0 tx=0x7f2f4c010bd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:49.133 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:49.126+0000 7f2f4a7fc700 1 -- 172.21.15.42:0/4150489922 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 317 v317) v1 ==== 95+0+1779 (secure 0 0 0) 0x7f2f4c0328c0 con 0x7f2f541223a0 2019-12-05T06:26:49.133 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 317 2019-12-05T06:26:49.133 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:26:49.133 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":317,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":317,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:26:46.246866+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":106,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11221},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11221":{"gid":11221,"name":"a","rank":0,"incarnation":314,"state":"up:active","state_seq":34,"addr":"172.21.15.42:6835/3427104185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3427104185},{"type":"v1","addr":"172.21.15.42:6835","nonce":3427104185}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:26:49.136 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:49.126+0000 7f2f5b68d700 1 -- 172.21.15.42:0/4150489922 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2f3c03add0 msgr2=0x7f2f3c03d2e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:49.136 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:49.126+0000 7f2f5b68d700 1 --2- 172.21.15.42:0/4150489922 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2f3c03add0 0x7f2f3c03d2e0 crc :-1 s=READY pgs=1109 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:49.137 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:49.126+0000 7f2f5b68d700 1 -- 172.21.15.42:0/4150489922 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2f541223a0 msgr2=0x7f2f54142530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:49.137 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:49.126+0000 7f2f5b68d700 1 --2- 172.21.15.42:0/4150489922 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2f541223a0 0x7f2f54142530 secure :-1 s=READY pgs=2669 cs=0 l=1 rx=0x7f2f54125db0 tx=0x7f2f4c010bd0).stop 2019-12-05T06:26:49.137 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:49.126+0000 7f2f5b68d700 1 -- 172.21.15.42:0/4150489922 shutdown_connections 2019-12-05T06:26:49.137 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:49.126+0000 7f2f5b68d700 1 --2- 172.21.15.42:0/4150489922 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2f3c03add0 0x7f2f3c03d2e0 unknown :-1 s=CLOSED pgs=1109 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:49.138 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:49.126+0000 7f2f5b68d700 1 --2- 172.21.15.42:0/4150489922 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2f541223a0 0x7f2f54142530 unknown :-1 s=CLOSED pgs=2669 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:49.138 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:49.126+0000 7f2f5b68d700 1 --2- 172.21.15.42:0/4150489922 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2f5407df60 0x7f2f54121e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:49.138 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:49.126+0000 7f2f5b68d700 1 --2- 172.21.15.42:0/4150489922 >> v2:172.21.15.40:3300/0 conn(0x7f2f54115750 0x7f2f541436a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:49.138 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:49.126+0000 7f2f5b68d700 1 -- 172.21.15.42:0/4150489922 shutdown_connections 2019-12-05T06:26:49.138 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:49.126+0000 7f2f5b68d700 1 -- 172.21.15.42:0/4150489922 wait complete. 2019-12-05T06:26:49.139 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:49.126+0000 7f2f5b68d700 1 -- 172.21.15.42:0/4150489922 >> 172.21.15.42:0/4150489922 conn(0x7f2f5408fde0 msgr2=0x7f2f540a4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:26:49.144 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2019-12-05T06:26:51.146 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:26:51.250 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:51.242+0000 7f9f3d63f700 1 Processor -- start 2019-12-05T06:26:51.250 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:51.242+0000 7f9f3d63f700 1 -- start start 2019-12-05T06:26:51.250 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:51.242+0000 7f9f3d63f700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f9f380a7740 0x7f9f3808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:26:51.250 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:51.242+0000 7f9f3d63f700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f9f3807e720 0x7f9f3807eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:26:51.250 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:51.242+0000 7f9f3d63f700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f380a0710 con 0x7f9f3809f2e0 2019-12-05T06:26:51.250 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:51.242+0000 7f9f3d63f700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f380a0b00 con 0x7f9f3807e720 2019-12-05T06:26:51.251 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:51.242+0000 7f9f3d63f700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f3808e5b0 con 0x7f9f380a7740 2019-12-05T06:26:51.251 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:51.242+0000 7f9f377fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f9f380a7740 0x7f9f3808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:26:51.251 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:51.242+0000 7f9f377fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f9f380a7740 0x7f9f3808df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:55344/0 (socket says 172.21.15.42:55344) 2019-12-05T06:26:51.251 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:51.242+0000 7f9f377fe700 1 -- 172.21.15.42:0/818977558 learned_addr learned my addr 172.21.15.42:0/818977558 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:26:51.251 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:51.242+0000 7f9f2ffff700 1 --2- 172.21.15.42:0/818977558 >> v2:172.21.15.40:3300/0 conn(0x7f9f3807e720 0x7f9f3807eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:26:51.251 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:51.242+0000 7f9f377fe700 1 -- 172.21.15.42:0/818977558 >> v2:172.21.15.40:3300/0 conn(0x7f9f3807e720 msgr2=0x7f9f3807eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:26:51.251 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:51.242+0000 7f9f377fe700 1 --2- 172.21.15.42:0/818977558 >> v2:172.21.15.40:3300/0 conn(0x7f9f3807e720 0x7f9f3807eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:26:51.252 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:51.242+0000 7f9f377fe700 1 -- 172.21.15.42:0/818977558 >> v1:172.21.15.42:6789/0 conn(0x7f9f3809f2e0 legacy=0x7f9f3809f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:51.252 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:51.242+0000 7f9f377fe700 1 -- 172.21.15.42:0/818977558 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f3807f0a0 con 0x7f9f380a7740 2019-12-05T06:26:51.252 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:51.242+0000 7f9f377fe700 1 --2- 172.21.15.42:0/818977558 >> v2:172.21.15.42:3300/0 conn(0x7f9f380a7740 0x7f9f3808df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f9f28002f30 tx=0x7f9f2800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:51.252 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:51.242+0000 7f9f377fe700 1 --2- 172.21.15.42:0/818977558 >> v2:172.21.15.42:3300/0 conn(0x7f9f380a7740 0x7f9f3808df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f9f28002f30 tx=0x7f9f2800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:51.252 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:51.242+0000 7f9f377fe700 1 --2- 172.21.15.42:0/818977558 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9f380a7740 0x7f9f3808df80 secure :-1 s=READY pgs=2671 cs=0 l=1 rx=0x7f9f28002f30 tx=0x7f9f2800fd30).ready entity=mon.1 client_cookie=e34273f9b291edde server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:26:51.252 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:51.242+0000 7f9f377fe700 1 --2- 172.21.15.42:0/818977558 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9f380a7740 0x7f9f3808df80 secure :-1 s=THROTTLE_DONE pgs=2671 cs=0 l=1 rx=0x7f9f28002f30 tx=0x7f9f2800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:51.253 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:51.242+0000 7f9f377fe700 1 --2- 172.21.15.42:0/818977558 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9f380a7740 0x7f9f3808df80 secure :-1 s=THROTTLE_DONE pgs=2671 cs=0 l=1 rx=0x7f9f28002f30 tx=0x7f9f2800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:51.253 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:51.242+0000 7f9f377fe700 1 --2- 172.21.15.42:0/818977558 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9f380a7740 0x7f9f3808df80 secure :-1 s=THROTTLE_DONE pgs=2671 cs=0 l=1 rx=0x7f9f28002f30 tx=0x7f9f2800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:51.253 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:51.242+0000 7f9f36ffd700 1 -- 172.21.15.42:0/818977558 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9f28011370 con 0x7f9f380a7740 2019-12-05T06:26:51.253 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:51.242+0000 7f9f36ffd700 1 -- 172.21.15.42:0/818977558 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f28008bb0 con 0x7f9f380a7740 2019-12-05T06:26:51.253 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:51.242+0000 7f9f36ffd700 1 -- 172.21.15.42:0/818977558 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9f28011950 con 0x7f9f380a7740 2019-12-05T06:26:51.253 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:51.242+0000 7f9f3d63f700 1 -- 172.21.15.42:0/818977558 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9f380a7740 msgr2=0x7f9f3808df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:51.253 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:51.242+0000 7f9f3d63f700 1 --2- 172.21.15.42:0/818977558 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9f380a7740 0x7f9f3808df80 secure :-1 s=READY pgs=2671 cs=0 l=1 rx=0x7f9f28002f30 tx=0x7f9f2800fd30).stop 2019-12-05T06:26:51.254 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:51.246+0000 7f9f3d63f700 1 -- 172.21.15.42:0/818977558 shutdown_connections 2019-12-05T06:26:51.254 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:51.246+0000 7f9f3d63f700 1 --2- 172.21.15.42:0/818977558 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9f380a7740 0x7f9f3808df80 unknown :-1 s=CLOSED pgs=2671 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:51.254 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:51.246+0000 7f9f3d63f700 1 --2- 172.21.15.42:0/818977558 >> v2:172.21.15.40:3300/0 conn(0x7f9f3807e720 0x7f9f3807eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:26:51.254 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:51.246+0000 7f9f3d63f700 1 -- 172.21.15.42:0/818977558 shutdown_connections 2019-12-05T06:26:51.254 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:51.246+0000 7f9f3d63f700 1 -- 172.21.15.42:0/818977558 wait complete. 2019-12-05T06:26:51.254 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:51.246+0000 7f9f3d63f700 1 -- 172.21.15.42:0/818977558 >> 172.21.15.42:0/818977558 conn(0x7f9f380aabe0 msgr2=0x7f9f380a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:26:51.257 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:51.246+0000 7f9f3d63f700 1 Processor -- start 2019-12-05T06:26:51.257 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:51.246+0000 7f9f3d63f700 1 -- start start 2019-12-05T06:26:51.257 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:51.250+0000 7f9f3d63f700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f9f38123690 0x7f9f38116ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:26:51.257 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:51.250+0000 7f9f3d63f700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f9f38132260 0x7f9f38138a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:26:51.257 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:51.250+0000 7f9f3d63f700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f38121000 con 0x7f9f38131e30 2019-12-05T06:26:51.257 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:51.250+0000 7f9f3d63f700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f38119a20 con 0x7f9f38123690 2019-12-05T06:26:51.258 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:51.250+0000 7f9f3d63f700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f38119cc0 con 0x7f9f38132260 2019-12-05T06:26:51.258 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:51.250+0000 7f9f37fff700 1 -- >> v2:172.21.15.42:3300/0 conn(0x7f9f38132260 msgr2=0x7f9f38138a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2019-12-05T06:26:51.258 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:51.250+0000 7f9f37fff700 1 -- >> v2:172.21.15.42:3300/0 conn(0x7f9f38132260 msgr2=0x7f9f38138a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2019-12-05T06:26:51.258 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:51.250+0000 7f9f37fff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f9f38132260 0x7f9f38138a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner read peer banner failed r=-1 ((1) Operation not permitted) 2019-12-05T06:26:51.258 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:51.250+0000 7f9f377fe700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f9f38123690 0x7f9f38116ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:26:51.258 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:51.250+0000 7f9f2ffff700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f9f38131e30 0x7f9f38118320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:36598/0 (socket says 172.21.15.42:36598) 2019-12-05T06:26:51.259 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:51.250+0000 7f9f37fff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f9f38132260 0x7f9f38138a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rx=0 tx=0)._fault waiting 0.200000 2019-12-05T06:26:51.259 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:51.250+0000 7f9f2ffff700 1 -- 172.21.15.42:0/1728863724 learned_addr learned my addr 172.21.15.42:0/1728863724 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:26:51.259 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:51.250+0000 7f9f377fe700 1 -- 172.21.15.42:0/1728863724 >> v2:172.21.15.42:3300/0 conn(0x7f9f38132260 msgr2=0x7f9f38138a70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2019-12-05T06:26:51.259 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:51.250+0000 7f9f377fe700 1 --2- 172.21.15.42:0/1728863724 >> v2:172.21.15.42:3300/0 conn(0x7f9f38132260 0x7f9f38138a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:51.259 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:51.250+0000 7f9f377fe700 1 -- 172.21.15.42:0/1728863724 >> v1:172.21.15.40:6789/0 conn(0x7f9f38131e30 legacy=0x7f9f38118320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:51.259 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:51.250+0000 7f9f377fe700 1 -- 172.21.15.42:0/1728863724 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f38121900 con 0x7f9f38123690 2019-12-05T06:26:51.259 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:51.250+0000 7f9f34ff9700 1 -- 172.21.15.42:0/1728863724 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 853690919 0 0) 0x7f9f38121000 con 0x7f9f38131e30 2019-12-05T06:26:51.260 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:51.250+0000 7f9f377fe700 1 --2- 172.21.15.42:0/1728863724 >> v2:172.21.15.40:3300/0 conn(0x7f9f38123690 0x7f9f38116ed0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f9f38117ec0 tx=0x7f9f28011f10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:51.260 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:51.250+0000 7f9f377fe700 1 --2- 172.21.15.42:0/1728863724 >> v2:172.21.15.40:3300/0 conn(0x7f9f38123690 0x7f9f38116ed0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f9f38117ec0 tx=0x7f9f28011f10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:51.260 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:51.250+0000 7f9f377fe700 1 --2- 172.21.15.42:0/1728863724 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9f38123690 0x7f9f38116ed0 secure :-1 s=READY pgs=1986 cs=0 l=1 rx=0x7f9f38117ec0 tx=0x7f9f28011f10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:26:51.260 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:51.250+0000 7f9f377fe700 1 --2- 172.21.15.42:0/1728863724 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9f38123690 0x7f9f38116ed0 secure :-1 s=THROTTLE_DONE pgs=1986 cs=0 l=1 rx=0x7f9f38117ec0 tx=0x7f9f28011f10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:51.260 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:51.250+0000 7f9f34ff9700 1 -- 172.21.15.42:0/1728863724 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9f28007340 con 0x7f9f38123690 2019-12-05T06:26:51.260 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:51.250+0000 7f9f377fe700 1 --2- 172.21.15.42:0/1728863724 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9f38123690 0x7f9f38116ed0 secure :-1 s=THROTTLE_DONE pgs=1986 cs=0 l=1 rx=0x7f9f38117ec0 tx=0x7f9f28011f10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:51.260 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:51.250+0000 7f9f377fe700 1 --2- 172.21.15.42:0/1728863724 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9f38123690 0x7f9f38116ed0 secure :-1 s=THROTTLE_DONE pgs=1986 cs=0 l=1 rx=0x7f9f38117ec0 tx=0x7f9f28011f10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:51.261 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:51.250+0000 7f9f34ff9700 1 -- 172.21.15.42:0/1728863724 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f28011570 con 0x7f9f38123690 2019-12-05T06:26:51.261 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:51.250+0000 7f9f3d63f700 1 -- 172.21.15.42:0/1728863724 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f38118b20 con 0x7f9f38123690 2019-12-05T06:26:51.261 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:51.250+0000 7f9f34ff9700 1 -- 172.21.15.42:0/1728863724 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9f28005540 con 0x7f9f38123690 2019-12-05T06:26:51.261 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:51.250+0000 7f9f3d63f700 1 -- 172.21.15.42:0/1728863724 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f38139160 con 0x7f9f38123690 2019-12-05T06:26:51.261 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:51.254+0000 7f9f377fe700 1 --2- 172.21.15.42:0/1728863724 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9f38123690 0x7f9f38116ed0 secure :-1 s=THROTTLE_DONE pgs=1986 cs=0 l=1 rx=0x7f9f38117ec0 tx=0x7f9f28011f10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:51.262 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:51.254+0000 7f9f377fe700 1 --2- 172.21.15.42:0/1728863724 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9f38123690 0x7f9f38116ed0 secure :-1 s=THROTTLE_DONE pgs=1986 cs=0 l=1 rx=0x7f9f38117ec0 tx=0x7f9f28011f10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:51.262 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:51.254+0000 7f9f34ff9700 1 -- 172.21.15.42:0/1728863724 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f9f280057e0 con 0x7f9f38123690 2019-12-05T06:26:51.262 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:51.254+0000 7f9f34ff9700 1 --2- 172.21.15.42:0/1728863724 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9f1c03ac70 0x7f9f1c03d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:26:51.262 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:51.254+0000 7f9f34ff9700 1 -- 172.21.15.42:0/1728863724 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 10933+0+0 (secure 0 0 0) 0x7f9f2804b370 con 0x7f9f38123690 2019-12-05T06:26:51.263 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:51.254+0000 7f9f2ffff700 1 --2- 172.21.15.42:0/1728863724 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9f1c03ac70 0x7f9f1c03d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:26:51.263 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:51.254+0000 7f9f2ffff700 1 --2- 172.21.15.42:0/1728863724 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9f1c03ac70 0x7f9f1c03d180 crc :-1 s=READY pgs=1111 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:26:51.263 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:51.254+0000 7f9f3d63f700 1 -- 172.21.15.42:0/1728863724 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f3805d110 con 0x7f9f38123690 2019-12-05T06:26:51.270 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:51.266+0000 7f9f377fe700 1 --2- 172.21.15.42:0/1728863724 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9f38123690 0x7f9f38116ed0 secure :-1 s=THROTTLE_DONE pgs=1986 cs=0 l=1 rx=0x7f9f38117ec0 tx=0x7f9f28011f10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:51.270 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:51.266+0000 7f9f34ff9700 1 -- 172.21.15.42:0/1728863724 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f9f28032330 con 0x7f9f38123690 2019-12-05T06:26:51.435 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:51.430+0000 7f9f3d63f700 1 -- 172.21.15.42:0/1728863724 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9f3805b8c0 con 0x7f9f38123690 2019-12-05T06:26:51.436 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:51.430+0000 7f9f377fe700 1 --2- 172.21.15.42:0/1728863724 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9f38123690 0x7f9f38116ed0 secure :-1 s=THROTTLE_DONE pgs=1986 cs=0 l=1 rx=0x7f9f38117ec0 tx=0x7f9f28011f10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:51.436 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:51.430+0000 7f9f34ff9700 1 -- 172.21.15.42:0/1728863724 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 317 v317) v1 ==== 95+0+1779 (secure 0 0 0) 0x7f9f28032590 con 0x7f9f38123690 2019-12-05T06:26:51.438 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 317 2019-12-05T06:26:51.438 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:26:51.438 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":317,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":317,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:26:46.246866+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":106,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11221},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11221":{"gid":11221,"name":"a","rank":0,"incarnation":314,"state":"up:active","state_seq":34,"addr":"172.21.15.42:6835/3427104185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3427104185},{"type":"v1","addr":"172.21.15.42:6835","nonce":3427104185}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:26:51.441 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:51.438+0000 7f9f3d63f700 1 -- 172.21.15.42:0/1728863724 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9f1c03ac70 msgr2=0x7f9f1c03d180 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:51.441 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:51.438+0000 7f9f3d63f700 1 --2- 172.21.15.42:0/1728863724 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9f1c03ac70 0x7f9f1c03d180 crc :-1 s=READY pgs=1111 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:51.441 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:51.438+0000 7f9f3d63f700 1 -- 172.21.15.42:0/1728863724 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9f38123690 msgr2=0x7f9f38116ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:51.442 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:51.438+0000 7f9f3d63f700 1 --2- 172.21.15.42:0/1728863724 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9f38123690 0x7f9f38116ed0 secure :-1 s=READY pgs=1986 cs=0 l=1 rx=0x7f9f38117ec0 tx=0x7f9f28011f10).stop 2019-12-05T06:26:51.442 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:51.438+0000 7f9f3d63f700 1 -- 172.21.15.42:0/1728863724 shutdown_connections 2019-12-05T06:26:51.442 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:51.438+0000 7f9f3d63f700 1 --2- 172.21.15.42:0/1728863724 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9f1c03ac70 0x7f9f1c03d180 unknown :-1 s=CLOSED pgs=1111 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:51.442 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:51.438+0000 7f9f3d63f700 1 --2- 172.21.15.42:0/1728863724 >> v2:172.21.15.42:3300/0 conn(0x7f9f38132260 0x7f9f38138a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:51.442 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:51.438+0000 7f9f3d63f700 1 --2- 172.21.15.42:0/1728863724 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9f38123690 0x7f9f38116ed0 unknown :-1 s=CLOSED pgs=1986 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:51.442 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:51.438+0000 7f9f3d63f700 1 -- 172.21.15.42:0/1728863724 shutdown_connections 2019-12-05T06:26:51.442 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:51.438+0000 7f9f3d63f700 1 -- 172.21.15.42:0/1728863724 wait complete. 2019-12-05T06:26:51.443 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:51.438+0000 7f9f3d63f700 1 -- 172.21.15.42:0/1728863724 >> 172.21.15.42:0/1728863724 conn(0x7f9f380a97e0 msgr2=0x7f9f3809f580 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:26:51.451 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2019-12-05T06:26:53.452 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:26:53.564 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:53.550+0000 7f69590d7700 1 Processor -- start 2019-12-05T06:26:53.564 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:53.550+0000 7f69590d7700 1 -- start start 2019-12-05T06:26:53.565 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:53.550+0000 7f69590d7700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f69540a7740 0x7f695408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:26:53.565 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:53.550+0000 7f69590d7700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f695407e720 0x7f695407eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:26:53.565 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:53.550+0000 7f69590d7700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69540a0710 con 0x7f695409f2e0 2019-12-05T06:26:53.565 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:53.550+0000 7f69590d7700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69540a0b00 con 0x7f69540a7740 2019-12-05T06:26:53.566 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:53.550+0000 7f69590d7700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f695408e5b0 con 0x7f695407e720 2019-12-05T06:26:53.566 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:53.550+0000 7f6952d9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f695407e720 0x7f695407eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:26:53.566 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:53.550+0000 7f6952d9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f695407e720 0x7f695407eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:55362/0 (socket says 172.21.15.42:55362) 2019-12-05T06:26:53.567 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:53.550+0000 7f6952d9d700 1 -- 172.21.15.42:0/565107858 learned_addr learned my addr 172.21.15.42:0/565107858 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:26:53.567 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:53.550+0000 7f695359e700 1 --2- 172.21.15.42:0/565107858 >> v2:172.21.15.40:3300/0 conn(0x7f69540a7740 0x7f695408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:26:53.567 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:53.550+0000 7f6952d9d700 1 -- 172.21.15.42:0/565107858 >> v2:172.21.15.40:3300/0 conn(0x7f69540a7740 msgr2=0x7f695408df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:26:53.567 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:53.550+0000 7f6952d9d700 1 --2- 172.21.15.42:0/565107858 >> v2:172.21.15.40:3300/0 conn(0x7f69540a7740 0x7f695408df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:26:53.567 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:53.550+0000 7f6952d9d700 1 -- 172.21.15.42:0/565107858 >> v1:172.21.15.40:6789/0 conn(0x7f695409f2e0 legacy=0x7f695409f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:53.568 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:53.550+0000 7f6952d9d700 1 -- 172.21.15.42:0/565107858 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f695407f0a0 con 0x7f695407e720 2019-12-05T06:26:53.568 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:53.550+0000 7f6952d9d700 1 --2- 172.21.15.42:0/565107858 >> v2:172.21.15.42:3300/0 conn(0x7f695407e720 0x7f695407eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f69440032b0 tx=0x7f69440100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:53.568 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:53.550+0000 7f6952d9d700 1 --2- 172.21.15.42:0/565107858 >> v2:172.21.15.42:3300/0 conn(0x7f695407e720 0x7f695407eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f69440032b0 tx=0x7f69440100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:53.568 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:53.550+0000 7f6952d9d700 1 --2- 172.21.15.42:0/565107858 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f695407e720 0x7f695407eb50 secure :-1 s=READY pgs=2672 cs=0 l=1 rx=0x7f69440032b0 tx=0x7f69440100b0).ready entity=mon.1 client_cookie=deca1257cdcfca74 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:26:53.568 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:53.550+0000 7f6952d9d700 1 --2- 172.21.15.42:0/565107858 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f695407e720 0x7f695407eb50 secure :-1 s=THROTTLE_DONE pgs=2672 cs=0 l=1 rx=0x7f69440032b0 tx=0x7f69440100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:53.568 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:53.550+0000 7f695259c700 1 -- 172.21.15.42:0/565107858 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6944011dd0 con 0x7f695407e720 2019-12-05T06:26:53.569 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:53.550+0000 7f6952d9d700 1 --2- 172.21.15.42:0/565107858 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f695407e720 0x7f695407eb50 secure :-1 s=THROTTLE_DONE pgs=2672 cs=0 l=1 rx=0x7f69440032b0 tx=0x7f69440100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:53.569 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:53.550+0000 7f695259c700 1 -- 172.21.15.42:0/565107858 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69440119f0 con 0x7f695407e720 2019-12-05T06:26:53.569 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:53.550+0000 7f6952d9d700 1 --2- 172.21.15.42:0/565107858 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f695407e720 0x7f695407eb50 secure :-1 s=THROTTLE_DONE pgs=2672 cs=0 l=1 rx=0x7f69440032b0 tx=0x7f69440100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:53.569 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:53.550+0000 7f69590d7700 1 -- 172.21.15.42:0/565107858 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f695407e720 msgr2=0x7f695407eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:53.569 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:53.550+0000 7f695259c700 1 -- 172.21.15.42:0/565107858 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f694400a540 con 0x7f695407e720 2019-12-05T06:26:53.569 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:53.550+0000 7f69590d7700 1 --2- 172.21.15.42:0/565107858 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f695407e720 0x7f695407eb50 secure :-1 s=READY pgs=2672 cs=0 l=1 rx=0x7f69440032b0 tx=0x7f69440100b0).stop 2019-12-05T06:26:53.569 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:53.554+0000 7f69590d7700 1 -- 172.21.15.42:0/565107858 shutdown_connections 2019-12-05T06:26:53.570 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:53.554+0000 7f69590d7700 1 --2- 172.21.15.42:0/565107858 >> v2:172.21.15.40:3300/0 conn(0x7f69540a7740 0x7f695408df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:26:53.570 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:53.554+0000 7f69590d7700 1 --2- 172.21.15.42:0/565107858 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f695407e720 0x7f695407eb50 unknown :-1 s=CLOSED pgs=2672 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:53.570 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:53.554+0000 7f69590d7700 1 -- 172.21.15.42:0/565107858 shutdown_connections 2019-12-05T06:26:53.570 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:53.554+0000 7f69590d7700 1 -- 172.21.15.42:0/565107858 wait complete. 2019-12-05T06:26:53.570 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:53.554+0000 7f69590d7700 1 -- 172.21.15.42:0/565107858 >> 172.21.15.42:0/565107858 conn(0x7f69540aabe0 msgr2=0x7f69540a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:26:53.570 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:53.554+0000 7f69590d7700 1 Processor -- start 2019-12-05T06:26:53.571 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:53.554+0000 7f69590d7700 1 -- start start 2019-12-05T06:26:53.571 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:53.554+0000 7f69590d7700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f695412ed90 0x7f695408ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:26:53.571 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:53.554+0000 7f69590d7700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f69540a4c50 0x7f69540742d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:26:53.571 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:53.554+0000 7f69590d7700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69540a56e0 con 0x7f69540a68b0 2019-12-05T06:26:53.572 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:53.554+0000 7f69590d7700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69540a5440 con 0x7f69540a4c50 2019-12-05T06:26:53.572 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:53.554+0000 7f69590d7700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69540a5080 con 0x7f695412ed90 2019-12-05T06:26:53.572 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:53.554+0000 7f69588d6700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f69540a4c50 0x7f69540742d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:26:53.572 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:53.554+0000 7f69588d6700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f69540a4c50 0x7f69540742d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:55372/0 (socket says 172.21.15.42:55372) 2019-12-05T06:26:53.572 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:53.554+0000 7f69588d6700 1 -- 172.21.15.42:0/2927943124 learned_addr learned my addr 172.21.15.42:0/2927943124 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:26:53.572 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:53.554+0000 7f6952d9d700 1 --2- 172.21.15.42:0/2927943124 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f695412ed90 0x7f695408ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:26:53.572 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:53.554+0000 7f694bfff700 1 -- 172.21.15.42:0/2927943124 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1826577095 0 0) 0x7f69540a56e0 con 0x7f69540a68b0 2019-12-05T06:26:53.573 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:53.554+0000 7f694bfff700 1 -- 172.21.15.42:0/2927943124 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f692c0030b0 con 0x7f69540a68b0 2019-12-05T06:26:53.573 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:53.558+0000 7f69588d6700 1 -- 172.21.15.42:0/2927943124 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f695412ed90 msgr2=0x7f695408ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:53.573 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:53.558+0000 7f69588d6700 1 --2- 172.21.15.42:0/2927943124 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f695412ed90 0x7f695408ecf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:53.573 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:53.558+0000 7f69588d6700 1 -- 172.21.15.42:0/2927943124 >> v1:172.21.15.40:6789/0 conn(0x7f69540a68b0 legacy=0x7f695412e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:53.573 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:53.558+0000 7f69588d6700 1 -- 172.21.15.42:0/2927943124 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f695413f060 con 0x7f69540a4c50 2019-12-05T06:26:53.573 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:53.558+0000 7f69588d6700 1 --2- 172.21.15.42:0/2927943124 >> v2:172.21.15.42:3300/0 conn(0x7f69540a4c50 0x7f69540742d0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f694c000e90 tx=0x7f694c0107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:53.574 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:53.558+0000 7f69588d6700 1 --2- 172.21.15.42:0/2927943124 >> v2:172.21.15.42:3300/0 conn(0x7f69540a4c50 0x7f69540742d0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f694c000e90 tx=0x7f694c0107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:53.574 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:53.558+0000 7f69588d6700 1 --2- 172.21.15.42:0/2927943124 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f69540a4c50 0x7f69540742d0 secure :-1 s=READY pgs=2673 cs=0 l=1 rx=0x7f694c000e90 tx=0x7f694c0107c0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:26:53.574 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:53.558+0000 7f69588d6700 1 --2- 172.21.15.42:0/2927943124 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f69540a4c50 0x7f69540742d0 secure :-1 s=THROTTLE_DONE pgs=2673 cs=0 l=1 rx=0x7f694c000e90 tx=0x7f694c0107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:53.574 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:53.558+0000 7f694bfff700 1 -- 172.21.15.42:0/2927943124 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f694c006a40 con 0x7f69540a4c50 2019-12-05T06:26:53.574 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:53.558+0000 7f69588d6700 1 --2- 172.21.15.42:0/2927943124 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f69540a4c50 0x7f69540742d0 secure :-1 s=THROTTLE_DONE pgs=2673 cs=0 l=1 rx=0x7f694c000e90 tx=0x7f694c0107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:53.575 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:53.558+0000 7f69588d6700 1 --2- 172.21.15.42:0/2927943124 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f69540a4c50 0x7f69540742d0 secure :-1 s=THROTTLE_DONE pgs=2673 cs=0 l=1 rx=0x7f694c000e90 tx=0x7f694c0107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:53.575 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:53.558+0000 7f694bfff700 1 -- 172.21.15.42:0/2927943124 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f694c006660 con 0x7f69540a4c50 2019-12-05T06:26:53.575 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:53.558+0000 7f694bfff700 1 -- 172.21.15.42:0/2927943124 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f694c011000 con 0x7f69540a4c50 2019-12-05T06:26:53.575 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:53.558+0000 7f69590d7700 1 -- 172.21.15.42:0/2927943124 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6954074820 con 0x7f69540a4c50 2019-12-05T06:26:53.575 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:53.558+0000 7f69590d7700 1 -- 172.21.15.42:0/2927943124 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6954074dd0 con 0x7f69540a4c50 2019-12-05T06:26:53.575 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:53.558+0000 7f69588d6700 1 --2- 172.21.15.42:0/2927943124 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f69540a4c50 0x7f69540742d0 secure :-1 s=THROTTLE_DONE pgs=2673 cs=0 l=1 rx=0x7f694c000e90 tx=0x7f694c0107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:53.575 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:53.558+0000 7f694bfff700 1 -- 172.21.15.42:0/2927943124 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f694c0112d0 con 0x7f69540a4c50 2019-12-05T06:26:53.576 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:53.558+0000 7f69588d6700 1 --2- 172.21.15.42:0/2927943124 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f69540a4c50 0x7f69540742d0 secure :-1 s=THROTTLE_DONE pgs=2673 cs=0 l=1 rx=0x7f694c000e90 tx=0x7f694c0107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:53.576 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:53.558+0000 7f694bfff700 1 --2- 172.21.15.42:0/2927943124 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f692c03b2c0 0x7f692c03d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:26:53.576 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:53.558+0000 7f694bfff700 1 -- 172.21.15.42:0/2927943124 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 10933+0+0 (secure 0 0 0) 0x7f694c04b010 con 0x7f69540a4c50 2019-12-05T06:26:53.576 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:53.558+0000 7f695359e700 1 --2- 172.21.15.42:0/2927943124 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f692c03b2c0 0x7f692c03d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:26:53.576 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:53.558+0000 7f695359e700 1 --2- 172.21.15.42:0/2927943124 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f692c03b2c0 0x7f692c03d7d0 crc :-1 s=READY pgs=1114 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:26:53.577 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:53.566+0000 7f69590d7700 1 -- 172.21.15.42:0/2927943124 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f695405d110 con 0x7f69540a4c50 2019-12-05T06:26:53.578 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:53.574+0000 7f69588d6700 1 --2- 172.21.15.42:0/2927943124 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f69540a4c50 0x7f69540742d0 secure :-1 s=THROTTLE_DONE pgs=2673 cs=0 l=1 rx=0x7f694c000e90 tx=0x7f694c0107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:53.578 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:53.574+0000 7f694bfff700 1 -- 172.21.15.42:0/2927943124 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f694c032090 con 0x7f69540a4c50 2019-12-05T06:26:53.745 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:53.742+0000 7f69590d7700 1 -- 172.21.15.42:0/2927943124 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f695405b8c0 con 0x7f69540a4c50 2019-12-05T06:26:53.745 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:53.742+0000 7f69588d6700 1 --2- 172.21.15.42:0/2927943124 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f69540a4c50 0x7f69540742d0 secure :-1 s=THROTTLE_DONE pgs=2673 cs=0 l=1 rx=0x7f694c000e90 tx=0x7f694c0107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:53.746 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:53.742+0000 7f694bfff700 1 -- 172.21.15.42:0/2927943124 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 318 v318) v1 ==== 95+0+2117 (secure 0 0 0) 0x7f694c0322f0 con 0x7f69540a4c50 2019-12-05T06:26:53.748 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 318 2019-12-05T06:26:53.748 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:26:53.748 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":318,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":11142,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/3155851600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3155851600},{"type":"v1","addr":"172.21.15.40:6833","nonce":3155851600}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":318}],"filesystems":[{"mdsmap":{"epoch":317,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:26:46.246866+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":106,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11221},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11221":{"gid":11221,"name":"a","rank":0,"incarnation":314,"state":"up:active","state_seq":34,"addr":"172.21.15.42:6835/3427104185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3427104185},{"type":"v1","addr":"172.21.15.42:6835","nonce":3427104185}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:26:53.751 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:53.746+0000 7f69590d7700 1 -- 172.21.15.42:0/2927943124 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f692c03b2c0 msgr2=0x7f692c03d7d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:53.751 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:53.746+0000 7f69590d7700 1 --2- 172.21.15.42:0/2927943124 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f692c03b2c0 0x7f692c03d7d0 crc :-1 s=READY pgs=1114 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:53.751 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:53.746+0000 7f69590d7700 1 -- 172.21.15.42:0/2927943124 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f69540a4c50 msgr2=0x7f69540742d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:53.751 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:53.746+0000 7f69590d7700 1 --2- 172.21.15.42:0/2927943124 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f69540a4c50 0x7f69540742d0 secure :-1 s=READY pgs=2673 cs=0 l=1 rx=0x7f694c000e90 tx=0x7f694c0107c0).stop 2019-12-05T06:26:53.751 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:53.746+0000 7f69590d7700 1 -- 172.21.15.42:0/2927943124 shutdown_connections 2019-12-05T06:26:53.752 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:53.746+0000 7f69590d7700 1 --2- 172.21.15.42:0/2927943124 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f692c03b2c0 0x7f692c03d7d0 unknown :-1 s=CLOSED pgs=1114 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:53.752 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:53.746+0000 7f69590d7700 1 --2- 172.21.15.42:0/2927943124 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f69540a4c50 0x7f69540742d0 unknown :-1 s=CLOSED pgs=2673 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:53.752 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:53.746+0000 7f69590d7700 1 --2- 172.21.15.42:0/2927943124 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f695412ed90 0x7f695408ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:53.752 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:53.746+0000 7f69590d7700 1 -- 172.21.15.42:0/2927943124 shutdown_connections 2019-12-05T06:26:53.752 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:53.746+0000 7f69590d7700 1 -- 172.21.15.42:0/2927943124 wait complete. 2019-12-05T06:26:53.752 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:26:53.746+0000 7f69590d7700 1 -- 172.21.15.42:0/2927943124 >> 172.21.15.42:0/2927943124 conn(0x7f69540a8110 msgr2=0x7f69540a97e0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:26:53.761 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b reported in up:standby state 2019-12-05T06:26:53.761 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 76 secs before thrashing 2019-12-05T06:27:14.675 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:27:14.679 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:27:44.741 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:27:44.749 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:28:09.762 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:28:09.877 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:09.866+0000 7f87c1db1700 1 Processor -- start 2019-12-05T06:28:09.877 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:09.866+0000 7f87c1db1700 1 -- start start 2019-12-05T06:28:09.877 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:09.870+0000 7f87c1db1700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f87bc0a7740 0x7f87bc08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:28:09.878 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:09.870+0000 7f87c1db1700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f87bc09fae0 0x7f87bc07eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:28:09.878 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:09.870+0000 7f87c1db1700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87bc0a0710 con 0x7f87bc07e720 2019-12-05T06:28:09.878 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:09.870+0000 7f87c1db1700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87bc0a0b00 con 0x7f87bc0a7740 2019-12-05T06:28:09.878 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:09.870+0000 7f87c1db1700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87bc08e5b0 con 0x7f87bc09fae0 2019-12-05T06:28:09.878 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:09.870+0000 7f87bbfff700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f87bc0a7740 0x7f87bc08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:28:09.878 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:09.870+0000 7f87c0b4f700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f87bc09fae0 0x7f87bc07eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:28:09.879 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:09.870+0000 7f87bbfff700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f87bc0a7740 0x7f87bc08df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.42:43480/0 (socket says 172.21.15.42:43480) 2019-12-05T06:28:09.879 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:09.870+0000 7f87c0b4f700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f87bc09fae0 0x7f87bc07eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:56134/0 (socket says 172.21.15.42:56134) 2019-12-05T06:28:09.879 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:09.870+0000 7f87bbfff700 1 -- 172.21.15.42:0/2858443203 learned_addr learned my addr 172.21.15.42:0/2858443203 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:28:09.879 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:09.870+0000 7f87bb7fe700 1 --1- 172.21.15.42:0/2858443203 >> v1:172.21.15.40:6789/0 conn(0x7f87bc07e720 0x7f87bc09f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:36912/0 (socket says 172.21.15.42:36912) 2019-12-05T06:28:09.879 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:09.870+0000 7f87c0b4f700 1 -- 172.21.15.42:0/2858443203 >> v2:172.21.15.40:3300/0 conn(0x7f87bc0a7740 msgr2=0x7f87bc08df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:28:09.879 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:09.870+0000 7f87c0b4f700 1 --2- 172.21.15.42:0/2858443203 >> v2:172.21.15.40:3300/0 conn(0x7f87bc0a7740 0x7f87bc08df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:28:09.879 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:09.870+0000 7f87c0b4f700 1 -- 172.21.15.42:0/2858443203 >> v1:172.21.15.40:6789/0 conn(0x7f87bc07e720 legacy=0x7f87bc09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:09.880 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:09.870+0000 7f87c0b4f700 1 -- 172.21.15.42:0/2858443203 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87bc07f0a0 con 0x7f87bc09fae0 2019-12-05T06:28:09.880 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:09.870+0000 7f87c0b4f700 1 --2- 172.21.15.42:0/2858443203 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f87bc09fae0 0x7f87bc07eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f87b4008b10 tx=0x7f87b400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:09.880 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:09.870+0000 7f87c0b4f700 1 --2- 172.21.15.42:0/2858443203 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f87bc09fae0 0x7f87bc07eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f87b4008b10 tx=0x7f87b400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:09.880 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:09.870+0000 7f87c0b4f700 1 --2- 172.21.15.42:0/2858443203 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f87bc09fae0 0x7f87bc07eb50 secure :-1 s=READY pgs=586 cs=0 l=1 rx=0x7f87b4008b10 tx=0x7f87b400fff0).ready entity=mon.2 client_cookie=689161c39f0f0084 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:28:09.880 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:09.870+0000 7f87c0b4f700 1 --2- 172.21.15.42:0/2858443203 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f87bc09fae0 0x7f87bc07eb50 secure :-1 s=THROTTLE_DONE pgs=586 cs=0 l=1 rx=0x7f87b4008b10 tx=0x7f87b400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:09.880 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:09.870+0000 7f87c0b4f700 1 --2- 172.21.15.42:0/2858443203 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f87bc09fae0 0x7f87bc07eb50 secure :-1 s=THROTTLE_DONE pgs=586 cs=0 l=1 rx=0x7f87b4008b10 tx=0x7f87b400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:09.881 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:09.870+0000 7f87c0b4f700 1 --2- 172.21.15.42:0/2858443203 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f87bc09fae0 0x7f87bc07eb50 secure :-1 s=THROTTLE_DONE pgs=586 cs=0 l=1 rx=0x7f87b4008b10 tx=0x7f87b400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:09.881 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:09.870+0000 7f87baffd700 1 -- 172.21.15.42:0/2858443203 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f87b4010900 con 0x7f87bc09fae0 2019-12-05T06:28:09.881 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:09.870+0000 7f87baffd700 1 -- 172.21.15.42:0/2858443203 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87b4008b50 con 0x7f87bc09fae0 2019-12-05T06:28:09.881 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:09.870+0000 7f87baffd700 1 -- 172.21.15.42:0/2858443203 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f87b40110c0 con 0x7f87bc09fae0 2019-12-05T06:28:09.881 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:09.870+0000 7f87c1db1700 1 -- 172.21.15.42:0/2858443203 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f87bc09fae0 msgr2=0x7f87bc07eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:09.881 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:09.870+0000 7f87c1db1700 1 --2- 172.21.15.42:0/2858443203 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f87bc09fae0 0x7f87bc07eb50 secure :-1 s=READY pgs=586 cs=0 l=1 rx=0x7f87b4008b10 tx=0x7f87b400fff0).stop 2019-12-05T06:28:09.881 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:09.874+0000 7f87c1db1700 1 -- 172.21.15.42:0/2858443203 shutdown_connections 2019-12-05T06:28:09.882 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:09.874+0000 7f87c1db1700 1 --2- 172.21.15.42:0/2858443203 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f87bc09fae0 0x7f87bc07eb50 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:09.882 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:09.874+0000 7f87c1db1700 1 --2- 172.21.15.42:0/2858443203 >> v2:172.21.15.40:3300/0 conn(0x7f87bc0a7740 0x7f87bc08df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:28:09.882 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:09.874+0000 7f87c1db1700 1 -- 172.21.15.42:0/2858443203 shutdown_connections 2019-12-05T06:28:09.882 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:09.874+0000 7f87c1db1700 1 -- 172.21.15.42:0/2858443203 wait complete. 2019-12-05T06:28:09.882 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:09.874+0000 7f87c1db1700 1 -- 172.21.15.42:0/2858443203 >> 172.21.15.42:0/2858443203 conn(0x7f87bc0aabe0 msgr2=0x7f87bc0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:28:09.886 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:09.874+0000 7f87c1db1700 1 Processor -- start 2019-12-05T06:28:09.886 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:09.874+0000 7f87c1db1700 1 -- start start 2019-12-05T06:28:09.886 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:09.878+0000 7f87c1db1700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f87bc0a4c60 0x7f87bc118a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:28:09.886 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:09.878+0000 7f87c1db1700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87bc11c5a0 con 0x7f87bc11be20 2019-12-05T06:28:09.886 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:09.878+0000 7f87c1db1700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87bc11c250 con 0x7f87bc117940 2019-12-05T06:28:09.887 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:09.878+0000 7f87c1db1700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87bc117400 con 0x7f87bc0a4c60 2019-12-05T06:28:09.887 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:09.878+0000 7f87bbfff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f87bc0a4c60 0x7f87bc118a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:28:09.887 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:09.878+0000 7f87c0b4f700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f87bc117940 0x7f87bc08f370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:55710/0 (socket says 172.21.15.42:55710) 2019-12-05T06:28:09.887 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:09.878+0000 7f87c0b4f700 1 -- 172.21.15.42:0/2292136487 learned_addr learned my addr 172.21.15.42:0/2292136487 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:28:09.887 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:09.878+0000 7f87b8ff9700 1 -- 172.21.15.42:0/2292136487 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 547112308 0 0) 0x7f87bc11c5a0 con 0x7f87bc11be20 2019-12-05T06:28:09.887 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:09.878+0000 7f87b8ff9700 1 -- 172.21.15.42:0/2292136487 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87a00030b0 con 0x7f87bc11be20 2019-12-05T06:28:09.888 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:09.878+0000 7f87bbfff700 1 -- 172.21.15.42:0/2292136487 >> v1:172.21.15.42:6789/0 conn(0x7f87bc117940 legacy=0x7f87bc08f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:09.888 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:09.878+0000 7f87b8ff9700 1 -- 172.21.15.42:0/2292136487 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3251196899 0 0) 0x7f87bc11c250 con 0x7f87bc117940 2019-12-05T06:28:09.888 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:09.878+0000 7f87bbfff700 1 -- 172.21.15.42:0/2292136487 >> v1:172.21.15.40:6789/0 conn(0x7f87bc11be20 legacy=0x7f87bc08eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:09.888 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:09.878+0000 7f87bbfff700 1 -- 172.21.15.42:0/2292136487 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87bc117d70 con 0x7f87bc0a4c60 2019-12-05T06:28:09.888 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:09.878+0000 7f87bbfff700 1 --2- 172.21.15.42:0/2292136487 >> v2:172.21.15.42:3300/0 conn(0x7f87bc0a4c60 0x7f87bc118a80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f87a4002f30 tx=0x7f87a4010a00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:09.888 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:09.878+0000 7f87bbfff700 1 --2- 172.21.15.42:0/2292136487 >> v2:172.21.15.42:3300/0 conn(0x7f87bc0a4c60 0x7f87bc118a80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f87a4002f30 tx=0x7f87a4010a00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:09.888 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:09.878+0000 7f87bbfff700 1 --2- 172.21.15.42:0/2292136487 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f87bc0a4c60 0x7f87bc118a80 secure :-1 s=READY pgs=2675 cs=0 l=1 rx=0x7f87a4002f30 tx=0x7f87a4010a00).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:28:09.889 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:09.878+0000 7f87bbfff700 1 --2- 172.21.15.42:0/2292136487 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f87bc0a4c60 0x7f87bc118a80 secure :-1 s=THROTTLE_DONE pgs=2675 cs=0 l=1 rx=0x7f87a4002f30 tx=0x7f87a4010a00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:09.889 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:09.878+0000 7f87b8ff9700 1 -- 172.21.15.42:0/2292136487 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f87a4008c90 con 0x7f87bc0a4c60 2019-12-05T06:28:09.889 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:09.878+0000 7f87bbfff700 1 --2- 172.21.15.42:0/2292136487 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f87bc0a4c60 0x7f87bc118a80 secure :-1 s=THROTTLE_DONE pgs=2675 cs=0 l=1 rx=0x7f87a4002f30 tx=0x7f87a4010a00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:09.889 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:09.878+0000 7f87bbfff700 1 --2- 172.21.15.42:0/2292136487 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f87bc0a4c60 0x7f87bc118a80 secure :-1 s=THROTTLE_DONE pgs=2675 cs=0 l=1 rx=0x7f87a4002f30 tx=0x7f87a4010a00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:09.889 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:09.878+0000 7f87b8ff9700 1 -- 172.21.15.42:0/2292136487 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87a40088b0 con 0x7f87bc0a4c60 2019-12-05T06:28:09.889 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:09.878+0000 7f87b8ff9700 1 -- 172.21.15.42:0/2292136487 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f87a4005920 con 0x7f87bc0a4c60 2019-12-05T06:28:09.890 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:09.878+0000 7f87c1db1700 1 -- 172.21.15.42:0/2292136487 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87bc0a5d40 con 0x7f87bc0a4c60 2019-12-05T06:28:09.890 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:09.878+0000 7f87c1db1700 1 -- 172.21.15.42:0/2292136487 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87bc078540 con 0x7f87bc0a4c60 2019-12-05T06:28:09.890 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:09.878+0000 7f87bbfff700 1 --2- 172.21.15.42:0/2292136487 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f87bc0a4c60 0x7f87bc118a80 secure :-1 s=THROTTLE_DONE pgs=2675 cs=0 l=1 rx=0x7f87a4002f30 tx=0x7f87a4010a00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:09.890 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:09.882+0000 7f87b8ff9700 1 -- 172.21.15.42:0/2292136487 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f87a4005bf0 con 0x7f87bc0a4c60 2019-12-05T06:28:09.890 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:09.882+0000 7f87bbfff700 1 --2- 172.21.15.42:0/2292136487 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f87bc0a4c60 0x7f87bc118a80 secure :-1 s=THROTTLE_DONE pgs=2675 cs=0 l=1 rx=0x7f87a4002f30 tx=0x7f87a4010a00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:09.890 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:09.882+0000 7f87b8ff9700 1 --2- 172.21.15.42:0/2292136487 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f87a003b220 0x7f87a003d730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:28:09.890 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:09.882+0000 7f87b8ff9700 1 -- 172.21.15.42:0/2292136487 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 10933+0+0 (secure 0 0 0) 0x7f87a404abb0 con 0x7f87bc0a4c60 2019-12-05T06:28:09.891 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:09.882+0000 7f87bb7fe700 1 --2- 172.21.15.42:0/2292136487 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f87a003b220 0x7f87a003d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:28:09.891 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:09.882+0000 7f87bb7fe700 1 --2- 172.21.15.42:0/2292136487 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f87a003b220 0x7f87a003d730 crc :-1 s=READY pgs=1116 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:28:09.891 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:09.882+0000 7f87c1db1700 1 -- 172.21.15.42:0/2292136487 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87bc05d110 con 0x7f87bc0a4c60 2019-12-05T06:28:09.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:09.890+0000 7f87bbfff700 1 --2- 172.21.15.42:0/2292136487 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f87bc0a4c60 0x7f87bc118a80 secure :-1 s=THROTTLE_DONE pgs=2675 cs=0 l=1 rx=0x7f87a4002f30 tx=0x7f87a4010a00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:09.897 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:09.890+0000 7f87b8ff9700 1 -- 172.21.15.42:0/2292136487 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f87a4031c30 con 0x7f87bc0a4c60 2019-12-05T06:28:10.067 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:10.062+0000 7f87c1db1700 1 -- 172.21.15.42:0/2292136487 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f87bc116900 con 0x7f87bc0a4c60 2019-12-05T06:28:10.068 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:10.062+0000 7f87bbfff700 1 --2- 172.21.15.42:0/2292136487 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f87bc0a4c60 0x7f87bc118a80 secure :-1 s=THROTTLE_DONE pgs=2675 cs=0 l=1 rx=0x7f87a4002f30 tx=0x7f87a4010a00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:10.068 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:10.062+0000 7f87b8ff9700 1 -- 172.21.15.42:0/2292136487 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 318 v318) v1 ==== 95+0+2117 (secure 0 0 0) 0x7f87a4031e90 con 0x7f87bc0a4c60 2019-12-05T06:28:10.068 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 318 2019-12-05T06:28:10.069 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:28:10.069 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":318,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":11142,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/3155851600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3155851600},{"type":"v1","addr":"172.21.15.40:6833","nonce":3155851600}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":318}],"filesystems":[{"mdsmap":{"epoch":317,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:26:46.246866+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":106,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11221},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11221":{"gid":11221,"name":"a","rank":0,"incarnation":314,"state":"up:active","state_seq":34,"addr":"172.21.15.42:6835/3427104185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3427104185},{"type":"v1","addr":"172.21.15.42:6835","nonce":3427104185}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:28:10.071 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:10.066+0000 7f87c1db1700 1 -- 172.21.15.42:0/2292136487 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f87a003b220 msgr2=0x7f87a003d730 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:10.071 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:10.066+0000 7f87c1db1700 1 --2- 172.21.15.42:0/2292136487 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f87a003b220 0x7f87a003d730 crc :-1 s=READY pgs=1116 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:10.071 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:10.066+0000 7f87c1db1700 1 -- 172.21.15.42:0/2292136487 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f87bc0a4c60 msgr2=0x7f87bc118a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:10.071 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:10.066+0000 7f87c1db1700 1 --2- 172.21.15.42:0/2292136487 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f87bc0a4c60 0x7f87bc118a80 secure :-1 s=READY pgs=2675 cs=0 l=1 rx=0x7f87a4002f30 tx=0x7f87a4010a00).stop 2019-12-05T06:28:10.072 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:10.066+0000 7f87c1db1700 1 -- 172.21.15.42:0/2292136487 shutdown_connections 2019-12-05T06:28:10.072 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:10.066+0000 7f87c1db1700 1 --2- 172.21.15.42:0/2292136487 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f87a003b220 0x7f87a003d730 unknown :-1 s=CLOSED pgs=1116 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:10.072 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:10.066+0000 7f87c1db1700 1 --2- 172.21.15.42:0/2292136487 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f87bc0a4c60 0x7f87bc118a80 unknown :-1 s=CLOSED pgs=2675 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:10.072 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:10.066+0000 7f87c1db1700 1 -- 172.21.15.42:0/2292136487 shutdown_connections 2019-12-05T06:28:10.072 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:10.066+0000 7f87c1db1700 1 -- 172.21.15.42:0/2292136487 wait complete. 2019-12-05T06:28:10.072 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:10.066+0000 7f87c1db1700 1 -- 172.21.15.42:0/2292136487 >> 172.21.15.42:0/2292136487 conn(0x7f87bc0aabe0 msgr2=0x7f87bc059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:28:10.080 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.a (rank=0) 2019-12-05T06:28:10.081 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2019-12-05T06:28:10.081 INFO:teuthology.orchestra.run:waiting for 300 2019-12-05T06:28:10.095 INFO:tasks.ceph.mds.a:Stopped 2019-12-05T06:28:10.095 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:28:10.265 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:10.254+0000 7fc465281700 1 Processor -- start 2019-12-05T06:28:10.265 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:10.254+0000 7fc465281700 1 -- start start 2019-12-05T06:28:10.265 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:10.254+0000 7fc465281700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc4600a7740 0x7fc46008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:28:10.266 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:10.254+0000 7fc465281700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc46009fae0 0x7fc46007eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:28:10.266 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:10.254+0000 7fc465281700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4600a0710 con 0x7fc46007e720 2019-12-05T06:28:10.266 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:10.254+0000 7fc465281700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4600a0b00 con 0x7fc46009fae0 2019-12-05T06:28:10.266 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:10.254+0000 7fc465281700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc46008e5b0 con 0x7fc4600a7740 2019-12-05T06:28:10.266 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:10.254+0000 7fc45f7fe700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc4600a7740 0x7fc46008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:28:10.266 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:10.254+0000 7fc45ffff700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc46009fae0 0x7fc46007eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:28:10.267 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:10.254+0000 7fc45f7fe700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc4600a7740 0x7fc46008df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:56146/0 (socket says 172.21.15.42:56146) 2019-12-05T06:28:10.267 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:10.254+0000 7fc45f7fe700 1 -- 172.21.15.42:0/1585873393 learned_addr learned my addr 172.21.15.42:0/1585873393 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:28:10.267 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:10.258+0000 7fc45f7fe700 1 -- 172.21.15.42:0/1585873393 >> v2:172.21.15.40:3300/0 conn(0x7fc46009fae0 msgr2=0x7fc46007eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:28:10.267 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:10.258+0000 7fc45f7fe700 1 --2- 172.21.15.42:0/1585873393 >> v2:172.21.15.40:3300/0 conn(0x7fc46009fae0 0x7fc46007eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:28:10.267 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:10.258+0000 7fc45f7fe700 1 -- 172.21.15.42:0/1585873393 >> v1:172.21.15.40:6789/0 conn(0x7fc46007e720 legacy=0x7fc46009f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:10.267 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:10.258+0000 7fc45f7fe700 1 -- 172.21.15.42:0/1585873393 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc46007f0a0 con 0x7fc4600a7740 2019-12-05T06:28:10.267 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:10.258+0000 7fc45f7fe700 1 --2- 172.21.15.42:0/1585873393 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc4600a7740 0x7fc46008df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fc450008790 tx=0x7fc45000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:10.268 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:10.258+0000 7fc45f7fe700 1 --2- 172.21.15.42:0/1585873393 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc4600a7740 0x7fc46008df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fc450008790 tx=0x7fc45000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:10.268 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:10.258+0000 7fc45f7fe700 1 --2- 172.21.15.42:0/1585873393 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc4600a7740 0x7fc46008df80 secure :-1 s=READY pgs=587 cs=0 l=1 rx=0x7fc450008790 tx=0x7fc45000fd30).ready entity=mon.2 client_cookie=79d30339b1a761f9 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:28:10.268 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:10.258+0000 7fc45f7fe700 1 --2- 172.21.15.42:0/1585873393 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc4600a7740 0x7fc46008df80 secure :-1 s=THROTTLE_DONE pgs=587 cs=0 l=1 rx=0x7fc450008790 tx=0x7fc45000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:10.268 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:10.258+0000 7fc45f7fe700 1 --2- 172.21.15.42:0/1585873393 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc4600a7740 0x7fc46008df80 secure :-1 s=THROTTLE_DONE pgs=587 cs=0 l=1 rx=0x7fc450008790 tx=0x7fc45000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:10.268 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:10.258+0000 7fc45e7fc700 1 -- 172.21.15.42:0/1585873393 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc450008c90 con 0x7fc4600a7740 2019-12-05T06:28:10.268 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:10.258+0000 7fc45f7fe700 1 --2- 172.21.15.42:0/1585873393 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc4600a7740 0x7fc46008df80 secure :-1 s=THROTTLE_DONE pgs=587 cs=0 l=1 rx=0x7fc450008790 tx=0x7fc45000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:10.268 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:10.258+0000 7fc45e7fc700 1 -- 172.21.15.42:0/1585873393 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4500088b0 con 0x7fc4600a7740 2019-12-05T06:28:10.269 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:10.258+0000 7fc45e7fc700 1 -- 172.21.15.42:0/1585873393 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc450011b10 con 0x7fc4600a7740 2019-12-05T06:28:10.269 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:10.258+0000 7fc465281700 1 -- 172.21.15.42:0/1585873393 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc4600a7740 msgr2=0x7fc46008df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:10.269 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:10.258+0000 7fc465281700 1 --2- 172.21.15.42:0/1585873393 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc4600a7740 0x7fc46008df80 secure :-1 s=READY pgs=587 cs=0 l=1 rx=0x7fc450008790 tx=0x7fc45000fd30).stop 2019-12-05T06:28:10.269 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:10.258+0000 7fc465281700 1 -- 172.21.15.42:0/1585873393 shutdown_connections 2019-12-05T06:28:10.269 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:10.258+0000 7fc465281700 1 --2- 172.21.15.42:0/1585873393 >> v2:172.21.15.40:3300/0 conn(0x7fc46009fae0 0x7fc46007eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:28:10.270 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:10.258+0000 7fc465281700 1 --2- 172.21.15.42:0/1585873393 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc4600a7740 0x7fc46008df80 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:10.270 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:10.258+0000 7fc465281700 1 -- 172.21.15.42:0/1585873393 shutdown_connections 2019-12-05T06:28:10.270 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:10.258+0000 7fc465281700 1 -- 172.21.15.42:0/1585873393 wait complete. 2019-12-05T06:28:10.270 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:10.258+0000 7fc465281700 1 -- 172.21.15.42:0/1585873393 >> 172.21.15.42:0/1585873393 conn(0x7fc4600aabe0 msgr2=0x7fc4600a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:28:10.270 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:10.262+0000 7fc465281700 1 Processor -- start 2019-12-05T06:28:10.270 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:10.262+0000 7fc465281700 1 -- start start 2019-12-05T06:28:10.274 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:10.262+0000 7fc465281700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc46008eeb0 0x7fc4600a6080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:28:10.274 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:10.262+0000 7fc465281700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc4600a4a00 0x7fc4600a6a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:28:10.274 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:10.262+0000 7fc465281700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc460116270 0x7fc46011ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:28:10.275 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:10.262+0000 7fc465281700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4600a4e30 con 0x7fc4600a4a00 2019-12-05T06:28:10.275 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:10.262+0000 7fc465281700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4600a5b40 con 0x7fc46008eeb0 2019-12-05T06:28:10.275 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:10.262+0000 7fc465281700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4600a5de0 con 0x7fc460116270 2019-12-05T06:28:10.275 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:10.262+0000 7fc45f7fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc46008eeb0 0x7fc4600a6080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:28:10.275 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:10.262+0000 7fc45f7fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc46008eeb0 0x7fc4600a6080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:55686/0 (socket says 172.21.15.42:55686) 2019-12-05T06:28:10.275 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:10.262+0000 7fc45f7fe700 1 -- 172.21.15.42:0/3220841165 learned_addr learned my addr 172.21.15.42:0/3220841165 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:28:10.276 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:10.262+0000 7fc45effd700 1 --2- 172.21.15.42:0/3220841165 >> v2:172.21.15.40:3300/0 conn(0x7fc4600a4a00 0x7fc4600a6a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:28:10.276 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:10.262+0000 7fc45ffff700 1 --2- 172.21.15.42:0/3220841165 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc460116270 0x7fc46011ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:28:10.276 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:10.262+0000 7fc45f7fe700 1 -- 172.21.15.42:0/3220841165 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc460116270 msgr2=0x7fc46011ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:10.276 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:10.262+0000 7fc45f7fe700 1 --2- 172.21.15.42:0/3220841165 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc460116270 0x7fc46011ea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:10.276 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:10.262+0000 7fc45f7fe700 1 -- 172.21.15.42:0/3220841165 >> v2:172.21.15.40:3300/0 conn(0x7fc4600a4a00 msgr2=0x7fc4600a6a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:10.276 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:10.262+0000 7fc45f7fe700 1 --2- 172.21.15.42:0/3220841165 >> v2:172.21.15.40:3300/0 conn(0x7fc4600a4a00 0x7fc4600a6a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:10.276 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:10.262+0000 7fc45f7fe700 1 -- 172.21.15.42:0/3220841165 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4600a7140 con 0x7fc46008eeb0 2019-12-05T06:28:10.277 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:10.262+0000 7fc45effd700 1 --2- 172.21.15.42:0/3220841165 >> v2:172.21.15.40:3300/0 conn(0x7fc4600a4a00 0x7fc4600a6a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:28:10.277 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:10.262+0000 7fc45f7fe700 1 --2- 172.21.15.42:0/3220841165 >> v2:172.21.15.42:3300/0 conn(0x7fc46008eeb0 0x7fc4600a6080 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fc4600a5d00 tx=0x7fc450008b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:10.277 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:10.262+0000 7fc45f7fe700 1 --2- 172.21.15.42:0/3220841165 >> v2:172.21.15.42:3300/0 conn(0x7fc46008eeb0 0x7fc4600a6080 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fc4600a5d00 tx=0x7fc450008b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:10.277 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:10.262+0000 7fc45f7fe700 1 --2- 172.21.15.42:0/3220841165 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc46008eeb0 0x7fc4600a6080 secure :-1 s=READY pgs=2676 cs=0 l=1 rx=0x7fc4600a5d00 tx=0x7fc450008b60).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:28:10.277 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:10.262+0000 7fc45f7fe700 1 --2- 172.21.15.42:0/3220841165 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc46008eeb0 0x7fc4600a6080 secure :-1 s=THROTTLE_DONE pgs=2676 cs=0 l=1 rx=0x7fc4600a5d00 tx=0x7fc450008b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:10.277 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:10.262+0000 7fc45f7fe700 1 --2- 172.21.15.42:0/3220841165 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc46008eeb0 0x7fc4600a6080 secure :-1 s=THROTTLE_DONE pgs=2676 cs=0 l=1 rx=0x7fc4600a5d00 tx=0x7fc450008b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:10.278 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:10.262+0000 7fc45f7fe700 1 --2- 172.21.15.42:0/3220841165 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc46008eeb0 0x7fc4600a6080 secure :-1 s=THROTTLE_DONE pgs=2676 cs=0 l=1 rx=0x7fc4600a5d00 tx=0x7fc450008b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:10.278 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:10.262+0000 7fc447fff700 1 -- 172.21.15.42:0/3220841165 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc45000f470 con 0x7fc46008eeb0 2019-12-05T06:28:10.278 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:10.262+0000 7fc447fff700 1 -- 172.21.15.42:0/3220841165 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc450011730 con 0x7fc46008eeb0 2019-12-05T06:28:10.278 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:10.262+0000 7fc465281700 1 -- 172.21.15.42:0/3220841165 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4601166a0 con 0x7fc46008eeb0 2019-12-05T06:28:10.278 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:10.262+0000 7fc447fff700 1 -- 172.21.15.42:0/3220841165 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc450005730 con 0x7fc46008eeb0 2019-12-05T06:28:10.278 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:10.262+0000 7fc465281700 1 -- 172.21.15.42:0/3220841165 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc46011ef90 con 0x7fc46008eeb0 2019-12-05T06:28:10.278 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:10.266+0000 7fc45f7fe700 1 --2- 172.21.15.42:0/3220841165 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc46008eeb0 0x7fc4600a6080 secure :-1 s=THROTTLE_DONE pgs=2676 cs=0 l=1 rx=0x7fc4600a5d00 tx=0x7fc450008b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:10.279 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:10.266+0000 7fc447fff700 1 -- 172.21.15.42:0/3220841165 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fc450005a40 con 0x7fc46008eeb0 2019-12-05T06:28:10.279 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:10.266+0000 7fc45f7fe700 1 --2- 172.21.15.42:0/3220841165 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc46008eeb0 0x7fc4600a6080 secure :-1 s=THROTTLE_DONE pgs=2676 cs=0 l=1 rx=0x7fc4600a5d00 tx=0x7fc450008b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:10.279 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:10.266+0000 7fc447fff700 1 --2- 172.21.15.42:0/3220841165 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc44003ae90 0x7fc44003d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:28:10.279 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:10.266+0000 7fc447fff700 1 -- 172.21.15.42:0/3220841165 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 10933+0+0 (secure 0 0 0) 0x7fc45004b730 con 0x7fc46008eeb0 2019-12-05T06:28:10.279 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:10.266+0000 7fc45effd700 1 --2- 172.21.15.42:0/3220841165 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc44003ae90 0x7fc44003d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:28:10.279 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:10.266+0000 7fc45effd700 1 --2- 172.21.15.42:0/3220841165 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc44003ae90 0x7fc44003d3a0 crc :-1 s=READY pgs=1117 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:28:10.279 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:10.270+0000 7fc465281700 1 -- 172.21.15.42:0/3220841165 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc46005d110 con 0x7fc46008eeb0 2019-12-05T06:28:10.280 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:10.278+0000 7fc45f7fe700 1 --2- 172.21.15.42:0/3220841165 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc46008eeb0 0x7fc4600a6080 secure :-1 s=THROTTLE_DONE pgs=2676 cs=0 l=1 rx=0x7fc4600a5d00 tx=0x7fc450008b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:10.281 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:10.278+0000 7fc447fff700 1 -- 172.21.15.42:0/3220841165 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fc450008de0 con 0x7fc46008eeb0 2019-12-05T06:28:10.455 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:10.450+0000 7fc465281700 1 -- 172.21.15.42:0/3220841165 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc46005b8c0 con 0x7fc46008eeb0 2019-12-05T06:28:10.456 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:10.454+0000 7fc45f7fe700 1 --2- 172.21.15.42:0/3220841165 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc46008eeb0 0x7fc4600a6080 secure :-1 s=THROTTLE_DONE pgs=2676 cs=0 l=1 rx=0x7fc4600a5d00 tx=0x7fc450008b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:10.456 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:10.454+0000 7fc447fff700 1 -- 172.21.15.42:0/3220841165 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 318 v318) v1 ==== 95+0+2117 (secure 0 0 0) 0x7fc450010c00 con 0x7fc46008eeb0 2019-12-05T06:28:10.456 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 318 2019-12-05T06:28:10.457 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:28:10.457 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":318,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":11142,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/3155851600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3155851600},{"type":"v1","addr":"172.21.15.40:6833","nonce":3155851600}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":318}],"filesystems":[{"mdsmap":{"epoch":317,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:26:46.246866+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":106,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11221},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11221":{"gid":11221,"name":"a","rank":0,"incarnation":314,"state":"up:active","state_seq":34,"addr":"172.21.15.42:6835/3427104185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3427104185},{"type":"v1","addr":"172.21.15.42:6835","nonce":3427104185}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:28:10.459 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:10.454+0000 7fc465281700 1 -- 172.21.15.42:0/3220841165 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc44003ae90 msgr2=0x7fc44003d3a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:10.459 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:10.454+0000 7fc465281700 1 --2- 172.21.15.42:0/3220841165 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc44003ae90 0x7fc44003d3a0 crc :-1 s=READY pgs=1117 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:10.459 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:10.454+0000 7fc465281700 1 -- 172.21.15.42:0/3220841165 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc46008eeb0 msgr2=0x7fc4600a6080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:10.459 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:10.454+0000 7fc465281700 1 --2- 172.21.15.42:0/3220841165 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc46008eeb0 0x7fc4600a6080 secure :-1 s=READY pgs=2676 cs=0 l=1 rx=0x7fc4600a5d00 tx=0x7fc450008b60).stop 2019-12-05T06:28:10.459 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:10.454+0000 7fc465281700 1 -- 172.21.15.42:0/3220841165 shutdown_connections 2019-12-05T06:28:10.459 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:10.454+0000 7fc465281700 1 --2- 172.21.15.42:0/3220841165 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc44003ae90 0x7fc44003d3a0 unknown :-1 s=CLOSED pgs=1117 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:10.460 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:10.454+0000 7fc465281700 1 --2- 172.21.15.42:0/3220841165 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc460116270 0x7fc46011ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:10.460 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:10.454+0000 7fc465281700 1 --2- 172.21.15.42:0/3220841165 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc46008eeb0 0x7fc4600a6080 unknown :-1 s=CLOSED pgs=2676 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:10.460 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:10.454+0000 7fc465281700 1 --2- 172.21.15.42:0/3220841165 >> v2:172.21.15.40:3300/0 conn(0x7fc4600a4a00 0x7fc4600a6a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:10.460 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:10.454+0000 7fc465281700 1 -- 172.21.15.42:0/3220841165 shutdown_connections 2019-12-05T06:28:10.460 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:10.454+0000 7fc465281700 1 -- 172.21.15.42:0/3220841165 wait complete. 2019-12-05T06:28:10.460 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:10.454+0000 7fc465281700 1 -- 172.21.15.42:0/3220841165 >> 172.21.15.42:0/3220841165 conn(0x7fc46008fde0 msgr2=0x7fc4600a9800 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:28:10.469 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:28:12.470 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:28:12.577 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:12.566+0000 7ff74f5e3700 1 Processor -- start 2019-12-05T06:28:12.577 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:12.566+0000 7ff74f5e3700 1 -- start start 2019-12-05T06:28:12.577 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:12.570+0000 7ff74f5e3700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff7480a7740 0x7ff74808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:28:12.577 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:12.570+0000 7ff74f5e3700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff74809fae0 0x7ff74807eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:28:12.577 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:12.570+0000 7ff74f5e3700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7480a0710 con 0x7ff74807e720 2019-12-05T06:28:12.578 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:12.570+0000 7ff74f5e3700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7480a0b00 con 0x7ff74809fae0 2019-12-05T06:28:12.578 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:12.570+0000 7ff74f5e3700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff74808e5b0 con 0x7ff7480a7740 2019-12-05T06:28:12.578 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:12.570+0000 7ff74db80700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff7480a7740 0x7ff74808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:28:12.578 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:12.570+0000 7ff74e381700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff74809fae0 0x7ff74807eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:28:12.578 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:12.570+0000 7ff74db80700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff7480a7740 0x7ff74808df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:56166/0 (socket says 172.21.15.42:56166) 2019-12-05T06:28:12.579 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:12.570+0000 7ff74e381700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff74809fae0 0x7ff74807eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.42:43520/0 (socket says 172.21.15.42:43520) 2019-12-05T06:28:12.579 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:12.570+0000 7ff74db80700 1 -- 172.21.15.42:0/1783752194 learned_addr learned my addr 172.21.15.42:0/1783752194 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:28:12.579 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:12.570+0000 7ff74e381700 1 -- 172.21.15.42:0/1783752194 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff7480a7740 msgr2=0x7ff74808df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:28:12.579 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:12.570+0000 7ff74e381700 1 --2- 172.21.15.42:0/1783752194 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff7480a7740 0x7ff74808df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:28:12.579 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:12.570+0000 7ff74e381700 1 -- 172.21.15.42:0/1783752194 >> v1:172.21.15.40:6789/0 conn(0x7ff74807e720 legacy=0x7ff74809f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:12.579 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:12.570+0000 7ff74cb7e700 1 -- 172.21.15.42:0/1783752194 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2357792940 0 0) 0x7ff7480a0710 con 0x7ff74807e720 2019-12-05T06:28:12.579 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:12.570+0000 7ff74e381700 1 -- 172.21.15.42:0/1783752194 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff74807f0a0 con 0x7ff74809fae0 2019-12-05T06:28:12.580 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:12.570+0000 7ff74db80700 1 --2- 172.21.15.42:0/1783752194 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff7480a7740 0x7ff74808df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:28:12.580 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:12.570+0000 7ff74e381700 1 --2- 172.21.15.42:0/1783752194 >> v2:172.21.15.40:3300/0 conn(0x7ff74809fae0 0x7ff74807eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7ff740008b10 tx=0x7ff74000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:12.580 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:12.570+0000 7ff74e381700 1 --2- 172.21.15.42:0/1783752194 >> v2:172.21.15.40:3300/0 conn(0x7ff74809fae0 0x7ff74807eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7ff740008b10 tx=0x7ff74000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:12.580 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:12.570+0000 7ff74e381700 1 --2- 172.21.15.42:0/1783752194 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff74809fae0 0x7ff74807eb50 secure :-1 s=READY pgs=2007 cs=0 l=1 rx=0x7ff740008b10 tx=0x7ff74000fff0).ready entity=mon.0 client_cookie=a1855abc6dfd2136 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:28:12.580 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:12.570+0000 7ff74e381700 1 --2- 172.21.15.42:0/1783752194 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff74809fae0 0x7ff74807eb50 secure :-1 s=THROTTLE_DONE pgs=2007 cs=0 l=1 rx=0x7ff740008b10 tx=0x7ff74000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:12.581 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:12.570+0000 7ff74cb7e700 1 -- 172.21.15.42:0/1783752194 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff740010900 con 0x7ff74809fae0 2019-12-05T06:28:12.581 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:12.570+0000 7ff74e381700 1 --2- 172.21.15.42:0/1783752194 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff74809fae0 0x7ff74807eb50 secure :-1 s=THROTTLE_DONE pgs=2007 cs=0 l=1 rx=0x7ff740008b10 tx=0x7ff74000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:12.581 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:12.570+0000 7ff74e381700 1 --2- 172.21.15.42:0/1783752194 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff74809fae0 0x7ff74807eb50 secure :-1 s=THROTTLE_DONE pgs=2007 cs=0 l=1 rx=0x7ff740008b10 tx=0x7ff74000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:12.581 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:12.570+0000 7ff74cb7e700 1 -- 172.21.15.42:0/1783752194 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff740008b50 con 0x7ff74809fae0 2019-12-05T06:28:12.582 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:12.570+0000 7ff74cb7e700 1 -- 172.21.15.42:0/1783752194 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff7400110c0 con 0x7ff74809fae0 2019-12-05T06:28:12.582 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:12.570+0000 7ff74f5e3700 1 -- 172.21.15.42:0/1783752194 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff74809fae0 msgr2=0x7ff74807eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:12.582 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:12.570+0000 7ff74f5e3700 1 --2- 172.21.15.42:0/1783752194 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff74809fae0 0x7ff74807eb50 secure :-1 s=READY pgs=2007 cs=0 l=1 rx=0x7ff740008b10 tx=0x7ff74000fff0).stop 2019-12-05T06:28:12.582 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:12.570+0000 7ff74f5e3700 1 -- 172.21.15.42:0/1783752194 shutdown_connections 2019-12-05T06:28:12.583 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:12.570+0000 7ff74f5e3700 1 --2- 172.21.15.42:0/1783752194 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff74809fae0 0x7ff74807eb50 unknown :-1 s=CLOSED pgs=2007 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:12.583 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:12.570+0000 7ff74f5e3700 1 --2- 172.21.15.42:0/1783752194 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff7480a7740 0x7ff74808df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:28:12.583 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:12.570+0000 7ff74f5e3700 1 -- 172.21.15.42:0/1783752194 shutdown_connections 2019-12-05T06:28:12.583 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:12.570+0000 7ff74f5e3700 1 -- 172.21.15.42:0/1783752194 wait complete. 2019-12-05T06:28:12.583 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:12.570+0000 7ff74f5e3700 1 -- 172.21.15.42:0/1783752194 >> 172.21.15.42:0/1783752194 conn(0x7ff7480aabe0 msgr2=0x7ff7480a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:28:12.586 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:12.574+0000 7ff74f5e3700 1 Processor -- start 2019-12-05T06:28:12.587 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:12.574+0000 7ff74f5e3700 1 -- start start 2019-12-05T06:28:12.587 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:12.574+0000 7ff74f5e3700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff748142b00 0x7ff748144a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:28:12.587 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:12.574+0000 7ff74f5e3700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff748116500 0x7ff748144fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:28:12.587 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:12.574+0000 7ff74f5e3700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff74811a6a0 0x7ff74811b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:28:12.587 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:12.574+0000 7ff74f5e3700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff748130380 con 0x7ff748116500 2019-12-05T06:28:12.587 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:12.574+0000 7ff74f5e3700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff748143010 con 0x7ff74811a6a0 2019-12-05T06:28:12.587 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:12.574+0000 7ff74f5e3700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7481432b0 con 0x7ff748142b00 2019-12-05T06:28:12.588 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:12.574+0000 7ff74e381700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff74811a6a0 0x7ff74811b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:28:12.588 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:12.574+0000 7ff74db80700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff748142b00 0x7ff748144a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:28:12.588 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:12.574+0000 7ff74d37f700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff748116500 0x7ff748144fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:28:12.588 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:12.574+0000 7ff74db80700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff748142b00 0x7ff748144a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:56172/0 (socket says 172.21.15.42:56172) 2019-12-05T06:28:12.588 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:12.574+0000 7ff74e381700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff74811a6a0 0x7ff74811b3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:55710/0 (socket says 172.21.15.42:55710) 2019-12-05T06:28:12.588 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:12.574+0000 7ff74db80700 1 -- 172.21.15.42:0/3714793412 learned_addr learned my addr 172.21.15.42:0/3714793412 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:28:12.589 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:12.578+0000 7ff74d37f700 1 -- 172.21.15.42:0/3714793412 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff748142b00 msgr2=0x7ff748144a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:12.589 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:12.578+0000 7ff74d37f700 1 --2- 172.21.15.42:0/3714793412 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff748142b00 0x7ff748144a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:12.589 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:12.578+0000 7ff74d37f700 1 -- 172.21.15.42:0/3714793412 >> v2:172.21.15.42:3300/0 conn(0x7ff74811a6a0 msgr2=0x7ff74811b3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:12.589 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:12.578+0000 7ff74d37f700 1 --2- 172.21.15.42:0/3714793412 >> v2:172.21.15.42:3300/0 conn(0x7ff74811a6a0 0x7ff74811b3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:12.589 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:12.578+0000 7ff74d37f700 1 -- 172.21.15.42:0/3714793412 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff74808f9b0 con 0x7ff748116500 2019-12-05T06:28:12.589 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:12.578+0000 7ff74db80700 1 --2- 172.21.15.42:0/3714793412 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff748142b00 0x7ff748144a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:28:12.590 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:12.578+0000 7ff74e381700 1 --2- 172.21.15.42:0/3714793412 >> v2:172.21.15.42:3300/0 conn(0x7ff74811a6a0 0x7ff74811b3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:28:12.590 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:12.578+0000 7ff74d37f700 1 --2- 172.21.15.42:0/3714793412 >> v2:172.21.15.40:3300/0 conn(0x7ff748116500 0x7ff748144fd0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7ff74811ab60 tx=0x7ff7440108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:12.590 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:12.578+0000 7ff74d37f700 1 --2- 172.21.15.42:0/3714793412 >> v2:172.21.15.40:3300/0 conn(0x7ff748116500 0x7ff748144fd0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7ff74811ab60 tx=0x7ff7440108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:12.590 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:12.578+0000 7ff74d37f700 1 --2- 172.21.15.42:0/3714793412 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff748116500 0x7ff748144fd0 secure :-1 s=READY pgs=2008 cs=0 l=1 rx=0x7ff74811ab60 tx=0x7ff7440108f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:28:12.590 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:12.578+0000 7ff74d37f700 1 --2- 172.21.15.42:0/3714793412 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff748116500 0x7ff748144fd0 secure :-1 s=THROTTLE_DONE pgs=2008 cs=0 l=1 rx=0x7ff74811ab60 tx=0x7ff7440108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:12.590 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:12.578+0000 7ff73a7fc700 1 -- 172.21.15.42:0/3714793412 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff744006a40 con 0x7ff748116500 2019-12-05T06:28:12.591 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:12.578+0000 7ff74d37f700 1 --2- 172.21.15.42:0/3714793412 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff748116500 0x7ff748144fd0 secure :-1 s=THROTTLE_DONE pgs=2008 cs=0 l=1 rx=0x7ff74811ab60 tx=0x7ff7440108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:12.591 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:12.578+0000 7ff74d37f700 1 --2- 172.21.15.42:0/3714793412 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff748116500 0x7ff748144fd0 secure :-1 s=THROTTLE_DONE pgs=2008 cs=0 l=1 rx=0x7ff74811ab60 tx=0x7ff7440108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:12.591 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:12.578+0000 7ff73a7fc700 1 -- 172.21.15.42:0/3714793412 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff744006660 con 0x7ff748116500 2019-12-05T06:28:12.591 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:12.578+0000 7ff74f5e3700 1 -- 172.21.15.42:0/3714793412 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff74811b900 con 0x7ff748116500 2019-12-05T06:28:12.591 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:12.578+0000 7ff73a7fc700 1 -- 172.21.15.42:0/3714793412 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff7440110b0 con 0x7ff748116500 2019-12-05T06:28:12.591 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:12.578+0000 7ff74f5e3700 1 -- 172.21.15.42:0/3714793412 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff74811bcb0 con 0x7ff748116500 2019-12-05T06:28:12.591 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:12.578+0000 7ff74d37f700 1 --2- 172.21.15.42:0/3714793412 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff748116500 0x7ff748144fd0 secure :-1 s=THROTTLE_DONE pgs=2008 cs=0 l=1 rx=0x7ff74811ab60 tx=0x7ff7440108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:12.592 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:12.578+0000 7ff73a7fc700 1 -- 172.21.15.42:0/3714793412 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7ff744011380 con 0x7ff748116500 2019-12-05T06:28:12.592 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:12.578+0000 7ff74d37f700 1 --2- 172.21.15.42:0/3714793412 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff748116500 0x7ff748144fd0 secure :-1 s=THROTTLE_DONE pgs=2008 cs=0 l=1 rx=0x7ff74811ab60 tx=0x7ff7440108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:12.592 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:12.578+0000 7ff73a7fc700 1 --2- 172.21.15.42:0/3714793412 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff73003ae60 0x7ff73003d370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:28:12.592 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:12.578+0000 7ff73a7fc700 1 -- 172.21.15.42:0/3714793412 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 10933+0+0 (secure 0 0 0) 0x7ff74404b0f0 con 0x7ff748116500 2019-12-05T06:28:12.592 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:12.582+0000 7ff74db80700 1 --2- 172.21.15.42:0/3714793412 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff73003ae60 0x7ff73003d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:28:12.593 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:12.582+0000 7ff74db80700 1 --2- 172.21.15.42:0/3714793412 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff73003ae60 0x7ff73003d370 crc :-1 s=READY pgs=1118 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:28:12.593 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:12.582+0000 7ff74f5e3700 1 -- 172.21.15.42:0/3714793412 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff74805d110 con 0x7ff748116500 2019-12-05T06:28:12.597 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:12.594+0000 7ff74d37f700 1 --2- 172.21.15.42:0/3714793412 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff748116500 0x7ff748144fd0 secure :-1 s=THROTTLE_DONE pgs=2008 cs=0 l=1 rx=0x7ff74811ab60 tx=0x7ff7440108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:12.597 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:12.594+0000 7ff73a7fc700 1 -- 172.21.15.42:0/3714793412 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7ff744032170 con 0x7ff748116500 2019-12-05T06:28:12.764 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:12.762+0000 7ff74f5e3700 1 -- 172.21.15.42:0/3714793412 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff74805b8c0 con 0x7ff748116500 2019-12-05T06:28:12.765 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:12.762+0000 7ff74d37f700 1 --2- 172.21.15.42:0/3714793412 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff748116500 0x7ff748144fd0 secure :-1 s=THROTTLE_DONE pgs=2008 cs=0 l=1 rx=0x7ff74811ab60 tx=0x7ff7440108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:12.765 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:12.762+0000 7ff73a7fc700 1 -- 172.21.15.42:0/3714793412 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 318 v318) v1 ==== 95+0+2117 (secure 0 0 0) 0x7ff7440323d0 con 0x7ff748116500 2019-12-05T06:28:12.767 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 318 2019-12-05T06:28:12.767 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:28:12.768 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":318,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":11142,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/3155851600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3155851600},{"type":"v1","addr":"172.21.15.40:6833","nonce":3155851600}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":318}],"filesystems":[{"mdsmap":{"epoch":317,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:26:46.246866+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":106,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11221},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11221":{"gid":11221,"name":"a","rank":0,"incarnation":314,"state":"up:active","state_seq":34,"addr":"172.21.15.42:6835/3427104185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3427104185},{"type":"v1","addr":"172.21.15.42:6835","nonce":3427104185}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:28:12.770 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:12.766+0000 7ff74f5e3700 1 -- 172.21.15.42:0/3714793412 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff73003ae60 msgr2=0x7ff73003d370 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:12.771 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:12.766+0000 7ff74f5e3700 1 --2- 172.21.15.42:0/3714793412 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff73003ae60 0x7ff73003d370 crc :-1 s=READY pgs=1118 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:12.771 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:12.766+0000 7ff74f5e3700 1 -- 172.21.15.42:0/3714793412 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff748116500 msgr2=0x7ff748144fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:12.771 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:12.766+0000 7ff74f5e3700 1 --2- 172.21.15.42:0/3714793412 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff748116500 0x7ff748144fd0 secure :-1 s=READY pgs=2008 cs=0 l=1 rx=0x7ff74811ab60 tx=0x7ff7440108f0).stop 2019-12-05T06:28:12.771 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:12.766+0000 7ff74f5e3700 1 -- 172.21.15.42:0/3714793412 shutdown_connections 2019-12-05T06:28:12.771 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:12.766+0000 7ff74f5e3700 1 --2- 172.21.15.42:0/3714793412 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff73003ae60 0x7ff73003d370 unknown :-1 s=CLOSED pgs=1118 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:12.772 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:12.766+0000 7ff74f5e3700 1 --2- 172.21.15.42:0/3714793412 >> v2:172.21.15.42:3300/0 conn(0x7ff74811a6a0 0x7ff74811b3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:12.772 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:12.766+0000 7ff74f5e3700 1 --2- 172.21.15.42:0/3714793412 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff748142b00 0x7ff748144a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:12.772 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:12.766+0000 7ff74f5e3700 1 --2- 172.21.15.42:0/3714793412 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff748116500 0x7ff748144fd0 unknown :-1 s=CLOSED pgs=2008 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:12.772 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:12.766+0000 7ff74f5e3700 1 -- 172.21.15.42:0/3714793412 shutdown_connections 2019-12-05T06:28:12.772 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:12.766+0000 7ff74f5e3700 1 -- 172.21.15.42:0/3714793412 wait complete. 2019-12-05T06:28:12.772 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:12.766+0000 7ff74f5e3700 1 -- 172.21.15.42:0/3714793412 >> 172.21.15.42:0/3714793412 conn(0x7ff74807e4f0 msgr2=0x7ff74811c610 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:28:12.781 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:28:14.782 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:28:14.823 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:28:14.827 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:28:14.886 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:14.874+0000 7fcd006de700 1 Processor -- start 2019-12-05T06:28:14.886 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:14.874+0000 7fcd006de700 1 -- start start 2019-12-05T06:28:14.886 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:14.874+0000 7fcd006de700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fccf808df70 0x7fccf809f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:28:14.886 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:14.874+0000 7fcd006de700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccf809fc50 con 0x7fccf809f820 2019-12-05T06:28:14.887 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:14.874+0000 7fcd006de700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccf80a0700 con 0x7fccf80a7730 2019-12-05T06:28:14.887 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:14.874+0000 7fcd006de700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fccf80a0af0 con 0x7fccf808df70 2019-12-05T06:28:14.887 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:14.874+0000 7fccfe47a700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fccf808df70 0x7fccf809f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:28:14.887 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:14.874+0000 7fccfe47a700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fccf808df70 0x7fccf809f2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:55716/0 (socket says 172.21.15.42:55716) 2019-12-05T06:28:14.887 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:14.874+0000 7fccfe47a700 1 -- 172.21.15.42:0/2533357068 learned_addr learned my addr 172.21.15.42:0/2533357068 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:28:14.888 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:14.878+0000 7fccfe47a700 1 -- 172.21.15.42:0/2533357068 >> v1:172.21.15.42:6789/0 conn(0x7fccf80a7730 legacy=0x7fccf807e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:14.888 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:14.878+0000 7fccfe47a700 1 -- 172.21.15.42:0/2533357068 >> v1:172.21.15.40:6789/0 conn(0x7fccf809f820 legacy=0x7fccf814efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:14.888 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:14.878+0000 7fccfe47a700 1 -- 172.21.15.42:0/2533357068 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccf807ef10 con 0x7fccf808df70 2019-12-05T06:28:14.888 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:14.878+0000 7fccfe47a700 1 --2- 172.21.15.42:0/2533357068 >> v2:172.21.15.42:3300/0 conn(0x7fccf808df70 0x7fccf809f2d0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fcce8008790 tx=0x7fcce800fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:14.888 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:14.878+0000 7fccfe47a700 1 --2- 172.21.15.42:0/2533357068 >> v2:172.21.15.42:3300/0 conn(0x7fccf808df70 0x7fccf809f2d0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fcce8008790 tx=0x7fcce800fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:14.888 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:14.878+0000 7fccfe47a700 1 --2- 172.21.15.42:0/2533357068 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fccf808df70 0x7fccf809f2d0 secure :-1 s=READY pgs=2678 cs=0 l=1 rx=0x7fcce8008790 tx=0x7fcce800fcb0).ready entity=mon.1 client_cookie=ced4bf75a77bdd1c server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:28:14.889 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:14.878+0000 7fccfe47a700 1 --2- 172.21.15.42:0/2533357068 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fccf808df70 0x7fccf809f2d0 secure :-1 s=THROTTLE_DONE pgs=2678 cs=0 l=1 rx=0x7fcce8008790 tx=0x7fcce800fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:14.890 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:14.878+0000 7fccfe47a700 1 --2- 172.21.15.42:0/2533357068 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fccf808df70 0x7fccf809f2d0 secure :-1 s=THROTTLE_DONE pgs=2678 cs=0 l=1 rx=0x7fcce8008790 tx=0x7fcce800fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:14.890 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:14.878+0000 7fccfdc79700 1 -- 172.21.15.42:0/2533357068 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcce8011520 con 0x7fccf808df70 2019-12-05T06:28:14.890 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:14.878+0000 7fccfe47a700 1 --2- 172.21.15.42:0/2533357068 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fccf808df70 0x7fccf809f2d0 secure :-1 s=THROTTLE_DONE pgs=2678 cs=0 l=1 rx=0x7fcce8008790 tx=0x7fcce800fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:14.890 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:14.878+0000 7fccfdc79700 1 -- 172.21.15.42:0/2533357068 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcce8008ca0 con 0x7fccf808df70 2019-12-05T06:28:14.890 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:14.878+0000 7fccfdc79700 1 -- 172.21.15.42:0/2533357068 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcce8011b00 con 0x7fccf808df70 2019-12-05T06:28:14.891 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:14.878+0000 7fcd006de700 1 -- 172.21.15.42:0/2533357068 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fccf808df70 msgr2=0x7fccf809f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:14.891 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:14.878+0000 7fcd006de700 1 --2- 172.21.15.42:0/2533357068 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fccf808df70 0x7fccf809f2d0 secure :-1 s=READY pgs=2678 cs=0 l=1 rx=0x7fcce8008790 tx=0x7fcce800fcb0).stop 2019-12-05T06:28:14.891 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:14.878+0000 7fcd006de700 1 -- 172.21.15.42:0/2533357068 shutdown_connections 2019-12-05T06:28:14.892 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:14.878+0000 7fcd006de700 1 --2- 172.21.15.42:0/2533357068 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fccf808df70 0x7fccf809f2d0 unknown :-1 s=CLOSED pgs=2678 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:14.892 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:14.878+0000 7fcd006de700 1 -- 172.21.15.42:0/2533357068 shutdown_connections 2019-12-05T06:28:14.892 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:14.878+0000 7fcd006de700 1 -- 172.21.15.42:0/2533357068 wait complete. 2019-12-05T06:28:14.892 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:14.878+0000 7fcd006de700 1 -- 172.21.15.42:0/2533357068 >> 172.21.15.42:0/2533357068 conn(0x7fccf80aabd0 msgr2=0x7fccf80a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:28:14.892 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:14.878+0000 7fcd006de700 1 Processor -- start 2019-12-05T06:28:14.893 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:14.878+0000 7fcd006de700 1 -- start start 2019-12-05T06:28:14.893 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:14.878+0000 7fcd006de700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fccf8115ba0 0x7fccf8113f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:28:14.893 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:14.878+0000 7fcd006de700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fccf807db60 0x7fccf8112fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:28:14.893 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:14.878+0000 7fcd006de700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fccf807df90 0x7fccf8113510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:28:14.893 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:14.878+0000 7fcd006de700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fccf8113d40 con 0x7fccf807df90 2019-12-05T06:28:14.893 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:14.878+0000 7fcd006de700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fccf80a54c0 con 0x7fccf8115ba0 2019-12-05T06:28:14.893 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:14.878+0000 7fcd006de700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccf80a5760 con 0x7fccf807db60 2019-12-05T06:28:14.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:14.878+0000 7fccfec7b700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fccf8115ba0 0x7fccf8113f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:28:14.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:14.878+0000 7fccfec7b700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fccf8115ba0 0x7fccf8113f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:55724/0 (socket says 172.21.15.42:55724) 2019-12-05T06:28:14.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:14.878+0000 7fccfec7b700 1 -- 172.21.15.42:0/2865578000 learned_addr learned my addr 172.21.15.42:0/2865578000 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:28:14.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:14.878+0000 7fccfe47a700 1 --2- 172.21.15.42:0/2865578000 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fccf807db60 0x7fccf8112fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:28:14.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:14.878+0000 7fccff47c700 1 --2- 172.21.15.42:0/2865578000 >> v2:172.21.15.40:3300/0 conn(0x7fccf807df90 0x7fccf8113510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:28:14.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:14.882+0000 7fccfec7b700 1 -- 172.21.15.42:0/2865578000 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fccf807db60 msgr2=0x7fccf8112fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:14.895 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:14.882+0000 7fccfec7b700 1 --2- 172.21.15.42:0/2865578000 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fccf807db60 0x7fccf8112fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:14.895 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:14.882+0000 7fccfec7b700 1 -- 172.21.15.42:0/2865578000 >> v2:172.21.15.40:3300/0 conn(0x7fccf807df90 msgr2=0x7fccf8113510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:14.895 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:14.882+0000 7fccfec7b700 1 --2- 172.21.15.42:0/2865578000 >> v2:172.21.15.40:3300/0 conn(0x7fccf807df90 0x7fccf8113510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:14.895 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:14.882+0000 7fccfec7b700 1 -- 172.21.15.42:0/2865578000 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccf80a5120 con 0x7fccf8115ba0 2019-12-05T06:28:14.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:14.882+0000 7fccfec7b700 1 --2- 172.21.15.42:0/2865578000 >> v2:172.21.15.42:3300/0 conn(0x7fccf8115ba0 0x7fccf8113f60 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fccf80a5680 tx=0x7fccf4010a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:14.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:14.882+0000 7fccfec7b700 1 --2- 172.21.15.42:0/2865578000 >> v2:172.21.15.42:3300/0 conn(0x7fccf8115ba0 0x7fccf8113f60 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fccf80a5680 tx=0x7fccf4010a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:14.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:14.882+0000 7fccfec7b700 1 --2- 172.21.15.42:0/2865578000 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fccf8115ba0 0x7fccf8113f60 secure :-1 s=READY pgs=2679 cs=0 l=1 rx=0x7fccf80a5680 tx=0x7fccf4010a20).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:28:14.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:14.882+0000 7fccfec7b700 1 --2- 172.21.15.42:0/2865578000 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fccf8115ba0 0x7fccf8113f60 secure :-1 s=THROTTLE_DONE pgs=2679 cs=0 l=1 rx=0x7fccf80a5680 tx=0x7fccf4010a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:14.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:14.882+0000 7fccef7fe700 1 -- 172.21.15.42:0/2865578000 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fccf400ca70 con 0x7fccf8115ba0 2019-12-05T06:28:14.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:14.882+0000 7fccfec7b700 1 --2- 172.21.15.42:0/2865578000 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fccf8115ba0 0x7fccf8113f60 secure :-1 s=THROTTLE_DONE pgs=2679 cs=0 l=1 rx=0x7fccf80a5680 tx=0x7fccf4010a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:14.897 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:14.882+0000 7fccef7fe700 1 -- 172.21.15.42:0/2865578000 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccf400c690 con 0x7fccf8115ba0 2019-12-05T06:28:14.897 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:14.882+0000 7fcd006de700 1 -- 172.21.15.42:0/2865578000 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccf81144b0 con 0x7fccf8115ba0 2019-12-05T06:28:14.897 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:14.882+0000 7fccfec7b700 1 --2- 172.21.15.42:0/2865578000 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fccf8115ba0 0x7fccf8113f60 secure :-1 s=THROTTLE_DONE pgs=2679 cs=0 l=1 rx=0x7fccf80a5680 tx=0x7fccf4010a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:14.897 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:14.882+0000 7fccef7fe700 1 -- 172.21.15.42:0/2865578000 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fccf40111a0 con 0x7fccf8115ba0 2019-12-05T06:28:14.897 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:14.882+0000 7fcd006de700 1 -- 172.21.15.42:0/2865578000 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccf808eb60 con 0x7fccf8115ba0 2019-12-05T06:28:14.897 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:14.882+0000 7fccfec7b700 1 --2- 172.21.15.42:0/2865578000 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fccf8115ba0 0x7fccf8113f60 secure :-1 s=THROTTLE_DONE pgs=2679 cs=0 l=1 rx=0x7fccf80a5680 tx=0x7fccf4010a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:14.900 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:14.882+0000 7fccef7fe700 1 -- 172.21.15.42:0/2865578000 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fccf4011480 con 0x7fccf8115ba0 2019-12-05T06:28:14.900 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:14.882+0000 7fccfec7b700 1 --2- 172.21.15.42:0/2865578000 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fccf8115ba0 0x7fccf8113f60 secure :-1 s=THROTTLE_DONE pgs=2679 cs=0 l=1 rx=0x7fccf80a5680 tx=0x7fccf4010a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:14.901 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:14.882+0000 7fccef7fe700 1 --2- 172.21.15.42:0/2865578000 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fcce003adb0 0x7fcce003d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:28:14.901 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:14.882+0000 7fccef7fe700 1 -- 172.21.15.42:0/2865578000 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 10933+0+0 (secure 0 0 0) 0x7fccf404b1c0 con 0x7fccf8115ba0 2019-12-05T06:28:14.901 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:14.882+0000 7fccfe47a700 1 --2- 172.21.15.42:0/2865578000 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fcce003adb0 0x7fcce003d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:28:14.901 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:14.882+0000 7fccfe47a700 1 --2- 172.21.15.42:0/2865578000 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fcce003adb0 0x7fcce003d2c0 crc :-1 s=READY pgs=1120 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:28:14.901 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:14.890+0000 7fcd006de700 1 -- 172.21.15.42:0/2865578000 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccf805d100 con 0x7fccf8115ba0 2019-12-05T06:28:14.901 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:14.894+0000 7fccfec7b700 1 --2- 172.21.15.42:0/2865578000 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fccf8115ba0 0x7fccf8113f60 secure :-1 s=THROTTLE_DONE pgs=2679 cs=0 l=1 rx=0x7fccf80a5680 tx=0x7fccf4010a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:14.902 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:14.894+0000 7fccef7fe700 1 -- 172.21.15.42:0/2865578000 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fccf40321c0 con 0x7fccf8115ba0 2019-12-05T06:28:15.066 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:15.062+0000 7fcd006de700 1 -- 172.21.15.42:0/2865578000 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fccf805b8b0 con 0x7fccf8115ba0 2019-12-05T06:28:15.066 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:15.062+0000 7fccfec7b700 1 --2- 172.21.15.42:0/2865578000 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fccf8115ba0 0x7fccf8113f60 secure :-1 s=THROTTLE_DONE pgs=2679 cs=0 l=1 rx=0x7fccf80a5680 tx=0x7fccf4010a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:15.066 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:15.062+0000 7fccef7fe700 1 -- 172.21.15.42:0/2865578000 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 318 v318) v1 ==== 95+0+2117 (secure 0 0 0) 0x7fccf4032420 con 0x7fccf8115ba0 2019-12-05T06:28:15.067 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:28:15.067 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":318,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":11142,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/3155851600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3155851600},{"type":"v1","addr":"172.21.15.40:6833","nonce":3155851600}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":318}],"filesystems":[{"mdsmap":{"epoch":317,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:26:46.246866+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":106,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11221},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11221":{"gid":11221,"name":"a","rank":0,"incarnation":314,"state":"up:active","state_seq":34,"addr":"172.21.15.42:6835/3427104185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3427104185},{"type":"v1","addr":"172.21.15.42:6835","nonce":3427104185}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:28:15.067 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 318 2019-12-05T06:28:15.069 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:15.066+0000 7fcd006de700 1 -- 172.21.15.42:0/2865578000 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fcce003adb0 msgr2=0x7fcce003d2c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:15.069 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:15.066+0000 7fcd006de700 1 --2- 172.21.15.42:0/2865578000 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fcce003adb0 0x7fcce003d2c0 crc :-1 s=READY pgs=1120 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:15.070 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:15.066+0000 7fcd006de700 1 -- 172.21.15.42:0/2865578000 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fccf8115ba0 msgr2=0x7fccf8113f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:15.070 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:15.066+0000 7fcd006de700 1 --2- 172.21.15.42:0/2865578000 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fccf8115ba0 0x7fccf8113f60 secure :-1 s=READY pgs=2679 cs=0 l=1 rx=0x7fccf80a5680 tx=0x7fccf4010a20).stop 2019-12-05T06:28:15.070 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:15.066+0000 7fcd006de700 1 -- 172.21.15.42:0/2865578000 shutdown_connections 2019-12-05T06:28:15.070 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:15.066+0000 7fcd006de700 1 --2- 172.21.15.42:0/2865578000 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fcce003adb0 0x7fcce003d2c0 unknown :-1 s=CLOSED pgs=1120 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:15.070 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:15.066+0000 7fcd006de700 1 --2- 172.21.15.42:0/2865578000 >> v2:172.21.15.40:3300/0 conn(0x7fccf807df90 0x7fccf8113510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:15.070 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:15.066+0000 7fcd006de700 1 --2- 172.21.15.42:0/2865578000 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fccf8115ba0 0x7fccf8113f60 unknown :-1 s=CLOSED pgs=2679 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:15.070 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:15.066+0000 7fcd006de700 1 --2- 172.21.15.42:0/2865578000 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fccf807db60 0x7fccf8112fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:15.071 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:15.066+0000 7fcd006de700 1 -- 172.21.15.42:0/2865578000 shutdown_connections 2019-12-05T06:28:15.071 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:15.066+0000 7fcd006de700 1 -- 172.21.15.42:0/2865578000 wait complete. 2019-12-05T06:28:15.071 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:15.066+0000 7fcd006de700 1 -- 172.21.15.42:0/2865578000 >> 172.21.15.42:0/2865578000 conn(0x7fccf805a180 msgr2=0x7fccf809f210 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:28:15.080 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:28:17.081 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:28:17.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:17.178+0000 7f501c6ba700 1 Processor -- start 2019-12-05T06:28:17.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:17.178+0000 7f501c6ba700 1 -- start start 2019-12-05T06:28:17.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:17.182+0000 7f501c6ba700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f50140a7740 0x7f501408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:28:17.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:17.182+0000 7f501c6ba700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f501407e720 0x7f501407eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:28:17.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:17.182+0000 7f501c6ba700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f501409f2e0 0x7f501409f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:28:17.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:17.182+0000 7f501c6ba700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50140a0b00 con 0x7f50140a7740 2019-12-05T06:28:17.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:17.182+0000 7f501c6ba700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f501408e5b0 con 0x7f501407e720 2019-12-05T06:28:17.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:17.182+0000 7f501c6ba700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f501408e850 con 0x7f501409f2e0 2019-12-05T06:28:17.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:17.182+0000 7f501a456700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f501407e720 0x7f501407eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:28:17.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:17.182+0000 7f501b458700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f501409f2e0 0x7f501409f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:28:17.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:17.182+0000 7f501a456700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f501407e720 0x7f501407eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:55736/0 (socket says 172.21.15.42:55736) 2019-12-05T06:28:17.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:17.182+0000 7f501a456700 1 -- 172.21.15.42:0/3588132361 learned_addr learned my addr 172.21.15.42:0/3588132361 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:28:17.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:17.182+0000 7f501ac57700 1 --2- 172.21.15.42:0/3588132361 >> v2:172.21.15.40:3300/0 conn(0x7f50140a7740 0x7f501408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:28:17.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:17.182+0000 7f501a456700 1 -- 172.21.15.42:0/3588132361 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f501409f2e0 msgr2=0x7f501409f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:28:17.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:17.182+0000 7f501a456700 1 --2- 172.21.15.42:0/3588132361 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f501409f2e0 0x7f501409f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:28:17.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:17.182+0000 7f501a456700 1 -- 172.21.15.42:0/3588132361 >> v2:172.21.15.40:3300/0 conn(0x7f50140a7740 msgr2=0x7f501408df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:28:17.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:17.182+0000 7f501a456700 1 --2- 172.21.15.42:0/3588132361 >> v2:172.21.15.40:3300/0 conn(0x7f50140a7740 0x7f501408df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:28:17.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:17.182+0000 7f501a456700 1 -- 172.21.15.42:0/3588132361 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f501407f0a0 con 0x7f501407e720 2019-12-05T06:28:17.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:17.182+0000 7f501b458700 1 --2- 172.21.15.42:0/3588132361 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f501409f2e0 0x7f501409f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:28:17.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:17.182+0000 7f501ac57700 1 --2- 172.21.15.42:0/3588132361 >> v2:172.21.15.40:3300/0 conn(0x7f50140a7740 0x7f501408df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:28:17.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:17.182+0000 7f501a456700 1 --2- 172.21.15.42:0/3588132361 >> v2:172.21.15.42:3300/0 conn(0x7f501407e720 0x7f501407eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f50140a0f50 tx=0x7f500800fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:17.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:17.182+0000 7f501a456700 1 --2- 172.21.15.42:0/3588132361 >> v2:172.21.15.42:3300/0 conn(0x7f501407e720 0x7f501407eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f50140a0f50 tx=0x7f500800fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:17.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:17.182+0000 7f501a456700 1 --2- 172.21.15.42:0/3588132361 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f501407e720 0x7f501407eb50 secure :-1 s=READY pgs=2680 cs=0 l=1 rx=0x7f50140a0f50 tx=0x7f500800fdb0).ready entity=mon.1 client_cookie=abd8548a5fce0cff server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:28:17.193 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:17.182+0000 7f501a456700 1 --2- 172.21.15.42:0/3588132361 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f501407e720 0x7f501407eb50 secure :-1 s=THROTTLE_DONE pgs=2680 cs=0 l=1 rx=0x7f50140a0f50 tx=0x7f500800fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:17.193 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:17.182+0000 7f501a456700 1 --2- 172.21.15.42:0/3588132361 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f501407e720 0x7f501407eb50 secure :-1 s=THROTTLE_DONE pgs=2680 cs=0 l=1 rx=0x7f50140a0f50 tx=0x7f500800fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:17.193 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:17.182+0000 7f5019c55700 1 -- 172.21.15.42:0/3588132361 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5008008c90 con 0x7f501407e720 2019-12-05T06:28:17.193 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:17.182+0000 7f501a456700 1 --2- 172.21.15.42:0/3588132361 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f501407e720 0x7f501407eb50 secure :-1 s=THROTTLE_DONE pgs=2680 cs=0 l=1 rx=0x7f50140a0f50 tx=0x7f500800fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:17.193 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:17.182+0000 7f5019c55700 1 -- 172.21.15.42:0/3588132361 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50080088b0 con 0x7f501407e720 2019-12-05T06:28:17.193 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:17.182+0000 7f5019c55700 1 -- 172.21.15.42:0/3588132361 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5008011b50 con 0x7f501407e720 2019-12-05T06:28:17.193 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:17.182+0000 7f501c6ba700 1 -- 172.21.15.42:0/3588132361 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f501407e720 msgr2=0x7f501407eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:17.194 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:17.182+0000 7f501c6ba700 1 --2- 172.21.15.42:0/3588132361 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f501407e720 0x7f501407eb50 secure :-1 s=READY pgs=2680 cs=0 l=1 rx=0x7f50140a0f50 tx=0x7f500800fdb0).stop 2019-12-05T06:28:17.194 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:17.182+0000 7f501c6ba700 1 -- 172.21.15.42:0/3588132361 shutdown_connections 2019-12-05T06:28:17.194 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:17.182+0000 7f501c6ba700 1 --2- 172.21.15.42:0/3588132361 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f501409f2e0 0x7f501409f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:28:17.194 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:17.182+0000 7f501c6ba700 1 --2- 172.21.15.42:0/3588132361 >> v2:172.21.15.40:3300/0 conn(0x7f50140a7740 0x7f501408df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:28:17.194 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:17.182+0000 7f501c6ba700 1 --2- 172.21.15.42:0/3588132361 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f501407e720 0x7f501407eb50 unknown :-1 s=CLOSED pgs=2680 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:17.194 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:17.182+0000 7f501c6ba700 1 -- 172.21.15.42:0/3588132361 shutdown_connections 2019-12-05T06:28:17.194 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:17.182+0000 7f501c6ba700 1 -- 172.21.15.42:0/3588132361 wait complete. 2019-12-05T06:28:17.195 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:17.182+0000 7f501c6ba700 1 -- 172.21.15.42:0/3588132361 >> 172.21.15.42:0/3588132361 conn(0x7f50140aabe0 msgr2=0x7f50140a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:28:17.200 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:17.186+0000 7f501c6ba700 1 Processor -- start 2019-12-05T06:28:17.200 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:17.186+0000 7f501c6ba700 1 -- start start 2019-12-05T06:28:17.201 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:17.186+0000 7f501c6ba700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5014142870 0x7f501411aa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:28:17.201 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:17.186+0000 7f501c6ba700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f50140a4c60 0x7f501407b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:28:17.201 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:17.186+0000 7f501c6ba700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50140a5d80 con 0x7f501407bf00 2019-12-05T06:28:17.201 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:17.186+0000 7f501c6ba700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f501411b0b0 con 0x7f50140a4c60 2019-12-05T06:28:17.201 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:17.186+0000 7f501c6ba700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f501411c7c0 con 0x7f5014142870 2019-12-05T06:28:17.204 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:17.186+0000 7f501b458700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f501407bf00 0x7f50140a5580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:55782/0 (socket says 172.21.15.42:55782) 2019-12-05T06:28:17.204 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:17.186+0000 7f501ac57700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5014142870 0x7f501411aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:28:17.204 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:17.186+0000 7f501b458700 1 -- 172.21.15.42:0/1384195834 learned_addr learned my addr 172.21.15.42:0/1384195834 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:28:17.205 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:17.186+0000 7f501a456700 1 --2- 172.21.15.42:0/1384195834 >> v2:172.21.15.40:3300/0 conn(0x7f50140a4c60 0x7f501407b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:28:17.205 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:17.186+0000 7f50137fe700 1 -- 172.21.15.42:0/1384195834 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 570175689 0 0) 0x7f50140a5d80 con 0x7f501407bf00 2019-12-05T06:28:17.205 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:17.186+0000 7f50137fe700 1 -- 172.21.15.42:0/1384195834 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ff40030b0 con 0x7f501407bf00 2019-12-05T06:28:17.205 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:17.186+0000 7f501ac57700 1 -- 172.21.15.42:0/1384195834 >> v2:172.21.15.40:3300/0 conn(0x7f50140a4c60 msgr2=0x7f501407b9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:17.205 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:17.186+0000 7f501ac57700 1 --2- 172.21.15.42:0/1384195834 >> v2:172.21.15.40:3300/0 conn(0x7f50140a4c60 0x7f501407b9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:17.206 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:17.186+0000 7f501ac57700 1 -- 172.21.15.42:0/1384195834 >> v1:172.21.15.42:6789/0 conn(0x7f501407bf00 legacy=0x7f50140a5580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:17.206 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:17.186+0000 7f501ac57700 1 -- 172.21.15.42:0/1384195834 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50141367b0 con 0x7f5014142870 2019-12-05T06:28:17.206 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:17.186+0000 7f501ac57700 1 --2- 172.21.15.42:0/1384195834 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5014142870 0x7f501411aa80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f501411a720 tx=0x7f500c010fd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:17.206 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:17.190+0000 7f501ac57700 1 --2- 172.21.15.42:0/1384195834 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5014142870 0x7f501411aa80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f501411a720 tx=0x7f500c010fd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:17.206 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:17.190+0000 7f501ac57700 1 --2- 172.21.15.42:0/1384195834 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5014142870 0x7f501411aa80 secure :-1 s=READY pgs=588 cs=0 l=1 rx=0x7f501411a720 tx=0x7f500c010fd0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:28:17.206 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:17.190+0000 7f501ac57700 1 --2- 172.21.15.42:0/1384195834 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5014142870 0x7f501411aa80 secure :-1 s=THROTTLE_DONE pgs=588 cs=0 l=1 rx=0x7f501411a720 tx=0x7f500c010fd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:17.207 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:17.190+0000 7f50137fe700 1 -- 172.21.15.42:0/1384195834 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f500c012950 con 0x7f5014142870 2019-12-05T06:28:17.207 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:17.190+0000 7f501ac57700 1 --2- 172.21.15.42:0/1384195834 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5014142870 0x7f501411aa80 secure :-1 s=THROTTLE_DONE pgs=588 cs=0 l=1 rx=0x7f501411a720 tx=0x7f500c010fd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:17.207 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:17.190+0000 7f501c6ba700 1 -- 172.21.15.42:0/1384195834 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50141357a0 con 0x7f5014142870 2019-12-05T06:28:17.207 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:17.190+0000 7f50137fe700 1 -- 172.21.15.42:0/1384195834 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f500c00e160 con 0x7f5014142870 2019-12-05T06:28:17.207 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:17.190+0000 7f501ac57700 1 --2- 172.21.15.42:0/1384195834 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5014142870 0x7f501411aa80 secure :-1 s=THROTTLE_DONE pgs=588 cs=0 l=1 rx=0x7f501411a720 tx=0x7f500c010fd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:17.208 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:17.190+0000 7f50137fe700 1 -- 172.21.15.42:0/1384195834 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f500c0058c0 con 0x7f5014142870 2019-12-05T06:28:17.208 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:17.190+0000 7f501c6ba700 1 -- 172.21.15.42:0/1384195834 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5014135d50 con 0x7f5014142870 2019-12-05T06:28:17.208 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:17.190+0000 7f501ac57700 1 -- 172.21.15.42:0/1384195834 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5014142870 msgr2=0x7f501411aa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2019-12-05T06:28:17.208 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:17.190+0000 7f501ac57700 1 -- 172.21.15.42:0/1384195834 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5014142870 msgr2=0x7f501411aa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2019-12-05T06:28:17.208 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:17.190+0000 7f501ac57700 1 --2- 172.21.15.42:0/1384195834 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5014142870 0x7f501411aa80 secure :-1 s=READY pgs=588 cs=0 l=1 rx=0x7f501411a720 tx=0x7f500c010fd0).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-12-05T06:28:17.208 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:17.190+0000 7f501ac57700 1 --2- 172.21.15.42:0/1384195834 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5014142870 0x7f501411aa80 secure :-1 s=READY pgs=588 cs=0 l=1 rx=0x7f501411a720 tx=0x7f500c010fd0).stop 2019-12-05T06:28:17.209 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:17.190+0000 7f50137fe700 1 -- 172.21.15.42:0/1384195834 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5014142870 msgr2=0x7f501411aa80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2019-12-05T06:28:17.209 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:17.190+0000 7f50137fe700 1 --2- 172.21.15.42:0/1384195834 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5014142870 0x7f501411aa80 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:17.209 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:17.190+0000 7f50137fe700 1 --2- 172.21.15.42:0/1384195834 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4ff40033c0 0x7f4ff40037f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:28:17.209 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:17.190+0000 7f50137fe700 1 --2- 172.21.15.42:0/1384195834 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4ff4003d40 0x7f4ff4004420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:28:17.209 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:17.190+0000 7f50137fe700 1 --2- 172.21.15.42:0/1384195834 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ff4004970 0x7f4ff4004da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:28:17.210 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:17.190+0000 7f50137fe700 1 -- 172.21.15.42:0/1384195834 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ff400b700 con 0x7f4ff4004970 2019-12-05T06:28:17.210 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:17.190+0000 7f50137fe700 1 -- 172.21.15.42:0/1384195834 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ff400b8f0 con 0x7f4ff40033c0 2019-12-05T06:28:17.210 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:17.190+0000 7f50137fe700 1 -- 172.21.15.42:0/1384195834 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ff400bb30 con 0x7f4ff4003d40 2019-12-05T06:28:17.210 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:17.190+0000 7f501ac57700 1 --2- 172.21.15.42:0/1384195834 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4ff40033c0 0x7f4ff40037f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:28:17.210 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:17.190+0000 7f501a456700 1 --2- 172.21.15.42:0/1384195834 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4ff4003d40 0x7f4ff4004420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:28:17.211 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:17.190+0000 7f501b458700 1 --2- 172.21.15.42:0/1384195834 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ff4004970 0x7f4ff4004da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:28:17.211 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:17.190+0000 7f501ac57700 1 -- 172.21.15.42:0/1384195834 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4ff4003d40 msgr2=0x7f4ff4004420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:17.211 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:17.190+0000 7f501ac57700 1 --2- 172.21.15.42:0/1384195834 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4ff4003d40 0x7f4ff4004420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:17.211 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:17.190+0000 7f501ac57700 1 -- 172.21.15.42:0/1384195834 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ff4004970 msgr2=0x7f4ff4004da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:17.211 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:17.190+0000 7f501ac57700 1 --2- 172.21.15.42:0/1384195834 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ff4004970 0x7f4ff4004da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:17.212 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:17.190+0000 7f501ac57700 1 -- 172.21.15.42:0/1384195834 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+,osdmap=0}) v3 -- 0x7f4ff400bef0 con 0x7f4ff40033c0 2019-12-05T06:28:17.212 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:17.190+0000 7f501ac57700 1 --2- 172.21.15.42:0/1384195834 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4ff40033c0 0x7f4ff40037f0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f501411a720 tx=0x7f500c007180).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:17.212 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:17.190+0000 7f501ac57700 1 --2- 172.21.15.42:0/1384195834 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4ff40033c0 0x7f4ff40037f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f501411a720 tx=0x7f500c007180).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:17.212 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:17.190+0000 7f501ac57700 1 --2- 172.21.15.42:0/1384195834 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4ff40033c0 0x7f4ff40037f0 secure :-1 s=READY pgs=2682 cs=0 l=1 rx=0x7f501411a720 tx=0x7f500c007180).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:28:17.212 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:17.190+0000 7f501ac57700 1 --2- 172.21.15.42:0/1384195834 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4ff40033c0 0x7f4ff40037f0 secure :-1 s=THROTTLE_DONE pgs=2682 cs=0 l=1 rx=0x7f501411a720 tx=0x7f500c007180).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:17.213 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:17.190+0000 7f50137fe700 1 -- 172.21.15.42:0/1384195834 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f500c00e5b0 con 0x7f4ff40033c0 2019-12-05T06:28:17.213 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:17.190+0000 7f501ac57700 1 --2- 172.21.15.42:0/1384195834 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4ff40033c0 0x7f4ff40037f0 secure :-1 s=THROTTLE_DONE pgs=2682 cs=0 l=1 rx=0x7f501411a720 tx=0x7f500c007180).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:17.213 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:17.190+0000 7f50137fe700 1 -- 172.21.15.42:0/1384195834 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f500c012b90 con 0x7f4ff40033c0 2019-12-05T06:28:17.213 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:17.190+0000 7f501ac57700 1 --2- 172.21.15.42:0/1384195834 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4ff40033c0 0x7f4ff40037f0 secure :-1 s=THROTTLE_DONE pgs=2682 cs=0 l=1 rx=0x7f501411a720 tx=0x7f500c007180).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:17.213 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:17.190+0000 7f50137fe700 1 -- 172.21.15.42:0/1384195834 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f500c00e7b0 con 0x7f4ff40033c0 2019-12-05T06:28:17.214 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:17.194+0000 7f501ac57700 1 --2- 172.21.15.42:0/1384195834 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4ff40033c0 0x7f4ff40037f0 secure :-1 s=THROTTLE_DONE pgs=2682 cs=0 l=1 rx=0x7f501411a720 tx=0x7f500c007180).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:17.214 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:17.194+0000 7f50137fe700 1 --2- 172.21.15.42:0/1384195834 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4ff4043dc0 0x7f4ff40462d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:28:17.214 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:17.194+0000 7f50137fe700 1 -- 172.21.15.42:0/1384195834 <== mon.1 v2:172.21.15.42:3300/0 4 ==== osd_map(106..106 src has 1..106) v4 ==== 10933+0+0 (secure 0 0 0) 0x7f500c04c9e0 con 0x7f4ff40033c0 2019-12-05T06:28:17.214 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:17.194+0000 7f501a456700 1 --2- 172.21.15.42:0/1384195834 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4ff4043dc0 0x7f4ff40462d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:28:17.214 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:17.194+0000 7f501a456700 1 --2- 172.21.15.42:0/1384195834 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4ff4043dc0 0x7f4ff40462d0 crc :-1 s=READY pgs=1121 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:28:17.215 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:17.194+0000 7f501c6ba700 1 -- 172.21.15.42:0/1384195834 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f501405d110 con 0x7f4ff40033c0 2019-12-05T06:28:17.215 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:17.202+0000 7f501ac57700 1 --2- 172.21.15.42:0/1384195834 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4ff40033c0 0x7f4ff40037f0 secure :-1 s=THROTTLE_DONE pgs=2682 cs=0 l=1 rx=0x7f501411a720 tx=0x7f500c007180).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:17.216 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:17.202+0000 7f50137fe700 1 -- 172.21.15.42:0/1384195834 <== mon.1 v2:172.21.15.42:3300/0 5 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f500c033a60 con 0x7f4ff40033c0 2019-12-05T06:28:17.369 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:17.366+0000 7f501c6ba700 1 -- 172.21.15.42:0/1384195834 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f501405b8c0 con 0x7f4ff40033c0 2019-12-05T06:28:17.370 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:17.366+0000 7f501ac57700 1 --2- 172.21.15.42:0/1384195834 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4ff40033c0 0x7f4ff40037f0 secure :-1 s=THROTTLE_DONE pgs=2682 cs=0 l=1 rx=0x7f501411a720 tx=0x7f500c007180).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:17.370 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:17.366+0000 7f50137fe700 1 -- 172.21.15.42:0/1384195834 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 318 v318) v1 ==== 95+0+2117 (secure 0 0 0) 0x7f500c033cc0 con 0x7f4ff40033c0 2019-12-05T06:28:17.370 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 318 2019-12-05T06:28:17.370 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:28:17.371 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":318,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":11142,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/3155851600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3155851600},{"type":"v1","addr":"172.21.15.40:6833","nonce":3155851600}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":318}],"filesystems":[{"mdsmap":{"epoch":317,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:26:46.246866+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":106,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11221},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11221":{"gid":11221,"name":"a","rank":0,"incarnation":314,"state":"up:active","state_seq":34,"addr":"172.21.15.42:6835/3427104185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3427104185},{"type":"v1","addr":"172.21.15.42:6835","nonce":3427104185}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:28:17.375 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:17.370+0000 7f501c6ba700 1 -- 172.21.15.42:0/1384195834 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4ff4043dc0 msgr2=0x7f4ff40462d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:17.376 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:17.370+0000 7f501c6ba700 1 --2- 172.21.15.42:0/1384195834 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4ff4043dc0 0x7f4ff40462d0 crc :-1 s=READY pgs=1121 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:17.376 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:17.370+0000 7f501c6ba700 1 -- 172.21.15.42:0/1384195834 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4ff40033c0 msgr2=0x7f4ff40037f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:17.376 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:17.370+0000 7f501c6ba700 1 --2- 172.21.15.42:0/1384195834 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4ff40033c0 0x7f4ff40037f0 secure :-1 s=READY pgs=2682 cs=0 l=1 rx=0x7f501411a720 tx=0x7f500c007180).stop 2019-12-05T06:28:17.376 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:17.370+0000 7f501b458700 1 -- 172.21.15.42:0/1384195834 reap_dead start 2019-12-05T06:28:17.376 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:17.370+0000 7f501c6ba700 1 -- 172.21.15.42:0/1384195834 shutdown_connections 2019-12-05T06:28:17.376 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:17.370+0000 7f501c6ba700 1 -- 172.21.15.42:0/1384195834 shutdown_connections 2019-12-05T06:28:17.376 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:17.370+0000 7f501c6ba700 1 -- 172.21.15.42:0/1384195834 wait complete. 2019-12-05T06:28:17.377 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:17.370+0000 7f501c6ba700 1 -- 172.21.15.42:0/1384195834 >> 172.21.15.42:0/1384195834 conn(0x7f50140a7740 msgr2=0x7f501407e640 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:28:17.387 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:28:19.388 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:28:19.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:19.482+0000 7f21b3c3f700 1 Processor -- start 2019-12-05T06:28:19.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:19.482+0000 7f21b3c3f700 1 -- start start 2019-12-05T06:28:19.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:19.482+0000 7f21b3c3f700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f21ac0a7740 0x7f21ac08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:28:19.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:19.482+0000 7f21b3c3f700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f21ac07e720 0x7f21ac07eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:28:19.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:19.482+0000 7f21b3c3f700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21ac0a0710 con 0x7f21ac09f2e0 2019-12-05T06:28:19.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:19.482+0000 7f21b3c3f700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21ac0a0b00 con 0x7f21ac07e720 2019-12-05T06:28:19.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:19.482+0000 7f21b3c3f700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21ac08e5b0 con 0x7f21ac0a7740 2019-12-05T06:28:19.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:19.482+0000 7f21b21dc700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f21ac0a7740 0x7f21ac08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:28:19.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:19.482+0000 7f21b21dc700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f21ac0a7740 0x7f21ac08df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:55754/0 (socket says 172.21.15.42:55754) 2019-12-05T06:28:19.496 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:19.482+0000 7f21b29dd700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f21ac09f2e0 0x7f21ac09f710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:55792/0 (socket says 172.21.15.42:55792) 2019-12-05T06:28:19.496 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:19.482+0000 7f21b21dc700 1 -- 172.21.15.42:0/1950964867 learned_addr learned my addr 172.21.15.42:0/1950964867 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:28:19.496 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:19.482+0000 7f21b19db700 1 --2- 172.21.15.42:0/1950964867 >> v2:172.21.15.40:3300/0 conn(0x7f21ac07e720 0x7f21ac07eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:28:19.496 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:19.486+0000 7f21b21dc700 1 -- 172.21.15.42:0/1950964867 >> v2:172.21.15.40:3300/0 conn(0x7f21ac07e720 msgr2=0x7f21ac07eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:28:19.497 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:19.486+0000 7f21b21dc700 1 --2- 172.21.15.42:0/1950964867 >> v2:172.21.15.40:3300/0 conn(0x7f21ac07e720 0x7f21ac07eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:28:19.497 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:19.486+0000 7f21b21dc700 1 -- 172.21.15.42:0/1950964867 >> v1:172.21.15.42:6789/0 conn(0x7f21ac09f2e0 legacy=0x7f21ac09f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:19.497 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:19.486+0000 7f21b21dc700 1 -- 172.21.15.42:0/1950964867 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21ac07f0a0 con 0x7f21ac0a7740 2019-12-05T06:28:19.497 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:19.486+0000 7f21b21dc700 1 --2- 172.21.15.42:0/1950964867 >> v2:172.21.15.42:3300/0 conn(0x7f21ac0a7740 0x7f21ac08df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f21a8002f30 tx=0x7f21a800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:19.497 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:19.486+0000 7f21b21dc700 1 --2- 172.21.15.42:0/1950964867 >> v2:172.21.15.42:3300/0 conn(0x7f21ac0a7740 0x7f21ac08df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f21a8002f30 tx=0x7f21a800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:19.498 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:19.486+0000 7f21b21dc700 1 --2- 172.21.15.42:0/1950964867 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f21ac0a7740 0x7f21ac08df80 secure :-1 s=READY pgs=2684 cs=0 l=1 rx=0x7f21a8002f30 tx=0x7f21a800fd30).ready entity=mon.1 client_cookie=7ce3e09835cebdef server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:28:19.498 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:19.486+0000 7f21b21dc700 1 --2- 172.21.15.42:0/1950964867 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f21ac0a7740 0x7f21ac08df80 secure :-1 s=THROTTLE_DONE pgs=2684 cs=0 l=1 rx=0x7f21a8002f30 tx=0x7f21a800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:19.498 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:19.486+0000 7f21b21dc700 1 --2- 172.21.15.42:0/1950964867 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f21ac0a7740 0x7f21ac08df80 secure :-1 s=THROTTLE_DONE pgs=2684 cs=0 l=1 rx=0x7f21a8002f30 tx=0x7f21a800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:19.498 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:19.486+0000 7f21b11da700 1 -- 172.21.15.42:0/1950964867 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f21a8011370 con 0x7f21ac0a7740 2019-12-05T06:28:19.499 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:19.486+0000 7f21b21dc700 1 --2- 172.21.15.42:0/1950964867 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f21ac0a7740 0x7f21ac08df80 secure :-1 s=THROTTLE_DONE pgs=2684 cs=0 l=1 rx=0x7f21a8002f30 tx=0x7f21a800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:19.499 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:19.486+0000 7f21b11da700 1 -- 172.21.15.42:0/1950964867 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21a8008bb0 con 0x7f21ac0a7740 2019-12-05T06:28:19.499 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:19.486+0000 7f21b11da700 1 -- 172.21.15.42:0/1950964867 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f21a8011950 con 0x7f21ac0a7740 2019-12-05T06:28:19.499 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:19.486+0000 7f21b3c3f700 1 -- 172.21.15.42:0/1950964867 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f21ac0a7740 msgr2=0x7f21ac08df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:19.499 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:19.486+0000 7f21b3c3f700 1 --2- 172.21.15.42:0/1950964867 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f21ac0a7740 0x7f21ac08df80 secure :-1 s=READY pgs=2684 cs=0 l=1 rx=0x7f21a8002f30 tx=0x7f21a800fd30).stop 2019-12-05T06:28:19.499 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:19.486+0000 7f21b3c3f700 1 -- 172.21.15.42:0/1950964867 shutdown_connections 2019-12-05T06:28:19.500 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:19.486+0000 7f21b3c3f700 1 --2- 172.21.15.42:0/1950964867 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f21ac0a7740 0x7f21ac08df80 unknown :-1 s=CLOSED pgs=2684 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:19.500 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:19.486+0000 7f21b3c3f700 1 --2- 172.21.15.42:0/1950964867 >> v2:172.21.15.40:3300/0 conn(0x7f21ac07e720 0x7f21ac07eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:28:19.500 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:19.486+0000 7f21b3c3f700 1 -- 172.21.15.42:0/1950964867 shutdown_connections 2019-12-05T06:28:19.500 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:19.486+0000 7f21b3c3f700 1 -- 172.21.15.42:0/1950964867 wait complete. 2019-12-05T06:28:19.500 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:19.486+0000 7f21b3c3f700 1 -- 172.21.15.42:0/1950964867 >> 172.21.15.42:0/1950964867 conn(0x7f21ac0aabe0 msgr2=0x7f21ac0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:28:19.500 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:19.490+0000 7f21b3c3f700 1 Processor -- start 2019-12-05T06:28:19.501 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:19.490+0000 7f21b3c3f700 1 -- start start 2019-12-05T06:28:19.521 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:19.490+0000 7f21b3c3f700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f21ac07ad20 0x7f21ac07b150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:28:19.521 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:19.490+0000 7f21b3c3f700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f21ac117e80 0x7f21ac1182b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:28:19.522 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:19.490+0000 7f21b3c3f700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21ac118f80 con 0x7f21ac116500 2019-12-05T06:28:19.522 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:19.490+0000 7f21b3c3f700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21ac117940 con 0x7f21ac07ad20 2019-12-05T06:28:19.522 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:19.490+0000 7f21b3c3f700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21ac117be0 con 0x7f21ac117e80 2019-12-05T06:28:19.522 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:19.490+0000 7f21b29dd700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f21ac117e80 0x7f21ac1182b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:28:19.522 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:19.490+0000 7f21b29dd700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f21ac117e80 0x7f21ac1182b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:55764/0 (socket says 172.21.15.42:55764) 2019-12-05T06:28:19.522 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:19.490+0000 7f21b29dd700 1 -- 172.21.15.42:0/1404259281 learned_addr learned my addr 172.21.15.42:0/1404259281 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:28:19.523 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:19.490+0000 7f21b19db700 1 --2- 172.21.15.42:0/1404259281 >> v2:172.21.15.40:3300/0 conn(0x7f21ac07ad20 0x7f21ac07b150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:28:19.523 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:19.494+0000 7f21a2ffd700 1 -- 172.21.15.42:0/1404259281 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1195151507 0 0) 0x7f21ac118f80 con 0x7f21ac116500 2019-12-05T06:28:19.523 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:19.494+0000 7f21a2ffd700 1 -- 172.21.15.42:0/1404259281 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21940030b0 con 0x7f21ac116500 2019-12-05T06:28:19.523 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:19.494+0000 7f21b29dd700 1 -- 172.21.15.42:0/1404259281 >> v2:172.21.15.40:3300/0 conn(0x7f21ac07ad20 msgr2=0x7f21ac07b150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:19.524 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:19.494+0000 7f21b29dd700 1 --2- 172.21.15.42:0/1404259281 >> v2:172.21.15.40:3300/0 conn(0x7f21ac07ad20 0x7f21ac07b150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:19.524 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:19.494+0000 7f21b29dd700 1 -- 172.21.15.42:0/1404259281 >> v1:172.21.15.40:6789/0 conn(0x7f21ac116500 legacy=0x7f21ac121090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:19.524 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:19.494+0000 7f21b29dd700 1 -- 172.21.15.42:0/1404259281 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21ac118800 con 0x7f21ac117e80 2019-12-05T06:28:19.524 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:19.494+0000 7f21b29dd700 1 --2- 172.21.15.42:0/1404259281 >> v2:172.21.15.42:3300/0 conn(0x7f21ac117e80 0x7f21ac1182b0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f21a4001210 tx=0x7f21a4010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:19.524 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:19.494+0000 7f21b29dd700 1 --2- 172.21.15.42:0/1404259281 >> v2:172.21.15.42:3300/0 conn(0x7f21ac117e80 0x7f21ac1182b0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f21a4001210 tx=0x7f21a4010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:19.524 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:19.494+0000 7f21b29dd700 1 --2- 172.21.15.42:0/1404259281 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f21ac117e80 0x7f21ac1182b0 secure :-1 s=READY pgs=2685 cs=0 l=1 rx=0x7f21a4001210 tx=0x7f21a4010b50).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:28:19.525 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:19.494+0000 7f21b29dd700 1 --2- 172.21.15.42:0/1404259281 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f21ac117e80 0x7f21ac1182b0 secure :-1 s=THROTTLE_DONE pgs=2685 cs=0 l=1 rx=0x7f21a4001210 tx=0x7f21a4010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:19.525 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:19.494+0000 7f21a2ffd700 1 -- 172.21.15.42:0/1404259281 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f21a400d0c0 con 0x7f21ac117e80 2019-12-05T06:28:19.525 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:19.494+0000 7f21b29dd700 1 --2- 172.21.15.42:0/1404259281 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f21ac117e80 0x7f21ac1182b0 secure :-1 s=THROTTLE_DONE pgs=2685 cs=0 l=1 rx=0x7f21a4001210 tx=0x7f21a4010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:19.525 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:19.494+0000 7f21b29dd700 1 --2- 172.21.15.42:0/1404259281 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f21ac117e80 0x7f21ac1182b0 secure :-1 s=THROTTLE_DONE pgs=2685 cs=0 l=1 rx=0x7f21a4001210 tx=0x7f21a4010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:19.525 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:19.494+0000 7f21a2ffd700 1 -- 172.21.15.42:0/1404259281 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21a40114a0 con 0x7f21ac117e80 2019-12-05T06:28:19.525 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:19.494+0000 7f21b3c3f700 1 -- 172.21.15.42:0/1404259281 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21ac13aad0 con 0x7f21ac117e80 2019-12-05T06:28:19.525 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:19.494+0000 7f21a2ffd700 1 -- 172.21.15.42:0/1404259281 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f21a400d6a0 con 0x7f21ac117e80 2019-12-05T06:28:19.526 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:19.494+0000 7f21b3c3f700 1 -- 172.21.15.42:0/1404259281 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21ac13b080 con 0x7f21ac117e80 2019-12-05T06:28:19.526 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:19.494+0000 7f21b29dd700 1 --2- 172.21.15.42:0/1404259281 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f21ac117e80 0x7f21ac1182b0 secure :-1 s=THROTTLE_DONE pgs=2685 cs=0 l=1 rx=0x7f21a4001210 tx=0x7f21a4010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:19.526 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:19.494+0000 7f21a2ffd700 1 -- 172.21.15.42:0/1404259281 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f21a400d970 con 0x7f21ac117e80 2019-12-05T06:28:19.526 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:19.494+0000 7f21b29dd700 1 --2- 172.21.15.42:0/1404259281 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f21ac117e80 0x7f21ac1182b0 secure :-1 s=THROTTLE_DONE pgs=2685 cs=0 l=1 rx=0x7f21a4001210 tx=0x7f21a4010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:19.526 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:19.494+0000 7f21a2ffd700 1 --2- 172.21.15.42:0/1404259281 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f219403b280 0x7f219403d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:28:19.526 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:19.494+0000 7f21a2ffd700 1 -- 172.21.15.42:0/1404259281 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 10933+0+0 (secure 0 0 0) 0x7f21a404b080 con 0x7f21ac117e80 2019-12-05T06:28:19.526 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:19.494+0000 7f21b21dc700 1 --2- 172.21.15.42:0/1404259281 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f219403b280 0x7f219403d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:28:19.527 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:19.498+0000 7f21b21dc700 1 --2- 172.21.15.42:0/1404259281 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f219403b280 0x7f219403d790 crc :-1 s=READY pgs=1122 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:28:19.527 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:19.498+0000 7f21b3c3f700 1 -- 172.21.15.42:0/1404259281 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21ac05d110 con 0x7f21ac117e80 2019-12-05T06:28:19.527 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:19.506+0000 7f21b29dd700 1 --2- 172.21.15.42:0/1404259281 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f21ac117e80 0x7f21ac1182b0 secure :-1 s=THROTTLE_DONE pgs=2685 cs=0 l=1 rx=0x7f21a4001210 tx=0x7f21a4010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:19.527 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:19.506+0000 7f21a2ffd700 1 -- 172.21.15.42:0/1404259281 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f21a4032100 con 0x7f21ac117e80 2019-12-05T06:28:19.683 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:19.678+0000 7f21b3c3f700 1 -- 172.21.15.42:0/1404259281 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f21ac121890 con 0x7f21ac117e80 2019-12-05T06:28:19.684 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:19.682+0000 7f21b29dd700 1 --2- 172.21.15.42:0/1404259281 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f21ac117e80 0x7f21ac1182b0 secure :-1 s=THROTTLE_DONE pgs=2685 cs=0 l=1 rx=0x7f21a4001210 tx=0x7f21a4010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:19.684 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:19.682+0000 7f21a2ffd700 1 -- 172.21.15.42:0/1404259281 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 318 v318) v1 ==== 95+0+2117 (secure 0 0 0) 0x7f21a4032360 con 0x7f21ac117e80 2019-12-05T06:28:19.686 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 318 2019-12-05T06:28:19.687 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:28:19.687 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":318,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":11142,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/3155851600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3155851600},{"type":"v1","addr":"172.21.15.40:6833","nonce":3155851600}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":318}],"filesystems":[{"mdsmap":{"epoch":317,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:26:46.246866+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":106,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11221},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11221":{"gid":11221,"name":"a","rank":0,"incarnation":314,"state":"up:active","state_seq":34,"addr":"172.21.15.42:6835/3427104185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3427104185},{"type":"v1","addr":"172.21.15.42:6835","nonce":3427104185}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:28:19.689 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:19.686+0000 7f21b3c3f700 1 -- 172.21.15.42:0/1404259281 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f219403b280 msgr2=0x7f219403d790 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:19.690 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:19.686+0000 7f21b3c3f700 1 --2- 172.21.15.42:0/1404259281 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f219403b280 0x7f219403d790 crc :-1 s=READY pgs=1122 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:19.690 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:19.686+0000 7f21b3c3f700 1 -- 172.21.15.42:0/1404259281 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f21ac117e80 msgr2=0x7f21ac1182b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:19.690 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:19.686+0000 7f21b3c3f700 1 --2- 172.21.15.42:0/1404259281 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f21ac117e80 0x7f21ac1182b0 secure :-1 s=READY pgs=2685 cs=0 l=1 rx=0x7f21a4001210 tx=0x7f21a4010b50).stop 2019-12-05T06:28:19.690 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:19.686+0000 7f21b3c3f700 1 -- 172.21.15.42:0/1404259281 shutdown_connections 2019-12-05T06:28:19.690 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:19.686+0000 7f21b3c3f700 1 --2- 172.21.15.42:0/1404259281 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f219403b280 0x7f219403d790 unknown :-1 s=CLOSED pgs=1122 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:19.690 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:19.686+0000 7f21b3c3f700 1 --2- 172.21.15.42:0/1404259281 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f21ac117e80 0x7f21ac1182b0 unknown :-1 s=CLOSED pgs=2685 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:19.691 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:19.686+0000 7f21b3c3f700 1 --2- 172.21.15.42:0/1404259281 >> v2:172.21.15.40:3300/0 conn(0x7f21ac07ad20 0x7f21ac07b150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:19.691 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:19.686+0000 7f21b3c3f700 1 -- 172.21.15.42:0/1404259281 shutdown_connections 2019-12-05T06:28:19.691 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:19.686+0000 7f21b3c3f700 1 -- 172.21.15.42:0/1404259281 wait complete. 2019-12-05T06:28:19.691 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:19.686+0000 7f21b3c3f700 1 -- 172.21.15.42:0/1404259281 >> 172.21.15.42:0/1404259281 conn(0x7f21ac0a97e0 msgr2=0x7f21ac09f580 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:28:19.700 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:28:21.701 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:28:21.820 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:21.802+0000 7f2667b75700 1 Processor -- start 2019-12-05T06:28:21.821 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:21.802+0000 7f2667b75700 1 -- start start 2019-12-05T06:28:21.821 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:21.802+0000 7f2667b75700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f26600a7730 0x7f266008df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:28:21.822 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:21.802+0000 7f2667b75700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f266007e710 0x7f266007eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:28:21.822 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:21.802+0000 7f2667b75700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26600a0700 con 0x7f266009f2d0 2019-12-05T06:28:21.822 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:21.802+0000 7f2667b75700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26600a0af0 con 0x7f266007e710 2019-12-05T06:28:21.823 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:21.802+0000 7f2667b75700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f266008e5a0 con 0x7f26600a7730 2019-12-05T06:28:21.823 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:21.802+0000 7f2665911700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f266007e710 0x7f266007eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:28:21.823 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:21.802+0000 7f2665911700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f266007e710 0x7f266007eb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:55774/0 (socket says 172.21.15.42:55774) 2019-12-05T06:28:21.823 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:21.802+0000 7f2665911700 1 -- 172.21.15.42:0/2740863094 learned_addr learned my addr 172.21.15.42:0/2740863094 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:28:21.825 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:21.802+0000 7f2666112700 1 --2- 172.21.15.42:0/2740863094 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f26600a7730 0x7f266008df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:28:21.825 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:21.806+0000 7f2665911700 1 -- 172.21.15.42:0/2740863094 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f26600a7730 msgr2=0x7f266008df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:28:21.825 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:21.806+0000 7f2665911700 1 --2- 172.21.15.42:0/2740863094 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f26600a7730 0x7f266008df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:28:21.825 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:21.806+0000 7f2665911700 1 -- 172.21.15.42:0/2740863094 >> v1:172.21.15.40:6789/0 conn(0x7f266009f2d0 legacy=0x7f266009f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:21.826 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:21.806+0000 7f2665911700 1 -- 172.21.15.42:0/2740863094 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f266007f090 con 0x7f266007e710 2019-12-05T06:28:21.826 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:21.806+0000 7f2666112700 1 --2- 172.21.15.42:0/2740863094 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f26600a7730 0x7f266008df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:28:21.826 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:21.806+0000 7f2665911700 1 --2- 172.21.15.42:0/2740863094 >> v2:172.21.15.42:3300/0 conn(0x7f266007e710 0x7f266007eb40 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f2650008790 tx=0x7f265000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:21.826 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:21.806+0000 7f2665911700 1 --2- 172.21.15.42:0/2740863094 >> v2:172.21.15.42:3300/0 conn(0x7f266007e710 0x7f266007eb40 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f2650008790 tx=0x7f265000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:21.826 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:21.806+0000 7f2665911700 1 --2- 172.21.15.42:0/2740863094 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f266007e710 0x7f266007eb40 secure :-1 s=READY pgs=2686 cs=0 l=1 rx=0x7f2650008790 tx=0x7f265000fd30).ready entity=mon.1 client_cookie=7b819d0106d0ac38 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:28:21.830 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:21.806+0000 7f2665911700 1 --2- 172.21.15.42:0/2740863094 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f266007e710 0x7f266007eb40 secure :-1 s=THROTTLE_DONE pgs=2686 cs=0 l=1 rx=0x7f2650008790 tx=0x7f265000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:21.831 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:21.806+0000 7f2665911700 1 --2- 172.21.15.42:0/2740863094 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f266007e710 0x7f266007eb40 secure :-1 s=THROTTLE_DONE pgs=2686 cs=0 l=1 rx=0x7f2650008790 tx=0x7f265000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:21.831 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:21.806+0000 7f2665110700 1 -- 172.21.15.42:0/2740863094 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f26500113b0 con 0x7f266007e710 2019-12-05T06:28:21.831 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:21.806+0000 7f2665911700 1 --2- 172.21.15.42:0/2740863094 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f266007e710 0x7f266007eb40 secure :-1 s=THROTTLE_DONE pgs=2686 cs=0 l=1 rx=0x7f2650008790 tx=0x7f265000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:21.831 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:21.806+0000 7f2665110700 1 -- 172.21.15.42:0/2740863094 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2650008bb0 con 0x7f266007e710 2019-12-05T06:28:21.832 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:21.806+0000 7f2665110700 1 -- 172.21.15.42:0/2740863094 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2650011990 con 0x7f266007e710 2019-12-05T06:28:21.832 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:21.806+0000 7f2667b75700 1 -- 172.21.15.42:0/2740863094 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f266007e710 msgr2=0x7f266007eb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:21.832 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:21.806+0000 7f2667b75700 1 --2- 172.21.15.42:0/2740863094 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f266007e710 0x7f266007eb40 secure :-1 s=READY pgs=2686 cs=0 l=1 rx=0x7f2650008790 tx=0x7f265000fd30).stop 2019-12-05T06:28:21.833 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:21.806+0000 7f2667b75700 1 -- 172.21.15.42:0/2740863094 shutdown_connections 2019-12-05T06:28:21.833 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:21.806+0000 7f2667b75700 1 --2- 172.21.15.42:0/2740863094 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f26600a7730 0x7f266008df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:28:21.833 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:21.806+0000 7f2667b75700 1 --2- 172.21.15.42:0/2740863094 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f266007e710 0x7f266007eb40 unknown :-1 s=CLOSED pgs=2686 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:21.834 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:21.806+0000 7f2667b75700 1 -- 172.21.15.42:0/2740863094 shutdown_connections 2019-12-05T06:28:21.834 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:21.806+0000 7f2667b75700 1 -- 172.21.15.42:0/2740863094 wait complete. 2019-12-05T06:28:21.834 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:21.806+0000 7f2667b75700 1 -- 172.21.15.42:0/2740863094 >> 172.21.15.42:0/2740863094 conn(0x7f26600aabd0 msgr2=0x7f26600a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:28:21.834 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:21.810+0000 7f2667b75700 1 Processor -- start 2019-12-05T06:28:21.834 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:21.810+0000 7f2667b75700 1 -- start start 2019-12-05T06:28:21.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:21.810+0000 7f2667b75700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2660115600 0x7f26600636e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:28:21.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:21.810+0000 7f2667b75700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f26600656b0 0x7f2660123bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:28:21.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:21.810+0000 7f2667b75700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2660124550 con 0x7f2660124120 2019-12-05T06:28:21.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:21.810+0000 7f2667b75700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2660063e50 con 0x7f2660115600 2019-12-05T06:28:21.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:21.810+0000 7f2667b75700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26600640f0 con 0x7f26600656b0 2019-12-05T06:28:21.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:21.810+0000 7f2666112700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2660115600 0x7f26600636e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:28:21.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:21.810+0000 7f2666112700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2660115600 0x7f26600636e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:55778/0 (socket says 172.21.15.42:55778) 2019-12-05T06:28:21.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:21.810+0000 7f2666112700 1 -- 172.21.15.42:0/2077754167 learned_addr learned my addr 172.21.15.42:0/2077754167 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:28:21.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:21.810+0000 7f2665911700 1 --2- 172.21.15.42:0/2077754167 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f26600656b0 0x7f2660123bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:28:21.837 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:21.810+0000 7f2666112700 1 -- 172.21.15.42:0/2077754167 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f26600656b0 msgr2=0x7f2660123bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:21.837 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:21.810+0000 7f2666112700 1 --2- 172.21.15.42:0/2077754167 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f26600656b0 0x7f2660123bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:21.837 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:21.810+0000 7f2666112700 1 -- 172.21.15.42:0/2077754167 >> v1:172.21.15.40:6789/0 conn(0x7f2660124120 legacy=0x7f26600a5e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:21.837 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:21.810+0000 7f2656ffd700 1 -- 172.21.15.42:0/2077754167 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1747201636 0 0) 0x7f2660124550 con 0x7f2660124120 2019-12-05T06:28:21.837 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:21.810+0000 7f2666112700 1 -- 172.21.15.42:0/2077754167 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2660065ae0 con 0x7f2660115600 2019-12-05T06:28:21.837 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:21.810+0000 7f2666112700 1 --2- 172.21.15.42:0/2077754167 >> v2:172.21.15.42:3300/0 conn(0x7f2660115600 0x7f26600636e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f265c0077a0 tx=0x7f265c010a00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:21.837 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:21.810+0000 7f2666112700 1 --2- 172.21.15.42:0/2077754167 >> v2:172.21.15.42:3300/0 conn(0x7f2660115600 0x7f26600636e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f265c0077a0 tx=0x7f265c010a00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:21.838 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:21.810+0000 7f2666112700 1 --2- 172.21.15.42:0/2077754167 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2660115600 0x7f26600636e0 secure :-1 s=READY pgs=2687 cs=0 l=1 rx=0x7f265c0077a0 tx=0x7f265c010a00).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:28:21.838 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:21.810+0000 7f2666112700 1 --2- 172.21.15.42:0/2077754167 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2660115600 0x7f26600636e0 secure :-1 s=THROTTLE_DONE pgs=2687 cs=0 l=1 rx=0x7f265c0077a0 tx=0x7f265c010a00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:21.838 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:21.810+0000 7f2656ffd700 1 -- 172.21.15.42:0/2077754167 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f265c009150 con 0x7f2660115600 2019-12-05T06:28:21.841 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:21.810+0000 7f2666112700 1 --2- 172.21.15.42:0/2077754167 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2660115600 0x7f26600636e0 secure :-1 s=THROTTLE_DONE pgs=2687 cs=0 l=1 rx=0x7f265c0077a0 tx=0x7f265c010a00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:21.841 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:21.810+0000 7f2666112700 1 --2- 172.21.15.42:0/2077754167 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2660115600 0x7f26600636e0 secure :-1 s=THROTTLE_DONE pgs=2687 cs=0 l=1 rx=0x7f265c0077a0 tx=0x7f265c010a00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:21.842 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:21.810+0000 7f2656ffd700 1 -- 172.21.15.42:0/2077754167 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f265c007b30 con 0x7f2660115600 2019-12-05T06:28:21.842 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:21.810+0000 7f2667b75700 1 -- 172.21.15.42:0/2077754167 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26600a6600 con 0x7f2660115600 2019-12-05T06:28:21.842 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:21.810+0000 7f2656ffd700 1 -- 172.21.15.42:0/2077754167 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f265c009730 con 0x7f2660115600 2019-12-05T06:28:21.843 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:21.810+0000 7f2667b75700 1 -- 172.21.15.42:0/2077754167 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2660131570 con 0x7f2660115600 2019-12-05T06:28:21.843 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:21.814+0000 7f2666112700 1 --2- 172.21.15.42:0/2077754167 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2660115600 0x7f26600636e0 secure :-1 s=THROTTLE_DONE pgs=2687 cs=0 l=1 rx=0x7f265c0077a0 tx=0x7f265c010a00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:21.843 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:21.814+0000 7f2656ffd700 1 -- 172.21.15.42:0/2077754167 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f265c007170 con 0x7f2660115600 2019-12-05T06:28:21.843 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:21.814+0000 7f2666112700 1 --2- 172.21.15.42:0/2077754167 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2660115600 0x7f26600636e0 secure :-1 s=THROTTLE_DONE pgs=2687 cs=0 l=1 rx=0x7f265c0077a0 tx=0x7f265c010a00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:21.843 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:21.814+0000 7f2656ffd700 1 --2- 172.21.15.42:0/2077754167 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f264803acd0 0x7f264803d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:28:21.844 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:21.814+0000 7f2656ffd700 1 -- 172.21.15.42:0/2077754167 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 10933+0+0 (secure 0 0 0) 0x7f265c04ab80 con 0x7f2660115600 2019-12-05T06:28:21.844 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:21.814+0000 7f2665911700 1 --2- 172.21.15.42:0/2077754167 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f264803acd0 0x7f264803d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:28:21.844 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:21.814+0000 7f2665911700 1 --2- 172.21.15.42:0/2077754167 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f264803acd0 0x7f264803d1e0 crc :-1 s=READY pgs=1123 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:28:21.845 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:21.818+0000 7f2667b75700 1 -- 172.21.15.42:0/2077754167 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f266005d100 con 0x7f2660115600 2019-12-05T06:28:21.845 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:21.826+0000 7f2666112700 1 --2- 172.21.15.42:0/2077754167 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2660115600 0x7f26600636e0 secure :-1 s=THROTTLE_DONE pgs=2687 cs=0 l=1 rx=0x7f265c0077a0 tx=0x7f265c010a00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:21.845 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:21.826+0000 7f2656ffd700 1 -- 172.21.15.42:0/2077754167 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f265c031be0 con 0x7f2660115600 2019-12-05T06:28:22.003 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:21.998+0000 7f2667b75700 1 -- 172.21.15.42:0/2077754167 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f266005b8b0 con 0x7f2660115600 2019-12-05T06:28:22.005 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:22.002+0000 7f2666112700 1 --2- 172.21.15.42:0/2077754167 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2660115600 0x7f26600636e0 secure :-1 s=THROTTLE_DONE pgs=2687 cs=0 l=1 rx=0x7f265c0077a0 tx=0x7f265c010a00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:22.006 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:22.002+0000 7f2656ffd700 1 -- 172.21.15.42:0/2077754167 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 318 v318) v1 ==== 95+0+2117 (secure 0 0 0) 0x7f265c031e40 con 0x7f2660115600 2019-12-05T06:28:22.006 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 318 2019-12-05T06:28:22.006 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:28:22.006 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":318,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":11142,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/3155851600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3155851600},{"type":"v1","addr":"172.21.15.40:6833","nonce":3155851600}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":318}],"filesystems":[{"mdsmap":{"epoch":317,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:26:46.246866+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":106,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11221},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11221":{"gid":11221,"name":"a","rank":0,"incarnation":314,"state":"up:active","state_seq":34,"addr":"172.21.15.42:6835/3427104185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3427104185},{"type":"v1","addr":"172.21.15.42:6835","nonce":3427104185}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:28:22.007 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:22.002+0000 7f2667b75700 1 -- 172.21.15.42:0/2077754167 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f264803acd0 msgr2=0x7f264803d1e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:22.007 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:22.002+0000 7f2667b75700 1 --2- 172.21.15.42:0/2077754167 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f264803acd0 0x7f264803d1e0 crc :-1 s=READY pgs=1123 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:22.008 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:22.002+0000 7f2667b75700 1 -- 172.21.15.42:0/2077754167 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2660115600 msgr2=0x7f26600636e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:22.008 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:22.002+0000 7f2667b75700 1 --2- 172.21.15.42:0/2077754167 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2660115600 0x7f26600636e0 secure :-1 s=READY pgs=2687 cs=0 l=1 rx=0x7f265c0077a0 tx=0x7f265c010a00).stop 2019-12-05T06:28:22.008 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:22.002+0000 7f2667b75700 1 -- 172.21.15.42:0/2077754167 shutdown_connections 2019-12-05T06:28:22.009 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:22.002+0000 7f2667b75700 1 --2- 172.21.15.42:0/2077754167 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f264803acd0 0x7f264803d1e0 unknown :-1 s=CLOSED pgs=1123 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:22.010 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:22.002+0000 7f2667b75700 1 --2- 172.21.15.42:0/2077754167 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2660115600 0x7f26600636e0 unknown :-1 s=CLOSED pgs=2687 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:22.010 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:22.002+0000 7f2667b75700 1 --2- 172.21.15.42:0/2077754167 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f26600656b0 0x7f2660123bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:22.011 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:22.002+0000 7f2667b75700 1 -- 172.21.15.42:0/2077754167 shutdown_connections 2019-12-05T06:28:22.012 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:22.002+0000 7f2667b75700 1 -- 172.21.15.42:0/2077754167 wait complete. 2019-12-05T06:28:22.012 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:22.002+0000 7f2667b75700 1 -- 172.21.15.42:0/2077754167 >> 172.21.15.42:0/2077754167 conn(0x7f26600a9a90 msgr2=0x7f26600596f0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:28:22.017 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:28:24.019 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:28:24.126 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:24.118+0000 7fa329768700 1 Processor -- start 2019-12-05T06:28:24.127 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:24.118+0000 7fa329768700 1 -- start start 2019-12-05T06:28:24.127 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:24.118+0000 7fa329768700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa3240a7740 0x7fa32408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:28:24.127 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:24.118+0000 7fa329768700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa32409fae0 0x7fa32407eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:28:24.127 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:24.118+0000 7fa329768700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3240a0710 con 0x7fa32407e720 2019-12-05T06:28:24.127 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:24.118+0000 7fa329768700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3240a0b00 con 0x7fa32409fae0 2019-12-05T06:28:24.128 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:24.118+0000 7fa329768700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa32408e5b0 con 0x7fa3240a7740 2019-12-05T06:28:24.128 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:24.118+0000 7fa3237fe700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa3240a7740 0x7fa32408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:28:24.128 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:24.118+0000 7fa322ffd700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fa32407e720 0x7fa32409f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:37034/0 (socket says 172.21.15.42:37034) 2019-12-05T06:28:24.128 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:24.118+0000 7fa322ffd700 1 -- 172.21.15.42:0/20786596 learned_addr learned my addr 172.21.15.42:0/20786596 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:28:24.128 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:24.118+0000 7fa323fff700 1 --2- 172.21.15.42:0/20786596 >> v2:172.21.15.40:3300/0 conn(0x7fa32409fae0 0x7fa32407eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:28:24.128 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:24.118+0000 7fa323fff700 1 -- 172.21.15.42:0/20786596 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa3240a7740 msgr2=0x7fa32408df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:28:24.129 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:24.118+0000 7fa323fff700 1 --2- 172.21.15.42:0/20786596 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa3240a7740 0x7fa32408df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:28:24.129 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:24.118+0000 7fa323fff700 1 -- 172.21.15.42:0/20786596 >> v1:172.21.15.40:6789/0 conn(0x7fa32407e720 legacy=0x7fa32409f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:24.129 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:24.118+0000 7fa3227fc700 1 -- 172.21.15.42:0/20786596 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 250542816 0 0) 0x7fa3240a0710 con 0x7fa32407e720 2019-12-05T06:28:24.129 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:24.118+0000 7fa323fff700 1 -- 172.21.15.42:0/20786596 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa32407f0a0 con 0x7fa32409fae0 2019-12-05T06:28:24.129 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:24.118+0000 7fa3237fe700 1 --2- 172.21.15.42:0/20786596 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa3240a7740 0x7fa32408df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:28:24.129 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:24.118+0000 7fa323fff700 1 --2- 172.21.15.42:0/20786596 >> v2:172.21.15.40:3300/0 conn(0x7fa32409fae0 0x7fa32407eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fa31c008790 tx=0x7fa31c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:24.129 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:24.118+0000 7fa323fff700 1 --2- 172.21.15.42:0/20786596 >> v2:172.21.15.40:3300/0 conn(0x7fa32409fae0 0x7fa32407eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fa31c008790 tx=0x7fa31c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:24.130 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:24.118+0000 7fa323fff700 1 --2- 172.21.15.42:0/20786596 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa32409fae0 0x7fa32407eb50 secure :-1 s=READY pgs=2014 cs=0 l=1 rx=0x7fa31c008790 tx=0x7fa31c00fc70).ready entity=mon.0 client_cookie=db7ebdfc4dde9220 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:28:24.130 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:24.122+0000 7fa323fff700 1 --2- 172.21.15.42:0/20786596 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa32409fae0 0x7fa32407eb50 secure :-1 s=THROTTLE_DONE pgs=2014 cs=0 l=1 rx=0x7fa31c008790 tx=0x7fa31c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:24.130 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:24.122+0000 7fa3227fc700 1 -- 172.21.15.42:0/20786596 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa31c008bb0 con 0x7fa32409fae0 2019-12-05T06:28:24.130 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:24.122+0000 7fa323fff700 1 --2- 172.21.15.42:0/20786596 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa32409fae0 0x7fa32407eb50 secure :-1 s=THROTTLE_DONE pgs=2014 cs=0 l=1 rx=0x7fa31c008790 tx=0x7fa31c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:24.130 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:24.122+0000 7fa3227fc700 1 -- 172.21.15.42:0/20786596 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa31c0087d0 con 0x7fa32409fae0 2019-12-05T06:28:24.130 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:24.122+0000 7fa323fff700 1 --2- 172.21.15.42:0/20786596 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa32409fae0 0x7fa32407eb50 secure :-1 s=THROTTLE_DONE pgs=2014 cs=0 l=1 rx=0x7fa31c008790 tx=0x7fa31c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:24.131 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:24.122+0000 7fa329768700 1 -- 172.21.15.42:0/20786596 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa32409fae0 msgr2=0x7fa32407eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:24.131 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:24.122+0000 7fa329768700 1 --2- 172.21.15.42:0/20786596 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa32409fae0 0x7fa32407eb50 secure :-1 s=READY pgs=2014 cs=0 l=1 rx=0x7fa31c008790 tx=0x7fa31c00fc70).stop 2019-12-05T06:28:24.131 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:24.122+0000 7fa3227fc700 1 -- 172.21.15.42:0/20786596 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa31c011860 con 0x7fa32409fae0 2019-12-05T06:28:24.131 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:24.122+0000 7fa329768700 1 -- 172.21.15.42:0/20786596 shutdown_connections 2019-12-05T06:28:24.131 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:24.122+0000 7fa329768700 1 --2- 172.21.15.42:0/20786596 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa32409fae0 0x7fa32407eb50 unknown :-1 s=CLOSED pgs=2014 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:24.132 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:24.122+0000 7fa329768700 1 --2- 172.21.15.42:0/20786596 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa3240a7740 0x7fa32408df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:28:24.132 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:24.122+0000 7fa329768700 1 -- 172.21.15.42:0/20786596 shutdown_connections 2019-12-05T06:28:24.132 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:24.122+0000 7fa329768700 1 -- 172.21.15.42:0/20786596 wait complete. 2019-12-05T06:28:24.132 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:24.122+0000 7fa329768700 1 -- 172.21.15.42:0/20786596 >> 172.21.15.42:0/20786596 conn(0x7fa3240aabe0 msgr2=0x7fa3240a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:28:24.136 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:24.122+0000 7fa329768700 1 Processor -- start 2019-12-05T06:28:24.136 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:24.122+0000 7fa329768700 1 -- start start 2019-12-05T06:28:24.136 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:24.126+0000 7fa329768700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa3241433b0 0x7fa324120e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:28:24.136 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:24.126+0000 7fa329768700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa32407de90 0x7fa324134580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:28:24.136 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:24.126+0000 7fa329768700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa324142e20 con 0x7fa32407da60 2019-12-05T06:28:24.137 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:24.126+0000 7fa329768700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa324143ef0 con 0x7fa32407de90 2019-12-05T06:28:24.137 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:24.126+0000 7fa329768700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa32411b9c0 con 0x7fa3241433b0 2019-12-05T06:28:24.137 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:24.126+0000 7fa3237fe700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa3241433b0 0x7fa324120e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:28:24.137 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:24.126+0000 7fa323fff700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa32407de90 0x7fa324134580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:28:24.137 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:24.126+0000 7fa323fff700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa32407de90 0x7fa324134580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.42:43612/0 (socket says 172.21.15.42:43612) 2019-12-05T06:28:24.137 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:24.126+0000 7fa3237fe700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa3241433b0 0x7fa324120e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:56260/0 (socket says 172.21.15.42:56260) 2019-12-05T06:28:24.138 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:24.126+0000 7fa323fff700 1 -- 172.21.15.42:0/3891911827 learned_addr learned my addr 172.21.15.42:0/3891911827 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:28:24.139 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:24.126+0000 7fa3237fe700 1 -- 172.21.15.42:0/3891911827 >> v2:172.21.15.40:3300/0 conn(0x7fa32407de90 msgr2=0x7fa324134580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:24.140 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:24.126+0000 7fa3237fe700 1 --2- 172.21.15.42:0/3891911827 >> v2:172.21.15.40:3300/0 conn(0x7fa32407de90 0x7fa324134580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:24.140 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:24.126+0000 7fa3237fe700 1 -- 172.21.15.42:0/3891911827 >> v1:172.21.15.40:6789/0 conn(0x7fa32407da60 legacy=0x7fa3241448c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:24.140 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:24.126+0000 7fa3237fe700 1 -- 172.21.15.42:0/3891911827 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa324121350 con 0x7fa3241433b0 2019-12-05T06:28:24.140 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:24.126+0000 7fa323fff700 1 --2- 172.21.15.42:0/3891911827 >> v2:172.21.15.40:3300/0 conn(0x7fa32407de90 0x7fa324134580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:28:24.140 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:24.126+0000 7fa3237fe700 1 --2- 172.21.15.42:0/3891911827 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa3241433b0 0x7fa324120e00 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fa3140032b0 tx=0x7fa314010fa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:24.140 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:24.126+0000 7fa3237fe700 1 --2- 172.21.15.42:0/3891911827 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa3241433b0 0x7fa324120e00 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fa3140032b0 tx=0x7fa314010fa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:24.140 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:24.126+0000 7fa3237fe700 1 --2- 172.21.15.42:0/3891911827 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa3241433b0 0x7fa324120e00 secure :-1 s=READY pgs=589 cs=0 l=1 rx=0x7fa3140032b0 tx=0x7fa314010fa0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:28:24.141 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:24.126+0000 7fa3237fe700 1 --2- 172.21.15.42:0/3891911827 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa3241433b0 0x7fa324120e00 secure :-1 s=THROTTLE_DONE pgs=589 cs=0 l=1 rx=0x7fa3140032b0 tx=0x7fa314010fa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:24.141 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:24.126+0000 7fa31bfff700 1 -- 172.21.15.42:0/3891911827 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa314012960 con 0x7fa3241433b0 2019-12-05T06:28:24.141 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:24.126+0000 7fa3237fe700 1 --2- 172.21.15.42:0/3891911827 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa3241433b0 0x7fa324120e00 secure :-1 s=THROTTLE_DONE pgs=589 cs=0 l=1 rx=0x7fa3140032b0 tx=0x7fa314010fa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:24.141 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:24.126+0000 7fa3237fe700 1 --2- 172.21.15.42:0/3891911827 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa3241433b0 0x7fa324120e00 secure :-1 s=THROTTLE_DONE pgs=589 cs=0 l=1 rx=0x7fa3140032b0 tx=0x7fa314010fa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:24.141 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:24.126+0000 7fa31bfff700 1 -- 172.21.15.42:0/3891911827 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa31400e160 con 0x7fa3241433b0 2019-12-05T06:28:24.141 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:24.126+0000 7fa329768700 1 -- 172.21.15.42:0/3891911827 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3241450c0 con 0x7fa3241433b0 2019-12-05T06:28:24.141 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:24.126+0000 7fa31bfff700 1 -- 172.21.15.42:0/3891911827 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa314005ab0 con 0x7fa3241433b0 2019-12-05T06:28:24.142 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:24.126+0000 7fa329768700 1 -- 172.21.15.42:0/3891911827 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa324134ad0 con 0x7fa3241433b0 2019-12-05T06:28:24.142 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:24.130+0000 7fa3237fe700 1 --2- 172.21.15.42:0/3891911827 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa3241433b0 0x7fa324120e00 secure :-1 s=THROTTLE_DONE pgs=589 cs=0 l=1 rx=0x7fa3140032b0 tx=0x7fa314010fa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:24.142 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:24.130+0000 7fa31bfff700 1 -- 172.21.15.42:0/3891911827 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fa314007180 con 0x7fa3241433b0 2019-12-05T06:28:24.142 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:24.130+0000 7fa3237fe700 1 --2- 172.21.15.42:0/3891911827 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa3241433b0 0x7fa324120e00 secure :-1 s=THROTTLE_DONE pgs=589 cs=0 l=1 rx=0x7fa3140032b0 tx=0x7fa314010fa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:24.142 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:24.130+0000 7fa31bfff700 1 --2- 172.21.15.42:0/3891911827 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa2fc03ad80 0x7fa2fc03d290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:28:24.142 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:24.130+0000 7fa31bfff700 1 -- 172.21.15.42:0/3891911827 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 10933+0+0 (secure 0 0 0) 0x7fa31404b0e0 con 0x7fa3241433b0 2019-12-05T06:28:24.142 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:24.130+0000 7fa322ffd700 1 --2- 172.21.15.42:0/3891911827 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa2fc03ad80 0x7fa2fc03d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:28:24.143 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:24.130+0000 7fa322ffd700 1 --2- 172.21.15.42:0/3891911827 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa2fc03ad80 0x7fa2fc03d290 crc :-1 s=READY pgs=1125 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:28:24.143 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:24.134+0000 7fa329768700 1 -- 172.21.15.42:0/3891911827 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa32405d110 con 0x7fa3241433b0 2019-12-05T06:28:24.146 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:24.142+0000 7fa3237fe700 1 --2- 172.21.15.42:0/3891911827 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa3241433b0 0x7fa324120e00 secure :-1 s=THROTTLE_DONE pgs=589 cs=0 l=1 rx=0x7fa3140032b0 tx=0x7fa314010fa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:24.147 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:24.142+0000 7fa31bfff700 1 -- 172.21.15.42:0/3891911827 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fa314032160 con 0x7fa3241433b0 2019-12-05T06:28:24.323 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:24.318+0000 7fa329768700 1 -- 172.21.15.42:0/3891911827 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa32405b8c0 con 0x7fa3241433b0 2019-12-05T06:28:24.325 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:24.318+0000 7fa3237fe700 1 --2- 172.21.15.42:0/3891911827 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa3241433b0 0x7fa324120e00 secure :-1 s=THROTTLE_DONE pgs=589 cs=0 l=1 rx=0x7fa3140032b0 tx=0x7fa314010fa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:24.325 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:24.318+0000 7fa31bfff700 1 -- 172.21.15.42:0/3891911827 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 318 v318) v1 ==== 95+0+2117 (secure 0 0 0) 0x7fa3140323c0 con 0x7fa3241433b0 2019-12-05T06:28:24.334 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 318 2019-12-05T06:28:24.334 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:24.322+0000 7fa329768700 1 -- 172.21.15.42:0/3891911827 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa2fc03ad80 msgr2=0x7fa2fc03d290 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:24.334 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:24.322+0000 7fa329768700 1 --2- 172.21.15.42:0/3891911827 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa2fc03ad80 0x7fa2fc03d290 crc :-1 s=READY pgs=1125 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:24.335 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:24.326+0000 7fa329768700 1 -- 172.21.15.42:0/3891911827 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa3241433b0 msgr2=0x7fa324120e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:24.335 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:24.326+0000 7fa329768700 1 --2- 172.21.15.42:0/3891911827 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa3241433b0 0x7fa324120e00 secure :-1 s=READY pgs=589 cs=0 l=1 rx=0x7fa3140032b0 tx=0x7fa314010fa0).stop 2019-12-05T06:28:24.335 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:24.326+0000 7fa329768700 1 -- 172.21.15.42:0/3891911827 shutdown_connections 2019-12-05T06:28:24.335 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:24.326+0000 7fa329768700 1 --2- 172.21.15.42:0/3891911827 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa2fc03ad80 0x7fa2fc03d290 unknown :-1 s=CLOSED pgs=1125 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:24.335 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:24.326+0000 7fa329768700 1 --2- 172.21.15.42:0/3891911827 >> v2:172.21.15.40:3300/0 conn(0x7fa32407de90 0x7fa324134580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:24.336 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:24.326+0000 7fa329768700 1 --2- 172.21.15.42:0/3891911827 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa3241433b0 0x7fa324120e00 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:24.336 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:24.326+0000 7fa329768700 1 -- 172.21.15.42:0/3891911827 shutdown_connections 2019-12-05T06:28:24.336 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:24.326+0000 7fa329768700 1 -- 172.21.15.42:0/3891911827 wait complete. 2019-12-05T06:28:24.336 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:24.326+0000 7fa329768700 1 -- 172.21.15.42:0/3891911827 >> 172.21.15.42:0/3891911827 conn(0x7fa32407e4f0 msgr2=0x7fa32411c610 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:28:24.336 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:28:24.337 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":318,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":11142,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/3155851600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3155851600},{"type":"v1","addr":"172.21.15.40:6833","nonce":3155851600}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":318}],"filesystems":[{"mdsmap":{"epoch":317,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:26:46.246866+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":106,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11221},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11221":{"gid":11221,"name":"a","rank":0,"incarnation":314,"state":"up:active","state_seq":34,"addr":"172.21.15.42:6835/3427104185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3427104185},{"type":"v1","addr":"172.21.15.42:6835","nonce":3427104185}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:28:24.338 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:28:26.339 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:28:26.445 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.434+0000 7fa42333d700 1 Processor -- start 2019-12-05T06:28:26.445 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.434+0000 7fa42333d700 1 -- start start 2019-12-05T06:28:26.446 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.438+0000 7fa42333d700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa41c0a7740 0x7fa41c08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:28:26.446 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.438+0000 7fa42333d700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa41c09fae0 0x7fa41c07eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:28:26.446 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.438+0000 7fa42333d700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa41c0a0710 con 0x7fa41c07e720 2019-12-05T06:28:26.446 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.438+0000 7fa42333d700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa41c0a0b00 con 0x7fa41c0a7740 2019-12-05T06:28:26.446 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.438+0000 7fa42333d700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa41c08e5b0 con 0x7fa41c09fae0 2019-12-05T06:28:26.446 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.438+0000 7fa4210d9700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fa41c07e720 0x7fa41c09f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:37050/0 (socket says 172.21.15.42:37050) 2019-12-05T06:28:26.447 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.438+0000 7fa4210d9700 1 -- 172.21.15.42:0/325418920 learned_addr learned my addr 172.21.15.42:0/325418920 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:28:26.447 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.438+0000 7fa4218da700 1 --2- 172.21.15.42:0/325418920 >> v2:172.21.15.40:3300/0 conn(0x7fa41c0a7740 0x7fa41c08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:28:26.447 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.438+0000 7fa4220db700 1 --2- 172.21.15.42:0/325418920 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa41c09fae0 0x7fa41c07eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:28:26.447 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.438+0000 7fa4208d8700 1 -- 172.21.15.42:0/325418920 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1620005810 0 0) 0x7fa41c0a0710 con 0x7fa41c07e720 2019-12-05T06:28:26.447 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.438+0000 7fa4208d8700 1 -- 172.21.15.42:0/325418920 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4080030b0 con 0x7fa41c07e720 2019-12-05T06:28:26.447 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.438+0000 7fa4220db700 1 -- 172.21.15.42:0/325418920 >> v2:172.21.15.40:3300/0 conn(0x7fa41c0a7740 msgr2=0x7fa41c08df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:28:26.447 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.438+0000 7fa4220db700 1 --2- 172.21.15.42:0/325418920 >> v2:172.21.15.40:3300/0 conn(0x7fa41c0a7740 0x7fa41c08df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:28:26.448 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.438+0000 7fa4220db700 1 -- 172.21.15.42:0/325418920 >> v1:172.21.15.40:6789/0 conn(0x7fa41c07e720 legacy=0x7fa41c09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:26.448 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.438+0000 7fa4208d8700 1 -- 172.21.15.42:0/325418920 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 410355904 0 0) 0x7fa4080030b0 con 0x7fa41c07e720 2019-12-05T06:28:26.448 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.438+0000 7fa4220db700 1 -- 172.21.15.42:0/325418920 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa41c07f0a0 con 0x7fa41c09fae0 2019-12-05T06:28:26.448 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.438+0000 7fa4220db700 1 --2- 172.21.15.42:0/325418920 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa41c09fae0 0x7fa41c07eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fa414008b10 tx=0x7fa40c0014a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:26.448 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.438+0000 7fa4220db700 1 --2- 172.21.15.42:0/325418920 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa41c09fae0 0x7fa41c07eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fa414008b10 tx=0x7fa40c0014a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:26.448 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.438+0000 7fa4220db700 1 --2- 172.21.15.42:0/325418920 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa41c09fae0 0x7fa41c07eb50 secure :-1 s=READY pgs=592 cs=0 l=1 rx=0x7fa414008b10 tx=0x7fa40c0014a0).ready entity=mon.2 client_cookie=cdded34e888da3da server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:28:26.448 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.438+0000 7fa4220db700 1 --2- 172.21.15.42:0/325418920 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa41c09fae0 0x7fa41c07eb50 secure :-1 s=THROTTLE_DONE pgs=592 cs=0 l=1 rx=0x7fa414008b10 tx=0x7fa40c0014a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:26.449 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.438+0000 7fa4220db700 1 --2- 172.21.15.42:0/325418920 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa41c09fae0 0x7fa41c07eb50 secure :-1 s=THROTTLE_DONE pgs=592 cs=0 l=1 rx=0x7fa414008b10 tx=0x7fa40c0014a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:26.449 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.438+0000 7fa4220db700 1 --2- 172.21.15.42:0/325418920 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa41c09fae0 0x7fa41c07eb50 secure :-1 s=THROTTLE_DONE pgs=592 cs=0 l=1 rx=0x7fa414008b10 tx=0x7fa40c0014a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:26.449 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.438+0000 7fa4208d8700 1 -- 172.21.15.42:0/325418920 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa40c001750 con 0x7fa41c09fae0 2019-12-05T06:28:26.449 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.438+0000 7fa4208d8700 1 -- 172.21.15.42:0/325418920 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa40c001560 con 0x7fa41c09fae0 2019-12-05T06:28:26.449 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.438+0000 7fa4208d8700 1 -- 172.21.15.42:0/325418920 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa4140109f0 con 0x7fa41c09fae0 2019-12-05T06:28:26.449 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.438+0000 7fa42333d700 1 -- 172.21.15.42:0/325418920 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa41c09fae0 msgr2=0x7fa41c07eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:26.449 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.438+0000 7fa42333d700 1 --2- 172.21.15.42:0/325418920 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa41c09fae0 0x7fa41c07eb50 secure :-1 s=READY pgs=592 cs=0 l=1 rx=0x7fa414008b10 tx=0x7fa40c0014a0).stop 2019-12-05T06:28:26.450 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.438+0000 7fa42333d700 1 -- 172.21.15.42:0/325418920 shutdown_connections 2019-12-05T06:28:26.450 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.438+0000 7fa42333d700 1 --2- 172.21.15.42:0/325418920 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa41c09fae0 0x7fa41c07eb50 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:26.450 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.438+0000 7fa42333d700 1 --2- 172.21.15.42:0/325418920 >> v2:172.21.15.40:3300/0 conn(0x7fa41c0a7740 0x7fa41c08df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:28:26.450 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.442+0000 7fa42333d700 1 -- 172.21.15.42:0/325418920 shutdown_connections 2019-12-05T06:28:26.450 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.442+0000 7fa42333d700 1 -- 172.21.15.42:0/325418920 wait complete. 2019-12-05T06:28:26.451 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.442+0000 7fa42333d700 1 -- 172.21.15.42:0/325418920 >> 172.21.15.42:0/325418920 conn(0x7fa41c0aabe0 msgr2=0x7fa41c0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:28:26.454 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.442+0000 7fa42333d700 1 Processor -- start 2019-12-05T06:28:26.454 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.442+0000 7fa42333d700 1 -- start start 2019-12-05T06:28:26.455 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.446+0000 7fa42333d700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa41c0a1850 0x7fa41c0a3c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:28:26.455 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.446+0000 7fa42333d700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa41c0a5cd0 0x7fa41c0a2ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:28:26.455 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.446+0000 7fa42333d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fa41c07ab20 0x7fa41c07af50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:28:26.455 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.446+0000 7fa42333d700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa41c0a6dd0 con 0x7fa41c0a5cd0 2019-12-05T06:28:26.455 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.446+0000 7fa42333d700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa41c0a4260 con 0x7fa41c07ab20 2019-12-05T06:28:26.455 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.446+0000 7fa42333d700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa41c0a2d50 con 0x7fa41c0a1850 2019-12-05T06:28:26.456 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.446+0000 7fa4220db700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fa41c07ab20 0x7fa41c07af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:28:26.456 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.446+0000 7fa4218da700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa41c0a1850 0x7fa41c0a3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:28:26.456 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.446+0000 7fa4210d9700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa41c0a5cd0 0x7fa41c0a2ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:28:26.456 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.446+0000 7fa4210d9700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa41c0a5cd0 0x7fa41c0a2ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.42:43624/0 (socket says 172.21.15.42:43624) 2019-12-05T06:28:26.456 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.446+0000 7fa4218da700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa41c0a1850 0x7fa41c0a3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:56276/0 (socket says 172.21.15.42:56276) 2019-12-05T06:28:26.456 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.446+0000 7fa4220db700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fa41c07ab20 0x7fa41c07af50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:55812/0 (socket says 172.21.15.42:55812) 2019-12-05T06:28:26.456 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.446+0000 7fa4218da700 1 -- 172.21.15.42:0/2827130513 learned_addr learned my addr 172.21.15.42:0/2827130513 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:28:26.457 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.446+0000 7fa4220db700 1 -- 172.21.15.42:0/2827130513 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa41c0a1850 msgr2=0x7fa41c0a3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:26.457 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.446+0000 7fa4220db700 1 --2- 172.21.15.42:0/2827130513 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa41c0a1850 0x7fa41c0a3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:26.457 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.446+0000 7fa4220db700 1 -- 172.21.15.42:0/2827130513 >> v2:172.21.15.40:3300/0 conn(0x7fa41c0a5cd0 msgr2=0x7fa41c0a2ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:26.457 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.446+0000 7fa4220db700 1 --2- 172.21.15.42:0/2827130513 >> v2:172.21.15.40:3300/0 conn(0x7fa41c0a5cd0 0x7fa41c0a2ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:26.457 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.446+0000 7fa4220db700 1 -- 172.21.15.42:0/2827130513 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa41c0a3540 con 0x7fa41c07ab20 2019-12-05T06:28:26.457 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.446+0000 7fa4218da700 1 --2- 172.21.15.42:0/2827130513 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa41c0a1850 0x7fa41c0a3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:28:26.457 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.446+0000 7fa4220db700 1 --2- 172.21.15.42:0/2827130513 >> v2:172.21.15.42:3300/0 conn(0x7fa41c07ab20 0x7fa41c07af50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fa41c0a4420 tx=0x7fa414007a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:26.458 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.446+0000 7fa4220db700 1 --2- 172.21.15.42:0/2827130513 >> v2:172.21.15.42:3300/0 conn(0x7fa41c07ab20 0x7fa41c07af50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fa41c0a4420 tx=0x7fa414007a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:26.458 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.446+0000 7fa4220db700 1 --2- 172.21.15.42:0/2827130513 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa41c07ab20 0x7fa41c07af50 secure :-1 s=READY pgs=2688 cs=0 l=1 rx=0x7fa41c0a4420 tx=0x7fa414007a90).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:28:26.458 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.446+0000 7fa4220db700 1 --2- 172.21.15.42:0/2827130513 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa41c07ab20 0x7fa41c07af50 secure :-1 s=THROTTLE_DONE pgs=2688 cs=0 l=1 rx=0x7fa41c0a4420 tx=0x7fa414007a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:26.458 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.446+0000 7fa4127fc700 1 -- 172.21.15.42:0/2827130513 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa414005ab0 con 0x7fa41c07ab20 2019-12-05T06:28:26.458 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.446+0000 7fa4220db700 1 --2- 172.21.15.42:0/2827130513 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa41c07ab20 0x7fa41c07af50 secure :-1 s=THROTTLE_DONE pgs=2688 cs=0 l=1 rx=0x7fa41c0a4420 tx=0x7fa414007a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:26.458 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.446+0000 7fa4127fc700 1 -- 172.21.15.42:0/2827130513 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa414008d40 con 0x7fa41c07ab20 2019-12-05T06:28:26.458 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.446+0000 7fa4220db700 1 --2- 172.21.15.42:0/2827130513 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa41c07ab20 0x7fa41c07af50 secure :-1 s=THROTTLE_DONE pgs=2688 cs=0 l=1 rx=0x7fa41c0a4420 tx=0x7fa414007a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:26.459 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.446+0000 7fa42333d700 1 -- 172.21.15.42:0/2827130513 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa41c08f080 con 0x7fa41c07ab20 2019-12-05T06:28:26.459 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.446+0000 7fa4127fc700 1 -- 172.21.15.42:0/2827130513 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa41400f680 con 0x7fa41c07ab20 2019-12-05T06:28:26.459 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.446+0000 7fa42333d700 1 -- 172.21.15.42:0/2827130513 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa41c131500 con 0x7fa41c07ab20 2019-12-05T06:28:26.459 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.446+0000 7fa4220db700 1 --2- 172.21.15.42:0/2827130513 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa41c07ab20 0x7fa41c07af50 secure :-1 s=THROTTLE_DONE pgs=2688 cs=0 l=1 rx=0x7fa41c0a4420 tx=0x7fa414007a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:26.459 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.446+0000 7fa4220db700 1 --2- 172.21.15.42:0/2827130513 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa41c07ab20 0x7fa41c07af50 secure :-1 s=THROTTLE_DONE pgs=2688 cs=0 l=1 rx=0x7fa41c0a4420 tx=0x7fa414007a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:26.459 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.446+0000 7fa4127fc700 1 -- 172.21.15.42:0/2827130513 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fa414015250 con 0x7fa41c07ab20 2019-12-05T06:28:26.459 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.450+0000 7fa4127fc700 1 --2- 172.21.15.42:0/2827130513 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa40403ae10 0x7fa40403d320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:28:26.460 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.450+0000 7fa4127fc700 1 -- 172.21.15.42:0/2827130513 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 11019+0+0 (secure 0 0 0) 0x7fa41404af90 con 0x7fa41c07ab20 2019-12-05T06:28:26.460 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.450+0000 7fa4218da700 1 --2- 172.21.15.42:0/2827130513 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa40403ae10 0x7fa40403d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:28:26.460 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.450+0000 7fa4218da700 1 --2- 172.21.15.42:0/2827130513 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa40403ae10 0x7fa40403d320 crc :-1 s=READY pgs=1126 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:28:26.460 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.450+0000 7fa42333d700 1 -- 172.21.15.42:0/2827130513 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa41c05d110 con 0x7fa41c07ab20 2019-12-05T06:28:26.464 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.458+0000 7fa4220db700 1 --2- 172.21.15.42:0/2827130513 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa41c07ab20 0x7fa41c07af50 secure :-1 s=THROTTLE_DONE pgs=2688 cs=0 l=1 rx=0x7fa41c0a4420 tx=0x7fa414007a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:26.464 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.458+0000 7fa4127fc700 1 -- 172.21.15.42:0/2827130513 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fa41400f950 con 0x7fa41c07ab20 2019-12-05T06:28:26.633 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.630+0000 7fa42333d700 1 -- 172.21.15.42:0/2827130513 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa41c0a2b30 con 0x7fa41c07ab20 2019-12-05T06:28:26.634 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.630+0000 7fa4220db700 1 --2- 172.21.15.42:0/2827130513 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa41c07ab20 0x7fa41c07af50 secure :-1 s=THROTTLE_DONE pgs=2688 cs=0 l=1 rx=0x7fa41c0a4420 tx=0x7fa414007a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:26.634 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.630+0000 7fa4127fc700 1 -- 172.21.15.42:0/2827130513 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 319 v319) v1 ==== 95+0+1778 (secure 0 0 0) 0x7fa414032d80 con 0x7fa41c07ab20 2019-12-05T06:28:26.634 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 319 2019-12-05T06:28:26.635 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:28:26.635 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":319,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":319,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:28:24.937170+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":107,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11142},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11142":{"gid":11142,"name":"b","rank":0,"incarnation":319,"state":"up:replay","state_seq":2,"addr":"172.21.15.40:6833/3155851600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3155851600},{"type":"v1","addr":"172.21.15.40:6833","nonce":3155851600}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:28:26.637 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.634+0000 7fa42333d700 1 -- 172.21.15.42:0/2827130513 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa40403ae10 msgr2=0x7fa40403d320 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:26.637 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.634+0000 7fa42333d700 1 --2- 172.21.15.42:0/2827130513 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa40403ae10 0x7fa40403d320 crc :-1 s=READY pgs=1126 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:26.637 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.634+0000 7fa42333d700 1 -- 172.21.15.42:0/2827130513 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa41c07ab20 msgr2=0x7fa41c07af50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:26.637 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.634+0000 7fa42333d700 1 --2- 172.21.15.42:0/2827130513 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa41c07ab20 0x7fa41c07af50 secure :-1 s=READY pgs=2688 cs=0 l=1 rx=0x7fa41c0a4420 tx=0x7fa414007a90).stop 2019-12-05T06:28:26.637 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.634+0000 7fa42333d700 1 -- 172.21.15.42:0/2827130513 shutdown_connections 2019-12-05T06:28:26.637 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.634+0000 7fa42333d700 1 --2- 172.21.15.42:0/2827130513 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa40403ae10 0x7fa40403d320 unknown :-1 s=CLOSED pgs=1126 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:26.638 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.634+0000 7fa42333d700 1 --2- 172.21.15.42:0/2827130513 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa41c07ab20 0x7fa41c07af50 unknown :-1 s=CLOSED pgs=2688 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:26.638 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.634+0000 7fa42333d700 1 --2- 172.21.15.42:0/2827130513 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa41c0a1850 0x7fa41c0a3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:26.638 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.634+0000 7fa42333d700 1 --2- 172.21.15.42:0/2827130513 >> v2:172.21.15.40:3300/0 conn(0x7fa41c0a5cd0 0x7fa41c0a2ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:26.639 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.634+0000 7fa42333d700 1 -- 172.21.15.42:0/2827130513 shutdown_connections 2019-12-05T06:28:26.639 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.634+0000 7fa42333d700 1 -- 172.21.15.42:0/2827130513 wait complete. 2019-12-05T06:28:26.639 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.634+0000 7fa42333d700 1 -- 172.21.15.42:0/2827130513 >> 172.21.15.42:0/2827130513 conn(0x7fa41c11d200 msgr2=0x7fa41c09f1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:28:26.647 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a down, removed from mdsmap 2019-12-05T06:28:26.647 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2019-12-05T06:28:26.647 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:28:26.750 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.742+0000 7fedabb56700 1 Processor -- start 2019-12-05T06:28:26.750 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.742+0000 7fedabb56700 1 -- start start 2019-12-05T06:28:26.751 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.742+0000 7fedabb56700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7feda40a7740 0x7feda408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:28:26.752 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.742+0000 7fedabb56700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feda40a0710 con 0x7feda409fae0 2019-12-05T06:28:26.752 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.742+0000 7fedabb56700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feda40a9ba0 con 0x7feda407e720 2019-12-05T06:28:26.752 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.742+0000 7fedabb56700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feda40a0b00 con 0x7feda40a7740 2019-12-05T06:28:26.754 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.742+0000 7feda98f2700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7feda407e720 0x7feda409f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:55854/0 (socket says 172.21.15.42:55854) 2019-12-05T06:28:26.754 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.742+0000 7feda98f2700 1 -- 172.21.15.42:0/3995585147 learned_addr learned my addr 172.21.15.42:0/3995585147 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:28:26.754 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.742+0000 7fedaa0f3700 1 --2- 172.21.15.42:0/3995585147 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7feda40a7740 0x7feda408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:28:26.755 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.742+0000 7feda90f1700 1 -- 172.21.15.42:0/3995585147 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2755922684 0 0) 0x7feda40a9ba0 con 0x7feda407e720 2019-12-05T06:28:26.755 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.742+0000 7feda90f1700 1 -- 172.21.15.42:0/3995585147 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed8c0030b0 con 0x7feda407e720 2019-12-05T06:28:26.755 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.742+0000 7feda90f1700 1 -- 172.21.15.42:0/3995585147 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3977703638 0 0) 0x7feda40a0710 con 0x7feda409fae0 2019-12-05T06:28:26.755 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.742+0000 7feda90f1700 1 -- 172.21.15.42:0/3995585147 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feda40a9ba0 con 0x7feda409fae0 2019-12-05T06:28:26.755 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.742+0000 7fedaa0f3700 1 -- 172.21.15.42:0/3995585147 >> v1:172.21.15.42:6789/0 conn(0x7feda407e720 legacy=0x7feda409f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:26.755 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.742+0000 7feda90f1700 1 -- 172.21.15.42:0/3995585147 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2994008482 0 0) 0x7fed8c0030b0 con 0x7feda407e720 2019-12-05T06:28:26.755 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.742+0000 7fedaa0f3700 1 -- 172.21.15.42:0/3995585147 >> v1:172.21.15.40:6789/0 conn(0x7feda409fae0 legacy=0x7feda414efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:26.756 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.742+0000 7fedaa0f3700 1 -- 172.21.15.42:0/3995585147 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feda407eb50 con 0x7feda40a7740 2019-12-05T06:28:26.756 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.742+0000 7fedaa0f3700 1 --2- 172.21.15.42:0/3995585147 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7feda40a7740 0x7feda408df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fed94008b10 tx=0x7feda00014a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:26.756 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.746+0000 7fedaa0f3700 1 --2- 172.21.15.42:0/3995585147 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7feda40a7740 0x7feda408df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fed94008b10 tx=0x7feda00014a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:26.756 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.746+0000 7fedaa0f3700 1 --2- 172.21.15.42:0/3995585147 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7feda40a7740 0x7feda408df80 secure :-1 s=READY pgs=593 cs=0 l=1 rx=0x7fed94008b10 tx=0x7feda00014a0).ready entity=mon.2 client_cookie=e0c956ef01b326cc server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:28:26.756 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.746+0000 7fedaa0f3700 1 --2- 172.21.15.42:0/3995585147 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7feda40a7740 0x7feda408df80 secure :-1 s=THROTTLE_DONE pgs=593 cs=0 l=1 rx=0x7fed94008b10 tx=0x7feda00014a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:26.756 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.746+0000 7fedaa0f3700 1 --2- 172.21.15.42:0/3995585147 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7feda40a7740 0x7feda408df80 secure :-1 s=THROTTLE_DONE pgs=593 cs=0 l=1 rx=0x7fed94008b10 tx=0x7feda00014a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:26.756 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.746+0000 7feda90f1700 1 -- 172.21.15.42:0/3995585147 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7feda0001750 con 0x7feda40a7740 2019-12-05T06:28:26.757 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.746+0000 7fedaa0f3700 1 --2- 172.21.15.42:0/3995585147 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7feda40a7740 0x7feda408df80 secure :-1 s=THROTTLE_DONE pgs=593 cs=0 l=1 rx=0x7fed94008b10 tx=0x7feda00014a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:26.757 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.746+0000 7feda90f1700 1 -- 172.21.15.42:0/3995585147 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feda0001560 con 0x7feda40a7740 2019-12-05T06:28:26.757 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.746+0000 7feda90f1700 1 -- 172.21.15.42:0/3995585147 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fed94011ac0 con 0x7feda40a7740 2019-12-05T06:28:26.757 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.746+0000 7fedabb56700 1 -- 172.21.15.42:0/3995585147 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7feda40a7740 msgr2=0x7feda408df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:26.757 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.746+0000 7fedabb56700 1 --2- 172.21.15.42:0/3995585147 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7feda40a7740 0x7feda408df80 secure :-1 s=READY pgs=593 cs=0 l=1 rx=0x7fed94008b10 tx=0x7feda00014a0).stop 2019-12-05T06:28:26.757 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.746+0000 7fedabb56700 1 -- 172.21.15.42:0/3995585147 shutdown_connections 2019-12-05T06:28:26.758 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.746+0000 7fedabb56700 1 --2- 172.21.15.42:0/3995585147 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7feda40a7740 0x7feda408df80 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:26.758 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.746+0000 7fedabb56700 1 -- 172.21.15.42:0/3995585147 shutdown_connections 2019-12-05T06:28:26.758 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.746+0000 7fedabb56700 1 -- 172.21.15.42:0/3995585147 wait complete. 2019-12-05T06:28:26.758 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.746+0000 7fedabb56700 1 -- 172.21.15.42:0/3995585147 >> 172.21.15.42:0/3995585147 conn(0x7feda40aabe0 msgr2=0x7feda40a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:28:26.761 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.746+0000 7fedabb56700 1 Processor -- start 2019-12-05T06:28:26.761 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.746+0000 7fedabb56700 1 -- start start 2019-12-05T06:28:26.762 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.746+0000 7fedabb56700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7feda41161c0 0x7feda40752e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:28:26.762 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.746+0000 7fedabb56700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feda408eb70 con 0x7feda40a6f40 2019-12-05T06:28:26.762 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.746+0000 7fedabb56700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feda41165f0 con 0x7feda411ae50 2019-12-05T06:28:26.762 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.746+0000 7fedabb56700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feda411a6b0 con 0x7feda41161c0 2019-12-05T06:28:26.762 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.746+0000 7fedaa0f3700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7feda40a6f40 0x7feda4133970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:37068/0 (socket says 172.21.15.42:37068) 2019-12-05T06:28:26.762 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.746+0000 7feda98f2700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7feda411ae50 0x7feda407d480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:55862/0 (socket says 172.21.15.42:55862) 2019-12-05T06:28:26.762 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.746+0000 7fedaa0f3700 1 -- 172.21.15.42:0/2195897317 learned_addr learned my addr 172.21.15.42:0/2195897317 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:28:26.763 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.746+0000 7fedaa8f4700 1 --2- 172.21.15.42:0/2195897317 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7feda41161c0 0x7feda40752e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:28:26.763 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.750+0000 7fed9affd700 1 -- 172.21.15.42:0/2195897317 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 899307231 0 0) 0x7feda408eb70 con 0x7feda40a6f40 2019-12-05T06:28:26.763 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.750+0000 7fed9affd700 1 -- 172.21.15.42:0/2195897317 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed840030b0 con 0x7feda40a6f40 2019-12-05T06:28:26.763 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.750+0000 7fed9affd700 1 -- 172.21.15.42:0/2195897317 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3940728584 0 0) 0x7feda41165f0 con 0x7feda411ae50 2019-12-05T06:28:26.765 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.750+0000 7fed9affd700 1 -- 172.21.15.42:0/2195897317 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feda408eb70 con 0x7feda411ae50 2019-12-05T06:28:26.765 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.750+0000 7fedaa8f4700 1 -- 172.21.15.42:0/2195897317 >> v1:172.21.15.42:6789/0 conn(0x7feda411ae50 legacy=0x7feda407d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:26.766 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.750+0000 7fedaa8f4700 1 -- 172.21.15.42:0/2195897317 >> v1:172.21.15.40:6789/0 conn(0x7feda40a6f40 legacy=0x7feda4133970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:26.766 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.750+0000 7fedaa8f4700 1 -- 172.21.15.42:0/2195897317 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feda4116820 con 0x7feda41161c0 2019-12-05T06:28:26.766 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.750+0000 7fedaa8f4700 1 --2- 172.21.15.42:0/2195897317 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7feda41161c0 0x7feda40752e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fed9c000e90 tx=0x7fed9c010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:26.766 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.750+0000 7fedaa8f4700 1 --2- 172.21.15.42:0/2195897317 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7feda41161c0 0x7feda40752e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fed9c000e90 tx=0x7fed9c010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:26.766 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.750+0000 7fedaa8f4700 1 --2- 172.21.15.42:0/2195897317 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7feda41161c0 0x7feda40752e0 secure :-1 s=READY pgs=594 cs=0 l=1 rx=0x7fed9c000e90 tx=0x7fed9c010720).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:28:26.766 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.750+0000 7fedaa8f4700 1 --2- 172.21.15.42:0/2195897317 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7feda41161c0 0x7feda40752e0 secure :-1 s=THROTTLE_DONE pgs=594 cs=0 l=1 rx=0x7fed9c000e90 tx=0x7fed9c010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:26.766 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.750+0000 7fedaa8f4700 1 --2- 172.21.15.42:0/2195897317 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7feda41161c0 0x7feda40752e0 secure :-1 s=THROTTLE_DONE pgs=594 cs=0 l=1 rx=0x7fed9c000e90 tx=0x7fed9c010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:26.767 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.750+0000 7fed9affd700 1 -- 172.21.15.42:0/2195897317 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fed9c006b30 con 0x7feda41161c0 2019-12-05T06:28:26.767 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.750+0000 7fedaa8f4700 1 --2- 172.21.15.42:0/2195897317 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7feda41161c0 0x7feda40752e0 secure :-1 s=THROTTLE_DONE pgs=594 cs=0 l=1 rx=0x7fed9c000e90 tx=0x7fed9c010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:26.767 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.750+0000 7fed9affd700 1 -- 172.21.15.42:0/2195897317 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed9c006750 con 0x7feda41161c0 2019-12-05T06:28:26.767 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.750+0000 7fedabb56700 1 -- 172.21.15.42:0/2195897317 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feda4075830 con 0x7feda41161c0 2019-12-05T06:28:26.767 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.750+0000 7fed9affd700 1 -- 172.21.15.42:0/2195897317 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fed9c007640 con 0x7feda41161c0 2019-12-05T06:28:26.767 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.750+0000 7fedabb56700 1 -- 172.21.15.42:0/2195897317 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feda4075d30 con 0x7feda41161c0 2019-12-05T06:28:26.767 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.750+0000 7fedaa8f4700 1 --2- 172.21.15.42:0/2195897317 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7feda41161c0 0x7feda40752e0 secure :-1 s=THROTTLE_DONE pgs=594 cs=0 l=1 rx=0x7fed9c000e90 tx=0x7fed9c010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:26.768 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.750+0000 7fed9affd700 1 -- 172.21.15.42:0/2195897317 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fed9c007950 con 0x7feda41161c0 2019-12-05T06:28:26.768 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.750+0000 7fedaa8f4700 1 --2- 172.21.15.42:0/2195897317 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7feda41161c0 0x7feda40752e0 secure :-1 s=THROTTLE_DONE pgs=594 cs=0 l=1 rx=0x7fed9c000e90 tx=0x7fed9c010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:26.768 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.750+0000 7fed9affd700 1 --2- 172.21.15.42:0/2195897317 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fed8403b520 0x7fed8403da30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:28:26.768 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.750+0000 7fed9affd700 1 -- 172.21.15.42:0/2195897317 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 11019+0+0 (secure 0 0 0) 0x7fed9c04afc0 con 0x7feda41161c0 2019-12-05T06:28:26.768 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.750+0000 7fedaa0f3700 1 --2- 172.21.15.42:0/2195897317 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fed8403b520 0x7fed8403da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:28:26.768 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.754+0000 7fedaa0f3700 1 --2- 172.21.15.42:0/2195897317 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fed8403b520 0x7fed8403da30 crc :-1 s=READY pgs=1127 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:28:26.769 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.758+0000 7fedabb56700 1 -- 172.21.15.42:0/2195897317 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feda405d110 con 0x7feda41161c0 2019-12-05T06:28:26.770 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.766+0000 7fedaa8f4700 1 --2- 172.21.15.42:0/2195897317 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7feda41161c0 0x7feda40752e0 secure :-1 s=THROTTLE_DONE pgs=594 cs=0 l=1 rx=0x7fed9c000e90 tx=0x7fed9c010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:26.770 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.766+0000 7fed9affd700 1 -- 172.21.15.42:0/2195897317 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fed9c011de0 con 0x7feda41161c0 2019-12-05T06:28:26.934 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.930+0000 7fedabb56700 1 -- 172.21.15.42:0/2195897317 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7feda405b8c0 con 0x7feda41161c0 2019-12-05T06:28:26.935 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.930+0000 7fedaa8f4700 1 --2- 172.21.15.42:0/2195897317 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7feda41161c0 0x7feda40752e0 secure :-1 s=THROTTLE_DONE pgs=594 cs=0 l=1 rx=0x7fed9c000e90 tx=0x7fed9c010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:26.935 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.930+0000 7fed9affd700 1 -- 172.21.15.42:0/2195897317 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 319 v319) v1 ==== 95+0+1778 (secure 0 0 0) 0x7fed9c031fe0 con 0x7feda41161c0 2019-12-05T06:28:26.935 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 319 2019-12-05T06:28:26.936 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:28:26.936 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":319,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":319,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:28:24.937170+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":107,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11142},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11142":{"gid":11142,"name":"b","rank":0,"incarnation":319,"state":"up:replay","state_seq":2,"addr":"172.21.15.40:6833/3155851600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3155851600},{"type":"v1","addr":"172.21.15.40:6833","nonce":3155851600}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:28:26.938 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.934+0000 7fedabb56700 1 -- 172.21.15.42:0/2195897317 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fed8403b520 msgr2=0x7fed8403da30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:26.938 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.934+0000 7fedabb56700 1 --2- 172.21.15.42:0/2195897317 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fed8403b520 0x7fed8403da30 crc :-1 s=READY pgs=1127 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:26.938 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.934+0000 7fedabb56700 1 -- 172.21.15.42:0/2195897317 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7feda41161c0 msgr2=0x7feda40752e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:26.938 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.934+0000 7fedabb56700 1 --2- 172.21.15.42:0/2195897317 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7feda41161c0 0x7feda40752e0 secure :-1 s=READY pgs=594 cs=0 l=1 rx=0x7fed9c000e90 tx=0x7fed9c010720).stop 2019-12-05T06:28:26.938 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.934+0000 7fedabb56700 1 -- 172.21.15.42:0/2195897317 shutdown_connections 2019-12-05T06:28:26.938 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.934+0000 7fedabb56700 1 --2- 172.21.15.42:0/2195897317 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fed8403b520 0x7fed8403da30 unknown :-1 s=CLOSED pgs=1127 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:26.939 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.934+0000 7fedabb56700 1 --2- 172.21.15.42:0/2195897317 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7feda41161c0 0x7feda40752e0 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:26.939 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.934+0000 7fedabb56700 1 -- 172.21.15.42:0/2195897317 shutdown_connections 2019-12-05T06:28:26.939 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.934+0000 7fedabb56700 1 -- 172.21.15.42:0/2195897317 wait complete. 2019-12-05T06:28:26.939 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:26.934+0000 7fedabb56700 1 -- 172.21.15.42:0/2195897317 >> 172.21.15.42:0/2195897317 conn(0x7feda4066fe0 msgr2=0x7feda411c790 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:28:26.949 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 319, "feature_flags": { "enable_multiple": false, "ever_enabled_multiple": false }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2019-12-05T05:03:35.604738+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 319, "ever_allowed_features": 2, "explicitly_allowed_features": 2, "failed": [], "flags": 18, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_11142": { "addr": "172.21.15.40:6833/3155851600", "addrs": { "addrvec": [ { "addr": "172.21.15.40:6832", "nonce": 3155851600, "type": "v2" }, { "addr": "172.21.15.40:6833", "nonce": 3155851600, "type": "v1" } ] }, "export_targets": [], "features": 4540138292836696063, "flags": 0, "gid": 11142, "incarnation": 319, "name": "b", "rank": 0, "state": "up:replay", "state_seq": 2 } }, "last_failure": 0, "last_failure_osd_epoch": 107, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 2, "min_compat_client": "0 (unknown)", "modified": "2019-12-05T06:28:24.937170+0000", "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 11142 } } } ], "standbys": [] } 2019-12-05T06:28:28.950 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:28:29.057 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:29.046+0000 7fbf0d617700 1 Processor -- start 2019-12-05T06:28:29.057 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:29.046+0000 7fbf0d617700 1 -- start start 2019-12-05T06:28:29.057 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:29.050+0000 7fbf0d617700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbf0808df80 0x7fbf0809f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:28:29.057 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:29.050+0000 7fbf0d617700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fbf0809f830 0x7fbf0814efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:28:29.058 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:29.050+0000 7fbf0d617700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf0809fc60 con 0x7fbf080a7740 2019-12-05T06:28:29.058 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:29.050+0000 7fbf0d617700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf080a0b00 con 0x7fbf0809f830 2019-12-05T06:28:29.058 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:29.050+0000 7fbf0d617700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf0808e5b0 con 0x7fbf0808df80 2019-12-05T06:28:29.058 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:29.050+0000 7fbf07fff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fbf0809f830 0x7fbf0814efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:28:29.058 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:29.050+0000 7fbf07fff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fbf0809f830 0x7fbf0814efe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:55836/0 (socket says 172.21.15.42:55836) 2019-12-05T06:28:29.058 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:29.050+0000 7fbf07fff700 1 -- 172.21.15.42:0/4133474262 learned_addr learned my addr 172.21.15.42:0/4133474262 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:28:29.059 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:29.050+0000 7fbf06ffd700 1 --2- 172.21.15.42:0/4133474262 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbf0808df80 0x7fbf0809f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:28:29.059 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:29.050+0000 7fbf07fff700 1 -- 172.21.15.42:0/4133474262 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbf0808df80 msgr2=0x7fbf0809f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:28:29.059 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:29.050+0000 7fbf07fff700 1 --2- 172.21.15.42:0/4133474262 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbf0808df80 0x7fbf0809f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:28:29.059 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:29.050+0000 7fbf07fff700 1 -- 172.21.15.42:0/4133474262 >> v1:172.21.15.42:6789/0 conn(0x7fbf080a7740 legacy=0x7fbf0807e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:29.059 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:29.050+0000 7fbf07fff700 1 -- 172.21.15.42:0/4133474262 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf080a0710 con 0x7fbf0809f830 2019-12-05T06:28:29.059 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:29.050+0000 7fbf07fff700 1 --2- 172.21.15.42:0/4133474262 >> v2:172.21.15.42:3300/0 conn(0x7fbf0809f830 0x7fbf0814efe0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fbf00008b10 tx=0x7fbf000100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:29.059 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:29.050+0000 7fbf07fff700 1 --2- 172.21.15.42:0/4133474262 >> v2:172.21.15.42:3300/0 conn(0x7fbf0809f830 0x7fbf0814efe0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fbf00008b10 tx=0x7fbf000100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:29.060 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:29.050+0000 7fbf07fff700 1 --2- 172.21.15.42:0/4133474262 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbf0809f830 0x7fbf0814efe0 secure :-1 s=READY pgs=2692 cs=0 l=1 rx=0x7fbf00008b10 tx=0x7fbf000100b0).ready entity=mon.1 client_cookie=904abd28f4690867 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:28:29.060 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:29.050+0000 7fbf07fff700 1 --2- 172.21.15.42:0/4133474262 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbf0809f830 0x7fbf0814efe0 secure :-1 s=THROTTLE_DONE pgs=2692 cs=0 l=1 rx=0x7fbf00008b10 tx=0x7fbf000100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:29.060 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:29.050+0000 7fbf067fc700 1 -- 172.21.15.42:0/4133474262 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbf00010da0 con 0x7fbf0809f830 2019-12-05T06:28:29.060 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:29.050+0000 7fbf07fff700 1 --2- 172.21.15.42:0/4133474262 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbf0809f830 0x7fbf0814efe0 secure :-1 s=THROTTLE_DONE pgs=2692 cs=0 l=1 rx=0x7fbf00008b10 tx=0x7fbf000100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:29.060 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:29.050+0000 7fbf07fff700 1 --2- 172.21.15.42:0/4133474262 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbf0809f830 0x7fbf0814efe0 secure :-1 s=THROTTLE_DONE pgs=2692 cs=0 l=1 rx=0x7fbf00008b10 tx=0x7fbf000100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:29.060 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:29.050+0000 7fbf067fc700 1 -- 172.21.15.42:0/4133474262 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf000109c0 con 0x7fbf0809f830 2019-12-05T06:28:29.060 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:29.050+0000 7fbf067fc700 1 -- 172.21.15.42:0/4133474262 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbf000114a0 con 0x7fbf0809f830 2019-12-05T06:28:29.061 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:29.050+0000 7fbf0d617700 1 -- 172.21.15.42:0/4133474262 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbf0809f830 msgr2=0x7fbf0814efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:29.061 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:29.050+0000 7fbf0d617700 1 --2- 172.21.15.42:0/4133474262 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbf0809f830 0x7fbf0814efe0 secure :-1 s=READY pgs=2692 cs=0 l=1 rx=0x7fbf00008b10 tx=0x7fbf000100b0).stop 2019-12-05T06:28:29.061 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:29.050+0000 7fbf0d617700 1 -- 172.21.15.42:0/4133474262 shutdown_connections 2019-12-05T06:28:29.061 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:29.050+0000 7fbf0d617700 1 --2- 172.21.15.42:0/4133474262 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbf0809f830 0x7fbf0814efe0 unknown :-1 s=CLOSED pgs=2692 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:29.061 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:29.050+0000 7fbf0d617700 1 --2- 172.21.15.42:0/4133474262 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbf0808df80 0x7fbf0809f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:28:29.062 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:29.050+0000 7fbf0d617700 1 -- 172.21.15.42:0/4133474262 shutdown_connections 2019-12-05T06:28:29.062 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:29.050+0000 7fbf0d617700 1 -- 172.21.15.42:0/4133474262 wait complete. 2019-12-05T06:28:29.062 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:29.050+0000 7fbf0d617700 1 -- 172.21.15.42:0/4133474262 >> 172.21.15.42:0/4133474262 conn(0x7fbf080aabe0 msgr2=0x7fbf080a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:28:29.066 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:29.054+0000 7fbf0d617700 1 Processor -- start 2019-12-05T06:28:29.066 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:29.054+0000 7fbf0d617700 1 -- start start 2019-12-05T06:28:29.066 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:29.054+0000 7fbf0d617700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fbf08117b70 0x7fbf081449b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:28:29.067 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:29.054+0000 7fbf0d617700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fbf08131cd0 0x7fbf080742d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:28:29.067 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:29.054+0000 7fbf0d617700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf08119ee0 con 0x7fbf08119ab0 2019-12-05T06:28:29.067 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:29.054+0000 7fbf0d617700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf0811a110 con 0x7fbf08131cd0 2019-12-05T06:28:29.067 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:29.054+0000 7fbf0d617700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf08118190 con 0x7fbf08117b70 2019-12-05T06:28:29.068 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:29.058+0000 7fbf077fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fbf08117b70 0x7fbf081449b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:28:29.068 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:29.058+0000 7fbf07fff700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fbf08131cd0 0x7fbf080742d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:28:29.068 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:29.058+0000 7fbf077fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fbf08117b70 0x7fbf081449b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:55838/0 (socket says 172.21.15.42:55838) 2019-12-05T06:28:29.068 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:29.058+0000 7fbf077fe700 1 -- 172.21.15.42:0/2650316802 learned_addr learned my addr 172.21.15.42:0/2650316802 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:28:29.068 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:29.058+0000 7fbf07fff700 1 -- 172.21.15.42:0/2650316802 >> v2:172.21.15.42:3300/0 conn(0x7fbf08117b70 msgr2=0x7fbf081449b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:29.069 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:29.058+0000 7fbf07fff700 1 --2- 172.21.15.42:0/2650316802 >> v2:172.21.15.42:3300/0 conn(0x7fbf08117b70 0x7fbf081449b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:29.069 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:29.058+0000 7fbf07fff700 1 -- 172.21.15.42:0/2650316802 >> v1:172.21.15.40:6789/0 conn(0x7fbf08119ab0 legacy=0x7fbf081314d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:29.069 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:29.058+0000 7fbeeffff700 1 -- 172.21.15.42:0/2650316802 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2396102306 0 0) 0x7fbf08119ee0 con 0x7fbf08119ab0 2019-12-05T06:28:29.069 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:29.058+0000 7fbf07fff700 1 -- 172.21.15.42:0/2650316802 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf081186d0 con 0x7fbf08131cd0 2019-12-05T06:28:29.069 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:29.058+0000 7fbf077fe700 1 --2- 172.21.15.42:0/2650316802 >> v2:172.21.15.42:3300/0 conn(0x7fbf08117b70 0x7fbf081449b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:28:29.069 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:29.058+0000 7fbf07fff700 1 --2- 172.21.15.42:0/2650316802 >> v2:172.21.15.40:3300/0 conn(0x7fbf08131cd0 0x7fbf080742d0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fbf00008b10 tx=0x7fbf00010cd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:29.069 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:29.058+0000 7fbf07fff700 1 --2- 172.21.15.42:0/2650316802 >> v2:172.21.15.40:3300/0 conn(0x7fbf08131cd0 0x7fbf080742d0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fbf00008b10 tx=0x7fbf00010cd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:29.070 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:29.058+0000 7fbf07fff700 1 --2- 172.21.15.42:0/2650316802 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbf08131cd0 0x7fbf080742d0 secure :-1 s=READY pgs=2020 cs=0 l=1 rx=0x7fbf00008b10 tx=0x7fbf00010cd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:28:29.070 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:29.058+0000 7fbf07fff700 1 --2- 172.21.15.42:0/2650316802 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbf08131cd0 0x7fbf080742d0 secure :-1 s=THROTTLE_DONE pgs=2020 cs=0 l=1 rx=0x7fbf00008b10 tx=0x7fbf00010cd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:29.070 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:29.058+0000 7fbeeffff700 1 -- 172.21.15.42:0/2650316802 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbf00011bb0 con 0x7fbf08131cd0 2019-12-05T06:28:29.070 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:29.058+0000 7fbf07fff700 1 --2- 172.21.15.42:0/2650316802 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbf08131cd0 0x7fbf080742d0 secure :-1 s=THROTTLE_DONE pgs=2020 cs=0 l=1 rx=0x7fbf00008b10 tx=0x7fbf00010cd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:29.070 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:29.058+0000 7fbf07fff700 1 --2- 172.21.15.42:0/2650316802 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbf08131cd0 0x7fbf080742d0 secure :-1 s=THROTTLE_DONE pgs=2020 cs=0 l=1 rx=0x7fbf00008b10 tx=0x7fbf00010cd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:29.070 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:29.058+0000 7fbeeffff700 1 -- 172.21.15.42:0/2650316802 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf000110c0 con 0x7fbf08131cd0 2019-12-05T06:28:29.070 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:29.058+0000 7fbeeffff700 1 -- 172.21.15.42:0/2650316802 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbf00005ab0 con 0x7fbf08131cd0 2019-12-05T06:28:29.071 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:29.058+0000 7fbf0d617700 1 -- 172.21.15.42:0/2650316802 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf08074820 con 0x7fbf08131cd0 2019-12-05T06:28:29.071 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:29.058+0000 7fbf0d617700 1 -- 172.21.15.42:0/2650316802 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf08074d70 con 0x7fbf08131cd0 2019-12-05T06:28:29.071 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:29.058+0000 7fbf07fff700 1 --2- 172.21.15.42:0/2650316802 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbf08131cd0 0x7fbf080742d0 secure :-1 s=THROTTLE_DONE pgs=2020 cs=0 l=1 rx=0x7fbf00008b10 tx=0x7fbf00010cd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:29.071 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:29.058+0000 7fbeeffff700 1 -- 172.21.15.42:0/2650316802 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fbf0000f280 con 0x7fbf08131cd0 2019-12-05T06:28:29.071 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:29.058+0000 7fbf07fff700 1 --2- 172.21.15.42:0/2650316802 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbf08131cd0 0x7fbf080742d0 secure :-1 s=THROTTLE_DONE pgs=2020 cs=0 l=1 rx=0x7fbf00008b10 tx=0x7fbf00010cd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:29.071 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:29.058+0000 7fbeeffff700 1 --2- 172.21.15.42:0/2650316802 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbee803ad00 0x7fbee803d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:28:29.072 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:29.058+0000 7fbeeffff700 1 -- 172.21.15.42:0/2650316802 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 11019+0+0 (secure 0 0 0) 0x7fbf0004b500 con 0x7fbf08131cd0 2019-12-05T06:28:29.072 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:29.062+0000 7fbf077fe700 1 --2- 172.21.15.42:0/2650316802 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbee803ad00 0x7fbee803d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:28:29.072 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:29.062+0000 7fbf077fe700 1 --2- 172.21.15.42:0/2650316802 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbee803ad00 0x7fbee803d210 crc :-1 s=READY pgs=1128 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:28:29.073 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:29.062+0000 7fbf0d617700 1 -- 172.21.15.42:0/2650316802 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf0805d110 con 0x7fbf08131cd0 2019-12-05T06:28:29.074 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:29.070+0000 7fbf07fff700 1 --2- 172.21.15.42:0/2650316802 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbf08131cd0 0x7fbf080742d0 secure :-1 s=THROTTLE_DONE pgs=2020 cs=0 l=1 rx=0x7fbf00008b10 tx=0x7fbf00010cd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:29.074 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:29.070+0000 7fbeeffff700 1 -- 172.21.15.42:0/2650316802 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fbf081186d0 con 0x7fbf08131cd0 2019-12-05T06:28:29.243 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:29.238+0000 7fbf0d617700 1 -- 172.21.15.42:0/2650316802 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbf0805b8c0 con 0x7fbf08131cd0 2019-12-05T06:28:29.244 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:29.242+0000 7fbf07fff700 1 --2- 172.21.15.42:0/2650316802 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbf08131cd0 0x7fbf080742d0 secure :-1 s=THROTTLE_DONE pgs=2020 cs=0 l=1 rx=0x7fbf00008b10 tx=0x7fbf00010cd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:29.244 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:29.242+0000 7fbeeffff700 1 -- 172.21.15.42:0/2650316802 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 321 v321) v1 ==== 95+0+1779 (secure 0 0 0) 0x7fbf00005d90 con 0x7fbf08131cd0 2019-12-05T06:28:29.247 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 321 2019-12-05T06:28:29.247 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:28:29.247 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":321,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":321,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:28:28.960516+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":107,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11142},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11142":{"gid":11142,"name":"b","rank":0,"incarnation":319,"state":"up:rejoin","state_seq":27,"addr":"172.21.15.40:6833/3155851600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3155851600},{"type":"v1","addr":"172.21.15.40:6833","nonce":3155851600}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:28:29.249 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:29.246+0000 7fbf0d617700 1 -- 172.21.15.42:0/2650316802 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbee803ad00 msgr2=0x7fbee803d210 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:29.250 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:29.246+0000 7fbf0d617700 1 --2- 172.21.15.42:0/2650316802 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbee803ad00 0x7fbee803d210 crc :-1 s=READY pgs=1128 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:29.250 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:29.246+0000 7fbf0d617700 1 -- 172.21.15.42:0/2650316802 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbf08131cd0 msgr2=0x7fbf080742d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:29.250 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:29.246+0000 7fbf0d617700 1 --2- 172.21.15.42:0/2650316802 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbf08131cd0 0x7fbf080742d0 secure :-1 s=READY pgs=2020 cs=0 l=1 rx=0x7fbf00008b10 tx=0x7fbf00010cd0).stop 2019-12-05T06:28:29.250 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:29.246+0000 7fbf0d617700 1 -- 172.21.15.42:0/2650316802 shutdown_connections 2019-12-05T06:28:29.250 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:29.246+0000 7fbf0d617700 1 --2- 172.21.15.42:0/2650316802 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbee803ad00 0x7fbee803d210 unknown :-1 s=CLOSED pgs=1128 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:29.250 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:29.246+0000 7fbf0d617700 1 --2- 172.21.15.42:0/2650316802 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbf08131cd0 0x7fbf080742d0 unknown :-1 s=CLOSED pgs=2020 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:29.251 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:29.246+0000 7fbf0d617700 1 --2- 172.21.15.42:0/2650316802 >> v2:172.21.15.42:3300/0 conn(0x7fbf08117b70 0x7fbf081449b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:29.251 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:29.246+0000 7fbf0d617700 1 -- 172.21.15.42:0/2650316802 shutdown_connections 2019-12-05T06:28:29.251 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:29.246+0000 7fbf0d617700 1 -- 172.21.15.42:0/2650316802 wait complete. 2019-12-05T06:28:29.251 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:29.246+0000 7fbf0d617700 1 -- 172.21.15.42:0/2650316802 >> 172.21.15.42:0/2650316802 conn(0x7fbf080a9a90 msgr2=0x7fbf0809f220 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:28:29.262 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2019-12-05T06:28:31.263 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:28:31.372 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:31.361+0000 7f7fe3685700 1 Processor -- start 2019-12-05T06:28:31.372 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:31.361+0000 7f7fe3685700 1 -- start start 2019-12-05T06:28:31.372 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:31.361+0000 7f7fe3685700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f7fdc0a7740 0x7f7fdc08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:28:31.372 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:31.361+0000 7f7fe3685700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7fdc07e720 0x7f7fdc07eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:28:31.372 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:31.361+0000 7f7fe3685700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fdc0a0710 con 0x7f7fdc09f2e0 2019-12-05T06:28:31.373 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:31.361+0000 7f7fe3685700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7fdc0a0b00 con 0x7f7fdc0a7740 2019-12-05T06:28:31.373 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:31.361+0000 7f7fe3685700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fdc08e5b0 con 0x7f7fdc07e720 2019-12-05T06:28:31.373 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:31.365+0000 7f7fe1c22700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f7fdc0a7740 0x7f7fdc08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:28:31.373 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:31.365+0000 7f7fe2423700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f7fdc09f2e0 0x7f7fdc09f710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:55892/0 (socket says 172.21.15.42:55892) 2019-12-05T06:28:31.373 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:31.365+0000 7f7fe2423700 1 -- 172.21.15.42:0/2845421887 learned_addr learned my addr 172.21.15.42:0/2845421887 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:28:31.373 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:31.365+0000 7f7fe1421700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7fdc07e720 0x7f7fdc07eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:28:31.373 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:31.365+0000 7f7fe0c20700 1 -- 172.21.15.42:0/2845421887 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1064175825 0 0) 0x7f7fdc0a0710 con 0x7f7fdc09f2e0 2019-12-05T06:28:31.374 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:31.365+0000 7f7fe1421700 1 -- 172.21.15.42:0/2845421887 >> v2:172.21.15.42:3300/0 conn(0x7f7fdc0a7740 msgr2=0x7f7fdc08df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:28:31.374 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:31.365+0000 7f7fe1421700 1 --2- 172.21.15.42:0/2845421887 >> v2:172.21.15.42:3300/0 conn(0x7f7fdc0a7740 0x7f7fdc08df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:28:31.374 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:31.365+0000 7f7fe1421700 1 -- 172.21.15.42:0/2845421887 >> v1:172.21.15.42:6789/0 conn(0x7f7fdc09f2e0 legacy=0x7f7fdc09f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:31.374 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:31.365+0000 7f7fe1421700 1 -- 172.21.15.42:0/2845421887 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fdc07f0a0 con 0x7f7fdc07e720 2019-12-05T06:28:31.374 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:31.365+0000 7f7fe1421700 1 --2- 172.21.15.42:0/2845421887 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7fdc07e720 0x7f7fdc07eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f7fcc008790 tx=0x7f7fcc00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:31.374 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:31.365+0000 7f7fe1421700 1 --2- 172.21.15.42:0/2845421887 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7fdc07e720 0x7f7fdc07eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f7fcc008790 tx=0x7f7fcc00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:31.374 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:31.365+0000 7f7fe1421700 1 --2- 172.21.15.42:0/2845421887 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7fdc07e720 0x7f7fdc07eb50 secure :-1 s=READY pgs=595 cs=0 l=1 rx=0x7f7fcc008790 tx=0x7f7fcc00fc70).ready entity=mon.2 client_cookie=18ea5f45a280b772 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:28:31.375 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:31.365+0000 7f7fe1421700 1 --2- 172.21.15.42:0/2845421887 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7fdc07e720 0x7f7fdc07eb50 secure :-1 s=THROTTLE_DONE pgs=595 cs=0 l=1 rx=0x7f7fcc008790 tx=0x7f7fcc00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:31.375 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:31.365+0000 7f7fe0c20700 1 -- 172.21.15.42:0/2845421887 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7fcc0112f0 con 0x7f7fdc07e720 2019-12-05T06:28:31.375 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:31.365+0000 7f7fe1421700 1 --2- 172.21.15.42:0/2845421887 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7fdc07e720 0x7f7fdc07eb50 secure :-1 s=THROTTLE_DONE pgs=595 cs=0 l=1 rx=0x7f7fcc008790 tx=0x7f7fcc00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:31.375 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:31.365+0000 7f7fe1421700 1 --2- 172.21.15.42:0/2845421887 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7fdc07e720 0x7f7fdc07eb50 secure :-1 s=THROTTLE_DONE pgs=595 cs=0 l=1 rx=0x7f7fcc008790 tx=0x7f7fcc00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:31.375 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:31.365+0000 7f7fe0c20700 1 -- 172.21.15.42:0/2845421887 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fcc008ad0 con 0x7f7fdc07e720 2019-12-05T06:28:31.375 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:31.365+0000 7f7fe0c20700 1 -- 172.21.15.42:0/2845421887 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7fcc0118d0 con 0x7f7fdc07e720 2019-12-05T06:28:31.376 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:31.365+0000 7f7fe3685700 1 -- 172.21.15.42:0/2845421887 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7fdc07e720 msgr2=0x7f7fdc07eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:31.376 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:31.365+0000 7f7fe3685700 1 --2- 172.21.15.42:0/2845421887 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7fdc07e720 0x7f7fdc07eb50 secure :-1 s=READY pgs=595 cs=0 l=1 rx=0x7f7fcc008790 tx=0x7f7fcc00fc70).stop 2019-12-05T06:28:31.376 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:31.365+0000 7f7fe3685700 1 -- 172.21.15.42:0/2845421887 shutdown_connections 2019-12-05T06:28:31.376 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:31.365+0000 7f7fe3685700 1 --2- 172.21.15.42:0/2845421887 >> v2:172.21.15.42:3300/0 conn(0x7f7fdc0a7740 0x7f7fdc08df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:28:31.376 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:31.365+0000 7f7fe3685700 1 --2- 172.21.15.42:0/2845421887 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7fdc07e720 0x7f7fdc07eb50 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:31.376 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:31.365+0000 7f7fe3685700 1 -- 172.21.15.42:0/2845421887 shutdown_connections 2019-12-05T06:28:31.377 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:31.365+0000 7f7fe3685700 1 -- 172.21.15.42:0/2845421887 wait complete. 2019-12-05T06:28:31.377 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:31.365+0000 7f7fe3685700 1 -- 172.21.15.42:0/2845421887 >> 172.21.15.42:0/2845421887 conn(0x7f7fdc0aabe0 msgr2=0x7f7fdc0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:28:31.381 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:31.369+0000 7f7fe3685700 1 Processor -- start 2019-12-05T06:28:31.381 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:31.369+0000 7f7fe3685700 1 -- start start 2019-12-05T06:28:31.381 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:31.369+0000 7f7fe3685700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f7fdc11b790 0x7f7fdc11e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:28:31.382 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:31.369+0000 7f7fe3685700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7fdc120830 0x7f7fdc11fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:28:31.382 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:31.369+0000 7f7fe3685700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7fdc120070 0x7f7fdc130090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:28:31.382 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:31.369+0000 7f7fe3685700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7fdc11f040 con 0x7f7fdc120830 2019-12-05T06:28:31.382 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:31.369+0000 7f7fe3685700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7fdc11bc60 con 0x7f7fdc11b790 2019-12-05T06:28:31.382 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:31.369+0000 7f7fe3685700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fdc11bf00 con 0x7f7fdc120070 2019-12-05T06:28:31.382 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:31.369+0000 7f7fe1c22700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f7fdc11b790 0x7f7fdc11e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:28:31.382 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:31.369+0000 7f7fe1c22700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f7fdc11b790 0x7f7fdc11e7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:55856/0 (socket says 172.21.15.42:55856) 2019-12-05T06:28:31.383 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:31.369+0000 7f7fe1c22700 1 -- 172.21.15.42:0/2424422974 learned_addr learned my addr 172.21.15.42:0/2424422974 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:28:31.383 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:31.369+0000 7f7fe2423700 1 --2- 172.21.15.42:0/2424422974 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7fdc120070 0x7f7fdc130090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:28:31.383 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:31.369+0000 7f7fe1421700 1 --2- 172.21.15.42:0/2424422974 >> v2:172.21.15.40:3300/0 conn(0x7f7fdc120830 0x7f7fdc11fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:28:31.383 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:31.369+0000 7f7fe1c22700 1 -- 172.21.15.42:0/2424422974 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7fdc120070 msgr2=0x7f7fdc130090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:31.383 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:31.369+0000 7f7fe1c22700 1 --2- 172.21.15.42:0/2424422974 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7fdc120070 0x7f7fdc130090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:31.383 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:31.369+0000 7f7fe1c22700 1 -- 172.21.15.42:0/2424422974 >> v2:172.21.15.40:3300/0 conn(0x7f7fdc120830 msgr2=0x7f7fdc11fb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:31.384 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:31.369+0000 7f7fe1c22700 1 --2- 172.21.15.42:0/2424422974 >> v2:172.21.15.40:3300/0 conn(0x7f7fdc120830 0x7f7fdc11fb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:31.384 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:31.369+0000 7f7fe1c22700 1 -- 172.21.15.42:0/2424422974 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fdc11dbe0 con 0x7f7fdc11b790 2019-12-05T06:28:31.384 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:31.369+0000 7f7fe1c22700 1 --2- 172.21.15.42:0/2424422974 >> v2:172.21.15.42:3300/0 conn(0x7f7fdc11b790 0x7f7fdc11e7a0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f7fd8003070 tx=0x7f7fd8010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:31.384 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:31.369+0000 7f7fe1c22700 1 --2- 172.21.15.42:0/2424422974 >> v2:172.21.15.42:3300/0 conn(0x7f7fdc11b790 0x7f7fdc11e7a0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f7fd8003070 tx=0x7f7fd8010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:31.384 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:31.369+0000 7f7fe1c22700 1 --2- 172.21.15.42:0/2424422974 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7fdc11b790 0x7f7fdc11e7a0 secure :-1 s=READY pgs=2694 cs=0 l=1 rx=0x7f7fd8003070 tx=0x7f7fd8010b60).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:28:31.384 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:31.369+0000 7f7fe1c22700 1 --2- 172.21.15.42:0/2424422974 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7fdc11b790 0x7f7fdc11e7a0 secure :-1 s=THROTTLE_DONE pgs=2694 cs=0 l=1 rx=0x7f7fd8003070 tx=0x7f7fd8010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:31.385 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:31.369+0000 7f7fd27fc700 1 -- 172.21.15.42:0/2424422974 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7fd8005920 con 0x7f7fdc11b790 2019-12-05T06:28:31.385 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:31.369+0000 7f7fe1c22700 1 --2- 172.21.15.42:0/2424422974 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7fdc11b790 0x7f7fdc11e7a0 secure :-1 s=THROTTLE_DONE pgs=2694 cs=0 l=1 rx=0x7f7fd8003070 tx=0x7f7fd8010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:31.385 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:31.373+0000 7f7fe1c22700 1 --2- 172.21.15.42:0/2424422974 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7fdc11b790 0x7f7fdc11e7a0 secure :-1 s=THROTTLE_DONE pgs=2694 cs=0 l=1 rx=0x7f7fd8003070 tx=0x7f7fd8010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:31.385 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:31.373+0000 7f7fd27fc700 1 -- 172.21.15.42:0/2424422974 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fd8005540 con 0x7f7fdc11b790 2019-12-05T06:28:31.385 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:31.373+0000 7f7fe3685700 1 -- 172.21.15.42:0/2424422974 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fdc11a260 con 0x7f7fdc11b790 2019-12-05T06:28:31.385 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:31.373+0000 7f7fd27fc700 1 -- 172.21.15.42:0/2424422974 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7fd800e440 con 0x7f7fdc11b790 2019-12-05T06:28:31.386 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:31.373+0000 7f7fe3685700 1 -- 172.21.15.42:0/2424422974 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fdc07de60 con 0x7f7fdc11b790 2019-12-05T06:28:31.386 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:31.373+0000 7f7fe1c22700 1 --2- 172.21.15.42:0/2424422974 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7fdc11b790 0x7f7fdc11e7a0 secure :-1 s=THROTTLE_DONE pgs=2694 cs=0 l=1 rx=0x7f7fd8003070 tx=0x7f7fd8010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:31.386 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:31.373+0000 7f7fd27fc700 1 -- 172.21.15.42:0/2424422974 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f7fd800e750 con 0x7f7fdc11b790 2019-12-05T06:28:31.386 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:31.373+0000 7f7fe1c22700 1 --2- 172.21.15.42:0/2424422974 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7fdc11b790 0x7f7fdc11e7a0 secure :-1 s=THROTTLE_DONE pgs=2694 cs=0 l=1 rx=0x7f7fd8003070 tx=0x7f7fd8010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:31.386 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:31.373+0000 7f7fd27fc700 1 --2- 172.21.15.42:0/2424422974 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f7fc403ad40 0x7f7fc403d250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:28:31.386 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:31.373+0000 7f7fd27fc700 1 -- 172.21.15.42:0/2424422974 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 11019+0+0 (secure 0 0 0) 0x7f7fd804ada0 con 0x7f7fdc11b790 2019-12-05T06:28:31.387 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:31.373+0000 7f7fe1421700 1 --2- 172.21.15.42:0/2424422974 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f7fc403ad40 0x7f7fc403d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:28:31.387 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:31.373+0000 7f7fe1421700 1 --2- 172.21.15.42:0/2424422974 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f7fc403ad40 0x7f7fc403d250 crc :-1 s=READY pgs=1129 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:28:31.387 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:31.377+0000 7f7fe3685700 1 -- 172.21.15.42:0/2424422974 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fdc05d110 con 0x7f7fdc11b790 2019-12-05T06:28:31.389 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:31.385+0000 7f7fe1c22700 1 --2- 172.21.15.42:0/2424422974 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7fdc11b790 0x7f7fdc11e7a0 secure :-1 s=THROTTLE_DONE pgs=2694 cs=0 l=1 rx=0x7f7fd8003070 tx=0x7f7fd8010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:31.389 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:31.385+0000 7f7fd27fc700 1 -- 172.21.15.42:0/2424422974 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f7fd8031e20 con 0x7f7fdc11b790 2019-12-05T06:28:31.565 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:31.561+0000 7f7fe3685700 1 -- 172.21.15.42:0/2424422974 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7fdc05b8c0 con 0x7f7fdc11b790 2019-12-05T06:28:31.566 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:31.561+0000 7f7fe1c22700 1 --2- 172.21.15.42:0/2424422974 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7fdc11b790 0x7f7fdc11e7a0 secure :-1 s=THROTTLE_DONE pgs=2694 cs=0 l=1 rx=0x7f7fd8003070 tx=0x7f7fd8010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:31.567 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:31.561+0000 7f7fd27fc700 1 -- 172.21.15.42:0/2424422974 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 322 v322) v1 ==== 95+0+1779 (secure 0 0 0) 0x7f7fd8032080 con 0x7f7fdc11b790 2019-12-05T06:28:31.567 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 322 2019-12-05T06:28:31.567 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:28:31.567 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":322,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":322,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:28:29.970815+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":107,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11142},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11142":{"gid":11142,"name":"b","rank":0,"incarnation":319,"state":"up:active","state_seq":28,"addr":"172.21.15.40:6833/3155851600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3155851600},{"type":"v1","addr":"172.21.15.40:6833","nonce":3155851600}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:28:31.570 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:31.565+0000 7f7fe3685700 1 -- 172.21.15.42:0/2424422974 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f7fc403ad40 msgr2=0x7f7fc403d250 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:31.570 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:31.565+0000 7f7fe3685700 1 --2- 172.21.15.42:0/2424422974 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f7fc403ad40 0x7f7fc403d250 crc :-1 s=READY pgs=1129 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:31.570 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:31.565+0000 7f7fe3685700 1 -- 172.21.15.42:0/2424422974 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7fdc11b790 msgr2=0x7f7fdc11e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:31.570 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:31.565+0000 7f7fe3685700 1 --2- 172.21.15.42:0/2424422974 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7fdc11b790 0x7f7fdc11e7a0 secure :-1 s=READY pgs=2694 cs=0 l=1 rx=0x7f7fd8003070 tx=0x7f7fd8010b60).stop 2019-12-05T06:28:31.570 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:31.565+0000 7f7fe3685700 1 -- 172.21.15.42:0/2424422974 shutdown_connections 2019-12-05T06:28:31.570 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:31.565+0000 7f7fe3685700 1 --2- 172.21.15.42:0/2424422974 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f7fc403ad40 0x7f7fc403d250 unknown :-1 s=CLOSED pgs=1129 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:31.570 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:31.565+0000 7f7fe3685700 1 --2- 172.21.15.42:0/2424422974 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7fdc120070 0x7f7fdc130090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:31.571 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:31.565+0000 7f7fe3685700 1 --2- 172.21.15.42:0/2424422974 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7fdc11b790 0x7f7fdc11e7a0 unknown :-1 s=CLOSED pgs=2694 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:31.571 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:31.565+0000 7f7fe3685700 1 --2- 172.21.15.42:0/2424422974 >> v2:172.21.15.40:3300/0 conn(0x7f7fdc120830 0x7f7fdc11fb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:31.571 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:31.565+0000 7f7fe3685700 1 -- 172.21.15.42:0/2424422974 shutdown_connections 2019-12-05T06:28:31.571 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:31.565+0000 7f7fe3685700 1 -- 172.21.15.42:0/2424422974 wait complete. 2019-12-05T06:28:31.571 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:31.565+0000 7f7fe3685700 1 -- 172.21.15.42:0/2424422974 >> 172.21.15.42:0/2424422974 conn(0x7f7fdc0a9c30 msgr2=0x7f7fdc0a8040 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:28:31.579 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b has gained rank=0, replacing gid=11221 2019-12-05T06:28:31.580 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 9 secs before reviving mds.a 2019-12-05T06:28:40.581 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.a 2019-12-05T06:28:40.581 INFO:tasks.ceph.mds.a:Restarting daemon 2019-12-05T06:28:40.581 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2019-12-05T06:28:40.585 INFO:tasks.ceph.mds.a:Started 2019-12-05T06:28:40.585 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:28:40.645 INFO:tasks.ceph.mds.a.smithi042.stdout:starting mds.a at 2019-12-05T06:28:40.688 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:40.677+0000 7f35b2422700 1 Processor -- start 2019-12-05T06:28:40.688 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:40.677+0000 7f35b2422700 1 -- start start 2019-12-05T06:28:40.688 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:40.681+0000 7f35b2422700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f35ac09fad0 0x7f35ac14ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:28:40.688 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:40.681+0000 7f35b2422700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35ac0a0700 con 0x7f35ac0a7730 2019-12-05T06:28:40.688 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:40.681+0000 7f35b2422700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35ac07ef10 con 0x7f35ac08df70 2019-12-05T06:28:40.689 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:40.681+0000 7f35b2422700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35ac0a0af0 con 0x7f35ac09fad0 2019-12-05T06:28:40.689 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:40.681+0000 7f35abfff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f35ac08df70 0x7f35ac09f2d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:55956/0 (socket says 172.21.15.42:55956) 2019-12-05T06:28:40.689 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:40.681+0000 7f35abfff700 1 -- 172.21.15.42:0/4266763072 learned_addr learned my addr 172.21.15.42:0/4266763072 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:28:40.689 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:40.681+0000 7f35b11c0700 1 --2- 172.21.15.42:0/4266763072 >> v2:172.21.15.42:3300/0 conn(0x7f35ac09fad0 0x7f35ac14ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:28:40.689 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:40.681+0000 7f35ab7fe700 1 -- 172.21.15.42:0/4266763072 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3200506833 0 0) 0x7f35ac07ef10 con 0x7f35ac08df70 2019-12-05T06:28:40.689 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:40.681+0000 7f35ab7fe700 1 -- 172.21.15.42:0/4266763072 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f359c0030b0 con 0x7f35ac08df70 2019-12-05T06:28:40.689 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:40.681+0000 7f35ab7fe700 1 -- 172.21.15.42:0/4266763072 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1907624893 0 0) 0x7f35ac0a0700 con 0x7f35ac0a7730 2019-12-05T06:28:40.690 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:40.681+0000 7f35ab7fe700 1 -- 172.21.15.42:0/4266763072 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35ac07ef10 con 0x7f35ac0a7730 2019-12-05T06:28:40.690 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:40.681+0000 7f35b11c0700 1 -- 172.21.15.42:0/4266763072 >> v1:172.21.15.42:6789/0 conn(0x7f35ac08df70 legacy=0x7f35ac09f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:40.690 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:40.681+0000 7f35b11c0700 1 -- 172.21.15.42:0/4266763072 >> v1:172.21.15.40:6789/0 conn(0x7f35ac0a7730 legacy=0x7f35ac07e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:40.690 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:40.681+0000 7f35b11c0700 1 -- 172.21.15.42:0/4266763072 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35ac0a9b90 con 0x7f35ac09fad0 2019-12-05T06:28:40.690 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:40.681+0000 7f35b11c0700 1 --2- 172.21.15.42:0/4266763072 >> v2:172.21.15.42:3300/0 conn(0x7f35ac09fad0 0x7f35ac14ffe0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f35a4008b10 tx=0x7f35a400ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:40.690 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:40.681+0000 7f35b11c0700 1 --2- 172.21.15.42:0/4266763072 >> v2:172.21.15.42:3300/0 conn(0x7f35ac09fad0 0x7f35ac14ffe0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f35a4008b10 tx=0x7f35a400ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:40.690 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:40.681+0000 7f35b11c0700 1 --2- 172.21.15.42:0/4266763072 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f35ac09fad0 0x7f35ac14ffe0 secure :-1 s=READY pgs=2699 cs=0 l=1 rx=0x7f35a4008b10 tx=0x7f35a400ff70).ready entity=mon.1 client_cookie=bc654637c837c2a0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:28:40.691 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:40.681+0000 7f35b11c0700 1 --2- 172.21.15.42:0/4266763072 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f35ac09fad0 0x7f35ac14ffe0 secure :-1 s=THROTTLE_DONE pgs=2699 cs=0 l=1 rx=0x7f35a4008b10 tx=0x7f35a400ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:40.691 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:40.681+0000 7f35ab7fe700 1 -- 172.21.15.42:0/4266763072 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f35a4010970 con 0x7f35ac09fad0 2019-12-05T06:28:40.691 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:40.681+0000 7f35b11c0700 1 --2- 172.21.15.42:0/4266763072 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f35ac09fad0 0x7f35ac14ffe0 secure :-1 s=THROTTLE_DONE pgs=2699 cs=0 l=1 rx=0x7f35a4008b10 tx=0x7f35a400ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:40.691 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:40.681+0000 7f35b11c0700 1 --2- 172.21.15.42:0/4266763072 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f35ac09fad0 0x7f35ac14ffe0 secure :-1 s=THROTTLE_DONE pgs=2699 cs=0 l=1 rx=0x7f35a4008b10 tx=0x7f35a400ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:40.691 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:40.681+0000 7f35ab7fe700 1 -- 172.21.15.42:0/4266763072 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35a4008b50 con 0x7f35ac09fad0 2019-12-05T06:28:40.691 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:40.681+0000 7f35ab7fe700 1 -- 172.21.15.42:0/4266763072 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f35a40110c0 con 0x7f35ac09fad0 2019-12-05T06:28:40.691 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:40.681+0000 7f35b2422700 1 -- 172.21.15.42:0/4266763072 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f35ac09fad0 msgr2=0x7f35ac14ffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:40.692 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:40.681+0000 7f35b2422700 1 --2- 172.21.15.42:0/4266763072 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f35ac09fad0 0x7f35ac14ffe0 secure :-1 s=READY pgs=2699 cs=0 l=1 rx=0x7f35a4008b10 tx=0x7f35a400ff70).stop 2019-12-05T06:28:40.692 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:40.681+0000 7f35b2422700 1 -- 172.21.15.42:0/4266763072 shutdown_connections 2019-12-05T06:28:40.692 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:40.681+0000 7f35b2422700 1 --2- 172.21.15.42:0/4266763072 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f35ac09fad0 0x7f35ac14ffe0 unknown :-1 s=CLOSED pgs=2699 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:40.692 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:40.681+0000 7f35b2422700 1 -- 172.21.15.42:0/4266763072 shutdown_connections 2019-12-05T06:28:40.692 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:40.681+0000 7f35b2422700 1 -- 172.21.15.42:0/4266763072 wait complete. 2019-12-05T06:28:40.692 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:40.681+0000 7f35b2422700 1 -- 172.21.15.42:0/4266763072 >> 172.21.15.42:0/4266763072 conn(0x7f35ac0aabd0 msgr2=0x7f35ac0a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:28:40.696 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:40.685+0000 7f35b2422700 1 Processor -- start 2019-12-05T06:28:40.696 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:40.685+0000 7f35b2422700 1 -- start start 2019-12-05T06:28:40.696 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:40.685+0000 7f35b2422700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f35ac0a1920 0x7f35ac144dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:28:40.696 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:40.685+0000 7f35b2422700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f35ac0a3e50 0x7f35ac0a2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:28:40.696 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:40.685+0000 7f35b2422700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35ac0a4280 con 0x7f35ac08eea0 2019-12-05T06:28:40.696 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:40.685+0000 7f35b2422700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35ac1454d0 con 0x7f35ac0a3e50 2019-12-05T06:28:40.697 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:40.685+0000 7f35b2422700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35ac0a3bb0 con 0x7f35ac0a1920 2019-12-05T06:28:40.697 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:40.685+0000 7f35abfff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f35ac08eea0 0x7f35ac143770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:55962/0 (socket says 172.21.15.42:55962) 2019-12-05T06:28:40.697 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:40.685+0000 7f35abfff700 1 -- 172.21.15.42:0/1559091325 learned_addr learned my addr 172.21.15.42:0/1559091325 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:28:40.697 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:40.685+0000 7f35b09bf700 1 --2- 172.21.15.42:0/1559091325 >> v2:172.21.15.42:3300/0 conn(0x7f35ac0a1920 0x7f35ac144dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:28:40.697 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:40.685+0000 7f35b11c0700 1 --2- 172.21.15.42:0/1559091325 >> v2:172.21.15.40:3300/0 conn(0x7f35ac0a3e50 0x7f35ac0a2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:28:40.697 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:40.685+0000 7f35a97fa700 1 -- 172.21.15.42:0/1559091325 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2624420117 0 0) 0x7f35ac0a4280 con 0x7f35ac08eea0 2019-12-05T06:28:40.698 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:40.685+0000 7f35a97fa700 1 -- 172.21.15.42:0/1559091325 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f358c0030b0 con 0x7f35ac08eea0 2019-12-05T06:28:40.698 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:40.685+0000 7f35b11c0700 1 -- 172.21.15.42:0/1559091325 >> v2:172.21.15.42:3300/0 conn(0x7f35ac0a1920 msgr2=0x7f35ac144dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:40.698 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:40.685+0000 7f35b11c0700 1 --2- 172.21.15.42:0/1559091325 >> v2:172.21.15.42:3300/0 conn(0x7f35ac0a1920 0x7f35ac144dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:40.698 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:40.685+0000 7f35b11c0700 1 -- 172.21.15.42:0/1559091325 >> v1:172.21.15.42:6789/0 conn(0x7f35ac08eea0 legacy=0x7f35ac143770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:40.698 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:40.685+0000 7f35b11c0700 1 -- 172.21.15.42:0/1559091325 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35ac0a3870 con 0x7f35ac0a3e50 2019-12-05T06:28:40.698 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:40.685+0000 7f35b09bf700 1 --2- 172.21.15.42:0/1559091325 >> v2:172.21.15.42:3300/0 conn(0x7f35ac0a1920 0x7f35ac144dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:28:40.698 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:40.685+0000 7f35b11c0700 1 --2- 172.21.15.42:0/1559091325 >> v2:172.21.15.40:3300/0 conn(0x7f35ac0a3e50 0x7f35ac0a2080 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f35ac145690 tx=0x7f35a4010de0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:40.699 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:40.685+0000 7f35b11c0700 1 --2- 172.21.15.42:0/1559091325 >> v2:172.21.15.40:3300/0 conn(0x7f35ac0a3e50 0x7f35ac0a2080 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f35ac145690 tx=0x7f35a4010de0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:40.699 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:40.685+0000 7f35b11c0700 1 --2- 172.21.15.42:0/1559091325 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f35ac0a3e50 0x7f35ac0a2080 secure :-1 s=READY pgs=2024 cs=0 l=1 rx=0x7f35ac145690 tx=0x7f35a4010de0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:28:40.699 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:40.685+0000 7f35b11c0700 1 --2- 172.21.15.42:0/1559091325 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f35ac0a3e50 0x7f35ac0a2080 secure :-1 s=THROTTLE_DONE pgs=2024 cs=0 l=1 rx=0x7f35ac145690 tx=0x7f35a4010de0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:40.699 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:40.685+0000 7f35a97fa700 1 -- 172.21.15.42:0/1559091325 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f35a4005ab0 con 0x7f35ac0a3e50 2019-12-05T06:28:40.699 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:40.685+0000 7f35b11c0700 1 --2- 172.21.15.42:0/1559091325 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f35ac0a3e50 0x7f35ac0a2080 secure :-1 s=THROTTLE_DONE pgs=2024 cs=0 l=1 rx=0x7f35ac145690 tx=0x7f35a4010de0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:40.699 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:40.685+0000 7f35b11c0700 1 --2- 172.21.15.42:0/1559091325 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f35ac0a3e50 0x7f35ac0a2080 secure :-1 s=THROTTLE_DONE pgs=2024 cs=0 l=1 rx=0x7f35ac145690 tx=0x7f35a4010de0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:40.699 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:40.685+0000 7f35a97fa700 1 -- 172.21.15.42:0/1559091325 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35a4010b70 con 0x7f35ac0a3e50 2019-12-05T06:28:40.699 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:40.685+0000 7f35b2422700 1 -- 172.21.15.42:0/1559091325 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35ac143f70 con 0x7f35ac0a3e50 2019-12-05T06:28:40.700 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:40.685+0000 7f35a97fa700 1 -- 172.21.15.42:0/1559091325 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f35a400f470 con 0x7f35ac0a3e50 2019-12-05T06:28:40.700 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:40.685+0000 7f35b2422700 1 -- 172.21.15.42:0/1559091325 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35ac144320 con 0x7f35ac0a3e50 2019-12-05T06:28:40.700 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:40.689+0000 7f35b11c0700 1 --2- 172.21.15.42:0/1559091325 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f35ac0a3e50 0x7f35ac0a2080 secure :-1 s=THROTTLE_DONE pgs=2024 cs=0 l=1 rx=0x7f35ac145690 tx=0x7f35a4010de0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:40.700 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:40.689+0000 7f35a97fa700 1 -- 172.21.15.42:0/1559091325 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f35a400f780 con 0x7f35ac0a3e50 2019-12-05T06:28:40.700 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:40.689+0000 7f35b11c0700 1 --2- 172.21.15.42:0/1559091325 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f35ac0a3e50 0x7f35ac0a2080 secure :-1 s=THROTTLE_DONE pgs=2024 cs=0 l=1 rx=0x7f35ac145690 tx=0x7f35a4010de0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:40.700 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:40.689+0000 7f35a97fa700 1 --2- 172.21.15.42:0/1559091325 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f358c03b2b0 0x7f358c03d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:28:40.700 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:40.689+0000 7f35a97fa700 1 -- 172.21.15.42:0/1559091325 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 11019+0+0 (secure 0 0 0) 0x7f35a404aef0 con 0x7f35ac0a3e50 2019-12-05T06:28:40.701 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:40.689+0000 7f35b09bf700 1 --2- 172.21.15.42:0/1559091325 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f358c03b2b0 0x7f358c03d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:28:40.701 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:40.689+0000 7f35b09bf700 1 --2- 172.21.15.42:0/1559091325 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f358c03b2b0 0x7f358c03d7c0 crc :-1 s=READY pgs=1130 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:28:40.701 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:40.693+0000 7f35b2422700 1 -- 172.21.15.42:0/1559091325 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35ac1445f0 con 0x7f35ac0a3e50 2019-12-05T06:28:40.709 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:40.705+0000 7f35b11c0700 1 --2- 172.21.15.42:0/1559091325 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f35ac0a3e50 0x7f35ac0a2080 secure :-1 s=THROTTLE_DONE pgs=2024 cs=0 l=1 rx=0x7f35ac145690 tx=0x7f35a4010de0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:40.709 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:40.705+0000 7f35a97fa700 1 -- 172.21.15.42:0/1559091325 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f35a4030bc0 con 0x7f35ac0a3e50 2019-12-05T06:28:40.878 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:40.873+0000 7f35b2422700 1 -- 172.21.15.42:0/1559091325 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f35ac05d100 con 0x7f35ac0a3e50 2019-12-05T06:28:40.879 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:40.873+0000 7f35b11c0700 1 --2- 172.21.15.42:0/1559091325 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f35ac0a3e50 0x7f35ac0a2080 secure :-1 s=THROTTLE_DONE pgs=2024 cs=0 l=1 rx=0x7f35ac145690 tx=0x7f35a4010de0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:40.879 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:40.873+0000 7f35a97fa700 1 -- 172.21.15.42:0/1559091325 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 322 v322) v1 ==== 95+0+1779 (secure 0 0 0) 0x7f35a4030de0 con 0x7f35ac0a3e50 2019-12-05T06:28:40.879 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 322 2019-12-05T06:28:40.880 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:28:40.880 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":322,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":322,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:28:29.970815+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":107,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11142},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11142":{"gid":11142,"name":"b","rank":0,"incarnation":319,"state":"up:active","state_seq":28,"addr":"172.21.15.40:6833/3155851600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3155851600},{"type":"v1","addr":"172.21.15.40:6833","nonce":3155851600}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:28:40.883 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:40.877+0000 7f35b2422700 1 -- 172.21.15.42:0/1559091325 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f358c03b2b0 msgr2=0x7f358c03d7c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:40.883 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:40.877+0000 7f35b2422700 1 --2- 172.21.15.42:0/1559091325 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f358c03b2b0 0x7f358c03d7c0 crc :-1 s=READY pgs=1130 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:40.883 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:40.877+0000 7f35b2422700 1 -- 172.21.15.42:0/1559091325 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f35ac0a3e50 msgr2=0x7f35ac0a2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:40.883 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:40.877+0000 7f35b2422700 1 --2- 172.21.15.42:0/1559091325 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f35ac0a3e50 0x7f35ac0a2080 secure :-1 s=READY pgs=2024 cs=0 l=1 rx=0x7f35ac145690 tx=0x7f35a4010de0).stop 2019-12-05T06:28:40.884 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:40.877+0000 7f35b2422700 1 -- 172.21.15.42:0/1559091325 shutdown_connections 2019-12-05T06:28:40.884 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:40.877+0000 7f35b2422700 1 --2- 172.21.15.42:0/1559091325 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f358c03b2b0 0x7f358c03d7c0 unknown :-1 s=CLOSED pgs=1130 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:40.884 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:40.877+0000 7f35b2422700 1 --2- 172.21.15.42:0/1559091325 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f35ac0a3e50 0x7f35ac0a2080 unknown :-1 s=CLOSED pgs=2024 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:40.884 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:40.877+0000 7f35b2422700 1 --2- 172.21.15.42:0/1559091325 >> v2:172.21.15.42:3300/0 conn(0x7f35ac0a1920 0x7f35ac144dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:40.884 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:40.877+0000 7f35b2422700 1 -- 172.21.15.42:0/1559091325 shutdown_connections 2019-12-05T06:28:40.884 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:40.877+0000 7f35b2422700 1 -- 172.21.15.42:0/1559091325 wait complete. 2019-12-05T06:28:40.884 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:40.877+0000 7f35b2422700 1 -- 172.21.15.42:0/1559091325 >> 172.21.15.42:0/1559091325 conn(0x7f35ac0a7730 msgr2=0x7f35ac07e4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:28:40.893 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2019-12-05T06:28:42.894 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:28:43.005 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:42.993+0000 7f107494e700 1 Processor -- start 2019-12-05T06:28:43.005 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:42.993+0000 7f107494e700 1 -- start start 2019-12-05T06:28:43.006 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:42.997+0000 7f107494e700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f107009fae0 0x7f107014fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:28:43.006 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:42.997+0000 7f107494e700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10700a0710 con 0x7f10700a7740 2019-12-05T06:28:43.006 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:42.997+0000 7f107494e700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f107007ef20 con 0x7f107008df80 2019-12-05T06:28:43.006 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:42.997+0000 7f107494e700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10700a0b00 con 0x7f107009fae0 2019-12-05T06:28:43.006 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:42.997+0000 7f106659c700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f107008df80 0x7f107009f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:55994/0 (socket says 172.21.15.42:55994) 2019-12-05T06:28:43.006 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:42.997+0000 7f106659c700 1 -- 172.21.15.42:0/2572017593 learned_addr learned my addr 172.21.15.42:0/2572017593 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:28:43.006 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:42.997+0000 7f106f59e700 1 --2- 172.21.15.42:0/2572017593 >> v2:172.21.15.40:3300/0 conn(0x7f107009fae0 0x7f107014fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:28:43.007 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:42.997+0000 7f106e59c700 1 -- 172.21.15.42:0/2572017593 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3070683530 0 0) 0x7f107007ef20 con 0x7f107008df80 2019-12-05T06:28:43.007 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:42.997+0000 7f106e59c700 1 -- 172.21.15.42:0/2572017593 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f105c0030b0 con 0x7f107008df80 2019-12-05T06:28:43.007 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:42.997+0000 7f106e59c700 1 -- 172.21.15.42:0/2572017593 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4178784543 0 0) 0x7f10700a0710 con 0x7f10700a7740 2019-12-05T06:28:43.007 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:42.997+0000 7f106e59c700 1 -- 172.21.15.42:0/2572017593 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f107007ef20 con 0x7f10700a7740 2019-12-05T06:28:43.007 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:42.997+0000 7f106f59e700 1 -- 172.21.15.42:0/2572017593 >> v1:172.21.15.42:6789/0 conn(0x7f107008df80 legacy=0x7f107009f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:43.007 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:42.997+0000 7f106f59e700 1 -- 172.21.15.42:0/2572017593 >> v1:172.21.15.40:6789/0 conn(0x7f10700a7740 legacy=0x7f107007e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:43.008 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:42.997+0000 7f106f59e700 1 -- 172.21.15.42:0/2572017593 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10700a9ba0 con 0x7f107009fae0 2019-12-05T06:28:43.008 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:42.997+0000 7f106f59e700 1 --2- 172.21.15.42:0/2572017593 >> v2:172.21.15.40:3300/0 conn(0x7f107009fae0 0x7f107014fff0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f1068008790 tx=0x7f106800fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:43.008 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:42.997+0000 7f106f59e700 1 --2- 172.21.15.42:0/2572017593 >> v2:172.21.15.40:3300/0 conn(0x7f107009fae0 0x7f107014fff0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f1068008790 tx=0x7f106800fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:43.008 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:42.997+0000 7f106f59e700 1 --2- 172.21.15.42:0/2572017593 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f107009fae0 0x7f107014fff0 secure :-1 s=READY pgs=2026 cs=0 l=1 rx=0x7f1068008790 tx=0x7f106800fbf0).ready entity=mon.0 client_cookie=adee73f0cc449644 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:28:43.008 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:42.997+0000 7f106f59e700 1 --2- 172.21.15.42:0/2572017593 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f107009fae0 0x7f107014fff0 secure :-1 s=THROTTLE_DONE pgs=2026 cs=0 l=1 rx=0x7f1068008790 tx=0x7f106800fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:43.008 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:42.997+0000 7f106e59c700 1 -- 172.21.15.42:0/2572017593 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1068008bb0 con 0x7f107009fae0 2019-12-05T06:28:43.008 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:42.997+0000 7f106f59e700 1 --2- 172.21.15.42:0/2572017593 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f107009fae0 0x7f107014fff0 secure :-1 s=THROTTLE_DONE pgs=2026 cs=0 l=1 rx=0x7f1068008790 tx=0x7f106800fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:43.009 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:42.997+0000 7f106f59e700 1 --2- 172.21.15.42:0/2572017593 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f107009fae0 0x7f107014fff0 secure :-1 s=THROTTLE_DONE pgs=2026 cs=0 l=1 rx=0x7f1068008790 tx=0x7f106800fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:43.009 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:42.997+0000 7f106e59c700 1 -- 172.21.15.42:0/2572017593 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10680087d0 con 0x7f107009fae0 2019-12-05T06:28:43.009 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:42.997+0000 7f106e59c700 1 -- 172.21.15.42:0/2572017593 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f10680118d0 con 0x7f107009fae0 2019-12-05T06:28:43.009 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:42.997+0000 7f107494e700 1 -- 172.21.15.42:0/2572017593 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f107009fae0 msgr2=0x7f107014fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:43.009 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:42.997+0000 7f107494e700 1 --2- 172.21.15.42:0/2572017593 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f107009fae0 0x7f107014fff0 secure :-1 s=READY pgs=2026 cs=0 l=1 rx=0x7f1068008790 tx=0x7f106800fbf0).stop 2019-12-05T06:28:43.009 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:43.001+0000 7f107494e700 1 -- 172.21.15.42:0/2572017593 shutdown_connections 2019-12-05T06:28:43.009 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:43.001+0000 7f107494e700 1 --2- 172.21.15.42:0/2572017593 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f107009fae0 0x7f107014fff0 unknown :-1 s=CLOSED pgs=2026 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:43.010 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:43.001+0000 7f107494e700 1 -- 172.21.15.42:0/2572017593 shutdown_connections 2019-12-05T06:28:43.010 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:43.001+0000 7f107494e700 1 -- 172.21.15.42:0/2572017593 wait complete. 2019-12-05T06:28:43.010 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:43.001+0000 7f107494e700 1 -- 172.21.15.42:0/2572017593 >> 172.21.15.42:0/2572017593 conn(0x7f10700aabe0 msgr2=0x7f10700a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:28:43.012 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:43.001+0000 7f107494e700 1 Processor -- start 2019-12-05T06:28:43.012 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:43.001+0000 7f107494e700 1 -- start start 2019-12-05T06:28:43.012 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:43.005+0000 7f107494e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f10700a5b10 0x7f1070142be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:28:43.012 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:43.005+0000 7f107494e700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1070143130 con 0x7f1070145270 2019-12-05T06:28:43.013 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:43.005+0000 7f107494e700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1070117100 con 0x7f1070144e40 2019-12-05T06:28:43.013 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:43.005+0000 7f107494e700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1070143410 con 0x7f10700a5b10 2019-12-05T06:28:43.013 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:43.005+0000 7f106ed9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f10700a5b10 0x7f1070142be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:28:43.013 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:43.005+0000 7f106659c700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f1070144e40 0x7f107011b180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:56000/0 (socket says 172.21.15.42:56000) 2019-12-05T06:28:43.013 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:43.005+0000 7f106659c700 1 -- 172.21.15.42:0/3160203288 learned_addr learned my addr 172.21.15.42:0/3160203288 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:28:43.013 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:43.005+0000 7f1067fff700 1 -- 172.21.15.42:0/3160203288 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3612064250 0 0) 0x7f10680110c0 con 0x7f1070145270 2019-12-05T06:28:43.013 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:43.005+0000 7f106ed9d700 1 -- 172.21.15.42:0/3160203288 >> v1:172.21.15.42:6789/0 conn(0x7f1070144e40 legacy=0x7f107011b180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:43.014 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:43.005+0000 7f106ed9d700 1 -- 172.21.15.42:0/3160203288 >> v1:172.21.15.40:6789/0 conn(0x7f1070145270 legacy=0x7f1070137800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:43.014 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:43.005+0000 7f106ed9d700 1 -- 172.21.15.42:0/3160203288 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f107011a6a0 con 0x7f10700a5b10 2019-12-05T06:28:43.014 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:43.005+0000 7f106ed9d700 1 --2- 172.21.15.42:0/3160203288 >> v2:172.21.15.42:3300/0 conn(0x7f10700a5b10 0x7f1070142be0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f10580010d0 tx=0x7f1058001110).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:43.014 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:43.005+0000 7f106ed9d700 1 --2- 172.21.15.42:0/3160203288 >> v2:172.21.15.42:3300/0 conn(0x7f10700a5b10 0x7f1070142be0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f10580010d0 tx=0x7f1058001110).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:43.014 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:43.005+0000 7f106ed9d700 1 --2- 172.21.15.42:0/3160203288 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f10700a5b10 0x7f1070142be0 secure :-1 s=READY pgs=2703 cs=0 l=1 rx=0x7f10580010d0 tx=0x7f1058001110).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:28:43.014 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:43.005+0000 7f106ed9d700 1 --2- 172.21.15.42:0/3160203288 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f10700a5b10 0x7f1070142be0 secure :-1 s=THROTTLE_DONE pgs=2703 cs=0 l=1 rx=0x7f10580010d0 tx=0x7f1058001110).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:43.014 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:43.005+0000 7f1067fff700 1 -- 172.21.15.42:0/3160203288 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f106000cce0 con 0x7f10700a5b10 2019-12-05T06:28:43.015 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:43.005+0000 7f106ed9d700 1 --2- 172.21.15.42:0/3160203288 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f10700a5b10 0x7f1070142be0 secure :-1 s=THROTTLE_DONE pgs=2703 cs=0 l=1 rx=0x7f10580010d0 tx=0x7f1058001110).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:43.015 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:43.005+0000 7f1067fff700 1 -- 172.21.15.42:0/3160203288 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f106000c900 con 0x7f10700a5b10 2019-12-05T06:28:43.015 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:43.005+0000 7f106ed9d700 1 --2- 172.21.15.42:0/3160203288 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f10700a5b10 0x7f1070142be0 secure :-1 s=THROTTLE_DONE pgs=2703 cs=0 l=1 rx=0x7f10580010d0 tx=0x7f1058001110).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:43.015 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:43.005+0000 7f107494e700 1 -- 172.21.15.42:0/3160203288 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f107011b980 con 0x7f10700a5b10 2019-12-05T06:28:43.015 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:43.005+0000 7f1067fff700 1 -- 172.21.15.42:0/3160203288 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f10600127b0 con 0x7f10700a5b10 2019-12-05T06:28:43.015 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:43.005+0000 7f107494e700 1 -- 172.21.15.42:0/3160203288 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1070139010 con 0x7f10700a5b10 2019-12-05T06:28:43.015 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:43.005+0000 7f106ed9d700 1 --2- 172.21.15.42:0/3160203288 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f10700a5b10 0x7f1070142be0 secure :-1 s=THROTTLE_DONE pgs=2703 cs=0 l=1 rx=0x7f10580010d0 tx=0x7f1058001110).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:43.016 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:43.009+0000 7f1067fff700 1 -- 172.21.15.42:0/3160203288 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f1060012ad0 con 0x7f10700a5b10 2019-12-05T06:28:43.017 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:43.009+0000 7f106ed9d700 1 --2- 172.21.15.42:0/3160203288 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f10700a5b10 0x7f1070142be0 secure :-1 s=THROTTLE_DONE pgs=2703 cs=0 l=1 rx=0x7f10580010d0 tx=0x7f1058001110).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:43.017 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:43.009+0000 7f1067fff700 1 --2- 172.21.15.42:0/3160203288 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f105403acb0 0x7f105403d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:28:43.017 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:43.009+0000 7f1067fff700 1 -- 172.21.15.42:0/3160203288 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 11019+0+0 (secure 0 0 0) 0x7f106004acb0 con 0x7f10700a5b10 2019-12-05T06:28:43.017 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:43.009+0000 7f106659c700 1 --2- 172.21.15.42:0/3160203288 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f105403acb0 0x7f105403d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:28:43.017 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:43.009+0000 7f106659c700 1 --2- 172.21.15.42:0/3160203288 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f105403acb0 0x7f105403d1c0 crc :-1 s=READY pgs=1131 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:28:43.017 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:43.009+0000 7f107494e700 1 -- 172.21.15.42:0/3160203288 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f107005d110 con 0x7f10700a5b10 2019-12-05T06:28:43.021 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:43.017+0000 7f106ed9d700 1 --2- 172.21.15.42:0/3160203288 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f10700a5b10 0x7f1070142be0 secure :-1 s=THROTTLE_DONE pgs=2703 cs=0 l=1 rx=0x7f10580010d0 tx=0x7f1058001110).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:43.022 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:43.017+0000 7f1067fff700 1 -- 172.21.15.42:0/3160203288 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f1060012de0 con 0x7f10700a5b10 2019-12-05T06:28:43.193 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:43.189+0000 7f107494e700 1 -- 172.21.15.42:0/3160203288 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f107005b8c0 con 0x7f10700a5b10 2019-12-05T06:28:43.194 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:43.189+0000 7f106ed9d700 1 --2- 172.21.15.42:0/3160203288 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f10700a5b10 0x7f1070142be0 secure :-1 s=THROTTLE_DONE pgs=2703 cs=0 l=1 rx=0x7f10580010d0 tx=0x7f1058001110).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:43.194 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:43.189+0000 7f1067fff700 1 -- 172.21.15.42:0/3160203288 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 322 v322) v1 ==== 95+0+1779 (secure 0 0 0) 0x7f1060031d30 con 0x7f10700a5b10 2019-12-05T06:28:43.194 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 322 2019-12-05T06:28:43.194 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:28:43.194 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":322,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":322,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:28:29.970815+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":107,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11142},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11142":{"gid":11142,"name":"b","rank":0,"incarnation":319,"state":"up:active","state_seq":28,"addr":"172.21.15.40:6833/3155851600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3155851600},{"type":"v1","addr":"172.21.15.40:6833","nonce":3155851600}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:28:43.197 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:43.193+0000 7f107494e700 1 -- 172.21.15.42:0/3160203288 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f105403acb0 msgr2=0x7f105403d1c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:43.197 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:43.193+0000 7f107494e700 1 --2- 172.21.15.42:0/3160203288 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f105403acb0 0x7f105403d1c0 crc :-1 s=READY pgs=1131 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:43.198 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:43.193+0000 7f107494e700 1 -- 172.21.15.42:0/3160203288 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f10700a5b10 msgr2=0x7f1070142be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:43.198 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:43.193+0000 7f107494e700 1 --2- 172.21.15.42:0/3160203288 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f10700a5b10 0x7f1070142be0 secure :-1 s=READY pgs=2703 cs=0 l=1 rx=0x7f10580010d0 tx=0x7f1058001110).stop 2019-12-05T06:28:43.198 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:43.193+0000 7f107494e700 1 -- 172.21.15.42:0/3160203288 shutdown_connections 2019-12-05T06:28:43.198 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:43.193+0000 7f107494e700 1 --2- 172.21.15.42:0/3160203288 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f105403acb0 0x7f105403d1c0 unknown :-1 s=CLOSED pgs=1131 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:43.199 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:43.193+0000 7f107494e700 1 --2- 172.21.15.42:0/3160203288 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f10700a5b10 0x7f1070142be0 unknown :-1 s=CLOSED pgs=2703 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:43.199 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:43.193+0000 7f107494e700 1 -- 172.21.15.42:0/3160203288 shutdown_connections 2019-12-05T06:28:43.199 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:43.193+0000 7f107494e700 1 -- 172.21.15.42:0/3160203288 wait complete. 2019-12-05T06:28:43.199 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:43.193+0000 7f107494e700 1 -- 172.21.15.42:0/3160203288 >> 172.21.15.42:0/3160203288 conn(0x7f107008fde0 msgr2=0x7f1070059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:28:43.207 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2019-12-05T06:28:44.903 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:28:44.906 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:28:45.207 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:28:45.318 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:45.305+0000 7faf24c9a700 1 Processor -- start 2019-12-05T06:28:45.318 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:45.305+0000 7faf24c9a700 1 -- start start 2019-12-05T06:28:45.318 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:45.309+0000 7faf24c9a700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7faf200a7740 0x7faf2008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:28:45.318 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:45.309+0000 7faf24c9a700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7faf2007e720 0x7faf2007eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:28:45.319 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:45.309+0000 7faf24c9a700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7faf2009f2e0 0x7faf2009f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:28:45.319 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:45.309+0000 7faf24c9a700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf200a0b00 con 0x7faf2007e720 2019-12-05T06:28:45.319 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:45.309+0000 7faf24c9a700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf2008e5b0 con 0x7faf200a7740 2019-12-05T06:28:45.319 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:45.309+0000 7faf24c9a700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf2008e850 con 0x7faf2009f2e0 2019-12-05T06:28:45.319 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:45.309+0000 7faf1ed9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7faf200a7740 0x7faf2008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:28:45.319 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:45.309+0000 7faf1ed9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7faf200a7740 0x7faf2008df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:55972/0 (socket says 172.21.15.42:55972) 2019-12-05T06:28:45.319 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:45.309+0000 7faf1ed9d700 1 -- 172.21.15.42:0/2786299297 learned_addr learned my addr 172.21.15.42:0/2786299297 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:28:45.320 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:45.309+0000 7faf1f59e700 1 --2- 172.21.15.42:0/2786299297 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7faf2009f2e0 0x7faf2009f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:28:45.320 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:45.309+0000 7faf1e59c700 1 --2- 172.21.15.42:0/2786299297 >> v2:172.21.15.40:3300/0 conn(0x7faf2007e720 0x7faf2007eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:28:45.320 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:45.309+0000 7faf1ed9d700 1 -- 172.21.15.42:0/2786299297 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7faf2009f2e0 msgr2=0x7faf2009f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:28:45.320 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:45.309+0000 7faf1ed9d700 1 --2- 172.21.15.42:0/2786299297 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7faf2009f2e0 0x7faf2009f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:28:45.320 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:45.309+0000 7faf1ed9d700 1 -- 172.21.15.42:0/2786299297 >> v2:172.21.15.40:3300/0 conn(0x7faf2007e720 msgr2=0x7faf2007eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:28:45.320 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:45.309+0000 7faf1ed9d700 1 --2- 172.21.15.42:0/2786299297 >> v2:172.21.15.40:3300/0 conn(0x7faf2007e720 0x7faf2007eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:28:45.321 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:45.309+0000 7faf1ed9d700 1 -- 172.21.15.42:0/2786299297 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf2007f0a0 con 0x7faf200a7740 2019-12-05T06:28:45.321 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:45.309+0000 7faf1ed9d700 1 --2- 172.21.15.42:0/2786299297 >> v2:172.21.15.42:3300/0 conn(0x7faf200a7740 0x7faf2008df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7faf200a0f50 tx=0x7faf1000fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:45.321 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:45.309+0000 7faf1ed9d700 1 --2- 172.21.15.42:0/2786299297 >> v2:172.21.15.42:3300/0 conn(0x7faf200a7740 0x7faf2008df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7faf200a0f50 tx=0x7faf1000fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:45.321 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:45.309+0000 7faf1ed9d700 1 --2- 172.21.15.42:0/2786299297 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf200a7740 0x7faf2008df80 secure :-1 s=READY pgs=2704 cs=0 l=1 rx=0x7faf200a0f50 tx=0x7faf1000fdb0).ready entity=mon.1 client_cookie=4fe2463c615a9b87 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:28:45.321 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:45.309+0000 7faf1ed9d700 1 --2- 172.21.15.42:0/2786299297 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf200a7740 0x7faf2008df80 secure :-1 s=THROTTLE_DONE pgs=2704 cs=0 l=1 rx=0x7faf200a0f50 tx=0x7faf1000fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:45.321 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:45.309+0000 7faf1dd9b700 1 -- 172.21.15.42:0/2786299297 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faf10008c90 con 0x7faf200a7740 2019-12-05T06:28:45.321 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:45.309+0000 7faf1ed9d700 1 --2- 172.21.15.42:0/2786299297 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf200a7740 0x7faf2008df80 secure :-1 s=THROTTLE_DONE pgs=2704 cs=0 l=1 rx=0x7faf200a0f50 tx=0x7faf1000fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:45.322 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:45.309+0000 7faf1ed9d700 1 --2- 172.21.15.42:0/2786299297 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf200a7740 0x7faf2008df80 secure :-1 s=THROTTLE_DONE pgs=2704 cs=0 l=1 rx=0x7faf200a0f50 tx=0x7faf1000fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:45.322 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:45.309+0000 7faf1dd9b700 1 -- 172.21.15.42:0/2786299297 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf100088b0 con 0x7faf200a7740 2019-12-05T06:28:45.322 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:45.309+0000 7faf1dd9b700 1 -- 172.21.15.42:0/2786299297 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faf10011b50 con 0x7faf200a7740 2019-12-05T06:28:45.322 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:45.309+0000 7faf24c9a700 1 -- 172.21.15.42:0/2786299297 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf200a7740 msgr2=0x7faf2008df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:45.322 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:45.309+0000 7faf24c9a700 1 --2- 172.21.15.42:0/2786299297 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf200a7740 0x7faf2008df80 secure :-1 s=READY pgs=2704 cs=0 l=1 rx=0x7faf200a0f50 tx=0x7faf1000fdb0).stop 2019-12-05T06:28:45.322 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:45.309+0000 7faf24c9a700 1 -- 172.21.15.42:0/2786299297 shutdown_connections 2019-12-05T06:28:45.322 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:45.309+0000 7faf24c9a700 1 --2- 172.21.15.42:0/2786299297 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7faf2009f2e0 0x7faf2009f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:28:45.323 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:45.309+0000 7faf24c9a700 1 --2- 172.21.15.42:0/2786299297 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf200a7740 0x7faf2008df80 unknown :-1 s=CLOSED pgs=2704 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:45.323 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:45.309+0000 7faf24c9a700 1 --2- 172.21.15.42:0/2786299297 >> v2:172.21.15.40:3300/0 conn(0x7faf2007e720 0x7faf2007eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:28:45.323 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:45.309+0000 7faf24c9a700 1 -- 172.21.15.42:0/2786299297 shutdown_connections 2019-12-05T06:28:45.323 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:45.309+0000 7faf24c9a700 1 -- 172.21.15.42:0/2786299297 wait complete. 2019-12-05T06:28:45.323 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:45.309+0000 7faf24c9a700 1 -- 172.21.15.42:0/2786299297 >> 172.21.15.42:0/2786299297 conn(0x7faf200aabe0 msgr2=0x7faf200a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:28:45.327 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:45.313+0000 7faf24c9a700 1 Processor -- start 2019-12-05T06:28:45.327 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:45.313+0000 7faf24c9a700 1 -- start start 2019-12-05T06:28:45.327 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:45.317+0000 7faf24c9a700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7faf200a3ad0 0x7faf2011d750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:28:45.327 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:45.317+0000 7faf24c9a700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7faf20123780 0x7faf20123bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:28:45.327 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:45.317+0000 7faf24c9a700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7faf20131630 0x7faf20131a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:28:45.327 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:45.317+0000 7faf24c9a700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf20121820 con 0x7faf20123780 2019-12-05T06:28:45.328 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:45.317+0000 7faf24c9a700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf20121ac0 con 0x7faf200a3ad0 2019-12-05T06:28:45.328 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:45.317+0000 7faf24c9a700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf20121510 con 0x7faf20131630 2019-12-05T06:28:45.328 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:45.317+0000 7faf1ed9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7faf200a3ad0 0x7faf2011d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:28:45.328 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:45.317+0000 7faf1ed9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7faf200a3ad0 0x7faf2011d750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:55978/0 (socket says 172.21.15.42:55978) 2019-12-05T06:28:45.328 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:45.317+0000 7faf1ed9d700 1 -- 172.21.15.42:0/1867356815 learned_addr learned my addr 172.21.15.42:0/1867356815 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:28:45.328 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:45.317+0000 7faf1e59c700 1 --2- 172.21.15.42:0/1867356815 >> v2:172.21.15.40:3300/0 conn(0x7faf20123780 0x7faf20123bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:28:45.329 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:45.317+0000 7faf1f59e700 1 --2- 172.21.15.42:0/1867356815 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7faf20131630 0x7faf20131a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:28:45.329 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:45.317+0000 7faf1ed9d700 1 -- 172.21.15.42:0/1867356815 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7faf20131630 msgr2=0x7faf20131a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:45.329 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:45.317+0000 7faf1ed9d700 1 --2- 172.21.15.42:0/1867356815 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7faf20131630 0x7faf20131a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:45.329 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:45.317+0000 7faf1ed9d700 1 -- 172.21.15.42:0/1867356815 >> v2:172.21.15.40:3300/0 conn(0x7faf20123780 msgr2=0x7faf20123bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:45.329 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:45.317+0000 7faf1ed9d700 1 --2- 172.21.15.42:0/1867356815 >> v2:172.21.15.40:3300/0 conn(0x7faf20123780 0x7faf20123bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:45.329 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:45.317+0000 7faf1ed9d700 1 -- 172.21.15.42:0/1867356815 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf201312c0 con 0x7faf200a3ad0 2019-12-05T06:28:45.329 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:45.317+0000 7faf1ed9d700 1 --2- 172.21.15.42:0/1867356815 >> v2:172.21.15.42:3300/0 conn(0x7faf200a3ad0 0x7faf2011d750 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7faf200a0f50 tx=0x7faf10008930).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:45.330 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:45.317+0000 7faf1ed9d700 1 --2- 172.21.15.42:0/1867356815 >> v2:172.21.15.42:3300/0 conn(0x7faf200a3ad0 0x7faf2011d750 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7faf200a0f50 tx=0x7faf10008930).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:45.330 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:45.317+0000 7faf1ed9d700 1 --2- 172.21.15.42:0/1867356815 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf200a3ad0 0x7faf2011d750 secure :-1 s=READY pgs=2705 cs=0 l=1 rx=0x7faf200a0f50 tx=0x7faf10008930).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:28:45.330 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:45.317+0000 7faf1ed9d700 1 --2- 172.21.15.42:0/1867356815 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf200a3ad0 0x7faf2011d750 secure :-1 s=THROTTLE_DONE pgs=2705 cs=0 l=1 rx=0x7faf200a0f50 tx=0x7faf10008930).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:45.330 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:45.317+0000 7faf1ed9d700 1 --2- 172.21.15.42:0/1867356815 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf200a3ad0 0x7faf2011d750 secure :-1 s=THROTTLE_DONE pgs=2705 cs=0 l=1 rx=0x7faf200a0f50 tx=0x7faf10008930).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:45.330 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:45.317+0000 7faf177fe700 1 -- 172.21.15.42:0/1867356815 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faf1000f280 con 0x7faf200a3ad0 2019-12-05T06:28:45.330 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:45.317+0000 7faf1ed9d700 1 --2- 172.21.15.42:0/1867356815 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf200a3ad0 0x7faf2011d750 secure :-1 s=THROTTLE_DONE pgs=2705 cs=0 l=1 rx=0x7faf200a0f50 tx=0x7faf10008930).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:45.330 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:45.317+0000 7faf177fe700 1 -- 172.21.15.42:0/1867356815 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf10011770 con 0x7faf200a3ad0 2019-12-05T06:28:45.331 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:45.317+0000 7faf24c9a700 1 -- 172.21.15.42:0/1867356815 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf201222a0 con 0x7faf200a3ad0 2019-12-05T06:28:45.331 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:45.317+0000 7faf177fe700 1 -- 172.21.15.42:0/1867356815 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faf10005540 con 0x7faf200a3ad0 2019-12-05T06:28:45.331 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:45.317+0000 7faf24c9a700 1 -- 172.21.15.42:0/1867356815 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf2008eb70 con 0x7faf200a3ad0 2019-12-05T06:28:45.331 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:45.317+0000 7faf1ed9d700 1 --2- 172.21.15.42:0/1867356815 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf200a3ad0 0x7faf2011d750 secure :-1 s=THROTTLE_DONE pgs=2705 cs=0 l=1 rx=0x7faf200a0f50 tx=0x7faf10008930).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:45.331 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:45.317+0000 7faf177fe700 1 -- 172.21.15.42:0/1867356815 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7faf10005850 con 0x7faf200a3ad0 2019-12-05T06:28:45.331 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:45.317+0000 7faf1ed9d700 1 --2- 172.21.15.42:0/1867356815 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf200a3ad0 0x7faf2011d750 secure :-1 s=THROTTLE_DONE pgs=2705 cs=0 l=1 rx=0x7faf200a0f50 tx=0x7faf10008930).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:45.331 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:45.321+0000 7faf177fe700 1 --2- 172.21.15.42:0/1867356815 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7faf0403ae10 0x7faf0403d320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:28:45.332 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:45.321+0000 7faf177fe700 1 -- 172.21.15.42:0/1867356815 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 11019+0+0 (secure 0 0 0) 0x7faf1008f470 con 0x7faf200a3ad0 2019-12-05T06:28:45.332 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:45.321+0000 7faf1e59c700 1 --2- 172.21.15.42:0/1867356815 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7faf0403ae10 0x7faf0403d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:28:45.332 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:45.321+0000 7faf1e59c700 1 --2- 172.21.15.42:0/1867356815 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7faf0403ae10 0x7faf0403d320 crc :-1 s=READY pgs=1134 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:28:45.332 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:45.321+0000 7faf24c9a700 1 -- 172.21.15.42:0/1867356815 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf20121c80 con 0x7faf200a3ad0 2019-12-05T06:28:45.335 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:45.329+0000 7faf1ed9d700 1 --2- 172.21.15.42:0/1867356815 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf200a3ad0 0x7faf2011d750 secure :-1 s=THROTTLE_DONE pgs=2705 cs=0 l=1 rx=0x7faf200a0f50 tx=0x7faf10008930).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:45.336 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:45.329+0000 7faf177fe700 1 -- 172.21.15.42:0/1867356815 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7faf10076710 con 0x7faf200a3ad0 2019-12-05T06:28:45.506 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:45.501+0000 7faf24c9a700 1 -- 172.21.15.42:0/1867356815 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faf2005d110 con 0x7faf200a3ad0 2019-12-05T06:28:45.507 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:45.501+0000 7faf1ed9d700 1 --2- 172.21.15.42:0/1867356815 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf200a3ad0 0x7faf2011d750 secure :-1 s=THROTTLE_DONE pgs=2705 cs=0 l=1 rx=0x7faf200a0f50 tx=0x7faf10008930).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:45.507 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:45.501+0000 7faf177fe700 1 -- 172.21.15.42:0/1867356815 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 323 v323) v1 ==== 95+0+2117 (secure 0 0 0) 0x7faf10007160 con 0x7faf200a3ad0 2019-12-05T06:28:45.507 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 323 2019-12-05T06:28:45.507 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:28:45.507 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":323,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":11265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/1985322937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1985322937},{"type":"v1","addr":"172.21.15.42:6835","nonce":1985322937}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":323}],"filesystems":[{"mdsmap":{"epoch":322,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:28:29.970815+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":107,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11142},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11142":{"gid":11142,"name":"b","rank":0,"incarnation":319,"state":"up:active","state_seq":28,"addr":"172.21.15.40:6833/3155851600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3155851600},{"type":"v1","addr":"172.21.15.40:6833","nonce":3155851600}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:28:45.511 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:45.505+0000 7faf24c9a700 1 -- 172.21.15.42:0/1867356815 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7faf0403ae10 msgr2=0x7faf0403d320 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:45.511 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:45.505+0000 7faf24c9a700 1 --2- 172.21.15.42:0/1867356815 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7faf0403ae10 0x7faf0403d320 crc :-1 s=READY pgs=1134 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:45.512 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:45.505+0000 7faf24c9a700 1 -- 172.21.15.42:0/1867356815 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf200a3ad0 msgr2=0x7faf2011d750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:45.512 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:45.505+0000 7faf24c9a700 1 --2- 172.21.15.42:0/1867356815 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf200a3ad0 0x7faf2011d750 secure :-1 s=READY pgs=2705 cs=0 l=1 rx=0x7faf200a0f50 tx=0x7faf10008930).stop 2019-12-05T06:28:45.512 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:45.505+0000 7faf24c9a700 1 -- 172.21.15.42:0/1867356815 shutdown_connections 2019-12-05T06:28:45.512 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:45.505+0000 7faf24c9a700 1 --2- 172.21.15.42:0/1867356815 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7faf0403ae10 0x7faf0403d320 unknown :-1 s=CLOSED pgs=1134 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:45.512 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:45.505+0000 7faf24c9a700 1 --2- 172.21.15.42:0/1867356815 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7faf20131630 0x7faf20131a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:45.513 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:45.505+0000 7faf24c9a700 1 --2- 172.21.15.42:0/1867356815 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf200a3ad0 0x7faf2011d750 unknown :-1 s=CLOSED pgs=2705 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:45.513 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:45.505+0000 7faf24c9a700 1 --2- 172.21.15.42:0/1867356815 >> v2:172.21.15.40:3300/0 conn(0x7faf20123780 0x7faf20123bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:45.513 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:45.505+0000 7faf24c9a700 1 -- 172.21.15.42:0/1867356815 shutdown_connections 2019-12-05T06:28:45.513 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:45.505+0000 7faf24c9a700 1 -- 172.21.15.42:0/1867356815 wait complete. 2019-12-05T06:28:45.513 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:28:45.505+0000 7faf24c9a700 1 -- 172.21.15.42:0/1867356815 >> 172.21.15.42:0/1867356815 conn(0x7faf200aabe0 msgr2=0x7faf200a9770 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:28:45.522 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a reported in up:standby state 2019-12-05T06:28:45.522 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 79 secs before thrashing 2019-12-05T06:29:14.987 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:29:14.990 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:29:15.022 INFO:tasks.ceph.mds.b.smithi040.stderr:2019-12-05T06:29:15.014+0000 7f0a29aa5700 -1 received signal: Hangup from (PID: 14725) UID: 0 2019-12-05T06:30:04.523 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:30:04.633 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.621+0000 7fb2360ba700 1 Processor -- start 2019-12-05T06:30:04.635 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.621+0000 7fb2360ba700 1 -- start start 2019-12-05T06:30:04.635 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.625+0000 7fb2360ba700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb23008df80 0x7fb23009f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:30:04.635 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.625+0000 7fb2360ba700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb23009fc60 con 0x7fb2300a7740 2019-12-05T06:30:04.635 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.625+0000 7fb2360ba700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2300a0710 con 0x7fb23009f830 2019-12-05T06:30:04.636 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.625+0000 7fb2360ba700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2300a0b00 con 0x7fb23008df80 2019-12-05T06:30:04.636 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.625+0000 7fb22f7fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb23008df80 0x7fb23009f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:30:04.636 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.625+0000 7fb22f7fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb23008df80 0x7fb23009f2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:56290/0 (socket says 172.21.15.42:56290) 2019-12-05T06:30:04.636 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.625+0000 7fb22f7fe700 1 -- 172.21.15.42:0/738881904 learned_addr learned my addr 172.21.15.42:0/738881904 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:30:04.637 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.625+0000 7fb234e58700 1 --1- 172.21.15.42:0/738881904 >> v1:172.21.15.42:6789/0 conn(0x7fb23009f830 0x7fb23014efe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:56330/0 (socket says 172.21.15.42:56330) 2019-12-05T06:30:04.637 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.625+0000 7fb22ffff700 1 --1- 172.21.15.42:0/738881904 >> v1:172.21.15.40:6789/0 conn(0x7fb2300a7740 0x7fb23007e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:37534/0 (socket says 172.21.15.42:37534) 2019-12-05T06:30:04.637 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.625+0000 7fb22f7fe700 1 -- 172.21.15.42:0/738881904 >> v1:172.21.15.42:6789/0 conn(0x7fb23009f830 legacy=0x7fb23014efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:30:04.637 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.625+0000 7fb22f7fe700 1 -- 172.21.15.42:0/738881904 >> v1:172.21.15.40:6789/0 conn(0x7fb2300a7740 legacy=0x7fb23007e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:04.637 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.625+0000 7fb22f7fe700 1 -- 172.21.15.42:0/738881904 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb23007ef20 con 0x7fb23008df80 2019-12-05T06:30:04.638 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.625+0000 7fb22f7fe700 1 --2- 172.21.15.42:0/738881904 >> v2:172.21.15.42:3300/0 conn(0x7fb23008df80 0x7fb23009f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fb218008790 tx=0x7fb2200014d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:04.638 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.625+0000 7fb22f7fe700 1 --2- 172.21.15.42:0/738881904 >> v2:172.21.15.42:3300/0 conn(0x7fb23008df80 0x7fb23009f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fb218008790 tx=0x7fb2200014d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:04.638 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.625+0000 7fb22f7fe700 1 --2- 172.21.15.42:0/738881904 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb23008df80 0x7fb23009f2e0 secure :-1 s=READY pgs=2709 cs=0 l=1 rx=0x7fb218008790 tx=0x7fb2200014d0).ready entity=mon.1 client_cookie=6cd224bafd8c06ec server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:30:04.638 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.625+0000 7fb22f7fe700 1 --2- 172.21.15.42:0/738881904 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb23008df80 0x7fb23009f2e0 secure :-1 s=THROTTLE_DONE pgs=2709 cs=0 l=1 rx=0x7fb218008790 tx=0x7fb2200014d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:04.638 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.625+0000 7fb22effd700 1 -- 172.21.15.42:0/738881904 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb218012060 con 0x7fb23008df80 2019-12-05T06:30:04.639 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.625+0000 7fb22f7fe700 1 --2- 172.21.15.42:0/738881904 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb23008df80 0x7fb23009f2e0 secure :-1 s=THROTTLE_DONE pgs=2709 cs=0 l=1 rx=0x7fb218008790 tx=0x7fb2200014d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:04.639 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.625+0000 7fb22f7fe700 1 --2- 172.21.15.42:0/738881904 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb23008df80 0x7fb23009f2e0 secure :-1 s=THROTTLE_DONE pgs=2709 cs=0 l=1 rx=0x7fb218008790 tx=0x7fb2200014d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:04.639 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.625+0000 7fb22effd700 1 -- 172.21.15.42:0/738881904 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2180110c0 con 0x7fb23008df80 2019-12-05T06:30:04.639 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.625+0000 7fb22effd700 1 -- 172.21.15.42:0/738881904 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb2180116a0 con 0x7fb23008df80 2019-12-05T06:30:04.640 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.625+0000 7fb2360ba700 1 -- 172.21.15.42:0/738881904 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb23008df80 msgr2=0x7fb23009f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:04.640 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.625+0000 7fb2360ba700 1 --2- 172.21.15.42:0/738881904 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb23008df80 0x7fb23009f2e0 secure :-1 s=READY pgs=2709 cs=0 l=1 rx=0x7fb218008790 tx=0x7fb2200014d0).stop 2019-12-05T06:30:04.640 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.625+0000 7fb2360ba700 1 -- 172.21.15.42:0/738881904 shutdown_connections 2019-12-05T06:30:04.640 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.625+0000 7fb2360ba700 1 --2- 172.21.15.42:0/738881904 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb23008df80 0x7fb23009f2e0 unknown :-1 s=CLOSED pgs=2709 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:04.641 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.625+0000 7fb2360ba700 1 -- 172.21.15.42:0/738881904 shutdown_connections 2019-12-05T06:30:04.641 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.625+0000 7fb2360ba700 1 -- 172.21.15.42:0/738881904 wait complete. 2019-12-05T06:30:04.641 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.625+0000 7fb2360ba700 1 -- 172.21.15.42:0/738881904 >> 172.21.15.42:0/738881904 conn(0x7fb2300aabe0 msgr2=0x7fb2300a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:30:04.641 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.629+0000 7fb2360ba700 1 Processor -- start 2019-12-05T06:30:04.642 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.629+0000 7fb2360ba700 1 -- start start 2019-12-05T06:30:04.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.629+0000 7fb2360ba700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb230121b50 0x7fb23007db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:30:04.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.629+0000 7fb2360ba700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb230118520 0x7fb230118950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:30:04.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.629+0000 7fb2360ba700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb23007e0c0 con 0x7fb23012e7b0 2019-12-05T06:30:04.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.629+0000 7fb2360ba700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb230118fc0 con 0x7fb230118520 2019-12-05T06:30:04.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.629+0000 7fb2360ba700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb23012e510 con 0x7fb230121b50 2019-12-05T06:30:04.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.629+0000 7fb22ffff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb230121b50 0x7fb23007db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:30:04.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.629+0000 7fb22ffff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb230121b50 0x7fb23007db70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:56294/0 (socket says 172.21.15.42:56294) 2019-12-05T06:30:04.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.629+0000 7fb22ffff700 1 -- 172.21.15.42:0/4014041323 learned_addr learned my addr 172.21.15.42:0/4014041323 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:30:04.650 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.629+0000 7fb22f7fe700 1 --2- 172.21.15.42:0/4014041323 >> v2:172.21.15.40:3300/0 conn(0x7fb230118520 0x7fb230118950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:30:04.650 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.629+0000 7fb22ffff700 1 -- 172.21.15.42:0/4014041323 >> v2:172.21.15.40:3300/0 conn(0x7fb230118520 msgr2=0x7fb230118950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:04.650 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.629+0000 7fb22ffff700 1 --2- 172.21.15.42:0/4014041323 >> v2:172.21.15.40:3300/0 conn(0x7fb230118520 0x7fb230118950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:04.650 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.629+0000 7fb22ffff700 1 -- 172.21.15.42:0/4014041323 >> v1:172.21.15.40:6789/0 conn(0x7fb23012e7b0 legacy=0x7fb23012ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:04.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.629+0000 7fb22ffff700 1 -- 172.21.15.42:0/4014041323 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb230136860 con 0x7fb230121b50 2019-12-05T06:30:04.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.629+0000 7fb22ffff700 1 --2- 172.21.15.42:0/4014041323 >> v2:172.21.15.42:3300/0 conn(0x7fb230121b50 0x7fb23007db70 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fb220001340 tx=0x7fb220010df0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:04.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.629+0000 7fb22f7fe700 1 --2- 172.21.15.42:0/4014041323 >> v2:172.21.15.40:3300/0 conn(0x7fb230118520 0x7fb230118950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:30:04.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.629+0000 7fb22ffff700 1 --2- 172.21.15.42:0/4014041323 >> v2:172.21.15.42:3300/0 conn(0x7fb230121b50 0x7fb23007db70 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fb220001340 tx=0x7fb220010df0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:04.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.629+0000 7fb22ffff700 1 --2- 172.21.15.42:0/4014041323 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb230121b50 0x7fb23007db70 secure :-1 s=READY pgs=2710 cs=0 l=1 rx=0x7fb220001340 tx=0x7fb220010df0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:30:04.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.633+0000 7fb22ffff700 1 --2- 172.21.15.42:0/4014041323 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb230121b50 0x7fb23007db70 secure :-1 s=THROTTLE_DONE pgs=2710 cs=0 l=1 rx=0x7fb220001340 tx=0x7fb220010df0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:04.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.633+0000 7fb22ffff700 1 --2- 172.21.15.42:0/4014041323 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb230121b50 0x7fb23007db70 secure :-1 s=THROTTLE_DONE pgs=2710 cs=0 l=1 rx=0x7fb220001340 tx=0x7fb220010df0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:04.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.633+0000 7fb22cff9700 1 -- 172.21.15.42:0/4014041323 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb220012b90 con 0x7fb230121b50 2019-12-05T06:30:04.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.633+0000 7fb22ffff700 1 --2- 172.21.15.42:0/4014041323 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb230121b50 0x7fb23007db70 secure :-1 s=THROTTLE_DONE pgs=2710 cs=0 l=1 rx=0x7fb220001340 tx=0x7fb220010df0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:04.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.633+0000 7fb22cff9700 1 -- 172.21.15.42:0/4014041323 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2200127b0 con 0x7fb230121b50 2019-12-05T06:30:04.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.633+0000 7fb22cff9700 1 -- 172.21.15.42:0/4014041323 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb22000c980 con 0x7fb230121b50 2019-12-05T06:30:04.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.633+0000 7fb2360ba700 1 -- 172.21.15.42:0/4014041323 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb230135850 con 0x7fb230121b50 2019-12-05T06:30:04.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.633+0000 7fb2360ba700 1 -- 172.21.15.42:0/4014041323 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb230135d80 con 0x7fb230121b50 2019-12-05T06:30:04.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.633+0000 7fb22ffff700 1 --2- 172.21.15.42:0/4014041323 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb230121b50 0x7fb23007db70 secure :-1 s=THROTTLE_DONE pgs=2710 cs=0 l=1 rx=0x7fb220001340 tx=0x7fb220010df0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:04.654 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.633+0000 7fb22cff9700 1 -- 172.21.15.42:0/4014041323 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fb22000cc90 con 0x7fb230121b50 2019-12-05T06:30:04.654 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.633+0000 7fb22ffff700 1 --2- 172.21.15.42:0/4014041323 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb230121b50 0x7fb23007db70 secure :-1 s=THROTTLE_DONE pgs=2710 cs=0 l=1 rx=0x7fb220001340 tx=0x7fb220010df0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:04.654 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.633+0000 7fb22cff9700 1 --2- 172.21.15.42:0/4014041323 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb21403adb0 0x7fb21403d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:30:04.654 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.633+0000 7fb22cff9700 1 -- 172.21.15.42:0/4014041323 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 11019+0+0 (secure 0 0 0) 0x7fb22004b080 con 0x7fb230121b50 2019-12-05T06:30:04.654 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.633+0000 7fb22f7fe700 1 --2- 172.21.15.42:0/4014041323 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb21403adb0 0x7fb21403d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:30:04.655 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.633+0000 7fb22f7fe700 1 --2- 172.21.15.42:0/4014041323 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb21403adb0 0x7fb21403d2c0 crc :-1 s=READY pgs=1137 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:30:04.656 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.637+0000 7fb2360ba700 1 -- 172.21.15.42:0/4014041323 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb23005d110 con 0x7fb230121b50 2019-12-05T06:30:04.657 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.645+0000 7fb22ffff700 1 --2- 172.21.15.42:0/4014041323 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb230121b50 0x7fb23007db70 secure :-1 s=THROTTLE_DONE pgs=2710 cs=0 l=1 rx=0x7fb220001340 tx=0x7fb220010df0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:04.657 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.645+0000 7fb22cff9700 1 -- 172.21.15.42:0/4014041323 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fb220032100 con 0x7fb230121b50 2019-12-05T06:30:04.830 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.821+0000 7fb2360ba700 1 -- 172.21.15.42:0/4014041323 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb23011a530 con 0x7fb230121b50 2019-12-05T06:30:04.830 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.821+0000 7fb22ffff700 1 --2- 172.21.15.42:0/4014041323 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb230121b50 0x7fb23007db70 secure :-1 s=THROTTLE_DONE pgs=2710 cs=0 l=1 rx=0x7fb220001340 tx=0x7fb220010df0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:04.830 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.821+0000 7fb22cff9700 1 -- 172.21.15.42:0/4014041323 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 323 v323) v1 ==== 95+0+2117 (secure 0 0 0) 0x7fb220032360 con 0x7fb230121b50 2019-12-05T06:30:04.830 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 323 2019-12-05T06:30:04.831 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:30:04.831 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":323,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":11265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/1985322937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1985322937},{"type":"v1","addr":"172.21.15.42:6835","nonce":1985322937}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":323}],"filesystems":[{"mdsmap":{"epoch":322,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:28:29.970815+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":107,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11142},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11142":{"gid":11142,"name":"b","rank":0,"incarnation":319,"state":"up:active","state_seq":28,"addr":"172.21.15.40:6833/3155851600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3155851600},{"type":"v1","addr":"172.21.15.40:6833","nonce":3155851600}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:30:04.833 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.825+0000 7fb2360ba700 1 -- 172.21.15.42:0/4014041323 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb21403adb0 msgr2=0x7fb21403d2c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:04.833 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.825+0000 7fb2360ba700 1 --2- 172.21.15.42:0/4014041323 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb21403adb0 0x7fb21403d2c0 crc :-1 s=READY pgs=1137 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:04.833 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.825+0000 7fb2360ba700 1 -- 172.21.15.42:0/4014041323 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb230121b50 msgr2=0x7fb23007db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:04.834 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.825+0000 7fb2360ba700 1 --2- 172.21.15.42:0/4014041323 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb230121b50 0x7fb23007db70 secure :-1 s=READY pgs=2710 cs=0 l=1 rx=0x7fb220001340 tx=0x7fb220010df0).stop 2019-12-05T06:30:04.834 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.825+0000 7fb2360ba700 1 -- 172.21.15.42:0/4014041323 shutdown_connections 2019-12-05T06:30:04.834 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.825+0000 7fb2360ba700 1 --2- 172.21.15.42:0/4014041323 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb21403adb0 0x7fb21403d2c0 unknown :-1 s=CLOSED pgs=1137 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:04.834 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.825+0000 7fb2360ba700 1 --2- 172.21.15.42:0/4014041323 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb230121b50 0x7fb23007db70 unknown :-1 s=CLOSED pgs=2710 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:04.834 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.825+0000 7fb2360ba700 1 --2- 172.21.15.42:0/4014041323 >> v2:172.21.15.40:3300/0 conn(0x7fb230118520 0x7fb230118950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:04.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.825+0000 7fb2360ba700 1 -- 172.21.15.42:0/4014041323 shutdown_connections 2019-12-05T06:30:04.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.825+0000 7fb2360ba700 1 -- 172.21.15.42:0/4014041323 wait complete. 2019-12-05T06:30:04.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.825+0000 7fb2360ba700 1 -- 172.21.15.42:0/4014041323 >> 172.21.15.42:0/4014041323 conn(0x7fb23005a190 msgr2=0x7fb2300a8040 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:30:04.841 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.b (rank=0) 2019-12-05T06:30:04.842 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2019-12-05T06:30:04.842 INFO:teuthology.orchestra.run:waiting for 300 2019-12-05T06:30:04.858 INFO:tasks.ceph.mds.b:Stopped 2019-12-05T06:30:04.859 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:30:04.961 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.949+0000 7f05a35a5700 1 Processor -- start 2019-12-05T06:30:04.961 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.949+0000 7f05a35a5700 1 -- start start 2019-12-05T06:30:04.961 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.953+0000 7f05a35a5700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f059c0a7740 0x7f059c08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:30:04.962 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.953+0000 7f05a35a5700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f059c09fae0 0x7f059c07eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:30:04.962 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.953+0000 7f05a35a5700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f059c0a0710 con 0x7f059c07e720 2019-12-05T06:30:04.962 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.953+0000 7f05a35a5700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f059c0a0b00 con 0x7f059c09fae0 2019-12-05T06:30:04.962 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.953+0000 7f05a35a5700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f059c08e5b0 con 0x7f059c0a7740 2019-12-05T06:30:04.962 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.953+0000 7f05a2343700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f059c09fae0 0x7f059c07eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:30:04.962 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.953+0000 7f05a2343700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f059c09fae0 0x7f059c07eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:56310/0 (socket says 172.21.15.42:56310) 2019-12-05T06:30:04.963 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.953+0000 7f05a2343700 1 -- 172.21.15.42:0/1061015707 learned_addr learned my addr 172.21.15.42:0/1061015707 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:30:04.963 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.953+0000 7f05a1b42700 1 --2- 172.21.15.42:0/1061015707 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f059c0a7740 0x7f059c08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:30:04.963 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.953+0000 7f05a0b40700 1 -- 172.21.15.42:0/1061015707 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3778378782 0 0) 0x7f059c0a0710 con 0x7f059c07e720 2019-12-05T06:30:04.963 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.953+0000 7f05a2343700 1 -- 172.21.15.42:0/1061015707 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f059c0a7740 msgr2=0x7f059c08df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:30:04.963 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.953+0000 7f05a2343700 1 --2- 172.21.15.42:0/1061015707 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f059c0a7740 0x7f059c08df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:30:04.963 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.953+0000 7f05a2343700 1 -- 172.21.15.42:0/1061015707 >> v1:172.21.15.40:6789/0 conn(0x7f059c07e720 legacy=0x7f059c09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:04.963 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.953+0000 7f05a2343700 1 -- 172.21.15.42:0/1061015707 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f059c07f0a0 con 0x7f059c09fae0 2019-12-05T06:30:04.964 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.953+0000 7f05a2343700 1 --2- 172.21.15.42:0/1061015707 >> v2:172.21.15.42:3300/0 conn(0x7f059c09fae0 0x7f059c07eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f0594008b10 tx=0x7f05940100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:04.964 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.953+0000 7f05a2343700 1 --2- 172.21.15.42:0/1061015707 >> v2:172.21.15.42:3300/0 conn(0x7f059c09fae0 0x7f059c07eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f0594008b10 tx=0x7f05940100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:04.964 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.953+0000 7f05a2343700 1 --2- 172.21.15.42:0/1061015707 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f059c09fae0 0x7f059c07eb50 secure :-1 s=READY pgs=2711 cs=0 l=1 rx=0x7f0594008b10 tx=0x7f05940100b0).ready entity=mon.1 client_cookie=570797868abd003f server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:30:04.964 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.953+0000 7f05a2343700 1 --2- 172.21.15.42:0/1061015707 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f059c09fae0 0x7f059c07eb50 secure :-1 s=THROTTLE_DONE pgs=2711 cs=0 l=1 rx=0x7f0594008b10 tx=0x7f05940100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:04.964 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.953+0000 7f05a0b40700 1 -- 172.21.15.42:0/1061015707 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0594010da0 con 0x7f059c09fae0 2019-12-05T06:30:04.964 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.953+0000 7f05a2343700 1 --2- 172.21.15.42:0/1061015707 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f059c09fae0 0x7f059c07eb50 secure :-1 s=THROTTLE_DONE pgs=2711 cs=0 l=1 rx=0x7f0594008b10 tx=0x7f05940100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:04.964 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.953+0000 7f05a2343700 1 --2- 172.21.15.42:0/1061015707 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f059c09fae0 0x7f059c07eb50 secure :-1 s=THROTTLE_DONE pgs=2711 cs=0 l=1 rx=0x7f0594008b10 tx=0x7f05940100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:04.965 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.953+0000 7f05a0b40700 1 -- 172.21.15.42:0/1061015707 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05940109c0 con 0x7f059c09fae0 2019-12-05T06:30:04.965 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.953+0000 7f05a0b40700 1 -- 172.21.15.42:0/1061015707 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f05940114a0 con 0x7f059c09fae0 2019-12-05T06:30:04.965 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.953+0000 7f05a35a5700 1 -- 172.21.15.42:0/1061015707 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f059c09fae0 msgr2=0x7f059c07eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:04.965 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.953+0000 7f05a35a5700 1 --2- 172.21.15.42:0/1061015707 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f059c09fae0 0x7f059c07eb50 secure :-1 s=READY pgs=2711 cs=0 l=1 rx=0x7f0594008b10 tx=0x7f05940100b0).stop 2019-12-05T06:30:04.965 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.957+0000 7f05a35a5700 1 -- 172.21.15.42:0/1061015707 shutdown_connections 2019-12-05T06:30:04.965 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.957+0000 7f05a35a5700 1 --2- 172.21.15.42:0/1061015707 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f059c09fae0 0x7f059c07eb50 unknown :-1 s=CLOSED pgs=2711 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:04.965 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.957+0000 7f05a35a5700 1 --2- 172.21.15.42:0/1061015707 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f059c0a7740 0x7f059c08df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:30:04.966 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.957+0000 7f05a35a5700 1 -- 172.21.15.42:0/1061015707 shutdown_connections 2019-12-05T06:30:04.966 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.957+0000 7f05a35a5700 1 -- 172.21.15.42:0/1061015707 wait complete. 2019-12-05T06:30:04.966 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.957+0000 7f05a35a5700 1 -- 172.21.15.42:0/1061015707 >> 172.21.15.42:0/1061015707 conn(0x7f059c0aabe0 msgr2=0x7f059c0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:30:04.970 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.957+0000 7f05a35a5700 1 Processor -- start 2019-12-05T06:30:04.971 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.957+0000 7f05a35a5700 1 -- start start 2019-12-05T06:30:04.971 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.961+0000 7f05a35a5700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f059c123f60 0x7f059c117f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:30:04.971 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.961+0000 7f05a35a5700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f059c11acc0 0x7f059c0a5d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:30:04.971 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.961+0000 7f05a35a5700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f059c11b0f0 con 0x7f059c11b400 2019-12-05T06:30:04.971 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.961+0000 7f05a35a5700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f059c07da20 con 0x7f059c11acc0 2019-12-05T06:30:04.971 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.961+0000 7f05a35a5700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f059c117c80 con 0x7f059c123f60 2019-12-05T06:30:04.972 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.961+0000 7f05a1341700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f059c11acc0 0x7f059c0a5d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:30:04.972 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.961+0000 7f05a1b42700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f059c123f60 0x7f059c117f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:30:04.972 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.961+0000 7f05a2343700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f059c11b400 0x7f059c07d220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:56354/0 (socket says 172.21.15.42:56354) 2019-12-05T06:30:04.972 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.961+0000 7f05a2343700 1 -- 172.21.15.42:0/3978189231 learned_addr learned my addr 172.21.15.42:0/3978189231 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:30:04.972 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.961+0000 7f05a1341700 1 -- 172.21.15.42:0/3978189231 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f059c123f60 msgr2=0x7f059c117f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:04.972 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.961+0000 7f05927fc700 1 -- 172.21.15.42:0/3978189231 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1466190450 0 0) 0x7f059c11b0f0 con 0x7f059c11b400 2019-12-05T06:30:04.972 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.961+0000 7f05a1341700 1 --2- 172.21.15.42:0/3978189231 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f059c123f60 0x7f059c117f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:04.973 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.961+0000 7f05a1341700 1 -- 172.21.15.42:0/3978189231 >> v1:172.21.15.42:6789/0 conn(0x7f059c11b400 legacy=0x7f059c07d220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:04.973 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.961+0000 7f05a1341700 1 -- 172.21.15.42:0/3978189231 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f059c0a6270 con 0x7f059c11acc0 2019-12-05T06:30:04.973 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.961+0000 7f05a1341700 1 --2- 172.21.15.42:0/3978189231 >> v2:172.21.15.42:3300/0 conn(0x7f059c11acc0 0x7f059c0a5d20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f058c000b80 tx=0x7f058c010810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:04.973 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.961+0000 7f05a1341700 1 --2- 172.21.15.42:0/3978189231 >> v2:172.21.15.42:3300/0 conn(0x7f059c11acc0 0x7f059c0a5d20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f058c000b80 tx=0x7f058c010810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:04.973 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.961+0000 7f05a1341700 1 --2- 172.21.15.42:0/3978189231 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f059c11acc0 0x7f059c0a5d20 secure :-1 s=READY pgs=2713 cs=0 l=1 rx=0x7f058c000b80 tx=0x7f058c010810).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:30:04.973 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.961+0000 7f05a1341700 1 --2- 172.21.15.42:0/3978189231 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f059c11acc0 0x7f059c0a5d20 secure :-1 s=THROTTLE_DONE pgs=2713 cs=0 l=1 rx=0x7f058c000b80 tx=0x7f058c010810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:04.973 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.961+0000 7f05927fc700 1 -- 172.21.15.42:0/3978189231 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f058c006a40 con 0x7f059c11acc0 2019-12-05T06:30:04.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.961+0000 7f05a1341700 1 --2- 172.21.15.42:0/3978189231 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f059c11acc0 0x7f059c0a5d20 secure :-1 s=THROTTLE_DONE pgs=2713 cs=0 l=1 rx=0x7f058c000b80 tx=0x7f058c010810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:04.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.961+0000 7f05a1341700 1 --2- 172.21.15.42:0/3978189231 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f059c11acc0 0x7f059c0a5d20 secure :-1 s=THROTTLE_DONE pgs=2713 cs=0 l=1 rx=0x7f058c000b80 tx=0x7f058c010810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:04.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.961+0000 7f05927fc700 1 -- 172.21.15.42:0/3978189231 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f058c006660 con 0x7f059c11acc0 2019-12-05T06:30:04.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.961+0000 7f05a35a5700 1 -- 172.21.15.42:0/3978189231 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f059c0a6be0 con 0x7f059c11acc0 2019-12-05T06:30:04.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.961+0000 7f05927fc700 1 -- 172.21.15.42:0/3978189231 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f058c011050 con 0x7f059c11acc0 2019-12-05T06:30:04.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.961+0000 7f05a35a5700 1 -- 172.21.15.42:0/3978189231 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f059c137660 con 0x7f059c11acc0 2019-12-05T06:30:04.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.961+0000 7f05a1341700 1 --2- 172.21.15.42:0/3978189231 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f059c11acc0 0x7f059c0a5d20 secure :-1 s=THROTTLE_DONE pgs=2713 cs=0 l=1 rx=0x7f058c000b80 tx=0x7f058c010810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:04.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.961+0000 7f05927fc700 1 -- 172.21.15.42:0/3978189231 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f058c011360 con 0x7f059c11acc0 2019-12-05T06:30:04.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.961+0000 7f05a1341700 1 --2- 172.21.15.42:0/3978189231 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f059c11acc0 0x7f059c0a5d20 secure :-1 s=THROTTLE_DONE pgs=2713 cs=0 l=1 rx=0x7f058c000b80 tx=0x7f058c010810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:04.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.965+0000 7f05927fc700 1 --2- 172.21.15.42:0/3978189231 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f058403ace0 0x7f058403d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:30:04.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.965+0000 7f05927fc700 1 -- 172.21.15.42:0/3978189231 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 11019+0+0 (secure 0 0 0) 0x7f058c04b0a0 con 0x7f059c11acc0 2019-12-05T06:30:04.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.965+0000 7f05a1b42700 1 --2- 172.21.15.42:0/3978189231 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f058403ace0 0x7f058403d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:30:04.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.965+0000 7f05a1b42700 1 --2- 172.21.15.42:0/3978189231 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f058403ace0 0x7f058403d1f0 crc :-1 s=READY pgs=1138 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:30:04.976 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.965+0000 7f05a35a5700 1 -- 172.21.15.42:0/3978189231 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f059c05d110 con 0x7f059c11acc0 2019-12-05T06:30:04.978 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.973+0000 7f05a1341700 1 --2- 172.21.15.42:0/3978189231 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f059c11acc0 0x7f059c0a5d20 secure :-1 s=THROTTLE_DONE pgs=2713 cs=0 l=1 rx=0x7f058c000b80 tx=0x7f058c010810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:04.978 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:04.973+0000 7f05927fc700 1 -- 172.21.15.42:0/3978189231 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f059c0a6270 con 0x7f059c11acc0 2019-12-05T06:30:05.135 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:05.129+0000 7f05a35a5700 1 -- 172.21.15.42:0/3978189231 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f059c05b8c0 con 0x7f059c11acc0 2019-12-05T06:30:05.136 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:05.129+0000 7f05a1341700 1 --2- 172.21.15.42:0/3978189231 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f059c11acc0 0x7f059c0a5d20 secure :-1 s=THROTTLE_DONE pgs=2713 cs=0 l=1 rx=0x7f058c000b80 tx=0x7f058c010810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:05.136 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:05.129+0000 7f05927fc700 1 -- 172.21.15.42:0/3978189231 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 323 v323) v1 ==== 95+0+2117 (secure 0 0 0) 0x7f058c0320c0 con 0x7f059c11acc0 2019-12-05T06:30:05.136 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 323 2019-12-05T06:30:05.136 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:30:05.137 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":323,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":11265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/1985322937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1985322937},{"type":"v1","addr":"172.21.15.42:6835","nonce":1985322937}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":323}],"filesystems":[{"mdsmap":{"epoch":322,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:28:29.970815+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":107,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11142},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11142":{"gid":11142,"name":"b","rank":0,"incarnation":319,"state":"up:active","state_seq":28,"addr":"172.21.15.40:6833/3155851600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3155851600},{"type":"v1","addr":"172.21.15.40:6833","nonce":3155851600}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:30:05.139 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:05.133+0000 7f05a35a5700 1 -- 172.21.15.42:0/3978189231 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f058403ace0 msgr2=0x7f058403d1f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:05.139 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:05.133+0000 7f05a35a5700 1 --2- 172.21.15.42:0/3978189231 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f058403ace0 0x7f058403d1f0 crc :-1 s=READY pgs=1138 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:05.139 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:05.133+0000 7f05a35a5700 1 -- 172.21.15.42:0/3978189231 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f059c11acc0 msgr2=0x7f059c0a5d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:05.139 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:05.133+0000 7f05a35a5700 1 --2- 172.21.15.42:0/3978189231 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f059c11acc0 0x7f059c0a5d20 secure :-1 s=READY pgs=2713 cs=0 l=1 rx=0x7f058c000b80 tx=0x7f058c010810).stop 2019-12-05T06:30:05.139 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:05.133+0000 7f05a35a5700 1 -- 172.21.15.42:0/3978189231 shutdown_connections 2019-12-05T06:30:05.140 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:05.133+0000 7f05a35a5700 1 --2- 172.21.15.42:0/3978189231 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f058403ace0 0x7f058403d1f0 unknown :-1 s=CLOSED pgs=1138 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:05.140 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:05.133+0000 7f05a35a5700 1 --2- 172.21.15.42:0/3978189231 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f059c123f60 0x7f059c117f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:05.140 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:05.133+0000 7f05a35a5700 1 --2- 172.21.15.42:0/3978189231 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f059c11acc0 0x7f059c0a5d20 unknown :-1 s=CLOSED pgs=2713 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:05.140 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:05.133+0000 7f05a35a5700 1 -- 172.21.15.42:0/3978189231 shutdown_connections 2019-12-05T06:30:05.140 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:05.133+0000 7f05a35a5700 1 -- 172.21.15.42:0/3978189231 wait complete. 2019-12-05T06:30:05.140 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:05.133+0000 7f05a35a5700 1 -- 172.21.15.42:0/3978189231 >> 172.21.15.42:0/3978189231 conn(0x7f059c09f3c0 msgr2=0x7f059c059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:30:05.154 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:30:07.161 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:30:07.273 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:07.261+0000 7ff75dbb0700 1 Processor -- start 2019-12-05T06:30:07.273 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:07.261+0000 7ff75dbb0700 1 -- start start 2019-12-05T06:30:07.274 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:07.265+0000 7ff75dbb0700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff7580a7740 0x7ff75808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:30:07.274 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:07.265+0000 7ff75dbb0700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff75809fae0 0x7ff75807eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:30:07.274 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:07.265+0000 7ff75dbb0700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7580a0710 con 0x7ff75807e720 2019-12-05T06:30:07.284 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:07.265+0000 7ff75dbb0700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7580a0b00 con 0x7ff75809fae0 2019-12-05T06:30:07.284 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:07.265+0000 7ff75dbb0700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff75808e5b0 con 0x7ff7580a7740 2019-12-05T06:30:07.284 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:07.265+0000 7ff74f7fe700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7ff75807e720 0x7ff75809f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:56366/0 (socket says 172.21.15.42:56366) 2019-12-05T06:30:07.285 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:07.265+0000 7ff74f7fe700 1 -- 172.21.15.42:0/690659035 learned_addr learned my addr 172.21.15.42:0/690659035 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:30:07.287 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:07.265+0000 7ff757fff700 1 --2- 172.21.15.42:0/690659035 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff7580a7740 0x7ff75808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:30:07.287 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:07.265+0000 7ff75c94e700 1 --2- 172.21.15.42:0/690659035 >> v2:172.21.15.42:3300/0 conn(0x7ff75809fae0 0x7ff75807eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:30:07.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:07.265+0000 7ff7577fe700 1 -- 172.21.15.42:0/690659035 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 873204384 0 0) 0x7ff7580a0710 con 0x7ff75807e720 2019-12-05T06:30:07.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:07.265+0000 7ff7577fe700 1 -- 172.21.15.42:0/690659035 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7440030b0 con 0x7ff75807e720 2019-12-05T06:30:07.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:07.265+0000 7ff75c94e700 1 -- 172.21.15.42:0/690659035 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff7580a7740 msgr2=0x7ff75808df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:30:07.291 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:07.265+0000 7ff75c94e700 1 --2- 172.21.15.42:0/690659035 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff7580a7740 0x7ff75808df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:30:07.292 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:07.265+0000 7ff75c94e700 1 -- 172.21.15.42:0/690659035 >> v1:172.21.15.42:6789/0 conn(0x7ff75807e720 legacy=0x7ff75809f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:07.292 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:07.265+0000 7ff75c94e700 1 -- 172.21.15.42:0/690659035 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff75807f0a0 con 0x7ff75809fae0 2019-12-05T06:30:07.292 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:07.265+0000 7ff757fff700 1 --2- 172.21.15.42:0/690659035 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff7580a7740 0x7ff75808df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:30:07.292 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:07.265+0000 7ff75c94e700 1 --2- 172.21.15.42:0/690659035 >> v2:172.21.15.42:3300/0 conn(0x7ff75809fae0 0x7ff75807eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7ff750008b10 tx=0x7ff75000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:07.293 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:07.265+0000 7ff75c94e700 1 --2- 172.21.15.42:0/690659035 >> v2:172.21.15.42:3300/0 conn(0x7ff75809fae0 0x7ff75807eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7ff750008b10 tx=0x7ff75000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:07.293 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:07.265+0000 7ff75c94e700 1 --2- 172.21.15.42:0/690659035 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff75809fae0 0x7ff75807eb50 secure :-1 s=READY pgs=2715 cs=0 l=1 rx=0x7ff750008b10 tx=0x7ff75000fff0).ready entity=mon.1 client_cookie=640b43aaaee37da6 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:30:07.293 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:07.265+0000 7ff75c94e700 1 --2- 172.21.15.42:0/690659035 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff75809fae0 0x7ff75807eb50 secure :-1 s=THROTTLE_DONE pgs=2715 cs=0 l=1 rx=0x7ff750008b10 tx=0x7ff75000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:07.294 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:07.265+0000 7ff7577fe700 1 -- 172.21.15.42:0/690659035 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff750010900 con 0x7ff75809fae0 2019-12-05T06:30:07.294 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:07.265+0000 7ff75c94e700 1 --2- 172.21.15.42:0/690659035 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff75809fae0 0x7ff75807eb50 secure :-1 s=THROTTLE_DONE pgs=2715 cs=0 l=1 rx=0x7ff750008b10 tx=0x7ff75000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:07.294 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:07.265+0000 7ff75c94e700 1 --2- 172.21.15.42:0/690659035 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff75809fae0 0x7ff75807eb50 secure :-1 s=THROTTLE_DONE pgs=2715 cs=0 l=1 rx=0x7ff750008b10 tx=0x7ff75000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:07.294 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:07.265+0000 7ff7577fe700 1 -- 172.21.15.42:0/690659035 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff750008b50 con 0x7ff75809fae0 2019-12-05T06:30:07.296 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:07.265+0000 7ff7577fe700 1 -- 172.21.15.42:0/690659035 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff7500110c0 con 0x7ff75809fae0 2019-12-05T06:30:07.297 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:07.265+0000 7ff75dbb0700 1 -- 172.21.15.42:0/690659035 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff75809fae0 msgr2=0x7ff75807eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:07.297 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:07.265+0000 7ff75dbb0700 1 --2- 172.21.15.42:0/690659035 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff75809fae0 0x7ff75807eb50 secure :-1 s=READY pgs=2715 cs=0 l=1 rx=0x7ff750008b10 tx=0x7ff75000fff0).stop 2019-12-05T06:30:07.298 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:07.265+0000 7ff75dbb0700 1 -- 172.21.15.42:0/690659035 shutdown_connections 2019-12-05T06:30:07.298 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:07.265+0000 7ff75dbb0700 1 --2- 172.21.15.42:0/690659035 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff75809fae0 0x7ff75807eb50 unknown :-1 s=CLOSED pgs=2715 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:07.299 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:07.265+0000 7ff75dbb0700 1 --2- 172.21.15.42:0/690659035 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff7580a7740 0x7ff75808df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:30:07.299 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:07.265+0000 7ff75dbb0700 1 -- 172.21.15.42:0/690659035 shutdown_connections 2019-12-05T06:30:07.299 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:07.265+0000 7ff75dbb0700 1 -- 172.21.15.42:0/690659035 wait complete. 2019-12-05T06:30:07.299 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:07.265+0000 7ff75dbb0700 1 -- 172.21.15.42:0/690659035 >> 172.21.15.42:0/690659035 conn(0x7ff7580aabe0 msgr2=0x7ff7580a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:30:07.325 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:07.269+0000 7ff75dbb0700 1 Processor -- start 2019-12-05T06:30:07.325 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:07.269+0000 7ff75dbb0700 1 -- start start 2019-12-05T06:30:07.326 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:07.269+0000 7ff75dbb0700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff75811baa0 0x7ff75811bed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:30:07.326 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:07.269+0000 7ff75dbb0700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff758118f80 0x7ff7581193b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:30:07.326 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:07.269+0000 7ff75dbb0700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff75812e320 con 0x7ff75811dff0 2019-12-05T06:30:07.326 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:07.269+0000 7ff75dbb0700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff758148710 con 0x7ff758118f80 2019-12-05T06:30:07.327 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:07.269+0000 7ff75dbb0700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff758118670 con 0x7ff75811baa0 2019-12-05T06:30:07.327 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:07.269+0000 7ff757fff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7ff75811dff0 0x7ff758116ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:56370/0 (socket says 172.21.15.42:56370) 2019-12-05T06:30:07.327 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:07.269+0000 7ff757fff700 1 -- 172.21.15.42:0/1877118190 learned_addr learned my addr 172.21.15.42:0/1877118190 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:30:07.327 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:07.273+0000 7ff74f7fe700 1 --2- 172.21.15.42:0/1877118190 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff75811baa0 0x7ff75811bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:30:07.327 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:07.273+0000 7ff75c94e700 1 --2- 172.21.15.42:0/1877118190 >> v2:172.21.15.40:3300/0 conn(0x7ff758118f80 0x7ff7581193b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:30:07.327 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:07.273+0000 7ff7557fa700 1 -- 172.21.15.42:0/1877118190 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2306414331 0 0) 0x7ff75812e320 con 0x7ff75811dff0 2019-12-05T06:30:07.328 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:07.273+0000 7ff7557fa700 1 -- 172.21.15.42:0/1877118190 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff73c0030b0 con 0x7ff75811dff0 2019-12-05T06:30:07.328 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:07.273+0000 7ff74f7fe700 1 -- 172.21.15.42:0/1877118190 >> v2:172.21.15.40:3300/0 conn(0x7ff758118f80 msgr2=0x7ff7581193b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:07.328 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:07.273+0000 7ff74f7fe700 1 --2- 172.21.15.42:0/1877118190 >> v2:172.21.15.40:3300/0 conn(0x7ff758118f80 0x7ff7581193b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:07.328 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:07.273+0000 7ff74f7fe700 1 -- 172.21.15.42:0/1877118190 >> v1:172.21.15.42:6789/0 conn(0x7ff75811dff0 legacy=0x7ff758116ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:07.328 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:07.273+0000 7ff74f7fe700 1 -- 172.21.15.42:0/1877118190 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff758136780 con 0x7ff75811baa0 2019-12-05T06:30:07.328 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:07.273+0000 7ff74f7fe700 1 --2- 172.21.15.42:0/1877118190 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff75811baa0 0x7ff75811bed0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7ff740004370 tx=0x7ff7400108b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:07.329 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:07.273+0000 7ff74f7fe700 1 --2- 172.21.15.42:0/1877118190 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff75811baa0 0x7ff75811bed0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7ff740004370 tx=0x7ff7400108b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:07.329 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:07.273+0000 7ff74f7fe700 1 --2- 172.21.15.42:0/1877118190 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff75811baa0 0x7ff75811bed0 secure :-1 s=READY pgs=596 cs=0 l=1 rx=0x7ff740004370 tx=0x7ff7400108b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:30:07.329 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:07.273+0000 7ff74f7fe700 1 --2- 172.21.15.42:0/1877118190 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff75811baa0 0x7ff75811bed0 secure :-1 s=THROTTLE_DONE pgs=596 cs=0 l=1 rx=0x7ff740004370 tx=0x7ff7400108b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:07.329 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:07.273+0000 7ff74f7fe700 1 --2- 172.21.15.42:0/1877118190 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff75811baa0 0x7ff75811bed0 secure :-1 s=THROTTLE_DONE pgs=596 cs=0 l=1 rx=0x7ff740004370 tx=0x7ff7400108b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:07.329 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:07.273+0000 7ff7557fa700 1 -- 172.21.15.42:0/1877118190 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff740005a50 con 0x7ff75811baa0 2019-12-05T06:30:07.330 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:07.273+0000 7ff74f7fe700 1 --2- 172.21.15.42:0/1877118190 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff75811baa0 0x7ff75811bed0 secure :-1 s=THROTTLE_DONE pgs=596 cs=0 l=1 rx=0x7ff740004370 tx=0x7ff7400108b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:07.330 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:07.273+0000 7ff7557fa700 1 -- 172.21.15.42:0/1877118190 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff740005670 con 0x7ff75811baa0 2019-12-05T06:30:07.330 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:07.273+0000 7ff75dbb0700 1 -- 172.21.15.42:0/1877118190 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff758135770 con 0x7ff75811baa0 2019-12-05T06:30:07.330 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:07.273+0000 7ff7557fa700 1 -- 172.21.15.42:0/1877118190 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff7400062b0 con 0x7ff75811baa0 2019-12-05T06:30:07.331 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:07.273+0000 7ff75dbb0700 1 -- 172.21.15.42:0/1877118190 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff758135c70 con 0x7ff75811baa0 2019-12-05T06:30:07.331 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:07.273+0000 7ff74f7fe700 1 --2- 172.21.15.42:0/1877118190 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff75811baa0 0x7ff75811bed0 secure :-1 s=THROTTLE_DONE pgs=596 cs=0 l=1 rx=0x7ff740004370 tx=0x7ff7400108b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:07.332 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:07.273+0000 7ff7557fa700 1 -- 172.21.15.42:0/1877118190 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7ff7400065c0 con 0x7ff75811baa0 2019-12-05T06:30:07.333 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:07.273+0000 7ff74f7fe700 1 --2- 172.21.15.42:0/1877118190 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff75811baa0 0x7ff75811bed0 secure :-1 s=THROTTLE_DONE pgs=596 cs=0 l=1 rx=0x7ff740004370 tx=0x7ff7400108b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:07.333 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:07.273+0000 7ff7557fa700 1 --2- 172.21.15.42:0/1877118190 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff73c03b230 0x7ff73c03d740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:30:07.334 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:07.273+0000 7ff7557fa700 1 -- 172.21.15.42:0/1877118190 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 11019+0+0 (secure 0 0 0) 0x7ff74004af20 con 0x7ff75811baa0 2019-12-05T06:30:07.334 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:07.277+0000 7ff757fff700 1 --2- 172.21.15.42:0/1877118190 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff73c03b230 0x7ff73c03d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:30:07.334 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:07.277+0000 7ff757fff700 1 --2- 172.21.15.42:0/1877118190 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff73c03b230 0x7ff73c03d740 crc :-1 s=READY pgs=1139 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:30:07.335 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:07.277+0000 7ff75dbb0700 1 -- 172.21.15.42:0/1877118190 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff75805d110 con 0x7ff75811baa0 2019-12-05T06:30:07.335 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:07.285+0000 7ff74f7fe700 1 --2- 172.21.15.42:0/1877118190 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff75811baa0 0x7ff75811bed0 secure :-1 s=THROTTLE_DONE pgs=596 cs=0 l=1 rx=0x7ff740004370 tx=0x7ff7400108b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:07.335 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:07.285+0000 7ff7557fa700 1 -- 172.21.15.42:0/1877118190 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7ff740031fa0 con 0x7ff75811baa0 2019-12-05T06:30:07.452 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:07.445+0000 7ff75dbb0700 1 -- 172.21.15.42:0/1877118190 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff75805b8c0 con 0x7ff75811baa0 2019-12-05T06:30:07.453 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:07.449+0000 7ff74f7fe700 1 --2- 172.21.15.42:0/1877118190 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff75811baa0 0x7ff75811bed0 secure :-1 s=THROTTLE_DONE pgs=596 cs=0 l=1 rx=0x7ff740004370 tx=0x7ff7400108b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:07.453 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:07.449+0000 7ff7557fa700 1 -- 172.21.15.42:0/1877118190 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 323 v323) v1 ==== 95+0+2117 (secure 0 0 0) 0x7ff740032200 con 0x7ff75811baa0 2019-12-05T06:30:07.456 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 323 2019-12-05T06:30:07.456 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:30:07.456 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":323,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":11265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/1985322937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1985322937},{"type":"v1","addr":"172.21.15.42:6835","nonce":1985322937}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":323}],"filesystems":[{"mdsmap":{"epoch":322,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:28:29.970815+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":107,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11142},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11142":{"gid":11142,"name":"b","rank":0,"incarnation":319,"state":"up:active","state_seq":28,"addr":"172.21.15.40:6833/3155851600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3155851600},{"type":"v1","addr":"172.21.15.40:6833","nonce":3155851600}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:30:07.458 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:07.453+0000 7ff75dbb0700 1 -- 172.21.15.42:0/1877118190 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff73c03b230 msgr2=0x7ff73c03d740 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:07.458 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:07.453+0000 7ff75dbb0700 1 --2- 172.21.15.42:0/1877118190 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff73c03b230 0x7ff73c03d740 crc :-1 s=READY pgs=1139 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:07.459 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:07.453+0000 7ff75dbb0700 1 -- 172.21.15.42:0/1877118190 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff75811baa0 msgr2=0x7ff75811bed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:07.460 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:07.453+0000 7ff75dbb0700 1 --2- 172.21.15.42:0/1877118190 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff75811baa0 0x7ff75811bed0 secure :-1 s=READY pgs=596 cs=0 l=1 rx=0x7ff740004370 tx=0x7ff7400108b0).stop 2019-12-05T06:30:07.460 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:07.453+0000 7ff75dbb0700 1 -- 172.21.15.42:0/1877118190 shutdown_connections 2019-12-05T06:30:07.460 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:07.453+0000 7ff75dbb0700 1 --2- 172.21.15.42:0/1877118190 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff73c03b230 0x7ff73c03d740 unknown :-1 s=CLOSED pgs=1139 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:07.460 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:07.453+0000 7ff75dbb0700 1 --2- 172.21.15.42:0/1877118190 >> v2:172.21.15.40:3300/0 conn(0x7ff758118f80 0x7ff7581193b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:07.460 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:07.453+0000 7ff75dbb0700 1 --2- 172.21.15.42:0/1877118190 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff75811baa0 0x7ff75811bed0 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:07.460 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:07.453+0000 7ff75dbb0700 1 -- 172.21.15.42:0/1877118190 shutdown_connections 2019-12-05T06:30:07.460 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:07.453+0000 7ff75dbb0700 1 -- 172.21.15.42:0/1877118190 wait complete. 2019-12-05T06:30:07.461 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:07.453+0000 7ff75dbb0700 1 -- 172.21.15.42:0/1877118190 >> 172.21.15.42:0/1877118190 conn(0x7ff7580a7f90 msgr2=0x7ff7580a9790 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:30:07.468 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:30:09.469 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:30:09.580 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:09.565+0000 7fec3b298700 1 Processor -- start 2019-12-05T06:30:09.581 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:09.565+0000 7fec3b298700 1 -- start start 2019-12-05T06:30:09.581 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:09.565+0000 7fec3b298700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fec3408df80 0x7fec3409f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:30:09.582 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:09.565+0000 7fec3b298700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fec3409f830 0x7fec3414efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:30:09.582 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:09.565+0000 7fec3b298700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec3409fc60 con 0x7fec340a7740 2019-12-05T06:30:09.582 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:09.565+0000 7fec3b298700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec340a0b00 con 0x7fec3409f830 2019-12-05T06:30:09.582 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:09.565+0000 7fec3b298700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec3408e5b0 con 0x7fec3408df80 2019-12-05T06:30:09.582 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:09.565+0000 7fec3a036700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fec3409f830 0x7fec3414efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:30:09.583 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:09.565+0000 7fec39835700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fec340a7740 0x7fec3407e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:37588/0 (socket says 172.21.15.42:37588) 2019-12-05T06:30:09.583 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:09.565+0000 7fec39835700 1 -- 172.21.15.42:0/573786693 learned_addr learned my addr 172.21.15.42:0/573786693 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:30:09.583 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:09.565+0000 7fec39034700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fec3408df80 0x7fec3409f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:30:09.583 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:09.565+0000 7fec38833700 1 -- 172.21.15.42:0/573786693 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2391430836 0 0) 0x7fec3409fc60 con 0x7fec340a7740 2019-12-05T06:30:09.584 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:09.565+0000 7fec38833700 1 -- 172.21.15.42:0/573786693 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec200030b0 con 0x7fec340a7740 2019-12-05T06:30:09.584 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:09.565+0000 7fec3a036700 1 -- 172.21.15.42:0/573786693 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fec3408df80 msgr2=0x7fec3409f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:30:09.584 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:09.565+0000 7fec3a036700 1 --2- 172.21.15.42:0/573786693 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fec3408df80 0x7fec3409f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:30:09.584 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:09.565+0000 7fec3a036700 1 -- 172.21.15.42:0/573786693 >> v1:172.21.15.40:6789/0 conn(0x7fec340a7740 legacy=0x7fec3407e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:09.584 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:09.565+0000 7fec3a036700 1 -- 172.21.15.42:0/573786693 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec340a0710 con 0x7fec3409f830 2019-12-05T06:30:09.585 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:09.565+0000 7fec39034700 1 --2- 172.21.15.42:0/573786693 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fec3408df80 0x7fec3409f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:30:09.585 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:09.565+0000 7fec3a036700 1 --2- 172.21.15.42:0/573786693 >> v2:172.21.15.42:3300/0 conn(0x7fec3409f830 0x7fec3414efe0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fec2c008790 tx=0x7fec2c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:09.585 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:09.565+0000 7fec3a036700 1 --2- 172.21.15.42:0/573786693 >> v2:172.21.15.42:3300/0 conn(0x7fec3409f830 0x7fec3414efe0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fec2c008790 tx=0x7fec2c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:09.585 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:09.565+0000 7fec3a036700 1 --2- 172.21.15.42:0/573786693 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fec3409f830 0x7fec3414efe0 secure :-1 s=READY pgs=2717 cs=0 l=1 rx=0x7fec2c008790 tx=0x7fec2c00fc70).ready entity=mon.1 client_cookie=18ce2d3eacc0ac4f server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:30:09.585 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:09.569+0000 7fec3a036700 1 --2- 172.21.15.42:0/573786693 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fec3409f830 0x7fec3414efe0 secure :-1 s=THROTTLE_DONE pgs=2717 cs=0 l=1 rx=0x7fec2c008790 tx=0x7fec2c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:09.585 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:09.569+0000 7fec38833700 1 -- 172.21.15.42:0/573786693 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fec2c008bb0 con 0x7fec3409f830 2019-12-05T06:30:09.586 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:09.569+0000 7fec3a036700 1 --2- 172.21.15.42:0/573786693 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fec3409f830 0x7fec3414efe0 secure :-1 s=THROTTLE_DONE pgs=2717 cs=0 l=1 rx=0x7fec2c008790 tx=0x7fec2c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:09.586 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:09.569+0000 7fec3a036700 1 --2- 172.21.15.42:0/573786693 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fec3409f830 0x7fec3414efe0 secure :-1 s=THROTTLE_DONE pgs=2717 cs=0 l=1 rx=0x7fec2c008790 tx=0x7fec2c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:09.586 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:09.569+0000 7fec38833700 1 -- 172.21.15.42:0/573786693 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec2c0087d0 con 0x7fec3409f830 2019-12-05T06:30:09.586 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:09.569+0000 7fec38833700 1 -- 172.21.15.42:0/573786693 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fec2c011860 con 0x7fec3409f830 2019-12-05T06:30:09.587 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:09.569+0000 7fec3b298700 1 -- 172.21.15.42:0/573786693 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fec3409f830 msgr2=0x7fec3414efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:09.587 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:09.569+0000 7fec3b298700 1 --2- 172.21.15.42:0/573786693 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fec3409f830 0x7fec3414efe0 secure :-1 s=READY pgs=2717 cs=0 l=1 rx=0x7fec2c008790 tx=0x7fec2c00fc70).stop 2019-12-05T06:30:09.587 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:09.569+0000 7fec3b298700 1 -- 172.21.15.42:0/573786693 shutdown_connections 2019-12-05T06:30:09.587 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:09.569+0000 7fec3b298700 1 --2- 172.21.15.42:0/573786693 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fec3409f830 0x7fec3414efe0 unknown :-1 s=CLOSED pgs=2717 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:09.587 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:09.569+0000 7fec3b298700 1 --2- 172.21.15.42:0/573786693 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fec3408df80 0x7fec3409f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:30:09.588 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:09.569+0000 7fec3b298700 1 -- 172.21.15.42:0/573786693 shutdown_connections 2019-12-05T06:30:09.588 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:09.569+0000 7fec3b298700 1 -- 172.21.15.42:0/573786693 wait complete. 2019-12-05T06:30:09.588 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:09.569+0000 7fec3b298700 1 -- 172.21.15.42:0/573786693 >> 172.21.15.42:0/573786693 conn(0x7fec340aabe0 msgr2=0x7fec340a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:30:09.588 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:09.569+0000 7fec3b298700 1 Processor -- start 2019-12-05T06:30:09.588 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:09.569+0000 7fec3b298700 1 -- start start 2019-12-05T06:30:09.588 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:09.573+0000 7fec3b298700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fec3407bdf0 0x7fec34138230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:30:09.589 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:09.573+0000 7fec3b298700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec34138780 con 0x7fec341220c0 2019-12-05T06:30:09.589 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:09.573+0000 7fec3b298700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec3407b2f0 con 0x7fec341323a0 2019-12-05T06:30:09.589 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:09.573+0000 7fec3b298700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec3407d780 con 0x7fec3407bdf0 2019-12-05T06:30:09.589 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:09.573+0000 7fec39034700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fec3407bdf0 0x7fec34138230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:30:09.589 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:09.573+0000 7fec3a036700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fec341323a0 0x7fec341327d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:56390/0 (socket says 172.21.15.42:56390) 2019-12-05T06:30:09.589 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:09.573+0000 7fec3a036700 1 -- 172.21.15.42:0/2355512604 learned_addr learned my addr 172.21.15.42:0/2355512604 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:30:09.590 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:09.573+0000 7fec267fc700 1 -- 172.21.15.42:0/2355512604 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 925083638 0 0) 0x7fec3407b2f0 con 0x7fec341323a0 2019-12-05T06:30:09.590 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:09.573+0000 7fec39034700 1 -- 172.21.15.42:0/2355512604 >> v1:172.21.15.42:6789/0 conn(0x7fec341323a0 legacy=0x7fec341327d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:09.590 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:09.573+0000 7fec39034700 1 -- 172.21.15.42:0/2355512604 >> v1:172.21.15.40:6789/0 conn(0x7fec341220c0 legacy=0x7fec34137a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:09.590 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:09.573+0000 7fec39034700 1 -- 172.21.15.42:0/2355512604 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec3407d390 con 0x7fec3407bdf0 2019-12-05T06:30:09.590 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:09.573+0000 7fec39034700 1 --2- 172.21.15.42:0/2355512604 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fec3407bdf0 0x7fec34138230 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fec28001090 tx=0x7fec280010d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:09.591 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:09.573+0000 7fec39034700 1 --2- 172.21.15.42:0/2355512604 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fec3407bdf0 0x7fec34138230 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fec28001090 tx=0x7fec280010d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:09.591 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:09.573+0000 7fec39034700 1 --2- 172.21.15.42:0/2355512604 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fec3407bdf0 0x7fec34138230 secure :-1 s=READY pgs=597 cs=0 l=1 rx=0x7fec28001090 tx=0x7fec280010d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:30:09.591 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:09.573+0000 7fec39034700 1 --2- 172.21.15.42:0/2355512604 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fec3407bdf0 0x7fec34138230 secure :-1 s=THROTTLE_DONE pgs=597 cs=0 l=1 rx=0x7fec28001090 tx=0x7fec280010d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:09.591 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:09.573+0000 7fec267fc700 1 -- 172.21.15.42:0/2355512604 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fec30007160 con 0x7fec3407bdf0 2019-12-05T06:30:09.592 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:09.573+0000 7fec39034700 1 --2- 172.21.15.42:0/2355512604 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fec3407bdf0 0x7fec34138230 secure :-1 s=THROTTLE_DONE pgs=597 cs=0 l=1 rx=0x7fec28001090 tx=0x7fec280010d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:09.592 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:09.573+0000 7fec39034700 1 --2- 172.21.15.42:0/2355512604 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fec3407bdf0 0x7fec34138230 secure :-1 s=THROTTLE_DONE pgs=597 cs=0 l=1 rx=0x7fec28001090 tx=0x7fec280010d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:09.592 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:09.573+0000 7fec267fc700 1 -- 172.21.15.42:0/2355512604 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec30005bc0 con 0x7fec3407bdf0 2019-12-05T06:30:09.592 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:09.573+0000 7fec3b298700 1 -- 172.21.15.42:0/2355512604 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec3407c220 con 0x7fec3407bdf0 2019-12-05T06:30:09.592 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:09.573+0000 7fec267fc700 1 -- 172.21.15.42:0/2355512604 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fec30007740 con 0x7fec3407bdf0 2019-12-05T06:30:09.593 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:09.573+0000 7fec3b298700 1 -- 172.21.15.42:0/2355512604 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec3411b360 con 0x7fec3407bdf0 2019-12-05T06:30:09.593 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:09.573+0000 7fec39034700 1 --2- 172.21.15.42:0/2355512604 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fec3407bdf0 0x7fec34138230 secure :-1 s=THROTTLE_DONE pgs=597 cs=0 l=1 rx=0x7fec28001090 tx=0x7fec280010d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:09.594 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:09.577+0000 7fec39034700 1 --2- 172.21.15.42:0/2355512604 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fec3407bdf0 0x7fec34138230 secure :-1 s=THROTTLE_DONE pgs=597 cs=0 l=1 rx=0x7fec28001090 tx=0x7fec280010d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:09.594 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:09.577+0000 7fec267fc700 1 -- 172.21.15.42:0/2355512604 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fec30012660 con 0x7fec3407bdf0 2019-12-05T06:30:09.595 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:09.577+0000 7fec267fc700 1 --2- 172.21.15.42:0/2355512604 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fec1c03ac80 0x7fec1c03d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:30:09.595 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:09.577+0000 7fec267fc700 1 -- 172.21.15.42:0/2355512604 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 11019+0+0 (secure 0 0 0) 0x7fec3004b210 con 0x7fec3407bdf0 2019-12-05T06:30:09.595 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:09.577+0000 7fec39835700 1 --2- 172.21.15.42:0/2355512604 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fec1c03ac80 0x7fec1c03d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:30:09.595 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:09.577+0000 7fec39835700 1 --2- 172.21.15.42:0/2355512604 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fec1c03ac80 0x7fec1c03d190 crc :-1 s=READY pgs=1140 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:30:09.595 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:09.581+0000 7fec3b298700 1 -- 172.21.15.42:0/2355512604 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec3405d110 con 0x7fec3407bdf0 2019-12-05T06:30:09.595 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:09.589+0000 7fec39034700 1 --2- 172.21.15.42:0/2355512604 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fec3407bdf0 0x7fec34138230 secure :-1 s=THROTTLE_DONE pgs=597 cs=0 l=1 rx=0x7fec28001090 tx=0x7fec280010d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:09.596 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:09.589+0000 7fec267fc700 1 -- 172.21.15.42:0/2355512604 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fec30032250 con 0x7fec3407bdf0 2019-12-05T06:30:09.762 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:09.757+0000 7fec3b298700 1 -- 172.21.15.42:0/2355512604 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fec3405b8c0 con 0x7fec3407bdf0 2019-12-05T06:30:09.763 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:09.757+0000 7fec39034700 1 --2- 172.21.15.42:0/2355512604 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fec3407bdf0 0x7fec34138230 secure :-1 s=THROTTLE_DONE pgs=597 cs=0 l=1 rx=0x7fec28001090 tx=0x7fec280010d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:09.763 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:09.757+0000 7fec267fc700 1 -- 172.21.15.42:0/2355512604 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 323 v323) v1 ==== 95+0+2117 (secure 0 0 0) 0x7fec300324b0 con 0x7fec3407bdf0 2019-12-05T06:30:09.765 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 323 2019-12-05T06:30:09.765 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:30:09.766 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":323,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":11265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/1985322937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1985322937},{"type":"v1","addr":"172.21.15.42:6835","nonce":1985322937}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":323}],"filesystems":[{"mdsmap":{"epoch":322,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:28:29.970815+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":107,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11142},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11142":{"gid":11142,"name":"b","rank":0,"incarnation":319,"state":"up:active","state_seq":28,"addr":"172.21.15.40:6833/3155851600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3155851600},{"type":"v1","addr":"172.21.15.40:6833","nonce":3155851600}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:30:09.768 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:09.761+0000 7fec3b298700 1 -- 172.21.15.42:0/2355512604 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fec1c03ac80 msgr2=0x7fec1c03d190 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:09.769 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:09.761+0000 7fec3b298700 1 --2- 172.21.15.42:0/2355512604 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fec1c03ac80 0x7fec1c03d190 crc :-1 s=READY pgs=1140 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:09.769 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:09.761+0000 7fec3b298700 1 -- 172.21.15.42:0/2355512604 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fec3407bdf0 msgr2=0x7fec34138230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:09.769 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:09.761+0000 7fec3b298700 1 --2- 172.21.15.42:0/2355512604 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fec3407bdf0 0x7fec34138230 secure :-1 s=READY pgs=597 cs=0 l=1 rx=0x7fec28001090 tx=0x7fec280010d0).stop 2019-12-05T06:30:09.769 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:09.761+0000 7fec3b298700 1 -- 172.21.15.42:0/2355512604 shutdown_connections 2019-12-05T06:30:09.770 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:09.761+0000 7fec3b298700 1 --2- 172.21.15.42:0/2355512604 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fec1c03ac80 0x7fec1c03d190 unknown :-1 s=CLOSED pgs=1140 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:09.770 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:09.761+0000 7fec3b298700 1 --2- 172.21.15.42:0/2355512604 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fec3407bdf0 0x7fec34138230 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:09.770 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:09.761+0000 7fec3b298700 1 -- 172.21.15.42:0/2355512604 shutdown_connections 2019-12-05T06:30:09.770 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:09.761+0000 7fec3b298700 1 -- 172.21.15.42:0/2355512604 wait complete. 2019-12-05T06:30:09.771 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:09.761+0000 7fec3b298700 1 -- 172.21.15.42:0/2355512604 >> 172.21.15.42:0/2355512604 conn(0x7fec340a9640 msgr2=0x7fec34059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:30:09.778 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:30:11.779 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:30:11.895 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:11.885+0000 7f05252b2700 1 Processor -- start 2019-12-05T06:30:11.895 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:11.885+0000 7f05252b2700 1 -- start start 2019-12-05T06:30:11.895 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:11.885+0000 7f05252b2700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f052008df80 0x7f052009f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:30:11.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:11.885+0000 7f05252b2700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f052009fc60 con 0x7f05200a7740 2019-12-05T06:30:11.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:11.885+0000 7f05252b2700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05200a0710 con 0x7f052009f830 2019-12-05T06:30:11.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:11.885+0000 7f05252b2700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05200a0b00 con 0x7f052008df80 2019-12-05T06:30:11.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:11.885+0000 7f051effd700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f052008df80 0x7f052009f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:30:11.897 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:11.885+0000 7f051effd700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f052008df80 0x7f052009f2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:56370/0 (socket says 172.21.15.42:56370) 2019-12-05T06:30:11.897 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:11.885+0000 7f051effd700 1 -- 172.21.15.42:0/2652778708 learned_addr learned my addr 172.21.15.42:0/2652778708 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:30:11.897 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:11.885+0000 7f051effd700 1 -- 172.21.15.42:0/2652778708 >> v1:172.21.15.42:6789/0 conn(0x7f052009f830 legacy=0x7f052014efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:11.897 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:11.885+0000 7f051effd700 1 -- 172.21.15.42:0/2652778708 >> v1:172.21.15.40:6789/0 conn(0x7f05200a7740 legacy=0x7f052007e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:11.897 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:11.885+0000 7f051e7fc700 1 -- 172.21.15.42:0/2652778708 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2751682971 0 0) 0x7f05200a0710 con 0x7f052009f830 2019-12-05T06:30:11.897 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:11.885+0000 7f051effd700 1 -- 172.21.15.42:0/2652778708 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f052007ef20 con 0x7f052008df80 2019-12-05T06:30:11.898 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:11.885+0000 7f051effd700 1 --2- 172.21.15.42:0/2652778708 >> v2:172.21.15.42:3300/0 conn(0x7f052008df80 0x7f052009f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f0510008790 tx=0x7f051000fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:11.898 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:11.885+0000 7f051effd700 1 --2- 172.21.15.42:0/2652778708 >> v2:172.21.15.42:3300/0 conn(0x7f052008df80 0x7f052009f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f0510008790 tx=0x7f051000fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:11.898 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:11.885+0000 7f051effd700 1 --2- 172.21.15.42:0/2652778708 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f052008df80 0x7f052009f2e0 secure :-1 s=READY pgs=2720 cs=0 l=1 rx=0x7f0510008790 tx=0x7f051000fcb0).ready entity=mon.1 client_cookie=cb2b6770707dd8e9 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:30:11.898 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:11.885+0000 7f051effd700 1 --2- 172.21.15.42:0/2652778708 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f052008df80 0x7f052009f2e0 secure :-1 s=THROTTLE_DONE pgs=2720 cs=0 l=1 rx=0x7f0510008790 tx=0x7f051000fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:11.898 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:11.885+0000 7f051effd700 1 --2- 172.21.15.42:0/2652778708 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f052008df80 0x7f052009f2e0 secure :-1 s=THROTTLE_DONE pgs=2720 cs=0 l=1 rx=0x7f0510008790 tx=0x7f051000fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:11.899 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:11.885+0000 7f051e7fc700 1 -- 172.21.15.42:0/2652778708 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0510011520 con 0x7f052008df80 2019-12-05T06:30:11.899 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:11.885+0000 7f051effd700 1 --2- 172.21.15.42:0/2652778708 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f052008df80 0x7f052009f2e0 secure :-1 s=THROTTLE_DONE pgs=2720 cs=0 l=1 rx=0x7f0510008790 tx=0x7f051000fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:11.899 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:11.885+0000 7f051e7fc700 1 -- 172.21.15.42:0/2652778708 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0510008ca0 con 0x7f052008df80 2019-12-05T06:30:11.899 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:11.885+0000 7f051e7fc700 1 -- 172.21.15.42:0/2652778708 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0510011b00 con 0x7f052008df80 2019-12-05T06:30:11.899 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:11.885+0000 7f05252b2700 1 -- 172.21.15.42:0/2652778708 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f052008df80 msgr2=0x7f052009f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:11.900 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:11.885+0000 7f05252b2700 1 --2- 172.21.15.42:0/2652778708 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f052008df80 0x7f052009f2e0 secure :-1 s=READY pgs=2720 cs=0 l=1 rx=0x7f0510008790 tx=0x7f051000fcb0).stop 2019-12-05T06:30:11.900 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:11.889+0000 7f05252b2700 1 -- 172.21.15.42:0/2652778708 shutdown_connections 2019-12-05T06:30:11.900 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:11.889+0000 7f05252b2700 1 --2- 172.21.15.42:0/2652778708 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f052008df80 0x7f052009f2e0 unknown :-1 s=CLOSED pgs=2720 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:11.900 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:11.889+0000 7f05252b2700 1 -- 172.21.15.42:0/2652778708 shutdown_connections 2019-12-05T06:30:11.901 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:11.889+0000 7f05252b2700 1 -- 172.21.15.42:0/2652778708 wait complete. 2019-12-05T06:30:11.901 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:11.889+0000 7f05252b2700 1 -- 172.21.15.42:0/2652778708 >> 172.21.15.42:0/2652778708 conn(0x7f05200aabe0 msgr2=0x7f05200a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:30:11.901 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:11.889+0000 7f05252b2700 1 Processor -- start 2019-12-05T06:30:11.901 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:11.889+0000 7f05252b2700 1 -- start start 2019-12-05T06:30:11.905 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:11.889+0000 7f05252b2700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0520119d70 0x7f052008f160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:30:11.905 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:11.889+0000 7f05252b2700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0520116200 0x7f05201328e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:30:11.905 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:11.889+0000 7f05252b2700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f0520116630 0x7f0520132e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:30:11.905 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:11.889+0000 7f05252b2700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0520116b70 con 0x7f0520116630 2019-12-05T06:30:11.905 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:11.889+0000 7f05252b2700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0520116e10 con 0x7f0520116200 2019-12-05T06:30:11.906 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:11.889+0000 7f05252b2700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0520133380 con 0x7f0520119d70 2019-12-05T06:30:11.906 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:11.889+0000 7f051f7fe700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0520119d70 0x7f052008f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:30:11.906 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:11.889+0000 7f051effd700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0520116200 0x7f05201328e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:30:11.906 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:11.889+0000 7f051effd700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0520116200 0x7f05201328e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:56374/0 (socket says 172.21.15.42:56374) 2019-12-05T06:30:11.906 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:11.889+0000 7f051effd700 1 -- 172.21.15.42:0/2273767344 learned_addr learned my addr 172.21.15.42:0/2273767344 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:30:11.906 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:11.889+0000 7f051ffff700 1 --2- 172.21.15.42:0/2273767344 >> v2:172.21.15.40:3300/0 conn(0x7f0520116630 0x7f0520132e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:30:11.907 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:11.893+0000 7f051f7fe700 1 -- 172.21.15.42:0/2273767344 >> v2:172.21.15.42:3300/0 conn(0x7f0520116200 msgr2=0x7f05201328e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:11.907 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:11.893+0000 7f051f7fe700 1 --2- 172.21.15.42:0/2273767344 >> v2:172.21.15.42:3300/0 conn(0x7f0520116200 0x7f05201328e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:11.907 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:11.893+0000 7f051f7fe700 1 -- 172.21.15.42:0/2273767344 >> v2:172.21.15.40:3300/0 conn(0x7f0520116630 msgr2=0x7f0520132e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:11.907 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:11.893+0000 7f051f7fe700 1 --2- 172.21.15.42:0/2273767344 >> v2:172.21.15.40:3300/0 conn(0x7f0520116630 0x7f0520132e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:11.907 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:11.893+0000 7f051f7fe700 1 -- 172.21.15.42:0/2273767344 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05201222b0 con 0x7f0520119d70 2019-12-05T06:30:11.908 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:11.893+0000 7f051f7fe700 1 --2- 172.21.15.42:0/2273767344 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0520119d70 0x7f052008f160 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f0520142160 tx=0x7f05080109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:11.908 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:11.893+0000 7f051f7fe700 1 --2- 172.21.15.42:0/2273767344 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0520119d70 0x7f052008f160 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f0520142160 tx=0x7f05080109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:11.908 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:11.893+0000 7f051f7fe700 1 --2- 172.21.15.42:0/2273767344 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0520119d70 0x7f052008f160 secure :-1 s=READY pgs=598 cs=0 l=1 rx=0x7f0520142160 tx=0x7f05080109a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:30:11.908 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:11.893+0000 7f051f7fe700 1 --2- 172.21.15.42:0/2273767344 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0520119d70 0x7f052008f160 secure :-1 s=THROTTLE_DONE pgs=598 cs=0 l=1 rx=0x7f0520142160 tx=0x7f05080109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:11.908 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:11.893+0000 7f051f7fe700 1 --2- 172.21.15.42:0/2273767344 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0520119d70 0x7f052008f160 secure :-1 s=THROTTLE_DONE pgs=598 cs=0 l=1 rx=0x7f0520142160 tx=0x7f05080109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:11.908 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:11.893+0000 7f051f7fe700 1 --2- 172.21.15.42:0/2273767344 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0520119d70 0x7f052008f160 secure :-1 s=THROTTLE_DONE pgs=598 cs=0 l=1 rx=0x7f0520142160 tx=0x7f05080109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:11.909 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:11.893+0000 7f0517fff700 1 -- 172.21.15.42:0/2273767344 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f050800cb90 con 0x7f0520119d70 2019-12-05T06:30:11.909 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:11.893+0000 7f0517fff700 1 -- 172.21.15.42:0/2273767344 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f050800c7b0 con 0x7f0520119d70 2019-12-05T06:30:11.909 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:11.893+0000 7f05252b2700 1 -- 172.21.15.42:0/2273767344 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05201225e0 con 0x7f0520119d70 2019-12-05T06:30:11.909 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:11.893+0000 7f0517fff700 1 -- 172.21.15.42:0/2273767344 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0508011120 con 0x7f0520119d70 2019-12-05T06:30:11.909 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:11.893+0000 7f05252b2700 1 -- 172.21.15.42:0/2273767344 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0520122b10 con 0x7f0520119d70 2019-12-05T06:30:11.910 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:11.893+0000 7f051f7fe700 1 --2- 172.21.15.42:0/2273767344 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0520119d70 0x7f052008f160 secure :-1 s=THROTTLE_DONE pgs=598 cs=0 l=1 rx=0x7f0520142160 tx=0x7f05080109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:11.910 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:11.893+0000 7f0517fff700 1 -- 172.21.15.42:0/2273767344 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f0508007430 con 0x7f0520119d70 2019-12-05T06:30:11.910 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:11.893+0000 7f051f7fe700 1 --2- 172.21.15.42:0/2273767344 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0520119d70 0x7f052008f160 secure :-1 s=THROTTLE_DONE pgs=598 cs=0 l=1 rx=0x7f0520142160 tx=0x7f05080109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:11.910 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:11.893+0000 7f0517fff700 1 --2- 172.21.15.42:0/2273767344 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f050403ae60 0x7f050403d370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:30:11.910 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:11.893+0000 7f0517fff700 1 -- 172.21.15.42:0/2273767344 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 11019+0+0 (secure 0 0 0) 0x7f050804b1a0 con 0x7f0520119d70 2019-12-05T06:30:11.910 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:11.893+0000 7f051effd700 1 --2- 172.21.15.42:0/2273767344 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f050403ae60 0x7f050403d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:30:11.911 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:11.893+0000 7f051effd700 1 --2- 172.21.15.42:0/2273767344 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f050403ae60 0x7f050403d370 crc :-1 s=READY pgs=1141 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:30:11.911 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:11.901+0000 7f05252b2700 1 -- 172.21.15.42:0/2273767344 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f052005d110 con 0x7f0520119d70 2019-12-05T06:30:11.914 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:11.909+0000 7f051f7fe700 1 --2- 172.21.15.42:0/2273767344 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0520119d70 0x7f052008f160 secure :-1 s=THROTTLE_DONE pgs=598 cs=0 l=1 rx=0x7f0520142160 tx=0x7f05080109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:11.915 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:11.909+0000 7f0517fff700 1 -- 172.21.15.42:0/2273767344 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f05080321e0 con 0x7f0520119d70 2019-12-05T06:30:12.081 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:12.077+0000 7f05252b2700 1 -- 172.21.15.42:0/2273767344 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f052005b8c0 con 0x7f0520119d70 2019-12-05T06:30:12.082 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:12.077+0000 7f051f7fe700 1 --2- 172.21.15.42:0/2273767344 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0520119d70 0x7f052008f160 secure :-1 s=THROTTLE_DONE pgs=598 cs=0 l=1 rx=0x7f0520142160 tx=0x7f05080109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:12.082 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:12.077+0000 7f0517fff700 1 -- 172.21.15.42:0/2273767344 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 323 v323) v1 ==== 95+0+2117 (secure 0 0 0) 0x7f0508032440 con 0x7f0520119d70 2019-12-05T06:30:12.084 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:30:12.084 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":323,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":11265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/1985322937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1985322937},{"type":"v1","addr":"172.21.15.42:6835","nonce":1985322937}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":323}],"filesystems":[{"mdsmap":{"epoch":322,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:28:29.970815+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":107,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11142},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11142":{"gid":11142,"name":"b","rank":0,"incarnation":319,"state":"up:active","state_seq":28,"addr":"172.21.15.40:6833/3155851600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3155851600},{"type":"v1","addr":"172.21.15.40:6833","nonce":3155851600}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:30:12.084 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 323 2019-12-05T06:30:12.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:12.081+0000 7f05252b2700 1 -- 172.21.15.42:0/2273767344 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f050403ae60 msgr2=0x7f050403d370 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:12.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:12.081+0000 7f05252b2700 1 --2- 172.21.15.42:0/2273767344 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f050403ae60 0x7f050403d370 crc :-1 s=READY pgs=1141 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:12.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:12.081+0000 7f05252b2700 1 -- 172.21.15.42:0/2273767344 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0520119d70 msgr2=0x7f052008f160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:12.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:12.081+0000 7f05252b2700 1 --2- 172.21.15.42:0/2273767344 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0520119d70 0x7f052008f160 secure :-1 s=READY pgs=598 cs=0 l=1 rx=0x7f0520142160 tx=0x7f05080109a0).stop 2019-12-05T06:30:12.088 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:12.081+0000 7f05252b2700 1 -- 172.21.15.42:0/2273767344 shutdown_connections 2019-12-05T06:30:12.088 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:12.081+0000 7f05252b2700 1 --2- 172.21.15.42:0/2273767344 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f050403ae60 0x7f050403d370 unknown :-1 s=CLOSED pgs=1141 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:12.088 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:12.081+0000 7f05252b2700 1 --2- 172.21.15.42:0/2273767344 >> v2:172.21.15.40:3300/0 conn(0x7f0520116630 0x7f0520132e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:12.088 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:12.081+0000 7f05252b2700 1 --2- 172.21.15.42:0/2273767344 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0520119d70 0x7f052008f160 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:12.088 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:12.081+0000 7f05252b2700 1 --2- 172.21.15.42:0/2273767344 >> v2:172.21.15.42:3300/0 conn(0x7f0520116200 0x7f05201328e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:12.088 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:12.081+0000 7f05252b2700 1 -- 172.21.15.42:0/2273767344 shutdown_connections 2019-12-05T06:30:12.089 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:12.081+0000 7f05252b2700 1 -- 172.21.15.42:0/2273767344 wait complete. 2019-12-05T06:30:12.089 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:12.081+0000 7f05252b2700 1 -- 172.21.15.42:0/2273767344 >> 172.21.15.42:0/2273767344 conn(0x7f05200a7f90 msgr2=0x7f052011da10 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:30:12.097 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:30:14.098 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:30:14.210 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:14.197+0000 7f55550de700 1 Processor -- start 2019-12-05T06:30:14.211 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:14.197+0000 7f55550de700 1 -- start start 2019-12-05T06:30:14.211 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:14.201+0000 7f55550de700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f55500a7740 0x7f555008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:30:14.211 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:14.201+0000 7f55550de700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f555007e720 0x7f555007eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:30:14.211 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:14.201+0000 7f55550de700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55500a0710 con 0x7f555009f2e0 2019-12-05T06:30:14.211 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:14.201+0000 7f55550de700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55500a0b00 con 0x7f55500a7740 2019-12-05T06:30:14.212 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:14.201+0000 7f55550de700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f555008e5b0 con 0x7f555007e720 2019-12-05T06:30:14.212 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:14.201+0000 7f554ed9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f555007e720 0x7f555007eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:30:14.212 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:14.201+0000 7f554ed9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f555007e720 0x7f555007eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:56380/0 (socket says 172.21.15.42:56380) 2019-12-05T06:30:14.212 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:14.201+0000 7f554ed9d700 1 -- 172.21.15.42:0/3936121901 learned_addr learned my addr 172.21.15.42:0/3936121901 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:30:14.212 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:14.201+0000 7f554f59e700 1 --2- 172.21.15.42:0/3936121901 >> v2:172.21.15.40:3300/0 conn(0x7f55500a7740 0x7f555008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:30:14.213 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:14.201+0000 7f554ed9d700 1 -- 172.21.15.42:0/3936121901 >> v2:172.21.15.40:3300/0 conn(0x7f55500a7740 msgr2=0x7f555008df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:30:14.213 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:14.201+0000 7f554ed9d700 1 --2- 172.21.15.42:0/3936121901 >> v2:172.21.15.40:3300/0 conn(0x7f55500a7740 0x7f555008df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:30:14.213 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:14.201+0000 7f554ed9d700 1 -- 172.21.15.42:0/3936121901 >> v1:172.21.15.42:6789/0 conn(0x7f555009f2e0 legacy=0x7f555009f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:14.213 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:14.201+0000 7f554e59c700 1 -- 172.21.15.42:0/3936121901 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1343532951 0 0) 0x7f55500a0710 con 0x7f555009f2e0 2019-12-05T06:30:14.213 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:14.201+0000 7f554ed9d700 1 -- 172.21.15.42:0/3936121901 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f555007f0a0 con 0x7f555007e720 2019-12-05T06:30:14.213 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:14.201+0000 7f554ed9d700 1 --2- 172.21.15.42:0/3936121901 >> v2:172.21.15.42:3300/0 conn(0x7f555007e720 0x7f555007eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f5544002f30 tx=0x7f554400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:14.213 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:14.201+0000 7f554ed9d700 1 --2- 172.21.15.42:0/3936121901 >> v2:172.21.15.42:3300/0 conn(0x7f555007e720 0x7f555007eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f5544002f30 tx=0x7f554400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:14.214 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:14.201+0000 7f554ed9d700 1 --2- 172.21.15.42:0/3936121901 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f555007e720 0x7f555007eb50 secure :-1 s=READY pgs=2722 cs=0 l=1 rx=0x7f5544002f30 tx=0x7f554400fd30).ready entity=mon.1 client_cookie=1350348a2012235a server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:30:14.214 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:14.201+0000 7f554ed9d700 1 --2- 172.21.15.42:0/3936121901 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f555007e720 0x7f555007eb50 secure :-1 s=THROTTLE_DONE pgs=2722 cs=0 l=1 rx=0x7f5544002f30 tx=0x7f554400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:14.214 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:14.201+0000 7f554e59c700 1 -- 172.21.15.42:0/3936121901 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5544011370 con 0x7f555007e720 2019-12-05T06:30:14.214 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:14.201+0000 7f554ed9d700 1 --2- 172.21.15.42:0/3936121901 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f555007e720 0x7f555007eb50 secure :-1 s=THROTTLE_DONE pgs=2722 cs=0 l=1 rx=0x7f5544002f30 tx=0x7f554400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:14.214 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:14.201+0000 7f554ed9d700 1 --2- 172.21.15.42:0/3936121901 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f555007e720 0x7f555007eb50 secure :-1 s=THROTTLE_DONE pgs=2722 cs=0 l=1 rx=0x7f5544002f30 tx=0x7f554400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:14.214 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:14.201+0000 7f554e59c700 1 -- 172.21.15.42:0/3936121901 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5544008bb0 con 0x7f555007e720 2019-12-05T06:30:14.214 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:14.201+0000 7f554e59c700 1 -- 172.21.15.42:0/3936121901 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5544011950 con 0x7f555007e720 2019-12-05T06:30:14.215 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:14.201+0000 7f55550de700 1 -- 172.21.15.42:0/3936121901 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f555007e720 msgr2=0x7f555007eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:14.215 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:14.201+0000 7f55550de700 1 --2- 172.21.15.42:0/3936121901 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f555007e720 0x7f555007eb50 secure :-1 s=READY pgs=2722 cs=0 l=1 rx=0x7f5544002f30 tx=0x7f554400fd30).stop 2019-12-05T06:30:14.215 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:14.201+0000 7f55550de700 1 -- 172.21.15.42:0/3936121901 shutdown_connections 2019-12-05T06:30:14.215 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:14.201+0000 7f55550de700 1 --2- 172.21.15.42:0/3936121901 >> v2:172.21.15.40:3300/0 conn(0x7f55500a7740 0x7f555008df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:30:14.215 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:14.201+0000 7f55550de700 1 --2- 172.21.15.42:0/3936121901 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f555007e720 0x7f555007eb50 unknown :-1 s=CLOSED pgs=2722 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:14.215 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:14.201+0000 7f55550de700 1 -- 172.21.15.42:0/3936121901 shutdown_connections 2019-12-05T06:30:14.218 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:14.201+0000 7f55550de700 1 -- 172.21.15.42:0/3936121901 wait complete. 2019-12-05T06:30:14.219 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:14.201+0000 7f55550de700 1 -- 172.21.15.42:0/3936121901 >> 172.21.15.42:0/3936121901 conn(0x7f55500aabe0 msgr2=0x7f55500a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:30:14.225 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:14.205+0000 7f55550de700 1 Processor -- start 2019-12-05T06:30:14.225 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:14.205+0000 7f55550de700 1 -- start start 2019-12-05T06:30:14.225 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:14.209+0000 7f55550de700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f555011b3c0 0x7f5550131e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:30:14.225 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:14.209+0000 7f55550de700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5550117e60 0x7f555006e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:30:14.225 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:14.209+0000 7f55550de700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f555011a670 con 0x7f55500a1510 2019-12-05T06:30:14.226 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:14.209+0000 7f55550de700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f555011abb0 con 0x7f555011b3c0 2019-12-05T06:30:14.226 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:14.209+0000 7f55550de700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f555011a8a0 con 0x7f5550117e60 2019-12-05T06:30:14.226 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:14.209+0000 7f554ed9d700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f555011b3c0 0x7f5550131e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:30:14.226 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:14.209+0000 7f55548dd700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5550117e60 0x7f555006e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:30:14.227 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:14.209+0000 7f554f59e700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f55500a1510 0x7f5550131680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:37632/0 (socket says 172.21.15.42:37632) 2019-12-05T06:30:14.227 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:14.209+0000 7f554ed9d700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f555011b3c0 0x7f5550131e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.42:44204/0 (socket says 172.21.15.42:44204) 2019-12-05T06:30:14.227 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:14.209+0000 7f554f59e700 1 -- 172.21.15.42:0/109882315 learned_addr learned my addr 172.21.15.42:0/109882315 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:30:14.227 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:14.209+0000 7f554ed9d700 1 -- 172.21.15.42:0/109882315 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5550117e60 msgr2=0x7f555006e2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:14.227 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:14.209+0000 7f554ed9d700 1 --2- 172.21.15.42:0/109882315 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5550117e60 0x7f555006e2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:14.228 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:14.209+0000 7f554ed9d700 1 -- 172.21.15.42:0/109882315 >> v1:172.21.15.40:6789/0 conn(0x7f55500a1510 legacy=0x7f5550131680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:14.229 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:14.209+0000 7f554ed9d700 1 -- 172.21.15.42:0/109882315 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f555006e7f0 con 0x7f555011b3c0 2019-12-05T06:30:14.230 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:14.209+0000 7f55548dd700 1 --2- 172.21.15.42:0/109882315 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5550117e60 0x7f555006e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:30:14.230 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:14.209+0000 7f554ed9d700 1 --2- 172.21.15.42:0/109882315 >> v2:172.21.15.40:3300/0 conn(0x7f555011b3c0 0x7f5550131e80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f5544002f30 tx=0x7f5544008e60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:14.230 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:14.209+0000 7f554ed9d700 1 --2- 172.21.15.42:0/109882315 >> v2:172.21.15.40:3300/0 conn(0x7f555011b3c0 0x7f5550131e80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f5544002f30 tx=0x7f5544008e60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:14.231 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:14.209+0000 7f554ed9d700 1 --2- 172.21.15.42:0/109882315 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f555011b3c0 0x7f5550131e80 secure :-1 s=READY pgs=2041 cs=0 l=1 rx=0x7f5544002f30 tx=0x7f5544008e60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:30:14.231 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:14.209+0000 7f554ed9d700 1 --2- 172.21.15.42:0/109882315 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f555011b3c0 0x7f5550131e80 secure :-1 s=THROTTLE_DONE pgs=2041 cs=0 l=1 rx=0x7f5544002f30 tx=0x7f5544008e60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:14.231 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:14.209+0000 7f5537fff700 1 -- 172.21.15.42:0/109882315 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5544007150 con 0x7f555011b3c0 2019-12-05T06:30:14.231 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:14.209+0000 7f554ed9d700 1 --2- 172.21.15.42:0/109882315 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f555011b3c0 0x7f5550131e80 secure :-1 s=THROTTLE_DONE pgs=2041 cs=0 l=1 rx=0x7f5544002f30 tx=0x7f5544008e60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:14.232 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:14.209+0000 7f554ed9d700 1 --2- 172.21.15.42:0/109882315 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f555011b3c0 0x7f5550131e80 secure :-1 s=THROTTLE_DONE pgs=2041 cs=0 l=1 rx=0x7f5544002f30 tx=0x7f5544008e60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:14.232 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:14.209+0000 7f5537fff700 1 -- 172.21.15.42:0/109882315 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5544011570 con 0x7f555011b3c0 2019-12-05T06:30:14.232 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:14.209+0000 7f5537fff700 1 -- 172.21.15.42:0/109882315 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5544005730 con 0x7f555011b3c0 2019-12-05T06:30:14.233 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:14.209+0000 7f55550de700 1 -- 172.21.15.42:0/109882315 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f555006eae0 con 0x7f555011b3c0 2019-12-05T06:30:14.233 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:14.209+0000 7f55550de700 1 -- 172.21.15.42:0/109882315 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f555006f010 con 0x7f555011b3c0 2019-12-05T06:30:14.233 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:14.209+0000 7f554ed9d700 1 --2- 172.21.15.42:0/109882315 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f555011b3c0 0x7f5550131e80 secure :-1 s=THROTTLE_DONE pgs=2041 cs=0 l=1 rx=0x7f5544002f30 tx=0x7f5544008e60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:14.233 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:14.213+0000 7f5537fff700 1 -- 172.21.15.42:0/109882315 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f5544005a10 con 0x7f555011b3c0 2019-12-05T06:30:14.233 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:14.213+0000 7f554ed9d700 1 --2- 172.21.15.42:0/109882315 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f555011b3c0 0x7f5550131e80 secure :-1 s=THROTTLE_DONE pgs=2041 cs=0 l=1 rx=0x7f5544002f30 tx=0x7f5544008e60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:14.234 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:14.213+0000 7f5537fff700 1 --2- 172.21.15.42:0/109882315 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f553003adb0 0x7f553003d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:30:14.234 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:14.213+0000 7f5537fff700 1 -- 172.21.15.42:0/109882315 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 11019+0+0 (secure 0 0 0) 0x7f554408f4a0 con 0x7f555011b3c0 2019-12-05T06:30:14.234 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:14.213+0000 7f554f59e700 1 --2- 172.21.15.42:0/109882315 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f553003adb0 0x7f553003d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:30:14.235 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:14.213+0000 7f554f59e700 1 --2- 172.21.15.42:0/109882315 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f553003adb0 0x7f553003d2c0 crc :-1 s=READY pgs=1142 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:30:14.235 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:14.213+0000 7f55550de700 1 -- 172.21.15.42:0/109882315 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f555005d110 con 0x7f555011b3c0 2019-12-05T06:30:14.235 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:14.225+0000 7f554ed9d700 1 --2- 172.21.15.42:0/109882315 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f555011b3c0 0x7f5550131e80 secure :-1 s=THROTTLE_DONE pgs=2041 cs=0 l=1 rx=0x7f5544002f30 tx=0x7f5544008e60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:14.236 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:14.225+0000 7f5537fff700 1 -- 172.21.15.42:0/109882315 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f5544076500 con 0x7f555011b3c0 2019-12-05T06:30:14.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:14.397+0000 7f55550de700 1 -- 172.21.15.42:0/109882315 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f555005b8c0 con 0x7f555011b3c0 2019-12-05T06:30:14.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:14.397+0000 7f554ed9d700 1 --2- 172.21.15.42:0/109882315 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f555011b3c0 0x7f5550131e80 secure :-1 s=THROTTLE_DONE pgs=2041 cs=0 l=1 rx=0x7f5544002f30 tx=0x7f5544008e60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:14.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:14.397+0000 7f5537fff700 1 -- 172.21.15.42:0/109882315 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 323 v323) v1 ==== 95+0+2117 (secure 0 0 0) 0x7f5544076760 con 0x7f555011b3c0 2019-12-05T06:30:14.404 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 323 2019-12-05T06:30:14.405 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:30:14.405 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":323,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":11265,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/1985322937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1985322937},{"type":"v1","addr":"172.21.15.42:6835","nonce":1985322937}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":323}],"filesystems":[{"mdsmap":{"epoch":322,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:28:29.970815+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":107,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11142},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11142":{"gid":11142,"name":"b","rank":0,"incarnation":319,"state":"up:active","state_seq":28,"addr":"172.21.15.40:6833/3155851600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3155851600},{"type":"v1","addr":"172.21.15.40:6833","nonce":3155851600}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:30:14.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:14.401+0000 7f55550de700 1 -- 172.21.15.42:0/109882315 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f553003adb0 msgr2=0x7f553003d2c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:14.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:14.401+0000 7f55550de700 1 --2- 172.21.15.42:0/109882315 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f553003adb0 0x7f553003d2c0 crc :-1 s=READY pgs=1142 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:14.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:14.401+0000 7f55550de700 1 -- 172.21.15.42:0/109882315 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f555011b3c0 msgr2=0x7f5550131e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:14.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:14.401+0000 7f55550de700 1 --2- 172.21.15.42:0/109882315 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f555011b3c0 0x7f5550131e80 secure :-1 s=READY pgs=2041 cs=0 l=1 rx=0x7f5544002f30 tx=0x7f5544008e60).stop 2019-12-05T06:30:14.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:14.401+0000 7f55550de700 1 -- 172.21.15.42:0/109882315 shutdown_connections 2019-12-05T06:30:14.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:14.401+0000 7f55550de700 1 --2- 172.21.15.42:0/109882315 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f553003adb0 0x7f553003d2c0 unknown :-1 s=CLOSED pgs=1142 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:14.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:14.401+0000 7f55550de700 1 --2- 172.21.15.42:0/109882315 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5550117e60 0x7f555006e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:14.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:14.401+0000 7f55550de700 1 --2- 172.21.15.42:0/109882315 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f555011b3c0 0x7f5550131e80 unknown :-1 s=CLOSED pgs=2041 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:14.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:14.401+0000 7f55550de700 1 -- 172.21.15.42:0/109882315 shutdown_connections 2019-12-05T06:30:14.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:14.401+0000 7f55550de700 1 -- 172.21.15.42:0/109882315 wait complete. 2019-12-05T06:30:14.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:14.401+0000 7f55550de700 1 -- 172.21.15.42:0/109882315 >> 172.21.15.42:0/109882315 conn(0x7f555008fda0 msgr2=0x7f55500a9790 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:30:14.417 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:30:16.419 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:30:16.515 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.501+0000 7f6a0131c700 1 Processor -- start 2019-12-05T06:30:16.516 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.501+0000 7f6a0131c700 1 -- start start 2019-12-05T06:30:16.516 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.501+0000 7f6a0131c700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f69fc0a7740 0x7f69fc08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:30:16.516 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.501+0000 7f6a0131c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f69fc09fae0 0x7f69fc07eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:30:16.516 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.501+0000 7f6a0131c700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69fc0a0710 con 0x7f69fc07e720 2019-12-05T06:30:16.517 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.501+0000 7f6a0131c700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69fc0a0b00 con 0x7f69fc09fae0 2019-12-05T06:30:16.517 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.501+0000 7f6a0131c700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69fc08e5b0 con 0x7f69fc0a7740 2019-12-05T06:30:16.517 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.501+0000 7f69fbfff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f69fc09fae0 0x7f69fc07eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:30:16.517 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.501+0000 7f69fbfff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f69fc09fae0 0x7f69fc07eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:56424/0 (socket says 172.21.15.42:56424) 2019-12-05T06:30:16.517 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.501+0000 7f69fbfff700 1 -- 172.21.15.42:0/2877751031 learned_addr learned my addr 172.21.15.42:0/2877751031 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:30:16.518 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.501+0000 7f69fb7fe700 1 --2- 172.21.15.42:0/2877751031 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f69fc0a7740 0x7f69fc08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:30:16.518 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.501+0000 7f69fbfff700 1 -- 172.21.15.42:0/2877751031 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f69fc0a7740 msgr2=0x7f69fc08df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:30:16.518 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.501+0000 7f69fbfff700 1 --2- 172.21.15.42:0/2877751031 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f69fc0a7740 0x7f69fc08df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:30:16.518 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.501+0000 7f69fbfff700 1 -- 172.21.15.42:0/2877751031 >> v1:172.21.15.40:6789/0 conn(0x7f69fc07e720 legacy=0x7f69fc09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:16.519 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.501+0000 7f69fbfff700 1 -- 172.21.15.42:0/2877751031 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69fc07f0a0 con 0x7f69fc09fae0 2019-12-05T06:30:16.519 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.501+0000 7f69fbfff700 1 --2- 172.21.15.42:0/2877751031 >> v2:172.21.15.42:3300/0 conn(0x7f69fc09fae0 0x7f69fc07eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f69f4008b10 tx=0x7f69f40100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:16.519 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.501+0000 7f69fbfff700 1 --2- 172.21.15.42:0/2877751031 >> v2:172.21.15.42:3300/0 conn(0x7f69fc09fae0 0x7f69fc07eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f69f4008b10 tx=0x7f69f40100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:16.519 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.501+0000 7f69fbfff700 1 --2- 172.21.15.42:0/2877751031 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f69fc09fae0 0x7f69fc07eb50 secure :-1 s=READY pgs=2723 cs=0 l=1 rx=0x7f69f4008b10 tx=0x7f69f40100b0).ready entity=mon.1 client_cookie=8241513e13768713 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:30:16.519 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.501+0000 7f69fbfff700 1 --2- 172.21.15.42:0/2877751031 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f69fc09fae0 0x7f69fc07eb50 secure :-1 s=THROTTLE_DONE pgs=2723 cs=0 l=1 rx=0x7f69f4008b10 tx=0x7f69f40100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:16.520 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.501+0000 7f69fbfff700 1 --2- 172.21.15.42:0/2877751031 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f69fc09fae0 0x7f69fc07eb50 secure :-1 s=THROTTLE_DONE pgs=2723 cs=0 l=1 rx=0x7f69f4008b10 tx=0x7f69f40100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:16.520 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.501+0000 7f69fbfff700 1 --2- 172.21.15.42:0/2877751031 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f69fc09fae0 0x7f69fc07eb50 secure :-1 s=THROTTLE_DONE pgs=2723 cs=0 l=1 rx=0x7f69f4008b10 tx=0x7f69f40100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:16.520 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.501+0000 7f69fa7fc700 1 -- 172.21.15.42:0/2877751031 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f69f4010da0 con 0x7f69fc09fae0 2019-12-05T06:30:16.520 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.501+0000 7f69fa7fc700 1 -- 172.21.15.42:0/2877751031 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69f40109c0 con 0x7f69fc09fae0 2019-12-05T06:30:16.521 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.501+0000 7f69fa7fc700 1 -- 172.21.15.42:0/2877751031 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f69f40114a0 con 0x7f69fc09fae0 2019-12-05T06:30:16.521 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.501+0000 7f6a0131c700 1 -- 172.21.15.42:0/2877751031 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f69fc09fae0 msgr2=0x7f69fc07eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:16.521 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.501+0000 7f6a0131c700 1 --2- 172.21.15.42:0/2877751031 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f69fc09fae0 0x7f69fc07eb50 secure :-1 s=READY pgs=2723 cs=0 l=1 rx=0x7f69f4008b10 tx=0x7f69f40100b0).stop 2019-12-05T06:30:16.521 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.501+0000 7f6a0131c700 1 -- 172.21.15.42:0/2877751031 shutdown_connections 2019-12-05T06:30:16.522 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.501+0000 7f6a0131c700 1 --2- 172.21.15.42:0/2877751031 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f69fc09fae0 0x7f69fc07eb50 unknown :-1 s=CLOSED pgs=2723 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:16.522 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.501+0000 7f6a0131c700 1 --2- 172.21.15.42:0/2877751031 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f69fc0a7740 0x7f69fc08df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:30:16.522 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.501+0000 7f6a0131c700 1 -- 172.21.15.42:0/2877751031 shutdown_connections 2019-12-05T06:30:16.522 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.501+0000 7f6a0131c700 1 -- 172.21.15.42:0/2877751031 wait complete. 2019-12-05T06:30:16.522 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.501+0000 7f6a0131c700 1 -- 172.21.15.42:0/2877751031 >> 172.21.15.42:0/2877751031 conn(0x7f69fc0aabe0 msgr2=0x7f69fc0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:30:16.523 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.505+0000 7f6a0131c700 1 Processor -- start 2019-12-05T06:30:16.523 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.505+0000 7f6a0131c700 1 -- start start 2019-12-05T06:30:16.523 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.505+0000 7f6a0131c700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f69fc11a150 0x7f69fc112090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:30:16.524 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.505+0000 7f6a0131c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f69fc114710 0x7f69fc14e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:30:16.524 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.505+0000 7f6a0131c700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69fc14ecd0 con 0x7f69fc14e8a0 2019-12-05T06:30:16.525 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.505+0000 7f6a0131c700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69fc112730 con 0x7f69fc114710 2019-12-05T06:30:16.525 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.505+0000 7f6a0131c700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69fc1129d0 con 0x7f69fc11a150 2019-12-05T06:30:16.525 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.505+0000 7f69faffd700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f69fc114710 0x7f69fc14e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:30:16.525 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.505+0000 7f69fb7fe700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f69fc11a150 0x7f69fc112090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:30:16.525 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.505+0000 7f69faffd700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f69fc114710 0x7f69fc14e350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:56428/0 (socket says 172.21.15.42:56428) 2019-12-05T06:30:16.526 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.505+0000 7f69faffd700 1 -- 172.21.15.42:0/3048977598 learned_addr learned my addr 172.21.15.42:0/3048977598 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:30:16.526 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.505+0000 7f69faffd700 1 -- 172.21.15.42:0/3048977598 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f69fc11a150 msgr2=0x7f69fc112090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:16.526 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.505+0000 7f69faffd700 1 --2- 172.21.15.42:0/3048977598 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f69fc11a150 0x7f69fc112090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:16.526 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.505+0000 7f69faffd700 1 -- 172.21.15.42:0/3048977598 >> v1:172.21.15.40:6789/0 conn(0x7f69fc14e8a0 legacy=0x7f69fc08ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:16.527 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.505+0000 7f69faffd700 1 -- 172.21.15.42:0/3048977598 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69fc112c70 con 0x7f69fc114710 2019-12-05T06:30:16.527 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.505+0000 7f69faffd700 1 --2- 172.21.15.42:0/3048977598 >> v2:172.21.15.42:3300/0 conn(0x7f69fc114710 0x7f69fc14e350 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f69fc113450 tx=0x7f69f00109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:16.527 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.505+0000 7f69faffd700 1 --2- 172.21.15.42:0/3048977598 >> v2:172.21.15.42:3300/0 conn(0x7f69fc114710 0x7f69fc14e350 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f69fc113450 tx=0x7f69f00109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:16.527 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.505+0000 7f69faffd700 1 --2- 172.21.15.42:0/3048977598 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f69fc114710 0x7f69fc14e350 secure :-1 s=READY pgs=2724 cs=0 l=1 rx=0x7f69fc113450 tx=0x7f69f00109a0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:30:16.528 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.505+0000 7f69faffd700 1 --2- 172.21.15.42:0/3048977598 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f69fc114710 0x7f69fc14e350 secure :-1 s=THROTTLE_DONE pgs=2724 cs=0 l=1 rx=0x7f69fc113450 tx=0x7f69f00109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:16.528 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.505+0000 7f69e3fff700 1 -- 172.21.15.42:0/3048977598 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f69f000caa0 con 0x7f69fc114710 2019-12-05T06:30:16.528 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.505+0000 7f69faffd700 1 --2- 172.21.15.42:0/3048977598 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f69fc114710 0x7f69fc14e350 secure :-1 s=THROTTLE_DONE pgs=2724 cs=0 l=1 rx=0x7f69fc113450 tx=0x7f69f00109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:16.528 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.505+0000 7f69faffd700 1 --2- 172.21.15.42:0/3048977598 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f69fc114710 0x7f69fc14e350 secure :-1 s=THROTTLE_DONE pgs=2724 cs=0 l=1 rx=0x7f69fc113450 tx=0x7f69f00109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:16.528 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.505+0000 7f69e3fff700 1 -- 172.21.15.42:0/3048977598 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69f000c6c0 con 0x7f69fc114710 2019-12-05T06:30:16.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.505+0000 7f69e3fff700 1 -- 172.21.15.42:0/3048977598 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f69f0011160 con 0x7f69fc114710 2019-12-05T06:30:16.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.505+0000 7f6a0131c700 1 -- 172.21.15.42:0/3048977598 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69fc08f610 con 0x7f69fc114710 2019-12-05T06:30:16.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.505+0000 7f6a0131c700 1 -- 172.21.15.42:0/3048977598 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69fc12f5e0 con 0x7f69fc114710 2019-12-05T06:30:16.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.505+0000 7f69faffd700 1 --2- 172.21.15.42:0/3048977598 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f69fc114710 0x7f69fc14e350 secure :-1 s=THROTTLE_DONE pgs=2724 cs=0 l=1 rx=0x7f69fc113450 tx=0x7f69f00109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:16.530 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.509+0000 7f69e3fff700 1 -- 172.21.15.42:0/3048977598 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f69f0011470 con 0x7f69fc114710 2019-12-05T06:30:16.530 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.509+0000 7f69faffd700 1 --2- 172.21.15.42:0/3048977598 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f69fc114710 0x7f69fc14e350 secure :-1 s=THROTTLE_DONE pgs=2724 cs=0 l=1 rx=0x7f69fc113450 tx=0x7f69f00109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:16.530 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.509+0000 7f69e3fff700 1 --2- 172.21.15.42:0/3048977598 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f69dc03ae00 0x7f69dc03d310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:30:16.530 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.509+0000 7f69e3fff700 1 -- 172.21.15.42:0/3048977598 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 11105+0+0 (secure 0 0 0) 0x7f69fc14ecd0 con 0x7f69fc114710 2019-12-05T06:30:16.530 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.509+0000 7f69fb7fe700 1 --2- 172.21.15.42:0/3048977598 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f69dc03ae00 0x7f69dc03d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:30:16.531 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.509+0000 7f69fb7fe700 1 --2- 172.21.15.42:0/3048977598 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f69dc03ae00 0x7f69dc03d310 crc :-1 s=READY pgs=1143 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:30:16.532 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.517+0000 7f6a0131c700 1 -- 172.21.15.42:0/3048977598 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69fc05d110 con 0x7f69fc114710 2019-12-05T06:30:16.532 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.517+0000 7f69faffd700 1 --2- 172.21.15.42:0/3048977598 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f69fc114710 0x7f69fc14e350 secure :-1 s=THROTTLE_DONE pgs=2724 cs=0 l=1 rx=0x7f69fc113450 tx=0x7f69f00109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:16.532 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.517+0000 7f69e3fff700 1 -- 172.21.15.42:0/3048977598 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f69f00321f0 con 0x7f69fc114710 2019-12-05T06:30:16.680 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.677+0000 7f6a0131c700 1 -- 172.21.15.42:0/3048977598 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f69fc05b8c0 con 0x7f69fc114710 2019-12-05T06:30:16.681 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.677+0000 7f69faffd700 1 --2- 172.21.15.42:0/3048977598 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f69fc114710 0x7f69fc14e350 secure :-1 s=THROTTLE_DONE pgs=2724 cs=0 l=1 rx=0x7f69fc113450 tx=0x7f69f00109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:16.681 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.677+0000 7f69e3fff700 1 -- 172.21.15.42:0/3048977598 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 324 v324) v1 ==== 95+0+1778 (secure 0 0 0) 0x7f69f0032450 con 0x7f69fc114710 2019-12-05T06:30:16.681 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 324 2019-12-05T06:30:16.682 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:30:16.682 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":324,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":324,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:30:14.969128+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":108,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11265":{"gid":11265,"name":"a","rank":0,"incarnation":324,"state":"up:replay","state_seq":2,"addr":"172.21.15.42:6835/1985322937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1985322937},{"type":"v1","addr":"172.21.15.42:6835","nonce":1985322937}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:30:16.684 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.677+0000 7f6a0131c700 1 -- 172.21.15.42:0/3048977598 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f69dc03ae00 msgr2=0x7f69dc03d310 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:16.684 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.677+0000 7f6a0131c700 1 --2- 172.21.15.42:0/3048977598 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f69dc03ae00 0x7f69dc03d310 crc :-1 s=READY pgs=1143 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:16.684 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.677+0000 7f6a0131c700 1 -- 172.21.15.42:0/3048977598 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f69fc114710 msgr2=0x7f69fc14e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:16.685 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.677+0000 7f6a0131c700 1 --2- 172.21.15.42:0/3048977598 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f69fc114710 0x7f69fc14e350 secure :-1 s=READY pgs=2724 cs=0 l=1 rx=0x7f69fc113450 tx=0x7f69f00109a0).stop 2019-12-05T06:30:16.685 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.677+0000 7f6a0131c700 1 -- 172.21.15.42:0/3048977598 shutdown_connections 2019-12-05T06:30:16.685 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.677+0000 7f6a0131c700 1 --2- 172.21.15.42:0/3048977598 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f69dc03ae00 0x7f69dc03d310 unknown :-1 s=CLOSED pgs=1143 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:16.685 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.677+0000 7f6a0131c700 1 --2- 172.21.15.42:0/3048977598 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f69fc11a150 0x7f69fc112090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:16.685 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.677+0000 7f6a0131c700 1 --2- 172.21.15.42:0/3048977598 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f69fc114710 0x7f69fc14e350 unknown :-1 s=CLOSED pgs=2724 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:16.685 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.677+0000 7f6a0131c700 1 -- 172.21.15.42:0/3048977598 shutdown_connections 2019-12-05T06:30:16.685 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.677+0000 7f6a0131c700 1 -- 172.21.15.42:0/3048977598 wait complete. 2019-12-05T06:30:16.686 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.677+0000 7f6a0131c700 1 -- 172.21.15.42:0/3048977598 >> 172.21.15.42:0/3048977598 conn(0x7f69fc0a8040 msgr2=0x7f69fc059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:30:16.692 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b down, removed from mdsmap 2019-12-05T06:30:16.693 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2019-12-05T06:30:16.693 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:30:16.778 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.769+0000 7f3023d6f700 1 Processor -- start 2019-12-05T06:30:16.778 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.769+0000 7f3023d6f700 1 -- start start 2019-12-05T06:30:16.778 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.769+0000 7f3023d6f700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f301c08df80 0x7f301c09f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:30:16.778 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.769+0000 7f3023d6f700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f301c09fc60 con 0x7f301c0a7740 2019-12-05T06:30:16.779 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.769+0000 7f3023d6f700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f301c0a0710 con 0x7f301c09f830 2019-12-05T06:30:16.779 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.769+0000 7f3023d6f700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f301c0a0b00 con 0x7f301c08df80 2019-12-05T06:30:16.779 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.769+0000 7f3022b0d700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f301c09f830 0x7f301c14efe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:56476/0 (socket says 172.21.15.42:56476) 2019-12-05T06:30:16.779 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.769+0000 7f3022b0d700 1 -- 172.21.15.42:0/1055142853 learned_addr learned my addr 172.21.15.42:0/1055142853 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:30:16.779 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.769+0000 7f3021b0b700 1 --2- 172.21.15.42:0/1055142853 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f301c08df80 0x7f301c09f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:30:16.779 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.769+0000 7f302130a700 1 -- 172.21.15.42:0/1055142853 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2228373535 0 0) 0x7f301c0a0710 con 0x7f301c09f830 2019-12-05T06:30:16.779 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.769+0000 7f302130a700 1 -- 172.21.15.42:0/1055142853 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30040030b0 con 0x7f301c09f830 2019-12-05T06:30:16.780 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.769+0000 7f302130a700 1 -- 172.21.15.42:0/1055142853 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1360352305 0 0) 0x7f301c09fc60 con 0x7f301c0a7740 2019-12-05T06:30:16.780 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.769+0000 7f302130a700 1 -- 172.21.15.42:0/1055142853 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f301c0a0710 con 0x7f301c0a7740 2019-12-05T06:30:16.780 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.769+0000 7f302130a700 1 -- 172.21.15.42:0/1055142853 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 169990222 0 0) 0x7f30040030b0 con 0x7f301c09f830 2019-12-05T06:30:16.780 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.769+0000 7f302130a700 1 -- 172.21.15.42:0/1055142853 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f301c08df80 msgr2=0x7f301c09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:30:16.780 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.769+0000 7f302130a700 1 --2- 172.21.15.42:0/1055142853 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f301c08df80 0x7f301c09f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:30:16.780 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.769+0000 7f302130a700 1 -- 172.21.15.42:0/1055142853 >> v1:172.21.15.40:6789/0 conn(0x7f301c0a7740 legacy=0x7f301c07e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:16.780 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.769+0000 7f302130a700 1 -- 172.21.15.42:0/1055142853 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f301c07ef20 con 0x7f301c09f830 2019-12-05T06:30:16.781 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.769+0000 7f302130a700 1 -- 172.21.15.42:0/1055142853 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7f300c001750 con 0x7f301c09f830 2019-12-05T06:30:16.781 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.769+0000 7f302130a700 1 -- 172.21.15.42:0/1055142853 >> v1:172.21.15.42:6789/0 conn(0x7f301c09f830 legacy=0x7f301c14efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:16.781 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.769+0000 7f302130a700 1 --2- 172.21.15.42:0/1055142853 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3004003940 0x7f3004003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:30:16.781 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.769+0000 7f302130a700 1 --2- 172.21.15.42:0/1055142853 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3004004420 0x7f3004004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:30:16.781 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.769+0000 7f302230c700 1 --2- 172.21.15.42:0/1055142853 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3004003940 0x7f3004003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:30:16.781 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.769+0000 7f302130a700 1 --2- 172.21.15.42:0/1055142853 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3004004e10 0x7f300404f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:30:16.781 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.769+0000 7f302130a700 1 -- 172.21.15.42:0/1055142853 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f301c0a0b00 con 0x7f3004004e10 2019-12-05T06:30:16.782 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.769+0000 7f302130a700 1 -- 172.21.15.42:0/1055142853 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f300404fc00 con 0x7f3004003940 2019-12-05T06:30:16.782 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.769+0000 7f302130a700 1 -- 172.21.15.42:0/1055142853 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f300404fe40 con 0x7f3004004420 2019-12-05T06:30:16.782 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.769+0000 7f3021b0b700 1 --2- 172.21.15.42:0/1055142853 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3004004420 0x7f3004004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:30:16.782 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.769+0000 7f3022b0d700 1 --2- 172.21.15.42:0/1055142853 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3004004e10 0x7f300404f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:30:16.782 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.769+0000 7f302230c700 1 -- 172.21.15.42:0/1055142853 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3004004420 msgr2=0x7f3004004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:30:16.782 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.769+0000 7f302230c700 1 --2- 172.21.15.42:0/1055142853 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3004004420 0x7f3004004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:30:16.782 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.769+0000 7f302230c700 1 -- 172.21.15.42:0/1055142853 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3004004e10 msgr2=0x7f300404f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:30:16.783 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.769+0000 7f302230c700 1 --2- 172.21.15.42:0/1055142853 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3004004e10 0x7f300404f680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:30:16.783 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.769+0000 7f302230c700 1 -- 172.21.15.42:0/1055142853 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3004050140 con 0x7f3004003940 2019-12-05T06:30:16.783 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.769+0000 7f3022b0d700 1 --2- 172.21.15.42:0/1055142853 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3004004e10 0x7f300404f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).send_auth_request state changed! 2019-12-05T06:30:16.783 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.769+0000 7f302230c700 1 --2- 172.21.15.42:0/1055142853 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3004003940 0x7f3004003d90 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f3014001360 tx=0x7f301400c1f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:16.783 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.769+0000 7f302230c700 1 --2- 172.21.15.42:0/1055142853 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3004003940 0x7f3004003d90 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f3014001360 tx=0x7f301400c1f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:16.783 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.773+0000 7f302230c700 1 --2- 172.21.15.42:0/1055142853 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3004003940 0x7f3004003d90 secure :-1 s=READY pgs=2726 cs=0 l=1 rx=0x7f3014001360 tx=0x7f301400c1f0).ready entity=mon.1 client_cookie=37004bd80ce63027 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:30:16.783 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.773+0000 7f302230c700 1 --2- 172.21.15.42:0/1055142853 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3004003940 0x7f3004003d90 secure :-1 s=THROTTLE_DONE pgs=2726 cs=0 l=1 rx=0x7f3014001360 tx=0x7f301400c1f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:16.784 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.773+0000 7f302230c700 1 --2- 172.21.15.42:0/1055142853 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3004003940 0x7f3004003d90 secure :-1 s=THROTTLE_DONE pgs=2726 cs=0 l=1 rx=0x7f3014001360 tx=0x7f301400c1f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:16.784 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.773+0000 7f302130a700 1 -- 172.21.15.42:0/1055142853 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f301400c740 con 0x7f3004003940 2019-12-05T06:30:16.784 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.773+0000 7f302230c700 1 --2- 172.21.15.42:0/1055142853 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3004003940 0x7f3004003d90 secure :-1 s=THROTTLE_DONE pgs=2726 cs=0 l=1 rx=0x7f3014001360 tx=0x7f301400c1f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:16.785 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.773+0000 7f302130a700 1 -- 172.21.15.42:0/1055142853 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3014010b80 con 0x7f3004003940 2019-12-05T06:30:16.785 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.773+0000 7f302130a700 1 -- 172.21.15.42:0/1055142853 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f301400cd20 con 0x7f3004003940 2019-12-05T06:30:16.785 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.773+0000 7f3023d6f700 1 -- 172.21.15.42:0/1055142853 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3004003940 msgr2=0x7f3004003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:16.785 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.773+0000 7f3023d6f700 1 --2- 172.21.15.42:0/1055142853 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3004003940 0x7f3004003d90 secure :-1 s=READY pgs=2726 cs=0 l=1 rx=0x7f3014001360 tx=0x7f301400c1f0).stop 2019-12-05T06:30:16.785 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.773+0000 7f3022b0d700 1 -- 172.21.15.42:0/1055142853 reap_dead start 2019-12-05T06:30:16.786 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.773+0000 7f3023d6f700 1 -- 172.21.15.42:0/1055142853 shutdown_connections 2019-12-05T06:30:16.786 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.773+0000 7f3023d6f700 1 -- 172.21.15.42:0/1055142853 shutdown_connections 2019-12-05T06:30:16.786 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.773+0000 7f3023d6f700 1 -- 172.21.15.42:0/1055142853 wait complete. 2019-12-05T06:30:16.786 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.773+0000 7f3023d6f700 1 -- 172.21.15.42:0/1055142853 >> 172.21.15.42:0/1055142853 conn(0x7f301c0aabe0 msgr2=0x7f301c0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:30:16.786 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.773+0000 7f3023d6f700 1 Processor -- start 2019-12-05T06:30:16.786 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.773+0000 7f3023d6f700 1 -- start start 2019-12-05T06:30:16.791 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.773+0000 7f3023d6f700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f301c08eeb0 0x7f301c146ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:30:16.791 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.773+0000 7f3023d6f700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f301c148080 con 0x7f301c147850 2019-12-05T06:30:16.791 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.773+0000 7f3023d6f700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f301c0a2fb0 con 0x7f301c147420 2019-12-05T06:30:16.792 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.773+0000 7f3023d6f700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f301c148790 con 0x7f301c08eeb0 2019-12-05T06:30:16.792 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.773+0000 7f3021b0b700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f301c147420 0x7f301c1305c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:56486/0 (socket says 172.21.15.42:56486) 2019-12-05T06:30:16.792 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.773+0000 7f3021b0b700 1 -- 172.21.15.42:0/195835570 learned_addr learned my addr 172.21.15.42:0/195835570 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:30:16.792 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.773+0000 7f302230c700 1 --2- 172.21.15.42:0/195835570 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f301c08eeb0 0x7f301c146ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:30:16.792 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.773+0000 7f3012ffd700 1 -- 172.21.15.42:0/195835570 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4157335098 0 0) 0x7f301c0a2fb0 con 0x7f301c147420 2019-12-05T06:30:16.792 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.773+0000 7f3012ffd700 1 -- 172.21.15.42:0/195835570 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ffc0030b0 con 0x7f301c147420 2019-12-05T06:30:16.793 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.773+0000 7f3012ffd700 1 -- 172.21.15.42:0/195835570 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1884086552 0 0) 0x7f301c148080 con 0x7f301c147850 2019-12-05T06:30:16.793 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.773+0000 7f3012ffd700 1 -- 172.21.15.42:0/195835570 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f301c0a2fb0 con 0x7f301c147850 2019-12-05T06:30:16.793 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.773+0000 7f3012ffd700 1 -- 172.21.15.42:0/195835570 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 3867698065 0 0) 0x7f2ffc0030b0 con 0x7f301c147420 2019-12-05T06:30:16.793 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.773+0000 7f3012ffd700 1 -- 172.21.15.42:0/195835570 --> v1:172.21.15.42:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f301c148080 con 0x7f301c147420 2019-12-05T06:30:16.793 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.773+0000 7f3012ffd700 1 -- 172.21.15.42:0/195835570 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7f3018001490 con 0x7f301c147420 2019-12-05T06:30:16.793 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.773+0000 7f3012ffd700 1 -- 172.21.15.42:0/195835570 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f301c08eeb0 msgr2=0x7f301c146ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:16.793 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.773+0000 7f3012ffd700 1 --2- 172.21.15.42:0/195835570 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f301c08eeb0 0x7f301c146ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:16.793 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.773+0000 7f3012ffd700 1 -- 172.21.15.42:0/195835570 >> v1:172.21.15.42:6789/0 conn(0x7f301c147420 legacy=0x7f301c1305c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:16.794 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.773+0000 7f3012ffd700 1 -- 172.21.15.42:0/195835570 >> v1:172.21.15.40:6789/0 conn(0x7f301c147850 legacy=0x7f301c130dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:16.794 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.773+0000 7f3012ffd700 1 --2- 172.21.15.42:0/195835570 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2ffc003ab0 0x7f2ffc004420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:30:16.794 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.773+0000 7f3012ffd700 1 --2- 172.21.15.42:0/195835570 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2ffc003f20 0x7f2ffc004970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:30:16.794 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.773+0000 7f3012ffd700 1 --2- 172.21.15.42:0/195835570 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2ffc004fd0 0x7f2ffc00b710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:30:16.795 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.773+0000 7f3012ffd700 1 -- 172.21.15.42:0/195835570 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f301c148790 con 0x7f2ffc004fd0 2019-12-05T06:30:16.795 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.773+0000 7f3012ffd700 1 -- 172.21.15.42:0/195835570 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ffc00be40 con 0x7f2ffc003f20 2019-12-05T06:30:16.795 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.773+0000 7f3012ffd700 1 -- 172.21.15.42:0/195835570 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ffc00c080 con 0x7f2ffc003ab0 2019-12-05T06:30:16.795 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.773+0000 7f3021b0b700 1 --2- 172.21.15.42:0/195835570 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2ffc003f20 0x7f2ffc004970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:30:16.795 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.773+0000 7f3022b0d700 1 --2- 172.21.15.42:0/195835570 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2ffc004fd0 0x7f2ffc00b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:30:16.795 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.777+0000 7f302230c700 1 --2- 172.21.15.42:0/195835570 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2ffc003ab0 0x7f2ffc004420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:30:16.795 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.777+0000 7f3021b0b700 1 -- 172.21.15.42:0/195835570 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2ffc003ab0 msgr2=0x7f2ffc004420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:16.796 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.777+0000 7f3021b0b700 1 --2- 172.21.15.42:0/195835570 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2ffc003ab0 0x7f2ffc004420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:16.796 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.777+0000 7f3021b0b700 1 -- 172.21.15.42:0/195835570 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2ffc004fd0 msgr2=0x7f2ffc00b710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:16.796 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.777+0000 7f3021b0b700 1 --2- 172.21.15.42:0/195835570 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2ffc004fd0 0x7f2ffc00b710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:16.796 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.777+0000 7f3021b0b700 1 -- 172.21.15.42:0/195835570 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f301c0a1510 con 0x7f2ffc003f20 2019-12-05T06:30:16.796 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.777+0000 7f3023d6f700 1 -- 172.21.15.42:0/195835570 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f301c08eeb0 con 0x7f2ffc003f20 2019-12-05T06:30:16.796 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.777+0000 7f302230c700 1 --2- 172.21.15.42:0/195835570 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2ffc003ab0 0x7f2ffc004420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:30:16.796 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.777+0000 7f3023d6f700 1 -- 172.21.15.42:0/195835570 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f301c146ed0 con 0x7f2ffc003f20 2019-12-05T06:30:16.797 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.777+0000 7f3021b0b700 1 --2- 172.21.15.42:0/195835570 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2ffc003f20 0x7f2ffc004970 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f3018007a50 tx=0x7f3018011730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:16.797 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.777+0000 7f3021b0b700 1 --2- 172.21.15.42:0/195835570 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2ffc003f20 0x7f2ffc004970 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f3018007a50 tx=0x7f3018011730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:16.797 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.777+0000 7f3021b0b700 1 --2- 172.21.15.42:0/195835570 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2ffc003f20 0x7f2ffc004970 secure :-1 s=READY pgs=2728 cs=0 l=1 rx=0x7f3018007a50 tx=0x7f3018011730).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:30:16.797 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.777+0000 7f3021b0b700 1 --2- 172.21.15.42:0/195835570 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2ffc003f20 0x7f2ffc004970 secure :-1 s=THROTTLE_DONE pgs=2728 cs=0 l=1 rx=0x7f3018007a50 tx=0x7f3018011730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:16.797 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.777+0000 7f3012ffd700 1 -- 172.21.15.42:0/195835570 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3018005840 con 0x7f2ffc003f20 2019-12-05T06:30:16.797 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.777+0000 7f3021b0b700 1 --2- 172.21.15.42:0/195835570 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2ffc003f20 0x7f2ffc004970 secure :-1 s=THROTTLE_DONE pgs=2728 cs=0 l=1 rx=0x7f3018007a50 tx=0x7f3018011730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:16.797 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.777+0000 7f3021b0b700 1 --2- 172.21.15.42:0/195835570 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2ffc003f20 0x7f2ffc004970 secure :-1 s=THROTTLE_DONE pgs=2728 cs=0 l=1 rx=0x7f3018007a50 tx=0x7f3018011730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:16.798 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.777+0000 7f3012ffd700 1 -- 172.21.15.42:0/195835570 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3018007160 con 0x7f2ffc003f20 2019-12-05T06:30:16.798 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.777+0000 7f3012ffd700 1 -- 172.21.15.42:0/195835570 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3018010250 con 0x7f2ffc003f20 2019-12-05T06:30:16.798 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.777+0000 7f3021b0b700 1 --2- 172.21.15.42:0/195835570 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2ffc003f20 0x7f2ffc004970 secure :-1 s=THROTTLE_DONE pgs=2728 cs=0 l=1 rx=0x7f3018007a50 tx=0x7f3018011730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:16.798 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.777+0000 7f3012ffd700 1 -- 172.21.15.42:0/195835570 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f3018010450 con 0x7f2ffc003f20 2019-12-05T06:30:16.798 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.777+0000 7f3021b0b700 1 --2- 172.21.15.42:0/195835570 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2ffc003f20 0x7f2ffc004970 secure :-1 s=THROTTLE_DONE pgs=2728 cs=0 l=1 rx=0x7f3018007a50 tx=0x7f3018011730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:16.798 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.777+0000 7f3012ffd700 1 --2- 172.21.15.42:0/195835570 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2ffc043f90 0x7f2ffc0464a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:30:16.798 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.777+0000 7f3012ffd700 1 -- 172.21.15.42:0/195835570 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 11105+0+0 (secure 0 0 0) 0x7f301804baf0 con 0x7f2ffc003f20 2019-12-05T06:30:16.799 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.777+0000 7f302230c700 1 --2- 172.21.15.42:0/195835570 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2ffc043f90 0x7f2ffc0464a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:30:16.799 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.777+0000 7f302230c700 1 --2- 172.21.15.42:0/195835570 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2ffc043f90 0x7f2ffc0464a0 crc :-1 s=READY pgs=1144 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:30:16.799 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.785+0000 7f3023d6f700 1 -- 172.21.15.42:0/195835570 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ff8003200 con 0x7f2ffc003f20 2019-12-05T06:30:16.800 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.793+0000 7f3021b0b700 1 --2- 172.21.15.42:0/195835570 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2ffc003f20 0x7f2ffc004970 secure :-1 s=THROTTLE_DONE pgs=2728 cs=0 l=1 rx=0x7f3018007a50 tx=0x7f3018011730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:16.800 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.793+0000 7f3012ffd700 1 -- 172.21.15.42:0/195835570 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f3018033980 con 0x7f2ffc003f20 2019-12-05T06:30:16.955 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.949+0000 7f3023d6f700 1 -- 172.21.15.42:0/195835570 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2ff8003630 con 0x7f2ffc003f20 2019-12-05T06:30:16.955 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.949+0000 7f3021b0b700 1 --2- 172.21.15.42:0/195835570 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2ffc003f20 0x7f2ffc004970 secure :-1 s=THROTTLE_DONE pgs=2728 cs=0 l=1 rx=0x7f3018007a50 tx=0x7f3018011730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:16.956 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.949+0000 7f3012ffd700 1 -- 172.21.15.42:0/195835570 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 324 v324) v1 ==== 95+0+1778 (secure 0 0 0) 0x7f3018032b90 con 0x7f2ffc003f20 2019-12-05T06:30:16.957 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 324 2019-12-05T06:30:16.957 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:30:16.957 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":324,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":324,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:30:14.969128+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":108,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11265":{"gid":11265,"name":"a","rank":0,"incarnation":324,"state":"up:replay","state_seq":2,"addr":"172.21.15.42:6835/1985322937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1985322937},{"type":"v1","addr":"172.21.15.42:6835","nonce":1985322937}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:30:16.959 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.953+0000 7f3023d6f700 1 -- 172.21.15.42:0/195835570 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2ffc043f90 msgr2=0x7f2ffc0464a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:16.959 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.953+0000 7f3023d6f700 1 --2- 172.21.15.42:0/195835570 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2ffc043f90 0x7f2ffc0464a0 crc :-1 s=READY pgs=1144 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:16.959 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.953+0000 7f3022b0d700 1 -- 172.21.15.42:0/195835570 reap_dead start 2019-12-05T06:30:16.960 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.953+0000 7f3023d6f700 1 -- 172.21.15.42:0/195835570 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2ffc003f20 msgr2=0x7f2ffc004970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:16.960 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.953+0000 7f3023d6f700 1 --2- 172.21.15.42:0/195835570 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2ffc003f20 0x7f2ffc004970 secure :-1 s=READY pgs=2728 cs=0 l=1 rx=0x7f3018007a50 tx=0x7f3018011730).stop 2019-12-05T06:30:16.961 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.953+0000 7f3023d6f700 1 -- 172.21.15.42:0/195835570 shutdown_connections 2019-12-05T06:30:16.961 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.953+0000 7f3023d6f700 1 --2- 172.21.15.42:0/195835570 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2ffc003f20 0x7f2ffc004970 unknown :-1 s=CLOSED pgs=2728 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:16.961 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.953+0000 7f3023d6f700 1 -- 172.21.15.42:0/195835570 shutdown_connections 2019-12-05T06:30:16.961 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.953+0000 7f3023d6f700 1 -- 172.21.15.42:0/195835570 wait complete. 2019-12-05T06:30:16.961 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:16.953+0000 7f3023d6f700 1 -- 172.21.15.42:0/195835570 >> 172.21.15.42:0/195835570 conn(0x7f301c0aabe0 msgr2=0x7f301c11cc40 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:30:16.968 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 324, "feature_flags": { "enable_multiple": false, "ever_enabled_multiple": false }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2019-12-05T05:03:35.604738+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 324, "ever_allowed_features": 2, "explicitly_allowed_features": 2, "failed": [], "flags": 18, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_11265": { "addr": "172.21.15.42:6835/1985322937", "addrs": { "addrvec": [ { "addr": "172.21.15.42:6834", "nonce": 1985322937, "type": "v2" }, { "addr": "172.21.15.42:6835", "nonce": 1985322937, "type": "v1" } ] }, "export_targets": [], "features": 4540138292836696063, "flags": 0, "gid": 11265, "incarnation": 324, "name": "a", "rank": 0, "state": "up:replay", "state_seq": 2 } }, "last_failure": 0, "last_failure_osd_epoch": 108, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 2, "min_compat_client": "0 (unknown)", "modified": "2019-12-05T06:30:14.969128+0000", "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 11265 } } } ], "standbys": [] } 2019-12-05T06:30:18.970 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:30:19.076 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:19.065+0000 7fc589e1f700 1 Processor -- start 2019-12-05T06:30:19.076 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:19.065+0000 7fc589e1f700 1 -- start start 2019-12-05T06:30:19.076 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:19.069+0000 7fc589e1f700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc5840a7740 0x7fc58408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:30:19.077 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:19.069+0000 7fc589e1f700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc58407e720 0x7fc58407eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:30:19.077 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:19.069+0000 7fc589e1f700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc58409f2e0 0x7fc58409f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:30:19.077 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:19.069+0000 7fc589e1f700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5840a0b00 con 0x7fc5840a7740 2019-12-05T06:30:19.077 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:19.069+0000 7fc589e1f700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc58408e5b0 con 0x7fc58407e720 2019-12-05T06:30:19.077 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:19.069+0000 7fc589e1f700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc58408e850 con 0x7fc58409f2e0 2019-12-05T06:30:19.078 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:19.069+0000 7fc5837fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc58407e720 0x7fc58407eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:30:19.078 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:19.069+0000 7fc583fff700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc5840a7740 0x7fc58408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:30:19.078 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:19.069+0000 7fc583fff700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc5840a7740 0x7fc58408df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.42:44278/0 (socket says 172.21.15.42:44278) 2019-12-05T06:30:19.078 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:19.069+0000 7fc583fff700 1 -- 172.21.15.42:0/868701412 learned_addr learned my addr 172.21.15.42:0/868701412 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:30:19.079 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:19.069+0000 7fc588bbd700 1 --2- 172.21.15.42:0/868701412 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc58409f2e0 0x7fc58409f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:30:19.079 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:19.069+0000 7fc583fff700 1 -- 172.21.15.42:0/868701412 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc58409f2e0 msgr2=0x7fc58409f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:30:19.079 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:19.069+0000 7fc583fff700 1 --2- 172.21.15.42:0/868701412 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc58409f2e0 0x7fc58409f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:30:19.079 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:19.069+0000 7fc583fff700 1 -- 172.21.15.42:0/868701412 >> v2:172.21.15.42:3300/0 conn(0x7fc58407e720 msgr2=0x7fc58407eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:30:19.080 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:19.069+0000 7fc583fff700 1 --2- 172.21.15.42:0/868701412 >> v2:172.21.15.42:3300/0 conn(0x7fc58407e720 0x7fc58407eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:30:19.080 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:19.069+0000 7fc583fff700 1 -- 172.21.15.42:0/868701412 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc58407f0a0 con 0x7fc5840a7740 2019-12-05T06:30:19.080 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:19.069+0000 7fc583fff700 1 --2- 172.21.15.42:0/868701412 >> v2:172.21.15.40:3300/0 conn(0x7fc5840a7740 0x7fc58408df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fc5840a0f50 tx=0x7fc57c010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:19.080 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:19.069+0000 7fc583fff700 1 --2- 172.21.15.42:0/868701412 >> v2:172.21.15.40:3300/0 conn(0x7fc5840a7740 0x7fc58408df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fc5840a0f50 tx=0x7fc57c010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:19.081 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:19.069+0000 7fc583fff700 1 --2- 172.21.15.42:0/868701412 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc5840a7740 0x7fc58408df80 secure :-1 s=READY pgs=2047 cs=0 l=1 rx=0x7fc5840a0f50 tx=0x7fc57c010130).ready entity=mon.0 client_cookie=b955f2ff0e338602 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:30:19.081 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:19.069+0000 7fc583fff700 1 --2- 172.21.15.42:0/868701412 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc5840a7740 0x7fc58408df80 secure :-1 s=THROTTLE_DONE pgs=2047 cs=0 l=1 rx=0x7fc5840a0f50 tx=0x7fc57c010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:19.081 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:19.069+0000 7fc582ffd700 1 -- 172.21.15.42:0/868701412 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc57c010de0 con 0x7fc5840a7740 2019-12-05T06:30:19.081 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:19.069+0000 7fc583fff700 1 --2- 172.21.15.42:0/868701412 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc5840a7740 0x7fc58408df80 secure :-1 s=THROTTLE_DONE pgs=2047 cs=0 l=1 rx=0x7fc5840a0f50 tx=0x7fc57c010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:19.082 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:19.069+0000 7fc583fff700 1 --2- 172.21.15.42:0/868701412 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc5840a7740 0x7fc58408df80 secure :-1 s=THROTTLE_DONE pgs=2047 cs=0 l=1 rx=0x7fc5840a0f50 tx=0x7fc57c010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:19.082 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:19.069+0000 7fc582ffd700 1 -- 172.21.15.42:0/868701412 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc57c010a00 con 0x7fc5840a7740 2019-12-05T06:30:19.082 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:19.069+0000 7fc582ffd700 1 -- 172.21.15.42:0/868701412 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc57c0114a0 con 0x7fc5840a7740 2019-12-05T06:30:19.083 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:19.069+0000 7fc589e1f700 1 -- 172.21.15.42:0/868701412 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc5840a7740 msgr2=0x7fc58408df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:19.083 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:19.069+0000 7fc589e1f700 1 --2- 172.21.15.42:0/868701412 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc5840a7740 0x7fc58408df80 secure :-1 s=READY pgs=2047 cs=0 l=1 rx=0x7fc5840a0f50 tx=0x7fc57c010130).stop 2019-12-05T06:30:19.083 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:19.069+0000 7fc589e1f700 1 -- 172.21.15.42:0/868701412 shutdown_connections 2019-12-05T06:30:19.083 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:19.069+0000 7fc589e1f700 1 --2- 172.21.15.42:0/868701412 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc58409f2e0 0x7fc58409f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:30:19.084 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:19.069+0000 7fc589e1f700 1 --2- 172.21.15.42:0/868701412 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc5840a7740 0x7fc58408df80 unknown :-1 s=CLOSED pgs=2047 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:19.084 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:19.069+0000 7fc589e1f700 1 --2- 172.21.15.42:0/868701412 >> v2:172.21.15.42:3300/0 conn(0x7fc58407e720 0x7fc58407eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:30:19.084 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:19.069+0000 7fc589e1f700 1 -- 172.21.15.42:0/868701412 shutdown_connections 2019-12-05T06:30:19.084 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:19.069+0000 7fc589e1f700 1 -- 172.21.15.42:0/868701412 wait complete. 2019-12-05T06:30:19.085 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:19.069+0000 7fc589e1f700 1 -- 172.21.15.42:0/868701412 >> 172.21.15.42:0/868701412 conn(0x7fc5840aabe0 msgr2=0x7fc5840a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:30:19.091 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:19.073+0000 7fc589e1f700 1 Processor -- start 2019-12-05T06:30:19.092 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:19.073+0000 7fc589e1f700 1 -- start start 2019-12-05T06:30:19.092 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:19.073+0000 7fc589e1f700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc5841211b0 0x7fc584122700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:30:19.092 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:19.073+0000 7fc589e1f700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc58411b4e0 0x7fc58411b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:30:19.092 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:19.073+0000 7fc589e1f700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc584122020 con 0x7fc5840a4a50 2019-12-05T06:30:19.093 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:19.073+0000 7fc589e1f700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc58411b280 con 0x7fc58411b4e0 2019-12-05T06:30:19.093 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:19.073+0000 7fc589e1f700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5841230a0 con 0x7fc5841211b0 2019-12-05T06:30:19.093 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:19.073+0000 7fc5837fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc58411b4e0 0x7fc58411b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:30:19.094 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:19.073+0000 7fc5837fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc58411b4e0 0x7fc58411b910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:56470/0 (socket says 172.21.15.42:56470) 2019-12-05T06:30:19.094 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:19.073+0000 7fc5837fe700 1 -- 172.21.15.42:0/1705809963 learned_addr learned my addr 172.21.15.42:0/1705809963 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:30:19.094 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:19.073+0000 7fc583fff700 1 --2- 172.21.15.42:0/1705809963 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc5841211b0 0x7fc584122700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:30:19.094 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:19.073+0000 7fc5837fe700 1 -- 172.21.15.42:0/1705809963 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc5841211b0 msgr2=0x7fc584122700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:19.094 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:19.073+0000 7fc5837fe700 1 --2- 172.21.15.42:0/1705809963 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc5841211b0 0x7fc584122700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:19.095 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:19.073+0000 7fc5837fe700 1 -- 172.21.15.42:0/1705809963 >> v1:172.21.15.40:6789/0 conn(0x7fc5840a4a50 legacy=0x7fc5840a4e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:19.095 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:19.073+0000 7fc5837fe700 1 -- 172.21.15.42:0/1705809963 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc58407e030 con 0x7fc58411b4e0 2019-12-05T06:30:19.095 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:19.073+0000 7fc580ff9700 1 -- 172.21.15.42:0/1705809963 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1109528833 0 0) 0x7fc584122020 con 0x7fc5840a4a50 2019-12-05T06:30:19.095 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:19.073+0000 7fc5837fe700 1 --2- 172.21.15.42:0/1705809963 >> v2:172.21.15.42:3300/0 conn(0x7fc58411b4e0 0x7fc58411b910 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fc56c003070 tx=0x7fc56c010bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:19.096 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:19.073+0000 7fc5837fe700 1 --2- 172.21.15.42:0/1705809963 >> v2:172.21.15.42:3300/0 conn(0x7fc58411b4e0 0x7fc58411b910 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fc56c003070 tx=0x7fc56c010bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:19.096 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:19.073+0000 7fc5837fe700 1 --2- 172.21.15.42:0/1705809963 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc58411b4e0 0x7fc58411b910 secure :-1 s=READY pgs=2729 cs=0 l=1 rx=0x7fc56c003070 tx=0x7fc56c010bc0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:30:19.096 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:19.073+0000 7fc5837fe700 1 --2- 172.21.15.42:0/1705809963 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc58411b4e0 0x7fc58411b910 secure :-1 s=THROTTLE_DONE pgs=2729 cs=0 l=1 rx=0x7fc56c003070 tx=0x7fc56c010bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:19.096 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:19.073+0000 7fc580ff9700 1 -- 172.21.15.42:0/1705809963 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc56c005840 con 0x7fc58411b4e0 2019-12-05T06:30:19.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:19.073+0000 7fc5837fe700 1 --2- 172.21.15.42:0/1705809963 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc58411b4e0 0x7fc58411b910 secure :-1 s=THROTTLE_DONE pgs=2729 cs=0 l=1 rx=0x7fc56c003070 tx=0x7fc56c010bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:19.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:19.073+0000 7fc5837fe700 1 --2- 172.21.15.42:0/1705809963 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc58411b4e0 0x7fc58411b910 secure :-1 s=THROTTLE_DONE pgs=2729 cs=0 l=1 rx=0x7fc56c003070 tx=0x7fc56c010bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:19.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:19.073+0000 7fc580ff9700 1 -- 172.21.15.42:0/1705809963 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc56c007160 con 0x7fc58411b4e0 2019-12-05T06:30:19.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:19.073+0000 7fc589e1f700 1 -- 172.21.15.42:0/1705809963 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc58411be60 con 0x7fc58411b4e0 2019-12-05T06:30:19.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:19.073+0000 7fc580ff9700 1 -- 172.21.15.42:0/1705809963 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc56c012240 con 0x7fc58411b4e0 2019-12-05T06:30:19.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:19.073+0000 7fc589e1f700 1 -- 172.21.15.42:0/1705809963 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5841215e0 con 0x7fc58411b4e0 2019-12-05T06:30:19.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:19.073+0000 7fc5837fe700 1 --2- 172.21.15.42:0/1705809963 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc58411b4e0 0x7fc58411b910 secure :-1 s=THROTTLE_DONE pgs=2729 cs=0 l=1 rx=0x7fc56c003070 tx=0x7fc56c010bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:19.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:19.077+0000 7fc580ff9700 1 -- 172.21.15.42:0/1705809963 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fc56c012550 con 0x7fc58411b4e0 2019-12-05T06:30:19.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:19.077+0000 7fc5837fe700 1 --2- 172.21.15.42:0/1705809963 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc58411b4e0 0x7fc58411b910 secure :-1 s=THROTTLE_DONE pgs=2729 cs=0 l=1 rx=0x7fc56c003070 tx=0x7fc56c010bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:19.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:19.077+0000 7fc580ff9700 1 --2- 172.21.15.42:0/1705809963 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc56803ac60 0x7fc56803d170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:30:19.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:19.077+0000 7fc580ff9700 1 -- 172.21.15.42:0/1705809963 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 11105+0+0 (secure 0 0 0) 0x7fc56c04b130 con 0x7fc58411b4e0 2019-12-05T06:30:19.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:19.077+0000 7fc583fff700 1 --2- 172.21.15.42:0/1705809963 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc56803ac60 0x7fc56803d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:30:19.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:19.077+0000 7fc583fff700 1 --2- 172.21.15.42:0/1705809963 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc56803ac60 0x7fc56803d170 crc :-1 s=READY pgs=1145 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:30:19.100 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:19.081+0000 7fc589e1f700 1 -- 172.21.15.42:0/1705809963 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc584123540 con 0x7fc58411b4e0 2019-12-05T06:30:19.100 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:19.085+0000 7fc5837fe700 1 --2- 172.21.15.42:0/1705809963 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc58411b4e0 0x7fc58411b910 secure :-1 s=THROTTLE_DONE pgs=2729 cs=0 l=1 rx=0x7fc56c003070 tx=0x7fc56c010bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:19.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:19.085+0000 7fc580ff9700 1 -- 172.21.15.42:0/1705809963 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fc56c0321b0 con 0x7fc58411b4e0 2019-12-05T06:30:19.255 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:19.249+0000 7fc589e1f700 1 -- 172.21.15.42:0/1705809963 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc58405d110 con 0x7fc58411b4e0 2019-12-05T06:30:19.256 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:19.249+0000 7fc5837fe700 1 --2- 172.21.15.42:0/1705809963 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc58411b4e0 0x7fc58411b910 secure :-1 s=THROTTLE_DONE pgs=2729 cs=0 l=1 rx=0x7fc56c003070 tx=0x7fc56c010bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:19.256 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:19.249+0000 7fc580ff9700 1 -- 172.21.15.42:0/1705809963 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 326 v326) v1 ==== 95+0+1779 (secure 0 0 0) 0x7fc56c032410 con 0x7fc58411b4e0 2019-12-05T06:30:19.256 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 326 2019-12-05T06:30:19.257 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:30:19.257 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":326,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":326,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:30:18.994955+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":108,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11265":{"gid":11265,"name":"a","rank":0,"incarnation":324,"state":"up:rejoin","state_seq":27,"addr":"172.21.15.42:6835/1985322937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1985322937},{"type":"v1","addr":"172.21.15.42:6835","nonce":1985322937}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:30:19.259 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:19.253+0000 7fc589e1f700 1 -- 172.21.15.42:0/1705809963 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc56803ac60 msgr2=0x7fc56803d170 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:19.259 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:19.253+0000 7fc589e1f700 1 --2- 172.21.15.42:0/1705809963 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc56803ac60 0x7fc56803d170 crc :-1 s=READY pgs=1145 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:19.259 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:19.253+0000 7fc589e1f700 1 -- 172.21.15.42:0/1705809963 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc58411b4e0 msgr2=0x7fc58411b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:19.259 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:19.253+0000 7fc589e1f700 1 --2- 172.21.15.42:0/1705809963 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc58411b4e0 0x7fc58411b910 secure :-1 s=READY pgs=2729 cs=0 l=1 rx=0x7fc56c003070 tx=0x7fc56c010bc0).stop 2019-12-05T06:30:19.259 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:19.253+0000 7fc589e1f700 1 -- 172.21.15.42:0/1705809963 shutdown_connections 2019-12-05T06:30:19.259 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:19.253+0000 7fc589e1f700 1 --2- 172.21.15.42:0/1705809963 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc56803ac60 0x7fc56803d170 unknown :-1 s=CLOSED pgs=1145 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:19.259 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:19.253+0000 7fc589e1f700 1 --2- 172.21.15.42:0/1705809963 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc5841211b0 0x7fc584122700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:19.260 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:19.253+0000 7fc589e1f700 1 --2- 172.21.15.42:0/1705809963 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc58411b4e0 0x7fc58411b910 unknown :-1 s=CLOSED pgs=2729 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:19.260 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:19.253+0000 7fc589e1f700 1 -- 172.21.15.42:0/1705809963 shutdown_connections 2019-12-05T06:30:19.260 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:19.253+0000 7fc589e1f700 1 -- 172.21.15.42:0/1705809963 wait complete. 2019-12-05T06:30:19.260 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:19.253+0000 7fc589e1f700 1 -- 172.21.15.42:0/1705809963 >> 172.21.15.42:0/1705809963 conn(0x7fc5840a82a0 msgr2=0x7fc58409f5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:30:19.268 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2019-12-05T06:30:21.269 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:30:21.374 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:21.365+0000 7f70f224a700 1 Processor -- start 2019-12-05T06:30:21.374 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:21.365+0000 7f70f224a700 1 -- start start 2019-12-05T06:30:21.374 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:21.365+0000 7f70f224a700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f70ec0a7740 0x7f70ec08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:30:21.375 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:21.365+0000 7f70f224a700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f70ec07e720 0x7f70ec07eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:30:21.375 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:21.365+0000 7f70f224a700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70ec0a0710 con 0x7f70ec09f2e0 2019-12-05T06:30:21.375 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:21.365+0000 7f70f224a700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70ec0a0b00 con 0x7f70ec07e720 2019-12-05T06:30:21.375 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:21.365+0000 7f70f224a700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70ec08e5b0 con 0x7f70ec0a7740 2019-12-05T06:30:21.375 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:21.365+0000 7f70ebfff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f70ec0a7740 0x7f70ec08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:30:21.376 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:21.365+0000 7f70ebfff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f70ec0a7740 0x7f70ec08df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:56500/0 (socket says 172.21.15.42:56500) 2019-12-05T06:30:21.376 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:21.365+0000 7f70ebfff700 1 -- 172.21.15.42:0/492165655 learned_addr learned my addr 172.21.15.42:0/492165655 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:30:21.376 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:21.365+0000 7f70f0fe8700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f70ec09f2e0 0x7f70ec09f710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:56542/0 (socket says 172.21.15.42:56542) 2019-12-05T06:30:21.376 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:21.365+0000 7f70eb7fe700 1 --2- 172.21.15.42:0/492165655 >> v2:172.21.15.40:3300/0 conn(0x7f70ec07e720 0x7f70ec07eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:30:21.376 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:21.365+0000 7f70ebfff700 1 -- 172.21.15.42:0/492165655 >> v2:172.21.15.40:3300/0 conn(0x7f70ec07e720 msgr2=0x7f70ec07eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:30:21.377 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:21.365+0000 7f70ebfff700 1 --2- 172.21.15.42:0/492165655 >> v2:172.21.15.40:3300/0 conn(0x7f70ec07e720 0x7f70ec07eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:30:21.377 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:21.365+0000 7f70ebfff700 1 -- 172.21.15.42:0/492165655 >> v1:172.21.15.42:6789/0 conn(0x7f70ec09f2e0 legacy=0x7f70ec09f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:21.377 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:21.365+0000 7f70ebfff700 1 -- 172.21.15.42:0/492165655 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70ec07f0a0 con 0x7f70ec0a7740 2019-12-05T06:30:21.377 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:21.365+0000 7f70eaffd700 1 -- 172.21.15.42:0/492165655 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2387996742 0 0) 0x7f70ec0a0710 con 0x7f70ec09f2e0 2019-12-05T06:30:21.377 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:21.365+0000 7f70ebfff700 1 --2- 172.21.15.42:0/492165655 >> v2:172.21.15.42:3300/0 conn(0x7f70ec0a7740 0x7f70ec08df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f70d4002f30 tx=0x7f70d400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:21.377 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:21.365+0000 7f70ebfff700 1 --2- 172.21.15.42:0/492165655 >> v2:172.21.15.42:3300/0 conn(0x7f70ec0a7740 0x7f70ec08df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f70d4002f30 tx=0x7f70d400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:21.377 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:21.365+0000 7f70ebfff700 1 --2- 172.21.15.42:0/492165655 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f70ec0a7740 0x7f70ec08df80 secure :-1 s=READY pgs=2731 cs=0 l=1 rx=0x7f70d4002f30 tx=0x7f70d400fd30).ready entity=mon.1 client_cookie=3e585da3bd76c0a8 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:30:21.378 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:21.365+0000 7f70ebfff700 1 --2- 172.21.15.42:0/492165655 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f70ec0a7740 0x7f70ec08df80 secure :-1 s=THROTTLE_DONE pgs=2731 cs=0 l=1 rx=0x7f70d4002f30 tx=0x7f70d400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:21.378 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:21.365+0000 7f70eaffd700 1 -- 172.21.15.42:0/492165655 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f70d4011370 con 0x7f70ec0a7740 2019-12-05T06:30:21.378 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:21.365+0000 7f70ebfff700 1 --2- 172.21.15.42:0/492165655 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f70ec0a7740 0x7f70ec08df80 secure :-1 s=THROTTLE_DONE pgs=2731 cs=0 l=1 rx=0x7f70d4002f30 tx=0x7f70d400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:21.378 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:21.365+0000 7f70ebfff700 1 --2- 172.21.15.42:0/492165655 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f70ec0a7740 0x7f70ec08df80 secure :-1 s=THROTTLE_DONE pgs=2731 cs=0 l=1 rx=0x7f70d4002f30 tx=0x7f70d400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:21.378 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:21.365+0000 7f70eaffd700 1 -- 172.21.15.42:0/492165655 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70d4008bb0 con 0x7f70ec0a7740 2019-12-05T06:30:21.379 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:21.365+0000 7f70eaffd700 1 -- 172.21.15.42:0/492165655 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f70d4011950 con 0x7f70ec0a7740 2019-12-05T06:30:21.379 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:21.369+0000 7f70f224a700 1 -- 172.21.15.42:0/492165655 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f70ec0a7740 msgr2=0x7f70ec08df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:21.379 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:21.369+0000 7f70f224a700 1 --2- 172.21.15.42:0/492165655 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f70ec0a7740 0x7f70ec08df80 secure :-1 s=READY pgs=2731 cs=0 l=1 rx=0x7f70d4002f30 tx=0x7f70d400fd30).stop 2019-12-05T06:30:21.379 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:21.369+0000 7f70f224a700 1 -- 172.21.15.42:0/492165655 shutdown_connections 2019-12-05T06:30:21.379 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:21.369+0000 7f70f224a700 1 --2- 172.21.15.42:0/492165655 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f70ec0a7740 0x7f70ec08df80 unknown :-1 s=CLOSED pgs=2731 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:21.379 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:21.369+0000 7f70f224a700 1 --2- 172.21.15.42:0/492165655 >> v2:172.21.15.40:3300/0 conn(0x7f70ec07e720 0x7f70ec07eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:30:21.379 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:21.369+0000 7f70f224a700 1 -- 172.21.15.42:0/492165655 shutdown_connections 2019-12-05T06:30:21.380 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:21.369+0000 7f70f224a700 1 -- 172.21.15.42:0/492165655 wait complete. 2019-12-05T06:30:21.380 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:21.369+0000 7f70f224a700 1 -- 172.21.15.42:0/492165655 >> 172.21.15.42:0/492165655 conn(0x7f70ec0aabe0 msgr2=0x7f70ec0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:30:21.383 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:21.373+0000 7f70f224a700 1 Processor -- start 2019-12-05T06:30:21.384 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:21.373+0000 7f70f224a700 1 -- start start 2019-12-05T06:30:21.384 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:21.373+0000 7f70f224a700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f70ec1242d0 0x7f70ec0a39d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:30:21.384 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:21.373+0000 7f70f224a700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f70ec07dd70 0x7f70ec137710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:30:21.384 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:21.373+0000 7f70f224a700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70ec0a2cd0 con 0x7f70ec07d940 2019-12-05T06:30:21.384 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:21.373+0000 7f70f224a700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70ec0a1f90 con 0x7f70ec07dd70 2019-12-05T06:30:21.385 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:21.373+0000 7f70f224a700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70ec0a2230 con 0x7f70ec1242d0 2019-12-05T06:30:21.385 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:21.373+0000 7f70ebfff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f70ec1242d0 0x7f70ec0a39d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:30:21.385 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:21.373+0000 7f70ebfff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f70ec1242d0 0x7f70ec0a39d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:56506/0 (socket says 172.21.15.42:56506) 2019-12-05T06:30:21.385 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:21.373+0000 7f70ebfff700 1 -- 172.21.15.42:0/3358662816 learned_addr learned my addr 172.21.15.42:0/3358662816 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:30:21.385 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:21.373+0000 7f70f0fe8700 1 --2- 172.21.15.42:0/3358662816 >> v2:172.21.15.40:3300/0 conn(0x7f70ec07dd70 0x7f70ec137710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:30:21.385 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:21.373+0000 7f70f0fe8700 1 -- 172.21.15.42:0/3358662816 >> v2:172.21.15.42:3300/0 conn(0x7f70ec1242d0 msgr2=0x7f70ec0a39d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:21.386 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:21.373+0000 7f70f0fe8700 1 --2- 172.21.15.42:0/3358662816 >> v2:172.21.15.42:3300/0 conn(0x7f70ec1242d0 0x7f70ec0a39d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:21.386 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:21.373+0000 7f70f0fe8700 1 -- 172.21.15.42:0/3358662816 >> v1:172.21.15.42:6789/0 conn(0x7f70ec07d940 legacy=0x7f70ec0a24d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:21.386 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:21.373+0000 7f70f0fe8700 1 -- 172.21.15.42:0/3358662816 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70ec11c070 con 0x7f70ec07dd70 2019-12-05T06:30:21.386 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:21.373+0000 7f70ebfff700 1 --2- 172.21.15.42:0/3358662816 >> v2:172.21.15.42:3300/0 conn(0x7f70ec1242d0 0x7f70ec0a39d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:30:21.386 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:21.373+0000 7f70f0fe8700 1 --2- 172.21.15.42:0/3358662816 >> v2:172.21.15.40:3300/0 conn(0x7f70ec07dd70 0x7f70ec137710 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f70ec0a0df0 tx=0x7f70e4010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:21.386 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:21.373+0000 7f70f0fe8700 1 --2- 172.21.15.42:0/3358662816 >> v2:172.21.15.40:3300/0 conn(0x7f70ec07dd70 0x7f70ec137710 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f70ec0a0df0 tx=0x7f70e4010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:21.387 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:21.373+0000 7f70f0fe8700 1 --2- 172.21.15.42:0/3358662816 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f70ec07dd70 0x7f70ec137710 secure :-1 s=READY pgs=2049 cs=0 l=1 rx=0x7f70ec0a0df0 tx=0x7f70e4010b60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:30:21.387 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:21.373+0000 7f70f0fe8700 1 --2- 172.21.15.42:0/3358662816 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f70ec07dd70 0x7f70ec137710 secure :-1 s=THROTTLE_DONE pgs=2049 cs=0 l=1 rx=0x7f70ec0a0df0 tx=0x7f70e4010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:21.387 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:21.373+0000 7f70e8ff9700 1 -- 172.21.15.42:0/3358662816 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f70e400d0c0 con 0x7f70ec07dd70 2019-12-05T06:30:21.387 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:21.373+0000 7f70f0fe8700 1 --2- 172.21.15.42:0/3358662816 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f70ec07dd70 0x7f70ec137710 secure :-1 s=THROTTLE_DONE pgs=2049 cs=0 l=1 rx=0x7f70ec0a0df0 tx=0x7f70e4010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:21.387 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:21.373+0000 7f70f0fe8700 1 --2- 172.21.15.42:0/3358662816 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f70ec07dd70 0x7f70ec137710 secure :-1 s=THROTTLE_DONE pgs=2049 cs=0 l=1 rx=0x7f70ec0a0df0 tx=0x7f70e4010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:21.388 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:21.373+0000 7f70e8ff9700 1 -- 172.21.15.42:0/3358662816 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70e4011430 con 0x7f70ec07dd70 2019-12-05T06:30:21.388 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:21.373+0000 7f70f224a700 1 -- 172.21.15.42:0/3358662816 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70ec0a4350 con 0x7f70ec07dd70 2019-12-05T06:30:21.388 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:21.373+0000 7f70e8ff9700 1 -- 172.21.15.42:0/3358662816 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f70e400d6a0 con 0x7f70ec07dd70 2019-12-05T06:30:21.388 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:21.377+0000 7f70f224a700 1 -- 172.21.15.42:0/3358662816 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70ec0a3f20 con 0x7f70ec07dd70 2019-12-05T06:30:21.388 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:21.377+0000 7f70f0fe8700 1 --2- 172.21.15.42:0/3358662816 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f70ec07dd70 0x7f70ec137710 secure :-1 s=THROTTLE_DONE pgs=2049 cs=0 l=1 rx=0x7f70ec0a0df0 tx=0x7f70e4010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:21.389 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:21.377+0000 7f70e8ff9700 1 -- 172.21.15.42:0/3358662816 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f70e400d9b0 con 0x7f70ec07dd70 2019-12-05T06:30:21.389 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:21.377+0000 7f70f0fe8700 1 --2- 172.21.15.42:0/3358662816 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f70ec07dd70 0x7f70ec137710 secure :-1 s=THROTTLE_DONE pgs=2049 cs=0 l=1 rx=0x7f70ec0a0df0 tx=0x7f70e4010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:21.389 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:21.377+0000 7f70e8ff9700 1 --2- 172.21.15.42:0/3358662816 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f70d003ae40 0x7f70d003d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:30:21.389 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:21.377+0000 7f70e8ff9700 1 -- 172.21.15.42:0/3358662816 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 11105+0+0 (secure 0 0 0) 0x7f70e404af30 con 0x7f70ec07dd70 2019-12-05T06:30:21.389 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:21.377+0000 7f70ebfff700 1 --2- 172.21.15.42:0/3358662816 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f70d003ae40 0x7f70d003d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:30:21.389 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:21.377+0000 7f70ebfff700 1 --2- 172.21.15.42:0/3358662816 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f70d003ae40 0x7f70d003d350 crc :-1 s=READY pgs=1146 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:30:21.390 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:21.381+0000 7f70f224a700 1 -- 172.21.15.42:0/3358662816 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70c4003200 con 0x7f70ec07dd70 2019-12-05T06:30:21.396 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:21.389+0000 7f70f0fe8700 1 --2- 172.21.15.42:0/3358662816 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f70ec07dd70 0x7f70ec137710 secure :-1 s=THROTTLE_DONE pgs=2049 cs=0 l=1 rx=0x7f70ec0a0df0 tx=0x7f70e4010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:21.396 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:21.389+0000 7f70e8ff9700 1 -- 172.21.15.42:0/3358662816 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f70e4031fb0 con 0x7f70ec07dd70 2019-12-05T06:30:21.566 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:21.561+0000 7f70f224a700 1 -- 172.21.15.42:0/3358662816 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f70c4003630 con 0x7f70ec07dd70 2019-12-05T06:30:21.567 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:21.561+0000 7f70f0fe8700 1 --2- 172.21.15.42:0/3358662816 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f70ec07dd70 0x7f70ec137710 secure :-1 s=THROTTLE_DONE pgs=2049 cs=0 l=1 rx=0x7f70ec0a0df0 tx=0x7f70e4010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:21.567 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:21.561+0000 7f70e8ff9700 1 -- 172.21.15.42:0/3358662816 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 327 v327) v1 ==== 95+0+1779 (secure 0 0 0) 0x7f70e4032210 con 0x7f70ec07dd70 2019-12-05T06:30:21.569 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 327 2019-12-05T06:30:21.569 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:30:21.570 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":327,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":327,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:30:20.005055+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":108,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11265":{"gid":11265,"name":"a","rank":0,"incarnation":324,"state":"up:active","state_seq":28,"addr":"172.21.15.42:6835/1985322937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1985322937},{"type":"v1","addr":"172.21.15.42:6835","nonce":1985322937}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:30:21.572 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:21.565+0000 7f70f224a700 1 -- 172.21.15.42:0/3358662816 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f70d003ae40 msgr2=0x7f70d003d350 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:21.572 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:21.565+0000 7f70f224a700 1 --2- 172.21.15.42:0/3358662816 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f70d003ae40 0x7f70d003d350 crc :-1 s=READY pgs=1146 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:21.573 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:21.565+0000 7f70f224a700 1 -- 172.21.15.42:0/3358662816 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f70ec07dd70 msgr2=0x7f70ec137710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:21.573 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:21.565+0000 7f70f224a700 1 --2- 172.21.15.42:0/3358662816 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f70ec07dd70 0x7f70ec137710 secure :-1 s=READY pgs=2049 cs=0 l=1 rx=0x7f70ec0a0df0 tx=0x7f70e4010b60).stop 2019-12-05T06:30:21.573 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:21.565+0000 7f70f224a700 1 -- 172.21.15.42:0/3358662816 shutdown_connections 2019-12-05T06:30:21.574 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:21.565+0000 7f70f224a700 1 --2- 172.21.15.42:0/3358662816 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f70d003ae40 0x7f70d003d350 unknown :-1 s=CLOSED pgs=1146 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:21.574 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:21.565+0000 7f70f224a700 1 --2- 172.21.15.42:0/3358662816 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f70ec07dd70 0x7f70ec137710 unknown :-1 s=CLOSED pgs=2049 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:21.575 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:21.565+0000 7f70f224a700 1 --2- 172.21.15.42:0/3358662816 >> v2:172.21.15.42:3300/0 conn(0x7f70ec1242d0 0x7f70ec0a39d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:21.575 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:21.565+0000 7f70f224a700 1 -- 172.21.15.42:0/3358662816 shutdown_connections 2019-12-05T06:30:21.575 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:21.565+0000 7f70f224a700 1 -- 172.21.15.42:0/3358662816 wait complete. 2019-12-05T06:30:21.575 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:21.565+0000 7f70f224a700 1 -- 172.21.15.42:0/3358662816 >> 172.21.15.42:0/3358662816 conn(0x7f70ec0aabe0 msgr2=0x7f70ec09f660 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:30:21.582 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a has gained rank=0, replacing gid=11142 2019-12-05T06:30:21.582 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 5 secs before reviving mds.b 2019-12-05T06:30:26.583 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.b 2019-12-05T06:30:26.584 INFO:tasks.ceph.mds.b:Restarting daemon 2019-12-05T06:30:26.584 INFO:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2019-12-05T06:30:26.589 INFO:tasks.ceph.mds.b:Started 2019-12-05T06:30:26.589 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:30:26.643 INFO:tasks.ceph.mds.b.smithi040.stdout:starting mds.b at 2019-12-05T06:30:26.695 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:26.685+0000 7f81479ec700 1 Processor -- start 2019-12-05T06:30:26.695 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:26.685+0000 7f81479ec700 1 -- start start 2019-12-05T06:30:26.696 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:26.685+0000 7f81479ec700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f81400a7740 0x7f814008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:30:26.696 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:26.685+0000 7f81479ec700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f814007e720 0x7f814007eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:30:26.696 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:26.685+0000 7f81479ec700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81400a0710 con 0x7f814009f2e0 2019-12-05T06:30:26.696 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:26.685+0000 7f81479ec700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81400a0b00 con 0x7f81400a7740 2019-12-05T06:30:26.697 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:26.685+0000 7f81479ec700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f814008e5b0 con 0x7f814007e720 2019-12-05T06:30:26.697 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:26.685+0000 7f8145f89700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f81400a7740 0x7f814008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:30:26.697 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:26.685+0000 7f8145f89700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f81400a7740 0x7f814008df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:56538/0 (socket says 172.21.15.42:56538) 2019-12-05T06:30:26.697 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:26.685+0000 7f8145f89700 1 -- 172.21.15.42:0/3478226071 learned_addr learned my addr 172.21.15.42:0/3478226071 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:30:26.697 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:26.685+0000 7f8145788700 1 --2- 172.21.15.42:0/3478226071 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f814007e720 0x7f814007eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:30:26.697 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:26.685+0000 7f8144f87700 1 -- 172.21.15.42:0/3478226071 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1628795630 0 0) 0x7f81400a0710 con 0x7f814009f2e0 2019-12-05T06:30:26.697 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:26.685+0000 7f8144f87700 1 -- 172.21.15.42:0/3478226071 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f812c0030b0 con 0x7f814009f2e0 2019-12-05T06:30:26.698 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:26.685+0000 7f8145f89700 1 -- 172.21.15.42:0/3478226071 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f814007e720 msgr2=0x7f814007eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:30:26.698 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:26.685+0000 7f8145f89700 1 --2- 172.21.15.42:0/3478226071 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f814007e720 0x7f814007eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:30:26.698 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:26.685+0000 7f8145f89700 1 -- 172.21.15.42:0/3478226071 >> v1:172.21.15.40:6789/0 conn(0x7f814009f2e0 legacy=0x7f814009f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:26.698 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:26.685+0000 7f8145f89700 1 -- 172.21.15.42:0/3478226071 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f814007f0a0 con 0x7f81400a7740 2019-12-05T06:30:26.698 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:26.685+0000 7f8145f89700 1 --2- 172.21.15.42:0/3478226071 >> v2:172.21.15.42:3300/0 conn(0x7f81400a7740 0x7f814008df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f813c008b10 tx=0x7f813c0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:26.698 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:26.685+0000 7f8145f89700 1 --2- 172.21.15.42:0/3478226071 >> v2:172.21.15.42:3300/0 conn(0x7f81400a7740 0x7f814008df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f813c008b10 tx=0x7f813c0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:26.699 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:26.685+0000 7f8145f89700 1 --2- 172.21.15.42:0/3478226071 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f81400a7740 0x7f814008df80 secure :-1 s=READY pgs=2734 cs=0 l=1 rx=0x7f813c008b10 tx=0x7f813c0100b0).ready entity=mon.1 client_cookie=e7ad7096b8c34ec8 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:30:26.699 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:26.685+0000 7f8145f89700 1 --2- 172.21.15.42:0/3478226071 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f81400a7740 0x7f814008df80 secure :-1 s=THROTTLE_DONE pgs=2734 cs=0 l=1 rx=0x7f813c008b10 tx=0x7f813c0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:26.699 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:26.685+0000 7f8145f89700 1 --2- 172.21.15.42:0/3478226071 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f81400a7740 0x7f814008df80 secure :-1 s=THROTTLE_DONE pgs=2734 cs=0 l=1 rx=0x7f813c008b10 tx=0x7f813c0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:26.699 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:26.685+0000 7f8144f87700 1 -- 172.21.15.42:0/3478226071 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f813c00a160 con 0x7f81400a7740 2019-12-05T06:30:26.699 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:26.685+0000 7f8145f89700 1 --2- 172.21.15.42:0/3478226071 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f81400a7740 0x7f814008df80 secure :-1 s=THROTTLE_DONE pgs=2734 cs=0 l=1 rx=0x7f813c008b10 tx=0x7f813c0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:26.699 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:26.685+0000 7f8144f87700 1 -- 172.21.15.42:0/3478226071 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f813c011a30 con 0x7f81400a7740 2019-12-05T06:30:26.699 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:26.685+0000 7f8144f87700 1 -- 172.21.15.42:0/3478226071 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f813c00a550 con 0x7f81400a7740 2019-12-05T06:30:26.700 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:26.685+0000 7f81479ec700 1 -- 172.21.15.42:0/3478226071 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f81400a7740 msgr2=0x7f814008df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:26.700 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:26.685+0000 7f81479ec700 1 --2- 172.21.15.42:0/3478226071 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f81400a7740 0x7f814008df80 secure :-1 s=READY pgs=2734 cs=0 l=1 rx=0x7f813c008b10 tx=0x7f813c0100b0).stop 2019-12-05T06:30:26.700 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:26.689+0000 7f81479ec700 1 -- 172.21.15.42:0/3478226071 shutdown_connections 2019-12-05T06:30:26.700 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:26.689+0000 7f81479ec700 1 --2- 172.21.15.42:0/3478226071 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f81400a7740 0x7f814008df80 unknown :-1 s=CLOSED pgs=2734 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:26.700 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:26.689+0000 7f81479ec700 1 --2- 172.21.15.42:0/3478226071 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f814007e720 0x7f814007eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:30:26.700 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:26.689+0000 7f81479ec700 1 -- 172.21.15.42:0/3478226071 shutdown_connections 2019-12-05T06:30:26.700 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:26.689+0000 7f81479ec700 1 -- 172.21.15.42:0/3478226071 wait complete. 2019-12-05T06:30:26.701 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:26.689+0000 7f81479ec700 1 -- 172.21.15.42:0/3478226071 >> 172.21.15.42:0/3478226071 conn(0x7f81400aabe0 msgr2=0x7f81400a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:30:26.704 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:26.693+0000 7f81479ec700 1 Processor -- start 2019-12-05T06:30:26.704 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:26.693+0000 7f81479ec700 1 -- start start 2019-12-05T06:30:26.705 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:26.693+0000 7f81479ec700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f81400a1af0 0x7f814007db50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:30:26.705 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:26.693+0000 7f81479ec700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f814011deb0 0x7f814011a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:30:26.705 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:26.693+0000 7f81479ec700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f814011a7b0 0x7f814011b110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:30:26.705 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:26.693+0000 7f81479ec700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81400a1ff0 con 0x7f814011a7b0 2019-12-05T06:30:26.705 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:26.693+0000 7f81479ec700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81400a2290 con 0x7f81400a1af0 2019-12-05T06:30:26.705 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:26.693+0000 7f81479ec700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81400a2530 con 0x7f814011deb0 2019-12-05T06:30:26.705 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:26.693+0000 7f8145f89700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f81400a1af0 0x7f814007db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:30:26.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:26.693+0000 7f8145f89700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f81400a1af0 0x7f814007db50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:56544/0 (socket says 172.21.15.42:56544) 2019-12-05T06:30:26.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:26.693+0000 7f8145f89700 1 -- 172.21.15.42:0/1599982983 learned_addr learned my addr 172.21.15.42:0/1599982983 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:30:26.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:26.693+0000 7f8145788700 1 --2- 172.21.15.42:0/1599982983 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f814011deb0 0x7f814011a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:30:26.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:26.693+0000 7f814678a700 1 --2- 172.21.15.42:0/1599982983 >> v2:172.21.15.40:3300/0 conn(0x7f814011a7b0 0x7f814011b110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:30:26.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:26.693+0000 7f8145f89700 1 -- 172.21.15.42:0/1599982983 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f814011deb0 msgr2=0x7f814011a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:26.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:26.693+0000 7f8145f89700 1 --2- 172.21.15.42:0/1599982983 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f814011deb0 0x7f814011a260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:26.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:26.693+0000 7f8145f89700 1 -- 172.21.15.42:0/1599982983 >> v2:172.21.15.40:3300/0 conn(0x7f814011a7b0 msgr2=0x7f814011b110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:26.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:26.693+0000 7f8145f89700 1 --2- 172.21.15.42:0/1599982983 >> v2:172.21.15.40:3300/0 conn(0x7f814011a7b0 0x7f814011b110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:26.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:26.693+0000 7f8145f89700 1 -- 172.21.15.42:0/1599982983 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f814007e0a0 con 0x7f81400a1af0 2019-12-05T06:30:26.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:26.693+0000 7f814678a700 1 --2- 172.21.15.42:0/1599982983 >> v2:172.21.15.40:3300/0 conn(0x7f814011a7b0 0x7f814011b110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).send_auth_request state changed! 2019-12-05T06:30:26.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:26.693+0000 7f8145f89700 1 --2- 172.21.15.42:0/1599982983 >> v2:172.21.15.42:3300/0 conn(0x7f81400a1af0 0x7f814007db50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f813c008b10 tx=0x7f813c011d60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:26.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:26.693+0000 7f8145f89700 1 --2- 172.21.15.42:0/1599982983 >> v2:172.21.15.42:3300/0 conn(0x7f81400a1af0 0x7f814007db50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f813c008b10 tx=0x7f813c011d60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:26.708 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:26.693+0000 7f8145f89700 1 --2- 172.21.15.42:0/1599982983 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f81400a1af0 0x7f814007db50 secure :-1 s=READY pgs=2735 cs=0 l=1 rx=0x7f813c008b10 tx=0x7f813c011d60).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:30:26.708 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:26.693+0000 7f8145f89700 1 --2- 172.21.15.42:0/1599982983 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f81400a1af0 0x7f814007db50 secure :-1 s=THROTTLE_DONE pgs=2735 cs=0 l=1 rx=0x7f813c008b10 tx=0x7f813c011d60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:26.708 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:26.693+0000 7f8145f89700 1 --2- 172.21.15.42:0/1599982983 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f81400a1af0 0x7f814007db50 secure :-1 s=THROTTLE_DONE pgs=2735 cs=0 l=1 rx=0x7f813c008b10 tx=0x7f813c011d60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:26.708 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:26.693+0000 7f81367fc700 1 -- 172.21.15.42:0/1599982983 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f813c00aca0 con 0x7f81400a1af0 2019-12-05T06:30:26.708 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:26.693+0000 7f8145f89700 1 --2- 172.21.15.42:0/1599982983 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f81400a1af0 0x7f814007db50 secure :-1 s=THROTTLE_DONE pgs=2735 cs=0 l=1 rx=0x7f813c008b10 tx=0x7f813c011d60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:26.708 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:26.693+0000 7f81367fc700 1 -- 172.21.15.42:0/1599982983 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f813c011e10 con 0x7f81400a1af0 2019-12-05T06:30:26.708 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:26.693+0000 7f81479ec700 1 -- 172.21.15.42:0/1599982983 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f814011b660 con 0x7f81400a1af0 2019-12-05T06:30:26.709 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:26.693+0000 7f81367fc700 1 -- 172.21.15.42:0/1599982983 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f813c005ca0 con 0x7f81400a1af0 2019-12-05T06:30:26.709 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:26.693+0000 7f81479ec700 1 -- 172.21.15.42:0/1599982983 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f814008eb70 con 0x7f81400a1af0 2019-12-05T06:30:26.709 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:26.697+0000 7f8145f89700 1 --2- 172.21.15.42:0/1599982983 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f81400a1af0 0x7f814007db50 secure :-1 s=THROTTLE_DONE pgs=2735 cs=0 l=1 rx=0x7f813c008b10 tx=0x7f813c011d60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:26.709 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:26.697+0000 7f8145f89700 1 --2- 172.21.15.42:0/1599982983 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f81400a1af0 0x7f814007db50 secure :-1 s=THROTTLE_DONE pgs=2735 cs=0 l=1 rx=0x7f813c008b10 tx=0x7f813c011d60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:26.709 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:26.697+0000 7f81367fc700 1 -- 172.21.15.42:0/1599982983 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f813c007150 con 0x7f81400a1af0 2019-12-05T06:30:26.709 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:26.697+0000 7f81367fc700 1 --2- 172.21.15.42:0/1599982983 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f812803ae90 0x7f812803d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:30:26.709 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:26.697+0000 7f81367fc700 1 -- 172.21.15.42:0/1599982983 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 11105+0+0 (secure 0 0 0) 0x7f813c04bb90 con 0x7f81400a1af0 2019-12-05T06:30:26.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:26.697+0000 7f8145788700 1 --2- 172.21.15.42:0/1599982983 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f812803ae90 0x7f812803d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:30:26.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:26.697+0000 7f8145788700 1 --2- 172.21.15.42:0/1599982983 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f812803ae90 0x7f812803d3a0 crc :-1 s=READY pgs=1147 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:30:26.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:26.697+0000 7f81479ec700 1 -- 172.21.15.42:0/1599982983 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f814005d110 con 0x7f81400a1af0 2019-12-05T06:30:26.711 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:26.705+0000 7f8145f89700 1 --2- 172.21.15.42:0/1599982983 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f81400a1af0 0x7f814007db50 secure :-1 s=THROTTLE_DONE pgs=2735 cs=0 l=1 rx=0x7f813c008b10 tx=0x7f813c011d60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:26.711 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:26.705+0000 7f81367fc700 1 -- 172.21.15.42:0/1599982983 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f813c032e30 con 0x7f81400a1af0 2019-12-05T06:30:26.873 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:26.865+0000 7f81479ec700 1 -- 172.21.15.42:0/1599982983 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f814005b8c0 con 0x7f81400a1af0 2019-12-05T06:30:26.873 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:26.869+0000 7f8145f89700 1 --2- 172.21.15.42:0/1599982983 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f81400a1af0 0x7f814007db50 secure :-1 s=THROTTLE_DONE pgs=2735 cs=0 l=1 rx=0x7f813c008b10 tx=0x7f813c011d60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:26.874 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:26.869+0000 7f81367fc700 1 -- 172.21.15.42:0/1599982983 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 327 v327) v1 ==== 95+0+1779 (secure 0 0 0) 0x7f813c0307e0 con 0x7f81400a1af0 2019-12-05T06:30:26.874 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 327 2019-12-05T06:30:26.874 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:30:26.874 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":327,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":327,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:30:20.005055+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":108,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11265":{"gid":11265,"name":"a","rank":0,"incarnation":324,"state":"up:active","state_seq":28,"addr":"172.21.15.42:6835/1985322937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1985322937},{"type":"v1","addr":"172.21.15.42:6835","nonce":1985322937}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:30:26.875 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:26.869+0000 7f81479ec700 1 -- 172.21.15.42:0/1599982983 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f812803ae90 msgr2=0x7f812803d3a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:26.876 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:26.869+0000 7f81479ec700 1 --2- 172.21.15.42:0/1599982983 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f812803ae90 0x7f812803d3a0 crc :-1 s=READY pgs=1147 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:26.876 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:26.869+0000 7f81479ec700 1 -- 172.21.15.42:0/1599982983 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f81400a1af0 msgr2=0x7f814007db50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:26.876 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:26.869+0000 7f81479ec700 1 --2- 172.21.15.42:0/1599982983 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f81400a1af0 0x7f814007db50 secure :-1 s=READY pgs=2735 cs=0 l=1 rx=0x7f813c008b10 tx=0x7f813c011d60).stop 2019-12-05T06:30:26.876 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:26.869+0000 7f81479ec700 1 -- 172.21.15.42:0/1599982983 shutdown_connections 2019-12-05T06:30:26.876 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:26.869+0000 7f81479ec700 1 --2- 172.21.15.42:0/1599982983 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f812803ae90 0x7f812803d3a0 unknown :-1 s=CLOSED pgs=1147 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:26.876 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:26.869+0000 7f81479ec700 1 --2- 172.21.15.42:0/1599982983 >> v2:172.21.15.40:3300/0 conn(0x7f814011a7b0 0x7f814011b110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:26.877 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:26.869+0000 7f81479ec700 1 --2- 172.21.15.42:0/1599982983 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f81400a1af0 0x7f814007db50 unknown :-1 s=CLOSED pgs=2735 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:26.877 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:26.869+0000 7f81479ec700 1 --2- 172.21.15.42:0/1599982983 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f814011deb0 0x7f814011a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:26.877 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:26.869+0000 7f81479ec700 1 -- 172.21.15.42:0/1599982983 shutdown_connections 2019-12-05T06:30:26.877 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:26.869+0000 7f81479ec700 1 -- 172.21.15.42:0/1599982983 wait complete. 2019-12-05T06:30:26.877 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:26.869+0000 7f81479ec700 1 -- 172.21.15.42:0/1599982983 >> 172.21.15.42:0/1599982983 conn(0x7f81400aabe0 msgr2=0x7f814011c3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:30:26.885 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2019-12-05T06:30:28.887 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:30:28.992 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:28.981+0000 7f5b04f5c700 1 Processor -- start 2019-12-05T06:30:28.992 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:28.981+0000 7f5b04f5c700 1 -- start start 2019-12-05T06:30:28.993 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:28.985+0000 7f5b04f5c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5b000a7740 0x7f5b0008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:30:28.993 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:28.985+0000 7f5b04f5c700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b000a0710 con 0x7f5b0009fae0 2019-12-05T06:30:28.993 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:28.985+0000 7f5b04f5c700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b000a9ba0 con 0x7f5b0007e720 2019-12-05T06:30:28.993 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:28.985+0000 7f5b04f5c700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b000a0b00 con 0x7f5b000a7740 2019-12-05T06:30:28.993 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:28.985+0000 7f5afed9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5b000a7740 0x7f5b0008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:30:28.993 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:28.985+0000 7f5afed9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5b000a7740 0x7f5b0008df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:56572/0 (socket says 172.21.15.42:56572) 2019-12-05T06:30:28.994 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:28.985+0000 7f5afed9d700 1 -- 172.21.15.42:0/3081036115 learned_addr learned my addr 172.21.15.42:0/3081036115 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:30:28.994 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:28.985+0000 7f5afed9d700 1 -- 172.21.15.42:0/3081036115 >> v1:172.21.15.42:6789/0 conn(0x7f5b0007e720 legacy=0x7f5b0009f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:28.994 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:28.985+0000 7f5afed9d700 1 -- 172.21.15.42:0/3081036115 >> v1:172.21.15.40:6789/0 conn(0x7f5b0009fae0 legacy=0x7f5b0014efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:28.994 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:28.985+0000 7f5afed9d700 1 -- 172.21.15.42:0/3081036115 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b0007eb50 con 0x7f5b000a7740 2019-12-05T06:30:28.994 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:28.985+0000 7f5afed9d700 1 --2- 172.21.15.42:0/3081036115 >> v2:172.21.15.42:3300/0 conn(0x7f5b000a7740 0x7f5b0008df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f5af0008b10 tx=0x7f5af0010030).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:28.994 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:28.985+0000 7f5afed9d700 1 --2- 172.21.15.42:0/3081036115 >> v2:172.21.15.42:3300/0 conn(0x7f5b000a7740 0x7f5b0008df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f5af0008b10 tx=0x7f5af0010030).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:28.994 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:28.985+0000 7f5afed9d700 1 --2- 172.21.15.42:0/3081036115 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5b000a7740 0x7f5b0008df80 secure :-1 s=READY pgs=2737 cs=0 l=1 rx=0x7f5af0008b10 tx=0x7f5af0010030).ready entity=mon.1 client_cookie=c0b6141960edf063 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:30:28.995 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:28.985+0000 7f5afed9d700 1 --2- 172.21.15.42:0/3081036115 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5b000a7740 0x7f5b0008df80 secure :-1 s=THROTTLE_DONE pgs=2737 cs=0 l=1 rx=0x7f5af0008b10 tx=0x7f5af0010030).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:28.995 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:28.985+0000 7f5afed9d700 1 --2- 172.21.15.42:0/3081036115 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5b000a7740 0x7f5b0008df80 secure :-1 s=THROTTLE_DONE pgs=2737 cs=0 l=1 rx=0x7f5af0008b10 tx=0x7f5af0010030).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:28.995 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:28.985+0000 7f5afe59c700 1 -- 172.21.15.42:0/3081036115 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5af000a160 con 0x7f5b000a7740 2019-12-05T06:30:28.995 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:28.985+0000 7f5afed9d700 1 --2- 172.21.15.42:0/3081036115 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5b000a7740 0x7f5b0008df80 secure :-1 s=THROTTLE_DONE pgs=2737 cs=0 l=1 rx=0x7f5af0008b10 tx=0x7f5af0010030).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:28.995 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:28.985+0000 7f5afe59c700 1 -- 172.21.15.42:0/3081036115 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5af0011aa0 con 0x7f5b000a7740 2019-12-05T06:30:28.995 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:28.985+0000 7f5afe59c700 1 -- 172.21.15.42:0/3081036115 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5af000a740 con 0x7f5b000a7740 2019-12-05T06:30:28.996 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:28.985+0000 7f5b04f5c700 1 -- 172.21.15.42:0/3081036115 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5b000a7740 msgr2=0x7f5b0008df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:28.996 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:28.985+0000 7f5b04f5c700 1 --2- 172.21.15.42:0/3081036115 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5b000a7740 0x7f5b0008df80 secure :-1 s=READY pgs=2737 cs=0 l=1 rx=0x7f5af0008b10 tx=0x7f5af0010030).stop 2019-12-05T06:30:28.996 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:28.985+0000 7f5b04f5c700 1 -- 172.21.15.42:0/3081036115 shutdown_connections 2019-12-05T06:30:28.996 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:28.985+0000 7f5b04f5c700 1 --2- 172.21.15.42:0/3081036115 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5b000a7740 0x7f5b0008df80 unknown :-1 s=CLOSED pgs=2737 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:28.996 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:28.985+0000 7f5b04f5c700 1 -- 172.21.15.42:0/3081036115 shutdown_connections 2019-12-05T06:30:28.996 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:28.985+0000 7f5b04f5c700 1 -- 172.21.15.42:0/3081036115 wait complete. 2019-12-05T06:30:28.996 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:28.985+0000 7f5b04f5c700 1 -- 172.21.15.42:0/3081036115 >> 172.21.15.42:0/3081036115 conn(0x7f5b000aabe0 msgr2=0x7f5b000a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:30:29.000 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:28.989+0000 7f5b04f5c700 1 Processor -- start 2019-12-05T06:30:29.001 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:28.989+0000 7f5b04f5c700 1 -- start start 2019-12-05T06:30:29.001 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:28.989+0000 7f5b04f5c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5b00145c60 0x7f5b0008f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:30:29.001 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:28.989+0000 7f5b04f5c700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5b00144b50 0x7f5b0008f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:30:29.001 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:28.989+0000 7f5b04f5c700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5b00114e70 0x7f5b001152a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:30:29.001 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:28.989+0000 7f5b04f5c700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b00146840 con 0x7f5b00114e70 2019-12-05T06:30:29.001 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:28.989+0000 7f5b04f5c700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b00146ae0 con 0x7f5b00145c60 2019-12-05T06:30:29.002 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:28.989+0000 7f5b04f5c700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b001464e0 con 0x7f5b00144b50 2019-12-05T06:30:29.002 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:28.989+0000 7f5afed9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5b00145c60 0x7f5b0008f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:30:29.002 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:28.989+0000 7f5afed9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5b00145c60 0x7f5b0008f1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:56578/0 (socket says 172.21.15.42:56578) 2019-12-05T06:30:29.002 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:28.989+0000 7f5afed9d700 1 -- 172.21.15.42:0/2270856388 learned_addr learned my addr 172.21.15.42:0/2270856388 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:30:29.002 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:28.989+0000 7f5af659c700 1 --2- 172.21.15.42:0/2270856388 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5b00144b50 0x7f5b0008f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:30:29.002 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:28.989+0000 7f5aff59e700 1 --2- 172.21.15.42:0/2270856388 >> v2:172.21.15.40:3300/0 conn(0x7f5b00114e70 0x7f5b001152a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:30:29.003 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:28.989+0000 7f5afed9d700 1 -- 172.21.15.42:0/2270856388 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5b00144b50 msgr2=0x7f5b0008f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:29.003 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:28.989+0000 7f5afed9d700 1 --2- 172.21.15.42:0/2270856388 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5b00144b50 0x7f5b0008f6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:29.003 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:28.989+0000 7f5afed9d700 1 -- 172.21.15.42:0/2270856388 >> v2:172.21.15.40:3300/0 conn(0x7f5b00114e70 msgr2=0x7f5b001152a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:29.003 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:28.989+0000 7f5afed9d700 1 --2- 172.21.15.42:0/2270856388 >> v2:172.21.15.40:3300/0 conn(0x7f5b00114e70 0x7f5b001152a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:29.003 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:28.989+0000 7f5afed9d700 1 -- 172.21.15.42:0/2270856388 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b00145860 con 0x7f5b00145c60 2019-12-05T06:30:29.003 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:28.989+0000 7f5afed9d700 1 --2- 172.21.15.42:0/2270856388 >> v2:172.21.15.42:3300/0 conn(0x7f5b00145c60 0x7f5b0008f1a0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f5af0008b10 tx=0x7f5af0011dd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:29.003 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:28.989+0000 7f5afed9d700 1 --2- 172.21.15.42:0/2270856388 >> v2:172.21.15.42:3300/0 conn(0x7f5b00145c60 0x7f5b0008f1a0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f5af0008b10 tx=0x7f5af0011dd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:29.004 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:28.989+0000 7f5afed9d700 1 --2- 172.21.15.42:0/2270856388 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5b00145c60 0x7f5b0008f1a0 secure :-1 s=READY pgs=2738 cs=0 l=1 rx=0x7f5af0008b10 tx=0x7f5af0011dd0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:30:29.004 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:28.989+0000 7f5afed9d700 1 --2- 172.21.15.42:0/2270856388 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5b00145c60 0x7f5b0008f1a0 secure :-1 s=THROTTLE_DONE pgs=2738 cs=0 l=1 rx=0x7f5af0008b10 tx=0x7f5af0011dd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:29.004 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:28.989+0000 7f5afed9d700 1 --2- 172.21.15.42:0/2270856388 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5b00145c60 0x7f5b0008f1a0 secure :-1 s=THROTTLE_DONE pgs=2738 cs=0 l=1 rx=0x7f5af0008b10 tx=0x7f5af0011dd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:29.004 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:28.989+0000 7f5af7fff700 1 -- 172.21.15.42:0/2270856388 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5af00058c0 con 0x7f5b00145c60 2019-12-05T06:30:29.004 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:28.989+0000 7f5afed9d700 1 --2- 172.21.15.42:0/2270856388 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5b00145c60 0x7f5b0008f1a0 secure :-1 s=THROTTLE_DONE pgs=2738 cs=0 l=1 rx=0x7f5af0008b10 tx=0x7f5af0011dd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:29.004 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:28.989+0000 7f5af7fff700 1 -- 172.21.15.42:0/2270856388 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5af000a360 con 0x7f5b00145c60 2019-12-05T06:30:29.005 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:28.989+0000 7f5af7fff700 1 -- 172.21.15.42:0/2270856388 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5af0007150 con 0x7f5b00145c60 2019-12-05T06:30:29.005 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:28.989+0000 7f5b04f5c700 1 -- 172.21.15.42:0/2270856388 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b00147c30 con 0x7f5b00145c60 2019-12-05T06:30:29.005 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:28.993+0000 7f5b04f5c700 1 -- 172.21.15.42:0/2270856388 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b00115990 con 0x7f5b00145c60 2019-12-05T06:30:29.005 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:28.993+0000 7f5afed9d700 1 --2- 172.21.15.42:0/2270856388 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5b00145c60 0x7f5b0008f1a0 secure :-1 s=THROTTLE_DONE pgs=2738 cs=0 l=1 rx=0x7f5af0008b10 tx=0x7f5af0011dd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:29.005 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:28.993+0000 7f5af7fff700 1 -- 172.21.15.42:0/2270856388 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f5af0007460 con 0x7f5b00145c60 2019-12-05T06:30:29.005 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:28.993+0000 7f5afed9d700 1 --2- 172.21.15.42:0/2270856388 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5b00145c60 0x7f5b0008f1a0 secure :-1 s=THROTTLE_DONE pgs=2738 cs=0 l=1 rx=0x7f5af0008b10 tx=0x7f5af0011dd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:29.006 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:28.993+0000 7f5af7fff700 1 --2- 172.21.15.42:0/2270856388 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5ae403ae60 0x7f5ae403d370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:30:29.006 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:28.993+0000 7f5af7fff700 1 -- 172.21.15.42:0/2270856388 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 11105+0+0 (secure 0 0 0) 0x7f5af004be80 con 0x7f5b00145c60 2019-12-05T06:30:29.006 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:28.993+0000 7f5af659c700 1 --2- 172.21.15.42:0/2270856388 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5ae403ae60 0x7f5ae403d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:30:29.006 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:28.993+0000 7f5af659c700 1 --2- 172.21.15.42:0/2270856388 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5ae403ae60 0x7f5ae403d370 crc :-1 s=READY pgs=1150 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:30:29.006 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:28.997+0000 7f5b04f5c700 1 -- 172.21.15.42:0/2270856388 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b0005d110 con 0x7f5b00145c60 2019-12-05T06:30:29.009 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:29.005+0000 7f5afed9d700 1 --2- 172.21.15.42:0/2270856388 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5b00145c60 0x7f5b0008f1a0 secure :-1 s=THROTTLE_DONE pgs=2738 cs=0 l=1 rx=0x7f5af0008b10 tx=0x7f5af0011dd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:29.009 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:29.005+0000 7f5af7fff700 1 -- 172.21.15.42:0/2270856388 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f5af0032f00 con 0x7f5b00145c60 2019-12-05T06:30:29.212 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:29.208+0000 7f5b04f5c700 1 -- 172.21.15.42:0/2270856388 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5b0005b8c0 con 0x7f5b00145c60 2019-12-05T06:30:29.213 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:29.208+0000 7f5afed9d700 1 --2- 172.21.15.42:0/2270856388 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5b00145c60 0x7f5b0008f1a0 secure :-1 s=THROTTLE_DONE pgs=2738 cs=0 l=1 rx=0x7f5af0008b10 tx=0x7f5af0011dd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:29.213 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:29.208+0000 7f5af7fff700 1 -- 172.21.15.42:0/2270856388 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 328 v328) v1 ==== 95+0+2117 (secure 0 0 0) 0x7f5af0033160 con 0x7f5b00145c60 2019-12-05T06:30:29.213 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 328 2019-12-05T06:30:29.214 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:30:29.214 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":328,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":11352,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/2055706790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2055706790},{"type":"v1","addr":"172.21.15.40:6833","nonce":2055706790}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":328}],"filesystems":[{"mdsmap":{"epoch":327,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:30:20.005055+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":108,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11265":{"gid":11265,"name":"a","rank":0,"incarnation":324,"state":"up:active","state_seq":28,"addr":"172.21.15.42:6835/1985322937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1985322937},{"type":"v1","addr":"172.21.15.42:6835","nonce":1985322937}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:30:29.216 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:29.208+0000 7f5b04f5c700 1 -- 172.21.15.42:0/2270856388 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5ae403ae60 msgr2=0x7f5ae403d370 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:29.216 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:29.208+0000 7f5b04f5c700 1 --2- 172.21.15.42:0/2270856388 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5ae403ae60 0x7f5ae403d370 crc :-1 s=READY pgs=1150 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:29.217 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:29.208+0000 7f5b04f5c700 1 -- 172.21.15.42:0/2270856388 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5b00145c60 msgr2=0x7f5b0008f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:29.217 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:29.208+0000 7f5b04f5c700 1 --2- 172.21.15.42:0/2270856388 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5b00145c60 0x7f5b0008f1a0 secure :-1 s=READY pgs=2738 cs=0 l=1 rx=0x7f5af0008b10 tx=0x7f5af0011dd0).stop 2019-12-05T06:30:29.217 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:29.212+0000 7f5b04f5c700 1 -- 172.21.15.42:0/2270856388 shutdown_connections 2019-12-05T06:30:29.217 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:29.212+0000 7f5b04f5c700 1 --2- 172.21.15.42:0/2270856388 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5ae403ae60 0x7f5ae403d370 unknown :-1 s=CLOSED pgs=1150 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:29.217 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:29.212+0000 7f5b04f5c700 1 --2- 172.21.15.42:0/2270856388 >> v2:172.21.15.40:3300/0 conn(0x7f5b00114e70 0x7f5b001152a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:29.217 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:29.212+0000 7f5b04f5c700 1 --2- 172.21.15.42:0/2270856388 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5b00145c60 0x7f5b0008f1a0 unknown :-1 s=CLOSED pgs=2738 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:29.217 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:29.212+0000 7f5b04f5c700 1 --2- 172.21.15.42:0/2270856388 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5b00144b50 0x7f5b0008f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:29.218 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:29.212+0000 7f5b04f5c700 1 -- 172.21.15.42:0/2270856388 shutdown_connections 2019-12-05T06:30:29.218 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:29.212+0000 7f5b04f5c700 1 -- 172.21.15.42:0/2270856388 wait complete. 2019-12-05T06:30:29.218 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:30:29.212+0000 7f5b04f5c700 1 -- 172.21.15.42:0/2270856388 >> 172.21.15.42:0/2270856388 conn(0x7f5b0007dff0 msgr2=0x7f5b00059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:30:29.227 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b reported in up:standby state 2019-12-05T06:30:29.228 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 31 secs before thrashing 2019-12-05T06:31:00.229 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:31:00.334 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.324+0000 7f256dcc8700 1 Processor -- start 2019-12-05T06:31:00.335 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.324+0000 7f256dcc8700 1 -- start start 2019-12-05T06:31:00.335 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.324+0000 7f256dcc8700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f25680a7740 0x7f256808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:31:00.335 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.324+0000 7f256dcc8700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f256807e720 0x7f256807eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:31:00.335 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.324+0000 7f256dcc8700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f256809f2e0 0x7f256809f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:31:00.336 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.324+0000 7f256dcc8700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25680a0b00 con 0x7f256809f2e0 2019-12-05T06:31:00.336 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.324+0000 7f256dcc8700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f256808e5b0 con 0x7f25680a7740 2019-12-05T06:31:00.336 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.324+0000 7f256dcc8700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f256808e850 con 0x7f256807e720 2019-12-05T06:31:00.336 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.324+0000 7f2567fff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f25680a7740 0x7f256808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:31:00.336 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.324+0000 7f2567fff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f25680a7740 0x7f256808df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:56726/0 (socket says 172.21.15.42:56726) 2019-12-05T06:31:00.336 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.324+0000 7f2567fff700 1 -- 172.21.15.42:0/2656449852 learned_addr learned my addr 172.21.15.42:0/2656449852 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:31:00.337 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.324+0000 7f256ca66700 1 --2- 172.21.15.42:0/2656449852 >> v2:172.21.15.40:3300/0 conn(0x7f256809f2e0 0x7f256809f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:31:00.337 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.324+0000 7f25677fe700 1 --2- 172.21.15.42:0/2656449852 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f256807e720 0x7f256807eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:31:00.337 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.324+0000 7f2567fff700 1 -- 172.21.15.42:0/2656449852 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f256807e720 msgr2=0x7f256807eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:31:00.337 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.324+0000 7f2567fff700 1 --2- 172.21.15.42:0/2656449852 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f256807e720 0x7f256807eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:31:00.337 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.324+0000 7f2567fff700 1 -- 172.21.15.42:0/2656449852 >> v2:172.21.15.40:3300/0 conn(0x7f256809f2e0 msgr2=0x7f256809f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:31:00.337 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.324+0000 7f2567fff700 1 --2- 172.21.15.42:0/2656449852 >> v2:172.21.15.40:3300/0 conn(0x7f256809f2e0 0x7f256809f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:31:00.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.324+0000 7f2567fff700 1 -- 172.21.15.42:0/2656449852 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f256807f0a0 con 0x7f25680a7740 2019-12-05T06:31:00.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.324+0000 7f25677fe700 1 --2- 172.21.15.42:0/2656449852 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f256807e720 0x7f256807eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).send_auth_request state changed! 2019-12-05T06:31:00.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.324+0000 7f256ca66700 1 --2- 172.21.15.42:0/2656449852 >> v2:172.21.15.40:3300/0 conn(0x7f256809f2e0 0x7f256809f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).send_auth_request state changed! 2019-12-05T06:31:00.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.324+0000 7f2567fff700 1 --2- 172.21.15.42:0/2656449852 >> v2:172.21.15.42:3300/0 conn(0x7f25680a7740 0x7f256808df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f256808e770 tx=0x7f255800fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:00.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.324+0000 7f2567fff700 1 --2- 172.21.15.42:0/2656449852 >> v2:172.21.15.42:3300/0 conn(0x7f25680a7740 0x7f256808df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f256808e770 tx=0x7f255800fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:00.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.324+0000 7f2567fff700 1 --2- 172.21.15.42:0/2656449852 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f25680a7740 0x7f256808df80 secure :-1 s=READY pgs=2740 cs=0 l=1 rx=0x7f256808e770 tx=0x7f255800fdb0).ready entity=mon.1 client_cookie=1c358cdb8e8f6889 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:31:00.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.328+0000 7f2567fff700 1 --2- 172.21.15.42:0/2656449852 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f25680a7740 0x7f256808df80 secure :-1 s=THROTTLE_DONE pgs=2740 cs=0 l=1 rx=0x7f256808e770 tx=0x7f255800fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:00.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.328+0000 7f2567fff700 1 --2- 172.21.15.42:0/2656449852 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f25680a7740 0x7f256808df80 secure :-1 s=THROTTLE_DONE pgs=2740 cs=0 l=1 rx=0x7f256808e770 tx=0x7f255800fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:00.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.328+0000 7f2567fff700 1 --2- 172.21.15.42:0/2656449852 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f25680a7740 0x7f256808df80 secure :-1 s=THROTTLE_DONE pgs=2740 cs=0 l=1 rx=0x7f256808e770 tx=0x7f255800fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:00.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.328+0000 7f2566ffd700 1 -- 172.21.15.42:0/2656449852 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2558008c90 con 0x7f25680a7740 2019-12-05T06:31:00.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.328+0000 7f2566ffd700 1 -- 172.21.15.42:0/2656449852 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25580088b0 con 0x7f25680a7740 2019-12-05T06:31:00.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.328+0000 7f2566ffd700 1 -- 172.21.15.42:0/2656449852 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2558011b50 con 0x7f25680a7740 2019-12-05T06:31:00.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.328+0000 7f256dcc8700 1 -- 172.21.15.42:0/2656449852 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f25680a7740 msgr2=0x7f256808df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:00.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.328+0000 7f256dcc8700 1 --2- 172.21.15.42:0/2656449852 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f25680a7740 0x7f256808df80 secure :-1 s=READY pgs=2740 cs=0 l=1 rx=0x7f256808e770 tx=0x7f255800fdb0).stop 2019-12-05T06:31:00.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.328+0000 7f256dcc8700 1 -- 172.21.15.42:0/2656449852 shutdown_connections 2019-12-05T06:31:00.341 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.328+0000 7f256dcc8700 1 --2- 172.21.15.42:0/2656449852 >> v2:172.21.15.40:3300/0 conn(0x7f256809f2e0 0x7f256809f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:31:00.341 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.328+0000 7f256dcc8700 1 --2- 172.21.15.42:0/2656449852 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f25680a7740 0x7f256808df80 unknown :-1 s=CLOSED pgs=2740 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:31:00.341 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.328+0000 7f256dcc8700 1 --2- 172.21.15.42:0/2656449852 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f256807e720 0x7f256807eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:31:00.341 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.328+0000 7f256dcc8700 1 -- 172.21.15.42:0/2656449852 shutdown_connections 2019-12-05T06:31:00.341 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.328+0000 7f256dcc8700 1 -- 172.21.15.42:0/2656449852 wait complete. 2019-12-05T06:31:00.342 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.328+0000 7f256dcc8700 1 -- 172.21.15.42:0/2656449852 >> 172.21.15.42:0/2656449852 conn(0x7f25680aabe0 msgr2=0x7f25680a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:31:00.346 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.328+0000 7f256dcc8700 1 Processor -- start 2019-12-05T06:31:00.347 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.328+0000 7f256dcc8700 1 -- start start 2019-12-05T06:31:00.347 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.332+0000 7f256dcc8700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f25681448c0 0x7f2568117bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:31:00.347 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.332+0000 7f256dcc8700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2568144cf0 con 0x7f2568119550 2019-12-05T06:31:00.347 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.332+0000 7f256dcc8700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2568117010 con 0x7f25680a5d00 2019-12-05T06:31:00.347 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.332+0000 7f256dcc8700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25681176b0 con 0x7f25681448c0 2019-12-05T06:31:00.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.332+0000 7f2567fff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f25680a5d00 0x7f256807b4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:56770/0 (socket says 172.21.15.42:56770) 2019-12-05T06:31:00.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.332+0000 7f2567fff700 1 -- 172.21.15.42:0/532886186 learned_addr learned my addr 172.21.15.42:0/532886186 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:31:00.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.332+0000 7f256ca66700 1 --2- 172.21.15.42:0/532886186 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f25681448c0 0x7f2568117bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:31:00.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.332+0000 7f2564ff9700 1 -- 172.21.15.42:0/532886186 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 121214212 0 0) 0x7f2568144cf0 con 0x7f2568119550 2019-12-05T06:31:00.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.332+0000 7f2564ff9700 1 -- 172.21.15.42:0/532886186 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25480030b0 con 0x7f2568119550 2019-12-05T06:31:00.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.332+0000 7f256ca66700 1 -- 172.21.15.42:0/532886186 >> v1:172.21.15.42:6789/0 conn(0x7f25680a5d00 legacy=0x7f256807b4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:00.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.332+0000 7f256ca66700 1 -- 172.21.15.42:0/532886186 >> v1:172.21.15.40:6789/0 conn(0x7f2568119550 legacy=0x7f2568135ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:00.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.332+0000 7f256ca66700 1 -- 172.21.15.42:0/532886186 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2568118140 con 0x7f25681448c0 2019-12-05T06:31:00.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.332+0000 7f256ca66700 1 --2- 172.21.15.42:0/532886186 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f25681448c0 0x7f2568117bf0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f2568142560 tx=0x7f2560010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:00.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.332+0000 7f256ca66700 1 --2- 172.21.15.42:0/532886186 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f25681448c0 0x7f2568117bf0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f2568142560 tx=0x7f2560010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:00.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.332+0000 7f256ca66700 1 --2- 172.21.15.42:0/532886186 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f25681448c0 0x7f2568117bf0 secure :-1 s=READY pgs=600 cs=0 l=1 rx=0x7f2568142560 tx=0x7f2560010ac0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:31:00.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.332+0000 7f256ca66700 1 --2- 172.21.15.42:0/532886186 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f25681448c0 0x7f2568117bf0 secure :-1 s=THROTTLE_DONE pgs=600 cs=0 l=1 rx=0x7f2568142560 tx=0x7f2560010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:00.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.332+0000 7f256ca66700 1 --2- 172.21.15.42:0/532886186 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f25681448c0 0x7f2568117bf0 secure :-1 s=THROTTLE_DONE pgs=600 cs=0 l=1 rx=0x7f2568142560 tx=0x7f2560010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:00.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.332+0000 7f256ca66700 1 --2- 172.21.15.42:0/532886186 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f25681448c0 0x7f2568117bf0 secure :-1 s=THROTTLE_DONE pgs=600 cs=0 l=1 rx=0x7f2568142560 tx=0x7f2560010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:00.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.336+0000 7f2564ff9700 1 -- 172.21.15.42:0/532886186 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2560007160 con 0x7f25681448c0 2019-12-05T06:31:00.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.336+0000 7f2564ff9700 1 -- 172.21.15.42:0/532886186 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f256000f0b0 con 0x7f25681448c0 2019-12-05T06:31:00.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.336+0000 7f2564ff9700 1 -- 172.21.15.42:0/532886186 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f256000e1c0 con 0x7f25681448c0 2019-12-05T06:31:00.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.336+0000 7f256dcc8700 1 -- 172.21.15.42:0/532886186 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2568119c50 con 0x7f25681448c0 2019-12-05T06:31:00.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.336+0000 7f256dcc8700 1 -- 172.21.15.42:0/532886186 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f256807bcb0 con 0x7f25681448c0 2019-12-05T06:31:00.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.336+0000 7f256ca66700 1 --2- 172.21.15.42:0/532886186 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f25681448c0 0x7f2568117bf0 secure :-1 s=THROTTLE_DONE pgs=600 cs=0 l=1 rx=0x7f2568142560 tx=0x7f2560010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:00.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.336+0000 7f2564ff9700 1 -- 172.21.15.42:0/532886186 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f256000e3c0 con 0x7f25681448c0 2019-12-05T06:31:00.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.336+0000 7f256ca66700 1 --2- 172.21.15.42:0/532886186 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f25681448c0 0x7f2568117bf0 secure :-1 s=THROTTLE_DONE pgs=600 cs=0 l=1 rx=0x7f2568142560 tx=0x7f2560010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:00.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.336+0000 7f2564ff9700 1 --2- 172.21.15.42:0/532886186 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f254803b280 0x7f254803d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:31:00.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.336+0000 7f2564ff9700 1 -- 172.21.15.42:0/532886186 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 11105+0+0 (secure 0 0 0) 0x7f256004af80 con 0x7f25681448c0 2019-12-05T06:31:00.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.336+0000 7f2567fff700 1 --2- 172.21.15.42:0/532886186 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f254803b280 0x7f254803d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:31:00.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.336+0000 7f2567fff700 1 --2- 172.21.15.42:0/532886186 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f254803b280 0x7f254803d790 crc :-1 s=READY pgs=1151 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:31:00.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.340+0000 7f256dcc8700 1 -- 172.21.15.42:0/532886186 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f256805d110 con 0x7f25681448c0 2019-12-05T06:31:00.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.348+0000 7f256ca66700 1 --2- 172.21.15.42:0/532886186 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f25681448c0 0x7f2568117bf0 secure :-1 s=THROTTLE_DONE pgs=600 cs=0 l=1 rx=0x7f2568142560 tx=0x7f2560010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:00.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.348+0000 7f2564ff9700 1 -- 172.21.15.42:0/532886186 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f2568118140 con 0x7f25681448c0 2019-12-05T06:31:00.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.524+0000 7f256dcc8700 1 -- 172.21.15.42:0/532886186 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f256809fd00 con 0x7f25681448c0 2019-12-05T06:31:00.530 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.524+0000 7f256ca66700 1 --2- 172.21.15.42:0/532886186 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f25681448c0 0x7f2568117bf0 secure :-1 s=THROTTLE_DONE pgs=600 cs=0 l=1 rx=0x7f2568142560 tx=0x7f2560010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:00.530 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.524+0000 7f2564ff9700 1 -- 172.21.15.42:0/532886186 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 328 v328) v1 ==== 95+0+2117 (secure 0 0 0) 0x7f2560032260 con 0x7f25681448c0 2019-12-05T06:31:00.532 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 328 2019-12-05T06:31:00.532 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:31:00.533 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":328,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":11352,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/2055706790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2055706790},{"type":"v1","addr":"172.21.15.40:6833","nonce":2055706790}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":328}],"filesystems":[{"mdsmap":{"epoch":327,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:30:20.005055+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":108,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11265":{"gid":11265,"name":"a","rank":0,"incarnation":324,"state":"up:active","state_seq":28,"addr":"172.21.15.42:6835/1985322937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1985322937},{"type":"v1","addr":"172.21.15.42:6835","nonce":1985322937}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:31:00.535 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.528+0000 7f256dcc8700 1 -- 172.21.15.42:0/532886186 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f254803b280 msgr2=0x7f254803d790 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:00.535 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.528+0000 7f256dcc8700 1 --2- 172.21.15.42:0/532886186 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f254803b280 0x7f254803d790 crc :-1 s=READY pgs=1151 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:31:00.535 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.528+0000 7f256dcc8700 1 -- 172.21.15.42:0/532886186 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f25681448c0 msgr2=0x7f2568117bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:00.535 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.528+0000 7f256dcc8700 1 --2- 172.21.15.42:0/532886186 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f25681448c0 0x7f2568117bf0 secure :-1 s=READY pgs=600 cs=0 l=1 rx=0x7f2568142560 tx=0x7f2560010ac0).stop 2019-12-05T06:31:00.536 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.528+0000 7f256dcc8700 1 -- 172.21.15.42:0/532886186 shutdown_connections 2019-12-05T06:31:00.536 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.528+0000 7f256dcc8700 1 --2- 172.21.15.42:0/532886186 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f254803b280 0x7f254803d790 unknown :-1 s=CLOSED pgs=1151 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:31:00.536 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.528+0000 7f256dcc8700 1 --2- 172.21.15.42:0/532886186 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f25681448c0 0x7f2568117bf0 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:31:00.536 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.528+0000 7f256dcc8700 1 -- 172.21.15.42:0/532886186 shutdown_connections 2019-12-05T06:31:00.536 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.528+0000 7f256dcc8700 1 -- 172.21.15.42:0/532886186 wait complete. 2019-12-05T06:31:00.536 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.528+0000 7f256dcc8700 1 -- 172.21.15.42:0/532886186 >> 172.21.15.42:0/532886186 conn(0x7f256811c590 msgr2=0x7f256811cde0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:31:00.545 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.a (rank=0) 2019-12-05T06:31:00.545 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2019-12-05T06:31:00.546 INFO:teuthology.orchestra.run:waiting for 300 2019-12-05T06:31:00.560 INFO:tasks.ceph.mds.a:Stopped 2019-12-05T06:31:00.560 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:31:00.732 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.720+0000 7f7bd8c92700 1 Processor -- start 2019-12-05T06:31:00.733 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.720+0000 7f7bd8c92700 1 -- start start 2019-12-05T06:31:00.733 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.720+0000 7f7bd8c92700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7bd409fae0 0x7f7bd414fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:31:00.734 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.720+0000 7f7bd8c92700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bd40a0710 con 0x7f7bd40a7740 2019-12-05T06:31:00.734 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.720+0000 7f7bd8c92700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bd407ef20 con 0x7f7bd408df80 2019-12-05T06:31:00.735 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.720+0000 7f7bd8c92700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bd40a0b00 con 0x7f7bd409fae0 2019-12-05T06:31:00.735 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.720+0000 7f7bd259c700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f7bd408df80 0x7f7bd409f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:56782/0 (socket says 172.21.15.42:56782) 2019-12-05T06:31:00.735 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.720+0000 7f7bd259c700 1 -- 172.21.15.42:0/1144140410 learned_addr learned my addr 172.21.15.42:0/1144140410 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:31:00.735 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.720+0000 7f7bd359e700 1 --2- 172.21.15.42:0/1144140410 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7bd409fae0 0x7f7bd414fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:31:00.736 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.724+0000 7f7bd1d9b700 1 -- 172.21.15.42:0/1144140410 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3378374642 0 0) 0x7f7bd407ef20 con 0x7f7bd408df80 2019-12-05T06:31:00.736 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.724+0000 7f7bd1d9b700 1 -- 172.21.15.42:0/1144140410 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7bc00030b0 con 0x7f7bd408df80 2019-12-05T06:31:00.736 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.724+0000 7f7bd1d9b700 1 -- 172.21.15.42:0/1144140410 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4038081223 0 0) 0x7f7bd40a0710 con 0x7f7bd40a7740 2019-12-05T06:31:00.737 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.724+0000 7f7bd359e700 1 -- 172.21.15.42:0/1144140410 >> v1:172.21.15.42:6789/0 conn(0x7f7bd408df80 legacy=0x7f7bd409f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:00.738 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.724+0000 7f7bd359e700 1 -- 172.21.15.42:0/1144140410 >> v1:172.21.15.40:6789/0 conn(0x7f7bd40a7740 legacy=0x7f7bd407e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:00.738 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.724+0000 7f7bd359e700 1 -- 172.21.15.42:0/1144140410 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bd40a9ba0 con 0x7f7bd409fae0 2019-12-05T06:31:00.738 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.724+0000 7f7bd359e700 1 --2- 172.21.15.42:0/1144140410 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7bd409fae0 0x7f7bd414fff0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f7bcc0032b0 tx=0x7f7bcc00ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:00.738 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.724+0000 7f7bd359e700 1 --2- 172.21.15.42:0/1144140410 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7bd409fae0 0x7f7bd414fff0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f7bcc0032b0 tx=0x7f7bcc00ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:00.739 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.724+0000 7f7bd359e700 1 --2- 172.21.15.42:0/1144140410 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7bd409fae0 0x7f7bd414fff0 secure :-1 s=READY pgs=601 cs=0 l=1 rx=0x7f7bcc0032b0 tx=0x7f7bcc00ff70).ready entity=mon.2 client_cookie=76ef541959391e1d server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:31:00.739 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.724+0000 7f7bd359e700 1 --2- 172.21.15.42:0/1144140410 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7bd409fae0 0x7f7bd414fff0 secure :-1 s=THROTTLE_DONE pgs=601 cs=0 l=1 rx=0x7f7bcc0032b0 tx=0x7f7bcc00ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:00.739 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.724+0000 7f7bd1d9b700 1 -- 172.21.15.42:0/1144140410 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7bcc0109b0 con 0x7f7bd409fae0 2019-12-05T06:31:00.739 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.724+0000 7f7bd359e700 1 --2- 172.21.15.42:0/1144140410 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7bd409fae0 0x7f7bd414fff0 secure :-1 s=THROTTLE_DONE pgs=601 cs=0 l=1 rx=0x7f7bcc0032b0 tx=0x7f7bcc00ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:00.740 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.724+0000 7f7bd359e700 1 --2- 172.21.15.42:0/1144140410 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7bd409fae0 0x7f7bd414fff0 secure :-1 s=THROTTLE_DONE pgs=601 cs=0 l=1 rx=0x7f7bcc0032b0 tx=0x7f7bcc00ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:00.740 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.724+0000 7f7bd1d9b700 1 -- 172.21.15.42:0/1144140410 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bcc008b50 con 0x7f7bd409fae0 2019-12-05T06:31:00.740 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.724+0000 7f7bd1d9b700 1 -- 172.21.15.42:0/1144140410 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7bcc0110c0 con 0x7f7bd409fae0 2019-12-05T06:31:00.741 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.724+0000 7f7bd8c92700 1 -- 172.21.15.42:0/1144140410 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7bd409fae0 msgr2=0x7f7bd414fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:00.741 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.724+0000 7f7bd8c92700 1 --2- 172.21.15.42:0/1144140410 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7bd409fae0 0x7f7bd414fff0 secure :-1 s=READY pgs=601 cs=0 l=1 rx=0x7f7bcc0032b0 tx=0x7f7bcc00ff70).stop 2019-12-05T06:31:00.741 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.724+0000 7f7bd8c92700 1 -- 172.21.15.42:0/1144140410 shutdown_connections 2019-12-05T06:31:00.741 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.724+0000 7f7bd8c92700 1 --2- 172.21.15.42:0/1144140410 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7bd409fae0 0x7f7bd414fff0 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:31:00.742 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.724+0000 7f7bd8c92700 1 -- 172.21.15.42:0/1144140410 shutdown_connections 2019-12-05T06:31:00.742 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.724+0000 7f7bd8c92700 1 -- 172.21.15.42:0/1144140410 wait complete. 2019-12-05T06:31:00.742 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.724+0000 7f7bd8c92700 1 -- 172.21.15.42:0/1144140410 >> 172.21.15.42:0/1144140410 conn(0x7f7bd40aabe0 msgr2=0x7f7bd40a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:31:00.751 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.728+0000 7f7bd8c92700 1 Processor -- start 2019-12-05T06:31:00.751 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.728+0000 7f7bd8c92700 1 -- start start 2019-12-05T06:31:00.751 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.728+0000 7f7bd8c92700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f7bd4117010 0x7f7bd4117440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:31:00.752 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.728+0000 7f7bd8c92700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7bd412a480 0x7f7bd412c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:31:00.752 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.728+0000 7f7bd8c92700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bd40a2ce0 con 0x7f7bd4145140 2019-12-05T06:31:00.752 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.728+0000 7f7bd8c92700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7bd40a3020 con 0x7f7bd4117010 2019-12-05T06:31:00.752 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.728+0000 7f7bd8c92700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bd40a32c0 con 0x7f7bd412a480 2019-12-05T06:31:00.753 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.728+0000 7f7bd259c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f7bd4117010 0x7f7bd4117440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:31:00.753 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.728+0000 7f7bd259c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f7bd4117010 0x7f7bd4117440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:56748/0 (socket says 172.21.15.42:56748) 2019-12-05T06:31:00.753 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.728+0000 7f7bd259c700 1 -- 172.21.15.42:0/3485788535 learned_addr learned my addr 172.21.15.42:0/3485788535 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:31:00.754 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.728+0000 7f7bd359e700 1 --2- 172.21.15.42:0/3485788535 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7bd412a480 0x7f7bd412c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:31:00.754 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.732+0000 7f7bd259c700 1 -- 172.21.15.42:0/3485788535 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7bd412a480 msgr2=0x7f7bd412c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:00.754 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.732+0000 7f7bd259c700 1 --2- 172.21.15.42:0/3485788535 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7bd412a480 0x7f7bd412c8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:31:00.755 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.732+0000 7f7bd259c700 1 -- 172.21.15.42:0/3485788535 >> v1:172.21.15.42:6789/0 conn(0x7f7bd4145140 legacy=0x7f7bd4079230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:00.755 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.732+0000 7f7bd259c700 1 -- 172.21.15.42:0/3485788535 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bd4079a30 con 0x7f7bd4117010 2019-12-05T06:31:00.755 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.732+0000 7f7bd359e700 1 --2- 172.21.15.42:0/3485788535 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7bd412a480 0x7f7bd412c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:31:00.756 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.732+0000 7f7bd259c700 1 --2- 172.21.15.42:0/3485788535 >> v2:172.21.15.42:3300/0 conn(0x7f7bd4117010 0x7f7bd4117440 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f7bd40a3b40 tx=0x7f7bbc012b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:00.757 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.732+0000 7f7bd259c700 1 --2- 172.21.15.42:0/3485788535 >> v2:172.21.15.42:3300/0 conn(0x7f7bd4117010 0x7f7bd4117440 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f7bd40a3b40 tx=0x7f7bbc012b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:00.757 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.732+0000 7f7bd259c700 1 --2- 172.21.15.42:0/3485788535 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7bd4117010 0x7f7bd4117440 secure :-1 s=READY pgs=2744 cs=0 l=1 rx=0x7f7bd40a3b40 tx=0x7f7bbc012b60).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:31:00.758 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.732+0000 7f7bd259c700 1 --2- 172.21.15.42:0/3485788535 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7bd4117010 0x7f7bd4117440 secure :-1 s=THROTTLE_DONE pgs=2744 cs=0 l=1 rx=0x7f7bd40a3b40 tx=0x7f7bbc012b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:00.758 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.732+0000 7f7bcb7fe700 1 -- 172.21.15.42:0/3485788535 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7bbc014940 con 0x7f7bd4117010 2019-12-05T06:31:00.758 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.732+0000 7f7bd259c700 1 --2- 172.21.15.42:0/3485788535 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7bd4117010 0x7f7bd4117440 secure :-1 s=THROTTLE_DONE pgs=2744 cs=0 l=1 rx=0x7f7bd40a3b40 tx=0x7f7bbc012b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:00.758 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.732+0000 7f7bd259c700 1 --2- 172.21.15.42:0/3485788535 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7bd4117010 0x7f7bd4117440 secure :-1 s=THROTTLE_DONE pgs=2744 cs=0 l=1 rx=0x7f7bd40a3b40 tx=0x7f7bbc012b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:00.758 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.732+0000 7f7bcb7fe700 1 -- 172.21.15.42:0/3485788535 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bbc014560 con 0x7f7bd4117010 2019-12-05T06:31:00.758 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.732+0000 7f7bd8c92700 1 -- 172.21.15.42:0/3485788535 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bd4117990 con 0x7f7bd4117010 2019-12-05T06:31:00.759 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.732+0000 7f7bcb7fe700 1 -- 172.21.15.42:0/3485788535 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7bbc005460 con 0x7f7bd4117010 2019-12-05T06:31:00.759 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.732+0000 7f7bd8c92700 1 -- 172.21.15.42:0/3485788535 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bd41448a0 con 0x7f7bd4117010 2019-12-05T06:31:00.759 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.732+0000 7f7bd259c700 1 --2- 172.21.15.42:0/3485788535 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7bd4117010 0x7f7bd4117440 secure :-1 s=THROTTLE_DONE pgs=2744 cs=0 l=1 rx=0x7f7bd40a3b40 tx=0x7f7bbc012b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:00.759 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.732+0000 7f7bcb7fe700 1 -- 172.21.15.42:0/3485788535 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f7bbc005770 con 0x7f7bd4117010 2019-12-05T06:31:00.759 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.732+0000 7f7bd259c700 1 --2- 172.21.15.42:0/3485788535 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7bd4117010 0x7f7bd4117440 secure :-1 s=THROTTLE_DONE pgs=2744 cs=0 l=1 rx=0x7f7bd40a3b40 tx=0x7f7bbc012b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:00.759 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.732+0000 7f7bcb7fe700 1 --2- 172.21.15.42:0/3485788535 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f7bb803adf0 0x7f7bb803d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:31:00.759 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.732+0000 7f7bcb7fe700 1 -- 172.21.15.42:0/3485788535 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 11105+0+0 (secure 0 0 0) 0x7f7bbc04cfc0 con 0x7f7bd4117010 2019-12-05T06:31:00.760 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.732+0000 7f7bd2d9d700 1 --2- 172.21.15.42:0/3485788535 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f7bb803adf0 0x7f7bb803d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:31:00.760 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.732+0000 7f7bd2d9d700 1 --2- 172.21.15.42:0/3485788535 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f7bb803adf0 0x7f7bb803d300 crc :-1 s=READY pgs=1152 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:31:00.760 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.736+0000 7f7bd8c92700 1 -- 172.21.15.42:0/3485788535 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bd405d110 con 0x7f7bd4117010 2019-12-05T06:31:00.760 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.740+0000 7f7bd259c700 1 --2- 172.21.15.42:0/3485788535 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7bd4117010 0x7f7bd4117440 secure :-1 s=THROTTLE_DONE pgs=2744 cs=0 l=1 rx=0x7f7bd40a3b40 tx=0x7f7bbc012b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:00.760 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.740+0000 7f7bcb7fe700 1 -- 172.21.15.42:0/3485788535 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f7bbc034040 con 0x7f7bd4117010 2019-12-05T06:31:00.909 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.904+0000 7f7bd8c92700 1 -- 172.21.15.42:0/3485788535 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7bd405b8c0 con 0x7f7bd4117010 2019-12-05T06:31:00.911 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.904+0000 7f7bd259c700 1 --2- 172.21.15.42:0/3485788535 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7bd4117010 0x7f7bd4117440 secure :-1 s=THROTTLE_DONE pgs=2744 cs=0 l=1 rx=0x7f7bd40a3b40 tx=0x7f7bbc012b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:00.911 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.904+0000 7f7bcb7fe700 1 -- 172.21.15.42:0/3485788535 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 328 v328) v1 ==== 95+0+2117 (secure 0 0 0) 0x7f7bbc0342a0 con 0x7f7bd4117010 2019-12-05T06:31:00.912 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 328 2019-12-05T06:31:00.912 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:31:00.912 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":328,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":11352,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/2055706790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2055706790},{"type":"v1","addr":"172.21.15.40:6833","nonce":2055706790}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":328}],"filesystems":[{"mdsmap":{"epoch":327,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:30:20.005055+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":108,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11265":{"gid":11265,"name":"a","rank":0,"incarnation":324,"state":"up:active","state_seq":28,"addr":"172.21.15.42:6835/1985322937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1985322937},{"type":"v1","addr":"172.21.15.42:6835","nonce":1985322937}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:31:00.914 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.908+0000 7f7bd8c92700 1 -- 172.21.15.42:0/3485788535 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f7bb803adf0 msgr2=0x7f7bb803d300 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:00.914 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.908+0000 7f7bd8c92700 1 --2- 172.21.15.42:0/3485788535 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f7bb803adf0 0x7f7bb803d300 crc :-1 s=READY pgs=1152 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:31:00.914 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.908+0000 7f7bd8c92700 1 -- 172.21.15.42:0/3485788535 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7bd4117010 msgr2=0x7f7bd4117440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:00.914 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.908+0000 7f7bd8c92700 1 --2- 172.21.15.42:0/3485788535 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7bd4117010 0x7f7bd4117440 secure :-1 s=READY pgs=2744 cs=0 l=1 rx=0x7f7bd40a3b40 tx=0x7f7bbc012b60).stop 2019-12-05T06:31:00.914 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.908+0000 7f7bd8c92700 1 -- 172.21.15.42:0/3485788535 shutdown_connections 2019-12-05T06:31:00.915 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.908+0000 7f7bd8c92700 1 --2- 172.21.15.42:0/3485788535 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f7bb803adf0 0x7f7bb803d300 unknown :-1 s=CLOSED pgs=1152 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:31:00.915 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.908+0000 7f7bd8c92700 1 --2- 172.21.15.42:0/3485788535 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7bd412a480 0x7f7bd412c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:31:00.915 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.908+0000 7f7bd8c92700 1 --2- 172.21.15.42:0/3485788535 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7bd4117010 0x7f7bd4117440 unknown :-1 s=CLOSED pgs=2744 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:31:00.915 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.908+0000 7f7bd8c92700 1 -- 172.21.15.42:0/3485788535 shutdown_connections 2019-12-05T06:31:00.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.908+0000 7f7bd8c92700 1 -- 172.21.15.42:0/3485788535 wait complete. 2019-12-05T06:31:00.918 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:00.908+0000 7f7bd8c92700 1 -- 172.21.15.42:0/3485788535 >> 172.21.15.42:0/3485788535 conn(0x7f7bd408ff40 msgr2=0x7f7bd40a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:31:00.923 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:31:02.924 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:31:03.027 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:03.016+0000 7f172db5c700 1 Processor -- start 2019-12-05T06:31:03.027 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:03.016+0000 7f172db5c700 1 -- start start 2019-12-05T06:31:03.027 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:03.016+0000 7f172db5c700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f17280a7730 0x7f172808df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:31:03.028 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:03.016+0000 7f172db5c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f172807e710 0x7f172807eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:31:03.028 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:03.016+0000 7f172db5c700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f172809f2d0 0x7f172809f700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:31:03.028 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:03.016+0000 7f172db5c700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17280a0af0 con 0x7f17280a7730 2019-12-05T06:31:03.028 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:03.016+0000 7f172db5c700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f172808e5a0 con 0x7f172807e710 2019-12-05T06:31:03.028 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:03.016+0000 7f172db5c700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f172808e840 con 0x7f172809f2d0 2019-12-05T06:31:03.028 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:03.016+0000 7f17277fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f172807e710 0x7f172807eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:31:03.028 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:03.016+0000 7f17277fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f172807e710 0x7f172807eb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:56768/0 (socket says 172.21.15.42:56768) 2019-12-05T06:31:03.029 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:03.016+0000 7f17277fe700 1 -- 172.21.15.42:0/3561281026 learned_addr learned my addr 172.21.15.42:0/3561281026 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:31:03.029 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:03.016+0000 7f1727fff700 1 --2- 172.21.15.42:0/3561281026 >> v2:172.21.15.40:3300/0 conn(0x7f17280a7730 0x7f172808df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:31:03.029 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:03.016+0000 7f172c8fa700 1 --2- 172.21.15.42:0/3561281026 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f172809f2d0 0x7f172809f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:31:03.029 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:03.016+0000 7f17277fe700 1 -- 172.21.15.42:0/3561281026 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f172809f2d0 msgr2=0x7f172809f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:31:03.029 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:03.016+0000 7f17277fe700 1 --2- 172.21.15.42:0/3561281026 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f172809f2d0 0x7f172809f700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:31:03.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:03.016+0000 7f17277fe700 1 -- 172.21.15.42:0/3561281026 >> v2:172.21.15.40:3300/0 conn(0x7f17280a7730 msgr2=0x7f172808df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:31:03.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:03.016+0000 7f17277fe700 1 --2- 172.21.15.42:0/3561281026 >> v2:172.21.15.40:3300/0 conn(0x7f17280a7730 0x7f172808df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:31:03.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:03.020+0000 7f17277fe700 1 -- 172.21.15.42:0/3561281026 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f172807f090 con 0x7f172807e710 2019-12-05T06:31:03.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:03.020+0000 7f172c8fa700 1 --2- 172.21.15.42:0/3561281026 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f172809f2d0 0x7f172809f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).send_auth_request state changed! 2019-12-05T06:31:03.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:03.020+0000 7f17277fe700 1 --2- 172.21.15.42:0/3561281026 >> v2:172.21.15.42:3300/0 conn(0x7f172807e710 0x7f172807eb40 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f17280a0f40 tx=0x7f171c00fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:03.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:03.020+0000 7f17277fe700 1 --2- 172.21.15.42:0/3561281026 >> v2:172.21.15.42:3300/0 conn(0x7f172807e710 0x7f172807eb40 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f17280a0f40 tx=0x7f171c00fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:03.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:03.020+0000 7f17277fe700 1 --2- 172.21.15.42:0/3561281026 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f172807e710 0x7f172807eb40 secure :-1 s=READY pgs=2745 cs=0 l=1 rx=0x7f17280a0f40 tx=0x7f171c00fdb0).ready entity=mon.1 client_cookie=fcf5bf3ddfe9d788 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:31:03.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:03.020+0000 7f17277fe700 1 --2- 172.21.15.42:0/3561281026 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f172807e710 0x7f172807eb40 secure :-1 s=THROTTLE_DONE pgs=2745 cs=0 l=1 rx=0x7f17280a0f40 tx=0x7f171c00fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:03.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:03.020+0000 7f1726ffd700 1 -- 172.21.15.42:0/3561281026 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f171c008c90 con 0x7f172807e710 2019-12-05T06:31:03.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:03.020+0000 7f17277fe700 1 --2- 172.21.15.42:0/3561281026 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f172807e710 0x7f172807eb40 secure :-1 s=THROTTLE_DONE pgs=2745 cs=0 l=1 rx=0x7f17280a0f40 tx=0x7f171c00fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:03.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:03.020+0000 7f1726ffd700 1 -- 172.21.15.42:0/3561281026 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f171c0088b0 con 0x7f172807e710 2019-12-05T06:31:03.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:03.020+0000 7f17277fe700 1 --2- 172.21.15.42:0/3561281026 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f172807e710 0x7f172807eb40 secure :-1 s=THROTTLE_DONE pgs=2745 cs=0 l=1 rx=0x7f17280a0f40 tx=0x7f171c00fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:03.032 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:03.020+0000 7f172db5c700 1 -- 172.21.15.42:0/3561281026 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f172807e710 msgr2=0x7f172807eb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:03.032 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:03.020+0000 7f1726ffd700 1 -- 172.21.15.42:0/3561281026 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f171c011b50 con 0x7f172807e710 2019-12-05T06:31:03.032 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:03.020+0000 7f172db5c700 1 --2- 172.21.15.42:0/3561281026 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f172807e710 0x7f172807eb40 secure :-1 s=READY pgs=2745 cs=0 l=1 rx=0x7f17280a0f40 tx=0x7f171c00fdb0).stop 2019-12-05T06:31:03.032 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:03.020+0000 7f172db5c700 1 -- 172.21.15.42:0/3561281026 shutdown_connections 2019-12-05T06:31:03.032 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:03.020+0000 7f172db5c700 1 --2- 172.21.15.42:0/3561281026 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f172809f2d0 0x7f172809f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:31:03.032 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:03.020+0000 7f172db5c700 1 --2- 172.21.15.42:0/3561281026 >> v2:172.21.15.40:3300/0 conn(0x7f17280a7730 0x7f172808df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:31:03.032 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:03.020+0000 7f172db5c700 1 --2- 172.21.15.42:0/3561281026 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f172807e710 0x7f172807eb40 unknown :-1 s=CLOSED pgs=2745 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:31:03.033 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:03.020+0000 7f172db5c700 1 -- 172.21.15.42:0/3561281026 shutdown_connections 2019-12-05T06:31:03.033 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:03.020+0000 7f172db5c700 1 -- 172.21.15.42:0/3561281026 wait complete. 2019-12-05T06:31:03.033 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:03.020+0000 7f172db5c700 1 -- 172.21.15.42:0/3561281026 >> 172.21.15.42:0/3561281026 conn(0x7f17280aabd0 msgr2=0x7f17280a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:31:03.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:03.020+0000 7f172db5c700 1 Processor -- start 2019-12-05T06:31:03.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:03.020+0000 7f172db5c700 1 -- start start 2019-12-05T06:31:03.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:03.024+0000 7f172db5c700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1728117940 0x7f17280a6420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:31:03.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:03.024+0000 7f172db5c700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f1728124df0 0x7f1728125220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:31:03.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:03.024+0000 7f172db5c700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17280a6970 con 0x7f17281121b0 2019-12-05T06:31:03.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:03.024+0000 7f172db5c700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1728114000 con 0x7f1728124df0 2019-12-05T06:31:03.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:03.024+0000 7f172db5c700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17281142a0 con 0x7f1728117940 2019-12-05T06:31:03.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:03.024+0000 7f1727fff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f17281121b0 0x7f1728114540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:56810/0 (socket says 172.21.15.42:56810) 2019-12-05T06:31:03.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:03.024+0000 7f1727fff700 1 -- 172.21.15.42:0/118900190 learned_addr learned my addr 172.21.15.42:0/118900190 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:31:03.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:03.024+0000 7f17277fe700 1 --2- 172.21.15.42:0/118900190 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1728117940 0x7f17280a6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:31:03.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:03.024+0000 7f172c8fa700 1 --2- 172.21.15.42:0/118900190 >> v2:172.21.15.40:3300/0 conn(0x7f1728124df0 0x7f1728125220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:31:03.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:03.024+0000 7f1724ff9700 1 -- 172.21.15.42:0/118900190 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1403671391 0 0) 0x7f17280a6970 con 0x7f17281121b0 2019-12-05T06:31:03.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:03.024+0000 7f1724ff9700 1 -- 172.21.15.42:0/118900190 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17080030b0 con 0x7f17281121b0 2019-12-05T06:31:03.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:03.024+0000 7f17277fe700 1 -- 172.21.15.42:0/118900190 >> v2:172.21.15.40:3300/0 conn(0x7f1728124df0 msgr2=0x7f1728125220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:03.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:03.024+0000 7f17277fe700 1 --2- 172.21.15.42:0/118900190 >> v2:172.21.15.40:3300/0 conn(0x7f1728124df0 0x7f1728125220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:31:03.039 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:03.024+0000 7f17277fe700 1 -- 172.21.15.42:0/118900190 >> v1:172.21.15.42:6789/0 conn(0x7f17281121b0 legacy=0x7f1728114540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:03.039 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:03.024+0000 7f17277fe700 1 -- 172.21.15.42:0/118900190 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1728112b40 con 0x7f1728117940 2019-12-05T06:31:03.039 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:03.024+0000 7f172c8fa700 1 --2- 172.21.15.42:0/118900190 >> v2:172.21.15.40:3300/0 conn(0x7f1728124df0 0x7f1728125220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:31:03.039 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:03.024+0000 7f17277fe700 1 --2- 172.21.15.42:0/118900190 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1728117940 0x7f17280a6420 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f17280a0f40 tx=0x7f171c008e30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:03.039 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:03.024+0000 7f17277fe700 1 --2- 172.21.15.42:0/118900190 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1728117940 0x7f17280a6420 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f17280a0f40 tx=0x7f171c008e30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:03.039 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:03.024+0000 7f17277fe700 1 --2- 172.21.15.42:0/118900190 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1728117940 0x7f17280a6420 secure :-1 s=READY pgs=602 cs=0 l=1 rx=0x7f17280a0f40 tx=0x7f171c008e30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:31:03.040 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:03.024+0000 7f17277fe700 1 --2- 172.21.15.42:0/118900190 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1728117940 0x7f17280a6420 secure :-1 s=THROTTLE_DONE pgs=602 cs=0 l=1 rx=0x7f17280a0f40 tx=0x7f171c008e30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:03.040 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:03.024+0000 7f1724ff9700 1 -- 172.21.15.42:0/118900190 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f171c00f680 con 0x7f1728117940 2019-12-05T06:31:03.040 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:03.024+0000 7f17277fe700 1 --2- 172.21.15.42:0/118900190 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1728117940 0x7f17280a6420 secure :-1 s=THROTTLE_DONE pgs=602 cs=0 l=1 rx=0x7f17280a0f40 tx=0x7f171c008e30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:03.041 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:03.024+0000 7f17277fe700 1 --2- 172.21.15.42:0/118900190 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1728117940 0x7f17280a6420 secure :-1 s=THROTTLE_DONE pgs=602 cs=0 l=1 rx=0x7f17280a0f40 tx=0x7f171c008e30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:03.041 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:03.024+0000 7f1724ff9700 1 -- 172.21.15.42:0/118900190 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f171c011770 con 0x7f1728117940 2019-12-05T06:31:03.041 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:03.024+0000 7f172db5c700 1 -- 172.21.15.42:0/118900190 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1728125770 con 0x7f1728117940 2019-12-05T06:31:03.041 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:03.024+0000 7f1724ff9700 1 -- 172.21.15.42:0/118900190 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f171c005920 con 0x7f1728117940 2019-12-05T06:31:03.041 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:03.024+0000 7f172db5c700 1 -- 172.21.15.42:0/118900190 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f172812c5c0 con 0x7f1728117940 2019-12-05T06:31:03.041 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:03.024+0000 7f17277fe700 1 --2- 172.21.15.42:0/118900190 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1728117940 0x7f17280a6420 secure :-1 s=THROTTLE_DONE pgs=602 cs=0 l=1 rx=0x7f17280a0f40 tx=0x7f171c008e30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:03.041 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:03.024+0000 7f1724ff9700 1 -- 172.21.15.42:0/118900190 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f171c005c30 con 0x7f1728117940 2019-12-05T06:31:03.042 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:03.024+0000 7f17277fe700 1 --2- 172.21.15.42:0/118900190 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1728117940 0x7f17280a6420 secure :-1 s=THROTTLE_DONE pgs=602 cs=0 l=1 rx=0x7f17280a0f40 tx=0x7f171c008e30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:03.042 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:03.028+0000 7f1724ff9700 1 --2- 172.21.15.42:0/118900190 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f170803b2b0 0x7f170803d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:31:03.042 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:03.028+0000 7f1724ff9700 1 -- 172.21.15.42:0/118900190 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 11105+0+0 (secure 0 0 0) 0x7f171c04b620 con 0x7f1728117940 2019-12-05T06:31:03.042 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:03.028+0000 7f1727fff700 1 --2- 172.21.15.42:0/118900190 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f170803b2b0 0x7f170803d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:31:03.042 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:03.028+0000 7f1727fff700 1 --2- 172.21.15.42:0/118900190 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f170803b2b0 0x7f170803d7c0 crc :-1 s=READY pgs=1153 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:31:03.043 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:03.032+0000 7f172db5c700 1 -- 172.21.15.42:0/118900190 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f172805d100 con 0x7f1728117940 2019-12-05T06:31:03.047 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:03.040+0000 7f17277fe700 1 --2- 172.21.15.42:0/118900190 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1728117940 0x7f17280a6420 secure :-1 s=THROTTLE_DONE pgs=602 cs=0 l=1 rx=0x7f17280a0f40 tx=0x7f171c008e30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:03.047 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:03.040+0000 7f1724ff9700 1 -- 172.21.15.42:0/118900190 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f171c0334b0 con 0x7f1728117940 2019-12-05T06:31:03.215 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:03.208+0000 7f172db5c700 1 -- 172.21.15.42:0/118900190 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f172805b8b0 con 0x7f1728117940 2019-12-05T06:31:03.216 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:03.212+0000 7f17277fe700 1 --2- 172.21.15.42:0/118900190 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1728117940 0x7f17280a6420 secure :-1 s=THROTTLE_DONE pgs=602 cs=0 l=1 rx=0x7f17280a0f40 tx=0x7f171c008e30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:03.216 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:03.212+0000 7f1724ff9700 1 -- 172.21.15.42:0/118900190 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 328 v328) v1 ==== 95+0+2117 (secure 0 0 0) 0x7f171c012ce0 con 0x7f1728117940 2019-12-05T06:31:03.218 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 328 2019-12-05T06:31:03.218 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:31:03.219 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":328,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":11352,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/2055706790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2055706790},{"type":"v1","addr":"172.21.15.40:6833","nonce":2055706790}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":328}],"filesystems":[{"mdsmap":{"epoch":327,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:30:20.005055+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":108,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11265":{"gid":11265,"name":"a","rank":0,"incarnation":324,"state":"up:active","state_seq":28,"addr":"172.21.15.42:6835/1985322937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1985322937},{"type":"v1","addr":"172.21.15.42:6835","nonce":1985322937}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:31:03.221 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:03.216+0000 7f172db5c700 1 -- 172.21.15.42:0/118900190 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f170803b2b0 msgr2=0x7f170803d7c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:03.221 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:03.216+0000 7f172db5c700 1 --2- 172.21.15.42:0/118900190 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f170803b2b0 0x7f170803d7c0 crc :-1 s=READY pgs=1153 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:31:03.221 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:03.216+0000 7f172db5c700 1 -- 172.21.15.42:0/118900190 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1728117940 msgr2=0x7f17280a6420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:03.222 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:03.216+0000 7f172db5c700 1 --2- 172.21.15.42:0/118900190 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1728117940 0x7f17280a6420 secure :-1 s=READY pgs=602 cs=0 l=1 rx=0x7f17280a0f40 tx=0x7f171c008e30).stop 2019-12-05T06:31:03.222 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:03.216+0000 7f172db5c700 1 -- 172.21.15.42:0/118900190 shutdown_connections 2019-12-05T06:31:03.222 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:03.216+0000 7f172db5c700 1 --2- 172.21.15.42:0/118900190 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f170803b2b0 0x7f170803d7c0 unknown :-1 s=CLOSED pgs=1153 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:31:03.222 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:03.216+0000 7f172db5c700 1 --2- 172.21.15.42:0/118900190 >> v2:172.21.15.40:3300/0 conn(0x7f1728124df0 0x7f1728125220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:31:03.222 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:03.216+0000 7f172db5c700 1 --2- 172.21.15.42:0/118900190 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1728117940 0x7f17280a6420 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:31:03.222 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:03.216+0000 7f172db5c700 1 -- 172.21.15.42:0/118900190 shutdown_connections 2019-12-05T06:31:03.223 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:03.216+0000 7f172db5c700 1 -- 172.21.15.42:0/118900190 wait complete. 2019-12-05T06:31:03.223 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:03.216+0000 7f172db5c700 1 -- 172.21.15.42:0/118900190 >> 172.21.15.42:0/118900190 conn(0x7f17280aabd0 msgr2=0x7f172809f500 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:31:03.231 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:31:05.232 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:31:05.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:05.328+0000 7f567ecb4700 1 Processor -- start 2019-12-05T06:31:05.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:05.328+0000 7f567ecb4700 1 -- start start 2019-12-05T06:31:05.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:05.328+0000 7f567ecb4700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f56780a7740 0x7f567808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:31:05.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:05.328+0000 7f567ecb4700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f567809fae0 0x7f567807eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:31:05.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:05.332+0000 7f567ecb4700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56780a0710 con 0x7f567807e720 2019-12-05T06:31:05.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:05.332+0000 7f567ecb4700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56780a0b00 con 0x7f56780a7740 2019-12-05T06:31:05.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:05.332+0000 7f567ecb4700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f567808e5b0 con 0x7f567809fae0 2019-12-05T06:31:05.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:05.332+0000 7f567d251700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f56780a7740 0x7f567808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:31:05.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:05.332+0000 7f567d251700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f56780a7740 0x7f567808df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:56782/0 (socket says 172.21.15.42:56782) 2019-12-05T06:31:05.341 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:05.332+0000 7f567d251700 1 -- 172.21.15.42:0/3368275827 learned_addr learned my addr 172.21.15.42:0/3368275827 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:31:05.341 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:05.332+0000 7f567da52700 1 --2- 172.21.15.42:0/3368275827 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f567809fae0 0x7f567807eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:31:05.341 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:05.332+0000 7f567d251700 1 -- 172.21.15.42:0/3368275827 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f567809fae0 msgr2=0x7f567807eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:31:05.341 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:05.332+0000 7f567d251700 1 --2- 172.21.15.42:0/3368275827 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f567809fae0 0x7f567807eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:31:05.341 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:05.332+0000 7f567d251700 1 -- 172.21.15.42:0/3368275827 >> v1:172.21.15.42:6789/0 conn(0x7f567807e720 legacy=0x7f567809f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:05.341 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:05.332+0000 7f567d251700 1 -- 172.21.15.42:0/3368275827 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f567807f0a0 con 0x7f56780a7740 2019-12-05T06:31:05.341 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:05.332+0000 7f567d251700 1 --2- 172.21.15.42:0/3368275827 >> v2:172.21.15.42:3300/0 conn(0x7f56780a7740 0x7f567808df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f5674008b10 tx=0x7f56740100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:05.342 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:05.332+0000 7f567d251700 1 --2- 172.21.15.42:0/3368275827 >> v2:172.21.15.42:3300/0 conn(0x7f56780a7740 0x7f567808df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f5674008b10 tx=0x7f56740100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:05.342 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:05.332+0000 7f567d251700 1 --2- 172.21.15.42:0/3368275827 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f56780a7740 0x7f567808df80 secure :-1 s=READY pgs=2748 cs=0 l=1 rx=0x7f5674008b10 tx=0x7f56740100b0).ready entity=mon.1 client_cookie=6298b205233db1d3 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:31:05.342 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:05.332+0000 7f567d251700 1 --2- 172.21.15.42:0/3368275827 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f56780a7740 0x7f567808df80 secure :-1 s=THROTTLE_DONE pgs=2748 cs=0 l=1 rx=0x7f5674008b10 tx=0x7f56740100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:05.342 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:05.332+0000 7f567d251700 1 --2- 172.21.15.42:0/3368275827 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f56780a7740 0x7f567808df80 secure :-1 s=THROTTLE_DONE pgs=2748 cs=0 l=1 rx=0x7f5674008b10 tx=0x7f56740100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:05.342 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:05.332+0000 7f567d251700 1 --2- 172.21.15.42:0/3368275827 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f56780a7740 0x7f567808df80 secure :-1 s=THROTTLE_DONE pgs=2748 cs=0 l=1 rx=0x7f5674008b10 tx=0x7f56740100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:05.342 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:05.332+0000 7f566ffff700 1 -- 172.21.15.42:0/3368275827 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5674010da0 con 0x7f56780a7740 2019-12-05T06:31:05.343 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:05.332+0000 7f566ffff700 1 -- 172.21.15.42:0/3368275827 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56740109c0 con 0x7f56780a7740 2019-12-05T06:31:05.343 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:05.332+0000 7f566ffff700 1 -- 172.21.15.42:0/3368275827 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f56740114a0 con 0x7f56780a7740 2019-12-05T06:31:05.343 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:05.332+0000 7f567ecb4700 1 -- 172.21.15.42:0/3368275827 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f56780a7740 msgr2=0x7f567808df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:05.343 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:05.332+0000 7f567ecb4700 1 --2- 172.21.15.42:0/3368275827 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f56780a7740 0x7f567808df80 secure :-1 s=READY pgs=2748 cs=0 l=1 rx=0x7f5674008b10 tx=0x7f56740100b0).stop 2019-12-05T06:31:05.343 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:05.332+0000 7f567ecb4700 1 -- 172.21.15.42:0/3368275827 shutdown_connections 2019-12-05T06:31:05.343 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:05.332+0000 7f567ecb4700 1 --2- 172.21.15.42:0/3368275827 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f567809fae0 0x7f567807eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:31:05.343 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:05.332+0000 7f567ecb4700 1 --2- 172.21.15.42:0/3368275827 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f56780a7740 0x7f567808df80 unknown :-1 s=CLOSED pgs=2748 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:31:05.344 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:05.332+0000 7f567ecb4700 1 -- 172.21.15.42:0/3368275827 shutdown_connections 2019-12-05T06:31:05.344 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:05.332+0000 7f567ecb4700 1 -- 172.21.15.42:0/3368275827 wait complete. 2019-12-05T06:31:05.344 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:05.332+0000 7f567ecb4700 1 -- 172.21.15.42:0/3368275827 >> 172.21.15.42:0/3368275827 conn(0x7f56780aabe0 msgr2=0x7f56780a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:31:05.347 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:05.336+0000 7f567ecb4700 1 Processor -- start 2019-12-05T06:31:05.347 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:05.336+0000 7f567ecb4700 1 -- start start 2019-12-05T06:31:05.347 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:05.336+0000 7f567ecb4700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5678121020 0x7f5678118550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:31:05.347 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:05.336+0000 7f567ecb4700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f567808f7a0 con 0x7f5678118aa0 2019-12-05T06:31:05.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:05.336+0000 7f567ecb4700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f567808f080 con 0x7f56781238a0 2019-12-05T06:31:05.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:05.336+0000 7f567ecb4700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5678121450 con 0x7f5678121020 2019-12-05T06:31:05.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:05.336+0000 7f567ca50700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5678121020 0x7f5678118550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:31:05.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:05.336+0000 7f567d251700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f56781238a0 0x7f56781224b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:56826/0 (socket says 172.21.15.42:56826) 2019-12-05T06:31:05.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:05.336+0000 7f567d251700 1 -- 172.21.15.42:0/3177536451 learned_addr learned my addr 172.21.15.42:0/3177536451 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:31:05.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:05.336+0000 7f567ca50700 1 -- 172.21.15.42:0/3177536451 >> v1:172.21.15.42:6789/0 conn(0x7f56781238a0 legacy=0x7f56781224b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:05.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:05.336+0000 7f567ca50700 1 -- 172.21.15.42:0/3177536451 >> v1:172.21.15.40:6789/0 conn(0x7f5678118aa0 legacy=0x7f5678138bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:05.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:05.336+0000 7f566dffb700 1 -- 172.21.15.42:0/3177536451 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3278639024 0 0) 0x7f567808f080 con 0x7f56781238a0 2019-12-05T06:31:05.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:05.336+0000 7f567ca50700 1 -- 172.21.15.42:0/3177536451 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5678122cb0 con 0x7f5678121020 2019-12-05T06:31:05.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:05.336+0000 7f567ca50700 1 --2- 172.21.15.42:0/3177536451 >> v2:172.21.15.42:3300/0 conn(0x7f5678121020 0x7f5678118550 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f5668001430 tx=0x7f566800c320).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:05.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:05.336+0000 7f567ca50700 1 --2- 172.21.15.42:0/3177536451 >> v2:172.21.15.42:3300/0 conn(0x7f5678121020 0x7f5678118550 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f5668001430 tx=0x7f566800c320).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:05.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:05.336+0000 7f567ca50700 1 --2- 172.21.15.42:0/3177536451 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5678121020 0x7f5678118550 secure :-1 s=READY pgs=2750 cs=0 l=1 rx=0x7f5668001430 tx=0x7f566800c320).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:31:05.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:05.336+0000 7f567ca50700 1 --2- 172.21.15.42:0/3177536451 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5678121020 0x7f5678118550 secure :-1 s=THROTTLE_DONE pgs=2750 cs=0 l=1 rx=0x7f5668001430 tx=0x7f566800c320).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:05.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:05.336+0000 7f566dffb700 1 -- 172.21.15.42:0/3177536451 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f566800cb70 con 0x7f5678121020 2019-12-05T06:31:05.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:05.336+0000 7f567ca50700 1 --2- 172.21.15.42:0/3177536451 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5678121020 0x7f5678118550 secure :-1 s=THROTTLE_DONE pgs=2750 cs=0 l=1 rx=0x7f5668001430 tx=0x7f566800c320).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:05.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:05.336+0000 7f566dffb700 1 -- 172.21.15.42:0/3177536451 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f566800c790 con 0x7f5678121020 2019-12-05T06:31:05.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:05.336+0000 7f567ca50700 1 --2- 172.21.15.42:0/3177536451 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5678121020 0x7f5678118550 secure :-1 s=THROTTLE_DONE pgs=2750 cs=0 l=1 rx=0x7f5668001430 tx=0x7f566800c320).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:05.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:05.336+0000 7f567ecb4700 1 -- 172.21.15.42:0/3177536451 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5678131770 con 0x7f5678121020 2019-12-05T06:31:05.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:05.336+0000 7f566dffb700 1 -- 172.21.15.42:0/3177536451 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5668011070 con 0x7f5678121020 2019-12-05T06:31:05.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:05.336+0000 7f567ecb4700 1 -- 172.21.15.42:0/3177536451 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f567813a3d0 con 0x7f5678121020 2019-12-05T06:31:05.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:05.340+0000 7f567ca50700 1 --2- 172.21.15.42:0/3177536451 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5678121020 0x7f5678118550 secure :-1 s=THROTTLE_DONE pgs=2750 cs=0 l=1 rx=0x7f5668001430 tx=0x7f566800c320).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:05.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:05.340+0000 7f566dffb700 1 -- 172.21.15.42:0/3177536451 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f5668011350 con 0x7f5678121020 2019-12-05T06:31:05.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:05.340+0000 7f567ca50700 1 --2- 172.21.15.42:0/3177536451 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5678121020 0x7f5678118550 secure :-1 s=THROTTLE_DONE pgs=2750 cs=0 l=1 rx=0x7f5668001430 tx=0x7f566800c320).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:05.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:05.340+0000 7f566dffb700 1 --2- 172.21.15.42:0/3177536451 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f565803ac50 0x7f565803d160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:31:05.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:05.340+0000 7f566dffb700 1 -- 172.21.15.42:0/3177536451 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 11105+0+0 (secure 0 0 0) 0x7f566804b2a0 con 0x7f5678121020 2019-12-05T06:31:05.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:05.340+0000 7f567d251700 1 --2- 172.21.15.42:0/3177536451 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f565803ac50 0x7f565803d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:31:05.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:05.340+0000 7f567d251700 1 --2- 172.21.15.42:0/3177536451 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f565803ac50 0x7f565803d160 crc :-1 s=READY pgs=1154 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:31:05.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:05.344+0000 7f567ecb4700 1 -- 172.21.15.42:0/3177536451 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f567805d110 con 0x7f5678121020 2019-12-05T06:31:05.359 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:05.352+0000 7f567ca50700 1 --2- 172.21.15.42:0/3177536451 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5678121020 0x7f5678118550 secure :-1 s=THROTTLE_DONE pgs=2750 cs=0 l=1 rx=0x7f5668001430 tx=0x7f566800c320).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:05.360 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:05.352+0000 7f566dffb700 1 -- 172.21.15.42:0/3177536451 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f5668032300 con 0x7f5678121020 2019-12-05T06:31:05.532 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:05.524+0000 7f567ecb4700 1 -- 172.21.15.42:0/3177536451 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f567805b8c0 con 0x7f5678121020 2019-12-05T06:31:05.533 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:05.528+0000 7f567ca50700 1 --2- 172.21.15.42:0/3177536451 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5678121020 0x7f5678118550 secure :-1 s=THROTTLE_DONE pgs=2750 cs=0 l=1 rx=0x7f5668001430 tx=0x7f566800c320).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:05.533 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:05.528+0000 7f566dffb700 1 -- 172.21.15.42:0/3177536451 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 328 v328) v1 ==== 95+0+2117 (secure 0 0 0) 0x7f5668032560 con 0x7f5678121020 2019-12-05T06:31:05.533 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 328 2019-12-05T06:31:05.537 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:31:05.537 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":328,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":11352,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/2055706790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2055706790},{"type":"v1","addr":"172.21.15.40:6833","nonce":2055706790}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":328}],"filesystems":[{"mdsmap":{"epoch":327,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:30:20.005055+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":108,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11265":{"gid":11265,"name":"a","rank":0,"incarnation":324,"state":"up:active","state_seq":28,"addr":"172.21.15.42:6835/1985322937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1985322937},{"type":"v1","addr":"172.21.15.42:6835","nonce":1985322937}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:31:05.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:05.528+0000 7f567ecb4700 1 -- 172.21.15.42:0/3177536451 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f565803ac50 msgr2=0x7f565803d160 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:05.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:05.528+0000 7f567ecb4700 1 --2- 172.21.15.42:0/3177536451 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f565803ac50 0x7f565803d160 crc :-1 s=READY pgs=1154 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:31:05.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:05.528+0000 7f567ecb4700 1 -- 172.21.15.42:0/3177536451 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5678121020 msgr2=0x7f5678118550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:05.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:05.528+0000 7f567ecb4700 1 --2- 172.21.15.42:0/3177536451 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5678121020 0x7f5678118550 secure :-1 s=READY pgs=2750 cs=0 l=1 rx=0x7f5668001430 tx=0x7f566800c320).stop 2019-12-05T06:31:05.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:05.528+0000 7f567ecb4700 1 -- 172.21.15.42:0/3177536451 shutdown_connections 2019-12-05T06:31:05.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:05.528+0000 7f567ecb4700 1 --2- 172.21.15.42:0/3177536451 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f565803ac50 0x7f565803d160 unknown :-1 s=CLOSED pgs=1154 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:31:05.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:05.528+0000 7f567ecb4700 1 --2- 172.21.15.42:0/3177536451 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5678121020 0x7f5678118550 unknown :-1 s=CLOSED pgs=2750 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:31:05.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:05.528+0000 7f567ecb4700 1 -- 172.21.15.42:0/3177536451 shutdown_connections 2019-12-05T06:31:05.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:05.528+0000 7f567ecb4700 1 -- 172.21.15.42:0/3177536451 wait complete. 2019-12-05T06:31:05.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:05.528+0000 7f567ecb4700 1 -- 172.21.15.42:0/3177536451 >> 172.21.15.42:0/3177536451 conn(0x7f56780a9640 msgr2=0x7f567809f410 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:31:05.546 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:31:07.547 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:31:07.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:07.636+0000 7fd351be6700 1 Processor -- start 2019-12-05T06:31:07.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:07.636+0000 7fd351be6700 1 -- start start 2019-12-05T06:31:07.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:07.640+0000 7fd351be6700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd34c08df70 0x7fd34c09f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:31:07.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:07.640+0000 7fd351be6700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd34c09f820 0x7fd34c14efd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:31:07.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:07.640+0000 7fd351be6700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd34c09fc50 con 0x7fd34c0a7730 2019-12-05T06:31:07.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:07.640+0000 7fd351be6700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd34c0a0af0 con 0x7fd34c09f820 2019-12-05T06:31:07.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:07.640+0000 7fd351be6700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd34c08e5a0 con 0x7fd34c08df70 2019-12-05T06:31:07.650 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:07.640+0000 7fd34b7fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd34c08df70 0x7fd34c09f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:31:07.650 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:07.640+0000 7fd34b7fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd34c08df70 0x7fd34c09f2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:56804/0 (socket says 172.21.15.42:56804) 2019-12-05T06:31:07.650 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:07.640+0000 7fd34b7fe700 1 -- 172.21.15.42:0/618205255 learned_addr learned my addr 172.21.15.42:0/618205255 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:31:07.650 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:07.640+0000 7fd350984700 1 --2- 172.21.15.42:0/618205255 >> v2:172.21.15.40:3300/0 conn(0x7fd34c09f820 0x7fd34c14efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:31:07.650 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:07.640+0000 7fd34b7fe700 1 -- 172.21.15.42:0/618205255 >> v2:172.21.15.40:3300/0 conn(0x7fd34c09f820 msgr2=0x7fd34c14efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:31:07.650 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:07.640+0000 7fd34b7fe700 1 --2- 172.21.15.42:0/618205255 >> v2:172.21.15.40:3300/0 conn(0x7fd34c09f820 0x7fd34c14efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:31:07.650 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:07.640+0000 7fd34b7fe700 1 -- 172.21.15.42:0/618205255 >> v1:172.21.15.40:6789/0 conn(0x7fd34c0a7730 legacy=0x7fd34c07e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:07.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:07.640+0000 7fd34b7fe700 1 -- 172.21.15.42:0/618205255 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd34c0a0700 con 0x7fd34c08df70 2019-12-05T06:31:07.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:07.640+0000 7fd34b7fe700 1 --2- 172.21.15.42:0/618205255 >> v2:172.21.15.42:3300/0 conn(0x7fd34c08df70 0x7fd34c09f2d0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fd3400032b0 tx=0x7fd3400100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:07.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:07.640+0000 7fd34b7fe700 1 --2- 172.21.15.42:0/618205255 >> v2:172.21.15.42:3300/0 conn(0x7fd34c08df70 0x7fd34c09f2d0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fd3400032b0 tx=0x7fd3400100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:07.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:07.640+0000 7fd34b7fe700 1 --2- 172.21.15.42:0/618205255 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd34c08df70 0x7fd34c09f2d0 secure :-1 s=READY pgs=2751 cs=0 l=1 rx=0x7fd3400032b0 tx=0x7fd3400100b0).ready entity=mon.1 client_cookie=d41afa2f1dd7d5e server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:31:07.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:07.640+0000 7fd34b7fe700 1 --2- 172.21.15.42:0/618205255 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd34c08df70 0x7fd34c09f2d0 secure :-1 s=THROTTLE_DONE pgs=2751 cs=0 l=1 rx=0x7fd3400032b0 tx=0x7fd3400100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:07.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:07.640+0000 7fd34affd700 1 -- 172.21.15.42:0/618205255 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd340010d60 con 0x7fd34c08df70 2019-12-05T06:31:07.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:07.640+0000 7fd34b7fe700 1 --2- 172.21.15.42:0/618205255 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd34c08df70 0x7fd34c09f2d0 secure :-1 s=THROTTLE_DONE pgs=2751 cs=0 l=1 rx=0x7fd3400032b0 tx=0x7fd3400100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:07.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:07.640+0000 7fd34affd700 1 -- 172.21.15.42:0/618205255 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd340010980 con 0x7fd34c08df70 2019-12-05T06:31:07.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:07.640+0000 7fd34b7fe700 1 --2- 172.21.15.42:0/618205255 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd34c08df70 0x7fd34c09f2d0 secure :-1 s=THROTTLE_DONE pgs=2751 cs=0 l=1 rx=0x7fd3400032b0 tx=0x7fd3400100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:07.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:07.640+0000 7fd34affd700 1 -- 172.21.15.42:0/618205255 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd3400114a0 con 0x7fd34c08df70 2019-12-05T06:31:07.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:07.640+0000 7fd351be6700 1 -- 172.21.15.42:0/618205255 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd34c08df70 msgr2=0x7fd34c09f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:07.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:07.640+0000 7fd351be6700 1 --2- 172.21.15.42:0/618205255 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd34c08df70 0x7fd34c09f2d0 secure :-1 s=READY pgs=2751 cs=0 l=1 rx=0x7fd3400032b0 tx=0x7fd3400100b0).stop 2019-12-05T06:31:07.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:07.640+0000 7fd351be6700 1 -- 172.21.15.42:0/618205255 shutdown_connections 2019-12-05T06:31:07.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:07.640+0000 7fd351be6700 1 --2- 172.21.15.42:0/618205255 >> v2:172.21.15.40:3300/0 conn(0x7fd34c09f820 0x7fd34c14efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:31:07.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:07.640+0000 7fd351be6700 1 --2- 172.21.15.42:0/618205255 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd34c08df70 0x7fd34c09f2d0 unknown :-1 s=CLOSED pgs=2751 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:31:07.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:07.640+0000 7fd351be6700 1 -- 172.21.15.42:0/618205255 shutdown_connections 2019-12-05T06:31:07.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:07.640+0000 7fd351be6700 1 -- 172.21.15.42:0/618205255 wait complete. 2019-12-05T06:31:07.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:07.640+0000 7fd351be6700 1 -- 172.21.15.42:0/618205255 >> 172.21.15.42:0/618205255 conn(0x7fd34c0aabd0 msgr2=0x7fd34c0a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:31:07.657 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:07.644+0000 7fd351be6700 1 Processor -- start 2019-12-05T06:31:07.657 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:07.644+0000 7fd351be6700 1 -- start start 2019-12-05T06:31:07.657 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:07.644+0000 7fd351be6700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd34c0a4c80 0x7fd34c11a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:31:07.658 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:07.648+0000 7fd351be6700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd34c11af20 0x7fd34c0762e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:31:07.658 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:07.648+0000 7fd351be6700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd34c11b350 con 0x7fd34c08f3f0 2019-12-05T06:31:07.658 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:07.648+0000 7fd351be6700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd34c0a51e0 con 0x7fd34c0a4c80 2019-12-05T06:31:07.658 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:07.648+0000 7fd351be6700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd34c0a5480 con 0x7fd34c11af20 2019-12-05T06:31:07.658 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:07.648+0000 7fd34bfff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fd34c08f3f0 0x7fd34c118060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:56846/0 (socket says 172.21.15.42:56846) 2019-12-05T06:31:07.659 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:07.648+0000 7fd34b7fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd34c0a4c80 0x7fd34c11a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:31:07.659 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:07.648+0000 7fd34bfff700 1 -- 172.21.15.42:0/233884738 learned_addr learned my addr 172.21.15.42:0/233884738 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:31:07.659 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:07.648+0000 7fd350984700 1 --2- 172.21.15.42:0/233884738 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd34c11af20 0x7fd34c0762e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:31:07.659 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:07.648+0000 7fd34b7fe700 1 -- 172.21.15.42:0/233884738 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd34c11af20 msgr2=0x7fd34c0762e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:07.659 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:07.648+0000 7fd34b7fe700 1 --2- 172.21.15.42:0/233884738 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd34c11af20 0x7fd34c0762e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:31:07.659 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:07.648+0000 7fd34b7fe700 1 -- 172.21.15.42:0/233884738 >> v1:172.21.15.42:6789/0 conn(0x7fd34c08f3f0 legacy=0x7fd34c118060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:07.660 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:07.648+0000 7fd348ff9700 1 -- 172.21.15.42:0/233884738 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1562484122 0 0) 0x7fd34c11b350 con 0x7fd34c08f3f0 2019-12-05T06:31:07.660 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:07.648+0000 7fd34b7fe700 1 -- 172.21.15.42:0/233884738 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd34c077840 con 0x7fd34c0a4c80 2019-12-05T06:31:07.660 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:07.648+0000 7fd34b7fe700 1 --2- 172.21.15.42:0/233884738 >> v2:172.21.15.42:3300/0 conn(0x7fd34c0a4c80 0x7fd34c11a9d0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fd34c119f30 tx=0x7fd340010f00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:07.660 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:07.648+0000 7fd34b7fe700 1 --2- 172.21.15.42:0/233884738 >> v2:172.21.15.42:3300/0 conn(0x7fd34c0a4c80 0x7fd34c11a9d0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fd34c119f30 tx=0x7fd340010f00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:07.660 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:07.648+0000 7fd34b7fe700 1 --2- 172.21.15.42:0/233884738 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd34c0a4c80 0x7fd34c11a9d0 secure :-1 s=READY pgs=2753 cs=0 l=1 rx=0x7fd34c119f30 tx=0x7fd340010f00).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:31:07.660 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:07.648+0000 7fd34b7fe700 1 --2- 172.21.15.42:0/233884738 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd34c0a4c80 0x7fd34c11a9d0 secure :-1 s=THROTTLE_DONE pgs=2753 cs=0 l=1 rx=0x7fd34c119f30 tx=0x7fd340010f00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:07.661 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:07.648+0000 7fd348ff9700 1 -- 172.21.15.42:0/233884738 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd340011c40 con 0x7fd34c0a4c80 2019-12-05T06:31:07.661 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:07.648+0000 7fd34b7fe700 1 --2- 172.21.15.42:0/233884738 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd34c0a4c80 0x7fd34c11a9d0 secure :-1 s=THROTTLE_DONE pgs=2753 cs=0 l=1 rx=0x7fd34c119f30 tx=0x7fd340010f00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:07.661 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:07.648+0000 7fd348ff9700 1 -- 172.21.15.42:0/233884738 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3400110c0 con 0x7fd34c0a4c80 2019-12-05T06:31:07.661 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:07.648+0000 7fd34b7fe700 1 --2- 172.21.15.42:0/233884738 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd34c0a4c80 0x7fd34c11a9d0 secure :-1 s=THROTTLE_DONE pgs=2753 cs=0 l=1 rx=0x7fd34c119f30 tx=0x7fd340010f00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:07.661 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:07.648+0000 7fd351be6700 1 -- 172.21.15.42:0/233884738 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd34c077b70 con 0x7fd34c0a4c80 2019-12-05T06:31:07.662 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:07.648+0000 7fd348ff9700 1 -- 172.21.15.42:0/233884738 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd340005ca0 con 0x7fd34c0a4c80 2019-12-05T06:31:07.662 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:07.648+0000 7fd351be6700 1 -- 172.21.15.42:0/233884738 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd34c078120 con 0x7fd34c0a4c80 2019-12-05T06:31:07.662 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:07.648+0000 7fd34b7fe700 1 --2- 172.21.15.42:0/233884738 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd34c0a4c80 0x7fd34c11a9d0 secure :-1 s=THROTTLE_DONE pgs=2753 cs=0 l=1 rx=0x7fd34c119f30 tx=0x7fd340010f00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:07.662 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:07.648+0000 7fd348ff9700 1 -- 172.21.15.42:0/233884738 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fd34000f280 con 0x7fd34c0a4c80 2019-12-05T06:31:07.662 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:07.648+0000 7fd34b7fe700 1 --2- 172.21.15.42:0/233884738 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd34c0a4c80 0x7fd34c11a9d0 secure :-1 s=THROTTLE_DONE pgs=2753 cs=0 l=1 rx=0x7fd34c119f30 tx=0x7fd340010f00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:07.662 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:07.652+0000 7fd348ff9700 1 --2- 172.21.15.42:0/233884738 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd32c03ac10 0x7fd32c03d120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:31:07.662 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:07.652+0000 7fd348ff9700 1 -- 172.21.15.42:0/233884738 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 11105+0+0 (secure 0 0 0) 0x7fd34004b630 con 0x7fd34c0a4c80 2019-12-05T06:31:07.663 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:07.652+0000 7fd34bfff700 1 --2- 172.21.15.42:0/233884738 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd32c03ac10 0x7fd32c03d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:31:07.663 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:07.652+0000 7fd34bfff700 1 --2- 172.21.15.42:0/233884738 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd32c03ac10 0x7fd32c03d120 crc :-1 s=READY pgs=1155 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:31:07.663 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:07.652+0000 7fd351be6700 1 -- 172.21.15.42:0/233884738 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd34c05d100 con 0x7fd34c0a4c80 2019-12-05T06:31:07.667 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:07.660+0000 7fd34b7fe700 1 --2- 172.21.15.42:0/233884738 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd34c0a4c80 0x7fd34c11a9d0 secure :-1 s=THROTTLE_DONE pgs=2753 cs=0 l=1 rx=0x7fd34c119f30 tx=0x7fd340010f00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:07.667 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:07.660+0000 7fd348ff9700 1 -- 172.21.15.42:0/233884738 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fd3400334a0 con 0x7fd34c0a4c80 2019-12-05T06:31:07.830 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:07.824+0000 7fd351be6700 1 -- 172.21.15.42:0/233884738 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd34c05b8b0 con 0x7fd34c0a4c80 2019-12-05T06:31:07.831 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:07.824+0000 7fd34b7fe700 1 --2- 172.21.15.42:0/233884738 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd34c0a4c80 0x7fd34c11a9d0 secure :-1 s=THROTTLE_DONE pgs=2753 cs=0 l=1 rx=0x7fd34c119f30 tx=0x7fd340010f00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:07.831 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:07.824+0000 7fd348ff9700 1 -- 172.21.15.42:0/233884738 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 328 v328) v1 ==== 95+0+2117 (secure 0 0 0) 0x7fd3400331b0 con 0x7fd34c0a4c80 2019-12-05T06:31:07.831 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 328 2019-12-05T06:31:07.831 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:31:07.831 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":328,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":11352,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/2055706790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2055706790},{"type":"v1","addr":"172.21.15.40:6833","nonce":2055706790}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":328}],"filesystems":[{"mdsmap":{"epoch":327,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:30:20.005055+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":108,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11265":{"gid":11265,"name":"a","rank":0,"incarnation":324,"state":"up:active","state_seq":28,"addr":"172.21.15.42:6835/1985322937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1985322937},{"type":"v1","addr":"172.21.15.42:6835","nonce":1985322937}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:31:07.833 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:07.828+0000 7fd351be6700 1 -- 172.21.15.42:0/233884738 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd32c03ac10 msgr2=0x7fd32c03d120 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:07.834 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:07.828+0000 7fd351be6700 1 --2- 172.21.15.42:0/233884738 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd32c03ac10 0x7fd32c03d120 crc :-1 s=READY pgs=1155 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:31:07.834 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:07.828+0000 7fd351be6700 1 -- 172.21.15.42:0/233884738 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd34c0a4c80 msgr2=0x7fd34c11a9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:07.834 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:07.828+0000 7fd351be6700 1 --2- 172.21.15.42:0/233884738 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd34c0a4c80 0x7fd34c11a9d0 secure :-1 s=READY pgs=2753 cs=0 l=1 rx=0x7fd34c119f30 tx=0x7fd340010f00).stop 2019-12-05T06:31:07.834 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:07.828+0000 7fd351be6700 1 -- 172.21.15.42:0/233884738 shutdown_connections 2019-12-05T06:31:07.834 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:07.828+0000 7fd351be6700 1 --2- 172.21.15.42:0/233884738 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd32c03ac10 0x7fd32c03d120 unknown :-1 s=CLOSED pgs=1155 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:31:07.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:07.828+0000 7fd351be6700 1 --2- 172.21.15.42:0/233884738 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd34c11af20 0x7fd34c0762e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:31:07.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:07.828+0000 7fd351be6700 1 --2- 172.21.15.42:0/233884738 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd34c0a4c80 0x7fd34c11a9d0 unknown :-1 s=CLOSED pgs=2753 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:31:07.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:07.828+0000 7fd351be6700 1 -- 172.21.15.42:0/233884738 shutdown_connections 2019-12-05T06:31:07.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:07.828+0000 7fd351be6700 1 -- 172.21.15.42:0/233884738 wait complete. 2019-12-05T06:31:07.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:07.828+0000 7fd351be6700 1 -- 172.21.15.42:0/233884738 >> 172.21.15.42:0/233884738 conn(0x7fd34c0a7f80 msgr2=0x7fd34c11bee0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:31:07.843 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:31:09.847 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:31:09.957 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:09.948+0000 7f4e1e1bb700 1 Processor -- start 2019-12-05T06:31:09.958 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:09.948+0000 7f4e1e1bb700 1 -- start start 2019-12-05T06:31:09.958 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:09.948+0000 7f4e1e1bb700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f4e180a7740 0x7f4e1808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:31:09.958 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:09.948+0000 7f4e1e1bb700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e180a0710 con 0x7f4e1809fae0 2019-12-05T06:31:09.958 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:09.948+0000 7f4e1e1bb700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e180a9ba0 con 0x7f4e1807e720 2019-12-05T06:31:09.958 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:09.948+0000 7f4e1e1bb700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e180a0b00 con 0x7f4e180a7740 2019-12-05T06:31:09.958 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:09.948+0000 7f4e17fff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f4e180a7740 0x7f4e1808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:31:09.959 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:09.948+0000 7f4e17fff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f4e180a7740 0x7f4e1808df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:56818/0 (socket says 172.21.15.42:56818) 2019-12-05T06:31:09.959 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:09.948+0000 7f4e17fff700 1 -- 172.21.15.42:0/3651252765 learned_addr learned my addr 172.21.15.42:0/3651252765 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:31:09.959 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:09.948+0000 7f4e177fe700 1 --1- 172.21.15.42:0/3651252765 >> v1:172.21.15.42:6789/0 conn(0x7f4e1807e720 0x7f4e1809f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:56858/0 (socket says 172.21.15.42:56858) 2019-12-05T06:31:09.959 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:09.948+0000 7f4e17fff700 1 -- 172.21.15.42:0/3651252765 >> v1:172.21.15.42:6789/0 conn(0x7f4e1807e720 legacy=0x7f4e1809f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:09.959 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:09.948+0000 7f4e17fff700 1 -- 172.21.15.42:0/3651252765 >> v1:172.21.15.40:6789/0 conn(0x7f4e1809fae0 legacy=0x7f4e1814efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:09.959 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:09.948+0000 7f4e17fff700 1 -- 172.21.15.42:0/3651252765 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e1807eb50 con 0x7f4e180a7740 2019-12-05T06:31:09.959 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:09.948+0000 7f4e17fff700 1 --2- 172.21.15.42:0/3651252765 >> v2:172.21.15.42:3300/0 conn(0x7f4e180a7740 0x7f4e1808df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f4e08008790 tx=0x7f4e0800fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:09.960 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:09.948+0000 7f4e17fff700 1 --2- 172.21.15.42:0/3651252765 >> v2:172.21.15.42:3300/0 conn(0x7f4e180a7740 0x7f4e1808df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f4e08008790 tx=0x7f4e0800fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:09.960 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:09.948+0000 7f4e17fff700 1 --2- 172.21.15.42:0/3651252765 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4e180a7740 0x7f4e1808df80 secure :-1 s=READY pgs=2755 cs=0 l=1 rx=0x7f4e08008790 tx=0x7f4e0800fcb0).ready entity=mon.1 client_cookie=fefb9a7274c71448 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:31:09.960 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:09.948+0000 7f4e17fff700 1 --2- 172.21.15.42:0/3651252765 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4e180a7740 0x7f4e1808df80 secure :-1 s=THROTTLE_DONE pgs=2755 cs=0 l=1 rx=0x7f4e08008790 tx=0x7f4e0800fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:09.960 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:09.948+0000 7f4e17fff700 1 --2- 172.21.15.42:0/3651252765 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4e180a7740 0x7f4e1808df80 secure :-1 s=THROTTLE_DONE pgs=2755 cs=0 l=1 rx=0x7f4e08008790 tx=0x7f4e0800fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:09.960 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:09.948+0000 7f4e16ffd700 1 -- 172.21.15.42:0/3651252765 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4e08011520 con 0x7f4e180a7740 2019-12-05T06:31:09.961 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:09.948+0000 7f4e17fff700 1 --2- 172.21.15.42:0/3651252765 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4e180a7740 0x7f4e1808df80 secure :-1 s=THROTTLE_DONE pgs=2755 cs=0 l=1 rx=0x7f4e08008790 tx=0x7f4e0800fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:09.961 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:09.948+0000 7f4e16ffd700 1 -- 172.21.15.42:0/3651252765 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e08008ca0 con 0x7f4e180a7740 2019-12-05T06:31:09.962 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:09.948+0000 7f4e16ffd700 1 -- 172.21.15.42:0/3651252765 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4e08011b00 con 0x7f4e180a7740 2019-12-05T06:31:09.962 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:09.948+0000 7f4e1e1bb700 1 -- 172.21.15.42:0/3651252765 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4e180a7740 msgr2=0x7f4e1808df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:09.962 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:09.948+0000 7f4e1e1bb700 1 --2- 172.21.15.42:0/3651252765 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4e180a7740 0x7f4e1808df80 secure :-1 s=READY pgs=2755 cs=0 l=1 rx=0x7f4e08008790 tx=0x7f4e0800fcb0).stop 2019-12-05T06:31:09.962 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:09.948+0000 7f4e1e1bb700 1 -- 172.21.15.42:0/3651252765 shutdown_connections 2019-12-05T06:31:09.962 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:09.948+0000 7f4e1e1bb700 1 --2- 172.21.15.42:0/3651252765 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4e180a7740 0x7f4e1808df80 unknown :-1 s=CLOSED pgs=2755 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:31:09.962 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:09.952+0000 7f4e1e1bb700 1 -- 172.21.15.42:0/3651252765 shutdown_connections 2019-12-05T06:31:09.963 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:09.952+0000 7f4e1e1bb700 1 -- 172.21.15.42:0/3651252765 wait complete. 2019-12-05T06:31:09.963 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:09.952+0000 7f4e1e1bb700 1 -- 172.21.15.42:0/3651252765 >> 172.21.15.42:0/3651252765 conn(0x7f4e180aabe0 msgr2=0x7f4e180a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:31:09.966 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:09.952+0000 7f4e1e1bb700 1 Processor -- start 2019-12-05T06:31:09.966 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:09.952+0000 7f4e1e1bb700 1 -- start start 2019-12-05T06:31:09.966 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:09.956+0000 7f4e1e1bb700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4e181476e0 0x7f4e181450b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:31:09.966 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:09.956+0000 7f4e1e1bb700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f4e18148460 0x7f4e18125ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:31:09.966 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:09.956+0000 7f4e1e1bb700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e18126430 con 0x7f4e18118ee0 2019-12-05T06:31:09.967 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:09.956+0000 7f4e1e1bb700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e181427c0 con 0x7f4e18148460 2019-12-05T06:31:09.967 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:09.956+0000 7f4e1e1bb700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e18147c80 con 0x7f4e181476e0 2019-12-05T06:31:09.967 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:09.956+0000 7f4e1cf59700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f4e18148460 0x7f4e18125ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:31:09.967 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:09.956+0000 7f4e17fff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f4e18118ee0 0x7f4e18142b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:56862/0 (socket says 172.21.15.42:56862) 2019-12-05T06:31:09.967 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:09.956+0000 7f4e1cf59700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f4e18148460 0x7f4e18125ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:56826/0 (socket says 172.21.15.42:56826) 2019-12-05T06:31:09.967 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:09.956+0000 7f4e17fff700 1 -- 172.21.15.42:0/3391415941 learned_addr learned my addr 172.21.15.42:0/3391415941 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:31:09.967 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:09.956+0000 7f4e177fe700 1 --2- 172.21.15.42:0/3391415941 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4e181476e0 0x7f4e181450b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:31:09.968 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:09.956+0000 7f4e1cf59700 1 -- 172.21.15.42:0/3391415941 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4e181476e0 msgr2=0x7f4e181450b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:09.968 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:09.956+0000 7f4e1cf59700 1 --2- 172.21.15.42:0/3391415941 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4e181476e0 0x7f4e181450b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:31:09.968 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:09.956+0000 7f4e14ff9700 1 -- 172.21.15.42:0/3391415941 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3393402535 0 0) 0x7f4e18126430 con 0x7f4e18118ee0 2019-12-05T06:31:09.968 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:09.956+0000 7f4e1cf59700 1 -- 172.21.15.42:0/3391415941 >> v1:172.21.15.42:6789/0 conn(0x7f4e18118ee0 legacy=0x7f4e18142b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:09.968 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:09.956+0000 7f4e1cf59700 1 -- 172.21.15.42:0/3391415941 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e18126660 con 0x7f4e18148460 2019-12-05T06:31:09.968 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:09.956+0000 7f4e1cf59700 1 --2- 172.21.15.42:0/3391415941 >> v2:172.21.15.42:3300/0 conn(0x7f4e18148460 0x7f4e18125ee0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f4e18119370 tx=0x7f4e10010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:09.968 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:09.956+0000 7f4e1cf59700 1 --2- 172.21.15.42:0/3391415941 >> v2:172.21.15.42:3300/0 conn(0x7f4e18148460 0x7f4e18125ee0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f4e18119370 tx=0x7f4e10010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:09.969 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:09.956+0000 7f4e1cf59700 1 --2- 172.21.15.42:0/3391415941 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4e18148460 0x7f4e18125ee0 secure :-1 s=READY pgs=2757 cs=0 l=1 rx=0x7f4e18119370 tx=0x7f4e10010b40).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:31:09.969 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:09.956+0000 7f4e1cf59700 1 --2- 172.21.15.42:0/3391415941 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4e18148460 0x7f4e18125ee0 secure :-1 s=THROTTLE_DONE pgs=2757 cs=0 l=1 rx=0x7f4e18119370 tx=0x7f4e10010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:09.969 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:09.956+0000 7f4e1cf59700 1 --2- 172.21.15.42:0/3391415941 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4e18148460 0x7f4e18125ee0 secure :-1 s=THROTTLE_DONE pgs=2757 cs=0 l=1 rx=0x7f4e18119370 tx=0x7f4e10010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:09.969 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:09.956+0000 7f4e14ff9700 1 -- 172.21.15.42:0/3391415941 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4e10006ac0 con 0x7f4e18148460 2019-12-05T06:31:09.969 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:09.956+0000 7f4e1cf59700 1 --2- 172.21.15.42:0/3391415941 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4e18148460 0x7f4e18125ee0 secure :-1 s=THROTTLE_DONE pgs=2757 cs=0 l=1 rx=0x7f4e18119370 tx=0x7f4e10010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:09.969 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:09.956+0000 7f4e14ff9700 1 -- 172.21.15.42:0/3391415941 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e100066e0 con 0x7f4e18148460 2019-12-05T06:31:09.970 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:09.956+0000 7f4e14ff9700 1 -- 172.21.15.42:0/3391415941 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4e10002730 con 0x7f4e18148460 2019-12-05T06:31:09.970 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:09.956+0000 7f4e1e1bb700 1 -- 172.21.15.42:0/3391415941 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e1812cd00 con 0x7f4e18148460 2019-12-05T06:31:09.970 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:09.956+0000 7f4e1e1bb700 1 -- 172.21.15.42:0/3391415941 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e1812d110 con 0x7f4e18148460 2019-12-05T06:31:09.970 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:09.960+0000 7f4e1cf59700 1 --2- 172.21.15.42:0/3391415941 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4e18148460 0x7f4e18125ee0 secure :-1 s=THROTTLE_DONE pgs=2757 cs=0 l=1 rx=0x7f4e18119370 tx=0x7f4e10010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:09.970 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:09.960+0000 7f4e14ff9700 1 -- 172.21.15.42:0/3391415941 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f4e10002a40 con 0x7f4e18148460 2019-12-05T06:31:09.970 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:09.960+0000 7f4e1cf59700 1 --2- 172.21.15.42:0/3391415941 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4e18148460 0x7f4e18125ee0 secure :-1 s=THROTTLE_DONE pgs=2757 cs=0 l=1 rx=0x7f4e18119370 tx=0x7f4e10010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:09.970 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:09.960+0000 7f4e14ff9700 1 --2- 172.21.15.42:0/3391415941 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4df803ac30 0x7f4df803d140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:31:09.971 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:09.960+0000 7f4e14ff9700 1 -- 172.21.15.42:0/3391415941 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 11105+0+0 (secure 0 0 0) 0x7f4e1004b1b0 con 0x7f4e18148460 2019-12-05T06:31:09.971 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:09.960+0000 7f4e17fff700 1 --2- 172.21.15.42:0/3391415941 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4df803ac30 0x7f4df803d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:31:09.972 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:09.960+0000 7f4e17fff700 1 --2- 172.21.15.42:0/3391415941 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4df803ac30 0x7f4df803d140 crc :-1 s=READY pgs=1156 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:31:09.972 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:09.960+0000 7f4e1e1bb700 1 -- 172.21.15.42:0/3391415941 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e1805d110 con 0x7f4e18148460 2019-12-05T06:31:09.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:09.968+0000 7f4e1cf59700 1 --2- 172.21.15.42:0/3391415941 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4e18148460 0x7f4e18125ee0 secure :-1 s=THROTTLE_DONE pgs=2757 cs=0 l=1 rx=0x7f4e18119370 tx=0x7f4e10010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:09.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:09.968+0000 7f4e14ff9700 1 -- 172.21.15.42:0/3391415941 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f4e10032230 con 0x7f4e18148460 2019-12-05T06:31:10.146 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:10.140+0000 7f4e1e1bb700 1 -- 172.21.15.42:0/3391415941 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4e1805b8c0 con 0x7f4e18148460 2019-12-05T06:31:10.147 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:10.140+0000 7f4e1cf59700 1 --2- 172.21.15.42:0/3391415941 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4e18148460 0x7f4e18125ee0 secure :-1 s=THROTTLE_DONE pgs=2757 cs=0 l=1 rx=0x7f4e18119370 tx=0x7f4e10010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:10.147 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:10.140+0000 7f4e14ff9700 1 -- 172.21.15.42:0/3391415941 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 328 v328) v1 ==== 95+0+2117 (secure 0 0 0) 0x7f4e10032490 con 0x7f4e18148460 2019-12-05T06:31:10.147 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 328 2019-12-05T06:31:10.148 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:31:10.148 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":328,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":11352,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/2055706790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2055706790},{"type":"v1","addr":"172.21.15.40:6833","nonce":2055706790}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":328}],"filesystems":[{"mdsmap":{"epoch":327,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:30:20.005055+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":108,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11265":{"gid":11265,"name":"a","rank":0,"incarnation":324,"state":"up:active","state_seq":28,"addr":"172.21.15.42:6835/1985322937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1985322937},{"type":"v1","addr":"172.21.15.42:6835","nonce":1985322937}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:31:10.150 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:10.144+0000 7f4e1e1bb700 1 -- 172.21.15.42:0/3391415941 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4df803ac30 msgr2=0x7f4df803d140 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:10.150 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:10.144+0000 7f4e1e1bb700 1 --2- 172.21.15.42:0/3391415941 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4df803ac30 0x7f4df803d140 crc :-1 s=READY pgs=1156 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:31:10.150 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:10.144+0000 7f4e1e1bb700 1 -- 172.21.15.42:0/3391415941 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4e18148460 msgr2=0x7f4e18125ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:10.150 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:10.144+0000 7f4e1e1bb700 1 --2- 172.21.15.42:0/3391415941 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4e18148460 0x7f4e18125ee0 secure :-1 s=READY pgs=2757 cs=0 l=1 rx=0x7f4e18119370 tx=0x7f4e10010b40).stop 2019-12-05T06:31:10.150 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:10.144+0000 7f4e1e1bb700 1 -- 172.21.15.42:0/3391415941 shutdown_connections 2019-12-05T06:31:10.151 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:10.144+0000 7f4e1e1bb700 1 --2- 172.21.15.42:0/3391415941 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4df803ac30 0x7f4df803d140 unknown :-1 s=CLOSED pgs=1156 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:31:10.151 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:10.144+0000 7f4e1e1bb700 1 --2- 172.21.15.42:0/3391415941 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4e18148460 0x7f4e18125ee0 unknown :-1 s=CLOSED pgs=2757 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:31:10.151 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:10.144+0000 7f4e1e1bb700 1 --2- 172.21.15.42:0/3391415941 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4e181476e0 0x7f4e181450b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:31:10.151 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:10.144+0000 7f4e1e1bb700 1 -- 172.21.15.42:0/3391415941 shutdown_connections 2019-12-05T06:31:10.151 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:10.144+0000 7f4e1e1bb700 1 -- 172.21.15.42:0/3391415941 wait complete. 2019-12-05T06:31:10.151 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:10.144+0000 7f4e1e1bb700 1 -- 172.21.15.42:0/3391415941 >> 172.21.15.42:0/3391415941 conn(0x7f4e1808fda0 msgr2=0x7f4e18059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:31:10.160 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:31:12.160 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:31:12.263 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:12.252+0000 7f3cad714700 1 Processor -- start 2019-12-05T06:31:12.263 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:12.252+0000 7f3cad714700 1 -- start start 2019-12-05T06:31:12.264 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:12.252+0000 7f3cad714700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3ca80a7740 0x7f3ca808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:31:12.264 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:12.252+0000 7f3cad714700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3ca807e720 0x7f3ca807eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:31:12.264 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:12.252+0000 7f3cad714700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ca80a0710 con 0x7f3ca809f2e0 2019-12-05T06:31:12.264 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:12.252+0000 7f3cad714700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ca80a0b00 con 0x7f3ca807e720 2019-12-05T06:31:12.264 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:12.252+0000 7f3cad714700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ca808e5b0 con 0x7f3ca80a7740 2019-12-05T06:31:12.265 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:12.252+0000 7f3c9ffff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3ca807e720 0x7f3ca807eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:31:12.265 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:12.252+0000 7f3c9ffff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3ca807e720 0x7f3ca807eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:56834/0 (socket says 172.21.15.42:56834) 2019-12-05T06:31:12.265 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:12.252+0000 7f3c9ffff700 1 -- 172.21.15.42:0/1282539614 learned_addr learned my addr 172.21.15.42:0/1282539614 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:31:12.265 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:12.252+0000 7f3ca77fe700 1 --2- 172.21.15.42:0/1282539614 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3ca80a7740 0x7f3ca808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:31:12.265 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:12.252+0000 7f3ca6ffd700 1 -- 172.21.15.42:0/1282539614 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1483533767 0 0) 0x7f3ca80a0710 con 0x7f3ca809f2e0 2019-12-05T06:31:12.265 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:12.252+0000 7f3ca6ffd700 1 -- 172.21.15.42:0/1282539614 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c940030b0 con 0x7f3ca809f2e0 2019-12-05T06:31:12.265 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:12.252+0000 7f3c9ffff700 1 -- 172.21.15.42:0/1282539614 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3ca80a7740 msgr2=0x7f3ca808df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:31:12.266 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:12.252+0000 7f3c9ffff700 1 --2- 172.21.15.42:0/1282539614 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3ca80a7740 0x7f3ca808df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:31:12.266 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:12.252+0000 7f3c9ffff700 1 -- 172.21.15.42:0/1282539614 >> v1:172.21.15.40:6789/0 conn(0x7f3ca809f2e0 legacy=0x7f3ca809f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:12.266 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:12.256+0000 7f3c9ffff700 1 -- 172.21.15.42:0/1282539614 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ca807f0a0 con 0x7f3ca807e720 2019-12-05T06:31:12.266 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:12.256+0000 7f3c9ffff700 1 --2- 172.21.15.42:0/1282539614 >> v2:172.21.15.42:3300/0 conn(0x7f3ca807e720 0x7f3ca807eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f3c90008790 tx=0x7f3c9000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:12.266 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:12.256+0000 7f3c9ffff700 1 --2- 172.21.15.42:0/1282539614 >> v2:172.21.15.42:3300/0 conn(0x7f3ca807e720 0x7f3ca807eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f3c90008790 tx=0x7f3c9000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:12.266 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:12.256+0000 7f3c9ffff700 1 --2- 172.21.15.42:0/1282539614 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3ca807e720 0x7f3ca807eb50 secure :-1 s=READY pgs=2758 cs=0 l=1 rx=0x7f3c90008790 tx=0x7f3c9000fd30).ready entity=mon.1 client_cookie=84a034ec516663cd server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:31:12.267 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:12.256+0000 7f3c9ffff700 1 --2- 172.21.15.42:0/1282539614 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3ca807e720 0x7f3ca807eb50 secure :-1 s=THROTTLE_DONE pgs=2758 cs=0 l=1 rx=0x7f3c90008790 tx=0x7f3c9000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:12.267 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:12.256+0000 7f3ca6ffd700 1 -- 172.21.15.42:0/1282539614 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3c900113b0 con 0x7f3ca807e720 2019-12-05T06:31:12.267 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:12.256+0000 7f3c9ffff700 1 --2- 172.21.15.42:0/1282539614 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3ca807e720 0x7f3ca807eb50 secure :-1 s=THROTTLE_DONE pgs=2758 cs=0 l=1 rx=0x7f3c90008790 tx=0x7f3c9000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:12.267 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:12.256+0000 7f3ca6ffd700 1 -- 172.21.15.42:0/1282539614 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c90008bb0 con 0x7f3ca807e720 2019-12-05T06:31:12.267 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:12.256+0000 7f3c9ffff700 1 --2- 172.21.15.42:0/1282539614 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3ca807e720 0x7f3ca807eb50 secure :-1 s=THROTTLE_DONE pgs=2758 cs=0 l=1 rx=0x7f3c90008790 tx=0x7f3c9000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:12.267 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:12.256+0000 7f3cad714700 1 -- 172.21.15.42:0/1282539614 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3ca807e720 msgr2=0x7f3ca807eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:12.268 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:12.256+0000 7f3ca6ffd700 1 -- 172.21.15.42:0/1282539614 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3c90011990 con 0x7f3ca807e720 2019-12-05T06:31:12.268 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:12.256+0000 7f3cad714700 1 --2- 172.21.15.42:0/1282539614 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3ca807e720 0x7f3ca807eb50 secure :-1 s=READY pgs=2758 cs=0 l=1 rx=0x7f3c90008790 tx=0x7f3c9000fd30).stop 2019-12-05T06:31:12.268 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:12.256+0000 7f3cad714700 1 -- 172.21.15.42:0/1282539614 shutdown_connections 2019-12-05T06:31:12.268 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:12.256+0000 7f3cad714700 1 --2- 172.21.15.42:0/1282539614 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3ca80a7740 0x7f3ca808df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:31:12.268 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:12.256+0000 7f3cad714700 1 --2- 172.21.15.42:0/1282539614 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3ca807e720 0x7f3ca807eb50 unknown :-1 s=CLOSED pgs=2758 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:31:12.268 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:12.256+0000 7f3cad714700 1 -- 172.21.15.42:0/1282539614 shutdown_connections 2019-12-05T06:31:12.269 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:12.256+0000 7f3cad714700 1 -- 172.21.15.42:0/1282539614 wait complete. 2019-12-05T06:31:12.269 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:12.256+0000 7f3cad714700 1 -- 172.21.15.42:0/1282539614 >> 172.21.15.42:0/1282539614 conn(0x7f3ca80aabe0 msgr2=0x7f3ca80a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:31:12.272 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:12.260+0000 7f3cad714700 1 Processor -- start 2019-12-05T06:31:12.272 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:12.260+0000 7f3cad714700 1 -- start start 2019-12-05T06:31:12.272 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:12.260+0000 7f3cad714700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f3ca8147fc0 0x7f3ca8119a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:31:12.273 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:12.260+0000 7f3cad714700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ca8142750 con 0x7f3ca811aa80 2019-12-05T06:31:12.273 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:12.260+0000 7f3cad714700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ca811b2e0 con 0x7f3ca811aeb0 2019-12-05T06:31:12.273 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:12.260+0000 7f3cad714700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ca811a5b0 con 0x7f3ca8147fc0 2019-12-05T06:31:12.273 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:12.260+0000 7f3ca7fff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f3ca811aeb0 0x7f3ca81359a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:56876/0 (socket says 172.21.15.42:56876) 2019-12-05T06:31:12.273 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:12.260+0000 7f3ca7fff700 1 -- 172.21.15.42:0/1472935500 learned_addr learned my addr 172.21.15.42:0/1472935500 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:31:12.273 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:12.260+0000 7f3ca77fe700 1 --2- 172.21.15.42:0/1472935500 >> v2:172.21.15.40:3300/0 conn(0x7f3ca8147fc0 0x7f3ca8119a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:31:12.273 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:12.260+0000 7f3ca4ff9700 1 -- 172.21.15.42:0/1472935500 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1937453788 0 0) 0x7f3ca811b2e0 con 0x7f3ca811aeb0 2019-12-05T06:31:12.274 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:12.260+0000 7f3ca4ff9700 1 -- 172.21.15.42:0/1472935500 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c8c0030b0 con 0x7f3ca811aeb0 2019-12-05T06:31:12.274 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:12.260+0000 7f3ca77fe700 1 -- 172.21.15.42:0/1472935500 >> v1:172.21.15.42:6789/0 conn(0x7f3ca811aeb0 legacy=0x7f3ca81359a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:12.274 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:12.260+0000 7f3ca77fe700 1 -- 172.21.15.42:0/1472935500 >> v1:172.21.15.40:6789/0 conn(0x7f3ca811aa80 legacy=0x7f3ca807b7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:12.274 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:12.260+0000 7f3ca77fe700 1 -- 172.21.15.42:0/1472935500 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ca807bfb0 con 0x7f3ca8147fc0 2019-12-05T06:31:12.274 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:12.260+0000 7f3ca77fe700 1 --2- 172.21.15.42:0/1472935500 >> v2:172.21.15.40:3300/0 conn(0x7f3ca8147fc0 0x7f3ca8119a80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f3c98002f30 tx=0x7f3c980109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:12.274 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:12.260+0000 7f3ca77fe700 1 --2- 172.21.15.42:0/1472935500 >> v2:172.21.15.40:3300/0 conn(0x7f3ca8147fc0 0x7f3ca8119a80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f3c98002f30 tx=0x7f3c980109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:12.274 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:12.260+0000 7f3ca77fe700 1 --2- 172.21.15.42:0/1472935500 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3ca8147fc0 0x7f3ca8119a80 secure :-1 s=READY pgs=2068 cs=0 l=1 rx=0x7f3c98002f30 tx=0x7f3c980109e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:31:12.275 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:12.260+0000 7f3ca77fe700 1 --2- 172.21.15.42:0/1472935500 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3ca8147fc0 0x7f3ca8119a80 secure :-1 s=THROTTLE_DONE pgs=2068 cs=0 l=1 rx=0x7f3c98002f30 tx=0x7f3c980109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:12.275 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:12.260+0000 7f3ca4ff9700 1 -- 172.21.15.42:0/1472935500 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3c98005920 con 0x7f3ca8147fc0 2019-12-05T06:31:12.275 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:12.260+0000 7f3ca77fe700 1 --2- 172.21.15.42:0/1472935500 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3ca8147fc0 0x7f3ca8119a80 secure :-1 s=THROTTLE_DONE pgs=2068 cs=0 l=1 rx=0x7f3c98002f30 tx=0x7f3c980109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:12.275 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:12.264+0000 7f3ca77fe700 1 --2- 172.21.15.42:0/1472935500 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3ca8147fc0 0x7f3ca8119a80 secure :-1 s=THROTTLE_DONE pgs=2068 cs=0 l=1 rx=0x7f3c98002f30 tx=0x7f3c980109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:12.275 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:12.264+0000 7f3ca4ff9700 1 -- 172.21.15.42:0/1472935500 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c98005540 con 0x7f3ca8147fc0 2019-12-05T06:31:12.276 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:12.264+0000 7f3cad714700 1 -- 172.21.15.42:0/1472935500 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ca81371b0 con 0x7f3ca8147fc0 2019-12-05T06:31:12.276 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:12.264+0000 7f3ca4ff9700 1 -- 172.21.15.42:0/1472935500 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3c98012510 con 0x7f3ca8147fc0 2019-12-05T06:31:12.276 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:12.264+0000 7f3cad714700 1 -- 172.21.15.42:0/1472935500 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ca8137680 con 0x7f3ca8147fc0 2019-12-05T06:31:12.276 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:12.264+0000 7f3ca77fe700 1 --2- 172.21.15.42:0/1472935500 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3ca8147fc0 0x7f3ca8119a80 secure :-1 s=THROTTLE_DONE pgs=2068 cs=0 l=1 rx=0x7f3c98002f30 tx=0x7f3c980109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:12.276 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:12.264+0000 7f3ca4ff9700 1 -- 172.21.15.42:0/1472935500 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f3c98012870 con 0x7f3ca8147fc0 2019-12-05T06:31:12.276 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:12.264+0000 7f3ca77fe700 1 --2- 172.21.15.42:0/1472935500 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3ca8147fc0 0x7f3ca8119a80 secure :-1 s=THROTTLE_DONE pgs=2068 cs=0 l=1 rx=0x7f3c98002f30 tx=0x7f3c980109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:12.276 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:12.264+0000 7f3ca4ff9700 1 --2- 172.21.15.42:0/1472935500 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3c8c03b150 0x7f3c8c03d660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:31:12.277 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:12.264+0000 7f3ca4ff9700 1 -- 172.21.15.42:0/1472935500 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 11105+0+0 (secure 0 0 0) 0x7f3c9804aec0 con 0x7f3ca8147fc0 2019-12-05T06:31:12.277 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:12.264+0000 7f3c9ffff700 1 --2- 172.21.15.42:0/1472935500 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3c8c03b150 0x7f3c8c03d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:31:12.277 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:12.264+0000 7f3c9ffff700 1 --2- 172.21.15.42:0/1472935500 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3c8c03b150 0x7f3c8c03d660 crc :-1 s=READY pgs=1157 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:31:12.277 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:12.268+0000 7f3cad714700 1 -- 172.21.15.42:0/1472935500 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ca805d110 con 0x7f3ca8147fc0 2019-12-05T06:31:12.282 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:12.276+0000 7f3ca77fe700 1 --2- 172.21.15.42:0/1472935500 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3ca8147fc0 0x7f3ca8119a80 secure :-1 s=THROTTLE_DONE pgs=2068 cs=0 l=1 rx=0x7f3c98002f30 tx=0x7f3c980109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:12.283 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:12.276+0000 7f3ca4ff9700 1 -- 172.21.15.42:0/1472935500 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f3c98031f20 con 0x7f3ca8147fc0 2019-12-05T06:31:12.453 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:12.448+0000 7f3cad714700 1 -- 172.21.15.42:0/1472935500 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3ca80a6990 con 0x7f3ca8147fc0 2019-12-05T06:31:12.454 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:12.448+0000 7f3ca77fe700 1 --2- 172.21.15.42:0/1472935500 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3ca8147fc0 0x7f3ca8119a80 secure :-1 s=THROTTLE_DONE pgs=2068 cs=0 l=1 rx=0x7f3c98002f30 tx=0x7f3c980109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:12.455 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:12.448+0000 7f3ca4ff9700 1 -- 172.21.15.42:0/1472935500 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 328 v328) v1 ==== 95+0+2117 (secure 0 0 0) 0x7f3c98032180 con 0x7f3ca8147fc0 2019-12-05T06:31:12.457 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 328 2019-12-05T06:31:12.457 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:31:12.457 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":328,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":11352,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/2055706790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2055706790},{"type":"v1","addr":"172.21.15.40:6833","nonce":2055706790}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":328}],"filesystems":[{"mdsmap":{"epoch":327,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:30:20.005055+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":108,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11265":{"gid":11265,"name":"a","rank":0,"incarnation":324,"state":"up:active","state_seq":28,"addr":"172.21.15.42:6835/1985322937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1985322937},{"type":"v1","addr":"172.21.15.42:6835","nonce":1985322937}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:31:12.460 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:12.452+0000 7f3cad714700 1 -- 172.21.15.42:0/1472935500 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3c8c03b150 msgr2=0x7f3c8c03d660 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:12.460 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:12.452+0000 7f3cad714700 1 --2- 172.21.15.42:0/1472935500 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3c8c03b150 0x7f3c8c03d660 crc :-1 s=READY pgs=1157 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:31:12.460 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:12.452+0000 7f3cad714700 1 -- 172.21.15.42:0/1472935500 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3ca8147fc0 msgr2=0x7f3ca8119a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:12.460 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:12.452+0000 7f3cad714700 1 --2- 172.21.15.42:0/1472935500 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3ca8147fc0 0x7f3ca8119a80 secure :-1 s=READY pgs=2068 cs=0 l=1 rx=0x7f3c98002f30 tx=0x7f3c980109e0).stop 2019-12-05T06:31:12.460 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:12.452+0000 7f3cad714700 1 -- 172.21.15.42:0/1472935500 shutdown_connections 2019-12-05T06:31:12.460 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:12.452+0000 7f3cad714700 1 --2- 172.21.15.42:0/1472935500 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3c8c03b150 0x7f3c8c03d660 unknown :-1 s=CLOSED pgs=1157 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:31:12.461 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:12.452+0000 7f3cad714700 1 --2- 172.21.15.42:0/1472935500 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3ca8147fc0 0x7f3ca8119a80 unknown :-1 s=CLOSED pgs=2068 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:31:12.461 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:12.452+0000 7f3cad714700 1 -- 172.21.15.42:0/1472935500 shutdown_connections 2019-12-05T06:31:12.461 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:12.452+0000 7f3cad714700 1 -- 172.21.15.42:0/1472935500 wait complete. 2019-12-05T06:31:12.461 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:12.452+0000 7f3cad714700 1 -- 172.21.15.42:0/1472935500 >> 172.21.15.42:0/1472935500 conn(0x7f3ca809f380 msgr2=0x7f3ca811c520 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:31:12.470 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:31:13.087 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:31:13.091 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:31:14.471 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:31:14.574 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:14.564+0000 7eff02964700 1 Processor -- start 2019-12-05T06:31:14.574 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:14.564+0000 7eff02964700 1 -- start start 2019-12-05T06:31:14.574 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:14.564+0000 7eff02964700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efefc0a7740 0x7efefc08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:31:14.575 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:14.564+0000 7eff02964700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7efefc09fae0 0x7efefc07eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:31:14.575 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:14.564+0000 7eff02964700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efefc0a0710 con 0x7efefc07e720 2019-12-05T06:31:14.575 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:14.564+0000 7eff02964700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efefc0a0b00 con 0x7efefc09fae0 2019-12-05T06:31:14.575 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:14.564+0000 7eff02964700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efefc08e5b0 con 0x7efefc0a7740 2019-12-05T06:31:14.575 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:14.564+0000 7eff01702700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7efefc09fae0 0x7efefc07eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:31:14.575 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:14.564+0000 7eff01702700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7efefc09fae0 0x7efefc07eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:56852/0 (socket says 172.21.15.42:56852) 2019-12-05T06:31:14.576 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:14.564+0000 7eff01702700 1 -- 172.21.15.42:0/272670444 learned_addr learned my addr 172.21.15.42:0/272670444 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:31:14.576 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:14.564+0000 7eff00f01700 1 --2- 172.21.15.42:0/272670444 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efefc0a7740 0x7efefc08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:31:14.576 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:14.564+0000 7eff01702700 1 -- 172.21.15.42:0/272670444 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efefc0a7740 msgr2=0x7efefc08df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:31:14.576 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:14.564+0000 7eff01702700 1 --2- 172.21.15.42:0/272670444 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efefc0a7740 0x7efefc08df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:31:14.576 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:14.564+0000 7eff01702700 1 -- 172.21.15.42:0/272670444 >> v1:172.21.15.42:6789/0 conn(0x7efefc07e720 legacy=0x7efefc09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:14.576 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:14.564+0000 7eff01702700 1 -- 172.21.15.42:0/272670444 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efefc07f0a0 con 0x7efefc09fae0 2019-12-05T06:31:14.576 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:14.564+0000 7eff01702700 1 --2- 172.21.15.42:0/272670444 >> v2:172.21.15.42:3300/0 conn(0x7efefc09fae0 0x7efefc07eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7efef4008b10 tx=0x7efef40100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:14.577 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:14.564+0000 7eff01702700 1 --2- 172.21.15.42:0/272670444 >> v2:172.21.15.42:3300/0 conn(0x7efefc09fae0 0x7efefc07eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7efef4008b10 tx=0x7efef40100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:14.577 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:14.564+0000 7eff01702700 1 --2- 172.21.15.42:0/272670444 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efefc09fae0 0x7efefc07eb50 secure :-1 s=READY pgs=2761 cs=0 l=1 rx=0x7efef4008b10 tx=0x7efef40100b0).ready entity=mon.1 client_cookie=c7933e0809516384 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:31:14.577 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:14.564+0000 7eff01702700 1 --2- 172.21.15.42:0/272670444 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efefc09fae0 0x7efefc07eb50 secure :-1 s=THROTTLE_DONE pgs=2761 cs=0 l=1 rx=0x7efef4008b10 tx=0x7efef40100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:14.577 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:14.564+0000 7eff01702700 1 --2- 172.21.15.42:0/272670444 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efefc09fae0 0x7efefc07eb50 secure :-1 s=THROTTLE_DONE pgs=2761 cs=0 l=1 rx=0x7efef4008b10 tx=0x7efef40100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:14.577 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:14.564+0000 7efefb7fe700 1 -- 172.21.15.42:0/272670444 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efef4010da0 con 0x7efefc09fae0 2019-12-05T06:31:14.577 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:14.564+0000 7eff01702700 1 --2- 172.21.15.42:0/272670444 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efefc09fae0 0x7efefc07eb50 secure :-1 s=THROTTLE_DONE pgs=2761 cs=0 l=1 rx=0x7efef4008b10 tx=0x7efef40100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:14.578 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:14.564+0000 7efefb7fe700 1 -- 172.21.15.42:0/272670444 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efef40109c0 con 0x7efefc09fae0 2019-12-05T06:31:14.578 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:14.564+0000 7efefb7fe700 1 -- 172.21.15.42:0/272670444 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efef40114a0 con 0x7efefc09fae0 2019-12-05T06:31:14.578 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:14.564+0000 7eff02964700 1 -- 172.21.15.42:0/272670444 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efefc09fae0 msgr2=0x7efefc07eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:14.578 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:14.564+0000 7eff02964700 1 --2- 172.21.15.42:0/272670444 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efefc09fae0 0x7efefc07eb50 secure :-1 s=READY pgs=2761 cs=0 l=1 rx=0x7efef4008b10 tx=0x7efef40100b0).stop 2019-12-05T06:31:14.578 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:14.568+0000 7eff02964700 1 -- 172.21.15.42:0/272670444 shutdown_connections 2019-12-05T06:31:14.578 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:14.568+0000 7eff02964700 1 --2- 172.21.15.42:0/272670444 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efefc09fae0 0x7efefc07eb50 unknown :-1 s=CLOSED pgs=2761 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:31:14.578 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:14.568+0000 7eff02964700 1 --2- 172.21.15.42:0/272670444 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efefc0a7740 0x7efefc08df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:31:14.579 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:14.568+0000 7eff02964700 1 -- 172.21.15.42:0/272670444 shutdown_connections 2019-12-05T06:31:14.579 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:14.568+0000 7eff02964700 1 -- 172.21.15.42:0/272670444 wait complete. 2019-12-05T06:31:14.579 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:14.568+0000 7eff02964700 1 -- 172.21.15.42:0/272670444 >> 172.21.15.42:0/272670444 conn(0x7efefc0aabe0 msgr2=0x7efefc0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:31:14.582 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:14.568+0000 7eff02964700 1 Processor -- start 2019-12-05T06:31:14.583 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:14.568+0000 7eff02964700 1 -- start start 2019-12-05T06:31:14.583 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:14.568+0000 7eff02964700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7efefc1224e0 0x7efefc1163d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:31:14.583 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:14.568+0000 7eff02964700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efefc11be90 0x7efefc132c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:31:14.583 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:14.568+0000 7eff02964700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7efefc1331a0 0x7efefc11a310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:31:14.583 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:14.568+0000 7eff02964700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efefc1329f0 con 0x7efefc1331a0 2019-12-05T06:31:14.583 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:14.568+0000 7eff02964700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efefc116c00 con 0x7efefc1224e0 2019-12-05T06:31:14.584 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:14.568+0000 7eff02964700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efefc116ea0 con 0x7efefc11be90 2019-12-05T06:31:14.584 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:14.568+0000 7eff00f01700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7efefc1224e0 0x7efefc1163d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:31:14.584 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:14.568+0000 7eff00f01700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7efefc1224e0 0x7efefc1163d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:56856/0 (socket says 172.21.15.42:56856) 2019-12-05T06:31:14.585 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:14.568+0000 7eff00f01700 1 -- 172.21.15.42:0/1865762958 learned_addr learned my addr 172.21.15.42:0/1865762958 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:31:14.585 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:14.568+0000 7eff01702700 1 --2- 172.21.15.42:0/1865762958 >> v2:172.21.15.40:3300/0 conn(0x7efefc1331a0 0x7efefc11a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:31:14.585 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:14.568+0000 7efefbfff700 1 --2- 172.21.15.42:0/1865762958 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efefc11be90 0x7efefc132c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:31:14.585 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:14.572+0000 7eff00f01700 1 -- 172.21.15.42:0/1865762958 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efefc11be90 msgr2=0x7efefc132c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:14.585 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:14.572+0000 7eff00f01700 1 --2- 172.21.15.42:0/1865762958 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efefc11be90 0x7efefc132c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:31:14.585 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:14.572+0000 7eff00f01700 1 -- 172.21.15.42:0/1865762958 >> v2:172.21.15.40:3300/0 conn(0x7efefc1331a0 msgr2=0x7efefc11a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:14.586 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:14.572+0000 7eff00f01700 1 --2- 172.21.15.42:0/1865762958 >> v2:172.21.15.40:3300/0 conn(0x7efefc1331a0 0x7efefc11a310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:31:14.586 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:14.572+0000 7eff00f01700 1 -- 172.21.15.42:0/1865762958 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efefc116920 con 0x7efefc1224e0 2019-12-05T06:31:14.586 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:14.572+0000 7efefbfff700 1 --2- 172.21.15.42:0/1865762958 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efefc11be90 0x7efefc132c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:31:14.586 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:14.572+0000 7eff00f01700 1 --2- 172.21.15.42:0/1865762958 >> v2:172.21.15.42:3300/0 conn(0x7efefc1224e0 0x7efefc1163d0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7efefc117060 tx=0x7efeec010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:14.586 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:14.572+0000 7eff00f01700 1 --2- 172.21.15.42:0/1865762958 >> v2:172.21.15.42:3300/0 conn(0x7efefc1224e0 0x7efefc1163d0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7efefc117060 tx=0x7efeec010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:14.586 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:14.572+0000 7eff00f01700 1 --2- 172.21.15.42:0/1865762958 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efefc1224e0 0x7efefc1163d0 secure :-1 s=READY pgs=2762 cs=0 l=1 rx=0x7efefc117060 tx=0x7efeec010b60).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:31:14.586 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:14.572+0000 7eff00f01700 1 --2- 172.21.15.42:0/1865762958 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efefc1224e0 0x7efefc1163d0 secure :-1 s=THROTTLE_DONE pgs=2762 cs=0 l=1 rx=0x7efefc117060 tx=0x7efeec010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:14.587 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:14.572+0000 7eff00f01700 1 --2- 172.21.15.42:0/1865762958 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efefc1224e0 0x7efefc1163d0 secure :-1 s=THROTTLE_DONE pgs=2762 cs=0 l=1 rx=0x7efefc117060 tx=0x7efeec010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:14.587 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:14.572+0000 7efef97fa700 1 -- 172.21.15.42:0/1865762958 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efeec005930 con 0x7efefc1224e0 2019-12-05T06:31:14.587 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:14.572+0000 7eff00f01700 1 --2- 172.21.15.42:0/1865762958 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efefc1224e0 0x7efefc1163d0 secure :-1 s=THROTTLE_DONE pgs=2762 cs=0 l=1 rx=0x7efefc117060 tx=0x7efeec010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:14.587 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:14.572+0000 7efef97fa700 1 -- 172.21.15.42:0/1865762958 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efeec007160 con 0x7efefc1224e0 2019-12-05T06:31:14.587 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:14.572+0000 7eff02964700 1 -- 172.21.15.42:0/1865762958 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efefc1335d0 con 0x7efefc1224e0 2019-12-05T06:31:14.587 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:14.572+0000 7efef97fa700 1 -- 172.21.15.42:0/1865762958 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efeec0121a0 con 0x7efefc1224e0 2019-12-05T06:31:14.587 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:14.572+0000 7eff02964700 1 -- 172.21.15.42:0/1865762958 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efefc11aa80 con 0x7efefc1224e0 2019-12-05T06:31:14.588 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:14.572+0000 7eff00f01700 1 --2- 172.21.15.42:0/1865762958 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efefc1224e0 0x7efefc1163d0 secure :-1 s=THROTTLE_DONE pgs=2762 cs=0 l=1 rx=0x7efefc117060 tx=0x7efeec010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:14.588 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:14.572+0000 7efef97fa700 1 -- 172.21.15.42:0/1865762958 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7efeec0124b0 con 0x7efefc1224e0 2019-12-05T06:31:14.588 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:14.572+0000 7eff00f01700 1 --2- 172.21.15.42:0/1865762958 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efefc1224e0 0x7efefc1163d0 secure :-1 s=THROTTLE_DONE pgs=2762 cs=0 l=1 rx=0x7efefc117060 tx=0x7efeec010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:14.588 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:14.572+0000 7efef97fa700 1 --2- 172.21.15.42:0/1865762958 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7efedc03ae90 0x7efedc03d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:31:14.588 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:14.572+0000 7efef97fa700 1 -- 172.21.15.42:0/1865762958 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 11105+0+0 (secure 0 0 0) 0x7efeec04ae80 con 0x7efefc1224e0 2019-12-05T06:31:14.589 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:14.572+0000 7efefbfff700 1 --2- 172.21.15.42:0/1865762958 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7efedc03ae90 0x7efedc03d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:31:14.589 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:14.572+0000 7efefbfff700 1 --2- 172.21.15.42:0/1865762958 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7efedc03ae90 0x7efedc03d3a0 crc :-1 s=READY pgs=1158 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:31:14.589 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:14.580+0000 7eff02964700 1 -- 172.21.15.42:0/1865762958 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efefc05d110 con 0x7efefc1224e0 2019-12-05T06:31:14.591 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:14.584+0000 7eff00f01700 1 --2- 172.21.15.42:0/1865762958 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efefc1224e0 0x7efefc1163d0 secure :-1 s=THROTTLE_DONE pgs=2762 cs=0 l=1 rx=0x7efefc117060 tx=0x7efeec010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:14.592 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:14.584+0000 7efef97fa700 1 -- 172.21.15.42:0/1865762958 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7efeec031f00 con 0x7efefc1224e0 2019-12-05T06:31:14.755 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:14.748+0000 7eff02964700 1 -- 172.21.15.42:0/1865762958 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efefc05b8c0 con 0x7efefc1224e0 2019-12-05T06:31:14.756 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:14.752+0000 7eff00f01700 1 --2- 172.21.15.42:0/1865762958 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efefc1224e0 0x7efefc1163d0 secure :-1 s=THROTTLE_DONE pgs=2762 cs=0 l=1 rx=0x7efefc117060 tx=0x7efeec010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:14.757 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:14.752+0000 7efef97fa700 1 -- 172.21.15.42:0/1865762958 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 328 v328) v1 ==== 95+0+2117 (secure 0 0 0) 0x7efeec032160 con 0x7efefc1224e0 2019-12-05T06:31:14.758 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:31:14.759 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":328,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":11352,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/2055706790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2055706790},{"type":"v1","addr":"172.21.15.40:6833","nonce":2055706790}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":328}],"filesystems":[{"mdsmap":{"epoch":327,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:30:20.005055+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":108,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11265":{"gid":11265,"name":"a","rank":0,"incarnation":324,"state":"up:active","state_seq":28,"addr":"172.21.15.42:6835/1985322937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1985322937},{"type":"v1","addr":"172.21.15.42:6835","nonce":1985322937}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:31:14.759 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 328 2019-12-05T06:31:14.760 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:14.756+0000 7eff02964700 1 -- 172.21.15.42:0/1865762958 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7efedc03ae90 msgr2=0x7efedc03d3a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:14.764 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:14.756+0000 7eff02964700 1 --2- 172.21.15.42:0/1865762958 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7efedc03ae90 0x7efedc03d3a0 crc :-1 s=READY pgs=1158 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:31:14.765 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:14.756+0000 7eff02964700 1 -- 172.21.15.42:0/1865762958 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efefc1224e0 msgr2=0x7efefc1163d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:14.765 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:14.756+0000 7eff02964700 1 --2- 172.21.15.42:0/1865762958 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efefc1224e0 0x7efefc1163d0 secure :-1 s=READY pgs=2762 cs=0 l=1 rx=0x7efefc117060 tx=0x7efeec010b60).stop 2019-12-05T06:31:14.765 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:14.756+0000 7eff02964700 1 -- 172.21.15.42:0/1865762958 shutdown_connections 2019-12-05T06:31:14.765 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:14.756+0000 7eff02964700 1 --2- 172.21.15.42:0/1865762958 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7efedc03ae90 0x7efedc03d3a0 unknown :-1 s=CLOSED pgs=1158 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:31:14.766 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:14.756+0000 7eff02964700 1 --2- 172.21.15.42:0/1865762958 >> v2:172.21.15.40:3300/0 conn(0x7efefc1331a0 0x7efefc11a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:31:14.766 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:14.756+0000 7eff02964700 1 --2- 172.21.15.42:0/1865762958 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efefc1224e0 0x7efefc1163d0 unknown :-1 s=CLOSED pgs=2762 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:31:14.766 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:14.756+0000 7eff02964700 1 --2- 172.21.15.42:0/1865762958 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efefc11be90 0x7efefc132c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:31:14.766 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:14.756+0000 7eff02964700 1 -- 172.21.15.42:0/1865762958 shutdown_connections 2019-12-05T06:31:14.766 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:14.756+0000 7eff02964700 1 -- 172.21.15.42:0/1865762958 wait complete. 2019-12-05T06:31:14.766 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:14.756+0000 7eff02964700 1 -- 172.21.15.42:0/1865762958 >> 172.21.15.42:0/1865762958 conn(0x7efefc0aabe0 msgr2=0x7efefc08eb70 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:31:14.772 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:31:16.773 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:31:16.882 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:16.872+0000 7f0348b73700 1 Processor -- start 2019-12-05T06:31:16.883 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:16.872+0000 7f0348b73700 1 -- start start 2019-12-05T06:31:16.883 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:16.872+0000 7f0348b73700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f03440a7740 0x7f034408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:31:16.883 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:16.872+0000 7f0348b73700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f034407e720 0x7f034407eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:31:16.884 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:16.872+0000 7f0348b73700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03440a0710 con 0x7f034409f2e0 2019-12-05T06:31:16.884 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:16.872+0000 7f0348b73700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03440a0b00 con 0x7f03440a7740 2019-12-05T06:31:16.884 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:16.872+0000 7f0348b73700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f034408e5b0 con 0x7f034407e720 2019-12-05T06:31:16.884 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:16.872+0000 7f0342d9d700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f03440a7740 0x7f034408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:31:16.885 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:16.872+0000 7f034259c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f034407e720 0x7f034407eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:31:16.885 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:16.872+0000 7f0342d9d700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f03440a7740 0x7f034408df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.42:44678/0 (socket says 172.21.15.42:44678) 2019-12-05T06:31:16.885 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:16.872+0000 7f0342d9d700 1 -- 172.21.15.42:0/548284676 learned_addr learned my addr 172.21.15.42:0/548284676 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:31:16.885 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:16.872+0000 7f0342d9d700 1 -- 172.21.15.42:0/548284676 >> v2:172.21.15.42:3300/0 conn(0x7f034407e720 msgr2=0x7f034407eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:31:16.885 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:16.872+0000 7f0342d9d700 1 --2- 172.21.15.42:0/548284676 >> v2:172.21.15.42:3300/0 conn(0x7f034407e720 0x7f034407eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:31:16.886 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:16.872+0000 7f0342d9d700 1 -- 172.21.15.42:0/548284676 >> v1:172.21.15.42:6789/0 conn(0x7f034409f2e0 legacy=0x7f034409f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:16.886 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:16.872+0000 7f0342d9d700 1 -- 172.21.15.42:0/548284676 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f034407f0a0 con 0x7f03440a7740 2019-12-05T06:31:16.886 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:16.872+0000 7f0342d9d700 1 --2- 172.21.15.42:0/548284676 >> v2:172.21.15.40:3300/0 conn(0x7f03440a7740 0x7f034408df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f032c002f30 tx=0x7f032c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:16.886 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:16.872+0000 7f0342d9d700 1 --2- 172.21.15.42:0/548284676 >> v2:172.21.15.40:3300/0 conn(0x7f03440a7740 0x7f034408df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f032c002f30 tx=0x7f032c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:16.887 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:16.872+0000 7f0342d9d700 1 --2- 172.21.15.42:0/548284676 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f03440a7740 0x7f034408df80 secure :-1 s=READY pgs=2069 cs=0 l=1 rx=0x7f032c002f30 tx=0x7f032c00fd30).ready entity=mon.0 client_cookie=3bb4043ce035312c server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:31:16.887 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:16.872+0000 7f0342d9d700 1 --2- 172.21.15.42:0/548284676 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f03440a7740 0x7f034408df80 secure :-1 s=THROTTLE_DONE pgs=2069 cs=0 l=1 rx=0x7f032c002f30 tx=0x7f032c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:16.887 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:16.872+0000 7f0341d9b700 1 -- 172.21.15.42:0/548284676 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f032c011370 con 0x7f03440a7740 2019-12-05T06:31:16.887 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:16.872+0000 7f0342d9d700 1 --2- 172.21.15.42:0/548284676 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f03440a7740 0x7f034408df80 secure :-1 s=THROTTLE_DONE pgs=2069 cs=0 l=1 rx=0x7f032c002f30 tx=0x7f032c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:16.888 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:16.872+0000 7f0341d9b700 1 -- 172.21.15.42:0/548284676 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f032c008bb0 con 0x7f03440a7740 2019-12-05T06:31:16.888 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:16.872+0000 7f0342d9d700 1 --2- 172.21.15.42:0/548284676 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f03440a7740 0x7f034408df80 secure :-1 s=THROTTLE_DONE pgs=2069 cs=0 l=1 rx=0x7f032c002f30 tx=0x7f032c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:16.888 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:16.872+0000 7f0341d9b700 1 -- 172.21.15.42:0/548284676 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f032c011950 con 0x7f03440a7740 2019-12-05T06:31:16.888 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:16.872+0000 7f0348b73700 1 -- 172.21.15.42:0/548284676 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f03440a7740 msgr2=0x7f034408df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:16.888 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:16.872+0000 7f0348b73700 1 --2- 172.21.15.42:0/548284676 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f03440a7740 0x7f034408df80 secure :-1 s=READY pgs=2069 cs=0 l=1 rx=0x7f032c002f30 tx=0x7f032c00fd30).stop 2019-12-05T06:31:16.889 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:16.876+0000 7f0348b73700 1 -- 172.21.15.42:0/548284676 shutdown_connections 2019-12-05T06:31:16.889 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:16.876+0000 7f0348b73700 1 --2- 172.21.15.42:0/548284676 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f03440a7740 0x7f034408df80 unknown :-1 s=CLOSED pgs=2069 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:31:16.889 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:16.876+0000 7f0348b73700 1 --2- 172.21.15.42:0/548284676 >> v2:172.21.15.42:3300/0 conn(0x7f034407e720 0x7f034407eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:31:16.889 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:16.876+0000 7f0348b73700 1 -- 172.21.15.42:0/548284676 shutdown_connections 2019-12-05T06:31:16.889 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:16.876+0000 7f0348b73700 1 -- 172.21.15.42:0/548284676 wait complete. 2019-12-05T06:31:16.890 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:16.876+0000 7f0348b73700 1 -- 172.21.15.42:0/548284676 >> 172.21.15.42:0/548284676 conn(0x7f03440aabe0 msgr2=0x7f03440a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:31:16.895 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:16.876+0000 7f0348b73700 1 Processor -- start 2019-12-05T06:31:16.895 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:16.876+0000 7f0348b73700 1 -- start start 2019-12-05T06:31:16.895 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:16.880+0000 7f0348b73700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f03440a5890 0x7f03441240c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:31:16.895 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:16.880+0000 7f0348b73700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03440a4a50 con 0x7f0344064cb0 2019-12-05T06:31:16.895 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:16.880+0000 7f0348b73700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03440a6fa0 con 0x7f0344064880 2019-12-05T06:31:16.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:16.880+0000 7f0348b73700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03440a71d0 con 0x7f03440a5890 2019-12-05T06:31:16.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:16.880+0000 7f0342d9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f03440a5890 0x7f03441240c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:31:16.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:16.880+0000 7f034259c700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f0344064880 0x7f0344063190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:56908/0 (socket says 172.21.15.42:56908) 2019-12-05T06:31:16.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:16.880+0000 7f034259c700 1 -- 172.21.15.42:0/3142050990 learned_addr learned my addr 172.21.15.42:0/3142050990 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:31:16.897 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:16.880+0000 7f033b7fe700 1 -- 172.21.15.42:0/3142050990 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 278258846 0 0) 0x7f03440a4a50 con 0x7f0344064cb0 2019-12-05T06:31:16.897 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:16.880+0000 7f033b7fe700 1 -- 172.21.15.42:0/3142050990 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f031c0030b0 con 0x7f0344064cb0 2019-12-05T06:31:16.897 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:16.880+0000 7f0342d9d700 1 -- 172.21.15.42:0/3142050990 >> v1:172.21.15.42:6789/0 conn(0x7f0344064880 legacy=0x7f0344063190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:16.897 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:16.880+0000 7f0342d9d700 1 -- 172.21.15.42:0/3142050990 >> v1:172.21.15.40:6789/0 conn(0x7f0344064cb0 legacy=0x7f0344063990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:16.897 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:16.880+0000 7f033b7fe700 1 -- 172.21.15.42:0/3142050990 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1995045354 0 0) 0x7f03440a6fa0 con 0x7f0344064880 2019-12-05T06:31:16.898 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:16.880+0000 7f0342d9d700 1 -- 172.21.15.42:0/3142050990 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f034407a330 con 0x7f03440a5890 2019-12-05T06:31:16.898 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:16.880+0000 7f0342d9d700 1 --2- 172.21.15.42:0/3142050990 >> v2:172.21.15.42:3300/0 conn(0x7f03440a5890 0x7f03441240c0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f032c002f30 tx=0x7f032c011ee0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:16.898 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:16.880+0000 7f0342d9d700 1 --2- 172.21.15.42:0/3142050990 >> v2:172.21.15.42:3300/0 conn(0x7f03440a5890 0x7f03441240c0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f032c002f30 tx=0x7f032c011ee0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:16.898 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:16.880+0000 7f0342d9d700 1 --2- 172.21.15.42:0/3142050990 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f03440a5890 0x7f03441240c0 secure :-1 s=READY pgs=2765 cs=0 l=1 rx=0x7f032c002f30 tx=0x7f032c011ee0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:31:16.898 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:16.880+0000 7f0342d9d700 1 --2- 172.21.15.42:0/3142050990 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f03440a5890 0x7f03441240c0 secure :-1 s=THROTTLE_DONE pgs=2765 cs=0 l=1 rx=0x7f032c002f30 tx=0x7f032c011ee0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:16.898 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:16.880+0000 7f033b7fe700 1 -- 172.21.15.42:0/3142050990 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f032c007340 con 0x7f03440a5890 2019-12-05T06:31:16.898 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:16.880+0000 7f0342d9d700 1 --2- 172.21.15.42:0/3142050990 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f03440a5890 0x7f03441240c0 secure :-1 s=THROTTLE_DONE pgs=2765 cs=0 l=1 rx=0x7f032c002f30 tx=0x7f032c011ee0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:16.898 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:16.880+0000 7f0342d9d700 1 --2- 172.21.15.42:0/3142050990 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f03440a5890 0x7f03441240c0 secure :-1 s=THROTTLE_DONE pgs=2765 cs=0 l=1 rx=0x7f032c002f30 tx=0x7f032c011ee0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:16.899 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:16.880+0000 7f033b7fe700 1 -- 172.21.15.42:0/3142050990 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f032c011570 con 0x7f03440a5890 2019-12-05T06:31:16.899 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:16.880+0000 7f0348b73700 1 -- 172.21.15.42:0/3142050990 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f034407a660 con 0x7f03440a5890 2019-12-05T06:31:16.899 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:16.880+0000 7f033b7fe700 1 -- 172.21.15.42:0/3142050990 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f032c005540 con 0x7f03440a5890 2019-12-05T06:31:16.899 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:16.880+0000 7f0348b73700 1 -- 172.21.15.42:0/3142050990 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f034407ab90 con 0x7f03440a5890 2019-12-05T06:31:16.899 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:16.880+0000 7f0342d9d700 1 --2- 172.21.15.42:0/3142050990 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f03440a5890 0x7f03441240c0 secure :-1 s=THROTTLE_DONE pgs=2765 cs=0 l=1 rx=0x7f032c002f30 tx=0x7f032c011ee0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:16.899 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:16.880+0000 7f033b7fe700 1 -- 172.21.15.42:0/3142050990 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f032c005850 con 0x7f03440a5890 2019-12-05T06:31:16.899 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:16.880+0000 7f0342d9d700 1 --2- 172.21.15.42:0/3142050990 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f03440a5890 0x7f03441240c0 secure :-1 s=THROTTLE_DONE pgs=2765 cs=0 l=1 rx=0x7f032c002f30 tx=0x7f032c011ee0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:16.900 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:16.880+0000 7f033b7fe700 1 --2- 172.21.15.42:0/3142050990 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f031c03b220 0x7f031c03d730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:31:16.900 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:16.884+0000 7f033b7fe700 1 -- 172.21.15.42:0/3142050990 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 11191+0+0 (secure 0 0 0) 0x7f032c08f2f0 con 0x7f03440a5890 2019-12-05T06:31:16.900 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:16.884+0000 7f034259c700 1 --2- 172.21.15.42:0/3142050990 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f031c03b220 0x7f031c03d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:31:16.900 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:16.884+0000 7f034259c700 1 --2- 172.21.15.42:0/3142050990 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f031c03b220 0x7f031c03d730 crc :-1 s=READY pgs=1159 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:31:16.900 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:16.888+0000 7f0348b73700 1 -- 172.21.15.42:0/3142050990 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f034405d110 con 0x7f03440a5890 2019-12-05T06:31:16.901 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:16.896+0000 7f0342d9d700 1 --2- 172.21.15.42:0/3142050990 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f03440a5890 0x7f03441240c0 secure :-1 s=THROTTLE_DONE pgs=2765 cs=0 l=1 rx=0x7f032c002f30 tx=0x7f032c011ee0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:16.901 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:16.896+0000 7f033b7fe700 1 -- 172.21.15.42:0/3142050990 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f032c076370 con 0x7f03440a5890 2019-12-05T06:31:17.074 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:17.064+0000 7f0348b73700 1 -- 172.21.15.42:0/3142050990 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f034405b8c0 con 0x7f03440a5890 2019-12-05T06:31:17.074 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:17.064+0000 7f0342d9d700 1 --2- 172.21.15.42:0/3142050990 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f03440a5890 0x7f03441240c0 secure :-1 s=THROTTLE_DONE pgs=2765 cs=0 l=1 rx=0x7f032c002f30 tx=0x7f032c011ee0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:17.075 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:17.064+0000 7f033b7fe700 1 -- 172.21.15.42:0/3142050990 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 329 v329) v1 ==== 95+0+1778 (secure 0 0 0) 0x7f032c0765d0 con 0x7f03440a5890 2019-12-05T06:31:17.075 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 329 2019-12-05T06:31:17.075 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:17.068+0000 7f0348b73700 1 -- 172.21.15.42:0/3142050990 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f031c03b220 msgr2=0x7f031c03d730 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:17.075 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:17.068+0000 7f0348b73700 1 --2- 172.21.15.42:0/3142050990 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f031c03b220 0x7f031c03d730 crc :-1 s=READY pgs=1159 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:31:17.076 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:17.068+0000 7f0348b73700 1 -- 172.21.15.42:0/3142050990 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f03440a5890 msgr2=0x7f03441240c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:17.076 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:17.068+0000 7f0348b73700 1 --2- 172.21.15.42:0/3142050990 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f03440a5890 0x7f03441240c0 secure :-1 s=READY pgs=2765 cs=0 l=1 rx=0x7f032c002f30 tx=0x7f032c011ee0).stop 2019-12-05T06:31:17.076 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:17.068+0000 7f0348b73700 1 -- 172.21.15.42:0/3142050990 shutdown_connections 2019-12-05T06:31:17.077 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:17.068+0000 7f0348b73700 1 --2- 172.21.15.42:0/3142050990 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f031c03b220 0x7f031c03d730 unknown :-1 s=CLOSED pgs=1159 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:31:17.077 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:17.068+0000 7f0348b73700 1 --2- 172.21.15.42:0/3142050990 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f03440a5890 0x7f03441240c0 unknown :-1 s=CLOSED pgs=2765 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:31:17.077 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:17.068+0000 7f0348b73700 1 -- 172.21.15.42:0/3142050990 shutdown_connections 2019-12-05T06:31:17.077 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:17.068+0000 7f0348b73700 1 -- 172.21.15.42:0/3142050990 wait complete. 2019-12-05T06:31:17.078 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:17.068+0000 7f0348b73700 1 -- 172.21.15.42:0/3142050990 >> 172.21.15.42:0/3142050990 conn(0x7f034408fda0 msgr2=0x7f034408ec60 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:31:17.078 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:31:17.078 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":329,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":329,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:31:14.986780+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":109,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11352},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11352":{"gid":11352,"name":"b","rank":0,"incarnation":329,"state":"up:replay","state_seq":1,"addr":"172.21.15.40:6833/2055706790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2055706790},{"type":"v1","addr":"172.21.15.40:6833","nonce":2055706790}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:31:17.084 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a down, removed from mdsmap 2019-12-05T06:31:17.084 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2019-12-05T06:31:17.085 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:31:17.202 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:17.188+0000 7f127e78a700 1 Processor -- start 2019-12-05T06:31:17.202 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:17.188+0000 7f127e78a700 1 -- start start 2019-12-05T06:31:17.202 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:17.192+0000 7f127e78a700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f127808df80 0x7f127809f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:31:17.202 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:17.192+0000 7f127e78a700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f127809f830 0x7f127814efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:31:17.203 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:17.192+0000 7f127e78a700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f127809fc60 con 0x7f12780a7740 2019-12-05T06:31:17.203 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:17.192+0000 7f127e78a700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12780a0b00 con 0x7f127809f830 2019-12-05T06:31:17.203 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:17.192+0000 7f127e78a700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f127808e5b0 con 0x7f127808df80 2019-12-05T06:31:17.203 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:17.192+0000 7f127d528700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f127809f830 0x7f127814efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:31:17.203 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:17.192+0000 7f127cd27700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f12780a7740 0x7f127807e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:38124/0 (socket says 172.21.15.42:38124) 2019-12-05T06:31:17.203 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:17.192+0000 7f1277fff700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f127808df80 0x7f127809f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:31:17.203 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:17.192+0000 7f127cd27700 1 -- 172.21.15.42:0/4174017403 learned_addr learned my addr 172.21.15.42:0/4174017403 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:31:17.204 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:17.192+0000 7f127d528700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f127809f830 0x7f127814efe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:56880/0 (socket says 172.21.15.42:56880) 2019-12-05T06:31:17.204 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:17.192+0000 7f12777fe700 1 -- 172.21.15.42:0/4174017403 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 697441920 0 0) 0x7f127809fc60 con 0x7f12780a7740 2019-12-05T06:31:17.204 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:17.192+0000 7f12777fe700 1 -- 172.21.15.42:0/4174017403 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12640030b0 con 0x7f12780a7740 2019-12-05T06:31:17.204 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:17.192+0000 7f1277fff700 1 -- 172.21.15.42:0/4174017403 >> v2:172.21.15.42:3300/0 conn(0x7f127809f830 msgr2=0x7f127814efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:31:17.204 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:17.192+0000 7f1277fff700 1 --2- 172.21.15.42:0/4174017403 >> v2:172.21.15.42:3300/0 conn(0x7f127809f830 0x7f127814efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:31:17.204 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:17.192+0000 7f1277fff700 1 -- 172.21.15.42:0/4174017403 >> v1:172.21.15.40:6789/0 conn(0x7f12780a7740 legacy=0x7f127807e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:17.204 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:17.192+0000 7f1277fff700 1 -- 172.21.15.42:0/4174017403 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12780a0710 con 0x7f127808df80 2019-12-05T06:31:17.205 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:17.192+0000 7f127d528700 1 --2- 172.21.15.42:0/4174017403 >> v2:172.21.15.42:3300/0 conn(0x7f127809f830 0x7f127814efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:31:17.205 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:17.192+0000 7f1277fff700 1 --2- 172.21.15.42:0/4174017403 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f127808df80 0x7f127809f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f1260008790 tx=0x7f126000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:17.205 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:17.192+0000 7f1277fff700 1 --2- 172.21.15.42:0/4174017403 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f127808df80 0x7f127809f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f1260008790 tx=0x7f126000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:17.205 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:17.192+0000 7f1277fff700 1 --2- 172.21.15.42:0/4174017403 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f127808df80 0x7f127809f2e0 secure :-1 s=READY pgs=603 cs=0 l=1 rx=0x7f1260008790 tx=0x7f126000fc70).ready entity=mon.2 client_cookie=8874adfa26c38fc7 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:31:17.205 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:17.192+0000 7f1277fff700 1 --2- 172.21.15.42:0/4174017403 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f127808df80 0x7f127809f2e0 secure :-1 s=THROTTLE_DONE pgs=603 cs=0 l=1 rx=0x7f1260008790 tx=0x7f126000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:17.205 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:17.192+0000 7f1277fff700 1 --2- 172.21.15.42:0/4174017403 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f127808df80 0x7f127809f2e0 secure :-1 s=THROTTLE_DONE pgs=603 cs=0 l=1 rx=0x7f1260008790 tx=0x7f126000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:17.205 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:17.192+0000 7f1277fff700 1 --2- 172.21.15.42:0/4174017403 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f127808df80 0x7f127809f2e0 secure :-1 s=THROTTLE_DONE pgs=603 cs=0 l=1 rx=0x7f1260008790 tx=0x7f126000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:17.206 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:17.192+0000 7f12777fe700 1 -- 172.21.15.42:0/4174017403 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1260008bb0 con 0x7f127808df80 2019-12-05T06:31:17.206 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:17.192+0000 7f12777fe700 1 -- 172.21.15.42:0/4174017403 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12600087d0 con 0x7f127808df80 2019-12-05T06:31:17.206 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:17.192+0000 7f12777fe700 1 -- 172.21.15.42:0/4174017403 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1260011860 con 0x7f127808df80 2019-12-05T06:31:17.206 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:17.192+0000 7f127e78a700 1 -- 172.21.15.42:0/4174017403 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f127808df80 msgr2=0x7f127809f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:17.206 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:17.192+0000 7f127e78a700 1 --2- 172.21.15.42:0/4174017403 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f127808df80 0x7f127809f2e0 secure :-1 s=READY pgs=603 cs=0 l=1 rx=0x7f1260008790 tx=0x7f126000fc70).stop 2019-12-05T06:31:17.206 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:17.192+0000 7f127e78a700 1 -- 172.21.15.42:0/4174017403 shutdown_connections 2019-12-05T06:31:17.206 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:17.192+0000 7f127e78a700 1 --2- 172.21.15.42:0/4174017403 >> v2:172.21.15.42:3300/0 conn(0x7f127809f830 0x7f127814efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:31:17.207 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:17.192+0000 7f127e78a700 1 --2- 172.21.15.42:0/4174017403 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f127808df80 0x7f127809f2e0 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:31:17.207 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:17.192+0000 7f127e78a700 1 -- 172.21.15.42:0/4174017403 shutdown_connections 2019-12-05T06:31:17.207 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:17.192+0000 7f127e78a700 1 -- 172.21.15.42:0/4174017403 wait complete. 2019-12-05T06:31:17.207 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:17.192+0000 7f127e78a700 1 -- 172.21.15.42:0/4174017403 >> 172.21.15.42:0/4174017403 conn(0x7f12780aabe0 msgr2=0x7f12780a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:31:17.207 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:17.196+0000 7f127e78a700 1 Processor -- start 2019-12-05T06:31:17.207 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:17.196+0000 7f127e78a700 1 -- start start 2019-12-05T06:31:17.210 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:17.196+0000 7f127e78a700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f12780a4ec0 0x7f12780a52f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:31:17.210 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:17.196+0000 7f127e78a700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12780a5840 con 0x7f127812caf0 2019-12-05T06:31:17.210 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:17.196+0000 7f127e78a700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f127812cf20 con 0x7f12781354f0 2019-12-05T06:31:17.210 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:17.196+0000 7f127e78a700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f127812d320 con 0x7f12780a4ec0 2019-12-05T06:31:17.211 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:17.196+0000 7f127cd27700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f127812caf0 0x7f12780a1e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:38128/0 (socket says 172.21.15.42:38128) 2019-12-05T06:31:17.211 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:17.196+0000 7f127cd27700 1 -- 172.21.15.42:0/708309494 learned_addr learned my addr 172.21.15.42:0/708309494 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:31:17.211 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:17.196+0000 7f1277fff700 1 --2- 172.21.15.42:0/708309494 >> v2:172.21.15.42:3300/0 conn(0x7f12780a4ec0 0x7f12780a52f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:31:17.211 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:17.196+0000 7f12757fa700 1 -- 172.21.15.42:0/708309494 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3514828414 0 0) 0x7f12780a5840 con 0x7f127812caf0 2019-12-05T06:31:17.211 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:17.196+0000 7f12757fa700 1 -- 172.21.15.42:0/708309494 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f125c0030b0 con 0x7f127812caf0 2019-12-05T06:31:17.211 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:17.196+0000 7f12757fa700 1 -- 172.21.15.42:0/708309494 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 3669402573 0 0) 0x7f125c0030b0 con 0x7f127812caf0 2019-12-05T06:31:17.211 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:17.196+0000 7f1277fff700 1 -- 172.21.15.42:0/708309494 >> v1:172.21.15.42:6789/0 conn(0x7f12781354f0 legacy=0x7f1278137930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:17.212 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:17.196+0000 7f1277fff700 1 -- 172.21.15.42:0/708309494 >> v1:172.21.15.40:6789/0 conn(0x7f127812caf0 legacy=0x7f12780a1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:17.212 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:17.196+0000 7f1277fff700 1 -- 172.21.15.42:0/708309494 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12780a4340 con 0x7f12780a4ec0 2019-12-05T06:31:17.212 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:17.196+0000 7f1277fff700 1 --2- 172.21.15.42:0/708309494 >> v2:172.21.15.42:3300/0 conn(0x7f12780a4ec0 0x7f12780a52f0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f1268001640 tx=0x7f1268001b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:17.212 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:17.196+0000 7f1277fff700 1 --2- 172.21.15.42:0/708309494 >> v2:172.21.15.42:3300/0 conn(0x7f12780a4ec0 0x7f12780a52f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f1268001640 tx=0x7f1268001b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:17.212 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:17.200+0000 7f1277fff700 1 --2- 172.21.15.42:0/708309494 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f12780a4ec0 0x7f12780a52f0 secure :-1 s=READY pgs=2767 cs=0 l=1 rx=0x7f1268001640 tx=0x7f1268001b70).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:31:17.212 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:17.200+0000 7f1277fff700 1 --2- 172.21.15.42:0/708309494 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f12780a4ec0 0x7f12780a52f0 secure :-1 s=THROTTLE_DONE pgs=2767 cs=0 l=1 rx=0x7f1268001640 tx=0x7f1268001b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:17.212 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:17.200+0000 7f12757fa700 1 -- 172.21.15.42:0/708309494 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1268001930 con 0x7f12780a4ec0 2019-12-05T06:31:17.213 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:17.200+0000 7f1277fff700 1 --2- 172.21.15.42:0/708309494 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f12780a4ec0 0x7f12780a52f0 secure :-1 s=THROTTLE_DONE pgs=2767 cs=0 l=1 rx=0x7f1268001640 tx=0x7f1268001b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:17.213 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:17.200+0000 7f12757fa700 1 -- 172.21.15.42:0/708309494 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1268001740 con 0x7f12780a4ec0 2019-12-05T06:31:17.213 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:17.200+0000 7f1277fff700 1 --2- 172.21.15.42:0/708309494 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f12780a4ec0 0x7f12780a52f0 secure :-1 s=THROTTLE_DONE pgs=2767 cs=0 l=1 rx=0x7f1268001640 tx=0x7f1268001b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:17.213 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:17.200+0000 7f12757fa700 1 -- 172.21.15.42:0/708309494 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1260011670 con 0x7f12780a4ec0 2019-12-05T06:31:17.213 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:17.200+0000 7f127e78a700 1 -- 172.21.15.42:0/708309494 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12780a2630 con 0x7f12780a4ec0 2019-12-05T06:31:17.213 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:17.200+0000 7f127e78a700 1 -- 172.21.15.42:0/708309494 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f127808eb70 con 0x7f12780a4ec0 2019-12-05T06:31:17.213 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:17.200+0000 7f1277fff700 1 --2- 172.21.15.42:0/708309494 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f12780a4ec0 0x7f12780a52f0 secure :-1 s=THROTTLE_DONE pgs=2767 cs=0 l=1 rx=0x7f1268001640 tx=0x7f1268001b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:17.214 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:17.200+0000 7f12757fa700 1 -- 172.21.15.42:0/708309494 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f1260011910 con 0x7f12780a4ec0 2019-12-05T06:31:17.215 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:17.200+0000 7f1277fff700 1 --2- 172.21.15.42:0/708309494 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f12780a4ec0 0x7f12780a52f0 secure :-1 s=THROTTLE_DONE pgs=2767 cs=0 l=1 rx=0x7f1268001640 tx=0x7f1268001b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:17.215 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:17.200+0000 7f12757fa700 1 --2- 172.21.15.42:0/708309494 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f125c03b1b0 0x7f125c03d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:31:17.215 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:17.200+0000 7f12757fa700 1 -- 172.21.15.42:0/708309494 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 11191+0+0 (secure 0 0 0) 0x7f126004b010 con 0x7f12780a4ec0 2019-12-05T06:31:17.216 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:17.200+0000 7f127cd27700 1 --2- 172.21.15.42:0/708309494 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f125c03b1b0 0x7f125c03d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:31:17.216 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:17.200+0000 7f127cd27700 1 --2- 172.21.15.42:0/708309494 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f125c03b1b0 0x7f125c03d6c0 crc :-1 s=READY pgs=1160 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:31:17.216 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:17.208+0000 7f127e78a700 1 -- 172.21.15.42:0/708309494 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f127805d110 con 0x7f12780a4ec0 2019-12-05T06:31:17.219 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:17.212+0000 7f1277fff700 1 --2- 172.21.15.42:0/708309494 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f12780a4ec0 0x7f12780a52f0 secure :-1 s=THROTTLE_DONE pgs=2767 cs=0 l=1 rx=0x7f1268001640 tx=0x7f1268001b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:17.220 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:17.212+0000 7f12757fa700 1 -- 172.21.15.42:0/708309494 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f12600322d0 con 0x7f12780a4ec0 2019-12-05T06:31:17.398 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:17.392+0000 7f127e78a700 1 -- 172.21.15.42:0/708309494 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f127805b8c0 con 0x7f12780a4ec0 2019-12-05T06:31:17.399 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:17.392+0000 7f1277fff700 1 --2- 172.21.15.42:0/708309494 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f12780a4ec0 0x7f12780a52f0 secure :-1 s=THROTTLE_DONE pgs=2767 cs=0 l=1 rx=0x7f1268001640 tx=0x7f1268001b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:17.399 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:17.392+0000 7f12757fa700 1 -- 172.21.15.42:0/708309494 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 329 v329) v1 ==== 95+0+1778 (secure 0 0 0) 0x7f1260007f10 con 0x7f12780a4ec0 2019-12-05T06:31:17.399 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 329 2019-12-05T06:31:17.399 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:31:17.399 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":329,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":329,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:31:14.986780+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":109,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11352},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11352":{"gid":11352,"name":"b","rank":0,"incarnation":329,"state":"up:replay","state_seq":1,"addr":"172.21.15.40:6833/2055706790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2055706790},{"type":"v1","addr":"172.21.15.40:6833","nonce":2055706790}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:31:17.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:17.396+0000 7f127e78a700 1 -- 172.21.15.42:0/708309494 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f125c03b1b0 msgr2=0x7f125c03d6c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:17.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:17.396+0000 7f127e78a700 1 --2- 172.21.15.42:0/708309494 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f125c03b1b0 0x7f125c03d6c0 crc :-1 s=READY pgs=1160 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:31:17.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:17.396+0000 7f127e78a700 1 -- 172.21.15.42:0/708309494 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f12780a4ec0 msgr2=0x7f12780a52f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:17.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:17.396+0000 7f127e78a700 1 --2- 172.21.15.42:0/708309494 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f12780a4ec0 0x7f12780a52f0 secure :-1 s=READY pgs=2767 cs=0 l=1 rx=0x7f1268001640 tx=0x7f1268001b70).stop 2019-12-05T06:31:17.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:17.396+0000 7f127e78a700 1 -- 172.21.15.42:0/708309494 shutdown_connections 2019-12-05T06:31:17.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:17.396+0000 7f127e78a700 1 --2- 172.21.15.42:0/708309494 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f125c03b1b0 0x7f125c03d6c0 unknown :-1 s=CLOSED pgs=1160 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:31:17.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:17.396+0000 7f127e78a700 1 --2- 172.21.15.42:0/708309494 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f12780a4ec0 0x7f12780a52f0 unknown :-1 s=CLOSED pgs=2767 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:31:17.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:17.396+0000 7f127e78a700 1 -- 172.21.15.42:0/708309494 shutdown_connections 2019-12-05T06:31:17.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:17.396+0000 7f127e78a700 1 -- 172.21.15.42:0/708309494 wait complete. 2019-12-05T06:31:17.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:17.396+0000 7f127e78a700 1 -- 172.21.15.42:0/708309494 >> 172.21.15.42:0/708309494 conn(0x7f12780aabe0 msgr2=0x7f12780a9790 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:31:17.412 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 329, "feature_flags": { "enable_multiple": false, "ever_enabled_multiple": false }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2019-12-05T05:03:35.604738+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 329, "ever_allowed_features": 2, "explicitly_allowed_features": 2, "failed": [], "flags": 18, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_11352": { "addr": "172.21.15.40:6833/2055706790", "addrs": { "addrvec": [ { "addr": "172.21.15.40:6832", "nonce": 2055706790, "type": "v2" }, { "addr": "172.21.15.40:6833", "nonce": 2055706790, "type": "v1" } ] }, "export_targets": [], "features": 4540138292836696063, "flags": 0, "gid": 11352, "incarnation": 329, "name": "b", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 109, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 2, "min_compat_client": "0 (unknown)", "modified": "2019-12-05T06:31:14.986780+0000", "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 11352 } } } ], "standbys": [] } 2019-12-05T06:31:19.413 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:31:19.524 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:19.512+0000 7f1d3fd92700 1 Processor -- start 2019-12-05T06:31:19.525 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:19.512+0000 7f1d3fd92700 1 -- start start 2019-12-05T06:31:19.525 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:19.512+0000 7f1d3fd92700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f1d380a7740 0x7f1d3808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:31:19.525 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:19.512+0000 7f1d3fd92700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f1d3809fae0 0x7f1d3807eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:31:19.525 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:19.512+0000 7f1d3fd92700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d380a0710 con 0x7f1d3807e720 2019-12-05T06:31:19.525 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:19.512+0000 7f1d3fd92700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d380a0b00 con 0x7f1d380a7740 2019-12-05T06:31:19.525 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:19.512+0000 7f1d3fd92700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d3808e5b0 con 0x7f1d3809fae0 2019-12-05T06:31:19.526 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:19.512+0000 7f1d3eb30700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f1d3809fae0 0x7f1d3807eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:31:19.526 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:19.512+0000 7f1d3eb30700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f1d3809fae0 0x7f1d3807eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:56894/0 (socket says 172.21.15.42:56894) 2019-12-05T06:31:19.526 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:19.512+0000 7f1d3eb30700 1 -- 172.21.15.42:0/42883669 learned_addr learned my addr 172.21.15.42:0/42883669 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:31:19.526 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:19.512+0000 7f1d3e32f700 1 --2- 172.21.15.42:0/42883669 >> v2:172.21.15.40:3300/0 conn(0x7f1d380a7740 0x7f1d3808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:31:19.526 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:19.512+0000 7f1d3eb30700 1 -- 172.21.15.42:0/42883669 >> v2:172.21.15.40:3300/0 conn(0x7f1d380a7740 msgr2=0x7f1d3808df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:31:19.526 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:19.512+0000 7f1d3eb30700 1 --2- 172.21.15.42:0/42883669 >> v2:172.21.15.40:3300/0 conn(0x7f1d380a7740 0x7f1d3808df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:31:19.526 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:19.512+0000 7f1d3eb30700 1 -- 172.21.15.42:0/42883669 >> v1:172.21.15.40:6789/0 conn(0x7f1d3807e720 legacy=0x7f1d3809f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:19.527 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:19.512+0000 7f1d3eb30700 1 -- 172.21.15.42:0/42883669 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d3807f0a0 con 0x7f1d3809fae0 2019-12-05T06:31:19.527 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:19.516+0000 7f1d3eb30700 1 --2- 172.21.15.42:0/42883669 >> v2:172.21.15.42:3300/0 conn(0x7f1d3809fae0 0x7f1d3807eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f1d300032b0 tx=0x7f1d300100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:19.527 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:19.516+0000 7f1d3eb30700 1 --2- 172.21.15.42:0/42883669 >> v2:172.21.15.42:3300/0 conn(0x7f1d3809fae0 0x7f1d3807eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f1d300032b0 tx=0x7f1d300100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:19.527 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:19.516+0000 7f1d3eb30700 1 --2- 172.21.15.42:0/42883669 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1d3809fae0 0x7f1d3807eb50 secure :-1 s=READY pgs=2768 cs=0 l=1 rx=0x7f1d300032b0 tx=0x7f1d300100b0).ready entity=mon.1 client_cookie=86e4759a663a6412 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:31:19.527 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:19.516+0000 7f1d3eb30700 1 --2- 172.21.15.42:0/42883669 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1d3809fae0 0x7f1d3807eb50 secure :-1 s=THROTTLE_DONE pgs=2768 cs=0 l=1 rx=0x7f1d300032b0 tx=0x7f1d300100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:19.527 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:19.516+0000 7f1d3eb30700 1 --2- 172.21.15.42:0/42883669 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1d3809fae0 0x7f1d3807eb50 secure :-1 s=THROTTLE_DONE pgs=2768 cs=0 l=1 rx=0x7f1d300032b0 tx=0x7f1d300100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:19.527 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:19.516+0000 7f1d3d32d700 1 -- 172.21.15.42:0/42883669 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1d30010d60 con 0x7f1d3809fae0 2019-12-05T06:31:19.528 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:19.516+0000 7f1d3eb30700 1 --2- 172.21.15.42:0/42883669 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1d3809fae0 0x7f1d3807eb50 secure :-1 s=THROTTLE_DONE pgs=2768 cs=0 l=1 rx=0x7f1d300032b0 tx=0x7f1d300100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:19.528 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:19.516+0000 7f1d3d32d700 1 -- 172.21.15.42:0/42883669 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d30010980 con 0x7f1d3809fae0 2019-12-05T06:31:19.528 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:19.516+0000 7f1d3d32d700 1 -- 172.21.15.42:0/42883669 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1d300114a0 con 0x7f1d3809fae0 2019-12-05T06:31:19.528 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:19.516+0000 7f1d3fd92700 1 -- 172.21.15.42:0/42883669 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1d3809fae0 msgr2=0x7f1d3807eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:19.528 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:19.516+0000 7f1d3fd92700 1 --2- 172.21.15.42:0/42883669 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1d3809fae0 0x7f1d3807eb50 secure :-1 s=READY pgs=2768 cs=0 l=1 rx=0x7f1d300032b0 tx=0x7f1d300100b0).stop 2019-12-05T06:31:19.528 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:19.516+0000 7f1d3fd92700 1 -- 172.21.15.42:0/42883669 shutdown_connections 2019-12-05T06:31:19.528 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:19.516+0000 7f1d3fd92700 1 --2- 172.21.15.42:0/42883669 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1d3809fae0 0x7f1d3807eb50 unknown :-1 s=CLOSED pgs=2768 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:31:19.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:19.516+0000 7f1d3fd92700 1 --2- 172.21.15.42:0/42883669 >> v2:172.21.15.40:3300/0 conn(0x7f1d380a7740 0x7f1d3808df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:31:19.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:19.516+0000 7f1d3fd92700 1 -- 172.21.15.42:0/42883669 shutdown_connections 2019-12-05T06:31:19.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:19.516+0000 7f1d3fd92700 1 -- 172.21.15.42:0/42883669 wait complete. 2019-12-05T06:31:19.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:19.516+0000 7f1d3fd92700 1 -- 172.21.15.42:0/42883669 >> 172.21.15.42:0/42883669 conn(0x7f1d380aabe0 msgr2=0x7f1d380a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:31:19.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:19.520+0000 7f1d3fd92700 1 Processor -- start 2019-12-05T06:31:19.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:19.520+0000 7f1d3fd92700 1 -- start start 2019-12-05T06:31:19.532 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:19.520+0000 7f1d3fd92700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f1d38119520 0x7f1d381168c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:31:19.532 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:19.520+0000 7f1d3fd92700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1d38116e10 0x7f1d38136ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:31:19.532 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:19.520+0000 7f1d3fd92700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d380a6af0 con 0x7f1d38118360 2019-12-05T06:31:19.532 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:19.520+0000 7f1d3fd92700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d38116660 con 0x7f1d38119520 2019-12-05T06:31:19.532 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:19.520+0000 7f1d3fd92700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d38119e90 con 0x7f1d38116e10 2019-12-05T06:31:19.533 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:19.520+0000 7f1d3db2e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f1d38119520 0x7f1d381168c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:31:19.533 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:19.520+0000 7f1d3e32f700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f1d38118360 0x7f1d3812fd20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:56936/0 (socket says 172.21.15.42:56936) 2019-12-05T06:31:19.533 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:19.520+0000 7f1d3db2e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f1d38119520 0x7f1d381168c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:56900/0 (socket says 172.21.15.42:56900) 2019-12-05T06:31:19.533 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:19.520+0000 7f1d3e32f700 1 -- 172.21.15.42:0/1522095549 learned_addr learned my addr 172.21.15.42:0/1522095549 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:31:19.533 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:19.520+0000 7f1d3eb30700 1 --2- 172.21.15.42:0/1522095549 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1d38116e10 0x7f1d38136ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:31:19.533 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:19.520+0000 7f1d3db2e700 1 -- 172.21.15.42:0/1522095549 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1d38116e10 msgr2=0x7f1d38136ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:19.533 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:19.520+0000 7f1d3db2e700 1 --2- 172.21.15.42:0/1522095549 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1d38116e10 0x7f1d38136ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:31:19.534 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:19.520+0000 7f1d3db2e700 1 -- 172.21.15.42:0/1522095549 >> v1:172.21.15.42:6789/0 conn(0x7f1d38118360 legacy=0x7f1d3812fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:19.534 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:19.520+0000 7f1d3db2e700 1 -- 172.21.15.42:0/1522095549 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d38117240 con 0x7f1d38119520 2019-12-05T06:31:19.534 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:19.520+0000 7f1d3db2e700 1 --2- 172.21.15.42:0/1522095549 >> v2:172.21.15.42:3300/0 conn(0x7f1d38119520 0x7f1d381168c0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f1d3811b560 tx=0x7f1d2800c320).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:19.534 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:19.524+0000 7f1d3db2e700 1 --2- 172.21.15.42:0/1522095549 >> v2:172.21.15.42:3300/0 conn(0x7f1d38119520 0x7f1d381168c0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f1d3811b560 tx=0x7f1d2800c320).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:19.534 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:19.524+0000 7f1d3db2e700 1 --2- 172.21.15.42:0/1522095549 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1d38119520 0x7f1d381168c0 secure :-1 s=READY pgs=2770 cs=0 l=1 rx=0x7f1d3811b560 tx=0x7f1d2800c320).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:31:19.534 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:19.524+0000 7f1d3db2e700 1 --2- 172.21.15.42:0/1522095549 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1d38119520 0x7f1d381168c0 secure :-1 s=THROTTLE_DONE pgs=2770 cs=0 l=1 rx=0x7f1d3811b560 tx=0x7f1d2800c320).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:19.535 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:19.524+0000 7f1d3db2e700 1 --2- 172.21.15.42:0/1522095549 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1d38119520 0x7f1d381168c0 secure :-1 s=THROTTLE_DONE pgs=2770 cs=0 l=1 rx=0x7f1d3811b560 tx=0x7f1d2800c320).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:19.535 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:19.524+0000 7f1d2effd700 1 -- 172.21.15.42:0/1522095549 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1d2800ca80 con 0x7f1d38119520 2019-12-05T06:31:19.535 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:19.524+0000 7f1d3db2e700 1 --2- 172.21.15.42:0/1522095549 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1d38119520 0x7f1d381168c0 secure :-1 s=THROTTLE_DONE pgs=2770 cs=0 l=1 rx=0x7f1d3811b560 tx=0x7f1d2800c320).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:19.535 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:19.524+0000 7f1d2effd700 1 -- 172.21.15.42:0/1522095549 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d2800c6a0 con 0x7f1d38119520 2019-12-05T06:31:19.535 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:19.524+0000 7f1d2effd700 1 -- 172.21.15.42:0/1522095549 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1d280110e0 con 0x7f1d38119520 2019-12-05T06:31:19.535 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:19.524+0000 7f1d3fd92700 1 -- 172.21.15.42:0/1522095549 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d38130520 con 0x7f1d38119520 2019-12-05T06:31:19.535 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:19.524+0000 7f1d3fd92700 1 -- 172.21.15.42:0/1522095549 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d381179e0 con 0x7f1d38119520 2019-12-05T06:31:19.536 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:19.524+0000 7f1d3db2e700 1 --2- 172.21.15.42:0/1522095549 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1d38119520 0x7f1d381168c0 secure :-1 s=THROTTLE_DONE pgs=2770 cs=0 l=1 rx=0x7f1d3811b560 tx=0x7f1d2800c320).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:19.536 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:19.524+0000 7f1d2effd700 1 -- 172.21.15.42:0/1522095549 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f1d280113f0 con 0x7f1d38119520 2019-12-05T06:31:19.536 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:19.524+0000 7f1d3db2e700 1 --2- 172.21.15.42:0/1522095549 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1d38119520 0x7f1d381168c0 secure :-1 s=THROTTLE_DONE pgs=2770 cs=0 l=1 rx=0x7f1d3811b560 tx=0x7f1d2800c320).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:19.536 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:19.524+0000 7f1d2effd700 1 --2- 172.21.15.42:0/1522095549 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f1d2003ae00 0x7f1d2003d310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:31:19.536 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:19.524+0000 7f1d2effd700 1 -- 172.21.15.42:0/1522095549 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 11191+0+0 (secure 0 0 0) 0x7f1d2804b190 con 0x7f1d38119520 2019-12-05T06:31:19.536 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:19.524+0000 7f1d3e32f700 1 --2- 172.21.15.42:0/1522095549 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f1d2003ae00 0x7f1d2003d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:31:19.536 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:19.524+0000 7f1d3e32f700 1 --2- 172.21.15.42:0/1522095549 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f1d2003ae00 0x7f1d2003d310 crc :-1 s=READY pgs=1161 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:31:19.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:19.528+0000 7f1d3fd92700 1 -- 172.21.15.42:0/1522095549 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d3805d110 con 0x7f1d38119520 2019-12-05T06:31:19.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:19.532+0000 7f1d3db2e700 1 --2- 172.21.15.42:0/1522095549 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1d38119520 0x7f1d381168c0 secure :-1 s=THROTTLE_DONE pgs=2770 cs=0 l=1 rx=0x7f1d3811b560 tx=0x7f1d2800c320).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:19.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:19.532+0000 7f1d2effd700 1 -- 172.21.15.42:0/1522095549 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f1d28032210 con 0x7f1d38119520 2019-12-05T06:31:19.703 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:19.696+0000 7f1d3fd92700 1 -- 172.21.15.42:0/1522095549 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1d3805b8c0 con 0x7f1d38119520 2019-12-05T06:31:19.704 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:19.696+0000 7f1d3db2e700 1 --2- 172.21.15.42:0/1522095549 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1d38119520 0x7f1d381168c0 secure :-1 s=THROTTLE_DONE pgs=2770 cs=0 l=1 rx=0x7f1d3811b560 tx=0x7f1d2800c320).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:19.704 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:19.696+0000 7f1d2effd700 1 -- 172.21.15.42:0/1522095549 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 331 v331) v1 ==== 95+0+1779 (secure 0 0 0) 0x7f1d28032470 con 0x7f1d38119520 2019-12-05T06:31:19.706 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 331 2019-12-05T06:31:19.706 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:31:19.706 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":331,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":331,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:31:19.132603+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":109,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11352},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11352":{"gid":11352,"name":"b","rank":0,"incarnation":329,"state":"up:rejoin","state_seq":15,"addr":"172.21.15.40:6833/2055706790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2055706790},{"type":"v1","addr":"172.21.15.40:6833","nonce":2055706790}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:31:19.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:19.704+0000 7f1d3fd92700 1 -- 172.21.15.42:0/1522095549 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f1d2003ae00 msgr2=0x7f1d2003d310 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:19.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:19.704+0000 7f1d3fd92700 1 --2- 172.21.15.42:0/1522095549 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f1d2003ae00 0x7f1d2003d310 crc :-1 s=READY pgs=1161 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:31:19.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:19.704+0000 7f1d3fd92700 1 -- 172.21.15.42:0/1522095549 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1d38119520 msgr2=0x7f1d381168c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:19.711 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:19.704+0000 7f1d3fd92700 1 --2- 172.21.15.42:0/1522095549 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1d38119520 0x7f1d381168c0 secure :-1 s=READY pgs=2770 cs=0 l=1 rx=0x7f1d3811b560 tx=0x7f1d2800c320).stop 2019-12-05T06:31:19.711 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:19.704+0000 7f1d3fd92700 1 -- 172.21.15.42:0/1522095549 shutdown_connections 2019-12-05T06:31:19.711 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:19.704+0000 7f1d3fd92700 1 --2- 172.21.15.42:0/1522095549 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f1d2003ae00 0x7f1d2003d310 unknown :-1 s=CLOSED pgs=1161 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:31:19.711 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:19.704+0000 7f1d3fd92700 1 --2- 172.21.15.42:0/1522095549 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1d38116e10 0x7f1d38136ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:31:19.711 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:19.704+0000 7f1d3fd92700 1 --2- 172.21.15.42:0/1522095549 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1d38119520 0x7f1d381168c0 unknown :-1 s=CLOSED pgs=2770 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:31:19.711 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:19.704+0000 7f1d3fd92700 1 -- 172.21.15.42:0/1522095549 shutdown_connections 2019-12-05T06:31:19.711 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:19.704+0000 7f1d3fd92700 1 -- 172.21.15.42:0/1522095549 wait complete. 2019-12-05T06:31:19.712 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:19.704+0000 7f1d3fd92700 1 -- 172.21.15.42:0/1522095549 >> 172.21.15.42:0/1522095549 conn(0x7f1d380aabe0 msgr2=0x7f1d380a7f90 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:31:19.724 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2019-12-05T06:31:21.725 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:31:21.831 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:21.820+0000 7f5dce809700 1 Processor -- start 2019-12-05T06:31:21.832 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:21.820+0000 7f5dce809700 1 -- start start 2019-12-05T06:31:21.832 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:21.820+0000 7f5dce809700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5dc80a7740 0x7f5dc808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:31:21.832 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:21.820+0000 7f5dce809700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5dc807e720 0x7f5dc807eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:31:21.832 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:21.820+0000 7f5dce809700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5dc809f2e0 0x7f5dc809f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:31:21.832 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:21.820+0000 7f5dce809700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5dc80a0b00 con 0x7f5dc807e720 2019-12-05T06:31:21.833 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:21.820+0000 7f5dce809700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5dc808e5b0 con 0x7f5dc80a7740 2019-12-05T06:31:21.833 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:21.820+0000 7f5dce809700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5dc808e850 con 0x7f5dc809f2e0 2019-12-05T06:31:21.833 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:21.820+0000 7f5dccda6700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5dc80a7740 0x7f5dc808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:31:21.833 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:21.820+0000 7f5dccda6700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5dc80a7740 0x7f5dc808df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:56912/0 (socket says 172.21.15.42:56912) 2019-12-05T06:31:21.833 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:21.820+0000 7f5dccda6700 1 -- 172.21.15.42:0/1173323260 learned_addr learned my addr 172.21.15.42:0/1173323260 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:31:21.833 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:21.820+0000 7f5dc7fff700 1 --2- 172.21.15.42:0/1173323260 >> v2:172.21.15.40:3300/0 conn(0x7f5dc807e720 0x7f5dc807eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:31:21.833 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:21.820+0000 7f5dcd5a7700 1 --2- 172.21.15.42:0/1173323260 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5dc809f2e0 0x7f5dc809f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:31:21.834 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:21.824+0000 7f5dccda6700 1 -- 172.21.15.42:0/1173323260 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5dc809f2e0 msgr2=0x7f5dc809f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:31:21.834 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:21.824+0000 7f5dccda6700 1 --2- 172.21.15.42:0/1173323260 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5dc809f2e0 0x7f5dc809f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:31:21.834 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:21.824+0000 7f5dccda6700 1 -- 172.21.15.42:0/1173323260 >> v2:172.21.15.40:3300/0 conn(0x7f5dc807e720 msgr2=0x7f5dc807eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:31:21.834 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:21.824+0000 7f5dccda6700 1 --2- 172.21.15.42:0/1173323260 >> v2:172.21.15.40:3300/0 conn(0x7f5dc807e720 0x7f5dc807eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:31:21.834 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:21.824+0000 7f5dccda6700 1 -- 172.21.15.42:0/1173323260 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dc807f0a0 con 0x7f5dc80a7740 2019-12-05T06:31:21.834 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:21.824+0000 7f5dc7fff700 1 --2- 172.21.15.42:0/1173323260 >> v2:172.21.15.40:3300/0 conn(0x7f5dc807e720 0x7f5dc807eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:31:21.834 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:21.824+0000 7f5dccda6700 1 --2- 172.21.15.42:0/1173323260 >> v2:172.21.15.42:3300/0 conn(0x7f5dc80a7740 0x7f5dc808df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f5dc80a0f50 tx=0x7f5db800fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:21.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:21.824+0000 7f5dccda6700 1 --2- 172.21.15.42:0/1173323260 >> v2:172.21.15.42:3300/0 conn(0x7f5dc80a7740 0x7f5dc808df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f5dc80a0f50 tx=0x7f5db800fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:21.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:21.824+0000 7f5dccda6700 1 --2- 172.21.15.42:0/1173323260 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5dc80a7740 0x7f5dc808df80 secure :-1 s=READY pgs=2772 cs=0 l=1 rx=0x7f5dc80a0f50 tx=0x7f5db800fdb0).ready entity=mon.1 client_cookie=8dbac0a20890f1c0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:31:21.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:21.824+0000 7f5dccda6700 1 --2- 172.21.15.42:0/1173323260 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5dc80a7740 0x7f5dc808df80 secure :-1 s=THROTTLE_DONE pgs=2772 cs=0 l=1 rx=0x7f5dc80a0f50 tx=0x7f5db800fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:21.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:21.824+0000 7f5dc77fe700 1 -- 172.21.15.42:0/1173323260 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5db8008c90 con 0x7f5dc80a7740 2019-12-05T06:31:21.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:21.824+0000 7f5dccda6700 1 --2- 172.21.15.42:0/1173323260 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5dc80a7740 0x7f5dc808df80 secure :-1 s=THROTTLE_DONE pgs=2772 cs=0 l=1 rx=0x7f5dc80a0f50 tx=0x7f5db800fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:21.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:21.824+0000 7f5dccda6700 1 --2- 172.21.15.42:0/1173323260 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5dc80a7740 0x7f5dc808df80 secure :-1 s=THROTTLE_DONE pgs=2772 cs=0 l=1 rx=0x7f5dc80a0f50 tx=0x7f5db800fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:21.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:21.824+0000 7f5dc77fe700 1 -- 172.21.15.42:0/1173323260 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5db80088b0 con 0x7f5dc80a7740 2019-12-05T06:31:21.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:21.824+0000 7f5dc77fe700 1 -- 172.21.15.42:0/1173323260 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5db8011b50 con 0x7f5dc80a7740 2019-12-05T06:31:21.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:21.824+0000 7f5dce809700 1 -- 172.21.15.42:0/1173323260 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5dc80a7740 msgr2=0x7f5dc808df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:21.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:21.824+0000 7f5dce809700 1 --2- 172.21.15.42:0/1173323260 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5dc80a7740 0x7f5dc808df80 secure :-1 s=READY pgs=2772 cs=0 l=1 rx=0x7f5dc80a0f50 tx=0x7f5db800fdb0).stop 2019-12-05T06:31:21.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:21.824+0000 7f5dce809700 1 -- 172.21.15.42:0/1173323260 shutdown_connections 2019-12-05T06:31:21.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:21.824+0000 7f5dce809700 1 --2- 172.21.15.42:0/1173323260 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5dc809f2e0 0x7f5dc809f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:31:21.848 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:21.824+0000 7f5dce809700 1 --2- 172.21.15.42:0/1173323260 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5dc80a7740 0x7f5dc808df80 unknown :-1 s=CLOSED pgs=2772 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:31:21.849 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:21.824+0000 7f5dce809700 1 --2- 172.21.15.42:0/1173323260 >> v2:172.21.15.40:3300/0 conn(0x7f5dc807e720 0x7f5dc807eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:31:21.849 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:21.824+0000 7f5dce809700 1 -- 172.21.15.42:0/1173323260 shutdown_connections 2019-12-05T06:31:21.849 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:21.824+0000 7f5dce809700 1 -- 172.21.15.42:0/1173323260 wait complete. 2019-12-05T06:31:21.849 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:21.824+0000 7f5dce809700 1 -- 172.21.15.42:0/1173323260 >> 172.21.15.42:0/1173323260 conn(0x7f5dc80aabe0 msgr2=0x7f5dc80a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:31:21.852 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:21.828+0000 7f5dce809700 1 Processor -- start 2019-12-05T06:31:21.853 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:21.828+0000 7f5dce809700 1 -- start start 2019-12-05T06:31:21.853 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:21.828+0000 7f5dce809700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5dc808eeb0 0x7f5dc81481d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:31:21.853 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:21.828+0000 7f5dce809700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5dc8065200 0x7f5dc807bc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:31:21.853 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:21.828+0000 7f5dce809700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5dc8065a20 con 0x7f5dc807b820 2019-12-05T06:31:21.853 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:21.828+0000 7f5dce809700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5dc807c1a0 con 0x7f5dc808eeb0 2019-12-05T06:31:21.853 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:21.828+0000 7f5dce809700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5dc8148820 con 0x7f5dc8065200 2019-12-05T06:31:21.853 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:21.828+0000 7f5dccda6700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5dc808eeb0 0x7f5dc81481d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:31:21.854 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:21.828+0000 7f5dcd5a7700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5dc8065200 0x7f5dc807bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:31:21.854 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:21.828+0000 7f5dccda6700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5dc808eeb0 0x7f5dc81481d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.42:44732/0 (socket says 172.21.15.42:44732) 2019-12-05T06:31:21.854 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:21.828+0000 7f5dccda6700 1 -- 172.21.15.42:0/286045105 learned_addr learned my addr 172.21.15.42:0/286045105 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:31:21.854 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:21.828+0000 7f5dccda6700 1 -- 172.21.15.42:0/286045105 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5dc8065200 msgr2=0x7f5dc807bc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:21.854 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:21.832+0000 7f5dccda6700 1 --2- 172.21.15.42:0/286045105 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5dc8065200 0x7f5dc807bc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:31:21.854 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:21.832+0000 7f5dccda6700 1 -- 172.21.15.42:0/286045105 >> v1:172.21.15.40:6789/0 conn(0x7f5dc807b820 legacy=0x7f5dc8064a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:21.854 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:21.832+0000 7f5dc57fa700 1 -- 172.21.15.42:0/286045105 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2776660128 0 0) 0x7f5dc8065a20 con 0x7f5dc807b820 2019-12-05T06:31:21.855 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:21.832+0000 7f5dccda6700 1 -- 172.21.15.42:0/286045105 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dc8145c80 con 0x7f5dc808eeb0 2019-12-05T06:31:21.855 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:21.832+0000 7f5dccda6700 1 --2- 172.21.15.42:0/286045105 >> v2:172.21.15.40:3300/0 conn(0x7f5dc808eeb0 0x7f5dc81481d0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f5dc81489e0 tx=0x7f5db8008b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:21.855 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:21.832+0000 7f5dccda6700 1 --2- 172.21.15.42:0/286045105 >> v2:172.21.15.40:3300/0 conn(0x7f5dc808eeb0 0x7f5dc81481d0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f5dc81489e0 tx=0x7f5db8008b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:21.855 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:21.832+0000 7f5dccda6700 1 --2- 172.21.15.42:0/286045105 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5dc808eeb0 0x7f5dc81481d0 secure :-1 s=READY pgs=2077 cs=0 l=1 rx=0x7f5dc81489e0 tx=0x7f5db8008b70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:31:21.855 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:21.832+0000 7f5dccda6700 1 --2- 172.21.15.42:0/286045105 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5dc808eeb0 0x7f5dc81481d0 secure :-1 s=THROTTLE_DONE pgs=2077 cs=0 l=1 rx=0x7f5dc81489e0 tx=0x7f5db8008b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:21.855 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:21.832+0000 7f5dc57fa700 1 -- 172.21.15.42:0/286045105 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5db800f470 con 0x7f5dc808eeb0 2019-12-05T06:31:21.855 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:21.832+0000 7f5dce809700 1 -- 172.21.15.42:0/286045105 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5dc8137970 con 0x7f5dc808eeb0 2019-12-05T06:31:21.856 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:21.832+0000 7f5dccda6700 1 --2- 172.21.15.42:0/286045105 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5dc808eeb0 0x7f5dc81481d0 secure :-1 s=THROTTLE_DONE pgs=2077 cs=0 l=1 rx=0x7f5dc81489e0 tx=0x7f5db8008b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:21.856 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:21.832+0000 7f5dc57fa700 1 -- 172.21.15.42:0/286045105 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5db8011770 con 0x7f5dc808eeb0 2019-12-05T06:31:21.856 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:21.832+0000 7f5dccda6700 1 --2- 172.21.15.42:0/286045105 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5dc808eeb0 0x7f5dc81481d0 secure :-1 s=THROTTLE_DONE pgs=2077 cs=0 l=1 rx=0x7f5dc81489e0 tx=0x7f5db8008b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:21.856 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:21.832+0000 7f5dce809700 1 -- 172.21.15.42:0/286045105 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5dc8137f20 con 0x7f5dc808eeb0 2019-12-05T06:31:21.859 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:21.832+0000 7f5dc57fa700 1 -- 172.21.15.42:0/286045105 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5db8005730 con 0x7f5dc808eeb0 2019-12-05T06:31:21.859 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:21.832+0000 7f5dccda6700 1 --2- 172.21.15.42:0/286045105 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5dc808eeb0 0x7f5dc81481d0 secure :-1 s=THROTTLE_DONE pgs=2077 cs=0 l=1 rx=0x7f5dc81489e0 tx=0x7f5db8008b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:21.860 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:21.832+0000 7f5dc57fa700 1 -- 172.21.15.42:0/286045105 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f5db80059d0 con 0x7f5dc808eeb0 2019-12-05T06:31:21.860 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:21.832+0000 7f5dccda6700 1 --2- 172.21.15.42:0/286045105 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5dc808eeb0 0x7f5dc81481d0 secure :-1 s=THROTTLE_DONE pgs=2077 cs=0 l=1 rx=0x7f5dc81489e0 tx=0x7f5db8008b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:21.860 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:21.832+0000 7f5dc57fa700 1 --2- 172.21.15.42:0/286045105 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5da803abd0 0x7f5da803d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:31:21.860 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:21.832+0000 7f5dc57fa700 1 -- 172.21.15.42:0/286045105 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 11191+0+0 (secure 0 0 0) 0x7f5db808f120 con 0x7f5dc808eeb0 2019-12-05T06:31:21.860 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:21.832+0000 7f5dc7fff700 1 --2- 172.21.15.42:0/286045105 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5da803abd0 0x7f5da803d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:31:21.860 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:21.832+0000 7f5dc7fff700 1 --2- 172.21.15.42:0/286045105 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5da803abd0 0x7f5da803d0e0 crc :-1 s=READY pgs=1162 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:31:21.860 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:21.836+0000 7f5dce809700 1 -- 172.21.15.42:0/286045105 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5dc805d110 con 0x7f5dc808eeb0 2019-12-05T06:31:21.861 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:21.844+0000 7f5dccda6700 1 --2- 172.21.15.42:0/286045105 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5dc808eeb0 0x7f5dc81481d0 secure :-1 s=THROTTLE_DONE pgs=2077 cs=0 l=1 rx=0x7f5dc81489e0 tx=0x7f5db8008b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:21.861 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:21.844+0000 7f5dc57fa700 1 -- 172.21.15.42:0/286045105 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f5db8076180 con 0x7f5dc808eeb0 2019-12-05T06:31:22.021 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:22.012+0000 7f5dce809700 1 -- 172.21.15.42:0/286045105 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5dc805b8c0 con 0x7f5dc808eeb0 2019-12-05T06:31:22.022 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:22.016+0000 7f5dccda6700 1 --2- 172.21.15.42:0/286045105 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5dc808eeb0 0x7f5dc81481d0 secure :-1 s=THROTTLE_DONE pgs=2077 cs=0 l=1 rx=0x7f5dc81489e0 tx=0x7f5db8008b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:22.022 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:22.016+0000 7f5dc57fa700 1 -- 172.21.15.42:0/286045105 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 332 v332) v1 ==== 95+0+1779 (secure 0 0 0) 0x7f5db807c810 con 0x7f5dc808eeb0 2019-12-05T06:31:22.022 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 332 2019-12-05T06:31:22.022 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:31:22.022 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":332,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":332,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:31:20.143327+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":109,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11352},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11352":{"gid":11352,"name":"b","rank":0,"incarnation":329,"state":"up:active","state_seq":16,"addr":"172.21.15.40:6833/2055706790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2055706790},{"type":"v1","addr":"172.21.15.40:6833","nonce":2055706790}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:31:22.026 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:22.020+0000 7f5dce809700 1 -- 172.21.15.42:0/286045105 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5da803abd0 msgr2=0x7f5da803d0e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:22.026 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:22.020+0000 7f5dce809700 1 --2- 172.21.15.42:0/286045105 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5da803abd0 0x7f5da803d0e0 crc :-1 s=READY pgs=1162 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:31:22.026 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:22.020+0000 7f5dce809700 1 -- 172.21.15.42:0/286045105 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5dc808eeb0 msgr2=0x7f5dc81481d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:22.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:22.020+0000 7f5dce809700 1 --2- 172.21.15.42:0/286045105 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5dc808eeb0 0x7f5dc81481d0 secure :-1 s=READY pgs=2077 cs=0 l=1 rx=0x7f5dc81489e0 tx=0x7f5db8008b70).stop 2019-12-05T06:31:22.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:22.020+0000 7f5dce809700 1 -- 172.21.15.42:0/286045105 shutdown_connections 2019-12-05T06:31:22.032 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:22.020+0000 7f5dce809700 1 --2- 172.21.15.42:0/286045105 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5da803abd0 0x7f5da803d0e0 unknown :-1 s=CLOSED pgs=1162 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:31:22.032 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:22.020+0000 7f5dce809700 1 --2- 172.21.15.42:0/286045105 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5dc8065200 0x7f5dc807bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:31:22.032 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:22.020+0000 7f5dce809700 1 --2- 172.21.15.42:0/286045105 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5dc808eeb0 0x7f5dc81481d0 unknown :-1 s=CLOSED pgs=2077 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:31:22.033 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:22.020+0000 7f5dce809700 1 -- 172.21.15.42:0/286045105 shutdown_connections 2019-12-05T06:31:22.033 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:22.020+0000 7f5dce809700 1 -- 172.21.15.42:0/286045105 wait complete. 2019-12-05T06:31:22.033 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:22.020+0000 7f5dce809700 1 -- 172.21.15.42:0/286045105 >> 172.21.15.42:0/286045105 conn(0x7f5dc80aabe0 msgr2=0x7f5dc807c720 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:31:22.040 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b has gained rank=0, replacing gid=11265 2019-12-05T06:31:22.040 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 7 secs before reviving mds.a 2019-12-05T06:31:29.040 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.a 2019-12-05T06:31:29.040 INFO:tasks.ceph.mds.a:Restarting daemon 2019-12-05T06:31:29.041 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2019-12-05T06:31:29.052 INFO:tasks.ceph.mds.a:Started 2019-12-05T06:31:29.052 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:31:29.103 INFO:tasks.ceph.mds.a.smithi042.stdout:starting mds.a at 2019-12-05T06:31:29.159 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:29.148+0000 7fe7fc7a4700 1 Processor -- start 2019-12-05T06:31:29.159 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:29.148+0000 7fe7fc7a4700 1 -- start start 2019-12-05T06:31:29.160 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:29.148+0000 7fe7fc7a4700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe7f40a7740 0x7fe7f408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:31:29.160 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:29.148+0000 7fe7fc7a4700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe7f407e720 0x7fe7f407eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:31:29.160 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:29.148+0000 7fe7fc7a4700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe7f409f2e0 0x7fe7f409f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:31:29.160 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:29.148+0000 7fe7fc7a4700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7f40a0b00 con 0x7fe7f40a7740 2019-12-05T06:31:29.160 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:29.148+0000 7fe7fc7a4700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7f408e5b0 con 0x7fe7f409f2e0 2019-12-05T06:31:29.160 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:29.148+0000 7fe7fc7a4700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7f408e850 con 0x7fe7f407e720 2019-12-05T06:31:29.160 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:29.152+0000 7fe7fb542700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe7f409f2e0 0x7fe7f409f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:31:29.161 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:29.152+0000 7fe7fb542700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe7f409f2e0 0x7fe7f409f710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:56976/0 (socket says 172.21.15.42:56976) 2019-12-05T06:31:29.161 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:29.152+0000 7fe7fb542700 1 -- 172.21.15.42:0/1686163707 learned_addr learned my addr 172.21.15.42:0/1686163707 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:31:29.161 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:29.152+0000 7fe7fa540700 1 --2- 172.21.15.42:0/1686163707 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe7f407e720 0x7fe7f407eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:31:29.161 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:29.152+0000 7fe7fad41700 1 --2- 172.21.15.42:0/1686163707 >> v2:172.21.15.40:3300/0 conn(0x7fe7f40a7740 0x7fe7f408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:31:29.161 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:29.152+0000 7fe7fb542700 1 -- 172.21.15.42:0/1686163707 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe7f407e720 msgr2=0x7fe7f407eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:31:29.161 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:29.152+0000 7fe7fb542700 1 --2- 172.21.15.42:0/1686163707 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe7f407e720 0x7fe7f407eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:31:29.162 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:29.152+0000 7fe7fb542700 1 -- 172.21.15.42:0/1686163707 >> v2:172.21.15.40:3300/0 conn(0x7fe7f40a7740 msgr2=0x7fe7f408df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:31:29.162 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:29.152+0000 7fe7fb542700 1 --2- 172.21.15.42:0/1686163707 >> v2:172.21.15.40:3300/0 conn(0x7fe7f40a7740 0x7fe7f408df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:31:29.162 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:29.152+0000 7fe7fb542700 1 -- 172.21.15.42:0/1686163707 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7f407f0a0 con 0x7fe7f409f2e0 2019-12-05T06:31:29.162 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:29.152+0000 7fe7fb542700 1 --2- 172.21.15.42:0/1686163707 >> v2:172.21.15.42:3300/0 conn(0x7fe7f409f2e0 0x7fe7f409f710 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fe7f408e770 tx=0x7fe7ec010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:29.162 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:29.152+0000 7fe7fb542700 1 --2- 172.21.15.42:0/1686163707 >> v2:172.21.15.42:3300/0 conn(0x7fe7f409f2e0 0x7fe7f409f710 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fe7f408e770 tx=0x7fe7ec010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:29.162 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:29.152+0000 7fe7fb542700 1 --2- 172.21.15.42:0/1686163707 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe7f409f2e0 0x7fe7f409f710 secure :-1 s=READY pgs=2775 cs=0 l=1 rx=0x7fe7f408e770 tx=0x7fe7ec010130).ready entity=mon.1 client_cookie=277cb351b9cf35dc server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:31:29.162 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:29.152+0000 7fe7fb542700 1 --2- 172.21.15.42:0/1686163707 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe7f409f2e0 0x7fe7f409f710 secure :-1 s=THROTTLE_DONE pgs=2775 cs=0 l=1 rx=0x7fe7f408e770 tx=0x7fe7ec010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:29.163 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:29.152+0000 7fe7fb542700 1 --2- 172.21.15.42:0/1686163707 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe7f409f2e0 0x7fe7f409f710 secure :-1 s=THROTTLE_DONE pgs=2775 cs=0 l=1 rx=0x7fe7f408e770 tx=0x7fe7ec010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:29.163 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:29.152+0000 7fe7fb542700 1 --2- 172.21.15.42:0/1686163707 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe7f409f2e0 0x7fe7f409f710 secure :-1 s=THROTTLE_DONE pgs=2775 cs=0 l=1 rx=0x7fe7f408e770 tx=0x7fe7ec010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:29.163 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:29.152+0000 7fe7f9d3f700 1 -- 172.21.15.42:0/1686163707 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe7ec010de0 con 0x7fe7f409f2e0 2019-12-05T06:31:29.163 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:29.152+0000 7fe7f9d3f700 1 -- 172.21.15.42:0/1686163707 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7ec010a00 con 0x7fe7f409f2e0 2019-12-05T06:31:29.163 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:29.152+0000 7fe7f9d3f700 1 -- 172.21.15.42:0/1686163707 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe7ec0114a0 con 0x7fe7f409f2e0 2019-12-05T06:31:29.163 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:29.152+0000 7fe7fc7a4700 1 -- 172.21.15.42:0/1686163707 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe7f409f2e0 msgr2=0x7fe7f409f710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:29.163 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:29.152+0000 7fe7fc7a4700 1 --2- 172.21.15.42:0/1686163707 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe7f409f2e0 0x7fe7f409f710 secure :-1 s=READY pgs=2775 cs=0 l=1 rx=0x7fe7f408e770 tx=0x7fe7ec010130).stop 2019-12-05T06:31:29.163 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:29.152+0000 7fe7fc7a4700 1 -- 172.21.15.42:0/1686163707 shutdown_connections 2019-12-05T06:31:29.164 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:29.152+0000 7fe7fc7a4700 1 --2- 172.21.15.42:0/1686163707 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe7f409f2e0 0x7fe7f409f710 unknown :-1 s=CLOSED pgs=2775 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:31:29.164 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:29.152+0000 7fe7fc7a4700 1 --2- 172.21.15.42:0/1686163707 >> v2:172.21.15.40:3300/0 conn(0x7fe7f40a7740 0x7fe7f408df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:31:29.164 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:29.152+0000 7fe7fc7a4700 1 --2- 172.21.15.42:0/1686163707 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe7f407e720 0x7fe7f407eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:31:29.164 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:29.152+0000 7fe7fc7a4700 1 -- 172.21.15.42:0/1686163707 shutdown_connections 2019-12-05T06:31:29.164 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:29.152+0000 7fe7fc7a4700 1 -- 172.21.15.42:0/1686163707 wait complete. 2019-12-05T06:31:29.164 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:29.152+0000 7fe7fc7a4700 1 -- 172.21.15.42:0/1686163707 >> 172.21.15.42:0/1686163707 conn(0x7fe7f40aabe0 msgr2=0x7fe7f40a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:31:29.168 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:29.156+0000 7fe7fc7a4700 1 Processor -- start 2019-12-05T06:31:29.168 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:29.156+0000 7fe7fc7a4700 1 -- start start 2019-12-05T06:31:29.168 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:29.156+0000 7fe7fc7a4700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe7f41236e0 0x7fe7f408eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:31:29.168 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:29.156+0000 7fe7fc7a4700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe7f408f4f0 0x7fe7f4128c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:31:29.169 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:29.156+0000 7fe7fc7a4700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7f40a65f0 con 0x7fe7f408f0c0 2019-12-05T06:31:29.169 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:29.156+0000 7fe7fc7a4700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7f407d110 con 0x7fe7f408f4f0 2019-12-05T06:31:29.169 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:29.156+0000 7fe7fc7a4700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7f407c0d0 con 0x7fe7f41236e0 2019-12-05T06:31:29.169 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:29.156+0000 7fe7fa540700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fe7f408f0c0 0x7fe7f4128400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:57018/0 (socket says 172.21.15.42:57018) 2019-12-05T06:31:29.169 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:29.156+0000 7fe7fa540700 1 -- 172.21.15.42:0/1267221432 learned_addr learned my addr 172.21.15.42:0/1267221432 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:31:29.169 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:29.156+0000 7fe7fb542700 1 --2- 172.21.15.42:0/1267221432 >> v2:172.21.15.40:3300/0 conn(0x7fe7f408f4f0 0x7fe7f4128c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:31:29.169 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:29.156+0000 7fe7fad41700 1 --2- 172.21.15.42:0/1267221432 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe7f41236e0 0x7fe7f408eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:31:29.170 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:29.156+0000 7fe7eb7fe700 1 -- 172.21.15.42:0/1267221432 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3277827144 0 0) 0x7fe7e4001ec0 con 0x7fe7f408f0c0 2019-12-05T06:31:29.170 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:29.156+0000 7fe7eb7fe700 1 -- 172.21.15.42:0/1267221432 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7dc0030b0 con 0x7fe7f408f0c0 2019-12-05T06:31:29.170 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:29.156+0000 7fe7fb542700 1 -- 172.21.15.42:0/1267221432 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe7f41236e0 msgr2=0x7fe7f408eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:29.170 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:29.156+0000 7fe7fb542700 1 --2- 172.21.15.42:0/1267221432 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe7f41236e0 0x7fe7f408eb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:31:29.170 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:29.156+0000 7fe7fb542700 1 -- 172.21.15.42:0/1267221432 >> v1:172.21.15.42:6789/0 conn(0x7fe7f408f0c0 legacy=0x7fe7f4128400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:29.170 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:29.156+0000 7fe7fb542700 1 -- 172.21.15.42:0/1267221432 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7f40a71b0 con 0x7fe7f408f4f0 2019-12-05T06:31:29.170 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:29.156+0000 7fe7fb542700 1 --2- 172.21.15.42:0/1267221432 >> v2:172.21.15.40:3300/0 conn(0x7fe7f408f4f0 0x7fe7f4128c00 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fe7f407c290 tx=0x7fe7ec010cb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:29.171 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:29.156+0000 7fe7fb542700 1 --2- 172.21.15.42:0/1267221432 >> v2:172.21.15.40:3300/0 conn(0x7fe7f408f4f0 0x7fe7f4128c00 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fe7f407c290 tx=0x7fe7ec010cb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:29.171 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:29.156+0000 7fe7fb542700 1 --2- 172.21.15.42:0/1267221432 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe7f408f4f0 0x7fe7f4128c00 secure :-1 s=READY pgs=2082 cs=0 l=1 rx=0x7fe7f407c290 tx=0x7fe7ec010cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:31:29.171 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:29.156+0000 7fe7fb542700 1 --2- 172.21.15.42:0/1267221432 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe7f408f4f0 0x7fe7f4128c00 secure :-1 s=THROTTLE_DONE pgs=2082 cs=0 l=1 rx=0x7fe7f407c290 tx=0x7fe7ec010cb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:29.171 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:29.160+0000 7fe7eb7fe700 1 -- 172.21.15.42:0/1267221432 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe7ec011b70 con 0x7fe7f408f4f0 2019-12-05T06:31:29.171 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:29.160+0000 7fe7fb542700 1 --2- 172.21.15.42:0/1267221432 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe7f408f4f0 0x7fe7f4128c00 secure :-1 s=THROTTLE_DONE pgs=2082 cs=0 l=1 rx=0x7fe7f407c290 tx=0x7fe7ec010cb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:29.171 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:29.160+0000 7fe7fb542700 1 --2- 172.21.15.42:0/1267221432 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe7f408f4f0 0x7fe7f4128c00 secure :-1 s=THROTTLE_DONE pgs=2082 cs=0 l=1 rx=0x7fe7f407c290 tx=0x7fe7ec010cb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:29.171 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:29.160+0000 7fe7eb7fe700 1 -- 172.21.15.42:0/1267221432 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7ec0110c0 con 0x7fe7f408f4f0 2019-12-05T06:31:29.172 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:29.160+0000 7fe7eb7fe700 1 -- 172.21.15.42:0/1267221432 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe7ec005ab0 con 0x7fe7f408f4f0 2019-12-05T06:31:29.172 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:29.160+0000 7fe7fc7a4700 1 -- 172.21.15.42:0/1267221432 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7f407cbf0 con 0x7fe7f408f4f0 2019-12-05T06:31:29.172 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:29.160+0000 7fe7fc7a4700 1 -- 172.21.15.42:0/1267221432 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7f408f920 con 0x7fe7f408f4f0 2019-12-05T06:31:29.172 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:29.160+0000 7fe7fb542700 1 --2- 172.21.15.42:0/1267221432 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe7f408f4f0 0x7fe7f4128c00 secure :-1 s=THROTTLE_DONE pgs=2082 cs=0 l=1 rx=0x7fe7f407c290 tx=0x7fe7ec010cb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:29.172 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:29.160+0000 7fe7eb7fe700 1 -- 172.21.15.42:0/1267221432 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fe7ec00f280 con 0x7fe7f408f4f0 2019-12-05T06:31:29.172 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:29.160+0000 7fe7fb542700 1 --2- 172.21.15.42:0/1267221432 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe7f408f4f0 0x7fe7f4128c00 secure :-1 s=THROTTLE_DONE pgs=2082 cs=0 l=1 rx=0x7fe7f407c290 tx=0x7fe7ec010cb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:29.172 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:29.160+0000 7fe7eb7fe700 1 --2- 172.21.15.42:0/1267221432 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe7dc03b2c0 0x7fe7dc03d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:31:29.173 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:29.160+0000 7fe7eb7fe700 1 -- 172.21.15.42:0/1267221432 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 11191+0+0 (secure 0 0 0) 0x7fe7ec08f580 con 0x7fe7f408f4f0 2019-12-05T06:31:29.173 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:29.160+0000 7fe7fad41700 1 --2- 172.21.15.42:0/1267221432 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe7dc03b2c0 0x7fe7dc03d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:31:29.173 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:29.160+0000 7fe7fad41700 1 --2- 172.21.15.42:0/1267221432 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe7dc03b2c0 0x7fe7dc03d7d0 crc :-1 s=READY pgs=1163 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:31:29.173 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:29.164+0000 7fe7fc7a4700 1 -- 172.21.15.42:0/1267221432 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7f405d110 con 0x7fe7f408f4f0 2019-12-05T06:31:29.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:29.176+0000 7fe7fb542700 1 --2- 172.21.15.42:0/1267221432 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe7f408f4f0 0x7fe7f4128c00 secure :-1 s=THROTTLE_DONE pgs=2082 cs=0 l=1 rx=0x7fe7f407c290 tx=0x7fe7ec010cb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:29.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:29.176+0000 7fe7eb7fe700 1 -- 172.21.15.42:0/1267221432 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fe7ec0760c0 con 0x7fe7f408f4f0 2019-12-05T06:31:29.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:29.344+0000 7fe7fc7a4700 1 -- 172.21.15.42:0/1267221432 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe7f405b8c0 con 0x7fe7f408f4f0 2019-12-05T06:31:29.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:29.344+0000 7fe7fb542700 1 --2- 172.21.15.42:0/1267221432 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe7f408f4f0 0x7fe7f4128c00 secure :-1 s=THROTTLE_DONE pgs=2082 cs=0 l=1 rx=0x7fe7f407c290 tx=0x7fe7ec010cb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:29.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:29.344+0000 7fe7eb7fe700 1 -- 172.21.15.42:0/1267221432 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 332 v332) v1 ==== 95+0+1779 (secure 0 0 0) 0x7fe7ec076600 con 0x7fe7f408f4f0 2019-12-05T06:31:29.352 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 332 2019-12-05T06:31:29.352 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:31:29.352 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":332,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":332,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:31:20.143327+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":109,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11352},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11352":{"gid":11352,"name":"b","rank":0,"incarnation":329,"state":"up:active","state_seq":16,"addr":"172.21.15.40:6833/2055706790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2055706790},{"type":"v1","addr":"172.21.15.40:6833","nonce":2055706790}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:31:29.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:29.348+0000 7fe7fc7a4700 1 -- 172.21.15.42:0/1267221432 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe7dc03b2c0 msgr2=0x7fe7dc03d7d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:29.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:29.348+0000 7fe7fc7a4700 1 --2- 172.21.15.42:0/1267221432 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe7dc03b2c0 0x7fe7dc03d7d0 crc :-1 s=READY pgs=1163 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:31:29.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:29.348+0000 7fe7fc7a4700 1 -- 172.21.15.42:0/1267221432 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe7f408f4f0 msgr2=0x7fe7f4128c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:29.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:29.348+0000 7fe7fc7a4700 1 --2- 172.21.15.42:0/1267221432 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe7f408f4f0 0x7fe7f4128c00 secure :-1 s=READY pgs=2082 cs=0 l=1 rx=0x7fe7f407c290 tx=0x7fe7ec010cb0).stop 2019-12-05T06:31:29.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:29.348+0000 7fe7fc7a4700 1 -- 172.21.15.42:0/1267221432 shutdown_connections 2019-12-05T06:31:29.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:29.348+0000 7fe7fc7a4700 1 --2- 172.21.15.42:0/1267221432 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe7dc03b2c0 0x7fe7dc03d7d0 unknown :-1 s=CLOSED pgs=1163 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:31:29.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:29.348+0000 7fe7fc7a4700 1 --2- 172.21.15.42:0/1267221432 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe7f408f4f0 0x7fe7f4128c00 unknown :-1 s=CLOSED pgs=2082 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:31:29.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:29.348+0000 7fe7fc7a4700 1 --2- 172.21.15.42:0/1267221432 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe7f41236e0 0x7fe7f408eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:31:29.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:29.348+0000 7fe7fc7a4700 1 -- 172.21.15.42:0/1267221432 shutdown_connections 2019-12-05T06:31:29.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:29.348+0000 7fe7fc7a4700 1 -- 172.21.15.42:0/1267221432 wait complete. 2019-12-05T06:31:29.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:29.348+0000 7fe7fc7a4700 1 -- 172.21.15.42:0/1267221432 >> 172.21.15.42:0/1267221432 conn(0x7fe7f409fa10 msgr2=0x7fe7f4059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:31:29.365 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2019-12-05T06:31:31.365 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:31:31.477 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:31.464+0000 7f9bbdc00700 1 Processor -- start 2019-12-05T06:31:31.477 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:31.464+0000 7f9bbdc00700 1 -- start start 2019-12-05T06:31:31.477 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:31.468+0000 7f9bbdc00700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f9bb80a7740 0x7f9bb808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:31:31.477 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:31.468+0000 7f9bbdc00700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9bb807e720 0x7f9bb807eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:31:31.478 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:31.468+0000 7f9bbdc00700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f9bb809f2e0 0x7f9bb809f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:31:31.478 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:31.468+0000 7f9bbdc00700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9bb80a0b00 con 0x7f9bb80a7740 2019-12-05T06:31:31.478 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:31.468+0000 7f9bbdc00700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9bb808e5b0 con 0x7f9bb809f2e0 2019-12-05T06:31:31.478 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:31.468+0000 7f9bbdc00700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bb808e850 con 0x7f9bb807e720 2019-12-05T06:31:31.479 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:31.468+0000 7f9bbc99e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f9bb809f2e0 0x7f9bb809f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:31:31.479 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:31.468+0000 7f9bb7fff700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f9bb80a7740 0x7f9bb808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:31:31.479 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:31.468+0000 7f9bb7fff700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f9bb80a7740 0x7f9bb808df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.42:44806/0 (socket says 172.21.15.42:44806) 2019-12-05T06:31:31.479 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:31.468+0000 7f9bb7fff700 1 -- 172.21.15.42:0/54908950 learned_addr learned my addr 172.21.15.42:0/54908950 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:31:31.479 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:31.468+0000 7f9bbc99e700 1 --2- 172.21.15.42:0/54908950 >> v2:172.21.15.42:3300/0 conn(0x7f9bb809f2e0 0x7f9bb809f710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:56994/0 (socket says 172.21.15.42:56994) 2019-12-05T06:31:31.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:31.468+0000 7f9bb77fe700 1 --2- 172.21.15.42:0/54908950 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9bb807e720 0x7f9bb807eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:31:31.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:31.468+0000 7f9bbc99e700 1 -- 172.21.15.42:0/54908950 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9bb807e720 msgr2=0x7f9bb807eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:31:31.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:31.468+0000 7f9bbc99e700 1 --2- 172.21.15.42:0/54908950 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9bb807e720 0x7f9bb807eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:31:31.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:31.468+0000 7f9bbc99e700 1 -- 172.21.15.42:0/54908950 >> v2:172.21.15.40:3300/0 conn(0x7f9bb80a7740 msgr2=0x7f9bb808df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:31:31.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:31.468+0000 7f9bbc99e700 1 --2- 172.21.15.42:0/54908950 >> v2:172.21.15.40:3300/0 conn(0x7f9bb80a7740 0x7f9bb808df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:31:31.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:31.468+0000 7f9bbc99e700 1 -- 172.21.15.42:0/54908950 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bb807f0a0 con 0x7f9bb809f2e0 2019-12-05T06:31:31.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:31.468+0000 7f9bbc99e700 1 --2- 172.21.15.42:0/54908950 >> v2:172.21.15.42:3300/0 conn(0x7f9bb809f2e0 0x7f9bb809f710 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f9bb808e770 tx=0x7f9bb000fcf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:31.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:31.468+0000 7f9bbc99e700 1 --2- 172.21.15.42:0/54908950 >> v2:172.21.15.42:3300/0 conn(0x7f9bb809f2e0 0x7f9bb809f710 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f9bb808e770 tx=0x7f9bb000fcf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:31.482 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:31.468+0000 7f9bbc99e700 1 --2- 172.21.15.42:0/54908950 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9bb809f2e0 0x7f9bb809f710 secure :-1 s=READY pgs=2777 cs=0 l=1 rx=0x7f9bb808e770 tx=0x7f9bb000fcf0).ready entity=mon.1 client_cookie=370e35b12f961595 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:31:31.482 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:31.468+0000 7f9bbc99e700 1 --2- 172.21.15.42:0/54908950 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9bb809f2e0 0x7f9bb809f710 secure :-1 s=THROTTLE_DONE pgs=2777 cs=0 l=1 rx=0x7f9bb808e770 tx=0x7f9bb000fcf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:31.482 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:31.468+0000 7f9bbc99e700 1 --2- 172.21.15.42:0/54908950 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9bb809f2e0 0x7f9bb809f710 secure :-1 s=THROTTLE_DONE pgs=2777 cs=0 l=1 rx=0x7f9bb808e770 tx=0x7f9bb000fcf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:31.482 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:31.468+0000 7f9bb6ffd700 1 -- 172.21.15.42:0/54908950 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9bb0008bb0 con 0x7f9bb809f2e0 2019-12-05T06:31:31.482 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:31.468+0000 7f9bbc99e700 1 --2- 172.21.15.42:0/54908950 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9bb809f2e0 0x7f9bb809f710 secure :-1 s=THROTTLE_DONE pgs=2777 cs=0 l=1 rx=0x7f9bb808e770 tx=0x7f9bb000fcf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:31.483 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:31.468+0000 7f9bb6ffd700 1 -- 172.21.15.42:0/54908950 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bb00087d0 con 0x7f9bb809f2e0 2019-12-05T06:31:31.483 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:31.468+0000 7f9bb6ffd700 1 -- 172.21.15.42:0/54908950 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9bb00118a0 con 0x7f9bb809f2e0 2019-12-05T06:31:31.483 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:31.468+0000 7f9bbdc00700 1 -- 172.21.15.42:0/54908950 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9bb809f2e0 msgr2=0x7f9bb809f710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:31.483 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:31.468+0000 7f9bbdc00700 1 --2- 172.21.15.42:0/54908950 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9bb809f2e0 0x7f9bb809f710 secure :-1 s=READY pgs=2777 cs=0 l=1 rx=0x7f9bb808e770 tx=0x7f9bb000fcf0).stop 2019-12-05T06:31:31.484 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:31.468+0000 7f9bbdc00700 1 -- 172.21.15.42:0/54908950 shutdown_connections 2019-12-05T06:31:31.484 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:31.468+0000 7f9bbdc00700 1 --2- 172.21.15.42:0/54908950 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9bb809f2e0 0x7f9bb809f710 unknown :-1 s=CLOSED pgs=2777 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:31:31.484 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:31.468+0000 7f9bbdc00700 1 --2- 172.21.15.42:0/54908950 >> v2:172.21.15.40:3300/0 conn(0x7f9bb80a7740 0x7f9bb808df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:31:31.484 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:31.468+0000 7f9bbdc00700 1 --2- 172.21.15.42:0/54908950 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9bb807e720 0x7f9bb807eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:31:31.484 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:31.468+0000 7f9bbdc00700 1 -- 172.21.15.42:0/54908950 shutdown_connections 2019-12-05T06:31:31.485 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:31.468+0000 7f9bbdc00700 1 -- 172.21.15.42:0/54908950 wait complete. 2019-12-05T06:31:31.485 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:31.468+0000 7f9bbdc00700 1 -- 172.21.15.42:0/54908950 >> 172.21.15.42:0/54908950 conn(0x7f9bb80aabe0 msgr2=0x7f9bb80a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:31:31.490 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:31.472+0000 7f9bbdc00700 1 Processor -- start 2019-12-05T06:31:31.490 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:31.472+0000 7f9bbdc00700 1 -- start start 2019-12-05T06:31:31.490 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:31.472+0000 7f9bbdc00700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f9bb80a67e0 0x7f9bb8063b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:31:31.491 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:31.472+0000 7f9bbdc00700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f9bb8064f40 0x7f9bb8065370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:31:31.491 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:31.472+0000 7f9bbdc00700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bb8147020 con 0x7f9bb8146bf0 2019-12-05T06:31:31.491 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:31.472+0000 7f9bbdc00700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9bb80641c0 con 0x7f9bb8064f40 2019-12-05T06:31:31.491 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:31.472+0000 7f9bbdc00700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9bb8064460 con 0x7f9bb80a67e0 2019-12-05T06:31:31.492 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:31.472+0000 7f9bb7fff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f9bb80a67e0 0x7f9bb8063b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:31:31.492 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:31.472+0000 7f9bb7fff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f9bb80a67e0 0x7f9bb8063b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:56996/0 (socket says 172.21.15.42:56996) 2019-12-05T06:31:31.492 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:31.472+0000 7f9bb7fff700 1 -- 172.21.15.42:0/644433533 learned_addr learned my addr 172.21.15.42:0/644433533 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:31:31.492 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:31.472+0000 7f9bbc99e700 1 --2- 172.21.15.42:0/644433533 >> v2:172.21.15.40:3300/0 conn(0x7f9bb8064f40 0x7f9bb8065370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:31:31.493 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:31.476+0000 7f9bb7fff700 1 -- 172.21.15.42:0/644433533 >> v2:172.21.15.40:3300/0 conn(0x7f9bb8064f40 msgr2=0x7f9bb8065370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:31.493 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:31.476+0000 7f9bb7fff700 1 --2- 172.21.15.42:0/644433533 >> v2:172.21.15.40:3300/0 conn(0x7f9bb8064f40 0x7f9bb8065370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:31:31.493 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:31.476+0000 7f9bb7fff700 1 -- 172.21.15.42:0/644433533 >> v1:172.21.15.40:6789/0 conn(0x7f9bb8146bf0 legacy=0x7f9bb8129fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:31.493 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:31.476+0000 7f9bb7fff700 1 -- 172.21.15.42:0/644433533 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bb8064700 con 0x7f9bb80a67e0 2019-12-05T06:31:31.493 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:31.476+0000 7f9bb7fff700 1 --2- 172.21.15.42:0/644433533 >> v2:172.21.15.42:3300/0 conn(0x7f9bb80a67e0 0x7f9bb8063b90 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f9ba0003070 tx=0x7f9ba0010b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:31.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:31.476+0000 7f9bb7fff700 1 --2- 172.21.15.42:0/644433533 >> v2:172.21.15.42:3300/0 conn(0x7f9bb80a67e0 0x7f9bb8063b90 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f9ba0003070 tx=0x7f9ba0010b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:31.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:31.476+0000 7f9bb7fff700 1 --2- 172.21.15.42:0/644433533 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9bb80a67e0 0x7f9bb8063b90 secure :-1 s=READY pgs=2778 cs=0 l=1 rx=0x7f9ba0003070 tx=0x7f9ba0010b00).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:31:31.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:31.476+0000 7f9bb7fff700 1 --2- 172.21.15.42:0/644433533 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9bb80a67e0 0x7f9bb8063b90 secure :-1 s=THROTTLE_DONE pgs=2778 cs=0 l=1 rx=0x7f9ba0003070 tx=0x7f9ba0010b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:31.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:31.476+0000 7f9bb7fff700 1 --2- 172.21.15.42:0/644433533 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9bb80a67e0 0x7f9bb8063b90 secure :-1 s=THROTTLE_DONE pgs=2778 cs=0 l=1 rx=0x7f9ba0003070 tx=0x7f9ba0010b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:31.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:31.476+0000 7f9bb4ff9700 1 -- 172.21.15.42:0/644433533 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9ba0008c90 con 0x7f9bb80a67e0 2019-12-05T06:31:31.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:31.476+0000 7f9bb7fff700 1 --2- 172.21.15.42:0/644433533 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9bb80a67e0 0x7f9bb8063b90 secure :-1 s=THROTTLE_DONE pgs=2778 cs=0 l=1 rx=0x7f9ba0003070 tx=0x7f9ba0010b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:31.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:31.476+0000 7f9bb4ff9700 1 -- 172.21.15.42:0/644433533 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ba00088b0 con 0x7f9bb80a67e0 2019-12-05T06:31:31.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:31.476+0000 7f9bbdc00700 1 -- 172.21.15.42:0/644433533 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bb812a7d0 con 0x7f9bb80a67e0 2019-12-05T06:31:31.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:31.476+0000 7f9bb4ff9700 1 -- 172.21.15.42:0/644433533 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9ba0005920 con 0x7f9bb80a67e0 2019-12-05T06:31:31.496 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:31.476+0000 7f9bbdc00700 1 -- 172.21.15.42:0/644433533 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bb8137950 con 0x7f9bb80a67e0 2019-12-05T06:31:31.496 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:31.476+0000 7f9bb7fff700 1 --2- 172.21.15.42:0/644433533 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9bb80a67e0 0x7f9bb8063b90 secure :-1 s=THROTTLE_DONE pgs=2778 cs=0 l=1 rx=0x7f9ba0003070 tx=0x7f9ba0010b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:31.496 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:31.476+0000 7f9bb4ff9700 1 -- 172.21.15.42:0/644433533 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f9ba0005c30 con 0x7f9bb80a67e0 2019-12-05T06:31:31.496 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:31.476+0000 7f9bb7fff700 1 --2- 172.21.15.42:0/644433533 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9bb80a67e0 0x7f9bb8063b90 secure :-1 s=THROTTLE_DONE pgs=2778 cs=0 l=1 rx=0x7f9ba0003070 tx=0x7f9ba0010b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:31.497 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:31.476+0000 7f9bb4ff9700 1 --2- 172.21.15.42:0/644433533 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9b9c03acf0 0x7f9b9c03d200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:31:31.497 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:31.476+0000 7f9bb4ff9700 1 -- 172.21.15.42:0/644433533 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 11191+0+0 (secure 0 0 0) 0x7f9ba004af80 con 0x7f9bb80a67e0 2019-12-05T06:31:31.497 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:31.480+0000 7f9bb77fe700 1 --2- 172.21.15.42:0/644433533 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9b9c03acf0 0x7f9b9c03d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:31:31.497 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:31.480+0000 7f9bb77fe700 1 --2- 172.21.15.42:0/644433533 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9b9c03acf0 0x7f9b9c03d200 crc :-1 s=READY pgs=1164 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:31:31.498 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:31.480+0000 7f9bbdc00700 1 -- 172.21.15.42:0/644433533 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bb805d110 con 0x7f9bb80a67e0 2019-12-05T06:31:31.498 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:31.488+0000 7f9bb7fff700 1 --2- 172.21.15.42:0/644433533 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9bb80a67e0 0x7f9bb8063b90 secure :-1 s=THROTTLE_DONE pgs=2778 cs=0 l=1 rx=0x7f9ba0003070 tx=0x7f9ba0010b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:31.499 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:31.488+0000 7f9bb4ff9700 1 -- 172.21.15.42:0/644433533 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f9ba0032000 con 0x7f9bb80a67e0 2019-12-05T06:31:31.661 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:31.656+0000 7f9bbdc00700 1 -- 172.21.15.42:0/644433533 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9bb805b8c0 con 0x7f9bb80a67e0 2019-12-05T06:31:31.661 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:31.656+0000 7f9bb7fff700 1 --2- 172.21.15.42:0/644433533 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9bb80a67e0 0x7f9bb8063b90 secure :-1 s=THROTTLE_DONE pgs=2778 cs=0 l=1 rx=0x7f9ba0003070 tx=0x7f9ba0010b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:31.661 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:31.656+0000 7f9bb4ff9700 1 -- 172.21.15.42:0/644433533 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 332 v332) v1 ==== 95+0+1779 (secure 0 0 0) 0x7f9ba002f0d0 con 0x7f9bb80a67e0 2019-12-05T06:31:31.662 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 332 2019-12-05T06:31:31.662 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:31:31.662 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":332,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":332,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:31:20.143327+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":109,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11352},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11352":{"gid":11352,"name":"b","rank":0,"incarnation":329,"state":"up:active","state_seq":16,"addr":"172.21.15.40:6833/2055706790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2055706790},{"type":"v1","addr":"172.21.15.40:6833","nonce":2055706790}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:31:31.664 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:31.656+0000 7f9bbdc00700 1 -- 172.21.15.42:0/644433533 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9b9c03acf0 msgr2=0x7f9b9c03d200 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:31.664 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:31.656+0000 7f9bbdc00700 1 --2- 172.21.15.42:0/644433533 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9b9c03acf0 0x7f9b9c03d200 crc :-1 s=READY pgs=1164 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:31:31.665 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:31.656+0000 7f9bbdc00700 1 -- 172.21.15.42:0/644433533 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9bb80a67e0 msgr2=0x7f9bb8063b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:31.665 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:31.656+0000 7f9bbdc00700 1 --2- 172.21.15.42:0/644433533 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9bb80a67e0 0x7f9bb8063b90 secure :-1 s=READY pgs=2778 cs=0 l=1 rx=0x7f9ba0003070 tx=0x7f9ba0010b00).stop 2019-12-05T06:31:31.665 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:31.660+0000 7f9bbdc00700 1 -- 172.21.15.42:0/644433533 shutdown_connections 2019-12-05T06:31:31.665 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:31.660+0000 7f9bbdc00700 1 --2- 172.21.15.42:0/644433533 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9b9c03acf0 0x7f9b9c03d200 unknown :-1 s=CLOSED pgs=1164 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:31:31.665 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:31.660+0000 7f9bbdc00700 1 --2- 172.21.15.42:0/644433533 >> v2:172.21.15.40:3300/0 conn(0x7f9bb8064f40 0x7f9bb8065370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:31:31.665 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:31.660+0000 7f9bbdc00700 1 --2- 172.21.15.42:0/644433533 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9bb80a67e0 0x7f9bb8063b90 unknown :-1 s=CLOSED pgs=2778 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:31:31.665 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:31.660+0000 7f9bbdc00700 1 -- 172.21.15.42:0/644433533 shutdown_connections 2019-12-05T06:31:31.666 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:31.660+0000 7f9bbdc00700 1 -- 172.21.15.42:0/644433533 wait complete. 2019-12-05T06:31:31.666 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:31.660+0000 7f9bbdc00700 1 -- 172.21.15.42:0/644433533 >> 172.21.15.42:0/644433533 conn(0x7f9bb80a7740 msgr2=0x7f9bb807e4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:31:31.674 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2019-12-05T06:31:33.675 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:31:33.784 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:33.772+0000 7ff7efc16700 1 Processor -- start 2019-12-05T06:31:33.785 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:33.772+0000 7ff7efc16700 1 -- start start 2019-12-05T06:31:33.785 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:33.776+0000 7ff7efc16700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff7e809fae0 0x7ff7e814fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:31:33.785 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:33.776+0000 7ff7efc16700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7e80a0710 con 0x7ff7e80a7740 2019-12-05T06:31:33.785 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:33.776+0000 7ff7efc16700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7e807ef20 con 0x7ff7e808df80 2019-12-05T06:31:33.785 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:33.776+0000 7ff7efc16700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7e80a0b00 con 0x7ff7e809fae0 2019-12-05T06:31:33.786 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:33.776+0000 7ff7ed9b2700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7ff7e808df80 0x7ff7e809f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:57050/0 (socket says 172.21.15.42:57050) 2019-12-05T06:31:33.786 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:33.776+0000 7ff7ed9b2700 1 -- 172.21.15.42:0/4157437868 learned_addr learned my addr 172.21.15.42:0/4157437868 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:31:33.786 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:33.776+0000 7ff7ee9b4700 1 --2- 172.21.15.42:0/4157437868 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff7e809fae0 0x7ff7e814fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:31:33.786 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:33.776+0000 7ff7ed1b1700 1 -- 172.21.15.42:0/4157437868 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1980371603 0 0) 0x7ff7e807ef20 con 0x7ff7e808df80 2019-12-05T06:31:33.786 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:33.776+0000 7ff7ed1b1700 1 -- 172.21.15.42:0/4157437868 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7d40030b0 con 0x7ff7e808df80 2019-12-05T06:31:33.786 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:33.776+0000 7ff7ed1b1700 1 -- 172.21.15.42:0/4157437868 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1159292888 0 0) 0x7ff7e80a0710 con 0x7ff7e80a7740 2019-12-05T06:31:33.786 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:33.776+0000 7ff7ed1b1700 1 -- 172.21.15.42:0/4157437868 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7e807ef20 con 0x7ff7e80a7740 2019-12-05T06:31:33.787 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:33.776+0000 7ff7ee9b4700 1 -- 172.21.15.42:0/4157437868 >> v1:172.21.15.42:6789/0 conn(0x7ff7e808df80 legacy=0x7ff7e809f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:33.787 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:33.776+0000 7ff7ee9b4700 1 -- 172.21.15.42:0/4157437868 >> v1:172.21.15.40:6789/0 conn(0x7ff7e80a7740 legacy=0x7ff7e807e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:33.787 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:33.776+0000 7ff7ee9b4700 1 -- 172.21.15.42:0/4157437868 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7e80a9ba0 con 0x7ff7e809fae0 2019-12-05T06:31:33.787 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:33.776+0000 7ff7ee9b4700 1 --2- 172.21.15.42:0/4157437868 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff7e809fae0 0x7ff7e814fff0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7ff7e00032b0 tx=0x7ff7e000ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:33.787 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:33.776+0000 7ff7ee9b4700 1 --2- 172.21.15.42:0/4157437868 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff7e809fae0 0x7ff7e814fff0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7ff7e00032b0 tx=0x7ff7e000ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:33.787 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:33.776+0000 7ff7ee9b4700 1 --2- 172.21.15.42:0/4157437868 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff7e809fae0 0x7ff7e814fff0 secure :-1 s=READY pgs=611 cs=0 l=1 rx=0x7ff7e00032b0 tx=0x7ff7e000ff70).ready entity=mon.2 client_cookie=6c6269a630843367 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:31:33.787 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:33.776+0000 7ff7ee9b4700 1 --2- 172.21.15.42:0/4157437868 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff7e809fae0 0x7ff7e814fff0 secure :-1 s=THROTTLE_DONE pgs=611 cs=0 l=1 rx=0x7ff7e00032b0 tx=0x7ff7e000ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:33.788 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:33.776+0000 7ff7ed1b1700 1 -- 172.21.15.42:0/4157437868 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff7e00109b0 con 0x7ff7e809fae0 2019-12-05T06:31:33.788 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:33.776+0000 7ff7ee9b4700 1 --2- 172.21.15.42:0/4157437868 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff7e809fae0 0x7ff7e814fff0 secure :-1 s=THROTTLE_DONE pgs=611 cs=0 l=1 rx=0x7ff7e00032b0 tx=0x7ff7e000ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:33.788 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:33.776+0000 7ff7ee9b4700 1 --2- 172.21.15.42:0/4157437868 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff7e809fae0 0x7ff7e814fff0 secure :-1 s=THROTTLE_DONE pgs=611 cs=0 l=1 rx=0x7ff7e00032b0 tx=0x7ff7e000ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:33.788 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:33.776+0000 7ff7ed1b1700 1 -- 172.21.15.42:0/4157437868 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7e0008b50 con 0x7ff7e809fae0 2019-12-05T06:31:33.788 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:33.776+0000 7ff7ed1b1700 1 -- 172.21.15.42:0/4157437868 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff7e00110c0 con 0x7ff7e809fae0 2019-12-05T06:31:33.788 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:33.776+0000 7ff7efc16700 1 -- 172.21.15.42:0/4157437868 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff7e809fae0 msgr2=0x7ff7e814fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:33.789 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:33.776+0000 7ff7efc16700 1 --2- 172.21.15.42:0/4157437868 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff7e809fae0 0x7ff7e814fff0 secure :-1 s=READY pgs=611 cs=0 l=1 rx=0x7ff7e00032b0 tx=0x7ff7e000ff70).stop 2019-12-05T06:31:33.789 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:33.776+0000 7ff7efc16700 1 -- 172.21.15.42:0/4157437868 shutdown_connections 2019-12-05T06:31:33.789 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:33.776+0000 7ff7efc16700 1 --2- 172.21.15.42:0/4157437868 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff7e809fae0 0x7ff7e814fff0 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:31:33.789 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:33.776+0000 7ff7efc16700 1 -- 172.21.15.42:0/4157437868 shutdown_connections 2019-12-05T06:31:33.789 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:33.776+0000 7ff7efc16700 1 -- 172.21.15.42:0/4157437868 wait complete. 2019-12-05T06:31:33.789 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:33.776+0000 7ff7efc16700 1 -- 172.21.15.42:0/4157437868 >> 172.21.15.42:0/4157437868 conn(0x7ff7e80aabe0 msgr2=0x7ff7e80a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:31:33.793 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:33.780+0000 7ff7efc16700 1 Processor -- start 2019-12-05T06:31:33.793 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:33.780+0000 7ff7efc16700 1 -- start start 2019-12-05T06:31:33.793 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:33.780+0000 7ff7efc16700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff7e80a3a60 0x7ff7e8112fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:31:33.793 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:33.780+0000 7ff7efc16700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff7e807d850 0x7ff7e807dc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:31:33.794 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:33.780+0000 7ff7efc16700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7e807e1d0 con 0x7ff7e8114230 2019-12-05T06:31:33.794 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:33.780+0000 7ff7efc16700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7e811a100 con 0x7ff7e80a3a60 2019-12-05T06:31:33.794 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:33.780+0000 7ff7efc16700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7e811a3a0 con 0x7ff7e807d850 2019-12-05T06:31:33.794 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:33.780+0000 7ff7ee9b4700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7ff7e8114230 0x7ff7e8114660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:57058/0 (socket says 172.21.15.42:57058) 2019-12-05T06:31:33.794 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:33.780+0000 7ff7ee9b4700 1 -- 172.21.15.42:0/954155938 learned_addr learned my addr 172.21.15.42:0/954155938 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:31:33.794 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:33.780+0000 7ff7ed9b2700 1 --2- 172.21.15.42:0/954155938 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff7e807d850 0x7ff7e807dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:31:33.794 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:33.780+0000 7ff7ee1b3700 1 --2- 172.21.15.42:0/954155938 >> v2:172.21.15.40:3300/0 conn(0x7ff7e80a3a60 0x7ff7e8112fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:31:33.795 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:33.780+0000 7ff7deffd700 1 -- 172.21.15.42:0/954155938 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2414869269 0 0) 0x7ff7e807e1d0 con 0x7ff7e8114230 2019-12-05T06:31:33.795 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:33.780+0000 7ff7deffd700 1 -- 172.21.15.42:0/954155938 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7d00030b0 con 0x7ff7e8114230 2019-12-05T06:31:33.795 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:33.784+0000 7ff7ed9b2700 1 -- 172.21.15.42:0/954155938 >> v2:172.21.15.40:3300/0 conn(0x7ff7e80a3a60 msgr2=0x7ff7e8112fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:33.795 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:33.784+0000 7ff7ed9b2700 1 --2- 172.21.15.42:0/954155938 >> v2:172.21.15.40:3300/0 conn(0x7ff7e80a3a60 0x7ff7e8112fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:31:33.795 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:33.784+0000 7ff7ed9b2700 1 -- 172.21.15.42:0/954155938 >> v1:172.21.15.42:6789/0 conn(0x7ff7e8114230 legacy=0x7ff7e8114660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:33.795 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:33.784+0000 7ff7ed9b2700 1 -- 172.21.15.42:0/954155938 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7e8113520 con 0x7ff7e807d850 2019-12-05T06:31:33.796 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:33.784+0000 7ff7ee1b3700 1 --2- 172.21.15.42:0/954155938 >> v2:172.21.15.40:3300/0 conn(0x7ff7e80a3a60 0x7ff7e8112fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:31:33.796 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:33.784+0000 7ff7ed9b2700 1 --2- 172.21.15.42:0/954155938 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff7e807d850 0x7ff7e807dc80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7ff7e4000b80 tx=0x7ff7e4010890).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:33.796 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:33.784+0000 7ff7ed9b2700 1 --2- 172.21.15.42:0/954155938 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff7e807d850 0x7ff7e807dc80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7ff7e4000b80 tx=0x7ff7e4010890).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:33.796 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:33.784+0000 7ff7ed9b2700 1 --2- 172.21.15.42:0/954155938 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff7e807d850 0x7ff7e807dc80 secure :-1 s=READY pgs=612 cs=0 l=1 rx=0x7ff7e4000b80 tx=0x7ff7e4010890).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:31:33.796 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:33.784+0000 7ff7ed9b2700 1 --2- 172.21.15.42:0/954155938 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff7e807d850 0x7ff7e807dc80 secure :-1 s=THROTTLE_DONE pgs=612 cs=0 l=1 rx=0x7ff7e4000b80 tx=0x7ff7e4010890).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:33.796 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:33.784+0000 7ff7deffd700 1 -- 172.21.15.42:0/954155938 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff7e4005b40 con 0x7ff7e807d850 2019-12-05T06:31:33.796 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:33.784+0000 7ff7efc16700 1 -- 172.21.15.42:0/954155938 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7e808ef40 con 0x7ff7e807d850 2019-12-05T06:31:33.797 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:33.784+0000 7ff7ed9b2700 1 --2- 172.21.15.42:0/954155938 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff7e807d850 0x7ff7e807dc80 secure :-1 s=THROTTLE_DONE pgs=612 cs=0 l=1 rx=0x7ff7e4000b80 tx=0x7ff7e4010890).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:33.797 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:33.784+0000 7ff7deffd700 1 -- 172.21.15.42:0/954155938 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7e4005760 con 0x7ff7e807d850 2019-12-05T06:31:33.797 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:33.784+0000 7ff7efc16700 1 -- 172.21.15.42:0/954155938 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7e812dee0 con 0x7ff7e807d850 2019-12-05T06:31:33.797 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:33.784+0000 7ff7ed9b2700 1 --2- 172.21.15.42:0/954155938 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff7e807d850 0x7ff7e807dc80 secure :-1 s=THROTTLE_DONE pgs=612 cs=0 l=1 rx=0x7ff7e4000b80 tx=0x7ff7e4010890).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:33.797 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:33.784+0000 7ff7deffd700 1 -- 172.21.15.42:0/954155938 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff7e4009440 con 0x7ff7e807d850 2019-12-05T06:31:33.797 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:33.784+0000 7ff7ed9b2700 1 --2- 172.21.15.42:0/954155938 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff7e807d850 0x7ff7e807dc80 secure :-1 s=THROTTLE_DONE pgs=612 cs=0 l=1 rx=0x7ff7e4000b80 tx=0x7ff7e4010890).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:33.798 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:33.784+0000 7ff7deffd700 1 -- 172.21.15.42:0/954155938 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7ff7e4009640 con 0x7ff7e807d850 2019-12-05T06:31:33.798 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:33.784+0000 7ff7ed9b2700 1 --2- 172.21.15.42:0/954155938 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff7e807d850 0x7ff7e807dc80 secure :-1 s=THROTTLE_DONE pgs=612 cs=0 l=1 rx=0x7ff7e4000b80 tx=0x7ff7e4010890).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:33.798 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:33.784+0000 7ff7deffd700 1 --2- 172.21.15.42:0/954155938 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff7d003b2a0 0x7ff7d003d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:31:33.798 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:33.784+0000 7ff7deffd700 1 -- 172.21.15.42:0/954155938 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 11191+0+0 (secure 0 0 0) 0x7ff7e404b020 con 0x7ff7e807d850 2019-12-05T06:31:33.798 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:33.784+0000 7ff7ee1b3700 1 --2- 172.21.15.42:0/954155938 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff7d003b2a0 0x7ff7d003d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:31:33.798 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:33.784+0000 7ff7ee1b3700 1 --2- 172.21.15.42:0/954155938 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff7d003b2a0 0x7ff7d003d7b0 crc :-1 s=READY pgs=1166 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:31:33.799 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:33.788+0000 7ff7efc16700 1 -- 172.21.15.42:0/954155938 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7e805d110 con 0x7ff7e807d850 2019-12-05T06:31:33.804 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:33.800+0000 7ff7ed9b2700 1 --2- 172.21.15.42:0/954155938 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff7e807d850 0x7ff7e807dc80 secure :-1 s=THROTTLE_DONE pgs=612 cs=0 l=1 rx=0x7ff7e4000b80 tx=0x7ff7e4010890).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:33.804 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:33.800+0000 7ff7deffd700 1 -- 172.21.15.42:0/954155938 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7ff7e4032020 con 0x7ff7e807d850 2019-12-05T06:31:33.986 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:33.980+0000 7ff7efc16700 1 -- 172.21.15.42:0/954155938 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff7e805b8c0 con 0x7ff7e807d850 2019-12-05T06:31:33.987 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:33.980+0000 7ff7ed9b2700 1 --2- 172.21.15.42:0/954155938 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff7e807d850 0x7ff7e807dc80 secure :-1 s=THROTTLE_DONE pgs=612 cs=0 l=1 rx=0x7ff7e4000b80 tx=0x7ff7e4010890).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:33.987 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:33.980+0000 7ff7deffd700 1 -- 172.21.15.42:0/954155938 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 333 v333) v1 ==== 95+0+2117 (secure 0 0 0) 0x7ff7e4032280 con 0x7ff7e807d850 2019-12-05T06:31:33.989 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:31:33.989 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":333,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":11433,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/2532474042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2532474042},{"type":"v1","addr":"172.21.15.42:6835","nonce":2532474042}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":333}],"filesystems":[{"mdsmap":{"epoch":332,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:31:20.143327+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":109,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11352},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11352":{"gid":11352,"name":"b","rank":0,"incarnation":329,"state":"up:active","state_seq":16,"addr":"172.21.15.40:6833/2055706790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2055706790},{"type":"v1","addr":"172.21.15.40:6833","nonce":2055706790}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:31:33.989 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 333 2019-12-05T06:31:33.992 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:33.984+0000 7ff7efc16700 1 -- 172.21.15.42:0/954155938 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff7d003b2a0 msgr2=0x7ff7d003d7b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:33.992 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:33.984+0000 7ff7efc16700 1 --2- 172.21.15.42:0/954155938 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff7d003b2a0 0x7ff7d003d7b0 crc :-1 s=READY pgs=1166 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:31:33.992 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:33.984+0000 7ff7efc16700 1 -- 172.21.15.42:0/954155938 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff7e807d850 msgr2=0x7ff7e807dc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:33.992 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:33.984+0000 7ff7efc16700 1 --2- 172.21.15.42:0/954155938 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff7e807d850 0x7ff7e807dc80 secure :-1 s=READY pgs=612 cs=0 l=1 rx=0x7ff7e4000b80 tx=0x7ff7e4010890).stop 2019-12-05T06:31:33.993 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:33.984+0000 7ff7efc16700 1 -- 172.21.15.42:0/954155938 shutdown_connections 2019-12-05T06:31:33.993 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:33.984+0000 7ff7efc16700 1 --2- 172.21.15.42:0/954155938 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff7d003b2a0 0x7ff7d003d7b0 unknown :-1 s=CLOSED pgs=1166 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:31:33.993 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:33.984+0000 7ff7efc16700 1 --2- 172.21.15.42:0/954155938 >> v2:172.21.15.40:3300/0 conn(0x7ff7e80a3a60 0x7ff7e8112fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:31:33.993 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:33.984+0000 7ff7efc16700 1 --2- 172.21.15.42:0/954155938 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff7e807d850 0x7ff7e807dc80 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:31:33.993 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:33.984+0000 7ff7efc16700 1 -- 172.21.15.42:0/954155938 shutdown_connections 2019-12-05T06:31:33.993 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:33.984+0000 7ff7efc16700 1 -- 172.21.15.42:0/954155938 wait complete. 2019-12-05T06:31:33.993 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:31:33.988+0000 7ff7efc16700 1 -- 172.21.15.42:0/954155938 >> 172.21.15.42:0/954155938 conn(0x7ff7e80a7f90 msgr2=0x7ff7e809f580 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:31:34.005 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a reported in up:standby state 2019-12-05T06:31:34.005 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 94 secs before thrashing 2019-12-05T06:31:43.184 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:31:43.189 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:32:13.245 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:32:13.249 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:32:43.320 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:32:43.333 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:33:08.007 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:33:08.108 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.095+0000 7f90db151700 1 Processor -- start 2019-12-05T06:33:08.108 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.095+0000 7f90db151700 1 -- start start 2019-12-05T06:33:08.108 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.099+0000 7f90db151700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f90d40a7740 0x7f90d408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:33:08.108 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.099+0000 7f90db151700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f90d407e720 0x7f90d407eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:33:08.109 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.099+0000 7f90db151700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90d40a0710 con 0x7f90d409f2e0 2019-12-05T06:33:08.109 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.099+0000 7f90db151700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90d40a0b00 con 0x7f90d407e720 2019-12-05T06:33:08.109 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.099+0000 7f90db151700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90d408e5b0 con 0x7f90d40a7740 2019-12-05T06:33:08.109 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.099+0000 7f90d8eed700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f90d407e720 0x7f90d407eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:33:08.109 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.099+0000 7f90d8eed700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f90d407e720 0x7f90d407eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:57376/0 (socket says 172.21.15.42:57376) 2019-12-05T06:33:08.109 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.099+0000 7f90d8eed700 1 -- 172.21.15.42:0/1942862765 learned_addr learned my addr 172.21.15.42:0/1942862765 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:33:08.109 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.099+0000 7f90d96ee700 1 --2- 172.21.15.42:0/1942862765 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f90d40a7740 0x7f90d408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:33:08.110 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.099+0000 7f90d8eed700 1 -- 172.21.15.42:0/1942862765 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f90d40a7740 msgr2=0x7f90d408df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:33:08.110 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.099+0000 7f90d8eed700 1 --2- 172.21.15.42:0/1942862765 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f90d40a7740 0x7f90d408df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:33:08.110 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.099+0000 7f90d8eed700 1 -- 172.21.15.42:0/1942862765 >> v1:172.21.15.42:6789/0 conn(0x7f90d409f2e0 legacy=0x7f90d409f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:08.110 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.099+0000 7f90d8eed700 1 -- 172.21.15.42:0/1942862765 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90d407f0a0 con 0x7f90d407e720 2019-12-05T06:33:08.110 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.099+0000 7f90d8eed700 1 --2- 172.21.15.42:0/1942862765 >> v2:172.21.15.42:3300/0 conn(0x7f90d407e720 0x7f90d407eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f90c4008790 tx=0x7f90c400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:08.110 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.099+0000 7f90d8eed700 1 --2- 172.21.15.42:0/1942862765 >> v2:172.21.15.42:3300/0 conn(0x7f90d407e720 0x7f90d407eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f90c4008790 tx=0x7f90c400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:08.110 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.099+0000 7f90d8eed700 1 --2- 172.21.15.42:0/1942862765 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f90d407e720 0x7f90d407eb50 secure :-1 s=READY pgs=2783 cs=0 l=1 rx=0x7f90c4008790 tx=0x7f90c400fd30).ready entity=mon.1 client_cookie=e6e817700318d55d server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:33:08.111 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.099+0000 7f90d8eed700 1 --2- 172.21.15.42:0/1942862765 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f90d407e720 0x7f90d407eb50 secure :-1 s=THROTTLE_DONE pgs=2783 cs=0 l=1 rx=0x7f90c4008790 tx=0x7f90c400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:08.111 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.099+0000 7f90cbfff700 1 -- 172.21.15.42:0/1942862765 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f90c40113b0 con 0x7f90d407e720 2019-12-05T06:33:08.111 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.099+0000 7f90d8eed700 1 --2- 172.21.15.42:0/1942862765 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f90d407e720 0x7f90d407eb50 secure :-1 s=THROTTLE_DONE pgs=2783 cs=0 l=1 rx=0x7f90c4008790 tx=0x7f90c400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:08.111 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.099+0000 7f90d8eed700 1 --2- 172.21.15.42:0/1942862765 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f90d407e720 0x7f90d407eb50 secure :-1 s=THROTTLE_DONE pgs=2783 cs=0 l=1 rx=0x7f90c4008790 tx=0x7f90c400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:08.111 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.099+0000 7f90cbfff700 1 -- 172.21.15.42:0/1942862765 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90c4008bb0 con 0x7f90d407e720 2019-12-05T06:33:08.111 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.099+0000 7f90cbfff700 1 -- 172.21.15.42:0/1942862765 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f90c4011990 con 0x7f90d407e720 2019-12-05T06:33:08.111 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.099+0000 7f90db151700 1 -- 172.21.15.42:0/1942862765 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f90d407e720 msgr2=0x7f90d407eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:08.112 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.099+0000 7f90db151700 1 --2- 172.21.15.42:0/1942862765 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f90d407e720 0x7f90d407eb50 secure :-1 s=READY pgs=2783 cs=0 l=1 rx=0x7f90c4008790 tx=0x7f90c400fd30).stop 2019-12-05T06:33:08.112 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.099+0000 7f90db151700 1 -- 172.21.15.42:0/1942862765 shutdown_connections 2019-12-05T06:33:08.112 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.099+0000 7f90db151700 1 --2- 172.21.15.42:0/1942862765 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f90d40a7740 0x7f90d408df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:33:08.112 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.099+0000 7f90db151700 1 --2- 172.21.15.42:0/1942862765 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f90d407e720 0x7f90d407eb50 unknown :-1 s=CLOSED pgs=2783 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:08.112 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.099+0000 7f90db151700 1 -- 172.21.15.42:0/1942862765 shutdown_connections 2019-12-05T06:33:08.112 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.099+0000 7f90db151700 1 -- 172.21.15.42:0/1942862765 wait complete. 2019-12-05T06:33:08.113 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.099+0000 7f90db151700 1 -- 172.21.15.42:0/1942862765 >> 172.21.15.42:0/1942862765 conn(0x7f90d40aabe0 msgr2=0x7f90d40a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:33:08.116 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.103+0000 7f90db151700 1 Processor -- start 2019-12-05T06:33:08.116 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.103+0000 7f90db151700 1 -- start start 2019-12-05T06:33:08.116 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.103+0000 7f90db151700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f90d4143c20 0x7f90d407a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:33:08.117 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.103+0000 7f90db151700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f90d4142ce0 0x7f90d4143110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:33:08.117 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.103+0000 7f90db151700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90d4144790 con 0x7f90d4113730 2019-12-05T06:33:08.117 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.103+0000 7f90db151700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90d4144250 con 0x7f90d4142ce0 2019-12-05T06:33:08.117 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.103+0000 7f90db151700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90d41444f0 con 0x7f90d4143c20 2019-12-05T06:33:08.117 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.107+0000 7f90d96ee700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f90d4113730 0x7f90d4078800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:38626/0 (socket says 172.21.15.42:38626) 2019-12-05T06:33:08.117 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.107+0000 7f90d8eed700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f90d4143c20 0x7f90d407a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:33:08.117 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.107+0000 7f90d96ee700 1 -- 172.21.15.42:0/538262218 learned_addr learned my addr 172.21.15.42:0/538262218 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:33:08.118 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.107+0000 7f90d9eef700 1 --2- 172.21.15.42:0/538262218 >> v2:172.21.15.40:3300/0 conn(0x7f90d4142ce0 0x7f90d4143110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:33:08.118 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.107+0000 7f90c9ffb700 1 -- 172.21.15.42:0/538262218 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2090600545 0 0) 0x7f90d4144790 con 0x7f90d4113730 2019-12-05T06:33:08.118 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.107+0000 7f90c9ffb700 1 -- 172.21.15.42:0/538262218 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90b40030b0 con 0x7f90d4113730 2019-12-05T06:33:08.118 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.107+0000 7f90d8eed700 1 -- 172.21.15.42:0/538262218 >> v2:172.21.15.40:3300/0 conn(0x7f90d4142ce0 msgr2=0x7f90d4143110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:08.118 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.107+0000 7f90d8eed700 1 --2- 172.21.15.42:0/538262218 >> v2:172.21.15.40:3300/0 conn(0x7f90d4142ce0 0x7f90d4143110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:08.118 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.107+0000 7f90d8eed700 1 -- 172.21.15.42:0/538262218 >> v1:172.21.15.40:6789/0 conn(0x7f90d4113730 legacy=0x7f90d4078800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:08.118 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.107+0000 7f90d8eed700 1 -- 172.21.15.42:0/538262218 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90d4079000 con 0x7f90d4143c20 2019-12-05T06:33:08.119 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.107+0000 7f90d8eed700 1 --2- 172.21.15.42:0/538262218 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f90d4143c20 0x7f90d407a5b0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f90c4008790 tx=0x7f90c4008e60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:08.119 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.107+0000 7f90d8eed700 1 --2- 172.21.15.42:0/538262218 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f90d4143c20 0x7f90d407a5b0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f90c4008790 tx=0x7f90c4008e60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:08.119 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.107+0000 7f90d8eed700 1 --2- 172.21.15.42:0/538262218 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f90d4143c20 0x7f90d407a5b0 secure :-1 s=READY pgs=613 cs=0 l=1 rx=0x7f90c4008790 tx=0x7f90c4008e60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:33:08.119 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.107+0000 7f90d8eed700 1 --2- 172.21.15.42:0/538262218 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f90d4143c20 0x7f90d407a5b0 secure :-1 s=THROTTLE_DONE pgs=613 cs=0 l=1 rx=0x7f90c4008790 tx=0x7f90c4008e60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:08.120 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.107+0000 7f90c9ffb700 1 -- 172.21.15.42:0/538262218 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f90c4007150 con 0x7f90d4143c20 2019-12-05T06:33:08.120 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.107+0000 7f90d8eed700 1 --2- 172.21.15.42:0/538262218 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f90d4143c20 0x7f90d407a5b0 secure :-1 s=THROTTLE_DONE pgs=613 cs=0 l=1 rx=0x7f90c4008790 tx=0x7f90c4008e60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:08.120 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.107+0000 7f90c9ffb700 1 -- 172.21.15.42:0/538262218 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90c40115b0 con 0x7f90d4143c20 2019-12-05T06:33:08.120 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.107+0000 7f90d8eed700 1 --2- 172.21.15.42:0/538262218 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f90d4143c20 0x7f90d407a5b0 secure :-1 s=THROTTLE_DONE pgs=613 cs=0 l=1 rx=0x7f90c4008790 tx=0x7f90c4008e60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:08.120 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.107+0000 7f90c9ffb700 1 -- 172.21.15.42:0/538262218 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f90c4005730 con 0x7f90d4143c20 2019-12-05T06:33:08.121 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.107+0000 7f90db151700 1 -- 172.21.15.42:0/538262218 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90d4143660 con 0x7f90d4143c20 2019-12-05T06:33:08.121 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.107+0000 7f90db151700 1 -- 172.21.15.42:0/538262218 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90d412e410 con 0x7f90d4143c20 2019-12-05T06:33:08.121 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.107+0000 7f90d8eed700 1 --2- 172.21.15.42:0/538262218 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f90d4143c20 0x7f90d407a5b0 secure :-1 s=THROTTLE_DONE pgs=613 cs=0 l=1 rx=0x7f90c4008790 tx=0x7f90c4008e60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:08.121 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.107+0000 7f90c9ffb700 1 -- 172.21.15.42:0/538262218 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f90c40059d0 con 0x7f90d4143c20 2019-12-05T06:33:08.121 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.107+0000 7f90d8eed700 1 --2- 172.21.15.42:0/538262218 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f90d4143c20 0x7f90d407a5b0 secure :-1 s=THROTTLE_DONE pgs=613 cs=0 l=1 rx=0x7f90c4008790 tx=0x7f90c4008e60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:08.121 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.107+0000 7f90c9ffb700 1 --2- 172.21.15.42:0/538262218 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f90b403b260 0x7f90b403d770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:33:08.121 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.107+0000 7f90c9ffb700 1 -- 172.21.15.42:0/538262218 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 11191+0+0 (secure 0 0 0) 0x7f90c404b500 con 0x7f90d4143c20 2019-12-05T06:33:08.122 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.111+0000 7f90d96ee700 1 --2- 172.21.15.42:0/538262218 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f90b403b260 0x7f90b403d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:33:08.122 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.111+0000 7f90d96ee700 1 --2- 172.21.15.42:0/538262218 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f90b403b260 0x7f90b403d770 crc :-1 s=READY pgs=1167 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:33:08.122 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.111+0000 7f90db151700 1 -- 172.21.15.42:0/538262218 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90d4144050 con 0x7f90d4143c20 2019-12-05T06:33:08.128 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.123+0000 7f90d8eed700 1 --2- 172.21.15.42:0/538262218 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f90d4143c20 0x7f90d407a5b0 secure :-1 s=THROTTLE_DONE pgs=613 cs=0 l=1 rx=0x7f90c4008790 tx=0x7f90c4008e60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:08.129 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.123+0000 7f90c9ffb700 1 -- 172.21.15.42:0/538262218 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f90c4005ce0 con 0x7f90d4143c20 2019-12-05T06:33:08.294 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.287+0000 7f90db151700 1 -- 172.21.15.42:0/538262218 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f90d405d110 con 0x7f90d4143c20 2019-12-05T06:33:08.295 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.287+0000 7f90d8eed700 1 --2- 172.21.15.42:0/538262218 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f90d4143c20 0x7f90d407a5b0 secure :-1 s=THROTTLE_DONE pgs=613 cs=0 l=1 rx=0x7f90c4008790 tx=0x7f90c4008e60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:08.295 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.287+0000 7f90c9ffb700 1 -- 172.21.15.42:0/538262218 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 333 v333) v1 ==== 95+0+2117 (secure 0 0 0) 0x7f90c4033080 con 0x7f90d4143c20 2019-12-05T06:33:08.297 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 333 2019-12-05T06:33:08.297 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:33:08.298 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":333,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":11433,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/2532474042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2532474042},{"type":"v1","addr":"172.21.15.42:6835","nonce":2532474042}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":333}],"filesystems":[{"mdsmap":{"epoch":332,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:31:20.143327+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":109,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11352},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11352":{"gid":11352,"name":"b","rank":0,"incarnation":329,"state":"up:active","state_seq":16,"addr":"172.21.15.40:6833/2055706790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2055706790},{"type":"v1","addr":"172.21.15.40:6833","nonce":2055706790}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:33:08.300 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.291+0000 7f90db151700 1 -- 172.21.15.42:0/538262218 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f90b403b260 msgr2=0x7f90b403d770 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:08.300 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.291+0000 7f90db151700 1 --2- 172.21.15.42:0/538262218 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f90b403b260 0x7f90b403d770 crc :-1 s=READY pgs=1167 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:08.300 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.291+0000 7f90db151700 1 -- 172.21.15.42:0/538262218 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f90d4143c20 msgr2=0x7f90d407a5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:08.300 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.291+0000 7f90db151700 1 --2- 172.21.15.42:0/538262218 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f90d4143c20 0x7f90d407a5b0 secure :-1 s=READY pgs=613 cs=0 l=1 rx=0x7f90c4008790 tx=0x7f90c4008e60).stop 2019-12-05T06:33:08.301 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.291+0000 7f90db151700 1 -- 172.21.15.42:0/538262218 shutdown_connections 2019-12-05T06:33:08.301 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.291+0000 7f90db151700 1 --2- 172.21.15.42:0/538262218 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f90b403b260 0x7f90b403d770 unknown :-1 s=CLOSED pgs=1167 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:08.301 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.291+0000 7f90db151700 1 --2- 172.21.15.42:0/538262218 >> v2:172.21.15.40:3300/0 conn(0x7f90d4142ce0 0x7f90d4143110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:08.301 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.291+0000 7f90db151700 1 --2- 172.21.15.42:0/538262218 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f90d4143c20 0x7f90d407a5b0 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:08.302 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.291+0000 7f90db151700 1 -- 172.21.15.42:0/538262218 shutdown_connections 2019-12-05T06:33:08.302 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.291+0000 7f90db151700 1 -- 172.21.15.42:0/538262218 wait complete. 2019-12-05T06:33:08.302 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.291+0000 7f90db151700 1 -- 172.21.15.42:0/538262218 >> 172.21.15.42:0/538262218 conn(0x7f90d4059700 msgr2=0x7f90d4059d20 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:33:08.309 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.b (rank=0) 2019-12-05T06:33:08.310 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2019-12-05T06:33:08.310 INFO:teuthology.orchestra.run:waiting for 300 2019-12-05T06:33:08.325 INFO:tasks.ceph.mds.b:Stopped 2019-12-05T06:33:08.326 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:33:08.417 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.403+0000 7fbfeaf64700 1 Processor -- start 2019-12-05T06:33:08.417 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.403+0000 7fbfeaf64700 1 -- start start 2019-12-05T06:33:08.417 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.407+0000 7fbfeaf64700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fbfe40a7740 0x7fbfe408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:33:08.418 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.407+0000 7fbfeaf64700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fbfe407e720 0x7fbfe407eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:33:08.418 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.407+0000 7fbfeaf64700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbfe409f2e0 0x7fbfe409f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:33:08.418 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.407+0000 7fbfeaf64700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfe40a0b00 con 0x7fbfe407e720 2019-12-05T06:33:08.418 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.407+0000 7fbfeaf64700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfe408e5b0 con 0x7fbfe40a7740 2019-12-05T06:33:08.418 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.407+0000 7fbfeaf64700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfe408e850 con 0x7fbfe409f2e0 2019-12-05T06:33:08.418 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.407+0000 7fbfe9501700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fbfe40a7740 0x7fbfe408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:33:08.418 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.407+0000 7fbfe9501700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fbfe40a7740 0x7fbfe408df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:57390/0 (socket says 172.21.15.42:57390) 2019-12-05T06:33:08.419 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.407+0000 7fbfe9501700 1 -- 172.21.15.42:0/3969015038 learned_addr learned my addr 172.21.15.42:0/3969015038 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:33:08.419 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.407+0000 7fbfe8d00700 1 --2- 172.21.15.42:0/3969015038 >> v2:172.21.15.40:3300/0 conn(0x7fbfe407e720 0x7fbfe407eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:33:08.419 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.407+0000 7fbfe9d02700 1 --2- 172.21.15.42:0/3969015038 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbfe409f2e0 0x7fbfe409f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:33:08.419 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.407+0000 7fbfe9501700 1 -- 172.21.15.42:0/3969015038 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbfe409f2e0 msgr2=0x7fbfe409f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:33:08.419 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.407+0000 7fbfe9501700 1 --2- 172.21.15.42:0/3969015038 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbfe409f2e0 0x7fbfe409f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:33:08.419 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.407+0000 7fbfe9501700 1 -- 172.21.15.42:0/3969015038 >> v2:172.21.15.40:3300/0 conn(0x7fbfe407e720 msgr2=0x7fbfe407eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:33:08.420 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.407+0000 7fbfe9501700 1 --2- 172.21.15.42:0/3969015038 >> v2:172.21.15.40:3300/0 conn(0x7fbfe407e720 0x7fbfe407eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:33:08.420 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.407+0000 7fbfe9501700 1 -- 172.21.15.42:0/3969015038 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfe407f0a0 con 0x7fbfe40a7740 2019-12-05T06:33:08.420 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.407+0000 7fbfe9501700 1 --2- 172.21.15.42:0/3969015038 >> v2:172.21.15.42:3300/0 conn(0x7fbfe40a7740 0x7fbfe408df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fbfe40a0f50 tx=0x7fbfe000fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:08.420 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.407+0000 7fbfe9501700 1 --2- 172.21.15.42:0/3969015038 >> v2:172.21.15.42:3300/0 conn(0x7fbfe40a7740 0x7fbfe408df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fbfe40a0f50 tx=0x7fbfe000fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:08.420 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.407+0000 7fbfe9501700 1 --2- 172.21.15.42:0/3969015038 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbfe40a7740 0x7fbfe408df80 secure :-1 s=READY pgs=2784 cs=0 l=1 rx=0x7fbfe40a0f50 tx=0x7fbfe000fdb0).ready entity=mon.1 client_cookie=d6336ee33c9fe478 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:33:08.420 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.407+0000 7fbfe9501700 1 --2- 172.21.15.42:0/3969015038 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbfe40a7740 0x7fbfe408df80 secure :-1 s=THROTTLE_DONE pgs=2784 cs=0 l=1 rx=0x7fbfe40a0f50 tx=0x7fbfe000fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:08.420 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.407+0000 7fbfe9501700 1 --2- 172.21.15.42:0/3969015038 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbfe40a7740 0x7fbfe408df80 secure :-1 s=THROTTLE_DONE pgs=2784 cs=0 l=1 rx=0x7fbfe40a0f50 tx=0x7fbfe000fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:08.421 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.407+0000 7fbfe9501700 1 --2- 172.21.15.42:0/3969015038 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbfe40a7740 0x7fbfe408df80 secure :-1 s=THROTTLE_DONE pgs=2784 cs=0 l=1 rx=0x7fbfe40a0f50 tx=0x7fbfe000fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:08.421 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.407+0000 7fbfdbfff700 1 -- 172.21.15.42:0/3969015038 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbfe0008c90 con 0x7fbfe40a7740 2019-12-05T06:33:08.421 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.407+0000 7fbfdbfff700 1 -- 172.21.15.42:0/3969015038 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfe00088b0 con 0x7fbfe40a7740 2019-12-05T06:33:08.421 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.407+0000 7fbfdbfff700 1 -- 172.21.15.42:0/3969015038 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbfe0011b50 con 0x7fbfe40a7740 2019-12-05T06:33:08.421 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.407+0000 7fbfeaf64700 1 -- 172.21.15.42:0/3969015038 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbfe40a7740 msgr2=0x7fbfe408df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:08.421 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.407+0000 7fbfeaf64700 1 --2- 172.21.15.42:0/3969015038 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbfe40a7740 0x7fbfe408df80 secure :-1 s=READY pgs=2784 cs=0 l=1 rx=0x7fbfe40a0f50 tx=0x7fbfe000fdb0).stop 2019-12-05T06:33:08.421 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.407+0000 7fbfeaf64700 1 -- 172.21.15.42:0/3969015038 shutdown_connections 2019-12-05T06:33:08.422 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.407+0000 7fbfeaf64700 1 --2- 172.21.15.42:0/3969015038 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbfe409f2e0 0x7fbfe409f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:33:08.422 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.407+0000 7fbfeaf64700 1 --2- 172.21.15.42:0/3969015038 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbfe40a7740 0x7fbfe408df80 unknown :-1 s=CLOSED pgs=2784 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:08.422 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.407+0000 7fbfeaf64700 1 --2- 172.21.15.42:0/3969015038 >> v2:172.21.15.40:3300/0 conn(0x7fbfe407e720 0x7fbfe407eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:33:08.422 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.407+0000 7fbfeaf64700 1 -- 172.21.15.42:0/3969015038 shutdown_connections 2019-12-05T06:33:08.422 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.407+0000 7fbfeaf64700 1 -- 172.21.15.42:0/3969015038 wait complete. 2019-12-05T06:33:08.424 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.407+0000 7fbfeaf64700 1 -- 172.21.15.42:0/3969015038 >> 172.21.15.42:0/3969015038 conn(0x7fbfe40aabe0 msgr2=0x7fbfe40a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:33:08.429 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.411+0000 7fbfeaf64700 1 Processor -- start 2019-12-05T06:33:08.429 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.411+0000 7fbfeaf64700 1 -- start start 2019-12-05T06:33:08.429 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.411+0000 7fbfeaf64700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fbfe4116500 0x7fbfe4117050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:33:08.430 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.411+0000 7fbfeaf64700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fbfe40a1d00 0x7fbfe40a2130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:33:08.430 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.411+0000 7fbfeaf64700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfe4137890 con 0x7fbfe40a34b0 2019-12-05T06:33:08.430 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.411+0000 7fbfeaf64700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfe40a2680 con 0x7fbfe40a1d00 2019-12-05T06:33:08.430 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.411+0000 7fbfeaf64700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfe4130440 con 0x7fbfe4116500 2019-12-05T06:33:08.430 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.411+0000 7fbfe8d00700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fbfe4116500 0x7fbfe4117050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:33:08.430 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.411+0000 7fbfe9501700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fbfe40a34b0 0x7fbfe4130b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:57438/0 (socket says 172.21.15.42:57438) 2019-12-05T06:33:08.431 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.411+0000 7fbfe9501700 1 -- 172.21.15.42:0/2782461906 learned_addr learned my addr 172.21.15.42:0/2782461906 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:33:08.431 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.415+0000 7fbfe9d02700 1 --2- 172.21.15.42:0/2782461906 >> v2:172.21.15.40:3300/0 conn(0x7fbfe40a1d00 0x7fbfe40a2130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:33:08.431 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.415+0000 7fbfd9ffb700 1 -- 172.21.15.42:0/2782461906 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 205135785 0 0) 0x7fbfe4137890 con 0x7fbfe40a34b0 2019-12-05T06:33:08.431 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.415+0000 7fbfe8d00700 1 -- 172.21.15.42:0/2782461906 >> v2:172.21.15.40:3300/0 conn(0x7fbfe40a1d00 msgr2=0x7fbfe40a2130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:08.431 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.415+0000 7fbfe8d00700 1 --2- 172.21.15.42:0/2782461906 >> v2:172.21.15.40:3300/0 conn(0x7fbfe40a1d00 0x7fbfe40a2130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:08.431 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.415+0000 7fbfe8d00700 1 -- 172.21.15.42:0/2782461906 >> v1:172.21.15.42:6789/0 conn(0x7fbfe40a34b0 legacy=0x7fbfe4130b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:08.431 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.415+0000 7fbfe8d00700 1 -- 172.21.15.42:0/2782461906 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfe4138ad0 con 0x7fbfe4116500 2019-12-05T06:33:08.432 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.415+0000 7fbfe8d00700 1 --2- 172.21.15.42:0/2782461906 >> v2:172.21.15.42:3300/0 conn(0x7fbfe4116500 0x7fbfe4117050 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fbfd4002f30 tx=0x7fbfd4010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:08.432 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.415+0000 7fbfe8d00700 1 --2- 172.21.15.42:0/2782461906 >> v2:172.21.15.42:3300/0 conn(0x7fbfe4116500 0x7fbfe4117050 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fbfd4002f30 tx=0x7fbfd4010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:08.432 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.415+0000 7fbfe8d00700 1 --2- 172.21.15.42:0/2782461906 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbfe4116500 0x7fbfe4117050 secure :-1 s=READY pgs=2786 cs=0 l=1 rx=0x7fbfd4002f30 tx=0x7fbfd4010ac0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:33:08.432 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.415+0000 7fbfe8d00700 1 --2- 172.21.15.42:0/2782461906 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbfe4116500 0x7fbfe4117050 secure :-1 s=THROTTLE_DONE pgs=2786 cs=0 l=1 rx=0x7fbfd4002f30 tx=0x7fbfd4010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:08.432 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.415+0000 7fbfe8d00700 1 --2- 172.21.15.42:0/2782461906 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbfe4116500 0x7fbfe4117050 secure :-1 s=THROTTLE_DONE pgs=2786 cs=0 l=1 rx=0x7fbfd4002f30 tx=0x7fbfd4010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:08.432 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.415+0000 7fbfd9ffb700 1 -- 172.21.15.42:0/2782461906 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbfd4012950 con 0x7fbfe4116500 2019-12-05T06:33:08.433 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.415+0000 7fbfe8d00700 1 --2- 172.21.15.42:0/2782461906 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbfe4116500 0x7fbfe4117050 secure :-1 s=THROTTLE_DONE pgs=2786 cs=0 l=1 rx=0x7fbfd4002f30 tx=0x7fbfd4010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:08.433 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.415+0000 7fbfd9ffb700 1 -- 172.21.15.42:0/2782461906 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfd4012570 con 0x7fbfe4116500 2019-12-05T06:33:08.433 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.415+0000 7fbfd9ffb700 1 -- 172.21.15.42:0/2782461906 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbfd4005540 con 0x7fbfe4116500 2019-12-05T06:33:08.433 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.415+0000 7fbfeaf64700 1 -- 172.21.15.42:0/2782461906 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfe4137ac0 con 0x7fbfe4116500 2019-12-05T06:33:08.433 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.415+0000 7fbfeaf64700 1 -- 172.21.15.42:0/2782461906 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfe4138070 con 0x7fbfe4116500 2019-12-05T06:33:08.434 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.415+0000 7fbfe8d00700 1 --2- 172.21.15.42:0/2782461906 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbfe4116500 0x7fbfe4117050 secure :-1 s=THROTTLE_DONE pgs=2786 cs=0 l=1 rx=0x7fbfd4002f30 tx=0x7fbfd4010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:08.434 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.415+0000 7fbfd9ffb700 1 -- 172.21.15.42:0/2782461906 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fbfd4007180 con 0x7fbfe4116500 2019-12-05T06:33:08.434 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.415+0000 7fbfe8d00700 1 --2- 172.21.15.42:0/2782461906 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbfe4116500 0x7fbfe4117050 secure :-1 s=THROTTLE_DONE pgs=2786 cs=0 l=1 rx=0x7fbfd4002f30 tx=0x7fbfd4010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:08.435 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.415+0000 7fbfd9ffb700 1 --2- 172.21.15.42:0/2782461906 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbfc403acd0 0x7fbfc403d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:33:08.435 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.415+0000 7fbfd9ffb700 1 -- 172.21.15.42:0/2782461906 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 11191+0+0 (secure 0 0 0) 0x7fbfd404af40 con 0x7fbfe4116500 2019-12-05T06:33:08.435 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.415+0000 7fbfe9501700 1 --2- 172.21.15.42:0/2782461906 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbfc403acd0 0x7fbfc403d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:33:08.435 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.415+0000 7fbfe9501700 1 --2- 172.21.15.42:0/2782461906 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbfc403acd0 0x7fbfc403d1e0 crc :-1 s=READY pgs=1168 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:33:08.435 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.419+0000 7fbfeaf64700 1 -- 172.21.15.42:0/2782461906 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfe4131300 con 0x7fbfe4116500 2019-12-05T06:33:08.436 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.427+0000 7fbfe8d00700 1 --2- 172.21.15.42:0/2782461906 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbfe4116500 0x7fbfe4117050 secure :-1 s=THROTTLE_DONE pgs=2786 cs=0 l=1 rx=0x7fbfd4002f30 tx=0x7fbfd4010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:08.436 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.427+0000 7fbfd9ffb700 1 -- 172.21.15.42:0/2782461906 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fbfd4031fc0 con 0x7fbfe4116500 2019-12-05T06:33:08.597 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.591+0000 7fbfeaf64700 1 -- 172.21.15.42:0/2782461906 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbfe405d110 con 0x7fbfe4116500 2019-12-05T06:33:08.598 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.591+0000 7fbfe8d00700 1 --2- 172.21.15.42:0/2782461906 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbfe4116500 0x7fbfe4117050 secure :-1 s=THROTTLE_DONE pgs=2786 cs=0 l=1 rx=0x7fbfd4002f30 tx=0x7fbfd4010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:08.598 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.591+0000 7fbfd9ffb700 1 -- 172.21.15.42:0/2782461906 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 333 v333) v1 ==== 95+0+2117 (secure 0 0 0) 0x7fbfd4032220 con 0x7fbfe4116500 2019-12-05T06:33:08.598 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 333 2019-12-05T06:33:08.599 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:33:08.599 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":333,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":11433,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/2532474042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2532474042},{"type":"v1","addr":"172.21.15.42:6835","nonce":2532474042}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":333}],"filesystems":[{"mdsmap":{"epoch":332,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:31:20.143327+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":109,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11352},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11352":{"gid":11352,"name":"b","rank":0,"incarnation":329,"state":"up:active","state_seq":16,"addr":"172.21.15.40:6833/2055706790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2055706790},{"type":"v1","addr":"172.21.15.40:6833","nonce":2055706790}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:33:08.601 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.595+0000 7fbfeaf64700 1 -- 172.21.15.42:0/2782461906 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbfc403acd0 msgr2=0x7fbfc403d1e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:08.601 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.595+0000 7fbfeaf64700 1 --2- 172.21.15.42:0/2782461906 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbfc403acd0 0x7fbfc403d1e0 crc :-1 s=READY pgs=1168 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:08.601 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.595+0000 7fbfeaf64700 1 -- 172.21.15.42:0/2782461906 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbfe4116500 msgr2=0x7fbfe4117050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:08.601 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.595+0000 7fbfeaf64700 1 --2- 172.21.15.42:0/2782461906 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbfe4116500 0x7fbfe4117050 secure :-1 s=READY pgs=2786 cs=0 l=1 rx=0x7fbfd4002f30 tx=0x7fbfd4010ac0).stop 2019-12-05T06:33:08.601 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.595+0000 7fbfeaf64700 1 -- 172.21.15.42:0/2782461906 shutdown_connections 2019-12-05T06:33:08.601 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.595+0000 7fbfeaf64700 1 --2- 172.21.15.42:0/2782461906 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbfc403acd0 0x7fbfc403d1e0 unknown :-1 s=CLOSED pgs=1168 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:08.602 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.595+0000 7fbfeaf64700 1 --2- 172.21.15.42:0/2782461906 >> v2:172.21.15.40:3300/0 conn(0x7fbfe40a1d00 0x7fbfe40a2130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:08.602 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.595+0000 7fbfeaf64700 1 --2- 172.21.15.42:0/2782461906 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbfe4116500 0x7fbfe4117050 unknown :-1 s=CLOSED pgs=2786 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:08.602 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.595+0000 7fbfeaf64700 1 -- 172.21.15.42:0/2782461906 shutdown_connections 2019-12-05T06:33:08.602 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.595+0000 7fbfeaf64700 1 -- 172.21.15.42:0/2782461906 wait complete. 2019-12-05T06:33:08.602 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:08.595+0000 7fbfeaf64700 1 -- 172.21.15.42:0/2782461906 >> 172.21.15.42:0/2782461906 conn(0x7fbfe40aabe0 msgr2=0x7fbfe4059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:33:08.610 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:33:10.611 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:33:10.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:10.707+0000 7f9e07624700 1 Processor -- start 2019-12-05T06:33:10.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:10.707+0000 7f9e07624700 1 -- start start 2019-12-05T06:33:10.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:10.707+0000 7f9e07624700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f9e000a7740 0x7f9e0008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:33:10.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:10.707+0000 7f9e07624700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e000a0710 con 0x7f9e0009fae0 2019-12-05T06:33:10.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:10.707+0000 7f9e07624700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e000a9ba0 con 0x7f9e0007e720 2019-12-05T06:33:10.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:10.707+0000 7f9e07624700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e000a0b00 con 0x7f9e000a7740 2019-12-05T06:33:10.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:10.707+0000 7f9e053c0700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f9e0007e720 0x7f9e0009f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:57448/0 (socket says 172.21.15.42:57448) 2019-12-05T06:33:10.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:10.707+0000 7f9e053c0700 1 -- 172.21.15.42:0/2385544224 learned_addr learned my addr 172.21.15.42:0/2385544224 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:33:10.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:10.707+0000 7f9e05bc1700 1 --2- 172.21.15.42:0/2385544224 >> v2:172.21.15.40:3300/0 conn(0x7f9e000a7740 0x7f9e0008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:33:10.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:10.707+0000 7f9e04bbf700 1 -- 172.21.15.42:0/2385544224 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2474941198 0 0) 0x7f9e000a9ba0 con 0x7f9e0007e720 2019-12-05T06:33:10.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:10.707+0000 7f9e04bbf700 1 -- 172.21.15.42:0/2385544224 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9dec0030b0 con 0x7f9e0007e720 2019-12-05T06:33:10.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:10.707+0000 7f9e04bbf700 1 -- 172.21.15.42:0/2385544224 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4072725644 0 0) 0x7f9e000a0710 con 0x7f9e0009fae0 2019-12-05T06:33:10.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:10.707+0000 7f9e04bbf700 1 -- 172.21.15.42:0/2385544224 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e000a9ba0 con 0x7f9e0009fae0 2019-12-05T06:33:10.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:10.707+0000 7f9e04bbf700 1 -- 172.21.15.42:0/2385544224 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 469582564 0 0) 0x7f9dec0030b0 con 0x7f9e0007e720 2019-12-05T06:33:10.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:10.707+0000 7f9e05bc1700 1 -- 172.21.15.42:0/2385544224 >> v1:172.21.15.42:6789/0 conn(0x7f9e0007e720 legacy=0x7f9e0009f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:10.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:10.707+0000 7f9e05bc1700 1 -- 172.21.15.42:0/2385544224 >> v1:172.21.15.40:6789/0 conn(0x7f9e0009fae0 legacy=0x7f9e0014efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:10.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:10.711+0000 7f9e05bc1700 1 -- 172.21.15.42:0/2385544224 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e0007eb50 con 0x7f9e000a7740 2019-12-05T06:33:10.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:10.711+0000 7f9e05bc1700 1 --2- 172.21.15.42:0/2385544224 >> v2:172.21.15.40:3300/0 conn(0x7f9e000a7740 0x7f9e0008df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f9dfc008790 tx=0x7f9df0001950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:10.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:10.711+0000 7f9e05bc1700 1 --2- 172.21.15.42:0/2385544224 >> v2:172.21.15.40:3300/0 conn(0x7f9e000a7740 0x7f9e0008df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f9dfc008790 tx=0x7f9df0001950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:10.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:10.711+0000 7f9e05bc1700 1 --2- 172.21.15.42:0/2385544224 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9e000a7740 0x7f9e0008df80 secure :-1 s=READY pgs=2100 cs=0 l=1 rx=0x7f9dfc008790 tx=0x7f9df0001950).ready entity=mon.0 client_cookie=95fb14b6562fc5a4 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:33:10.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:10.711+0000 7f9e05bc1700 1 --2- 172.21.15.42:0/2385544224 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9e000a7740 0x7f9e0008df80 secure :-1 s=THROTTLE_DONE pgs=2100 cs=0 l=1 rx=0x7f9dfc008790 tx=0x7f9df0001950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:10.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:10.711+0000 7f9e04bbf700 1 -- 172.21.15.42:0/2385544224 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9df0001750 con 0x7f9e000a7740 2019-12-05T06:33:10.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:10.711+0000 7f9e05bc1700 1 --2- 172.21.15.42:0/2385544224 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9e000a7740 0x7f9e0008df80 secure :-1 s=THROTTLE_DONE pgs=2100 cs=0 l=1 rx=0x7f9dfc008790 tx=0x7f9df0001950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:10.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:10.711+0000 7f9e04bbf700 1 -- 172.21.15.42:0/2385544224 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9df0001560 con 0x7f9e000a7740 2019-12-05T06:33:10.724 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:10.711+0000 7f9e05bc1700 1 --2- 172.21.15.42:0/2385544224 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9e000a7740 0x7f9e0008df80 secure :-1 s=THROTTLE_DONE pgs=2100 cs=0 l=1 rx=0x7f9dfc008790 tx=0x7f9df0001950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:10.724 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:10.711+0000 7f9e04bbf700 1 -- 172.21.15.42:0/2385544224 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9dfc011360 con 0x7f9e000a7740 2019-12-05T06:33:10.724 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:10.711+0000 7f9e07624700 1 -- 172.21.15.42:0/2385544224 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9e000a7740 msgr2=0x7f9e0008df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:10.724 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:10.711+0000 7f9e07624700 1 --2- 172.21.15.42:0/2385544224 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9e000a7740 0x7f9e0008df80 secure :-1 s=READY pgs=2100 cs=0 l=1 rx=0x7f9dfc008790 tx=0x7f9df0001950).stop 2019-12-05T06:33:10.724 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:10.711+0000 7f9e07624700 1 -- 172.21.15.42:0/2385544224 shutdown_connections 2019-12-05T06:33:10.724 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:10.711+0000 7f9e07624700 1 --2- 172.21.15.42:0/2385544224 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9e000a7740 0x7f9e0008df80 unknown :-1 s=CLOSED pgs=2100 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:10.724 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:10.711+0000 7f9e07624700 1 -- 172.21.15.42:0/2385544224 shutdown_connections 2019-12-05T06:33:10.725 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:10.711+0000 7f9e07624700 1 -- 172.21.15.42:0/2385544224 wait complete. 2019-12-05T06:33:10.725 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:10.711+0000 7f9e07624700 1 -- 172.21.15.42:0/2385544224 >> 172.21.15.42:0/2385544224 conn(0x7f9e000aabe0 msgr2=0x7f9e000a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:33:10.729 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:10.715+0000 7f9e07624700 1 Processor -- start 2019-12-05T06:33:10.729 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:10.715+0000 7f9e07624700 1 -- start start 2019-12-05T06:33:10.729 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:10.715+0000 7f9e07624700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9e00116500 0x7f9e0011f190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:33:10.729 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:10.715+0000 7f9e07624700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f9e001195c0 0x7f9e000702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:33:10.729 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:10.715+0000 7f9e07624700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e00070810 con 0x7f9e00120dd0 2019-12-05T06:33:10.730 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:10.715+0000 7f9e07624700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e0011f9a0 con 0x7f9e001195c0 2019-12-05T06:33:10.730 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:10.715+0000 7f9e07624700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e0011fc40 con 0x7f9e00116500 2019-12-05T06:33:10.730 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:10.715+0000 7f9e063c2700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f9e001195c0 0x7f9e000702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:33:10.730 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:10.715+0000 7f9e063c2700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f9e001195c0 0x7f9e000702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:57420/0 (socket says 172.21.15.42:57420) 2019-12-05T06:33:10.730 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:10.715+0000 7f9e053c0700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f9e00120dd0 0x7f9e001201c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:38664/0 (socket says 172.21.15.42:38664) 2019-12-05T06:33:10.730 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:10.715+0000 7f9e063c2700 1 -- 172.21.15.42:0/4036200934 learned_addr learned my addr 172.21.15.42:0/4036200934 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:33:10.731 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:10.715+0000 7f9e05bc1700 1 --2- 172.21.15.42:0/4036200934 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9e00116500 0x7f9e0011f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:33:10.731 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:10.715+0000 7f9e063c2700 1 -- 172.21.15.42:0/4036200934 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9e00116500 msgr2=0x7f9e0011f190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:10.731 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:10.715+0000 7f9e063c2700 1 --2- 172.21.15.42:0/4036200934 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9e00116500 0x7f9e0011f190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:10.731 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:10.715+0000 7f9e063c2700 1 -- 172.21.15.42:0/4036200934 >> v1:172.21.15.40:6789/0 conn(0x7f9e00120dd0 legacy=0x7f9e001201c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:10.731 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:10.715+0000 7f9df67fc700 1 -- 172.21.15.42:0/4036200934 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2250202253 0 0) 0x7f9e00070810 con 0x7f9e00120dd0 2019-12-05T06:33:10.731 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:10.719+0000 7f9e063c2700 1 -- 172.21.15.42:0/4036200934 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e0011f6e0 con 0x7f9e001195c0 2019-12-05T06:33:10.732 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:10.719+0000 7f9e063c2700 1 --2- 172.21.15.42:0/4036200934 >> v2:172.21.15.42:3300/0 conn(0x7f9e001195c0 0x7f9e000702c0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f9e0011fb60 tx=0x7f9df8010b10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:10.732 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:10.719+0000 7f9e063c2700 1 --2- 172.21.15.42:0/4036200934 >> v2:172.21.15.42:3300/0 conn(0x7f9e001195c0 0x7f9e000702c0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f9e0011fb60 tx=0x7f9df8010b10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:10.732 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:10.719+0000 7f9e063c2700 1 --2- 172.21.15.42:0/4036200934 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9e001195c0 0x7f9e000702c0 secure :-1 s=READY pgs=2788 cs=0 l=1 rx=0x7f9e0011fb60 tx=0x7f9df8010b10).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:33:10.733 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:10.719+0000 7f9e063c2700 1 --2- 172.21.15.42:0/4036200934 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9e001195c0 0x7f9e000702c0 secure :-1 s=THROTTLE_DONE pgs=2788 cs=0 l=1 rx=0x7f9e0011fb60 tx=0x7f9df8010b10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:10.733 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:10.719+0000 7f9df67fc700 1 -- 172.21.15.42:0/4036200934 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9df8006dc0 con 0x7f9e001195c0 2019-12-05T06:33:10.734 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:10.719+0000 7f9e063c2700 1 --2- 172.21.15.42:0/4036200934 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9e001195c0 0x7f9e000702c0 secure :-1 s=THROTTLE_DONE pgs=2788 cs=0 l=1 rx=0x7f9e0011fb60 tx=0x7f9df8010b10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:10.734 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:10.719+0000 7f9e07624700 1 -- 172.21.15.42:0/4036200934 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e00070a40 con 0x7f9e001195c0 2019-12-05T06:33:10.735 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:10.719+0000 7f9df67fc700 1 -- 172.21.15.42:0/4036200934 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9df80069e0 con 0x7f9e001195c0 2019-12-05T06:33:10.735 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:10.719+0000 7f9e063c2700 1 --2- 172.21.15.42:0/4036200934 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9e001195c0 0x7f9e000702c0 secure :-1 s=THROTTLE_DONE pgs=2788 cs=0 l=1 rx=0x7f9e0011fb60 tx=0x7f9df8010b10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:10.735 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:10.719+0000 7f9df67fc700 1 -- 172.21.15.42:0/4036200934 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9df8012570 con 0x7f9e001195c0 2019-12-05T06:33:10.736 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:10.719+0000 7f9e07624700 1 -- 172.21.15.42:0/4036200934 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e00070ff0 con 0x7f9e001195c0 2019-12-05T06:33:10.736 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:10.719+0000 7f9e063c2700 1 --2- 172.21.15.42:0/4036200934 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9e001195c0 0x7f9e000702c0 secure :-1 s=THROTTLE_DONE pgs=2788 cs=0 l=1 rx=0x7f9e0011fb60 tx=0x7f9df8010b10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:10.736 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:10.719+0000 7f9df67fc700 1 -- 172.21.15.42:0/4036200934 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f9df8012850 con 0x7f9e001195c0 2019-12-05T06:33:10.736 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:10.719+0000 7f9e063c2700 1 --2- 172.21.15.42:0/4036200934 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9e001195c0 0x7f9e000702c0 secure :-1 s=THROTTLE_DONE pgs=2788 cs=0 l=1 rx=0x7f9e0011fb60 tx=0x7f9df8010b10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:10.736 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:10.719+0000 7f9df67fc700 1 --2- 172.21.15.42:0/4036200934 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9de803ac00 0x7f9de803d110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:33:10.736 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:10.719+0000 7f9df67fc700 1 -- 172.21.15.42:0/4036200934 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 11191+0+0 (secure 0 0 0) 0x7f9df804b5b0 con 0x7f9e001195c0 2019-12-05T06:33:10.737 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:10.719+0000 7f9e05bc1700 1 --2- 172.21.15.42:0/4036200934 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9de803ac00 0x7f9de803d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:33:10.737 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:10.719+0000 7f9e05bc1700 1 --2- 172.21.15.42:0/4036200934 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9de803ac00 0x7f9de803d110 crc :-1 s=READY pgs=1169 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:33:10.737 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:10.723+0000 7f9e07624700 1 -- 172.21.15.42:0/4036200934 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e0005d110 con 0x7f9e001195c0 2019-12-05T06:33:10.738 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:10.731+0000 7f9e063c2700 1 --2- 172.21.15.42:0/4036200934 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9e001195c0 0x7f9e000702c0 secure :-1 s=THROTTLE_DONE pgs=2788 cs=0 l=1 rx=0x7f9e0011fb60 tx=0x7f9df8010b10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:10.738 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:10.731+0000 7f9df67fc700 1 -- 172.21.15.42:0/4036200934 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f9e0011f6e0 con 0x7f9e001195c0 2019-12-05T06:33:10.906 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:10.899+0000 7f9e07624700 1 -- 172.21.15.42:0/4036200934 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9e0005b8c0 con 0x7f9e001195c0 2019-12-05T06:33:10.907 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:10.899+0000 7f9e063c2700 1 --2- 172.21.15.42:0/4036200934 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9e001195c0 0x7f9e000702c0 secure :-1 s=THROTTLE_DONE pgs=2788 cs=0 l=1 rx=0x7f9e0011fb60 tx=0x7f9df8010b10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:10.907 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:10.899+0000 7f9df67fc700 1 -- 172.21.15.42:0/4036200934 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 333 v333) v1 ==== 95+0+2117 (secure 0 0 0) 0x7f9df80324f0 con 0x7f9e001195c0 2019-12-05T06:33:10.908 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 333 2019-12-05T06:33:10.908 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:33:10.908 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":333,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":11433,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/2532474042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2532474042},{"type":"v1","addr":"172.21.15.42:6835","nonce":2532474042}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":333}],"filesystems":[{"mdsmap":{"epoch":332,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:31:20.143327+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":109,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11352},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11352":{"gid":11352,"name":"b","rank":0,"incarnation":329,"state":"up:active","state_seq":16,"addr":"172.21.15.40:6833/2055706790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2055706790},{"type":"v1","addr":"172.21.15.40:6833","nonce":2055706790}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:33:10.910 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:10.903+0000 7f9e07624700 1 -- 172.21.15.42:0/4036200934 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9de803ac00 msgr2=0x7f9de803d110 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:10.910 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:10.903+0000 7f9e07624700 1 --2- 172.21.15.42:0/4036200934 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9de803ac00 0x7f9de803d110 crc :-1 s=READY pgs=1169 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:10.910 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:10.903+0000 7f9e07624700 1 -- 172.21.15.42:0/4036200934 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9e001195c0 msgr2=0x7f9e000702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:10.910 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:10.903+0000 7f9e07624700 1 --2- 172.21.15.42:0/4036200934 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9e001195c0 0x7f9e000702c0 secure :-1 s=READY pgs=2788 cs=0 l=1 rx=0x7f9e0011fb60 tx=0x7f9df8010b10).stop 2019-12-05T06:33:10.911 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:10.903+0000 7f9e07624700 1 -- 172.21.15.42:0/4036200934 shutdown_connections 2019-12-05T06:33:10.911 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:10.903+0000 7f9e07624700 1 --2- 172.21.15.42:0/4036200934 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9de803ac00 0x7f9de803d110 unknown :-1 s=CLOSED pgs=1169 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:10.911 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:10.903+0000 7f9e07624700 1 --2- 172.21.15.42:0/4036200934 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9e001195c0 0x7f9e000702c0 unknown :-1 s=CLOSED pgs=2788 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:10.911 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:10.903+0000 7f9e07624700 1 --2- 172.21.15.42:0/4036200934 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9e00116500 0x7f9e0011f190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:10.911 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:10.903+0000 7f9e07624700 1 -- 172.21.15.42:0/4036200934 shutdown_connections 2019-12-05T06:33:10.911 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:10.903+0000 7f9e07624700 1 -- 172.21.15.42:0/4036200934 wait complete. 2019-12-05T06:33:10.911 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:10.903+0000 7f9e07624700 1 -- 172.21.15.42:0/4036200934 >> 172.21.15.42:0/4036200934 conn(0x7f9e0007ee20 msgr2=0x7f9e0009f3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:33:10.919 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:33:12.920 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:33:13.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:13.019+0000 7fe76ea11700 1 Processor -- start 2019-12-05T06:33:13.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:13.019+0000 7fe76ea11700 1 -- start start 2019-12-05T06:33:13.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:13.019+0000 7fe76ea11700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe7680a7740 0x7fe76808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:33:13.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:13.019+0000 7fe76ea11700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe76807e720 0x7fe76807eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:33:13.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:13.019+0000 7fe76ea11700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7680a0710 con 0x7fe76809f2e0 2019-12-05T06:33:13.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:13.019+0000 7fe76ea11700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7680a0b00 con 0x7fe7680a7740 2019-12-05T06:33:13.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:13.019+0000 7fe76ea11700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe76808e5b0 con 0x7fe76807e720 2019-12-05T06:33:13.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:13.019+0000 7fe76cfae700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe7680a7740 0x7fe76808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:33:13.032 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:13.019+0000 7fe76cfae700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe7680a7740 0x7fe76808df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:57428/0 (socket says 172.21.15.42:57428) 2019-12-05T06:33:13.032 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:13.019+0000 7fe76cfae700 1 -- 172.21.15.42:0/1247636357 learned_addr learned my addr 172.21.15.42:0/1247636357 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:33:13.032 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:13.019+0000 7fe767fff700 1 --2- 172.21.15.42:0/1247636357 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe76807e720 0x7fe76807eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:33:13.032 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:13.019+0000 7fe7677fe700 1 -- 172.21.15.42:0/1247636357 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3934095808 0 0) 0x7fe7680a0710 con 0x7fe76809f2e0 2019-12-05T06:33:13.032 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:13.019+0000 7fe7677fe700 1 -- 172.21.15.42:0/1247636357 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7480030b0 con 0x7fe76809f2e0 2019-12-05T06:33:13.032 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:13.019+0000 7fe767fff700 1 -- 172.21.15.42:0/1247636357 >> v2:172.21.15.42:3300/0 conn(0x7fe7680a7740 msgr2=0x7fe76808df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:33:13.033 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:13.019+0000 7fe767fff700 1 --2- 172.21.15.42:0/1247636357 >> v2:172.21.15.42:3300/0 conn(0x7fe7680a7740 0x7fe76808df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:33:13.033 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:13.019+0000 7fe767fff700 1 -- 172.21.15.42:0/1247636357 >> v1:172.21.15.40:6789/0 conn(0x7fe76809f2e0 legacy=0x7fe76809f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:13.033 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:13.019+0000 7fe767fff700 1 -- 172.21.15.42:0/1247636357 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe76807f0a0 con 0x7fe76807e720 2019-12-05T06:33:13.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:13.019+0000 7fe76cfae700 1 --2- 172.21.15.42:0/1247636357 >> v2:172.21.15.42:3300/0 conn(0x7fe7680a7740 0x7fe76808df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:33:13.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:13.019+0000 7fe767fff700 1 --2- 172.21.15.42:0/1247636357 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe76807e720 0x7fe76807eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fe758008b10 tx=0x7fe75800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:13.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:13.019+0000 7fe767fff700 1 --2- 172.21.15.42:0/1247636357 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe76807e720 0x7fe76807eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fe758008b10 tx=0x7fe75800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:13.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:13.019+0000 7fe767fff700 1 --2- 172.21.15.42:0/1247636357 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe76807e720 0x7fe76807eb50 secure :-1 s=READY pgs=614 cs=0 l=1 rx=0x7fe758008b10 tx=0x7fe75800fff0).ready entity=mon.2 client_cookie=77f3798ec397139f server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:33:13.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:13.019+0000 7fe767fff700 1 --2- 172.21.15.42:0/1247636357 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe76807e720 0x7fe76807eb50 secure :-1 s=THROTTLE_DONE pgs=614 cs=0 l=1 rx=0x7fe758008b10 tx=0x7fe75800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:13.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:13.019+0000 7fe7677fe700 1 -- 172.21.15.42:0/1247636357 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe758011a50 con 0x7fe76807e720 2019-12-05T06:33:13.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:13.019+0000 7fe767fff700 1 --2- 172.21.15.42:0/1247636357 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe76807e720 0x7fe76807eb50 secure :-1 s=THROTTLE_DONE pgs=614 cs=0 l=1 rx=0x7fe758008b10 tx=0x7fe75800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:13.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:13.023+0000 7fe7677fe700 1 -- 172.21.15.42:0/1247636357 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe758011670 con 0x7fe76807e720 2019-12-05T06:33:13.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:13.023+0000 7fe767fff700 1 --2- 172.21.15.42:0/1247636357 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe76807e720 0x7fe76807eb50 secure :-1 s=THROTTLE_DONE pgs=614 cs=0 l=1 rx=0x7fe758008b10 tx=0x7fe75800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:13.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:13.023+0000 7fe7677fe700 1 -- 172.21.15.42:0/1247636357 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe75800a350 con 0x7fe76807e720 2019-12-05T06:33:13.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:13.023+0000 7fe76ea11700 1 -- 172.21.15.42:0/1247636357 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe76807e720 msgr2=0x7fe76807eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:13.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:13.023+0000 7fe76ea11700 1 --2- 172.21.15.42:0/1247636357 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe76807e720 0x7fe76807eb50 secure :-1 s=READY pgs=614 cs=0 l=1 rx=0x7fe758008b10 tx=0x7fe75800fff0).stop 2019-12-05T06:33:13.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:13.023+0000 7fe76ea11700 1 -- 172.21.15.42:0/1247636357 shutdown_connections 2019-12-05T06:33:13.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:13.023+0000 7fe76ea11700 1 --2- 172.21.15.42:0/1247636357 >> v2:172.21.15.42:3300/0 conn(0x7fe7680a7740 0x7fe76808df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:33:13.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:13.023+0000 7fe76ea11700 1 --2- 172.21.15.42:0/1247636357 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe76807e720 0x7fe76807eb50 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:13.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:13.023+0000 7fe76ea11700 1 -- 172.21.15.42:0/1247636357 shutdown_connections 2019-12-05T06:33:13.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:13.023+0000 7fe76ea11700 1 -- 172.21.15.42:0/1247636357 wait complete. 2019-12-05T06:33:13.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:13.023+0000 7fe76ea11700 1 -- 172.21.15.42:0/1247636357 >> 172.21.15.42:0/1247636357 conn(0x7fe7680aabe0 msgr2=0x7fe7680a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:33:13.045 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:13.027+0000 7fe76ea11700 1 Processor -- start 2019-12-05T06:33:13.045 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:13.027+0000 7fe76ea11700 1 -- start start 2019-12-05T06:33:13.045 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:13.027+0000 7fe76ea11700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe76807ddc0 0x7fe768121830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:33:13.045 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:13.027+0000 7fe76ea11700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe768130e50 0x7fe768059c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:33:13.045 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:13.027+0000 7fe76ea11700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe768117cf0 con 0x7fe7681178c0 2019-12-05T06:33:13.045 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:13.027+0000 7fe76ea11700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe768117fd0 con 0x7fe768130e50 2019-12-05T06:33:13.045 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:13.027+0000 7fe76ea11700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe768118270 con 0x7fe76807ddc0 2019-12-05T06:33:13.046 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:13.027+0000 7fe76cfae700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe76807ddc0 0x7fe768121830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:33:13.046 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:13.027+0000 7fe76cfae700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe76807ddc0 0x7fe768121830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:57434/0 (socket says 172.21.15.42:57434) 2019-12-05T06:33:13.046 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:13.027+0000 7fe76cfae700 1 -- 172.21.15.42:0/1181194889 learned_addr learned my addr 172.21.15.42:0/1181194889 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:33:13.046 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:13.027+0000 7fe767fff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fe7681178c0 0x7fe768130650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:57476/0 (socket says 172.21.15.42:57476) 2019-12-05T06:33:13.046 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:13.027+0000 7fe76d7af700 1 --2- 172.21.15.42:0/1181194889 >> v2:172.21.15.40:3300/0 conn(0x7fe768130e50 0x7fe768059c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:33:13.046 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:13.027+0000 7fe7657fa700 1 -- 172.21.15.42:0/1181194889 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 452818441 0 0) 0x7fe758007c40 con 0x7fe7681178c0 2019-12-05T06:33:13.047 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:13.027+0000 7fe7657fa700 1 -- 172.21.15.42:0/1181194889 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7400030b0 con 0x7fe7681178c0 2019-12-05T06:33:13.047 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:13.027+0000 7fe76cfae700 1 -- 172.21.15.42:0/1181194889 >> v2:172.21.15.42:3300/0 conn(0x7fe76807ddc0 msgr2=0x7fe768121830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2019-12-05T06:33:13.047 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:13.027+0000 7fe76cfae700 1 -- 172.21.15.42:0/1181194889 >> v2:172.21.15.42:3300/0 conn(0x7fe76807ddc0 msgr2=0x7fe768121830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2019-12-05T06:33:13.047 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:13.027+0000 7fe76cfae700 1 --2- 172.21.15.42:0/1181194889 >> v2:172.21.15.42:3300/0 conn(0x7fe76807ddc0 0x7fe768121830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-12-05T06:33:13.047 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:13.027+0000 7fe76cfae700 1 --2- 172.21.15.42:0/1181194889 >> v2:172.21.15.42:3300/0 conn(0x7fe76807ddc0 0x7fe768121830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rx=0 tx=0)._fault waiting 0.200000 2019-12-05T06:33:13.047 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:13.027+0000 7fe76d7af700 1 -- 172.21.15.42:0/1181194889 >> v2:172.21.15.42:3300/0 conn(0x7fe76807ddc0 msgr2=0x7fe768121830 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2019-12-05T06:33:13.047 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:13.027+0000 7fe76d7af700 1 --2- 172.21.15.42:0/1181194889 >> v2:172.21.15.42:3300/0 conn(0x7fe76807ddc0 0x7fe768121830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:13.047 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:13.027+0000 7fe76d7af700 1 -- 172.21.15.42:0/1181194889 >> v1:172.21.15.42:6789/0 conn(0x7fe7681178c0 legacy=0x7fe768130650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:13.048 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:13.027+0000 7fe76d7af700 1 -- 172.21.15.42:0/1181194889 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe76805a160 con 0x7fe768130e50 2019-12-05T06:33:13.048 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:13.027+0000 7fe76d7af700 1 --2- 172.21.15.42:0/1181194889 >> v2:172.21.15.40:3300/0 conn(0x7fe768130e50 0x7fe768059c10 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fe768118190 tx=0x7fe7600129f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:13.048 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:13.027+0000 7fe76d7af700 1 --2- 172.21.15.42:0/1181194889 >> v2:172.21.15.40:3300/0 conn(0x7fe768130e50 0x7fe768059c10 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fe768118190 tx=0x7fe7600129f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:13.048 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:13.027+0000 7fe76d7af700 1 --2- 172.21.15.42:0/1181194889 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe768130e50 0x7fe768059c10 secure :-1 s=READY pgs=2103 cs=0 l=1 rx=0x7fe768118190 tx=0x7fe7600129f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:33:13.048 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:13.027+0000 7fe76d7af700 1 --2- 172.21.15.42:0/1181194889 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe768130e50 0x7fe768059c10 secure :-1 s=THROTTLE_DONE pgs=2103 cs=0 l=1 rx=0x7fe768118190 tx=0x7fe7600129f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:13.048 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:13.027+0000 7fe7657fa700 1 -- 172.21.15.42:0/1181194889 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe7600035d0 con 0x7fe768130e50 2019-12-05T06:33:13.049 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:13.027+0000 7fe76d7af700 1 --2- 172.21.15.42:0/1181194889 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe768130e50 0x7fe768059c10 secure :-1 s=THROTTLE_DONE pgs=2103 cs=0 l=1 rx=0x7fe768118190 tx=0x7fe7600129f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:13.049 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:13.027+0000 7fe76d7af700 1 --2- 172.21.15.42:0/1181194889 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe768130e50 0x7fe768059c10 secure :-1 s=THROTTLE_DONE pgs=2103 cs=0 l=1 rx=0x7fe768118190 tx=0x7fe7600129f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:13.049 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:13.027+0000 7fe7657fa700 1 -- 172.21.15.42:0/1181194889 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7600031f0 con 0x7fe768130e50 2019-12-05T06:33:13.049 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:13.027+0000 7fe76ea11700 1 -- 172.21.15.42:0/1181194889 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe76808eb70 con 0x7fe768130e50 2019-12-05T06:33:13.049 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:13.027+0000 7fe7657fa700 1 -- 172.21.15.42:0/1181194889 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe760003bb0 con 0x7fe768130e50 2019-12-05T06:33:13.050 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:13.031+0000 7fe76ea11700 1 -- 172.21.15.42:0/1181194889 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe76808f040 con 0x7fe768130e50 2019-12-05T06:33:13.050 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:13.031+0000 7fe76d7af700 1 --2- 172.21.15.42:0/1181194889 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe768130e50 0x7fe768059c10 secure :-1 s=THROTTLE_DONE pgs=2103 cs=0 l=1 rx=0x7fe768118190 tx=0x7fe7600129f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:13.051 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:13.031+0000 7fe7657fa700 1 -- 172.21.15.42:0/1181194889 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fe760009460 con 0x7fe768130e50 2019-12-05T06:33:13.051 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:13.031+0000 7fe76d7af700 1 --2- 172.21.15.42:0/1181194889 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe768130e50 0x7fe768059c10 secure :-1 s=THROTTLE_DONE pgs=2103 cs=0 l=1 rx=0x7fe768118190 tx=0x7fe7600129f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:13.052 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:13.031+0000 7fe7657fa700 1 --2- 172.21.15.42:0/1181194889 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe74003b270 0x7fe74003d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:33:13.052 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:13.031+0000 7fe7657fa700 1 -- 172.21.15.42:0/1181194889 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 11191+0+0 (secure 0 0 0) 0x7fe76004cf70 con 0x7fe768130e50 2019-12-05T06:33:13.052 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:13.031+0000 7fe76cfae700 1 --2- 172.21.15.42:0/1181194889 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe74003b270 0x7fe74003d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:33:13.052 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:13.031+0000 7fe76cfae700 1 --2- 172.21.15.42:0/1181194889 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe74003b270 0x7fe74003d780 crc :-1 s=READY pgs=1170 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:33:13.052 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:13.035+0000 7fe76ea11700 1 -- 172.21.15.42:0/1181194889 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe76805d110 con 0x7fe768130e50 2019-12-05T06:33:13.053 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:13.043+0000 7fe76d7af700 1 --2- 172.21.15.42:0/1181194889 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe768130e50 0x7fe768059c10 secure :-1 s=THROTTLE_DONE pgs=2103 cs=0 l=1 rx=0x7fe768118190 tx=0x7fe7600129f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:13.053 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:13.043+0000 7fe7657fa700 1 -- 172.21.15.42:0/1181194889 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fe760033f10 con 0x7fe768130e50 2019-12-05T06:33:13.212 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:13.207+0000 7fe76ea11700 1 -- 172.21.15.42:0/1181194889 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe76805b8c0 con 0x7fe768130e50 2019-12-05T06:33:13.214 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:13.207+0000 7fe76d7af700 1 --2- 172.21.15.42:0/1181194889 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe768130e50 0x7fe768059c10 secure :-1 s=THROTTLE_DONE pgs=2103 cs=0 l=1 rx=0x7fe768118190 tx=0x7fe7600129f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:13.214 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:13.207+0000 7fe7657fa700 1 -- 172.21.15.42:0/1181194889 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 333 v333) v1 ==== 95+0+2117 (secure 0 0 0) 0x7fe760034170 con 0x7fe768130e50 2019-12-05T06:33:13.215 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 333 2019-12-05T06:33:13.216 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:33:13.216 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":333,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":11433,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/2532474042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2532474042},{"type":"v1","addr":"172.21.15.42:6835","nonce":2532474042}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":333}],"filesystems":[{"mdsmap":{"epoch":332,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:31:20.143327+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":109,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11352},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11352":{"gid":11352,"name":"b","rank":0,"incarnation":329,"state":"up:active","state_seq":16,"addr":"172.21.15.40:6833/2055706790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2055706790},{"type":"v1","addr":"172.21.15.40:6833","nonce":2055706790}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:33:13.219 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:13.211+0000 7fe76ea11700 1 -- 172.21.15.42:0/1181194889 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe74003b270 msgr2=0x7fe74003d780 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:13.219 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:13.211+0000 7fe76ea11700 1 --2- 172.21.15.42:0/1181194889 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe74003b270 0x7fe74003d780 crc :-1 s=READY pgs=1170 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:13.220 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:13.211+0000 7fe76ea11700 1 -- 172.21.15.42:0/1181194889 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe768130e50 msgr2=0x7fe768059c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:13.220 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:13.211+0000 7fe76ea11700 1 --2- 172.21.15.42:0/1181194889 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe768130e50 0x7fe768059c10 secure :-1 s=READY pgs=2103 cs=0 l=1 rx=0x7fe768118190 tx=0x7fe7600129f0).stop 2019-12-05T06:33:13.220 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:13.211+0000 7fe76ea11700 1 -- 172.21.15.42:0/1181194889 shutdown_connections 2019-12-05T06:33:13.220 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:13.211+0000 7fe76ea11700 1 --2- 172.21.15.42:0/1181194889 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe74003b270 0x7fe74003d780 unknown :-1 s=CLOSED pgs=1170 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:13.220 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:13.211+0000 7fe76ea11700 1 --2- 172.21.15.42:0/1181194889 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe768130e50 0x7fe768059c10 unknown :-1 s=CLOSED pgs=2103 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:13.220 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:13.211+0000 7fe76ea11700 1 --2- 172.21.15.42:0/1181194889 >> v2:172.21.15.42:3300/0 conn(0x7fe76807ddc0 0x7fe768121830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:13.220 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:13.211+0000 7fe76ea11700 1 -- 172.21.15.42:0/1181194889 shutdown_connections 2019-12-05T06:33:13.221 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:13.211+0000 7fe76ea11700 1 -- 172.21.15.42:0/1181194889 wait complete. 2019-12-05T06:33:13.221 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:13.211+0000 7fe76ea11700 1 -- 172.21.15.42:0/1181194889 >> 172.21.15.42:0/1181194889 conn(0x7fe7680a9640 msgr2=0x7fe76809f460 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:33:13.233 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:33:13.395 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:33:13.398 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:33:15.233 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:33:15.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:15.327+0000 7f75af984700 1 Processor -- start 2019-12-05T06:33:15.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:15.327+0000 7f75af984700 1 -- start start 2019-12-05T06:33:15.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:15.327+0000 7f75af984700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f75a808df80 0x7f75a809f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:33:15.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:15.327+0000 7f75af984700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f75a809f830 0x7f75a814efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:33:15.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:15.327+0000 7f75af984700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75a809fc60 con 0x7f75a80a7740 2019-12-05T06:33:15.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:15.327+0000 7f75af984700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75a80a0b00 con 0x7f75a809f830 2019-12-05T06:33:15.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:15.327+0000 7f75af984700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75a808e5b0 con 0x7f75a808df80 2019-12-05T06:33:15.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:15.327+0000 7f75ae722700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f75a809f830 0x7f75a814efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:33:15.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:15.327+0000 7f75ae722700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f75a809f830 0x7f75a814efe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:57448/0 (socket says 172.21.15.42:57448) 2019-12-05T06:33:15.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:15.327+0000 7f75ae722700 1 -- 172.21.15.42:0/2205187532 learned_addr learned my addr 172.21.15.42:0/2205187532 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:33:15.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:15.327+0000 7f75ad720700 1 --2- 172.21.15.42:0/2205187532 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f75a808df80 0x7f75a809f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:33:15.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:15.327+0000 7f75ae722700 1 -- 172.21.15.42:0/2205187532 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f75a808df80 msgr2=0x7f75a809f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:33:15.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:15.327+0000 7f75ae722700 1 --2- 172.21.15.42:0/2205187532 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f75a808df80 0x7f75a809f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:33:15.341 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:15.327+0000 7f75ae722700 1 -- 172.21.15.42:0/2205187532 >> v1:172.21.15.40:6789/0 conn(0x7f75a80a7740 legacy=0x7f75a807e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:15.341 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:15.327+0000 7f75ae722700 1 -- 172.21.15.42:0/2205187532 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75a80a0710 con 0x7f75a809f830 2019-12-05T06:33:15.341 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:15.327+0000 7f75ae722700 1 --2- 172.21.15.42:0/2205187532 >> v2:172.21.15.42:3300/0 conn(0x7f75a809f830 0x7f75a814efe0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f75a0008b10 tx=0x7f75a00100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:15.341 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:15.327+0000 7f75ae722700 1 --2- 172.21.15.42:0/2205187532 >> v2:172.21.15.42:3300/0 conn(0x7f75a809f830 0x7f75a814efe0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f75a0008b10 tx=0x7f75a00100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:15.341 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:15.327+0000 7f75ae722700 1 --2- 172.21.15.42:0/2205187532 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f75a809f830 0x7f75a814efe0 secure :-1 s=READY pgs=2791 cs=0 l=1 rx=0x7f75a0008b10 tx=0x7f75a00100b0).ready entity=mon.1 client_cookie=c86d176842865a23 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:33:15.341 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:15.331+0000 7f75ae722700 1 --2- 172.21.15.42:0/2205187532 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f75a809f830 0x7f75a814efe0 secure :-1 s=THROTTLE_DONE pgs=2791 cs=0 l=1 rx=0x7f75a0008b10 tx=0x7f75a00100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:15.342 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:15.331+0000 7f75acf1f700 1 -- 172.21.15.42:0/2205187532 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f75a0010da0 con 0x7f75a809f830 2019-12-05T06:33:15.342 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:15.331+0000 7f75ae722700 1 --2- 172.21.15.42:0/2205187532 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f75a809f830 0x7f75a814efe0 secure :-1 s=THROTTLE_DONE pgs=2791 cs=0 l=1 rx=0x7f75a0008b10 tx=0x7f75a00100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:15.342 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:15.331+0000 7f75acf1f700 1 -- 172.21.15.42:0/2205187532 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75a00109c0 con 0x7f75a809f830 2019-12-05T06:33:15.342 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:15.331+0000 7f75ae722700 1 --2- 172.21.15.42:0/2205187532 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f75a809f830 0x7f75a814efe0 secure :-1 s=THROTTLE_DONE pgs=2791 cs=0 l=1 rx=0x7f75a0008b10 tx=0x7f75a00100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:15.342 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:15.331+0000 7f75af984700 1 -- 172.21.15.42:0/2205187532 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f75a809f830 msgr2=0x7f75a814efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:15.342 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:15.331+0000 7f75af984700 1 --2- 172.21.15.42:0/2205187532 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f75a809f830 0x7f75a814efe0 secure :-1 s=READY pgs=2791 cs=0 l=1 rx=0x7f75a0008b10 tx=0x7f75a00100b0).stop 2019-12-05T06:33:15.342 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:15.331+0000 7f75acf1f700 1 -- 172.21.15.42:0/2205187532 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f75a00114a0 con 0x7f75a809f830 2019-12-05T06:33:15.343 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:15.331+0000 7f75af984700 1 -- 172.21.15.42:0/2205187532 shutdown_connections 2019-12-05T06:33:15.343 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:15.331+0000 7f75af984700 1 --2- 172.21.15.42:0/2205187532 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f75a809f830 0x7f75a814efe0 unknown :-1 s=CLOSED pgs=2791 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:15.343 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:15.331+0000 7f75af984700 1 --2- 172.21.15.42:0/2205187532 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f75a808df80 0x7f75a809f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:33:15.343 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:15.331+0000 7f75af984700 1 -- 172.21.15.42:0/2205187532 shutdown_connections 2019-12-05T06:33:15.343 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:15.331+0000 7f75af984700 1 -- 172.21.15.42:0/2205187532 wait complete. 2019-12-05T06:33:15.343 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:15.331+0000 7f75af984700 1 -- 172.21.15.42:0/2205187532 >> 172.21.15.42:0/2205187532 conn(0x7f75a80aabe0 msgr2=0x7f75a80a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:33:15.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:15.331+0000 7f75af984700 1 Processor -- start 2019-12-05T06:33:15.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:15.331+0000 7f75af984700 1 -- start start 2019-12-05T06:33:15.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:15.335+0000 7f75af984700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f75a811a650 0x7f75a81161c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:33:15.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:15.335+0000 7f75af984700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f75a8117930 0x7f75a8118260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:33:15.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:15.335+0000 7f75af984700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f75a81187b0 0x7f75a8122230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:33:15.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:15.335+0000 7f75af984700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75a81219d0 con 0x7f75a81187b0 2019-12-05T06:33:15.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:15.335+0000 7f75af984700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75a811aba0 con 0x7f75a811a650 2019-12-05T06:33:15.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:15.335+0000 7f75af984700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75a8121c70 con 0x7f75a8117930 2019-12-05T06:33:15.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:15.335+0000 7f75adf21700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f75a811a650 0x7f75a81161c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:33:15.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:15.335+0000 7f75ad720700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f75a8117930 0x7f75a8118260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:33:15.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:15.335+0000 7f75ad720700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f75a8117930 0x7f75a8118260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:57916/0 (socket says 172.21.15.42:57916) 2019-12-05T06:33:15.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:15.335+0000 7f75adf21700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f75a811a650 0x7f75a81161c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:57452/0 (socket says 172.21.15.42:57452) 2019-12-05T06:33:15.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:15.335+0000 7f75ad720700 1 -- 172.21.15.42:0/656424770 learned_addr learned my addr 172.21.15.42:0/656424770 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:33:15.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:15.335+0000 7f75ae722700 1 --2- 172.21.15.42:0/656424770 >> v2:172.21.15.40:3300/0 conn(0x7f75a81187b0 0x7f75a8122230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:33:15.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:15.335+0000 7f75ad720700 1 -- 172.21.15.42:0/656424770 >> v2:172.21.15.42:3300/0 conn(0x7f75a811a650 msgr2=0x7f75a81161c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:15.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:15.335+0000 7f75ad720700 1 --2- 172.21.15.42:0/656424770 >> v2:172.21.15.42:3300/0 conn(0x7f75a811a650 0x7f75a81161c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:15.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:15.335+0000 7f75ad720700 1 -- 172.21.15.42:0/656424770 >> v2:172.21.15.40:3300/0 conn(0x7f75a81187b0 msgr2=0x7f75a8122230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:15.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:15.335+0000 7f75ad720700 1 --2- 172.21.15.42:0/656424770 >> v2:172.21.15.40:3300/0 conn(0x7f75a81187b0 0x7f75a8122230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:15.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:15.335+0000 7f75ad720700 1 -- 172.21.15.42:0/656424770 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75a8116710 con 0x7f75a8117930 2019-12-05T06:33:15.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:15.335+0000 7f75ad720700 1 --2- 172.21.15.42:0/656424770 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f75a8117930 0x7f75a8118260 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f75a8121b90 tx=0x7f7598010cc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:15.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:15.335+0000 7f75ad720700 1 --2- 172.21.15.42:0/656424770 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f75a8117930 0x7f75a8118260 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f75a8121b90 tx=0x7f7598010cc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:15.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:15.335+0000 7f75ad720700 1 --2- 172.21.15.42:0/656424770 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f75a8117930 0x7f75a8118260 secure :-1 s=READY pgs=615 cs=0 l=1 rx=0x7f75a8121b90 tx=0x7f7598010cc0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:33:15.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:15.335+0000 7f75ad720700 1 --2- 172.21.15.42:0/656424770 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f75a8117930 0x7f75a8118260 secure :-1 s=THROTTLE_DONE pgs=615 cs=0 l=1 rx=0x7f75a8121b90 tx=0x7f7598010cc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:15.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:15.335+0000 7f759e7fc700 1 -- 172.21.15.42:0/656424770 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7598005930 con 0x7f75a8117930 2019-12-05T06:33:15.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:15.335+0000 7f75ad720700 1 --2- 172.21.15.42:0/656424770 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f75a8117930 0x7f75a8118260 secure :-1 s=THROTTLE_DONE pgs=615 cs=0 l=1 rx=0x7f75a8121b90 tx=0x7f7598010cc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:15.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:15.335+0000 7f75ad720700 1 --2- 172.21.15.42:0/656424770 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f75a8117930 0x7f75a8118260 secure :-1 s=THROTTLE_DONE pgs=615 cs=0 l=1 rx=0x7f75a8121b90 tx=0x7f7598010cc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:15.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:15.335+0000 7f759e7fc700 1 -- 172.21.15.42:0/656424770 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7598007160 con 0x7f75a8117930 2019-12-05T06:33:15.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:15.335+0000 7f75af984700 1 -- 172.21.15.42:0/656424770 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75a8122780 con 0x7f75a8117930 2019-12-05T06:33:15.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:15.335+0000 7f759e7fc700 1 -- 172.21.15.42:0/656424770 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7598012300 con 0x7f75a8117930 2019-12-05T06:33:15.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:15.335+0000 7f75af984700 1 -- 172.21.15.42:0/656424770 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75a8122c50 con 0x7f75a8117930 2019-12-05T06:33:15.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:15.335+0000 7f75ad720700 1 --2- 172.21.15.42:0/656424770 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f75a8117930 0x7f75a8118260 secure :-1 s=THROTTLE_DONE pgs=615 cs=0 l=1 rx=0x7f75a8121b90 tx=0x7f7598010cc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:15.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:15.335+0000 7f759e7fc700 1 -- 172.21.15.42:0/656424770 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f7598012610 con 0x7f75a8117930 2019-12-05T06:33:15.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:15.335+0000 7f75ad720700 1 --2- 172.21.15.42:0/656424770 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f75a8117930 0x7f75a8118260 secure :-1 s=THROTTLE_DONE pgs=615 cs=0 l=1 rx=0x7f75a8121b90 tx=0x7f7598010cc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:15.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:15.339+0000 7f759e7fc700 1 --2- 172.21.15.42:0/656424770 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f759003ae00 0x7f759003d310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:33:15.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:15.339+0000 7f759e7fc700 1 -- 172.21.15.42:0/656424770 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 11191+0+0 (secure 0 0 0) 0x7f759804b4a0 con 0x7f75a8117930 2019-12-05T06:33:15.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:15.339+0000 7f75adf21700 1 --2- 172.21.15.42:0/656424770 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f759003ae00 0x7f759003d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:33:15.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:15.339+0000 7f75adf21700 1 --2- 172.21.15.42:0/656424770 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f759003ae00 0x7f759003d310 crc :-1 s=READY pgs=1171 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:33:15.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:15.343+0000 7f75af984700 1 -- 172.21.15.42:0/656424770 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75a805d110 con 0x7f75a8117930 2019-12-05T06:33:15.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:15.351+0000 7f75ad720700 1 --2- 172.21.15.42:0/656424770 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f75a8117930 0x7f75a8118260 secure :-1 s=THROTTLE_DONE pgs=615 cs=0 l=1 rx=0x7f75a8121b90 tx=0x7f7598010cc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:15.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:15.351+0000 7f759e7fc700 1 -- 172.21.15.42:0/656424770 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f7598032520 con 0x7f75a8117930 2019-12-05T06:33:15.525 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:15.519+0000 7f75af984700 1 -- 172.21.15.42:0/656424770 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f75a805b8c0 con 0x7f75a8117930 2019-12-05T06:33:15.527 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:15.519+0000 7f75ad720700 1 -- 172.21.15.42:0/656424770 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f75a8117930 msgr2=0x7f75a8118260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2019-12-05T06:33:15.527 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:15.519+0000 7f75ad720700 1 -- 172.21.15.42:0/656424770 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f75a8117930 msgr2=0x7f75a8118260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2019-12-05T06:33:15.527 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:15.519+0000 7f75ad720700 1 --2- 172.21.15.42:0/656424770 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f75a8117930 0x7f75a8118260 secure :-1 s=READY pgs=615 cs=0 l=1 rx=0x7f75a8121b90 tx=0x7f7598010cc0).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-12-05T06:33:15.528 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:15.519+0000 7f75ad720700 1 --2- 172.21.15.42:0/656424770 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f75a8117930 0x7f75a8118260 secure :-1 s=READY pgs=615 cs=0 l=1 rx=0x7f75a8121b90 tx=0x7f7598010cc0).stop 2019-12-05T06:33:15.528 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:15.519+0000 7f759e7fc700 1 -- 172.21.15.42:0/656424770 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f75a8117930 msgr2=0x7f75a8118260 unknown :-1 s=STATE_CLOSED l=1).mark_down 2019-12-05T06:33:15.528 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:15.519+0000 7f759e7fc700 1 --2- 172.21.15.42:0/656424770 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f75a8117930 0x7f75a8118260 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:15.528 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:15.519+0000 7f759e7fc700 1 --2- 172.21.15.42:0/656424770 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7590044410 0x7f7590046850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:33:15.528 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:15.519+0000 7f759e7fc700 1 --2- 172.21.15.42:0/656424770 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7590046ec0 0x7f75900493a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:33:15.528 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:15.519+0000 7f759e7fc700 1 --2- 172.21.15.42:0/656424770 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7590049b00 0x7f759004c010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:33:15.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:15.519+0000 7f759e7fc700 1 -- 172.21.15.42:0/656424770 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f759004c7a0 con 0x7f7590046ec0 2019-12-05T06:33:15.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:15.519+0000 7f759e7fc700 1 -- 172.21.15.42:0/656424770 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f759004c990 con 0x7f7590049b00 2019-12-05T06:33:15.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:15.519+0000 7f759e7fc700 1 -- 172.21.15.42:0/656424770 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f759004cbd0 con 0x7f7590044410 2019-12-05T06:33:15.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:15.519+0000 7f75ae722700 1 --2- 172.21.15.42:0/656424770 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7590046ec0 0x7f75900493a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:33:15.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:15.519+0000 7f75adf21700 1 --2- 172.21.15.42:0/656424770 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7590049b00 0x7f759004c010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:33:15.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:15.519+0000 7f75ad720700 1 --2- 172.21.15.42:0/656424770 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7590044410 0x7f7590046850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:33:15.531 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:15.519+0000 7f75ae722700 1 -- 172.21.15.42:0/656424770 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7590044410 msgr2=0x7f7590046850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:15.531 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:15.519+0000 7f75ae722700 1 --2- 172.21.15.42:0/656424770 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7590044410 0x7f7590046850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:15.531 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:15.519+0000 7f75ae722700 1 -- 172.21.15.42:0/656424770 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7590049b00 msgr2=0x7f759004c010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:15.531 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:15.519+0000 7f75ae722700 1 --2- 172.21.15.42:0/656424770 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7590049b00 0x7f759004c010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:15.532 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:15.519+0000 7f75ae722700 1 -- 172.21.15.42:0/656424770 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f759004ce70 con 0x7f7590046ec0 2019-12-05T06:33:15.532 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:15.519+0000 7f75ae722700 1 -- 172.21.15.42:0/656424770 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f75a0011780 con 0x7f7590046ec0 2019-12-05T06:33:15.532 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:15.519+0000 7f75ae722700 1 --2- 172.21.15.42:0/656424770 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7590046ec0 0x7f75900493a0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f75a0008b10 tx=0x7f75a0011980).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:15.532 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:15.519+0000 7f75ae722700 1 --2- 172.21.15.42:0/656424770 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7590046ec0 0x7f75900493a0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f75a0008b10 tx=0x7f75a0011980).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:15.532 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:15.519+0000 7f75ae722700 1 --2- 172.21.15.42:0/656424770 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7590046ec0 0x7f75900493a0 secure :-1 s=READY pgs=2105 cs=0 l=1 rx=0x7f75a0008b10 tx=0x7f75a0011980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:33:15.532 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:15.523+0000 7f75ae722700 1 --2- 172.21.15.42:0/656424770 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7590046ec0 0x7f75900493a0 secure :-1 s=THROTTLE_DONE pgs=2105 cs=0 l=1 rx=0x7f75a0008b10 tx=0x7f75a0011980).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:15.532 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:15.523+0000 7f759e7fc700 1 -- 172.21.15.42:0/656424770 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f75a0011780 con 0x7f7590046ec0 2019-12-05T06:33:15.533 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:15.523+0000 7f75ae722700 1 --2- 172.21.15.42:0/656424770 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7590046ec0 0x7f75900493a0 secure :-1 s=THROTTLE_DONE pgs=2105 cs=0 l=1 rx=0x7f75a0008b10 tx=0x7f75a0011980).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:15.533 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:15.523+0000 7f759e7fc700 1 -- 172.21.15.42:0/656424770 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75a00110c0 con 0x7f7590046ec0 2019-12-05T06:33:15.533 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:15.523+0000 7f75ae722700 1 --2- 172.21.15.42:0/656424770 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7590046ec0 0x7f75900493a0 secure :-1 s=THROTTLE_DONE pgs=2105 cs=0 l=1 rx=0x7f75a0008b10 tx=0x7f75a0011980).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:15.533 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:15.523+0000 7f75ae722700 1 --2- 172.21.15.42:0/656424770 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7590046ec0 0x7f75900493a0 secure :-1 s=THROTTLE_DONE pgs=2105 cs=0 l=1 rx=0x7f75a0008b10 tx=0x7f75a0011980).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:15.533 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:15.523+0000 7f759e7fc700 1 -- 172.21.15.42:0/656424770 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f75a000f280 con 0x7f7590046ec0 2019-12-05T06:33:15.533 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:15.523+0000 7f759e7fc700 1 -- 172.21.15.42:0/656424770 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 333 v333) v1 ==== 95+0+2117 (secure 0 0 0) 0x7f75a004bbc0 con 0x7f7590046ec0 2019-12-05T06:33:15.534 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 333 2019-12-05T06:33:15.534 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:33:15.534 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":333,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":11433,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/2532474042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2532474042},{"type":"v1","addr":"172.21.15.42:6835","nonce":2532474042}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":333}],"filesystems":[{"mdsmap":{"epoch":332,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:31:20.143327+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":109,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11352},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11352":{"gid":11352,"name":"b","rank":0,"incarnation":329,"state":"up:active","state_seq":16,"addr":"172.21.15.40:6833/2055706790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2055706790},{"type":"v1","addr":"172.21.15.40:6833","nonce":2055706790}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:33:15.536 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:15.527+0000 7f75af984700 1 -- 172.21.15.42:0/656424770 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f759003ae00 msgr2=0x7f759003d310 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:15.536 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:15.527+0000 7f75af984700 1 --2- 172.21.15.42:0/656424770 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f759003ae00 0x7f759003d310 crc :-1 s=READY pgs=1171 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:15.536 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:15.527+0000 7f75af984700 1 -- 172.21.15.42:0/656424770 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7590046ec0 msgr2=0x7f75900493a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:15.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:15.527+0000 7f75af984700 1 --2- 172.21.15.42:0/656424770 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7590046ec0 0x7f75900493a0 secure :-1 s=READY pgs=2105 cs=0 l=1 rx=0x7f75a0008b10 tx=0x7f75a0011980).stop 2019-12-05T06:33:15.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:15.527+0000 7f75ae722700 1 -- 172.21.15.42:0/656424770 reap_dead start 2019-12-05T06:33:15.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:15.527+0000 7f75ae722700 1 -- 172.21.15.42:0/656424770 reap_dead start 2019-12-05T06:33:15.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:15.527+0000 7f75af984700 1 -- 172.21.15.42:0/656424770 shutdown_connections 2019-12-05T06:33:15.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:15.527+0000 7f75af984700 1 -- 172.21.15.42:0/656424770 shutdown_connections 2019-12-05T06:33:15.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:15.527+0000 7f75af984700 1 -- 172.21.15.42:0/656424770 wait complete. 2019-12-05T06:33:15.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:15.527+0000 7f75af984700 1 -- 172.21.15.42:0/656424770 >> 172.21.15.42:0/656424770 conn(0x7f75a809f4c0 msgr2=0x7f75a80a97c0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:33:15.548 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:33:17.549 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:33:17.654 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:17.643+0000 7f75df653700 1 Processor -- start 2019-12-05T06:33:17.654 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:17.643+0000 7f75df653700 1 -- start start 2019-12-05T06:33:17.654 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:17.643+0000 7f75df653700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f75d80a7740 0x7f75d808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:33:17.655 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:17.643+0000 7f75df653700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75d80a0710 con 0x7f75d809fae0 2019-12-05T06:33:17.655 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:17.643+0000 7f75df653700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75d80a9ba0 con 0x7f75d807e720 2019-12-05T06:33:17.655 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:17.643+0000 7f75df653700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75d80a0b00 con 0x7f75d80a7740 2019-12-05T06:33:17.655 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:17.643+0000 7f75dd3ef700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f75d807e720 0x7f75d809f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:57504/0 (socket says 172.21.15.42:57504) 2019-12-05T06:33:17.655 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:17.643+0000 7f75dd3ef700 1 -- 172.21.15.42:0/3118019790 learned_addr learned my addr 172.21.15.42:0/3118019790 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:33:17.655 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:17.643+0000 7f75ddbf0700 1 --2- 172.21.15.42:0/3118019790 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f75d80a7740 0x7f75d808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:33:17.655 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:17.643+0000 7f75dcbee700 1 -- 172.21.15.42:0/3118019790 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3507686754 0 0) 0x7f75d80a9ba0 con 0x7f75d807e720 2019-12-05T06:33:17.656 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:17.643+0000 7f75dcbee700 1 -- 172.21.15.42:0/3118019790 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75c40030b0 con 0x7f75d807e720 2019-12-05T06:33:17.656 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:17.643+0000 7f75dcbee700 1 -- 172.21.15.42:0/3118019790 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3504870926 0 0) 0x7f75d80a0710 con 0x7f75d809fae0 2019-12-05T06:33:17.656 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:17.643+0000 7f75ddbf0700 1 -- 172.21.15.42:0/3118019790 >> v1:172.21.15.42:6789/0 conn(0x7f75d807e720 legacy=0x7f75d809f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:17.656 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:17.643+0000 7f75ddbf0700 1 -- 172.21.15.42:0/3118019790 >> v1:172.21.15.40:6789/0 conn(0x7f75d809fae0 legacy=0x7f75d814efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:17.656 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:17.643+0000 7f75ddbf0700 1 -- 172.21.15.42:0/3118019790 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75d807eb50 con 0x7f75d80a7740 2019-12-05T06:33:17.656 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:17.643+0000 7f75ddbf0700 1 --2- 172.21.15.42:0/3118019790 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f75d80a7740 0x7f75d808df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f75d40032b0 tx=0x7f75d400ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:17.657 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:17.643+0000 7f75ddbf0700 1 --2- 172.21.15.42:0/3118019790 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f75d80a7740 0x7f75d808df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f75d40032b0 tx=0x7f75d400ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:17.657 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:17.643+0000 7f75ddbf0700 1 --2- 172.21.15.42:0/3118019790 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f75d80a7740 0x7f75d808df80 secure :-1 s=READY pgs=616 cs=0 l=1 rx=0x7f75d40032b0 tx=0x7f75d400ff70).ready entity=mon.2 client_cookie=3a9a67cd73c0e950 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:33:17.657 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:17.643+0000 7f75ddbf0700 1 --2- 172.21.15.42:0/3118019790 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f75d80a7740 0x7f75d808df80 secure :-1 s=THROTTLE_DONE pgs=616 cs=0 l=1 rx=0x7f75d40032b0 tx=0x7f75d400ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:17.657 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:17.643+0000 7f75dcbee700 1 -- 172.21.15.42:0/3118019790 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f75d4011a10 con 0x7f75d80a7740 2019-12-05T06:33:17.657 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:17.643+0000 7f75ddbf0700 1 --2- 172.21.15.42:0/3118019790 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f75d80a7740 0x7f75d808df80 secure :-1 s=THROTTLE_DONE pgs=616 cs=0 l=1 rx=0x7f75d40032b0 tx=0x7f75d400ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:17.657 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:17.643+0000 7f75dcbee700 1 -- 172.21.15.42:0/3118019790 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75d4011630 con 0x7f75d80a7740 2019-12-05T06:33:17.657 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:17.643+0000 7f75ddbf0700 1 --2- 172.21.15.42:0/3118019790 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f75d80a7740 0x7f75d808df80 secure :-1 s=THROTTLE_DONE pgs=616 cs=0 l=1 rx=0x7f75d40032b0 tx=0x7f75d400ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:17.658 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:17.643+0000 7f75dcbee700 1 -- 172.21.15.42:0/3118019790 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f75d400a160 con 0x7f75d80a7740 2019-12-05T06:33:17.658 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:17.647+0000 7f75df653700 1 -- 172.21.15.42:0/3118019790 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f75d80a7740 msgr2=0x7f75d808df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:17.658 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:17.647+0000 7f75df653700 1 --2- 172.21.15.42:0/3118019790 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f75d80a7740 0x7f75d808df80 secure :-1 s=READY pgs=616 cs=0 l=1 rx=0x7f75d40032b0 tx=0x7f75d400ff70).stop 2019-12-05T06:33:17.658 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:17.647+0000 7f75df653700 1 -- 172.21.15.42:0/3118019790 shutdown_connections 2019-12-05T06:33:17.658 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:17.647+0000 7f75df653700 1 --2- 172.21.15.42:0/3118019790 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f75d80a7740 0x7f75d808df80 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:17.658 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:17.647+0000 7f75df653700 1 -- 172.21.15.42:0/3118019790 shutdown_connections 2019-12-05T06:33:17.659 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:17.647+0000 7f75df653700 1 -- 172.21.15.42:0/3118019790 wait complete. 2019-12-05T06:33:17.659 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:17.647+0000 7f75df653700 1 -- 172.21.15.42:0/3118019790 >> 172.21.15.42:0/3118019790 conn(0x7f75d80aabe0 msgr2=0x7f75d80a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:33:17.664 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:17.647+0000 7f75df653700 1 Processor -- start 2019-12-05T06:33:17.664 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:17.651+0000 7f75df653700 1 -- start start 2019-12-05T06:33:17.664 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:17.651+0000 7f75df653700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f75d8126880 0x7f75d8116530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:33:17.665 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:17.651+0000 7f75df653700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f75d8142aa0 0x7f75d807a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:33:17.665 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:17.651+0000 7f75df653700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75d8143480 con 0x7f75d80a3f60 2019-12-05T06:33:17.665 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:17.651+0000 7f75df653700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75d81431e0 con 0x7f75d8126880 2019-12-05T06:33:17.665 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:17.651+0000 7f75df653700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75d80a6230 con 0x7f75d8142aa0 2019-12-05T06:33:17.665 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:17.651+0000 7f75de3f1700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f75d8142aa0 0x7f75d807a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:33:17.665 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:17.651+0000 7f75ddbf0700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f75d8126880 0x7f75d8116530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:33:17.665 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:17.651+0000 7f75de3f1700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f75d8142aa0 0x7f75d807a860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:57474/0 (socket says 172.21.15.42:57474) 2019-12-05T06:33:17.666 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:17.651+0000 7f75de3f1700 1 -- 172.21.15.42:0/1550013536 learned_addr learned my addr 172.21.15.42:0/1550013536 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:33:17.666 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:17.651+0000 7f75de3f1700 1 -- 172.21.15.42:0/1550013536 >> v2:172.21.15.40:3300/0 conn(0x7f75d8126880 msgr2=0x7f75d8116530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:17.666 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:17.651+0000 7f75de3f1700 1 --2- 172.21.15.42:0/1550013536 >> v2:172.21.15.40:3300/0 conn(0x7f75d8126880 0x7f75d8116530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:17.666 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:17.651+0000 7f75de3f1700 1 -- 172.21.15.42:0/1550013536 >> v1:172.21.15.40:6789/0 conn(0x7f75d80a3f60 legacy=0x7f75d80a64d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:17.666 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:17.651+0000 7f75de3f1700 1 -- 172.21.15.42:0/1550013536 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75d80a4390 con 0x7f75d8142aa0 2019-12-05T06:33:17.666 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:17.651+0000 7f75ddbf0700 1 --2- 172.21.15.42:0/1550013536 >> v2:172.21.15.40:3300/0 conn(0x7f75d8126880 0x7f75d8116530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:33:17.667 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:17.651+0000 7f75de3f1700 1 --2- 172.21.15.42:0/1550013536 >> v2:172.21.15.42:3300/0 conn(0x7f75d8142aa0 0x7f75d807a860 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f75d0000b80 tx=0x7f75d0010780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:17.667 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:17.651+0000 7f75de3f1700 1 --2- 172.21.15.42:0/1550013536 >> v2:172.21.15.42:3300/0 conn(0x7f75d8142aa0 0x7f75d807a860 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f75d0000b80 tx=0x7f75d0010780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:17.667 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:17.651+0000 7f75de3f1700 1 --2- 172.21.15.42:0/1550013536 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f75d8142aa0 0x7f75d807a860 secure :-1 s=READY pgs=2793 cs=0 l=1 rx=0x7f75d0000b80 tx=0x7f75d0010780).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:33:17.667 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:17.651+0000 7f75de3f1700 1 --2- 172.21.15.42:0/1550013536 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f75d8142aa0 0x7f75d807a860 secure :-1 s=THROTTLE_DONE pgs=2793 cs=0 l=1 rx=0x7f75d0000b80 tx=0x7f75d0010780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:17.667 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:17.651+0000 7f75ce7fc700 1 -- 172.21.15.42:0/1550013536 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f75d0006a40 con 0x7f75d8142aa0 2019-12-05T06:33:17.667 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:17.651+0000 7f75de3f1700 1 --2- 172.21.15.42:0/1550013536 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f75d8142aa0 0x7f75d807a860 secure :-1 s=THROTTLE_DONE pgs=2793 cs=0 l=1 rx=0x7f75d0000b80 tx=0x7f75d0010780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:17.668 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:17.651+0000 7f75de3f1700 1 --2- 172.21.15.42:0/1550013536 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f75d8142aa0 0x7f75d807a860 secure :-1 s=THROTTLE_DONE pgs=2793 cs=0 l=1 rx=0x7f75d0000b80 tx=0x7f75d0010780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:17.668 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:17.651+0000 7f75ce7fc700 1 -- 172.21.15.42:0/1550013536 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75d0006660 con 0x7f75d8142aa0 2019-12-05T06:33:17.668 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:17.651+0000 7f75df653700 1 -- 172.21.15.42:0/1550013536 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75d8142ed0 con 0x7f75d8142aa0 2019-12-05T06:33:17.668 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:17.651+0000 7f75ce7fc700 1 -- 172.21.15.42:0/1550013536 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f75d0010fc0 con 0x7f75d8142aa0 2019-12-05T06:33:17.668 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:17.651+0000 7f75df653700 1 -- 172.21.15.42:0/1550013536 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75d807afd0 con 0x7f75d8142aa0 2019-12-05T06:33:17.669 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:17.655+0000 7f75de3f1700 1 --2- 172.21.15.42:0/1550013536 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f75d8142aa0 0x7f75d807a860 secure :-1 s=THROTTLE_DONE pgs=2793 cs=0 l=1 rx=0x7f75d0000b80 tx=0x7f75d0010780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:17.669 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:17.655+0000 7f75ce7fc700 1 -- 172.21.15.42:0/1550013536 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f75d00112e0 con 0x7f75d8142aa0 2019-12-05T06:33:17.669 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:17.655+0000 7f75de3f1700 1 --2- 172.21.15.42:0/1550013536 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f75d8142aa0 0x7f75d807a860 secure :-1 s=THROTTLE_DONE pgs=2793 cs=0 l=1 rx=0x7f75d0000b80 tx=0x7f75d0010780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:17.669 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:17.655+0000 7f75ce7fc700 1 --2- 172.21.15.42:0/1550013536 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f75c003adf0 0x7f75c003d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:33:17.669 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:17.655+0000 7f75ce7fc700 1 -- 172.21.15.42:0/1550013536 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 11191+0+0 (secure 0 0 0) 0x7f75d004b110 con 0x7f75d8142aa0 2019-12-05T06:33:17.669 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:17.655+0000 7f75ddbf0700 1 --2- 172.21.15.42:0/1550013536 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f75c003adf0 0x7f75c003d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:33:17.670 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:17.655+0000 7f75ddbf0700 1 --2- 172.21.15.42:0/1550013536 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f75c003adf0 0x7f75c003d300 crc :-1 s=READY pgs=1172 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:33:17.670 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:17.655+0000 7f75df653700 1 -- 172.21.15.42:0/1550013536 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75d805d110 con 0x7f75d8142aa0 2019-12-05T06:33:17.671 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:17.663+0000 7f75de3f1700 1 --2- 172.21.15.42:0/1550013536 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f75d8142aa0 0x7f75d807a860 secure :-1 s=THROTTLE_DONE pgs=2793 cs=0 l=1 rx=0x7f75d0000b80 tx=0x7f75d0010780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:17.671 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:17.663+0000 7f75ce7fc700 1 -- 172.21.15.42:0/1550013536 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f75d80a4390 con 0x7f75d8142aa0 2019-12-05T06:33:17.841 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:17.835+0000 7f75df653700 1 -- 172.21.15.42:0/1550013536 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f75d805b8c0 con 0x7f75d8142aa0 2019-12-05T06:33:17.842 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:17.835+0000 7f75de3f1700 1 --2- 172.21.15.42:0/1550013536 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f75d8142aa0 0x7f75d807a860 secure :-1 s=THROTTLE_DONE pgs=2793 cs=0 l=1 rx=0x7f75d0000b80 tx=0x7f75d0010780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:17.843 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:17.835+0000 7f75ce7fc700 1 -- 172.21.15.42:0/1550013536 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 333 v333) v1 ==== 95+0+2117 (secure 0 0 0) 0x7f75d0032110 con 0x7f75d8142aa0 2019-12-05T06:33:17.843 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 333 2019-12-05T06:33:17.843 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:33:17.843 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":333,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":11433,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/2532474042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2532474042},{"type":"v1","addr":"172.21.15.42:6835","nonce":2532474042}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":333}],"filesystems":[{"mdsmap":{"epoch":332,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:31:20.143327+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":109,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11352},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11352":{"gid":11352,"name":"b","rank":0,"incarnation":329,"state":"up:active","state_seq":16,"addr":"172.21.15.40:6833/2055706790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2055706790},{"type":"v1","addr":"172.21.15.40:6833","nonce":2055706790}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:33:17.845 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:17.839+0000 7f75df653700 1 -- 172.21.15.42:0/1550013536 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f75c003adf0 msgr2=0x7f75c003d300 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:17.845 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:17.839+0000 7f75df653700 1 --2- 172.21.15.42:0/1550013536 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f75c003adf0 0x7f75c003d300 crc :-1 s=READY pgs=1172 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:17.845 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:17.839+0000 7f75df653700 1 -- 172.21.15.42:0/1550013536 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f75d8142aa0 msgr2=0x7f75d807a860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:17.845 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:17.839+0000 7f75df653700 1 --2- 172.21.15.42:0/1550013536 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f75d8142aa0 0x7f75d807a860 secure :-1 s=READY pgs=2793 cs=0 l=1 rx=0x7f75d0000b80 tx=0x7f75d0010780).stop 2019-12-05T06:33:17.846 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:17.839+0000 7f75df653700 1 -- 172.21.15.42:0/1550013536 shutdown_connections 2019-12-05T06:33:17.846 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:17.839+0000 7f75df653700 1 --2- 172.21.15.42:0/1550013536 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f75c003adf0 0x7f75c003d300 unknown :-1 s=CLOSED pgs=1172 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:17.846 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:17.839+0000 7f75df653700 1 --2- 172.21.15.42:0/1550013536 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f75d8142aa0 0x7f75d807a860 unknown :-1 s=CLOSED pgs=2793 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:17.846 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:17.839+0000 7f75df653700 1 --2- 172.21.15.42:0/1550013536 >> v2:172.21.15.40:3300/0 conn(0x7f75d8126880 0x7f75d8116530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:17.846 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:17.839+0000 7f75df653700 1 -- 172.21.15.42:0/1550013536 shutdown_connections 2019-12-05T06:33:17.846 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:17.839+0000 7f75df653700 1 -- 172.21.15.42:0/1550013536 wait complete. 2019-12-05T06:33:17.846 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:17.839+0000 7f75df653700 1 -- 172.21.15.42:0/1550013536 >> 172.21.15.42:0/1550013536 conn(0x7f75d814fbe0 msgr2=0x7f75d8059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:33:17.855 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:33:19.856 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:33:19.963 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:19.951+0000 7f8117b46700 1 Processor -- start 2019-12-05T06:33:19.964 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:19.951+0000 7f8117b46700 1 -- start start 2019-12-05T06:33:19.964 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:19.951+0000 7f8117b46700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f81100a7740 0x7f811008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:33:19.964 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:19.951+0000 7f8117b46700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f811009fae0 0x7f811007eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:33:19.965 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:19.951+0000 7f8117b46700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81100a0710 con 0x7f811007e720 2019-12-05T06:33:19.965 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:19.951+0000 7f8117b46700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81100a0b00 con 0x7f811009fae0 2019-12-05T06:33:19.965 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:19.951+0000 7f8117b46700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f811008e5b0 con 0x7f81100a7740 2019-12-05T06:33:19.965 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:19.951+0000 7f81158e2700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f811007e720 0x7f811009f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:57518/0 (socket says 172.21.15.42:57518) 2019-12-05T06:33:19.965 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:19.951+0000 7f81158e2700 1 -- 172.21.15.42:0/1043586133 learned_addr learned my addr 172.21.15.42:0/1043586133 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:33:19.965 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:19.951+0000 7f81168e4700 1 --2- 172.21.15.42:0/1043586133 >> v2:172.21.15.40:3300/0 conn(0x7f811009fae0 0x7f811007eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:33:19.965 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:19.951+0000 7f81160e3700 1 --2- 172.21.15.42:0/1043586133 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f81100a7740 0x7f811008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:33:19.966 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:19.951+0000 7f81168e4700 1 -- 172.21.15.42:0/1043586133 >> v2:172.21.15.40:3300/0 conn(0x7f811009fae0 msgr2=0x7f811007eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2019-12-05T06:33:19.966 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:19.951+0000 7f81168e4700 1 -- 172.21.15.42:0/1043586133 >> v2:172.21.15.40:3300/0 conn(0x7f811009fae0 msgr2=0x7f811007eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2019-12-05T06:33:19.966 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:19.951+0000 7f81168e4700 1 --2- 172.21.15.42:0/1043586133 >> v2:172.21.15.40:3300/0 conn(0x7f811009fae0 0x7f811007eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-12-05T06:33:19.966 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:19.951+0000 7f81168e4700 1 --2- 172.21.15.42:0/1043586133 >> v2:172.21.15.40:3300/0 conn(0x7f811009fae0 0x7f811007eb50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rx=0 tx=0)._fault waiting 0.200000 2019-12-05T06:33:19.966 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:19.951+0000 7f81150e1700 1 -- 172.21.15.42:0/1043586133 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3489264041 0 0) 0x7f81100a0710 con 0x7f811007e720 2019-12-05T06:33:19.966 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:19.951+0000 7f81150e1700 1 -- 172.21.15.42:0/1043586133 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80fc0030b0 con 0x7f811007e720 2019-12-05T06:33:19.967 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:19.951+0000 7f81158e2700 1 -- 172.21.15.42:0/1043586133 >> v1:172.21.15.42:6789/0 conn(0x7f811007e720 legacy=0x7f811009f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2019-12-05T06:33:19.967 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:19.951+0000 7f81158e2700 1 -- 172.21.15.42:0/1043586133 >> v1:172.21.15.42:6789/0 conn(0x7f811007e720 legacy=0x7f811009f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2019-12-05T06:33:19.967 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:19.951+0000 7f81158e2700 1 --1- 172.21.15.42:0/1043586133 >> v1:172.21.15.42:6789/0 conn(0x7f811007e720 0x7f811009f2e0 :-1 s=OPENED pgs=2794 cs=1 l=1).handle_message read tag failed 2019-12-05T06:33:19.967 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:19.951+0000 7f81158e2700 1 --1- 172.21.15.42:0/1043586133 >> v1:172.21.15.42:6789/0 conn(0x7f811007e720 0x7f811009f2e0 :-1 s=OPENED pgs=2794 cs=1 l=1).fault on lossy channel, failing 2019-12-05T06:33:19.967 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:19.951+0000 7f81160e3700 1 -- 172.21.15.42:0/1043586133 >> v2:172.21.15.40:3300/0 conn(0x7f811009fae0 msgr2=0x7f811007eb50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2019-12-05T06:33:19.967 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:19.951+0000 7f81160e3700 1 --2- 172.21.15.42:0/1043586133 >> v2:172.21.15.40:3300/0 conn(0x7f811009fae0 0x7f811007eb50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:33:19.968 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:19.955+0000 7f81160e3700 1 -- 172.21.15.42:0/1043586133 >> v1:172.21.15.42:6789/0 conn(0x7f811007e720 legacy=0x7f811009f2e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2019-12-05T06:33:19.968 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:19.955+0000 7f81160e3700 1 -- 172.21.15.42:0/1043586133 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f811007f0a0 con 0x7f81100a7740 2019-12-05T06:33:19.968 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:19.955+0000 7f81160e3700 1 --2- 172.21.15.42:0/1043586133 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f81100a7740 0x7f811008df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f810c002f30 tx=0x7f810c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:19.968 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:19.955+0000 7f81160e3700 1 --2- 172.21.15.42:0/1043586133 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f81100a7740 0x7f811008df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f810c002f30 tx=0x7f810c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:19.969 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:19.955+0000 7f81160e3700 1 --2- 172.21.15.42:0/1043586133 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f81100a7740 0x7f811008df80 secure :-1 s=READY pgs=617 cs=0 l=1 rx=0x7f810c002f30 tx=0x7f810c00fc70).ready entity=mon.2 client_cookie=cd89d93874041727 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:33:19.969 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:19.955+0000 7f81160e3700 1 --2- 172.21.15.42:0/1043586133 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f81100a7740 0x7f811008df80 secure :-1 s=THROTTLE_DONE pgs=617 cs=0 l=1 rx=0x7f810c002f30 tx=0x7f810c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:19.969 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:19.955+0000 7f81150e1700 1 -- 172.21.15.42:0/1043586133 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f810c008bb0 con 0x7f81100a7740 2019-12-05T06:33:19.969 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:19.955+0000 7f81160e3700 1 --2- 172.21.15.42:0/1043586133 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f81100a7740 0x7f811008df80 secure :-1 s=THROTTLE_DONE pgs=617 cs=0 l=1 rx=0x7f810c002f30 tx=0x7f810c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:19.969 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:19.955+0000 7f81160e3700 1 --2- 172.21.15.42:0/1043586133 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f81100a7740 0x7f811008df80 secure :-1 s=THROTTLE_DONE pgs=617 cs=0 l=1 rx=0x7f810c002f30 tx=0x7f810c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:19.970 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:19.955+0000 7f81150e1700 1 -- 172.21.15.42:0/1043586133 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f810c0087d0 con 0x7f81100a7740 2019-12-05T06:33:19.970 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:19.955+0000 7f81150e1700 1 -- 172.21.15.42:0/1043586133 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f810c011820 con 0x7f81100a7740 2019-12-05T06:33:19.970 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:19.955+0000 7f8117b46700 1 -- 172.21.15.42:0/1043586133 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f81100a7740 msgr2=0x7f811008df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:19.970 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:19.955+0000 7f8117b46700 1 --2- 172.21.15.42:0/1043586133 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f81100a7740 0x7f811008df80 secure :-1 s=READY pgs=617 cs=0 l=1 rx=0x7f810c002f30 tx=0x7f810c00fc70).stop 2019-12-05T06:33:19.970 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:19.955+0000 7f8117b46700 1 -- 172.21.15.42:0/1043586133 shutdown_connections 2019-12-05T06:33:19.970 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:19.955+0000 7f8117b46700 1 --2- 172.21.15.42:0/1043586133 >> v2:172.21.15.40:3300/0 conn(0x7f811009fae0 0x7f811007eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:33:19.971 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:19.955+0000 7f8117b46700 1 --2- 172.21.15.42:0/1043586133 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f81100a7740 0x7f811008df80 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:19.971 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:19.955+0000 7f8117b46700 1 -- 172.21.15.42:0/1043586133 shutdown_connections 2019-12-05T06:33:19.971 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:19.955+0000 7f8117b46700 1 -- 172.21.15.42:0/1043586133 wait complete. 2019-12-05T06:33:19.971 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:19.955+0000 7f8117b46700 1 -- 172.21.15.42:0/1043586133 >> 172.21.15.42:0/1043586133 conn(0x7f81100aabe0 msgr2=0x7f81100a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:33:19.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:19.959+0000 7f8117b46700 1 Processor -- start 2019-12-05T06:33:19.976 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:19.959+0000 7f8117b46700 1 -- start start 2019-12-05T06:33:19.976 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:19.959+0000 7f8117b46700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f81101186a0 0x7f811011f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:33:19.976 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:19.959+0000 7f8117b46700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8110120750 0x7f8110120b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:33:19.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:19.959+0000 7f8117b46700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f811012d3d0 con 0x7f811012fa70 2019-12-05T06:33:19.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:19.959+0000 7f8117b46700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f811011e7e0 con 0x7f81101186a0 2019-12-05T06:33:19.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:19.959+0000 7f8117b46700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f811011fac0 con 0x7f8110120750 2019-12-05T06:33:19.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:19.959+0000 7f81160e3700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f81101186a0 0x7f811011f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:33:19.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:19.959+0000 7f81158e2700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8110120750 0x7f8110120b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:33:19.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:19.959+0000 7f81168e4700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f811012fa70 0x7f811012fea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:38734/0 (socket says 172.21.15.42:38734) 2019-12-05T06:33:19.978 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:19.959+0000 7f81168e4700 1 -- 172.21.15.42:0/994898894 learned_addr learned my addr 172.21.15.42:0/994898894 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:33:19.978 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:19.959+0000 7f81158e2700 1 -- 172.21.15.42:0/994898894 >> v2:172.21.15.40:3300/0 conn(0x7f81101186a0 msgr2=0x7f811011f1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:19.978 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:19.959+0000 7f81158e2700 1 --2- 172.21.15.42:0/994898894 >> v2:172.21.15.40:3300/0 conn(0x7f81101186a0 0x7f811011f1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:19.978 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:19.959+0000 7f81158e2700 1 -- 172.21.15.42:0/994898894 >> v1:172.21.15.40:6789/0 conn(0x7f811012fa70 legacy=0x7f811012fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:19.978 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:19.959+0000 7f81158e2700 1 -- 172.21.15.42:0/994898894 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81101210d0 con 0x7f8110120750 2019-12-05T06:33:19.978 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:19.959+0000 7f8102ffd700 1 -- 172.21.15.42:0/994898894 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1077413098 0 0) 0x7f811012d3d0 con 0x7f811012fa70 2019-12-05T06:33:19.979 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:19.959+0000 7f81160e3700 1 --2- 172.21.15.42:0/994898894 >> v2:172.21.15.40:3300/0 conn(0x7f81101186a0 0x7f811011f1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:33:19.979 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:19.959+0000 7f81158e2700 1 --2- 172.21.15.42:0/994898894 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8110120750 0x7f8110120b80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f811011e450 tx=0x7f8104010920).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:19.980 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:19.959+0000 7f81158e2700 1 --2- 172.21.15.42:0/994898894 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8110120750 0x7f8110120b80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f811011e450 tx=0x7f8104010920).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:19.980 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:19.959+0000 7f81158e2700 1 --2- 172.21.15.42:0/994898894 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8110120750 0x7f8110120b80 secure :-1 s=READY pgs=618 cs=0 l=1 rx=0x7f811011e450 tx=0x7f8104010920).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:33:19.980 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:19.963+0000 7f81158e2700 1 --2- 172.21.15.42:0/994898894 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8110120750 0x7f8110120b80 secure :-1 s=THROTTLE_DONE pgs=618 cs=0 l=1 rx=0x7f811011e450 tx=0x7f8104010920).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:19.980 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:19.963+0000 7f81158e2700 1 --2- 172.21.15.42:0/994898894 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8110120750 0x7f8110120b80 secure :-1 s=THROTTLE_DONE pgs=618 cs=0 l=1 rx=0x7f811011e450 tx=0x7f8104010920).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:19.980 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:19.963+0000 7f8102ffd700 1 -- 172.21.15.42:0/994898894 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8104005b40 con 0x7f8110120750 2019-12-05T06:33:19.980 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:19.963+0000 7f81158e2700 1 --2- 172.21.15.42:0/994898894 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8110120750 0x7f8110120b80 secure :-1 s=THROTTLE_DONE pgs=618 cs=0 l=1 rx=0x7f811011e450 tx=0x7f8104010920).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:19.981 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:19.963+0000 7f8102ffd700 1 -- 172.21.15.42:0/994898894 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8104005760 con 0x7f8110120750 2019-12-05T06:33:19.981 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:19.963+0000 7f8102ffd700 1 -- 172.21.15.42:0/994898894 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f81040062b0 con 0x7f8110120750 2019-12-05T06:33:19.981 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:19.963+0000 7f8117b46700 1 -- 172.21.15.42:0/994898894 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f811011f700 con 0x7f8110120750 2019-12-05T06:33:19.981 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:19.963+0000 7f8117b46700 1 -- 172.21.15.42:0/994898894 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8110118ad0 con 0x7f8110120750 2019-12-05T06:33:19.981 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:19.963+0000 7f81158e2700 1 --2- 172.21.15.42:0/994898894 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8110120750 0x7f8110120b80 secure :-1 s=THROTTLE_DONE pgs=618 cs=0 l=1 rx=0x7f811011e450 tx=0x7f8104010920).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:19.981 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:19.963+0000 7f8102ffd700 1 -- 172.21.15.42:0/994898894 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f81040065c0 con 0x7f8110120750 2019-12-05T06:33:19.981 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:19.963+0000 7f81158e2700 1 --2- 172.21.15.42:0/994898894 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8110120750 0x7f8110120b80 secure :-1 s=THROTTLE_DONE pgs=618 cs=0 l=1 rx=0x7f811011e450 tx=0x7f8104010920).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:19.984 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:19.963+0000 7f8102ffd700 1 --2- 172.21.15.42:0/994898894 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f80f803ad10 0x7f80f803d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:33:19.985 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:19.963+0000 7f8102ffd700 1 -- 172.21.15.42:0/994898894 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 11191+0+0 (secure 0 0 0) 0x7f810404af60 con 0x7f8110120750 2019-12-05T06:33:19.985 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:19.963+0000 7f81160e3700 1 --2- 172.21.15.42:0/994898894 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f80f803ad10 0x7f80f803d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:33:19.985 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:19.963+0000 7f81160e3700 1 --2- 172.21.15.42:0/994898894 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f80f803ad10 0x7f80f803d220 crc :-1 s=READY pgs=1173 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:33:19.985 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:19.967+0000 7f8117b46700 1 -- 172.21.15.42:0/994898894 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f811005d110 con 0x7f8110120750 2019-12-05T06:33:19.986 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:19.975+0000 7f81158e2700 1 --2- 172.21.15.42:0/994898894 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8110120750 0x7f8110120b80 secure :-1 s=THROTTLE_DONE pgs=618 cs=0 l=1 rx=0x7f811011e450 tx=0x7f8104010920).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:19.987 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:19.975+0000 7f8102ffd700 1 -- 172.21.15.42:0/994898894 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f8104031fe0 con 0x7f8110120750 2019-12-05T06:33:20.155 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:20.147+0000 7f8117b46700 1 -- 172.21.15.42:0/994898894 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f811005b8c0 con 0x7f8110120750 2019-12-05T06:33:20.156 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:20.151+0000 7f81158e2700 1 --2- 172.21.15.42:0/994898894 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8110120750 0x7f8110120b80 secure :-1 s=THROTTLE_DONE pgs=618 cs=0 l=1 rx=0x7f811011e450 tx=0x7f8104010920).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:20.157 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:20.151+0000 7f8102ffd700 1 -- 172.21.15.42:0/994898894 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 333 v333) v1 ==== 95+0+2117 (secure 0 0 0) 0x7f8104032240 con 0x7f8110120750 2019-12-05T06:33:20.158 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:33:20.159 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":333,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":11433,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/2532474042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2532474042},{"type":"v1","addr":"172.21.15.42:6835","nonce":2532474042}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":333}],"filesystems":[{"mdsmap":{"epoch":332,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:31:20.143327+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":109,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11352},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11352":{"gid":11352,"name":"b","rank":0,"incarnation":329,"state":"up:active","state_seq":16,"addr":"172.21.15.40:6833/2055706790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2055706790},{"type":"v1","addr":"172.21.15.40:6833","nonce":2055706790}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:33:20.159 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 333 2019-12-05T06:33:20.164 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:20.155+0000 7f8117b46700 1 -- 172.21.15.42:0/994898894 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f80f803ad10 msgr2=0x7f80f803d220 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:20.165 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:20.155+0000 7f8117b46700 1 --2- 172.21.15.42:0/994898894 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f80f803ad10 0x7f80f803d220 crc :-1 s=READY pgs=1173 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:20.165 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:20.155+0000 7f8117b46700 1 -- 172.21.15.42:0/994898894 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8110120750 msgr2=0x7f8110120b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:20.165 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:20.155+0000 7f8117b46700 1 --2- 172.21.15.42:0/994898894 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8110120750 0x7f8110120b80 secure :-1 s=READY pgs=618 cs=0 l=1 rx=0x7f811011e450 tx=0x7f8104010920).stop 2019-12-05T06:33:20.166 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:20.155+0000 7f8117b46700 1 -- 172.21.15.42:0/994898894 shutdown_connections 2019-12-05T06:33:20.166 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:20.155+0000 7f8117b46700 1 --2- 172.21.15.42:0/994898894 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f80f803ad10 0x7f80f803d220 unknown :-1 s=CLOSED pgs=1173 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:20.166 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:20.155+0000 7f8117b46700 1 --2- 172.21.15.42:0/994898894 >> v2:172.21.15.40:3300/0 conn(0x7f81101186a0 0x7f811011f1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:20.166 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:20.155+0000 7f8117b46700 1 --2- 172.21.15.42:0/994898894 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8110120750 0x7f8110120b80 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:20.166 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:20.155+0000 7f8117b46700 1 -- 172.21.15.42:0/994898894 shutdown_connections 2019-12-05T06:33:20.166 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:20.155+0000 7f8117b46700 1 -- 172.21.15.42:0/994898894 wait complete. 2019-12-05T06:33:20.167 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:20.155+0000 7f8117b46700 1 -- 172.21.15.42:0/994898894 >> 172.21.15.42:0/994898894 conn(0x7f81100a7740 msgr2=0x7f811008eb70 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:33:20.174 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:33:22.175 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:33:22.284 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:22.271+0000 7f01b59c7700 1 Processor -- start 2019-12-05T06:33:22.284 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:22.271+0000 7f01b59c7700 1 -- start start 2019-12-05T06:33:22.284 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:22.271+0000 7f01b59c7700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f01b00a7740 0x7f01b008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:33:22.285 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:22.271+0000 7f01b59c7700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01b00a0710 con 0x7f01b007e720 2019-12-05T06:33:22.285 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:22.271+0000 7f01b59c7700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01b00a9ba0 con 0x7f01b009fae0 2019-12-05T06:33:22.285 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:22.271+0000 7f01b59c7700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01b00a0b00 con 0x7f01b00a7740 2019-12-05T06:33:22.285 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:22.271+0000 7f01affff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f01b009fae0 0x7f01b014efe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:57536/0 (socket says 172.21.15.42:57536) 2019-12-05T06:33:22.285 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:22.271+0000 7f01affff700 1 -- 172.21.15.42:0/2171741638 learned_addr learned my addr 172.21.15.42:0/2171741638 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:33:22.285 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:22.271+0000 7f01af7fe700 1 --2- 172.21.15.42:0/2171741638 >> v2:172.21.15.40:3300/0 conn(0x7f01b00a7740 0x7f01b008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:33:22.285 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:22.275+0000 7f01ae7fc700 1 -- 172.21.15.42:0/2171741638 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4169155812 0 0) 0x7f01b00a9ba0 con 0x7f01b009fae0 2019-12-05T06:33:22.286 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:22.275+0000 7f01ae7fc700 1 -- 172.21.15.42:0/2171741638 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f019c0030b0 con 0x7f01b009fae0 2019-12-05T06:33:22.286 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:22.275+0000 7f01af7fe700 1 -- 172.21.15.42:0/2171741638 >> v1:172.21.15.42:6789/0 conn(0x7f01b009fae0 legacy=0x7f01b014efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:22.286 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:22.275+0000 7f01ae7fc700 1 -- 172.21.15.42:0/2171741638 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 232464634 0 0) 0x7f01b00a0710 con 0x7f01b007e720 2019-12-05T06:33:22.286 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:22.275+0000 7f01af7fe700 1 -- 172.21.15.42:0/2171741638 >> v1:172.21.15.40:6789/0 conn(0x7f01b007e720 legacy=0x7f01b009f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:22.286 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:22.275+0000 7f01af7fe700 1 -- 172.21.15.42:0/2171741638 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01b007eb50 con 0x7f01b00a7740 2019-12-05T06:33:22.286 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:22.275+0000 7f01af7fe700 1 --2- 172.21.15.42:0/2171741638 >> v2:172.21.15.40:3300/0 conn(0x7f01b00a7740 0x7f01b008df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f0198008790 tx=0x7f019800fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:22.286 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:22.275+0000 7f01af7fe700 1 --2- 172.21.15.42:0/2171741638 >> v2:172.21.15.40:3300/0 conn(0x7f01b00a7740 0x7f01b008df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f0198008790 tx=0x7f019800fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:22.287 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:22.275+0000 7f01af7fe700 1 --2- 172.21.15.42:0/2171741638 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f01b00a7740 0x7f01b008df80 secure :-1 s=READY pgs=2110 cs=0 l=1 rx=0x7f0198008790 tx=0x7f019800fbf0).ready entity=mon.0 client_cookie=53cd4fdcc9388971 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:33:22.287 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:22.275+0000 7f01af7fe700 1 --2- 172.21.15.42:0/2171741638 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f01b00a7740 0x7f01b008df80 secure :-1 s=THROTTLE_DONE pgs=2110 cs=0 l=1 rx=0x7f0198008790 tx=0x7f019800fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:22.287 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:22.275+0000 7f01ae7fc700 1 -- 172.21.15.42:0/2171741638 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0198011270 con 0x7f01b00a7740 2019-12-05T06:33:22.287 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:22.275+0000 7f01af7fe700 1 --2- 172.21.15.42:0/2171741638 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f01b00a7740 0x7f01b008df80 secure :-1 s=THROTTLE_DONE pgs=2110 cs=0 l=1 rx=0x7f0198008790 tx=0x7f019800fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:22.287 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:22.275+0000 7f01af7fe700 1 --2- 172.21.15.42:0/2171741638 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f01b00a7740 0x7f01b008df80 secure :-1 s=THROTTLE_DONE pgs=2110 cs=0 l=1 rx=0x7f0198008790 tx=0x7f019800fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:22.287 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:22.275+0000 7f01ae7fc700 1 -- 172.21.15.42:0/2171741638 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0198008bc0 con 0x7f01b00a7740 2019-12-05T06:33:22.287 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:22.275+0000 7f01ae7fc700 1 -- 172.21.15.42:0/2171741638 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0198011850 con 0x7f01b00a7740 2019-12-05T06:33:22.288 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:22.275+0000 7f01b59c7700 1 -- 172.21.15.42:0/2171741638 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f01b00a7740 msgr2=0x7f01b008df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:22.288 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:22.275+0000 7f01b59c7700 1 --2- 172.21.15.42:0/2171741638 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f01b00a7740 0x7f01b008df80 secure :-1 s=READY pgs=2110 cs=0 l=1 rx=0x7f0198008790 tx=0x7f019800fbf0).stop 2019-12-05T06:33:22.288 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:22.275+0000 7f01b59c7700 1 -- 172.21.15.42:0/2171741638 shutdown_connections 2019-12-05T06:33:22.288 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:22.275+0000 7f01b59c7700 1 --2- 172.21.15.42:0/2171741638 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f01b00a7740 0x7f01b008df80 unknown :-1 s=CLOSED pgs=2110 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:22.288 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:22.275+0000 7f01b59c7700 1 -- 172.21.15.42:0/2171741638 shutdown_connections 2019-12-05T06:33:22.288 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:22.275+0000 7f01b59c7700 1 -- 172.21.15.42:0/2171741638 wait complete. 2019-12-05T06:33:22.288 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:22.275+0000 7f01b59c7700 1 -- 172.21.15.42:0/2171741638 >> 172.21.15.42:0/2171741638 conn(0x7f01b00aabe0 msgr2=0x7f01b00a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:33:22.292 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:22.279+0000 7f01b59c7700 1 Processor -- start 2019-12-05T06:33:22.292 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:22.279+0000 7f01b59c7700 1 -- start start 2019-12-05T06:33:22.292 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:22.279+0000 7f01b59c7700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f01b0122780 0x7f01b0124480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:33:22.293 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:22.279+0000 7f01b59c7700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f01b01173e0 0x7f01b007d840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:33:22.293 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:22.279+0000 7f01b59c7700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01b0122200 con 0x7f01b0120fc0 2019-12-05T06:33:22.293 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:22.279+0000 7f01b59c7700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01b0122da0 con 0x7f01b0122780 2019-12-05T06:33:22.293 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:22.279+0000 7f01b59c7700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01b0123040 con 0x7f01b01173e0 2019-12-05T06:33:22.293 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:22.279+0000 7f01affff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f01b0120fc0 0x7f01b0118f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:57542/0 (socket says 172.21.15.42:57542) 2019-12-05T06:33:22.294 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:22.279+0000 7f01affff700 1 -- 172.21.15.42:0/3948241578 learned_addr learned my addr 172.21.15.42:0/3948241578 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:33:22.295 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:22.279+0000 7f01af7fe700 1 --2- 172.21.15.42:0/3948241578 >> v2:172.21.15.40:3300/0 conn(0x7f01b0122780 0x7f01b0124480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:33:22.295 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:22.279+0000 7f01aeffd700 1 --2- 172.21.15.42:0/3948241578 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f01b01173e0 0x7f01b007d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:33:22.295 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:22.279+0000 7f01a7fff700 1 -- 172.21.15.42:0/3948241578 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3736535884 0 0) 0x7f01b0122200 con 0x7f01b0120fc0 2019-12-05T06:33:22.295 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:22.279+0000 7f01a7fff700 1 -- 172.21.15.42:0/3948241578 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01940030b0 con 0x7f01b0120fc0 2019-12-05T06:33:22.295 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:22.279+0000 7f01af7fe700 1 -- 172.21.15.42:0/3948241578 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f01b01173e0 msgr2=0x7f01b007d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:22.295 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:22.279+0000 7f01af7fe700 1 --2- 172.21.15.42:0/3948241578 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f01b01173e0 0x7f01b007d840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:22.295 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:22.283+0000 7f01af7fe700 1 -- 172.21.15.42:0/3948241578 >> v1:172.21.15.42:6789/0 conn(0x7f01b0120fc0 legacy=0x7f01b0118f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:22.296 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:22.283+0000 7f01af7fe700 1 -- 172.21.15.42:0/3948241578 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01b007dd90 con 0x7f01b0122780 2019-12-05T06:33:22.296 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:22.283+0000 7f01af7fe700 1 --2- 172.21.15.42:0/3948241578 >> v2:172.21.15.40:3300/0 conn(0x7f01b0122780 0x7f01b0124480 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f01b0122f60 tx=0x7f0198008e70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:22.296 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:22.283+0000 7f01af7fe700 1 --2- 172.21.15.42:0/3948241578 >> v2:172.21.15.40:3300/0 conn(0x7f01b0122780 0x7f01b0124480 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f01b0122f60 tx=0x7f0198008e70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:22.296 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:22.283+0000 7f01af7fe700 1 --2- 172.21.15.42:0/3948241578 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f01b0122780 0x7f01b0124480 secure :-1 s=READY pgs=2111 cs=0 l=1 rx=0x7f01b0122f60 tx=0x7f0198008e70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:33:22.296 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:22.283+0000 7f01af7fe700 1 --2- 172.21.15.42:0/3948241578 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f01b0122780 0x7f01b0124480 secure :-1 s=THROTTLE_DONE pgs=2111 cs=0 l=1 rx=0x7f01b0122f60 tx=0x7f0198008e70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:22.297 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:22.283+0000 7f01a7fff700 1 -- 172.21.15.42:0/3948241578 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0198007340 con 0x7f01b0122780 2019-12-05T06:33:22.297 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:22.283+0000 7f01af7fe700 1 --2- 172.21.15.42:0/3948241578 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f01b0122780 0x7f01b0124480 secure :-1 s=THROTTLE_DONE pgs=2111 cs=0 l=1 rx=0x7f01b0122f60 tx=0x7f0198008e70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:22.299 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:22.283+0000 7f01a7fff700 1 -- 172.21.15.42:0/3948241578 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0198011470 con 0x7f01b0122780 2019-12-05T06:33:22.299 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:22.283+0000 7f01af7fe700 1 --2- 172.21.15.42:0/3948241578 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f01b0122780 0x7f01b0124480 secure :-1 s=THROTTLE_DONE pgs=2111 cs=0 l=1 rx=0x7f01b0122f60 tx=0x7f0198008e70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:22.299 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:22.283+0000 7f01b59c7700 1 -- 172.21.15.42:0/3948241578 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01b01213f0 con 0x7f01b0122780 2019-12-05T06:33:22.299 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:22.283+0000 7f01a7fff700 1 -- 172.21.15.42:0/3948241578 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0198005540 con 0x7f01b0122780 2019-12-05T06:33:22.299 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:22.283+0000 7f01b59c7700 1 -- 172.21.15.42:0/3948241578 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01b0121920 con 0x7f01b0122780 2019-12-05T06:33:22.300 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:22.283+0000 7f01af7fe700 1 --2- 172.21.15.42:0/3948241578 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f01b0122780 0x7f01b0124480 secure :-1 s=THROTTLE_DONE pgs=2111 cs=0 l=1 rx=0x7f01b0122f60 tx=0x7f0198008e70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:22.300 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:22.283+0000 7f01a7fff700 1 -- 172.21.15.42:0/3948241578 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f0198005820 con 0x7f01b0122780 2019-12-05T06:33:22.300 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:22.283+0000 7f01af7fe700 1 --2- 172.21.15.42:0/3948241578 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f01b0122780 0x7f01b0124480 secure :-1 s=THROTTLE_DONE pgs=2111 cs=0 l=1 rx=0x7f01b0122f60 tx=0x7f0198008e70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:22.300 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:22.283+0000 7f01a7fff700 1 --2- 172.21.15.42:0/3948241578 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f019403b270 0x7f019403d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:33:22.300 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:22.283+0000 7f01a7fff700 1 -- 172.21.15.42:0/3948241578 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 11191+0+0 (secure 0 0 0) 0x7f019804bb20 con 0x7f01b0122780 2019-12-05T06:33:22.300 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:22.283+0000 7f01aeffd700 1 --2- 172.21.15.42:0/3948241578 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f019403b270 0x7f019403d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:33:22.300 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:22.287+0000 7f01aeffd700 1 --2- 172.21.15.42:0/3948241578 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f019403b270 0x7f019403d780 crc :-1 s=READY pgs=1174 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:33:22.301 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:22.287+0000 7f01b59c7700 1 -- 172.21.15.42:0/3948241578 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01b0123200 con 0x7f01b0122780 2019-12-05T06:33:22.303 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:22.295+0000 7f01af7fe700 1 --2- 172.21.15.42:0/3948241578 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f01b0122780 0x7f01b0124480 secure :-1 s=THROTTLE_DONE pgs=2111 cs=0 l=1 rx=0x7f01b0122f60 tx=0x7f0198008e70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:22.304 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:22.295+0000 7f01a7fff700 1 -- 172.21.15.42:0/3948241578 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f0198032b80 con 0x7f01b0122780 2019-12-05T06:33:22.471 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:22.463+0000 7f01b59c7700 1 -- 172.21.15.42:0/3948241578 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f01b005d110 con 0x7f01b0122780 2019-12-05T06:33:22.472 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:22.467+0000 7f01af7fe700 1 --2- 172.21.15.42:0/3948241578 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f01b0122780 0x7f01b0124480 secure :-1 s=THROTTLE_DONE pgs=2111 cs=0 l=1 rx=0x7f01b0122f60 tx=0x7f0198008e70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:22.473 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:22.467+0000 7f01a7fff700 1 -- 172.21.15.42:0/3948241578 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 333 v333) v1 ==== 95+0+2117 (secure 0 0 0) 0x7f0198032de0 con 0x7f01b0122780 2019-12-05T06:33:22.475 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 333 2019-12-05T06:33:22.475 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:33:22.475 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":333,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":11433,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/2532474042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2532474042},{"type":"v1","addr":"172.21.15.42:6835","nonce":2532474042}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":333}],"filesystems":[{"mdsmap":{"epoch":332,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:31:20.143327+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":109,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11352},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11352":{"gid":11352,"name":"b","rank":0,"incarnation":329,"state":"up:active","state_seq":16,"addr":"172.21.15.40:6833/2055706790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2055706790},{"type":"v1","addr":"172.21.15.40:6833","nonce":2055706790}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:33:22.478 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:22.471+0000 7f01b59c7700 1 -- 172.21.15.42:0/3948241578 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f019403b270 msgr2=0x7f019403d780 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:22.479 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:22.471+0000 7f01b59c7700 1 --2- 172.21.15.42:0/3948241578 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f019403b270 0x7f019403d780 crc :-1 s=READY pgs=1174 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:22.479 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:22.471+0000 7f01b59c7700 1 -- 172.21.15.42:0/3948241578 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f01b0122780 msgr2=0x7f01b0124480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:22.479 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:22.471+0000 7f01b59c7700 1 --2- 172.21.15.42:0/3948241578 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f01b0122780 0x7f01b0124480 secure :-1 s=READY pgs=2111 cs=0 l=1 rx=0x7f01b0122f60 tx=0x7f0198008e70).stop 2019-12-05T06:33:22.479 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:22.471+0000 7f01b59c7700 1 -- 172.21.15.42:0/3948241578 shutdown_connections 2019-12-05T06:33:22.479 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:22.471+0000 7f01b59c7700 1 --2- 172.21.15.42:0/3948241578 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f019403b270 0x7f019403d780 unknown :-1 s=CLOSED pgs=1174 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:22.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:22.471+0000 7f01b59c7700 1 --2- 172.21.15.42:0/3948241578 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f01b0122780 0x7f01b0124480 unknown :-1 s=CLOSED pgs=2111 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:22.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:22.471+0000 7f01b59c7700 1 --2- 172.21.15.42:0/3948241578 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f01b01173e0 0x7f01b007d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:22.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:22.471+0000 7f01b59c7700 1 -- 172.21.15.42:0/3948241578 shutdown_connections 2019-12-05T06:33:22.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:22.471+0000 7f01b59c7700 1 -- 172.21.15.42:0/3948241578 wait complete. 2019-12-05T06:33:22.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:22.471+0000 7f01b59c7700 1 -- 172.21.15.42:0/3948241578 >> 172.21.15.42:0/3948241578 conn(0x7f01b007e670 msgr2=0x7f01b011d960 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:33:22.494 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:33:24.495 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:33:24.599 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:24.587+0000 7f30e8bfd700 1 Processor -- start 2019-12-05T06:33:24.599 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:24.587+0000 7f30e8bfd700 1 -- start start 2019-12-05T06:33:24.600 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:24.587+0000 7f30e8bfd700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f30e40a7740 0x7f30e408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:33:24.600 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:24.587+0000 7f30e8bfd700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f30e409fae0 0x7f30e407eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:33:24.600 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:24.587+0000 7f30e8bfd700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30e40a0710 con 0x7f30e407e720 2019-12-05T06:33:24.600 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:24.587+0000 7f30e8bfd700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30e40a0b00 con 0x7f30e40a7740 2019-12-05T06:33:24.600 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:24.587+0000 7f30e8bfd700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30e408e5b0 con 0x7f30e409fae0 2019-12-05T06:33:24.600 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:24.587+0000 7f30e359e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f30e409fae0 0x7f30e407eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:33:24.601 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:24.587+0000 7f30e359e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f30e409fae0 0x7f30e407eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:57512/0 (socket says 172.21.15.42:57512) 2019-12-05T06:33:24.601 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:24.587+0000 7f30e359e700 1 -- 172.21.15.42:0/3243621108 learned_addr learned my addr 172.21.15.42:0/3243621108 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:33:24.601 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:24.587+0000 7f30e2d9d700 1 --2- 172.21.15.42:0/3243621108 >> v2:172.21.15.40:3300/0 conn(0x7f30e40a7740 0x7f30e408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:33:24.601 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:24.591+0000 7f30e359e700 1 -- 172.21.15.42:0/3243621108 >> v2:172.21.15.40:3300/0 conn(0x7f30e40a7740 msgr2=0x7f30e408df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:33:24.601 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:24.591+0000 7f30e359e700 1 --2- 172.21.15.42:0/3243621108 >> v2:172.21.15.40:3300/0 conn(0x7f30e40a7740 0x7f30e408df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:33:24.601 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:24.591+0000 7f30e359e700 1 -- 172.21.15.42:0/3243621108 >> v1:172.21.15.40:6789/0 conn(0x7f30e407e720 legacy=0x7f30e409f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:24.601 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:24.591+0000 7f30e359e700 1 -- 172.21.15.42:0/3243621108 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30e407f0a0 con 0x7f30e409fae0 2019-12-05T06:33:24.602 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:24.591+0000 7f30e359e700 1 --2- 172.21.15.42:0/3243621108 >> v2:172.21.15.42:3300/0 conn(0x7f30e409fae0 0x7f30e407eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f30dc0032b0 tx=0x7f30dc0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:24.602 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:24.591+0000 7f30e359e700 1 --2- 172.21.15.42:0/3243621108 >> v2:172.21.15.42:3300/0 conn(0x7f30e409fae0 0x7f30e407eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f30dc0032b0 tx=0x7f30dc0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:24.602 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:24.591+0000 7f30e359e700 1 --2- 172.21.15.42:0/3243621108 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f30e409fae0 0x7f30e407eb50 secure :-1 s=READY pgs=2797 cs=0 l=1 rx=0x7f30dc0032b0 tx=0x7f30dc0100b0).ready entity=mon.1 client_cookie=a67a38baaa5e8d5d server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:33:24.602 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:24.591+0000 7f30e359e700 1 --2- 172.21.15.42:0/3243621108 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f30e409fae0 0x7f30e407eb50 secure :-1 s=THROTTLE_DONE pgs=2797 cs=0 l=1 rx=0x7f30dc0032b0 tx=0x7f30dc0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:24.602 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:24.591+0000 7f30e359e700 1 --2- 172.21.15.42:0/3243621108 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f30e409fae0 0x7f30e407eb50 secure :-1 s=THROTTLE_DONE pgs=2797 cs=0 l=1 rx=0x7f30dc0032b0 tx=0x7f30dc0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:24.602 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:24.591+0000 7f30e359e700 1 --2- 172.21.15.42:0/3243621108 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f30e409fae0 0x7f30e407eb50 secure :-1 s=THROTTLE_DONE pgs=2797 cs=0 l=1 rx=0x7f30dc0032b0 tx=0x7f30dc0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:24.602 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:24.591+0000 7f30e259c700 1 -- 172.21.15.42:0/3243621108 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f30dc010d60 con 0x7f30e409fae0 2019-12-05T06:33:24.602 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:24.591+0000 7f30e259c700 1 -- 172.21.15.42:0/3243621108 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30dc010980 con 0x7f30e409fae0 2019-12-05T06:33:24.603 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:24.591+0000 7f30e259c700 1 -- 172.21.15.42:0/3243621108 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f30dc0114a0 con 0x7f30e409fae0 2019-12-05T06:33:24.603 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:24.591+0000 7f30e8bfd700 1 -- 172.21.15.42:0/3243621108 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f30e409fae0 msgr2=0x7f30e407eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:24.603 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:24.591+0000 7f30e8bfd700 1 --2- 172.21.15.42:0/3243621108 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f30e409fae0 0x7f30e407eb50 secure :-1 s=READY pgs=2797 cs=0 l=1 rx=0x7f30dc0032b0 tx=0x7f30dc0100b0).stop 2019-12-05T06:33:24.603 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:24.591+0000 7f30e8bfd700 1 -- 172.21.15.42:0/3243621108 shutdown_connections 2019-12-05T06:33:24.603 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:24.591+0000 7f30e8bfd700 1 --2- 172.21.15.42:0/3243621108 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f30e409fae0 0x7f30e407eb50 unknown :-1 s=CLOSED pgs=2797 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:24.603 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:24.591+0000 7f30e8bfd700 1 --2- 172.21.15.42:0/3243621108 >> v2:172.21.15.40:3300/0 conn(0x7f30e40a7740 0x7f30e408df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:33:24.603 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:24.591+0000 7f30e8bfd700 1 -- 172.21.15.42:0/3243621108 shutdown_connections 2019-12-05T06:33:24.604 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:24.591+0000 7f30e8bfd700 1 -- 172.21.15.42:0/3243621108 wait complete. 2019-12-05T06:33:24.604 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:24.591+0000 7f30e8bfd700 1 -- 172.21.15.42:0/3243621108 >> 172.21.15.42:0/3243621108 conn(0x7f30e40aabe0 msgr2=0x7f30e40a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:33:24.609 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:24.595+0000 7f30e8bfd700 1 Processor -- start 2019-12-05T06:33:24.609 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:24.595+0000 7f30e8bfd700 1 -- start start 2019-12-05T06:33:24.610 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:24.595+0000 7f30e8bfd700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f30e4063bc0 0x7f30e408ee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:33:24.610 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:24.595+0000 7f30e8bfd700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30e407c010 con 0x7f30e41140a0 2019-12-05T06:33:24.610 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:24.595+0000 7f30e8bfd700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30e4065300 con 0x7f30e4064ed0 2019-12-05T06:33:24.610 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:24.595+0000 7f30e8bfd700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30e4065530 con 0x7f30e4063bc0 2019-12-05T06:33:24.610 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:24.595+0000 7f30da59c700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f30e4064ed0 0x7f30e407b810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:57554/0 (socket says 172.21.15.42:57554) 2019-12-05T06:33:24.611 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:24.595+0000 7f30da59c700 1 -- 172.21.15.42:0/2886450152 learned_addr learned my addr 172.21.15.42:0/2886450152 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:33:24.611 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:24.595+0000 7f30e2d9d700 1 --2- 172.21.15.42:0/2886450152 >> v2:172.21.15.40:3300/0 conn(0x7f30e4063bc0 0x7f30e408ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:33:24.611 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:24.595+0000 7f30dbfff700 1 -- 172.21.15.42:0/2886450152 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 592358572 0 0) 0x7f30e4065300 con 0x7f30e4064ed0 2019-12-05T06:33:24.611 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:24.595+0000 7f30dbfff700 1 -- 172.21.15.42:0/2886450152 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30c80030b0 con 0x7f30e4064ed0 2019-12-05T06:33:24.611 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:24.599+0000 7f30dbfff700 1 -- 172.21.15.42:0/2886450152 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 413417639 0 0) 0x7f30c80030b0 con 0x7f30e4064ed0 2019-12-05T06:33:24.612 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:24.599+0000 7f30dbfff700 1 -- 172.21.15.42:0/2886450152 --> v1:172.21.15.42:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f30e4065300 con 0x7f30e4064ed0 2019-12-05T06:33:24.612 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:24.599+0000 7f30dbfff700 1 -- 172.21.15.42:0/2886450152 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7f30cc001a60 con 0x7f30e4064ed0 2019-12-05T06:33:24.612 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:24.599+0000 7f30e2d9d700 1 -- 172.21.15.42:0/2886450152 >> v1:172.21.15.42:6789/0 conn(0x7f30e4064ed0 legacy=0x7f30e407b810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:24.612 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:24.599+0000 7f30e2d9d700 1 -- 172.21.15.42:0/2886450152 >> v1:172.21.15.40:6789/0 conn(0x7f30e41140a0 legacy=0x7f30e41144d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:24.612 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:24.599+0000 7f30e2d9d700 1 -- 172.21.15.42:0/2886450152 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30e4065a60 con 0x7f30e4063bc0 2019-12-05T06:33:24.612 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:24.599+0000 7f30e2d9d700 1 --2- 172.21.15.42:0/2886450152 >> v2:172.21.15.40:3300/0 conn(0x7f30e4063bc0 0x7f30e408ee90 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f30dc008290 tx=0x7f30dc0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:24.613 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:24.599+0000 7f30e2d9d700 1 --2- 172.21.15.42:0/2886450152 >> v2:172.21.15.40:3300/0 conn(0x7f30e4063bc0 0x7f30e408ee90 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f30dc008290 tx=0x7f30dc0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:24.613 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:24.599+0000 7f30e2d9d700 1 --2- 172.21.15.42:0/2886450152 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f30e4063bc0 0x7f30e408ee90 secure :-1 s=READY pgs=2114 cs=0 l=1 rx=0x7f30dc008290 tx=0x7f30dc0100b0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:33:24.613 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:24.599+0000 7f30e2d9d700 1 --2- 172.21.15.42:0/2886450152 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f30e4063bc0 0x7f30e408ee90 secure :-1 s=THROTTLE_DONE pgs=2114 cs=0 l=1 rx=0x7f30dc008290 tx=0x7f30dc0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:24.613 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:24.599+0000 7f30dbfff700 1 -- 172.21.15.42:0/2886450152 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f30d4011e00 con 0x7f30e4063bc0 2019-12-05T06:33:24.613 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:24.599+0000 7f30e2d9d700 1 --2- 172.21.15.42:0/2886450152 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f30e4063bc0 0x7f30e408ee90 secure :-1 s=THROTTLE_DONE pgs=2114 cs=0 l=1 rx=0x7f30dc008290 tx=0x7f30dc0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:24.613 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:24.599+0000 7f30e2d9d700 1 --2- 172.21.15.42:0/2886450152 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f30e4063bc0 0x7f30e408ee90 secure :-1 s=THROTTLE_DONE pgs=2114 cs=0 l=1 rx=0x7f30dc008290 tx=0x7f30dc0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:24.614 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:24.599+0000 7f30dbfff700 1 -- 172.21.15.42:0/2886450152 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30d40010c0 con 0x7f30e4063bc0 2019-12-05T06:33:24.614 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:24.599+0000 7f30dbfff700 1 -- 172.21.15.42:0/2886450152 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f30d4001880 con 0x7f30e4063bc0 2019-12-05T06:33:24.614 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:24.599+0000 7f30e8bfd700 1 -- 172.21.15.42:0/2886450152 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30e41325a0 con 0x7f30e4063bc0 2019-12-05T06:33:24.614 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:24.599+0000 7f30e8bfd700 1 -- 172.21.15.42:0/2886450152 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30e4132b20 con 0x7f30e4063bc0 2019-12-05T06:33:24.614 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:24.599+0000 7f30e2d9d700 1 --2- 172.21.15.42:0/2886450152 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f30e4063bc0 0x7f30e408ee90 secure :-1 s=THROTTLE_DONE pgs=2114 cs=0 l=1 rx=0x7f30dc008290 tx=0x7f30dc0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:24.615 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:24.599+0000 7f30dbfff700 1 -- 172.21.15.42:0/2886450152 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f30d4001b90 con 0x7f30e4063bc0 2019-12-05T06:33:24.615 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:24.599+0000 7f30e2d9d700 1 --2- 172.21.15.42:0/2886450152 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f30e4063bc0 0x7f30e408ee90 secure :-1 s=THROTTLE_DONE pgs=2114 cs=0 l=1 rx=0x7f30dc008290 tx=0x7f30dc0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:24.615 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:24.599+0000 7f30dbfff700 1 --2- 172.21.15.42:0/2886450152 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f30c803b640 0x7f30c803db50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:33:24.615 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:24.599+0000 7f30dbfff700 1 -- 172.21.15.42:0/2886450152 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 11191+0+0 (secure 0 0 0) 0x7f30d404a6d0 con 0x7f30e4063bc0 2019-12-05T06:33:24.615 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:24.599+0000 7f30da59c700 1 --2- 172.21.15.42:0/2886450152 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f30c803b640 0x7f30c803db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:33:24.615 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:24.603+0000 7f30da59c700 1 --2- 172.21.15.42:0/2886450152 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f30c803b640 0x7f30c803db50 crc :-1 s=READY pgs=1175 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:33:24.616 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:24.603+0000 7f30e8bfd700 1 -- 172.21.15.42:0/2886450152 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30e405d110 con 0x7f30e4063bc0 2019-12-05T06:33:24.619 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:24.611+0000 7f30e2d9d700 1 --2- 172.21.15.42:0/2886450152 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f30e4063bc0 0x7f30e408ee90 secure :-1 s=THROTTLE_DONE pgs=2114 cs=0 l=1 rx=0x7f30dc008290 tx=0x7f30dc0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:24.619 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:24.611+0000 7f30dbfff700 1 -- 172.21.15.42:0/2886450152 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f30d4031750 con 0x7f30e4063bc0 2019-12-05T06:33:24.788 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:24.783+0000 7f30e8bfd700 1 -- 172.21.15.42:0/2886450152 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f30e405b8c0 con 0x7f30e4063bc0 2019-12-05T06:33:24.789 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:24.783+0000 7f30e2d9d700 1 --2- 172.21.15.42:0/2886450152 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f30e4063bc0 0x7f30e408ee90 secure :-1 s=THROTTLE_DONE pgs=2114 cs=0 l=1 rx=0x7f30dc008290 tx=0x7f30dc0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:24.789 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:24.783+0000 7f30dbfff700 1 -- 172.21.15.42:0/2886450152 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 333 v333) v1 ==== 95+0+2117 (secure 0 0 0) 0x7f30d40319b0 con 0x7f30e4063bc0 2019-12-05T06:33:24.791 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:33:24.792 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":333,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":11433,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/2532474042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2532474042},{"type":"v1","addr":"172.21.15.42:6835","nonce":2532474042}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":333}],"filesystems":[{"mdsmap":{"epoch":332,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:31:20.143327+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":109,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11352},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11352":{"gid":11352,"name":"b","rank":0,"incarnation":329,"state":"up:active","state_seq":16,"addr":"172.21.15.40:6833/2055706790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2055706790},{"type":"v1","addr":"172.21.15.40:6833","nonce":2055706790}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:33:24.792 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 333 2019-12-05T06:33:24.794 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:24.787+0000 7f30e8bfd700 1 -- 172.21.15.42:0/2886450152 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f30c803b640 msgr2=0x7f30c803db50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:24.794 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:24.787+0000 7f30e8bfd700 1 --2- 172.21.15.42:0/2886450152 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f30c803b640 0x7f30c803db50 crc :-1 s=READY pgs=1175 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:24.794 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:24.787+0000 7f30e8bfd700 1 -- 172.21.15.42:0/2886450152 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f30e4063bc0 msgr2=0x7f30e408ee90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:24.794 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:24.787+0000 7f30e8bfd700 1 --2- 172.21.15.42:0/2886450152 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f30e4063bc0 0x7f30e408ee90 secure :-1 s=READY pgs=2114 cs=0 l=1 rx=0x7f30dc008290 tx=0x7f30dc0100b0).stop 2019-12-05T06:33:24.794 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:24.787+0000 7f30e8bfd700 1 -- 172.21.15.42:0/2886450152 shutdown_connections 2019-12-05T06:33:24.794 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:24.787+0000 7f30e8bfd700 1 --2- 172.21.15.42:0/2886450152 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f30c803b640 0x7f30c803db50 unknown :-1 s=CLOSED pgs=1175 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:24.795 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:24.787+0000 7f30e8bfd700 1 --2- 172.21.15.42:0/2886450152 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f30e4063bc0 0x7f30e408ee90 unknown :-1 s=CLOSED pgs=2114 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:24.795 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:24.787+0000 7f30e8bfd700 1 -- 172.21.15.42:0/2886450152 shutdown_connections 2019-12-05T06:33:24.795 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:24.787+0000 7f30e8bfd700 1 -- 172.21.15.42:0/2886450152 wait complete. 2019-12-05T06:33:24.795 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:24.787+0000 7f30e8bfd700 1 -- 172.21.15.42:0/2886450152 >> 172.21.15.42:0/2886450152 conn(0x7f30e40a7f90 msgr2=0x7f30e4116220 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:33:24.804 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:33:26.805 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:33:26.913 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:26.899+0000 7f5dc307e700 1 Processor -- start 2019-12-05T06:33:26.914 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:26.899+0000 7f5dc307e700 1 -- start start 2019-12-05T06:33:26.914 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:26.903+0000 7f5dc307e700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5dbc08df80 0x7f5dbc09f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:33:26.914 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:26.903+0000 7f5dc307e700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5dbc09f830 0x7f5dbc14efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:33:26.914 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:26.903+0000 7f5dc307e700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5dbc09fc60 con 0x7f5dbc0a7740 2019-12-05T06:33:26.914 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:26.903+0000 7f5dc307e700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5dbc0a0b00 con 0x7f5dbc09f830 2019-12-05T06:33:26.915 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:26.903+0000 7f5dc307e700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5dbc08e5b0 con 0x7f5dbc08df80 2019-12-05T06:33:26.915 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:26.903+0000 7f5dc0e1a700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5dbc08df80 0x7f5dbc09f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:33:26.915 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:26.903+0000 7f5dc0e1a700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5dbc08df80 0x7f5dbc09f2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:58010/0 (socket says 172.21.15.42:58010) 2019-12-05T06:33:26.915 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:26.903+0000 7f5dc0e1a700 1 -- 172.21.15.42:0/693861412 learned_addr learned my addr 172.21.15.42:0/693861412 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:33:26.915 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:26.903+0000 7f5dc1e1c700 1 --2- 172.21.15.42:0/693861412 >> v2:172.21.15.40:3300/0 conn(0x7f5dbc09f830 0x7f5dbc14efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:33:26.915 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:26.903+0000 7f5dc0e1a700 1 -- 172.21.15.42:0/693861412 >> v2:172.21.15.40:3300/0 conn(0x7f5dbc09f830 msgr2=0x7f5dbc14efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:33:26.915 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:26.903+0000 7f5dc0e1a700 1 --2- 172.21.15.42:0/693861412 >> v2:172.21.15.40:3300/0 conn(0x7f5dbc09f830 0x7f5dbc14efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:33:26.916 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:26.903+0000 7f5dc0e1a700 1 -- 172.21.15.42:0/693861412 >> v1:172.21.15.40:6789/0 conn(0x7f5dbc0a7740 legacy=0x7f5dbc07e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:26.916 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:26.903+0000 7f5dc0e1a700 1 -- 172.21.15.42:0/693861412 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dbc0a0710 con 0x7f5dbc08df80 2019-12-05T06:33:26.916 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:26.903+0000 7f5dc0e1a700 1 --2- 172.21.15.42:0/693861412 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5dbc08df80 0x7f5dbc09f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f5db0008790 tx=0x7f5db000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:26.916 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:26.903+0000 7f5dc0e1a700 1 --2- 172.21.15.42:0/693861412 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5dbc08df80 0x7f5dbc09f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f5db0008790 tx=0x7f5db000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:26.916 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:26.903+0000 7f5dc0e1a700 1 --2- 172.21.15.42:0/693861412 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5dbc08df80 0x7f5dbc09f2e0 secure :-1 s=READY pgs=619 cs=0 l=1 rx=0x7f5db0008790 tx=0x7f5db000fd30).ready entity=mon.2 client_cookie=8d0e2f9d0879769d server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:33:26.916 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:26.903+0000 7f5dc0e1a700 1 --2- 172.21.15.42:0/693861412 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5dbc08df80 0x7f5dbc09f2e0 secure :-1 s=THROTTLE_DONE pgs=619 cs=0 l=1 rx=0x7f5db0008790 tx=0x7f5db000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:26.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:26.903+0000 7f5dc0e1a700 1 --2- 172.21.15.42:0/693861412 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5dbc08df80 0x7f5dbc09f2e0 secure :-1 s=THROTTLE_DONE pgs=619 cs=0 l=1 rx=0x7f5db0008790 tx=0x7f5db000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:26.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:26.903+0000 7f5dc0e1a700 1 --2- 172.21.15.42:0/693861412 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5dbc08df80 0x7f5dbc09f2e0 secure :-1 s=THROTTLE_DONE pgs=619 cs=0 l=1 rx=0x7f5db0008790 tx=0x7f5db000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:26.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:26.903+0000 7f5daffff700 1 -- 172.21.15.42:0/693861412 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5db0008c90 con 0x7f5dbc08df80 2019-12-05T06:33:26.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:26.903+0000 7f5daffff700 1 -- 172.21.15.42:0/693861412 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5db00088b0 con 0x7f5dbc08df80 2019-12-05T06:33:26.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:26.903+0000 7f5daffff700 1 -- 172.21.15.42:0/693861412 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5db0011b10 con 0x7f5dbc08df80 2019-12-05T06:33:26.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:26.903+0000 7f5dc307e700 1 -- 172.21.15.42:0/693861412 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5dbc08df80 msgr2=0x7f5dbc09f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:26.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:26.903+0000 7f5dc307e700 1 --2- 172.21.15.42:0/693861412 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5dbc08df80 0x7f5dbc09f2e0 secure :-1 s=READY pgs=619 cs=0 l=1 rx=0x7f5db0008790 tx=0x7f5db000fd30).stop 2019-12-05T06:33:26.918 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:26.907+0000 7f5dc307e700 1 -- 172.21.15.42:0/693861412 shutdown_connections 2019-12-05T06:33:26.918 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:26.907+0000 7f5dc307e700 1 --2- 172.21.15.42:0/693861412 >> v2:172.21.15.40:3300/0 conn(0x7f5dbc09f830 0x7f5dbc14efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:33:26.918 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:26.907+0000 7f5dc307e700 1 --2- 172.21.15.42:0/693861412 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5dbc08df80 0x7f5dbc09f2e0 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:26.918 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:26.907+0000 7f5dc307e700 1 -- 172.21.15.42:0/693861412 shutdown_connections 2019-12-05T06:33:26.918 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:26.907+0000 7f5dc307e700 1 -- 172.21.15.42:0/693861412 wait complete. 2019-12-05T06:33:26.918 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:26.907+0000 7f5dc307e700 1 -- 172.21.15.42:0/693861412 >> 172.21.15.42:0/693861412 conn(0x7f5dbc0aabe0 msgr2=0x7f5dbc0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:33:26.921 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:26.907+0000 7f5dc307e700 1 Processor -- start 2019-12-05T06:33:26.921 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:26.907+0000 7f5dc307e700 1 -- start start 2019-12-05T06:33:26.922 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:26.907+0000 7f5dc307e700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5dbc0a6b70 0x7f5dbc11a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:33:26.922 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:26.907+0000 7f5dc307e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5dbc1120f0 0x7f5dbc0a4a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:33:26.922 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:26.907+0000 7f5dc307e700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5dbc117d30 con 0x7f5dbc117900 2019-12-05T06:33:26.922 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:26.907+0000 7f5dc307e700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5dbc0a51d0 con 0x7f5dbc0a6b70 2019-12-05T06:33:26.922 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:26.907+0000 7f5dc307e700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5dbc11a280 con 0x7f5dbc1120f0 2019-12-05T06:33:26.922 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:26.907+0000 7f5dc0e1a700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5dbc1120f0 0x7f5dbc0a4a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:33:26.922 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:26.907+0000 7f5dc0e1a700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5dbc1120f0 0x7f5dbc0a4a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:57548/0 (socket says 172.21.15.42:57548) 2019-12-05T06:33:26.923 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:26.907+0000 7f5dc0e1a700 1 -- 172.21.15.42:0/4235938973 learned_addr learned my addr 172.21.15.42:0/4235938973 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:33:26.923 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:26.907+0000 7f5dc161b700 1 --2- 172.21.15.42:0/4235938973 >> v2:172.21.15.40:3300/0 conn(0x7f5dbc0a6b70 0x7f5dbc11a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:33:26.923 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:26.911+0000 7f5dc161b700 1 -- 172.21.15.42:0/4235938973 >> v2:172.21.15.42:3300/0 conn(0x7f5dbc1120f0 msgr2=0x7f5dbc0a4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:26.923 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:26.911+0000 7f5dc161b700 1 --2- 172.21.15.42:0/4235938973 >> v2:172.21.15.42:3300/0 conn(0x7f5dbc1120f0 0x7f5dbc0a4a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:26.923 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:26.911+0000 7f5dc161b700 1 -- 172.21.15.42:0/4235938973 >> v1:172.21.15.42:6789/0 conn(0x7f5dbc117900 legacy=0x7f5dbc07db50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:26.923 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:26.911+0000 7f5dc161b700 1 -- 172.21.15.42:0/4235938973 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dbc09f2e0 con 0x7f5dbc0a6b70 2019-12-05T06:33:26.923 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:26.911+0000 7f5dc161b700 1 --2- 172.21.15.42:0/4235938973 >> v2:172.21.15.40:3300/0 conn(0x7f5dbc0a6b70 0x7f5dbc11a7c0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f5dbc11a110 tx=0x7f5db4010c70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:26.924 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:26.911+0000 7f5dc161b700 1 --2- 172.21.15.42:0/4235938973 >> v2:172.21.15.40:3300/0 conn(0x7f5dbc0a6b70 0x7f5dbc11a7c0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f5dbc11a110 tx=0x7f5db4010c70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:26.924 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:26.911+0000 7f5dc161b700 1 --2- 172.21.15.42:0/4235938973 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5dbc0a6b70 0x7f5dbc11a7c0 secure :-1 s=READY pgs=2116 cs=0 l=1 rx=0x7f5dbc11a110 tx=0x7f5db4010c70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:33:26.924 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:26.911+0000 7f5dc161b700 1 --2- 172.21.15.42:0/4235938973 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5dbc0a6b70 0x7f5dbc11a7c0 secure :-1 s=THROTTLE_DONE pgs=2116 cs=0 l=1 rx=0x7f5dbc11a110 tx=0x7f5db4010c70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:26.924 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:26.911+0000 7f5dadffb700 1 -- 172.21.15.42:0/4235938973 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5db400ca20 con 0x7f5dbc0a6b70 2019-12-05T06:33:26.924 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:26.911+0000 7f5dc161b700 1 --2- 172.21.15.42:0/4235938973 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5dbc0a6b70 0x7f5dbc11a7c0 secure :-1 s=THROTTLE_DONE pgs=2116 cs=0 l=1 rx=0x7f5dbc11a110 tx=0x7f5db4010c70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:26.924 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:26.911+0000 7f5dc307e700 1 -- 172.21.15.42:0/4235938973 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5dbc09f550 con 0x7f5dbc0a6b70 2019-12-05T06:33:26.925 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:26.911+0000 7f5dadffb700 1 -- 172.21.15.42:0/4235938973 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5db40101c0 con 0x7f5dbc0a6b70 2019-12-05T06:33:26.925 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:26.911+0000 7f5dc161b700 1 --2- 172.21.15.42:0/4235938973 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5dbc0a6b70 0x7f5dbc11a7c0 secure :-1 s=THROTTLE_DONE pgs=2116 cs=0 l=1 rx=0x7f5dbc11a110 tx=0x7f5db4010c70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:26.925 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:26.911+0000 7f5dadffb700 1 -- 172.21.15.42:0/4235938973 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5db40124a0 con 0x7f5dbc0a6b70 2019-12-05T06:33:26.925 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:26.911+0000 7f5dc307e700 1 -- 172.21.15.42:0/4235938973 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5dbc137100 con 0x7f5dbc0a6b70 2019-12-05T06:33:26.925 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:26.911+0000 7f5dc161b700 1 --2- 172.21.15.42:0/4235938973 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5dbc0a6b70 0x7f5dbc11a7c0 secure :-1 s=THROTTLE_DONE pgs=2116 cs=0 l=1 rx=0x7f5dbc11a110 tx=0x7f5db4010c70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:26.925 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:26.911+0000 7f5dadffb700 1 -- 172.21.15.42:0/4235938973 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f5db4012800 con 0x7f5dbc0a6b70 2019-12-05T06:33:26.925 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:26.911+0000 7f5dc161b700 1 --2- 172.21.15.42:0/4235938973 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5dbc0a6b70 0x7f5dbc11a7c0 secure :-1 s=THROTTLE_DONE pgs=2116 cs=0 l=1 rx=0x7f5dbc11a110 tx=0x7f5db4010c70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:26.926 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:26.911+0000 7f5dadffb700 1 --2- 172.21.15.42:0/4235938973 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5d9c03adb0 0x7f5d9c03d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:33:26.926 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:26.911+0000 7f5dadffb700 1 -- 172.21.15.42:0/4235938973 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 11277+0+0 (secure 0 0 0) 0x7f5db404b3f0 con 0x7f5dbc0a6b70 2019-12-05T06:33:26.926 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:26.911+0000 7f5dc0e1a700 1 --2- 172.21.15.42:0/4235938973 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5d9c03adb0 0x7f5d9c03d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:33:26.926 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:26.911+0000 7f5dc0e1a700 1 --2- 172.21.15.42:0/4235938973 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5d9c03adb0 0x7f5d9c03d2c0 crc :-1 s=READY pgs=1176 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:33:26.926 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:26.915+0000 7f5dc307e700 1 -- 172.21.15.42:0/4235938973 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5dbc05d110 con 0x7f5dbc0a6b70 2019-12-05T06:33:26.933 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:26.927+0000 7f5dc161b700 1 --2- 172.21.15.42:0/4235938973 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5dbc0a6b70 0x7f5dbc11a7c0 secure :-1 s=THROTTLE_DONE pgs=2116 cs=0 l=1 rx=0x7f5dbc11a110 tx=0x7f5db4010c70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:26.933 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:26.927+0000 7f5dadffb700 1 -- 172.21.15.42:0/4235938973 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f5db40323f0 con 0x7f5dbc0a6b70 2019-12-05T06:33:27.096 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:27.091+0000 7f5dc307e700 1 -- 172.21.15.42:0/4235938973 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5dbc05b8c0 con 0x7f5dbc0a6b70 2019-12-05T06:33:27.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:27.091+0000 7f5dc161b700 1 --2- 172.21.15.42:0/4235938973 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5dbc0a6b70 0x7f5dbc11a7c0 secure :-1 s=THROTTLE_DONE pgs=2116 cs=0 l=1 rx=0x7f5dbc11a110 tx=0x7f5db4010c70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:27.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:27.091+0000 7f5dadffb700 1 -- 172.21.15.42:0/4235938973 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 334 v334) v1 ==== 95+0+1778 (secure 0 0 0) 0x7f5db4032650 con 0x7f5dbc0a6b70 2019-12-05T06:33:27.099 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 334 2019-12-05T06:33:27.099 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:33:27.100 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":334,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":334,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:33:25.052967+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":110,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11433},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11433":{"gid":11433,"name":"a","rank":0,"incarnation":334,"state":"up:replay","state_seq":2,"addr":"172.21.15.42:6835/2532474042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2532474042},{"type":"v1","addr":"172.21.15.42:6835","nonce":2532474042}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:33:27.103 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:27.095+0000 7f5dc307e700 1 -- 172.21.15.42:0/4235938973 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5d9c03adb0 msgr2=0x7f5d9c03d2c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:27.103 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:27.095+0000 7f5dc307e700 1 --2- 172.21.15.42:0/4235938973 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5d9c03adb0 0x7f5d9c03d2c0 crc :-1 s=READY pgs=1176 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:27.103 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:27.095+0000 7f5dc307e700 1 -- 172.21.15.42:0/4235938973 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5dbc0a6b70 msgr2=0x7f5dbc11a7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:27.103 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:27.095+0000 7f5dc307e700 1 --2- 172.21.15.42:0/4235938973 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5dbc0a6b70 0x7f5dbc11a7c0 secure :-1 s=READY pgs=2116 cs=0 l=1 rx=0x7f5dbc11a110 tx=0x7f5db4010c70).stop 2019-12-05T06:33:27.104 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:27.095+0000 7f5dc307e700 1 -- 172.21.15.42:0/4235938973 shutdown_connections 2019-12-05T06:33:27.104 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:27.095+0000 7f5dc307e700 1 --2- 172.21.15.42:0/4235938973 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5d9c03adb0 0x7f5d9c03d2c0 unknown :-1 s=CLOSED pgs=1176 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:27.104 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:27.095+0000 7f5dc307e700 1 --2- 172.21.15.42:0/4235938973 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5dbc0a6b70 0x7f5dbc11a7c0 unknown :-1 s=CLOSED pgs=2116 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:27.104 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:27.095+0000 7f5dc307e700 1 --2- 172.21.15.42:0/4235938973 >> v2:172.21.15.42:3300/0 conn(0x7f5dbc1120f0 0x7f5dbc0a4a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:27.105 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:27.095+0000 7f5dc307e700 1 -- 172.21.15.42:0/4235938973 shutdown_connections 2019-12-05T06:33:27.105 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:27.095+0000 7f5dc307e700 1 -- 172.21.15.42:0/4235938973 wait complete. 2019-12-05T06:33:27.105 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:27.095+0000 7f5dc307e700 1 -- 172.21.15.42:0/4235938973 >> 172.21.15.42:0/4235938973 conn(0x7f5dbc0a7f90 msgr2=0x7f5dbc118880 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:33:27.120 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b down, removed from mdsmap 2019-12-05T06:33:27.122 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2019-12-05T06:33:27.122 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:33:27.219 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:27.207+0000 7f84af2cc700 1 Processor -- start 2019-12-05T06:33:27.219 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:27.207+0000 7f84af2cc700 1 -- start start 2019-12-05T06:33:27.220 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:27.211+0000 7f84af2cc700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f84a80a7740 0x7f84a808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:33:27.220 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:27.211+0000 7f84af2cc700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f84a807e720 0x7f84a807eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:33:27.220 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:27.211+0000 7f84af2cc700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84a80a0710 con 0x7f84a809f2e0 2019-12-05T06:33:27.220 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:27.211+0000 7f84af2cc700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84a80a0b00 con 0x7f84a807e720 2019-12-05T06:33:27.220 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:27.211+0000 7f84af2cc700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84a808e5b0 con 0x7f84a80a7740 2019-12-05T06:33:27.220 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:27.211+0000 7f84ad068700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f84a807e720 0x7f84a807eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:33:27.221 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:27.211+0000 7f84ad068700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f84a807e720 0x7f84a807eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:57558/0 (socket says 172.21.15.42:57558) 2019-12-05T06:33:27.221 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:27.211+0000 7f84ad068700 1 -- 172.21.15.42:0/549555085 learned_addr learned my addr 172.21.15.42:0/549555085 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:33:27.221 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:27.211+0000 7f84ad869700 1 --2- 172.21.15.42:0/549555085 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f84a80a7740 0x7f84a808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:33:27.221 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:27.211+0000 7f84ad068700 1 -- 172.21.15.42:0/549555085 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f84a80a7740 msgr2=0x7f84a808df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:33:27.221 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:27.211+0000 7f84ad068700 1 --2- 172.21.15.42:0/549555085 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f84a80a7740 0x7f84a808df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:33:27.221 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:27.211+0000 7f84ad068700 1 -- 172.21.15.42:0/549555085 >> v1:172.21.15.40:6789/0 conn(0x7f84a809f2e0 legacy=0x7f84a809f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:27.222 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:27.211+0000 7f84ad068700 1 -- 172.21.15.42:0/549555085 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84a807f0a0 con 0x7f84a807e720 2019-12-05T06:33:27.222 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:27.211+0000 7f84ad068700 1 --2- 172.21.15.42:0/549555085 >> v2:172.21.15.42:3300/0 conn(0x7f84a807e720 0x7f84a807eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f8498008790 tx=0x7f849800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:27.222 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:27.211+0000 7f84ad068700 1 --2- 172.21.15.42:0/549555085 >> v2:172.21.15.42:3300/0 conn(0x7f84a807e720 0x7f84a807eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f8498008790 tx=0x7f849800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:27.222 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:27.211+0000 7f84ad068700 1 --2- 172.21.15.42:0/549555085 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f84a807e720 0x7f84a807eb50 secure :-1 s=READY pgs=2802 cs=0 l=1 rx=0x7f8498008790 tx=0x7f849800fd30).ready entity=mon.1 client_cookie=960df4a4043b08de server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:33:27.222 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:27.211+0000 7f84ad068700 1 --2- 172.21.15.42:0/549555085 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f84a807e720 0x7f84a807eb50 secure :-1 s=THROTTLE_DONE pgs=2802 cs=0 l=1 rx=0x7f8498008790 tx=0x7f849800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:27.222 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:27.211+0000 7f84ad068700 1 --2- 172.21.15.42:0/549555085 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f84a807e720 0x7f84a807eb50 secure :-1 s=THROTTLE_DONE pgs=2802 cs=0 l=1 rx=0x7f8498008790 tx=0x7f849800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:27.223 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:27.211+0000 7f84ac867700 1 -- 172.21.15.42:0/549555085 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f84980113b0 con 0x7f84a807e720 2019-12-05T06:33:27.223 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:27.211+0000 7f84ad068700 1 --2- 172.21.15.42:0/549555085 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f84a807e720 0x7f84a807eb50 secure :-1 s=THROTTLE_DONE pgs=2802 cs=0 l=1 rx=0x7f8498008790 tx=0x7f849800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:27.223 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:27.211+0000 7f84ac867700 1 -- 172.21.15.42:0/549555085 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8498008bb0 con 0x7f84a807e720 2019-12-05T06:33:27.223 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:27.211+0000 7f84ac867700 1 -- 172.21.15.42:0/549555085 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8498011990 con 0x7f84a807e720 2019-12-05T06:33:27.223 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:27.211+0000 7f84af2cc700 1 -- 172.21.15.42:0/549555085 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f84a807e720 msgr2=0x7f84a807eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:27.223 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:27.211+0000 7f84af2cc700 1 --2- 172.21.15.42:0/549555085 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f84a807e720 0x7f84a807eb50 secure :-1 s=READY pgs=2802 cs=0 l=1 rx=0x7f8498008790 tx=0x7f849800fd30).stop 2019-12-05T06:33:27.224 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:27.211+0000 7f84af2cc700 1 -- 172.21.15.42:0/549555085 shutdown_connections 2019-12-05T06:33:27.224 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:27.211+0000 7f84af2cc700 1 --2- 172.21.15.42:0/549555085 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f84a80a7740 0x7f84a808df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:33:27.224 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:27.211+0000 7f84af2cc700 1 --2- 172.21.15.42:0/549555085 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f84a807e720 0x7f84a807eb50 unknown :-1 s=CLOSED pgs=2802 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:27.224 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:27.211+0000 7f84af2cc700 1 -- 172.21.15.42:0/549555085 shutdown_connections 2019-12-05T06:33:27.225 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:27.211+0000 7f84af2cc700 1 -- 172.21.15.42:0/549555085 wait complete. 2019-12-05T06:33:27.225 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:27.211+0000 7f84af2cc700 1 -- 172.21.15.42:0/549555085 >> 172.21.15.42:0/549555085 conn(0x7f84a80aabe0 msgr2=0x7f84a80a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:33:27.232 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:27.215+0000 7f84af2cc700 1 Processor -- start 2019-12-05T06:33:27.233 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:27.215+0000 7f84af2cc700 1 -- start start 2019-12-05T06:33:27.233 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:27.215+0000 7f84af2cc700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f84a8118a80 0x7f84a812d640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:33:27.233 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:27.215+0000 7f84af2cc700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f84a812db90 0x7f84a81179a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:33:27.233 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:27.215+0000 7f84af2cc700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84a8118320 con 0x7f84a8117ef0 2019-12-05T06:33:27.234 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:27.215+0000 7f84af2cc700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84a812d3a0 con 0x7f84a8118a80 2019-12-05T06:33:27.234 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:27.215+0000 7f84af2cc700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84a8143590 con 0x7f84a812db90 2019-12-05T06:33:27.234 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:27.215+0000 7f84ad068700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f84a812db90 0x7f84a81179a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:33:27.234 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:27.215+0000 7f84ad068700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f84a812db90 0x7f84a81179a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:57564/0 (socket says 172.21.15.42:57564) 2019-12-05T06:33:27.235 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:27.215+0000 7f84ad068700 1 -- 172.21.15.42:0/3766010567 learned_addr learned my addr 172.21.15.42:0/3766010567 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:33:27.235 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:27.215+0000 7f84ad869700 1 --2- 172.21.15.42:0/3766010567 >> v2:172.21.15.40:3300/0 conn(0x7f84a8118a80 0x7f84a812d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:33:27.236 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:27.215+0000 7f84ad068700 1 -- 172.21.15.42:0/3766010567 >> v2:172.21.15.40:3300/0 conn(0x7f84a8118a80 msgr2=0x7f84a812d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:27.236 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:27.215+0000 7f84ad068700 1 --2- 172.21.15.42:0/3766010567 >> v2:172.21.15.40:3300/0 conn(0x7f84a8118a80 0x7f84a812d640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:27.236 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:27.215+0000 7f84ad068700 1 -- 172.21.15.42:0/3766010567 >> v1:172.21.15.42:6789/0 conn(0x7f84a8117ef0 legacy=0x7f84a8130540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:27.237 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:27.215+0000 7f849e7fc700 1 -- 172.21.15.42:0/3766010567 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3064744461 0 0) 0x7f84a8118320 con 0x7f84a8117ef0 2019-12-05T06:33:27.237 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:27.215+0000 7f84ad068700 1 -- 172.21.15.42:0/3766010567 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84a812cf80 con 0x7f84a812db90 2019-12-05T06:33:27.238 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:27.215+0000 7f84ad068700 1 --2- 172.21.15.42:0/3766010567 >> v2:172.21.15.42:3300/0 conn(0x7f84a812db90 0x7f84a81179a0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f8498008790 tx=0x7f8498008e60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:27.238 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:27.215+0000 7f84ad068700 1 --2- 172.21.15.42:0/3766010567 >> v2:172.21.15.42:3300/0 conn(0x7f84a812db90 0x7f84a81179a0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f8498008790 tx=0x7f8498008e60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:27.238 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:27.215+0000 7f84ad068700 1 --2- 172.21.15.42:0/3766010567 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f84a812db90 0x7f84a81179a0 secure :-1 s=READY pgs=2804 cs=0 l=1 rx=0x7f8498008790 tx=0x7f8498008e60).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:33:27.239 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:27.215+0000 7f84ad068700 1 --2- 172.21.15.42:0/3766010567 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f84a812db90 0x7f84a81179a0 secure :-1 s=THROTTLE_DONE pgs=2804 cs=0 l=1 rx=0x7f8498008790 tx=0x7f8498008e60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:27.240 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:27.215+0000 7f849e7fc700 1 -- 172.21.15.42:0/3766010567 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8498007150 con 0x7f84a812db90 2019-12-05T06:33:27.240 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:27.215+0000 7f84ad068700 1 --2- 172.21.15.42:0/3766010567 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f84a812db90 0x7f84a81179a0 secure :-1 s=THROTTLE_DONE pgs=2804 cs=0 l=1 rx=0x7f8498008790 tx=0x7f8498008e60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:27.240 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:27.215+0000 7f849e7fc700 1 -- 172.21.15.42:0/3766010567 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84980115b0 con 0x7f84a812db90 2019-12-05T06:33:27.240 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:27.215+0000 7f84ad068700 1 --2- 172.21.15.42:0/3766010567 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f84a812db90 0x7f84a81179a0 secure :-1 s=THROTTLE_DONE pgs=2804 cs=0 l=1 rx=0x7f8498008790 tx=0x7f8498008e60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:27.241 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:27.215+0000 7f84af2cc700 1 -- 172.21.15.42:0/3766010567 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84a8118eb0 con 0x7f84a812db90 2019-12-05T06:33:27.241 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:27.215+0000 7f849e7fc700 1 -- 172.21.15.42:0/3766010567 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8498005730 con 0x7f84a812db90 2019-12-05T06:33:27.241 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:27.215+0000 7f84af2cc700 1 -- 172.21.15.42:0/3766010567 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84a807cfb0 con 0x7f84a812db90 2019-12-05T06:33:27.241 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:27.215+0000 7f84ad068700 1 --2- 172.21.15.42:0/3766010567 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f84a812db90 0x7f84a81179a0 secure :-1 s=THROTTLE_DONE pgs=2804 cs=0 l=1 rx=0x7f8498008790 tx=0x7f8498008e60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:27.242 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:27.219+0000 7f849e7fc700 1 -- 172.21.15.42:0/3766010567 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f84980059d0 con 0x7f84a812db90 2019-12-05T06:33:27.242 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:27.219+0000 7f84ad068700 1 --2- 172.21.15.42:0/3766010567 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f84a812db90 0x7f84a81179a0 secure :-1 s=THROTTLE_DONE pgs=2804 cs=0 l=1 rx=0x7f8498008790 tx=0x7f8498008e60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:27.242 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:27.219+0000 7f849e7fc700 1 --2- 172.21.15.42:0/3766010567 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f849003acc0 0x7f849003d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:33:27.242 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:27.219+0000 7f849e7fc700 1 -- 172.21.15.42:0/3766010567 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 11277+0+0 (secure 0 0 0) 0x7f849804b580 con 0x7f84a812db90 2019-12-05T06:33:27.243 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:27.219+0000 7f84ad869700 1 --2- 172.21.15.42:0/3766010567 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f849003acc0 0x7f849003d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:33:27.243 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:27.219+0000 7f84ad869700 1 --2- 172.21.15.42:0/3766010567 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f849003acc0 0x7f849003d1d0 crc :-1 s=READY pgs=1177 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:33:27.243 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:27.223+0000 7f84af2cc700 1 -- 172.21.15.42:0/3766010567 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84a805d110 con 0x7f84a812db90 2019-12-05T06:33:27.244 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:27.227+0000 7f84ad068700 1 --2- 172.21.15.42:0/3766010567 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f84a812db90 0x7f84a81179a0 secure :-1 s=THROTTLE_DONE pgs=2804 cs=0 l=1 rx=0x7f8498008790 tx=0x7f8498008e60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:27.244 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:27.227+0000 7f849e7fc700 1 -- 172.21.15.42:0/3766010567 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f8498032840 con 0x7f84a812db90 2019-12-05T06:33:27.395 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:27.387+0000 7f84af2cc700 1 -- 172.21.15.42:0/3766010567 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f84a805b8c0 con 0x7f84a812db90 2019-12-05T06:33:27.396 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:27.387+0000 7f84ad068700 1 --2- 172.21.15.42:0/3766010567 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f84a812db90 0x7f84a81179a0 secure :-1 s=THROTTLE_DONE pgs=2804 cs=0 l=1 rx=0x7f8498008790 tx=0x7f8498008e60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:27.396 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:27.387+0000 7f849e7fc700 1 -- 172.21.15.42:0/3766010567 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 334 v334) v1 ==== 95+0+1778 (secure 0 0 0) 0x7f84980302a0 con 0x7f84a812db90 2019-12-05T06:33:27.396 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 334 2019-12-05T06:33:27.397 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:33:27.397 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":334,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":334,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:33:25.052967+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":110,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11433},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11433":{"gid":11433,"name":"a","rank":0,"incarnation":334,"state":"up:replay","state_seq":2,"addr":"172.21.15.42:6835/2532474042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2532474042},{"type":"v1","addr":"172.21.15.42:6835","nonce":2532474042}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:33:27.399 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:27.391+0000 7f84af2cc700 1 -- 172.21.15.42:0/3766010567 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f849003acc0 msgr2=0x7f849003d1d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:27.399 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:27.391+0000 7f84af2cc700 1 --2- 172.21.15.42:0/3766010567 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f849003acc0 0x7f849003d1d0 crc :-1 s=READY pgs=1177 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:27.399 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:27.391+0000 7f84af2cc700 1 -- 172.21.15.42:0/3766010567 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f84a812db90 msgr2=0x7f84a81179a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:27.399 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:27.391+0000 7f84af2cc700 1 --2- 172.21.15.42:0/3766010567 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f84a812db90 0x7f84a81179a0 secure :-1 s=READY pgs=2804 cs=0 l=1 rx=0x7f8498008790 tx=0x7f8498008e60).stop 2019-12-05T06:33:27.399 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:27.391+0000 7f84af2cc700 1 -- 172.21.15.42:0/3766010567 shutdown_connections 2019-12-05T06:33:27.399 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:27.391+0000 7f84af2cc700 1 --2- 172.21.15.42:0/3766010567 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f849003acc0 0x7f849003d1d0 unknown :-1 s=CLOSED pgs=1177 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:27.400 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:27.391+0000 7f84af2cc700 1 --2- 172.21.15.42:0/3766010567 >> v2:172.21.15.40:3300/0 conn(0x7f84a8118a80 0x7f84a812d640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:27.400 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:27.391+0000 7f84af2cc700 1 --2- 172.21.15.42:0/3766010567 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f84a812db90 0x7f84a81179a0 unknown :-1 s=CLOSED pgs=2804 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:27.400 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:27.391+0000 7f84af2cc700 1 -- 172.21.15.42:0/3766010567 shutdown_connections 2019-12-05T06:33:27.400 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:27.391+0000 7f84af2cc700 1 -- 172.21.15.42:0/3766010567 wait complete. 2019-12-05T06:33:27.400 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:27.391+0000 7f84af2cc700 1 -- 172.21.15.42:0/3766010567 >> 172.21.15.42:0/3766010567 conn(0x7f84a807da60 msgr2=0x7f84a80a6a60 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:33:27.408 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 334, "feature_flags": { "enable_multiple": false, "ever_enabled_multiple": false }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2019-12-05T05:03:35.604738+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 334, "ever_allowed_features": 2, "explicitly_allowed_features": 2, "failed": [], "flags": 18, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_11433": { "addr": "172.21.15.42:6835/2532474042", "addrs": { "addrvec": [ { "addr": "172.21.15.42:6834", "nonce": 2532474042, "type": "v2" }, { "addr": "172.21.15.42:6835", "nonce": 2532474042, "type": "v1" } ] }, "export_targets": [], "features": 4540138292836696063, "flags": 0, "gid": 11433, "incarnation": 334, "name": "a", "rank": 0, "state": "up:replay", "state_seq": 2 } }, "last_failure": 0, "last_failure_osd_epoch": 110, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 2, "min_compat_client": "0 (unknown)", "modified": "2019-12-05T06:33:25.052967+0000", "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 11433 } } } ], "standbys": [] } 2019-12-05T06:33:29.409 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:33:29.536 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:29.511+0000 7f9f3c04b700 1 Processor -- start 2019-12-05T06:33:29.536 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:29.511+0000 7f9f3c04b700 1 -- start start 2019-12-05T06:33:29.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:29.511+0000 7f9f3c04b700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f9f340a7740 0x7f9f3408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:33:29.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:29.511+0000 7f9f3c04b700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9f3409fae0 0x7f9f3407eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:33:29.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:29.511+0000 7f9f3c04b700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f340a0710 con 0x7f9f3407e720 2019-12-05T06:33:29.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:29.511+0000 7f9f3c04b700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f340a0b00 con 0x7f9f340a7740 2019-12-05T06:33:29.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:29.511+0000 7f9f3c04b700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f3408e5b0 con 0x7f9f3409fae0 2019-12-05T06:33:29.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:29.511+0000 7f9f3a5e8700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f9f340a7740 0x7f9f3408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:33:29.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:29.511+0000 7f9f39de7700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f9f3407e720 0x7f9f3409f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:57610/0 (socket says 172.21.15.42:57610) 2019-12-05T06:33:29.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:29.511+0000 7f9f39de7700 1 -- 172.21.15.42:0/595916724 learned_addr learned my addr 172.21.15.42:0/595916724 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:33:29.539 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:29.515+0000 7f9f3ade9700 1 --2- 172.21.15.42:0/595916724 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9f3409fae0 0x7f9f3407eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:33:29.539 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:29.515+0000 7f9f3a5e8700 1 -- 172.21.15.42:0/595916724 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9f3409fae0 msgr2=0x7f9f3407eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:33:29.539 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:29.515+0000 7f9f3a5e8700 1 --2- 172.21.15.42:0/595916724 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9f3409fae0 0x7f9f3407eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:33:29.539 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:29.515+0000 7f9f3a5e8700 1 -- 172.21.15.42:0/595916724 >> v1:172.21.15.42:6789/0 conn(0x7f9f3407e720 legacy=0x7f9f3409f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:29.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:29.515+0000 7f9f395e6700 1 -- 172.21.15.42:0/595916724 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3174690857 0 0) 0x7f9f340a0710 con 0x7f9f3407e720 2019-12-05T06:33:29.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:29.515+0000 7f9f3a5e8700 1 -- 172.21.15.42:0/595916724 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f3407f0a0 con 0x7f9f340a7740 2019-12-05T06:33:29.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:29.515+0000 7f9f3a5e8700 1 --2- 172.21.15.42:0/595916724 >> v2:172.21.15.42:3300/0 conn(0x7f9f340a7740 0x7f9f3408df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f9f30008790 tx=0x7f9f3000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:29.541 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:29.515+0000 7f9f3a5e8700 1 --2- 172.21.15.42:0/595916724 >> v2:172.21.15.42:3300/0 conn(0x7f9f340a7740 0x7f9f3408df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f9f30008790 tx=0x7f9f3000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:29.541 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:29.515+0000 7f9f3a5e8700 1 --2- 172.21.15.42:0/595916724 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9f340a7740 0x7f9f3408df80 secure :-1 s=READY pgs=2806 cs=0 l=1 rx=0x7f9f30008790 tx=0x7f9f3000fc70).ready entity=mon.1 client_cookie=ce850cb1c7f825c8 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:33:29.541 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:29.515+0000 7f9f3a5e8700 1 --2- 172.21.15.42:0/595916724 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9f340a7740 0x7f9f3408df80 secure :-1 s=THROTTLE_DONE pgs=2806 cs=0 l=1 rx=0x7f9f30008790 tx=0x7f9f3000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:29.541 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:29.515+0000 7f9f395e6700 1 -- 172.21.15.42:0/595916724 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9f30008bb0 con 0x7f9f340a7740 2019-12-05T06:33:29.542 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:29.515+0000 7f9f3a5e8700 1 --2- 172.21.15.42:0/595916724 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9f340a7740 0x7f9f3408df80 secure :-1 s=THROTTLE_DONE pgs=2806 cs=0 l=1 rx=0x7f9f30008790 tx=0x7f9f3000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:29.542 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:29.515+0000 7f9f395e6700 1 -- 172.21.15.42:0/595916724 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f300087d0 con 0x7f9f340a7740 2019-12-05T06:33:29.542 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:29.515+0000 7f9f3a5e8700 1 --2- 172.21.15.42:0/595916724 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9f340a7740 0x7f9f3408df80 secure :-1 s=THROTTLE_DONE pgs=2806 cs=0 l=1 rx=0x7f9f30008790 tx=0x7f9f3000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:29.542 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:29.515+0000 7f9f3c04b700 1 -- 172.21.15.42:0/595916724 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9f340a7740 msgr2=0x7f9f3408df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:29.543 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:29.515+0000 7f9f395e6700 1 -- 172.21.15.42:0/595916724 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9f30011860 con 0x7f9f340a7740 2019-12-05T06:33:29.543 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:29.515+0000 7f9f3c04b700 1 --2- 172.21.15.42:0/595916724 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9f340a7740 0x7f9f3408df80 secure :-1 s=READY pgs=2806 cs=0 l=1 rx=0x7f9f30008790 tx=0x7f9f3000fc70).stop 2019-12-05T06:33:29.543 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:29.515+0000 7f9f3c04b700 1 -- 172.21.15.42:0/595916724 shutdown_connections 2019-12-05T06:33:29.543 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:29.515+0000 7f9f3c04b700 1 --2- 172.21.15.42:0/595916724 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9f3409fae0 0x7f9f3407eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:33:29.544 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:29.515+0000 7f9f3c04b700 1 --2- 172.21.15.42:0/595916724 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9f340a7740 0x7f9f3408df80 unknown :-1 s=CLOSED pgs=2806 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:29.544 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:29.515+0000 7f9f3c04b700 1 -- 172.21.15.42:0/595916724 shutdown_connections 2019-12-05T06:33:29.544 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:29.515+0000 7f9f3c04b700 1 -- 172.21.15.42:0/595916724 wait complete. 2019-12-05T06:33:29.544 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:29.515+0000 7f9f3c04b700 1 -- 172.21.15.42:0/595916724 >> 172.21.15.42:0/595916724 conn(0x7f9f340aabe0 msgr2=0x7f9f340a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:33:29.545 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:29.515+0000 7f9f3c04b700 1 Processor -- start 2019-12-05T06:33:29.545 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:29.519+0000 7f9f3c04b700 1 -- start start 2019-12-05T06:33:29.545 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:29.519+0000 7f9f3c04b700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9f3408ed10 0x7f9f340a5f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:33:29.546 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:29.519+0000 7f9f3c04b700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f9f341174d0 0x7f9f3411abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:33:29.546 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:29.519+0000 7f9f3c04b700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f340a5a10 con 0x7f9f341161c0 2019-12-05T06:33:29.546 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:29.519+0000 7f9f3c04b700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f340a64d0 con 0x7f9f341174d0 2019-12-05T06:33:29.547 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:29.519+0000 7f9f3c04b700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f340a4ba0 con 0x7f9f3408ed10 2019-12-05T06:33:29.547 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:29.519+0000 7f9f3a5e8700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f9f341161c0 0x7f9f3411a3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:57620/0 (socket says 172.21.15.42:57620) 2019-12-05T06:33:29.547 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:29.519+0000 7f9f3a5e8700 1 -- 172.21.15.42:0/894554718 learned_addr learned my addr 172.21.15.42:0/894554718 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:33:29.548 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:29.519+0000 7f9f3ade9700 1 --2- 172.21.15.42:0/894554718 >> v2:172.21.15.40:3300/0 conn(0x7f9f341174d0 0x7f9f3411abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:33:29.548 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:29.519+0000 7f9f39de7700 1 --2- 172.21.15.42:0/894554718 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9f3408ed10 0x7f9f340a5f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:33:29.548 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:29.519+0000 7f9f2affd700 1 -- 172.21.15.42:0/894554718 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 869858325 0 0) 0x7f9f340a5a10 con 0x7f9f341161c0 2019-12-05T06:33:29.548 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:29.519+0000 7f9f2affd700 1 -- 172.21.15.42:0/894554718 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f140030b0 con 0x7f9f341161c0 2019-12-05T06:33:29.549 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:29.519+0000 7f9f39de7700 1 -- 172.21.15.42:0/894554718 >> v2:172.21.15.40:3300/0 conn(0x7f9f341174d0 msgr2=0x7f9f3411abc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:29.549 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:29.519+0000 7f9f39de7700 1 --2- 172.21.15.42:0/894554718 >> v2:172.21.15.40:3300/0 conn(0x7f9f341174d0 0x7f9f3411abc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:29.549 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:29.519+0000 7f9f39de7700 1 -- 172.21.15.42:0/894554718 >> v1:172.21.15.42:6789/0 conn(0x7f9f341161c0 legacy=0x7f9f3411a3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:29.550 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:29.519+0000 7f9f39de7700 1 -- 172.21.15.42:0/894554718 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f3408f140 con 0x7f9f3408ed10 2019-12-05T06:33:29.550 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:29.519+0000 7f9f3ade9700 1 --2- 172.21.15.42:0/894554718 >> v2:172.21.15.40:3300/0 conn(0x7f9f341174d0 0x7f9f3411abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:33:29.550 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:29.519+0000 7f9f39de7700 1 --2- 172.21.15.42:0/894554718 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9f3408ed10 0x7f9f340a5f80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f9f24004720 tx=0x7f9f24010c20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:29.550 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:29.519+0000 7f9f39de7700 1 --2- 172.21.15.42:0/894554718 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9f3408ed10 0x7f9f340a5f80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f9f24004720 tx=0x7f9f24010c20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:29.551 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:29.519+0000 7f9f39de7700 1 --2- 172.21.15.42:0/894554718 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9f3408ed10 0x7f9f340a5f80 secure :-1 s=READY pgs=620 cs=0 l=1 rx=0x7f9f24004720 tx=0x7f9f24010c20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:33:29.552 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:29.519+0000 7f9f39de7700 1 --2- 172.21.15.42:0/894554718 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9f3408ed10 0x7f9f340a5f80 secure :-1 s=THROTTLE_DONE pgs=620 cs=0 l=1 rx=0x7f9f24004720 tx=0x7f9f24010c20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:29.552 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:29.519+0000 7f9f39de7700 1 --2- 172.21.15.42:0/894554718 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9f3408ed10 0x7f9f340a5f80 secure :-1 s=THROTTLE_DONE pgs=620 cs=0 l=1 rx=0x7f9f24004720 tx=0x7f9f24010c20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:29.552 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:29.519+0000 7f9f2affd700 1 -- 172.21.15.42:0/894554718 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9f24011950 con 0x7f9f3408ed10 2019-12-05T06:33:29.552 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:29.519+0000 7f9f39de7700 1 --2- 172.21.15.42:0/894554718 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9f3408ed10 0x7f9f340a5f80 secure :-1 s=THROTTLE_DONE pgs=620 cs=0 l=1 rx=0x7f9f24004720 tx=0x7f9f24010c20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:29.553 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:29.519+0000 7f9f2affd700 1 -- 172.21.15.42:0/894554718 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f24011570 con 0x7f9f3408ed10 2019-12-05T06:33:29.553 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:29.519+0000 7f9f3c04b700 1 -- 172.21.15.42:0/894554718 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f3411b110 con 0x7f9f3408ed10 2019-12-05T06:33:29.553 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:29.519+0000 7f9f2affd700 1 -- 172.21.15.42:0/894554718 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9f24011f70 con 0x7f9f3408ed10 2019-12-05T06:33:29.553 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:29.519+0000 7f9f3c04b700 1 -- 172.21.15.42:0/894554718 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f34117900 con 0x7f9f3408ed10 2019-12-05T06:33:29.554 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:29.523+0000 7f9f39de7700 1 --2- 172.21.15.42:0/894554718 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9f3408ed10 0x7f9f340a5f80 secure :-1 s=THROTTLE_DONE pgs=620 cs=0 l=1 rx=0x7f9f24004720 tx=0x7f9f24010c20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:29.554 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:29.523+0000 7f9f39de7700 1 --2- 172.21.15.42:0/894554718 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9f3408ed10 0x7f9f340a5f80 secure :-1 s=THROTTLE_DONE pgs=620 cs=0 l=1 rx=0x7f9f24004720 tx=0x7f9f24010c20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:29.554 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:29.523+0000 7f9f2affd700 1 -- 172.21.15.42:0/894554718 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f9f24012280 con 0x7f9f3408ed10 2019-12-05T06:33:29.555 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:29.523+0000 7f9f2affd700 1 --2- 172.21.15.42:0/894554718 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9f1403b280 0x7f9f1403d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:33:29.555 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:29.523+0000 7f9f2affd700 1 -- 172.21.15.42:0/894554718 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 11277+0+0 (secure 0 0 0) 0x7f9f2404b290 con 0x7f9f3408ed10 2019-12-05T06:33:29.555 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:29.523+0000 7f9f3a5e8700 1 --2- 172.21.15.42:0/894554718 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9f1403b280 0x7f9f1403d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:33:29.555 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:29.523+0000 7f9f3a5e8700 1 --2- 172.21.15.42:0/894554718 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9f1403b280 0x7f9f1403d790 crc :-1 s=READY pgs=1178 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:33:29.556 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:29.527+0000 7f9f3c04b700 1 -- 172.21.15.42:0/894554718 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f3405d110 con 0x7f9f3408ed10 2019-12-05T06:33:29.557 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:29.539+0000 7f9f39de7700 1 --2- 172.21.15.42:0/894554718 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9f3408ed10 0x7f9f340a5f80 secure :-1 s=THROTTLE_DONE pgs=620 cs=0 l=1 rx=0x7f9f24004720 tx=0x7f9f24010c20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:29.557 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:29.539+0000 7f9f2affd700 1 -- 172.21.15.42:0/894554718 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f9f24032270 con 0x7f9f3408ed10 2019-12-05T06:33:29.712 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:29.703+0000 7f9f3c04b700 1 -- 172.21.15.42:0/894554718 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9f3405b8c0 con 0x7f9f3408ed10 2019-12-05T06:33:29.713 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:29.707+0000 7f9f39de7700 1 --2- 172.21.15.42:0/894554718 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9f3408ed10 0x7f9f340a5f80 secure :-1 s=THROTTLE_DONE pgs=620 cs=0 l=1 rx=0x7f9f24004720 tx=0x7f9f24010c20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:29.713 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:29.707+0000 7f9f2affd700 1 -- 172.21.15.42:0/894554718 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 336 v336) v1 ==== 95+0+1779 (secure 0 0 0) 0x7f9f240324d0 con 0x7f9f3408ed10 2019-12-05T06:33:29.715 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 336 2019-12-05T06:33:29.715 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:33:29.716 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":336,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":336,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:33:29.160235+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":110,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11433},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11433":{"gid":11433,"name":"a","rank":0,"incarnation":334,"state":"up:rejoin","state_seq":32,"addr":"172.21.15.42:6835/2532474042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2532474042},{"type":"v1","addr":"172.21.15.42:6835","nonce":2532474042}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:33:29.718 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:29.711+0000 7f9f3c04b700 1 -- 172.21.15.42:0/894554718 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9f1403b280 msgr2=0x7f9f1403d790 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:29.718 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:29.711+0000 7f9f3c04b700 1 --2- 172.21.15.42:0/894554718 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9f1403b280 0x7f9f1403d790 crc :-1 s=READY pgs=1178 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:29.718 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:29.711+0000 7f9f3c04b700 1 -- 172.21.15.42:0/894554718 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9f3408ed10 msgr2=0x7f9f340a5f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:29.718 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:29.711+0000 7f9f3c04b700 1 --2- 172.21.15.42:0/894554718 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9f3408ed10 0x7f9f340a5f80 secure :-1 s=READY pgs=620 cs=0 l=1 rx=0x7f9f24004720 tx=0x7f9f24010c20).stop 2019-12-05T06:33:29.718 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:29.711+0000 7f9f3c04b700 1 -- 172.21.15.42:0/894554718 shutdown_connections 2019-12-05T06:33:29.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:29.711+0000 7f9f3c04b700 1 --2- 172.21.15.42:0/894554718 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9f1403b280 0x7f9f1403d790 unknown :-1 s=CLOSED pgs=1178 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:29.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:29.711+0000 7f9f3c04b700 1 --2- 172.21.15.42:0/894554718 >> v2:172.21.15.40:3300/0 conn(0x7f9f341174d0 0x7f9f3411abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:29.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:29.711+0000 7f9f3c04b700 1 --2- 172.21.15.42:0/894554718 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9f3408ed10 0x7f9f340a5f80 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:29.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:29.711+0000 7f9f3c04b700 1 -- 172.21.15.42:0/894554718 shutdown_connections 2019-12-05T06:33:29.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:29.711+0000 7f9f3c04b700 1 -- 172.21.15.42:0/894554718 wait complete. 2019-12-05T06:33:29.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:29.711+0000 7f9f3c04b700 1 -- 172.21.15.42:0/894554718 >> 172.21.15.42:0/894554718 conn(0x7f9f3408fda0 msgr2=0x7f9f34141710 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:33:29.728 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2019-12-05T06:33:31.729 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:33:31.834 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:31.819+0000 7f5d4caaa700 1 Processor -- start 2019-12-05T06:33:31.834 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:31.819+0000 7f5d4caaa700 1 -- start start 2019-12-05T06:33:31.834 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:31.823+0000 7f5d4caaa700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5d4808df80 0x7f5d4809f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:33:31.834 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:31.823+0000 7f5d4caaa700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5d4809f830 0x7f5d4814efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:33:31.834 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:31.823+0000 7f5d4caaa700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d4809fc60 con 0x7f5d480a7740 2019-12-05T06:33:31.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:31.823+0000 7f5d4caaa700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d480a0b00 con 0x7f5d4808df80 2019-12-05T06:33:31.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:31.823+0000 7f5d4caaa700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d4808e5b0 con 0x7f5d4809f830 2019-12-05T06:33:31.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:31.823+0000 7f5d4659c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5d4808df80 0x7f5d4809f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:33:31.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:31.823+0000 7f5d4659c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5d4808df80 0x7f5d4809f2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:57604/0 (socket says 172.21.15.42:57604) 2019-12-05T06:33:31.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:31.823+0000 7f5d4659c700 1 -- 172.21.15.42:0/436205055 learned_addr learned my addr 172.21.15.42:0/436205055 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:33:31.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:31.823+0000 7f5d4759e700 1 --2- 172.21.15.42:0/436205055 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5d4809f830 0x7f5d4814efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:33:31.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:31.823+0000 7f5d4659c700 1 -- 172.21.15.42:0/436205055 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5d4809f830 msgr2=0x7f5d4814efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:33:31.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:31.823+0000 7f5d4659c700 1 --2- 172.21.15.42:0/436205055 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5d4809f830 0x7f5d4814efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:33:31.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:31.823+0000 7f5d4659c700 1 -- 172.21.15.42:0/436205055 >> v1:172.21.15.42:6789/0 conn(0x7f5d480a7740 legacy=0x7f5d4807e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:31.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:31.823+0000 7f5d4659c700 1 -- 172.21.15.42:0/436205055 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d480a0710 con 0x7f5d4808df80 2019-12-05T06:33:31.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:31.823+0000 7f5d4759e700 1 --2- 172.21.15.42:0/436205055 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5d4809f830 0x7f5d4814efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).send_auth_request state changed! 2019-12-05T06:33:31.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:31.823+0000 7f5d4659c700 1 --2- 172.21.15.42:0/436205055 >> v2:172.21.15.42:3300/0 conn(0x7f5d4808df80 0x7f5d4809f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f5d3c008790 tx=0x7f5d3c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:31.837 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:31.823+0000 7f5d4659c700 1 --2- 172.21.15.42:0/436205055 >> v2:172.21.15.42:3300/0 conn(0x7f5d4808df80 0x7f5d4809f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f5d3c008790 tx=0x7f5d3c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:31.837 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:31.823+0000 7f5d4659c700 1 --2- 172.21.15.42:0/436205055 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5d4808df80 0x7f5d4809f2e0 secure :-1 s=READY pgs=2809 cs=0 l=1 rx=0x7f5d3c008790 tx=0x7f5d3c00fd30).ready entity=mon.1 client_cookie=43ef7e6cb0ed5d25 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:33:31.837 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:31.823+0000 7f5d4659c700 1 --2- 172.21.15.42:0/436205055 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5d4808df80 0x7f5d4809f2e0 secure :-1 s=THROTTLE_DONE pgs=2809 cs=0 l=1 rx=0x7f5d3c008790 tx=0x7f5d3c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:31.837 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:31.823+0000 7f5d45d9b700 1 -- 172.21.15.42:0/436205055 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5d3c008c90 con 0x7f5d4808df80 2019-12-05T06:33:31.837 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:31.823+0000 7f5d4659c700 1 --2- 172.21.15.42:0/436205055 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5d4808df80 0x7f5d4809f2e0 secure :-1 s=THROTTLE_DONE pgs=2809 cs=0 l=1 rx=0x7f5d3c008790 tx=0x7f5d3c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:31.838 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:31.823+0000 7f5d45d9b700 1 -- 172.21.15.42:0/436205055 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d3c0088b0 con 0x7f5d4808df80 2019-12-05T06:33:31.838 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:31.823+0000 7f5d4659c700 1 --2- 172.21.15.42:0/436205055 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5d4808df80 0x7f5d4809f2e0 secure :-1 s=THROTTLE_DONE pgs=2809 cs=0 l=1 rx=0x7f5d3c008790 tx=0x7f5d3c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:31.838 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:31.823+0000 7f5d45d9b700 1 -- 172.21.15.42:0/436205055 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5d3c011b10 con 0x7f5d4808df80 2019-12-05T06:33:31.838 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:31.823+0000 7f5d4caaa700 1 -- 172.21.15.42:0/436205055 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5d4808df80 msgr2=0x7f5d4809f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:31.838 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:31.823+0000 7f5d4caaa700 1 --2- 172.21.15.42:0/436205055 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5d4808df80 0x7f5d4809f2e0 secure :-1 s=READY pgs=2809 cs=0 l=1 rx=0x7f5d3c008790 tx=0x7f5d3c00fd30).stop 2019-12-05T06:33:31.838 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:31.827+0000 7f5d4caaa700 1 -- 172.21.15.42:0/436205055 shutdown_connections 2019-12-05T06:33:31.839 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:31.827+0000 7f5d4caaa700 1 --2- 172.21.15.42:0/436205055 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5d4809f830 0x7f5d4814efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:33:31.839 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:31.827+0000 7f5d4caaa700 1 --2- 172.21.15.42:0/436205055 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5d4808df80 0x7f5d4809f2e0 unknown :-1 s=CLOSED pgs=2809 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:31.839 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:31.827+0000 7f5d4caaa700 1 -- 172.21.15.42:0/436205055 shutdown_connections 2019-12-05T06:33:31.839 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:31.827+0000 7f5d4caaa700 1 -- 172.21.15.42:0/436205055 wait complete. 2019-12-05T06:33:31.839 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:31.827+0000 7f5d4caaa700 1 -- 172.21.15.42:0/436205055 >> 172.21.15.42:0/436205055 conn(0x7f5d480aabe0 msgr2=0x7f5d480a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:33:31.852 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:31.827+0000 7f5d4caaa700 1 Processor -- start 2019-12-05T06:33:31.852 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:31.827+0000 7f5d4caaa700 1 -- start start 2019-12-05T06:33:31.852 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:31.831+0000 7f5d4caaa700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5d4808eeb0 0x7f5d48077c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:33:31.852 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:31.831+0000 7f5d4caaa700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d48078190 con 0x7f5d481180c0 2019-12-05T06:33:31.852 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:31.831+0000 7f5d4caaa700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d4813adc0 con 0x7f5d48131d00 2019-12-05T06:33:31.853 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:31.831+0000 7f5d4caaa700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d48124340 con 0x7f5d4808eeb0 2019-12-05T06:33:31.853 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:31.831+0000 7f5d4659c700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f5d48131d00 0x7f5d48132130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:57646/0 (socket says 172.21.15.42:57646) 2019-12-05T06:33:31.853 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:31.831+0000 7f5d4659c700 1 -- 172.21.15.42:0/3420394407 learned_addr learned my addr 172.21.15.42:0/3420394407 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:33:31.856 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:31.831+0000 7f5d46d9d700 1 --2- 172.21.15.42:0/3420394407 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5d4808eeb0 0x7f5d48077c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:33:31.856 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:31.831+0000 7f5d377fe700 1 -- 172.21.15.42:0/3420394407 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4234286838 0 0) 0x7f5d4813adc0 con 0x7f5d48131d00 2019-12-05T06:33:31.857 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:31.831+0000 7f5d377fe700 1 -- 172.21.15.42:0/3420394407 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d2c0030b0 con 0x7f5d48131d00 2019-12-05T06:33:31.858 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:31.831+0000 7f5d377fe700 1 -- 172.21.15.42:0/3420394407 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 909562639 0 0) 0x7f5d48078190 con 0x7f5d481180c0 2019-12-05T06:33:31.858 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:31.831+0000 7f5d377fe700 1 -- 172.21.15.42:0/3420394407 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d4813adc0 con 0x7f5d481180c0 2019-12-05T06:33:31.858 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:31.831+0000 7f5d46d9d700 1 -- 172.21.15.42:0/3420394407 >> v1:172.21.15.42:6789/0 conn(0x7f5d48131d00 legacy=0x7f5d48132130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:31.858 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:31.831+0000 7f5d46d9d700 1 -- 172.21.15.42:0/3420394407 >> v1:172.21.15.40:6789/0 conn(0x7f5d481180c0 legacy=0x7f5d481184f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:31.858 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:31.831+0000 7f5d46d9d700 1 -- 172.21.15.42:0/3420394407 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d4813c000 con 0x7f5d4808eeb0 2019-12-05T06:33:31.859 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:31.831+0000 7f5d46d9d700 1 --2- 172.21.15.42:0/3420394407 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5d4808eeb0 0x7f5d48077c40 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f5d38001210 tx=0x7f5d3800c6a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:31.859 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:31.831+0000 7f5d46d9d700 1 --2- 172.21.15.42:0/3420394407 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5d4808eeb0 0x7f5d48077c40 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f5d38001210 tx=0x7f5d3800c6a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:31.859 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:31.831+0000 7f5d46d9d700 1 --2- 172.21.15.42:0/3420394407 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5d4808eeb0 0x7f5d48077c40 secure :-1 s=READY pgs=621 cs=0 l=1 rx=0x7f5d38001210 tx=0x7f5d3800c6a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:33:31.859 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:31.831+0000 7f5d46d9d700 1 --2- 172.21.15.42:0/3420394407 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5d4808eeb0 0x7f5d48077c40 secure :-1 s=THROTTLE_DONE pgs=621 cs=0 l=1 rx=0x7f5d38001210 tx=0x7f5d3800c6a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:31.860 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:31.831+0000 7f5d377fe700 1 -- 172.21.15.42:0/3420394407 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5d380101c0 con 0x7f5d4808eeb0 2019-12-05T06:33:31.860 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:31.831+0000 7f5d46d9d700 1 --2- 172.21.15.42:0/3420394407 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5d4808eeb0 0x7f5d48077c40 secure :-1 s=THROTTLE_DONE pgs=621 cs=0 l=1 rx=0x7f5d38001210 tx=0x7f5d3800c6a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:31.860 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:31.831+0000 7f5d46d9d700 1 --2- 172.21.15.42:0/3420394407 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5d4808eeb0 0x7f5d48077c40 secure :-1 s=THROTTLE_DONE pgs=621 cs=0 l=1 rx=0x7f5d38001210 tx=0x7f5d3800c6a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:31.860 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:31.831+0000 7f5d377fe700 1 -- 172.21.15.42:0/3420394407 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d3800cb10 con 0x7f5d4808eeb0 2019-12-05T06:33:31.861 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:31.831+0000 7f5d4caaa700 1 -- 172.21.15.42:0/3420394407 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d48139db0 con 0x7f5d4808eeb0 2019-12-05T06:33:31.861 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:31.831+0000 7f5d377fe700 1 -- 172.21.15.42:0/3420394407 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5d380124a0 con 0x7f5d4808eeb0 2019-12-05T06:33:31.861 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:31.831+0000 7f5d4caaa700 1 -- 172.21.15.42:0/3420394407 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d4813a2e0 con 0x7f5d4808eeb0 2019-12-05T06:33:31.861 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:31.835+0000 7f5d46d9d700 1 --2- 172.21.15.42:0/3420394407 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5d4808eeb0 0x7f5d48077c40 secure :-1 s=THROTTLE_DONE pgs=621 cs=0 l=1 rx=0x7f5d38001210 tx=0x7f5d3800c6a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:31.861 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:31.835+0000 7f5d377fe700 1 -- 172.21.15.42:0/3420394407 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f5d38012800 con 0x7f5d4808eeb0 2019-12-05T06:33:31.862 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:31.835+0000 7f5d46d9d700 1 --2- 172.21.15.42:0/3420394407 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5d4808eeb0 0x7f5d48077c40 secure :-1 s=THROTTLE_DONE pgs=621 cs=0 l=1 rx=0x7f5d38001210 tx=0x7f5d3800c6a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:31.862 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:31.835+0000 7f5d377fe700 1 --2- 172.21.15.42:0/3420394407 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5d2c03b410 0x7f5d2c03d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:33:31.862 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:31.835+0000 7f5d377fe700 1 -- 172.21.15.42:0/3420394407 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 11277+0+0 (secure 0 0 0) 0x7f5d3804b1b0 con 0x7f5d4808eeb0 2019-12-05T06:33:31.862 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:31.835+0000 7f5d4659c700 1 --2- 172.21.15.42:0/3420394407 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5d2c03b410 0x7f5d2c03d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:33:31.863 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:31.835+0000 7f5d4659c700 1 --2- 172.21.15.42:0/3420394407 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5d2c03b410 0x7f5d2c03d920 crc :-1 s=READY pgs=1179 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:33:31.863 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:31.835+0000 7f5d4caaa700 1 -- 172.21.15.42:0/3420394407 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d4805d110 con 0x7f5d4808eeb0 2019-12-05T06:33:31.863 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:31.843+0000 7f5d46d9d700 1 --2- 172.21.15.42:0/3420394407 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5d4808eeb0 0x7f5d48077c40 secure :-1 s=THROTTLE_DONE pgs=621 cs=0 l=1 rx=0x7f5d38001210 tx=0x7f5d3800c6a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:31.864 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:31.843+0000 7f5d377fe700 1 -- 172.21.15.42:0/3420394407 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f5d38032230 con 0x7f5d4808eeb0 2019-12-05T06:33:32.021 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:32.011+0000 7f5d4caaa700 1 -- 172.21.15.42:0/3420394407 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5d480a71e0 con 0x7f5d4808eeb0 2019-12-05T06:33:32.022 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:32.015+0000 7f5d46d9d700 1 --2- 172.21.15.42:0/3420394407 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5d4808eeb0 0x7f5d48077c40 secure :-1 s=THROTTLE_DONE pgs=621 cs=0 l=1 rx=0x7f5d38001210 tx=0x7f5d3800c6a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:32.022 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:32.015+0000 7f5d377fe700 1 -- 172.21.15.42:0/3420394407 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 337 v337) v1 ==== 95+0+1779 (secure 0 0 0) 0x7f5d38032490 con 0x7f5d4808eeb0 2019-12-05T06:33:32.023 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:33:32.023 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":337,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":337,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:33:30.171201+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":110,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11433},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11433":{"gid":11433,"name":"a","rank":0,"incarnation":334,"state":"up:active","state_seq":33,"addr":"172.21.15.42:6835/2532474042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2532474042},{"type":"v1","addr":"172.21.15.42:6835","nonce":2532474042}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:33:32.023 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 337 2019-12-05T06:33:32.032 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:32.019+0000 7f5d4caaa700 1 -- 172.21.15.42:0/3420394407 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5d2c03b410 msgr2=0x7f5d2c03d920 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:32.032 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:32.019+0000 7f5d4caaa700 1 --2- 172.21.15.42:0/3420394407 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5d2c03b410 0x7f5d2c03d920 crc :-1 s=READY pgs=1179 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:32.032 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:32.019+0000 7f5d4caaa700 1 -- 172.21.15.42:0/3420394407 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5d4808eeb0 msgr2=0x7f5d48077c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:32.032 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:32.019+0000 7f5d4caaa700 1 --2- 172.21.15.42:0/3420394407 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5d4808eeb0 0x7f5d48077c40 secure :-1 s=READY pgs=621 cs=0 l=1 rx=0x7f5d38001210 tx=0x7f5d3800c6a0).stop 2019-12-05T06:33:32.032 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:32.019+0000 7f5d4caaa700 1 -- 172.21.15.42:0/3420394407 shutdown_connections 2019-12-05T06:33:32.033 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:32.019+0000 7f5d4caaa700 1 --2- 172.21.15.42:0/3420394407 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5d2c03b410 0x7f5d2c03d920 unknown :-1 s=CLOSED pgs=1179 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:32.033 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:32.019+0000 7f5d4caaa700 1 --2- 172.21.15.42:0/3420394407 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5d4808eeb0 0x7f5d48077c40 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:32.033 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:32.019+0000 7f5d4caaa700 1 -- 172.21.15.42:0/3420394407 shutdown_connections 2019-12-05T06:33:32.033 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:32.019+0000 7f5d4caaa700 1 -- 172.21.15.42:0/3420394407 wait complete. 2019-12-05T06:33:32.033 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:32.019+0000 7f5d4caaa700 1 -- 172.21.15.42:0/3420394407 >> 172.21.15.42:0/3420394407 conn(0x7f5d480a6d70 msgr2=0x7f5d4811c9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:33:32.037 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a has gained rank=0, replacing gid=11352 2019-12-05T06:33:32.037 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 9 secs before reviving mds.b 2019-12-05T06:33:41.038 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.b 2019-12-05T06:33:41.038 INFO:tasks.ceph.mds.b:Restarting daemon 2019-12-05T06:33:41.038 INFO:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2019-12-05T06:33:41.042 INFO:tasks.ceph.mds.b:Started 2019-12-05T06:33:41.042 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:33:41.101 INFO:tasks.ceph.mds.b.smithi040.stdout:starting mds.b at 2019-12-05T06:33:41.150 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:41.135+0000 7fccc112f700 1 Processor -- start 2019-12-05T06:33:41.151 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:41.135+0000 7fccc112f700 1 -- start start 2019-12-05T06:33:41.151 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:41.139+0000 7fccc112f700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fccbc0a7740 0x7fccbc08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:33:41.151 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:41.139+0000 7fccc112f700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fccbc09fae0 0x7fccbc07eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:33:41.151 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:41.139+0000 7fccc112f700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccbc0a0710 con 0x7fccbc07e720 2019-12-05T06:33:41.151 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:41.139+0000 7fccc112f700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fccbc0a0b00 con 0x7fccbc0a7740 2019-12-05T06:33:41.151 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:41.139+0000 7fccc112f700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fccbc08e5b0 con 0x7fccbc09fae0 2019-12-05T06:33:41.152 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:41.139+0000 7fccc092e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fccbc09fae0 0x7fccbc07eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:33:41.152 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:41.139+0000 7fccc092e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fccbc09fae0 0x7fccbc07eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:57646/0 (socket says 172.21.15.42:57646) 2019-12-05T06:33:41.152 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:41.139+0000 7fccc092e700 1 -- 172.21.15.42:0/219469214 learned_addr learned my addr 172.21.15.42:0/219469214 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:33:41.152 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:41.139+0000 7fccbad9d700 1 --1- 172.21.15.42:0/219469214 >> v1:172.21.15.42:6789/0 conn(0x7fccbc07e720 0x7fccbc09f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:57682/0 (socket says 172.21.15.42:57682) 2019-12-05T06:33:41.152 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:41.139+0000 7fccbb59e700 1 --2- 172.21.15.42:0/219469214 >> v2:172.21.15.40:3300/0 conn(0x7fccbc0a7740 0x7fccbc08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:33:41.152 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:41.139+0000 7fccc092e700 1 -- 172.21.15.42:0/219469214 >> v2:172.21.15.40:3300/0 conn(0x7fccbc0a7740 msgr2=0x7fccbc08df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:33:41.152 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:41.139+0000 7fccc092e700 1 --2- 172.21.15.42:0/219469214 >> v2:172.21.15.40:3300/0 conn(0x7fccbc0a7740 0x7fccbc08df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:33:41.153 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:41.139+0000 7fccc092e700 1 -- 172.21.15.42:0/219469214 >> v1:172.21.15.42:6789/0 conn(0x7fccbc07e720 legacy=0x7fccbc09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:41.153 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:41.139+0000 7fccc092e700 1 -- 172.21.15.42:0/219469214 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccbc07f0a0 con 0x7fccbc09fae0 2019-12-05T06:33:41.153 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:41.139+0000 7fccba59c700 1 -- 172.21.15.42:0/219469214 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2766464427 0 0) 0x7fccbc0a0710 con 0x7fccbc07e720 2019-12-05T06:33:41.153 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:41.139+0000 7fccc092e700 1 --2- 172.21.15.42:0/219469214 >> v2:172.21.15.42:3300/0 conn(0x7fccbc09fae0 0x7fccbc07eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fccb40032b0 tx=0x7fccb40100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:41.153 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:41.139+0000 7fccc092e700 1 --2- 172.21.15.42:0/219469214 >> v2:172.21.15.42:3300/0 conn(0x7fccbc09fae0 0x7fccbc07eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fccb40032b0 tx=0x7fccb40100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:41.154 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:41.139+0000 7fccc092e700 1 --2- 172.21.15.42:0/219469214 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fccbc09fae0 0x7fccbc07eb50 secure :-1 s=READY pgs=2813 cs=0 l=1 rx=0x7fccb40032b0 tx=0x7fccb40100b0).ready entity=mon.1 client_cookie=5a4164ed0d8dd6af server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:33:41.154 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:41.139+0000 7fccc092e700 1 --2- 172.21.15.42:0/219469214 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fccbc09fae0 0x7fccbc07eb50 secure :-1 s=THROTTLE_DONE pgs=2813 cs=0 l=1 rx=0x7fccb40032b0 tx=0x7fccb40100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:41.154 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:41.139+0000 7fccc092e700 1 --2- 172.21.15.42:0/219469214 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fccbc09fae0 0x7fccbc07eb50 secure :-1 s=THROTTLE_DONE pgs=2813 cs=0 l=1 rx=0x7fccb40032b0 tx=0x7fccb40100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:41.154 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:41.139+0000 7fccba59c700 1 -- 172.21.15.42:0/219469214 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fccb4010d60 con 0x7fccbc09fae0 2019-12-05T06:33:41.154 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:41.139+0000 7fccc092e700 1 --2- 172.21.15.42:0/219469214 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fccbc09fae0 0x7fccbc07eb50 secure :-1 s=THROTTLE_DONE pgs=2813 cs=0 l=1 rx=0x7fccb40032b0 tx=0x7fccb40100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:41.155 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:41.139+0000 7fccba59c700 1 -- 172.21.15.42:0/219469214 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccb4010980 con 0x7fccbc09fae0 2019-12-05T06:33:41.155 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:41.139+0000 7fccba59c700 1 -- 172.21.15.42:0/219469214 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fccb40114a0 con 0x7fccbc09fae0 2019-12-05T06:33:41.155 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:41.139+0000 7fccc112f700 1 -- 172.21.15.42:0/219469214 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fccbc09fae0 msgr2=0x7fccbc07eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:41.155 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:41.139+0000 7fccc112f700 1 --2- 172.21.15.42:0/219469214 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fccbc09fae0 0x7fccbc07eb50 secure :-1 s=READY pgs=2813 cs=0 l=1 rx=0x7fccb40032b0 tx=0x7fccb40100b0).stop 2019-12-05T06:33:41.156 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:41.139+0000 7fccc112f700 1 -- 172.21.15.42:0/219469214 shutdown_connections 2019-12-05T06:33:41.156 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:41.139+0000 7fccc112f700 1 --2- 172.21.15.42:0/219469214 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fccbc09fae0 0x7fccbc07eb50 unknown :-1 s=CLOSED pgs=2813 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:41.156 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:41.139+0000 7fccc112f700 1 --2- 172.21.15.42:0/219469214 >> v2:172.21.15.40:3300/0 conn(0x7fccbc0a7740 0x7fccbc08df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:33:41.156 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:41.139+0000 7fccc112f700 1 -- 172.21.15.42:0/219469214 shutdown_connections 2019-12-05T06:33:41.157 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:41.139+0000 7fccc112f700 1 -- 172.21.15.42:0/219469214 wait complete. 2019-12-05T06:33:41.157 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:41.139+0000 7fccc112f700 1 -- 172.21.15.42:0/219469214 >> 172.21.15.42:0/219469214 conn(0x7fccbc0aabe0 msgr2=0x7fccbc0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:33:41.157 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:41.143+0000 7fccc112f700 1 Processor -- start 2019-12-05T06:33:41.157 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:41.143+0000 7fccc112f700 1 -- start start 2019-12-05T06:33:41.157 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:41.143+0000 7fccc112f700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fccbc064b50 0x7fccbc1326f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:33:41.158 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:41.143+0000 7fccc112f700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fccbc132c40 0x7fccbc133070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:33:41.158 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:41.143+0000 7fccc112f700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccbc064f80 con 0x7fccbc1426d0 2019-12-05T06:33:41.158 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:41.143+0000 7fccc112f700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fccbc065950 con 0x7fccbc132c40 2019-12-05T06:33:41.158 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:41.143+0000 7fccc112f700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fccbc142b00 con 0x7fccbc064b50 2019-12-05T06:33:41.159 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:41.143+0000 7fccbad9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fccbc064b50 0x7fccbc1326f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:33:41.159 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:41.143+0000 7fccbad9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fccbc064b50 0x7fccbc1326f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:57648/0 (socket says 172.21.15.42:57648) 2019-12-05T06:33:41.159 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:41.143+0000 7fccbad9d700 1 -- 172.21.15.42:0/1862133425 learned_addr learned my addr 172.21.15.42:0/1862133425 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:33:41.167 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:41.143+0000 7fccc092e700 1 --2- 172.21.15.42:0/1862133425 >> v2:172.21.15.40:3300/0 conn(0x7fccbc132c40 0x7fccbc133070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:33:41.169 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:41.143+0000 7fccc092e700 1 -- 172.21.15.42:0/1862133425 >> v2:172.21.15.42:3300/0 conn(0x7fccbc064b50 msgr2=0x7fccbc1326f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:41.169 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:41.143+0000 7fccc092e700 1 --2- 172.21.15.42:0/1862133425 >> v2:172.21.15.42:3300/0 conn(0x7fccbc064b50 0x7fccbc1326f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:41.170 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:41.143+0000 7fccc092e700 1 -- 172.21.15.42:0/1862133425 >> v1:172.21.15.42:6789/0 conn(0x7fccbc1426d0 legacy=0x7fccbc12f130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:41.170 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:41.143+0000 7fccc092e700 1 -- 172.21.15.42:0/1862133425 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccbc12f930 con 0x7fccbc132c40 2019-12-05T06:33:41.170 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:41.143+0000 7fccc092e700 1 --2- 172.21.15.42:0/1862133425 >> v2:172.21.15.40:3300/0 conn(0x7fccbc132c40 0x7fccbc133070 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fccb40032b0 tx=0x7fccb4010f00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:41.170 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:41.143+0000 7fccc092e700 1 --2- 172.21.15.42:0/1862133425 >> v2:172.21.15.40:3300/0 conn(0x7fccbc132c40 0x7fccbc133070 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fccb40032b0 tx=0x7fccb4010f00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:41.170 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:41.143+0000 7fccc092e700 1 --2- 172.21.15.42:0/1862133425 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fccbc132c40 0x7fccbc133070 secure :-1 s=READY pgs=2123 cs=0 l=1 rx=0x7fccb40032b0 tx=0x7fccb4010f00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:33:41.170 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:41.143+0000 7fccc092e700 1 --2- 172.21.15.42:0/1862133425 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fccbc132c40 0x7fccbc133070 secure :-1 s=THROTTLE_DONE pgs=2123 cs=0 l=1 rx=0x7fccb40032b0 tx=0x7fccb4010f00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:41.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:41.143+0000 7fccc092e700 1 --2- 172.21.15.42:0/1862133425 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fccbc132c40 0x7fccbc133070 secure :-1 s=THROTTLE_DONE pgs=2123 cs=0 l=1 rx=0x7fccb40032b0 tx=0x7fccb4010f00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:41.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:41.143+0000 7fcca3fff700 1 -- 172.21.15.42:0/1862133425 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fccb4011c30 con 0x7fccbc132c40 2019-12-05T06:33:41.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:41.143+0000 7fccc092e700 1 --2- 172.21.15.42:0/1862133425 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fccbc132c40 0x7fccbc133070 secure :-1 s=THROTTLE_DONE pgs=2123 cs=0 l=1 rx=0x7fccb40032b0 tx=0x7fccb4010f00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:41.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:41.143+0000 7fcca3fff700 1 -- 172.21.15.42:0/1862133425 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccb40110c0 con 0x7fccbc132c40 2019-12-05T06:33:41.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:41.143+0000 7fcca3fff700 1 -- 172.21.15.42:0/1862133425 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fccb4005ca0 con 0x7fccbc132c40 2019-12-05T06:33:41.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:41.143+0000 7fccc112f700 1 -- 172.21.15.42:0/1862133425 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccbc063da0 con 0x7fccbc132c40 2019-12-05T06:33:41.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:41.143+0000 7fccc112f700 1 -- 172.21.15.42:0/1862133425 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccbc063190 con 0x7fccbc132c40 2019-12-05T06:33:41.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:41.147+0000 7fccc092e700 1 --2- 172.21.15.42:0/1862133425 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fccbc132c40 0x7fccbc133070 secure :-1 s=THROTTLE_DONE pgs=2123 cs=0 l=1 rx=0x7fccb40032b0 tx=0x7fccb4010f00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:41.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:41.147+0000 7fcca3fff700 1 -- 172.21.15.42:0/1862133425 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fccb400f280 con 0x7fccbc132c40 2019-12-05T06:33:41.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:41.147+0000 7fccc092e700 1 --2- 172.21.15.42:0/1862133425 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fccbc132c40 0x7fccbc133070 secure :-1 s=THROTTLE_DONE pgs=2123 cs=0 l=1 rx=0x7fccb40032b0 tx=0x7fccb4010f00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:41.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:41.147+0000 7fcca3fff700 1 --2- 172.21.15.42:0/1862133425 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fcc9c03ade0 0x7fcc9c03d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:33:41.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:41.147+0000 7fcca3fff700 1 -- 172.21.15.42:0/1862133425 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 11277+0+0 (secure 0 0 0) 0x7fccb404b620 con 0x7fccbc132c40 2019-12-05T06:33:41.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:41.147+0000 7fccbb59e700 1 --2- 172.21.15.42:0/1862133425 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fcc9c03ade0 0x7fcc9c03d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:33:41.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:41.147+0000 7fccbb59e700 1 --2- 172.21.15.42:0/1862133425 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fcc9c03ade0 0x7fcc9c03d2f0 crc :-1 s=READY pgs=1180 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:33:41.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:41.151+0000 7fccc112f700 1 -- 172.21.15.42:0/1862133425 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccbc05d110 con 0x7fccbc132c40 2019-12-05T06:33:41.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:41.159+0000 7fccc092e700 1 --2- 172.21.15.42:0/1862133425 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fccbc132c40 0x7fccbc133070 secure :-1 s=THROTTLE_DONE pgs=2123 cs=0 l=1 rx=0x7fccb40032b0 tx=0x7fccb4010f00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:41.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:41.159+0000 7fcca3fff700 1 -- 172.21.15.42:0/1862133425 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fccb40326a0 con 0x7fccbc132c40 2019-12-05T06:33:41.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:41.339+0000 7fccc112f700 1 -- 172.21.15.42:0/1862133425 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fccbc05b8c0 con 0x7fccbc132c40 2019-12-05T06:33:41.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:41.343+0000 7fccc092e700 1 --2- 172.21.15.42:0/1862133425 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fccbc132c40 0x7fccbc133070 secure :-1 s=THROTTLE_DONE pgs=2123 cs=0 l=1 rx=0x7fccb40032b0 tx=0x7fccb4010f00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:41.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:41.343+0000 7fcca3fff700 1 -- 172.21.15.42:0/1862133425 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 338 v338) v1 ==== 95+0+2116 (secure 0 0 0) 0x7fccb4032900 con 0x7fccbc132c40 2019-12-05T06:33:41.350 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 338 2019-12-05T06:33:41.351 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:33:41.351 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":338,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7910,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/4253974783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":4253974783},{"type":"v1","addr":"172.21.15.40:6833","nonce":4253974783}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":338}],"filesystems":[{"mdsmap":{"epoch":337,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:33:30.171201+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":110,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11433},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11433":{"gid":11433,"name":"a","rank":0,"incarnation":334,"state":"up:active","state_seq":33,"addr":"172.21.15.42:6835/2532474042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2532474042},{"type":"v1","addr":"172.21.15.42:6835","nonce":2532474042}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:33:41.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:41.347+0000 7fccc112f700 1 -- 172.21.15.42:0/1862133425 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fcc9c03ade0 msgr2=0x7fcc9c03d2f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:41.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:41.347+0000 7fccc112f700 1 --2- 172.21.15.42:0/1862133425 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fcc9c03ade0 0x7fcc9c03d2f0 crc :-1 s=READY pgs=1180 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:41.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:41.347+0000 7fccc112f700 1 -- 172.21.15.42:0/1862133425 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fccbc132c40 msgr2=0x7fccbc133070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:41.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:41.347+0000 7fccc112f700 1 --2- 172.21.15.42:0/1862133425 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fccbc132c40 0x7fccbc133070 secure :-1 s=READY pgs=2123 cs=0 l=1 rx=0x7fccb40032b0 tx=0x7fccb4010f00).stop 2019-12-05T06:33:41.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:41.347+0000 7fccc112f700 1 -- 172.21.15.42:0/1862133425 shutdown_connections 2019-12-05T06:33:41.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:41.347+0000 7fccc112f700 1 --2- 172.21.15.42:0/1862133425 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fcc9c03ade0 0x7fcc9c03d2f0 unknown :-1 s=CLOSED pgs=1180 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:41.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:41.347+0000 7fccc112f700 1 --2- 172.21.15.42:0/1862133425 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fccbc132c40 0x7fccbc133070 unknown :-1 s=CLOSED pgs=2123 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:41.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:41.347+0000 7fccc112f700 1 --2- 172.21.15.42:0/1862133425 >> v2:172.21.15.42:3300/0 conn(0x7fccbc064b50 0x7fccbc1326f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:41.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:41.347+0000 7fccc112f700 1 -- 172.21.15.42:0/1862133425 shutdown_connections 2019-12-05T06:33:41.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:41.347+0000 7fccc112f700 1 -- 172.21.15.42:0/1862133425 wait complete. 2019-12-05T06:33:41.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:33:41.347+0000 7fccc112f700 1 -- 172.21.15.42:0/1862133425 >> 172.21.15.42:0/1862133425 conn(0x7fccbc0aabe0 msgr2=0x7fccbc11d480 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:33:41.364 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b reported in up:standby state 2019-12-05T06:33:41.364 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 99 secs before thrashing 2019-12-05T06:33:43.451 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:33:43.455 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:34:13.516 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:34:13.521 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:34:43.589 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:34:43.593 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:35:13.647 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:35:13.654 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:35:20.365 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:35:20.464 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.454+0000 7f53ab9a3700 1 Processor -- start 2019-12-05T06:35:20.464 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.454+0000 7f53ab9a3700 1 -- start start 2019-12-05T06:35:20.464 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.454+0000 7f53ab9a3700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f53a40a7730 0x7f53a408df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:35:20.465 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.454+0000 7f53ab9a3700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f53a407e710 0x7f53a407eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:35:20.465 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.454+0000 7f53ab9a3700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53a40a0700 con 0x7f53a409f2d0 2019-12-05T06:35:20.465 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.454+0000 7f53ab9a3700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53a40a0af0 con 0x7f53a407e710 2019-12-05T06:35:20.465 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.454+0000 7f53ab9a3700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53a408e5a0 con 0x7f53a40a7730 2019-12-05T06:35:20.465 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.454+0000 7f53a9f40700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f53a40a7730 0x7f53a408df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:35:20.465 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.454+0000 7f53a9f40700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f53a40a7730 0x7f53a408df70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:58114/0 (socket says 172.21.15.42:58114) 2019-12-05T06:35:20.466 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.454+0000 7f53a9f40700 1 -- 172.21.15.42:0/3759019056 learned_addr learned my addr 172.21.15.42:0/3759019056 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:35:20.466 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.454+0000 7f53a973f700 1 --2- 172.21.15.42:0/3759019056 >> v2:172.21.15.40:3300/0 conn(0x7f53a407e710 0x7f53a407eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:35:20.466 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.458+0000 7f53a9f40700 1 -- 172.21.15.42:0/3759019056 >> v2:172.21.15.40:3300/0 conn(0x7f53a407e710 msgr2=0x7f53a407eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:35:20.466 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.458+0000 7f53a9f40700 1 --2- 172.21.15.42:0/3759019056 >> v2:172.21.15.40:3300/0 conn(0x7f53a407e710 0x7f53a407eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:35:20.466 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.458+0000 7f53a9f40700 1 -- 172.21.15.42:0/3759019056 >> v1:172.21.15.42:6789/0 conn(0x7f53a409f2d0 legacy=0x7f53a409f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:35:20.466 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.458+0000 7f53a8f3e700 1 -- 172.21.15.42:0/3759019056 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1285251566 0 0) 0x7f53a40a0700 con 0x7f53a409f2d0 2019-12-05T06:35:20.466 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.458+0000 7f53a9f40700 1 -- 172.21.15.42:0/3759019056 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53a407f090 con 0x7f53a40a7730 2019-12-05T06:35:20.467 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.458+0000 7f53a9f40700 1 --2- 172.21.15.42:0/3759019056 >> v2:172.21.15.42:3300/0 conn(0x7f53a40a7730 0x7f53a408df70 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f53940032b0 tx=0x7f53940100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:20.467 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.458+0000 7f53a9f40700 1 --2- 172.21.15.42:0/3759019056 >> v2:172.21.15.42:3300/0 conn(0x7f53a40a7730 0x7f53a408df70 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f53940032b0 tx=0x7f53940100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:20.467 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.458+0000 7f53a9f40700 1 --2- 172.21.15.42:0/3759019056 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f53a40a7730 0x7f53a408df70 secure :-1 s=READY pgs=2816 cs=0 l=1 rx=0x7f53940032b0 tx=0x7f53940100b0).ready entity=mon.1 client_cookie=fb7a72f05a3b25c server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:35:20.467 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.458+0000 7f53a9f40700 1 --2- 172.21.15.42:0/3759019056 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f53a40a7730 0x7f53a408df70 secure :-1 s=THROTTLE_DONE pgs=2816 cs=0 l=1 rx=0x7f53940032b0 tx=0x7f53940100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:20.467 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.458+0000 7f53a8f3e700 1 -- 172.21.15.42:0/3759019056 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5394011dd0 con 0x7f53a40a7730 2019-12-05T06:35:20.467 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.458+0000 7f53a9f40700 1 --2- 172.21.15.42:0/3759019056 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f53a40a7730 0x7f53a408df70 secure :-1 s=THROTTLE_DONE pgs=2816 cs=0 l=1 rx=0x7f53940032b0 tx=0x7f53940100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:20.468 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.458+0000 7f53a9f40700 1 --2- 172.21.15.42:0/3759019056 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f53a40a7730 0x7f53a408df70 secure :-1 s=THROTTLE_DONE pgs=2816 cs=0 l=1 rx=0x7f53940032b0 tx=0x7f53940100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:20.468 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.458+0000 7f53a8f3e700 1 -- 172.21.15.42:0/3759019056 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53940119f0 con 0x7f53a40a7730 2019-12-05T06:35:20.468 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.458+0000 7f53a8f3e700 1 -- 172.21.15.42:0/3759019056 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f539400a540 con 0x7f53a40a7730 2019-12-05T06:35:20.468 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.458+0000 7f53ab9a3700 1 -- 172.21.15.42:0/3759019056 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f53a40a7730 msgr2=0x7f53a408df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:35:20.468 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.458+0000 7f53ab9a3700 1 --2- 172.21.15.42:0/3759019056 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f53a40a7730 0x7f53a408df70 secure :-1 s=READY pgs=2816 cs=0 l=1 rx=0x7f53940032b0 tx=0x7f53940100b0).stop 2019-12-05T06:35:20.468 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.458+0000 7f53ab9a3700 1 -- 172.21.15.42:0/3759019056 shutdown_connections 2019-12-05T06:35:20.468 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.458+0000 7f53ab9a3700 1 --2- 172.21.15.42:0/3759019056 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f53a40a7730 0x7f53a408df70 unknown :-1 s=CLOSED pgs=2816 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:35:20.469 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.458+0000 7f53ab9a3700 1 --2- 172.21.15.42:0/3759019056 >> v2:172.21.15.40:3300/0 conn(0x7f53a407e710 0x7f53a407eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:35:20.469 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.458+0000 7f53ab9a3700 1 -- 172.21.15.42:0/3759019056 shutdown_connections 2019-12-05T06:35:20.469 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.458+0000 7f53ab9a3700 1 -- 172.21.15.42:0/3759019056 wait complete. 2019-12-05T06:35:20.469 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.458+0000 7f53ab9a3700 1 -- 172.21.15.42:0/3759019056 >> 172.21.15.42:0/3759019056 conn(0x7f53a40aabd0 msgr2=0x7f53a40a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:35:20.472 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.462+0000 7f53ab9a3700 1 Processor -- start 2019-12-05T06:35:20.472 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.462+0000 7f53ab9a3700 1 -- start start 2019-12-05T06:35:20.473 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.462+0000 7f53ab9a3700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f53a40a3a50 0x7f53a411eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:35:20.473 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.462+0000 7f53ab9a3700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f53a40722b0 0x7f53a40746f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:35:20.473 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.462+0000 7f53ab9a3700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53a411a470 con 0x7f53a411f2f0 2019-12-05T06:35:20.473 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.462+0000 7f53ab9a3700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53a411b020 con 0x7f53a40a3a50 2019-12-05T06:35:20.473 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.462+0000 7f53ab9a3700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53a411b2c0 con 0x7f53a40722b0 2019-12-05T06:35:20.473 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.462+0000 7f53a973f700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f53a411f2f0 0x7f53a411b560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:58158/0 (socket says 172.21.15.42:58158) 2019-12-05T06:35:20.474 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.462+0000 7f53a973f700 1 -- 172.21.15.42:0/3558192279 learned_addr learned my addr 172.21.15.42:0/3558192279 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:35:20.474 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.462+0000 7f53a9f40700 1 --2- 172.21.15.42:0/3558192279 >> v2:172.21.15.42:3300/0 conn(0x7f53a40a3a50 0x7f53a411eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:35:20.474 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.462+0000 7f53aa741700 1 --2- 172.21.15.42:0/3558192279 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f53a40722b0 0x7f53a40746f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:35:20.474 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.462+0000 7f53a9f40700 1 -- 172.21.15.42:0/3558192279 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f53a40722b0 msgr2=0x7f53a40746f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:35:20.474 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.462+0000 7f53a9f40700 1 --2- 172.21.15.42:0/3558192279 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f53a40722b0 0x7f53a40746f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:35:20.474 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.462+0000 7f53a9f40700 1 -- 172.21.15.42:0/3558192279 >> v1:172.21.15.42:6789/0 conn(0x7f53a411f2f0 legacy=0x7f53a411b560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:35:20.474 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.462+0000 7f53a9f40700 1 -- 172.21.15.42:0/3558192279 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53a411f820 con 0x7f53a40a3a50 2019-12-05T06:35:20.475 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.462+0000 7f53a27fc700 1 -- 172.21.15.42:0/3558192279 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3890817178 0 0) 0x7f53a411a470 con 0x7f53a411f2f0 2019-12-05T06:35:20.475 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.462+0000 7f53aa741700 1 --2- 172.21.15.42:0/3558192279 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f53a40722b0 0x7f53a40746f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:35:20.475 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.462+0000 7f53a9f40700 1 --2- 172.21.15.42:0/3558192279 >> v2:172.21.15.42:3300/0 conn(0x7f53a40a3a50 0x7f53a411eda0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f53a411b1e0 tx=0x7f5394011d60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:20.475 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.466+0000 7f53a9f40700 1 --2- 172.21.15.42:0/3558192279 >> v2:172.21.15.42:3300/0 conn(0x7f53a40a3a50 0x7f53a411eda0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f53a411b1e0 tx=0x7f5394011d60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:20.475 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.466+0000 7f53a9f40700 1 --2- 172.21.15.42:0/3558192279 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f53a40a3a50 0x7f53a411eda0 secure :-1 s=READY pgs=2818 cs=0 l=1 rx=0x7f53a411b1e0 tx=0x7f5394011d60).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:35:20.475 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.466+0000 7f53a9f40700 1 --2- 172.21.15.42:0/3558192279 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f53a40a3a50 0x7f53a411eda0 secure :-1 s=THROTTLE_DONE pgs=2818 cs=0 l=1 rx=0x7f53a411b1e0 tx=0x7f5394011d60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:20.475 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.466+0000 7f53a27fc700 1 -- 172.21.15.42:0/3558192279 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f539400ac50 con 0x7f53a40a3a50 2019-12-05T06:35:20.476 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.466+0000 7f53ab9a3700 1 -- 172.21.15.42:0/3558192279 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53a411bd60 con 0x7f53a40a3a50 2019-12-05T06:35:20.476 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.466+0000 7f53a9f40700 1 --2- 172.21.15.42:0/3558192279 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f53a40a3a50 0x7f53a411eda0 secure :-1 s=THROTTLE_DONE pgs=2818 cs=0 l=1 rx=0x7f53a411b1e0 tx=0x7f5394011d60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:20.476 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.466+0000 7f53a27fc700 1 -- 172.21.15.42:0/3558192279 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f539400a160 con 0x7f53a40a3a50 2019-12-05T06:35:20.476 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.466+0000 7f53a9f40700 1 --2- 172.21.15.42:0/3558192279 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f53a40a3a50 0x7f53a411eda0 secure :-1 s=THROTTLE_DONE pgs=2818 cs=0 l=1 rx=0x7f53a411b1e0 tx=0x7f5394011d60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:20.476 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.466+0000 7f53a27fc700 1 -- 172.21.15.42:0/3558192279 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5394005ca0 con 0x7f53a40a3a50 2019-12-05T06:35:20.476 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.466+0000 7f53ab9a3700 1 -- 172.21.15.42:0/3558192279 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53a4074c40 con 0x7f53a40a3a50 2019-12-05T06:35:20.477 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.466+0000 7f53a9f40700 1 --2- 172.21.15.42:0/3558192279 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f53a40a3a50 0x7f53a411eda0 secure :-1 s=THROTTLE_DONE pgs=2818 cs=0 l=1 rx=0x7f53a411b1e0 tx=0x7f5394011d60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:20.477 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.466+0000 7f53a27fc700 1 -- 172.21.15.42:0/3558192279 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f5394007150 con 0x7f53a40a3a50 2019-12-05T06:35:20.477 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.466+0000 7f53a9f40700 1 --2- 172.21.15.42:0/3558192279 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f53a40a3a50 0x7f53a411eda0 secure :-1 s=THROTTLE_DONE pgs=2818 cs=0 l=1 rx=0x7f53a411b1e0 tx=0x7f5394011d60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:20.477 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.466+0000 7f53a27fc700 1 --2- 172.21.15.42:0/3558192279 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f538403acf0 0x7f538403d200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:35:20.477 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.466+0000 7f53a27fc700 1 -- 172.21.15.42:0/3558192279 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 11277+0+0 (secure 0 0 0) 0x7f539404bcb0 con 0x7f53a40a3a50 2019-12-05T06:35:20.477 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.466+0000 7f53a973f700 1 --2- 172.21.15.42:0/3558192279 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f538403acf0 0x7f538403d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:35:20.477 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.466+0000 7f53a973f700 1 --2- 172.21.15.42:0/3558192279 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f538403acf0 0x7f538403d200 crc :-1 s=READY pgs=1186 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:35:20.478 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.470+0000 7f53ab9a3700 1 -- 172.21.15.42:0/3558192279 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53a405d100 con 0x7f53a40a3a50 2019-12-05T06:35:20.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.474+0000 7f53a9f40700 1 --2- 172.21.15.42:0/3558192279 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f53a40a3a50 0x7f53a411eda0 secure :-1 s=THROTTLE_DONE pgs=2818 cs=0 l=1 rx=0x7f53a411b1e0 tx=0x7f5394011d60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:20.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.474+0000 7f53a27fc700 1 -- 172.21.15.42:0/3558192279 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f5394032d10 con 0x7f53a40a3a50 2019-12-05T06:35:20.641 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.638+0000 7f53ab9a3700 1 -- 172.21.15.42:0/3558192279 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f53a405b8b0 con 0x7f53a40a3a50 2019-12-05T06:35:20.642 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.638+0000 7f53a9f40700 1 --2- 172.21.15.42:0/3558192279 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f53a40a3a50 0x7f53a411eda0 secure :-1 s=THROTTLE_DONE pgs=2818 cs=0 l=1 rx=0x7f53a411b1e0 tx=0x7f5394011d60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:20.642 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.638+0000 7f53a27fc700 1 -- 172.21.15.42:0/3558192279 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 338 v338) v1 ==== 95+0+2116 (secure 0 0 0) 0x7f5394032f70 con 0x7f53a40a3a50 2019-12-05T06:35:20.644 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 338 2019-12-05T06:35:20.644 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:35:20.644 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":338,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7910,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/4253974783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":4253974783},{"type":"v1","addr":"172.21.15.40:6833","nonce":4253974783}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":338}],"filesystems":[{"mdsmap":{"epoch":337,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:33:30.171201+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":110,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11433},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11433":{"gid":11433,"name":"a","rank":0,"incarnation":334,"state":"up:active","state_seq":33,"addr":"172.21.15.42:6835/2532474042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2532474042},{"type":"v1","addr":"172.21.15.42:6835","nonce":2532474042}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:35:20.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.642+0000 7f53ab9a3700 1 -- 172.21.15.42:0/3558192279 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f538403acf0 msgr2=0x7f538403d200 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:35:20.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.642+0000 7f53ab9a3700 1 --2- 172.21.15.42:0/3558192279 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f538403acf0 0x7f538403d200 crc :-1 s=READY pgs=1186 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:35:20.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.642+0000 7f53ab9a3700 1 -- 172.21.15.42:0/3558192279 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f53a40a3a50 msgr2=0x7f53a411eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:35:20.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.642+0000 7f53ab9a3700 1 --2- 172.21.15.42:0/3558192279 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f53a40a3a50 0x7f53a411eda0 secure :-1 s=READY pgs=2818 cs=0 l=1 rx=0x7f53a411b1e0 tx=0x7f5394011d60).stop 2019-12-05T06:35:20.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.642+0000 7f53ab9a3700 1 -- 172.21.15.42:0/3558192279 shutdown_connections 2019-12-05T06:35:20.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.642+0000 7f53ab9a3700 1 --2- 172.21.15.42:0/3558192279 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f538403acf0 0x7f538403d200 unknown :-1 s=CLOSED pgs=1186 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:35:20.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.642+0000 7f53ab9a3700 1 --2- 172.21.15.42:0/3558192279 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f53a40722b0 0x7f53a40746f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:35:20.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.642+0000 7f53ab9a3700 1 --2- 172.21.15.42:0/3558192279 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f53a40a3a50 0x7f53a411eda0 unknown :-1 s=CLOSED pgs=2818 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:35:20.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.642+0000 7f53ab9a3700 1 -- 172.21.15.42:0/3558192279 shutdown_connections 2019-12-05T06:35:20.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.642+0000 7f53ab9a3700 1 -- 172.21.15.42:0/3558192279 wait complete. 2019-12-05T06:35:20.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.642+0000 7f53ab9a3700 1 -- 172.21.15.42:0/3558192279 >> 172.21.15.42:0/3558192279 conn(0x7f53a40aabd0 msgr2=0x7f53a409f650 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:35:20.660 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.a (rank=0) 2019-12-05T06:35:20.662 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2019-12-05T06:35:20.662 INFO:teuthology.orchestra.run:waiting for 300 2019-12-05T06:35:20.678 INFO:tasks.ceph.mds.a:Stopped 2019-12-05T06:35:20.679 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:35:20.847 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.838+0000 7f63516b3700 1 Processor -- start 2019-12-05T06:35:20.848 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.838+0000 7f63516b3700 1 -- start start 2019-12-05T06:35:20.848 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.838+0000 7f63516b3700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f634c09fae0 0x7f634c14fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:35:20.848 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.838+0000 7f63516b3700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f634c0a0710 con 0x7f634c08df80 2019-12-05T06:35:20.848 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.838+0000 7f63516b3700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f634c07ef20 con 0x7f634c0a7740 2019-12-05T06:35:20.848 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.838+0000 7f63516b3700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f634c0a0b00 con 0x7f634c09fae0 2019-12-05T06:35:20.848 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.838+0000 7f634b7fe700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f634c0a7740 0x7f634c07e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:58168/0 (socket says 172.21.15.42:58168) 2019-12-05T06:35:20.848 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.838+0000 7f634b7fe700 1 -- 172.21.15.42:0/4113587974 learned_addr learned my addr 172.21.15.42:0/4113587974 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:35:20.849 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.838+0000 7f634bfff700 1 --2- 172.21.15.42:0/4113587974 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f634c09fae0 0x7f634c14fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:35:20.849 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.838+0000 7f634a7fc700 1 -- 172.21.15.42:0/4113587974 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4041389181 0 0) 0x7f634c07ef20 con 0x7f634c0a7740 2019-12-05T06:35:20.849 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.838+0000 7f634a7fc700 1 -- 172.21.15.42:0/4113587974 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63340030b0 con 0x7f634c0a7740 2019-12-05T06:35:20.849 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.838+0000 7f634a7fc700 1 -- 172.21.15.42:0/4113587974 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1273185663 0 0) 0x7f634c0a0710 con 0x7f634c08df80 2019-12-05T06:35:20.849 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.838+0000 7f634a7fc700 1 -- 172.21.15.42:0/4113587974 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f634c07ef20 con 0x7f634c08df80 2019-12-05T06:35:20.849 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.838+0000 7f634a7fc700 1 -- 172.21.15.42:0/4113587974 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 632323972 0 0) 0x7f63340030b0 con 0x7f634c0a7740 2019-12-05T06:35:20.849 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.838+0000 7f634bfff700 1 -- 172.21.15.42:0/4113587974 >> v1:172.21.15.42:6789/0 conn(0x7f634c0a7740 legacy=0x7f634c07e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:35:20.850 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.838+0000 7f634bfff700 1 -- 172.21.15.42:0/4113587974 >> v1:172.21.15.40:6789/0 conn(0x7f634c08df80 legacy=0x7f634c09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:35:20.850 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.838+0000 7f634bfff700 1 -- 172.21.15.42:0/4113587974 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f634c0a9ba0 con 0x7f634c09fae0 2019-12-05T06:35:20.850 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.838+0000 7f634bfff700 1 --2- 172.21.15.42:0/4113587974 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f634c09fae0 0x7f634c14fff0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f6344008b10 tx=0x7f633c001950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:20.853 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.838+0000 7f634bfff700 1 --2- 172.21.15.42:0/4113587974 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f634c09fae0 0x7f634c14fff0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f6344008b10 tx=0x7f633c001950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:20.853 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.838+0000 7f634bfff700 1 --2- 172.21.15.42:0/4113587974 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f634c09fae0 0x7f634c14fff0 secure :-1 s=READY pgs=623 cs=0 l=1 rx=0x7f6344008b10 tx=0x7f633c001950).ready entity=mon.2 client_cookie=ce192ffb76d8f518 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:35:20.853 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.842+0000 7f634bfff700 1 --2- 172.21.15.42:0/4113587974 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f634c09fae0 0x7f634c14fff0 secure :-1 s=THROTTLE_DONE pgs=623 cs=0 l=1 rx=0x7f6344008b10 tx=0x7f633c001950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:20.853 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.842+0000 7f634a7fc700 1 -- 172.21.15.42:0/4113587974 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f633c001750 con 0x7f634c09fae0 2019-12-05T06:35:20.853 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.842+0000 7f634bfff700 1 --2- 172.21.15.42:0/4113587974 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f634c09fae0 0x7f634c14fff0 secure :-1 s=THROTTLE_DONE pgs=623 cs=0 l=1 rx=0x7f6344008b10 tx=0x7f633c001950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:20.853 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.842+0000 7f634a7fc700 1 -- 172.21.15.42:0/4113587974 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f633c001560 con 0x7f634c09fae0 2019-12-05T06:35:20.854 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.842+0000 7f634bfff700 1 --2- 172.21.15.42:0/4113587974 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f634c09fae0 0x7f634c14fff0 secure :-1 s=THROTTLE_DONE pgs=623 cs=0 l=1 rx=0x7f6344008b10 tx=0x7f633c001950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:20.854 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.842+0000 7f634a7fc700 1 -- 172.21.15.42:0/4113587974 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6344010a60 con 0x7f634c09fae0 2019-12-05T06:35:20.854 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.842+0000 7f63516b3700 1 -- 172.21.15.42:0/4113587974 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f634c09fae0 msgr2=0x7f634c14fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:35:20.854 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.842+0000 7f63516b3700 1 --2- 172.21.15.42:0/4113587974 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f634c09fae0 0x7f634c14fff0 secure :-1 s=READY pgs=623 cs=0 l=1 rx=0x7f6344008b10 tx=0x7f633c001950).stop 2019-12-05T06:35:20.854 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.842+0000 7f63516b3700 1 -- 172.21.15.42:0/4113587974 shutdown_connections 2019-12-05T06:35:20.854 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.842+0000 7f63516b3700 1 --2- 172.21.15.42:0/4113587974 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f634c09fae0 0x7f634c14fff0 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:35:20.854 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.842+0000 7f63516b3700 1 -- 172.21.15.42:0/4113587974 shutdown_connections 2019-12-05T06:35:20.855 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.842+0000 7f63516b3700 1 -- 172.21.15.42:0/4113587974 wait complete. 2019-12-05T06:35:20.855 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.842+0000 7f63516b3700 1 -- 172.21.15.42:0/4113587974 >> 172.21.15.42:0/4113587974 conn(0x7f634c0aabe0 msgr2=0x7f634c0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:35:20.858 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.846+0000 7f63516b3700 1 Processor -- start 2019-12-05T06:35:20.858 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.846+0000 7f63516b3700 1 -- start start 2019-12-05T06:35:20.859 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.846+0000 7f63516b3700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f634c117ec0 0x7f634c11ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:35:20.859 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.846+0000 7f63516b3700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f634c11bf90 0x7f634c13aae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:35:20.859 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.846+0000 7f63516b3700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f634c07dfe0 con 0x7f634c118d30 2019-12-05T06:35:20.859 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.846+0000 7f63516b3700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f634c118950 con 0x7f634c11bf90 2019-12-05T06:35:20.859 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.846+0000 7f63516b3700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f634c12d8a0 con 0x7f634c117ec0 2019-12-05T06:35:20.859 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.846+0000 7f634affd700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f634c117ec0 0x7f634c11ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:35:20.859 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.846+0000 7f634affd700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f634c117ec0 0x7f634c11ba40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:58138/0 (socket says 172.21.15.42:58138) 2019-12-05T06:35:20.860 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.846+0000 7f634affd700 1 -- 172.21.15.42:0/3016398709 learned_addr learned my addr 172.21.15.42:0/3016398709 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:35:20.860 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.846+0000 7f634bfff700 1 --2- 172.21.15.42:0/3016398709 >> v2:172.21.15.40:3300/0 conn(0x7f634c11bf90 0x7f634c13aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:35:20.860 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.846+0000 7f634affd700 1 -- 172.21.15.42:0/3016398709 >> v2:172.21.15.40:3300/0 conn(0x7f634c11bf90 msgr2=0x7f634c13aae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:35:20.860 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.846+0000 7f634affd700 1 --2- 172.21.15.42:0/3016398709 >> v2:172.21.15.40:3300/0 conn(0x7f634c11bf90 0x7f634c13aae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:35:20.860 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.846+0000 7f634affd700 1 -- 172.21.15.42:0/3016398709 >> v1:172.21.15.40:6789/0 conn(0x7f634c118d30 legacy=0x7f634c12d0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:35:20.860 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.846+0000 7f634affd700 1 -- 172.21.15.42:0/3016398709 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f634c11c3c0 con 0x7f634c117ec0 2019-12-05T06:35:20.860 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.846+0000 7f634affd700 1 --2- 172.21.15.42:0/3016398709 >> v2:172.21.15.42:3300/0 conn(0x7f634c117ec0 0x7f634c11ba40 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f6340000b80 tx=0x7f634000c340).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:20.861 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.846+0000 7f634affd700 1 --2- 172.21.15.42:0/3016398709 >> v2:172.21.15.42:3300/0 conn(0x7f634c117ec0 0x7f634c11ba40 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f6340000b80 tx=0x7f634000c340).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:20.861 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.846+0000 7f634affd700 1 --2- 172.21.15.42:0/3016398709 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f634c117ec0 0x7f634c11ba40 secure :-1 s=READY pgs=2820 cs=0 l=1 rx=0x7f6340000b80 tx=0x7f634000c340).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:35:20.861 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.846+0000 7f634affd700 1 --2- 172.21.15.42:0/3016398709 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f634c117ec0 0x7f634c11ba40 secure :-1 s=THROTTLE_DONE pgs=2820 cs=0 l=1 rx=0x7f6340000b80 tx=0x7f634000c340).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:20.861 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.846+0000 7f634affd700 1 --2- 172.21.15.42:0/3016398709 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f634c117ec0 0x7f634c11ba40 secure :-1 s=THROTTLE_DONE pgs=2820 cs=0 l=1 rx=0x7f6340000b80 tx=0x7f634000c340).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:20.861 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.846+0000 7f634affd700 1 --2- 172.21.15.42:0/3016398709 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f634c117ec0 0x7f634c11ba40 secure :-1 s=THROTTLE_DONE pgs=2820 cs=0 l=1 rx=0x7f6340000b80 tx=0x7f634000c340).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:20.861 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.846+0000 7f6333fff700 1 -- 172.21.15.42:0/3016398709 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f634000caa0 con 0x7f634c117ec0 2019-12-05T06:35:20.861 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.846+0000 7f6333fff700 1 -- 172.21.15.42:0/3016398709 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f634000c6c0 con 0x7f634c117ec0 2019-12-05T06:35:20.862 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.846+0000 7f6333fff700 1 -- 172.21.15.42:0/3016398709 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f634000d6a0 con 0x7f634c117ec0 2019-12-05T06:35:20.862 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.846+0000 7f63516b3700 1 -- 172.21.15.42:0/3016398709 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f634c118510 con 0x7f634c117ec0 2019-12-05T06:35:20.862 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.846+0000 7f63516b3700 1 -- 172.21.15.42:0/3016398709 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f634c11a740 con 0x7f634c117ec0 2019-12-05T06:35:20.862 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.850+0000 7f634affd700 1 --2- 172.21.15.42:0/3016398709 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f634c117ec0 0x7f634c11ba40 secure :-1 s=THROTTLE_DONE pgs=2820 cs=0 l=1 rx=0x7f6340000b80 tx=0x7f634000c340).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:20.862 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.850+0000 7f6333fff700 1 -- 172.21.15.42:0/3016398709 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f634000d9b0 con 0x7f634c117ec0 2019-12-05T06:35:20.862 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.850+0000 7f634affd700 1 --2- 172.21.15.42:0/3016398709 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f634c117ec0 0x7f634c11ba40 secure :-1 s=THROTTLE_DONE pgs=2820 cs=0 l=1 rx=0x7f6340000b80 tx=0x7f634000c340).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:20.862 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.850+0000 7f6333fff700 1 --2- 172.21.15.42:0/3016398709 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f632c03adb0 0x7f632c03d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:35:20.863 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.850+0000 7f6333fff700 1 -- 172.21.15.42:0/3016398709 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 11277+0+0 (secure 0 0 0) 0x7f634004b2a0 con 0x7f634c117ec0 2019-12-05T06:35:20.863 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.850+0000 7f634b7fe700 1 --2- 172.21.15.42:0/3016398709 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f632c03adb0 0x7f632c03d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:35:20.863 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.850+0000 7f634b7fe700 1 --2- 172.21.15.42:0/3016398709 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f632c03adb0 0x7f632c03d2c0 crc :-1 s=READY pgs=1187 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:35:20.863 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.854+0000 7f63516b3700 1 -- 172.21.15.42:0/3016398709 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f634c05d110 con 0x7f634c117ec0 2019-12-05T06:35:20.864 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.858+0000 7f634affd700 1 --2- 172.21.15.42:0/3016398709 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f634c117ec0 0x7f634c11ba40 secure :-1 s=THROTTLE_DONE pgs=2820 cs=0 l=1 rx=0x7f6340000b80 tx=0x7f634000c340).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:20.864 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:20.858+0000 7f6333fff700 1 -- 172.21.15.42:0/3016398709 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f63400322c0 con 0x7f634c117ec0 2019-12-05T06:35:21.027 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:21.022+0000 7f63516b3700 1 -- 172.21.15.42:0/3016398709 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f634c1182f0 con 0x7f634c117ec0 2019-12-05T06:35:21.028 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:21.022+0000 7f634affd700 1 --2- 172.21.15.42:0/3016398709 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f634c117ec0 0x7f634c11ba40 secure :-1 s=THROTTLE_DONE pgs=2820 cs=0 l=1 rx=0x7f6340000b80 tx=0x7f634000c340).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:21.028 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:21.022+0000 7f6333fff700 1 -- 172.21.15.42:0/3016398709 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 338 v338) v1 ==== 95+0+2116 (secure 0 0 0) 0x7f6340032520 con 0x7f634c117ec0 2019-12-05T06:35:21.028 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 338 2019-12-05T06:35:21.028 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:35:21.029 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":338,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7910,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/4253974783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":4253974783},{"type":"v1","addr":"172.21.15.40:6833","nonce":4253974783}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":338}],"filesystems":[{"mdsmap":{"epoch":337,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:33:30.171201+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":110,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11433},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11433":{"gid":11433,"name":"a","rank":0,"incarnation":334,"state":"up:active","state_seq":33,"addr":"172.21.15.42:6835/2532474042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2532474042},{"type":"v1","addr":"172.21.15.42:6835","nonce":2532474042}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:35:21.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:21.026+0000 7f63516b3700 1 -- 172.21.15.42:0/3016398709 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f632c03adb0 msgr2=0x7f632c03d2c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:35:21.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:21.026+0000 7f63516b3700 1 --2- 172.21.15.42:0/3016398709 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f632c03adb0 0x7f632c03d2c0 crc :-1 s=READY pgs=1187 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:35:21.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:21.026+0000 7f63516b3700 1 -- 172.21.15.42:0/3016398709 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f634c117ec0 msgr2=0x7f634c11ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:35:21.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:21.026+0000 7f63516b3700 1 --2- 172.21.15.42:0/3016398709 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f634c117ec0 0x7f634c11ba40 secure :-1 s=READY pgs=2820 cs=0 l=1 rx=0x7f6340000b80 tx=0x7f634000c340).stop 2019-12-05T06:35:21.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:21.026+0000 7f63516b3700 1 -- 172.21.15.42:0/3016398709 shutdown_connections 2019-12-05T06:35:21.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:21.026+0000 7f63516b3700 1 --2- 172.21.15.42:0/3016398709 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f632c03adb0 0x7f632c03d2c0 unknown :-1 s=CLOSED pgs=1187 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:35:21.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:21.026+0000 7f63516b3700 1 --2- 172.21.15.42:0/3016398709 >> v2:172.21.15.40:3300/0 conn(0x7f634c11bf90 0x7f634c13aae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:35:21.032 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:21.026+0000 7f63516b3700 1 --2- 172.21.15.42:0/3016398709 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f634c117ec0 0x7f634c11ba40 unknown :-1 s=CLOSED pgs=2820 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:35:21.032 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:21.026+0000 7f63516b3700 1 -- 172.21.15.42:0/3016398709 shutdown_connections 2019-12-05T06:35:21.032 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:21.026+0000 7f63516b3700 1 -- 172.21.15.42:0/3016398709 wait complete. 2019-12-05T06:35:21.032 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:21.026+0000 7f63516b3700 1 -- 172.21.15.42:0/3016398709 >> 172.21.15.42:0/3016398709 conn(0x7f634c0aabe0 msgr2=0x7f634c09f340 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:35:21.041 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:35:23.041 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:35:23.147 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:23.138+0000 7f7509cfb700 1 Processor -- start 2019-12-05T06:35:23.147 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:23.138+0000 7f7509cfb700 1 -- start start 2019-12-05T06:35:23.147 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:23.138+0000 7f7509cfb700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f750408df80 0x7f750409f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:35:23.147 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:23.138+0000 7f7509cfb700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f750409f830 0x7f750414efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:35:23.147 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:23.138+0000 7f7509cfb700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f750409fc60 con 0x7f75040a7740 2019-12-05T06:35:23.148 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:23.138+0000 7f7509cfb700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75040a0b00 con 0x7f750408df80 2019-12-05T06:35:23.148 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:23.138+0000 7f7509cfb700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f750408e5b0 con 0x7f750409f830 2019-12-05T06:35:23.148 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:23.138+0000 7f7508a99700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f750409f830 0x7f750414efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:35:23.148 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:23.138+0000 7f7508a99700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f750409f830 0x7f750414efe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:58162/0 (socket says 172.21.15.42:58162) 2019-12-05T06:35:23.148 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:23.138+0000 7f7503fff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f75040a7740 0x7f750407e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:58196/0 (socket says 172.21.15.42:58196) 2019-12-05T06:35:23.148 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:23.138+0000 7f7508a99700 1 -- 172.21.15.42:0/3960066636 learned_addr learned my addr 172.21.15.42:0/3960066636 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:35:23.148 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:23.138+0000 7f74fb7fe700 1 --2- 172.21.15.42:0/3960066636 >> v2:172.21.15.40:3300/0 conn(0x7f750408df80 0x7f750409f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:35:23.149 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:23.138+0000 7f7508a99700 1 -- 172.21.15.42:0/3960066636 >> v2:172.21.15.40:3300/0 conn(0x7f750408df80 msgr2=0x7f750409f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:35:23.149 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:23.138+0000 7f7508a99700 1 --2- 172.21.15.42:0/3960066636 >> v2:172.21.15.40:3300/0 conn(0x7f750408df80 0x7f750409f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:35:23.149 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:23.138+0000 7f7508a99700 1 -- 172.21.15.42:0/3960066636 >> v1:172.21.15.42:6789/0 conn(0x7f75040a7740 legacy=0x7f750407e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:35:23.149 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:23.138+0000 7f75037fe700 1 -- 172.21.15.42:0/3960066636 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4056120627 0 0) 0x7f750409fc60 con 0x7f75040a7740 2019-12-05T06:35:23.149 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:23.138+0000 7f7508a99700 1 -- 172.21.15.42:0/3960066636 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75040a0710 con 0x7f750409f830 2019-12-05T06:35:23.149 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:23.138+0000 7f74fb7fe700 1 --2- 172.21.15.42:0/3960066636 >> v2:172.21.15.40:3300/0 conn(0x7f750408df80 0x7f750409f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:35:23.155 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:23.138+0000 7f7508a99700 1 --2- 172.21.15.42:0/3960066636 >> v2:172.21.15.42:3300/0 conn(0x7f750409f830 0x7f750414efe0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f74fc0032b0 tx=0x7f74fc0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:23.155 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:23.142+0000 7f7508a99700 1 --2- 172.21.15.42:0/3960066636 >> v2:172.21.15.42:3300/0 conn(0x7f750409f830 0x7f750414efe0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f74fc0032b0 tx=0x7f74fc0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:23.155 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:23.142+0000 7f7508a99700 1 --2- 172.21.15.42:0/3960066636 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f750409f830 0x7f750414efe0 secure :-1 s=READY pgs=2822 cs=0 l=1 rx=0x7f74fc0032b0 tx=0x7f74fc0100b0).ready entity=mon.1 client_cookie=2fc6f6211f76bcd8 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:35:23.155 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:23.142+0000 7f7508a99700 1 --2- 172.21.15.42:0/3960066636 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f750409f830 0x7f750414efe0 secure :-1 s=THROTTLE_DONE pgs=2822 cs=0 l=1 rx=0x7f74fc0032b0 tx=0x7f74fc0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:23.155 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:23.142+0000 7f75037fe700 1 -- 172.21.15.42:0/3960066636 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f74fc010d60 con 0x7f750409f830 2019-12-05T06:35:23.155 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:23.142+0000 7f7508a99700 1 --2- 172.21.15.42:0/3960066636 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f750409f830 0x7f750414efe0 secure :-1 s=THROTTLE_DONE pgs=2822 cs=0 l=1 rx=0x7f74fc0032b0 tx=0x7f74fc0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:23.156 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:23.142+0000 7f7508a99700 1 --2- 172.21.15.42:0/3960066636 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f750409f830 0x7f750414efe0 secure :-1 s=THROTTLE_DONE pgs=2822 cs=0 l=1 rx=0x7f74fc0032b0 tx=0x7f74fc0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:23.156 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:23.142+0000 7f75037fe700 1 -- 172.21.15.42:0/3960066636 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74fc010980 con 0x7f750409f830 2019-12-05T06:35:23.156 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:23.142+0000 7f75037fe700 1 -- 172.21.15.42:0/3960066636 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f74fc0114a0 con 0x7f750409f830 2019-12-05T06:35:23.156 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:23.142+0000 7f7509cfb700 1 -- 172.21.15.42:0/3960066636 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f750409f830 msgr2=0x7f750414efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:35:23.156 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:23.142+0000 7f7509cfb700 1 --2- 172.21.15.42:0/3960066636 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f750409f830 0x7f750414efe0 secure :-1 s=READY pgs=2822 cs=0 l=1 rx=0x7f74fc0032b0 tx=0x7f74fc0100b0).stop 2019-12-05T06:35:23.156 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:23.142+0000 7f7509cfb700 1 -- 172.21.15.42:0/3960066636 shutdown_connections 2019-12-05T06:35:23.156 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:23.142+0000 7f7509cfb700 1 --2- 172.21.15.42:0/3960066636 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f750409f830 0x7f750414efe0 unknown :-1 s=CLOSED pgs=2822 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:35:23.157 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:23.142+0000 7f7509cfb700 1 --2- 172.21.15.42:0/3960066636 >> v2:172.21.15.40:3300/0 conn(0x7f750408df80 0x7f750409f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:35:23.157 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:23.142+0000 7f7509cfb700 1 -- 172.21.15.42:0/3960066636 shutdown_connections 2019-12-05T06:35:23.157 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:23.142+0000 7f7509cfb700 1 -- 172.21.15.42:0/3960066636 wait complete. 2019-12-05T06:35:23.157 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:23.142+0000 7f7509cfb700 1 -- 172.21.15.42:0/3960066636 >> 172.21.15.42:0/3960066636 conn(0x7f75040aabe0 msgr2=0x7f75040a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:35:23.161 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:23.146+0000 7f7509cfb700 1 Processor -- start 2019-12-05T06:35:23.161 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:23.146+0000 7f7509cfb700 1 -- start start 2019-12-05T06:35:23.161 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:23.146+0000 7f7509cfb700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f750407ce70 0x7f750407d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:35:23.161 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:23.146+0000 7f7509cfb700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f75041357d0 0x7f7504137c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:35:23.161 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:23.146+0000 7f7509cfb700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75040a2910 con 0x7f750411bac0 2019-12-05T06:35:23.161 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:23.146+0000 7f7509cfb700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f750407c930 con 0x7f750407ce70 2019-12-05T06:35:23.162 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:23.146+0000 7f7509cfb700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f750407cbd0 con 0x7f75041357d0 2019-12-05T06:35:23.162 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:23.146+0000 7f7503fff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f750411bac0 0x7f75040a1ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:58202/0 (socket says 172.21.15.42:58202) 2019-12-05T06:35:23.162 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:23.146+0000 7f7503fff700 1 -- 172.21.15.42:0/648862053 learned_addr learned my addr 172.21.15.42:0/648862053 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:35:23.162 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:23.146+0000 7f7508a99700 1 --2- 172.21.15.42:0/648862053 >> v2:172.21.15.42:3300/0 conn(0x7f75041357d0 0x7f7504137c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:35:23.163 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:23.146+0000 7f74fb7fe700 1 --2- 172.21.15.42:0/648862053 >> v2:172.21.15.40:3300/0 conn(0x7f750407ce70 0x7f750407d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:35:23.164 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:23.146+0000 7f7508a99700 1 -- 172.21.15.42:0/648862053 >> v2:172.21.15.40:3300/0 conn(0x7f750407ce70 msgr2=0x7f750407d2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:35:23.164 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:23.146+0000 7f7508a99700 1 --2- 172.21.15.42:0/648862053 >> v2:172.21.15.40:3300/0 conn(0x7f750407ce70 0x7f750407d2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:35:23.164 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:23.146+0000 7f7508a99700 1 -- 172.21.15.42:0/648862053 >> v1:172.21.15.42:6789/0 conn(0x7f750411bac0 legacy=0x7f75040a1ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:35:23.164 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:23.146+0000 7f75017fa700 1 -- 172.21.15.42:0/648862053 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1069670410 0 0) 0x7f75040a2910 con 0x7f750411bac0 2019-12-05T06:35:23.164 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:23.146+0000 7f7508a99700 1 -- 172.21.15.42:0/648862053 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75040a6db0 con 0x7f75041357d0 2019-12-05T06:35:23.164 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:23.146+0000 7f74fb7fe700 1 --2- 172.21.15.42:0/648862053 >> v2:172.21.15.40:3300/0 conn(0x7f750407ce70 0x7f750407d2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:35:23.165 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:23.146+0000 7f7508a99700 1 --2- 172.21.15.42:0/648862053 >> v2:172.21.15.42:3300/0 conn(0x7f75041357d0 0x7f7504137c10 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f74fc0032b0 tx=0x7f74fc010cb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:23.165 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:23.146+0000 7f7508a99700 1 --2- 172.21.15.42:0/648862053 >> v2:172.21.15.42:3300/0 conn(0x7f75041357d0 0x7f7504137c10 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f74fc0032b0 tx=0x7f74fc010cb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:23.165 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:23.146+0000 7f7508a99700 1 --2- 172.21.15.42:0/648862053 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f75041357d0 0x7f7504137c10 secure :-1 s=READY pgs=2824 cs=0 l=1 rx=0x7f74fc0032b0 tx=0x7f74fc010cb0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:35:23.165 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:23.146+0000 7f7508a99700 1 --2- 172.21.15.42:0/648862053 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f75041357d0 0x7f7504137c10 secure :-1 s=THROTTLE_DONE pgs=2824 cs=0 l=1 rx=0x7f74fc0032b0 tx=0x7f74fc010cb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:23.165 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:23.146+0000 7f75017fa700 1 -- 172.21.15.42:0/648862053 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f74fc011bb0 con 0x7f75041357d0 2019-12-05T06:35:23.166 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:23.146+0000 7f7508a99700 1 --2- 172.21.15.42:0/648862053 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f75041357d0 0x7f7504137c10 secure :-1 s=THROTTLE_DONE pgs=2824 cs=0 l=1 rx=0x7f74fc0032b0 tx=0x7f74fc010cb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:23.166 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:23.146+0000 7f7508a99700 1 --2- 172.21.15.42:0/648862053 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f75041357d0 0x7f7504137c10 secure :-1 s=THROTTLE_DONE pgs=2824 cs=0 l=1 rx=0x7f74fc0032b0 tx=0x7f74fc010cb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:23.166 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:23.146+0000 7f75017fa700 1 -- 172.21.15.42:0/648862053 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74fc0110c0 con 0x7f75041357d0 2019-12-05T06:35:23.166 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:23.146+0000 7f7509cfb700 1 -- 172.21.15.42:0/648862053 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75041310d0 con 0x7f75041357d0 2019-12-05T06:35:23.166 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:23.146+0000 7f75017fa700 1 -- 172.21.15.42:0/648862053 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f74fc005ab0 con 0x7f75041357d0 2019-12-05T06:35:23.166 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:23.146+0000 7f7509cfb700 1 -- 172.21.15.42:0/648862053 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7504130390 con 0x7f75041357d0 2019-12-05T06:35:23.166 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:23.150+0000 7f7508a99700 1 --2- 172.21.15.42:0/648862053 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f75041357d0 0x7f7504137c10 secure :-1 s=THROTTLE_DONE pgs=2824 cs=0 l=1 rx=0x7f74fc0032b0 tx=0x7f74fc010cb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:23.167 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:23.150+0000 7f75017fa700 1 -- 172.21.15.42:0/648862053 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f74fc00f280 con 0x7f75041357d0 2019-12-05T06:35:23.167 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:23.150+0000 7f7508a99700 1 --2- 172.21.15.42:0/648862053 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f75041357d0 0x7f7504137c10 secure :-1 s=THROTTLE_DONE pgs=2824 cs=0 l=1 rx=0x7f74fc0032b0 tx=0x7f74fc010cb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:23.167 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:23.150+0000 7f75017fa700 1 --2- 172.21.15.42:0/648862053 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f74e803ce40 0x7f74e803f350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:35:23.167 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:23.150+0000 7f75017fa700 1 -- 172.21.15.42:0/648862053 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 11277+0+0 (secure 0 0 0) 0x7f74fc04b290 con 0x7f75041357d0 2019-12-05T06:35:23.167 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:23.150+0000 7f7503fff700 1 --2- 172.21.15.42:0/648862053 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f74e803ce40 0x7f74e803f350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:35:23.167 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:23.150+0000 7f7503fff700 1 --2- 172.21.15.42:0/648862053 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f74e803ce40 0x7f74e803f350 crc :-1 s=READY pgs=1188 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:35:23.167 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:23.154+0000 7f7509cfb700 1 -- 172.21.15.42:0/648862053 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f750405d110 con 0x7f75041357d0 2019-12-05T06:35:23.168 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:23.158+0000 7f7508a99700 1 --2- 172.21.15.42:0/648862053 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f75041357d0 0x7f7504137c10 secure :-1 s=THROTTLE_DONE pgs=2824 cs=0 l=1 rx=0x7f74fc0032b0 tx=0x7f74fc010cb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:23.168 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:23.158+0000 7f75017fa700 1 -- 172.21.15.42:0/648862053 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f74fc005d80 con 0x7f75041357d0 2019-12-05T06:35:23.333 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:23.330+0000 7f7509cfb700 1 -- 172.21.15.42:0/648862053 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f750405b8c0 con 0x7f75041357d0 2019-12-05T06:35:23.334 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:23.330+0000 7f7508a99700 1 --2- 172.21.15.42:0/648862053 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f75041357d0 0x7f7504137c10 secure :-1 s=THROTTLE_DONE pgs=2824 cs=0 l=1 rx=0x7f74fc0032b0 tx=0x7f74fc010cb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:23.334 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:23.330+0000 7f75017fa700 1 -- 172.21.15.42:0/648862053 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 338 v338) v1 ==== 95+0+2116 (secure 0 0 0) 0x7f74fc034d80 con 0x7f75041357d0 2019-12-05T06:35:23.334 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 338 2019-12-05T06:35:23.334 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:35:23.334 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":338,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7910,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/4253974783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":4253974783},{"type":"v1","addr":"172.21.15.40:6833","nonce":4253974783}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":338}],"filesystems":[{"mdsmap":{"epoch":337,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:33:30.171201+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":110,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11433},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11433":{"gid":11433,"name":"a","rank":0,"incarnation":334,"state":"up:active","state_seq":33,"addr":"172.21.15.42:6835/2532474042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2532474042},{"type":"v1","addr":"172.21.15.42:6835","nonce":2532474042}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:35:23.336 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:23.330+0000 7f7509cfb700 1 -- 172.21.15.42:0/648862053 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f74e803ce40 msgr2=0x7f74e803f350 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:35:23.337 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:23.330+0000 7f7509cfb700 1 --2- 172.21.15.42:0/648862053 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f74e803ce40 0x7f74e803f350 crc :-1 s=READY pgs=1188 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:35:23.337 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:23.330+0000 7f7509cfb700 1 -- 172.21.15.42:0/648862053 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f75041357d0 msgr2=0x7f7504137c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:35:23.337 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:23.330+0000 7f7509cfb700 1 --2- 172.21.15.42:0/648862053 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f75041357d0 0x7f7504137c10 secure :-1 s=READY pgs=2824 cs=0 l=1 rx=0x7f74fc0032b0 tx=0x7f74fc010cb0).stop 2019-12-05T06:35:23.337 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:23.334+0000 7f7509cfb700 1 -- 172.21.15.42:0/648862053 shutdown_connections 2019-12-05T06:35:23.337 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:23.334+0000 7f7509cfb700 1 --2- 172.21.15.42:0/648862053 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f74e803ce40 0x7f74e803f350 unknown :-1 s=CLOSED pgs=1188 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:35:23.337 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:23.334+0000 7f7509cfb700 1 --2- 172.21.15.42:0/648862053 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f75041357d0 0x7f7504137c10 unknown :-1 s=CLOSED pgs=2824 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:35:23.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:23.334+0000 7f7509cfb700 1 --2- 172.21.15.42:0/648862053 >> v2:172.21.15.40:3300/0 conn(0x7f750407ce70 0x7f750407d2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:35:23.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:23.334+0000 7f7509cfb700 1 -- 172.21.15.42:0/648862053 shutdown_connections 2019-12-05T06:35:23.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:23.334+0000 7f7509cfb700 1 -- 172.21.15.42:0/648862053 wait complete. 2019-12-05T06:35:23.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:23.334+0000 7f7509cfb700 1 -- 172.21.15.42:0/648862053 >> 172.21.15.42:0/648862053 conn(0x7f750411d360 msgr2=0x7f750407e6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:35:23.347 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:35:25.348 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:35:25.461 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:25.450+0000 7f9367b78700 1 Processor -- start 2019-12-05T06:35:25.461 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:25.450+0000 7f9367b78700 1 -- start start 2019-12-05T06:35:25.461 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:25.454+0000 7f9367b78700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f93600a7740 0x7f936008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:35:25.462 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:25.454+0000 7f9367b78700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f936007e720 0x7f936007eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:35:25.462 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:25.454+0000 7f9367b78700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f936009f2e0 0x7f936009f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:35:25.462 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:25.454+0000 7f9367b78700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93600a0b00 con 0x7f93600a7740 2019-12-05T06:35:25.462 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:25.454+0000 7f9367b78700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f936008e5b0 con 0x7f936007e720 2019-12-05T06:35:25.462 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:25.454+0000 7f9367b78700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f936008e850 con 0x7f936009f2e0 2019-12-05T06:35:25.462 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:25.454+0000 7f9365914700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f936007e720 0x7f936007eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:35:25.462 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:25.454+0000 7f9365914700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f936007e720 0x7f936007eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:58174/0 (socket says 172.21.15.42:58174) 2019-12-05T06:35:25.463 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:25.454+0000 7f9365914700 1 -- 172.21.15.42:0/1902182536 learned_addr learned my addr 172.21.15.42:0/1902182536 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:35:25.463 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:25.454+0000 7f9366916700 1 --2- 172.21.15.42:0/1902182536 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f936009f2e0 0x7f936009f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:35:25.463 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:25.454+0000 7f9366115700 1 --2- 172.21.15.42:0/1902182536 >> v2:172.21.15.40:3300/0 conn(0x7f93600a7740 0x7f936008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:35:25.463 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:25.454+0000 7f9365914700 1 -- 172.21.15.42:0/1902182536 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f936009f2e0 msgr2=0x7f936009f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:35:25.463 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:25.454+0000 7f9365914700 1 --2- 172.21.15.42:0/1902182536 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f936009f2e0 0x7f936009f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:35:25.463 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:25.454+0000 7f9365914700 1 -- 172.21.15.42:0/1902182536 >> v2:172.21.15.40:3300/0 conn(0x7f93600a7740 msgr2=0x7f936008df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:35:25.464 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:25.454+0000 7f9365914700 1 --2- 172.21.15.42:0/1902182536 >> v2:172.21.15.40:3300/0 conn(0x7f93600a7740 0x7f936008df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:35:25.464 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:25.454+0000 7f9365914700 1 -- 172.21.15.42:0/1902182536 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f936007f0a0 con 0x7f936007e720 2019-12-05T06:35:25.464 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:25.454+0000 7f9365914700 1 --2- 172.21.15.42:0/1902182536 >> v2:172.21.15.42:3300/0 conn(0x7f936007e720 0x7f936007eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f93600a0f50 tx=0x7f935400fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:25.464 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:25.454+0000 7f9365914700 1 --2- 172.21.15.42:0/1902182536 >> v2:172.21.15.42:3300/0 conn(0x7f936007e720 0x7f936007eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f93600a0f50 tx=0x7f935400fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:25.464 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:25.454+0000 7f9365914700 1 --2- 172.21.15.42:0/1902182536 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f936007e720 0x7f936007eb50 secure :-1 s=READY pgs=2825 cs=0 l=1 rx=0x7f93600a0f50 tx=0x7f935400fdb0).ready entity=mon.1 client_cookie=cef2c65f8db39adf server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:35:25.464 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:25.454+0000 7f9365914700 1 --2- 172.21.15.42:0/1902182536 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f936007e720 0x7f936007eb50 secure :-1 s=THROTTLE_DONE pgs=2825 cs=0 l=1 rx=0x7f93600a0f50 tx=0x7f935400fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:25.464 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:25.454+0000 7f9365113700 1 -- 172.21.15.42:0/1902182536 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9354008c90 con 0x7f936007e720 2019-12-05T06:35:25.465 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:25.454+0000 7f9365914700 1 --2- 172.21.15.42:0/1902182536 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f936007e720 0x7f936007eb50 secure :-1 s=THROTTLE_DONE pgs=2825 cs=0 l=1 rx=0x7f93600a0f50 tx=0x7f935400fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:25.465 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:25.454+0000 7f9365914700 1 --2- 172.21.15.42:0/1902182536 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f936007e720 0x7f936007eb50 secure :-1 s=THROTTLE_DONE pgs=2825 cs=0 l=1 rx=0x7f93600a0f50 tx=0x7f935400fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:25.465 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:25.454+0000 7f9365113700 1 -- 172.21.15.42:0/1902182536 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93540088b0 con 0x7f936007e720 2019-12-05T06:35:25.465 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:25.454+0000 7f9365113700 1 -- 172.21.15.42:0/1902182536 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9354011b50 con 0x7f936007e720 2019-12-05T06:35:25.465 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:25.454+0000 7f9367b78700 1 -- 172.21.15.42:0/1902182536 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f936007e720 msgr2=0x7f936007eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:35:25.465 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:25.454+0000 7f9367b78700 1 --2- 172.21.15.42:0/1902182536 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f936007e720 0x7f936007eb50 secure :-1 s=READY pgs=2825 cs=0 l=1 rx=0x7f93600a0f50 tx=0x7f935400fdb0).stop 2019-12-05T06:35:25.466 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:25.458+0000 7f9367b78700 1 -- 172.21.15.42:0/1902182536 shutdown_connections 2019-12-05T06:35:25.467 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:25.458+0000 7f9367b78700 1 --2- 172.21.15.42:0/1902182536 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f936009f2e0 0x7f936009f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:35:25.467 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:25.458+0000 7f9367b78700 1 --2- 172.21.15.42:0/1902182536 >> v2:172.21.15.40:3300/0 conn(0x7f93600a7740 0x7f936008df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:35:25.468 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:25.458+0000 7f9367b78700 1 --2- 172.21.15.42:0/1902182536 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f936007e720 0x7f936007eb50 unknown :-1 s=CLOSED pgs=2825 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:35:25.469 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:25.458+0000 7f9367b78700 1 -- 172.21.15.42:0/1902182536 shutdown_connections 2019-12-05T06:35:25.469 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:25.458+0000 7f9367b78700 1 -- 172.21.15.42:0/1902182536 wait complete. 2019-12-05T06:35:25.469 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:25.458+0000 7f9367b78700 1 -- 172.21.15.42:0/1902182536 >> 172.21.15.42:0/1902182536 conn(0x7f93600aabe0 msgr2=0x7f93600a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:35:25.472 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:25.458+0000 7f9367b78700 1 Processor -- start 2019-12-05T06:35:25.473 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:25.458+0000 7f9367b78700 1 -- start start 2019-12-05T06:35:25.473 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:25.458+0000 7f9367b78700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f936007c050 0x7f936008ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:35:25.473 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:25.458+0000 7f9367b78700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f936011a220 con 0x7f9360121300 2019-12-05T06:35:25.473 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:25.458+0000 7f9367b78700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f936011ae00 con 0x7f936008f4e0 2019-12-05T06:35:25.473 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:25.458+0000 7f9367b78700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f936011b8a0 con 0x7f936007c050 2019-12-05T06:35:25.473 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:25.458+0000 7f9365914700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f936008f4e0 0x7f9360120b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:58218/0 (socket says 172.21.15.42:58218) 2019-12-05T06:35:25.474 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:25.458+0000 7f9365914700 1 -- 172.21.15.42:0/2598495544 learned_addr learned my addr 172.21.15.42:0/2598495544 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:35:25.474 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:25.458+0000 7f9366115700 1 --2- 172.21.15.42:0/2598495544 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f936007c050 0x7f936008ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:35:25.474 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:25.462+0000 7f9352ffd700 1 -- 172.21.15.42:0/2598495544 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2792998059 0 0) 0x7f936011ae00 con 0x7f936008f4e0 2019-12-05T06:35:25.474 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:25.462+0000 7f9352ffd700 1 -- 172.21.15.42:0/2598495544 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93480030b0 con 0x7f936008f4e0 2019-12-05T06:35:25.474 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:25.462+0000 7f9366115700 1 -- 172.21.15.42:0/2598495544 >> v1:172.21.15.42:6789/0 conn(0x7f936008f4e0 legacy=0x7f9360120b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:35:25.474 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:25.462+0000 7f9366115700 1 -- 172.21.15.42:0/2598495544 >> v1:172.21.15.40:6789/0 conn(0x7f9360121300 legacy=0x7f936012d7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:35:25.474 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:25.462+0000 7f9352ffd700 1 -- 172.21.15.42:0/2598495544 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1223836908 0 0) 0x7f936011a220 con 0x7f9360121300 2019-12-05T06:35:25.475 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:25.462+0000 7f9366115700 1 -- 172.21.15.42:0/2598495544 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f936008f910 con 0x7f936007c050 2019-12-05T06:35:25.475 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:25.462+0000 7f9366115700 1 --2- 172.21.15.42:0/2598495544 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f936007c050 0x7f936008ef90 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f935c002f30 tx=0x7f935c010990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:25.475 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:25.462+0000 7f9366115700 1 --2- 172.21.15.42:0/2598495544 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f936007c050 0x7f936008ef90 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f935c002f30 tx=0x7f935c010990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:25.475 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:25.462+0000 7f9366115700 1 --2- 172.21.15.42:0/2598495544 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f936007c050 0x7f936008ef90 secure :-1 s=READY pgs=624 cs=0 l=1 rx=0x7f935c002f30 tx=0x7f935c010990).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:35:25.475 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:25.462+0000 7f9366115700 1 --2- 172.21.15.42:0/2598495544 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f936007c050 0x7f936008ef90 secure :-1 s=THROTTLE_DONE pgs=624 cs=0 l=1 rx=0x7f935c002f30 tx=0x7f935c010990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:25.475 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:25.462+0000 7f9352ffd700 1 -- 172.21.15.42:0/2598495544 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f935c0016b0 con 0x7f936007c050 2019-12-05T06:35:25.475 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:25.462+0000 7f9366115700 1 --2- 172.21.15.42:0/2598495544 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f936007c050 0x7f936008ef90 secure :-1 s=THROTTLE_DONE pgs=624 cs=0 l=1 rx=0x7f935c002f30 tx=0x7f935c010990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:25.476 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:25.462+0000 7f9352ffd700 1 -- 172.21.15.42:0/2598495544 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f935c0012d0 con 0x7f936007c050 2019-12-05T06:35:25.476 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:25.462+0000 7f9366115700 1 --2- 172.21.15.42:0/2598495544 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f936007c050 0x7f936008ef90 secure :-1 s=THROTTLE_DONE pgs=624 cs=0 l=1 rx=0x7f935c002f30 tx=0x7f935c010990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:25.476 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:25.462+0000 7f9367b78700 1 -- 172.21.15.42:0/2598495544 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9360123bf0 con 0x7f936007c050 2019-12-05T06:35:25.476 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:25.462+0000 7f9352ffd700 1 -- 172.21.15.42:0/2598495544 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f935c001c90 con 0x7f936007c050 2019-12-05T06:35:25.476 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:25.462+0000 7f9367b78700 1 -- 172.21.15.42:0/2598495544 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f936007c480 con 0x7f936007c050 2019-12-05T06:35:25.484 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:25.462+0000 7f9366115700 1 --2- 172.21.15.42:0/2598495544 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f936007c050 0x7f936008ef90 secure :-1 s=THROTTLE_DONE pgs=624 cs=0 l=1 rx=0x7f935c002f30 tx=0x7f935c010990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:25.484 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:25.462+0000 7f9352ffd700 1 -- 172.21.15.42:0/2598495544 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f935c00d1b0 con 0x7f936007c050 2019-12-05T06:35:25.484 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:25.462+0000 7f9366115700 1 --2- 172.21.15.42:0/2598495544 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f936007c050 0x7f936008ef90 secure :-1 s=THROTTLE_DONE pgs=624 cs=0 l=1 rx=0x7f935c002f30 tx=0x7f935c010990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:25.485 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:25.462+0000 7f9352ffd700 1 --2- 172.21.15.42:0/2598495544 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f934803b1e0 0x7f934803d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:35:25.485 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:25.462+0000 7f9352ffd700 1 -- 172.21.15.42:0/2598495544 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 11277+0+0 (secure 0 0 0) 0x7f935c04aaf0 con 0x7f936007c050 2019-12-05T06:35:25.485 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:25.462+0000 7f9365914700 1 --2- 172.21.15.42:0/2598495544 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f934803b1e0 0x7f934803d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:35:25.485 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:25.466+0000 7f9365914700 1 --2- 172.21.15.42:0/2598495544 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f934803b1e0 0x7f934803d6f0 crc :-1 s=READY pgs=1189 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:35:25.485 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:25.466+0000 7f9367b78700 1 -- 172.21.15.42:0/2598495544 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f936005d110 con 0x7f936007c050 2019-12-05T06:35:25.486 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:25.478+0000 7f9366115700 1 --2- 172.21.15.42:0/2598495544 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f936007c050 0x7f936008ef90 secure :-1 s=THROTTLE_DONE pgs=624 cs=0 l=1 rx=0x7f935c002f30 tx=0x7f935c010990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:25.486 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:25.478+0000 7f9352ffd700 1 -- 172.21.15.42:0/2598495544 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f935c031ad0 con 0x7f936007c050 2019-12-05T06:35:25.658 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:25.654+0000 7f9367b78700 1 -- 172.21.15.42:0/2598495544 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f936005b8c0 con 0x7f936007c050 2019-12-05T06:35:25.659 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:25.654+0000 7f9366115700 1 --2- 172.21.15.42:0/2598495544 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f936007c050 0x7f936008ef90 secure :-1 s=THROTTLE_DONE pgs=624 cs=0 l=1 rx=0x7f935c002f30 tx=0x7f935c010990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:25.659 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:25.654+0000 7f9352ffd700 1 -- 172.21.15.42:0/2598495544 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 338 v338) v1 ==== 95+0+2116 (secure 0 0 0) 0x7f935c031d30 con 0x7f936007c050 2019-12-05T06:35:25.661 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 338 2019-12-05T06:35:25.661 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:35:25.661 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":338,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7910,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/4253974783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":4253974783},{"type":"v1","addr":"172.21.15.40:6833","nonce":4253974783}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":338}],"filesystems":[{"mdsmap":{"epoch":337,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:33:30.171201+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":110,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11433},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11433":{"gid":11433,"name":"a","rank":0,"incarnation":334,"state":"up:active","state_seq":33,"addr":"172.21.15.42:6835/2532474042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2532474042},{"type":"v1","addr":"172.21.15.42:6835","nonce":2532474042}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:35:25.664 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:25.658+0000 7f9367b78700 1 -- 172.21.15.42:0/2598495544 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f934803b1e0 msgr2=0x7f934803d6f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:35:25.664 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:25.658+0000 7f9367b78700 1 --2- 172.21.15.42:0/2598495544 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f934803b1e0 0x7f934803d6f0 crc :-1 s=READY pgs=1189 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:35:25.664 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:25.658+0000 7f9367b78700 1 -- 172.21.15.42:0/2598495544 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f936007c050 msgr2=0x7f936008ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:35:25.664 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:25.658+0000 7f9367b78700 1 --2- 172.21.15.42:0/2598495544 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f936007c050 0x7f936008ef90 secure :-1 s=READY pgs=624 cs=0 l=1 rx=0x7f935c002f30 tx=0x7f935c010990).stop 2019-12-05T06:35:25.664 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:25.658+0000 7f9367b78700 1 -- 172.21.15.42:0/2598495544 shutdown_connections 2019-12-05T06:35:25.672 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:25.658+0000 7f9367b78700 1 --2- 172.21.15.42:0/2598495544 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f934803b1e0 0x7f934803d6f0 unknown :-1 s=CLOSED pgs=1189 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:35:25.672 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:25.658+0000 7f9367b78700 1 --2- 172.21.15.42:0/2598495544 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f936007c050 0x7f936008ef90 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:35:25.673 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:25.658+0000 7f9367b78700 1 -- 172.21.15.42:0/2598495544 shutdown_connections 2019-12-05T06:35:25.673 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:25.658+0000 7f9367b78700 1 -- 172.21.15.42:0/2598495544 wait complete. 2019-12-05T06:35:25.674 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:25.658+0000 7f9367b78700 1 -- 172.21.15.42:0/2598495544 >> 172.21.15.42:0/2598495544 conn(0x7f936008fda0 msgr2=0x7f9360059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:35:25.676 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:35:27.676 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:35:27.793 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:27.782+0000 7ff036f3e700 1 Processor -- start 2019-12-05T06:35:27.793 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:27.782+0000 7ff036f3e700 1 -- start start 2019-12-05T06:35:27.793 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:27.786+0000 7ff036f3e700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff0300a7740 0x7ff03008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:35:27.793 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:27.786+0000 7ff036f3e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff03007e720 0x7ff03007eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:35:27.793 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:27.786+0000 7ff036f3e700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0300a0710 con 0x7ff03009f2e0 2019-12-05T06:35:27.793 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:27.786+0000 7ff036f3e700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0300a0b00 con 0x7ff03007e720 2019-12-05T06:35:27.794 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:27.786+0000 7ff036f3e700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff03008e5b0 con 0x7ff0300a7740 2019-12-05T06:35:27.794 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:27.786+0000 7ff034cda700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff03007e720 0x7ff03007eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:35:27.794 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:27.786+0000 7ff034cda700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff03007e720 0x7ff03007eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:58190/0 (socket says 172.21.15.42:58190) 2019-12-05T06:35:27.794 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:27.786+0000 7ff034cda700 1 -- 172.21.15.42:0/953976286 learned_addr learned my addr 172.21.15.42:0/953976286 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:35:27.794 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:27.786+0000 7ff0354db700 1 --2- 172.21.15.42:0/953976286 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff0300a7740 0x7ff03008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:35:27.794 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:27.786+0000 7ff034cda700 1 -- 172.21.15.42:0/953976286 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff0300a7740 msgr2=0x7ff03008df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:35:27.794 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:27.786+0000 7ff034cda700 1 --2- 172.21.15.42:0/953976286 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff0300a7740 0x7ff03008df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:35:27.795 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:27.786+0000 7ff034cda700 1 -- 172.21.15.42:0/953976286 >> v1:172.21.15.40:6789/0 conn(0x7ff03009f2e0 legacy=0x7ff03009f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:35:27.795 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:27.786+0000 7ff034cda700 1 -- 172.21.15.42:0/953976286 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff03007f0a0 con 0x7ff03007e720 2019-12-05T06:35:27.795 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:27.786+0000 7ff034cda700 1 --2- 172.21.15.42:0/953976286 >> v2:172.21.15.42:3300/0 conn(0x7ff03007e720 0x7ff03007eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7ff02c008790 tx=0x7ff02c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:27.795 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:27.786+0000 7ff034cda700 1 --2- 172.21.15.42:0/953976286 >> v2:172.21.15.42:3300/0 conn(0x7ff03007e720 0x7ff03007eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7ff02c008790 tx=0x7ff02c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:27.795 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:27.786+0000 7ff034cda700 1 --2- 172.21.15.42:0/953976286 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff03007e720 0x7ff03007eb50 secure :-1 s=READY pgs=2827 cs=0 l=1 rx=0x7ff02c008790 tx=0x7ff02c00fd30).ready entity=mon.1 client_cookie=922e282f5fea4db8 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:35:27.795 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:27.786+0000 7ff034cda700 1 --2- 172.21.15.42:0/953976286 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff03007e720 0x7ff03007eb50 secure :-1 s=THROTTLE_DONE pgs=2827 cs=0 l=1 rx=0x7ff02c008790 tx=0x7ff02c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:27.795 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:27.786+0000 7ff027fff700 1 -- 172.21.15.42:0/953976286 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff02c0113b0 con 0x7ff03007e720 2019-12-05T06:35:27.796 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:27.786+0000 7ff034cda700 1 --2- 172.21.15.42:0/953976286 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff03007e720 0x7ff03007eb50 secure :-1 s=THROTTLE_DONE pgs=2827 cs=0 l=1 rx=0x7ff02c008790 tx=0x7ff02c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:27.796 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:27.786+0000 7ff034cda700 1 --2- 172.21.15.42:0/953976286 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff03007e720 0x7ff03007eb50 secure :-1 s=THROTTLE_DONE pgs=2827 cs=0 l=1 rx=0x7ff02c008790 tx=0x7ff02c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:27.796 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:27.786+0000 7ff027fff700 1 -- 172.21.15.42:0/953976286 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff02c008bb0 con 0x7ff03007e720 2019-12-05T06:35:27.796 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:27.786+0000 7ff027fff700 1 -- 172.21.15.42:0/953976286 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff02c011990 con 0x7ff03007e720 2019-12-05T06:35:27.796 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:27.786+0000 7ff036f3e700 1 -- 172.21.15.42:0/953976286 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff03007e720 msgr2=0x7ff03007eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:35:27.797 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:27.786+0000 7ff036f3e700 1 --2- 172.21.15.42:0/953976286 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff03007e720 0x7ff03007eb50 secure :-1 s=READY pgs=2827 cs=0 l=1 rx=0x7ff02c008790 tx=0x7ff02c00fd30).stop 2019-12-05T06:35:27.797 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:27.786+0000 7ff036f3e700 1 -- 172.21.15.42:0/953976286 shutdown_connections 2019-12-05T06:35:27.797 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:27.786+0000 7ff036f3e700 1 --2- 172.21.15.42:0/953976286 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff0300a7740 0x7ff03008df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:35:27.797 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:27.786+0000 7ff036f3e700 1 --2- 172.21.15.42:0/953976286 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff03007e720 0x7ff03007eb50 unknown :-1 s=CLOSED pgs=2827 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:35:27.797 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:27.786+0000 7ff036f3e700 1 -- 172.21.15.42:0/953976286 shutdown_connections 2019-12-05T06:35:27.797 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:27.786+0000 7ff036f3e700 1 -- 172.21.15.42:0/953976286 wait complete. 2019-12-05T06:35:27.797 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:27.786+0000 7ff036f3e700 1 -- 172.21.15.42:0/953976286 >> 172.21.15.42:0/953976286 conn(0x7ff0300aabe0 msgr2=0x7ff0300a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:35:27.801 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:27.790+0000 7ff036f3e700 1 Processor -- start 2019-12-05T06:35:27.801 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:27.790+0000 7ff036f3e700 1 -- start start 2019-12-05T06:35:27.801 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:27.790+0000 7ff036f3e700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff03012cf30 0x7ff030124510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:35:27.801 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:27.790+0000 7ff036f3e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff03007d5b0 0x7ff030137940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:35:27.801 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:27.790+0000 7ff036f3e700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff030124a60 con 0x7ff03007d180 2019-12-05T06:35:27.801 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:27.790+0000 7ff036f3e700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0301304f0 con 0x7ff03007d5b0 2019-12-05T06:35:27.802 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:27.790+0000 7ff036f3e700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff030130750 con 0x7ff03012cf30 2019-12-05T06:35:27.802 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:27.790+0000 7ff035cdc700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff03007d5b0 0x7ff030137940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:35:27.802 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:27.790+0000 7ff034cda700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7ff03007d180 0x7ff030130c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:58236/0 (socket says 172.21.15.42:58236) 2019-12-05T06:35:27.802 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:27.790+0000 7ff034cda700 1 -- 172.21.15.42:0/2944149954 learned_addr learned my addr 172.21.15.42:0/2944149954 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:35:27.802 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:27.790+0000 7ff0354db700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff03012cf30 0x7ff030124510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:35:27.802 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:27.794+0000 7ff025ffb700 1 -- 172.21.15.42:0/2944149954 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3537220456 0 0) 0x7ff030124a60 con 0x7ff03007d180 2019-12-05T06:35:27.802 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:27.794+0000 7ff025ffb700 1 -- 172.21.15.42:0/2944149954 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0100030b0 con 0x7ff03007d180 2019-12-05T06:35:27.803 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:27.794+0000 7ff035cdc700 1 -- 172.21.15.42:0/2944149954 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff03012cf30 msgr2=0x7ff030124510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:35:27.803 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:27.794+0000 7ff035cdc700 1 --2- 172.21.15.42:0/2944149954 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff03012cf30 0x7ff030124510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:35:27.803 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:27.794+0000 7ff035cdc700 1 -- 172.21.15.42:0/2944149954 >> v1:172.21.15.42:6789/0 conn(0x7ff03007d180 legacy=0x7ff030130c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:35:27.803 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:27.794+0000 7ff035cdc700 1 -- 172.21.15.42:0/2944149954 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff03011f0d0 con 0x7ff03007d5b0 2019-12-05T06:35:27.803 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:27.794+0000 7ff035cdc700 1 --2- 172.21.15.42:0/2944149954 >> v2:172.21.15.42:3300/0 conn(0x7ff03007d5b0 0x7ff030137940 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7ff028001210 tx=0x7ff028010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:27.803 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:27.794+0000 7ff035cdc700 1 --2- 172.21.15.42:0/2944149954 >> v2:172.21.15.42:3300/0 conn(0x7ff03007d5b0 0x7ff030137940 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7ff028001210 tx=0x7ff028010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:27.804 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:27.794+0000 7ff035cdc700 1 --2- 172.21.15.42:0/2944149954 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff03007d5b0 0x7ff030137940 secure :-1 s=READY pgs=2829 cs=0 l=1 rx=0x7ff028001210 tx=0x7ff028010b40).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:35:27.804 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:27.794+0000 7ff035cdc700 1 --2- 172.21.15.42:0/2944149954 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff03007d5b0 0x7ff030137940 secure :-1 s=THROTTLE_DONE pgs=2829 cs=0 l=1 rx=0x7ff028001210 tx=0x7ff028010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:27.804 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:27.794+0000 7ff025ffb700 1 -- 172.21.15.42:0/2944149954 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff028006ac0 con 0x7ff03007d5b0 2019-12-05T06:35:27.804 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:27.794+0000 7ff035cdc700 1 --2- 172.21.15.42:0/2944149954 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff03007d5b0 0x7ff030137940 secure :-1 s=THROTTLE_DONE pgs=2829 cs=0 l=1 rx=0x7ff028001210 tx=0x7ff028010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:27.804 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:27.794+0000 7ff035cdc700 1 --2- 172.21.15.42:0/2944149954 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff03007d5b0 0x7ff030137940 secure :-1 s=THROTTLE_DONE pgs=2829 cs=0 l=1 rx=0x7ff028001210 tx=0x7ff028010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:27.804 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:27.794+0000 7ff025ffb700 1 -- 172.21.15.42:0/2944149954 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0280066e0 con 0x7ff03007d5b0 2019-12-05T06:35:27.804 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:27.794+0000 7ff036f3e700 1 -- 172.21.15.42:0/2944149954 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff03012dca0 con 0x7ff03007d5b0 2019-12-05T06:35:27.805 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:27.794+0000 7ff025ffb700 1 -- 172.21.15.42:0/2944149954 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff028002730 con 0x7ff03007d5b0 2019-12-05T06:35:27.805 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:27.794+0000 7ff036f3e700 1 -- 172.21.15.42:0/2944149954 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff030137e90 con 0x7ff03007d5b0 2019-12-05T06:35:27.805 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:27.794+0000 7ff035cdc700 1 --2- 172.21.15.42:0/2944149954 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff03007d5b0 0x7ff030137940 secure :-1 s=THROTTLE_DONE pgs=2829 cs=0 l=1 rx=0x7ff028001210 tx=0x7ff028010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:27.805 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:27.794+0000 7ff025ffb700 1 -- 172.21.15.42:0/2944149954 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7ff028002a40 con 0x7ff03007d5b0 2019-12-05T06:35:27.805 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:27.794+0000 7ff035cdc700 1 --2- 172.21.15.42:0/2944149954 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff03007d5b0 0x7ff030137940 secure :-1 s=THROTTLE_DONE pgs=2829 cs=0 l=1 rx=0x7ff028001210 tx=0x7ff028010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:27.805 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:27.794+0000 7ff025ffb700 1 --2- 172.21.15.42:0/2944149954 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff01003b270 0x7ff01003d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:35:27.805 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:27.798+0000 7ff025ffb700 1 -- 172.21.15.42:0/2944149954 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 11277+0+0 (secure 0 0 0) 0x7ff02804b250 con 0x7ff03007d5b0 2019-12-05T06:35:27.806 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:27.798+0000 7ff0354db700 1 --2- 172.21.15.42:0/2944149954 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff01003b270 0x7ff01003d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:35:27.806 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:27.798+0000 7ff0354db700 1 --2- 172.21.15.42:0/2944149954 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff01003b270 0x7ff01003d780 crc :-1 s=READY pgs=1190 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:35:27.806 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:27.798+0000 7ff036f3e700 1 -- 172.21.15.42:0/2944149954 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff03005d110 con 0x7ff03007d5b0 2019-12-05T06:35:27.813 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:27.810+0000 7ff035cdc700 1 --2- 172.21.15.42:0/2944149954 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff03007d5b0 0x7ff030137940 secure :-1 s=THROTTLE_DONE pgs=2829 cs=0 l=1 rx=0x7ff028001210 tx=0x7ff028010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:27.814 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:27.810+0000 7ff025ffb700 1 -- 172.21.15.42:0/2944149954 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7ff0280322d0 con 0x7ff03007d5b0 2019-12-05T06:35:27.983 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:27.978+0000 7ff036f3e700 1 -- 172.21.15.42:0/2944149954 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff0300a71b0 con 0x7ff03007d5b0 2019-12-05T06:35:27.984 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:27.978+0000 7ff035cdc700 1 --2- 172.21.15.42:0/2944149954 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff03007d5b0 0x7ff030137940 secure :-1 s=THROTTLE_DONE pgs=2829 cs=0 l=1 rx=0x7ff028001210 tx=0x7ff028010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:27.984 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:27.978+0000 7ff025ffb700 1 -- 172.21.15.42:0/2944149954 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 338 v338) v1 ==== 95+0+2116 (secure 0 0 0) 0x7ff028032530 con 0x7ff03007d5b0 2019-12-05T06:35:27.986 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 338 2019-12-05T06:35:27.986 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:35:27.986 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":338,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7910,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/4253974783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":4253974783},{"type":"v1","addr":"172.21.15.40:6833","nonce":4253974783}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":338}],"filesystems":[{"mdsmap":{"epoch":337,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:33:30.171201+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":110,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11433},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11433":{"gid":11433,"name":"a","rank":0,"incarnation":334,"state":"up:active","state_seq":33,"addr":"172.21.15.42:6835/2532474042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2532474042},{"type":"v1","addr":"172.21.15.42:6835","nonce":2532474042}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:35:27.989 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:27.986+0000 7ff036f3e700 1 -- 172.21.15.42:0/2944149954 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff01003b270 msgr2=0x7ff01003d780 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:35:27.989 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:27.986+0000 7ff036f3e700 1 --2- 172.21.15.42:0/2944149954 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff01003b270 0x7ff01003d780 crc :-1 s=READY pgs=1190 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:35:27.989 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:27.986+0000 7ff036f3e700 1 -- 172.21.15.42:0/2944149954 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff03007d5b0 msgr2=0x7ff030137940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:35:27.990 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:27.986+0000 7ff036f3e700 1 --2- 172.21.15.42:0/2944149954 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff03007d5b0 0x7ff030137940 secure :-1 s=READY pgs=2829 cs=0 l=1 rx=0x7ff028001210 tx=0x7ff028010b40).stop 2019-12-05T06:35:27.990 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:27.986+0000 7ff036f3e700 1 -- 172.21.15.42:0/2944149954 shutdown_connections 2019-12-05T06:35:27.990 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:27.986+0000 7ff036f3e700 1 --2- 172.21.15.42:0/2944149954 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff01003b270 0x7ff01003d780 unknown :-1 s=CLOSED pgs=1190 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:35:27.990 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:27.986+0000 7ff036f3e700 1 --2- 172.21.15.42:0/2944149954 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff03007d5b0 0x7ff030137940 unknown :-1 s=CLOSED pgs=2829 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:35:27.990 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:27.986+0000 7ff036f3e700 1 --2- 172.21.15.42:0/2944149954 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff03012cf30 0x7ff030124510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:35:27.990 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:27.986+0000 7ff036f3e700 1 -- 172.21.15.42:0/2944149954 shutdown_connections 2019-12-05T06:35:27.990 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:27.986+0000 7ff036f3e700 1 -- 172.21.15.42:0/2944149954 wait complete. 2019-12-05T06:35:27.991 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:27.986+0000 7ff036f3e700 1 -- 172.21.15.42:0/2944149954 >> 172.21.15.42:0/2944149954 conn(0x7ff0300a9870 msgr2=0x7ff03011c610 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:35:28.000 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:35:30.001 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:35:30.109 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:30.098+0000 7fd6ff25a700 1 Processor -- start 2019-12-05T06:35:30.110 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:30.098+0000 7fd6ff25a700 1 -- start start 2019-12-05T06:35:30.110 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:30.102+0000 7fd6ff25a700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd6f809fae0 0x7fd6f814fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:35:30.110 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:30.102+0000 7fd6ff25a700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6f80a0710 con 0x7fd6f80a7740 2019-12-05T06:35:30.110 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:30.102+0000 7fd6ff25a700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6f807ef20 con 0x7fd6f808df80 2019-12-05T06:35:30.111 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:30.102+0000 7fd6ff25a700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6f80a0b00 con 0x7fd6f809fae0 2019-12-05T06:35:30.111 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:30.102+0000 7fd6fdff8700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd6f809fae0 0x7fd6f814fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:35:30.111 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:30.102+0000 7fd6fcff6700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fd6f808df80 0x7fd6f809f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:58250/0 (socket says 172.21.15.42:58250) 2019-12-05T06:35:30.111 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:30.102+0000 7fd6fcff6700 1 -- 172.21.15.42:0/944271103 learned_addr learned my addr 172.21.15.42:0/944271103 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:35:30.111 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:30.102+0000 7fd6ebfff700 1 -- 172.21.15.42:0/944271103 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2613884586 0 0) 0x7fd6f807ef20 con 0x7fd6f808df80 2019-12-05T06:35:30.112 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:30.102+0000 7fd6ebfff700 1 -- 172.21.15.42:0/944271103 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6e40030b0 con 0x7fd6f808df80 2019-12-05T06:35:30.112 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:30.102+0000 7fd6fdff8700 1 -- 172.21.15.42:0/944271103 >> v1:172.21.15.42:6789/0 conn(0x7fd6f808df80 legacy=0x7fd6f809f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:35:30.112 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:30.102+0000 7fd6fdff8700 1 -- 172.21.15.42:0/944271103 >> v1:172.21.15.40:6789/0 conn(0x7fd6f80a7740 legacy=0x7fd6f807e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:35:30.112 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:30.102+0000 7fd6fdff8700 1 -- 172.21.15.42:0/944271103 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6f80a9ba0 con 0x7fd6f809fae0 2019-12-05T06:35:30.112 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:30.102+0000 7fd6fdff8700 1 --2- 172.21.15.42:0/944271103 >> v2:172.21.15.40:3300/0 conn(0x7fd6f809fae0 0x7fd6f814fff0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fd6f0008b10 tx=0x7fd6f000ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:30.112 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:30.102+0000 7fd6fdff8700 1 --2- 172.21.15.42:0/944271103 >> v2:172.21.15.40:3300/0 conn(0x7fd6f809fae0 0x7fd6f814fff0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fd6f0008b10 tx=0x7fd6f000ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:30.113 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:30.102+0000 7fd6fdff8700 1 --2- 172.21.15.42:0/944271103 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd6f809fae0 0x7fd6f814fff0 secure :-1 s=READY pgs=2133 cs=0 l=1 rx=0x7fd6f0008b10 tx=0x7fd6f000ff70).ready entity=mon.0 client_cookie=ba192109a72e76f9 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:35:30.113 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:30.102+0000 7fd6fdff8700 1 --2- 172.21.15.42:0/944271103 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd6f809fae0 0x7fd6f814fff0 secure :-1 s=THROTTLE_DONE pgs=2133 cs=0 l=1 rx=0x7fd6f0008b10 tx=0x7fd6f000ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:30.113 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:30.102+0000 7fd6ebfff700 1 -- 172.21.15.42:0/944271103 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd6f0010970 con 0x7fd6f809fae0 2019-12-05T06:35:30.113 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:30.102+0000 7fd6fdff8700 1 --2- 172.21.15.42:0/944271103 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd6f809fae0 0x7fd6f814fff0 secure :-1 s=THROTTLE_DONE pgs=2133 cs=0 l=1 rx=0x7fd6f0008b10 tx=0x7fd6f000ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:30.114 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:30.102+0000 7fd6fdff8700 1 --2- 172.21.15.42:0/944271103 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd6f809fae0 0x7fd6f814fff0 secure :-1 s=THROTTLE_DONE pgs=2133 cs=0 l=1 rx=0x7fd6f0008b10 tx=0x7fd6f000ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:30.114 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:30.102+0000 7fd6ebfff700 1 -- 172.21.15.42:0/944271103 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6f0008b50 con 0x7fd6f809fae0 2019-12-05T06:35:30.114 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:30.102+0000 7fd6ebfff700 1 -- 172.21.15.42:0/944271103 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd6f00110c0 con 0x7fd6f809fae0 2019-12-05T06:35:30.114 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:30.102+0000 7fd6ff25a700 1 -- 172.21.15.42:0/944271103 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd6f809fae0 msgr2=0x7fd6f814fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:35:30.114 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:30.102+0000 7fd6ff25a700 1 --2- 172.21.15.42:0/944271103 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd6f809fae0 0x7fd6f814fff0 secure :-1 s=READY pgs=2133 cs=0 l=1 rx=0x7fd6f0008b10 tx=0x7fd6f000ff70).stop 2019-12-05T06:35:30.115 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:30.106+0000 7fd6ff25a700 1 -- 172.21.15.42:0/944271103 shutdown_connections 2019-12-05T06:35:30.115 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:30.106+0000 7fd6ff25a700 1 --2- 172.21.15.42:0/944271103 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd6f809fae0 0x7fd6f814fff0 unknown :-1 s=CLOSED pgs=2133 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:35:30.115 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:30.106+0000 7fd6ff25a700 1 -- 172.21.15.42:0/944271103 shutdown_connections 2019-12-05T06:35:30.115 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:30.106+0000 7fd6ff25a700 1 -- 172.21.15.42:0/944271103 wait complete. 2019-12-05T06:35:30.116 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:30.106+0000 7fd6ff25a700 1 -- 172.21.15.42:0/944271103 >> 172.21.15.42:0/944271103 conn(0x7fd6f80aabe0 msgr2=0x7fd6f80a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:35:30.120 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:30.106+0000 7fd6ff25a700 1 Processor -- start 2019-12-05T06:35:30.120 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:30.106+0000 7fd6ff25a700 1 -- start start 2019-12-05T06:35:30.120 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:30.110+0000 7fd6ff25a700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd6f8143270 0x7fd6f812ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:35:30.121 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:30.110+0000 7fd6ff25a700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6f8143da0 con 0x7fd6f807e810 2019-12-05T06:35:30.121 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:30.110+0000 7fd6ff25a700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6f8079450 con 0x7fd6f80a3b20 2019-12-05T06:35:30.121 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:30.110+0000 7fd6ff25a700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6f81446d0 con 0x7fd6f8143270 2019-12-05T06:35:30.121 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:30.110+0000 7fd6fcff6700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fd6f80a3b20 0x7fd6f8142a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:58252/0 (socket says 172.21.15.42:58252) 2019-12-05T06:35:30.122 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:30.110+0000 7fd6fcff6700 1 -- 172.21.15.42:0/2542795097 learned_addr learned my addr 172.21.15.42:0/2542795097 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:35:30.122 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:30.110+0000 7fd6fdff8700 1 --2- 172.21.15.42:0/2542795097 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd6f8143270 0x7fd6f812ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:35:30.122 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:30.110+0000 7fd6e9ffb700 1 -- 172.21.15.42:0/2542795097 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2676729716 0 0) 0x7fd6f8079450 con 0x7fd6f80a3b20 2019-12-05T06:35:30.122 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:30.110+0000 7fd6e9ffb700 1 -- 172.21.15.42:0/2542795097 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6d80030b0 con 0x7fd6f80a3b20 2019-12-05T06:35:30.122 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:30.110+0000 7fd6e9ffb700 1 -- 172.21.15.42:0/2542795097 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3187797307 0 0) 0x7fd6f8143da0 con 0x7fd6f807e810 2019-12-05T06:35:30.123 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:30.110+0000 7fd6e9ffb700 1 -- 172.21.15.42:0/2542795097 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6f8079450 con 0x7fd6f807e810 2019-12-05T06:35:30.123 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:30.110+0000 7fd6fdff8700 1 -- 172.21.15.42:0/2542795097 >> v1:172.21.15.42:6789/0 conn(0x7fd6f80a3b20 legacy=0x7fd6f8142a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:35:30.123 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:30.110+0000 7fd6fdff8700 1 -- 172.21.15.42:0/2542795097 >> v1:172.21.15.40:6789/0 conn(0x7fd6f807e810 legacy=0x7fd6f8078c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:35:30.123 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:30.110+0000 7fd6fdff8700 1 -- 172.21.15.42:0/2542795097 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6f80a3f50 con 0x7fd6f8143270 2019-12-05T06:35:30.123 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:30.110+0000 7fd6fdff8700 1 --2- 172.21.15.42:0/2542795097 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd6f8143270 0x7fd6f812ecf0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fd6f0008b10 tx=0x7fd6f0007ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:30.123 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:30.110+0000 7fd6fdff8700 1 --2- 172.21.15.42:0/2542795097 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd6f8143270 0x7fd6f812ecf0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fd6f0008b10 tx=0x7fd6f0007ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:30.124 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:30.110+0000 7fd6fdff8700 1 --2- 172.21.15.42:0/2542795097 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd6f8143270 0x7fd6f812ecf0 secure :-1 s=READY pgs=625 cs=0 l=1 rx=0x7fd6f0008b10 tx=0x7fd6f0007ae0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:35:30.124 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:30.110+0000 7fd6fdff8700 1 --2- 172.21.15.42:0/2542795097 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd6f8143270 0x7fd6f812ecf0 secure :-1 s=THROTTLE_DONE pgs=625 cs=0 l=1 rx=0x7fd6f0008b10 tx=0x7fd6f0007ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:30.124 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:30.110+0000 7fd6fdff8700 1 --2- 172.21.15.42:0/2542795097 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd6f8143270 0x7fd6f812ecf0 secure :-1 s=THROTTLE_DONE pgs=625 cs=0 l=1 rx=0x7fd6f0008b10 tx=0x7fd6f0007ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:30.124 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:30.110+0000 7fd6e9ffb700 1 -- 172.21.15.42:0/2542795097 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd6f0005ab0 con 0x7fd6f8143270 2019-12-05T06:35:30.124 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:30.110+0000 7fd6fdff8700 1 --2- 172.21.15.42:0/2542795097 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd6f8143270 0x7fd6f812ecf0 secure :-1 s=THROTTLE_DONE pgs=625 cs=0 l=1 rx=0x7fd6f0008b10 tx=0x7fd6f0007ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:30.124 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:30.110+0000 7fd6e9ffb700 1 -- 172.21.15.42:0/2542795097 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6f0010b70 con 0x7fd6f8143270 2019-12-05T06:35:30.125 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:30.110+0000 7fd6ff25a700 1 -- 172.21.15.42:0/2542795097 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6f812f240 con 0x7fd6f8143270 2019-12-05T06:35:30.125 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:30.110+0000 7fd6e9ffb700 1 -- 172.21.15.42:0/2542795097 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd6f000f680 con 0x7fd6f8143270 2019-12-05T06:35:30.125 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:30.110+0000 7fd6ff25a700 1 -- 172.21.15.42:0/2542795097 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6f812f740 con 0x7fd6f8143270 2019-12-05T06:35:30.125 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:30.110+0000 7fd6fdff8700 1 --2- 172.21.15.42:0/2542795097 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd6f8143270 0x7fd6f812ecf0 secure :-1 s=THROTTLE_DONE pgs=625 cs=0 l=1 rx=0x7fd6f0008b10 tx=0x7fd6f0007ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:30.125 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:30.114+0000 7fd6fdff8700 1 --2- 172.21.15.42:0/2542795097 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd6f8143270 0x7fd6f812ecf0 secure :-1 s=THROTTLE_DONE pgs=625 cs=0 l=1 rx=0x7fd6f0008b10 tx=0x7fd6f0007ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:30.125 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:30.114+0000 7fd6e9ffb700 1 -- 172.21.15.42:0/2542795097 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fd6f000f880 con 0x7fd6f8143270 2019-12-05T06:35:30.125 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:30.114+0000 7fd6e9ffb700 1 --2- 172.21.15.42:0/2542795097 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd6d803b4f0 0x7fd6d803da00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:35:30.126 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:30.114+0000 7fd6e9ffb700 1 -- 172.21.15.42:0/2542795097 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 11277+0+0 (secure 0 0 0) 0x7fd6f004afc0 con 0x7fd6f8143270 2019-12-05T06:35:30.126 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:30.114+0000 7fd6fd7f7700 1 --2- 172.21.15.42:0/2542795097 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd6d803b4f0 0x7fd6d803da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:35:30.126 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:30.114+0000 7fd6fd7f7700 1 --2- 172.21.15.42:0/2542795097 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd6d803b4f0 0x7fd6d803da00 crc :-1 s=READY pgs=1191 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:35:30.126 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:30.114+0000 7fd6ff25a700 1 -- 172.21.15.42:0/2542795097 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6f805d110 con 0x7fd6f8143270 2019-12-05T06:35:30.130 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:30.126+0000 7fd6fdff8700 1 --2- 172.21.15.42:0/2542795097 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd6f8143270 0x7fd6f812ecf0 secure :-1 s=THROTTLE_DONE pgs=625 cs=0 l=1 rx=0x7fd6f0008b10 tx=0x7fd6f0007ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:30.130 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:30.126+0000 7fd6e9ffb700 1 -- 172.21.15.42:0/2542795097 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fd6f0032000 con 0x7fd6f8143270 2019-12-05T06:35:30.300 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:30.294+0000 7fd6ff25a700 1 -- 172.21.15.42:0/2542795097 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd6f805b8c0 con 0x7fd6f8143270 2019-12-05T06:35:30.301 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:30.298+0000 7fd6fdff8700 1 --2- 172.21.15.42:0/2542795097 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd6f8143270 0x7fd6f812ecf0 secure :-1 s=THROTTLE_DONE pgs=625 cs=0 l=1 rx=0x7fd6f0008b10 tx=0x7fd6f0007ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:30.301 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:30.298+0000 7fd6e9ffb700 1 -- 172.21.15.42:0/2542795097 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 338 v338) v1 ==== 95+0+2116 (secure 0 0 0) 0x7fd6f0032260 con 0x7fd6f8143270 2019-12-05T06:35:30.303 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:35:30.303 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":338,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7910,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/4253974783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":4253974783},{"type":"v1","addr":"172.21.15.40:6833","nonce":4253974783}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":338}],"filesystems":[{"mdsmap":{"epoch":337,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:33:30.171201+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":110,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11433},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11433":{"gid":11433,"name":"a","rank":0,"incarnation":334,"state":"up:active","state_seq":33,"addr":"172.21.15.42:6835/2532474042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2532474042},{"type":"v1","addr":"172.21.15.42:6835","nonce":2532474042}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:35:30.303 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 338 2019-12-05T06:35:30.306 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:30.302+0000 7fd6ff25a700 1 -- 172.21.15.42:0/2542795097 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd6d803b4f0 msgr2=0x7fd6d803da00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:35:30.306 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:30.302+0000 7fd6ff25a700 1 --2- 172.21.15.42:0/2542795097 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd6d803b4f0 0x7fd6d803da00 crc :-1 s=READY pgs=1191 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:35:30.306 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:30.302+0000 7fd6ff25a700 1 -- 172.21.15.42:0/2542795097 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd6f8143270 msgr2=0x7fd6f812ecf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:35:30.307 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:30.302+0000 7fd6ff25a700 1 --2- 172.21.15.42:0/2542795097 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd6f8143270 0x7fd6f812ecf0 secure :-1 s=READY pgs=625 cs=0 l=1 rx=0x7fd6f0008b10 tx=0x7fd6f0007ae0).stop 2019-12-05T06:35:30.307 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:30.302+0000 7fd6ff25a700 1 -- 172.21.15.42:0/2542795097 shutdown_connections 2019-12-05T06:35:30.307 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:30.302+0000 7fd6ff25a700 1 --2- 172.21.15.42:0/2542795097 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd6d803b4f0 0x7fd6d803da00 unknown :-1 s=CLOSED pgs=1191 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:35:30.307 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:30.302+0000 7fd6ff25a700 1 --2- 172.21.15.42:0/2542795097 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd6f8143270 0x7fd6f812ecf0 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:35:30.307 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:30.302+0000 7fd6ff25a700 1 -- 172.21.15.42:0/2542795097 shutdown_connections 2019-12-05T06:35:30.307 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:30.302+0000 7fd6ff25a700 1 -- 172.21.15.42:0/2542795097 wait complete. 2019-12-05T06:35:30.308 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:30.302+0000 7fd6ff25a700 1 -- 172.21.15.42:0/2542795097 >> 172.21.15.42:0/2542795097 conn(0x7fd6f808fde0 msgr2=0x7fd6f8059ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:35:30.315 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:35:32.317 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:35:32.423 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:32.414+0000 7fb5af26a700 1 Processor -- start 2019-12-05T06:35:32.423 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:32.414+0000 7fb5af26a700 1 -- start start 2019-12-05T06:35:32.423 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:32.414+0000 7fb5af26a700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb5a80a7740 0x7fb5a808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:35:32.423 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:32.414+0000 7fb5af26a700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5a80a0710 con 0x7fb5a807e720 2019-12-05T06:35:32.424 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:32.414+0000 7fb5af26a700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5a80a9ba0 con 0x7fb5a809fae0 2019-12-05T06:35:32.424 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:32.414+0000 7fb5af26a700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5a80a0b00 con 0x7fb5a80a7740 2019-12-05T06:35:32.424 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:32.414+0000 7fb5ae008700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fb5a809fae0 0x7fb5a814efe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:58264/0 (socket says 172.21.15.42:58264) 2019-12-05T06:35:32.424 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:32.414+0000 7fb5ae008700 1 -- 172.21.15.42:0/1702411157 learned_addr learned my addr 172.21.15.42:0/1702411157 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:35:32.424 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:32.414+0000 7fb5ad807700 1 --2- 172.21.15.42:0/1702411157 >> v2:172.21.15.40:3300/0 conn(0x7fb5a80a7740 0x7fb5a808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:35:32.424 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:32.414+0000 7fb5ac805700 1 -- 172.21.15.42:0/1702411157 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3169695247 0 0) 0x7fb5a80a0710 con 0x7fb5a807e720 2019-12-05T06:35:32.425 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:32.414+0000 7fb5ac805700 1 -- 172.21.15.42:0/1702411157 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb5900030b0 con 0x7fb5a807e720 2019-12-05T06:35:32.425 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:32.414+0000 7fb5ac805700 1 -- 172.21.15.42:0/1702411157 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2714622417 0 0) 0x7fb5a80a9ba0 con 0x7fb5a809fae0 2019-12-05T06:35:32.425 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:32.414+0000 7fb5ac805700 1 -- 172.21.15.42:0/1702411157 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb5a80a0710 con 0x7fb5a809fae0 2019-12-05T06:35:32.425 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:32.414+0000 7fb5ad807700 1 -- 172.21.15.42:0/1702411157 >> v1:172.21.15.42:6789/0 conn(0x7fb5a809fae0 legacy=0x7fb5a814efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:35:32.425 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:32.414+0000 7fb5ad807700 1 -- 172.21.15.42:0/1702411157 >> v1:172.21.15.40:6789/0 conn(0x7fb5a807e720 legacy=0x7fb5a809f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:35:32.426 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:32.414+0000 7fb5ad807700 1 -- 172.21.15.42:0/1702411157 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5a807eb50 con 0x7fb5a80a7740 2019-12-05T06:35:32.426 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:32.414+0000 7fb5ad807700 1 --2- 172.21.15.42:0/1702411157 >> v2:172.21.15.40:3300/0 conn(0x7fb5a80a7740 0x7fb5a808df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fb59c008790 tx=0x7fb59c00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:32.426 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:32.414+0000 7fb5ad807700 1 --2- 172.21.15.42:0/1702411157 >> v2:172.21.15.40:3300/0 conn(0x7fb5a80a7740 0x7fb5a808df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fb59c008790 tx=0x7fb59c00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:32.426 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:32.414+0000 7fb5ad807700 1 --2- 172.21.15.42:0/1702411157 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb5a80a7740 0x7fb5a808df80 secure :-1 s=READY pgs=2136 cs=0 l=1 rx=0x7fb59c008790 tx=0x7fb59c00fbf0).ready entity=mon.0 client_cookie=7172b63e7e6c662e server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:35:32.426 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:32.418+0000 7fb5ad807700 1 --2- 172.21.15.42:0/1702411157 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb5a80a7740 0x7fb5a808df80 secure :-1 s=THROTTLE_DONE pgs=2136 cs=0 l=1 rx=0x7fb59c008790 tx=0x7fb59c00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:32.427 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:32.418+0000 7fb5ac805700 1 -- 172.21.15.42:0/1702411157 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb59c011270 con 0x7fb5a80a7740 2019-12-05T06:35:32.427 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:32.418+0000 7fb5ad807700 1 --2- 172.21.15.42:0/1702411157 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb5a80a7740 0x7fb5a808df80 secure :-1 s=THROTTLE_DONE pgs=2136 cs=0 l=1 rx=0x7fb59c008790 tx=0x7fb59c00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:32.427 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:32.418+0000 7fb5ad807700 1 --2- 172.21.15.42:0/1702411157 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb5a80a7740 0x7fb5a808df80 secure :-1 s=THROTTLE_DONE pgs=2136 cs=0 l=1 rx=0x7fb59c008790 tx=0x7fb59c00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:32.427 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:32.418+0000 7fb5ac805700 1 -- 172.21.15.42:0/1702411157 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb59c008bc0 con 0x7fb5a80a7740 2019-12-05T06:35:32.427 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:32.418+0000 7fb5ac805700 1 -- 172.21.15.42:0/1702411157 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb59c011850 con 0x7fb5a80a7740 2019-12-05T06:35:32.427 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:32.418+0000 7fb5af26a700 1 -- 172.21.15.42:0/1702411157 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb5a80a7740 msgr2=0x7fb5a808df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:35:32.427 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:32.418+0000 7fb5af26a700 1 --2- 172.21.15.42:0/1702411157 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb5a80a7740 0x7fb5a808df80 secure :-1 s=READY pgs=2136 cs=0 l=1 rx=0x7fb59c008790 tx=0x7fb59c00fbf0).stop 2019-12-05T06:35:32.428 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:32.418+0000 7fb5af26a700 1 -- 172.21.15.42:0/1702411157 shutdown_connections 2019-12-05T06:35:32.428 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:32.418+0000 7fb5af26a700 1 --2- 172.21.15.42:0/1702411157 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb5a80a7740 0x7fb5a808df80 unknown :-1 s=CLOSED pgs=2136 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:35:32.428 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:32.418+0000 7fb5af26a700 1 -- 172.21.15.42:0/1702411157 shutdown_connections 2019-12-05T06:35:32.428 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:32.418+0000 7fb5af26a700 1 -- 172.21.15.42:0/1702411157 wait complete. 2019-12-05T06:35:32.428 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:32.418+0000 7fb5af26a700 1 -- 172.21.15.42:0/1702411157 >> 172.21.15.42:0/1702411157 conn(0x7fb5a80aabe0 msgr2=0x7fb5a80a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:35:32.433 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:32.418+0000 7fb5af26a700 1 Processor -- start 2019-12-05T06:35:32.433 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:32.418+0000 7fb5af26a700 1 -- start start 2019-12-05T06:35:32.433 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:32.422+0000 7fb5af26a700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb5a811c1d0 0x7fb5a807d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:35:32.433 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:32.422+0000 7fb5af26a700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb5a807dda0 0x7fb5a812fe60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:35:32.434 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:32.422+0000 7fb5af26a700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5a80a6c10 con 0x7fb5a8142530 2019-12-05T06:35:32.434 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:32.422+0000 7fb5af26a700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5a807e1d0 con 0x7fb5a811c1d0 2019-12-05T06:35:32.434 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:32.422+0000 7fb5af26a700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5a8144fe0 con 0x7fb5a807dda0 2019-12-05T06:35:32.434 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:32.422+0000 7fb5ad807700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fb5a8142530 0x7fb5a80a4fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:58266/0 (socket says 172.21.15.42:58266) 2019-12-05T06:35:32.434 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:32.422+0000 7fb5ad807700 1 -- 172.21.15.42:0/3133875435 learned_addr learned my addr 172.21.15.42:0/3133875435 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:35:32.435 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:32.422+0000 7fb5ae008700 1 --2- 172.21.15.42:0/3133875435 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb5a807dda0 0x7fb5a812fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:35:32.435 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:32.422+0000 7fb5ad006700 1 --2- 172.21.15.42:0/3133875435 >> v2:172.21.15.40:3300/0 conn(0x7fb5a811c1d0 0x7fb5a807d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:35:32.435 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:32.422+0000 7fb5a67fc700 1 -- 172.21.15.42:0/3133875435 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1121835216 0 0) 0x7fb5a80a6c10 con 0x7fb5a8142530 2019-12-05T06:35:32.435 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:32.422+0000 7fb5a67fc700 1 -- 172.21.15.42:0/3133875435 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb5880030b0 con 0x7fb5a8142530 2019-12-05T06:35:32.435 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:32.422+0000 7fb5ae008700 1 -- 172.21.15.42:0/3133875435 >> v2:172.21.15.40:3300/0 conn(0x7fb5a811c1d0 msgr2=0x7fb5a807d850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:35:32.435 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:32.422+0000 7fb5ae008700 1 --2- 172.21.15.42:0/3133875435 >> v2:172.21.15.40:3300/0 conn(0x7fb5a811c1d0 0x7fb5a807d850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:35:32.435 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:32.422+0000 7fb5ae008700 1 -- 172.21.15.42:0/3133875435 >> v1:172.21.15.42:6789/0 conn(0x7fb5a8142530 legacy=0x7fb5a80a4fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:35:32.436 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:32.422+0000 7fb5a67fc700 1 -- 172.21.15.42:0/3133875435 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 2838485432 0 0) 0x7fb5880030b0 con 0x7fb5a8142530 2019-12-05T06:35:32.436 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:32.422+0000 7fb5ae008700 1 -- 172.21.15.42:0/3133875435 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5a80a57a0 con 0x7fb5a807dda0 2019-12-05T06:35:32.436 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:32.422+0000 7fb5ae008700 1 --2- 172.21.15.42:0/3133875435 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb5a807dda0 0x7fb5a812fe60 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fb59c007e90 tx=0x7fb59c010100).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:32.436 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:32.422+0000 7fb5ae008700 1 --2- 172.21.15.42:0/3133875435 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb5a807dda0 0x7fb5a812fe60 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fb59c007e90 tx=0x7fb59c010100).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:32.436 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:32.422+0000 7fb5ae008700 1 --2- 172.21.15.42:0/3133875435 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb5a807dda0 0x7fb5a812fe60 secure :-1 s=READY pgs=626 cs=0 l=1 rx=0x7fb59c007e90 tx=0x7fb59c010100).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:35:32.436 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:32.422+0000 7fb5ae008700 1 --2- 172.21.15.42:0/3133875435 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb5a807dda0 0x7fb5a812fe60 secure :-1 s=THROTTLE_DONE pgs=626 cs=0 l=1 rx=0x7fb59c007e90 tx=0x7fb59c010100).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:32.437 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:32.422+0000 7fb5a67fc700 1 -- 172.21.15.42:0/3133875435 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb59c011270 con 0x7fb5a807dda0 2019-12-05T06:35:32.437 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:32.422+0000 7fb5ae008700 1 --2- 172.21.15.42:0/3133875435 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb5a807dda0 0x7fb5a812fe60 secure :-1 s=THROTTLE_DONE pgs=626 cs=0 l=1 rx=0x7fb59c007e90 tx=0x7fb59c010100).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:32.437 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:32.422+0000 7fb5a67fc700 1 -- 172.21.15.42:0/3133875435 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb59c011470 con 0x7fb5a807dda0 2019-12-05T06:35:32.437 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:32.422+0000 7fb5ae008700 1 --2- 172.21.15.42:0/3133875435 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb5a807dda0 0x7fb5a812fe60 secure :-1 s=THROTTLE_DONE pgs=626 cs=0 l=1 rx=0x7fb59c007e90 tx=0x7fb59c010100).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:32.437 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:32.422+0000 7fb5af26a700 1 -- 172.21.15.42:0/3133875435 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5a80a4a50 con 0x7fb5a807dda0 2019-12-05T06:35:32.437 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:32.422+0000 7fb5a67fc700 1 -- 172.21.15.42:0/3133875435 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb5a00122e0 con 0x7fb5a807dda0 2019-12-05T06:35:32.438 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:32.422+0000 7fb5af26a700 1 -- 172.21.15.42:0/3133875435 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5a8063fe0 con 0x7fb5a807dda0 2019-12-05T06:35:32.438 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:32.422+0000 7fb5ae008700 1 --2- 172.21.15.42:0/3133875435 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb5a807dda0 0x7fb5a812fe60 secure :-1 s=THROTTLE_DONE pgs=626 cs=0 l=1 rx=0x7fb59c007e90 tx=0x7fb59c010100).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:32.438 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:32.426+0000 7fb5a67fc700 1 -- 172.21.15.42:0/3133875435 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fb5a0012580 con 0x7fb5a807dda0 2019-12-05T06:35:32.438 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:32.426+0000 7fb5ae008700 1 --2- 172.21.15.42:0/3133875435 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb5a807dda0 0x7fb5a812fe60 secure :-1 s=THROTTLE_DONE pgs=626 cs=0 l=1 rx=0x7fb59c007e90 tx=0x7fb59c010100).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:32.438 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:32.426+0000 7fb5a67fc700 1 --2- 172.21.15.42:0/3133875435 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb58803b230 0x7fb58803d740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:35:32.438 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:32.426+0000 7fb5a67fc700 1 -- 172.21.15.42:0/3133875435 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 11277+0+0 (secure 0 0 0) 0x7fb5a004ad60 con 0x7fb5a807dda0 2019-12-05T06:35:32.439 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:32.426+0000 7fb5ad807700 1 --2- 172.21.15.42:0/3133875435 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb58803b230 0x7fb58803d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:35:32.439 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:32.426+0000 7fb5ad807700 1 --2- 172.21.15.42:0/3133875435 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb58803b230 0x7fb58803d740 crc :-1 s=READY pgs=1192 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:35:32.439 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:32.430+0000 7fb5af26a700 1 -- 172.21.15.42:0/3133875435 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5a805d110 con 0x7fb5a807dda0 2019-12-05T06:35:32.443 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:32.438+0000 7fb5ae008700 1 --2- 172.21.15.42:0/3133875435 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb5a807dda0 0x7fb5a812fe60 secure :-1 s=THROTTLE_DONE pgs=626 cs=0 l=1 rx=0x7fb59c007e90 tx=0x7fb59c010100).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:32.443 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:32.438+0000 7fb5a67fc700 1 -- 172.21.15.42:0/3133875435 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fb5a0031da0 con 0x7fb5a807dda0 2019-12-05T06:35:32.616 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:32.610+0000 7fb5af26a700 1 -- 172.21.15.42:0/3133875435 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb5a81505d0 con 0x7fb5a807dda0 2019-12-05T06:35:32.617 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:32.614+0000 7fb5ae008700 1 --2- 172.21.15.42:0/3133875435 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb5a807dda0 0x7fb5a812fe60 secure :-1 s=THROTTLE_DONE pgs=626 cs=0 l=1 rx=0x7fb59c007e90 tx=0x7fb59c010100).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:32.617 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:32.614+0000 7fb5a67fc700 1 -- 172.21.15.42:0/3133875435 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 338 v338) v1 ==== 95+0+2116 (secure 0 0 0) 0x7fb5a0032000 con 0x7fb5a807dda0 2019-12-05T06:35:32.619 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:35:32.619 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":338,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7910,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/4253974783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":4253974783},{"type":"v1","addr":"172.21.15.40:6833","nonce":4253974783}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":338}],"filesystems":[{"mdsmap":{"epoch":337,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:33:30.171201+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":110,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11433},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11433":{"gid":11433,"name":"a","rank":0,"incarnation":334,"state":"up:active","state_seq":33,"addr":"172.21.15.42:6835/2532474042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2532474042},{"type":"v1","addr":"172.21.15.42:6835","nonce":2532474042}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:35:32.619 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 338 2019-12-05T06:35:32.621 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:32.618+0000 7fb5af26a700 1 -- 172.21.15.42:0/3133875435 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb58803b230 msgr2=0x7fb58803d740 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:35:32.621 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:32.618+0000 7fb5af26a700 1 --2- 172.21.15.42:0/3133875435 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb58803b230 0x7fb58803d740 crc :-1 s=READY pgs=1192 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:35:32.622 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:32.618+0000 7fb5af26a700 1 -- 172.21.15.42:0/3133875435 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb5a807dda0 msgr2=0x7fb5a812fe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:35:32.622 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:32.618+0000 7fb5af26a700 1 --2- 172.21.15.42:0/3133875435 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb5a807dda0 0x7fb5a812fe60 secure :-1 s=READY pgs=626 cs=0 l=1 rx=0x7fb59c007e90 tx=0x7fb59c010100).stop 2019-12-05T06:35:32.622 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:32.618+0000 7fb5af26a700 1 -- 172.21.15.42:0/3133875435 shutdown_connections 2019-12-05T06:35:32.622 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:32.618+0000 7fb5af26a700 1 --2- 172.21.15.42:0/3133875435 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb58803b230 0x7fb58803d740 unknown :-1 s=CLOSED pgs=1192 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:35:32.622 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:32.618+0000 7fb5af26a700 1 --2- 172.21.15.42:0/3133875435 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb5a807dda0 0x7fb5a812fe60 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:35:32.623 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:32.618+0000 7fb5af26a700 1 --2- 172.21.15.42:0/3133875435 >> v2:172.21.15.40:3300/0 conn(0x7fb5a811c1d0 0x7fb5a807d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:35:32.623 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:32.618+0000 7fb5af26a700 1 -- 172.21.15.42:0/3133875435 shutdown_connections 2019-12-05T06:35:32.623 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:32.618+0000 7fb5af26a700 1 -- 172.21.15.42:0/3133875435 wait complete. 2019-12-05T06:35:32.624 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:32.618+0000 7fb5af26a700 1 -- 172.21.15.42:0/3133875435 >> 172.21.15.42:0/3133875435 conn(0x7fb5a808fde0 msgr2=0x7fb5a80a96e0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:35:32.632 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:35:34.632 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:35:34.744 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:34.734+0000 7f3f98be4700 1 Processor -- start 2019-12-05T06:35:34.745 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:34.734+0000 7f3f98be4700 1 -- start start 2019-12-05T06:35:34.745 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:34.734+0000 7f3f98be4700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f3f940a7730 0x7f3f9408df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:35:34.745 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:34.734+0000 7f3f98be4700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3f9407e710 0x7f3f9407eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:35:34.745 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:34.734+0000 7f3f98be4700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f940a0700 con 0x7f3f9409f2d0 2019-12-05T06:35:34.745 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:34.734+0000 7f3f98be4700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f940a0af0 con 0x7f3f940a7730 2019-12-05T06:35:34.745 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:34.734+0000 7f3f98be4700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f9408e5a0 con 0x7f3f9407e710 2019-12-05T06:35:34.745 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:34.734+0000 7f3f9259c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3f9407e710 0x7f3f9407eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:35:34.746 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:34.738+0000 7f3f9259c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3f9407e710 0x7f3f9407eb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:58240/0 (socket says 172.21.15.42:58240) 2019-12-05T06:35:34.746 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:34.738+0000 7f3f9259c700 1 -- 172.21.15.42:0/1926615716 learned_addr learned my addr 172.21.15.42:0/1926615716 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:35:34.746 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:34.738+0000 7f3f9359e700 1 --1- 172.21.15.42:0/1926615716 >> v1:172.21.15.42:6789/0 conn(0x7f3f9409f2d0 0x7f3f9409f700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:58280/0 (socket says 172.21.15.42:58280) 2019-12-05T06:35:34.746 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:34.738+0000 7f3f92d9d700 1 --2- 172.21.15.42:0/1926615716 >> v2:172.21.15.40:3300/0 conn(0x7f3f940a7730 0x7f3f9408df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:35:34.746 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:34.738+0000 7f3f9259c700 1 -- 172.21.15.42:0/1926615716 >> v2:172.21.15.40:3300/0 conn(0x7f3f940a7730 msgr2=0x7f3f9408df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:35:34.746 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:34.738+0000 7f3f9259c700 1 --2- 172.21.15.42:0/1926615716 >> v2:172.21.15.40:3300/0 conn(0x7f3f940a7730 0x7f3f9408df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:35:34.746 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:34.738+0000 7f3f9259c700 1 -- 172.21.15.42:0/1926615716 >> v1:172.21.15.42:6789/0 conn(0x7f3f9409f2d0 legacy=0x7f3f9409f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:35:34.747 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:34.738+0000 7f3f9259c700 1 -- 172.21.15.42:0/1926615716 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f9407f090 con 0x7f3f9407e710 2019-12-05T06:35:34.747 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:34.738+0000 7f3f9259c700 1 --2- 172.21.15.42:0/1926615716 >> v2:172.21.15.42:3300/0 conn(0x7f3f9407e710 0x7f3f9407eb40 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f3f7c002f30 tx=0x7f3f7c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:34.747 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:34.738+0000 7f3f9259c700 1 --2- 172.21.15.42:0/1926615716 >> v2:172.21.15.42:3300/0 conn(0x7f3f9407e710 0x7f3f9407eb40 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f3f7c002f30 tx=0x7f3f7c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:34.747 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:34.738+0000 7f3f9259c700 1 --2- 172.21.15.42:0/1926615716 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3f9407e710 0x7f3f9407eb40 secure :-1 s=READY pgs=2835 cs=0 l=1 rx=0x7f3f7c002f30 tx=0x7f3f7c00fd30).ready entity=mon.1 client_cookie=adf677a4e87ac454 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:35:34.747 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:34.738+0000 7f3f9259c700 1 --2- 172.21.15.42:0/1926615716 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3f9407e710 0x7f3f9407eb40 secure :-1 s=THROTTLE_DONE pgs=2835 cs=0 l=1 rx=0x7f3f7c002f30 tx=0x7f3f7c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:34.747 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:34.738+0000 7f3f91d9b700 1 -- 172.21.15.42:0/1926615716 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3f7c011370 con 0x7f3f9407e710 2019-12-05T06:35:34.747 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:34.738+0000 7f3f9259c700 1 --2- 172.21.15.42:0/1926615716 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3f9407e710 0x7f3f9407eb40 secure :-1 s=THROTTLE_DONE pgs=2835 cs=0 l=1 rx=0x7f3f7c002f30 tx=0x7f3f7c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:34.748 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:34.738+0000 7f3f9259c700 1 --2- 172.21.15.42:0/1926615716 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3f9407e710 0x7f3f9407eb40 secure :-1 s=THROTTLE_DONE pgs=2835 cs=0 l=1 rx=0x7f3f7c002f30 tx=0x7f3f7c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:34.748 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:34.738+0000 7f3f91d9b700 1 -- 172.21.15.42:0/1926615716 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f7c008bb0 con 0x7f3f9407e710 2019-12-05T06:35:34.748 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:34.738+0000 7f3f91d9b700 1 -- 172.21.15.42:0/1926615716 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3f7c011950 con 0x7f3f9407e710 2019-12-05T06:35:34.748 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:34.738+0000 7f3f98be4700 1 -- 172.21.15.42:0/1926615716 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3f9407e710 msgr2=0x7f3f9407eb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:35:34.748 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:34.738+0000 7f3f98be4700 1 --2- 172.21.15.42:0/1926615716 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3f9407e710 0x7f3f9407eb40 secure :-1 s=READY pgs=2835 cs=0 l=1 rx=0x7f3f7c002f30 tx=0x7f3f7c00fd30).stop 2019-12-05T06:35:34.748 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:34.738+0000 7f3f98be4700 1 -- 172.21.15.42:0/1926615716 shutdown_connections 2019-12-05T06:35:34.748 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:34.738+0000 7f3f98be4700 1 --2- 172.21.15.42:0/1926615716 >> v2:172.21.15.40:3300/0 conn(0x7f3f940a7730 0x7f3f9408df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:35:34.749 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:34.738+0000 7f3f98be4700 1 --2- 172.21.15.42:0/1926615716 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3f9407e710 0x7f3f9407eb40 unknown :-1 s=CLOSED pgs=2835 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:35:34.749 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:34.738+0000 7f3f98be4700 1 -- 172.21.15.42:0/1926615716 shutdown_connections 2019-12-05T06:35:34.749 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:34.738+0000 7f3f98be4700 1 -- 172.21.15.42:0/1926615716 wait complete. 2019-12-05T06:35:34.749 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:34.738+0000 7f3f98be4700 1 -- 172.21.15.42:0/1926615716 >> 172.21.15.42:0/1926615716 conn(0x7f3f940aabd0 msgr2=0x7f3f940a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:35:34.752 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:34.742+0000 7f3f98be4700 1 Processor -- start 2019-12-05T06:35:34.752 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:34.742+0000 7f3f98be4700 1 -- start start 2019-12-05T06:35:34.753 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:34.742+0000 7f3f98be4700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f3f940780e0 0x7f3f94117e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:35:34.753 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:34.742+0000 7f3f98be4700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3f94127fe0 0x7f3f9412f430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:35:34.753 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:34.742+0000 7f3f98be4700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f941122b0 con 0x7f3f94115370 2019-12-05T06:35:34.753 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:34.742+0000 7f3f98be4700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f94078b40 con 0x7f3f940780e0 2019-12-05T06:35:34.753 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:34.742+0000 7f3f98be4700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f94078510 con 0x7f3f94127fe0 2019-12-05T06:35:34.753 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:34.742+0000 7f3f9359e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3f94127fe0 0x7f3f9412f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:35:34.754 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:34.742+0000 7f3f9259c700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f3f94115370 0x7f3f94113500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:58284/0 (socket says 172.21.15.42:58284) 2019-12-05T06:35:34.754 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:34.742+0000 7f3f9259c700 1 -- 172.21.15.42:0/549154793 learned_addr learned my addr 172.21.15.42:0/549154793 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:35:34.754 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:34.742+0000 7f3f92d9d700 1 --2- 172.21.15.42:0/549154793 >> v2:172.21.15.40:3300/0 conn(0x7f3f940780e0 0x7f3f94117e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:35:34.754 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:34.746+0000 7f3f8b7fe700 1 -- 172.21.15.42:0/549154793 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2946917041 0 0) 0x7f3f7c008470 con 0x7f3f94115370 2019-12-05T06:35:34.754 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:34.746+0000 7f3f9359e700 1 -- 172.21.15.42:0/549154793 >> v2:172.21.15.40:3300/0 conn(0x7f3f940780e0 msgr2=0x7f3f94117e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:35:34.754 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:34.746+0000 7f3f9359e700 1 --2- 172.21.15.42:0/549154793 >> v2:172.21.15.40:3300/0 conn(0x7f3f940780e0 0x7f3f94117e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:35:34.754 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:34.746+0000 7f3f9359e700 1 -- 172.21.15.42:0/549154793 >> v1:172.21.15.42:6789/0 conn(0x7f3f94115370 legacy=0x7f3f94113500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:35:34.755 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:34.746+0000 7f3f9359e700 1 -- 172.21.15.42:0/549154793 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f94113d00 con 0x7f3f94127fe0 2019-12-05T06:35:34.755 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:34.746+0000 7f3f92d9d700 1 --2- 172.21.15.42:0/549154793 >> v2:172.21.15.40:3300/0 conn(0x7f3f940780e0 0x7f3f94117e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:35:34.755 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:34.746+0000 7f3f9359e700 1 --2- 172.21.15.42:0/549154793 >> v2:172.21.15.42:3300/0 conn(0x7f3f94127fe0 0x7f3f9412f430 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f3f940a73b0 tx=0x7f3f8c010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:34.755 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:34.746+0000 7f3f9359e700 1 --2- 172.21.15.42:0/549154793 >> v2:172.21.15.42:3300/0 conn(0x7f3f94127fe0 0x7f3f9412f430 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f3f940a73b0 tx=0x7f3f8c010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:34.755 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:34.746+0000 7f3f9359e700 1 --2- 172.21.15.42:0/549154793 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3f94127fe0 0x7f3f9412f430 secure :-1 s=READY pgs=2837 cs=0 l=1 rx=0x7f3f940a73b0 tx=0x7f3f8c010b50).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:35:34.755 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:34.746+0000 7f3f9359e700 1 --2- 172.21.15.42:0/549154793 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3f94127fe0 0x7f3f9412f430 secure :-1 s=THROTTLE_DONE pgs=2837 cs=0 l=1 rx=0x7f3f940a73b0 tx=0x7f3f8c010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:34.755 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:34.746+0000 7f3f8b7fe700 1 -- 172.21.15.42:0/549154793 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3f8c013060 con 0x7f3f94127fe0 2019-12-05T06:35:34.755 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:34.746+0000 7f3f9359e700 1 --2- 172.21.15.42:0/549154793 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3f94127fe0 0x7f3f9412f430 secure :-1 s=THROTTLE_DONE pgs=2837 cs=0 l=1 rx=0x7f3f940a73b0 tx=0x7f3f8c010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:34.756 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:34.746+0000 7f3f8b7fe700 1 -- 172.21.15.42:0/549154793 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f8c0046f0 con 0x7f3f94127fe0 2019-12-05T06:35:34.756 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:34.746+0000 7f3f9359e700 1 --2- 172.21.15.42:0/549154793 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3f94127fe0 0x7f3f9412f430 secure :-1 s=THROTTLE_DONE pgs=2837 cs=0 l=1 rx=0x7f3f940a73b0 tx=0x7f3f8c010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:34.756 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:34.746+0000 7f3f98be4700 1 -- 172.21.15.42:0/549154793 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f941185a0 con 0x7f3f94127fe0 2019-12-05T06:35:34.756 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:34.746+0000 7f3f8b7fe700 1 -- 172.21.15.42:0/549154793 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3f8c0124d0 con 0x7f3f94127fe0 2019-12-05T06:35:34.756 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:34.746+0000 7f3f98be4700 1 -- 172.21.15.42:0/549154793 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f941157a0 con 0x7f3f94127fe0 2019-12-05T06:35:34.756 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:34.746+0000 7f3f9359e700 1 --2- 172.21.15.42:0/549154793 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3f94127fe0 0x7f3f9412f430 secure :-1 s=THROTTLE_DONE pgs=2837 cs=0 l=1 rx=0x7f3f940a73b0 tx=0x7f3f8c010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:34.756 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:34.746+0000 7f3f8b7fe700 1 -- 172.21.15.42:0/549154793 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f3f8c012830 con 0x7f3f94127fe0 2019-12-05T06:35:34.757 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:34.746+0000 7f3f9359e700 1 --2- 172.21.15.42:0/549154793 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3f94127fe0 0x7f3f9412f430 secure :-1 s=THROTTLE_DONE pgs=2837 cs=0 l=1 rx=0x7f3f940a73b0 tx=0x7f3f8c010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:34.757 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:34.746+0000 7f3f8b7fe700 1 --2- 172.21.15.42:0/549154793 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3f7803abc0 0x7f3f7803d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:35:34.757 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:34.746+0000 7f3f8b7fe700 1 -- 172.21.15.42:0/549154793 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 11277+0+0 (secure 0 0 0) 0x7f3f8c04b130 con 0x7f3f94127fe0 2019-12-05T06:35:34.757 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:34.746+0000 7f3f92d9d700 1 --2- 172.21.15.42:0/549154793 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3f7803abc0 0x7f3f7803d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:35:34.757 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:34.746+0000 7f3f92d9d700 1 --2- 172.21.15.42:0/549154793 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3f7803abc0 0x7f3f7803d0d0 crc :-1 s=READY pgs=1193 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:35:34.757 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:34.750+0000 7f3f98be4700 1 -- 172.21.15.42:0/549154793 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f9405d100 con 0x7f3f94127fe0 2019-12-05T06:35:34.759 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:34.754+0000 7f3f9359e700 1 --2- 172.21.15.42:0/549154793 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3f94127fe0 0x7f3f9412f430 secure :-1 s=THROTTLE_DONE pgs=2837 cs=0 l=1 rx=0x7f3f940a73b0 tx=0x7f3f8c010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:34.759 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:34.754+0000 7f3f8b7fe700 1 -- 172.21.15.42:0/549154793 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f3f94113d00 con 0x7f3f94127fe0 2019-12-05T06:35:34.919 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:34.914+0000 7f3f98be4700 1 -- 172.21.15.42:0/549154793 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3f9405b8b0 con 0x7f3f94127fe0 2019-12-05T06:35:34.920 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:34.918+0000 7f3f9359e700 1 --2- 172.21.15.42:0/549154793 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3f94127fe0 0x7f3f9412f430 secure :-1 s=THROTTLE_DONE pgs=2837 cs=0 l=1 rx=0x7f3f940a73b0 tx=0x7f3f8c010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:34.921 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:34.918+0000 7f3f8b7fe700 1 -- 172.21.15.42:0/549154793 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 338 v338) v1 ==== 95+0+2116 (secure 0 0 0) 0x7f3f8c0323f0 con 0x7f3f94127fe0 2019-12-05T06:35:34.921 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 338 2019-12-05T06:35:34.921 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:35:34.921 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":338,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7910,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/4253974783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":4253974783},{"type":"v1","addr":"172.21.15.40:6833","nonce":4253974783}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":338}],"filesystems":[{"mdsmap":{"epoch":337,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:33:30.171201+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":110,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11433},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11433":{"gid":11433,"name":"a","rank":0,"incarnation":334,"state":"up:active","state_seq":33,"addr":"172.21.15.42:6835/2532474042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2532474042},{"type":"v1","addr":"172.21.15.42:6835","nonce":2532474042}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:35:34.923 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:34.918+0000 7f3f98be4700 1 -- 172.21.15.42:0/549154793 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3f7803abc0 msgr2=0x7f3f7803d0d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:35:34.923 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:34.918+0000 7f3f98be4700 1 --2- 172.21.15.42:0/549154793 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3f7803abc0 0x7f3f7803d0d0 crc :-1 s=READY pgs=1193 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:35:34.923 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:34.918+0000 7f3f98be4700 1 -- 172.21.15.42:0/549154793 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3f94127fe0 msgr2=0x7f3f9412f430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:35:34.923 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:34.918+0000 7f3f98be4700 1 --2- 172.21.15.42:0/549154793 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3f94127fe0 0x7f3f9412f430 secure :-1 s=READY pgs=2837 cs=0 l=1 rx=0x7f3f940a73b0 tx=0x7f3f8c010b50).stop 2019-12-05T06:35:34.924 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:34.918+0000 7f3f98be4700 1 -- 172.21.15.42:0/549154793 shutdown_connections 2019-12-05T06:35:34.924 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:34.918+0000 7f3f98be4700 1 --2- 172.21.15.42:0/549154793 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3f7803abc0 0x7f3f7803d0d0 unknown :-1 s=CLOSED pgs=1193 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:35:34.924 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:34.918+0000 7f3f98be4700 1 --2- 172.21.15.42:0/549154793 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3f94127fe0 0x7f3f9412f430 unknown :-1 s=CLOSED pgs=2837 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:35:34.924 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:34.918+0000 7f3f98be4700 1 --2- 172.21.15.42:0/549154793 >> v2:172.21.15.40:3300/0 conn(0x7f3f940780e0 0x7f3f94117e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:35:34.925 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:34.918+0000 7f3f98be4700 1 -- 172.21.15.42:0/549154793 shutdown_connections 2019-12-05T06:35:34.925 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:34.918+0000 7f3f98be4700 1 -- 172.21.15.42:0/549154793 wait complete. 2019-12-05T06:35:34.925 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:34.918+0000 7f3f98be4700 1 -- 172.21.15.42:0/549154793 >> 172.21.15.42:0/549154793 conn(0x7f3f9408fdb0 msgr2=0x7f3f940a97d0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:35:34.933 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:35:36.934 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:35:37.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.026+0000 7fdfdf959700 1 Processor -- start 2019-12-05T06:35:37.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.026+0000 7fdfdf959700 1 -- start start 2019-12-05T06:35:37.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.026+0000 7fdfdf959700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fdfd80a7730 0x7fdfd808df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:35:37.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.026+0000 7fdfdf959700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fdfd809fad0 0x7fdfd807eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:35:37.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.026+0000 7fdfdf959700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfd80a0700 con 0x7fdfd807e710 2019-12-05T06:35:37.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.026+0000 7fdfdf959700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdfd80a0af0 con 0x7fdfd809fad0 2019-12-05T06:35:37.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.026+0000 7fdfdf959700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdfd808e5a0 con 0x7fdfd80a7730 2019-12-05T06:35:37.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.026+0000 7fdfddef6700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fdfd80a7730 0x7fdfd808df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:35:37.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.026+0000 7fdfddef6700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fdfd80a7730 0x7fdfd808df70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:58252/0 (socket says 172.21.15.42:58252) 2019-12-05T06:35:37.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.026+0000 7fdfddef6700 1 -- 172.21.15.42:0/500149480 learned_addr learned my addr 172.21.15.42:0/500149480 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:35:37.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.026+0000 7fdfdd6f5700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fdfd807e710 0x7fdfd809f2d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:58292/0 (socket says 172.21.15.42:58292) 2019-12-05T06:35:37.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.026+0000 7fdfde6f7700 1 --2- 172.21.15.42:0/500149480 >> v2:172.21.15.40:3300/0 conn(0x7fdfd809fad0 0x7fdfd807eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:35:37.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.026+0000 7fdfddef6700 1 -- 172.21.15.42:0/500149480 >> v2:172.21.15.40:3300/0 conn(0x7fdfd809fad0 msgr2=0x7fdfd807eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:35:37.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.026+0000 7fdfddef6700 1 --2- 172.21.15.42:0/500149480 >> v2:172.21.15.40:3300/0 conn(0x7fdfd809fad0 0x7fdfd807eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:35:37.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.026+0000 7fdfddef6700 1 -- 172.21.15.42:0/500149480 >> v1:172.21.15.42:6789/0 conn(0x7fdfd807e710 legacy=0x7fdfd809f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:35:37.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.026+0000 7fdfddef6700 1 -- 172.21.15.42:0/500149480 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfd807f090 con 0x7fdfd80a7730 2019-12-05T06:35:37.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.026+0000 7fdfdcef4700 1 -- 172.21.15.42:0/500149480 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3915155232 0 0) 0x7fdfd80a0700 con 0x7fdfd807e710 2019-12-05T06:35:37.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.026+0000 7fdfddef6700 1 --2- 172.21.15.42:0/500149480 >> v2:172.21.15.42:3300/0 conn(0x7fdfd80a7730 0x7fdfd808df70 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fdfd00032b0 tx=0x7fdfd00100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:37.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.026+0000 7fdfddef6700 1 --2- 172.21.15.42:0/500149480 >> v2:172.21.15.42:3300/0 conn(0x7fdfd80a7730 0x7fdfd808df70 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fdfd00032b0 tx=0x7fdfd00100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:37.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.026+0000 7fdfddef6700 1 --2- 172.21.15.42:0/500149480 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdfd80a7730 0x7fdfd808df70 secure :-1 s=READY pgs=2839 cs=0 l=1 rx=0x7fdfd00032b0 tx=0x7fdfd00100b0).ready entity=mon.1 client_cookie=b09b8d1aa62e1b81 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:35:37.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.026+0000 7fdfddef6700 1 --2- 172.21.15.42:0/500149480 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdfd80a7730 0x7fdfd808df70 secure :-1 s=THROTTLE_DONE pgs=2839 cs=0 l=1 rx=0x7fdfd00032b0 tx=0x7fdfd00100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:37.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.026+0000 7fdfdcef4700 1 -- 172.21.15.42:0/500149480 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdfd0010d60 con 0x7fdfd80a7730 2019-12-05T06:35:37.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.026+0000 7fdfddef6700 1 --2- 172.21.15.42:0/500149480 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdfd80a7730 0x7fdfd808df70 secure :-1 s=THROTTLE_DONE pgs=2839 cs=0 l=1 rx=0x7fdfd00032b0 tx=0x7fdfd00100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:37.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.026+0000 7fdfddef6700 1 --2- 172.21.15.42:0/500149480 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdfd80a7730 0x7fdfd808df70 secure :-1 s=THROTTLE_DONE pgs=2839 cs=0 l=1 rx=0x7fdfd00032b0 tx=0x7fdfd00100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:37.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.026+0000 7fdfdcef4700 1 -- 172.21.15.42:0/500149480 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfd0010980 con 0x7fdfd80a7730 2019-12-05T06:35:37.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.030+0000 7fdfdcef4700 1 -- 172.21.15.42:0/500149480 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdfd00114a0 con 0x7fdfd80a7730 2019-12-05T06:35:37.039 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.030+0000 7fdfdf959700 1 -- 172.21.15.42:0/500149480 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdfd80a7730 msgr2=0x7fdfd808df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:35:37.039 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.030+0000 7fdfdf959700 1 --2- 172.21.15.42:0/500149480 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdfd80a7730 0x7fdfd808df70 secure :-1 s=READY pgs=2839 cs=0 l=1 rx=0x7fdfd00032b0 tx=0x7fdfd00100b0).stop 2019-12-05T06:35:37.039 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.030+0000 7fdfdf959700 1 -- 172.21.15.42:0/500149480 shutdown_connections 2019-12-05T06:35:37.039 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.030+0000 7fdfdf959700 1 --2- 172.21.15.42:0/500149480 >> v2:172.21.15.40:3300/0 conn(0x7fdfd809fad0 0x7fdfd807eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:35:37.039 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.030+0000 7fdfdf959700 1 --2- 172.21.15.42:0/500149480 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdfd80a7730 0x7fdfd808df70 unknown :-1 s=CLOSED pgs=2839 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:35:37.039 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.030+0000 7fdfdf959700 1 -- 172.21.15.42:0/500149480 shutdown_connections 2019-12-05T06:35:37.040 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.030+0000 7fdfdf959700 1 -- 172.21.15.42:0/500149480 wait complete. 2019-12-05T06:35:37.040 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.030+0000 7fdfdf959700 1 -- 172.21.15.42:0/500149480 >> 172.21.15.42:0/500149480 conn(0x7fdfd80aabd0 msgr2=0x7fdfd80a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:35:37.044 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.030+0000 7fdfdf959700 1 Processor -- start 2019-12-05T06:35:37.044 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.030+0000 7fdfdf959700 1 -- start start 2019-12-05T06:35:37.044 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.034+0000 7fdfdf959700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fdfd8123590 0x7fdfd8070ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:35:37.044 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.034+0000 7fdfdf959700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfd8123ca0 con 0x7fdfd8116350 2019-12-05T06:35:37.044 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.034+0000 7fdfdf959700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfd8071000 con 0x7fdfd8123160 2019-12-05T06:35:37.045 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.034+0000 7fdfdf959700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdfd8121a90 con 0x7fdfd8123590 2019-12-05T06:35:37.045 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.034+0000 7fdfdd6f5700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fdfd8123160 0x7fdfd80702b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:58298/0 (socket says 172.21.15.42:58298) 2019-12-05T06:35:37.045 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.034+0000 7fdfdd6f5700 1 -- 172.21.15.42:0/3012068774 learned_addr learned my addr 172.21.15.42:0/3012068774 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:35:37.045 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.034+0000 7fdfde6f7700 1 --2- 172.21.15.42:0/3012068774 >> v2:172.21.15.40:3300/0 conn(0x7fdfd8123590 0x7fdfd8070ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:35:37.045 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.034+0000 7fdfce7fc700 1 -- 172.21.15.42:0/3012068774 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 883351839 0 0) 0x7fdfd8071000 con 0x7fdfd8123160 2019-12-05T06:35:37.045 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.034+0000 7fdfce7fc700 1 -- 172.21.15.42:0/3012068774 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdfc00030b0 con 0x7fdfd8123160 2019-12-05T06:35:37.046 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.034+0000 7fdfde6f7700 1 -- 172.21.15.42:0/3012068774 >> v1:172.21.15.42:6789/0 conn(0x7fdfd8123160 legacy=0x7fdfd80702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:35:37.046 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.034+0000 7fdfce7fc700 1 -- 172.21.15.42:0/3012068774 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2226177506 0 0) 0x7fdfd8123ca0 con 0x7fdfd8116350 2019-12-05T06:35:37.046 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.034+0000 7fdfde6f7700 1 -- 172.21.15.42:0/3012068774 >> v1:172.21.15.40:6789/0 conn(0x7fdfd8116350 legacy=0x7fdfd8118f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:35:37.046 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.034+0000 7fdfde6f7700 1 -- 172.21.15.42:0/3012068774 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfd8121110 con 0x7fdfd8123590 2019-12-05T06:35:37.046 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.034+0000 7fdfde6f7700 1 --2- 172.21.15.42:0/3012068774 >> v2:172.21.15.40:3300/0 conn(0x7fdfd8123590 0x7fdfd8070ab0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fdfd81222f0 tx=0x7fdfd4010950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:37.046 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.034+0000 7fdfde6f7700 1 --2- 172.21.15.42:0/3012068774 >> v2:172.21.15.40:3300/0 conn(0x7fdfd8123590 0x7fdfd8070ab0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fdfd81222f0 tx=0x7fdfd4010950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:37.046 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.034+0000 7fdfde6f7700 1 --2- 172.21.15.42:0/3012068774 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdfd8123590 0x7fdfd8070ab0 secure :-1 s=READY pgs=2138 cs=0 l=1 rx=0x7fdfd81222f0 tx=0x7fdfd4010950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:35:37.047 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.034+0000 7fdfde6f7700 1 --2- 172.21.15.42:0/3012068774 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdfd8123590 0x7fdfd8070ab0 secure :-1 s=THROTTLE_DONE pgs=2138 cs=0 l=1 rx=0x7fdfd81222f0 tx=0x7fdfd4010950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:37.047 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.034+0000 7fdfce7fc700 1 -- 172.21.15.42:0/3012068774 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdfd4005b70 con 0x7fdfd8123590 2019-12-05T06:35:37.047 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.034+0000 7fdfde6f7700 1 --2- 172.21.15.42:0/3012068774 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdfd8123590 0x7fdfd8070ab0 secure :-1 s=THROTTLE_DONE pgs=2138 cs=0 l=1 rx=0x7fdfd81222f0 tx=0x7fdfd4010950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:37.047 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.034+0000 7fdfde6f7700 1 --2- 172.21.15.42:0/3012068774 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdfd8123590 0x7fdfd8070ab0 secure :-1 s=THROTTLE_DONE pgs=2138 cs=0 l=1 rx=0x7fdfd81222f0 tx=0x7fdfd4010950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:37.047 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.034+0000 7fdfce7fc700 1 -- 172.21.15.42:0/3012068774 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfd40071a0 con 0x7fdfd8123590 2019-12-05T06:35:37.047 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.034+0000 7fdfdf959700 1 -- 172.21.15.42:0/3012068774 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfd8078530 con 0x7fdfd8123590 2019-12-05T06:35:37.048 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.034+0000 7fdfce7fc700 1 -- 172.21.15.42:0/3012068774 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdfd40062b0 con 0x7fdfd8123590 2019-12-05T06:35:37.048 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.034+0000 7fdfdf959700 1 -- 172.21.15.42:0/3012068774 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfd8078910 con 0x7fdfd8123590 2019-12-05T06:35:37.048 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.034+0000 7fdfde6f7700 1 --2- 172.21.15.42:0/3012068774 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdfd8123590 0x7fdfd8070ab0 secure :-1 s=THROTTLE_DONE pgs=2138 cs=0 l=1 rx=0x7fdfd81222f0 tx=0x7fdfd4010950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:37.048 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.034+0000 7fdfce7fc700 1 -- 172.21.15.42:0/3012068774 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fdfd4007ae0 con 0x7fdfd8123590 2019-12-05T06:35:37.048 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.034+0000 7fdfde6f7700 1 --2- 172.21.15.42:0/3012068774 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdfd8123590 0x7fdfd8070ab0 secure :-1 s=THROTTLE_DONE pgs=2138 cs=0 l=1 rx=0x7fdfd81222f0 tx=0x7fdfd4010950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:37.048 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.038+0000 7fdfce7fc700 1 --2- 172.21.15.42:0/3012068774 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fdfc003b210 0x7fdfc003d720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:35:37.049 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.038+0000 7fdfce7fc700 1 -- 172.21.15.42:0/3012068774 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 11363+0+0 (secure 0 0 0) 0x7fdfd404a940 con 0x7fdfd8123590 2019-12-05T06:35:37.049 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.038+0000 7fdfddef6700 1 --2- 172.21.15.42:0/3012068774 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fdfc003b210 0x7fdfc003d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:35:37.049 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.038+0000 7fdfddef6700 1 --2- 172.21.15.42:0/3012068774 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fdfc003b210 0x7fdfc003d720 crc :-1 s=READY pgs=1194 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:35:37.050 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.042+0000 7fdfdf959700 1 -- 172.21.15.42:0/3012068774 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfd805d100 con 0x7fdfd8123590 2019-12-05T06:35:37.053 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.050+0000 7fdfde6f7700 1 --2- 172.21.15.42:0/3012068774 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdfd8123590 0x7fdfd8070ab0 secure :-1 s=THROTTLE_DONE pgs=2138 cs=0 l=1 rx=0x7fdfd81222f0 tx=0x7fdfd4010950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:37.053 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.050+0000 7fdfce7fc700 1 -- 172.21.15.42:0/3012068774 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fdfd40319c0 con 0x7fdfd8123590 2019-12-05T06:35:37.215 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.210+0000 7fdfdf959700 1 -- 172.21.15.42:0/3012068774 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdfd805b8b0 con 0x7fdfd8123590 2019-12-05T06:35:37.216 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.210+0000 7fdfde6f7700 1 --2- 172.21.15.42:0/3012068774 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdfd8123590 0x7fdfd8070ab0 secure :-1 s=THROTTLE_DONE pgs=2138 cs=0 l=1 rx=0x7fdfd81222f0 tx=0x7fdfd4010950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:37.216 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.210+0000 7fdfce7fc700 1 -- 172.21.15.42:0/3012068774 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 339 v339) v1 ==== 95+0+1775 (secure 0 0 0) 0x7fdfd4031c20 con 0x7fdfd8123590 2019-12-05T06:35:37.216 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 339 2019-12-05T06:35:37.216 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:35:37.217 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":339,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":339,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:35:35.091849+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":111,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7910},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7910":{"gid":7910,"name":"b","rank":0,"incarnation":339,"state":"up:replay","state_seq":1,"addr":"172.21.15.40:6833/4253974783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":4253974783},{"type":"v1","addr":"172.21.15.40:6833","nonce":4253974783}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:35:37.218 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.214+0000 7fdfdf959700 1 -- 172.21.15.42:0/3012068774 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fdfc003b210 msgr2=0x7fdfc003d720 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:35:37.218 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.214+0000 7fdfdf959700 1 --2- 172.21.15.42:0/3012068774 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fdfc003b210 0x7fdfc003d720 crc :-1 s=READY pgs=1194 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:35:37.219 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.214+0000 7fdfdf959700 1 -- 172.21.15.42:0/3012068774 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdfd8123590 msgr2=0x7fdfd8070ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:35:37.219 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.214+0000 7fdfdf959700 1 --2- 172.21.15.42:0/3012068774 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdfd8123590 0x7fdfd8070ab0 secure :-1 s=READY pgs=2138 cs=0 l=1 rx=0x7fdfd81222f0 tx=0x7fdfd4010950).stop 2019-12-05T06:35:37.219 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.214+0000 7fdfdf959700 1 -- 172.21.15.42:0/3012068774 shutdown_connections 2019-12-05T06:35:37.219 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.214+0000 7fdfdf959700 1 --2- 172.21.15.42:0/3012068774 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fdfc003b210 0x7fdfc003d720 unknown :-1 s=CLOSED pgs=1194 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:35:37.219 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.214+0000 7fdfdf959700 1 --2- 172.21.15.42:0/3012068774 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdfd8123590 0x7fdfd8070ab0 unknown :-1 s=CLOSED pgs=2138 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:35:37.219 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.214+0000 7fdfdf959700 1 -- 172.21.15.42:0/3012068774 shutdown_connections 2019-12-05T06:35:37.220 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.214+0000 7fdfdf959700 1 -- 172.21.15.42:0/3012068774 wait complete. 2019-12-05T06:35:37.220 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.214+0000 7fdfdf959700 1 -- 172.21.15.42:0/3012068774 >> 172.21.15.42:0/3012068774 conn(0x7fdfd808fd90 msgr2=0x7fdfd80a9630 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:35:37.228 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a down, removed from mdsmap 2019-12-05T06:35:37.228 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2019-12-05T06:35:37.228 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:35:37.336 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.326+0000 7fb824837700 1 Processor -- start 2019-12-05T06:35:37.336 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.326+0000 7fb824837700 1 -- start start 2019-12-05T06:35:37.337 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.326+0000 7fb824837700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb8200a7740 0x7fb82008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:35:37.337 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.326+0000 7fb824837700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8200a0710 con 0x7fb82009fae0 2019-12-05T06:35:37.337 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.326+0000 7fb824837700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8200a9ba0 con 0x7fb82007e720 2019-12-05T06:35:37.337 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.326+0000 7fb824837700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8200a0b00 con 0x7fb8200a7740 2019-12-05T06:35:37.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.326+0000 7fb81659c700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fb82007e720 0x7fb82009f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:58306/0 (socket says 172.21.15.42:58306) 2019-12-05T06:35:37.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.326+0000 7fb81659c700 1 -- 172.21.15.42:0/3708457074 learned_addr learned my addr 172.21.15.42:0/3708457074 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:35:37.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.326+0000 7fb81ed9d700 1 --2- 172.21.15.42:0/3708457074 >> v2:172.21.15.40:3300/0 conn(0x7fb8200a7740 0x7fb82008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:35:37.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.326+0000 7fb81e59c700 1 -- 172.21.15.42:0/3708457074 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 64658016 0 0) 0x7fb8200a9ba0 con 0x7fb82007e720 2019-12-05T06:35:37.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.326+0000 7fb81e59c700 1 -- 172.21.15.42:0/3708457074 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb80c0030b0 con 0x7fb82007e720 2019-12-05T06:35:37.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.326+0000 7fb81e59c700 1 -- 172.21.15.42:0/3708457074 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4058455652 0 0) 0x7fb8200a0710 con 0x7fb82009fae0 2019-12-05T06:35:37.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.326+0000 7fb81e59c700 1 -- 172.21.15.42:0/3708457074 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8200a9ba0 con 0x7fb82009fae0 2019-12-05T06:35:37.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.326+0000 7fb81e59c700 1 -- 172.21.15.42:0/3708457074 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2136136423 0 0) 0x7fb80c0030b0 con 0x7fb82007e720 2019-12-05T06:35:37.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.326+0000 7fb81e59c700 1 -- 172.21.15.42:0/3708457074 >> v2:172.21.15.40:3300/0 conn(0x7fb8200a7740 msgr2=0x7fb82008df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:35:37.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.326+0000 7fb81e59c700 1 --2- 172.21.15.42:0/3708457074 >> v2:172.21.15.40:3300/0 conn(0x7fb8200a7740 0x7fb82008df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:35:37.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.326+0000 7fb81e59c700 1 -- 172.21.15.42:0/3708457074 >> v1:172.21.15.40:6789/0 conn(0x7fb82009fae0 legacy=0x7fb82014efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:35:37.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.330+0000 7fb81e59c700 1 -- 172.21.15.42:0/3708457074 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb82007eb50 con 0x7fb82007e720 2019-12-05T06:35:37.341 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.330+0000 7fb81e59c700 1 -- 172.21.15.42:0/3708457074 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7fb808001750 con 0x7fb82007e720 2019-12-05T06:35:37.341 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.330+0000 7fb81e59c700 1 -- 172.21.15.42:0/3708457074 >> v1:172.21.15.42:6789/0 conn(0x7fb82007e720 legacy=0x7fb82009f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:35:37.341 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.330+0000 7fb81e59c700 1 --2- 172.21.15.42:0/3708457074 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb80c003960 0x7fb80c003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:35:37.341 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.330+0000 7fb81e59c700 1 --2- 172.21.15.42:0/3708457074 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb80c004470 0x7fb80c004940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:35:37.342 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.330+0000 7fb81659c700 1 --2- 172.21.15.42:0/3708457074 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb80c004470 0x7fb80c004940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:35:37.342 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.330+0000 7fb81e59c700 1 --2- 172.21.15.42:0/3708457074 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb80c004fa0 0x7fb80c04f700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:35:37.343 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.330+0000 7fb81ed9d700 1 --2- 172.21.15.42:0/3708457074 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb80c003960 0x7fb80c003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:35:37.343 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.330+0000 7fb81e59c700 1 -- 172.21.15.42:0/3708457074 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8200a0b00 con 0x7fb80c003960 2019-12-05T06:35:37.343 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.330+0000 7fb81e59c700 1 -- 172.21.15.42:0/3708457074 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb80c04fec0 con 0x7fb80c004470 2019-12-05T06:35:37.344 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.330+0000 7fb81e59c700 1 -- 172.21.15.42:0/3708457074 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb80c050100 con 0x7fb80c004fa0 2019-12-05T06:35:37.344 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.330+0000 7fb81f59e700 1 --2- 172.21.15.42:0/3708457074 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb80c004fa0 0x7fb80c04f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:35:37.344 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.330+0000 7fb81ed9d700 1 -- 172.21.15.42:0/3708457074 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb80c004fa0 msgr2=0x7fb80c04f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:35:37.344 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.330+0000 7fb81ed9d700 1 --2- 172.21.15.42:0/3708457074 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb80c004fa0 0x7fb80c04f700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:35:37.345 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.330+0000 7fb81ed9d700 1 -- 172.21.15.42:0/3708457074 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb80c004470 msgr2=0x7fb80c004940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:35:37.345 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.330+0000 7fb81ed9d700 1 --2- 172.21.15.42:0/3708457074 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb80c004470 0x7fb80c004940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:35:37.345 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.330+0000 7fb81f59e700 1 -- 172.21.15.42:0/3708457074 reap_dead start 2019-12-05T06:35:37.345 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.330+0000 7fb81ed9d700 1 -- 172.21.15.42:0/3708457074 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb80c050400 con 0x7fb80c003960 2019-12-05T06:35:37.347 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.330+0000 7fb81659c700 1 --2- 172.21.15.42:0/3708457074 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb80c004470 0x7fb80c004940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:35:37.347 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.330+0000 7fb81ed9d700 1 --2- 172.21.15.42:0/3708457074 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb80c003960 0x7fb80c003db0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fb810002f30 tx=0x7fb81000ffe0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:37.347 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.330+0000 7fb81ed9d700 1 --2- 172.21.15.42:0/3708457074 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb80c003960 0x7fb80c003db0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fb810002f30 tx=0x7fb81000ffe0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:37.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.330+0000 7fb81ed9d700 1 --2- 172.21.15.42:0/3708457074 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb80c003960 0x7fb80c003db0 secure :-1 s=READY pgs=2140 cs=0 l=1 rx=0x7fb810002f30 tx=0x7fb81000ffe0).ready entity=mon.0 client_cookie=4e2ece728d2d9b64 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:35:37.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.330+0000 7fb81ed9d700 1 --2- 172.21.15.42:0/3708457074 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb80c003960 0x7fb80c003db0 secure :-1 s=THROTTLE_DONE pgs=2140 cs=0 l=1 rx=0x7fb810002f30 tx=0x7fb81000ffe0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:37.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.330+0000 7fb81e59c700 1 -- 172.21.15.42:0/3708457074 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb810010d10 con 0x7fb80c003960 2019-12-05T06:35:37.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.330+0000 7fb81ed9d700 1 --2- 172.21.15.42:0/3708457074 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb80c003960 0x7fb80c003db0 secure :-1 s=THROTTLE_DONE pgs=2140 cs=0 l=1 rx=0x7fb810002f30 tx=0x7fb81000ffe0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:37.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.330+0000 7fb81ed9d700 1 --2- 172.21.15.42:0/3708457074 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb80c003960 0x7fb80c003db0 secure :-1 s=THROTTLE_DONE pgs=2140 cs=0 l=1 rx=0x7fb810002f30 tx=0x7fb81000ffe0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:37.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.330+0000 7fb81e59c700 1 -- 172.21.15.42:0/3708457074 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb810010930 con 0x7fb80c003960 2019-12-05T06:35:37.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.330+0000 7fb81e59c700 1 -- 172.21.15.42:0/3708457074 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb810008bd0 con 0x7fb80c003960 2019-12-05T06:35:37.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.330+0000 7fb824837700 1 -- 172.21.15.42:0/3708457074 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb80c003960 msgr2=0x7fb80c003db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:35:37.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.330+0000 7fb824837700 1 --2- 172.21.15.42:0/3708457074 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb80c003960 0x7fb80c003db0 secure :-1 s=READY pgs=2140 cs=0 l=1 rx=0x7fb810002f30 tx=0x7fb81000ffe0).stop 2019-12-05T06:35:37.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.330+0000 7fb824837700 1 -- 172.21.15.42:0/3708457074 shutdown_connections 2019-12-05T06:35:37.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.330+0000 7fb824837700 1 --2- 172.21.15.42:0/3708457074 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb80c003960 0x7fb80c003db0 unknown :-1 s=CLOSED pgs=2140 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:35:37.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.330+0000 7fb824837700 1 -- 172.21.15.42:0/3708457074 shutdown_connections 2019-12-05T06:35:37.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.330+0000 7fb824837700 1 -- 172.21.15.42:0/3708457074 wait complete. 2019-12-05T06:35:37.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.330+0000 7fb824837700 1 -- 172.21.15.42:0/3708457074 >> 172.21.15.42:0/3708457074 conn(0x7fb8200aabe0 msgr2=0x7fb8200a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:35:37.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.334+0000 7fb824837700 1 Processor -- start 2019-12-05T06:35:37.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.334+0000 7fb824837700 1 -- start start 2019-12-05T06:35:37.358 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.334+0000 7fb824837700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb82007a9f0 0x7fb820077c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:35:37.358 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.334+0000 7fb824837700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8200795a0 con 0x7fb820078180 2019-12-05T06:35:37.358 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.334+0000 7fb824837700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb82007bb80 con 0x7fb82011c500 2019-12-05T06:35:37.358 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.334+0000 7fb824837700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8201455b0 con 0x7fb82007a9f0 2019-12-05T06:35:37.359 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.334+0000 7fb81ed9d700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fb82011c500 0x7fb82007b380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:58316/0 (socket says 172.21.15.42:58316) 2019-12-05T06:35:37.359 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.334+0000 7fb81ed9d700 1 -- 172.21.15.42:0/1958806479 learned_addr learned my addr 172.21.15.42:0/1958806479 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:35:37.359 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.334+0000 7fb81659c700 1 --2- 172.21.15.42:0/1958806479 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb82007a9f0 0x7fb820077c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:35:37.359 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.334+0000 7fb817fff700 1 -- 172.21.15.42:0/1958806479 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3009783725 0 0) 0x7fb82007bb80 con 0x7fb82011c500 2019-12-05T06:35:37.360 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.334+0000 7fb817fff700 1 -- 172.21.15.42:0/1958806479 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8040030b0 con 0x7fb82011c500 2019-12-05T06:35:37.360 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.334+0000 7fb817fff700 1 -- 172.21.15.42:0/1958806479 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2033102746 0 0) 0x7fb8200795a0 con 0x7fb820078180 2019-12-05T06:35:37.360 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.334+0000 7fb817fff700 1 -- 172.21.15.42:0/1958806479 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb82007bb80 con 0x7fb820078180 2019-12-05T06:35:37.360 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.334+0000 7fb81659c700 1 -- 172.21.15.42:0/1958806479 >> v1:172.21.15.42:6789/0 conn(0x7fb82011c500 legacy=0x7fb82007b380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:35:37.361 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.334+0000 7fb81659c700 1 -- 172.21.15.42:0/1958806479 >> v1:172.21.15.40:6789/0 conn(0x7fb820078180 legacy=0x7fb820134b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:35:37.361 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.334+0000 7fb81659c700 1 -- 172.21.15.42:0/1958806479 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8200785b0 con 0x7fb82007a9f0 2019-12-05T06:35:37.361 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.334+0000 7fb81659c700 1 --2- 172.21.15.42:0/1958806479 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb82007a9f0 0x7fb820077c30 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fb808001520 tx=0x7fb80800c490).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:37.361 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.334+0000 7fb81659c700 1 --2- 172.21.15.42:0/1958806479 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb82007a9f0 0x7fb820077c30 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fb808001520 tx=0x7fb80800c490).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:37.362 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.334+0000 7fb81659c700 1 --2- 172.21.15.42:0/1958806479 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb82007a9f0 0x7fb820077c30 secure :-1 s=READY pgs=627 cs=0 l=1 rx=0x7fb808001520 tx=0x7fb80800c490).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:35:37.362 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.334+0000 7fb81659c700 1 --2- 172.21.15.42:0/1958806479 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb82007a9f0 0x7fb820077c30 secure :-1 s=THROTTLE_DONE pgs=627 cs=0 l=1 rx=0x7fb808001520 tx=0x7fb80800c490).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:37.362 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.334+0000 7fb817fff700 1 -- 172.21.15.42:0/1958806479 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb80800ec60 con 0x7fb82007a9f0 2019-12-05T06:35:37.362 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.334+0000 7fb81659c700 1 --2- 172.21.15.42:0/1958806479 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb82007a9f0 0x7fb820077c30 secure :-1 s=THROTTLE_DONE pgs=627 cs=0 l=1 rx=0x7fb808001520 tx=0x7fb80800c490).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:37.362 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.334+0000 7fb81659c700 1 --2- 172.21.15.42:0/1958806479 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb82007a9f0 0x7fb820077c30 secure :-1 s=THROTTLE_DONE pgs=627 cs=0 l=1 rx=0x7fb808001520 tx=0x7fb80800c490).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:37.363 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.334+0000 7fb817fff700 1 -- 172.21.15.42:0/1958806479 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb80800e880 con 0x7fb82007a9f0 2019-12-05T06:35:37.363 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.334+0000 7fb824837700 1 -- 172.21.15.42:0/1958806479 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb820078880 con 0x7fb82007a9f0 2019-12-05T06:35:37.363 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.334+0000 7fb817fff700 1 -- 172.21.15.42:0/1958806479 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb808012a60 con 0x7fb82007a9f0 2019-12-05T06:35:37.363 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.334+0000 7fb824837700 1 -- 172.21.15.42:0/1958806479 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb820136350 con 0x7fb82007a9f0 2019-12-05T06:35:37.364 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.338+0000 7fb81659c700 1 --2- 172.21.15.42:0/1958806479 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb82007a9f0 0x7fb820077c30 secure :-1 s=THROTTLE_DONE pgs=627 cs=0 l=1 rx=0x7fb808001520 tx=0x7fb80800c490).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:37.364 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.338+0000 7fb817fff700 1 -- 172.21.15.42:0/1958806479 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fb808005450 con 0x7fb82007a9f0 2019-12-05T06:35:37.364 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.338+0000 7fb81659c700 1 --2- 172.21.15.42:0/1958806479 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb82007a9f0 0x7fb820077c30 secure :-1 s=THROTTLE_DONE pgs=627 cs=0 l=1 rx=0x7fb808001520 tx=0x7fb80800c490).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:37.364 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.338+0000 7fb817fff700 1 --2- 172.21.15.42:0/1958806479 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb80403b490 0x7fb80403d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:35:37.365 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.338+0000 7fb817fff700 1 -- 172.21.15.42:0/1958806479 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 11363+0+0 (secure 0 0 0) 0x7fb80800b5c0 con 0x7fb82007a9f0 2019-12-05T06:35:37.365 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.338+0000 7fb81ed9d700 1 --2- 172.21.15.42:0/1958806479 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb80403b490 0x7fb80403d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:35:37.365 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.338+0000 7fb81ed9d700 1 --2- 172.21.15.42:0/1958806479 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb80403b490 0x7fb80403d9a0 crc :-1 s=READY pgs=1195 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:35:37.365 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.342+0000 7fb824837700 1 -- 172.21.15.42:0/1958806479 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb82005d110 con 0x7fb82007a9f0 2019-12-05T06:35:37.366 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.350+0000 7fb81659c700 1 --2- 172.21.15.42:0/1958806479 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb82007a9f0 0x7fb820077c30 secure :-1 s=THROTTLE_DONE pgs=627 cs=0 l=1 rx=0x7fb808001520 tx=0x7fb80800c490).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:37.366 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.350+0000 7fb817fff700 1 -- 172.21.15.42:0/1958806479 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fb808032230 con 0x7fb82007a9f0 2019-12-05T06:35:37.528 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.522+0000 7fb824837700 1 -- 172.21.15.42:0/1958806479 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb82005ae30 con 0x7fb82007a9f0 2019-12-05T06:35:37.528 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.522+0000 7fb81659c700 1 --2- 172.21.15.42:0/1958806479 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb82007a9f0 0x7fb820077c30 secure :-1 s=THROTTLE_DONE pgs=627 cs=0 l=1 rx=0x7fb808001520 tx=0x7fb80800c490).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:37.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.522+0000 7fb817fff700 1 -- 172.21.15.42:0/1958806479 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 339 v339) v1 ==== 95+0+1775 (secure 0 0 0) 0x7fb808032490 con 0x7fb82007a9f0 2019-12-05T06:35:37.529 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 339 2019-12-05T06:35:37.529 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:35:37.529 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":339,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":339,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:35:35.091849+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":111,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7910},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7910":{"gid":7910,"name":"b","rank":0,"incarnation":339,"state":"up:replay","state_seq":1,"addr":"172.21.15.40:6833/4253974783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":4253974783},{"type":"v1","addr":"172.21.15.40:6833","nonce":4253974783}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:35:37.532 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.526+0000 7fb824837700 1 -- 172.21.15.42:0/1958806479 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb80403b490 msgr2=0x7fb80403d9a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:35:37.532 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.526+0000 7fb824837700 1 --2- 172.21.15.42:0/1958806479 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb80403b490 0x7fb80403d9a0 crc :-1 s=READY pgs=1195 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:35:37.532 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.526+0000 7fb824837700 1 -- 172.21.15.42:0/1958806479 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb82007a9f0 msgr2=0x7fb820077c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:35:37.533 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.526+0000 7fb824837700 1 --2- 172.21.15.42:0/1958806479 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb82007a9f0 0x7fb820077c30 secure :-1 s=READY pgs=627 cs=0 l=1 rx=0x7fb808001520 tx=0x7fb80800c490).stop 2019-12-05T06:35:37.533 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.526+0000 7fb824837700 1 -- 172.21.15.42:0/1958806479 shutdown_connections 2019-12-05T06:35:37.533 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.526+0000 7fb824837700 1 --2- 172.21.15.42:0/1958806479 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb80403b490 0x7fb80403d9a0 unknown :-1 s=CLOSED pgs=1195 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:35:37.533 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.526+0000 7fb824837700 1 --2- 172.21.15.42:0/1958806479 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb82007a9f0 0x7fb820077c30 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:35:37.534 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.526+0000 7fb824837700 1 -- 172.21.15.42:0/1958806479 shutdown_connections 2019-12-05T06:35:37.534 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.526+0000 7fb824837700 1 -- 172.21.15.42:0/1958806479 wait complete. 2019-12-05T06:35:37.534 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:37.526+0000 7fb824837700 1 -- 172.21.15.42:0/1958806479 >> 172.21.15.42:0/1958806479 conn(0x7fb82007df10 msgr2=0x7fb82007c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:35:37.540 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 339, "feature_flags": { "enable_multiple": false, "ever_enabled_multiple": false }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2019-12-05T05:03:35.604738+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 339, "ever_allowed_features": 2, "explicitly_allowed_features": 2, "failed": [], "flags": 18, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_7910": { "addr": "172.21.15.40:6833/4253974783", "addrs": { "addrvec": [ { "addr": "172.21.15.40:6832", "nonce": 4253974783, "type": "v2" }, { "addr": "172.21.15.40:6833", "nonce": 4253974783, "type": "v1" } ] }, "export_targets": [], "features": 4540138292836696063, "flags": 0, "gid": 7910, "incarnation": 339, "name": "b", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 111, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 2, "min_compat_client": "0 (unknown)", "modified": "2019-12-05T06:35:35.091849+0000", "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7910 } } } ], "standbys": [] } 2019-12-05T06:35:39.541 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:35:39.646 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:39.638+0000 7f58842f0700 1 Processor -- start 2019-12-05T06:35:39.647 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:39.638+0000 7f58842f0700 1 -- start start 2019-12-05T06:35:39.647 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:39.638+0000 7f58842f0700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f587c0a7740 0x7f587c08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:35:39.647 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:39.638+0000 7f58842f0700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f587c09fae0 0x7f587c07eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:35:39.647 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:39.638+0000 7f58842f0700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f587c0a0710 con 0x7f587c07e720 2019-12-05T06:35:39.647 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:39.638+0000 7f58842f0700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f587c0a0b00 con 0x7f587c0a7740 2019-12-05T06:35:39.647 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:39.638+0000 7f58842f0700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f587c08e5b0 con 0x7f587c09fae0 2019-12-05T06:35:39.647 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:39.638+0000 7f588308e700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f587c09fae0 0x7f587c07eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:35:39.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:39.638+0000 7f588288d700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f587c0a7740 0x7f587c08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:35:39.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:39.638+0000 7f588308e700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f587c09fae0 0x7f587c07eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:58760/0 (socket says 172.21.15.42:58760) 2019-12-05T06:35:39.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:39.638+0000 7f588308e700 1 -- 172.21.15.42:0/2132265533 learned_addr learned my addr 172.21.15.42:0/2132265533 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:35:39.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:39.638+0000 7f588288d700 1 --2- 172.21.15.42:0/2132265533 >> v2:172.21.15.40:3300/0 conn(0x7f587c0a7740 0x7f587c08df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.42:46106/0 (socket says 172.21.15.42:46106) 2019-12-05T06:35:39.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:39.638+0000 7f588308e700 1 -- 172.21.15.42:0/2132265533 >> v2:172.21.15.40:3300/0 conn(0x7f587c0a7740 msgr2=0x7f587c08df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:35:39.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:39.638+0000 7f588308e700 1 --2- 172.21.15.42:0/2132265533 >> v2:172.21.15.40:3300/0 conn(0x7f587c0a7740 0x7f587c08df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:35:39.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:39.638+0000 7f588308e700 1 -- 172.21.15.42:0/2132265533 >> v1:172.21.15.40:6789/0 conn(0x7f587c07e720 legacy=0x7f587c09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:35:39.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:39.638+0000 7f588308e700 1 -- 172.21.15.42:0/2132265533 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f587c07f0a0 con 0x7f587c09fae0 2019-12-05T06:35:39.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:39.638+0000 7f588308e700 1 --2- 172.21.15.42:0/2132265533 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f587c09fae0 0x7f587c07eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f586c008790 tx=0x7f586c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:39.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:39.638+0000 7f588308e700 1 --2- 172.21.15.42:0/2132265533 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f587c09fae0 0x7f587c07eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f586c008790 tx=0x7f586c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:39.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:39.638+0000 7f588308e700 1 --2- 172.21.15.42:0/2132265533 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f587c09fae0 0x7f587c07eb50 secure :-1 s=READY pgs=628 cs=0 l=1 rx=0x7f586c008790 tx=0x7f586c00fd30).ready entity=mon.2 client_cookie=a14380a7285b57e1 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:35:39.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:39.642+0000 7f588308e700 1 --2- 172.21.15.42:0/2132265533 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f587c09fae0 0x7f587c07eb50 secure :-1 s=THROTTLE_DONE pgs=628 cs=0 l=1 rx=0x7f586c008790 tx=0x7f586c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:39.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:39.642+0000 7f588188b700 1 -- 172.21.15.42:0/2132265533 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f586c008c90 con 0x7f587c09fae0 2019-12-05T06:35:39.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:39.642+0000 7f588308e700 1 --2- 172.21.15.42:0/2132265533 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f587c09fae0 0x7f587c07eb50 secure :-1 s=THROTTLE_DONE pgs=628 cs=0 l=1 rx=0x7f586c008790 tx=0x7f586c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:39.650 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:39.642+0000 7f588308e700 1 --2- 172.21.15.42:0/2132265533 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f587c09fae0 0x7f587c07eb50 secure :-1 s=THROTTLE_DONE pgs=628 cs=0 l=1 rx=0x7f586c008790 tx=0x7f586c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:39.650 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:39.642+0000 7f588188b700 1 -- 172.21.15.42:0/2132265533 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f586c0088b0 con 0x7f587c09fae0 2019-12-05T06:35:39.650 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:39.642+0000 7f588188b700 1 -- 172.21.15.42:0/2132265533 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f586c011b10 con 0x7f587c09fae0 2019-12-05T06:35:39.650 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:39.642+0000 7f58842f0700 1 -- 172.21.15.42:0/2132265533 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f587c09fae0 msgr2=0x7f587c07eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:35:39.650 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:39.642+0000 7f58842f0700 1 --2- 172.21.15.42:0/2132265533 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f587c09fae0 0x7f587c07eb50 secure :-1 s=READY pgs=628 cs=0 l=1 rx=0x7f586c008790 tx=0x7f586c00fd30).stop 2019-12-05T06:35:39.650 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:39.642+0000 7f58842f0700 1 -- 172.21.15.42:0/2132265533 shutdown_connections 2019-12-05T06:35:39.650 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:39.642+0000 7f58842f0700 1 --2- 172.21.15.42:0/2132265533 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f587c09fae0 0x7f587c07eb50 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:35:39.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:39.642+0000 7f58842f0700 1 --2- 172.21.15.42:0/2132265533 >> v2:172.21.15.40:3300/0 conn(0x7f587c0a7740 0x7f587c08df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:35:39.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:39.642+0000 7f58842f0700 1 -- 172.21.15.42:0/2132265533 shutdown_connections 2019-12-05T06:35:39.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:39.642+0000 7f58842f0700 1 -- 172.21.15.42:0/2132265533 wait complete. 2019-12-05T06:35:39.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:39.642+0000 7f58842f0700 1 -- 172.21.15.42:0/2132265533 >> 172.21.15.42:0/2132265533 conn(0x7f587c0aabe0 msgr2=0x7f587c0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:35:39.655 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:39.642+0000 7f58842f0700 1 Processor -- start 2019-12-05T06:35:39.655 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:39.646+0000 7f58842f0700 1 -- start start 2019-12-05T06:35:39.655 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:39.646+0000 7f58842f0700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f587c0a4a70 0x7f587c1381d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:35:39.655 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:39.646+0000 7f58842f0700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f587c0a4ea0 con 0x7f587c1120f0 2019-12-05T06:35:39.656 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:39.646+0000 7f58842f0700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f587c12fd70 con 0x7f587c1182f0 2019-12-05T06:35:39.656 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:39.646+0000 7f58842f0700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f587c0a51f0 con 0x7f587c0a4a70 2019-12-05T06:35:39.656 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:39.646+0000 7f588308e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f587c0a4a70 0x7f587c1381d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:35:39.656 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:39.646+0000 7f588208c700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f587c1182f0 0x7f587c11a730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:58336/0 (socket says 172.21.15.42:58336) 2019-12-05T06:35:39.656 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:39.646+0000 7f588208c700 1 -- 172.21.15.42:0/3404098597 learned_addr learned my addr 172.21.15.42:0/3404098597 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:35:39.657 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:39.646+0000 7f58737fe700 1 -- 172.21.15.42:0/3404098597 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2131626944 0 0) 0x7f587c0a4ea0 con 0x7f587c1120f0 2019-12-05T06:35:39.657 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:39.646+0000 7f58737fe700 1 -- 172.21.15.42:0/3404098597 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58640030b0 con 0x7f587c1120f0 2019-12-05T06:35:39.657 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:39.646+0000 7f588308e700 1 -- 172.21.15.42:0/3404098597 >> v1:172.21.15.42:6789/0 conn(0x7f587c1182f0 legacy=0x7f587c11a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:35:39.657 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:39.646+0000 7f588308e700 1 -- 172.21.15.42:0/3404098597 >> v1:172.21.15.40:6789/0 conn(0x7f587c1120f0 legacy=0x7f587c130500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:35:39.657 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:39.646+0000 7f588308e700 1 -- 172.21.15.42:0/3404098597 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f587c11af30 con 0x7f587c0a4a70 2019-12-05T06:35:39.657 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:39.646+0000 7f588308e700 1 --2- 172.21.15.42:0/3404098597 >> v2:172.21.15.42:3300/0 conn(0x7f587c0a4a70 0x7f587c1381d0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f586c008790 tx=0x7f586c008ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:39.657 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:39.646+0000 7f588308e700 1 --2- 172.21.15.42:0/3404098597 >> v2:172.21.15.42:3300/0 conn(0x7f587c0a4a70 0x7f587c1381d0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f586c008790 tx=0x7f586c008ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:39.658 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:39.646+0000 7f588308e700 1 --2- 172.21.15.42:0/3404098597 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f587c0a4a70 0x7f587c1381d0 secure :-1 s=READY pgs=2844 cs=0 l=1 rx=0x7f586c008790 tx=0x7f586c008ae0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:35:39.658 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:39.646+0000 7f588308e700 1 --2- 172.21.15.42:0/3404098597 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f587c0a4a70 0x7f587c1381d0 secure :-1 s=THROTTLE_DONE pgs=2844 cs=0 l=1 rx=0x7f586c008790 tx=0x7f586c008ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:39.658 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:39.646+0000 7f58737fe700 1 -- 172.21.15.42:0/3404098597 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f586c00f470 con 0x7f587c0a4a70 2019-12-05T06:35:39.658 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:39.646+0000 7f588308e700 1 --2- 172.21.15.42:0/3404098597 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f587c0a4a70 0x7f587c1381d0 secure :-1 s=THROTTLE_DONE pgs=2844 cs=0 l=1 rx=0x7f586c008790 tx=0x7f586c008ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:39.658 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:39.646+0000 7f588308e700 1 --2- 172.21.15.42:0/3404098597 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f587c0a4a70 0x7f587c1381d0 secure :-1 s=THROTTLE_DONE pgs=2844 cs=0 l=1 rx=0x7f586c008790 tx=0x7f586c008ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:39.658 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:39.646+0000 7f58737fe700 1 -- 172.21.15.42:0/3404098597 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f586c011730 con 0x7f587c0a4a70 2019-12-05T06:35:39.658 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:39.646+0000 7f58842f0700 1 -- 172.21.15.42:0/3404098597 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f587c0a9790 con 0x7f587c0a4a70 2019-12-05T06:35:39.659 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:39.646+0000 7f58737fe700 1 -- 172.21.15.42:0/3404098597 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f586c005730 con 0x7f587c0a4a70 2019-12-05T06:35:39.659 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:39.646+0000 7f58842f0700 1 -- 172.21.15.42:0/3404098597 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f587c0a9c90 con 0x7f587c0a4a70 2019-12-05T06:35:39.659 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:39.650+0000 7f588308e700 1 --2- 172.21.15.42:0/3404098597 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f587c0a4a70 0x7f587c1381d0 secure :-1 s=THROTTLE_DONE pgs=2844 cs=0 l=1 rx=0x7f586c008790 tx=0x7f586c008ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:39.659 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:39.650+0000 7f58737fe700 1 -- 172.21.15.42:0/3404098597 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f586c005a00 con 0x7f587c0a4a70 2019-12-05T06:35:39.659 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:39.650+0000 7f588308e700 1 --2- 172.21.15.42:0/3404098597 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f587c0a4a70 0x7f587c1381d0 secure :-1 s=THROTTLE_DONE pgs=2844 cs=0 l=1 rx=0x7f586c008790 tx=0x7f586c008ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:39.659 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:39.650+0000 7f58737fe700 1 --2- 172.21.15.42:0/3404098597 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f586403c320 0x7f586403e830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:35:39.659 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:39.650+0000 7f58737fe700 1 -- 172.21.15.42:0/3404098597 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 11363+0+0 (secure 0 0 0) 0x7f586c04b690 con 0x7f587c0a4a70 2019-12-05T06:35:39.660 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:39.650+0000 7f588288d700 1 --2- 172.21.15.42:0/3404098597 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f586403c320 0x7f586403e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:35:39.660 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:39.650+0000 7f588288d700 1 --2- 172.21.15.42:0/3404098597 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f586403c320 0x7f586403e830 crc :-1 s=READY pgs=1196 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:35:39.660 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:39.650+0000 7f58842f0700 1 -- 172.21.15.42:0/3404098597 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f587c05d110 con 0x7f587c0a4a70 2019-12-05T06:35:39.662 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:39.658+0000 7f588308e700 1 --2- 172.21.15.42:0/3404098597 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f587c0a4a70 0x7f587c1381d0 secure :-1 s=THROTTLE_DONE pgs=2844 cs=0 l=1 rx=0x7f586c008790 tx=0x7f586c008ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:39.663 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:39.658+0000 7f58737fe700 1 -- 172.21.15.42:0/3404098597 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f586c032970 con 0x7f587c0a4a70 2019-12-05T06:35:39.838 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:39.834+0000 7f58842f0700 1 -- 172.21.15.42:0/3404098597 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f587c05b8c0 con 0x7f587c0a4a70 2019-12-05T06:35:39.839 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:39.834+0000 7f588308e700 1 --2- 172.21.15.42:0/3404098597 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f587c0a4a70 0x7f587c1381d0 secure :-1 s=THROTTLE_DONE pgs=2844 cs=0 l=1 rx=0x7f586c008790 tx=0x7f586c008ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:39.839 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:39.834+0000 7f58737fe700 1 -- 172.21.15.42:0/3404098597 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 341 v341) v1 ==== 95+0+1776 (secure 0 0 0) 0x7f586c005d10 con 0x7f587c0a4a70 2019-12-05T06:35:39.839 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 341 2019-12-05T06:35:39.840 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:35:39.840 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":341,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":341,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:35:39.277824+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":111,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7910},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7910":{"gid":7910,"name":"b","rank":0,"incarnation":339,"state":"up:rejoin","state_seq":32,"addr":"172.21.15.40:6833/4253974783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":4253974783},{"type":"v1","addr":"172.21.15.40:6833","nonce":4253974783}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:35:39.842 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:39.838+0000 7f58842f0700 1 -- 172.21.15.42:0/3404098597 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f586403c320 msgr2=0x7f586403e830 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:35:39.842 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:39.838+0000 7f58842f0700 1 --2- 172.21.15.42:0/3404098597 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f586403c320 0x7f586403e830 crc :-1 s=READY pgs=1196 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:35:39.843 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:39.838+0000 7f58842f0700 1 -- 172.21.15.42:0/3404098597 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f587c0a4a70 msgr2=0x7f587c1381d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:35:39.843 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:39.838+0000 7f58842f0700 1 --2- 172.21.15.42:0/3404098597 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f587c0a4a70 0x7f587c1381d0 secure :-1 s=READY pgs=2844 cs=0 l=1 rx=0x7f586c008790 tx=0x7f586c008ae0).stop 2019-12-05T06:35:39.843 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:39.838+0000 7f58842f0700 1 -- 172.21.15.42:0/3404098597 shutdown_connections 2019-12-05T06:35:39.843 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:39.838+0000 7f58842f0700 1 --2- 172.21.15.42:0/3404098597 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f586403c320 0x7f586403e830 unknown :-1 s=CLOSED pgs=1196 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:35:39.843 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:39.838+0000 7f58842f0700 1 --2- 172.21.15.42:0/3404098597 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f587c0a4a70 0x7f587c1381d0 unknown :-1 s=CLOSED pgs=2844 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:35:39.843 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:39.838+0000 7f58842f0700 1 -- 172.21.15.42:0/3404098597 shutdown_connections 2019-12-05T06:35:39.843 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:39.838+0000 7f58842f0700 1 -- 172.21.15.42:0/3404098597 wait complete. 2019-12-05T06:35:39.844 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:39.838+0000 7f58842f0700 1 -- 172.21.15.42:0/3404098597 >> 172.21.15.42:0/3404098597 conn(0x7f587c0aabe0 msgr2=0x7f587c118880 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:35:39.854 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2019-12-05T06:35:41.855 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:35:41.961 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:41.950+0000 7f544cd52700 1 Processor -- start 2019-12-05T06:35:41.961 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:41.950+0000 7f544cd52700 1 -- start start 2019-12-05T06:35:41.961 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:41.954+0000 7f544cd52700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f54480a7740 0x7f544808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:35:41.961 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:41.954+0000 7f544cd52700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f544807e720 0x7f544807eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:35:41.962 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:41.954+0000 7f544cd52700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54480a0710 con 0x7f544809f2e0 2019-12-05T06:35:41.962 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:41.954+0000 7f544cd52700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54480a0b00 con 0x7f544807e720 2019-12-05T06:35:41.962 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:41.954+0000 7f544cd52700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f544808e5b0 con 0x7f54480a7740 2019-12-05T06:35:41.962 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:41.954+0000 7f544759e700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f544809f2e0 0x7f544809f710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:58352/0 (socket says 172.21.15.42:58352) 2019-12-05T06:35:41.962 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:41.954+0000 7f544759e700 1 -- 172.21.15.42:0/4292060383 learned_addr learned my addr 172.21.15.42:0/4292060383 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:35:41.962 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:41.954+0000 7f544659c700 1 --2- 172.21.15.42:0/4292060383 >> v2:172.21.15.40:3300/0 conn(0x7f544807e720 0x7f544807eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:35:41.963 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:41.954+0000 7f5446d9d700 1 --2- 172.21.15.42:0/4292060383 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f54480a7740 0x7f544808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:35:41.963 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:41.954+0000 7f5445d9b700 1 -- 172.21.15.42:0/4292060383 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1547236273 0 0) 0x7f54480a0710 con 0x7f544809f2e0 2019-12-05T06:35:41.963 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:41.954+0000 7f5445d9b700 1 -- 172.21.15.42:0/4292060383 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54340030b0 con 0x7f544809f2e0 2019-12-05T06:35:41.963 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:41.954+0000 7f544659c700 1 -- 172.21.15.42:0/4292060383 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f54480a7740 msgr2=0x7f544808df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:35:41.963 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:41.954+0000 7f544659c700 1 --2- 172.21.15.42:0/4292060383 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f54480a7740 0x7f544808df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:35:41.963 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:41.954+0000 7f544659c700 1 -- 172.21.15.42:0/4292060383 >> v1:172.21.15.42:6789/0 conn(0x7f544809f2e0 legacy=0x7f544809f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:35:41.963 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:41.954+0000 7f544659c700 1 -- 172.21.15.42:0/4292060383 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f544807f0a0 con 0x7f544807e720 2019-12-05T06:35:41.963 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:41.954+0000 7f5446d9d700 1 --2- 172.21.15.42:0/4292060383 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f54480a7740 0x7f544808df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:35:41.964 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:41.954+0000 7f544659c700 1 --2- 172.21.15.42:0/4292060383 >> v2:172.21.15.40:3300/0 conn(0x7f544807e720 0x7f544807eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f5430008790 tx=0x7f543000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:41.964 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:41.954+0000 7f544659c700 1 --2- 172.21.15.42:0/4292060383 >> v2:172.21.15.40:3300/0 conn(0x7f544807e720 0x7f544807eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f5430008790 tx=0x7f543000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:41.964 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:41.954+0000 7f544659c700 1 --2- 172.21.15.42:0/4292060383 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f544807e720 0x7f544807eb50 secure :-1 s=READY pgs=2144 cs=0 l=1 rx=0x7f5430008790 tx=0x7f543000fc70).ready entity=mon.0 client_cookie=ef21067085db3571 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:35:41.964 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:41.954+0000 7f544659c700 1 --2- 172.21.15.42:0/4292060383 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f544807e720 0x7f544807eb50 secure :-1 s=THROTTLE_DONE pgs=2144 cs=0 l=1 rx=0x7f5430008790 tx=0x7f543000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:41.965 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:41.954+0000 7f5445d9b700 1 -- 172.21.15.42:0/4292060383 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f54300112f0 con 0x7f544807e720 2019-12-05T06:35:41.965 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:41.954+0000 7f544659c700 1 --2- 172.21.15.42:0/4292060383 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f544807e720 0x7f544807eb50 secure :-1 s=THROTTLE_DONE pgs=2144 cs=0 l=1 rx=0x7f5430008790 tx=0x7f543000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:41.966 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:41.954+0000 7f544659c700 1 --2- 172.21.15.42:0/4292060383 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f544807e720 0x7f544807eb50 secure :-1 s=THROTTLE_DONE pgs=2144 cs=0 l=1 rx=0x7f5430008790 tx=0x7f543000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:41.966 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:41.954+0000 7f5445d9b700 1 -- 172.21.15.42:0/4292060383 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5430008ad0 con 0x7f544807e720 2019-12-05T06:35:41.966 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:41.954+0000 7f5445d9b700 1 -- 172.21.15.42:0/4292060383 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f54300118d0 con 0x7f544807e720 2019-12-05T06:35:41.966 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:41.954+0000 7f544cd52700 1 -- 172.21.15.42:0/4292060383 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f544807e720 msgr2=0x7f544807eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:35:41.966 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:41.954+0000 7f544cd52700 1 --2- 172.21.15.42:0/4292060383 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f544807e720 0x7f544807eb50 secure :-1 s=READY pgs=2144 cs=0 l=1 rx=0x7f5430008790 tx=0x7f543000fc70).stop 2019-12-05T06:35:41.966 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:41.954+0000 7f544cd52700 1 -- 172.21.15.42:0/4292060383 shutdown_connections 2019-12-05T06:35:41.967 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:41.954+0000 7f544cd52700 1 --2- 172.21.15.42:0/4292060383 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f54480a7740 0x7f544808df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:35:41.967 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:41.954+0000 7f544cd52700 1 --2- 172.21.15.42:0/4292060383 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f544807e720 0x7f544807eb50 unknown :-1 s=CLOSED pgs=2144 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:35:41.967 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:41.954+0000 7f544cd52700 1 -- 172.21.15.42:0/4292060383 shutdown_connections 2019-12-05T06:35:41.968 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:41.954+0000 7f544cd52700 1 -- 172.21.15.42:0/4292060383 wait complete. 2019-12-05T06:35:41.968 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:41.954+0000 7f544cd52700 1 -- 172.21.15.42:0/4292060383 >> 172.21.15.42:0/4292060383 conn(0x7f54480aabe0 msgr2=0x7f54480a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:35:41.989 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:41.958+0000 7f544cd52700 1 Processor -- start 2019-12-05T06:35:41.989 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:41.958+0000 7f544cd52700 1 -- start start 2019-12-05T06:35:41.989 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:41.958+0000 7f544cd52700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5448119420 0x7f544811abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:35:41.989 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:41.958+0000 7f544cd52700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f544811b120 con 0x7f544807d920 2019-12-05T06:35:41.989 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:41.958+0000 7f544cd52700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5448118c90 con 0x7f5448117920 2019-12-05T06:35:41.989 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:41.958+0000 7f544cd52700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f544811b470 con 0x7f5448119420 2019-12-05T06:35:41.990 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:41.962+0000 7f544759e700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f5448117920 0x7f54481375c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:58356/0 (socket says 172.21.15.42:58356) 2019-12-05T06:35:41.990 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:41.962+0000 7f544759e700 1 -- 172.21.15.42:0/3194527021 learned_addr learned my addr 172.21.15.42:0/3194527021 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:35:41.990 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:41.962+0000 7f5446d9d700 1 --2- 172.21.15.42:0/3194527021 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5448119420 0x7f544811abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:35:41.990 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:41.962+0000 7f543f7fe700 1 -- 172.21.15.42:0/3194527021 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2895478958 0 0) 0x7f5448118c90 con 0x7f5448117920 2019-12-05T06:35:41.990 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:41.962+0000 7f543f7fe700 1 -- 172.21.15.42:0/3194527021 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f542c0030b0 con 0x7f5448117920 2019-12-05T06:35:41.990 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:41.962+0000 7f543f7fe700 1 -- 172.21.15.42:0/3194527021 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2151426384 0 0) 0x7f544811b120 con 0x7f544807d920 2019-12-05T06:35:41.996 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:41.962+0000 7f543f7fe700 1 -- 172.21.15.42:0/3194527021 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5448118c90 con 0x7f544807d920 2019-12-05T06:35:41.996 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:41.962+0000 7f5446d9d700 1 -- 172.21.15.42:0/3194527021 >> v1:172.21.15.42:6789/0 conn(0x7f5448117920 legacy=0x7f54481375c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:35:41.996 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:41.962+0000 7f5446d9d700 1 -- 172.21.15.42:0/3194527021 >> v1:172.21.15.40:6789/0 conn(0x7f544807d920 legacy=0x7f5448117120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:35:41.997 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:41.962+0000 7f5446d9d700 1 -- 172.21.15.42:0/3194527021 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f544807dd50 con 0x7f5448119420 2019-12-05T06:35:41.997 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:41.962+0000 7f5446d9d700 1 --2- 172.21.15.42:0/3194527021 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5448119420 0x7f544811abd0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f54380032b0 tx=0x7f5438010f50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:41.997 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:41.962+0000 7f5446d9d700 1 --2- 172.21.15.42:0/3194527021 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5448119420 0x7f544811abd0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f54380032b0 tx=0x7f5438010f50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:41.997 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:41.962+0000 7f5446d9d700 1 --2- 172.21.15.42:0/3194527021 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5448119420 0x7f544811abd0 secure :-1 s=READY pgs=629 cs=0 l=1 rx=0x7f54380032b0 tx=0x7f5438010f50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:35:41.997 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:41.962+0000 7f5446d9d700 1 --2- 172.21.15.42:0/3194527021 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5448119420 0x7f544811abd0 secure :-1 s=THROTTLE_DONE pgs=629 cs=0 l=1 rx=0x7f54380032b0 tx=0x7f5438010f50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:41.998 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:41.962+0000 7f543f7fe700 1 -- 172.21.15.42:0/3194527021 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5438012960 con 0x7f5448119420 2019-12-05T06:35:41.998 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:41.962+0000 7f5446d9d700 1 --2- 172.21.15.42:0/3194527021 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5448119420 0x7f544811abd0 secure :-1 s=THROTTLE_DONE pgs=629 cs=0 l=1 rx=0x7f54380032b0 tx=0x7f5438010f50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:41.998 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:41.962+0000 7f543f7fe700 1 -- 172.21.15.42:0/3194527021 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f543800e160 con 0x7f5448119420 2019-12-05T06:35:41.998 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:41.962+0000 7f5446d9d700 1 --2- 172.21.15.42:0/3194527021 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5448119420 0x7f544811abd0 secure :-1 s=THROTTLE_DONE pgs=629 cs=0 l=1 rx=0x7f54380032b0 tx=0x7f5438010f50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:41.998 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:41.962+0000 7f543f7fe700 1 -- 172.21.15.42:0/3194527021 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f54380058c0 con 0x7f5448119420 2019-12-05T06:35:41.998 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:41.962+0000 7f544cd52700 1 -- 172.21.15.42:0/3194527021 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f544807e020 con 0x7f5448119420 2019-12-05T06:35:41.999 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:41.962+0000 7f544cd52700 1 -- 172.21.15.42:0/3194527021 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f544808eb70 con 0x7f5448119420 2019-12-05T06:35:41.999 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:41.962+0000 7f5446d9d700 1 --2- 172.21.15.42:0/3194527021 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5448119420 0x7f544811abd0 secure :-1 s=THROTTLE_DONE pgs=629 cs=0 l=1 rx=0x7f54380032b0 tx=0x7f5438010f50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:41.999 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:41.962+0000 7f543f7fe700 1 -- 172.21.15.42:0/3194527021 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f5438005c20 con 0x7f5448119420 2019-12-05T06:35:41.999 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:41.966+0000 7f5446d9d700 1 --2- 172.21.15.42:0/3194527021 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5448119420 0x7f544811abd0 secure :-1 s=THROTTLE_DONE pgs=629 cs=0 l=1 rx=0x7f54380032b0 tx=0x7f5438010f50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:41.999 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:41.966+0000 7f543f7fe700 1 --2- 172.21.15.42:0/3194527021 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f542c03d610 0x7f542c03fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:35:41.999 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:41.966+0000 7f543f7fe700 1 -- 172.21.15.42:0/3194527021 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 11363+0+0 (secure 0 0 0) 0x7f543804af50 con 0x7f5448119420 2019-12-05T06:35:41.999 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:41.966+0000 7f544659c700 1 --2- 172.21.15.42:0/3194527021 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f542c03d610 0x7f542c03fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:35:42.000 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:41.966+0000 7f544659c700 1 --2- 172.21.15.42:0/3194527021 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f542c03d610 0x7f542c03fb20 crc :-1 s=READY pgs=1197 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:35:42.000 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:41.966+0000 7f544cd52700 1 -- 172.21.15.42:0/3194527021 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f544805d110 con 0x7f5448119420 2019-12-05T06:35:42.000 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:41.974+0000 7f5446d9d700 1 --2- 172.21.15.42:0/3194527021 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5448119420 0x7f544811abd0 secure :-1 s=THROTTLE_DONE pgs=629 cs=0 l=1 rx=0x7f54380032b0 tx=0x7f5438010f50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:42.000 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:41.974+0000 7f543f7fe700 1 -- 172.21.15.42:0/3194527021 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f5438031fd0 con 0x7f5448119420 2019-12-05T06:35:42.151 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:42.146+0000 7f544cd52700 1 -- 172.21.15.42:0/3194527021 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f544805b8c0 con 0x7f5448119420 2019-12-05T06:35:42.152 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:42.146+0000 7f5446d9d700 1 --2- 172.21.15.42:0/3194527021 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5448119420 0x7f544811abd0 secure :-1 s=THROTTLE_DONE pgs=629 cs=0 l=1 rx=0x7f54380032b0 tx=0x7f5438010f50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:42.152 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:42.146+0000 7f543f7fe700 1 -- 172.21.15.42:0/3194527021 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 342 v342) v1 ==== 95+0+1776 (secure 0 0 0) 0x7f5438032230 con 0x7f5448119420 2019-12-05T06:35:42.154 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:35:42.154 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":342,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":342,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:35:40.288210+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":111,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7910},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7910":{"gid":7910,"name":"b","rank":0,"incarnation":339,"state":"up:active","state_seq":33,"addr":"172.21.15.40:6833/4253974783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":4253974783},{"type":"v1","addr":"172.21.15.40:6833","nonce":4253974783}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:35:42.154 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 342 2019-12-05T06:35:42.156 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:42.150+0000 7f544cd52700 1 -- 172.21.15.42:0/3194527021 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f542c03d610 msgr2=0x7f542c03fb20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:35:42.156 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:42.150+0000 7f544cd52700 1 --2- 172.21.15.42:0/3194527021 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f542c03d610 0x7f542c03fb20 crc :-1 s=READY pgs=1197 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:35:42.157 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:42.150+0000 7f544cd52700 1 -- 172.21.15.42:0/3194527021 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5448119420 msgr2=0x7f544811abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:35:42.157 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:42.150+0000 7f544cd52700 1 --2- 172.21.15.42:0/3194527021 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5448119420 0x7f544811abd0 secure :-1 s=READY pgs=629 cs=0 l=1 rx=0x7f54380032b0 tx=0x7f5438010f50).stop 2019-12-05T06:35:42.157 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:42.150+0000 7f544cd52700 1 -- 172.21.15.42:0/3194527021 shutdown_connections 2019-12-05T06:35:42.157 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:42.150+0000 7f544cd52700 1 --2- 172.21.15.42:0/3194527021 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f542c03d610 0x7f542c03fb20 unknown :-1 s=CLOSED pgs=1197 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:35:42.158 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:42.150+0000 7f544cd52700 1 --2- 172.21.15.42:0/3194527021 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5448119420 0x7f544811abd0 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:35:42.158 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:42.154+0000 7f544cd52700 1 -- 172.21.15.42:0/3194527021 shutdown_connections 2019-12-05T06:35:42.158 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:42.154+0000 7f544cd52700 1 -- 172.21.15.42:0/3194527021 wait complete. 2019-12-05T06:35:42.158 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:42.154+0000 7f544cd52700 1 -- 172.21.15.42:0/3194527021 >> 172.21.15.42:0/3194527021 conn(0x7f544807e670 msgr2=0x7f544809f5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:35:42.167 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b has gained rank=0, replacing gid=11433 2019-12-05T06:35:42.167 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 1 secs before reviving mds.a 2019-12-05T06:35:43.169 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.a 2019-12-05T06:35:43.169 INFO:tasks.ceph.mds.a:Restarting daemon 2019-12-05T06:35:43.169 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2019-12-05T06:35:43.173 INFO:tasks.ceph.mds.a:Started 2019-12-05T06:35:43.173 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:35:43.236 INFO:tasks.ceph.mds.a.smithi042.stdout:starting mds.a at 2019-12-05T06:35:43.285 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:43.274+0000 7fe90e614700 1 Processor -- start 2019-12-05T06:35:43.285 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:43.274+0000 7fe90e614700 1 -- start start 2019-12-05T06:35:43.285 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:43.278+0000 7fe90e614700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe9080a7730 0x7fe90808df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:35:43.285 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:43.278+0000 7fe90e614700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe90809fad0 0x7fe90807eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:35:43.286 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:43.278+0000 7fe90e614700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9080a0700 con 0x7fe90807e710 2019-12-05T06:35:43.286 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:43.278+0000 7fe90e614700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9080a0af0 con 0x7fe9080a7730 2019-12-05T06:35:43.286 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:43.278+0000 7fe90e614700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe90808e5a0 con 0x7fe90809fad0 2019-12-05T06:35:43.286 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:43.278+0000 7fe90cbb1700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe9080a7730 0x7fe90808df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:35:43.286 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:43.278+0000 7fe90d3b2700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe90809fad0 0x7fe90807eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:35:43.286 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:43.278+0000 7fe90cbb1700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe9080a7730 0x7fe90808df70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.42:46158/0 (socket says 172.21.15.42:46158) 2019-12-05T06:35:43.287 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:43.278+0000 7fe907fff700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fe90807e710 0x7fe90809f2d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:39590/0 (socket says 172.21.15.42:39590) 2019-12-05T06:35:43.287 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:43.278+0000 7fe90cbb1700 1 -- 172.21.15.42:0/428511763 learned_addr learned my addr 172.21.15.42:0/428511763 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:35:43.287 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:43.278+0000 7fe90d3b2700 1 -- 172.21.15.42:0/428511763 >> v2:172.21.15.40:3300/0 conn(0x7fe9080a7730 msgr2=0x7fe90808df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:35:43.287 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:43.278+0000 7fe90d3b2700 1 --2- 172.21.15.42:0/428511763 >> v2:172.21.15.40:3300/0 conn(0x7fe9080a7730 0x7fe90808df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:35:43.287 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:43.278+0000 7fe90d3b2700 1 -- 172.21.15.42:0/428511763 >> v1:172.21.15.40:6789/0 conn(0x7fe90807e710 legacy=0x7fe90809f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:35:43.288 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:43.278+0000 7fe90d3b2700 1 -- 172.21.15.42:0/428511763 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe90807f090 con 0x7fe90809fad0 2019-12-05T06:35:43.288 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:43.278+0000 7fe90d3b2700 1 --2- 172.21.15.42:0/428511763 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe90809fad0 0x7fe90807eb40 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fe900008b10 tx=0x7fe90000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:43.288 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:43.278+0000 7fe90d3b2700 1 --2- 172.21.15.42:0/428511763 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe90809fad0 0x7fe90807eb40 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fe900008b10 tx=0x7fe90000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:43.288 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:43.278+0000 7fe90d3b2700 1 --2- 172.21.15.42:0/428511763 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe90809fad0 0x7fe90807eb40 secure :-1 s=READY pgs=630 cs=0 l=1 rx=0x7fe900008b10 tx=0x7fe90000fff0).ready entity=mon.2 client_cookie=e3ddcb92dd4c8bb7 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:35:43.288 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:43.278+0000 7fe90d3b2700 1 --2- 172.21.15.42:0/428511763 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe90809fad0 0x7fe90807eb40 secure :-1 s=THROTTLE_DONE pgs=630 cs=0 l=1 rx=0x7fe900008b10 tx=0x7fe90000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:43.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:43.278+0000 7fe9077fe700 1 -- 172.21.15.42:0/428511763 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe900010900 con 0x7fe90809fad0 2019-12-05T06:35:43.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:43.278+0000 7fe90d3b2700 1 --2- 172.21.15.42:0/428511763 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe90809fad0 0x7fe90807eb40 secure :-1 s=THROTTLE_DONE pgs=630 cs=0 l=1 rx=0x7fe900008b10 tx=0x7fe90000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:43.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:43.278+0000 7fe90d3b2700 1 --2- 172.21.15.42:0/428511763 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe90809fad0 0x7fe90807eb40 secure :-1 s=THROTTLE_DONE pgs=630 cs=0 l=1 rx=0x7fe900008b10 tx=0x7fe90000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:43.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:43.278+0000 7fe9077fe700 1 -- 172.21.15.42:0/428511763 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe900008b50 con 0x7fe90809fad0 2019-12-05T06:35:43.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:43.278+0000 7fe9077fe700 1 -- 172.21.15.42:0/428511763 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe9000110c0 con 0x7fe90809fad0 2019-12-05T06:35:43.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:43.278+0000 7fe90e614700 1 -- 172.21.15.42:0/428511763 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe90809fad0 msgr2=0x7fe90807eb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:35:43.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:43.278+0000 7fe90e614700 1 --2- 172.21.15.42:0/428511763 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe90809fad0 0x7fe90807eb40 secure :-1 s=READY pgs=630 cs=0 l=1 rx=0x7fe900008b10 tx=0x7fe90000fff0).stop 2019-12-05T06:35:43.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:43.278+0000 7fe90e614700 1 -- 172.21.15.42:0/428511763 shutdown_connections 2019-12-05T06:35:43.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:43.278+0000 7fe90e614700 1 --2- 172.21.15.42:0/428511763 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe90809fad0 0x7fe90807eb40 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:35:43.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:43.278+0000 7fe90e614700 1 --2- 172.21.15.42:0/428511763 >> v2:172.21.15.40:3300/0 conn(0x7fe9080a7730 0x7fe90808df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:35:43.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:43.278+0000 7fe90e614700 1 -- 172.21.15.42:0/428511763 shutdown_connections 2019-12-05T06:35:43.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:43.278+0000 7fe90e614700 1 -- 172.21.15.42:0/428511763 wait complete. 2019-12-05T06:35:43.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:43.278+0000 7fe90e614700 1 -- 172.21.15.42:0/428511763 >> 172.21.15.42:0/428511763 conn(0x7fe9080aabd0 msgr2=0x7fe9080a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:35:43.294 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:43.282+0000 7fe90e614700 1 Processor -- start 2019-12-05T06:35:43.295 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:43.282+0000 7fe90e614700 1 -- start start 2019-12-05T06:35:43.295 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:43.282+0000 7fe90e614700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe90811bbd0 0x7fe90812ce30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:35:43.295 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:43.282+0000 7fe90e614700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe90811c000 con 0x7fe90811ca80 2019-12-05T06:35:43.295 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:43.282+0000 7fe90e614700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe90811a5c0 con 0x7fe908123070 2019-12-05T06:35:43.295 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:43.282+0000 7fe90e614700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9081234a0 con 0x7fe90811bbd0 2019-12-05T06:35:43.295 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:43.282+0000 7fe907fff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fe908123070 0x7fe908135830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:58388/0 (socket says 172.21.15.42:58388) 2019-12-05T06:35:43.296 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:43.282+0000 7fe907fff700 1 -- 172.21.15.42:0/4177991330 learned_addr learned my addr 172.21.15.42:0/4177991330 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:35:43.296 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:43.282+0000 7fe90d3b2700 1 --2- 172.21.15.42:0/4177991330 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe90811bbd0 0x7fe90812ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:35:43.296 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:43.282+0000 7fe9057fa700 1 -- 172.21.15.42:0/4177991330 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1097382851 0 0) 0x7fe90811a5c0 con 0x7fe908123070 2019-12-05T06:35:43.296 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:43.282+0000 7fe9057fa700 1 -- 172.21.15.42:0/4177991330 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8e80030b0 con 0x7fe908123070 2019-12-05T06:35:43.296 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:43.282+0000 7fe9057fa700 1 -- 172.21.15.42:0/4177991330 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1931300027 0 0) 0x7fe90811c000 con 0x7fe90811ca80 2019-12-05T06:35:43.297 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:43.282+0000 7fe9057fa700 1 -- 172.21.15.42:0/4177991330 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe90811a5c0 con 0x7fe90811ca80 2019-12-05T06:35:43.297 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:43.282+0000 7fe90d3b2700 1 -- 172.21.15.42:0/4177991330 >> v1:172.21.15.42:6789/0 conn(0x7fe908123070 legacy=0x7fe908135830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:35:43.297 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:43.282+0000 7fe90d3b2700 1 -- 172.21.15.42:0/4177991330 >> v1:172.21.15.40:6789/0 conn(0x7fe90811ca80 legacy=0x7fe90807db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:35:43.297 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:43.282+0000 7fe90d3b2700 1 -- 172.21.15.42:0/4177991330 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe90813b070 con 0x7fe90811bbd0 2019-12-05T06:35:43.297 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:43.282+0000 7fe90d3b2700 1 --2- 172.21.15.42:0/4177991330 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe90811bbd0 0x7fe90812ce30 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fe900008b10 tx=0x7fe900008a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:43.297 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:43.286+0000 7fe90d3b2700 1 --2- 172.21.15.42:0/4177991330 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe90811bbd0 0x7fe90812ce30 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fe900008b10 tx=0x7fe900008a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:43.298 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:43.286+0000 7fe90d3b2700 1 --2- 172.21.15.42:0/4177991330 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe90811bbd0 0x7fe90812ce30 secure :-1 s=READY pgs=631 cs=0 l=1 rx=0x7fe900008b10 tx=0x7fe900008a20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:35:43.298 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:43.286+0000 7fe90d3b2700 1 --2- 172.21.15.42:0/4177991330 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe90811bbd0 0x7fe90812ce30 secure :-1 s=THROTTLE_DONE pgs=631 cs=0 l=1 rx=0x7fe900008b10 tx=0x7fe900008a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:43.298 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:43.286+0000 7fe9057fa700 1 -- 172.21.15.42:0/4177991330 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe900005ab0 con 0x7fe90811bbd0 2019-12-05T06:35:43.298 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:43.286+0000 7fe90d3b2700 1 --2- 172.21.15.42:0/4177991330 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe90811bbd0 0x7fe90812ce30 secure :-1 s=THROTTLE_DONE pgs=631 cs=0 l=1 rx=0x7fe900008b10 tx=0x7fe900008a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:43.298 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:43.286+0000 7fe9057fa700 1 -- 172.21.15.42:0/4177991330 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe900010b00 con 0x7fe90811bbd0 2019-12-05T06:35:43.298 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:43.286+0000 7fe90e614700 1 -- 172.21.15.42:0/4177991330 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe908139050 con 0x7fe90811bbd0 2019-12-05T06:35:43.299 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:43.286+0000 7fe90d3b2700 1 --2- 172.21.15.42:0/4177991330 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe90811bbd0 0x7fe90812ce30 secure :-1 s=THROTTLE_DONE pgs=631 cs=0 l=1 rx=0x7fe900008b10 tx=0x7fe900008a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:43.299 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:43.286+0000 7fe9057fa700 1 -- 172.21.15.42:0/4177991330 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe90000f680 con 0x7fe90811bbd0 2019-12-05T06:35:43.299 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:43.286+0000 7fe90e614700 1 -- 172.21.15.42:0/4177991330 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9081395d0 con 0x7fe90811bbd0 2019-12-05T06:35:43.299 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:43.286+0000 7fe90d3b2700 1 --2- 172.21.15.42:0/4177991330 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe90811bbd0 0x7fe90812ce30 secure :-1 s=THROTTLE_DONE pgs=631 cs=0 l=1 rx=0x7fe900008b10 tx=0x7fe900008a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:43.299 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:43.286+0000 7fe9057fa700 1 -- 172.21.15.42:0/4177991330 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fe9000113a0 con 0x7fe90811bbd0 2019-12-05T06:35:43.300 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:43.286+0000 7fe90d3b2700 1 --2- 172.21.15.42:0/4177991330 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe90811bbd0 0x7fe90812ce30 secure :-1 s=THROTTLE_DONE pgs=631 cs=0 l=1 rx=0x7fe900008b10 tx=0x7fe900008a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:43.300 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:43.286+0000 7fe9057fa700 1 --2- 172.21.15.42:0/4177991330 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe8e803b440 0x7fe8e803d950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:35:43.300 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:43.286+0000 7fe9057fa700 1 -- 172.21.15.42:0/4177991330 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 11363+0+0 (secure 0 0 0) 0x7fe90008f1a0 con 0x7fe90811bbd0 2019-12-05T06:35:43.300 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:43.286+0000 7fe90cbb1700 1 --2- 172.21.15.42:0/4177991330 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe8e803b440 0x7fe8e803d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:35:43.300 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:43.286+0000 7fe90cbb1700 1 --2- 172.21.15.42:0/4177991330 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe8e803b440 0x7fe8e803d950 crc :-1 s=READY pgs=1198 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:35:43.300 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:43.290+0000 7fe90e614700 1 -- 172.21.15.42:0/4177991330 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe90805d100 con 0x7fe90811bbd0 2019-12-05T06:35:43.302 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:43.298+0000 7fe90d3b2700 1 --2- 172.21.15.42:0/4177991330 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe90811bbd0 0x7fe90812ce30 secure :-1 s=THROTTLE_DONE pgs=631 cs=0 l=1 rx=0x7fe900008b10 tx=0x7fe900008a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:43.302 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:43.298+0000 7fe9057fa700 1 -- 172.21.15.42:0/4177991330 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fe900076460 con 0x7fe90811bbd0 2019-12-05T06:35:43.486 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:43.482+0000 7fe90e614700 1 -- 172.21.15.42:0/4177991330 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe90805b8b0 con 0x7fe90811bbd0 2019-12-05T06:35:43.487 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:43.482+0000 7fe90d3b2700 1 --2- 172.21.15.42:0/4177991330 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe90811bbd0 0x7fe90812ce30 secure :-1 s=THROTTLE_DONE pgs=631 cs=0 l=1 rx=0x7fe900008b10 tx=0x7fe900008a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:43.487 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:43.482+0000 7fe9057fa700 1 -- 172.21.15.42:0/4177991330 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 342 v342) v1 ==== 95+0+1776 (secure 0 0 0) 0x7fe900011ca0 con 0x7fe90811bbd0 2019-12-05T06:35:43.488 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 342 2019-12-05T06:35:43.488 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:35:43.488 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":342,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":342,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:35:40.288210+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":111,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7910},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7910":{"gid":7910,"name":"b","rank":0,"incarnation":339,"state":"up:active","state_seq":33,"addr":"172.21.15.40:6833/4253974783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":4253974783},{"type":"v1","addr":"172.21.15.40:6833","nonce":4253974783}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:35:43.490 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:43.486+0000 7fe90e614700 1 -- 172.21.15.42:0/4177991330 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe8e803b440 msgr2=0x7fe8e803d950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:35:43.490 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:43.486+0000 7fe90e614700 1 --2- 172.21.15.42:0/4177991330 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe8e803b440 0x7fe8e803d950 crc :-1 s=READY pgs=1198 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:35:43.490 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:43.486+0000 7fe90e614700 1 -- 172.21.15.42:0/4177991330 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe90811bbd0 msgr2=0x7fe90812ce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:35:43.490 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:43.486+0000 7fe90e614700 1 --2- 172.21.15.42:0/4177991330 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe90811bbd0 0x7fe90812ce30 secure :-1 s=READY pgs=631 cs=0 l=1 rx=0x7fe900008b10 tx=0x7fe900008a20).stop 2019-12-05T06:35:43.491 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:43.486+0000 7fe90e614700 1 -- 172.21.15.42:0/4177991330 shutdown_connections 2019-12-05T06:35:43.491 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:43.486+0000 7fe90e614700 1 --2- 172.21.15.42:0/4177991330 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe8e803b440 0x7fe8e803d950 unknown :-1 s=CLOSED pgs=1198 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:35:43.491 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:43.486+0000 7fe90e614700 1 --2- 172.21.15.42:0/4177991330 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe90811bbd0 0x7fe90812ce30 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:35:43.491 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:43.486+0000 7fe90e614700 1 -- 172.21.15.42:0/4177991330 shutdown_connections 2019-12-05T06:35:43.492 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:43.486+0000 7fe90e614700 1 -- 172.21.15.42:0/4177991330 wait complete. 2019-12-05T06:35:43.492 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:43.486+0000 7fe90e614700 1 -- 172.21.15.42:0/4177991330 >> 172.21.15.42:0/4177991330 conn(0x7fe9080a8070 msgr2=0x7fe9080a6ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:35:43.499 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2019-12-05T06:35:43.728 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:35:43.732 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:35:45.500 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:35:45.606 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:45.598+0000 7f241bd8c700 1 Processor -- start 2019-12-05T06:35:45.606 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:45.598+0000 7f241bd8c700 1 -- start start 2019-12-05T06:35:45.607 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:45.598+0000 7f241bd8c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f241408df80 0x7f241409f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:35:45.607 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:45.598+0000 7f241bd8c700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f241409f830 0x7f241414efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:35:45.607 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:45.598+0000 7f241bd8c700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f241409fc60 con 0x7f24140a7740 2019-12-05T06:35:45.607 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:45.598+0000 7f241bd8c700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24140a0b00 con 0x7f241409f830 2019-12-05T06:35:45.607 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:45.598+0000 7f241bd8c700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f241408e5b0 con 0x7f241408df80 2019-12-05T06:35:45.608 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:45.598+0000 7f2419b28700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f241408df80 0x7f241409f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:35:45.608 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:45.598+0000 7f2419b28700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f241408df80 0x7f241409f2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:58362/0 (socket says 172.21.15.42:58362) 2019-12-05T06:35:45.608 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:45.598+0000 7f241a329700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f24140a7740 0x7f241407e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:58398/0 (socket says 172.21.15.42:58398) 2019-12-05T06:35:45.608 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:45.598+0000 7f2419b28700 1 -- 172.21.15.42:0/3681512470 learned_addr learned my addr 172.21.15.42:0/3681512470 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:35:45.608 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:45.598+0000 7f241ab2a700 1 --2- 172.21.15.42:0/3681512470 >> v2:172.21.15.40:3300/0 conn(0x7f241409f830 0x7f241414efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:35:45.608 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:45.598+0000 7f2419b28700 1 -- 172.21.15.42:0/3681512470 >> v2:172.21.15.40:3300/0 conn(0x7f241409f830 msgr2=0x7f241414efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:35:45.608 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:45.598+0000 7f2419b28700 1 --2- 172.21.15.42:0/3681512470 >> v2:172.21.15.40:3300/0 conn(0x7f241409f830 0x7f241414efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:35:45.609 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:45.598+0000 7f2419b28700 1 -- 172.21.15.42:0/3681512470 >> v1:172.21.15.42:6789/0 conn(0x7f24140a7740 legacy=0x7f241407e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:35:45.609 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:45.598+0000 7f2419b28700 1 -- 172.21.15.42:0/3681512470 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24140a0710 con 0x7f241408df80 2019-12-05T06:35:45.609 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:45.598+0000 7f2419b28700 1 --2- 172.21.15.42:0/3681512470 >> v2:172.21.15.42:3300/0 conn(0x7f241408df80 0x7f241409f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f2404002f30 tx=0x7f240400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:45.609 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:45.598+0000 7f2419b28700 1 --2- 172.21.15.42:0/3681512470 >> v2:172.21.15.42:3300/0 conn(0x7f241408df80 0x7f241409f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f2404002f30 tx=0x7f240400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:45.609 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:45.598+0000 7f2419b28700 1 --2- 172.21.15.42:0/3681512470 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f241408df80 0x7f241409f2e0 secure :-1 s=READY pgs=2852 cs=0 l=1 rx=0x7f2404002f30 tx=0x7f240400fd30).ready entity=mon.1 client_cookie=6790a6dff5368f3a server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:35:45.609 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:45.598+0000 7f2419b28700 1 --2- 172.21.15.42:0/3681512470 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f241408df80 0x7f241409f2e0 secure :-1 s=THROTTLE_DONE pgs=2852 cs=0 l=1 rx=0x7f2404002f30 tx=0x7f240400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:45.610 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:45.602+0000 7f2419b28700 1 --2- 172.21.15.42:0/3681512470 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f241408df80 0x7f241409f2e0 secure :-1 s=THROTTLE_DONE pgs=2852 cs=0 l=1 rx=0x7f2404002f30 tx=0x7f240400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:45.610 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:45.602+0000 7f2419327700 1 -- 172.21.15.42:0/3681512470 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2404008c90 con 0x7f241408df80 2019-12-05T06:35:45.610 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:45.602+0000 7f2419b28700 1 --2- 172.21.15.42:0/3681512470 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f241408df80 0x7f241409f2e0 secure :-1 s=THROTTLE_DONE pgs=2852 cs=0 l=1 rx=0x7f2404002f30 tx=0x7f240400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:45.610 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:45.602+0000 7f2419327700 1 -- 172.21.15.42:0/3681512470 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24040088b0 con 0x7f241408df80 2019-12-05T06:35:45.610 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:45.602+0000 7f2419327700 1 -- 172.21.15.42:0/3681512470 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2404011ad0 con 0x7f241408df80 2019-12-05T06:35:45.610 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:45.602+0000 7f241bd8c700 1 -- 172.21.15.42:0/3681512470 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f241408df80 msgr2=0x7f241409f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:35:45.611 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:45.602+0000 7f241bd8c700 1 --2- 172.21.15.42:0/3681512470 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f241408df80 0x7f241409f2e0 secure :-1 s=READY pgs=2852 cs=0 l=1 rx=0x7f2404002f30 tx=0x7f240400fd30).stop 2019-12-05T06:35:45.611 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:45.602+0000 7f241bd8c700 1 -- 172.21.15.42:0/3681512470 shutdown_connections 2019-12-05T06:35:45.611 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:45.602+0000 7f241bd8c700 1 --2- 172.21.15.42:0/3681512470 >> v2:172.21.15.40:3300/0 conn(0x7f241409f830 0x7f241414efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:35:45.611 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:45.602+0000 7f241bd8c700 1 --2- 172.21.15.42:0/3681512470 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f241408df80 0x7f241409f2e0 unknown :-1 s=CLOSED pgs=2852 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:35:45.611 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:45.602+0000 7f241bd8c700 1 -- 172.21.15.42:0/3681512470 shutdown_connections 2019-12-05T06:35:45.611 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:45.602+0000 7f241bd8c700 1 -- 172.21.15.42:0/3681512470 wait complete. 2019-12-05T06:35:45.612 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:45.602+0000 7f241bd8c700 1 -- 172.21.15.42:0/3681512470 >> 172.21.15.42:0/3681512470 conn(0x7f24140aabe0 msgr2=0x7f24140a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:35:45.615 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:45.602+0000 7f241bd8c700 1 Processor -- start 2019-12-05T06:35:45.615 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:45.602+0000 7f241bd8c700 1 -- start start 2019-12-05T06:35:45.616 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:45.606+0000 7f241bd8c700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2414116500 0x7f241411ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:35:45.616 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:45.606+0000 7f241bd8c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f241411b930 0x7f241408f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:35:45.616 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:45.606+0000 7f241bd8c700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2414118d90 con 0x7f241411b500 2019-12-05T06:35:45.616 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:45.606+0000 7f241bd8c700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f241407c700 con 0x7f241411b930 2019-12-05T06:35:45.616 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:45.606+0000 7f241bd8c700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f241407c9a0 con 0x7f2414116500 2019-12-05T06:35:45.616 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:45.606+0000 7f241ab2a700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f241411b930 0x7f241408f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:35:45.616 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:45.606+0000 7f241ab2a700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f241411b930 0x7f241408f370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:58368/0 (socket says 172.21.15.42:58368) 2019-12-05T06:35:45.617 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:45.606+0000 7f241ab2a700 1 -- 172.21.15.42:0/3311526742 learned_addr learned my addr 172.21.15.42:0/3311526742 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:35:45.617 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:45.606+0000 7f241a329700 1 --2- 172.21.15.42:0/3311526742 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2414116500 0x7f241411ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:35:45.617 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:45.606+0000 7f240affd700 1 -- 172.21.15.42:0/3311526742 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2674960206 0 0) 0x7f2414118d90 con 0x7f241411b500 2019-12-05T06:35:45.617 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:45.606+0000 7f240affd700 1 -- 172.21.15.42:0/3311526742 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23fc0030b0 con 0x7f241411b500 2019-12-05T06:35:45.617 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:45.606+0000 7f241ab2a700 1 -- 172.21.15.42:0/3311526742 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2414116500 msgr2=0x7f241411ab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:35:45.618 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:45.606+0000 7f241ab2a700 1 --2- 172.21.15.42:0/3311526742 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2414116500 0x7f241411ab60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:35:45.618 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:45.606+0000 7f241ab2a700 1 -- 172.21.15.42:0/3311526742 >> v1:172.21.15.40:6789/0 conn(0x7f241411b500 legacy=0x7f241408eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:35:45.618 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:45.606+0000 7f241ab2a700 1 -- 172.21.15.42:0/3311526742 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f241408f8c0 con 0x7f241411b930 2019-12-05T06:35:45.618 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:45.606+0000 7f241ab2a700 1 --2- 172.21.15.42:0/3311526742 >> v2:172.21.15.42:3300/0 conn(0x7f241411b930 0x7f241408f370 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f2414118fc0 tx=0x7f240c010d10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:45.618 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:45.606+0000 7f241ab2a700 1 --2- 172.21.15.42:0/3311526742 >> v2:172.21.15.42:3300/0 conn(0x7f241411b930 0x7f241408f370 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f2414118fc0 tx=0x7f240c010d10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:45.618 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:45.606+0000 7f241ab2a700 1 --2- 172.21.15.42:0/3311526742 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f241411b930 0x7f241408f370 secure :-1 s=READY pgs=2853 cs=0 l=1 rx=0x7f2414118fc0 tx=0x7f240c010d10).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:35:45.618 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:45.606+0000 7f241ab2a700 1 --2- 172.21.15.42:0/3311526742 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f241411b930 0x7f241408f370 secure :-1 s=THROTTLE_DONE pgs=2853 cs=0 l=1 rx=0x7f2414118fc0 tx=0x7f240c010d10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:45.619 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:45.606+0000 7f240affd700 1 -- 172.21.15.42:0/3311526742 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f240c007350 con 0x7f241411b930 2019-12-05T06:35:45.619 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:45.606+0000 7f241ab2a700 1 --2- 172.21.15.42:0/3311526742 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f241411b930 0x7f241408f370 secure :-1 s=THROTTLE_DONE pgs=2853 cs=0 l=1 rx=0x7f2414118fc0 tx=0x7f240c010d10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:45.619 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:45.606+0000 7f241ab2a700 1 --2- 172.21.15.42:0/3311526742 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f241411b930 0x7f241408f370 secure :-1 s=THROTTLE_DONE pgs=2853 cs=0 l=1 rx=0x7f2414118fc0 tx=0x7f240c010d10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:45.619 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:45.606+0000 7f240affd700 1 -- 172.21.15.42:0/3311526742 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f240c005cb0 con 0x7f241411b930 2019-12-05T06:35:45.619 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:45.606+0000 7f241bd8c700 1 -- 172.21.15.42:0/3311526742 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f241407cfd0 con 0x7f241411b930 2019-12-05T06:35:45.619 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:45.606+0000 7f240affd700 1 -- 172.21.15.42:0/3311526742 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f240c012390 con 0x7f241411b930 2019-12-05T06:35:45.619 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:45.606+0000 7f241bd8c700 1 -- 172.21.15.42:0/3311526742 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f241411bd60 con 0x7f241411b930 2019-12-05T06:35:45.620 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:45.610+0000 7f241ab2a700 1 --2- 172.21.15.42:0/3311526742 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f241411b930 0x7f241408f370 secure :-1 s=THROTTLE_DONE pgs=2853 cs=0 l=1 rx=0x7f2414118fc0 tx=0x7f240c010d10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:45.620 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:45.610+0000 7f240affd700 1 -- 172.21.15.42:0/3311526742 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f240c0126a0 con 0x7f241411b930 2019-12-05T06:35:45.620 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:45.610+0000 7f241ab2a700 1 --2- 172.21.15.42:0/3311526742 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f241411b930 0x7f241408f370 secure :-1 s=THROTTLE_DONE pgs=2853 cs=0 l=1 rx=0x7f2414118fc0 tx=0x7f240c010d10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:45.620 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:45.610+0000 7f240affd700 1 --2- 172.21.15.42:0/3311526742 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f23fc03b280 0x7f23fc03d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:35:45.620 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:45.610+0000 7f240affd700 1 -- 172.21.15.42:0/3311526742 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 11363+0+0 (secure 0 0 0) 0x7f240c04b780 con 0x7f241411b930 2019-12-05T06:35:45.620 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:45.610+0000 7f241a329700 1 --2- 172.21.15.42:0/3311526742 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f23fc03b280 0x7f23fc03d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:35:45.621 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:45.610+0000 7f241a329700 1 -- 172.21.15.42:0/3311526742 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f23fc03b280 msgr2=0x7f23fc03d790 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2019-12-05T06:35:45.621 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:45.610+0000 7f241a329700 1 -- 172.21.15.42:0/3311526742 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f23fc03b280 msgr2=0x7f23fc03d790 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2019-12-05T06:35:45.621 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:45.610+0000 7f241a329700 1 --2- 172.21.15.42:0/3311526742 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f23fc03b280 0x7f23fc03d790 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-12-05T06:35:45.621 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:45.610+0000 7f241a329700 1 --2- 172.21.15.42:0/3311526742 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f23fc03b280 0x7f23fc03d790 crc :-1 s=START_CONNECT pgs=0 cs=0 l=1 rx=0 tx=0)._fault waiting 0.200000 2019-12-05T06:35:45.621 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:45.614+0000 7f241bd8c700 1 -- 172.21.15.42:0/3311526742 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f241405d110 con 0x7f241411b930 2019-12-05T06:35:45.626 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:45.622+0000 7f241ab2a700 1 --2- 172.21.15.42:0/3311526742 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f241411b930 0x7f241408f370 secure :-1 s=THROTTLE_DONE pgs=2853 cs=0 l=1 rx=0x7f2414118fc0 tx=0x7f240c010d10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:45.626 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:45.622+0000 7f240affd700 1 -- 172.21.15.42:0/3311526742 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f240c032800 con 0x7f241411b930 2019-12-05T06:35:45.795 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:45.790+0000 7f241bd8c700 1 -- 172.21.15.42:0/3311526742 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f241405b8c0 con 0x7f241411b930 2019-12-05T06:35:45.796 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:45.790+0000 7f241ab2a700 1 --2- 172.21.15.42:0/3311526742 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f241411b930 0x7f241408f370 secure :-1 s=THROTTLE_DONE pgs=2853 cs=0 l=1 rx=0x7f2414118fc0 tx=0x7f240c010d10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:45.797 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:45.790+0000 7f240affd700 1 -- 172.21.15.42:0/3311526742 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 342 v342) v1 ==== 95+0+1776 (secure 0 0 0) 0x7f240c032a60 con 0x7f241411b930 2019-12-05T06:35:45.797 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 342 2019-12-05T06:35:45.797 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:35:45.797 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":342,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":342,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:35:40.288210+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":111,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7910},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7910":{"gid":7910,"name":"b","rank":0,"incarnation":339,"state":"up:active","state_seq":33,"addr":"172.21.15.40:6833/4253974783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":4253974783},{"type":"v1","addr":"172.21.15.40:6833","nonce":4253974783}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:35:45.799 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:45.794+0000 7f241bd8c700 1 -- 172.21.15.42:0/3311526742 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f23fc03b280 msgr2=0x7f23fc03d790 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2019-12-05T06:35:45.799 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:45.794+0000 7f241bd8c700 1 --2- 172.21.15.42:0/3311526742 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f23fc03b280 0x7f23fc03d790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:35:45.799 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:45.794+0000 7f241bd8c700 1 -- 172.21.15.42:0/3311526742 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f241411b930 msgr2=0x7f241408f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:35:45.799 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:45.794+0000 7f241bd8c700 1 --2- 172.21.15.42:0/3311526742 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f241411b930 0x7f241408f370 secure :-1 s=READY pgs=2853 cs=0 l=1 rx=0x7f2414118fc0 tx=0x7f240c010d10).stop 2019-12-05T06:35:45.799 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:45.794+0000 7f241bd8c700 1 -- 172.21.15.42:0/3311526742 shutdown_connections 2019-12-05T06:35:45.800 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:45.794+0000 7f241bd8c700 1 --2- 172.21.15.42:0/3311526742 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f23fc03b280 0x7f23fc03d790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:35:45.800 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:45.794+0000 7f241bd8c700 1 --2- 172.21.15.42:0/3311526742 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f241411b930 0x7f241408f370 unknown :-1 s=CLOSED pgs=2853 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:35:45.800 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:45.794+0000 7f241bd8c700 1 --2- 172.21.15.42:0/3311526742 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2414116500 0x7f241411ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:35:45.800 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:45.794+0000 7f241bd8c700 1 -- 172.21.15.42:0/3311526742 shutdown_connections 2019-12-05T06:35:45.800 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:45.794+0000 7f241bd8c700 1 -- 172.21.15.42:0/3311526742 wait complete. 2019-12-05T06:35:45.800 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:45.794+0000 7f241bd8c700 1 -- 172.21.15.42:0/3311526742 >> 172.21.15.42:0/3311526742 conn(0x7f24140a97e0 msgr2=0x7f2414059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:35:45.809 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2019-12-05T06:35:47.809 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:35:47.919 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:47.910+0000 7f4a87690700 1 Processor -- start 2019-12-05T06:35:47.919 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:47.910+0000 7f4a87690700 1 -- start start 2019-12-05T06:35:47.919 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:47.910+0000 7f4a87690700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f4a8009fae0 0x7f4a8014fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:35:47.919 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:47.910+0000 7f4a87690700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a800a0710 con 0x7f4a800a7740 2019-12-05T06:35:47.920 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:47.910+0000 7f4a87690700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a8007ef20 con 0x7f4a8008df80 2019-12-05T06:35:47.920 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:47.910+0000 7f4a87690700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a800a0b00 con 0x7f4a8009fae0 2019-12-05T06:35:47.920 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:47.910+0000 7f4a8542c700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f4a8008df80 0x7f4a8009f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:58416/0 (socket says 172.21.15.42:58416) 2019-12-05T06:35:47.920 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:47.910+0000 7f4a8542c700 1 -- 172.21.15.42:0/1309763817 learned_addr learned my addr 172.21.15.42:0/1309763817 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:35:47.920 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:47.910+0000 7f4a8642e700 1 --2- 172.21.15.42:0/1309763817 >> v2:172.21.15.42:3300/0 conn(0x7f4a8009fae0 0x7f4a8014fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:35:47.921 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:47.910+0000 7f4a84c2b700 1 -- 172.21.15.42:0/1309763817 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3336430781 0 0) 0x7f4a8007ef20 con 0x7f4a8008df80 2019-12-05T06:35:47.921 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:47.910+0000 7f4a84c2b700 1 -- 172.21.15.42:0/1309763817 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a6c0030b0 con 0x7f4a8008df80 2019-12-05T06:35:47.921 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:47.910+0000 7f4a84c2b700 1 -- 172.21.15.42:0/1309763817 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2837622319 0 0) 0x7f4a800a0710 con 0x7f4a800a7740 2019-12-05T06:35:47.921 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:47.910+0000 7f4a8642e700 1 -- 172.21.15.42:0/1309763817 >> v1:172.21.15.42:6789/0 conn(0x7f4a8008df80 legacy=0x7f4a8009f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:35:47.921 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:47.910+0000 7f4a8642e700 1 -- 172.21.15.42:0/1309763817 >> v1:172.21.15.40:6789/0 conn(0x7f4a800a7740 legacy=0x7f4a8007e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:35:47.921 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:47.910+0000 7f4a8642e700 1 -- 172.21.15.42:0/1309763817 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a800a9ba0 con 0x7f4a8009fae0 2019-12-05T06:35:47.921 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:47.910+0000 7f4a8642e700 1 --2- 172.21.15.42:0/1309763817 >> v2:172.21.15.42:3300/0 conn(0x7f4a8009fae0 0x7f4a8014fff0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f4a78008b10 tx=0x7f4a7800ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:47.922 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:47.914+0000 7f4a8642e700 1 --2- 172.21.15.42:0/1309763817 >> v2:172.21.15.42:3300/0 conn(0x7f4a8009fae0 0x7f4a8014fff0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f4a78008b10 tx=0x7f4a7800ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:47.922 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:47.914+0000 7f4a8642e700 1 --2- 172.21.15.42:0/1309763817 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4a8009fae0 0x7f4a8014fff0 secure :-1 s=READY pgs=2855 cs=0 l=1 rx=0x7f4a78008b10 tx=0x7f4a7800ff70).ready entity=mon.1 client_cookie=5112f766a41650e3 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:35:47.922 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:47.914+0000 7f4a8642e700 1 --2- 172.21.15.42:0/1309763817 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4a8009fae0 0x7f4a8014fff0 secure :-1 s=THROTTLE_DONE pgs=2855 cs=0 l=1 rx=0x7f4a78008b10 tx=0x7f4a7800ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:47.922 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:47.914+0000 7f4a84c2b700 1 -- 172.21.15.42:0/1309763817 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4a78010970 con 0x7f4a8009fae0 2019-12-05T06:35:47.922 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:47.914+0000 7f4a8642e700 1 --2- 172.21.15.42:0/1309763817 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4a8009fae0 0x7f4a8014fff0 secure :-1 s=THROTTLE_DONE pgs=2855 cs=0 l=1 rx=0x7f4a78008b10 tx=0x7f4a7800ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:47.922 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:47.914+0000 7f4a84c2b700 1 -- 172.21.15.42:0/1309763817 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a78008b50 con 0x7f4a8009fae0 2019-12-05T06:35:47.923 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:47.914+0000 7f4a8642e700 1 --2- 172.21.15.42:0/1309763817 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4a8009fae0 0x7f4a8014fff0 secure :-1 s=THROTTLE_DONE pgs=2855 cs=0 l=1 rx=0x7f4a78008b10 tx=0x7f4a7800ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:47.923 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:47.914+0000 7f4a87690700 1 -- 172.21.15.42:0/1309763817 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4a8009fae0 msgr2=0x7f4a8014fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:35:47.923 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:47.914+0000 7f4a84c2b700 1 -- 172.21.15.42:0/1309763817 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4a780110c0 con 0x7f4a8009fae0 2019-12-05T06:35:47.923 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:47.914+0000 7f4a87690700 1 --2- 172.21.15.42:0/1309763817 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4a8009fae0 0x7f4a8014fff0 secure :-1 s=READY pgs=2855 cs=0 l=1 rx=0x7f4a78008b10 tx=0x7f4a7800ff70).stop 2019-12-05T06:35:47.923 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:47.914+0000 7f4a87690700 1 -- 172.21.15.42:0/1309763817 shutdown_connections 2019-12-05T06:35:47.923 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:47.914+0000 7f4a87690700 1 --2- 172.21.15.42:0/1309763817 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4a8009fae0 0x7f4a8014fff0 unknown :-1 s=CLOSED pgs=2855 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:35:47.923 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:47.914+0000 7f4a87690700 1 -- 172.21.15.42:0/1309763817 shutdown_connections 2019-12-05T06:35:47.924 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:47.914+0000 7f4a87690700 1 -- 172.21.15.42:0/1309763817 wait complete. 2019-12-05T06:35:47.924 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:47.914+0000 7f4a87690700 1 -- 172.21.15.42:0/1309763817 >> 172.21.15.42:0/1309763817 conn(0x7f4a800aabe0 msgr2=0x7f4a800a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:35:47.931 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:47.918+0000 7f4a87690700 1 Processor -- start 2019-12-05T06:35:47.931 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:47.918+0000 7f4a87690700 1 -- start start 2019-12-05T06:35:47.931 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:47.918+0000 7f4a87690700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f4a80115170 0x7f4a800a6c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:35:47.931 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:47.918+0000 7f4a87690700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f4a8007b440 0x7f4a8007a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:35:47.931 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:47.918+0000 7f4a87690700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4a8007b870 0x7f4a8007adb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:35:47.932 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:47.918+0000 7f4a87690700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a80115960 con 0x7f4a80115170 2019-12-05T06:35:47.932 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:47.918+0000 7f4a87690700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a80115c00 con 0x7f4a8007b440 2019-12-05T06:35:47.932 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:47.918+0000 7f4a87690700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a80119220 con 0x7f4a8007b870 2019-12-05T06:35:47.932 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:47.918+0000 7f4a8542c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f4a8007b440 0x7f4a8007a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:35:47.932 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:47.918+0000 7f4a8542c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f4a8007b440 0x7f4a8007a860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:58384/0 (socket says 172.21.15.42:58384) 2019-12-05T06:35:47.932 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:47.918+0000 7f4a8542c700 1 -- 172.21.15.42:0/832425480 learned_addr learned my addr 172.21.15.42:0/832425480 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:35:47.932 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:47.918+0000 7f4a85c2d700 1 --2- 172.21.15.42:0/832425480 >> v2:172.21.15.40:3300/0 conn(0x7f4a80115170 0x7f4a800a6c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:35:47.933 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:47.918+0000 7f4a8642e700 1 --2- 172.21.15.42:0/832425480 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4a8007b870 0x7f4a8007adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:35:47.933 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:47.918+0000 7f4a8542c700 1 -- 172.21.15.42:0/832425480 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4a8007b870 msgr2=0x7f4a8007adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:35:47.933 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:47.918+0000 7f4a8542c700 1 --2- 172.21.15.42:0/832425480 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4a8007b870 0x7f4a8007adb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:35:47.933 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:47.918+0000 7f4a8542c700 1 -- 172.21.15.42:0/832425480 >> v2:172.21.15.40:3300/0 conn(0x7f4a80115170 msgr2=0x7f4a800a6c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:35:47.933 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:47.918+0000 7f4a8542c700 1 --2- 172.21.15.42:0/832425480 >> v2:172.21.15.40:3300/0 conn(0x7f4a80115170 0x7f4a800a6c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:35:47.933 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:47.918+0000 7f4a8542c700 1 -- 172.21.15.42:0/832425480 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a8014fd20 con 0x7f4a8007b440 2019-12-05T06:35:47.933 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:47.918+0000 7f4a85c2d700 1 --2- 172.21.15.42:0/832425480 >> v2:172.21.15.40:3300/0 conn(0x7f4a80115170 0x7f4a800a6c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:35:47.934 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:47.918+0000 7f4a8542c700 1 --2- 172.21.15.42:0/832425480 >> v2:172.21.15.42:3300/0 conn(0x7f4a8007b440 0x7f4a8007a860 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f4a70000b80 tx=0x7f4a70010970).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:47.934 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:47.918+0000 7f4a8542c700 1 --2- 172.21.15.42:0/832425480 >> v2:172.21.15.42:3300/0 conn(0x7f4a8007b440 0x7f4a8007a860 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f4a70000b80 tx=0x7f4a70010970).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:47.934 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:47.918+0000 7f4a8542c700 1 --2- 172.21.15.42:0/832425480 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4a8007b440 0x7f4a8007a860 secure :-1 s=READY pgs=2856 cs=0 l=1 rx=0x7f4a70000b80 tx=0x7f4a70010970).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:35:47.934 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:47.918+0000 7f4a8542c700 1 --2- 172.21.15.42:0/832425480 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4a8007b440 0x7f4a8007a860 secure :-1 s=THROTTLE_DONE pgs=2856 cs=0 l=1 rx=0x7f4a70000b80 tx=0x7f4a70010970).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:47.934 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:47.918+0000 7f4a7e7fc700 1 -- 172.21.15.42:0/832425480 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4a70005b40 con 0x7f4a8007b440 2019-12-05T06:35:47.934 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:47.918+0000 7f4a8542c700 1 --2- 172.21.15.42:0/832425480 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4a8007b440 0x7f4a8007a860 secure :-1 s=THROTTLE_DONE pgs=2856 cs=0 l=1 rx=0x7f4a70000b80 tx=0x7f4a70010970).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:47.934 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:47.918+0000 7f4a8542c700 1 --2- 172.21.15.42:0/832425480 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4a8007b440 0x7f4a8007a860 secure :-1 s=THROTTLE_DONE pgs=2856 cs=0 l=1 rx=0x7f4a70000b80 tx=0x7f4a70010970).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:47.935 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:47.918+0000 7f4a7e7fc700 1 -- 172.21.15.42:0/832425480 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a70005760 con 0x7f4a8007b440 2019-12-05T06:35:47.935 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:47.918+0000 7f4a7e7fc700 1 -- 172.21.15.42:0/832425480 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4a700062b0 con 0x7f4a8007b440 2019-12-05T06:35:47.935 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:47.918+0000 7f4a87690700 1 -- 172.21.15.42:0/832425480 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a80126940 con 0x7f4a8007b440 2019-12-05T06:35:47.935 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:47.922+0000 7f4a87690700 1 -- 172.21.15.42:0/832425480 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a8008ed10 con 0x7f4a8007b440 2019-12-05T06:35:47.935 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:47.922+0000 7f4a8542c700 1 --2- 172.21.15.42:0/832425480 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4a8007b440 0x7f4a8007a860 secure :-1 s=THROTTLE_DONE pgs=2856 cs=0 l=1 rx=0x7f4a70000b80 tx=0x7f4a70010970).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:47.935 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:47.922+0000 7f4a7e7fc700 1 -- 172.21.15.42:0/832425480 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f4a700065c0 con 0x7f4a8007b440 2019-12-05T06:35:47.935 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:47.922+0000 7f4a8542c700 1 --2- 172.21.15.42:0/832425480 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4a8007b440 0x7f4a8007a860 secure :-1 s=THROTTLE_DONE pgs=2856 cs=0 l=1 rx=0x7f4a70000b80 tx=0x7f4a70010970).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:47.936 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:47.922+0000 7f4a7e7fc700 1 --2- 172.21.15.42:0/832425480 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4a6403ae60 0x7f4a6403d370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:35:47.936 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:47.922+0000 7f4a7e7fc700 1 -- 172.21.15.42:0/832425480 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 11363+0+0 (secure 0 0 0) 0x7f4a7004af90 con 0x7f4a8007b440 2019-12-05T06:35:47.936 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:47.922+0000 7f4a85c2d700 1 --2- 172.21.15.42:0/832425480 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4a6403ae60 0x7f4a6403d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:35:47.936 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:47.922+0000 7f4a85c2d700 1 --2- 172.21.15.42:0/832425480 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4a6403ae60 0x7f4a6403d370 crc :-1 s=READY pgs=1200 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:35:47.937 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:47.926+0000 7f4a87690700 1 -- 172.21.15.42:0/832425480 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a8005d110 con 0x7f4a8007b440 2019-12-05T06:35:47.939 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:47.934+0000 7f4a8542c700 1 --2- 172.21.15.42:0/832425480 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4a8007b440 0x7f4a8007a860 secure :-1 s=THROTTLE_DONE pgs=2856 cs=0 l=1 rx=0x7f4a70000b80 tx=0x7f4a70010970).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:47.940 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:47.934+0000 7f4a7e7fc700 1 -- 172.21.15.42:0/832425480 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f4a70032010 con 0x7f4a8007b440 2019-12-05T06:35:48.106 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:48.102+0000 7f4a87690700 1 -- 172.21.15.42:0/832425480 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4a8005b8c0 con 0x7f4a8007b440 2019-12-05T06:35:48.107 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:48.102+0000 7f4a8542c700 1 --2- 172.21.15.42:0/832425480 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4a8007b440 0x7f4a8007a860 secure :-1 s=THROTTLE_DONE pgs=2856 cs=0 l=1 rx=0x7f4a70000b80 tx=0x7f4a70010970).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:48.107 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:48.102+0000 7f4a7e7fc700 1 -- 172.21.15.42:0/832425480 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 343 v343) v1 ==== 95+0+2111 (secure 0 0 0) 0x7f4a70032270 con 0x7f4a8007b440 2019-12-05T06:35:48.109 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 343 2019-12-05T06:35:48.109 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:35:48.109 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":343,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":11887,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/958796360","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":958796360},{"type":"v1","addr":"172.21.15.42:6835","nonce":958796360}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":343}],"filesystems":[{"mdsmap":{"epoch":342,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:35:40.288210+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":111,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7910},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7910":{"gid":7910,"name":"b","rank":0,"incarnation":339,"state":"up:active","state_seq":33,"addr":"172.21.15.40:6833/4253974783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":4253974783},{"type":"v1","addr":"172.21.15.40:6833","nonce":4253974783}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:35:48.113 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:48.110+0000 7f4a87690700 1 -- 172.21.15.42:0/832425480 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4a6403ae60 msgr2=0x7f4a6403d370 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:35:48.114 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:48.110+0000 7f4a87690700 1 --2- 172.21.15.42:0/832425480 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4a6403ae60 0x7f4a6403d370 crc :-1 s=READY pgs=1200 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:35:48.114 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:48.110+0000 7f4a87690700 1 -- 172.21.15.42:0/832425480 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4a8007b440 msgr2=0x7f4a8007a860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:35:48.114 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:48.110+0000 7f4a87690700 1 --2- 172.21.15.42:0/832425480 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4a8007b440 0x7f4a8007a860 secure :-1 s=READY pgs=2856 cs=0 l=1 rx=0x7f4a70000b80 tx=0x7f4a70010970).stop 2019-12-05T06:35:48.114 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:48.110+0000 7f4a87690700 1 -- 172.21.15.42:0/832425480 shutdown_connections 2019-12-05T06:35:48.114 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:48.110+0000 7f4a87690700 1 --2- 172.21.15.42:0/832425480 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4a6403ae60 0x7f4a6403d370 unknown :-1 s=CLOSED pgs=1200 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:35:48.114 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:48.110+0000 7f4a87690700 1 --2- 172.21.15.42:0/832425480 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4a8007b870 0x7f4a8007adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:35:48.125 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:48.110+0000 7f4a87690700 1 --2- 172.21.15.42:0/832425480 >> v2:172.21.15.40:3300/0 conn(0x7f4a80115170 0x7f4a800a6c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:35:48.125 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:48.110+0000 7f4a87690700 1 --2- 172.21.15.42:0/832425480 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4a8007b440 0x7f4a8007a860 unknown :-1 s=CLOSED pgs=2856 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:35:48.125 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:48.110+0000 7f4a87690700 1 -- 172.21.15.42:0/832425480 shutdown_connections 2019-12-05T06:35:48.126 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:48.110+0000 7f4a87690700 1 -- 172.21.15.42:0/832425480 wait complete. 2019-12-05T06:35:48.126 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:35:48.110+0000 7f4a87690700 1 -- 172.21.15.42:0/832425480 >> 172.21.15.42:0/832425480 conn(0x7f4a8008fef0 msgr2=0x7f4a80119490 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:35:48.129 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a reported in up:standby state 2019-12-05T06:35:48.129 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 114 secs before thrashing 2019-12-05T06:36:13.802 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:36:13.815 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:36:43.884 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:36:43.889 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:37:13.950 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:37:13.955 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:37:42.130 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:37:42.241 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.229+0000 7fd3c7263700 1 Processor -- start 2019-12-05T06:37:42.241 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.229+0000 7fd3c7263700 1 -- start start 2019-12-05T06:37:42.241 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.233+0000 7fd3c7263700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd3c008df80 0x7fd3c009f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:37:42.241 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.233+0000 7fd3c7263700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd3c009f830 0x7fd3c014efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:37:42.241 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.233+0000 7fd3c7263700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3c009fc60 con 0x7fd3c00a7740 2019-12-05T06:37:42.241 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.233+0000 7fd3c7263700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3c00a0b00 con 0x7fd3c009f830 2019-12-05T06:37:42.242 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.233+0000 7fd3c7263700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3c008e5b0 con 0x7fd3c008df80 2019-12-05T06:37:42.242 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.233+0000 7fd3c5800700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fd3c00a7740 0x7fd3c007e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:40146/0 (socket says 172.21.15.42:40146) 2019-12-05T06:37:42.242 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.233+0000 7fd3c4fff700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd3c008df80 0x7fd3c009f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:37:42.242 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.233+0000 7fd3c5800700 1 -- 172.21.15.42:0/4015941223 learned_addr learned my addr 172.21.15.42:0/4015941223 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:37:42.242 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.233+0000 7fd3c6001700 1 --2- 172.21.15.42:0/4015941223 >> v2:172.21.15.40:3300/0 conn(0x7fd3c009f830 0x7fd3c014efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:37:42.242 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.233+0000 7fd3b7fff700 1 -- 172.21.15.42:0/4015941223 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1190868282 0 0) 0x7fd3c009fc60 con 0x7fd3c00a7740 2019-12-05T06:37:42.242 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.233+0000 7fd3b7fff700 1 -- 172.21.15.42:0/4015941223 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3ac0030b0 con 0x7fd3c00a7740 2019-12-05T06:37:42.243 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.233+0000 7fd3c4fff700 1 -- 172.21.15.42:0/4015941223 >> v2:172.21.15.40:3300/0 conn(0x7fd3c009f830 msgr2=0x7fd3c014efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:37:42.243 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.233+0000 7fd3c4fff700 1 --2- 172.21.15.42:0/4015941223 >> v2:172.21.15.40:3300/0 conn(0x7fd3c009f830 0x7fd3c014efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:37:42.243 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.233+0000 7fd3c4fff700 1 -- 172.21.15.42:0/4015941223 >> v1:172.21.15.40:6789/0 conn(0x7fd3c00a7740 legacy=0x7fd3c007e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:37:42.243 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.233+0000 7fd3c4fff700 1 -- 172.21.15.42:0/4015941223 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3c00a0710 con 0x7fd3c008df80 2019-12-05T06:37:42.243 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.233+0000 7fd3c6001700 1 --2- 172.21.15.42:0/4015941223 >> v2:172.21.15.40:3300/0 conn(0x7fd3c009f830 0x7fd3c014efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:37:42.243 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.233+0000 7fd3c4fff700 1 --2- 172.21.15.42:0/4015941223 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd3c008df80 0x7fd3c009f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fd3b0008790 tx=0x7fd3b000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:42.243 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.233+0000 7fd3c4fff700 1 --2- 172.21.15.42:0/4015941223 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd3c008df80 0x7fd3c009f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fd3b0008790 tx=0x7fd3b000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:42.244 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.233+0000 7fd3c4fff700 1 --2- 172.21.15.42:0/4015941223 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd3c008df80 0x7fd3c009f2e0 secure :-1 s=READY pgs=642 cs=0 l=1 rx=0x7fd3b0008790 tx=0x7fd3b000fc70).ready entity=mon.2 client_cookie=7e6e1325df9fc4ea server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:37:42.244 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.233+0000 7fd3c4fff700 1 --2- 172.21.15.42:0/4015941223 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd3c008df80 0x7fd3c009f2e0 secure :-1 s=THROTTLE_DONE pgs=642 cs=0 l=1 rx=0x7fd3b0008790 tx=0x7fd3b000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:42.244 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.233+0000 7fd3b7fff700 1 -- 172.21.15.42:0/4015941223 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd3b0008bb0 con 0x7fd3c008df80 2019-12-05T06:37:42.244 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.233+0000 7fd3c4fff700 1 --2- 172.21.15.42:0/4015941223 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd3c008df80 0x7fd3c009f2e0 secure :-1 s=THROTTLE_DONE pgs=642 cs=0 l=1 rx=0x7fd3b0008790 tx=0x7fd3b000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:42.244 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.233+0000 7fd3c4fff700 1 --2- 172.21.15.42:0/4015941223 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd3c008df80 0x7fd3c009f2e0 secure :-1 s=THROTTLE_DONE pgs=642 cs=0 l=1 rx=0x7fd3b0008790 tx=0x7fd3b000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:42.244 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.233+0000 7fd3b7fff700 1 -- 172.21.15.42:0/4015941223 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3b00087d0 con 0x7fd3c008df80 2019-12-05T06:37:42.245 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.233+0000 7fd3b7fff700 1 -- 172.21.15.42:0/4015941223 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd3b0011860 con 0x7fd3c008df80 2019-12-05T06:37:42.245 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.233+0000 7fd3c7263700 1 -- 172.21.15.42:0/4015941223 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd3c008df80 msgr2=0x7fd3c009f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:37:42.245 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.233+0000 7fd3c7263700 1 --2- 172.21.15.42:0/4015941223 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd3c008df80 0x7fd3c009f2e0 secure :-1 s=READY pgs=642 cs=0 l=1 rx=0x7fd3b0008790 tx=0x7fd3b000fc70).stop 2019-12-05T06:37:42.245 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.233+0000 7fd3c7263700 1 -- 172.21.15.42:0/4015941223 shutdown_connections 2019-12-05T06:37:42.245 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.233+0000 7fd3c7263700 1 --2- 172.21.15.42:0/4015941223 >> v2:172.21.15.40:3300/0 conn(0x7fd3c009f830 0x7fd3c014efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:37:42.245 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.233+0000 7fd3c7263700 1 --2- 172.21.15.42:0/4015941223 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd3c008df80 0x7fd3c009f2e0 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:37:42.245 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.233+0000 7fd3c7263700 1 -- 172.21.15.42:0/4015941223 shutdown_connections 2019-12-05T06:37:42.246 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.233+0000 7fd3c7263700 1 -- 172.21.15.42:0/4015941223 wait complete. 2019-12-05T06:37:42.246 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.233+0000 7fd3c7263700 1 -- 172.21.15.42:0/4015941223 >> 172.21.15.42:0/4015941223 conn(0x7fd3c00aabe0 msgr2=0x7fd3c00a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:37:42.251 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.237+0000 7fd3c7263700 1 Processor -- start 2019-12-05T06:37:42.251 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.237+0000 7fd3c7263700 1 -- start start 2019-12-05T06:37:42.252 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.237+0000 7fd3c7263700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd3c0063190 0x7fd3c0065870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:37:42.252 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.237+0000 7fd3c7263700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd3c0142530 0x7fd3c013ab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:37:42.252 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.237+0000 7fd3c7263700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3c0064270 con 0x7fd3c00a9980 2019-12-05T06:37:42.252 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.237+0000 7fd3c7263700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3c0064560 con 0x7fd3c0142530 2019-12-05T06:37:42.252 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.237+0000 7fd3c7263700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3c007cdf0 con 0x7fd3c0063190 2019-12-05T06:37:42.252 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.237+0000 7fd3c4fff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fd3c00a9980 0x7fd3c0079e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:58946/0 (socket says 172.21.15.42:58946) 2019-12-05T06:37:42.253 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.237+0000 7fd3c4fff700 1 -- 172.21.15.42:0/3075182687 learned_addr learned my addr 172.21.15.42:0/3075182687 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:37:42.253 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.237+0000 7fd3c6001700 1 --2- 172.21.15.42:0/3075182687 >> v2:172.21.15.40:3300/0 conn(0x7fd3c0142530 0x7fd3c013ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:37:42.253 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.237+0000 7fd3c5800700 1 --2- 172.21.15.42:0/3075182687 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd3c0063190 0x7fd3c0065870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:37:42.253 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.241+0000 7fd3b5ffb700 1 -- 172.21.15.42:0/3075182687 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3580332475 0 0) 0x7fd3c0064270 con 0x7fd3c00a9980 2019-12-05T06:37:42.253 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.241+0000 7fd3b5ffb700 1 -- 172.21.15.42:0/3075182687 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3a00030b0 con 0x7fd3c00a9980 2019-12-05T06:37:42.253 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.241+0000 7fd3c6001700 1 -- 172.21.15.42:0/3075182687 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd3c0063190 msgr2=0x7fd3c0065870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:37:42.253 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.241+0000 7fd3c6001700 1 --2- 172.21.15.42:0/3075182687 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd3c0063190 0x7fd3c0065870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:37:42.254 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.241+0000 7fd3c6001700 1 -- 172.21.15.42:0/3075182687 >> v1:172.21.15.42:6789/0 conn(0x7fd3c00a9980 legacy=0x7fd3c0079e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:37:42.254 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.241+0000 7fd3c6001700 1 -- 172.21.15.42:0/3075182687 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3c007c6c0 con 0x7fd3c0142530 2019-12-05T06:37:42.254 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.241+0000 7fd3c6001700 1 --2- 172.21.15.42:0/3075182687 >> v2:172.21.15.40:3300/0 conn(0x7fd3c0142530 0x7fd3c013ab00 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fd3c0065190 tx=0x7fd3b8010f00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:42.254 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.241+0000 7fd3c6001700 1 --2- 172.21.15.42:0/3075182687 >> v2:172.21.15.40:3300/0 conn(0x7fd3c0142530 0x7fd3c013ab00 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fd3c0065190 tx=0x7fd3b8010f00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:42.254 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.241+0000 7fd3c6001700 1 --2- 172.21.15.42:0/3075182687 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd3c0142530 0x7fd3c013ab00 secure :-1 s=READY pgs=2169 cs=0 l=1 rx=0x7fd3c0065190 tx=0x7fd3b8010f00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:37:42.255 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.241+0000 7fd3c6001700 1 --2- 172.21.15.42:0/3075182687 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd3c0142530 0x7fd3c013ab00 secure :-1 s=THROTTLE_DONE pgs=2169 cs=0 l=1 rx=0x7fd3c0065190 tx=0x7fd3b8010f00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:42.255 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.241+0000 7fd3c6001700 1 --2- 172.21.15.42:0/3075182687 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd3c0142530 0x7fd3c013ab00 secure :-1 s=THROTTLE_DONE pgs=2169 cs=0 l=1 rx=0x7fd3c0065190 tx=0x7fd3b8010f00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:42.255 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.241+0000 7fd3b5ffb700 1 -- 172.21.15.42:0/3075182687 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd3b80058c0 con 0x7fd3c0142530 2019-12-05T06:37:42.255 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.241+0000 7fd3c6001700 1 --2- 172.21.15.42:0/3075182687 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd3c0142530 0x7fd3c013ab00 secure :-1 s=THROTTLE_DONE pgs=2169 cs=0 l=1 rx=0x7fd3c0065190 tx=0x7fd3b8010f00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:42.255 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.241+0000 7fd3b5ffb700 1 -- 172.21.15.42:0/3075182687 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3b800e160 con 0x7fd3c0142530 2019-12-05T06:37:42.255 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.241+0000 7fd3c7263700 1 -- 172.21.15.42:0/3075182687 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3c007a7b0 con 0x7fd3c0142530 2019-12-05T06:37:42.255 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.241+0000 7fd3b5ffb700 1 -- 172.21.15.42:0/3075182687 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd3b800c850 con 0x7fd3c0142530 2019-12-05T06:37:42.255 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.241+0000 7fd3c7263700 1 -- 172.21.15.42:0/3075182687 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3c0142b00 con 0x7fd3c0142530 2019-12-05T06:37:42.256 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.241+0000 7fd3c6001700 1 --2- 172.21.15.42:0/3075182687 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd3c0142530 0x7fd3c013ab00 secure :-1 s=THROTTLE_DONE pgs=2169 cs=0 l=1 rx=0x7fd3c0065190 tx=0x7fd3b8010f00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:42.256 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.241+0000 7fd3b5ffb700 1 -- 172.21.15.42:0/3075182687 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fd3b800cb60 con 0x7fd3c0142530 2019-12-05T06:37:42.256 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.241+0000 7fd3c6001700 1 --2- 172.21.15.42:0/3075182687 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd3c0142530 0x7fd3c013ab00 secure :-1 s=THROTTLE_DONE pgs=2169 cs=0 l=1 rx=0x7fd3c0065190 tx=0x7fd3b8010f00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:42.256 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.245+0000 7fd3b5ffb700 1 --2- 172.21.15.42:0/3075182687 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd3a003b280 0x7fd3a003d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:37:42.256 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.245+0000 7fd3b5ffb700 1 -- 172.21.15.42:0/3075182687 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 11363+0+0 (secure 0 0 0) 0x7fd3b804af20 con 0x7fd3c0142530 2019-12-05T06:37:42.256 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.245+0000 7fd3c5800700 1 --2- 172.21.15.42:0/3075182687 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd3a003b280 0x7fd3a003d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:37:42.257 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.245+0000 7fd3c5800700 1 --2- 172.21.15.42:0/3075182687 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd3a003b280 0x7fd3a003d790 crc :-1 s=READY pgs=1202 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:37:42.257 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.245+0000 7fd3c7263700 1 -- 172.21.15.42:0/3075182687 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3c005d110 con 0x7fd3c0142530 2019-12-05T06:37:42.269 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.253+0000 7fd3c6001700 1 --2- 172.21.15.42:0/3075182687 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd3c0142530 0x7fd3c013ab00 secure :-1 s=THROTTLE_DONE pgs=2169 cs=0 l=1 rx=0x7fd3c0065190 tx=0x7fd3b8010f00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:42.270 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.253+0000 7fd3b5ffb700 1 -- 172.21.15.42:0/3075182687 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fd3b8031fa0 con 0x7fd3c0142530 2019-12-05T06:37:42.426 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.421+0000 7fd3c7263700 1 -- 172.21.15.42:0/3075182687 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd3c005b8c0 con 0x7fd3c0142530 2019-12-05T06:37:42.427 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.421+0000 7fd3c6001700 1 --2- 172.21.15.42:0/3075182687 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd3c0142530 0x7fd3c013ab00 secure :-1 s=THROTTLE_DONE pgs=2169 cs=0 l=1 rx=0x7fd3c0065190 tx=0x7fd3b8010f00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:42.427 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.421+0000 7fd3b5ffb700 1 -- 172.21.15.42:0/3075182687 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 343 v343) v1 ==== 95+0+2111 (secure 0 0 0) 0x7fd3b8032200 con 0x7fd3c0142530 2019-12-05T06:37:42.429 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 343 2019-12-05T06:37:42.429 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:37:42.430 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":343,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":11887,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/958796360","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":958796360},{"type":"v1","addr":"172.21.15.42:6835","nonce":958796360}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":343}],"filesystems":[{"mdsmap":{"epoch":342,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:35:40.288210+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":111,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7910},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7910":{"gid":7910,"name":"b","rank":0,"incarnation":339,"state":"up:active","state_seq":33,"addr":"172.21.15.40:6833/4253974783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":4253974783},{"type":"v1","addr":"172.21.15.40:6833","nonce":4253974783}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:37:42.434 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.429+0000 7fd3c7263700 1 -- 172.21.15.42:0/3075182687 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd3a003b280 msgr2=0x7fd3a003d790 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:37:42.434 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.429+0000 7fd3c7263700 1 --2- 172.21.15.42:0/3075182687 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd3a003b280 0x7fd3a003d790 crc :-1 s=READY pgs=1202 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:37:42.434 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.429+0000 7fd3c7263700 1 -- 172.21.15.42:0/3075182687 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd3c0142530 msgr2=0x7fd3c013ab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:37:42.434 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.429+0000 7fd3c7263700 1 --2- 172.21.15.42:0/3075182687 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd3c0142530 0x7fd3c013ab00 secure :-1 s=READY pgs=2169 cs=0 l=1 rx=0x7fd3c0065190 tx=0x7fd3b8010f00).stop 2019-12-05T06:37:42.435 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.429+0000 7fd3c7263700 1 -- 172.21.15.42:0/3075182687 shutdown_connections 2019-12-05T06:37:42.435 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.429+0000 7fd3c7263700 1 --2- 172.21.15.42:0/3075182687 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd3a003b280 0x7fd3a003d790 unknown :-1 s=CLOSED pgs=1202 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:37:42.435 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.429+0000 7fd3c7263700 1 --2- 172.21.15.42:0/3075182687 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd3c0142530 0x7fd3c013ab00 unknown :-1 s=CLOSED pgs=2169 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:37:42.435 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.429+0000 7fd3c7263700 1 --2- 172.21.15.42:0/3075182687 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd3c0063190 0x7fd3c0065870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:37:42.435 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.429+0000 7fd3c7263700 1 -- 172.21.15.42:0/3075182687 shutdown_connections 2019-12-05T06:37:42.435 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.429+0000 7fd3c7263700 1 -- 172.21.15.42:0/3075182687 wait complete. 2019-12-05T06:37:42.436 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.429+0000 7fd3c7263700 1 -- 172.21.15.42:0/3075182687 >> 172.21.15.42:0/3075182687 conn(0x7fd3c00a82a0 msgr2=0x7fd3c007e5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:37:42.447 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.b (rank=0) 2019-12-05T06:37:42.447 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2019-12-05T06:37:42.448 INFO:teuthology.orchestra.run:waiting for 300 2019-12-05T06:37:42.465 INFO:tasks.ceph.mds.b:Stopped 2019-12-05T06:37:42.466 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:37:42.581 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.569+0000 7f0982b8e700 1 Processor -- start 2019-12-05T06:37:42.582 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.569+0000 7f0982b8e700 1 -- start start 2019-12-05T06:37:42.582 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.573+0000 7f0982b8e700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f097c0a7740 0x7f097c08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:37:42.582 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.573+0000 7f0982b8e700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f097c09fae0 0x7f097c07eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:37:42.582 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.573+0000 7f0982b8e700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f097c0a0710 con 0x7f097c07e720 2019-12-05T06:37:42.582 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.573+0000 7f0982b8e700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f097c0a0b00 con 0x7f097c09fae0 2019-12-05T06:37:42.583 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.573+0000 7f0982b8e700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f097c08e5b0 con 0x7f097c0a7740 2019-12-05T06:37:42.583 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.573+0000 7f098112b700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f097c0a7740 0x7f097c08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:37:42.583 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.573+0000 7f098192c700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f097c09fae0 0x7f097c07eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:37:42.583 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.573+0000 7f098092a700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f097c07e720 0x7f097c09f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:40162/0 (socket says 172.21.15.42:40162) 2019-12-05T06:37:42.583 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.573+0000 7f098092a700 1 -- 172.21.15.42:0/3768019649 learned_addr learned my addr 172.21.15.42:0/3768019649 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:37:42.583 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.573+0000 7f098112b700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f097c0a7740 0x7f097c08df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:59384/0 (socket says 172.21.15.42:59384) 2019-12-05T06:37:42.583 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.573+0000 7f098192c700 1 -- 172.21.15.42:0/3768019649 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f097c0a7740 msgr2=0x7f097c08df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:37:42.584 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.573+0000 7f098192c700 1 --2- 172.21.15.42:0/3768019649 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f097c0a7740 0x7f097c08df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:37:42.584 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.573+0000 7f098192c700 1 -- 172.21.15.42:0/3768019649 >> v1:172.21.15.40:6789/0 conn(0x7f097c07e720 legacy=0x7f097c09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:37:42.584 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.573+0000 7f098192c700 1 -- 172.21.15.42:0/3768019649 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f097c07f0a0 con 0x7f097c09fae0 2019-12-05T06:37:42.584 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.573+0000 7f0973fff700 1 -- 172.21.15.42:0/3768019649 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3564943935 0 0) 0x7f097c0a0710 con 0x7f097c07e720 2019-12-05T06:37:42.584 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.573+0000 7f098112b700 1 --2- 172.21.15.42:0/3768019649 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f097c0a7740 0x7f097c08df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:37:42.584 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.573+0000 7f098192c700 1 --2- 172.21.15.42:0/3768019649 >> v2:172.21.15.40:3300/0 conn(0x7f097c09fae0 0x7f097c07eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f0974008b10 tx=0x7f097400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:42.585 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.573+0000 7f098192c700 1 --2- 172.21.15.42:0/3768019649 >> v2:172.21.15.40:3300/0 conn(0x7f097c09fae0 0x7f097c07eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f0974008b10 tx=0x7f097400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:42.585 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.573+0000 7f098192c700 1 --2- 172.21.15.42:0/3768019649 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f097c09fae0 0x7f097c07eb50 secure :-1 s=READY pgs=2171 cs=0 l=1 rx=0x7f0974008b10 tx=0x7f097400fff0).ready entity=mon.0 client_cookie=9127ccd55354d6a7 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:37:42.585 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.573+0000 7f098192c700 1 --2- 172.21.15.42:0/3768019649 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f097c09fae0 0x7f097c07eb50 secure :-1 s=THROTTLE_DONE pgs=2171 cs=0 l=1 rx=0x7f0974008b10 tx=0x7f097400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:42.585 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.573+0000 7f0973fff700 1 -- 172.21.15.42:0/3768019649 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0974010900 con 0x7f097c09fae0 2019-12-05T06:37:42.585 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.573+0000 7f098192c700 1 --2- 172.21.15.42:0/3768019649 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f097c09fae0 0x7f097c07eb50 secure :-1 s=THROTTLE_DONE pgs=2171 cs=0 l=1 rx=0x7f0974008b10 tx=0x7f097400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:42.586 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.573+0000 7f0973fff700 1 -- 172.21.15.42:0/3768019649 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0974008b50 con 0x7f097c09fae0 2019-12-05T06:37:42.586 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.573+0000 7f098192c700 1 --2- 172.21.15.42:0/3768019649 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f097c09fae0 0x7f097c07eb50 secure :-1 s=THROTTLE_DONE pgs=2171 cs=0 l=1 rx=0x7f0974008b10 tx=0x7f097400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:42.586 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.573+0000 7f0973fff700 1 -- 172.21.15.42:0/3768019649 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f09740110c0 con 0x7f097c09fae0 2019-12-05T06:37:42.586 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.573+0000 7f0982b8e700 1 -- 172.21.15.42:0/3768019649 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f097c09fae0 msgr2=0x7f097c07eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:37:42.586 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.573+0000 7f0982b8e700 1 --2- 172.21.15.42:0/3768019649 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f097c09fae0 0x7f097c07eb50 secure :-1 s=READY pgs=2171 cs=0 l=1 rx=0x7f0974008b10 tx=0x7f097400fff0).stop 2019-12-05T06:37:42.587 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.573+0000 7f0982b8e700 1 -- 172.21.15.42:0/3768019649 shutdown_connections 2019-12-05T06:37:42.587 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.573+0000 7f0982b8e700 1 --2- 172.21.15.42:0/3768019649 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f097c09fae0 0x7f097c07eb50 unknown :-1 s=CLOSED pgs=2171 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:37:42.587 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.573+0000 7f0982b8e700 1 --2- 172.21.15.42:0/3768019649 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f097c0a7740 0x7f097c08df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:37:42.587 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.573+0000 7f0982b8e700 1 -- 172.21.15.42:0/3768019649 shutdown_connections 2019-12-05T06:37:42.587 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.573+0000 7f0982b8e700 1 -- 172.21.15.42:0/3768019649 wait complete. 2019-12-05T06:37:42.587 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.573+0000 7f0982b8e700 1 -- 172.21.15.42:0/3768019649 >> 172.21.15.42:0/3768019649 conn(0x7f097c0aabe0 msgr2=0x7f097c0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:37:42.592 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.577+0000 7f0982b8e700 1 Processor -- start 2019-12-05T06:37:42.593 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.577+0000 7f0982b8e700 1 -- start start 2019-12-05T06:37:42.593 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.577+0000 7f0982b8e700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f097c11b050 0x7f097c08f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:37:42.593 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.577+0000 7f0982b8e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f097c144a80 0x7f097c144eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:37:42.593 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.577+0000 7f0982b8e700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f097c145400 con 0x7f097c116500 2019-12-05T06:37:42.594 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.577+0000 7f0982b8e700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f097c11b680 con 0x7f097c144a80 2019-12-05T06:37:42.594 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.577+0000 7f0982b8e700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f097c11b920 con 0x7f097c11b050 2019-12-05T06:37:42.594 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.577+0000 7f098192c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f097c144a80 0x7f097c144eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:37:42.594 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.577+0000 7f098092a700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f097c11b050 0x7f097c08f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:37:42.595 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.577+0000 7f098192c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f097c144a80 0x7f097c144eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:58926/0 (socket says 172.21.15.42:58926) 2019-12-05T06:37:42.595 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.577+0000 7f098192c700 1 -- 172.21.15.42:0/3930272723 learned_addr learned my addr 172.21.15.42:0/3930272723 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:37:42.595 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.581+0000 7f098092a700 1 -- 172.21.15.42:0/3930272723 >> v2:172.21.15.42:3300/0 conn(0x7f097c144a80 msgr2=0x7f097c144eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:37:42.595 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.581+0000 7f098092a700 1 --2- 172.21.15.42:0/3930272723 >> v2:172.21.15.42:3300/0 conn(0x7f097c144a80 0x7f097c144eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:37:42.595 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.581+0000 7f098092a700 1 -- 172.21.15.42:0/3930272723 >> v1:172.21.15.42:6789/0 conn(0x7f097c116500 legacy=0x7f097c142c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:37:42.595 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.581+0000 7f098092a700 1 -- 172.21.15.42:0/3930272723 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f097c143480 con 0x7f097c11b050 2019-12-05T06:37:42.596 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.581+0000 7f098192c700 1 --2- 172.21.15.42:0/3930272723 >> v2:172.21.15.42:3300/0 conn(0x7f097c144a80 0x7f097c144eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:37:42.596 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.581+0000 7f098092a700 1 --2- 172.21.15.42:0/3930272723 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f097c11b050 0x7f097c08f5a0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f096c000b80 tx=0x7f096c010880).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:42.596 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.581+0000 7f098092a700 1 --2- 172.21.15.42:0/3930272723 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f097c11b050 0x7f097c08f5a0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f096c000b80 tx=0x7f096c010880).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:42.596 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.581+0000 7f098092a700 1 --2- 172.21.15.42:0/3930272723 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f097c11b050 0x7f097c08f5a0 secure :-1 s=READY pgs=643 cs=0 l=1 rx=0x7f096c000b80 tx=0x7f096c010880).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:37:42.597 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.581+0000 7f098092a700 1 --2- 172.21.15.42:0/3930272723 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f097c11b050 0x7f097c08f5a0 secure :-1 s=THROTTLE_DONE pgs=643 cs=0 l=1 rx=0x7f096c000b80 tx=0x7f096c010880).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:42.597 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.581+0000 7f098092a700 1 --2- 172.21.15.42:0/3930272723 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f097c11b050 0x7f097c08f5a0 secure :-1 s=THROTTLE_DONE pgs=643 cs=0 l=1 rx=0x7f096c000b80 tx=0x7f096c010880).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:42.597 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.581+0000 7f098092a700 1 --2- 172.21.15.42:0/3930272723 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f097c11b050 0x7f097c08f5a0 secure :-1 s=THROTTLE_DONE pgs=643 cs=0 l=1 rx=0x7f096c000b80 tx=0x7f096c010880).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:42.597 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.581+0000 7f0971ffb700 1 -- 172.21.15.42:0/3930272723 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f096c0115b0 con 0x7f097c11b050 2019-12-05T06:37:42.597 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.581+0000 7f0971ffb700 1 -- 172.21.15.42:0/3930272723 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f096c0111d0 con 0x7f097c11b050 2019-12-05T06:37:42.597 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.581+0000 7f0982b8e700 1 -- 172.21.15.42:0/3930272723 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f097c130380 con 0x7f097c11b050 2019-12-05T06:37:42.598 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.581+0000 7f0971ffb700 1 -- 172.21.15.42:0/3930272723 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f096c011b90 con 0x7f097c11b050 2019-12-05T06:37:42.598 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.581+0000 7f0982b8e700 1 -- 172.21.15.42:0/3930272723 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f097c1379f0 con 0x7f097c11b050 2019-12-05T06:37:42.598 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.581+0000 7f098092a700 1 --2- 172.21.15.42:0/3930272723 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f097c11b050 0x7f097c08f5a0 secure :-1 s=THROTTLE_DONE pgs=643 cs=0 l=1 rx=0x7f096c000b80 tx=0x7f096c010880).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:42.598 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.581+0000 7f0971ffb700 1 -- 172.21.15.42:0/3930272723 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f096c00b330 con 0x7f097c11b050 2019-12-05T06:37:42.599 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.581+0000 7f098092a700 1 --2- 172.21.15.42:0/3930272723 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f097c11b050 0x7f097c08f5a0 secure :-1 s=THROTTLE_DONE pgs=643 cs=0 l=1 rx=0x7f096c000b80 tx=0x7f096c010880).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:42.599 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.581+0000 7f0971ffb700 1 --2- 172.21.15.42:0/3930272723 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f095c03ae30 0x7f095c03d340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:37:42.599 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.585+0000 7f0971ffb700 1 -- 172.21.15.42:0/3930272723 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 11363+0+0 (secure 0 0 0) 0x7f096c04b1e0 con 0x7f097c11b050 2019-12-05T06:37:42.600 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.585+0000 7f098112b700 1 --2- 172.21.15.42:0/3930272723 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f095c03ae30 0x7f095c03d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:37:42.600 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.585+0000 7f098112b700 1 --2- 172.21.15.42:0/3930272723 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f095c03ae30 0x7f095c03d340 crc :-1 s=READY pgs=1203 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:37:42.600 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.585+0000 7f0982b8e700 1 -- 172.21.15.42:0/3930272723 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f097c11b480 con 0x7f097c11b050 2019-12-05T06:37:42.601 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.597+0000 7f098092a700 1 --2- 172.21.15.42:0/3930272723 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f097c11b050 0x7f097c08f5a0 secure :-1 s=THROTTLE_DONE pgs=643 cs=0 l=1 rx=0x7f096c000b80 tx=0x7f096c010880).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:42.602 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.597+0000 7f0971ffb700 1 -- 172.21.15.42:0/3930272723 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f096c032260 con 0x7f097c11b050 2019-12-05T06:37:42.769 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.765+0000 7f0982b8e700 1 -- 172.21.15.42:0/3930272723 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f097c05d110 con 0x7f097c11b050 2019-12-05T06:37:42.771 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.765+0000 7f098092a700 1 --2- 172.21.15.42:0/3930272723 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f097c11b050 0x7f097c08f5a0 secure :-1 s=THROTTLE_DONE pgs=643 cs=0 l=1 rx=0x7f096c000b80 tx=0x7f096c010880).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:42.771 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.765+0000 7f0971ffb700 1 -- 172.21.15.42:0/3930272723 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 343 v343) v1 ==== 95+0+2111 (secure 0 0 0) 0x7f096c0324c0 con 0x7f097c11b050 2019-12-05T06:37:42.773 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:37:42.773 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":343,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":11887,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/958796360","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":958796360},{"type":"v1","addr":"172.21.15.42:6835","nonce":958796360}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":343}],"filesystems":[{"mdsmap":{"epoch":342,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:35:40.288210+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":111,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7910},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7910":{"gid":7910,"name":"b","rank":0,"incarnation":339,"state":"up:active","state_seq":33,"addr":"172.21.15.40:6833/4253974783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":4253974783},{"type":"v1","addr":"172.21.15.40:6833","nonce":4253974783}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:37:42.773 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 343 2019-12-05T06:37:42.775 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.769+0000 7f0982b8e700 1 -- 172.21.15.42:0/3930272723 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f095c03ae30 msgr2=0x7f095c03d340 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:37:42.775 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.769+0000 7f0982b8e700 1 --2- 172.21.15.42:0/3930272723 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f095c03ae30 0x7f095c03d340 crc :-1 s=READY pgs=1203 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:37:42.776 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.769+0000 7f0982b8e700 1 -- 172.21.15.42:0/3930272723 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f097c11b050 msgr2=0x7f097c08f5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:37:42.776 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.769+0000 7f0982b8e700 1 --2- 172.21.15.42:0/3930272723 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f097c11b050 0x7f097c08f5a0 secure :-1 s=READY pgs=643 cs=0 l=1 rx=0x7f096c000b80 tx=0x7f096c010880).stop 2019-12-05T06:37:42.776 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.769+0000 7f0982b8e700 1 -- 172.21.15.42:0/3930272723 shutdown_connections 2019-12-05T06:37:42.776 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.769+0000 7f0982b8e700 1 --2- 172.21.15.42:0/3930272723 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f095c03ae30 0x7f095c03d340 unknown :-1 s=CLOSED pgs=1203 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:37:42.776 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.769+0000 7f0982b8e700 1 --2- 172.21.15.42:0/3930272723 >> v2:172.21.15.42:3300/0 conn(0x7f097c144a80 0x7f097c144eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:37:42.776 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.769+0000 7f0982b8e700 1 --2- 172.21.15.42:0/3930272723 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f097c11b050 0x7f097c08f5a0 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:37:42.777 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.769+0000 7f0982b8e700 1 -- 172.21.15.42:0/3930272723 shutdown_connections 2019-12-05T06:37:42.777 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.769+0000 7f0982b8e700 1 -- 172.21.15.42:0/3930272723 wait complete. 2019-12-05T06:37:42.777 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:42.769+0000 7f0982b8e700 1 -- 172.21.15.42:0/3930272723 >> 172.21.15.42:0/3930272723 conn(0x7f097c07e4f0 msgr2=0x7f097c11c610 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:37:42.785 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:37:44.014 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:37:44.018 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:37:44.786 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:37:44.892 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:44.881+0000 7feb1a0a9700 1 Processor -- start 2019-12-05T06:37:44.893 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:44.881+0000 7feb1a0a9700 1 -- start start 2019-12-05T06:37:44.893 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:44.885+0000 7feb1a0a9700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7feb1408df80 0x7feb1409f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:37:44.893 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:44.885+0000 7feb1a0a9700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7feb1409f830 0x7feb1414efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:37:44.893 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:44.885+0000 7feb1a0a9700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb1409fc60 con 0x7feb140a7740 2019-12-05T06:37:44.893 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:44.885+0000 7feb1a0a9700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb140a0b00 con 0x7feb1409f830 2019-12-05T06:37:44.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:44.885+0000 7feb1a0a9700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb1408e5b0 con 0x7feb1408df80 2019-12-05T06:37:44.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:44.885+0000 7feb18e47700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7feb1409f830 0x7feb1414efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:37:44.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:44.885+0000 7feb13fff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7feb140a7740 0x7feb1407e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:58974/0 (socket says 172.21.15.42:58974) 2019-12-05T06:37:44.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:44.885+0000 7feb18e47700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7feb1409f830 0x7feb1414efe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:58938/0 (socket says 172.21.15.42:58938) 2019-12-05T06:37:44.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:44.885+0000 7feb13fff700 1 -- 172.21.15.42:0/2192602788 learned_addr learned my addr 172.21.15.42:0/2192602788 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:37:44.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:44.885+0000 7feb137fe700 1 --2- 172.21.15.42:0/2192602788 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7feb1408df80 0x7feb1409f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:37:44.895 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:44.885+0000 7feb18e47700 1 -- 172.21.15.42:0/2192602788 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7feb1408df80 msgr2=0x7feb1409f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:37:44.895 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:44.885+0000 7feb18e47700 1 --2- 172.21.15.42:0/2192602788 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7feb1408df80 0x7feb1409f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:37:44.895 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:44.885+0000 7feb18e47700 1 -- 172.21.15.42:0/2192602788 >> v1:172.21.15.42:6789/0 conn(0x7feb140a7740 legacy=0x7feb1407e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:37:44.895 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:44.885+0000 7feb18e47700 1 -- 172.21.15.42:0/2192602788 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb140a0710 con 0x7feb1409f830 2019-12-05T06:37:44.895 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:44.885+0000 7feb18e47700 1 --2- 172.21.15.42:0/2192602788 >> v2:172.21.15.42:3300/0 conn(0x7feb1409f830 0x7feb1414efe0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7feb0c008b10 tx=0x7feb0c00fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:44.895 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:44.885+0000 7feb18e47700 1 --2- 172.21.15.42:0/2192602788 >> v2:172.21.15.42:3300/0 conn(0x7feb1409f830 0x7feb1414efe0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7feb0c008b10 tx=0x7feb0c00fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:44.895 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:44.885+0000 7feb18e47700 1 --2- 172.21.15.42:0/2192602788 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feb1409f830 0x7feb1414efe0 secure :-1 s=READY pgs=2861 cs=0 l=1 rx=0x7feb0c008b10 tx=0x7feb0c00fff0).ready entity=mon.1 client_cookie=94b96a7ddf19a685 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:37:44.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:44.885+0000 7feb18e47700 1 --2- 172.21.15.42:0/2192602788 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feb1409f830 0x7feb1414efe0 secure :-1 s=THROTTLE_DONE pgs=2861 cs=0 l=1 rx=0x7feb0c008b10 tx=0x7feb0c00fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:44.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:44.885+0000 7feb12ffd700 1 -- 172.21.15.42:0/2192602788 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7feb0c010900 con 0x7feb1409f830 2019-12-05T06:37:44.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:44.885+0000 7feb18e47700 1 --2- 172.21.15.42:0/2192602788 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feb1409f830 0x7feb1414efe0 secure :-1 s=THROTTLE_DONE pgs=2861 cs=0 l=1 rx=0x7feb0c008b10 tx=0x7feb0c00fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:44.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:44.885+0000 7feb18e47700 1 --2- 172.21.15.42:0/2192602788 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feb1409f830 0x7feb1414efe0 secure :-1 s=THROTTLE_DONE pgs=2861 cs=0 l=1 rx=0x7feb0c008b10 tx=0x7feb0c00fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:44.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:44.885+0000 7feb12ffd700 1 -- 172.21.15.42:0/2192602788 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb0c008b50 con 0x7feb1409f830 2019-12-05T06:37:44.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:44.885+0000 7feb12ffd700 1 -- 172.21.15.42:0/2192602788 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7feb0c0110c0 con 0x7feb1409f830 2019-12-05T06:37:44.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:44.885+0000 7feb1a0a9700 1 -- 172.21.15.42:0/2192602788 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feb1409f830 msgr2=0x7feb1414efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:37:44.897 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:44.885+0000 7feb1a0a9700 1 --2- 172.21.15.42:0/2192602788 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feb1409f830 0x7feb1414efe0 secure :-1 s=READY pgs=2861 cs=0 l=1 rx=0x7feb0c008b10 tx=0x7feb0c00fff0).stop 2019-12-05T06:37:44.897 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:44.885+0000 7feb1a0a9700 1 -- 172.21.15.42:0/2192602788 shutdown_connections 2019-12-05T06:37:44.897 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:44.885+0000 7feb1a0a9700 1 --2- 172.21.15.42:0/2192602788 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feb1409f830 0x7feb1414efe0 unknown :-1 s=CLOSED pgs=2861 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:37:44.897 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:44.885+0000 7feb1a0a9700 1 --2- 172.21.15.42:0/2192602788 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7feb1408df80 0x7feb1409f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:37:44.897 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:44.885+0000 7feb1a0a9700 1 -- 172.21.15.42:0/2192602788 shutdown_connections 2019-12-05T06:37:44.897 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:44.885+0000 7feb1a0a9700 1 -- 172.21.15.42:0/2192602788 wait complete. 2019-12-05T06:37:44.898 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:44.885+0000 7feb1a0a9700 1 -- 172.21.15.42:0/2192602788 >> 172.21.15.42:0/2192602788 conn(0x7feb140aabe0 msgr2=0x7feb140a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:37:44.900 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:44.889+0000 7feb1a0a9700 1 Processor -- start 2019-12-05T06:37:44.901 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:44.889+0000 7feb1a0a9700 1 -- start start 2019-12-05T06:37:44.901 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:44.889+0000 7feb1a0a9700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7feb141172a0 0x7feb14123790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:37:44.901 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:44.889+0000 7feb1a0a9700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7feb140a2fa0 0x7feb1412dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:37:44.901 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:44.889+0000 7feb1a0a9700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb1407dc60 con 0x7feb14123ce0 2019-12-05T06:37:44.901 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:44.893+0000 7feb1a0a9700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb14124110 con 0x7feb140a2fa0 2019-12-05T06:37:44.901 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:44.893+0000 7feb1a0a9700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb14142530 con 0x7feb141172a0 2019-12-05T06:37:44.902 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:44.893+0000 7feb137fe700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7feb14123ce0 0x7feb140a27a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:58982/0 (socket says 172.21.15.42:58982) 2019-12-05T06:37:44.902 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:44.893+0000 7feb18e47700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7feb140a2fa0 0x7feb1412dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:37:44.902 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:44.893+0000 7feb137fe700 1 -- 172.21.15.42:0/3781309810 learned_addr learned my addr 172.21.15.42:0/3781309810 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:37:44.902 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:44.893+0000 7feb13fff700 1 --2- 172.21.15.42:0/3781309810 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7feb141172a0 0x7feb14123790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:37:44.903 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:44.893+0000 7feb18e47700 1 -- 172.21.15.42:0/3781309810 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7feb141172a0 msgr2=0x7feb14123790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:37:44.903 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:44.893+0000 7feb18e47700 1 --2- 172.21.15.42:0/3781309810 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7feb141172a0 0x7feb14123790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:37:44.903 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:44.893+0000 7feb18e47700 1 -- 172.21.15.42:0/3781309810 >> v1:172.21.15.42:6789/0 conn(0x7feb14123ce0 legacy=0x7feb140a27a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:37:44.903 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:44.893+0000 7feb10ff9700 1 -- 172.21.15.42:0/3781309810 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2421885085 0 0) 0x7feb1407dc60 con 0x7feb14123ce0 2019-12-05T06:37:44.904 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:44.893+0000 7feb18e47700 1 -- 172.21.15.42:0/3781309810 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb140a15a0 con 0x7feb140a2fa0 2019-12-05T06:37:44.904 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:44.893+0000 7feb13fff700 1 --2- 172.21.15.42:0/3781309810 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7feb141172a0 0x7feb14123790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:37:44.904 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:44.893+0000 7feb18e47700 1 --2- 172.21.15.42:0/3781309810 >> v2:172.21.15.42:3300/0 conn(0x7feb140a2fa0 0x7feb1412dbc0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7feb141426f0 tx=0x7feb0c010cf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:44.904 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:44.893+0000 7feb18e47700 1 --2- 172.21.15.42:0/3781309810 >> v2:172.21.15.42:3300/0 conn(0x7feb140a2fa0 0x7feb1412dbc0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7feb141426f0 tx=0x7feb0c010cf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:44.904 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:44.893+0000 7feb18e47700 1 --2- 172.21.15.42:0/3781309810 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feb140a2fa0 0x7feb1412dbc0 secure :-1 s=READY pgs=2863 cs=0 l=1 rx=0x7feb141426f0 tx=0x7feb0c010cf0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:37:44.904 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:44.893+0000 7feb18e47700 1 --2- 172.21.15.42:0/3781309810 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feb140a2fa0 0x7feb1412dbc0 secure :-1 s=THROTTLE_DONE pgs=2863 cs=0 l=1 rx=0x7feb141426f0 tx=0x7feb0c010cf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:44.904 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:44.893+0000 7feb10ff9700 1 -- 172.21.15.42:0/3781309810 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7feb0c005ab0 con 0x7feb140a2fa0 2019-12-05T06:37:44.905 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:44.893+0000 7feb18e47700 1 --2- 172.21.15.42:0/3781309810 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feb140a2fa0 0x7feb1412dbc0 secure :-1 s=THROTTLE_DONE pgs=2863 cs=0 l=1 rx=0x7feb141426f0 tx=0x7feb0c010cf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:44.905 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:44.893+0000 7feb1a0a9700 1 -- 172.21.15.42:0/3781309810 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb141176d0 con 0x7feb140a2fa0 2019-12-05T06:37:44.905 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:44.893+0000 7feb10ff9700 1 -- 172.21.15.42:0/3781309810 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb0c010b00 con 0x7feb140a2fa0 2019-12-05T06:37:44.905 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:44.893+0000 7feb18e47700 1 --2- 172.21.15.42:0/3781309810 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feb140a2fa0 0x7feb1412dbc0 secure :-1 s=THROTTLE_DONE pgs=2863 cs=0 l=1 rx=0x7feb141426f0 tx=0x7feb0c010cf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:44.905 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:44.893+0000 7feb10ff9700 1 -- 172.21.15.42:0/3781309810 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7feb0c00f680 con 0x7feb140a2fa0 2019-12-05T06:37:44.905 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:44.893+0000 7feb1a0a9700 1 -- 172.21.15.42:0/3781309810 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb1412e110 con 0x7feb140a2fa0 2019-12-05T06:37:44.905 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:44.893+0000 7feb18e47700 1 --2- 172.21.15.42:0/3781309810 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feb140a2fa0 0x7feb1412dbc0 secure :-1 s=THROTTLE_DONE pgs=2863 cs=0 l=1 rx=0x7feb141426f0 tx=0x7feb0c010cf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:44.906 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:44.893+0000 7feb10ff9700 1 -- 172.21.15.42:0/3781309810 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7feb0c011360 con 0x7feb140a2fa0 2019-12-05T06:37:44.906 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:44.893+0000 7feb18e47700 1 --2- 172.21.15.42:0/3781309810 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feb140a2fa0 0x7feb1412dbc0 secure :-1 s=THROTTLE_DONE pgs=2863 cs=0 l=1 rx=0x7feb141426f0 tx=0x7feb0c010cf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:44.906 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:44.893+0000 7feb10ff9700 1 --2- 172.21.15.42:0/3781309810 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7feaf403ac50 0x7feaf403d160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:37:44.906 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:44.893+0000 7feb10ff9700 1 -- 172.21.15.42:0/3781309810 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 11363+0+0 (secure 0 0 0) 0x7feb0c04b050 con 0x7feb140a2fa0 2019-12-05T06:37:44.906 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:44.893+0000 7feb13fff700 1 --2- 172.21.15.42:0/3781309810 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7feaf403ac50 0x7feaf403d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:37:44.906 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:44.897+0000 7feb13fff700 1 --2- 172.21.15.42:0/3781309810 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7feaf403ac50 0x7feaf403d160 crc :-1 s=READY pgs=1204 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:37:44.907 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:44.897+0000 7feb1a0a9700 1 -- 172.21.15.42:0/3781309810 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb1405d110 con 0x7feb140a2fa0 2019-12-05T06:37:44.910 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:44.905+0000 7feb18e47700 1 --2- 172.21.15.42:0/3781309810 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feb140a2fa0 0x7feb1412dbc0 secure :-1 s=THROTTLE_DONE pgs=2863 cs=0 l=1 rx=0x7feb141426f0 tx=0x7feb0c010cf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:44.910 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:44.905+0000 7feb10ff9700 1 -- 172.21.15.42:0/3781309810 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7feb0c011c60 con 0x7feb140a2fa0 2019-12-05T06:37:45.080 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:45.073+0000 7feb1a0a9700 1 -- 172.21.15.42:0/3781309810 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7feb1405b8c0 con 0x7feb140a2fa0 2019-12-05T06:37:45.081 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:45.077+0000 7feb18e47700 1 --2- 172.21.15.42:0/3781309810 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feb140a2fa0 0x7feb1412dbc0 secure :-1 s=THROTTLE_DONE pgs=2863 cs=0 l=1 rx=0x7feb141426f0 tx=0x7feb0c010cf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:45.081 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:45.077+0000 7feb10ff9700 1 -- 172.21.15.42:0/3781309810 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 343 v343) v1 ==== 95+0+2111 (secure 0 0 0) 0x7feb0c02fd80 con 0x7feb140a2fa0 2019-12-05T06:37:45.081 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 343 2019-12-05T06:37:45.082 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:37:45.082 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":343,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":11887,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/958796360","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":958796360},{"type":"v1","addr":"172.21.15.42:6835","nonce":958796360}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":343}],"filesystems":[{"mdsmap":{"epoch":342,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:35:40.288210+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":111,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7910},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7910":{"gid":7910,"name":"b","rank":0,"incarnation":339,"state":"up:active","state_seq":33,"addr":"172.21.15.40:6833/4253974783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":4253974783},{"type":"v1","addr":"172.21.15.40:6833","nonce":4253974783}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:37:45.084 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:45.077+0000 7feb1a0a9700 1 -- 172.21.15.42:0/3781309810 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7feaf403ac50 msgr2=0x7feaf403d160 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:37:45.084 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:45.077+0000 7feb1a0a9700 1 --2- 172.21.15.42:0/3781309810 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7feaf403ac50 0x7feaf403d160 crc :-1 s=READY pgs=1204 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:37:45.084 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:45.077+0000 7feb1a0a9700 1 -- 172.21.15.42:0/3781309810 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feb140a2fa0 msgr2=0x7feb1412dbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:37:45.084 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:45.077+0000 7feb1a0a9700 1 --2- 172.21.15.42:0/3781309810 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feb140a2fa0 0x7feb1412dbc0 secure :-1 s=READY pgs=2863 cs=0 l=1 rx=0x7feb141426f0 tx=0x7feb0c010cf0).stop 2019-12-05T06:37:45.084 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:45.077+0000 7feb1a0a9700 1 -- 172.21.15.42:0/3781309810 shutdown_connections 2019-12-05T06:37:45.084 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:45.077+0000 7feb1a0a9700 1 --2- 172.21.15.42:0/3781309810 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7feaf403ac50 0x7feaf403d160 unknown :-1 s=CLOSED pgs=1204 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:37:45.085 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:45.077+0000 7feb1a0a9700 1 --2- 172.21.15.42:0/3781309810 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feb140a2fa0 0x7feb1412dbc0 unknown :-1 s=CLOSED pgs=2863 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:37:45.085 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:45.077+0000 7feb1a0a9700 1 --2- 172.21.15.42:0/3781309810 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7feb141172a0 0x7feb14123790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:37:45.085 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:45.077+0000 7feb1a0a9700 1 -- 172.21.15.42:0/3781309810 shutdown_connections 2019-12-05T06:37:45.085 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:45.077+0000 7feb1a0a9700 1 -- 172.21.15.42:0/3781309810 wait complete. 2019-12-05T06:37:45.085 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:45.077+0000 7feb1a0a9700 1 -- 172.21.15.42:0/3781309810 >> 172.21.15.42:0/3781309810 conn(0x7feb140a82a0 msgr2=0x7feb14115e80 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:37:45.093 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:37:47.094 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:37:47.198 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:47.189+0000 7f6a07080700 1 Processor -- start 2019-12-05T06:37:47.199 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:47.189+0000 7f6a07080700 1 -- start start 2019-12-05T06:37:47.199 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:47.189+0000 7f6a07080700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f6a000a7730 0x7f6a0008df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:37:47.199 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:47.189+0000 7f6a07080700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6a0007e710 0x7f6a0007eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:37:47.199 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:47.189+0000 7f6a07080700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a000a0700 con 0x7f6a0009f2d0 2019-12-05T06:37:47.200 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:47.189+0000 7f6a07080700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a000a0af0 con 0x7f6a000a7730 2019-12-05T06:37:47.200 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:47.189+0000 7f6a07080700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a0008e5a0 con 0x7f6a0007e710 2019-12-05T06:37:47.200 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:47.189+0000 7f6a04e1c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6a0007e710 0x7f6a0007eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:37:47.200 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:47.189+0000 7f6a04e1c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6a0007e710 0x7f6a0007eb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:58952/0 (socket says 172.21.15.42:58952) 2019-12-05T06:37:47.201 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:47.189+0000 7f6a04e1c700 1 -- 172.21.15.42:0/3982008532 learned_addr learned my addr 172.21.15.42:0/3982008532 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:37:47.201 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:47.189+0000 7f6a0561d700 1 --2- 172.21.15.42:0/3982008532 >> v2:172.21.15.40:3300/0 conn(0x7f6a000a7730 0x7f6a0008df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:37:47.201 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:47.189+0000 7f6a04e1c700 1 -- 172.21.15.42:0/3982008532 >> v2:172.21.15.40:3300/0 conn(0x7f6a000a7730 msgr2=0x7f6a0008df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:37:47.201 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:47.189+0000 7f6a04e1c700 1 --2- 172.21.15.42:0/3982008532 >> v2:172.21.15.40:3300/0 conn(0x7f6a000a7730 0x7f6a0008df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:37:47.202 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:47.189+0000 7f6a04e1c700 1 -- 172.21.15.42:0/3982008532 >> v1:172.21.15.40:6789/0 conn(0x7f6a0009f2d0 legacy=0x7f6a0009f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:37:47.202 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:47.189+0000 7f6a04e1c700 1 -- 172.21.15.42:0/3982008532 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a0007f090 con 0x7f6a0007e710 2019-12-05T06:37:47.202 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:47.189+0000 7f6a04e1c700 1 --2- 172.21.15.42:0/3982008532 >> v2:172.21.15.42:3300/0 conn(0x7f6a0007e710 0x7f6a0007eb40 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f69fc002f30 tx=0x7f69fc00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:47.202 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:47.189+0000 7f6a04e1c700 1 --2- 172.21.15.42:0/3982008532 >> v2:172.21.15.42:3300/0 conn(0x7f6a0007e710 0x7f6a0007eb40 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f69fc002f30 tx=0x7f69fc00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:47.202 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:47.189+0000 7f6a04e1c700 1 --2- 172.21.15.42:0/3982008532 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6a0007e710 0x7f6a0007eb40 secure :-1 s=READY pgs=2864 cs=0 l=1 rx=0x7f69fc002f30 tx=0x7f69fc00fd30).ready entity=mon.1 client_cookie=1082fca9cd2859bb server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:37:47.203 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:47.189+0000 7f6a04e1c700 1 --2- 172.21.15.42:0/3982008532 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6a0007e710 0x7f6a0007eb40 secure :-1 s=THROTTLE_DONE pgs=2864 cs=0 l=1 rx=0x7f69fc002f30 tx=0x7f69fc00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:47.203 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:47.189+0000 7f69f7fff700 1 -- 172.21.15.42:0/3982008532 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f69fc011370 con 0x7f6a0007e710 2019-12-05T06:37:47.203 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:47.189+0000 7f6a04e1c700 1 --2- 172.21.15.42:0/3982008532 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6a0007e710 0x7f6a0007eb40 secure :-1 s=THROTTLE_DONE pgs=2864 cs=0 l=1 rx=0x7f69fc002f30 tx=0x7f69fc00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:47.203 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:47.189+0000 7f69f7fff700 1 -- 172.21.15.42:0/3982008532 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69fc008bb0 con 0x7f6a0007e710 2019-12-05T06:37:47.203 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:47.189+0000 7f6a04e1c700 1 --2- 172.21.15.42:0/3982008532 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6a0007e710 0x7f6a0007eb40 secure :-1 s=THROTTLE_DONE pgs=2864 cs=0 l=1 rx=0x7f69fc002f30 tx=0x7f69fc00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:47.204 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:47.189+0000 7f69f7fff700 1 -- 172.21.15.42:0/3982008532 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f69fc011950 con 0x7f6a0007e710 2019-12-05T06:37:47.204 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:47.189+0000 7f6a07080700 1 -- 172.21.15.42:0/3982008532 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6a0007e710 msgr2=0x7f6a0007eb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:37:47.204 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:47.189+0000 7f6a07080700 1 --2- 172.21.15.42:0/3982008532 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6a0007e710 0x7f6a0007eb40 secure :-1 s=READY pgs=2864 cs=0 l=1 rx=0x7f69fc002f30 tx=0x7f69fc00fd30).stop 2019-12-05T06:37:47.204 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:47.193+0000 7f6a07080700 1 -- 172.21.15.42:0/3982008532 shutdown_connections 2019-12-05T06:37:47.204 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:47.193+0000 7f6a07080700 1 --2- 172.21.15.42:0/3982008532 >> v2:172.21.15.40:3300/0 conn(0x7f6a000a7730 0x7f6a0008df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:37:47.205 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:47.193+0000 7f6a07080700 1 --2- 172.21.15.42:0/3982008532 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6a0007e710 0x7f6a0007eb40 unknown :-1 s=CLOSED pgs=2864 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:37:47.205 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:47.193+0000 7f6a07080700 1 -- 172.21.15.42:0/3982008532 shutdown_connections 2019-12-05T06:37:47.205 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:47.193+0000 7f6a07080700 1 -- 172.21.15.42:0/3982008532 wait complete. 2019-12-05T06:37:47.205 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:47.193+0000 7f6a07080700 1 -- 172.21.15.42:0/3982008532 >> 172.21.15.42:0/3982008532 conn(0x7f6a000aabd0 msgr2=0x7f6a000a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:37:47.206 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:47.193+0000 7f6a07080700 1 Processor -- start 2019-12-05T06:37:47.206 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:47.193+0000 7f6a07080700 1 -- start start 2019-12-05T06:37:47.211 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:47.193+0000 7f6a07080700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6a000a3a50 0x7f6a0014dad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:37:47.211 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:47.193+0000 7f6a07080700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f6a000a6e40 0x7f6a0014e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:37:47.211 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:47.193+0000 7f6a07080700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a000a6390 con 0x7f6a0007b110 2019-12-05T06:37:47.212 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:47.193+0000 7f6a07080700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a0007bb90 con 0x7f6a000a6e40 2019-12-05T06:37:47.212 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:47.193+0000 7f6a07080700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a00128120 con 0x7f6a000a3a50 2019-12-05T06:37:47.212 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:47.193+0000 7f6a04e1c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6a000a3a50 0x7f6a0014dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:37:47.212 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:47.193+0000 7f6a04e1c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6a000a3a50 0x7f6a0014dad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:58960/0 (socket says 172.21.15.42:58960) 2019-12-05T06:37:47.212 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:47.193+0000 7f6a04e1c700 1 -- 172.21.15.42:0/3732256600 learned_addr learned my addr 172.21.15.42:0/3732256600 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:37:47.213 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:47.193+0000 7f6a05e1e700 1 --2- 172.21.15.42:0/3732256600 >> v2:172.21.15.40:3300/0 conn(0x7f6a000a6e40 0x7f6a0014e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:37:47.213 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:47.193+0000 7f6a04e1c700 1 -- 172.21.15.42:0/3732256600 >> v2:172.21.15.40:3300/0 conn(0x7f6a000a6e40 msgr2=0x7f6a0014e020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:37:47.213 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:47.193+0000 7f6a04e1c700 1 --2- 172.21.15.42:0/3732256600 >> v2:172.21.15.40:3300/0 conn(0x7f6a000a6e40 0x7f6a0014e020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:37:47.213 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:47.193+0000 7f6a04e1c700 1 -- 172.21.15.42:0/3732256600 >> v1:172.21.15.40:6789/0 conn(0x7f6a0007b110 legacy=0x7f6a000a5b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:37:47.213 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:47.193+0000 7f6a04e1c700 1 -- 172.21.15.42:0/3732256600 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a0007a080 con 0x7f6a000a3a50 2019-12-05T06:37:47.213 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:47.193+0000 7f6a04e1c700 1 --2- 172.21.15.42:0/3732256600 >> v2:172.21.15.42:3300/0 conn(0x7f6a000a3a50 0x7f6a0014dad0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f6a000a3fe0 tx=0x7f69fc008de0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:47.214 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:47.197+0000 7f6a04e1c700 1 --2- 172.21.15.42:0/3732256600 >> v2:172.21.15.42:3300/0 conn(0x7f6a000a3a50 0x7f6a0014dad0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f6a000a3fe0 tx=0x7f69fc008de0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:47.214 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:47.197+0000 7f6a04e1c700 1 --2- 172.21.15.42:0/3732256600 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6a000a3a50 0x7f6a0014dad0 secure :-1 s=READY pgs=2865 cs=0 l=1 rx=0x7f6a000a3fe0 tx=0x7f69fc008de0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:37:47.214 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:47.197+0000 7f6a04e1c700 1 --2- 172.21.15.42:0/3732256600 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6a000a3a50 0x7f6a0014dad0 secure :-1 s=THROTTLE_DONE pgs=2865 cs=0 l=1 rx=0x7f6a000a3fe0 tx=0x7f69fc008de0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:47.214 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:47.197+0000 7f69f5ffb700 1 -- 172.21.15.42:0/3732256600 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f69fc007150 con 0x7f6a000a3a50 2019-12-05T06:37:47.214 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:47.197+0000 7f6a07080700 1 -- 172.21.15.42:0/3732256600 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a0014e570 con 0x7f6a000a3a50 2019-12-05T06:37:47.214 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:47.197+0000 7f6a04e1c700 1 --2- 172.21.15.42:0/3732256600 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6a000a3a50 0x7f6a0014dad0 secure :-1 s=THROTTLE_DONE pgs=2865 cs=0 l=1 rx=0x7f6a000a3fe0 tx=0x7f69fc008de0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:47.215 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:47.197+0000 7f69f5ffb700 1 -- 172.21.15.42:0/3732256600 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69fc011570 con 0x7f6a000a3a50 2019-12-05T06:37:47.215 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:47.197+0000 7f6a07080700 1 -- 172.21.15.42:0/3732256600 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a0012f840 con 0x7f6a000a3a50 2019-12-05T06:37:47.215 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:47.197+0000 7f6a04e1c700 1 --2- 172.21.15.42:0/3732256600 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6a000a3a50 0x7f6a0014dad0 secure :-1 s=THROTTLE_DONE pgs=2865 cs=0 l=1 rx=0x7f6a000a3fe0 tx=0x7f69fc008de0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:47.215 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:47.197+0000 7f69f5ffb700 1 -- 172.21.15.42:0/3732256600 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f69fc005730 con 0x7f6a000a3a50 2019-12-05T06:37:47.215 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:47.197+0000 7f6a04e1c700 1 --2- 172.21.15.42:0/3732256600 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6a000a3a50 0x7f6a0014dad0 secure :-1 s=THROTTLE_DONE pgs=2865 cs=0 l=1 rx=0x7f6a000a3fe0 tx=0x7f69fc008de0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:47.215 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:47.197+0000 7f69f5ffb700 1 -- 172.21.15.42:0/3732256600 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f69fc005a10 con 0x7f6a000a3a50 2019-12-05T06:37:47.216 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:47.197+0000 7f6a04e1c700 1 --2- 172.21.15.42:0/3732256600 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6a000a3a50 0x7f6a0014dad0 secure :-1 s=THROTTLE_DONE pgs=2865 cs=0 l=1 rx=0x7f6a000a3fe0 tx=0x7f69fc008de0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:47.216 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:47.197+0000 7f69f5ffb700 1 --2- 172.21.15.42:0/3732256600 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f69e003adb0 0x7f69e003d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:37:47.216 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:47.197+0000 7f69f5ffb700 1 -- 172.21.15.42:0/3732256600 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 11363+0+0 (secure 0 0 0) 0x7f69fc04b560 con 0x7f6a000a3a50 2019-12-05T06:37:47.216 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:47.197+0000 7f6a0561d700 1 --2- 172.21.15.42:0/3732256600 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f69e003adb0 0x7f69e003d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:37:47.216 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:47.197+0000 7f6a0561d700 1 --2- 172.21.15.42:0/3732256600 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f69e003adb0 0x7f69e003d2c0 crc :-1 s=READY pgs=1205 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:37:47.216 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:47.205+0000 7f6a07080700 1 -- 172.21.15.42:0/3732256600 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a000aae00 con 0x7f6a000a3a50 2019-12-05T06:37:47.217 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:47.209+0000 7f6a04e1c700 1 --2- 172.21.15.42:0/3732256600 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6a000a3a50 0x7f6a0014dad0 secure :-1 s=THROTTLE_DONE pgs=2865 cs=0 l=1 rx=0x7f6a000a3fe0 tx=0x7f69fc008de0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:47.217 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:47.209+0000 7f69f5ffb700 1 -- 172.21.15.42:0/3732256600 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f69fc032820 con 0x7f6a000a3a50 2019-12-05T06:37:47.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:47.377+0000 7f6a07080700 1 -- 172.21.15.42:0/3732256600 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6a0005b8b0 con 0x7f6a000a3a50 2019-12-05T06:37:47.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:47.377+0000 7f6a04e1c700 1 --2- 172.21.15.42:0/3732256600 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6a000a3a50 0x7f6a0014dad0 secure :-1 s=THROTTLE_DONE pgs=2865 cs=0 l=1 rx=0x7f6a000a3fe0 tx=0x7f69fc008de0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:47.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:47.377+0000 7f69f5ffb700 1 -- 172.21.15.42:0/3732256600 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 343 v343) v1 ==== 95+0+2111 (secure 0 0 0) 0x7f69fc00ebd0 con 0x7f6a000a3a50 2019-12-05T06:37:47.408 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 343 2019-12-05T06:37:47.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:47.377+0000 7f6a07080700 1 -- 172.21.15.42:0/3732256600 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f69e003adb0 msgr2=0x7f69e003d2c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:37:47.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:47.377+0000 7f6a07080700 1 --2- 172.21.15.42:0/3732256600 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f69e003adb0 0x7f69e003d2c0 crc :-1 s=READY pgs=1205 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:37:47.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:47.377+0000 7f6a07080700 1 -- 172.21.15.42:0/3732256600 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6a000a3a50 msgr2=0x7f6a0014dad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:37:47.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:47.377+0000 7f6a07080700 1 --2- 172.21.15.42:0/3732256600 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6a000a3a50 0x7f6a0014dad0 secure :-1 s=READY pgs=2865 cs=0 l=1 rx=0x7f6a000a3fe0 tx=0x7f69fc008de0).stop 2019-12-05T06:37:47.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:47.377+0000 7f6a07080700 1 -- 172.21.15.42:0/3732256600 shutdown_connections 2019-12-05T06:37:47.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:47.377+0000 7f6a07080700 1 --2- 172.21.15.42:0/3732256600 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f69e003adb0 0x7f69e003d2c0 unknown :-1 s=CLOSED pgs=1205 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:37:47.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:47.377+0000 7f6a07080700 1 --2- 172.21.15.42:0/3732256600 >> v2:172.21.15.40:3300/0 conn(0x7f6a000a6e40 0x7f6a0014e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:37:47.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:47.377+0000 7f6a07080700 1 --2- 172.21.15.42:0/3732256600 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6a000a3a50 0x7f6a0014dad0 unknown :-1 s=CLOSED pgs=2865 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:37:47.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:47.377+0000 7f6a07080700 1 -- 172.21.15.42:0/3732256600 shutdown_connections 2019-12-05T06:37:47.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:47.377+0000 7f6a07080700 1 -- 172.21.15.42:0/3732256600 wait complete. 2019-12-05T06:37:47.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:47.377+0000 7f6a07080700 1 -- 172.21.15.42:0/3732256600 >> 172.21.15.42:0/3732256600 conn(0x7f6a00116dc0 msgr2=0x7f6a00059bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:37:47.410 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:37:47.410 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":343,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":11887,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/958796360","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":958796360},{"type":"v1","addr":"172.21.15.42:6835","nonce":958796360}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":343}],"filesystems":[{"mdsmap":{"epoch":342,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:35:40.288210+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":111,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7910},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7910":{"gid":7910,"name":"b","rank":0,"incarnation":339,"state":"up:active","state_seq":33,"addr":"172.21.15.40:6833/4253974783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":4253974783},{"type":"v1","addr":"172.21.15.40:6833","nonce":4253974783}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:37:47.411 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:37:49.412 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:37:49.520 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:49.509+0000 7f0a2d16a700 1 Processor -- start 2019-12-05T06:37:49.520 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:49.509+0000 7f0a2d16a700 1 -- start start 2019-12-05T06:37:49.521 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:49.513+0000 7f0a2d16a700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0a2808df80 0x7f0a2809f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:37:49.521 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:49.513+0000 7f0a2d16a700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0a2809f830 0x7f0a2814efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:37:49.521 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:49.513+0000 7f0a2d16a700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a2809fc60 con 0x7f0a280a7740 2019-12-05T06:37:49.521 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:49.513+0000 7f0a2d16a700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a280a0b00 con 0x7f0a2809f830 2019-12-05T06:37:49.522 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:49.513+0000 7f0a2d16a700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a2808e5b0 con 0x7f0a2808df80 2019-12-05T06:37:49.522 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:49.513+0000 7f0a2759e700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f0a280a7740 0x7f0a2807e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:59006/0 (socket says 172.21.15.42:59006) 2019-12-05T06:37:49.522 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:49.513+0000 7f0a2759e700 1 -- 172.21.15.42:0/18886128 learned_addr learned my addr 172.21.15.42:0/18886128 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:37:49.522 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:49.513+0000 7f0a2c969700 1 --2- 172.21.15.42:0/18886128 >> v2:172.21.15.42:3300/0 conn(0x7f0a2809f830 0x7f0a2814efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:37:49.522 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:49.513+0000 7f0a26d9d700 1 --2- 172.21.15.42:0/18886128 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0a2808df80 0x7f0a2809f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:37:49.523 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:49.513+0000 7f0a2659c700 1 -- 172.21.15.42:0/18886128 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1622669983 0 0) 0x7f0a2809fc60 con 0x7f0a280a7740 2019-12-05T06:37:49.523 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:49.513+0000 7f0a2659c700 1 -- 172.21.15.42:0/18886128 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a140030b0 con 0x7f0a280a7740 2019-12-05T06:37:49.523 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:49.513+0000 7f0a2c969700 1 -- 172.21.15.42:0/18886128 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0a2808df80 msgr2=0x7f0a2809f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:37:49.523 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:49.513+0000 7f0a2c969700 1 --2- 172.21.15.42:0/18886128 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0a2808df80 0x7f0a2809f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:37:49.524 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:49.513+0000 7f0a2c969700 1 -- 172.21.15.42:0/18886128 >> v1:172.21.15.42:6789/0 conn(0x7f0a280a7740 legacy=0x7f0a2807e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:37:49.524 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:49.513+0000 7f0a2c969700 1 -- 172.21.15.42:0/18886128 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a280a0710 con 0x7f0a2809f830 2019-12-05T06:37:49.524 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:49.513+0000 7f0a2c969700 1 --2- 172.21.15.42:0/18886128 >> v2:172.21.15.42:3300/0 conn(0x7f0a2809f830 0x7f0a2814efe0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f0a20008b10 tx=0x7f0a2000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:49.524 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:49.513+0000 7f0a2c969700 1 --2- 172.21.15.42:0/18886128 >> v2:172.21.15.42:3300/0 conn(0x7f0a2809f830 0x7f0a2814efe0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f0a20008b10 tx=0x7f0a2000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:49.524 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:49.513+0000 7f0a2c969700 1 --2- 172.21.15.42:0/18886128 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0a2809f830 0x7f0a2814efe0 secure :-1 s=READY pgs=2867 cs=0 l=1 rx=0x7f0a20008b10 tx=0x7f0a2000fff0).ready entity=mon.1 client_cookie=1374550127206ae5 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:37:49.524 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:49.513+0000 7f0a2c969700 1 --2- 172.21.15.42:0/18886128 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0a2809f830 0x7f0a2814efe0 secure :-1 s=THROTTLE_DONE pgs=2867 cs=0 l=1 rx=0x7f0a20008b10 tx=0x7f0a2000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:49.525 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:49.513+0000 7f0a2c969700 1 --2- 172.21.15.42:0/18886128 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0a2809f830 0x7f0a2814efe0 secure :-1 s=THROTTLE_DONE pgs=2867 cs=0 l=1 rx=0x7f0a20008b10 tx=0x7f0a2000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:49.525 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:49.513+0000 7f0a2659c700 1 -- 172.21.15.42:0/18886128 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0a20010900 con 0x7f0a2809f830 2019-12-05T06:37:49.525 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:49.513+0000 7f0a2c969700 1 --2- 172.21.15.42:0/18886128 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0a2809f830 0x7f0a2814efe0 secure :-1 s=THROTTLE_DONE pgs=2867 cs=0 l=1 rx=0x7f0a20008b10 tx=0x7f0a2000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:49.526 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:49.513+0000 7f0a2659c700 1 -- 172.21.15.42:0/18886128 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a20008b50 con 0x7f0a2809f830 2019-12-05T06:37:49.526 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:49.513+0000 7f0a2659c700 1 -- 172.21.15.42:0/18886128 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0a200110c0 con 0x7f0a2809f830 2019-12-05T06:37:49.526 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:49.513+0000 7f0a2d16a700 1 -- 172.21.15.42:0/18886128 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0a2809f830 msgr2=0x7f0a2814efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:37:49.526 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:49.513+0000 7f0a2d16a700 1 --2- 172.21.15.42:0/18886128 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0a2809f830 0x7f0a2814efe0 secure :-1 s=READY pgs=2867 cs=0 l=1 rx=0x7f0a20008b10 tx=0x7f0a2000fff0).stop 2019-12-05T06:37:49.526 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:49.513+0000 7f0a2d16a700 1 -- 172.21.15.42:0/18886128 shutdown_connections 2019-12-05T06:37:49.527 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:49.513+0000 7f0a2d16a700 1 --2- 172.21.15.42:0/18886128 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0a2809f830 0x7f0a2814efe0 unknown :-1 s=CLOSED pgs=2867 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:37:49.527 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:49.513+0000 7f0a2d16a700 1 --2- 172.21.15.42:0/18886128 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0a2808df80 0x7f0a2809f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:37:49.527 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:49.513+0000 7f0a2d16a700 1 -- 172.21.15.42:0/18886128 shutdown_connections 2019-12-05T06:37:49.527 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:49.513+0000 7f0a2d16a700 1 -- 172.21.15.42:0/18886128 wait complete. 2019-12-05T06:37:49.527 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:49.513+0000 7f0a2d16a700 1 -- 172.21.15.42:0/18886128 >> 172.21.15.42:0/18886128 conn(0x7f0a280aabe0 msgr2=0x7f0a280a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:37:49.532 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:49.517+0000 7f0a2d16a700 1 Processor -- start 2019-12-05T06:37:49.532 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:49.517+0000 7f0a2d16a700 1 -- start start 2019-12-05T06:37:49.532 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:49.517+0000 7f0a2d16a700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0a28143610 0x7f0a281190f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:37:49.532 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:49.517+0000 7f0a2d16a700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0a28119640 0x7f0a2807b830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:37:49.532 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:49.517+0000 7f0a2d16a700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a28145210 con 0x7f0a28147fc0 2019-12-05T06:37:49.533 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:49.517+0000 7f0a2d16a700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a28142e30 con 0x7f0a28119640 2019-12-05T06:37:49.533 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:49.517+0000 7f0a2d16a700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a281430d0 con 0x7f0a28143610 2019-12-05T06:37:49.533 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:49.517+0000 7f0a26d9d700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f0a28147fc0 0x7f0a28144a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:59014/0 (socket says 172.21.15.42:59014) 2019-12-05T06:37:49.533 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:49.517+0000 7f0a26d9d700 1 -- 172.21.15.42:0/165036947 learned_addr learned my addr 172.21.15.42:0/165036947 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:37:49.533 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:49.517+0000 7f0a2c969700 1 --2- 172.21.15.42:0/165036947 >> v2:172.21.15.42:3300/0 conn(0x7f0a28119640 0x7f0a2807b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:37:49.533 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:49.517+0000 7f0a2759e700 1 --2- 172.21.15.42:0/165036947 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0a28143610 0x7f0a281190f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:37:49.541 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:49.517+0000 7f0a2759e700 1 -- 172.21.15.42:0/165036947 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0a28143610 msgr2=0x7f0a281190f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2019-12-05T06:37:49.541 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:49.517+0000 7f0a2759e700 1 -- 172.21.15.42:0/165036947 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0a28143610 msgr2=0x7f0a281190f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2019-12-05T06:37:49.541 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:49.517+0000 7f0a2759e700 1 --2- 172.21.15.42:0/165036947 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0a28143610 0x7f0a281190f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-12-05T06:37:49.541 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:49.517+0000 7f0a2759e700 1 --2- 172.21.15.42:0/165036947 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0a28143610 0x7f0a281190f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rx=0 tx=0)._fault waiting 0.200000 2019-12-05T06:37:49.541 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:49.517+0000 7f0a2c969700 1 -- 172.21.15.42:0/165036947 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0a28143610 msgr2=0x7f0a281190f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2019-12-05T06:37:49.542 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:49.517+0000 7f0a2c969700 1 --2- 172.21.15.42:0/165036947 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0a28143610 0x7f0a281190f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:37:49.542 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:49.517+0000 7f0a2c969700 1 -- 172.21.15.42:0/165036947 >> v1:172.21.15.42:6789/0 conn(0x7f0a28147fc0 legacy=0x7f0a28144a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:37:49.542 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:49.517+0000 7f0a1ffff700 1 -- 172.21.15.42:0/165036947 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1984634908 0 0) 0x7f0a28145210 con 0x7f0a28147fc0 2019-12-05T06:37:49.542 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:49.517+0000 7f0a2c969700 1 -- 172.21.15.42:0/165036947 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a2807bd80 con 0x7f0a28119640 2019-12-05T06:37:49.542 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:49.517+0000 7f0a2c969700 1 --2- 172.21.15.42:0/165036947 >> v2:172.21.15.42:3300/0 conn(0x7f0a28119640 0x7f0a2807b830 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f0a28142ff0 tx=0x7f0a200087f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:49.542 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:49.517+0000 7f0a2c969700 1 --2- 172.21.15.42:0/165036947 >> v2:172.21.15.42:3300/0 conn(0x7f0a28119640 0x7f0a2807b830 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f0a28142ff0 tx=0x7f0a200087f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:49.542 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:49.517+0000 7f0a2c969700 1 --2- 172.21.15.42:0/165036947 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0a28119640 0x7f0a2807b830 secure :-1 s=READY pgs=2869 cs=0 l=1 rx=0x7f0a28142ff0 tx=0x7f0a200087f0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:37:49.543 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:49.517+0000 7f0a2c969700 1 --2- 172.21.15.42:0/165036947 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0a28119640 0x7f0a2807b830 secure :-1 s=THROTTLE_DONE pgs=2869 cs=0 l=1 rx=0x7f0a28142ff0 tx=0x7f0a200087f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:49.543 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:49.517+0000 7f0a1ffff700 1 -- 172.21.15.42:0/165036947 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0a20005ab0 con 0x7f0a28119640 2019-12-05T06:37:49.543 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:49.517+0000 7f0a2c969700 1 --2- 172.21.15.42:0/165036947 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0a28119640 0x7f0a2807b830 secure :-1 s=THROTTLE_DONE pgs=2869 cs=0 l=1 rx=0x7f0a28142ff0 tx=0x7f0a200087f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:49.543 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:49.521+0000 7f0a2c969700 1 --2- 172.21.15.42:0/165036947 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0a28119640 0x7f0a2807b830 secure :-1 s=THROTTLE_DONE pgs=2869 cs=0 l=1 rx=0x7f0a28142ff0 tx=0x7f0a200087f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:49.543 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:49.521+0000 7f0a1ffff700 1 -- 172.21.15.42:0/165036947 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a20010b00 con 0x7f0a28119640 2019-12-05T06:37:49.543 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:49.521+0000 7f0a2d16a700 1 -- 172.21.15.42:0/165036947 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a2807c050 con 0x7f0a28119640 2019-12-05T06:37:49.544 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:49.521+0000 7f0a1ffff700 1 -- 172.21.15.42:0/165036947 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0a2000f470 con 0x7f0a28119640 2019-12-05T06:37:49.544 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:49.521+0000 7f0a2d16a700 1 -- 172.21.15.42:0/165036947 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a28135840 con 0x7f0a28119640 2019-12-05T06:37:49.544 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:49.521+0000 7f0a2c969700 1 --2- 172.21.15.42:0/165036947 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0a28119640 0x7f0a2807b830 secure :-1 s=THROTTLE_DONE pgs=2869 cs=0 l=1 rx=0x7f0a28142ff0 tx=0x7f0a200087f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:49.544 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:49.521+0000 7f0a1ffff700 1 -- 172.21.15.42:0/165036947 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f0a2000f670 con 0x7f0a28119640 2019-12-05T06:37:49.545 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:49.521+0000 7f0a2c969700 1 --2- 172.21.15.42:0/165036947 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0a28119640 0x7f0a2807b830 secure :-1 s=THROTTLE_DONE pgs=2869 cs=0 l=1 rx=0x7f0a28142ff0 tx=0x7f0a200087f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:49.545 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:49.521+0000 7f0a1ffff700 1 --2- 172.21.15.42:0/165036947 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0a0c03ad50 0x7f0a0c03d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:37:49.545 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:49.521+0000 7f0a1ffff700 1 -- 172.21.15.42:0/165036947 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 11363+0+0 (secure 0 0 0) 0x7f0a2008f060 con 0x7f0a28119640 2019-12-05T06:37:49.545 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:49.521+0000 7f0a2759e700 1 --2- 172.21.15.42:0/165036947 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0a0c03ad50 0x7f0a0c03d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:37:49.545 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:49.521+0000 7f0a2759e700 1 --2- 172.21.15.42:0/165036947 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0a0c03ad50 0x7f0a0c03d260 crc :-1 s=READY pgs=1206 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:37:49.545 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:49.525+0000 7f0a2d16a700 1 -- 172.21.15.42:0/165036947 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a2805d110 con 0x7f0a28119640 2019-12-05T06:37:49.546 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:49.533+0000 7f0a2c969700 1 --2- 172.21.15.42:0/165036947 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0a28119640 0x7f0a2807b830 secure :-1 s=THROTTLE_DONE pgs=2869 cs=0 l=1 rx=0x7f0a28142ff0 tx=0x7f0a200087f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:49.546 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:49.533+0000 7f0a1ffff700 1 -- 172.21.15.42:0/165036947 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f0a200760e0 con 0x7f0a28119640 2019-12-05T06:37:49.702 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:49.697+0000 7f0a2d16a700 1 -- 172.21.15.42:0/165036947 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0a2805b8c0 con 0x7f0a28119640 2019-12-05T06:37:49.703 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:49.697+0000 7f0a2c969700 1 --2- 172.21.15.42:0/165036947 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0a28119640 0x7f0a2807b830 secure :-1 s=THROTTLE_DONE pgs=2869 cs=0 l=1 rx=0x7f0a28142ff0 tx=0x7f0a200087f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:49.703 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:49.697+0000 7f0a1ffff700 1 -- 172.21.15.42:0/165036947 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 343 v343) v1 ==== 95+0+2111 (secure 0 0 0) 0x7f0a2000f980 con 0x7f0a28119640 2019-12-05T06:37:49.703 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 343 2019-12-05T06:37:49.703 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:37:49.703 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":343,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":11887,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/958796360","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":958796360},{"type":"v1","addr":"172.21.15.42:6835","nonce":958796360}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":343}],"filesystems":[{"mdsmap":{"epoch":342,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:35:40.288210+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":111,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7910},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7910":{"gid":7910,"name":"b","rank":0,"incarnation":339,"state":"up:active","state_seq":33,"addr":"172.21.15.40:6833/4253974783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":4253974783},{"type":"v1","addr":"172.21.15.40:6833","nonce":4253974783}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:37:49.705 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:49.701+0000 7f0a2d16a700 1 -- 172.21.15.42:0/165036947 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0a0c03ad50 msgr2=0x7f0a0c03d260 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:37:49.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:49.701+0000 7f0a2d16a700 1 --2- 172.21.15.42:0/165036947 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0a0c03ad50 0x7f0a0c03d260 crc :-1 s=READY pgs=1206 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:37:49.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:49.701+0000 7f0a2d16a700 1 -- 172.21.15.42:0/165036947 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0a28119640 msgr2=0x7f0a2807b830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:37:49.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:49.701+0000 7f0a2d16a700 1 --2- 172.21.15.42:0/165036947 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0a28119640 0x7f0a2807b830 secure :-1 s=READY pgs=2869 cs=0 l=1 rx=0x7f0a28142ff0 tx=0x7f0a200087f0).stop 2019-12-05T06:37:49.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:49.701+0000 7f0a2d16a700 1 -- 172.21.15.42:0/165036947 shutdown_connections 2019-12-05T06:37:49.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:49.701+0000 7f0a2d16a700 1 --2- 172.21.15.42:0/165036947 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0a0c03ad50 0x7f0a0c03d260 unknown :-1 s=CLOSED pgs=1206 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:37:49.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:49.701+0000 7f0a2d16a700 1 --2- 172.21.15.42:0/165036947 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0a28119640 0x7f0a2807b830 unknown :-1 s=CLOSED pgs=2869 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:37:49.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:49.701+0000 7f0a2d16a700 1 --2- 172.21.15.42:0/165036947 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0a28143610 0x7f0a281190f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:37:49.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:49.701+0000 7f0a2d16a700 1 -- 172.21.15.42:0/165036947 shutdown_connections 2019-12-05T06:37:49.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:49.701+0000 7f0a2d16a700 1 -- 172.21.15.42:0/165036947 wait complete. 2019-12-05T06:37:49.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:49.701+0000 7f0a2d16a700 1 -- 172.21.15.42:0/165036947 >> 172.21.15.42:0/165036947 conn(0x7f0a280aabe0 msgr2=0x7f0a2811c470 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:37:49.714 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:37:51.716 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:37:51.826 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:51.809+0000 7f88db38c700 1 Processor -- start 2019-12-05T06:37:51.827 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:51.809+0000 7f88db38c700 1 -- start start 2019-12-05T06:37:51.827 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:51.809+0000 7f88db38c700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f88d40a7740 0x7f88d408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:37:51.827 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:51.809+0000 7f88db38c700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88d40a0710 con 0x7f88d407e720 2019-12-05T06:37:51.828 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:51.809+0000 7f88db38c700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88d40a9ba0 con 0x7f88d409fae0 2019-12-05T06:37:51.828 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:51.809+0000 7f88db38c700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88d40a0b00 con 0x7f88d40a7740 2019-12-05T06:37:51.828 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:51.809+0000 7f88da12a700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f88d409fae0 0x7f88d414efe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:59024/0 (socket says 172.21.15.42:59024) 2019-12-05T06:37:51.829 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:51.809+0000 7f88da12a700 1 -- 172.21.15.42:0/2799215998 learned_addr learned my addr 172.21.15.42:0/2799215998 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:37:51.829 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:51.809+0000 7f88d9929700 1 --2- 172.21.15.42:0/2799215998 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f88d40a7740 0x7f88d408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:37:51.829 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:51.809+0000 7f88d8927700 1 -- 172.21.15.42:0/2799215998 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1968697038 0 0) 0x7f88d40a9ba0 con 0x7f88d409fae0 2019-12-05T06:37:51.829 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:51.809+0000 7f88d8927700 1 -- 172.21.15.42:0/2799215998 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88c00030b0 con 0x7f88d409fae0 2019-12-05T06:37:51.829 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:51.809+0000 7f88d8927700 1 -- 172.21.15.42:0/2799215998 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1653465793 0 0) 0x7f88d40a0710 con 0x7f88d407e720 2019-12-05T06:37:51.829 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:51.809+0000 7f88d8927700 1 -- 172.21.15.42:0/2799215998 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88d40a9ba0 con 0x7f88d407e720 2019-12-05T06:37:51.830 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:51.809+0000 7f88d9929700 1 -- 172.21.15.42:0/2799215998 >> v1:172.21.15.42:6789/0 conn(0x7f88d409fae0 legacy=0x7f88d414efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:37:51.830 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:51.809+0000 7f88d9929700 1 -- 172.21.15.42:0/2799215998 >> v1:172.21.15.40:6789/0 conn(0x7f88d407e720 legacy=0x7f88d409f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:37:51.830 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:51.809+0000 7f88d9929700 1 -- 172.21.15.42:0/2799215998 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88d407eb50 con 0x7f88d40a7740 2019-12-05T06:37:51.830 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:51.809+0000 7f88d9929700 1 --2- 172.21.15.42:0/2799215998 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f88d40a7740 0x7f88d408df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f88d0002f30 tx=0x7f88d000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:51.830 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:51.813+0000 7f88d9929700 1 --2- 172.21.15.42:0/2799215998 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f88d40a7740 0x7f88d408df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f88d0002f30 tx=0x7f88d000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:51.831 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:51.813+0000 7f88d9929700 1 --2- 172.21.15.42:0/2799215998 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f88d40a7740 0x7f88d408df80 secure :-1 s=READY pgs=644 cs=0 l=1 rx=0x7f88d0002f30 tx=0x7f88d000fbf0).ready entity=mon.2 client_cookie=5385b578108438f5 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:37:51.832 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:51.813+0000 7f88d9929700 1 --2- 172.21.15.42:0/2799215998 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f88d40a7740 0x7f88d408df80 secure :-1 s=THROTTLE_DONE pgs=644 cs=0 l=1 rx=0x7f88d0002f30 tx=0x7f88d000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:51.832 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:51.813+0000 7f88d8927700 1 -- 172.21.15.42:0/2799215998 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f88d00112b0 con 0x7f88d40a7740 2019-12-05T06:37:51.832 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:51.813+0000 7f88d9929700 1 --2- 172.21.15.42:0/2799215998 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f88d40a7740 0x7f88d408df80 secure :-1 s=THROTTLE_DONE pgs=644 cs=0 l=1 rx=0x7f88d0002f30 tx=0x7f88d000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:51.832 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:51.813+0000 7f88d9929700 1 --2- 172.21.15.42:0/2799215998 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f88d40a7740 0x7f88d408df80 secure :-1 s=THROTTLE_DONE pgs=644 cs=0 l=1 rx=0x7f88d0002f30 tx=0x7f88d000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:51.832 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:51.813+0000 7f88d8927700 1 -- 172.21.15.42:0/2799215998 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88d0008bc0 con 0x7f88d40a7740 2019-12-05T06:37:51.833 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:51.813+0000 7f88d8927700 1 -- 172.21.15.42:0/2799215998 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f88d0011890 con 0x7f88d40a7740 2019-12-05T06:37:51.833 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:51.813+0000 7f88db38c700 1 -- 172.21.15.42:0/2799215998 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f88d40a7740 msgr2=0x7f88d408df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:37:51.833 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:51.813+0000 7f88db38c700 1 --2- 172.21.15.42:0/2799215998 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f88d40a7740 0x7f88d408df80 secure :-1 s=READY pgs=644 cs=0 l=1 rx=0x7f88d0002f30 tx=0x7f88d000fbf0).stop 2019-12-05T06:37:51.833 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:51.813+0000 7f88db38c700 1 -- 172.21.15.42:0/2799215998 shutdown_connections 2019-12-05T06:37:51.833 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:51.813+0000 7f88db38c700 1 --2- 172.21.15.42:0/2799215998 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f88d40a7740 0x7f88d408df80 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:37:51.833 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:51.813+0000 7f88db38c700 1 -- 172.21.15.42:0/2799215998 shutdown_connections 2019-12-05T06:37:51.834 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:51.813+0000 7f88db38c700 1 -- 172.21.15.42:0/2799215998 wait complete. 2019-12-05T06:37:51.834 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:51.813+0000 7f88db38c700 1 -- 172.21.15.42:0/2799215998 >> 172.21.15.42:0/2799215998 conn(0x7f88d40aabe0 msgr2=0x7f88d40a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:37:51.834 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:51.813+0000 7f88db38c700 1 Processor -- start 2019-12-05T06:37:51.834 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:51.813+0000 7f88db38c700 1 -- start start 2019-12-05T06:37:51.834 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:51.817+0000 7f88db38c700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f88d41222a0 0x7f88d40a58f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:37:51.834 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:51.817+0000 7f88db38c700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88d41266a0 con 0x7f88d4126270 2019-12-05T06:37:51.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:51.817+0000 7f88db38c700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88d41226d0 con 0x7f88d4116850 2019-12-05T06:37:51.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:51.817+0000 7f88db38c700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88d4125a90 con 0x7f88d41222a0 2019-12-05T06:37:51.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:51.817+0000 7f88d9929700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f88d4116850 0x7f88d40a3700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:59026/0 (socket says 172.21.15.42:59026) 2019-12-05T06:37:51.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:51.817+0000 7f88d9929700 1 -- 172.21.15.42:0/1660099857 learned_addr learned my addr 172.21.15.42:0/1660099857 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:37:51.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:51.817+0000 7f88da12a700 1 --2- 172.21.15.42:0/1660099857 >> v2:172.21.15.40:3300/0 conn(0x7f88d41222a0 0x7f88d40a58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:37:51.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:51.817+0000 7f88c67fc700 1 -- 172.21.15.42:0/1660099857 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4055578942 0 0) 0x7f88d41226d0 con 0x7f88d4116850 2019-12-05T06:37:51.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:51.817+0000 7f88c67fc700 1 -- 172.21.15.42:0/1660099857 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88bc0030b0 con 0x7f88d4116850 2019-12-05T06:37:51.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:51.817+0000 7f88c67fc700 1 -- 172.21.15.42:0/1660099857 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 810144720 0 0) 0x7f88d41266a0 con 0x7f88d4126270 2019-12-05T06:37:51.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:51.817+0000 7f88c67fc700 1 -- 172.21.15.42:0/1660099857 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88d41226d0 con 0x7f88d4126270 2019-12-05T06:37:51.837 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:51.817+0000 7f88c67fc700 1 -- 172.21.15.42:0/1660099857 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 1177610487 0 0) 0x7f88bc0030b0 con 0x7f88d4116850 2019-12-05T06:37:51.837 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:51.817+0000 7f88c67fc700 1 -- 172.21.15.42:0/1660099857 --> v1:172.21.15.42:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f88d41266a0 con 0x7f88d4116850 2019-12-05T06:37:51.837 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:51.817+0000 7f88c67fc700 1 -- 172.21.15.42:0/1660099857 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7f88d00112b0 con 0x7f88d4116850 2019-12-05T06:37:51.838 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:51.817+0000 7f88c67fc700 1 -- 172.21.15.42:0/1660099857 >> v2:172.21.15.40:3300/0 conn(0x7f88d41222a0 msgr2=0x7f88d40a58f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:37:51.838 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:51.817+0000 7f88c67fc700 1 --2- 172.21.15.42:0/1660099857 >> v2:172.21.15.40:3300/0 conn(0x7f88d41222a0 0x7f88d40a58f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:37:51.838 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:51.817+0000 7f88c67fc700 1 -- 172.21.15.42:0/1660099857 >> v1:172.21.15.42:6789/0 conn(0x7f88d4116850 legacy=0x7f88d40a3700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:37:51.838 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:51.817+0000 7f88c67fc700 1 -- 172.21.15.42:0/1660099857 >> v1:172.21.15.40:6789/0 conn(0x7f88d4126270 legacy=0x7f88d412aca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:37:51.839 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:51.817+0000 7f88c67fc700 1 --2- 172.21.15.42:0/1660099857 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f88bc003b00 0x7f88bc004420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:37:51.839 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:51.817+0000 7f88c67fc700 1 --2- 172.21.15.42:0/1660099857 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f88bc004970 0x7f88bc004da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:37:51.839 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:51.817+0000 7f88c67fc700 1 --2- 172.21.15.42:0/1660099857 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f88bc009660 0x7f88bc00baa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:37:51.839 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:51.817+0000 7f88c67fc700 1 -- 172.21.15.42:0/1660099857 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88d4125a90 con 0x7f88bc009660 2019-12-05T06:37:51.839 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:51.817+0000 7f88c67fc700 1 -- 172.21.15.42:0/1660099857 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88bc00bff0 con 0x7f88bc004970 2019-12-05T06:37:51.839 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:51.817+0000 7f88c67fc700 1 -- 172.21.15.42:0/1660099857 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88bc00c200 con 0x7f88bc003b00 2019-12-05T06:37:51.840 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:51.817+0000 7f88da12a700 1 --2- 172.21.15.42:0/1660099857 >> v2:172.21.15.40:3300/0 conn(0x7f88d41222a0 0x7f88d40a58f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:37:51.840 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:51.817+0000 7f88d9929700 1 --2- 172.21.15.42:0/1660099857 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f88bc003b00 0x7f88bc004420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:37:51.840 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:51.817+0000 7f88d9128700 1 --2- 172.21.15.42:0/1660099857 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f88bc004970 0x7f88bc004da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:37:51.841 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:51.817+0000 7f88da12a700 1 --2- 172.21.15.42:0/1660099857 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f88bc009660 0x7f88bc00baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:37:51.841 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:51.817+0000 7f88d9128700 1 -- 172.21.15.42:0/1660099857 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f88bc003b00 msgr2=0x7f88bc004420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:37:51.843 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:51.817+0000 7f88d9128700 1 --2- 172.21.15.42:0/1660099857 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f88bc003b00 0x7f88bc004420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:37:51.843 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:51.817+0000 7f88d9128700 1 -- 172.21.15.42:0/1660099857 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f88bc009660 msgr2=0x7f88bc00baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:37:51.843 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:51.817+0000 7f88d9128700 1 --2- 172.21.15.42:0/1660099857 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f88bc009660 0x7f88bc00baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:37:51.843 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:51.817+0000 7f88d9128700 1 -- 172.21.15.42:0/1660099857 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88d414acd0 con 0x7f88bc004970 2019-12-05T06:37:51.843 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:51.817+0000 7f88da12a700 1 --2- 172.21.15.42:0/1660099857 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f88bc009660 0x7f88bc00baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:37:51.844 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:51.817+0000 7f88db38c700 1 -- 172.21.15.42:0/1660099857 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88d40a3f00 con 0x7f88bc004970 2019-12-05T06:37:51.844 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:51.817+0000 7f88da12a700 1 -- 172.21.15.42:0/1660099857 reap_dead start 2019-12-05T06:37:51.844 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:51.817+0000 7f88d9128700 1 --2- 172.21.15.42:0/1660099857 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f88bc004970 0x7f88bc004da0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f88c80011b0 tx=0x7f88c8010b80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:51.844 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:51.817+0000 7f88d9929700 1 --2- 172.21.15.42:0/1660099857 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f88bc003b00 0x7f88bc004420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:37:51.845 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:51.817+0000 7f88db38c700 1 -- 172.21.15.42:0/1660099857 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88d41222a0 con 0x7f88bc004970 2019-12-05T06:37:51.845 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:51.817+0000 7f88d9128700 1 --2- 172.21.15.42:0/1660099857 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f88bc004970 0x7f88bc004da0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f88c80011b0 tx=0x7f88c8010b80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:51.845 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:51.817+0000 7f88d9128700 1 --2- 172.21.15.42:0/1660099857 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f88bc004970 0x7f88bc004da0 secure :-1 s=READY pgs=2872 cs=0 l=1 rx=0x7f88c80011b0 tx=0x7f88c8010b80).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:37:51.846 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:51.817+0000 7f88d9128700 1 --2- 172.21.15.42:0/1660099857 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f88bc004970 0x7f88bc004da0 secure :-1 s=THROTTLE_DONE pgs=2872 cs=0 l=1 rx=0x7f88c80011b0 tx=0x7f88c8010b80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:51.846 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:51.817+0000 7f88d9128700 1 --2- 172.21.15.42:0/1660099857 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f88bc004970 0x7f88bc004da0 secure :-1 s=THROTTLE_DONE pgs=2872 cs=0 l=1 rx=0x7f88c80011b0 tx=0x7f88c8010b80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:51.846 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:51.817+0000 7f88c67fc700 1 -- 172.21.15.42:0/1660099857 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f88c8007810 con 0x7f88bc004970 2019-12-05T06:37:51.846 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:51.817+0000 7f88d9128700 1 --2- 172.21.15.42:0/1660099857 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f88bc004970 0x7f88bc004da0 secure :-1 s=THROTTLE_DONE pgs=2872 cs=0 l=1 rx=0x7f88c80011b0 tx=0x7f88c8010b80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:51.847 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:51.817+0000 7f88c67fc700 1 -- 172.21.15.42:0/1660099857 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88c8007430 con 0x7f88bc004970 2019-12-05T06:37:51.847 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:51.817+0000 7f88c67fc700 1 -- 172.21.15.42:0/1660099857 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f88c8007df0 con 0x7f88bc004970 2019-12-05T06:37:51.847 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:51.821+0000 7f88d9128700 1 --2- 172.21.15.42:0/1660099857 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f88bc004970 0x7f88bc004da0 secure :-1 s=THROTTLE_DONE pgs=2872 cs=0 l=1 rx=0x7f88c80011b0 tx=0x7f88c8010b80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:51.848 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:51.821+0000 7f88c67fc700 1 -- 172.21.15.42:0/1660099857 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f88c8006360 con 0x7f88bc004970 2019-12-05T06:37:51.848 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:51.821+0000 7f88d9128700 1 --2- 172.21.15.42:0/1660099857 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f88bc004970 0x7f88bc004da0 secure :-1 s=THROTTLE_DONE pgs=2872 cs=0 l=1 rx=0x7f88c80011b0 tx=0x7f88c8010b80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:51.848 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:51.821+0000 7f88c67fc700 1 --2- 172.21.15.42:0/1660099857 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f88bc03ece0 0x7f88bc0411f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:37:51.849 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:51.821+0000 7f88c67fc700 1 -- 172.21.15.42:0/1660099857 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 11363+0+0 (secure 0 0 0) 0x7f88c804b210 con 0x7f88bc004970 2019-12-05T06:37:51.849 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:51.821+0000 7f88d9929700 1 --2- 172.21.15.42:0/1660099857 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f88bc03ece0 0x7f88bc0411f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:37:51.851 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:51.821+0000 7f88d9929700 1 --2- 172.21.15.42:0/1660099857 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f88bc03ece0 0x7f88bc0411f0 crc :-1 s=READY pgs=1207 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:37:51.853 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:51.825+0000 7f88db38c700 1 -- 172.21.15.42:0/1660099857 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88b0003200 con 0x7f88bc004970 2019-12-05T06:37:51.853 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:51.833+0000 7f88d9128700 1 --2- 172.21.15.42:0/1660099857 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f88bc004970 0x7f88bc004da0 secure :-1 s=THROTTLE_DONE pgs=2872 cs=0 l=1 rx=0x7f88c80011b0 tx=0x7f88c8010b80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:51.853 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:51.833+0000 7f88c67fc700 1 -- 172.21.15.42:0/1660099857 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f88c8032290 con 0x7f88bc004970 2019-12-05T06:37:52.010 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:52.005+0000 7f88db38c700 1 -- 172.21.15.42:0/1660099857 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f88b0003630 con 0x7f88bc004970 2019-12-05T06:37:52.012 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:52.005+0000 7f88d9128700 1 --2- 172.21.15.42:0/1660099857 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f88bc004970 0x7f88bc004da0 secure :-1 s=THROTTLE_DONE pgs=2872 cs=0 l=1 rx=0x7f88c80011b0 tx=0x7f88c8010b80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:52.012 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:52.005+0000 7f88c67fc700 1 -- 172.21.15.42:0/1660099857 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 343 v343) v1 ==== 95+0+2111 (secure 0 0 0) 0x7f88c80324f0 con 0x7f88bc004970 2019-12-05T06:37:52.012 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 343 2019-12-05T06:37:52.012 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:37:52.013 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":343,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":11887,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/958796360","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":958796360},{"type":"v1","addr":"172.21.15.42:6835","nonce":958796360}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":343}],"filesystems":[{"mdsmap":{"epoch":342,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:35:40.288210+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":111,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7910},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7910":{"gid":7910,"name":"b","rank":0,"incarnation":339,"state":"up:active","state_seq":33,"addr":"172.21.15.40:6833/4253974783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":4253974783},{"type":"v1","addr":"172.21.15.40:6833","nonce":4253974783}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:37:52.014 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:52.009+0000 7f88db38c700 1 -- 172.21.15.42:0/1660099857 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f88bc03ece0 msgr2=0x7f88bc0411f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:37:52.014 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:52.009+0000 7f88db38c700 1 --2- 172.21.15.42:0/1660099857 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f88bc03ece0 0x7f88bc0411f0 crc :-1 s=READY pgs=1207 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:37:52.014 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:52.009+0000 7f88db38c700 1 -- 172.21.15.42:0/1660099857 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f88bc004970 msgr2=0x7f88bc004da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:37:52.015 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:52.009+0000 7f88db38c700 1 --2- 172.21.15.42:0/1660099857 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f88bc004970 0x7f88bc004da0 secure :-1 s=READY pgs=2872 cs=0 l=1 rx=0x7f88c80011b0 tx=0x7f88c8010b80).stop 2019-12-05T06:37:52.015 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:52.009+0000 7f88db38c700 1 -- 172.21.15.42:0/1660099857 shutdown_connections 2019-12-05T06:37:52.015 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:52.009+0000 7f88db38c700 1 --2- 172.21.15.42:0/1660099857 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f88bc03ece0 0x7f88bc0411f0 unknown :-1 s=CLOSED pgs=1207 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:37:52.015 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:52.009+0000 7f88db38c700 1 --2- 172.21.15.42:0/1660099857 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f88bc004970 0x7f88bc004da0 unknown :-1 s=CLOSED pgs=2872 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:37:52.015 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:52.009+0000 7f88db38c700 1 -- 172.21.15.42:0/1660099857 shutdown_connections 2019-12-05T06:37:52.016 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:52.009+0000 7f88db38c700 1 -- 172.21.15.42:0/1660099857 wait complete. 2019-12-05T06:37:52.016 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:52.009+0000 7f88db38c700 1 -- 172.21.15.42:0/1660099857 >> 172.21.15.42:0/1660099857 conn(0x7f88d408fda0 msgr2=0x7f88d409f460 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:37:52.025 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:37:54.027 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:37:54.133 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:54.121+0000 7f2964d45700 1 Processor -- start 2019-12-05T06:37:54.133 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:54.121+0000 7f2964d45700 1 -- start start 2019-12-05T06:37:54.133 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:54.121+0000 7f2964d45700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f29600a7740 0x7f296008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:37:54.133 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:54.121+0000 7f2964d45700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f296009fae0 0x7f296007eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:37:54.133 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:54.121+0000 7f2964d45700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29600a0710 con 0x7f296007e720 2019-12-05T06:37:54.134 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:54.121+0000 7f2964d45700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29600a0b00 con 0x7f296009fae0 2019-12-05T06:37:54.134 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:54.121+0000 7f2964d45700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f296008e5b0 con 0x7f29600a7740 2019-12-05T06:37:54.134 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:54.121+0000 7f295f59e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f296009fae0 0x7f296007eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:37:54.134 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:54.121+0000 7f295f59e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f296009fae0 0x7f296007eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:59004/0 (socket says 172.21.15.42:59004) 2019-12-05T06:37:54.134 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:54.121+0000 7f295e59c700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f296007e720 0x7f296009f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:40248/0 (socket says 172.21.15.42:40248) 2019-12-05T06:37:54.134 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:54.121+0000 7f295f59e700 1 -- 172.21.15.42:0/598202121 learned_addr learned my addr 172.21.15.42:0/598202121 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:37:54.134 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:54.121+0000 7f295ed9d700 1 --2- 172.21.15.42:0/598202121 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f29600a7740 0x7f296008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:37:54.135 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:54.121+0000 7f295dd9b700 1 -- 172.21.15.42:0/598202121 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3194195286 0 0) 0x7f29600a0710 con 0x7f296007e720 2019-12-05T06:37:54.135 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:54.121+0000 7f295dd9b700 1 -- 172.21.15.42:0/598202121 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f294c0030b0 con 0x7f296007e720 2019-12-05T06:37:54.135 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:54.121+0000 7f295f59e700 1 -- 172.21.15.42:0/598202121 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f29600a7740 msgr2=0x7f296008df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:37:54.135 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:54.121+0000 7f295f59e700 1 --2- 172.21.15.42:0/598202121 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f29600a7740 0x7f296008df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:37:54.135 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:54.121+0000 7f295f59e700 1 -- 172.21.15.42:0/598202121 >> v1:172.21.15.40:6789/0 conn(0x7f296007e720 legacy=0x7f296009f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:37:54.135 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:54.121+0000 7f295f59e700 1 -- 172.21.15.42:0/598202121 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f296007f0a0 con 0x7f296009fae0 2019-12-05T06:37:54.135 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:54.121+0000 7f295ed9d700 1 --2- 172.21.15.42:0/598202121 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f29600a7740 0x7f296008df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:37:54.136 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:54.121+0000 7f295f59e700 1 --2- 172.21.15.42:0/598202121 >> v2:172.21.15.42:3300/0 conn(0x7f296009fae0 0x7f296007eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f2958008790 tx=0x7f295800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:54.136 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:54.121+0000 7f295f59e700 1 --2- 172.21.15.42:0/598202121 >> v2:172.21.15.42:3300/0 conn(0x7f296009fae0 0x7f296007eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f2958008790 tx=0x7f295800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:54.136 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:54.121+0000 7f295f59e700 1 --2- 172.21.15.42:0/598202121 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f296009fae0 0x7f296007eb50 secure :-1 s=READY pgs=2873 cs=0 l=1 rx=0x7f2958008790 tx=0x7f295800fd30).ready entity=mon.1 client_cookie=e73a40f52617e523 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:37:54.136 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:54.121+0000 7f295f59e700 1 --2- 172.21.15.42:0/598202121 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f296009fae0 0x7f296007eb50 secure :-1 s=THROTTLE_DONE pgs=2873 cs=0 l=1 rx=0x7f2958008790 tx=0x7f295800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:54.136 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:54.121+0000 7f295dd9b700 1 -- 172.21.15.42:0/598202121 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2958008c90 con 0x7f296009fae0 2019-12-05T06:37:54.136 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:54.121+0000 7f295f59e700 1 --2- 172.21.15.42:0/598202121 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f296009fae0 0x7f296007eb50 secure :-1 s=THROTTLE_DONE pgs=2873 cs=0 l=1 rx=0x7f2958008790 tx=0x7f295800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:54.137 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:54.121+0000 7f295dd9b700 1 -- 172.21.15.42:0/598202121 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29580088b0 con 0x7f296009fae0 2019-12-05T06:37:54.137 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:54.121+0000 7f2964d45700 1 -- 172.21.15.42:0/598202121 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f296009fae0 msgr2=0x7f296007eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:37:54.137 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:54.121+0000 7f295f59e700 1 --2- 172.21.15.42:0/598202121 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f296009fae0 0x7f296007eb50 secure :-1 s=THROTTLE_DONE pgs=2873 cs=0 l=1 rx=0x7f2958008790 tx=0x7f295800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:54.137 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:54.121+0000 7f295dd9b700 1 -- 172.21.15.42:0/598202121 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2958011b10 con 0x7f296009fae0 2019-12-05T06:37:54.137 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:54.121+0000 7f2964d45700 1 --2- 172.21.15.42:0/598202121 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f296009fae0 0x7f296007eb50 secure :-1 s=READY pgs=2873 cs=0 l=1 rx=0x7f2958008790 tx=0x7f295800fd30).stop 2019-12-05T06:37:54.137 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:54.125+0000 7f2964d45700 1 -- 172.21.15.42:0/598202121 shutdown_connections 2019-12-05T06:37:54.137 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:54.125+0000 7f2964d45700 1 --2- 172.21.15.42:0/598202121 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f296009fae0 0x7f296007eb50 unknown :-1 s=CLOSED pgs=2873 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:37:54.137 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:54.125+0000 7f2964d45700 1 --2- 172.21.15.42:0/598202121 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f29600a7740 0x7f296008df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:37:54.138 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:54.125+0000 7f2964d45700 1 -- 172.21.15.42:0/598202121 shutdown_connections 2019-12-05T06:37:54.138 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:54.125+0000 7f2964d45700 1 -- 172.21.15.42:0/598202121 wait complete. 2019-12-05T06:37:54.138 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:54.125+0000 7f2964d45700 1 -- 172.21.15.42:0/598202121 >> 172.21.15.42:0/598202121 conn(0x7f29600aabe0 msgr2=0x7f29600a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:37:54.138 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:54.125+0000 7f2964d45700 1 Processor -- start 2019-12-05T06:37:54.138 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:54.125+0000 7f2964d45700 1 -- start start 2019-12-05T06:37:54.138 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:54.129+0000 7f2964d45700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f29600a2500 0x7f2960130120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:37:54.138 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:54.129+0000 7f2964d45700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f29600a33a0 0x7f29601429d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:37:54.139 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:54.129+0000 7f2964d45700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29600a37d0 con 0x7f2960142f20 2019-12-05T06:37:54.139 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:54.129+0000 7f2964d45700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29600a2930 con 0x7f29600a2500 2019-12-05T06:37:54.139 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:54.129+0000 7f2964d45700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2960130670 con 0x7f29600a33a0 2019-12-05T06:37:54.139 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:54.129+0000 7f295ed9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f29600a2500 0x7f2960130120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:37:54.139 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:54.129+0000 7f295ed9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f29600a2500 0x7f2960130120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:59008/0 (socket says 172.21.15.42:59008) 2019-12-05T06:37:54.139 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:54.129+0000 7f295ed9d700 1 -- 172.21.15.42:0/3479679288 learned_addr learned my addr 172.21.15.42:0/3479679288 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:37:54.140 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:54.129+0000 7f295e59c700 1 --2- 172.21.15.42:0/3479679288 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f29600a33a0 0x7f29601429d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:37:54.142 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:54.129+0000 7f295ed9d700 1 -- 172.21.15.42:0/3479679288 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f29600a33a0 msgr2=0x7f29601429d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:37:54.142 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:54.129+0000 7f295ed9d700 1 --2- 172.21.15.42:0/3479679288 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f29600a33a0 0x7f29601429d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:37:54.142 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:54.129+0000 7f295ed9d700 1 -- 172.21.15.42:0/3479679288 >> v1:172.21.15.40:6789/0 conn(0x7f2960142f20 legacy=0x7f29601336e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:37:54.143 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:54.129+0000 7f29577fe700 1 -- 172.21.15.42:0/3479679288 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2770546937 0 0) 0x7f29600a37d0 con 0x7f2960142f20 2019-12-05T06:37:54.143 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:54.129+0000 7f295ed9d700 1 -- 172.21.15.42:0/3479679288 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2960133ee0 con 0x7f29600a2500 2019-12-05T06:37:54.143 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:54.129+0000 7f295ed9d700 1 --2- 172.21.15.42:0/3479679288 >> v2:172.21.15.42:3300/0 conn(0x7f29600a2500 0x7f2960130120 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f2950003270 tx=0x7f29500110b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:54.143 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:54.129+0000 7f295ed9d700 1 --2- 172.21.15.42:0/3479679288 >> v2:172.21.15.42:3300/0 conn(0x7f29600a2500 0x7f2960130120 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f2950003270 tx=0x7f29500110b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:54.143 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:54.129+0000 7f295ed9d700 1 --2- 172.21.15.42:0/3479679288 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f29600a2500 0x7f2960130120 secure :-1 s=READY pgs=2874 cs=0 l=1 rx=0x7f2950003270 tx=0x7f29500110b0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:37:54.143 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:54.129+0000 7f295ed9d700 1 --2- 172.21.15.42:0/3479679288 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f29600a2500 0x7f2960130120 secure :-1 s=THROTTLE_DONE pgs=2874 cs=0 l=1 rx=0x7f2950003270 tx=0x7f29500110b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:54.143 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:54.129+0000 7f29577fe700 1 -- 172.21.15.42:0/3479679288 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2950012a30 con 0x7f29600a2500 2019-12-05T06:37:54.144 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:54.129+0000 7f295ed9d700 1 --2- 172.21.15.42:0/3479679288 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f29600a2500 0x7f2960130120 secure :-1 s=THROTTLE_DONE pgs=2874 cs=0 l=1 rx=0x7f2950003270 tx=0x7f29500110b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:54.144 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:54.129+0000 7f295ed9d700 1 --2- 172.21.15.42:0/3479679288 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f29600a2500 0x7f2960130120 secure :-1 s=THROTTLE_DONE pgs=2874 cs=0 l=1 rx=0x7f2950003270 tx=0x7f29500110b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:54.144 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:54.129+0000 7f29577fe700 1 -- 172.21.15.42:0/3479679288 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f295000e160 con 0x7f29600a2500 2019-12-05T06:37:54.144 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:54.129+0000 7f2964d45700 1 -- 172.21.15.42:0/3479679288 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29601341b0 con 0x7f29600a2500 2019-12-05T06:37:54.144 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:54.129+0000 7f29577fe700 1 -- 172.21.15.42:0/3479679288 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2950005ab0 con 0x7f29600a2500 2019-12-05T06:37:54.144 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:54.129+0000 7f2964d45700 1 -- 172.21.15.42:0/3479679288 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f296011de40 con 0x7f29600a2500 2019-12-05T06:37:54.144 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:54.129+0000 7f295ed9d700 1 --2- 172.21.15.42:0/3479679288 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f29600a2500 0x7f2960130120 secure :-1 s=THROTTLE_DONE pgs=2874 cs=0 l=1 rx=0x7f2950003270 tx=0x7f29500110b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:54.145 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:54.129+0000 7f29577fe700 1 -- 172.21.15.42:0/3479679288 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f2950007180 con 0x7f29600a2500 2019-12-05T06:37:54.145 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:54.129+0000 7f295ed9d700 1 --2- 172.21.15.42:0/3479679288 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f29600a2500 0x7f2960130120 secure :-1 s=THROTTLE_DONE pgs=2874 cs=0 l=1 rx=0x7f2950003270 tx=0x7f29500110b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:54.145 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:54.133+0000 7f29577fe700 1 --2- 172.21.15.42:0/3479679288 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f294403abf0 0x7f294403d100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:37:54.145 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:54.133+0000 7f29577fe700 1 -- 172.21.15.42:0/3479679288 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 11363+0+0 (secure 0 0 0) 0x7f295004b250 con 0x7f29600a2500 2019-12-05T06:37:54.145 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:54.133+0000 7f295e59c700 1 --2- 172.21.15.42:0/3479679288 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f294403abf0 0x7f294403d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:37:54.146 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:54.133+0000 7f295e59c700 1 --2- 172.21.15.42:0/3479679288 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f294403abf0 0x7f294403d100 crc :-1 s=READY pgs=1208 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:37:54.146 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:54.137+0000 7f2964d45700 1 -- 172.21.15.42:0/3479679288 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f296005d110 con 0x7f29600a2500 2019-12-05T06:37:54.149 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:54.145+0000 7f295ed9d700 1 --2- 172.21.15.42:0/3479679288 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f29600a2500 0x7f2960130120 secure :-1 s=THROTTLE_DONE pgs=2874 cs=0 l=1 rx=0x7f2950003270 tx=0x7f29500110b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:54.149 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:54.145+0000 7f29577fe700 1 -- 172.21.15.42:0/3479679288 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f2950005d90 con 0x7f29600a2500 2019-12-05T06:37:54.321 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:54.317+0000 7f2964d45700 1 -- 172.21.15.42:0/3479679288 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f296005b8c0 con 0x7f29600a2500 2019-12-05T06:37:54.322 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:54.317+0000 7f295ed9d700 1 --2- 172.21.15.42:0/3479679288 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f29600a2500 0x7f2960130120 secure :-1 s=THROTTLE_DONE pgs=2874 cs=0 l=1 rx=0x7f2950003270 tx=0x7f29500110b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:54.322 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:54.317+0000 7f29577fe700 1 -- 172.21.15.42:0/3479679288 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 343 v343) v1 ==== 95+0+2111 (secure 0 0 0) 0x7f29500322b0 con 0x7f29600a2500 2019-12-05T06:37:54.322 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 343 2019-12-05T06:37:54.322 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:37:54.323 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":343,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":11887,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/958796360","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":958796360},{"type":"v1","addr":"172.21.15.42:6835","nonce":958796360}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":343}],"filesystems":[{"mdsmap":{"epoch":342,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:35:40.288210+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":111,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7910},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7910":{"gid":7910,"name":"b","rank":0,"incarnation":339,"state":"up:active","state_seq":33,"addr":"172.21.15.40:6833/4253974783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":4253974783},{"type":"v1","addr":"172.21.15.40:6833","nonce":4253974783}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:37:54.324 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:54.317+0000 7f2964d45700 1 -- 172.21.15.42:0/3479679288 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f294403abf0 msgr2=0x7f294403d100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:37:54.325 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:54.317+0000 7f2964d45700 1 --2- 172.21.15.42:0/3479679288 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f294403abf0 0x7f294403d100 crc :-1 s=READY pgs=1208 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:37:54.325 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:54.321+0000 7f2964d45700 1 -- 172.21.15.42:0/3479679288 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f29600a2500 msgr2=0x7f2960130120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:37:54.325 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:54.321+0000 7f2964d45700 1 --2- 172.21.15.42:0/3479679288 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f29600a2500 0x7f2960130120 secure :-1 s=READY pgs=2874 cs=0 l=1 rx=0x7f2950003270 tx=0x7f29500110b0).stop 2019-12-05T06:37:54.325 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:54.321+0000 7f2964d45700 1 -- 172.21.15.42:0/3479679288 shutdown_connections 2019-12-05T06:37:54.325 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:54.321+0000 7f2964d45700 1 --2- 172.21.15.42:0/3479679288 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f294403abf0 0x7f294403d100 unknown :-1 s=CLOSED pgs=1208 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:37:54.326 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:54.321+0000 7f2964d45700 1 --2- 172.21.15.42:0/3479679288 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f29600a2500 0x7f2960130120 unknown :-1 s=CLOSED pgs=2874 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:37:54.326 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:54.321+0000 7f2964d45700 1 --2- 172.21.15.42:0/3479679288 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f29600a33a0 0x7f29601429d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:37:54.326 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:54.321+0000 7f2964d45700 1 -- 172.21.15.42:0/3479679288 shutdown_connections 2019-12-05T06:37:54.326 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:54.321+0000 7f2964d45700 1 -- 172.21.15.42:0/3479679288 wait complete. 2019-12-05T06:37:54.326 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:54.321+0000 7f2964d45700 1 -- 172.21.15.42:0/3479679288 >> 172.21.15.42:0/3479679288 conn(0x7f29600aabe0 msgr2=0x7f296007e4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:37:54.334 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:37:56.336 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:37:56.428 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.417+0000 7f2b1b190700 1 Processor -- start 2019-12-05T06:37:56.428 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.417+0000 7f2b1b190700 1 -- start start 2019-12-05T06:37:56.428 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.417+0000 7f2b1b190700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2b1408df80 0x7f2b1409f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:37:56.428 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.417+0000 7f2b1b190700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b1409fc60 con 0x7f2b140a7740 2019-12-05T06:37:56.429 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.417+0000 7f2b1b190700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b140a0710 con 0x7f2b1409f830 2019-12-05T06:37:56.429 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.421+0000 7f2b1b190700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b140a0b00 con 0x7f2b1408df80 2019-12-05T06:37:56.429 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.421+0000 7f2b19f2e700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f2b1409f830 0x7f2b1414efe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:59074/0 (socket says 172.21.15.42:59074) 2019-12-05T06:37:56.429 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.421+0000 7f2b19f2e700 1 -- 172.21.15.42:0/1169211566 learned_addr learned my addr 172.21.15.42:0/1169211566 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:37:56.429 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.421+0000 7f2b18f2c700 1 --2- 172.21.15.42:0/1169211566 >> v2:172.21.15.40:3300/0 conn(0x7f2b1408df80 0x7f2b1409f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:37:56.429 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.421+0000 7f2b0bfff700 1 -- 172.21.15.42:0/1169211566 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1888035363 0 0) 0x7f2b140a0710 con 0x7f2b1409f830 2019-12-05T06:37:56.429 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.421+0000 7f2b0bfff700 1 -- 172.21.15.42:0/1169211566 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b000030b0 con 0x7f2b1409f830 2019-12-05T06:37:56.430 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.421+0000 7f2b18f2c700 1 -- 172.21.15.42:0/1169211566 >> v1:172.21.15.42:6789/0 conn(0x7f2b1409f830 legacy=0x7f2b1414efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:37:56.430 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.421+0000 7f2b18f2c700 1 -- 172.21.15.42:0/1169211566 >> v1:172.21.15.40:6789/0 conn(0x7f2b140a7740 legacy=0x7f2b1407e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:37:56.430 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.421+0000 7f2b18f2c700 1 -- 172.21.15.42:0/1169211566 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b1407ef20 con 0x7f2b1408df80 2019-12-05T06:37:56.430 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.421+0000 7f2b18f2c700 1 --2- 172.21.15.42:0/1169211566 >> v2:172.21.15.40:3300/0 conn(0x7f2b1408df80 0x7f2b1409f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f2b04008790 tx=0x7f2b0400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:56.430 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.421+0000 7f2b18f2c700 1 --2- 172.21.15.42:0/1169211566 >> v2:172.21.15.40:3300/0 conn(0x7f2b1408df80 0x7f2b1409f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f2b04008790 tx=0x7f2b0400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:56.430 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.421+0000 7f2b18f2c700 1 --2- 172.21.15.42:0/1169211566 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2b1408df80 0x7f2b1409f2e0 secure :-1 s=READY pgs=2179 cs=0 l=1 rx=0x7f2b04008790 tx=0x7f2b0400fbf0).ready entity=mon.0 client_cookie=b503711a95e20806 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:37:56.431 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.421+0000 7f2b18f2c700 1 --2- 172.21.15.42:0/1169211566 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2b1408df80 0x7f2b1409f2e0 secure :-1 s=THROTTLE_DONE pgs=2179 cs=0 l=1 rx=0x7f2b04008790 tx=0x7f2b0400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:56.431 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.421+0000 7f2b0bfff700 1 -- 172.21.15.42:0/1169211566 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2b04011270 con 0x7f2b1408df80 2019-12-05T06:37:56.431 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.421+0000 7f2b18f2c700 1 --2- 172.21.15.42:0/1169211566 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2b1408df80 0x7f2b1409f2e0 secure :-1 s=THROTTLE_DONE pgs=2179 cs=0 l=1 rx=0x7f2b04008790 tx=0x7f2b0400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:56.431 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.421+0000 7f2b0bfff700 1 -- 172.21.15.42:0/1169211566 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b04008bc0 con 0x7f2b1408df80 2019-12-05T06:37:56.431 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.421+0000 7f2b18f2c700 1 --2- 172.21.15.42:0/1169211566 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2b1408df80 0x7f2b1409f2e0 secure :-1 s=THROTTLE_DONE pgs=2179 cs=0 l=1 rx=0x7f2b04008790 tx=0x7f2b0400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:56.431 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.421+0000 7f2b0bfff700 1 -- 172.21.15.42:0/1169211566 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2b04011850 con 0x7f2b1408df80 2019-12-05T06:37:56.431 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.421+0000 7f2b1b190700 1 -- 172.21.15.42:0/1169211566 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2b1408df80 msgr2=0x7f2b1409f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:37:56.432 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.421+0000 7f2b1b190700 1 --2- 172.21.15.42:0/1169211566 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2b1408df80 0x7f2b1409f2e0 secure :-1 s=READY pgs=2179 cs=0 l=1 rx=0x7f2b04008790 tx=0x7f2b0400fbf0).stop 2019-12-05T06:37:56.432 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.421+0000 7f2b1b190700 1 -- 172.21.15.42:0/1169211566 shutdown_connections 2019-12-05T06:37:56.432 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.421+0000 7f2b1b190700 1 --2- 172.21.15.42:0/1169211566 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2b1408df80 0x7f2b1409f2e0 unknown :-1 s=CLOSED pgs=2179 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:37:56.432 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.421+0000 7f2b1b190700 1 -- 172.21.15.42:0/1169211566 shutdown_connections 2019-12-05T06:37:56.432 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.421+0000 7f2b1b190700 1 -- 172.21.15.42:0/1169211566 wait complete. 2019-12-05T06:37:56.436 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.421+0000 7f2b1b190700 1 -- 172.21.15.42:0/1169211566 >> 172.21.15.42:0/1169211566 conn(0x7f2b140aabe0 msgr2=0x7f2b140a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:37:56.439 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.425+0000 7f2b1b190700 1 Processor -- start 2019-12-05T06:37:56.439 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.425+0000 7f2b1b190700 1 -- start start 2019-12-05T06:37:56.439 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.425+0000 7f2b1b190700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2b14133860 0x7f2b1411b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:37:56.440 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.425+0000 7f2b1b190700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2b1411cf30 0x7f2b1411bf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:37:56.440 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.425+0000 7f2b1b190700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b14123f80 con 0x7f2b1411a560 2019-12-05T06:37:56.440 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.425+0000 7f2b1b190700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b14122630 con 0x7f2b14133860 2019-12-05T06:37:56.440 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.425+0000 7f2b1b190700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b141228d0 con 0x7f2b1411cf30 2019-12-05T06:37:56.440 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.425+0000 7f2b19f2e700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2b1411cf30 0x7f2b1411bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:37:56.441 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.425+0000 7f2b18f2c700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2b14133860 0x7f2b1411b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:37:56.441 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.425+0000 7f2b19f2e700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2b1411cf30 0x7f2b1411bf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:59508/0 (socket says 172.21.15.42:59508) 2019-12-05T06:37:56.441 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.425+0000 7f2b19f2e700 1 -- 172.21.15.42:0/3811296835 learned_addr learned my addr 172.21.15.42:0/3811296835 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:37:56.441 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.425+0000 7f2b18f2c700 1 -- 172.21.15.42:0/3811296835 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2b1411cf30 msgr2=0x7f2b1411bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:37:56.441 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.425+0000 7f2b18f2c700 1 --2- 172.21.15.42:0/3811296835 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2b1411cf30 0x7f2b1411bf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:37:56.442 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.425+0000 7f2b18f2c700 1 -- 172.21.15.42:0/3811296835 >> v1:172.21.15.40:6789/0 conn(0x7f2b1411a560 legacy=0x7f2b14120ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:37:56.442 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.425+0000 7f2b18f2c700 1 -- 172.21.15.42:0/3811296835 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b141220b0 con 0x7f2b14133860 2019-12-05T06:37:56.442 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.425+0000 7f2b09ffb700 1 -- 172.21.15.42:0/3811296835 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2330392842 0 0) 0x7f2b14123f80 con 0x7f2b1411a560 2019-12-05T06:37:56.442 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.425+0000 7f2b18f2c700 1 --2- 172.21.15.42:0/3811296835 >> v2:172.21.15.40:3300/0 conn(0x7f2b14133860 0x7f2b1411b9b0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f2b141227f0 tx=0x7f2b04008e70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:56.443 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.425+0000 7f2b18f2c700 1 --2- 172.21.15.42:0/3811296835 >> v2:172.21.15.40:3300/0 conn(0x7f2b14133860 0x7f2b1411b9b0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f2b141227f0 tx=0x7f2b04008e70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:56.443 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.425+0000 7f2b18f2c700 1 --2- 172.21.15.42:0/3811296835 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2b14133860 0x7f2b1411b9b0 secure :-1 s=READY pgs=2181 cs=0 l=1 rx=0x7f2b141227f0 tx=0x7f2b04008e70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:37:56.443 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.425+0000 7f2b18f2c700 1 --2- 172.21.15.42:0/3811296835 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2b14133860 0x7f2b1411b9b0 secure :-1 s=THROTTLE_DONE pgs=2181 cs=0 l=1 rx=0x7f2b141227f0 tx=0x7f2b04008e70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:56.443 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.425+0000 7f2b09ffb700 1 -- 172.21.15.42:0/3811296835 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2b0400f360 con 0x7f2b14133860 2019-12-05T06:37:56.443 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.425+0000 7f2b18f2c700 1 --2- 172.21.15.42:0/3811296835 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2b14133860 0x7f2b1411b9b0 secure :-1 s=THROTTLE_DONE pgs=2181 cs=0 l=1 rx=0x7f2b141227f0 tx=0x7f2b04008e70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:56.444 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.425+0000 7f2b09ffb700 1 -- 172.21.15.42:0/3811296835 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b04011470 con 0x7f2b14133860 2019-12-05T06:37:56.444 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.425+0000 7f2b18f2c700 1 --2- 172.21.15.42:0/3811296835 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2b14133860 0x7f2b1411b9b0 secure :-1 s=THROTTLE_DONE pgs=2181 cs=0 l=1 rx=0x7f2b141227f0 tx=0x7f2b04008e70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:56.444 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.425+0000 7f2b1b190700 1 -- 172.21.15.42:0/3811296835 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b14122380 con 0x7f2b14133860 2019-12-05T06:37:56.444 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.425+0000 7f2b09ffb700 1 -- 172.21.15.42:0/3811296835 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2b04007340 con 0x7f2b14133860 2019-12-05T06:37:56.445 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.425+0000 7f2b1b190700 1 -- 172.21.15.42:0/3811296835 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b14077300 con 0x7f2b14133860 2019-12-05T06:37:56.445 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.429+0000 7f2b18f2c700 1 --2- 172.21.15.42:0/3811296835 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2b14133860 0x7f2b1411b9b0 secure :-1 s=THROTTLE_DONE pgs=2181 cs=0 l=1 rx=0x7f2b141227f0 tx=0x7f2b04008e70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:56.445 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.429+0000 7f2b09ffb700 1 -- 172.21.15.42:0/3811296835 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f2b04005540 con 0x7f2b14133860 2019-12-05T06:37:56.445 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.429+0000 7f2b18f2c700 1 --2- 172.21.15.42:0/3811296835 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2b14133860 0x7f2b1411b9b0 secure :-1 s=THROTTLE_DONE pgs=2181 cs=0 l=1 rx=0x7f2b141227f0 tx=0x7f2b04008e70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:56.446 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.429+0000 7f2b09ffb700 1 --2- 172.21.15.42:0/3811296835 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2af403abf0 0x7f2af403d100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:37:56.446 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.429+0000 7f2b09ffb700 1 -- 172.21.15.42:0/3811296835 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 11449+0+0 (secure 0 0 0) 0x7f2b0404bbb0 con 0x7f2b14133860 2019-12-05T06:37:56.446 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.429+0000 7f2b1972d700 1 --2- 172.21.15.42:0/3811296835 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2af403abf0 0x7f2af403d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:37:56.446 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.429+0000 7f2b1972d700 1 --2- 172.21.15.42:0/3811296835 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2af403abf0 0x7f2af403d100 crc :-1 s=READY pgs=1209 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:37:56.446 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.433+0000 7f2b1b190700 1 -- 172.21.15.42:0/3811296835 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b1411c450 con 0x7f2b14133860 2019-12-05T06:37:56.447 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.441+0000 7f2b18f2c700 1 --2- 172.21.15.42:0/3811296835 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2b14133860 0x7f2b1411b9b0 secure :-1 s=THROTTLE_DONE pgs=2181 cs=0 l=1 rx=0x7f2b141227f0 tx=0x7f2b04008e70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:56.447 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.441+0000 7f2b09ffb700 1 -- 172.21.15.42:0/3811296835 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f2b04030890 con 0x7f2b14133860 2019-12-05T06:37:56.608 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.601+0000 7f2b1b190700 1 -- 172.21.15.42:0/3811296835 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2b1408e650 con 0x7f2b14133860 2019-12-05T06:37:56.609 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.605+0000 7f2b18f2c700 1 --2- 172.21.15.42:0/3811296835 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2b14133860 0x7f2b1411b9b0 secure :-1 s=THROTTLE_DONE pgs=2181 cs=0 l=1 rx=0x7f2b141227f0 tx=0x7f2b04008e70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:56.609 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.605+0000 7f2b09ffb700 1 -- 172.21.15.42:0/3811296835 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 344 v344) v1 ==== 95+0+1775 (secure 0 0 0) 0x7f2b04033a40 con 0x7f2b14133860 2019-12-05T06:37:56.609 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 344 2019-12-05T06:37:56.609 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:37:56.610 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":344,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":344,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:37:55.151481+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11887},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11887":{"gid":11887,"name":"a","rank":0,"incarnation":344,"state":"up:replay","state_seq":2,"addr":"172.21.15.42:6835/958796360","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":958796360},{"type":"v1","addr":"172.21.15.42:6835","nonce":958796360}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:37:56.613 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.605+0000 7f2b1b190700 1 -- 172.21.15.42:0/3811296835 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2af403abf0 msgr2=0x7f2af403d100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:37:56.613 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.609+0000 7f2b1b190700 1 --2- 172.21.15.42:0/3811296835 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2af403abf0 0x7f2af403d100 crc :-1 s=READY pgs=1209 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:37:56.613 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.609+0000 7f2b1b190700 1 -- 172.21.15.42:0/3811296835 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2b14133860 msgr2=0x7f2b1411b9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:37:56.614 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.609+0000 7f2b1b190700 1 --2- 172.21.15.42:0/3811296835 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2b14133860 0x7f2b1411b9b0 secure :-1 s=READY pgs=2181 cs=0 l=1 rx=0x7f2b141227f0 tx=0x7f2b04008e70).stop 2019-12-05T06:37:56.614 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.609+0000 7f2b1b190700 1 -- 172.21.15.42:0/3811296835 shutdown_connections 2019-12-05T06:37:56.614 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.609+0000 7f2b1b190700 1 --2- 172.21.15.42:0/3811296835 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2af403abf0 0x7f2af403d100 unknown :-1 s=CLOSED pgs=1209 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:37:56.614 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.609+0000 7f2b1b190700 1 --2- 172.21.15.42:0/3811296835 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2b1411cf30 0x7f2b1411bf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:37:56.615 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.609+0000 7f2b1b190700 1 --2- 172.21.15.42:0/3811296835 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2b14133860 0x7f2b1411b9b0 unknown :-1 s=CLOSED pgs=2181 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:37:56.615 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.609+0000 7f2b1b190700 1 -- 172.21.15.42:0/3811296835 shutdown_connections 2019-12-05T06:37:56.616 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.609+0000 7f2b1b190700 1 -- 172.21.15.42:0/3811296835 wait complete. 2019-12-05T06:37:56.616 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.609+0000 7f2b1b190700 1 -- 172.21.15.42:0/3811296835 >> 172.21.15.42:0/3811296835 conn(0x7f2b140a9770 msgr2=0x7f2b1407e600 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:37:56.625 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b down, removed from mdsmap 2019-12-05T06:37:56.691 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2019-12-05T06:37:56.692 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:37:56.773 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.765+0000 7f6622dc3700 1 Processor -- start 2019-12-05T06:37:56.774 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.765+0000 7f6622dc3700 1 -- start start 2019-12-05T06:37:56.774 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.765+0000 7f6622dc3700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f661c09fae0 0x7f661c14fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:37:56.774 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.765+0000 7f6622dc3700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f661c0a0710 con 0x7f661c08df80 2019-12-05T06:37:56.774 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.765+0000 7f6622dc3700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f661c07ef20 con 0x7f661c0a7740 2019-12-05T06:37:56.774 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.765+0000 7f6622dc3700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f661c0a0b00 con 0x7f661c09fae0 2019-12-05T06:37:56.774 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.765+0000 7f6621b61700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f661c09fae0 0x7f661c14fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:37:56.775 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.765+0000 7f6621360700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f661c0a7740 0x7f661c07e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:59084/0 (socket says 172.21.15.42:59084) 2019-12-05T06:37:56.775 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.765+0000 7f6621360700 1 -- 172.21.15.42:0/1786576346 learned_addr learned my addr 172.21.15.42:0/1786576346 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:37:56.775 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.765+0000 7f6621b61700 1 -- 172.21.15.42:0/1786576346 >> v1:172.21.15.42:6789/0 conn(0x7f661c0a7740 legacy=0x7f661c07e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:37:56.775 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.765+0000 7f6621b61700 1 -- 172.21.15.42:0/1786576346 >> v1:172.21.15.40:6789/0 conn(0x7f661c08df80 legacy=0x7f661c09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:37:56.775 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.765+0000 7f6621b61700 1 -- 172.21.15.42:0/1786576346 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f661c0a9ba0 con 0x7f661c09fae0 2019-12-05T06:37:56.775 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.765+0000 7f6621b61700 1 --2- 172.21.15.42:0/1786576346 >> v2:172.21.15.42:3300/0 conn(0x7f661c09fae0 0x7f661c14fff0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f6614008b10 tx=0x7f661400ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:56.776 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.765+0000 7f6621b61700 1 --2- 172.21.15.42:0/1786576346 >> v2:172.21.15.42:3300/0 conn(0x7f661c09fae0 0x7f661c14fff0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f6614008b10 tx=0x7f661400ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:56.776 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.765+0000 7f6621b61700 1 --2- 172.21.15.42:0/1786576346 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f661c09fae0 0x7f661c14fff0 secure :-1 s=READY pgs=2877 cs=0 l=1 rx=0x7f6614008b10 tx=0x7f661400ff70).ready entity=mon.1 client_cookie=7b63f8ccd3389912 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:37:56.776 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.765+0000 7f6621b61700 1 --2- 172.21.15.42:0/1786576346 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f661c09fae0 0x7f661c14fff0 secure :-1 s=THROTTLE_DONE pgs=2877 cs=0 l=1 rx=0x7f6614008b10 tx=0x7f661400ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:56.776 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.765+0000 7f6621b61700 1 --2- 172.21.15.42:0/1786576346 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f661c09fae0 0x7f661c14fff0 secure :-1 s=THROTTLE_DONE pgs=2877 cs=0 l=1 rx=0x7f6614008b10 tx=0x7f661400ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:56.776 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.765+0000 7f6613fff700 1 -- 172.21.15.42:0/1786576346 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6614010970 con 0x7f661c09fae0 2019-12-05T06:37:56.777 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.769+0000 7f6621b61700 1 --2- 172.21.15.42:0/1786576346 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f661c09fae0 0x7f661c14fff0 secure :-1 s=THROTTLE_DONE pgs=2877 cs=0 l=1 rx=0x7f6614008b10 tx=0x7f661400ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:56.777 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.769+0000 7f6613fff700 1 -- 172.21.15.42:0/1786576346 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6614008b50 con 0x7f661c09fae0 2019-12-05T06:37:56.777 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.769+0000 7f6613fff700 1 -- 172.21.15.42:0/1786576346 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f66140110c0 con 0x7f661c09fae0 2019-12-05T06:37:56.777 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.769+0000 7f6622dc3700 1 -- 172.21.15.42:0/1786576346 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f661c09fae0 msgr2=0x7f661c14fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:37:56.777 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.769+0000 7f6622dc3700 1 --2- 172.21.15.42:0/1786576346 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f661c09fae0 0x7f661c14fff0 secure :-1 s=READY pgs=2877 cs=0 l=1 rx=0x7f6614008b10 tx=0x7f661400ff70).stop 2019-12-05T06:37:56.777 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.769+0000 7f6622dc3700 1 -- 172.21.15.42:0/1786576346 shutdown_connections 2019-12-05T06:37:56.778 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.769+0000 7f6622dc3700 1 --2- 172.21.15.42:0/1786576346 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f661c09fae0 0x7f661c14fff0 unknown :-1 s=CLOSED pgs=2877 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:37:56.778 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.769+0000 7f6622dc3700 1 -- 172.21.15.42:0/1786576346 shutdown_connections 2019-12-05T06:37:56.778 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.769+0000 7f6622dc3700 1 -- 172.21.15.42:0/1786576346 wait complete. 2019-12-05T06:37:56.778 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.769+0000 7f6622dc3700 1 -- 172.21.15.42:0/1786576346 >> 172.21.15.42:0/1786576346 conn(0x7f661c0aabe0 msgr2=0x7f661c0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:37:56.781 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.769+0000 7f6622dc3700 1 Processor -- start 2019-12-05T06:37:56.781 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.769+0000 7f6622dc3700 1 -- start start 2019-12-05T06:37:56.781 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.773+0000 7f6622dc3700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f661c0a63c0 0x7f661c142b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:37:56.781 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.773+0000 7f6622dc3700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f661c143860 0x7f661c126ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:37:56.781 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.773+0000 7f6622dc3700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f661c143c90 con 0x7f661c07e9b0 2019-12-05T06:37:56.782 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.773+0000 7f6622dc3700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f661c143fd0 con 0x7f661c0a63c0 2019-12-05T06:37:56.782 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.773+0000 7f6622dc3700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f661c144270 con 0x7f661c143860 2019-12-05T06:37:56.782 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.773+0000 7f6621360700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f661c07e9b0 0x7f661c0a5bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:59090/0 (socket says 172.21.15.42:59090) 2019-12-05T06:37:56.782 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.773+0000 7f6621360700 1 -- 172.21.15.42:0/2220808452 learned_addr learned my addr 172.21.15.42:0/2220808452 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:37:56.782 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.773+0000 7f6620b5f700 1 --2- 172.21.15.42:0/2220808452 >> v2:172.21.15.42:3300/0 conn(0x7f661c0a63c0 0x7f661c142b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:37:56.782 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.773+0000 7f6621b61700 1 --2- 172.21.15.42:0/2220808452 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f661c143860 0x7f661c126ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:37:56.782 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.773+0000 7f6620b5f700 1 -- 172.21.15.42:0/2220808452 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f661c143860 msgr2=0x7f661c126ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:37:56.783 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.773+0000 7f6620b5f700 1 --2- 172.21.15.42:0/2220808452 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f661c143860 0x7f661c126ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:37:56.783 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.773+0000 7f6620b5f700 1 -- 172.21.15.42:0/2220808452 >> v1:172.21.15.42:6789/0 conn(0x7f661c07e9b0 legacy=0x7f661c0a5bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:37:56.783 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.773+0000 7f6620b5f700 1 -- 172.21.15.42:0/2220808452 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f661c1273f0 con 0x7f661c0a63c0 2019-12-05T06:37:56.783 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.773+0000 7f6611ffb700 1 -- 172.21.15.42:0/2220808452 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2716976799 0 0) 0x7f661c143c90 con 0x7f661c07e9b0 2019-12-05T06:37:56.783 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.773+0000 7f6621b61700 1 --2- 172.21.15.42:0/2220808452 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f661c143860 0x7f661c126ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:37:56.783 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.773+0000 7f6620b5f700 1 --2- 172.21.15.42:0/2220808452 >> v2:172.21.15.42:3300/0 conn(0x7f661c0a63c0 0x7f661c142b20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f661c144190 tx=0x7f660c00c320).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:56.783 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.773+0000 7f6620b5f700 1 --2- 172.21.15.42:0/2220808452 >> v2:172.21.15.42:3300/0 conn(0x7f661c0a63c0 0x7f661c142b20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f661c144190 tx=0x7f660c00c320).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:56.784 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.773+0000 7f6620b5f700 1 --2- 172.21.15.42:0/2220808452 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f661c0a63c0 0x7f661c142b20 secure :-1 s=READY pgs=2879 cs=0 l=1 rx=0x7f661c144190 tx=0x7f660c00c320).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:37:56.784 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.773+0000 7f6620b5f700 1 --2- 172.21.15.42:0/2220808452 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f661c0a63c0 0x7f661c142b20 secure :-1 s=THROTTLE_DONE pgs=2879 cs=0 l=1 rx=0x7f661c144190 tx=0x7f660c00c320).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:56.784 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.773+0000 7f6611ffb700 1 -- 172.21.15.42:0/2220808452 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f660c00ca80 con 0x7f661c0a63c0 2019-12-05T06:37:56.784 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.773+0000 7f6620b5f700 1 --2- 172.21.15.42:0/2220808452 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f661c0a63c0 0x7f661c142b20 secure :-1 s=THROTTLE_DONE pgs=2879 cs=0 l=1 rx=0x7f661c144190 tx=0x7f660c00c320).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:56.784 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.773+0000 7f6620b5f700 1 --2- 172.21.15.42:0/2220808452 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f661c0a63c0 0x7f661c142b20 secure :-1 s=THROTTLE_DONE pgs=2879 cs=0 l=1 rx=0x7f661c144190 tx=0x7f660c00c320).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:56.784 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.773+0000 7f6611ffb700 1 -- 172.21.15.42:0/2220808452 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f660c00c6a0 con 0x7f661c0a63c0 2019-12-05T06:37:56.784 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.773+0000 7f6622dc3700 1 -- 172.21.15.42:0/2220808452 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f661c127660 con 0x7f661c0a63c0 2019-12-05T06:37:56.785 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.773+0000 7f6611ffb700 1 -- 172.21.15.42:0/2220808452 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f660c0110e0 con 0x7f661c0a63c0 2019-12-05T06:37:56.785 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.773+0000 7f6622dc3700 1 -- 172.21.15.42:0/2220808452 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f661c12e0d0 con 0x7f661c0a63c0 2019-12-05T06:37:56.785 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.773+0000 7f6620b5f700 1 --2- 172.21.15.42:0/2220808452 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f661c0a63c0 0x7f661c142b20 secure :-1 s=THROTTLE_DONE pgs=2879 cs=0 l=1 rx=0x7f661c144190 tx=0x7f660c00c320).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:56.785 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.773+0000 7f6611ffb700 1 -- 172.21.15.42:0/2220808452 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f660c0113b0 con 0x7f661c0a63c0 2019-12-05T06:37:56.785 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.773+0000 7f6620b5f700 1 --2- 172.21.15.42:0/2220808452 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f661c0a63c0 0x7f661c142b20 secure :-1 s=THROTTLE_DONE pgs=2879 cs=0 l=1 rx=0x7f661c144190 tx=0x7f660c00c320).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:56.785 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.773+0000 7f6611ffb700 1 --2- 172.21.15.42:0/2220808452 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f65fc03ace0 0x7f65fc03d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:37:56.785 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.773+0000 7f6611ffb700 1 -- 172.21.15.42:0/2220808452 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 11449+0+0 (secure 0 0 0) 0x7f660c04b110 con 0x7f661c0a63c0 2019-12-05T06:37:56.786 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.773+0000 7f6621360700 1 --2- 172.21.15.42:0/2220808452 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f65fc03ace0 0x7f65fc03d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:37:56.786 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.773+0000 7f6621360700 1 --2- 172.21.15.42:0/2220808452 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f65fc03ace0 0x7f65fc03d1f0 crc :-1 s=READY pgs=1210 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:37:56.786 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.781+0000 7f6622dc3700 1 -- 172.21.15.42:0/2220808452 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f661c05d110 con 0x7f661c0a63c0 2019-12-05T06:37:56.791 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.785+0000 7f6620b5f700 1 --2- 172.21.15.42:0/2220808452 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f661c0a63c0 0x7f661c142b20 secure :-1 s=THROTTLE_DONE pgs=2879 cs=0 l=1 rx=0x7f661c144190 tx=0x7f660c00c320).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:56.792 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.785+0000 7f6611ffb700 1 -- 172.21.15.42:0/2220808452 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f660c032190 con 0x7f661c0a63c0 2019-12-05T06:37:56.952 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.945+0000 7f6622dc3700 1 -- 172.21.15.42:0/2220808452 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f661c05b8c0 con 0x7f661c0a63c0 2019-12-05T06:37:56.952 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.945+0000 7f6620b5f700 1 --2- 172.21.15.42:0/2220808452 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f661c0a63c0 0x7f661c142b20 secure :-1 s=THROTTLE_DONE pgs=2879 cs=0 l=1 rx=0x7f661c144190 tx=0x7f660c00c320).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:56.952 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.945+0000 7f6611ffb700 1 -- 172.21.15.42:0/2220808452 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 344 v344) v1 ==== 95+0+1775 (secure 0 0 0) 0x7f660c0323f0 con 0x7f661c0a63c0 2019-12-05T06:37:56.953 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 344 2019-12-05T06:37:56.953 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:37:56.953 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":344,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":344,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:37:55.151481+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11887},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11887":{"gid":11887,"name":"a","rank":0,"incarnation":344,"state":"up:replay","state_seq":2,"addr":"172.21.15.42:6835/958796360","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":958796360},{"type":"v1","addr":"172.21.15.42:6835","nonce":958796360}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:37:56.955 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.949+0000 7f6622dc3700 1 -- 172.21.15.42:0/2220808452 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f65fc03ace0 msgr2=0x7f65fc03d1f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:37:56.955 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.949+0000 7f6622dc3700 1 --2- 172.21.15.42:0/2220808452 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f65fc03ace0 0x7f65fc03d1f0 crc :-1 s=READY pgs=1210 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:37:56.955 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.949+0000 7f6622dc3700 1 -- 172.21.15.42:0/2220808452 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f661c0a63c0 msgr2=0x7f661c142b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:37:56.956 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.949+0000 7f6622dc3700 1 --2- 172.21.15.42:0/2220808452 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f661c0a63c0 0x7f661c142b20 secure :-1 s=READY pgs=2879 cs=0 l=1 rx=0x7f661c144190 tx=0x7f660c00c320).stop 2019-12-05T06:37:56.956 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.949+0000 7f6622dc3700 1 -- 172.21.15.42:0/2220808452 shutdown_connections 2019-12-05T06:37:56.956 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.949+0000 7f6622dc3700 1 --2- 172.21.15.42:0/2220808452 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f65fc03ace0 0x7f65fc03d1f0 unknown :-1 s=CLOSED pgs=1210 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:37:56.956 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.949+0000 7f6622dc3700 1 --2- 172.21.15.42:0/2220808452 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f661c143860 0x7f661c126ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:37:56.956 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.949+0000 7f6622dc3700 1 --2- 172.21.15.42:0/2220808452 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f661c0a63c0 0x7f661c142b20 unknown :-1 s=CLOSED pgs=2879 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:37:56.956 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.949+0000 7f6622dc3700 1 -- 172.21.15.42:0/2220808452 shutdown_connections 2019-12-05T06:37:56.956 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.949+0000 7f6622dc3700 1 -- 172.21.15.42:0/2220808452 wait complete. 2019-12-05T06:37:56.957 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:56.949+0000 7f6622dc3700 1 -- 172.21.15.42:0/2220808452 >> 172.21.15.42:0/2220808452 conn(0x7f661c1150a0 msgr2=0x7f661c059b20 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:37:56.964 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 344, "feature_flags": { "enable_multiple": false, "ever_enabled_multiple": false }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2019-12-05T05:03:35.604738+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 344, "ever_allowed_features": 2, "explicitly_allowed_features": 2, "failed": [], "flags": 18, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_11887": { "addr": "172.21.15.42:6835/958796360", "addrs": { "addrvec": [ { "addr": "172.21.15.42:6834", "nonce": 958796360, "type": "v2" }, { "addr": "172.21.15.42:6835", "nonce": 958796360, "type": "v1" } ] }, "export_targets": [], "features": 4540138292836696063, "flags": 0, "gid": 11887, "incarnation": 344, "name": "a", "rank": 0, "state": "up:replay", "state_seq": 2 } }, "last_failure": 0, "last_failure_osd_epoch": 112, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 2, "min_compat_client": "0 (unknown)", "modified": "2019-12-05T06:37:55.151481+0000", "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 11887 } } } ], "standbys": [] } 2019-12-05T06:37:58.966 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:37:59.072 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:59.061+0000 7f57d9ad0700 1 Processor -- start 2019-12-05T06:37:59.073 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:59.061+0000 7f57d9ad0700 1 -- start start 2019-12-05T06:37:59.073 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:59.061+0000 7f57d9ad0700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f57d40a7740 0x7f57d408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:37:59.073 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:59.061+0000 7f57d9ad0700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57d40a0710 con 0x7f57d407e720 2019-12-05T06:37:59.073 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:59.061+0000 7f57d9ad0700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57d40a9ba0 con 0x7f57d409fae0 2019-12-05T06:37:59.073 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:59.061+0000 7f57d9ad0700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57d40a0b00 con 0x7f57d40a7740 2019-12-05T06:37:59.073 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:59.065+0000 7f57d886e700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f57d409fae0 0x7f57d414efe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:59104/0 (socket says 172.21.15.42:59104) 2019-12-05T06:37:59.074 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:59.065+0000 7f57d886e700 1 -- 172.21.15.42:0/482221284 learned_addr learned my addr 172.21.15.42:0/482221284 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:37:59.074 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:59.065+0000 7f57d3fff700 1 --2- 172.21.15.42:0/482221284 >> v2:172.21.15.40:3300/0 conn(0x7f57d40a7740 0x7f57d408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:37:59.074 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:59.065+0000 7f57d2ffd700 1 -- 172.21.15.42:0/482221284 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2528046860 0 0) 0x7f57d40a9ba0 con 0x7f57d409fae0 2019-12-05T06:37:59.074 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:59.065+0000 7f57d2ffd700 1 -- 172.21.15.42:0/482221284 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57c00030b0 con 0x7f57d409fae0 2019-12-05T06:37:59.074 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:59.065+0000 7f57d2ffd700 1 -- 172.21.15.42:0/482221284 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3312894839 0 0) 0x7f57c00030b0 con 0x7f57d409fae0 2019-12-05T06:37:59.074 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:59.065+0000 7f57d3fff700 1 -- 172.21.15.42:0/482221284 >> v1:172.21.15.42:6789/0 conn(0x7f57d409fae0 legacy=0x7f57d414efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:37:59.075 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:59.065+0000 7f57d3fff700 1 -- 172.21.15.42:0/482221284 >> v1:172.21.15.40:6789/0 conn(0x7f57d407e720 legacy=0x7f57d409f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:37:59.075 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:59.065+0000 7f57d3fff700 1 -- 172.21.15.42:0/482221284 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57d407eb50 con 0x7f57d40a7740 2019-12-05T06:37:59.075 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:59.065+0000 7f57d3fff700 1 --2- 172.21.15.42:0/482221284 >> v2:172.21.15.40:3300/0 conn(0x7f57d40a7740 0x7f57d408df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f57bc001110 tx=0x7f57cc001990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:59.075 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:59.065+0000 7f57d3fff700 1 --2- 172.21.15.42:0/482221284 >> v2:172.21.15.40:3300/0 conn(0x7f57d40a7740 0x7f57d408df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f57bc001110 tx=0x7f57cc001990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:59.075 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:59.065+0000 7f57d3fff700 1 --2- 172.21.15.42:0/482221284 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f57d40a7740 0x7f57d408df80 secure :-1 s=READY pgs=2184 cs=0 l=1 rx=0x7f57bc001110 tx=0x7f57cc001990).ready entity=mon.0 client_cookie=7aee9d0ba39287 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:37:59.076 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:59.065+0000 7f57d3fff700 1 --2- 172.21.15.42:0/482221284 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f57d40a7740 0x7f57d408df80 secure :-1 s=THROTTLE_DONE pgs=2184 cs=0 l=1 rx=0x7f57bc001110 tx=0x7f57cc001990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:59.076 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:59.065+0000 7f57d2ffd700 1 -- 172.21.15.42:0/482221284 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f57cc001750 con 0x7f57d40a7740 2019-12-05T06:37:59.076 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:59.065+0000 7f57d3fff700 1 --2- 172.21.15.42:0/482221284 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f57d40a7740 0x7f57d408df80 secure :-1 s=THROTTLE_DONE pgs=2184 cs=0 l=1 rx=0x7f57bc001110 tx=0x7f57cc001990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:59.076 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:59.065+0000 7f57d3fff700 1 --2- 172.21.15.42:0/482221284 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f57d40a7740 0x7f57d408df80 secure :-1 s=THROTTLE_DONE pgs=2184 cs=0 l=1 rx=0x7f57bc001110 tx=0x7f57cc001990).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:59.077 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:59.065+0000 7f57d2ffd700 1 -- 172.21.15.42:0/482221284 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57cc001560 con 0x7f57d40a7740 2019-12-05T06:37:59.077 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:59.065+0000 7f57d2ffd700 1 -- 172.21.15.42:0/482221284 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f57c4010a60 con 0x7f57d40a7740 2019-12-05T06:37:59.077 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:59.065+0000 7f57d9ad0700 1 -- 172.21.15.42:0/482221284 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f57d40a7740 msgr2=0x7f57d408df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:37:59.077 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:59.065+0000 7f57d9ad0700 1 --2- 172.21.15.42:0/482221284 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f57d40a7740 0x7f57d408df80 secure :-1 s=READY pgs=2184 cs=0 l=1 rx=0x7f57bc001110 tx=0x7f57cc001990).stop 2019-12-05T06:37:59.077 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:59.065+0000 7f57d9ad0700 1 -- 172.21.15.42:0/482221284 shutdown_connections 2019-12-05T06:37:59.077 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:59.065+0000 7f57d9ad0700 1 --2- 172.21.15.42:0/482221284 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f57d40a7740 0x7f57d408df80 unknown :-1 s=CLOSED pgs=2184 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:37:59.078 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:59.065+0000 7f57d9ad0700 1 -- 172.21.15.42:0/482221284 shutdown_connections 2019-12-05T06:37:59.078 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:59.065+0000 7f57d9ad0700 1 -- 172.21.15.42:0/482221284 wait complete. 2019-12-05T06:37:59.078 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:59.065+0000 7f57d9ad0700 1 -- 172.21.15.42:0/482221284 >> 172.21.15.42:0/482221284 conn(0x7f57d40aabe0 msgr2=0x7f57d40a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:37:59.082 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:59.069+0000 7f57d9ad0700 1 Processor -- start 2019-12-05T06:37:59.082 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:59.069+0000 7f57d9ad0700 1 -- start start 2019-12-05T06:37:59.083 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:59.069+0000 7f57d9ad0700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f57d4116900 0x7f57d40a3970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:37:59.083 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:59.069+0000 7f57d9ad0700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f57d4144ed0 0x7f57d412ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:37:59.083 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:59.069+0000 7f57d9ad0700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57d4116d30 con 0x7f57d40a9ad0 2019-12-05T06:37:59.083 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:59.069+0000 7f57d9ad0700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57d40a3ec0 con 0x7f57d4116900 2019-12-05T06:37:59.083 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:59.069+0000 7f57d9ad0700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57d4144840 con 0x7f57d4144ed0 2019-12-05T06:37:59.083 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:59.069+0000 7f57d886e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f57d4144ed0 0x7f57d412ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:37:59.083 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:59.069+0000 7f57d886e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f57d4144ed0 0x7f57d412ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:59070/0 (socket says 172.21.15.42:59070) 2019-12-05T06:37:59.084 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:59.069+0000 7f57d886e700 1 -- 172.21.15.42:0/284521588 learned_addr learned my addr 172.21.15.42:0/284521588 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:37:59.084 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:59.069+0000 7f57d37fe700 1 --2- 172.21.15.42:0/284521588 >> v2:172.21.15.40:3300/0 conn(0x7f57d4116900 0x7f57d40a3970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:37:59.084 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:59.069+0000 7f57d0ff9700 1 -- 172.21.15.42:0/284521588 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3562061320 0 0) 0x7f57d4116d30 con 0x7f57d40a9ad0 2019-12-05T06:37:59.084 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:59.069+0000 7f57d886e700 1 -- 172.21.15.42:0/284521588 >> v2:172.21.15.40:3300/0 conn(0x7f57d4116900 msgr2=0x7f57d40a3970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:37:59.084 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:59.069+0000 7f57d886e700 1 --2- 172.21.15.42:0/284521588 >> v2:172.21.15.40:3300/0 conn(0x7f57d4116900 0x7f57d40a3970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:37:59.084 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:59.069+0000 7f57d886e700 1 -- 172.21.15.42:0/284521588 >> v1:172.21.15.40:6789/0 conn(0x7f57d40a9ad0 legacy=0x7f57d407b900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:37:59.084 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:59.069+0000 7f57d886e700 1 -- 172.21.15.42:0/284521588 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57d407c100 con 0x7f57d4144ed0 2019-12-05T06:37:59.085 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:59.069+0000 7f57d886e700 1 --2- 172.21.15.42:0/284521588 >> v2:172.21.15.42:3300/0 conn(0x7f57d4144ed0 0x7f57d412ef20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f57d4128b10 tx=0x7f57cc0108a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:59.085 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:59.069+0000 7f57d886e700 1 --2- 172.21.15.42:0/284521588 >> v2:172.21.15.42:3300/0 conn(0x7f57d4144ed0 0x7f57d412ef20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f57d4128b10 tx=0x7f57cc0108a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:59.085 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:59.069+0000 7f57d886e700 1 --2- 172.21.15.42:0/284521588 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f57d4144ed0 0x7f57d412ef20 secure :-1 s=READY pgs=2881 cs=0 l=1 rx=0x7f57d4128b10 tx=0x7f57cc0108a0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:37:59.085 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:59.073+0000 7f57d886e700 1 --2- 172.21.15.42:0/284521588 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f57d4144ed0 0x7f57d412ef20 secure :-1 s=THROTTLE_DONE pgs=2881 cs=0 l=1 rx=0x7f57d4128b10 tx=0x7f57cc0108a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:59.085 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:59.073+0000 7f57d0ff9700 1 -- 172.21.15.42:0/284521588 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f57cc011680 con 0x7f57d4144ed0 2019-12-05T06:37:59.085 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:59.073+0000 7f57d886e700 1 --2- 172.21.15.42:0/284521588 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f57d4144ed0 0x7f57d412ef20 secure :-1 s=THROTTLE_DONE pgs=2881 cs=0 l=1 rx=0x7f57d4128b10 tx=0x7f57cc0108a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:59.086 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:59.073+0000 7f57d886e700 1 --2- 172.21.15.42:0/284521588 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f57d4144ed0 0x7f57d412ef20 secure :-1 s=THROTTLE_DONE pgs=2881 cs=0 l=1 rx=0x7f57d4128b10 tx=0x7f57cc0108a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:59.086 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:59.073+0000 7f57d0ff9700 1 -- 172.21.15.42:0/284521588 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57cc0112a0 con 0x7f57d4144ed0 2019-12-05T06:37:59.086 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:59.073+0000 7f57d9ad0700 1 -- 172.21.15.42:0/284521588 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57d4145300 con 0x7f57d4144ed0 2019-12-05T06:37:59.086 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:59.073+0000 7f57d0ff9700 1 -- 172.21.15.42:0/284521588 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f57cc011c80 con 0x7f57d4144ed0 2019-12-05T06:37:59.086 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:59.073+0000 7f57d9ad0700 1 -- 172.21.15.42:0/284521588 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57d4150230 con 0x7f57d4144ed0 2019-12-05T06:37:59.086 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:59.073+0000 7f57d886e700 1 --2- 172.21.15.42:0/284521588 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f57d4144ed0 0x7f57d412ef20 secure :-1 s=THROTTLE_DONE pgs=2881 cs=0 l=1 rx=0x7f57d4128b10 tx=0x7f57cc0108a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:59.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:59.073+0000 7f57d0ff9700 1 -- 172.21.15.42:0/284521588 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f57cc00b330 con 0x7f57d4144ed0 2019-12-05T06:37:59.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:59.073+0000 7f57d886e700 1 --2- 172.21.15.42:0/284521588 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f57d4144ed0 0x7f57d412ef20 secure :-1 s=THROTTLE_DONE pgs=2881 cs=0 l=1 rx=0x7f57d4128b10 tx=0x7f57cc0108a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:59.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:59.073+0000 7f57d0ff9700 1 --2- 172.21.15.42:0/284521588 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f57b803ac90 0x7f57b803d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:37:59.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:59.073+0000 7f57d0ff9700 1 -- 172.21.15.42:0/284521588 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 11449+0+0 (secure 0 0 0) 0x7f57cc04b1e0 con 0x7f57d4144ed0 2019-12-05T06:37:59.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:59.073+0000 7f57d3fff700 1 --2- 172.21.15.42:0/284521588 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f57b803ac90 0x7f57b803d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:37:59.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:59.073+0000 7f57d3fff700 1 --2- 172.21.15.42:0/284521588 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f57b803ac90 0x7f57b803d1a0 crc :-1 s=READY pgs=1211 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:37:59.088 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:59.077+0000 7f57d9ad0700 1 -- 172.21.15.42:0/284521588 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57d4144a00 con 0x7f57d4144ed0 2019-12-05T06:37:59.090 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:59.085+0000 7f57d886e700 1 --2- 172.21.15.42:0/284521588 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f57d4144ed0 0x7f57d412ef20 secure :-1 s=THROTTLE_DONE pgs=2881 cs=0 l=1 rx=0x7f57d4128b10 tx=0x7f57cc0108a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:59.090 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:59.085+0000 7f57d0ff9700 1 -- 172.21.15.42:0/284521588 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f57cc032200 con 0x7f57d4144ed0 2019-12-05T06:37:59.276 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:59.269+0000 7f57d9ad0700 1 -- 172.21.15.42:0/284521588 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f57d405d110 con 0x7f57d4144ed0 2019-12-05T06:37:59.276 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:59.269+0000 7f57d886e700 1 --2- 172.21.15.42:0/284521588 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f57d4144ed0 0x7f57d412ef20 secure :-1 s=THROTTLE_DONE pgs=2881 cs=0 l=1 rx=0x7f57d4128b10 tx=0x7f57cc0108a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:59.276 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:59.269+0000 7f57d0ff9700 1 -- 172.21.15.42:0/284521588 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 346 v346) v1 ==== 95+0+1776 (secure 0 0 0) 0x7f57cc032460 con 0x7f57d4144ed0 2019-12-05T06:37:59.277 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 346 2019-12-05T06:37:59.278 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:37:59.278 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":346,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":346,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:37:59.180074+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11887},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11887":{"gid":11887,"name":"a","rank":0,"incarnation":344,"state":"up:rejoin","state_seq":36,"addr":"172.21.15.42:6835/958796360","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":958796360},{"type":"v1","addr":"172.21.15.42:6835","nonce":958796360}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:37:59.280 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:59.273+0000 7f57d9ad0700 1 -- 172.21.15.42:0/284521588 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f57b803ac90 msgr2=0x7f57b803d1a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:37:59.280 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:59.273+0000 7f57d9ad0700 1 --2- 172.21.15.42:0/284521588 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f57b803ac90 0x7f57b803d1a0 crc :-1 s=READY pgs=1211 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:37:59.281 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:59.273+0000 7f57d9ad0700 1 -- 172.21.15.42:0/284521588 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f57d4144ed0 msgr2=0x7f57d412ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:37:59.281 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:59.273+0000 7f57d9ad0700 1 --2- 172.21.15.42:0/284521588 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f57d4144ed0 0x7f57d412ef20 secure :-1 s=READY pgs=2881 cs=0 l=1 rx=0x7f57d4128b10 tx=0x7f57cc0108a0).stop 2019-12-05T06:37:59.281 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:59.273+0000 7f57d9ad0700 1 -- 172.21.15.42:0/284521588 shutdown_connections 2019-12-05T06:37:59.281 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:59.273+0000 7f57d9ad0700 1 --2- 172.21.15.42:0/284521588 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f57b803ac90 0x7f57b803d1a0 unknown :-1 s=CLOSED pgs=1211 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:37:59.282 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:59.273+0000 7f57d9ad0700 1 --2- 172.21.15.42:0/284521588 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f57d4144ed0 0x7f57d412ef20 unknown :-1 s=CLOSED pgs=2881 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:37:59.282 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:59.273+0000 7f57d9ad0700 1 --2- 172.21.15.42:0/284521588 >> v2:172.21.15.40:3300/0 conn(0x7f57d4116900 0x7f57d40a3970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:37:59.282 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:59.273+0000 7f57d9ad0700 1 -- 172.21.15.42:0/284521588 shutdown_connections 2019-12-05T06:37:59.282 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:59.273+0000 7f57d9ad0700 1 -- 172.21.15.42:0/284521588 wait complete. 2019-12-05T06:37:59.282 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:37:59.273+0000 7f57d9ad0700 1 -- 172.21.15.42:0/284521588 >> 172.21.15.42:0/284521588 conn(0x7f57d40a7740 msgr2=0x7f57d407dab0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:37:59.290 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2019-12-05T06:38:01.290 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:38:01.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:01.393+0000 7f21e24da700 1 Processor -- start 2019-12-05T06:38:01.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:01.393+0000 7f21e24da700 1 -- start start 2019-12-05T06:38:01.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:01.393+0000 7f21e24da700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f21dc08df80 0x7f21dc09f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:38:01.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:01.393+0000 7f21e24da700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f21dc09f830 0x7f21dc14efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:38:01.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:01.393+0000 7f21e24da700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21dc09fc60 con 0x7f21dc0a7740 2019-12-05T06:38:01.468 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:01.393+0000 7f21e24da700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21dc0a0b00 con 0x7f21dc08df80 2019-12-05T06:38:01.469 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:01.393+0000 7f21e24da700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21dc08e5b0 con 0x7f21dc09f830 2019-12-05T06:38:01.469 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:01.393+0000 7f21dbfff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f21dc08df80 0x7f21dc09f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:38:01.469 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:01.393+0000 7f21dbfff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f21dc08df80 0x7f21dc09f2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:59090/0 (socket says 172.21.15.42:59090) 2019-12-05T06:38:01.469 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:01.393+0000 7f21dbfff700 1 -- 172.21.15.42:0/506875904 learned_addr learned my addr 172.21.15.42:0/506875904 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:38:01.469 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:01.393+0000 7f21e1278700 1 --2- 172.21.15.42:0/506875904 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f21dc09f830 0x7f21dc14efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:38:01.469 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:01.393+0000 7f21dbfff700 1 -- 172.21.15.42:0/506875904 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f21dc09f830 msgr2=0x7f21dc14efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:38:01.469 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:01.393+0000 7f21dbfff700 1 --2- 172.21.15.42:0/506875904 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f21dc09f830 0x7f21dc14efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:38:01.470 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:01.393+0000 7f21dbfff700 1 -- 172.21.15.42:0/506875904 >> v1:172.21.15.40:6789/0 conn(0x7f21dc0a7740 legacy=0x7f21dc07e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:38:01.470 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:01.393+0000 7f21dbfff700 1 -- 172.21.15.42:0/506875904 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21dc0a0710 con 0x7f21dc08df80 2019-12-05T06:38:01.470 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:01.393+0000 7f21dbfff700 1 --2- 172.21.15.42:0/506875904 >> v2:172.21.15.42:3300/0 conn(0x7f21dc08df80 0x7f21dc09f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f21c8008790 tx=0x7f21c800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:38:01.470 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:01.393+0000 7f21dbfff700 1 --2- 172.21.15.42:0/506875904 >> v2:172.21.15.42:3300/0 conn(0x7f21dc08df80 0x7f21dc09f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f21c8008790 tx=0x7f21c800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:38:01.470 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:01.393+0000 7f21dbfff700 1 --2- 172.21.15.42:0/506875904 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f21dc08df80 0x7f21dc09f2e0 secure :-1 s=READY pgs=2882 cs=0 l=1 rx=0x7f21c8008790 tx=0x7f21c800fd30).ready entity=mon.1 client_cookie=2241966976eddbc9 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:38:01.470 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:01.393+0000 7f21dbfff700 1 --2- 172.21.15.42:0/506875904 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f21dc08df80 0x7f21dc09f2e0 secure :-1 s=THROTTLE_DONE pgs=2882 cs=0 l=1 rx=0x7f21c8008790 tx=0x7f21c800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:38:01.470 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:01.393+0000 7f21db7fe700 1 -- 172.21.15.42:0/506875904 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f21c8008c90 con 0x7f21dc08df80 2019-12-05T06:38:01.471 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:01.393+0000 7f21dbfff700 1 --2- 172.21.15.42:0/506875904 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f21dc08df80 0x7f21dc09f2e0 secure :-1 s=THROTTLE_DONE pgs=2882 cs=0 l=1 rx=0x7f21c8008790 tx=0x7f21c800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:38:01.471 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:01.393+0000 7f21db7fe700 1 -- 172.21.15.42:0/506875904 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21c80088b0 con 0x7f21dc08df80 2019-12-05T06:38:01.471 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:01.393+0000 7f21dbfff700 1 --2- 172.21.15.42:0/506875904 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f21dc08df80 0x7f21dc09f2e0 secure :-1 s=THROTTLE_DONE pgs=2882 cs=0 l=1 rx=0x7f21c8008790 tx=0x7f21c800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:38:01.471 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:01.393+0000 7f21db7fe700 1 -- 172.21.15.42:0/506875904 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f21c8011b10 con 0x7f21dc08df80 2019-12-05T06:38:01.471 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:01.393+0000 7f21e24da700 1 -- 172.21.15.42:0/506875904 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f21dc08df80 msgr2=0x7f21dc09f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:38:01.471 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:01.393+0000 7f21e24da700 1 --2- 172.21.15.42:0/506875904 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f21dc08df80 0x7f21dc09f2e0 secure :-1 s=READY pgs=2882 cs=0 l=1 rx=0x7f21c8008790 tx=0x7f21c800fd30).stop 2019-12-05T06:38:01.471 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:01.397+0000 7f21e24da700 1 -- 172.21.15.42:0/506875904 shutdown_connections 2019-12-05T06:38:01.472 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:01.397+0000 7f21e24da700 1 --2- 172.21.15.42:0/506875904 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f21dc09f830 0x7f21dc14efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:38:01.472 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:01.397+0000 7f21e24da700 1 --2- 172.21.15.42:0/506875904 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f21dc08df80 0x7f21dc09f2e0 unknown :-1 s=CLOSED pgs=2882 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:38:01.472 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:01.397+0000 7f21e24da700 1 -- 172.21.15.42:0/506875904 shutdown_connections 2019-12-05T06:38:01.472 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:01.397+0000 7f21e24da700 1 -- 172.21.15.42:0/506875904 wait complete. 2019-12-05T06:38:01.472 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:01.397+0000 7f21e24da700 1 -- 172.21.15.42:0/506875904 >> 172.21.15.42:0/506875904 conn(0x7f21dc0aabe0 msgr2=0x7f21dc0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:38:01.476 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:01.397+0000 7f21e24da700 1 Processor -- start 2019-12-05T06:38:01.476 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:01.397+0000 7f21e24da700 1 -- start start 2019-12-05T06:38:01.476 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:01.401+0000 7f21e24da700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f21dc0a6f20 0x7f21dc11c020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:38:01.477 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:01.401+0000 7f21e24da700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f21dc1194f0 0x7f21dc1203c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:38:01.477 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:01.401+0000 7f21e24da700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21dc11ba90 con 0x7f21dc11a5f0 2019-12-05T06:38:01.477 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:01.401+0000 7f21e24da700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21dc120b30 con 0x7f21dc1194f0 2019-12-05T06:38:01.477 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:01.401+0000 7f21e24da700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21dc11bd80 con 0x7f21dc0a6f20 2019-12-05T06:38:01.477 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:01.401+0000 7f21e0a77700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f21dc0a6f20 0x7f21dc11c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:38:01.477 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:01.401+0000 7f21e0a77700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f21dc0a6f20 0x7f21dc11c020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:59094/0 (socket says 172.21.15.42:59094) 2019-12-05T06:38:01.478 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:01.401+0000 7f21e0a77700 1 -- 172.21.15.42:0/474995034 learned_addr learned my addr 172.21.15.42:0/474995034 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:38:01.478 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:01.401+0000 7f21dbfff700 1 --2- 172.21.15.42:0/474995034 >> v2:172.21.15.40:3300/0 conn(0x7f21dc1194f0 0x7f21dc1203c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:38:01.478 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:01.401+0000 7f21e0a77700 1 -- 172.21.15.42:0/474995034 >> v2:172.21.15.40:3300/0 conn(0x7f21dc1194f0 msgr2=0x7f21dc1203c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:38:01.478 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:01.401+0000 7f21e0a77700 1 --2- 172.21.15.42:0/474995034 >> v2:172.21.15.40:3300/0 conn(0x7f21dc1194f0 0x7f21dc1203c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:38:01.478 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:01.401+0000 7f21e0a77700 1 -- 172.21.15.42:0/474995034 >> v1:172.21.15.40:6789/0 conn(0x7f21dc11a5f0 legacy=0x7f21dc11edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:38:01.478 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:01.401+0000 7f21e0a77700 1 -- 172.21.15.42:0/474995034 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21dc11aa20 con 0x7f21dc0a6f20 2019-12-05T06:38:01.478 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:01.401+0000 7f21e0a77700 1 --2- 172.21.15.42:0/474995034 >> v2:172.21.15.42:3300/0 conn(0x7f21dc0a6f20 0x7f21dc11c020 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f21d0000e90 tx=0x7f21d00109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:38:01.478 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:01.401+0000 7f21e0a77700 1 --2- 172.21.15.42:0/474995034 >> v2:172.21.15.42:3300/0 conn(0x7f21dc0a6f20 0x7f21dc11c020 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f21d0000e90 tx=0x7f21d00109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:38:01.479 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:01.401+0000 7f21e0a77700 1 --2- 172.21.15.42:0/474995034 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f21dc0a6f20 0x7f21dc11c020 secure :-1 s=READY pgs=2883 cs=0 l=1 rx=0x7f21d0000e90 tx=0x7f21d00109a0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:38:01.479 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:01.401+0000 7f21e0a77700 1 --2- 172.21.15.42:0/474995034 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f21dc0a6f20 0x7f21dc11c020 secure :-1 s=THROTTLE_DONE pgs=2883 cs=0 l=1 rx=0x7f21d0000e90 tx=0x7f21d00109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:38:01.479 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:01.401+0000 7f21e0a77700 1 --2- 172.21.15.42:0/474995034 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f21dc0a6f20 0x7f21dc11c020 secure :-1 s=THROTTLE_DONE pgs=2883 cs=0 l=1 rx=0x7f21d0000e90 tx=0x7f21d00109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:38:01.479 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:01.401+0000 7f21e0a77700 1 --2- 172.21.15.42:0/474995034 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f21dc0a6f20 0x7f21dc11c020 secure :-1 s=THROTTLE_DONE pgs=2883 cs=0 l=1 rx=0x7f21d0000e90 tx=0x7f21d00109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:38:01.479 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:01.401+0000 7f21d97fa700 1 -- 172.21.15.42:0/474995034 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f21d000caa0 con 0x7f21dc0a6f20 2019-12-05T06:38:01.479 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:01.401+0000 7f21d97fa700 1 -- 172.21.15.42:0/474995034 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21d000c6c0 con 0x7f21dc0a6f20 2019-12-05T06:38:01.479 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:01.401+0000 7f21e24da700 1 -- 172.21.15.42:0/474995034 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21dc11f5b0 con 0x7f21dc0a6f20 2019-12-05T06:38:01.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:01.401+0000 7f21d97fa700 1 -- 172.21.15.42:0/474995034 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f21d0012210 con 0x7f21dc0a6f20 2019-12-05T06:38:01.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:01.401+0000 7f21e24da700 1 -- 172.21.15.42:0/474995034 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21dc11fb30 con 0x7f21dc0a6f20 2019-12-05T06:38:01.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:01.401+0000 7f21e0a77700 1 --2- 172.21.15.42:0/474995034 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f21dc0a6f20 0x7f21dc11c020 secure :-1 s=THROTTLE_DONE pgs=2883 cs=0 l=1 rx=0x7f21d0000e90 tx=0x7f21d00109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:38:01.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:01.405+0000 7f21d97fa700 1 -- 172.21.15.42:0/474995034 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f21d0012410 con 0x7f21dc0a6f20 2019-12-05T06:38:01.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:01.405+0000 7f21e0a77700 1 --2- 172.21.15.42:0/474995034 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f21dc0a6f20 0x7f21dc11c020 secure :-1 s=THROTTLE_DONE pgs=2883 cs=0 l=1 rx=0x7f21d0000e90 tx=0x7f21d00109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:38:01.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:01.405+0000 7f21d97fa700 1 --2- 172.21.15.42:0/474995034 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f21bc03ad50 0x7f21bc03d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:38:01.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:01.405+0000 7f21d97fa700 1 -- 172.21.15.42:0/474995034 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 11449+0+0 (secure 0 0 0) 0x7f21d004b270 con 0x7f21dc0a6f20 2019-12-05T06:38:01.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:01.405+0000 7f21dbfff700 1 --2- 172.21.15.42:0/474995034 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f21bc03ad50 0x7f21bc03d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:38:01.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:01.405+0000 7f21dbfff700 1 --2- 172.21.15.42:0/474995034 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f21bc03ad50 0x7f21bc03d260 crc :-1 s=READY pgs=1212 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:38:01.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:01.405+0000 7f21e24da700 1 -- 172.21.15.42:0/474995034 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21dc05d110 con 0x7f21dc0a6f20 2019-12-05T06:38:01.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:01.413+0000 7f21e0a77700 1 --2- 172.21.15.42:0/474995034 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f21dc0a6f20 0x7f21dc11c020 secure :-1 s=THROTTLE_DONE pgs=2883 cs=0 l=1 rx=0x7f21d0000e90 tx=0x7f21d00109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:38:01.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:01.413+0000 7f21d97fa700 1 -- 172.21.15.42:0/474995034 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f21d0032290 con 0x7f21dc0a6f20 2019-12-05T06:38:01.608 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:01.601+0000 7f21e24da700 1 -- 172.21.15.42:0/474995034 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f21dc05b8c0 con 0x7f21dc0a6f20 2019-12-05T06:38:01.609 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:01.605+0000 7f21e0a77700 1 --2- 172.21.15.42:0/474995034 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f21dc0a6f20 0x7f21dc11c020 secure :-1 s=THROTTLE_DONE pgs=2883 cs=0 l=1 rx=0x7f21d0000e90 tx=0x7f21d00109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:38:01.609 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:01.605+0000 7f21d97fa700 1 -- 172.21.15.42:0/474995034 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 347 v347) v1 ==== 95+0+1776 (secure 0 0 0) 0x7f21d00324f0 con 0x7f21dc0a6f20 2019-12-05T06:38:01.609 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 347 2019-12-05T06:38:01.609 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:38:01.610 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":347,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":347,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:38:00.190088+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11887},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11887":{"gid":11887,"name":"a","rank":0,"incarnation":344,"state":"up:active","state_seq":37,"addr":"172.21.15.42:6835/958796360","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":958796360},{"type":"v1","addr":"172.21.15.42:6835","nonce":958796360}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:38:01.612 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:01.605+0000 7f21e24da700 1 -- 172.21.15.42:0/474995034 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f21bc03ad50 msgr2=0x7f21bc03d260 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:38:01.612 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:01.605+0000 7f21e24da700 1 --2- 172.21.15.42:0/474995034 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f21bc03ad50 0x7f21bc03d260 crc :-1 s=READY pgs=1212 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:38:01.612 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:01.605+0000 7f21e24da700 1 -- 172.21.15.42:0/474995034 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f21dc0a6f20 msgr2=0x7f21dc11c020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:38:01.612 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:01.605+0000 7f21e24da700 1 --2- 172.21.15.42:0/474995034 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f21dc0a6f20 0x7f21dc11c020 secure :-1 s=READY pgs=2883 cs=0 l=1 rx=0x7f21d0000e90 tx=0x7f21d00109a0).stop 2019-12-05T06:38:01.613 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:01.605+0000 7f21e24da700 1 -- 172.21.15.42:0/474995034 shutdown_connections 2019-12-05T06:38:01.613 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:01.605+0000 7f21e24da700 1 --2- 172.21.15.42:0/474995034 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f21bc03ad50 0x7f21bc03d260 unknown :-1 s=CLOSED pgs=1212 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:38:01.613 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:01.605+0000 7f21e24da700 1 --2- 172.21.15.42:0/474995034 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f21dc0a6f20 0x7f21dc11c020 unknown :-1 s=CLOSED pgs=2883 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:38:01.613 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:01.605+0000 7f21e24da700 1 --2- 172.21.15.42:0/474995034 >> v2:172.21.15.40:3300/0 conn(0x7f21dc1194f0 0x7f21dc1203c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:38:01.613 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:01.605+0000 7f21e24da700 1 -- 172.21.15.42:0/474995034 shutdown_connections 2019-12-05T06:38:01.613 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:01.605+0000 7f21e24da700 1 -- 172.21.15.42:0/474995034 wait complete. 2019-12-05T06:38:01.613 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:01.605+0000 7f21e24da700 1 -- 172.21.15.42:0/474995034 >> 172.21.15.42:0/474995034 conn(0x7f21dc0a97e0 msgr2=0x7f21dc059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:38:01.624 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a has gained rank=0, replacing gid=7910 2019-12-05T06:38:01.624 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 1 secs before reviving mds.b 2019-12-05T06:38:02.625 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.b 2019-12-05T06:38:02.625 INFO:tasks.ceph.mds.b:Restarting daemon 2019-12-05T06:38:02.626 INFO:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2019-12-05T06:38:02.629 INFO:tasks.ceph.mds.b:Started 2019-12-05T06:38:02.630 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:38:02.691 INFO:tasks.ceph.mds.b.smithi040.stdout:starting mds.b at 2019-12-05T06:38:02.747 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:02.737+0000 7f67cdb8e700 1 Processor -- start 2019-12-05T06:38:02.748 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:02.737+0000 7f67cdb8e700 1 -- start start 2019-12-05T06:38:02.748 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:02.737+0000 7f67cdb8e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f67c808df80 0x7f67c809f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:38:02.748 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:02.737+0000 7f67cdb8e700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f67c809f830 0x7f67c814efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:38:02.748 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:02.737+0000 7f67cdb8e700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67c809fc60 con 0x7f67c80a7740 2019-12-05T06:38:02.748 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:02.737+0000 7f67cdb8e700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67c80a0b00 con 0x7f67c809f830 2019-12-05T06:38:02.749 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:02.737+0000 7f67cdb8e700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67c808e5b0 con 0x7f67c808df80 2019-12-05T06:38:02.749 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:02.737+0000 7f67bf7fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f67c808df80 0x7f67c809f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:38:02.749 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:02.737+0000 7f67bf7fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f67c808df80 0x7f67c809f2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:59110/0 (socket says 172.21.15.42:59110) 2019-12-05T06:38:02.749 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:02.737+0000 7f67bf7fe700 1 -- 172.21.15.42:0/2270766307 learned_addr learned my addr 172.21.15.42:0/2270766307 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:38:02.749 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:02.741+0000 7f67cc92c700 1 --2- 172.21.15.42:0/2270766307 >> v2:172.21.15.40:3300/0 conn(0x7f67c809f830 0x7f67c814efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:38:02.749 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:02.741+0000 7f67bf7fe700 1 -- 172.21.15.42:0/2270766307 >> v2:172.21.15.40:3300/0 conn(0x7f67c809f830 msgr2=0x7f67c814efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:38:02.749 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:02.741+0000 7f67bf7fe700 1 --2- 172.21.15.42:0/2270766307 >> v2:172.21.15.40:3300/0 conn(0x7f67c809f830 0x7f67c814efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:38:02.750 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:02.741+0000 7f67bf7fe700 1 -- 172.21.15.42:0/2270766307 >> v1:172.21.15.40:6789/0 conn(0x7f67c80a7740 legacy=0x7f67c807e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:38:02.750 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:02.741+0000 7f67bf7fe700 1 -- 172.21.15.42:0/2270766307 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67c80a0710 con 0x7f67c808df80 2019-12-05T06:38:02.750 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:02.741+0000 7f67bf7fe700 1 --2- 172.21.15.42:0/2270766307 >> v2:172.21.15.42:3300/0 conn(0x7f67c808df80 0x7f67c809f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f67b0002f30 tx=0x7f67b000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:38:02.750 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:02.741+0000 7f67bf7fe700 1 --2- 172.21.15.42:0/2270766307 >> v2:172.21.15.42:3300/0 conn(0x7f67c808df80 0x7f67c809f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f67b0002f30 tx=0x7f67b000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:38:02.751 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:02.741+0000 7f67bf7fe700 1 --2- 172.21.15.42:0/2270766307 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f67c808df80 0x7f67c809f2e0 secure :-1 s=READY pgs=2886 cs=0 l=1 rx=0x7f67b0002f30 tx=0x7f67b000fd30).ready entity=mon.1 client_cookie=db73f92a480a3761 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:38:02.751 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:02.741+0000 7f67bf7fe700 1 --2- 172.21.15.42:0/2270766307 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f67c808df80 0x7f67c809f2e0 secure :-1 s=THROTTLE_DONE pgs=2886 cs=0 l=1 rx=0x7f67b0002f30 tx=0x7f67b000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:38:02.751 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:02.741+0000 7f67bf7fe700 1 --2- 172.21.15.42:0/2270766307 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f67c808df80 0x7f67c809f2e0 secure :-1 s=THROTTLE_DONE pgs=2886 cs=0 l=1 rx=0x7f67b0002f30 tx=0x7f67b000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:38:02.751 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:02.741+0000 7f67c77fe700 1 -- 172.21.15.42:0/2270766307 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f67b0008c90 con 0x7f67c808df80 2019-12-05T06:38:02.751 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:02.741+0000 7f67bf7fe700 1 --2- 172.21.15.42:0/2270766307 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f67c808df80 0x7f67c809f2e0 secure :-1 s=THROTTLE_DONE pgs=2886 cs=0 l=1 rx=0x7f67b0002f30 tx=0x7f67b000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:38:02.751 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:02.741+0000 7f67c77fe700 1 -- 172.21.15.42:0/2270766307 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67b00088b0 con 0x7f67c808df80 2019-12-05T06:38:02.751 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:02.741+0000 7f67c77fe700 1 -- 172.21.15.42:0/2270766307 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f67b0011ad0 con 0x7f67c808df80 2019-12-05T06:38:02.752 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:02.741+0000 7f67cdb8e700 1 -- 172.21.15.42:0/2270766307 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f67c808df80 msgr2=0x7f67c809f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:38:02.752 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:02.741+0000 7f67cdb8e700 1 --2- 172.21.15.42:0/2270766307 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f67c808df80 0x7f67c809f2e0 secure :-1 s=READY pgs=2886 cs=0 l=1 rx=0x7f67b0002f30 tx=0x7f67b000fd30).stop 2019-12-05T06:38:02.752 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:02.741+0000 7f67cdb8e700 1 -- 172.21.15.42:0/2270766307 shutdown_connections 2019-12-05T06:38:02.752 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:02.741+0000 7f67cdb8e700 1 --2- 172.21.15.42:0/2270766307 >> v2:172.21.15.40:3300/0 conn(0x7f67c809f830 0x7f67c814efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:38:02.752 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:02.741+0000 7f67cdb8e700 1 --2- 172.21.15.42:0/2270766307 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f67c808df80 0x7f67c809f2e0 unknown :-1 s=CLOSED pgs=2886 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:38:02.752 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:02.741+0000 7f67cdb8e700 1 -- 172.21.15.42:0/2270766307 shutdown_connections 2019-12-05T06:38:02.790 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:02.741+0000 7f67cdb8e700 1 -- 172.21.15.42:0/2270766307 wait complete. 2019-12-05T06:38:02.791 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:02.741+0000 7f67cdb8e700 1 -- 172.21.15.42:0/2270766307 >> 172.21.15.42:0/2270766307 conn(0x7f67c80aabe0 msgr2=0x7f67c80a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:38:02.796 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:02.745+0000 7f67cdb8e700 1 Processor -- start 2019-12-05T06:38:02.796 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:02.745+0000 7f67cdb8e700 1 -- start start 2019-12-05T06:38:02.796 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:02.745+0000 7f67cdb8e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f67c808eeb0 0x7f67c8065470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:38:02.796 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:02.745+0000 7f67cdb8e700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f67c81267f0 0x7f67c8063eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:38:02.796 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:02.745+0000 7f67cdb8e700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67c80a4240 con 0x7f67c80a3e10 2019-12-05T06:38:02.797 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:02.745+0000 7f67cdb8e700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67c8065d40 con 0x7f67c81267f0 2019-12-05T06:38:02.797 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:02.745+0000 7f67cdb8e700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67c8065aa0 con 0x7f67c808eeb0 2019-12-05T06:38:02.797 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:02.745+0000 7f67c7fff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f67c808eeb0 0x7f67c8065470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:38:02.797 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:02.745+0000 7f67c7fff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f67c808eeb0 0x7f67c8065470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:59114/0 (socket says 172.21.15.42:59114) 2019-12-05T06:38:02.797 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:02.745+0000 7f67c7fff700 1 -- 172.21.15.42:0/3690884851 learned_addr learned my addr 172.21.15.42:0/3690884851 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:38:02.797 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:02.745+0000 7f67bf7fe700 1 --2- 172.21.15.42:0/3690884851 >> v2:172.21.15.40:3300/0 conn(0x7f67c81267f0 0x7f67c8063eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:38:02.797 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:02.745+0000 7f67c7fff700 1 -- 172.21.15.42:0/3690884851 >> v2:172.21.15.40:3300/0 conn(0x7f67c81267f0 msgr2=0x7f67c8063eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:38:02.798 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:02.745+0000 7f67c7fff700 1 --2- 172.21.15.42:0/3690884851 >> v2:172.21.15.40:3300/0 conn(0x7f67c81267f0 0x7f67c8063eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:38:02.798 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:02.745+0000 7f67c7fff700 1 -- 172.21.15.42:0/3690884851 >> v1:172.21.15.40:6789/0 conn(0x7f67c80a3e10 legacy=0x7f67c8127070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:38:02.798 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:02.745+0000 7f67c7fff700 1 -- 172.21.15.42:0/3690884851 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67c8132210 con 0x7f67c808eeb0 2019-12-05T06:38:02.798 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:02.745+0000 7f67c7fff700 1 --2- 172.21.15.42:0/3690884851 >> v2:172.21.15.42:3300/0 conn(0x7f67c808eeb0 0x7f67c8065470 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f67b8000e90 tx=0x7f67b80109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:38:02.798 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:02.745+0000 7f67c7fff700 1 --2- 172.21.15.42:0/3690884851 >> v2:172.21.15.42:3300/0 conn(0x7f67c808eeb0 0x7f67c8065470 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f67b8000e90 tx=0x7f67b80109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:38:02.798 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:02.745+0000 7f67c7fff700 1 --2- 172.21.15.42:0/3690884851 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f67c808eeb0 0x7f67c8065470 secure :-1 s=READY pgs=2887 cs=0 l=1 rx=0x7f67b8000e90 tx=0x7f67b80109a0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:38:02.798 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:02.749+0000 7f67c7fff700 1 --2- 172.21.15.42:0/3690884851 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f67c808eeb0 0x7f67c8065470 secure :-1 s=THROTTLE_DONE pgs=2887 cs=0 l=1 rx=0x7f67b8000e90 tx=0x7f67b80109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:38:02.799 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:02.749+0000 7f67c7fff700 1 --2- 172.21.15.42:0/3690884851 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f67c808eeb0 0x7f67c8065470 secure :-1 s=THROTTLE_DONE pgs=2887 cs=0 l=1 rx=0x7f67b8000e90 tx=0x7f67b80109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:38:02.799 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:02.749+0000 7f67c57fa700 1 -- 172.21.15.42:0/3690884851 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f67b800caa0 con 0x7f67c808eeb0 2019-12-05T06:38:02.799 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:02.749+0000 7f67c7fff700 1 --2- 172.21.15.42:0/3690884851 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f67c808eeb0 0x7f67c8065470 secure :-1 s=THROTTLE_DONE pgs=2887 cs=0 l=1 rx=0x7f67b8000e90 tx=0x7f67b80109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:38:02.799 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:02.749+0000 7f67c57fa700 1 -- 172.21.15.42:0/3690884851 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67b800c6c0 con 0x7f67c808eeb0 2019-12-05T06:38:02.799 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:02.749+0000 7f67cdb8e700 1 -- 172.21.15.42:0/3690884851 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67c8131200 con 0x7f67c808eeb0 2019-12-05T06:38:02.799 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:02.749+0000 7f67c57fa700 1 -- 172.21.15.42:0/3690884851 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f67b8012210 con 0x7f67c808eeb0 2019-12-05T06:38:02.799 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:02.749+0000 7f67cdb8e700 1 -- 172.21.15.42:0/3690884851 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67c8131730 con 0x7f67c808eeb0 2019-12-05T06:38:02.800 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:02.749+0000 7f67c7fff700 1 --2- 172.21.15.42:0/3690884851 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f67c808eeb0 0x7f67c8065470 secure :-1 s=THROTTLE_DONE pgs=2887 cs=0 l=1 rx=0x7f67b8000e90 tx=0x7f67b80109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:38:02.800 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:02.749+0000 7f67c57fa700 1 -- 172.21.15.42:0/3690884851 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f67b8012520 con 0x7f67c808eeb0 2019-12-05T06:38:02.800 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:02.749+0000 7f67c7fff700 1 --2- 172.21.15.42:0/3690884851 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f67c808eeb0 0x7f67c8065470 secure :-1 s=THROTTLE_DONE pgs=2887 cs=0 l=1 rx=0x7f67b8000e90 tx=0x7f67b80109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:38:02.800 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:02.749+0000 7f67c57fa700 1 --2- 172.21.15.42:0/3690884851 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f67ac03ad20 0x7f67ac03d230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:38:02.800 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:02.749+0000 7f67c57fa700 1 -- 172.21.15.42:0/3690884851 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 11449+0+0 (secure 0 0 0) 0x7f67b804b2a0 con 0x7f67c808eeb0 2019-12-05T06:38:02.800 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:02.749+0000 7f67bf7fe700 1 --2- 172.21.15.42:0/3690884851 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f67ac03ad20 0x7f67ac03d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:38:02.800 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:02.749+0000 7f67bf7fe700 1 --2- 172.21.15.42:0/3690884851 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f67ac03ad20 0x7f67ac03d230 crc :-1 s=READY pgs=1213 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:38:02.801 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:02.753+0000 7f67cdb8e700 1 -- 172.21.15.42:0/3690884851 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67a0003200 con 0x7f67c808eeb0 2019-12-05T06:38:02.801 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:02.761+0000 7f67c7fff700 1 --2- 172.21.15.42:0/3690884851 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f67c808eeb0 0x7f67c8065470 secure :-1 s=THROTTLE_DONE pgs=2887 cs=0 l=1 rx=0x7f67b8000e90 tx=0x7f67b80109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:38:02.801 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:02.761+0000 7f67c57fa700 1 -- 172.21.15.42:0/3690884851 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f67b8032320 con 0x7f67c808eeb0 2019-12-05T06:38:02.952 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:02.945+0000 7f67cdb8e700 1 -- 172.21.15.42:0/3690884851 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f67a0003630 con 0x7f67c808eeb0 2019-12-05T06:38:02.953 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:02.949+0000 7f67c7fff700 1 --2- 172.21.15.42:0/3690884851 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f67c808eeb0 0x7f67c8065470 secure :-1 s=THROTTLE_DONE pgs=2887 cs=0 l=1 rx=0x7f67b8000e90 tx=0x7f67b80109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:38:02.953 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:02.949+0000 7f67c57fa700 1 -- 172.21.15.42:0/3690884851 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 347 v347) v1 ==== 95+0+1776 (secure 0 0 0) 0x7f67b8032580 con 0x7f67c808eeb0 2019-12-05T06:38:02.953 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 347 2019-12-05T06:38:02.953 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:38:02.953 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":347,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":347,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:38:00.190088+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11887},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11887":{"gid":11887,"name":"a","rank":0,"incarnation":344,"state":"up:active","state_seq":37,"addr":"172.21.15.42:6835/958796360","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":958796360},{"type":"v1","addr":"172.21.15.42:6835","nonce":958796360}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:38:02.956 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:02.949+0000 7f67cdb8e700 1 -- 172.21.15.42:0/3690884851 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f67ac03ad20 msgr2=0x7f67ac03d230 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:38:02.956 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:02.949+0000 7f67cdb8e700 1 --2- 172.21.15.42:0/3690884851 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f67ac03ad20 0x7f67ac03d230 crc :-1 s=READY pgs=1213 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:38:02.956 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:02.949+0000 7f67cdb8e700 1 -- 172.21.15.42:0/3690884851 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f67c808eeb0 msgr2=0x7f67c8065470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:38:02.957 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:02.949+0000 7f67cdb8e700 1 --2- 172.21.15.42:0/3690884851 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f67c808eeb0 0x7f67c8065470 secure :-1 s=READY pgs=2887 cs=0 l=1 rx=0x7f67b8000e90 tx=0x7f67b80109a0).stop 2019-12-05T06:38:02.957 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:02.949+0000 7f67cdb8e700 1 -- 172.21.15.42:0/3690884851 shutdown_connections 2019-12-05T06:38:02.957 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:02.949+0000 7f67cdb8e700 1 --2- 172.21.15.42:0/3690884851 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f67ac03ad20 0x7f67ac03d230 unknown :-1 s=CLOSED pgs=1213 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:38:02.957 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:02.949+0000 7f67cdb8e700 1 --2- 172.21.15.42:0/3690884851 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f67c808eeb0 0x7f67c8065470 unknown :-1 s=CLOSED pgs=2887 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:38:02.957 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:02.949+0000 7f67cdb8e700 1 --2- 172.21.15.42:0/3690884851 >> v2:172.21.15.40:3300/0 conn(0x7f67c81267f0 0x7f67c8063eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:38:02.957 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:02.949+0000 7f67cdb8e700 1 -- 172.21.15.42:0/3690884851 shutdown_connections 2019-12-05T06:38:02.958 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:02.949+0000 7f67cdb8e700 1 -- 172.21.15.42:0/3690884851 wait complete. 2019-12-05T06:38:02.958 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:02.949+0000 7f67cdb8e700 1 -- 172.21.15.42:0/3690884851 >> 172.21.15.42:0/3690884851 conn(0x7f67c80a83e0 msgr2=0x7f67c80a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:38:02.966 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2019-12-05T06:38:04.967 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:38:05.079 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:05.065+0000 7f65805b2700 1 Processor -- start 2019-12-05T06:38:05.079 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:05.065+0000 7f65805b2700 1 -- start start 2019-12-05T06:38:05.079 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:05.069+0000 7f65805b2700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f657808df80 0x7f657809f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:38:05.080 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:05.069+0000 7f65805b2700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f657809f830 0x7f657814efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:38:05.080 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:05.069+0000 7f65805b2700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f657809fc60 con 0x7f65780a7740 2019-12-05T06:38:05.080 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:05.069+0000 7f65805b2700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65780a0b00 con 0x7f657808df80 2019-12-05T06:38:05.080 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:05.069+0000 7f65805b2700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f657808e5b0 con 0x7f657809f830 2019-12-05T06:38:05.080 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:05.069+0000 7f657eb4f700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f65780a7740 0x7f657807e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:59170/0 (socket says 172.21.15.42:59170) 2019-12-05T06:38:05.081 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:05.069+0000 7f657eb4f700 1 -- 172.21.15.42:0/4055123684 learned_addr learned my addr 172.21.15.42:0/4055123684 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:38:05.081 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:05.069+0000 7f657e34e700 1 --2- 172.21.15.42:0/4055123684 >> v2:172.21.15.40:3300/0 conn(0x7f657808df80 0x7f657809f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:38:05.081 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:05.069+0000 7f657f350700 1 --2- 172.21.15.42:0/4055123684 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f657809f830 0x7f657814efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:38:05.081 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:05.069+0000 7f657db4d700 1 -- 172.21.15.42:0/4055123684 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3507805314 0 0) 0x7f657809fc60 con 0x7f65780a7740 2019-12-05T06:38:05.081 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:05.069+0000 7f657db4d700 1 -- 172.21.15.42:0/4055123684 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65640030b0 con 0x7f65780a7740 2019-12-05T06:38:05.081 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:05.069+0000 7f657db4d700 1 -- 172.21.15.42:0/4055123684 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1809056255 0 0) 0x7f65640030b0 con 0x7f65780a7740 2019-12-05T06:38:05.082 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:05.069+0000 7f657db4d700 1 -- 172.21.15.42:0/4055123684 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f657809f830 msgr2=0x7f657814efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:38:05.082 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:05.069+0000 7f657db4d700 1 --2- 172.21.15.42:0/4055123684 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f657809f830 0x7f657814efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:38:05.082 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:05.069+0000 7f657db4d700 1 -- 172.21.15.42:0/4055123684 >> v2:172.21.15.40:3300/0 conn(0x7f657808df80 msgr2=0x7f657809f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:38:05.082 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:05.069+0000 7f657db4d700 1 --2- 172.21.15.42:0/4055123684 >> v2:172.21.15.40:3300/0 conn(0x7f657808df80 0x7f657809f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:38:05.082 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:05.069+0000 7f657db4d700 1 -- 172.21.15.42:0/4055123684 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65780a0710 con 0x7f65780a7740 2019-12-05T06:38:05.082 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:05.069+0000 7f657db4d700 1 -- 172.21.15.42:0/4055123684 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7f656c001750 con 0x7f65780a7740 2019-12-05T06:38:05.082 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:05.069+0000 7f657db4d700 1 -- 172.21.15.42:0/4055123684 >> v1:172.21.15.42:6789/0 conn(0x7f65780a7740 legacy=0x7f657807e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:38:05.083 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:05.069+0000 7f657db4d700 1 --2- 172.21.15.42:0/4055123684 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f65640036e0 0x7f6564003b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:38:05.083 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:05.069+0000 7f657db4d700 1 --2- 172.21.15.42:0/4055123684 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6564004420 0x7f6564004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:38:05.083 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:05.069+0000 7f657db4d700 1 --2- 172.21.15.42:0/4055123684 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6564004da0 0x7f656404f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:38:05.083 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:05.069+0000 7f657db4d700 1 -- 172.21.15.42:0/4055123684 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65780a0b00 con 0x7f6564004da0 2019-12-05T06:38:05.083 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:05.069+0000 7f657db4d700 1 -- 172.21.15.42:0/4055123684 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f657808e5b0 con 0x7f6564004420 2019-12-05T06:38:05.083 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:05.069+0000 7f657db4d700 1 -- 172.21.15.42:0/4055123684 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f656404fbd0 con 0x7f65640036e0 2019-12-05T06:38:05.083 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:05.069+0000 7f657e34e700 1 --2- 172.21.15.42:0/4055123684 >> v2:172.21.15.40:3300/0 conn(0x7f657808df80 0x7f657809f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:38:05.084 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:05.069+0000 7f657e34e700 1 --2- 172.21.15.42:0/4055123684 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6564004420 0x7f6564004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:38:05.084 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:05.069+0000 7f657eb4f700 1 --2- 172.21.15.42:0/4055123684 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f65640036e0 0x7f6564003b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:38:05.084 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:05.069+0000 7f657f350700 1 --2- 172.21.15.42:0/4055123684 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6564004da0 0x7f656404f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:38:05.084 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:05.069+0000 7f657e34e700 1 -- 172.21.15.42:0/4055123684 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f65640036e0 msgr2=0x7f6564003b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:38:05.084 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:05.069+0000 7f657e34e700 1 --2- 172.21.15.42:0/4055123684 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f65640036e0 0x7f6564003b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:38:05.084 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:05.069+0000 7f657e34e700 1 -- 172.21.15.42:0/4055123684 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6564004da0 msgr2=0x7f656404f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:38:05.085 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:05.069+0000 7f657e34e700 1 --2- 172.21.15.42:0/4055123684 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6564004da0 0x7f656404f680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:38:05.085 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:05.069+0000 7f657e34e700 1 -- 172.21.15.42:0/4055123684 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f656404fd90 con 0x7f6564004420 2019-12-05T06:38:05.085 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:05.069+0000 7f657e34e700 1 --2- 172.21.15.42:0/4055123684 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6564004420 0x7f6564004850 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f6574002f30 tx=0x7f657400bed0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:38:05.085 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:05.069+0000 7f657e34e700 1 --2- 172.21.15.42:0/4055123684 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6564004420 0x7f6564004850 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f6574002f30 tx=0x7f657400bed0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:38:05.085 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:05.069+0000 7f657e34e700 1 --2- 172.21.15.42:0/4055123684 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6564004420 0x7f6564004850 secure :-1 s=READY pgs=2889 cs=0 l=1 rx=0x7f6574002f30 tx=0x7f657400bed0).ready entity=mon.1 client_cookie=1df87521cdc2a239 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:38:05.085 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:05.069+0000 7f657e34e700 1 --2- 172.21.15.42:0/4055123684 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6564004420 0x7f6564004850 secure :-1 s=THROTTLE_DONE pgs=2889 cs=0 l=1 rx=0x7f6574002f30 tx=0x7f657400bed0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:38:05.086 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:05.069+0000 7f657e34e700 1 --2- 172.21.15.42:0/4055123684 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6564004420 0x7f6564004850 secure :-1 s=THROTTLE_DONE pgs=2889 cs=0 l=1 rx=0x7f6574002f30 tx=0x7f657400bed0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:38:05.086 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:05.069+0000 7f657db4d700 1 -- 172.21.15.42:0/4055123684 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f657400e370 con 0x7f6564004420 2019-12-05T06:38:05.086 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:05.069+0000 7f657e34e700 1 --2- 172.21.15.42:0/4055123684 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6564004420 0x7f6564004850 secure :-1 s=THROTTLE_DONE pgs=2889 cs=0 l=1 rx=0x7f6574002f30 tx=0x7f657400bed0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:38:05.086 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:05.069+0000 7f657db4d700 1 -- 172.21.15.42:0/4055123684 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6574011b00 con 0x7f6564004420 2019-12-05T06:38:05.086 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:05.069+0000 7f657db4d700 1 -- 172.21.15.42:0/4055123684 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f657400e950 con 0x7f6564004420 2019-12-05T06:38:05.086 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:05.069+0000 7f65805b2700 1 -- 172.21.15.42:0/4055123684 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6564004420 msgr2=0x7f6564004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:38:05.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:05.069+0000 7f65805b2700 1 --2- 172.21.15.42:0/4055123684 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6564004420 0x7f6564004850 secure :-1 s=READY pgs=2889 cs=0 l=1 rx=0x7f6574002f30 tx=0x7f657400bed0).stop 2019-12-05T06:38:05.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:05.069+0000 7f657f350700 1 -- 172.21.15.42:0/4055123684 reap_dead start 2019-12-05T06:38:05.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:05.073+0000 7f65805b2700 1 -- 172.21.15.42:0/4055123684 shutdown_connections 2019-12-05T06:38:05.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:05.073+0000 7f65805b2700 1 -- 172.21.15.42:0/4055123684 shutdown_connections 2019-12-05T06:38:05.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:05.073+0000 7f65805b2700 1 -- 172.21.15.42:0/4055123684 wait complete. 2019-12-05T06:38:05.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:05.073+0000 7f65805b2700 1 -- 172.21.15.42:0/4055123684 >> 172.21.15.42:0/4055123684 conn(0x7f65780aabe0 msgr2=0x7f65780a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:38:05.093 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:05.073+0000 7f65805b2700 1 Processor -- start 2019-12-05T06:38:05.093 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:05.073+0000 7f65805b2700 1 -- start start 2019-12-05T06:38:05.094 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:05.077+0000 7f65805b2700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f657811bea0 0x7f6578117f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:38:05.094 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:05.077+0000 7f65805b2700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f6578135210 0x7f6578077300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:38:05.094 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:05.077+0000 7f65805b2700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6578141f70 con 0x7f65781275f0 2019-12-05T06:38:05.094 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:05.077+0000 7f65805b2700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6578117640 con 0x7f6578135210 2019-12-05T06:38:05.094 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:05.077+0000 7f65805b2700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6578118590 con 0x7f657811bea0 2019-12-05T06:38:05.094 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:05.077+0000 7f657eb4f700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f657811bea0 0x7f6578117f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:38:05.095 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:05.077+0000 7f657eb4f700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f657811bea0 0x7f6578117f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:59146/0 (socket says 172.21.15.42:59146) 2019-12-05T06:38:05.095 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:05.077+0000 7f657eb4f700 1 -- 172.21.15.42:0/3698538598 learned_addr learned my addr 172.21.15.42:0/3698538598 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:38:05.095 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:05.077+0000 7f657f350700 1 --2- 172.21.15.42:0/3698538598 >> v2:172.21.15.40:3300/0 conn(0x7f6578135210 0x7f6578077300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:38:05.095 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:05.077+0000 7f657eb4f700 1 -- 172.21.15.42:0/3698538598 >> v2:172.21.15.40:3300/0 conn(0x7f6578135210 msgr2=0x7f6578077300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:38:05.095 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:05.077+0000 7f657eb4f700 1 --2- 172.21.15.42:0/3698538598 >> v2:172.21.15.40:3300/0 conn(0x7f6578135210 0x7f6578077300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:38:05.095 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:05.077+0000 7f657eb4f700 1 -- 172.21.15.42:0/3698538598 >> v1:172.21.15.40:6789/0 conn(0x7f65781275f0 legacy=0x7f6578134a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:38:05.096 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:05.077+0000 7f657eb4f700 1 -- 172.21.15.42:0/3698538598 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f657805ae30 con 0x7f657811bea0 2019-12-05T06:38:05.096 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:05.077+0000 7f657eb4f700 1 --2- 172.21.15.42:0/3698538598 >> v2:172.21.15.42:3300/0 conn(0x7f657811bea0 0x7f6578117f70 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f656c001e80 tx=0x7f656c0082f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:38:05.096 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:05.077+0000 7f657eb4f700 1 --2- 172.21.15.42:0/3698538598 >> v2:172.21.15.42:3300/0 conn(0x7f657811bea0 0x7f6578117f70 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f656c001e80 tx=0x7f656c0082f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:38:05.096 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:05.077+0000 7f657eb4f700 1 --2- 172.21.15.42:0/3698538598 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f657811bea0 0x7f6578117f70 secure :-1 s=READY pgs=2890 cs=0 l=1 rx=0x7f656c001e80 tx=0x7f656c0082f0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:38:05.096 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:05.077+0000 7f657eb4f700 1 --2- 172.21.15.42:0/3698538598 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f657811bea0 0x7f6578117f70 secure :-1 s=THROTTLE_DONE pgs=2890 cs=0 l=1 rx=0x7f656c001e80 tx=0x7f656c0082f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:38:05.096 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:05.077+0000 7f657eb4f700 1 --2- 172.21.15.42:0/3698538598 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f657811bea0 0x7f6578117f70 secure :-1 s=THROTTLE_DONE pgs=2890 cs=0 l=1 rx=0x7f656c001e80 tx=0x7f656c0082f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:38:05.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:05.077+0000 7f656b7fe700 1 -- 172.21.15.42:0/3698538598 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f656c00dc20 con 0x7f657811bea0 2019-12-05T06:38:05.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:05.077+0000 7f657eb4f700 1 --2- 172.21.15.42:0/3698538598 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f657811bea0 0x7f6578117f70 secure :-1 s=THROTTLE_DONE pgs=2890 cs=0 l=1 rx=0x7f656c001e80 tx=0x7f656c0082f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:38:05.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:05.077+0000 7f656b7fe700 1 -- 172.21.15.42:0/3698538598 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f656c00d840 con 0x7f657811bea0 2019-12-05T06:38:05.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:05.077+0000 7f656b7fe700 1 -- 172.21.15.42:0/3698538598 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f656c012b30 con 0x7f657811bea0 2019-12-05T06:38:05.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:05.077+0000 7f65805b2700 1 -- 172.21.15.42:0/3698538598 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6578127a20 con 0x7f657811bea0 2019-12-05T06:38:05.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:05.077+0000 7f65805b2700 1 -- 172.21.15.42:0/3698538598 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6578077850 con 0x7f657811bea0 2019-12-05T06:38:05.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:05.081+0000 7f657eb4f700 1 --2- 172.21.15.42:0/3698538598 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f657811bea0 0x7f6578117f70 secure :-1 s=THROTTLE_DONE pgs=2890 cs=0 l=1 rx=0x7f656c001e80 tx=0x7f656c0082f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:38:05.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:05.081+0000 7f656b7fe700 1 -- 172.21.15.42:0/3698538598 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f656c00e070 con 0x7f657811bea0 2019-12-05T06:38:05.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:05.081+0000 7f657eb4f700 1 --2- 172.21.15.42:0/3698538598 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f657811bea0 0x7f6578117f70 secure :-1 s=THROTTLE_DONE pgs=2890 cs=0 l=1 rx=0x7f656c001e80 tx=0x7f656c0082f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:38:05.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:05.081+0000 7f656b7fe700 1 --2- 172.21.15.42:0/3698538598 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f656003ace0 0x7f656003d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:38:05.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:05.081+0000 7f656b7fe700 1 -- 172.21.15.42:0/3698538598 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 11449+0+0 (secure 0 0 0) 0x7f656c04b150 con 0x7f657811bea0 2019-12-05T06:38:05.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:05.081+0000 7f657e34e700 1 --2- 172.21.15.42:0/3698538598 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f656003ace0 0x7f656003d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:38:05.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:05.081+0000 7f657e34e700 1 --2- 172.21.15.42:0/3698538598 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f656003ace0 0x7f656003d1f0 crc :-1 s=READY pgs=1214 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:38:05.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:05.081+0000 7f65805b2700 1 -- 172.21.15.42:0/3698538598 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f657805d110 con 0x7f657811bea0 2019-12-05T06:38:05.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:05.093+0000 7f657eb4f700 1 --2- 172.21.15.42:0/3698538598 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f657811bea0 0x7f6578117f70 secure :-1 s=THROTTLE_DONE pgs=2890 cs=0 l=1 rx=0x7f656c001e80 tx=0x7f656c0082f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:38:05.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:05.093+0000 7f656b7fe700 1 -- 172.21.15.42:0/3698538598 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f656c032410 con 0x7f657811bea0 2019-12-05T06:38:05.274 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:05.261+0000 7f65805b2700 1 -- 172.21.15.42:0/3698538598 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f657805b8c0 con 0x7f657811bea0 2019-12-05T06:38:05.274 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:05.265+0000 7f657eb4f700 1 --2- 172.21.15.42:0/3698538598 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f657811bea0 0x7f6578117f70 secure :-1 s=THROTTLE_DONE pgs=2890 cs=0 l=1 rx=0x7f656c001e80 tx=0x7f656c0082f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:38:05.274 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:05.265+0000 7f656b7fe700 1 -- 172.21.15.42:0/3698538598 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 347 v347) v1 ==== 95+0+1776 (secure 0 0 0) 0x7f656c032fc0 con 0x7f657811bea0 2019-12-05T06:38:05.277 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 347 2019-12-05T06:38:05.278 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:05.265+0000 7f65805b2700 1 -- 172.21.15.42:0/3698538598 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f656003ace0 msgr2=0x7f656003d1f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:38:05.278 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:05.265+0000 7f65805b2700 1 --2- 172.21.15.42:0/3698538598 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f656003ace0 0x7f656003d1f0 crc :-1 s=READY pgs=1214 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:38:05.278 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:05.265+0000 7f65805b2700 1 -- 172.21.15.42:0/3698538598 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f657811bea0 msgr2=0x7f6578117f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:38:05.278 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:05.265+0000 7f65805b2700 1 --2- 172.21.15.42:0/3698538598 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f657811bea0 0x7f6578117f70 secure :-1 s=READY pgs=2890 cs=0 l=1 rx=0x7f656c001e80 tx=0x7f656c0082f0).stop 2019-12-05T06:38:05.278 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:05.265+0000 7f65805b2700 1 -- 172.21.15.42:0/3698538598 shutdown_connections 2019-12-05T06:38:05.278 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:05.265+0000 7f65805b2700 1 --2- 172.21.15.42:0/3698538598 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f656003ace0 0x7f656003d1f0 unknown :-1 s=CLOSED pgs=1214 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:38:05.278 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:05.265+0000 7f65805b2700 1 --2- 172.21.15.42:0/3698538598 >> v2:172.21.15.40:3300/0 conn(0x7f6578135210 0x7f6578077300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:38:05.279 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:05.265+0000 7f65805b2700 1 --2- 172.21.15.42:0/3698538598 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f657811bea0 0x7f6578117f70 unknown :-1 s=CLOSED pgs=2890 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:38:05.279 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:05.265+0000 7f65805b2700 1 -- 172.21.15.42:0/3698538598 shutdown_connections 2019-12-05T06:38:05.282 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:05.265+0000 7f65805b2700 1 -- 172.21.15.42:0/3698538598 wait complete. 2019-12-05T06:38:05.282 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:05.265+0000 7f65805b2700 1 -- 172.21.15.42:0/3698538598 >> 172.21.15.42:0/3698538598 conn(0x7f657811d010 msgr2=0x7f6578059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:38:05.282 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:38:05.283 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":347,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":347,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:38:00.190088+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11887},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11887":{"gid":11887,"name":"a","rank":0,"incarnation":344,"state":"up:active","state_seq":37,"addr":"172.21.15.42:6835/958796360","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":958796360},{"type":"v1","addr":"172.21.15.42:6835","nonce":958796360}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:38:05.286 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2019-12-05T06:38:07.288 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:38:07.387 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:07.377+0000 7fdfadee7700 1 Processor -- start 2019-12-05T06:38:07.387 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:07.377+0000 7fdfadee7700 1 -- start start 2019-12-05T06:38:07.387 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:07.377+0000 7fdfadee7700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fdfa809fae0 0x7fdfa814fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:38:07.387 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:07.377+0000 7fdfadee7700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfa80a0710 con 0x7fdfa80a7740 2019-12-05T06:38:07.387 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:07.377+0000 7fdfadee7700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfa807ef20 con 0x7fdfa808df80 2019-12-05T06:38:07.388 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:07.377+0000 7fdfadee7700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdfa80a0b00 con 0x7fdfa809fae0 2019-12-05T06:38:07.388 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:07.377+0000 7fdfa77fe700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fdfa808df80 0x7fdfa809f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:59212/0 (socket says 172.21.15.42:59212) 2019-12-05T06:38:07.388 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:07.377+0000 7fdfa77fe700 1 -- 172.21.15.42:0/1676338485 learned_addr learned my addr 172.21.15.42:0/1676338485 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:38:07.388 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:07.377+0000 7fdfacc85700 1 --2- 172.21.15.42:0/1676338485 >> v2:172.21.15.40:3300/0 conn(0x7fdfa809fae0 0x7fdfa814fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:38:07.388 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:07.377+0000 7fdfa6ffd700 1 -- 172.21.15.42:0/1676338485 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 566197172 0 0) 0x7fdfa807ef20 con 0x7fdfa808df80 2019-12-05T06:38:07.388 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:07.377+0000 7fdfa6ffd700 1 -- 172.21.15.42:0/1676338485 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf940030b0 con 0x7fdfa808df80 2019-12-05T06:38:07.388 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:07.377+0000 7fdfa6ffd700 1 -- 172.21.15.42:0/1676338485 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2547918315 0 0) 0x7fdf940030b0 con 0x7fdfa808df80 2019-12-05T06:38:07.389 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:07.377+0000 7fdfa6ffd700 1 -- 172.21.15.42:0/1676338485 >> v2:172.21.15.40:3300/0 conn(0x7fdfa809fae0 msgr2=0x7fdfa814fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:38:07.389 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:07.377+0000 7fdfa6ffd700 1 --2- 172.21.15.42:0/1676338485 >> v2:172.21.15.40:3300/0 conn(0x7fdfa809fae0 0x7fdfa814fff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:38:07.389 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:07.377+0000 7fdfa6ffd700 1 -- 172.21.15.42:0/1676338485 >> v1:172.21.15.40:6789/0 conn(0x7fdfa80a7740 legacy=0x7fdfa807e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:38:07.389 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:07.377+0000 7fdfa6ffd700 1 -- 172.21.15.42:0/1676338485 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfa80a9ba0 con 0x7fdfa808df80 2019-12-05T06:38:07.389 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:07.377+0000 7fdfa6ffd700 1 -- 172.21.15.42:0/1676338485 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7fdf90001750 con 0x7fdfa808df80 2019-12-05T06:38:07.389 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:07.377+0000 7fdfa6ffd700 1 -- 172.21.15.42:0/1676338485 >> v1:172.21.15.42:6789/0 conn(0x7fdfa808df80 legacy=0x7fdfa809f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:38:07.389 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:07.377+0000 7fdfa6ffd700 1 --2- 172.21.15.42:0/1676338485 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdf94003730 0x7fdf94003ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:38:07.390 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:07.377+0000 7fdfa6ffd700 1 --2- 172.21.15.42:0/1676338485 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf94004420 0x7fdf94004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:38:07.390 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:07.377+0000 7fdfa6ffd700 1 --2- 172.21.15.42:0/1676338485 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdf94004da0 0x7fdf9400b670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:38:07.390 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:07.377+0000 7fdfa6ffd700 1 -- 172.21.15.42:0/1676338485 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfa80a0b00 con 0x7fdf94004420 2019-12-05T06:38:07.390 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:07.377+0000 7fdfa6ffd700 1 -- 172.21.15.42:0/1676338485 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf9404fc90 con 0x7fdf94004da0 2019-12-05T06:38:07.390 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:07.377+0000 7fdfa6ffd700 1 -- 172.21.15.42:0/1676338485 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf9404fed0 con 0x7fdf94003730 2019-12-05T06:38:07.390 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:07.377+0000 7fdfacc85700 1 --2- 172.21.15.42:0/1676338485 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdf94004da0 0x7fdf9400b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:38:07.397 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:07.377+0000 7fdfa77fe700 1 --2- 172.21.15.42:0/1676338485 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf94004420 0x7fdf94004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:38:07.397 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:07.377+0000 7fdfa7fff700 1 --2- 172.21.15.42:0/1676338485 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdf94003730 0x7fdf94003ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:38:07.397 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:07.377+0000 7fdfacc85700 1 -- 172.21.15.42:0/1676338485 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdf94003730 msgr2=0x7fdf94003ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:38:07.398 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:07.377+0000 7fdfacc85700 1 --2- 172.21.15.42:0/1676338485 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdf94003730 0x7fdf94003ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:38:07.398 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:07.377+0000 7fdfacc85700 1 -- 172.21.15.42:0/1676338485 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf94004420 msgr2=0x7fdf94004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:38:07.398 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:07.377+0000 7fdfacc85700 1 --2- 172.21.15.42:0/1676338485 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf94004420 0x7fdf94004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:38:07.398 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:07.377+0000 7fdfacc85700 1 -- 172.21.15.42:0/1676338485 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf940501d0 con 0x7fdf94004da0 2019-12-05T06:38:07.398 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:07.377+0000 7fdfa7fff700 1 --2- 172.21.15.42:0/1676338485 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdf94003730 0x7fdf94003ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:38:07.399 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:07.377+0000 7fdfacc85700 1 --2- 172.21.15.42:0/1676338485 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdf94004da0 0x7fdf9400b670 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fdfa0002f30 tx=0x7fdfa000ffa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:38:07.399 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:07.377+0000 7fdfacc85700 1 -- 172.21.15.42:0/1676338485 reap_dead start 2019-12-05T06:38:07.399 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:07.377+0000 7fdfacc85700 1 --2- 172.21.15.42:0/1676338485 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdf94004da0 0x7fdf9400b670 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fdfa0002f30 tx=0x7fdfa000ffa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:38:07.399 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:07.377+0000 7fdfacc85700 1 --2- 172.21.15.42:0/1676338485 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdf94004da0 0x7fdf9400b670 secure :-1 s=READY pgs=2892 cs=0 l=1 rx=0x7fdfa0002f30 tx=0x7fdfa000ffa0).ready entity=mon.1 client_cookie=876d359087fcc3ba server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:38:07.400 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:07.377+0000 7fdfacc85700 1 --2- 172.21.15.42:0/1676338485 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdf94004da0 0x7fdf9400b670 secure :-1 s=THROTTLE_DONE pgs=2892 cs=0 l=1 rx=0x7fdfa0002f30 tx=0x7fdfa000ffa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:38:07.400 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:07.377+0000 7fdfacc85700 1 --2- 172.21.15.42:0/1676338485 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdf94004da0 0x7fdf9400b670 secure :-1 s=THROTTLE_DONE pgs=2892 cs=0 l=1 rx=0x7fdfa0002f30 tx=0x7fdfa000ffa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:38:07.400 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:07.377+0000 7fdfa6ffd700 1 -- 172.21.15.42:0/1676338485 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdfa0010cd0 con 0x7fdf94004da0 2019-12-05T06:38:07.400 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:07.377+0000 7fdfacc85700 1 --2- 172.21.15.42:0/1676338485 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdf94004da0 0x7fdf9400b670 secure :-1 s=THROTTLE_DONE pgs=2892 cs=0 l=1 rx=0x7fdfa0002f30 tx=0x7fdfa000ffa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:38:07.400 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:07.377+0000 7fdfa6ffd700 1 -- 172.21.15.42:0/1676338485 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfa00108f0 con 0x7fdf94004da0 2019-12-05T06:38:07.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:07.377+0000 7fdfa6ffd700 1 -- 172.21.15.42:0/1676338485 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdfa0008bb0 con 0x7fdf94004da0 2019-12-05T06:38:07.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:07.377+0000 7fdfadee7700 1 -- 172.21.15.42:0/1676338485 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdf94004da0 msgr2=0x7fdf9400b670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:38:07.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:07.377+0000 7fdfadee7700 1 --2- 172.21.15.42:0/1676338485 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdf94004da0 0x7fdf9400b670 secure :-1 s=READY pgs=2892 cs=0 l=1 rx=0x7fdfa0002f30 tx=0x7fdfa000ffa0).stop 2019-12-05T06:38:07.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:07.381+0000 7fdfadee7700 1 -- 172.21.15.42:0/1676338485 shutdown_connections 2019-12-05T06:38:07.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:07.381+0000 7fdfadee7700 1 --2- 172.21.15.42:0/1676338485 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdf94004da0 0x7fdf9400b670 unknown :-1 s=CLOSED pgs=2892 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:38:07.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:07.381+0000 7fdfadee7700 1 -- 172.21.15.42:0/1676338485 shutdown_connections 2019-12-05T06:38:07.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:07.381+0000 7fdfadee7700 1 -- 172.21.15.42:0/1676338485 wait complete. 2019-12-05T06:38:07.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:07.381+0000 7fdfadee7700 1 -- 172.21.15.42:0/1676338485 >> 172.21.15.42:0/1676338485 conn(0x7fdfa80aabe0 msgr2=0x7fdfa80a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:38:07.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:07.381+0000 7fdfadee7700 1 Processor -- start 2019-12-05T06:38:07.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:07.381+0000 7fdfadee7700 1 -- start start 2019-12-05T06:38:07.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:07.381+0000 7fdfadee7700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fdfa80a2760 0x7fdfa811a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:38:07.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:07.381+0000 7fdfadee7700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfa80a2b90 con 0x7fdfa80702c0 2019-12-05T06:38:07.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:07.381+0000 7fdfadee7700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfa811bcb0 con 0x7fdfa8126660 2019-12-05T06:38:07.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:07.381+0000 7fdfadee7700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdfa80a24c0 con 0x7fdfa80a2760 2019-12-05T06:38:07.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:07.381+0000 7fdfa7fff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fdfa8126660 0x7fdfa811b3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:59222/0 (socket says 172.21.15.42:59222) 2019-12-05T06:38:07.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:07.381+0000 7fdfa7fff700 1 -- 172.21.15.42:0/1668855972 learned_addr learned my addr 172.21.15.42:0/1668855972 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:38:07.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:07.381+0000 7fdfa77fe700 1 --2- 172.21.15.42:0/1668855972 >> v2:172.21.15.40:3300/0 conn(0x7fdfa80a2760 0x7fdfa811a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:38:07.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:07.385+0000 7fdfa4ff9700 1 -- 172.21.15.42:0/1668855972 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2477564226 0 0) 0x7fdfa811bcb0 con 0x7fdfa8126660 2019-12-05T06:38:07.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:07.385+0000 7fdfa4ff9700 1 -- 172.21.15.42:0/1668855972 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf8c0030b0 con 0x7fdfa8126660 2019-12-05T06:38:07.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:07.385+0000 7fdfa77fe700 1 -- 172.21.15.42:0/1668855972 >> v1:172.21.15.42:6789/0 conn(0x7fdfa8126660 legacy=0x7fdfa811b3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:38:07.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:07.385+0000 7fdfa77fe700 1 -- 172.21.15.42:0/1668855972 >> v1:172.21.15.40:6789/0 conn(0x7fdfa80702c0 legacy=0x7fdfa80706f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:38:07.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:07.385+0000 7fdfa4ff9700 1 -- 172.21.15.42:0/1668855972 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3927448164 0 0) 0x7fdfa80a2b90 con 0x7fdfa80702c0 2019-12-05T06:38:07.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:07.385+0000 7fdfa77fe700 1 -- 172.21.15.42:0/1668855972 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfa8070ef0 con 0x7fdfa80a2760 2019-12-05T06:38:07.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:07.385+0000 7fdfa77fe700 1 --2- 172.21.15.42:0/1668855972 >> v2:172.21.15.40:3300/0 conn(0x7fdfa80a2760 0x7fdfa811a1b0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fdf90000fe0 tx=0x7fdf90010910).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:38:07.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:07.385+0000 7fdfa77fe700 1 --2- 172.21.15.42:0/1668855972 >> v2:172.21.15.40:3300/0 conn(0x7fdfa80a2760 0x7fdfa811a1b0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fdf90000fe0 tx=0x7fdf90010910).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:38:07.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:07.385+0000 7fdfa77fe700 1 --2- 172.21.15.42:0/1668855972 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdfa80a2760 0x7fdfa811a1b0 secure :-1 s=READY pgs=2195 cs=0 l=1 rx=0x7fdf90000fe0 tx=0x7fdf90010910).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:38:07.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:07.385+0000 7fdfa77fe700 1 --2- 172.21.15.42:0/1668855972 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdfa80a2760 0x7fdfa811a1b0 secure :-1 s=THROTTLE_DONE pgs=2195 cs=0 l=1 rx=0x7fdf90000fe0 tx=0x7fdf90010910).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:38:07.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:07.385+0000 7fdfa4ff9700 1 -- 172.21.15.42:0/1668855972 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdf90004370 con 0x7fdfa80a2760 2019-12-05T06:38:07.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:07.385+0000 7fdfa77fe700 1 --2- 172.21.15.42:0/1668855972 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdfa80a2760 0x7fdfa811a1b0 secure :-1 s=THROTTLE_DONE pgs=2195 cs=0 l=1 rx=0x7fdf90000fe0 tx=0x7fdf90010910).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:38:07.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:07.385+0000 7fdfa77fe700 1 --2- 172.21.15.42:0/1668855972 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdfa80a2760 0x7fdfa811a1b0 secure :-1 s=THROTTLE_DONE pgs=2195 cs=0 l=1 rx=0x7fdf90000fe0 tx=0x7fdf90010910).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:38:07.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:07.385+0000 7fdfa4ff9700 1 -- 172.21.15.42:0/1668855972 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf9000cb80 con 0x7fdfa80a2760 2019-12-05T06:38:07.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:07.385+0000 7fdfadee7700 1 -- 172.21.15.42:0/1668855972 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfa8078540 con 0x7fdfa80a2760 2019-12-05T06:38:07.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:07.385+0000 7fdfa4ff9700 1 -- 172.21.15.42:0/1668855972 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdf90004950 con 0x7fdfa80a2760 2019-12-05T06:38:07.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:07.385+0000 7fdfadee7700 1 -- 172.21.15.42:0/1668855972 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfa8078a40 con 0x7fdfa80a2760 2019-12-05T06:38:07.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:07.385+0000 7fdfa77fe700 1 --2- 172.21.15.42:0/1668855972 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdfa80a2760 0x7fdfa811a1b0 secure :-1 s=THROTTLE_DONE pgs=2195 cs=0 l=1 rx=0x7fdf90000fe0 tx=0x7fdf90010910).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:38:07.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:07.385+0000 7fdfa4ff9700 1 -- 172.21.15.42:0/1668855972 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fdf90004c20 con 0x7fdfa80a2760 2019-12-05T06:38:07.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:07.385+0000 7fdfa77fe700 1 --2- 172.21.15.42:0/1668855972 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdfa80a2760 0x7fdfa811a1b0 secure :-1 s=THROTTLE_DONE pgs=2195 cs=0 l=1 rx=0x7fdf90000fe0 tx=0x7fdf90010910).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:38:07.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:07.385+0000 7fdfa4ff9700 1 --2- 172.21.15.42:0/1668855972 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fdf8c03b210 0x7fdf8c03d720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:38:07.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:07.385+0000 7fdfa4ff9700 1 -- 172.21.15.42:0/1668855972 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 11449+0+0 (secure 0 0 0) 0x7fdf9004aaa0 con 0x7fdfa80a2760 2019-12-05T06:38:07.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:07.385+0000 7fdfa7fff700 1 --2- 172.21.15.42:0/1668855972 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fdf8c03b210 0x7fdf8c03d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:38:07.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:07.385+0000 7fdfa7fff700 1 --2- 172.21.15.42:0/1668855972 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fdf8c03b210 0x7fdf8c03d720 crc :-1 s=READY pgs=1216 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:38:07.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:07.393+0000 7fdfadee7700 1 -- 172.21.15.42:0/1668855972 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfa805d110 con 0x7fdfa80a2760 2019-12-05T06:38:07.411 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:07.401+0000 7fdfa77fe700 1 --2- 172.21.15.42:0/1668855972 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdfa80a2760 0x7fdfa811a1b0 secure :-1 s=THROTTLE_DONE pgs=2195 cs=0 l=1 rx=0x7fdf90000fe0 tx=0x7fdf90010910).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:38:07.411 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:07.401+0000 7fdfa4ff9700 1 -- 172.21.15.42:0/1668855972 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fdf90031b20 con 0x7fdfa80a2760 2019-12-05T06:38:07.578 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:07.573+0000 7fdfadee7700 1 -- 172.21.15.42:0/1668855972 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdfa805b8c0 con 0x7fdfa80a2760 2019-12-05T06:38:07.578 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:07.573+0000 7fdfa77fe700 1 --2- 172.21.15.42:0/1668855972 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdfa80a2760 0x7fdfa811a1b0 secure :-1 s=THROTTLE_DONE pgs=2195 cs=0 l=1 rx=0x7fdf90000fe0 tx=0x7fdf90010910).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:38:07.579 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:07.573+0000 7fdfa4ff9700 1 -- 172.21.15.42:0/1668855972 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 348 v348) v1 ==== 95+0+2113 (secure 0 0 0) 0x7fdf90031d80 con 0x7fdfa80a2760 2019-12-05T06:38:07.581 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 348 2019-12-05T06:38:07.581 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:38:07.581 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":348,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":8009,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/1777669201","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1777669201},{"type":"v1","addr":"172.21.15.40:6833","nonce":1777669201}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":348}],"filesystems":[{"mdsmap":{"epoch":347,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:38:00.190088+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11887},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11887":{"gid":11887,"name":"a","rank":0,"incarnation":344,"state":"up:active","state_seq":37,"addr":"172.21.15.42:6835/958796360","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":958796360},{"type":"v1","addr":"172.21.15.42:6835","nonce":958796360}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:38:07.585 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:07.581+0000 7fdfadee7700 1 -- 172.21.15.42:0/1668855972 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fdf8c03b210 msgr2=0x7fdf8c03d720 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:38:07.585 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:07.581+0000 7fdfadee7700 1 --2- 172.21.15.42:0/1668855972 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fdf8c03b210 0x7fdf8c03d720 crc :-1 s=READY pgs=1216 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:38:07.585 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:07.581+0000 7fdfadee7700 1 -- 172.21.15.42:0/1668855972 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdfa80a2760 msgr2=0x7fdfa811a1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:38:07.585 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:07.581+0000 7fdfadee7700 1 --2- 172.21.15.42:0/1668855972 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdfa80a2760 0x7fdfa811a1b0 secure :-1 s=READY pgs=2195 cs=0 l=1 rx=0x7fdf90000fe0 tx=0x7fdf90010910).stop 2019-12-05T06:38:07.585 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:07.581+0000 7fdfadee7700 1 -- 172.21.15.42:0/1668855972 shutdown_connections 2019-12-05T06:38:07.586 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:07.581+0000 7fdfadee7700 1 --2- 172.21.15.42:0/1668855972 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fdf8c03b210 0x7fdf8c03d720 unknown :-1 s=CLOSED pgs=1216 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:38:07.586 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:07.581+0000 7fdfadee7700 1 --2- 172.21.15.42:0/1668855972 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdfa80a2760 0x7fdfa811a1b0 secure :-1 s=CLOSED pgs=2195 cs=0 l=1 rx=0x7fdf90000fe0 tx=0x7fdf90010910).stop 2019-12-05T06:38:07.586 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:07.581+0000 7fdfadee7700 1 -- 172.21.15.42:0/1668855972 shutdown_connections 2019-12-05T06:38:07.586 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:07.581+0000 7fdfadee7700 1 -- 172.21.15.42:0/1668855972 wait complete. 2019-12-05T06:38:07.586 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:38:07.581+0000 7fdfadee7700 1 -- 172.21.15.42:0/1668855972 >> 172.21.15.42:0/1668855972 conn(0x7fdfa80a89f0 msgr2=0x7fdfa8059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:38:07.595 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b reported in up:standby state 2019-12-05T06:38:07.597 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 67 secs before thrashing 2019-12-05T06:38:14.088 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:38:14.092 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:38:44.162 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:38:44.181 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:39:14.253 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:39:14.257 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:39:14.597 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:39:14.698 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:14.688+0000 7fe6565af700 1 Processor -- start 2019-12-05T06:39:14.698 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:14.688+0000 7fe6565af700 1 -- start start 2019-12-05T06:39:14.698 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:14.688+0000 7fe6565af700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe6500a7730 0x7fe65008df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:39:14.698 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:14.688+0000 7fe6565af700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe65007e710 0x7fe65007eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:39:14.698 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:14.688+0000 7fe6565af700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6500a0700 con 0x7fe65009f2d0 2019-12-05T06:39:14.699 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:14.688+0000 7fe6565af700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6500a0af0 con 0x7fe65007e710 2019-12-05T06:39:14.699 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:14.688+0000 7fe6565af700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe65008e5a0 con 0x7fe6500a7730 2019-12-05T06:39:14.699 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:14.688+0000 7fe654b4c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe6500a7730 0x7fe65008df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:14.699 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:14.688+0000 7fe654b4c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe6500a7730 0x7fe65008df70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:59958/0 (socket says 172.21.15.42:59958) 2019-12-05T06:39:14.699 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:14.688+0000 7fe654b4c700 1 -- 172.21.15.42:0/2712917793 learned_addr learned my addr 172.21.15.42:0/2712917793 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:39:14.699 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:14.688+0000 7fe65534d700 1 --1- 172.21.15.42:0/2712917793 >> v1:172.21.15.42:6789/0 conn(0x7fe65009f2d0 0x7fe65009f700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:60000/0 (socket says 172.21.15.42:60000) 2019-12-05T06:39:14.699 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:14.688+0000 7fe64ffff700 1 --2- 172.21.15.42:0/2712917793 >> v2:172.21.15.40:3300/0 conn(0x7fe65007e710 0x7fe65007eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:14.700 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:14.688+0000 7fe654b4c700 1 -- 172.21.15.42:0/2712917793 >> v2:172.21.15.40:3300/0 conn(0x7fe65007e710 msgr2=0x7fe65007eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:39:14.700 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:14.688+0000 7fe654b4c700 1 --2- 172.21.15.42:0/2712917793 >> v2:172.21.15.40:3300/0 conn(0x7fe65007e710 0x7fe65007eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:39:14.700 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:14.688+0000 7fe654b4c700 1 -- 172.21.15.42:0/2712917793 >> v1:172.21.15.42:6789/0 conn(0x7fe65009f2d0 legacy=0x7fe65009f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:14.700 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:14.688+0000 7fe654b4c700 1 -- 172.21.15.42:0/2712917793 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe65007f090 con 0x7fe6500a7730 2019-12-05T06:39:14.700 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:14.688+0000 7fe64f7fe700 1 -- 172.21.15.42:0/2712917793 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 13353460 0 0) 0x7fe6500a0700 con 0x7fe65009f2d0 2019-12-05T06:39:14.701 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:14.688+0000 7fe654b4c700 1 --2- 172.21.15.42:0/2712917793 >> v2:172.21.15.42:3300/0 conn(0x7fe6500a7730 0x7fe65008df70 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fe644002f30 tx=0x7fe64400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:14.701 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:14.688+0000 7fe654b4c700 1 --2- 172.21.15.42:0/2712917793 >> v2:172.21.15.42:3300/0 conn(0x7fe6500a7730 0x7fe65008df70 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fe644002f30 tx=0x7fe64400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:14.701 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:14.688+0000 7fe654b4c700 1 --2- 172.21.15.42:0/2712917793 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe6500a7730 0x7fe65008df70 secure :-1 s=READY pgs=2899 cs=0 l=1 rx=0x7fe644002f30 tx=0x7fe64400fd30).ready entity=mon.1 client_cookie=c73bcabc300a2b32 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:39:14.701 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:14.688+0000 7fe654b4c700 1 --2- 172.21.15.42:0/2712917793 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe6500a7730 0x7fe65008df70 secure :-1 s=THROTTLE_DONE pgs=2899 cs=0 l=1 rx=0x7fe644002f30 tx=0x7fe64400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:14.701 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:14.688+0000 7fe64f7fe700 1 -- 172.21.15.42:0/2712917793 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe644011370 con 0x7fe6500a7730 2019-12-05T06:39:14.702 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:14.688+0000 7fe654b4c700 1 --2- 172.21.15.42:0/2712917793 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe6500a7730 0x7fe65008df70 secure :-1 s=THROTTLE_DONE pgs=2899 cs=0 l=1 rx=0x7fe644002f30 tx=0x7fe64400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:14.702 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:14.688+0000 7fe654b4c700 1 --2- 172.21.15.42:0/2712917793 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe6500a7730 0x7fe65008df70 secure :-1 s=THROTTLE_DONE pgs=2899 cs=0 l=1 rx=0x7fe644002f30 tx=0x7fe64400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:14.702 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:14.688+0000 7fe64f7fe700 1 -- 172.21.15.42:0/2712917793 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe644008bb0 con 0x7fe6500a7730 2019-12-05T06:39:14.702 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:14.688+0000 7fe64f7fe700 1 -- 172.21.15.42:0/2712917793 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe644011950 con 0x7fe6500a7730 2019-12-05T06:39:14.702 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:14.688+0000 7fe6565af700 1 -- 172.21.15.42:0/2712917793 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe6500a7730 msgr2=0x7fe65008df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:14.702 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:14.688+0000 7fe6565af700 1 --2- 172.21.15.42:0/2712917793 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe6500a7730 0x7fe65008df70 secure :-1 s=READY pgs=2899 cs=0 l=1 rx=0x7fe644002f30 tx=0x7fe64400fd30).stop 2019-12-05T06:39:14.703 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:14.692+0000 7fe6565af700 1 -- 172.21.15.42:0/2712917793 shutdown_connections 2019-12-05T06:39:14.703 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:14.692+0000 7fe6565af700 1 --2- 172.21.15.42:0/2712917793 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe6500a7730 0x7fe65008df70 unknown :-1 s=CLOSED pgs=2899 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:14.703 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:14.692+0000 7fe6565af700 1 --2- 172.21.15.42:0/2712917793 >> v2:172.21.15.40:3300/0 conn(0x7fe65007e710 0x7fe65007eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:39:14.703 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:14.692+0000 7fe6565af700 1 -- 172.21.15.42:0/2712917793 shutdown_connections 2019-12-05T06:39:14.703 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:14.692+0000 7fe6565af700 1 -- 172.21.15.42:0/2712917793 wait complete. 2019-12-05T06:39:14.703 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:14.692+0000 7fe6565af700 1 -- 172.21.15.42:0/2712917793 >> 172.21.15.42:0/2712917793 conn(0x7fe6500aabd0 msgr2=0x7fe6500a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:39:14.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:14.692+0000 7fe6565af700 1 Processor -- start 2019-12-05T06:39:14.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:14.692+0000 7fe6565af700 1 -- start start 2019-12-05T06:39:14.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:14.692+0000 7fe6565af700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe65008f7a0 0x7fe65008ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:39:14.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:14.692+0000 7fe6565af700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe650063cf0 0x7fe650064120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:39:14.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:14.692+0000 7fe6565af700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6501453e0 con 0x7fe6500655b0 2019-12-05T06:39:14.708 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:14.692+0000 7fe6565af700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe650065ac0 con 0x7fe65008f7a0 2019-12-05T06:39:14.708 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:14.692+0000 7fe6565af700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe650065d60 con 0x7fe650063cf0 2019-12-05T06:39:14.708 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:14.692+0000 7fe65534d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe650063cf0 0x7fe650064120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:14.708 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:14.692+0000 7fe654b4c700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fe6500655b0 0x7fe650144be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:60002/0 (socket says 172.21.15.42:60002) 2019-12-05T06:39:14.708 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:14.692+0000 7fe654b4c700 1 -- 172.21.15.42:0/3820730336 learned_addr learned my addr 172.21.15.42:0/3820730336 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:39:14.708 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:14.692+0000 7fe64ffff700 1 --2- 172.21.15.42:0/3820730336 >> v2:172.21.15.40:3300/0 conn(0x7fe65008f7a0 0x7fe65008ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:14.708 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:14.692+0000 7fe65534d700 1 -- 172.21.15.42:0/3820730336 >> v2:172.21.15.40:3300/0 conn(0x7fe65008f7a0 msgr2=0x7fe65008ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:14.709 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:14.692+0000 7fe65534d700 1 --2- 172.21.15.42:0/3820730336 >> v2:172.21.15.40:3300/0 conn(0x7fe65008f7a0 0x7fe65008ecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:14.709 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:14.692+0000 7fe65534d700 1 -- 172.21.15.42:0/3820730336 >> v1:172.21.15.42:6789/0 conn(0x7fe6500655b0 legacy=0x7fe650144be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:14.709 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:14.692+0000 7fe65534d700 1 -- 172.21.15.42:0/3820730336 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe65008f210 con 0x7fe650063cf0 2019-12-05T06:39:14.709 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:14.692+0000 7fe64ffff700 1 --2- 172.21.15.42:0/3820730336 >> v2:172.21.15.40:3300/0 conn(0x7fe65008f7a0 0x7fe65008ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:39:14.709 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:14.692+0000 7fe65534d700 1 --2- 172.21.15.42:0/3820730336 >> v2:172.21.15.42:3300/0 conn(0x7fe650063cf0 0x7fe650064120 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fe648001210 tx=0x7fe648010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:14.709 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:14.692+0000 7fe65534d700 1 --2- 172.21.15.42:0/3820730336 >> v2:172.21.15.42:3300/0 conn(0x7fe650063cf0 0x7fe650064120 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fe648001210 tx=0x7fe648010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:14.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:14.692+0000 7fe65534d700 1 --2- 172.21.15.42:0/3820730336 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe650063cf0 0x7fe650064120 secure :-1 s=READY pgs=2901 cs=0 l=1 rx=0x7fe648001210 tx=0x7fe648010b50).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:39:14.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:14.696+0000 7fe65534d700 1 --2- 172.21.15.42:0/3820730336 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe650063cf0 0x7fe650064120 secure :-1 s=THROTTLE_DONE pgs=2901 cs=0 l=1 rx=0x7fe648001210 tx=0x7fe648010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:14.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:14.696+0000 7fe64d7fa700 1 -- 172.21.15.42:0/3820730336 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe64800d0c0 con 0x7fe650063cf0 2019-12-05T06:39:14.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:14.696+0000 7fe65534d700 1 --2- 172.21.15.42:0/3820730336 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe650063cf0 0x7fe650064120 secure :-1 s=THROTTLE_DONE pgs=2901 cs=0 l=1 rx=0x7fe648001210 tx=0x7fe648010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:14.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:14.696+0000 7fe65534d700 1 --2- 172.21.15.42:0/3820730336 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe650063cf0 0x7fe650064120 secure :-1 s=THROTTLE_DONE pgs=2901 cs=0 l=1 rx=0x7fe648001210 tx=0x7fe648010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:14.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:14.696+0000 7fe64d7fa700 1 -- 172.21.15.42:0/3820730336 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe648011590 con 0x7fe650063cf0 2019-12-05T06:39:14.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:14.696+0000 7fe64d7fa700 1 -- 172.21.15.42:0/3820730336 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe64800d6a0 con 0x7fe650063cf0 2019-12-05T06:39:14.711 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:14.696+0000 7fe6565af700 1 -- 172.21.15.42:0/3820730336 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe650064670 con 0x7fe650063cf0 2019-12-05T06:39:14.711 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:14.696+0000 7fe6565af700 1 -- 172.21.15.42:0/3820730336 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe650129e60 con 0x7fe650063cf0 2019-12-05T06:39:14.711 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:14.696+0000 7fe65534d700 1 --2- 172.21.15.42:0/3820730336 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe650063cf0 0x7fe650064120 secure :-1 s=THROTTLE_DONE pgs=2901 cs=0 l=1 rx=0x7fe648001210 tx=0x7fe648010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:14.711 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:14.696+0000 7fe64d7fa700 1 -- 172.21.15.42:0/3820730336 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fe64800d980 con 0x7fe650063cf0 2019-12-05T06:39:14.711 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:14.696+0000 7fe65534d700 1 --2- 172.21.15.42:0/3820730336 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe650063cf0 0x7fe650064120 secure :-1 s=THROTTLE_DONE pgs=2901 cs=0 l=1 rx=0x7fe648001210 tx=0x7fe648010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:14.711 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:14.696+0000 7fe64d7fa700 1 --2- 172.21.15.42:0/3820730336 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe63003adc0 0x7fe63003d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:39:14.712 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:14.696+0000 7fe64d7fa700 1 -- 172.21.15.42:0/3820730336 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 11449+0+0 (secure 0 0 0) 0x7fe64804b160 con 0x7fe650063cf0 2019-12-05T06:39:14.712 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:14.696+0000 7fe654b4c700 1 --2- 172.21.15.42:0/3820730336 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe63003adc0 0x7fe63003d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:14.712 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:14.696+0000 7fe654b4c700 1 --2- 172.21.15.42:0/3820730336 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe63003adc0 0x7fe63003d2d0 crc :-1 s=READY pgs=1217 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:39:14.712 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:14.704+0000 7fe6565af700 1 -- 172.21.15.42:0/3820730336 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe65005d100 con 0x7fe650063cf0 2019-12-05T06:39:14.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:14.708+0000 7fe65534d700 1 --2- 172.21.15.42:0/3820730336 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe650063cf0 0x7fe650064120 secure :-1 s=THROTTLE_DONE pgs=2901 cs=0 l=1 rx=0x7fe648001210 tx=0x7fe648010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:14.715 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:14.708+0000 7fe64d7fa700 1 -- 172.21.15.42:0/3820730336 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fe650064670 con 0x7fe650063cf0 2019-12-05T06:39:14.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:14.892+0000 7fe6565af700 1 -- 172.21.15.42:0/3820730336 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe65005b8b0 con 0x7fe650063cf0 2019-12-05T06:39:14.897 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:14.892+0000 7fe65534d700 1 --2- 172.21.15.42:0/3820730336 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe650063cf0 0x7fe650064120 secure :-1 s=THROTTLE_DONE pgs=2901 cs=0 l=1 rx=0x7fe648001210 tx=0x7fe648010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:14.897 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:14.892+0000 7fe64d7fa700 1 -- 172.21.15.42:0/3820730336 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 348 v348) v1 ==== 95+0+2113 (secure 0 0 0) 0x7fe6480320a0 con 0x7fe650063cf0 2019-12-05T06:39:14.897 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 348 2019-12-05T06:39:14.897 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:39:14.898 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":348,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":8009,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/1777669201","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1777669201},{"type":"v1","addr":"172.21.15.40:6833","nonce":1777669201}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":348}],"filesystems":[{"mdsmap":{"epoch":347,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:38:00.190088+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11887},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11887":{"gid":11887,"name":"a","rank":0,"incarnation":344,"state":"up:active","state_seq":37,"addr":"172.21.15.42:6835/958796360","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":958796360},{"type":"v1","addr":"172.21.15.42:6835","nonce":958796360}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:39:14.900 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:14.892+0000 7fe6565af700 1 -- 172.21.15.42:0/3820730336 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe63003adc0 msgr2=0x7fe63003d2d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:14.900 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:14.892+0000 7fe6565af700 1 --2- 172.21.15.42:0/3820730336 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe63003adc0 0x7fe63003d2d0 crc :-1 s=READY pgs=1217 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:14.900 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:14.892+0000 7fe6565af700 1 -- 172.21.15.42:0/3820730336 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe650063cf0 msgr2=0x7fe650064120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:14.900 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:14.892+0000 7fe6565af700 1 --2- 172.21.15.42:0/3820730336 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe650063cf0 0x7fe650064120 secure :-1 s=READY pgs=2901 cs=0 l=1 rx=0x7fe648001210 tx=0x7fe648010b50).stop 2019-12-05T06:39:14.901 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:14.892+0000 7fe6565af700 1 -- 172.21.15.42:0/3820730336 shutdown_connections 2019-12-05T06:39:14.901 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:14.892+0000 7fe6565af700 1 --2- 172.21.15.42:0/3820730336 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe63003adc0 0x7fe63003d2d0 unknown :-1 s=CLOSED pgs=1217 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:14.901 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:14.892+0000 7fe6565af700 1 --2- 172.21.15.42:0/3820730336 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe650063cf0 0x7fe650064120 unknown :-1 s=CLOSED pgs=2901 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:14.901 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:14.892+0000 7fe6565af700 1 --2- 172.21.15.42:0/3820730336 >> v2:172.21.15.40:3300/0 conn(0x7fe65008f7a0 0x7fe65008ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:14.901 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:14.892+0000 7fe6565af700 1 -- 172.21.15.42:0/3820730336 shutdown_connections 2019-12-05T06:39:14.901 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:14.892+0000 7fe6565af700 1 -- 172.21.15.42:0/3820730336 wait complete. 2019-12-05T06:39:14.902 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:14.892+0000 7fe6565af700 1 -- 172.21.15.42:0/3820730336 >> 172.21.15.42:0/3820730336 conn(0x7fe6500a89c0 msgr2=0x7fe6500a9780 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:39:14.910 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.a (rank=0) 2019-12-05T06:39:14.911 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2019-12-05T06:39:14.911 INFO:teuthology.orchestra.run:waiting for 300 2019-12-05T06:39:14.920 INFO:tasks.ceph.mds.a:Stopped 2019-12-05T06:39:14.921 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:39:15.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:15.084+0000 7fca6d203700 1 Processor -- start 2019-12-05T06:39:15.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:15.084+0000 7fca6d203700 1 -- start start 2019-12-05T06:39:15.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:15.088+0000 7fca6d203700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fca6808df70 0x7fca6809f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:39:15.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:15.088+0000 7fca6d203700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca6809fc50 con 0x7fca6809f820 2019-12-05T06:39:15.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:15.088+0000 7fca6d203700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca680a0700 con 0x7fca680a7730 2019-12-05T06:39:15.097 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:15.088+0000 7fca6d203700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca680a0af0 con 0x7fca6808df70 2019-12-05T06:39:15.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:15.088+0000 7fca66d9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fca6808df70 0x7fca6809f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:15.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:15.088+0000 7fca66d9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fca6808df70 0x7fca6809f2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:59978/0 (socket says 172.21.15.42:59978) 2019-12-05T06:39:15.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:15.088+0000 7fca66d9d700 1 -- 172.21.15.42:0/2923352863 learned_addr learned my addr 172.21.15.42:0/2923352863 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:39:15.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:15.088+0000 7fca6659c700 1 -- 172.21.15.42:0/2923352863 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1507360098 0 0) 0x7fca6809fc50 con 0x7fca6809f820 2019-12-05T06:39:15.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:15.088+0000 7fca66d9d700 1 -- 172.21.15.42:0/2923352863 >> v1:172.21.15.42:6789/0 conn(0x7fca680a7730 legacy=0x7fca6807e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:15.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:15.088+0000 7fca66d9d700 1 -- 172.21.15.42:0/2923352863 >> v1:172.21.15.40:6789/0 conn(0x7fca6809f820 legacy=0x7fca6814efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:15.098 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:15.088+0000 7fca66d9d700 1 -- 172.21.15.42:0/2923352863 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca6807ef10 con 0x7fca6808df70 2019-12-05T06:39:15.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:15.088+0000 7fca66d9d700 1 --2- 172.21.15.42:0/2923352863 >> v2:172.21.15.42:3300/0 conn(0x7fca6808df70 0x7fca6809f2d0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fca54008790 tx=0x7fca5400fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:15.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:15.088+0000 7fca66d9d700 1 --2- 172.21.15.42:0/2923352863 >> v2:172.21.15.42:3300/0 conn(0x7fca6808df70 0x7fca6809f2d0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fca54008790 tx=0x7fca5400fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:15.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:15.088+0000 7fca66d9d700 1 --2- 172.21.15.42:0/2923352863 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fca6808df70 0x7fca6809f2d0 secure :-1 s=READY pgs=2903 cs=0 l=1 rx=0x7fca54008790 tx=0x7fca5400fcb0).ready entity=mon.1 client_cookie=a752b78fb1dc85b server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:39:15.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:15.088+0000 7fca66d9d700 1 --2- 172.21.15.42:0/2923352863 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fca6808df70 0x7fca6809f2d0 secure :-1 s=THROTTLE_DONE pgs=2903 cs=0 l=1 rx=0x7fca54008790 tx=0x7fca5400fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:15.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:15.088+0000 7fca66d9d700 1 --2- 172.21.15.42:0/2923352863 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fca6808df70 0x7fca6809f2d0 secure :-1 s=THROTTLE_DONE pgs=2903 cs=0 l=1 rx=0x7fca54008790 tx=0x7fca5400fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:15.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:15.088+0000 7fca6659c700 1 -- 172.21.15.42:0/2923352863 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fca54011520 con 0x7fca6808df70 2019-12-05T06:39:15.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:15.088+0000 7fca66d9d700 1 --2- 172.21.15.42:0/2923352863 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fca6808df70 0x7fca6809f2d0 secure :-1 s=THROTTLE_DONE pgs=2903 cs=0 l=1 rx=0x7fca54008790 tx=0x7fca5400fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:15.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:15.088+0000 7fca6659c700 1 -- 172.21.15.42:0/2923352863 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca54008ca0 con 0x7fca6808df70 2019-12-05T06:39:15.100 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:15.088+0000 7fca6659c700 1 -- 172.21.15.42:0/2923352863 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fca54011b00 con 0x7fca6808df70 2019-12-05T06:39:15.100 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:15.088+0000 7fca6d203700 1 -- 172.21.15.42:0/2923352863 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fca6808df70 msgr2=0x7fca6809f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:15.100 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:15.088+0000 7fca6d203700 1 --2- 172.21.15.42:0/2923352863 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fca6808df70 0x7fca6809f2d0 secure :-1 s=READY pgs=2903 cs=0 l=1 rx=0x7fca54008790 tx=0x7fca5400fcb0).stop 2019-12-05T06:39:15.100 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:15.088+0000 7fca6d203700 1 -- 172.21.15.42:0/2923352863 shutdown_connections 2019-12-05T06:39:15.100 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:15.088+0000 7fca6d203700 1 --2- 172.21.15.42:0/2923352863 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fca6808df70 0x7fca6809f2d0 unknown :-1 s=CLOSED pgs=2903 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:15.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:15.088+0000 7fca6d203700 1 -- 172.21.15.42:0/2923352863 shutdown_connections 2019-12-05T06:39:15.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:15.088+0000 7fca6d203700 1 -- 172.21.15.42:0/2923352863 wait complete. 2019-12-05T06:39:15.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:15.088+0000 7fca6d203700 1 -- 172.21.15.42:0/2923352863 >> 172.21.15.42:0/2923352863 conn(0x7fca680aabd0 msgr2=0x7fca680a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:39:15.104 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:15.092+0000 7fca6d203700 1 Processor -- start 2019-12-05T06:39:15.104 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:15.092+0000 7fca6d203700 1 -- start start 2019-12-05T06:39:15.104 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:15.092+0000 7fca6d203700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fca681164f0 0x7fca6807c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:39:15.104 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:15.092+0000 7fca6d203700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fca6807ae70 0x7fca6807b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:39:15.105 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:15.092+0000 7fca6d203700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca6811bec0 con 0x7fca6807cf70 2019-12-05T06:39:15.105 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:15.092+0000 7fca6d203700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca6811c0f0 con 0x7fca6807ae70 2019-12-05T06:39:15.105 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:15.092+0000 7fca6d203700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca6807d3a0 con 0x7fca681164f0 2019-12-05T06:39:15.105 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:15.092+0000 7fca66d9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fca6807ae70 0x7fca6807b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:15.105 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:15.092+0000 7fca6ca02700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fca6807cf70 0x7fca6811b6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:60022/0 (socket says 172.21.15.42:60022) 2019-12-05T06:39:15.105 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:15.092+0000 7fca6ca02700 1 -- 172.21.15.42:0/2742744224 learned_addr learned my addr 172.21.15.42:0/2742744224 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:39:15.105 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:15.092+0000 7fca66d9d700 1 --2- 172.21.15.42:0/2742744224 >> v2:172.21.15.42:3300/0 conn(0x7fca6807ae70 0x7fca6807b2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:59986/0 (socket says 172.21.15.42:59986) 2019-12-05T06:39:15.106 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:15.092+0000 7fca6759e700 1 --2- 172.21.15.42:0/2742744224 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fca681164f0 0x7fca6807c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:15.106 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:15.092+0000 7fca5ffff700 1 -- 172.21.15.42:0/2742744224 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1824422439 0 0) 0x7fca6811bec0 con 0x7fca6807cf70 2019-12-05T06:39:15.106 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:15.096+0000 7fca5ffff700 1 -- 172.21.15.42:0/2742744224 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca440030b0 con 0x7fca6807cf70 2019-12-05T06:39:15.106 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:15.096+0000 7fca66d9d700 1 -- 172.21.15.42:0/2742744224 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fca681164f0 msgr2=0x7fca6807c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:15.106 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:15.096+0000 7fca66d9d700 1 --2- 172.21.15.42:0/2742744224 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fca681164f0 0x7fca6807c8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:15.106 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:15.096+0000 7fca66d9d700 1 -- 172.21.15.42:0/2742744224 >> v1:172.21.15.42:6789/0 conn(0x7fca6807cf70 legacy=0x7fca6811b6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:15.106 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:15.096+0000 7fca66d9d700 1 -- 172.21.15.42:0/2742744224 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca6807d590 con 0x7fca6807ae70 2019-12-05T06:39:15.107 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:15.096+0000 7fca66d9d700 1 --2- 172.21.15.42:0/2742744224 >> v2:172.21.15.42:3300/0 conn(0x7fca6807ae70 0x7fca6807b2a0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fca54008790 tx=0x7fca540115e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:15.107 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:15.096+0000 7fca66d9d700 1 --2- 172.21.15.42:0/2742744224 >> v2:172.21.15.42:3300/0 conn(0x7fca6807ae70 0x7fca6807b2a0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fca54008790 tx=0x7fca540115e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:15.107 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:15.096+0000 7fca66d9d700 1 --2- 172.21.15.42:0/2742744224 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fca6807ae70 0x7fca6807b2a0 secure :-1 s=READY pgs=2905 cs=0 l=1 rx=0x7fca54008790 tx=0x7fca540115e0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:39:15.107 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:15.096+0000 7fca66d9d700 1 --2- 172.21.15.42:0/2742744224 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fca6807ae70 0x7fca6807b2a0 secure :-1 s=THROTTLE_DONE pgs=2905 cs=0 l=1 rx=0x7fca54008790 tx=0x7fca540115e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:15.107 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:15.096+0000 7fca5ffff700 1 -- 172.21.15.42:0/2742744224 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fca54007150 con 0x7fca6807ae70 2019-12-05T06:39:15.107 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:15.096+0000 7fca66d9d700 1 --2- 172.21.15.42:0/2742744224 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fca6807ae70 0x7fca6807b2a0 secure :-1 s=THROTTLE_DONE pgs=2905 cs=0 l=1 rx=0x7fca54008790 tx=0x7fca540115e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:15.107 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:15.096+0000 7fca66d9d700 1 --2- 172.21.15.42:0/2742744224 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fca6807ae70 0x7fca6807b2a0 secure :-1 s=THROTTLE_DONE pgs=2905 cs=0 l=1 rx=0x7fca54008790 tx=0x7fca540115e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:15.108 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:15.096+0000 7fca5ffff700 1 -- 172.21.15.42:0/2742744224 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca54011720 con 0x7fca6807ae70 2019-12-05T06:39:15.108 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:15.096+0000 7fca6d203700 1 -- 172.21.15.42:0/2742744224 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca6807d7a0 con 0x7fca6807ae70 2019-12-05T06:39:15.108 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:15.096+0000 7fca5ffff700 1 -- 172.21.15.42:0/2742744224 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fca54005730 con 0x7fca6807ae70 2019-12-05T06:39:15.108 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:15.096+0000 7fca6d203700 1 -- 172.21.15.42:0/2742744224 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca6813adc0 con 0x7fca6807ae70 2019-12-05T06:39:15.108 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:15.096+0000 7fca66d9d700 1 --2- 172.21.15.42:0/2742744224 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fca6807ae70 0x7fca6807b2a0 secure :-1 s=THROTTLE_DONE pgs=2905 cs=0 l=1 rx=0x7fca54008790 tx=0x7fca540115e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:15.108 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:15.096+0000 7fca5ffff700 1 -- 172.21.15.42:0/2742744224 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fca54005a00 con 0x7fca6807ae70 2019-12-05T06:39:15.108 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:15.096+0000 7fca66d9d700 1 --2- 172.21.15.42:0/2742744224 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fca6807ae70 0x7fca6807b2a0 secure :-1 s=THROTTLE_DONE pgs=2905 cs=0 l=1 rx=0x7fca54008790 tx=0x7fca540115e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:15.109 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:15.096+0000 7fca5ffff700 1 --2- 172.21.15.42:0/2742744224 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fca4403b2b0 0x7fca4403d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:39:15.109 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:15.096+0000 7fca5ffff700 1 -- 172.21.15.42:0/2742744224 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 11449+0+0 (secure 0 0 0) 0x7fca5404b7c0 con 0x7fca6807ae70 2019-12-05T06:39:15.109 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:15.096+0000 7fca6759e700 1 --2- 172.21.15.42:0/2742744224 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fca4403b2b0 0x7fca4403d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:15.109 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:15.100+0000 7fca6759e700 1 --2- 172.21.15.42:0/2742744224 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fca4403b2b0 0x7fca4403d7c0 crc :-1 s=READY pgs=1218 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:39:15.109 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:15.104+0000 7fca6d203700 1 -- 172.21.15.42:0/2742744224 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca6805d100 con 0x7fca6807ae70 2019-12-05T06:39:15.118 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:15.112+0000 7fca66d9d700 1 --2- 172.21.15.42:0/2742744224 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fca6807ae70 0x7fca6807b2a0 secure :-1 s=THROTTLE_DONE pgs=2905 cs=0 l=1 rx=0x7fca54008790 tx=0x7fca540115e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:15.118 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:15.112+0000 7fca5ffff700 1 -- 172.21.15.42:0/2742744224 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fca54032840 con 0x7fca6807ae70 2019-12-05T06:39:15.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:15.280+0000 7fca6d203700 1 -- 172.21.15.42:0/2742744224 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fca6805b8b0 con 0x7fca6807ae70 2019-12-05T06:39:15.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:15.280+0000 7fca66d9d700 1 --2- 172.21.15.42:0/2742744224 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fca6807ae70 0x7fca6807b2a0 secure :-1 s=THROTTLE_DONE pgs=2905 cs=0 l=1 rx=0x7fca54008790 tx=0x7fca540115e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:15.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:15.280+0000 7fca5ffff700 1 -- 172.21.15.42:0/2742744224 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 348 v348) v1 ==== 95+0+2113 (secure 0 0 0) 0x7fca54032aa0 con 0x7fca6807ae70 2019-12-05T06:39:15.290 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 348 2019-12-05T06:39:15.290 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:39:15.290 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":348,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":8009,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/1777669201","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1777669201},{"type":"v1","addr":"172.21.15.40:6833","nonce":1777669201}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":348}],"filesystems":[{"mdsmap":{"epoch":347,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:38:00.190088+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11887},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11887":{"gid":11887,"name":"a","rank":0,"incarnation":344,"state":"up:active","state_seq":37,"addr":"172.21.15.42:6835/958796360","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":958796360},{"type":"v1","addr":"172.21.15.42:6835","nonce":958796360}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:39:15.292 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:15.284+0000 7fca6d203700 1 -- 172.21.15.42:0/2742744224 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fca4403b2b0 msgr2=0x7fca4403d7c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:15.293 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:15.284+0000 7fca6d203700 1 --2- 172.21.15.42:0/2742744224 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fca4403b2b0 0x7fca4403d7c0 crc :-1 s=READY pgs=1218 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:15.293 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:15.284+0000 7fca6d203700 1 -- 172.21.15.42:0/2742744224 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fca6807ae70 msgr2=0x7fca6807b2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:15.293 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:15.284+0000 7fca6d203700 1 --2- 172.21.15.42:0/2742744224 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fca6807ae70 0x7fca6807b2a0 secure :-1 s=READY pgs=2905 cs=0 l=1 rx=0x7fca54008790 tx=0x7fca540115e0).stop 2019-12-05T06:39:15.293 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:15.284+0000 7fca6d203700 1 -- 172.21.15.42:0/2742744224 shutdown_connections 2019-12-05T06:39:15.293 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:15.284+0000 7fca6d203700 1 --2- 172.21.15.42:0/2742744224 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fca4403b2b0 0x7fca4403d7c0 unknown :-1 s=CLOSED pgs=1218 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:15.294 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:15.284+0000 7fca6d203700 1 --2- 172.21.15.42:0/2742744224 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fca681164f0 0x7fca6807c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:15.294 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:15.284+0000 7fca6d203700 1 --2- 172.21.15.42:0/2742744224 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fca6807ae70 0x7fca6807b2a0 unknown :-1 s=CLOSED pgs=2905 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:15.294 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:15.284+0000 7fca6d203700 1 -- 172.21.15.42:0/2742744224 shutdown_connections 2019-12-05T06:39:15.294 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:15.284+0000 7fca6d203700 1 -- 172.21.15.42:0/2742744224 wait complete. 2019-12-05T06:39:15.295 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:15.284+0000 7fca6d203700 1 -- 172.21.15.42:0/2742744224 >> 172.21.15.42:0/2742744224 conn(0x7fca6808fd90 msgr2=0x7fca680596f0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:39:15.301 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:39:17.303 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:39:17.414 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:17.400+0000 7ff08826b700 1 Processor -- start 2019-12-05T06:39:17.415 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:17.400+0000 7ff08826b700 1 -- start start 2019-12-05T06:39:17.415 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:17.400+0000 7ff08826b700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff08008df80 0x7ff08009f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:39:17.415 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:17.400+0000 7ff08826b700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff08009f830 0x7ff08014efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:39:17.416 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:17.400+0000 7ff08826b700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff08009fc60 con 0x7ff0800a7740 2019-12-05T06:39:17.416 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:17.400+0000 7ff08826b700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0800a0b00 con 0x7ff08008df80 2019-12-05T06:39:17.416 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:17.400+0000 7ff08826b700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff08008e5b0 con 0x7ff08009f830 2019-12-05T06:39:17.417 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:17.400+0000 7ff086808700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7ff0800a7740 0x7ff08007e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:60034/0 (socket says 172.21.15.42:60034) 2019-12-05T06:39:17.417 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:17.400+0000 7ff086808700 1 -- 172.21.15.42:0/2183535660 learned_addr learned my addr 172.21.15.42:0/2183535660 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:39:17.417 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:17.400+0000 7ff087009700 1 --2- 172.21.15.42:0/2183535660 >> v2:172.21.15.42:3300/0 conn(0x7ff08009f830 0x7ff08014efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:17.417 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:17.404+0000 7ff086007700 1 --2- 172.21.15.42:0/2183535660 >> v2:172.21.15.40:3300/0 conn(0x7ff08008df80 0x7ff08009f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:17.418 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:17.404+0000 7ff085806700 1 -- 172.21.15.42:0/2183535660 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 977882549 0 0) 0x7ff08009fc60 con 0x7ff0800a7740 2019-12-05T06:39:17.418 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:17.404+0000 7ff085806700 1 -- 172.21.15.42:0/2183535660 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff06c0030b0 con 0x7ff0800a7740 2019-12-05T06:39:17.418 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:17.404+0000 7ff087009700 1 -- 172.21.15.42:0/2183535660 >> v2:172.21.15.40:3300/0 conn(0x7ff08008df80 msgr2=0x7ff08009f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:39:17.418 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:17.404+0000 7ff087009700 1 --2- 172.21.15.42:0/2183535660 >> v2:172.21.15.40:3300/0 conn(0x7ff08008df80 0x7ff08009f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:39:17.419 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:17.404+0000 7ff087009700 1 -- 172.21.15.42:0/2183535660 >> v1:172.21.15.42:6789/0 conn(0x7ff0800a7740 legacy=0x7ff08007e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:17.419 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:17.404+0000 7ff087009700 1 -- 172.21.15.42:0/2183535660 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0800a0710 con 0x7ff08009f830 2019-12-05T06:39:17.419 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:17.404+0000 7ff087009700 1 --2- 172.21.15.42:0/2183535660 >> v2:172.21.15.42:3300/0 conn(0x7ff08009f830 0x7ff08014efe0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7ff0780032b0 tx=0x7ff07800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:17.420 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:17.404+0000 7ff087009700 1 --2- 172.21.15.42:0/2183535660 >> v2:172.21.15.42:3300/0 conn(0x7ff08009f830 0x7ff08014efe0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7ff0780032b0 tx=0x7ff07800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:17.420 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:17.404+0000 7ff087009700 1 --2- 172.21.15.42:0/2183535660 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff08009f830 0x7ff08014efe0 secure :-1 s=READY pgs=2907 cs=0 l=1 rx=0x7ff0780032b0 tx=0x7ff07800fff0).ready entity=mon.1 client_cookie=6706f7ab58c1ca5a server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:39:17.420 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:17.404+0000 7ff087009700 1 --2- 172.21.15.42:0/2183535660 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff08009f830 0x7ff08014efe0 secure :-1 s=THROTTLE_DONE pgs=2907 cs=0 l=1 rx=0x7ff0780032b0 tx=0x7ff07800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:17.421 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:17.404+0000 7ff087009700 1 --2- 172.21.15.42:0/2183535660 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff08009f830 0x7ff08014efe0 secure :-1 s=THROTTLE_DONE pgs=2907 cs=0 l=1 rx=0x7ff0780032b0 tx=0x7ff07800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:17.421 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:17.404+0000 7ff085806700 1 -- 172.21.15.42:0/2183535660 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff0780108c0 con 0x7ff08009f830 2019-12-05T06:39:17.421 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:17.404+0000 7ff087009700 1 --2- 172.21.15.42:0/2183535660 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff08009f830 0x7ff08014efe0 secure :-1 s=THROTTLE_DONE pgs=2907 cs=0 l=1 rx=0x7ff0780032b0 tx=0x7ff07800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:17.421 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:17.404+0000 7ff085806700 1 -- 172.21.15.42:0/2183535660 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff078008b50 con 0x7ff08009f830 2019-12-05T06:39:17.422 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:17.404+0000 7ff085806700 1 -- 172.21.15.42:0/2183535660 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff0780110c0 con 0x7ff08009f830 2019-12-05T06:39:17.422 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:17.404+0000 7ff08826b700 1 -- 172.21.15.42:0/2183535660 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff08009f830 msgr2=0x7ff08014efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:17.422 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:17.404+0000 7ff08826b700 1 --2- 172.21.15.42:0/2183535660 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff08009f830 0x7ff08014efe0 secure :-1 s=READY pgs=2907 cs=0 l=1 rx=0x7ff0780032b0 tx=0x7ff07800fff0).stop 2019-12-05T06:39:17.422 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:17.404+0000 7ff08826b700 1 -- 172.21.15.42:0/2183535660 shutdown_connections 2019-12-05T06:39:17.423 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:17.404+0000 7ff08826b700 1 --2- 172.21.15.42:0/2183535660 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff08009f830 0x7ff08014efe0 unknown :-1 s=CLOSED pgs=2907 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:17.423 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:17.404+0000 7ff08826b700 1 --2- 172.21.15.42:0/2183535660 >> v2:172.21.15.40:3300/0 conn(0x7ff08008df80 0x7ff08009f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:39:17.423 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:17.404+0000 7ff08826b700 1 -- 172.21.15.42:0/2183535660 shutdown_connections 2019-12-05T06:39:17.423 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:17.404+0000 7ff08826b700 1 -- 172.21.15.42:0/2183535660 wait complete. 2019-12-05T06:39:17.423 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:17.404+0000 7ff08826b700 1 -- 172.21.15.42:0/2183535660 >> 172.21.15.42:0/2183535660 conn(0x7ff0800aabe0 msgr2=0x7ff0800a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:39:17.425 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:17.408+0000 7ff08826b700 1 Processor -- start 2019-12-05T06:39:17.427 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:17.408+0000 7ff08826b700 1 -- start start 2019-12-05T06:39:17.435 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:17.408+0000 7ff08826b700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff0801167d0 0x7ff0800a9930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:39:17.435 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:17.408+0000 7ff08826b700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff080120e10 0x7ff080119080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:39:17.435 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:17.408+0000 7ff08826b700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff080121a90 con 0x7ff0801209e0 2019-12-05T06:39:17.435 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:17.408+0000 7ff08826b700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff080116c00 con 0x7ff0801167d0 2019-12-05T06:39:17.435 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:17.408+0000 7ff08826b700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff080116e70 con 0x7ff080120e10 2019-12-05T06:39:17.435 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:17.408+0000 7ff086007700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7ff0801209e0 0x7ff080118880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:60042/0 (socket says 172.21.15.42:60042) 2019-12-05T06:39:17.436 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:17.408+0000 7ff086007700 1 -- 172.21.15.42:0/832007931 learned_addr learned my addr 172.21.15.42:0/832007931 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:39:17.436 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:17.408+0000 7ff086808700 1 --2- 172.21.15.42:0/832007931 >> v2:172.21.15.40:3300/0 conn(0x7ff0801167d0 0x7ff0800a9930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:17.436 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:17.408+0000 7ff087009700 1 --2- 172.21.15.42:0/832007931 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff080120e10 0x7ff080119080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:17.436 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:17.408+0000 7ff0777fe700 1 -- 172.21.15.42:0/832007931 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2409083129 0 0) 0x7ff080121a90 con 0x7ff0801209e0 2019-12-05T06:39:17.436 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:17.408+0000 7ff0777fe700 1 -- 172.21.15.42:0/832007931 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0680030b0 con 0x7ff0801209e0 2019-12-05T06:39:17.436 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:17.412+0000 7ff086808700 1 -- 172.21.15.42:0/832007931 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff080120e10 msgr2=0x7ff080119080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:17.437 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:17.412+0000 7ff086808700 1 --2- 172.21.15.42:0/832007931 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff080120e10 0x7ff080119080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:17.437 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:17.412+0000 7ff086808700 1 -- 172.21.15.42:0/832007931 >> v1:172.21.15.42:6789/0 conn(0x7ff0801209e0 legacy=0x7ff080118880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:17.437 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:17.412+0000 7ff086808700 1 -- 172.21.15.42:0/832007931 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0801195d0 con 0x7ff0801167d0 2019-12-05T06:39:17.437 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:17.412+0000 7ff087009700 1 --2- 172.21.15.42:0/832007931 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff080120e10 0x7ff080119080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:39:17.437 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:17.412+0000 7ff086808700 1 --2- 172.21.15.42:0/832007931 >> v2:172.21.15.40:3300/0 conn(0x7ff0801167d0 0x7ff0800a9930 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7ff080117030 tx=0x7ff07c0108e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:17.437 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:17.412+0000 7ff086808700 1 --2- 172.21.15.42:0/832007931 >> v2:172.21.15.40:3300/0 conn(0x7ff0801167d0 0x7ff0800a9930 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7ff080117030 tx=0x7ff07c0108e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:17.438 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:17.412+0000 7ff086808700 1 --2- 172.21.15.42:0/832007931 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff0801167d0 0x7ff0800a9930 secure :-1 s=READY pgs=2202 cs=0 l=1 rx=0x7ff080117030 tx=0x7ff07c0108e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:39:17.438 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:17.412+0000 7ff086808700 1 --2- 172.21.15.42:0/832007931 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff0801167d0 0x7ff0800a9930 secure :-1 s=THROTTLE_DONE pgs=2202 cs=0 l=1 rx=0x7ff080117030 tx=0x7ff07c0108e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:17.438 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:17.412+0000 7ff0777fe700 1 -- 172.21.15.42:0/832007931 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff07c005b40 con 0x7ff0801167d0 2019-12-05T06:39:17.438 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:17.412+0000 7ff086808700 1 --2- 172.21.15.42:0/832007931 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff0801167d0 0x7ff0800a9930 secure :-1 s=THROTTLE_DONE pgs=2202 cs=0 l=1 rx=0x7ff080117030 tx=0x7ff07c0108e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:17.438 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:17.412+0000 7ff086808700 1 --2- 172.21.15.42:0/832007931 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff0801167d0 0x7ff0800a9930 secure :-1 s=THROTTLE_DONE pgs=2202 cs=0 l=1 rx=0x7ff080117030 tx=0x7ff07c0108e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:17.438 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:17.412+0000 7ff0777fe700 1 -- 172.21.15.42:0/832007931 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff07c005760 con 0x7ff0801167d0 2019-12-05T06:39:17.438 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:17.412+0000 7ff08826b700 1 -- 172.21.15.42:0/832007931 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff080121240 con 0x7ff0801167d0 2019-12-05T06:39:17.439 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:17.412+0000 7ff0777fe700 1 -- 172.21.15.42:0/832007931 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff07c0062b0 con 0x7ff0801167d0 2019-12-05T06:39:17.439 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:17.412+0000 7ff08826b700 1 -- 172.21.15.42:0/832007931 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0801224f0 con 0x7ff0801167d0 2019-12-05T06:39:17.439 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:17.412+0000 7ff086808700 1 --2- 172.21.15.42:0/832007931 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff0801167d0 0x7ff0800a9930 secure :-1 s=THROTTLE_DONE pgs=2202 cs=0 l=1 rx=0x7ff080117030 tx=0x7ff07c0108e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:17.439 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:17.412+0000 7ff0777fe700 1 -- 172.21.15.42:0/832007931 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7ff07c0065c0 con 0x7ff0801167d0 2019-12-05T06:39:17.439 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:17.412+0000 7ff086808700 1 --2- 172.21.15.42:0/832007931 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff0801167d0 0x7ff0800a9930 secure :-1 s=THROTTLE_DONE pgs=2202 cs=0 l=1 rx=0x7ff080117030 tx=0x7ff07c0108e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:17.439 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:17.412+0000 7ff0777fe700 1 --2- 172.21.15.42:0/832007931 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff06803b260 0x7ff06803d770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:39:17.440 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:17.412+0000 7ff0777fe700 1 -- 172.21.15.42:0/832007931 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 11449+0+0 (secure 0 0 0) 0x7ff07c04afc0 con 0x7ff0801167d0 2019-12-05T06:39:17.440 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:17.412+0000 7ff086007700 1 --2- 172.21.15.42:0/832007931 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff06803b260 0x7ff06803d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:17.440 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:17.416+0000 7ff086007700 1 --2- 172.21.15.42:0/832007931 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff06803b260 0x7ff06803d770 crc :-1 s=READY pgs=1219 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:39:17.440 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:17.416+0000 7ff08826b700 1 -- 172.21.15.42:0/832007931 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff08005d110 con 0x7ff0801167d0 2019-12-05T06:39:17.440 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:17.424+0000 7ff086808700 1 --2- 172.21.15.42:0/832007931 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff0801167d0 0x7ff0800a9930 secure :-1 s=THROTTLE_DONE pgs=2202 cs=0 l=1 rx=0x7ff080117030 tx=0x7ff07c0108e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:17.440 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:17.424+0000 7ff0777fe700 1 -- 172.21.15.42:0/832007931 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7ff07c032040 con 0x7ff0801167d0 2019-12-05T06:39:17.602 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:17.596+0000 7ff08826b700 1 -- 172.21.15.42:0/832007931 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff08005b8c0 con 0x7ff0801167d0 2019-12-05T06:39:17.603 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:17.596+0000 7ff086808700 1 --2- 172.21.15.42:0/832007931 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff0801167d0 0x7ff0800a9930 secure :-1 s=THROTTLE_DONE pgs=2202 cs=0 l=1 rx=0x7ff080117030 tx=0x7ff07c0108e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:17.603 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:17.596+0000 7ff0777fe700 1 -- 172.21.15.42:0/832007931 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 348 v348) v1 ==== 95+0+2113 (secure 0 0 0) 0x7ff07c0322a0 con 0x7ff0801167d0 2019-12-05T06:39:17.605 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 348 2019-12-05T06:39:17.605 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:39:17.605 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":348,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":8009,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/1777669201","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1777669201},{"type":"v1","addr":"172.21.15.40:6833","nonce":1777669201}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":348}],"filesystems":[{"mdsmap":{"epoch":347,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:38:00.190088+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11887},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11887":{"gid":11887,"name":"a","rank":0,"incarnation":344,"state":"up:active","state_seq":37,"addr":"172.21.15.42:6835/958796360","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":958796360},{"type":"v1","addr":"172.21.15.42:6835","nonce":958796360}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:39:17.610 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:17.600+0000 7ff08826b700 1 -- 172.21.15.42:0/832007931 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff06803b260 msgr2=0x7ff06803d770 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:17.610 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:17.600+0000 7ff08826b700 1 --2- 172.21.15.42:0/832007931 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff06803b260 0x7ff06803d770 crc :-1 s=READY pgs=1219 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:17.611 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:17.600+0000 7ff08826b700 1 -- 172.21.15.42:0/832007931 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff0801167d0 msgr2=0x7ff0800a9930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:17.611 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:17.600+0000 7ff08826b700 1 --2- 172.21.15.42:0/832007931 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff0801167d0 0x7ff0800a9930 secure :-1 s=READY pgs=2202 cs=0 l=1 rx=0x7ff080117030 tx=0x7ff07c0108e0).stop 2019-12-05T06:39:17.612 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:17.600+0000 7ff08826b700 1 -- 172.21.15.42:0/832007931 shutdown_connections 2019-12-05T06:39:17.612 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:17.600+0000 7ff08826b700 1 --2- 172.21.15.42:0/832007931 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff06803b260 0x7ff06803d770 unknown :-1 s=CLOSED pgs=1219 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:17.612 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:17.600+0000 7ff08826b700 1 --2- 172.21.15.42:0/832007931 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff080120e10 0x7ff080119080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:17.612 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:17.600+0000 7ff08826b700 1 --2- 172.21.15.42:0/832007931 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff0801167d0 0x7ff0800a9930 unknown :-1 s=CLOSED pgs=2202 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:17.612 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:17.600+0000 7ff08826b700 1 -- 172.21.15.42:0/832007931 shutdown_connections 2019-12-05T06:39:17.613 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:17.600+0000 7ff08826b700 1 -- 172.21.15.42:0/832007931 wait complete. 2019-12-05T06:39:17.613 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:17.600+0000 7ff08826b700 1 -- 172.21.15.42:0/832007931 >> 172.21.15.42:0/832007931 conn(0x7ff0800a7f90 msgr2=0x7ff080111db0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:39:17.623 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:39:19.623 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:39:19.750 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:19.720+0000 7f6e10357700 1 Processor -- start 2019-12-05T06:39:19.751 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:19.720+0000 7f6e10357700 1 -- start start 2019-12-05T06:39:19.751 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:19.720+0000 7f6e10357700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f6e0808df80 0x7f6e0809f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:39:19.751 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:19.720+0000 7f6e10357700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e0809fc60 con 0x7f6e0809f830 2019-12-05T06:39:19.751 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:19.720+0000 7f6e10357700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e080a0710 con 0x7f6e080a7740 2019-12-05T06:39:19.751 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:19.720+0000 7f6e10357700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e080a0b00 con 0x7f6e0808df80 2019-12-05T06:39:19.752 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:19.720+0000 7f6e0e8f4700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f6e080a7740 0x7f6e0807e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:60050/0 (socket says 172.21.15.42:60050) 2019-12-05T06:39:19.752 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:19.724+0000 7f6e0e8f4700 1 -- 172.21.15.42:0/1030621961 learned_addr learned my addr 172.21.15.42:0/1030621961 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:39:19.752 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:19.724+0000 7f6e0e0f3700 1 --2- 172.21.15.42:0/1030621961 >> v2:172.21.15.40:3300/0 conn(0x7f6e0808df80 0x7f6e0809f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:19.752 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:19.724+0000 7f6e0d8f2700 1 -- 172.21.15.42:0/1030621961 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 81082658 0 0) 0x7f6e080a0710 con 0x7f6e080a7740 2019-12-05T06:39:19.752 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:19.724+0000 7f6e0d8f2700 1 -- 172.21.15.42:0/1030621961 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6df40030b0 con 0x7f6e080a7740 2019-12-05T06:39:19.752 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:19.724+0000 7f6e0e0f3700 1 -- 172.21.15.42:0/1030621961 >> v1:172.21.15.42:6789/0 conn(0x7f6e080a7740 legacy=0x7f6e0807e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:19.753 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:19.724+0000 7f6e0e0f3700 1 -- 172.21.15.42:0/1030621961 >> v1:172.21.15.40:6789/0 conn(0x7f6e0809f830 legacy=0x7f6e0814efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:19.753 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:19.724+0000 7f6e0d8f2700 1 -- 172.21.15.42:0/1030621961 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3133298040 0 0) 0x7f6e0809fc60 con 0x7f6e0809f830 2019-12-05T06:39:19.753 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:19.724+0000 7f6e0e0f3700 1 -- 172.21.15.42:0/1030621961 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e0807ef20 con 0x7f6e0808df80 2019-12-05T06:39:19.753 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:19.724+0000 7f6e0e0f3700 1 --2- 172.21.15.42:0/1030621961 >> v2:172.21.15.40:3300/0 conn(0x7f6e0808df80 0x7f6e0809f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f6e04008b10 tx=0x7f6e0400ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:19.753 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:19.724+0000 7f6e0e0f3700 1 --2- 172.21.15.42:0/1030621961 >> v2:172.21.15.40:3300/0 conn(0x7f6e0808df80 0x7f6e0809f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f6e04008b10 tx=0x7f6e0400ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:19.753 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:19.724+0000 7f6e0e0f3700 1 --2- 172.21.15.42:0/1030621961 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6e0808df80 0x7f6e0809f2e0 secure :-1 s=READY pgs=2204 cs=0 l=1 rx=0x7f6e04008b10 tx=0x7f6e0400ff70).ready entity=mon.0 client_cookie=9d42753bf7360432 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:39:19.753 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:19.724+0000 7f6e0e0f3700 1 --2- 172.21.15.42:0/1030621961 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6e0808df80 0x7f6e0809f2e0 secure :-1 s=THROTTLE_DONE pgs=2204 cs=0 l=1 rx=0x7f6e04008b10 tx=0x7f6e0400ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:19.754 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:19.724+0000 7f6e0e0f3700 1 --2- 172.21.15.42:0/1030621961 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6e0808df80 0x7f6e0809f2e0 secure :-1 s=THROTTLE_DONE pgs=2204 cs=0 l=1 rx=0x7f6e04008b10 tx=0x7f6e0400ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:19.754 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:19.724+0000 7f6e0d8f2700 1 -- 172.21.15.42:0/1030621961 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6e040119d0 con 0x7f6e0808df80 2019-12-05T06:39:19.754 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:19.724+0000 7f6e0e0f3700 1 --2- 172.21.15.42:0/1030621961 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6e0808df80 0x7f6e0809f2e0 secure :-1 s=THROTTLE_DONE pgs=2204 cs=0 l=1 rx=0x7f6e04008b10 tx=0x7f6e0400ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:19.754 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:19.724+0000 7f6e0d8f2700 1 -- 172.21.15.42:0/1030621961 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e040115f0 con 0x7f6e0808df80 2019-12-05T06:39:19.754 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:19.724+0000 7f6e0d8f2700 1 -- 172.21.15.42:0/1030621961 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6e0400a160 con 0x7f6e0808df80 2019-12-05T06:39:19.754 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:19.724+0000 7f6e10357700 1 -- 172.21.15.42:0/1030621961 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6e0808df80 msgr2=0x7f6e0809f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:19.754 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:19.724+0000 7f6e10357700 1 --2- 172.21.15.42:0/1030621961 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6e0808df80 0x7f6e0809f2e0 secure :-1 s=READY pgs=2204 cs=0 l=1 rx=0x7f6e04008b10 tx=0x7f6e0400ff70).stop 2019-12-05T06:39:19.755 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:19.724+0000 7f6e10357700 1 -- 172.21.15.42:0/1030621961 shutdown_connections 2019-12-05T06:39:19.755 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:19.724+0000 7f6e10357700 1 --2- 172.21.15.42:0/1030621961 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6e0808df80 0x7f6e0809f2e0 unknown :-1 s=CLOSED pgs=2204 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:19.755 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:19.724+0000 7f6e10357700 1 -- 172.21.15.42:0/1030621961 shutdown_connections 2019-12-05T06:39:19.755 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:19.724+0000 7f6e10357700 1 -- 172.21.15.42:0/1030621961 wait complete. 2019-12-05T06:39:19.755 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:19.724+0000 7f6e10357700 1 -- 172.21.15.42:0/1030621961 >> 172.21.15.42:0/1030621961 conn(0x7f6e080aabe0 msgr2=0x7f6e080a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:39:19.755 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:19.728+0000 7f6e10357700 1 Processor -- start 2019-12-05T06:39:19.756 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:19.728+0000 7f6e10357700 1 -- start start 2019-12-05T06:39:19.756 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:19.728+0000 7f6e10357700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6e08144b80 0x7f6e08118c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:39:19.756 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:19.728+0000 7f6e10357700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e08119990 con 0x7f6e081191c0 2019-12-05T06:39:19.756 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:19.728+0000 7f6e10357700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e0811c030 con 0x7f6e0811b090 2019-12-05T06:39:19.756 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:19.728+0000 7f6e10357700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e08117a20 con 0x7f6e08144b80 2019-12-05T06:39:19.756 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:19.728+0000 7f6e0e0f3700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f6e0811b090 0x7f6e0808eb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:60058/0 (socket says 172.21.15.42:60058) 2019-12-05T06:39:19.756 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:19.728+0000 7f6e0e0f3700 1 -- 172.21.15.42:0/2843713873 learned_addr learned my addr 172.21.15.42:0/2843713873 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:39:19.757 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:19.732+0000 7f6e0e8f4700 1 --2- 172.21.15.42:0/2843713873 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6e08144b80 0x7f6e08118c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:19.757 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:19.732+0000 7f6dfb7fe700 1 -- 172.21.15.42:0/2843713873 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3142181842 0 0) 0x7f6e0811c030 con 0x7f6e0811b090 2019-12-05T06:39:19.757 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:19.732+0000 7f6dfb7fe700 1 -- 172.21.15.42:0/2843713873 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6df00030b0 con 0x7f6e0811b090 2019-12-05T06:39:19.757 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:19.732+0000 7f6dfb7fe700 1 -- 172.21.15.42:0/2843713873 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2389908543 0 0) 0x7f6e08119990 con 0x7f6e081191c0 2019-12-05T06:39:19.757 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:19.732+0000 7f6dfb7fe700 1 -- 172.21.15.42:0/2843713873 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e0811c030 con 0x7f6e081191c0 2019-12-05T06:39:19.757 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:19.732+0000 7f6e0e8f4700 1 -- 172.21.15.42:0/2843713873 >> v1:172.21.15.42:6789/0 conn(0x7f6e0811b090 legacy=0x7f6e0808eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:19.757 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:19.732+0000 7f6e0e8f4700 1 -- 172.21.15.42:0/2843713873 >> v1:172.21.15.40:6789/0 conn(0x7f6e081191c0 legacy=0x7f6e0808f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:19.758 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:19.732+0000 7f6e0e8f4700 1 -- 172.21.15.42:0/2843713873 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e08078310 con 0x7f6e08144b80 2019-12-05T06:39:19.758 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:19.732+0000 7f6e0e8f4700 1 --2- 172.21.15.42:0/2843713873 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6e08144b80 0x7f6e08118c70 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f6e0811ae50 tx=0x7f6dfc010810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:19.758 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:19.732+0000 7f6e0e8f4700 1 --2- 172.21.15.42:0/2843713873 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6e08144b80 0x7f6e08118c70 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f6e0811ae50 tx=0x7f6dfc010810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:19.758 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:19.732+0000 7f6e0e8f4700 1 --2- 172.21.15.42:0/2843713873 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6e08144b80 0x7f6e08118c70 secure :-1 s=READY pgs=648 cs=0 l=1 rx=0x7f6e0811ae50 tx=0x7f6dfc010810).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:39:19.758 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:19.732+0000 7f6e0e8f4700 1 --2- 172.21.15.42:0/2843713873 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6e08144b80 0x7f6e08118c70 secure :-1 s=THROTTLE_DONE pgs=648 cs=0 l=1 rx=0x7f6e0811ae50 tx=0x7f6dfc010810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:19.758 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:19.732+0000 7f6dfb7fe700 1 -- 172.21.15.42:0/2843713873 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6dfc005c30 con 0x7f6e08144b80 2019-12-05T06:39:19.759 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:19.732+0000 7f6e0e8f4700 1 --2- 172.21.15.42:0/2843713873 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6e08144b80 0x7f6e08118c70 secure :-1 s=THROTTLE_DONE pgs=648 cs=0 l=1 rx=0x7f6e0811ae50 tx=0x7f6dfc010810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:19.759 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:19.732+0000 7f6dfb7fe700 1 -- 172.21.15.42:0/2843713873 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6dfc005850 con 0x7f6e08144b80 2019-12-05T06:39:19.759 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:19.732+0000 7f6e0e8f4700 1 --2- 172.21.15.42:0/2843713873 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6e08144b80 0x7f6e08118c70 secure :-1 s=THROTTLE_DONE pgs=648 cs=0 l=1 rx=0x7f6e0811ae50 tx=0x7f6dfc010810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:19.759 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:19.732+0000 7f6e10357700 1 -- 172.21.15.42:0/2843713873 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e080762f0 con 0x7f6e08144b80 2019-12-05T06:39:19.759 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:19.732+0000 7f6dfb7fe700 1 -- 172.21.15.42:0/2843713873 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6dfc0064a0 con 0x7f6e08144b80 2019-12-05T06:39:19.759 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:19.732+0000 7f6e10357700 1 -- 172.21.15.42:0/2843713873 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e08076620 con 0x7f6e08144b80 2019-12-05T06:39:19.759 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:19.732+0000 7f6e0e8f4700 1 --2- 172.21.15.42:0/2843713873 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6e08144b80 0x7f6e08118c70 secure :-1 s=THROTTLE_DONE pgs=648 cs=0 l=1 rx=0x7f6e0811ae50 tx=0x7f6dfc010810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:19.760 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:19.732+0000 7f6dfb7fe700 1 -- 172.21.15.42:0/2843713873 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f6dfc006780 con 0x7f6e08144b80 2019-12-05T06:39:19.760 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:19.732+0000 7f6e0e8f4700 1 --2- 172.21.15.42:0/2843713873 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6e08144b80 0x7f6e08118c70 secure :-1 s=THROTTLE_DONE pgs=648 cs=0 l=1 rx=0x7f6e0811ae50 tx=0x7f6dfc010810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:19.760 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:19.736+0000 7f6dfb7fe700 1 --2- 172.21.15.42:0/2843713873 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6df003b420 0x7f6df003d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:39:19.760 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:19.736+0000 7f6dfb7fe700 1 -- 172.21.15.42:0/2843713873 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 11449+0+0 (secure 0 0 0) 0x7f6dfc04af20 con 0x7f6e08144b80 2019-12-05T06:39:19.760 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:19.736+0000 7f6e0e0f3700 1 --2- 172.21.15.42:0/2843713873 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6df003b420 0x7f6df003d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:19.760 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:19.736+0000 7f6e0e0f3700 1 --2- 172.21.15.42:0/2843713873 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6df003b420 0x7f6df003d930 crc :-1 s=READY pgs=1220 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:39:19.760 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:19.736+0000 7f6e10357700 1 -- 172.21.15.42:0/2843713873 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e0805d110 con 0x7f6e08144b80 2019-12-05T06:39:19.761 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:19.744+0000 7f6e0e8f4700 1 --2- 172.21.15.42:0/2843713873 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6e08144b80 0x7f6e08118c70 secure :-1 s=THROTTLE_DONE pgs=648 cs=0 l=1 rx=0x7f6e0811ae50 tx=0x7f6dfc010810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:19.761 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:19.744+0000 7f6dfb7fe700 1 -- 172.21.15.42:0/2843713873 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f6dfc031f80 con 0x7f6e08144b80 2019-12-05T06:39:19.924 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:19.916+0000 7f6e10357700 1 -- 172.21.15.42:0/2843713873 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6e080a8cb0 con 0x7f6e08144b80 2019-12-05T06:39:19.925 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:19.920+0000 7f6e0e8f4700 1 --2- 172.21.15.42:0/2843713873 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6e08144b80 0x7f6e08118c70 secure :-1 s=THROTTLE_DONE pgs=648 cs=0 l=1 rx=0x7f6e0811ae50 tx=0x7f6dfc010810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:19.925 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:19.920+0000 7f6dfb7fe700 1 -- 172.21.15.42:0/2843713873 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 348 v348) v1 ==== 95+0+2113 (secure 0 0 0) 0x7f6dfc0321e0 con 0x7f6e08144b80 2019-12-05T06:39:19.927 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 348 2019-12-05T06:39:19.927 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:39:19.928 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":348,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":8009,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/1777669201","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1777669201},{"type":"v1","addr":"172.21.15.40:6833","nonce":1777669201}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":348}],"filesystems":[{"mdsmap":{"epoch":347,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:38:00.190088+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11887},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11887":{"gid":11887,"name":"a","rank":0,"incarnation":344,"state":"up:active","state_seq":37,"addr":"172.21.15.42:6835/958796360","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":958796360},{"type":"v1","addr":"172.21.15.42:6835","nonce":958796360}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:39:19.930 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:19.924+0000 7f6e10357700 1 -- 172.21.15.42:0/2843713873 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6df003b420 msgr2=0x7f6df003d930 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:19.931 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:19.924+0000 7f6e10357700 1 --2- 172.21.15.42:0/2843713873 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6df003b420 0x7f6df003d930 crc :-1 s=READY pgs=1220 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:19.931 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:19.924+0000 7f6e10357700 1 -- 172.21.15.42:0/2843713873 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6e08144b80 msgr2=0x7f6e08118c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:19.931 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:19.924+0000 7f6e10357700 1 --2- 172.21.15.42:0/2843713873 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6e08144b80 0x7f6e08118c70 secure :-1 s=READY pgs=648 cs=0 l=1 rx=0x7f6e0811ae50 tx=0x7f6dfc010810).stop 2019-12-05T06:39:19.932 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:19.924+0000 7f6e10357700 1 -- 172.21.15.42:0/2843713873 shutdown_connections 2019-12-05T06:39:19.932 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:19.924+0000 7f6e10357700 1 --2- 172.21.15.42:0/2843713873 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6df003b420 0x7f6df003d930 unknown :-1 s=CLOSED pgs=1220 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:19.932 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:19.924+0000 7f6e10357700 1 --2- 172.21.15.42:0/2843713873 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6e08144b80 0x7f6e08118c70 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:19.933 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:19.924+0000 7f6e10357700 1 -- 172.21.15.42:0/2843713873 shutdown_connections 2019-12-05T06:39:19.933 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:19.924+0000 7f6e10357700 1 -- 172.21.15.42:0/2843713873 wait complete. 2019-12-05T06:39:19.933 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:19.924+0000 7f6e10357700 1 -- 172.21.15.42:0/2843713873 >> 172.21.15.42:0/2843713873 conn(0x7f6e0808fda0 msgr2=0x7f6e08059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:39:19.944 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:39:21.945 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:39:22.073 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:22.036+0000 7f9995a6b700 1 Processor -- start 2019-12-05T06:39:22.073 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:22.036+0000 7f9995a6b700 1 -- start start 2019-12-05T06:39:22.073 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:22.036+0000 7f9995a6b700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f99900a7740 0x7f999008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:39:22.073 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:22.036+0000 7f9995a6b700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99900a0710 con 0x7f999007e720 2019-12-05T06:39:22.083 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:22.036+0000 7f9995a6b700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99900a9ba0 con 0x7f999009fae0 2019-12-05T06:39:22.083 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:22.036+0000 7f9995a6b700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99900a0b00 con 0x7f99900a7740 2019-12-05T06:39:22.083 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:22.040+0000 7f998ffff700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f99900a7740 0x7f999008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:22.083 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:22.040+0000 7f9994809700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f999009fae0 0x7f999014efe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:60070/0 (socket says 172.21.15.42:60070) 2019-12-05T06:39:22.083 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:22.040+0000 7f9994809700 1 -- 172.21.15.42:0/2403618097 learned_addr learned my addr 172.21.15.42:0/2403618097 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:39:22.083 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:22.040+0000 7f998effd700 1 -- 172.21.15.42:0/2403618097 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4191172121 0 0) 0x7f99900a9ba0 con 0x7f999009fae0 2019-12-05T06:39:22.084 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:22.040+0000 7f998effd700 1 -- 172.21.15.42:0/2403618097 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f997c0030b0 con 0x7f999009fae0 2019-12-05T06:39:22.084 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:22.040+0000 7f998ffff700 1 -- 172.21.15.42:0/2403618097 >> v1:172.21.15.42:6789/0 conn(0x7f999009fae0 legacy=0x7f999014efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:22.084 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:22.040+0000 7f998ffff700 1 -- 172.21.15.42:0/2403618097 >> v1:172.21.15.40:6789/0 conn(0x7f999007e720 legacy=0x7f999009f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:22.085 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:22.040+0000 7f998ffff700 1 -- 172.21.15.42:0/2403618097 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f999007eb50 con 0x7f99900a7740 2019-12-05T06:39:22.085 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:22.040+0000 7f998ffff700 1 --2- 172.21.15.42:0/2403618097 >> v2:172.21.15.40:3300/0 conn(0x7f99900a7740 0x7f999008df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f9980008b10 tx=0x7f998000ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:22.085 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:22.040+0000 7f998ffff700 1 --2- 172.21.15.42:0/2403618097 >> v2:172.21.15.40:3300/0 conn(0x7f99900a7740 0x7f999008df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f9980008b10 tx=0x7f998000ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:22.085 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:22.040+0000 7f998ffff700 1 --2- 172.21.15.42:0/2403618097 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f99900a7740 0x7f999008df80 secure :-1 s=READY pgs=2207 cs=0 l=1 rx=0x7f9980008b10 tx=0x7f998000ff70).ready entity=mon.0 client_cookie=6df91ac7c2b78d37 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:39:22.086 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:22.040+0000 7f998ffff700 1 --2- 172.21.15.42:0/2403618097 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f99900a7740 0x7f999008df80 secure :-1 s=THROTTLE_DONE pgs=2207 cs=0 l=1 rx=0x7f9980008b10 tx=0x7f998000ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:22.086 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:22.040+0000 7f998effd700 1 -- 172.21.15.42:0/2403618097 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f99800119d0 con 0x7f99900a7740 2019-12-05T06:39:22.086 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:22.040+0000 7f998ffff700 1 --2- 172.21.15.42:0/2403618097 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f99900a7740 0x7f999008df80 secure :-1 s=THROTTLE_DONE pgs=2207 cs=0 l=1 rx=0x7f9980008b10 tx=0x7f998000ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:22.086 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:22.040+0000 7f998ffff700 1 --2- 172.21.15.42:0/2403618097 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f99900a7740 0x7f999008df80 secure :-1 s=THROTTLE_DONE pgs=2207 cs=0 l=1 rx=0x7f9980008b10 tx=0x7f998000ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:22.086 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:22.040+0000 7f998effd700 1 -- 172.21.15.42:0/2403618097 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99800115f0 con 0x7f99900a7740 2019-12-05T06:39:22.086 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:22.040+0000 7f998effd700 1 -- 172.21.15.42:0/2403618097 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f998000a160 con 0x7f99900a7740 2019-12-05T06:39:22.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:22.040+0000 7f9995a6b700 1 -- 172.21.15.42:0/2403618097 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f99900a7740 msgr2=0x7f999008df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:22.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:22.040+0000 7f9995a6b700 1 --2- 172.21.15.42:0/2403618097 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f99900a7740 0x7f999008df80 secure :-1 s=READY pgs=2207 cs=0 l=1 rx=0x7f9980008b10 tx=0x7f998000ff70).stop 2019-12-05T06:39:22.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:22.040+0000 7f9995a6b700 1 -- 172.21.15.42:0/2403618097 shutdown_connections 2019-12-05T06:39:22.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:22.040+0000 7f9995a6b700 1 --2- 172.21.15.42:0/2403618097 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f99900a7740 0x7f999008df80 unknown :-1 s=CLOSED pgs=2207 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:22.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:22.040+0000 7f9995a6b700 1 -- 172.21.15.42:0/2403618097 shutdown_connections 2019-12-05T06:39:22.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:22.040+0000 7f9995a6b700 1 -- 172.21.15.42:0/2403618097 wait complete. 2019-12-05T06:39:22.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:22.040+0000 7f9995a6b700 1 -- 172.21.15.42:0/2403618097 >> 172.21.15.42:0/2403618097 conn(0x7f99900aabe0 msgr2=0x7f99900a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:39:22.088 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:22.044+0000 7f9995a6b700 1 Processor -- start 2019-12-05T06:39:22.089 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:22.044+0000 7f9995a6b700 1 -- start start 2019-12-05T06:39:22.089 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:22.044+0000 7f9995a6b700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f99900a3a60 0x7f99900797a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:39:22.089 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:22.044+0000 7f9995a6b700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9990142870 0x7f9990142ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:39:22.089 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:22.044+0000 7f9995a6b700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9990143ad0 con 0x7f99901436a0 2019-12-05T06:39:22.089 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:22.044+0000 7f9995a6b700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9990143db0 con 0x7f99900a3a60 2019-12-05T06:39:22.089 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:22.044+0000 7f9995a6b700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99901425d0 con 0x7f9990142870 2019-12-05T06:39:22.089 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:22.044+0000 7f998ffff700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f99900a3a60 0x7f99900797a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:22.090 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:22.044+0000 7f9994809700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9990142870 0x7f9990142ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:22.090 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:22.044+0000 7f998ffff700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f99900a3a60 0x7f99900797a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.42:47850/0 (socket says 172.21.15.42:47850) 2019-12-05T06:39:22.090 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:22.044+0000 7f998ffff700 1 -- 172.21.15.42:0/2900566787 learned_addr learned my addr 172.21.15.42:0/2900566787 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:39:22.090 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:22.044+0000 7f998ffff700 1 -- 172.21.15.42:0/2900566787 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9990142870 msgr2=0x7f9990142ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:22.090 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:22.044+0000 7f998ffff700 1 --2- 172.21.15.42:0/2900566787 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9990142870 0x7f9990142ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:22.090 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:22.044+0000 7f998cff9700 1 -- 172.21.15.42:0/2900566787 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3278297531 0 0) 0x7f99900a0710 con 0x7f99901436a0 2019-12-05T06:39:22.090 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:22.044+0000 7f998ffff700 1 -- 172.21.15.42:0/2900566787 >> v1:172.21.15.40:6789/0 conn(0x7f99901436a0 legacy=0x7f9990078a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:22.091 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:22.044+0000 7f998ffff700 1 -- 172.21.15.42:0/2900566787 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9990079250 con 0x7f99900a3a60 2019-12-05T06:39:22.091 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:22.044+0000 7f998ffff700 1 --2- 172.21.15.42:0/2900566787 >> v2:172.21.15.40:3300/0 conn(0x7f99900a3a60 0x7f99900797a0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f9990143f70 tx=0x7f99800087f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:22.091 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:22.048+0000 7f998ffff700 1 --2- 172.21.15.42:0/2900566787 >> v2:172.21.15.40:3300/0 conn(0x7f99900a3a60 0x7f99900797a0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f9990143f70 tx=0x7f99800087f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:22.091 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:22.048+0000 7f998ffff700 1 --2- 172.21.15.42:0/2900566787 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f99900a3a60 0x7f99900797a0 secure :-1 s=READY pgs=2209 cs=0 l=1 rx=0x7f9990143f70 tx=0x7f99800087f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:39:22.091 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:22.048+0000 7f998ffff700 1 --2- 172.21.15.42:0/2900566787 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f99900a3a60 0x7f99900797a0 secure :-1 s=THROTTLE_DONE pgs=2209 cs=0 l=1 rx=0x7f9990143f70 tx=0x7f99800087f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:22.091 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:22.048+0000 7f998cff9700 1 -- 172.21.15.42:0/2900566787 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f998000f760 con 0x7f99900a3a60 2019-12-05T06:39:22.091 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:22.048+0000 7f998ffff700 1 --2- 172.21.15.42:0/2900566787 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f99900a3a60 0x7f99900797a0 secure :-1 s=THROTTLE_DONE pgs=2209 cs=0 l=1 rx=0x7f9990143f70 tx=0x7f99800087f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:22.092 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:22.048+0000 7f998ffff700 1 --2- 172.21.15.42:0/2900566787 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f99900a3a60 0x7f99900797a0 secure :-1 s=THROTTLE_DONE pgs=2209 cs=0 l=1 rx=0x7f9990143f70 tx=0x7f99800087f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:22.092 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:22.048+0000 7f998cff9700 1 -- 172.21.15.42:0/2900566787 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9980011bd0 con 0x7f99900a3a60 2019-12-05T06:39:22.092 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:22.048+0000 7f9995a6b700 1 -- 172.21.15.42:0/2900566787 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99901431f0 con 0x7f99900a3a60 2019-12-05T06:39:22.092 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:22.048+0000 7f998cff9700 1 -- 172.21.15.42:0/2900566787 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9980005ab0 con 0x7f99900a3a60 2019-12-05T06:39:22.092 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:22.048+0000 7f9995a6b700 1 -- 172.21.15.42:0/2900566787 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f999012df40 con 0x7f99900a3a60 2019-12-05T06:39:22.092 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:22.048+0000 7f998ffff700 1 --2- 172.21.15.42:0/2900566787 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f99900a3a60 0x7f99900797a0 secure :-1 s=THROTTLE_DONE pgs=2209 cs=0 l=1 rx=0x7f9990143f70 tx=0x7f99800087f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:22.092 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:22.048+0000 7f998cff9700 1 -- 172.21.15.42:0/2900566787 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f9980007150 con 0x7f99900a3a60 2019-12-05T06:39:22.093 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:22.048+0000 7f998ffff700 1 --2- 172.21.15.42:0/2900566787 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f99900a3a60 0x7f99900797a0 secure :-1 s=THROTTLE_DONE pgs=2209 cs=0 l=1 rx=0x7f9990143f70 tx=0x7f99800087f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:22.093 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:22.048+0000 7f998cff9700 1 --2- 172.21.15.42:0/2900566787 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f997403acb0 0x7f997403d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:39:22.093 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:22.048+0000 7f998cff9700 1 -- 172.21.15.42:0/2900566787 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 11449+0+0 (secure 0 0 0) 0x7f998004b7b0 con 0x7f99900a3a60 2019-12-05T06:39:22.093 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:22.048+0000 7f998f7fe700 1 --2- 172.21.15.42:0/2900566787 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f997403acb0 0x7f997403d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:22.093 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:22.048+0000 7f998f7fe700 1 --2- 172.21.15.42:0/2900566787 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f997403acb0 0x7f997403d1c0 crc :-1 s=READY pgs=1221 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:39:22.093 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:22.052+0000 7f9995a6b700 1 -- 172.21.15.42:0/2900566787 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f999005d110 con 0x7f99900a3a60 2019-12-05T06:39:22.093 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:22.060+0000 7f998ffff700 1 --2- 172.21.15.42:0/2900566787 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f99900a3a60 0x7f99900797a0 secure :-1 s=THROTTLE_DONE pgs=2209 cs=0 l=1 rx=0x7f9990143f70 tx=0x7f99800087f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:22.094 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:22.060+0000 7f998cff9700 1 -- 172.21.15.42:0/2900566787 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f9980032830 con 0x7f99900a3a60 2019-12-05T06:39:22.235 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:22.228+0000 7f9995a6b700 1 -- 172.21.15.42:0/2900566787 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f999007c6f0 con 0x7f99900a3a60 2019-12-05T06:39:22.236 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:22.232+0000 7f998ffff700 1 --2- 172.21.15.42:0/2900566787 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f99900a3a60 0x7f99900797a0 secure :-1 s=THROTTLE_DONE pgs=2209 cs=0 l=1 rx=0x7f9990143f70 tx=0x7f99800087f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:22.236 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:22.232+0000 7f998cff9700 1 -- 172.21.15.42:0/2900566787 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 348 v348) v1 ==== 95+0+2113 (secure 0 0 0) 0x7f998000eda0 con 0x7f99900a3a60 2019-12-05T06:39:22.238 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 348 2019-12-05T06:39:22.239 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:39:22.239 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":348,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":8009,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/1777669201","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1777669201},{"type":"v1","addr":"172.21.15.40:6833","nonce":1777669201}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":348}],"filesystems":[{"mdsmap":{"epoch":347,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:38:00.190088+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11887},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11887":{"gid":11887,"name":"a","rank":0,"incarnation":344,"state":"up:active","state_seq":37,"addr":"172.21.15.42:6835/958796360","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":958796360},{"type":"v1","addr":"172.21.15.42:6835","nonce":958796360}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:39:22.241 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:22.236+0000 7f9995a6b700 1 -- 172.21.15.42:0/2900566787 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f997403acb0 msgr2=0x7f997403d1c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:22.241 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:22.236+0000 7f9995a6b700 1 --2- 172.21.15.42:0/2900566787 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f997403acb0 0x7f997403d1c0 crc :-1 s=READY pgs=1221 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:22.241 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:22.236+0000 7f9995a6b700 1 -- 172.21.15.42:0/2900566787 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f99900a3a60 msgr2=0x7f99900797a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:22.242 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:22.236+0000 7f9995a6b700 1 --2- 172.21.15.42:0/2900566787 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f99900a3a60 0x7f99900797a0 secure :-1 s=READY pgs=2209 cs=0 l=1 rx=0x7f9990143f70 tx=0x7f99800087f0).stop 2019-12-05T06:39:22.242 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:22.236+0000 7f9995a6b700 1 -- 172.21.15.42:0/2900566787 shutdown_connections 2019-12-05T06:39:22.242 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:22.236+0000 7f9995a6b700 1 --2- 172.21.15.42:0/2900566787 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f997403acb0 0x7f997403d1c0 unknown :-1 s=CLOSED pgs=1221 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:22.242 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:22.236+0000 7f9995a6b700 1 --2- 172.21.15.42:0/2900566787 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9990142870 0x7f9990142ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:22.242 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:22.236+0000 7f9995a6b700 1 --2- 172.21.15.42:0/2900566787 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f99900a3a60 0x7f99900797a0 unknown :-1 s=CLOSED pgs=2209 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:22.243 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:22.236+0000 7f9995a6b700 1 -- 172.21.15.42:0/2900566787 shutdown_connections 2019-12-05T06:39:22.243 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:22.236+0000 7f9995a6b700 1 -- 172.21.15.42:0/2900566787 wait complete. 2019-12-05T06:39:22.243 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:22.236+0000 7f9995a6b700 1 -- 172.21.15.42:0/2900566787 >> 172.21.15.42:0/2900566787 conn(0x7f99900a01a0 msgr2=0x7f99900a9640 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:39:22.252 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:39:24.253 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:39:24.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:24.344+0000 7fe217960700 1 Processor -- start 2019-12-05T06:39:24.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:24.344+0000 7fe217960700 1 -- start start 2019-12-05T06:39:24.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:24.344+0000 7fe217960700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe21008df80 0x7fe21009f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:39:24.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:24.344+0000 7fe217960700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe21009fc60 con 0x7fe2100a7740 2019-12-05T06:39:24.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:24.344+0000 7fe217960700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2100a0710 con 0x7fe21009f830 2019-12-05T06:39:24.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:24.344+0000 7fe217960700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2100a0b00 con 0x7fe21008df80 2019-12-05T06:39:24.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:24.344+0000 7fe2166fe700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fe21009f830 0x7fe21014efe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:60084/0 (socket says 172.21.15.42:60084) 2019-12-05T06:39:24.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:24.344+0000 7fe2166fe700 1 -- 172.21.15.42:0/3372992130 learned_addr learned my addr 172.21.15.42:0/3372992130 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:39:24.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:24.344+0000 7fe2156fc700 1 --2- 172.21.15.42:0/3372992130 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe21008df80 0x7fe21009f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:24.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:24.344+0000 7fe214efb700 1 -- 172.21.15.42:0/3372992130 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 233358437 0 0) 0x7fe2100a0710 con 0x7fe21009f830 2019-12-05T06:39:24.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:24.344+0000 7fe214efb700 1 -- 172.21.15.42:0/3372992130 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1fc0030b0 con 0x7fe21009f830 2019-12-05T06:39:24.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:24.344+0000 7fe214efb700 1 -- 172.21.15.42:0/3372992130 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3707039924 0 0) 0x7fe21009fc60 con 0x7fe2100a7740 2019-12-05T06:39:24.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:24.344+0000 7fe2156fc700 1 -- 172.21.15.42:0/3372992130 >> v1:172.21.15.42:6789/0 conn(0x7fe21009f830 legacy=0x7fe21014efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:24.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:24.344+0000 7fe2156fc700 1 -- 172.21.15.42:0/3372992130 >> v1:172.21.15.40:6789/0 conn(0x7fe2100a7740 legacy=0x7fe21007e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:24.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:24.344+0000 7fe2156fc700 1 -- 172.21.15.42:0/3372992130 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe21007ef20 con 0x7fe21008df80 2019-12-05T06:39:24.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:24.344+0000 7fe2156fc700 1 --2- 172.21.15.42:0/3372992130 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe21008df80 0x7fe21009f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fe204002f30 tx=0x7fe20400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:24.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:24.344+0000 7fe2156fc700 1 --2- 172.21.15.42:0/3372992130 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe21008df80 0x7fe21009f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fe204002f30 tx=0x7fe20400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:24.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:24.344+0000 7fe2156fc700 1 --2- 172.21.15.42:0/3372992130 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe21008df80 0x7fe21009f2e0 secure :-1 s=READY pgs=649 cs=0 l=1 rx=0x7fe204002f30 tx=0x7fe20400fbf0).ready entity=mon.2 client_cookie=f003a4d33e696652 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:39:24.358 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:24.348+0000 7fe2156fc700 1 --2- 172.21.15.42:0/3372992130 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe21008df80 0x7fe21009f2e0 secure :-1 s=THROTTLE_DONE pgs=649 cs=0 l=1 rx=0x7fe204002f30 tx=0x7fe20400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:24.358 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:24.348+0000 7fe214efb700 1 -- 172.21.15.42:0/3372992130 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe2040112b0 con 0x7fe21008df80 2019-12-05T06:39:24.358 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:24.348+0000 7fe2156fc700 1 --2- 172.21.15.42:0/3372992130 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe21008df80 0x7fe21009f2e0 secure :-1 s=THROTTLE_DONE pgs=649 cs=0 l=1 rx=0x7fe204002f30 tx=0x7fe20400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:24.359 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:24.348+0000 7fe2156fc700 1 --2- 172.21.15.42:0/3372992130 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe21008df80 0x7fe21009f2e0 secure :-1 s=THROTTLE_DONE pgs=649 cs=0 l=1 rx=0x7fe204002f30 tx=0x7fe20400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:24.359 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:24.348+0000 7fe214efb700 1 -- 172.21.15.42:0/3372992130 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe204008bc0 con 0x7fe21008df80 2019-12-05T06:39:24.359 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:24.348+0000 7fe214efb700 1 -- 172.21.15.42:0/3372992130 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe204011890 con 0x7fe21008df80 2019-12-05T06:39:24.359 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:24.348+0000 7fe217960700 1 -- 172.21.15.42:0/3372992130 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe21008df80 msgr2=0x7fe21009f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:24.360 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:24.348+0000 7fe217960700 1 --2- 172.21.15.42:0/3372992130 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe21008df80 0x7fe21009f2e0 secure :-1 s=READY pgs=649 cs=0 l=1 rx=0x7fe204002f30 tx=0x7fe20400fbf0).stop 2019-12-05T06:39:24.360 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:24.348+0000 7fe217960700 1 -- 172.21.15.42:0/3372992130 shutdown_connections 2019-12-05T06:39:24.360 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:24.348+0000 7fe217960700 1 --2- 172.21.15.42:0/3372992130 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe21008df80 0x7fe21009f2e0 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:24.360 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:24.348+0000 7fe217960700 1 -- 172.21.15.42:0/3372992130 shutdown_connections 2019-12-05T06:39:24.361 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:24.348+0000 7fe217960700 1 -- 172.21.15.42:0/3372992130 wait complete. 2019-12-05T06:39:24.361 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:24.348+0000 7fe217960700 1 -- 172.21.15.42:0/3372992130 >> 172.21.15.42:0/3372992130 conn(0x7fe2100aabe0 msgr2=0x7fe2100a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:39:24.366 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:24.348+0000 7fe217960700 1 Processor -- start 2019-12-05T06:39:24.366 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:24.348+0000 7fe217960700 1 -- start start 2019-12-05T06:39:24.366 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:24.348+0000 7fe217960700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe2101197a0 0x7fe210079b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:39:24.367 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:24.348+0000 7fe217960700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe21007a0a0 con 0x7fe210078560 2019-12-05T06:39:24.367 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:24.348+0000 7fe217960700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe21007b230 con 0x7fe21014f110 2019-12-05T06:39:24.367 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:24.348+0000 7fe217960700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe21007b460 con 0x7fe2101197a0 2019-12-05T06:39:24.367 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:24.348+0000 7fe215efd700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe2101197a0 0x7fe210079b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:24.367 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:24.348+0000 7fe215efd700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe2101197a0 0x7fe210079b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:60048/0 (socket says 172.21.15.42:60048) 2019-12-05T06:39:24.368 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:24.348+0000 7fe215efd700 1 -- 172.21.15.42:0/3995874684 learned_addr learned my addr 172.21.15.42:0/3995874684 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:39:24.368 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:24.352+0000 7fe2027fc700 1 -- 172.21.15.42:0/3995874684 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2695989624 0 0) 0x7fe21007a0a0 con 0x7fe210078560 2019-12-05T06:39:24.368 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:24.352+0000 7fe2027fc700 1 -- 172.21.15.42:0/3995874684 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1f80030b0 con 0x7fe210078560 2019-12-05T06:39:24.368 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:24.352+0000 7fe215efd700 1 -- 172.21.15.42:0/3995874684 >> v1:172.21.15.42:6789/0 conn(0x7fe21014f110 legacy=0x7fe21014f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:24.368 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:24.352+0000 7fe215efd700 1 -- 172.21.15.42:0/3995874684 >> v1:172.21.15.40:6789/0 conn(0x7fe210078560 legacy=0x7fe2100a35b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:24.369 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:24.352+0000 7fe215efd700 1 -- 172.21.15.42:0/3995874684 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe210130970 con 0x7fe2101197a0 2019-12-05T06:39:24.369 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:24.352+0000 7fe215efd700 1 --2- 172.21.15.42:0/3995874684 >> v2:172.21.15.42:3300/0 conn(0x7fe2101197a0 0x7fe210079b50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fe20c001460 tx=0x7fe20c0108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:24.369 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:24.352+0000 7fe215efd700 1 --2- 172.21.15.42:0/3995874684 >> v2:172.21.15.42:3300/0 conn(0x7fe2101197a0 0x7fe210079b50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fe20c001460 tx=0x7fe20c0108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:24.370 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:24.352+0000 7fe215efd700 1 --2- 172.21.15.42:0/3995874684 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe2101197a0 0x7fe210079b50 secure :-1 s=READY pgs=2914 cs=0 l=1 rx=0x7fe20c001460 tx=0x7fe20c0108f0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:39:24.370 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:24.352+0000 7fe215efd700 1 --2- 172.21.15.42:0/3995874684 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe2101197a0 0x7fe210079b50 secure :-1 s=THROTTLE_DONE pgs=2914 cs=0 l=1 rx=0x7fe20c001460 tx=0x7fe20c0108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:24.370 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:24.352+0000 7fe2027fc700 1 -- 172.21.15.42:0/3995874684 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe20c00ca80 con 0x7fe2101197a0 2019-12-05T06:39:24.370 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:24.352+0000 7fe215efd700 1 --2- 172.21.15.42:0/3995874684 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe2101197a0 0x7fe210079b50 secure :-1 s=THROTTLE_DONE pgs=2914 cs=0 l=1 rx=0x7fe20c001460 tx=0x7fe20c0108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:24.370 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:24.352+0000 7fe217960700 1 -- 172.21.15.42:0/3995874684 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe21012e950 con 0x7fe2101197a0 2019-12-05T06:39:24.370 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:24.352+0000 7fe2027fc700 1 -- 172.21.15.42:0/3995874684 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe20c00c6a0 con 0x7fe2101197a0 2019-12-05T06:39:24.371 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:24.352+0000 7fe215efd700 1 --2- 172.21.15.42:0/3995874684 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe2101197a0 0x7fe210079b50 secure :-1 s=THROTTLE_DONE pgs=2914 cs=0 l=1 rx=0x7fe20c001460 tx=0x7fe20c0108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:24.371 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:24.352+0000 7fe2027fc700 1 -- 172.21.15.42:0/3995874684 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe20c0110b0 con 0x7fe2101197a0 2019-12-05T06:39:24.371 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:24.352+0000 7fe217960700 1 -- 172.21.15.42:0/3995874684 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe21012eed0 con 0x7fe2101197a0 2019-12-05T06:39:24.371 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:24.352+0000 7fe215efd700 1 --2- 172.21.15.42:0/3995874684 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe2101197a0 0x7fe210079b50 secure :-1 s=THROTTLE_DONE pgs=2914 cs=0 l=1 rx=0x7fe20c001460 tx=0x7fe20c0108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:24.371 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:24.352+0000 7fe2027fc700 1 -- 172.21.15.42:0/3995874684 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fe20c011350 con 0x7fe2101197a0 2019-12-05T06:39:24.371 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:24.352+0000 7fe215efd700 1 --2- 172.21.15.42:0/3995874684 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe2101197a0 0x7fe210079b50 secure :-1 s=THROTTLE_DONE pgs=2914 cs=0 l=1 rx=0x7fe20c001460 tx=0x7fe20c0108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:24.372 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:24.352+0000 7fe2027fc700 1 --2- 172.21.15.42:0/3995874684 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe1f803b1c0 0x7fe1f803d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:39:24.372 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:24.356+0000 7fe2027fc700 1 -- 172.21.15.42:0/3995874684 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 11449+0+0 (secure 0 0 0) 0x7fe20c04b280 con 0x7fe2101197a0 2019-12-05T06:39:24.372 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:24.356+0000 7fe2156fc700 1 --2- 172.21.15.42:0/3995874684 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe1f803b1c0 0x7fe1f803d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:24.372 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:24.356+0000 7fe2156fc700 1 --2- 172.21.15.42:0/3995874684 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe1f803b1c0 0x7fe1f803d6d0 crc :-1 s=READY pgs=1222 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:39:24.372 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:24.360+0000 7fe217960700 1 -- 172.21.15.42:0/3995874684 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe21005d110 con 0x7fe2101197a0 2019-12-05T06:39:24.373 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:24.364+0000 7fe215efd700 1 --2- 172.21.15.42:0/3995874684 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe2101197a0 0x7fe210079b50 secure :-1 s=THROTTLE_DONE pgs=2914 cs=0 l=1 rx=0x7fe20c001460 tx=0x7fe20c0108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:24.373 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:24.364+0000 7fe2027fc700 1 -- 172.21.15.42:0/3995874684 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fe20c032300 con 0x7fe2101197a0 2019-12-05T06:39:24.539 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:24.532+0000 7fe217960700 1 -- 172.21.15.42:0/3995874684 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe21005b8c0 con 0x7fe2101197a0 2019-12-05T06:39:24.539 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:24.532+0000 7fe215efd700 1 --2- 172.21.15.42:0/3995874684 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe2101197a0 0x7fe210079b50 secure :-1 s=THROTTLE_DONE pgs=2914 cs=0 l=1 rx=0x7fe20c001460 tx=0x7fe20c0108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:24.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:24.532+0000 7fe2027fc700 1 -- 172.21.15.42:0/3995874684 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 348 v348) v1 ==== 95+0+2113 (secure 0 0 0) 0x7fe20c032560 con 0x7fe2101197a0 2019-12-05T06:39:24.540 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 348 2019-12-05T06:39:24.541 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:39:24.541 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":348,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":8009,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/1777669201","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1777669201},{"type":"v1","addr":"172.21.15.40:6833","nonce":1777669201}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":348}],"filesystems":[{"mdsmap":{"epoch":347,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:38:00.190088+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11887},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11887":{"gid":11887,"name":"a","rank":0,"incarnation":344,"state":"up:active","state_seq":37,"addr":"172.21.15.42:6835/958796360","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":958796360},{"type":"v1","addr":"172.21.15.42:6835","nonce":958796360}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:39:24.542 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:24.536+0000 7fe217960700 1 -- 172.21.15.42:0/3995874684 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe1f803b1c0 msgr2=0x7fe1f803d6d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:24.543 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:24.536+0000 7fe217960700 1 --2- 172.21.15.42:0/3995874684 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe1f803b1c0 0x7fe1f803d6d0 crc :-1 s=READY pgs=1222 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:24.543 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:24.536+0000 7fe217960700 1 -- 172.21.15.42:0/3995874684 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe2101197a0 msgr2=0x7fe210079b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:24.543 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:24.536+0000 7fe217960700 1 --2- 172.21.15.42:0/3995874684 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe2101197a0 0x7fe210079b50 secure :-1 s=READY pgs=2914 cs=0 l=1 rx=0x7fe20c001460 tx=0x7fe20c0108f0).stop 2019-12-05T06:39:24.543 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:24.536+0000 7fe217960700 1 -- 172.21.15.42:0/3995874684 shutdown_connections 2019-12-05T06:39:24.544 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:24.536+0000 7fe217960700 1 --2- 172.21.15.42:0/3995874684 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe1f803b1c0 0x7fe1f803d6d0 unknown :-1 s=CLOSED pgs=1222 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:24.544 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:24.536+0000 7fe217960700 1 --2- 172.21.15.42:0/3995874684 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe2101197a0 0x7fe210079b50 unknown :-1 s=CLOSED pgs=2914 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:24.544 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:24.536+0000 7fe217960700 1 -- 172.21.15.42:0/3995874684 shutdown_connections 2019-12-05T06:39:24.544 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:24.536+0000 7fe217960700 1 -- 172.21.15.42:0/3995874684 wait complete. 2019-12-05T06:39:24.544 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:24.536+0000 7fe217960700 1 -- 172.21.15.42:0/3995874684 >> 172.21.15.42:0/3995874684 conn(0x7fe2100a9640 msgr2=0x7fe21007e600 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:39:24.554 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:39:26.555 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:39:26.666 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:26.652+0000 7f8231411700 1 Processor -- start 2019-12-05T06:39:26.667 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:26.652+0000 7f8231411700 1 -- start start 2019-12-05T06:39:26.667 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:26.656+0000 7f8231411700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f822c0a7740 0x7f822c08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:39:26.667 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:26.656+0000 7f8231411700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f822c07e720 0x7f822c07eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:39:26.667 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:26.656+0000 7f8231411700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f822c09f2e0 0x7f822c09f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:39:26.668 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:26.656+0000 7f8231411700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f822c0a0b00 con 0x7f822c0a7740 2019-12-05T06:39:26.668 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:26.656+0000 7f8231411700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f822c08e5b0 con 0x7f822c09f2e0 2019-12-05T06:39:26.668 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:26.656+0000 7f8231411700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f822c08e850 con 0x7f822c07e720 2019-12-05T06:39:26.671 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:26.656+0000 7f822bfff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f822c09f2e0 0x7f822c09f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:26.672 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:26.656+0000 7f822bfff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f822c09f2e0 0x7f822c09f710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:60064/0 (socket says 172.21.15.42:60064) 2019-12-05T06:39:26.672 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:26.656+0000 7f822bfff700 1 -- 172.21.15.42:0/1305560984 learned_addr learned my addr 172.21.15.42:0/1305560984 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:39:26.672 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:26.656+0000 7f822affd700 1 --2- 172.21.15.42:0/1305560984 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f822c07e720 0x7f822c07eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:26.672 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:26.656+0000 7f822b7fe700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f822c0a7740 0x7f822c08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:26.673 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:26.656+0000 7f822bfff700 1 -- 172.21.15.42:0/1305560984 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f822c07e720 msgr2=0x7f822c07eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:39:26.673 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:26.656+0000 7f822bfff700 1 --2- 172.21.15.42:0/1305560984 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f822c07e720 0x7f822c07eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:39:26.673 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:26.656+0000 7f822bfff700 1 -- 172.21.15.42:0/1305560984 >> v2:172.21.15.40:3300/0 conn(0x7f822c0a7740 msgr2=0x7f822c08df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:39:26.673 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:26.656+0000 7f822bfff700 1 --2- 172.21.15.42:0/1305560984 >> v2:172.21.15.40:3300/0 conn(0x7f822c0a7740 0x7f822c08df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:39:26.674 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:26.656+0000 7f822bfff700 1 -- 172.21.15.42:0/1305560984 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f822c07f0a0 con 0x7f822c09f2e0 2019-12-05T06:39:26.674 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:26.656+0000 7f822bfff700 1 --2- 172.21.15.42:0/1305560984 >> v2:172.21.15.42:3300/0 conn(0x7f822c09f2e0 0x7f822c09f710 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f822c08e770 tx=0x7f822400fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:26.674 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:26.656+0000 7f822bfff700 1 --2- 172.21.15.42:0/1305560984 >> v2:172.21.15.42:3300/0 conn(0x7f822c09f2e0 0x7f822c09f710 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f822c08e770 tx=0x7f822400fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:26.674 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:26.656+0000 7f822bfff700 1 --2- 172.21.15.42:0/1305560984 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f822c09f2e0 0x7f822c09f710 secure :-1 s=READY pgs=2915 cs=0 l=1 rx=0x7f822c08e770 tx=0x7f822400fdb0).ready entity=mon.1 client_cookie=9c12c2cf371eea1e server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:39:26.674 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:26.656+0000 7f822bfff700 1 --2- 172.21.15.42:0/1305560984 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f822c09f2e0 0x7f822c09f710 secure :-1 s=THROTTLE_DONE pgs=2915 cs=0 l=1 rx=0x7f822c08e770 tx=0x7f822400fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:26.675 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:26.656+0000 7f822a7fc700 1 -- 172.21.15.42:0/1305560984 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8224008c90 con 0x7f822c09f2e0 2019-12-05T06:39:26.675 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:26.656+0000 7f822bfff700 1 --2- 172.21.15.42:0/1305560984 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f822c09f2e0 0x7f822c09f710 secure :-1 s=THROTTLE_DONE pgs=2915 cs=0 l=1 rx=0x7f822c08e770 tx=0x7f822400fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:26.675 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:26.656+0000 7f822bfff700 1 --2- 172.21.15.42:0/1305560984 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f822c09f2e0 0x7f822c09f710 secure :-1 s=THROTTLE_DONE pgs=2915 cs=0 l=1 rx=0x7f822c08e770 tx=0x7f822400fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:26.675 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:26.656+0000 7f822a7fc700 1 -- 172.21.15.42:0/1305560984 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82240088b0 con 0x7f822c09f2e0 2019-12-05T06:39:26.676 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:26.656+0000 7f822a7fc700 1 -- 172.21.15.42:0/1305560984 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8224011b50 con 0x7f822c09f2e0 2019-12-05T06:39:26.676 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:26.656+0000 7f8231411700 1 -- 172.21.15.42:0/1305560984 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f822c09f2e0 msgr2=0x7f822c09f710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:26.676 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:26.656+0000 7f8231411700 1 --2- 172.21.15.42:0/1305560984 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f822c09f2e0 0x7f822c09f710 secure :-1 s=READY pgs=2915 cs=0 l=1 rx=0x7f822c08e770 tx=0x7f822400fdb0).stop 2019-12-05T06:39:26.677 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:26.656+0000 7f8231411700 1 -- 172.21.15.42:0/1305560984 shutdown_connections 2019-12-05T06:39:26.677 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:26.656+0000 7f8231411700 1 --2- 172.21.15.42:0/1305560984 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f822c09f2e0 0x7f822c09f710 unknown :-1 s=CLOSED pgs=2915 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:26.677 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:26.656+0000 7f8231411700 1 --2- 172.21.15.42:0/1305560984 >> v2:172.21.15.40:3300/0 conn(0x7f822c0a7740 0x7f822c08df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:39:26.677 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:26.656+0000 7f8231411700 1 --2- 172.21.15.42:0/1305560984 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f822c07e720 0x7f822c07eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:39:26.677 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:26.656+0000 7f8231411700 1 -- 172.21.15.42:0/1305560984 shutdown_connections 2019-12-05T06:39:26.678 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:26.656+0000 7f8231411700 1 -- 172.21.15.42:0/1305560984 wait complete. 2019-12-05T06:39:26.678 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:26.656+0000 7f8231411700 1 -- 172.21.15.42:0/1305560984 >> 172.21.15.42:0/1305560984 conn(0x7f822c0aabe0 msgr2=0x7f822c0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:39:26.678 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:26.660+0000 7f8231411700 1 Processor -- start 2019-12-05T06:39:26.678 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:26.660+0000 7f8231411700 1 -- start start 2019-12-05T06:39:26.685 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:26.660+0000 7f8231411700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f822c11d970 0x7f822c0a9640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:39:26.685 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:26.660+0000 7f8231411700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f822c1236f0 0x7f822c123b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:39:26.685 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:26.660+0000 7f8231411700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f822c0a9b90 con 0x7f822c131630 2019-12-05T06:39:26.685 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:26.660+0000 7f8231411700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f822c121350 con 0x7f822c11d970 2019-12-05T06:39:26.686 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:26.660+0000 7f8231411700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f822c118c00 con 0x7f822c1236f0 2019-12-05T06:39:26.686 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:26.664+0000 7f822b7fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f822c11d970 0x7f822c0a9640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:26.686 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:26.664+0000 7f822b7fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f822c11d970 0x7f822c0a9640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:60068/0 (socket says 172.21.15.42:60068) 2019-12-05T06:39:26.686 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:26.664+0000 7f822b7fe700 1 -- 172.21.15.42:0/3016208223 learned_addr learned my addr 172.21.15.42:0/3016208223 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:39:26.687 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:26.664+0000 7f822affd700 1 --2- 172.21.15.42:0/3016208223 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f822c1236f0 0x7f822c123b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:26.687 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:26.664+0000 7f8213fff700 1 -- 172.21.15.42:0/3016208223 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3671233592 0 0) 0x7f822c0a9b90 con 0x7f822c131630 2019-12-05T06:39:26.687 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:26.664+0000 7f822b7fe700 1 -- 172.21.15.42:0/3016208223 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f822c1236f0 msgr2=0x7f822c123b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:26.687 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:26.664+0000 7f822b7fe700 1 --2- 172.21.15.42:0/3016208223 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f822c1236f0 0x7f822c123b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:26.687 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:26.664+0000 7f822b7fe700 1 -- 172.21.15.42:0/3016208223 >> v1:172.21.15.40:6789/0 conn(0x7f822c131630 legacy=0x7f822c131a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:26.688 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:26.664+0000 7f822b7fe700 1 -- 172.21.15.42:0/3016208223 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f822c124070 con 0x7f822c11d970 2019-12-05T06:39:26.688 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:26.664+0000 7f822b7fe700 1 --2- 172.21.15.42:0/3016208223 >> v2:172.21.15.42:3300/0 conn(0x7f822c11d970 0x7f822c0a9640 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f821c0033f0 tx=0x7f821c010f60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:26.688 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:26.664+0000 7f822b7fe700 1 --2- 172.21.15.42:0/3016208223 >> v2:172.21.15.42:3300/0 conn(0x7f822c11d970 0x7f822c0a9640 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f821c0033f0 tx=0x7f821c010f60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:26.688 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:26.664+0000 7f822b7fe700 1 --2- 172.21.15.42:0/3016208223 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f822c11d970 0x7f822c0a9640 secure :-1 s=READY pgs=2916 cs=0 l=1 rx=0x7f821c0033f0 tx=0x7f821c010f60).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:39:26.689 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:26.664+0000 7f822b7fe700 1 --2- 172.21.15.42:0/3016208223 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f822c11d970 0x7f822c0a9640 secure :-1 s=THROTTLE_DONE pgs=2916 cs=0 l=1 rx=0x7f821c0033f0 tx=0x7f821c010f60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:26.689 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:26.664+0000 7f8213fff700 1 -- 172.21.15.42:0/3016208223 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f821c007350 con 0x7f822c11d970 2019-12-05T06:39:26.689 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:26.664+0000 7f822b7fe700 1 --2- 172.21.15.42:0/3016208223 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f822c11d970 0x7f822c0a9640 secure :-1 s=THROTTLE_DONE pgs=2916 cs=0 l=1 rx=0x7f821c0033f0 tx=0x7f821c010f60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:26.689 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:26.664+0000 7f822b7fe700 1 --2- 172.21.15.42:0/3016208223 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f822c11d970 0x7f822c0a9640 secure :-1 s=THROTTLE_DONE pgs=2916 cs=0 l=1 rx=0x7f821c0033f0 tx=0x7f821c010f60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:26.690 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:26.664+0000 7f8213fff700 1 -- 172.21.15.42:0/3016208223 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f821c005cb0 con 0x7f822c11d970 2019-12-05T06:39:26.690 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:26.664+0000 7f8231411700 1 -- 172.21.15.42:0/3016208223 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f822c1193e0 con 0x7f822c11d970 2019-12-05T06:39:26.690 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:26.664+0000 7f8213fff700 1 -- 172.21.15.42:0/3016208223 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f821c0127d0 con 0x7f822c11d970 2019-12-05T06:39:26.690 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:26.664+0000 7f8231411700 1 -- 172.21.15.42:0/3016208223 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f822c120340 con 0x7f822c11d970 2019-12-05T06:39:26.690 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:26.664+0000 7f822b7fe700 1 --2- 172.21.15.42:0/3016208223 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f822c11d970 0x7f822c0a9640 secure :-1 s=THROTTLE_DONE pgs=2916 cs=0 l=1 rx=0x7f821c0033f0 tx=0x7f821c010f60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:26.691 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:26.664+0000 7f8213fff700 1 -- 172.21.15.42:0/3016208223 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f821c010180 con 0x7f822c11d970 2019-12-05T06:39:26.691 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:26.664+0000 7f822b7fe700 1 --2- 172.21.15.42:0/3016208223 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f822c11d970 0x7f822c0a9640 secure :-1 s=THROTTLE_DONE pgs=2916 cs=0 l=1 rx=0x7f821c0033f0 tx=0x7f821c010f60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:26.691 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:26.668+0000 7f8213fff700 1 --2- 172.21.15.42:0/3016208223 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f820c03ad10 0x7f820c03d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:39:26.691 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:26.668+0000 7f8213fff700 1 -- 172.21.15.42:0/3016208223 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 11449+0+0 (secure 0 0 0) 0x7f821c0090a0 con 0x7f822c11d970 2019-12-05T06:39:26.691 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:26.668+0000 7f822affd700 1 --2- 172.21.15.42:0/3016208223 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f820c03ad10 0x7f820c03d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:26.692 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:26.668+0000 7f822affd700 1 --2- 172.21.15.42:0/3016208223 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f820c03ad10 0x7f820c03d220 crc :-1 s=READY pgs=1223 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:39:26.692 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:26.668+0000 7f8231411700 1 -- 172.21.15.42:0/3016208223 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f822c05d110 con 0x7f822c11d970 2019-12-05T06:39:26.692 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:26.672+0000 7f822b7fe700 1 --2- 172.21.15.42:0/3016208223 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f822c11d970 0x7f822c0a9640 secure :-1 s=THROTTLE_DONE pgs=2916 cs=0 l=1 rx=0x7f821c0033f0 tx=0x7f821c010f60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:26.692 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:26.672+0000 7f8213fff700 1 -- 172.21.15.42:0/3016208223 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f822c124070 con 0x7f822c11d970 2019-12-05T06:39:26.845 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:26.840+0000 7f8231411700 1 -- 172.21.15.42:0/3016208223 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f822c08faf0 con 0x7f822c11d970 2019-12-05T06:39:26.862 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:26.840+0000 7f822b7fe700 1 --2- 172.21.15.42:0/3016208223 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f822c11d970 0x7f822c0a9640 secure :-1 s=THROTTLE_DONE pgs=2916 cs=0 l=1 rx=0x7f821c0033f0 tx=0x7f821c010f60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:26.862 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:26.840+0000 7f8213fff700 1 -- 172.21.15.42:0/3016208223 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 348 v348) v1 ==== 95+0+2113 (secure 0 0 0) 0x7f821c00ede0 con 0x7f822c11d970 2019-12-05T06:39:26.862 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 348 2019-12-05T06:39:26.863 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:26.840+0000 7f8231411700 1 -- 172.21.15.42:0/3016208223 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f820c03ad10 msgr2=0x7f820c03d220 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:26.863 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:26.840+0000 7f8231411700 1 --2- 172.21.15.42:0/3016208223 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f820c03ad10 0x7f820c03d220 crc :-1 s=READY pgs=1223 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:26.863 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:26.840+0000 7f8231411700 1 -- 172.21.15.42:0/3016208223 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f822c11d970 msgr2=0x7f822c0a9640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:26.863 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:26.840+0000 7f8231411700 1 --2- 172.21.15.42:0/3016208223 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f822c11d970 0x7f822c0a9640 secure :-1 s=READY pgs=2916 cs=0 l=1 rx=0x7f821c0033f0 tx=0x7f821c010f60).stop 2019-12-05T06:39:26.863 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:26.844+0000 7f8231411700 1 -- 172.21.15.42:0/3016208223 shutdown_connections 2019-12-05T06:39:26.864 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:26.844+0000 7f8231411700 1 --2- 172.21.15.42:0/3016208223 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f820c03ad10 0x7f820c03d220 unknown :-1 s=CLOSED pgs=1223 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:26.864 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:26.844+0000 7f8231411700 1 --2- 172.21.15.42:0/3016208223 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f822c11d970 0x7f822c0a9640 unknown :-1 s=CLOSED pgs=2916 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:26.864 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:26.844+0000 7f8231411700 1 --2- 172.21.15.42:0/3016208223 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f822c1236f0 0x7f822c123b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:26.864 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:26.844+0000 7f8231411700 1 -- 172.21.15.42:0/3016208223 shutdown_connections 2019-12-05T06:39:26.864 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:26.844+0000 7f8231411700 1 -- 172.21.15.42:0/3016208223 wait complete. 2019-12-05T06:39:26.864 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:26.844+0000 7f8231411700 1 -- 172.21.15.42:0/3016208223 >> 172.21.15.42:0/3016208223 conn(0x7f822c0a82a0 msgr2=0x7f822c07e4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:39:26.865 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:39:26.865 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":348,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":8009,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/1777669201","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1777669201},{"type":"v1","addr":"172.21.15.40:6833","nonce":1777669201}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":348}],"filesystems":[{"mdsmap":{"epoch":347,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:38:00.190088+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11887},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11887":{"gid":11887,"name":"a","rank":0,"incarnation":344,"state":"up:active","state_seq":37,"addr":"172.21.15.42:6835/958796360","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":958796360},{"type":"v1","addr":"172.21.15.42:6835","nonce":958796360}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:39:26.865 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:39:28.868 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:39:29.007 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:28.972+0000 7f29defdc700 1 Processor -- start 2019-12-05T06:39:29.008 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:28.972+0000 7f29defdc700 1 -- start start 2019-12-05T06:39:29.008 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:28.976+0000 7f29defdc700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f29d808df80 0x7f29d809f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:39:29.008 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:28.976+0000 7f29defdc700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f29d809f830 0x7f29d814efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:39:29.009 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:28.976+0000 7f29defdc700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29d809fc60 con 0x7f29d80a7740 2019-12-05T06:39:29.009 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:28.976+0000 7f29defdc700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29d80a0b00 con 0x7f29d809f830 2019-12-05T06:39:29.009 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:28.976+0000 7f29defdc700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29d808e5b0 con 0x7f29d808df80 2019-12-05T06:39:29.009 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:28.976+0000 7f29dd579700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f29d80a7740 0x7f29d807e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:60116/0 (socket says 172.21.15.42:60116) 2019-12-05T06:39:29.010 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:28.976+0000 7f29dd579700 1 -- 172.21.15.42:0/1532671380 learned_addr learned my addr 172.21.15.42:0/1532671380 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:39:29.010 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:28.976+0000 7f29dcd78700 1 --2- 172.21.15.42:0/1532671380 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f29d808df80 0x7f29d809f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:29.010 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:28.976+0000 7f29ddd7a700 1 --2- 172.21.15.42:0/1532671380 >> v2:172.21.15.40:3300/0 conn(0x7f29d809f830 0x7f29d814efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:29.010 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:28.976+0000 7f29cffff700 1 -- 172.21.15.42:0/1532671380 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 540970069 0 0) 0x7f29d809fc60 con 0x7f29d80a7740 2019-12-05T06:39:29.011 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:28.976+0000 7f29cffff700 1 -- 172.21.15.42:0/1532671380 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29c40030b0 con 0x7f29d80a7740 2019-12-05T06:39:29.011 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:28.976+0000 7f29cffff700 1 -- 172.21.15.42:0/1532671380 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2775688731 0 0) 0x7f29c40030b0 con 0x7f29d80a7740 2019-12-05T06:39:29.011 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:28.976+0000 7f29cffff700 1 -- 172.21.15.42:0/1532671380 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f29d808df80 msgr2=0x7f29d809f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:39:29.011 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:28.976+0000 7f29cffff700 1 --2- 172.21.15.42:0/1532671380 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f29d808df80 0x7f29d809f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:39:29.012 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:28.976+0000 7f29cffff700 1 -- 172.21.15.42:0/1532671380 >> v2:172.21.15.40:3300/0 conn(0x7f29d809f830 msgr2=0x7f29d814efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:39:29.012 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:28.976+0000 7f29cffff700 1 --2- 172.21.15.42:0/1532671380 >> v2:172.21.15.40:3300/0 conn(0x7f29d809f830 0x7f29d814efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:39:29.012 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:28.976+0000 7f29cffff700 1 -- 172.21.15.42:0/1532671380 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29d80a0710 con 0x7f29d80a7740 2019-12-05T06:39:29.012 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:28.976+0000 7f29cffff700 1 -- 172.21.15.42:0/1532671380 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7f29d4001750 con 0x7f29d80a7740 2019-12-05T06:39:29.013 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:28.976+0000 7f29dcd78700 1 --2- 172.21.15.42:0/1532671380 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f29d808df80 0x7f29d809f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:39:29.013 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:28.976+0000 7f29cffff700 1 -- 172.21.15.42:0/1532671380 >> v1:172.21.15.42:6789/0 conn(0x7f29d80a7740 legacy=0x7f29d807e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:29.014 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:28.976+0000 7f29cffff700 1 --2- 172.21.15.42:0/1532671380 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f29c40036e0 0x7f29c4003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:39:29.014 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:28.976+0000 7f29cffff700 1 --2- 172.21.15.42:0/1532671380 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f29c4004420 0x7f29c4004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:39:29.015 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:28.976+0000 7f29cffff700 1 --2- 172.21.15.42:0/1532671380 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f29c4004da0 0x7f29c404f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:39:29.015 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:28.976+0000 7f29cffff700 1 -- 172.21.15.42:0/1532671380 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29d80a0b00 con 0x7f29c4004da0 2019-12-05T06:39:29.015 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:28.976+0000 7f29cffff700 1 -- 172.21.15.42:0/1532671380 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29d808e5b0 con 0x7f29c40036e0 2019-12-05T06:39:29.015 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:28.976+0000 7f29cffff700 1 -- 172.21.15.42:0/1532671380 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29c404fbd0 con 0x7f29c4004420 2019-12-05T06:39:29.016 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:28.976+0000 7f29dd579700 1 --2- 172.21.15.42:0/1532671380 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f29c40036e0 0x7f29c4003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:29.016 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:28.976+0000 7f29ddd7a700 1 --2- 172.21.15.42:0/1532671380 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f29c4004da0 0x7f29c404f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:29.016 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:28.976+0000 7f29dcd78700 1 --2- 172.21.15.42:0/1532671380 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f29c4004420 0x7f29c4004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:29.016 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:28.976+0000 7f29dcd78700 1 -- 172.21.15.42:0/1532671380 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f29c4004420 msgr2=0x7f29c4004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2019-12-05T06:39:29.017 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:28.976+0000 7f29dcd78700 1 -- 172.21.15.42:0/1532671380 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f29c4004420 msgr2=0x7f29c4004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2019-12-05T06:39:29.017 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:28.976+0000 7f29dcd78700 1 --2- 172.21.15.42:0/1532671380 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f29c4004420 0x7f29c4004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-12-05T06:39:29.017 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:28.976+0000 7f29dcd78700 1 --2- 172.21.15.42:0/1532671380 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f29c4004420 0x7f29c4004850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rx=0 tx=0)._fault waiting 0.200000 2019-12-05T06:39:29.017 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:28.976+0000 7f29dd579700 1 -- 172.21.15.42:0/1532671380 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f29c4004420 msgr2=0x7f29c4004850 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2019-12-05T06:39:29.018 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:28.976+0000 7f29dd579700 1 --2- 172.21.15.42:0/1532671380 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f29c4004420 0x7f29c4004850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:39:29.018 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:28.976+0000 7f29dd579700 1 -- 172.21.15.42:0/1532671380 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f29c4004da0 msgr2=0x7f29c404f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:39:29.018 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:28.976+0000 7f29dd579700 1 --2- 172.21.15.42:0/1532671380 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f29c4004da0 0x7f29c404f680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:39:29.018 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:28.976+0000 7f29dd579700 1 -- 172.21.15.42:0/1532671380 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29c404fd90 con 0x7f29c40036e0 2019-12-05T06:39:29.019 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:28.976+0000 7f29dd579700 1 --2- 172.21.15.42:0/1532671380 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f29c40036e0 0x7f29c4003b50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f29d4001a50 tx=0x7f29d400c220).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:29.019 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:28.976+0000 7f29dd579700 1 --2- 172.21.15.42:0/1532671380 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f29c40036e0 0x7f29c4003b50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f29d4001a50 tx=0x7f29d400c220).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:29.019 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:28.976+0000 7f29dd579700 1 --2- 172.21.15.42:0/1532671380 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f29c40036e0 0x7f29c4003b50 secure :-1 s=READY pgs=2918 cs=0 l=1 rx=0x7f29d4001a50 tx=0x7f29d400c220).ready entity=mon.1 client_cookie=dc18d7fdd5b8dbad server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:39:29.019 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:28.976+0000 7f29dd579700 1 --2- 172.21.15.42:0/1532671380 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f29c40036e0 0x7f29c4003b50 secure :-1 s=THROTTLE_DONE pgs=2918 cs=0 l=1 rx=0x7f29d4001a50 tx=0x7f29d400c220).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:29.020 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:28.976+0000 7f29cffff700 1 -- 172.21.15.42:0/1532671380 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f29d4010350 con 0x7f29c40036e0 2019-12-05T06:39:29.020 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:28.976+0000 7f29dd579700 1 --2- 172.21.15.42:0/1532671380 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f29c40036e0 0x7f29c4003b50 secure :-1 s=THROTTLE_DONE pgs=2918 cs=0 l=1 rx=0x7f29d4001a50 tx=0x7f29d400c220).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:29.026 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:28.976+0000 7f29cffff700 1 -- 172.21.15.42:0/1532671380 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29d400edf0 con 0x7f29c40036e0 2019-12-05T06:39:29.026 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:28.976+0000 7f29dd579700 1 --2- 172.21.15.42:0/1532671380 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f29c40036e0 0x7f29c4003b50 secure :-1 s=THROTTLE_DONE pgs=2918 cs=0 l=1 rx=0x7f29d4001a50 tx=0x7f29d400c220).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:29.027 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:28.976+0000 7f29cffff700 1 -- 172.21.15.42:0/1532671380 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f29d4010930 con 0x7f29c40036e0 2019-12-05T06:39:29.027 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:28.976+0000 7f29defdc700 1 -- 172.21.15.42:0/1532671380 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f29c40036e0 msgr2=0x7f29c4003b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:29.027 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:28.976+0000 7f29defdc700 1 --2- 172.21.15.42:0/1532671380 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f29c40036e0 0x7f29c4003b50 secure :-1 s=READY pgs=2918 cs=0 l=1 rx=0x7f29d4001a50 tx=0x7f29d400c220).stop 2019-12-05T06:39:29.027 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:28.976+0000 7f29ddd7a700 1 -- 172.21.15.42:0/1532671380 reap_dead start 2019-12-05T06:39:29.028 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:28.980+0000 7f29defdc700 1 -- 172.21.15.42:0/1532671380 shutdown_connections 2019-12-05T06:39:29.028 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:28.980+0000 7f29defdc700 1 -- 172.21.15.42:0/1532671380 shutdown_connections 2019-12-05T06:39:29.028 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:28.980+0000 7f29defdc700 1 -- 172.21.15.42:0/1532671380 wait complete. 2019-12-05T06:39:29.028 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:28.980+0000 7f29defdc700 1 -- 172.21.15.42:0/1532671380 >> 172.21.15.42:0/1532671380 conn(0x7f29d80aabe0 msgr2=0x7f29d80a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:39:29.028 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:28.980+0000 7f29defdc700 1 Processor -- start 2019-12-05T06:39:29.029 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:28.984+0000 7f29defdc700 1 -- start start 2019-12-05T06:39:29.029 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:28.984+0000 7f29defdc700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f29d80a9980 0x7f29d8116680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:39:29.029 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:28.984+0000 7f29defdc700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f29d8117df0 0x7f29d8111db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:39:29.029 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:28.984+0000 7f29defdc700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f29d8112300 0x7f29d8112730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:39:29.029 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:28.984+0000 7f29defdc700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29d811a770 con 0x7f29d8117df0 2019-12-05T06:39:29.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:28.984+0000 7f29defdc700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29d81178b0 con 0x7f29d80a9980 2019-12-05T06:39:29.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:28.984+0000 7f29defdc700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29d8117b50 con 0x7f29d8112300 2019-12-05T06:39:29.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:28.984+0000 7f29dd579700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f29d80a9980 0x7f29d8116680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:29.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:28.984+0000 7f29dd579700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f29d80a9980 0x7f29d8116680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:60090/0 (socket says 172.21.15.42:60090) 2019-12-05T06:39:29.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:28.984+0000 7f29dd579700 1 -- 172.21.15.42:0/2846557641 learned_addr learned my addr 172.21.15.42:0/2846557641 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:39:29.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:28.984+0000 7f29ddd7a700 1 --2- 172.21.15.42:0/2846557641 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f29d8112300 0x7f29d8112730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:29.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:28.984+0000 7f29dcd78700 1 --2- 172.21.15.42:0/2846557641 >> v2:172.21.15.40:3300/0 conn(0x7f29d8117df0 0x7f29d8111db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:29.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:28.984+0000 7f29dd579700 1 -- 172.21.15.42:0/2846557641 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f29d8112300 msgr2=0x7f29d8112730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:29.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:28.984+0000 7f29dd579700 1 --2- 172.21.15.42:0/2846557641 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f29d8112300 0x7f29d8112730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:29.032 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:28.984+0000 7f29dd579700 1 -- 172.21.15.42:0/2846557641 >> v2:172.21.15.40:3300/0 conn(0x7f29d8117df0 msgr2=0x7f29d8111db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:29.032 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:28.984+0000 7f29dd579700 1 --2- 172.21.15.42:0/2846557641 >> v2:172.21.15.40:3300/0 conn(0x7f29d8117df0 0x7f29d8111db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:29.032 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:28.984+0000 7f29dd579700 1 -- 172.21.15.42:0/2846557641 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29d80a4840 con 0x7f29d80a9980 2019-12-05T06:39:29.032 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:28.984+0000 7f29ddd7a700 1 --2- 172.21.15.42:0/2846557641 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f29d8112300 0x7f29d8112730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:39:29.032 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:28.984+0000 7f29dd579700 1 --2- 172.21.15.42:0/2846557641 >> v2:172.21.15.42:3300/0 conn(0x7f29d80a9980 0x7f29d8116680 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f29d811a930 tx=0x7f29d4010160).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:29.033 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:28.984+0000 7f29dd579700 1 --2- 172.21.15.42:0/2846557641 >> v2:172.21.15.42:3300/0 conn(0x7f29d80a9980 0x7f29d8116680 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f29d811a930 tx=0x7f29d4010160).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:29.033 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:28.984+0000 7f29dd579700 1 --2- 172.21.15.42:0/2846557641 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f29d80a9980 0x7f29d8116680 secure :-1 s=READY pgs=2919 cs=0 l=1 rx=0x7f29d811a930 tx=0x7f29d4010160).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:39:29.033 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:28.984+0000 7f29dd579700 1 --2- 172.21.15.42:0/2846557641 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f29d80a9980 0x7f29d8116680 secure :-1 s=THROTTLE_DONE pgs=2919 cs=0 l=1 rx=0x7f29d811a930 tx=0x7f29d4010160).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:29.033 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:28.984+0000 7f29cdffb700 1 -- 172.21.15.42:0/2846557641 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f29d4004400 con 0x7f29d80a9980 2019-12-05T06:39:29.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:28.984+0000 7f29dd579700 1 --2- 172.21.15.42:0/2846557641 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f29d80a9980 0x7f29d8116680 secure :-1 s=THROTTLE_DONE pgs=2919 cs=0 l=1 rx=0x7f29d811a930 tx=0x7f29d4010160).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:29.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:28.984+0000 7f29dd579700 1 --2- 172.21.15.42:0/2846557641 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f29d80a9980 0x7f29d8116680 secure :-1 s=THROTTLE_DONE pgs=2919 cs=0 l=1 rx=0x7f29d811a930 tx=0x7f29d4010160).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:29.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:28.984+0000 7f29cdffb700 1 -- 172.21.15.42:0/2846557641 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29d4010550 con 0x7f29d80a9980 2019-12-05T06:39:29.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:28.984+0000 7f29cdffb700 1 -- 172.21.15.42:0/2846557641 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f29d40049e0 con 0x7f29d80a9980 2019-12-05T06:39:29.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:28.984+0000 7f29defdc700 1 -- 172.21.15.42:0/2846557641 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29d811b350 con 0x7f29d80a9980 2019-12-05T06:39:29.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:28.984+0000 7f29defdc700 1 -- 172.21.15.42:0/2846557641 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29d811af20 con 0x7f29d80a9980 2019-12-05T06:39:29.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:28.988+0000 7f29dd579700 1 --2- 172.21.15.42:0/2846557641 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f29d80a9980 0x7f29d8116680 secure :-1 s=THROTTLE_DONE pgs=2919 cs=0 l=1 rx=0x7f29d811a930 tx=0x7f29d4010160).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:29.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:28.988+0000 7f29cdffb700 1 -- 172.21.15.42:0/2846557641 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f29d40152b0 con 0x7f29d80a9980 2019-12-05T06:39:29.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:28.988+0000 7f29dd579700 1 --2- 172.21.15.42:0/2846557641 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f29d80a9980 0x7f29d8116680 secure :-1 s=THROTTLE_DONE pgs=2919 cs=0 l=1 rx=0x7f29d811a930 tx=0x7f29d4010160).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:29.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:28.988+0000 7f29cdffb700 1 --2- 172.21.15.42:0/2846557641 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f29b803acf0 0x7f29b803d200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:39:29.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:28.988+0000 7f29cdffb700 1 -- 172.21.15.42:0/2846557641 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 11449+0+0 (secure 0 0 0) 0x7f29d404b3e0 con 0x7f29d80a9980 2019-12-05T06:39:29.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:28.988+0000 7f29dcd78700 1 --2- 172.21.15.42:0/2846557641 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f29b803acf0 0x7f29b803d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:29.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:28.988+0000 7f29dcd78700 1 --2- 172.21.15.42:0/2846557641 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f29b803acf0 0x7f29b803d200 crc :-1 s=READY pgs=1224 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:39:29.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:28.988+0000 7f29defdc700 1 -- 172.21.15.42:0/2846557641 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29d805d110 con 0x7f29d80a9980 2019-12-05T06:39:29.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:28.996+0000 7f29dd579700 1 --2- 172.21.15.42:0/2846557641 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f29d80a9980 0x7f29d8116680 secure :-1 s=THROTTLE_DONE pgs=2919 cs=0 l=1 rx=0x7f29d811a930 tx=0x7f29d4010160).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:29.039 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:28.996+0000 7f29cdffb700 1 -- 172.21.15.42:0/2846557641 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f29d4015bb0 con 0x7f29d80a9980 2019-12-05T06:39:29.165 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:29.160+0000 7f29defdc700 1 -- 172.21.15.42:0/2846557641 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f29d811b840 con 0x7f29d80a9980 2019-12-05T06:39:29.166 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:29.160+0000 7f29dd579700 1 --2- 172.21.15.42:0/2846557641 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f29d80a9980 0x7f29d8116680 secure :-1 s=THROTTLE_DONE pgs=2919 cs=0 l=1 rx=0x7f29d811a930 tx=0x7f29d4010160).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:29.166 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:29.160+0000 7f29cdffb700 1 -- 172.21.15.42:0/2846557641 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 348 v348) v1 ==== 95+0+2113 (secure 0 0 0) 0x7f29d4033290 con 0x7f29d80a9980 2019-12-05T06:39:29.169 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 348 2019-12-05T06:39:29.170 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:39:29.170 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":348,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":8009,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/1777669201","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1777669201},{"type":"v1","addr":"172.21.15.40:6833","nonce":1777669201}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":348}],"filesystems":[{"mdsmap":{"epoch":347,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:38:00.190088+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11887},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11887":{"gid":11887,"name":"a","rank":0,"incarnation":344,"state":"up:active","state_seq":37,"addr":"172.21.15.42:6835/958796360","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":958796360},{"type":"v1","addr":"172.21.15.42:6835","nonce":958796360}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:39:29.172 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:29.164+0000 7f29defdc700 1 -- 172.21.15.42:0/2846557641 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f29b803acf0 msgr2=0x7f29b803d200 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:29.172 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:29.164+0000 7f29defdc700 1 --2- 172.21.15.42:0/2846557641 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f29b803acf0 0x7f29b803d200 crc :-1 s=READY pgs=1224 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:29.173 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:29.164+0000 7f29defdc700 1 -- 172.21.15.42:0/2846557641 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f29d80a9980 msgr2=0x7f29d8116680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:29.173 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:29.164+0000 7f29defdc700 1 --2- 172.21.15.42:0/2846557641 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f29d80a9980 0x7f29d8116680 secure :-1 s=READY pgs=2919 cs=0 l=1 rx=0x7f29d811a930 tx=0x7f29d4010160).stop 2019-12-05T06:39:29.173 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:29.164+0000 7f29defdc700 1 -- 172.21.15.42:0/2846557641 shutdown_connections 2019-12-05T06:39:29.173 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:29.164+0000 7f29defdc700 1 --2- 172.21.15.42:0/2846557641 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f29b803acf0 0x7f29b803d200 unknown :-1 s=CLOSED pgs=1224 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:29.173 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:29.164+0000 7f29defdc700 1 --2- 172.21.15.42:0/2846557641 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f29d8112300 0x7f29d8112730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:29.173 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:29.164+0000 7f29defdc700 1 --2- 172.21.15.42:0/2846557641 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f29d80a9980 0x7f29d8116680 unknown :-1 s=CLOSED pgs=2919 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:29.174 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:29.164+0000 7f29defdc700 1 --2- 172.21.15.42:0/2846557641 >> v2:172.21.15.40:3300/0 conn(0x7f29d8117df0 0x7f29d8111db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:29.174 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:29.164+0000 7f29defdc700 1 -- 172.21.15.42:0/2846557641 shutdown_connections 2019-12-05T06:39:29.175 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:29.164+0000 7f29defdc700 1 -- 172.21.15.42:0/2846557641 wait complete. 2019-12-05T06:39:29.175 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:29.164+0000 7f29defdc700 1 -- 172.21.15.42:0/2846557641 >> 172.21.15.42:0/2846557641 conn(0x7f29d805a200 msgr2=0x7f29d807e8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:39:29.181 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:39:31.192 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:39:31.306 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.292+0000 7fde13858700 1 Processor -- start 2019-12-05T06:39:31.306 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.296+0000 7fde13858700 1 -- start start 2019-12-05T06:39:31.307 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.296+0000 7fde13858700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fde0c0a7740 0x7fde0c08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:39:31.307 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.296+0000 7fde13858700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fde0c07e720 0x7fde0c07eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:39:31.307 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.296+0000 7fde13858700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fde0c09f2e0 0x7fde0c09f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:39:31.307 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.296+0000 7fde13858700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde0c0a0b00 con 0x7fde0c0a7740 2019-12-05T06:39:31.307 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.296+0000 7fde13858700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde0c08e5b0 con 0x7fde0c07e720 2019-12-05T06:39:31.307 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.296+0000 7fde13858700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde0c08e850 con 0x7fde0c09f2e0 2019-12-05T06:39:31.307 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.296+0000 7fde11df5700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fde0c0a7740 0x7fde0c08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:31.308 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.296+0000 7fde115f4700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fde0c07e720 0x7fde0c07eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:31.308 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.296+0000 7fde11df5700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fde0c0a7740 0x7fde0c08df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.42:47914/0 (socket says 172.21.15.42:47914) 2019-12-05T06:39:31.308 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.296+0000 7fde11df5700 1 -- 172.21.15.42:0/3747920023 learned_addr learned my addr 172.21.15.42:0/3747920023 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:39:31.308 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.296+0000 7fde125f6700 1 --2- 172.21.15.42:0/3747920023 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fde0c09f2e0 0x7fde0c09f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:31.308 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.296+0000 7fde115f4700 1 -- 172.21.15.42:0/3747920023 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fde0c09f2e0 msgr2=0x7fde0c09f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:39:31.308 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.296+0000 7fde115f4700 1 --2- 172.21.15.42:0/3747920023 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fde0c09f2e0 0x7fde0c09f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:39:31.309 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.296+0000 7fde115f4700 1 -- 172.21.15.42:0/3747920023 >> v2:172.21.15.40:3300/0 conn(0x7fde0c0a7740 msgr2=0x7fde0c08df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:39:31.309 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.296+0000 7fde115f4700 1 --2- 172.21.15.42:0/3747920023 >> v2:172.21.15.40:3300/0 conn(0x7fde0c0a7740 0x7fde0c08df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:39:31.309 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.296+0000 7fde115f4700 1 -- 172.21.15.42:0/3747920023 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde0c07f0a0 con 0x7fde0c07e720 2019-12-05T06:39:31.309 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.296+0000 7fde115f4700 1 --2- 172.21.15.42:0/3747920023 >> v2:172.21.15.42:3300/0 conn(0x7fde0c07e720 0x7fde0c07eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fde0c0a0f50 tx=0x7fde0000fcf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:31.309 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.296+0000 7fde115f4700 1 --2- 172.21.15.42:0/3747920023 >> v2:172.21.15.42:3300/0 conn(0x7fde0c07e720 0x7fde0c07eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fde0c0a0f50 tx=0x7fde0000fcf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:31.309 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.296+0000 7fde115f4700 1 --2- 172.21.15.42:0/3747920023 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fde0c07e720 0x7fde0c07eb50 secure :-1 s=READY pgs=2920 cs=0 l=1 rx=0x7fde0c0a0f50 tx=0x7fde0000fcf0).ready entity=mon.1 client_cookie=1db474e16df98dce server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:39:31.309 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.296+0000 7fde115f4700 1 --2- 172.21.15.42:0/3747920023 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fde0c07e720 0x7fde0c07eb50 secure :-1 s=THROTTLE_DONE pgs=2920 cs=0 l=1 rx=0x7fde0c0a0f50 tx=0x7fde0000fcf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:31.310 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.296+0000 7fde10df3700 1 -- 172.21.15.42:0/3747920023 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fde00008bb0 con 0x7fde0c07e720 2019-12-05T06:39:31.310 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.296+0000 7fde115f4700 1 --2- 172.21.15.42:0/3747920023 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fde0c07e720 0x7fde0c07eb50 secure :-1 s=THROTTLE_DONE pgs=2920 cs=0 l=1 rx=0x7fde0c0a0f50 tx=0x7fde0000fcf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:31.310 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.296+0000 7fde10df3700 1 -- 172.21.15.42:0/3747920023 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde000087d0 con 0x7fde0c07e720 2019-12-05T06:39:31.310 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.296+0000 7fde115f4700 1 --2- 172.21.15.42:0/3747920023 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fde0c07e720 0x7fde0c07eb50 secure :-1 s=THROTTLE_DONE pgs=2920 cs=0 l=1 rx=0x7fde0c0a0f50 tx=0x7fde0000fcf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:31.310 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.296+0000 7fde10df3700 1 -- 172.21.15.42:0/3747920023 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fde000118a0 con 0x7fde0c07e720 2019-12-05T06:39:31.310 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.296+0000 7fde13858700 1 -- 172.21.15.42:0/3747920023 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fde0c07e720 msgr2=0x7fde0c07eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:31.310 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.296+0000 7fde13858700 1 --2- 172.21.15.42:0/3747920023 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fde0c07e720 0x7fde0c07eb50 secure :-1 s=READY pgs=2920 cs=0 l=1 rx=0x7fde0c0a0f50 tx=0x7fde0000fcf0).stop 2019-12-05T06:39:31.311 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.300+0000 7fde13858700 1 -- 172.21.15.42:0/3747920023 shutdown_connections 2019-12-05T06:39:31.311 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.300+0000 7fde13858700 1 --2- 172.21.15.42:0/3747920023 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fde0c09f2e0 0x7fde0c09f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:39:31.311 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.300+0000 7fde13858700 1 --2- 172.21.15.42:0/3747920023 >> v2:172.21.15.40:3300/0 conn(0x7fde0c0a7740 0x7fde0c08df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:39:31.311 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.300+0000 7fde13858700 1 --2- 172.21.15.42:0/3747920023 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fde0c07e720 0x7fde0c07eb50 unknown :-1 s=CLOSED pgs=2920 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:31.311 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.300+0000 7fde13858700 1 -- 172.21.15.42:0/3747920023 shutdown_connections 2019-12-05T06:39:31.311 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.300+0000 7fde13858700 1 -- 172.21.15.42:0/3747920023 wait complete. 2019-12-05T06:39:31.311 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.300+0000 7fde13858700 1 -- 172.21.15.42:0/3747920023 >> 172.21.15.42:0/3747920023 conn(0x7fde0c0aabe0 msgr2=0x7fde0c0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:39:31.315 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.300+0000 7fde13858700 1 Processor -- start 2019-12-05T06:39:31.315 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.300+0000 7fde13858700 1 -- start start 2019-12-05T06:39:31.315 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.304+0000 7fde13858700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fde0c07d410 0x7fde0c07ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:39:31.315 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.304+0000 7fde13858700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fde0c0a6bf0 0x7fde0c120bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:39:31.316 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.304+0000 7fde13858700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde0c121120 con 0x7fde0c11a920 2019-12-05T06:39:31.316 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.304+0000 7fde13858700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde0c07d840 con 0x7fde0c0a6bf0 2019-12-05T06:39:31.316 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.304+0000 7fde13858700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde0c0a7020 con 0x7fde0c07d410 2019-12-05T06:39:31.316 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.304+0000 7fde11df5700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fde0c11a920 0x7fde0c07b020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:60142/0 (socket says 172.21.15.42:60142) 2019-12-05T06:39:31.316 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.304+0000 7fde11df5700 1 -- 172.21.15.42:0/490641615 learned_addr learned my addr 172.21.15.42:0/490641615 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:39:31.316 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.304+0000 7fde125f6700 1 --2- 172.21.15.42:0/490641615 >> v2:172.21.15.40:3300/0 conn(0x7fde0c0a6bf0 0x7fde0c120bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:31.317 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.304+0000 7fde115f4700 1 --2- 172.21.15.42:0/490641615 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fde0c07d410 0x7fde0c07ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:31.317 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.304+0000 7fde0a7fc700 1 -- 172.21.15.42:0/490641615 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1027003349 0 0) 0x7fde0c121120 con 0x7fde0c11a920 2019-12-05T06:39:31.317 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.304+0000 7fde0a7fc700 1 -- 172.21.15.42:0/490641615 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fddec0030b0 con 0x7fde0c11a920 2019-12-05T06:39:31.317 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.304+0000 7fde125f6700 1 -- 172.21.15.42:0/490641615 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fde0c07d410 msgr2=0x7fde0c07ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:31.317 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.304+0000 7fde125f6700 1 --2- 172.21.15.42:0/490641615 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fde0c07d410 0x7fde0c07ba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:31.317 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.304+0000 7fde125f6700 1 -- 172.21.15.42:0/490641615 >> v1:172.21.15.42:6789/0 conn(0x7fde0c11a920 legacy=0x7fde0c07b020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:31.318 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.304+0000 7fde125f6700 1 -- 172.21.15.42:0/490641615 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde0c139a00 con 0x7fde0c0a6bf0 2019-12-05T06:39:31.318 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.304+0000 7fde125f6700 1 --2- 172.21.15.42:0/490641615 >> v2:172.21.15.40:3300/0 conn(0x7fde0c0a6bf0 0x7fde0c120bd0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fde04002f30 tx=0x7fde04010980).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:31.318 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.304+0000 7fde125f6700 1 --2- 172.21.15.42:0/490641615 >> v2:172.21.15.40:3300/0 conn(0x7fde0c0a6bf0 0x7fde0c120bd0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fde04002f30 tx=0x7fde04010980).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:31.318 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.304+0000 7fde125f6700 1 --2- 172.21.15.42:0/490641615 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fde0c0a6bf0 0x7fde0c120bd0 secure :-1 s=READY pgs=2214 cs=0 l=1 rx=0x7fde04002f30 tx=0x7fde04010980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:39:31.318 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.304+0000 7fde125f6700 1 --2- 172.21.15.42:0/490641615 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fde0c0a6bf0 0x7fde0c120bd0 secure :-1 s=THROTTLE_DONE pgs=2214 cs=0 l=1 rx=0x7fde04002f30 tx=0x7fde04010980).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:31.318 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.304+0000 7fde0a7fc700 1 -- 172.21.15.42:0/490641615 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fde04002690 con 0x7fde0c0a6bf0 2019-12-05T06:39:31.319 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.304+0000 7fde125f6700 1 --2- 172.21.15.42:0/490641615 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fde0c0a6bf0 0x7fde0c120bd0 secure :-1 s=THROTTLE_DONE pgs=2214 cs=0 l=1 rx=0x7fde04002f30 tx=0x7fde04010980).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:31.319 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.304+0000 7fde125f6700 1 --2- 172.21.15.42:0/490641615 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fde0c0a6bf0 0x7fde0c120bd0 secure :-1 s=THROTTLE_DONE pgs=2214 cs=0 l=1 rx=0x7fde04002f30 tx=0x7fde04010980).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:31.319 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.304+0000 7fde0a7fc700 1 -- 172.21.15.42:0/490641615 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde040022b0 con 0x7fde0c0a6bf0 2019-12-05T06:39:31.319 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.304+0000 7fde0a7fc700 1 -- 172.21.15.42:0/490641615 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fde04002c70 con 0x7fde0c0a6bf0 2019-12-05T06:39:31.319 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.304+0000 7fde13858700 1 -- 172.21.15.42:0/490641615 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde0c1389f0 con 0x7fde0c0a6bf0 2019-12-05T06:39:31.319 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.304+0000 7fde13858700 1 -- 172.21.15.42:0/490641615 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde0c138f70 con 0x7fde0c0a6bf0 2019-12-05T06:39:31.320 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.308+0000 7fde125f6700 1 --2- 172.21.15.42:0/490641615 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fde0c0a6bf0 0x7fde0c120bd0 secure :-1 s=THROTTLE_DONE pgs=2214 cs=0 l=1 rx=0x7fde04002f30 tx=0x7fde04010980).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:31.320 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.308+0000 7fde0a7fc700 1 -- 172.21.15.42:0/490641615 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fde04007170 con 0x7fde0c0a6bf0 2019-12-05T06:39:31.320 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.308+0000 7fde125f6700 1 --2- 172.21.15.42:0/490641615 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fde0c0a6bf0 0x7fde0c120bd0 secure :-1 s=THROTTLE_DONE pgs=2214 cs=0 l=1 rx=0x7fde04002f30 tx=0x7fde04010980).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:31.320 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.308+0000 7fde0a7fc700 1 --2- 172.21.15.42:0/490641615 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fddec03b2c0 0x7fddec03d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:39:31.320 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.308+0000 7fde0a7fc700 1 -- 172.21.15.42:0/490641615 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 11535+0+0 (secure 0 0 0) 0x7fde0404aa70 con 0x7fde0c0a6bf0 2019-12-05T06:39:31.320 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.308+0000 7fde11df5700 1 --2- 172.21.15.42:0/490641615 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fddec03b2c0 0x7fddec03d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:31.321 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.308+0000 7fde11df5700 1 --2- 172.21.15.42:0/490641615 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fddec03b2c0 0x7fddec03d7d0 crc :-1 s=READY pgs=1225 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:39:31.321 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.308+0000 7fde13858700 1 -- 172.21.15.42:0/490641615 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde0c05d110 con 0x7fde0c0a6bf0 2019-12-05T06:39:31.335 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.320+0000 7fde125f6700 1 --2- 172.21.15.42:0/490641615 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fde0c0a6bf0 0x7fde0c120bd0 secure :-1 s=THROTTLE_DONE pgs=2214 cs=0 l=1 rx=0x7fde04002f30 tx=0x7fde04010980).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:31.335 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.320+0000 7fde0a7fc700 1 -- 172.21.15.42:0/490641615 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fde04031af0 con 0x7fde0c0a6bf0 2019-12-05T06:39:31.498 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.492+0000 7fde13858700 1 -- 172.21.15.42:0/490641615 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fde0c05b8c0 con 0x7fde0c0a6bf0 2019-12-05T06:39:31.558 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.492+0000 7fde125f6700 1 --2- 172.21.15.42:0/490641615 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fde0c0a6bf0 0x7fde0c120bd0 secure :-1 s=THROTTLE_DONE pgs=2214 cs=0 l=1 rx=0x7fde04002f30 tx=0x7fde04010980).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:31.558 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.492+0000 7fde0a7fc700 1 -- 172.21.15.42:0/490641615 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 349 v349) v1 ==== 95+0+1775 (secure 0 0 0) 0x7fde04031d50 con 0x7fde0c0a6bf0 2019-12-05T06:39:31.559 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 349 2019-12-05T06:39:31.559 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.496+0000 7fde13858700 1 -- 172.21.15.42:0/490641615 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fddec03b2c0 msgr2=0x7fddec03d7d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:31.559 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.496+0000 7fde13858700 1 --2- 172.21.15.42:0/490641615 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fddec03b2c0 0x7fddec03d7d0 crc :-1 s=READY pgs=1225 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:31.559 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.496+0000 7fde13858700 1 -- 172.21.15.42:0/490641615 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fde0c0a6bf0 msgr2=0x7fde0c120bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:31.559 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.496+0000 7fde13858700 1 --2- 172.21.15.42:0/490641615 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fde0c0a6bf0 0x7fde0c120bd0 secure :-1 s=READY pgs=2214 cs=0 l=1 rx=0x7fde04002f30 tx=0x7fde04010980).stop 2019-12-05T06:39:31.559 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.496+0000 7fde13858700 1 -- 172.21.15.42:0/490641615 shutdown_connections 2019-12-05T06:39:31.560 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.496+0000 7fde13858700 1 --2- 172.21.15.42:0/490641615 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fddec03b2c0 0x7fddec03d7d0 unknown :-1 s=CLOSED pgs=1225 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:31.560 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.496+0000 7fde13858700 1 --2- 172.21.15.42:0/490641615 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fde0c0a6bf0 0x7fde0c120bd0 unknown :-1 s=CLOSED pgs=2214 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:31.560 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.496+0000 7fde13858700 1 --2- 172.21.15.42:0/490641615 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fde0c07d410 0x7fde0c07ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:31.560 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.496+0000 7fde13858700 1 -- 172.21.15.42:0/490641615 shutdown_connections 2019-12-05T06:39:31.560 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.496+0000 7fde13858700 1 -- 172.21.15.42:0/490641615 wait complete. 2019-12-05T06:39:31.560 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.496+0000 7fde13858700 1 -- 172.21.15.42:0/490641615 >> 172.21.15.42:0/490641615 conn(0x7fde0c08fda0 msgr2=0x7fde0c09f040 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:39:31.560 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:39:31.561 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":349,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":349,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:39:30.177868+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":113,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8009},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8009":{"gid":8009,"name":"b","rank":0,"incarnation":349,"state":"up:replay","state_seq":2,"addr":"172.21.15.40:6833/1777669201","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1777669201},{"type":"v1","addr":"172.21.15.40:6833","nonce":1777669201}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:39:31.561 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a down, removed from mdsmap 2019-12-05T06:39:31.561 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2019-12-05T06:39:31.562 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:39:31.698 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.668+0000 7fb0af884700 1 Processor -- start 2019-12-05T06:39:31.698 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.668+0000 7fb0af884700 1 -- start start 2019-12-05T06:39:31.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.668+0000 7fb0af884700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb0a80a7730 0x7fb0a808df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:39:31.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.668+0000 7fb0af884700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0a80a0700 con 0x7fb0a807e710 2019-12-05T06:39:31.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.668+0000 7fb0af884700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0a80a9b90 con 0x7fb0a809fad0 2019-12-05T06:39:31.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.668+0000 7fb0af884700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0a80a0af0 con 0x7fb0a80a7730 2019-12-05T06:39:31.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.668+0000 7fb0ade21700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb0a80a7730 0x7fb0a808df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:31.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.668+0000 7fb0ae622700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fb0a809fad0 0x7fb0a814efd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:60154/0 (socket says 172.21.15.42:60154) 2019-12-05T06:39:31.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.668+0000 7fb0ae622700 1 -- 172.21.15.42:0/2881318005 learned_addr learned my addr 172.21.15.42:0/2881318005 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:39:31.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.668+0000 7fb0ade21700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb0a80a7730 0x7fb0a808df70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:60114/0 (socket says 172.21.15.42:60114) 2019-12-05T06:39:31.708 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.668+0000 7fb0ad620700 1 --1- 172.21.15.42:0/2881318005 >> v1:172.21.15.40:6789/0 conn(0x7fb0a807e710 0x7fb0a809f2d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:41358/0 (socket says 172.21.15.42:41358) 2019-12-05T06:39:31.708 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.668+0000 7fb0ade21700 1 -- 172.21.15.42:0/2881318005 >> v1:172.21.15.42:6789/0 conn(0x7fb0a809fad0 legacy=0x7fb0a814efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:31.708 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.668+0000 7fb0ade21700 1 -- 172.21.15.42:0/2881318005 >> v1:172.21.15.40:6789/0 conn(0x7fb0a807e710 legacy=0x7fb0a809f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:31.708 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.668+0000 7fb0ace1f700 1 -- 172.21.15.42:0/2881318005 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 879749371 0 0) 0x7fb0a80a0700 con 0x7fb0a807e710 2019-12-05T06:39:31.708 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.668+0000 7fb0ade21700 1 -- 172.21.15.42:0/2881318005 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0a807eb40 con 0x7fb0a80a7730 2019-12-05T06:39:31.709 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.668+0000 7fb0ade21700 1 --2- 172.21.15.42:0/2881318005 >> v2:172.21.15.42:3300/0 conn(0x7fb0a80a7730 0x7fb0a808df70 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fb09c008790 tx=0x7fb09c00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:31.709 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.668+0000 7fb0ade21700 1 --2- 172.21.15.42:0/2881318005 >> v2:172.21.15.42:3300/0 conn(0x7fb0a80a7730 0x7fb0a808df70 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fb09c008790 tx=0x7fb09c00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:31.709 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.668+0000 7fb0ade21700 1 --2- 172.21.15.42:0/2881318005 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb0a80a7730 0x7fb0a808df70 secure :-1 s=READY pgs=2923 cs=0 l=1 rx=0x7fb09c008790 tx=0x7fb09c00fbf0).ready entity=mon.1 client_cookie=99649329ce637107 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:39:31.709 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.668+0000 7fb0ade21700 1 --2- 172.21.15.42:0/2881318005 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb0a80a7730 0x7fb0a808df70 secure :-1 s=THROTTLE_DONE pgs=2923 cs=0 l=1 rx=0x7fb09c008790 tx=0x7fb09c00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:31.709 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.672+0000 7fb0ace1f700 1 -- 172.21.15.42:0/2881318005 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb09c011270 con 0x7fb0a80a7730 2019-12-05T06:39:31.709 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.672+0000 7fb0ade21700 1 --2- 172.21.15.42:0/2881318005 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb0a80a7730 0x7fb0a808df70 secure :-1 s=THROTTLE_DONE pgs=2923 cs=0 l=1 rx=0x7fb09c008790 tx=0x7fb09c00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:31.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.672+0000 7fb0ace1f700 1 -- 172.21.15.42:0/2881318005 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb09c008bc0 con 0x7fb0a80a7730 2019-12-05T06:39:31.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.672+0000 7fb0ade21700 1 --2- 172.21.15.42:0/2881318005 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb0a80a7730 0x7fb0a808df70 secure :-1 s=THROTTLE_DONE pgs=2923 cs=0 l=1 rx=0x7fb09c008790 tx=0x7fb09c00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:31.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.672+0000 7fb0af884700 1 -- 172.21.15.42:0/2881318005 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb0a80a7730 msgr2=0x7fb0a808df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:31.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.672+0000 7fb0ace1f700 1 -- 172.21.15.42:0/2881318005 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb09c011850 con 0x7fb0a80a7730 2019-12-05T06:39:31.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.672+0000 7fb0af884700 1 --2- 172.21.15.42:0/2881318005 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb0a80a7730 0x7fb0a808df70 secure :-1 s=READY pgs=2923 cs=0 l=1 rx=0x7fb09c008790 tx=0x7fb09c00fbf0).stop 2019-12-05T06:39:31.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.672+0000 7fb0af884700 1 -- 172.21.15.42:0/2881318005 shutdown_connections 2019-12-05T06:39:31.711 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.672+0000 7fb0af884700 1 --2- 172.21.15.42:0/2881318005 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb0a80a7730 0x7fb0a808df70 unknown :-1 s=CLOSED pgs=2923 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:31.711 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.672+0000 7fb0af884700 1 -- 172.21.15.42:0/2881318005 shutdown_connections 2019-12-05T06:39:31.711 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.672+0000 7fb0af884700 1 -- 172.21.15.42:0/2881318005 wait complete. 2019-12-05T06:39:31.711 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.672+0000 7fb0af884700 1 -- 172.21.15.42:0/2881318005 >> 172.21.15.42:0/2881318005 conn(0x7fb0a80aabd0 msgr2=0x7fb0a80a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:39:31.711 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.672+0000 7fb0af884700 1 Processor -- start 2019-12-05T06:39:31.712 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.672+0000 7fb0af884700 1 -- start start 2019-12-05T06:39:31.712 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.672+0000 7fb0af884700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb0a80a67d0 0x7fb0a8064ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:39:31.712 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.672+0000 7fb0af884700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb0a8144a40 0x7fb0a8063580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:39:31.712 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.672+0000 7fb0af884700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0a8122fe0 con 0x7fb0a8065540 2019-12-05T06:39:31.713 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.672+0000 7fb0af884700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0a8065a20 con 0x7fb0a8144a40 2019-12-05T06:39:31.713 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.672+0000 7fb0af884700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0a8063180 con 0x7fb0a80a67d0 2019-12-05T06:39:31.713 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.672+0000 7fb0ae622700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb0a8144a40 0x7fb0a8063580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:31.713 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.672+0000 7fb0ae622700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb0a8144a40 0x7fb0a8063580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:60120/0 (socket says 172.21.15.42:60120) 2019-12-05T06:39:31.713 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.672+0000 7fb0ae622700 1 -- 172.21.15.42:0/2619120533 learned_addr learned my addr 172.21.15.42:0/2619120533 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:39:31.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.672+0000 7fb0ade21700 1 --2- 172.21.15.42:0/2619120533 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb0a80a67d0 0x7fb0a8064ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:31.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.676+0000 7fb0ae622700 1 -- 172.21.15.42:0/2619120533 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb0a80a67d0 msgr2=0x7fb0a8064ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:31.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.676+0000 7fb0ae622700 1 --2- 172.21.15.42:0/2619120533 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb0a80a67d0 0x7fb0a8064ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:31.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.676+0000 7fb0ae622700 1 -- 172.21.15.42:0/2619120533 >> v1:172.21.15.40:6789/0 conn(0x7fb0a8065540 legacy=0x7fb0a81227e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:31.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.676+0000 7fb0ae622700 1 -- 172.21.15.42:0/2619120533 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0a8063ad0 con 0x7fb0a8144a40 2019-12-05T06:39:31.715 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.676+0000 7fb0a67fc700 1 -- 172.21.15.42:0/2619120533 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 793197532 0 0) 0x7fb0a8122fe0 con 0x7fb0a8065540 2019-12-05T06:39:31.715 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.676+0000 7fb0ade21700 1 --2- 172.21.15.42:0/2619120533 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb0a80a67d0 0x7fb0a8064ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:39:31.715 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.676+0000 7fb0ae622700 1 --2- 172.21.15.42:0/2619120533 >> v2:172.21.15.42:3300/0 conn(0x7fb0a8144a40 0x7fb0a8063580 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fb0a8128cf0 tx=0x7fb0a0010830).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:31.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.676+0000 7fb0ae622700 1 --2- 172.21.15.42:0/2619120533 >> v2:172.21.15.42:3300/0 conn(0x7fb0a8144a40 0x7fb0a8063580 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fb0a8128cf0 tx=0x7fb0a0010830).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:31.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.676+0000 7fb0ae622700 1 --2- 172.21.15.42:0/2619120533 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb0a8144a40 0x7fb0a8063580 secure :-1 s=READY pgs=2924 cs=0 l=1 rx=0x7fb0a8128cf0 tx=0x7fb0a0010830).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:39:31.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.676+0000 7fb0ae622700 1 --2- 172.21.15.42:0/2619120533 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb0a8144a40 0x7fb0a8063580 secure :-1 s=THROTTLE_DONE pgs=2924 cs=0 l=1 rx=0x7fb0a8128cf0 tx=0x7fb0a0010830).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:31.718 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.676+0000 7fb0a67fc700 1 -- 172.21.15.42:0/2619120533 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb0a0011520 con 0x7fb0a8144a40 2019-12-05T06:39:31.718 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.676+0000 7fb0ae622700 1 --2- 172.21.15.42:0/2619120533 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb0a8144a40 0x7fb0a8063580 secure :-1 s=THROTTLE_DONE pgs=2924 cs=0 l=1 rx=0x7fb0a8128cf0 tx=0x7fb0a0010830).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:31.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.676+0000 7fb0ae622700 1 --2- 172.21.15.42:0/2619120533 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb0a8144a40 0x7fb0a8063580 secure :-1 s=THROTTLE_DONE pgs=2924 cs=0 l=1 rx=0x7fb0a8128cf0 tx=0x7fb0a0010830).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:31.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.676+0000 7fb0a67fc700 1 -- 172.21.15.42:0/2619120533 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0a0011140 con 0x7fb0a8144a40 2019-12-05T06:39:31.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.676+0000 7fb0af884700 1 -- 172.21.15.42:0/2619120533 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0a8063da0 con 0x7fb0a8144a40 2019-12-05T06:39:31.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.676+0000 7fb0a67fc700 1 -- 172.21.15.42:0/2619120533 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb0a0011b00 con 0x7fb0a8144a40 2019-12-05T06:39:31.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.676+0000 7fb0af884700 1 -- 172.21.15.42:0/2619120533 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0a8130110 con 0x7fb0a8144a40 2019-12-05T06:39:31.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.676+0000 7fb0ae622700 1 --2- 172.21.15.42:0/2619120533 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb0a8144a40 0x7fb0a8063580 secure :-1 s=THROTTLE_DONE pgs=2924 cs=0 l=1 rx=0x7fb0a8128cf0 tx=0x7fb0a0010830).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:31.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.676+0000 7fb0a67fc700 1 -- 172.21.15.42:0/2619120533 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fb0a000b330 con 0x7fb0a8144a40 2019-12-05T06:39:31.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.676+0000 7fb0ae622700 1 --2- 172.21.15.42:0/2619120533 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb0a8144a40 0x7fb0a8063580 secure :-1 s=THROTTLE_DONE pgs=2924 cs=0 l=1 rx=0x7fb0a8128cf0 tx=0x7fb0a0010830).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:31.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.676+0000 7fb0a67fc700 1 --2- 172.21.15.42:0/2619120533 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb08803acf0 0x7fb08803d200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:39:31.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.676+0000 7fb0a67fc700 1 -- 172.21.15.42:0/2619120533 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 11535+0+0 (secure 0 0 0) 0x7fb0a004b0a0 con 0x7fb0a8144a40 2019-12-05T06:39:31.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.676+0000 7fb0ade21700 1 --2- 172.21.15.42:0/2619120533 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb08803acf0 0x7fb08803d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:31.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.680+0000 7fb0ade21700 1 --2- 172.21.15.42:0/2619120533 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb08803acf0 0x7fb08803d200 crc :-1 s=READY pgs=1226 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:39:31.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.684+0000 7fb0af884700 1 -- 172.21.15.42:0/2619120533 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0a805d100 con 0x7fb0a8144a40 2019-12-05T06:39:31.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.696+0000 7fb0ae622700 1 --2- 172.21.15.42:0/2619120533 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb0a8144a40 0x7fb0a8063580 secure :-1 s=THROTTLE_DONE pgs=2924 cs=0 l=1 rx=0x7fb0a8128cf0 tx=0x7fb0a0010830).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:31.724 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.696+0000 7fb0a67fc700 1 -- 172.21.15.42:0/2619120533 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fb0a0032120 con 0x7fb0a8144a40 2019-12-05T06:39:31.871 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.864+0000 7fb0af884700 1 -- 172.21.15.42:0/2619120533 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb0a805b8b0 con 0x7fb0a8144a40 2019-12-05T06:39:31.873 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.868+0000 7fb0ae622700 1 --2- 172.21.15.42:0/2619120533 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb0a8144a40 0x7fb0a8063580 secure :-1 s=THROTTLE_DONE pgs=2924 cs=0 l=1 rx=0x7fb0a8128cf0 tx=0x7fb0a0010830).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:31.873 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.868+0000 7fb0a67fc700 1 -- 172.21.15.42:0/2619120533 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 349 v349) v1 ==== 95+0+1775 (secure 0 0 0) 0x7fb0a0032380 con 0x7fb0a8144a40 2019-12-05T06:39:31.873 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 349 2019-12-05T06:39:31.874 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:39:31.874 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":349,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":349,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:39:30.177868+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":113,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8009},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8009":{"gid":8009,"name":"b","rank":0,"incarnation":349,"state":"up:replay","state_seq":2,"addr":"172.21.15.40:6833/1777669201","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1777669201},{"type":"v1","addr":"172.21.15.40:6833","nonce":1777669201}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:39:31.876 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.868+0000 7fb0af884700 1 -- 172.21.15.42:0/2619120533 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb08803acf0 msgr2=0x7fb08803d200 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:31.876 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.868+0000 7fb0af884700 1 --2- 172.21.15.42:0/2619120533 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb08803acf0 0x7fb08803d200 crc :-1 s=READY pgs=1226 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:31.877 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.868+0000 7fb0af884700 1 -- 172.21.15.42:0/2619120533 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb0a8144a40 msgr2=0x7fb0a8063580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:31.877 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.868+0000 7fb0af884700 1 --2- 172.21.15.42:0/2619120533 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb0a8144a40 0x7fb0a8063580 secure :-1 s=READY pgs=2924 cs=0 l=1 rx=0x7fb0a8128cf0 tx=0x7fb0a0010830).stop 2019-12-05T06:39:31.878 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.868+0000 7fb0af884700 1 -- 172.21.15.42:0/2619120533 shutdown_connections 2019-12-05T06:39:31.878 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.868+0000 7fb0af884700 1 --2- 172.21.15.42:0/2619120533 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb08803acf0 0x7fb08803d200 unknown :-1 s=CLOSED pgs=1226 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:31.880 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.868+0000 7fb0af884700 1 --2- 172.21.15.42:0/2619120533 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb0a8144a40 0x7fb0a8063580 unknown :-1 s=CLOSED pgs=2924 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:31.880 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.868+0000 7fb0af884700 1 --2- 172.21.15.42:0/2619120533 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb0a80a67d0 0x7fb0a8064ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:31.880 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.868+0000 7fb0af884700 1 -- 172.21.15.42:0/2619120533 shutdown_connections 2019-12-05T06:39:31.881 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.868+0000 7fb0af884700 1 -- 172.21.15.42:0/2619120533 wait complete. 2019-12-05T06:39:31.881 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:31.868+0000 7fb0af884700 1 -- 172.21.15.42:0/2619120533 >> 172.21.15.42:0/2619120533 conn(0x7fb0a80a6d20 msgr2=0x7fb0a80596f0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:39:31.894 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 349, "feature_flags": { "enable_multiple": false, "ever_enabled_multiple": false }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2019-12-05T05:03:35.604738+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 349, "ever_allowed_features": 2, "explicitly_allowed_features": 2, "failed": [], "flags": 18, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_8009": { "addr": "172.21.15.40:6833/1777669201", "addrs": { "addrvec": [ { "addr": "172.21.15.40:6832", "nonce": 1777669201, "type": "v2" }, { "addr": "172.21.15.40:6833", "nonce": 1777669201, "type": "v1" } ] }, "export_targets": [], "features": 4540138292836696063, "flags": 0, "gid": 8009, "incarnation": 349, "name": "b", "rank": 0, "state": "up:replay", "state_seq": 2 } }, "last_failure": 0, "last_failure_osd_epoch": 113, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 2, "min_compat_client": "0 (unknown)", "modified": "2019-12-05T06:39:30.177868+0000", "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 8009 } } } ], "standbys": [] } 2019-12-05T06:39:33.894 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:39:34.005 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:33.996+0000 7f34bf123700 1 Processor -- start 2019-12-05T06:39:34.006 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:33.996+0000 7f34bf123700 1 -- start start 2019-12-05T06:39:34.006 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:33.996+0000 7f34bf123700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f34b80a7740 0x7f34b808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:39:34.006 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:33.996+0000 7f34bf123700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f34b807e720 0x7f34b807eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:39:34.006 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:33.996+0000 7f34bf123700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34b80a0710 con 0x7f34b809f2e0 2019-12-05T06:39:34.007 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:33.996+0000 7f34bf123700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34b80a0b00 con 0x7f34b80a7740 2019-12-05T06:39:34.007 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:33.996+0000 7f34bf123700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34b808e5b0 con 0x7f34b807e720 2019-12-05T06:39:34.007 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:33.996+0000 7f34bcebf700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f34b807e720 0x7f34b807eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:34.007 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:33.996+0000 7f34bd6c0700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f34b80a7740 0x7f34b808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:34.007 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:33.996+0000 7f34bcebf700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f34b807e720 0x7f34b807eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:60128/0 (socket says 172.21.15.42:60128) 2019-12-05T06:39:34.008 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:33.996+0000 7f34bd6c0700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f34b80a7740 0x7f34b808df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.42:47942/0 (socket says 172.21.15.42:47942) 2019-12-05T06:39:34.008 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:33.996+0000 7f34bcebf700 1 -- 172.21.15.42:0/1667142733 learned_addr learned my addr 172.21.15.42:0/1667142733 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:39:34.008 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:33.996+0000 7f34bcebf700 1 -- 172.21.15.42:0/1667142733 >> v2:172.21.15.40:3300/0 conn(0x7f34b80a7740 msgr2=0x7f34b808df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:39:34.008 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:33.996+0000 7f34bcebf700 1 --2- 172.21.15.42:0/1667142733 >> v2:172.21.15.40:3300/0 conn(0x7f34b80a7740 0x7f34b808df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:39:34.009 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:33.996+0000 7f34bcebf700 1 -- 172.21.15.42:0/1667142733 >> v1:172.21.15.40:6789/0 conn(0x7f34b809f2e0 legacy=0x7f34b809f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:34.010 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:33.996+0000 7f34bcebf700 1 -- 172.21.15.42:0/1667142733 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34b807f0a0 con 0x7f34b807e720 2019-12-05T06:39:34.010 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:33.996+0000 7f34bcebf700 1 --2- 172.21.15.42:0/1667142733 >> v2:172.21.15.42:3300/0 conn(0x7f34b807e720 0x7f34b807eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f34a8002f30 tx=0x7f34a800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:34.010 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:33.996+0000 7f34bcebf700 1 --2- 172.21.15.42:0/1667142733 >> v2:172.21.15.42:3300/0 conn(0x7f34b807e720 0x7f34b807eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f34a8002f30 tx=0x7f34a800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:34.010 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:33.996+0000 7f34bcebf700 1 --2- 172.21.15.42:0/1667142733 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f34b807e720 0x7f34b807eb50 secure :-1 s=READY pgs=2925 cs=0 l=1 rx=0x7f34a8002f30 tx=0x7f34a800fd30).ready entity=mon.1 client_cookie=3f6f9b5d72c485e5 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:39:34.010 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:33.996+0000 7f34bcebf700 1 --2- 172.21.15.42:0/1667142733 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f34b807e720 0x7f34b807eb50 secure :-1 s=THROTTLE_DONE pgs=2925 cs=0 l=1 rx=0x7f34a8002f30 tx=0x7f34a800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:34.011 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:33.996+0000 7f34bcebf700 1 --2- 172.21.15.42:0/1667142733 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f34b807e720 0x7f34b807eb50 secure :-1 s=THROTTLE_DONE pgs=2925 cs=0 l=1 rx=0x7f34a8002f30 tx=0x7f34a800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:34.011 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:33.996+0000 7f34affff700 1 -- 172.21.15.42:0/1667142733 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f34a8011370 con 0x7f34b807e720 2019-12-05T06:39:34.011 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:33.996+0000 7f34bcebf700 1 --2- 172.21.15.42:0/1667142733 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f34b807e720 0x7f34b807eb50 secure :-1 s=THROTTLE_DONE pgs=2925 cs=0 l=1 rx=0x7f34a8002f30 tx=0x7f34a800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:34.011 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:33.996+0000 7f34affff700 1 -- 172.21.15.42:0/1667142733 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34a8008bb0 con 0x7f34b807e720 2019-12-05T06:39:34.011 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:33.996+0000 7f34affff700 1 -- 172.21.15.42:0/1667142733 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f34a8011950 con 0x7f34b807e720 2019-12-05T06:39:34.011 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:33.996+0000 7f34bf123700 1 -- 172.21.15.42:0/1667142733 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f34b807e720 msgr2=0x7f34b807eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:34.011 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:33.996+0000 7f34bf123700 1 --2- 172.21.15.42:0/1667142733 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f34b807e720 0x7f34b807eb50 secure :-1 s=READY pgs=2925 cs=0 l=1 rx=0x7f34a8002f30 tx=0x7f34a800fd30).stop 2019-12-05T06:39:34.012 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:33.996+0000 7f34bf123700 1 -- 172.21.15.42:0/1667142733 shutdown_connections 2019-12-05T06:39:34.012 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:33.996+0000 7f34bf123700 1 --2- 172.21.15.42:0/1667142733 >> v2:172.21.15.40:3300/0 conn(0x7f34b80a7740 0x7f34b808df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:39:34.012 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:33.996+0000 7f34bf123700 1 --2- 172.21.15.42:0/1667142733 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f34b807e720 0x7f34b807eb50 unknown :-1 s=CLOSED pgs=2925 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:34.012 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:33.996+0000 7f34bf123700 1 -- 172.21.15.42:0/1667142733 shutdown_connections 2019-12-05T06:39:34.013 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:33.996+0000 7f34bf123700 1 -- 172.21.15.42:0/1667142733 wait complete. 2019-12-05T06:39:34.013 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:33.996+0000 7f34bf123700 1 -- 172.21.15.42:0/1667142733 >> 172.21.15.42:0/1667142733 conn(0x7f34b80aabe0 msgr2=0x7f34b80a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:39:34.013 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:34.000+0000 7f34bf123700 1 Processor -- start 2019-12-05T06:39:34.014 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:34.000+0000 7f34bf123700 1 -- start start 2019-12-05T06:39:34.019 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:34.000+0000 7f34bf123700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f34b8119ca0 0x7f34b80a9770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:39:34.019 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:34.000+0000 7f34bf123700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f34b8116bd0 0x7f34b8117000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:39:34.019 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:34.000+0000 7f34bf123700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34b811adf0 con 0x7f34b8111db0 2019-12-05T06:39:34.019 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:34.000+0000 7f34bf123700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34b811b0c0 con 0x7f34b8116bd0 2019-12-05T06:39:34.019 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:34.000+0000 7f34bf123700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34b811b360 con 0x7f34b8119ca0 2019-12-05T06:39:34.020 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:34.000+0000 7f34bcebf700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f34b8116bd0 0x7f34b8117000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:34.020 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:34.000+0000 7f34bd6c0700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f34b8119ca0 0x7f34b80a9770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:34.020 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:34.000+0000 7f34bdec1700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f34b8111db0 0x7f34b81121e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:41380/0 (socket says 172.21.15.42:41380) 2019-12-05T06:39:34.020 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:34.000+0000 7f34bdec1700 1 -- 172.21.15.42:0/719619071 learned_addr learned my addr 172.21.15.42:0/719619071 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:39:34.020 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:34.000+0000 7f34adffb700 1 -- 172.21.15.42:0/719619071 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2285787023 0 0) 0x7f34b811adf0 con 0x7f34b8111db0 2019-12-05T06:39:34.020 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:34.000+0000 7f34adffb700 1 -- 172.21.15.42:0/719619071 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34980030b0 con 0x7f34b8111db0 2019-12-05T06:39:34.021 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:34.000+0000 7f34bd6c0700 1 -- 172.21.15.42:0/719619071 >> v2:172.21.15.42:3300/0 conn(0x7f34b8116bd0 msgr2=0x7f34b8117000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:34.021 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:34.000+0000 7f34bd6c0700 1 --2- 172.21.15.42:0/719619071 >> v2:172.21.15.42:3300/0 conn(0x7f34b8116bd0 0x7f34b8117000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:34.021 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:34.000+0000 7f34bd6c0700 1 -- 172.21.15.42:0/719619071 >> v1:172.21.15.40:6789/0 conn(0x7f34b8111db0 legacy=0x7f34b81121e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:34.021 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:34.000+0000 7f34bd6c0700 1 -- 172.21.15.42:0/719619071 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34b811a0d0 con 0x7f34b8119ca0 2019-12-05T06:39:34.021 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:34.000+0000 7f34bcebf700 1 --2- 172.21.15.42:0/719619071 >> v2:172.21.15.42:3300/0 conn(0x7f34b8116bd0 0x7f34b8117000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:39:34.022 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:34.000+0000 7f34bd6c0700 1 --2- 172.21.15.42:0/719619071 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f34b8119ca0 0x7f34b80a9770 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f34b4002f30 tx=0x7f34b40109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:34.022 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:34.000+0000 7f34bd6c0700 1 --2- 172.21.15.42:0/719619071 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f34b8119ca0 0x7f34b80a9770 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f34b4002f30 tx=0x7f34b40109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:34.022 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:34.000+0000 7f34bd6c0700 1 --2- 172.21.15.42:0/719619071 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f34b8119ca0 0x7f34b80a9770 secure :-1 s=READY pgs=650 cs=0 l=1 rx=0x7f34b4002f30 tx=0x7f34b40109e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:39:34.022 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:34.004+0000 7f34bd6c0700 1 --2- 172.21.15.42:0/719619071 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f34b8119ca0 0x7f34b80a9770 secure :-1 s=THROTTLE_DONE pgs=650 cs=0 l=1 rx=0x7f34b4002f30 tx=0x7f34b40109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:34.022 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:34.004+0000 7f34adffb700 1 -- 172.21.15.42:0/719619071 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f34b4005920 con 0x7f34b8119ca0 2019-12-05T06:39:34.022 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:34.004+0000 7f34bd6c0700 1 --2- 172.21.15.42:0/719619071 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f34b8119ca0 0x7f34b80a9770 secure :-1 s=THROTTLE_DONE pgs=650 cs=0 l=1 rx=0x7f34b4002f30 tx=0x7f34b40109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:34.022 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:34.004+0000 7f34bf123700 1 -- 172.21.15.42:0/719619071 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34b80a9cc0 con 0x7f34b8119ca0 2019-12-05T06:39:34.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:34.004+0000 7f34adffb700 1 -- 172.21.15.42:0/719619071 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34b4005540 con 0x7f34b8119ca0 2019-12-05T06:39:34.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:34.004+0000 7f34bd6c0700 1 --2- 172.21.15.42:0/719619071 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f34b8119ca0 0x7f34b80a9770 secure :-1 s=THROTTLE_DONE pgs=650 cs=0 l=1 rx=0x7f34b4002f30 tx=0x7f34b40109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:34.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:34.004+0000 7f34adffb700 1 -- 172.21.15.42:0/719619071 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f34b4012490 con 0x7f34b8119ca0 2019-12-05T06:39:34.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:34.004+0000 7f34bf123700 1 -- 172.21.15.42:0/719619071 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34b8117550 con 0x7f34b8119ca0 2019-12-05T06:39:34.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:34.004+0000 7f34bd6c0700 1 --2- 172.21.15.42:0/719619071 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f34b8119ca0 0x7f34b80a9770 secure :-1 s=THROTTLE_DONE pgs=650 cs=0 l=1 rx=0x7f34b4002f30 tx=0x7f34b40109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:34.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:34.004+0000 7f34adffb700 1 -- 172.21.15.42:0/719619071 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f34b4007180 con 0x7f34b8119ca0 2019-12-05T06:39:34.024 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:34.004+0000 7f34bd6c0700 1 --2- 172.21.15.42:0/719619071 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f34b8119ca0 0x7f34b80a9770 secure :-1 s=THROTTLE_DONE pgs=650 cs=0 l=1 rx=0x7f34b4002f30 tx=0x7f34b40109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:34.024 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:34.004+0000 7f34adffb700 1 --2- 172.21.15.42:0/719619071 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f349803b1c0 0x7f349803d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:39:34.024 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:34.004+0000 7f34adffb700 1 -- 172.21.15.42:0/719619071 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 11535+0+0 (secure 0 0 0) 0x7f34b404ad70 con 0x7f34b8119ca0 2019-12-05T06:39:34.025 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:34.004+0000 7f34bcebf700 1 --2- 172.21.15.42:0/719619071 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f349803b1c0 0x7f349803d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:34.025 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:34.004+0000 7f34bcebf700 1 --2- 172.21.15.42:0/719619071 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f349803b1c0 0x7f349803d6d0 crc :-1 s=READY pgs=1227 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:39:34.025 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:34.012+0000 7f34bf123700 1 -- 172.21.15.42:0/719619071 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34b805d110 con 0x7f34b8119ca0 2019-12-05T06:39:34.026 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:34.020+0000 7f34bd6c0700 1 --2- 172.21.15.42:0/719619071 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f34b8119ca0 0x7f34b80a9770 secure :-1 s=THROTTLE_DONE pgs=650 cs=0 l=1 rx=0x7f34b4002f30 tx=0x7f34b40109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:34.026 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:34.020+0000 7f34adffb700 1 -- 172.21.15.42:0/719619071 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f34b4031dd0 con 0x7f34b8119ca0 2019-12-05T06:39:34.198 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:34.192+0000 7f34bf123700 1 -- 172.21.15.42:0/719619071 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f34b805b8c0 con 0x7f34b8119ca0 2019-12-05T06:39:34.199 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:34.192+0000 7f34bd6c0700 1 --2- 172.21.15.42:0/719619071 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f34b8119ca0 0x7f34b80a9770 secure :-1 s=THROTTLE_DONE pgs=650 cs=0 l=1 rx=0x7f34b4002f30 tx=0x7f34b40109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:34.199 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:34.192+0000 7f34adffb700 1 -- 172.21.15.42:0/719619071 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 349 v349) v1 ==== 95+0+1775 (secure 0 0 0) 0x7f34b4032030 con 0x7f34b8119ca0 2019-12-05T06:39:34.201 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 349 2019-12-05T06:39:34.202 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:39:34.202 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":349,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":349,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:39:30.177868+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":113,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8009},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8009":{"gid":8009,"name":"b","rank":0,"incarnation":349,"state":"up:replay","state_seq":2,"addr":"172.21.15.40:6833/1777669201","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1777669201},{"type":"v1","addr":"172.21.15.40:6833","nonce":1777669201}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:39:34.205 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:34.196+0000 7f34bf123700 1 -- 172.21.15.42:0/719619071 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f349803b1c0 msgr2=0x7f349803d6d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:34.205 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:34.196+0000 7f34bf123700 1 --2- 172.21.15.42:0/719619071 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f349803b1c0 0x7f349803d6d0 crc :-1 s=READY pgs=1227 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:34.205 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:34.196+0000 7f34bf123700 1 -- 172.21.15.42:0/719619071 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f34b8119ca0 msgr2=0x7f34b80a9770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:34.206 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:34.196+0000 7f34bf123700 1 --2- 172.21.15.42:0/719619071 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f34b8119ca0 0x7f34b80a9770 secure :-1 s=READY pgs=650 cs=0 l=1 rx=0x7f34b4002f30 tx=0x7f34b40109e0).stop 2019-12-05T06:39:34.206 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:34.196+0000 7f34bf123700 1 -- 172.21.15.42:0/719619071 shutdown_connections 2019-12-05T06:39:34.206 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:34.196+0000 7f34bf123700 1 --2- 172.21.15.42:0/719619071 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f349803b1c0 0x7f349803d6d0 unknown :-1 s=CLOSED pgs=1227 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:34.206 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:34.196+0000 7f34bf123700 1 --2- 172.21.15.42:0/719619071 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f34b8119ca0 0x7f34b80a9770 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:34.207 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:34.196+0000 7f34bf123700 1 --2- 172.21.15.42:0/719619071 >> v2:172.21.15.42:3300/0 conn(0x7f34b8116bd0 0x7f34b8117000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:34.207 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:34.196+0000 7f34bf123700 1 -- 172.21.15.42:0/719619071 shutdown_connections 2019-12-05T06:39:34.207 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:34.196+0000 7f34bf123700 1 -- 172.21.15.42:0/719619071 wait complete. 2019-12-05T06:39:34.207 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:34.196+0000 7f34bf123700 1 -- 172.21.15.42:0/719619071 >> 172.21.15.42:0/719619071 conn(0x7f34b80aabe0 msgr2=0x7f34b811be20 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:39:34.215 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2019-12-05T06:39:36.216 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:39:36.347 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:36.324+0000 7f183ac0a700 1 Processor -- start 2019-12-05T06:39:36.347 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:36.324+0000 7f183ac0a700 1 -- start start 2019-12-05T06:39:36.347 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:36.328+0000 7f183ac0a700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f183408df70 0x7f183409f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:39:36.347 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:36.328+0000 7f183ac0a700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f183409f820 0x7f183414efd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:39:36.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:36.328+0000 7f183ac0a700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f183409fc50 con 0x7f18340a7730 2019-12-05T06:39:36.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:36.328+0000 7f183ac0a700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18340a0af0 con 0x7f183408df70 2019-12-05T06:39:36.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:36.328+0000 7f183ac0a700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f183408e5a0 con 0x7f183409f820 2019-12-05T06:39:36.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:36.328+0000 7f18389a6700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f183408df70 0x7f183409f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:36.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:36.328+0000 7f18391a7700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f18340a7730 0x7f183407e710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:60186/0 (socket says 172.21.15.42:60186) 2019-12-05T06:39:36.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:36.328+0000 7f18391a7700 1 -- 172.21.15.42:0/3878155896 learned_addr learned my addr 172.21.15.42:0/3878155896 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:39:36.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:36.328+0000 7f18399a8700 1 --2- 172.21.15.42:0/3878155896 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f183409f820 0x7f183414efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:36.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:36.328+0000 7f18389a6700 1 -- 172.21.15.42:0/3878155896 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f183409f820 msgr2=0x7f183414efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:39:36.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:36.328+0000 7f18389a6700 1 --2- 172.21.15.42:0/3878155896 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f183409f820 0x7f183414efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:39:36.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:36.328+0000 7f18389a6700 1 -- 172.21.15.42:0/3878155896 >> v1:172.21.15.42:6789/0 conn(0x7f18340a7730 legacy=0x7f183407e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:36.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:36.328+0000 7f18389a6700 1 -- 172.21.15.42:0/3878155896 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18340a0700 con 0x7f183408df70 2019-12-05T06:39:36.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:36.328+0000 7f182bfff700 1 -- 172.21.15.42:0/3878155896 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2786793370 0 0) 0x7f183409fc50 con 0x7f18340a7730 2019-12-05T06:39:36.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:36.328+0000 7f18389a6700 1 --2- 172.21.15.42:0/3878155896 >> v2:172.21.15.42:3300/0 conn(0x7f183408df70 0x7f183409f2d0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f1824008790 tx=0x7f182400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:36.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:36.328+0000 7f18389a6700 1 --2- 172.21.15.42:0/3878155896 >> v2:172.21.15.42:3300/0 conn(0x7f183408df70 0x7f183409f2d0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f1824008790 tx=0x7f182400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:36.358 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:36.328+0000 7f18389a6700 1 --2- 172.21.15.42:0/3878155896 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f183408df70 0x7f183409f2d0 secure :-1 s=READY pgs=2927 cs=0 l=1 rx=0x7f1824008790 tx=0x7f182400fc70).ready entity=mon.1 client_cookie=a1b37f0097e2aab3 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:39:36.358 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:36.328+0000 7f18389a6700 1 --2- 172.21.15.42:0/3878155896 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f183408df70 0x7f183409f2d0 secure :-1 s=THROTTLE_DONE pgs=2927 cs=0 l=1 rx=0x7f1824008790 tx=0x7f182400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:36.358 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:36.328+0000 7f182bfff700 1 -- 172.21.15.42:0/3878155896 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1824008bb0 con 0x7f183408df70 2019-12-05T06:39:36.358 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:36.328+0000 7f18389a6700 1 --2- 172.21.15.42:0/3878155896 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f183408df70 0x7f183409f2d0 secure :-1 s=THROTTLE_DONE pgs=2927 cs=0 l=1 rx=0x7f1824008790 tx=0x7f182400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:36.358 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:36.328+0000 7f18389a6700 1 --2- 172.21.15.42:0/3878155896 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f183408df70 0x7f183409f2d0 secure :-1 s=THROTTLE_DONE pgs=2927 cs=0 l=1 rx=0x7f1824008790 tx=0x7f182400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:36.359 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:36.328+0000 7f182bfff700 1 -- 172.21.15.42:0/3878155896 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18240087d0 con 0x7f183408df70 2019-12-05T06:39:36.359 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:36.328+0000 7f182bfff700 1 -- 172.21.15.42:0/3878155896 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1824011860 con 0x7f183408df70 2019-12-05T06:39:36.359 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:36.328+0000 7f183ac0a700 1 -- 172.21.15.42:0/3878155896 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f183408df70 msgr2=0x7f183409f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:36.360 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:36.328+0000 7f183ac0a700 1 --2- 172.21.15.42:0/3878155896 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f183408df70 0x7f183409f2d0 secure :-1 s=READY pgs=2927 cs=0 l=1 rx=0x7f1824008790 tx=0x7f182400fc70).stop 2019-12-05T06:39:36.360 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:36.328+0000 7f183ac0a700 1 -- 172.21.15.42:0/3878155896 shutdown_connections 2019-12-05T06:39:36.360 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:36.328+0000 7f183ac0a700 1 --2- 172.21.15.42:0/3878155896 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f183409f820 0x7f183414efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:39:36.360 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:36.328+0000 7f183ac0a700 1 --2- 172.21.15.42:0/3878155896 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f183408df70 0x7f183409f2d0 unknown :-1 s=CLOSED pgs=2927 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:36.360 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:36.328+0000 7f183ac0a700 1 -- 172.21.15.42:0/3878155896 shutdown_connections 2019-12-05T06:39:36.361 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:36.328+0000 7f183ac0a700 1 -- 172.21.15.42:0/3878155896 wait complete. 2019-12-05T06:39:36.361 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:36.328+0000 7f183ac0a700 1 -- 172.21.15.42:0/3878155896 >> 172.21.15.42:0/3878155896 conn(0x7f18340aabd0 msgr2=0x7f18340a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:39:36.361 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:36.332+0000 7f183ac0a700 1 Processor -- start 2019-12-05T06:39:36.361 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:36.332+0000 7f183ac0a700 1 -- start start 2019-12-05T06:39:36.361 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:36.332+0000 7f183ac0a700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f183411b890 0x7f183407d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:39:36.362 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:36.332+0000 7f183ac0a700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f18341300a0 0x7f18340752d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:39:36.362 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:36.332+0000 7f183ac0a700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f183407d9c0 con 0x7f1834130d20 2019-12-05T06:39:36.362 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:36.332+0000 7f183ac0a700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1834130a80 con 0x7f18341300a0 2019-12-05T06:39:36.362 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:36.332+0000 7f183ac0a700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f183411bcc0 con 0x7f183411b890 2019-12-05T06:39:36.363 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:36.332+0000 7f18391a7700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f1834130d20 0x7f1834121e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:60192/0 (socket says 172.21.15.42:60192) 2019-12-05T06:39:36.363 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:36.332+0000 7f18391a7700 1 -- 172.21.15.42:0/17156745 learned_addr learned my addr 172.21.15.42:0/17156745 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:39:36.364 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:36.332+0000 7f18399a8700 1 --2- 172.21.15.42:0/17156745 >> v2:172.21.15.42:3300/0 conn(0x7f18341300a0 0x7f18340752d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:36.364 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:36.332+0000 7f18389a6700 1 --2- 172.21.15.42:0/17156745 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f183411b890 0x7f183407d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:36.364 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:36.332+0000 7f1829ffb700 1 -- 172.21.15.42:0/17156745 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 983792262 0 0) 0x7f183407d9c0 con 0x7f1834130d20 2019-12-05T06:39:36.365 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:36.332+0000 7f1829ffb700 1 -- 172.21.15.42:0/17156745 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18140030b0 con 0x7f1834130d20 2019-12-05T06:39:36.365 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:36.332+0000 7f18389a6700 1 -- 172.21.15.42:0/17156745 >> v2:172.21.15.42:3300/0 conn(0x7f18341300a0 msgr2=0x7f18340752d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:36.365 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:36.332+0000 7f18389a6700 1 --2- 172.21.15.42:0/17156745 >> v2:172.21.15.42:3300/0 conn(0x7f18341300a0 0x7f18340752d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:36.365 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:36.332+0000 7f18389a6700 1 -- 172.21.15.42:0/17156745 >> v1:172.21.15.42:6789/0 conn(0x7f1834130d20 legacy=0x7f1834121e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:36.366 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:36.332+0000 7f18389a6700 1 -- 172.21.15.42:0/17156745 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1834141e90 con 0x7f183411b890 2019-12-05T06:39:36.366 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:36.332+0000 7f18399a8700 1 --2- 172.21.15.42:0/17156745 >> v2:172.21.15.42:3300/0 conn(0x7f18341300a0 0x7f18340752d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:39:36.366 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:36.332+0000 7f18389a6700 1 --2- 172.21.15.42:0/17156745 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f183411b890 0x7f183407d470 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f1824008790 tx=0x7f1824008a80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:36.366 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:36.336+0000 7f18389a6700 1 --2- 172.21.15.42:0/17156745 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f183411b890 0x7f183407d470 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f1824008790 tx=0x7f1824008a80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:36.367 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:36.336+0000 7f18389a6700 1 --2- 172.21.15.42:0/17156745 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f183411b890 0x7f183407d470 secure :-1 s=READY pgs=651 cs=0 l=1 rx=0x7f1824008790 tx=0x7f1824008a80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:39:36.367 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:36.336+0000 7f18389a6700 1 --2- 172.21.15.42:0/17156745 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f183411b890 0x7f183407d470 secure :-1 s=THROTTLE_DONE pgs=651 cs=0 l=1 rx=0x7f1824008790 tx=0x7f1824008a80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:36.367 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:36.336+0000 7f1829ffb700 1 -- 172.21.15.42:0/17156745 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1824005540 con 0x7f183411b890 2019-12-05T06:39:36.367 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:36.336+0000 7f18389a6700 1 --2- 172.21.15.42:0/17156745 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f183411b890 0x7f183407d470 secure :-1 s=THROTTLE_DONE pgs=651 cs=0 l=1 rx=0x7f1824008790 tx=0x7f1824008a80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:36.367 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:36.336+0000 7f18389a6700 1 --2- 172.21.15.42:0/17156745 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f183411b890 0x7f183407d470 secure :-1 s=THROTTLE_DONE pgs=651 cs=0 l=1 rx=0x7f1824008790 tx=0x7f1824008a80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:36.368 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:36.336+0000 7f1829ffb700 1 -- 172.21.15.42:0/17156745 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1824008db0 con 0x7f183411b890 2019-12-05T06:39:36.368 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:36.336+0000 7f183ac0a700 1 -- 172.21.15.42:0/17156745 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18341304d0 con 0x7f183411b890 2019-12-05T06:39:36.368 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:36.336+0000 7f1829ffb700 1 -- 172.21.15.42:0/17156745 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1824005b20 con 0x7f183411b890 2019-12-05T06:39:36.368 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:36.336+0000 7f183ac0a700 1 -- 172.21.15.42:0/17156745 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1834075820 con 0x7f183411b890 2019-12-05T06:39:36.369 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:36.336+0000 7f18389a6700 1 --2- 172.21.15.42:0/17156745 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f183411b890 0x7f183407d470 secure :-1 s=THROTTLE_DONE pgs=651 cs=0 l=1 rx=0x7f1824008790 tx=0x7f1824008a80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:36.369 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:36.336+0000 7f1829ffb700 1 -- 172.21.15.42:0/17156745 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f1824013400 con 0x7f183411b890 2019-12-05T06:39:36.369 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:36.336+0000 7f18389a6700 1 --2- 172.21.15.42:0/17156745 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f183411b890 0x7f183407d470 secure :-1 s=THROTTLE_DONE pgs=651 cs=0 l=1 rx=0x7f1824008790 tx=0x7f1824008a80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:36.369 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:36.336+0000 7f1829ffb700 1 --2- 172.21.15.42:0/17156745 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f181403b2a0 0x7f181403d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:39:36.370 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:36.336+0000 7f1829ffb700 1 -- 172.21.15.42:0/17156745 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 11535+0+0 (secure 0 0 0) 0x7f182404b720 con 0x7f183411b890 2019-12-05T06:39:36.370 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:36.336+0000 7f18391a7700 1 --2- 172.21.15.42:0/17156745 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f181403b2a0 0x7f181403d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:36.370 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:36.336+0000 7f18391a7700 1 --2- 172.21.15.42:0/17156745 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f181403b2a0 0x7f181403d7b0 crc :-1 s=READY pgs=1228 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:39:36.371 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:36.344+0000 7f183ac0a700 1 -- 172.21.15.42:0/17156745 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f183405d100 con 0x7f183411b890 2019-12-05T06:39:36.371 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:36.352+0000 7f18389a6700 1 --2- 172.21.15.42:0/17156745 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f183411b890 0x7f183407d470 secure :-1 s=THROTTLE_DONE pgs=651 cs=0 l=1 rx=0x7f1824008790 tx=0x7f1824008a80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:36.371 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:36.352+0000 7f1829ffb700 1 -- 172.21.15.42:0/17156745 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f1824013d00 con 0x7f183411b890 2019-12-05T06:39:36.528 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:36.520+0000 7f183ac0a700 1 -- 172.21.15.42:0/17156745 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f183405b8b0 con 0x7f183411b890 2019-12-05T06:39:36.528 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:36.524+0000 7f18389a6700 1 --2- 172.21.15.42:0/17156745 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f183411b890 0x7f183407d470 secure :-1 s=THROTTLE_DONE pgs=651 cs=0 l=1 rx=0x7f1824008790 tx=0x7f1824008a80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:36.528 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:36.524+0000 7f1829ffb700 1 -- 172.21.15.42:0/17156745 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 352 v352) v1 ==== 95+0+1782 (secure 0 0 0) 0x7f1824030480 con 0x7f183411b890 2019-12-05T06:39:36.529 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:39:36.529 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":352,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":352,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:39:36.474528+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":113,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8009},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8009":{"gid":8009,"name":"b","rank":0,"incarnation":349,"state":"up:clientreplay","state_seq":26,"addr":"172.21.15.40:6833/1777669201","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1777669201},{"type":"v1","addr":"172.21.15.40:6833","nonce":1777669201}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:39:36.529 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 352 2019-12-05T06:39:36.531 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:36.524+0000 7f183ac0a700 1 -- 172.21.15.42:0/17156745 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f181403b2a0 msgr2=0x7f181403d7b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:36.532 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:36.524+0000 7f183ac0a700 1 --2- 172.21.15.42:0/17156745 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f181403b2a0 0x7f181403d7b0 crc :-1 s=READY pgs=1228 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:36.532 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:36.524+0000 7f183ac0a700 1 -- 172.21.15.42:0/17156745 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f183411b890 msgr2=0x7f183407d470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:36.532 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:36.524+0000 7f183ac0a700 1 --2- 172.21.15.42:0/17156745 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f183411b890 0x7f183407d470 secure :-1 s=READY pgs=651 cs=0 l=1 rx=0x7f1824008790 tx=0x7f1824008a80).stop 2019-12-05T06:39:36.532 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:36.524+0000 7f183ac0a700 1 -- 172.21.15.42:0/17156745 shutdown_connections 2019-12-05T06:39:36.533 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:36.524+0000 7f183ac0a700 1 --2- 172.21.15.42:0/17156745 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f181403b2a0 0x7f181403d7b0 unknown :-1 s=CLOSED pgs=1228 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:36.533 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:36.524+0000 7f183ac0a700 1 --2- 172.21.15.42:0/17156745 >> v2:172.21.15.42:3300/0 conn(0x7f18341300a0 0x7f18340752d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:36.533 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:36.524+0000 7f183ac0a700 1 --2- 172.21.15.42:0/17156745 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f183411b890 0x7f183407d470 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:36.533 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:36.524+0000 7f183ac0a700 1 -- 172.21.15.42:0/17156745 shutdown_connections 2019-12-05T06:39:36.533 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:36.524+0000 7f183ac0a700 1 -- 172.21.15.42:0/17156745 wait complete. 2019-12-05T06:39:36.533 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:36.524+0000 7f183ac0a700 1 -- 172.21.15.42:0/17156745 >> 172.21.15.42:0/17156745 conn(0x7f183411c4a0 msgr2=0x7f183411cbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:39:36.541 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2019-12-05T06:39:38.544 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:39:38.626 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:38.616+0000 7f78893b0700 1 Processor -- start 2019-12-05T06:39:38.664 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:38.616+0000 7f78893b0700 1 -- start start 2019-12-05T06:39:38.665 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:38.616+0000 7f78893b0700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f788408df80 0x7f788409f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:39:38.665 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:38.616+0000 7f78893b0700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f788409fc60 con 0x7f78840a7740 2019-12-05T06:39:38.665 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:38.616+0000 7f78893b0700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78840a0710 con 0x7f788409f830 2019-12-05T06:39:38.665 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:38.616+0000 7f78893b0700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78840a0b00 con 0x7f788408df80 2019-12-05T06:39:38.665 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:38.616+0000 7f7883fff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f788409f830 0x7f788414efe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:60250/0 (socket says 172.21.15.42:60250) 2019-12-05T06:39:38.666 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:38.616+0000 7f7883fff700 1 -- 172.21.15.42:0/3087907757 learned_addr learned my addr 172.21.15.42:0/3087907757 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:39:38.666 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:38.616+0000 7f7882ffd700 1 --2- 172.21.15.42:0/3087907757 >> v2:172.21.15.40:3300/0 conn(0x7f788408df80 0x7f788409f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:38.666 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:38.616+0000 7f78827fc700 1 -- 172.21.15.42:0/3087907757 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2993611860 0 0) 0x7f78840a0710 con 0x7f788409f830 2019-12-05T06:39:38.666 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:38.616+0000 7f78827fc700 1 -- 172.21.15.42:0/3087907757 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f786c0030b0 con 0x7f788409f830 2019-12-05T06:39:38.666 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:38.616+0000 7f7882ffd700 1 -- 172.21.15.42:0/3087907757 >> v1:172.21.15.42:6789/0 conn(0x7f788409f830 legacy=0x7f788414efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:38.666 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:38.616+0000 7f7882ffd700 1 -- 172.21.15.42:0/3087907757 >> v1:172.21.15.40:6789/0 conn(0x7f78840a7740 legacy=0x7f788407e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:38.666 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:38.616+0000 7f78827fc700 1 -- 172.21.15.42:0/3087907757 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1538685693 0 0) 0x7f788409fc60 con 0x7f78840a7740 2019-12-05T06:39:38.667 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:38.616+0000 7f7882ffd700 1 -- 172.21.15.42:0/3087907757 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f788407ef20 con 0x7f788408df80 2019-12-05T06:39:38.667 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:38.616+0000 7f7882ffd700 1 --2- 172.21.15.42:0/3087907757 >> v2:172.21.15.40:3300/0 conn(0x7f788408df80 0x7f788409f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f787c001d10 tx=0x7f787c001820).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:38.667 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:38.616+0000 7f7882ffd700 1 --2- 172.21.15.42:0/3087907757 >> v2:172.21.15.40:3300/0 conn(0x7f788408df80 0x7f788409f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f787c001d10 tx=0x7f787c001820).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:38.667 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:38.616+0000 7f7882ffd700 1 --2- 172.21.15.42:0/3087907757 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f788408df80 0x7f788409f2e0 secure :-1 s=READY pgs=2220 cs=0 l=1 rx=0x7f787c001d10 tx=0x7f787c001820).ready entity=mon.0 client_cookie=fea1c45fe243bb13 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:39:38.667 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:38.620+0000 7f7882ffd700 1 --2- 172.21.15.42:0/3087907757 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f788408df80 0x7f788409f2e0 secure :-1 s=THROTTLE_DONE pgs=2220 cs=0 l=1 rx=0x7f787c001d10 tx=0x7f787c001820).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:38.667 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:38.620+0000 7f7882ffd700 1 --2- 172.21.15.42:0/3087907757 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f788408df80 0x7f788409f2e0 secure :-1 s=THROTTLE_DONE pgs=2220 cs=0 l=1 rx=0x7f787c001d10 tx=0x7f787c001820).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:38.667 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:38.620+0000 7f78827fc700 1 -- 172.21.15.42:0/3087907757 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f787c001ad0 con 0x7f788408df80 2019-12-05T06:39:38.667 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:38.620+0000 7f7882ffd700 1 --2- 172.21.15.42:0/3087907757 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f788408df80 0x7f788409f2e0 secure :-1 s=THROTTLE_DONE pgs=2220 cs=0 l=1 rx=0x7f787c001d10 tx=0x7f787c001820).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:38.668 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:38.620+0000 7f78827fc700 1 -- 172.21.15.42:0/3087907757 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f787c0018e0 con 0x7f788408df80 2019-12-05T06:39:38.668 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:38.620+0000 7f78827fc700 1 -- 172.21.15.42:0/3087907757 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7878011170 con 0x7f788408df80 2019-12-05T06:39:38.668 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:38.620+0000 7f78893b0700 1 -- 172.21.15.42:0/3087907757 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f788408df80 msgr2=0x7f788409f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:38.668 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:38.620+0000 7f78893b0700 1 --2- 172.21.15.42:0/3087907757 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f788408df80 0x7f788409f2e0 secure :-1 s=READY pgs=2220 cs=0 l=1 rx=0x7f787c001d10 tx=0x7f787c001820).stop 2019-12-05T06:39:38.668 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:38.620+0000 7f78893b0700 1 -- 172.21.15.42:0/3087907757 shutdown_connections 2019-12-05T06:39:38.668 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:38.620+0000 7f78893b0700 1 --2- 172.21.15.42:0/3087907757 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f788408df80 0x7f788409f2e0 unknown :-1 s=CLOSED pgs=2220 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:38.669 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:38.620+0000 7f78893b0700 1 -- 172.21.15.42:0/3087907757 shutdown_connections 2019-12-05T06:39:38.669 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:38.620+0000 7f78893b0700 1 -- 172.21.15.42:0/3087907757 wait complete. 2019-12-05T06:39:38.669 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:38.620+0000 7f78893b0700 1 -- 172.21.15.42:0/3087907757 >> 172.21.15.42:0/3087907757 conn(0x7f78840aabe0 msgr2=0x7f78840a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:39:38.669 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:38.620+0000 7f78893b0700 1 Processor -- start 2019-12-05T06:39:38.669 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:38.620+0000 7f78893b0700 1 -- start start 2019-12-05T06:39:38.674 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:38.620+0000 7f78893b0700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f7884122980 0x7f78841447c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:39:38.674 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:38.620+0000 7f78893b0700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7884142c00 0x7f78840a3db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:39:38.674 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:38.620+0000 7f78893b0700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f78841435d0 0x7f7884143a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:39:38.674 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:38.620+0000 7f78893b0700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78841221a0 con 0x7f7884142c00 2019-12-05T06:39:38.675 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:38.620+0000 7f78893b0700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7884122440 con 0x7f7884122980 2019-12-05T06:39:38.675 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:38.620+0000 7f78893b0700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78841226e0 con 0x7f78841435d0 2019-12-05T06:39:38.675 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:38.620+0000 7f78837fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f7884122980 0x7f78841447c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:38.675 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:38.620+0000 7f78837fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f7884122980 0x7f78841447c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:60214/0 (socket says 172.21.15.42:60214) 2019-12-05T06:39:38.675 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:38.620+0000 7f78837fe700 1 -- 172.21.15.42:0/3418117069 learned_addr learned my addr 172.21.15.42:0/3418117069 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:39:38.676 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:38.620+0000 7f7883fff700 1 --2- 172.21.15.42:0/3418117069 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f78841435d0 0x7f7884143a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:38.676 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:38.620+0000 7f7882ffd700 1 --2- 172.21.15.42:0/3418117069 >> v2:172.21.15.40:3300/0 conn(0x7f7884142c00 0x7f78840a3db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:38.676 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:38.620+0000 7f7882ffd700 1 -- 172.21.15.42:0/3418117069 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f78841435d0 msgr2=0x7f7884143a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:38.676 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:38.620+0000 7f7882ffd700 1 --2- 172.21.15.42:0/3418117069 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f78841435d0 0x7f7884143a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:38.677 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:38.620+0000 7f7882ffd700 1 -- 172.21.15.42:0/3418117069 >> v2:172.21.15.42:3300/0 conn(0x7f7884122980 msgr2=0x7f78841447c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:38.677 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:38.620+0000 7f7882ffd700 1 --2- 172.21.15.42:0/3418117069 >> v2:172.21.15.42:3300/0 conn(0x7f7884122980 0x7f78841447c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:38.677 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:38.620+0000 7f7882ffd700 1 -- 172.21.15.42:0/3418117069 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7884121ed0 con 0x7f7884142c00 2019-12-05T06:39:38.677 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:38.624+0000 7f7883fff700 1 --2- 172.21.15.42:0/3418117069 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f78841435d0 0x7f7884143a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:39:38.678 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:38.624+0000 7f7882ffd700 1 --2- 172.21.15.42:0/3418117069 >> v2:172.21.15.40:3300/0 conn(0x7f7884142c00 0x7f78840a3db0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f7884122360 tx=0x7f7878008570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:38.678 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:38.624+0000 7f78837fe700 1 --2- 172.21.15.42:0/3418117069 >> v2:172.21.15.42:3300/0 conn(0x7f7884122980 0x7f78841447c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:39:38.678 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:38.624+0000 7f7882ffd700 1 --2- 172.21.15.42:0/3418117069 >> v2:172.21.15.40:3300/0 conn(0x7f7884142c00 0x7f78840a3db0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f7884122360 tx=0x7f7878008570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:38.678 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:38.624+0000 7f7882ffd700 1 --2- 172.21.15.42:0/3418117069 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7884142c00 0x7f78840a3db0 secure :-1 s=READY pgs=2221 cs=0 l=1 rx=0x7f7884122360 tx=0x7f7878008570).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:39:38.679 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:38.624+0000 7f7882ffd700 1 --2- 172.21.15.42:0/3418117069 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7884142c00 0x7f78840a3db0 secure :-1 s=THROTTLE_DONE pgs=2221 cs=0 l=1 rx=0x7f7884122360 tx=0x7f7878008570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:38.679 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:38.624+0000 7f786bfff700 1 -- 172.21.15.42:0/3418117069 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7878007340 con 0x7f7884142c00 2019-12-05T06:39:38.679 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:38.624+0000 7f7882ffd700 1 --2- 172.21.15.42:0/3418117069 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7884142c00 0x7f78840a3db0 secure :-1 s=THROTTLE_DONE pgs=2221 cs=0 l=1 rx=0x7f7884122360 tx=0x7f7878008570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:38.680 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:38.624+0000 7f7882ffd700 1 --2- 172.21.15.42:0/3418117069 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7884142c00 0x7f78840a3db0 secure :-1 s=THROTTLE_DONE pgs=2221 cs=0 l=1 rx=0x7f7884122360 tx=0x7f7878008570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:38.680 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:38.624+0000 7f786bfff700 1 -- 172.21.15.42:0/3418117069 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7878008b80 con 0x7f7884142c00 2019-12-05T06:39:38.680 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:38.624+0000 7f786bfff700 1 -- 172.21.15.42:0/3418117069 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7878011b30 con 0x7f7884142c00 2019-12-05T06:39:38.680 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:38.624+0000 7f78893b0700 1 -- 172.21.15.42:0/3418117069 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7884143f50 con 0x7f7884142c00 2019-12-05T06:39:38.681 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:38.624+0000 7f78893b0700 1 -- 172.21.15.42:0/3418117069 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7884143030 con 0x7f7884142c00 2019-12-05T06:39:38.681 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:38.624+0000 7f7882ffd700 1 --2- 172.21.15.42:0/3418117069 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7884142c00 0x7f78840a3db0 secure :-1 s=THROTTLE_DONE pgs=2221 cs=0 l=1 rx=0x7f7884122360 tx=0x7f7878008570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:38.681 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:38.624+0000 7f786bfff700 1 -- 172.21.15.42:0/3418117069 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f7878005540 con 0x7f7884142c00 2019-12-05T06:39:38.681 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:38.624+0000 7f7882ffd700 1 --2- 172.21.15.42:0/3418117069 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7884142c00 0x7f78840a3db0 secure :-1 s=THROTTLE_DONE pgs=2221 cs=0 l=1 rx=0x7f7884122360 tx=0x7f7878008570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:38.682 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:38.624+0000 7f786bfff700 1 --2- 172.21.15.42:0/3418117069 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f786403ae90 0x7f786403d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:39:38.682 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:38.624+0000 7f786bfff700 1 -- 172.21.15.42:0/3418117069 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 11535+0+0 (secure 0 0 0) 0x7f787804b1d0 con 0x7f7884142c00 2019-12-05T06:39:38.682 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:38.624+0000 7f78837fe700 1 --2- 172.21.15.42:0/3418117069 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f786403ae90 0x7f786403d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:38.683 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:38.624+0000 7f78837fe700 1 --2- 172.21.15.42:0/3418117069 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f786403ae90 0x7f786403d3a0 crc :-1 s=READY pgs=1229 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:39:38.683 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:38.632+0000 7f78893b0700 1 -- 172.21.15.42:0/3418117069 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f788405d110 con 0x7f7884142c00 2019-12-05T06:39:38.683 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:38.636+0000 7f7882ffd700 1 --2- 172.21.15.42:0/3418117069 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7884142c00 0x7f78840a3db0 secure :-1 s=THROTTLE_DONE pgs=2221 cs=0 l=1 rx=0x7f7884122360 tx=0x7f7878008570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:38.683 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:38.636+0000 7f786bfff700 1 -- 172.21.15.42:0/3418117069 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f78780330e0 con 0x7f7884142c00 2019-12-05T06:39:38.845 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:38.840+0000 7f78893b0700 1 -- 172.21.15.42:0/3418117069 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f788405b8c0 con 0x7f7884142c00 2019-12-05T06:39:38.847 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:38.840+0000 7f7882ffd700 1 --2- 172.21.15.42:0/3418117069 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7884142c00 0x7f78840a3db0 secure :-1 s=THROTTLE_DONE pgs=2221 cs=0 l=1 rx=0x7f7884122360 tx=0x7f7878008570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:38.847 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:38.840+0000 7f786bfff700 1 -- 172.21.15.42:0/3418117069 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 353 v353) v1 ==== 95+0+1776 (secure 0 0 0) 0x7f7878005850 con 0x7f7884142c00 2019-12-05T06:39:38.847 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 353 2019-12-05T06:39:38.848 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:39:38.848 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":353,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":353,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:39:37.486362+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":113,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8009},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8009":{"gid":8009,"name":"b","rank":0,"incarnation":349,"state":"up:active","state_seq":27,"addr":"172.21.15.40:6833/1777669201","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1777669201},{"type":"v1","addr":"172.21.15.40:6833","nonce":1777669201}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:39:38.857 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:38.844+0000 7f78893b0700 1 -- 172.21.15.42:0/3418117069 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f786403ae90 msgr2=0x7f786403d3a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:38.857 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:38.844+0000 7f78893b0700 1 --2- 172.21.15.42:0/3418117069 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f786403ae90 0x7f786403d3a0 crc :-1 s=READY pgs=1229 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:38.857 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:38.844+0000 7f78893b0700 1 -- 172.21.15.42:0/3418117069 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7884142c00 msgr2=0x7f78840a3db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:38.858 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:38.844+0000 7f78893b0700 1 --2- 172.21.15.42:0/3418117069 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7884142c00 0x7f78840a3db0 secure :-1 s=READY pgs=2221 cs=0 l=1 rx=0x7f7884122360 tx=0x7f7878008570).stop 2019-12-05T06:39:38.858 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:38.844+0000 7f78893b0700 1 -- 172.21.15.42:0/3418117069 shutdown_connections 2019-12-05T06:39:38.858 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:38.844+0000 7f78893b0700 1 --2- 172.21.15.42:0/3418117069 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f786403ae90 0x7f786403d3a0 unknown :-1 s=CLOSED pgs=1229 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:38.858 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:38.844+0000 7f78893b0700 1 --2- 172.21.15.42:0/3418117069 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f78841435d0 0x7f7884143a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:38.858 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:38.844+0000 7f78893b0700 1 --2- 172.21.15.42:0/3418117069 >> v2:172.21.15.42:3300/0 conn(0x7f7884122980 0x7f78841447c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:38.859 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:38.844+0000 7f78893b0700 1 --2- 172.21.15.42:0/3418117069 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7884142c00 0x7f78840a3db0 unknown :-1 s=CLOSED pgs=2221 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:38.859 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:38.844+0000 7f78893b0700 1 -- 172.21.15.42:0/3418117069 shutdown_connections 2019-12-05T06:39:38.859 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:38.844+0000 7f78893b0700 1 -- 172.21.15.42:0/3418117069 wait complete. 2019-12-05T06:39:38.859 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:38.844+0000 7f78893b0700 1 -- 172.21.15.42:0/3418117069 >> 172.21.15.42:0/3418117069 conn(0x7f788409f3f0 msgr2=0x7f78840a9790 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:39:38.860 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b has gained rank=0, replacing gid=11887 2019-12-05T06:39:38.860 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 1 secs before reviving mds.a 2019-12-05T06:39:39.861 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.a 2019-12-05T06:39:39.862 INFO:tasks.ceph.mds.a:Restarting daemon 2019-12-05T06:39:39.862 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2019-12-05T06:39:39.865 INFO:tasks.ceph.mds.a:Started 2019-12-05T06:39:39.865 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:39:39.920 INFO:tasks.ceph.mds.a.smithi042.stdout:starting mds.a at 2019-12-05T06:39:39.957 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:39.948+0000 7f5bbf42c700 1 Processor -- start 2019-12-05T06:39:39.957 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:39.948+0000 7f5bbf42c700 1 -- start start 2019-12-05T06:39:39.957 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:39.948+0000 7f5bbf42c700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5bb80a7740 0x7f5bb808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:39:39.958 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:39.948+0000 7f5bbf42c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5bb807e720 0x7f5bb807eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:39:39.958 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:39.948+0000 7f5bbf42c700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bb80a0710 con 0x7f5bb809f2e0 2019-12-05T06:39:39.958 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:39.948+0000 7f5bbf42c700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5bb80a0b00 con 0x7f5bb80a7740 2019-12-05T06:39:39.958 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:39.948+0000 7f5bbf42c700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5bb808e5b0 con 0x7f5bb807e720 2019-12-05T06:39:39.958 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:39.948+0000 7f5bbd1c8700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5bb807e720 0x7f5bb807eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:39.958 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:39.948+0000 7f5bbd1c8700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5bb807e720 0x7f5bb807eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:60248/0 (socket says 172.21.15.42:60248) 2019-12-05T06:39:39.959 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:39.948+0000 7f5bbd1c8700 1 -- 172.21.15.42:0/1296215870 learned_addr learned my addr 172.21.15.42:0/1296215870 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:39:39.959 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:39.948+0000 7f5bbd9c9700 1 --2- 172.21.15.42:0/1296215870 >> v2:172.21.15.40:3300/0 conn(0x7f5bb80a7740 0x7f5bb808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:39.959 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:39.948+0000 7f5bbd1c8700 1 -- 172.21.15.42:0/1296215870 >> v2:172.21.15.40:3300/0 conn(0x7f5bb80a7740 msgr2=0x7f5bb808df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:39:39.959 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:39.948+0000 7f5bbd1c8700 1 --2- 172.21.15.42:0/1296215870 >> v2:172.21.15.40:3300/0 conn(0x7f5bb80a7740 0x7f5bb808df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:39:39.959 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:39.948+0000 7f5bbd1c8700 1 -- 172.21.15.42:0/1296215870 >> v1:172.21.15.40:6789/0 conn(0x7f5bb809f2e0 legacy=0x7f5bb809f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:39.959 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:39.948+0000 7f5bbd1c8700 1 -- 172.21.15.42:0/1296215870 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bb807f0a0 con 0x7f5bb807e720 2019-12-05T06:39:39.959 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:39.948+0000 7f5bbd1c8700 1 --2- 172.21.15.42:0/1296215870 >> v2:172.21.15.42:3300/0 conn(0x7f5bb807e720 0x7f5bb807eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f5ba8002f30 tx=0x7f5ba800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:39.960 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:39.948+0000 7f5bbd1c8700 1 --2- 172.21.15.42:0/1296215870 >> v2:172.21.15.42:3300/0 conn(0x7f5bb807e720 0x7f5bb807eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f5ba8002f30 tx=0x7f5ba800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:39.960 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:39.948+0000 7f5bbd1c8700 1 --2- 172.21.15.42:0/1296215870 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5bb807e720 0x7f5bb807eb50 secure :-1 s=READY pgs=2932 cs=0 l=1 rx=0x7f5ba8002f30 tx=0x7f5ba800fd30).ready entity=mon.1 client_cookie=45f3051e13df2305 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:39:39.960 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:39.948+0000 7f5bbd1c8700 1 --2- 172.21.15.42:0/1296215870 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5bb807e720 0x7f5bb807eb50 secure :-1 s=THROTTLE_DONE pgs=2932 cs=0 l=1 rx=0x7f5ba8002f30 tx=0x7f5ba800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:39.960 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:39.948+0000 7f5bbd1c8700 1 --2- 172.21.15.42:0/1296215870 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5bb807e720 0x7f5bb807eb50 secure :-1 s=THROTTLE_DONE pgs=2932 cs=0 l=1 rx=0x7f5ba8002f30 tx=0x7f5ba800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:39.960 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:39.948+0000 7f5bbd1c8700 1 --2- 172.21.15.42:0/1296215870 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5bb807e720 0x7f5bb807eb50 secure :-1 s=THROTTLE_DONE pgs=2932 cs=0 l=1 rx=0x7f5ba8002f30 tx=0x7f5ba800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:39.960 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:39.948+0000 7f5bbc9c7700 1 -- 172.21.15.42:0/1296215870 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5ba8011370 con 0x7f5bb807e720 2019-12-05T06:39:39.961 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:39.948+0000 7f5bbc9c7700 1 -- 172.21.15.42:0/1296215870 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ba8008bb0 con 0x7f5bb807e720 2019-12-05T06:39:39.961 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:39.948+0000 7f5bbc9c7700 1 -- 172.21.15.42:0/1296215870 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5ba8011950 con 0x7f5bb807e720 2019-12-05T06:39:39.961 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:39.948+0000 7f5bbf42c700 1 -- 172.21.15.42:0/1296215870 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5bb807e720 msgr2=0x7f5bb807eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:39.961 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:39.948+0000 7f5bbf42c700 1 --2- 172.21.15.42:0/1296215870 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5bb807e720 0x7f5bb807eb50 secure :-1 s=READY pgs=2932 cs=0 l=1 rx=0x7f5ba8002f30 tx=0x7f5ba800fd30).stop 2019-12-05T06:39:39.961 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:39.948+0000 7f5bbf42c700 1 -- 172.21.15.42:0/1296215870 shutdown_connections 2019-12-05T06:39:39.961 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:39.948+0000 7f5bbf42c700 1 --2- 172.21.15.42:0/1296215870 >> v2:172.21.15.40:3300/0 conn(0x7f5bb80a7740 0x7f5bb808df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:39:39.962 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:39.948+0000 7f5bbf42c700 1 --2- 172.21.15.42:0/1296215870 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5bb807e720 0x7f5bb807eb50 unknown :-1 s=CLOSED pgs=2932 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:39.963 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:39.948+0000 7f5bbf42c700 1 -- 172.21.15.42:0/1296215870 shutdown_connections 2019-12-05T06:39:39.963 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:39.948+0000 7f5bbf42c700 1 -- 172.21.15.42:0/1296215870 wait complete. 2019-12-05T06:39:39.963 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:39.948+0000 7f5bbf42c700 1 -- 172.21.15.42:0/1296215870 >> 172.21.15.42:0/1296215870 conn(0x7f5bb80aabe0 msgr2=0x7f5bb80a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:39:39.963 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:39.952+0000 7f5bbf42c700 1 Processor -- start 2019-12-05T06:39:39.964 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:39.952+0000 7f5bbf42c700 1 -- start start 2019-12-05T06:39:39.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:39.952+0000 7f5bbf42c700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5bb8064460 0x7f5bb80a6de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:39:39.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:39.952+0000 7f5bbf42c700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5bb80a1e40 0x7f5bb80a2270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:39:39.976 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:39.952+0000 7f5bbf42c700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bb80a3f60 con 0x7f5bb80a3550 2019-12-05T06:39:39.976 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:39.952+0000 7f5bbf42c700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5bb80a3a20 con 0x7f5bb8064460 2019-12-05T06:39:39.976 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:39.952+0000 7f5bbf42c700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bb80a3cc0 con 0x7f5bb80a1e40 2019-12-05T06:39:39.976 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:39.952+0000 7f5bbd9c9700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f5bb80a3550 0x7f5bb807d710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:60290/0 (socket says 172.21.15.42:60290) 2019-12-05T06:39:39.976 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:39.952+0000 7f5bbd1c8700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5bb8064460 0x7f5bb80a6de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:39.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:39.952+0000 7f5bbd9c9700 1 -- 172.21.15.42:0/1233631997 learned_addr learned my addr 172.21.15.42:0/1233631997 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:39:39.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:39.952+0000 7f5bbe1ca700 1 --2- 172.21.15.42:0/1233631997 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5bb80a1e40 0x7f5bb80a2270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:39.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:39.952+0000 7f5bae7fc700 1 -- 172.21.15.42:0/1233631997 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3910817607 0 0) 0x7f5bb80a3f60 con 0x7f5bb80a3550 2019-12-05T06:39:39.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:39.952+0000 7f5bbd1c8700 1 -- 172.21.15.42:0/1233631997 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5bb80a1e40 msgr2=0x7f5bb80a2270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:39.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:39.952+0000 7f5bbd1c8700 1 --2- 172.21.15.42:0/1233631997 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5bb80a1e40 0x7f5bb80a2270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:39.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:39.952+0000 7f5bbd1c8700 1 -- 172.21.15.42:0/1233631997 >> v1:172.21.15.42:6789/0 conn(0x7f5bb80a3550 legacy=0x7f5bb807d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:39.978 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:39.952+0000 7f5bbd1c8700 1 -- 172.21.15.42:0/1233631997 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bb80a1670 con 0x7f5bb8064460 2019-12-05T06:39:39.978 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:39.952+0000 7f5bbd1c8700 1 --2- 172.21.15.42:0/1233631997 >> v2:172.21.15.40:3300/0 conn(0x7f5bb8064460 0x7f5bb80a6de0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f5bb80a3be0 tx=0x7f5ba8011e90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:39.978 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:39.952+0000 7f5bbd1c8700 1 --2- 172.21.15.42:0/1233631997 >> v2:172.21.15.40:3300/0 conn(0x7f5bb8064460 0x7f5bb80a6de0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f5bb80a3be0 tx=0x7f5ba8011e90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:39.978 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:39.952+0000 7f5bbd1c8700 1 --2- 172.21.15.42:0/1233631997 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5bb8064460 0x7f5bb80a6de0 secure :-1 s=READY pgs=2227 cs=0 l=1 rx=0x7f5bb80a3be0 tx=0x7f5ba8011e90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:39:39.978 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:39.952+0000 7f5bbd1c8700 1 --2- 172.21.15.42:0/1233631997 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5bb8064460 0x7f5bb80a6de0 secure :-1 s=THROTTLE_DONE pgs=2227 cs=0 l=1 rx=0x7f5bb80a3be0 tx=0x7f5ba8011e90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:39.978 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:39.952+0000 7f5bbd1c8700 1 --2- 172.21.15.42:0/1233631997 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5bb8064460 0x7f5bb80a6de0 secure :-1 s=THROTTLE_DONE pgs=2227 cs=0 l=1 rx=0x7f5bb80a3be0 tx=0x7f5ba8011e90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:39.979 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:39.952+0000 7f5bae7fc700 1 -- 172.21.15.42:0/1233631997 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5ba8007340 con 0x7f5bb8064460 2019-12-05T06:39:39.979 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:39.952+0000 7f5bbd1c8700 1 --2- 172.21.15.42:0/1233631997 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5bb8064460 0x7f5bb80a6de0 secure :-1 s=THROTTLE_DONE pgs=2227 cs=0 l=1 rx=0x7f5bb80a3be0 tx=0x7f5ba8011e90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:39.979 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:39.952+0000 7f5bae7fc700 1 -- 172.21.15.42:0/1233631997 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ba8011570 con 0x7f5bb8064460 2019-12-05T06:39:39.979 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:39.952+0000 7f5bbf42c700 1 -- 172.21.15.42:0/1233631997 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bb807df10 con 0x7f5bb8064460 2019-12-05T06:39:39.979 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:39.952+0000 7f5bae7fc700 1 -- 172.21.15.42:0/1233631997 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5ba8005540 con 0x7f5bb8064460 2019-12-05T06:39:39.979 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:39.952+0000 7f5bbf42c700 1 -- 172.21.15.42:0/1233631997 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bb8137be0 con 0x7f5bb8064460 2019-12-05T06:39:39.979 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:39.952+0000 7f5bbd1c8700 1 --2- 172.21.15.42:0/1233631997 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5bb8064460 0x7f5bb80a6de0 secure :-1 s=THROTTLE_DONE pgs=2227 cs=0 l=1 rx=0x7f5bb80a3be0 tx=0x7f5ba8011e90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:39.980 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:39.956+0000 7f5bae7fc700 1 -- 172.21.15.42:0/1233631997 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f5ba8005850 con 0x7f5bb8064460 2019-12-05T06:39:39.980 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:39.956+0000 7f5bbd1c8700 1 --2- 172.21.15.42:0/1233631997 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5bb8064460 0x7f5bb80a6de0 secure :-1 s=THROTTLE_DONE pgs=2227 cs=0 l=1 rx=0x7f5bb80a3be0 tx=0x7f5ba8011e90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:39.980 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:39.956+0000 7f5bae7fc700 1 --2- 172.21.15.42:0/1233631997 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5ba003acf0 0x7f5ba003d200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:39:39.980 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:39.956+0000 7f5bbf42c700 1 -- 172.21.15.42:0/1233631997 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5bb805d110 con 0x7f5bb8064460 2019-12-05T06:39:39.981 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:39.956+0000 7f5bae7fc700 1 -- 172.21.15.42:0/1233631997 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 11535+0+0 (secure 0 0 0) 0x7f5ba804b3a0 con 0x7f5bb8064460 2019-12-05T06:39:39.981 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:39.956+0000 7f5bbd9c9700 1 --2- 172.21.15.42:0/1233631997 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5ba003acf0 0x7f5ba003d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:39.981 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:39.956+0000 7f5bbd9c9700 1 --2- 172.21.15.42:0/1233631997 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5ba003acf0 0x7f5ba003d200 crc :-1 s=READY pgs=1230 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:39:39.981 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:39.960+0000 7f5bbd1c8700 1 --2- 172.21.15.42:0/1233631997 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5bb8064460 0x7f5bb80a6de0 secure :-1 s=THROTTLE_DONE pgs=2227 cs=0 l=1 rx=0x7f5bb80a3be0 tx=0x7f5ba8011e90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:39.981 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:39.960+0000 7f5bae7fc700 1 -- 172.21.15.42:0/1233631997 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f5bb80a1670 con 0x7f5bb8064460 2019-12-05T06:39:40.175 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:40.168+0000 7f5bbf42c700 1 -- 172.21.15.42:0/1233631997 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5bb80aadf0 con 0x7f5bb8064460 2019-12-05T06:39:40.176 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:40.168+0000 7f5bbd1c8700 1 --2- 172.21.15.42:0/1233631997 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5bb8064460 0x7f5bb80a6de0 secure :-1 s=THROTTLE_DONE pgs=2227 cs=0 l=1 rx=0x7f5bb80a3be0 tx=0x7f5ba8011e90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:40.176 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:40.168+0000 7f5bae7fc700 1 -- 172.21.15.42:0/1233631997 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 353 v353) v1 ==== 95+0+1776 (secure 0 0 0) 0x7f5ba8032420 con 0x7f5bb8064460 2019-12-05T06:39:40.176 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 353 2019-12-05T06:39:40.177 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:39:40.177 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":353,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":353,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:39:37.486362+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":113,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8009},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8009":{"gid":8009,"name":"b","rank":0,"incarnation":349,"state":"up:active","state_seq":27,"addr":"172.21.15.40:6833/1777669201","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1777669201},{"type":"v1","addr":"172.21.15.40:6833","nonce":1777669201}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:39:40.179 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:40.172+0000 7f5bbf42c700 1 -- 172.21.15.42:0/1233631997 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5ba003acf0 msgr2=0x7f5ba003d200 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:40.180 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:40.172+0000 7f5bbf42c700 1 --2- 172.21.15.42:0/1233631997 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5ba003acf0 0x7f5ba003d200 crc :-1 s=READY pgs=1230 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:40.180 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:40.172+0000 7f5bbf42c700 1 -- 172.21.15.42:0/1233631997 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5bb8064460 msgr2=0x7f5bb80a6de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:40.180 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:40.172+0000 7f5bbf42c700 1 --2- 172.21.15.42:0/1233631997 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5bb8064460 0x7f5bb80a6de0 secure :-1 s=READY pgs=2227 cs=0 l=1 rx=0x7f5bb80a3be0 tx=0x7f5ba8011e90).stop 2019-12-05T06:39:40.180 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:40.172+0000 7f5bbf42c700 1 -- 172.21.15.42:0/1233631997 shutdown_connections 2019-12-05T06:39:40.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:40.172+0000 7f5bbf42c700 1 --2- 172.21.15.42:0/1233631997 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5ba003acf0 0x7f5ba003d200 unknown :-1 s=CLOSED pgs=1230 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:40.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:40.172+0000 7f5bbf42c700 1 --2- 172.21.15.42:0/1233631997 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5bb80a1e40 0x7f5bb80a2270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:40.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:40.172+0000 7f5bbf42c700 1 --2- 172.21.15.42:0/1233631997 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5bb8064460 0x7f5bb80a6de0 unknown :-1 s=CLOSED pgs=2227 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:40.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:40.172+0000 7f5bbf42c700 1 -- 172.21.15.42:0/1233631997 shutdown_connections 2019-12-05T06:39:40.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:40.172+0000 7f5bbf42c700 1 -- 172.21.15.42:0/1233631997 wait complete. 2019-12-05T06:39:40.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:40.172+0000 7f5bbf42c700 1 -- 172.21.15.42:0/1233631997 >> 172.21.15.42:0/1233631997 conn(0x7f5bb808fde0 msgr2=0x7f5bb80a9850 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:39:40.190 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2019-12-05T06:39:42.192 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:39:42.306 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:42.284+0000 7f5cfb9dd700 1 Processor -- start 2019-12-05T06:39:42.307 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:42.284+0000 7f5cfb9dd700 1 -- start start 2019-12-05T06:39:42.307 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:42.284+0000 7f5cfb9dd700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5cf40a7740 0x7f5cf408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:39:42.307 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:42.284+0000 7f5cfb9dd700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5cf40a0710 con 0x7f5cf407e720 2019-12-05T06:39:42.307 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:42.284+0000 7f5cfb9dd700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5cf40a9ba0 con 0x7f5cf409fae0 2019-12-05T06:39:42.307 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:42.284+0000 7f5cfb9dd700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5cf40a0b00 con 0x7f5cf40a7740 2019-12-05T06:39:42.308 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:42.284+0000 7f5cfa77b700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f5cf409fae0 0x7f5cf414efe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:60332/0 (socket says 172.21.15.42:60332) 2019-12-05T06:39:42.308 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:42.284+0000 7f5cfa77b700 1 -- 172.21.15.42:0/2841547563 learned_addr learned my addr 172.21.15.42:0/2841547563 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:39:42.308 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:42.284+0000 7f5cf9f7a700 1 --2- 172.21.15.42:0/2841547563 >> v2:172.21.15.40:3300/0 conn(0x7f5cf40a7740 0x7f5cf408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:42.308 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:42.284+0000 7f5cf8f78700 1 -- 172.21.15.42:0/2841547563 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1907954499 0 0) 0x7f5cf40a9ba0 con 0x7f5cf409fae0 2019-12-05T06:39:42.309 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:42.284+0000 7f5cf8f78700 1 -- 172.21.15.42:0/2841547563 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ce00030b0 con 0x7f5cf409fae0 2019-12-05T06:39:42.309 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:42.284+0000 7f5cf9f7a700 1 -- 172.21.15.42:0/2841547563 >> v1:172.21.15.42:6789/0 conn(0x7f5cf409fae0 legacy=0x7f5cf414efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:42.309 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:42.284+0000 7f5cf9f7a700 1 -- 172.21.15.42:0/2841547563 >> v1:172.21.15.40:6789/0 conn(0x7f5cf407e720 legacy=0x7f5cf409f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:42.309 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:42.284+0000 7f5cf9f7a700 1 -- 172.21.15.42:0/2841547563 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cf407eb50 con 0x7f5cf40a7740 2019-12-05T06:39:42.309 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:42.284+0000 7f5cf9f7a700 1 --2- 172.21.15.42:0/2841547563 >> v2:172.21.15.40:3300/0 conn(0x7f5cf40a7740 0x7f5cf408df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f5ce4008790 tx=0x7f5ce400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:42.310 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:42.288+0000 7f5cf9f7a700 1 --2- 172.21.15.42:0/2841547563 >> v2:172.21.15.40:3300/0 conn(0x7f5cf40a7740 0x7f5cf408df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f5ce4008790 tx=0x7f5ce400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:42.310 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:42.288+0000 7f5cf9f7a700 1 --2- 172.21.15.42:0/2841547563 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5cf40a7740 0x7f5cf408df80 secure :-1 s=READY pgs=2229 cs=0 l=1 rx=0x7f5ce4008790 tx=0x7f5ce400fbf0).ready entity=mon.0 client_cookie=c3eb166c6e7118ab server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:39:42.310 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:42.288+0000 7f5cf9f7a700 1 --2- 172.21.15.42:0/2841547563 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5cf40a7740 0x7f5cf408df80 secure :-1 s=THROTTLE_DONE pgs=2229 cs=0 l=1 rx=0x7f5ce4008790 tx=0x7f5ce400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:42.310 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:42.288+0000 7f5cf8f78700 1 -- 172.21.15.42:0/2841547563 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5ce4011270 con 0x7f5cf40a7740 2019-12-05T06:39:42.310 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:42.288+0000 7f5cf9f7a700 1 --2- 172.21.15.42:0/2841547563 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5cf40a7740 0x7f5cf408df80 secure :-1 s=THROTTLE_DONE pgs=2229 cs=0 l=1 rx=0x7f5ce4008790 tx=0x7f5ce400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:42.311 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:42.288+0000 7f5cf9f7a700 1 --2- 172.21.15.42:0/2841547563 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5cf40a7740 0x7f5cf408df80 secure :-1 s=THROTTLE_DONE pgs=2229 cs=0 l=1 rx=0x7f5ce4008790 tx=0x7f5ce400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:42.311 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:42.288+0000 7f5cf8f78700 1 -- 172.21.15.42:0/2841547563 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ce4008bc0 con 0x7f5cf40a7740 2019-12-05T06:39:42.311 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:42.288+0000 7f5cf8f78700 1 -- 172.21.15.42:0/2841547563 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5ce4011850 con 0x7f5cf40a7740 2019-12-05T06:39:42.311 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:42.288+0000 7f5cfb9dd700 1 -- 172.21.15.42:0/2841547563 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5cf40a7740 msgr2=0x7f5cf408df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:42.312 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:42.288+0000 7f5cfb9dd700 1 --2- 172.21.15.42:0/2841547563 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5cf40a7740 0x7f5cf408df80 secure :-1 s=READY pgs=2229 cs=0 l=1 rx=0x7f5ce4008790 tx=0x7f5ce400fbf0).stop 2019-12-05T06:39:42.312 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:42.288+0000 7f5cfb9dd700 1 -- 172.21.15.42:0/2841547563 shutdown_connections 2019-12-05T06:39:42.312 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:42.288+0000 7f5cfb9dd700 1 --2- 172.21.15.42:0/2841547563 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5cf40a7740 0x7f5cf408df80 unknown :-1 s=CLOSED pgs=2229 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:42.312 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:42.288+0000 7f5cfb9dd700 1 -- 172.21.15.42:0/2841547563 shutdown_connections 2019-12-05T06:39:42.312 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:42.288+0000 7f5cfb9dd700 1 -- 172.21.15.42:0/2841547563 wait complete. 2019-12-05T06:39:42.313 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:42.288+0000 7f5cfb9dd700 1 -- 172.21.15.42:0/2841547563 >> 172.21.15.42:0/2841547563 conn(0x7f5cf40aabe0 msgr2=0x7f5cf40a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:39:42.314 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:42.288+0000 7f5cfb9dd700 1 Processor -- start 2019-12-05T06:39:42.314 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:42.288+0000 7f5cfb9dd700 1 -- start start 2019-12-05T06:39:42.315 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:42.288+0000 7f5cfb9dd700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5cf4118ee0 0x7f5cf40a2e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:39:42.315 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:42.288+0000 7f5cfb9dd700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5cf40a33c0 0x7f5cf4125f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:39:42.315 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:42.288+0000 7f5cfb9dd700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5cf4145ed0 con 0x7f5cf4146750 2019-12-05T06:39:42.315 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:42.288+0000 7f5cfb9dd700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5cf4146210 con 0x7f5cf40a33c0 2019-12-05T06:39:42.316 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:42.288+0000 7f5cfb9dd700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5cf41464b0 con 0x7f5cf4118ee0 2019-12-05T06:39:42.316 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:42.288+0000 7f5cf9f7a700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f5cf4146750 0x7f5cf4147f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:60334/0 (socket says 172.21.15.42:60334) 2019-12-05T06:39:42.318 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:42.288+0000 7f5cf9f7a700 1 -- 172.21.15.42:0/546217330 learned_addr learned my addr 172.21.15.42:0/546217330 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:39:42.318 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:42.288+0000 7f5cf9779700 1 --2- 172.21.15.42:0/546217330 >> v2:172.21.15.42:3300/0 conn(0x7f5cf4118ee0 0x7f5cf40a2e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:42.318 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:42.288+0000 7f5cfa77b700 1 --2- 172.21.15.42:0/546217330 >> v2:172.21.15.40:3300/0 conn(0x7f5cf40a33c0 0x7f5cf4125f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:42.318 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:42.292+0000 7f5cea7fc700 1 -- 172.21.15.42:0/546217330 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2576626258 0 0) 0x7f5cf4145ed0 con 0x7f5cf4146750 2019-12-05T06:39:42.319 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:42.292+0000 7f5cea7fc700 1 -- 172.21.15.42:0/546217330 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5cdc0030b0 con 0x7f5cf4146750 2019-12-05T06:39:42.320 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:42.292+0000 7f5cf9779700 1 -- 172.21.15.42:0/546217330 >> v2:172.21.15.40:3300/0 conn(0x7f5cf40a33c0 msgr2=0x7f5cf4125f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:42.320 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:42.292+0000 7f5cf9779700 1 --2- 172.21.15.42:0/546217330 >> v2:172.21.15.40:3300/0 conn(0x7f5cf40a33c0 0x7f5cf4125f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:42.321 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:42.292+0000 7f5cf9779700 1 -- 172.21.15.42:0/546217330 >> v1:172.21.15.42:6789/0 conn(0x7f5cf4146750 legacy=0x7f5cf4147f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:42.321 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:42.292+0000 7f5cf9779700 1 -- 172.21.15.42:0/546217330 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cf40a37f0 con 0x7f5cf4118ee0 2019-12-05T06:39:42.321 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:42.292+0000 7f5cf9779700 1 --2- 172.21.15.42:0/546217330 >> v2:172.21.15.42:3300/0 conn(0x7f5cf4118ee0 0x7f5cf40a2e70 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f5cf0000e90 tx=0x7f5cf00108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:42.321 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:42.292+0000 7f5cf9779700 1 --2- 172.21.15.42:0/546217330 >> v2:172.21.15.42:3300/0 conn(0x7f5cf4118ee0 0x7f5cf40a2e70 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f5cf0000e90 tx=0x7f5cf00108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:42.321 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:42.292+0000 7f5cf9779700 1 --2- 172.21.15.42:0/546217330 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5cf4118ee0 0x7f5cf40a2e70 secure :-1 s=READY pgs=2936 cs=0 l=1 rx=0x7f5cf0000e90 tx=0x7f5cf00108f0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:39:42.321 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:42.292+0000 7f5cf9779700 1 --2- 172.21.15.42:0/546217330 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5cf4118ee0 0x7f5cf40a2e70 secure :-1 s=THROTTLE_DONE pgs=2936 cs=0 l=1 rx=0x7f5cf0000e90 tx=0x7f5cf00108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:42.322 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:42.292+0000 7f5cea7fc700 1 -- 172.21.15.42:0/546217330 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5cf000cb90 con 0x7f5cf4118ee0 2019-12-05T06:39:42.322 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:42.292+0000 7f5cf9779700 1 --2- 172.21.15.42:0/546217330 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5cf4118ee0 0x7f5cf40a2e70 secure :-1 s=THROTTLE_DONE pgs=2936 cs=0 l=1 rx=0x7f5cf0000e90 tx=0x7f5cf00108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:42.322 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:42.292+0000 7f5cfb9dd700 1 -- 172.21.15.42:0/546217330 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5cf41264d0 con 0x7f5cf4118ee0 2019-12-05T06:39:42.322 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:42.292+0000 7f5cea7fc700 1 -- 172.21.15.42:0/546217330 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cf000c7b0 con 0x7f5cf4118ee0 2019-12-05T06:39:42.322 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:42.292+0000 7f5cf9779700 1 --2- 172.21.15.42:0/546217330 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5cf4118ee0 0x7f5cf40a2e70 secure :-1 s=THROTTLE_DONE pgs=2936 cs=0 l=1 rx=0x7f5cf0000e90 tx=0x7f5cf00108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:42.322 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:42.292+0000 7f5cea7fc700 1 -- 172.21.15.42:0/546217330 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5cf00110b0 con 0x7f5cf4118ee0 2019-12-05T06:39:42.322 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:42.292+0000 7f5cfb9dd700 1 -- 172.21.15.42:0/546217330 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5cf4126a00 con 0x7f5cf4118ee0 2019-12-05T06:39:42.323 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:42.292+0000 7f5cf9779700 1 --2- 172.21.15.42:0/546217330 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5cf4118ee0 0x7f5cf40a2e70 secure :-1 s=THROTTLE_DONE pgs=2936 cs=0 l=1 rx=0x7f5cf0000e90 tx=0x7f5cf00108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:42.323 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:42.292+0000 7f5cea7fc700 1 -- 172.21.15.42:0/546217330 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f5cf00113d0 con 0x7f5cf4118ee0 2019-12-05T06:39:42.323 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:42.292+0000 7f5cf9779700 1 --2- 172.21.15.42:0/546217330 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5cf4118ee0 0x7f5cf40a2e70 secure :-1 s=THROTTLE_DONE pgs=2936 cs=0 l=1 rx=0x7f5cf0000e90 tx=0x7f5cf00108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:42.323 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:42.292+0000 7f5cea7fc700 1 --2- 172.21.15.42:0/546217330 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5cdc03b200 0x7f5cdc03d710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:39:42.323 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:42.292+0000 7f5cea7fc700 1 -- 172.21.15.42:0/546217330 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 11535+0+0 (secure 0 0 0) 0x7f5cf004b260 con 0x7f5cf4118ee0 2019-12-05T06:39:42.323 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:42.292+0000 7f5cf9f7a700 1 --2- 172.21.15.42:0/546217330 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5cdc03b200 0x7f5cdc03d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:42.324 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:42.292+0000 7f5cf9f7a700 1 --2- 172.21.15.42:0/546217330 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5cdc03b200 0x7f5cdc03d710 crc :-1 s=READY pgs=1231 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:39:42.324 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:42.300+0000 7f5cfb9dd700 1 -- 172.21.15.42:0/546217330 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5cf405d110 con 0x7f5cf4118ee0 2019-12-05T06:39:42.325 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:42.304+0000 7f5cf9779700 1 --2- 172.21.15.42:0/546217330 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5cf4118ee0 0x7f5cf40a2e70 secure :-1 s=THROTTLE_DONE pgs=2936 cs=0 l=1 rx=0x7f5cf0000e90 tx=0x7f5cf00108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:42.325 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:42.304+0000 7f5cea7fc700 1 -- 172.21.15.42:0/546217330 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f5cf0032260 con 0x7f5cf4118ee0 2019-12-05T06:39:42.493 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:42.488+0000 7f5cfb9dd700 1 -- 172.21.15.42:0/546217330 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5cf405b8c0 con 0x7f5cf4118ee0 2019-12-05T06:39:42.493 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:42.488+0000 7f5cf9779700 1 --2- 172.21.15.42:0/546217330 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5cf4118ee0 0x7f5cf40a2e70 secure :-1 s=THROTTLE_DONE pgs=2936 cs=0 l=1 rx=0x7f5cf0000e90 tx=0x7f5cf00108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:42.493 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:42.488+0000 7f5cea7fc700 1 -- 172.21.15.42:0/546217330 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 353 v353) v1 ==== 95+0+1776 (secure 0 0 0) 0x7f5cf00324c0 con 0x7f5cf4118ee0 2019-12-05T06:39:42.494 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 353 2019-12-05T06:39:42.494 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:39:42.495 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":353,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":353,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:39:37.486362+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":113,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8009},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8009":{"gid":8009,"name":"b","rank":0,"incarnation":349,"state":"up:active","state_seq":27,"addr":"172.21.15.40:6833/1777669201","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1777669201},{"type":"v1","addr":"172.21.15.40:6833","nonce":1777669201}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:39:42.497 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:42.492+0000 7f5cfb9dd700 1 -- 172.21.15.42:0/546217330 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5cdc03b200 msgr2=0x7f5cdc03d710 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:42.497 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:42.492+0000 7f5cfb9dd700 1 --2- 172.21.15.42:0/546217330 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5cdc03b200 0x7f5cdc03d710 crc :-1 s=READY pgs=1231 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:42.497 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:42.492+0000 7f5cfb9dd700 1 -- 172.21.15.42:0/546217330 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5cf4118ee0 msgr2=0x7f5cf40a2e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:42.498 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:42.492+0000 7f5cfb9dd700 1 --2- 172.21.15.42:0/546217330 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5cf4118ee0 0x7f5cf40a2e70 secure :-1 s=READY pgs=2936 cs=0 l=1 rx=0x7f5cf0000e90 tx=0x7f5cf00108f0).stop 2019-12-05T06:39:42.498 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:42.492+0000 7f5cfb9dd700 1 -- 172.21.15.42:0/546217330 shutdown_connections 2019-12-05T06:39:42.498 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:42.492+0000 7f5cfb9dd700 1 --2- 172.21.15.42:0/546217330 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5cdc03b200 0x7f5cdc03d710 unknown :-1 s=CLOSED pgs=1231 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:42.498 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:42.492+0000 7f5cfb9dd700 1 --2- 172.21.15.42:0/546217330 >> v2:172.21.15.40:3300/0 conn(0x7f5cf40a33c0 0x7f5cf4125f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:42.498 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:42.492+0000 7f5cfb9dd700 1 --2- 172.21.15.42:0/546217330 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5cf4118ee0 0x7f5cf40a2e70 unknown :-1 s=CLOSED pgs=2936 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:42.498 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:42.492+0000 7f5cfb9dd700 1 -- 172.21.15.42:0/546217330 shutdown_connections 2019-12-05T06:39:42.498 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:42.492+0000 7f5cfb9dd700 1 -- 172.21.15.42:0/546217330 wait complete. 2019-12-05T06:39:42.499 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:42.492+0000 7f5cfb9dd700 1 -- 172.21.15.42:0/546217330 >> 172.21.15.42:0/546217330 conn(0x7f5cf40a6c30 msgr2=0x7f5cf40a9770 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:39:42.511 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2019-12-05T06:39:44.512 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:39:44.608 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:44.596+0000 7fa088175700 1 Processor -- start 2019-12-05T06:39:44.608 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:44.596+0000 7fa088175700 1 -- start start 2019-12-05T06:39:44.608 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:44.596+0000 7fa088175700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa0800a7740 0x7fa08008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:39:44.608 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:44.596+0000 7fa088175700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa08009fae0 0x7fa08007eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:39:44.609 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:44.596+0000 7fa088175700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0800a0710 con 0x7fa08007e720 2019-12-05T06:39:44.609 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:44.596+0000 7fa088175700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0800a0b00 con 0x7fa0800a7740 2019-12-05T06:39:44.609 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:44.596+0000 7fa088175700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa08008e5b0 con 0x7fa08009fae0 2019-12-05T06:39:44.609 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:44.596+0000 7fa085f11700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fa08007e720 0x7fa08009f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:60376/0 (socket says 172.21.15.42:60376) 2019-12-05T06:39:44.609 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:44.596+0000 7fa085f11700 1 -- 172.21.15.42:0/830784594 learned_addr learned my addr 172.21.15.42:0/830784594 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:39:44.610 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:44.596+0000 7fa086712700 1 --2- 172.21.15.42:0/830784594 >> v2:172.21.15.40:3300/0 conn(0x7fa0800a7740 0x7fa08008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:44.610 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:44.596+0000 7fa086f13700 1 --2- 172.21.15.42:0/830784594 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa08009fae0 0x7fa08007eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:44.610 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:44.596+0000 7fa085710700 1 -- 172.21.15.42:0/830784594 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2712495316 0 0) 0x7fa0800a0710 con 0x7fa08007e720 2019-12-05T06:39:44.610 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:44.596+0000 7fa085710700 1 -- 172.21.15.42:0/830784594 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa06c0030b0 con 0x7fa08007e720 2019-12-05T06:39:44.610 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:44.600+0000 7fa086712700 1 -- 172.21.15.42:0/830784594 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa08009fae0 msgr2=0x7fa08007eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:39:44.611 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:44.600+0000 7fa086712700 1 --2- 172.21.15.42:0/830784594 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa08009fae0 0x7fa08007eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:39:44.611 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:44.600+0000 7fa085710700 1 -- 172.21.15.42:0/830784594 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3221979088 0 0) 0x7fa06c0030b0 con 0x7fa08007e720 2019-12-05T06:39:44.611 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:44.600+0000 7fa086712700 1 -- 172.21.15.42:0/830784594 >> v1:172.21.15.42:6789/0 conn(0x7fa08007e720 legacy=0x7fa08009f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:44.611 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:44.600+0000 7fa086712700 1 -- 172.21.15.42:0/830784594 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa08007f0a0 con 0x7fa0800a7740 2019-12-05T06:39:44.626 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:44.600+0000 7fa086f13700 1 --2- 172.21.15.42:0/830784594 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa08009fae0 0x7fa08007eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:39:44.627 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:44.600+0000 7fa086712700 1 --2- 172.21.15.42:0/830784594 >> v2:172.21.15.40:3300/0 conn(0x7fa0800a7740 0x7fa08008df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fa07c008790 tx=0x7fa0700019d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:44.627 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:44.600+0000 7fa086712700 1 --2- 172.21.15.42:0/830784594 >> v2:172.21.15.40:3300/0 conn(0x7fa0800a7740 0x7fa08008df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fa07c008790 tx=0x7fa0700019d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:44.627 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:44.600+0000 7fa086712700 1 --2- 172.21.15.42:0/830784594 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa0800a7740 0x7fa08008df80 secure :-1 s=READY pgs=2230 cs=0 l=1 rx=0x7fa07c008790 tx=0x7fa0700019d0).ready entity=mon.0 client_cookie=9e98be3b8181ddec server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:39:44.628 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:44.600+0000 7fa086712700 1 --2- 172.21.15.42:0/830784594 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa0800a7740 0x7fa08008df80 secure :-1 s=THROTTLE_DONE pgs=2230 cs=0 l=1 rx=0x7fa07c008790 tx=0x7fa0700019d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:44.628 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:44.600+0000 7fa085710700 1 -- 172.21.15.42:0/830784594 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa070001750 con 0x7fa0800a7740 2019-12-05T06:39:44.628 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:44.600+0000 7fa086712700 1 --2- 172.21.15.42:0/830784594 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa0800a7740 0x7fa08008df80 secure :-1 s=THROTTLE_DONE pgs=2230 cs=0 l=1 rx=0x7fa07c008790 tx=0x7fa0700019d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:44.628 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:44.600+0000 7fa086712700 1 --2- 172.21.15.42:0/830784594 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa0800a7740 0x7fa08008df80 secure :-1 s=THROTTLE_DONE pgs=2230 cs=0 l=1 rx=0x7fa07c008790 tx=0x7fa0700019d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:44.628 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:44.600+0000 7fa085710700 1 -- 172.21.15.42:0/830784594 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa070001560 con 0x7fa0800a7740 2019-12-05T06:39:44.629 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:44.600+0000 7fa085710700 1 -- 172.21.15.42:0/830784594 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa07c008da0 con 0x7fa0800a7740 2019-12-05T06:39:44.629 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:44.600+0000 7fa088175700 1 -- 172.21.15.42:0/830784594 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa0800a7740 msgr2=0x7fa08008df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:44.629 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:44.600+0000 7fa088175700 1 --2- 172.21.15.42:0/830784594 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa0800a7740 0x7fa08008df80 secure :-1 s=READY pgs=2230 cs=0 l=1 rx=0x7fa07c008790 tx=0x7fa0700019d0).stop 2019-12-05T06:39:44.629 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:44.600+0000 7fa088175700 1 -- 172.21.15.42:0/830784594 shutdown_connections 2019-12-05T06:39:44.629 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:44.600+0000 7fa088175700 1 --2- 172.21.15.42:0/830784594 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa08009fae0 0x7fa08007eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:39:44.630 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:44.600+0000 7fa088175700 1 --2- 172.21.15.42:0/830784594 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa0800a7740 0x7fa08008df80 unknown :-1 s=CLOSED pgs=2230 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:44.630 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:44.600+0000 7fa088175700 1 -- 172.21.15.42:0/830784594 shutdown_connections 2019-12-05T06:39:44.630 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:44.600+0000 7fa088175700 1 -- 172.21.15.42:0/830784594 wait complete. 2019-12-05T06:39:44.630 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:44.600+0000 7fa088175700 1 -- 172.21.15.42:0/830784594 >> 172.21.15.42:0/830784594 conn(0x7fa0800aabe0 msgr2=0x7fa0800a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:39:44.642 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:44.604+0000 7fa088175700 1 Processor -- start 2019-12-05T06:39:44.643 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:44.604+0000 7fa088175700 1 -- start start 2019-12-05T06:39:44.643 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:44.604+0000 7fa088175700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa0801181a0 0x7fa08008f340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:39:44.643 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:44.604+0000 7fa088175700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fa08012e150 0x7fa080141150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:39:44.643 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:44.604+0000 7fa088175700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa08008f890 con 0x7fa08012e580 2019-12-05T06:39:44.643 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:44.604+0000 7fa088175700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa08012f570 con 0x7fa08012e150 2019-12-05T06:39:44.643 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:44.604+0000 7fa088175700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa08012f810 con 0x7fa0801181a0 2019-12-05T06:39:44.644 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:44.604+0000 7fa085f11700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fa08012e150 0x7fa080141150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:44.644 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:44.604+0000 7fa085f11700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fa08012e150 0x7fa080141150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:60344/0 (socket says 172.21.15.42:60344) 2019-12-05T06:39:44.644 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:44.604+0000 7fa085f11700 1 -- 172.21.15.42:0/98812727 learned_addr learned my addr 172.21.15.42:0/98812727 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:39:44.644 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:44.608+0000 7fa086712700 1 --2- 172.21.15.42:0/98812727 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa0801181a0 0x7fa08008f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:44.645 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:44.608+0000 7fa085f11700 1 -- 172.21.15.42:0/98812727 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa0801181a0 msgr2=0x7fa08008f340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:44.645 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:44.608+0000 7fa085f11700 1 --2- 172.21.15.42:0/98812727 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa0801181a0 0x7fa08008f340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:44.646 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:44.608+0000 7fa085f11700 1 -- 172.21.15.42:0/98812727 >> v1:172.21.15.40:6789/0 conn(0x7fa08012e580 legacy=0x7fa0801416a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:44.646 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:44.608+0000 7fa085f11700 1 -- 172.21.15.42:0/98812727 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa080141ea0 con 0x7fa08012e150 2019-12-05T06:39:44.647 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:44.608+0000 7fa085f11700 1 --2- 172.21.15.42:0/98812727 >> v2:172.21.15.42:3300/0 conn(0x7fa08012e150 0x7fa080141150 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fa08008ea20 tx=0x7fa070032aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:44.647 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:44.608+0000 7fa085f11700 1 --2- 172.21.15.42:0/98812727 >> v2:172.21.15.42:3300/0 conn(0x7fa08012e150 0x7fa080141150 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fa08008ea20 tx=0x7fa070032aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:44.647 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:44.608+0000 7fa085f11700 1 --2- 172.21.15.42:0/98812727 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa08012e150 0x7fa080141150 secure :-1 s=READY pgs=2938 cs=0 l=1 rx=0x7fa08008ea20 tx=0x7fa070032aa0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:39:44.647 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:44.608+0000 7fa085f11700 1 --2- 172.21.15.42:0/98812727 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa08012e150 0x7fa080141150 secure :-1 s=THROTTLE_DONE pgs=2938 cs=0 l=1 rx=0x7fa08008ea20 tx=0x7fa070032aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:44.647 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:44.608+0000 7fa085f11700 1 --2- 172.21.15.42:0/98812727 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa08012e150 0x7fa080141150 secure :-1 s=THROTTLE_DONE pgs=2938 cs=0 l=1 rx=0x7fa08008ea20 tx=0x7fa070032aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:44.647 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:44.608+0000 7fa085f11700 1 --2- 172.21.15.42:0/98812727 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa08012e150 0x7fa080141150 secure :-1 s=THROTTLE_DONE pgs=2938 cs=0 l=1 rx=0x7fa08008ea20 tx=0x7fa070032aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:44.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:44.608+0000 7fa076ffd700 1 -- 172.21.15.42:0/98812727 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa07002fbb0 con 0x7fa08012e150 2019-12-05T06:39:44.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:44.608+0000 7fa076ffd700 1 -- 172.21.15.42:0/98812727 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa07002f7d0 con 0x7fa08012e150 2019-12-05T06:39:44.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:44.608+0000 7fa088175700 1 -- 172.21.15.42:0/98812727 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa080143d40 con 0x7fa08012e150 2019-12-05T06:39:44.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:44.608+0000 7fa076ffd700 1 -- 172.21.15.42:0/98812727 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa070004620 con 0x7fa08012e150 2019-12-05T06:39:44.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:44.608+0000 7fa088175700 1 -- 172.21.15.42:0/98812727 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0801446d0 con 0x7fa08012e150 2019-12-05T06:39:44.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:44.608+0000 7fa085f11700 1 --2- 172.21.15.42:0/98812727 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa08012e150 0x7fa080141150 secure :-1 s=THROTTLE_DONE pgs=2938 cs=0 l=1 rx=0x7fa08008ea20 tx=0x7fa070032aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:44.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:44.608+0000 7fa076ffd700 1 -- 172.21.15.42:0/98812727 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fa070004820 con 0x7fa08012e150 2019-12-05T06:39:44.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:44.608+0000 7fa085f11700 1 --2- 172.21.15.42:0/98812727 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa08012e150 0x7fa080141150 secure :-1 s=THROTTLE_DONE pgs=2938 cs=0 l=1 rx=0x7fa08008ea20 tx=0x7fa070032aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:44.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:44.608+0000 7fa076ffd700 1 --2- 172.21.15.42:0/98812727 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa06803ae30 0x7fa06803d340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:39:44.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:44.608+0000 7fa076ffd700 1 -- 172.21.15.42:0/98812727 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 11535+0+0 (secure 0 0 0) 0x7fa07006cfd0 con 0x7fa08012e150 2019-12-05T06:39:44.650 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:44.608+0000 7fa086712700 1 --2- 172.21.15.42:0/98812727 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa06803ae30 0x7fa06803d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:44.650 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:44.608+0000 7fa086712700 1 --2- 172.21.15.42:0/98812727 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa06803ae30 0x7fa06803d340 crc :-1 s=READY pgs=1233 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:39:44.650 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:44.612+0000 7fa088175700 1 -- 172.21.15.42:0/98812727 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa08005d110 con 0x7fa08012e150 2019-12-05T06:39:44.650 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:44.616+0000 7fa085f11700 1 --2- 172.21.15.42:0/98812727 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa08012e150 0x7fa080141150 secure :-1 s=THROTTLE_DONE pgs=2938 cs=0 l=1 rx=0x7fa08008ea20 tx=0x7fa070032aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:44.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:44.616+0000 7fa076ffd700 1 -- 172.21.15.42:0/98812727 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fa070053ff0 con 0x7fa08012e150 2019-12-05T06:39:44.794 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:44.788+0000 7fa088175700 1 -- 172.21.15.42:0/98812727 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa08005b8c0 con 0x7fa08012e150 2019-12-05T06:39:44.795 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:44.788+0000 7fa085f11700 1 --2- 172.21.15.42:0/98812727 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa08012e150 0x7fa080141150 secure :-1 s=THROTTLE_DONE pgs=2938 cs=0 l=1 rx=0x7fa08008ea20 tx=0x7fa070032aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:44.795 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:44.788+0000 7fa076ffd700 1 -- 172.21.15.42:0/98812727 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 354 v354) v1 ==== 95+0+2111 (secure 0 0 0) 0x7fa070054250 con 0x7fa08012e150 2019-12-05T06:39:44.797 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 354 2019-12-05T06:39:44.797 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:39:44.798 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":354,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":11868,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/406829134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":406829134},{"type":"v1","addr":"172.21.15.42:6835","nonce":406829134}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":354}],"filesystems":[{"mdsmap":{"epoch":353,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:39:37.486362+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":113,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8009},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8009":{"gid":8009,"name":"b","rank":0,"incarnation":349,"state":"up:active","state_seq":27,"addr":"172.21.15.40:6833/1777669201","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1777669201},{"type":"v1","addr":"172.21.15.40:6833","nonce":1777669201}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:39:44.799 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:44.792+0000 7fa088175700 1 -- 172.21.15.42:0/98812727 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa06803ae30 msgr2=0x7fa06803d340 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:44.800 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:44.792+0000 7fa088175700 1 --2- 172.21.15.42:0/98812727 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa06803ae30 0x7fa06803d340 crc :-1 s=READY pgs=1233 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:44.800 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:44.792+0000 7fa088175700 1 -- 172.21.15.42:0/98812727 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa08012e150 msgr2=0x7fa080141150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:44.800 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:44.792+0000 7fa088175700 1 --2- 172.21.15.42:0/98812727 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa08012e150 0x7fa080141150 secure :-1 s=READY pgs=2938 cs=0 l=1 rx=0x7fa08008ea20 tx=0x7fa070032aa0).stop 2019-12-05T06:39:44.801 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:44.792+0000 7fa088175700 1 -- 172.21.15.42:0/98812727 shutdown_connections 2019-12-05T06:39:44.801 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:44.792+0000 7fa088175700 1 --2- 172.21.15.42:0/98812727 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa06803ae30 0x7fa06803d340 unknown :-1 s=CLOSED pgs=1233 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:44.801 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:44.792+0000 7fa088175700 1 --2- 172.21.15.42:0/98812727 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa0801181a0 0x7fa08008f340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:44.801 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:44.792+0000 7fa088175700 1 --2- 172.21.15.42:0/98812727 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa08012e150 0x7fa080141150 unknown :-1 s=CLOSED pgs=2938 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:44.801 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:44.792+0000 7fa088175700 1 -- 172.21.15.42:0/98812727 shutdown_connections 2019-12-05T06:39:44.802 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:44.792+0000 7fa088175700 1 -- 172.21.15.42:0/98812727 wait complete. 2019-12-05T06:39:44.802 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:44.792+0000 7fa088175700 1 -- 172.21.15.42:0/98812727 >> 172.21.15.42:0/98812727 conn(0x7fa08011c470 msgr2=0x7fa08009f380 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:39:44.815 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a reported in up:standby state 2019-12-05T06:39:44.815 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 7 secs before thrashing 2019-12-05T06:39:44.842 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:39:44.853 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:39:51.816 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:39:51.926 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:51.912+0000 7fac1d84b700 1 Processor -- start 2019-12-05T06:39:51.928 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:51.912+0000 7fac1d84b700 1 -- start start 2019-12-05T06:39:51.928 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:51.916+0000 7fac1d84b700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fac180a7730 0x7fac1808df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:39:51.929 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:51.916+0000 7fac1d84b700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac180a0700 con 0x7fac1809fad0 2019-12-05T06:39:51.929 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:51.916+0000 7fac1d84b700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac180a9b90 con 0x7fac1807e710 2019-12-05T06:39:51.929 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:51.916+0000 7fac1d84b700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac180a0af0 con 0x7fac180a7730 2019-12-05T06:39:51.929 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:51.916+0000 7fac177fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fac180a7730 0x7fac1808df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:51.929 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:51.916+0000 7fac177fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fac180a7730 0x7fac1808df70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:60416/0 (socket says 172.21.15.42:60416) 2019-12-05T06:39:51.929 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:51.916+0000 7fac177fe700 1 -- 172.21.15.42:0/2287760410 learned_addr learned my addr 172.21.15.42:0/2287760410 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:39:51.930 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:51.916+0000 7fac177fe700 1 -- 172.21.15.42:0/2287760410 >> v1:172.21.15.42:6789/0 conn(0x7fac1807e710 legacy=0x7fac1809f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:51.930 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:51.916+0000 7fac177fe700 1 -- 172.21.15.42:0/2287760410 >> v1:172.21.15.40:6789/0 conn(0x7fac1809fad0 legacy=0x7fac1814efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:51.930 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:51.916+0000 7fac177fe700 1 -- 172.21.15.42:0/2287760410 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac1807eb40 con 0x7fac180a7730 2019-12-05T06:39:51.930 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:51.916+0000 7fac177fe700 1 --2- 172.21.15.42:0/2287760410 >> v2:172.21.15.42:3300/0 conn(0x7fac180a7730 0x7fac1808df70 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fac08008790 tx=0x7fac0800fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:51.930 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:51.916+0000 7fac177fe700 1 --2- 172.21.15.42:0/2287760410 >> v2:172.21.15.42:3300/0 conn(0x7fac180a7730 0x7fac1808df70 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fac08008790 tx=0x7fac0800fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:51.930 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:51.916+0000 7fac177fe700 1 --2- 172.21.15.42:0/2287760410 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fac180a7730 0x7fac1808df70 secure :-1 s=READY pgs=2941 cs=0 l=1 rx=0x7fac08008790 tx=0x7fac0800fcb0).ready entity=mon.1 client_cookie=b3d62d811534eff2 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:39:51.931 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:51.916+0000 7fac177fe700 1 --2- 172.21.15.42:0/2287760410 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fac180a7730 0x7fac1808df70 secure :-1 s=THROTTLE_DONE pgs=2941 cs=0 l=1 rx=0x7fac08008790 tx=0x7fac0800fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:51.931 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:51.916+0000 7fac177fe700 1 --2- 172.21.15.42:0/2287760410 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fac180a7730 0x7fac1808df70 secure :-1 s=THROTTLE_DONE pgs=2941 cs=0 l=1 rx=0x7fac08008790 tx=0x7fac0800fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:51.931 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:51.916+0000 7fac167fc700 1 -- 172.21.15.42:0/2287760410 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fac08011520 con 0x7fac180a7730 2019-12-05T06:39:51.931 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:51.916+0000 7fac177fe700 1 --2- 172.21.15.42:0/2287760410 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fac180a7730 0x7fac1808df70 secure :-1 s=THROTTLE_DONE pgs=2941 cs=0 l=1 rx=0x7fac08008790 tx=0x7fac0800fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:51.932 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:51.916+0000 7fac167fc700 1 -- 172.21.15.42:0/2287760410 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac08008ca0 con 0x7fac180a7730 2019-12-05T06:39:51.932 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:51.916+0000 7fac167fc700 1 -- 172.21.15.42:0/2287760410 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fac08011b00 con 0x7fac180a7730 2019-12-05T06:39:51.932 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:51.916+0000 7fac1d84b700 1 -- 172.21.15.42:0/2287760410 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fac180a7730 msgr2=0x7fac1808df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:51.932 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:51.916+0000 7fac1d84b700 1 --2- 172.21.15.42:0/2287760410 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fac180a7730 0x7fac1808df70 secure :-1 s=READY pgs=2941 cs=0 l=1 rx=0x7fac08008790 tx=0x7fac0800fcb0).stop 2019-12-05T06:39:51.933 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:51.920+0000 7fac1d84b700 1 -- 172.21.15.42:0/2287760410 shutdown_connections 2019-12-05T06:39:51.933 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:51.920+0000 7fac1d84b700 1 --2- 172.21.15.42:0/2287760410 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fac180a7730 0x7fac1808df70 unknown :-1 s=CLOSED pgs=2941 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:51.933 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:51.920+0000 7fac1d84b700 1 -- 172.21.15.42:0/2287760410 shutdown_connections 2019-12-05T06:39:51.933 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:51.920+0000 7fac1d84b700 1 -- 172.21.15.42:0/2287760410 wait complete. 2019-12-05T06:39:51.934 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:51.920+0000 7fac1d84b700 1 -- 172.21.15.42:0/2287760410 >> 172.21.15.42:0/2287760410 conn(0x7fac180aabd0 msgr2=0x7fac180a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:39:51.939 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:51.920+0000 7fac1d84b700 1 Processor -- start 2019-12-05T06:39:51.939 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:51.920+0000 7fac1d84b700 1 -- start start 2019-12-05T06:39:51.939 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:51.924+0000 7fac1d84b700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fac1808eea0 0x7fac1812f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:39:51.940 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:51.924+0000 7fac1d84b700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac18144150 con 0x7fac18118ed0 2019-12-05T06:39:51.940 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:51.924+0000 7fac1d84b700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac1808f2d0 con 0x7fac18143d20 2019-12-05T06:39:51.940 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:51.924+0000 7fac1d84b700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac1808f7c0 con 0x7fac1808eea0 2019-12-05T06:39:51.940 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:51.924+0000 7fac16ffd700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fac18143d20 0x7fac180a5970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:60460/0 (socket says 172.21.15.42:60460) 2019-12-05T06:39:51.941 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:51.924+0000 7fac16ffd700 1 -- 172.21.15.42:0/3232227116 learned_addr learned my addr 172.21.15.42:0/3232227116 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:39:51.941 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:51.924+0000 7fac17fff700 1 --2- 172.21.15.42:0/3232227116 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fac1808eea0 0x7fac1812f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:51.941 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:51.924+0000 7fabfffff700 1 -- 172.21.15.42:0/3232227116 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 56659813 0 0) 0x7fac1808f2d0 con 0x7fac18143d20 2019-12-05T06:39:51.941 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:51.924+0000 7fabfffff700 1 -- 172.21.15.42:0/3232227116 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabf80030b0 con 0x7fac18143d20 2019-12-05T06:39:51.941 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:51.924+0000 7fabfffff700 1 -- 172.21.15.42:0/3232227116 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1671458329 0 0) 0x7fac18144150 con 0x7fac18118ed0 2019-12-05T06:39:51.941 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:51.924+0000 7fabfffff700 1 -- 172.21.15.42:0/3232227116 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac1808f2d0 con 0x7fac18118ed0 2019-12-05T06:39:51.943 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:51.924+0000 7fabfffff700 1 -- 172.21.15.42:0/3232227116 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 3466113659 0 0) 0x7fabf80030b0 con 0x7fac18143d20 2019-12-05T06:39:51.944 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:51.924+0000 7fabfffff700 1 -- 172.21.15.42:0/3232227116 --> v1:172.21.15.42:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fac18144150 con 0x7fac18143d20 2019-12-05T06:39:51.944 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:51.924+0000 7fabfffff700 1 -- 172.21.15.42:0/3232227116 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7fac0c001a10 con 0x7fac18143d20 2019-12-05T06:39:51.945 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:51.924+0000 7fac17fff700 1 -- 172.21.15.42:0/3232227116 >> v1:172.21.15.42:6789/0 conn(0x7fac18143d20 legacy=0x7fac180a5970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:51.945 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:51.924+0000 7fac17fff700 1 -- 172.21.15.42:0/3232227116 >> v1:172.21.15.40:6789/0 conn(0x7fac18118ed0 legacy=0x7fac180a5170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:51.945 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:51.924+0000 7fac17fff700 1 -- 172.21.15.42:0/3232227116 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac18148920 con 0x7fac1808eea0 2019-12-05T06:39:51.945 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:51.924+0000 7fac17fff700 1 --2- 172.21.15.42:0/3232227116 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fac1808eea0 0x7fac1812f090 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fac180a9090 tx=0x7fac0800fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:51.946 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:51.924+0000 7fac17fff700 1 --2- 172.21.15.42:0/3232227116 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fac1808eea0 0x7fac1812f090 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fac180a9090 tx=0x7fac0800fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:51.946 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:51.924+0000 7fac17fff700 1 --2- 172.21.15.42:0/3232227116 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fac1808eea0 0x7fac1812f090 secure :-1 s=READY pgs=652 cs=0 l=1 rx=0x7fac180a9090 tx=0x7fac0800fcb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:39:51.947 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:51.924+0000 7fac17fff700 1 --2- 172.21.15.42:0/3232227116 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fac1808eea0 0x7fac1812f090 secure :-1 s=THROTTLE_DONE pgs=652 cs=0 l=1 rx=0x7fac180a9090 tx=0x7fac0800fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:51.947 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:51.924+0000 7fabfffff700 1 -- 172.21.15.42:0/3232227116 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fac10006bb0 con 0x7fac1808eea0 2019-12-05T06:39:51.947 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:51.924+0000 7fac17fff700 1 --2- 172.21.15.42:0/3232227116 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fac1808eea0 0x7fac1812f090 secure :-1 s=THROTTLE_DONE pgs=652 cs=0 l=1 rx=0x7fac180a9090 tx=0x7fac0800fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:51.948 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:51.924+0000 7fac17fff700 1 --2- 172.21.15.42:0/3232227116 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fac1808eea0 0x7fac1812f090 secure :-1 s=THROTTLE_DONE pgs=652 cs=0 l=1 rx=0x7fac180a9090 tx=0x7fac0800fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:51.948 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:51.924+0000 7fabfffff700 1 -- 172.21.15.42:0/3232227116 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac100067d0 con 0x7fac1808eea0 2019-12-05T06:39:51.949 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:51.924+0000 7fac1d84b700 1 -- 172.21.15.42:0/3232227116 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac18148590 con 0x7fac1808eea0 2019-12-05T06:39:51.950 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:51.924+0000 7fabfffff700 1 -- 172.21.15.42:0/3232227116 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fac10002410 con 0x7fac1808eea0 2019-12-05T06:39:51.950 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:51.924+0000 7fac1d84b700 1 -- 172.21.15.42:0/3232227116 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac18142be0 con 0x7fac1808eea0 2019-12-05T06:39:51.950 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:51.928+0000 7fac17fff700 1 --2- 172.21.15.42:0/3232227116 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fac1808eea0 0x7fac1812f090 secure :-1 s=THROTTLE_DONE pgs=652 cs=0 l=1 rx=0x7fac180a9090 tx=0x7fac0800fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:51.950 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:51.928+0000 7fabfffff700 1 -- 172.21.15.42:0/3232227116 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fac10002720 con 0x7fac1808eea0 2019-12-05T06:39:51.951 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:51.928+0000 7fac17fff700 1 --2- 172.21.15.42:0/3232227116 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fac1808eea0 0x7fac1812f090 secure :-1 s=THROTTLE_DONE pgs=652 cs=0 l=1 rx=0x7fac180a9090 tx=0x7fac0800fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:51.951 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:51.928+0000 7fabfffff700 1 --2- 172.21.15.42:0/3232227116 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fabf803b860 0x7fabf803dd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:39:51.951 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:51.928+0000 7fabfffff700 1 -- 172.21.15.42:0/3232227116 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 11535+0+0 (secure 0 0 0) 0x7fac1004adf0 con 0x7fac1808eea0 2019-12-05T06:39:51.952 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:51.928+0000 7fac177fe700 1 --2- 172.21.15.42:0/3232227116 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fabf803b860 0x7fabf803dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:51.952 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:51.928+0000 7fac177fe700 1 --2- 172.21.15.42:0/3232227116 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fabf803b860 0x7fabf803dd70 crc :-1 s=READY pgs=1234 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:39:51.952 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:51.928+0000 7fac1d84b700 1 -- 172.21.15.42:0/3232227116 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac1805d100 con 0x7fac1808eea0 2019-12-05T06:39:51.953 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:51.932+0000 7fac17fff700 1 --2- 172.21.15.42:0/3232227116 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fac1808eea0 0x7fac1812f090 secure :-1 s=THROTTLE_DONE pgs=652 cs=0 l=1 rx=0x7fac180a9090 tx=0x7fac0800fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:51.953 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:51.932+0000 7fabfffff700 1 -- 172.21.15.42:0/3232227116 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fac18148920 con 0x7fac1808eea0 2019-12-05T06:39:52.107 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:52.100+0000 7fac1d84b700 1 -- 172.21.15.42:0/3232227116 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fac1805b8b0 con 0x7fac1808eea0 2019-12-05T06:39:52.108 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:52.104+0000 7fac17fff700 1 --2- 172.21.15.42:0/3232227116 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fac1808eea0 0x7fac1812f090 secure :-1 s=THROTTLE_DONE pgs=652 cs=0 l=1 rx=0x7fac180a9090 tx=0x7fac0800fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:52.109 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:52.104+0000 7fabfffff700 1 -- 172.21.15.42:0/3232227116 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 354 v354) v1 ==== 95+0+2111 (secure 0 0 0) 0x7fac10031e70 con 0x7fac1808eea0 2019-12-05T06:39:52.110 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 354 2019-12-05T06:39:52.111 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:39:52.111 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":354,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":11868,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/406829134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":406829134},{"type":"v1","addr":"172.21.15.42:6835","nonce":406829134}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":354}],"filesystems":[{"mdsmap":{"epoch":353,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:39:37.486362+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":113,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8009},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8009":{"gid":8009,"name":"b","rank":0,"incarnation":349,"state":"up:active","state_seq":27,"addr":"172.21.15.40:6833/1777669201","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1777669201},{"type":"v1","addr":"172.21.15.40:6833","nonce":1777669201}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:39:52.114 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:52.108+0000 7fac1d84b700 1 -- 172.21.15.42:0/3232227116 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fabf803b860 msgr2=0x7fabf803dd70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:52.114 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:52.108+0000 7fac1d84b700 1 --2- 172.21.15.42:0/3232227116 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fabf803b860 0x7fabf803dd70 crc :-1 s=READY pgs=1234 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:52.114 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:52.108+0000 7fac1d84b700 1 -- 172.21.15.42:0/3232227116 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fac1808eea0 msgr2=0x7fac1812f090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:52.114 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:52.108+0000 7fac1d84b700 1 --2- 172.21.15.42:0/3232227116 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fac1808eea0 0x7fac1812f090 secure :-1 s=READY pgs=652 cs=0 l=1 rx=0x7fac180a9090 tx=0x7fac0800fcb0).stop 2019-12-05T06:39:52.114 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:52.108+0000 7fac1d84b700 1 -- 172.21.15.42:0/3232227116 shutdown_connections 2019-12-05T06:39:52.115 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:52.108+0000 7fac1d84b700 1 --2- 172.21.15.42:0/3232227116 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fabf803b860 0x7fabf803dd70 unknown :-1 s=CLOSED pgs=1234 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:52.115 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:52.108+0000 7fac1d84b700 1 --2- 172.21.15.42:0/3232227116 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fac1808eea0 0x7fac1812f090 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:52.116 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:52.108+0000 7fac1d84b700 1 -- 172.21.15.42:0/3232227116 shutdown_connections 2019-12-05T06:39:52.116 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:52.108+0000 7fac1d84b700 1 -- 172.21.15.42:0/3232227116 wait complete. 2019-12-05T06:39:52.116 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:52.108+0000 7fac1d84b700 1 -- 172.21.15.42:0/3232227116 >> 172.21.15.42:0/3232227116 conn(0x7fac1808fd90 msgr2=0x7fac180596f0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:39:52.124 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.b (rank=0) 2019-12-05T06:39:52.125 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2019-12-05T06:39:52.125 INFO:teuthology.orchestra.run:waiting for 300 2019-12-05T06:39:52.137 INFO:tasks.ceph.mds.b:Stopped 2019-12-05T06:39:52.137 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:39:52.442 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:52.224+0000 7f16df9a1700 1 Processor -- start 2019-12-05T06:39:52.442 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:52.224+0000 7f16df9a1700 1 -- start start 2019-12-05T06:39:52.442 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:52.224+0000 7f16df9a1700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f16d80a7740 0x7f16d808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:39:52.442 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:52.224+0000 7f16df9a1700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16d80a0710 con 0x7f16d809fae0 2019-12-05T06:39:52.442 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:52.224+0000 7f16df9a1700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16d80a9ba0 con 0x7f16d807e720 2019-12-05T06:39:52.443 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:52.224+0000 7f16df9a1700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16d80a0b00 con 0x7f16d80a7740 2019-12-05T06:39:52.443 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:52.224+0000 7f16dd73d700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f16d807e720 0x7f16d809f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:60470/0 (socket says 172.21.15.42:60470) 2019-12-05T06:39:52.443 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:52.224+0000 7f16dd73d700 1 -- 172.21.15.42:0/601015169 learned_addr learned my addr 172.21.15.42:0/601015169 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:39:52.443 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:52.224+0000 7f16ddf3e700 1 --2- 172.21.15.42:0/601015169 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f16d80a7740 0x7f16d808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:52.443 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:52.224+0000 7f16dcf3c700 1 -- 172.21.15.42:0/601015169 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2027602155 0 0) 0x7f16d80a9ba0 con 0x7f16d807e720 2019-12-05T06:39:52.443 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:52.224+0000 7f16dcf3c700 1 -- 172.21.15.42:0/601015169 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16c40030b0 con 0x7f16d807e720 2019-12-05T06:39:52.444 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:52.224+0000 7f16dcf3c700 1 -- 172.21.15.42:0/601015169 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 302037258 0 0) 0x7f16c40030b0 con 0x7f16d807e720 2019-12-05T06:39:52.444 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:52.224+0000 7f16dcf3c700 1 -- 172.21.15.42:0/601015169 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f16d80a7740 msgr2=0x7f16d808df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:39:52.444 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:52.224+0000 7f16dcf3c700 1 --2- 172.21.15.42:0/601015169 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f16d80a7740 0x7f16d808df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:39:52.444 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:52.224+0000 7f16dcf3c700 1 -- 172.21.15.42:0/601015169 >> v1:172.21.15.40:6789/0 conn(0x7f16d809fae0 legacy=0x7f16d814efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:52.444 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:52.224+0000 7f16dcf3c700 1 -- 172.21.15.42:0/601015169 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16d807eb50 con 0x7f16d807e720 2019-12-05T06:39:52.444 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:52.224+0000 7f16dcf3c700 1 -- 172.21.15.42:0/601015169 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7f16d4001750 con 0x7f16d807e720 2019-12-05T06:39:52.444 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:52.224+0000 7f16dcf3c700 1 -- 172.21.15.42:0/601015169 >> v1:172.21.15.42:6789/0 conn(0x7f16d807e720 legacy=0x7f16d809f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:52.445 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:52.224+0000 7f16dcf3c700 1 --2- 172.21.15.42:0/601015169 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f16c4003710 0x7f16c4003b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:39:52.445 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:52.224+0000 7f16dcf3c700 1 --2- 172.21.15.42:0/601015169 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f16c4004420 0x7f16c4004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:39:52.445 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:52.224+0000 7f16dcf3c700 1 --2- 172.21.15.42:0/601015169 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f16c4004da0 0x7f16c400b670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:39:52.446 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:52.224+0000 7f16dcf3c700 1 -- 172.21.15.42:0/601015169 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16d80a0b00 con 0x7f16c4004da0 2019-12-05T06:39:52.446 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:52.224+0000 7f16dcf3c700 1 -- 172.21.15.42:0/601015169 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16c40051d0 con 0x7f16c4003710 2019-12-05T06:39:52.447 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:52.224+0000 7f16dcf3c700 1 -- 172.21.15.42:0/601015169 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16c404fbd0 con 0x7f16c4004420 2019-12-05T06:39:52.447 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:52.224+0000 7f16ddf3e700 1 --2- 172.21.15.42:0/601015169 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f16c4003710 0x7f16c4003b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:52.447 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:52.224+0000 7f16dd73d700 1 --2- 172.21.15.42:0/601015169 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f16c4004420 0x7f16c4004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:52.447 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:52.224+0000 7f16de73f700 1 --2- 172.21.15.42:0/601015169 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f16c4004da0 0x7f16c400b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:52.447 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:52.228+0000 7f16ddf3e700 1 -- 172.21.15.42:0/601015169 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f16c4004420 msgr2=0x7f16c4004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:39:52.447 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:52.228+0000 7f16ddf3e700 1 --2- 172.21.15.42:0/601015169 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f16c4004420 0x7f16c4004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:39:52.448 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:52.228+0000 7f16ddf3e700 1 -- 172.21.15.42:0/601015169 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f16c4004da0 msgr2=0x7f16c400b670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:39:52.448 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:52.228+0000 7f16ddf3e700 1 --2- 172.21.15.42:0/601015169 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f16c4004da0 0x7f16c400b670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:39:52.448 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:52.228+0000 7f16ddf3e700 1 -- 172.21.15.42:0/601015169 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16c404fed0 con 0x7f16c4003710 2019-12-05T06:39:52.448 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:52.228+0000 7f16ddf3e700 1 --2- 172.21.15.42:0/601015169 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f16c4003710 0x7f16c4003b80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f16cc002f30 tx=0x7f16cc00ffe0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:52.448 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:52.228+0000 7f16ddf3e700 1 -- 172.21.15.42:0/601015169 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f16c4003710 msgr2=0x7f16c4003b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2019-12-05T06:39:52.448 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:52.228+0000 7f16ddf3e700 1 -- 172.21.15.42:0/601015169 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f16c4003710 msgr2=0x7f16c4003b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2019-12-05T06:39:52.449 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:52.228+0000 7f16ddf3e700 1 --2- 172.21.15.42:0/601015169 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f16c4003710 0x7f16c4003b80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f16cc002f30 tx=0x7f16cc00ffe0).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-12-05T06:39:52.449 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:52.228+0000 7f16ddf3e700 1 --2- 172.21.15.42:0/601015169 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f16c4003710 0x7f16c4003b80 secure :-1 s=START_CONNECT pgs=0 cs=0 l=0 rx=0x7f16cc002f30 tx=0x7f16cc00ffe0)._fault waiting 0.200000 2019-12-05T06:39:52.449 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:52.432+0000 7f16ddf3e700 1 --2- 172.21.15.42:0/601015169 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f16c4003710 0x7f16c4003b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:52.449 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:52.432+0000 7f16ddf3e700 0 --2- 172.21.15.42:0/601015169 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f16c4003710 0x7f16c4003b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2019-12-05T06:39:52.449 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:52.432+0000 7f16ddf3e700 1 --2- 172.21.15.42:0/601015169 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f16c4003710 0x7f16c4003b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:39:52.449 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:52.432+0000 7f16de73f700 1 -- 172.21.15.42:0/601015169 reap_dead start 2019-12-05T06:39:52.450 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:52.432+0000 7f16dcf3c700 1 -- 172.21.15.42:0/601015169 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f16c4003710 msgr2=0x7f16c4003b80 unknown :-1 s=STATE_CLOSED l=0).mark_down 2019-12-05T06:39:52.450 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:52.432+0000 7f16dcf3c700 1 --2- 172.21.15.42:0/601015169 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f16c4003710 0x7f16c4003b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:39:52.450 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:52.432+0000 7f16dcf3c700 1 --2- 172.21.15.42:0/601015169 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f16c4004420 0x7f16c4004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:39:52.450 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:52.432+0000 7f16dcf3c700 1 --2- 172.21.15.42:0/601015169 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f16c4009660 0x7f16c40501a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:39:52.450 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:52.432+0000 7f16dcf3c700 1 --2- 172.21.15.42:0/601015169 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f16c40506f0 0x7f16c4052b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:39:52.450 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:52.432+0000 7f16dcf3c700 1 -- 172.21.15.42:0/601015169 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16c4053210 con 0x7f16c4009660 2019-12-05T06:39:52.450 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:52.432+0000 7f16dcf3c700 1 -- 172.21.15.42:0/601015169 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16c4053400 con 0x7f16c40506f0 2019-12-05T06:39:52.451 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:52.432+0000 7f16dcf3c700 1 -- 172.21.15.42:0/601015169 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16c4053640 con 0x7f16c4004420 2019-12-05T06:39:52.451 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:52.432+0000 7f16de73f700 1 --2- 172.21.15.42:0/601015169 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f16c40506f0 0x7f16c4052b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:52.451 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:52.432+0000 7f16ddf3e700 1 --2- 172.21.15.42:0/601015169 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f16c4004420 0x7f16c4004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:52.451 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:52.432+0000 7f16dd73d700 1 --2- 172.21.15.42:0/601015169 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f16c4009660 0x7f16c40501a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:52.452 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:52.432+0000 7f16de73f700 1 -- 172.21.15.42:0/601015169 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f16c4004420 msgr2=0x7f16c4004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:39:52.452 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:52.432+0000 7f16de73f700 1 --2- 172.21.15.42:0/601015169 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f16c4004420 0x7f16c4004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:39:52.452 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:52.432+0000 7f16de73f700 1 -- 172.21.15.42:0/601015169 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f16c4009660 msgr2=0x7f16c40501a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:39:52.452 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:52.432+0000 7f16de73f700 1 --2- 172.21.15.42:0/601015169 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f16c4009660 0x7f16c40501a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:39:52.452 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:52.432+0000 7f16de73f700 1 -- 172.21.15.42:0/601015169 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=2+}) v3 -- 0x7f16c4003460 con 0x7f16c40506f0 2019-12-05T06:39:52.452 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:52.432+0000 7f16de73f700 1 --2- 172.21.15.42:0/601015169 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f16c40506f0 0x7f16c4052b30 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f16d805e480 tx=0x7f16d0010030).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:52.453 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:52.432+0000 7f16de73f700 1 --2- 172.21.15.42:0/601015169 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f16c40506f0 0x7f16c4052b30 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f16d805e480 tx=0x7f16d0010030).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:52.453 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:52.432+0000 7f16de73f700 1 --2- 172.21.15.42:0/601015169 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f16c40506f0 0x7f16c4052b30 secure :-1 s=READY pgs=2945 cs=0 l=1 rx=0x7f16d805e480 tx=0x7f16d0010030).ready entity=mon.1 client_cookie=6e15637c4418ef01 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:39:52.453 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:52.432+0000 7f16de73f700 1 --2- 172.21.15.42:0/601015169 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f16c40506f0 0x7f16c4052b30 secure :-1 s=THROTTLE_DONE pgs=2945 cs=0 l=1 rx=0x7f16d805e480 tx=0x7f16d0010030).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:52.453 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:52.432+0000 7f16dcf3c700 1 -- 172.21.15.42:0/601015169 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f16d0003540 con 0x7f16c40506f0 2019-12-05T06:39:52.454 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:52.432+0000 7f16de73f700 1 --2- 172.21.15.42:0/601015169 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f16c40506f0 0x7f16c4052b30 secure :-1 s=THROTTLE_DONE pgs=2945 cs=0 l=1 rx=0x7f16d805e480 tx=0x7f16d0010030).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:52.455 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:52.432+0000 7f16dcf3c700 1 -- 172.21.15.42:0/601015169 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16d0003160 con 0x7f16c40506f0 2019-12-05T06:39:52.455 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:52.432+0000 7f16df9a1700 1 -- 172.21.15.42:0/601015169 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f16c40506f0 msgr2=0x7f16c4052b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:52.455 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:52.432+0000 7f16df9a1700 1 --2- 172.21.15.42:0/601015169 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f16c40506f0 0x7f16c4052b30 secure :-1 s=READY pgs=2945 cs=0 l=1 rx=0x7f16d805e480 tx=0x7f16d0010030).stop 2019-12-05T06:39:52.456 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:52.432+0000 7f16df9a1700 1 -- 172.21.15.42:0/601015169 shutdown_connections 2019-12-05T06:39:52.456 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:52.432+0000 7f16df9a1700 1 --2- 172.21.15.42:0/601015169 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f16c40506f0 0x7f16c4052b30 unknown :-1 s=CLOSED pgs=2945 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:52.456 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:52.432+0000 7f16df9a1700 1 --2- 172.21.15.42:0/601015169 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f16c4009660 0x7f16c40501a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:39:52.456 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:52.432+0000 7f16df9a1700 1 --2- 172.21.15.42:0/601015169 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f16c4004420 0x7f16c4004850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:39:52.456 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:52.432+0000 7f16df9a1700 1 -- 172.21.15.42:0/601015169 shutdown_connections 2019-12-05T06:39:52.462 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:52.432+0000 7f16df9a1700 1 -- 172.21.15.42:0/601015169 wait complete. 2019-12-05T06:39:52.462 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:52.432+0000 7f16df9a1700 1 -- 172.21.15.42:0/601015169 >> 172.21.15.42:0/601015169 conn(0x7f16d80aabe0 msgr2=0x7f16d80a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:39:52.471 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:52.436+0000 7f16df9a1700 1 Processor -- start 2019-12-05T06:39:52.471 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:52.436+0000 7f16df9a1700 1 -- start start 2019-12-05T06:39:52.471 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:52.440+0000 7f16df9a1700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f16d8065980 0x7f16d8062880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:39:52.472 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:52.440+0000 7f16df9a1700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16d805fe20 con 0x7f16d805f9f0 2019-12-05T06:39:52.472 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:52.440+0000 7f16df9a1700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16d8060830 con 0x7f16d805f5c0 2019-12-05T06:39:52.472 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:52.440+0000 7f16df9a1700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16d8060e00 con 0x7f16d8065980 2019-12-05T06:39:52.472 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:52.440+0000 7f16dd73d700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f16d805f5c0 0x7f16d8133520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:60494/0 (socket says 172.21.15.42:60494) 2019-12-05T06:39:52.472 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:52.440+0000 7f16dd73d700 1 -- 172.21.15.42:0/3421968293 learned_addr learned my addr 172.21.15.42:0/3421968293 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:39:52.473 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:52.440+0000 7f16ddf3e700 1 --2- 172.21.15.42:0/3421968293 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f16d8065980 0x7f16d8062880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:52.473 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:52.440+0000 7f16ca7fc700 1 -- 172.21.15.42:0/3421968293 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4140150097 0 0) 0x7f16d8060830 con 0x7f16d805f5c0 2019-12-05T06:39:52.473 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:52.440+0000 7f16ca7fc700 1 -- 172.21.15.42:0/3421968293 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16c00030b0 con 0x7f16d805f5c0 2019-12-05T06:39:52.473 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:52.440+0000 7f16ca7fc700 1 -- 172.21.15.42:0/3421968293 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3338320538 0 0) 0x7f16d805fe20 con 0x7f16d805f9f0 2019-12-05T06:39:52.473 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:52.440+0000 7f16ca7fc700 1 -- 172.21.15.42:0/3421968293 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16d8060830 con 0x7f16d805f9f0 2019-12-05T06:39:52.474 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:52.440+0000 7f16ddf3e700 1 -- 172.21.15.42:0/3421968293 >> v1:172.21.15.42:6789/0 conn(0x7f16d805f5c0 legacy=0x7f16d8133520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:52.474 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:52.440+0000 7f16ddf3e700 1 -- 172.21.15.42:0/3421968293 >> v1:172.21.15.40:6789/0 conn(0x7f16d805f9f0 legacy=0x7f16d8133d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:52.474 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:52.440+0000 7f16ddf3e700 1 -- 172.21.15.42:0/3421968293 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16d8060b40 con 0x7f16d8065980 2019-12-05T06:39:52.474 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:52.440+0000 7f16ddf3e700 1 --2- 172.21.15.42:0/3421968293 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f16d8065980 0x7f16d8062880 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f16cc0103f0 tx=0x7f16cc010ed0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:52.475 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:52.440+0000 7f16ddf3e700 1 --2- 172.21.15.42:0/3421968293 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f16d8065980 0x7f16d8062880 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f16cc0103f0 tx=0x7f16cc010ed0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:52.475 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:52.440+0000 7f16ddf3e700 1 --2- 172.21.15.42:0/3421968293 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f16d8065980 0x7f16d8062880 secure :-1 s=READY pgs=653 cs=0 l=1 rx=0x7f16cc0103f0 tx=0x7f16cc010ed0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:39:52.475 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:52.440+0000 7f16ddf3e700 1 --2- 172.21.15.42:0/3421968293 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f16d8065980 0x7f16d8062880 secure :-1 s=THROTTLE_DONE pgs=653 cs=0 l=1 rx=0x7f16cc0103f0 tx=0x7f16cc010ed0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:52.475 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:52.440+0000 7f16ca7fc700 1 -- 172.21.15.42:0/3421968293 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f16cc008b20 con 0x7f16d8065980 2019-12-05T06:39:52.476 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:52.440+0000 7f16ddf3e700 1 --2- 172.21.15.42:0/3421968293 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f16d8065980 0x7f16d8062880 secure :-1 s=THROTTLE_DONE pgs=653 cs=0 l=1 rx=0x7f16cc0103f0 tx=0x7f16cc010ed0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:52.476 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:52.440+0000 7f16ddf3e700 1 --2- 172.21.15.42:0/3421968293 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f16d8065980 0x7f16d8062880 secure :-1 s=THROTTLE_DONE pgs=653 cs=0 l=1 rx=0x7f16cc0103f0 tx=0x7f16cc010ed0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:52.476 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:52.440+0000 7f16ca7fc700 1 -- 172.21.15.42:0/3421968293 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16cc008740 con 0x7f16d8065980 2019-12-05T06:39:52.476 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:52.440+0000 7f16df9a1700 1 -- 172.21.15.42:0/3421968293 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16d8077300 con 0x7f16d8065980 2019-12-05T06:39:52.476 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:52.440+0000 7f16ca7fc700 1 -- 172.21.15.42:0/3421968293 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f16cc005730 con 0x7f16d8065980 2019-12-05T06:39:52.477 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:52.440+0000 7f16df9a1700 1 -- 172.21.15.42:0/3421968293 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16d8077880 con 0x7f16d8065980 2019-12-05T06:39:52.477 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:52.444+0000 7f16ddf3e700 1 --2- 172.21.15.42:0/3421968293 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f16d8065980 0x7f16d8062880 secure :-1 s=THROTTLE_DONE pgs=653 cs=0 l=1 rx=0x7f16cc0103f0 tx=0x7f16cc010ed0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:52.477 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:52.444+0000 7f16ca7fc700 1 -- 172.21.15.42:0/3421968293 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f16cc005930 con 0x7f16d8065980 2019-12-05T06:39:52.477 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:52.444+0000 7f16ddf3e700 1 --2- 172.21.15.42:0/3421968293 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f16d8065980 0x7f16d8062880 secure :-1 s=THROTTLE_DONE pgs=653 cs=0 l=1 rx=0x7f16cc0103f0 tx=0x7f16cc010ed0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:52.477 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:52.444+0000 7f16ca7fc700 1 --2- 172.21.15.42:0/3421968293 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f16c003b4c0 0x7f16c003d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:39:52.477 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:52.444+0000 7f16ca7fc700 1 -- 172.21.15.42:0/3421968293 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 11535+0+0 (secure 0 0 0) 0x7f16cc04a610 con 0x7f16d8065980 2019-12-05T06:39:52.478 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:52.444+0000 7f16dd73d700 1 --2- 172.21.15.42:0/3421968293 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f16c003b4c0 0x7f16c003d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:52.478 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:52.444+0000 7f16dd73d700 1 --2- 172.21.15.42:0/3421968293 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f16c003b4c0 0x7f16c003d9d0 crc :-1 s=READY pgs=1235 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:39:52.496 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:52.488+0000 7f16df9a1700 1 -- 172.21.15.42:0/3421968293 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16d805d110 con 0x7f16d8065980 2019-12-05T06:39:52.505 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:52.500+0000 7f16ddf3e700 1 --2- 172.21.15.42:0/3421968293 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f16d8065980 0x7f16d8062880 secure :-1 s=THROTTLE_DONE pgs=653 cs=0 l=1 rx=0x7f16cc0103f0 tx=0x7f16cc010ed0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:52.505 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:52.500+0000 7f16ca7fc700 1 -- 172.21.15.42:0/3421968293 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f16d8060b40 con 0x7f16d8065980 2019-12-05T06:39:52.673 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:52.668+0000 7f16df9a1700 1 -- 172.21.15.42:0/3421968293 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f16d805b8c0 con 0x7f16d8065980 2019-12-05T06:39:52.675 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:52.668+0000 7f16ddf3e700 1 --2- 172.21.15.42:0/3421968293 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f16d8065980 0x7f16d8062880 secure :-1 s=THROTTLE_DONE pgs=653 cs=0 l=1 rx=0x7f16cc0103f0 tx=0x7f16cc010ed0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:52.675 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:52.668+0000 7f16ca7fc700 1 -- 172.21.15.42:0/3421968293 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 354 v354) v1 ==== 95+0+2111 (secure 0 0 0) 0x7f16cc031690 con 0x7f16d8065980 2019-12-05T06:39:52.677 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 354 2019-12-05T06:39:52.678 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:39:52.678 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":354,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":11868,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/406829134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":406829134},{"type":"v1","addr":"172.21.15.42:6835","nonce":406829134}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":354}],"filesystems":[{"mdsmap":{"epoch":353,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:39:37.486362+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":113,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8009},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8009":{"gid":8009,"name":"b","rank":0,"incarnation":349,"state":"up:active","state_seq":27,"addr":"172.21.15.40:6833/1777669201","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1777669201},{"type":"v1","addr":"172.21.15.40:6833","nonce":1777669201}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:39:52.680 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:52.672+0000 7f16df9a1700 1 -- 172.21.15.42:0/3421968293 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f16c003b4c0 msgr2=0x7f16c003d9d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:52.680 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:52.672+0000 7f16df9a1700 1 --2- 172.21.15.42:0/3421968293 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f16c003b4c0 0x7f16c003d9d0 crc :-1 s=READY pgs=1235 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:52.680 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:52.672+0000 7f16df9a1700 1 -- 172.21.15.42:0/3421968293 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f16d8065980 msgr2=0x7f16d8062880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:52.680 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:52.672+0000 7f16df9a1700 1 --2- 172.21.15.42:0/3421968293 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f16d8065980 0x7f16d8062880 secure :-1 s=READY pgs=653 cs=0 l=1 rx=0x7f16cc0103f0 tx=0x7f16cc010ed0).stop 2019-12-05T06:39:52.681 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:52.672+0000 7f16df9a1700 1 -- 172.21.15.42:0/3421968293 shutdown_connections 2019-12-05T06:39:52.681 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:52.672+0000 7f16df9a1700 1 --2- 172.21.15.42:0/3421968293 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f16c003b4c0 0x7f16c003d9d0 unknown :-1 s=CLOSED pgs=1235 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:52.681 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:52.672+0000 7f16df9a1700 1 --2- 172.21.15.42:0/3421968293 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f16d8065980 0x7f16d8062880 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:52.681 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:52.672+0000 7f16df9a1700 1 -- 172.21.15.42:0/3421968293 shutdown_connections 2019-12-05T06:39:52.682 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:52.672+0000 7f16df9a1700 1 -- 172.21.15.42:0/3421968293 wait complete. 2019-12-05T06:39:52.682 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:52.672+0000 7f16df9a1700 1 -- 172.21.15.42:0/3421968293 >> 172.21.15.42:0/3421968293 conn(0x7f16d80a6fa0 msgr2=0x7f16d8059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:39:52.690 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:39:54.691 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:39:54.799 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:54.788+0000 7f4e58738700 1 Processor -- start 2019-12-05T06:39:54.799 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:54.788+0000 7f4e58738700 1 -- start start 2019-12-05T06:39:54.799 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:54.788+0000 7f4e58738700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f4e500a7740 0x7f4e5008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:39:54.799 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:54.788+0000 7f4e58738700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f4e5007e720 0x7f4e5007eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:39:54.800 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:54.788+0000 7f4e58738700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e500a0710 con 0x7f4e5009f2e0 2019-12-05T06:39:54.800 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:54.788+0000 7f4e58738700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e500a0b00 con 0x7f4e500a7740 2019-12-05T06:39:54.800 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:54.788+0000 7f4e58738700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e5008e5b0 con 0x7f4e5007e720 2019-12-05T06:39:54.800 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:54.788+0000 7f4e564d4700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f4e5007e720 0x7f4e5007eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:54.800 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:54.788+0000 7f4e564d4700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f4e5007e720 0x7f4e5007eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:60464/0 (socket says 172.21.15.42:60464) 2019-12-05T06:39:54.800 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:54.788+0000 7f4e564d4700 1 -- 172.21.15.42:0/497533375 learned_addr learned my addr 172.21.15.42:0/497533375 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:39:54.800 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:54.788+0000 7f4e56cd5700 1 --2- 172.21.15.42:0/497533375 >> v2:172.21.15.40:3300/0 conn(0x7f4e500a7740 0x7f4e5008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:54.801 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:54.788+0000 7f4e564d4700 1 -- 172.21.15.42:0/497533375 >> v2:172.21.15.40:3300/0 conn(0x7f4e500a7740 msgr2=0x7f4e5008df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:39:54.801 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:54.788+0000 7f4e564d4700 1 --2- 172.21.15.42:0/497533375 >> v2:172.21.15.40:3300/0 conn(0x7f4e500a7740 0x7f4e5008df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:39:54.801 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:54.788+0000 7f4e564d4700 1 -- 172.21.15.42:0/497533375 >> v1:172.21.15.42:6789/0 conn(0x7f4e5009f2e0 legacy=0x7f4e5009f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:54.801 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:54.788+0000 7f4e564d4700 1 -- 172.21.15.42:0/497533375 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e5007f0a0 con 0x7f4e5007e720 2019-12-05T06:39:54.801 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:54.788+0000 7f4e564d4700 1 --2- 172.21.15.42:0/497533375 >> v2:172.21.15.42:3300/0 conn(0x7f4e5007e720 0x7f4e5007eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f4e40002f30 tx=0x7f4e4000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:54.801 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:54.792+0000 7f4e564d4700 1 --2- 172.21.15.42:0/497533375 >> v2:172.21.15.42:3300/0 conn(0x7f4e5007e720 0x7f4e5007eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f4e40002f30 tx=0x7f4e4000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:54.801 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:54.792+0000 7f4e564d4700 1 --2- 172.21.15.42:0/497533375 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4e5007e720 0x7f4e5007eb50 secure :-1 s=READY pgs=2948 cs=0 l=1 rx=0x7f4e40002f30 tx=0x7f4e4000fd30).ready entity=mon.1 client_cookie=58a2aef05e8d0489 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:39:54.802 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:54.792+0000 7f4e564d4700 1 --2- 172.21.15.42:0/497533375 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4e5007e720 0x7f4e5007eb50 secure :-1 s=THROTTLE_DONE pgs=2948 cs=0 l=1 rx=0x7f4e40002f30 tx=0x7f4e4000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:54.802 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:54.792+0000 7f4e564d4700 1 --2- 172.21.15.42:0/497533375 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4e5007e720 0x7f4e5007eb50 secure :-1 s=THROTTLE_DONE pgs=2948 cs=0 l=1 rx=0x7f4e40002f30 tx=0x7f4e4000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:54.802 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:54.792+0000 7f4e564d4700 1 --2- 172.21.15.42:0/497533375 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4e5007e720 0x7f4e5007eb50 secure :-1 s=THROTTLE_DONE pgs=2948 cs=0 l=1 rx=0x7f4e40002f30 tx=0x7f4e4000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:54.802 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:54.792+0000 7f4e55cd3700 1 -- 172.21.15.42:0/497533375 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4e40011370 con 0x7f4e5007e720 2019-12-05T06:39:54.802 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:54.792+0000 7f4e55cd3700 1 -- 172.21.15.42:0/497533375 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e40008bb0 con 0x7f4e5007e720 2019-12-05T06:39:54.802 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:54.792+0000 7f4e55cd3700 1 -- 172.21.15.42:0/497533375 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4e40011950 con 0x7f4e5007e720 2019-12-05T06:39:54.802 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:54.792+0000 7f4e58738700 1 -- 172.21.15.42:0/497533375 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4e5007e720 msgr2=0x7f4e5007eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:54.803 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:54.792+0000 7f4e58738700 1 --2- 172.21.15.42:0/497533375 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4e5007e720 0x7f4e5007eb50 secure :-1 s=READY pgs=2948 cs=0 l=1 rx=0x7f4e40002f30 tx=0x7f4e4000fd30).stop 2019-12-05T06:39:54.803 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:54.792+0000 7f4e58738700 1 -- 172.21.15.42:0/497533375 shutdown_connections 2019-12-05T06:39:54.803 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:54.792+0000 7f4e58738700 1 --2- 172.21.15.42:0/497533375 >> v2:172.21.15.40:3300/0 conn(0x7f4e500a7740 0x7f4e5008df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:39:54.803 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:54.792+0000 7f4e58738700 1 --2- 172.21.15.42:0/497533375 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4e5007e720 0x7f4e5007eb50 unknown :-1 s=CLOSED pgs=2948 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:54.803 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:54.792+0000 7f4e58738700 1 -- 172.21.15.42:0/497533375 shutdown_connections 2019-12-05T06:39:54.803 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:54.792+0000 7f4e58738700 1 -- 172.21.15.42:0/497533375 wait complete. 2019-12-05T06:39:54.803 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:54.792+0000 7f4e58738700 1 -- 172.21.15.42:0/497533375 >> 172.21.15.42:0/497533375 conn(0x7f4e500aabe0 msgr2=0x7f4e500a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:39:54.809 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:54.796+0000 7f4e58738700 1 Processor -- start 2019-12-05T06:39:54.809 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:54.796+0000 7f4e58738700 1 -- start start 2019-12-05T06:39:54.809 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:54.796+0000 7f4e58738700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4e5007b6f0 0x7f4e50079350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:39:54.809 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:54.796+0000 7f4e58738700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e500a4a50 con 0x7f4e500798a0 2019-12-05T06:39:54.810 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:54.796+0000 7f4e58738700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e5007c290 con 0x7f4e500a9910 2019-12-05T06:39:54.810 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:54.796+0000 7f4e58738700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e50127cc0 con 0x7f4e5007b6f0 2019-12-05T06:39:54.810 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:54.796+0000 7f4e56cd5700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f4e500a9910 0x7f4e501274c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:60508/0 (socket says 172.21.15.42:60508) 2019-12-05T06:39:54.810 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:54.796+0000 7f4e56cd5700 1 -- 172.21.15.42:0/259418401 learned_addr learned my addr 172.21.15.42:0/259418401 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:39:54.810 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:54.796+0000 7f4e564d4700 1 --2- 172.21.15.42:0/259418401 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4e5007b6f0 0x7f4e50079350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:54.811 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:54.796+0000 7f4e477fe700 1 -- 172.21.15.42:0/259418401 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3302804631 0 0) 0x7f4e5007c290 con 0x7f4e500a9910 2019-12-05T06:39:54.811 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:54.796+0000 7f4e477fe700 1 -- 172.21.15.42:0/259418401 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e380030b0 con 0x7f4e500a9910 2019-12-05T06:39:54.811 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:54.796+0000 7f4e477fe700 1 -- 172.21.15.42:0/259418401 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3352248069 0 0) 0x7f4e500a4a50 con 0x7f4e500798a0 2019-12-05T06:39:54.811 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:54.796+0000 7f4e477fe700 1 -- 172.21.15.42:0/259418401 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e5007c290 con 0x7f4e500798a0 2019-12-05T06:39:54.811 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:54.796+0000 7f4e477fe700 1 -- 172.21.15.42:0/259418401 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 115602590 0 0) 0x7f4e380030b0 con 0x7f4e500a9910 2019-12-05T06:39:54.811 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:54.796+0000 7f4e564d4700 1 -- 172.21.15.42:0/259418401 >> v1:172.21.15.42:6789/0 conn(0x7f4e500a9910 legacy=0x7f4e501274c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:54.811 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:54.796+0000 7f4e564d4700 1 -- 172.21.15.42:0/259418401 >> v1:172.21.15.40:6789/0 conn(0x7f4e500798a0 legacy=0x7f4e501317c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:54.812 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:54.796+0000 7f4e564d4700 1 -- 172.21.15.42:0/259418401 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e5007bb20 con 0x7f4e5007b6f0 2019-12-05T06:39:54.812 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:54.800+0000 7f4e564d4700 1 --2- 172.21.15.42:0/259418401 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4e5007b6f0 0x7f4e50079350 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f4e4c007940 tx=0x7f4e4c002f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:54.812 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:54.800+0000 7f4e564d4700 1 --2- 172.21.15.42:0/259418401 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4e5007b6f0 0x7f4e50079350 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f4e4c007940 tx=0x7f4e4c002f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:54.812 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:54.800+0000 7f4e564d4700 1 --2- 172.21.15.42:0/259418401 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4e5007b6f0 0x7f4e50079350 secure :-1 s=READY pgs=654 cs=0 l=1 rx=0x7f4e4c007940 tx=0x7f4e4c002f70).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:39:54.812 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:54.800+0000 7f4e564d4700 1 --2- 172.21.15.42:0/259418401 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4e5007b6f0 0x7f4e50079350 secure :-1 s=THROTTLE_DONE pgs=654 cs=0 l=1 rx=0x7f4e4c007940 tx=0x7f4e4c002f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:54.812 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:54.800+0000 7f4e477fe700 1 -- 172.21.15.42:0/259418401 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4e4c001490 con 0x7f4e5007b6f0 2019-12-05T06:39:54.812 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:54.800+0000 7f4e564d4700 1 --2- 172.21.15.42:0/259418401 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4e5007b6f0 0x7f4e50079350 secure :-1 s=THROTTLE_DONE pgs=654 cs=0 l=1 rx=0x7f4e4c007940 tx=0x7f4e4c002f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:54.813 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:54.800+0000 7f4e564d4700 1 --2- 172.21.15.42:0/259418401 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4e5007b6f0 0x7f4e50079350 secure :-1 s=THROTTLE_DONE pgs=654 cs=0 l=1 rx=0x7f4e4c007940 tx=0x7f4e4c002f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:54.813 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:54.800+0000 7f4e477fe700 1 -- 172.21.15.42:0/259418401 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e4c0012a0 con 0x7f4e5007b6f0 2019-12-05T06:39:54.813 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:54.800+0000 7f4e58738700 1 -- 172.21.15.42:0/259418401 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e50079cd0 con 0x7f4e5007b6f0 2019-12-05T06:39:54.813 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:54.800+0000 7f4e477fe700 1 -- 172.21.15.42:0/259418401 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4e40007340 con 0x7f4e5007b6f0 2019-12-05T06:39:54.813 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:54.800+0000 7f4e58738700 1 -- 172.21.15.42:0/259418401 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e50132fd0 con 0x7f4e5007b6f0 2019-12-05T06:39:54.813 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:54.800+0000 7f4e564d4700 1 --2- 172.21.15.42:0/259418401 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4e5007b6f0 0x7f4e50079350 secure :-1 s=THROTTLE_DONE pgs=654 cs=0 l=1 rx=0x7f4e4c007940 tx=0x7f4e4c002f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:54.813 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:54.800+0000 7f4e477fe700 1 -- 172.21.15.42:0/259418401 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f4e4000f360 con 0x7f4e5007b6f0 2019-12-05T06:39:54.814 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:54.800+0000 7f4e564d4700 1 --2- 172.21.15.42:0/259418401 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4e5007b6f0 0x7f4e50079350 secure :-1 s=THROTTLE_DONE pgs=654 cs=0 l=1 rx=0x7f4e4c007940 tx=0x7f4e4c002f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:54.814 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:54.800+0000 7f4e477fe700 1 --2- 172.21.15.42:0/259418401 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4e3803b420 0x7f4e3803d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:39:54.814 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:54.800+0000 7f4e477fe700 1 -- 172.21.15.42:0/259418401 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 11535+0+0 (secure 0 0 0) 0x7f4e4004ace0 con 0x7f4e5007b6f0 2019-12-05T06:39:54.814 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:54.800+0000 7f4e56cd5700 1 --2- 172.21.15.42:0/259418401 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4e3803b420 0x7f4e3803d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:54.814 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:54.804+0000 7f4e56cd5700 1 --2- 172.21.15.42:0/259418401 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4e3803b420 0x7f4e3803d930 crc :-1 s=READY pgs=1236 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:39:54.815 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:54.804+0000 7f4e58738700 1 -- 172.21.15.42:0/259418401 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e5005d110 con 0x7f4e5007b6f0 2019-12-05T06:39:54.819 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:54.812+0000 7f4e564d4700 1 --2- 172.21.15.42:0/259418401 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4e5007b6f0 0x7f4e50079350 secure :-1 s=THROTTLE_DONE pgs=654 cs=0 l=1 rx=0x7f4e4c007940 tx=0x7f4e4c002f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:54.819 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:54.812+0000 7f4e477fe700 1 -- 172.21.15.42:0/259418401 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f4e40031fa0 con 0x7f4e5007b6f0 2019-12-05T06:39:55.000 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:54.980+0000 7f4e58738700 1 -- 172.21.15.42:0/259418401 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4e5005b8c0 con 0x7f4e5007b6f0 2019-12-05T06:39:55.000 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:54.980+0000 7f4e564d4700 1 --2- 172.21.15.42:0/259418401 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4e5007b6f0 0x7f4e50079350 secure :-1 s=THROTTLE_DONE pgs=654 cs=0 l=1 rx=0x7f4e4c007940 tx=0x7f4e4c002f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:55.001 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:54.980+0000 7f4e477fe700 1 -- 172.21.15.42:0/259418401 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 354 v354) v1 ==== 95+0+2111 (secure 0 0 0) 0x7f4e4000f670 con 0x7f4e5007b6f0 2019-12-05T06:39:55.001 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 354 2019-12-05T06:39:55.001 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:54.984+0000 7f4e58738700 1 -- 172.21.15.42:0/259418401 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4e3803b420 msgr2=0x7f4e3803d930 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:55.001 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:54.984+0000 7f4e58738700 1 --2- 172.21.15.42:0/259418401 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4e3803b420 0x7f4e3803d930 crc :-1 s=READY pgs=1236 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:55.002 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:54.984+0000 7f4e58738700 1 -- 172.21.15.42:0/259418401 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4e5007b6f0 msgr2=0x7f4e50079350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:55.002 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:54.984+0000 7f4e58738700 1 --2- 172.21.15.42:0/259418401 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4e5007b6f0 0x7f4e50079350 secure :-1 s=READY pgs=654 cs=0 l=1 rx=0x7f4e4c007940 tx=0x7f4e4c002f70).stop 2019-12-05T06:39:55.002 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:54.984+0000 7f4e58738700 1 -- 172.21.15.42:0/259418401 shutdown_connections 2019-12-05T06:39:55.002 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:54.984+0000 7f4e58738700 1 --2- 172.21.15.42:0/259418401 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4e3803b420 0x7f4e3803d930 unknown :-1 s=CLOSED pgs=1236 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:55.003 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:54.984+0000 7f4e58738700 1 --2- 172.21.15.42:0/259418401 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4e5007b6f0 0x7f4e50079350 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:55.003 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:54.984+0000 7f4e58738700 1 -- 172.21.15.42:0/259418401 shutdown_connections 2019-12-05T06:39:55.003 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:54.984+0000 7f4e58738700 1 -- 172.21.15.42:0/259418401 wait complete. 2019-12-05T06:39:55.003 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:54.984+0000 7f4e58738700 1 -- 172.21.15.42:0/259418401 >> 172.21.15.42:0/259418401 conn(0x7f4e500a7f90 msgr2=0x7f4e5009f200 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:39:55.004 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:39:55.004 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":354,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":11868,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/406829134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":406829134},{"type":"v1","addr":"172.21.15.42:6835","nonce":406829134}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":354}],"filesystems":[{"mdsmap":{"epoch":353,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:39:37.486362+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":113,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8009},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8009":{"gid":8009,"name":"b","rank":0,"incarnation":349,"state":"up:active","state_seq":27,"addr":"172.21.15.40:6833/1777669201","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1777669201},{"type":"v1","addr":"172.21.15.40:6833","nonce":1777669201}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:39:55.006 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:39:57.006 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:39:57.115 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:57.104+0000 7f8b392ac700 1 Processor -- start 2019-12-05T06:39:57.116 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:57.104+0000 7f8b392ac700 1 -- start start 2019-12-05T06:39:57.116 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:57.104+0000 7f8b392ac700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8b340a7730 0x7f8b3408df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:39:57.116 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:57.104+0000 7f8b392ac700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8b3409fad0 0x7f8b3407eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:39:57.116 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:57.104+0000 7f8b392ac700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b340a0700 con 0x7f8b3407e710 2019-12-05T06:39:57.116 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:57.104+0000 7f8b392ac700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b340a0af0 con 0x7f8b3409fad0 2019-12-05T06:39:57.116 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:57.104+0000 7f8b392ac700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b3408e5a0 con 0x7f8b340a7730 2019-12-05T06:39:57.117 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:57.104+0000 7f8b32ffd700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f8b3407e710 0x7f8b3409f2d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:60520/0 (socket says 172.21.15.42:60520) 2019-12-05T06:39:57.117 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:57.104+0000 7f8b32ffd700 1 -- 172.21.15.42:0/3213415287 learned_addr learned my addr 172.21.15.42:0/3213415287 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:39:57.117 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:57.104+0000 7f8b33fff700 1 --2- 172.21.15.42:0/3213415287 >> v2:172.21.15.40:3300/0 conn(0x7f8b3409fad0 0x7f8b3407eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:57.117 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:57.104+0000 7f8b337fe700 1 --2- 172.21.15.42:0/3213415287 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8b340a7730 0x7f8b3408df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:57.117 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:57.104+0000 7f8b327fc700 1 -- 172.21.15.42:0/3213415287 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2450130597 0 0) 0x7f8b340a0700 con 0x7f8b3407e710 2019-12-05T06:39:57.118 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:57.104+0000 7f8b327fc700 1 -- 172.21.15.42:0/3213415287 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b200030b0 con 0x7f8b3407e710 2019-12-05T06:39:57.118 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:57.108+0000 7f8b33fff700 1 -- 172.21.15.42:0/3213415287 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8b340a7730 msgr2=0x7f8b3408df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:39:57.118 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:57.108+0000 7f8b33fff700 1 --2- 172.21.15.42:0/3213415287 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8b340a7730 0x7f8b3408df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:39:57.118 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:57.108+0000 7f8b33fff700 1 -- 172.21.15.42:0/3213415287 >> v1:172.21.15.42:6789/0 conn(0x7f8b3407e710 legacy=0x7f8b3409f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:57.118 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:57.108+0000 7f8b33fff700 1 -- 172.21.15.42:0/3213415287 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b3407f090 con 0x7f8b3409fad0 2019-12-05T06:39:57.119 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:57.108+0000 7f8b337fe700 1 --2- 172.21.15.42:0/3213415287 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8b340a7730 0x7f8b3408df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:39:57.119 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:57.108+0000 7f8b33fff700 1 --2- 172.21.15.42:0/3213415287 >> v2:172.21.15.40:3300/0 conn(0x7f8b3409fad0 0x7f8b3407eb40 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f8b2c008790 tx=0x7f8b2c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:57.119 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:57.108+0000 7f8b33fff700 1 --2- 172.21.15.42:0/3213415287 >> v2:172.21.15.40:3300/0 conn(0x7f8b3409fad0 0x7f8b3407eb40 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f8b2c008790 tx=0x7f8b2c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:57.119 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:57.108+0000 7f8b33fff700 1 --2- 172.21.15.42:0/3213415287 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8b3409fad0 0x7f8b3407eb40 secure :-1 s=READY pgs=2239 cs=0 l=1 rx=0x7f8b2c008790 tx=0x7f8b2c00fc70).ready entity=mon.0 client_cookie=17966938c952ef53 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:39:57.119 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:57.108+0000 7f8b33fff700 1 --2- 172.21.15.42:0/3213415287 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8b3409fad0 0x7f8b3407eb40 secure :-1 s=THROTTLE_DONE pgs=2239 cs=0 l=1 rx=0x7f8b2c008790 tx=0x7f8b2c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:57.119 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:57.108+0000 7f8b327fc700 1 -- 172.21.15.42:0/3213415287 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8b2c008bb0 con 0x7f8b3409fad0 2019-12-05T06:39:57.119 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:57.108+0000 7f8b33fff700 1 --2- 172.21.15.42:0/3213415287 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8b3409fad0 0x7f8b3407eb40 secure :-1 s=THROTTLE_DONE pgs=2239 cs=0 l=1 rx=0x7f8b2c008790 tx=0x7f8b2c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:57.120 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:57.108+0000 7f8b327fc700 1 -- 172.21.15.42:0/3213415287 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b2c0087d0 con 0x7f8b3409fad0 2019-12-05T06:39:57.120 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:57.108+0000 7f8b33fff700 1 --2- 172.21.15.42:0/3213415287 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8b3409fad0 0x7f8b3407eb40 secure :-1 s=THROTTLE_DONE pgs=2239 cs=0 l=1 rx=0x7f8b2c008790 tx=0x7f8b2c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:57.120 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:57.108+0000 7f8b327fc700 1 -- 172.21.15.42:0/3213415287 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8b2c011860 con 0x7f8b3409fad0 2019-12-05T06:39:57.120 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:57.108+0000 7f8b392ac700 1 -- 172.21.15.42:0/3213415287 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8b3409fad0 msgr2=0x7f8b3407eb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:57.120 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:57.108+0000 7f8b392ac700 1 --2- 172.21.15.42:0/3213415287 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8b3409fad0 0x7f8b3407eb40 secure :-1 s=READY pgs=2239 cs=0 l=1 rx=0x7f8b2c008790 tx=0x7f8b2c00fc70).stop 2019-12-05T06:39:57.120 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:57.108+0000 7f8b392ac700 1 -- 172.21.15.42:0/3213415287 shutdown_connections 2019-12-05T06:39:57.121 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:57.108+0000 7f8b392ac700 1 --2- 172.21.15.42:0/3213415287 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8b3409fad0 0x7f8b3407eb40 unknown :-1 s=CLOSED pgs=2239 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:57.121 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:57.108+0000 7f8b392ac700 1 --2- 172.21.15.42:0/3213415287 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8b340a7730 0x7f8b3408df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:39:57.121 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:57.108+0000 7f8b392ac700 1 -- 172.21.15.42:0/3213415287 shutdown_connections 2019-12-05T06:39:57.121 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:57.108+0000 7f8b392ac700 1 -- 172.21.15.42:0/3213415287 wait complete. 2019-12-05T06:39:57.121 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:57.108+0000 7f8b392ac700 1 -- 172.21.15.42:0/3213415287 >> 172.21.15.42:0/3213415287 conn(0x7f8b340aabd0 msgr2=0x7f8b340a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:39:57.124 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:57.112+0000 7f8b392ac700 1 Processor -- start 2019-12-05T06:39:57.125 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:57.112+0000 7f8b392ac700 1 -- start start 2019-12-05T06:39:57.125 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:57.112+0000 7f8b392ac700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f8b3411a510 0x7f8b3407d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:39:57.125 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:57.112+0000 7f8b392ac700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8b34121320 0x7f8b3407dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:39:57.125 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:57.112+0000 7f8b392ac700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8b34117610 0x7f8b34117a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:39:57.125 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:57.112+0000 7f8b392ac700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b3411baf0 con 0x7f8b34121320 2019-12-05T06:39:57.126 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:57.112+0000 7f8b392ac700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b3411bd90 con 0x7f8b3411a510 2019-12-05T06:39:57.126 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:57.112+0000 7f8b392ac700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b3411c030 con 0x7f8b34117610 2019-12-05T06:39:57.126 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:57.112+0000 7f8b337fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f8b3411a510 0x7f8b3407d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:57.126 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:57.112+0000 7f8b337fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f8b3411a510 0x7f8b3407d7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:60486/0 (socket says 172.21.15.42:60486) 2019-12-05T06:39:57.126 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:57.112+0000 7f8b337fe700 1 -- 172.21.15.42:0/4275302609 learned_addr learned my addr 172.21.15.42:0/4275302609 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:39:57.126 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:57.112+0000 7f8b33fff700 1 --2- 172.21.15.42:0/4275302609 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8b34117610 0x7f8b34117a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:57.126 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:57.112+0000 7f8b32ffd700 1 --2- 172.21.15.42:0/4275302609 >> v2:172.21.15.40:3300/0 conn(0x7f8b34121320 0x7f8b3407dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:57.127 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:57.116+0000 7f8b337fe700 1 -- 172.21.15.42:0/4275302609 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8b34117610 msgr2=0x7f8b34117a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:57.127 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:57.116+0000 7f8b337fe700 1 --2- 172.21.15.42:0/4275302609 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8b34117610 0x7f8b34117a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:57.127 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:57.116+0000 7f8b337fe700 1 -- 172.21.15.42:0/4275302609 >> v2:172.21.15.40:3300/0 conn(0x7f8b34121320 msgr2=0x7f8b3407dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:57.127 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:57.116+0000 7f8b337fe700 1 --2- 172.21.15.42:0/4275302609 >> v2:172.21.15.40:3300/0 conn(0x7f8b34121320 0x7f8b3407dd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:57.127 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:57.116+0000 7f8b337fe700 1 -- 172.21.15.42:0/4275302609 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b34117f90 con 0x7f8b3411a510 2019-12-05T06:39:57.127 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:57.116+0000 7f8b337fe700 1 --2- 172.21.15.42:0/4275302609 >> v2:172.21.15.42:3300/0 conn(0x7f8b3411a510 0x7f8b3407d7b0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f8b3411bf50 tx=0x7f8b1c010db0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:57.127 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:57.116+0000 7f8b337fe700 1 --2- 172.21.15.42:0/4275302609 >> v2:172.21.15.42:3300/0 conn(0x7f8b3411a510 0x7f8b3407d7b0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f8b3411bf50 tx=0x7f8b1c010db0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:57.128 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:57.116+0000 7f8b337fe700 1 --2- 172.21.15.42:0/4275302609 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8b3411a510 0x7f8b3407d7b0 secure :-1 s=READY pgs=2951 cs=0 l=1 rx=0x7f8b3411bf50 tx=0x7f8b1c010db0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:39:57.128 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:57.116+0000 7f8b337fe700 1 --2- 172.21.15.42:0/4275302609 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8b3411a510 0x7f8b3407d7b0 secure :-1 s=THROTTLE_DONE pgs=2951 cs=0 l=1 rx=0x7f8b3411bf50 tx=0x7f8b1c010db0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:57.128 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:57.116+0000 7f8b2bfff700 1 -- 172.21.15.42:0/4275302609 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8b1c0126f0 con 0x7f8b3411a510 2019-12-05T06:39:57.128 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:57.116+0000 7f8b337fe700 1 --2- 172.21.15.42:0/4275302609 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8b3411a510 0x7f8b3407d7b0 secure :-1 s=THROTTLE_DONE pgs=2951 cs=0 l=1 rx=0x7f8b3411bf50 tx=0x7f8b1c010db0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:57.128 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:57.116+0000 7f8b337fe700 1 --2- 172.21.15.42:0/4275302609 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8b3411a510 0x7f8b3407d7b0 secure :-1 s=THROTTLE_DONE pgs=2951 cs=0 l=1 rx=0x7f8b3411bf50 tx=0x7f8b1c010db0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:57.128 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:57.116+0000 7f8b2bfff700 1 -- 172.21.15.42:0/4275302609 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b1c007150 con 0x7f8b3411a510 2019-12-05T06:39:57.128 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:57.116+0000 7f8b392ac700 1 -- 172.21.15.42:0/4275302609 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b3411b770 con 0x7f8b3411a510 2019-12-05T06:39:57.129 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:57.116+0000 7f8b2bfff700 1 -- 172.21.15.42:0/4275302609 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8b1c012cd0 con 0x7f8b3411a510 2019-12-05T06:39:57.129 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:57.116+0000 7f8b392ac700 1 -- 172.21.15.42:0/4275302609 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b3411dab0 con 0x7f8b3411a510 2019-12-05T06:39:57.129 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:57.116+0000 7f8b337fe700 1 --2- 172.21.15.42:0/4275302609 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8b3411a510 0x7f8b3407d7b0 secure :-1 s=THROTTLE_DONE pgs=2951 cs=0 l=1 rx=0x7f8b3411bf50 tx=0x7f8b1c010db0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:57.129 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:57.116+0000 7f8b2bfff700 1 -- 172.21.15.42:0/4275302609 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f8b1c005540 con 0x7f8b3411a510 2019-12-05T06:39:57.129 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:57.116+0000 7f8b337fe700 1 --2- 172.21.15.42:0/4275302609 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8b3411a510 0x7f8b3407d7b0 secure :-1 s=THROTTLE_DONE pgs=2951 cs=0 l=1 rx=0x7f8b3411bf50 tx=0x7f8b1c010db0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:57.129 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:57.116+0000 7f8b2bfff700 1 --2- 172.21.15.42:0/4275302609 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8b1803ae10 0x7f8b1803d320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:39:57.130 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:57.116+0000 7f8b2bfff700 1 -- 172.21.15.42:0/4275302609 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 11535+0+0 (secure 0 0 0) 0x7f8b1c04b5c0 con 0x7f8b3411a510 2019-12-05T06:39:57.130 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:57.120+0000 7f8b32ffd700 1 --2- 172.21.15.42:0/4275302609 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8b1803ae10 0x7f8b1803d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:57.130 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:57.120+0000 7f8b32ffd700 1 --2- 172.21.15.42:0/4275302609 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8b1803ae10 0x7f8b1803d320 crc :-1 s=READY pgs=1237 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:39:57.131 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:57.120+0000 7f8b392ac700 1 -- 172.21.15.42:0/4275302609 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b3405d100 con 0x7f8b3411a510 2019-12-05T06:39:57.133 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:57.128+0000 7f8b337fe700 1 --2- 172.21.15.42:0/4275302609 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8b3411a510 0x7f8b3407d7b0 secure :-1 s=THROTTLE_DONE pgs=2951 cs=0 l=1 rx=0x7f8b3411bf50 tx=0x7f8b1c010db0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:57.133 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:57.128+0000 7f8b2bfff700 1 -- 172.21.15.42:0/4275302609 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f8b1c032620 con 0x7f8b3411a510 2019-12-05T06:39:57.304 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:57.300+0000 7f8b392ac700 1 -- 172.21.15.42:0/4275302609 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8b3405b8b0 con 0x7f8b3411a510 2019-12-05T06:39:57.305 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:57.300+0000 7f8b337fe700 1 --2- 172.21.15.42:0/4275302609 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8b3411a510 0x7f8b3407d7b0 secure :-1 s=THROTTLE_DONE pgs=2951 cs=0 l=1 rx=0x7f8b3411bf50 tx=0x7f8b1c010db0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:57.305 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:57.300+0000 7f8b2bfff700 1 -- 172.21.15.42:0/4275302609 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 354 v354) v1 ==== 95+0+2111 (secure 0 0 0) 0x7f8b1c032880 con 0x7f8b3411a510 2019-12-05T06:39:57.308 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:39:57.308 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":354,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":11868,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/406829134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":406829134},{"type":"v1","addr":"172.21.15.42:6835","nonce":406829134}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":354}],"filesystems":[{"mdsmap":{"epoch":353,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:39:37.486362+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":113,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8009},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8009":{"gid":8009,"name":"b","rank":0,"incarnation":349,"state":"up:active","state_seq":27,"addr":"172.21.15.40:6833/1777669201","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1777669201},{"type":"v1","addr":"172.21.15.40:6833","nonce":1777669201}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:39:57.308 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 354 2019-12-05T06:39:57.310 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:57.304+0000 7f8b392ac700 1 -- 172.21.15.42:0/4275302609 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8b1803ae10 msgr2=0x7f8b1803d320 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:57.311 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:57.304+0000 7f8b392ac700 1 --2- 172.21.15.42:0/4275302609 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8b1803ae10 0x7f8b1803d320 crc :-1 s=READY pgs=1237 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:57.311 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:57.304+0000 7f8b392ac700 1 -- 172.21.15.42:0/4275302609 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8b3411a510 msgr2=0x7f8b3407d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:57.311 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:57.304+0000 7f8b392ac700 1 --2- 172.21.15.42:0/4275302609 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8b3411a510 0x7f8b3407d7b0 secure :-1 s=READY pgs=2951 cs=0 l=1 rx=0x7f8b3411bf50 tx=0x7f8b1c010db0).stop 2019-12-05T06:39:57.311 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:57.304+0000 7f8b392ac700 1 -- 172.21.15.42:0/4275302609 shutdown_connections 2019-12-05T06:39:57.312 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:57.304+0000 7f8b392ac700 1 --2- 172.21.15.42:0/4275302609 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8b1803ae10 0x7f8b1803d320 unknown :-1 s=CLOSED pgs=1237 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:57.312 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:57.304+0000 7f8b392ac700 1 --2- 172.21.15.42:0/4275302609 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8b34117610 0x7f8b34117a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:57.312 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:57.304+0000 7f8b392ac700 1 --2- 172.21.15.42:0/4275302609 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8b3411a510 0x7f8b3407d7b0 unknown :-1 s=CLOSED pgs=2951 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:57.312 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:57.304+0000 7f8b392ac700 1 --2- 172.21.15.42:0/4275302609 >> v2:172.21.15.40:3300/0 conn(0x7f8b34121320 0x7f8b3407dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:57.313 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:57.304+0000 7f8b392ac700 1 -- 172.21.15.42:0/4275302609 shutdown_connections 2019-12-05T06:39:57.313 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:57.304+0000 7f8b392ac700 1 -- 172.21.15.42:0/4275302609 wait complete. 2019-12-05T06:39:57.313 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:57.304+0000 7f8b392ac700 1 -- 172.21.15.42:0/4275302609 >> 172.21.15.42:0/4275302609 conn(0x7f8b3407e580 msgr2=0x7f8b340a9780 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:39:57.321 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:39:59.323 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:39:59.428 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:59.416+0000 7f054def3700 1 Processor -- start 2019-12-05T06:39:59.428 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:59.416+0000 7f054def3700 1 -- start start 2019-12-05T06:39:59.428 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:59.420+0000 7f054def3700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f05480a7740 0x7f054808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:39:59.429 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:59.420+0000 7f054def3700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05480a0710 con 0x7f054809fae0 2019-12-05T06:39:59.429 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:59.420+0000 7f054def3700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05480a9ba0 con 0x7f054807e720 2019-12-05T06:39:59.430 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:59.420+0000 7f054def3700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05480a0b00 con 0x7f05480a7740 2019-12-05T06:39:59.430 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:59.420+0000 7f05477fe700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f054807e720 0x7f054809f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:60534/0 (socket says 172.21.15.42:60534) 2019-12-05T06:39:59.431 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:59.420+0000 7f05477fe700 1 -- 172.21.15.42:0/970278298 learned_addr learned my addr 172.21.15.42:0/970278298 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:39:59.431 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:59.420+0000 7f0547fff700 1 --2- 172.21.15.42:0/970278298 >> v2:172.21.15.42:3300/0 conn(0x7f05480a7740 0x7f054808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:59.431 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:59.420+0000 7f0547fff700 1 -- 172.21.15.42:0/970278298 >> v1:172.21.15.42:6789/0 conn(0x7f054807e720 legacy=0x7f054809f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:59.431 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:59.420+0000 7f0547fff700 1 -- 172.21.15.42:0/970278298 >> v1:172.21.15.40:6789/0 conn(0x7f054809fae0 legacy=0x7f054814efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:59.431 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:59.420+0000 7f0547fff700 1 -- 172.21.15.42:0/970278298 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f054807eb50 con 0x7f05480a7740 2019-12-05T06:39:59.431 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:59.420+0000 7f0547fff700 1 --2- 172.21.15.42:0/970278298 >> v2:172.21.15.42:3300/0 conn(0x7f05480a7740 0x7f054808df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f0538008b10 tx=0x7f053800ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:59.431 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:59.420+0000 7f0547fff700 1 --2- 172.21.15.42:0/970278298 >> v2:172.21.15.42:3300/0 conn(0x7f05480a7740 0x7f054808df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f0538008b10 tx=0x7f053800ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:59.432 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:59.420+0000 7f0547fff700 1 --2- 172.21.15.42:0/970278298 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f05480a7740 0x7f054808df80 secure :-1 s=READY pgs=2953 cs=0 l=1 rx=0x7f0538008b10 tx=0x7f053800ff70).ready entity=mon.1 client_cookie=80313b84bfb250d6 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:39:59.432 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:59.420+0000 7f0547fff700 1 --2- 172.21.15.42:0/970278298 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f05480a7740 0x7f054808df80 secure :-1 s=THROTTLE_DONE pgs=2953 cs=0 l=1 rx=0x7f0538008b10 tx=0x7f053800ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:59.432 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:59.420+0000 7f0546ffd700 1 -- 172.21.15.42:0/970278298 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f05380119d0 con 0x7f05480a7740 2019-12-05T06:39:59.432 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:59.420+0000 7f0547fff700 1 --2- 172.21.15.42:0/970278298 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f05480a7740 0x7f054808df80 secure :-1 s=THROTTLE_DONE pgs=2953 cs=0 l=1 rx=0x7f0538008b10 tx=0x7f053800ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:59.432 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:59.420+0000 7f0547fff700 1 --2- 172.21.15.42:0/970278298 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f05480a7740 0x7f054808df80 secure :-1 s=THROTTLE_DONE pgs=2953 cs=0 l=1 rx=0x7f0538008b10 tx=0x7f053800ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:59.432 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:59.420+0000 7f0546ffd700 1 -- 172.21.15.42:0/970278298 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05380115f0 con 0x7f05480a7740 2019-12-05T06:39:59.433 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:59.420+0000 7f0546ffd700 1 -- 172.21.15.42:0/970278298 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f053800a160 con 0x7f05480a7740 2019-12-05T06:39:59.433 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:59.420+0000 7f054def3700 1 -- 172.21.15.42:0/970278298 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f05480a7740 msgr2=0x7f054808df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:59.433 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:59.420+0000 7f054def3700 1 --2- 172.21.15.42:0/970278298 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f05480a7740 0x7f054808df80 secure :-1 s=READY pgs=2953 cs=0 l=1 rx=0x7f0538008b10 tx=0x7f053800ff70).stop 2019-12-05T06:39:59.433 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:59.420+0000 7f054def3700 1 -- 172.21.15.42:0/970278298 shutdown_connections 2019-12-05T06:39:59.433 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:59.420+0000 7f054def3700 1 --2- 172.21.15.42:0/970278298 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f05480a7740 0x7f054808df80 unknown :-1 s=CLOSED pgs=2953 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:59.433 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:59.420+0000 7f054def3700 1 -- 172.21.15.42:0/970278298 shutdown_connections 2019-12-05T06:39:59.433 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:59.420+0000 7f054def3700 1 -- 172.21.15.42:0/970278298 wait complete. 2019-12-05T06:39:59.434 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:59.420+0000 7f054def3700 1 -- 172.21.15.42:0/970278298 >> 172.21.15.42:0/970278298 conn(0x7f05480aabe0 msgr2=0x7f05480a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:39:59.438 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:59.424+0000 7f054def3700 1 Processor -- start 2019-12-05T06:39:59.438 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:59.424+0000 7f054def3700 1 -- start start 2019-12-05T06:39:59.438 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:59.424+0000 7f054def3700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f0548123880 0x7f054805a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:39:59.439 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:59.424+0000 7f054def3700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f054812bce0 con 0x7f0548124060 2019-12-05T06:39:59.439 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:59.424+0000 7f054def3700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f054812cf20 con 0x7f0548124490 2019-12-05T06:39:59.439 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:59.424+0000 7f054def3700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0548115f00 con 0x7f0548123880 2019-12-05T06:39:59.439 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:59.424+0000 7f054cc91700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f0548124490 0x7f0548114470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:60544/0 (socket says 172.21.15.42:60544) 2019-12-05T06:39:59.439 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:59.424+0000 7f054cc91700 1 -- 172.21.15.42:0/1043681058 learned_addr learned my addr 172.21.15.42:0/1043681058 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:39:59.440 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:59.424+0000 7f0547fff700 1 --2- 172.21.15.42:0/1043681058 >> v2:172.21.15.40:3300/0 conn(0x7f0548123880 0x7f054805a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:59.440 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:59.424+0000 7f0544ff9700 1 -- 172.21.15.42:0/1043681058 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 376393946 0 0) 0x7f054812cf20 con 0x7f0548124490 2019-12-05T06:39:59.440 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:59.424+0000 7f0544ff9700 1 -- 172.21.15.42:0/1043681058 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f052c0030b0 con 0x7f0548124490 2019-12-05T06:39:59.440 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:59.424+0000 7f0544ff9700 1 -- 172.21.15.42:0/1043681058 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 765318401 0 0) 0x7f052c0030b0 con 0x7f0548124490 2019-12-05T06:39:59.440 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:59.424+0000 7f0544ff9700 1 -- 172.21.15.42:0/1043681058 --> v1:172.21.15.42:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f054812cf20 con 0x7f0548124490 2019-12-05T06:39:59.440 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:59.424+0000 7f0544ff9700 1 -- 172.21.15.42:0/1043681058 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7f0540001920 con 0x7f0548124490 2019-12-05T06:39:59.441 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:59.424+0000 7f0544ff9700 1 -- 172.21.15.42:0/1043681058 >> v2:172.21.15.40:3300/0 conn(0x7f0548123880 msgr2=0x7f054805a070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:59.441 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:59.424+0000 7f0544ff9700 1 --2- 172.21.15.42:0/1043681058 >> v2:172.21.15.40:3300/0 conn(0x7f0548123880 0x7f054805a070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:59.441 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:59.424+0000 7f0544ff9700 1 -- 172.21.15.42:0/1043681058 >> v1:172.21.15.42:6789/0 conn(0x7f0548124490 legacy=0x7f0548114470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:59.441 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:59.424+0000 7f0544ff9700 1 -- 172.21.15.42:0/1043681058 >> v1:172.21.15.40:6789/0 conn(0x7f0548124060 legacy=0x7f054807b770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:59.441 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:59.424+0000 7f0544ff9700 1 --2- 172.21.15.42:0/1043681058 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f052c0038a0 0x7f052c003d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:39:59.441 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:59.424+0000 7f0544ff9700 1 --2- 172.21.15.42:0/1043681058 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f052c004420 0x7f052c004850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:39:59.441 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:59.424+0000 7f0544ff9700 1 --2- 172.21.15.42:0/1043681058 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f052c004eb0 0x7f052c00b5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:39:59.442 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:59.424+0000 7f0544ff9700 1 -- 172.21.15.42:0/1043681058 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0548115f00 con 0x7f052c004eb0 2019-12-05T06:39:59.442 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:59.424+0000 7f0544ff9700 1 -- 172.21.15.42:0/1043681058 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f052c00bbe0 con 0x7f052c0038a0 2019-12-05T06:39:59.442 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:59.424+0000 7f0544ff9700 1 -- 172.21.15.42:0/1043681058 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f052c00be20 con 0x7f052c004420 2019-12-05T06:39:59.442 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:59.424+0000 7f0547fff700 1 --2- 172.21.15.42:0/1043681058 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f052c0038a0 0x7f052c003d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:59.442 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:59.424+0000 7f054cc91700 1 --2- 172.21.15.42:0/1043681058 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f052c004eb0 0x7f052c00b5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:59.442 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:59.424+0000 7f05477fe700 1 --2- 172.21.15.42:0/1043681058 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f052c004420 0x7f052c004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:59.442 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:59.424+0000 7f0547fff700 1 -- 172.21.15.42:0/1043681058 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f052c004420 msgr2=0x7f052c004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:59.442 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:59.424+0000 7f0547fff700 1 --2- 172.21.15.42:0/1043681058 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f052c004420 0x7f052c004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:59.443 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:59.428+0000 7f0547fff700 1 -- 172.21.15.42:0/1043681058 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f052c004eb0 msgr2=0x7f052c00b5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:59.443 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:59.428+0000 7f0547fff700 1 --2- 172.21.15.42:0/1043681058 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f052c004eb0 0x7f052c00b5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:59.443 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:59.428+0000 7f0547fff700 1 -- 172.21.15.42:0/1043681058 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f054812e160 con 0x7f052c0038a0 2019-12-05T06:39:59.443 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:59.428+0000 7f054def3700 1 -- 172.21.15.42:0/1043681058 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f054812bf10 con 0x7f052c0038a0 2019-12-05T06:39:59.443 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:59.428+0000 7f054cc91700 1 --2- 172.21.15.42:0/1043681058 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f052c004eb0 0x7f052c00b5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:39:59.443 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:59.428+0000 7f054cc91700 1 -- 172.21.15.42:0/1043681058 reap_dead start 2019-12-05T06:39:59.444 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:59.428+0000 7f054def3700 1 -- 172.21.15.42:0/1043681058 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0548123880 con 0x7f052c0038a0 2019-12-05T06:39:59.444 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:59.428+0000 7f0547fff700 1 --2- 172.21.15.42:0/1043681058 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f052c0038a0 0x7f052c003d10 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f0538008b10 tx=0x7f053800f880).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:59.444 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:59.428+0000 7f0547fff700 1 --2- 172.21.15.42:0/1043681058 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f052c0038a0 0x7f052c003d10 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f0538008b10 tx=0x7f053800f880).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:59.444 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:59.428+0000 7f0547fff700 1 --2- 172.21.15.42:0/1043681058 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f052c0038a0 0x7f052c003d10 secure :-1 s=READY pgs=2955 cs=0 l=1 rx=0x7f0538008b10 tx=0x7f053800f880).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:39:59.445 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:59.428+0000 7f0547fff700 1 --2- 172.21.15.42:0/1043681058 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f052c0038a0 0x7f052c003d10 secure :-1 s=THROTTLE_DONE pgs=2955 cs=0 l=1 rx=0x7f0538008b10 tx=0x7f053800f880).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:59.445 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:59.428+0000 7f0544ff9700 1 -- 172.21.15.42:0/1043681058 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0538005ab0 con 0x7f052c0038a0 2019-12-05T06:39:59.445 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:59.428+0000 7f0547fff700 1 --2- 172.21.15.42:0/1043681058 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f052c0038a0 0x7f052c003d10 secure :-1 s=THROTTLE_DONE pgs=2955 cs=0 l=1 rx=0x7f0538008b10 tx=0x7f053800f880).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:59.445 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:59.428+0000 7f0544ff9700 1 -- 172.21.15.42:0/1043681058 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0538011bd0 con 0x7f052c0038a0 2019-12-05T06:39:59.445 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:59.428+0000 7f0547fff700 1 --2- 172.21.15.42:0/1043681058 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f052c0038a0 0x7f052c003d10 secure :-1 s=THROTTLE_DONE pgs=2955 cs=0 l=1 rx=0x7f0538008b10 tx=0x7f053800f880).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:59.445 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:59.428+0000 7f0544ff9700 1 -- 172.21.15.42:0/1043681058 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0538007340 con 0x7f052c0038a0 2019-12-05T06:39:59.445 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:59.428+0000 7f0547fff700 1 --2- 172.21.15.42:0/1043681058 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f052c0038a0 0x7f052c003d10 secure :-1 s=THROTTLE_DONE pgs=2955 cs=0 l=1 rx=0x7f0538008b10 tx=0x7f053800f880).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:59.446 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:59.428+0000 7f0544ff9700 1 -- 172.21.15.42:0/1043681058 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f0538007540 con 0x7f052c0038a0 2019-12-05T06:39:59.446 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:59.428+0000 7f0547fff700 1 --2- 172.21.15.42:0/1043681058 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f052c0038a0 0x7f052c003d10 secure :-1 s=THROTTLE_DONE pgs=2955 cs=0 l=1 rx=0x7f0538008b10 tx=0x7f053800f880).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:59.446 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:59.428+0000 7f0544ff9700 1 --2- 172.21.15.42:0/1043681058 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f052c03e950 0x7f052c040e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:39:59.447 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:59.428+0000 7f0544ff9700 1 -- 172.21.15.42:0/1043681058 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 11535+0+0 (secure 0 0 0) 0x7f053804b850 con 0x7f052c0038a0 2019-12-05T06:39:59.447 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:59.428+0000 7f05477fe700 1 --2- 172.21.15.42:0/1043681058 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f052c03e950 0x7f052c040e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:59.447 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:59.428+0000 7f05477fe700 1 --2- 172.21.15.42:0/1043681058 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f052c03e950 0x7f052c040e60 crc :-1 s=READY pgs=1238 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:39:59.448 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:59.432+0000 7f054def3700 1 -- 172.21.15.42:0/1043681058 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0520003200 con 0x7f052c0038a0 2019-12-05T06:39:59.448 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:59.440+0000 7f0547fff700 1 --2- 172.21.15.42:0/1043681058 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f052c0038a0 0x7f052c003d10 secure :-1 s=THROTTLE_DONE pgs=2955 cs=0 l=1 rx=0x7f0538008b10 tx=0x7f053800f880).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:59.448 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:59.440+0000 7f0544ff9700 1 -- 172.21.15.42:0/1043681058 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f05380328b0 con 0x7f052c0038a0 2019-12-05T06:39:59.618 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:59.612+0000 7f054def3700 1 -- 172.21.15.42:0/1043681058 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0520003630 con 0x7f052c0038a0 2019-12-05T06:39:59.619 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:59.612+0000 7f0547fff700 1 --2- 172.21.15.42:0/1043681058 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f052c0038a0 0x7f052c003d10 secure :-1 s=THROTTLE_DONE pgs=2955 cs=0 l=1 rx=0x7f0538008b10 tx=0x7f053800f880).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:59.620 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:59.612+0000 7f0544ff9700 1 -- 172.21.15.42:0/1043681058 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 354 v354) v1 ==== 95+0+2111 (secure 0 0 0) 0x7f0538032b10 con 0x7f052c0038a0 2019-12-05T06:39:59.620 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 354 2019-12-05T06:39:59.620 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:39:59.620 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":354,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":11868,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/406829134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":406829134},{"type":"v1","addr":"172.21.15.42:6835","nonce":406829134}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":354}],"filesystems":[{"mdsmap":{"epoch":353,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:39:37.486362+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":113,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8009},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8009":{"gid":8009,"name":"b","rank":0,"incarnation":349,"state":"up:active","state_seq":27,"addr":"172.21.15.40:6833/1777669201","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1777669201},{"type":"v1","addr":"172.21.15.40:6833","nonce":1777669201}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:39:59.622 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:59.616+0000 7f054def3700 1 -- 172.21.15.42:0/1043681058 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f052c03e950 msgr2=0x7f052c040e60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:59.623 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:59.616+0000 7f054def3700 1 --2- 172.21.15.42:0/1043681058 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f052c03e950 0x7f052c040e60 crc :-1 s=READY pgs=1238 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:59.623 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:59.616+0000 7f054def3700 1 -- 172.21.15.42:0/1043681058 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f052c0038a0 msgr2=0x7f052c003d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:59.623 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:59.616+0000 7f054def3700 1 --2- 172.21.15.42:0/1043681058 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f052c0038a0 0x7f052c003d10 secure :-1 s=READY pgs=2955 cs=0 l=1 rx=0x7f0538008b10 tx=0x7f053800f880).stop 2019-12-05T06:39:59.623 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:59.616+0000 7f054def3700 1 -- 172.21.15.42:0/1043681058 shutdown_connections 2019-12-05T06:39:59.624 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:59.616+0000 7f054def3700 1 --2- 172.21.15.42:0/1043681058 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f052c03e950 0x7f052c040e60 unknown :-1 s=CLOSED pgs=1238 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:59.624 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:59.616+0000 7f054def3700 1 --2- 172.21.15.42:0/1043681058 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f052c0038a0 0x7f052c003d10 unknown :-1 s=CLOSED pgs=2955 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:59.624 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:59.616+0000 7f054def3700 1 -- 172.21.15.42:0/1043681058 shutdown_connections 2019-12-05T06:39:59.624 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:59.616+0000 7f054def3700 1 -- 172.21.15.42:0/1043681058 wait complete. 2019-12-05T06:39:59.624 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:39:59.616+0000 7f054def3700 1 -- 172.21.15.42:0/1043681058 >> 172.21.15.42:0/1043681058 conn(0x7f05480a7f90 msgr2=0x7f054807da50 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:39:59.632 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:40:01.633 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:40:01.741 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:01.728+0000 7f569da43700 1 Processor -- start 2019-12-05T06:40:01.741 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:01.728+0000 7f569da43700 1 -- start start 2019-12-05T06:40:01.741 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:01.732+0000 7f569da43700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f569808df80 0x7f569809f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:40:01.741 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:01.732+0000 7f569da43700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f569809fc60 con 0x7f56980a7740 2019-12-05T06:40:01.741 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:01.732+0000 7f569da43700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56980a0710 con 0x7f569809f830 2019-12-05T06:40:01.742 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:01.732+0000 7f569da43700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56980a0b00 con 0x7f569808df80 2019-12-05T06:40:01.742 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:01.732+0000 7f5696ffd700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f569808df80 0x7f569809f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:40:01.742 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:01.732+0000 7f5697fff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f569809f830 0x7f569814efe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:60562/0 (socket says 172.21.15.42:60562) 2019-12-05T06:40:01.742 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:01.732+0000 7f5697fff700 1 -- 172.21.15.42:0/4238778397 learned_addr learned my addr 172.21.15.42:0/4238778397 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:40:01.742 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:01.732+0000 7f56967fc700 1 -- 172.21.15.42:0/4238778397 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 575406635 0 0) 0x7f569809fc60 con 0x7f56980a7740 2019-12-05T06:40:01.742 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:01.732+0000 7f5696ffd700 1 -- 172.21.15.42:0/4238778397 >> v1:172.21.15.42:6789/0 conn(0x7f569809f830 legacy=0x7f569814efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:01.743 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:01.732+0000 7f5696ffd700 1 -- 172.21.15.42:0/4238778397 >> v1:172.21.15.40:6789/0 conn(0x7f56980a7740 legacy=0x7f569807e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:01.743 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:01.732+0000 7f5696ffd700 1 -- 172.21.15.42:0/4238778397 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f569807ef20 con 0x7f569808df80 2019-12-05T06:40:01.743 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:01.732+0000 7f5696ffd700 1 --2- 172.21.15.42:0/4238778397 >> v2:172.21.15.42:3300/0 conn(0x7f569808df80 0x7f569809f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f5680008790 tx=0x7f5688001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:01.743 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:01.732+0000 7f5696ffd700 1 --2- 172.21.15.42:0/4238778397 >> v2:172.21.15.42:3300/0 conn(0x7f569808df80 0x7f569809f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f5680008790 tx=0x7f5688001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:01.743 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:01.732+0000 7f5696ffd700 1 --2- 172.21.15.42:0/4238778397 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f569808df80 0x7f569809f2e0 secure :-1 s=READY pgs=2961 cs=0 l=1 rx=0x7f5680008790 tx=0x7f5688001150).ready entity=mon.1 client_cookie=d6070ebc7a982404 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:40:01.743 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:01.732+0000 7f5696ffd700 1 --2- 172.21.15.42:0/4238778397 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f569808df80 0x7f569809f2e0 secure :-1 s=THROTTLE_DONE pgs=2961 cs=0 l=1 rx=0x7f5680008790 tx=0x7f5688001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:01.743 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:01.732+0000 7f56967fc700 1 -- 172.21.15.42:0/4238778397 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5680008bb0 con 0x7f569808df80 2019-12-05T06:40:01.744 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:01.732+0000 7f5696ffd700 1 --2- 172.21.15.42:0/4238778397 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f569808df80 0x7f569809f2e0 secure :-1 s=THROTTLE_DONE pgs=2961 cs=0 l=1 rx=0x7f5680008790 tx=0x7f5688001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:01.744 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:01.732+0000 7f5696ffd700 1 --2- 172.21.15.42:0/4238778397 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f569808df80 0x7f569809f2e0 secure :-1 s=THROTTLE_DONE pgs=2961 cs=0 l=1 rx=0x7f5680008790 tx=0x7f5688001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:01.744 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:01.732+0000 7f56967fc700 1 -- 172.21.15.42:0/4238778397 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56800087d0 con 0x7f569808df80 2019-12-05T06:40:01.744 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:01.732+0000 7f56967fc700 1 -- 172.21.15.42:0/4238778397 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f56800117d0 con 0x7f569808df80 2019-12-05T06:40:01.744 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:01.732+0000 7f569da43700 1 -- 172.21.15.42:0/4238778397 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f569808df80 msgr2=0x7f569809f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:01.744 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:01.732+0000 7f569da43700 1 --2- 172.21.15.42:0/4238778397 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f569808df80 0x7f569809f2e0 secure :-1 s=READY pgs=2961 cs=0 l=1 rx=0x7f5680008790 tx=0x7f5688001150).stop 2019-12-05T06:40:01.744 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:01.732+0000 7f569da43700 1 -- 172.21.15.42:0/4238778397 shutdown_connections 2019-12-05T06:40:01.745 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:01.736+0000 7f569da43700 1 --2- 172.21.15.42:0/4238778397 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f569808df80 0x7f569809f2e0 unknown :-1 s=CLOSED pgs=2961 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:40:01.745 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:01.736+0000 7f569da43700 1 -- 172.21.15.42:0/4238778397 shutdown_connections 2019-12-05T06:40:01.745 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:01.736+0000 7f569da43700 1 -- 172.21.15.42:0/4238778397 wait complete. 2019-12-05T06:40:01.745 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:01.736+0000 7f569da43700 1 -- 172.21.15.42:0/4238778397 >> 172.21.15.42:0/4238778397 conn(0x7f56980aabe0 msgr2=0x7f56980a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:40:01.748 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:01.736+0000 7f569da43700 1 Processor -- start 2019-12-05T06:40:01.748 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:01.736+0000 7f569da43700 1 -- start start 2019-12-05T06:40:01.748 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:01.740+0000 7f569da43700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f56980a1dc0 0x7f569811aed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:40:01.748 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:01.740+0000 7f569da43700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56980a57d0 con 0x7f56980a3b30 2019-12-05T06:40:01.749 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:01.740+0000 7f569da43700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56980a4390 con 0x7f56980a3f60 2019-12-05T06:40:01.749 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:01.740+0000 7f569da43700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56980a2550 con 0x7f56980a1dc0 2019-12-05T06:40:01.749 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:01.740+0000 7f56977fe700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f56980a1dc0 0x7f569811aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:40:01.749 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:01.740+0000 7f56977fe700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f56980a1dc0 0x7f569811aed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.42:48342/0 (socket says 172.21.15.42:48342) 2019-12-05T06:40:01.749 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:01.740+0000 7f56977fe700 1 -- 172.21.15.42:0/2250127523 learned_addr learned my addr 172.21.15.42:0/2250127523 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:40:01.750 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:01.740+0000 7f568ffff700 1 -- 172.21.15.42:0/2250127523 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2081116200 0 0) 0x7f56980a4390 con 0x7f56980a3f60 2019-12-05T06:40:01.750 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:01.740+0000 7f568ffff700 1 -- 172.21.15.42:0/2250127523 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f567c0030b0 con 0x7f56980a3f60 2019-12-05T06:40:01.750 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:01.740+0000 7f56977fe700 1 -- 172.21.15.42:0/2250127523 >> v1:172.21.15.42:6789/0 conn(0x7f56980a3f60 legacy=0x7f5698130aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:01.751 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:01.740+0000 7f568ffff700 1 -- 172.21.15.42:0/2250127523 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 791328577 0 0) 0x7f56980a57d0 con 0x7f56980a3b30 2019-12-05T06:40:01.751 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:01.740+0000 7f56977fe700 1 -- 172.21.15.42:0/2250127523 >> v1:172.21.15.40:6789/0 conn(0x7f56980a3b30 legacy=0x7f56981302a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:01.751 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:01.740+0000 7f56977fe700 1 -- 172.21.15.42:0/2250127523 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f569811b420 con 0x7f56980a1dc0 2019-12-05T06:40:01.751 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:01.740+0000 7f56977fe700 1 --2- 172.21.15.42:0/2250127523 >> v2:172.21.15.40:3300/0 conn(0x7f56980a1dc0 0x7f569811aed0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f5690001210 tx=0x7f5690010c70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:01.751 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:01.740+0000 7f56977fe700 1 --2- 172.21.15.42:0/2250127523 >> v2:172.21.15.40:3300/0 conn(0x7f56980a1dc0 0x7f569811aed0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f5690001210 tx=0x7f5690010c70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:01.751 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:01.740+0000 7f56977fe700 1 --2- 172.21.15.42:0/2250127523 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f56980a1dc0 0x7f569811aed0 secure :-1 s=READY pgs=2247 cs=0 l=1 rx=0x7f5690001210 tx=0x7f5690010c70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:40:01.751 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:01.740+0000 7f56977fe700 1 --2- 172.21.15.42:0/2250127523 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f56980a1dc0 0x7f569811aed0 secure :-1 s=THROTTLE_DONE pgs=2247 cs=0 l=1 rx=0x7f5690001210 tx=0x7f5690010c70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:01.752 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:01.740+0000 7f568ffff700 1 -- 172.21.15.42:0/2250127523 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f569000ca40 con 0x7f56980a1dc0 2019-12-05T06:40:01.752 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:01.740+0000 7f56977fe700 1 --2- 172.21.15.42:0/2250127523 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f56980a1dc0 0x7f569811aed0 secure :-1 s=THROTTLE_DONE pgs=2247 cs=0 l=1 rx=0x7f5690001210 tx=0x7f5690010c70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:01.752 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:01.740+0000 7f569da43700 1 -- 172.21.15.42:0/2250127523 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56980a1510 con 0x7f56980a1dc0 2019-12-05T06:40:01.752 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:01.740+0000 7f568ffff700 1 -- 172.21.15.42:0/2250127523 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56900101c0 con 0x7f56980a1dc0 2019-12-05T06:40:01.752 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:01.740+0000 7f56977fe700 1 --2- 172.21.15.42:0/2250127523 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f56980a1dc0 0x7f569811aed0 secure :-1 s=THROTTLE_DONE pgs=2247 cs=0 l=1 rx=0x7f5690001210 tx=0x7f5690010c70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:01.752 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:01.740+0000 7f568ffff700 1 -- 172.21.15.42:0/2250127523 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f56900124e0 con 0x7f56980a1dc0 2019-12-05T06:40:01.752 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:01.740+0000 7f569da43700 1 -- 172.21.15.42:0/2250127523 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56980a21f0 con 0x7f56980a1dc0 2019-12-05T06:40:01.753 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:01.740+0000 7f56977fe700 1 --2- 172.21.15.42:0/2250127523 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f56980a1dc0 0x7f569811aed0 secure :-1 s=THROTTLE_DONE pgs=2247 cs=0 l=1 rx=0x7f5690001210 tx=0x7f5690010c70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:01.754 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:01.744+0000 7f568ffff700 1 -- 172.21.15.42:0/2250127523 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f56900127c0 con 0x7f56980a1dc0 2019-12-05T06:40:01.754 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:01.744+0000 7f56977fe700 1 --2- 172.21.15.42:0/2250127523 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f56980a1dc0 0x7f569811aed0 secure :-1 s=THROTTLE_DONE pgs=2247 cs=0 l=1 rx=0x7f5690001210 tx=0x7f5690010c70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:01.754 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:01.744+0000 7f568ffff700 1 --2- 172.21.15.42:0/2250127523 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f567c03b1c0 0x7f567c03d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:40:01.754 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:01.744+0000 7f568ffff700 1 -- 172.21.15.42:0/2250127523 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 11535+0+0 (secure 0 0 0) 0x7f569004b350 con 0x7f56980a1dc0 2019-12-05T06:40:01.755 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:01.744+0000 7f5696ffd700 1 --2- 172.21.15.42:0/2250127523 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f567c03b1c0 0x7f567c03d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:40:01.755 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:01.744+0000 7f5696ffd700 1 --2- 172.21.15.42:0/2250127523 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f567c03b1c0 0x7f567c03d6d0 crc :-1 s=READY pgs=1239 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:40:01.755 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:01.744+0000 7f569da43700 1 -- 172.21.15.42:0/2250127523 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f569805d110 con 0x7f56980a1dc0 2019-12-05T06:40:01.761 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:01.756+0000 7f56977fe700 1 --2- 172.21.15.42:0/2250127523 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f56980a1dc0 0x7f569811aed0 secure :-1 s=THROTTLE_DONE pgs=2247 cs=0 l=1 rx=0x7f5690001210 tx=0x7f5690010c70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:01.761 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:01.756+0000 7f568ffff700 1 -- 172.21.15.42:0/2250127523 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f56900323b0 con 0x7f56980a1dc0 2019-12-05T06:40:01.936 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:01.928+0000 7f569da43700 1 -- 172.21.15.42:0/2250127523 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f569805b8c0 con 0x7f56980a1dc0 2019-12-05T06:40:01.937 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:01.932+0000 7f56977fe700 1 --2- 172.21.15.42:0/2250127523 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f56980a1dc0 0x7f569811aed0 secure :-1 s=THROTTLE_DONE pgs=2247 cs=0 l=1 rx=0x7f5690001210 tx=0x7f5690010c70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:01.937 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:01.932+0000 7f568ffff700 1 -- 172.21.15.42:0/2250127523 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 354 v354) v1 ==== 95+0+2111 (secure 0 0 0) 0x7f5690032610 con 0x7f56980a1dc0 2019-12-05T06:40:01.939 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:40:01.939 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":354,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":11868,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/406829134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":406829134},{"type":"v1","addr":"172.21.15.42:6835","nonce":406829134}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":354}],"filesystems":[{"mdsmap":{"epoch":353,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:39:37.486362+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":113,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8009},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8009":{"gid":8009,"name":"b","rank":0,"incarnation":349,"state":"up:active","state_seq":27,"addr":"172.21.15.40:6833/1777669201","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1777669201},{"type":"v1","addr":"172.21.15.40:6833","nonce":1777669201}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:40:01.939 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 354 2019-12-05T06:40:01.942 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:01.936+0000 7f569da43700 1 -- 172.21.15.42:0/2250127523 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f567c03b1c0 msgr2=0x7f567c03d6d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:01.942 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:01.936+0000 7f569da43700 1 --2- 172.21.15.42:0/2250127523 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f567c03b1c0 0x7f567c03d6d0 crc :-1 s=READY pgs=1239 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:40:01.942 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:01.936+0000 7f569da43700 1 -- 172.21.15.42:0/2250127523 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f56980a1dc0 msgr2=0x7f569811aed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:01.942 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:01.936+0000 7f569da43700 1 --2- 172.21.15.42:0/2250127523 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f56980a1dc0 0x7f569811aed0 secure :-1 s=READY pgs=2247 cs=0 l=1 rx=0x7f5690001210 tx=0x7f5690010c70).stop 2019-12-05T06:40:01.942 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:01.936+0000 7f569da43700 1 -- 172.21.15.42:0/2250127523 shutdown_connections 2019-12-05T06:40:01.943 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:01.936+0000 7f569da43700 1 --2- 172.21.15.42:0/2250127523 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f567c03b1c0 0x7f567c03d6d0 unknown :-1 s=CLOSED pgs=1239 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:40:01.943 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:01.936+0000 7f569da43700 1 --2- 172.21.15.42:0/2250127523 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f56980a1dc0 0x7f569811aed0 unknown :-1 s=CLOSED pgs=2247 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:40:01.943 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:01.936+0000 7f569da43700 1 -- 172.21.15.42:0/2250127523 shutdown_connections 2019-12-05T06:40:01.943 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:01.936+0000 7f569da43700 1 -- 172.21.15.42:0/2250127523 wait complete. 2019-12-05T06:40:01.944 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:01.936+0000 7f569da43700 1 -- 172.21.15.42:0/2250127523 >> 172.21.15.42:0/2250127523 conn(0x7f56980a82a0 msgr2=0x7f569811bed0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:40:01.952 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:40:03.954 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:40:04.070 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:04.052+0000 7f7a33f1c700 1 Processor -- start 2019-12-05T06:40:04.070 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:04.052+0000 7f7a33f1c700 1 -- start start 2019-12-05T06:40:04.070 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:04.052+0000 7f7a33f1c700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7a2c08df80 0x7f7a2c09f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:40:04.072 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:04.052+0000 7f7a33f1c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f7a2c09f830 0x7f7a2c14efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:40:04.073 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:04.052+0000 7f7a33f1c700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a2c09fc60 con 0x7f7a2c0a7740 2019-12-05T06:40:04.073 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:04.052+0000 7f7a33f1c700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a2c0a0b00 con 0x7f7a2c09f830 2019-12-05T06:40:04.073 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:04.052+0000 7f7a33f1c700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a2c08e5b0 con 0x7f7a2c08df80 2019-12-05T06:40:04.074 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:04.052+0000 7f7a324b9700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f7a2c0a7740 0x7f7a2c07e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:60574/0 (socket says 172.21.15.42:60574) 2019-12-05T06:40:04.074 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:04.052+0000 7f7a324b9700 1 -- 172.21.15.42:0/3189339508 learned_addr learned my addr 172.21.15.42:0/3189339508 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:40:04.075 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:04.052+0000 7f7a32cba700 1 --2- 172.21.15.42:0/3189339508 >> v2:172.21.15.42:3300/0 conn(0x7f7a2c09f830 0x7f7a2c14efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:40:04.075 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:04.052+0000 7f7a31cb8700 1 --2- 172.21.15.42:0/3189339508 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7a2c08df80 0x7f7a2c09f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:40:04.075 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:04.052+0000 7f7a314b7700 1 -- 172.21.15.42:0/3189339508 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3047760305 0 0) 0x7f7a2c09fc60 con 0x7f7a2c0a7740 2019-12-05T06:40:04.075 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:04.052+0000 7f7a314b7700 1 -- 172.21.15.42:0/3189339508 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a180030b0 con 0x7f7a2c0a7740 2019-12-05T06:40:04.076 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:04.052+0000 7f7a32cba700 1 -- 172.21.15.42:0/3189339508 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7a2c08df80 msgr2=0x7f7a2c09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:40:04.076 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:04.052+0000 7f7a32cba700 1 --2- 172.21.15.42:0/3189339508 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7a2c08df80 0x7f7a2c09f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:40:04.076 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:04.052+0000 7f7a32cba700 1 -- 172.21.15.42:0/3189339508 >> v1:172.21.15.42:6789/0 conn(0x7f7a2c0a7740 legacy=0x7f7a2c07e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:04.076 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:04.052+0000 7f7a32cba700 1 -- 172.21.15.42:0/3189339508 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a2c0a0710 con 0x7f7a2c09f830 2019-12-05T06:40:04.077 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:04.052+0000 7f7a32cba700 1 --2- 172.21.15.42:0/3189339508 >> v2:172.21.15.42:3300/0 conn(0x7f7a2c09f830 0x7f7a2c14efe0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f7a24008b10 tx=0x7f7a2400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:04.077 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:04.052+0000 7f7a32cba700 1 --2- 172.21.15.42:0/3189339508 >> v2:172.21.15.42:3300/0 conn(0x7f7a2c09f830 0x7f7a2c14efe0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f7a24008b10 tx=0x7f7a2400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:04.078 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:04.052+0000 7f7a32cba700 1 --2- 172.21.15.42:0/3189339508 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7a2c09f830 0x7f7a2c14efe0 secure :-1 s=READY pgs=2965 cs=0 l=1 rx=0x7f7a24008b10 tx=0x7f7a2400fff0).ready entity=mon.1 client_cookie=a5e3e38611500e16 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:40:04.078 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:04.052+0000 7f7a32cba700 1 --2- 172.21.15.42:0/3189339508 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7a2c09f830 0x7f7a2c14efe0 secure :-1 s=THROTTLE_DONE pgs=2965 cs=0 l=1 rx=0x7f7a24008b10 tx=0x7f7a2400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:04.078 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:04.052+0000 7f7a314b7700 1 -- 172.21.15.42:0/3189339508 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7a24010900 con 0x7f7a2c09f830 2019-12-05T06:40:04.078 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:04.052+0000 7f7a32cba700 1 --2- 172.21.15.42:0/3189339508 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7a2c09f830 0x7f7a2c14efe0 secure :-1 s=THROTTLE_DONE pgs=2965 cs=0 l=1 rx=0x7f7a24008b10 tx=0x7f7a2400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:04.079 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:04.052+0000 7f7a32cba700 1 --2- 172.21.15.42:0/3189339508 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7a2c09f830 0x7f7a2c14efe0 secure :-1 s=THROTTLE_DONE pgs=2965 cs=0 l=1 rx=0x7f7a24008b10 tx=0x7f7a2400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:04.079 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:04.052+0000 7f7a314b7700 1 -- 172.21.15.42:0/3189339508 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a24008b50 con 0x7f7a2c09f830 2019-12-05T06:40:04.080 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:04.052+0000 7f7a314b7700 1 -- 172.21.15.42:0/3189339508 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7a240110c0 con 0x7f7a2c09f830 2019-12-05T06:40:04.080 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:04.052+0000 7f7a33f1c700 1 -- 172.21.15.42:0/3189339508 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7a2c09f830 msgr2=0x7f7a2c14efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:04.080 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:04.052+0000 7f7a33f1c700 1 --2- 172.21.15.42:0/3189339508 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7a2c09f830 0x7f7a2c14efe0 secure :-1 s=READY pgs=2965 cs=0 l=1 rx=0x7f7a24008b10 tx=0x7f7a2400fff0).stop 2019-12-05T06:40:04.094 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:04.056+0000 7f7a33f1c700 1 -- 172.21.15.42:0/3189339508 shutdown_connections 2019-12-05T06:40:04.094 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:04.056+0000 7f7a33f1c700 1 --2- 172.21.15.42:0/3189339508 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7a2c09f830 0x7f7a2c14efe0 unknown :-1 s=CLOSED pgs=2965 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:40:04.096 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:04.056+0000 7f7a33f1c700 1 --2- 172.21.15.42:0/3189339508 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7a2c08df80 0x7f7a2c09f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:40:04.096 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:04.056+0000 7f7a33f1c700 1 -- 172.21.15.42:0/3189339508 shutdown_connections 2019-12-05T06:40:04.096 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:04.056+0000 7f7a33f1c700 1 -- 172.21.15.42:0/3189339508 wait complete. 2019-12-05T06:40:04.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:04.056+0000 7f7a33f1c700 1 -- 172.21.15.42:0/3189339508 >> 172.21.15.42:0/3189339508 conn(0x7f7a2c0aabe0 msgr2=0x7f7a2c0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:40:04.100 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:04.056+0000 7f7a33f1c700 1 Processor -- start 2019-12-05T06:40:04.100 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:04.056+0000 7f7a33f1c700 1 -- start start 2019-12-05T06:40:04.108 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:04.060+0000 7f7a33f1c700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7a2c116700 0x7f7a2c116b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:40:04.109 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:04.060+0000 7f7a33f1c700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a2c127550 con 0x7f7a2c117dd0 2019-12-05T06:40:04.109 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:04.060+0000 7f7a33f1c700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a2c1304c0 con 0x7f7a2c08eeb0 2019-12-05T06:40:04.109 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:04.060+0000 7f7a33f1c700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a2c146450 con 0x7f7a2c116700 2019-12-05T06:40:04.109 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:04.060+0000 7f7a31cb8700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f7a2c08eeb0 0x7f7a2c126d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:60584/0 (socket says 172.21.15.42:60584) 2019-12-05T06:40:04.111 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:04.060+0000 7f7a31cb8700 1 -- 172.21.15.42:0/4194495651 learned_addr learned my addr 172.21.15.42:0/4194495651 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:40:04.112 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:04.060+0000 7f7a32cba700 1 --2- 172.21.15.42:0/4194495651 >> v2:172.21.15.40:3300/0 conn(0x7f7a2c116700 0x7f7a2c116b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:40:04.112 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:04.060+0000 7f7a22ffd700 1 -- 172.21.15.42:0/4194495651 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1680932562 0 0) 0x7f7a2c1304c0 con 0x7f7a2c08eeb0 2019-12-05T06:40:04.114 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:04.060+0000 7f7a22ffd700 1 -- 172.21.15.42:0/4194495651 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a140030b0 con 0x7f7a2c08eeb0 2019-12-05T06:40:04.114 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:04.060+0000 7f7a22ffd700 1 -- 172.21.15.42:0/4194495651 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2598789612 0 0) 0x7f7a2c127550 con 0x7f7a2c117dd0 2019-12-05T06:40:04.115 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:04.060+0000 7f7a22ffd700 1 -- 172.21.15.42:0/4194495651 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a2c1304c0 con 0x7f7a2c117dd0 2019-12-05T06:40:04.118 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:04.060+0000 7f7a32cba700 1 -- 172.21.15.42:0/4194495651 >> v1:172.21.15.42:6789/0 conn(0x7f7a2c08eeb0 legacy=0x7f7a2c126d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:04.118 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:04.060+0000 7f7a32cba700 1 -- 172.21.15.42:0/4194495651 >> v1:172.21.15.40:6789/0 conn(0x7f7a2c117dd0 legacy=0x7f7a2c145b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:04.118 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:04.060+0000 7f7a32cba700 1 -- 172.21.15.42:0/4194495651 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a2c13a940 con 0x7f7a2c116700 2019-12-05T06:40:04.118 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:04.060+0000 7f7a32cba700 1 --2- 172.21.15.42:0/4194495651 >> v2:172.21.15.40:3300/0 conn(0x7f7a2c116700 0x7f7a2c116b30 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f7a24008b10 tx=0x7f7a240112d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:04.120 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:04.060+0000 7f7a32cba700 1 --2- 172.21.15.42:0/4194495651 >> v2:172.21.15.40:3300/0 conn(0x7f7a2c116700 0x7f7a2c116b30 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f7a24008b10 tx=0x7f7a240112d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:04.120 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:04.060+0000 7f7a32cba700 1 --2- 172.21.15.42:0/4194495651 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7a2c116700 0x7f7a2c116b30 secure :-1 s=READY pgs=2252 cs=0 l=1 rx=0x7f7a24008b10 tx=0x7f7a240112d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:40:04.120 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:04.060+0000 7f7a32cba700 1 --2- 172.21.15.42:0/4194495651 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7a2c116700 0x7f7a2c116b30 secure :-1 s=THROTTLE_DONE pgs=2252 cs=0 l=1 rx=0x7f7a24008b10 tx=0x7f7a240112d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:04.121 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:04.060+0000 7f7a22ffd700 1 -- 172.21.15.42:0/4194495651 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7a24005ab0 con 0x7f7a2c116700 2019-12-05T06:40:04.121 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:04.060+0000 7f7a32cba700 1 --2- 172.21.15.42:0/4194495651 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7a2c116700 0x7f7a2c116b30 secure :-1 s=THROTTLE_DONE pgs=2252 cs=0 l=1 rx=0x7f7a24008b10 tx=0x7f7a240112d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:04.121 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:04.060+0000 7f7a22ffd700 1 -- 172.21.15.42:0/4194495651 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a24010b00 con 0x7f7a2c116700 2019-12-05T06:40:04.121 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:04.060+0000 7f7a32cba700 1 --2- 172.21.15.42:0/4194495651 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7a2c116700 0x7f7a2c116b30 secure :-1 s=THROTTLE_DONE pgs=2252 cs=0 l=1 rx=0x7f7a24008b10 tx=0x7f7a240112d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:04.124 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:04.060+0000 7f7a22ffd700 1 -- 172.21.15.42:0/4194495651 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7a2400f470 con 0x7f7a2c116700 2019-12-05T06:40:04.125 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:04.060+0000 7f7a33f1c700 1 -- 172.21.15.42:0/4194495651 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a2c138920 con 0x7f7a2c116700 2019-12-05T06:40:04.125 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:04.060+0000 7f7a33f1c700 1 -- 172.21.15.42:0/4194495651 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a2c138e20 con 0x7f7a2c116700 2019-12-05T06:40:04.126 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:04.060+0000 7f7a32cba700 1 --2- 172.21.15.42:0/4194495651 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7a2c116700 0x7f7a2c116b30 secure :-1 s=THROTTLE_DONE pgs=2252 cs=0 l=1 rx=0x7f7a24008b10 tx=0x7f7a240112d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:04.126 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:04.064+0000 7f7a22ffd700 1 -- 172.21.15.42:0/4194495651 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f7a2400f6c0 con 0x7f7a2c116700 2019-12-05T06:40:04.127 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:04.064+0000 7f7a32cba700 1 --2- 172.21.15.42:0/4194495651 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7a2c116700 0x7f7a2c116b30 secure :-1 s=THROTTLE_DONE pgs=2252 cs=0 l=1 rx=0x7f7a24008b10 tx=0x7f7a240112d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:04.127 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:04.064+0000 7f7a22ffd700 1 --2- 172.21.15.42:0/4194495651 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f7a1403b4a0 0x7f7a1403d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:40:04.128 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:04.064+0000 7f7a22ffd700 1 -- 172.21.15.42:0/4194495651 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 11535+0+0 (secure 0 0 0) 0x7f7a2408ef90 con 0x7f7a2c116700 2019-12-05T06:40:04.128 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:04.064+0000 7f7a324b9700 1 --2- 172.21.15.42:0/4194495651 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f7a1403b4a0 0x7f7a1403d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:40:04.128 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:04.064+0000 7f7a324b9700 1 --2- 172.21.15.42:0/4194495651 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f7a1403b4a0 0x7f7a1403d9b0 crc :-1 s=READY pgs=1240 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:40:04.131 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:04.068+0000 7f7a33f1c700 1 -- 172.21.15.42:0/4194495651 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a2c05d110 con 0x7f7a2c116700 2019-12-05T06:40:04.132 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:04.076+0000 7f7a32cba700 1 --2- 172.21.15.42:0/4194495651 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7a2c116700 0x7f7a2c116b30 secure :-1 s=THROTTLE_DONE pgs=2252 cs=0 l=1 rx=0x7f7a24008b10 tx=0x7f7a240112d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:04.132 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:04.076+0000 7f7a22ffd700 1 -- 172.21.15.42:0/4194495651 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f7a24075ff0 con 0x7f7a2c116700 2019-12-05T06:40:04.253 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:04.248+0000 7f7a33f1c700 1 -- 172.21.15.42:0/4194495651 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7a2c05b8c0 con 0x7f7a2c116700 2019-12-05T06:40:04.253 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:04.248+0000 7f7a32cba700 1 --2- 172.21.15.42:0/4194495651 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7a2c116700 0x7f7a2c116b30 secure :-1 s=THROTTLE_DONE pgs=2252 cs=0 l=1 rx=0x7f7a24008b10 tx=0x7f7a240112d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:04.254 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:04.248+0000 7f7a22ffd700 1 -- 172.21.15.42:0/4194495651 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 354 v354) v1 ==== 95+0+2111 (secure 0 0 0) 0x7f7a24076250 con 0x7f7a2c116700 2019-12-05T06:40:04.256 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 354 2019-12-05T06:40:04.256 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:40:04.256 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":354,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":11868,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/406829134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":406829134},{"type":"v1","addr":"172.21.15.42:6835","nonce":406829134}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":354}],"filesystems":[{"mdsmap":{"epoch":353,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:39:37.486362+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":113,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8009},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8009":{"gid":8009,"name":"b","rank":0,"incarnation":349,"state":"up:active","state_seq":27,"addr":"172.21.15.40:6833/1777669201","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1777669201},{"type":"v1","addr":"172.21.15.40:6833","nonce":1777669201}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:40:04.259 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:04.252+0000 7f7a33f1c700 1 -- 172.21.15.42:0/4194495651 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f7a1403b4a0 msgr2=0x7f7a1403d9b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:04.259 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:04.252+0000 7f7a33f1c700 1 --2- 172.21.15.42:0/4194495651 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f7a1403b4a0 0x7f7a1403d9b0 crc :-1 s=READY pgs=1240 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:40:04.259 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:04.252+0000 7f7a33f1c700 1 -- 172.21.15.42:0/4194495651 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7a2c116700 msgr2=0x7f7a2c116b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:04.259 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:04.252+0000 7f7a33f1c700 1 --2- 172.21.15.42:0/4194495651 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7a2c116700 0x7f7a2c116b30 secure :-1 s=READY pgs=2252 cs=0 l=1 rx=0x7f7a24008b10 tx=0x7f7a240112d0).stop 2019-12-05T06:40:04.259 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:04.252+0000 7f7a33f1c700 1 -- 172.21.15.42:0/4194495651 shutdown_connections 2019-12-05T06:40:04.259 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:04.252+0000 7f7a33f1c700 1 --2- 172.21.15.42:0/4194495651 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f7a1403b4a0 0x7f7a1403d9b0 unknown :-1 s=CLOSED pgs=1240 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:40:04.260 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:04.252+0000 7f7a33f1c700 1 --2- 172.21.15.42:0/4194495651 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7a2c116700 0x7f7a2c116b30 unknown :-1 s=CLOSED pgs=2252 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:40:04.260 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:04.252+0000 7f7a33f1c700 1 -- 172.21.15.42:0/4194495651 shutdown_connections 2019-12-05T06:40:04.260 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:04.252+0000 7f7a33f1c700 1 -- 172.21.15.42:0/4194495651 wait complete. 2019-12-05T06:40:04.260 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:04.252+0000 7f7a33f1c700 1 -- 172.21.15.42:0/4194495651 >> 172.21.15.42:0/4194495651 conn(0x7f7a2c0aabe0 msgr2=0x7f7a2c11c470 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:40:04.268 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:40:06.269 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:40:06.368 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.352+0000 7f2756200700 1 Processor -- start 2019-12-05T06:40:06.369 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.352+0000 7f2756200700 1 -- start start 2019-12-05T06:40:06.369 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.352+0000 7f2756200700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f27500a7740 0x7f275008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:40:06.369 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.352+0000 7f2756200700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f275009fae0 0x7f275007eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:40:06.370 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.352+0000 7f2756200700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27500a0710 con 0x7f275007e720 2019-12-05T06:40:06.371 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.352+0000 7f2756200700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27500a0b00 con 0x7f27500a7740 2019-12-05T06:40:06.371 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.352+0000 7f2756200700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f275008e5b0 con 0x7f275009fae0 2019-12-05T06:40:06.371 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.352+0000 7f274ffff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f27500a7740 0x7f275008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:40:06.372 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.352+0000 7f274ffff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f27500a7740 0x7f275008df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:60566/0 (socket says 172.21.15.42:60566) 2019-12-05T06:40:06.372 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.352+0000 7f274ffff700 1 -- 172.21.15.42:0/3932073377 learned_addr learned my addr 172.21.15.42:0/3932073377 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:40:06.372 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.352+0000 7f274ffff700 1 -- 172.21.15.42:0/3932073377 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f275009fae0 msgr2=0x7f275007eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:40:06.372 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.352+0000 7f274ffff700 1 --2- 172.21.15.42:0/3932073377 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f275009fae0 0x7f275007eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:40:06.372 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.352+0000 7f274ffff700 1 -- 172.21.15.42:0/3932073377 >> v1:172.21.15.40:6789/0 conn(0x7f275007e720 legacy=0x7f275009f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:40:06.373 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.352+0000 7f274ffff700 1 -- 172.21.15.42:0/3932073377 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f275007f0a0 con 0x7f27500a7740 2019-12-05T06:40:06.373 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.352+0000 7f274ffff700 1 --2- 172.21.15.42:0/3932073377 >> v2:172.21.15.42:3300/0 conn(0x7f27500a7740 0x7f275008df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f2748008b10 tx=0x7f27480100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:06.373 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.352+0000 7f274ffff700 1 --2- 172.21.15.42:0/3932073377 >> v2:172.21.15.42:3300/0 conn(0x7f27500a7740 0x7f275008df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f2748008b10 tx=0x7f27480100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:06.373 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.352+0000 7f274ffff700 1 --2- 172.21.15.42:0/3932073377 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f27500a7740 0x7f275008df80 secure :-1 s=READY pgs=2967 cs=0 l=1 rx=0x7f2748008b10 tx=0x7f27480100b0).ready entity=mon.1 client_cookie=c4667fd8390fd662 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:40:06.373 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.352+0000 7f274ffff700 1 --2- 172.21.15.42:0/3932073377 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f27500a7740 0x7f275008df80 secure :-1 s=THROTTLE_DONE pgs=2967 cs=0 l=1 rx=0x7f2748008b10 tx=0x7f27480100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:06.373 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.352+0000 7f274effd700 1 -- 172.21.15.42:0/3932073377 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2748010da0 con 0x7f27500a7740 2019-12-05T06:40:06.374 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.352+0000 7f274ffff700 1 --2- 172.21.15.42:0/3932073377 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f27500a7740 0x7f275008df80 secure :-1 s=THROTTLE_DONE pgs=2967 cs=0 l=1 rx=0x7f2748008b10 tx=0x7f27480100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:06.374 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.352+0000 7f274ffff700 1 --2- 172.21.15.42:0/3932073377 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f27500a7740 0x7f275008df80 secure :-1 s=THROTTLE_DONE pgs=2967 cs=0 l=1 rx=0x7f2748008b10 tx=0x7f27480100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:06.374 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.352+0000 7f274effd700 1 -- 172.21.15.42:0/3932073377 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27480109c0 con 0x7f27500a7740 2019-12-05T06:40:06.374 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.352+0000 7f274effd700 1 -- 172.21.15.42:0/3932073377 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f27480114a0 con 0x7f27500a7740 2019-12-05T06:40:06.374 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.352+0000 7f2756200700 1 -- 172.21.15.42:0/3932073377 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f27500a7740 msgr2=0x7f275008df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:06.374 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.352+0000 7f2756200700 1 --2- 172.21.15.42:0/3932073377 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f27500a7740 0x7f275008df80 secure :-1 s=READY pgs=2967 cs=0 l=1 rx=0x7f2748008b10 tx=0x7f27480100b0).stop 2019-12-05T06:40:06.374 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.352+0000 7f2756200700 1 -- 172.21.15.42:0/3932073377 shutdown_connections 2019-12-05T06:40:06.375 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.352+0000 7f2756200700 1 --2- 172.21.15.42:0/3932073377 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f275009fae0 0x7f275007eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:40:06.375 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.352+0000 7f2756200700 1 --2- 172.21.15.42:0/3932073377 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f27500a7740 0x7f275008df80 unknown :-1 s=CLOSED pgs=2967 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:40:06.375 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.352+0000 7f2756200700 1 -- 172.21.15.42:0/3932073377 shutdown_connections 2019-12-05T06:40:06.375 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.352+0000 7f2756200700 1 -- 172.21.15.42:0/3932073377 wait complete. 2019-12-05T06:40:06.375 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.352+0000 7f2756200700 1 -- 172.21.15.42:0/3932073377 >> 172.21.15.42:0/3932073377 conn(0x7f27500aabe0 msgr2=0x7f27500a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:40:06.375 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.356+0000 7f2756200700 1 Processor -- start 2019-12-05T06:40:06.375 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.356+0000 7f2756200700 1 -- start start 2019-12-05T06:40:06.376 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.356+0000 7f2756200700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f27500a3a60 0x7f2750142530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:40:06.376 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.356+0000 7f2756200700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f275007cbd0 0x7f2750138e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:40:06.376 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.356+0000 7f2756200700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2750143830 con 0x7f275007c7a0 2019-12-05T06:40:06.376 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.356+0000 7f2756200700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2750143bd0 con 0x7f275007cbd0 2019-12-05T06:40:06.376 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.356+0000 7f2756200700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2750143e70 con 0x7f27500a3a60 2019-12-05T06:40:06.376 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.356+0000 7f2754f9e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f275007cbd0 0x7f2750138e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:40:06.377 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.356+0000 7f2754f9e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f275007cbd0 0x7f2750138e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:60576/0 (socket says 172.21.15.42:60576) 2019-12-05T06:40:06.377 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.356+0000 7f2754f9e700 1 -- 172.21.15.42:0/3507614114 learned_addr learned my addr 172.21.15.42:0/3507614114 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:40:06.377 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.356+0000 7f274ffff700 1 --2- 172.21.15.42:0/3507614114 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f27500a3a60 0x7f2750142530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:40:06.377 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.356+0000 7f274cff9700 1 -- 172.21.15.42:0/3507614114 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 516101052 0 0) 0x7f2750143830 con 0x7f275007c7a0 2019-12-05T06:40:06.377 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.356+0000 7f274cff9700 1 -- 172.21.15.42:0/3507614114 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27340030b0 con 0x7f275007c7a0 2019-12-05T06:40:06.377 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.356+0000 7f2754f9e700 1 -- 172.21.15.42:0/3507614114 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f27500a3a60 msgr2=0x7f2750142530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:06.378 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.356+0000 7f2754f9e700 1 --2- 172.21.15.42:0/3507614114 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f27500a3a60 0x7f2750142530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:40:06.378 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.356+0000 7f2754f9e700 1 -- 172.21.15.42:0/3507614114 >> v1:172.21.15.42:6789/0 conn(0x7f275007c7a0 legacy=0x7f27500a5a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:06.378 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.356+0000 7f2754f9e700 1 -- 172.21.15.42:0/3507614114 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27500a6290 con 0x7f275007cbd0 2019-12-05T06:40:06.378 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.356+0000 7f2754f9e700 1 --2- 172.21.15.42:0/3507614114 >> v2:172.21.15.42:3300/0 conn(0x7f275007cbd0 0x7f2750138e40 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f2750143d90 tx=0x7f27400106f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:06.378 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.356+0000 7f2754f9e700 1 --2- 172.21.15.42:0/3507614114 >> v2:172.21.15.42:3300/0 conn(0x7f275007cbd0 0x7f2750138e40 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f2750143d90 tx=0x7f27400106f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:06.378 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.356+0000 7f2754f9e700 1 --2- 172.21.15.42:0/3507614114 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f275007cbd0 0x7f2750138e40 secure :-1 s=READY pgs=2969 cs=0 l=1 rx=0x7f2750143d90 tx=0x7f27400106f0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:40:06.378 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.356+0000 7f2754f9e700 1 --2- 172.21.15.42:0/3507614114 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f275007cbd0 0x7f2750138e40 secure :-1 s=THROTTLE_DONE pgs=2969 cs=0 l=1 rx=0x7f2750143d90 tx=0x7f27400106f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:06.379 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.356+0000 7f274cff9700 1 -- 172.21.15.42:0/3507614114 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2740005940 con 0x7f275007cbd0 2019-12-05T06:40:06.379 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.356+0000 7f2756200700 1 -- 172.21.15.42:0/3507614114 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2750142a80 con 0x7f275007cbd0 2019-12-05T06:40:06.379 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.356+0000 7f2756200700 1 -- 172.21.15.42:0/3507614114 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2750139390 con 0x7f275007cbd0 2019-12-05T06:40:06.379 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.360+0000 7f2754f9e700 1 --2- 172.21.15.42:0/3507614114 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f275007cbd0 0x7f2750138e40 secure :-1 s=THROTTLE_DONE pgs=2969 cs=0 l=1 rx=0x7f2750143d90 tx=0x7f27400106f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:06.379 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.360+0000 7f2754f9e700 1 --2- 172.21.15.42:0/3507614114 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f275007cbd0 0x7f2750138e40 secure :-1 s=THROTTLE_DONE pgs=2969 cs=0 l=1 rx=0x7f2750143d90 tx=0x7f27400106f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:06.380 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.360+0000 7f274cff9700 1 -- 172.21.15.42:0/3507614114 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2740007160 con 0x7f275007cbd0 2019-12-05T06:40:06.380 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.360+0000 7f274cff9700 1 -- 172.21.15.42:0/3507614114 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2740008630 con 0x7f275007cbd0 2019-12-05T06:40:06.380 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.360+0000 7f2754f9e700 1 --2- 172.21.15.42:0/3507614114 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f275007cbd0 0x7f2750138e40 secure :-1 s=THROTTLE_DONE pgs=2969 cs=0 l=1 rx=0x7f2750143d90 tx=0x7f27400106f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:06.380 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.360+0000 7f274cff9700 1 -- 172.21.15.42:0/3507614114 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f2740008830 con 0x7f275007cbd0 2019-12-05T06:40:06.380 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.360+0000 7f2754f9e700 1 --2- 172.21.15.42:0/3507614114 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f275007cbd0 0x7f2750138e40 secure :-1 s=THROTTLE_DONE pgs=2969 cs=0 l=1 rx=0x7f2750143d90 tx=0x7f27400106f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:06.380 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.360+0000 7f274cff9700 1 --2- 172.21.15.42:0/3507614114 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f273403b260 0x7f273403d770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:40:06.381 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.360+0000 7f274cff9700 1 -- 172.21.15.42:0/3507614114 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 11621+0+0 (secure 0 0 0) 0x7f274004ac40 con 0x7f275007cbd0 2019-12-05T06:40:06.381 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.360+0000 7f274ffff700 1 --2- 172.21.15.42:0/3507614114 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f273403b260 0x7f273403d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:40:06.381 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.360+0000 7f274ffff700 1 --2- 172.21.15.42:0/3507614114 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f273403b260 0x7f273403d770 crc :-1 s=READY pgs=1241 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:40:06.382 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.368+0000 7f2756200700 1 -- 172.21.15.42:0/3507614114 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2728003200 con 0x7f275007cbd0 2019-12-05T06:40:06.382 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.372+0000 7f2754f9e700 1 --2- 172.21.15.42:0/3507614114 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f275007cbd0 0x7f2750138e40 secure :-1 s=THROTTLE_DONE pgs=2969 cs=0 l=1 rx=0x7f2750143d90 tx=0x7f27400106f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:06.382 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.372+0000 7f274cff9700 1 -- 172.21.15.42:0/3507614114 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f2740031ce0 con 0x7f275007cbd0 2019-12-05T06:40:06.569 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.564+0000 7f2756200700 1 -- 172.21.15.42:0/3507614114 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2728003630 con 0x7f275007cbd0 2019-12-05T06:40:06.571 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.564+0000 7f2754f9e700 1 --2- 172.21.15.42:0/3507614114 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f275007cbd0 0x7f2750138e40 secure :-1 s=THROTTLE_DONE pgs=2969 cs=0 l=1 rx=0x7f2750143d90 tx=0x7f27400106f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:06.571 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.564+0000 7f274cff9700 1 -- 172.21.15.42:0/3507614114 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 355 v355) v1 ==== 95+0+1775 (secure 0 0 0) 0x7f2740031f40 con 0x7f275007cbd0 2019-12-05T06:40:06.571 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 355 2019-12-05T06:40:06.571 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:40:06.572 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":355,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":355,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:40:05.188099+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":114,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11868},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11868":{"gid":11868,"name":"a","rank":0,"incarnation":355,"state":"up:replay","state_seq":2,"addr":"172.21.15.42:6835/406829134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":406829134},{"type":"v1","addr":"172.21.15.42:6835","nonce":406829134}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:40:06.573 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.568+0000 7f2756200700 1 -- 172.21.15.42:0/3507614114 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f273403b260 msgr2=0x7f273403d770 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:06.574 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.568+0000 7f2756200700 1 --2- 172.21.15.42:0/3507614114 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f273403b260 0x7f273403d770 crc :-1 s=READY pgs=1241 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:40:06.574 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.568+0000 7f2756200700 1 -- 172.21.15.42:0/3507614114 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f275007cbd0 msgr2=0x7f2750138e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:06.574 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.568+0000 7f2756200700 1 --2- 172.21.15.42:0/3507614114 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f275007cbd0 0x7f2750138e40 secure :-1 s=READY pgs=2969 cs=0 l=1 rx=0x7f2750143d90 tx=0x7f27400106f0).stop 2019-12-05T06:40:06.574 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.568+0000 7f2756200700 1 -- 172.21.15.42:0/3507614114 shutdown_connections 2019-12-05T06:40:06.576 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.568+0000 7f2756200700 1 --2- 172.21.15.42:0/3507614114 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f273403b260 0x7f273403d770 unknown :-1 s=CLOSED pgs=1241 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:40:06.576 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.568+0000 7f2756200700 1 --2- 172.21.15.42:0/3507614114 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f275007cbd0 0x7f2750138e40 unknown :-1 s=CLOSED pgs=2969 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:40:06.576 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.568+0000 7f2756200700 1 --2- 172.21.15.42:0/3507614114 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f27500a3a60 0x7f2750142530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:40:06.577 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.568+0000 7f2756200700 1 -- 172.21.15.42:0/3507614114 shutdown_connections 2019-12-05T06:40:06.577 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.568+0000 7f2756200700 1 -- 172.21.15.42:0/3507614114 wait complete. 2019-12-05T06:40:06.577 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.568+0000 7f2756200700 1 -- 172.21.15.42:0/3507614114 >> 172.21.15.42:0/3507614114 conn(0x7f275008ef90 msgr2=0x7f275011bc00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:40:06.581 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b down, removed from mdsmap 2019-12-05T06:40:06.582 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2019-12-05T06:40:06.582 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:40:06.677 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.668+0000 7fcc6f4fa700 1 Processor -- start 2019-12-05T06:40:06.678 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.668+0000 7fcc6f4fa700 1 -- start start 2019-12-05T06:40:06.678 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.668+0000 7fcc6f4fa700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fcc6808df80 0x7fcc6809f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:40:06.678 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.668+0000 7fcc6f4fa700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcc6809f830 0x7fcc6814efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:40:06.678 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.668+0000 7fcc6f4fa700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc6809fc60 con 0x7fcc680a7740 2019-12-05T06:40:06.678 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.668+0000 7fcc6f4fa700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc680a0b00 con 0x7fcc6808df80 2019-12-05T06:40:06.679 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.668+0000 7fcc6f4fa700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc6808e5b0 con 0x7fcc6809f830 2019-12-05T06:40:06.679 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.668+0000 7fcc6da97700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fcc680a7740 0x7fcc6807e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:60618/0 (socket says 172.21.15.42:60618) 2019-12-05T06:40:06.679 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.668+0000 7fcc6da97700 1 -- 172.21.15.42:0/376869148 learned_addr learned my addr 172.21.15.42:0/376869148 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:40:06.679 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.668+0000 7fcc6d296700 1 --2- 172.21.15.42:0/376869148 >> v2:172.21.15.40:3300/0 conn(0x7fcc6808df80 0x7fcc6809f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:40:06.679 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.668+0000 7fcc6e298700 1 --2- 172.21.15.42:0/376869148 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcc6809f830 0x7fcc6814efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:40:06.680 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.668+0000 7fcc6ca95700 1 -- 172.21.15.42:0/376869148 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 664597941 0 0) 0x7fcc6809fc60 con 0x7fcc680a7740 2019-12-05T06:40:06.680 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.668+0000 7fcc6ca95700 1 -- 172.21.15.42:0/376869148 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc540030b0 con 0x7fcc680a7740 2019-12-05T06:40:06.680 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.668+0000 7fcc6ca95700 1 -- 172.21.15.42:0/376869148 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1443067154 0 0) 0x7fcc540030b0 con 0x7fcc680a7740 2019-12-05T06:40:06.680 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.668+0000 7fcc6ca95700 1 -- 172.21.15.42:0/376869148 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcc6809f830 msgr2=0x7fcc6814efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:40:06.680 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.668+0000 7fcc6ca95700 1 --2- 172.21.15.42:0/376869148 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcc6809f830 0x7fcc6814efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:40:06.681 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.668+0000 7fcc6ca95700 1 -- 172.21.15.42:0/376869148 >> v2:172.21.15.40:3300/0 conn(0x7fcc6808df80 msgr2=0x7fcc6809f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:40:06.681 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.668+0000 7fcc6ca95700 1 --2- 172.21.15.42:0/376869148 >> v2:172.21.15.40:3300/0 conn(0x7fcc6808df80 0x7fcc6809f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:40:06.681 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.668+0000 7fcc6ca95700 1 -- 172.21.15.42:0/376869148 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc680a0710 con 0x7fcc680a7740 2019-12-05T06:40:06.681 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.668+0000 7fcc6ca95700 1 -- 172.21.15.42:0/376869148 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7fcc60001750 con 0x7fcc680a7740 2019-12-05T06:40:06.681 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.668+0000 7fcc6ca95700 1 -- 172.21.15.42:0/376869148 >> v1:172.21.15.42:6789/0 conn(0x7fcc680a7740 legacy=0x7fcc6807e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:06.681 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.668+0000 7fcc6ca95700 1 --2- 172.21.15.42:0/376869148 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc540036e0 0x7fcc54003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:40:06.681 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.668+0000 7fcc6ca95700 1 --2- 172.21.15.42:0/376869148 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcc54004420 0x7fcc54004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:40:06.682 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.668+0000 7fcc6ca95700 1 --2- 172.21.15.42:0/376869148 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcc54004da0 0x7fcc5404f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:40:06.682 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.668+0000 7fcc6ca95700 1 -- 172.21.15.42:0/376869148 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc680a0b00 con 0x7fcc540036e0 2019-12-05T06:40:06.682 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.668+0000 7fcc6ca95700 1 -- 172.21.15.42:0/376869148 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc6808e5b0 con 0x7fcc54004da0 2019-12-05T06:40:06.682 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.668+0000 7fcc6ca95700 1 -- 172.21.15.42:0/376869148 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc54005250 con 0x7fcc54004420 2019-12-05T06:40:06.682 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.668+0000 7fcc6e298700 1 --2- 172.21.15.42:0/376869148 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcc54004da0 0x7fcc5404f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:40:06.682 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.668+0000 7fcc6da97700 1 --2- 172.21.15.42:0/376869148 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc540036e0 0x7fcc54003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:40:06.682 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.668+0000 7fcc6e298700 1 -- 172.21.15.42:0/376869148 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcc54004420 msgr2=0x7fcc54004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:40:06.683 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.668+0000 7fcc6d296700 1 --2- 172.21.15.42:0/376869148 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcc54004420 0x7fcc54004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:40:06.683 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.668+0000 7fcc6e298700 1 --2- 172.21.15.42:0/376869148 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcc54004420 0x7fcc54004850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:40:06.683 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.668+0000 7fcc6da97700 1 -- 172.21.15.42:0/376869148 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc540036e0 msgr2=0x7fcc54003b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2019-12-05T06:40:06.683 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.668+0000 7fcc6da97700 1 -- 172.21.15.42:0/376869148 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc540036e0 msgr2=0x7fcc54003b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2019-12-05T06:40:06.683 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.668+0000 7fcc6e298700 1 -- 172.21.15.42:0/376869148 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc540036e0 msgr2=0x7fcc54003b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:40:06.683 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.668+0000 7fcc6da97700 1 --2- 172.21.15.42:0/376869148 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc540036e0 0x7fcc54003b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-12-05T06:40:06.684 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.668+0000 7fcc6da97700 1 --2- 172.21.15.42:0/376869148 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc540036e0 0x7fcc54003b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rx=0 tx=0)._fault waiting 0.200000 2019-12-05T06:40:06.684 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.668+0000 7fcc6e298700 1 --2- 172.21.15.42:0/376869148 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc540036e0 0x7fcc54003b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:40:06.684 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.668+0000 7fcc6e298700 1 -- 172.21.15.42:0/376869148 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc5404fd10 con 0x7fcc54004da0 2019-12-05T06:40:06.684 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.668+0000 7fcc6e298700 1 --2- 172.21.15.42:0/376869148 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcc54004da0 0x7fcc5404f680 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fcc64002f30 tx=0x7fcc6400ffa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:06.684 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.668+0000 7fcc6e298700 1 --2- 172.21.15.42:0/376869148 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcc54004da0 0x7fcc5404f680 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fcc64002f30 tx=0x7fcc6400ffa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:06.684 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.668+0000 7fcc6e298700 1 --2- 172.21.15.42:0/376869148 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcc54004da0 0x7fcc5404f680 secure :-1 s=READY pgs=2971 cs=0 l=1 rx=0x7fcc64002f30 tx=0x7fcc6400ffa0).ready entity=mon.1 client_cookie=8dca77a017bfd13c server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:40:06.685 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.672+0000 7fcc6e298700 1 --2- 172.21.15.42:0/376869148 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcc54004da0 0x7fcc5404f680 secure :-1 s=THROTTLE_DONE pgs=2971 cs=0 l=1 rx=0x7fcc64002f30 tx=0x7fcc6400ffa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:06.685 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.672+0000 7fcc6ca95700 1 -- 172.21.15.42:0/376869148 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcc64010cd0 con 0x7fcc54004da0 2019-12-05T06:40:06.685 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.672+0000 7fcc6e298700 1 --2- 172.21.15.42:0/376869148 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcc54004da0 0x7fcc5404f680 secure :-1 s=THROTTLE_DONE pgs=2971 cs=0 l=1 rx=0x7fcc64002f30 tx=0x7fcc6400ffa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:06.685 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.672+0000 7fcc6e298700 1 --2- 172.21.15.42:0/376869148 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcc54004da0 0x7fcc5404f680 secure :-1 s=THROTTLE_DONE pgs=2971 cs=0 l=1 rx=0x7fcc64002f30 tx=0x7fcc6400ffa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:06.685 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.672+0000 7fcc6ca95700 1 -- 172.21.15.42:0/376869148 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc640108f0 con 0x7fcc54004da0 2019-12-05T06:40:06.685 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.672+0000 7fcc6ca95700 1 -- 172.21.15.42:0/376869148 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcc64008bb0 con 0x7fcc54004da0 2019-12-05T06:40:06.685 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.672+0000 7fcc6f4fa700 1 -- 172.21.15.42:0/376869148 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcc54004da0 msgr2=0x7fcc5404f680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:06.686 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.672+0000 7fcc6f4fa700 1 --2- 172.21.15.42:0/376869148 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcc54004da0 0x7fcc5404f680 secure :-1 s=READY pgs=2971 cs=0 l=1 rx=0x7fcc64002f30 tx=0x7fcc6400ffa0).stop 2019-12-05T06:40:06.686 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.672+0000 7fcc6e298700 1 -- 172.21.15.42:0/376869148 reap_dead start 2019-12-05T06:40:06.686 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.672+0000 7fcc6f4fa700 1 -- 172.21.15.42:0/376869148 shutdown_connections 2019-12-05T06:40:06.686 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.672+0000 7fcc6f4fa700 1 -- 172.21.15.42:0/376869148 shutdown_connections 2019-12-05T06:40:06.686 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.672+0000 7fcc6f4fa700 1 -- 172.21.15.42:0/376869148 wait complete. 2019-12-05T06:40:06.686 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.672+0000 7fcc6f4fa700 1 -- 172.21.15.42:0/376869148 >> 172.21.15.42:0/376869148 conn(0x7fcc680aabe0 msgr2=0x7fcc680a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:40:06.690 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.672+0000 7fcc6f4fa700 1 Processor -- start 2019-12-05T06:40:06.690 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.672+0000 7fcc6f4fa700 1 -- start start 2019-12-05T06:40:06.691 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.672+0000 7fcc6f4fa700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fcc68144f10 0x7fcc68138a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:40:06.691 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.672+0000 7fcc6f4fa700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc68145340 con 0x7fcc6811bfb0 2019-12-05T06:40:06.691 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.672+0000 7fcc6f4fa700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc68144460 con 0x7fcc68142870 2019-12-05T06:40:06.691 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.672+0000 7fcc6f4fa700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc6811b240 con 0x7fcc68144f10 2019-12-05T06:40:06.691 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.672+0000 7fcc6d296700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fcc68142870 0x7fcc6807d850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:60632/0 (socket says 172.21.15.42:60632) 2019-12-05T06:40:06.691 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.672+0000 7fcc6d296700 1 -- 172.21.15.42:0/3052504463 learned_addr learned my addr 172.21.15.42:0/3052504463 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:40:06.691 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.672+0000 7fcc6e298700 1 --2- 172.21.15.42:0/3052504463 >> v2:172.21.15.40:3300/0 conn(0x7fcc68144f10 0x7fcc68138a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:40:06.692 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.672+0000 7fcc5a7fc700 1 -- 172.21.15.42:0/3052504463 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2549543812 0 0) 0x7fcc68144460 con 0x7fcc68142870 2019-12-05T06:40:06.692 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.672+0000 7fcc5a7fc700 1 -- 172.21.15.42:0/3052504463 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc500030b0 con 0x7fcc68142870 2019-12-05T06:40:06.692 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.672+0000 7fcc5a7fc700 1 -- 172.21.15.42:0/3052504463 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3144168819 0 0) 0x7fcc68145340 con 0x7fcc6811bfb0 2019-12-05T06:40:06.692 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.672+0000 7fcc5a7fc700 1 -- 172.21.15.42:0/3052504463 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc68144460 con 0x7fcc6811bfb0 2019-12-05T06:40:06.692 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.672+0000 7fcc5a7fc700 1 -- 172.21.15.42:0/3052504463 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 907591444 0 0) 0x7fcc500030b0 con 0x7fcc68142870 2019-12-05T06:40:06.692 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.676+0000 7fcc5a7fc700 1 -- 172.21.15.42:0/3052504463 --> v1:172.21.15.42:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcc68145340 con 0x7fcc68142870 2019-12-05T06:40:06.692 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.676+0000 7fcc5a7fc700 1 -- 172.21.15.42:0/3052504463 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7fcc5c001620 con 0x7fcc68142870 2019-12-05T06:40:06.693 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.676+0000 7fcc6e298700 1 -- 172.21.15.42:0/3052504463 >> v1:172.21.15.42:6789/0 conn(0x7fcc68142870 legacy=0x7fcc6807d850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:06.693 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.676+0000 7fcc6e298700 1 -- 172.21.15.42:0/3052504463 >> v1:172.21.15.40:6789/0 conn(0x7fcc6811bfb0 legacy=0x7fcc6811aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:06.693 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.676+0000 7fcc6e298700 1 -- 172.21.15.42:0/3052504463 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc6807e050 con 0x7fcc68144f10 2019-12-05T06:40:06.693 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.676+0000 7fcc6e298700 1 --2- 172.21.15.42:0/3052504463 >> v2:172.21.15.40:3300/0 conn(0x7fcc68144f10 0x7fcc68138a60 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fcc680a14d0 tx=0x7fcc60000e90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:06.693 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.676+0000 7fcc6e298700 1 --2- 172.21.15.42:0/3052504463 >> v2:172.21.15.40:3300/0 conn(0x7fcc68144f10 0x7fcc68138a60 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fcc680a14d0 tx=0x7fcc60000e90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:06.693 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.676+0000 7fcc6e298700 1 --2- 172.21.15.42:0/3052504463 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc68144f10 0x7fcc68138a60 secure :-1 s=READY pgs=2254 cs=0 l=1 rx=0x7fcc680a14d0 tx=0x7fcc60000e90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:40:06.693 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.676+0000 7fcc6e298700 1 --2- 172.21.15.42:0/3052504463 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc68144f10 0x7fcc68138a60 secure :-1 s=THROTTLE_DONE pgs=2254 cs=0 l=1 rx=0x7fcc680a14d0 tx=0x7fcc60000e90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:06.694 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.676+0000 7fcc5a7fc700 1 -- 172.21.15.42:0/3052504463 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcc64010b60 con 0x7fcc68144f10 2019-12-05T06:40:06.694 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.676+0000 7fcc6e298700 1 --2- 172.21.15.42:0/3052504463 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc68144f10 0x7fcc68138a60 secure :-1 s=THROTTLE_DONE pgs=2254 cs=0 l=1 rx=0x7fcc680a14d0 tx=0x7fcc60000e90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:06.694 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.676+0000 7fcc6f4fa700 1 -- 172.21.15.42:0/3052504463 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc68138fb0 con 0x7fcc68144f10 2019-12-05T06:40:06.694 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.676+0000 7fcc5a7fc700 1 -- 172.21.15.42:0/3052504463 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc640087d0 con 0x7fcc68144f10 2019-12-05T06:40:06.694 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.676+0000 7fcc6e298700 1 --2- 172.21.15.42:0/3052504463 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc68144f10 0x7fcc68138a60 secure :-1 s=THROTTLE_DONE pgs=2254 cs=0 l=1 rx=0x7fcc680a14d0 tx=0x7fcc60000e90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:06.694 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.676+0000 7fcc5a7fc700 1 -- 172.21.15.42:0/3052504463 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcc6400f3e0 con 0x7fcc68144f10 2019-12-05T06:40:06.694 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.676+0000 7fcc6f4fa700 1 -- 172.21.15.42:0/3052504463 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc6808eb70 con 0x7fcc68144f10 2019-12-05T06:40:06.695 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.676+0000 7fcc6e298700 1 --2- 172.21.15.42:0/3052504463 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc68144f10 0x7fcc68138a60 secure :-1 s=THROTTLE_DONE pgs=2254 cs=0 l=1 rx=0x7fcc680a14d0 tx=0x7fcc60000e90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:06.695 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.676+0000 7fcc5a7fc700 1 -- 172.21.15.42:0/3052504463 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fcc6400f5e0 con 0x7fcc68144f10 2019-12-05T06:40:06.695 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.676+0000 7fcc6e298700 1 --2- 172.21.15.42:0/3052504463 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc68144f10 0x7fcc68138a60 secure :-1 s=THROTTLE_DONE pgs=2254 cs=0 l=1 rx=0x7fcc680a14d0 tx=0x7fcc60000e90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:06.695 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.676+0000 7fcc5a7fc700 1 --2- 172.21.15.42:0/3052504463 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fcc5003b820 0x7fcc5003dd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:40:06.695 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.676+0000 7fcc5a7fc700 1 -- 172.21.15.42:0/3052504463 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 11621+0+0 (secure 0 0 0) 0x7fcc6404a960 con 0x7fcc68144f10 2019-12-05T06:40:06.696 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.676+0000 7fcc6da97700 1 --2- 172.21.15.42:0/3052504463 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fcc5003b820 0x7fcc5003dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:40:06.696 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.676+0000 7fcc6da97700 1 --2- 172.21.15.42:0/3052504463 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fcc5003b820 0x7fcc5003dd30 crc :-1 s=READY pgs=1242 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:40:06.696 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.684+0000 7fcc6f4fa700 1 -- 172.21.15.42:0/3052504463 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc6805d110 con 0x7fcc68144f10 2019-12-05T06:40:06.701 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.696+0000 7fcc6e298700 1 --2- 172.21.15.42:0/3052504463 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc68144f10 0x7fcc68138a60 secure :-1 s=THROTTLE_DONE pgs=2254 cs=0 l=1 rx=0x7fcc680a14d0 tx=0x7fcc60000e90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:06.701 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.696+0000 7fcc5a7fc700 1 -- 172.21.15.42:0/3052504463 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fcc640319c0 con 0x7fcc68144f10 2019-12-05T06:40:06.861 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.856+0000 7fcc6f4fa700 1 -- 172.21.15.42:0/3052504463 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcc6805b8c0 con 0x7fcc68144f10 2019-12-05T06:40:06.862 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.856+0000 7fcc6e298700 1 --2- 172.21.15.42:0/3052504463 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc68144f10 0x7fcc68138a60 secure :-1 s=THROTTLE_DONE pgs=2254 cs=0 l=1 rx=0x7fcc680a14d0 tx=0x7fcc60000e90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:06.862 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.856+0000 7fcc5a7fc700 1 -- 172.21.15.42:0/3052504463 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 355 v355) v1 ==== 95+0+1775 (secure 0 0 0) 0x7fcc64031c20 con 0x7fcc68144f10 2019-12-05T06:40:06.862 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 355 2019-12-05T06:40:06.863 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:40:06.863 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":355,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":355,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:40:05.188099+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":114,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11868},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11868":{"gid":11868,"name":"a","rank":0,"incarnation":355,"state":"up:replay","state_seq":2,"addr":"172.21.15.42:6835/406829134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":406829134},{"type":"v1","addr":"172.21.15.42:6835","nonce":406829134}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:40:06.865 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.860+0000 7fcc6f4fa700 1 -- 172.21.15.42:0/3052504463 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fcc5003b820 msgr2=0x7fcc5003dd30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:06.865 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.860+0000 7fcc6f4fa700 1 --2- 172.21.15.42:0/3052504463 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fcc5003b820 0x7fcc5003dd30 crc :-1 s=READY pgs=1242 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:40:06.865 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.860+0000 7fcc6f4fa700 1 -- 172.21.15.42:0/3052504463 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc68144f10 msgr2=0x7fcc68138a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:06.865 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.860+0000 7fcc6f4fa700 1 --2- 172.21.15.42:0/3052504463 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc68144f10 0x7fcc68138a60 secure :-1 s=READY pgs=2254 cs=0 l=1 rx=0x7fcc680a14d0 tx=0x7fcc60000e90).stop 2019-12-05T06:40:06.866 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.860+0000 7fcc6f4fa700 1 -- 172.21.15.42:0/3052504463 shutdown_connections 2019-12-05T06:40:06.866 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.860+0000 7fcc6f4fa700 1 --2- 172.21.15.42:0/3052504463 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fcc5003b820 0x7fcc5003dd30 unknown :-1 s=CLOSED pgs=1242 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:40:06.866 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.860+0000 7fcc6f4fa700 1 --2- 172.21.15.42:0/3052504463 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc68144f10 0x7fcc68138a60 unknown :-1 s=CLOSED pgs=2254 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:40:06.866 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.860+0000 7fcc6f4fa700 1 -- 172.21.15.42:0/3052504463 shutdown_connections 2019-12-05T06:40:06.866 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.860+0000 7fcc6f4fa700 1 -- 172.21.15.42:0/3052504463 wait complete. 2019-12-05T06:40:06.866 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:06.860+0000 7fcc6f4fa700 1 -- 172.21.15.42:0/3052504463 >> 172.21.15.42:0/3052504463 conn(0x7fcc680a9770 msgr2=0x7fcc6811c470 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:40:06.874 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 355, "feature_flags": { "enable_multiple": false, "ever_enabled_multiple": false }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2019-12-05T05:03:35.604738+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 355, "ever_allowed_features": 2, "explicitly_allowed_features": 2, "failed": [], "flags": 18, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_11868": { "addr": "172.21.15.42:6835/406829134", "addrs": { "addrvec": [ { "addr": "172.21.15.42:6834", "nonce": 406829134, "type": "v2" }, { "addr": "172.21.15.42:6835", "nonce": 406829134, "type": "v1" } ] }, "export_targets": [], "features": 4540138292836696063, "flags": 0, "gid": 11868, "incarnation": 355, "name": "a", "rank": 0, "state": "up:replay", "state_seq": 2 } }, "last_failure": 0, "last_failure_osd_epoch": 114, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 2, "min_compat_client": "0 (unknown)", "modified": "2019-12-05T06:40:05.188099+0000", "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 11868 } } } ], "standbys": [] } 2019-12-05T06:40:08.876 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:40:08.970 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:08.960+0000 7fc8d604e700 1 Processor -- start 2019-12-05T06:40:08.970 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:08.960+0000 7fc8d604e700 1 -- start start 2019-12-05T06:40:08.970 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:08.960+0000 7fc8d604e700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc8d009fae0 0x7fc8d014fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:40:08.970 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:08.960+0000 7fc8d604e700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8d00a0710 con 0x7fc8d00a7740 2019-12-05T06:40:08.971 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:08.960+0000 7fc8d604e700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8d007ef20 con 0x7fc8d008df80 2019-12-05T06:40:08.971 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:08.960+0000 7fc8d604e700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8d00a0b00 con 0x7fc8d009fae0 2019-12-05T06:40:08.971 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:08.960+0000 7fc8cf7fe700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fc8d008df80 0x7fc8d009f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:60646/0 (socket says 172.21.15.42:60646) 2019-12-05T06:40:08.971 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:08.960+0000 7fc8cf7fe700 1 -- 172.21.15.42:0/2142054388 learned_addr learned my addr 172.21.15.42:0/2142054388 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:40:08.971 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:08.960+0000 7fc8d4dec700 1 --2- 172.21.15.42:0/2142054388 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc8d009fae0 0x7fc8d014fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:40:08.971 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:08.960+0000 7fc8ceffd700 1 -- 172.21.15.42:0/2142054388 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1159138372 0 0) 0x7fc8d007ef20 con 0x7fc8d008df80 2019-12-05T06:40:08.972 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:08.960+0000 7fc8ceffd700 1 -- 172.21.15.42:0/2142054388 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8bc0030b0 con 0x7fc8d008df80 2019-12-05T06:40:08.972 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:08.960+0000 7fc8ceffd700 1 -- 172.21.15.42:0/2142054388 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 372249223 0 0) 0x7fc8bc0030b0 con 0x7fc8d008df80 2019-12-05T06:40:08.972 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:08.960+0000 7fc8ceffd700 1 -- 172.21.15.42:0/2142054388 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc8d009fae0 msgr2=0x7fc8d014fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:40:08.972 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:08.960+0000 7fc8ceffd700 1 --2- 172.21.15.42:0/2142054388 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc8d009fae0 0x7fc8d014fff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:40:08.972 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:08.960+0000 7fc8ceffd700 1 -- 172.21.15.42:0/2142054388 >> v1:172.21.15.40:6789/0 conn(0x7fc8d00a7740 legacy=0x7fc8d007e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:08.972 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:08.960+0000 7fc8ceffd700 1 -- 172.21.15.42:0/2142054388 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8d00a9ba0 con 0x7fc8d008df80 2019-12-05T06:40:08.973 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:08.960+0000 7fc8ceffd700 1 -- 172.21.15.42:0/2142054388 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7fc8b8001750 con 0x7fc8d008df80 2019-12-05T06:40:08.973 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:08.960+0000 7fc8ceffd700 1 -- 172.21.15.42:0/2142054388 >> v1:172.21.15.42:6789/0 conn(0x7fc8d008df80 legacy=0x7fc8d009f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:08.973 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:08.960+0000 7fc8ceffd700 1 --2- 172.21.15.42:0/2142054388 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc8bc003710 0x7fc8bc003b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:40:08.973 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:08.960+0000 7fc8ceffd700 1 --2- 172.21.15.42:0/2142054388 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc8bc004420 0x7fc8bc004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:40:08.973 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:08.960+0000 7fc8ceffd700 1 --2- 172.21.15.42:0/2142054388 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc8bc004da0 0x7fc8bc00b670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:40:08.973 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:08.960+0000 7fc8ceffd700 1 -- 172.21.15.42:0/2142054388 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8d00a0b00 con 0x7fc8bc004420 2019-12-05T06:40:08.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:08.960+0000 7fc8cffff700 1 --2- 172.21.15.42:0/2142054388 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc8bc003710 0x7fc8bc003b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:40:08.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:08.960+0000 7fc8ceffd700 1 -- 172.21.15.42:0/2142054388 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8bc0051d0 con 0x7fc8bc003710 2019-12-05T06:40:08.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:08.960+0000 7fc8ceffd700 1 -- 172.21.15.42:0/2142054388 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8bc04fbd0 con 0x7fc8bc004da0 2019-12-05T06:40:08.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:08.960+0000 7fc8cf7fe700 1 --2- 172.21.15.42:0/2142054388 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc8bc004420 0x7fc8bc004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:40:08.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:08.960+0000 7fc8d4dec700 1 --2- 172.21.15.42:0/2142054388 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc8bc004da0 0x7fc8bc00b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:40:08.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:08.960+0000 7fc8cffff700 1 -- 172.21.15.42:0/2142054388 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc8bc004da0 msgr2=0x7fc8bc00b670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:40:08.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:08.960+0000 7fc8cffff700 1 --2- 172.21.15.42:0/2142054388 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc8bc004da0 0x7fc8bc00b670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:40:08.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:08.960+0000 7fc8cffff700 1 -- 172.21.15.42:0/2142054388 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc8bc004420 msgr2=0x7fc8bc004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:40:08.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:08.960+0000 7fc8cffff700 1 --2- 172.21.15.42:0/2142054388 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc8bc004420 0x7fc8bc004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:40:08.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:08.960+0000 7fc8cffff700 1 -- 172.21.15.42:0/2142054388 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8bc04fe50 con 0x7fc8bc003710 2019-12-05T06:40:08.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:08.960+0000 7fc8cf7fe700 1 --2- 172.21.15.42:0/2142054388 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc8bc004420 0x7fc8bc004850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:40:08.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:08.960+0000 7fc8cffff700 1 --2- 172.21.15.42:0/2142054388 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc8bc003710 0x7fc8bc003b40 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fc8c0001360 tx=0x7fc8c0010100).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:08.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:08.960+0000 7fc8cffff700 1 --2- 172.21.15.42:0/2142054388 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc8bc003710 0x7fc8bc003b40 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fc8c0001360 tx=0x7fc8c0010100).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:08.976 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:08.960+0000 7fc8cffff700 1 --2- 172.21.15.42:0/2142054388 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc8bc003710 0x7fc8bc003b40 secure :-1 s=READY pgs=2974 cs=0 l=1 rx=0x7fc8c0001360 tx=0x7fc8c0010100).ready entity=mon.1 client_cookie=272cc6b5bca0fc72 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:40:08.976 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:08.960+0000 7fc8cffff700 1 --2- 172.21.15.42:0/2142054388 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc8bc003710 0x7fc8bc003b40 secure :-1 s=THROTTLE_DONE pgs=2974 cs=0 l=1 rx=0x7fc8c0001360 tx=0x7fc8c0010100).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:08.976 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:08.960+0000 7fc8ceffd700 1 -- 172.21.15.42:0/2142054388 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc8c0006ac0 con 0x7fc8bc003710 2019-12-05T06:40:08.976 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:08.960+0000 7fc8cffff700 1 --2- 172.21.15.42:0/2142054388 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc8bc003710 0x7fc8bc003b40 secure :-1 s=THROTTLE_DONE pgs=2974 cs=0 l=1 rx=0x7fc8c0001360 tx=0x7fc8c0010100).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:08.976 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:08.960+0000 7fc8cffff700 1 --2- 172.21.15.42:0/2142054388 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc8bc003710 0x7fc8bc003b40 secure :-1 s=THROTTLE_DONE pgs=2974 cs=0 l=1 rx=0x7fc8c0001360 tx=0x7fc8c0010100).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:08.976 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:08.960+0000 7fc8ceffd700 1 -- 172.21.15.42:0/2142054388 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8c00066e0 con 0x7fc8bc003710 2019-12-05T06:40:08.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:08.960+0000 7fc8ceffd700 1 -- 172.21.15.42:0/2142054388 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc8c0002730 con 0x7fc8bc003710 2019-12-05T06:40:08.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:08.960+0000 7fc8d604e700 1 -- 172.21.15.42:0/2142054388 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc8bc003710 msgr2=0x7fc8bc003b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:08.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:08.960+0000 7fc8d604e700 1 --2- 172.21.15.42:0/2142054388 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc8bc003710 0x7fc8bc003b40 secure :-1 s=READY pgs=2974 cs=0 l=1 rx=0x7fc8c0001360 tx=0x7fc8c0010100).stop 2019-12-05T06:40:08.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:08.960+0000 7fc8d4dec700 1 -- 172.21.15.42:0/2142054388 reap_dead start 2019-12-05T06:40:08.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:08.964+0000 7fc8d604e700 1 -- 172.21.15.42:0/2142054388 shutdown_connections 2019-12-05T06:40:08.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:08.964+0000 7fc8d604e700 1 -- 172.21.15.42:0/2142054388 shutdown_connections 2019-12-05T06:40:08.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:08.964+0000 7fc8d604e700 1 -- 172.21.15.42:0/2142054388 wait complete. 2019-12-05T06:40:08.978 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:08.964+0000 7fc8d604e700 1 -- 172.21.15.42:0/2142054388 >> 172.21.15.42:0/2142054388 conn(0x7fc8d00aabe0 msgr2=0x7fc8d00a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:40:08.981 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:08.964+0000 7fc8d604e700 1 Processor -- start 2019-12-05T06:40:08.981 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:08.964+0000 7fc8d604e700 1 -- start start 2019-12-05T06:40:08.981 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:08.964+0000 7fc8d604e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc8d011ffc0 0x7fc8d00a6030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:40:08.981 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:08.964+0000 7fc8d604e700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8d011bfc0 con 0x7fc8d008f370 2019-12-05T06:40:08.982 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:08.964+0000 7fc8d604e700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8d008f7a0 con 0x7fc8d00a4ae0 2019-12-05T06:40:08.982 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:08.964+0000 7fc8d604e700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8d00a6ec0 con 0x7fc8d011ffc0 2019-12-05T06:40:08.982 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:08.964+0000 7fc8cffff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc8d011ffc0 0x7fc8d00a6030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:40:08.982 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:08.964+0000 7fc8cffff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc8d011ffc0 0x7fc8d00a6030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:60616/0 (socket says 172.21.15.42:60616) 2019-12-05T06:40:08.982 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:08.964+0000 7fc8cffff700 1 -- 172.21.15.42:0/3205793822 learned_addr learned my addr 172.21.15.42:0/3205793822 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:40:08.982 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:08.964+0000 7fc8cf7fe700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fc8d00a4ae0 0x7fc8d008eb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:60656/0 (socket says 172.21.15.42:60656) 2019-12-05T06:40:08.983 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:08.968+0000 7fc8cffff700 1 -- 172.21.15.42:0/3205793822 >> v1:172.21.15.42:6789/0 conn(0x7fc8d00a4ae0 legacy=0x7fc8d008eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:08.983 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:08.968+0000 7fc8cffff700 1 -- 172.21.15.42:0/3205793822 >> v1:172.21.15.40:6789/0 conn(0x7fc8d008f370 legacy=0x7fc8d0137c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:08.983 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:08.968+0000 7fc8ccff9700 1 -- 172.21.15.42:0/3205793822 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1549018671 0 0) 0x7fc8d008f7a0 con 0x7fc8d00a4ae0 2019-12-05T06:40:08.983 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:08.968+0000 7fc8cffff700 1 -- 172.21.15.42:0/3205793822 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8d005ae30 con 0x7fc8d011ffc0 2019-12-05T06:40:08.983 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:08.968+0000 7fc8cffff700 1 --2- 172.21.15.42:0/3205793822 >> v2:172.21.15.42:3300/0 conn(0x7fc8d011ffc0 0x7fc8d00a6030 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fc8c8000b20 tx=0x7fc8b800b6f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:08.983 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:08.968+0000 7fc8cffff700 1 --2- 172.21.15.42:0/3205793822 >> v2:172.21.15.42:3300/0 conn(0x7fc8d011ffc0 0x7fc8d00a6030 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fc8c8000b20 tx=0x7fc8b800b6f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:08.983 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:08.968+0000 7fc8cffff700 1 --2- 172.21.15.42:0/3205793822 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc8d011ffc0 0x7fc8d00a6030 secure :-1 s=READY pgs=2976 cs=0 l=1 rx=0x7fc8c8000b20 tx=0x7fc8b800b6f0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:40:08.984 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:08.968+0000 7fc8cffff700 1 --2- 172.21.15.42:0/3205793822 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc8d011ffc0 0x7fc8d00a6030 secure :-1 s=THROTTLE_DONE pgs=2976 cs=0 l=1 rx=0x7fc8c8000b20 tx=0x7fc8b800b6f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:08.984 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:08.968+0000 7fc8ccff9700 1 -- 172.21.15.42:0/3205793822 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc8c0002750 con 0x7fc8d011ffc0 2019-12-05T06:40:08.984 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:08.968+0000 7fc8cffff700 1 --2- 172.21.15.42:0/3205793822 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc8d011ffc0 0x7fc8d00a6030 secure :-1 s=THROTTLE_DONE pgs=2976 cs=0 l=1 rx=0x7fc8c8000b20 tx=0x7fc8b800b6f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:08.984 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:08.968+0000 7fc8cffff700 1 --2- 172.21.15.42:0/3205793822 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc8d011ffc0 0x7fc8d00a6030 secure :-1 s=THROTTLE_DONE pgs=2976 cs=0 l=1 rx=0x7fc8c8000b20 tx=0x7fc8b800b6f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:08.984 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:08.968+0000 7fc8ccff9700 1 -- 172.21.15.42:0/3205793822 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8c0006cc0 con 0x7fc8d011ffc0 2019-12-05T06:40:08.984 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:08.968+0000 7fc8ccff9700 1 -- 172.21.15.42:0/3205793822 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc8c0002d30 con 0x7fc8d011ffc0 2019-12-05T06:40:08.985 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:08.968+0000 7fc8d604e700 1 -- 172.21.15.42:0/3205793822 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8d00a6c00 con 0x7fc8d011ffc0 2019-12-05T06:40:08.985 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:08.968+0000 7fc8d604e700 1 -- 172.21.15.42:0/3205793822 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8d00a4f10 con 0x7fc8d011ffc0 2019-12-05T06:40:08.985 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:08.968+0000 7fc8cffff700 1 --2- 172.21.15.42:0/3205793822 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc8d011ffc0 0x7fc8d00a6030 secure :-1 s=THROTTLE_DONE pgs=2976 cs=0 l=1 rx=0x7fc8c8000b20 tx=0x7fc8b800b6f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:08.985 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:08.968+0000 7fc8ccff9700 1 -- 172.21.15.42:0/3205793822 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fc8c0009200 con 0x7fc8d011ffc0 2019-12-05T06:40:08.985 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:08.968+0000 7fc8cffff700 1 --2- 172.21.15.42:0/3205793822 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc8d011ffc0 0x7fc8d00a6030 secure :-1 s=THROTTLE_DONE pgs=2976 cs=0 l=1 rx=0x7fc8c8000b20 tx=0x7fc8b800b6f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:08.985 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:08.968+0000 7fc8ccff9700 1 --2- 172.21.15.42:0/3205793822 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc8b403abf0 0x7fc8b403d100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:40:08.985 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:08.968+0000 7fc8ccff9700 1 -- 172.21.15.42:0/3205793822 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 11621+0+0 (secure 0 0 0) 0x7fc8c004b1b0 con 0x7fc8d011ffc0 2019-12-05T06:40:08.986 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:08.968+0000 7fc8cf7fe700 1 --2- 172.21.15.42:0/3205793822 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc8b403abf0 0x7fc8b403d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:40:08.986 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:08.968+0000 7fc8cf7fe700 1 --2- 172.21.15.42:0/3205793822 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc8b403abf0 0x7fc8b403d100 crc :-1 s=READY pgs=1243 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:40:08.986 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:08.976+0000 7fc8d604e700 1 -- 172.21.15.42:0/3205793822 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8d005d110 con 0x7fc8d011ffc0 2019-12-05T06:40:08.987 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:08.980+0000 7fc8cffff700 1 --2- 172.21.15.42:0/3205793822 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc8d011ffc0 0x7fc8d00a6030 secure :-1 s=THROTTLE_DONE pgs=2976 cs=0 l=1 rx=0x7fc8c8000b20 tx=0x7fc8b800b6f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:08.988 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:08.980+0000 7fc8ccff9700 1 -- 172.21.15.42:0/3205793822 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fc8d00a6c00 con 0x7fc8d011ffc0 2019-12-05T06:40:09.159 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:09.152+0000 7fc8d604e700 1 -- 172.21.15.42:0/3205793822 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc8d005b8c0 con 0x7fc8d011ffc0 2019-12-05T06:40:09.163 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:09.152+0000 7fc8cffff700 1 --2- 172.21.15.42:0/3205793822 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc8d011ffc0 0x7fc8d00a6030 secure :-1 s=THROTTLE_DONE pgs=2976 cs=0 l=1 rx=0x7fc8c8000b20 tx=0x7fc8b800b6f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:09.163 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:09.152+0000 7fc8ccff9700 1 -- 172.21.15.42:0/3205793822 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 355 v355) v1 ==== 95+0+1775 (secure 0 0 0) 0x7fc8c00321f0 con 0x7fc8d011ffc0 2019-12-05T06:40:09.163 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 355 2019-12-05T06:40:09.163 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:09.156+0000 7fc8d604e700 1 -- 172.21.15.42:0/3205793822 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc8b403abf0 msgr2=0x7fc8b403d100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:09.164 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:09.156+0000 7fc8d604e700 1 --2- 172.21.15.42:0/3205793822 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc8b403abf0 0x7fc8b403d100 crc :-1 s=READY pgs=1243 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:40:09.164 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:09.156+0000 7fc8d604e700 1 -- 172.21.15.42:0/3205793822 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc8d011ffc0 msgr2=0x7fc8d00a6030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:09.164 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:09.156+0000 7fc8d604e700 1 --2- 172.21.15.42:0/3205793822 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc8d011ffc0 0x7fc8d00a6030 secure :-1 s=READY pgs=2976 cs=0 l=1 rx=0x7fc8c8000b20 tx=0x7fc8b800b6f0).stop 2019-12-05T06:40:09.164 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:09.156+0000 7fc8d604e700 1 -- 172.21.15.42:0/3205793822 shutdown_connections 2019-12-05T06:40:09.164 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:09.156+0000 7fc8d604e700 1 --2- 172.21.15.42:0/3205793822 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc8b403abf0 0x7fc8b403d100 unknown :-1 s=CLOSED pgs=1243 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:40:09.164 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:09.156+0000 7fc8d604e700 1 --2- 172.21.15.42:0/3205793822 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc8d011ffc0 0x7fc8d00a6030 unknown :-1 s=CLOSED pgs=2976 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:40:09.164 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:09.156+0000 7fc8d604e700 1 -- 172.21.15.42:0/3205793822 shutdown_connections 2019-12-05T06:40:09.165 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:09.156+0000 7fc8d604e700 1 -- 172.21.15.42:0/3205793822 wait complete. 2019-12-05T06:40:09.165 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:09.156+0000 7fc8d604e700 1 -- 172.21.15.42:0/3205793822 >> 172.21.15.42:0/3205793822 conn(0x7fc8d00a8940 msgr2=0x7fc8d00a00a0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:40:09.165 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:40:09.168 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":355,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":355,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:40:05.188099+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":114,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11868},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11868":{"gid":11868,"name":"a","rank":0,"incarnation":355,"state":"up:replay","state_seq":2,"addr":"172.21.15.42:6835/406829134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":406829134},{"type":"v1","addr":"172.21.15.42:6835","nonce":406829134}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:40:09.172 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2019-12-05T06:40:11.173 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:40:11.288 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:11.272+0000 7fa4e2829700 1 Processor -- start 2019-12-05T06:40:11.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:11.272+0000 7fa4e2829700 1 -- start start 2019-12-05T06:40:11.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:11.272+0000 7fa4e2829700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa4dc08df70 0x7fa4dc09f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:40:11.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:11.272+0000 7fa4e2829700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4dc09fc50 con 0x7fa4dc09f820 2019-12-05T06:40:11.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:11.272+0000 7fa4e2829700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4dc0a0700 con 0x7fa4dc0a7730 2019-12-05T06:40:11.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:11.272+0000 7fa4e2829700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4dc0a0af0 con 0x7fa4dc08df70 2019-12-05T06:40:11.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:11.272+0000 7fa4e0dc6700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fa4dc0a7730 0x7fa4dc07e710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:60668/0 (socket says 172.21.15.42:60668) 2019-12-05T06:40:11.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:11.272+0000 7fa4e0dc6700 1 -- 172.21.15.42:0/899370394 learned_addr learned my addr 172.21.15.42:0/899370394 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:40:11.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:11.272+0000 7fa4dbfff700 1 --2- 172.21.15.42:0/899370394 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa4dc08df70 0x7fa4dc09f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:40:11.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:11.272+0000 7fa4db7fe700 1 -- 172.21.15.42:0/899370394 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1464374609 0 0) 0x7fa4dc0a0700 con 0x7fa4dc0a7730 2019-12-05T06:40:11.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:11.272+0000 7fa4db7fe700 1 -- 172.21.15.42:0/899370394 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4c80030b0 con 0x7fa4dc0a7730 2019-12-05T06:40:11.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:11.276+0000 7fa4db7fe700 1 -- 172.21.15.42:0/899370394 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2887853017 0 0) 0x7fa4dc09fc50 con 0x7fa4dc09f820 2019-12-05T06:40:11.291 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:11.276+0000 7fa4db7fe700 1 -- 172.21.15.42:0/899370394 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4dc0a0700 con 0x7fa4dc09f820 2019-12-05T06:40:11.291 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:11.276+0000 7fa4dbfff700 1 -- 172.21.15.42:0/899370394 >> v1:172.21.15.42:6789/0 conn(0x7fa4dc0a7730 legacy=0x7fa4dc07e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:11.291 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:11.276+0000 7fa4db7fe700 1 -- 172.21.15.42:0/899370394 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 16152606 0 0) 0x7fa4c80030b0 con 0x7fa4dc0a7730 2019-12-05T06:40:11.291 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:11.276+0000 7fa4dbfff700 1 -- 172.21.15.42:0/899370394 >> v1:172.21.15.40:6789/0 conn(0x7fa4dc09f820 legacy=0x7fa4dc14efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:11.291 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:11.276+0000 7fa4dbfff700 1 -- 172.21.15.42:0/899370394 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4dc07ef10 con 0x7fa4dc08df70 2019-12-05T06:40:11.291 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:11.276+0000 7fa4dbfff700 1 --2- 172.21.15.42:0/899370394 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa4dc08df70 0x7fa4dc09f2d0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fa4c4008b10 tx=0x7fa4d4001950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:11.291 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:11.276+0000 7fa4dbfff700 1 --2- 172.21.15.42:0/899370394 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa4dc08df70 0x7fa4dc09f2d0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fa4c4008b10 tx=0x7fa4d4001950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:11.292 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:11.276+0000 7fa4dbfff700 1 --2- 172.21.15.42:0/899370394 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa4dc08df70 0x7fa4dc09f2d0 secure :-1 s=READY pgs=656 cs=0 l=1 rx=0x7fa4c4008b10 tx=0x7fa4d4001950).ready entity=mon.2 client_cookie=86344dba8ee73c9d server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:40:11.292 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:11.276+0000 7fa4dbfff700 1 --2- 172.21.15.42:0/899370394 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa4dc08df70 0x7fa4dc09f2d0 secure :-1 s=THROTTLE_DONE pgs=656 cs=0 l=1 rx=0x7fa4c4008b10 tx=0x7fa4d4001950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:11.292 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:11.276+0000 7fa4db7fe700 1 -- 172.21.15.42:0/899370394 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa4d4001750 con 0x7fa4dc08df70 2019-12-05T06:40:11.292 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:11.276+0000 7fa4dbfff700 1 --2- 172.21.15.42:0/899370394 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa4dc08df70 0x7fa4dc09f2d0 secure :-1 s=THROTTLE_DONE pgs=656 cs=0 l=1 rx=0x7fa4c4008b10 tx=0x7fa4d4001950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:11.292 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:11.276+0000 7fa4db7fe700 1 -- 172.21.15.42:0/899370394 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4d4001560 con 0x7fa4dc08df70 2019-12-05T06:40:11.292 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:11.276+0000 7fa4dbfff700 1 --2- 172.21.15.42:0/899370394 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa4dc08df70 0x7fa4dc09f2d0 secure :-1 s=THROTTLE_DONE pgs=656 cs=0 l=1 rx=0x7fa4c4008b10 tx=0x7fa4d4001950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:11.292 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:11.276+0000 7fa4db7fe700 1 -- 172.21.15.42:0/899370394 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa4c4011ac0 con 0x7fa4dc08df70 2019-12-05T06:40:11.293 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:11.276+0000 7fa4e2829700 1 -- 172.21.15.42:0/899370394 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa4dc08df70 msgr2=0x7fa4dc09f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:11.293 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:11.276+0000 7fa4e2829700 1 --2- 172.21.15.42:0/899370394 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa4dc08df70 0x7fa4dc09f2d0 secure :-1 s=READY pgs=656 cs=0 l=1 rx=0x7fa4c4008b10 tx=0x7fa4d4001950).stop 2019-12-05T06:40:11.293 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:11.276+0000 7fa4e2829700 1 -- 172.21.15.42:0/899370394 shutdown_connections 2019-12-05T06:40:11.293 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:11.276+0000 7fa4e2829700 1 --2- 172.21.15.42:0/899370394 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa4dc08df70 0x7fa4dc09f2d0 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:40:11.293 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:11.276+0000 7fa4e2829700 1 -- 172.21.15.42:0/899370394 shutdown_connections 2019-12-05T06:40:11.293 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:11.276+0000 7fa4e2829700 1 -- 172.21.15.42:0/899370394 wait complete. 2019-12-05T06:40:11.293 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:11.276+0000 7fa4e2829700 1 -- 172.21.15.42:0/899370394 >> 172.21.15.42:0/899370394 conn(0x7fa4dc0aabd0 msgr2=0x7fa4dc0a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:40:11.294 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:11.276+0000 7fa4e2829700 1 Processor -- start 2019-12-05T06:40:11.294 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:11.276+0000 7fa4e2829700 1 -- start start 2019-12-05T06:40:11.294 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:11.280+0000 7fa4e2829700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa4dc11da60 0x7fa4dc11aae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:40:11.294 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:11.280+0000 7fa4e2829700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fa4dc11c0d0 0x7fa4dc132640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:40:11.294 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:11.280+0000 7fa4e2829700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4dc11a490 con 0x7fa4dc11b030 2019-12-05T06:40:11.294 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:11.280+0000 7fa4e2829700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4dc11de90 con 0x7fa4dc11c0d0 2019-12-05T06:40:11.294 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:11.280+0000 7fa4e2829700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4dc11a840 con 0x7fa4dc11da60 2019-12-05T06:40:11.295 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:11.280+0000 7fa4e15c7700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fa4dc11c0d0 0x7fa4dc132640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:40:11.295 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:11.280+0000 7fa4dbfff700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fa4dc11b030 0x7fa4dc11b8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:41882/0 (socket says 172.21.15.42:41882) 2019-12-05T06:40:11.295 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:11.280+0000 7fa4dbfff700 1 -- 172.21.15.42:0/1616130640 learned_addr learned my addr 172.21.15.42:0/1616130640 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:40:11.295 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:11.280+0000 7fa4e0dc6700 1 --2- 172.21.15.42:0/1616130640 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa4dc11da60 0x7fa4dc11aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:40:11.295 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:11.280+0000 7fa4d97fa700 1 -- 172.21.15.42:0/1616130640 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2076063447 0 0) 0x7fa4dc11a490 con 0x7fa4dc11b030 2019-12-05T06:40:11.295 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:11.280+0000 7fa4d97fa700 1 -- 172.21.15.42:0/1616130640 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4c00030b0 con 0x7fa4dc11b030 2019-12-05T06:40:11.295 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:11.280+0000 7fa4e15c7700 1 -- 172.21.15.42:0/1616130640 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa4dc11da60 msgr2=0x7fa4dc11aae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:11.296 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:11.280+0000 7fa4e15c7700 1 --2- 172.21.15.42:0/1616130640 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa4dc11da60 0x7fa4dc11aae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:40:11.296 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:11.280+0000 7fa4e15c7700 1 -- 172.21.15.42:0/1616130640 >> v1:172.21.15.40:6789/0 conn(0x7fa4dc11b030 legacy=0x7fa4dc11b8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:11.296 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:11.280+0000 7fa4e15c7700 1 -- 172.21.15.42:0/1616130640 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4dc116fc0 con 0x7fa4dc11c0d0 2019-12-05T06:40:11.296 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:11.280+0000 7fa4e15c7700 1 --2- 172.21.15.42:0/1616130640 >> v2:172.21.15.42:3300/0 conn(0x7fa4dc11c0d0 0x7fa4dc132640 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fa4cc000e90 tx=0x7fa4cc0107a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:11.303 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:11.280+0000 7fa4e15c7700 1 --2- 172.21.15.42:0/1616130640 >> v2:172.21.15.42:3300/0 conn(0x7fa4dc11c0d0 0x7fa4dc132640 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fa4cc000e90 tx=0x7fa4cc0107a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:11.303 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:11.280+0000 7fa4e15c7700 1 --2- 172.21.15.42:0/1616130640 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa4dc11c0d0 0x7fa4dc132640 secure :-1 s=READY pgs=2978 cs=0 l=1 rx=0x7fa4cc000e90 tx=0x7fa4cc0107a0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:40:11.303 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:11.280+0000 7fa4e15c7700 1 --2- 172.21.15.42:0/1616130640 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa4dc11c0d0 0x7fa4dc132640 secure :-1 s=THROTTLE_DONE pgs=2978 cs=0 l=1 rx=0x7fa4cc000e90 tx=0x7fa4cc0107a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:11.303 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:11.280+0000 7fa4d97fa700 1 -- 172.21.15.42:0/1616130640 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa4cc006a40 con 0x7fa4dc11c0d0 2019-12-05T06:40:11.304 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:11.280+0000 7fa4e15c7700 1 --2- 172.21.15.42:0/1616130640 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa4dc11c0d0 0x7fa4dc132640 secure :-1 s=THROTTLE_DONE pgs=2978 cs=0 l=1 rx=0x7fa4cc000e90 tx=0x7fa4cc0107a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:11.304 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:11.280+0000 7fa4e15c7700 1 --2- 172.21.15.42:0/1616130640 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa4dc11c0d0 0x7fa4dc132640 secure :-1 s=THROTTLE_DONE pgs=2978 cs=0 l=1 rx=0x7fa4cc000e90 tx=0x7fa4cc0107a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:11.304 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:11.280+0000 7fa4d97fa700 1 -- 172.21.15.42:0/1616130640 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4cc006660 con 0x7fa4dc11c0d0 2019-12-05T06:40:11.304 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:11.280+0000 7fa4e2829700 1 -- 172.21.15.42:0/1616130640 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4dc132b90 con 0x7fa4dc11c0d0 2019-12-05T06:40:11.304 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:11.280+0000 7fa4d97fa700 1 -- 172.21.15.42:0/1616130640 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa4cc007640 con 0x7fa4dc11c0d0 2019-12-05T06:40:11.304 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:11.280+0000 7fa4e2829700 1 -- 172.21.15.42:0/1616130640 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4dc133140 con 0x7fa4dc11c0d0 2019-12-05T06:40:11.304 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:11.280+0000 7fa4e15c7700 1 --2- 172.21.15.42:0/1616130640 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa4dc11c0d0 0x7fa4dc132640 secure :-1 s=THROTTLE_DONE pgs=2978 cs=0 l=1 rx=0x7fa4cc000e90 tx=0x7fa4cc0107a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:11.305 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:11.280+0000 7fa4d97fa700 1 -- 172.21.15.42:0/1616130640 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fa4cc007960 con 0x7fa4dc11c0d0 2019-12-05T06:40:11.305 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:11.280+0000 7fa4d97fa700 1 --2- 172.21.15.42:0/1616130640 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa4c003b230 0x7fa4c003d740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:40:11.305 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:11.280+0000 7fa4e0dc6700 1 --2- 172.21.15.42:0/1616130640 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa4c003b230 0x7fa4c003d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:40:11.305 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:11.280+0000 7fa4e15c7700 1 --2- 172.21.15.42:0/1616130640 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa4dc11c0d0 0x7fa4dc132640 secure :-1 s=THROTTLE_DONE pgs=2978 cs=0 l=1 rx=0x7fa4cc000e90 tx=0x7fa4cc0107a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:11.305 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:11.280+0000 7fa4d97fa700 1 -- 172.21.15.42:0/1616130640 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 11621+0+0 (secure 0 0 0) 0x7fa4cc031c20 con 0x7fa4dc11c0d0 2019-12-05T06:40:11.305 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:11.280+0000 7fa4e0dc6700 1 --2- 172.21.15.42:0/1616130640 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa4c003b230 0x7fa4c003d740 crc :-1 s=READY pgs=1244 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:40:11.306 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:11.288+0000 7fa4e2829700 1 -- 172.21.15.42:0/1616130640 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4dc05d100 con 0x7fa4dc11c0d0 2019-12-05T06:40:11.306 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:11.292+0000 7fa4e15c7700 1 --2- 172.21.15.42:0/1616130640 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa4dc11c0d0 0x7fa4dc132640 secure :-1 s=THROTTLE_DONE pgs=2978 cs=0 l=1 rx=0x7fa4cc000e90 tx=0x7fa4cc0107a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:11.306 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:11.292+0000 7fa4d97fa700 1 -- 172.21.15.42:0/1616130640 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fa4cc031e80 con 0x7fa4dc11c0d0 2019-12-05T06:40:11.460 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:11.452+0000 7fa4e2829700 1 -- 172.21.15.42:0/1616130640 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa4dc05b8b0 con 0x7fa4dc11c0d0 2019-12-05T06:40:11.461 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:11.456+0000 7fa4e15c7700 1 --2- 172.21.15.42:0/1616130640 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa4dc11c0d0 0x7fa4dc132640 secure :-1 s=THROTTLE_DONE pgs=2978 cs=0 l=1 rx=0x7fa4cc000e90 tx=0x7fa4cc0107a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:11.461 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:11.456+0000 7fa4d97fa700 1 -- 172.21.15.42:0/1616130640 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 358 v358) v1 ==== 95+0+1782 (secure 0 0 0) 0x7fa4cc007c70 con 0x7fa4dc11c0d0 2019-12-05T06:40:11.461 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 358 2019-12-05T06:40:11.461 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:40:11.462 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":358,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":358,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:40:11.226184+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":114,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11868},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11868":{"gid":11868,"name":"a","rank":0,"incarnation":355,"state":"up:clientreplay","state_seq":11,"addr":"172.21.15.42:6835/406829134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":406829134},{"type":"v1","addr":"172.21.15.42:6835","nonce":406829134}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:40:11.463 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:11.456+0000 7fa4e2829700 1 -- 172.21.15.42:0/1616130640 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa4c003b230 msgr2=0x7fa4c003d740 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:11.463 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:11.456+0000 7fa4e2829700 1 --2- 172.21.15.42:0/1616130640 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa4c003b230 0x7fa4c003d740 crc :-1 s=READY pgs=1244 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:40:11.463 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:11.456+0000 7fa4e2829700 1 -- 172.21.15.42:0/1616130640 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa4dc11c0d0 msgr2=0x7fa4dc132640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:11.464 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:11.456+0000 7fa4e2829700 1 --2- 172.21.15.42:0/1616130640 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa4dc11c0d0 0x7fa4dc132640 secure :-1 s=READY pgs=2978 cs=0 l=1 rx=0x7fa4cc000e90 tx=0x7fa4cc0107a0).stop 2019-12-05T06:40:11.464 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:11.456+0000 7fa4e2829700 1 -- 172.21.15.42:0/1616130640 shutdown_connections 2019-12-05T06:40:11.464 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:11.456+0000 7fa4e2829700 1 --2- 172.21.15.42:0/1616130640 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa4c003b230 0x7fa4c003d740 unknown :-1 s=CLOSED pgs=1244 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:40:11.464 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:11.456+0000 7fa4e2829700 1 --2- 172.21.15.42:0/1616130640 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa4dc11c0d0 0x7fa4dc132640 unknown :-1 s=CLOSED pgs=2978 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:40:11.464 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:11.456+0000 7fa4e2829700 1 --2- 172.21.15.42:0/1616130640 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa4dc11da60 0x7fa4dc11aae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:40:11.464 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:11.456+0000 7fa4e2829700 1 -- 172.21.15.42:0/1616130640 shutdown_connections 2019-12-05T06:40:11.465 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:11.456+0000 7fa4e2829700 1 -- 172.21.15.42:0/1616130640 wait complete. 2019-12-05T06:40:11.465 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:11.456+0000 7fa4e2829700 1 -- 172.21.15.42:0/1616130640 >> 172.21.15.42:0/1616130640 conn(0x7fa4dc0a7f80 msgr2=0x7fa4dc08eb60 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:40:11.475 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2019-12-05T06:40:13.476 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:40:13.575 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:13.564+0000 7fdf8a8f9700 1 Processor -- start 2019-12-05T06:40:13.575 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:13.564+0000 7fdf8a8f9700 1 -- start start 2019-12-05T06:40:13.575 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:13.564+0000 7fdf8a8f9700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fdf8408df80 0x7fdf8409f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:40:13.576 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:13.564+0000 7fdf8a8f9700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdf8409f830 0x7fdf8414efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:40:13.576 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:13.564+0000 7fdf8a8f9700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf8409fc60 con 0x7fdf840a7740 2019-12-05T06:40:13.576 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:13.564+0000 7fdf8a8f9700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf840a0b00 con 0x7fdf8408df80 2019-12-05T06:40:13.576 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:13.564+0000 7fdf8a8f9700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf8408e5b0 con 0x7fdf8409f830 2019-12-05T06:40:13.576 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:13.564+0000 7fdf88e96700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fdf840a7740 0x7fdf8407e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:60720/0 (socket says 172.21.15.42:60720) 2019-12-05T06:40:13.576 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:13.564+0000 7fdf88e96700 1 -- 172.21.15.42:0/1397076336 learned_addr learned my addr 172.21.15.42:0/1397076336 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:40:13.577 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:13.564+0000 7fdf7bfff700 1 --2- 172.21.15.42:0/1397076336 >> v2:172.21.15.40:3300/0 conn(0x7fdf8408df80 0x7fdf8409f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:40:13.577 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:13.564+0000 7fdf89697700 1 --2- 172.21.15.42:0/1397076336 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdf8409f830 0x7fdf8414efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:40:13.577 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:13.564+0000 7fdf7b7fe700 1 -- 172.21.15.42:0/1397076336 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2531991426 0 0) 0x7fdf8409fc60 con 0x7fdf840a7740 2019-12-05T06:40:13.577 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:13.564+0000 7fdf7b7fe700 1 -- 172.21.15.42:0/1397076336 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf6c0030b0 con 0x7fdf840a7740 2019-12-05T06:40:13.578 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:13.564+0000 7fdf7b7fe700 1 -- 172.21.15.42:0/1397076336 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1412728339 0 0) 0x7fdf6c0030b0 con 0x7fdf840a7740 2019-12-05T06:40:13.578 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:13.564+0000 7fdf7bfff700 1 -- 172.21.15.42:0/1397076336 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdf8409f830 msgr2=0x7fdf8414efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:40:13.578 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:13.564+0000 7fdf7bfff700 1 --2- 172.21.15.42:0/1397076336 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdf8409f830 0x7fdf8414efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:40:13.578 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:13.564+0000 7fdf7bfff700 1 -- 172.21.15.42:0/1397076336 >> v1:172.21.15.42:6789/0 conn(0x7fdf840a7740 legacy=0x7fdf8407e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:13.578 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:13.564+0000 7fdf7bfff700 1 -- 172.21.15.42:0/1397076336 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf840a0710 con 0x7fdf8408df80 2019-12-05T06:40:13.578 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:13.564+0000 7fdf89697700 1 --2- 172.21.15.42:0/1397076336 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdf8409f830 0x7fdf8414efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:40:13.579 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:13.564+0000 7fdf7bfff700 1 --2- 172.21.15.42:0/1397076336 >> v2:172.21.15.40:3300/0 conn(0x7fdf8408df80 0x7fdf8409f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fdf74008790 tx=0x7fdf800019d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:13.579 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:13.568+0000 7fdf7bfff700 1 --2- 172.21.15.42:0/1397076336 >> v2:172.21.15.40:3300/0 conn(0x7fdf8408df80 0x7fdf8409f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fdf74008790 tx=0x7fdf800019d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:13.579 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:13.568+0000 7fdf7bfff700 1 --2- 172.21.15.42:0/1397076336 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf8408df80 0x7fdf8409f2e0 secure :-1 s=READY pgs=2259 cs=0 l=1 rx=0x7fdf74008790 tx=0x7fdf800019d0).ready entity=mon.0 client_cookie=6320a3c2db3d8ba5 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:40:13.579 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:13.568+0000 7fdf7bfff700 1 --2- 172.21.15.42:0/1397076336 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf8408df80 0x7fdf8409f2e0 secure :-1 s=THROTTLE_DONE pgs=2259 cs=0 l=1 rx=0x7fdf74008790 tx=0x7fdf800019d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:13.579 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:13.568+0000 7fdf7b7fe700 1 -- 172.21.15.42:0/1397076336 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdf80001750 con 0x7fdf8408df80 2019-12-05T06:40:13.579 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:13.568+0000 7fdf7bfff700 1 --2- 172.21.15.42:0/1397076336 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf8408df80 0x7fdf8409f2e0 secure :-1 s=THROTTLE_DONE pgs=2259 cs=0 l=1 rx=0x7fdf74008790 tx=0x7fdf800019d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:13.580 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:13.568+0000 7fdf7b7fe700 1 -- 172.21.15.42:0/1397076336 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf80001560 con 0x7fdf8408df80 2019-12-05T06:40:13.580 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:13.568+0000 7fdf7bfff700 1 --2- 172.21.15.42:0/1397076336 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf8408df80 0x7fdf8409f2e0 secure :-1 s=THROTTLE_DONE pgs=2259 cs=0 l=1 rx=0x7fdf74008790 tx=0x7fdf800019d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:13.580 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:13.568+0000 7fdf7b7fe700 1 -- 172.21.15.42:0/1397076336 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdf74008da0 con 0x7fdf8408df80 2019-12-05T06:40:13.580 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:13.568+0000 7fdf8a8f9700 1 -- 172.21.15.42:0/1397076336 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf8408df80 msgr2=0x7fdf8409f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:13.580 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:13.568+0000 7fdf8a8f9700 1 --2- 172.21.15.42:0/1397076336 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf8408df80 0x7fdf8409f2e0 secure :-1 s=READY pgs=2259 cs=0 l=1 rx=0x7fdf74008790 tx=0x7fdf800019d0).stop 2019-12-05T06:40:13.580 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:13.568+0000 7fdf8a8f9700 1 -- 172.21.15.42:0/1397076336 shutdown_connections 2019-12-05T06:40:13.581 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:13.568+0000 7fdf8a8f9700 1 --2- 172.21.15.42:0/1397076336 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdf8409f830 0x7fdf8414efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:40:13.581 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:13.568+0000 7fdf8a8f9700 1 --2- 172.21.15.42:0/1397076336 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf8408df80 0x7fdf8409f2e0 unknown :-1 s=CLOSED pgs=2259 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:40:13.581 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:13.568+0000 7fdf8a8f9700 1 -- 172.21.15.42:0/1397076336 shutdown_connections 2019-12-05T06:40:13.581 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:13.568+0000 7fdf8a8f9700 1 -- 172.21.15.42:0/1397076336 wait complete. 2019-12-05T06:40:13.581 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:13.568+0000 7fdf8a8f9700 1 -- 172.21.15.42:0/1397076336 >> 172.21.15.42:0/1397076336 conn(0x7fdf840aabe0 msgr2=0x7fdf840a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:40:13.584 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:13.572+0000 7fdf8a8f9700 1 Processor -- start 2019-12-05T06:40:13.587 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:13.572+0000 7fdf8a8f9700 1 -- start start 2019-12-05T06:40:13.587 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:13.572+0000 7fdf8a8f9700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fdf840a9ad0 0x7fdf84117810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:40:13.587 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:13.572+0000 7fdf8a8f9700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdf84122470 0x7fdf841228a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:40:13.587 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:13.572+0000 7fdf8a8f9700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf84065b50 con 0x7fdf84120450 2019-12-05T06:40:13.587 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:13.572+0000 7fdf8a8f9700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf84117030 con 0x7fdf840a9ad0 2019-12-05T06:40:13.588 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:13.572+0000 7fdf8a8f9700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf841172d0 con 0x7fdf84122470 2019-12-05T06:40:13.588 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:13.572+0000 7fdf88e96700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fdf840a9ad0 0x7fdf84117810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:40:13.588 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:13.572+0000 7fdf7bfff700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdf84122470 0x7fdf841228a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:40:13.588 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:13.572+0000 7fdf89697700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fdf84120450 0x7fdf84120880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:41936/0 (socket says 172.21.15.42:41936) 2019-12-05T06:40:13.588 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:13.572+0000 7fdf89697700 1 -- 172.21.15.42:0/1142396217 learned_addr learned my addr 172.21.15.42:0/1142396217 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:40:13.588 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:13.572+0000 7fdf88e96700 1 -- 172.21.15.42:0/1142396217 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdf84122470 msgr2=0x7fdf841228a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:13.589 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:13.572+0000 7fdf88e96700 1 --2- 172.21.15.42:0/1142396217 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdf84122470 0x7fdf841228a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:40:13.589 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:13.572+0000 7fdf88e96700 1 -- 172.21.15.42:0/1142396217 >> v1:172.21.15.40:6789/0 conn(0x7fdf84120450 legacy=0x7fdf84120880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:13.589 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:13.572+0000 7fdf88e96700 1 -- 172.21.15.42:0/1142396217 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf84116c10 con 0x7fdf840a9ad0 2019-12-05T06:40:13.589 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:13.572+0000 7fdf797fa700 1 -- 172.21.15.42:0/1142396217 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 361580902 0 0) 0x7fdf84065b50 con 0x7fdf84120450 2019-12-05T06:40:13.589 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:13.572+0000 7fdf7bfff700 1 --2- 172.21.15.42:0/1142396217 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdf84122470 0x7fdf841228a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:40:13.589 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:13.572+0000 7fdf88e96700 1 --2- 172.21.15.42:0/1142396217 >> v2:172.21.15.40:3300/0 conn(0x7fdf840a9ad0 0x7fdf84117810 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fdf841171f0 tx=0x7fdf800109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:13.590 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:13.572+0000 7fdf88e96700 1 --2- 172.21.15.42:0/1142396217 >> v2:172.21.15.40:3300/0 conn(0x7fdf840a9ad0 0x7fdf84117810 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fdf841171f0 tx=0x7fdf800109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:13.590 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:13.572+0000 7fdf88e96700 1 --2- 172.21.15.42:0/1142396217 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf840a9ad0 0x7fdf84117810 secure :-1 s=READY pgs=2261 cs=0 l=1 rx=0x7fdf841171f0 tx=0x7fdf800109e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:40:13.590 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:13.572+0000 7fdf88e96700 1 --2- 172.21.15.42:0/1142396217 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf840a9ad0 0x7fdf84117810 secure :-1 s=THROTTLE_DONE pgs=2261 cs=0 l=1 rx=0x7fdf841171f0 tx=0x7fdf800109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:13.590 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:13.572+0000 7fdf797fa700 1 -- 172.21.15.42:0/1142396217 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdf8000c890 con 0x7fdf840a9ad0 2019-12-05T06:40:13.590 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:13.572+0000 7fdf88e96700 1 --2- 172.21.15.42:0/1142396217 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf840a9ad0 0x7fdf84117810 secure :-1 s=THROTTLE_DONE pgs=2261 cs=0 l=1 rx=0x7fdf841171f0 tx=0x7fdf800109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:13.590 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:13.572+0000 7fdf88e96700 1 --2- 172.21.15.42:0/1142396217 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf840a9ad0 0x7fdf84117810 secure :-1 s=THROTTLE_DONE pgs=2261 cs=0 l=1 rx=0x7fdf841171f0 tx=0x7fdf800109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:13.590 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:13.576+0000 7fdf797fa700 1 -- 172.21.15.42:0/1142396217 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf8000c4b0 con 0x7fdf840a9ad0 2019-12-05T06:40:13.591 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:13.576+0000 7fdf8a8f9700 1 -- 172.21.15.42:0/1142396217 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf84122df0 con 0x7fdf840a9ad0 2019-12-05T06:40:13.591 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:13.576+0000 7fdf797fa700 1 -- 172.21.15.42:0/1142396217 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdf80004430 con 0x7fdf840a9ad0 2019-12-05T06:40:13.591 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:13.576+0000 7fdf8a8f9700 1 -- 172.21.15.42:0/1142396217 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf84121080 con 0x7fdf840a9ad0 2019-12-05T06:40:13.591 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:13.576+0000 7fdf88e96700 1 --2- 172.21.15.42:0/1142396217 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf840a9ad0 0x7fdf84117810 secure :-1 s=THROTTLE_DONE pgs=2261 cs=0 l=1 rx=0x7fdf841171f0 tx=0x7fdf800109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:13.591 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:13.576+0000 7fdf88e96700 1 --2- 172.21.15.42:0/1142396217 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf840a9ad0 0x7fdf84117810 secure :-1 s=THROTTLE_DONE pgs=2261 cs=0 l=1 rx=0x7fdf841171f0 tx=0x7fdf800109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:13.592 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:13.576+0000 7fdf797fa700 1 -- 172.21.15.42:0/1142396217 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fdf80004700 con 0x7fdf840a9ad0 2019-12-05T06:40:13.592 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:13.576+0000 7fdf797fa700 1 --2- 172.21.15.42:0/1142396217 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fdf6403acf0 0x7fdf6403d200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:40:13.592 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:13.576+0000 7fdf797fa700 1 -- 172.21.15.42:0/1142396217 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 11621+0+0 (secure 0 0 0) 0x7fdf8004abc0 con 0x7fdf840a9ad0 2019-12-05T06:40:13.592 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:13.576+0000 7fdf7bfff700 1 --2- 172.21.15.42:0/1142396217 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fdf6403acf0 0x7fdf6403d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:40:13.592 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:13.576+0000 7fdf7bfff700 1 --2- 172.21.15.42:0/1142396217 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fdf6403acf0 0x7fdf6403d200 crc :-1 s=READY pgs=1246 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:40:13.593 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:13.580+0000 7fdf8a8f9700 1 -- 172.21.15.42:0/1142396217 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf8405d110 con 0x7fdf840a9ad0 2019-12-05T06:40:13.594 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:13.588+0000 7fdf88e96700 1 --2- 172.21.15.42:0/1142396217 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf840a9ad0 0x7fdf84117810 secure :-1 s=THROTTLE_DONE pgs=2261 cs=0 l=1 rx=0x7fdf841171f0 tx=0x7fdf800109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:13.594 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:13.588+0000 7fdf797fa700 1 -- 172.21.15.42:0/1142396217 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fdf80031c00 con 0x7fdf840a9ad0 2019-12-05T06:40:13.769 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:13.764+0000 7fdf8a8f9700 1 -- 172.21.15.42:0/1142396217 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdf8405b8c0 con 0x7fdf840a9ad0 2019-12-05T06:40:13.770 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:13.764+0000 7fdf88e96700 1 --2- 172.21.15.42:0/1142396217 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf840a9ad0 0x7fdf84117810 secure :-1 s=THROTTLE_DONE pgs=2261 cs=0 l=1 rx=0x7fdf841171f0 tx=0x7fdf800109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:13.771 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:13.764+0000 7fdf797fa700 1 -- 172.21.15.42:0/1142396217 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 359 v359) v1 ==== 95+0+1776 (secure 0 0 0) 0x7fdf80031e60 con 0x7fdf840a9ad0 2019-12-05T06:40:13.773 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 359 2019-12-05T06:40:13.773 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:40:13.773 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":359,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":359,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:40:13.247006+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":114,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11868},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11868":{"gid":11868,"name":"a","rank":0,"incarnation":355,"state":"up:active","state_seq":12,"addr":"172.21.15.42:6835/406829134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":406829134},{"type":"v1","addr":"172.21.15.42:6835","nonce":406829134}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:40:13.777 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:13.768+0000 7fdf8a8f9700 1 -- 172.21.15.42:0/1142396217 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fdf6403acf0 msgr2=0x7fdf6403d200 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:13.777 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:13.768+0000 7fdf8a8f9700 1 --2- 172.21.15.42:0/1142396217 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fdf6403acf0 0x7fdf6403d200 crc :-1 s=READY pgs=1246 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:40:13.777 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:13.768+0000 7fdf8a8f9700 1 -- 172.21.15.42:0/1142396217 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf840a9ad0 msgr2=0x7fdf84117810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:13.777 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:13.768+0000 7fdf8a8f9700 1 --2- 172.21.15.42:0/1142396217 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf840a9ad0 0x7fdf84117810 secure :-1 s=READY pgs=2261 cs=0 l=1 rx=0x7fdf841171f0 tx=0x7fdf800109e0).stop 2019-12-05T06:40:13.777 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:13.768+0000 7fdf8a8f9700 1 -- 172.21.15.42:0/1142396217 shutdown_connections 2019-12-05T06:40:13.777 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:13.768+0000 7fdf8a8f9700 1 --2- 172.21.15.42:0/1142396217 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fdf6403acf0 0x7fdf6403d200 unknown :-1 s=CLOSED pgs=1246 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:40:13.778 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:13.768+0000 7fdf8a8f9700 1 --2- 172.21.15.42:0/1142396217 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf840a9ad0 0x7fdf84117810 unknown :-1 s=CLOSED pgs=2261 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:40:13.778 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:13.768+0000 7fdf8a8f9700 1 --2- 172.21.15.42:0/1142396217 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdf84122470 0x7fdf841228a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:40:13.778 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:13.768+0000 7fdf8a8f9700 1 -- 172.21.15.42:0/1142396217 shutdown_connections 2019-12-05T06:40:13.778 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:13.768+0000 7fdf8a8f9700 1 -- 172.21.15.42:0/1142396217 wait complete. 2019-12-05T06:40:13.778 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:13.768+0000 7fdf8a8f9700 1 -- 172.21.15.42:0/1142396217 >> 172.21.15.42:0/1142396217 conn(0x7fdf840a82a0 msgr2=0x7fdf8411bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:40:13.785 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a has gained rank=0, replacing gid=8009 2019-12-05T06:40:13.785 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 6 secs before reviving mds.b 2019-12-05T06:40:14.941 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:40:14.945 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:40:19.786 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.b 2019-12-05T06:40:19.787 INFO:tasks.ceph.mds.b:Restarting daemon 2019-12-05T06:40:19.787 INFO:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2019-12-05T06:40:19.791 INFO:tasks.ceph.mds.b:Started 2019-12-05T06:40:19.791 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:40:19.850 INFO:tasks.ceph.mds.b.smithi040.stdout:starting mds.b at 2019-12-05T06:40:19.895 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:19.880+0000 7f0ad1b1f700 1 Processor -- start 2019-12-05T06:40:19.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:19.880+0000 7f0ad1b1f700 1 -- start start 2019-12-05T06:40:19.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:19.884+0000 7f0ad1b1f700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0acc0a7740 0x7f0acc08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:40:19.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:19.884+0000 7f0ad1b1f700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0acc09fae0 0x7f0acc07eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:40:19.906 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:19.884+0000 7f0ad1b1f700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0acc0a0710 con 0x7f0acc07e720 2019-12-05T06:40:19.913 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:19.884+0000 7f0ad1b1f700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0acc0a0b00 con 0x7f0acc09fae0 2019-12-05T06:40:19.913 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:19.884+0000 7f0ad1b1f700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0acc08e5b0 con 0x7f0acc0a7740 2019-12-05T06:40:19.913 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:19.884+0000 7f0ad08bd700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0acc09fae0 0x7f0acc07eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:40:19.913 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:19.884+0000 7f0ac3fff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f0acc07e720 0x7f0acc09f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:60814/0 (socket says 172.21.15.42:60814) 2019-12-05T06:40:19.914 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:19.884+0000 7f0ad08bd700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0acc09fae0 0x7f0acc07eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:60778/0 (socket says 172.21.15.42:60778) 2019-12-05T06:40:19.914 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:19.884+0000 7f0ac3fff700 1 -- 172.21.15.42:0/4027488509 learned_addr learned my addr 172.21.15.42:0/4027488509 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:40:19.914 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:19.884+0000 7f0acbfff700 1 --2- 172.21.15.42:0/4027488509 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0acc0a7740 0x7f0acc08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:40:19.915 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:19.884+0000 7f0ad08bd700 1 -- 172.21.15.42:0/4027488509 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0acc0a7740 msgr2=0x7f0acc08df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:40:19.915 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:19.884+0000 7f0ad08bd700 1 --2- 172.21.15.42:0/4027488509 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0acc0a7740 0x7f0acc08df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:40:19.915 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:19.884+0000 7f0ad08bd700 1 -- 172.21.15.42:0/4027488509 >> v1:172.21.15.42:6789/0 conn(0x7f0acc07e720 legacy=0x7f0acc09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:19.915 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:19.884+0000 7f0ad08bd700 1 -- 172.21.15.42:0/4027488509 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0acc07f0a0 con 0x7f0acc09fae0 2019-12-05T06:40:19.916 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:19.884+0000 7f0acb7fe700 1 -- 172.21.15.42:0/4027488509 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1018903591 0 0) 0x7f0acc0a0710 con 0x7f0acc07e720 2019-12-05T06:40:19.916 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:19.884+0000 7f0ad08bd700 1 --2- 172.21.15.42:0/4027488509 >> v2:172.21.15.42:3300/0 conn(0x7f0acc09fae0 0x7f0acc07eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f0ac4008b10 tx=0x7f0ac400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:19.916 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:19.884+0000 7f0ad08bd700 1 --2- 172.21.15.42:0/4027488509 >> v2:172.21.15.42:3300/0 conn(0x7f0acc09fae0 0x7f0acc07eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f0ac4008b10 tx=0x7f0ac400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:19.916 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:19.884+0000 7f0ad08bd700 1 --2- 172.21.15.42:0/4027488509 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0acc09fae0 0x7f0acc07eb50 secure :-1 s=READY pgs=2983 cs=0 l=1 rx=0x7f0ac4008b10 tx=0x7f0ac400fff0).ready entity=mon.1 client_cookie=c61a2bcaa7fca3ba server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:40:19.916 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:19.884+0000 7f0ad08bd700 1 --2- 172.21.15.42:0/4027488509 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0acc09fae0 0x7f0acc07eb50 secure :-1 s=THROTTLE_DONE pgs=2983 cs=0 l=1 rx=0x7f0ac4008b10 tx=0x7f0ac400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:19.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:19.884+0000 7f0acb7fe700 1 -- 172.21.15.42:0/4027488509 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0ac4010900 con 0x7f0acc09fae0 2019-12-05T06:40:19.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:19.884+0000 7f0ad08bd700 1 --2- 172.21.15.42:0/4027488509 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0acc09fae0 0x7f0acc07eb50 secure :-1 s=THROTTLE_DONE pgs=2983 cs=0 l=1 rx=0x7f0ac4008b10 tx=0x7f0ac400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:19.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:19.884+0000 7f0ad08bd700 1 --2- 172.21.15.42:0/4027488509 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0acc09fae0 0x7f0acc07eb50 secure :-1 s=THROTTLE_DONE pgs=2983 cs=0 l=1 rx=0x7f0ac4008b10 tx=0x7f0ac400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:19.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:19.884+0000 7f0acb7fe700 1 -- 172.21.15.42:0/4027488509 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ac4008b50 con 0x7f0acc09fae0 2019-12-05T06:40:19.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:19.884+0000 7f0acb7fe700 1 -- 172.21.15.42:0/4027488509 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0ac40110c0 con 0x7f0acc09fae0 2019-12-05T06:40:19.918 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:19.884+0000 7f0ad1b1f700 1 -- 172.21.15.42:0/4027488509 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0acc09fae0 msgr2=0x7f0acc07eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:19.918 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:19.884+0000 7f0ad1b1f700 1 --2- 172.21.15.42:0/4027488509 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0acc09fae0 0x7f0acc07eb50 secure :-1 s=READY pgs=2983 cs=0 l=1 rx=0x7f0ac4008b10 tx=0x7f0ac400fff0).stop 2019-12-05T06:40:19.918 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:19.884+0000 7f0ad1b1f700 1 -- 172.21.15.42:0/4027488509 shutdown_connections 2019-12-05T06:40:19.918 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:19.884+0000 7f0ad1b1f700 1 --2- 172.21.15.42:0/4027488509 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0acc09fae0 0x7f0acc07eb50 unknown :-1 s=CLOSED pgs=2983 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:40:19.918 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:19.884+0000 7f0ad1b1f700 1 --2- 172.21.15.42:0/4027488509 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0acc0a7740 0x7f0acc08df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:40:19.918 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:19.884+0000 7f0ad1b1f700 1 -- 172.21.15.42:0/4027488509 shutdown_connections 2019-12-05T06:40:19.919 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:19.884+0000 7f0ad1b1f700 1 -- 172.21.15.42:0/4027488509 wait complete. 2019-12-05T06:40:19.919 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:19.884+0000 7f0ad1b1f700 1 -- 172.21.15.42:0/4027488509 >> 172.21.15.42:0/4027488509 conn(0x7f0acc0aabe0 msgr2=0x7f0acc0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:40:19.919 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:19.888+0000 7f0ad1b1f700 1 Processor -- start 2019-12-05T06:40:19.919 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:19.888+0000 7f0ad1b1f700 1 -- start start 2019-12-05T06:40:19.919 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:19.888+0000 7f0ad1b1f700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0acc0a4a50 0x7f0acc11ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:40:19.920 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:19.888+0000 7f0ad1b1f700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0acc11f020 con 0x7f0acc12ae50 2019-12-05T06:40:19.920 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:19.888+0000 7f0ad1b1f700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0acc11b880 con 0x7f0acc0722c0 2019-12-05T06:40:19.920 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:19.888+0000 7f0ad1b1f700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0acc11bc70 con 0x7f0acc0a4a50 2019-12-05T06:40:19.921 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:19.888+0000 7f0ad08bd700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f0acc0722c0 0x7f0acc074700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:60822/0 (socket says 172.21.15.42:60822) 2019-12-05T06:40:19.922 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:19.888+0000 7f0ad08bd700 1 -- 172.21.15.42:0/409099344 learned_addr learned my addr 172.21.15.42:0/409099344 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:40:19.922 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:19.888+0000 7f0acbfff700 1 --2- 172.21.15.42:0/409099344 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0acc0a4a50 0x7f0acc11ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:40:19.922 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:19.888+0000 7f0ac97fa700 1 -- 172.21.15.42:0/409099344 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2371070797 0 0) 0x7f0acc11b880 con 0x7f0acc0722c0 2019-12-05T06:40:19.922 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:19.888+0000 7f0ac97fa700 1 -- 172.21.15.42:0/409099344 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ab00030b0 con 0x7f0acc0722c0 2019-12-05T06:40:19.922 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:19.888+0000 7f0ac97fa700 1 -- 172.21.15.42:0/409099344 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2900466251 0 0) 0x7f0acc11f020 con 0x7f0acc12ae50 2019-12-05T06:40:19.922 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:19.888+0000 7f0ac97fa700 1 -- 172.21.15.42:0/409099344 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0acc11b880 con 0x7f0acc12ae50 2019-12-05T06:40:19.923 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:19.888+0000 7f0ac97fa700 1 -- 172.21.15.42:0/409099344 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 416207108 0 0) 0x7f0ab00030b0 con 0x7f0acc0722c0 2019-12-05T06:40:19.923 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:19.888+0000 7f0ac97fa700 1 -- 172.21.15.42:0/409099344 --> v1:172.21.15.42:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0acc11f020 con 0x7f0acc0722c0 2019-12-05T06:40:19.923 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:19.888+0000 7f0ac97fa700 1 -- 172.21.15.42:0/409099344 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7f0ac4010900 con 0x7f0acc0722c0 2019-12-05T06:40:19.923 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:19.888+0000 7f0acbfff700 1 -- 172.21.15.42:0/409099344 >> v1:172.21.15.42:6789/0 conn(0x7f0acc0722c0 legacy=0x7f0acc074700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:19.923 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:19.888+0000 7f0acbfff700 1 -- 172.21.15.42:0/409099344 >> v1:172.21.15.40:6789/0 conn(0x7f0acc12ae50 legacy=0x7f0acc12b280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:19.923 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:19.888+0000 7f0acbfff700 1 -- 172.21.15.42:0/409099344 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0acc11f960 con 0x7f0acc0a4a50 2019-12-05T06:40:19.923 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:19.888+0000 7f0acbfff700 1 --2- 172.21.15.42:0/409099344 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0acc0a4a50 0x7f0acc11ead0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f0abc002f30 tx=0x7f0ab40010d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:19.924 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:19.888+0000 7f0acbfff700 1 -- 172.21.15.42:0/409099344 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0acc0a4a50 msgr2=0x7f0acc11ead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2019-12-05T06:40:19.924 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:19.888+0000 7f0acbfff700 1 -- 172.21.15.42:0/409099344 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0acc0a4a50 msgr2=0x7f0acc11ead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2019-12-05T06:40:19.924 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:19.888+0000 7f0acbfff700 1 --2- 172.21.15.42:0/409099344 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0acc0a4a50 0x7f0acc11ead0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f0abc002f30 tx=0x7f0ab40010d0).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-12-05T06:40:19.924 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:19.888+0000 7f0acbfff700 1 --2- 172.21.15.42:0/409099344 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0acc0a4a50 0x7f0acc11ead0 secure :-1 s=START_CONNECT pgs=0 cs=0 l=1 rx=0x7f0abc002f30 tx=0x7f0ab40010d0)._fault waiting 0.200000 2019-12-05T06:40:20.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:20.092+0000 7f0acbfff700 1 --2- 172.21.15.42:0/409099344 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0acc0a4a50 0x7f0acc11ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:40:20.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:20.092+0000 7f0acbfff700 0 --2- 172.21.15.42:0/409099344 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0acc0a4a50 0x7f0acc11ead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2019-12-05T06:40:20.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:20.092+0000 7f0acbfff700 1 --2- 172.21.15.42:0/409099344 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0acc0a4a50 0x7f0acc11ead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:40:20.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:20.092+0000 7f0ac97fa700 1 -- 172.21.15.42:0/409099344 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0acc0a4a50 msgr2=0x7f0acc11ead0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2019-12-05T06:40:20.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:20.092+0000 7f0ac97fa700 1 --2- 172.21.15.42:0/409099344 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0acc0a4a50 0x7f0acc11ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:40:20.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:20.092+0000 7f0ac97fa700 1 --2- 172.21.15.42:0/409099344 >> v2:172.21.15.40:3300/0 conn(0x7f0ab0003ae0 0x7f0ab0004420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:40:20.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:20.092+0000 7f0ac97fa700 1 --2- 172.21.15.42:0/409099344 >> v2:172.21.15.42:3300/0 conn(0x7f0ab0004970 0x7f0ab0004da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:40:20.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:20.092+0000 7f0ac97fa700 1 -- 172.21.15.42:0/409099344 --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ab00040e0 con 0x7f0ab0003ae0 2019-12-05T06:40:20.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:20.092+0000 7f0ac97fa700 1 -- 172.21.15.42:0/409099344 --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ab000da70 con 0x7f0ab0004970 2019-12-05T06:40:20.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:20.092+0000 7f0ac3fff700 1 --2- 172.21.15.42:0/409099344 >> v2:172.21.15.42:3300/0 conn(0x7f0ab0004970 0x7f0ab0004da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:40:20.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:20.092+0000 7f0acbfff700 1 --2- 172.21.15.42:0/409099344 >> v2:172.21.15.40:3300/0 conn(0x7f0ab0003ae0 0x7f0ab0004420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:40:20.103 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:20.092+0000 7f0ac3fff700 1 -- 172.21.15.42:0/409099344 >> v2:172.21.15.40:3300/0 conn(0x7f0ab0003ae0 msgr2=0x7f0ab0004420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:20.103 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:20.092+0000 7f0ac3fff700 1 --2- 172.21.15.42:0/409099344 >> v2:172.21.15.40:3300/0 conn(0x7f0ab0003ae0 0x7f0ab0004420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:40:20.103 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:20.092+0000 7f0ac3fff700 1 -- 172.21.15.42:0/409099344 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ab000dc30 con 0x7f0ab0004970 2019-12-05T06:40:20.103 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:20.092+0000 7f0ac3fff700 1 --2- 172.21.15.42:0/409099344 >> v2:172.21.15.42:3300/0 conn(0x7f0ab0004970 0x7f0ab0004da0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f0ab4001230 tx=0x7f0ab4010b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:20.103 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:20.092+0000 7f0ac3fff700 1 --2- 172.21.15.42:0/409099344 >> v2:172.21.15.42:3300/0 conn(0x7f0ab0004970 0x7f0ab0004da0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f0ab4001230 tx=0x7f0ab4010b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:20.104 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:20.092+0000 7f0ac3fff700 1 --2- 172.21.15.42:0/409099344 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0ab0004970 0x7f0ab0004da0 secure :-1 s=READY pgs=2985 cs=0 l=1 rx=0x7f0ab4001230 tx=0x7f0ab4010b30).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:40:20.104 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:20.092+0000 7f0ac3fff700 1 --2- 172.21.15.42:0/409099344 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0ab0004970 0x7f0ab0004da0 secure :-1 s=THROTTLE_DONE pgs=2985 cs=0 l=1 rx=0x7f0ab4001230 tx=0x7f0ab4010b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:20.104 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:20.092+0000 7f0ac97fa700 1 -- 172.21.15.42:0/409099344 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0ab400c9c0 con 0x7f0ab0004970 2019-12-05T06:40:20.104 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:20.092+0000 7f0ac3fff700 1 --2- 172.21.15.42:0/409099344 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0ab0004970 0x7f0ab0004da0 secure :-1 s=THROTTLE_DONE pgs=2985 cs=0 l=1 rx=0x7f0ab4001230 tx=0x7f0ab4010b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:20.104 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:20.092+0000 7f0ac3fff700 1 --2- 172.21.15.42:0/409099344 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0ab0004970 0x7f0ab0004da0 secure :-1 s=THROTTLE_DONE pgs=2985 cs=0 l=1 rx=0x7f0ab4001230 tx=0x7f0ab4010b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:20.105 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:20.092+0000 7f0ac97fa700 1 -- 172.21.15.42:0/409099344 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ab4010250 con 0x7f0ab0004970 2019-12-05T06:40:20.105 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:20.092+0000 7f0ac97fa700 1 -- 172.21.15.42:0/409099344 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0ab40121b0 con 0x7f0ab0004970 2019-12-05T06:40:20.105 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:20.092+0000 7f0ad1b1f700 1 -- 172.21.15.42:0/409099344 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0acc13f150 con 0x7f0ab0004970 2019-12-05T06:40:20.105 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:20.092+0000 7f0ad1b1f700 1 -- 172.21.15.42:0/409099344 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0acc13f650 con 0x7f0ab0004970 2019-12-05T06:40:20.105 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:20.092+0000 7f0ac3fff700 1 --2- 172.21.15.42:0/409099344 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0ab0004970 0x7f0ab0004da0 secure :-1 s=THROTTLE_DONE pgs=2985 cs=0 l=1 rx=0x7f0ab4001230 tx=0x7f0ab4010b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:20.105 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:20.092+0000 7f0ac97fa700 1 -- 172.21.15.42:0/409099344 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f0ab4012480 con 0x7f0ab0004970 2019-12-05T06:40:20.106 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:20.092+0000 7f0ac3fff700 1 --2- 172.21.15.42:0/409099344 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0ab0004970 0x7f0ab0004da0 secure :-1 s=THROTTLE_DONE pgs=2985 cs=0 l=1 rx=0x7f0ab4001230 tx=0x7f0ab4010b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:20.106 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:20.092+0000 7f0ac97fa700 1 --2- 172.21.15.42:0/409099344 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0ab0045ad0 0x7f0ab0047fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:40:20.106 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:20.092+0000 7f0ac97fa700 1 -- 172.21.15.42:0/409099344 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 11621+0+0 (secure 0 0 0) 0x7f0ab404adc0 con 0x7f0ab0004970 2019-12-05T06:40:20.106 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:20.092+0000 7f0acbfff700 1 --2- 172.21.15.42:0/409099344 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0ab0045ad0 0x7f0ab0047fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:40:20.106 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:20.092+0000 7f0acbfff700 1 --2- 172.21.15.42:0/409099344 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0ab0045ad0 0x7f0ab0047fe0 crc :-1 s=READY pgs=1248 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:40:20.107 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:20.096+0000 7f0ad1b1f700 1 -- 172.21.15.42:0/409099344 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0acc05d110 con 0x7f0ab0004970 2019-12-05T06:40:20.107 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:20.100+0000 7f0ac3fff700 1 --2- 172.21.15.42:0/409099344 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0ab0004970 0x7f0ab0004da0 secure :-1 s=THROTTLE_DONE pgs=2985 cs=0 l=1 rx=0x7f0ab4001230 tx=0x7f0ab4010b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:20.107 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:20.100+0000 7f0ac97fa700 1 -- 172.21.15.42:0/409099344 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f0ab4031e40 con 0x7f0ab0004970 2019-12-05T06:40:20.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:20.284+0000 7f0ad1b1f700 1 -- 172.21.15.42:0/409099344 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0acc05b8c0 con 0x7f0ab0004970 2019-12-05T06:40:20.292 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:20.284+0000 7f0ac3fff700 1 --2- 172.21.15.42:0/409099344 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0ab0004970 0x7f0ab0004da0 secure :-1 s=THROTTLE_DONE pgs=2985 cs=0 l=1 rx=0x7f0ab4001230 tx=0x7f0ab4010b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:20.292 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:20.284+0000 7f0ac97fa700 1 -- 172.21.15.42:0/409099344 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 359 v359) v1 ==== 95+0+1776 (secure 0 0 0) 0x7f0ab40320a0 con 0x7f0ab0004970 2019-12-05T06:40:20.292 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 359 2019-12-05T06:40:20.292 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:40:20.293 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":359,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":359,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:40:13.247006+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":114,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11868},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11868":{"gid":11868,"name":"a","rank":0,"incarnation":355,"state":"up:active","state_seq":12,"addr":"172.21.15.42:6835/406829134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":406829134},{"type":"v1","addr":"172.21.15.42:6835","nonce":406829134}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:40:20.294 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:20.288+0000 7f0ad1b1f700 1 -- 172.21.15.42:0/409099344 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0ab0045ad0 msgr2=0x7f0ab0047fe0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:20.294 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:20.288+0000 7f0ad1b1f700 1 --2- 172.21.15.42:0/409099344 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0ab0045ad0 0x7f0ab0047fe0 crc :-1 s=READY pgs=1248 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:40:20.294 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:20.288+0000 7f0ad1b1f700 1 -- 172.21.15.42:0/409099344 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0ab0004970 msgr2=0x7f0ab0004da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:20.294 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:20.288+0000 7f0ad08bd700 1 -- 172.21.15.42:0/409099344 reap_dead start 2019-12-05T06:40:20.294 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:20.288+0000 7f0ad1b1f700 1 --2- 172.21.15.42:0/409099344 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0ab0004970 0x7f0ab0004da0 secure :-1 s=READY pgs=2985 cs=0 l=1 rx=0x7f0ab4001230 tx=0x7f0ab4010b30).stop 2019-12-05T06:40:20.295 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:20.288+0000 7f0ad1b1f700 1 -- 172.21.15.42:0/409099344 shutdown_connections 2019-12-05T06:40:20.295 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:20.288+0000 7f0ad1b1f700 1 --2- 172.21.15.42:0/409099344 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0ab0004970 0x7f0ab0004da0 unknown :-1 s=CLOSED pgs=2985 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:40:20.295 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:20.288+0000 7f0ad1b1f700 1 -- 172.21.15.42:0/409099344 shutdown_connections 2019-12-05T06:40:20.295 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:20.288+0000 7f0ad1b1f700 1 -- 172.21.15.42:0/409099344 wait complete. 2019-12-05T06:40:20.295 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:20.288+0000 7f0ad1b1f700 1 -- 172.21.15.42:0/409099344 >> 172.21.15.42:0/409099344 conn(0x7f0acc0a8410 msgr2=0x7f0acc07e5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:40:20.309 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2019-12-05T06:40:22.310 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:40:22.411 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:22.400+0000 7f390a363700 1 Processor -- start 2019-12-05T06:40:22.411 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:22.400+0000 7f390a363700 1 -- start start 2019-12-05T06:40:22.411 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:22.400+0000 7f390a363700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f390408df80 0x7f390409f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:40:22.411 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:22.400+0000 7f390a363700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f390409fc60 con 0x7f390409f830 2019-12-05T06:40:22.411 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:22.400+0000 7f390a363700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39040a0710 con 0x7f39040a7740 2019-12-05T06:40:22.411 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:22.400+0000 7f390a363700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39040a0b00 con 0x7f390408df80 2019-12-05T06:40:22.412 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:22.400+0000 7f3903fff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f390408df80 0x7f390409f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:40:22.412 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:22.400+0000 7f3903fff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f390408df80 0x7f390409f2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:60822/0 (socket says 172.21.15.42:60822) 2019-12-05T06:40:22.412 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:22.400+0000 7f3903fff700 1 -- 172.21.15.42:0/3404871313 learned_addr learned my addr 172.21.15.42:0/3404871313 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:40:22.412 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:22.400+0000 7f3903fff700 1 -- 172.21.15.42:0/3404871313 >> v1:172.21.15.42:6789/0 conn(0x7f39040a7740 legacy=0x7f390407e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:22.412 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:22.400+0000 7f3903fff700 1 -- 172.21.15.42:0/3404871313 >> v1:172.21.15.40:6789/0 conn(0x7f390409f830 legacy=0x7f390414efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:22.412 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:22.400+0000 7f3903fff700 1 -- 172.21.15.42:0/3404871313 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f390407ef20 con 0x7f390408df80 2019-12-05T06:40:22.413 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:22.400+0000 7f3903fff700 1 --2- 172.21.15.42:0/3404871313 >> v2:172.21.15.42:3300/0 conn(0x7f390408df80 0x7f390409f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f38f4008790 tx=0x7f38f400fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:22.413 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:22.400+0000 7f3903fff700 1 --2- 172.21.15.42:0/3404871313 >> v2:172.21.15.42:3300/0 conn(0x7f390408df80 0x7f390409f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f38f4008790 tx=0x7f38f400fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:22.413 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:22.400+0000 7f3903fff700 1 --2- 172.21.15.42:0/3404871313 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f390408df80 0x7f390409f2e0 secure :-1 s=READY pgs=2987 cs=0 l=1 rx=0x7f38f4008790 tx=0x7f38f400fcb0).ready entity=mon.1 client_cookie=7abe43f2d7cc27 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:40:22.413 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:22.400+0000 7f3903fff700 1 --2- 172.21.15.42:0/3404871313 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f390408df80 0x7f390409f2e0 secure :-1 s=THROTTLE_DONE pgs=2987 cs=0 l=1 rx=0x7f38f4008790 tx=0x7f38f400fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:22.413 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:22.400+0000 7f3903fff700 1 --2- 172.21.15.42:0/3404871313 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f390408df80 0x7f390409f2e0 secure :-1 s=THROTTLE_DONE pgs=2987 cs=0 l=1 rx=0x7f38f4008790 tx=0x7f38f400fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:22.413 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:22.400+0000 7f39037fe700 1 -- 172.21.15.42:0/3404871313 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f38f4011520 con 0x7f390408df80 2019-12-05T06:40:22.413 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:22.400+0000 7f3903fff700 1 --2- 172.21.15.42:0/3404871313 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f390408df80 0x7f390409f2e0 secure :-1 s=THROTTLE_DONE pgs=2987 cs=0 l=1 rx=0x7f38f4008790 tx=0x7f38f400fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:22.413 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:22.400+0000 7f39037fe700 1 -- 172.21.15.42:0/3404871313 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38f4008ca0 con 0x7f390408df80 2019-12-05T06:40:22.414 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:22.400+0000 7f39037fe700 1 -- 172.21.15.42:0/3404871313 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f38f4011b00 con 0x7f390408df80 2019-12-05T06:40:22.414 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:22.400+0000 7f390a363700 1 -- 172.21.15.42:0/3404871313 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f390408df80 msgr2=0x7f390409f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:22.414 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:22.400+0000 7f390a363700 1 --2- 172.21.15.42:0/3404871313 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f390408df80 0x7f390409f2e0 secure :-1 s=READY pgs=2987 cs=0 l=1 rx=0x7f38f4008790 tx=0x7f38f400fcb0).stop 2019-12-05T06:40:22.414 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:22.404+0000 7f390a363700 1 -- 172.21.15.42:0/3404871313 shutdown_connections 2019-12-05T06:40:22.414 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:22.404+0000 7f390a363700 1 --2- 172.21.15.42:0/3404871313 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f390408df80 0x7f390409f2e0 unknown :-1 s=CLOSED pgs=2987 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:40:22.414 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:22.404+0000 7f390a363700 1 -- 172.21.15.42:0/3404871313 shutdown_connections 2019-12-05T06:40:22.414 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:22.404+0000 7f390a363700 1 -- 172.21.15.42:0/3404871313 wait complete. 2019-12-05T06:40:22.415 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:22.404+0000 7f390a363700 1 -- 172.21.15.42:0/3404871313 >> 172.21.15.42:0/3404871313 conn(0x7f39040aabe0 msgr2=0x7f39040a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:40:22.415 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:22.404+0000 7f390a363700 1 Processor -- start 2019-12-05T06:40:22.415 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:22.404+0000 7f390a363700 1 -- start start 2019-12-05T06:40:22.424 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:22.404+0000 7f390a363700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f390411bd80 0x7f39040a5de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:40:22.424 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:22.404+0000 7f390a363700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f39040646c0 0x7f390407d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:40:22.424 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:22.404+0000 7f390a363700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39040a48d0 con 0x7f39040a1510 2019-12-05T06:40:22.424 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:22.404+0000 7f390a363700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3904064c40 con 0x7f390411bd80 2019-12-05T06:40:22.424 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:22.404+0000 7f390a363700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3904064ee0 con 0x7f39040646c0 2019-12-05T06:40:22.425 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:22.404+0000 7f3908900700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f39040a1510 0x7f39040a55e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:60866/0 (socket says 172.21.15.42:60866) 2019-12-05T06:40:22.425 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:22.404+0000 7f3908900700 1 -- 172.21.15.42:0/1008392916 learned_addr learned my addr 172.21.15.42:0/1008392916 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:40:22.425 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:22.408+0000 7f3903fff700 1 --2- 172.21.15.42:0/1008392916 >> v2:172.21.15.40:3300/0 conn(0x7f390411bd80 0x7f39040a5de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:40:22.425 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:22.408+0000 7f3909101700 1 --2- 172.21.15.42:0/1008392916 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f39040646c0 0x7f390407d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:40:22.425 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:22.408+0000 7f39017fa700 1 -- 172.21.15.42:0/1008392916 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1964032473 0 0) 0x7f39040a48d0 con 0x7f39040a1510 2019-12-05T06:40:22.425 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:22.408+0000 7f39017fa700 1 -- 172.21.15.42:0/1008392916 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38dc0030b0 con 0x7f39040a1510 2019-12-05T06:40:22.425 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:22.408+0000 7f3903fff700 1 -- 172.21.15.42:0/1008392916 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f39040646c0 msgr2=0x7f390407d390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:22.426 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:22.408+0000 7f3903fff700 1 --2- 172.21.15.42:0/1008392916 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f39040646c0 0x7f390407d390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:40:22.426 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:22.408+0000 7f3903fff700 1 -- 172.21.15.42:0/1008392916 >> v1:172.21.15.42:6789/0 conn(0x7f39040a1510 legacy=0x7f39040a55e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:22.426 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:22.408+0000 7f39017fa700 1 -- 172.21.15.42:0/1008392916 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 3318543674 0 0) 0x7f38dc0030b0 con 0x7f39040a1510 2019-12-05T06:40:22.426 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:22.408+0000 7f3903fff700 1 -- 172.21.15.42:0/1008392916 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39040a52a0 con 0x7f390411bd80 2019-12-05T06:40:22.426 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:22.408+0000 7f3909101700 1 --2- 172.21.15.42:0/1008392916 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f39040646c0 0x7f390407d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:40:22.426 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:22.408+0000 7f3903fff700 1 --2- 172.21.15.42:0/1008392916 >> v2:172.21.15.40:3300/0 conn(0x7f390411bd80 0x7f39040a5de0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f38ec001c50 tx=0x7f38ec001c90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:22.427 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:22.408+0000 7f3903fff700 1 --2- 172.21.15.42:0/1008392916 >> v2:172.21.15.40:3300/0 conn(0x7f390411bd80 0x7f39040a5de0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f38ec001c50 tx=0x7f38ec001c90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:22.427 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:22.408+0000 7f3903fff700 1 --2- 172.21.15.42:0/1008392916 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f390411bd80 0x7f39040a5de0 secure :-1 s=READY pgs=2265 cs=0 l=1 rx=0x7f38ec001c50 tx=0x7f38ec001c90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:40:22.427 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:22.408+0000 7f3903fff700 1 --2- 172.21.15.42:0/1008392916 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f390411bd80 0x7f39040a5de0 secure :-1 s=THROTTLE_DONE pgs=2265 cs=0 l=1 rx=0x7f38ec001c50 tx=0x7f38ec001c90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:22.427 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:22.408+0000 7f39017fa700 1 -- 172.21.15.42:0/1008392916 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f38ec001a10 con 0x7f390411bd80 2019-12-05T06:40:22.428 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:22.408+0000 7f3903fff700 1 --2- 172.21.15.42:0/1008392916 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f390411bd80 0x7f39040a5de0 secure :-1 s=THROTTLE_DONE pgs=2265 cs=0 l=1 rx=0x7f38ec001c50 tx=0x7f38ec001c90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:22.428 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:22.408+0000 7f390a363700 1 -- 172.21.15.42:0/1008392916 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f390407d8e0 con 0x7f390411bd80 2019-12-05T06:40:22.428 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:22.408+0000 7f39017fa700 1 -- 172.21.15.42:0/1008392916 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38ec001820 con 0x7f390411bd80 2019-12-05T06:40:22.429 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:22.408+0000 7f3903fff700 1 --2- 172.21.15.42:0/1008392916 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f390411bd80 0x7f39040a5de0 secure :-1 s=THROTTLE_DONE pgs=2265 cs=0 l=1 rx=0x7f38ec001c50 tx=0x7f38ec001c90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:22.429 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:22.408+0000 7f39017fa700 1 -- 172.21.15.42:0/1008392916 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f38f4007340 con 0x7f390411bd80 2019-12-05T06:40:22.429 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:22.408+0000 7f390a363700 1 -- 172.21.15.42:0/1008392916 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f390407de60 con 0x7f390411bd80 2019-12-05T06:40:22.430 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:22.408+0000 7f3903fff700 1 --2- 172.21.15.42:0/1008392916 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f390411bd80 0x7f39040a5de0 secure :-1 s=THROTTLE_DONE pgs=2265 cs=0 l=1 rx=0x7f38ec001c50 tx=0x7f38ec001c90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:22.430 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:22.412+0000 7f39017fa700 1 -- 172.21.15.42:0/1008392916 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f38f4005540 con 0x7f390411bd80 2019-12-05T06:40:22.430 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:22.412+0000 7f3903fff700 1 --2- 172.21.15.42:0/1008392916 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f390411bd80 0x7f39040a5de0 secure :-1 s=THROTTLE_DONE pgs=2265 cs=0 l=1 rx=0x7f38ec001c50 tx=0x7f38ec001c90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:22.430 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:22.412+0000 7f39017fa700 1 --2- 172.21.15.42:0/1008392916 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f38dc03b210 0x7f38dc03d720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:40:22.431 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:22.412+0000 7f39017fa700 1 -- 172.21.15.42:0/1008392916 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 11621+0+0 (secure 0 0 0) 0x7f38f404b250 con 0x7f390411bd80 2019-12-05T06:40:22.431 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:22.412+0000 7f3908900700 1 --2- 172.21.15.42:0/1008392916 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f38dc03b210 0x7f38dc03d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:40:22.431 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:22.412+0000 7f3908900700 1 --2- 172.21.15.42:0/1008392916 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f38dc03b210 0x7f38dc03d720 crc :-1 s=READY pgs=1250 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:40:22.431 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:22.416+0000 7f390a363700 1 -- 172.21.15.42:0/1008392916 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f390405d110 con 0x7f390411bd80 2019-12-05T06:40:22.431 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:22.424+0000 7f3903fff700 1 --2- 172.21.15.42:0/1008392916 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f390411bd80 0x7f39040a5de0 secure :-1 s=THROTTLE_DONE pgs=2265 cs=0 l=1 rx=0x7f38ec001c50 tx=0x7f38ec001c90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:22.432 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:22.424+0000 7f39017fa700 1 -- 172.21.15.42:0/1008392916 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f38f40330c0 con 0x7f390411bd80 2019-12-05T06:40:22.604 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:22.596+0000 7f390a363700 1 -- 172.21.15.42:0/1008392916 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f390405b8c0 con 0x7f390411bd80 2019-12-05T06:40:22.605 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:22.600+0000 7f3903fff700 1 --2- 172.21.15.42:0/1008392916 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f390411bd80 0x7f39040a5de0 secure :-1 s=THROTTLE_DONE pgs=2265 cs=0 l=1 rx=0x7f38ec001c50 tx=0x7f38ec001c90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:22.605 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:22.600+0000 7f39017fa700 1 -- 172.21.15.42:0/1008392916 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 360 v360) v1 ==== 95+0+2111 (secure 0 0 0) 0x7f38f400dbb0 con 0x7f390411bd80 2019-12-05T06:40:22.607 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 360 2019-12-05T06:40:22.607 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:40:22.607 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":360,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":11976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/474728597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":474728597},{"type":"v1","addr":"172.21.15.40:6833","nonce":474728597}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":360}],"filesystems":[{"mdsmap":{"epoch":359,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:40:13.247006+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":114,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11868},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11868":{"gid":11868,"name":"a","rank":0,"incarnation":355,"state":"up:active","state_seq":12,"addr":"172.21.15.42:6835/406829134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":406829134},{"type":"v1","addr":"172.21.15.42:6835","nonce":406829134}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:40:22.610 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:22.604+0000 7f390a363700 1 -- 172.21.15.42:0/1008392916 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f38dc03b210 msgr2=0x7f38dc03d720 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:22.610 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:22.604+0000 7f390a363700 1 --2- 172.21.15.42:0/1008392916 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f38dc03b210 0x7f38dc03d720 crc :-1 s=READY pgs=1250 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:40:22.610 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:22.604+0000 7f390a363700 1 -- 172.21.15.42:0/1008392916 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f390411bd80 msgr2=0x7f39040a5de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:22.611 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:22.604+0000 7f390a363700 1 --2- 172.21.15.42:0/1008392916 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f390411bd80 0x7f39040a5de0 secure :-1 s=READY pgs=2265 cs=0 l=1 rx=0x7f38ec001c50 tx=0x7f38ec001c90).stop 2019-12-05T06:40:22.611 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:22.604+0000 7f390a363700 1 -- 172.21.15.42:0/1008392916 shutdown_connections 2019-12-05T06:40:22.611 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:22.604+0000 7f390a363700 1 --2- 172.21.15.42:0/1008392916 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f38dc03b210 0x7f38dc03d720 unknown :-1 s=CLOSED pgs=1250 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:40:22.611 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:22.604+0000 7f390a363700 1 --2- 172.21.15.42:0/1008392916 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f39040646c0 0x7f390407d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:40:22.611 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:22.604+0000 7f390a363700 1 --2- 172.21.15.42:0/1008392916 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f390411bd80 0x7f39040a5de0 unknown :-1 s=CLOSED pgs=2265 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:40:22.611 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:22.604+0000 7f390a363700 1 -- 172.21.15.42:0/1008392916 shutdown_connections 2019-12-05T06:40:22.612 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:22.604+0000 7f390a363700 1 -- 172.21.15.42:0/1008392916 wait complete. 2019-12-05T06:40:22.612 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:22.604+0000 7f390a363700 1 -- 172.21.15.42:0/1008392916 >> 172.21.15.42:0/1008392916 conn(0x7f39040a7740 msgr2=0x7f390408eb70 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:40:22.620 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b reported in up:standby state 2019-12-05T06:40:22.620 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 2 secs before thrashing 2019-12-05T06:40:24.621 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:40:24.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:24.711+0000 7f57f0c1f700 1 Processor -- start 2019-12-05T06:40:24.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:24.711+0000 7f57f0c1f700 1 -- start start 2019-12-05T06:40:24.724 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:24.711+0000 7f57f0c1f700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f57ec08df80 0x7f57ec09f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:40:24.724 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:24.711+0000 7f57f0c1f700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f57ec09f830 0x7f57ec14efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:40:24.724 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:24.711+0000 7f57f0c1f700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57ec09fc60 con 0x7f57ec0a7740 2019-12-05T06:40:24.724 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:24.711+0000 7f57f0c1f700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57ec0a0b00 con 0x7f57ec09f830 2019-12-05T06:40:24.724 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:24.711+0000 7f57f0c1f700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57ec08e5b0 con 0x7f57ec08df80 2019-12-05T06:40:24.725 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:24.711+0000 7f57ead9d700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f57ec0a7740 0x7f57ec07e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:60916/0 (socket says 172.21.15.42:60916) 2019-12-05T06:40:24.725 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:24.711+0000 7f57ead9d700 1 -- 172.21.15.42:0/3718015313 learned_addr learned my addr 172.21.15.42:0/3718015313 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:40:24.725 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:24.711+0000 7f57e3fff700 1 --2- 172.21.15.42:0/3718015313 >> v2:172.21.15.42:3300/0 conn(0x7f57ec08df80 0x7f57ec09f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:40:24.725 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:24.715+0000 7f57eb59e700 1 --2- 172.21.15.42:0/3718015313 >> v2:172.21.15.40:3300/0 conn(0x7f57ec09f830 0x7f57ec14efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:40:24.725 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:24.715+0000 7f57ea59c700 1 -- 172.21.15.42:0/3718015313 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 29549778 0 0) 0x7f57ec09fc60 con 0x7f57ec0a7740 2019-12-05T06:40:24.725 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:24.715+0000 7f57ea59c700 1 -- 172.21.15.42:0/3718015313 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57d80030b0 con 0x7f57ec0a7740 2019-12-05T06:40:24.725 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:24.715+0000 7f57e3fff700 1 -- 172.21.15.42:0/3718015313 >> v2:172.21.15.40:3300/0 conn(0x7f57ec09f830 msgr2=0x7f57ec14efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:40:24.726 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:24.715+0000 7f57e3fff700 1 --2- 172.21.15.42:0/3718015313 >> v2:172.21.15.40:3300/0 conn(0x7f57ec09f830 0x7f57ec14efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:40:24.726 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:24.715+0000 7f57e3fff700 1 -- 172.21.15.42:0/3718015313 >> v1:172.21.15.42:6789/0 conn(0x7f57ec0a7740 legacy=0x7f57ec07e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:24.726 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:24.715+0000 7f57e3fff700 1 -- 172.21.15.42:0/3718015313 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57ec0a0710 con 0x7f57ec08df80 2019-12-05T06:40:24.726 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:24.715+0000 7f57e3fff700 1 --2- 172.21.15.42:0/3718015313 >> v2:172.21.15.42:3300/0 conn(0x7f57ec08df80 0x7f57ec09f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f57d4002f30 tx=0x7f57d400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:24.726 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:24.715+0000 7f57e3fff700 1 --2- 172.21.15.42:0/3718015313 >> v2:172.21.15.42:3300/0 conn(0x7f57ec08df80 0x7f57ec09f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f57d4002f30 tx=0x7f57d400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:24.726 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:24.715+0000 7f57e3fff700 1 --2- 172.21.15.42:0/3718015313 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f57ec08df80 0x7f57ec09f2e0 secure :-1 s=READY pgs=2990 cs=0 l=1 rx=0x7f57d4002f30 tx=0x7f57d400fc70).ready entity=mon.1 client_cookie=397e7ebd596c598d server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:40:24.726 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:24.715+0000 7f57e3fff700 1 --2- 172.21.15.42:0/3718015313 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f57ec08df80 0x7f57ec09f2e0 secure :-1 s=THROTTLE_DONE pgs=2990 cs=0 l=1 rx=0x7f57d4002f30 tx=0x7f57d400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:24.727 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:24.715+0000 7f57ea59c700 1 -- 172.21.15.42:0/3718015313 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f57d4008bb0 con 0x7f57ec08df80 2019-12-05T06:40:24.727 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:24.715+0000 7f57e3fff700 1 --2- 172.21.15.42:0/3718015313 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f57ec08df80 0x7f57ec09f2e0 secure :-1 s=THROTTLE_DONE pgs=2990 cs=0 l=1 rx=0x7f57d4002f30 tx=0x7f57d400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:24.727 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:24.715+0000 7f57e3fff700 1 --2- 172.21.15.42:0/3718015313 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f57ec08df80 0x7f57ec09f2e0 secure :-1 s=THROTTLE_DONE pgs=2990 cs=0 l=1 rx=0x7f57d4002f30 tx=0x7f57d400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:24.727 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:24.715+0000 7f57ea59c700 1 -- 172.21.15.42:0/3718015313 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57d40087d0 con 0x7f57ec08df80 2019-12-05T06:40:24.727 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:24.715+0000 7f57ea59c700 1 -- 172.21.15.42:0/3718015313 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f57d4011820 con 0x7f57ec08df80 2019-12-05T06:40:24.727 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:24.715+0000 7f57f0c1f700 1 -- 172.21.15.42:0/3718015313 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f57ec08df80 msgr2=0x7f57ec09f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:24.728 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:24.715+0000 7f57f0c1f700 1 --2- 172.21.15.42:0/3718015313 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f57ec08df80 0x7f57ec09f2e0 secure :-1 s=READY pgs=2990 cs=0 l=1 rx=0x7f57d4002f30 tx=0x7f57d400fc70).stop 2019-12-05T06:40:24.728 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:24.715+0000 7f57f0c1f700 1 -- 172.21.15.42:0/3718015313 shutdown_connections 2019-12-05T06:40:24.728 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:24.715+0000 7f57f0c1f700 1 --2- 172.21.15.42:0/3718015313 >> v2:172.21.15.40:3300/0 conn(0x7f57ec09f830 0x7f57ec14efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:40:24.728 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:24.715+0000 7f57f0c1f700 1 --2- 172.21.15.42:0/3718015313 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f57ec08df80 0x7f57ec09f2e0 unknown :-1 s=CLOSED pgs=2990 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:40:24.728 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:24.715+0000 7f57f0c1f700 1 -- 172.21.15.42:0/3718015313 shutdown_connections 2019-12-05T06:40:24.728 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:24.715+0000 7f57f0c1f700 1 -- 172.21.15.42:0/3718015313 wait complete. 2019-12-05T06:40:24.729 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:24.715+0000 7f57f0c1f700 1 -- 172.21.15.42:0/3718015313 >> 172.21.15.42:0/3718015313 conn(0x7f57ec0aabe0 msgr2=0x7f57ec0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:40:24.734 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:24.719+0000 7f57f0c1f700 1 Processor -- start 2019-12-05T06:40:24.735 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:24.719+0000 7f57f0c1f700 1 -- start start 2019-12-05T06:40:24.735 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:24.719+0000 7f57f0c1f700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f57ec123c30 0x7f57ec1197a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:40:24.735 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:24.719+0000 7f57f0c1f700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f57ec1171a0 0x7f57ec118510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:40:24.736 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:24.719+0000 7f57f0c1f700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57ec1219b0 con 0x7f57ec118a60 2019-12-05T06:40:24.736 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:24.719+0000 7f57f0c1f700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57ec121d30 con 0x7f57ec123c30 2019-12-05T06:40:24.736 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:24.719+0000 7f57f0c1f700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57ec121fd0 con 0x7f57ec1171a0 2019-12-05T06:40:24.736 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:24.719+0000 7f57ead9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f57ec123c30 0x7f57ec1197a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:40:24.736 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:24.719+0000 7f57ead9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f57ec123c30 0x7f57ec1197a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:60884/0 (socket says 172.21.15.42:60884) 2019-12-05T06:40:24.736 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:24.719+0000 7f57ead9d700 1 -- 172.21.15.42:0/940959579 learned_addr learned my addr 172.21.15.42:0/940959579 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:40:24.737 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:24.719+0000 7f57e3fff700 1 --2- 172.21.15.42:0/940959579 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f57ec1171a0 0x7f57ec118510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:40:24.737 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:24.719+0000 7f57ead9d700 1 -- 172.21.15.42:0/940959579 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f57ec1171a0 msgr2=0x7f57ec118510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:24.737 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:24.719+0000 7f57ead9d700 1 --2- 172.21.15.42:0/940959579 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f57ec1171a0 0x7f57ec118510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:40:24.737 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:24.719+0000 7f57ead9d700 1 -- 172.21.15.42:0/940959579 >> v1:172.21.15.40:6789/0 conn(0x7f57ec118a60 legacy=0x7f57ec08eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:24.737 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:24.719+0000 7f57ead9d700 1 -- 172.21.15.42:0/940959579 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57ec1175d0 con 0x7f57ec123c30 2019-12-05T06:40:24.737 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:24.719+0000 7f57ead9d700 1 --2- 172.21.15.42:0/940959579 >> v2:172.21.15.42:3300/0 conn(0x7f57ec123c30 0x7f57ec1197a0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f57ec121ef0 tx=0x7f57dc00c430).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:24.737 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:24.719+0000 7f57ead9d700 1 --2- 172.21.15.42:0/940959579 >> v2:172.21.15.42:3300/0 conn(0x7f57ec123c30 0x7f57ec1197a0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f57ec121ef0 tx=0x7f57dc00c430).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:24.738 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:24.719+0000 7f57ead9d700 1 --2- 172.21.15.42:0/940959579 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f57ec123c30 0x7f57ec1197a0 secure :-1 s=READY pgs=2991 cs=0 l=1 rx=0x7f57ec121ef0 tx=0x7f57dc00c430).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:40:24.738 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:24.719+0000 7f57ead9d700 1 --2- 172.21.15.42:0/940959579 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f57ec123c30 0x7f57ec1197a0 secure :-1 s=THROTTLE_DONE pgs=2991 cs=0 l=1 rx=0x7f57ec121ef0 tx=0x7f57dc00c430).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:24.738 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:24.719+0000 7f57e37fe700 1 -- 172.21.15.42:0/940959579 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f57dc00c9a0 con 0x7f57ec123c30 2019-12-05T06:40:24.738 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:24.719+0000 7f57ead9d700 1 --2- 172.21.15.42:0/940959579 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f57ec123c30 0x7f57ec1197a0 secure :-1 s=THROTTLE_DONE pgs=2991 cs=0 l=1 rx=0x7f57ec121ef0 tx=0x7f57dc00c430).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:24.738 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:24.719+0000 7f57e37fe700 1 -- 172.21.15.42:0/940959579 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57dc010250 con 0x7f57ec123c30 2019-12-05T06:40:24.738 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:24.719+0000 7f57ead9d700 1 --2- 172.21.15.42:0/940959579 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f57ec123c30 0x7f57ec1197a0 secure :-1 s=THROTTLE_DONE pgs=2991 cs=0 l=1 rx=0x7f57ec121ef0 tx=0x7f57dc00c430).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:24.738 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:24.719+0000 7f57e37fe700 1 -- 172.21.15.42:0/940959579 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f57dc00d0c0 con 0x7f57ec123c30 2019-12-05T06:40:24.739 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:24.719+0000 7f57f0c1f700 1 -- 172.21.15.42:0/940959579 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57ec08f370 con 0x7f57ec123c30 2019-12-05T06:40:24.739 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:24.719+0000 7f57f0c1f700 1 -- 172.21.15.42:0/940959579 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57ec08f8a0 con 0x7f57ec123c30 2019-12-05T06:40:24.739 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:24.719+0000 7f57ead9d700 1 --2- 172.21.15.42:0/940959579 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f57ec123c30 0x7f57ec1197a0 secure :-1 s=THROTTLE_DONE pgs=2991 cs=0 l=1 rx=0x7f57ec121ef0 tx=0x7f57dc00c430).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:24.739 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:24.723+0000 7f57e37fe700 1 -- 172.21.15.42:0/940959579 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f57dc00d3e0 con 0x7f57ec123c30 2019-12-05T06:40:24.739 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:24.723+0000 7f57ead9d700 1 --2- 172.21.15.42:0/940959579 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f57ec123c30 0x7f57ec1197a0 secure :-1 s=THROTTLE_DONE pgs=2991 cs=0 l=1 rx=0x7f57ec121ef0 tx=0x7f57dc00c430).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:24.739 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:24.723+0000 7f57e37fe700 1 --2- 172.21.15.42:0/940959579 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f57d003ae70 0x7f57d003d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:40:24.739 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:24.723+0000 7f57e37fe700 1 -- 172.21.15.42:0/940959579 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 11621+0+0 (secure 0 0 0) 0x7f57dc04ac10 con 0x7f57ec123c30 2019-12-05T06:40:24.740 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:24.723+0000 7f57e3fff700 1 --2- 172.21.15.42:0/940959579 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f57d003ae70 0x7f57d003d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:40:24.740 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:24.723+0000 7f57e3fff700 1 --2- 172.21.15.42:0/940959579 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f57d003ae70 0x7f57d003d380 crc :-1 s=READY pgs=1251 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:40:24.740 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:24.727+0000 7f57f0c1f700 1 -- 172.21.15.42:0/940959579 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57ec05d110 con 0x7f57ec123c30 2019-12-05T06:40:24.740 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:24.731+0000 7f57ead9d700 1 --2- 172.21.15.42:0/940959579 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f57ec123c30 0x7f57ec1197a0 secure :-1 s=THROTTLE_DONE pgs=2991 cs=0 l=1 rx=0x7f57ec121ef0 tx=0x7f57dc00c430).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:24.741 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:24.731+0000 7f57e37fe700 1 -- 172.21.15.42:0/940959579 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f57dc031c10 con 0x7f57ec123c30 2019-12-05T06:40:24.902 INFO:tasks.workunit.client.0.smithi042.stderr:Checking out files: 1% (101/9108) Checking out files: 1% (128/9108) Checking out files: 2% (183/9108) Checking out files: 3% (274/9108) Checking out files: 3% (289/9108) Checking out files: 4% (365/9108) Checking out files: 4% (419/9108) Checking out files: 5% (456/9108) Checking out files: 6% (547/9108) Checking out files: 6% (618/9108) Checking out files: 7% (638/9108) Checking out files: 8% (729/9108) Checking out files: 9% (820/9108) Checking out files: 9% (864/9108) Checking out files: 10% (911/9108) Checking out files: 11% (1002/9108) Checking out files: 12% (1093/9108) Checking out files: 12% (1145/9108) Checking out files: 13% (1185/9108) Checking out files: 13% (1201/9108) Checking out files: 13% (1234/9108) Checking out files: 13% (1237/9108) Checking out files: 14% (1276/9108) Checking out files: 14% (1331/9108) Checking out files: 15% (1367/9108) Checking out files: 15% (1434/9108) Checking out files: 16% (1458/9108) Checking out files: 17% (1549/9108) Checking out files: 17% (1563/9108) Checking out files: 18% (1640/9108) Checking out files: 18% (1698/9108) Checking out files: 18% (1710/9108) Checking out files: 19% (1731/9108) Checking out files: 19% (1812/9108) Checking out files: 20% (1822/9108) Checking out files: 21% (1913/9108) Checking out files: 21% (1954/9108) Checking out files: 22% (2004/9108) Checking out files: 23% (2095/9108) Checking out files: 23% (2110/9108) Checking out files: 24% (2186/9108) Checking out files: 24% (2188/9108) Checking out files: 24% (2252/9108) Checking out files: 25% (2277/9108) Checking out files: 26% (2369/9108) Checking out files: 26% (2397/9108) Checking out files: 26% (2422/9108) Checking out files: 27% (2460/9108) Checking out files: 28% (2551/9108) Checking out files: 28% (2611/9108) Checking out files: 29% (2642/9108) Checking out files: 30% (2733/9108) Checking out files: 30% (2763/9108) Checking out files: 31% (2824/9108) Checking out files: 31% (2831/9108) Checking out files: 32% (2915/9108) Checking out files: 32% (2973/9108) Checking out files: 33% (3006/9108) Checking out files: 34% (3097/9108) Checking out files: 34% (3159/9108) Checking out files: 35% (3188/9108) Checking out files: 35% (3252/9108) Checking out files: 36% (3279/9108) Checking out files: 37% (3370/9108) Checking out files: 38% (3462/9108) Checking out files: 38% (3528/9108) Checking out files: 39% (3553/9108) Checking out files: 40% (3644/9108) Checking out files: 40% (3671/9108) Checking out files: 41% (3735/9108) Checking out files: 42% (3826/9108) Checking out files: 42% (3858/9108) Checking out files: 43% (3917/9108) Checking out files: 43% (3981/9108) Checking out files: 44% (4008/9108) Checking out files: 44% (4079/9108) Checking out files: 45% (4099/9108) Checking out files: 46% (4190/9108) Checking out files: 47% (4281/9108) Checking out files: 47% (4340/9108) Checking out files: 48% (4372/9108) Checking out files: 48% (4396/9108) Checking out files: 49% (4463/9108) Checking out files: 49% (4533/9108) Checking out files: 50% (4554/9108) Checking out files: 51% (4646/9108) Checking out files: 51% (4663/9108) Checking out files: 51% (4711/9108) Checking out files: 52% (4737/9108) Checking out files: 52% (4791/9108) Checking out files: 53% (4828/9108) Checking out files: 54% (4919/9108) Checking out files: 54% (4983/9108) Checking out files: 55% (5010/9108) Checking out files: 56% (5101/9108) Checking out files: 56% (5121/9108) Checking out files: 57% (5192/9108) Checking out files: 58% (5283/9108) Checking out files: 58% (5291/9108) Checking out files: 59% (5374/9108) Checking out files: 60% (5465/9108) Checking out files: 60% (5546/9108) Checking out files: 61% (5556/9108) Checking out files: 61% (5584/9108) Checking out files: 61% (5588/9108) Checking out files: 62% (5647/9108) Checking out files: 63% (5739/9108) Checking out files: 63% (5764/9108) Checking out files: 64% (5830/9108) Checking out files: 65% (5921/9108) Checking out files: 65% (5959/9108) Checking out files: 66% (6012/9108) Checking out files: 66% (6061/9108) Checking out files: 67% (6103/9108) Checking out files: 68% (6194/9108) Checking out files: 68% (6210/9108) Checking out files: 69% (6285/9108) Checking out files: 69% (6335/9108) Checking out files: 70% (6376/9108) Checking out files: 70% (6437/9108) Checking out files: 70% (6464/9108) Checking out files: 71% (6467/9108) Checking out files: 71% (6522/9108) Checking out files: 71% (6555/9108) Checking out files: 72% (6558/9108) Checking out files: 72% (6572/9108) Checking out files: 72% (6598/9108) Checking out files: 73% (6649/9108) Checking out files: 74% (6740/9108) Checking out files: 74% (6769/9108) Checking out files: 75% (6831/9108) Checking out files: 75% (6868/9108) Checking out files: 76% (6923/9108) Checking out files: 76% (7003/9108) Checking out files: 77% (7014/9108) Checking out files: 78% (7105/9108) Checking out files: 78% (7145/9108) Checking out files: 79% (7196/9108) Checking out files: 79% (7207/9108) Checking out files: 79% (7210/9108) Checking out files: 79% (7258/9108) Checking out files: 80% (7287/9108) Checking out files: 80% (7316/9108) Checking out files: 81% (7378/9108) Checking out files: 82% (7469/9108) Checking out files: 82% (7524/9108) Checking out files: 82% (7545/9108) Checking out files: 83% (7560/9108) Checking out files: 83% (7635/9108) Checking out files: 84% (7651/9108) Checking out files: 84% (7656/9108) Checking out files: 85% (7742/9108) Checking out files: 85% (7793/9108) Checking out files: 85% (7830/9108) Checking out files: 86% (7833/9108) Checking out files: 86% (7886/9108) Checking out files: 87% (7924/9108) Checking out files: 87% (7967/9108) Checking out files: 88% (8016/9108) Checking out files: 89% (8107/9108) Checking out files: 89% (8174/9108) Checking out files: 90% (8198/9108) Checking out files: 90% (8212/9108) Checking out files: 91% (8289/9108) Checking out files: 92% (8380/9108) Checking out files: 92% (8462/9108) Checking out files: 93% (8471/9108) Checking out files: 94% (8562/9108) Checking out files: 94% (8623/9108) Checking out files: 95% (8653/9108) Checking out files: 95% (8715/9108) Checking out files: 95% (8736/9108) Checking out files: 95% (8738/9108) Checking out files: 96% (8744/9108) Checking out files: 97% (8835/9108) Checking out files: 97% (8903/9108) Checking out files: 98% (8926/9108) Checking out files: 99% (9017/9108) Checking out files: 99% (9086/9108) Checking out files: 100% (9108/9108) Checking out files: 100% (9108/9108), done. 2019-12-05T06:40:25.108 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:24.927+0000 7f57f0c1f700 1 -- 172.21.15.42:0/940959579 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f57ec05b8c0 con 0x7f57ec123c30 2019-12-05T06:40:25.109 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:24.927+0000 7f57ead9d700 1 --2- 172.21.15.42:0/940959579 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f57ec123c30 0x7f57ec1197a0 secure :-1 s=THROTTLE_DONE pgs=2991 cs=0 l=1 rx=0x7f57ec121ef0 tx=0x7f57dc00c430).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:25.109 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:24.927+0000 7f57e37fe700 1 -- 172.21.15.42:0/940959579 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 360 v360) v1 ==== 95+0+2111 (secure 0 0 0) 0x7f57dc031e70 con 0x7f57ec123c30 2019-12-05T06:40:25.109 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 360 2019-12-05T06:40:25.109 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:24.931+0000 7f57f0c1f700 1 -- 172.21.15.42:0/940959579 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f57d003ae70 msgr2=0x7f57d003d380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:25.110 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:24.931+0000 7f57f0c1f700 1 --2- 172.21.15.42:0/940959579 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f57d003ae70 0x7f57d003d380 crc :-1 s=READY pgs=1251 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:40:25.110 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:24.931+0000 7f57f0c1f700 1 -- 172.21.15.42:0/940959579 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f57ec123c30 msgr2=0x7f57ec1197a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:25.110 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:24.931+0000 7f57f0c1f700 1 --2- 172.21.15.42:0/940959579 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f57ec123c30 0x7f57ec1197a0 secure :-1 s=READY pgs=2991 cs=0 l=1 rx=0x7f57ec121ef0 tx=0x7f57dc00c430).stop 2019-12-05T06:40:25.110 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:24.943+0000 7f57f0c1f700 1 -- 172.21.15.42:0/940959579 shutdown_connections 2019-12-05T06:40:25.110 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:24.943+0000 7f57f0c1f700 1 --2- 172.21.15.42:0/940959579 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f57d003ae70 0x7f57d003d380 unknown :-1 s=CLOSED pgs=1251 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:40:25.110 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:24.943+0000 7f57f0c1f700 1 --2- 172.21.15.42:0/940959579 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f57ec123c30 0x7f57ec1197a0 unknown :-1 s=CLOSED pgs=2991 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:40:25.111 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:24.943+0000 7f57f0c1f700 1 --2- 172.21.15.42:0/940959579 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f57ec1171a0 0x7f57ec118510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:40:25.111 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:24.943+0000 7f57f0c1f700 1 -- 172.21.15.42:0/940959579 shutdown_connections 2019-12-05T06:40:25.111 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:24.943+0000 7f57f0c1f700 1 -- 172.21.15.42:0/940959579 wait complete. 2019-12-05T06:40:25.111 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:24.943+0000 7f57f0c1f700 1 -- 172.21.15.42:0/940959579 >> 172.21.15.42:0/940959579 conn(0x7f57ec0a9820 msgr2=0x7f57ec059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:40:25.111 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:40:25.112 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":360,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":11976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/474728597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":474728597},{"type":"v1","addr":"172.21.15.40:6833","nonce":474728597}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":360}],"filesystems":[{"mdsmap":{"epoch":359,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:40:13.247006+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":114,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11868},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11868":{"gid":11868,"name":"a","rank":0,"incarnation":355,"state":"up:active","state_seq":12,"addr":"172.21.15.42:6835/406829134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":406829134},{"type":"v1","addr":"172.21.15.42:6835","nonce":406829134}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:40:25.112 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.a (rank=0) 2019-12-05T06:40:25.113 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2019-12-05T06:40:25.113 INFO:teuthology.orchestra.run:waiting for 300 2019-12-05T06:40:25.122 INFO:tasks.ceph.mds.a:Stopped 2019-12-05T06:40:25.122 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:40:25.292 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:25.279+0000 7fb973a6e700 1 Processor -- start 2019-12-05T06:40:25.293 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:25.279+0000 7fb973a6e700 1 -- start start 2019-12-05T06:40:25.293 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:25.283+0000 7fb973a6e700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb96c0a7740 0x7fb96c08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:40:25.293 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:25.283+0000 7fb973a6e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb96c09fae0 0x7fb96c07eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:40:25.293 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:25.283+0000 7fb973a6e700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb96c0a0710 con 0x7fb96c07e720 2019-12-05T06:40:25.293 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:25.283+0000 7fb973a6e700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb96c0a0b00 con 0x7fb96c09fae0 2019-12-05T06:40:25.293 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:25.283+0000 7fb973a6e700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb96c08e5b0 con 0x7fb96c0a7740 2019-12-05T06:40:25.294 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:25.283+0000 7fb97280c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb96c09fae0 0x7fb96c07eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:40:25.294 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:25.283+0000 7fb97280c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb96c09fae0 0x7fb96c07eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:60918/0 (socket says 172.21.15.42:60918) 2019-12-05T06:40:25.294 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:25.283+0000 7fb97280c700 1 -- 172.21.15.42:0/2378724247 learned_addr learned my addr 172.21.15.42:0/2378724247 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:40:25.294 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:25.283+0000 7fb97200b700 1 --2- 172.21.15.42:0/2378724247 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb96c0a7740 0x7fb96c08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:40:25.294 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:25.283+0000 7fb97280c700 1 -- 172.21.15.42:0/2378724247 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb96c0a7740 msgr2=0x7fb96c08df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:40:25.294 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:25.283+0000 7fb97280c700 1 --2- 172.21.15.42:0/2378724247 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb96c0a7740 0x7fb96c08df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:40:25.294 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:25.283+0000 7fb97280c700 1 -- 172.21.15.42:0/2378724247 >> v1:172.21.15.40:6789/0 conn(0x7fb96c07e720 legacy=0x7fb96c09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:25.295 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:25.283+0000 7fb97280c700 1 -- 172.21.15.42:0/2378724247 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb96c07f0a0 con 0x7fb96c09fae0 2019-12-05T06:40:25.295 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:25.283+0000 7fb97280c700 1 --2- 172.21.15.42:0/2378724247 >> v2:172.21.15.42:3300/0 conn(0x7fb96c09fae0 0x7fb96c07eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fb964008b10 tx=0x7fb9640100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:25.295 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:25.283+0000 7fb97280c700 1 --2- 172.21.15.42:0/2378724247 >> v2:172.21.15.42:3300/0 conn(0x7fb96c09fae0 0x7fb96c07eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fb964008b10 tx=0x7fb9640100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:25.295 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:25.283+0000 7fb97280c700 1 --2- 172.21.15.42:0/2378724247 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb96c09fae0 0x7fb96c07eb50 secure :-1 s=READY pgs=2992 cs=0 l=1 rx=0x7fb964008b10 tx=0x7fb9640100b0).ready entity=mon.1 client_cookie=be3b418c4faea86f server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:40:25.295 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:25.283+0000 7fb97280c700 1 --2- 172.21.15.42:0/2378724247 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb96c09fae0 0x7fb96c07eb50 secure :-1 s=THROTTLE_DONE pgs=2992 cs=0 l=1 rx=0x7fb964008b10 tx=0x7fb9640100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:25.295 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:25.283+0000 7fb971009700 1 -- 172.21.15.42:0/2378724247 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb964010da0 con 0x7fb96c09fae0 2019-12-05T06:40:25.295 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:25.283+0000 7fb97280c700 1 --2- 172.21.15.42:0/2378724247 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb96c09fae0 0x7fb96c07eb50 secure :-1 s=THROTTLE_DONE pgs=2992 cs=0 l=1 rx=0x7fb964008b10 tx=0x7fb9640100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:25.296 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:25.283+0000 7fb97280c700 1 --2- 172.21.15.42:0/2378724247 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb96c09fae0 0x7fb96c07eb50 secure :-1 s=THROTTLE_DONE pgs=2992 cs=0 l=1 rx=0x7fb964008b10 tx=0x7fb9640100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:25.296 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:25.283+0000 7fb971009700 1 -- 172.21.15.42:0/2378724247 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9640109c0 con 0x7fb96c09fae0 2019-12-05T06:40:25.296 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:25.283+0000 7fb971009700 1 -- 172.21.15.42:0/2378724247 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb9640114a0 con 0x7fb96c09fae0 2019-12-05T06:40:25.296 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:25.283+0000 7fb973a6e700 1 -- 172.21.15.42:0/2378724247 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb96c09fae0 msgr2=0x7fb96c07eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:25.296 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:25.283+0000 7fb973a6e700 1 --2- 172.21.15.42:0/2378724247 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb96c09fae0 0x7fb96c07eb50 secure :-1 s=READY pgs=2992 cs=0 l=1 rx=0x7fb964008b10 tx=0x7fb9640100b0).stop 2019-12-05T06:40:25.296 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:25.283+0000 7fb973a6e700 1 -- 172.21.15.42:0/2378724247 shutdown_connections 2019-12-05T06:40:25.297 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:25.283+0000 7fb973a6e700 1 --2- 172.21.15.42:0/2378724247 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb96c09fae0 0x7fb96c07eb50 unknown :-1 s=CLOSED pgs=2992 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:40:25.297 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:25.283+0000 7fb973a6e700 1 --2- 172.21.15.42:0/2378724247 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb96c0a7740 0x7fb96c08df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:40:25.297 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:25.283+0000 7fb973a6e700 1 -- 172.21.15.42:0/2378724247 shutdown_connections 2019-12-05T06:40:25.297 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:25.283+0000 7fb973a6e700 1 -- 172.21.15.42:0/2378724247 wait complete. 2019-12-05T06:40:25.297 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:25.283+0000 7fb973a6e700 1 -- 172.21.15.42:0/2378724247 >> 172.21.15.42:0/2378724247 conn(0x7fb96c0aabe0 msgr2=0x7fb96c0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:40:25.300 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:25.287+0000 7fb973a6e700 1 Processor -- start 2019-12-05T06:40:25.301 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:25.287+0000 7fb973a6e700 1 -- start start 2019-12-05T06:40:25.301 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:25.291+0000 7fb973a6e700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb96c116500 0x7fb96c1175a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:40:25.301 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:25.291+0000 7fb973a6e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb96c122a80 0x7fb96c122eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:40:25.301 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:25.291+0000 7fb973a6e700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb96c11e930 con 0x7fb96c08eb70 2019-12-05T06:40:25.301 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:25.291+0000 7fb973a6e700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb96c125450 con 0x7fb96c116500 2019-12-05T06:40:25.301 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:25.291+0000 7fb973a6e700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb96c117f70 con 0x7fb96c122a80 2019-12-05T06:40:25.301 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:25.291+0000 7fb97180a700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb96c122a80 0x7fb96c122eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:40:25.302 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:25.291+0000 7fb97200b700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb96c116500 0x7fb96c1175a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:40:25.302 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:25.291+0000 7fb97200b700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb96c116500 0x7fb96c1175a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.42:48736/0 (socket says 172.21.15.42:48736) 2019-12-05T06:40:25.302 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:25.291+0000 7fb97180a700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb96c122a80 0x7fb96c122eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:60924/0 (socket says 172.21.15.42:60924) 2019-12-05T06:40:25.302 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:25.291+0000 7fb97200b700 1 -- 172.21.15.42:0/1146629283 learned_addr learned my addr 172.21.15.42:0/1146629283 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:40:25.302 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:25.291+0000 7fb97180a700 1 -- 172.21.15.42:0/1146629283 >> v2:172.21.15.40:3300/0 conn(0x7fb96c116500 msgr2=0x7fb96c1175a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:25.302 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:25.291+0000 7fb97180a700 1 --2- 172.21.15.42:0/1146629283 >> v2:172.21.15.40:3300/0 conn(0x7fb96c116500 0x7fb96c1175a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:40:25.302 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:25.291+0000 7fb97180a700 1 -- 172.21.15.42:0/1146629283 >> v1:172.21.15.40:6789/0 conn(0x7fb96c08eb70 legacy=0x7fb96c08efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:25.303 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:25.291+0000 7fb97180a700 1 -- 172.21.15.42:0/1146629283 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb96c124190 con 0x7fb96c122a80 2019-12-05T06:40:25.303 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:25.291+0000 7fb97180a700 1 --2- 172.21.15.42:0/1146629283 >> v2:172.21.15.42:3300/0 conn(0x7fb96c122a80 0x7fb96c122eb0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fb95c000e90 tx=0x7fb95c0108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:25.303 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:25.291+0000 7fb97180a700 1 --2- 172.21.15.42:0/1146629283 >> v2:172.21.15.42:3300/0 conn(0x7fb96c122a80 0x7fb96c122eb0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fb95c000e90 tx=0x7fb95c0108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:25.303 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:25.291+0000 7fb97180a700 1 --2- 172.21.15.42:0/1146629283 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb96c122a80 0x7fb96c122eb0 secure :-1 s=READY pgs=2993 cs=0 l=1 rx=0x7fb95c000e90 tx=0x7fb95c0108f0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:40:25.303 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:25.291+0000 7fb97180a700 1 --2- 172.21.15.42:0/1146629283 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb96c122a80 0x7fb96c122eb0 secure :-1 s=THROTTLE_DONE pgs=2993 cs=0 l=1 rx=0x7fb95c000e90 tx=0x7fb95c0108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:25.303 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:25.291+0000 7fb962ffd700 1 -- 172.21.15.42:0/1146629283 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb95c00cb90 con 0x7fb96c122a80 2019-12-05T06:40:25.303 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:25.291+0000 7fb97180a700 1 --2- 172.21.15.42:0/1146629283 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb96c122a80 0x7fb96c122eb0 secure :-1 s=THROTTLE_DONE pgs=2993 cs=0 l=1 rx=0x7fb95c000e90 tx=0x7fb95c0108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:25.304 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:25.291+0000 7fb97180a700 1 --2- 172.21.15.42:0/1146629283 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb96c122a80 0x7fb96c122eb0 secure :-1 s=THROTTLE_DONE pgs=2993 cs=0 l=1 rx=0x7fb95c000e90 tx=0x7fb95c0108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:25.304 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:25.291+0000 7fb962ffd700 1 -- 172.21.15.42:0/1146629283 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb95c00c7b0 con 0x7fb96c122a80 2019-12-05T06:40:25.304 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:25.291+0000 7fb973a6e700 1 -- 172.21.15.42:0/1146629283 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb96c117af0 con 0x7fb96c122a80 2019-12-05T06:40:25.304 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:25.291+0000 7fb962ffd700 1 -- 172.21.15.42:0/1146629283 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb95c0110b0 con 0x7fb96c122a80 2019-12-05T06:40:25.304 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:25.291+0000 7fb973a6e700 1 -- 172.21.15.42:0/1146629283 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb96c1235a0 con 0x7fb96c122a80 2019-12-05T06:40:25.304 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:25.291+0000 7fb97180a700 1 --2- 172.21.15.42:0/1146629283 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb96c122a80 0x7fb96c122eb0 secure :-1 s=THROTTLE_DONE pgs=2993 cs=0 l=1 rx=0x7fb95c000e90 tx=0x7fb95c0108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:25.304 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:25.291+0000 7fb962ffd700 1 -- 172.21.15.42:0/1146629283 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fb95c0113c0 con 0x7fb96c122a80 2019-12-05T06:40:25.305 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:25.291+0000 7fb97180a700 1 --2- 172.21.15.42:0/1146629283 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb96c122a80 0x7fb96c122eb0 secure :-1 s=THROTTLE_DONE pgs=2993 cs=0 l=1 rx=0x7fb95c000e90 tx=0x7fb95c0108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:25.305 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:25.291+0000 7fb962ffd700 1 --2- 172.21.15.42:0/1146629283 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb95403ae40 0x7fb95403d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:40:25.305 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:25.291+0000 7fb962ffd700 1 -- 172.21.15.42:0/1146629283 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 11621+0+0 (secure 0 0 0) 0x7fb95c04b160 con 0x7fb96c122a80 2019-12-05T06:40:25.305 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:25.291+0000 7fb97200b700 1 --2- 172.21.15.42:0/1146629283 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb95403ae40 0x7fb95403d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:40:25.305 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:25.295+0000 7fb97200b700 1 --2- 172.21.15.42:0/1146629283 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb95403ae40 0x7fb95403d350 crc :-1 s=READY pgs=1252 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:40:25.311 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:25.295+0000 7fb973a6e700 1 -- 172.21.15.42:0/1146629283 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb96c05d110 con 0x7fb96c122a80 2019-12-05T06:40:25.311 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:25.303+0000 7fb97180a700 1 --2- 172.21.15.42:0/1146629283 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb96c122a80 0x7fb96c122eb0 secure :-1 s=THROTTLE_DONE pgs=2993 cs=0 l=1 rx=0x7fb95c000e90 tx=0x7fb95c0108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:25.312 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:25.303+0000 7fb962ffd700 1 -- 172.21.15.42:0/1146629283 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fb95c0321e0 con 0x7fb96c122a80 2019-12-05T06:40:25.484 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:25.475+0000 7fb973a6e700 1 -- 172.21.15.42:0/1146629283 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb96c05b8c0 con 0x7fb96c122a80 2019-12-05T06:40:25.485 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:25.479+0000 7fb97180a700 1 --2- 172.21.15.42:0/1146629283 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb96c122a80 0x7fb96c122eb0 secure :-1 s=THROTTLE_DONE pgs=2993 cs=0 l=1 rx=0x7fb95c000e90 tx=0x7fb95c0108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:25.485 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:25.479+0000 7fb962ffd700 1 -- 172.21.15.42:0/1146629283 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 360 v360) v1 ==== 95+0+2111 (secure 0 0 0) 0x7fb95c032440 con 0x7fb96c122a80 2019-12-05T06:40:25.485 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 360 2019-12-05T06:40:25.486 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:40:25.486 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":360,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":11976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/474728597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":474728597},{"type":"v1","addr":"172.21.15.40:6833","nonce":474728597}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":360}],"filesystems":[{"mdsmap":{"epoch":359,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:40:13.247006+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":114,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11868},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11868":{"gid":11868,"name":"a","rank":0,"incarnation":355,"state":"up:active","state_seq":12,"addr":"172.21.15.42:6835/406829134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":406829134},{"type":"v1","addr":"172.21.15.42:6835","nonce":406829134}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:40:25.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:25.479+0000 7fb973a6e700 1 -- 172.21.15.42:0/1146629283 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb95403ae40 msgr2=0x7fb95403d350 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:25.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:25.479+0000 7fb973a6e700 1 --2- 172.21.15.42:0/1146629283 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb95403ae40 0x7fb95403d350 crc :-1 s=READY pgs=1252 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:40:25.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:25.479+0000 7fb973a6e700 1 -- 172.21.15.42:0/1146629283 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb96c122a80 msgr2=0x7fb96c122eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:25.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:25.479+0000 7fb973a6e700 1 --2- 172.21.15.42:0/1146629283 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb96c122a80 0x7fb96c122eb0 secure :-1 s=READY pgs=2993 cs=0 l=1 rx=0x7fb95c000e90 tx=0x7fb95c0108f0).stop 2019-12-05T06:40:25.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:25.479+0000 7fb973a6e700 1 -- 172.21.15.42:0/1146629283 shutdown_connections 2019-12-05T06:40:25.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:25.479+0000 7fb973a6e700 1 --2- 172.21.15.42:0/1146629283 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb95403ae40 0x7fb95403d350 unknown :-1 s=CLOSED pgs=1252 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:40:25.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:25.479+0000 7fb973a6e700 1 --2- 172.21.15.42:0/1146629283 >> v2:172.21.15.40:3300/0 conn(0x7fb96c116500 0x7fb96c1175a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:40:25.501 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:25.479+0000 7fb973a6e700 1 --2- 172.21.15.42:0/1146629283 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb96c122a80 0x7fb96c122eb0 unknown :-1 s=CLOSED pgs=2993 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:40:25.501 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:25.479+0000 7fb973a6e700 1 -- 172.21.15.42:0/1146629283 shutdown_connections 2019-12-05T06:40:25.501 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:25.479+0000 7fb973a6e700 1 -- 172.21.15.42:0/1146629283 wait complete. 2019-12-05T06:40:25.502 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:25.479+0000 7fb973a6e700 1 -- 172.21.15.42:0/1146629283 >> 172.21.15.42:0/1146629283 conn(0x7fb96c0a98b0 msgr2=0x7fb96c11db90 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:40:25.503 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:40:27.506 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:40:27.618 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:27.603+0000 7f754fcfe700 1 Processor -- start 2019-12-05T06:40:27.619 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:27.603+0000 7f754fcfe700 1 -- start start 2019-12-05T06:40:27.619 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:27.607+0000 7f754fcfe700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f754809fae0 0x7f754814fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:40:27.619 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:27.607+0000 7f754fcfe700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75480a0710 con 0x7f754808df80 2019-12-05T06:40:27.620 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:27.607+0000 7f754fcfe700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f754807ef20 con 0x7f75480a7740 2019-12-05T06:40:27.620 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:27.607+0000 7f754fcfe700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75480a0b00 con 0x7f754809fae0 2019-12-05T06:40:27.620 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:27.607+0000 7f754e29b700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f75480a7740 0x7f754807e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:60976/0 (socket says 172.21.15.42:60976) 2019-12-05T06:40:27.621 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:27.607+0000 7f754e29b700 1 -- 172.21.15.42:0/3240524825 learned_addr learned my addr 172.21.15.42:0/3240524825 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:40:27.621 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:27.607+0000 7f754ea9c700 1 --2- 172.21.15.42:0/3240524825 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f754809fae0 0x7f754814fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:40:27.622 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:27.607+0000 7f754d299700 1 -- 172.21.15.42:0/3240524825 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3249863520 0 0) 0x7f754807ef20 con 0x7f75480a7740 2019-12-05T06:40:27.622 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:27.607+0000 7f754d299700 1 -- 172.21.15.42:0/3240524825 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75340030b0 con 0x7f75480a7740 2019-12-05T06:40:27.622 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:27.607+0000 7f754d299700 1 -- 172.21.15.42:0/3240524825 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4136609673 0 0) 0x7f75480a0710 con 0x7f754808df80 2019-12-05T06:40:27.623 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:27.607+0000 7f754d299700 1 -- 172.21.15.42:0/3240524825 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f754807ef20 con 0x7f754808df80 2019-12-05T06:40:27.623 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:27.607+0000 7f754ea9c700 1 -- 172.21.15.42:0/3240524825 >> v1:172.21.15.42:6789/0 conn(0x7f75480a7740 legacy=0x7f754807e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:27.624 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:27.607+0000 7f754ea9c700 1 -- 172.21.15.42:0/3240524825 >> v1:172.21.15.40:6789/0 conn(0x7f754808df80 legacy=0x7f754809f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:27.624 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:27.607+0000 7f754ea9c700 1 -- 172.21.15.42:0/3240524825 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75480a9ba0 con 0x7f754809fae0 2019-12-05T06:40:27.624 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:27.607+0000 7f754ea9c700 1 --2- 172.21.15.42:0/3240524825 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f754809fae0 0x7f754814fff0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f75400032b0 tx=0x7f754000ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:27.624 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:27.607+0000 7f754ea9c700 1 --2- 172.21.15.42:0/3240524825 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f754809fae0 0x7f754814fff0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f75400032b0 tx=0x7f754000ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:27.624 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:27.607+0000 7f754ea9c700 1 --2- 172.21.15.42:0/3240524825 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f754809fae0 0x7f754814fff0 secure :-1 s=READY pgs=657 cs=0 l=1 rx=0x7f75400032b0 tx=0x7f754000ff70).ready entity=mon.2 client_cookie=822060e12ef1750f server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:40:27.625 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:27.607+0000 7f754ea9c700 1 --2- 172.21.15.42:0/3240524825 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f754809fae0 0x7f754814fff0 secure :-1 s=THROTTLE_DONE pgs=657 cs=0 l=1 rx=0x7f75400032b0 tx=0x7f754000ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:27.625 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:27.607+0000 7f754d299700 1 -- 172.21.15.42:0/3240524825 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f75400109b0 con 0x7f754809fae0 2019-12-05T06:40:27.626 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:27.607+0000 7f754ea9c700 1 --2- 172.21.15.42:0/3240524825 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f754809fae0 0x7f754814fff0 secure :-1 s=THROTTLE_DONE pgs=657 cs=0 l=1 rx=0x7f75400032b0 tx=0x7f754000ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:27.626 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:27.607+0000 7f754ea9c700 1 --2- 172.21.15.42:0/3240524825 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f754809fae0 0x7f754814fff0 secure :-1 s=THROTTLE_DONE pgs=657 cs=0 l=1 rx=0x7f75400032b0 tx=0x7f754000ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:27.626 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:27.607+0000 7f754d299700 1 -- 172.21.15.42:0/3240524825 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7540008b50 con 0x7f754809fae0 2019-12-05T06:40:27.627 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:27.607+0000 7f754d299700 1 -- 172.21.15.42:0/3240524825 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f75400110c0 con 0x7f754809fae0 2019-12-05T06:40:27.627 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:27.607+0000 7f754fcfe700 1 -- 172.21.15.42:0/3240524825 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f754809fae0 msgr2=0x7f754814fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:27.627 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:27.607+0000 7f754fcfe700 1 --2- 172.21.15.42:0/3240524825 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f754809fae0 0x7f754814fff0 secure :-1 s=READY pgs=657 cs=0 l=1 rx=0x7f75400032b0 tx=0x7f754000ff70).stop 2019-12-05T06:40:27.628 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:27.611+0000 7f754fcfe700 1 -- 172.21.15.42:0/3240524825 shutdown_connections 2019-12-05T06:40:27.628 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:27.611+0000 7f754fcfe700 1 --2- 172.21.15.42:0/3240524825 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f754809fae0 0x7f754814fff0 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:40:27.628 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:27.611+0000 7f754fcfe700 1 -- 172.21.15.42:0/3240524825 shutdown_connections 2019-12-05T06:40:27.628 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:27.611+0000 7f754fcfe700 1 -- 172.21.15.42:0/3240524825 wait complete. 2019-12-05T06:40:27.630 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:27.611+0000 7f754fcfe700 1 -- 172.21.15.42:0/3240524825 >> 172.21.15.42:0/3240524825 conn(0x7f75480aabe0 msgr2=0x7f75480a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:40:27.630 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:27.611+0000 7f754fcfe700 1 Processor -- start 2019-12-05T06:40:27.630 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:27.611+0000 7f754fcfe700 1 -- start start 2019-12-05T06:40:27.637 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:27.615+0000 7f754fcfe700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f754814fd20 0x7f75481141a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:40:27.637 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:27.615+0000 7f754fcfe700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f75481133f0 0x7f754812dec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:40:27.637 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:27.615+0000 7f754fcfe700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7548142530 con 0x7f75480a3a60 2019-12-05T06:40:27.637 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:27.615+0000 7f754fcfe700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7548113f90 con 0x7f75481133f0 2019-12-05T06:40:27.639 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:27.615+0000 7f754fcfe700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7548142800 con 0x7f754814fd20 2019-12-05T06:40:27.639 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:27.615+0000 7f754da9a700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f75480a3a60 0x7f754807d970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:60984/0 (socket says 172.21.15.42:60984) 2019-12-05T06:40:27.640 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:27.615+0000 7f754da9a700 1 -- 172.21.15.42:0/2450383712 learned_addr learned my addr 172.21.15.42:0/2450383712 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:40:27.640 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:27.615+0000 7f754e29b700 1 --2- 172.21.15.42:0/2450383712 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f754814fd20 0x7f75481141a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:40:27.640 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:27.615+0000 7f754ea9c700 1 --2- 172.21.15.42:0/2450383712 >> v2:172.21.15.40:3300/0 conn(0x7f75481133f0 0x7f754812dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:40:27.640 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:27.615+0000 7f753effd700 1 -- 172.21.15.42:0/2450383712 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 165811041 0 0) 0x7f7548142530 con 0x7f75480a3a60 2019-12-05T06:40:27.640 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:27.615+0000 7f753effd700 1 -- 172.21.15.42:0/2450383712 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75300030b0 con 0x7f75480a3a60 2019-12-05T06:40:27.640 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:27.615+0000 7f754ea9c700 1 -- 172.21.15.42:0/2450383712 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f754814fd20 msgr2=0x7f75481141a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:27.641 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:27.615+0000 7f754ea9c700 1 --2- 172.21.15.42:0/2450383712 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f754814fd20 0x7f75481141a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:40:27.641 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:27.615+0000 7f754ea9c700 1 -- 172.21.15.42:0/2450383712 >> v1:172.21.15.42:6789/0 conn(0x7f75480a3a60 legacy=0x7f754807d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:27.641 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:27.615+0000 7f754ea9c700 1 -- 172.21.15.42:0/2450383712 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7548113820 con 0x7f75481133f0 2019-12-05T06:40:27.641 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:27.615+0000 7f754e29b700 1 --2- 172.21.15.42:0/2450383712 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f754814fd20 0x7f75481141a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:40:27.641 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:27.615+0000 7f754ea9c700 1 --2- 172.21.15.42:0/2450383712 >> v2:172.21.15.40:3300/0 conn(0x7f75481133f0 0x7f754812dec0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f75400032b0 tx=0x7f75400110c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:27.641 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:27.615+0000 7f754ea9c700 1 --2- 172.21.15.42:0/2450383712 >> v2:172.21.15.40:3300/0 conn(0x7f75481133f0 0x7f754812dec0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f75400032b0 tx=0x7f75400110c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:27.642 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:27.615+0000 7f754ea9c700 1 --2- 172.21.15.42:0/2450383712 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f75481133f0 0x7f754812dec0 secure :-1 s=READY pgs=2270 cs=0 l=1 rx=0x7f75400032b0 tx=0x7f75400110c0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:40:27.642 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:27.615+0000 7f754ea9c700 1 --2- 172.21.15.42:0/2450383712 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f75481133f0 0x7f754812dec0 secure :-1 s=THROTTLE_DONE pgs=2270 cs=0 l=1 rx=0x7f75400032b0 tx=0x7f75400110c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:27.642 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:27.615+0000 7f753effd700 1 -- 172.21.15.42:0/2450383712 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7540005ab0 con 0x7f75481133f0 2019-12-05T06:40:27.642 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:27.615+0000 7f754ea9c700 1 --2- 172.21.15.42:0/2450383712 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f75481133f0 0x7f754812dec0 secure :-1 s=THROTTLE_DONE pgs=2270 cs=0 l=1 rx=0x7f75400032b0 tx=0x7f75400110c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:27.642 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:27.615+0000 7f753effd700 1 -- 172.21.15.42:0/2450383712 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7540010bb0 con 0x7f75481133f0 2019-12-05T06:40:27.642 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:27.615+0000 7f754ea9c700 1 --2- 172.21.15.42:0/2450383712 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f75481133f0 0x7f754812dec0 secure :-1 s=THROTTLE_DONE pgs=2270 cs=0 l=1 rx=0x7f75400032b0 tx=0x7f75400110c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:27.643 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:27.615+0000 7f753effd700 1 -- 172.21.15.42:0/2450383712 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f754000f470 con 0x7f75481133f0 2019-12-05T06:40:27.643 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:27.615+0000 7f754fcfe700 1 -- 172.21.15.42:0/2450383712 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7548113b90 con 0x7f75481133f0 2019-12-05T06:40:27.643 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:27.615+0000 7f754fcfe700 1 -- 172.21.15.42:0/2450383712 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f754812e410 con 0x7f75481133f0 2019-12-05T06:40:27.643 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:27.615+0000 7f754ea9c700 1 --2- 172.21.15.42:0/2450383712 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f75481133f0 0x7f754812dec0 secure :-1 s=THROTTLE_DONE pgs=2270 cs=0 l=1 rx=0x7f75400032b0 tx=0x7f75400110c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:27.643 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:27.619+0000 7f753effd700 1 -- 172.21.15.42:0/2450383712 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f754000f710 con 0x7f75481133f0 2019-12-05T06:40:27.643 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:27.619+0000 7f754ea9c700 1 --2- 172.21.15.42:0/2450383712 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f75481133f0 0x7f754812dec0 secure :-1 s=THROTTLE_DONE pgs=2270 cs=0 l=1 rx=0x7f75400032b0 tx=0x7f75400110c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:27.644 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:27.619+0000 7f753effd700 1 --2- 172.21.15.42:0/2450383712 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f753003b280 0x7f753003d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:40:27.644 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:27.619+0000 7f753effd700 1 -- 172.21.15.42:0/2450383712 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 11621+0+0 (secure 0 0 0) 0x7f754004af30 con 0x7f75481133f0 2019-12-05T06:40:27.644 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:27.619+0000 7f754e29b700 1 --2- 172.21.15.42:0/2450383712 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f753003b280 0x7f753003d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:40:27.644 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:27.619+0000 7f754e29b700 1 --2- 172.21.15.42:0/2450383712 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f753003b280 0x7f753003d790 crc :-1 s=READY pgs=1253 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:40:27.645 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:27.619+0000 7f754fcfe700 1 -- 172.21.15.42:0/2450383712 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f754805d110 con 0x7f75481133f0 2019-12-05T06:40:27.645 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:27.631+0000 7f754ea9c700 1 --2- 172.21.15.42:0/2450383712 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f75481133f0 0x7f754812dec0 secure :-1 s=THROTTLE_DONE pgs=2270 cs=0 l=1 rx=0x7f75400032b0 tx=0x7f75400110c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:27.645 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:27.631+0000 7f753effd700 1 -- 172.21.15.42:0/2450383712 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f7540032170 con 0x7f75481133f0 2019-12-05T06:40:27.808 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:27.803+0000 7f754fcfe700 1 -- 172.21.15.42:0/2450383712 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f754805b8c0 con 0x7f75481133f0 2019-12-05T06:40:27.810 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:27.803+0000 7f754ea9c700 1 --2- 172.21.15.42:0/2450383712 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f75481133f0 0x7f754812dec0 secure :-1 s=THROTTLE_DONE pgs=2270 cs=0 l=1 rx=0x7f75400032b0 tx=0x7f75400110c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:27.810 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:27.803+0000 7f753effd700 1 -- 172.21.15.42:0/2450383712 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 360 v360) v1 ==== 95+0+2111 (secure 0 0 0) 0x7f7540032d20 con 0x7f75481133f0 2019-12-05T06:40:27.812 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:40:27.812 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":360,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":11976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/474728597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":474728597},{"type":"v1","addr":"172.21.15.40:6833","nonce":474728597}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":360}],"filesystems":[{"mdsmap":{"epoch":359,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:40:13.247006+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":114,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11868},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11868":{"gid":11868,"name":"a","rank":0,"incarnation":355,"state":"up:active","state_seq":12,"addr":"172.21.15.42:6835/406829134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":406829134},{"type":"v1","addr":"172.21.15.42:6835","nonce":406829134}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:40:27.812 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 360 2019-12-05T06:40:27.815 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:27.807+0000 7f754fcfe700 1 -- 172.21.15.42:0/2450383712 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f753003b280 msgr2=0x7f753003d790 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:27.815 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:27.807+0000 7f754fcfe700 1 --2- 172.21.15.42:0/2450383712 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f753003b280 0x7f753003d790 crc :-1 s=READY pgs=1253 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:40:27.815 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:27.807+0000 7f754fcfe700 1 -- 172.21.15.42:0/2450383712 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f75481133f0 msgr2=0x7f754812dec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:27.817 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:27.807+0000 7f754fcfe700 1 --2- 172.21.15.42:0/2450383712 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f75481133f0 0x7f754812dec0 secure :-1 s=READY pgs=2270 cs=0 l=1 rx=0x7f75400032b0 tx=0x7f75400110c0).stop 2019-12-05T06:40:27.817 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:27.807+0000 7f754fcfe700 1 -- 172.21.15.42:0/2450383712 shutdown_connections 2019-12-05T06:40:27.817 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:27.807+0000 7f754fcfe700 1 --2- 172.21.15.42:0/2450383712 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f753003b280 0x7f753003d790 unknown :-1 s=CLOSED pgs=1253 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:40:27.817 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:27.807+0000 7f754fcfe700 1 --2- 172.21.15.42:0/2450383712 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f75481133f0 0x7f754812dec0 unknown :-1 s=CLOSED pgs=2270 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:40:27.817 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:27.807+0000 7f754fcfe700 1 --2- 172.21.15.42:0/2450383712 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f754814fd20 0x7f75481141a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:40:27.818 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:27.807+0000 7f754fcfe700 1 -- 172.21.15.42:0/2450383712 shutdown_connections 2019-12-05T06:40:27.818 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:27.807+0000 7f754fcfe700 1 -- 172.21.15.42:0/2450383712 wait complete. 2019-12-05T06:40:27.818 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:27.807+0000 7f754fcfe700 1 -- 172.21.15.42:0/2450383712 >> 172.21.15.42:0/2450383712 conn(0x7f754809f040 msgr2=0x7f754809f580 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:40:27.824 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:40:29.825 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:40:29.935 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:29.919+0000 7f0601975700 1 Processor -- start 2019-12-05T06:40:29.935 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:29.919+0000 7f0601975700 1 -- start start 2019-12-05T06:40:29.935 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:29.919+0000 7f0601975700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f05fc0a7740 0x7f05fc08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:40:29.935 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:29.919+0000 7f0601975700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f05fc07e720 0x7f05fc07eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:40:29.935 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:29.919+0000 7f0601975700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05fc0a0710 con 0x7f05fc09f2e0 2019-12-05T06:40:29.936 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:29.919+0000 7f0601975700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05fc0a0b00 con 0x7f05fc0a7740 2019-12-05T06:40:29.936 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:29.919+0000 7f0601975700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05fc08e5b0 con 0x7f05fc07e720 2019-12-05T06:40:29.936 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:29.919+0000 7f05fb7fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f05fc0a7740 0x7f05fc08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:40:29.936 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:29.919+0000 7f05fbfff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f05fc09f2e0 0x7f05fc09f710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:60996/0 (socket says 172.21.15.42:60996) 2019-12-05T06:40:29.936 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:29.919+0000 7f05fbfff700 1 -- 172.21.15.42:0/3410748333 learned_addr learned my addr 172.21.15.42:0/3410748333 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:40:29.937 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:29.919+0000 7f05faffd700 1 --2- 172.21.15.42:0/3410748333 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f05fc07e720 0x7f05fc07eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:40:29.937 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:29.923+0000 7f05fb7fe700 1 -- 172.21.15.42:0/3410748333 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f05fc07e720 msgr2=0x7f05fc07eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:40:29.937 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:29.923+0000 7f05fb7fe700 1 --2- 172.21.15.42:0/3410748333 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f05fc07e720 0x7f05fc07eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:40:29.937 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:29.923+0000 7f05fb7fe700 1 -- 172.21.15.42:0/3410748333 >> v1:172.21.15.42:6789/0 conn(0x7f05fc09f2e0 legacy=0x7f05fc09f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:29.937 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:29.923+0000 7f05fb7fe700 1 -- 172.21.15.42:0/3410748333 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05fc07f0a0 con 0x7f05fc0a7740 2019-12-05T06:40:29.938 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:29.923+0000 7f05fa7fc700 1 -- 172.21.15.42:0/3410748333 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1044108574 0 0) 0x7f05fc0a0710 con 0x7f05fc09f2e0 2019-12-05T06:40:29.938 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:29.923+0000 7f05fb7fe700 1 --2- 172.21.15.42:0/3410748333 >> v2:172.21.15.42:3300/0 conn(0x7f05fc0a7740 0x7f05fc08df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f05ec008790 tx=0x7f05ec00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:29.938 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:29.923+0000 7f05fb7fe700 1 --2- 172.21.15.42:0/3410748333 >> v2:172.21.15.42:3300/0 conn(0x7f05fc0a7740 0x7f05fc08df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f05ec008790 tx=0x7f05ec00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:29.938 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:29.923+0000 7f05fb7fe700 1 --2- 172.21.15.42:0/3410748333 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f05fc0a7740 0x7f05fc08df80 secure :-1 s=READY pgs=2997 cs=0 l=1 rx=0x7f05ec008790 tx=0x7f05ec00fc70).ready entity=mon.1 client_cookie=6d2612fa88b9641d server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:40:29.938 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:29.923+0000 7f05fb7fe700 1 --2- 172.21.15.42:0/3410748333 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f05fc0a7740 0x7f05fc08df80 secure :-1 s=THROTTLE_DONE pgs=2997 cs=0 l=1 rx=0x7f05ec008790 tx=0x7f05ec00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:29.938 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:29.923+0000 7f05fa7fc700 1 -- 172.21.15.42:0/3410748333 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f05ec0112f0 con 0x7f05fc0a7740 2019-12-05T06:40:29.938 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:29.923+0000 7f05fb7fe700 1 --2- 172.21.15.42:0/3410748333 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f05fc0a7740 0x7f05fc08df80 secure :-1 s=THROTTLE_DONE pgs=2997 cs=0 l=1 rx=0x7f05ec008790 tx=0x7f05ec00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:29.939 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:29.923+0000 7f05fb7fe700 1 --2- 172.21.15.42:0/3410748333 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f05fc0a7740 0x7f05fc08df80 secure :-1 s=THROTTLE_DONE pgs=2997 cs=0 l=1 rx=0x7f05ec008790 tx=0x7f05ec00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:29.939 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:29.923+0000 7f05fa7fc700 1 -- 172.21.15.42:0/3410748333 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05ec008ad0 con 0x7f05fc0a7740 2019-12-05T06:40:29.939 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:29.923+0000 7f05fa7fc700 1 -- 172.21.15.42:0/3410748333 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f05ec0118d0 con 0x7f05fc0a7740 2019-12-05T06:40:29.939 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:29.923+0000 7f0601975700 1 -- 172.21.15.42:0/3410748333 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f05fc0a7740 msgr2=0x7f05fc08df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:29.939 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:29.923+0000 7f0601975700 1 --2- 172.21.15.42:0/3410748333 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f05fc0a7740 0x7f05fc08df80 secure :-1 s=READY pgs=2997 cs=0 l=1 rx=0x7f05ec008790 tx=0x7f05ec00fc70).stop 2019-12-05T06:40:29.939 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:29.923+0000 7f0601975700 1 -- 172.21.15.42:0/3410748333 shutdown_connections 2019-12-05T06:40:29.939 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:29.923+0000 7f0601975700 1 --2- 172.21.15.42:0/3410748333 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f05fc0a7740 0x7f05fc08df80 unknown :-1 s=CLOSED pgs=2997 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:40:29.940 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:29.923+0000 7f0601975700 1 --2- 172.21.15.42:0/3410748333 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f05fc07e720 0x7f05fc07eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:40:29.940 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:29.923+0000 7f0601975700 1 -- 172.21.15.42:0/3410748333 shutdown_connections 2019-12-05T06:40:29.940 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:29.923+0000 7f0601975700 1 -- 172.21.15.42:0/3410748333 wait complete. 2019-12-05T06:40:29.941 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:29.923+0000 7f0601975700 1 -- 172.21.15.42:0/3410748333 >> 172.21.15.42:0/3410748333 conn(0x7f05fc0aabe0 msgr2=0x7f05fc0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:40:29.941 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:29.923+0000 7f0601975700 1 Processor -- start 2019-12-05T06:40:29.941 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:29.923+0000 7f0601975700 1 -- start start 2019-12-05T06:40:29.942 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:29.927+0000 7f0601975700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f05fc12f710 0x7f05fc143900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:40:29.942 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:29.927+0000 7f0601975700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05fc07c1d0 con 0x7f05fc117960 2019-12-05T06:40:29.943 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:29.927+0000 7f0601975700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05fc143e50 con 0x7f05fc08f080 2019-12-05T06:40:29.943 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:29.927+0000 7f0601975700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05fc144210 con 0x7f05fc12f710 2019-12-05T06:40:29.943 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:29.927+0000 7f05fb7fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f05fc12f710 0x7f05fc143900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:40:29.944 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:29.927+0000 7f05fbfff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f05fc08f080 0x7f05fc07b9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:32768/0 (socket says 172.21.15.42:32768) 2019-12-05T06:40:29.944 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:29.927+0000 7f05fbfff700 1 -- 172.21.15.42:0/4279832535 learned_addr learned my addr 172.21.15.42:0/4279832535 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:40:29.944 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:29.927+0000 7f05fb7fe700 1 -- 172.21.15.42:0/4279832535 >> v1:172.21.15.42:6789/0 conn(0x7f05fc08f080 legacy=0x7f05fc07b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:29.944 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:29.927+0000 7f05e3fff700 1 -- 172.21.15.42:0/4279832535 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1704764275 0 0) 0x7f05fc07c1d0 con 0x7f05fc117960 2019-12-05T06:40:29.944 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:29.927+0000 7f05fb7fe700 1 -- 172.21.15.42:0/4279832535 >> v1:172.21.15.40:6789/0 conn(0x7f05fc117960 legacy=0x7f05fc1169f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:29.945 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:29.927+0000 7f05fb7fe700 1 -- 172.21.15.42:0/4279832535 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05fc139b90 con 0x7f05fc12f710 2019-12-05T06:40:29.945 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:29.927+0000 7f05fb7fe700 1 --2- 172.21.15.42:0/4279832535 >> v2:172.21.15.42:3300/0 conn(0x7f05fc12f710 0x7f05fc143900 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f05fc0a7360 tx=0x7f05f40013c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:29.945 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:29.927+0000 7f05fb7fe700 1 --2- 172.21.15.42:0/4279832535 >> v2:172.21.15.42:3300/0 conn(0x7f05fc12f710 0x7f05fc143900 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f05fc0a7360 tx=0x7f05f40013c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:29.945 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:29.927+0000 7f05fb7fe700 1 --2- 172.21.15.42:0/4279832535 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f05fc12f710 0x7f05fc143900 secure :-1 s=READY pgs=2999 cs=0 l=1 rx=0x7f05fc0a7360 tx=0x7f05f40013c0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:40:29.945 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:29.927+0000 7f05fb7fe700 1 --2- 172.21.15.42:0/4279832535 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f05fc12f710 0x7f05fc143900 secure :-1 s=THROTTLE_DONE pgs=2999 cs=0 l=1 rx=0x7f05fc0a7360 tx=0x7f05f40013c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:29.946 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:29.927+0000 7f05fb7fe700 1 --2- 172.21.15.42:0/4279832535 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f05fc12f710 0x7f05fc143900 secure :-1 s=THROTTLE_DONE pgs=2999 cs=0 l=1 rx=0x7f05fc0a7360 tx=0x7f05f40013c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:29.946 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:29.927+0000 7f05e3fff700 1 -- 172.21.15.42:0/4279832535 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f05ec008ad0 con 0x7f05fc12f710 2019-12-05T06:40:29.946 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:29.927+0000 7f05fb7fe700 1 --2- 172.21.15.42:0/4279832535 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f05fc12f710 0x7f05fc143900 secure :-1 s=THROTTLE_DONE pgs=2999 cs=0 l=1 rx=0x7f05fc0a7360 tx=0x7f05f40013c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:29.946 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:29.927+0000 7f05e3fff700 1 -- 172.21.15.42:0/4279832535 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05ec0114f0 con 0x7f05fc12f710 2019-12-05T06:40:29.946 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:29.927+0000 7f0601975700 1 -- 172.21.15.42:0/4279832535 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05fc137b70 con 0x7f05fc12f710 2019-12-05T06:40:29.946 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:29.927+0000 7f05e3fff700 1 -- 172.21.15.42:0/4279832535 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f05ec007340 con 0x7f05fc12f710 2019-12-05T06:40:29.946 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:29.927+0000 7f0601975700 1 -- 172.21.15.42:0/4279832535 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05fc138120 con 0x7f05fc12f710 2019-12-05T06:40:29.948 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:29.927+0000 7f05fb7fe700 1 --2- 172.21.15.42:0/4279832535 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f05fc12f710 0x7f05fc143900 secure :-1 s=THROTTLE_DONE pgs=2999 cs=0 l=1 rx=0x7f05fc0a7360 tx=0x7f05f40013c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:29.948 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:29.927+0000 7f05e3fff700 1 -- 172.21.15.42:0/4279832535 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f05ec005540 con 0x7f05fc12f710 2019-12-05T06:40:29.949 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:29.931+0000 7f05fb7fe700 1 --2- 172.21.15.42:0/4279832535 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f05fc12f710 0x7f05fc143900 secure :-1 s=THROTTLE_DONE pgs=2999 cs=0 l=1 rx=0x7f05fc0a7360 tx=0x7f05f40013c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:29.949 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:29.931+0000 7f05e3fff700 1 --2- 172.21.15.42:0/4279832535 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f05dc03ac90 0x7f05dc03d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:40:29.949 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:29.931+0000 7f05e3fff700 1 -- 172.21.15.42:0/4279832535 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 11621+0+0 (secure 0 0 0) 0x7f05ec08f740 con 0x7f05fc12f710 2019-12-05T06:40:29.950 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:29.931+0000 7f05faffd700 1 --2- 172.21.15.42:0/4279832535 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f05dc03ac90 0x7f05dc03d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:40:29.950 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:29.931+0000 7f05faffd700 1 --2- 172.21.15.42:0/4279832535 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f05dc03ac90 0x7f05dc03d1a0 crc :-1 s=READY pgs=1254 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:40:29.950 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:29.935+0000 7f0601975700 1 -- 172.21.15.42:0/4279832535 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05fc05d110 con 0x7f05fc12f710 2019-12-05T06:40:29.950 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:29.939+0000 7f05fb7fe700 1 --2- 172.21.15.42:0/4279832535 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f05fc12f710 0x7f05fc143900 secure :-1 s=THROTTLE_DONE pgs=2999 cs=0 l=1 rx=0x7f05fc0a7360 tx=0x7f05f40013c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:29.950 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:29.939+0000 7f05e3fff700 1 -- 172.21.15.42:0/4279832535 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f05ec0775d0 con 0x7f05fc12f710 2019-12-05T06:40:30.112 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:30.103+0000 7f0601975700 1 -- 172.21.15.42:0/4279832535 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f05fc05b8c0 con 0x7f05fc12f710 2019-12-05T06:40:30.113 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:30.103+0000 7f05fb7fe700 1 --2- 172.21.15.42:0/4279832535 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f05fc12f710 0x7f05fc143900 secure :-1 s=THROTTLE_DONE pgs=2999 cs=0 l=1 rx=0x7f05fc0a7360 tx=0x7f05f40013c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:30.113 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:30.103+0000 7f05e3fff700 1 -- 172.21.15.42:0/4279832535 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 360 v360) v1 ==== 95+0+2111 (secure 0 0 0) 0x7f05ec056c40 con 0x7f05fc12f710 2019-12-05T06:40:30.113 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 360 2019-12-05T06:40:30.113 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:40:30.114 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":360,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":11976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/474728597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":474728597},{"type":"v1","addr":"172.21.15.40:6833","nonce":474728597}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":360}],"filesystems":[{"mdsmap":{"epoch":359,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:40:13.247006+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":114,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11868},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11868":{"gid":11868,"name":"a","rank":0,"incarnation":355,"state":"up:active","state_seq":12,"addr":"172.21.15.42:6835/406829134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":406829134},{"type":"v1","addr":"172.21.15.42:6835","nonce":406829134}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:40:30.115 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:30.107+0000 7f0601975700 1 -- 172.21.15.42:0/4279832535 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f05dc03ac90 msgr2=0x7f05dc03d1a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:30.115 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:30.107+0000 7f0601975700 1 --2- 172.21.15.42:0/4279832535 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f05dc03ac90 0x7f05dc03d1a0 crc :-1 s=READY pgs=1254 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:40:30.115 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:30.107+0000 7f0601975700 1 -- 172.21.15.42:0/4279832535 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f05fc12f710 msgr2=0x7f05fc143900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:30.115 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:30.107+0000 7f0601975700 1 --2- 172.21.15.42:0/4279832535 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f05fc12f710 0x7f05fc143900 secure :-1 s=READY pgs=2999 cs=0 l=1 rx=0x7f05fc0a7360 tx=0x7f05f40013c0).stop 2019-12-05T06:40:30.115 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:30.107+0000 7f0601975700 1 -- 172.21.15.42:0/4279832535 shutdown_connections 2019-12-05T06:40:30.116 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:30.107+0000 7f0601975700 1 --2- 172.21.15.42:0/4279832535 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f05dc03ac90 0x7f05dc03d1a0 unknown :-1 s=CLOSED pgs=1254 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:40:30.116 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:30.107+0000 7f0601975700 1 --2- 172.21.15.42:0/4279832535 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f05fc12f710 0x7f05fc143900 unknown :-1 s=CLOSED pgs=2999 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:40:30.116 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:30.107+0000 7f0601975700 1 -- 172.21.15.42:0/4279832535 shutdown_connections 2019-12-05T06:40:30.116 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:30.107+0000 7f0601975700 1 -- 172.21.15.42:0/4279832535 wait complete. 2019-12-05T06:40:30.116 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:30.107+0000 7f0601975700 1 -- 172.21.15.42:0/4279832535 >> 172.21.15.42:0/4279832535 conn(0x7f05fc07e680 msgr2=0x7f05fc11d600 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:40:30.125 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:40:32.126 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:40:32.237 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:32.223+0000 7fb6333aa700 1 Processor -- start 2019-12-05T06:40:32.237 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:32.223+0000 7fb6333aa700 1 -- start start 2019-12-05T06:40:32.237 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:32.223+0000 7fb6333aa700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb62c0a7730 0x7fb62c08df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:40:32.237 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:32.223+0000 7fb6333aa700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb62c07e710 0x7fb62c07eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:40:32.238 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:32.223+0000 7fb6333aa700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb62c09f2d0 0x7fb62c09f700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:40:32.238 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:32.223+0000 7fb6333aa700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb62c0a0af0 con 0x7fb62c09f2d0 2019-12-05T06:40:32.238 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:32.223+0000 7fb6333aa700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb62c08e5a0 con 0x7fb62c0a7730 2019-12-05T06:40:32.238 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:32.223+0000 7fb6333aa700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb62c08e840 con 0x7fb62c07e710 2019-12-05T06:40:32.238 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:32.227+0000 7fb631947700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb62c0a7730 0x7fb62c08df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:40:32.238 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:32.227+0000 7fb631947700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb62c0a7730 0x7fb62c08df70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:60974/0 (socket says 172.21.15.42:60974) 2019-12-05T06:40:32.238 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:32.227+0000 7fb631947700 1 -- 172.21.15.42:0/1060706133 learned_addr learned my addr 172.21.15.42:0/1060706133 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:40:32.239 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:32.227+0000 7fb631146700 1 --2- 172.21.15.42:0/1060706133 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb62c07e710 0x7fb62c07eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:40:32.239 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:32.227+0000 7fb632148700 1 --2- 172.21.15.42:0/1060706133 >> v2:172.21.15.40:3300/0 conn(0x7fb62c09f2d0 0x7fb62c09f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:40:32.239 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:32.227+0000 7fb631947700 1 -- 172.21.15.42:0/1060706133 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb62c07e710 msgr2=0x7fb62c07eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:40:32.239 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:32.227+0000 7fb631947700 1 --2- 172.21.15.42:0/1060706133 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb62c07e710 0x7fb62c07eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:40:32.239 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:32.227+0000 7fb631947700 1 -- 172.21.15.42:0/1060706133 >> v2:172.21.15.40:3300/0 conn(0x7fb62c09f2d0 msgr2=0x7fb62c09f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:40:32.240 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:32.227+0000 7fb631947700 1 --2- 172.21.15.42:0/1060706133 >> v2:172.21.15.40:3300/0 conn(0x7fb62c09f2d0 0x7fb62c09f700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:40:32.240 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:32.227+0000 7fb631947700 1 -- 172.21.15.42:0/1060706133 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb62c07f090 con 0x7fb62c0a7730 2019-12-05T06:40:32.240 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:32.227+0000 7fb631947700 1 --2- 172.21.15.42:0/1060706133 >> v2:172.21.15.42:3300/0 conn(0x7fb62c0a7730 0x7fb62c08df70 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fb62c08e760 tx=0x7fb62000fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:32.240 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:32.227+0000 7fb631947700 1 --2- 172.21.15.42:0/1060706133 >> v2:172.21.15.42:3300/0 conn(0x7fb62c0a7730 0x7fb62c08df70 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fb62c08e760 tx=0x7fb62000fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:32.240 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:32.227+0000 7fb631947700 1 --2- 172.21.15.42:0/1060706133 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb62c0a7730 0x7fb62c08df70 secure :-1 s=READY pgs=3000 cs=0 l=1 rx=0x7fb62c08e760 tx=0x7fb62000fdb0).ready entity=mon.1 client_cookie=17fcb10f3da6773d server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:40:32.241 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:32.227+0000 7fb631947700 1 --2- 172.21.15.42:0/1060706133 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb62c0a7730 0x7fb62c08df70 secure :-1 s=THROTTLE_DONE pgs=3000 cs=0 l=1 rx=0x7fb62c08e760 tx=0x7fb62000fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:32.241 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:32.227+0000 7fb631947700 1 --2- 172.21.15.42:0/1060706133 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb62c0a7730 0x7fb62c08df70 secure :-1 s=THROTTLE_DONE pgs=3000 cs=0 l=1 rx=0x7fb62c08e760 tx=0x7fb62000fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:32.241 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:32.227+0000 7fb630945700 1 -- 172.21.15.42:0/1060706133 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb620008c90 con 0x7fb62c0a7730 2019-12-05T06:40:32.241 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:32.227+0000 7fb631947700 1 --2- 172.21.15.42:0/1060706133 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb62c0a7730 0x7fb62c08df70 secure :-1 s=THROTTLE_DONE pgs=3000 cs=0 l=1 rx=0x7fb62c08e760 tx=0x7fb62000fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:32.242 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:32.227+0000 7fb630945700 1 -- 172.21.15.42:0/1060706133 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6200088b0 con 0x7fb62c0a7730 2019-12-05T06:40:32.242 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:32.227+0000 7fb630945700 1 -- 172.21.15.42:0/1060706133 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb620011b50 con 0x7fb62c0a7730 2019-12-05T06:40:32.242 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:32.227+0000 7fb6333aa700 1 -- 172.21.15.42:0/1060706133 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb62c0a7730 msgr2=0x7fb62c08df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:32.242 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:32.227+0000 7fb6333aa700 1 --2- 172.21.15.42:0/1060706133 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb62c0a7730 0x7fb62c08df70 secure :-1 s=READY pgs=3000 cs=0 l=1 rx=0x7fb62c08e760 tx=0x7fb62000fdb0).stop 2019-12-05T06:40:32.242 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:32.227+0000 7fb6333aa700 1 -- 172.21.15.42:0/1060706133 shutdown_connections 2019-12-05T06:40:32.242 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:32.227+0000 7fb6333aa700 1 --2- 172.21.15.42:0/1060706133 >> v2:172.21.15.40:3300/0 conn(0x7fb62c09f2d0 0x7fb62c09f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:40:32.242 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:32.227+0000 7fb6333aa700 1 --2- 172.21.15.42:0/1060706133 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb62c0a7730 0x7fb62c08df70 unknown :-1 s=CLOSED pgs=3000 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:40:32.243 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:32.227+0000 7fb6333aa700 1 --2- 172.21.15.42:0/1060706133 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb62c07e710 0x7fb62c07eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:40:32.243 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:32.227+0000 7fb6333aa700 1 -- 172.21.15.42:0/1060706133 shutdown_connections 2019-12-05T06:40:32.243 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:32.227+0000 7fb6333aa700 1 -- 172.21.15.42:0/1060706133 wait complete. 2019-12-05T06:40:32.243 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:32.227+0000 7fb6333aa700 1 -- 172.21.15.42:0/1060706133 >> 172.21.15.42:0/1060706133 conn(0x7fb62c0aabd0 msgr2=0x7fb62c0a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:40:32.249 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:32.231+0000 7fb6333aa700 1 Processor -- start 2019-12-05T06:40:32.249 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:32.231+0000 7fb6333aa700 1 -- start start 2019-12-05T06:40:32.249 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:32.231+0000 7fb6333aa700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb62c063de0 0x7fb62c0722b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:40:32.250 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:32.231+0000 7fb6333aa700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb62c072800 0x7fb62c074c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:40:32.250 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:32.231+0000 7fb6333aa700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb62c0a6990 con 0x7fb62c07c9b0 2019-12-05T06:40:32.250 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:32.231+0000 7fb6333aa700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb62c065300 con 0x7fb62c063de0 2019-12-05T06:40:32.250 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:32.231+0000 7fb6333aa700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb62c0655a0 con 0x7fb62c072800 2019-12-05T06:40:32.251 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:32.231+0000 7fb631146700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb62c063de0 0x7fb62c0722b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:40:32.251 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:32.231+0000 7fb632148700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb62c072800 0x7fb62c074c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:40:32.251 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:32.231+0000 7fb631947700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fb62c07c9b0 0x7fb62c0635e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:42226/0 (socket says 172.21.15.42:42226) 2019-12-05T06:40:32.251 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:32.231+0000 7fb631947700 1 -- 172.21.15.42:0/1118235015 learned_addr learned my addr 172.21.15.42:0/1118235015 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:40:32.252 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:32.235+0000 7fb631146700 1 -- 172.21.15.42:0/1118235015 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb62c072800 msgr2=0x7fb62c074c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:32.252 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:32.235+0000 7fb631146700 1 --2- 172.21.15.42:0/1118235015 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb62c072800 0x7fb62c074c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:40:32.252 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:32.235+0000 7fb631146700 1 -- 172.21.15.42:0/1118235015 >> v1:172.21.15.40:6789/0 conn(0x7fb62c07c9b0 legacy=0x7fb62c0635e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:32.252 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:32.235+0000 7fb61e7fc700 1 -- 172.21.15.42:0/1118235015 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3764797457 0 0) 0x7fb62c0a6990 con 0x7fb62c07c9b0 2019-12-05T06:40:32.252 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:32.235+0000 7fb631146700 1 -- 172.21.15.42:0/1118235015 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb62c064f40 con 0x7fb62c063de0 2019-12-05T06:40:32.252 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:32.235+0000 7fb631146700 1 --2- 172.21.15.42:0/1118235015 >> v2:172.21.15.40:3300/0 conn(0x7fb62c063de0 0x7fb62c0722b0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fb62c0654c0 tx=0x7fb628010e80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:32.252 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:32.235+0000 7fb631146700 1 --2- 172.21.15.42:0/1118235015 >> v2:172.21.15.40:3300/0 conn(0x7fb62c063de0 0x7fb62c0722b0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fb62c0654c0 tx=0x7fb628010e80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:32.253 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:32.235+0000 7fb631146700 1 --2- 172.21.15.42:0/1118235015 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb62c063de0 0x7fb62c0722b0 secure :-1 s=READY pgs=2273 cs=0 l=1 rx=0x7fb62c0654c0 tx=0x7fb628010e80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:40:32.253 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:32.235+0000 7fb631146700 1 --2- 172.21.15.42:0/1118235015 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb62c063de0 0x7fb62c0722b0 secure :-1 s=THROTTLE_DONE pgs=2273 cs=0 l=1 rx=0x7fb62c0654c0 tx=0x7fb628010e80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:32.254 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:32.235+0000 7fb61e7fc700 1 -- 172.21.15.42:0/1118235015 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb628012cd0 con 0x7fb62c063de0 2019-12-05T06:40:32.254 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:32.235+0000 7fb631146700 1 --2- 172.21.15.42:0/1118235015 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb62c063de0 0x7fb62c0722b0 secure :-1 s=THROTTLE_DONE pgs=2273 cs=0 l=1 rx=0x7fb62c0654c0 tx=0x7fb628010e80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:32.254 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:32.235+0000 7fb61e7fc700 1 -- 172.21.15.42:0/1118235015 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6280128f0 con 0x7fb62c063de0 2019-12-05T06:40:32.254 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:32.235+0000 7fb631146700 1 --2- 172.21.15.42:0/1118235015 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb62c063de0 0x7fb62c0722b0 secure :-1 s=THROTTLE_DONE pgs=2273 cs=0 l=1 rx=0x7fb62c0654c0 tx=0x7fb628010e80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:32.254 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:32.235+0000 7fb6333aa700 1 -- 172.21.15.42:0/1118235015 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb62c0a5e70 con 0x7fb62c063de0 2019-12-05T06:40:32.255 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:32.235+0000 7fb61e7fc700 1 -- 172.21.15.42:0/1118235015 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb628005ca0 con 0x7fb62c063de0 2019-12-05T06:40:32.255 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:32.235+0000 7fb6333aa700 1 -- 172.21.15.42:0/1118235015 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb62c063180 con 0x7fb62c063de0 2019-12-05T06:40:32.256 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:32.235+0000 7fb631146700 1 --2- 172.21.15.42:0/1118235015 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb62c063de0 0x7fb62c0722b0 secure :-1 s=THROTTLE_DONE pgs=2273 cs=0 l=1 rx=0x7fb62c0654c0 tx=0x7fb628010e80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:32.256 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:32.235+0000 7fb61e7fc700 1 -- 172.21.15.42:0/1118235015 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fb628007150 con 0x7fb62c063de0 2019-12-05T06:40:32.256 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:32.235+0000 7fb631146700 1 --2- 172.21.15.42:0/1118235015 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb62c063de0 0x7fb62c0722b0 secure :-1 s=THROTTLE_DONE pgs=2273 cs=0 l=1 rx=0x7fb62c0654c0 tx=0x7fb628010e80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:32.256 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:32.235+0000 7fb61e7fc700 1 --2- 172.21.15.42:0/1118235015 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb61403abc0 0x7fb61403d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:40:32.256 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:32.235+0000 7fb61e7fc700 1 -- 172.21.15.42:0/1118235015 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 11621+0+0 (secure 0 0 0) 0x7fb6280090a0 con 0x7fb62c063de0 2019-12-05T06:40:32.256 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:32.235+0000 7fb631947700 1 --2- 172.21.15.42:0/1118235015 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb61403abc0 0x7fb61403d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:40:32.256 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:32.239+0000 7fb631947700 1 --2- 172.21.15.42:0/1118235015 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb61403abc0 0x7fb61403d0d0 crc :-1 s=READY pgs=1255 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:40:32.257 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:32.239+0000 7fb6333aa700 1 -- 172.21.15.42:0/1118235015 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb62c05d100 con 0x7fb62c063de0 2019-12-05T06:40:32.257 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:32.247+0000 7fb631146700 1 --2- 172.21.15.42:0/1118235015 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb62c063de0 0x7fb62c0722b0 secure :-1 s=THROTTLE_DONE pgs=2273 cs=0 l=1 rx=0x7fb62c0654c0 tx=0x7fb628010e80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:32.257 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:32.247+0000 7fb61e7fc700 1 -- 172.21.15.42:0/1118235015 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fb628032890 con 0x7fb62c063de0 2019-12-05T06:40:32.430 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:32.419+0000 7fb6333aa700 1 -- 172.21.15.42:0/1118235015 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb62c05b8b0 con 0x7fb62c063de0 2019-12-05T06:40:32.431 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:32.419+0000 7fb631146700 1 --2- 172.21.15.42:0/1118235015 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb62c063de0 0x7fb62c0722b0 secure :-1 s=THROTTLE_DONE pgs=2273 cs=0 l=1 rx=0x7fb62c0654c0 tx=0x7fb628010e80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:32.431 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:32.419+0000 7fb61e7fc700 1 -- 172.21.15.42:0/1118235015 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 360 v360) v1 ==== 95+0+2111 (secure 0 0 0) 0x7fb628032af0 con 0x7fb62c063de0 2019-12-05T06:40:32.432 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:40:32.432 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":360,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":11976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/474728597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":474728597},{"type":"v1","addr":"172.21.15.40:6833","nonce":474728597}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":360}],"filesystems":[{"mdsmap":{"epoch":359,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:40:13.247006+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":114,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11868},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11868":{"gid":11868,"name":"a","rank":0,"incarnation":355,"state":"up:active","state_seq":12,"addr":"172.21.15.42:6835/406829134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":406829134},{"type":"v1","addr":"172.21.15.42:6835","nonce":406829134}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:40:32.432 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 360 2019-12-05T06:40:32.434 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:32.423+0000 7fb6333aa700 1 -- 172.21.15.42:0/1118235015 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb61403abc0 msgr2=0x7fb61403d0d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:32.434 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:32.423+0000 7fb6333aa700 1 --2- 172.21.15.42:0/1118235015 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb61403abc0 0x7fb61403d0d0 crc :-1 s=READY pgs=1255 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:40:32.434 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:32.423+0000 7fb6333aa700 1 -- 172.21.15.42:0/1118235015 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb62c063de0 msgr2=0x7fb62c0722b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:32.434 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:32.423+0000 7fb6333aa700 1 --2- 172.21.15.42:0/1118235015 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb62c063de0 0x7fb62c0722b0 secure :-1 s=READY pgs=2273 cs=0 l=1 rx=0x7fb62c0654c0 tx=0x7fb628010e80).stop 2019-12-05T06:40:32.434 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:32.423+0000 7fb6333aa700 1 -- 172.21.15.42:0/1118235015 shutdown_connections 2019-12-05T06:40:32.435 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:32.423+0000 7fb6333aa700 1 --2- 172.21.15.42:0/1118235015 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb61403abc0 0x7fb61403d0d0 unknown :-1 s=CLOSED pgs=1255 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:40:32.435 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:32.423+0000 7fb6333aa700 1 --2- 172.21.15.42:0/1118235015 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb62c072800 0x7fb62c074c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:40:32.435 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:32.423+0000 7fb6333aa700 1 --2- 172.21.15.42:0/1118235015 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb62c063de0 0x7fb62c0722b0 unknown :-1 s=CLOSED pgs=2273 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:40:32.435 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:32.427+0000 7fb6333aa700 1 -- 172.21.15.42:0/1118235015 shutdown_connections 2019-12-05T06:40:32.435 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:32.427+0000 7fb6333aa700 1 -- 172.21.15.42:0/1118235015 wait complete. 2019-12-05T06:40:32.435 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:32.427+0000 7fb6333aa700 1 -- 172.21.15.42:0/1118235015 >> 172.21.15.42:0/1118235015 conn(0x7fb62c0a6be0 msgr2=0x7fb62c07e4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:40:32.447 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:40:34.448 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:40:34.549 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:34.539+0000 7f85b2eea700 1 Processor -- start 2019-12-05T06:40:34.549 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:34.539+0000 7f85b2eea700 1 -- start start 2019-12-05T06:40:34.550 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:34.539+0000 7f85b2eea700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f85ac0a7740 0x7f85ac08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:40:34.550 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:34.539+0000 7f85b2eea700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85ac0a0710 con 0x7f85ac09fae0 2019-12-05T06:40:34.550 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:34.539+0000 7f85b2eea700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85ac0a9ba0 con 0x7f85ac07e720 2019-12-05T06:40:34.550 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:34.539+0000 7f85b2eea700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85ac0a0b00 con 0x7f85ac0a7740 2019-12-05T06:40:34.550 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:34.539+0000 7f85b0c86700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f85ac07e720 0x7f85ac09f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:32794/0 (socket says 172.21.15.42:32794) 2019-12-05T06:40:34.550 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:34.539+0000 7f85b0c86700 1 -- 172.21.15.42:0/77293781 learned_addr learned my addr 172.21.15.42:0/77293781 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:40:34.551 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:34.539+0000 7f85b1487700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f85ac0a7740 0x7f85ac08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:40:34.551 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:34.539+0000 7f859ffff700 1 -- 172.21.15.42:0/77293781 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 564941418 0 0) 0x7f85ac0a9ba0 con 0x7f85ac07e720 2019-12-05T06:40:34.551 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:34.539+0000 7f859ffff700 1 -- 172.21.15.42:0/77293781 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85980030b0 con 0x7f85ac07e720 2019-12-05T06:40:34.551 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:34.539+0000 7f85b1487700 1 -- 172.21.15.42:0/77293781 >> v1:172.21.15.42:6789/0 conn(0x7f85ac07e720 legacy=0x7f85ac09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:34.551 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:34.539+0000 7f85b1487700 1 -- 172.21.15.42:0/77293781 >> v1:172.21.15.40:6789/0 conn(0x7f85ac09fae0 legacy=0x7f85ac14efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:34.551 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:34.539+0000 7f85b1487700 1 -- 172.21.15.42:0/77293781 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85ac07eb50 con 0x7f85ac0a7740 2019-12-05T06:40:34.551 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:34.539+0000 7f85b1487700 1 --2- 172.21.15.42:0/77293781 >> v2:172.21.15.42:3300/0 conn(0x7f85ac0a7740 0x7f85ac08df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f85a8008790 tx=0x7f85a800fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:34.552 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:34.539+0000 7f85b1487700 1 --2- 172.21.15.42:0/77293781 >> v2:172.21.15.42:3300/0 conn(0x7f85ac0a7740 0x7f85ac08df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f85a8008790 tx=0x7f85a800fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:34.552 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:34.539+0000 7f85b1487700 1 --2- 172.21.15.42:0/77293781 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f85ac0a7740 0x7f85ac08df80 secure :-1 s=READY pgs=3002 cs=0 l=1 rx=0x7f85a8008790 tx=0x7f85a800fbf0).ready entity=mon.1 client_cookie=947319f34f8f7eb4 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:40:34.552 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:34.539+0000 7f85b1487700 1 --2- 172.21.15.42:0/77293781 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f85ac0a7740 0x7f85ac08df80 secure :-1 s=THROTTLE_DONE pgs=3002 cs=0 l=1 rx=0x7f85a8008790 tx=0x7f85a800fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:34.552 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:34.539+0000 7f859ffff700 1 -- 172.21.15.42:0/77293781 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f85a8011270 con 0x7f85ac0a7740 2019-12-05T06:40:34.552 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:34.539+0000 7f85b1487700 1 --2- 172.21.15.42:0/77293781 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f85ac0a7740 0x7f85ac08df80 secure :-1 s=THROTTLE_DONE pgs=3002 cs=0 l=1 rx=0x7f85a8008790 tx=0x7f85a800fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:34.552 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:34.539+0000 7f859ffff700 1 -- 172.21.15.42:0/77293781 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85a8008bc0 con 0x7f85ac0a7740 2019-12-05T06:40:34.553 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:34.539+0000 7f85b1487700 1 --2- 172.21.15.42:0/77293781 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f85ac0a7740 0x7f85ac08df80 secure :-1 s=THROTTLE_DONE pgs=3002 cs=0 l=1 rx=0x7f85a8008790 tx=0x7f85a800fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:34.553 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:34.539+0000 7f859ffff700 1 -- 172.21.15.42:0/77293781 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f85a8011850 con 0x7f85ac0a7740 2019-12-05T06:40:34.553 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:34.539+0000 7f85b2eea700 1 -- 172.21.15.42:0/77293781 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f85ac0a7740 msgr2=0x7f85ac08df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:34.553 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:34.539+0000 7f85b2eea700 1 --2- 172.21.15.42:0/77293781 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f85ac0a7740 0x7f85ac08df80 secure :-1 s=READY pgs=3002 cs=0 l=1 rx=0x7f85a8008790 tx=0x7f85a800fbf0).stop 2019-12-05T06:40:34.553 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:34.543+0000 7f85b2eea700 1 -- 172.21.15.42:0/77293781 shutdown_connections 2019-12-05T06:40:34.553 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:34.543+0000 7f85b2eea700 1 --2- 172.21.15.42:0/77293781 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f85ac0a7740 0x7f85ac08df80 unknown :-1 s=CLOSED pgs=3002 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:40:34.553 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:34.543+0000 7f85b2eea700 1 -- 172.21.15.42:0/77293781 shutdown_connections 2019-12-05T06:40:34.554 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:34.543+0000 7f85b2eea700 1 -- 172.21.15.42:0/77293781 wait complete. 2019-12-05T06:40:34.554 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:34.543+0000 7f85b2eea700 1 -- 172.21.15.42:0/77293781 >> 172.21.15.42:0/77293781 conn(0x7f85ac0aabe0 msgr2=0x7f85ac0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:40:34.557 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:34.543+0000 7f85b2eea700 1 Processor -- start 2019-12-05T06:40:34.557 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:34.543+0000 7f85b2eea700 1 -- start start 2019-12-05T06:40:34.557 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:34.547+0000 7f85b2eea700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f85ac07b950 0x7f85ac144730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:40:34.558 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:34.547+0000 7f85b2eea700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85ac144c80 con 0x7f85ac1221a0 2019-12-05T06:40:34.558 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:34.547+0000 7f85b2eea700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85ac14ff10 con 0x7f85ac142cf0 2019-12-05T06:40:34.558 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:34.547+0000 7f85b2eea700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85ac144f60 con 0x7f85ac07b950 2019-12-05T06:40:34.558 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:34.547+0000 7f85b1487700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f85ac142cf0 0x7f85ac07b150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:32802/0 (socket says 172.21.15.42:32802) 2019-12-05T06:40:34.558 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:34.547+0000 7f85b1487700 1 -- 172.21.15.42:0/1541255093 learned_addr learned my addr 172.21.15.42:0/1541255093 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:40:34.558 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:34.547+0000 7f85b0c86700 1 --2- 172.21.15.42:0/1541255093 >> v2:172.21.15.40:3300/0 conn(0x7f85ac07b950 0x7f85ac144730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:40:34.559 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:34.547+0000 7f859dffb700 1 -- 172.21.15.42:0/1541255093 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2693226945 0 0) 0x7f85ac14ff10 con 0x7f85ac142cf0 2019-12-05T06:40:34.559 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:34.547+0000 7f859dffb700 1 -- 172.21.15.42:0/1541255093 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f858c0030b0 con 0x7f85ac142cf0 2019-12-05T06:40:34.559 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:34.547+0000 7f85b0c86700 1 -- 172.21.15.42:0/1541255093 >> v1:172.21.15.42:6789/0 conn(0x7f85ac142cf0 legacy=0x7f85ac07b150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:34.559 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:34.547+0000 7f85b0c86700 1 -- 172.21.15.42:0/1541255093 >> v1:172.21.15.40:6789/0 conn(0x7f85ac1221a0 legacy=0x7f85ac1225d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:34.559 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:34.547+0000 7f85b0c86700 1 -- 172.21.15.42:0/1541255093 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85ac12fb40 con 0x7f85ac07b950 2019-12-05T06:40:34.559 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:34.547+0000 7f85b0c86700 1 --2- 172.21.15.42:0/1541255093 >> v2:172.21.15.40:3300/0 conn(0x7f85ac07b950 0x7f85ac144730 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f85ac0aaf70 tx=0x7f85a0010bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:34.560 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:34.547+0000 7f85b0c86700 1 --2- 172.21.15.42:0/1541255093 >> v2:172.21.15.40:3300/0 conn(0x7f85ac07b950 0x7f85ac144730 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f85ac0aaf70 tx=0x7f85a0010bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:34.560 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:34.547+0000 7f85b0c86700 1 --2- 172.21.15.42:0/1541255093 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f85ac07b950 0x7f85ac144730 secure :-1 s=READY pgs=2276 cs=0 l=1 rx=0x7f85ac0aaf70 tx=0x7f85a0010bc0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:40:34.560 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:34.547+0000 7f85b0c86700 1 --2- 172.21.15.42:0/1541255093 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f85ac07b950 0x7f85ac144730 secure :-1 s=THROTTLE_DONE pgs=2276 cs=0 l=1 rx=0x7f85ac0aaf70 tx=0x7f85a0010bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:34.560 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:34.547+0000 7f859dffb700 1 -- 172.21.15.42:0/1541255093 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f85a0012240 con 0x7f85ac07b950 2019-12-05T06:40:34.560 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:34.547+0000 7f85b0c86700 1 --2- 172.21.15.42:0/1541255093 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f85ac07b950 0x7f85ac144730 secure :-1 s=THROTTLE_DONE pgs=2276 cs=0 l=1 rx=0x7f85ac0aaf70 tx=0x7f85a0010bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:34.560 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:34.547+0000 7f859dffb700 1 -- 172.21.15.42:0/1541255093 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85a0005bd0 con 0x7f85ac07b950 2019-12-05T06:40:34.560 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:34.547+0000 7f85b0c86700 1 --2- 172.21.15.42:0/1541255093 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f85ac07b950 0x7f85ac144730 secure :-1 s=THROTTLE_DONE pgs=2276 cs=0 l=1 rx=0x7f85ac0aaf70 tx=0x7f85a0010bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:34.561 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:34.547+0000 7f85b2eea700 1 -- 172.21.15.42:0/1541255093 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85ac12db20 con 0x7f85ac07b950 2019-12-05T06:40:34.561 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:34.547+0000 7f859dffb700 1 -- 172.21.15.42:0/1541255093 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f85a0012820 con 0x7f85ac07b950 2019-12-05T06:40:34.561 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:34.547+0000 7f85b2eea700 1 -- 172.21.15.42:0/1541255093 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85ac08eb70 con 0x7f85ac07b950 2019-12-05T06:40:34.561 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:34.547+0000 7f85b0c86700 1 --2- 172.21.15.42:0/1541255093 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f85ac07b950 0x7f85ac144730 secure :-1 s=THROTTLE_DONE pgs=2276 cs=0 l=1 rx=0x7f85ac0aaf70 tx=0x7f85a0010bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:34.561 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:34.551+0000 7f859dffb700 1 -- 172.21.15.42:0/1541255093 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f85a0012b40 con 0x7f85ac07b950 2019-12-05T06:40:34.561 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:34.551+0000 7f85b0c86700 1 --2- 172.21.15.42:0/1541255093 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f85ac07b950 0x7f85ac144730 secure :-1 s=THROTTLE_DONE pgs=2276 cs=0 l=1 rx=0x7f85ac0aaf70 tx=0x7f85a0010bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:34.561 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:34.551+0000 7f859dffb700 1 --2- 172.21.15.42:0/1541255093 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f858c03d3a0 0x7f858c03f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:40:34.562 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:34.551+0000 7f859dffb700 1 -- 172.21.15.42:0/1541255093 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 11621+0+0 (secure 0 0 0) 0x7f85a004b590 con 0x7f85ac07b950 2019-12-05T06:40:34.562 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:34.551+0000 7f85b1487700 1 --2- 172.21.15.42:0/1541255093 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f858c03d3a0 0x7f858c03f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:40:34.562 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:34.551+0000 7f85b1487700 1 --2- 172.21.15.42:0/1541255093 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f858c03d3a0 0x7f858c03f8b0 crc :-1 s=READY pgs=1256 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:40:34.562 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:34.555+0000 7f85b2eea700 1 -- 172.21.15.42:0/1541255093 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85ac144520 con 0x7f85ac07b950 2019-12-05T06:40:34.572 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:34.567+0000 7f85b0c86700 1 --2- 172.21.15.42:0/1541255093 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f85ac07b950 0x7f85ac144730 secure :-1 s=THROTTLE_DONE pgs=2276 cs=0 l=1 rx=0x7f85ac0aaf70 tx=0x7f85a0010bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:34.572 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:34.567+0000 7f859dffb700 1 -- 172.21.15.42:0/1541255093 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f85a0032610 con 0x7f85ac07b950 2019-12-05T06:40:34.740 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:34.731+0000 7f85b2eea700 1 -- 172.21.15.42:0/1541255093 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f85ac05b8c0 con 0x7f85ac07b950 2019-12-05T06:40:34.741 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:34.735+0000 7f85b0c86700 1 --2- 172.21.15.42:0/1541255093 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f85ac07b950 0x7f85ac144730 secure :-1 s=THROTTLE_DONE pgs=2276 cs=0 l=1 rx=0x7f85ac0aaf70 tx=0x7f85a0010bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:34.741 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:34.735+0000 7f859dffb700 1 -- 172.21.15.42:0/1541255093 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 360 v360) v1 ==== 95+0+2111 (secure 0 0 0) 0x7f85a0032870 con 0x7f85ac07b950 2019-12-05T06:40:34.743 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 360 2019-12-05T06:40:34.743 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:40:34.743 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":360,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":11976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/474728597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":474728597},{"type":"v1","addr":"172.21.15.40:6833","nonce":474728597}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":360}],"filesystems":[{"mdsmap":{"epoch":359,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:40:13.247006+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":114,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11868},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11868":{"gid":11868,"name":"a","rank":0,"incarnation":355,"state":"up:active","state_seq":12,"addr":"172.21.15.42:6835/406829134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":406829134},{"type":"v1","addr":"172.21.15.42:6835","nonce":406829134}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:40:34.746 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:34.739+0000 7f85b2eea700 1 -- 172.21.15.42:0/1541255093 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f858c03d3a0 msgr2=0x7f858c03f8b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:34.746 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:34.739+0000 7f85b2eea700 1 --2- 172.21.15.42:0/1541255093 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f858c03d3a0 0x7f858c03f8b0 crc :-1 s=READY pgs=1256 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:40:34.746 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:34.739+0000 7f85b2eea700 1 -- 172.21.15.42:0/1541255093 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f85ac07b950 msgr2=0x7f85ac144730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:34.746 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:34.739+0000 7f85b2eea700 1 --2- 172.21.15.42:0/1541255093 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f85ac07b950 0x7f85ac144730 secure :-1 s=READY pgs=2276 cs=0 l=1 rx=0x7f85ac0aaf70 tx=0x7f85a0010bc0).stop 2019-12-05T06:40:34.746 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:34.739+0000 7f85b2eea700 1 -- 172.21.15.42:0/1541255093 shutdown_connections 2019-12-05T06:40:34.747 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:34.739+0000 7f85b2eea700 1 --2- 172.21.15.42:0/1541255093 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f858c03d3a0 0x7f858c03f8b0 unknown :-1 s=CLOSED pgs=1256 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:40:34.747 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:34.739+0000 7f85b2eea700 1 --2- 172.21.15.42:0/1541255093 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f85ac07b950 0x7f85ac144730 unknown :-1 s=CLOSED pgs=2276 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:40:34.747 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:34.739+0000 7f85b2eea700 1 -- 172.21.15.42:0/1541255093 shutdown_connections 2019-12-05T06:40:34.747 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:34.739+0000 7f85b2eea700 1 -- 172.21.15.42:0/1541255093 wait complete. 2019-12-05T06:40:34.747 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:34.739+0000 7f85b2eea700 1 -- 172.21.15.42:0/1541255093 >> 172.21.15.42:0/1541255093 conn(0x7f85ac07dd40 msgr2=0x7f85ac059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:40:34.756 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:40:36.757 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:40:36.871 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:36.851+0000 7fe777ffb700 1 Processor -- start 2019-12-05T06:40:36.871 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:36.851+0000 7fe777ffb700 1 -- start start 2019-12-05T06:40:36.871 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:36.855+0000 7fe777ffb700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe77008df80 0x7fe77009f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:40:36.871 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:36.855+0000 7fe777ffb700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe77009fc60 con 0x7fe77009f830 2019-12-05T06:40:36.871 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:36.855+0000 7fe777ffb700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7700a0710 con 0x7fe7700a7740 2019-12-05T06:40:36.872 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:36.855+0000 7fe777ffb700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7700a0b00 con 0x7fe77008df80 2019-12-05T06:40:36.872 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:36.855+0000 7fe775d97700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe77008df80 0x7fe77009f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:40:36.872 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:36.855+0000 7fe776598700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fe7700a7740 0x7fe77007e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:32808/0 (socket says 172.21.15.42:32808) 2019-12-05T06:40:36.872 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:36.855+0000 7fe776598700 1 -- 172.21.15.42:0/3815296532 learned_addr learned my addr 172.21.15.42:0/3815296532 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:40:36.872 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:36.855+0000 7fe775d97700 1 -- 172.21.15.42:0/3815296532 >> v1:172.21.15.42:6789/0 conn(0x7fe7700a7740 legacy=0x7fe77007e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:36.872 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:36.855+0000 7fe775d97700 1 -- 172.21.15.42:0/3815296532 >> v1:172.21.15.40:6789/0 conn(0x7fe77009f830 legacy=0x7fe77014efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:36.873 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:36.855+0000 7fe775d97700 1 -- 172.21.15.42:0/3815296532 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe77007ef20 con 0x7fe77008df80 2019-12-05T06:40:36.873 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:36.855+0000 7fe775d97700 1 --2- 172.21.15.42:0/3815296532 >> v2:172.21.15.42:3300/0 conn(0x7fe77008df80 0x7fe77009f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fe760008790 tx=0x7fe76000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:36.873 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:36.855+0000 7fe775d97700 1 --2- 172.21.15.42:0/3815296532 >> v2:172.21.15.42:3300/0 conn(0x7fe77008df80 0x7fe77009f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fe760008790 tx=0x7fe76000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:36.873 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:36.855+0000 7fe775d97700 1 --2- 172.21.15.42:0/3815296532 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe77008df80 0x7fe77009f2e0 secure :-1 s=READY pgs=3005 cs=0 l=1 rx=0x7fe760008790 tx=0x7fe76000fbf0).ready entity=mon.1 client_cookie=62ccc0f555eed368 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:40:36.873 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:36.855+0000 7fe775d97700 1 --2- 172.21.15.42:0/3815296532 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe77008df80 0x7fe77009f2e0 secure :-1 s=THROTTLE_DONE pgs=3005 cs=0 l=1 rx=0x7fe760008790 tx=0x7fe76000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:36.873 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:36.855+0000 7fe775596700 1 -- 172.21.15.42:0/3815296532 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe760011270 con 0x7fe77008df80 2019-12-05T06:40:36.873 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:36.855+0000 7fe775d97700 1 --2- 172.21.15.42:0/3815296532 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe77008df80 0x7fe77009f2e0 secure :-1 s=THROTTLE_DONE pgs=3005 cs=0 l=1 rx=0x7fe760008790 tx=0x7fe76000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:36.874 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:36.855+0000 7fe775d97700 1 --2- 172.21.15.42:0/3815296532 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe77008df80 0x7fe77009f2e0 secure :-1 s=THROTTLE_DONE pgs=3005 cs=0 l=1 rx=0x7fe760008790 tx=0x7fe76000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:36.874 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:36.855+0000 7fe775596700 1 -- 172.21.15.42:0/3815296532 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe760008bc0 con 0x7fe77008df80 2019-12-05T06:40:36.874 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:36.855+0000 7fe775596700 1 -- 172.21.15.42:0/3815296532 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe760011850 con 0x7fe77008df80 2019-12-05T06:40:36.874 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:36.855+0000 7fe777ffb700 1 -- 172.21.15.42:0/3815296532 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe77008df80 msgr2=0x7fe77009f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:36.874 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:36.855+0000 7fe777ffb700 1 --2- 172.21.15.42:0/3815296532 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe77008df80 0x7fe77009f2e0 secure :-1 s=READY pgs=3005 cs=0 l=1 rx=0x7fe760008790 tx=0x7fe76000fbf0).stop 2019-12-05T06:40:36.874 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:36.855+0000 7fe777ffb700 1 -- 172.21.15.42:0/3815296532 shutdown_connections 2019-12-05T06:40:36.874 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:36.855+0000 7fe777ffb700 1 --2- 172.21.15.42:0/3815296532 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe77008df80 0x7fe77009f2e0 unknown :-1 s=CLOSED pgs=3005 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:40:36.875 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:36.855+0000 7fe777ffb700 1 -- 172.21.15.42:0/3815296532 shutdown_connections 2019-12-05T06:40:36.875 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:36.855+0000 7fe777ffb700 1 -- 172.21.15.42:0/3815296532 wait complete. 2019-12-05T06:40:36.875 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:36.855+0000 7fe777ffb700 1 -- 172.21.15.42:0/3815296532 >> 172.21.15.42:0/3815296532 conn(0x7fe7700aabe0 msgr2=0x7fe7700a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:40:36.875 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:36.859+0000 7fe777ffb700 1 Processor -- start 2019-12-05T06:40:36.875 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:36.859+0000 7fe777ffb700 1 -- start start 2019-12-05T06:40:36.875 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:36.859+0000 7fe777ffb700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe770122400 0x7fe7700784e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:40:36.875 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:36.859+0000 7fe777ffb700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe770119440 0x7fe770119870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:40:36.876 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:36.859+0000 7fe777ffb700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe77011a500 con 0x7fe770118180 2019-12-05T06:40:36.876 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:36.859+0000 7fe777ffb700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe770118700 con 0x7fe770119440 2019-12-05T06:40:36.876 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:36.859+0000 7fe777ffb700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7700a3520 con 0x7fe770122400 2019-12-05T06:40:36.876 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:36.859+0000 7fe776d99700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe770119440 0x7fe770119870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:40:36.876 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:36.859+0000 7fe776598700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe770122400 0x7fe7700784e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:40:36.876 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:36.859+0000 7fe776d99700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe770119440 0x7fe770119870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.42:48828/0 (socket says 172.21.15.42:48828) 2019-12-05T06:40:36.876 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:36.859+0000 7fe776d99700 1 -- 172.21.15.42:0/615490989 learned_addr learned my addr 172.21.15.42:0/615490989 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:40:36.877 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:36.863+0000 7fe776d99700 1 -- 172.21.15.42:0/615490989 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe770122400 msgr2=0x7fe7700784e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:36.877 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:36.863+0000 7fe776d99700 1 --2- 172.21.15.42:0/615490989 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe770122400 0x7fe7700784e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:40:36.877 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:36.863+0000 7fe776d99700 1 -- 172.21.15.42:0/615490989 >> v1:172.21.15.40:6789/0 conn(0x7fe770118180 legacy=0x7fe7700a3a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:36.877 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:36.863+0000 7fe776d99700 1 -- 172.21.15.42:0/615490989 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe770119dc0 con 0x7fe770119440 2019-12-05T06:40:36.877 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:36.863+0000 7fe776598700 1 --2- 172.21.15.42:0/615490989 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe770122400 0x7fe7700784e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:40:36.877 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:36.863+0000 7fe776d99700 1 --2- 172.21.15.42:0/615490989 >> v2:172.21.15.40:3300/0 conn(0x7fe770119440 0x7fe770119870 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fe760008370 tx=0x7fe76000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:36.877 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:36.863+0000 7fe776d99700 1 --2- 172.21.15.42:0/615490989 >> v2:172.21.15.40:3300/0 conn(0x7fe770119440 0x7fe770119870 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fe760008370 tx=0x7fe76000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:36.878 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:36.863+0000 7fe776d99700 1 --2- 172.21.15.42:0/615490989 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe770119440 0x7fe770119870 secure :-1 s=READY pgs=2279 cs=0 l=1 rx=0x7fe760008370 tx=0x7fe76000fbf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:40:36.878 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:36.863+0000 7fe776d99700 1 --2- 172.21.15.42:0/615490989 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe770119440 0x7fe770119870 secure :-1 s=THROTTLE_DONE pgs=2279 cs=0 l=1 rx=0x7fe760008370 tx=0x7fe76000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:36.878 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:36.863+0000 7fe766ffd700 1 -- 172.21.15.42:0/615490989 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe768006ac0 con 0x7fe770119440 2019-12-05T06:40:36.878 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:36.863+0000 7fe776d99700 1 --2- 172.21.15.42:0/615490989 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe770119440 0x7fe770119870 secure :-1 s=THROTTLE_DONE pgs=2279 cs=0 l=1 rx=0x7fe760008370 tx=0x7fe76000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:36.878 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:36.863+0000 7fe776d99700 1 --2- 172.21.15.42:0/615490989 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe770119440 0x7fe770119870 secure :-1 s=THROTTLE_DONE pgs=2279 cs=0 l=1 rx=0x7fe760008370 tx=0x7fe76000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:36.878 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:36.863+0000 7fe766ffd700 1 -- 172.21.15.42:0/615490989 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7680066e0 con 0x7fe770119440 2019-12-05T06:40:36.878 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:36.863+0000 7fe766ffd700 1 -- 172.21.15.42:0/615490989 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe768002820 con 0x7fe770119440 2019-12-05T06:40:36.879 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:36.863+0000 7fe777ffb700 1 -- 172.21.15.42:0/615490989 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe77012ce10 con 0x7fe770119440 2019-12-05T06:40:36.879 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:36.863+0000 7fe777ffb700 1 -- 172.21.15.42:0/615490989 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe77012d3c0 con 0x7fe770119440 2019-12-05T06:40:36.880 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:36.863+0000 7fe776d99700 1 --2- 172.21.15.42:0/615490989 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe770119440 0x7fe770119870 secure :-1 s=THROTTLE_DONE pgs=2279 cs=0 l=1 rx=0x7fe760008370 tx=0x7fe76000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:36.880 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:36.863+0000 7fe766ffd700 1 -- 172.21.15.42:0/615490989 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fe768002ab0 con 0x7fe770119440 2019-12-05T06:40:36.880 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:36.863+0000 7fe776d99700 1 --2- 172.21.15.42:0/615490989 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe770119440 0x7fe770119870 secure :-1 s=THROTTLE_DONE pgs=2279 cs=0 l=1 rx=0x7fe760008370 tx=0x7fe76000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:36.880 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:36.867+0000 7fe766ffd700 1 --2- 172.21.15.42:0/615490989 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe75803acd0 0x7fe75803d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:40:36.880 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:36.867+0000 7fe766ffd700 1 -- 172.21.15.42:0/615490989 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 11621+0+0 (secure 0 0 0) 0x7fe76804aec0 con 0x7fe770119440 2019-12-05T06:40:36.881 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:36.867+0000 7fe776598700 1 --2- 172.21.15.42:0/615490989 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe75803acd0 0x7fe75803d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:40:36.881 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:36.867+0000 7fe776598700 1 --2- 172.21.15.42:0/615490989 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe75803acd0 0x7fe75803d1e0 crc :-1 s=READY pgs=1257 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:40:36.881 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:36.867+0000 7fe777ffb700 1 -- 172.21.15.42:0/615490989 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe77005d110 con 0x7fe770119440 2019-12-05T06:40:36.883 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:36.875+0000 7fe776d99700 1 --2- 172.21.15.42:0/615490989 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe770119440 0x7fe770119870 secure :-1 s=THROTTLE_DONE pgs=2279 cs=0 l=1 rx=0x7fe760008370 tx=0x7fe76000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:36.883 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:36.875+0000 7fe766ffd700 1 -- 172.21.15.42:0/615490989 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fe768031f20 con 0x7fe770119440 2019-12-05T06:40:37.052 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:37.043+0000 7fe777ffb700 1 -- 172.21.15.42:0/615490989 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe77005b8c0 con 0x7fe770119440 2019-12-05T06:40:37.052 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:37.047+0000 7fe776d99700 1 --2- 172.21.15.42:0/615490989 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe770119440 0x7fe770119870 secure :-1 s=THROTTLE_DONE pgs=2279 cs=0 l=1 rx=0x7fe760008370 tx=0x7fe76000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:37.053 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:37.047+0000 7fe766ffd700 1 -- 172.21.15.42:0/615490989 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 360 v360) v1 ==== 95+0+2111 (secure 0 0 0) 0x7fe768032180 con 0x7fe770119440 2019-12-05T06:40:37.055 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 360 2019-12-05T06:40:37.055 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:40:37.055 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":360,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":11976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/474728597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":474728597},{"type":"v1","addr":"172.21.15.40:6833","nonce":474728597}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":360}],"filesystems":[{"mdsmap":{"epoch":359,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:40:13.247006+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":114,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11868},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11868":{"gid":11868,"name":"a","rank":0,"incarnation":355,"state":"up:active","state_seq":12,"addr":"172.21.15.42:6835/406829134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":406829134},{"type":"v1","addr":"172.21.15.42:6835","nonce":406829134}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:40:37.064 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:37.051+0000 7fe777ffb700 1 -- 172.21.15.42:0/615490989 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe75803acd0 msgr2=0x7fe75803d1e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:37.064 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:37.051+0000 7fe777ffb700 1 --2- 172.21.15.42:0/615490989 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe75803acd0 0x7fe75803d1e0 crc :-1 s=READY pgs=1257 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:40:37.064 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:37.051+0000 7fe777ffb700 1 -- 172.21.15.42:0/615490989 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe770119440 msgr2=0x7fe770119870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:37.065 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:37.051+0000 7fe777ffb700 1 --2- 172.21.15.42:0/615490989 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe770119440 0x7fe770119870 secure :-1 s=READY pgs=2279 cs=0 l=1 rx=0x7fe760008370 tx=0x7fe76000fbf0).stop 2019-12-05T06:40:37.065 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:37.051+0000 7fe777ffb700 1 -- 172.21.15.42:0/615490989 shutdown_connections 2019-12-05T06:40:37.065 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:37.051+0000 7fe777ffb700 1 --2- 172.21.15.42:0/615490989 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe75803acd0 0x7fe75803d1e0 unknown :-1 s=CLOSED pgs=1257 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:40:37.065 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:37.051+0000 7fe777ffb700 1 --2- 172.21.15.42:0/615490989 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe770119440 0x7fe770119870 unknown :-1 s=CLOSED pgs=2279 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:40:37.065 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:37.051+0000 7fe777ffb700 1 --2- 172.21.15.42:0/615490989 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe770122400 0x7fe7700784e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:40:37.065 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:37.051+0000 7fe777ffb700 1 -- 172.21.15.42:0/615490989 shutdown_connections 2019-12-05T06:40:37.065 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:37.051+0000 7fe777ffb700 1 -- 172.21.15.42:0/615490989 wait complete. 2019-12-05T06:40:37.066 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:37.051+0000 7fe777ffb700 1 -- 172.21.15.42:0/615490989 >> 172.21.15.42:0/615490989 conn(0x7fe7700aabe0 msgr2=0x7fe77009f240 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:40:37.069 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:40:39.070 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:40:39.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:39.167+0000 7f0bc435f700 1 Processor -- start 2019-12-05T06:40:39.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:39.167+0000 7f0bc435f700 1 -- start start 2019-12-05T06:40:39.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:39.171+0000 7f0bc435f700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0bbc08df80 0x7f0bbc09f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:40:39.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:39.171+0000 7f0bc435f700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bbc09fc60 con 0x7f0bbc0a7740 2019-12-05T06:40:39.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:39.171+0000 7f0bc435f700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bbc0a0710 con 0x7f0bbc09f830 2019-12-05T06:40:39.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:39.171+0000 7f0bc435f700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bbc0a0b00 con 0x7f0bbc08df80 2019-12-05T06:40:39.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:39.171+0000 7f0bc30fd700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f0bbc09f830 0x7f0bbc14efe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:32830/0 (socket says 172.21.15.42:32830) 2019-12-05T06:40:39.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:39.171+0000 7f0bc30fd700 1 -- 172.21.15.42:0/3758706501 learned_addr learned my addr 172.21.15.42:0/3758706501 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:40:39.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:39.171+0000 7f0bc20fb700 1 --2- 172.21.15.42:0/3758706501 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0bbc08df80 0x7f0bbc09f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:40:39.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:39.171+0000 7f0bc18fa700 1 -- 172.21.15.42:0/3758706501 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 29159327 0 0) 0x7f0bbc0a0710 con 0x7f0bbc09f830 2019-12-05T06:40:39.184 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:39.171+0000 7f0bc18fa700 1 -- 172.21.15.42:0/3758706501 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ba80030b0 con 0x7f0bbc09f830 2019-12-05T06:40:39.184 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:39.171+0000 7f0bc18fa700 1 -- 172.21.15.42:0/3758706501 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3685222266 0 0) 0x7f0bbc09fc60 con 0x7f0bbc0a7740 2019-12-05T06:40:39.184 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:39.171+0000 7f0bc18fa700 1 -- 172.21.15.42:0/3758706501 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0bbc0a0710 con 0x7f0bbc0a7740 2019-12-05T06:40:39.184 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:39.171+0000 7f0bc18fa700 1 -- 172.21.15.42:0/3758706501 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1216671920 0 0) 0x7f0ba80030b0 con 0x7f0bbc09f830 2019-12-05T06:40:39.184 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:39.171+0000 7f0bc18fa700 1 -- 172.21.15.42:0/3758706501 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0bbc08df80 msgr2=0x7f0bbc09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:40:39.184 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:39.171+0000 7f0bc18fa700 1 --2- 172.21.15.42:0/3758706501 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0bbc08df80 0x7f0bbc09f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:40:39.185 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:39.171+0000 7f0bc18fa700 1 -- 172.21.15.42:0/3758706501 >> v1:172.21.15.40:6789/0 conn(0x7f0bbc0a7740 legacy=0x7f0bbc07e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:39.185 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:39.171+0000 7f0bc18fa700 1 -- 172.21.15.42:0/3758706501 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bbc07ef20 con 0x7f0bbc09f830 2019-12-05T06:40:39.185 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:39.171+0000 7f0bc18fa700 1 -- 172.21.15.42:0/3758706501 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7f0bb4001ad0 con 0x7f0bbc09f830 2019-12-05T06:40:39.185 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:39.171+0000 7f0bc18fa700 1 -- 172.21.15.42:0/3758706501 >> v1:172.21.15.42:6789/0 conn(0x7f0bbc09f830 legacy=0x7f0bbc14efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:39.185 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:39.171+0000 7f0bc18fa700 1 --2- 172.21.15.42:0/3758706501 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0ba8003940 0x7f0ba8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:40:39.185 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:39.171+0000 7f0bc18fa700 1 --2- 172.21.15.42:0/3758706501 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0ba8004420 0x7f0ba8004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:40:39.186 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:39.171+0000 7f0bc28fc700 1 --2- 172.21.15.42:0/3758706501 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0ba8003940 0x7f0ba8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:40:39.186 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:39.171+0000 7f0bc18fa700 1 --2- 172.21.15.42:0/3758706501 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0ba8004e10 0x7f0ba804f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:40:39.186 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:39.171+0000 7f0bc18fa700 1 -- 172.21.15.42:0/3758706501 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bbc0a0b00 con 0x7f0ba8004e10 2019-12-05T06:40:39.186 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:39.171+0000 7f0bc18fa700 1 -- 172.21.15.42:0/3758706501 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ba804fc00 con 0x7f0ba8003940 2019-12-05T06:40:39.186 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:39.171+0000 7f0bc18fa700 1 -- 172.21.15.42:0/3758706501 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ba804fe40 con 0x7f0ba8004420 2019-12-05T06:40:39.186 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:39.171+0000 7f0bc20fb700 1 --2- 172.21.15.42:0/3758706501 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0ba8004420 0x7f0ba8004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:40:39.187 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:39.171+0000 7f0bc28fc700 1 -- 172.21.15.42:0/3758706501 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0ba8004420 msgr2=0x7f0ba8004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:40:39.187 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:39.171+0000 7f0bc28fc700 1 --2- 172.21.15.42:0/3758706501 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0ba8004420 0x7f0ba8004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:40:39.187 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:39.171+0000 7f0bc28fc700 1 -- 172.21.15.42:0/3758706501 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0ba8004e10 msgr2=0x7f0ba804f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:40:39.187 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:39.171+0000 7f0bc30fd700 1 --2- 172.21.15.42:0/3758706501 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0ba8004e10 0x7f0ba804f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:40:39.187 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:39.171+0000 7f0bc28fc700 1 --2- 172.21.15.42:0/3758706501 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0ba8004e10 0x7f0ba804f680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:40:39.187 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:39.171+0000 7f0bc28fc700 1 -- 172.21.15.42:0/3758706501 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ba8050140 con 0x7f0ba8003940 2019-12-05T06:40:39.187 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:39.171+0000 7f0bc30fd700 1 --2- 172.21.15.42:0/3758706501 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0ba8004e10 0x7f0ba804f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).send_auth_request state changed! 2019-12-05T06:40:39.188 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:39.171+0000 7f0bc28fc700 1 --2- 172.21.15.42:0/3758706501 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0ba8003940 0x7f0ba8003d90 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f0bb8000fe0 tx=0x7f0bb800be70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:39.188 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:39.171+0000 7f0bc28fc700 1 --2- 172.21.15.42:0/3758706501 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0ba8003940 0x7f0ba8003d90 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f0bb8000fe0 tx=0x7f0bb800be70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:39.188 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:39.171+0000 7f0bc28fc700 1 --2- 172.21.15.42:0/3758706501 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0ba8003940 0x7f0ba8003d90 secure :-1 s=READY pgs=3007 cs=0 l=1 rx=0x7f0bb8000fe0 tx=0x7f0bb800be70).ready entity=mon.1 client_cookie=513ccc0317f9dbcb server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:40:39.188 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:39.171+0000 7f0bc28fc700 1 --2- 172.21.15.42:0/3758706501 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0ba8003940 0x7f0ba8003d90 secure :-1 s=THROTTLE_DONE pgs=3007 cs=0 l=1 rx=0x7f0bb8000fe0 tx=0x7f0bb800be70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:39.188 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:39.171+0000 7f0bc18fa700 1 -- 172.21.15.42:0/3758706501 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0bb8010be0 con 0x7f0ba8003940 2019-12-05T06:40:39.188 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:39.171+0000 7f0bc28fc700 1 --2- 172.21.15.42:0/3758706501 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0ba8003940 0x7f0ba8003d90 secure :-1 s=THROTTLE_DONE pgs=3007 cs=0 l=1 rx=0x7f0bb8000fe0 tx=0x7f0bb800be70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:39.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:39.171+0000 7f0bc28fc700 1 --2- 172.21.15.42:0/3758706501 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0ba8003940 0x7f0ba8003d90 secure :-1 s=THROTTLE_DONE pgs=3007 cs=0 l=1 rx=0x7f0bb8000fe0 tx=0x7f0bb800be70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:39.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:39.171+0000 7f0bc18fa700 1 -- 172.21.15.42:0/3758706501 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bb8010800 con 0x7f0ba8003940 2019-12-05T06:40:39.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:39.171+0000 7f0bc18fa700 1 -- 172.21.15.42:0/3758706501 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0bb800c5b0 con 0x7f0ba8003940 2019-12-05T06:40:39.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:39.171+0000 7f0bc435f700 1 -- 172.21.15.42:0/3758706501 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0ba8003940 msgr2=0x7f0ba8003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:39.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:39.171+0000 7f0bc435f700 1 --2- 172.21.15.42:0/3758706501 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0ba8003940 0x7f0ba8003d90 secure :-1 s=READY pgs=3007 cs=0 l=1 rx=0x7f0bb8000fe0 tx=0x7f0bb800be70).stop 2019-12-05T06:40:39.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:39.171+0000 7f0bc30fd700 1 -- 172.21.15.42:0/3758706501 reap_dead start 2019-12-05T06:40:39.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:39.171+0000 7f0bc435f700 1 -- 172.21.15.42:0/3758706501 shutdown_connections 2019-12-05T06:40:39.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:39.171+0000 7f0bc435f700 1 -- 172.21.15.42:0/3758706501 shutdown_connections 2019-12-05T06:40:39.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:39.171+0000 7f0bc435f700 1 -- 172.21.15.42:0/3758706501 wait complete. 2019-12-05T06:40:39.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:39.171+0000 7f0bc435f700 1 -- 172.21.15.42:0/3758706501 >> 172.21.15.42:0/3758706501 conn(0x7f0bbc0aabe0 msgr2=0x7f0bbc0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:40:39.193 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:39.175+0000 7f0bc435f700 1 Processor -- start 2019-12-05T06:40:39.194 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:39.175+0000 7f0bc435f700 1 -- start start 2019-12-05T06:40:39.194 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:39.179+0000 7f0bc435f700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0bbc07be00 0x7f0bbc0a29e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:40:39.194 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:39.179+0000 7f0bc435f700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bbc0a1510 con 0x7f0bbc0a2f30 2019-12-05T06:40:39.194 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:39.179+0000 7f0bc435f700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bbc1486f0 con 0x7f0bbc08eeb0 2019-12-05T06:40:39.194 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:39.179+0000 7f0bc435f700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bbc076c30 con 0x7f0bbc07be00 2019-12-05T06:40:39.194 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:39.179+0000 7f0bc28fc700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f0bbc08eeb0 0x7f0bbc1417b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:32838/0 (socket says 172.21.15.42:32838) 2019-12-05T06:40:39.194 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:39.179+0000 7f0bc28fc700 1 -- 172.21.15.42:0/2605886244 learned_addr learned my addr 172.21.15.42:0/2605886244 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:40:39.195 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:39.179+0000 7f0bc20fb700 1 --2- 172.21.15.42:0/2605886244 >> v2:172.21.15.42:3300/0 conn(0x7f0bbc07be00 0x7f0bbc0a29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:40:39.195 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:39.179+0000 7f0bc20fb700 1 -- 172.21.15.42:0/2605886244 >> v1:172.21.15.42:6789/0 conn(0x7f0bbc08eeb0 legacy=0x7f0bbc1417b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:39.195 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:39.179+0000 7f0bc20fb700 1 -- 172.21.15.42:0/2605886244 >> v1:172.21.15.40:6789/0 conn(0x7f0bbc0a2f30 legacy=0x7f0bbc1258c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:39.195 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:39.179+0000 7f0baf7fe700 1 -- 172.21.15.42:0/2605886244 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4184034562 0 0) 0x7f0bbc1486f0 con 0x7f0bbc08eeb0 2019-12-05T06:40:39.195 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:39.179+0000 7f0bc20fb700 1 -- 172.21.15.42:0/2605886244 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bbc141fb0 con 0x7f0bbc07be00 2019-12-05T06:40:39.195 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:39.179+0000 7f0bc20fb700 1 --2- 172.21.15.42:0/2605886244 >> v2:172.21.15.42:3300/0 conn(0x7f0bbc07be00 0x7f0bbc0a29e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f0bb40011d0 tx=0x7f0bb4001fa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:39.196 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:39.179+0000 7f0bc20fb700 1 --2- 172.21.15.42:0/2605886244 >> v2:172.21.15.42:3300/0 conn(0x7f0bbc07be00 0x7f0bbc0a29e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f0bb40011d0 tx=0x7f0bb4001fa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:39.196 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:39.179+0000 7f0bc20fb700 1 --2- 172.21.15.42:0/2605886244 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0bbc07be00 0x7f0bbc0a29e0 secure :-1 s=READY pgs=3009 cs=0 l=1 rx=0x7f0bb40011d0 tx=0x7f0bb4001fa0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:40:39.196 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:39.179+0000 7f0bc20fb700 1 --2- 172.21.15.42:0/2605886244 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0bbc07be00 0x7f0bbc0a29e0 secure :-1 s=THROTTLE_DONE pgs=3009 cs=0 l=1 rx=0x7f0bb40011d0 tx=0x7f0bb4001fa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:39.196 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:39.179+0000 7f0baf7fe700 1 -- 172.21.15.42:0/2605886244 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0bb00117b0 con 0x7f0bbc07be00 2019-12-05T06:40:39.196 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:39.179+0000 7f0bc435f700 1 -- 172.21.15.42:0/2605886244 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bbc0a3360 con 0x7f0bbc07be00 2019-12-05T06:40:39.196 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:39.179+0000 7f0bc20fb700 1 --2- 172.21.15.42:0/2605886244 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0bbc07be00 0x7f0bbc0a29e0 secure :-1 s=THROTTLE_DONE pgs=3009 cs=0 l=1 rx=0x7f0bb40011d0 tx=0x7f0bb4001fa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:39.197 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:39.179+0000 7f0baf7fe700 1 -- 172.21.15.42:0/2605886244 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bb00113d0 con 0x7f0bbc07be00 2019-12-05T06:40:39.197 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:39.179+0000 7f0bc20fb700 1 --2- 172.21.15.42:0/2605886244 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0bbc07be00 0x7f0bbc0a29e0 secure :-1 s=THROTTLE_DONE pgs=3009 cs=0 l=1 rx=0x7f0bb40011d0 tx=0x7f0bb4001fa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:39.197 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:39.179+0000 7f0bc435f700 1 -- 172.21.15.42:0/2605886244 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bbc1270d0 con 0x7f0bbc07be00 2019-12-05T06:40:39.197 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:39.179+0000 7f0baf7fe700 1 -- 172.21.15.42:0/2605886244 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0bb000d160 con 0x7f0bbc07be00 2019-12-05T06:40:39.197 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:39.179+0000 7f0bc20fb700 1 --2- 172.21.15.42:0/2605886244 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0bbc07be00 0x7f0bbc0a29e0 secure :-1 s=THROTTLE_DONE pgs=3009 cs=0 l=1 rx=0x7f0bb40011d0 tx=0x7f0bb4001fa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:39.197 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:39.183+0000 7f0baf7fe700 1 -- 172.21.15.42:0/2605886244 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f0bb0007170 con 0x7f0bbc07be00 2019-12-05T06:40:39.198 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:39.183+0000 7f0bc20fb700 1 --2- 172.21.15.42:0/2605886244 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0bbc07be00 0x7f0bbc0a29e0 secure :-1 s=THROTTLE_DONE pgs=3009 cs=0 l=1 rx=0x7f0bb40011d0 tx=0x7f0bb4001fa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:39.198 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:39.183+0000 7f0baf7fe700 1 --2- 172.21.15.42:0/2605886244 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0ba403abe0 0x7f0ba403d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:40:39.198 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:39.183+0000 7f0baf7fe700 1 -- 172.21.15.42:0/2605886244 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 11621+0+0 (secure 0 0 0) 0x7f0bb004ac10 con 0x7f0bbc07be00 2019-12-05T06:40:39.198 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:39.183+0000 7f0bc28fc700 1 --2- 172.21.15.42:0/2605886244 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0ba403abe0 0x7f0ba403d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:40:39.198 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:39.183+0000 7f0bc28fc700 1 --2- 172.21.15.42:0/2605886244 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0ba403abe0 0x7f0ba403d0f0 crc :-1 s=READY pgs=1259 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:40:39.198 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:39.183+0000 7f0bc435f700 1 -- 172.21.15.42:0/2605886244 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bbc05d110 con 0x7f0bbc07be00 2019-12-05T06:40:39.199 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:39.191+0000 7f0bc20fb700 1 --2- 172.21.15.42:0/2605886244 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0bbc07be00 0x7f0bbc0a29e0 secure :-1 s=THROTTLE_DONE pgs=3009 cs=0 l=1 rx=0x7f0bb40011d0 tx=0x7f0bb4001fa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:39.199 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:39.191+0000 7f0baf7fe700 1 -- 172.21.15.42:0/2605886244 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f0bb0031c70 con 0x7f0bbc07be00 2019-12-05T06:40:39.366 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:39.359+0000 7f0bc435f700 1 -- 172.21.15.42:0/2605886244 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0bbc05b8c0 con 0x7f0bbc07be00 2019-12-05T06:40:39.367 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:39.359+0000 7f0bc20fb700 1 --2- 172.21.15.42:0/2605886244 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0bbc07be00 0x7f0bbc0a29e0 secure :-1 s=THROTTLE_DONE pgs=3009 cs=0 l=1 rx=0x7f0bb40011d0 tx=0x7f0bb4001fa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:39.367 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:39.359+0000 7f0baf7fe700 1 -- 172.21.15.42:0/2605886244 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 360 v360) v1 ==== 95+0+2111 (secure 0 0 0) 0x7f0bb0031ed0 con 0x7f0bbc07be00 2019-12-05T06:40:39.367 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 360 2019-12-05T06:40:39.367 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:40:39.368 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":360,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":11976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/474728597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":474728597},{"type":"v1","addr":"172.21.15.40:6833","nonce":474728597}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":360}],"filesystems":[{"mdsmap":{"epoch":359,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:40:13.247006+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":114,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11868},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11868":{"gid":11868,"name":"a","rank":0,"incarnation":355,"state":"up:active","state_seq":12,"addr":"172.21.15.42:6835/406829134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":406829134},{"type":"v1","addr":"172.21.15.42:6835","nonce":406829134}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:40:39.369 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:39.363+0000 7f0bc435f700 1 -- 172.21.15.42:0/2605886244 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0ba403abe0 msgr2=0x7f0ba403d0f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:39.370 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:39.363+0000 7f0bc435f700 1 --2- 172.21.15.42:0/2605886244 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0ba403abe0 0x7f0ba403d0f0 crc :-1 s=READY pgs=1259 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:40:39.370 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:39.363+0000 7f0bc435f700 1 -- 172.21.15.42:0/2605886244 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0bbc07be00 msgr2=0x7f0bbc0a29e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:39.370 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:39.363+0000 7f0bc435f700 1 --2- 172.21.15.42:0/2605886244 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0bbc07be00 0x7f0bbc0a29e0 secure :-1 s=READY pgs=3009 cs=0 l=1 rx=0x7f0bb40011d0 tx=0x7f0bb4001fa0).stop 2019-12-05T06:40:39.370 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:39.363+0000 7f0bc435f700 1 -- 172.21.15.42:0/2605886244 shutdown_connections 2019-12-05T06:40:39.370 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:39.363+0000 7f0bc435f700 1 --2- 172.21.15.42:0/2605886244 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f0ba403abe0 0x7f0ba403d0f0 unknown :-1 s=CLOSED pgs=1259 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:40:39.371 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:39.363+0000 7f0bc435f700 1 --2- 172.21.15.42:0/2605886244 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0bbc07be00 0x7f0bbc0a29e0 unknown :-1 s=CLOSED pgs=3009 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:40:39.371 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:39.363+0000 7f0bc435f700 1 -- 172.21.15.42:0/2605886244 shutdown_connections 2019-12-05T06:40:39.371 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:39.363+0000 7f0bc435f700 1 -- 172.21.15.42:0/2605886244 wait complete. 2019-12-05T06:40:39.371 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:39.363+0000 7f0bc435f700 1 -- 172.21.15.42:0/2605886244 >> 172.21.15.42:0/2605886244 conn(0x7f0bbc0aabe0 msgr2=0x7f0bbc11cc40 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:40:39.379 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:40:41.380 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:40:41.490 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.479+0000 7fdee8185700 1 Processor -- start 2019-12-05T06:40:41.491 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.479+0000 7fdee8185700 1 -- start start 2019-12-05T06:40:41.491 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.479+0000 7fdee8185700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdee00a7740 0x7fdee008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:40:41.491 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.479+0000 7fdee8185700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fdee009fae0 0x7fdee007eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:40:41.491 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.479+0000 7fdee8185700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdee00a0710 con 0x7fdee007e720 2019-12-05T06:40:41.491 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.479+0000 7fdee8185700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdee00a0b00 con 0x7fdee009fae0 2019-12-05T06:40:41.491 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.479+0000 7fdee8185700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdee008e5b0 con 0x7fdee00a7740 2019-12-05T06:40:41.492 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.479+0000 7fdee6f23700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fdee009fae0 0x7fdee007eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:40:41.492 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.479+0000 7fdee5f21700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fdee007e720 0x7fdee009f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:42288/0 (socket says 172.21.15.42:42288) 2019-12-05T06:40:41.492 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.479+0000 7fdee5f21700 1 -- 172.21.15.42:0/885063315 learned_addr learned my addr 172.21.15.42:0/885063315 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:40:41.492 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.479+0000 7fdee6722700 1 --2- 172.21.15.42:0/885063315 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdee00a7740 0x7fdee008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:40:41.492 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.479+0000 7fdee5720700 1 -- 172.21.15.42:0/885063315 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 985171242 0 0) 0x7fdee00a0710 con 0x7fdee007e720 2019-12-05T06:40:41.492 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.479+0000 7fdee5720700 1 -- 172.21.15.42:0/885063315 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdecc0030b0 con 0x7fdee007e720 2019-12-05T06:40:41.492 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.479+0000 7fdee6f23700 1 -- 172.21.15.42:0/885063315 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdee00a7740 msgr2=0x7fdee008df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:40:41.493 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.479+0000 7fdee6f23700 1 --2- 172.21.15.42:0/885063315 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdee00a7740 0x7fdee008df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:40:41.493 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.479+0000 7fdee6f23700 1 -- 172.21.15.42:0/885063315 >> v1:172.21.15.40:6789/0 conn(0x7fdee007e720 legacy=0x7fdee009f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:41.493 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.479+0000 7fdee6f23700 1 -- 172.21.15.42:0/885063315 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdee007f0a0 con 0x7fdee009fae0 2019-12-05T06:40:41.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.479+0000 7fdee6f23700 1 --2- 172.21.15.42:0/885063315 >> v2:172.21.15.42:3300/0 conn(0x7fdee009fae0 0x7fdee007eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fded8008b10 tx=0x7fded800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:41.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.479+0000 7fdee6f23700 1 --2- 172.21.15.42:0/885063315 >> v2:172.21.15.42:3300/0 conn(0x7fdee009fae0 0x7fdee007eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fded8008b10 tx=0x7fded800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:41.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.479+0000 7fdee6f23700 1 --2- 172.21.15.42:0/885063315 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdee009fae0 0x7fdee007eb50 secure :-1 s=READY pgs=3010 cs=0 l=1 rx=0x7fded8008b10 tx=0x7fded800fff0).ready entity=mon.1 client_cookie=fed057eac53a8271 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:40:41.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.479+0000 7fdee6f23700 1 --2- 172.21.15.42:0/885063315 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdee009fae0 0x7fdee007eb50 secure :-1 s=THROTTLE_DONE pgs=3010 cs=0 l=1 rx=0x7fded8008b10 tx=0x7fded800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:41.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.479+0000 7fdee5720700 1 -- 172.21.15.42:0/885063315 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fded8010900 con 0x7fdee009fae0 2019-12-05T06:40:41.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.479+0000 7fdee6f23700 1 --2- 172.21.15.42:0/885063315 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdee009fae0 0x7fdee007eb50 secure :-1 s=THROTTLE_DONE pgs=3010 cs=0 l=1 rx=0x7fded8008b10 tx=0x7fded800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:41.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.479+0000 7fdee5720700 1 -- 172.21.15.42:0/885063315 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fded8008b50 con 0x7fdee009fae0 2019-12-05T06:40:41.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.479+0000 7fdee6f23700 1 --2- 172.21.15.42:0/885063315 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdee009fae0 0x7fdee007eb50 secure :-1 s=THROTTLE_DONE pgs=3010 cs=0 l=1 rx=0x7fded8008b10 tx=0x7fded800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:41.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.479+0000 7fdee5720700 1 -- 172.21.15.42:0/885063315 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fded80110c0 con 0x7fdee009fae0 2019-12-05T06:40:41.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.479+0000 7fdee8185700 1 -- 172.21.15.42:0/885063315 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdee009fae0 msgr2=0x7fdee007eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:41.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.479+0000 7fdee8185700 1 --2- 172.21.15.42:0/885063315 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdee009fae0 0x7fdee007eb50 secure :-1 s=READY pgs=3010 cs=0 l=1 rx=0x7fded8008b10 tx=0x7fded800fff0).stop 2019-12-05T06:40:41.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.483+0000 7fdee8185700 1 -- 172.21.15.42:0/885063315 shutdown_connections 2019-12-05T06:40:41.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.483+0000 7fdee8185700 1 --2- 172.21.15.42:0/885063315 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdee009fae0 0x7fdee007eb50 unknown :-1 s=CLOSED pgs=3010 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:40:41.496 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.483+0000 7fdee8185700 1 --2- 172.21.15.42:0/885063315 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdee00a7740 0x7fdee008df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:40:41.496 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.483+0000 7fdee8185700 1 -- 172.21.15.42:0/885063315 shutdown_connections 2019-12-05T06:40:41.496 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.483+0000 7fdee8185700 1 -- 172.21.15.42:0/885063315 wait complete. 2019-12-05T06:40:41.496 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.483+0000 7fdee8185700 1 -- 172.21.15.42:0/885063315 >> 172.21.15.42:0/885063315 conn(0x7fdee00aabe0 msgr2=0x7fdee00a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:40:41.499 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.483+0000 7fdee8185700 1 Processor -- start 2019-12-05T06:40:41.499 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.483+0000 7fdee8185700 1 -- start start 2019-12-05T06:40:41.499 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.487+0000 7fdee8185700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fdee01194f0 0x7fdee0142bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:40:41.500 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.487+0000 7fdee8185700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdee01302c0 con 0x7fdee0118660 2019-12-05T06:40:41.500 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.487+0000 7fdee8185700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdee011b1e0 con 0x7fdee011adb0 2019-12-05T06:40:41.500 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.487+0000 7fdee8185700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdee011b410 con 0x7fdee01194f0 2019-12-05T06:40:41.500 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.487+0000 7fdee6722700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fdee01194f0 0x7fdee0142bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:40:41.500 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.487+0000 7fdee6722700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fdee01194f0 0x7fdee0142bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.42:48862/0 (socket says 172.21.15.42:48862) 2019-12-05T06:40:41.500 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.487+0000 7fdee5f21700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fdee011adb0 0x7fdee007db50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:32854/0 (socket says 172.21.15.42:32854) 2019-12-05T06:40:41.501 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.487+0000 7fdee6722700 1 -- 172.21.15.42:0/2060509178 learned_addr learned my addr 172.21.15.42:0/2060509178 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:40:41.501 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.487+0000 7fdee6722700 1 -- 172.21.15.42:0/2060509178 >> v1:172.21.15.42:6789/0 conn(0x7fdee011adb0 legacy=0x7fdee007db50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:41.501 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.487+0000 7fdee6722700 1 -- 172.21.15.42:0/2060509178 >> v1:172.21.15.40:6789/0 conn(0x7fdee0118660 legacy=0x7fdee0137710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:41.502 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.487+0000 7fded2ffd700 1 -- 172.21.15.42:0/2060509178 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3345476445 0 0) 0x7fdee01302c0 con 0x7fdee0118660 2019-12-05T06:40:41.502 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.487+0000 7fdee6722700 1 -- 172.21.15.42:0/2060509178 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdee0118a90 con 0x7fdee01194f0 2019-12-05T06:40:41.502 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.487+0000 7fdee6722700 1 --2- 172.21.15.42:0/2060509178 >> v2:172.21.15.40:3300/0 conn(0x7fdee01194f0 0x7fdee0142bf0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fded4003070 tx=0x7fded4010a60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:41.502 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.487+0000 7fdee6722700 1 --2- 172.21.15.42:0/2060509178 >> v2:172.21.15.40:3300/0 conn(0x7fdee01194f0 0x7fdee0142bf0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fded4003070 tx=0x7fded4010a60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:41.502 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.487+0000 7fdee6722700 1 --2- 172.21.15.42:0/2060509178 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdee01194f0 0x7fdee0142bf0 secure :-1 s=READY pgs=2284 cs=0 l=1 rx=0x7fded4003070 tx=0x7fded4010a60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:40:41.502 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.487+0000 7fdee6722700 1 --2- 172.21.15.42:0/2060509178 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdee01194f0 0x7fdee0142bf0 secure :-1 s=THROTTLE_DONE pgs=2284 cs=0 l=1 rx=0x7fded4003070 tx=0x7fded4010a60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:41.502 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.487+0000 7fded2ffd700 1 -- 172.21.15.42:0/2060509178 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fded4005930 con 0x7fdee01194f0 2019-12-05T06:40:41.503 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.487+0000 7fdee6722700 1 --2- 172.21.15.42:0/2060509178 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdee01194f0 0x7fdee0142bf0 secure :-1 s=THROTTLE_DONE pgs=2284 cs=0 l=1 rx=0x7fded4003070 tx=0x7fded4010a60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:41.503 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.487+0000 7fdee6722700 1 --2- 172.21.15.42:0/2060509178 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdee01194f0 0x7fdee0142bf0 secure :-1 s=THROTTLE_DONE pgs=2284 cs=0 l=1 rx=0x7fded4003070 tx=0x7fded4010a60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:41.503 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.487+0000 7fded2ffd700 1 -- 172.21.15.42:0/2060509178 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fded4007160 con 0x7fdee01194f0 2019-12-05T06:40:41.503 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.487+0000 7fdee8185700 1 -- 172.21.15.42:0/2060509178 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdee0118d60 con 0x7fdee01194f0 2019-12-05T06:40:41.503 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.487+0000 7fded2ffd700 1 -- 172.21.15.42:0/2060509178 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fded40120e0 con 0x7fdee01194f0 2019-12-05T06:40:41.504 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.487+0000 7fdee8185700 1 -- 172.21.15.42:0/2060509178 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdee0138f20 con 0x7fdee01194f0 2019-12-05T06:40:41.504 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.487+0000 7fdee6722700 1 --2- 172.21.15.42:0/2060509178 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdee01194f0 0x7fdee0142bf0 secure :-1 s=THROTTLE_DONE pgs=2284 cs=0 l=1 rx=0x7fded4003070 tx=0x7fded4010a60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:41.504 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.491+0000 7fded2ffd700 1 -- 172.21.15.42:0/2060509178 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fded4012400 con 0x7fdee01194f0 2019-12-05T06:40:41.504 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.491+0000 7fdee6722700 1 --2- 172.21.15.42:0/2060509178 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdee01194f0 0x7fdee0142bf0 secure :-1 s=THROTTLE_DONE pgs=2284 cs=0 l=1 rx=0x7fded4003070 tx=0x7fded4010a60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:41.504 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.491+0000 7fded2ffd700 1 --2- 172.21.15.42:0/2060509178 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fdec803abf0 0x7fdec803d100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:40:41.504 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.491+0000 7fded2ffd700 1 -- 172.21.15.42:0/2060509178 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 11707+0+0 (secure 0 0 0) 0x7fded404ae70 con 0x7fdee01194f0 2019-12-05T06:40:41.505 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.491+0000 7fdee5f21700 1 --2- 172.21.15.42:0/2060509178 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fdec803abf0 0x7fdec803d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:40:41.505 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.491+0000 7fdee5f21700 1 --2- 172.21.15.42:0/2060509178 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fdec803abf0 0x7fdec803d100 crc :-1 s=READY pgs=1260 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:40:41.505 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.495+0000 7fdee8185700 1 -- 172.21.15.42:0/2060509178 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdee005d110 con 0x7fdee01194f0 2019-12-05T06:40:41.507 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.499+0000 7fdee6722700 1 --2- 172.21.15.42:0/2060509178 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdee01194f0 0x7fdee0142bf0 secure :-1 s=THROTTLE_DONE pgs=2284 cs=0 l=1 rx=0x7fded4003070 tx=0x7fded4010a60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:41.507 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.499+0000 7fded2ffd700 1 -- 172.21.15.42:0/2060509178 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fded4031ed0 con 0x7fdee01194f0 2019-12-05T06:40:41.677 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.667+0000 7fdee8185700 1 -- 172.21.15.42:0/2060509178 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdee005b8c0 con 0x7fdee01194f0 2019-12-05T06:40:41.677 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.671+0000 7fdee6722700 1 --2- 172.21.15.42:0/2060509178 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdee01194f0 0x7fdee0142bf0 secure :-1 s=THROTTLE_DONE pgs=2284 cs=0 l=1 rx=0x7fded4003070 tx=0x7fded4010a60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:41.677 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.671+0000 7fded2ffd700 1 -- 172.21.15.42:0/2060509178 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 361 v361) v1 ==== 95+0+1775 (secure 0 0 0) 0x7fded4032130 con 0x7fdee01194f0 2019-12-05T06:40:41.679 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 361 2019-12-05T06:40:41.679 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:40:41.680 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":361,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":361,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:40:40.198989+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":115,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11976},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11976":{"gid":11976,"name":"b","rank":0,"incarnation":361,"state":"up:replay","state_seq":1,"addr":"172.21.15.40:6833/474728597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":474728597},{"type":"v1","addr":"172.21.15.40:6833","nonce":474728597}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:40:41.682 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.675+0000 7fdee8185700 1 -- 172.21.15.42:0/2060509178 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fdec803abf0 msgr2=0x7fdec803d100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:41.682 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.675+0000 7fdee8185700 1 --2- 172.21.15.42:0/2060509178 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fdec803abf0 0x7fdec803d100 crc :-1 s=READY pgs=1260 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:40:41.683 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.675+0000 7fdee8185700 1 -- 172.21.15.42:0/2060509178 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdee01194f0 msgr2=0x7fdee0142bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:41.683 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.675+0000 7fdee8185700 1 --2- 172.21.15.42:0/2060509178 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdee01194f0 0x7fdee0142bf0 secure :-1 s=READY pgs=2284 cs=0 l=1 rx=0x7fded4003070 tx=0x7fded4010a60).stop 2019-12-05T06:40:41.683 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.675+0000 7fdee8185700 1 -- 172.21.15.42:0/2060509178 shutdown_connections 2019-12-05T06:40:41.691 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.675+0000 7fdee8185700 1 --2- 172.21.15.42:0/2060509178 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fdec803abf0 0x7fdec803d100 unknown :-1 s=CLOSED pgs=1260 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:40:41.691 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.675+0000 7fdee8185700 1 --2- 172.21.15.42:0/2060509178 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdee01194f0 0x7fdee0142bf0 unknown :-1 s=CLOSED pgs=2284 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:40:41.691 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.675+0000 7fdee8185700 1 -- 172.21.15.42:0/2060509178 shutdown_connections 2019-12-05T06:40:41.691 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.675+0000 7fdee8185700 1 -- 172.21.15.42:0/2060509178 wait complete. 2019-12-05T06:40:41.691 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.675+0000 7fdee8185700 1 -- 172.21.15.42:0/2060509178 >> 172.21.15.42:0/2060509178 conn(0x7fdee00a7f90 msgr2=0x7fdee00a9790 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:40:41.693 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a down, removed from mdsmap 2019-12-05T06:40:41.693 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2019-12-05T06:40:41.693 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:40:41.791 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.779+0000 7f630b20e700 1 Processor -- start 2019-12-05T06:40:41.791 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.779+0000 7f630b20e700 1 -- start start 2019-12-05T06:40:41.791 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.779+0000 7f630b20e700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f630408df80 0x7f630409f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:40:41.791 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.779+0000 7f630b20e700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f630409fc60 con 0x7f630409f830 2019-12-05T06:40:41.791 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.779+0000 7f630b20e700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63040a0710 con 0x7f63040a7740 2019-12-05T06:40:41.792 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.779+0000 7f630b20e700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63040a0b00 con 0x7f630408df80 2019-12-05T06:40:41.792 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.779+0000 7f63097ab700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f63040a7740 0x7f630407e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:32860/0 (socket says 172.21.15.42:32860) 2019-12-05T06:40:41.792 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.779+0000 7f63097ab700 1 -- 172.21.15.42:0/639763723 learned_addr learned my addr 172.21.15.42:0/639763723 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:40:41.792 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.779+0000 7f6308faa700 1 --2- 172.21.15.42:0/639763723 >> v2:172.21.15.40:3300/0 conn(0x7f630408df80 0x7f630409f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:40:41.792 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.779+0000 7f62f7fff700 1 -- 172.21.15.42:0/639763723 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3838388921 0 0) 0x7f63040a0710 con 0x7f63040a7740 2019-12-05T06:40:41.792 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.779+0000 7f62f7fff700 1 -- 172.21.15.42:0/639763723 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62f00030b0 con 0x7f63040a7740 2019-12-05T06:40:41.793 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.779+0000 7f6308faa700 1 -- 172.21.15.42:0/639763723 >> v1:172.21.15.42:6789/0 conn(0x7f63040a7740 legacy=0x7f630407e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:41.793 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.779+0000 7f6308faa700 1 -- 172.21.15.42:0/639763723 >> v1:172.21.15.40:6789/0 conn(0x7f630409f830 legacy=0x7f630414efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:41.793 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.779+0000 7f62f7fff700 1 -- 172.21.15.42:0/639763723 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2722681920 0 0) 0x7f630409fc60 con 0x7f630409f830 2019-12-05T06:40:41.793 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.779+0000 7f6308faa700 1 -- 172.21.15.42:0/639763723 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f630407ef20 con 0x7f630408df80 2019-12-05T06:40:41.794 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.779+0000 7f6308faa700 1 --2- 172.21.15.42:0/639763723 >> v2:172.21.15.40:3300/0 conn(0x7f630408df80 0x7f630409f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f62f8008b10 tx=0x7f62f800ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:41.794 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.779+0000 7f6308faa700 1 --2- 172.21.15.42:0/639763723 >> v2:172.21.15.40:3300/0 conn(0x7f630408df80 0x7f630409f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f62f8008b10 tx=0x7f62f800ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:41.794 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.779+0000 7f6308faa700 1 --2- 172.21.15.42:0/639763723 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f630408df80 0x7f630409f2e0 secure :-1 s=READY pgs=2286 cs=0 l=1 rx=0x7f62f8008b10 tx=0x7f62f800ff70).ready entity=mon.0 client_cookie=2bffe4ee2a1ea73a server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:40:41.794 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.779+0000 7f6308faa700 1 --2- 172.21.15.42:0/639763723 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f630408df80 0x7f630409f2e0 secure :-1 s=THROTTLE_DONE pgs=2286 cs=0 l=1 rx=0x7f62f8008b10 tx=0x7f62f800ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:41.794 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.779+0000 7f62f7fff700 1 -- 172.21.15.42:0/639763723 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f62f80119d0 con 0x7f630408df80 2019-12-05T06:40:41.794 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.779+0000 7f6308faa700 1 --2- 172.21.15.42:0/639763723 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f630408df80 0x7f630409f2e0 secure :-1 s=THROTTLE_DONE pgs=2286 cs=0 l=1 rx=0x7f62f8008b10 tx=0x7f62f800ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:41.795 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.779+0000 7f6308faa700 1 --2- 172.21.15.42:0/639763723 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f630408df80 0x7f630409f2e0 secure :-1 s=THROTTLE_DONE pgs=2286 cs=0 l=1 rx=0x7f62f8008b10 tx=0x7f62f800ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:41.795 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.779+0000 7f62f7fff700 1 -- 172.21.15.42:0/639763723 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62f80115f0 con 0x7f630408df80 2019-12-05T06:40:41.795 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.779+0000 7f62f7fff700 1 -- 172.21.15.42:0/639763723 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f62f800a160 con 0x7f630408df80 2019-12-05T06:40:41.795 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.779+0000 7f630b20e700 1 -- 172.21.15.42:0/639763723 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f630408df80 msgr2=0x7f630409f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:41.796 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.779+0000 7f630b20e700 1 --2- 172.21.15.42:0/639763723 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f630408df80 0x7f630409f2e0 secure :-1 s=READY pgs=2286 cs=0 l=1 rx=0x7f62f8008b10 tx=0x7f62f800ff70).stop 2019-12-05T06:40:41.796 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.783+0000 7f630b20e700 1 -- 172.21.15.42:0/639763723 shutdown_connections 2019-12-05T06:40:41.796 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.783+0000 7f630b20e700 1 --2- 172.21.15.42:0/639763723 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f630408df80 0x7f630409f2e0 unknown :-1 s=CLOSED pgs=2286 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:40:41.796 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.783+0000 7f630b20e700 1 -- 172.21.15.42:0/639763723 shutdown_connections 2019-12-05T06:40:41.796 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.783+0000 7f630b20e700 1 -- 172.21.15.42:0/639763723 wait complete. 2019-12-05T06:40:41.796 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.783+0000 7f630b20e700 1 -- 172.21.15.42:0/639763723 >> 172.21.15.42:0/639763723 conn(0x7f63040aabe0 msgr2=0x7f63040a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:40:41.797 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.783+0000 7f630b20e700 1 Processor -- start 2019-12-05T06:40:41.797 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.783+0000 7f630b20e700 1 -- start start 2019-12-05T06:40:41.801 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.783+0000 7f630b20e700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f63041431f0 0x7f6304138520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:40:41.801 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.783+0000 7f630b20e700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6304117c70 con 0x7f6304144740 2019-12-05T06:40:41.801 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.783+0000 7f630b20e700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6304143620 con 0x7f6304117840 2019-12-05T06:40:41.802 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.783+0000 7f630b20e700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6304144b70 con 0x7f63041431f0 2019-12-05T06:40:41.802 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.783+0000 7f6308faa700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f6304117840 0x7f630407db50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:32866/0 (socket says 172.21.15.42:32866) 2019-12-05T06:40:41.802 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.783+0000 7f6308faa700 1 -- 172.21.15.42:0/1529070977 learned_addr learned my addr 172.21.15.42:0/1529070977 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:40:41.802 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.783+0000 7f6309fac700 1 --2- 172.21.15.42:0/1529070977 >> v2:172.21.15.40:3300/0 conn(0x7f63041431f0 0x7f6304138520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:40:41.802 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.787+0000 7f62f5ffb700 1 -- 172.21.15.42:0/1529070977 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2041281980 0 0) 0x7f6304117c70 con 0x7f6304144740 2019-12-05T06:40:41.803 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.787+0000 7f62f5ffb700 1 -- 172.21.15.42:0/1529070977 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62e40030b0 con 0x7f6304144740 2019-12-05T06:40:41.803 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.787+0000 7f62f5ffb700 1 -- 172.21.15.42:0/1529070977 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 489716921 0 0) 0x7f6304143620 con 0x7f6304117840 2019-12-05T06:40:41.803 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.787+0000 7f62f5ffb700 1 -- 172.21.15.42:0/1529070977 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6304117c70 con 0x7f6304117840 2019-12-05T06:40:41.804 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.787+0000 7f6309fac700 1 -- 172.21.15.42:0/1529070977 >> v1:172.21.15.42:6789/0 conn(0x7f6304117840 legacy=0x7f630407db50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:41.804 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.787+0000 7f6309fac700 1 -- 172.21.15.42:0/1529070977 >> v1:172.21.15.40:6789/0 conn(0x7f6304144740 legacy=0x7f630412cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:41.804 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.787+0000 7f6309fac700 1 -- 172.21.15.42:0/1529070977 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6304143850 con 0x7f63041431f0 2019-12-05T06:40:41.805 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.787+0000 7f6309fac700 1 --2- 172.21.15.42:0/1529070977 >> v2:172.21.15.40:3300/0 conn(0x7f63041431f0 0x7f6304138520 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f62fc000e90 tx=0x7f62fc010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:41.806 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.787+0000 7f6309fac700 1 --2- 172.21.15.42:0/1529070977 >> v2:172.21.15.40:3300/0 conn(0x7f63041431f0 0x7f6304138520 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f62fc000e90 tx=0x7f62fc010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:41.806 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.787+0000 7f6309fac700 1 --2- 172.21.15.42:0/1529070977 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f63041431f0 0x7f6304138520 secure :-1 s=READY pgs=2288 cs=0 l=1 rx=0x7f62fc000e90 tx=0x7f62fc010750).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:40:41.806 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.787+0000 7f6309fac700 1 --2- 172.21.15.42:0/1529070977 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f63041431f0 0x7f6304138520 secure :-1 s=THROTTLE_DONE pgs=2288 cs=0 l=1 rx=0x7f62fc000e90 tx=0x7f62fc010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:41.806 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.787+0000 7f6309fac700 1 --2- 172.21.15.42:0/1529070977 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f63041431f0 0x7f6304138520 secure :-1 s=THROTTLE_DONE pgs=2288 cs=0 l=1 rx=0x7f62fc000e90 tx=0x7f62fc010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:41.806 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.787+0000 7f62f5ffb700 1 -- 172.21.15.42:0/1529070977 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f62fc006940 con 0x7f63041431f0 2019-12-05T06:40:41.807 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.787+0000 7f6309fac700 1 --2- 172.21.15.42:0/1529070977 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f63041431f0 0x7f6304138520 secure :-1 s=THROTTLE_DONE pgs=2288 cs=0 l=1 rx=0x7f62fc000e90 tx=0x7f62fc010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:41.807 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.787+0000 7f62f5ffb700 1 -- 172.21.15.42:0/1529070977 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62fc011e10 con 0x7f63041431f0 2019-12-05T06:40:41.807 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.787+0000 7f630b20e700 1 -- 172.21.15.42:0/1529070977 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6304138a70 con 0x7f63041431f0 2019-12-05T06:40:41.807 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.787+0000 7f62f5ffb700 1 -- 172.21.15.42:0/1529070977 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f62fc010da0 con 0x7f63041431f0 2019-12-05T06:40:41.807 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.787+0000 7f630b20e700 1 -- 172.21.15.42:0/1529070977 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6304138ff0 con 0x7f63041431f0 2019-12-05T06:40:41.807 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.787+0000 7f6309fac700 1 --2- 172.21.15.42:0/1529070977 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f63041431f0 0x7f6304138520 secure :-1 s=THROTTLE_DONE pgs=2288 cs=0 l=1 rx=0x7f62fc000e90 tx=0x7f62fc010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:41.807 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.787+0000 7f62f5ffb700 1 -- 172.21.15.42:0/1529070977 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f62fc0110b0 con 0x7f63041431f0 2019-12-05T06:40:41.808 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.787+0000 7f6309fac700 1 --2- 172.21.15.42:0/1529070977 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f63041431f0 0x7f6304138520 secure :-1 s=THROTTLE_DONE pgs=2288 cs=0 l=1 rx=0x7f62fc000e90 tx=0x7f62fc010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:41.808 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.787+0000 7f62f5ffb700 1 --2- 172.21.15.42:0/1529070977 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f62e403b4a0 0x7f62e403d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:40:41.808 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.787+0000 7f62f5ffb700 1 -- 172.21.15.42:0/1529070977 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 11707+0+0 (secure 0 0 0) 0x7f62fc04add0 con 0x7f63041431f0 2019-12-05T06:40:41.808 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.791+0000 7f63097ab700 1 --2- 172.21.15.42:0/1529070977 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f62e403b4a0 0x7f62e403d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:40:41.808 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.791+0000 7f63097ab700 1 --2- 172.21.15.42:0/1529070977 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f62e403b4a0 0x7f62e403d9b0 crc :-1 s=READY pgs=1261 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:40:41.809 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.795+0000 7f630b20e700 1 -- 172.21.15.42:0/1529070977 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f630405d110 con 0x7f63041431f0 2019-12-05T06:40:41.810 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.803+0000 7f6309fac700 1 --2- 172.21.15.42:0/1529070977 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f63041431f0 0x7f6304138520 secure :-1 s=THROTTLE_DONE pgs=2288 cs=0 l=1 rx=0x7f62fc000e90 tx=0x7f62fc010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:41.810 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.803+0000 7f62f5ffb700 1 -- 172.21.15.42:0/1529070977 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f62fc031e50 con 0x7f63041431f0 2019-12-05T06:40:41.980 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.971+0000 7f630b20e700 1 -- 172.21.15.42:0/1529070977 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f630405b8c0 con 0x7f63041431f0 2019-12-05T06:40:41.980 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.975+0000 7f6309fac700 1 --2- 172.21.15.42:0/1529070977 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f63041431f0 0x7f6304138520 secure :-1 s=THROTTLE_DONE pgs=2288 cs=0 l=1 rx=0x7f62fc000e90 tx=0x7f62fc010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:41.981 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.975+0000 7f62f5ffb700 1 -- 172.21.15.42:0/1529070977 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 361 v361) v1 ==== 95+0+1775 (secure 0 0 0) 0x7f62fc0320b0 con 0x7f63041431f0 2019-12-05T06:40:41.981 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 361 2019-12-05T06:40:41.981 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:40:41.981 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":361,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":361,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:40:40.198989+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":115,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11976},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11976":{"gid":11976,"name":"b","rank":0,"incarnation":361,"state":"up:replay","state_seq":1,"addr":"172.21.15.40:6833/474728597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":474728597},{"type":"v1","addr":"172.21.15.40:6833","nonce":474728597}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:40:41.984 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.975+0000 7f630b20e700 1 -- 172.21.15.42:0/1529070977 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f62e403b4a0 msgr2=0x7f62e403d9b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:41.984 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.975+0000 7f630b20e700 1 --2- 172.21.15.42:0/1529070977 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f62e403b4a0 0x7f62e403d9b0 crc :-1 s=READY pgs=1261 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:40:41.984 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.975+0000 7f630b20e700 1 -- 172.21.15.42:0/1529070977 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f63041431f0 msgr2=0x7f6304138520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:41.984 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.975+0000 7f630b20e700 1 --2- 172.21.15.42:0/1529070977 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f63041431f0 0x7f6304138520 secure :-1 s=READY pgs=2288 cs=0 l=1 rx=0x7f62fc000e90 tx=0x7f62fc010750).stop 2019-12-05T06:40:41.984 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.975+0000 7f630b20e700 1 -- 172.21.15.42:0/1529070977 shutdown_connections 2019-12-05T06:40:41.985 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.975+0000 7f630b20e700 1 --2- 172.21.15.42:0/1529070977 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f62e403b4a0 0x7f62e403d9b0 unknown :-1 s=CLOSED pgs=1261 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:40:41.985 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.975+0000 7f630b20e700 1 --2- 172.21.15.42:0/1529070977 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f63041431f0 0x7f6304138520 unknown :-1 s=CLOSED pgs=2288 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:40:41.985 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.975+0000 7f630b20e700 1 -- 172.21.15.42:0/1529070977 shutdown_connections 2019-12-05T06:40:41.985 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.975+0000 7f630b20e700 1 -- 172.21.15.42:0/1529070977 wait complete. 2019-12-05T06:40:41.985 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:41.975+0000 7f630b20e700 1 -- 172.21.15.42:0/1529070977 >> 172.21.15.42:0/1529070977 conn(0x7f630408fde0 msgr2=0x7f630409f5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:40:41.999 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 361, "feature_flags": { "enable_multiple": false, "ever_enabled_multiple": false }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2019-12-05T05:03:35.604738+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 361, "ever_allowed_features": 2, "explicitly_allowed_features": 2, "failed": [], "flags": 18, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_11976": { "addr": "172.21.15.40:6833/474728597", "addrs": { "addrvec": [ { "addr": "172.21.15.40:6832", "nonce": 474728597, "type": "v2" }, { "addr": "172.21.15.40:6833", "nonce": 474728597, "type": "v1" } ] }, "export_targets": [], "features": 4540138292836696063, "flags": 0, "gid": 11976, "incarnation": 361, "name": "b", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 115, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 2, "min_compat_client": "0 (unknown)", "modified": "2019-12-05T06:40:40.198989+0000", "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 11976 } } } ], "standbys": [] } 2019-12-05T06:40:44.001 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:40:44.109 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:44.095+0000 7ff83fab5700 1 Processor -- start 2019-12-05T06:40:44.109 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:44.095+0000 7ff83fab5700 1 -- start start 2019-12-05T06:40:44.109 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:44.099+0000 7ff83fab5700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff8380a7740 0x7ff83808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:40:44.109 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:44.099+0000 7ff83fab5700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8380a0710 con 0x7ff83809fae0 2019-12-05T06:40:44.110 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:44.099+0000 7ff83fab5700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8380a9ba0 con 0x7ff83807e720 2019-12-05T06:40:44.110 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:44.099+0000 7ff83fab5700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8380a0b00 con 0x7ff8380a7740 2019-12-05T06:40:44.110 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:44.099+0000 7ff83d851700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7ff83807e720 0x7ff83809f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:32874/0 (socket says 172.21.15.42:32874) 2019-12-05T06:40:44.110 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:44.099+0000 7ff83d851700 1 -- 172.21.15.42:0/3063087444 learned_addr learned my addr 172.21.15.42:0/3063087444 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:40:44.110 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:44.099+0000 7ff83e052700 1 --2- 172.21.15.42:0/3063087444 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff8380a7740 0x7ff83808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:40:44.110 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:44.099+0000 7ff83d050700 1 -- 172.21.15.42:0/3063087444 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3777232853 0 0) 0x7ff8380a9ba0 con 0x7ff83807e720 2019-12-05T06:40:44.111 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:44.099+0000 7ff83d050700 1 -- 172.21.15.42:0/3063087444 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8240030b0 con 0x7ff83807e720 2019-12-05T06:40:44.111 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:44.099+0000 7ff83d050700 1 -- 172.21.15.42:0/3063087444 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 5027884 0 0) 0x7ff8380a0710 con 0x7ff83809fae0 2019-12-05T06:40:44.111 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:44.099+0000 7ff83d050700 1 -- 172.21.15.42:0/3063087444 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8380a9ba0 con 0x7ff83809fae0 2019-12-05T06:40:44.111 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:44.099+0000 7ff83e052700 1 -- 172.21.15.42:0/3063087444 >> v1:172.21.15.42:6789/0 conn(0x7ff83807e720 legacy=0x7ff83809f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:44.111 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:44.099+0000 7ff83e052700 1 -- 172.21.15.42:0/3063087444 >> v1:172.21.15.40:6789/0 conn(0x7ff83809fae0 legacy=0x7ff83814efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:44.111 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:44.099+0000 7ff83e052700 1 -- 172.21.15.42:0/3063087444 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff83807eb50 con 0x7ff8380a7740 2019-12-05T06:40:44.111 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:44.099+0000 7ff83e052700 1 --2- 172.21.15.42:0/3063087444 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff8380a7740 0x7ff83808df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7ff834002f30 tx=0x7ff83400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:44.112 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:44.099+0000 7ff83e052700 1 --2- 172.21.15.42:0/3063087444 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff8380a7740 0x7ff83808df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7ff834002f30 tx=0x7ff83400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:44.112 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:44.099+0000 7ff83e052700 1 --2- 172.21.15.42:0/3063087444 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff8380a7740 0x7ff83808df80 secure :-1 s=READY pgs=658 cs=0 l=1 rx=0x7ff834002f30 tx=0x7ff83400fbf0).ready entity=mon.2 client_cookie=ecc4b4851942d4a server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:40:44.112 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:44.099+0000 7ff83e052700 1 --2- 172.21.15.42:0/3063087444 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff8380a7740 0x7ff83808df80 secure :-1 s=THROTTLE_DONE pgs=658 cs=0 l=1 rx=0x7ff834002f30 tx=0x7ff83400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:44.112 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:44.099+0000 7ff83e052700 1 --2- 172.21.15.42:0/3063087444 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff8380a7740 0x7ff83808df80 secure :-1 s=THROTTLE_DONE pgs=658 cs=0 l=1 rx=0x7ff834002f30 tx=0x7ff83400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:44.112 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:44.099+0000 7ff83d050700 1 -- 172.21.15.42:0/3063087444 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff8340112b0 con 0x7ff8380a7740 2019-12-05T06:40:44.112 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:44.099+0000 7ff83e052700 1 --2- 172.21.15.42:0/3063087444 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff8380a7740 0x7ff83808df80 secure :-1 s=THROTTLE_DONE pgs=658 cs=0 l=1 rx=0x7ff834002f30 tx=0x7ff83400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:44.112 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:44.099+0000 7ff83d050700 1 -- 172.21.15.42:0/3063087444 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff834008bc0 con 0x7ff8380a7740 2019-12-05T06:40:44.113 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:44.099+0000 7ff83d050700 1 -- 172.21.15.42:0/3063087444 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff834011890 con 0x7ff8380a7740 2019-12-05T06:40:44.113 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:44.099+0000 7ff83fab5700 1 -- 172.21.15.42:0/3063087444 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff8380a7740 msgr2=0x7ff83808df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:44.113 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:44.099+0000 7ff83fab5700 1 --2- 172.21.15.42:0/3063087444 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff8380a7740 0x7ff83808df80 secure :-1 s=READY pgs=658 cs=0 l=1 rx=0x7ff834002f30 tx=0x7ff83400fbf0).stop 2019-12-05T06:40:44.114 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:44.099+0000 7ff83fab5700 1 -- 172.21.15.42:0/3063087444 shutdown_connections 2019-12-05T06:40:44.114 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:44.099+0000 7ff83fab5700 1 --2- 172.21.15.42:0/3063087444 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff8380a7740 0x7ff83808df80 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:40:44.114 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:44.099+0000 7ff83fab5700 1 -- 172.21.15.42:0/3063087444 shutdown_connections 2019-12-05T06:40:44.115 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:44.103+0000 7ff83fab5700 1 -- 172.21.15.42:0/3063087444 wait complete. 2019-12-05T06:40:44.115 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:44.103+0000 7ff83fab5700 1 -- 172.21.15.42:0/3063087444 >> 172.21.15.42:0/3063087444 conn(0x7ff8380aabe0 msgr2=0x7ff8380a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:40:44.119 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:44.103+0000 7ff83fab5700 1 Processor -- start 2019-12-05T06:40:44.119 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:44.103+0000 7ff83fab5700 1 -- start start 2019-12-05T06:40:44.119 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:44.103+0000 7ff83fab5700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff838142ff0 0x7ff838125860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:40:44.119 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:44.103+0000 7ff83fab5700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8380a58b0 con 0x7ff83807cfb0 2019-12-05T06:40:44.120 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:44.103+0000 7ff83fab5700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8380a6600 con 0x7ff8380a5480 2019-12-05T06:40:44.120 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:44.103+0000 7ff83fab5700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff838145280 con 0x7ff838142ff0 2019-12-05T06:40:44.120 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:44.103+0000 7ff83d851700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff838142ff0 0x7ff838125860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:40:44.120 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:44.103+0000 7ff83d851700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff838142ff0 0x7ff838125860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.42:48892/0 (socket says 172.21.15.42:48892) 2019-12-05T06:40:44.120 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:44.103+0000 7ff83e853700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7ff8380a5480 0x7ff83812ccb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:32884/0 (socket says 172.21.15.42:32884) 2019-12-05T06:40:44.120 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:44.103+0000 7ff83d851700 1 -- 172.21.15.42:0/1601901561 learned_addr learned my addr 172.21.15.42:0/1601901561 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:40:44.121 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:44.103+0000 7ff82affd700 1 -- 172.21.15.42:0/1601901561 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 121183319 0 0) 0x7ff8380a6600 con 0x7ff8380a5480 2019-12-05T06:40:44.121 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:44.103+0000 7ff82affd700 1 -- 172.21.15.42:0/1601901561 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8200030b0 con 0x7ff8380a5480 2019-12-05T06:40:44.121 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:44.103+0000 7ff82affd700 1 -- 172.21.15.42:0/1601901561 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1368143733 0 0) 0x7ff8380a58b0 con 0x7ff83807cfb0 2019-12-05T06:40:44.121 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:44.107+0000 7ff83d851700 1 -- 172.21.15.42:0/1601901561 >> v1:172.21.15.42:6789/0 conn(0x7ff8380a5480 legacy=0x7ff83812ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:44.121 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:44.107+0000 7ff83d851700 1 -- 172.21.15.42:0/1601901561 >> v1:172.21.15.40:6789/0 conn(0x7ff83807cfb0 legacy=0x7ff838144a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:44.121 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:44.107+0000 7ff83d851700 1 -- 172.21.15.42:0/1601901561 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8380a6830 con 0x7ff838142ff0 2019-12-05T06:40:44.121 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:44.107+0000 7ff83d851700 1 --2- 172.21.15.42:0/1601901561 >> v2:172.21.15.40:3300/0 conn(0x7ff838142ff0 0x7ff838125860 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7ff83807f1c0 tx=0x7ff82c00c780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:44.122 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:44.107+0000 7ff83d851700 1 --2- 172.21.15.42:0/1601901561 >> v2:172.21.15.40:3300/0 conn(0x7ff838142ff0 0x7ff838125860 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7ff83807f1c0 tx=0x7ff82c00c780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:44.122 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:44.107+0000 7ff83d851700 1 --2- 172.21.15.42:0/1601901561 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff838142ff0 0x7ff838125860 secure :-1 s=READY pgs=2291 cs=0 l=1 rx=0x7ff83807f1c0 tx=0x7ff82c00c780).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:40:44.122 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:44.107+0000 7ff83d851700 1 --2- 172.21.15.42:0/1601901561 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff838142ff0 0x7ff838125860 secure :-1 s=THROTTLE_DONE pgs=2291 cs=0 l=1 rx=0x7ff83807f1c0 tx=0x7ff82c00c780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:44.122 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:44.107+0000 7ff83d851700 1 --2- 172.21.15.42:0/1601901561 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff838142ff0 0x7ff838125860 secure :-1 s=THROTTLE_DONE pgs=2291 cs=0 l=1 rx=0x7ff83807f1c0 tx=0x7ff82c00c780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:44.122 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:44.107+0000 7ff82affd700 1 -- 172.21.15.42:0/1601901561 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff82c010440 con 0x7ff838142ff0 2019-12-05T06:40:44.122 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:44.107+0000 7ff83d851700 1 --2- 172.21.15.42:0/1601901561 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff838142ff0 0x7ff838125860 secure :-1 s=THROTTLE_DONE pgs=2291 cs=0 l=1 rx=0x7ff83807f1c0 tx=0x7ff82c00c780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:44.122 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:44.107+0000 7ff82affd700 1 -- 172.21.15.42:0/1601901561 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff82c00cc40 con 0x7ff838142ff0 2019-12-05T06:40:44.122 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:44.107+0000 7ff83fab5700 1 -- 172.21.15.42:0/1601901561 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8380a6b00 con 0x7ff838142ff0 2019-12-05T06:40:44.123 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:44.107+0000 7ff82affd700 1 -- 172.21.15.42:0/1601901561 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff82c012530 con 0x7ff838142ff0 2019-12-05T06:40:44.123 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:44.107+0000 7ff83fab5700 1 -- 172.21.15.42:0/1601901561 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff83808eb70 con 0x7ff838142ff0 2019-12-05T06:40:44.123 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:44.107+0000 7ff83d851700 1 --2- 172.21.15.42:0/1601901561 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff838142ff0 0x7ff838125860 secure :-1 s=THROTTLE_DONE pgs=2291 cs=0 l=1 rx=0x7ff83807f1c0 tx=0x7ff82c00c780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:44.123 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:44.107+0000 7ff82affd700 1 -- 172.21.15.42:0/1601901561 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7ff82c012800 con 0x7ff838142ff0 2019-12-05T06:40:44.123 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:44.107+0000 7ff83d851700 1 --2- 172.21.15.42:0/1601901561 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff838142ff0 0x7ff838125860 secure :-1 s=THROTTLE_DONE pgs=2291 cs=0 l=1 rx=0x7ff83807f1c0 tx=0x7ff82c00c780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:44.123 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:44.107+0000 7ff82affd700 1 --2- 172.21.15.42:0/1601901561 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff82003b1a0 0x7ff82003d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:40:44.124 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:44.107+0000 7ff82affd700 1 -- 172.21.15.42:0/1601901561 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 11707+0+0 (secure 0 0 0) 0x7ff82c04b0c0 con 0x7ff838142ff0 2019-12-05T06:40:44.125 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:44.107+0000 7ff83e052700 1 --2- 172.21.15.42:0/1601901561 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff82003b1a0 0x7ff82003d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:40:44.126 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:44.111+0000 7ff83e052700 1 --2- 172.21.15.42:0/1601901561 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff82003b1a0 0x7ff82003d6b0 crc :-1 s=READY pgs=1262 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:40:44.126 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:44.115+0000 7ff83fab5700 1 -- 172.21.15.42:0/1601901561 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff83805d110 con 0x7ff838142ff0 2019-12-05T06:40:44.130 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:44.123+0000 7ff83d851700 1 --2- 172.21.15.42:0/1601901561 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff838142ff0 0x7ff838125860 secure :-1 s=THROTTLE_DONE pgs=2291 cs=0 l=1 rx=0x7ff83807f1c0 tx=0x7ff82c00c780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:44.130 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:44.123+0000 7ff82affd700 1 -- 172.21.15.42:0/1601901561 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7ff82c032140 con 0x7ff838142ff0 2019-12-05T06:40:44.294 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:44.287+0000 7ff83fab5700 1 -- 172.21.15.42:0/1601901561 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff83805b8c0 con 0x7ff838142ff0 2019-12-05T06:40:44.294 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:44.287+0000 7ff83d851700 1 --2- 172.21.15.42:0/1601901561 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff838142ff0 0x7ff838125860 secure :-1 s=THROTTLE_DONE pgs=2291 cs=0 l=1 rx=0x7ff83807f1c0 tx=0x7ff82c00c780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:44.295 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:44.287+0000 7ff82affd700 1 -- 172.21.15.42:0/1601901561 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 361 v361) v1 ==== 95+0+1775 (secure 0 0 0) 0x7ff82c0323a0 con 0x7ff838142ff0 2019-12-05T06:40:44.297 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:40:44.297 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":361,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":361,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:40:40.198989+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":115,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11976},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11976":{"gid":11976,"name":"b","rank":0,"incarnation":361,"state":"up:replay","state_seq":1,"addr":"172.21.15.40:6833/474728597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":474728597},{"type":"v1","addr":"172.21.15.40:6833","nonce":474728597}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:40:44.297 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 361 2019-12-05T06:40:44.300 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:44.291+0000 7ff83fab5700 1 -- 172.21.15.42:0/1601901561 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff82003b1a0 msgr2=0x7ff82003d6b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:44.300 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:44.291+0000 7ff83fab5700 1 --2- 172.21.15.42:0/1601901561 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff82003b1a0 0x7ff82003d6b0 crc :-1 s=READY pgs=1262 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:40:44.300 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:44.291+0000 7ff83fab5700 1 -- 172.21.15.42:0/1601901561 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff838142ff0 msgr2=0x7ff838125860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:44.300 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:44.291+0000 7ff83fab5700 1 --2- 172.21.15.42:0/1601901561 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff838142ff0 0x7ff838125860 secure :-1 s=READY pgs=2291 cs=0 l=1 rx=0x7ff83807f1c0 tx=0x7ff82c00c780).stop 2019-12-05T06:40:44.300 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:44.291+0000 7ff83fab5700 1 -- 172.21.15.42:0/1601901561 shutdown_connections 2019-12-05T06:40:44.301 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:44.291+0000 7ff83fab5700 1 --2- 172.21.15.42:0/1601901561 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff82003b1a0 0x7ff82003d6b0 unknown :-1 s=CLOSED pgs=1262 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:40:44.301 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:44.291+0000 7ff83fab5700 1 --2- 172.21.15.42:0/1601901561 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff838142ff0 0x7ff838125860 unknown :-1 s=CLOSED pgs=2291 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:40:44.301 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:44.291+0000 7ff83fab5700 1 -- 172.21.15.42:0/1601901561 shutdown_connections 2019-12-05T06:40:44.301 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:44.291+0000 7ff83fab5700 1 -- 172.21.15.42:0/1601901561 wait complete. 2019-12-05T06:40:44.301 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:44.291+0000 7ff83fab5700 1 -- 172.21.15.42:0/1601901561 >> 172.21.15.42:0/1601901561 conn(0x7ff8380a7740 msgr2=0x7ff838059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:40:44.310 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2019-12-05T06:40:45.025 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:40:45.028 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:40:46.311 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:40:46.418 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:46.407+0000 7f18cc1f1700 1 Processor -- start 2019-12-05T06:40:46.419 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:46.407+0000 7f18cc1f1700 1 -- start start 2019-12-05T06:40:46.419 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:46.407+0000 7f18cc1f1700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f18c40a7740 0x7f18c408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:40:46.419 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:46.407+0000 7f18cc1f1700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f18c407e720 0x7f18c407eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:40:46.420 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:46.407+0000 7f18cc1f1700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f18c409f2e0 0x7f18c409f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:40:46.420 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:46.407+0000 7f18cc1f1700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18c40a0b00 con 0x7f18c407e720 2019-12-05T06:40:46.420 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:46.407+0000 7f18cc1f1700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18c408e5b0 con 0x7f18c40a7740 2019-12-05T06:40:46.420 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:46.407+0000 7f18cc1f1700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18c408e850 con 0x7f18c409f2e0 2019-12-05T06:40:46.420 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:46.407+0000 7f18ca78e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f18c40a7740 0x7f18c408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:40:46.420 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:46.407+0000 7f18ca78e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f18c40a7740 0x7f18c408df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:32856/0 (socket says 172.21.15.42:32856) 2019-12-05T06:40:46.421 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:46.407+0000 7f18ca78e700 1 -- 172.21.15.42:0/429286700 learned_addr learned my addr 172.21.15.42:0/429286700 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:40:46.421 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:46.407+0000 7f18c9f8d700 1 --2- 172.21.15.42:0/429286700 >> v2:172.21.15.40:3300/0 conn(0x7f18c407e720 0x7f18c407eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:40:46.421 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:46.407+0000 7f18caf8f700 1 --2- 172.21.15.42:0/429286700 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f18c409f2e0 0x7f18c409f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:40:46.421 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:46.407+0000 7f18ca78e700 1 -- 172.21.15.42:0/429286700 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f18c409f2e0 msgr2=0x7f18c409f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:40:46.421 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:46.407+0000 7f18ca78e700 1 --2- 172.21.15.42:0/429286700 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f18c409f2e0 0x7f18c409f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:40:46.421 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:46.407+0000 7f18ca78e700 1 -- 172.21.15.42:0/429286700 >> v2:172.21.15.40:3300/0 conn(0x7f18c407e720 msgr2=0x7f18c407eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:40:46.421 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:46.407+0000 7f18ca78e700 1 --2- 172.21.15.42:0/429286700 >> v2:172.21.15.40:3300/0 conn(0x7f18c407e720 0x7f18c407eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:40:46.422 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:46.407+0000 7f18ca78e700 1 -- 172.21.15.42:0/429286700 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18c407f0a0 con 0x7f18c40a7740 2019-12-05T06:40:46.422 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:46.407+0000 7f18c9f8d700 1 --2- 172.21.15.42:0/429286700 >> v2:172.21.15.40:3300/0 conn(0x7f18c407e720 0x7f18c407eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:40:46.422 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:46.407+0000 7f18caf8f700 1 --2- 172.21.15.42:0/429286700 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f18c409f2e0 0x7f18c409f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:40:46.422 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:46.407+0000 7f18ca78e700 1 --2- 172.21.15.42:0/429286700 >> v2:172.21.15.42:3300/0 conn(0x7f18c40a7740 0x7f18c408df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f18c40a0f50 tx=0x7f18b400fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:46.422 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:46.407+0000 7f18ca78e700 1 --2- 172.21.15.42:0/429286700 >> v2:172.21.15.42:3300/0 conn(0x7f18c40a7740 0x7f18c408df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f18c40a0f50 tx=0x7f18b400fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:46.422 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:46.407+0000 7f18ca78e700 1 --2- 172.21.15.42:0/429286700 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f18c40a7740 0x7f18c408df80 secure :-1 s=READY pgs=3016 cs=0 l=1 rx=0x7f18c40a0f50 tx=0x7f18b400fdb0).ready entity=mon.1 client_cookie=a6cda7a87440687f server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:40:46.423 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:46.407+0000 7f18ca78e700 1 --2- 172.21.15.42:0/429286700 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f18c40a7740 0x7f18c408df80 secure :-1 s=THROTTLE_DONE pgs=3016 cs=0 l=1 rx=0x7f18c40a0f50 tx=0x7f18b400fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:46.423 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:46.407+0000 7f18c978c700 1 -- 172.21.15.42:0/429286700 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f18b4008c90 con 0x7f18c40a7740 2019-12-05T06:40:46.423 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:46.407+0000 7f18ca78e700 1 --2- 172.21.15.42:0/429286700 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f18c40a7740 0x7f18c408df80 secure :-1 s=THROTTLE_DONE pgs=3016 cs=0 l=1 rx=0x7f18c40a0f50 tx=0x7f18b400fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:46.423 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:46.407+0000 7f18ca78e700 1 --2- 172.21.15.42:0/429286700 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f18c40a7740 0x7f18c408df80 secure :-1 s=THROTTLE_DONE pgs=3016 cs=0 l=1 rx=0x7f18c40a0f50 tx=0x7f18b400fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:46.423 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:46.407+0000 7f18c978c700 1 -- 172.21.15.42:0/429286700 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18b40088b0 con 0x7f18c40a7740 2019-12-05T06:40:46.423 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:46.407+0000 7f18c978c700 1 -- 172.21.15.42:0/429286700 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f18b4011b50 con 0x7f18c40a7740 2019-12-05T06:40:46.424 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:46.407+0000 7f18cc1f1700 1 -- 172.21.15.42:0/429286700 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f18c40a7740 msgr2=0x7f18c408df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:46.424 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:46.407+0000 7f18cc1f1700 1 --2- 172.21.15.42:0/429286700 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f18c40a7740 0x7f18c408df80 secure :-1 s=READY pgs=3016 cs=0 l=1 rx=0x7f18c40a0f50 tx=0x7f18b400fdb0).stop 2019-12-05T06:40:46.424 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:46.411+0000 7f18cc1f1700 1 -- 172.21.15.42:0/429286700 shutdown_connections 2019-12-05T06:40:46.424 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:46.411+0000 7f18cc1f1700 1 --2- 172.21.15.42:0/429286700 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f18c409f2e0 0x7f18c409f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:40:46.424 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:46.411+0000 7f18cc1f1700 1 --2- 172.21.15.42:0/429286700 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f18c40a7740 0x7f18c408df80 unknown :-1 s=CLOSED pgs=3016 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:40:46.424 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:46.411+0000 7f18cc1f1700 1 --2- 172.21.15.42:0/429286700 >> v2:172.21.15.40:3300/0 conn(0x7f18c407e720 0x7f18c407eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:40:46.424 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:46.411+0000 7f18cc1f1700 1 -- 172.21.15.42:0/429286700 shutdown_connections 2019-12-05T06:40:46.425 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:46.411+0000 7f18cc1f1700 1 -- 172.21.15.42:0/429286700 wait complete. 2019-12-05T06:40:46.425 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:46.411+0000 7f18cc1f1700 1 -- 172.21.15.42:0/429286700 >> 172.21.15.42:0/429286700 conn(0x7f18c40aabe0 msgr2=0x7f18c40a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:40:46.432 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:46.411+0000 7f18cc1f1700 1 Processor -- start 2019-12-05T06:40:46.432 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:46.411+0000 7f18cc1f1700 1 -- start start 2019-12-05T06:40:46.433 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:46.415+0000 7f18cc1f1700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f18c40a9790 0x7f18c41387b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:40:46.434 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:46.415+0000 7f18cc1f1700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f18c4138d00 0x7f18c413b140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:40:46.434 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:46.415+0000 7f18cc1f1700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18c4116620 con 0x7f18c4130fc0 2019-12-05T06:40:46.434 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:46.415+0000 7f18cc1f1700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18c41307e0 con 0x7f18c40a9790 2019-12-05T06:40:46.434 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:46.415+0000 7f18cc1f1700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18c4130a80 con 0x7f18c4138d00 2019-12-05T06:40:46.434 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:46.415+0000 7f18caf8f700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f18c4138d00 0x7f18c413b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:40:46.435 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:46.415+0000 7f18caf8f700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f18c4138d00 0x7f18c413b140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:32866/0 (socket says 172.21.15.42:32866) 2019-12-05T06:40:46.435 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:46.415+0000 7f18caf8f700 1 -- 172.21.15.42:0/3198711175 learned_addr learned my addr 172.21.15.42:0/3198711175 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:40:46.435 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:46.415+0000 7f18c9f8d700 1 --2- 172.21.15.42:0/3198711175 >> v2:172.21.15.40:3300/0 conn(0x7f18c40a9790 0x7f18c41387b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:40:46.435 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:46.415+0000 7f18caf8f700 1 -- 172.21.15.42:0/3198711175 >> v2:172.21.15.40:3300/0 conn(0x7f18c40a9790 msgr2=0x7f18c41387b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:46.435 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:46.415+0000 7f18caf8f700 1 --2- 172.21.15.42:0/3198711175 >> v2:172.21.15.40:3300/0 conn(0x7f18c40a9790 0x7f18c41387b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:40:46.436 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:46.415+0000 7f18caf8f700 1 -- 172.21.15.42:0/3198711175 >> v1:172.21.15.40:6789/0 conn(0x7f18c4130fc0 legacy=0x7f18c4116a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:46.436 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:46.415+0000 7f18caf8f700 1 -- 172.21.15.42:0/3198711175 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18c4121320 con 0x7f18c4138d00 2019-12-05T06:40:46.436 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:46.415+0000 7f18caf8f700 1 --2- 172.21.15.42:0/3198711175 >> v2:172.21.15.42:3300/0 conn(0x7f18c4138d00 0x7f18c413b140 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f18bc002f30 tx=0x7f18bc010800).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:46.437 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:46.415+0000 7f18caf8f700 1 --2- 172.21.15.42:0/3198711175 >> v2:172.21.15.42:3300/0 conn(0x7f18c4138d00 0x7f18c413b140 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f18bc002f30 tx=0x7f18bc010800).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:46.437 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:46.415+0000 7f18caf8f700 1 --2- 172.21.15.42:0/3198711175 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f18c4138d00 0x7f18c413b140 secure :-1 s=READY pgs=3017 cs=0 l=1 rx=0x7f18bc002f30 tx=0x7f18bc010800).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:40:46.437 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:46.415+0000 7f18caf8f700 1 --2- 172.21.15.42:0/3198711175 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f18c4138d00 0x7f18c413b140 secure :-1 s=THROTTLE_DONE pgs=3017 cs=0 l=1 rx=0x7f18bc002f30 tx=0x7f18bc010800).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:46.437 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:46.415+0000 7f18baffd700 1 -- 172.21.15.42:0/3198711175 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f18bc005b20 con 0x7f18c4138d00 2019-12-05T06:40:46.437 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:46.415+0000 7f18caf8f700 1 --2- 172.21.15.42:0/3198711175 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f18c4138d00 0x7f18c413b140 secure :-1 s=THROTTLE_DONE pgs=3017 cs=0 l=1 rx=0x7f18bc002f30 tx=0x7f18bc010800).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:46.438 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:46.415+0000 7f18caf8f700 1 --2- 172.21.15.42:0/3198711175 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f18c4138d00 0x7f18c413b140 secure :-1 s=THROTTLE_DONE pgs=3017 cs=0 l=1 rx=0x7f18bc002f30 tx=0x7f18bc010800).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:46.438 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:46.415+0000 7f18baffd700 1 -- 172.21.15.42:0/3198711175 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18bc007200 con 0x7f18c4138d00 2019-12-05T06:40:46.438 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:46.415+0000 7f18cc1f1700 1 -- 172.21.15.42:0/3198711175 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18c40a9bc0 con 0x7f18c4138d00 2019-12-05T06:40:46.439 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:46.415+0000 7f18baffd700 1 -- 172.21.15.42:0/3198711175 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f18bc0062b0 con 0x7f18c4138d00 2019-12-05T06:40:46.439 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:46.415+0000 7f18cc1f1700 1 -- 172.21.15.42:0/3198711175 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18c4115e80 con 0x7f18c4138d00 2019-12-05T06:40:46.439 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:46.415+0000 7f18caf8f700 1 -- 172.21.15.42:0/3198711175 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f18c4138d00 msgr2=0x7f18c413b140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2019-12-05T06:40:46.439 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:46.415+0000 7f18caf8f700 1 -- 172.21.15.42:0/3198711175 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f18c4138d00 msgr2=0x7f18c413b140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2019-12-05T06:40:46.440 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:46.415+0000 7f18caf8f700 1 --2- 172.21.15.42:0/3198711175 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f18c4138d00 0x7f18c413b140 secure :-1 s=READY pgs=3017 cs=0 l=1 rx=0x7f18bc002f30 tx=0x7f18bc010800).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-12-05T06:40:46.440 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:46.415+0000 7f18caf8f700 1 --2- 172.21.15.42:0/3198711175 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f18c4138d00 0x7f18c413b140 secure :-1 s=READY pgs=3017 cs=0 l=1 rx=0x7f18bc002f30 tx=0x7f18bc010800).stop 2019-12-05T06:40:46.440 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:46.415+0000 7f18baffd700 1 -- 172.21.15.42:0/3198711175 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f18c4138d00 msgr2=0x7f18c413b140 unknown :-1 s=STATE_CLOSED l=1).mark_down 2019-12-05T06:40:46.440 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:46.415+0000 7f18baffd700 1 --2- 172.21.15.42:0/3198711175 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f18c4138d00 0x7f18c413b140 unknown :-1 s=CLOSED pgs=3017 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:40:46.441 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:46.415+0000 7f18baffd700 1 --2- 172.21.15.42:0/3198711175 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f18a4002ff0 0x7f18a4005490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:40:46.441 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:46.415+0000 7f18baffd700 1 --2- 172.21.15.42:0/3198711175 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f18a4005b20 0x7f18a4008000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:40:46.441 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:46.415+0000 7f18baffd700 1 --2- 172.21.15.42:0/3198711175 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f18a4008780 0x7f18a400ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:40:46.441 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:46.415+0000 7f18baffd700 1 -- 172.21.15.42:0/3198711175 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18a400b420 con 0x7f18a4002ff0 2019-12-05T06:40:46.441 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:46.415+0000 7f18baffd700 1 -- 172.21.15.42:0/3198711175 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18a400b610 con 0x7f18a4008780 2019-12-05T06:40:46.441 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:46.415+0000 7f18baffd700 1 -- 172.21.15.42:0/3198711175 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18a400b850 con 0x7f18a4005b20 2019-12-05T06:40:46.442 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:46.419+0000 7f18ca78e700 1 --2- 172.21.15.42:0/3198711175 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f18a4002ff0 0x7f18a4005490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:40:46.442 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:46.419+0000 7f18caf8f700 1 --2- 172.21.15.42:0/3198711175 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f18a4008780 0x7f18a400ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:40:46.442 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:46.419+0000 7f18c9f8d700 1 --2- 172.21.15.42:0/3198711175 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f18a4005b20 0x7f18a4008000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:40:46.442 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:46.419+0000 7f18ca78e700 1 -- 172.21.15.42:0/3198711175 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f18a4005b20 msgr2=0x7f18a4008000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:46.442 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:46.419+0000 7f18ca78e700 1 --2- 172.21.15.42:0/3198711175 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f18a4005b20 0x7f18a4008000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:40:46.442 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:46.419+0000 7f18ca78e700 1 -- 172.21.15.42:0/3198711175 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f18a4008780 msgr2=0x7f18a400ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:46.443 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:46.419+0000 7f18ca78e700 1 --2- 172.21.15.42:0/3198711175 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f18a4008780 0x7f18a400ac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:40:46.443 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:46.419+0000 7f18caf8f700 1 -- 172.21.15.42:0/3198711175 reap_dead start 2019-12-05T06:40:46.443 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:46.419+0000 7f18ca78e700 1 -- 172.21.15.42:0/3198711175 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+,osdmap=0}) v3 -- 0x7f18a400bc10 con 0x7f18a4002ff0 2019-12-05T06:40:46.443 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:46.419+0000 7f18ca78e700 1 --2- 172.21.15.42:0/3198711175 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f18a4002ff0 0x7f18a4005490 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f18b40101c0 tx=0x7f18b4008be0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:46.444 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:46.419+0000 7f18ca78e700 1 --2- 172.21.15.42:0/3198711175 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f18a4002ff0 0x7f18a4005490 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f18b40101c0 tx=0x7f18b4008be0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:46.444 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:46.419+0000 7f18ca78e700 1 --2- 172.21.15.42:0/3198711175 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f18a4002ff0 0x7f18a4005490 secure :-1 s=READY pgs=2293 cs=0 l=1 rx=0x7f18b40101c0 tx=0x7f18b4008be0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:40:46.444 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:46.419+0000 7f18ca78e700 1 --2- 172.21.15.42:0/3198711175 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f18a4002ff0 0x7f18a4005490 secure :-1 s=THROTTLE_DONE pgs=2293 cs=0 l=1 rx=0x7f18b40101c0 tx=0x7f18b4008be0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:46.444 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:46.419+0000 7f18baffd700 1 -- 172.21.15.42:0/3198711175 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f18b400f680 con 0x7f18a4002ff0 2019-12-05T06:40:46.444 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:46.419+0000 7f18ca78e700 1 --2- 172.21.15.42:0/3198711175 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f18a4002ff0 0x7f18a4005490 secure :-1 s=THROTTLE_DONE pgs=2293 cs=0 l=1 rx=0x7f18b40101c0 tx=0x7f18b4008be0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:46.444 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:46.419+0000 7f18baffd700 1 -- 172.21.15.42:0/3198711175 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18b4011770 con 0x7f18a4002ff0 2019-12-05T06:40:46.444 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:46.419+0000 7f18ca78e700 1 --2- 172.21.15.42:0/3198711175 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f18a4002ff0 0x7f18a4005490 secure :-1 s=THROTTLE_DONE pgs=2293 cs=0 l=1 rx=0x7f18b40101c0 tx=0x7f18b4008be0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:46.445 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:46.419+0000 7f18baffd700 1 -- 172.21.15.42:0/3198711175 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f18b4005540 con 0x7f18a4002ff0 2019-12-05T06:40:46.445 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:46.419+0000 7f18ca78e700 1 --2- 172.21.15.42:0/3198711175 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f18a4002ff0 0x7f18a4005490 secure :-1 s=THROTTLE_DONE pgs=2293 cs=0 l=1 rx=0x7f18b40101c0 tx=0x7f18b4008be0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:46.446 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:46.423+0000 7f18baffd700 1 --2- 172.21.15.42:0/3198711175 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f18a403e820 0x7f18a4040d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:40:46.446 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:46.423+0000 7f18baffd700 1 -- 172.21.15.42:0/3198711175 <== mon.0 v2:172.21.15.40:3300/0 4 ==== osd_map(115..115 src has 1..115) v4 ==== 11707+0+0 (secure 0 0 0) 0x7f18b404ab10 con 0x7f18a4002ff0 2019-12-05T06:40:46.446 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:46.423+0000 7f18c9f8d700 1 --2- 172.21.15.42:0/3198711175 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f18a403e820 0x7f18a4040d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:40:46.446 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:46.423+0000 7f18c9f8d700 1 --2- 172.21.15.42:0/3198711175 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f18a403e820 0x7f18a4040d30 crc :-1 s=READY pgs=1263 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:40:46.447 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:46.423+0000 7f18cc1f1700 1 -- 172.21.15.42:0/3198711175 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18c405d110 con 0x7f18a4002ff0 2019-12-05T06:40:46.447 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:46.431+0000 7f18ca78e700 1 --2- 172.21.15.42:0/3198711175 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f18a4002ff0 0x7f18a4005490 secure :-1 s=THROTTLE_DONE pgs=2293 cs=0 l=1 rx=0x7f18b40101c0 tx=0x7f18b4008be0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:46.447 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:46.431+0000 7f18baffd700 1 -- 172.21.15.42:0/3198711175 <== mon.0 v2:172.21.15.40:3300/0 5 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f18b4031b90 con 0x7f18a4002ff0 2019-12-05T06:40:46.609 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:46.603+0000 7f18cc1f1700 1 -- 172.21.15.42:0/3198711175 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f18c405b8c0 con 0x7f18a4002ff0 2019-12-05T06:40:46.610 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:46.603+0000 7f18ca78e700 1 --2- 172.21.15.42:0/3198711175 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f18a4002ff0 0x7f18a4005490 secure :-1 s=THROTTLE_DONE pgs=2293 cs=0 l=1 rx=0x7f18b40101c0 tx=0x7f18b4008be0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:46.610 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:46.603+0000 7f18baffd700 1 -- 172.21.15.42:0/3198711175 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 364 v364) v1 ==== 95+0+1782 (secure 0 0 0) 0x7f18b4031df0 con 0x7f18a4002ff0 2019-12-05T06:40:46.612 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 364 2019-12-05T06:40:46.612 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:40:46.613 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":364,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":364,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:40:46.507615+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":115,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11976},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11976":{"gid":11976,"name":"b","rank":0,"incarnation":361,"state":"up:clientreplay","state_seq":10,"addr":"172.21.15.40:6833/474728597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":474728597},{"type":"v1","addr":"172.21.15.40:6833","nonce":474728597}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:40:46.615 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:46.607+0000 7f18cc1f1700 1 -- 172.21.15.42:0/3198711175 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f18a403e820 msgr2=0x7f18a4040d30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:46.615 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:46.607+0000 7f18cc1f1700 1 --2- 172.21.15.42:0/3198711175 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f18a403e820 0x7f18a4040d30 crc :-1 s=READY pgs=1263 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:40:46.615 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:46.607+0000 7f18cc1f1700 1 -- 172.21.15.42:0/3198711175 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f18a4002ff0 msgr2=0x7f18a4005490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:46.616 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:46.607+0000 7f18cc1f1700 1 --2- 172.21.15.42:0/3198711175 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f18a4002ff0 0x7f18a4005490 secure :-1 s=READY pgs=2293 cs=0 l=1 rx=0x7f18b40101c0 tx=0x7f18b4008be0).stop 2019-12-05T06:40:46.616 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:46.607+0000 7f18cc1f1700 1 -- 172.21.15.42:0/3198711175 shutdown_connections 2019-12-05T06:40:46.616 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:46.607+0000 7f18cc1f1700 1 --2- 172.21.15.42:0/3198711175 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f18a403e820 0x7f18a4040d30 unknown :-1 s=CLOSED pgs=1263 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:40:46.616 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:46.607+0000 7f18cc1f1700 1 --2- 172.21.15.42:0/3198711175 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f18a4002ff0 0x7f18a4005490 unknown :-1 s=CLOSED pgs=2293 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:40:46.617 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:46.607+0000 7f18cc1f1700 1 --2- 172.21.15.42:0/3198711175 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f18c4138d00 0x7f18c413b140 unknown :-1 s=CLOSED pgs=3017 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:40:46.617 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:46.607+0000 7f18cc1f1700 1 -- 172.21.15.42:0/3198711175 shutdown_connections 2019-12-05T06:40:46.617 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:46.607+0000 7f18cc1f1700 1 -- 172.21.15.42:0/3198711175 wait complete. 2019-12-05T06:40:46.617 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:46.607+0000 7f18cc1f1700 1 -- 172.21.15.42:0/3198711175 >> 172.21.15.42:0/3198711175 conn(0x7f18c407e4f0 msgr2=0x7f18c4059ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:40:46.625 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2019-12-05T06:40:47.592 INFO:tasks.workunit.client.0.smithi042.stderr:+ cd ceph 2019-12-05T06:40:47.594 INFO:tasks.workunit.client.0.smithi042.stderr:+ seq 1 21 2019-12-05T06:40:47.595 INFO:tasks.workunit.client.0.smithi042.stderr:+ versions=1 2019-12-05T06:40:47.596 INFO:tasks.workunit.client.0.smithi042.stderr:2 2019-12-05T06:40:47.596 INFO:tasks.workunit.client.0.smithi042.stderr:3 2019-12-05T06:40:47.596 INFO:tasks.workunit.client.0.smithi042.stderr:4 2019-12-05T06:40:47.596 INFO:tasks.workunit.client.0.smithi042.stderr:5 2019-12-05T06:40:47.597 INFO:tasks.workunit.client.0.smithi042.stderr:6 2019-12-05T06:40:47.597 INFO:tasks.workunit.client.0.smithi042.stderr:7 2019-12-05T06:40:47.597 INFO:tasks.workunit.client.0.smithi042.stderr:8 2019-12-05T06:40:47.597 INFO:tasks.workunit.client.0.smithi042.stderr:9 2019-12-05T06:40:47.598 INFO:tasks.workunit.client.0.smithi042.stderr:10 2019-12-05T06:40:47.598 INFO:tasks.workunit.client.0.smithi042.stderr:11 2019-12-05T06:40:47.598 INFO:tasks.workunit.client.0.smithi042.stderr:12 2019-12-05T06:40:47.598 INFO:tasks.workunit.client.0.smithi042.stderr:13 2019-12-05T06:40:47.599 INFO:tasks.workunit.client.0.smithi042.stderr:14 2019-12-05T06:40:47.599 INFO:tasks.workunit.client.0.smithi042.stderr:15 2019-12-05T06:40:47.599 INFO:tasks.workunit.client.0.smithi042.stderr:16 2019-12-05T06:40:47.599 INFO:tasks.workunit.client.0.smithi042.stderr:17 2019-12-05T06:40:47.599 INFO:tasks.workunit.client.0.smithi042.stderr:18 2019-12-05T06:40:47.600 INFO:tasks.workunit.client.0.smithi042.stderr:19 2019-12-05T06:40:47.600 INFO:tasks.workunit.client.0.smithi042.stderr:20 2019-12-05T06:40:47.600 INFO:tasks.workunit.client.0.smithi042.stderr:21 2019-12-05T06:40:47.600 INFO:tasks.workunit.client.0.smithi042.stderr:+ ver=v0.1 2019-12-05T06:40:47.600 INFO:tasks.workunit.client.0.smithi042.stderr:+ echo v0.1 2019-12-05T06:40:47.600 INFO:tasks.workunit.client.0.smithi042.stderr:+ git reset --hard v0.1 2019-12-05T06:40:47.601 INFO:tasks.workunit.client.0.smithi042.stdout:v0.1 2019-12-05T06:40:48.626 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:40:48.753 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:48.731+0000 7f697cb61700 1 Processor -- start 2019-12-05T06:40:48.753 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:48.731+0000 7f697cb61700 1 -- start start 2019-12-05T06:40:48.753 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:48.735+0000 7f697cb61700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f69780a7740 0x7f697808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:40:48.754 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:48.735+0000 7f697cb61700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f697807e720 0x7f697807eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:40:48.754 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:48.735+0000 7f697cb61700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f697809f2e0 0x7f697809f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:40:48.754 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:48.735+0000 7f697cb61700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69780a0b00 con 0x7f697807e720 2019-12-05T06:40:48.754 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:48.735+0000 7f697cb61700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f697808e5b0 con 0x7f69780a7740 2019-12-05T06:40:48.754 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:48.735+0000 7f697cb61700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f697808e850 con 0x7f697809f2e0 2019-12-05T06:40:48.754 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:48.735+0000 7f6976d9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f69780a7740 0x7f697808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:40:48.754 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:48.735+0000 7f697659c700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f697807e720 0x7f697807eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:40:48.755 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:48.735+0000 7f6976d9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f69780a7740 0x7f697808df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:32886/0 (socket says 172.21.15.42:32886) 2019-12-05T06:40:48.755 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:48.735+0000 7f6976d9d700 1 -- 172.21.15.42:0/1851239538 learned_addr learned my addr 172.21.15.42:0/1851239538 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:40:48.755 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:48.735+0000 7f697759e700 1 --2- 172.21.15.42:0/1851239538 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f697809f2e0 0x7f697809f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:40:48.755 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:48.735+0000 7f6976d9d700 1 -- 172.21.15.42:0/1851239538 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f697809f2e0 msgr2=0x7f697809f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:40:48.755 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:48.735+0000 7f6976d9d700 1 --2- 172.21.15.42:0/1851239538 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f697809f2e0 0x7f697809f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:40:48.755 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:48.735+0000 7f6976d9d700 1 -- 172.21.15.42:0/1851239538 >> v2:172.21.15.40:3300/0 conn(0x7f697807e720 msgr2=0x7f697807eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:40:48.755 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:48.735+0000 7f6976d9d700 1 --2- 172.21.15.42:0/1851239538 >> v2:172.21.15.40:3300/0 conn(0x7f697807e720 0x7f697807eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:40:48.756 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:48.735+0000 7f6976d9d700 1 -- 172.21.15.42:0/1851239538 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f697807f0a0 con 0x7f69780a7740 2019-12-05T06:40:48.756 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:48.735+0000 7f6976d9d700 1 --2- 172.21.15.42:0/1851239538 >> v2:172.21.15.42:3300/0 conn(0x7f69780a7740 0x7f697808df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f69780a0f50 tx=0x7f6968010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:48.756 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:48.735+0000 7f6976d9d700 1 --2- 172.21.15.42:0/1851239538 >> v2:172.21.15.42:3300/0 conn(0x7f69780a7740 0x7f697808df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f69780a0f50 tx=0x7f6968010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:48.756 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:48.735+0000 7f6976d9d700 1 --2- 172.21.15.42:0/1851239538 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f69780a7740 0x7f697808df80 secure :-1 s=READY pgs=3018 cs=0 l=1 rx=0x7f69780a0f50 tx=0x7f6968010130).ready entity=mon.1 client_cookie=8fb5807b6b39388 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:40:48.757 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:48.735+0000 7f6976d9d700 1 --2- 172.21.15.42:0/1851239538 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f69780a7740 0x7f697808df80 secure :-1 s=THROTTLE_DONE pgs=3018 cs=0 l=1 rx=0x7f69780a0f50 tx=0x7f6968010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:48.757 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:48.735+0000 7f6975d9b700 1 -- 172.21.15.42:0/1851239538 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6968010de0 con 0x7f69780a7740 2019-12-05T06:40:48.757 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:48.735+0000 7f6976d9d700 1 --2- 172.21.15.42:0/1851239538 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f69780a7740 0x7f697808df80 secure :-1 s=THROTTLE_DONE pgs=3018 cs=0 l=1 rx=0x7f69780a0f50 tx=0x7f6968010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:48.757 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:48.735+0000 7f6975d9b700 1 -- 172.21.15.42:0/1851239538 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6968010a00 con 0x7f69780a7740 2019-12-05T06:40:48.757 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:48.735+0000 7f6976d9d700 1 --2- 172.21.15.42:0/1851239538 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f69780a7740 0x7f697808df80 secure :-1 s=THROTTLE_DONE pgs=3018 cs=0 l=1 rx=0x7f69780a0f50 tx=0x7f6968010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:48.757 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:48.735+0000 7f6975d9b700 1 -- 172.21.15.42:0/1851239538 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f69680114a0 con 0x7f69780a7740 2019-12-05T06:40:48.757 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:48.735+0000 7f697cb61700 1 -- 172.21.15.42:0/1851239538 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f69780a7740 msgr2=0x7f697808df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:48.758 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:48.735+0000 7f697cb61700 1 --2- 172.21.15.42:0/1851239538 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f69780a7740 0x7f697808df80 secure :-1 s=READY pgs=3018 cs=0 l=1 rx=0x7f69780a0f50 tx=0x7f6968010130).stop 2019-12-05T06:40:48.758 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:48.739+0000 7f697cb61700 1 -- 172.21.15.42:0/1851239538 shutdown_connections 2019-12-05T06:40:48.758 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:48.739+0000 7f697cb61700 1 --2- 172.21.15.42:0/1851239538 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f697809f2e0 0x7f697809f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:40:48.758 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:48.739+0000 7f697cb61700 1 --2- 172.21.15.42:0/1851239538 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f69780a7740 0x7f697808df80 unknown :-1 s=CLOSED pgs=3018 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:40:48.758 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:48.739+0000 7f697cb61700 1 --2- 172.21.15.42:0/1851239538 >> v2:172.21.15.40:3300/0 conn(0x7f697807e720 0x7f697807eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:40:48.758 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:48.739+0000 7f697cb61700 1 -- 172.21.15.42:0/1851239538 shutdown_connections 2019-12-05T06:40:48.758 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:48.739+0000 7f697cb61700 1 -- 172.21.15.42:0/1851239538 wait complete. 2019-12-05T06:40:48.759 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:48.739+0000 7f697cb61700 1 -- 172.21.15.42:0/1851239538 >> 172.21.15.42:0/1851239538 conn(0x7f69780aabe0 msgr2=0x7f69780a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:40:48.759 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:48.739+0000 7f697cb61700 1 Processor -- start 2019-12-05T06:40:48.759 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:48.739+0000 7f697cb61700 1 -- start start 2019-12-05T06:40:48.762 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:48.743+0000 7f697cb61700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f69781244b0 0x7f69781451c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:40:48.762 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:48.743+0000 7f697cb61700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f697811b850 0x7f69781300c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:40:48.762 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:48.743+0000 7f697cb61700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f697811a680 con 0x7f697807db50 2019-12-05T06:40:48.762 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:48.743+0000 7f697cb61700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f697811bc80 con 0x7f697811b850 2019-12-05T06:40:48.762 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:48.743+0000 7f697cb61700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6978142680 con 0x7f69781244b0 2019-12-05T06:40:48.763 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:48.743+0000 7f697759e700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f697807db50 0x7f6978120780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:32934/0 (socket says 172.21.15.42:32934) 2019-12-05T06:40:48.763 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:48.743+0000 7f6976d9d700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f69781244b0 0x7f69781451c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:40:48.763 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:48.743+0000 7f697659c700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f697811b850 0x7f69781300c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:40:48.763 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:48.743+0000 7f697759e700 1 -- 172.21.15.42:0/2655566744 learned_addr learned my addr 172.21.15.42:0/2655566744 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:40:48.763 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:48.743+0000 7f6976d9d700 1 -- 172.21.15.42:0/2655566744 >> v2:172.21.15.40:3300/0 conn(0x7f697811b850 msgr2=0x7f69781300c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:48.763 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:48.743+0000 7f6976d9d700 1 --2- 172.21.15.42:0/2655566744 >> v2:172.21.15.40:3300/0 conn(0x7f697811b850 0x7f69781300c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:40:48.763 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:48.743+0000 7f696f7fe700 1 -- 172.21.15.42:0/2655566744 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1478906148 0 0) 0x7f697811a680 con 0x7f697807db50 2019-12-05T06:40:48.764 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:48.743+0000 7f6976d9d700 1 -- 172.21.15.42:0/2655566744 >> v1:172.21.15.42:6789/0 conn(0x7f697807db50 legacy=0x7f6978120780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:48.764 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:48.743+0000 7f6976d9d700 1 -- 172.21.15.42:0/2655566744 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6978130610 con 0x7f69781244b0 2019-12-05T06:40:48.764 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:48.743+0000 7f6976d9d700 1 --2- 172.21.15.42:0/2655566744 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f69781244b0 0x7f69781451c0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f69780a0f50 tx=0x7f6968007a60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:48.764 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:48.743+0000 7f6976d9d700 1 --2- 172.21.15.42:0/2655566744 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f69781244b0 0x7f69781451c0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f69780a0f50 tx=0x7f6968007a60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:48.764 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:48.743+0000 7f6976d9d700 1 --2- 172.21.15.42:0/2655566744 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f69781244b0 0x7f69781451c0 secure :-1 s=READY pgs=659 cs=0 l=1 rx=0x7f69780a0f50 tx=0x7f6968007a60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:40:48.764 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:48.743+0000 7f6976d9d700 1 --2- 172.21.15.42:0/2655566744 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f69781244b0 0x7f69781451c0 secure :-1 s=THROTTLE_DONE pgs=659 cs=0 l=1 rx=0x7f69780a0f50 tx=0x7f6968007a60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:48.764 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:48.743+0000 7f696f7fe700 1 -- 172.21.15.42:0/2655566744 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6968011bc0 con 0x7f69781244b0 2019-12-05T06:40:48.765 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:48.743+0000 7f6976d9d700 1 --2- 172.21.15.42:0/2655566744 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f69781244b0 0x7f69781451c0 secure :-1 s=THROTTLE_DONE pgs=659 cs=0 l=1 rx=0x7f69780a0f50 tx=0x7f6968007a60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:48.765 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:48.743+0000 7f696f7fe700 1 -- 172.21.15.42:0/2655566744 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69680110c0 con 0x7f69781244b0 2019-12-05T06:40:48.765 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:48.743+0000 7f697cb61700 1 -- 172.21.15.42:0/2655566744 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f697807df80 con 0x7f69781244b0 2019-12-05T06:40:48.765 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:48.743+0000 7f6976d9d700 1 --2- 172.21.15.42:0/2655566744 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f69781244b0 0x7f69781451c0 secure :-1 s=THROTTLE_DONE pgs=659 cs=0 l=1 rx=0x7f69780a0f50 tx=0x7f6968007a60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:48.765 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:48.743+0000 7f696f7fe700 1 -- 172.21.15.42:0/2655566744 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6968005ab0 con 0x7f69781244b0 2019-12-05T06:40:48.765 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:48.743+0000 7f697cb61700 1 -- 172.21.15.42:0/2655566744 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6978120f80 con 0x7f69781244b0 2019-12-05T06:40:48.765 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:48.743+0000 7f6976d9d700 1 --2- 172.21.15.42:0/2655566744 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f69781244b0 0x7f69781451c0 secure :-1 s=THROTTLE_DONE pgs=659 cs=0 l=1 rx=0x7f69780a0f50 tx=0x7f6968007a60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:48.765 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:48.747+0000 7f696f7fe700 1 -- 172.21.15.42:0/2655566744 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f696800f280 con 0x7f69781244b0 2019-12-05T06:40:48.766 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:48.747+0000 7f6976d9d700 1 --2- 172.21.15.42:0/2655566744 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f69781244b0 0x7f69781451c0 secure :-1 s=THROTTLE_DONE pgs=659 cs=0 l=1 rx=0x7f69780a0f50 tx=0x7f6968007a60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:48.766 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:48.747+0000 7f696f7fe700 1 --2- 172.21.15.42:0/2655566744 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f695c03ac00 0x7f695c03d110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:40:48.766 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:48.747+0000 7f696f7fe700 1 -- 172.21.15.42:0/2655566744 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 11707+0+0 (secure 0 0 0) 0x7f6968091ab0 con 0x7f69781244b0 2019-12-05T06:40:48.766 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:48.747+0000 7f697659c700 1 --2- 172.21.15.42:0/2655566744 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f695c03ac00 0x7f695c03d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:40:48.766 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:48.747+0000 7f697659c700 1 --2- 172.21.15.42:0/2655566744 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f695c03ac00 0x7f695c03d110 crc :-1 s=READY pgs=1265 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:40:48.766 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:48.747+0000 7f697cb61700 1 -- 172.21.15.42:0/2655566744 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f697805d110 con 0x7f69781244b0 2019-12-05T06:40:48.767 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:48.755+0000 7f6976d9d700 1 --2- 172.21.15.42:0/2655566744 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f69781244b0 0x7f69781451c0 secure :-1 s=THROTTLE_DONE pgs=659 cs=0 l=1 rx=0x7f69780a0f50 tx=0x7f6968007a60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:48.767 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:48.755+0000 7f696f7fe700 1 -- 172.21.15.42:0/2655566744 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f6978130610 con 0x7f69781244b0 2019-12-05T06:40:48.928 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:48.919+0000 7f697cb61700 1 -- 172.21.15.42:0/2655566744 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f697805b8c0 con 0x7f69781244b0 2019-12-05T06:40:48.929 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:48.923+0000 7f6976d9d700 1 --2- 172.21.15.42:0/2655566744 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f69781244b0 0x7f69781451c0 secure :-1 s=THROTTLE_DONE pgs=659 cs=0 l=1 rx=0x7f69780a0f50 tx=0x7f6968007a60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:48.929 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:48.923+0000 7f696f7fe700 1 -- 172.21.15.42:0/2655566744 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 365 v365) v1 ==== 95+0+1776 (secure 0 0 0) 0x7f6968078b10 con 0x7f69781244b0 2019-12-05T06:40:48.931 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:40:48.931 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":365,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":365,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:40:47.519742+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":115,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11976},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11976":{"gid":11976,"name":"b","rank":0,"incarnation":361,"state":"up:active","state_seq":11,"addr":"172.21.15.40:6833/474728597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":474728597},{"type":"v1","addr":"172.21.15.40:6833","nonce":474728597}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:40:48.932 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 365 2019-12-05T06:40:48.934 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:48.927+0000 7f697cb61700 1 -- 172.21.15.42:0/2655566744 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f695c03ac00 msgr2=0x7f695c03d110 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:48.935 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:48.927+0000 7f697cb61700 1 --2- 172.21.15.42:0/2655566744 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f695c03ac00 0x7f695c03d110 crc :-1 s=READY pgs=1265 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:40:48.935 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:48.927+0000 7f697cb61700 1 -- 172.21.15.42:0/2655566744 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f69781244b0 msgr2=0x7f69781451c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:48.935 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:48.927+0000 7f697cb61700 1 --2- 172.21.15.42:0/2655566744 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f69781244b0 0x7f69781451c0 secure :-1 s=READY pgs=659 cs=0 l=1 rx=0x7f69780a0f50 tx=0x7f6968007a60).stop 2019-12-05T06:40:48.936 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:48.927+0000 7f697cb61700 1 -- 172.21.15.42:0/2655566744 shutdown_connections 2019-12-05T06:40:48.936 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:48.927+0000 7f697cb61700 1 --2- 172.21.15.42:0/2655566744 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f695c03ac00 0x7f695c03d110 unknown :-1 s=CLOSED pgs=1265 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:40:48.936 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:48.927+0000 7f697cb61700 1 --2- 172.21.15.42:0/2655566744 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f69781244b0 0x7f69781451c0 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:40:48.936 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:48.927+0000 7f697cb61700 1 --2- 172.21.15.42:0/2655566744 >> v2:172.21.15.40:3300/0 conn(0x7f697811b850 0x7f69781300c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:40:48.937 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:48.927+0000 7f697cb61700 1 -- 172.21.15.42:0/2655566744 shutdown_connections 2019-12-05T06:40:48.937 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:48.927+0000 7f697cb61700 1 -- 172.21.15.42:0/2655566744 wait complete. 2019-12-05T06:40:48.937 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:48.927+0000 7f697cb61700 1 -- 172.21.15.42:0/2655566744 >> 172.21.15.42:0/2655566744 conn(0x7f69780aabe0 msgr2=0x7f697809f620 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:40:48.945 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b has gained rank=0, replacing gid=11868 2019-12-05T06:40:48.946 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 0 secs before reviving mds.a 2019-12-05T06:40:48.946 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.a 2019-12-05T06:40:48.946 INFO:tasks.ceph.mds.a:Restarting daemon 2019-12-05T06:40:48.946 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2019-12-05T06:40:48.950 INFO:tasks.ceph.mds.a:Started 2019-12-05T06:40:48.951 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:40:49.006 INFO:tasks.ceph.mds.a.smithi042.stdout:starting mds.a at 2019-12-05T06:40:49.050 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:49.035+0000 7ff9f48c4700 1 Processor -- start 2019-12-05T06:40:49.051 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:49.035+0000 7ff9f48c4700 1 -- start start 2019-12-05T06:40:49.051 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:49.039+0000 7ff9f48c4700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff9f00a7740 0x7ff9f008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:40:49.051 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:49.039+0000 7ff9f48c4700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9f00a0710 con 0x7ff9f009fae0 2019-12-05T06:40:49.052 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:49.039+0000 7ff9f48c4700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9f00a9ba0 con 0x7ff9f007e720 2019-12-05T06:40:49.052 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:49.039+0000 7ff9f48c4700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9f00a0b00 con 0x7ff9f00a7740 2019-12-05T06:40:49.052 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:49.039+0000 7ff9eed9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff9f00a7740 0x7ff9f008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:40:49.052 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:49.039+0000 7ff9eed9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff9f00a7740 0x7ff9f008df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:32912/0 (socket says 172.21.15.42:32912) 2019-12-05T06:40:49.053 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:49.039+0000 7ff9ee59c700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7ff9f007e720 0x7ff9f009f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:32950/0 (socket says 172.21.15.42:32950) 2019-12-05T06:40:49.053 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:49.039+0000 7ff9eed9d700 1 -- 172.21.15.42:0/4122666140 learned_addr learned my addr 172.21.15.42:0/4122666140 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:40:49.053 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:49.039+0000 7ff9eed9d700 1 -- 172.21.15.42:0/4122666140 >> v1:172.21.15.42:6789/0 conn(0x7ff9f007e720 legacy=0x7ff9f009f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:49.053 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:49.039+0000 7ff9eed9d700 1 -- 172.21.15.42:0/4122666140 >> v1:172.21.15.40:6789/0 conn(0x7ff9f009fae0 legacy=0x7ff9f014efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:49.054 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:49.039+0000 7ff9eed9d700 1 -- 172.21.15.42:0/4122666140 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9f007eb50 con 0x7ff9f00a7740 2019-12-05T06:40:49.054 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:49.039+0000 7ff9eed9d700 1 --2- 172.21.15.42:0/4122666140 >> v2:172.21.15.42:3300/0 conn(0x7ff9f00a7740 0x7ff9f008df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7ff9e0008790 tx=0x7ff9e000fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:49.054 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:49.039+0000 7ff9eed9d700 1 --2- 172.21.15.42:0/4122666140 >> v2:172.21.15.42:3300/0 conn(0x7ff9f00a7740 0x7ff9f008df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7ff9e0008790 tx=0x7ff9e000fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:49.054 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:49.039+0000 7ff9eed9d700 1 --2- 172.21.15.42:0/4122666140 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff9f00a7740 0x7ff9f008df80 secure :-1 s=READY pgs=3024 cs=0 l=1 rx=0x7ff9e0008790 tx=0x7ff9e000fcb0).ready entity=mon.1 client_cookie=dc3eaeba3d8b63d1 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:40:49.055 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:49.039+0000 7ff9eed9d700 1 --2- 172.21.15.42:0/4122666140 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff9f00a7740 0x7ff9f008df80 secure :-1 s=THROTTLE_DONE pgs=3024 cs=0 l=1 rx=0x7ff9e0008790 tx=0x7ff9e000fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:49.055 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:49.039+0000 7ff9edd9b700 1 -- 172.21.15.42:0/4122666140 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff9e0011520 con 0x7ff9f00a7740 2019-12-05T06:40:49.055 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:49.039+0000 7ff9eed9d700 1 --2- 172.21.15.42:0/4122666140 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff9f00a7740 0x7ff9f008df80 secure :-1 s=THROTTLE_DONE pgs=3024 cs=0 l=1 rx=0x7ff9e0008790 tx=0x7ff9e000fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:49.056 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:49.039+0000 7ff9eed9d700 1 --2- 172.21.15.42:0/4122666140 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff9f00a7740 0x7ff9f008df80 secure :-1 s=THROTTLE_DONE pgs=3024 cs=0 l=1 rx=0x7ff9e0008790 tx=0x7ff9e000fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:49.056 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:49.039+0000 7ff9edd9b700 1 -- 172.21.15.42:0/4122666140 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9e0008ca0 con 0x7ff9f00a7740 2019-12-05T06:40:49.056 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:49.039+0000 7ff9edd9b700 1 -- 172.21.15.42:0/4122666140 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff9e0011b00 con 0x7ff9f00a7740 2019-12-05T06:40:49.056 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:49.039+0000 7ff9f48c4700 1 -- 172.21.15.42:0/4122666140 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff9f00a7740 msgr2=0x7ff9f008df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:49.057 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:49.039+0000 7ff9f48c4700 1 --2- 172.21.15.42:0/4122666140 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff9f00a7740 0x7ff9f008df80 secure :-1 s=READY pgs=3024 cs=0 l=1 rx=0x7ff9e0008790 tx=0x7ff9e000fcb0).stop 2019-12-05T06:40:49.057 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:49.043+0000 7ff9f48c4700 1 -- 172.21.15.42:0/4122666140 shutdown_connections 2019-12-05T06:40:49.057 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:49.043+0000 7ff9f48c4700 1 --2- 172.21.15.42:0/4122666140 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff9f00a7740 0x7ff9f008df80 unknown :-1 s=CLOSED pgs=3024 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:40:49.057 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:49.043+0000 7ff9f48c4700 1 -- 172.21.15.42:0/4122666140 shutdown_connections 2019-12-05T06:40:49.058 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:49.043+0000 7ff9f48c4700 1 -- 172.21.15.42:0/4122666140 wait complete. 2019-12-05T06:40:49.058 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:49.043+0000 7ff9f48c4700 1 -- 172.21.15.42:0/4122666140 >> 172.21.15.42:0/4122666140 conn(0x7ff9f00aabe0 msgr2=0x7ff9f00a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:40:49.064 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:49.043+0000 7ff9f48c4700 1 Processor -- start 2019-12-05T06:40:49.064 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:49.043+0000 7ff9f48c4700 1 -- start start 2019-12-05T06:40:49.065 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:49.047+0000 7ff9f48c4700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff9f0063220 0x7ff9f0065180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:40:49.065 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:49.047+0000 7ff9f48c4700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9f007e1f0 con 0x7ff9f00a3340 2019-12-05T06:40:49.065 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:49.047+0000 7ff9f48c4700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9f0131fc0 con 0x7ff9f011a260 2019-12-05T06:40:49.065 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:49.047+0000 7ff9f48c4700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9f00637a0 con 0x7ff9f0063220 2019-12-05T06:40:49.065 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:49.047+0000 7ff9ee59c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff9f0063220 0x7ff9f0065180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:40:49.066 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:49.047+0000 7ff9eed9d700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7ff9f011a260 0x7ff9f007d9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:32956/0 (socket says 172.21.15.42:32956) 2019-12-05T06:40:49.066 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:49.047+0000 7ff9eed9d700 1 -- 172.21.15.42:0/3614588925 learned_addr learned my addr 172.21.15.42:0/3614588925 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:40:49.066 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:49.047+0000 7ff9ee59c700 1 --2- 172.21.15.42:0/3614588925 >> v2:172.21.15.42:3300/0 conn(0x7ff9f0063220 0x7ff9f0065180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:32920/0 (socket says 172.21.15.42:32920) 2019-12-05T06:40:49.066 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:49.047+0000 7ff9ee59c700 1 -- 172.21.15.42:0/3614588925 >> v1:172.21.15.42:6789/0 conn(0x7ff9f011a260 legacy=0x7ff9f007d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:49.067 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:49.047+0000 7ff9ee59c700 1 -- 172.21.15.42:0/3614588925 >> v1:172.21.15.40:6789/0 conn(0x7ff9f00a3340 legacy=0x7ff9f00a3770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:49.067 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:49.047+0000 7ff9df7fe700 1 -- 172.21.15.42:0/3614588925 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3080710274 0 0) 0x7ff9f0131fc0 con 0x7ff9f011a260 2019-12-05T06:40:49.067 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:49.047+0000 7ff9ee59c700 1 -- 172.21.15.42:0/3614588925 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9f0133200 con 0x7ff9f0063220 2019-12-05T06:40:49.067 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:49.047+0000 7ff9ee59c700 1 --2- 172.21.15.42:0/3614588925 >> v2:172.21.15.42:3300/0 conn(0x7ff9f0063220 0x7ff9f0065180 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7ff9e80013c0 tx=0x7ff9e8001400).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:49.067 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:49.047+0000 7ff9ee59c700 1 --2- 172.21.15.42:0/3614588925 >> v2:172.21.15.42:3300/0 conn(0x7ff9f0063220 0x7ff9f0065180 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7ff9e80013c0 tx=0x7ff9e8001400).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:49.067 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:49.047+0000 7ff9ee59c700 1 --2- 172.21.15.42:0/3614588925 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff9f0063220 0x7ff9f0065180 secure :-1 s=READY pgs=3026 cs=0 l=1 rx=0x7ff9e80013c0 tx=0x7ff9e8001400).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:40:49.068 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:49.047+0000 7ff9ee59c700 1 --2- 172.21.15.42:0/3614588925 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff9f0063220 0x7ff9f0065180 secure :-1 s=THROTTLE_DONE pgs=3026 cs=0 l=1 rx=0x7ff9e80013c0 tx=0x7ff9e8001400).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:49.068 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:49.047+0000 7ff9df7fe700 1 -- 172.21.15.42:0/3614588925 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff9e4006850 con 0x7ff9f0063220 2019-12-05T06:40:49.068 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:49.047+0000 7ff9ee59c700 1 --2- 172.21.15.42:0/3614588925 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff9f0063220 0x7ff9f0065180 secure :-1 s=THROTTLE_DONE pgs=3026 cs=0 l=1 rx=0x7ff9e80013c0 tx=0x7ff9e8001400).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:49.068 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:49.047+0000 7ff9ee59c700 1 --2- 172.21.15.42:0/3614588925 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff9f0063220 0x7ff9f0065180 secure :-1 s=THROTTLE_DONE pgs=3026 cs=0 l=1 rx=0x7ff9e80013c0 tx=0x7ff9e8001400).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:49.068 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:49.047+0000 7ff9df7fe700 1 -- 172.21.15.42:0/3614588925 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9e4006470 con 0x7ff9f0063220 2019-12-05T06:40:49.069 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:49.047+0000 7ff9df7fe700 1 -- 172.21.15.42:0/3614588925 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff9e40110b0 con 0x7ff9f0063220 2019-12-05T06:40:49.069 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:49.047+0000 7ff9f48c4700 1 -- 172.21.15.42:0/3614588925 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9f0130fb0 con 0x7ff9f0063220 2019-12-05T06:40:49.069 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:49.047+0000 7ff9f48c4700 1 -- 172.21.15.42:0/3614588925 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9f01314e0 con 0x7ff9f0063220 2019-12-05T06:40:49.069 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:49.047+0000 7ff9ee59c700 1 --2- 172.21.15.42:0/3614588925 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff9f0063220 0x7ff9f0065180 secure :-1 s=THROTTLE_DONE pgs=3026 cs=0 l=1 rx=0x7ff9e80013c0 tx=0x7ff9e8001400).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:49.070 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:49.047+0000 7ff9df7fe700 1 -- 172.21.15.42:0/3614588925 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7ff9e40113c0 con 0x7ff9f0063220 2019-12-05T06:40:49.070 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:49.047+0000 7ff9ee59c700 1 --2- 172.21.15.42:0/3614588925 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff9f0063220 0x7ff9f0065180 secure :-1 s=THROTTLE_DONE pgs=3026 cs=0 l=1 rx=0x7ff9e80013c0 tx=0x7ff9e8001400).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:49.071 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:49.047+0000 7ff9df7fe700 1 --2- 172.21.15.42:0/3614588925 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff9d403ac80 0x7ff9d403d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:40:49.071 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:49.047+0000 7ff9df7fe700 1 -- 172.21.15.42:0/3614588925 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 11707+0+0 (secure 0 0 0) 0x7ff9e404b020 con 0x7ff9f0063220 2019-12-05T06:40:49.071 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:49.047+0000 7ff9eed9d700 1 --2- 172.21.15.42:0/3614588925 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff9d403ac80 0x7ff9d403d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:40:49.071 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:49.051+0000 7ff9eed9d700 1 --2- 172.21.15.42:0/3614588925 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff9d403ac80 0x7ff9d403d190 crc :-1 s=READY pgs=1266 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:40:49.071 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:49.051+0000 7ff9f48c4700 1 -- 172.21.15.42:0/3614588925 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9f005d110 con 0x7ff9f0063220 2019-12-05T06:40:49.072 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:49.059+0000 7ff9ee59c700 1 --2- 172.21.15.42:0/3614588925 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff9f0063220 0x7ff9f0065180 secure :-1 s=THROTTLE_DONE pgs=3026 cs=0 l=1 rx=0x7ff9e80013c0 tx=0x7ff9e8001400).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:49.072 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:49.059+0000 7ff9df7fe700 1 -- 172.21.15.42:0/3614588925 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7ff9e40320a0 con 0x7ff9f0063220 2019-12-05T06:40:49.235 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:49.227+0000 7ff9f48c4700 1 -- 172.21.15.42:0/3614588925 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff9f005b8c0 con 0x7ff9f0063220 2019-12-05T06:40:49.236 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:49.227+0000 7ff9ee59c700 1 --2- 172.21.15.42:0/3614588925 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff9f0063220 0x7ff9f0065180 secure :-1 s=THROTTLE_DONE pgs=3026 cs=0 l=1 rx=0x7ff9e80013c0 tx=0x7ff9e8001400).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:49.236 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:49.227+0000 7ff9df7fe700 1 -- 172.21.15.42:0/3614588925 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 365 v365) v1 ==== 95+0+1776 (secure 0 0 0) 0x7ff9e4032300 con 0x7ff9f0063220 2019-12-05T06:40:49.236 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 365 2019-12-05T06:40:49.236 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:40:49.236 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":365,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":365,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:40:47.519742+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":115,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11976},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11976":{"gid":11976,"name":"b","rank":0,"incarnation":361,"state":"up:active","state_seq":11,"addr":"172.21.15.40:6833/474728597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":474728597},{"type":"v1","addr":"172.21.15.40:6833","nonce":474728597}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:40:49.239 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:49.231+0000 7ff9f48c4700 1 -- 172.21.15.42:0/3614588925 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff9d403ac80 msgr2=0x7ff9d403d190 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:49.239 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:49.231+0000 7ff9f48c4700 1 --2- 172.21.15.42:0/3614588925 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff9d403ac80 0x7ff9d403d190 crc :-1 s=READY pgs=1266 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:40:49.239 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:49.231+0000 7ff9f48c4700 1 -- 172.21.15.42:0/3614588925 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff9f0063220 msgr2=0x7ff9f0065180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:49.239 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:49.231+0000 7ff9f48c4700 1 --2- 172.21.15.42:0/3614588925 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff9f0063220 0x7ff9f0065180 secure :-1 s=READY pgs=3026 cs=0 l=1 rx=0x7ff9e80013c0 tx=0x7ff9e8001400).stop 2019-12-05T06:40:49.239 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:49.231+0000 7ff9f48c4700 1 -- 172.21.15.42:0/3614588925 shutdown_connections 2019-12-05T06:40:49.239 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:49.231+0000 7ff9f48c4700 1 --2- 172.21.15.42:0/3614588925 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff9d403ac80 0x7ff9d403d190 unknown :-1 s=CLOSED pgs=1266 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:40:49.240 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:49.231+0000 7ff9f48c4700 1 --2- 172.21.15.42:0/3614588925 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff9f0063220 0x7ff9f0065180 unknown :-1 s=CLOSED pgs=3026 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:40:49.240 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:49.231+0000 7ff9f48c4700 1 -- 172.21.15.42:0/3614588925 shutdown_connections 2019-12-05T06:40:49.240 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:49.231+0000 7ff9f48c4700 1 -- 172.21.15.42:0/3614588925 wait complete. 2019-12-05T06:40:49.240 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:49.231+0000 7ff9f48c4700 1 -- 172.21.15.42:0/3614588925 >> 172.21.15.42:0/3614588925 conn(0x7ff9f008fde0 msgr2=0x7ff9f0059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:40:49.250 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2019-12-05T06:40:51.257 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:40:51.371 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:51.351+0000 7f3d017cf700 1 Processor -- start 2019-12-05T06:40:51.371 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:51.351+0000 7f3d017cf700 1 -- start start 2019-12-05T06:40:51.371 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:51.351+0000 7f3d017cf700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f3cfc0a7740 0x7f3cfc08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:40:51.371 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:51.351+0000 7f3d017cf700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cfc0a0710 con 0x7f3cfc07e720 2019-12-05T06:40:51.371 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:51.351+0000 7f3d017cf700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cfc0a9ba0 con 0x7f3cfc09fae0 2019-12-05T06:40:51.371 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:51.351+0000 7f3d017cf700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3cfc0a0b00 con 0x7f3cfc0a7740 2019-12-05T06:40:51.372 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:51.351+0000 7f3cfb7fe700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f3cfc0a7740 0x7f3cfc08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:40:51.372 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:51.351+0000 7f3cfb7fe700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f3cfc0a7740 0x7f3cfc08df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.42:48992/0 (socket says 172.21.15.42:48992) 2019-12-05T06:40:51.372 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:51.351+0000 7f3cfb7fe700 1 -- 172.21.15.42:0/4011157008 learned_addr learned my addr 172.21.15.42:0/4011157008 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:40:51.372 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:51.351+0000 7f3cfaffd700 1 --1- 172.21.15.42:0/4011157008 >> v1:172.21.15.40:6789/0 conn(0x7f3cfc07e720 0x7f3cfc09f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:42424/0 (socket says 172.21.15.42:42424) 2019-12-05T06:40:51.372 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:51.351+0000 7f3cfb7fe700 1 -- 172.21.15.42:0/4011157008 >> v1:172.21.15.42:6789/0 conn(0x7f3cfc09fae0 legacy=0x7f3cfc14efe0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2019-12-05T06:40:51.373 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:51.351+0000 7f3cfb7fe700 1 -- 172.21.15.42:0/4011157008 >> v1:172.21.15.40:6789/0 conn(0x7f3cfc07e720 legacy=0x7f3cfc09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:51.373 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:51.351+0000 7f3cfb7fe700 1 -- 172.21.15.42:0/4011157008 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cfc07eb50 con 0x7f3cfc0a7740 2019-12-05T06:40:51.373 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:51.351+0000 7f3cfb7fe700 1 --2- 172.21.15.42:0/4011157008 >> v2:172.21.15.40:3300/0 conn(0x7f3cfc0a7740 0x7f3cfc08df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f3cec008790 tx=0x7f3cec00fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:51.373 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:51.351+0000 7f3cfb7fe700 1 --2- 172.21.15.42:0/4011157008 >> v2:172.21.15.40:3300/0 conn(0x7f3cfc0a7740 0x7f3cfc08df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f3cec008790 tx=0x7f3cec00fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:51.373 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:51.351+0000 7f3cfb7fe700 1 --2- 172.21.15.42:0/4011157008 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3cfc0a7740 0x7f3cfc08df80 secure :-1 s=READY pgs=2298 cs=0 l=1 rx=0x7f3cec008790 tx=0x7f3cec00fcb0).ready entity=mon.0 client_cookie=dcedd1d58f252d8 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:40:51.373 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:51.351+0000 7f3cfb7fe700 1 --2- 172.21.15.42:0/4011157008 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3cfc0a7740 0x7f3cfc08df80 secure :-1 s=THROTTLE_DONE pgs=2298 cs=0 l=1 rx=0x7f3cec008790 tx=0x7f3cec00fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:51.373 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:51.351+0000 7f3cfb7fe700 1 --2- 172.21.15.42:0/4011157008 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3cfc0a7740 0x7f3cfc08df80 secure :-1 s=THROTTLE_DONE pgs=2298 cs=0 l=1 rx=0x7f3cec008790 tx=0x7f3cec00fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:51.374 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:51.351+0000 7f3cfa7fc700 1 -- 172.21.15.42:0/4011157008 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3cec011520 con 0x7f3cfc0a7740 2019-12-05T06:40:51.374 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:51.351+0000 7f3cfb7fe700 1 --2- 172.21.15.42:0/4011157008 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3cfc0a7740 0x7f3cfc08df80 secure :-1 s=THROTTLE_DONE pgs=2298 cs=0 l=1 rx=0x7f3cec008790 tx=0x7f3cec00fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:51.375 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:51.351+0000 7f3cfa7fc700 1 -- 172.21.15.42:0/4011157008 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cec008ca0 con 0x7f3cfc0a7740 2019-12-05T06:40:51.375 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:51.351+0000 7f3cfa7fc700 1 -- 172.21.15.42:0/4011157008 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3cec011b00 con 0x7f3cfc0a7740 2019-12-05T06:40:51.375 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:51.351+0000 7f3d017cf700 1 -- 172.21.15.42:0/4011157008 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3cfc0a7740 msgr2=0x7f3cfc08df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:51.375 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:51.351+0000 7f3d017cf700 1 --2- 172.21.15.42:0/4011157008 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3cfc0a7740 0x7f3cfc08df80 secure :-1 s=READY pgs=2298 cs=0 l=1 rx=0x7f3cec008790 tx=0x7f3cec00fcb0).stop 2019-12-05T06:40:51.375 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:51.355+0000 7f3d017cf700 1 -- 172.21.15.42:0/4011157008 shutdown_connections 2019-12-05T06:40:51.375 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:51.355+0000 7f3d017cf700 1 --2- 172.21.15.42:0/4011157008 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3cfc0a7740 0x7f3cfc08df80 unknown :-1 s=CLOSED pgs=2298 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:40:51.375 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:51.355+0000 7f3d017cf700 1 -- 172.21.15.42:0/4011157008 shutdown_connections 2019-12-05T06:40:51.376 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:51.355+0000 7f3d017cf700 1 -- 172.21.15.42:0/4011157008 wait complete. 2019-12-05T06:40:51.376 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:51.355+0000 7f3d017cf700 1 -- 172.21.15.42:0/4011157008 >> 172.21.15.42:0/4011157008 conn(0x7f3cfc0aabe0 msgr2=0x7f3cfc0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:40:51.376 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:51.359+0000 7f3d017cf700 1 Processor -- start 2019-12-05T06:40:51.376 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:51.359+0000 7f3d017cf700 1 -- start start 2019-12-05T06:40:51.376 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:51.359+0000 7f3d017cf700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3cfc07d7c0 0x7f3cfc07dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:40:51.376 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:51.359+0000 7f3d017cf700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f3cfc125950 0x7f3cfc125d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:40:51.377 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:51.359+0000 7f3d017cf700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cfc07e140 con 0x7f3cfc11d6b0 2019-12-05T06:40:51.377 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:51.359+0000 7f3d017cf700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3cfc0a4e70 con 0x7f3cfc125950 2019-12-05T06:40:51.377 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:51.359+0000 7f3d017cf700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3cfc0a5110 con 0x7f3cfc07d7c0 2019-12-05T06:40:51.377 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:51.359+0000 7f3cfaffd700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3cfc07d7c0 0x7f3cfc07dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:40:51.377 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:51.359+0000 7f3cfb7fe700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f3cfc11d6b0 0x7f3cfc0a6420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:32986/0 (socket says 172.21.15.42:32986) 2019-12-05T06:40:51.377 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:51.359+0000 7f3cfb7fe700 1 -- 172.21.15.42:0/1820107076 learned_addr learned my addr 172.21.15.42:0/1820107076 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:40:51.378 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:51.359+0000 7f3cfbfff700 1 --2- 172.21.15.42:0/1820107076 >> v2:172.21.15.40:3300/0 conn(0x7f3cfc125950 0x7f3cfc125d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:40:51.378 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:51.359+0000 7f3cfaffd700 1 -- 172.21.15.42:0/1820107076 >> v2:172.21.15.40:3300/0 conn(0x7f3cfc125950 msgr2=0x7f3cfc125d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:51.378 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:51.359+0000 7f3cfaffd700 1 --2- 172.21.15.42:0/1820107076 >> v2:172.21.15.40:3300/0 conn(0x7f3cfc125950 0x7f3cfc125d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:40:51.378 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:51.359+0000 7f3cfaffd700 1 -- 172.21.15.42:0/1820107076 >> v1:172.21.15.42:6789/0 conn(0x7f3cfc11d6b0 legacy=0x7f3cfc0a6420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:51.378 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:51.359+0000 7f3cfaffd700 1 -- 172.21.15.42:0/1820107076 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cfc1262d0 con 0x7f3cfc07d7c0 2019-12-05T06:40:51.379 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:51.359+0000 7f3ce3fff700 1 -- 172.21.15.42:0/1820107076 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3269175975 0 0) 0x7f3cfc07e140 con 0x7f3cfc11d6b0 2019-12-05T06:40:51.379 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:51.359+0000 7f3cfaffd700 1 --2- 172.21.15.42:0/1820107076 >> v2:172.21.15.42:3300/0 conn(0x7f3cfc07d7c0 0x7f3cfc07dbf0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f3cf0000e90 tx=0x7f3cf0010940).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:51.380 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:51.359+0000 7f3cfaffd700 1 --2- 172.21.15.42:0/1820107076 >> v2:172.21.15.42:3300/0 conn(0x7f3cfc07d7c0 0x7f3cfc07dbf0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f3cf0000e90 tx=0x7f3cf0010940).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:51.380 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:51.359+0000 7f3cfaffd700 1 --2- 172.21.15.42:0/1820107076 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3cfc07d7c0 0x7f3cfc07dbf0 secure :-1 s=READY pgs=3028 cs=0 l=1 rx=0x7f3cf0000e90 tx=0x7f3cf0010940).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:40:51.380 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:51.359+0000 7f3cfaffd700 1 --2- 172.21.15.42:0/1820107076 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3cfc07d7c0 0x7f3cfc07dbf0 secure :-1 s=THROTTLE_DONE pgs=3028 cs=0 l=1 rx=0x7f3cf0000e90 tx=0x7f3cf0010940).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:51.380 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:51.359+0000 7f3cfaffd700 1 --2- 172.21.15.42:0/1820107076 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3cfc07d7c0 0x7f3cfc07dbf0 secure :-1 s=THROTTLE_DONE pgs=3028 cs=0 l=1 rx=0x7f3cf0000e90 tx=0x7f3cf0010940).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:51.380 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:51.359+0000 7f3ce3fff700 1 -- 172.21.15.42:0/1820107076 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3cf00116a0 con 0x7f3cfc07d7c0 2019-12-05T06:40:51.380 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:51.359+0000 7f3cfaffd700 1 --2- 172.21.15.42:0/1820107076 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3cfc07d7c0 0x7f3cfc07dbf0 secure :-1 s=THROTTLE_DONE pgs=3028 cs=0 l=1 rx=0x7f3cf0000e90 tx=0x7f3cf0010940).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:51.381 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:51.359+0000 7f3ce3fff700 1 -- 172.21.15.42:0/1820107076 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cf00112c0 con 0x7f3cfc07d7c0 2019-12-05T06:40:51.381 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:51.359+0000 7f3d017cf700 1 -- 172.21.15.42:0/1820107076 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cfc11c830 con 0x7f3cfc07d7c0 2019-12-05T06:40:51.381 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:51.359+0000 7f3ce3fff700 1 -- 172.21.15.42:0/1820107076 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3cf0011c80 con 0x7f3cfc07d7c0 2019-12-05T06:40:51.381 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:51.359+0000 7f3d017cf700 1 -- 172.21.15.42:0/1820107076 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cfc130360 con 0x7f3cfc07d7c0 2019-12-05T06:40:51.381 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:51.359+0000 7f3cfaffd700 1 --2- 172.21.15.42:0/1820107076 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3cfc07d7c0 0x7f3cfc07dbf0 secure :-1 s=THROTTLE_DONE pgs=3028 cs=0 l=1 rx=0x7f3cf0000e90 tx=0x7f3cf0010940).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:51.381 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:51.359+0000 7f3ce3fff700 1 -- 172.21.15.42:0/1820107076 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f3cf0006460 con 0x7f3cfc07d7c0 2019-12-05T06:40:51.381 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:51.359+0000 7f3cfaffd700 1 --2- 172.21.15.42:0/1820107076 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3cfc07d7c0 0x7f3cfc07dbf0 secure :-1 s=THROTTLE_DONE pgs=3028 cs=0 l=1 rx=0x7f3cf0000e90 tx=0x7f3cf0010940).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:51.382 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:51.359+0000 7f3ce3fff700 1 --2- 172.21.15.42:0/1820107076 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3cdc03ac50 0x7f3cdc03d160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:40:51.382 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:51.359+0000 7f3ce3fff700 1 -- 172.21.15.42:0/1820107076 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 11707+0+0 (secure 0 0 0) 0x7f3cf004ae90 con 0x7f3cfc07d7c0 2019-12-05T06:40:51.382 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:51.359+0000 7f3cfb7fe700 1 --2- 172.21.15.42:0/1820107076 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3cdc03ac50 0x7f3cdc03d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:40:51.382 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:51.363+0000 7f3cfb7fe700 1 --2- 172.21.15.42:0/1820107076 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3cdc03ac50 0x7f3cdc03d160 crc :-1 s=READY pgs=1267 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:40:51.383 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:51.363+0000 7f3d017cf700 1 -- 172.21.15.42:0/1820107076 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cfc05d110 con 0x7f3cfc07d7c0 2019-12-05T06:40:51.383 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:51.367+0000 7f3cfaffd700 1 --2- 172.21.15.42:0/1820107076 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3cfc07d7c0 0x7f3cfc07dbf0 secure :-1 s=THROTTLE_DONE pgs=3028 cs=0 l=1 rx=0x7f3cf0000e90 tx=0x7f3cf0010940).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:51.383 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:51.367+0000 7f3ce3fff700 1 -- 172.21.15.42:0/1820107076 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f3cf0031eb0 con 0x7f3cfc07d7c0 2019-12-05T06:40:51.543 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:51.535+0000 7f3d017cf700 1 -- 172.21.15.42:0/1820107076 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3cfc05b8c0 con 0x7f3cfc07d7c0 2019-12-05T06:40:51.544 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:51.539+0000 7f3cfaffd700 1 --2- 172.21.15.42:0/1820107076 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3cfc07d7c0 0x7f3cfc07dbf0 secure :-1 s=THROTTLE_DONE pgs=3028 cs=0 l=1 rx=0x7f3cf0000e90 tx=0x7f3cf0010940).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:51.544 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:51.539+0000 7f3ce3fff700 1 -- 172.21.15.42:0/1820107076 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 365 v365) v1 ==== 95+0+1776 (secure 0 0 0) 0x7f3cf0032110 con 0x7f3cfc07d7c0 2019-12-05T06:40:51.546 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 365 2019-12-05T06:40:51.547 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:40:51.547 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":365,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":365,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:40:47.519742+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":115,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11976},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11976":{"gid":11976,"name":"b","rank":0,"incarnation":361,"state":"up:active","state_seq":11,"addr":"172.21.15.40:6833/474728597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":474728597},{"type":"v1","addr":"172.21.15.40:6833","nonce":474728597}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:40:51.550 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:51.543+0000 7f3d017cf700 1 -- 172.21.15.42:0/1820107076 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3cdc03ac50 msgr2=0x7f3cdc03d160 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:51.550 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:51.543+0000 7f3d017cf700 1 --2- 172.21.15.42:0/1820107076 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3cdc03ac50 0x7f3cdc03d160 crc :-1 s=READY pgs=1267 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:40:51.551 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:51.543+0000 7f3d017cf700 1 -- 172.21.15.42:0/1820107076 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3cfc07d7c0 msgr2=0x7f3cfc07dbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:51.551 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:51.543+0000 7f3d017cf700 1 --2- 172.21.15.42:0/1820107076 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3cfc07d7c0 0x7f3cfc07dbf0 secure :-1 s=READY pgs=3028 cs=0 l=1 rx=0x7f3cf0000e90 tx=0x7f3cf0010940).stop 2019-12-05T06:40:51.551 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:51.543+0000 7f3d017cf700 1 -- 172.21.15.42:0/1820107076 shutdown_connections 2019-12-05T06:40:51.551 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:51.543+0000 7f3d017cf700 1 --2- 172.21.15.42:0/1820107076 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3cdc03ac50 0x7f3cdc03d160 unknown :-1 s=CLOSED pgs=1267 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:40:51.551 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:51.543+0000 7f3d017cf700 1 --2- 172.21.15.42:0/1820107076 >> v2:172.21.15.40:3300/0 conn(0x7f3cfc125950 0x7f3cfc125d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:40:51.551 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:51.543+0000 7f3d017cf700 1 --2- 172.21.15.42:0/1820107076 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3cfc07d7c0 0x7f3cfc07dbf0 unknown :-1 s=CLOSED pgs=3028 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:40:51.552 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:51.543+0000 7f3d017cf700 1 -- 172.21.15.42:0/1820107076 shutdown_connections 2019-12-05T06:40:51.552 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:51.543+0000 7f3d017cf700 1 -- 172.21.15.42:0/1820107076 wait complete. 2019-12-05T06:40:51.552 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:51.543+0000 7f3d017cf700 1 -- 172.21.15.42:0/1820107076 >> 172.21.15.42:0/1820107076 conn(0x7f3cfc07ee20 msgr2=0x7f3cfc08eb70 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:40:51.565 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2019-12-05T06:40:53.566 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:40:53.677 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:53.663+0000 7fbc195b8700 1 Processor -- start 2019-12-05T06:40:53.677 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:53.663+0000 7fbc195b8700 1 -- start start 2019-12-05T06:40:53.678 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:53.667+0000 7fbc195b8700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fbc1408df80 0x7fbc1409f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:40:53.678 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:53.667+0000 7fbc195b8700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc1409fc60 con 0x7fbc1409f830 2019-12-05T06:40:53.678 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:53.667+0000 7fbc195b8700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc140a0710 con 0x7fbc140a7740 2019-12-05T06:40:53.678 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:53.667+0000 7fbc195b8700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc140a0b00 con 0x7fbc1408df80 2019-12-05T06:40:53.678 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:53.667+0000 7fbc12ffd700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fbc1408df80 0x7fbc1409f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:40:53.678 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:53.667+0000 7fbc137fe700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fbc140a7740 0x7fbc1407e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:33014/0 (socket says 172.21.15.42:33014) 2019-12-05T06:40:53.678 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:53.667+0000 7fbc12ffd700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fbc1408df80 0x7fbc1409f2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:32978/0 (socket says 172.21.15.42:32978) 2019-12-05T06:40:53.679 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:53.667+0000 7fbc137fe700 1 -- 172.21.15.42:0/40210060 learned_addr learned my addr 172.21.15.42:0/40210060 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:40:53.679 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:53.667+0000 7fbc12ffd700 1 -- 172.21.15.42:0/40210060 >> v1:172.21.15.42:6789/0 conn(0x7fbc140a7740 legacy=0x7fbc1407e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:53.679 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:53.667+0000 7fbc12ffd700 1 -- 172.21.15.42:0/40210060 >> v1:172.21.15.40:6789/0 conn(0x7fbc1409f830 legacy=0x7fbc1414efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:53.679 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:53.667+0000 7fbc12ffd700 1 -- 172.21.15.42:0/40210060 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc1407ef20 con 0x7fbc1408df80 2019-12-05T06:40:53.679 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:53.667+0000 7fbc12ffd700 1 --2- 172.21.15.42:0/40210060 >> v2:172.21.15.42:3300/0 conn(0x7fbc1408df80 0x7fbc1409f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fbc08008790 tx=0x7fbc0800fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:53.679 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:53.667+0000 7fbc12ffd700 1 --2- 172.21.15.42:0/40210060 >> v2:172.21.15.42:3300/0 conn(0x7fbc1408df80 0x7fbc1409f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fbc08008790 tx=0x7fbc0800fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:53.679 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:53.667+0000 7fbc12ffd700 1 --2- 172.21.15.42:0/40210060 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbc1408df80 0x7fbc1409f2e0 secure :-1 s=READY pgs=3030 cs=0 l=1 rx=0x7fbc08008790 tx=0x7fbc0800fbf0).ready entity=mon.1 client_cookie=202b3f9eee0f239c server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:40:53.680 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:53.667+0000 7fbc12ffd700 1 --2- 172.21.15.42:0/40210060 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbc1408df80 0x7fbc1409f2e0 secure :-1 s=THROTTLE_DONE pgs=3030 cs=0 l=1 rx=0x7fbc08008790 tx=0x7fbc0800fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:53.680 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:53.667+0000 7fbc127fc700 1 -- 172.21.15.42:0/40210060 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbc08011270 con 0x7fbc1408df80 2019-12-05T06:40:53.680 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:53.667+0000 7fbc12ffd700 1 --2- 172.21.15.42:0/40210060 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbc1408df80 0x7fbc1409f2e0 secure :-1 s=THROTTLE_DONE pgs=3030 cs=0 l=1 rx=0x7fbc08008790 tx=0x7fbc0800fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:53.686 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:53.667+0000 7fbc127fc700 1 -- 172.21.15.42:0/40210060 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc08008bc0 con 0x7fbc1408df80 2019-12-05T06:40:53.686 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:53.667+0000 7fbc12ffd700 1 --2- 172.21.15.42:0/40210060 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbc1408df80 0x7fbc1409f2e0 secure :-1 s=THROTTLE_DONE pgs=3030 cs=0 l=1 rx=0x7fbc08008790 tx=0x7fbc0800fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:53.686 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:53.667+0000 7fbc127fc700 1 -- 172.21.15.42:0/40210060 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbc08011850 con 0x7fbc1408df80 2019-12-05T06:40:53.686 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:53.667+0000 7fbc195b8700 1 -- 172.21.15.42:0/40210060 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbc1408df80 msgr2=0x7fbc1409f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:53.687 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:53.667+0000 7fbc195b8700 1 --2- 172.21.15.42:0/40210060 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbc1408df80 0x7fbc1409f2e0 secure :-1 s=READY pgs=3030 cs=0 l=1 rx=0x7fbc08008790 tx=0x7fbc0800fbf0).stop 2019-12-05T06:40:53.687 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:53.671+0000 7fbc195b8700 1 -- 172.21.15.42:0/40210060 shutdown_connections 2019-12-05T06:40:53.687 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:53.671+0000 7fbc195b8700 1 --2- 172.21.15.42:0/40210060 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbc1408df80 0x7fbc1409f2e0 unknown :-1 s=CLOSED pgs=3030 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:40:53.687 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:53.671+0000 7fbc195b8700 1 -- 172.21.15.42:0/40210060 shutdown_connections 2019-12-05T06:40:53.687 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:53.671+0000 7fbc195b8700 1 -- 172.21.15.42:0/40210060 wait complete. 2019-12-05T06:40:53.687 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:53.671+0000 7fbc195b8700 1 -- 172.21.15.42:0/40210060 >> 172.21.15.42:0/40210060 conn(0x7fbc140aabe0 msgr2=0x7fbc140a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:40:53.692 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:53.671+0000 7fbc195b8700 1 Processor -- start 2019-12-05T06:40:53.692 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:53.671+0000 7fbc195b8700 1 -- start start 2019-12-05T06:40:53.692 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:53.675+0000 7fbc195b8700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fbc14146ff0 0x7fbc14142a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:40:53.693 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:53.675+0000 7fbc195b8700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbc14125a90 0x7fbc14125ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:40:53.693 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:53.675+0000 7fbc195b8700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc14126410 con 0x7fbc14118af0 2019-12-05T06:40:53.693 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:53.675+0000 7fbc195b8700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc14142f90 con 0x7fbc14146ff0 2019-12-05T06:40:53.693 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:53.675+0000 7fbc195b8700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc14147610 con 0x7fbc14125a90 2019-12-05T06:40:53.693 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:53.675+0000 7fbc137fe700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fbc14118af0 0x7fbc14077110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:33020/0 (socket says 172.21.15.42:33020) 2019-12-05T06:40:53.694 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:53.675+0000 7fbc137fe700 1 -- 172.21.15.42:0/3358138319 learned_addr learned my addr 172.21.15.42:0/3358138319 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:40:53.694 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:53.675+0000 7fbc12ffd700 1 --2- 172.21.15.42:0/3358138319 >> v2:172.21.15.40:3300/0 conn(0x7fbc14146ff0 0x7fbc14142a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:40:53.694 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:53.675+0000 7fbc13fff700 1 --2- 172.21.15.42:0/3358138319 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbc14125a90 0x7fbc14125ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:40:53.694 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:53.675+0000 7fbbfbfff700 1 -- 172.21.15.42:0/3358138319 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1258275561 0 0) 0x7fbc14126410 con 0x7fbc14118af0 2019-12-05T06:40:53.694 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:53.675+0000 7fbbfbfff700 1 -- 172.21.15.42:0/3358138319 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbbf40030b0 con 0x7fbc14118af0 2019-12-05T06:40:53.695 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:53.675+0000 7fbc13fff700 1 -- 172.21.15.42:0/3358138319 >> v2:172.21.15.40:3300/0 conn(0x7fbc14146ff0 msgr2=0x7fbc14142a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:53.695 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:53.675+0000 7fbc13fff700 1 --2- 172.21.15.42:0/3358138319 >> v2:172.21.15.40:3300/0 conn(0x7fbc14146ff0 0x7fbc14142a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:40:53.695 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:53.675+0000 7fbc13fff700 1 -- 172.21.15.42:0/3358138319 >> v1:172.21.15.42:6789/0 conn(0x7fbc14118af0 legacy=0x7fbc14077110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:53.695 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:53.675+0000 7fbc13fff700 1 -- 172.21.15.42:0/3358138319 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc1408eb70 con 0x7fbc14125a90 2019-12-05T06:40:53.695 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:53.675+0000 7fbc13fff700 1 --2- 172.21.15.42:0/3358138319 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbc14125a90 0x7fbc14125ec0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fbc0c001210 tx=0x7fbc0c010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:53.696 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:53.675+0000 7fbc13fff700 1 --2- 172.21.15.42:0/3358138319 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbc14125a90 0x7fbc14125ec0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fbc0c001210 tx=0x7fbc0c010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:53.696 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:53.675+0000 7fbc13fff700 1 --2- 172.21.15.42:0/3358138319 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbc14125a90 0x7fbc14125ec0 secure :-1 s=READY pgs=660 cs=0 l=1 rx=0x7fbc0c001210 tx=0x7fbc0c010b40).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:40:53.696 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:53.675+0000 7fbc13fff700 1 --2- 172.21.15.42:0/3358138319 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbc14125a90 0x7fbc14125ec0 secure :-1 s=THROTTLE_DONE pgs=660 cs=0 l=1 rx=0x7fbc0c001210 tx=0x7fbc0c010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:53.697 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:53.675+0000 7fbbfbfff700 1 -- 172.21.15.42:0/3358138319 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbc0c006dc0 con 0x7fbc14125a90 2019-12-05T06:40:53.697 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:53.675+0000 7fbc13fff700 1 --2- 172.21.15.42:0/3358138319 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbc14125a90 0x7fbc14125ec0 secure :-1 s=THROTTLE_DONE pgs=660 cs=0 l=1 rx=0x7fbc0c001210 tx=0x7fbc0c010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:53.697 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:53.675+0000 7fbc195b8700 1 -- 172.21.15.42:0/3358138319 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc1408ee40 con 0x7fbc14125a90 2019-12-05T06:40:53.697 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:53.675+0000 7fbbfbfff700 1 -- 172.21.15.42:0/3358138319 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc0c0069e0 con 0x7fbc14125a90 2019-12-05T06:40:53.697 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:53.675+0000 7fbc13fff700 1 --2- 172.21.15.42:0/3358138319 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbc14125a90 0x7fbc14125ec0 secure :-1 s=THROTTLE_DONE pgs=660 cs=0 l=1 rx=0x7fbc0c001210 tx=0x7fbc0c010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:53.698 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:53.675+0000 7fbbfbfff700 1 -- 172.21.15.42:0/3358138319 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbc0c002620 con 0x7fbc14125a90 2019-12-05T06:40:53.698 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:53.675+0000 7fbc195b8700 1 -- 172.21.15.42:0/3358138319 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc1408f370 con 0x7fbc14125a90 2019-12-05T06:40:53.698 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:53.675+0000 7fbc13fff700 1 --2- 172.21.15.42:0/3358138319 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbc14125a90 0x7fbc14125ec0 secure :-1 s=THROTTLE_DONE pgs=660 cs=0 l=1 rx=0x7fbc0c001210 tx=0x7fbc0c010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:53.698 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:53.675+0000 7fbbfbfff700 1 -- 172.21.15.42:0/3358138319 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fbc0c002900 con 0x7fbc14125a90 2019-12-05T06:40:53.698 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:53.675+0000 7fbc13fff700 1 --2- 172.21.15.42:0/3358138319 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbc14125a90 0x7fbc14125ec0 secure :-1 s=THROTTLE_DONE pgs=660 cs=0 l=1 rx=0x7fbc0c001210 tx=0x7fbc0c010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:53.698 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:53.679+0000 7fbbfbfff700 1 --2- 172.21.15.42:0/3358138319 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbbf403b2a0 0x7fbbf403d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:40:53.699 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:53.679+0000 7fbbfbfff700 1 -- 172.21.15.42:0/3358138319 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 11707+0+0 (secure 0 0 0) 0x7fbc0c04b4b0 con 0x7fbc14125a90 2019-12-05T06:40:53.699 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:53.679+0000 7fbc137fe700 1 --2- 172.21.15.42:0/3358138319 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbbf403b2a0 0x7fbbf403d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:40:53.699 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:53.679+0000 7fbc137fe700 1 --2- 172.21.15.42:0/3358138319 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbbf403b2a0 0x7fbbf403d7b0 crc :-1 s=READY pgs=1269 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:40:53.699 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:53.679+0000 7fbc195b8700 1 -- 172.21.15.42:0/3358138319 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc1405d110 con 0x7fbc14125a90 2019-12-05T06:40:53.700 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:53.691+0000 7fbc13fff700 1 --2- 172.21.15.42:0/3358138319 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbc14125a90 0x7fbc14125ec0 secure :-1 s=THROTTLE_DONE pgs=660 cs=0 l=1 rx=0x7fbc0c001210 tx=0x7fbc0c010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:53.700 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:53.691+0000 7fbbfbfff700 1 -- 172.21.15.42:0/3358138319 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fbc0c032510 con 0x7fbc14125a90 2019-12-05T06:40:53.869 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:53.859+0000 7fbc195b8700 1 -- 172.21.15.42:0/3358138319 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbc14078480 con 0x7fbc14125a90 2019-12-05T06:40:53.869 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:53.863+0000 7fbc13fff700 1 --2- 172.21.15.42:0/3358138319 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbc14125a90 0x7fbc14125ec0 secure :-1 s=THROTTLE_DONE pgs=660 cs=0 l=1 rx=0x7fbc0c001210 tx=0x7fbc0c010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:53.869 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:53.863+0000 7fbbfbfff700 1 -- 172.21.15.42:0/3358138319 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 366 v366) v1 ==== 95+0+2114 (secure 0 0 0) 0x7fbc0c032770 con 0x7fbc14125a90 2019-12-05T06:40:53.870 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 366 2019-12-05T06:40:53.870 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:40:53.871 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":366,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":12358,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/1922398668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1922398668},{"type":"v1","addr":"172.21.15.42:6835","nonce":1922398668}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":366}],"filesystems":[{"mdsmap":{"epoch":365,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:40:47.519742+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":115,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11976},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11976":{"gid":11976,"name":"b","rank":0,"incarnation":361,"state":"up:active","state_seq":11,"addr":"172.21.15.40:6833/474728597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":474728597},{"type":"v1","addr":"172.21.15.40:6833","nonce":474728597}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:40:53.875 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:53.867+0000 7fbc195b8700 1 -- 172.21.15.42:0/3358138319 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbbf403b2a0 msgr2=0x7fbbf403d7b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:53.876 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:53.867+0000 7fbc195b8700 1 --2- 172.21.15.42:0/3358138319 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbbf403b2a0 0x7fbbf403d7b0 crc :-1 s=READY pgs=1269 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:40:53.876 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:53.867+0000 7fbc195b8700 1 -- 172.21.15.42:0/3358138319 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbc14125a90 msgr2=0x7fbc14125ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:53.876 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:53.867+0000 7fbc195b8700 1 --2- 172.21.15.42:0/3358138319 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbc14125a90 0x7fbc14125ec0 secure :-1 s=READY pgs=660 cs=0 l=1 rx=0x7fbc0c001210 tx=0x7fbc0c010b40).stop 2019-12-05T06:40:53.876 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:53.867+0000 7fbc195b8700 1 -- 172.21.15.42:0/3358138319 shutdown_connections 2019-12-05T06:40:53.877 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:53.867+0000 7fbc195b8700 1 --2- 172.21.15.42:0/3358138319 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbbf403b2a0 0x7fbbf403d7b0 unknown :-1 s=CLOSED pgs=1269 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:40:53.877 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:53.867+0000 7fbc195b8700 1 --2- 172.21.15.42:0/3358138319 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbc14125a90 0x7fbc14125ec0 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:40:53.877 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:53.867+0000 7fbc195b8700 1 --2- 172.21.15.42:0/3358138319 >> v2:172.21.15.40:3300/0 conn(0x7fbc14146ff0 0x7fbc14142a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:40:53.877 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:53.867+0000 7fbc195b8700 1 -- 172.21.15.42:0/3358138319 shutdown_connections 2019-12-05T06:40:53.877 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:53.867+0000 7fbc195b8700 1 -- 172.21.15.42:0/3358138319 wait complete. 2019-12-05T06:40:53.878 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:40:53.867+0000 7fbc195b8700 1 -- 172.21.15.42:0/3358138319 >> 172.21.15.42:0/3358138319 conn(0x7fbc1408fda0 msgr2=0x7fbc1407e4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:40:53.885 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a reported in up:standby state 2019-12-05T06:40:53.886 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 96 secs before thrashing 2019-12-05T06:41:15.101 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:41:15.105 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:41:45.174 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:41:45.179 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:42:15.235 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:42:15.238 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:42:29.886 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:42:30.419 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.406+0000 7fd2686f3700 1 Processor -- start 2019-12-05T06:42:30.419 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.406+0000 7fd2686f3700 1 -- start start 2019-12-05T06:42:30.419 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.406+0000 7fd2686f3700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd2600a7740 0x7fd26008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:42:30.419 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.406+0000 7fd2686f3700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd26009fae0 0x7fd26007eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:42:30.420 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.406+0000 7fd2686f3700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2600a0710 con 0x7fd26007e720 2019-12-05T06:42:30.420 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.406+0000 7fd2686f3700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2600a0b00 con 0x7fd26009fae0 2019-12-05T06:42:30.420 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.406+0000 7fd2686f3700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd26008e5b0 con 0x7fd2600a7740 2019-12-05T06:42:30.420 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.406+0000 7fd267491700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd26009fae0 0x7fd26007eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:42:30.420 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.406+0000 7fd267491700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd26009fae0 0x7fd26007eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:33646/0 (socket says 172.21.15.42:33646) 2019-12-05T06:42:30.420 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.406+0000 7fd267491700 1 -- 172.21.15.42:0/2543718927 learned_addr learned my addr 172.21.15.42:0/2543718927 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:42:30.421 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.406+0000 7fd26648f700 1 --1- 172.21.15.42:0/2543718927 >> v1:172.21.15.42:6789/0 conn(0x7fd26007e720 0x7fd26009f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:33682/0 (socket says 172.21.15.42:33682) 2019-12-05T06:42:30.421 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.406+0000 7fd266c90700 1 --2- 172.21.15.42:0/2543718927 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd2600a7740 0x7fd26008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:42:30.421 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.406+0000 7fd267491700 1 -- 172.21.15.42:0/2543718927 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd2600a7740 msgr2=0x7fd26008df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:42:30.421 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.406+0000 7fd267491700 1 --2- 172.21.15.42:0/2543718927 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd2600a7740 0x7fd26008df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:42:30.421 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.406+0000 7fd267491700 1 -- 172.21.15.42:0/2543718927 >> v1:172.21.15.42:6789/0 conn(0x7fd26007e720 legacy=0x7fd26009f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:42:30.422 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.406+0000 7fd267491700 1 -- 172.21.15.42:0/2543718927 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd26007f0a0 con 0x7fd26009fae0 2019-12-05T06:42:30.422 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.406+0000 7fd267491700 1 --2- 172.21.15.42:0/2543718927 >> v2:172.21.15.42:3300/0 conn(0x7fd26009fae0 0x7fd26007eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fd258008b10 tx=0x7fd2580100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:30.422 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.406+0000 7fd267491700 1 --2- 172.21.15.42:0/2543718927 >> v2:172.21.15.42:3300/0 conn(0x7fd26009fae0 0x7fd26007eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fd258008b10 tx=0x7fd2580100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:30.422 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.406+0000 7fd267491700 1 --2- 172.21.15.42:0/2543718927 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd26009fae0 0x7fd26007eb50 secure :-1 s=READY pgs=3036 cs=0 l=1 rx=0x7fd258008b10 tx=0x7fd2580100b0).ready entity=mon.1 client_cookie=71734ed1ccabfb52 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:42:30.422 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.406+0000 7fd267491700 1 --2- 172.21.15.42:0/2543718927 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd26009fae0 0x7fd26007eb50 secure :-1 s=THROTTLE_DONE pgs=3036 cs=0 l=1 rx=0x7fd258008b10 tx=0x7fd2580100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:30.422 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.406+0000 7fd265c8e700 1 -- 172.21.15.42:0/2543718927 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd258010da0 con 0x7fd26009fae0 2019-12-05T06:42:30.423 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.406+0000 7fd267491700 1 --2- 172.21.15.42:0/2543718927 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd26009fae0 0x7fd26007eb50 secure :-1 s=THROTTLE_DONE pgs=3036 cs=0 l=1 rx=0x7fd258008b10 tx=0x7fd2580100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:30.423 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.410+0000 7fd267491700 1 --2- 172.21.15.42:0/2543718927 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd26009fae0 0x7fd26007eb50 secure :-1 s=THROTTLE_DONE pgs=3036 cs=0 l=1 rx=0x7fd258008b10 tx=0x7fd2580100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:30.423 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.410+0000 7fd265c8e700 1 -- 172.21.15.42:0/2543718927 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2580109c0 con 0x7fd26009fae0 2019-12-05T06:42:30.423 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.410+0000 7fd265c8e700 1 -- 172.21.15.42:0/2543718927 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd2580114a0 con 0x7fd26009fae0 2019-12-05T06:42:30.423 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.410+0000 7fd2686f3700 1 -- 172.21.15.42:0/2543718927 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd26009fae0 msgr2=0x7fd26007eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:42:30.423 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.410+0000 7fd2686f3700 1 --2- 172.21.15.42:0/2543718927 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd26009fae0 0x7fd26007eb50 secure :-1 s=READY pgs=3036 cs=0 l=1 rx=0x7fd258008b10 tx=0x7fd2580100b0).stop 2019-12-05T06:42:30.423 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.410+0000 7fd2686f3700 1 -- 172.21.15.42:0/2543718927 shutdown_connections 2019-12-05T06:42:30.424 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.410+0000 7fd2686f3700 1 --2- 172.21.15.42:0/2543718927 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd26009fae0 0x7fd26007eb50 unknown :-1 s=CLOSED pgs=3036 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:42:30.424 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.410+0000 7fd2686f3700 1 --2- 172.21.15.42:0/2543718927 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd2600a7740 0x7fd26008df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:42:30.424 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.410+0000 7fd2686f3700 1 -- 172.21.15.42:0/2543718927 shutdown_connections 2019-12-05T06:42:30.424 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.410+0000 7fd2686f3700 1 -- 172.21.15.42:0/2543718927 wait complete. 2019-12-05T06:42:30.424 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.410+0000 7fd2686f3700 1 -- 172.21.15.42:0/2543718927 >> 172.21.15.42:0/2543718927 conn(0x7fd2600aabe0 msgr2=0x7fd2600a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:42:30.429 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.414+0000 7fd2686f3700 1 Processor -- start 2019-12-05T06:42:30.429 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.414+0000 7fd2686f3700 1 -- start start 2019-12-05T06:42:30.430 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.414+0000 7fd2686f3700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd260116500 0x7fd2601175a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:42:30.430 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.414+0000 7fd2686f3700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd260122a80 0x7fd260122eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:42:30.430 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.414+0000 7fd2686f3700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd26011e930 con 0x7fd26008eb70 2019-12-05T06:42:30.430 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.414+0000 7fd2686f3700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd260125450 con 0x7fd260122a80 2019-12-05T06:42:30.430 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.414+0000 7fd2686f3700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd260117f70 con 0x7fd260116500 2019-12-05T06:42:30.430 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.414+0000 7fd266c90700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd260116500 0x7fd2601175a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:42:30.431 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.414+0000 7fd266c90700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd260116500 0x7fd2601175a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:34114/0 (socket says 172.21.15.42:34114) 2019-12-05T06:42:30.431 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.414+0000 7fd266c90700 1 -- 172.21.15.42:0/4076414804 learned_addr learned my addr 172.21.15.42:0/4076414804 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:42:30.431 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.414+0000 7fd26648f700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd260122a80 0x7fd260122eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:42:30.431 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.414+0000 7fd267491700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fd26008eb70 0x7fd26008efa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:43128/0 (socket says 172.21.15.42:43128) 2019-12-05T06:42:30.431 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.414+0000 7fd2577fe700 1 -- 172.21.15.42:0/4076414804 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3112382984 0 0) 0x7fd26011e930 con 0x7fd26008eb70 2019-12-05T06:42:30.431 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.414+0000 7fd266c90700 1 -- 172.21.15.42:0/4076414804 >> v2:172.21.15.40:3300/0 conn(0x7fd260122a80 msgr2=0x7fd260122eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:42:30.431 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.414+0000 7fd266c90700 1 --2- 172.21.15.42:0/4076414804 >> v2:172.21.15.40:3300/0 conn(0x7fd260122a80 0x7fd260122eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:42:30.432 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.414+0000 7fd266c90700 1 -- 172.21.15.42:0/4076414804 >> v1:172.21.15.40:6789/0 conn(0x7fd26008eb70 legacy=0x7fd26008efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:42:30.432 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.414+0000 7fd266c90700 1 -- 172.21.15.42:0/4076414804 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd260124190 con 0x7fd260116500 2019-12-05T06:42:30.432 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.414+0000 7fd266c90700 1 --2- 172.21.15.42:0/4076414804 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd260116500 0x7fd2601175a0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fd25c003070 tx=0x7fd25c010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:30.432 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.414+0000 7fd266c90700 1 --2- 172.21.15.42:0/4076414804 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd260116500 0x7fd2601175a0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fd25c003070 tx=0x7fd25c010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:30.432 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.414+0000 7fd266c90700 1 --2- 172.21.15.42:0/4076414804 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd260116500 0x7fd2601175a0 secure :-1 s=READY pgs=663 cs=0 l=1 rx=0x7fd25c003070 tx=0x7fd25c010b60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:42:30.432 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.414+0000 7fd266c90700 1 --2- 172.21.15.42:0/4076414804 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd260116500 0x7fd2601175a0 secure :-1 s=THROTTLE_DONE pgs=663 cs=0 l=1 rx=0x7fd25c003070 tx=0x7fd25c010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:30.432 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.414+0000 7fd2577fe700 1 -- 172.21.15.42:0/4076414804 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd25c005540 con 0x7fd260116500 2019-12-05T06:42:30.433 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.414+0000 7fd266c90700 1 --2- 172.21.15.42:0/4076414804 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd260116500 0x7fd2601175a0 secure :-1 s=THROTTLE_DONE pgs=663 cs=0 l=1 rx=0x7fd25c003070 tx=0x7fd25c010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:30.433 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.414+0000 7fd2577fe700 1 -- 172.21.15.42:0/4076414804 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd25c007150 con 0x7fd260116500 2019-12-05T06:42:30.433 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.414+0000 7fd266c90700 1 --2- 172.21.15.42:0/4076414804 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd260116500 0x7fd2601175a0 secure :-1 s=THROTTLE_DONE pgs=663 cs=0 l=1 rx=0x7fd25c003070 tx=0x7fd25c010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:30.433 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.414+0000 7fd2686f3700 1 -- 172.21.15.42:0/4076414804 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd260117af0 con 0x7fd260116500 2019-12-05T06:42:30.433 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.414+0000 7fd2577fe700 1 -- 172.21.15.42:0/4076414804 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd25c005b20 con 0x7fd260116500 2019-12-05T06:42:30.433 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.414+0000 7fd2686f3700 1 -- 172.21.15.42:0/4076414804 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2601235a0 con 0x7fd260116500 2019-12-05T06:42:30.433 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.414+0000 7fd266c90700 1 --2- 172.21.15.42:0/4076414804 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd260116500 0x7fd2601175a0 secure :-1 s=THROTTLE_DONE pgs=663 cs=0 l=1 rx=0x7fd25c003070 tx=0x7fd25c010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:30.433 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.418+0000 7fd2577fe700 1 -- 172.21.15.42:0/4076414804 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fd25c0123c0 con 0x7fd260116500 2019-12-05T06:42:30.434 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.418+0000 7fd266c90700 1 --2- 172.21.15.42:0/4076414804 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd260116500 0x7fd2601175a0 secure :-1 s=THROTTLE_DONE pgs=663 cs=0 l=1 rx=0x7fd25c003070 tx=0x7fd25c010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:30.434 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.418+0000 7fd2577fe700 1 --2- 172.21.15.42:0/4076414804 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd24803acd0 0x7fd24803d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:42:30.434 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.418+0000 7fd2577fe700 1 -- 172.21.15.42:0/4076414804 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 11707+0+0 (secure 0 0 0) 0x7fd25c04af00 con 0x7fd260116500 2019-12-05T06:42:30.434 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.418+0000 7fd26648f700 1 --2- 172.21.15.42:0/4076414804 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd24803acd0 0x7fd24803d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:42:30.434 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.418+0000 7fd26648f700 1 --2- 172.21.15.42:0/4076414804 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd24803acd0 0x7fd24803d1e0 crc :-1 s=READY pgs=1272 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:42:30.434 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.422+0000 7fd2686f3700 1 -- 172.21.15.42:0/4076414804 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd26005d110 con 0x7fd260116500 2019-12-05T06:42:30.436 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.430+0000 7fd266c90700 1 --2- 172.21.15.42:0/4076414804 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd260116500 0x7fd2601175a0 secure :-1 s=THROTTLE_DONE pgs=663 cs=0 l=1 rx=0x7fd25c003070 tx=0x7fd25c010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:30.436 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.430+0000 7fd2577fe700 1 -- 172.21.15.42:0/4076414804 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fd25c005dc0 con 0x7fd260116500 2019-12-05T06:42:30.643 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.634+0000 7fd2686f3700 1 -- 172.21.15.42:0/4076414804 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd26005b8c0 con 0x7fd260116500 2019-12-05T06:42:30.645 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.638+0000 7fd266c90700 1 --2- 172.21.15.42:0/4076414804 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd260116500 0x7fd2601175a0 secure :-1 s=THROTTLE_DONE pgs=663 cs=0 l=1 rx=0x7fd25c003070 tx=0x7fd25c010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:30.645 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.638+0000 7fd2577fe700 1 -- 172.21.15.42:0/4076414804 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 367 v367) v1 ==== 95+0+2114 (secure 0 0 0) 0x7fd25c032120 con 0x7fd260116500 2019-12-05T06:42:30.651 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 367 2019-12-05T06:42:30.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.642+0000 7fd2686f3700 1 -- 172.21.15.42:0/4076414804 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd24803acd0 msgr2=0x7fd24803d1e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:42:30.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.642+0000 7fd2686f3700 1 --2- 172.21.15.42:0/4076414804 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd24803acd0 0x7fd24803d1e0 crc :-1 s=READY pgs=1272 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:42:30.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.642+0000 7fd2686f3700 1 -- 172.21.15.42:0/4076414804 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd260116500 msgr2=0x7fd2601175a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:42:30.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.642+0000 7fd2686f3700 1 --2- 172.21.15.42:0/4076414804 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd260116500 0x7fd2601175a0 secure :-1 s=READY pgs=663 cs=0 l=1 rx=0x7fd25c003070 tx=0x7fd25c010b60).stop 2019-12-05T06:42:30.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.642+0000 7fd2686f3700 1 -- 172.21.15.42:0/4076414804 shutdown_connections 2019-12-05T06:42:30.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.642+0000 7fd2686f3700 1 --2- 172.21.15.42:0/4076414804 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd24803acd0 0x7fd24803d1e0 unknown :-1 s=CLOSED pgs=1272 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:42:30.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.642+0000 7fd2686f3700 1 --2- 172.21.15.42:0/4076414804 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd260116500 0x7fd2601175a0 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:42:30.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.642+0000 7fd2686f3700 1 --2- 172.21.15.42:0/4076414804 >> v2:172.21.15.40:3300/0 conn(0x7fd260122a80 0x7fd260122eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:42:30.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.642+0000 7fd2686f3700 1 -- 172.21.15.42:0/4076414804 shutdown_connections 2019-12-05T06:42:30.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.642+0000 7fd2686f3700 1 -- 172.21.15.42:0/4076414804 wait complete. 2019-12-05T06:42:30.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.642+0000 7fd2686f3700 1 -- 172.21.15.42:0/4076414804 >> 172.21.15.42:0/4076414804 conn(0x7fd2600a98b0 msgr2=0x7fd26011db90 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:42:30.653 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:42:30.653 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":367,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":12358,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/1922398668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1922398668},{"type":"v1","addr":"172.21.15.42:6835","nonce":1922398668}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":366}],"filesystems":[{"mdsmap":{"epoch":367,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:41:51.221717+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":115,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11976},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11976":{"gid":11976,"name":"b","rank":0,"incarnation":361,"state":"up:active","state_seq":27,"addr":"172.21.15.40:6833/474728597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":474728597},{"type":"v1","addr":"172.21.15.40:6833","nonce":474728597}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:42:30.663 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.b (rank=0) 2019-12-05T06:42:30.663 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2019-12-05T06:42:30.663 INFO:teuthology.orchestra.run:waiting for 300 2019-12-05T06:42:30.680 INFO:tasks.ceph.mds.b:Stopped 2019-12-05T06:42:30.680 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:42:30.779 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.766+0000 7f677defa700 1 Processor -- start 2019-12-05T06:42:30.779 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.766+0000 7f677defa700 1 -- start start 2019-12-05T06:42:30.780 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.766+0000 7f677defa700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f67780a7740 0x7f677808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:42:30.780 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.766+0000 7f677defa700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f677807e720 0x7f677807eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:42:30.780 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.766+0000 7f677defa700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67780a0710 con 0x7f677809f2e0 2019-12-05T06:42:30.780 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.766+0000 7f677defa700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67780a0b00 con 0x7f67780a7740 2019-12-05T06:42:30.780 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.766+0000 7f677defa700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f677808e5b0 con 0x7f677807e720 2019-12-05T06:42:30.781 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.766+0000 7f677cc98700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f677809f2e0 0x7f677809f710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:33702/0 (socket says 172.21.15.42:33702) 2019-12-05T06:42:30.781 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.766+0000 7f677cc98700 1 -- 172.21.15.42:0/3547362893 learned_addr learned my addr 172.21.15.42:0/3547362893 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:42:30.781 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.766+0000 7f6777fff700 1 --2- 172.21.15.42:0/3547362893 >> v2:172.21.15.40:3300/0 conn(0x7f67780a7740 0x7f677808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:42:30.781 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.766+0000 7f67777fe700 1 --2- 172.21.15.42:0/3547362893 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f677807e720 0x7f677807eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:42:30.781 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.770+0000 7f6776ffd700 1 -- 172.21.15.42:0/3547362893 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1657365813 0 0) 0x7f67780a0710 con 0x7f677809f2e0 2019-12-05T06:42:30.781 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.770+0000 7f6776ffd700 1 -- 172.21.15.42:0/3547362893 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67600030b0 con 0x7f677809f2e0 2019-12-05T06:42:30.781 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.770+0000 7f6777fff700 1 -- 172.21.15.42:0/3547362893 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f677807e720 msgr2=0x7f677807eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:42:30.782 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.770+0000 7f6777fff700 1 --2- 172.21.15.42:0/3547362893 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f677807e720 0x7f677807eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:42:30.782 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.770+0000 7f6777fff700 1 -- 172.21.15.42:0/3547362893 >> v1:172.21.15.42:6789/0 conn(0x7f677809f2e0 legacy=0x7f677809f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:42:30.782 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.770+0000 7f6777fff700 1 -- 172.21.15.42:0/3547362893 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f677807f0a0 con 0x7f67780a7740 2019-12-05T06:42:30.782 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.770+0000 7f6777fff700 1 --2- 172.21.15.42:0/3547362893 >> v2:172.21.15.40:3300/0 conn(0x7f67780a7740 0x7f677808df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f6768008790 tx=0x7f676800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:30.782 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.770+0000 7f6777fff700 1 --2- 172.21.15.42:0/3547362893 >> v2:172.21.15.40:3300/0 conn(0x7f67780a7740 0x7f677808df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f6768008790 tx=0x7f676800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:30.782 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.770+0000 7f6777fff700 1 --2- 172.21.15.42:0/3547362893 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f67780a7740 0x7f677808df80 secure :-1 s=READY pgs=2303 cs=0 l=1 rx=0x7f6768008790 tx=0x7f676800fc70).ready entity=mon.0 client_cookie=47095b7052800dd6 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:42:30.782 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.770+0000 7f6777fff700 1 --2- 172.21.15.42:0/3547362893 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f67780a7740 0x7f677808df80 secure :-1 s=THROTTLE_DONE pgs=2303 cs=0 l=1 rx=0x7f6768008790 tx=0x7f676800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:30.783 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.770+0000 7f6777fff700 1 --2- 172.21.15.42:0/3547362893 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f67780a7740 0x7f677808df80 secure :-1 s=THROTTLE_DONE pgs=2303 cs=0 l=1 rx=0x7f6768008790 tx=0x7f676800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:30.783 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.770+0000 7f6776ffd700 1 -- 172.21.15.42:0/3547362893 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f67680112f0 con 0x7f67780a7740 2019-12-05T06:42:30.783 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.770+0000 7f6777fff700 1 --2- 172.21.15.42:0/3547362893 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f67780a7740 0x7f677808df80 secure :-1 s=THROTTLE_DONE pgs=2303 cs=0 l=1 rx=0x7f6768008790 tx=0x7f676800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:30.783 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.770+0000 7f6776ffd700 1 -- 172.21.15.42:0/3547362893 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6768008ad0 con 0x7f67780a7740 2019-12-05T06:42:30.783 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.770+0000 7f6776ffd700 1 -- 172.21.15.42:0/3547362893 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f67680118d0 con 0x7f67780a7740 2019-12-05T06:42:30.783 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.770+0000 7f677defa700 1 -- 172.21.15.42:0/3547362893 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f67780a7740 msgr2=0x7f677808df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:42:30.783 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.770+0000 7f677defa700 1 --2- 172.21.15.42:0/3547362893 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f67780a7740 0x7f677808df80 secure :-1 s=READY pgs=2303 cs=0 l=1 rx=0x7f6768008790 tx=0x7f676800fc70).stop 2019-12-05T06:42:30.784 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.770+0000 7f677defa700 1 -- 172.21.15.42:0/3547362893 shutdown_connections 2019-12-05T06:42:30.784 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.770+0000 7f677defa700 1 --2- 172.21.15.42:0/3547362893 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f67780a7740 0x7f677808df80 unknown :-1 s=CLOSED pgs=2303 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:42:30.784 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.770+0000 7f677defa700 1 --2- 172.21.15.42:0/3547362893 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f677807e720 0x7f677807eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:42:30.784 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.770+0000 7f677defa700 1 -- 172.21.15.42:0/3547362893 shutdown_connections 2019-12-05T06:42:30.784 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.770+0000 7f677defa700 1 -- 172.21.15.42:0/3547362893 wait complete. 2019-12-05T06:42:30.784 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.770+0000 7f677defa700 1 -- 172.21.15.42:0/3547362893 >> 172.21.15.42:0/3547362893 conn(0x7f67780aabe0 msgr2=0x7f67780a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:42:30.788 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.774+0000 7f677defa700 1 Processor -- start 2019-12-05T06:42:30.788 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.774+0000 7f677defa700 1 -- start start 2019-12-05T06:42:30.789 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.774+0000 7f677defa700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f6778119c00 0x7f677811b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:42:30.789 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.774+0000 7f677defa700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67781173e0 con 0x7f677811bc70 2019-12-05T06:42:30.789 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.774+0000 7f677defa700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6778118470 con 0x7f6778116500 2019-12-05T06:42:30.789 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.774+0000 7f677defa700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67781187b0 con 0x7f6778119c00 2019-12-05T06:42:30.789 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.774+0000 7f6777fff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f6778116500 0x7f677811d9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:33704/0 (socket says 172.21.15.42:33704) 2019-12-05T06:42:30.789 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.774+0000 7f6777fff700 1 -- 172.21.15.42:0/1327756981 learned_addr learned my addr 172.21.15.42:0/1327756981 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:42:30.789 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.774+0000 7f67777fe700 1 --2- 172.21.15.42:0/1327756981 >> v2:172.21.15.40:3300/0 conn(0x7f6778119c00 0x7f677811b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:42:30.790 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.774+0000 7f6774ff9700 1 -- 172.21.15.42:0/1327756981 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3207142575 0 0) 0x7f6778118470 con 0x7f6778116500 2019-12-05T06:42:30.790 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.774+0000 7f6774ff9700 1 -- 172.21.15.42:0/1327756981 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67580030b0 con 0x7f6778116500 2019-12-05T06:42:30.790 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.774+0000 7f67777fe700 1 -- 172.21.15.42:0/1327756981 >> v1:172.21.15.42:6789/0 conn(0x7f6778116500 legacy=0x7f677811d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:42:30.790 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.774+0000 7f6774ff9700 1 -- 172.21.15.42:0/1327756981 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1401884819 0 0) 0x7f67781173e0 con 0x7f677811bc70 2019-12-05T06:42:30.790 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.774+0000 7f67777fe700 1 -- 172.21.15.42:0/1327756981 >> v1:172.21.15.40:6789/0 conn(0x7f677811bc70 legacy=0x7f6778137690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:42:30.790 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.774+0000 7f67777fe700 1 -- 172.21.15.42:0/1327756981 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6778130240 con 0x7f6778119c00 2019-12-05T06:42:30.791 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.774+0000 7f67777fe700 1 --2- 172.21.15.42:0/1327756981 >> v2:172.21.15.40:3300/0 conn(0x7f6778119c00 0x7f677811b720 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f676c002f30 tx=0x7f676c010af0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:30.791 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.774+0000 7f67777fe700 1 --2- 172.21.15.42:0/1327756981 >> v2:172.21.15.40:3300/0 conn(0x7f6778119c00 0x7f677811b720 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f676c002f30 tx=0x7f676c010af0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:30.791 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.774+0000 7f67777fe700 1 --2- 172.21.15.42:0/1327756981 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6778119c00 0x7f677811b720 secure :-1 s=READY pgs=2305 cs=0 l=1 rx=0x7f676c002f30 tx=0x7f676c010af0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:42:30.791 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.778+0000 7f67777fe700 1 --2- 172.21.15.42:0/1327756981 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6778119c00 0x7f677811b720 secure :-1 s=THROTTLE_DONE pgs=2305 cs=0 l=1 rx=0x7f676c002f30 tx=0x7f676c010af0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:30.795 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.778+0000 7f6774ff9700 1 -- 172.21.15.42:0/1327756981 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f676c0125a0 con 0x7f6778119c00 2019-12-05T06:42:30.796 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.778+0000 7f67777fe700 1 --2- 172.21.15.42:0/1327756981 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6778119c00 0x7f677811b720 secure :-1 s=THROTTLE_DONE pgs=2305 cs=0 l=1 rx=0x7f676c002f30 tx=0x7f676c010af0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:30.796 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.778+0000 7f67777fe700 1 --2- 172.21.15.42:0/1327756981 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6778119c00 0x7f677811b720 secure :-1 s=THROTTLE_DONE pgs=2305 cs=0 l=1 rx=0x7f676c002f30 tx=0x7f676c010af0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:30.797 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.778+0000 7f6774ff9700 1 -- 172.21.15.42:0/1327756981 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f676c007150 con 0x7f6778119c00 2019-12-05T06:42:30.797 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.778+0000 7f6774ff9700 1 -- 172.21.15.42:0/1327756981 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f676c012b80 con 0x7f6778119c00 2019-12-05T06:42:30.797 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.778+0000 7f677defa700 1 -- 172.21.15.42:0/1327756981 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f677811c0a0 con 0x7f6778119c00 2019-12-05T06:42:30.797 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.778+0000 7f677defa700 1 -- 172.21.15.42:0/1327756981 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6778138ea0 con 0x7f6778119c00 2019-12-05T06:42:30.797 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.778+0000 7f67777fe700 1 --2- 172.21.15.42:0/1327756981 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6778119c00 0x7f677811b720 secure :-1 s=THROTTLE_DONE pgs=2305 cs=0 l=1 rx=0x7f676c002f30 tx=0x7f676c010af0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:30.797 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.778+0000 7f6774ff9700 1 -- 172.21.15.42:0/1327756981 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f676c005540 con 0x7f6778119c00 2019-12-05T06:42:30.797 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.778+0000 7f67777fe700 1 --2- 172.21.15.42:0/1327756981 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6778119c00 0x7f677811b720 secure :-1 s=THROTTLE_DONE pgs=2305 cs=0 l=1 rx=0x7f676c002f30 tx=0x7f676c010af0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:30.798 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.778+0000 7f6774ff9700 1 --2- 172.21.15.42:0/1327756981 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f675803b210 0x7f675803d720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:42:30.798 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.778+0000 7f6774ff9700 1 -- 172.21.15.42:0/1327756981 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 11707+0+0 (secure 0 0 0) 0x7f676c04afb0 con 0x7f6778119c00 2019-12-05T06:42:30.798 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.778+0000 7f6777fff700 1 --2- 172.21.15.42:0/1327756981 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f675803b210 0x7f675803d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:42:30.798 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.778+0000 7f6777fff700 1 --2- 172.21.15.42:0/1327756981 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f675803b210 0x7f675803d720 crc :-1 s=READY pgs=1273 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:42:30.803 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.782+0000 7f677defa700 1 -- 172.21.15.42:0/1327756981 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f677811e850 con 0x7f6778119c00 2019-12-05T06:42:30.803 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.790+0000 7f67777fe700 1 --2- 172.21.15.42:0/1327756981 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6778119c00 0x7f677811b720 secure :-1 s=THROTTLE_DONE pgs=2305 cs=0 l=1 rx=0x7f676c002f30 tx=0x7f676c010af0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:30.803 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.790+0000 7f6774ff9700 1 -- 172.21.15.42:0/1327756981 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f676c032030 con 0x7f6778119c00 2019-12-05T06:42:30.969 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.962+0000 7f677defa700 1 -- 172.21.15.42:0/1327756981 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f677805d110 con 0x7f6778119c00 2019-12-05T06:42:30.971 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.962+0000 7f67777fe700 1 --2- 172.21.15.42:0/1327756981 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6778119c00 0x7f677811b720 secure :-1 s=THROTTLE_DONE pgs=2305 cs=0 l=1 rx=0x7f676c002f30 tx=0x7f676c010af0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:30.971 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.962+0000 7f6774ff9700 1 -- 172.21.15.42:0/1327756981 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 367 v367) v1 ==== 95+0+2114 (secure 0 0 0) 0x7f676c032290 con 0x7f6778119c00 2019-12-05T06:42:30.973 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:42:30.974 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":367,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":12358,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/1922398668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1922398668},{"type":"v1","addr":"172.21.15.42:6835","nonce":1922398668}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":366}],"filesystems":[{"mdsmap":{"epoch":367,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:41:51.221717+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":115,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11976},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11976":{"gid":11976,"name":"b","rank":0,"incarnation":361,"state":"up:active","state_seq":27,"addr":"172.21.15.40:6833/474728597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":474728597},{"type":"v1","addr":"172.21.15.40:6833","nonce":474728597}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:42:30.974 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 367 2019-12-05T06:42:30.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.966+0000 7f677defa700 1 -- 172.21.15.42:0/1327756981 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f675803b210 msgr2=0x7f675803d720 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:42:30.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.966+0000 7f677defa700 1 --2- 172.21.15.42:0/1327756981 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f675803b210 0x7f675803d720 crc :-1 s=READY pgs=1273 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:42:30.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.966+0000 7f677defa700 1 -- 172.21.15.42:0/1327756981 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6778119c00 msgr2=0x7f677811b720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:42:30.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.966+0000 7f677defa700 1 --2- 172.21.15.42:0/1327756981 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6778119c00 0x7f677811b720 secure :-1 s=READY pgs=2305 cs=0 l=1 rx=0x7f676c002f30 tx=0x7f676c010af0).stop 2019-12-05T06:42:30.976 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.966+0000 7f677defa700 1 -- 172.21.15.42:0/1327756981 shutdown_connections 2019-12-05T06:42:30.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.966+0000 7f677defa700 1 --2- 172.21.15.42:0/1327756981 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f675803b210 0x7f675803d720 unknown :-1 s=CLOSED pgs=1273 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:42:30.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.966+0000 7f677defa700 1 --2- 172.21.15.42:0/1327756981 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6778119c00 0x7f677811b720 unknown :-1 s=CLOSED pgs=2305 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:42:30.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.966+0000 7f677defa700 1 -- 172.21.15.42:0/1327756981 shutdown_connections 2019-12-05T06:42:30.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.966+0000 7f677defa700 1 -- 172.21.15.42:0/1327756981 wait complete. 2019-12-05T06:42:30.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:30.966+0000 7f677defa700 1 -- 172.21.15.42:0/1327756981 >> 172.21.15.42:0/1327756981 conn(0x7f677807e920 msgr2=0x7f677811c4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:42:30.991 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:42:32.992 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:42:33.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:33.086+0000 7f647409e700 1 Processor -- start 2019-12-05T06:42:33.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:33.086+0000 7f647409e700 1 -- start start 2019-12-05T06:42:33.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:33.086+0000 7f647409e700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f646c0a7740 0x7f646c08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:42:33.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:33.086+0000 7f647409e700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f646c07e720 0x7f646c07eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:42:33.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:33.090+0000 7f647409e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f646c09f2e0 0x7f646c09f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:42:33.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:33.090+0000 7f647409e700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f646c0a0b00 con 0x7f646c0a7740 2019-12-05T06:42:33.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:33.090+0000 7f647409e700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f646c08e5b0 con 0x7f646c09f2e0 2019-12-05T06:42:33.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:33.090+0000 7f647409e700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f646c08e850 con 0x7f646c07e720 2019-12-05T06:42:33.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:33.090+0000 7f6472e3c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f646c09f2e0 0x7f646c09f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:42:33.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:33.090+0000 7f6472e3c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f646c09f2e0 0x7f646c09f710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:33684/0 (socket says 172.21.15.42:33684) 2019-12-05T06:42:33.103 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:33.090+0000 7f6471e3a700 1 --2- 172.21.15.42:0/3841259258 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f646c07e720 0x7f646c07eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:42:33.103 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:33.090+0000 7f6472e3c700 1 -- 172.21.15.42:0/3841259258 learned_addr learned my addr 172.21.15.42:0/3841259258 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:42:33.103 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:33.090+0000 7f647263b700 1 --2- 172.21.15.42:0/3841259258 >> v2:172.21.15.40:3300/0 conn(0x7f646c0a7740 0x7f646c08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:42:33.103 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:33.090+0000 7f6471e3a700 1 -- 172.21.15.42:0/3841259258 >> v2:172.21.15.42:3300/0 conn(0x7f646c09f2e0 msgr2=0x7f646c09f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:42:33.103 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:33.090+0000 7f6471e3a700 1 --2- 172.21.15.42:0/3841259258 >> v2:172.21.15.42:3300/0 conn(0x7f646c09f2e0 0x7f646c09f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:42:33.103 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:33.090+0000 7f6471e3a700 1 -- 172.21.15.42:0/3841259258 >> v2:172.21.15.40:3300/0 conn(0x7f646c0a7740 msgr2=0x7f646c08df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:42:33.103 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:33.090+0000 7f6471e3a700 1 --2- 172.21.15.42:0/3841259258 >> v2:172.21.15.40:3300/0 conn(0x7f646c0a7740 0x7f646c08df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:42:33.103 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:33.090+0000 7f6471e3a700 1 -- 172.21.15.42:0/3841259258 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f646c07f0a0 con 0x7f646c07e720 2019-12-05T06:42:33.104 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:33.090+0000 7f6472e3c700 1 --2- 172.21.15.42:0/3841259258 >> v2:172.21.15.42:3300/0 conn(0x7f646c09f2e0 0x7f646c09f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:42:33.104 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:33.090+0000 7f647263b700 1 --2- 172.21.15.42:0/3841259258 >> v2:172.21.15.40:3300/0 conn(0x7f646c0a7740 0x7f646c08df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:42:33.104 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:33.090+0000 7f6471e3a700 1 --2- 172.21.15.42:0/3841259258 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f646c07e720 0x7f646c07eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f64680032b0 tx=0x7f6468010070).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:33.104 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:33.090+0000 7f6471e3a700 1 --2- 172.21.15.42:0/3841259258 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f646c07e720 0x7f646c07eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f64680032b0 tx=0x7f6468010070).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:33.104 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:33.090+0000 7f6471e3a700 1 --2- 172.21.15.42:0/3841259258 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f646c07e720 0x7f646c07eb50 secure :-1 s=READY pgs=666 cs=0 l=1 rx=0x7f64680032b0 tx=0x7f6468010070).ready entity=mon.2 client_cookie=559aca5bf44cf100 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:42:33.104 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:33.090+0000 7f6471e3a700 1 --2- 172.21.15.42:0/3841259258 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f646c07e720 0x7f646c07eb50 secure :-1 s=THROTTLE_DONE pgs=666 cs=0 l=1 rx=0x7f64680032b0 tx=0x7f6468010070).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:33.104 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:33.090+0000 7f6471639700 1 -- 172.21.15.42:0/3841259258 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6468010940 con 0x7f646c07e720 2019-12-05T06:42:33.105 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:33.090+0000 7f6471e3a700 1 --2- 172.21.15.42:0/3841259258 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f646c07e720 0x7f646c07eb50 secure :-1 s=THROTTLE_DONE pgs=666 cs=0 l=1 rx=0x7f64680032b0 tx=0x7f6468010070).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:33.105 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:33.090+0000 7f6471639700 1 -- 172.21.15.42:0/3841259258 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6468008b50 con 0x7f646c07e720 2019-12-05T06:42:33.105 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:33.090+0000 7f6471e3a700 1 --2- 172.21.15.42:0/3841259258 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f646c07e720 0x7f646c07eb50 secure :-1 s=THROTTLE_DONE pgs=666 cs=0 l=1 rx=0x7f64680032b0 tx=0x7f6468010070).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:33.105 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:33.090+0000 7f647409e700 1 -- 172.21.15.42:0/3841259258 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f646c07e720 msgr2=0x7f646c07eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:42:33.105 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:33.090+0000 7f6471639700 1 -- 172.21.15.42:0/3841259258 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f64680110c0 con 0x7f646c07e720 2019-12-05T06:42:33.105 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:33.090+0000 7f647409e700 1 --2- 172.21.15.42:0/3841259258 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f646c07e720 0x7f646c07eb50 secure :-1 s=READY pgs=666 cs=0 l=1 rx=0x7f64680032b0 tx=0x7f6468010070).stop 2019-12-05T06:42:33.105 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:33.090+0000 7f647409e700 1 -- 172.21.15.42:0/3841259258 shutdown_connections 2019-12-05T06:42:33.106 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:33.090+0000 7f647409e700 1 --2- 172.21.15.42:0/3841259258 >> v2:172.21.15.42:3300/0 conn(0x7f646c09f2e0 0x7f646c09f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:42:33.106 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:33.090+0000 7f647409e700 1 --2- 172.21.15.42:0/3841259258 >> v2:172.21.15.40:3300/0 conn(0x7f646c0a7740 0x7f646c08df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:42:33.106 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:33.090+0000 7f647409e700 1 --2- 172.21.15.42:0/3841259258 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f646c07e720 0x7f646c07eb50 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:42:33.106 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:33.090+0000 7f647409e700 1 -- 172.21.15.42:0/3841259258 shutdown_connections 2019-12-05T06:42:33.106 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:33.090+0000 7f647409e700 1 -- 172.21.15.42:0/3841259258 wait complete. 2019-12-05T06:42:33.106 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:33.090+0000 7f647409e700 1 -- 172.21.15.42:0/3841259258 >> 172.21.15.42:0/3841259258 conn(0x7f646c0aabe0 msgr2=0x7f646c0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:42:33.109 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:33.094+0000 7f647409e700 1 Processor -- start 2019-12-05T06:42:33.109 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:33.094+0000 7f647409e700 1 -- start start 2019-12-05T06:42:33.110 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:33.094+0000 7f647409e700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f646c116500 0x7f646c1431a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:42:33.110 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:33.094+0000 7f647409e700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f646c1448c0 0x7f646c144cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:42:33.110 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:33.094+0000 7f647409e700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f646c145240 con 0x7f646c134390 2019-12-05T06:42:33.110 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:33.094+0000 7f647409e700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f646c1436f0 con 0x7f646c1448c0 2019-12-05T06:42:33.110 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:33.094+0000 7f647409e700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f646c133bb0 con 0x7f646c116500 2019-12-05T06:42:33.110 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:33.094+0000 7f6471e3a700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f646c116500 0x7f646c1431a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:42:33.110 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:33.094+0000 7f6472e3c700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f646c1448c0 0x7f646c144cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:42:33.111 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:33.094+0000 7f6471e3a700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f646c116500 0x7f646c1431a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:34152/0 (socket says 172.21.15.42:34152) 2019-12-05T06:42:33.111 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:33.094+0000 7f6471e3a700 1 -- 172.21.15.42:0/2022208976 learned_addr learned my addr 172.21.15.42:0/2022208976 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:42:33.111 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:33.098+0000 7f6471e3a700 1 -- 172.21.15.42:0/2022208976 >> v2:172.21.15.40:3300/0 conn(0x7f646c1448c0 msgr2=0x7f646c144cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:42:33.111 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:33.098+0000 7f6471e3a700 1 --2- 172.21.15.42:0/2022208976 >> v2:172.21.15.40:3300/0 conn(0x7f646c1448c0 0x7f646c144cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:42:33.111 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:33.098+0000 7f6471e3a700 1 -- 172.21.15.42:0/2022208976 >> v1:172.21.15.40:6789/0 conn(0x7f646c134390 legacy=0x7f646c1301c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:42:33.111 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:33.098+0000 7f6471e3a700 1 -- 172.21.15.42:0/2022208976 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f646c133780 con 0x7f646c116500 2019-12-05T06:42:33.111 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:33.098+0000 7f6472e3c700 1 --2- 172.21.15.42:0/2022208976 >> v2:172.21.15.40:3300/0 conn(0x7f646c1448c0 0x7f646c144cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:42:33.112 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:33.098+0000 7f6471e3a700 1 --2- 172.21.15.42:0/2022208976 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f646c116500 0x7f646c1431a0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f64680032b0 tx=0x7f64680109c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:33.112 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:33.098+0000 7f6471e3a700 1 --2- 172.21.15.42:0/2022208976 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f646c116500 0x7f646c1431a0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f64680032b0 tx=0x7f64680109c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:33.112 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:33.098+0000 7f6471e3a700 1 --2- 172.21.15.42:0/2022208976 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f646c116500 0x7f646c1431a0 secure :-1 s=READY pgs=667 cs=0 l=1 rx=0x7f64680032b0 tx=0x7f64680109c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:42:33.112 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:33.098+0000 7f6471e3a700 1 --2- 172.21.15.42:0/2022208976 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f646c116500 0x7f646c1431a0 secure :-1 s=THROTTLE_DONE pgs=667 cs=0 l=1 rx=0x7f64680032b0 tx=0x7f64680109c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:33.112 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:33.098+0000 7f6471e3a700 1 --2- 172.21.15.42:0/2022208976 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f646c116500 0x7f646c1431a0 secure :-1 s=THROTTLE_DONE pgs=667 cs=0 l=1 rx=0x7f64680032b0 tx=0x7f64680109c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:33.112 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:33.098+0000 7f645effd700 1 -- 172.21.15.42:0/2022208976 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6468005ab0 con 0x7f646c116500 2019-12-05T06:42:33.112 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:33.098+0000 7f6471e3a700 1 --2- 172.21.15.42:0/2022208976 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f646c116500 0x7f646c1431a0 secure :-1 s=THROTTLE_DONE pgs=667 cs=0 l=1 rx=0x7f64680032b0 tx=0x7f64680109c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:33.113 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:33.098+0000 7f645effd700 1 -- 172.21.15.42:0/2022208976 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6468010b40 con 0x7f646c116500 2019-12-05T06:42:33.113 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:33.098+0000 7f645effd700 1 -- 172.21.15.42:0/2022208976 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f646800f680 con 0x7f646c116500 2019-12-05T06:42:33.113 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:33.098+0000 7f647409e700 1 -- 172.21.15.42:0/2022208976 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f646c077300 con 0x7f646c116500 2019-12-05T06:42:33.113 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:33.098+0000 7f647409e700 1 -- 172.21.15.42:0/2022208976 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f646c0777d0 con 0x7f646c116500 2019-12-05T06:42:33.113 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:33.098+0000 7f6471e3a700 1 --2- 172.21.15.42:0/2022208976 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f646c116500 0x7f646c1431a0 secure :-1 s=THROTTLE_DONE pgs=667 cs=0 l=1 rx=0x7f64680032b0 tx=0x7f64680109c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:33.113 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:33.098+0000 7f645effd700 1 -- 172.21.15.42:0/2022208976 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f6468011360 con 0x7f646c116500 2019-12-05T06:42:33.113 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:33.098+0000 7f6471e3a700 1 --2- 172.21.15.42:0/2022208976 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f646c116500 0x7f646c1431a0 secure :-1 s=THROTTLE_DONE pgs=667 cs=0 l=1 rx=0x7f64680032b0 tx=0x7f64680109c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:33.114 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:33.098+0000 7f645effd700 1 --2- 172.21.15.42:0/2022208976 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f645403add0 0x7f645403d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:42:33.114 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:33.098+0000 7f645effd700 1 -- 172.21.15.42:0/2022208976 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 11707+0+0 (secure 0 0 0) 0x7f646808f050 con 0x7f646c116500 2019-12-05T06:42:33.114 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:33.098+0000 7f647263b700 1 --2- 172.21.15.42:0/2022208976 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f645403add0 0x7f645403d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:42:33.114 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:33.102+0000 7f647263b700 1 --2- 172.21.15.42:0/2022208976 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f645403add0 0x7f645403d2e0 crc :-1 s=READY pgs=1274 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:42:33.114 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:33.102+0000 7f647409e700 1 -- 172.21.15.42:0/2022208976 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f646c05d110 con 0x7f646c116500 2019-12-05T06:42:33.120 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:33.110+0000 7f6471e3a700 1 --2- 172.21.15.42:0/2022208976 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f646c116500 0x7f646c1431a0 secure :-1 s=THROTTLE_DONE pgs=667 cs=0 l=1 rx=0x7f64680032b0 tx=0x7f64680109c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:33.120 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:33.114+0000 7f645effd700 1 -- 172.21.15.42:0/2022208976 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f646800f990 con 0x7f646c116500 2019-12-05T06:42:33.291 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:33.282+0000 7f647409e700 1 -- 172.21.15.42:0/2022208976 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f646c05b8c0 con 0x7f646c116500 2019-12-05T06:42:33.292 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:33.282+0000 7f6471e3a700 1 --2- 172.21.15.42:0/2022208976 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f646c116500 0x7f646c1431a0 secure :-1 s=THROTTLE_DONE pgs=667 cs=0 l=1 rx=0x7f64680032b0 tx=0x7f64680109c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:33.292 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:33.282+0000 7f645effd700 1 -- 172.21.15.42:0/2022208976 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 367 v367) v1 ==== 95+0+2114 (secure 0 0 0) 0x7f6468011c60 con 0x7f646c116500 2019-12-05T06:42:33.294 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:42:33.295 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":367,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":12358,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/1922398668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1922398668},{"type":"v1","addr":"172.21.15.42:6835","nonce":1922398668}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":366}],"filesystems":[{"mdsmap":{"epoch":367,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:41:51.221717+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":115,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11976},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11976":{"gid":11976,"name":"b","rank":0,"incarnation":361,"state":"up:active","state_seq":27,"addr":"172.21.15.40:6833/474728597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":474728597},{"type":"v1","addr":"172.21.15.40:6833","nonce":474728597}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:42:33.295 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 367 2019-12-05T06:42:33.297 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:33.290+0000 7f647409e700 1 -- 172.21.15.42:0/2022208976 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f645403add0 msgr2=0x7f645403d2e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:42:33.297 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:33.290+0000 7f647409e700 1 --2- 172.21.15.42:0/2022208976 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f645403add0 0x7f645403d2e0 crc :-1 s=READY pgs=1274 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:42:33.297 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:33.290+0000 7f647409e700 1 -- 172.21.15.42:0/2022208976 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f646c116500 msgr2=0x7f646c1431a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:42:33.298 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:33.290+0000 7f647409e700 1 --2- 172.21.15.42:0/2022208976 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f646c116500 0x7f646c1431a0 secure :-1 s=READY pgs=667 cs=0 l=1 rx=0x7f64680032b0 tx=0x7f64680109c0).stop 2019-12-05T06:42:33.298 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:33.290+0000 7f647409e700 1 -- 172.21.15.42:0/2022208976 shutdown_connections 2019-12-05T06:42:33.304 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:33.290+0000 7f647409e700 1 --2- 172.21.15.42:0/2022208976 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f645403add0 0x7f645403d2e0 unknown :-1 s=CLOSED pgs=1274 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:42:33.304 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:33.290+0000 7f647409e700 1 --2- 172.21.15.42:0/2022208976 >> v2:172.21.15.40:3300/0 conn(0x7f646c1448c0 0x7f646c144cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:42:33.304 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:33.290+0000 7f647409e700 1 --2- 172.21.15.42:0/2022208976 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f646c116500 0x7f646c1431a0 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:42:33.305 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:33.290+0000 7f647409e700 1 -- 172.21.15.42:0/2022208976 shutdown_connections 2019-12-05T06:42:33.305 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:33.290+0000 7f647409e700 1 -- 172.21.15.42:0/2022208976 wait complete. 2019-12-05T06:42:33.305 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:33.290+0000 7f647409e700 1 -- 172.21.15.42:0/2022208976 >> 172.21.15.42:0/2022208976 conn(0x7f646c0a7f90 msgr2=0x7f646c09f3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:42:33.308 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:42:35.310 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:42:35.419 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:35.406+0000 7fcecc7a4700 1 Processor -- start 2019-12-05T06:42:35.419 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:35.406+0000 7fcecc7a4700 1 -- start start 2019-12-05T06:42:35.420 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:35.406+0000 7fcecc7a4700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fcec408df80 0x7fcec409f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:42:35.420 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:35.406+0000 7fcecc7a4700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcec409f830 0x7fcec414efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:42:35.420 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:35.406+0000 7fcecc7a4700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcec409fc60 con 0x7fcec40a7740 2019-12-05T06:42:35.420 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:35.406+0000 7fcecc7a4700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcec40a0b00 con 0x7fcec408df80 2019-12-05T06:42:35.421 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:35.406+0000 7fcecc7a4700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcec408e5b0 con 0x7fcec409f830 2019-12-05T06:42:35.421 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:35.406+0000 7fceca540700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fcec408df80 0x7fcec409f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:42:35.421 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:35.406+0000 7fcecad41700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fcec40a7740 0x7fcec407e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:33734/0 (socket says 172.21.15.42:33734) 2019-12-05T06:42:35.421 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:35.406+0000 7fceca540700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fcec408df80 0x7fcec409f2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:33698/0 (socket says 172.21.15.42:33698) 2019-12-05T06:42:35.422 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:35.406+0000 7fcecad41700 1 -- 172.21.15.42:0/3782364640 learned_addr learned my addr 172.21.15.42:0/3782364640 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:42:35.422 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:35.406+0000 7fcecb542700 1 --2- 172.21.15.42:0/3782364640 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcec409f830 0x7fcec414efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:42:35.422 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:35.410+0000 7fceca540700 1 -- 172.21.15.42:0/3782364640 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcec409f830 msgr2=0x7fcec414efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:42:35.422 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:35.410+0000 7fceca540700 1 --2- 172.21.15.42:0/3782364640 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcec409f830 0x7fcec414efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:42:35.423 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:35.410+0000 7fceca540700 1 -- 172.21.15.42:0/3782364640 >> v1:172.21.15.42:6789/0 conn(0x7fcec40a7740 legacy=0x7fcec407e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:42:35.423 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:35.410+0000 7fceca540700 1 -- 172.21.15.42:0/3782364640 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcec40a0710 con 0x7fcec408df80 2019-12-05T06:42:35.424 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:35.410+0000 7fcecb542700 1 --2- 172.21.15.42:0/3782364640 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcec409f830 0x7fcec414efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:42:35.424 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:35.410+0000 7fceca540700 1 --2- 172.21.15.42:0/3782364640 >> v2:172.21.15.42:3300/0 conn(0x7fcec408df80 0x7fcec409f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fceb4008790 tx=0x7fceb400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:35.425 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:35.410+0000 7fceca540700 1 --2- 172.21.15.42:0/3782364640 >> v2:172.21.15.42:3300/0 conn(0x7fcec408df80 0x7fcec409f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fceb4008790 tx=0x7fceb400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:35.426 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:35.410+0000 7fceca540700 1 --2- 172.21.15.42:0/3782364640 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcec408df80 0x7fcec409f2e0 secure :-1 s=READY pgs=3040 cs=0 l=1 rx=0x7fceb4008790 tx=0x7fceb400fc70).ready entity=mon.1 client_cookie=7a5d6f822f13c8fc server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:42:35.426 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:35.410+0000 7fceca540700 1 --2- 172.21.15.42:0/3782364640 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcec408df80 0x7fcec409f2e0 secure :-1 s=THROTTLE_DONE pgs=3040 cs=0 l=1 rx=0x7fceb4008790 tx=0x7fceb400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:35.426 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:35.410+0000 7fcec9d3f700 1 -- 172.21.15.42:0/3782364640 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fceb4008bb0 con 0x7fcec408df80 2019-12-05T06:42:35.426 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:35.410+0000 7fceca540700 1 --2- 172.21.15.42:0/3782364640 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcec408df80 0x7fcec409f2e0 secure :-1 s=THROTTLE_DONE pgs=3040 cs=0 l=1 rx=0x7fceb4008790 tx=0x7fceb400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:35.427 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:35.410+0000 7fceca540700 1 --2- 172.21.15.42:0/3782364640 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcec408df80 0x7fcec409f2e0 secure :-1 s=THROTTLE_DONE pgs=3040 cs=0 l=1 rx=0x7fceb4008790 tx=0x7fceb400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:35.427 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:35.410+0000 7fcec9d3f700 1 -- 172.21.15.42:0/3782364640 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fceb40087d0 con 0x7fcec408df80 2019-12-05T06:42:35.427 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:35.410+0000 7fcec9d3f700 1 -- 172.21.15.42:0/3782364640 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fceb4011860 con 0x7fcec408df80 2019-12-05T06:42:35.428 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:35.410+0000 7fcecc7a4700 1 -- 172.21.15.42:0/3782364640 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcec408df80 msgr2=0x7fcec409f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:42:35.428 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:35.410+0000 7fcecc7a4700 1 --2- 172.21.15.42:0/3782364640 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcec408df80 0x7fcec409f2e0 secure :-1 s=READY pgs=3040 cs=0 l=1 rx=0x7fceb4008790 tx=0x7fceb400fc70).stop 2019-12-05T06:42:35.428 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:35.410+0000 7fcecc7a4700 1 -- 172.21.15.42:0/3782364640 shutdown_connections 2019-12-05T06:42:35.428 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:35.410+0000 7fcecc7a4700 1 --2- 172.21.15.42:0/3782364640 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcec409f830 0x7fcec414efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:42:35.428 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:35.410+0000 7fcecc7a4700 1 --2- 172.21.15.42:0/3782364640 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcec408df80 0x7fcec409f2e0 unknown :-1 s=CLOSED pgs=3040 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:42:35.429 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:35.410+0000 7fcecc7a4700 1 -- 172.21.15.42:0/3782364640 shutdown_connections 2019-12-05T06:42:35.429 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:35.410+0000 7fcecc7a4700 1 -- 172.21.15.42:0/3782364640 wait complete. 2019-12-05T06:42:35.429 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:35.410+0000 7fcecc7a4700 1 -- 172.21.15.42:0/3782364640 >> 172.21.15.42:0/3782364640 conn(0x7fcec40aabe0 msgr2=0x7fcec40a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:42:35.436 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:35.414+0000 7fcecc7a4700 1 Processor -- start 2019-12-05T06:42:35.436 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:35.414+0000 7fcecc7a4700 1 -- start start 2019-12-05T06:42:35.436 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:35.414+0000 7fcecc7a4700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fcec407c4e0 0x7fcec41161c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:42:35.437 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:35.414+0000 7fcecc7a4700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcec411b1b0 0x7fcec411a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:42:35.437 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:35.414+0000 7fcecc7a4700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcec4118e90 con 0x7fcec411ab40 2019-12-05T06:42:35.437 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:35.414+0000 7fcecc7a4700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcec411c350 con 0x7fcec407c4e0 2019-12-05T06:42:35.437 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:35.414+0000 7fcecc7a4700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcec407bfa0 con 0x7fcec411b1b0 2019-12-05T06:42:35.437 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:35.414+0000 7fcecad41700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fcec407c4e0 0x7fcec41161c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:42:35.438 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:35.414+0000 7fcecad41700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fcec407c4e0 0x7fcec41161c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:33702/0 (socket says 172.21.15.42:33702) 2019-12-05T06:42:35.438 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:35.414+0000 7fcecad41700 1 -- 172.21.15.42:0/856925434 learned_addr learned my addr 172.21.15.42:0/856925434 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:42:35.438 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:35.414+0000 7fceca540700 1 --2- 172.21.15.42:0/856925434 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcec411b1b0 0x7fcec411a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:42:35.438 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:35.414+0000 7fcecad41700 1 -- 172.21.15.42:0/856925434 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcec411b1b0 msgr2=0x7fcec411a5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:42:35.439 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:35.414+0000 7fcecad41700 1 --2- 172.21.15.42:0/856925434 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcec411b1b0 0x7fcec411a5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:42:35.439 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:35.414+0000 7fcecad41700 1 -- 172.21.15.42:0/856925434 >> v1:172.21.15.42:6789/0 conn(0x7fcec411ab40 legacy=0x7fcec408eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:42:35.439 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:35.414+0000 7fcecad41700 1 -- 172.21.15.42:0/856925434 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcec4116710 con 0x7fcec407c4e0 2019-12-05T06:42:35.440 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:35.414+0000 7fcecad41700 1 --2- 172.21.15.42:0/856925434 >> v2:172.21.15.42:3300/0 conn(0x7fcec407c4e0 0x7fcec41161c0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fcec00043a0 tx=0x7fcec0010a00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:35.440 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:35.414+0000 7fcecad41700 1 --2- 172.21.15.42:0/856925434 >> v2:172.21.15.42:3300/0 conn(0x7fcec407c4e0 0x7fcec41161c0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fcec00043a0 tx=0x7fcec0010a00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:35.440 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:35.414+0000 7fcecad41700 1 --2- 172.21.15.42:0/856925434 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcec407c4e0 0x7fcec41161c0 secure :-1 s=READY pgs=3042 cs=0 l=1 rx=0x7fcec00043a0 tx=0x7fcec0010a00).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:42:35.440 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:35.414+0000 7fcecad41700 1 --2- 172.21.15.42:0/856925434 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcec407c4e0 0x7fcec41161c0 secure :-1 s=THROTTLE_DONE pgs=3042 cs=0 l=1 rx=0x7fcec00043a0 tx=0x7fcec0010a00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:35.441 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:35.414+0000 7fcecad41700 1 --2- 172.21.15.42:0/856925434 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcec407c4e0 0x7fcec41161c0 secure :-1 s=THROTTLE_DONE pgs=3042 cs=0 l=1 rx=0x7fcec00043a0 tx=0x7fcec0010a00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:35.441 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:35.414+0000 7fcebb7fe700 1 -- 172.21.15.42:0/856925434 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcec000c890 con 0x7fcec407c4e0 2019-12-05T06:42:35.441 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:35.414+0000 7fcecad41700 1 --2- 172.21.15.42:0/856925434 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcec407c4e0 0x7fcec41161c0 secure :-1 s=THROTTLE_DONE pgs=3042 cs=0 l=1 rx=0x7fcec00043a0 tx=0x7fcec0010a00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:35.441 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:35.414+0000 7fcebb7fe700 1 -- 172.21.15.42:0/856925434 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcec000c4b0 con 0x7fcec407c4e0 2019-12-05T06:42:35.442 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:35.418+0000 7fcecc7a4700 1 -- 172.21.15.42:0/856925434 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcec407b320 con 0x7fcec407c4e0 2019-12-05T06:42:35.442 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:35.418+0000 7fcebb7fe700 1 -- 172.21.15.42:0/856925434 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcec0004430 con 0x7fcec407c4e0 2019-12-05T06:42:35.442 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:35.418+0000 7fcecc7a4700 1 -- 172.21.15.42:0/856925434 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcec411b5e0 con 0x7fcec407c4e0 2019-12-05T06:42:35.442 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:35.418+0000 7fcecad41700 1 --2- 172.21.15.42:0/856925434 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcec407c4e0 0x7fcec41161c0 secure :-1 s=THROTTLE_DONE pgs=3042 cs=0 l=1 rx=0x7fcec00043a0 tx=0x7fcec0010a00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:35.443 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:35.418+0000 7fcebb7fe700 1 -- 172.21.15.42:0/856925434 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fcec0004740 con 0x7fcec407c4e0 2019-12-05T06:42:35.443 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:35.418+0000 7fcecad41700 1 --2- 172.21.15.42:0/856925434 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcec407c4e0 0x7fcec41161c0 secure :-1 s=THROTTLE_DONE pgs=3042 cs=0 l=1 rx=0x7fcec00043a0 tx=0x7fcec0010a00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:35.443 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:35.418+0000 7fcebb7fe700 1 --2- 172.21.15.42:0/856925434 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fceac03ad80 0x7fceac03d290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:42:35.444 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:35.418+0000 7fcebb7fe700 1 -- 172.21.15.42:0/856925434 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 11707+0+0 (secure 0 0 0) 0x7fcec004ad00 con 0x7fcec407c4e0 2019-12-05T06:42:35.444 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:35.418+0000 7fceca540700 1 --2- 172.21.15.42:0/856925434 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fceac03ad80 0x7fceac03d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:42:35.444 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:35.418+0000 7fceca540700 1 --2- 172.21.15.42:0/856925434 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fceac03ad80 0x7fceac03d290 crc :-1 s=READY pgs=1275 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:42:35.444 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:35.422+0000 7fcecc7a4700 1 -- 172.21.15.42:0/856925434 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcec405d110 con 0x7fcec407c4e0 2019-12-05T06:42:35.445 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:35.426+0000 7fcecad41700 1 --2- 172.21.15.42:0/856925434 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcec407c4e0 0x7fcec41161c0 secure :-1 s=THROTTLE_DONE pgs=3042 cs=0 l=1 rx=0x7fcec00043a0 tx=0x7fcec0010a00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:35.445 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:35.426+0000 7fcebb7fe700 1 -- 172.21.15.42:0/856925434 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fcec0031d80 con 0x7fcec407c4e0 2019-12-05T06:42:35.600 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:35.590+0000 7fcecc7a4700 1 -- 172.21.15.42:0/856925434 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcec407bd80 con 0x7fcec407c4e0 2019-12-05T06:42:35.600 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:35.590+0000 7fcecad41700 1 --2- 172.21.15.42:0/856925434 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcec407c4e0 0x7fcec41161c0 secure :-1 s=THROTTLE_DONE pgs=3042 cs=0 l=1 rx=0x7fcec00043a0 tx=0x7fcec0010a00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:35.601 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:35.590+0000 7fcebb7fe700 1 -- 172.21.15.42:0/856925434 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 367 v367) v1 ==== 95+0+2114 (secure 0 0 0) 0x7fcec0031fe0 con 0x7fcec407c4e0 2019-12-05T06:42:35.608 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 367 2019-12-05T06:42:35.608 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:42:35.608 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":367,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":12358,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/1922398668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1922398668},{"type":"v1","addr":"172.21.15.42:6835","nonce":1922398668}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":366}],"filesystems":[{"mdsmap":{"epoch":367,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:41:51.221717+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":115,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11976},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11976":{"gid":11976,"name":"b","rank":0,"incarnation":361,"state":"up:active","state_seq":27,"addr":"172.21.15.40:6833/474728597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":474728597},{"type":"v1","addr":"172.21.15.40:6833","nonce":474728597}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:42:35.610 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:35.594+0000 7fcecc7a4700 1 -- 172.21.15.42:0/856925434 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fceac03ad80 msgr2=0x7fceac03d290 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:42:35.610 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:35.594+0000 7fcecc7a4700 1 --2- 172.21.15.42:0/856925434 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fceac03ad80 0x7fceac03d290 crc :-1 s=READY pgs=1275 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:42:35.610 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:35.594+0000 7fcecc7a4700 1 -- 172.21.15.42:0/856925434 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcec407c4e0 msgr2=0x7fcec41161c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:42:35.610 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:35.594+0000 7fcecc7a4700 1 --2- 172.21.15.42:0/856925434 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcec407c4e0 0x7fcec41161c0 secure :-1 s=READY pgs=3042 cs=0 l=1 rx=0x7fcec00043a0 tx=0x7fcec0010a00).stop 2019-12-05T06:42:35.610 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:35.594+0000 7fcecc7a4700 1 -- 172.21.15.42:0/856925434 shutdown_connections 2019-12-05T06:42:35.611 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:35.594+0000 7fcecc7a4700 1 --2- 172.21.15.42:0/856925434 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fceac03ad80 0x7fceac03d290 unknown :-1 s=CLOSED pgs=1275 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:42:35.611 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:35.594+0000 7fcecc7a4700 1 --2- 172.21.15.42:0/856925434 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcec407c4e0 0x7fcec41161c0 unknown :-1 s=CLOSED pgs=3042 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:42:35.611 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:35.594+0000 7fcecc7a4700 1 --2- 172.21.15.42:0/856925434 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcec411b1b0 0x7fcec411a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:42:35.611 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:35.594+0000 7fcecc7a4700 1 -- 172.21.15.42:0/856925434 shutdown_connections 2019-12-05T06:42:35.611 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:35.594+0000 7fcecc7a4700 1 -- 172.21.15.42:0/856925434 wait complete. 2019-12-05T06:42:35.611 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:35.594+0000 7fcecc7a4700 1 -- 172.21.15.42:0/856925434 >> 172.21.15.42:0/856925434 conn(0x7fcec40a82a0 msgr2=0x7fcec4059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:42:35.612 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:42:37.614 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:42:37.718 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:37.706+0000 7ff73bde1700 1 Processor -- start 2019-12-05T06:42:37.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:37.706+0000 7ff73bde1700 1 -- start start 2019-12-05T06:42:37.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:37.706+0000 7ff73bde1700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff7340a7740 0x7ff73408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:42:37.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:37.706+0000 7ff73bde1700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff73407e720 0x7ff73407eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:42:37.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:37.706+0000 7ff73bde1700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff73409f2e0 0x7ff73409f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:42:37.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:37.706+0000 7ff73bde1700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7340a0b00 con 0x7ff73409f2e0 2019-12-05T06:42:37.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:37.706+0000 7ff73bde1700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff73408e5b0 con 0x7ff7340a7740 2019-12-05T06:42:37.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:37.706+0000 7ff73bde1700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff73408e850 con 0x7ff73407e720 2019-12-05T06:42:37.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:37.706+0000 7ff73a37e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff7340a7740 0x7ff73408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:42:37.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:37.706+0000 7ff73a37e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff7340a7740 0x7ff73408df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:33714/0 (socket says 172.21.15.42:33714) 2019-12-05T06:42:37.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:37.706+0000 7ff73ab7f700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff73409f2e0 0x7ff73409f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:42:37.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:37.706+0000 7ff73a37e700 1 -- 172.21.15.42:0/3084811284 learned_addr learned my addr 172.21.15.42:0/3084811284 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:42:37.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:37.706+0000 7ff739b7d700 1 --2- 172.21.15.42:0/3084811284 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff73407e720 0x7ff73407eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:42:37.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:37.706+0000 7ff73a37e700 1 -- 172.21.15.42:0/3084811284 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff73407e720 msgr2=0x7ff73407eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:42:37.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:37.706+0000 7ff73a37e700 1 --2- 172.21.15.42:0/3084811284 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff73407e720 0x7ff73407eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:42:37.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:37.706+0000 7ff73a37e700 1 -- 172.21.15.42:0/3084811284 >> v2:172.21.15.40:3300/0 conn(0x7ff73409f2e0 msgr2=0x7ff73409f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:42:37.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:37.706+0000 7ff73a37e700 1 --2- 172.21.15.42:0/3084811284 >> v2:172.21.15.40:3300/0 conn(0x7ff73409f2e0 0x7ff73409f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:42:37.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:37.706+0000 7ff73a37e700 1 -- 172.21.15.42:0/3084811284 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff73407f0a0 con 0x7ff7340a7740 2019-12-05T06:42:37.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:37.706+0000 7ff73a37e700 1 --2- 172.21.15.42:0/3084811284 >> v2:172.21.15.42:3300/0 conn(0x7ff7340a7740 0x7ff73408df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7ff73408e770 tx=0x7ff72800fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:37.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:37.706+0000 7ff73a37e700 1 --2- 172.21.15.42:0/3084811284 >> v2:172.21.15.42:3300/0 conn(0x7ff7340a7740 0x7ff73408df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7ff73408e770 tx=0x7ff72800fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:37.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:37.706+0000 7ff73a37e700 1 --2- 172.21.15.42:0/3084811284 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff7340a7740 0x7ff73408df80 secure :-1 s=READY pgs=3043 cs=0 l=1 rx=0x7ff73408e770 tx=0x7ff72800fdb0).ready entity=mon.1 client_cookie=7290cfd23836881c server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:42:37.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:37.706+0000 7ff73a37e700 1 --2- 172.21.15.42:0/3084811284 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff7340a7740 0x7ff73408df80 secure :-1 s=THROTTLE_DONE pgs=3043 cs=0 l=1 rx=0x7ff73408e770 tx=0x7ff72800fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:37.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:37.706+0000 7ff73937c700 1 -- 172.21.15.42:0/3084811284 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff728008c90 con 0x7ff7340a7740 2019-12-05T06:42:37.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:37.706+0000 7ff73a37e700 1 --2- 172.21.15.42:0/3084811284 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff7340a7740 0x7ff73408df80 secure :-1 s=THROTTLE_DONE pgs=3043 cs=0 l=1 rx=0x7ff73408e770 tx=0x7ff72800fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:37.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:37.706+0000 7ff73a37e700 1 --2- 172.21.15.42:0/3084811284 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff7340a7740 0x7ff73408df80 secure :-1 s=THROTTLE_DONE pgs=3043 cs=0 l=1 rx=0x7ff73408e770 tx=0x7ff72800fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:37.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:37.706+0000 7ff73937c700 1 -- 172.21.15.42:0/3084811284 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7280088b0 con 0x7ff7340a7740 2019-12-05T06:42:37.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:37.706+0000 7ff73937c700 1 -- 172.21.15.42:0/3084811284 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff728011b50 con 0x7ff7340a7740 2019-12-05T06:42:37.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:37.706+0000 7ff73bde1700 1 -- 172.21.15.42:0/3084811284 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff7340a7740 msgr2=0x7ff73408df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:42:37.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:37.706+0000 7ff73bde1700 1 --2- 172.21.15.42:0/3084811284 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff7340a7740 0x7ff73408df80 secure :-1 s=READY pgs=3043 cs=0 l=1 rx=0x7ff73408e770 tx=0x7ff72800fdb0).stop 2019-12-05T06:42:37.724 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:37.710+0000 7ff73bde1700 1 -- 172.21.15.42:0/3084811284 shutdown_connections 2019-12-05T06:42:37.724 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:37.710+0000 7ff73bde1700 1 --2- 172.21.15.42:0/3084811284 >> v2:172.21.15.40:3300/0 conn(0x7ff73409f2e0 0x7ff73409f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:42:37.724 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:37.710+0000 7ff73bde1700 1 --2- 172.21.15.42:0/3084811284 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff7340a7740 0x7ff73408df80 unknown :-1 s=CLOSED pgs=3043 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:42:37.724 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:37.710+0000 7ff73bde1700 1 --2- 172.21.15.42:0/3084811284 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff73407e720 0x7ff73407eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:42:37.724 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:37.710+0000 7ff73bde1700 1 -- 172.21.15.42:0/3084811284 shutdown_connections 2019-12-05T06:42:37.724 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:37.710+0000 7ff73bde1700 1 -- 172.21.15.42:0/3084811284 wait complete. 2019-12-05T06:42:37.725 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:37.710+0000 7ff73bde1700 1 -- 172.21.15.42:0/3084811284 >> 172.21.15.42:0/3084811284 conn(0x7ff7340aabe0 msgr2=0x7ff7340a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:42:37.725 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:37.710+0000 7ff73bde1700 1 Processor -- start 2019-12-05T06:42:37.725 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:37.710+0000 7ff73bde1700 1 -- start start 2019-12-05T06:42:37.725 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:37.710+0000 7ff73bde1700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff734135690 0x7ff734137ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:42:37.725 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:37.710+0000 7ff73bde1700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff734130030 con 0x7ff73407bfb0 2019-12-05T06:42:37.725 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:37.710+0000 7ff73bde1700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff734063190 con 0x7ff73411c850 2019-12-05T06:42:37.726 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:37.710+0000 7ff73bde1700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7340a56d0 con 0x7ff734135690 2019-12-05T06:42:37.726 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:37.710+0000 7ff73ab7f700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff734135690 0x7ff734137ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:42:37.726 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:37.710+0000 7ff73a37e700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7ff73411c850 0x7ff73412f830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:33756/0 (socket says 172.21.15.42:33756) 2019-12-05T06:42:37.726 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:37.710+0000 7ff73a37e700 1 -- 172.21.15.42:0/2115692113 learned_addr learned my addr 172.21.15.42:0/2115692113 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:42:37.726 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:37.710+0000 7ff73ab7f700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff734135690 0x7ff734137ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:33722/0 (socket says 172.21.15.42:33722) 2019-12-05T06:42:37.729 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:37.710+0000 7ff726ffd700 1 -- 172.21.15.42:0/2115692113 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3265100732 0 0) 0x7ff734130030 con 0x7ff73407bfb0 2019-12-05T06:42:37.729 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:37.710+0000 7ff73ab7f700 1 -- 172.21.15.42:0/2115692113 >> v1:172.21.15.42:6789/0 conn(0x7ff73411c850 legacy=0x7ff73412f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:42:37.729 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:37.710+0000 7ff73ab7f700 1 -- 172.21.15.42:0/2115692113 >> v1:172.21.15.40:6789/0 conn(0x7ff73407bfb0 legacy=0x7ff7340a58e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:42:37.730 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:37.710+0000 7ff73ab7f700 1 -- 172.21.15.42:0/2115692113 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7340a4ee0 con 0x7ff734135690 2019-12-05T06:42:37.730 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:37.710+0000 7ff73ab7f700 1 --2- 172.21.15.42:0/2115692113 >> v2:172.21.15.42:3300/0 conn(0x7ff734135690 0x7ff734137ad0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7ff728008370 tx=0x7ff72800fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:37.730 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:37.710+0000 7ff73ab7f700 1 --2- 172.21.15.42:0/2115692113 >> v2:172.21.15.42:3300/0 conn(0x7ff734135690 0x7ff734137ad0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7ff728008370 tx=0x7ff72800fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:37.730 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:37.710+0000 7ff73ab7f700 1 --2- 172.21.15.42:0/2115692113 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff734135690 0x7ff734137ad0 secure :-1 s=READY pgs=3045 cs=0 l=1 rx=0x7ff728008370 tx=0x7ff72800fdb0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:42:37.730 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:37.714+0000 7ff73ab7f700 1 --2- 172.21.15.42:0/2115692113 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff734135690 0x7ff734137ad0 secure :-1 s=THROTTLE_DONE pgs=3045 cs=0 l=1 rx=0x7ff728008370 tx=0x7ff72800fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:37.730 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:37.714+0000 7ff726ffd700 1 -- 172.21.15.42:0/2115692113 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff72c005840 con 0x7ff734135690 2019-12-05T06:42:37.731 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:37.714+0000 7ff73ab7f700 1 --2- 172.21.15.42:0/2115692113 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff734135690 0x7ff734137ad0 secure :-1 s=THROTTLE_DONE pgs=3045 cs=0 l=1 rx=0x7ff728008370 tx=0x7ff72800fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:37.731 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:37.714+0000 7ff73ab7f700 1 --2- 172.21.15.42:0/2115692113 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff734135690 0x7ff734137ad0 secure :-1 s=THROTTLE_DONE pgs=3045 cs=0 l=1 rx=0x7ff728008370 tx=0x7ff72800fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:37.731 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:37.714+0000 7ff726ffd700 1 -- 172.21.15.42:0/2115692113 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff72c007160 con 0x7ff734135690 2019-12-05T06:42:37.731 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:37.714+0000 7ff73bde1700 1 -- 172.21.15.42:0/2115692113 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7340a60e0 con 0x7ff734135690 2019-12-05T06:42:37.731 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:37.714+0000 7ff726ffd700 1 -- 172.21.15.42:0/2115692113 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff72c0060c0 con 0x7ff734135690 2019-12-05T06:42:37.731 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:37.714+0000 7ff73bde1700 1 -- 172.21.15.42:0/2115692113 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff734138020 con 0x7ff734135690 2019-12-05T06:42:37.731 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:37.714+0000 7ff73ab7f700 1 --2- 172.21.15.42:0/2115692113 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff734135690 0x7ff734137ad0 secure :-1 s=THROTTLE_DONE pgs=3045 cs=0 l=1 rx=0x7ff728008370 tx=0x7ff72800fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:37.732 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:37.714+0000 7ff726ffd700 1 -- 172.21.15.42:0/2115692113 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7ff72c006390 con 0x7ff734135690 2019-12-05T06:42:37.732 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:37.714+0000 7ff73ab7f700 1 --2- 172.21.15.42:0/2115692113 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff734135690 0x7ff734137ad0 secure :-1 s=THROTTLE_DONE pgs=3045 cs=0 l=1 rx=0x7ff728008370 tx=0x7ff72800fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:37.732 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:37.714+0000 7ff726ffd700 1 --2- 172.21.15.42:0/2115692113 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff71c03abe0 0x7ff71c03d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:42:37.732 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:37.714+0000 7ff726ffd700 1 -- 172.21.15.42:0/2115692113 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 11707+0+0 (secure 0 0 0) 0x7ff72c04aae0 con 0x7ff734135690 2019-12-05T06:42:37.732 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:37.714+0000 7ff73a37e700 1 --2- 172.21.15.42:0/2115692113 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff71c03abe0 0x7ff71c03d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:42:37.732 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:37.714+0000 7ff73a37e700 1 --2- 172.21.15.42:0/2115692113 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff71c03abe0 0x7ff71c03d0f0 crc :-1 s=READY pgs=1276 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:42:37.733 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:37.722+0000 7ff73bde1700 1 -- 172.21.15.42:0/2115692113 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff73405d110 con 0x7ff734135690 2019-12-05T06:42:37.734 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:37.726+0000 7ff73ab7f700 1 --2- 172.21.15.42:0/2115692113 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff734135690 0x7ff734137ad0 secure :-1 s=THROTTLE_DONE pgs=3045 cs=0 l=1 rx=0x7ff728008370 tx=0x7ff72800fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:37.734 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:37.726+0000 7ff726ffd700 1 -- 172.21.15.42:0/2115692113 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7ff72c031b60 con 0x7ff734135690 2019-12-05T06:42:37.901 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:37.894+0000 7ff73bde1700 1 -- 172.21.15.42:0/2115692113 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff73405b8c0 con 0x7ff734135690 2019-12-05T06:42:37.902 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:37.894+0000 7ff73ab7f700 1 --2- 172.21.15.42:0/2115692113 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff734135690 0x7ff734137ad0 secure :-1 s=THROTTLE_DONE pgs=3045 cs=0 l=1 rx=0x7ff728008370 tx=0x7ff72800fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:37.902 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:37.894+0000 7ff726ffd700 1 -- 172.21.15.42:0/2115692113 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 367 v367) v1 ==== 95+0+2114 (secure 0 0 0) 0x7ff72c031dc0 con 0x7ff734135690 2019-12-05T06:42:37.904 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:42:37.904 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":367,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":12358,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/1922398668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1922398668},{"type":"v1","addr":"172.21.15.42:6835","nonce":1922398668}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":366}],"filesystems":[{"mdsmap":{"epoch":367,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:41:51.221717+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":115,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11976},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11976":{"gid":11976,"name":"b","rank":0,"incarnation":361,"state":"up:active","state_seq":27,"addr":"172.21.15.40:6833/474728597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":474728597},{"type":"v1","addr":"172.21.15.40:6833","nonce":474728597}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:42:37.905 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 367 2019-12-05T06:42:37.908 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:37.898+0000 7ff73bde1700 1 -- 172.21.15.42:0/2115692113 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff71c03abe0 msgr2=0x7ff71c03d0f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:42:37.908 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:37.898+0000 7ff73bde1700 1 --2- 172.21.15.42:0/2115692113 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff71c03abe0 0x7ff71c03d0f0 crc :-1 s=READY pgs=1276 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:42:37.909 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:37.898+0000 7ff73bde1700 1 -- 172.21.15.42:0/2115692113 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff734135690 msgr2=0x7ff734137ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:42:37.909 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:37.898+0000 7ff73bde1700 1 --2- 172.21.15.42:0/2115692113 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff734135690 0x7ff734137ad0 secure :-1 s=READY pgs=3045 cs=0 l=1 rx=0x7ff728008370 tx=0x7ff72800fdb0).stop 2019-12-05T06:42:37.909 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:37.902+0000 7ff73bde1700 1 -- 172.21.15.42:0/2115692113 shutdown_connections 2019-12-05T06:42:37.909 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:37.902+0000 7ff73bde1700 1 --2- 172.21.15.42:0/2115692113 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff71c03abe0 0x7ff71c03d0f0 unknown :-1 s=CLOSED pgs=1276 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:42:37.909 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:37.902+0000 7ff73bde1700 1 --2- 172.21.15.42:0/2115692113 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff734135690 0x7ff734137ad0 unknown :-1 s=CLOSED pgs=3045 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:42:37.910 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:37.902+0000 7ff73bde1700 1 -- 172.21.15.42:0/2115692113 shutdown_connections 2019-12-05T06:42:37.910 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:37.902+0000 7ff73bde1700 1 -- 172.21.15.42:0/2115692113 wait complete. 2019-12-05T06:42:37.910 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:37.902+0000 7ff73bde1700 1 -- 172.21.15.42:0/2115692113 >> 172.21.15.42:0/2115692113 conn(0x7ff73407e680 msgr2=0x7ff73411d340 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:42:37.922 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:42:39.923 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:42:40.045 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:40.030+0000 7f3187ee0700 1 Processor -- start 2019-12-05T06:42:40.046 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:40.030+0000 7f3187ee0700 1 -- start start 2019-12-05T06:42:40.046 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:40.034+0000 7f3187ee0700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f31800a7740 0x7f318008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:42:40.046 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:40.034+0000 7f3187ee0700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f318009fae0 0x7f318007eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:42:40.046 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:40.034+0000 7f3187ee0700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31800a0710 con 0x7f318007e720 2019-12-05T06:42:40.046 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:40.034+0000 7f3187ee0700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31800a0b00 con 0x7f318009fae0 2019-12-05T06:42:40.046 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:40.034+0000 7f3187ee0700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f318008e5b0 con 0x7f31800a7740 2019-12-05T06:42:40.047 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:40.034+0000 7f318647d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f31800a7740 0x7f318008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:42:40.047 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:40.034+0000 7f3185c7c700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f318007e720 0x7f318009f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:33768/0 (socket says 172.21.15.42:33768) 2019-12-05T06:42:40.047 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:40.034+0000 7f3185c7c700 1 -- 172.21.15.42:0/1006500380 learned_addr learned my addr 172.21.15.42:0/1006500380 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:42:40.047 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:40.034+0000 7f3186c7e700 1 --2- 172.21.15.42:0/1006500380 >> v2:172.21.15.40:3300/0 conn(0x7f318009fae0 0x7f318007eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:42:40.047 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:40.034+0000 7f318647d700 1 -- 172.21.15.42:0/1006500380 >> v2:172.21.15.40:3300/0 conn(0x7f318009fae0 msgr2=0x7f318007eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:42:40.047 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:40.034+0000 7f318647d700 1 --2- 172.21.15.42:0/1006500380 >> v2:172.21.15.40:3300/0 conn(0x7f318009fae0 0x7f318007eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:42:40.047 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:40.034+0000 7f318647d700 1 -- 172.21.15.42:0/1006500380 >> v1:172.21.15.42:6789/0 conn(0x7f318007e720 legacy=0x7f318009f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:42:40.048 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:40.034+0000 7f318547b700 1 -- 172.21.15.42:0/1006500380 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3924198903 0 0) 0x7f31800a0710 con 0x7f318007e720 2019-12-05T06:42:40.048 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:40.034+0000 7f318647d700 1 -- 172.21.15.42:0/1006500380 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f318007f0a0 con 0x7f31800a7740 2019-12-05T06:42:40.048 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:40.034+0000 7f318647d700 1 --2- 172.21.15.42:0/1006500380 >> v2:172.21.15.42:3300/0 conn(0x7f31800a7740 0x7f318008df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f317c002f30 tx=0x7f317c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:40.048 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:40.034+0000 7f318647d700 1 --2- 172.21.15.42:0/1006500380 >> v2:172.21.15.42:3300/0 conn(0x7f31800a7740 0x7f318008df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f317c002f30 tx=0x7f317c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:40.049 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:40.034+0000 7f318647d700 1 --2- 172.21.15.42:0/1006500380 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f31800a7740 0x7f318008df80 secure :-1 s=READY pgs=3047 cs=0 l=1 rx=0x7f317c002f30 tx=0x7f317c00fc70).ready entity=mon.1 client_cookie=855e08bcaba95055 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:42:40.049 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:40.034+0000 7f318647d700 1 --2- 172.21.15.42:0/1006500380 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f31800a7740 0x7f318008df80 secure :-1 s=THROTTLE_DONE pgs=3047 cs=0 l=1 rx=0x7f317c002f30 tx=0x7f317c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:40.049 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:40.034+0000 7f318547b700 1 -- 172.21.15.42:0/1006500380 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f317c008bb0 con 0x7f31800a7740 2019-12-05T06:42:40.049 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:40.034+0000 7f318647d700 1 --2- 172.21.15.42:0/1006500380 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f31800a7740 0x7f318008df80 secure :-1 s=THROTTLE_DONE pgs=3047 cs=0 l=1 rx=0x7f317c002f30 tx=0x7f317c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:40.049 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:40.034+0000 7f318647d700 1 --2- 172.21.15.42:0/1006500380 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f31800a7740 0x7f318008df80 secure :-1 s=THROTTLE_DONE pgs=3047 cs=0 l=1 rx=0x7f317c002f30 tx=0x7f317c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:40.050 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:40.034+0000 7f318547b700 1 -- 172.21.15.42:0/1006500380 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f317c0087d0 con 0x7f31800a7740 2019-12-05T06:42:40.050 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:40.034+0000 7f318547b700 1 -- 172.21.15.42:0/1006500380 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f317c011820 con 0x7f31800a7740 2019-12-05T06:42:40.050 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:40.034+0000 7f3187ee0700 1 -- 172.21.15.42:0/1006500380 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f31800a7740 msgr2=0x7f318008df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:42:40.050 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:40.034+0000 7f3187ee0700 1 --2- 172.21.15.42:0/1006500380 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f31800a7740 0x7f318008df80 secure :-1 s=READY pgs=3047 cs=0 l=1 rx=0x7f317c002f30 tx=0x7f317c00fc70).stop 2019-12-05T06:42:40.050 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:40.038+0000 7f3187ee0700 1 -- 172.21.15.42:0/1006500380 shutdown_connections 2019-12-05T06:42:40.050 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:40.038+0000 7f3187ee0700 1 --2- 172.21.15.42:0/1006500380 >> v2:172.21.15.40:3300/0 conn(0x7f318009fae0 0x7f318007eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:42:40.051 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:40.038+0000 7f3187ee0700 1 --2- 172.21.15.42:0/1006500380 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f31800a7740 0x7f318008df80 unknown :-1 s=CLOSED pgs=3047 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:42:40.051 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:40.038+0000 7f3187ee0700 1 -- 172.21.15.42:0/1006500380 shutdown_connections 2019-12-05T06:42:40.051 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:40.038+0000 7f3187ee0700 1 -- 172.21.15.42:0/1006500380 wait complete. 2019-12-05T06:42:40.051 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:40.038+0000 7f3187ee0700 1 -- 172.21.15.42:0/1006500380 >> 172.21.15.42:0/1006500380 conn(0x7f31800aabe0 msgr2=0x7f31800a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:42:40.054 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:40.038+0000 7f3187ee0700 1 Processor -- start 2019-12-05T06:42:40.054 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:40.038+0000 7f3187ee0700 1 -- start start 2019-12-05T06:42:40.054 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:40.038+0000 7f3187ee0700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3180117370 0x7f31800a3720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:42:40.054 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:40.038+0000 7f3187ee0700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3180119010 0x7f3180119440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:42:40.055 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:40.038+0000 7f3187ee0700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31800a3c70 con 0x7f318008f420 2019-12-05T06:42:40.055 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:40.038+0000 7f3187ee0700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3180117b50 con 0x7f3180117370 2019-12-05T06:42:40.055 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:40.038+0000 7f3187ee0700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3180117df0 con 0x7f3180119010 2019-12-05T06:42:40.055 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:40.042+0000 7f318647d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3180117370 0x7f31800a3720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:42:40.055 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:40.042+0000 7f318647d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3180117370 0x7f31800a3720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:33738/0 (socket says 172.21.15.42:33738) 2019-12-05T06:42:40.056 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:40.042+0000 7f318647d700 1 -- 172.21.15.42:0/2927600380 learned_addr learned my addr 172.21.15.42:0/2927600380 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:42:40.056 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:40.042+0000 7f3186c7e700 1 --2- 172.21.15.42:0/2927600380 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3180119010 0x7f3180119440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:42:40.056 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:40.042+0000 7f318647d700 1 -- 172.21.15.42:0/2927600380 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3180119010 msgr2=0x7f3180119440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:42:40.056 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:40.042+0000 7f318647d700 1 --2- 172.21.15.42:0/2927600380 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3180119010 0x7f3180119440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:42:40.056 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:40.042+0000 7f318647d700 1 -- 172.21.15.42:0/2927600380 >> v1:172.21.15.42:6789/0 conn(0x7f318008f420 legacy=0x7f318011b1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:42:40.057 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:40.042+0000 7f318647d700 1 -- 172.21.15.42:0/2927600380 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31801177a0 con 0x7f3180117370 2019-12-05T06:42:40.057 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:40.042+0000 7f318647d700 1 --2- 172.21.15.42:0/2927600380 >> v2:172.21.15.42:3300/0 conn(0x7f3180117370 0x7f31800a3720 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f31800a3ea0 tx=0x7f317c008a40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:40.057 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:40.042+0000 7f318647d700 1 --2- 172.21.15.42:0/2927600380 >> v2:172.21.15.42:3300/0 conn(0x7f3180117370 0x7f31800a3720 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f31800a3ea0 tx=0x7f317c008a40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:40.057 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:40.042+0000 7f318647d700 1 --2- 172.21.15.42:0/2927600380 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3180117370 0x7f31800a3720 secure :-1 s=READY pgs=3049 cs=0 l=1 rx=0x7f31800a3ea0 tx=0x7f317c008a40).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:42:40.057 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:40.042+0000 7f318647d700 1 --2- 172.21.15.42:0/2927600380 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3180117370 0x7f31800a3720 secure :-1 s=THROTTLE_DONE pgs=3049 cs=0 l=1 rx=0x7f31800a3ea0 tx=0x7f317c008a40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:40.057 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:40.042+0000 7f318647d700 1 --2- 172.21.15.42:0/2927600380 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3180117370 0x7f31800a3720 secure :-1 s=THROTTLE_DONE pgs=3049 cs=0 l=1 rx=0x7f31800a3ea0 tx=0x7f317c008a40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:40.058 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:40.042+0000 7f3172ffd700 1 -- 172.21.15.42:0/2927600380 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f317c00f490 con 0x7f3180117370 2019-12-05T06:42:40.058 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:40.042+0000 7f318647d700 1 --2- 172.21.15.42:0/2927600380 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3180117370 0x7f31800a3720 secure :-1 s=THROTTLE_DONE pgs=3049 cs=0 l=1 rx=0x7f31800a3ea0 tx=0x7f317c008a40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:40.058 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:40.042+0000 7f3172ffd700 1 -- 172.21.15.42:0/2927600380 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f317c008db0 con 0x7f3180117370 2019-12-05T06:42:40.058 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:40.042+0000 7f3187ee0700 1 -- 172.21.15.42:0/2927600380 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f318011b9c0 con 0x7f3180117370 2019-12-05T06:42:40.058 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:40.042+0000 7f3172ffd700 1 -- 172.21.15.42:0/2927600380 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f317c005920 con 0x7f3180117370 2019-12-05T06:42:40.058 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:40.042+0000 7f3187ee0700 1 -- 172.21.15.42:0/2927600380 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3180119990 con 0x7f3180117370 2019-12-05T06:42:40.058 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:40.042+0000 7f318647d700 1 --2- 172.21.15.42:0/2927600380 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3180117370 0x7f31800a3720 secure :-1 s=THROTTLE_DONE pgs=3049 cs=0 l=1 rx=0x7f31800a3ea0 tx=0x7f317c008a40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:40.059 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:40.042+0000 7f3172ffd700 1 -- 172.21.15.42:0/2927600380 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f317c0130b0 con 0x7f3180117370 2019-12-05T06:42:40.059 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:40.042+0000 7f318647d700 1 --2- 172.21.15.42:0/2927600380 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3180117370 0x7f31800a3720 secure :-1 s=THROTTLE_DONE pgs=3049 cs=0 l=1 rx=0x7f31800a3ea0 tx=0x7f317c008a40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:40.059 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:40.042+0000 7f3172ffd700 1 --2- 172.21.15.42:0/2927600380 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f316803ad80 0x7f316803d290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:42:40.059 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:40.042+0000 7f3172ffd700 1 -- 172.21.15.42:0/2927600380 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 11707+0+0 (secure 0 0 0) 0x7f317c04b720 con 0x7f3180117370 2019-12-05T06:42:40.059 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:40.046+0000 7f3185c7c700 1 --2- 172.21.15.42:0/2927600380 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f316803ad80 0x7f316803d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:42:40.059 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:40.046+0000 7f3185c7c700 1 --2- 172.21.15.42:0/2927600380 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f316803ad80 0x7f316803d290 crc :-1 s=READY pgs=1277 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:42:40.060 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:40.046+0000 7f3187ee0700 1 -- 172.21.15.42:0/2927600380 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f318005d110 con 0x7f3180117370 2019-12-05T06:42:40.062 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:40.054+0000 7f318647d700 1 --2- 172.21.15.42:0/2927600380 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3180117370 0x7f31800a3720 secure :-1 s=THROTTLE_DONE pgs=3049 cs=0 l=1 rx=0x7f31800a3ea0 tx=0x7f317c008a40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:40.063 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:40.054+0000 7f3172ffd700 1 -- 172.21.15.42:0/2927600380 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f317c0327a0 con 0x7f3180117370 2019-12-05T06:42:40.230 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:40.222+0000 7f3187ee0700 1 -- 172.21.15.42:0/2927600380 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f318005b8c0 con 0x7f3180117370 2019-12-05T06:42:40.231 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:40.222+0000 7f318647d700 1 --2- 172.21.15.42:0/2927600380 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3180117370 0x7f31800a3720 secure :-1 s=THROTTLE_DONE pgs=3049 cs=0 l=1 rx=0x7f31800a3ea0 tx=0x7f317c008a40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:40.231 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:40.222+0000 7f3172ffd700 1 -- 172.21.15.42:0/2927600380 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 367 v367) v1 ==== 95+0+2114 (secure 0 0 0) 0x7f317c005c30 con 0x7f3180117370 2019-12-05T06:42:40.232 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 367 2019-12-05T06:42:40.232 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:42:40.232 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":367,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":12358,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/1922398668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1922398668},{"type":"v1","addr":"172.21.15.42:6835","nonce":1922398668}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":366}],"filesystems":[{"mdsmap":{"epoch":367,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:41:51.221717+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":115,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11976},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11976":{"gid":11976,"name":"b","rank":0,"incarnation":361,"state":"up:active","state_seq":27,"addr":"172.21.15.40:6833/474728597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":474728597},{"type":"v1","addr":"172.21.15.40:6833","nonce":474728597}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:42:40.235 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:40.226+0000 7f3187ee0700 1 -- 172.21.15.42:0/2927600380 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f316803ad80 msgr2=0x7f316803d290 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:42:40.235 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:40.226+0000 7f3187ee0700 1 --2- 172.21.15.42:0/2927600380 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f316803ad80 0x7f316803d290 crc :-1 s=READY pgs=1277 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:42:40.235 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:40.226+0000 7f3187ee0700 1 -- 172.21.15.42:0/2927600380 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3180117370 msgr2=0x7f31800a3720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:42:40.236 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:40.226+0000 7f3187ee0700 1 --2- 172.21.15.42:0/2927600380 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3180117370 0x7f31800a3720 secure :-1 s=READY pgs=3049 cs=0 l=1 rx=0x7f31800a3ea0 tx=0x7f317c008a40).stop 2019-12-05T06:42:40.236 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:40.226+0000 7f3187ee0700 1 -- 172.21.15.42:0/2927600380 shutdown_connections 2019-12-05T06:42:40.236 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:40.226+0000 7f3187ee0700 1 --2- 172.21.15.42:0/2927600380 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f316803ad80 0x7f316803d290 unknown :-1 s=CLOSED pgs=1277 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:42:40.236 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:40.226+0000 7f3187ee0700 1 --2- 172.21.15.42:0/2927600380 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3180119010 0x7f3180119440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:42:40.236 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:40.226+0000 7f3187ee0700 1 --2- 172.21.15.42:0/2927600380 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3180117370 0x7f31800a3720 unknown :-1 s=CLOSED pgs=3049 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:42:40.237 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:40.226+0000 7f3187ee0700 1 -- 172.21.15.42:0/2927600380 shutdown_connections 2019-12-05T06:42:40.237 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:40.226+0000 7f3187ee0700 1 -- 172.21.15.42:0/2927600380 wait complete. 2019-12-05T06:42:40.237 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:40.226+0000 7f3187ee0700 1 -- 172.21.15.42:0/2927600380 >> 172.21.15.42:0/2927600380 conn(0x7f31800a82a0 msgr2=0x7f318011c4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:42:40.243 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:42:42.249 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:42:42.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:42.338+0000 7fccfc862700 1 Processor -- start 2019-12-05T06:42:42.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:42.338+0000 7fccfc862700 1 -- start start 2019-12-05T06:42:42.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:42.338+0000 7fccfc862700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fccf80a7740 0x7fccf808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:42:42.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:42.338+0000 7fccfc862700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fccf809fae0 0x7fccf807eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:42:42.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:42.338+0000 7fccfc862700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccf80a0710 con 0x7fccf807e720 2019-12-05T06:42:42.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:42.338+0000 7fccfc862700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fccf80a0b00 con 0x7fccf809fae0 2019-12-05T06:42:42.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:42.338+0000 7fccfc862700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccf808e5b0 con 0x7fccf80a7740 2019-12-05T06:42:42.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:42.338+0000 7fccf759e700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fccf809fae0 0x7fccf807eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:42:42.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:42.338+0000 7fccee59c700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fccf807e720 0x7fccf809f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:43222/0 (socket says 172.21.15.42:43222) 2019-12-05T06:42:42.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:42.338+0000 7fccee59c700 1 -- 172.21.15.42:0/2926499263 learned_addr learned my addr 172.21.15.42:0/2926499263 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:42:42.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:42.338+0000 7fccf6d9d700 1 --2- 172.21.15.42:0/2926499263 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fccf80a7740 0x7fccf808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:42:42.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:42.342+0000 7fccf759e700 1 -- 172.21.15.42:0/2926499263 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fccf80a7740 msgr2=0x7fccf808df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:42:42.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:42.342+0000 7fccf759e700 1 --2- 172.21.15.42:0/2926499263 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fccf80a7740 0x7fccf808df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:42:42.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:42.342+0000 7fccf759e700 1 -- 172.21.15.42:0/2926499263 >> v1:172.21.15.40:6789/0 conn(0x7fccf807e720 legacy=0x7fccf809f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:42:42.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:42.342+0000 7fccf759e700 1 -- 172.21.15.42:0/2926499263 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccf807f0a0 con 0x7fccf809fae0 2019-12-05T06:42:42.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:42.342+0000 7fccf6d9d700 1 --2- 172.21.15.42:0/2926499263 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fccf80a7740 0x7fccf808df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:42:42.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:42.342+0000 7fccf759e700 1 --2- 172.21.15.42:0/2926499263 >> v2:172.21.15.40:3300/0 conn(0x7fccf809fae0 0x7fccf807eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fccf0008b10 tx=0x7fccf000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:42.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:42.342+0000 7fccf759e700 1 --2- 172.21.15.42:0/2926499263 >> v2:172.21.15.40:3300/0 conn(0x7fccf809fae0 0x7fccf807eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fccf0008b10 tx=0x7fccf000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:42.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:42.342+0000 7fccf759e700 1 --2- 172.21.15.42:0/2926499263 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fccf809fae0 0x7fccf807eb50 secure :-1 s=READY pgs=2311 cs=0 l=1 rx=0x7fccf0008b10 tx=0x7fccf000fff0).ready entity=mon.0 client_cookie=102d9d9f84d880da server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:42:42.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:42.342+0000 7fccf759e700 1 --2- 172.21.15.42:0/2926499263 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fccf809fae0 0x7fccf807eb50 secure :-1 s=THROTTLE_DONE pgs=2311 cs=0 l=1 rx=0x7fccf0008b10 tx=0x7fccf000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:42.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:42.342+0000 7fccf659c700 1 -- 172.21.15.42:0/2926499263 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fccf0010900 con 0x7fccf809fae0 2019-12-05T06:42:42.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:42.342+0000 7fccf759e700 1 --2- 172.21.15.42:0/2926499263 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fccf809fae0 0x7fccf807eb50 secure :-1 s=THROTTLE_DONE pgs=2311 cs=0 l=1 rx=0x7fccf0008b10 tx=0x7fccf000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:42.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:42.342+0000 7fccf759e700 1 --2- 172.21.15.42:0/2926499263 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fccf809fae0 0x7fccf807eb50 secure :-1 s=THROTTLE_DONE pgs=2311 cs=0 l=1 rx=0x7fccf0008b10 tx=0x7fccf000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:42.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:42.342+0000 7fccf659c700 1 -- 172.21.15.42:0/2926499263 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccf0008b50 con 0x7fccf809fae0 2019-12-05T06:42:42.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:42.342+0000 7fccf659c700 1 -- 172.21.15.42:0/2926499263 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fccf00110c0 con 0x7fccf809fae0 2019-12-05T06:42:42.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:42.342+0000 7fccfc862700 1 -- 172.21.15.42:0/2926499263 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fccf809fae0 msgr2=0x7fccf807eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:42:42.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:42.342+0000 7fccfc862700 1 --2- 172.21.15.42:0/2926499263 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fccf809fae0 0x7fccf807eb50 secure :-1 s=READY pgs=2311 cs=0 l=1 rx=0x7fccf0008b10 tx=0x7fccf000fff0).stop 2019-12-05T06:42:42.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:42.342+0000 7fccfc862700 1 -- 172.21.15.42:0/2926499263 shutdown_connections 2019-12-05T06:42:42.358 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:42.342+0000 7fccfc862700 1 --2- 172.21.15.42:0/2926499263 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fccf809fae0 0x7fccf807eb50 unknown :-1 s=CLOSED pgs=2311 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:42:42.358 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:42.342+0000 7fccfc862700 1 --2- 172.21.15.42:0/2926499263 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fccf80a7740 0x7fccf808df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:42:42.358 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:42.342+0000 7fccfc862700 1 -- 172.21.15.42:0/2926499263 shutdown_connections 2019-12-05T06:42:42.358 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:42.342+0000 7fccfc862700 1 -- 172.21.15.42:0/2926499263 wait complete. 2019-12-05T06:42:42.359 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:42.342+0000 7fccfc862700 1 -- 172.21.15.42:0/2926499263 >> 172.21.15.42:0/2926499263 conn(0x7fccf80aabe0 msgr2=0x7fccf80a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:42:42.363 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:42.346+0000 7fccfc862700 1 Processor -- start 2019-12-05T06:42:42.364 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:42.346+0000 7fccfc862700 1 -- start start 2019-12-05T06:42:42.364 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:42.346+0000 7fccfc862700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fccf80a6bc0 0x7fccf811e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:42:42.364 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:42.346+0000 7fccfc862700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fccf81302c0 0x7fccf81191b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:42:42.364 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:42.346+0000 7fccfc862700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccf8119f00 con 0x7fccf81306f0 2019-12-05T06:42:42.364 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:42.346+0000 7fccfc862700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fccf807c810 con 0x7fccf80a6bc0 2019-12-05T06:42:42.365 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:42.346+0000 7fccfc862700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccf807d480 con 0x7fccf81302c0 2019-12-05T06:42:42.365 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:42.346+0000 7fccf759e700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fccf81306f0 0x7fccf8119700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:33792/0 (socket says 172.21.15.42:33792) 2019-12-05T06:42:42.365 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:42.346+0000 7fccf759e700 1 -- 172.21.15.42:0/3234519052 learned_addr learned my addr 172.21.15.42:0/3234519052 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:42:42.365 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:42.346+0000 7fccf6d9d700 1 --2- 172.21.15.42:0/3234519052 >> v2:172.21.15.40:3300/0 conn(0x7fccf80a6bc0 0x7fccf811e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:42:42.365 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:42.346+0000 7fccee59c700 1 --2- 172.21.15.42:0/3234519052 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fccf81302c0 0x7fccf81191b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:42:42.365 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:42.346+0000 7fccee59c700 1 -- 172.21.15.42:0/3234519052 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fccf81302c0 msgr2=0x7fccf81191b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2019-12-05T06:42:42.365 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:42.346+0000 7fccee59c700 1 -- 172.21.15.42:0/3234519052 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fccf81302c0 msgr2=0x7fccf81191b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2019-12-05T06:42:42.366 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:42.346+0000 7fccee59c700 1 --2- 172.21.15.42:0/3234519052 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fccf81302c0 0x7fccf81191b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-12-05T06:42:42.366 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:42.346+0000 7fccee59c700 1 --2- 172.21.15.42:0/3234519052 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fccf81302c0 0x7fccf81191b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rx=0 tx=0)._fault waiting 0.200000 2019-12-05T06:42:42.366 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:42.346+0000 7fcceffff700 1 -- 172.21.15.42:0/3234519052 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1051306482 0 0) 0x7fccf8119f00 con 0x7fccf81306f0 2019-12-05T06:42:42.366 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:42.346+0000 7fcceffff700 1 -- 172.21.15.42:0/3234519052 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fccdc0030b0 con 0x7fccf81306f0 2019-12-05T06:42:42.366 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:42.346+0000 7fccf6d9d700 1 -- 172.21.15.42:0/3234519052 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fccf81302c0 msgr2=0x7fccf81191b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2019-12-05T06:42:42.367 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:42.346+0000 7fccf6d9d700 1 --2- 172.21.15.42:0/3234519052 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fccf81302c0 0x7fccf81191b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:42:42.367 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:42.346+0000 7fccf6d9d700 1 -- 172.21.15.42:0/3234519052 >> v1:172.21.15.42:6789/0 conn(0x7fccf81306f0 legacy=0x7fccf8119700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:42:42.367 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:42.346+0000 7fccf6d9d700 1 -- 172.21.15.42:0/3234519052 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccf807d110 con 0x7fccf80a6bc0 2019-12-05T06:42:42.367 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:42.346+0000 7fccf6d9d700 1 --2- 172.21.15.42:0/3234519052 >> v2:172.21.15.40:3300/0 conn(0x7fccf80a6bc0 0x7fccf811e0a0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fcce8002f30 tx=0x7fcce8010c20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:42.368 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:42.350+0000 7fccf6d9d700 1 --2- 172.21.15.42:0/3234519052 >> v2:172.21.15.40:3300/0 conn(0x7fccf80a6bc0 0x7fccf811e0a0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fcce8002f30 tx=0x7fcce8010c20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:42.368 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:42.350+0000 7fccf6d9d700 1 --2- 172.21.15.42:0/3234519052 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fccf80a6bc0 0x7fccf811e0a0 secure :-1 s=READY pgs=2312 cs=0 l=1 rx=0x7fcce8002f30 tx=0x7fcce8010c20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:42:42.368 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:42.350+0000 7fccf6d9d700 1 --2- 172.21.15.42:0/3234519052 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fccf80a6bc0 0x7fccf811e0a0 secure :-1 s=THROTTLE_DONE pgs=2312 cs=0 l=1 rx=0x7fcce8002f30 tx=0x7fcce8010c20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:42.368 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:42.350+0000 7fcceffff700 1 -- 172.21.15.42:0/3234519052 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcce800d0c0 con 0x7fccf80a6bc0 2019-12-05T06:42:42.368 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:42.350+0000 7fccfc862700 1 -- 172.21.15.42:0/3234519052 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccf811e5f0 con 0x7fccf80a6bc0 2019-12-05T06:42:42.368 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:42.350+0000 7fccf6d9d700 1 --2- 172.21.15.42:0/3234519052 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fccf80a6bc0 0x7fccf811e0a0 secure :-1 s=THROTTLE_DONE pgs=2312 cs=0 l=1 rx=0x7fcce8002f30 tx=0x7fcce8010c20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:42.369 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:42.350+0000 7fccfc862700 1 -- 172.21.15.42:0/3234519052 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccf807b5d0 con 0x7fccf80a6bc0 2019-12-05T06:42:42.369 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:42.350+0000 7fccf6d9d700 1 --2- 172.21.15.42:0/3234519052 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fccf80a6bc0 0x7fccf811e0a0 secure :-1 s=THROTTLE_DONE pgs=2312 cs=0 l=1 rx=0x7fcce8002f30 tx=0x7fcce8010c20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:42.369 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:42.350+0000 7fcceffff700 1 -- 172.21.15.42:0/3234519052 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcce8007150 con 0x7fccf80a6bc0 2019-12-05T06:42:42.369 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:42.350+0000 7fcceffff700 1 -- 172.21.15.42:0/3234519052 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcce8012a00 con 0x7fccf80a6bc0 2019-12-05T06:42:42.369 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:42.350+0000 7fccf6d9d700 1 --2- 172.21.15.42:0/3234519052 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fccf80a6bc0 0x7fccf811e0a0 secure :-1 s=THROTTLE_DONE pgs=2312 cs=0 l=1 rx=0x7fcce8002f30 tx=0x7fcce8010c20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:42.370 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:42.350+0000 7fcceffff700 1 -- 172.21.15.42:0/3234519052 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fcce8013060 con 0x7fccf80a6bc0 2019-12-05T06:42:42.370 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:42.350+0000 7fccf6d9d700 1 --2- 172.21.15.42:0/3234519052 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fccf80a6bc0 0x7fccf811e0a0 secure :-1 s=THROTTLE_DONE pgs=2312 cs=0 l=1 rx=0x7fcce8002f30 tx=0x7fcce8010c20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:42.370 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:42.350+0000 7fcceffff700 1 --2- 172.21.15.42:0/3234519052 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fccdc03b2b0 0x7fccdc03d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:42:42.370 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:42.350+0000 7fcceffff700 1 -- 172.21.15.42:0/3234519052 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 11707+0+0 (secure 0 0 0) 0x7fcce804b010 con 0x7fccf80a6bc0 2019-12-05T06:42:42.371 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:42.350+0000 7fccee59c700 1 --2- 172.21.15.42:0/3234519052 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fccdc03b2b0 0x7fccdc03d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:42:42.371 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:42.350+0000 7fccee59c700 1 --2- 172.21.15.42:0/3234519052 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fccdc03b2b0 0x7fccdc03d7c0 crc :-1 s=READY pgs=1278 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:42:42.371 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:42.354+0000 7fccfc862700 1 -- 172.21.15.42:0/3234519052 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccf805d110 con 0x7fccf80a6bc0 2019-12-05T06:42:42.372 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:42.366+0000 7fccf6d9d700 1 --2- 172.21.15.42:0/3234519052 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fccf80a6bc0 0x7fccf811e0a0 secure :-1 s=THROTTLE_DONE pgs=2312 cs=0 l=1 rx=0x7fcce8002f30 tx=0x7fcce8010c20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:42.372 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:42.366+0000 7fcceffff700 1 -- 172.21.15.42:0/3234519052 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fcce8032070 con 0x7fccf80a6bc0 2019-12-05T06:42:42.548 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:42.534+0000 7fccfc862700 1 -- 172.21.15.42:0/3234519052 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fccf805b8c0 con 0x7fccf80a6bc0 2019-12-05T06:42:42.549 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:42.534+0000 7fccf6d9d700 1 --2- 172.21.15.42:0/3234519052 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fccf80a6bc0 0x7fccf811e0a0 secure :-1 s=THROTTLE_DONE pgs=2312 cs=0 l=1 rx=0x7fcce8002f30 tx=0x7fcce8010c20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:42.549 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:42.534+0000 7fcceffff700 1 -- 172.21.15.42:0/3234519052 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 367 v367) v1 ==== 95+0+2114 (secure 0 0 0) 0x7fcce80322d0 con 0x7fccf80a6bc0 2019-12-05T06:42:42.549 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 367 2019-12-05T06:42:42.550 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:42.538+0000 7fccfc862700 1 -- 172.21.15.42:0/3234519052 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fccdc03b2b0 msgr2=0x7fccdc03d7c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:42:42.550 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:42.538+0000 7fccfc862700 1 --2- 172.21.15.42:0/3234519052 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fccdc03b2b0 0x7fccdc03d7c0 crc :-1 s=READY pgs=1278 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:42:42.550 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:42.538+0000 7fccfc862700 1 -- 172.21.15.42:0/3234519052 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fccf80a6bc0 msgr2=0x7fccf811e0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:42:42.550 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:42.538+0000 7fccfc862700 1 --2- 172.21.15.42:0/3234519052 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fccf80a6bc0 0x7fccf811e0a0 secure :-1 s=READY pgs=2312 cs=0 l=1 rx=0x7fcce8002f30 tx=0x7fcce8010c20).stop 2019-12-05T06:42:42.551 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:42.538+0000 7fccfc862700 1 -- 172.21.15.42:0/3234519052 shutdown_connections 2019-12-05T06:42:42.551 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:42.538+0000 7fccfc862700 1 --2- 172.21.15.42:0/3234519052 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fccdc03b2b0 0x7fccdc03d7c0 unknown :-1 s=CLOSED pgs=1278 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:42:42.551 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:42.538+0000 7fccfc862700 1 --2- 172.21.15.42:0/3234519052 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fccf80a6bc0 0x7fccf811e0a0 unknown :-1 s=CLOSED pgs=2312 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:42:42.551 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:42.538+0000 7fccfc862700 1 --2- 172.21.15.42:0/3234519052 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fccf81302c0 0x7fccf81191b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:42:42.552 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:42.538+0000 7fccfc862700 1 -- 172.21.15.42:0/3234519052 shutdown_connections 2019-12-05T06:42:42.552 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:42.538+0000 7fccfc862700 1 -- 172.21.15.42:0/3234519052 wait complete. 2019-12-05T06:42:42.552 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:42.538+0000 7fccfc862700 1 -- 172.21.15.42:0/3234519052 >> 172.21.15.42:0/3234519052 conn(0x7fccf80a9820 msgr2=0x7fccf809f620 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:42:42.552 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:42:42.553 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":367,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":12358,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/1922398668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1922398668},{"type":"v1","addr":"172.21.15.42:6835","nonce":1922398668}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":366}],"filesystems":[{"mdsmap":{"epoch":367,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:41:51.221717+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":115,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11976},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11976":{"gid":11976,"name":"b","rank":0,"incarnation":361,"state":"up:active","state_seq":27,"addr":"172.21.15.40:6833/474728597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":474728597},{"type":"v1","addr":"172.21.15.40:6833","nonce":474728597}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:42:42.558 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:42:44.558 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:42:44.668 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:44.650+0000 7f942a0cb700 1 Processor -- start 2019-12-05T06:42:44.668 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:44.650+0000 7f942a0cb700 1 -- start start 2019-12-05T06:42:44.668 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:44.650+0000 7f942a0cb700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f94240a7740 0x7f942408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:42:44.668 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:44.650+0000 7f942a0cb700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f942407e720 0x7f942407eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:42:44.668 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:44.650+0000 7f942a0cb700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94240a0710 con 0x7f942409f2e0 2019-12-05T06:42:44.668 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:44.650+0000 7f942a0cb700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94240a0b00 con 0x7f94240a7740 2019-12-05T06:42:44.669 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:44.650+0000 7f942a0cb700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f942408e5b0 con 0x7f942407e720 2019-12-05T06:42:44.669 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:44.650+0000 7f94237fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f942407e720 0x7f942407eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:42:44.669 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:44.650+0000 7f94237fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f942407e720 0x7f942407eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:33760/0 (socket says 172.21.15.42:33760) 2019-12-05T06:42:44.669 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:44.650+0000 7f94237fe700 1 -- 172.21.15.42:0/293663922 learned_addr learned my addr 172.21.15.42:0/293663922 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:42:44.669 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:44.650+0000 7f9423fff700 1 --2- 172.21.15.42:0/293663922 >> v2:172.21.15.40:3300/0 conn(0x7f94240a7740 0x7f942408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:42:44.669 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:44.650+0000 7f94237fe700 1 -- 172.21.15.42:0/293663922 >> v2:172.21.15.40:3300/0 conn(0x7f94240a7740 msgr2=0x7f942408df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:42:44.669 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:44.650+0000 7f94237fe700 1 --2- 172.21.15.42:0/293663922 >> v2:172.21.15.40:3300/0 conn(0x7f94240a7740 0x7f942408df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:42:44.670 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:44.650+0000 7f94237fe700 1 -- 172.21.15.42:0/293663922 >> v1:172.21.15.42:6789/0 conn(0x7f942409f2e0 legacy=0x7f942409f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:42:44.670 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:44.650+0000 7f94237fe700 1 -- 172.21.15.42:0/293663922 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f942407f0a0 con 0x7f942407e720 2019-12-05T06:42:44.670 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:44.650+0000 7f94237fe700 1 --2- 172.21.15.42:0/293663922 >> v2:172.21.15.42:3300/0 conn(0x7f942407e720 0x7f942407eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f940c002f30 tx=0x7f940c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:44.670 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:44.650+0000 7f94237fe700 1 --2- 172.21.15.42:0/293663922 >> v2:172.21.15.42:3300/0 conn(0x7f942407e720 0x7f942407eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f940c002f30 tx=0x7f940c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:44.670 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:44.650+0000 7f94237fe700 1 --2- 172.21.15.42:0/293663922 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f942407e720 0x7f942407eb50 secure :-1 s=READY pgs=3052 cs=0 l=1 rx=0x7f940c002f30 tx=0x7f940c00fd30).ready entity=mon.1 client_cookie=448f2d9915bc56b server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:42:44.670 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:44.650+0000 7f94237fe700 1 --2- 172.21.15.42:0/293663922 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f942407e720 0x7f942407eb50 secure :-1 s=THROTTLE_DONE pgs=3052 cs=0 l=1 rx=0x7f940c002f30 tx=0x7f940c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:44.670 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:44.650+0000 7f94237fe700 1 --2- 172.21.15.42:0/293663922 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f942407e720 0x7f942407eb50 secure :-1 s=THROTTLE_DONE pgs=3052 cs=0 l=1 rx=0x7f940c002f30 tx=0x7f940c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:44.671 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:44.650+0000 7f9422ffd700 1 -- 172.21.15.42:0/293663922 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f940c011370 con 0x7f942407e720 2019-12-05T06:42:44.671 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:44.650+0000 7f94237fe700 1 --2- 172.21.15.42:0/293663922 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f942407e720 0x7f942407eb50 secure :-1 s=THROTTLE_DONE pgs=3052 cs=0 l=1 rx=0x7f940c002f30 tx=0x7f940c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:44.671 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:44.650+0000 7f9422ffd700 1 -- 172.21.15.42:0/293663922 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f940c008bb0 con 0x7f942407e720 2019-12-05T06:42:44.671 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:44.654+0000 7f9422ffd700 1 -- 172.21.15.42:0/293663922 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f940c011950 con 0x7f942407e720 2019-12-05T06:42:44.671 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:44.654+0000 7f942a0cb700 1 -- 172.21.15.42:0/293663922 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f942407e720 msgr2=0x7f942407eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:42:44.671 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:44.654+0000 7f942a0cb700 1 --2- 172.21.15.42:0/293663922 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f942407e720 0x7f942407eb50 secure :-1 s=READY pgs=3052 cs=0 l=1 rx=0x7f940c002f30 tx=0x7f940c00fd30).stop 2019-12-05T06:42:44.672 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:44.654+0000 7f942a0cb700 1 -- 172.21.15.42:0/293663922 shutdown_connections 2019-12-05T06:42:44.672 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:44.654+0000 7f942a0cb700 1 --2- 172.21.15.42:0/293663922 >> v2:172.21.15.40:3300/0 conn(0x7f94240a7740 0x7f942408df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:42:44.672 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:44.654+0000 7f942a0cb700 1 --2- 172.21.15.42:0/293663922 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f942407e720 0x7f942407eb50 unknown :-1 s=CLOSED pgs=3052 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:42:44.672 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:44.654+0000 7f942a0cb700 1 -- 172.21.15.42:0/293663922 shutdown_connections 2019-12-05T06:42:44.672 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:44.654+0000 7f942a0cb700 1 -- 172.21.15.42:0/293663922 wait complete. 2019-12-05T06:42:44.672 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:44.654+0000 7f942a0cb700 1 -- 172.21.15.42:0/293663922 >> 172.21.15.42:0/293663922 conn(0x7f94240aabe0 msgr2=0x7f94240a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:42:44.673 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:44.654+0000 7f942a0cb700 1 Processor -- start 2019-12-05T06:42:44.673 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:44.654+0000 7f942a0cb700 1 -- start start 2019-12-05T06:42:44.673 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:44.654+0000 7f942a0cb700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f942412ed90 0x7f942408ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:42:44.673 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:44.654+0000 7f942a0cb700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f94240a4c50 0x7f94240742d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:42:44.674 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:44.654+0000 7f942a0cb700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94240a56e0 con 0x7f94240a68b0 2019-12-05T06:42:44.674 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:44.654+0000 7f942a0cb700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94240a5440 con 0x7f94240a4c50 2019-12-05T06:42:44.674 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:44.654+0000 7f942a0cb700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94240a5080 con 0x7f942412ed90 2019-12-05T06:42:44.674 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:44.654+0000 7f9423fff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f94240a68b0 0x7f942412e590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:33804/0 (socket says 172.21.15.42:33804) 2019-12-05T06:42:44.674 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:44.654+0000 7f9423fff700 1 -- 172.21.15.42:0/1860225871 learned_addr learned my addr 172.21.15.42:0/1860225871 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:42:44.674 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:44.654+0000 7f9428e69700 1 --2- 172.21.15.42:0/1860225871 >> v2:172.21.15.42:3300/0 conn(0x7f94240a4c50 0x7f94240742d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:42:44.675 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:44.658+0000 7f94237fe700 1 --2- 172.21.15.42:0/1860225871 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f942412ed90 0x7f942408ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:42:44.675 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:44.658+0000 7f9420ff9700 1 -- 172.21.15.42:0/1860225871 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1617765439 0 0) 0x7f94240a56e0 con 0x7f94240a68b0 2019-12-05T06:42:44.675 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:44.658+0000 7f9420ff9700 1 -- 172.21.15.42:0/1860225871 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93fc0030b0 con 0x7f94240a68b0 2019-12-05T06:42:44.675 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:44.658+0000 7f9428e69700 1 -- 172.21.15.42:0/1860225871 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f942412ed90 msgr2=0x7f942408ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:42:44.676 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:44.658+0000 7f9428e69700 1 --2- 172.21.15.42:0/1860225871 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f942412ed90 0x7f942408ecf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:42:44.676 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:44.658+0000 7f9428e69700 1 -- 172.21.15.42:0/1860225871 >> v1:172.21.15.42:6789/0 conn(0x7f94240a68b0 legacy=0x7f942412e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:42:44.676 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:44.658+0000 7f9420ff9700 1 -- 172.21.15.42:0/1860225871 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 3741886093 0 0) 0x7f93fc0030b0 con 0x7f94240a68b0 2019-12-05T06:42:44.676 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:44.658+0000 7f9428e69700 1 -- 172.21.15.42:0/1860225871 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f942413f060 con 0x7f94240a4c50 2019-12-05T06:42:44.677 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:44.658+0000 7f94237fe700 1 --2- 172.21.15.42:0/1860225871 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f942412ed90 0x7f942408ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:42:44.677 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:44.658+0000 7f9428e69700 1 --2- 172.21.15.42:0/1860225871 >> v2:172.21.15.42:3300/0 conn(0x7f94240a4c50 0x7f94240742d0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f94140016d0 tx=0x7f9414002f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:44.677 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:44.658+0000 7f9428e69700 1 --2- 172.21.15.42:0/1860225871 >> v2:172.21.15.42:3300/0 conn(0x7f94240a4c50 0x7f94240742d0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f94140016d0 tx=0x7f9414002f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:44.677 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:44.658+0000 7f9428e69700 1 --2- 172.21.15.42:0/1860225871 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f94240a4c50 0x7f94240742d0 secure :-1 s=READY pgs=3054 cs=0 l=1 rx=0x7f94140016d0 tx=0x7f9414002f70).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:42:44.677 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:44.658+0000 7f9428e69700 1 --2- 172.21.15.42:0/1860225871 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f94240a4c50 0x7f94240742d0 secure :-1 s=THROTTLE_DONE pgs=3054 cs=0 l=1 rx=0x7f94140016d0 tx=0x7f9414002f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:44.677 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:44.658+0000 7f9420ff9700 1 -- 172.21.15.42:0/1860225871 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9414001490 con 0x7f94240a4c50 2019-12-05T06:42:44.677 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:44.658+0000 7f9428e69700 1 --2- 172.21.15.42:0/1860225871 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f94240a4c50 0x7f94240742d0 secure :-1 s=THROTTLE_DONE pgs=3054 cs=0 l=1 rx=0x7f94140016d0 tx=0x7f9414002f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:44.678 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:44.658+0000 7f9428e69700 1 --2- 172.21.15.42:0/1860225871 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f94240a4c50 0x7f94240742d0 secure :-1 s=THROTTLE_DONE pgs=3054 cs=0 l=1 rx=0x7f94140016d0 tx=0x7f9414002f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:44.678 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:44.658+0000 7f9420ff9700 1 -- 172.21.15.42:0/1860225871 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94140012a0 con 0x7f94240a4c50 2019-12-05T06:42:44.678 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:44.658+0000 7f942a0cb700 1 -- 172.21.15.42:0/1860225871 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9424074820 con 0x7f94240a4c50 2019-12-05T06:42:44.678 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:44.658+0000 7f9420ff9700 1 -- 172.21.15.42:0/1860225871 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f941c002220 con 0x7f94240a4c50 2019-12-05T06:42:44.678 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:44.658+0000 7f942a0cb700 1 -- 172.21.15.42:0/1860225871 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9424074dd0 con 0x7f94240a4c50 2019-12-05T06:42:44.678 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:44.658+0000 7f9428e69700 1 --2- 172.21.15.42:0/1860225871 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f94240a4c50 0x7f94240742d0 secure :-1 s=THROTTLE_DONE pgs=3054 cs=0 l=1 rx=0x7f94140016d0 tx=0x7f9414002f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:44.678 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:44.658+0000 7f9420ff9700 1 -- 172.21.15.42:0/1860225871 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f941c002530 con 0x7f94240a4c50 2019-12-05T06:42:44.679 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:44.658+0000 7f9428e69700 1 --2- 172.21.15.42:0/1860225871 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f94240a4c50 0x7f94240742d0 secure :-1 s=THROTTLE_DONE pgs=3054 cs=0 l=1 rx=0x7f94140016d0 tx=0x7f9414002f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:44.679 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:44.658+0000 7f9420ff9700 1 --2- 172.21.15.42:0/1860225871 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f93fc03b280 0x7f93fc03d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:42:44.679 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:44.658+0000 7f9420ff9700 1 -- 172.21.15.42:0/1860225871 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 11707+0+0 (secure 0 0 0) 0x7f941c04abf0 con 0x7f94240a4c50 2019-12-05T06:42:44.679 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:44.658+0000 7f9423fff700 1 --2- 172.21.15.42:0/1860225871 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f93fc03b280 0x7f93fc03d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:42:44.680 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:44.662+0000 7f9423fff700 1 --2- 172.21.15.42:0/1860225871 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f93fc03b280 0x7f93fc03d790 crc :-1 s=READY pgs=1279 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:42:44.680 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:44.670+0000 7f942a0cb700 1 -- 172.21.15.42:0/1860225871 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f942405d110 con 0x7f94240a4c50 2019-12-05T06:42:44.681 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:44.674+0000 7f9428e69700 1 --2- 172.21.15.42:0/1860225871 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f94240a4c50 0x7f94240742d0 secure :-1 s=THROTTLE_DONE pgs=3054 cs=0 l=1 rx=0x7f94140016d0 tx=0x7f9414002f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:44.682 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:44.674+0000 7f9420ff9700 1 -- 172.21.15.42:0/1860225871 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f941c031c70 con 0x7f94240a4c50 2019-12-05T06:42:44.850 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:44.842+0000 7f942a0cb700 1 -- 172.21.15.42:0/1860225871 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f942405b8c0 con 0x7f94240a4c50 2019-12-05T06:42:44.852 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:44.842+0000 7f9428e69700 1 --2- 172.21.15.42:0/1860225871 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f94240a4c50 0x7f94240742d0 secure :-1 s=THROTTLE_DONE pgs=3054 cs=0 l=1 rx=0x7f94140016d0 tx=0x7f9414002f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:44.852 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:44.842+0000 7f9420ff9700 1 -- 172.21.15.42:0/1860225871 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 367 v367) v1 ==== 95+0+2114 (secure 0 0 0) 0x7f941c031ed0 con 0x7f94240a4c50 2019-12-05T06:42:44.853 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:42:44.853 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":367,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":12358,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/1922398668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1922398668},{"type":"v1","addr":"172.21.15.42:6835","nonce":1922398668}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":366}],"filesystems":[{"mdsmap":{"epoch":367,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:41:51.221717+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":115,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11976},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11976":{"gid":11976,"name":"b","rank":0,"incarnation":361,"state":"up:active","state_seq":27,"addr":"172.21.15.40:6833/474728597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":474728597},{"type":"v1","addr":"172.21.15.40:6833","nonce":474728597}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:42:44.854 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 367 2019-12-05T06:42:44.856 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:44.846+0000 7f942a0cb700 1 -- 172.21.15.42:0/1860225871 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f93fc03b280 msgr2=0x7f93fc03d790 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:42:44.856 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:44.846+0000 7f942a0cb700 1 --2- 172.21.15.42:0/1860225871 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f93fc03b280 0x7f93fc03d790 crc :-1 s=READY pgs=1279 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:42:44.856 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:44.846+0000 7f942a0cb700 1 -- 172.21.15.42:0/1860225871 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f94240a4c50 msgr2=0x7f94240742d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:42:44.856 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:44.846+0000 7f942a0cb700 1 --2- 172.21.15.42:0/1860225871 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f94240a4c50 0x7f94240742d0 secure :-1 s=READY pgs=3054 cs=0 l=1 rx=0x7f94140016d0 tx=0x7f9414002f70).stop 2019-12-05T06:42:44.856 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:44.846+0000 7f942a0cb700 1 -- 172.21.15.42:0/1860225871 shutdown_connections 2019-12-05T06:42:44.856 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:44.846+0000 7f942a0cb700 1 --2- 172.21.15.42:0/1860225871 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f93fc03b280 0x7f93fc03d790 unknown :-1 s=CLOSED pgs=1279 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:42:44.857 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:44.846+0000 7f942a0cb700 1 --2- 172.21.15.42:0/1860225871 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f94240a4c50 0x7f94240742d0 unknown :-1 s=CLOSED pgs=3054 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:42:44.857 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:44.846+0000 7f942a0cb700 1 --2- 172.21.15.42:0/1860225871 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f942412ed90 0x7f942408ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:42:44.857 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:44.846+0000 7f942a0cb700 1 -- 172.21.15.42:0/1860225871 shutdown_connections 2019-12-05T06:42:44.857 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:44.846+0000 7f942a0cb700 1 -- 172.21.15.42:0/1860225871 wait complete. 2019-12-05T06:42:44.857 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:44.846+0000 7f942a0cb700 1 -- 172.21.15.42:0/1860225871 >> 172.21.15.42:0/1860225871 conn(0x7f94240a8110 msgr2=0x7f94240a97e0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:42:44.869 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:42:45.424 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:42:45.429 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:42:46.872 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:42:46.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:46.970+0000 7ff70288c700 1 Processor -- start 2019-12-05T06:42:46.978 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:46.970+0000 7ff70288c700 1 -- start start 2019-12-05T06:42:46.978 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:46.970+0000 7ff70288c700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff6fc08df80 0x7ff6fc09f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:42:46.978 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:46.970+0000 7ff70288c700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6fc09fc60 con 0x7ff6fc0a7740 2019-12-05T06:42:46.978 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:46.970+0000 7ff70288c700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6fc0a0710 con 0x7ff6fc09f830 2019-12-05T06:42:46.978 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:46.970+0000 7ff70288c700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6fc0a0b00 con 0x7ff6fc08df80 2019-12-05T06:42:46.978 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:46.970+0000 7ff70162a700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7ff6fc09f830 0x7ff6fc14efe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:33830/0 (socket says 172.21.15.42:33830) 2019-12-05T06:42:46.979 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:46.970+0000 7ff70162a700 1 -- 172.21.15.42:0/614625717 learned_addr learned my addr 172.21.15.42:0/614625717 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:42:46.979 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:46.970+0000 7ff6fbfff700 1 --2- 172.21.15.42:0/614625717 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff6fc08df80 0x7ff6fc09f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:42:46.979 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:46.970+0000 7ff6fb7fe700 1 -- 172.21.15.42:0/614625717 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3115399393 0 0) 0x7ff6fc0a0710 con 0x7ff6fc09f830 2019-12-05T06:42:46.979 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:46.970+0000 7ff6fb7fe700 1 -- 172.21.15.42:0/614625717 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6e80030b0 con 0x7ff6fc09f830 2019-12-05T06:42:46.979 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:46.970+0000 7ff6fb7fe700 1 -- 172.21.15.42:0/614625717 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1990915484 0 0) 0x7ff6e80030b0 con 0x7ff6fc09f830 2019-12-05T06:42:46.980 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:46.970+0000 7ff6fb7fe700 1 -- 172.21.15.42:0/614625717 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff6fc08df80 msgr2=0x7ff6fc09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:42:46.980 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:46.970+0000 7ff6fb7fe700 1 --2- 172.21.15.42:0/614625717 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff6fc08df80 0x7ff6fc09f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:42:46.980 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:46.970+0000 7ff6fb7fe700 1 -- 172.21.15.42:0/614625717 >> v1:172.21.15.40:6789/0 conn(0x7ff6fc0a7740 legacy=0x7ff6fc07e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:42:46.980 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:46.970+0000 7ff6fb7fe700 1 -- 172.21.15.42:0/614625717 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6fc07ef20 con 0x7ff6fc09f830 2019-12-05T06:42:46.980 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:46.970+0000 7ff6fb7fe700 1 -- 172.21.15.42:0/614625717 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7ff6f4001ad0 con 0x7ff6fc09f830 2019-12-05T06:42:46.981 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:46.970+0000 7ff6fb7fe700 1 -- 172.21.15.42:0/614625717 >> v1:172.21.15.42:6789/0 conn(0x7ff6fc09f830 legacy=0x7ff6fc14efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:42:46.981 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:46.970+0000 7ff6fb7fe700 1 --2- 172.21.15.42:0/614625717 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff6e8003710 0x7ff6e8003b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:42:46.981 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:46.970+0000 7ff6fb7fe700 1 --2- 172.21.15.42:0/614625717 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff6e8004420 0x7ff6e8004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:42:46.981 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:46.970+0000 7ff6fb7fe700 1 --2- 172.21.15.42:0/614625717 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff6e8004da0 0x7ff6e800b670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:42:46.981 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:46.970+0000 7ff6fb7fe700 1 -- 172.21.15.42:0/614625717 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6fc0a0b00 con 0x7ff6e8004420 2019-12-05T06:42:46.981 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:46.970+0000 7ff6fb7fe700 1 -- 172.21.15.42:0/614625717 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6e8005240 con 0x7ff6e8003710 2019-12-05T06:42:46.981 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:46.970+0000 7ff6fb7fe700 1 -- 172.21.15.42:0/614625717 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6e804fc20 con 0x7ff6e8004da0 2019-12-05T06:42:46.982 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:46.970+0000 7ff700e29700 1 --2- 172.21.15.42:0/614625717 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff6e8003710 0x7ff6e8003b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:42:46.982 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:46.970+0000 7ff70162a700 1 --2- 172.21.15.42:0/614625717 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff6e8004da0 0x7ff6e800b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:42:46.982 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:46.970+0000 7ff6fbfff700 1 --2- 172.21.15.42:0/614625717 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff6e8004420 0x7ff6e8004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:42:46.983 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:46.970+0000 7ff700e29700 1 -- 172.21.15.42:0/614625717 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff6e8004da0 msgr2=0x7ff6e800b670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:42:46.983 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:46.970+0000 7ff700e29700 1 --2- 172.21.15.42:0/614625717 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff6e8004da0 0x7ff6e800b670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:42:46.983 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:46.970+0000 7ff700e29700 1 -- 172.21.15.42:0/614625717 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff6e8004420 msgr2=0x7ff6e8004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:42:46.983 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:46.970+0000 7ff700e29700 1 --2- 172.21.15.42:0/614625717 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff6e8004420 0x7ff6e8004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:42:46.984 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:46.970+0000 7ff700e29700 1 -- 172.21.15.42:0/614625717 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6e804ff20 con 0x7ff6e8003710 2019-12-05T06:42:46.984 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:46.970+0000 7ff70162a700 1 --2- 172.21.15.42:0/614625717 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff6e8004da0 0x7ff6e800b670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:42:46.984 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:46.970+0000 7ff700e29700 1 --2- 172.21.15.42:0/614625717 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff6e8003710 0x7ff6e8003b80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7ff6ec001000 tx=0x7ff6ec00fcd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:46.984 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:46.970+0000 7ff700e29700 1 --2- 172.21.15.42:0/614625717 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff6e8003710 0x7ff6e8003b80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7ff6ec001000 tx=0x7ff6ec00fcd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:46.984 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:46.970+0000 7ff700e29700 1 --2- 172.21.15.42:0/614625717 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff6e8003710 0x7ff6e8003b80 secure :-1 s=READY pgs=3056 cs=0 l=1 rx=0x7ff6ec001000 tx=0x7ff6ec00fcd0).ready entity=mon.1 client_cookie=4e9104efeb739490 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:42:46.984 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:46.970+0000 7ff700e29700 1 --2- 172.21.15.42:0/614625717 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff6e8003710 0x7ff6e8003b80 secure :-1 s=THROTTLE_DONE pgs=3056 cs=0 l=1 rx=0x7ff6ec001000 tx=0x7ff6ec00fcd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:46.985 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:46.970+0000 7ff6fb7fe700 1 -- 172.21.15.42:0/614625717 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff6ec003540 con 0x7ff6e8003710 2019-12-05T06:42:46.985 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:46.970+0000 7ff700e29700 1 --2- 172.21.15.42:0/614625717 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff6e8003710 0x7ff6e8003b80 secure :-1 s=THROTTLE_DONE pgs=3056 cs=0 l=1 rx=0x7ff6ec001000 tx=0x7ff6ec00fcd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:46.985 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:46.970+0000 7ff700e29700 1 --2- 172.21.15.42:0/614625717 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff6e8003710 0x7ff6e8003b80 secure :-1 s=THROTTLE_DONE pgs=3056 cs=0 l=1 rx=0x7ff6ec001000 tx=0x7ff6ec00fcd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:46.985 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:46.970+0000 7ff6fb7fe700 1 -- 172.21.15.42:0/614625717 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6ec003160 con 0x7ff6e8003710 2019-12-05T06:42:46.985 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:46.970+0000 7ff6fb7fe700 1 -- 172.21.15.42:0/614625717 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff6ec003b20 con 0x7ff6e8003710 2019-12-05T06:42:46.985 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:46.970+0000 7ff70288c700 1 -- 172.21.15.42:0/614625717 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff6e8003710 msgr2=0x7ff6e8003b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:42:46.986 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:46.970+0000 7ff70288c700 1 --2- 172.21.15.42:0/614625717 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff6e8003710 0x7ff6e8003b80 secure :-1 s=READY pgs=3056 cs=0 l=1 rx=0x7ff6ec001000 tx=0x7ff6ec00fcd0).stop 2019-12-05T06:42:46.986 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:46.970+0000 7ff70162a700 1 -- 172.21.15.42:0/614625717 reap_dead start 2019-12-05T06:42:46.986 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:46.974+0000 7ff70288c700 1 -- 172.21.15.42:0/614625717 shutdown_connections 2019-12-05T06:42:46.986 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:46.974+0000 7ff70288c700 1 -- 172.21.15.42:0/614625717 shutdown_connections 2019-12-05T06:42:46.986 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:46.974+0000 7ff70288c700 1 -- 172.21.15.42:0/614625717 wait complete. 2019-12-05T06:42:46.986 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:46.974+0000 7ff70288c700 1 -- 172.21.15.42:0/614625717 >> 172.21.15.42:0/614625717 conn(0x7ff6fc0aabe0 msgr2=0x7ff6fc0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:42:46.990 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:46.974+0000 7ff70288c700 1 Processor -- start 2019-12-05T06:42:46.990 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:46.974+0000 7ff70288c700 1 -- start start 2019-12-05T06:42:46.991 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:46.974+0000 7ff70288c700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff6fc123a80 0x7ff6fc11e100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:42:46.991 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:46.974+0000 7ff70288c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff6fc1248d0 0x7ff6fc138e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:42:46.991 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:46.974+0000 7ff70288c700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6fc0a59c0 con 0x7ff6fc120b10 2019-12-05T06:42:46.991 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:46.974+0000 7ff70288c700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6fc07b6f0 con 0x7ff6fc1248d0 2019-12-05T06:42:46.991 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:46.974+0000 7ff70288c700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6fc11dbc0 con 0x7ff6fc123a80 2019-12-05T06:42:46.991 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:46.974+0000 7ff70162a700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff6fc1248d0 0x7ff6fc138e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:42:46.992 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:46.974+0000 7ff6fbfff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7ff6fc120b10 0x7ff6fc1240d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:33838/0 (socket says 172.21.15.42:33838) 2019-12-05T06:42:46.992 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:46.974+0000 7ff6fbfff700 1 -- 172.21.15.42:0/2935792880 learned_addr learned my addr 172.21.15.42:0/2935792880 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:42:46.992 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:46.974+0000 7ff700e29700 1 --2- 172.21.15.42:0/2935792880 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff6fc123a80 0x7ff6fc11e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:42:46.992 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:46.974+0000 7ff70162a700 1 -- 172.21.15.42:0/2935792880 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff6fc123a80 msgr2=0x7ff6fc11e100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:42:46.992 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:46.974+0000 7ff70162a700 1 --2- 172.21.15.42:0/2935792880 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff6fc123a80 0x7ff6fc11e100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:42:46.993 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:46.974+0000 7ff6f97fa700 1 -- 172.21.15.42:0/2935792880 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3653250893 0 0) 0x7ff6fc0a59c0 con 0x7ff6fc120b10 2019-12-05T06:42:46.993 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:46.978+0000 7ff70162a700 1 -- 172.21.15.42:0/2935792880 >> v1:172.21.15.42:6789/0 conn(0x7ff6fc120b10 legacy=0x7ff6fc1240d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:42:46.993 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:46.978+0000 7ff70162a700 1 -- 172.21.15.42:0/2935792880 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6fc05ae30 con 0x7ff6fc1248d0 2019-12-05T06:42:46.994 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:46.978+0000 7ff700e29700 1 --2- 172.21.15.42:0/2935792880 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff6fc123a80 0x7ff6fc11e100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:42:46.994 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:46.978+0000 7ff70162a700 1 --2- 172.21.15.42:0/2935792880 >> v2:172.21.15.42:3300/0 conn(0x7ff6fc1248d0 0x7ff6fc138e20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7ff6f40017a0 tx=0x7ff6f400ca00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:46.994 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:46.978+0000 7ff70162a700 1 --2- 172.21.15.42:0/2935792880 >> v2:172.21.15.42:3300/0 conn(0x7ff6fc1248d0 0x7ff6fc138e20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7ff6f40017a0 tx=0x7ff6f400ca00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:46.994 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:46.978+0000 7ff70162a700 1 --2- 172.21.15.42:0/2935792880 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff6fc1248d0 0x7ff6fc138e20 secure :-1 s=READY pgs=3058 cs=0 l=1 rx=0x7ff6f40017a0 tx=0x7ff6f400ca00).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:42:46.994 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:46.978+0000 7ff70162a700 1 --2- 172.21.15.42:0/2935792880 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff6fc1248d0 0x7ff6fc138e20 secure :-1 s=THROTTLE_DONE pgs=3058 cs=0 l=1 rx=0x7ff6f40017a0 tx=0x7ff6f400ca00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:46.995 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:46.978+0000 7ff6f97fa700 1 -- 172.21.15.42:0/2935792880 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff6f4004b60 con 0x7ff6fc1248d0 2019-12-05T06:42:46.995 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:46.978+0000 7ff70162a700 1 --2- 172.21.15.42:0/2935792880 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff6fc1248d0 0x7ff6fc138e20 secure :-1 s=THROTTLE_DONE pgs=3058 cs=0 l=1 rx=0x7ff6f40017a0 tx=0x7ff6f400ca00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:46.995 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:46.978+0000 7ff70162a700 1 --2- 172.21.15.42:0/2935792880 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff6fc1248d0 0x7ff6fc138e20 secure :-1 s=THROTTLE_DONE pgs=3058 cs=0 l=1 rx=0x7ff6f40017a0 tx=0x7ff6f400ca00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:46.995 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:46.978+0000 7ff6f97fa700 1 -- 172.21.15.42:0/2935792880 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6f4004780 con 0x7ff6fc1248d0 2019-12-05T06:42:46.995 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:46.978+0000 7ff70288c700 1 -- 172.21.15.42:0/2935792880 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6fc0a4f40 con 0x7ff6fc1248d0 2019-12-05T06:42:46.995 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:46.978+0000 7ff6f97fa700 1 -- 172.21.15.42:0/2935792880 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff6f400d2b0 con 0x7ff6fc1248d0 2019-12-05T06:42:46.996 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:46.978+0000 7ff70288c700 1 -- 172.21.15.42:0/2935792880 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6fc120f40 con 0x7ff6fc1248d0 2019-12-05T06:42:46.996 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:46.978+0000 7ff70162a700 1 --2- 172.21.15.42:0/2935792880 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff6fc1248d0 0x7ff6fc138e20 secure :-1 s=THROTTLE_DONE pgs=3058 cs=0 l=1 rx=0x7ff6f40017a0 tx=0x7ff6f400ca00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:46.996 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:46.978+0000 7ff6f97fa700 1 -- 172.21.15.42:0/2935792880 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7ff6f4013060 con 0x7ff6fc1248d0 2019-12-05T06:42:46.996 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:46.978+0000 7ff70162a700 1 --2- 172.21.15.42:0/2935792880 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff6fc1248d0 0x7ff6fc138e20 secure :-1 s=THROTTLE_DONE pgs=3058 cs=0 l=1 rx=0x7ff6f40017a0 tx=0x7ff6f400ca00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:46.996 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:46.978+0000 7ff6f97fa700 1 --2- 172.21.15.42:0/2935792880 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff6e003ac70 0x7ff6e003d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:42:46.998 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:46.978+0000 7ff6f97fa700 1 -- 172.21.15.42:0/2935792880 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 11793+0+0 (secure 0 0 0) 0x7ff6f404ab10 con 0x7ff6fc1248d0 2019-12-05T06:42:46.998 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:46.978+0000 7ff700e29700 1 --2- 172.21.15.42:0/2935792880 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff6e003ac70 0x7ff6e003d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:42:46.998 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:46.978+0000 7ff700e29700 1 --2- 172.21.15.42:0/2935792880 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff6e003ac70 0x7ff6e003d180 crc :-1 s=READY pgs=1280 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:42:46.998 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:46.986+0000 7ff70288c700 1 -- 172.21.15.42:0/2935792880 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6fc05d110 con 0x7ff6fc1248d0 2019-12-05T06:42:47.000 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:46.990+0000 7ff70162a700 1 --2- 172.21.15.42:0/2935792880 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff6fc1248d0 0x7ff6fc138e20 secure :-1 s=THROTTLE_DONE pgs=3058 cs=0 l=1 rx=0x7ff6f40017a0 tx=0x7ff6f400ca00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:47.000 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:46.990+0000 7ff6f97fa700 1 -- 172.21.15.42:0/2935792880 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7ff6f4031b90 con 0x7ff6fc1248d0 2019-12-05T06:42:47.161 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:47.158+0000 7ff70288c700 1 -- 172.21.15.42:0/2935792880 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff6fc05b8c0 con 0x7ff6fc1248d0 2019-12-05T06:42:47.162 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:47.158+0000 7ff70162a700 1 --2- 172.21.15.42:0/2935792880 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff6fc1248d0 0x7ff6fc138e20 secure :-1 s=THROTTLE_DONE pgs=3058 cs=0 l=1 rx=0x7ff6f40017a0 tx=0x7ff6f400ca00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:47.162 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:47.158+0000 7ff6f97fa700 1 -- 172.21.15.42:0/2935792880 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 368 v368) v1 ==== 95+0+1778 (secure 0 0 0) 0x7ff6f4031df0 con 0x7ff6fc1248d0 2019-12-05T06:42:47.162 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 368 2019-12-05T06:42:47.162 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:42:47.162 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":368,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":368,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:42:45.235691+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":116,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12358},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12358":{"gid":12358,"name":"a","rank":0,"incarnation":368,"state":"up:replay","state_seq":2,"addr":"172.21.15.42:6835/1922398668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1922398668},{"type":"v1","addr":"172.21.15.42:6835","nonce":1922398668}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:42:47.168 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:47.162+0000 7ff70288c700 1 -- 172.21.15.42:0/2935792880 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff6e003ac70 msgr2=0x7ff6e003d180 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:42:47.169 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:47.162+0000 7ff70288c700 1 --2- 172.21.15.42:0/2935792880 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff6e003ac70 0x7ff6e003d180 crc :-1 s=READY pgs=1280 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:42:47.169 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:47.162+0000 7ff70288c700 1 -- 172.21.15.42:0/2935792880 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff6fc1248d0 msgr2=0x7ff6fc138e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:42:47.169 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:47.162+0000 7ff70288c700 1 --2- 172.21.15.42:0/2935792880 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff6fc1248d0 0x7ff6fc138e20 secure :-1 s=READY pgs=3058 cs=0 l=1 rx=0x7ff6f40017a0 tx=0x7ff6f400ca00).stop 2019-12-05T06:42:47.169 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:47.162+0000 7ff70288c700 1 -- 172.21.15.42:0/2935792880 shutdown_connections 2019-12-05T06:42:47.169 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:47.162+0000 7ff70288c700 1 --2- 172.21.15.42:0/2935792880 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7ff6e003ac70 0x7ff6e003d180 unknown :-1 s=CLOSED pgs=1280 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:42:47.170 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:47.162+0000 7ff70288c700 1 --2- 172.21.15.42:0/2935792880 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff6fc1248d0 0x7ff6fc138e20 unknown :-1 s=CLOSED pgs=3058 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:42:47.170 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:47.162+0000 7ff70288c700 1 --2- 172.21.15.42:0/2935792880 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff6fc123a80 0x7ff6fc11e100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:42:47.170 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:47.162+0000 7ff70288c700 1 -- 172.21.15.42:0/2935792880 shutdown_connections 2019-12-05T06:42:47.170 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:47.162+0000 7ff70288c700 1 -- 172.21.15.42:0/2935792880 wait complete. 2019-12-05T06:42:47.171 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:47.162+0000 7ff70288c700 1 -- 172.21.15.42:0/2935792880 >> 172.21.15.42:0/2935792880 conn(0x7ff6fc0a0f50 msgr2=0x7ff6fc07e450 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:42:47.180 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b down, removed from mdsmap 2019-12-05T06:42:47.180 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2019-12-05T06:42:47.181 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:42:47.292 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:47.282+0000 7f6e8f3d7700 1 Processor -- start 2019-12-05T06:42:47.292 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:47.282+0000 7f6e8f3d7700 1 -- start start 2019-12-05T06:42:47.292 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:47.282+0000 7f6e8f3d7700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f6e880a7740 0x7f6e8808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:42:47.292 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:47.282+0000 7f6e8f3d7700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6e8809fae0 0x7f6e8807eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:42:47.293 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:47.282+0000 7f6e8f3d7700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e880a0710 con 0x7f6e8807e720 2019-12-05T06:42:47.293 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:47.282+0000 7f6e8f3d7700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e880a0b00 con 0x7f6e880a7740 2019-12-05T06:42:47.293 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:47.282+0000 7f6e8f3d7700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e8808e5b0 con 0x7f6e8809fae0 2019-12-05T06:42:47.293 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:47.286+0000 7f6e8e175700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6e8809fae0 0x7f6e8807eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:42:47.294 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:47.286+0000 7f6e8d974700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f6e880a7740 0x7f6e8808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:42:47.295 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:47.286+0000 7f6e8e175700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6e8809fae0 0x7f6e8807eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:33812/0 (socket says 172.21.15.42:33812) 2019-12-05T06:42:47.296 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:47.286+0000 7f6e8e175700 1 -- 172.21.15.42:0/774045224 learned_addr learned my addr 172.21.15.42:0/774045224 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:42:47.296 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:47.286+0000 7f6e8e175700 1 -- 172.21.15.42:0/774045224 >> v2:172.21.15.40:3300/0 conn(0x7f6e880a7740 msgr2=0x7f6e8808df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:42:47.296 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:47.286+0000 7f6e8e175700 1 --2- 172.21.15.42:0/774045224 >> v2:172.21.15.40:3300/0 conn(0x7f6e880a7740 0x7f6e8808df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:42:47.296 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:47.286+0000 7f6e8e175700 1 -- 172.21.15.42:0/774045224 >> v1:172.21.15.40:6789/0 conn(0x7f6e8807e720 legacy=0x7f6e8809f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:42:47.297 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:47.286+0000 7f6e8e175700 1 -- 172.21.15.42:0/774045224 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e8807f0a0 con 0x7f6e8809fae0 2019-12-05T06:42:47.297 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:47.286+0000 7f6e8d974700 1 --2- 172.21.15.42:0/774045224 >> v2:172.21.15.40:3300/0 conn(0x7f6e880a7740 0x7f6e8808df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:42:47.297 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:47.286+0000 7f6e8e175700 1 --2- 172.21.15.42:0/774045224 >> v2:172.21.15.42:3300/0 conn(0x7f6e8809fae0 0x7f6e8807eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f6e800032b0 tx=0x7f6e800100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:47.298 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:47.286+0000 7f6e8e175700 1 --2- 172.21.15.42:0/774045224 >> v2:172.21.15.42:3300/0 conn(0x7f6e8809fae0 0x7f6e8807eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f6e800032b0 tx=0x7f6e800100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:47.298 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:47.286+0000 7f6e8e175700 1 --2- 172.21.15.42:0/774045224 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6e8809fae0 0x7f6e8807eb50 secure :-1 s=READY pgs=3059 cs=0 l=1 rx=0x7f6e800032b0 tx=0x7f6e800100b0).ready entity=mon.1 client_cookie=413e6ca4cda4de92 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:42:47.298 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:47.286+0000 7f6e8e175700 1 --2- 172.21.15.42:0/774045224 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6e8809fae0 0x7f6e8807eb50 secure :-1 s=THROTTLE_DONE pgs=3059 cs=0 l=1 rx=0x7f6e800032b0 tx=0x7f6e800100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:47.298 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:47.286+0000 7f6e8c972700 1 -- 172.21.15.42:0/774045224 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6e80010d60 con 0x7f6e8809fae0 2019-12-05T06:42:47.299 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:47.286+0000 7f6e8e175700 1 --2- 172.21.15.42:0/774045224 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6e8809fae0 0x7f6e8807eb50 secure :-1 s=THROTTLE_DONE pgs=3059 cs=0 l=1 rx=0x7f6e800032b0 tx=0x7f6e800100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:47.299 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:47.286+0000 7f6e8e175700 1 --2- 172.21.15.42:0/774045224 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6e8809fae0 0x7f6e8807eb50 secure :-1 s=THROTTLE_DONE pgs=3059 cs=0 l=1 rx=0x7f6e800032b0 tx=0x7f6e800100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:47.299 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:47.286+0000 7f6e8c972700 1 -- 172.21.15.42:0/774045224 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e80010980 con 0x7f6e8809fae0 2019-12-05T06:42:47.299 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:47.286+0000 7f6e8c972700 1 -- 172.21.15.42:0/774045224 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6e800114a0 con 0x7f6e8809fae0 2019-12-05T06:42:47.300 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:47.286+0000 7f6e8f3d7700 1 -- 172.21.15.42:0/774045224 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6e8809fae0 msgr2=0x7f6e8807eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:42:47.300 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:47.286+0000 7f6e8f3d7700 1 --2- 172.21.15.42:0/774045224 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6e8809fae0 0x7f6e8807eb50 secure :-1 s=READY pgs=3059 cs=0 l=1 rx=0x7f6e800032b0 tx=0x7f6e800100b0).stop 2019-12-05T06:42:47.303 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:47.286+0000 7f6e8f3d7700 1 -- 172.21.15.42:0/774045224 shutdown_connections 2019-12-05T06:42:47.304 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:47.286+0000 7f6e8f3d7700 1 --2- 172.21.15.42:0/774045224 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6e8809fae0 0x7f6e8807eb50 unknown :-1 s=CLOSED pgs=3059 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:42:47.304 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:47.286+0000 7f6e8f3d7700 1 --2- 172.21.15.42:0/774045224 >> v2:172.21.15.40:3300/0 conn(0x7f6e880a7740 0x7f6e8808df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:42:47.304 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:47.286+0000 7f6e8f3d7700 1 -- 172.21.15.42:0/774045224 shutdown_connections 2019-12-05T06:42:47.305 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:47.286+0000 7f6e8f3d7700 1 -- 172.21.15.42:0/774045224 wait complete. 2019-12-05T06:42:47.305 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:47.286+0000 7f6e8f3d7700 1 -- 172.21.15.42:0/774045224 >> 172.21.15.42:0/774045224 conn(0x7f6e880aabe0 msgr2=0x7f6e880a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:42:47.320 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:47.290+0000 7f6e8f3d7700 1 Processor -- start 2019-12-05T06:42:47.320 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:47.290+0000 7f6e8f3d7700 1 -- start start 2019-12-05T06:42:47.320 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:47.290+0000 7f6e8f3d7700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6e88123c00 0x7f6e88127960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:42:47.321 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:47.290+0000 7f6e8f3d7700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6e88147820 0x7f6e88147c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:42:47.321 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:47.290+0000 7f6e8f3d7700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e881457a0 con 0x7f6e880a1b90 2019-12-05T06:42:47.321 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:47.290+0000 7f6e8f3d7700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e881271c0 con 0x7f6e88147820 2019-12-05T06:42:47.321 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:47.290+0000 7f6e8f3d7700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e88127420 con 0x7f6e88123c00 2019-12-05T06:42:47.321 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:47.290+0000 7f6e8e175700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6e88147820 0x7f6e88147c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:42:47.321 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:47.290+0000 7f6e8e175700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6e88147820 0x7f6e88147c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:33816/0 (socket says 172.21.15.42:33816) 2019-12-05T06:42:47.322 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:47.290+0000 7f6e8e175700 1 -- 172.21.15.42:0/682885735 learned_addr learned my addr 172.21.15.42:0/682885735 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:42:47.322 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:47.290+0000 7f6e8d974700 1 --2- 172.21.15.42:0/682885735 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6e88123c00 0x7f6e88127960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:42:47.322 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:47.290+0000 7f6e8e175700 1 -- 172.21.15.42:0/682885735 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6e88123c00 msgr2=0x7f6e88127960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:42:47.322 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:47.290+0000 7f6e8e175700 1 --2- 172.21.15.42:0/682885735 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6e88123c00 0x7f6e88127960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:42:47.322 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:47.290+0000 7f6e8e175700 1 -- 172.21.15.42:0/682885735 >> v1:172.21.15.40:6789/0 conn(0x7f6e880a1b90 legacy=0x7f6e8811db80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:42:47.322 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:47.290+0000 7f6e8e175700 1 -- 172.21.15.42:0/682885735 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e88127eb0 con 0x7f6e88147820 2019-12-05T06:42:47.322 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:47.290+0000 7f6e8e175700 1 --2- 172.21.15.42:0/682885735 >> v2:172.21.15.42:3300/0 conn(0x7f6e88147820 0x7f6e88147c50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f6e881275e0 tx=0x7f6e80010cb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:47.323 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:47.290+0000 7f6e8e175700 1 --2- 172.21.15.42:0/682885735 >> v2:172.21.15.42:3300/0 conn(0x7f6e88147820 0x7f6e88147c50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f6e881275e0 tx=0x7f6e80010cb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:47.323 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:47.290+0000 7f6e8e175700 1 --2- 172.21.15.42:0/682885735 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6e88147820 0x7f6e88147c50 secure :-1 s=READY pgs=3060 cs=0 l=1 rx=0x7f6e881275e0 tx=0x7f6e80010cb0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:42:47.323 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:47.290+0000 7f6e8e175700 1 --2- 172.21.15.42:0/682885735 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6e88147820 0x7f6e88147c50 secure :-1 s=THROTTLE_DONE pgs=3060 cs=0 l=1 rx=0x7f6e881275e0 tx=0x7f6e80010cb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:47.323 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:47.290+0000 7f6e7e7fc700 1 -- 172.21.15.42:0/682885735 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6e80011bb0 con 0x7f6e88147820 2019-12-05T06:42:47.323 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:47.290+0000 7f6e8e175700 1 --2- 172.21.15.42:0/682885735 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6e88147820 0x7f6e88147c50 secure :-1 s=THROTTLE_DONE pgs=3060 cs=0 l=1 rx=0x7f6e881275e0 tx=0x7f6e80010cb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:47.323 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:47.290+0000 7f6e8e175700 1 --2- 172.21.15.42:0/682885735 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6e88147820 0x7f6e88147c50 secure :-1 s=THROTTLE_DONE pgs=3060 cs=0 l=1 rx=0x7f6e881275e0 tx=0x7f6e80010cb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:47.324 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:47.290+0000 7f6e7e7fc700 1 -- 172.21.15.42:0/682885735 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e800110c0 con 0x7f6e88147820 2019-12-05T06:42:47.324 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:47.290+0000 7f6e8f3d7700 1 -- 172.21.15.42:0/682885735 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e8812e610 con 0x7f6e88147820 2019-12-05T06:42:47.324 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:47.290+0000 7f6e7e7fc700 1 -- 172.21.15.42:0/682885735 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6e80005ab0 con 0x7f6e88147820 2019-12-05T06:42:47.324 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:47.290+0000 7f6e8f3d7700 1 -- 172.21.15.42:0/682885735 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e8812ebc0 con 0x7f6e88147820 2019-12-05T06:42:47.324 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:47.290+0000 7f6e8e175700 1 --2- 172.21.15.42:0/682885735 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6e88147820 0x7f6e88147c50 secure :-1 s=THROTTLE_DONE pgs=3060 cs=0 l=1 rx=0x7f6e881275e0 tx=0x7f6e80010cb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:47.325 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:47.290+0000 7f6e7e7fc700 1 -- 172.21.15.42:0/682885735 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f6e8000f280 con 0x7f6e88147820 2019-12-05T06:42:47.325 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:47.290+0000 7f6e8e175700 1 --2- 172.21.15.42:0/682885735 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6e88147820 0x7f6e88147c50 secure :-1 s=THROTTLE_DONE pgs=3060 cs=0 l=1 rx=0x7f6e881275e0 tx=0x7f6e80010cb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:47.325 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:47.294+0000 7f6e7e7fc700 1 --2- 172.21.15.42:0/682885735 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6e7003ad20 0x7f6e7003d230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:42:47.325 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:47.294+0000 7f6e7e7fc700 1 -- 172.21.15.42:0/682885735 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 11793+0+0 (secure 0 0 0) 0x7f6e8004b2d0 con 0x7f6e88147820 2019-12-05T06:42:47.325 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:47.294+0000 7f6e8d974700 1 --2- 172.21.15.42:0/682885735 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6e7003ad20 0x7f6e7003d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:42:47.325 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:47.294+0000 7f6e8d974700 1 --2- 172.21.15.42:0/682885735 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6e7003ad20 0x7f6e7003d230 crc :-1 s=READY pgs=1281 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:42:47.325 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:47.298+0000 7f6e8f3d7700 1 -- 172.21.15.42:0/682885735 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e8805d110 con 0x7f6e88147820 2019-12-05T06:42:47.326 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:47.302+0000 7f6e8e175700 1 --2- 172.21.15.42:0/682885735 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6e88147820 0x7f6e88147c50 secure :-1 s=THROTTLE_DONE pgs=3060 cs=0 l=1 rx=0x7f6e881275e0 tx=0x7f6e80010cb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:47.326 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:47.302+0000 7f6e7e7fc700 1 -- 172.21.15.42:0/682885735 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f6e80005d90 con 0x7f6e88147820 2019-12-05T06:42:47.485 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:47.482+0000 7f6e8f3d7700 1 -- 172.21.15.42:0/682885735 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6e880a71e0 con 0x7f6e88147820 2019-12-05T06:42:47.486 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:47.482+0000 7f6e8e175700 1 --2- 172.21.15.42:0/682885735 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6e88147820 0x7f6e88147c50 secure :-1 s=THROTTLE_DONE pgs=3060 cs=0 l=1 rx=0x7f6e881275e0 tx=0x7f6e80010cb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:47.486 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:47.482+0000 7f6e7e7fc700 1 -- 172.21.15.42:0/682885735 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 368 v368) v1 ==== 95+0+1778 (secure 0 0 0) 0x7f6e800082b0 con 0x7f6e88147820 2019-12-05T06:42:47.486 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 368 2019-12-05T06:42:47.487 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:42:47.487 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":368,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":368,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:42:45.235691+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":116,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12358},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12358":{"gid":12358,"name":"a","rank":0,"incarnation":368,"state":"up:replay","state_seq":2,"addr":"172.21.15.42:6835/1922398668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1922398668},{"type":"v1","addr":"172.21.15.42:6835","nonce":1922398668}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:42:47.491 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:47.486+0000 7f6e8f3d7700 1 -- 172.21.15.42:0/682885735 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6e7003ad20 msgr2=0x7f6e7003d230 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:42:47.492 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:47.486+0000 7f6e8f3d7700 1 --2- 172.21.15.42:0/682885735 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6e7003ad20 0x7f6e7003d230 crc :-1 s=READY pgs=1281 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:42:47.492 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:47.486+0000 7f6e8f3d7700 1 -- 172.21.15.42:0/682885735 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6e88147820 msgr2=0x7f6e88147c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:42:47.492 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:47.486+0000 7f6e8f3d7700 1 --2- 172.21.15.42:0/682885735 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6e88147820 0x7f6e88147c50 secure :-1 s=READY pgs=3060 cs=0 l=1 rx=0x7f6e881275e0 tx=0x7f6e80010cb0).stop 2019-12-05T06:42:47.492 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:47.486+0000 7f6e8f3d7700 1 -- 172.21.15.42:0/682885735 shutdown_connections 2019-12-05T06:42:47.492 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:47.486+0000 7f6e8f3d7700 1 --2- 172.21.15.42:0/682885735 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6e7003ad20 0x7f6e7003d230 unknown :-1 s=CLOSED pgs=1281 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:42:47.493 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:47.486+0000 7f6e8f3d7700 1 --2- 172.21.15.42:0/682885735 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6e88147820 0x7f6e88147c50 unknown :-1 s=CLOSED pgs=3060 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:42:47.493 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:47.486+0000 7f6e8f3d7700 1 --2- 172.21.15.42:0/682885735 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6e88123c00 0x7f6e88127960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:42:47.493 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:47.486+0000 7f6e8f3d7700 1 -- 172.21.15.42:0/682885735 shutdown_connections 2019-12-05T06:42:47.493 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:47.486+0000 7f6e8f3d7700 1 -- 172.21.15.42:0/682885735 wait complete. 2019-12-05T06:42:47.493 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:47.486+0000 7f6e8f3d7700 1 -- 172.21.15.42:0/682885735 >> 172.21.15.42:0/682885735 conn(0x7f6e8807de90 msgr2=0x7f6e880a9790 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:42:47.503 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 368, "feature_flags": { "enable_multiple": false, "ever_enabled_multiple": false }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2019-12-05T05:03:35.604738+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 368, "ever_allowed_features": 2, "explicitly_allowed_features": 2, "failed": [], "flags": 18, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_12358": { "addr": "172.21.15.42:6835/1922398668", "addrs": { "addrvec": [ { "addr": "172.21.15.42:6834", "nonce": 1922398668, "type": "v2" }, { "addr": "172.21.15.42:6835", "nonce": 1922398668, "type": "v1" } ] }, "export_targets": [], "features": 4540138292836696063, "flags": 0, "gid": 12358, "incarnation": 368, "name": "a", "rank": 0, "state": "up:replay", "state_seq": 2 } }, "last_failure": 0, "last_failure_osd_epoch": 116, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 2, "min_compat_client": "0 (unknown)", "modified": "2019-12-05T06:42:45.235691+0000", "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 12358 } } } ], "standbys": [] } 2019-12-05T06:42:49.504 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:42:49.613 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:49.590+0000 7efc8b2b2700 1 Processor -- start 2019-12-05T06:42:49.613 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:49.590+0000 7efc8b2b2700 1 -- start start 2019-12-05T06:42:49.614 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:49.594+0000 7efc8b2b2700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7efc840a7740 0x7efc8408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:42:49.614 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:49.594+0000 7efc8b2b2700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc840a0710 con 0x7efc8409fae0 2019-12-05T06:42:49.614 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:49.594+0000 7efc8b2b2700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc840a9ba0 con 0x7efc8407e720 2019-12-05T06:42:49.614 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:49.594+0000 7efc8b2b2700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc840a0b00 con 0x7efc840a7740 2019-12-05T06:42:49.614 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:49.594+0000 7efc8904e700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7efc8407e720 0x7efc8409f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:33864/0 (socket says 172.21.15.42:33864) 2019-12-05T06:42:49.614 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:49.594+0000 7efc8904e700 1 -- 172.21.15.42:0/1845320145 learned_addr learned my addr 172.21.15.42:0/1845320145 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:42:49.614 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:49.594+0000 7efc8984f700 1 --2- 172.21.15.42:0/1845320145 >> v2:172.21.15.40:3300/0 conn(0x7efc840a7740 0x7efc8408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:42:49.615 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:49.594+0000 7efc8884d700 1 -- 172.21.15.42:0/1845320145 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2139279750 0 0) 0x7efc840a9ba0 con 0x7efc8407e720 2019-12-05T06:42:49.615 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:49.594+0000 7efc8884d700 1 -- 172.21.15.42:0/1845320145 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc700030b0 con 0x7efc8407e720 2019-12-05T06:42:49.615 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:49.594+0000 7efc8884d700 1 -- 172.21.15.42:0/1845320145 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 936378593 0 0) 0x7efc700030b0 con 0x7efc8407e720 2019-12-05T06:42:49.615 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:49.594+0000 7efc8884d700 1 -- 172.21.15.42:0/1845320145 >> v2:172.21.15.40:3300/0 conn(0x7efc840a7740 msgr2=0x7efc8408df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:42:49.615 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:49.594+0000 7efc8884d700 1 --2- 172.21.15.42:0/1845320145 >> v2:172.21.15.40:3300/0 conn(0x7efc840a7740 0x7efc8408df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:42:49.615 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:49.594+0000 7efc8884d700 1 -- 172.21.15.42:0/1845320145 >> v1:172.21.15.40:6789/0 conn(0x7efc8409fae0 legacy=0x7efc8414efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:42:49.615 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:49.594+0000 7efc8884d700 1 -- 172.21.15.42:0/1845320145 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc8407eb50 con 0x7efc8407e720 2019-12-05T06:42:49.616 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:49.594+0000 7efc8884d700 1 -- 172.21.15.42:0/1845320145 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7efc74001750 con 0x7efc8407e720 2019-12-05T06:42:49.623 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:49.594+0000 7efc8884d700 1 -- 172.21.15.42:0/1845320145 >> v1:172.21.15.42:6789/0 conn(0x7efc8407e720 legacy=0x7efc8409f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:42:49.623 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:49.594+0000 7efc8884d700 1 --2- 172.21.15.42:0/1845320145 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efc700036e0 0x7efc70003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:42:49.623 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:49.594+0000 7efc8884d700 1 --2- 172.21.15.42:0/1845320145 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efc70004420 0x7efc70004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:42:49.623 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:49.594+0000 7efc8884d700 1 --2- 172.21.15.42:0/1845320145 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efc70004da0 0x7efc7000b670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:42:49.623 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:49.594+0000 7efc8884d700 1 -- 172.21.15.42:0/1845320145 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc840a0b00 con 0x7efc70004420 2019-12-05T06:42:49.624 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:49.594+0000 7efc8a050700 1 --2- 172.21.15.42:0/1845320145 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efc70004da0 0x7efc7000b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:42:49.624 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:49.594+0000 7efc8884d700 1 -- 172.21.15.42:0/1845320145 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc7004fc60 con 0x7efc70004da0 2019-12-05T06:42:49.624 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:49.594+0000 7efc8884d700 1 -- 172.21.15.42:0/1845320145 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc7004fea0 con 0x7efc700036e0 2019-12-05T06:42:49.624 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:49.594+0000 7efc8984f700 1 --2- 172.21.15.42:0/1845320145 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efc700036e0 0x7efc70003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:42:49.624 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:49.594+0000 7efc8904e700 1 --2- 172.21.15.42:0/1845320145 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efc70004420 0x7efc70004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:42:49.624 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:49.594+0000 7efc8a050700 1 -- 172.21.15.42:0/1845320145 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efc700036e0 msgr2=0x7efc70003b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:42:49.624 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:49.594+0000 7efc8a050700 1 --2- 172.21.15.42:0/1845320145 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efc700036e0 0x7efc70003b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:42:49.625 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:49.594+0000 7efc8a050700 1 -- 172.21.15.42:0/1845320145 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efc70004420 msgr2=0x7efc70004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:42:49.625 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:49.594+0000 7efc8a050700 1 --2- 172.21.15.42:0/1845320145 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efc70004420 0x7efc70004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:42:49.625 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:49.594+0000 7efc8a050700 1 -- 172.21.15.42:0/1845320145 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc700501a0 con 0x7efc70004da0 2019-12-05T06:42:49.625 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:49.594+0000 7efc8904e700 1 --2- 172.21.15.42:0/1845320145 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efc70004420 0x7efc70004850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:42:49.625 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:49.594+0000 7efc8a050700 1 --2- 172.21.15.42:0/1845320145 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efc70004da0 0x7efc7000b670 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7efc7c001380 tx=0x7efc7c010050).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:49.625 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:49.594+0000 7efc8a050700 1 -- 172.21.15.42:0/1845320145 reap_dead start 2019-12-05T06:42:49.625 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:49.594+0000 7efc8a050700 1 --2- 172.21.15.42:0/1845320145 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efc70004da0 0x7efc7000b670 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7efc7c001380 tx=0x7efc7c010050).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:49.626 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:49.594+0000 7efc8a050700 1 --2- 172.21.15.42:0/1845320145 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efc70004da0 0x7efc7000b670 secure :-1 s=READY pgs=3063 cs=0 l=1 rx=0x7efc7c001380 tx=0x7efc7c010050).ready entity=mon.1 client_cookie=f92033337d8575d server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:42:49.626 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:49.594+0000 7efc8a050700 1 --2- 172.21.15.42:0/1845320145 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efc70004da0 0x7efc7000b670 secure :-1 s=THROTTLE_DONE pgs=3063 cs=0 l=1 rx=0x7efc7c001380 tx=0x7efc7c010050).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:49.626 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:49.594+0000 7efc8884d700 1 -- 172.21.15.42:0/1845320145 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efc7c006ac0 con 0x7efc70004da0 2019-12-05T06:42:49.626 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:49.594+0000 7efc8a050700 1 --2- 172.21.15.42:0/1845320145 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efc70004da0 0x7efc7000b670 secure :-1 s=THROTTLE_DONE pgs=3063 cs=0 l=1 rx=0x7efc7c001380 tx=0x7efc7c010050).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:49.626 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:49.594+0000 7efc8884d700 1 -- 172.21.15.42:0/1845320145 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc7c0066e0 con 0x7efc70004da0 2019-12-05T06:42:49.626 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:49.594+0000 7efc8a050700 1 --2- 172.21.15.42:0/1845320145 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efc70004da0 0x7efc7000b670 secure :-1 s=THROTTLE_DONE pgs=3063 cs=0 l=1 rx=0x7efc7c001380 tx=0x7efc7c010050).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:49.626 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:49.594+0000 7efc8884d700 1 -- 172.21.15.42:0/1845320145 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efc7c002750 con 0x7efc70004da0 2019-12-05T06:42:49.627 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:49.594+0000 7efc8b2b2700 1 -- 172.21.15.42:0/1845320145 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efc70004da0 msgr2=0x7efc7000b670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:42:49.627 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:49.594+0000 7efc8b2b2700 1 --2- 172.21.15.42:0/1845320145 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efc70004da0 0x7efc7000b670 secure :-1 s=READY pgs=3063 cs=0 l=1 rx=0x7efc7c001380 tx=0x7efc7c010050).stop 2019-12-05T06:42:49.627 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:49.594+0000 7efc8b2b2700 1 -- 172.21.15.42:0/1845320145 shutdown_connections 2019-12-05T06:42:49.627 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:49.594+0000 7efc8b2b2700 1 --2- 172.21.15.42:0/1845320145 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efc70004da0 0x7efc7000b670 unknown :-1 s=CLOSED pgs=3063 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:42:49.631 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:49.594+0000 7efc8b2b2700 1 -- 172.21.15.42:0/1845320145 shutdown_connections 2019-12-05T06:42:49.631 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:49.594+0000 7efc8b2b2700 1 -- 172.21.15.42:0/1845320145 wait complete. 2019-12-05T06:42:49.631 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:49.594+0000 7efc8b2b2700 1 -- 172.21.15.42:0/1845320145 >> 172.21.15.42:0/1845320145 conn(0x7efc840aabe0 msgr2=0x7efc840a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:42:49.632 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:49.598+0000 7efc8b2b2700 1 Processor -- start 2019-12-05T06:42:49.632 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:49.598+0000 7efc8b2b2700 1 -- start start 2019-12-05T06:42:49.632 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:49.598+0000 7efc8b2b2700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7efc84132dd0 0x7efc84135210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:42:49.633 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:49.598+0000 7efc8b2b2700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc840a5700 con 0x7efc84067320 2019-12-05T06:42:49.633 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:49.598+0000 7efc8b2b2700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc840a6a20 con 0x7efc8407b370 2019-12-05T06:42:49.633 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:49.598+0000 7efc8b2b2700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc840a4840 con 0x7efc84132dd0 2019-12-05T06:42:49.633 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:49.598+0000 7efc8904e700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7efc8407b370 0x7efc8407b7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:33878/0 (socket says 172.21.15.42:33878) 2019-12-05T06:42:49.633 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:49.598+0000 7efc8904e700 1 -- 172.21.15.42:0/2061810027 learned_addr learned my addr 172.21.15.42:0/2061810027 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:42:49.633 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:49.598+0000 7efc8a050700 1 --2- 172.21.15.42:0/2061810027 >> v2:172.21.15.40:3300/0 conn(0x7efc84132dd0 0x7efc84135210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:42:49.633 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:49.598+0000 7efc7a7fc700 1 -- 172.21.15.42:0/2061810027 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2306819207 0 0) 0x7efc840a6a20 con 0x7efc8407b370 2019-12-05T06:42:49.634 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:49.598+0000 7efc7a7fc700 1 -- 172.21.15.42:0/2061810027 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc6c0030b0 con 0x7efc8407b370 2019-12-05T06:42:49.634 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:49.598+0000 7efc7a7fc700 1 -- 172.21.15.42:0/2061810027 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2428966841 0 0) 0x7efc840a5700 con 0x7efc84067320 2019-12-05T06:42:49.634 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:49.598+0000 7efc7a7fc700 1 -- 172.21.15.42:0/2061810027 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc840a6a20 con 0x7efc84067320 2019-12-05T06:42:49.634 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:49.598+0000 7efc7a7fc700 1 -- 172.21.15.42:0/2061810027 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 899787826 0 0) 0x7efc6c0030b0 con 0x7efc8407b370 2019-12-05T06:42:49.634 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:49.598+0000 7efc7a7fc700 1 -- 172.21.15.42:0/2061810027 --> v1:172.21.15.42:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efc840a5700 con 0x7efc8407b370 2019-12-05T06:42:49.634 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:49.598+0000 7efc7a7fc700 1 -- 172.21.15.42:0/2061810027 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7efc7400a6c0 con 0x7efc8407b370 2019-12-05T06:42:49.634 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:49.598+0000 7efc7a7fc700 1 -- 172.21.15.42:0/2061810027 >> v2:172.21.15.40:3300/0 conn(0x7efc84132dd0 msgr2=0x7efc84135210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:42:49.635 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:49.598+0000 7efc7a7fc700 1 --2- 172.21.15.42:0/2061810027 >> v2:172.21.15.40:3300/0 conn(0x7efc84132dd0 0x7efc84135210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:42:49.635 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:49.598+0000 7efc7a7fc700 1 -- 172.21.15.42:0/2061810027 >> v1:172.21.15.42:6789/0 conn(0x7efc8407b370 legacy=0x7efc8407b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:42:49.635 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:49.598+0000 7efc7a7fc700 1 -- 172.21.15.42:0/2061810027 >> v1:172.21.15.40:6789/0 conn(0x7efc84067320 legacy=0x7efc840a4f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:42:49.635 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:49.598+0000 7efc7a7fc700 1 --2- 172.21.15.42:0/2061810027 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efc6c003ab0 0x7efc6c004420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:42:49.635 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:49.598+0000 7efc7a7fc700 1 --2- 172.21.15.42:0/2061810027 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efc6c004970 0x7efc6c004da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:42:49.635 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:49.602+0000 7efc7a7fc700 1 --2- 172.21.15.42:0/2061810027 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efc6c009660 0x7efc6c00baa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:42:49.640 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:49.602+0000 7efc7a7fc700 1 -- 172.21.15.42:0/2061810027 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc840a4840 con 0x7efc6c003ab0 2019-12-05T06:42:49.640 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:49.602+0000 7efc7a7fc700 1 -- 172.21.15.42:0/2061810027 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc6c00bff0 con 0x7efc6c009660 2019-12-05T06:42:49.641 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:49.602+0000 7efc7a7fc700 1 -- 172.21.15.42:0/2061810027 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc6c00c1b0 con 0x7efc6c004970 2019-12-05T06:42:49.641 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:49.602+0000 7efc8a050700 1 --2- 172.21.15.42:0/2061810027 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efc6c009660 0x7efc6c00baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:42:49.641 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:49.602+0000 7efc8984f700 1 --2- 172.21.15.42:0/2061810027 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efc6c003ab0 0x7efc6c004420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:42:49.641 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:49.602+0000 7efc8904e700 1 --2- 172.21.15.42:0/2061810027 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efc6c004970 0x7efc6c004da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:42:49.641 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:49.602+0000 7efc8a050700 1 -- 172.21.15.42:0/2061810027 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efc6c004970 msgr2=0x7efc6c004da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:42:49.641 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:49.602+0000 7efc8a050700 1 --2- 172.21.15.42:0/2061810027 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efc6c004970 0x7efc6c004da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:42:49.641 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:49.602+0000 7efc8a050700 1 -- 172.21.15.42:0/2061810027 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efc6c003ab0 msgr2=0x7efc6c004420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:42:49.642 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:49.602+0000 7efc8a050700 1 --2- 172.21.15.42:0/2061810027 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efc6c003ab0 0x7efc6c004420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:42:49.642 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:49.602+0000 7efc8a050700 1 -- 172.21.15.42:0/2061810027 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc8405ae30 con 0x7efc6c009660 2019-12-05T06:42:49.642 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:49.602+0000 7efc8b2b2700 1 -- 172.21.15.42:0/2061810027 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc840a2f40 con 0x7efc6c009660 2019-12-05T06:42:49.642 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:49.602+0000 7efc8a050700 1 --2- 172.21.15.42:0/2061810027 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efc6c009660 0x7efc6c00baa0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7efc7c001380 tx=0x7efc7c011960).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:49.642 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:49.602+0000 7efc8a050700 1 -- 172.21.15.42:0/2061810027 reap_dead start 2019-12-05T06:42:49.642 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:49.602+0000 7efc8b2b2700 1 -- 172.21.15.42:0/2061810027 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc840a6140 con 0x7efc6c009660 2019-12-05T06:42:49.642 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:49.602+0000 7efc8a050700 1 --2- 172.21.15.42:0/2061810027 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efc6c009660 0x7efc6c00baa0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7efc7c001380 tx=0x7efc7c011960).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:49.643 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:49.602+0000 7efc8a050700 1 --2- 172.21.15.42:0/2061810027 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efc6c009660 0x7efc6c00baa0 secure :-1 s=READY pgs=3065 cs=0 l=1 rx=0x7efc7c001380 tx=0x7efc7c011960).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:42:49.643 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:49.602+0000 7efc8a050700 1 --2- 172.21.15.42:0/2061810027 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efc6c009660 0x7efc6c00baa0 secure :-1 s=THROTTLE_DONE pgs=3065 cs=0 l=1 rx=0x7efc7c001380 tx=0x7efc7c011960).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:49.643 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:49.602+0000 7efc8a050700 1 --2- 172.21.15.42:0/2061810027 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efc6c009660 0x7efc6c00baa0 secure :-1 s=THROTTLE_DONE pgs=3065 cs=0 l=1 rx=0x7efc7c001380 tx=0x7efc7c011960).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:49.643 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:49.602+0000 7efc7a7fc700 1 -- 172.21.15.42:0/2061810027 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efc7c011260 con 0x7efc6c009660 2019-12-05T06:42:49.643 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:49.602+0000 7efc8a050700 1 --2- 172.21.15.42:0/2061810027 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efc6c009660 0x7efc6c00baa0 secure :-1 s=THROTTLE_DONE pgs=3065 cs=0 l=1 rx=0x7efc7c001380 tx=0x7efc7c011960).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:49.643 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:49.602+0000 7efc7a7fc700 1 -- 172.21.15.42:0/2061810027 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc7c006cc0 con 0x7efc6c009660 2019-12-05T06:42:49.644 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:49.602+0000 7efc7a7fc700 1 -- 172.21.15.42:0/2061810027 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efc7c002920 con 0x7efc6c009660 2019-12-05T06:42:49.644 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:49.602+0000 7efc8a050700 1 --2- 172.21.15.42:0/2061810027 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efc6c009660 0x7efc6c00baa0 secure :-1 s=THROTTLE_DONE pgs=3065 cs=0 l=1 rx=0x7efc7c001380 tx=0x7efc7c011960).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:49.644 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:49.602+0000 7efc7a7fc700 1 -- 172.21.15.42:0/2061810027 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7efc7c002b20 con 0x7efc6c009660 2019-12-05T06:42:49.644 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:49.602+0000 7efc8a050700 1 --2- 172.21.15.42:0/2061810027 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efc6c009660 0x7efc6c00baa0 secure :-1 s=THROTTLE_DONE pgs=3065 cs=0 l=1 rx=0x7efc7c001380 tx=0x7efc7c011960).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:49.644 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:49.602+0000 7efc7a7fc700 1 --2- 172.21.15.42:0/2061810027 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7efc6c03eb00 0x7efc6c041010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:42:49.644 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:49.602+0000 7efc7a7fc700 1 -- 172.21.15.42:0/2061810027 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 11793+0+0 (secure 0 0 0) 0x7efc7c04c610 con 0x7efc6c009660 2019-12-05T06:42:49.645 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:49.602+0000 7efc8984f700 1 --2- 172.21.15.42:0/2061810027 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7efc6c03eb00 0x7efc6c041010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:42:49.645 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:49.602+0000 7efc8984f700 1 --2- 172.21.15.42:0/2061810027 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7efc6c03eb00 0x7efc6c041010 crc :-1 s=READY pgs=1282 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:42:49.645 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:49.606+0000 7efc8b2b2700 1 -- 172.21.15.42:0/2061810027 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc60003200 con 0x7efc6c009660 2019-12-05T06:42:49.646 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:49.610+0000 7efc8a050700 1 --2- 172.21.15.42:0/2061810027 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efc6c009660 0x7efc6c00baa0 secure :-1 s=THROTTLE_DONE pgs=3065 cs=0 l=1 rx=0x7efc7c001380 tx=0x7efc7c011960).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:49.646 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:49.610+0000 7efc7a7fc700 1 -- 172.21.15.42:0/2061810027 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7efc7c0338f0 con 0x7efc6c009660 2019-12-05T06:42:49.779 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:49.766+0000 7efc8b2b2700 1 -- 172.21.15.42:0/2061810027 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efc60003630 con 0x7efc6c009660 2019-12-05T06:42:49.779 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:49.770+0000 7efc8a050700 1 --2- 172.21.15.42:0/2061810027 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efc6c009660 0x7efc6c00baa0 secure :-1 s=THROTTLE_DONE pgs=3065 cs=0 l=1 rx=0x7efc7c001380 tx=0x7efc7c011960).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:49.779 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:49.770+0000 7efc7a7fc700 1 -- 172.21.15.42:0/2061810027 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 368 v368) v1 ==== 95+0+1778 (secure 0 0 0) 0x7efc7c0313e0 con 0x7efc6c009660 2019-12-05T06:42:49.779 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 368 2019-12-05T06:42:49.780 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:49.770+0000 7efc8b2b2700 1 -- 172.21.15.42:0/2061810027 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7efc6c03eb00 msgr2=0x7efc6c041010 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:42:49.780 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:49.770+0000 7efc8b2b2700 1 --2- 172.21.15.42:0/2061810027 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7efc6c03eb00 0x7efc6c041010 crc :-1 s=READY pgs=1282 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:42:49.780 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:49.770+0000 7efc8b2b2700 1 -- 172.21.15.42:0/2061810027 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efc6c009660 msgr2=0x7efc6c00baa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:42:49.780 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:49.770+0000 7efc8b2b2700 1 --2- 172.21.15.42:0/2061810027 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efc6c009660 0x7efc6c00baa0 secure :-1 s=READY pgs=3065 cs=0 l=1 rx=0x7efc7c001380 tx=0x7efc7c011960).stop 2019-12-05T06:42:49.781 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:49.770+0000 7efc8b2b2700 1 -- 172.21.15.42:0/2061810027 shutdown_connections 2019-12-05T06:42:49.781 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:49.770+0000 7efc8b2b2700 1 --2- 172.21.15.42:0/2061810027 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7efc6c03eb00 0x7efc6c041010 unknown :-1 s=CLOSED pgs=1282 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:42:49.781 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:49.770+0000 7efc8b2b2700 1 --2- 172.21.15.42:0/2061810027 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efc6c009660 0x7efc6c00baa0 unknown :-1 s=CLOSED pgs=3065 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:42:49.781 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:49.770+0000 7efc8b2b2700 1 -- 172.21.15.42:0/2061810027 shutdown_connections 2019-12-05T06:42:49.791 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:49.770+0000 7efc8b2b2700 1 -- 172.21.15.42:0/2061810027 wait complete. 2019-12-05T06:42:49.792 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:49.770+0000 7efc8b2b2700 1 -- 172.21.15.42:0/2061810027 >> 172.21.15.42:0/2061810027 conn(0x7efc840aabe0 msgr2=0x7efc841183a0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:42:49.792 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:42:49.792 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":368,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":368,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:42:45.235691+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":116,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12358},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12358":{"gid":12358,"name":"a","rank":0,"incarnation":368,"state":"up:replay","state_seq":2,"addr":"172.21.15.42:6835/1922398668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1922398668},{"type":"v1","addr":"172.21.15.42:6835","nonce":1922398668}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:42:49.794 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2019-12-05T06:42:51.795 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:42:51.900 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:51.890+0000 7fad7711e700 1 Processor -- start 2019-12-05T06:42:51.900 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:51.890+0000 7fad7711e700 1 -- start start 2019-12-05T06:42:51.900 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:51.894+0000 7fad7711e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fad7008df80 0x7fad7009f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:42:51.900 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:51.894+0000 7fad7711e700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fad7009f830 0x7fad7014efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:42:51.900 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:51.894+0000 7fad7711e700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad7009fc60 con 0x7fad700a7740 2019-12-05T06:42:51.901 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:51.894+0000 7fad7711e700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad700a0b00 con 0x7fad7008df80 2019-12-05T06:42:51.901 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:51.894+0000 7fad7711e700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad7008e5b0 con 0x7fad7009f830 2019-12-05T06:42:51.901 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:51.894+0000 7fad74eba700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fad7008df80 0x7fad7009f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:42:51.901 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:51.894+0000 7fad74eba700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fad7008df80 0x7fad7009f2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:33868/0 (socket says 172.21.15.42:33868) 2019-12-05T06:42:51.901 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:51.894+0000 7fad74eba700 1 -- 172.21.15.42:0/3424788955 learned_addr learned my addr 172.21.15.42:0/3424788955 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:42:51.901 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:51.894+0000 7fad75ebc700 1 --2- 172.21.15.42:0/3424788955 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fad7009f830 0x7fad7014efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:42:51.902 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:51.894+0000 7fad74eba700 1 -- 172.21.15.42:0/3424788955 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fad7009f830 msgr2=0x7fad7014efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:42:51.902 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:51.894+0000 7fad74eba700 1 --2- 172.21.15.42:0/3424788955 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fad7009f830 0x7fad7014efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:42:51.902 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:51.894+0000 7fad74eba700 1 -- 172.21.15.42:0/3424788955 >> v1:172.21.15.40:6789/0 conn(0x7fad700a7740 legacy=0x7fad7007e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:42:51.902 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:51.894+0000 7fad67fff700 1 -- 172.21.15.42:0/3424788955 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1304026262 0 0) 0x7fad7009fc60 con 0x7fad700a7740 2019-12-05T06:42:51.902 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:51.894+0000 7fad74eba700 1 -- 172.21.15.42:0/3424788955 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad700a0710 con 0x7fad7008df80 2019-12-05T06:42:51.902 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:51.894+0000 7fad74eba700 1 --2- 172.21.15.42:0/3424788955 >> v2:172.21.15.42:3300/0 conn(0x7fad7008df80 0x7fad7009f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fad60008790 tx=0x7fad6000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:51.902 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:51.894+0000 7fad74eba700 1 --2- 172.21.15.42:0/3424788955 >> v2:172.21.15.42:3300/0 conn(0x7fad7008df80 0x7fad7009f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fad60008790 tx=0x7fad6000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:51.903 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:51.894+0000 7fad74eba700 1 --2- 172.21.15.42:0/3424788955 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fad7008df80 0x7fad7009f2e0 secure :-1 s=READY pgs=3067 cs=0 l=1 rx=0x7fad60008790 tx=0x7fad6000fd30).ready entity=mon.1 client_cookie=782a68286f93294 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:42:51.903 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:51.894+0000 7fad74eba700 1 --2- 172.21.15.42:0/3424788955 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fad7008df80 0x7fad7009f2e0 secure :-1 s=THROTTLE_DONE pgs=3067 cs=0 l=1 rx=0x7fad60008790 tx=0x7fad6000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:51.903 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:51.894+0000 7fad67fff700 1 -- 172.21.15.42:0/3424788955 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fad60008c90 con 0x7fad7008df80 2019-12-05T06:42:51.903 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:51.894+0000 7fad74eba700 1 --2- 172.21.15.42:0/3424788955 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fad7008df80 0x7fad7009f2e0 secure :-1 s=THROTTLE_DONE pgs=3067 cs=0 l=1 rx=0x7fad60008790 tx=0x7fad6000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:51.903 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:51.894+0000 7fad74eba700 1 --2- 172.21.15.42:0/3424788955 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fad7008df80 0x7fad7009f2e0 secure :-1 s=THROTTLE_DONE pgs=3067 cs=0 l=1 rx=0x7fad60008790 tx=0x7fad6000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:51.903 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:51.894+0000 7fad67fff700 1 -- 172.21.15.42:0/3424788955 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad600088b0 con 0x7fad7008df80 2019-12-05T06:42:51.904 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:51.894+0000 7fad67fff700 1 -- 172.21.15.42:0/3424788955 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fad60011b10 con 0x7fad7008df80 2019-12-05T06:42:51.904 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:51.894+0000 7fad7711e700 1 -- 172.21.15.42:0/3424788955 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fad7008df80 msgr2=0x7fad7009f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:42:51.904 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:51.894+0000 7fad7711e700 1 --2- 172.21.15.42:0/3424788955 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fad7008df80 0x7fad7009f2e0 secure :-1 s=READY pgs=3067 cs=0 l=1 rx=0x7fad60008790 tx=0x7fad6000fd30).stop 2019-12-05T06:42:51.904 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:51.894+0000 7fad7711e700 1 -- 172.21.15.42:0/3424788955 shutdown_connections 2019-12-05T06:42:51.904 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:51.894+0000 7fad7711e700 1 --2- 172.21.15.42:0/3424788955 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fad7009f830 0x7fad7014efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:42:51.904 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:51.894+0000 7fad7711e700 1 --2- 172.21.15.42:0/3424788955 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fad7008df80 0x7fad7009f2e0 unknown :-1 s=CLOSED pgs=3067 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:42:51.904 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:51.894+0000 7fad7711e700 1 -- 172.21.15.42:0/3424788955 shutdown_connections 2019-12-05T06:42:51.905 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:51.894+0000 7fad7711e700 1 -- 172.21.15.42:0/3424788955 wait complete. 2019-12-05T06:42:51.905 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:51.894+0000 7fad7711e700 1 -- 172.21.15.42:0/3424788955 >> 172.21.15.42:0/3424788955 conn(0x7fad700aabe0 msgr2=0x7fad700a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:42:51.908 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:51.898+0000 7fad7711e700 1 Processor -- start 2019-12-05T06:42:51.908 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:51.898+0000 7fad7711e700 1 -- start start 2019-12-05T06:42:51.908 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:51.898+0000 7fad7711e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fad70119640 0x7fad701339c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:42:51.908 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:51.898+0000 7fad7711e700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fad70138e00 0x7fad7013b240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:42:51.908 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:51.898+0000 7fad7711e700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad70133f10 con 0x7fad701161c0 2019-12-05T06:42:51.908 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:51.898+0000 7fad7711e700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad70119a70 con 0x7fad70138e00 2019-12-05T06:42:51.909 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:51.898+0000 7fad7711e700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad70119100 con 0x7fad70119640 2019-12-05T06:42:51.909 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:51.898+0000 7fad756bb700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fad70119640 0x7fad701339c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:42:51.909 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:51.898+0000 7fad74eba700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fad701161c0 0x7fad7011a470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:33910/0 (socket says 172.21.15.42:33910) 2019-12-05T06:42:51.909 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:51.898+0000 7fad74eba700 1 -- 172.21.15.42:0/3228803726 learned_addr learned my addr 172.21.15.42:0/3228803726 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:42:51.909 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:51.902+0000 7fad75ebc700 1 --2- 172.21.15.42:0/3228803726 >> v2:172.21.15.40:3300/0 conn(0x7fad70138e00 0x7fad7013b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:42:51.909 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:51.902+0000 7fad65ffb700 1 -- 172.21.15.42:0/3228803726 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4134391119 0 0) 0x7fad70133f10 con 0x7fad701161c0 2019-12-05T06:42:51.909 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:51.902+0000 7fad65ffb700 1 -- 172.21.15.42:0/3228803726 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad500030b0 con 0x7fad701161c0 2019-12-05T06:42:51.910 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:51.902+0000 7fad756bb700 1 -- 172.21.15.42:0/3228803726 >> v2:172.21.15.40:3300/0 conn(0x7fad70138e00 msgr2=0x7fad7013b240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:42:51.910 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:51.902+0000 7fad756bb700 1 --2- 172.21.15.42:0/3228803726 >> v2:172.21.15.40:3300/0 conn(0x7fad70138e00 0x7fad7013b240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:42:51.910 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:51.902+0000 7fad756bb700 1 -- 172.21.15.42:0/3228803726 >> v1:172.21.15.42:6789/0 conn(0x7fad701161c0 legacy=0x7fad7011a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:42:51.910 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:51.902+0000 7fad65ffb700 1 -- 172.21.15.42:0/3228803726 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 1359772460 0 0) 0x7fad500030b0 con 0x7fad701161c0 2019-12-05T06:42:51.910 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:51.902+0000 7fad756bb700 1 -- 172.21.15.42:0/3228803726 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad7011ada0 con 0x7fad70119640 2019-12-05T06:42:51.910 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:51.902+0000 7fad756bb700 1 --2- 172.21.15.42:0/3228803726 >> v2:172.21.15.42:3300/0 conn(0x7fad70119640 0x7fad701339c0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fad60010180 tx=0x7fad600082d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:51.910 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:51.902+0000 7fad756bb700 1 --2- 172.21.15.42:0/3228803726 >> v2:172.21.15.42:3300/0 conn(0x7fad70119640 0x7fad701339c0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fad60010180 tx=0x7fad600082d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:51.911 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:51.902+0000 7fad756bb700 1 --2- 172.21.15.42:0/3228803726 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fad70119640 0x7fad701339c0 secure :-1 s=READY pgs=3069 cs=0 l=1 rx=0x7fad60010180 tx=0x7fad600082d0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:42:51.911 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:51.902+0000 7fad756bb700 1 --2- 172.21.15.42:0/3228803726 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fad70119640 0x7fad701339c0 secure :-1 s=THROTTLE_DONE pgs=3069 cs=0 l=1 rx=0x7fad60010180 tx=0x7fad600082d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:51.911 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:51.902+0000 7fad756bb700 1 --2- 172.21.15.42:0/3228803726 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fad70119640 0x7fad701339c0 secure :-1 s=THROTTLE_DONE pgs=3069 cs=0 l=1 rx=0x7fad60010180 tx=0x7fad600082d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:51.911 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:51.902+0000 7fad65ffb700 1 -- 172.21.15.42:0/3228803726 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fad60011db0 con 0x7fad70119640 2019-12-05T06:42:51.911 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:51.902+0000 7fad756bb700 1 --2- 172.21.15.42:0/3228803726 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fad70119640 0x7fad701339c0 secure :-1 s=THROTTLE_DONE pgs=3069 cs=0 l=1 rx=0x7fad60010180 tx=0x7fad600082d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:51.911 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:51.902+0000 7fad65ffb700 1 -- 172.21.15.42:0/3228803726 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad60011730 con 0x7fad70119640 2019-12-05T06:42:51.911 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:51.902+0000 7fad7711e700 1 -- 172.21.15.42:0/3228803726 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad7011c1f0 con 0x7fad70119640 2019-12-05T06:42:51.911 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:51.902+0000 7fad65ffb700 1 -- 172.21.15.42:0/3228803726 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fad6c00c740 con 0x7fad70119640 2019-12-05T06:42:51.912 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:51.902+0000 7fad7711e700 1 -- 172.21.15.42:0/3228803726 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad7007d000 con 0x7fad70119640 2019-12-05T06:42:51.912 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:51.902+0000 7fad756bb700 1 --2- 172.21.15.42:0/3228803726 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fad70119640 0x7fad701339c0 secure :-1 s=THROTTLE_DONE pgs=3069 cs=0 l=1 rx=0x7fad60010180 tx=0x7fad600082d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:51.912 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:51.902+0000 7fad65ffb700 1 -- 172.21.15.42:0/3228803726 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fad6c00c940 con 0x7fad70119640 2019-12-05T06:42:51.912 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:51.902+0000 7fad756bb700 1 --2- 172.21.15.42:0/3228803726 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fad70119640 0x7fad701339c0 secure :-1 s=THROTTLE_DONE pgs=3069 cs=0 l=1 rx=0x7fad60010180 tx=0x7fad600082d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:51.916 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:51.902+0000 7fad65ffb700 1 --2- 172.21.15.42:0/3228803726 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fad5003b210 0x7fad5003d720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:42:51.916 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:51.902+0000 7fad65ffb700 1 -- 172.21.15.42:0/3228803726 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 11793+0+0 (secure 0 0 0) 0x7fad6c04abd0 con 0x7fad70119640 2019-12-05T06:42:51.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:51.902+0000 7fad74eba700 1 --2- 172.21.15.42:0/3228803726 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fad5003b210 0x7fad5003d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:42:51.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:51.902+0000 7fad74eba700 1 --2- 172.21.15.42:0/3228803726 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fad5003b210 0x7fad5003d720 crc :-1 s=READY pgs=1283 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:42:51.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:51.906+0000 7fad7711e700 1 -- 172.21.15.42:0/3228803726 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad7005d110 con 0x7fad70119640 2019-12-05T06:42:51.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:51.910+0000 7fad756bb700 1 --2- 172.21.15.42:0/3228803726 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fad70119640 0x7fad701339c0 secure :-1 s=THROTTLE_DONE pgs=3069 cs=0 l=1 rx=0x7fad60010180 tx=0x7fad600082d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:51.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:51.910+0000 7fad65ffb700 1 -- 172.21.15.42:0/3228803726 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fad7011ada0 con 0x7fad70119640 2019-12-05T06:42:52.077 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:52.074+0000 7fad7711e700 1 -- 172.21.15.42:0/3228803726 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fad7005b8c0 con 0x7fad70119640 2019-12-05T06:42:52.078 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:52.074+0000 7fad756bb700 1 --2- 172.21.15.42:0/3228803726 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fad70119640 0x7fad701339c0 secure :-1 s=THROTTLE_DONE pgs=3069 cs=0 l=1 rx=0x7fad60010180 tx=0x7fad600082d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:52.078 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:52.074+0000 7fad65ffb700 1 -- 172.21.15.42:0/3228803726 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 369 v369) v1 ==== 95+0+1782 (secure 0 0 0) 0x7fad6c031bf0 con 0x7fad70119640 2019-12-05T06:42:52.078 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 369 2019-12-05T06:42:52.078 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:42:52.078 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":369,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":369,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:42:51.514000+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":116,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12358},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12358":{"gid":12358,"name":"a","rank":0,"incarnation":368,"state":"up:reconnect","state_seq":32,"addr":"172.21.15.42:6835/1922398668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1922398668},{"type":"v1","addr":"172.21.15.42:6835","nonce":1922398668}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:42:52.081 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:52.074+0000 7fad7711e700 1 -- 172.21.15.42:0/3228803726 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fad5003b210 msgr2=0x7fad5003d720 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:42:52.081 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:52.074+0000 7fad7711e700 1 --2- 172.21.15.42:0/3228803726 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fad5003b210 0x7fad5003d720 crc :-1 s=READY pgs=1283 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:42:52.081 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:52.074+0000 7fad7711e700 1 -- 172.21.15.42:0/3228803726 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fad70119640 msgr2=0x7fad701339c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:42:52.081 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:52.074+0000 7fad7711e700 1 --2- 172.21.15.42:0/3228803726 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fad70119640 0x7fad701339c0 secure :-1 s=READY pgs=3069 cs=0 l=1 rx=0x7fad60010180 tx=0x7fad600082d0).stop 2019-12-05T06:42:52.081 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:52.078+0000 7fad7711e700 1 -- 172.21.15.42:0/3228803726 shutdown_connections 2019-12-05T06:42:52.082 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:52.078+0000 7fad7711e700 1 --2- 172.21.15.42:0/3228803726 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fad5003b210 0x7fad5003d720 unknown :-1 s=CLOSED pgs=1283 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:42:52.082 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:52.078+0000 7fad7711e700 1 --2- 172.21.15.42:0/3228803726 >> v2:172.21.15.40:3300/0 conn(0x7fad70138e00 0x7fad7013b240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:42:52.082 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:52.078+0000 7fad7711e700 1 --2- 172.21.15.42:0/3228803726 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fad70119640 0x7fad701339c0 unknown :-1 s=CLOSED pgs=3069 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:42:52.082 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:52.078+0000 7fad7711e700 1 -- 172.21.15.42:0/3228803726 shutdown_connections 2019-12-05T06:42:52.082 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:52.078+0000 7fad7711e700 1 -- 172.21.15.42:0/3228803726 wait complete. 2019-12-05T06:42:52.082 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:52.078+0000 7fad7711e700 1 -- 172.21.15.42:0/3228803726 >> 172.21.15.42:0/3228803726 conn(0x7fad700a7740 msgr2=0x7fad70059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:42:52.090 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2019-12-05T06:42:54.091 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:42:54.173 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:54.166+0000 7fe83c4c6700 1 Processor -- start 2019-12-05T06:42:54.173 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:54.166+0000 7fe83c4c6700 1 -- start start 2019-12-05T06:42:54.173 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:54.166+0000 7fe83c4c6700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe83408df70 0x7fe83409f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:42:54.173 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:54.166+0000 7fe83c4c6700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe83409fc50 con 0x7fe83409f820 2019-12-05T06:42:54.174 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:54.166+0000 7fe83c4c6700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8340a0700 con 0x7fe8340a7730 2019-12-05T06:42:54.174 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:54.166+0000 7fe83c4c6700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8340a0af0 con 0x7fe83408df70 2019-12-05T06:42:54.174 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:54.166+0000 7fe83a262700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe83408df70 0x7fe83409f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:42:54.174 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:54.166+0000 7fe83a262700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe83408df70 0x7fe83409f2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:33890/0 (socket says 172.21.15.42:33890) 2019-12-05T06:42:54.174 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:54.166+0000 7fe83a262700 1 -- 172.21.15.42:0/3710712666 learned_addr learned my addr 172.21.15.42:0/3710712666 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:42:54.174 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:54.166+0000 7fe83aa63700 1 --1- 172.21.15.42:0/3710712666 >> v1:172.21.15.42:6789/0 conn(0x7fe8340a7730 0x7fe83407e710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:33926/0 (socket says 172.21.15.42:33926) 2019-12-05T06:42:54.174 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:54.166+0000 7fe83a262700 1 -- 172.21.15.42:0/3710712666 >> v1:172.21.15.42:6789/0 conn(0x7fe8340a7730 legacy=0x7fe83407e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:42:54.175 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:54.166+0000 7fe83a262700 1 -- 172.21.15.42:0/3710712666 >> v1:172.21.15.40:6789/0 conn(0x7fe83409f820 legacy=0x7fe83414efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:42:54.175 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:54.166+0000 7fe83a262700 1 -- 172.21.15.42:0/3710712666 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe83407ef10 con 0x7fe83408df70 2019-12-05T06:42:54.175 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:54.166+0000 7fe83a262700 1 --2- 172.21.15.42:0/3710712666 >> v2:172.21.15.42:3300/0 conn(0x7fe83408df70 0x7fe83409f2d0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fe824008790 tx=0x7fe82400fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:54.175 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:54.166+0000 7fe83a262700 1 --2- 172.21.15.42:0/3710712666 >> v2:172.21.15.42:3300/0 conn(0x7fe83408df70 0x7fe83409f2d0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fe824008790 tx=0x7fe82400fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:54.175 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:54.166+0000 7fe83a262700 1 --2- 172.21.15.42:0/3710712666 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe83408df70 0x7fe83409f2d0 secure :-1 s=READY pgs=3071 cs=0 l=1 rx=0x7fe824008790 tx=0x7fe82400fcb0).ready entity=mon.1 client_cookie=134d537626109c9 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:42:54.175 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:54.166+0000 7fe83a262700 1 --2- 172.21.15.42:0/3710712666 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe83408df70 0x7fe83409f2d0 secure :-1 s=THROTTLE_DONE pgs=3071 cs=0 l=1 rx=0x7fe824008790 tx=0x7fe82400fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:54.175 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:54.166+0000 7fe839a61700 1 -- 172.21.15.42:0/3710712666 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe824011520 con 0x7fe83408df70 2019-12-05T06:42:54.176 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:54.166+0000 7fe83a262700 1 --2- 172.21.15.42:0/3710712666 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe83408df70 0x7fe83409f2d0 secure :-1 s=THROTTLE_DONE pgs=3071 cs=0 l=1 rx=0x7fe824008790 tx=0x7fe82400fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:54.176 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:54.166+0000 7fe83a262700 1 --2- 172.21.15.42:0/3710712666 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe83408df70 0x7fe83409f2d0 secure :-1 s=THROTTLE_DONE pgs=3071 cs=0 l=1 rx=0x7fe824008790 tx=0x7fe82400fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:54.176 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:54.166+0000 7fe839a61700 1 -- 172.21.15.42:0/3710712666 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe824008ca0 con 0x7fe83408df70 2019-12-05T06:42:54.176 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:54.166+0000 7fe839a61700 1 -- 172.21.15.42:0/3710712666 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe824011b00 con 0x7fe83408df70 2019-12-05T06:42:54.176 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:54.166+0000 7fe83c4c6700 1 -- 172.21.15.42:0/3710712666 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe83408df70 msgr2=0x7fe83409f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:42:54.176 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:54.166+0000 7fe83c4c6700 1 --2- 172.21.15.42:0/3710712666 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe83408df70 0x7fe83409f2d0 secure :-1 s=READY pgs=3071 cs=0 l=1 rx=0x7fe824008790 tx=0x7fe82400fcb0).stop 2019-12-05T06:42:54.176 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:54.170+0000 7fe83c4c6700 1 -- 172.21.15.42:0/3710712666 shutdown_connections 2019-12-05T06:42:54.176 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:54.170+0000 7fe83c4c6700 1 --2- 172.21.15.42:0/3710712666 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe83408df70 0x7fe83409f2d0 unknown :-1 s=CLOSED pgs=3071 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:42:54.177 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:54.170+0000 7fe83c4c6700 1 -- 172.21.15.42:0/3710712666 shutdown_connections 2019-12-05T06:42:54.177 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:54.170+0000 7fe83c4c6700 1 -- 172.21.15.42:0/3710712666 wait complete. 2019-12-05T06:42:54.177 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:54.170+0000 7fe83c4c6700 1 -- 172.21.15.42:0/3710712666 >> 172.21.15.42:0/3710712666 conn(0x7fe8340aabd0 msgr2=0x7fe8340a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:42:54.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:54.170+0000 7fe83c4c6700 1 Processor -- start 2019-12-05T06:42:54.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:54.170+0000 7fe83c4c6700 1 -- start start 2019-12-05T06:42:54.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:54.170+0000 7fe83c4c6700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe834115d50 0x7fe834112fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:42:54.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:54.170+0000 7fe83c4c6700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe834113530 con 0x7fe834122a00 2019-12-05T06:42:54.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:54.170+0000 7fe83c4c6700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe834145440 con 0x7fe834143250 2019-12-05T06:42:54.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:54.170+0000 7fe83c4c6700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8341441b0 con 0x7fe834115d50 2019-12-05T06:42:54.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:54.170+0000 7fe83b264700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fe834143250 0x7fe83412cd80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:33936/0 (socket says 172.21.15.42:33936) 2019-12-05T06:42:54.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:54.170+0000 7fe83b264700 1 -- 172.21.15.42:0/546453364 learned_addr learned my addr 172.21.15.42:0/546453364 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:42:54.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:54.170+0000 7fe83aa63700 1 --2- 172.21.15.42:0/546453364 >> v2:172.21.15.40:3300/0 conn(0x7fe834115d50 0x7fe834112fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:42:54.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:54.174+0000 7fe82b7fe700 1 -- 172.21.15.42:0/546453364 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1283444912 0 0) 0x7fe834145440 con 0x7fe834143250 2019-12-05T06:42:54.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:54.174+0000 7fe82b7fe700 1 -- 172.21.15.42:0/546453364 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe81c0030b0 con 0x7fe834143250 2019-12-05T06:42:54.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:54.174+0000 7fe82b7fe700 1 -- 172.21.15.42:0/546453364 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2491080146 0 0) 0x7fe834113530 con 0x7fe834122a00 2019-12-05T06:42:54.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:54.174+0000 7fe83aa63700 1 -- 172.21.15.42:0/546453364 >> v1:172.21.15.42:6789/0 conn(0x7fe834143250 legacy=0x7fe83412cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:42:54.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:54.174+0000 7fe83aa63700 1 -- 172.21.15.42:0/546453364 >> v1:172.21.15.40:6789/0 conn(0x7fe834122a00 legacy=0x7fe834142a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:42:54.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:54.174+0000 7fe83aa63700 1 -- 172.21.15.42:0/546453364 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe834143d50 con 0x7fe834115d50 2019-12-05T06:42:54.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:54.174+0000 7fe83aa63700 1 --2- 172.21.15.42:0/546453364 >> v2:172.21.15.40:3300/0 conn(0x7fe834115d50 0x7fe834112fe0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fe82c001e70 tx=0x7fe82c001980).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:54.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:54.174+0000 7fe83aa63700 1 --2- 172.21.15.42:0/546453364 >> v2:172.21.15.40:3300/0 conn(0x7fe834115d50 0x7fe834112fe0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fe82c001e70 tx=0x7fe82c001980).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:54.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:54.174+0000 7fe83aa63700 1 --2- 172.21.15.42:0/546453364 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe834115d50 0x7fe834112fe0 secure :-1 s=READY pgs=2325 cs=0 l=1 rx=0x7fe82c001e70 tx=0x7fe82c001980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:42:54.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:54.174+0000 7fe83aa63700 1 --2- 172.21.15.42:0/546453364 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe834115d50 0x7fe834112fe0 secure :-1 s=THROTTLE_DONE pgs=2325 cs=0 l=1 rx=0x7fe82c001e70 tx=0x7fe82c001980).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:54.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:54.174+0000 7fe82b7fe700 1 -- 172.21.15.42:0/546453364 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe82c001c30 con 0x7fe834115d50 2019-12-05T06:42:54.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:54.174+0000 7fe83aa63700 1 --2- 172.21.15.42:0/546453364 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe834115d50 0x7fe834112fe0 secure :-1 s=THROTTLE_DONE pgs=2325 cs=0 l=1 rx=0x7fe82c001e70 tx=0x7fe82c001980).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:54.184 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:54.174+0000 7fe83aa63700 1 --2- 172.21.15.42:0/546453364 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe834115d50 0x7fe834112fe0 secure :-1 s=THROTTLE_DONE pgs=2325 cs=0 l=1 rx=0x7fe82c001e70 tx=0x7fe82c001980).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:54.184 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:54.174+0000 7fe82b7fe700 1 -- 172.21.15.42:0/546453364 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe82c001a40 con 0x7fe834115d50 2019-12-05T06:42:54.184 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:54.174+0000 7fe83c4c6700 1 -- 172.21.15.42:0/546453364 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe834122e30 con 0x7fe834115d50 2019-12-05T06:42:54.184 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:54.174+0000 7fe82b7fe700 1 -- 172.21.15.42:0/546453364 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe830011760 con 0x7fe834115d50 2019-12-05T06:42:54.184 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:54.174+0000 7fe83c4c6700 1 -- 172.21.15.42:0/546453364 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe83412e590 con 0x7fe834115d50 2019-12-05T06:42:54.184 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:54.174+0000 7fe83aa63700 1 --2- 172.21.15.42:0/546453364 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe834115d50 0x7fe834112fe0 secure :-1 s=THROTTLE_DONE pgs=2325 cs=0 l=1 rx=0x7fe82c001e70 tx=0x7fe82c001980).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:54.184 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:54.174+0000 7fe82b7fe700 1 -- 172.21.15.42:0/546453364 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fe830011a40 con 0x7fe834115d50 2019-12-05T06:42:54.185 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:54.174+0000 7fe83aa63700 1 --2- 172.21.15.42:0/546453364 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe834115d50 0x7fe834112fe0 secure :-1 s=THROTTLE_DONE pgs=2325 cs=0 l=1 rx=0x7fe82c001e70 tx=0x7fe82c001980).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:54.185 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:54.174+0000 7fe82b7fe700 1 --2- 172.21.15.42:0/546453364 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe81c03b1e0 0x7fe81c03d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:42:54.185 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:54.174+0000 7fe82b7fe700 1 -- 172.21.15.42:0/546453364 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 11793+0+0 (secure 0 0 0) 0x7fe83004a9b0 con 0x7fe834115d50 2019-12-05T06:42:54.185 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:54.174+0000 7fe83a262700 1 --2- 172.21.15.42:0/546453364 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe81c03b1e0 0x7fe81c03d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:42:54.185 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:54.178+0000 7fe83a262700 1 --2- 172.21.15.42:0/546453364 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe81c03b1e0 0x7fe81c03d6f0 crc :-1 s=READY pgs=1284 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:42:54.186 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:54.182+0000 7fe83c4c6700 1 -- 172.21.15.42:0/546453364 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe83405d100 con 0x7fe834115d50 2019-12-05T06:42:54.197 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:54.186+0000 7fe83aa63700 1 --2- 172.21.15.42:0/546453364 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe834115d50 0x7fe834112fe0 secure :-1 s=THROTTLE_DONE pgs=2325 cs=0 l=1 rx=0x7fe82c001e70 tx=0x7fe82c001980).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:54.198 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:54.186+0000 7fe82b7fe700 1 -- 172.21.15.42:0/546453364 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fe830031a10 con 0x7fe834115d50 2019-12-05T06:42:54.359 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:54.354+0000 7fe83c4c6700 1 -- 172.21.15.42:0/546453364 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe83405b8b0 con 0x7fe834115d50 2019-12-05T06:42:54.360 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:54.358+0000 7fe83aa63700 1 --2- 172.21.15.42:0/546453364 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe834115d50 0x7fe834112fe0 secure :-1 s=THROTTLE_DONE pgs=2325 cs=0 l=1 rx=0x7fe82c001e70 tx=0x7fe82c001980).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:42:54.361 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:54.358+0000 7fe82b7fe700 1 -- 172.21.15.42:0/546453364 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 371 v371) v1 ==== 95+0+1779 (secure 0 0 0) 0x7fe830031c70 con 0x7fe834115d50 2019-12-05T06:42:54.362 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:42:54.363 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":371,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":371,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:42:53.534430+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":116,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12358},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12358":{"gid":12358,"name":"a","rank":0,"incarnation":368,"state":"up:active","state_seq":34,"addr":"172.21.15.42:6835/1922398668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1922398668},{"type":"v1","addr":"172.21.15.42:6835","nonce":1922398668}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:42:54.363 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 371 2019-12-05T06:42:54.365 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:54.362+0000 7fe83c4c6700 1 -- 172.21.15.42:0/546453364 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe81c03b1e0 msgr2=0x7fe81c03d6f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:42:54.365 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:54.362+0000 7fe83c4c6700 1 --2- 172.21.15.42:0/546453364 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe81c03b1e0 0x7fe81c03d6f0 crc :-1 s=READY pgs=1284 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:42:54.366 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:54.362+0000 7fe83c4c6700 1 -- 172.21.15.42:0/546453364 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe834115d50 msgr2=0x7fe834112fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:42:54.366 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:54.362+0000 7fe83c4c6700 1 --2- 172.21.15.42:0/546453364 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe834115d50 0x7fe834112fe0 secure :-1 s=READY pgs=2325 cs=0 l=1 rx=0x7fe82c001e70 tx=0x7fe82c001980).stop 2019-12-05T06:42:54.366 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:54.362+0000 7fe83c4c6700 1 -- 172.21.15.42:0/546453364 shutdown_connections 2019-12-05T06:42:54.366 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:54.362+0000 7fe83c4c6700 1 --2- 172.21.15.42:0/546453364 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fe81c03b1e0 0x7fe81c03d6f0 unknown :-1 s=CLOSED pgs=1284 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:42:54.366 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:54.362+0000 7fe83c4c6700 1 --2- 172.21.15.42:0/546453364 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe834115d50 0x7fe834112fe0 unknown :-1 s=CLOSED pgs=2325 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:42:54.366 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:54.362+0000 7fe83c4c6700 1 -- 172.21.15.42:0/546453364 shutdown_connections 2019-12-05T06:42:54.366 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:54.362+0000 7fe83c4c6700 1 -- 172.21.15.42:0/546453364 wait complete. 2019-12-05T06:42:54.367 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:42:54.362+0000 7fe83c4c6700 1 -- 172.21.15.42:0/546453364 >> 172.21.15.42:0/546453364 conn(0x7fe8340aabd0 msgr2=0x7fe834111da0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:42:54.375 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a has gained rank=0, replacing gid=11976 2019-12-05T06:42:54.375 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 7 secs before reviving mds.b 2019-12-05T06:43:01.376 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.b 2019-12-05T06:43:01.376 INFO:tasks.ceph.mds.b:Restarting daemon 2019-12-05T06:43:01.376 INFO:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2019-12-05T06:43:01.380 INFO:tasks.ceph.mds.b:Started 2019-12-05T06:43:01.380 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:43:01.427 INFO:tasks.ceph.mds.b.smithi040.stdout:starting mds.b at 2019-12-05T06:43:01.471 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:01.462+0000 7fd4ad145700 1 Processor -- start 2019-12-05T06:43:01.472 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:01.462+0000 7fd4ad145700 1 -- start start 2019-12-05T06:43:01.472 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:01.466+0000 7fd4ad145700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd4a80a7740 0x7fd4a808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:43:01.472 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:01.466+0000 7fd4ad145700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4a80a0710 con 0x7fd4a809fae0 2019-12-05T06:43:01.472 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:01.466+0000 7fd4ad145700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4a80a9ba0 con 0x7fd4a807e720 2019-12-05T06:43:01.473 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:01.466+0000 7fd4ad145700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4a80a0b00 con 0x7fd4a80a7740 2019-12-05T06:43:01.473 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:01.466+0000 7fd4a759e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd4a80a7740 0x7fd4a808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:01.473 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:01.466+0000 7fd4a6d9d700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fd4a807e720 0x7fd4a809f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:34006/0 (socket says 172.21.15.42:34006) 2019-12-05T06:43:01.473 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:01.466+0000 7fd4a759e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd4a80a7740 0x7fd4a808df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:33968/0 (socket says 172.21.15.42:33968) 2019-12-05T06:43:01.476 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:01.466+0000 7fd4a6d9d700 1 -- 172.21.15.42:0/3418402906 learned_addr learned my addr 172.21.15.42:0/3418402906 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:43:01.476 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:01.466+0000 7fd4a759e700 1 -- 172.21.15.42:0/3418402906 >> v1:172.21.15.42:6789/0 conn(0x7fd4a807e720 legacy=0x7fd4a809f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:01.477 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:01.466+0000 7fd4a759e700 1 -- 172.21.15.42:0/3418402906 >> v1:172.21.15.40:6789/0 conn(0x7fd4a809fae0 legacy=0x7fd4a814efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:43:01.477 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:01.466+0000 7fd4a759e700 1 -- 172.21.15.42:0/3418402906 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4a807eb50 con 0x7fd4a80a7740 2019-12-05T06:43:01.477 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:01.466+0000 7fd4a659c700 1 -- 172.21.15.42:0/3418402906 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 271394664 0 0) 0x7fd4a80a9ba0 con 0x7fd4a807e720 2019-12-05T06:43:01.477 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:01.466+0000 7fd4a759e700 1 --2- 172.21.15.42:0/3418402906 >> v2:172.21.15.42:3300/0 conn(0x7fd4a80a7740 0x7fd4a808df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fd490008790 tx=0x7fd49000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:01.477 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:01.466+0000 7fd4a759e700 1 --2- 172.21.15.42:0/3418402906 >> v2:172.21.15.42:3300/0 conn(0x7fd4a80a7740 0x7fd4a808df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fd490008790 tx=0x7fd49000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:01.478 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:01.466+0000 7fd4a759e700 1 --2- 172.21.15.42:0/3418402906 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd4a80a7740 0x7fd4a808df80 secure :-1 s=READY pgs=3076 cs=0 l=1 rx=0x7fd490008790 tx=0x7fd49000fbf0).ready entity=mon.1 client_cookie=1268f039854331a8 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:43:01.478 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:01.466+0000 7fd4a759e700 1 --2- 172.21.15.42:0/3418402906 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd4a80a7740 0x7fd4a808df80 secure :-1 s=THROTTLE_DONE pgs=3076 cs=0 l=1 rx=0x7fd490008790 tx=0x7fd49000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:01.478 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:01.466+0000 7fd4a659c700 1 -- 172.21.15.42:0/3418402906 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd490011270 con 0x7fd4a80a7740 2019-12-05T06:43:01.479 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:01.466+0000 7fd4a759e700 1 --2- 172.21.15.42:0/3418402906 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd4a80a7740 0x7fd4a808df80 secure :-1 s=THROTTLE_DONE pgs=3076 cs=0 l=1 rx=0x7fd490008790 tx=0x7fd49000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:01.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:01.466+0000 7fd4a759e700 1 --2- 172.21.15.42:0/3418402906 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd4a80a7740 0x7fd4a808df80 secure :-1 s=THROTTLE_DONE pgs=3076 cs=0 l=1 rx=0x7fd490008790 tx=0x7fd49000fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:01.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:01.466+0000 7fd4a659c700 1 -- 172.21.15.42:0/3418402906 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd490008bc0 con 0x7fd4a80a7740 2019-12-05T06:43:01.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:01.466+0000 7fd4a659c700 1 -- 172.21.15.42:0/3418402906 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd490011850 con 0x7fd4a80a7740 2019-12-05T06:43:01.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:01.466+0000 7fd4ad145700 1 -- 172.21.15.42:0/3418402906 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd4a80a7740 msgr2=0x7fd4a808df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:01.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:01.466+0000 7fd4ad145700 1 --2- 172.21.15.42:0/3418402906 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd4a80a7740 0x7fd4a808df80 secure :-1 s=READY pgs=3076 cs=0 l=1 rx=0x7fd490008790 tx=0x7fd49000fbf0).stop 2019-12-05T06:43:01.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:01.466+0000 7fd4ad145700 1 -- 172.21.15.42:0/3418402906 shutdown_connections 2019-12-05T06:43:01.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:01.466+0000 7fd4ad145700 1 --2- 172.21.15.42:0/3418402906 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd4a80a7740 0x7fd4a808df80 unknown :-1 s=CLOSED pgs=3076 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:01.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:01.466+0000 7fd4ad145700 1 -- 172.21.15.42:0/3418402906 shutdown_connections 2019-12-05T06:43:01.482 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:01.466+0000 7fd4ad145700 1 -- 172.21.15.42:0/3418402906 wait complete. 2019-12-05T06:43:01.483 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:01.466+0000 7fd4ad145700 1 -- 172.21.15.42:0/3418402906 >> 172.21.15.42:0/3418402906 conn(0x7fd4a80aabe0 msgr2=0x7fd4a80a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:43:01.501 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:01.466+0000 7fd4ad145700 1 Processor -- start 2019-12-05T06:43:01.501 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:01.466+0000 7fd4ad145700 1 -- start start 2019-12-05T06:43:01.501 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:01.470+0000 7fd4ad145700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd4a811b500 0x7fd4a80a1c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:43:01.502 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:01.470+0000 7fd4ad145700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd4a811b930 0x7fd4a813a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:43:01.502 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:01.470+0000 7fd4ad145700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4a8130360 con 0x7fd4a81196e0 2019-12-05T06:43:01.502 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:01.470+0000 7fd4ad145700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4a8119b10 con 0x7fd4a811b930 2019-12-05T06:43:01.502 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:01.470+0000 7fd4ad145700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4a80a21c0 con 0x7fd4a811b500 2019-12-05T06:43:01.502 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:01.470+0000 7fd4ac944700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd4a811b930 0x7fd4a813a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:01.503 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:01.470+0000 7fd4ac944700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd4a811b930 0x7fd4a813a7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:33976/0 (socket says 172.21.15.42:33976) 2019-12-05T06:43:01.503 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:01.470+0000 7fd4ac944700 1 -- 172.21.15.42:0/195169185 learned_addr learned my addr 172.21.15.42:0/195169185 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:43:01.504 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:01.470+0000 7fd4a6d9d700 1 --2- 172.21.15.42:0/195169185 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd4a811b500 0x7fd4a80a1c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:01.504 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:01.470+0000 7fd4ac944700 1 -- 172.21.15.42:0/195169185 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd4a811b500 msgr2=0x7fd4a80a1c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:01.504 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:01.470+0000 7fd4ac944700 1 --2- 172.21.15.42:0/195169185 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd4a811b500 0x7fd4a80a1c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:01.504 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:01.470+0000 7fd4ac944700 1 -- 172.21.15.42:0/195169185 >> v1:172.21.15.40:6789/0 conn(0x7fd4a81196e0 legacy=0x7fd4a811a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:01.505 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:01.470+0000 7fd4ac944700 1 -- 172.21.15.42:0/195169185 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4a80a2380 con 0x7fd4a811b930 2019-12-05T06:43:01.505 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:01.470+0000 7fd4ac944700 1 --2- 172.21.15.42:0/195169185 >> v2:172.21.15.42:3300/0 conn(0x7fd4a811b930 0x7fd4a813a7e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fd4a0003270 tx=0x7fd4a0010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:01.505 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:01.470+0000 7fd4ac944700 1 --2- 172.21.15.42:0/195169185 >> v2:172.21.15.42:3300/0 conn(0x7fd4a811b930 0x7fd4a813a7e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fd4a0003270 tx=0x7fd4a0010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:01.505 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:01.470+0000 7fd4ac944700 1 --2- 172.21.15.42:0/195169185 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd4a811b930 0x7fd4a813a7e0 secure :-1 s=READY pgs=3077 cs=0 l=1 rx=0x7fd4a0003270 tx=0x7fd4a0010b50).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:43:01.505 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:01.470+0000 7fd4ac944700 1 --2- 172.21.15.42:0/195169185 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd4a811b930 0x7fd4a813a7e0 secure :-1 s=THROTTLE_DONE pgs=3077 cs=0 l=1 rx=0x7fd4a0003270 tx=0x7fd4a0010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:01.506 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:01.470+0000 7fd4ac944700 1 --2- 172.21.15.42:0/195169185 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd4a811b930 0x7fd4a813a7e0 secure :-1 s=THROTTLE_DONE pgs=3077 cs=0 l=1 rx=0x7fd4a0003270 tx=0x7fd4a0010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:01.506 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:01.470+0000 7fd4ac944700 1 --2- 172.21.15.42:0/195169185 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd4a811b930 0x7fd4a813a7e0 secure :-1 s=THROTTLE_DONE pgs=3077 cs=0 l=1 rx=0x7fd4a0003270 tx=0x7fd4a0010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:01.506 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:01.470+0000 7fd49ffff700 1 -- 172.21.15.42:0/195169185 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd4a0008a20 con 0x7fd4a811b930 2019-12-05T06:43:01.506 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:01.470+0000 7fd49ffff700 1 -- 172.21.15.42:0/195169185 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4a0005b40 con 0x7fd4a811b930 2019-12-05T06:43:01.506 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:01.470+0000 7fd4ad145700 1 -- 172.21.15.42:0/195169185 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4a813ad30 con 0x7fd4a811b930 2019-12-05T06:43:01.507 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:01.470+0000 7fd49ffff700 1 -- 172.21.15.42:0/195169185 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd4a0007160 con 0x7fd4a811b930 2019-12-05T06:43:01.507 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:01.470+0000 7fd4ad145700 1 -- 172.21.15.42:0/195169185 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4a813b110 con 0x7fd4a811b930 2019-12-05T06:43:01.507 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:01.470+0000 7fd4ac944700 1 --2- 172.21.15.42:0/195169185 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd4a811b930 0x7fd4a813a7e0 secure :-1 s=THROTTLE_DONE pgs=3077 cs=0 l=1 rx=0x7fd4a0003270 tx=0x7fd4a0010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:01.507 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:01.470+0000 7fd49ffff700 1 -- 172.21.15.42:0/195169185 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fd4a0007470 con 0x7fd4a811b930 2019-12-05T06:43:01.507 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:01.470+0000 7fd4ac944700 1 --2- 172.21.15.42:0/195169185 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd4a811b930 0x7fd4a813a7e0 secure :-1 s=THROTTLE_DONE pgs=3077 cs=0 l=1 rx=0x7fd4a0003270 tx=0x7fd4a0010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:01.508 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:01.470+0000 7fd49ffff700 1 --2- 172.21.15.42:0/195169185 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd48c03ad80 0x7fd48c03d290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:43:01.508 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:01.470+0000 7fd49ffff700 1 -- 172.21.15.42:0/195169185 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 11793+0+0 (secure 0 0 0) 0x7fd4a0013060 con 0x7fd4a811b930 2019-12-05T06:43:01.508 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:01.470+0000 7fd4a759e700 1 --2- 172.21.15.42:0/195169185 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd48c03ad80 0x7fd48c03d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:01.508 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:01.470+0000 7fd4a759e700 1 --2- 172.21.15.42:0/195169185 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd48c03ad80 0x7fd48c03d290 crc :-1 s=READY pgs=1285 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:43:01.508 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:01.478+0000 7fd4ad145700 1 -- 172.21.15.42:0/195169185 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4a805d110 con 0x7fd4a811b930 2019-12-05T06:43:01.508 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:01.486+0000 7fd4ac944700 1 --2- 172.21.15.42:0/195169185 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd4a811b930 0x7fd4a813a7e0 secure :-1 s=THROTTLE_DONE pgs=3077 cs=0 l=1 rx=0x7fd4a0003270 tx=0x7fd4a0010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:01.509 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:01.486+0000 7fd49ffff700 1 -- 172.21.15.42:0/195169185 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fd4a00321a0 con 0x7fd4a811b930 2019-12-05T06:43:01.658 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:01.654+0000 7fd4ad145700 1 -- 172.21.15.42:0/195169185 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd4a805b8c0 con 0x7fd4a811b930 2019-12-05T06:43:01.660 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:01.654+0000 7fd4ac944700 1 --2- 172.21.15.42:0/195169185 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd4a811b930 0x7fd4a813a7e0 secure :-1 s=THROTTLE_DONE pgs=3077 cs=0 l=1 rx=0x7fd4a0003270 tx=0x7fd4a0010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:01.660 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:01.654+0000 7fd49ffff700 1 -- 172.21.15.42:0/195169185 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 371 v371) v1 ==== 95+0+1779 (secure 0 0 0) 0x7fd4a0032400 con 0x7fd4a811b930 2019-12-05T06:43:01.660 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 371 2019-12-05T06:43:01.661 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:43:01.661 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":371,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":371,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:42:53.534430+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":116,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12358},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12358":{"gid":12358,"name":"a","rank":0,"incarnation":368,"state":"up:active","state_seq":34,"addr":"172.21.15.42:6835/1922398668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1922398668},{"type":"v1","addr":"172.21.15.42:6835","nonce":1922398668}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:43:01.663 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:01.658+0000 7fd4ad145700 1 -- 172.21.15.42:0/195169185 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd48c03ad80 msgr2=0x7fd48c03d290 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:01.663 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:01.658+0000 7fd4ad145700 1 --2- 172.21.15.42:0/195169185 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd48c03ad80 0x7fd48c03d290 crc :-1 s=READY pgs=1285 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:01.663 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:01.658+0000 7fd4ad145700 1 -- 172.21.15.42:0/195169185 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd4a811b930 msgr2=0x7fd4a813a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:01.663 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:01.658+0000 7fd4ad145700 1 --2- 172.21.15.42:0/195169185 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd4a811b930 0x7fd4a813a7e0 secure :-1 s=READY pgs=3077 cs=0 l=1 rx=0x7fd4a0003270 tx=0x7fd4a0010b50).stop 2019-12-05T06:43:01.663 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:01.658+0000 7fd4ad145700 1 -- 172.21.15.42:0/195169185 shutdown_connections 2019-12-05T06:43:01.664 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:01.658+0000 7fd4ad145700 1 --2- 172.21.15.42:0/195169185 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd48c03ad80 0x7fd48c03d290 unknown :-1 s=CLOSED pgs=1285 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:01.664 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:01.658+0000 7fd4ad145700 1 --2- 172.21.15.42:0/195169185 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd4a811b930 0x7fd4a813a7e0 unknown :-1 s=CLOSED pgs=3077 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:01.664 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:01.658+0000 7fd4ad145700 1 --2- 172.21.15.42:0/195169185 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd4a811b500 0x7fd4a80a1c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:01.664 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:01.658+0000 7fd4ad145700 1 -- 172.21.15.42:0/195169185 shutdown_connections 2019-12-05T06:43:01.664 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:01.658+0000 7fd4ad145700 1 -- 172.21.15.42:0/195169185 wait complete. 2019-12-05T06:43:01.664 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:01.658+0000 7fd4ad145700 1 -- 172.21.15.42:0/195169185 >> 172.21.15.42:0/195169185 conn(0x7fd4a807ee20 msgr2=0x7fd4a80a9790 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:43:01.676 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2019-12-05T06:43:03.677 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:43:03.785 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:03.774+0000 7fab6b57f700 1 Processor -- start 2019-12-05T06:43:03.785 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:03.774+0000 7fab6b57f700 1 -- start start 2019-12-05T06:43:03.786 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:03.778+0000 7fab6b57f700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fab640a7740 0x7fab6408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:43:03.786 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:03.778+0000 7fab6b57f700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fab6409fae0 0x7fab6407eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:43:03.786 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:03.778+0000 7fab6b57f700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab640a0710 con 0x7fab6407e720 2019-12-05T06:43:03.786 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:03.778+0000 7fab6b57f700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab640a0b00 con 0x7fab6409fae0 2019-12-05T06:43:03.786 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:03.778+0000 7fab6b57f700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab6408e5b0 con 0x7fab640a7740 2019-12-05T06:43:03.787 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:03.778+0000 7fab6a31d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fab6409fae0 0x7fab6407eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:03.787 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:03.778+0000 7fab6a31d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fab6409fae0 0x7fab6407eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:33994/0 (socket says 172.21.15.42:33994) 2019-12-05T06:43:03.787 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:03.778+0000 7fab6a31d700 1 -- 172.21.15.42:0/1758710569 learned_addr learned my addr 172.21.15.42:0/1758710569 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:43:03.787 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:03.778+0000 7fab69b1c700 1 --2- 172.21.15.42:0/1758710569 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fab640a7740 0x7fab6408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:03.787 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:03.778+0000 7fab6a31d700 1 -- 172.21.15.42:0/1758710569 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fab640a7740 msgr2=0x7fab6408df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:43:03.788 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:03.778+0000 7fab6a31d700 1 --2- 172.21.15.42:0/1758710569 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fab640a7740 0x7fab6408df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:43:03.788 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:03.778+0000 7fab6a31d700 1 -- 172.21.15.42:0/1758710569 >> v1:172.21.15.42:6789/0 conn(0x7fab6407e720 legacy=0x7fab6409f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:03.788 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:03.778+0000 7fab6a31d700 1 -- 172.21.15.42:0/1758710569 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab6407f0a0 con 0x7fab6409fae0 2019-12-05T06:43:03.788 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:03.778+0000 7fab69b1c700 1 --2- 172.21.15.42:0/1758710569 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fab640a7740 0x7fab6408df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).send_auth_request state changed! 2019-12-05T06:43:03.788 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:03.778+0000 7fab6a31d700 1 --2- 172.21.15.42:0/1758710569 >> v2:172.21.15.42:3300/0 conn(0x7fab6409fae0 0x7fab6407eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fab5c008b10 tx=0x7fab5c0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:03.788 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:03.778+0000 7fab6a31d700 1 --2- 172.21.15.42:0/1758710569 >> v2:172.21.15.42:3300/0 conn(0x7fab6409fae0 0x7fab6407eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fab5c008b10 tx=0x7fab5c0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:03.789 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:03.778+0000 7fab6a31d700 1 --2- 172.21.15.42:0/1758710569 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fab6409fae0 0x7fab6407eb50 secure :-1 s=READY pgs=3079 cs=0 l=1 rx=0x7fab5c008b10 tx=0x7fab5c0100b0).ready entity=mon.1 client_cookie=ecb6da20fe7db57c server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:43:03.789 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:03.778+0000 7fab6a31d700 1 --2- 172.21.15.42:0/1758710569 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fab6409fae0 0x7fab6407eb50 secure :-1 s=THROTTLE_DONE pgs=3079 cs=0 l=1 rx=0x7fab5c008b10 tx=0x7fab5c0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:03.789 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:03.778+0000 7fab68b1a700 1 -- 172.21.15.42:0/1758710569 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fab5c010da0 con 0x7fab6409fae0 2019-12-05T06:43:03.789 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:03.778+0000 7fab6a31d700 1 --2- 172.21.15.42:0/1758710569 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fab6409fae0 0x7fab6407eb50 secure :-1 s=THROTTLE_DONE pgs=3079 cs=0 l=1 rx=0x7fab5c008b10 tx=0x7fab5c0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:03.789 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:03.778+0000 7fab6a31d700 1 --2- 172.21.15.42:0/1758710569 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fab6409fae0 0x7fab6407eb50 secure :-1 s=THROTTLE_DONE pgs=3079 cs=0 l=1 rx=0x7fab5c008b10 tx=0x7fab5c0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:03.790 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:03.778+0000 7fab68b1a700 1 -- 172.21.15.42:0/1758710569 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab5c0109c0 con 0x7fab6409fae0 2019-12-05T06:43:03.790 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:03.778+0000 7fab68b1a700 1 -- 172.21.15.42:0/1758710569 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fab5c0114a0 con 0x7fab6409fae0 2019-12-05T06:43:03.790 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:03.778+0000 7fab6b57f700 1 -- 172.21.15.42:0/1758710569 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fab6409fae0 msgr2=0x7fab6407eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:03.790 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:03.778+0000 7fab6b57f700 1 --2- 172.21.15.42:0/1758710569 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fab6409fae0 0x7fab6407eb50 secure :-1 s=READY pgs=3079 cs=0 l=1 rx=0x7fab5c008b10 tx=0x7fab5c0100b0).stop 2019-12-05T06:43:03.790 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:03.782+0000 7fab6b57f700 1 -- 172.21.15.42:0/1758710569 shutdown_connections 2019-12-05T06:43:03.790 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:03.782+0000 7fab6b57f700 1 --2- 172.21.15.42:0/1758710569 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fab6409fae0 0x7fab6407eb50 unknown :-1 s=CLOSED pgs=3079 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:03.790 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:03.782+0000 7fab6b57f700 1 --2- 172.21.15.42:0/1758710569 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fab640a7740 0x7fab6408df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:43:03.791 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:03.782+0000 7fab6b57f700 1 -- 172.21.15.42:0/1758710569 shutdown_connections 2019-12-05T06:43:03.791 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:03.782+0000 7fab6b57f700 1 -- 172.21.15.42:0/1758710569 wait complete. 2019-12-05T06:43:03.791 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:03.782+0000 7fab6b57f700 1 -- 172.21.15.42:0/1758710569 >> 172.21.15.42:0/1758710569 conn(0x7fab640aabe0 msgr2=0x7fab640a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:43:03.795 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:03.782+0000 7fab6b57f700 1 Processor -- start 2019-12-05T06:43:03.795 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:03.782+0000 7fab6b57f700 1 -- start start 2019-12-05T06:43:03.795 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:03.782+0000 7fab6b57f700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fab64117b50 0x7fab641162e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:43:03.796 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:03.782+0000 7fab6b57f700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab6407bda0 con 0x7fab640a4090 2019-12-05T06:43:03.796 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:03.782+0000 7fab6b57f700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab64133650 con 0x7fab64126c20 2019-12-05T06:43:03.796 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:03.782+0000 7fab6b57f700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab64116a60 con 0x7fab64117b50 2019-12-05T06:43:03.796 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:03.786+0000 7fab69b1c700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fab64126c20 0x7fab6411aa40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:34034/0 (socket says 172.21.15.42:34034) 2019-12-05T06:43:03.820 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:03.786+0000 7fab69b1c700 1 -- 172.21.15.42:0/1295841647 learned_addr learned my addr 172.21.15.42:0/1295841647 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:43:03.820 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:03.786+0000 7fab6931b700 1 --2- 172.21.15.42:0/1295841647 >> v2:172.21.15.40:3300/0 conn(0x7fab64117b50 0x7fab641162e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:03.821 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:03.786+0000 7fab5a7fc700 1 -- 172.21.15.42:0/1295841647 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 312265033 0 0) 0x7fab64133650 con 0x7fab64126c20 2019-12-05T06:43:03.821 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:03.786+0000 7fab5a7fc700 1 -- 172.21.15.42:0/1295841647 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab4c0030b0 con 0x7fab64126c20 2019-12-05T06:43:03.821 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:03.786+0000 7fab5a7fc700 1 -- 172.21.15.42:0/1295841647 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 2416700315 0 0) 0x7fab4c0030b0 con 0x7fab64126c20 2019-12-05T06:43:03.821 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:03.786+0000 7fab5a7fc700 1 -- 172.21.15.42:0/1295841647 --> v1:172.21.15.42:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fab64133650 con 0x7fab64126c20 2019-12-05T06:43:03.821 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:03.786+0000 7fab5a7fc700 1 -- 172.21.15.42:0/1295841647 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7fab60001620 con 0x7fab64126c20 2019-12-05T06:43:03.821 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:03.786+0000 7fab5a7fc700 1 -- 172.21.15.42:0/1295841647 >> v2:172.21.15.40:3300/0 conn(0x7fab64117b50 msgr2=0x7fab641162e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:03.822 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:03.786+0000 7fab5a7fc700 1 --2- 172.21.15.42:0/1295841647 >> v2:172.21.15.40:3300/0 conn(0x7fab64117b50 0x7fab641162e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:03.822 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:03.786+0000 7fab5a7fc700 1 -- 172.21.15.42:0/1295841647 >> v1:172.21.15.42:6789/0 conn(0x7fab64126c20 legacy=0x7fab6411aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:03.823 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:03.786+0000 7fab5a7fc700 1 -- 172.21.15.42:0/1295841647 >> v1:172.21.15.40:6789/0 conn(0x7fab640a4090 legacy=0x7fab6407b5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:03.823 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:03.786+0000 7fab5a7fc700 1 --2- 172.21.15.42:0/1295841647 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab4c0038a0 0x7fab4c003d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:43:03.823 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:03.786+0000 7fab5a7fc700 1 --2- 172.21.15.42:0/1295841647 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fab4c004420 0x7fab4c004850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:43:03.823 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:03.786+0000 7fab5a7fc700 1 --2- 172.21.15.42:0/1295841647 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fab4c004ee0 0x7fab4c00b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:43:03.823 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:03.786+0000 7fab5a7fc700 1 -- 172.21.15.42:0/1295841647 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab64116a60 con 0x7fab4c0038a0 2019-12-05T06:43:03.823 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:03.786+0000 7fab5a7fc700 1 -- 172.21.15.42:0/1295841647 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab4c00bcc0 con 0x7fab4c004420 2019-12-05T06:43:03.824 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:03.786+0000 7fab5a7fc700 1 -- 172.21.15.42:0/1295841647 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab4c00bf00 con 0x7fab4c004ee0 2019-12-05T06:43:03.824 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:03.786+0000 7fab6931b700 1 --2- 172.21.15.42:0/1295841647 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fab4c004420 0x7fab4c004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:03.824 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:03.786+0000 7fab69b1c700 1 --2- 172.21.15.42:0/1295841647 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab4c0038a0 0x7fab4c003d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:03.824 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:03.786+0000 7fab6a31d700 1 --2- 172.21.15.42:0/1295841647 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fab4c004ee0 0x7fab4c00b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:03.824 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:03.786+0000 7fab6931b700 1 -- 172.21.15.42:0/1295841647 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fab4c004ee0 msgr2=0x7fab4c00b670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:03.825 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:03.786+0000 7fab6931b700 1 --2- 172.21.15.42:0/1295841647 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fab4c004ee0 0x7fab4c00b670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:03.825 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:03.786+0000 7fab6931b700 1 -- 172.21.15.42:0/1295841647 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab4c0038a0 msgr2=0x7fab4c003d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:03.825 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:03.786+0000 7fab6931b700 1 --2- 172.21.15.42:0/1295841647 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab4c0038a0 0x7fab4c003d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:03.825 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:03.786+0000 7fab6931b700 1 -- 172.21.15.42:0/1295841647 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab64134890 con 0x7fab4c004420 2019-12-05T06:43:03.825 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:03.786+0000 7fab6b57f700 1 -- 172.21.15.42:0/1295841647 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab64132640 con 0x7fab4c004420 2019-12-05T06:43:03.826 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:03.786+0000 7fab6a31d700 1 --2- 172.21.15.42:0/1295841647 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fab4c004ee0 0x7fab4c00b670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:43:03.826 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:03.786+0000 7fab6a31d700 1 -- 172.21.15.42:0/1295841647 reap_dead start 2019-12-05T06:43:03.826 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:03.786+0000 7fab6931b700 1 --2- 172.21.15.42:0/1295841647 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fab4c004420 0x7fab4c004850 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fab54000e90 tx=0x7fab54010cb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:03.826 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:03.786+0000 7fab6b57f700 1 -- 172.21.15.42:0/1295841647 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab64117b50 con 0x7fab4c004420 2019-12-05T06:43:03.826 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:03.786+0000 7fab6931b700 1 --2- 172.21.15.42:0/1295841647 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fab4c004420 0x7fab4c004850 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fab54000e90 tx=0x7fab54010cb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:03.826 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:03.786+0000 7fab6931b700 1 --2- 172.21.15.42:0/1295841647 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fab4c004420 0x7fab4c004850 secure :-1 s=READY pgs=3081 cs=0 l=1 rx=0x7fab54000e90 tx=0x7fab54010cb0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:43:03.826 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:03.786+0000 7fab6931b700 1 --2- 172.21.15.42:0/1295841647 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fab4c004420 0x7fab4c004850 secure :-1 s=THROTTLE_DONE pgs=3081 cs=0 l=1 rx=0x7fab54000e90 tx=0x7fab54010cb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:03.827 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:03.786+0000 7fab5a7fc700 1 -- 172.21.15.42:0/1295841647 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fab54007810 con 0x7fab4c004420 2019-12-05T06:43:03.827 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:03.786+0000 7fab6931b700 1 --2- 172.21.15.42:0/1295841647 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fab4c004420 0x7fab4c004850 secure :-1 s=THROTTLE_DONE pgs=3081 cs=0 l=1 rx=0x7fab54000e90 tx=0x7fab54010cb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:03.827 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:03.786+0000 7fab5a7fc700 1 -- 172.21.15.42:0/1295841647 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab54007430 con 0x7fab4c004420 2019-12-05T06:43:03.827 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:03.786+0000 7fab6931b700 1 --2- 172.21.15.42:0/1295841647 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fab4c004420 0x7fab4c004850 secure :-1 s=THROTTLE_DONE pgs=3081 cs=0 l=1 rx=0x7fab54000e90 tx=0x7fab54010cb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:03.827 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:03.786+0000 7fab5a7fc700 1 -- 172.21.15.42:0/1295841647 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fab54006360 con 0x7fab4c004420 2019-12-05T06:43:03.827 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:03.786+0000 7fab6931b700 1 --2- 172.21.15.42:0/1295841647 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fab4c004420 0x7fab4c004850 secure :-1 s=THROTTLE_DONE pgs=3081 cs=0 l=1 rx=0x7fab54000e90 tx=0x7fab54010cb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:03.827 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:03.790+0000 7fab5a7fc700 1 -- 172.21.15.42:0/1295841647 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fab54006560 con 0x7fab4c004420 2019-12-05T06:43:03.828 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:03.790+0000 7fab6931b700 1 --2- 172.21.15.42:0/1295841647 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fab4c004420 0x7fab4c004850 secure :-1 s=THROTTLE_DONE pgs=3081 cs=0 l=1 rx=0x7fab54000e90 tx=0x7fab54010cb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:03.828 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:03.790+0000 7fab5a7fc700 1 --2- 172.21.15.42:0/1295841647 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fab4c03e9d0 0x7fab4c040ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:43:03.828 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:03.790+0000 7fab5a7fc700 1 -- 172.21.15.42:0/1295841647 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 11793+0+0 (secure 0 0 0) 0x7fab5404b550 con 0x7fab4c004420 2019-12-05T06:43:03.828 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:03.790+0000 7fab69b1c700 1 --2- 172.21.15.42:0/1295841647 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fab4c03e9d0 0x7fab4c040ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:03.844 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:03.790+0000 7fab69b1c700 1 --2- 172.21.15.42:0/1295841647 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fab4c03e9d0 0x7fab4c040ee0 crc :-1 s=READY pgs=1286 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:43:03.845 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:03.790+0000 7fab6b57f700 1 -- 172.21.15.42:0/1295841647 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab40003200 con 0x7fab4c004420 2019-12-05T06:43:03.845 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:03.794+0000 7fab6931b700 1 --2- 172.21.15.42:0/1295841647 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fab4c004420 0x7fab4c004850 secure :-1 s=THROTTLE_DONE pgs=3081 cs=0 l=1 rx=0x7fab54000e90 tx=0x7fab54010cb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:03.846 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:03.798+0000 7fab5a7fc700 1 -- 172.21.15.42:0/1295841647 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fab540325b0 con 0x7fab4c004420 2019-12-05T06:43:04.006 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:03.982+0000 7fab6b57f700 1 -- 172.21.15.42:0/1295841647 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fab40003630 con 0x7fab4c004420 2019-12-05T06:43:04.006 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:03.982+0000 7fab6931b700 1 --2- 172.21.15.42:0/1295841647 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fab4c004420 0x7fab4c004850 secure :-1 s=THROTTLE_DONE pgs=3081 cs=0 l=1 rx=0x7fab54000e90 tx=0x7fab54010cb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:04.007 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:03.982+0000 7fab5a7fc700 1 -- 172.21.15.42:0/1295841647 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 371 v371) v1 ==== 95+0+1779 (secure 0 0 0) 0x7fab54032810 con 0x7fab4c004420 2019-12-05T06:43:04.007 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 371 2019-12-05T06:43:04.007 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:03.986+0000 7fab6b57f700 1 -- 172.21.15.42:0/1295841647 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fab4c03e9d0 msgr2=0x7fab4c040ee0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:04.008 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:03.986+0000 7fab6b57f700 1 --2- 172.21.15.42:0/1295841647 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fab4c03e9d0 0x7fab4c040ee0 crc :-1 s=READY pgs=1286 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:04.008 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:03.986+0000 7fab6b57f700 1 -- 172.21.15.42:0/1295841647 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fab4c004420 msgr2=0x7fab4c004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:04.008 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:03.986+0000 7fab6b57f700 1 --2- 172.21.15.42:0/1295841647 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fab4c004420 0x7fab4c004850 secure :-1 s=READY pgs=3081 cs=0 l=1 rx=0x7fab54000e90 tx=0x7fab54010cb0).stop 2019-12-05T06:43:04.008 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:03.986+0000 7fab6b57f700 1 -- 172.21.15.42:0/1295841647 shutdown_connections 2019-12-05T06:43:04.008 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:03.986+0000 7fab6b57f700 1 --2- 172.21.15.42:0/1295841647 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fab4c03e9d0 0x7fab4c040ee0 unknown :-1 s=CLOSED pgs=1286 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:04.008 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:03.986+0000 7fab6b57f700 1 --2- 172.21.15.42:0/1295841647 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fab4c004420 0x7fab4c004850 unknown :-1 s=CLOSED pgs=3081 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:04.008 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:03.986+0000 7fab6b57f700 1 -- 172.21.15.42:0/1295841647 shutdown_connections 2019-12-05T06:43:04.009 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:03.986+0000 7fab6b57f700 1 -- 172.21.15.42:0/1295841647 wait complete. 2019-12-05T06:43:04.009 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:03.986+0000 7fab6b57f700 1 -- 172.21.15.42:0/1295841647 >> 172.21.15.42:0/1295841647 conn(0x7fab6408eb70 msgr2=0x7fab64117010 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:43:04.009 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:43:04.009 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":371,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":371,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:42:53.534430+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":116,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12358},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12358":{"gid":12358,"name":"a","rank":0,"incarnation":368,"state":"up:active","state_seq":34,"addr":"172.21.15.42:6835/1922398668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1922398668},{"type":"v1","addr":"172.21.15.42:6835","nonce":1922398668}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:43:04.010 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2019-12-05T06:43:06.011 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:43:06.120 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:06.110+0000 7fd0d5441700 1 Processor -- start 2019-12-05T06:43:06.120 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:06.110+0000 7fd0d5441700 1 -- start start 2019-12-05T06:43:06.121 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:06.114+0000 7fd0d5441700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd0d00a7740 0x7fd0d008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:43:06.121 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:06.114+0000 7fd0d5441700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd0d007e720 0x7fd0d007eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:43:06.121 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:06.114+0000 7fd0d5441700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0d00a0710 con 0x7fd0d009f2e0 2019-12-05T06:43:06.121 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:06.114+0000 7fd0d5441700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0d00a0b00 con 0x7fd0d007e720 2019-12-05T06:43:06.121 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:06.114+0000 7fd0d5441700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0d008e5b0 con 0x7fd0d00a7740 2019-12-05T06:43:06.121 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:06.114+0000 7fd0cf7fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd0d00a7740 0x7fd0d008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:06.122 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:06.114+0000 7fd0cffff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fd0d009f2e0 0x7fd0d009f710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:34062/0 (socket says 172.21.15.42:34062) 2019-12-05T06:43:06.122 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:06.114+0000 7fd0cffff700 1 -- 172.21.15.42:0/1174588842 learned_addr learned my addr 172.21.15.42:0/1174588842 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:43:06.122 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:06.114+0000 7fd0ceffd700 1 --2- 172.21.15.42:0/1174588842 >> v2:172.21.15.40:3300/0 conn(0x7fd0d007e720 0x7fd0d007eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:06.122 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:06.114+0000 7fd0cf7fe700 1 -- 172.21.15.42:0/1174588842 >> v2:172.21.15.40:3300/0 conn(0x7fd0d007e720 msgr2=0x7fd0d007eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:43:06.122 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:06.114+0000 7fd0cf7fe700 1 --2- 172.21.15.42:0/1174588842 >> v2:172.21.15.40:3300/0 conn(0x7fd0d007e720 0x7fd0d007eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:43:06.122 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:06.114+0000 7fd0cf7fe700 1 -- 172.21.15.42:0/1174588842 >> v1:172.21.15.42:6789/0 conn(0x7fd0d009f2e0 legacy=0x7fd0d009f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:06.122 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:06.114+0000 7fd0ce7fc700 1 -- 172.21.15.42:0/1174588842 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2845279778 0 0) 0x7fd0d00a0710 con 0x7fd0d009f2e0 2019-12-05T06:43:06.123 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:06.114+0000 7fd0cf7fe700 1 -- 172.21.15.42:0/1174588842 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0d007f0a0 con 0x7fd0d00a7740 2019-12-05T06:43:06.123 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:06.114+0000 7fd0cf7fe700 1 --2- 172.21.15.42:0/1174588842 >> v2:172.21.15.42:3300/0 conn(0x7fd0d00a7740 0x7fd0d008df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fd0b8002f30 tx=0x7fd0b800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:06.123 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:06.114+0000 7fd0cf7fe700 1 --2- 172.21.15.42:0/1174588842 >> v2:172.21.15.42:3300/0 conn(0x7fd0d00a7740 0x7fd0d008df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fd0b8002f30 tx=0x7fd0b800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:06.123 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:06.114+0000 7fd0cf7fe700 1 --2- 172.21.15.42:0/1174588842 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd0d00a7740 0x7fd0d008df80 secure :-1 s=READY pgs=3083 cs=0 l=1 rx=0x7fd0b8002f30 tx=0x7fd0b800fc70).ready entity=mon.1 client_cookie=194468e8cf75e9ec server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:43:06.123 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:06.114+0000 7fd0cf7fe700 1 --2- 172.21.15.42:0/1174588842 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd0d00a7740 0x7fd0d008df80 secure :-1 s=THROTTLE_DONE pgs=3083 cs=0 l=1 rx=0x7fd0b8002f30 tx=0x7fd0b800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:06.124 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:06.114+0000 7fd0cf7fe700 1 --2- 172.21.15.42:0/1174588842 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd0d00a7740 0x7fd0d008df80 secure :-1 s=THROTTLE_DONE pgs=3083 cs=0 l=1 rx=0x7fd0b8002f30 tx=0x7fd0b800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:06.124 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:06.114+0000 7fd0ce7fc700 1 -- 172.21.15.42:0/1174588842 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd0b80112b0 con 0x7fd0d00a7740 2019-12-05T06:43:06.124 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:06.114+0000 7fd0cf7fe700 1 --2- 172.21.15.42:0/1174588842 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd0d00a7740 0x7fd0d008df80 secure :-1 s=THROTTLE_DONE pgs=3083 cs=0 l=1 rx=0x7fd0b8002f30 tx=0x7fd0b800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:06.124 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:06.114+0000 7fd0ce7fc700 1 -- 172.21.15.42:0/1174588842 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0b8008ad0 con 0x7fd0d00a7740 2019-12-05T06:43:06.124 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:06.114+0000 7fd0ce7fc700 1 -- 172.21.15.42:0/1174588842 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd0b8011890 con 0x7fd0d00a7740 2019-12-05T06:43:06.124 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:06.114+0000 7fd0d5441700 1 -- 172.21.15.42:0/1174588842 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd0d00a7740 msgr2=0x7fd0d008df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:06.125 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:06.114+0000 7fd0d5441700 1 --2- 172.21.15.42:0/1174588842 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd0d00a7740 0x7fd0d008df80 secure :-1 s=READY pgs=3083 cs=0 l=1 rx=0x7fd0b8002f30 tx=0x7fd0b800fc70).stop 2019-12-05T06:43:06.125 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:06.114+0000 7fd0d5441700 1 -- 172.21.15.42:0/1174588842 shutdown_connections 2019-12-05T06:43:06.125 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:06.114+0000 7fd0d5441700 1 --2- 172.21.15.42:0/1174588842 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd0d00a7740 0x7fd0d008df80 unknown :-1 s=CLOSED pgs=3083 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:06.125 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:06.114+0000 7fd0d5441700 1 --2- 172.21.15.42:0/1174588842 >> v2:172.21.15.40:3300/0 conn(0x7fd0d007e720 0x7fd0d007eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:43:06.125 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:06.114+0000 7fd0d5441700 1 -- 172.21.15.42:0/1174588842 shutdown_connections 2019-12-05T06:43:06.126 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:06.114+0000 7fd0d5441700 1 -- 172.21.15.42:0/1174588842 wait complete. 2019-12-05T06:43:06.126 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:06.114+0000 7fd0d5441700 1 -- 172.21.15.42:0/1174588842 >> 172.21.15.42:0/1174588842 conn(0x7fd0d00aabe0 msgr2=0x7fd0d00a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:43:06.129 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:06.118+0000 7fd0d5441700 1 Processor -- start 2019-12-05T06:43:06.129 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:06.118+0000 7fd0d5441700 1 -- start start 2019-12-05T06:43:06.129 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:06.118+0000 7fd0d5441700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd0d011eab0 0x7fd0d01211c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:43:06.129 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:06.118+0000 7fd0d5441700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd0d0124510 0x7fd0d0120050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:43:06.130 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:06.118+0000 7fd0d5441700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd0d01205a0 0x7fd0d0133a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:43:06.130 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:06.118+0000 7fd0d5441700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0d0133630 con 0x7fd0d0124510 2019-12-05T06:43:06.130 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:06.118+0000 7fd0d5441700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0d011f880 con 0x7fd0d01205a0 2019-12-05T06:43:06.130 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:06.118+0000 7fd0d5441700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0d01217f0 con 0x7fd0d011eab0 2019-12-05T06:43:06.131 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:06.118+0000 7fd0cffff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd0d01205a0 0x7fd0d0133a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:06.131 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:06.118+0000 7fd0cffff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd0d01205a0 0x7fd0d0133a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:34032/0 (socket says 172.21.15.42:34032) 2019-12-05T06:43:06.131 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:06.118+0000 7fd0ceffd700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd0d0124510 0x7fd0d0120050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:06.131 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:06.118+0000 7fd0cf7fe700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd0d011eab0 0x7fd0d01211c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:06.131 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:06.118+0000 7fd0cffff700 1 -- 172.21.15.42:0/1396928794 learned_addr learned my addr 172.21.15.42:0/1396928794 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:43:06.131 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:06.122+0000 7fd0cffff700 1 -- 172.21.15.42:0/1396928794 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd0d011eab0 msgr2=0x7fd0d01211c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:06.132 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:06.122+0000 7fd0cffff700 1 --2- 172.21.15.42:0/1396928794 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd0d011eab0 0x7fd0d01211c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:06.132 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:06.122+0000 7fd0cffff700 1 -- 172.21.15.42:0/1396928794 >> v2:172.21.15.40:3300/0 conn(0x7fd0d0124510 msgr2=0x7fd0d0120050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:06.132 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:06.122+0000 7fd0cffff700 1 --2- 172.21.15.42:0/1396928794 >> v2:172.21.15.40:3300/0 conn(0x7fd0d0124510 0x7fd0d0120050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:06.132 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:06.122+0000 7fd0cffff700 1 -- 172.21.15.42:0/1396928794 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0d01209d0 con 0x7fd0d01205a0 2019-12-05T06:43:06.132 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:06.122+0000 7fd0ceffd700 1 --2- 172.21.15.42:0/1396928794 >> v2:172.21.15.40:3300/0 conn(0x7fd0d0124510 0x7fd0d0120050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:43:06.132 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:06.122+0000 7fd0cffff700 1 --2- 172.21.15.42:0/1396928794 >> v2:172.21.15.42:3300/0 conn(0x7fd0d01205a0 0x7fd0d0133a20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fd0d01311a0 tx=0x7fd0c8010c20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:06.132 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:06.122+0000 7fd0cffff700 1 --2- 172.21.15.42:0/1396928794 >> v2:172.21.15.42:3300/0 conn(0x7fd0d01205a0 0x7fd0d0133a20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fd0d01311a0 tx=0x7fd0c8010c20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:06.133 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:06.122+0000 7fd0cffff700 1 --2- 172.21.15.42:0/1396928794 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd0d01205a0 0x7fd0d0133a20 secure :-1 s=READY pgs=3084 cs=0 l=1 rx=0x7fd0d01311a0 tx=0x7fd0c8010c20).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:43:06.133 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:06.122+0000 7fd0cffff700 1 --2- 172.21.15.42:0/1396928794 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd0d01205a0 0x7fd0d0133a20 secure :-1 s=THROTTLE_DONE pgs=3084 cs=0 l=1 rx=0x7fd0d01311a0 tx=0x7fd0c8010c20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:06.133 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:06.122+0000 7fd0cffff700 1 --2- 172.21.15.42:0/1396928794 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd0d01205a0 0x7fd0d0133a20 secure :-1 s=THROTTLE_DONE pgs=3084 cs=0 l=1 rx=0x7fd0d01311a0 tx=0x7fd0c8010c20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:06.133 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:06.122+0000 7fd0c7fff700 1 -- 172.21.15.42:0/1396928794 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd0c8004b90 con 0x7fd0d01205a0 2019-12-05T06:43:06.134 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:06.122+0000 7fd0cffff700 1 --2- 172.21.15.42:0/1396928794 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd0d01205a0 0x7fd0d0133a20 secure :-1 s=THROTTLE_DONE pgs=3084 cs=0 l=1 rx=0x7fd0d01311a0 tx=0x7fd0c8010c20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:06.134 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:06.122+0000 7fd0c7fff700 1 -- 172.21.15.42:0/1396928794 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0c80047b0 con 0x7fd0d01205a0 2019-12-05T06:43:06.134 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:06.122+0000 7fd0d5441700 1 -- 172.21.15.42:0/1396928794 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0d0121a90 con 0x7fd0d01205a0 2019-12-05T06:43:06.134 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:06.122+0000 7fd0c7fff700 1 -- 172.21.15.42:0/1396928794 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd0c8012790 con 0x7fd0d01205a0 2019-12-05T06:43:06.134 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:06.122+0000 7fd0d5441700 1 -- 172.21.15.42:0/1396928794 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0d0133f70 con 0x7fd0d01205a0 2019-12-05T06:43:06.134 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:06.122+0000 7fd0cffff700 1 --2- 172.21.15.42:0/1396928794 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd0d01205a0 0x7fd0d0133a20 secure :-1 s=THROTTLE_DONE pgs=3084 cs=0 l=1 rx=0x7fd0d01311a0 tx=0x7fd0c8010c20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:06.134 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:06.122+0000 7fd0c7fff700 1 -- 172.21.15.42:0/1396928794 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fd0c8012aa0 con 0x7fd0d01205a0 2019-12-05T06:43:06.134 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:06.122+0000 7fd0cffff700 1 --2- 172.21.15.42:0/1396928794 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd0d01205a0 0x7fd0d0133a20 secure :-1 s=THROTTLE_DONE pgs=3084 cs=0 l=1 rx=0x7fd0d01311a0 tx=0x7fd0c8010c20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:06.135 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:06.122+0000 7fd0c7fff700 1 --2- 172.21.15.42:0/1396928794 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd0b403ae30 0x7fd0b403d340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:43:06.135 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:06.122+0000 7fd0c7fff700 1 -- 172.21.15.42:0/1396928794 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 11793+0+0 (secure 0 0 0) 0x7fd0c804b4f0 con 0x7fd0d01205a0 2019-12-05T06:43:06.135 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:06.122+0000 7fd0cf7fe700 1 --2- 172.21.15.42:0/1396928794 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd0b403ae30 0x7fd0b403d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:06.135 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:06.126+0000 7fd0cf7fe700 1 --2- 172.21.15.42:0/1396928794 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd0b403ae30 0x7fd0b403d340 crc :-1 s=READY pgs=1287 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:43:06.136 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:06.126+0000 7fd0d5441700 1 -- 172.21.15.42:0/1396928794 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0d005d110 con 0x7fd0d01205a0 2019-12-05T06:43:06.138 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:06.134+0000 7fd0cffff700 1 --2- 172.21.15.42:0/1396928794 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd0d01205a0 0x7fd0d0133a20 secure :-1 s=THROTTLE_DONE pgs=3084 cs=0 l=1 rx=0x7fd0d01311a0 tx=0x7fd0c8010c20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:06.139 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:06.134+0000 7fd0c7fff700 1 -- 172.21.15.42:0/1396928794 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fd0d01209d0 con 0x7fd0d01205a0 2019-12-05T06:43:06.323 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:06.318+0000 7fd0d5441700 1 -- 172.21.15.42:0/1396928794 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd0d005b8c0 con 0x7fd0d01205a0 2019-12-05T06:43:06.324 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:06.322+0000 7fd0cffff700 1 --2- 172.21.15.42:0/1396928794 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd0d01205a0 0x7fd0d0133a20 secure :-1 s=THROTTLE_DONE pgs=3084 cs=0 l=1 rx=0x7fd0d01311a0 tx=0x7fd0c8010c20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:06.324 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:06.322+0000 7fd0c7fff700 1 -- 172.21.15.42:0/1396928794 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 372 v372) v1 ==== 95+0+2108 (secure 0 0 0) 0x7fd0c80324b0 con 0x7fd0d01205a0 2019-12-05T06:43:06.324 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 372 2019-12-05T06:43:06.325 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:43:06.325 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":372,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":12496,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/3599619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3599619},{"type":"v1","addr":"172.21.15.40:6833","nonce":3599619}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":372}],"filesystems":[{"mdsmap":{"epoch":371,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:42:53.534430+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":116,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12358},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12358":{"gid":12358,"name":"a","rank":0,"incarnation":368,"state":"up:active","state_seq":34,"addr":"172.21.15.42:6835/1922398668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1922398668},{"type":"v1","addr":"172.21.15.42:6835","nonce":1922398668}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:43:06.327 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:06.322+0000 7fd0d5441700 1 -- 172.21.15.42:0/1396928794 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd0b403ae30 msgr2=0x7fd0b403d340 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:06.328 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:06.322+0000 7fd0d5441700 1 --2- 172.21.15.42:0/1396928794 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd0b403ae30 0x7fd0b403d340 crc :-1 s=READY pgs=1287 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:06.328 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:06.322+0000 7fd0d5441700 1 -- 172.21.15.42:0/1396928794 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd0d01205a0 msgr2=0x7fd0d0133a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:06.328 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:06.322+0000 7fd0d5441700 1 --2- 172.21.15.42:0/1396928794 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd0d01205a0 0x7fd0d0133a20 secure :-1 s=READY pgs=3084 cs=0 l=1 rx=0x7fd0d01311a0 tx=0x7fd0c8010c20).stop 2019-12-05T06:43:06.328 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:06.322+0000 7fd0d5441700 1 -- 172.21.15.42:0/1396928794 shutdown_connections 2019-12-05T06:43:06.328 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:06.322+0000 7fd0d5441700 1 --2- 172.21.15.42:0/1396928794 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd0b403ae30 0x7fd0b403d340 unknown :-1 s=CLOSED pgs=1287 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:06.329 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:06.322+0000 7fd0d5441700 1 --2- 172.21.15.42:0/1396928794 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd0d01205a0 0x7fd0d0133a20 unknown :-1 s=CLOSED pgs=3084 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:06.329 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:06.322+0000 7fd0d5441700 1 --2- 172.21.15.42:0/1396928794 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd0d011eab0 0x7fd0d01211c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:06.329 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:06.322+0000 7fd0d5441700 1 --2- 172.21.15.42:0/1396928794 >> v2:172.21.15.40:3300/0 conn(0x7fd0d0124510 0x7fd0d0120050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:06.329 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:06.322+0000 7fd0d5441700 1 -- 172.21.15.42:0/1396928794 shutdown_connections 2019-12-05T06:43:06.329 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:06.322+0000 7fd0d5441700 1 -- 172.21.15.42:0/1396928794 wait complete. 2019-12-05T06:43:06.329 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:06.322+0000 7fd0d5441700 1 -- 172.21.15.42:0/1396928794 >> 172.21.15.42:0/1396928794 conn(0x7fd0d008fde0 msgr2=0x7fd0d009f650 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:43:06.341 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b reported in up:standby state 2019-12-05T06:43:06.341 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 6 secs before thrashing 2019-12-05T06:43:12.343 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:43:12.433 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.426+0000 7f129369e700 1 Processor -- start 2019-12-05T06:43:12.434 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.426+0000 7f129369e700 1 -- start start 2019-12-05T06:43:12.434 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.426+0000 7f129369e700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f128c08df70 0x7f128c09f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:43:12.434 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.426+0000 7f129369e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f128c09f820 0x7f128c14efd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:43:12.434 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.426+0000 7f129369e700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f128c09fc50 con 0x7f128c0a7730 2019-12-05T06:43:12.435 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.426+0000 7f129369e700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f128c0a0af0 con 0x7f128c08df70 2019-12-05T06:43:12.435 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.426+0000 7f129369e700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f128c08e5a0 con 0x7f128c09f820 2019-12-05T06:43:12.435 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.426+0000 7f129243c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f128c09f820 0x7f128c14efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:12.435 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.426+0000 7f1291c3b700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f128c0a7730 0x7f128c07e710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:34132/0 (socket says 172.21.15.42:34132) 2019-12-05T06:43:12.435 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.426+0000 7f1291c3b700 1 -- 172.21.15.42:0/243994104 learned_addr learned my addr 172.21.15.42:0/243994104 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:43:12.435 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.426+0000 7f129143a700 1 --2- 172.21.15.42:0/243994104 >> v2:172.21.15.40:3300/0 conn(0x7f128c08df70 0x7f128c09f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:12.436 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.426+0000 7f129243c700 1 -- 172.21.15.42:0/243994104 >> v2:172.21.15.40:3300/0 conn(0x7f128c08df70 msgr2=0x7f128c09f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:43:12.436 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.426+0000 7f129243c700 1 --2- 172.21.15.42:0/243994104 >> v2:172.21.15.40:3300/0 conn(0x7f128c08df70 0x7f128c09f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:43:12.436 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.426+0000 7f129243c700 1 -- 172.21.15.42:0/243994104 >> v1:172.21.15.42:6789/0 conn(0x7f128c0a7730 legacy=0x7f128c07e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:12.436 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.426+0000 7f129243c700 1 -- 172.21.15.42:0/243994104 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f128c0a0700 con 0x7f128c09f820 2019-12-05T06:43:12.436 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.426+0000 7f129243c700 1 --2- 172.21.15.42:0/243994104 >> v2:172.21.15.42:3300/0 conn(0x7f128c09f820 0x7f128c14efd0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f12840032b0 tx=0x7f128400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:12.436 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.426+0000 7f129243c700 1 --2- 172.21.15.42:0/243994104 >> v2:172.21.15.42:3300/0 conn(0x7f128c09f820 0x7f128c14efd0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f12840032b0 tx=0x7f128400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:12.437 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.426+0000 7f129243c700 1 --2- 172.21.15.42:0/243994104 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f128c09f820 0x7f128c14efd0 secure :-1 s=READY pgs=3086 cs=0 l=1 rx=0x7f12840032b0 tx=0x7f128400fff0).ready entity=mon.1 client_cookie=df946ffe963b528a server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:43:12.437 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.426+0000 7f129243c700 1 --2- 172.21.15.42:0/243994104 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f128c09f820 0x7f128c14efd0 secure :-1 s=THROTTLE_DONE pgs=3086 cs=0 l=1 rx=0x7f12840032b0 tx=0x7f128400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:12.437 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.426+0000 7f1290c39700 1 -- 172.21.15.42:0/243994104 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f12840108c0 con 0x7f128c09f820 2019-12-05T06:43:12.437 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.426+0000 7f129243c700 1 --2- 172.21.15.42:0/243994104 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f128c09f820 0x7f128c14efd0 secure :-1 s=THROTTLE_DONE pgs=3086 cs=0 l=1 rx=0x7f12840032b0 tx=0x7f128400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:12.437 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.426+0000 7f129243c700 1 --2- 172.21.15.42:0/243994104 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f128c09f820 0x7f128c14efd0 secure :-1 s=THROTTLE_DONE pgs=3086 cs=0 l=1 rx=0x7f12840032b0 tx=0x7f128400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:12.437 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.426+0000 7f1290c39700 1 -- 172.21.15.42:0/243994104 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1284008b50 con 0x7f128c09f820 2019-12-05T06:43:12.437 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.426+0000 7f1290c39700 1 -- 172.21.15.42:0/243994104 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f12840110c0 con 0x7f128c09f820 2019-12-05T06:43:12.438 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.426+0000 7f129369e700 1 -- 172.21.15.42:0/243994104 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f128c09f820 msgr2=0x7f128c14efd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:12.438 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.426+0000 7f129369e700 1 --2- 172.21.15.42:0/243994104 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f128c09f820 0x7f128c14efd0 secure :-1 s=READY pgs=3086 cs=0 l=1 rx=0x7f12840032b0 tx=0x7f128400fff0).stop 2019-12-05T06:43:12.438 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.430+0000 7f129369e700 1 -- 172.21.15.42:0/243994104 shutdown_connections 2019-12-05T06:43:12.438 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.430+0000 7f129369e700 1 --2- 172.21.15.42:0/243994104 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f128c09f820 0x7f128c14efd0 unknown :-1 s=CLOSED pgs=3086 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:12.438 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.430+0000 7f129369e700 1 --2- 172.21.15.42:0/243994104 >> v2:172.21.15.40:3300/0 conn(0x7f128c08df70 0x7f128c09f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:43:12.438 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.430+0000 7f129369e700 1 -- 172.21.15.42:0/243994104 shutdown_connections 2019-12-05T06:43:12.438 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.430+0000 7f129369e700 1 -- 172.21.15.42:0/243994104 wait complete. 2019-12-05T06:43:12.439 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.430+0000 7f129369e700 1 -- 172.21.15.42:0/243994104 >> 172.21.15.42:0/243994104 conn(0x7f128c0aabd0 msgr2=0x7f128c0a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:43:12.442 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.430+0000 7f129369e700 1 Processor -- start 2019-12-05T06:43:12.442 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.430+0000 7f129369e700 1 -- start start 2019-12-05T06:43:12.442 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.430+0000 7f129369e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f128c0a52a0 0x7f128c078850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:43:12.442 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.430+0000 7f129369e700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f128c0a56d0 con 0x7f128c12a250 2019-12-05T06:43:12.442 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.430+0000 7f129369e700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f128c0a5cc0 con 0x7f128c0a6f30 2019-12-05T06:43:12.442 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.430+0000 7f129369e700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f128c08eeb0 con 0x7f128c0a52a0 2019-12-05T06:43:12.442 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.434+0000 7f129143a700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f128c0a52a0 0x7f128c078850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:12.443 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.434+0000 7f129143a700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f128c0a52a0 0x7f128c078850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:34098/0 (socket says 172.21.15.42:34098) 2019-12-05T06:43:12.443 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.434+0000 7f129143a700 1 -- 172.21.15.42:0/1645317258 learned_addr learned my addr 172.21.15.42:0/1645317258 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:43:12.443 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.434+0000 7f1291c3b700 1 --1- 172.21.15.42:0/1645317258 >> v1:172.21.15.42:6789/0 conn(0x7f128c0a6f30 0x7f128c08f110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:34138/0 (socket says 172.21.15.42:34138) 2019-12-05T06:43:12.443 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.434+0000 7f129143a700 1 -- 172.21.15.42:0/1645317258 >> v1:172.21.15.42:6789/0 conn(0x7f128c0a6f30 legacy=0x7f128c08f110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:12.443 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.434+0000 7f129143a700 1 -- 172.21.15.42:0/1645317258 >> v1:172.21.15.40:6789/0 conn(0x7f128c12a250 legacy=0x7f128c12a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:12.443 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.434+0000 7f129143a700 1 -- 172.21.15.42:0/1645317258 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f128c08f910 con 0x7f128c0a52a0 2019-12-05T06:43:12.443 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.434+0000 7f129143a700 1 --2- 172.21.15.42:0/1645317258 >> v2:172.21.15.42:3300/0 conn(0x7f128c0a52a0 0x7f128c078850 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f127c007980 tx=0x7f127c010b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:12.444 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.434+0000 7f129143a700 1 --2- 172.21.15.42:0/1645317258 >> v2:172.21.15.42:3300/0 conn(0x7f128c0a52a0 0x7f128c078850 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f127c007980 tx=0x7f127c010b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:12.444 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.434+0000 7f129143a700 1 --2- 172.21.15.42:0/1645317258 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f128c0a52a0 0x7f128c078850 secure :-1 s=READY pgs=3088 cs=0 l=1 rx=0x7f127c007980 tx=0x7f127c010b30).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:43:12.444 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.434+0000 7f129143a700 1 --2- 172.21.15.42:0/1645317258 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f128c0a52a0 0x7f128c078850 secure :-1 s=THROTTLE_DONE pgs=3088 cs=0 l=1 rx=0x7f127c007980 tx=0x7f127c010b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:12.444 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.434+0000 7f129143a700 1 --2- 172.21.15.42:0/1645317258 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f128c0a52a0 0x7f128c078850 secure :-1 s=THROTTLE_DONE pgs=3088 cs=0 l=1 rx=0x7f127c007980 tx=0x7f127c010b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:12.444 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.434+0000 7f129143a700 1 --2- 172.21.15.42:0/1645317258 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f128c0a52a0 0x7f128c078850 secure :-1 s=THROTTLE_DONE pgs=3088 cs=0 l=1 rx=0x7f127c007980 tx=0x7f127c010b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:12.444 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.434+0000 7f128a7fc700 1 -- 172.21.15.42:0/1645317258 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f127c005980 con 0x7f128c0a52a0 2019-12-05T06:43:12.445 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.434+0000 7f128a7fc700 1 -- 172.21.15.42:0/1645317258 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f127c007160 con 0x7f128c0a52a0 2019-12-05T06:43:12.445 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.434+0000 7f129369e700 1 -- 172.21.15.42:0/1645317258 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f128c132280 con 0x7f128c0a52a0 2019-12-05T06:43:12.445 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.434+0000 7f128a7fc700 1 -- 172.21.15.42:0/1645317258 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f127c0121b0 con 0x7f128c0a52a0 2019-12-05T06:43:12.445 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.434+0000 7f129369e700 1 -- 172.21.15.42:0/1645317258 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f128c132780 con 0x7f128c0a52a0 2019-12-05T06:43:12.445 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.434+0000 7f129143a700 1 --2- 172.21.15.42:0/1645317258 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f128c0a52a0 0x7f128c078850 secure :-1 s=THROTTLE_DONE pgs=3088 cs=0 l=1 rx=0x7f127c007980 tx=0x7f127c010b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:12.445 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.434+0000 7f128a7fc700 1 -- 172.21.15.42:0/1645317258 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f127c007bc0 con 0x7f128c0a52a0 2019-12-05T06:43:12.446 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.434+0000 7f129143a700 1 --2- 172.21.15.42:0/1645317258 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f128c0a52a0 0x7f128c078850 secure :-1 s=THROTTLE_DONE pgs=3088 cs=0 l=1 rx=0x7f127c007980 tx=0x7f127c010b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:12.446 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.434+0000 7f128a7fc700 1 --2- 172.21.15.42:0/1645317258 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f127003adb0 0x7f127003d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:43:12.446 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.434+0000 7f128a7fc700 1 -- 172.21.15.42:0/1645317258 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 11793+0+0 (secure 0 0 0) 0x7f127c04b080 con 0x7f128c0a52a0 2019-12-05T06:43:12.446 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.434+0000 7f1291c3b700 1 --2- 172.21.15.42:0/1645317258 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f127003adb0 0x7f127003d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:12.446 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.438+0000 7f1291c3b700 1 --2- 172.21.15.42:0/1645317258 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f127003adb0 0x7f127003d2c0 crc :-1 s=READY pgs=1289 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:43:12.447 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.442+0000 7f129369e700 1 -- 172.21.15.42:0/1645317258 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f128c129e20 con 0x7f128c0a52a0 2019-12-05T06:43:12.451 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.446+0000 7f129143a700 1 --2- 172.21.15.42:0/1645317258 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f128c0a52a0 0x7f128c078850 secure :-1 s=THROTTLE_DONE pgs=3088 cs=0 l=1 rx=0x7f127c007980 tx=0x7f127c010b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:12.451 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.446+0000 7f128a7fc700 1 -- 172.21.15.42:0/1645317258 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f127c032100 con 0x7f128c0a52a0 2019-12-05T06:43:12.617 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.614+0000 7f129369e700 1 -- 172.21.15.42:0/1645317258 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f128c05d100 con 0x7f128c0a52a0 2019-12-05T06:43:12.618 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.614+0000 7f129143a700 1 --2- 172.21.15.42:0/1645317258 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f128c0a52a0 0x7f128c078850 secure :-1 s=THROTTLE_DONE pgs=3088 cs=0 l=1 rx=0x7f127c007980 tx=0x7f127c010b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:12.618 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.614+0000 7f128a7fc700 1 -- 172.21.15.42:0/1645317258 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 372 v372) v1 ==== 95+0+2108 (secure 0 0 0) 0x7f127c032360 con 0x7f128c0a52a0 2019-12-05T06:43:12.618 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 372 2019-12-05T06:43:12.618 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:43:12.619 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":372,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":12496,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/3599619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3599619},{"type":"v1","addr":"172.21.15.40:6833","nonce":3599619}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":372}],"filesystems":[{"mdsmap":{"epoch":371,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:42:53.534430+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":116,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12358},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12358":{"gid":12358,"name":"a","rank":0,"incarnation":368,"state":"up:active","state_seq":34,"addr":"172.21.15.42:6835/1922398668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1922398668},{"type":"v1","addr":"172.21.15.42:6835","nonce":1922398668}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:43:12.620 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.618+0000 7f129369e700 1 -- 172.21.15.42:0/1645317258 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f127003adb0 msgr2=0x7f127003d2c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:12.621 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.618+0000 7f129369e700 1 --2- 172.21.15.42:0/1645317258 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f127003adb0 0x7f127003d2c0 crc :-1 s=READY pgs=1289 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:12.621 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.618+0000 7f129369e700 1 -- 172.21.15.42:0/1645317258 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f128c0a52a0 msgr2=0x7f128c078850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:12.621 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.618+0000 7f129369e700 1 --2- 172.21.15.42:0/1645317258 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f128c0a52a0 0x7f128c078850 secure :-1 s=READY pgs=3088 cs=0 l=1 rx=0x7f127c007980 tx=0x7f127c010b30).stop 2019-12-05T06:43:12.622 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.618+0000 7f129369e700 1 -- 172.21.15.42:0/1645317258 shutdown_connections 2019-12-05T06:43:12.622 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.618+0000 7f129369e700 1 --2- 172.21.15.42:0/1645317258 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f127003adb0 0x7f127003d2c0 unknown :-1 s=CLOSED pgs=1289 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:12.622 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.618+0000 7f129369e700 1 --2- 172.21.15.42:0/1645317258 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f128c0a52a0 0x7f128c078850 unknown :-1 s=CLOSED pgs=3088 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:12.622 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.618+0000 7f129369e700 1 -- 172.21.15.42:0/1645317258 shutdown_connections 2019-12-05T06:43:12.623 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.618+0000 7f129369e700 1 -- 172.21.15.42:0/1645317258 wait complete. 2019-12-05T06:43:12.623 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.618+0000 7f129369e700 1 -- 172.21.15.42:0/1645317258 >> 172.21.15.42:0/1645317258 conn(0x7f128c14ebc0 msgr2=0x7f128c09f210 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:43:12.633 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.a (rank=0) 2019-12-05T06:43:12.633 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2019-12-05T06:43:12.633 INFO:teuthology.orchestra.run:waiting for 300 2019-12-05T06:43:12.643 INFO:tasks.ceph.mds.a:Stopped 2019-12-05T06:43:12.643 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:43:12.808 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.798+0000 7f93a51ae700 1 Processor -- start 2019-12-05T06:43:12.809 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.798+0000 7f93a51ae700 1 -- start start 2019-12-05T06:43:12.809 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.802+0000 7f93a51ae700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f93a009fae0 0x7f93a014fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:43:12.809 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.802+0000 7f93a51ae700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93a00a0710 con 0x7f93a00a7740 2019-12-05T06:43:12.809 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.802+0000 7f93a51ae700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93a007ef20 con 0x7f93a008df80 2019-12-05T06:43:12.809 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.802+0000 7f93a51ae700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93a00a0b00 con 0x7f93a009fae0 2019-12-05T06:43:12.810 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.802+0000 7f93a49ad700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f93a009fae0 0x7f93a014fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:12.810 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.802+0000 7f939ed9d700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f93a008df80 0x7f93a009f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:34148/0 (socket says 172.21.15.42:34148) 2019-12-05T06:43:12.810 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.802+0000 7f93a49ad700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f93a009fae0 0x7f93a014fff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:34110/0 (socket says 172.21.15.42:34110) 2019-12-05T06:43:12.810 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.802+0000 7f939ed9d700 1 -- 172.21.15.42:0/1921227797 learned_addr learned my addr 172.21.15.42:0/1921227797 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:43:12.810 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.802+0000 7f93a49ad700 1 -- 172.21.15.42:0/1921227797 >> v1:172.21.15.42:6789/0 conn(0x7f93a008df80 legacy=0x7f93a009f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:12.810 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.802+0000 7f93a49ad700 1 -- 172.21.15.42:0/1921227797 >> v1:172.21.15.40:6789/0 conn(0x7f93a00a7740 legacy=0x7f93a007e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:12.810 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.802+0000 7f939e59c700 1 -- 172.21.15.42:0/1921227797 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2509166154 0 0) 0x7f93a007ef20 con 0x7f93a008df80 2019-12-05T06:43:12.811 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.802+0000 7f93a49ad700 1 -- 172.21.15.42:0/1921227797 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93a00a9ba0 con 0x7f93a009fae0 2019-12-05T06:43:12.811 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.802+0000 7f93a49ad700 1 --2- 172.21.15.42:0/1921227797 >> v2:172.21.15.42:3300/0 conn(0x7f93a009fae0 0x7f93a014fff0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f9398008790 tx=0x7f9394001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:12.811 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.802+0000 7f93a49ad700 1 --2- 172.21.15.42:0/1921227797 >> v2:172.21.15.42:3300/0 conn(0x7f93a009fae0 0x7f93a014fff0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f9398008790 tx=0x7f9394001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:12.811 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.802+0000 7f93a49ad700 1 --2- 172.21.15.42:0/1921227797 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f93a009fae0 0x7f93a014fff0 secure :-1 s=READY pgs=3090 cs=0 l=1 rx=0x7f9398008790 tx=0x7f9394001150).ready entity=mon.1 client_cookie=f805432d62e98432 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:43:12.811 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.802+0000 7f93a49ad700 1 --2- 172.21.15.42:0/1921227797 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f93a009fae0 0x7f93a014fff0 secure :-1 s=THROTTLE_DONE pgs=3090 cs=0 l=1 rx=0x7f9398008790 tx=0x7f9394001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:12.811 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.802+0000 7f93a49ad700 1 --2- 172.21.15.42:0/1921227797 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f93a009fae0 0x7f93a014fff0 secure :-1 s=THROTTLE_DONE pgs=3090 cs=0 l=1 rx=0x7f9398008790 tx=0x7f9394001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:12.812 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.802+0000 7f939e59c700 1 -- 172.21.15.42:0/1921227797 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9398008bb0 con 0x7f93a009fae0 2019-12-05T06:43:12.812 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.802+0000 7f93a49ad700 1 --2- 172.21.15.42:0/1921227797 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f93a009fae0 0x7f93a014fff0 secure :-1 s=THROTTLE_DONE pgs=3090 cs=0 l=1 rx=0x7f9398008790 tx=0x7f9394001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:12.812 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.802+0000 7f939e59c700 1 -- 172.21.15.42:0/1921227797 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93980087d0 con 0x7f93a009fae0 2019-12-05T06:43:12.812 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.802+0000 7f939e59c700 1 -- 172.21.15.42:0/1921227797 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f93980117d0 con 0x7f93a009fae0 2019-12-05T06:43:12.812 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.802+0000 7f93a51ae700 1 -- 172.21.15.42:0/1921227797 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f93a009fae0 msgr2=0x7f93a014fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:12.812 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.802+0000 7f93a51ae700 1 --2- 172.21.15.42:0/1921227797 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f93a009fae0 0x7f93a014fff0 secure :-1 s=READY pgs=3090 cs=0 l=1 rx=0x7f9398008790 tx=0x7f9394001150).stop 2019-12-05T06:43:12.813 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.802+0000 7f93a51ae700 1 -- 172.21.15.42:0/1921227797 shutdown_connections 2019-12-05T06:43:12.813 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.802+0000 7f93a51ae700 1 --2- 172.21.15.42:0/1921227797 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f93a009fae0 0x7f93a014fff0 unknown :-1 s=CLOSED pgs=3090 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:12.813 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.802+0000 7f93a51ae700 1 -- 172.21.15.42:0/1921227797 shutdown_connections 2019-12-05T06:43:12.813 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.802+0000 7f93a51ae700 1 -- 172.21.15.42:0/1921227797 wait complete. 2019-12-05T06:43:12.813 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.802+0000 7f93a51ae700 1 -- 172.21.15.42:0/1921227797 >> 172.21.15.42:0/1921227797 conn(0x7f93a00aabe0 msgr2=0x7f93a00a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:43:12.818 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.806+0000 7f93a51ae700 1 Processor -- start 2019-12-05T06:43:12.818 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.806+0000 7f93a51ae700 1 -- start start 2019-12-05T06:43:12.818 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.806+0000 7f93a51ae700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f93a007c7f0 0x7f93a012d070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:43:12.818 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.806+0000 7f93a51ae700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f93a012d5c0 0x7f93a007ae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:43:12.818 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.806+0000 7f93a51ae700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f93a007b3d0 0x7f93a012b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:43:12.818 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.806+0000 7f93a51ae700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93a012e010 con 0x7f93a012d5c0 2019-12-05T06:43:12.819 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.806+0000 7f93a51ae700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93a012dad0 con 0x7f93a007c7f0 2019-12-05T06:43:12.819 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.806+0000 7f93a51ae700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93a012dd70 con 0x7f93a007b3d0 2019-12-05T06:43:12.819 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.806+0000 7f939f59e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f93a007c7f0 0x7f93a012d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:12.819 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.806+0000 7f939f59e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f93a007c7f0 0x7f93a012d070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:34114/0 (socket says 172.21.15.42:34114) 2019-12-05T06:43:12.819 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.806+0000 7f939f59e700 1 -- 172.21.15.42:0/2257065976 learned_addr learned my addr 172.21.15.42:0/2257065976 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:43:12.820 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.806+0000 7f939ed9d700 1 --2- 172.21.15.42:0/2257065976 >> v2:172.21.15.40:3300/0 conn(0x7f93a012d5c0 0x7f93a007ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:12.820 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.806+0000 7f93a49ad700 1 --2- 172.21.15.42:0/2257065976 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f93a007b3d0 0x7f93a012b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:12.820 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.806+0000 7f939f59e700 1 -- 172.21.15.42:0/2257065976 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f93a007b3d0 msgr2=0x7f93a012b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:12.820 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.806+0000 7f939f59e700 1 --2- 172.21.15.42:0/2257065976 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f93a007b3d0 0x7f93a012b0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:12.820 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.806+0000 7f939f59e700 1 -- 172.21.15.42:0/2257065976 >> v2:172.21.15.40:3300/0 conn(0x7f93a012d5c0 msgr2=0x7f93a007ae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:12.821 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.806+0000 7f939f59e700 1 --2- 172.21.15.42:0/2257065976 >> v2:172.21.15.40:3300/0 conn(0x7f93a012d5c0 0x7f93a007ae80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:12.821 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.806+0000 7f939f59e700 1 -- 172.21.15.42:0/2257065976 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93a007d340 con 0x7f93a007c7f0 2019-12-05T06:43:12.821 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.806+0000 7f939ed9d700 1 --2- 172.21.15.42:0/2257065976 >> v2:172.21.15.40:3300/0 conn(0x7f93a012d5c0 0x7f93a007ae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:43:12.821 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.806+0000 7f93a49ad700 1 --2- 172.21.15.42:0/2257065976 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f93a007b3d0 0x7f93a012b0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:43:12.821 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.810+0000 7f939f59e700 1 --2- 172.21.15.42:0/2257065976 >> v2:172.21.15.42:3300/0 conn(0x7f93a007c7f0 0x7f93a012d070 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f93a012df30 tx=0x7f9390010e20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:12.821 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.810+0000 7f939f59e700 1 --2- 172.21.15.42:0/2257065976 >> v2:172.21.15.42:3300/0 conn(0x7f93a007c7f0 0x7f93a012d070 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f93a012df30 tx=0x7f9390010e20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:12.821 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.810+0000 7f939f59e700 1 --2- 172.21.15.42:0/2257065976 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f93a007c7f0 0x7f93a012d070 secure :-1 s=READY pgs=3091 cs=0 l=1 rx=0x7f93a012df30 tx=0x7f9390010e20).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:43:12.822 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.810+0000 7f939f59e700 1 --2- 172.21.15.42:0/2257065976 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f93a007c7f0 0x7f93a012d070 secure :-1 s=THROTTLE_DONE pgs=3091 cs=0 l=1 rx=0x7f93a012df30 tx=0x7f9390010e20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:12.822 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.810+0000 7f9387fff700 1 -- 172.21.15.42:0/2257065976 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f939000ca40 con 0x7f93a007c7f0 2019-12-05T06:43:12.822 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.810+0000 7f939f59e700 1 --2- 172.21.15.42:0/2257065976 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f93a007c7f0 0x7f93a012d070 secure :-1 s=THROTTLE_DONE pgs=3091 cs=0 l=1 rx=0x7f93a012df30 tx=0x7f9390010e20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:12.822 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.810+0000 7f939f59e700 1 --2- 172.21.15.42:0/2257065976 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f93a007c7f0 0x7f93a012d070 secure :-1 s=THROTTLE_DONE pgs=3091 cs=0 l=1 rx=0x7f93a012df30 tx=0x7f9390010e20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:12.822 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.810+0000 7f9387fff700 1 -- 172.21.15.42:0/2257065976 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93900101c0 con 0x7f93a007c7f0 2019-12-05T06:43:12.823 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.810+0000 7f93a51ae700 1 -- 172.21.15.42:0/2257065976 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93a012b620 con 0x7f93a007c7f0 2019-12-05T06:43:12.823 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.810+0000 7f9387fff700 1 -- 172.21.15.42:0/2257065976 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9390012690 con 0x7f93a007c7f0 2019-12-05T06:43:12.823 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.810+0000 7f93a51ae700 1 -- 172.21.15.42:0/2257065976 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93a012bbd0 con 0x7f93a007c7f0 2019-12-05T06:43:12.823 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.810+0000 7f939f59e700 1 --2- 172.21.15.42:0/2257065976 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f93a007c7f0 0x7f93a012d070 secure :-1 s=THROTTLE_DONE pgs=3091 cs=0 l=1 rx=0x7f93a012df30 tx=0x7f9390010e20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:12.824 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.810+0000 7f9387fff700 1 -- 172.21.15.42:0/2257065976 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f9390012930 con 0x7f93a007c7f0 2019-12-05T06:43:12.824 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.810+0000 7f939f59e700 1 --2- 172.21.15.42:0/2257065976 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f93a007c7f0 0x7f93a012d070 secure :-1 s=THROTTLE_DONE pgs=3091 cs=0 l=1 rx=0x7f93a012df30 tx=0x7f9390010e20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:12.824 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.810+0000 7f9387fff700 1 --2- 172.21.15.42:0/2257065976 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f938003ad20 0x7f938003d230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:43:12.824 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.810+0000 7f9387fff700 1 -- 172.21.15.42:0/2257065976 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 11793+0+0 (secure 0 0 0) 0x7f939004b270 con 0x7f93a007c7f0 2019-12-05T06:43:12.825 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.810+0000 7f939ed9d700 1 --2- 172.21.15.42:0/2257065976 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f938003ad20 0x7f938003d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:12.825 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.810+0000 7f939ed9d700 1 --2- 172.21.15.42:0/2257065976 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f938003ad20 0x7f938003d230 crc :-1 s=READY pgs=1290 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:43:12.826 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.814+0000 7f93a51ae700 1 -- 172.21.15.42:0/2257065976 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93a005d110 con 0x7f93a007c7f0 2019-12-05T06:43:12.827 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.822+0000 7f939f59e700 1 --2- 172.21.15.42:0/2257065976 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f93a007c7f0 0x7f93a012d070 secure :-1 s=THROTTLE_DONE pgs=3091 cs=0 l=1 rx=0x7f93a012df30 tx=0x7f9390010e20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:12.827 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.822+0000 7f9387fff700 1 -- 172.21.15.42:0/2257065976 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f93a012bbd0 con 0x7f93a007c7f0 2019-12-05T06:43:12.995 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.990+0000 7f93a51ae700 1 -- 172.21.15.42:0/2257065976 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f93a00a71e0 con 0x7f93a007c7f0 2019-12-05T06:43:12.996 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.994+0000 7f939f59e700 1 --2- 172.21.15.42:0/2257065976 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f93a007c7f0 0x7f93a012d070 secure :-1 s=THROTTLE_DONE pgs=3091 cs=0 l=1 rx=0x7f93a012df30 tx=0x7f9390010e20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:12.997 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.994+0000 7f9387fff700 1 -- 172.21.15.42:0/2257065976 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 372 v372) v1 ==== 95+0+2108 (secure 0 0 0) 0x7f9390032210 con 0x7f93a007c7f0 2019-12-05T06:43:12.997 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 372 2019-12-05T06:43:12.997 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:43:12.997 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":372,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":12496,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/3599619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3599619},{"type":"v1","addr":"172.21.15.40:6833","nonce":3599619}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":372}],"filesystems":[{"mdsmap":{"epoch":371,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:42:53.534430+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":116,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12358},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12358":{"gid":12358,"name":"a","rank":0,"incarnation":368,"state":"up:active","state_seq":34,"addr":"172.21.15.42:6835/1922398668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1922398668},{"type":"v1","addr":"172.21.15.42:6835","nonce":1922398668}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:43:12.999 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.994+0000 7f93a51ae700 1 -- 172.21.15.42:0/2257065976 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f938003ad20 msgr2=0x7f938003d230 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:12.999 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.994+0000 7f93a51ae700 1 --2- 172.21.15.42:0/2257065976 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f938003ad20 0x7f938003d230 crc :-1 s=READY pgs=1290 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:13.000 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.994+0000 7f93a51ae700 1 -- 172.21.15.42:0/2257065976 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f93a007c7f0 msgr2=0x7f93a012d070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:13.000 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.994+0000 7f93a51ae700 1 --2- 172.21.15.42:0/2257065976 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f93a007c7f0 0x7f93a012d070 secure :-1 s=READY pgs=3091 cs=0 l=1 rx=0x7f93a012df30 tx=0x7f9390010e20).stop 2019-12-05T06:43:13.000 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.994+0000 7f93a51ae700 1 -- 172.21.15.42:0/2257065976 shutdown_connections 2019-12-05T06:43:13.000 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.994+0000 7f93a51ae700 1 --2- 172.21.15.42:0/2257065976 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f938003ad20 0x7f938003d230 unknown :-1 s=CLOSED pgs=1290 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:13.000 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.994+0000 7f93a51ae700 1 --2- 172.21.15.42:0/2257065976 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f93a007b3d0 0x7f93a012b0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:13.000 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.994+0000 7f93a51ae700 1 --2- 172.21.15.42:0/2257065976 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f93a007c7f0 0x7f93a012d070 unknown :-1 s=CLOSED pgs=3091 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:13.000 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.994+0000 7f93a51ae700 1 --2- 172.21.15.42:0/2257065976 >> v2:172.21.15.40:3300/0 conn(0x7f93a012d5c0 0x7f93a007ae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:13.001 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.994+0000 7f93a51ae700 1 -- 172.21.15.42:0/2257065976 shutdown_connections 2019-12-05T06:43:13.001 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.994+0000 7f93a51ae700 1 -- 172.21.15.42:0/2257065976 wait complete. 2019-12-05T06:43:13.001 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:12.994+0000 7f93a51ae700 1 -- 172.21.15.42:0/2257065976 >> 172.21.15.42:0/2257065976 conn(0x7f93a0066fe0 msgr2=0x7f93a011c8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:43:13.009 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:43:15.025 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:43:15.131 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:15.122+0000 7f990cc8e700 1 Processor -- start 2019-12-05T06:43:15.131 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:15.122+0000 7f990cc8e700 1 -- start start 2019-12-05T06:43:15.131 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:15.122+0000 7f990cc8e700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f99080a7740 0x7f990808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:43:15.131 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:15.122+0000 7f990cc8e700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f990809fae0 0x7f990807eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:43:15.132 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:15.122+0000 7f990cc8e700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99080a0710 con 0x7f990807e720 2019-12-05T06:43:15.132 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:15.122+0000 7f990cc8e700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99080a0b00 con 0x7f99080a7740 2019-12-05T06:43:15.132 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:15.122+0000 7f990cc8e700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f990808e5b0 con 0x7f990809fae0 2019-12-05T06:43:15.132 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:15.122+0000 7f990659c700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f990807e720 0x7f990809f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:34174/0 (socket says 172.21.15.42:34174) 2019-12-05T06:43:15.132 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:15.122+0000 7f990659c700 1 -- 172.21.15.42:0/2506820425 learned_addr learned my addr 172.21.15.42:0/2506820425 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:43:15.132 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:15.122+0000 7f9906d9d700 1 --2- 172.21.15.42:0/2506820425 >> v2:172.21.15.40:3300/0 conn(0x7f99080a7740 0x7f990808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:15.132 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:15.122+0000 7f990759e700 1 --2- 172.21.15.42:0/2506820425 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f990809fae0 0x7f990807eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:15.133 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:15.126+0000 7f9905d9b700 1 -- 172.21.15.42:0/2506820425 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4052473709 0 0) 0x7f99080a0710 con 0x7f990807e720 2019-12-05T06:43:15.133 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:15.126+0000 7f9905d9b700 1 -- 172.21.15.42:0/2506820425 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98f00030b0 con 0x7f990807e720 2019-12-05T06:43:15.133 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:15.126+0000 7f990759e700 1 -- 172.21.15.42:0/2506820425 >> v2:172.21.15.40:3300/0 conn(0x7f99080a7740 msgr2=0x7f990808df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:43:15.133 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:15.126+0000 7f990759e700 1 --2- 172.21.15.42:0/2506820425 >> v2:172.21.15.40:3300/0 conn(0x7f99080a7740 0x7f990808df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:43:15.133 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:15.126+0000 7f990759e700 1 -- 172.21.15.42:0/2506820425 >> v1:172.21.15.42:6789/0 conn(0x7f990807e720 legacy=0x7f990809f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:15.133 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:15.126+0000 7f990759e700 1 -- 172.21.15.42:0/2506820425 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f990807f0a0 con 0x7f990809fae0 2019-12-05T06:43:15.133 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:15.126+0000 7f990759e700 1 --2- 172.21.15.42:0/2506820425 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f990809fae0 0x7f990807eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f9900008b10 tx=0x7f990000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:15.134 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:15.126+0000 7f990759e700 1 --2- 172.21.15.42:0/2506820425 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f990809fae0 0x7f990807eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f9900008b10 tx=0x7f990000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:15.134 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:15.126+0000 7f990759e700 1 --2- 172.21.15.42:0/2506820425 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f990809fae0 0x7f990807eb50 secure :-1 s=READY pgs=672 cs=0 l=1 rx=0x7f9900008b10 tx=0x7f990000fff0).ready entity=mon.2 client_cookie=971d7b24448bac9a server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:43:15.134 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:15.126+0000 7f990759e700 1 --2- 172.21.15.42:0/2506820425 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f990809fae0 0x7f990807eb50 secure :-1 s=THROTTLE_DONE pgs=672 cs=0 l=1 rx=0x7f9900008b10 tx=0x7f990000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:15.134 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:15.126+0000 7f9905d9b700 1 -- 172.21.15.42:0/2506820425 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9900010900 con 0x7f990809fae0 2019-12-05T06:43:15.134 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:15.126+0000 7f990759e700 1 --2- 172.21.15.42:0/2506820425 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f990809fae0 0x7f990807eb50 secure :-1 s=THROTTLE_DONE pgs=672 cs=0 l=1 rx=0x7f9900008b10 tx=0x7f990000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:15.134 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:15.126+0000 7f990759e700 1 --2- 172.21.15.42:0/2506820425 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f990809fae0 0x7f990807eb50 secure :-1 s=THROTTLE_DONE pgs=672 cs=0 l=1 rx=0x7f9900008b10 tx=0x7f990000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:15.134 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:15.126+0000 7f9905d9b700 1 -- 172.21.15.42:0/2506820425 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9900008b50 con 0x7f990809fae0 2019-12-05T06:43:15.135 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:15.126+0000 7f9905d9b700 1 -- 172.21.15.42:0/2506820425 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f99000110c0 con 0x7f990809fae0 2019-12-05T06:43:15.135 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:15.126+0000 7f990cc8e700 1 -- 172.21.15.42:0/2506820425 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f990809fae0 msgr2=0x7f990807eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:15.135 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:15.126+0000 7f990cc8e700 1 --2- 172.21.15.42:0/2506820425 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f990809fae0 0x7f990807eb50 secure :-1 s=READY pgs=672 cs=0 l=1 rx=0x7f9900008b10 tx=0x7f990000fff0).stop 2019-12-05T06:43:15.135 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:15.126+0000 7f990cc8e700 1 -- 172.21.15.42:0/2506820425 shutdown_connections 2019-12-05T06:43:15.135 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:15.126+0000 7f990cc8e700 1 --2- 172.21.15.42:0/2506820425 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f990809fae0 0x7f990807eb50 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:15.135 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:15.126+0000 7f990cc8e700 1 --2- 172.21.15.42:0/2506820425 >> v2:172.21.15.40:3300/0 conn(0x7f99080a7740 0x7f990808df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:43:15.135 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:15.126+0000 7f990cc8e700 1 -- 172.21.15.42:0/2506820425 shutdown_connections 2019-12-05T06:43:15.136 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:15.126+0000 7f990cc8e700 1 -- 172.21.15.42:0/2506820425 wait complete. 2019-12-05T06:43:15.136 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:15.126+0000 7f990cc8e700 1 -- 172.21.15.42:0/2506820425 >> 172.21.15.42:0/2506820425 conn(0x7f99080aabe0 msgr2=0x7f99080a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:43:15.139 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:15.130+0000 7f990cc8e700 1 Processor -- start 2019-12-05T06:43:15.139 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:15.130+0000 7f990cc8e700 1 -- start start 2019-12-05T06:43:15.140 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:15.130+0000 7f990cc8e700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f9908130160 0x7f9908130590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:43:15.140 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:15.130+0000 7f990cc8e700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9908130ae0 con 0x7f990812cf50 2019-12-05T06:43:15.140 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:15.130+0000 7f990cc8e700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99081395d0 con 0x7f990811b950 2019-12-05T06:43:15.140 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:15.130+0000 7f990cc8e700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9908130dc0 con 0x7f9908130160 2019-12-05T06:43:15.140 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:15.130+0000 7f9906d9d700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f990811b950 0x7f990807ae80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:34178/0 (socket says 172.21.15.42:34178) 2019-12-05T06:43:15.140 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:15.130+0000 7f9906d9d700 1 -- 172.21.15.42:0/1777107090 learned_addr learned my addr 172.21.15.42:0/1777107090 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:43:15.141 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:15.130+0000 7f990659c700 1 --2- 172.21.15.42:0/1777107090 >> v2:172.21.15.40:3300/0 conn(0x7f9908130160 0x7f9908130590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:15.141 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:15.130+0000 7f98f77fe700 1 -- 172.21.15.42:0/1777107090 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 898765296 0 0) 0x7f99081395d0 con 0x7f990811b950 2019-12-05T06:43:15.141 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:15.130+0000 7f98f77fe700 1 -- 172.21.15.42:0/1777107090 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98ec0030b0 con 0x7f990811b950 2019-12-05T06:43:15.141 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:15.130+0000 7f990659c700 1 -- 172.21.15.42:0/1777107090 >> v1:172.21.15.42:6789/0 conn(0x7f990811b950 legacy=0x7f990807ae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:15.141 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:15.130+0000 7f990659c700 1 -- 172.21.15.42:0/1777107090 >> v1:172.21.15.40:6789/0 conn(0x7f990812cf50 legacy=0x7f990812d380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:15.141 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:15.130+0000 7f990659c700 1 -- 172.21.15.42:0/1777107090 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f990813a810 con 0x7f9908130160 2019-12-05T06:43:15.141 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:15.130+0000 7f990659c700 1 --2- 172.21.15.42:0/1777107090 >> v2:172.21.15.40:3300/0 conn(0x7f9908130160 0x7f9908130590 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f98fc0043a0 tx=0x7f98fc010860).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:15.142 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:15.130+0000 7f990659c700 1 --2- 172.21.15.42:0/1777107090 >> v2:172.21.15.40:3300/0 conn(0x7f9908130160 0x7f9908130590 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f98fc0043a0 tx=0x7f98fc010860).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:15.142 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:15.130+0000 7f990659c700 1 --2- 172.21.15.42:0/1777107090 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9908130160 0x7f9908130590 secure :-1 s=READY pgs=2334 cs=0 l=1 rx=0x7f98fc0043a0 tx=0x7f98fc010860).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:43:15.142 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:15.130+0000 7f990659c700 1 --2- 172.21.15.42:0/1777107090 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9908130160 0x7f9908130590 secure :-1 s=THROTTLE_DONE pgs=2334 cs=0 l=1 rx=0x7f98fc0043a0 tx=0x7f98fc010860).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:15.142 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:15.130+0000 7f98f77fe700 1 -- 172.21.15.42:0/1777107090 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f98fc005c30 con 0x7f9908130160 2019-12-05T06:43:15.142 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:15.130+0000 7f990659c700 1 --2- 172.21.15.42:0/1777107090 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9908130160 0x7f9908130590 secure :-1 s=THROTTLE_DONE pgs=2334 cs=0 l=1 rx=0x7f98fc0043a0 tx=0x7f98fc010860).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:15.142 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:15.130+0000 7f990cc8e700 1 -- 172.21.15.42:0/1777107090 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99081385c0 con 0x7f9908130160 2019-12-05T06:43:15.142 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:15.130+0000 7f98f77fe700 1 -- 172.21.15.42:0/1777107090 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98fc005850 con 0x7f9908130160 2019-12-05T06:43:15.143 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:15.130+0000 7f990659c700 1 --2- 172.21.15.42:0/1777107090 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9908130160 0x7f9908130590 secure :-1 s=THROTTLE_DONE pgs=2334 cs=0 l=1 rx=0x7f98fc0043a0 tx=0x7f98fc010860).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:15.143 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:15.130+0000 7f990cc8e700 1 -- 172.21.15.42:0/1777107090 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9908138b70 con 0x7f9908130160 2019-12-05T06:43:15.143 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:15.130+0000 7f98f77fe700 1 -- 172.21.15.42:0/1777107090 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f98fc009630 con 0x7f9908130160 2019-12-05T06:43:15.143 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:15.134+0000 7f990659c700 1 --2- 172.21.15.42:0/1777107090 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9908130160 0x7f9908130590 secure :-1 s=THROTTLE_DONE pgs=2334 cs=0 l=1 rx=0x7f98fc0043a0 tx=0x7f98fc010860).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:15.143 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:15.134+0000 7f98f77fe700 1 -- 172.21.15.42:0/1777107090 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f98fc009830 con 0x7f9908130160 2019-12-05T06:43:15.143 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:15.134+0000 7f990659c700 1 --2- 172.21.15.42:0/1777107090 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9908130160 0x7f9908130590 secure :-1 s=THROTTLE_DONE pgs=2334 cs=0 l=1 rx=0x7f98fc0043a0 tx=0x7f98fc010860).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:15.144 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:15.134+0000 7f98f77fe700 1 --2- 172.21.15.42:0/1777107090 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f98ec03d3d0 0x7f98ec03f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:43:15.144 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:15.134+0000 7f98f77fe700 1 -- 172.21.15.42:0/1777107090 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 11793+0+0 (secure 0 0 0) 0x7f98fc04b0c0 con 0x7f9908130160 2019-12-05T06:43:15.144 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:15.134+0000 7f9906d9d700 1 --2- 172.21.15.42:0/1777107090 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f98ec03d3d0 0x7f98ec03f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:15.144 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:15.134+0000 7f9906d9d700 1 --2- 172.21.15.42:0/1777107090 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f98ec03d3d0 0x7f98ec03f8e0 crc :-1 s=READY pgs=1292 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:43:15.145 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:15.138+0000 7f990cc8e700 1 -- 172.21.15.42:0/1777107090 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f990805d110 con 0x7f9908130160 2019-12-05T06:43:15.151 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:15.146+0000 7f990659c700 1 --2- 172.21.15.42:0/1777107090 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9908130160 0x7f9908130590 secure :-1 s=THROTTLE_DONE pgs=2334 cs=0 l=1 rx=0x7f98fc0043a0 tx=0x7f98fc010860).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:15.151 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:15.146+0000 7f98f77fe700 1 -- 172.21.15.42:0/1777107090 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f98fc032120 con 0x7f9908130160 2019-12-05T06:43:15.320 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:15.314+0000 7f990cc8e700 1 -- 172.21.15.42:0/1777107090 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f990805b8c0 con 0x7f9908130160 2019-12-05T06:43:15.358 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:15.318+0000 7f990659c700 1 --2- 172.21.15.42:0/1777107090 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9908130160 0x7f9908130590 secure :-1 s=THROTTLE_DONE pgs=2334 cs=0 l=1 rx=0x7f98fc0043a0 tx=0x7f98fc010860).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:15.359 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:15.318+0000 7f98f77fe700 1 -- 172.21.15.42:0/1777107090 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 372 v372) v1 ==== 95+0+2108 (secure 0 0 0) 0x7f98fc032380 con 0x7f9908130160 2019-12-05T06:43:15.359 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 372 2019-12-05T06:43:15.359 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:15.322+0000 7f990cc8e700 1 -- 172.21.15.42:0/1777107090 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f98ec03d3d0 msgr2=0x7f98ec03f8e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:15.360 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:15.322+0000 7f990cc8e700 1 --2- 172.21.15.42:0/1777107090 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f98ec03d3d0 0x7f98ec03f8e0 crc :-1 s=READY pgs=1292 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:15.360 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:15.322+0000 7f990cc8e700 1 -- 172.21.15.42:0/1777107090 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9908130160 msgr2=0x7f9908130590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:15.360 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:15.322+0000 7f990cc8e700 1 --2- 172.21.15.42:0/1777107090 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9908130160 0x7f9908130590 secure :-1 s=READY pgs=2334 cs=0 l=1 rx=0x7f98fc0043a0 tx=0x7f98fc010860).stop 2019-12-05T06:43:15.360 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:15.322+0000 7f990cc8e700 1 -- 172.21.15.42:0/1777107090 shutdown_connections 2019-12-05T06:43:15.361 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:15.322+0000 7f990cc8e700 1 --2- 172.21.15.42:0/1777107090 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f98ec03d3d0 0x7f98ec03f8e0 unknown :-1 s=CLOSED pgs=1292 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:15.361 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:15.322+0000 7f990cc8e700 1 --2- 172.21.15.42:0/1777107090 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9908130160 0x7f9908130590 unknown :-1 s=CLOSED pgs=2334 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:15.361 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:15.322+0000 7f990cc8e700 1 -- 172.21.15.42:0/1777107090 shutdown_connections 2019-12-05T06:43:15.361 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:15.322+0000 7f990cc8e700 1 -- 172.21.15.42:0/1777107090 wait complete. 2019-12-05T06:43:15.362 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:15.322+0000 7f990cc8e700 1 -- 172.21.15.42:0/1777107090 >> 172.21.15.42:0/1777107090 conn(0x7f990807e5d0 msgr2=0x7f9908059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:43:15.362 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:43:15.362 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":372,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":12496,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/3599619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3599619},{"type":"v1","addr":"172.21.15.40:6833","nonce":3599619}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":372}],"filesystems":[{"mdsmap":{"epoch":371,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:42:53.534430+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":116,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12358},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12358":{"gid":12358,"name":"a","rank":0,"incarnation":368,"state":"up:active","state_seq":34,"addr":"172.21.15.42:6835/1922398668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1922398668},{"type":"v1","addr":"172.21.15.42:6835","nonce":1922398668}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:43:15.363 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:43:15.495 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:43:15.499 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:43:17.364 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:43:17.472 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:17.462+0000 7f8b46a1e700 1 Processor -- start 2019-12-05T06:43:17.472 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:17.462+0000 7f8b46a1e700 1 -- start start 2019-12-05T06:43:17.472 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:17.462+0000 7f8b46a1e700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8b4008df80 0x7f8b4009f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:43:17.473 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:17.462+0000 7f8b46a1e700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8b4009f830 0x7f8b4014efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:43:17.473 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:17.462+0000 7f8b46a1e700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b4009fc60 con 0x7f8b400a7740 2019-12-05T06:43:17.473 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:17.462+0000 7f8b46a1e700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b400a0b00 con 0x7f8b4009f830 2019-12-05T06:43:17.473 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:17.462+0000 7f8b46a1e700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b4008e5b0 con 0x7f8b4008df80 2019-12-05T06:43:17.473 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:17.462+0000 7f8b457bc700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8b4009f830 0x7f8b4014efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:17.473 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:17.462+0000 7f8b3ffff700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8b4008df80 0x7f8b4009f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:17.473 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:17.462+0000 7f8b457bc700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8b4009f830 0x7f8b4014efe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.42:50202/0 (socket says 172.21.15.42:50202) 2019-12-05T06:43:17.474 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:17.462+0000 7f8b457bc700 1 -- 172.21.15.42:0/2311144092 learned_addr learned my addr 172.21.15.42:0/2311144092 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:43:17.474 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:17.462+0000 7f8b44fbb700 1 --1- 172.21.15.42:0/2311144092 >> v1:172.21.15.40:6789/0 conn(0x7f8b400a7740 0x7f8b4007e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:43630/0 (socket says 172.21.15.42:43630) 2019-12-05T06:43:17.474 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:17.462+0000 7f8b3ffff700 1 -- 172.21.15.42:0/2311144092 >> v2:172.21.15.40:3300/0 conn(0x7f8b4009f830 msgr2=0x7f8b4014efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:43:17.474 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:17.462+0000 7f8b3ffff700 1 --2- 172.21.15.42:0/2311144092 >> v2:172.21.15.40:3300/0 conn(0x7f8b4009f830 0x7f8b4014efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:43:17.474 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:17.462+0000 7f8b3ffff700 1 -- 172.21.15.42:0/2311144092 >> v1:172.21.15.40:6789/0 conn(0x7f8b400a7740 legacy=0x7f8b4007e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:17.474 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:17.462+0000 7f8b3ffff700 1 -- 172.21.15.42:0/2311144092 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b400a0710 con 0x7f8b4008df80 2019-12-05T06:43:17.474 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:17.462+0000 7f8b3ffff700 1 --2- 172.21.15.42:0/2311144092 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8b4008df80 0x7f8b4009f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f8b28008b10 tx=0x7f8b2800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:17.475 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:17.462+0000 7f8b3ffff700 1 --2- 172.21.15.42:0/2311144092 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8b4008df80 0x7f8b4009f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f8b28008b10 tx=0x7f8b2800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:17.475 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:17.462+0000 7f8b3ffff700 1 --2- 172.21.15.42:0/2311144092 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8b4008df80 0x7f8b4009f2e0 secure :-1 s=READY pgs=673 cs=0 l=1 rx=0x7f8b28008b10 tx=0x7f8b2800fff0).ready entity=mon.2 client_cookie=f0abc81c791236be server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:43:17.475 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:17.462+0000 7f8b3ffff700 1 --2- 172.21.15.42:0/2311144092 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8b4008df80 0x7f8b4009f2e0 secure :-1 s=THROTTLE_DONE pgs=673 cs=0 l=1 rx=0x7f8b28008b10 tx=0x7f8b2800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:17.475 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:17.462+0000 7f8b3ffff700 1 --2- 172.21.15.42:0/2311144092 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8b4008df80 0x7f8b4009f2e0 secure :-1 s=THROTTLE_DONE pgs=673 cs=0 l=1 rx=0x7f8b28008b10 tx=0x7f8b2800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:17.475 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:17.462+0000 7f8b3f7fe700 1 -- 172.21.15.42:0/2311144092 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8b28010900 con 0x7f8b4008df80 2019-12-05T06:43:17.475 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:17.462+0000 7f8b3ffff700 1 --2- 172.21.15.42:0/2311144092 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8b4008df80 0x7f8b4009f2e0 secure :-1 s=THROTTLE_DONE pgs=673 cs=0 l=1 rx=0x7f8b28008b10 tx=0x7f8b2800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:17.475 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:17.462+0000 7f8b3f7fe700 1 -- 172.21.15.42:0/2311144092 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b28008b50 con 0x7f8b4008df80 2019-12-05T06:43:17.476 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:17.466+0000 7f8b3f7fe700 1 -- 172.21.15.42:0/2311144092 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8b280110c0 con 0x7f8b4008df80 2019-12-05T06:43:17.476 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:17.466+0000 7f8b46a1e700 1 -- 172.21.15.42:0/2311144092 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8b4008df80 msgr2=0x7f8b4009f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:17.476 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:17.466+0000 7f8b46a1e700 1 --2- 172.21.15.42:0/2311144092 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8b4008df80 0x7f8b4009f2e0 secure :-1 s=READY pgs=673 cs=0 l=1 rx=0x7f8b28008b10 tx=0x7f8b2800fff0).stop 2019-12-05T06:43:17.476 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:17.466+0000 7f8b46a1e700 1 -- 172.21.15.42:0/2311144092 shutdown_connections 2019-12-05T06:43:17.476 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:17.466+0000 7f8b46a1e700 1 --2- 172.21.15.42:0/2311144092 >> v2:172.21.15.40:3300/0 conn(0x7f8b4009f830 0x7f8b4014efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:43:17.476 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:17.466+0000 7f8b46a1e700 1 --2- 172.21.15.42:0/2311144092 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8b4008df80 0x7f8b4009f2e0 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:17.476 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:17.466+0000 7f8b46a1e700 1 -- 172.21.15.42:0/2311144092 shutdown_connections 2019-12-05T06:43:17.477 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:17.466+0000 7f8b46a1e700 1 -- 172.21.15.42:0/2311144092 wait complete. 2019-12-05T06:43:17.477 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:17.466+0000 7f8b46a1e700 1 -- 172.21.15.42:0/2311144092 >> 172.21.15.42:0/2311144092 conn(0x7f8b400aabe0 msgr2=0x7f8b400a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:43:17.477 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:17.466+0000 7f8b46a1e700 1 Processor -- start 2019-12-05T06:43:17.477 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:17.466+0000 7f8b46a1e700 1 -- start start 2019-12-05T06:43:17.477 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:17.466+0000 7f8b46a1e700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8b40125660 0x7f8b40123270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:43:17.477 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:17.466+0000 7f8b46a1e700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8b40124b10 0x7f8b401186e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:43:17.477 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:17.466+0000 7f8b46a1e700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b40119eb0 con 0x7f8b401246e0 2019-12-05T06:43:17.478 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:17.466+0000 7f8b46a1e700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b40119710 con 0x7f8b40125660 2019-12-05T06:43:17.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:17.466+0000 7f8b46a1e700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b40118c30 con 0x7f8b40124b10 2019-12-05T06:43:17.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:17.466+0000 7f8b3ffff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f8b401246e0 0x7f8b40117ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:34194/0 (socket says 172.21.15.42:34194) 2019-12-05T06:43:17.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:17.466+0000 7f8b3ffff700 1 -- 172.21.15.42:0/1591662448 learned_addr learned my addr 172.21.15.42:0/1591662448 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:43:17.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:17.466+0000 7f8b457bc700 1 --2- 172.21.15.42:0/1591662448 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8b40124b10 0x7f8b401186e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:17.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:17.466+0000 7f8b44fbb700 1 --2- 172.21.15.42:0/1591662448 >> v2:172.21.15.40:3300/0 conn(0x7f8b40125660 0x7f8b40123270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:17.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:17.466+0000 7f8b3d7fa700 1 -- 172.21.15.42:0/1591662448 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1758823075 0 0) 0x7f8b280110c0 con 0x7f8b401246e0 2019-12-05T06:43:17.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:17.466+0000 7f8b3d7fa700 1 -- 172.21.15.42:0/1591662448 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b240030b0 con 0x7f8b401246e0 2019-12-05T06:43:17.486 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:17.470+0000 7f8b3d7fa700 1 -- 172.21.15.42:0/1591662448 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 2957426290 0 0) 0x7f8b240030b0 con 0x7f8b401246e0 2019-12-05T06:43:17.486 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:17.470+0000 7f8b457bc700 1 -- 172.21.15.42:0/1591662448 >> v2:172.21.15.40:3300/0 conn(0x7f8b40125660 msgr2=0x7f8b40123270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:17.487 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:17.470+0000 7f8b457bc700 1 --2- 172.21.15.42:0/1591662448 >> v2:172.21.15.40:3300/0 conn(0x7f8b40125660 0x7f8b40123270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:17.487 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:17.470+0000 7f8b457bc700 1 -- 172.21.15.42:0/1591662448 >> v1:172.21.15.42:6789/0 conn(0x7f8b401246e0 legacy=0x7f8b40117ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:17.487 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:17.470+0000 7f8b457bc700 1 -- 172.21.15.42:0/1591662448 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b40124f40 con 0x7f8b40124b10 2019-12-05T06:43:17.487 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:17.470+0000 7f8b44fbb700 1 --2- 172.21.15.42:0/1591662448 >> v2:172.21.15.40:3300/0 conn(0x7f8b40125660 0x7f8b40123270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:43:17.488 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:17.470+0000 7f8b457bc700 1 --2- 172.21.15.42:0/1591662448 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8b40124b10 0x7f8b401186e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f8b28008f60 tx=0x7f8b28010580).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:17.488 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:17.470+0000 7f8b457bc700 1 --2- 172.21.15.42:0/1591662448 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8b40124b10 0x7f8b401186e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f8b28008f60 tx=0x7f8b28010580).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:17.488 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:17.470+0000 7f8b457bc700 1 --2- 172.21.15.42:0/1591662448 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8b40124b10 0x7f8b401186e0 secure :-1 s=READY pgs=674 cs=0 l=1 rx=0x7f8b28008f60 tx=0x7f8b28010580).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:43:17.488 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:17.470+0000 7f8b457bc700 1 --2- 172.21.15.42:0/1591662448 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8b40124b10 0x7f8b401186e0 secure :-1 s=THROTTLE_DONE pgs=674 cs=0 l=1 rx=0x7f8b28008f60 tx=0x7f8b28010580).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:17.489 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:17.470+0000 7f8b3d7fa700 1 -- 172.21.15.42:0/1591662448 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8b28010900 con 0x7f8b40124b10 2019-12-05T06:43:17.489 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:17.470+0000 7f8b457bc700 1 --2- 172.21.15.42:0/1591662448 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8b40124b10 0x7f8b401186e0 secure :-1 s=THROTTLE_DONE pgs=674 cs=0 l=1 rx=0x7f8b28008f60 tx=0x7f8b28010580).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:17.489 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:17.470+0000 7f8b3d7fa700 1 -- 172.21.15.42:0/1591662448 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b28010b00 con 0x7f8b40124b10 2019-12-05T06:43:17.490 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:17.470+0000 7f8b457bc700 1 --2- 172.21.15.42:0/1591662448 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8b40124b10 0x7f8b401186e0 secure :-1 s=THROTTLE_DONE pgs=674 cs=0 l=1 rx=0x7f8b28008f60 tx=0x7f8b28010580).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:17.490 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:17.470+0000 7f8b3d7fa700 1 -- 172.21.15.42:0/1591662448 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8b38005b20 con 0x7f8b40124b10 2019-12-05T06:43:17.490 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:17.470+0000 7f8b46a1e700 1 -- 172.21.15.42:0/1591662448 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b40125a90 con 0x7f8b40124b10 2019-12-05T06:43:17.491 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:17.470+0000 7f8b46a1e700 1 -- 172.21.15.42:0/1591662448 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b401237c0 con 0x7f8b40124b10 2019-12-05T06:43:17.491 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:17.470+0000 7f8b457bc700 1 --2- 172.21.15.42:0/1591662448 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8b40124b10 0x7f8b401186e0 secure :-1 s=THROTTLE_DONE pgs=674 cs=0 l=1 rx=0x7f8b28008f60 tx=0x7f8b28010580).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:17.491 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:17.470+0000 7f8b3d7fa700 1 -- 172.21.15.42:0/1591662448 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f8b380088b0 con 0x7f8b40124b10 2019-12-05T06:43:17.491 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:17.470+0000 7f8b457bc700 1 --2- 172.21.15.42:0/1591662448 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8b40124b10 0x7f8b401186e0 secure :-1 s=THROTTLE_DONE pgs=674 cs=0 l=1 rx=0x7f8b28008f60 tx=0x7f8b28010580).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:17.491 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:17.474+0000 7f8b3d7fa700 1 --2- 172.21.15.42:0/1591662448 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8b2403b1e0 0x7f8b2403d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:43:17.492 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:17.474+0000 7f8b3d7fa700 1 -- 172.21.15.42:0/1591662448 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 11793+0+0 (secure 0 0 0) 0x7f8b3804a830 con 0x7f8b40124b10 2019-12-05T06:43:17.492 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:17.474+0000 7f8b44fbb700 1 --2- 172.21.15.42:0/1591662448 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8b2403b1e0 0x7f8b2403d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:17.492 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:17.474+0000 7f8b44fbb700 1 --2- 172.21.15.42:0/1591662448 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8b2403b1e0 0x7f8b2403d6f0 crc :-1 s=READY pgs=1293 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:43:17.492 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:17.478+0000 7f8b46a1e700 1 -- 172.21.15.42:0/1591662448 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b4005d110 con 0x7f8b40124b10 2019-12-05T06:43:17.493 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:17.486+0000 7f8b457bc700 1 --2- 172.21.15.42:0/1591662448 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8b40124b10 0x7f8b401186e0 secure :-1 s=THROTTLE_DONE pgs=674 cs=0 l=1 rx=0x7f8b28008f60 tx=0x7f8b28010580).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:17.493 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:17.486+0000 7f8b3d7fa700 1 -- 172.21.15.42:0/1591662448 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f8b40124f40 con 0x7f8b40124b10 2019-12-05T06:43:17.666 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:17.662+0000 7f8b46a1e700 1 -- 172.21.15.42:0/1591662448 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8b4005b8c0 con 0x7f8b40124b10 2019-12-05T06:43:17.668 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:17.662+0000 7f8b457bc700 1 --2- 172.21.15.42:0/1591662448 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8b40124b10 0x7f8b401186e0 secure :-1 s=THROTTLE_DONE pgs=674 cs=0 l=1 rx=0x7f8b28008f60 tx=0x7f8b28010580).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:17.668 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:17.662+0000 7f8b3d7fa700 1 -- 172.21.15.42:0/1591662448 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 372 v372) v1 ==== 95+0+2108 (secure 0 0 0) 0x7f8b38031890 con 0x7f8b40124b10 2019-12-05T06:43:17.675 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 372 2019-12-05T06:43:17.675 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:43:17.676 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":372,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":12496,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/3599619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3599619},{"type":"v1","addr":"172.21.15.40:6833","nonce":3599619}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":372}],"filesystems":[{"mdsmap":{"epoch":371,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:42:53.534430+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":116,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12358},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12358":{"gid":12358,"name":"a","rank":0,"incarnation":368,"state":"up:active","state_seq":34,"addr":"172.21.15.42:6835/1922398668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1922398668},{"type":"v1","addr":"172.21.15.42:6835","nonce":1922398668}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:43:17.680 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:17.666+0000 7f8b46a1e700 1 -- 172.21.15.42:0/1591662448 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8b2403b1e0 msgr2=0x7f8b2403d6f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:17.681 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:17.666+0000 7f8b46a1e700 1 --2- 172.21.15.42:0/1591662448 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8b2403b1e0 0x7f8b2403d6f0 crc :-1 s=READY pgs=1293 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:17.681 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:17.666+0000 7f8b46a1e700 1 -- 172.21.15.42:0/1591662448 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8b40124b10 msgr2=0x7f8b401186e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:17.681 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:17.666+0000 7f8b46a1e700 1 --2- 172.21.15.42:0/1591662448 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8b40124b10 0x7f8b401186e0 secure :-1 s=READY pgs=674 cs=0 l=1 rx=0x7f8b28008f60 tx=0x7f8b28010580).stop 2019-12-05T06:43:17.681 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:17.666+0000 7f8b46a1e700 1 -- 172.21.15.42:0/1591662448 shutdown_connections 2019-12-05T06:43:17.681 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:17.666+0000 7f8b46a1e700 1 --2- 172.21.15.42:0/1591662448 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8b2403b1e0 0x7f8b2403d6f0 unknown :-1 s=CLOSED pgs=1293 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:17.682 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:17.666+0000 7f8b46a1e700 1 --2- 172.21.15.42:0/1591662448 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8b40124b10 0x7f8b401186e0 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:17.682 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:17.666+0000 7f8b46a1e700 1 --2- 172.21.15.42:0/1591662448 >> v2:172.21.15.40:3300/0 conn(0x7f8b40125660 0x7f8b40123270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:17.682 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:17.666+0000 7f8b46a1e700 1 -- 172.21.15.42:0/1591662448 shutdown_connections 2019-12-05T06:43:17.682 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:17.666+0000 7f8b46a1e700 1 -- 172.21.15.42:0/1591662448 wait complete. 2019-12-05T06:43:17.682 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:17.666+0000 7f8b46a1e700 1 -- 172.21.15.42:0/1591662448 >> 172.21.15.42:0/1591662448 conn(0x7f8b400a6bb0 msgr2=0x7f8b400a3720 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:43:17.684 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:43:19.684 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:43:19.793 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:19.782+0000 7f28794ee700 1 Processor -- start 2019-12-05T06:43:19.794 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:19.782+0000 7f28794ee700 1 -- start start 2019-12-05T06:43:19.794 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:19.782+0000 7f28794ee700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f287408df70 0x7f287409f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:43:19.794 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:19.782+0000 7f28794ee700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f287409f820 0x7f287414efd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:43:19.795 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:19.782+0000 7f28794ee700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f287409fc50 con 0x7f28740a7730 2019-12-05T06:43:19.795 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:19.782+0000 7f28794ee700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28740a0af0 con 0x7f287409f820 2019-12-05T06:43:19.795 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:19.782+0000 7f28794ee700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f287408e5a0 con 0x7f287408df70 2019-12-05T06:43:19.795 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:19.786+0000 7f2873fff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f287409f820 0x7f287414efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:19.796 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:19.786+0000 7f2873fff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f287409f820 0x7f287414efd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:34170/0 (socket says 172.21.15.42:34170) 2019-12-05T06:43:19.796 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:19.786+0000 7f2873fff700 1 -- 172.21.15.42:0/775100287 learned_addr learned my addr 172.21.15.42:0/775100287 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:43:19.796 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:19.786+0000 7f2872ffd700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f287408df70 0x7f287409f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:19.796 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:19.786+0000 7f28737fe700 1 --1- 172.21.15.42:0/775100287 >> v1:172.21.15.40:6789/0 conn(0x7f28740a7730 0x7f287407e710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:43646/0 (socket says 172.21.15.42:43646) 2019-12-05T06:43:19.797 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:19.786+0000 7f2873fff700 1 -- 172.21.15.42:0/775100287 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f287408df70 msgr2=0x7f287409f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:43:19.797 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:19.786+0000 7f2873fff700 1 --2- 172.21.15.42:0/775100287 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f287408df70 0x7f287409f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:43:19.797 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:19.786+0000 7f2873fff700 1 -- 172.21.15.42:0/775100287 >> v1:172.21.15.40:6789/0 conn(0x7f28740a7730 legacy=0x7f287407e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:19.797 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:19.786+0000 7f2873fff700 1 -- 172.21.15.42:0/775100287 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28740a0700 con 0x7f287409f820 2019-12-05T06:43:19.797 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:19.786+0000 7f2872ffd700 1 --2- 172.21.15.42:0/775100287 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f287408df70 0x7f287409f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:43:19.798 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:19.786+0000 7f28727fc700 1 -- 172.21.15.42:0/775100287 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 346936378 0 0) 0x7f287409fc50 con 0x7f28740a7730 2019-12-05T06:43:19.798 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:19.786+0000 7f2873fff700 1 --2- 172.21.15.42:0/775100287 >> v2:172.21.15.42:3300/0 conn(0x7f287409f820 0x7f287414efd0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f286c008790 tx=0x7f286c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:19.798 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:19.786+0000 7f2873fff700 1 --2- 172.21.15.42:0/775100287 >> v2:172.21.15.42:3300/0 conn(0x7f287409f820 0x7f287414efd0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f286c008790 tx=0x7f286c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:19.798 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:19.786+0000 7f2873fff700 1 --2- 172.21.15.42:0/775100287 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f287409f820 0x7f287414efd0 secure :-1 s=READY pgs=3095 cs=0 l=1 rx=0x7f286c008790 tx=0x7f286c00fd30).ready entity=mon.1 client_cookie=2aec96db799868e server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:43:19.798 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:19.786+0000 7f2873fff700 1 --2- 172.21.15.42:0/775100287 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f287409f820 0x7f287414efd0 secure :-1 s=THROTTLE_DONE pgs=3095 cs=0 l=1 rx=0x7f286c008790 tx=0x7f286c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:19.798 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:19.786+0000 7f28727fc700 1 -- 172.21.15.42:0/775100287 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f286c008c90 con 0x7f287409f820 2019-12-05T06:43:19.799 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:19.786+0000 7f2873fff700 1 --2- 172.21.15.42:0/775100287 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f287409f820 0x7f287414efd0 secure :-1 s=THROTTLE_DONE pgs=3095 cs=0 l=1 rx=0x7f286c008790 tx=0x7f286c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:19.799 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:19.786+0000 7f2873fff700 1 --2- 172.21.15.42:0/775100287 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f287409f820 0x7f287414efd0 secure :-1 s=THROTTLE_DONE pgs=3095 cs=0 l=1 rx=0x7f286c008790 tx=0x7f286c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:19.799 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:19.786+0000 7f28727fc700 1 -- 172.21.15.42:0/775100287 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f286c0088b0 con 0x7f287409f820 2019-12-05T06:43:19.799 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:19.786+0000 7f28727fc700 1 -- 172.21.15.42:0/775100287 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f286c011b10 con 0x7f287409f820 2019-12-05T06:43:19.799 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:19.786+0000 7f28794ee700 1 -- 172.21.15.42:0/775100287 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f287409f820 msgr2=0x7f287414efd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:19.800 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:19.786+0000 7f28794ee700 1 --2- 172.21.15.42:0/775100287 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f287409f820 0x7f287414efd0 secure :-1 s=READY pgs=3095 cs=0 l=1 rx=0x7f286c008790 tx=0x7f286c00fd30).stop 2019-12-05T06:43:19.800 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:19.786+0000 7f28794ee700 1 -- 172.21.15.42:0/775100287 shutdown_connections 2019-12-05T06:43:19.800 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:19.786+0000 7f28794ee700 1 --2- 172.21.15.42:0/775100287 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f287409f820 0x7f287414efd0 unknown :-1 s=CLOSED pgs=3095 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:19.800 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:19.786+0000 7f28794ee700 1 --2- 172.21.15.42:0/775100287 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f287408df70 0x7f287409f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:43:19.800 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:19.786+0000 7f28794ee700 1 -- 172.21.15.42:0/775100287 shutdown_connections 2019-12-05T06:43:19.801 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:19.786+0000 7f28794ee700 1 -- 172.21.15.42:0/775100287 wait complete. 2019-12-05T06:43:19.801 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:19.786+0000 7f28794ee700 1 -- 172.21.15.42:0/775100287 >> 172.21.15.42:0/775100287 conn(0x7f28740aabd0 msgr2=0x7f28740a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:43:19.809 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:19.790+0000 7f28794ee700 1 Processor -- start 2019-12-05T06:43:19.809 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:19.790+0000 7f28794ee700 1 -- start start 2019-12-05T06:43:19.810 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:19.790+0000 7f28794ee700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f28740a3740 0x7f2874120c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:43:19.810 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:19.794+0000 7f28794ee700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28741211d0 con 0x7f287412e810 2019-12-05T06:43:19.810 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:19.794+0000 7f28794ee700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2874076ae0 con 0x7f28740a3b70 2019-12-05T06:43:19.810 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:19.794+0000 7f28794ee700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28741195e0 con 0x7f28740a3740 2019-12-05T06:43:19.810 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:19.794+0000 7f2872ffd700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f28740a3b70 0x7f287412e010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:34212/0 (socket says 172.21.15.42:34212) 2019-12-05T06:43:19.811 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:19.794+0000 7f2872ffd700 1 -- 172.21.15.42:0/3250481110 learned_addr learned my addr 172.21.15.42:0/3250481110 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:43:19.811 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:19.794+0000 7f28737fe700 1 --2- 172.21.15.42:0/3250481110 >> v2:172.21.15.40:3300/0 conn(0x7f28740a3740 0x7f2874120c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:19.811 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:19.794+0000 7f286bfff700 1 -- 172.21.15.42:0/3250481110 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3270508461 0 0) 0x7f2874076ae0 con 0x7f28740a3b70 2019-12-05T06:43:19.811 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:19.794+0000 7f286bfff700 1 -- 172.21.15.42:0/3250481110 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28580030b0 con 0x7f28740a3b70 2019-12-05T06:43:19.811 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:19.794+0000 7f286bfff700 1 -- 172.21.15.42:0/3250481110 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4224015112 0 0) 0x7f28741211d0 con 0x7f287412e810 2019-12-05T06:43:19.812 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:19.794+0000 7f286bfff700 1 -- 172.21.15.42:0/3250481110 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2874076ae0 con 0x7f287412e810 2019-12-05T06:43:19.812 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:19.794+0000 7f286bfff700 1 -- 172.21.15.42:0/3250481110 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 1475955134 0 0) 0x7f28580030b0 con 0x7f28740a3b70 2019-12-05T06:43:19.812 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:19.794+0000 7f286bfff700 1 -- 172.21.15.42:0/3250481110 --> v1:172.21.15.42:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f28741211d0 con 0x7f28740a3b70 2019-12-05T06:43:19.812 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:19.794+0000 7f286bfff700 1 -- 172.21.15.42:0/3250481110 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7f285c002500 con 0x7f28740a3b70 2019-12-05T06:43:19.813 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:19.794+0000 7f286bfff700 1 -- 172.21.15.42:0/3250481110 >> v2:172.21.15.40:3300/0 conn(0x7f28740a3740 msgr2=0x7f2874120c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:19.813 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:19.794+0000 7f286bfff700 1 --2- 172.21.15.42:0/3250481110 >> v2:172.21.15.40:3300/0 conn(0x7f28740a3740 0x7f2874120c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:19.813 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:19.794+0000 7f286bfff700 1 -- 172.21.15.42:0/3250481110 >> v1:172.21.15.42:6789/0 conn(0x7f28740a3b70 legacy=0x7f287412e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:19.813 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:19.794+0000 7f286bfff700 1 -- 172.21.15.42:0/3250481110 >> v1:172.21.15.40:6789/0 conn(0x7f287412e810 legacy=0x7f28740742c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:19.813 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:19.794+0000 7f286bfff700 1 --2- 172.21.15.42:0/3250481110 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2858003ab0 0x7f2858004420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:43:19.814 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:19.794+0000 7f286bfff700 1 --2- 172.21.15.42:0/3250481110 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2858004970 0x7f2858004da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:43:19.814 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:19.794+0000 7f286bfff700 1 --2- 172.21.15.42:0/3250481110 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2858009660 0x7f285800baa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:43:19.814 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:19.794+0000 7f286bfff700 1 -- 172.21.15.42:0/3250481110 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28741195e0 con 0x7f2858003ab0 2019-12-05T06:43:19.814 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:19.794+0000 7f286bfff700 1 -- 172.21.15.42:0/3250481110 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f285800bff0 con 0x7f2858004970 2019-12-05T06:43:19.814 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:19.794+0000 7f286bfff700 1 -- 172.21.15.42:0/3250481110 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f285800c1b0 con 0x7f2858009660 2019-12-05T06:43:19.815 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:19.794+0000 7f2872ffd700 1 --2- 172.21.15.42:0/3250481110 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2858004970 0x7f2858004da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:19.815 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:19.794+0000 7f28737fe700 1 --2- 172.21.15.42:0/3250481110 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2858003ab0 0x7f2858004420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:19.815 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:19.794+0000 7f2873fff700 1 --2- 172.21.15.42:0/3250481110 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2858009660 0x7f285800baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:19.815 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:19.794+0000 7f2872ffd700 1 -- 172.21.15.42:0/3250481110 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2858009660 msgr2=0x7f285800baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:19.815 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:19.794+0000 7f2872ffd700 1 --2- 172.21.15.42:0/3250481110 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2858009660 0x7f285800baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:19.815 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:19.794+0000 7f2872ffd700 1 -- 172.21.15.42:0/3250481110 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2858003ab0 msgr2=0x7f2858004420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:19.815 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:19.794+0000 7f2872ffd700 1 --2- 172.21.15.42:0/3250481110 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2858003ab0 0x7f2858004420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:19.816 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:19.794+0000 7f2872ffd700 1 -- 172.21.15.42:0/3250481110 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2874077d20 con 0x7f2858004970 2019-12-05T06:43:19.816 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:19.794+0000 7f28794ee700 1 -- 172.21.15.42:0/3250481110 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2874075ad0 con 0x7f2858004970 2019-12-05T06:43:19.816 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:19.794+0000 7f2872ffd700 1 --2- 172.21.15.42:0/3250481110 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2858004970 0x7f2858004da0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f285c0022d0 tx=0x7f285c011970).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:19.816 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:19.794+0000 7f2873fff700 1 --2- 172.21.15.42:0/3250481110 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2858009660 0x7f285800baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:43:19.816 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:19.794+0000 7f2873fff700 1 -- 172.21.15.42:0/3250481110 reap_dead start 2019-12-05T06:43:19.816 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:19.794+0000 7f28794ee700 1 -- 172.21.15.42:0/3250481110 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28740a3b70 con 0x7f2858004970 2019-12-05T06:43:19.816 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:19.794+0000 7f2872ffd700 1 --2- 172.21.15.42:0/3250481110 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2858004970 0x7f2858004da0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f285c0022d0 tx=0x7f285c011970).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:19.817 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:19.794+0000 7f2872ffd700 1 --2- 172.21.15.42:0/3250481110 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2858004970 0x7f2858004da0 secure :-1 s=READY pgs=3097 cs=0 l=1 rx=0x7f285c0022d0 tx=0x7f285c011970).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:43:19.817 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:19.794+0000 7f2872ffd700 1 --2- 172.21.15.42:0/3250481110 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2858004970 0x7f2858004da0 secure :-1 s=THROTTLE_DONE pgs=3097 cs=0 l=1 rx=0x7f285c0022d0 tx=0x7f285c011970).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:19.817 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:19.794+0000 7f2872ffd700 1 --2- 172.21.15.42:0/3250481110 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2858004970 0x7f2858004da0 secure :-1 s=THROTTLE_DONE pgs=3097 cs=0 l=1 rx=0x7f285c0022d0 tx=0x7f285c011970).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:19.817 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:19.794+0000 7f2872ffd700 1 --2- 172.21.15.42:0/3250481110 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2858004970 0x7f2858004da0 secure :-1 s=THROTTLE_DONE pgs=3097 cs=0 l=1 rx=0x7f285c0022d0 tx=0x7f285c011970).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:19.817 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:19.794+0000 7f286bfff700 1 -- 172.21.15.42:0/3250481110 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f285c005840 con 0x7f2858004970 2019-12-05T06:43:19.817 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:19.794+0000 7f286bfff700 1 -- 172.21.15.42:0/3250481110 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f285c007160 con 0x7f2858004970 2019-12-05T06:43:19.818 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:19.794+0000 7f286bfff700 1 -- 172.21.15.42:0/3250481110 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f285c00f2d0 con 0x7f2858004970 2019-12-05T06:43:19.818 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:19.794+0000 7f2872ffd700 1 --2- 172.21.15.42:0/3250481110 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2858004970 0x7f2858004da0 secure :-1 s=THROTTLE_DONE pgs=3097 cs=0 l=1 rx=0x7f285c0022d0 tx=0x7f285c011970).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:19.818 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:19.798+0000 7f286bfff700 1 -- 172.21.15.42:0/3250481110 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f285c00f4d0 con 0x7f2858004970 2019-12-05T06:43:19.819 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:19.798+0000 7f2872ffd700 1 --2- 172.21.15.42:0/3250481110 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2858004970 0x7f2858004da0 secure :-1 s=THROTTLE_DONE pgs=3097 cs=0 l=1 rx=0x7f285c0022d0 tx=0x7f285c011970).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:19.819 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:19.798+0000 7f286bfff700 1 --2- 172.21.15.42:0/3250481110 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f285803ec00 0x7f2858041110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:43:19.819 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:19.798+0000 7f286bfff700 1 -- 172.21.15.42:0/3250481110 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 11793+0+0 (secure 0 0 0) 0x7f285c04bf70 con 0x7f2858004970 2019-12-05T06:43:19.819 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:19.798+0000 7f28737fe700 1 --2- 172.21.15.42:0/3250481110 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f285803ec00 0x7f2858041110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:19.820 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:19.798+0000 7f28737fe700 1 --2- 172.21.15.42:0/3250481110 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f285803ec00 0x7f2858041110 crc :-1 s=READY pgs=1294 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:43:19.820 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:19.798+0000 7f28794ee700 1 -- 172.21.15.42:0/3250481110 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f284c003200 con 0x7f2858004970 2019-12-05T06:43:19.820 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:19.806+0000 7f2872ffd700 1 --2- 172.21.15.42:0/3250481110 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2858004970 0x7f2858004da0 secure :-1 s=THROTTLE_DONE pgs=3097 cs=0 l=1 rx=0x7f285c0022d0 tx=0x7f285c011970).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:19.821 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:19.806+0000 7f286bfff700 1 -- 172.21.15.42:0/3250481110 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f285c00f0b0 con 0x7f2858004970 2019-12-05T06:43:19.979 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:19.974+0000 7f28794ee700 1 -- 172.21.15.42:0/3250481110 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f284c003630 con 0x7f2858004970 2019-12-05T06:43:19.980 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:19.974+0000 7f2872ffd700 1 --2- 172.21.15.42:0/3250481110 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2858004970 0x7f2858004da0 secure :-1 s=THROTTLE_DONE pgs=3097 cs=0 l=1 rx=0x7f285c0022d0 tx=0x7f285c011970).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:19.981 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:19.978+0000 7f286bfff700 1 -- 172.21.15.42:0/3250481110 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 372 v372) v1 ==== 95+0+2108 (secure 0 0 0) 0x7f285c032fb0 con 0x7f2858004970 2019-12-05T06:43:19.981 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 372 2019-12-05T06:43:19.982 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:43:19.982 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":372,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":12496,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/3599619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3599619},{"type":"v1","addr":"172.21.15.40:6833","nonce":3599619}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":372}],"filesystems":[{"mdsmap":{"epoch":371,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:42:53.534430+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":116,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12358},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12358":{"gid":12358,"name":"a","rank":0,"incarnation":368,"state":"up:active","state_seq":34,"addr":"172.21.15.42:6835/1922398668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1922398668},{"type":"v1","addr":"172.21.15.42:6835","nonce":1922398668}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:43:19.983 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:19.978+0000 7f28794ee700 1 -- 172.21.15.42:0/3250481110 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f285803ec00 msgr2=0x7f2858041110 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:19.983 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:19.978+0000 7f28794ee700 1 --2- 172.21.15.42:0/3250481110 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f285803ec00 0x7f2858041110 crc :-1 s=READY pgs=1294 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:19.985 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:19.978+0000 7f28794ee700 1 -- 172.21.15.42:0/3250481110 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2858004970 msgr2=0x7f2858004da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:19.985 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:19.978+0000 7f28794ee700 1 --2- 172.21.15.42:0/3250481110 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2858004970 0x7f2858004da0 secure :-1 s=READY pgs=3097 cs=0 l=1 rx=0x7f285c0022d0 tx=0x7f285c011970).stop 2019-12-05T06:43:19.985 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:19.978+0000 7f28794ee700 1 -- 172.21.15.42:0/3250481110 shutdown_connections 2019-12-05T06:43:19.985 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:19.978+0000 7f28794ee700 1 --2- 172.21.15.42:0/3250481110 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f285803ec00 0x7f2858041110 unknown :-1 s=CLOSED pgs=1294 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:19.985 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:19.978+0000 7f28794ee700 1 --2- 172.21.15.42:0/3250481110 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2858004970 0x7f2858004da0 unknown :-1 s=CLOSED pgs=3097 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:19.986 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:19.978+0000 7f28794ee700 1 -- 172.21.15.42:0/3250481110 shutdown_connections 2019-12-05T06:43:19.986 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:19.978+0000 7f28794ee700 1 -- 172.21.15.42:0/3250481110 wait complete. 2019-12-05T06:43:19.986 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:19.978+0000 7f28794ee700 1 -- 172.21.15.42:0/3250481110 >> 172.21.15.42:0/3250481110 conn(0x7f2874067070 msgr2=0x7f28740a6b30 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:43:19.994 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:43:21.998 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:43:22.112 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:22.102+0000 7fed93bfe700 1 Processor -- start 2019-12-05T06:43:22.112 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:22.102+0000 7fed93bfe700 1 -- start start 2019-12-05T06:43:22.113 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:22.102+0000 7fed93bfe700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fed8c0a7740 0x7fed8c08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:43:22.113 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:22.102+0000 7fed93bfe700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fed8c09fae0 0x7fed8c07eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:43:22.113 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:22.102+0000 7fed93bfe700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed8c0a0710 con 0x7fed8c07e720 2019-12-05T06:43:22.113 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:22.102+0000 7fed93bfe700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed8c0a0b00 con 0x7fed8c0a7740 2019-12-05T06:43:22.114 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:22.102+0000 7fed93bfe700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed8c08e5b0 con 0x7fed8c09fae0 2019-12-05T06:43:22.114 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:22.102+0000 7fed9199a700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fed8c07e720 0x7fed8c09f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:34226/0 (socket says 172.21.15.42:34226) 2019-12-05T06:43:22.114 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:22.102+0000 7fed9199a700 1 -- 172.21.15.42:0/2052350525 learned_addr learned my addr 172.21.15.42:0/2052350525 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:43:22.118 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:22.102+0000 7fed9219b700 1 --2- 172.21.15.42:0/2052350525 >> v2:172.21.15.40:3300/0 conn(0x7fed8c0a7740 0x7fed8c08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:22.118 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:22.106+0000 7fed9299c700 1 --2- 172.21.15.42:0/2052350525 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fed8c09fae0 0x7fed8c07eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:22.118 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:22.106+0000 7fed91199700 1 -- 172.21.15.42:0/2052350525 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 964217299 0 0) 0x7fed8c0a0710 con 0x7fed8c07e720 2019-12-05T06:43:22.118 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:22.106+0000 7fed91199700 1 -- 172.21.15.42:0/2052350525 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed780030b0 con 0x7fed8c07e720 2019-12-05T06:43:22.118 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:22.106+0000 7fed9219b700 1 -- 172.21.15.42:0/2052350525 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fed8c09fae0 msgr2=0x7fed8c07eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:43:22.118 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:22.106+0000 7fed9219b700 1 --2- 172.21.15.42:0/2052350525 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fed8c09fae0 0x7fed8c07eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:43:22.119 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:22.106+0000 7fed9219b700 1 -- 172.21.15.42:0/2052350525 >> v1:172.21.15.42:6789/0 conn(0x7fed8c07e720 legacy=0x7fed8c09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:22.119 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:22.106+0000 7fed91199700 1 -- 172.21.15.42:0/2052350525 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 174731359 0 0) 0x7fed780030b0 con 0x7fed8c07e720 2019-12-05T06:43:22.119 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:22.106+0000 7fed9219b700 1 -- 172.21.15.42:0/2052350525 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed8c07f0a0 con 0x7fed8c0a7740 2019-12-05T06:43:22.119 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:22.106+0000 7fed9219b700 1 --2- 172.21.15.42:0/2052350525 >> v2:172.21.15.40:3300/0 conn(0x7fed8c0a7740 0x7fed8c08df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fed88008790 tx=0x7fed7c0019d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:22.119 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:22.106+0000 7fed9219b700 1 --2- 172.21.15.42:0/2052350525 >> v2:172.21.15.40:3300/0 conn(0x7fed8c0a7740 0x7fed8c08df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fed88008790 tx=0x7fed7c0019d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:22.120 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:22.106+0000 7fed9219b700 1 --2- 172.21.15.42:0/2052350525 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fed8c0a7740 0x7fed8c08df80 secure :-1 s=READY pgs=2338 cs=0 l=1 rx=0x7fed88008790 tx=0x7fed7c0019d0).ready entity=mon.0 client_cookie=6387c96ac153669f server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:43:22.120 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:22.106+0000 7fed9219b700 1 --2- 172.21.15.42:0/2052350525 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fed8c0a7740 0x7fed8c08df80 secure :-1 s=THROTTLE_DONE pgs=2338 cs=0 l=1 rx=0x7fed88008790 tx=0x7fed7c0019d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:22.120 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:22.106+0000 7fed91199700 1 -- 172.21.15.42:0/2052350525 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fed7c001750 con 0x7fed8c0a7740 2019-12-05T06:43:22.120 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:22.106+0000 7fed9219b700 1 --2- 172.21.15.42:0/2052350525 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fed8c0a7740 0x7fed8c08df80 secure :-1 s=THROTTLE_DONE pgs=2338 cs=0 l=1 rx=0x7fed88008790 tx=0x7fed7c0019d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:22.120 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:22.106+0000 7fed9219b700 1 --2- 172.21.15.42:0/2052350525 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fed8c0a7740 0x7fed8c08df80 secure :-1 s=THROTTLE_DONE pgs=2338 cs=0 l=1 rx=0x7fed88008790 tx=0x7fed7c0019d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:22.120 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:22.106+0000 7fed91199700 1 -- 172.21.15.42:0/2052350525 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed7c001560 con 0x7fed8c0a7740 2019-12-05T06:43:22.120 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:22.106+0000 7fed91199700 1 -- 172.21.15.42:0/2052350525 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fed88008da0 con 0x7fed8c0a7740 2019-12-05T06:43:22.121 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:22.106+0000 7fed93bfe700 1 -- 172.21.15.42:0/2052350525 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fed8c0a7740 msgr2=0x7fed8c08df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:22.121 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:22.106+0000 7fed93bfe700 1 --2- 172.21.15.42:0/2052350525 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fed8c0a7740 0x7fed8c08df80 secure :-1 s=READY pgs=2338 cs=0 l=1 rx=0x7fed88008790 tx=0x7fed7c0019d0).stop 2019-12-05T06:43:22.121 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:22.106+0000 7fed93bfe700 1 -- 172.21.15.42:0/2052350525 shutdown_connections 2019-12-05T06:43:22.121 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:22.106+0000 7fed93bfe700 1 --2- 172.21.15.42:0/2052350525 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fed8c09fae0 0x7fed8c07eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:43:22.121 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:22.106+0000 7fed93bfe700 1 --2- 172.21.15.42:0/2052350525 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fed8c0a7740 0x7fed8c08df80 unknown :-1 s=CLOSED pgs=2338 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:22.121 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:22.106+0000 7fed93bfe700 1 -- 172.21.15.42:0/2052350525 shutdown_connections 2019-12-05T06:43:22.121 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:22.106+0000 7fed93bfe700 1 -- 172.21.15.42:0/2052350525 wait complete. 2019-12-05T06:43:22.122 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:22.106+0000 7fed93bfe700 1 -- 172.21.15.42:0/2052350525 >> 172.21.15.42:0/2052350525 conn(0x7fed8c0aabe0 msgr2=0x7fed8c0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:43:22.122 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:22.110+0000 7fed93bfe700 1 Processor -- start 2019-12-05T06:43:22.122 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:22.110+0000 7fed93bfe700 1 -- start start 2019-12-05T06:43:22.124 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:22.110+0000 7fed93bfe700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fed8c11a2f0 0x7fed8c123040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:43:22.125 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:22.110+0000 7fed93bfe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fed8c0651b0 0x7fed8c0655e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:43:22.125 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:22.110+0000 7fed93bfe700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed8c121430 con 0x7fed8c123f00 2019-12-05T06:43:22.125 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:22.110+0000 7fed93bfe700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed8c1239c0 con 0x7fed8c11a2f0 2019-12-05T06:43:22.125 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:22.110+0000 7fed93bfe700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed8c123c60 con 0x7fed8c0651b0 2019-12-05T06:43:22.125 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:22.110+0000 7fed9299c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fed8c0651b0 0x7fed8c0655e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:22.125 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:22.110+0000 7fed9299c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fed8c0651b0 0x7fed8c0655e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:34194/0 (socket says 172.21.15.42:34194) 2019-12-05T06:43:22.125 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:22.110+0000 7fed9219b700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fed8c11a2f0 0x7fed8c123040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:22.126 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:22.110+0000 7fed9299c700 1 -- 172.21.15.42:0/929762996 learned_addr learned my addr 172.21.15.42:0/929762996 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:43:22.126 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:22.110+0000 7fed9199a700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fed8c123f00 0x7fed8c120c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:34232/0 (socket says 172.21.15.42:34232) 2019-12-05T06:43:22.126 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:22.110+0000 7fed9299c700 1 -- 172.21.15.42:0/929762996 >> v2:172.21.15.40:3300/0 conn(0x7fed8c11a2f0 msgr2=0x7fed8c123040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:22.126 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:22.110+0000 7fed9299c700 1 --2- 172.21.15.42:0/929762996 >> v2:172.21.15.40:3300/0 conn(0x7fed8c11a2f0 0x7fed8c123040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:22.126 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:22.110+0000 7fed9299c700 1 -- 172.21.15.42:0/929762996 >> v1:172.21.15.42:6789/0 conn(0x7fed8c123f00 legacy=0x7fed8c120c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:22.126 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:22.110+0000 7fed9299c700 1 -- 172.21.15.42:0/929762996 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed8c123700 con 0x7fed8c0651b0 2019-12-05T06:43:22.126 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:22.110+0000 7fed9299c700 1 --2- 172.21.15.42:0/929762996 >> v2:172.21.15.42:3300/0 conn(0x7fed8c0651b0 0x7fed8c0655e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fed840032b0 tx=0x7fed84010d00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:22.127 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:22.110+0000 7fed9299c700 1 --2- 172.21.15.42:0/929762996 >> v2:172.21.15.42:3300/0 conn(0x7fed8c0651b0 0x7fed8c0655e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fed840032b0 tx=0x7fed84010d00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:22.127 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:22.110+0000 7fed9299c700 1 --2- 172.21.15.42:0/929762996 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fed8c0651b0 0x7fed8c0655e0 secure :-1 s=READY pgs=3100 cs=0 l=1 rx=0x7fed840032b0 tx=0x7fed84010d00).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:43:22.127 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:22.110+0000 7fed9299c700 1 --2- 172.21.15.42:0/929762996 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fed8c0651b0 0x7fed8c0655e0 secure :-1 s=THROTTLE_DONE pgs=3100 cs=0 l=1 rx=0x7fed840032b0 tx=0x7fed84010d00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:22.127 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:22.110+0000 7fed82ffd700 1 -- 172.21.15.42:0/929762996 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fed84012a30 con 0x7fed8c0651b0 2019-12-05T06:43:22.127 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:22.110+0000 7fed9299c700 1 --2- 172.21.15.42:0/929762996 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fed8c0651b0 0x7fed8c0655e0 secure :-1 s=THROTTLE_DONE pgs=3100 cs=0 l=1 rx=0x7fed840032b0 tx=0x7fed84010d00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:22.127 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:22.110+0000 7fed9299c700 1 --2- 172.21.15.42:0/929762996 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fed8c0651b0 0x7fed8c0655e0 secure :-1 s=THROTTLE_DONE pgs=3100 cs=0 l=1 rx=0x7fed840032b0 tx=0x7fed84010d00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:22.127 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:22.110+0000 7fed82ffd700 1 -- 172.21.15.42:0/929762996 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed84012650 con 0x7fed8c0651b0 2019-12-05T06:43:22.128 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:22.110+0000 7fed93bfe700 1 -- 172.21.15.42:0/929762996 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed8c119d60 con 0x7fed8c0651b0 2019-12-05T06:43:22.128 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:22.110+0000 7fed82ffd700 1 -- 172.21.15.42:0/929762996 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fed8400e760 con 0x7fed8c0651b0 2019-12-05T06:43:22.128 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:22.110+0000 7fed93bfe700 1 -- 172.21.15.42:0/929762996 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed8c124330 con 0x7fed8c0651b0 2019-12-05T06:43:22.128 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:22.110+0000 7fed9299c700 1 --2- 172.21.15.42:0/929762996 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fed8c0651b0 0x7fed8c0655e0 secure :-1 s=THROTTLE_DONE pgs=3100 cs=0 l=1 rx=0x7fed840032b0 tx=0x7fed84010d00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:22.128 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:22.114+0000 7fed82ffd700 1 -- 172.21.15.42:0/929762996 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fed8400ea70 con 0x7fed8c0651b0 2019-12-05T06:43:22.128 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:22.114+0000 7fed9299c700 1 --2- 172.21.15.42:0/929762996 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fed8c0651b0 0x7fed8c0655e0 secure :-1 s=THROTTLE_DONE pgs=3100 cs=0 l=1 rx=0x7fed840032b0 tx=0x7fed84010d00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:22.129 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:22.114+0000 7fed82ffd700 1 --2- 172.21.15.42:0/929762996 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fed7403ad20 0x7fed7403d230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:43:22.129 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:22.114+0000 7fed82ffd700 1 -- 172.21.15.42:0/929762996 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 11793+0+0 (secure 0 0 0) 0x7fed8404b800 con 0x7fed8c0651b0 2019-12-05T06:43:22.129 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:22.114+0000 7fed9219b700 1 --2- 172.21.15.42:0/929762996 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fed7403ad20 0x7fed7403d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:22.129 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:22.114+0000 7fed9219b700 1 --2- 172.21.15.42:0/929762996 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fed7403ad20 0x7fed7403d230 crc :-1 s=READY pgs=1295 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:43:22.129 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:22.118+0000 7fed93bfe700 1 -- 172.21.15.42:0/929762996 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed8c05d110 con 0x7fed8c0651b0 2019-12-05T06:43:22.130 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:22.126+0000 7fed9299c700 1 --2- 172.21.15.42:0/929762996 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fed8c0651b0 0x7fed8c0655e0 secure :-1 s=THROTTLE_DONE pgs=3100 cs=0 l=1 rx=0x7fed840032b0 tx=0x7fed84010d00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:22.130 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:22.126+0000 7fed82ffd700 1 -- 172.21.15.42:0/929762996 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fed8c123700 con 0x7fed8c0651b0 2019-12-05T06:43:22.296 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:22.294+0000 7fed93bfe700 1 -- 172.21.15.42:0/929762996 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fed8c05b8c0 con 0x7fed8c0651b0 2019-12-05T06:43:22.297 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:22.294+0000 7fed9299c700 1 --2- 172.21.15.42:0/929762996 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fed8c0651b0 0x7fed8c0655e0 secure :-1 s=THROTTLE_DONE pgs=3100 cs=0 l=1 rx=0x7fed840032b0 tx=0x7fed84010d00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:22.297 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:22.294+0000 7fed82ffd700 1 -- 172.21.15.42:0/929762996 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 372 v372) v1 ==== 95+0+2108 (secure 0 0 0) 0x7fed84032820 con 0x7fed8c0651b0 2019-12-05T06:43:22.299 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:43:22.299 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":372,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":12496,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/3599619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3599619},{"type":"v1","addr":"172.21.15.40:6833","nonce":3599619}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":372}],"filesystems":[{"mdsmap":{"epoch":371,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:42:53.534430+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":116,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12358},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12358":{"gid":12358,"name":"a","rank":0,"incarnation":368,"state":"up:active","state_seq":34,"addr":"172.21.15.42:6835/1922398668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1922398668},{"type":"v1","addr":"172.21.15.42:6835","nonce":1922398668}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:43:22.300 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 372 2019-12-05T06:43:22.302 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:22.298+0000 7fed93bfe700 1 -- 172.21.15.42:0/929762996 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fed7403ad20 msgr2=0x7fed7403d230 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:22.302 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:22.298+0000 7fed93bfe700 1 --2- 172.21.15.42:0/929762996 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fed7403ad20 0x7fed7403d230 crc :-1 s=READY pgs=1295 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:22.302 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:22.298+0000 7fed93bfe700 1 -- 172.21.15.42:0/929762996 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fed8c0651b0 msgr2=0x7fed8c0655e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:22.303 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:22.298+0000 7fed93bfe700 1 --2- 172.21.15.42:0/929762996 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fed8c0651b0 0x7fed8c0655e0 secure :-1 s=READY pgs=3100 cs=0 l=1 rx=0x7fed840032b0 tx=0x7fed84010d00).stop 2019-12-05T06:43:22.303 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:22.298+0000 7fed93bfe700 1 -- 172.21.15.42:0/929762996 shutdown_connections 2019-12-05T06:43:22.303 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:22.298+0000 7fed93bfe700 1 --2- 172.21.15.42:0/929762996 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fed7403ad20 0x7fed7403d230 unknown :-1 s=CLOSED pgs=1295 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:22.303 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:22.298+0000 7fed93bfe700 1 --2- 172.21.15.42:0/929762996 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fed8c0651b0 0x7fed8c0655e0 unknown :-1 s=CLOSED pgs=3100 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:22.303 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:22.298+0000 7fed93bfe700 1 --2- 172.21.15.42:0/929762996 >> v2:172.21.15.40:3300/0 conn(0x7fed8c11a2f0 0x7fed8c123040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:22.303 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:22.298+0000 7fed93bfe700 1 -- 172.21.15.42:0/929762996 shutdown_connections 2019-12-05T06:43:22.303 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:22.298+0000 7fed93bfe700 1 -- 172.21.15.42:0/929762996 wait complete. 2019-12-05T06:43:22.304 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:22.298+0000 7fed93bfe700 1 -- 172.21.15.42:0/929762996 >> 172.21.15.42:0/929762996 conn(0x7fed8c0a9640 msgr2=0x7fed8c09f200 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:43:22.312 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:43:24.313 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:43:24.420 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:24.410+0000 7f2de04b0700 1 Processor -- start 2019-12-05T06:43:24.420 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:24.410+0000 7f2de04b0700 1 -- start start 2019-12-05T06:43:24.426 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:24.414+0000 7f2de04b0700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2dd80a7740 0x7f2dd808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:43:24.426 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:24.414+0000 7f2de04b0700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2dd807e720 0x7f2dd807eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:43:24.427 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:24.414+0000 7f2de04b0700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dd80a0710 con 0x7f2dd809f2e0 2019-12-05T06:43:24.427 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:24.414+0000 7f2de04b0700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2dd80a0b00 con 0x7f2dd80a7740 2019-12-05T06:43:24.427 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:24.414+0000 7f2de04b0700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2dd808e5b0 con 0x7f2dd807e720 2019-12-05T06:43:24.427 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:24.414+0000 7f2dde24c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2dd807e720 0x7f2dd807eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:24.427 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:24.414+0000 7f2ddf24e700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f2dd809f2e0 0x7f2dd809f710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:34242/0 (socket says 172.21.15.42:34242) 2019-12-05T06:43:24.427 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:24.414+0000 7f2ddf24e700 1 -- 172.21.15.42:0/3750391020 learned_addr learned my addr 172.21.15.42:0/3750391020 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:43:24.427 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:24.414+0000 7f2ddea4d700 1 --2- 172.21.15.42:0/3750391020 >> v2:172.21.15.40:3300/0 conn(0x7f2dd80a7740 0x7f2dd808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:24.428 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:24.414+0000 7f2dde24c700 1 -- 172.21.15.42:0/3750391020 >> v2:172.21.15.40:3300/0 conn(0x7f2dd80a7740 msgr2=0x7f2dd808df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:43:24.428 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:24.414+0000 7f2dde24c700 1 --2- 172.21.15.42:0/3750391020 >> v2:172.21.15.40:3300/0 conn(0x7f2dd80a7740 0x7f2dd808df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:43:24.428 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:24.414+0000 7f2dde24c700 1 -- 172.21.15.42:0/3750391020 >> v1:172.21.15.42:6789/0 conn(0x7f2dd809f2e0 legacy=0x7f2dd809f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:24.428 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:24.414+0000 7f2ddda4b700 1 -- 172.21.15.42:0/3750391020 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3935829347 0 0) 0x7f2dd80a0710 con 0x7f2dd809f2e0 2019-12-05T06:43:24.428 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:24.414+0000 7f2dde24c700 1 -- 172.21.15.42:0/3750391020 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dd807f0a0 con 0x7f2dd807e720 2019-12-05T06:43:24.428 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:24.414+0000 7f2dde24c700 1 --2- 172.21.15.42:0/3750391020 >> v2:172.21.15.42:3300/0 conn(0x7f2dd807e720 0x7f2dd807eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f2dcc0032b0 tx=0x7f2dcc00fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:24.428 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:24.414+0000 7f2dde24c700 1 --2- 172.21.15.42:0/3750391020 >> v2:172.21.15.42:3300/0 conn(0x7f2dd807e720 0x7f2dd807eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f2dcc0032b0 tx=0x7f2dcc00fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:24.429 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:24.414+0000 7f2dde24c700 1 --2- 172.21.15.42:0/3750391020 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2dd807e720 0x7f2dd807eb50 secure :-1 s=READY pgs=3102 cs=0 l=1 rx=0x7f2dcc0032b0 tx=0x7f2dcc00fff0).ready entity=mon.1 client_cookie=b193b6d1eb4418b server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:43:24.429 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:24.414+0000 7f2dde24c700 1 --2- 172.21.15.42:0/3750391020 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2dd807e720 0x7f2dd807eb50 secure :-1 s=THROTTLE_DONE pgs=3102 cs=0 l=1 rx=0x7f2dcc0032b0 tx=0x7f2dcc00fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:24.429 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:24.414+0000 7f2ddda4b700 1 -- 172.21.15.42:0/3750391020 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2dcc011a10 con 0x7f2dd807e720 2019-12-05T06:43:24.429 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:24.414+0000 7f2dde24c700 1 --2- 172.21.15.42:0/3750391020 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2dd807e720 0x7f2dd807eb50 secure :-1 s=THROTTLE_DONE pgs=3102 cs=0 l=1 rx=0x7f2dcc0032b0 tx=0x7f2dcc00fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:24.429 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:24.414+0000 7f2dde24c700 1 --2- 172.21.15.42:0/3750391020 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2dd807e720 0x7f2dd807eb50 secure :-1 s=THROTTLE_DONE pgs=3102 cs=0 l=1 rx=0x7f2dcc0032b0 tx=0x7f2dcc00fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:24.429 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:24.414+0000 7f2ddda4b700 1 -- 172.21.15.42:0/3750391020 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2dcc011630 con 0x7f2dd807e720 2019-12-05T06:43:24.429 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:24.414+0000 7f2ddda4b700 1 -- 172.21.15.42:0/3750391020 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2dcc00a160 con 0x7f2dd807e720 2019-12-05T06:43:24.430 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:24.414+0000 7f2de04b0700 1 -- 172.21.15.42:0/3750391020 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2dd807e720 msgr2=0x7f2dd807eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:24.430 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:24.414+0000 7f2de04b0700 1 --2- 172.21.15.42:0/3750391020 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2dd807e720 0x7f2dd807eb50 secure :-1 s=READY pgs=3102 cs=0 l=1 rx=0x7f2dcc0032b0 tx=0x7f2dcc00fff0).stop 2019-12-05T06:43:24.430 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:24.414+0000 7f2de04b0700 1 -- 172.21.15.42:0/3750391020 shutdown_connections 2019-12-05T06:43:24.430 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:24.414+0000 7f2de04b0700 1 --2- 172.21.15.42:0/3750391020 >> v2:172.21.15.40:3300/0 conn(0x7f2dd80a7740 0x7f2dd808df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:43:24.430 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:24.414+0000 7f2de04b0700 1 --2- 172.21.15.42:0/3750391020 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2dd807e720 0x7f2dd807eb50 unknown :-1 s=CLOSED pgs=3102 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:24.430 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:24.414+0000 7f2de04b0700 1 -- 172.21.15.42:0/3750391020 shutdown_connections 2019-12-05T06:43:24.430 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:24.414+0000 7f2de04b0700 1 -- 172.21.15.42:0/3750391020 wait complete. 2019-12-05T06:43:24.431 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:24.414+0000 7f2de04b0700 1 -- 172.21.15.42:0/3750391020 >> 172.21.15.42:0/3750391020 conn(0x7f2dd80aabe0 msgr2=0x7f2dd80a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:43:24.434 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:24.418+0000 7f2de04b0700 1 Processor -- start 2019-12-05T06:43:24.435 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:24.418+0000 7f2de04b0700 1 -- start start 2019-12-05T06:43:24.435 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:24.418+0000 7f2de04b0700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2dd808eb50 0x7f2dd8116da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:43:24.435 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:24.418+0000 7f2de04b0700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2dd8146050 0x7f2dd812fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:43:24.435 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:24.418+0000 7f2de04b0700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2dd8146480 0x7f2dd8125ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:43:24.435 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:24.418+0000 7f2de04b0700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2dd812f5a0 con 0x7f2dd8146480 2019-12-05T06:43:24.435 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:24.418+0000 7f2de04b0700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2dd812f840 con 0x7f2dd8146050 2019-12-05T06:43:24.436 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:24.418+0000 7f2de04b0700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2dd812fae0 con 0x7f2dd808eb50 2019-12-05T06:43:24.436 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:24.418+0000 7f2dde24c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2dd8146050 0x7f2dd812fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:24.436 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:24.418+0000 7f2dde24c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2dd8146050 0x7f2dd812fd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:34208/0 (socket says 172.21.15.42:34208) 2019-12-05T06:43:24.436 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:24.418+0000 7f2dde24c700 1 -- 172.21.15.42:0/1530727986 learned_addr learned my addr 172.21.15.42:0/1530727986 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:43:24.436 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:24.418+0000 7f2ddea4d700 1 --2- 172.21.15.42:0/1530727986 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2dd808eb50 0x7f2dd8116da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:24.436 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:24.418+0000 7f2ddf24e700 1 --2- 172.21.15.42:0/1530727986 >> v2:172.21.15.40:3300/0 conn(0x7f2dd8146480 0x7f2dd8125ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:24.436 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:24.418+0000 7f2dde24c700 1 -- 172.21.15.42:0/1530727986 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2dd808eb50 msgr2=0x7f2dd8116da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:24.437 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:24.418+0000 7f2dde24c700 1 --2- 172.21.15.42:0/1530727986 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2dd808eb50 0x7f2dd8116da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:24.437 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:24.418+0000 7f2dde24c700 1 -- 172.21.15.42:0/1530727986 >> v2:172.21.15.40:3300/0 conn(0x7f2dd8146480 msgr2=0x7f2dd8125ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:24.437 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:24.418+0000 7f2dde24c700 1 --2- 172.21.15.42:0/1530727986 >> v2:172.21.15.40:3300/0 conn(0x7f2dd8146480 0x7f2dd8125ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:24.437 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:24.418+0000 7f2dde24c700 1 -- 172.21.15.42:0/1530727986 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dd81302d0 con 0x7f2dd8146050 2019-12-05T06:43:24.437 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:24.418+0000 7f2dde24c700 1 --2- 172.21.15.42:0/1530727986 >> v2:172.21.15.42:3300/0 conn(0x7f2dd8146050 0x7f2dd812fd80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f2dd812f760 tx=0x7f2dcc011f80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:24.437 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:24.418+0000 7f2dde24c700 1 --2- 172.21.15.42:0/1530727986 >> v2:172.21.15.42:3300/0 conn(0x7f2dd8146050 0x7f2dd812fd80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f2dd812f760 tx=0x7f2dcc011f80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:24.437 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:24.418+0000 7f2dde24c700 1 --2- 172.21.15.42:0/1530727986 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2dd8146050 0x7f2dd812fd80 secure :-1 s=READY pgs=3103 cs=0 l=1 rx=0x7f2dd812f760 tx=0x7f2dcc011f80).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:43:24.438 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:24.418+0000 7f2dde24c700 1 --2- 172.21.15.42:0/1530727986 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2dd8146050 0x7f2dd812fd80 secure :-1 s=THROTTLE_DONE pgs=3103 cs=0 l=1 rx=0x7f2dd812f760 tx=0x7f2dcc011f80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:24.438 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:24.418+0000 7f2dde24c700 1 --2- 172.21.15.42:0/1530727986 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2dd8146050 0x7f2dd812fd80 secure :-1 s=THROTTLE_DONE pgs=3103 cs=0 l=1 rx=0x7f2dd812f760 tx=0x7f2dcc011f80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:24.438 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:24.418+0000 7f2dcb7fe700 1 -- 172.21.15.42:0/1530727986 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2dcc00f800 con 0x7f2dd8146050 2019-12-05T06:43:24.438 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:24.418+0000 7f2dde24c700 1 --2- 172.21.15.42:0/1530727986 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2dd8146050 0x7f2dd812fd80 secure :-1 s=THROTTLE_DONE pgs=3103 cs=0 l=1 rx=0x7f2dd812f760 tx=0x7f2dcc011f80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:24.438 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:24.418+0000 7f2dcb7fe700 1 -- 172.21.15.42:0/1530727986 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2dcc011c10 con 0x7f2dd8146050 2019-12-05T06:43:24.438 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:24.418+0000 7f2de04b0700 1 -- 172.21.15.42:0/1530727986 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2dd8126230 con 0x7f2dd8146050 2019-12-05T06:43:24.438 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:24.418+0000 7f2dcb7fe700 1 -- 172.21.15.42:0/1530727986 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2dcc005ca0 con 0x7f2dd8146050 2019-12-05T06:43:24.439 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:24.418+0000 7f2de04b0700 1 -- 172.21.15.42:0/1530727986 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2dd80a59c0 con 0x7f2dd8146050 2019-12-05T06:43:24.439 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:24.422+0000 7f2dde24c700 1 --2- 172.21.15.42:0/1530727986 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2dd8146050 0x7f2dd812fd80 secure :-1 s=THROTTLE_DONE pgs=3103 cs=0 l=1 rx=0x7f2dd812f760 tx=0x7f2dcc011f80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:24.439 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:24.422+0000 7f2dcb7fe700 1 -- 172.21.15.42:0/1530727986 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f2dcc007150 con 0x7f2dd8146050 2019-12-05T06:43:24.439 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:24.422+0000 7f2dde24c700 1 --2- 172.21.15.42:0/1530727986 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2dd8146050 0x7f2dd812fd80 secure :-1 s=THROTTLE_DONE pgs=3103 cs=0 l=1 rx=0x7f2dd812f760 tx=0x7f2dcc011f80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:24.439 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:24.422+0000 7f2dcb7fe700 1 --2- 172.21.15.42:0/1530727986 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2dc003ae60 0x7f2dc003d370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:43:24.439 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:24.422+0000 7f2dcb7fe700 1 -- 172.21.15.42:0/1530727986 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 11793+0+0 (secure 0 0 0) 0x7f2dcc04b830 con 0x7f2dd8146050 2019-12-05T06:43:24.439 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:24.422+0000 7f2ddea4d700 1 --2- 172.21.15.42:0/1530727986 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2dc003ae60 0x7f2dc003d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:24.439 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:24.422+0000 7f2ddea4d700 1 -- 172.21.15.42:0/1530727986 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2dc003ae60 msgr2=0x7f2dc003d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2019-12-05T06:43:24.440 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:24.422+0000 7f2ddea4d700 1 -- 172.21.15.42:0/1530727986 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2dc003ae60 msgr2=0x7f2dc003d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2019-12-05T06:43:24.440 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:24.422+0000 7f2ddea4d700 1 --2- 172.21.15.42:0/1530727986 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2dc003ae60 0x7f2dc003d370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-12-05T06:43:24.440 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:24.422+0000 7f2ddea4d700 1 --2- 172.21.15.42:0/1530727986 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2dc003ae60 0x7f2dc003d370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rx=0 tx=0)._fault waiting 0.200000 2019-12-05T06:43:24.440 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:24.426+0000 7f2de04b0700 1 -- 172.21.15.42:0/1530727986 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2dd805d110 con 0x7f2dd8146050 2019-12-05T06:43:24.440 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:24.434+0000 7f2dde24c700 1 --2- 172.21.15.42:0/1530727986 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2dd8146050 0x7f2dd812fd80 secure :-1 s=THROTTLE_DONE pgs=3103 cs=0 l=1 rx=0x7f2dd812f760 tx=0x7f2dcc011f80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:24.441 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:24.434+0000 7f2dcb7fe700 1 -- 172.21.15.42:0/1530727986 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f2dcc032870 con 0x7f2dd8146050 2019-12-05T06:43:24.610 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:24.606+0000 7f2de04b0700 1 -- 172.21.15.42:0/1530727986 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2dd805b8c0 con 0x7f2dd8146050 2019-12-05T06:43:24.611 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:24.606+0000 7f2dde24c700 1 --2- 172.21.15.42:0/1530727986 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2dd8146050 0x7f2dd812fd80 secure :-1 s=THROTTLE_DONE pgs=3103 cs=0 l=1 rx=0x7f2dd812f760 tx=0x7f2dcc011f80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:24.611 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:24.606+0000 7f2dcb7fe700 1 -- 172.21.15.42:0/1530727986 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 372 v372) v1 ==== 95+0+2108 (secure 0 0 0) 0x7f2dcc032ad0 con 0x7f2dd8146050 2019-12-05T06:43:24.611 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 372 2019-12-05T06:43:24.611 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:43:24.611 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":372,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":12496,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/3599619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3599619},{"type":"v1","addr":"172.21.15.40:6833","nonce":3599619}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":372}],"filesystems":[{"mdsmap":{"epoch":371,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:42:53.534430+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":116,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12358},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12358":{"gid":12358,"name":"a","rank":0,"incarnation":368,"state":"up:active","state_seq":34,"addr":"172.21.15.42:6835/1922398668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1922398668},{"type":"v1","addr":"172.21.15.42:6835","nonce":1922398668}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:43:24.613 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:24.606+0000 7f2de04b0700 1 -- 172.21.15.42:0/1530727986 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2dc003ae60 msgr2=0x7f2dc003d370 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2019-12-05T06:43:24.613 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:24.606+0000 7f2de04b0700 1 --2- 172.21.15.42:0/1530727986 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2dc003ae60 0x7f2dc003d370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:24.613 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:24.606+0000 7f2de04b0700 1 -- 172.21.15.42:0/1530727986 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2dd8146050 msgr2=0x7f2dd812fd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:24.613 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:24.606+0000 7f2de04b0700 1 --2- 172.21.15.42:0/1530727986 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2dd8146050 0x7f2dd812fd80 secure :-1 s=READY pgs=3103 cs=0 l=1 rx=0x7f2dd812f760 tx=0x7f2dcc011f80).stop 2019-12-05T06:43:24.613 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:24.606+0000 7f2de04b0700 1 -- 172.21.15.42:0/1530727986 shutdown_connections 2019-12-05T06:43:24.613 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:24.606+0000 7f2de04b0700 1 --2- 172.21.15.42:0/1530727986 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2dc003ae60 0x7f2dc003d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:24.614 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:24.606+0000 7f2de04b0700 1 --2- 172.21.15.42:0/1530727986 >> v2:172.21.15.40:3300/0 conn(0x7f2dd8146480 0x7f2dd8125ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:24.614 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:24.606+0000 7f2de04b0700 1 --2- 172.21.15.42:0/1530727986 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2dd808eb50 0x7f2dd8116da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:24.614 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:24.606+0000 7f2de04b0700 1 --2- 172.21.15.42:0/1530727986 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2dd8146050 0x7f2dd812fd80 unknown :-1 s=CLOSED pgs=3103 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:24.614 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:24.606+0000 7f2de04b0700 1 -- 172.21.15.42:0/1530727986 shutdown_connections 2019-12-05T06:43:24.614 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:24.606+0000 7f2de04b0700 1 -- 172.21.15.42:0/1530727986 wait complete. 2019-12-05T06:43:24.614 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:24.606+0000 7f2de04b0700 1 -- 172.21.15.42:0/1530727986 >> 172.21.15.42:0/1530727986 conn(0x7f2dd807ddc0 msgr2=0x7f2dd80a7f90 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:43:24.622 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:43:26.623 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:43:26.732 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:26.718+0000 7f977ad0c700 1 Processor -- start 2019-12-05T06:43:26.733 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:26.718+0000 7f977ad0c700 1 -- start start 2019-12-05T06:43:26.733 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:26.718+0000 7f977ad0c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f977408df80 0x7f977409f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:43:26.733 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:26.718+0000 7f977ad0c700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f977409f830 0x7f977414efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:43:26.733 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:26.718+0000 7f977ad0c700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f977409fc60 con 0x7f97740a7740 2019-12-05T06:43:26.734 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:26.718+0000 7f977ad0c700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97740a0b00 con 0x7f977408df80 2019-12-05T06:43:26.734 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:26.718+0000 7f977ad0c700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f977408e5b0 con 0x7f977409f830 2019-12-05T06:43:26.734 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:26.718+0000 7f97792a9700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f97740a7740 0x7f977407e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:34254/0 (socket says 172.21.15.42:34254) 2019-12-05T06:43:26.734 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:26.718+0000 7f97792a9700 1 -- 172.21.15.42:0/1786933330 learned_addr learned my addr 172.21.15.42:0/1786933330 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:43:26.735 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:26.718+0000 7f9778aa8700 1 --2- 172.21.15.42:0/1786933330 >> v2:172.21.15.42:3300/0 conn(0x7f977408df80 0x7f977409f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:26.735 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:26.718+0000 7f9779aaa700 1 --2- 172.21.15.42:0/1786933330 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f977409f830 0x7f977414efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:26.735 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:26.718+0000 7f9767fff700 1 -- 172.21.15.42:0/1786933330 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 308767908 0 0) 0x7f977409fc60 con 0x7f97740a7740 2019-12-05T06:43:26.735 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:26.718+0000 7f9767fff700 1 -- 172.21.15.42:0/1786933330 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97600030b0 con 0x7f97740a7740 2019-12-05T06:43:26.736 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:26.718+0000 7f9778aa8700 1 -- 172.21.15.42:0/1786933330 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f977409f830 msgr2=0x7f977414efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:43:26.736 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:26.718+0000 7f9767fff700 1 -- 172.21.15.42:0/1786933330 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2883622034 0 0) 0x7f97600030b0 con 0x7f97740a7740 2019-12-05T06:43:26.737 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:26.718+0000 7f9778aa8700 1 --2- 172.21.15.42:0/1786933330 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f977409f830 0x7f977414efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:43:26.738 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:26.718+0000 7f9778aa8700 1 -- 172.21.15.42:0/1786933330 >> v1:172.21.15.42:6789/0 conn(0x7f97740a7740 legacy=0x7f977407e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:26.738 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:26.718+0000 7f9778aa8700 1 -- 172.21.15.42:0/1786933330 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97740a0710 con 0x7f977408df80 2019-12-05T06:43:26.739 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:26.718+0000 7f9779aaa700 1 --2- 172.21.15.42:0/1786933330 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f977409f830 0x7f977414efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:43:26.739 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:26.718+0000 7f9778aa8700 1 --2- 172.21.15.42:0/1786933330 >> v2:172.21.15.42:3300/0 conn(0x7f977408df80 0x7f977409f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f9770008790 tx=0x7f976c001d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:26.739 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:26.718+0000 7f9778aa8700 1 --2- 172.21.15.42:0/1786933330 >> v2:172.21.15.42:3300/0 conn(0x7f977408df80 0x7f977409f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f9770008790 tx=0x7f976c001d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:26.739 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:26.718+0000 7f9778aa8700 1 --2- 172.21.15.42:0/1786933330 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f977408df80 0x7f977409f2e0 secure :-1 s=READY pgs=3105 cs=0 l=1 rx=0x7f9770008790 tx=0x7f976c001d50).ready entity=mon.1 client_cookie=3062ed511c09c158 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:43:26.740 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:26.718+0000 7f9778aa8700 1 --2- 172.21.15.42:0/1786933330 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f977408df80 0x7f977409f2e0 secure :-1 s=THROTTLE_DONE pgs=3105 cs=0 l=1 rx=0x7f9770008790 tx=0x7f976c001d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:26.740 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:26.718+0000 7f9767fff700 1 -- 172.21.15.42:0/1786933330 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f976c001ad0 con 0x7f977408df80 2019-12-05T06:43:26.740 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:26.718+0000 7f9778aa8700 1 --2- 172.21.15.42:0/1786933330 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f977408df80 0x7f977409f2e0 secure :-1 s=THROTTLE_DONE pgs=3105 cs=0 l=1 rx=0x7f9770008790 tx=0x7f976c001d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:26.740 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:26.718+0000 7f9778aa8700 1 --2- 172.21.15.42:0/1786933330 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f977408df80 0x7f977409f2e0 secure :-1 s=THROTTLE_DONE pgs=3105 cs=0 l=1 rx=0x7f9770008790 tx=0x7f976c001d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:26.741 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:26.718+0000 7f9767fff700 1 -- 172.21.15.42:0/1786933330 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f976c0018e0 con 0x7f977408df80 2019-12-05T06:43:26.741 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:26.718+0000 7f9767fff700 1 -- 172.21.15.42:0/1786933330 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9770008da0 con 0x7f977408df80 2019-12-05T06:43:26.741 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:26.718+0000 7f977ad0c700 1 -- 172.21.15.42:0/1786933330 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f977408df80 msgr2=0x7f977409f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:26.741 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:26.718+0000 7f977ad0c700 1 --2- 172.21.15.42:0/1786933330 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f977408df80 0x7f977409f2e0 secure :-1 s=READY pgs=3105 cs=0 l=1 rx=0x7f9770008790 tx=0x7f976c001d50).stop 2019-12-05T06:43:26.742 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:26.718+0000 7f977ad0c700 1 -- 172.21.15.42:0/1786933330 shutdown_connections 2019-12-05T06:43:26.742 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:26.718+0000 7f977ad0c700 1 --2- 172.21.15.42:0/1786933330 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f977409f830 0x7f977414efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:43:26.742 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:26.718+0000 7f977ad0c700 1 --2- 172.21.15.42:0/1786933330 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f977408df80 0x7f977409f2e0 unknown :-1 s=CLOSED pgs=3105 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:26.742 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:26.718+0000 7f977ad0c700 1 -- 172.21.15.42:0/1786933330 shutdown_connections 2019-12-05T06:43:26.743 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:26.718+0000 7f977ad0c700 1 -- 172.21.15.42:0/1786933330 wait complete. 2019-12-05T06:43:26.743 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:26.718+0000 7f977ad0c700 1 -- 172.21.15.42:0/1786933330 >> 172.21.15.42:0/1786933330 conn(0x7f97740aabe0 msgr2=0x7f97740a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:43:26.743 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:26.722+0000 7f977ad0c700 1 Processor -- start 2019-12-05T06:43:26.743 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:26.722+0000 7f977ad0c700 1 -- start start 2019-12-05T06:43:26.744 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:26.722+0000 7f977ad0c700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f97740a4f20 0x7f9774121580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:43:26.744 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:26.722+0000 7f977ad0c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f9774139930 0x7f977413bd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:43:26.744 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:26.722+0000 7f977ad0c700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97741314c0 con 0x7f9774123740 2019-12-05T06:43:26.744 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:26.722+0000 7f977ad0c700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97740a54a0 con 0x7f9774139930 2019-12-05T06:43:26.745 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:26.722+0000 7f977ad0c700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97740a5740 con 0x7f97740a4f20 2019-12-05T06:43:26.745 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:26.722+0000 7f9779aaa700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f9774139930 0x7f977413bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:26.745 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:26.722+0000 7f97792a9700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f97740a4f20 0x7f9774121580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:26.746 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:26.722+0000 7f9779aaa700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f9774139930 0x7f977413bd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:34226/0 (socket says 172.21.15.42:34226) 2019-12-05T06:43:26.746 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:26.722+0000 7f97792a9700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f97740a4f20 0x7f9774121580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:34688/0 (socket says 172.21.15.42:34688) 2019-12-05T06:43:26.746 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:26.722+0000 7f9779aaa700 1 -- 172.21.15.42:0/2098562089 learned_addr learned my addr 172.21.15.42:0/2098562089 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:43:26.747 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:26.722+0000 7f9779aaa700 1 -- 172.21.15.42:0/2098562089 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f97740a4f20 msgr2=0x7f9774121580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:26.747 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:26.722+0000 7f9779aaa700 1 --2- 172.21.15.42:0/2098562089 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f97740a4f20 0x7f9774121580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:26.747 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:26.722+0000 7f9779aaa700 1 -- 172.21.15.42:0/2098562089 >> v1:172.21.15.42:6789/0 conn(0x7f9774123740 legacy=0x7f977411b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:26.747 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:26.722+0000 7f9779aaa700 1 -- 172.21.15.42:0/2098562089 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f977411b900 con 0x7f9774139930 2019-12-05T06:43:26.749 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:26.726+0000 7f9779aaa700 1 --2- 172.21.15.42:0/2098562089 >> v2:172.21.15.42:3300/0 conn(0x7f9774139930 0x7f977413bd70 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f977411bca0 tx=0x7f9768010b80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:26.749 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:26.726+0000 7f9779aaa700 1 --2- 172.21.15.42:0/2098562089 >> v2:172.21.15.42:3300/0 conn(0x7f9774139930 0x7f977413bd70 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f977411bca0 tx=0x7f9768010b80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:26.749 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:26.726+0000 7f9779aaa700 1 --2- 172.21.15.42:0/2098562089 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9774139930 0x7f977413bd70 secure :-1 s=READY pgs=3107 cs=0 l=1 rx=0x7f977411bca0 tx=0x7f9768010b80).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:43:26.749 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:26.726+0000 7f9779aaa700 1 --2- 172.21.15.42:0/2098562089 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9774139930 0x7f977413bd70 secure :-1 s=THROTTLE_DONE pgs=3107 cs=0 l=1 rx=0x7f977411bca0 tx=0x7f9768010b80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:26.750 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:26.726+0000 7f9765ffb700 1 -- 172.21.15.42:0/2098562089 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f976800d0c0 con 0x7f9774139930 2019-12-05T06:43:26.750 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:26.726+0000 7f9779aaa700 1 --2- 172.21.15.42:0/2098562089 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9774139930 0x7f977413bd70 secure :-1 s=THROTTLE_DONE pgs=3107 cs=0 l=1 rx=0x7f977411bca0 tx=0x7f9768010b80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:26.750 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:26.726+0000 7f9765ffb700 1 -- 172.21.15.42:0/2098562089 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9768013060 con 0x7f9774139930 2019-12-05T06:43:26.750 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:26.726+0000 7f9779aaa700 1 --2- 172.21.15.42:0/2098562089 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9774139930 0x7f977413bd70 secure :-1 s=THROTTLE_DONE pgs=3107 cs=0 l=1 rx=0x7f977411bca0 tx=0x7f9768010b80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:26.751 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:26.726+0000 7f977ad0c700 1 -- 172.21.15.42:0/2098562089 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9774123b70 con 0x7f9774139930 2019-12-05T06:43:26.751 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:26.726+0000 7f9765ffb700 1 -- 172.21.15.42:0/2098562089 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9768007340 con 0x7f9774139930 2019-12-05T06:43:26.751 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:26.726+0000 7f977ad0c700 1 -- 172.21.15.42:0/2098562089 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9774132190 con 0x7f9774139930 2019-12-05T06:43:26.751 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:26.726+0000 7f9779aaa700 1 --2- 172.21.15.42:0/2098562089 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9774139930 0x7f977413bd70 secure :-1 s=THROTTLE_DONE pgs=3107 cs=0 l=1 rx=0x7f977411bca0 tx=0x7f9768010b80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:26.751 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:26.726+0000 7f9765ffb700 1 -- 172.21.15.42:0/2098562089 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f9768005a80 con 0x7f9774139930 2019-12-05T06:43:26.752 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:26.726+0000 7f9779aaa700 1 --2- 172.21.15.42:0/2098562089 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9774139930 0x7f977413bd70 secure :-1 s=THROTTLE_DONE pgs=3107 cs=0 l=1 rx=0x7f977411bca0 tx=0x7f9768010b80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:26.752 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:26.726+0000 7f9765ffb700 1 --2- 172.21.15.42:0/2098562089 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f975403ad60 0x7f975403d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:43:26.752 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:26.726+0000 7f9765ffb700 1 -- 172.21.15.42:0/2098562089 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 11879+0+0 (secure 0 0 0) 0x7f976804afe0 con 0x7f9774139930 2019-12-05T06:43:26.752 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:26.726+0000 7f97792a9700 1 --2- 172.21.15.42:0/2098562089 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f975403ad60 0x7f975403d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:26.753 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:26.726+0000 7f97792a9700 1 --2- 172.21.15.42:0/2098562089 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f975403ad60 0x7f975403d270 crc :-1 s=READY pgs=1296 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:43:26.754 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:26.734+0000 7f977ad0c700 1 -- 172.21.15.42:0/2098562089 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f977405d110 con 0x7f9774139930 2019-12-05T06:43:26.754 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:26.738+0000 7f9779aaa700 1 --2- 172.21.15.42:0/2098562089 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9774139930 0x7f977413bd70 secure :-1 s=THROTTLE_DONE pgs=3107 cs=0 l=1 rx=0x7f977411bca0 tx=0x7f9768010b80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:26.754 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:26.738+0000 7f9765ffb700 1 -- 172.21.15.42:0/2098562089 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f9768031fe0 con 0x7f9774139930 2019-12-05T06:43:26.909 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:26.906+0000 7f977ad0c700 1 -- 172.21.15.42:0/2098562089 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f977405b8c0 con 0x7f9774139930 2019-12-05T06:43:26.910 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:26.906+0000 7f9779aaa700 1 --2- 172.21.15.42:0/2098562089 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9774139930 0x7f977413bd70 secure :-1 s=THROTTLE_DONE pgs=3107 cs=0 l=1 rx=0x7f977411bca0 tx=0x7f9768010b80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:26.910 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:26.906+0000 7f9765ffb700 1 -- 172.21.15.42:0/2098562089 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 373 v373) v1 ==== 95+0+1769 (secure 0 0 0) 0x7f9768005d90 con 0x7f9774139930 2019-12-05T06:43:26.910 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 373 2019-12-05T06:43:26.910 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:43:26.910 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":373,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":373,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:43:25.248554+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":117,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12496},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12496":{"gid":12496,"name":"b","rank":0,"incarnation":373,"state":"up:replay","state_seq":2,"addr":"172.21.15.40:6833/3599619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3599619},{"type":"v1","addr":"172.21.15.40:6833","nonce":3599619}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:43:26.912 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:26.906+0000 7f977ad0c700 1 -- 172.21.15.42:0/2098562089 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f975403ad60 msgr2=0x7f975403d270 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:26.913 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:26.906+0000 7f977ad0c700 1 --2- 172.21.15.42:0/2098562089 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f975403ad60 0x7f975403d270 crc :-1 s=READY pgs=1296 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:26.913 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:26.906+0000 7f977ad0c700 1 -- 172.21.15.42:0/2098562089 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9774139930 msgr2=0x7f977413bd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:26.913 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:26.906+0000 7f977ad0c700 1 --2- 172.21.15.42:0/2098562089 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9774139930 0x7f977413bd70 secure :-1 s=READY pgs=3107 cs=0 l=1 rx=0x7f977411bca0 tx=0x7f9768010b80).stop 2019-12-05T06:43:26.913 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:26.910+0000 7f977ad0c700 1 -- 172.21.15.42:0/2098562089 shutdown_connections 2019-12-05T06:43:26.913 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:26.910+0000 7f977ad0c700 1 --2- 172.21.15.42:0/2098562089 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f975403ad60 0x7f975403d270 unknown :-1 s=CLOSED pgs=1296 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:26.913 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:26.910+0000 7f977ad0c700 1 --2- 172.21.15.42:0/2098562089 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9774139930 0x7f977413bd70 unknown :-1 s=CLOSED pgs=3107 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:26.914 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:26.910+0000 7f977ad0c700 1 --2- 172.21.15.42:0/2098562089 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f97740a4f20 0x7f9774121580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:26.914 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:26.910+0000 7f977ad0c700 1 -- 172.21.15.42:0/2098562089 shutdown_connections 2019-12-05T06:43:26.914 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:26.910+0000 7f977ad0c700 1 -- 172.21.15.42:0/2098562089 wait complete. 2019-12-05T06:43:26.914 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:26.910+0000 7f977ad0c700 1 -- 172.21.15.42:0/2098562089 >> 172.21.15.42:0/2098562089 conn(0x7f977411c610 msgr2=0x7f977411da30 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:43:26.933 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a down, removed from mdsmap 2019-12-05T06:43:26.933 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2019-12-05T06:43:26.933 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:43:27.053 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:27.034+0000 7f994e44b700 1 Processor -- start 2019-12-05T06:43:27.053 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:27.034+0000 7f994e44b700 1 -- start start 2019-12-05T06:43:27.053 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:27.034+0000 7f994e44b700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f994808df80 0x7f994809f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:43:27.053 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:27.034+0000 7f994e44b700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f994809fc60 con 0x7f99480a7740 2019-12-05T06:43:27.053 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:27.034+0000 7f994e44b700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99480a0710 con 0x7f994809f830 2019-12-05T06:43:27.053 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:27.034+0000 7f994e44b700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99480a0b00 con 0x7f994808df80 2019-12-05T06:43:27.054 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:27.034+0000 7f994d1e9700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f994809f830 0x7f994814efe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:34272/0 (socket says 172.21.15.42:34272) 2019-12-05T06:43:27.054 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:27.034+0000 7f994d1e9700 1 -- 172.21.15.42:0/2575173278 learned_addr learned my addr 172.21.15.42:0/2575173278 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:43:27.054 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:27.034+0000 7f9947fff700 1 --2- 172.21.15.42:0/2575173278 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f994808df80 0x7f994809f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:27.054 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:27.038+0000 7f99477fe700 1 -- 172.21.15.42:0/2575173278 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2203756602 0 0) 0x7f99480a0710 con 0x7f994809f830 2019-12-05T06:43:27.054 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:27.038+0000 7f99477fe700 1 -- 172.21.15.42:0/2575173278 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99300030b0 con 0x7f994809f830 2019-12-05T06:43:27.054 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:27.038+0000 7f99477fe700 1 -- 172.21.15.42:0/2575173278 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2386961860 0 0) 0x7f994809fc60 con 0x7f99480a7740 2019-12-05T06:43:27.054 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:27.038+0000 7f99477fe700 1 -- 172.21.15.42:0/2575173278 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99480a0710 con 0x7f99480a7740 2019-12-05T06:43:27.055 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:27.038+0000 7f9947fff700 1 -- 172.21.15.42:0/2575173278 >> v1:172.21.15.42:6789/0 conn(0x7f994809f830 legacy=0x7f994814efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:27.055 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:27.038+0000 7f9947fff700 1 -- 172.21.15.42:0/2575173278 >> v1:172.21.15.40:6789/0 conn(0x7f99480a7740 legacy=0x7f994807e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:27.055 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:27.038+0000 7f9947fff700 1 -- 172.21.15.42:0/2575173278 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f994807ef20 con 0x7f994808df80 2019-12-05T06:43:27.055 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:27.038+0000 7f9947fff700 1 --2- 172.21.15.42:0/2575173278 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f994808df80 0x7f994809f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f993c002f30 tx=0x7f993c00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:27.055 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:27.038+0000 7f9947fff700 1 --2- 172.21.15.42:0/2575173278 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f994808df80 0x7f994809f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f993c002f30 tx=0x7f993c00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:27.056 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:27.038+0000 7f9947fff700 1 --2- 172.21.15.42:0/2575173278 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f994808df80 0x7f994809f2e0 secure :-1 s=READY pgs=675 cs=0 l=1 rx=0x7f993c002f30 tx=0x7f993c00fbf0).ready entity=mon.2 client_cookie=629470e674b001d8 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:43:27.056 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:27.038+0000 7f9947fff700 1 --2- 172.21.15.42:0/2575173278 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f994808df80 0x7f994809f2e0 secure :-1 s=THROTTLE_DONE pgs=675 cs=0 l=1 rx=0x7f993c002f30 tx=0x7f993c00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:27.057 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:27.038+0000 7f99477fe700 1 -- 172.21.15.42:0/2575173278 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f993c0112b0 con 0x7f994808df80 2019-12-05T06:43:27.057 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:27.038+0000 7f9947fff700 1 --2- 172.21.15.42:0/2575173278 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f994808df80 0x7f994809f2e0 secure :-1 s=THROTTLE_DONE pgs=675 cs=0 l=1 rx=0x7f993c002f30 tx=0x7f993c00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:27.057 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:27.038+0000 7f99477fe700 1 -- 172.21.15.42:0/2575173278 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f993c008bc0 con 0x7f994808df80 2019-12-05T06:43:27.057 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:27.038+0000 7f9947fff700 1 --2- 172.21.15.42:0/2575173278 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f994808df80 0x7f994809f2e0 secure :-1 s=THROTTLE_DONE pgs=675 cs=0 l=1 rx=0x7f993c002f30 tx=0x7f993c00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:27.057 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:27.038+0000 7f99477fe700 1 -- 172.21.15.42:0/2575173278 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f993c011890 con 0x7f994808df80 2019-12-05T06:43:27.057 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:27.038+0000 7f994e44b700 1 -- 172.21.15.42:0/2575173278 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f994808df80 msgr2=0x7f994809f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:27.058 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:27.038+0000 7f994e44b700 1 --2- 172.21.15.42:0/2575173278 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f994808df80 0x7f994809f2e0 secure :-1 s=READY pgs=675 cs=0 l=1 rx=0x7f993c002f30 tx=0x7f993c00fbf0).stop 2019-12-05T06:43:27.058 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:27.038+0000 7f994e44b700 1 -- 172.21.15.42:0/2575173278 shutdown_connections 2019-12-05T06:43:27.058 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:27.038+0000 7f994e44b700 1 --2- 172.21.15.42:0/2575173278 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f994808df80 0x7f994809f2e0 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:27.058 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:27.038+0000 7f994e44b700 1 -- 172.21.15.42:0/2575173278 shutdown_connections 2019-12-05T06:43:27.058 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:27.038+0000 7f994e44b700 1 -- 172.21.15.42:0/2575173278 wait complete. 2019-12-05T06:43:27.058 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:27.038+0000 7f994e44b700 1 -- 172.21.15.42:0/2575173278 >> 172.21.15.42:0/2575173278 conn(0x7f99480aabe0 msgr2=0x7f99480a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:43:27.058 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:27.042+0000 7f994e44b700 1 Processor -- start 2019-12-05T06:43:27.059 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:27.042+0000 7f994e44b700 1 -- start start 2019-12-05T06:43:27.059 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:27.042+0000 7f994e44b700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f9948144970 0x7f99480a3180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:43:27.059 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:27.042+0000 7f994e44b700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f994807a820 con 0x7f9948122ba0 2019-12-05T06:43:27.059 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:27.042+0000 7f994e44b700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99480a36d0 con 0x7f9948122770 2019-12-05T06:43:27.059 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:27.042+0000 7f994e44b700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9948144da0 con 0x7f9948144970 2019-12-05T06:43:27.059 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:27.042+0000 7f994c9e8700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f9948144970 0x7f99480a3180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:27.059 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:27.042+0000 7f994c9e8700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f9948144970 0x7f99480a3180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:34238/0 (socket says 172.21.15.42:34238) 2019-12-05T06:43:27.059 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:27.042+0000 7f994c9e8700 1 -- 172.21.15.42:0/3972904194 learned_addr learned my addr 172.21.15.42:0/3972904194 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:43:27.060 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:27.042+0000 7f99457fa700 1 -- 172.21.15.42:0/3972904194 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1715043706 0 0) 0x7f994807a820 con 0x7f9948122ba0 2019-12-05T06:43:27.060 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:27.042+0000 7f99457fa700 1 -- 172.21.15.42:0/3972904194 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99280030b0 con 0x7f9948122ba0 2019-12-05T06:43:27.060 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:27.042+0000 7f994c9e8700 1 -- 172.21.15.42:0/3972904194 >> v1:172.21.15.42:6789/0 conn(0x7f9948122770 legacy=0x7f994807ab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:27.060 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:27.042+0000 7f994c9e8700 1 -- 172.21.15.42:0/3972904194 >> v1:172.21.15.40:6789/0 conn(0x7f9948122ba0 legacy=0x7f994812ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:27.060 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:27.042+0000 7f994c9e8700 1 -- 172.21.15.42:0/3972904194 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9948143d80 con 0x7f9948144970 2019-12-05T06:43:27.060 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:27.042+0000 7f994c9e8700 1 --2- 172.21.15.42:0/3972904194 >> v2:172.21.15.42:3300/0 conn(0x7f9948144970 0x7f99480a3180 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f99480a1260 tx=0x7f9938010c70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:27.061 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:27.042+0000 7f994c9e8700 1 --2- 172.21.15.42:0/3972904194 >> v2:172.21.15.42:3300/0 conn(0x7f9948144970 0x7f99480a3180 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f99480a1260 tx=0x7f9938010c70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:27.061 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:27.042+0000 7f994c9e8700 1 --2- 172.21.15.42:0/3972904194 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9948144970 0x7f99480a3180 secure :-1 s=READY pgs=3110 cs=0 l=1 rx=0x7f99480a1260 tx=0x7f9938010c70).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:43:27.061 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:27.042+0000 7f994c9e8700 1 --2- 172.21.15.42:0/3972904194 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9948144970 0x7f99480a3180 secure :-1 s=THROTTLE_DONE pgs=3110 cs=0 l=1 rx=0x7f99480a1260 tx=0x7f9938010c70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:27.061 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:27.042+0000 7f99457fa700 1 -- 172.21.15.42:0/3972904194 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f993800cb20 con 0x7f9948144970 2019-12-05T06:43:27.062 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:27.046+0000 7f994c9e8700 1 --2- 172.21.15.42:0/3972904194 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9948144970 0x7f99480a3180 secure :-1 s=THROTTLE_DONE pgs=3110 cs=0 l=1 rx=0x7f99480a1260 tx=0x7f9938010c70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:27.062 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:27.046+0000 7f994c9e8700 1 --2- 172.21.15.42:0/3972904194 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9948144970 0x7f99480a3180 secure :-1 s=THROTTLE_DONE pgs=3110 cs=0 l=1 rx=0x7f99480a1260 tx=0x7f9938010c70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:27.062 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:27.046+0000 7f99457fa700 1 -- 172.21.15.42:0/3972904194 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f993800c740 con 0x7f9948144970 2019-12-05T06:43:27.062 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:27.046+0000 7f994e44b700 1 -- 172.21.15.42:0/3972904194 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f994807b360 con 0x7f9948144970 2019-12-05T06:43:27.062 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:27.046+0000 7f99457fa700 1 -- 172.21.15.42:0/3972904194 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9938012860 con 0x7f9948144970 2019-12-05T06:43:27.063 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:27.046+0000 7f994e44b700 1 -- 172.21.15.42:0/3972904194 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f994812e440 con 0x7f9948144970 2019-12-05T06:43:27.063 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:27.046+0000 7f994c9e8700 1 --2- 172.21.15.42:0/3972904194 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9948144970 0x7f99480a3180 secure :-1 s=THROTTLE_DONE pgs=3110 cs=0 l=1 rx=0x7f99480a1260 tx=0x7f9938010c70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:27.063 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:27.046+0000 7f99457fa700 1 -- 172.21.15.42:0/3972904194 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f9938012bc0 con 0x7f9948144970 2019-12-05T06:43:27.063 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:27.046+0000 7f994c9e8700 1 --2- 172.21.15.42:0/3972904194 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9948144970 0x7f99480a3180 secure :-1 s=THROTTLE_DONE pgs=3110 cs=0 l=1 rx=0x7f99480a1260 tx=0x7f9938010c70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:27.063 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:27.046+0000 7f99457fa700 1 --2- 172.21.15.42:0/3972904194 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f992803b1c0 0x7f992803d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:43:27.063 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:27.046+0000 7f99457fa700 1 -- 172.21.15.42:0/3972904194 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 11879+0+0 (secure 0 0 0) 0x7f993804b3b0 con 0x7f9948144970 2019-12-05T06:43:27.064 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:27.046+0000 7f9947fff700 1 --2- 172.21.15.42:0/3972904194 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f992803b1c0 0x7f992803d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:27.064 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:27.046+0000 7f9947fff700 1 --2- 172.21.15.42:0/3972904194 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f992803b1c0 0x7f992803d6d0 crc :-1 s=READY pgs=1297 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:43:27.064 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:27.050+0000 7f994e44b700 1 -- 172.21.15.42:0/3972904194 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f994805d110 con 0x7f9948144970 2019-12-05T06:43:27.065 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:27.058+0000 7f994c9e8700 1 --2- 172.21.15.42:0/3972904194 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9948144970 0x7f99480a3180 secure :-1 s=THROTTLE_DONE pgs=3110 cs=0 l=1 rx=0x7f99480a1260 tx=0x7f9938010c70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:27.065 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:27.058+0000 7f99457fa700 1 -- 172.21.15.42:0/3972904194 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f9938032410 con 0x7f9948144970 2019-12-05T06:43:27.231 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:27.226+0000 7f994e44b700 1 -- 172.21.15.42:0/3972904194 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f994805b8c0 con 0x7f9948144970 2019-12-05T06:43:27.232 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:27.226+0000 7f994c9e8700 1 --2- 172.21.15.42:0/3972904194 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9948144970 0x7f99480a3180 secure :-1 s=THROTTLE_DONE pgs=3110 cs=0 l=1 rx=0x7f99480a1260 tx=0x7f9938010c70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:27.232 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:27.226+0000 7f99457fa700 1 -- 172.21.15.42:0/3972904194 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 373 v373) v1 ==== 95+0+1769 (secure 0 0 0) 0x7f9938032670 con 0x7f9948144970 2019-12-05T06:43:27.233 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 373 2019-12-05T06:43:27.233 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:43:27.233 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":373,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":373,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:43:25.248554+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":117,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12496},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12496":{"gid":12496,"name":"b","rank":0,"incarnation":373,"state":"up:replay","state_seq":2,"addr":"172.21.15.40:6833/3599619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3599619},{"type":"v1","addr":"172.21.15.40:6833","nonce":3599619}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:43:27.235 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:27.230+0000 7f994e44b700 1 -- 172.21.15.42:0/3972904194 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f992803b1c0 msgr2=0x7f992803d6d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:27.236 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:27.230+0000 7f994e44b700 1 --2- 172.21.15.42:0/3972904194 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f992803b1c0 0x7f992803d6d0 crc :-1 s=READY pgs=1297 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:27.236 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:27.230+0000 7f994e44b700 1 -- 172.21.15.42:0/3972904194 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9948144970 msgr2=0x7f99480a3180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:27.237 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:27.230+0000 7f994e44b700 1 --2- 172.21.15.42:0/3972904194 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9948144970 0x7f99480a3180 secure :-1 s=READY pgs=3110 cs=0 l=1 rx=0x7f99480a1260 tx=0x7f9938010c70).stop 2019-12-05T06:43:27.237 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:27.230+0000 7f994e44b700 1 -- 172.21.15.42:0/3972904194 shutdown_connections 2019-12-05T06:43:27.237 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:27.230+0000 7f994e44b700 1 --2- 172.21.15.42:0/3972904194 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f992803b1c0 0x7f992803d6d0 unknown :-1 s=CLOSED pgs=1297 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:27.238 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:27.230+0000 7f994e44b700 1 --2- 172.21.15.42:0/3972904194 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9948144970 0x7f99480a3180 unknown :-1 s=CLOSED pgs=3110 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:27.238 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:27.230+0000 7f994e44b700 1 -- 172.21.15.42:0/3972904194 shutdown_connections 2019-12-05T06:43:27.238 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:27.230+0000 7f994e44b700 1 -- 172.21.15.42:0/3972904194 wait complete. 2019-12-05T06:43:27.238 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:27.230+0000 7f994e44b700 1 -- 172.21.15.42:0/3972904194 >> 172.21.15.42:0/3972904194 conn(0x7f99480a9ae0 msgr2=0x7f994807e450 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:43:27.245 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 373, "feature_flags": { "enable_multiple": false, "ever_enabled_multiple": false }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2019-12-05T05:03:35.604738+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 373, "ever_allowed_features": 2, "explicitly_allowed_features": 2, "failed": [], "flags": 18, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_12496": { "addr": "172.21.15.40:6833/3599619", "addrs": { "addrvec": [ { "addr": "172.21.15.40:6832", "nonce": 3599619, "type": "v2" }, { "addr": "172.21.15.40:6833", "nonce": 3599619, "type": "v1" } ] }, "export_targets": [], "features": 4540138292836696063, "flags": 0, "gid": 12496, "incarnation": 373, "name": "b", "rank": 0, "state": "up:replay", "state_seq": 2 } }, "last_failure": 0, "last_failure_osd_epoch": 117, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 2, "min_compat_client": "0 (unknown)", "modified": "2019-12-05T06:43:25.248554+0000", "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 12496 } } } ], "standbys": [] } 2019-12-05T06:43:29.246 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:43:29.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:29.346+0000 7f75b6dbc700 1 Processor -- start 2019-12-05T06:43:29.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:29.346+0000 7f75b6dbc700 1 -- start start 2019-12-05T06:43:29.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:29.346+0000 7f75b6dbc700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f75b008df80 0x7f75b009f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:43:29.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:29.346+0000 7f75b6dbc700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f75b009f830 0x7f75b014efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:43:29.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:29.346+0000 7f75b6dbc700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75b009fc60 con 0x7f75b00a7740 2019-12-05T06:43:29.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:29.346+0000 7f75b6dbc700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75b00a0b00 con 0x7f75b009f830 2019-12-05T06:43:29.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:29.346+0000 7f75b6dbc700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75b008e5b0 con 0x7f75b008df80 2019-12-05T06:43:29.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:29.350+0000 7f75b5359700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f75b00a7740 0x7f75b007e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:34282/0 (socket says 172.21.15.42:34282) 2019-12-05T06:43:29.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:29.350+0000 7f75b5359700 1 -- 172.21.15.42:0/983947442 learned_addr learned my addr 172.21.15.42:0/983947442 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:43:29.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:29.350+0000 7f75b5b5a700 1 --2- 172.21.15.42:0/983947442 >> v2:172.21.15.42:3300/0 conn(0x7f75b009f830 0x7f75b014efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:29.358 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:29.350+0000 7f75b4b58700 1 --2- 172.21.15.42:0/983947442 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f75b008df80 0x7f75b009f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:29.358 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:29.350+0000 7f75a3fff700 1 -- 172.21.15.42:0/983947442 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2905784529 0 0) 0x7f75b009fc60 con 0x7f75b00a7740 2019-12-05T06:43:29.358 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:29.350+0000 7f75a3fff700 1 -- 172.21.15.42:0/983947442 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f759c0030b0 con 0x7f75b00a7740 2019-12-05T06:43:29.358 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:29.350+0000 7f75b5b5a700 1 -- 172.21.15.42:0/983947442 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f75b008df80 msgr2=0x7f75b009f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:43:29.358 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:29.350+0000 7f75b5b5a700 1 --2- 172.21.15.42:0/983947442 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f75b008df80 0x7f75b009f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:43:29.358 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:29.350+0000 7f75b5b5a700 1 -- 172.21.15.42:0/983947442 >> v1:172.21.15.42:6789/0 conn(0x7f75b00a7740 legacy=0x7f75b007e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:29.358 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:29.350+0000 7f75b5b5a700 1 -- 172.21.15.42:0/983947442 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75b00a0710 con 0x7f75b009f830 2019-12-05T06:43:29.359 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:29.350+0000 7f75b4b58700 1 --2- 172.21.15.42:0/983947442 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f75b008df80 0x7f75b009f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:43:29.359 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:29.350+0000 7f75b5b5a700 1 --2- 172.21.15.42:0/983947442 >> v2:172.21.15.42:3300/0 conn(0x7f75b009f830 0x7f75b014efe0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f75a8008b10 tx=0x7f75a800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:29.359 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:29.350+0000 7f75b5b5a700 1 --2- 172.21.15.42:0/983947442 >> v2:172.21.15.42:3300/0 conn(0x7f75b009f830 0x7f75b014efe0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f75a8008b10 tx=0x7f75a800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:29.359 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:29.350+0000 7f75b5b5a700 1 --2- 172.21.15.42:0/983947442 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f75b009f830 0x7f75b014efe0 secure :-1 s=READY pgs=3112 cs=0 l=1 rx=0x7f75a8008b10 tx=0x7f75a800fff0).ready entity=mon.1 client_cookie=9b694cfe4ed8e806 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:43:29.359 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:29.350+0000 7f75b5b5a700 1 --2- 172.21.15.42:0/983947442 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f75b009f830 0x7f75b014efe0 secure :-1 s=THROTTLE_DONE pgs=3112 cs=0 l=1 rx=0x7f75a8008b10 tx=0x7f75a800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:29.359 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:29.350+0000 7f75a3fff700 1 -- 172.21.15.42:0/983947442 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f75a8010900 con 0x7f75b009f830 2019-12-05T06:43:29.360 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:29.350+0000 7f75b5b5a700 1 --2- 172.21.15.42:0/983947442 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f75b009f830 0x7f75b014efe0 secure :-1 s=THROTTLE_DONE pgs=3112 cs=0 l=1 rx=0x7f75a8008b10 tx=0x7f75a800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:29.360 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:29.350+0000 7f75b5b5a700 1 --2- 172.21.15.42:0/983947442 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f75b009f830 0x7f75b014efe0 secure :-1 s=THROTTLE_DONE pgs=3112 cs=0 l=1 rx=0x7f75a8008b10 tx=0x7f75a800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:29.360 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:29.350+0000 7f75a3fff700 1 -- 172.21.15.42:0/983947442 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75a8008b50 con 0x7f75b009f830 2019-12-05T06:43:29.360 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:29.350+0000 7f75a3fff700 1 -- 172.21.15.42:0/983947442 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f75a80110c0 con 0x7f75b009f830 2019-12-05T06:43:29.361 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:29.350+0000 7f75b6dbc700 1 -- 172.21.15.42:0/983947442 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f75b009f830 msgr2=0x7f75b014efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:29.361 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:29.350+0000 7f75b6dbc700 1 --2- 172.21.15.42:0/983947442 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f75b009f830 0x7f75b014efe0 secure :-1 s=READY pgs=3112 cs=0 l=1 rx=0x7f75a8008b10 tx=0x7f75a800fff0).stop 2019-12-05T06:43:29.361 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:29.350+0000 7f75b6dbc700 1 -- 172.21.15.42:0/983947442 shutdown_connections 2019-12-05T06:43:29.362 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:29.350+0000 7f75b6dbc700 1 --2- 172.21.15.42:0/983947442 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f75b009f830 0x7f75b014efe0 unknown :-1 s=CLOSED pgs=3112 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:29.362 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:29.350+0000 7f75b6dbc700 1 --2- 172.21.15.42:0/983947442 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f75b008df80 0x7f75b009f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:43:29.362 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:29.350+0000 7f75b6dbc700 1 -- 172.21.15.42:0/983947442 shutdown_connections 2019-12-05T06:43:29.363 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:29.350+0000 7f75b6dbc700 1 -- 172.21.15.42:0/983947442 wait complete. 2019-12-05T06:43:29.364 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:29.350+0000 7f75b6dbc700 1 -- 172.21.15.42:0/983947442 >> 172.21.15.42:0/983947442 conn(0x7f75b00aabe0 msgr2=0x7f75b00a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:43:29.368 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:29.354+0000 7f75b6dbc700 1 Processor -- start 2019-12-05T06:43:29.368 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:29.354+0000 7f75b6dbc700 1 -- start start 2019-12-05T06:43:29.368 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:29.354+0000 7f75b6dbc700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f75b01242b0 0x7f75b01386e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:43:29.368 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:29.354+0000 7f75b6dbc700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f75b0138c30 0x7f75b013b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:43:29.368 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:29.354+0000 7f75b6dbc700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75b007e120 con 0x7f75b0122d80 2019-12-05T06:43:29.368 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:29.354+0000 7f75b6dbc700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75b0117840 con 0x7f75b01242b0 2019-12-05T06:43:29.369 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:29.354+0000 7f75b6dbc700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75b0117ae0 con 0x7f75b0138c30 2019-12-05T06:43:29.370 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:29.354+0000 7f75b4b58700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f75b01242b0 0x7f75b01386e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:29.370 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:29.354+0000 7f75b5b5a700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f75b0138c30 0x7f75b013b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:29.370 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:29.354+0000 7f75b4b58700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f75b01242b0 0x7f75b01386e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.42:50300/0 (socket says 172.21.15.42:50300) 2019-12-05T06:43:29.370 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:29.354+0000 7f75b4b58700 1 -- 172.21.15.42:0/386082013 learned_addr learned my addr 172.21.15.42:0/386082013 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:43:29.370 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:29.354+0000 7f75b4b58700 1 -- 172.21.15.42:0/386082013 >> v2:172.21.15.42:3300/0 conn(0x7f75b0138c30 msgr2=0x7f75b013b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:29.371 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:29.354+0000 7f75b4b58700 1 --2- 172.21.15.42:0/386082013 >> v2:172.21.15.42:3300/0 conn(0x7f75b0138c30 0x7f75b013b070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:29.371 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:29.354+0000 7f75b4b58700 1 -- 172.21.15.42:0/386082013 >> v1:172.21.15.40:6789/0 conn(0x7f75b0122d80 legacy=0x7f75b0121380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:29.371 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:29.354+0000 7f75b4b58700 1 -- 172.21.15.42:0/386082013 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75b0117d80 con 0x7f75b01242b0 2019-12-05T06:43:29.371 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:29.354+0000 7f75b4b58700 1 --2- 172.21.15.42:0/386082013 >> v2:172.21.15.40:3300/0 conn(0x7f75b01242b0 0x7f75b01386e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f75b0123960 tx=0x7f75ac010db0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:29.371 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:29.354+0000 7f75b4b58700 1 --2- 172.21.15.42:0/386082013 >> v2:172.21.15.40:3300/0 conn(0x7f75b01242b0 0x7f75b01386e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f75b0123960 tx=0x7f75ac010db0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:29.371 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:29.354+0000 7f75b4b58700 1 --2- 172.21.15.42:0/386082013 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f75b01242b0 0x7f75b01386e0 secure :-1 s=READY pgs=2342 cs=0 l=1 rx=0x7f75b0123960 tx=0x7f75ac010db0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:43:29.371 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:29.354+0000 7f75b4b58700 1 --2- 172.21.15.42:0/386082013 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f75b01242b0 0x7f75b01386e0 secure :-1 s=THROTTLE_DONE pgs=2342 cs=0 l=1 rx=0x7f75b0123960 tx=0x7f75ac010db0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:29.372 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:29.354+0000 7f75a1ffb700 1 -- 172.21.15.42:0/386082013 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f75ac0058e0 con 0x7f75b01242b0 2019-12-05T06:43:29.372 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:29.354+0000 7f75b4b58700 1 --2- 172.21.15.42:0/386082013 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f75b01242b0 0x7f75b01386e0 secure :-1 s=THROTTLE_DONE pgs=2342 cs=0 l=1 rx=0x7f75b0123960 tx=0x7f75ac010db0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:29.372 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:29.354+0000 7f75a1ffb700 1 -- 172.21.15.42:0/386082013 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75ac007160 con 0x7f75b01242b0 2019-12-05T06:43:29.372 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:29.354+0000 7f75b4b58700 1 --2- 172.21.15.42:0/386082013 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f75b01242b0 0x7f75b01386e0 secure :-1 s=THROTTLE_DONE pgs=2342 cs=0 l=1 rx=0x7f75b0123960 tx=0x7f75ac010db0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:29.372 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:29.354+0000 7f75b6dbc700 1 -- 172.21.15.42:0/386082013 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75b0130270 con 0x7f75b01242b0 2019-12-05T06:43:29.372 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:29.354+0000 7f75a1ffb700 1 -- 172.21.15.42:0/386082013 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f75ac00e3f0 con 0x7f75b01242b0 2019-12-05T06:43:29.374 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:29.354+0000 7f75b6dbc700 1 -- 172.21.15.42:0/386082013 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75b0121b80 con 0x7f75b01242b0 2019-12-05T06:43:29.374 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:29.358+0000 7f75b4b58700 1 --2- 172.21.15.42:0/386082013 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f75b01242b0 0x7f75b01386e0 secure :-1 s=THROTTLE_DONE pgs=2342 cs=0 l=1 rx=0x7f75b0123960 tx=0x7f75ac010db0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:29.374 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:29.358+0000 7f75a1ffb700 1 -- 172.21.15.42:0/386082013 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f75ac010180 con 0x7f75b01242b0 2019-12-05T06:43:29.374 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:29.358+0000 7f75b4b58700 1 --2- 172.21.15.42:0/386082013 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f75b01242b0 0x7f75b01386e0 secure :-1 s=THROTTLE_DONE pgs=2342 cs=0 l=1 rx=0x7f75b0123960 tx=0x7f75ac010db0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:29.374 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:29.358+0000 7f75a1ffb700 1 --2- 172.21.15.42:0/386082013 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f759003aca0 0x7f759003d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:43:29.375 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:29.358+0000 7f75a1ffb700 1 -- 172.21.15.42:0/386082013 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 11879+0+0 (secure 0 0 0) 0x7f75ac04b130 con 0x7f75b01242b0 2019-12-05T06:43:29.375 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:29.358+0000 7f75b5359700 1 --2- 172.21.15.42:0/386082013 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f759003aca0 0x7f759003d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:29.375 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:29.358+0000 7f75b5359700 1 --2- 172.21.15.42:0/386082013 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f759003aca0 0x7f759003d1b0 crc :-1 s=READY pgs=1298 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:43:29.375 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:29.362+0000 7f75b6dbc700 1 -- 172.21.15.42:0/386082013 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75b005d110 con 0x7f75b01242b0 2019-12-05T06:43:29.376 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:29.370+0000 7f75b4b58700 1 --2- 172.21.15.42:0/386082013 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f75b01242b0 0x7f75b01386e0 secure :-1 s=THROTTLE_DONE pgs=2342 cs=0 l=1 rx=0x7f75b0123960 tx=0x7f75ac010db0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:29.376 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:29.370+0000 7f75a1ffb700 1 -- 172.21.15.42:0/386082013 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f75ac032390 con 0x7f75b01242b0 2019-12-05T06:43:29.548 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:29.542+0000 7f75b6dbc700 1 -- 172.21.15.42:0/386082013 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f75b005b8c0 con 0x7f75b01242b0 2019-12-05T06:43:29.549 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:29.546+0000 7f75b4b58700 1 --2- 172.21.15.42:0/386082013 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f75b01242b0 0x7f75b01386e0 secure :-1 s=THROTTLE_DONE pgs=2342 cs=0 l=1 rx=0x7f75b0123960 tx=0x7f75ac010db0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:29.549 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:29.546+0000 7f75a1ffb700 1 -- 172.21.15.42:0/386082013 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 373 v373) v1 ==== 95+0+1769 (secure 0 0 0) 0x7f75ac032f40 con 0x7f75b01242b0 2019-12-05T06:43:29.550 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 373 2019-12-05T06:43:29.550 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:43:29.550 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":373,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":373,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:43:25.248554+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":117,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12496},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12496":{"gid":12496,"name":"b","rank":0,"incarnation":373,"state":"up:replay","state_seq":2,"addr":"172.21.15.40:6833/3599619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3599619},{"type":"v1","addr":"172.21.15.40:6833","nonce":3599619}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:43:29.552 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:29.546+0000 7f75b6dbc700 1 -- 172.21.15.42:0/386082013 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f759003aca0 msgr2=0x7f759003d1b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:29.552 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:29.546+0000 7f75b6dbc700 1 --2- 172.21.15.42:0/386082013 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f759003aca0 0x7f759003d1b0 crc :-1 s=READY pgs=1298 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:29.552 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:29.546+0000 7f75b6dbc700 1 -- 172.21.15.42:0/386082013 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f75b01242b0 msgr2=0x7f75b01386e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:29.552 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:29.546+0000 7f75b6dbc700 1 --2- 172.21.15.42:0/386082013 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f75b01242b0 0x7f75b01386e0 secure :-1 s=READY pgs=2342 cs=0 l=1 rx=0x7f75b0123960 tx=0x7f75ac010db0).stop 2019-12-05T06:43:29.553 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:29.546+0000 7f75b6dbc700 1 -- 172.21.15.42:0/386082013 shutdown_connections 2019-12-05T06:43:29.553 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:29.546+0000 7f75b6dbc700 1 --2- 172.21.15.42:0/386082013 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f759003aca0 0x7f759003d1b0 unknown :-1 s=CLOSED pgs=1298 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:29.553 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:29.546+0000 7f75b6dbc700 1 --2- 172.21.15.42:0/386082013 >> v2:172.21.15.42:3300/0 conn(0x7f75b0138c30 0x7f75b013b070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:29.553 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:29.546+0000 7f75b6dbc700 1 --2- 172.21.15.42:0/386082013 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f75b01242b0 0x7f75b01386e0 unknown :-1 s=CLOSED pgs=2342 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:29.553 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:29.546+0000 7f75b6dbc700 1 -- 172.21.15.42:0/386082013 shutdown_connections 2019-12-05T06:43:29.553 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:29.546+0000 7f75b6dbc700 1 -- 172.21.15.42:0/386082013 wait complete. 2019-12-05T06:43:29.554 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:29.546+0000 7f75b6dbc700 1 -- 172.21.15.42:0/386082013 >> 172.21.15.42:0/386082013 conn(0x7f75b0067080 msgr2=0x7f75b00a6b40 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:43:29.562 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2019-12-05T06:43:31.562 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:43:31.663 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:31.654+0000 7f3b97e38700 1 Processor -- start 2019-12-05T06:43:31.663 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:31.654+0000 7f3b97e38700 1 -- start start 2019-12-05T06:43:31.664 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:31.654+0000 7f3b97e38700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3b900a7740 0x7f3b9008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:43:31.664 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:31.654+0000 7f3b97e38700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3b9007e720 0x7f3b9007eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:43:31.664 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:31.654+0000 7f3b97e38700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b900a0710 con 0x7f3b9009f2e0 2019-12-05T06:43:31.664 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:31.654+0000 7f3b97e38700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b900a0b00 con 0x7f3b9007e720 2019-12-05T06:43:31.664 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:31.654+0000 7f3b97e38700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b9008e5b0 con 0x7f3b900a7740 2019-12-05T06:43:31.664 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:31.654+0000 7f3b95bd4700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3b9007e720 0x7f3b9007eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:31.664 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:31.654+0000 7f3b95bd4700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3b9007e720 0x7f3b9007eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:34262/0 (socket says 172.21.15.42:34262) 2019-12-05T06:43:31.665 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:31.654+0000 7f3b95bd4700 1 -- 172.21.15.42:0/627552462 learned_addr learned my addr 172.21.15.42:0/627552462 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:43:31.665 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:31.654+0000 7f3b963d5700 1 --2- 172.21.15.42:0/627552462 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3b900a7740 0x7f3b9008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:31.665 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:31.658+0000 7f3b95bd4700 1 -- 172.21.15.42:0/627552462 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3b900a7740 msgr2=0x7f3b9008df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:43:31.665 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:31.658+0000 7f3b95bd4700 1 --2- 172.21.15.42:0/627552462 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3b900a7740 0x7f3b9008df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:43:31.665 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:31.658+0000 7f3b95bd4700 1 -- 172.21.15.42:0/627552462 >> v1:172.21.15.40:6789/0 conn(0x7f3b9009f2e0 legacy=0x7f3b9009f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:31.665 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:31.658+0000 7f3b95bd4700 1 -- 172.21.15.42:0/627552462 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b9007f0a0 con 0x7f3b9007e720 2019-12-05T06:43:31.666 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:31.658+0000 7f3b95bd4700 1 --2- 172.21.15.42:0/627552462 >> v2:172.21.15.42:3300/0 conn(0x7f3b9007e720 0x7f3b9007eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f3b80008790 tx=0x7f3b8000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:31.666 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:31.658+0000 7f3b95bd4700 1 --2- 172.21.15.42:0/627552462 >> v2:172.21.15.42:3300/0 conn(0x7f3b9007e720 0x7f3b9007eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f3b80008790 tx=0x7f3b8000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:31.666 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:31.658+0000 7f3b95bd4700 1 --2- 172.21.15.42:0/627552462 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3b9007e720 0x7f3b9007eb50 secure :-1 s=READY pgs=3113 cs=0 l=1 rx=0x7f3b80008790 tx=0x7f3b8000fd30).ready entity=mon.1 client_cookie=5b617aa54bf188ac server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:43:31.666 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:31.658+0000 7f3b95bd4700 1 --2- 172.21.15.42:0/627552462 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3b9007e720 0x7f3b9007eb50 secure :-1 s=THROTTLE_DONE pgs=3113 cs=0 l=1 rx=0x7f3b80008790 tx=0x7f3b8000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:31.666 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:31.658+0000 7f3b953d3700 1 -- 172.21.15.42:0/627552462 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3b800113b0 con 0x7f3b9007e720 2019-12-05T06:43:31.666 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:31.658+0000 7f3b95bd4700 1 --2- 172.21.15.42:0/627552462 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3b9007e720 0x7f3b9007eb50 secure :-1 s=THROTTLE_DONE pgs=3113 cs=0 l=1 rx=0x7f3b80008790 tx=0x7f3b8000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:31.667 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:31.658+0000 7f3b95bd4700 1 --2- 172.21.15.42:0/627552462 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3b9007e720 0x7f3b9007eb50 secure :-1 s=THROTTLE_DONE pgs=3113 cs=0 l=1 rx=0x7f3b80008790 tx=0x7f3b8000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:31.667 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:31.658+0000 7f3b953d3700 1 -- 172.21.15.42:0/627552462 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b80008bb0 con 0x7f3b9007e720 2019-12-05T06:43:31.667 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:31.658+0000 7f3b953d3700 1 -- 172.21.15.42:0/627552462 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3b80011990 con 0x7f3b9007e720 2019-12-05T06:43:31.667 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:31.658+0000 7f3b97e38700 1 -- 172.21.15.42:0/627552462 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3b9007e720 msgr2=0x7f3b9007eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:31.667 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:31.658+0000 7f3b97e38700 1 --2- 172.21.15.42:0/627552462 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3b9007e720 0x7f3b9007eb50 secure :-1 s=READY pgs=3113 cs=0 l=1 rx=0x7f3b80008790 tx=0x7f3b8000fd30).stop 2019-12-05T06:43:31.667 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:31.658+0000 7f3b97e38700 1 -- 172.21.15.42:0/627552462 shutdown_connections 2019-12-05T06:43:31.667 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:31.658+0000 7f3b97e38700 1 --2- 172.21.15.42:0/627552462 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3b900a7740 0x7f3b9008df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:43:31.668 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:31.658+0000 7f3b97e38700 1 --2- 172.21.15.42:0/627552462 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3b9007e720 0x7f3b9007eb50 unknown :-1 s=CLOSED pgs=3113 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:31.668 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:31.658+0000 7f3b97e38700 1 -- 172.21.15.42:0/627552462 shutdown_connections 2019-12-05T06:43:31.668 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:31.658+0000 7f3b97e38700 1 -- 172.21.15.42:0/627552462 wait complete. 2019-12-05T06:43:31.668 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:31.658+0000 7f3b97e38700 1 -- 172.21.15.42:0/627552462 >> 172.21.15.42:0/627552462 conn(0x7f3b900aabe0 msgr2=0x7f3b900a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:43:31.671 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:31.662+0000 7f3b97e38700 1 Processor -- start 2019-12-05T06:43:31.671 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:31.662+0000 7f3b97e38700 1 -- start start 2019-12-05T06:43:31.671 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:31.662+0000 7f3b97e38700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3b901188c0 0x7f3b90116ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:43:31.671 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:31.662+0000 7f3b97e38700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3b901319a0 0x7f3b90131dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:43:31.672 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:31.662+0000 7f3b97e38700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b9007c760 con 0x7f3b901192c0 2019-12-05T06:43:31.672 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:31.662+0000 7f3b97e38700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b901175e0 con 0x7f3b901188c0 2019-12-05T06:43:31.672 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:31.662+0000 7f3b97e38700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b90117880 con 0x7f3b901319a0 2019-12-05T06:43:31.672 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:31.662+0000 7f3b95bd4700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3b901188c0 0x7f3b90116ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:31.672 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:31.662+0000 7f3b95bd4700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3b901188c0 0x7f3b90116ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:34268/0 (socket says 172.21.15.42:34268) 2019-12-05T06:43:31.672 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:31.662+0000 7f3b95bd4700 1 -- 172.21.15.42:0/3051417836 learned_addr learned my addr 172.21.15.42:0/3051417836 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:43:31.673 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:31.662+0000 7f3b96bd6700 1 --2- 172.21.15.42:0/3051417836 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3b901319a0 0x7f3b90131dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:31.673 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:31.662+0000 7f3b95bd4700 1 -- 172.21.15.42:0/3051417836 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3b901319a0 msgr2=0x7f3b90131dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:31.673 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:31.662+0000 7f3b95bd4700 1 --2- 172.21.15.42:0/3051417836 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3b901319a0 0x7f3b90131dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:31.673 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:31.662+0000 7f3b95bd4700 1 -- 172.21.15.42:0/3051417836 >> v1:172.21.15.40:6789/0 conn(0x7f3b901192c0 legacy=0x7f3b9007bf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:31.673 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:31.662+0000 7f3b95bd4700 1 -- 172.21.15.42:0/3051417836 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b90132320 con 0x7f3b901188c0 2019-12-05T06:43:31.673 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:31.662+0000 7f3b95bd4700 1 --2- 172.21.15.42:0/3051417836 >> v2:172.21.15.42:3300/0 conn(0x7f3b901188c0 0x7f3b90116ed0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f3b901177a0 tx=0x7f3b80008bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:31.674 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:31.662+0000 7f3b95bd4700 1 --2- 172.21.15.42:0/3051417836 >> v2:172.21.15.42:3300/0 conn(0x7f3b901188c0 0x7f3b90116ed0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f3b901177a0 tx=0x7f3b80008bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:31.674 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:31.662+0000 7f3b95bd4700 1 --2- 172.21.15.42:0/3051417836 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3b901188c0 0x7f3b90116ed0 secure :-1 s=READY pgs=3114 cs=0 l=1 rx=0x7f3b901177a0 tx=0x7f3b80008bb0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:43:31.674 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:31.666+0000 7f3b95bd4700 1 --2- 172.21.15.42:0/3051417836 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3b901188c0 0x7f3b90116ed0 secure :-1 s=THROTTLE_DONE pgs=3114 cs=0 l=1 rx=0x7f3b901177a0 tx=0x7f3b80008bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:31.674 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:31.666+0000 7f3b95bd4700 1 --2- 172.21.15.42:0/3051417836 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3b901188c0 0x7f3b90116ed0 secure :-1 s=THROTTLE_DONE pgs=3114 cs=0 l=1 rx=0x7f3b901177a0 tx=0x7f3b80008bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:31.675 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:31.666+0000 7f3b95bd4700 1 --2- 172.21.15.42:0/3051417836 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3b901188c0 0x7f3b90116ed0 secure :-1 s=THROTTLE_DONE pgs=3114 cs=0 l=1 rx=0x7f3b901177a0 tx=0x7f3b80008bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:31.675 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:31.666+0000 7f3b86ffd700 1 -- 172.21.15.42:0/3051417836 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3b80007340 con 0x7f3b901188c0 2019-12-05T06:43:31.675 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:31.666+0000 7f3b86ffd700 1 -- 172.21.15.42:0/3051417836 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b800115b0 con 0x7f3b901188c0 2019-12-05T06:43:31.675 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:31.666+0000 7f3b97e38700 1 -- 172.21.15.42:0/3051417836 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b90138c00 con 0x7f3b901188c0 2019-12-05T06:43:31.675 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:31.666+0000 7f3b86ffd700 1 -- 172.21.15.42:0/3051417836 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3b80005540 con 0x7f3b901188c0 2019-12-05T06:43:31.675 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:31.666+0000 7f3b97e38700 1 -- 172.21.15.42:0/3051417836 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b90139180 con 0x7f3b901188c0 2019-12-05T06:43:31.676 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:31.666+0000 7f3b95bd4700 1 --2- 172.21.15.42:0/3051417836 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3b901188c0 0x7f3b90116ed0 secure :-1 s=THROTTLE_DONE pgs=3114 cs=0 l=1 rx=0x7f3b901177a0 tx=0x7f3b80008bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:31.676 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:31.666+0000 7f3b86ffd700 1 -- 172.21.15.42:0/3051417836 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f3b80005810 con 0x7f3b901188c0 2019-12-05T06:43:31.676 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:31.666+0000 7f3b95bd4700 1 --2- 172.21.15.42:0/3051417836 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3b901188c0 0x7f3b90116ed0 secure :-1 s=THROTTLE_DONE pgs=3114 cs=0 l=1 rx=0x7f3b901177a0 tx=0x7f3b80008bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:31.676 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:31.666+0000 7f3b86ffd700 1 --2- 172.21.15.42:0/3051417836 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3b7803ae60 0x7f3b7803d370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:43:31.676 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:31.666+0000 7f3b86ffd700 1 -- 172.21.15.42:0/3051417836 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 11879+0+0 (secure 0 0 0) 0x7f3b8004b420 con 0x7f3b901188c0 2019-12-05T06:43:31.676 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:31.666+0000 7f3b963d5700 1 --2- 172.21.15.42:0/3051417836 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3b7803ae60 0x7f3b7803d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:31.676 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:31.666+0000 7f3b963d5700 1 --2- 172.21.15.42:0/3051417836 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3b7803ae60 0x7f3b7803d370 crc :-1 s=READY pgs=1299 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:43:31.677 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:31.670+0000 7f3b97e38700 1 -- 172.21.15.42:0/3051417836 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b90118cf0 con 0x7f3b901188c0 2019-12-05T06:43:31.679 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:31.674+0000 7f3b95bd4700 1 --2- 172.21.15.42:0/3051417836 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3b901188c0 0x7f3b90116ed0 secure :-1 s=THROTTLE_DONE pgs=3114 cs=0 l=1 rx=0x7f3b901177a0 tx=0x7f3b80008bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:31.679 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:31.674+0000 7f3b86ffd700 1 -- 172.21.15.42:0/3051417836 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f3b800324a0 con 0x7f3b901188c0 2019-12-05T06:43:31.841 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:31.838+0000 7f3b97e38700 1 -- 172.21.15.42:0/3051417836 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3b9005d110 con 0x7f3b901188c0 2019-12-05T06:43:31.842 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:31.838+0000 7f3b95bd4700 1 --2- 172.21.15.42:0/3051417836 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3b901188c0 0x7f3b90116ed0 secure :-1 s=THROTTLE_DONE pgs=3114 cs=0 l=1 rx=0x7f3b901177a0 tx=0x7f3b80008bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:31.842 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:31.838+0000 7f3b86ffd700 1 -- 172.21.15.42:0/3051417836 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 374 v374) v1 ==== 95+0+1772 (secure 0 0 0) 0x7f3b80032700 con 0x7f3b901188c0 2019-12-05T06:43:31.842 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 374 2019-12-05T06:43:31.843 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:43:31.843 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":374,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":374,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:43:31.524798+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":117,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12496},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12496":{"gid":12496,"name":"b","rank":0,"incarnation":373,"state":"up:reconnect","state_seq":9,"addr":"172.21.15.40:6833/3599619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3599619},{"type":"v1","addr":"172.21.15.40:6833","nonce":3599619}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:43:31.844 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:31.842+0000 7f3b97e38700 1 -- 172.21.15.42:0/3051417836 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3b7803ae60 msgr2=0x7f3b7803d370 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:31.845 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:31.842+0000 7f3b97e38700 1 --2- 172.21.15.42:0/3051417836 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3b7803ae60 0x7f3b7803d370 crc :-1 s=READY pgs=1299 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:31.845 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:31.842+0000 7f3b97e38700 1 -- 172.21.15.42:0/3051417836 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3b901188c0 msgr2=0x7f3b90116ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:31.845 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:31.842+0000 7f3b97e38700 1 --2- 172.21.15.42:0/3051417836 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3b901188c0 0x7f3b90116ed0 secure :-1 s=READY pgs=3114 cs=0 l=1 rx=0x7f3b901177a0 tx=0x7f3b80008bb0).stop 2019-12-05T06:43:31.845 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:31.842+0000 7f3b97e38700 1 -- 172.21.15.42:0/3051417836 shutdown_connections 2019-12-05T06:43:31.845 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:31.842+0000 7f3b97e38700 1 --2- 172.21.15.42:0/3051417836 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3b7803ae60 0x7f3b7803d370 unknown :-1 s=CLOSED pgs=1299 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:31.845 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:31.842+0000 7f3b97e38700 1 --2- 172.21.15.42:0/3051417836 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3b901319a0 0x7f3b90131dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:31.846 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:31.842+0000 7f3b97e38700 1 --2- 172.21.15.42:0/3051417836 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3b901188c0 0x7f3b90116ed0 unknown :-1 s=CLOSED pgs=3114 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:31.846 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:31.842+0000 7f3b97e38700 1 -- 172.21.15.42:0/3051417836 shutdown_connections 2019-12-05T06:43:31.846 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:31.842+0000 7f3b97e38700 1 -- 172.21.15.42:0/3051417836 wait complete. 2019-12-05T06:43:31.846 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:31.842+0000 7f3b97e38700 1 -- 172.21.15.42:0/3051417836 >> 172.21.15.42:0/3051417836 conn(0x7f3b900aabe0 msgr2=0x7f3b900a9790 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:43:31.855 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2019-12-05T06:43:33.858 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:43:33.941 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:33.934+0000 7fac7c8e5700 1 Processor -- start 2019-12-05T06:43:33.942 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:33.934+0000 7fac7c8e5700 1 -- start start 2019-12-05T06:43:33.942 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:33.934+0000 7fac7c8e5700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fac7808df80 0x7fac7809f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:43:33.942 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:33.934+0000 7fac7c8e5700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac7809fc60 con 0x7fac7809f830 2019-12-05T06:43:33.942 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:33.934+0000 7fac7c8e5700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac780a0710 con 0x7fac780a7740 2019-12-05T06:43:33.942 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:33.934+0000 7fac7c8e5700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac780a0b00 con 0x7fac7808df80 2019-12-05T06:43:33.942 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:33.934+0000 7fac7659c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fac7808df80 0x7fac7809f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:33.943 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:33.934+0000 7fac7659c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fac7808df80 0x7fac7809f2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:34278/0 (socket says 172.21.15.42:34278) 2019-12-05T06:43:33.943 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:33.934+0000 7fac7659c700 1 -- 172.21.15.42:0/3358792524 learned_addr learned my addr 172.21.15.42:0/3358792524 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:43:33.943 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:33.934+0000 7fac7659c700 1 -- 172.21.15.42:0/3358792524 >> v1:172.21.15.42:6789/0 conn(0x7fac780a7740 legacy=0x7fac7807e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:33.943 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:33.934+0000 7fac7659c700 1 -- 172.21.15.42:0/3358792524 >> v1:172.21.15.40:6789/0 conn(0x7fac7809f830 legacy=0x7fac7814efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:33.943 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:33.934+0000 7fac7659c700 1 -- 172.21.15.42:0/3358792524 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac7807ef20 con 0x7fac7808df80 2019-12-05T06:43:33.943 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:33.934+0000 7fac7659c700 1 --2- 172.21.15.42:0/3358792524 >> v2:172.21.15.42:3300/0 conn(0x7fac7808df80 0x7fac7809f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fac68008b10 tx=0x7fac68010030).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:33.943 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:33.934+0000 7fac7659c700 1 --2- 172.21.15.42:0/3358792524 >> v2:172.21.15.42:3300/0 conn(0x7fac7808df80 0x7fac7809f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fac68008b10 tx=0x7fac68010030).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:33.944 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:33.934+0000 7fac7659c700 1 --2- 172.21.15.42:0/3358792524 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fac7808df80 0x7fac7809f2e0 secure :-1 s=READY pgs=3116 cs=0 l=1 rx=0x7fac68008b10 tx=0x7fac68010030).ready entity=mon.1 client_cookie=5e11b4bfc09de697 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:43:33.944 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:33.934+0000 7fac7659c700 1 --2- 172.21.15.42:0/3358792524 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fac7808df80 0x7fac7809f2e0 secure :-1 s=THROTTLE_DONE pgs=3116 cs=0 l=1 rx=0x7fac68008b10 tx=0x7fac68010030).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:33.944 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:33.934+0000 7fac7659c700 1 --2- 172.21.15.42:0/3358792524 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fac7808df80 0x7fac7809f2e0 secure :-1 s=THROTTLE_DONE pgs=3116 cs=0 l=1 rx=0x7fac68008b10 tx=0x7fac68010030).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:33.944 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:33.934+0000 7fac7659c700 1 --2- 172.21.15.42:0/3358792524 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fac7808df80 0x7fac7809f2e0 secure :-1 s=THROTTLE_DONE pgs=3116 cs=0 l=1 rx=0x7fac68008b10 tx=0x7fac68010030).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:33.944 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:33.934+0000 7fac75d9b700 1 -- 172.21.15.42:0/3358792524 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fac6800a160 con 0x7fac7808df80 2019-12-05T06:43:33.944 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:33.934+0000 7fac75d9b700 1 -- 172.21.15.42:0/3358792524 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac68011aa0 con 0x7fac7808df80 2019-12-05T06:43:33.944 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:33.934+0000 7fac75d9b700 1 -- 172.21.15.42:0/3358792524 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fac6800a740 con 0x7fac7808df80 2019-12-05T06:43:33.945 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:33.934+0000 7fac7c8e5700 1 -- 172.21.15.42:0/3358792524 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fac7808df80 msgr2=0x7fac7809f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:33.945 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:33.934+0000 7fac7c8e5700 1 --2- 172.21.15.42:0/3358792524 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fac7808df80 0x7fac7809f2e0 secure :-1 s=READY pgs=3116 cs=0 l=1 rx=0x7fac68008b10 tx=0x7fac68010030).stop 2019-12-05T06:43:33.945 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:33.938+0000 7fac7c8e5700 1 -- 172.21.15.42:0/3358792524 shutdown_connections 2019-12-05T06:43:33.945 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:33.938+0000 7fac7c8e5700 1 --2- 172.21.15.42:0/3358792524 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fac7808df80 0x7fac7809f2e0 unknown :-1 s=CLOSED pgs=3116 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:33.945 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:33.938+0000 7fac7c8e5700 1 -- 172.21.15.42:0/3358792524 shutdown_connections 2019-12-05T06:43:33.945 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:33.938+0000 7fac7c8e5700 1 -- 172.21.15.42:0/3358792524 wait complete. 2019-12-05T06:43:33.946 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:33.938+0000 7fac7c8e5700 1 -- 172.21.15.42:0/3358792524 >> 172.21.15.42:0/3358792524 conn(0x7fac780aabe0 msgr2=0x7fac780a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:43:33.950 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:33.938+0000 7fac7c8e5700 1 Processor -- start 2019-12-05T06:43:33.950 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:33.938+0000 7fac7c8e5700 1 -- start start 2019-12-05T06:43:33.950 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:33.938+0000 7fac7c8e5700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fac78145f70 0x7fac78118900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:43:33.950 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:33.938+0000 7fac7c8e5700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac7812ee70 con 0x7fac780a3a60 2019-12-05T06:43:33.950 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:33.938+0000 7fac7c8e5700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac781300b0 con 0x7fac78117770 2019-12-05T06:43:33.950 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:33.938+0000 7fac7c8e5700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac78119010 con 0x7fac78145f70 2019-12-05T06:43:33.951 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:33.938+0000 7fac7759e700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fac78117770 0x7fac78117ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:34326/0 (socket says 172.21.15.42:34326) 2019-12-05T06:43:33.951 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:33.938+0000 7fac7659c700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fac78145f70 0x7fac78118900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:33.951 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:33.938+0000 7fac7759e700 1 -- 172.21.15.42:0/512059523 learned_addr learned my addr 172.21.15.42:0/512059523 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:43:33.951 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:33.938+0000 7fac677fe700 1 -- 172.21.15.42:0/512059523 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3841056834 0 0) 0x7fac7812ee70 con 0x7fac780a3a60 2019-12-05T06:43:33.951 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:33.938+0000 7fac677fe700 1 -- 172.21.15.42:0/512059523 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac5c0030b0 con 0x7fac780a3a60 2019-12-05T06:43:33.951 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:33.938+0000 7fac677fe700 1 -- 172.21.15.42:0/512059523 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 880250103 0 0) 0x7fac781300b0 con 0x7fac78117770 2019-12-05T06:43:33.951 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:33.938+0000 7fac677fe700 1 -- 172.21.15.42:0/512059523 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac7812ee70 con 0x7fac78117770 2019-12-05T06:43:33.952 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:33.938+0000 7fac7659c700 1 -- 172.21.15.42:0/512059523 >> v1:172.21.15.42:6789/0 conn(0x7fac78117770 legacy=0x7fac78117ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:33.952 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:33.938+0000 7fac7659c700 1 -- 172.21.15.42:0/512059523 >> v1:172.21.15.40:6789/0 conn(0x7fac780a3a60 legacy=0x7fac780781d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:33.952 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:33.938+0000 7fac7659c700 1 -- 172.21.15.42:0/512059523 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac781312f0 con 0x7fac78145f70 2019-12-05T06:43:33.952 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:33.938+0000 7fac7659c700 1 --2- 172.21.15.42:0/512059523 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fac78145f70 0x7fac78118900 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fac68008b10 tx=0x7fac68011cd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:33.952 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:33.942+0000 7fac7659c700 1 --2- 172.21.15.42:0/512059523 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fac78145f70 0x7fac78118900 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fac68008b10 tx=0x7fac68011cd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:33.952 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:33.942+0000 7fac7659c700 1 --2- 172.21.15.42:0/512059523 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fac78145f70 0x7fac78118900 secure :-1 s=READY pgs=676 cs=0 l=1 rx=0x7fac68008b10 tx=0x7fac68011cd0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:43:33.953 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:33.942+0000 7fac7659c700 1 --2- 172.21.15.42:0/512059523 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fac78145f70 0x7fac78118900 secure :-1 s=THROTTLE_DONE pgs=676 cs=0 l=1 rx=0x7fac68008b10 tx=0x7fac68011cd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:33.953 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:33.942+0000 7fac677fe700 1 -- 172.21.15.42:0/512059523 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fac680058c0 con 0x7fac78145f70 2019-12-05T06:43:33.953 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:33.942+0000 7fac7659c700 1 --2- 172.21.15.42:0/512059523 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fac78145f70 0x7fac78118900 secure :-1 s=THROTTLE_DONE pgs=676 cs=0 l=1 rx=0x7fac68008b10 tx=0x7fac68011cd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:33.953 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:33.942+0000 7fac7659c700 1 --2- 172.21.15.42:0/512059523 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fac78145f70 0x7fac78118900 secure :-1 s=THROTTLE_DONE pgs=676 cs=0 l=1 rx=0x7fac68008b10 tx=0x7fac68011cd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:33.953 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:33.942+0000 7fac677fe700 1 -- 172.21.15.42:0/512059523 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac6800a360 con 0x7fac78145f70 2019-12-05T06:43:33.953 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:33.942+0000 7fac7c8e5700 1 -- 172.21.15.42:0/512059523 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac781302e0 con 0x7fac78145f70 2019-12-05T06:43:33.953 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:33.942+0000 7fac677fe700 1 -- 172.21.15.42:0/512059523 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fac68007150 con 0x7fac78145f70 2019-12-05T06:43:33.954 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:33.942+0000 7fac7c8e5700 1 -- 172.21.15.42:0/512059523 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac781306c0 con 0x7fac78145f70 2019-12-05T06:43:33.954 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:33.942+0000 7fac7659c700 1 --2- 172.21.15.42:0/512059523 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fac78145f70 0x7fac78118900 secure :-1 s=THROTTLE_DONE pgs=676 cs=0 l=1 rx=0x7fac68008b10 tx=0x7fac68011cd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:33.954 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:33.942+0000 7fac677fe700 1 -- 172.21.15.42:0/512059523 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fac68007460 con 0x7fac78145f70 2019-12-05T06:43:33.954 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:33.942+0000 7fac7659c700 1 --2- 172.21.15.42:0/512059523 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fac78145f70 0x7fac78118900 secure :-1 s=THROTTLE_DONE pgs=676 cs=0 l=1 rx=0x7fac68008b10 tx=0x7fac68011cd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:33.954 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:33.942+0000 7fac677fe700 1 --2- 172.21.15.42:0/512059523 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fac5c03b540 0x7fac5c03da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:43:33.954 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:33.942+0000 7fac677fe700 1 -- 172.21.15.42:0/512059523 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 11879+0+0 (secure 0 0 0) 0x7fac6804bd20 con 0x7fac78145f70 2019-12-05T06:43:33.954 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:33.942+0000 7fac76d9d700 1 --2- 172.21.15.42:0/512059523 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fac5c03b540 0x7fac5c03da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:33.955 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:33.942+0000 7fac76d9d700 1 --2- 172.21.15.42:0/512059523 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fac5c03b540 0x7fac5c03da50 crc :-1 s=READY pgs=1300 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:43:33.955 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:33.950+0000 7fac7c8e5700 1 -- 172.21.15.42:0/512059523 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac7805d110 con 0x7fac78145f70 2019-12-05T06:43:33.969 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:33.958+0000 7fac7659c700 1 --2- 172.21.15.42:0/512059523 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fac78145f70 0x7fac78118900 secure :-1 s=THROTTLE_DONE pgs=676 cs=0 l=1 rx=0x7fac68008b10 tx=0x7fac68011cd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:33.969 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:33.958+0000 7fac677fe700 1 -- 172.21.15.42:0/512059523 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fac68032da0 con 0x7fac78145f70 2019-12-05T06:43:34.143 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:34.138+0000 7fac7c8e5700 1 -- 172.21.15.42:0/512059523 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fac781505d0 con 0x7fac78145f70 2019-12-05T06:43:34.144 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:34.138+0000 7fac7659c700 1 --2- 172.21.15.42:0/512059523 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fac78145f70 0x7fac78118900 secure :-1 s=THROTTLE_DONE pgs=676 cs=0 l=1 rx=0x7fac68008b10 tx=0x7fac68011cd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:34.144 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:34.138+0000 7fac677fe700 1 -- 172.21.15.42:0/512059523 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 376 v376) v1 ==== 95+0+1770 (secure 0 0 0) 0x7fac68033000 con 0x7fac78145f70 2019-12-05T06:43:34.144 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 376 2019-12-05T06:43:34.144 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:43:34.144 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":376,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":376,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:43:33.543372+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":117,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12496},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12496":{"gid":12496,"name":"b","rank":0,"incarnation":373,"state":"up:active","state_seq":11,"addr":"172.21.15.40:6833/3599619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3599619},{"type":"v1","addr":"172.21.15.40:6833","nonce":3599619}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:43:34.146 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:34.142+0000 7fac7c8e5700 1 -- 172.21.15.42:0/512059523 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fac5c03b540 msgr2=0x7fac5c03da50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:34.146 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:34.142+0000 7fac7c8e5700 1 --2- 172.21.15.42:0/512059523 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fac5c03b540 0x7fac5c03da50 crc :-1 s=READY pgs=1300 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:34.146 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:34.142+0000 7fac7c8e5700 1 -- 172.21.15.42:0/512059523 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fac78145f70 msgr2=0x7fac78118900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:34.146 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:34.142+0000 7fac7c8e5700 1 --2- 172.21.15.42:0/512059523 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fac78145f70 0x7fac78118900 secure :-1 s=READY pgs=676 cs=0 l=1 rx=0x7fac68008b10 tx=0x7fac68011cd0).stop 2019-12-05T06:43:34.146 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:34.142+0000 7fac7c8e5700 1 -- 172.21.15.42:0/512059523 shutdown_connections 2019-12-05T06:43:34.147 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:34.142+0000 7fac7c8e5700 1 --2- 172.21.15.42:0/512059523 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fac5c03b540 0x7fac5c03da50 unknown :-1 s=CLOSED pgs=1300 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:34.147 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:34.142+0000 7fac7c8e5700 1 --2- 172.21.15.42:0/512059523 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fac78145f70 0x7fac78118900 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:34.147 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:34.142+0000 7fac7c8e5700 1 -- 172.21.15.42:0/512059523 shutdown_connections 2019-12-05T06:43:34.147 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:34.142+0000 7fac7c8e5700 1 -- 172.21.15.42:0/512059523 wait complete. 2019-12-05T06:43:34.147 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:34.142+0000 7fac7c8e5700 1 -- 172.21.15.42:0/512059523 >> 172.21.15.42:0/512059523 conn(0x7fac780aabe0 msgr2=0x7fac7809f240 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:43:34.159 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b has gained rank=0, replacing gid=12358 2019-12-05T06:43:34.159 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 6 secs before reviving mds.a 2019-12-05T06:43:40.160 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.a 2019-12-05T06:43:40.160 INFO:tasks.ceph.mds.a:Restarting daemon 2019-12-05T06:43:40.161 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2019-12-05T06:43:40.167 INFO:tasks.ceph.mds.a:Started 2019-12-05T06:43:40.168 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:43:40.227 INFO:tasks.ceph.mds.a.smithi042.stdout:starting mds.a at 2019-12-05T06:43:40.267 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:40.258+0000 7f35e782c700 1 Processor -- start 2019-12-05T06:43:40.267 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:40.258+0000 7f35e782c700 1 -- start start 2019-12-05T06:43:40.267 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:40.262+0000 7f35e782c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f35e00a7740 0x7f35e008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:43:40.267 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:40.262+0000 7f35e782c700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f35e007e720 0x7f35e007eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:43:40.267 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:40.262+0000 7f35e782c700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f35e009f2e0 0x7f35e009f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:43:40.268 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:40.262+0000 7f35e782c700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35e00a0b00 con 0x7f35e007e720 2019-12-05T06:43:40.268 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:40.262+0000 7f35e782c700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35e008e5b0 con 0x7f35e00a7740 2019-12-05T06:43:40.268 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:40.262+0000 7f35e782c700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35e008e850 con 0x7f35e009f2e0 2019-12-05T06:43:40.268 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:40.262+0000 7f35e5dc9700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f35e00a7740 0x7f35e008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:40.268 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:40.262+0000 7f35e5dc9700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f35e00a7740 0x7f35e008df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:34326/0 (socket says 172.21.15.42:34326) 2019-12-05T06:43:40.268 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:40.262+0000 7f35e5dc9700 1 -- 172.21.15.42:0/4056783417 learned_addr learned my addr 172.21.15.42:0/4056783417 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:43:40.268 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:40.262+0000 7f35e65ca700 1 --2- 172.21.15.42:0/4056783417 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f35e009f2e0 0x7f35e009f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:40.269 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:40.262+0000 7f35e55c8700 1 --2- 172.21.15.42:0/4056783417 >> v2:172.21.15.40:3300/0 conn(0x7f35e007e720 0x7f35e007eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:40.269 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:40.262+0000 7f35e5dc9700 1 -- 172.21.15.42:0/4056783417 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f35e009f2e0 msgr2=0x7f35e009f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:43:40.269 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:40.262+0000 7f35e5dc9700 1 --2- 172.21.15.42:0/4056783417 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f35e009f2e0 0x7f35e009f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:43:40.269 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:40.262+0000 7f35e5dc9700 1 -- 172.21.15.42:0/4056783417 >> v2:172.21.15.40:3300/0 conn(0x7f35e007e720 msgr2=0x7f35e007eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:43:40.269 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:40.262+0000 7f35e5dc9700 1 --2- 172.21.15.42:0/4056783417 >> v2:172.21.15.40:3300/0 conn(0x7f35e007e720 0x7f35e007eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:43:40.269 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:40.262+0000 7f35e5dc9700 1 -- 172.21.15.42:0/4056783417 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35e007f0a0 con 0x7f35e00a7740 2019-12-05T06:43:40.269 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:40.262+0000 7f35e5dc9700 1 --2- 172.21.15.42:0/4056783417 >> v2:172.21.15.42:3300/0 conn(0x7f35e00a7740 0x7f35e008df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f35e00a0f50 tx=0x7f35d400fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:40.270 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:40.262+0000 7f35e5dc9700 1 --2- 172.21.15.42:0/4056783417 >> v2:172.21.15.42:3300/0 conn(0x7f35e00a7740 0x7f35e008df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f35e00a0f50 tx=0x7f35d400fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:40.270 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:40.262+0000 7f35e5dc9700 1 --2- 172.21.15.42:0/4056783417 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f35e00a7740 0x7f35e008df80 secure :-1 s=READY pgs=3119 cs=0 l=1 rx=0x7f35e00a0f50 tx=0x7f35d400fdb0).ready entity=mon.1 client_cookie=873be7e43efd0a2a server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:43:40.270 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:40.262+0000 7f35e5dc9700 1 --2- 172.21.15.42:0/4056783417 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f35e00a7740 0x7f35e008df80 secure :-1 s=THROTTLE_DONE pgs=3119 cs=0 l=1 rx=0x7f35e00a0f50 tx=0x7f35d400fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:40.271 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:40.262+0000 7f35e4dc7700 1 -- 172.21.15.42:0/4056783417 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f35d4008c90 con 0x7f35e00a7740 2019-12-05T06:43:40.271 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:40.262+0000 7f35e5dc9700 1 --2- 172.21.15.42:0/4056783417 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f35e00a7740 0x7f35e008df80 secure :-1 s=THROTTLE_DONE pgs=3119 cs=0 l=1 rx=0x7f35e00a0f50 tx=0x7f35d400fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:40.271 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:40.262+0000 7f35e4dc7700 1 -- 172.21.15.42:0/4056783417 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35d40088b0 con 0x7f35e00a7740 2019-12-05T06:43:40.271 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:40.262+0000 7f35e5dc9700 1 --2- 172.21.15.42:0/4056783417 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f35e00a7740 0x7f35e008df80 secure :-1 s=THROTTLE_DONE pgs=3119 cs=0 l=1 rx=0x7f35e00a0f50 tx=0x7f35d400fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:40.271 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:40.262+0000 7f35e4dc7700 1 -- 172.21.15.42:0/4056783417 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f35d4011b50 con 0x7f35e00a7740 2019-12-05T06:43:40.272 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:40.262+0000 7f35e782c700 1 -- 172.21.15.42:0/4056783417 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f35e00a7740 msgr2=0x7f35e008df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:40.272 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:40.262+0000 7f35e782c700 1 --2- 172.21.15.42:0/4056783417 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f35e00a7740 0x7f35e008df80 secure :-1 s=READY pgs=3119 cs=0 l=1 rx=0x7f35e00a0f50 tx=0x7f35d400fdb0).stop 2019-12-05T06:43:40.273 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:40.262+0000 7f35e782c700 1 -- 172.21.15.42:0/4056783417 shutdown_connections 2019-12-05T06:43:40.273 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:40.262+0000 7f35e782c700 1 --2- 172.21.15.42:0/4056783417 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f35e009f2e0 0x7f35e009f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:43:40.273 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:40.262+0000 7f35e782c700 1 --2- 172.21.15.42:0/4056783417 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f35e00a7740 0x7f35e008df80 unknown :-1 s=CLOSED pgs=3119 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:40.273 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:40.262+0000 7f35e782c700 1 --2- 172.21.15.42:0/4056783417 >> v2:172.21.15.40:3300/0 conn(0x7f35e007e720 0x7f35e007eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:43:40.273 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:40.262+0000 7f35e782c700 1 -- 172.21.15.42:0/4056783417 shutdown_connections 2019-12-05T06:43:40.273 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:40.262+0000 7f35e782c700 1 -- 172.21.15.42:0/4056783417 wait complete. 2019-12-05T06:43:40.274 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:40.262+0000 7f35e782c700 1 -- 172.21.15.42:0/4056783417 >> 172.21.15.42:0/4056783417 conn(0x7f35e00aabe0 msgr2=0x7f35e00a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:43:40.277 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:40.262+0000 7f35e782c700 1 Processor -- start 2019-12-05T06:43:40.277 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:40.262+0000 7f35e782c700 1 -- start start 2019-12-05T06:43:40.277 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:40.266+0000 7f35e782c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f35e0121510 0x7f35e011d750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:43:40.277 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:40.266+0000 7f35e782c700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f35e0131630 0x7f35e0131a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:43:40.277 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:40.266+0000 7f35e782c700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35e01312c0 con 0x7f35e00a3b10 2019-12-05T06:43:40.278 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:40.266+0000 7f35e782c700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35e0121ac0 con 0x7f35e0131630 2019-12-05T06:43:40.278 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:40.266+0000 7f35e782c700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35e0123f80 con 0x7f35e0121510 2019-12-05T06:43:40.278 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:40.266+0000 7f35e55c8700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f35e0121510 0x7f35e011d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:40.278 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:40.266+0000 7f35e55c8700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f35e0121510 0x7f35e011d750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:34334/0 (socket says 172.21.15.42:34334) 2019-12-05T06:43:40.278 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:40.266+0000 7f35e55c8700 1 -- 172.21.15.42:0/2997562406 learned_addr learned my addr 172.21.15.42:0/2997562406 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:43:40.278 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:40.266+0000 7f35e65ca700 1 --2- 172.21.15.42:0/2997562406 >> v2:172.21.15.40:3300/0 conn(0x7f35e0131630 0x7f35e0131a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:40.278 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:40.266+0000 7f35e55c8700 1 -- 172.21.15.42:0/2997562406 >> v2:172.21.15.40:3300/0 conn(0x7f35e0131630 msgr2=0x7f35e0131a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:40.279 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:40.266+0000 7f35e55c8700 1 --2- 172.21.15.42:0/2997562406 >> v2:172.21.15.40:3300/0 conn(0x7f35e0131630 0x7f35e0131a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:40.279 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:40.266+0000 7f35e55c8700 1 -- 172.21.15.42:0/2997562406 >> v1:172.21.15.40:6789/0 conn(0x7f35e00a3b10 legacy=0x7f35e0123780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:40.279 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:40.266+0000 7f35e55c8700 1 -- 172.21.15.42:0/2997562406 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35e0131fb0 con 0x7f35e0121510 2019-12-05T06:43:40.279 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:40.266+0000 7f35e55c8700 1 --2- 172.21.15.42:0/2997562406 >> v2:172.21.15.42:3300/0 conn(0x7f35e0121510 0x7f35e011d750 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f35dc003070 tx=0x7f35dc010bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:40.279 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:40.266+0000 7f35e55c8700 1 --2- 172.21.15.42:0/2997562406 >> v2:172.21.15.42:3300/0 conn(0x7f35e0121510 0x7f35e011d750 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f35dc003070 tx=0x7f35dc010bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:40.279 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:40.266+0000 7f35e55c8700 1 --2- 172.21.15.42:0/2997562406 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f35e0121510 0x7f35e011d750 secure :-1 s=READY pgs=3120 cs=0 l=1 rx=0x7f35dc003070 tx=0x7f35dc010bc0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:43:40.280 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:40.266+0000 7f35e55c8700 1 --2- 172.21.15.42:0/2997562406 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f35e0121510 0x7f35e011d750 secure :-1 s=THROTTLE_DONE pgs=3120 cs=0 l=1 rx=0x7f35dc003070 tx=0x7f35dc010bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:40.280 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:40.266+0000 7f35d27fc700 1 -- 172.21.15.42:0/2997562406 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f35dc005890 con 0x7f35e0121510 2019-12-05T06:43:40.280 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:40.266+0000 7f35e55c8700 1 --2- 172.21.15.42:0/2997562406 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f35e0121510 0x7f35e011d750 secure :-1 s=THROTTLE_DONE pgs=3120 cs=0 l=1 rx=0x7f35dc003070 tx=0x7f35dc010bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:40.280 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:40.266+0000 7f35e55c8700 1 --2- 172.21.15.42:0/2997562406 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f35e0121510 0x7f35e011d750 secure :-1 s=THROTTLE_DONE pgs=3120 cs=0 l=1 rx=0x7f35dc003070 tx=0x7f35dc010bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:40.280 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:40.266+0000 7f35d27fc700 1 -- 172.21.15.42:0/2997562406 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35dc007160 con 0x7f35e0121510 2019-12-05T06:43:40.281 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:40.266+0000 7f35e782c700 1 -- 172.21.15.42:0/2997562406 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35e0122240 con 0x7f35e0121510 2019-12-05T06:43:40.281 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:40.266+0000 7f35d27fc700 1 -- 172.21.15.42:0/2997562406 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f35dc012240 con 0x7f35e0121510 2019-12-05T06:43:40.281 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:40.266+0000 7f35e782c700 1 -- 172.21.15.42:0/2997562406 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35e0078530 con 0x7f35e0121510 2019-12-05T06:43:40.281 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:40.266+0000 7f35e55c8700 1 --2- 172.21.15.42:0/2997562406 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f35e0121510 0x7f35e011d750 secure :-1 s=THROTTLE_DONE pgs=3120 cs=0 l=1 rx=0x7f35dc003070 tx=0x7f35dc010bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:40.281 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:40.266+0000 7f35d27fc700 1 -- 172.21.15.42:0/2997562406 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f35dc012550 con 0x7f35e0121510 2019-12-05T06:43:40.281 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:40.266+0000 7f35e55c8700 1 --2- 172.21.15.42:0/2997562406 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f35e0121510 0x7f35e011d750 secure :-1 s=THROTTLE_DONE pgs=3120 cs=0 l=1 rx=0x7f35dc003070 tx=0x7f35dc010bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:40.281 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:40.266+0000 7f35d27fc700 1 --2- 172.21.15.42:0/2997562406 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f35c803ae10 0x7f35c803d320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:43:40.282 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:40.266+0000 7f35d27fc700 1 -- 172.21.15.42:0/2997562406 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 11879+0+0 (secure 0 0 0) 0x7f35dc04b1f0 con 0x7f35e0121510 2019-12-05T06:43:40.282 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:40.266+0000 7f35e5dc9700 1 --2- 172.21.15.42:0/2997562406 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f35c803ae10 0x7f35c803d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:40.282 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:40.266+0000 7f35e5dc9700 1 --2- 172.21.15.42:0/2997562406 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f35c803ae10 0x7f35c803d320 crc :-1 s=READY pgs=1301 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:43:40.283 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:40.274+0000 7f35e782c700 1 -- 172.21.15.42:0/2997562406 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35e0121c80 con 0x7f35e0121510 2019-12-05T06:43:40.286 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:40.282+0000 7f35e55c8700 1 --2- 172.21.15.42:0/2997562406 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f35e0121510 0x7f35e011d750 secure :-1 s=THROTTLE_DONE pgs=3120 cs=0 l=1 rx=0x7f35dc003070 tx=0x7f35dc010bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:40.286 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:40.282+0000 7f35d27fc700 1 -- 172.21.15.42:0/2997562406 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f35dc032270 con 0x7f35e0121510 2019-12-05T06:43:40.455 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:40.450+0000 7f35e782c700 1 -- 172.21.15.42:0/2997562406 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f35e005d110 con 0x7f35e0121510 2019-12-05T06:43:40.456 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:40.454+0000 7f35e55c8700 1 --2- 172.21.15.42:0/2997562406 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f35e0121510 0x7f35e011d750 secure :-1 s=THROTTLE_DONE pgs=3120 cs=0 l=1 rx=0x7f35dc003070 tx=0x7f35dc010bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:40.457 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:40.454+0000 7f35d27fc700 1 -- 172.21.15.42:0/2997562406 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 376 v376) v1 ==== 95+0+1770 (secure 0 0 0) 0x7f35dc0324d0 con 0x7f35e0121510 2019-12-05T06:43:40.457 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 376 2019-12-05T06:43:40.457 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:43:40.457 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":376,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":376,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:43:33.543372+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":117,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12496},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12496":{"gid":12496,"name":"b","rank":0,"incarnation":373,"state":"up:active","state_seq":11,"addr":"172.21.15.40:6833/3599619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3599619},{"type":"v1","addr":"172.21.15.40:6833","nonce":3599619}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:43:40.458 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:40.454+0000 7f35e782c700 1 -- 172.21.15.42:0/2997562406 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f35c803ae10 msgr2=0x7f35c803d320 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:40.459 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:40.454+0000 7f35e782c700 1 --2- 172.21.15.42:0/2997562406 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f35c803ae10 0x7f35c803d320 crc :-1 s=READY pgs=1301 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:40.459 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:40.454+0000 7f35e782c700 1 -- 172.21.15.42:0/2997562406 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f35e0121510 msgr2=0x7f35e011d750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:40.459 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:40.454+0000 7f35e782c700 1 --2- 172.21.15.42:0/2997562406 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f35e0121510 0x7f35e011d750 secure :-1 s=READY pgs=3120 cs=0 l=1 rx=0x7f35dc003070 tx=0x7f35dc010bc0).stop 2019-12-05T06:43:40.460 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:40.454+0000 7f35e782c700 1 -- 172.21.15.42:0/2997562406 shutdown_connections 2019-12-05T06:43:40.460 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:40.454+0000 7f35e782c700 1 --2- 172.21.15.42:0/2997562406 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f35c803ae10 0x7f35c803d320 unknown :-1 s=CLOSED pgs=1301 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:40.460 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:40.454+0000 7f35e782c700 1 --2- 172.21.15.42:0/2997562406 >> v2:172.21.15.40:3300/0 conn(0x7f35e0131630 0x7f35e0131a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:40.460 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:40.454+0000 7f35e782c700 1 --2- 172.21.15.42:0/2997562406 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f35e0121510 0x7f35e011d750 unknown :-1 s=CLOSED pgs=3120 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:40.461 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:40.454+0000 7f35e782c700 1 -- 172.21.15.42:0/2997562406 shutdown_connections 2019-12-05T06:43:40.461 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:40.454+0000 7f35e782c700 1 -- 172.21.15.42:0/2997562406 wait complete. 2019-12-05T06:43:40.461 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:40.454+0000 7f35e782c700 1 -- 172.21.15.42:0/2997562406 >> 172.21.15.42:0/2997562406 conn(0x7f35e00aabe0 msgr2=0x7f35e00a9770 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:43:40.475 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2019-12-05T06:43:42.476 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:43:42.577 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:42.566+0000 7f4a8a326700 1 Processor -- start 2019-12-05T06:43:42.577 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:42.566+0000 7f4a8a326700 1 -- start start 2019-12-05T06:43:42.577 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:42.570+0000 7f4a8a326700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f4a840a7730 0x7f4a8408df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:43:42.577 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:42.570+0000 7f4a8a326700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4a8409fad0 0x7f4a8407eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:43:42.577 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:42.570+0000 7f4a8a326700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a840a0700 con 0x7f4a8407e710 2019-12-05T06:43:42.578 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:42.570+0000 7f4a8a326700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a840a0af0 con 0x7f4a840a7730 2019-12-05T06:43:42.578 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:42.570+0000 7f4a8a326700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a8408e5a0 con 0x7f4a8409fad0 2019-12-05T06:43:42.578 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:42.570+0000 7f4a83fff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f4a8407e710 0x7f4a8409f2d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:34388/0 (socket says 172.21.15.42:34388) 2019-12-05T06:43:42.578 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:42.570+0000 7f4a83fff700 1 -- 172.21.15.42:0/383931379 learned_addr learned my addr 172.21.15.42:0/383931379 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:43:42.578 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:42.570+0000 7f4a888c3700 1 --2- 172.21.15.42:0/383931379 >> v2:172.21.15.40:3300/0 conn(0x7f4a840a7730 0x7f4a8408df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:42.578 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:42.570+0000 7f4a837fe700 1 -- 172.21.15.42:0/383931379 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3936427274 0 0) 0x7f4a840a0700 con 0x7f4a8407e710 2019-12-05T06:43:42.578 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:42.570+0000 7f4a837fe700 1 -- 172.21.15.42:0/383931379 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a740030b0 con 0x7f4a8407e710 2019-12-05T06:43:42.579 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:42.570+0000 7f4a888c3700 1 -- 172.21.15.42:0/383931379 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4a8409fad0 msgr2=0x7f4a8407eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:43:42.579 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:42.570+0000 7f4a888c3700 1 --2- 172.21.15.42:0/383931379 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4a8409fad0 0x7f4a8407eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:43:42.580 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:42.570+0000 7f4a888c3700 1 -- 172.21.15.42:0/383931379 >> v1:172.21.15.42:6789/0 conn(0x7f4a8407e710 legacy=0x7f4a8409f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:42.580 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:42.570+0000 7f4a888c3700 1 -- 172.21.15.42:0/383931379 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a8407f090 con 0x7f4a840a7730 2019-12-05T06:43:42.580 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:42.570+0000 7f4a888c3700 1 --2- 172.21.15.42:0/383931379 >> v2:172.21.15.40:3300/0 conn(0x7f4a840a7730 0x7f4a8408df70 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f4a78008790 tx=0x7f4a7800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:42.580 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:42.570+0000 7f4a888c3700 1 --2- 172.21.15.42:0/383931379 >> v2:172.21.15.40:3300/0 conn(0x7f4a840a7730 0x7f4a8408df70 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f4a78008790 tx=0x7f4a7800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:42.581 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:42.570+0000 7f4a888c3700 1 --2- 172.21.15.42:0/383931379 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4a840a7730 0x7f4a8408df70 secure :-1 s=READY pgs=2353 cs=0 l=1 rx=0x7f4a78008790 tx=0x7f4a7800fc70).ready entity=mon.0 client_cookie=b1cc4b076f754a3c server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:43:42.581 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:42.570+0000 7f4a888c3700 1 --2- 172.21.15.42:0/383931379 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4a840a7730 0x7f4a8408df70 secure :-1 s=THROTTLE_DONE pgs=2353 cs=0 l=1 rx=0x7f4a78008790 tx=0x7f4a7800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:42.581 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:42.570+0000 7f4a888c3700 1 --2- 172.21.15.42:0/383931379 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4a840a7730 0x7f4a8408df70 secure :-1 s=THROTTLE_DONE pgs=2353 cs=0 l=1 rx=0x7f4a78008790 tx=0x7f4a7800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:42.581 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:42.570+0000 7f4a837fe700 1 -- 172.21.15.42:0/383931379 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4a78008bb0 con 0x7f4a840a7730 2019-12-05T06:43:42.582 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:42.570+0000 7f4a888c3700 1 --2- 172.21.15.42:0/383931379 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4a840a7730 0x7f4a8408df70 secure :-1 s=THROTTLE_DONE pgs=2353 cs=0 l=1 rx=0x7f4a78008790 tx=0x7f4a7800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:42.582 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:42.570+0000 7f4a837fe700 1 -- 172.21.15.42:0/383931379 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a780087d0 con 0x7f4a840a7730 2019-12-05T06:43:42.582 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:42.570+0000 7f4a837fe700 1 -- 172.21.15.42:0/383931379 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4a78011950 con 0x7f4a840a7730 2019-12-05T06:43:42.582 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:42.570+0000 7f4a8a326700 1 -- 172.21.15.42:0/383931379 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4a840a7730 msgr2=0x7f4a8408df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:42.582 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:42.570+0000 7f4a8a326700 1 --2- 172.21.15.42:0/383931379 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4a840a7730 0x7f4a8408df70 secure :-1 s=READY pgs=2353 cs=0 l=1 rx=0x7f4a78008790 tx=0x7f4a7800fc70).stop 2019-12-05T06:43:42.582 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:42.570+0000 7f4a8a326700 1 -- 172.21.15.42:0/383931379 shutdown_connections 2019-12-05T06:43:42.583 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:42.570+0000 7f4a8a326700 1 --2- 172.21.15.42:0/383931379 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4a8409fad0 0x7f4a8407eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:43:42.583 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:42.570+0000 7f4a8a326700 1 --2- 172.21.15.42:0/383931379 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4a840a7730 0x7f4a8408df70 unknown :-1 s=CLOSED pgs=2353 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:42.583 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:42.570+0000 7f4a8a326700 1 -- 172.21.15.42:0/383931379 shutdown_connections 2019-12-05T06:43:42.583 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:42.570+0000 7f4a8a326700 1 -- 172.21.15.42:0/383931379 wait complete. 2019-12-05T06:43:42.584 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:42.570+0000 7f4a8a326700 1 -- 172.21.15.42:0/383931379 >> 172.21.15.42:0/383931379 conn(0x7f4a840aabd0 msgr2=0x7f4a840a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:43:42.587 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:42.574+0000 7f4a8a326700 1 Processor -- start 2019-12-05T06:43:42.588 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:42.574+0000 7f4a8a326700 1 -- start start 2019-12-05T06:43:42.588 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:42.574+0000 7f4a8a326700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4a840783e0 0x7f4a8411d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:43:42.588 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:42.574+0000 7f4a8a326700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a840776b0 con 0x7f4a84079460 2019-12-05T06:43:42.588 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:42.574+0000 7f4a8a326700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a84124400 con 0x7f4a8411d8b0 2019-12-05T06:43:42.589 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:42.574+0000 7f4a8a326700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a840a2680 con 0x7f4a840783e0 2019-12-05T06:43:42.589 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:42.574+0000 7f4a890c4700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f4a8411d8b0 0x7f4a8412e030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:34396/0 (socket says 172.21.15.42:34396) 2019-12-05T06:43:42.589 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:42.574+0000 7f4a890c4700 1 -- 172.21.15.42:0/508136662 learned_addr learned my addr 172.21.15.42:0/508136662 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:43:42.589 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:42.574+0000 7f4a888c3700 1 --2- 172.21.15.42:0/508136662 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4a840783e0 0x7f4a8411d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:42.589 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:42.574+0000 7f4a817fa700 1 -- 172.21.15.42:0/508136662 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3087046255 0 0) 0x7f4a84124400 con 0x7f4a8411d8b0 2019-12-05T06:43:42.589 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:42.574+0000 7f4a817fa700 1 -- 172.21.15.42:0/508136662 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a640030b0 con 0x7f4a8411d8b0 2019-12-05T06:43:42.590 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:42.574+0000 7f4a888c3700 1 -- 172.21.15.42:0/508136662 >> v1:172.21.15.42:6789/0 conn(0x7f4a8411d8b0 legacy=0x7f4a8412e030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:42.590 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:42.574+0000 7f4a888c3700 1 -- 172.21.15.42:0/508136662 >> v1:172.21.15.40:6789/0 conn(0x7f4a84079460 legacy=0x7f4a84123c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:42.590 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:42.574+0000 7f4a888c3700 1 -- 172.21.15.42:0/508136662 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a84078fe0 con 0x7f4a840783e0 2019-12-05T06:43:42.590 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:42.574+0000 7f4a888c3700 1 --2- 172.21.15.42:0/508136662 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4a840783e0 0x7f4a8411d360 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f4a78008790 tx=0x7f4a78011ed0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:42.590 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:42.574+0000 7f4a888c3700 1 --2- 172.21.15.42:0/508136662 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4a840783e0 0x7f4a8411d360 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f4a78008790 tx=0x7f4a78011ed0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:42.590 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:42.574+0000 7f4a888c3700 1 --2- 172.21.15.42:0/508136662 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4a840783e0 0x7f4a8411d360 secure :-1 s=READY pgs=677 cs=0 l=1 rx=0x7f4a78008790 tx=0x7f4a78011ed0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:43:42.590 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:42.574+0000 7f4a888c3700 1 --2- 172.21.15.42:0/508136662 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4a840783e0 0x7f4a8411d360 secure :-1 s=THROTTLE_DONE pgs=677 cs=0 l=1 rx=0x7f4a78008790 tx=0x7f4a78011ed0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:42.591 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:42.574+0000 7f4a817fa700 1 -- 172.21.15.42:0/508136662 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4a7800f280 con 0x7f4a840783e0 2019-12-05T06:43:42.591 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:42.574+0000 7f4a8a326700 1 -- 172.21.15.42:0/508136662 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a8412f840 con 0x7f4a840783e0 2019-12-05T06:43:42.591 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:42.574+0000 7f4a888c3700 1 --2- 172.21.15.42:0/508136662 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4a840783e0 0x7f4a8411d360 secure :-1 s=THROTTLE_DONE pgs=677 cs=0 l=1 rx=0x7f4a78008790 tx=0x7f4a78011ed0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:42.591 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:42.574+0000 7f4a888c3700 1 --2- 172.21.15.42:0/508136662 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4a840783e0 0x7f4a8411d360 secure :-1 s=THROTTLE_DONE pgs=677 cs=0 l=1 rx=0x7f4a78008790 tx=0x7f4a78011ed0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:42.591 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:42.578+0000 7f4a817fa700 1 -- 172.21.15.42:0/508136662 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a78008db0 con 0x7f4a840783e0 2019-12-05T06:43:42.591 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:42.578+0000 7f4a8a326700 1 -- 172.21.15.42:0/508136662 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a8412fd10 con 0x7f4a840783e0 2019-12-05T06:43:42.592 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:42.578+0000 7f4a817fa700 1 -- 172.21.15.42:0/508136662 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4a78005540 con 0x7f4a840783e0 2019-12-05T06:43:42.592 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:42.578+0000 7f4a888c3700 1 --2- 172.21.15.42:0/508136662 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4a840783e0 0x7f4a8411d360 secure :-1 s=THROTTLE_DONE pgs=677 cs=0 l=1 rx=0x7f4a78008790 tx=0x7f4a78011ed0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:42.592 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:42.578+0000 7f4a817fa700 1 -- 172.21.15.42:0/508136662 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f4a78005850 con 0x7f4a840783e0 2019-12-05T06:43:42.592 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:42.578+0000 7f4a888c3700 1 --2- 172.21.15.42:0/508136662 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4a840783e0 0x7f4a8411d360 secure :-1 s=THROTTLE_DONE pgs=677 cs=0 l=1 rx=0x7f4a78008790 tx=0x7f4a78011ed0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:42.593 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:42.578+0000 7f4a817fa700 1 --2- 172.21.15.42:0/508136662 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4a6403b250 0x7f4a6403d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:43:42.593 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:42.578+0000 7f4a817fa700 1 -- 172.21.15.42:0/508136662 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 11879+0+0 (secure 0 0 0) 0x7f4a7804b6e0 con 0x7f4a840783e0 2019-12-05T06:43:42.593 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:42.578+0000 7f4a83fff700 1 --2- 172.21.15.42:0/508136662 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4a6403b250 0x7f4a6403d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:42.593 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:42.578+0000 7f4a83fff700 1 --2- 172.21.15.42:0/508136662 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4a6403b250 0x7f4a6403d760 crc :-1 s=READY pgs=1302 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:43:42.595 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:42.586+0000 7f4a8a326700 1 -- 172.21.15.42:0/508136662 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a5c003200 con 0x7f4a840783e0 2019-12-05T06:43:42.595 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:42.590+0000 7f4a888c3700 1 --2- 172.21.15.42:0/508136662 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4a840783e0 0x7f4a8411d360 secure :-1 s=THROTTLE_DONE pgs=677 cs=0 l=1 rx=0x7f4a78008790 tx=0x7f4a78011ed0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:42.596 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:42.590+0000 7f4a817fa700 1 -- 172.21.15.42:0/508136662 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f4a780329c0 con 0x7f4a840783e0 2019-12-05T06:43:42.784 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:42.778+0000 7f4a8a326700 1 -- 172.21.15.42:0/508136662 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4a5c003630 con 0x7f4a840783e0 2019-12-05T06:43:42.785 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:42.782+0000 7f4a888c3700 1 --2- 172.21.15.42:0/508136662 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4a840783e0 0x7f4a8411d360 secure :-1 s=THROTTLE_DONE pgs=677 cs=0 l=1 rx=0x7f4a78008790 tx=0x7f4a78011ed0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:42.785 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:42.782+0000 7f4a817fa700 1 -- 172.21.15.42:0/508136662 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 376 v376) v1 ==== 95+0+1770 (secure 0 0 0) 0x7f4a78005b60 con 0x7f4a840783e0 2019-12-05T06:43:42.785 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 376 2019-12-05T06:43:42.786 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:43:42.786 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":376,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":376,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:43:33.543372+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":117,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12496},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12496":{"gid":12496,"name":"b","rank":0,"incarnation":373,"state":"up:active","state_seq":11,"addr":"172.21.15.40:6833/3599619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3599619},{"type":"v1","addr":"172.21.15.40:6833","nonce":3599619}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:43:42.791 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:42.782+0000 7f4a8a326700 1 -- 172.21.15.42:0/508136662 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4a6403b250 msgr2=0x7f4a6403d760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:42.791 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:42.782+0000 7f4a8a326700 1 --2- 172.21.15.42:0/508136662 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4a6403b250 0x7f4a6403d760 crc :-1 s=READY pgs=1302 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:42.793 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:42.782+0000 7f4a8a326700 1 -- 172.21.15.42:0/508136662 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4a840783e0 msgr2=0x7f4a8411d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:42.793 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:42.782+0000 7f4a8a326700 1 --2- 172.21.15.42:0/508136662 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4a840783e0 0x7f4a8411d360 secure :-1 s=READY pgs=677 cs=0 l=1 rx=0x7f4a78008790 tx=0x7f4a78011ed0).stop 2019-12-05T06:43:42.793 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:42.782+0000 7f4a8a326700 1 -- 172.21.15.42:0/508136662 shutdown_connections 2019-12-05T06:43:42.793 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:42.782+0000 7f4a8a326700 1 --2- 172.21.15.42:0/508136662 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4a6403b250 0x7f4a6403d760 unknown :-1 s=CLOSED pgs=1302 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:42.793 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:42.782+0000 7f4a8a326700 1 --2- 172.21.15.42:0/508136662 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4a840783e0 0x7f4a8411d360 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:42.793 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:42.782+0000 7f4a8a326700 1 -- 172.21.15.42:0/508136662 shutdown_connections 2019-12-05T06:43:42.793 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:42.782+0000 7f4a8a326700 1 -- 172.21.15.42:0/508136662 wait complete. 2019-12-05T06:43:42.794 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:42.782+0000 7f4a8a326700 1 -- 172.21.15.42:0/508136662 >> 172.21.15.42:0/508136662 conn(0x7f4a8409f030 msgr2=0x7f4a840a9780 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:43:42.803 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2019-12-05T06:43:44.804 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:43:44.910 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:44.898+0000 7f9a48596700 1 Processor -- start 2019-12-05T06:43:44.910 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:44.898+0000 7f9a48596700 1 -- start start 2019-12-05T06:43:44.910 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:44.898+0000 7f9a48596700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f9a400a7740 0x7f9a4008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:43:44.911 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:44.898+0000 7f9a48596700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f9a4007e720 0x7f9a4007eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:43:44.911 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:44.898+0000 7f9a48596700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9a4009f2e0 0x7f9a4009f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:43:44.911 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:44.898+0000 7f9a48596700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a400a0b00 con 0x7f9a400a7740 2019-12-05T06:43:44.911 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:44.898+0000 7f9a48596700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a4008e5b0 con 0x7f9a4007e720 2019-12-05T06:43:44.912 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:44.898+0000 7f9a48596700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a4008e850 con 0x7f9a4009f2e0 2019-12-05T06:43:44.912 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:44.898+0000 7f9a46b33700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f9a400a7740 0x7f9a4008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:44.913 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:44.898+0000 7f9a46b33700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f9a400a7740 0x7f9a4008df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.42:50446/0 (socket says 172.21.15.42:50446) 2019-12-05T06:43:44.913 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:44.898+0000 7f9a46b33700 1 -- 172.21.15.42:0/3514715672 learned_addr learned my addr 172.21.15.42:0/3514715672 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:43:44.914 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:44.898+0000 7f9a46b33700 1 -- 172.21.15.42:0/3514715672 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9a4009f2e0 msgr2=0x7f9a4009f710 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2019-12-05T06:43:44.914 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:44.898+0000 7f9a46b33700 1 --2- 172.21.15.42:0/3514715672 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9a4009f2e0 0x7f9a4009f710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:43:44.914 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:44.898+0000 7f9a46b33700 1 -- 172.21.15.42:0/3514715672 >> v2:172.21.15.42:3300/0 conn(0x7f9a4007e720 msgr2=0x7f9a4007eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:43:44.914 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:44.898+0000 7f9a46b33700 1 --2- 172.21.15.42:0/3514715672 >> v2:172.21.15.42:3300/0 conn(0x7f9a4007e720 0x7f9a4007eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:43:44.915 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:44.898+0000 7f9a46b33700 1 -- 172.21.15.42:0/3514715672 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a4007f0a0 con 0x7f9a400a7740 2019-12-05T06:43:44.915 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:44.898+0000 7f9a46b33700 1 --2- 172.21.15.42:0/3514715672 >> v2:172.21.15.40:3300/0 conn(0x7f9a400a7740 0x7f9a4008df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f9a400a0f50 tx=0x7f9a38010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:44.916 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:44.898+0000 7f9a46b33700 1 --2- 172.21.15.42:0/3514715672 >> v2:172.21.15.40:3300/0 conn(0x7f9a400a7740 0x7f9a4008df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f9a400a0f50 tx=0x7f9a38010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:44.916 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:44.898+0000 7f9a46b33700 1 --2- 172.21.15.42:0/3514715672 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9a400a7740 0x7f9a4008df80 secure :-1 s=READY pgs=2357 cs=0 l=1 rx=0x7f9a400a0f50 tx=0x7f9a38010130).ready entity=mon.0 client_cookie=d86b8e6998fa957a server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:43:44.916 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:44.902+0000 7f9a46b33700 1 --2- 172.21.15.42:0/3514715672 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9a400a7740 0x7f9a4008df80 secure :-1 s=THROTTLE_DONE pgs=2357 cs=0 l=1 rx=0x7f9a400a0f50 tx=0x7f9a38010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:44.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:44.902+0000 7f9a46b33700 1 --2- 172.21.15.42:0/3514715672 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9a400a7740 0x7f9a4008df80 secure :-1 s=THROTTLE_DONE pgs=2357 cs=0 l=1 rx=0x7f9a400a0f50 tx=0x7f9a38010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:44.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:44.902+0000 7f9a45b31700 1 -- 172.21.15.42:0/3514715672 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9a380110c0 con 0x7f9a400a7740 2019-12-05T06:43:44.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:44.902+0000 7f9a46b33700 1 --2- 172.21.15.42:0/3514715672 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9a400a7740 0x7f9a4008df80 secure :-1 s=THROTTLE_DONE pgs=2357 cs=0 l=1 rx=0x7f9a400a0f50 tx=0x7f9a38010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:44.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:44.902+0000 7f9a45b31700 1 -- 172.21.15.42:0/3514715672 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a38010af0 con 0x7f9a400a7740 2019-12-05T06:43:44.918 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:44.902+0000 7f9a48596700 1 -- 172.21.15.42:0/3514715672 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9a400a7740 msgr2=0x7f9a4008df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:44.918 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:44.902+0000 7f9a48596700 1 --2- 172.21.15.42:0/3514715672 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9a400a7740 0x7f9a4008df80 secure :-1 s=READY pgs=2357 cs=0 l=1 rx=0x7f9a400a0f50 tx=0x7f9a38010130).stop 2019-12-05T06:43:44.918 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:44.902+0000 7f9a45b31700 1 -- 172.21.15.42:0/3514715672 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9a380116a0 con 0x7f9a400a7740 2019-12-05T06:43:44.918 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:44.902+0000 7f9a48596700 1 -- 172.21.15.42:0/3514715672 shutdown_connections 2019-12-05T06:43:44.919 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:44.902+0000 7f9a48596700 1 --2- 172.21.15.42:0/3514715672 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9a4009f2e0 0x7f9a4009f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:43:44.919 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:44.902+0000 7f9a48596700 1 --2- 172.21.15.42:0/3514715672 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9a400a7740 0x7f9a4008df80 unknown :-1 s=CLOSED pgs=2357 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:44.919 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:44.902+0000 7f9a48596700 1 --2- 172.21.15.42:0/3514715672 >> v2:172.21.15.42:3300/0 conn(0x7f9a4007e720 0x7f9a4007eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:43:44.919 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:44.902+0000 7f9a48596700 1 -- 172.21.15.42:0/3514715672 shutdown_connections 2019-12-05T06:43:44.920 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:44.902+0000 7f9a48596700 1 -- 172.21.15.42:0/3514715672 wait complete. 2019-12-05T06:43:44.920 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:44.902+0000 7f9a48596700 1 -- 172.21.15.42:0/3514715672 >> 172.21.15.42:0/3514715672 conn(0x7f9a400aabe0 msgr2=0x7f9a400a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:43:44.921 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:44.902+0000 7f9a48596700 1 Processor -- start 2019-12-05T06:43:44.921 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:44.902+0000 7f9a48596700 1 -- start start 2019-12-05T06:43:44.921 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:44.902+0000 7f9a48596700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f9a4011e9a0 0x7f9a400a52c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:43:44.921 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:44.902+0000 7f9a48596700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f9a4012a810 0x7f9a4012ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:43:44.922 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:44.902+0000 7f9a48596700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a400a2c40 con 0x7f9a400a5810 2019-12-05T06:43:44.922 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:44.902+0000 7f9a48596700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a400a5cf0 con 0x7f9a4012a810 2019-12-05T06:43:44.922 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:44.902+0000 7f9a48596700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a4012a570 con 0x7f9a4011e9a0 2019-12-05T06:43:44.922 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:44.902+0000 7f9a46b33700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f9a4011e9a0 0x7f9a400a52c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:44.923 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:44.902+0000 7f9a46b33700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f9a4011e9a0 0x7f9a400a52c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:34402/0 (socket says 172.21.15.42:34402) 2019-12-05T06:43:44.926 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:44.902+0000 7f9a46b33700 1 -- 172.21.15.42:0/3073935784 learned_addr learned my addr 172.21.15.42:0/3073935784 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:43:44.927 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:44.902+0000 7f9a47334700 1 --2- 172.21.15.42:0/3073935784 >> v2:172.21.15.40:3300/0 conn(0x7f9a4012a810 0x7f9a4012ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:44.927 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:44.906+0000 7f9a46b33700 1 -- 172.21.15.42:0/3073935784 >> v2:172.21.15.40:3300/0 conn(0x7f9a4012a810 msgr2=0x7f9a4012ac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:44.927 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:44.906+0000 7f9a46b33700 1 --2- 172.21.15.42:0/3073935784 >> v2:172.21.15.40:3300/0 conn(0x7f9a4012a810 0x7f9a4012ac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:44.928 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:44.906+0000 7f9a46b33700 1 -- 172.21.15.42:0/3073935784 >> v1:172.21.15.42:6789/0 conn(0x7f9a400a5810 legacy=0x7f9a4007d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:44.928 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:44.906+0000 7f9a46b33700 1 -- 172.21.15.42:0/3073935784 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a4007e080 con 0x7f9a4011e9a0 2019-12-05T06:43:44.928 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:44.906+0000 7f9a46b33700 1 --2- 172.21.15.42:0/3073935784 >> v2:172.21.15.42:3300/0 conn(0x7f9a4011e9a0 0x7f9a400a52c0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f9a4008e500 tx=0x7f9a38010da0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:44.928 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:44.906+0000 7f9a46b33700 1 --2- 172.21.15.42:0/3073935784 >> v2:172.21.15.42:3300/0 conn(0x7f9a4011e9a0 0x7f9a400a52c0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f9a4008e500 tx=0x7f9a38010da0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:44.929 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:44.906+0000 7f9a46b33700 1 --2- 172.21.15.42:0/3073935784 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9a4011e9a0 0x7f9a400a52c0 secure :-1 s=READY pgs=3124 cs=0 l=1 rx=0x7f9a4008e500 tx=0x7f9a38010da0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:43:44.929 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:44.906+0000 7f9a46b33700 1 --2- 172.21.15.42:0/3073935784 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9a4011e9a0 0x7f9a400a52c0 secure :-1 s=THROTTLE_DONE pgs=3124 cs=0 l=1 rx=0x7f9a4008e500 tx=0x7f9a38010da0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:44.929 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:44.906+0000 7f9a46b33700 1 --2- 172.21.15.42:0/3073935784 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9a4011e9a0 0x7f9a400a52c0 secure :-1 s=THROTTLE_DONE pgs=3124 cs=0 l=1 rx=0x7f9a4008e500 tx=0x7f9a38010da0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:44.929 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:44.906+0000 7f9a46b33700 1 --2- 172.21.15.42:0/3073935784 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9a4011e9a0 0x7f9a400a52c0 secure :-1 s=THROTTLE_DONE pgs=3124 cs=0 l=1 rx=0x7f9a4008e500 tx=0x7f9a38010da0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:44.930 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:44.906+0000 7f9a377fe700 1 -- 172.21.15.42:0/3073935784 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9a38011df0 con 0x7f9a4011e9a0 2019-12-05T06:43:44.930 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:44.906+0000 7f9a377fe700 1 -- 172.21.15.42:0/3073935784 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a380112c0 con 0x7f9a4011e9a0 2019-12-05T06:43:44.930 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:44.906+0000 7f9a377fe700 1 -- 172.21.15.42:0/3073935784 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9a38005ca0 con 0x7f9a4011e9a0 2019-12-05T06:43:44.930 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:44.906+0000 7f9a48596700 1 -- 172.21.15.42:0/3073935784 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a4012b190 con 0x7f9a4011e9a0 2019-12-05T06:43:44.931 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:44.906+0000 7f9a48596700 1 -- 172.21.15.42:0/3073935784 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a40131a60 con 0x7f9a4011e9a0 2019-12-05T06:43:44.931 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:44.906+0000 7f9a46b33700 1 --2- 172.21.15.42:0/3073935784 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9a4011e9a0 0x7f9a400a52c0 secure :-1 s=THROTTLE_DONE pgs=3124 cs=0 l=1 rx=0x7f9a4008e500 tx=0x7f9a38010da0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:44.933 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:44.906+0000 7f9a377fe700 1 -- 172.21.15.42:0/3073935784 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f9a3800f280 con 0x7f9a4011e9a0 2019-12-05T06:43:44.933 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:44.906+0000 7f9a46b33700 1 --2- 172.21.15.42:0/3073935784 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9a4011e9a0 0x7f9a400a52c0 secure :-1 s=THROTTLE_DONE pgs=3124 cs=0 l=1 rx=0x7f9a4008e500 tx=0x7f9a38010da0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:44.934 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:44.906+0000 7f9a377fe700 1 --2- 172.21.15.42:0/3073935784 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9a2803ae00 0x7f9a2803d310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:43:44.934 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:44.906+0000 7f9a377fe700 1 -- 172.21.15.42:0/3073935784 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 11879+0+0 (secure 0 0 0) 0x7f9a3804b800 con 0x7f9a4011e9a0 2019-12-05T06:43:44.934 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:44.906+0000 7f9a46332700 1 --2- 172.21.15.42:0/3073935784 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9a2803ae00 0x7f9a2803d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:44.935 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:44.906+0000 7f9a46332700 1 --2- 172.21.15.42:0/3073935784 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9a2803ae00 0x7f9a2803d310 crc :-1 s=READY pgs=1304 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:43:44.935 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:44.914+0000 7f9a48596700 1 -- 172.21.15.42:0/3073935784 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a400a5eb0 con 0x7f9a4011e9a0 2019-12-05T06:43:44.935 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:44.918+0000 7f9a46b33700 1 --2- 172.21.15.42:0/3073935784 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9a4011e9a0 0x7f9a400a52c0 secure :-1 s=THROTTLE_DONE pgs=3124 cs=0 l=1 rx=0x7f9a4008e500 tx=0x7f9a38010da0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:44.936 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:44.918+0000 7f9a377fe700 1 -- 172.21.15.42:0/3073935784 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f9a38032880 con 0x7f9a4011e9a0 2019-12-05T06:43:45.084 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:45.078+0000 7f9a48596700 1 -- 172.21.15.42:0/3073935784 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9a4005d110 con 0x7f9a4011e9a0 2019-12-05T06:43:45.089 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:45.078+0000 7f9a46b33700 1 --2- 172.21.15.42:0/3073935784 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9a4011e9a0 0x7f9a400a52c0 secure :-1 s=THROTTLE_DONE pgs=3124 cs=0 l=1 rx=0x7f9a4008e500 tx=0x7f9a38010da0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:45.089 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:45.082+0000 7f9a377fe700 1 -- 172.21.15.42:0/3073935784 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 377 v377) v1 ==== 95+0+2108 (secure 0 0 0) 0x7f9a38033390 con 0x7f9a4011e9a0 2019-12-05T06:43:45.089 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 377 2019-12-05T06:43:45.089 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:45.082+0000 7f9a48596700 1 -- 172.21.15.42:0/3073935784 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9a2803ae00 msgr2=0x7f9a2803d310 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:45.089 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:45.082+0000 7f9a48596700 1 --2- 172.21.15.42:0/3073935784 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9a2803ae00 0x7f9a2803d310 crc :-1 s=READY pgs=1304 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:45.089 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:45.082+0000 7f9a48596700 1 -- 172.21.15.42:0/3073935784 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9a4011e9a0 msgr2=0x7f9a400a52c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:45.090 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:45.082+0000 7f9a48596700 1 --2- 172.21.15.42:0/3073935784 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9a4011e9a0 0x7f9a400a52c0 secure :-1 s=READY pgs=3124 cs=0 l=1 rx=0x7f9a4008e500 tx=0x7f9a38010da0).stop 2019-12-05T06:43:45.090 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:45.082+0000 7f9a48596700 1 -- 172.21.15.42:0/3073935784 shutdown_connections 2019-12-05T06:43:45.090 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:45.082+0000 7f9a48596700 1 --2- 172.21.15.42:0/3073935784 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9a2803ae00 0x7f9a2803d310 unknown :-1 s=CLOSED pgs=1304 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:45.090 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:45.082+0000 7f9a48596700 1 --2- 172.21.15.42:0/3073935784 >> v2:172.21.15.40:3300/0 conn(0x7f9a4012a810 0x7f9a4012ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:45.090 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:45.082+0000 7f9a48596700 1 --2- 172.21.15.42:0/3073935784 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9a4011e9a0 0x7f9a400a52c0 unknown :-1 s=CLOSED pgs=3124 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:45.090 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:45.082+0000 7f9a48596700 1 -- 172.21.15.42:0/3073935784 shutdown_connections 2019-12-05T06:43:45.091 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:45.082+0000 7f9a48596700 1 -- 172.21.15.42:0/3073935784 wait complete. 2019-12-05T06:43:45.091 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:43:45.082+0000 7f9a48596700 1 -- 172.21.15.42:0/3073935784 >> 172.21.15.42:0/3073935784 conn(0x7f9a400a81d0 msgr2=0x7f9a4008eb70 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:43:45.091 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:43:45.091 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":377,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":12303,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/2489540211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2489540211},{"type":"v1","addr":"172.21.15.42:6835","nonce":2489540211}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":377}],"filesystems":[{"mdsmap":{"epoch":376,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:43:33.543372+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":117,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12496},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12496":{"gid":12496,"name":"b","rank":0,"incarnation":373,"state":"up:active","state_seq":11,"addr":"172.21.15.40:6833/3599619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3599619},{"type":"v1","addr":"172.21.15.40:6833","nonce":3599619}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:43:45.098 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a reported in up:standby state 2019-12-05T06:43:45.098 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 27 secs before thrashing 2019-12-05T06:43:45.560 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:43:45.573 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:44:12.099 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:44:12.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.182+0000 7f7c98013700 1 Processor -- start 2019-12-05T06:44:12.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.182+0000 7f7c98013700 1 -- start start 2019-12-05T06:44:12.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.182+0000 7f7c98013700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f7c900a7740 0x7f7c9008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:44:12.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.182+0000 7f7c98013700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7c9007e720 0x7f7c9007eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:44:12.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.182+0000 7f7c98013700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c900a0710 con 0x7f7c9009f2e0 2019-12-05T06:44:12.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.182+0000 7f7c98013700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c900a0b00 con 0x7f7c900a7740 2019-12-05T06:44:12.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.182+0000 7f7c98013700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c9008e5b0 con 0x7f7c9007e720 2019-12-05T06:44:12.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.182+0000 7f7c965b0700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f7c900a7740 0x7f7c9008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:12.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.182+0000 7f7c965b0700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f7c900a7740 0x7f7c9008df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:34574/0 (socket says 172.21.15.42:34574) 2019-12-05T06:44:12.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.182+0000 7f7c965b0700 1 -- 172.21.15.42:0/440511468 learned_addr learned my addr 172.21.15.42:0/440511468 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:44:12.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.182+0000 7f7c95daf700 1 --2- 172.21.15.42:0/440511468 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7c9007e720 0x7f7c9007eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:12.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.182+0000 7f7c965b0700 1 -- 172.21.15.42:0/440511468 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7c9007e720 msgr2=0x7f7c9007eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:44:12.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.182+0000 7f7c965b0700 1 --2- 172.21.15.42:0/440511468 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7c9007e720 0x7f7c9007eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:44:12.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.182+0000 7f7c965b0700 1 -- 172.21.15.42:0/440511468 >> v1:172.21.15.40:6789/0 conn(0x7f7c9009f2e0 legacy=0x7f7c9009f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:12.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.182+0000 7f7c965b0700 1 -- 172.21.15.42:0/440511468 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c9007f0a0 con 0x7f7c900a7740 2019-12-05T06:44:12.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.182+0000 7f7c965b0700 1 --2- 172.21.15.42:0/440511468 >> v2:172.21.15.42:3300/0 conn(0x7f7c900a7740 0x7f7c9008df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f7c88008b10 tx=0x7f7c880100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:12.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.182+0000 7f7c965b0700 1 --2- 172.21.15.42:0/440511468 >> v2:172.21.15.42:3300/0 conn(0x7f7c900a7740 0x7f7c9008df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f7c88008b10 tx=0x7f7c880100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:12.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.182+0000 7f7c965b0700 1 --2- 172.21.15.42:0/440511468 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7c900a7740 0x7f7c9008df80 secure :-1 s=READY pgs=3126 cs=0 l=1 rx=0x7f7c88008b10 tx=0x7f7c880100b0).ready entity=mon.1 client_cookie=60ebe9f4bcebed3 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:44:12.193 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.182+0000 7f7c965b0700 1 --2- 172.21.15.42:0/440511468 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7c900a7740 0x7f7c9008df80 secure :-1 s=THROTTLE_DONE pgs=3126 cs=0 l=1 rx=0x7f7c88008b10 tx=0x7f7c880100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:12.193 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.182+0000 7f7c955ae700 1 -- 172.21.15.42:0/440511468 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7c8800a160 con 0x7f7c900a7740 2019-12-05T06:44:12.193 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.182+0000 7f7c965b0700 1 --2- 172.21.15.42:0/440511468 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7c900a7740 0x7f7c9008df80 secure :-1 s=THROTTLE_DONE pgs=3126 cs=0 l=1 rx=0x7f7c88008b10 tx=0x7f7c880100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:12.193 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.182+0000 7f7c965b0700 1 --2- 172.21.15.42:0/440511468 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7c900a7740 0x7f7c9008df80 secure :-1 s=THROTTLE_DONE pgs=3126 cs=0 l=1 rx=0x7f7c88008b10 tx=0x7f7c880100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:12.194 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.182+0000 7f7c955ae700 1 -- 172.21.15.42:0/440511468 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c88011a30 con 0x7f7c900a7740 2019-12-05T06:44:12.194 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.182+0000 7f7c955ae700 1 -- 172.21.15.42:0/440511468 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7c8800a550 con 0x7f7c900a7740 2019-12-05T06:44:12.194 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.182+0000 7f7c98013700 1 -- 172.21.15.42:0/440511468 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7c900a7740 msgr2=0x7f7c9008df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:12.194 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.182+0000 7f7c98013700 1 --2- 172.21.15.42:0/440511468 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7c900a7740 0x7f7c9008df80 secure :-1 s=READY pgs=3126 cs=0 l=1 rx=0x7f7c88008b10 tx=0x7f7c880100b0).stop 2019-12-05T06:44:12.194 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.186+0000 7f7c98013700 1 -- 172.21.15.42:0/440511468 shutdown_connections 2019-12-05T06:44:12.195 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.186+0000 7f7c98013700 1 --2- 172.21.15.42:0/440511468 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7c900a7740 0x7f7c9008df80 unknown :-1 s=CLOSED pgs=3126 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:12.195 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.186+0000 7f7c98013700 1 --2- 172.21.15.42:0/440511468 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7c9007e720 0x7f7c9007eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:44:12.195 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.186+0000 7f7c98013700 1 -- 172.21.15.42:0/440511468 shutdown_connections 2019-12-05T06:44:12.195 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.186+0000 7f7c98013700 1 -- 172.21.15.42:0/440511468 wait complete. 2019-12-05T06:44:12.195 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.186+0000 7f7c98013700 1 -- 172.21.15.42:0/440511468 >> 172.21.15.42:0/440511468 conn(0x7f7c900aabe0 msgr2=0x7f7c900a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:44:12.199 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.186+0000 7f7c98013700 1 Processor -- start 2019-12-05T06:44:12.199 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.186+0000 7f7c98013700 1 -- start start 2019-12-05T06:44:12.200 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.190+0000 7f7c98013700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7c901429f0 0x7f7c901236c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:44:12.200 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.190+0000 7f7c98013700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f7c900a2fc0 0x7f7c900a33f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:44:12.200 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.190+0000 7f7c98013700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c9007c100 con 0x7f7c90123c10 2019-12-05T06:44:12.200 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.190+0000 7f7c98013700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c90123420 con 0x7f7c901429f0 2019-12-05T06:44:12.200 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.190+0000 7f7c98013700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c900a3940 con 0x7f7c900a2fc0 2019-12-05T06:44:12.201 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.190+0000 7f7c95daf700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f7c90123c10 0x7f7c9007b900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:34620/0 (socket says 172.21.15.42:34620) 2019-12-05T06:44:12.201 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.190+0000 7f7c95daf700 1 -- 172.21.15.42:0/2427241838 learned_addr learned my addr 172.21.15.42:0/2427241838 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:44:12.201 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.190+0000 7f7c965b0700 1 --2- 172.21.15.42:0/2427241838 >> v2:172.21.15.40:3300/0 conn(0x7f7c901429f0 0x7f7c901236c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:12.201 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.190+0000 7f7c96db1700 1 --2- 172.21.15.42:0/2427241838 >> v2:172.21.15.42:3300/0 conn(0x7f7c900a2fc0 0x7f7c900a33f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:12.201 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.190+0000 7f7c8effd700 1 -- 172.21.15.42:0/2427241838 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2138766757 0 0) 0x7f7c9007c100 con 0x7f7c90123c10 2019-12-05T06:44:12.201 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.190+0000 7f7c8effd700 1 -- 172.21.15.42:0/2427241838 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c700030b0 con 0x7f7c90123c10 2019-12-05T06:44:12.201 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.190+0000 7f7c965b0700 1 -- 172.21.15.42:0/2427241838 >> v2:172.21.15.42:3300/0 conn(0x7f7c900a2fc0 msgr2=0x7f7c900a33f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:12.202 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.190+0000 7f7c965b0700 1 --2- 172.21.15.42:0/2427241838 >> v2:172.21.15.42:3300/0 conn(0x7f7c900a2fc0 0x7f7c900a33f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:12.202 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.190+0000 7f7c965b0700 1 -- 172.21.15.42:0/2427241838 >> v1:172.21.15.42:6789/0 conn(0x7f7c90123c10 legacy=0x7f7c9007b900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:12.202 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.190+0000 7f7c965b0700 1 -- 172.21.15.42:0/2427241838 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c90123000 con 0x7f7c901429f0 2019-12-05T06:44:12.202 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.190+0000 7f7c965b0700 1 --2- 172.21.15.42:0/2427241838 >> v2:172.21.15.40:3300/0 conn(0x7f7c901429f0 0x7f7c901236c0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f7c90143320 tx=0x7f7c88011ce0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:12.202 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.190+0000 7f7c965b0700 1 --2- 172.21.15.42:0/2427241838 >> v2:172.21.15.40:3300/0 conn(0x7f7c901429f0 0x7f7c901236c0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f7c90143320 tx=0x7f7c88011ce0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:12.202 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.190+0000 7f7c965b0700 1 --2- 172.21.15.42:0/2427241838 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7c901429f0 0x7f7c901236c0 secure :-1 s=READY pgs=2359 cs=0 l=1 rx=0x7f7c90143320 tx=0x7f7c88011ce0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:44:12.203 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.190+0000 7f7c965b0700 1 --2- 172.21.15.42:0/2427241838 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7c901429f0 0x7f7c901236c0 secure :-1 s=THROTTLE_DONE pgs=2359 cs=0 l=1 rx=0x7f7c90143320 tx=0x7f7c88011ce0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:12.203 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.190+0000 7f7c8effd700 1 -- 172.21.15.42:0/2427241838 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7c8800ace0 con 0x7f7c901429f0 2019-12-05T06:44:12.203 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.190+0000 7f7c98013700 1 -- 172.21.15.42:0/2427241838 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c90143d40 con 0x7f7c901429f0 2019-12-05T06:44:12.203 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.190+0000 7f7c965b0700 1 --2- 172.21.15.42:0/2427241838 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7c901429f0 0x7f7c901236c0 secure :-1 s=THROTTLE_DONE pgs=2359 cs=0 l=1 rx=0x7f7c90143320 tx=0x7f7c88011ce0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:12.203 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.190+0000 7f7c8effd700 1 -- 172.21.15.42:0/2427241838 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c88011e10 con 0x7f7c901429f0 2019-12-05T06:44:12.203 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.190+0000 7f7c965b0700 1 --2- 172.21.15.42:0/2427241838 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7c901429f0 0x7f7c901236c0 secure :-1 s=THROTTLE_DONE pgs=2359 cs=0 l=1 rx=0x7f7c90143320 tx=0x7f7c88011ce0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:12.204 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.190+0000 7f7c98013700 1 -- 172.21.15.42:0/2427241838 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c900a9860 con 0x7f7c901429f0 2019-12-05T06:44:12.204 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.190+0000 7f7c8effd700 1 -- 172.21.15.42:0/2427241838 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7c88005ca0 con 0x7f7c901429f0 2019-12-05T06:44:12.204 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.190+0000 7f7c965b0700 1 --2- 172.21.15.42:0/2427241838 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7c901429f0 0x7f7c901236c0 secure :-1 s=THROTTLE_DONE pgs=2359 cs=0 l=1 rx=0x7f7c90143320 tx=0x7f7c88011ce0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:12.204 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.194+0000 7f7c8effd700 1 -- 172.21.15.42:0/2427241838 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f7c88007160 con 0x7f7c901429f0 2019-12-05T06:44:12.204 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.194+0000 7f7c965b0700 1 --2- 172.21.15.42:0/2427241838 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7c901429f0 0x7f7c901236c0 secure :-1 s=THROTTLE_DONE pgs=2359 cs=0 l=1 rx=0x7f7c90143320 tx=0x7f7c88011ce0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:12.205 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.194+0000 7f7c8effd700 1 --2- 172.21.15.42:0/2427241838 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f7c7003b270 0x7f7c7003d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:44:12.205 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.194+0000 7f7c8effd700 1 -- 172.21.15.42:0/2427241838 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 11879+0+0 (secure 0 0 0) 0x7f7c8804b690 con 0x7f7c901429f0 2019-12-05T06:44:12.205 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.194+0000 7f7c95daf700 1 --2- 172.21.15.42:0/2427241838 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f7c7003b270 0x7f7c7003d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:12.205 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.194+0000 7f7c95daf700 1 --2- 172.21.15.42:0/2427241838 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f7c7003b270 0x7f7c7003d780 crc :-1 s=READY pgs=1305 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:44:12.205 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.198+0000 7f7c98013700 1 -- 172.21.15.42:0/2427241838 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c9005d110 con 0x7f7c901429f0 2019-12-05T06:44:12.209 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.206+0000 7f7c965b0700 1 --2- 172.21.15.42:0/2427241838 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7c901429f0 0x7f7c901236c0 secure :-1 s=THROTTLE_DONE pgs=2359 cs=0 l=1 rx=0x7f7c90143320 tx=0x7f7c88011ce0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:12.209 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.206+0000 7f7c8effd700 1 -- 172.21.15.42:0/2427241838 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f7c880326f0 con 0x7f7c901429f0 2019-12-05T06:44:12.380 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.378+0000 7f7c98013700 1 -- 172.21.15.42:0/2427241838 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7c9005b8c0 con 0x7f7c901429f0 2019-12-05T06:44:12.381 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.378+0000 7f7c965b0700 1 --2- 172.21.15.42:0/2427241838 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7c901429f0 0x7f7c901236c0 secure :-1 s=THROTTLE_DONE pgs=2359 cs=0 l=1 rx=0x7f7c90143320 tx=0x7f7c88011ce0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:12.381 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.378+0000 7f7c8effd700 1 -- 172.21.15.42:0/2427241838 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 377 v377) v1 ==== 95+0+2108 (secure 0 0 0) 0x7f7c88032950 con 0x7f7c901429f0 2019-12-05T06:44:12.383 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:44:12.383 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":377,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":12303,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/2489540211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2489540211},{"type":"v1","addr":"172.21.15.42:6835","nonce":2489540211}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":377}],"filesystems":[{"mdsmap":{"epoch":376,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:43:33.543372+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":117,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12496},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12496":{"gid":12496,"name":"b","rank":0,"incarnation":373,"state":"up:active","state_seq":11,"addr":"172.21.15.40:6833/3599619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3599619},{"type":"v1","addr":"172.21.15.40:6833","nonce":3599619}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:44:12.384 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 377 2019-12-05T06:44:12.386 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.382+0000 7f7c98013700 1 -- 172.21.15.42:0/2427241838 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f7c7003b270 msgr2=0x7f7c7003d780 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:12.386 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.382+0000 7f7c98013700 1 --2- 172.21.15.42:0/2427241838 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f7c7003b270 0x7f7c7003d780 crc :-1 s=READY pgs=1305 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:12.387 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.382+0000 7f7c98013700 1 -- 172.21.15.42:0/2427241838 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7c901429f0 msgr2=0x7f7c901236c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:12.387 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.382+0000 7f7c98013700 1 --2- 172.21.15.42:0/2427241838 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7c901429f0 0x7f7c901236c0 secure :-1 s=READY pgs=2359 cs=0 l=1 rx=0x7f7c90143320 tx=0x7f7c88011ce0).stop 2019-12-05T06:44:12.387 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.382+0000 7f7c98013700 1 -- 172.21.15.42:0/2427241838 shutdown_connections 2019-12-05T06:44:12.387 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.382+0000 7f7c98013700 1 --2- 172.21.15.42:0/2427241838 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f7c7003b270 0x7f7c7003d780 unknown :-1 s=CLOSED pgs=1305 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:12.387 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.382+0000 7f7c98013700 1 --2- 172.21.15.42:0/2427241838 >> v2:172.21.15.42:3300/0 conn(0x7f7c900a2fc0 0x7f7c900a33f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:12.387 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.382+0000 7f7c98013700 1 --2- 172.21.15.42:0/2427241838 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7c901429f0 0x7f7c901236c0 unknown :-1 s=CLOSED pgs=2359 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:12.387 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.382+0000 7f7c98013700 1 -- 172.21.15.42:0/2427241838 shutdown_connections 2019-12-05T06:44:12.388 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.382+0000 7f7c98013700 1 -- 172.21.15.42:0/2427241838 wait complete. 2019-12-05T06:44:12.388 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.382+0000 7f7c98013700 1 -- 172.21.15.42:0/2427241838 >> 172.21.15.42:0/2427241838 conn(0x7f7c900a8940 msgr2=0x7f7c9008f0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:44:12.396 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.b (rank=0) 2019-12-05T06:44:12.396 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2019-12-05T06:44:12.397 INFO:teuthology.orchestra.run:waiting for 300 2019-12-05T06:44:12.414 INFO:tasks.ceph.mds.b:Stopped 2019-12-05T06:44:12.414 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:44:12.520 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.506+0000 7faf3641a700 1 Processor -- start 2019-12-05T06:44:12.520 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.506+0000 7faf3641a700 1 -- start start 2019-12-05T06:44:12.520 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.506+0000 7faf3641a700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7faf300a7740 0x7faf3008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:44:12.521 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.506+0000 7faf3641a700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf300a0710 con 0x7faf3009fae0 2019-12-05T06:44:12.521 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.506+0000 7faf3641a700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf300a9ba0 con 0x7faf3007e720 2019-12-05T06:44:12.521 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.506+0000 7faf3641a700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf300a0b00 con 0x7faf300a7740 2019-12-05T06:44:12.521 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.506+0000 7faf2ffff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7faf3007e720 0x7faf3009f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:34628/0 (socket says 172.21.15.42:34628) 2019-12-05T06:44:12.521 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.506+0000 7faf2ffff700 1 -- 172.21.15.42:0/523580613 learned_addr learned my addr 172.21.15.42:0/523580613 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:44:12.522 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.506+0000 7faf349b7700 1 --2- 172.21.15.42:0/523580613 >> v2:172.21.15.40:3300/0 conn(0x7faf300a7740 0x7faf3008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:12.522 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.506+0000 7faf2f7fe700 1 -- 172.21.15.42:0/523580613 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 369033803 0 0) 0x7faf300a9ba0 con 0x7faf3007e720 2019-12-05T06:44:12.522 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.506+0000 7faf2f7fe700 1 -- 172.21.15.42:0/523580613 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf1c0030b0 con 0x7faf3007e720 2019-12-05T06:44:12.522 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.506+0000 7faf2f7fe700 1 -- 172.21.15.42:0/523580613 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2517789301 0 0) 0x7faf1c0030b0 con 0x7faf3007e720 2019-12-05T06:44:12.522 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.506+0000 7faf2f7fe700 1 -- 172.21.15.42:0/523580613 >> v2:172.21.15.40:3300/0 conn(0x7faf300a7740 msgr2=0x7faf3008df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:44:12.522 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.506+0000 7faf2f7fe700 1 --2- 172.21.15.42:0/523580613 >> v2:172.21.15.40:3300/0 conn(0x7faf300a7740 0x7faf3008df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:44:12.523 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.506+0000 7faf2f7fe700 1 -- 172.21.15.42:0/523580613 >> v1:172.21.15.40:6789/0 conn(0x7faf3009fae0 legacy=0x7faf3014efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:12.523 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.506+0000 7faf2f7fe700 1 -- 172.21.15.42:0/523580613 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf3007eb50 con 0x7faf3007e720 2019-12-05T06:44:12.523 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.506+0000 7faf2f7fe700 1 -- 172.21.15.42:0/523580613 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7faf18001750 con 0x7faf3007e720 2019-12-05T06:44:12.523 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.506+0000 7faf2f7fe700 1 -- 172.21.15.42:0/523580613 >> v1:172.21.15.42:6789/0 conn(0x7faf3007e720 legacy=0x7faf3009f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:12.523 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.506+0000 7faf2f7fe700 1 --2- 172.21.15.42:0/523580613 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf1c0036e0 0x7faf1c003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:44:12.524 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.506+0000 7faf2f7fe700 1 --2- 172.21.15.42:0/523580613 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7faf1c004420 0x7faf1c004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:44:12.524 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.506+0000 7faf2f7fe700 1 --2- 172.21.15.42:0/523580613 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faf1c004da0 0x7faf1c00b670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:44:12.524 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.506+0000 7faf2f7fe700 1 -- 172.21.15.42:0/523580613 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf300a0b00 con 0x7faf1c004da0 2019-12-05T06:44:12.524 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.506+0000 7faf349b7700 1 --2- 172.21.15.42:0/523580613 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf1c0036e0 0x7faf1c003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:12.524 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.510+0000 7faf2f7fe700 1 -- 172.21.15.42:0/523580613 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf1c04fc60 con 0x7faf1c0036e0 2019-12-05T06:44:12.525 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.510+0000 7faf2f7fe700 1 -- 172.21.15.42:0/523580613 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf1c04fea0 con 0x7faf1c004420 2019-12-05T06:44:12.525 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.510+0000 7faf2ffff700 1 --2- 172.21.15.42:0/523580613 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7faf1c004420 0x7faf1c004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:12.525 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.510+0000 7faf351b8700 1 --2- 172.21.15.42:0/523580613 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faf1c004da0 0x7faf1c00b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:12.525 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.510+0000 7faf349b7700 1 -- 172.21.15.42:0/523580613 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7faf1c004420 msgr2=0x7faf1c004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:44:12.525 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.510+0000 7faf349b7700 1 --2- 172.21.15.42:0/523580613 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7faf1c004420 0x7faf1c004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:44:12.525 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.510+0000 7faf349b7700 1 -- 172.21.15.42:0/523580613 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faf1c004da0 msgr2=0x7faf1c00b670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:44:12.526 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.510+0000 7faf349b7700 1 --2- 172.21.15.42:0/523580613 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faf1c004da0 0x7faf1c00b670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:44:12.526 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.510+0000 7faf349b7700 1 -- 172.21.15.42:0/523580613 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf1c0501a0 con 0x7faf1c0036e0 2019-12-05T06:44:12.526 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.510+0000 7faf351b8700 1 -- 172.21.15.42:0/523580613 reap_dead start 2019-12-05T06:44:12.526 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.510+0000 7faf349b7700 1 --2- 172.21.15.42:0/523580613 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf1c0036e0 0x7faf1c003b50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7faf20002f30 tx=0x7faf2000ffe0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:12.526 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.510+0000 7faf349b7700 1 --2- 172.21.15.42:0/523580613 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf1c0036e0 0x7faf1c003b50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7faf20002f30 tx=0x7faf2000ffe0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:12.526 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.510+0000 7faf349b7700 1 --2- 172.21.15.42:0/523580613 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf1c0036e0 0x7faf1c003b50 secure :-1 s=READY pgs=3129 cs=0 l=1 rx=0x7faf20002f30 tx=0x7faf2000ffe0).ready entity=mon.1 client_cookie=45cd0b369a036f73 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:44:12.527 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.510+0000 7faf349b7700 1 --2- 172.21.15.42:0/523580613 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf1c0036e0 0x7faf1c003b50 secure :-1 s=THROTTLE_DONE pgs=3129 cs=0 l=1 rx=0x7faf20002f30 tx=0x7faf2000ffe0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:12.527 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.510+0000 7faf349b7700 1 --2- 172.21.15.42:0/523580613 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf1c0036e0 0x7faf1c003b50 secure :-1 s=THROTTLE_DONE pgs=3129 cs=0 l=1 rx=0x7faf20002f30 tx=0x7faf2000ffe0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:12.527 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.510+0000 7faf2f7fe700 1 -- 172.21.15.42:0/523580613 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faf20010d10 con 0x7faf1c0036e0 2019-12-05T06:44:12.527 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.510+0000 7faf349b7700 1 --2- 172.21.15.42:0/523580613 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf1c0036e0 0x7faf1c003b50 secure :-1 s=THROTTLE_DONE pgs=3129 cs=0 l=1 rx=0x7faf20002f30 tx=0x7faf2000ffe0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:12.527 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.510+0000 7faf2f7fe700 1 -- 172.21.15.42:0/523580613 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf20010930 con 0x7faf1c0036e0 2019-12-05T06:44:12.527 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.510+0000 7faf2f7fe700 1 -- 172.21.15.42:0/523580613 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faf20008bb0 con 0x7faf1c0036e0 2019-12-05T06:44:12.528 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.510+0000 7faf3641a700 1 -- 172.21.15.42:0/523580613 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf1c0036e0 msgr2=0x7faf1c003b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:12.528 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.510+0000 7faf3641a700 1 --2- 172.21.15.42:0/523580613 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf1c0036e0 0x7faf1c003b50 secure :-1 s=READY pgs=3129 cs=0 l=1 rx=0x7faf20002f30 tx=0x7faf2000ffe0).stop 2019-12-05T06:44:12.528 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.510+0000 7faf3641a700 1 -- 172.21.15.42:0/523580613 shutdown_connections 2019-12-05T06:44:12.528 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.510+0000 7faf3641a700 1 --2- 172.21.15.42:0/523580613 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf1c0036e0 0x7faf1c003b50 unknown :-1 s=CLOSED pgs=3129 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:12.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.510+0000 7faf3641a700 1 -- 172.21.15.42:0/523580613 shutdown_connections 2019-12-05T06:44:12.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.510+0000 7faf3641a700 1 -- 172.21.15.42:0/523580613 wait complete. 2019-12-05T06:44:12.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.510+0000 7faf3641a700 1 -- 172.21.15.42:0/523580613 >> 172.21.15.42:0/523580613 conn(0x7faf300aabe0 msgr2=0x7faf300a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:44:12.530 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.510+0000 7faf3641a700 1 Processor -- start 2019-12-05T06:44:12.530 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.514+0000 7faf3641a700 1 -- start start 2019-12-05T06:44:12.530 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.514+0000 7faf3641a700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7faf3011c7b0 0x7faf30065390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:44:12.530 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.514+0000 7faf3641a700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7faf30064620 0x7faf300a5320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:44:12.532 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.514+0000 7faf3641a700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf30148850 con 0x7faf300658e0 2019-12-05T06:44:12.533 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.514+0000 7faf3641a700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf30064ba0 con 0x7faf3011c7b0 2019-12-05T06:44:12.533 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.514+0000 7faf3641a700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf30064e40 con 0x7faf30064620 2019-12-05T06:44:12.533 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.514+0000 7faf351b8700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7faf30064620 0x7faf300a5320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:12.533 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.514+0000 7faf351b8700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7faf30064620 0x7faf300a5320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:34602/0 (socket says 172.21.15.42:34602) 2019-12-05T06:44:12.534 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.514+0000 7faf351b8700 1 -- 172.21.15.42:0/4204920769 learned_addr learned my addr 172.21.15.42:0/4204920769 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:44:12.534 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.514+0000 7faf349b7700 1 --2- 172.21.15.42:0/4204920769 >> v2:172.21.15.40:3300/0 conn(0x7faf3011c7b0 0x7faf30065390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:12.534 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.514+0000 7faf351b8700 1 -- 172.21.15.42:0/4204920769 >> v2:172.21.15.40:3300/0 conn(0x7faf3011c7b0 msgr2=0x7faf30065390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:12.534 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.514+0000 7faf351b8700 1 --2- 172.21.15.42:0/4204920769 >> v2:172.21.15.40:3300/0 conn(0x7faf3011c7b0 0x7faf30065390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:12.535 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.514+0000 7faf351b8700 1 -- 172.21.15.42:0/4204920769 >> v1:172.21.15.40:6789/0 conn(0x7faf300658e0 legacy=0x7faf300a4b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:12.535 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.514+0000 7faf351b8700 1 -- 172.21.15.42:0/4204920769 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf300a5870 con 0x7faf30064620 2019-12-05T06:44:12.535 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.514+0000 7faf349b7700 1 --2- 172.21.15.42:0/4204920769 >> v2:172.21.15.40:3300/0 conn(0x7faf3011c7b0 0x7faf30065390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:44:12.535 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.514+0000 7faf351b8700 1 --2- 172.21.15.42:0/4204920769 >> v2:172.21.15.42:3300/0 conn(0x7faf30064620 0x7faf300a5320 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7faf3005bd60 tx=0x7faf28010800).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:12.535 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.514+0000 7faf351b8700 1 --2- 172.21.15.42:0/4204920769 >> v2:172.21.15.42:3300/0 conn(0x7faf30064620 0x7faf300a5320 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7faf3005bd60 tx=0x7faf28010800).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:12.535 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.514+0000 7faf351b8700 1 --2- 172.21.15.42:0/4204920769 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf30064620 0x7faf300a5320 secure :-1 s=READY pgs=3130 cs=0 l=1 rx=0x7faf3005bd60 tx=0x7faf28010800).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:44:12.536 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.514+0000 7faf351b8700 1 --2- 172.21.15.42:0/4204920769 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf30064620 0x7faf300a5320 secure :-1 s=THROTTLE_DONE pgs=3130 cs=0 l=1 rx=0x7faf3005bd60 tx=0x7faf28010800).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:12.536 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.514+0000 7faf351b8700 1 --2- 172.21.15.42:0/4204920769 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf30064620 0x7faf300a5320 secure :-1 s=THROTTLE_DONE pgs=3130 cs=0 l=1 rx=0x7faf3005bd60 tx=0x7faf28010800).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:12.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.514+0000 7faf351b8700 1 --2- 172.21.15.42:0/4204920769 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf30064620 0x7faf300a5320 secure :-1 s=THROTTLE_DONE pgs=3130 cs=0 l=1 rx=0x7faf3005bd60 tx=0x7faf28010800).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:12.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.514+0000 7faf2d7fa700 1 -- 172.21.15.42:0/4204920769 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faf28011e00 con 0x7faf30064620 2019-12-05T06:44:12.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.514+0000 7faf2d7fa700 1 -- 172.21.15.42:0/4204920769 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf28006ad0 con 0x7faf30064620 2019-12-05T06:44:12.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.514+0000 7faf3641a700 1 -- 172.21.15.42:0/4204920769 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf30130870 con 0x7faf30064620 2019-12-05T06:44:12.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.514+0000 7faf2d7fa700 1 -- 172.21.15.42:0/4204920769 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faf28001cb0 con 0x7faf30064620 2019-12-05T06:44:12.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.514+0000 7faf3641a700 1 -- 172.21.15.42:0/4204920769 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf30063330 con 0x7faf30064620 2019-12-05T06:44:12.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.514+0000 7faf351b8700 1 --2- 172.21.15.42:0/4204920769 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf30064620 0x7faf300a5320 secure :-1 s=THROTTLE_DONE pgs=3130 cs=0 l=1 rx=0x7faf3005bd60 tx=0x7faf28010800).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:12.539 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.518+0000 7faf2d7fa700 1 -- 172.21.15.42:0/4204920769 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7faf2800a280 con 0x7faf30064620 2019-12-05T06:44:12.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.518+0000 7faf351b8700 1 --2- 172.21.15.42:0/4204920769 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf30064620 0x7faf300a5320 secure :-1 s=THROTTLE_DONE pgs=3130 cs=0 l=1 rx=0x7faf3005bd60 tx=0x7faf28010800).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:12.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.518+0000 7faf2d7fa700 1 --2- 172.21.15.42:0/4204920769 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7faf1403acf0 0x7faf1403d200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:44:12.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.518+0000 7faf2d7fa700 1 -- 172.21.15.42:0/4204920769 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 11879+0+0 (secure 0 0 0) 0x7faf2804ac40 con 0x7faf30064620 2019-12-05T06:44:12.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.518+0000 7faf349b7700 1 --2- 172.21.15.42:0/4204920769 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7faf1403acf0 0x7faf1403d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:12.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.518+0000 7faf349b7700 1 --2- 172.21.15.42:0/4204920769 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7faf1403acf0 0x7faf1403d200 crc :-1 s=READY pgs=1306 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:44:12.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.522+0000 7faf3641a700 1 -- 172.21.15.42:0/4204920769 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf3005d110 con 0x7faf30064620 2019-12-05T06:44:12.541 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.526+0000 7faf351b8700 1 --2- 172.21.15.42:0/4204920769 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf30064620 0x7faf300a5320 secure :-1 s=THROTTLE_DONE pgs=3130 cs=0 l=1 rx=0x7faf3005bd60 tx=0x7faf28010800).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:12.541 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.526+0000 7faf2d7fa700 1 -- 172.21.15.42:0/4204920769 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7faf28031c60 con 0x7faf30064620 2019-12-05T06:44:12.696 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.694+0000 7faf3641a700 1 -- 172.21.15.42:0/4204920769 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faf300a65a0 con 0x7faf30064620 2019-12-05T06:44:12.697 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.694+0000 7faf351b8700 1 --2- 172.21.15.42:0/4204920769 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf30064620 0x7faf300a5320 secure :-1 s=THROTTLE_DONE pgs=3130 cs=0 l=1 rx=0x7faf3005bd60 tx=0x7faf28010800).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:12.697 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.694+0000 7faf2d7fa700 1 -- 172.21.15.42:0/4204920769 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 377 v377) v1 ==== 95+0+2108 (secure 0 0 0) 0x7faf28031ec0 con 0x7faf30064620 2019-12-05T06:44:12.699 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 377 2019-12-05T06:44:12.699 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:44:12.700 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":377,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":12303,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/2489540211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2489540211},{"type":"v1","addr":"172.21.15.42:6835","nonce":2489540211}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":377}],"filesystems":[{"mdsmap":{"epoch":376,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:43:33.543372+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":117,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12496},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12496":{"gid":12496,"name":"b","rank":0,"incarnation":373,"state":"up:active","state_seq":11,"addr":"172.21.15.40:6833/3599619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3599619},{"type":"v1","addr":"172.21.15.40:6833","nonce":3599619}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:44:12.704 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.698+0000 7faf3641a700 1 -- 172.21.15.42:0/4204920769 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7faf1403acf0 msgr2=0x7faf1403d200 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:12.704 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.698+0000 7faf3641a700 1 --2- 172.21.15.42:0/4204920769 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7faf1403acf0 0x7faf1403d200 crc :-1 s=READY pgs=1306 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:12.704 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.698+0000 7faf3641a700 1 -- 172.21.15.42:0/4204920769 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf30064620 msgr2=0x7faf300a5320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:12.705 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.698+0000 7faf3641a700 1 --2- 172.21.15.42:0/4204920769 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf30064620 0x7faf300a5320 secure :-1 s=READY pgs=3130 cs=0 l=1 rx=0x7faf3005bd60 tx=0x7faf28010800).stop 2019-12-05T06:44:12.705 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.698+0000 7faf3641a700 1 -- 172.21.15.42:0/4204920769 shutdown_connections 2019-12-05T06:44:12.705 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.698+0000 7faf3641a700 1 --2- 172.21.15.42:0/4204920769 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7faf1403acf0 0x7faf1403d200 unknown :-1 s=CLOSED pgs=1306 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:12.705 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.698+0000 7faf3641a700 1 --2- 172.21.15.42:0/4204920769 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf30064620 0x7faf300a5320 unknown :-1 s=CLOSED pgs=3130 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:12.705 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.698+0000 7faf3641a700 1 --2- 172.21.15.42:0/4204920769 >> v2:172.21.15.40:3300/0 conn(0x7faf3011c7b0 0x7faf30065390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:12.705 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.698+0000 7faf3641a700 1 -- 172.21.15.42:0/4204920769 shutdown_connections 2019-12-05T06:44:12.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.698+0000 7faf3641a700 1 -- 172.21.15.42:0/4204920769 wait complete. 2019-12-05T06:44:12.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:12.698+0000 7faf3641a700 1 -- 172.21.15.42:0/4204920769 >> 172.21.15.42:0/4204920769 conn(0x7faf300aabe0 msgr2=0x7faf300a9790 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:44:12.719 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:44:14.720 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:44:14.828 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:14.814+0000 7f9b3e555700 1 Processor -- start 2019-12-05T06:44:14.828 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:14.818+0000 7f9b3e555700 1 -- start start 2019-12-05T06:44:14.828 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:14.818+0000 7f9b3e555700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f9b3808df70 0x7f9b3809f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:44:14.828 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:14.818+0000 7f9b3e555700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9b3809f820 0x7f9b3814efd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:44:14.828 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:14.818+0000 7f9b3e555700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b3809fc50 con 0x7f9b380a7730 2019-12-05T06:44:14.829 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:14.818+0000 7f9b3e555700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b380a0af0 con 0x7f9b3808df70 2019-12-05T06:44:14.829 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:14.818+0000 7f9b3e555700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b3808e5a0 con 0x7f9b3809f820 2019-12-05T06:44:14.829 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:14.818+0000 7f9b3caf2700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f9b380a7730 0x7f9b3807e710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:34658/0 (socket says 172.21.15.42:34658) 2019-12-05T06:44:14.829 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:14.818+0000 7f9b3caf2700 1 -- 172.21.15.42:0/3246287604 learned_addr learned my addr 172.21.15.42:0/3246287604 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:44:14.830 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:14.818+0000 7f9b3d2f3700 1 --2- 172.21.15.42:0/3246287604 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9b3809f820 0x7f9b3814efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:14.830 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:14.818+0000 7f9b2ffff700 1 --2- 172.21.15.42:0/3246287604 >> v2:172.21.15.40:3300/0 conn(0x7f9b3808df70 0x7f9b3809f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:14.830 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:14.818+0000 7f9b2f7fe700 1 -- 172.21.15.42:0/3246287604 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1686097680 0 0) 0x7f9b3809fc50 con 0x7f9b380a7730 2019-12-05T06:44:14.830 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:14.818+0000 7f9b2f7fe700 1 -- 172.21.15.42:0/3246287604 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b200030b0 con 0x7f9b380a7730 2019-12-05T06:44:14.831 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:14.818+0000 7f9b2f7fe700 1 -- 172.21.15.42:0/3246287604 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2689015311 0 0) 0x7f9b200030b0 con 0x7f9b380a7730 2019-12-05T06:44:14.832 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:14.818+0000 7f9b2f7fe700 1 -- 172.21.15.42:0/3246287604 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9b3809f820 msgr2=0x7f9b3814efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:44:14.832 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:14.818+0000 7f9b2f7fe700 1 --2- 172.21.15.42:0/3246287604 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9b3809f820 0x7f9b3814efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:44:14.833 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:14.818+0000 7f9b2f7fe700 1 -- 172.21.15.42:0/3246287604 >> v2:172.21.15.40:3300/0 conn(0x7f9b3808df70 msgr2=0x7f9b3809f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:44:14.833 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:14.818+0000 7f9b2f7fe700 1 --2- 172.21.15.42:0/3246287604 >> v2:172.21.15.40:3300/0 conn(0x7f9b3808df70 0x7f9b3809f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:44:14.833 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:14.818+0000 7f9b2f7fe700 1 -- 172.21.15.42:0/3246287604 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b380a0700 con 0x7f9b380a7730 2019-12-05T06:44:14.833 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:14.818+0000 7f9b2f7fe700 1 -- 172.21.15.42:0/3246287604 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7f9b34001750 con 0x7f9b380a7730 2019-12-05T06:44:14.834 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:14.818+0000 7f9b2f7fe700 1 -- 172.21.15.42:0/3246287604 >> v1:172.21.15.42:6789/0 conn(0x7f9b380a7730 legacy=0x7f9b3807e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:14.834 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:14.818+0000 7f9b2f7fe700 1 --2- 172.21.15.42:0/3246287604 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9b20003760 0x7f9b20003bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:44:14.834 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:14.818+0000 7f9b2f7fe700 1 --2- 172.21.15.42:0/3246287604 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9b20004420 0x7f9b20004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:44:14.834 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:14.818+0000 7f9b2f7fe700 1 --2- 172.21.15.42:0/3246287604 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9b20004df0 0x7f9b2004f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:44:14.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:14.818+0000 7f9b2f7fe700 1 -- 172.21.15.42:0/3246287604 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b380a0af0 con 0x7f9b20003760 2019-12-05T06:44:14.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:14.818+0000 7f9b2f7fe700 1 -- 172.21.15.42:0/3246287604 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b3808e5a0 con 0x7f9b20004420 2019-12-05T06:44:14.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:14.818+0000 7f9b2f7fe700 1 -- 172.21.15.42:0/3246287604 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b2004fbd0 con 0x7f9b20004df0 2019-12-05T06:44:14.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:14.818+0000 7f9b2ffff700 1 --2- 172.21.15.42:0/3246287604 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9b20004420 0x7f9b20004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:14.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:14.818+0000 7f9b3d2f3700 1 --2- 172.21.15.42:0/3246287604 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9b20004df0 0x7f9b2004f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:14.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:14.818+0000 7f9b3caf2700 1 --2- 172.21.15.42:0/3246287604 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9b20003760 0x7f9b20003bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:14.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:14.822+0000 7f9b3caf2700 1 -- 172.21.15.42:0/3246287604 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9b20004df0 msgr2=0x7f9b2004f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:44:14.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:14.822+0000 7f9b3caf2700 1 --2- 172.21.15.42:0/3246287604 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9b20004df0 0x7f9b2004f680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:44:14.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:14.822+0000 7f9b3caf2700 1 -- 172.21.15.42:0/3246287604 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9b20004420 msgr2=0x7f9b20004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:44:14.837 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:14.822+0000 7f9b3caf2700 1 --2- 172.21.15.42:0/3246287604 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9b20004420 0x7f9b20004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:44:14.837 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:14.822+0000 7f9b3caf2700 1 -- 172.21.15.42:0/3246287604 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b2004fdf0 con 0x7f9b20003760 2019-12-05T06:44:14.837 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:14.822+0000 7f9b2ffff700 1 --2- 172.21.15.42:0/3246287604 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9b20004420 0x7f9b20004850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:44:14.837 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:14.822+0000 7f9b3d2f3700 1 --2- 172.21.15.42:0/3246287604 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9b20004df0 0x7f9b2004f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:44:14.838 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:14.822+0000 7f9b3caf2700 1 --2- 172.21.15.42:0/3246287604 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9b20003760 0x7f9b20003bd0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f9b34000fc0 tx=0x7f9b3400c120).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:14.838 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:14.822+0000 7f9b3caf2700 1 --2- 172.21.15.42:0/3246287604 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9b20003760 0x7f9b20003bd0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f9b34000fc0 tx=0x7f9b3400c120).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:14.838 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:14.822+0000 7f9b3caf2700 1 --2- 172.21.15.42:0/3246287604 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9b20003760 0x7f9b20003bd0 secure :-1 s=READY pgs=2362 cs=0 l=1 rx=0x7f9b34000fc0 tx=0x7f9b3400c120).ready entity=mon.0 client_cookie=f780f6e589a29eda server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:44:14.838 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:14.822+0000 7f9b3caf2700 1 --2- 172.21.15.42:0/3246287604 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9b20003760 0x7f9b20003bd0 secure :-1 s=THROTTLE_DONE pgs=2362 cs=0 l=1 rx=0x7f9b34000fc0 tx=0x7f9b3400c120).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:14.839 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:14.822+0000 7f9b2f7fe700 1 -- 172.21.15.42:0/3246287604 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9b3400ec40 con 0x7f9b20003760 2019-12-05T06:44:14.839 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:14.822+0000 7f9b3caf2700 1 --2- 172.21.15.42:0/3246287604 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9b20003760 0x7f9b20003bd0 secure :-1 s=THROTTLE_DONE pgs=2362 cs=0 l=1 rx=0x7f9b34000fc0 tx=0x7f9b3400c120).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:14.839 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:14.822+0000 7f9b3caf2700 1 --2- 172.21.15.42:0/3246287604 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9b20003760 0x7f9b20003bd0 secure :-1 s=THROTTLE_DONE pgs=2362 cs=0 l=1 rx=0x7f9b34000fc0 tx=0x7f9b3400c120).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:14.839 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:14.822+0000 7f9b2f7fe700 1 -- 172.21.15.42:0/3246287604 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b3400e860 con 0x7f9b20003760 2019-12-05T06:44:14.840 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:14.822+0000 7f9b2f7fe700 1 -- 172.21.15.42:0/3246287604 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9b34010530 con 0x7f9b20003760 2019-12-05T06:44:14.840 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:14.822+0000 7f9b3e555700 1 -- 172.21.15.42:0/3246287604 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9b20003760 msgr2=0x7f9b20003bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:14.840 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:14.822+0000 7f9b3e555700 1 --2- 172.21.15.42:0/3246287604 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9b20003760 0x7f9b20003bd0 secure :-1 s=READY pgs=2362 cs=0 l=1 rx=0x7f9b34000fc0 tx=0x7f9b3400c120).stop 2019-12-05T06:44:14.840 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:14.822+0000 7f9b3d2f3700 1 -- 172.21.15.42:0/3246287604 reap_dead start 2019-12-05T06:44:14.840 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:14.822+0000 7f9b3e555700 1 -- 172.21.15.42:0/3246287604 shutdown_connections 2019-12-05T06:44:14.840 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:14.822+0000 7f9b3e555700 1 -- 172.21.15.42:0/3246287604 shutdown_connections 2019-12-05T06:44:14.840 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:14.822+0000 7f9b3e555700 1 -- 172.21.15.42:0/3246287604 wait complete. 2019-12-05T06:44:14.841 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:14.822+0000 7f9b3e555700 1 -- 172.21.15.42:0/3246287604 >> 172.21.15.42:0/3246287604 conn(0x7f9b380aabd0 msgr2=0x7f9b380a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:44:14.844 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:14.826+0000 7f9b3e555700 1 Processor -- start 2019-12-05T06:44:14.844 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:14.826+0000 7f9b3e555700 1 -- start start 2019-12-05T06:44:14.845 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:14.826+0000 7f9b3e555700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9b3808f4d0 0x7f9b3807c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:44:14.845 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:14.826+0000 7f9b3e555700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b380a3180 con 0x7f9b38136970 2019-12-05T06:44:14.845 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:14.826+0000 7f9b3e555700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b38145390 con 0x7f9b3812df70 2019-12-05T06:44:14.845 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:14.826+0000 7f9b3e555700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b3808f900 con 0x7f9b3808f4d0 2019-12-05T06:44:14.845 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:14.826+0000 7f9b2ffff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f9b3812df70 0x7f9b3812e3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:34674/0 (socket says 172.21.15.42:34674) 2019-12-05T06:44:14.845 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:14.826+0000 7f9b2ffff700 1 -- 172.21.15.42:0/4060673762 learned_addr learned my addr 172.21.15.42:0/4060673762 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:44:14.846 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:14.826+0000 7f9b3caf2700 1 --2- 172.21.15.42:0/4060673762 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9b3808f4d0 0x7f9b3807c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:14.846 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:14.826+0000 7f9b2d7fa700 1 -- 172.21.15.42:0/4060673762 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1318443938 0 0) 0x7f9b38145390 con 0x7f9b3812df70 2019-12-05T06:44:14.846 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:14.826+0000 7f9b2d7fa700 1 -- 172.21.15.42:0/4060673762 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b180030b0 con 0x7f9b3812df70 2019-12-05T06:44:14.846 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:14.826+0000 7f9b2d7fa700 1 -- 172.21.15.42:0/4060673762 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3405689884 0 0) 0x7f9b380a3180 con 0x7f9b38136970 2019-12-05T06:44:14.846 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:14.826+0000 7f9b2d7fa700 1 -- 172.21.15.42:0/4060673762 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b38145390 con 0x7f9b38136970 2019-12-05T06:44:14.846 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:14.826+0000 7f9b3caf2700 1 -- 172.21.15.42:0/4060673762 >> v1:172.21.15.42:6789/0 conn(0x7f9b3812df70 legacy=0x7f9b3812e3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:14.846 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:14.826+0000 7f9b3caf2700 1 -- 172.21.15.42:0/4060673762 >> v1:172.21.15.40:6789/0 conn(0x7f9b38136970 legacy=0x7f9b38138db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:14.847 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:14.826+0000 7f9b3caf2700 1 -- 172.21.15.42:0/4060673762 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b380a3f30 con 0x7f9b3808f4d0 2019-12-05T06:44:14.847 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:14.826+0000 7f9b3caf2700 1 --2- 172.21.15.42:0/4060673762 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9b3808f4d0 0x7f9b3807c370 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f9b34000fc0 tx=0x7f9b3400ea50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:14.847 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:14.826+0000 7f9b3caf2700 1 --2- 172.21.15.42:0/4060673762 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9b3808f4d0 0x7f9b3807c370 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f9b34000fc0 tx=0x7f9b3400ea50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:14.847 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:14.826+0000 7f9b3caf2700 1 --2- 172.21.15.42:0/4060673762 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9b3808f4d0 0x7f9b3807c370 secure :-1 s=READY pgs=680 cs=0 l=1 rx=0x7f9b34000fc0 tx=0x7f9b3400ea50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:44:14.847 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:14.826+0000 7f9b3caf2700 1 --2- 172.21.15.42:0/4060673762 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9b3808f4d0 0x7f9b3807c370 secure :-1 s=THROTTLE_DONE pgs=680 cs=0 l=1 rx=0x7f9b34000fc0 tx=0x7f9b3400ea50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:14.847 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:14.826+0000 7f9b2d7fa700 1 -- 172.21.15.42:0/4060673762 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9b34001690 con 0x7f9b3808f4d0 2019-12-05T06:44:14.848 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:14.826+0000 7f9b3caf2700 1 --2- 172.21.15.42:0/4060673762 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9b3808f4d0 0x7f9b3807c370 secure :-1 s=THROTTLE_DONE pgs=680 cs=0 l=1 rx=0x7f9b34000fc0 tx=0x7f9b3400ea50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:14.848 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:14.826+0000 7f9b3caf2700 1 --2- 172.21.15.42:0/4060673762 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9b3808f4d0 0x7f9b3807c370 secure :-1 s=THROTTLE_DONE pgs=680 cs=0 l=1 rx=0x7f9b34000fc0 tx=0x7f9b3400ea50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:14.848 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:14.826+0000 7f9b2d7fa700 1 -- 172.21.15.42:0/4060673762 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b34010150 con 0x7f9b3808f4d0 2019-12-05T06:44:14.848 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:14.826+0000 7f9b3e555700 1 -- 172.21.15.42:0/4060673762 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b38065ba0 con 0x7f9b3808f4d0 2019-12-05T06:44:14.848 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:14.826+0000 7f9b2d7fa700 1 -- 172.21.15.42:0/4060673762 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9b34004400 con 0x7f9b3808f4d0 2019-12-05T06:44:14.848 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:14.826+0000 7f9b3e555700 1 -- 172.21.15.42:0/4060673762 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b3807c8c0 con 0x7f9b3808f4d0 2019-12-05T06:44:14.848 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:14.830+0000 7f9b3caf2700 1 --2- 172.21.15.42:0/4060673762 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9b3808f4d0 0x7f9b3807c370 secure :-1 s=THROTTLE_DONE pgs=680 cs=0 l=1 rx=0x7f9b34000fc0 tx=0x7f9b3400ea50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:14.849 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:14.830+0000 7f9b2d7fa700 1 -- 172.21.15.42:0/4060673762 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f9b340046d0 con 0x7f9b3808f4d0 2019-12-05T06:44:14.849 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:14.830+0000 7f9b3caf2700 1 --2- 172.21.15.42:0/4060673762 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9b3808f4d0 0x7f9b3807c370 secure :-1 s=THROTTLE_DONE pgs=680 cs=0 l=1 rx=0x7f9b34000fc0 tx=0x7f9b3400ea50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:14.849 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:14.830+0000 7f9b2d7fa700 1 --2- 172.21.15.42:0/4060673762 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9b1803b460 0x7f9b1803d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:44:14.849 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:14.830+0000 7f9b2d7fa700 1 -- 172.21.15.42:0/4060673762 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 11879+0+0 (secure 0 0 0) 0x7f9b3404abd0 con 0x7f9b3808f4d0 2019-12-05T06:44:14.849 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:14.830+0000 7f9b2ffff700 1 --2- 172.21.15.42:0/4060673762 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9b1803b460 0x7f9b1803d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:14.850 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:14.830+0000 7f9b2ffff700 1 --2- 172.21.15.42:0/4060673762 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9b1803b460 0x7f9b1803d970 crc :-1 s=READY pgs=1307 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:44:14.850 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:14.830+0000 7f9b3e555700 1 -- 172.21.15.42:0/4060673762 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b3805d100 con 0x7f9b3808f4d0 2019-12-05T06:44:14.851 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:14.842+0000 7f9b3caf2700 1 --2- 172.21.15.42:0/4060673762 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9b3808f4d0 0x7f9b3807c370 secure :-1 s=THROTTLE_DONE pgs=680 cs=0 l=1 rx=0x7f9b34000fc0 tx=0x7f9b3400ea50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:14.851 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:14.842+0000 7f9b2d7fa700 1 -- 172.21.15.42:0/4060673762 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f9b34031c50 con 0x7f9b3808f4d0 2019-12-05T06:44:15.022 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:15.010+0000 7f9b3e555700 1 -- 172.21.15.42:0/4060673762 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9b3805b8b0 con 0x7f9b3808f4d0 2019-12-05T06:44:15.022 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:15.010+0000 7f9b3caf2700 1 --2- 172.21.15.42:0/4060673762 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9b3808f4d0 0x7f9b3807c370 secure :-1 s=THROTTLE_DONE pgs=680 cs=0 l=1 rx=0x7f9b34000fc0 tx=0x7f9b3400ea50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:15.022 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:15.010+0000 7f9b2d7fa700 1 -- 172.21.15.42:0/4060673762 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 377 v377) v1 ==== 95+0+2108 (secure 0 0 0) 0x7f9b340150d0 con 0x7f9b3808f4d0 2019-12-05T06:44:15.022 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 377 2019-12-05T06:44:15.022 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:15.014+0000 7f9b3e555700 1 -- 172.21.15.42:0/4060673762 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9b1803b460 msgr2=0x7f9b1803d970 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:15.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:15.014+0000 7f9b3e555700 1 --2- 172.21.15.42:0/4060673762 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9b1803b460 0x7f9b1803d970 crc :-1 s=READY pgs=1307 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:15.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:15.014+0000 7f9b3e555700 1 -- 172.21.15.42:0/4060673762 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9b3808f4d0 msgr2=0x7f9b3807c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:15.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:15.014+0000 7f9b3e555700 1 --2- 172.21.15.42:0/4060673762 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9b3808f4d0 0x7f9b3807c370 secure :-1 s=READY pgs=680 cs=0 l=1 rx=0x7f9b34000fc0 tx=0x7f9b3400ea50).stop 2019-12-05T06:44:15.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:15.014+0000 7f9b3e555700 1 -- 172.21.15.42:0/4060673762 shutdown_connections 2019-12-05T06:44:15.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:15.014+0000 7f9b3e555700 1 --2- 172.21.15.42:0/4060673762 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9b1803b460 0x7f9b1803d970 unknown :-1 s=CLOSED pgs=1307 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:15.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:15.014+0000 7f9b3e555700 1 --2- 172.21.15.42:0/4060673762 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9b3808f4d0 0x7f9b3807c370 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:15.023 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:15.014+0000 7f9b3e555700 1 -- 172.21.15.42:0/4060673762 shutdown_connections 2019-12-05T06:44:15.024 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:15.014+0000 7f9b3e555700 1 -- 172.21.15.42:0/4060673762 wait complete. 2019-12-05T06:44:15.024 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:15.014+0000 7f9b3e555700 1 -- 172.21.15.42:0/4060673762 >> 172.21.15.42:0/4060673762 conn(0x7f9b380a9630 msgr2=0x7f9b3807e750 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:44:15.024 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:44:15.024 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":377,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":12303,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/2489540211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2489540211},{"type":"v1","addr":"172.21.15.42:6835","nonce":2489540211}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":377}],"filesystems":[{"mdsmap":{"epoch":376,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:43:33.543372+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":117,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12496},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12496":{"gid":12496,"name":"b","rank":0,"incarnation":373,"state":"up:active","state_seq":11,"addr":"172.21.15.40:6833/3599619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3599619},{"type":"v1","addr":"172.21.15.40:6833","nonce":3599619}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:44:15.033 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:44:15.643 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:44:15.647 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:44:17.033 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:44:17.139 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:17.130+0000 7f5a64955700 1 Processor -- start 2019-12-05T06:44:17.140 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:17.130+0000 7f5a64955700 1 -- start start 2019-12-05T06:44:17.140 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:17.130+0000 7f5a64955700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5a6008df70 0x7f5a6009f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:44:17.140 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:17.130+0000 7f5a64955700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5a6009f820 0x7f5a6014efd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:44:17.140 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:17.130+0000 7f5a64955700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a6009fc50 con 0x7f5a600a7730 2019-12-05T06:44:17.140 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:17.130+0000 7f5a64955700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a600a0af0 con 0x7f5a6009f820 2019-12-05T06:44:17.140 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:17.130+0000 7f5a64955700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a6008e5a0 con 0x7f5a6008df70 2019-12-05T06:44:17.141 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:17.130+0000 7f5a5f59e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5a6009f820 0x7f5a6014efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:17.141 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:17.130+0000 7f5a5f59e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5a6009f820 0x7f5a6014efd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:34646/0 (socket says 172.21.15.42:34646) 2019-12-05T06:44:17.141 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:17.130+0000 7f5a5f59e700 1 -- 172.21.15.42:0/3563082696 learned_addr learned my addr 172.21.15.42:0/3563082696 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:44:17.141 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:17.130+0000 7f5a5e59c700 1 --2- 172.21.15.42:0/3563082696 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5a6008df70 0x7f5a6009f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:17.141 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:17.134+0000 7f5a5f59e700 1 -- 172.21.15.42:0/3563082696 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5a6008df70 msgr2=0x7f5a6009f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:44:17.141 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:17.134+0000 7f5a5f59e700 1 --2- 172.21.15.42:0/3563082696 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5a6008df70 0x7f5a6009f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:44:17.141 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:17.134+0000 7f5a5f59e700 1 -- 172.21.15.42:0/3563082696 >> v1:172.21.15.40:6789/0 conn(0x7f5a600a7730 legacy=0x7f5a6007e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:17.142 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:17.134+0000 7f5a5dd9b700 1 -- 172.21.15.42:0/3563082696 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2978402409 0 0) 0x7f5a6009fc50 con 0x7f5a600a7730 2019-12-05T06:44:17.142 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:17.134+0000 7f5a5f59e700 1 -- 172.21.15.42:0/3563082696 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a600a0700 con 0x7f5a6009f820 2019-12-05T06:44:17.142 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:17.134+0000 7f5a5e59c700 1 --2- 172.21.15.42:0/3563082696 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5a6008df70 0x7f5a6009f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:44:17.142 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:17.134+0000 7f5a5f59e700 1 --2- 172.21.15.42:0/3563082696 >> v2:172.21.15.42:3300/0 conn(0x7f5a6009f820 0x7f5a6014efd0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f5a58008790 tx=0x7f5a5800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:17.142 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:17.134+0000 7f5a5f59e700 1 --2- 172.21.15.42:0/3563082696 >> v2:172.21.15.42:3300/0 conn(0x7f5a6009f820 0x7f5a6014efd0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f5a58008790 tx=0x7f5a5800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:17.142 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:17.134+0000 7f5a5f59e700 1 --2- 172.21.15.42:0/3563082696 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5a6009f820 0x7f5a6014efd0 secure :-1 s=READY pgs=3133 cs=0 l=1 rx=0x7f5a58008790 tx=0x7f5a5800fd30).ready entity=mon.1 client_cookie=2e0bb5da4d721331 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:44:17.142 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:17.134+0000 7f5a5f59e700 1 --2- 172.21.15.42:0/3563082696 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5a6009f820 0x7f5a6014efd0 secure :-1 s=THROTTLE_DONE pgs=3133 cs=0 l=1 rx=0x7f5a58008790 tx=0x7f5a5800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:17.143 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:17.134+0000 7f5a5dd9b700 1 -- 172.21.15.42:0/3563082696 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5a58008c90 con 0x7f5a6009f820 2019-12-05T06:44:17.143 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:17.134+0000 7f5a5f59e700 1 --2- 172.21.15.42:0/3563082696 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5a6009f820 0x7f5a6014efd0 secure :-1 s=THROTTLE_DONE pgs=3133 cs=0 l=1 rx=0x7f5a58008790 tx=0x7f5a5800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:17.143 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:17.134+0000 7f5a5f59e700 1 --2- 172.21.15.42:0/3563082696 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5a6009f820 0x7f5a6014efd0 secure :-1 s=THROTTLE_DONE pgs=3133 cs=0 l=1 rx=0x7f5a58008790 tx=0x7f5a5800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:17.143 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:17.134+0000 7f5a5dd9b700 1 -- 172.21.15.42:0/3563082696 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a580088b0 con 0x7f5a6009f820 2019-12-05T06:44:17.143 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:17.134+0000 7f5a5dd9b700 1 -- 172.21.15.42:0/3563082696 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5a58011b10 con 0x7f5a6009f820 2019-12-05T06:44:17.143 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:17.134+0000 7f5a64955700 1 -- 172.21.15.42:0/3563082696 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5a6009f820 msgr2=0x7f5a6014efd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:17.144 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:17.134+0000 7f5a64955700 1 --2- 172.21.15.42:0/3563082696 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5a6009f820 0x7f5a6014efd0 secure :-1 s=READY pgs=3133 cs=0 l=1 rx=0x7f5a58008790 tx=0x7f5a5800fd30).stop 2019-12-05T06:44:17.144 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:17.134+0000 7f5a64955700 1 -- 172.21.15.42:0/3563082696 shutdown_connections 2019-12-05T06:44:17.144 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:17.134+0000 7f5a64955700 1 --2- 172.21.15.42:0/3563082696 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5a6009f820 0x7f5a6014efd0 unknown :-1 s=CLOSED pgs=3133 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:17.144 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:17.134+0000 7f5a64955700 1 --2- 172.21.15.42:0/3563082696 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5a6008df70 0x7f5a6009f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:44:17.144 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:17.134+0000 7f5a64955700 1 -- 172.21.15.42:0/3563082696 shutdown_connections 2019-12-05T06:44:17.144 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:17.134+0000 7f5a64955700 1 -- 172.21.15.42:0/3563082696 wait complete. 2019-12-05T06:44:17.144 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:17.134+0000 7f5a64955700 1 -- 172.21.15.42:0/3563082696 >> 172.21.15.42:0/3563082696 conn(0x7f5a600aabd0 msgr2=0x7f5a600a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:44:17.148 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:17.138+0000 7f5a64955700 1 Processor -- start 2019-12-05T06:44:17.148 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:17.138+0000 7f5a64955700 1 -- start start 2019-12-05T06:44:17.148 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:17.138+0000 7f5a64955700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5a600a6c50 0x7f5a6012e360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:44:17.148 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:17.138+0000 7f5a64955700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5a6011c8f0 0x7f5a60120670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:44:17.148 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:17.138+0000 7f5a64955700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5a60120bc0 0x7f5a6011a330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:44:17.149 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:17.138+0000 7f5a64955700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a600a6470 con 0x7f5a60120bc0 2019-12-05T06:44:17.149 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:17.138+0000 7f5a64955700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a600a6710 con 0x7f5a6011c8f0 2019-12-05T06:44:17.149 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:17.138+0000 7f5a64955700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a600a69b0 con 0x7f5a600a6c50 2019-12-05T06:44:17.149 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:17.138+0000 7f5a5e59c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5a6011c8f0 0x7f5a60120670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:17.149 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:17.138+0000 7f5a5e59c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5a6011c8f0 0x7f5a60120670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:34648/0 (socket says 172.21.15.42:34648) 2019-12-05T06:44:17.149 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:17.138+0000 7f5a5e59c700 1 -- 172.21.15.42:0/1017445656 learned_addr learned my addr 172.21.15.42:0/1017445656 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:44:17.150 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:17.138+0000 7f5a5f59e700 1 --2- 172.21.15.42:0/1017445656 >> v2:172.21.15.40:3300/0 conn(0x7f5a60120bc0 0x7f5a6011a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:17.150 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:17.138+0000 7f5a5ed9d700 1 --2- 172.21.15.42:0/1017445656 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5a600a6c50 0x7f5a6012e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:17.150 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:17.138+0000 7f5a5e59c700 1 -- 172.21.15.42:0/1017445656 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5a600a6c50 msgr2=0x7f5a6012e360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:17.150 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:17.138+0000 7f5a5e59c700 1 --2- 172.21.15.42:0/1017445656 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5a600a6c50 0x7f5a6012e360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:17.150 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:17.138+0000 7f5a5e59c700 1 -- 172.21.15.42:0/1017445656 >> v2:172.21.15.40:3300/0 conn(0x7f5a60120bc0 msgr2=0x7f5a6011a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:17.150 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:17.138+0000 7f5a5e59c700 1 --2- 172.21.15.42:0/1017445656 >> v2:172.21.15.40:3300/0 conn(0x7f5a60120bc0 0x7f5a6011a330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:17.151 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:17.138+0000 7f5a5e59c700 1 -- 172.21.15.42:0/1017445656 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a600a61a0 con 0x7f5a6011c8f0 2019-12-05T06:44:17.151 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:17.138+0000 7f5a5ed9d700 1 --2- 172.21.15.42:0/1017445656 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5a600a6c50 0x7f5a6012e360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:44:17.151 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:17.138+0000 7f5a5e59c700 1 --2- 172.21.15.42:0/1017445656 >> v2:172.21.15.42:3300/0 conn(0x7f5a6011c8f0 0x7f5a60120670 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f5a6011b460 tx=0x7f5a500111e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:17.151 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:17.138+0000 7f5a5e59c700 1 --2- 172.21.15.42:0/1017445656 >> v2:172.21.15.42:3300/0 conn(0x7f5a6011c8f0 0x7f5a60120670 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f5a6011b460 tx=0x7f5a500111e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:17.151 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:17.138+0000 7f5a5e59c700 1 --2- 172.21.15.42:0/1017445656 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5a6011c8f0 0x7f5a60120670 secure :-1 s=READY pgs=3134 cs=0 l=1 rx=0x7f5a6011b460 tx=0x7f5a500111e0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:44:17.151 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:17.138+0000 7f5a5e59c700 1 --2- 172.21.15.42:0/1017445656 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5a6011c8f0 0x7f5a60120670 secure :-1 s=THROTTLE_DONE pgs=3134 cs=0 l=1 rx=0x7f5a6011b460 tx=0x7f5a500111e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:17.151 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:17.138+0000 7f5a5e59c700 1 --2- 172.21.15.42:0/1017445656 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5a6011c8f0 0x7f5a60120670 secure :-1 s=THROTTLE_DONE pgs=3134 cs=0 l=1 rx=0x7f5a6011b460 tx=0x7f5a500111e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:17.152 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:17.138+0000 7f5a577fe700 1 -- 172.21.15.42:0/1017445656 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5a50007160 con 0x7f5a6011c8f0 2019-12-05T06:44:17.152 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:17.138+0000 7f5a5e59c700 1 --2- 172.21.15.42:0/1017445656 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5a6011c8f0 0x7f5a60120670 secure :-1 s=THROTTLE_DONE pgs=3134 cs=0 l=1 rx=0x7f5a6011b460 tx=0x7f5a500111e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:17.152 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:17.138+0000 7f5a577fe700 1 -- 172.21.15.42:0/1017445656 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a50005bc0 con 0x7f5a6011c8f0 2019-12-05T06:44:17.152 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:17.138+0000 7f5a64955700 1 -- 172.21.15.42:0/1017445656 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a6012e8b0 con 0x7f5a6011c8f0 2019-12-05T06:44:17.152 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:17.138+0000 7f5a577fe700 1 -- 172.21.15.42:0/1017445656 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5a500127e0 con 0x7f5a6011c8f0 2019-12-05T06:44:17.152 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:17.138+0000 7f5a64955700 1 -- 172.21.15.42:0/1017445656 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a6011cd20 con 0x7f5a6011c8f0 2019-12-05T06:44:17.152 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:17.142+0000 7f5a5e59c700 1 --2- 172.21.15.42:0/1017445656 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5a6011c8f0 0x7f5a60120670 secure :-1 s=THROTTLE_DONE pgs=3134 cs=0 l=1 rx=0x7f5a6011b460 tx=0x7f5a500111e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:17.153 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:17.142+0000 7f5a577fe700 1 -- 172.21.15.42:0/1017445656 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f5a50012af0 con 0x7f5a6011c8f0 2019-12-05T06:44:17.153 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:17.142+0000 7f5a5e59c700 1 --2- 172.21.15.42:0/1017445656 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5a6011c8f0 0x7f5a60120670 secure :-1 s=THROTTLE_DONE pgs=3134 cs=0 l=1 rx=0x7f5a6011b460 tx=0x7f5a500111e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:17.153 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:17.142+0000 7f5a577fe700 1 --2- 172.21.15.42:0/1017445656 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5a4403ad40 0x7f5a4403d250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:44:17.153 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:17.142+0000 7f5a577fe700 1 -- 172.21.15.42:0/1017445656 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 11879+0+0 (secure 0 0 0) 0x7f5a5004b2d0 con 0x7f5a6011c8f0 2019-12-05T06:44:17.153 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:17.142+0000 7f5a5ed9d700 1 --2- 172.21.15.42:0/1017445656 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5a4403ad40 0x7f5a4403d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:17.153 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:17.142+0000 7f5a5ed9d700 1 --2- 172.21.15.42:0/1017445656 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5a4403ad40 0x7f5a4403d250 crc :-1 s=READY pgs=1308 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:44:17.154 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:17.146+0000 7f5a64955700 1 -- 172.21.15.42:0/1017445656 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a6005d100 con 0x7f5a6011c8f0 2019-12-05T06:44:17.158 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:17.154+0000 7f5a5e59c700 1 --2- 172.21.15.42:0/1017445656 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5a6011c8f0 0x7f5a60120670 secure :-1 s=THROTTLE_DONE pgs=3134 cs=0 l=1 rx=0x7f5a6011b460 tx=0x7f5a500111e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:17.158 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:17.154+0000 7f5a577fe700 1 -- 172.21.15.42:0/1017445656 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f5a500322f0 con 0x7f5a6011c8f0 2019-12-05T06:44:17.327 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:17.322+0000 7f5a64955700 1 -- 172.21.15.42:0/1017445656 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5a6005b8b0 con 0x7f5a6011c8f0 2019-12-05T06:44:17.328 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:17.322+0000 7f5a5e59c700 1 --2- 172.21.15.42:0/1017445656 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5a6011c8f0 0x7f5a60120670 secure :-1 s=THROTTLE_DONE pgs=3134 cs=0 l=1 rx=0x7f5a6011b460 tx=0x7f5a500111e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:17.328 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:17.322+0000 7f5a577fe700 1 -- 172.21.15.42:0/1017445656 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 377 v377) v1 ==== 95+0+2108 (secure 0 0 0) 0x7f5a50032550 con 0x7f5a6011c8f0 2019-12-05T06:44:17.328 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 377 2019-12-05T06:44:17.328 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:44:17.328 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":377,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":12303,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/2489540211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2489540211},{"type":"v1","addr":"172.21.15.42:6835","nonce":2489540211}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":377}],"filesystems":[{"mdsmap":{"epoch":376,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:43:33.543372+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":117,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12496},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12496":{"gid":12496,"name":"b","rank":0,"incarnation":373,"state":"up:active","state_seq":11,"addr":"172.21.15.40:6833/3599619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3599619},{"type":"v1","addr":"172.21.15.40:6833","nonce":3599619}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:44:17.330 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:17.326+0000 7f5a64955700 1 -- 172.21.15.42:0/1017445656 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5a4403ad40 msgr2=0x7f5a4403d250 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:17.331 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:17.326+0000 7f5a64955700 1 --2- 172.21.15.42:0/1017445656 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5a4403ad40 0x7f5a4403d250 crc :-1 s=READY pgs=1308 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:17.331 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:17.326+0000 7f5a64955700 1 -- 172.21.15.42:0/1017445656 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5a6011c8f0 msgr2=0x7f5a60120670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:17.331 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:17.326+0000 7f5a64955700 1 --2- 172.21.15.42:0/1017445656 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5a6011c8f0 0x7f5a60120670 secure :-1 s=READY pgs=3134 cs=0 l=1 rx=0x7f5a6011b460 tx=0x7f5a500111e0).stop 2019-12-05T06:44:17.331 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:17.326+0000 7f5a64955700 1 -- 172.21.15.42:0/1017445656 shutdown_connections 2019-12-05T06:44:17.331 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:17.326+0000 7f5a64955700 1 --2- 172.21.15.42:0/1017445656 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5a4403ad40 0x7f5a4403d250 unknown :-1 s=CLOSED pgs=1308 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:17.331 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:17.326+0000 7f5a64955700 1 --2- 172.21.15.42:0/1017445656 >> v2:172.21.15.40:3300/0 conn(0x7f5a60120bc0 0x7f5a6011a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:17.331 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:17.326+0000 7f5a64955700 1 --2- 172.21.15.42:0/1017445656 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5a600a6c50 0x7f5a6012e360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:17.332 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:17.326+0000 7f5a64955700 1 --2- 172.21.15.42:0/1017445656 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5a6011c8f0 0x7f5a60120670 unknown :-1 s=CLOSED pgs=3134 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:17.332 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:17.326+0000 7f5a64955700 1 -- 172.21.15.42:0/1017445656 shutdown_connections 2019-12-05T06:44:17.332 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:17.326+0000 7f5a64955700 1 -- 172.21.15.42:0/1017445656 wait complete. 2019-12-05T06:44:17.332 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:17.326+0000 7f5a64955700 1 -- 172.21.15.42:0/1017445656 >> 172.21.15.42:0/1017445656 conn(0x7f5a600596f0 msgr2=0x7f5a60141460 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:44:17.340 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:44:19.341 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:44:19.446 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:19.438+0000 7f4702472700 1 Processor -- start 2019-12-05T06:44:19.446 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:19.438+0000 7f4702472700 1 -- start start 2019-12-05T06:44:19.446 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:19.438+0000 7f4702472700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f46fc0a7740 0x7f46fc08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:44:19.447 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:19.438+0000 7f4702472700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f46fc07e720 0x7f46fc07eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:44:19.447 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:19.438+0000 7f4702472700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f46fc09f2e0 0x7f46fc09f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:44:19.447 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:19.438+0000 7f4702472700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46fc0a0b00 con 0x7f46fc09f2e0 2019-12-05T06:44:19.447 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:19.438+0000 7f4702472700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46fc08e5b0 con 0x7f46fc07e720 2019-12-05T06:44:19.447 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:19.438+0000 7f4702472700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46fc08e850 con 0x7f46fc0a7740 2019-12-05T06:44:19.447 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:19.438+0000 7f46f3fff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f46fc07e720 0x7f46fc07eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:19.448 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:19.438+0000 7f46f3fff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f46fc07e720 0x7f46fc07eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:34660/0 (socket says 172.21.15.42:34660) 2019-12-05T06:44:19.449 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:19.438+0000 7f46f3fff700 1 -- 172.21.15.42:0/2131644167 learned_addr learned my addr 172.21.15.42:0/2131644167 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:44:19.449 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:19.438+0000 7f4700a0f700 1 --2- 172.21.15.42:0/2131644167 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f46fc0a7740 0x7f46fc08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:19.449 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:19.438+0000 7f4701210700 1 --2- 172.21.15.42:0/2131644167 >> v2:172.21.15.40:3300/0 conn(0x7f46fc09f2e0 0x7f46fc09f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:19.449 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:19.438+0000 7f46f3fff700 1 -- 172.21.15.42:0/2131644167 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f46fc0a7740 msgr2=0x7f46fc08df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:44:19.449 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:19.438+0000 7f46f3fff700 1 --2- 172.21.15.42:0/2131644167 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f46fc0a7740 0x7f46fc08df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:44:19.450 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:19.438+0000 7f46f3fff700 1 -- 172.21.15.42:0/2131644167 >> v2:172.21.15.40:3300/0 conn(0x7f46fc09f2e0 msgr2=0x7f46fc09f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:44:19.450 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:19.438+0000 7f46f3fff700 1 --2- 172.21.15.42:0/2131644167 >> v2:172.21.15.40:3300/0 conn(0x7f46fc09f2e0 0x7f46fc09f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:44:19.450 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:19.438+0000 7f46f3fff700 1 -- 172.21.15.42:0/2131644167 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46fc07f0a0 con 0x7f46fc07e720 2019-12-05T06:44:19.450 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:19.438+0000 7f46f3fff700 1 --2- 172.21.15.42:0/2131644167 >> v2:172.21.15.42:3300/0 conn(0x7f46fc07e720 0x7f46fc07eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f46fc08e770 tx=0x7f46ec00fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:19.450 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:19.438+0000 7f46f3fff700 1 --2- 172.21.15.42:0/2131644167 >> v2:172.21.15.42:3300/0 conn(0x7f46fc07e720 0x7f46fc07eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f46fc08e770 tx=0x7f46ec00fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:19.450 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:19.438+0000 7f46f3fff700 1 --2- 172.21.15.42:0/2131644167 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f46fc07e720 0x7f46fc07eb50 secure :-1 s=READY pgs=3135 cs=0 l=1 rx=0x7f46fc08e770 tx=0x7f46ec00fdb0).ready entity=mon.1 client_cookie=b4ef029a1b6e554c server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:44:19.450 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:19.438+0000 7f46f3fff700 1 --2- 172.21.15.42:0/2131644167 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f46fc07e720 0x7f46fc07eb50 secure :-1 s=THROTTLE_DONE pgs=3135 cs=0 l=1 rx=0x7f46fc08e770 tx=0x7f46ec00fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:19.451 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:19.438+0000 7f46f37fe700 1 -- 172.21.15.42:0/2131644167 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f46ec008c90 con 0x7f46fc07e720 2019-12-05T06:44:19.451 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:19.438+0000 7f46f3fff700 1 --2- 172.21.15.42:0/2131644167 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f46fc07e720 0x7f46fc07eb50 secure :-1 s=THROTTLE_DONE pgs=3135 cs=0 l=1 rx=0x7f46fc08e770 tx=0x7f46ec00fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:19.451 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:19.438+0000 7f46f3fff700 1 --2- 172.21.15.42:0/2131644167 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f46fc07e720 0x7f46fc07eb50 secure :-1 s=THROTTLE_DONE pgs=3135 cs=0 l=1 rx=0x7f46fc08e770 tx=0x7f46ec00fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:19.451 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:19.438+0000 7f46f37fe700 1 -- 172.21.15.42:0/2131644167 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46ec0088b0 con 0x7f46fc07e720 2019-12-05T06:44:19.451 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:19.438+0000 7f46f37fe700 1 -- 172.21.15.42:0/2131644167 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f46ec011b50 con 0x7f46fc07e720 2019-12-05T06:44:19.451 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:19.438+0000 7f4702472700 1 -- 172.21.15.42:0/2131644167 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f46fc07e720 msgr2=0x7f46fc07eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:19.452 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:19.438+0000 7f4702472700 1 --2- 172.21.15.42:0/2131644167 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f46fc07e720 0x7f46fc07eb50 secure :-1 s=READY pgs=3135 cs=0 l=1 rx=0x7f46fc08e770 tx=0x7f46ec00fdb0).stop 2019-12-05T06:44:19.452 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:19.442+0000 7f4702472700 1 -- 172.21.15.42:0/2131644167 shutdown_connections 2019-12-05T06:44:19.452 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:19.442+0000 7f4702472700 1 --2- 172.21.15.42:0/2131644167 >> v2:172.21.15.40:3300/0 conn(0x7f46fc09f2e0 0x7f46fc09f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:44:19.452 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:19.442+0000 7f4702472700 1 --2- 172.21.15.42:0/2131644167 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f46fc0a7740 0x7f46fc08df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:44:19.453 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:19.442+0000 7f4702472700 1 --2- 172.21.15.42:0/2131644167 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f46fc07e720 0x7f46fc07eb50 unknown :-1 s=CLOSED pgs=3135 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:19.453 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:19.442+0000 7f4702472700 1 -- 172.21.15.42:0/2131644167 shutdown_connections 2019-12-05T06:44:19.453 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:19.442+0000 7f4702472700 1 -- 172.21.15.42:0/2131644167 wait complete. 2019-12-05T06:44:19.453 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:19.442+0000 7f4702472700 1 -- 172.21.15.42:0/2131644167 >> 172.21.15.42:0/2131644167 conn(0x7f46fc0aabe0 msgr2=0x7f46fc0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:44:19.460 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:19.442+0000 7f4702472700 1 Processor -- start 2019-12-05T06:44:19.460 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:19.446+0000 7f4702472700 1 -- start start 2019-12-05T06:44:19.460 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:19.446+0000 7f4702472700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f46fc0a6d90 0x7f46fc11ab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:44:19.460 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:19.446+0000 7f4702472700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f46fc1314b0 0x7f46fc1318e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:44:19.460 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:19.446+0000 7f4702472700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46fc131e30 con 0x7f46fc142cc0 2019-12-05T06:44:19.461 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:19.446+0000 7f4702472700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46fc11b130 con 0x7f46fc0a6d90 2019-12-05T06:44:19.461 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:19.446+0000 7f4702472700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46fc11b3d0 con 0x7f46fc1314b0 2019-12-05T06:44:19.461 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:19.446+0000 7f4700a0f700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f46fc0a6d90 0x7f46fc11ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:19.461 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:19.446+0000 7f4700a0f700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f46fc0a6d90 0x7f46fc11ab00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:34664/0 (socket says 172.21.15.42:34664) 2019-12-05T06:44:19.461 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:19.446+0000 7f46f3fff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f46fc142cc0 0x7f46fc07bb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:34704/0 (socket says 172.21.15.42:34704) 2019-12-05T06:44:19.461 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:19.446+0000 7f4700a0f700 1 -- 172.21.15.42:0/2790740195 learned_addr learned my addr 172.21.15.42:0/2790740195 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:44:19.462 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:19.446+0000 7f4701210700 1 --2- 172.21.15.42:0/2790740195 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f46fc1314b0 0x7f46fc1318e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:19.462 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:19.446+0000 7f4700a0f700 1 -- 172.21.15.42:0/2790740195 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f46fc1314b0 msgr2=0x7f46fc1318e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:19.462 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:19.446+0000 7f4700a0f700 1 --2- 172.21.15.42:0/2790740195 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f46fc1314b0 0x7f46fc1318e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:19.462 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:19.446+0000 7f4700a0f700 1 -- 172.21.15.42:0/2790740195 >> v1:172.21.15.42:6789/0 conn(0x7f46fc142cc0 legacy=0x7f46fc07bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:19.462 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:19.446+0000 7f4700a0f700 1 -- 172.21.15.42:0/2790740195 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46fc139800 con 0x7f46fc0a6d90 2019-12-05T06:44:19.462 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:19.446+0000 7f4700a0f700 1 --2- 172.21.15.42:0/2790740195 >> v2:172.21.15.42:3300/0 conn(0x7f46fc0a6d90 0x7f46fc11ab00 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f46fc1313d0 tx=0x7f46f8010a10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:19.463 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:19.446+0000 7f4700a0f700 1 --2- 172.21.15.42:0/2790740195 >> v2:172.21.15.42:3300/0 conn(0x7f46fc0a6d90 0x7f46fc11ab00 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f46fc1313d0 tx=0x7f46f8010a10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:19.463 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:19.446+0000 7f4700a0f700 1 --2- 172.21.15.42:0/2790740195 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f46fc0a6d90 0x7f46fc11ab00 secure :-1 s=READY pgs=3137 cs=0 l=1 rx=0x7f46fc1313d0 tx=0x7f46f8010a10).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:44:19.463 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:19.446+0000 7f4700a0f700 1 --2- 172.21.15.42:0/2790740195 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f46fc0a6d90 0x7f46fc11ab00 secure :-1 s=THROTTLE_DONE pgs=3137 cs=0 l=1 rx=0x7f46fc1313d0 tx=0x7f46f8010a10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:19.463 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:19.446+0000 7f46f17fa700 1 -- 172.21.15.42:0/2790740195 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f46f8005930 con 0x7f46fc0a6d90 2019-12-05T06:44:19.463 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:19.446+0000 7f4700a0f700 1 --2- 172.21.15.42:0/2790740195 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f46fc0a6d90 0x7f46fc11ab00 secure :-1 s=THROTTLE_DONE pgs=3137 cs=0 l=1 rx=0x7f46fc1313d0 tx=0x7f46f8010a10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:19.464 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:19.446+0000 7f4700a0f700 1 --2- 172.21.15.42:0/2790740195 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f46fc0a6d90 0x7f46fc11ab00 secure :-1 s=THROTTLE_DONE pgs=3137 cs=0 l=1 rx=0x7f46fc1313d0 tx=0x7f46f8010a10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:19.464 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:19.446+0000 7f46f17fa700 1 -- 172.21.15.42:0/2790740195 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46f8007160 con 0x7f46fc0a6d90 2019-12-05T06:44:19.464 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:19.446+0000 7f4702472700 1 -- 172.21.15.42:0/2790740195 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46fc1387f0 con 0x7f46fc0a6d90 2019-12-05T06:44:19.464 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:19.446+0000 7f46f17fa700 1 -- 172.21.15.42:0/2790740195 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f46f80060c0 con 0x7f46fc0a6d90 2019-12-05T06:44:19.464 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:19.446+0000 7f4702472700 1 -- 172.21.15.42:0/2790740195 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46fc138cf0 con 0x7f46fc0a6d90 2019-12-05T06:44:19.464 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:19.450+0000 7f4700a0f700 1 --2- 172.21.15.42:0/2790740195 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f46fc0a6d90 0x7f46fc11ab00 secure :-1 s=THROTTLE_DONE pgs=3137 cs=0 l=1 rx=0x7f46fc1313d0 tx=0x7f46f8010a10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:19.465 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:19.450+0000 7f46f17fa700 1 -- 172.21.15.42:0/2790740195 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f46f8007a20 con 0x7f46fc0a6d90 2019-12-05T06:44:19.465 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:19.450+0000 7f4700a0f700 1 --2- 172.21.15.42:0/2790740195 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f46fc0a6d90 0x7f46fc11ab00 secure :-1 s=THROTTLE_DONE pgs=3137 cs=0 l=1 rx=0x7f46fc1313d0 tx=0x7f46f8010a10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:19.465 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:19.450+0000 7f46f17fa700 1 --2- 172.21.15.42:0/2790740195 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f46dc03ae10 0x7f46dc03d320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:44:19.465 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:19.450+0000 7f46f17fa700 1 -- 172.21.15.42:0/2790740195 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 11879+0+0 (secure 0 0 0) 0x7f46f804ab30 con 0x7f46fc0a6d90 2019-12-05T06:44:19.465 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:19.450+0000 7f46f3fff700 1 --2- 172.21.15.42:0/2790740195 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f46dc03ae10 0x7f46dc03d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:19.465 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:19.450+0000 7f46f3fff700 1 --2- 172.21.15.42:0/2790740195 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f46dc03ae10 0x7f46dc03d320 crc :-1 s=READY pgs=1309 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:44:19.466 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:19.454+0000 7f4702472700 1 -- 172.21.15.42:0/2790740195 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46fc05d110 con 0x7f46fc0a6d90 2019-12-05T06:44:19.466 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:19.458+0000 7f4700a0f700 1 --2- 172.21.15.42:0/2790740195 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f46fc0a6d90 0x7f46fc11ab00 secure :-1 s=THROTTLE_DONE pgs=3137 cs=0 l=1 rx=0x7f46fc1313d0 tx=0x7f46f8010a10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:19.466 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:19.458+0000 7f46f17fa700 1 -- 172.21.15.42:0/2790740195 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f46f8031bb0 con 0x7f46fc0a6d90 2019-12-05T06:44:19.636 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:19.630+0000 7f4702472700 1 -- 172.21.15.42:0/2790740195 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f46fc05b8c0 con 0x7f46fc0a6d90 2019-12-05T06:44:19.637 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:19.634+0000 7f4700a0f700 1 --2- 172.21.15.42:0/2790740195 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f46fc0a6d90 0x7f46fc11ab00 secure :-1 s=THROTTLE_DONE pgs=3137 cs=0 l=1 rx=0x7f46fc1313d0 tx=0x7f46f8010a10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:19.638 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:19.634+0000 7f46f17fa700 1 -- 172.21.15.42:0/2790740195 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 377 v377) v1 ==== 95+0+2108 (secure 0 0 0) 0x7f46f8031e10 con 0x7f46fc0a6d90 2019-12-05T06:44:19.638 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 377 2019-12-05T06:44:19.638 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:44:19.638 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":377,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":12303,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/2489540211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2489540211},{"type":"v1","addr":"172.21.15.42:6835","nonce":2489540211}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":377}],"filesystems":[{"mdsmap":{"epoch":376,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:43:33.543372+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":117,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12496},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12496":{"gid":12496,"name":"b","rank":0,"incarnation":373,"state":"up:active","state_seq":11,"addr":"172.21.15.40:6833/3599619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3599619},{"type":"v1","addr":"172.21.15.40:6833","nonce":3599619}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:44:19.639 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:19.634+0000 7f4702472700 1 -- 172.21.15.42:0/2790740195 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f46dc03ae10 msgr2=0x7f46dc03d320 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:19.640 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:19.634+0000 7f4702472700 1 --2- 172.21.15.42:0/2790740195 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f46dc03ae10 0x7f46dc03d320 crc :-1 s=READY pgs=1309 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:19.640 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:19.634+0000 7f4702472700 1 -- 172.21.15.42:0/2790740195 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f46fc0a6d90 msgr2=0x7f46fc11ab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:19.640 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:19.634+0000 7f4702472700 1 --2- 172.21.15.42:0/2790740195 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f46fc0a6d90 0x7f46fc11ab00 secure :-1 s=READY pgs=3137 cs=0 l=1 rx=0x7f46fc1313d0 tx=0x7f46f8010a10).stop 2019-12-05T06:44:19.640 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:19.634+0000 7f4702472700 1 -- 172.21.15.42:0/2790740195 shutdown_connections 2019-12-05T06:44:19.640 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:19.634+0000 7f4702472700 1 --2- 172.21.15.42:0/2790740195 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f46dc03ae10 0x7f46dc03d320 unknown :-1 s=CLOSED pgs=1309 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:19.640 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:19.634+0000 7f4702472700 1 --2- 172.21.15.42:0/2790740195 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f46fc1314b0 0x7f46fc1318e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:19.640 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:19.634+0000 7f4702472700 1 --2- 172.21.15.42:0/2790740195 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f46fc0a6d90 0x7f46fc11ab00 unknown :-1 s=CLOSED pgs=3137 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:19.641 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:19.634+0000 7f4702472700 1 -- 172.21.15.42:0/2790740195 shutdown_connections 2019-12-05T06:44:19.641 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:19.634+0000 7f4702472700 1 -- 172.21.15.42:0/2790740195 wait complete. 2019-12-05T06:44:19.641 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:19.634+0000 7f4702472700 1 -- 172.21.15.42:0/2790740195 >> 172.21.15.42:0/2790740195 conn(0x7f46fc09f450 msgr2=0x7f46fc11bcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:44:19.649 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:44:21.650 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:44:21.755 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:21.746+0000 7fdbed44a700 1 Processor -- start 2019-12-05T06:44:21.755 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:21.746+0000 7fdbed44a700 1 -- start start 2019-12-05T06:44:21.755 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:21.746+0000 7fdbed44a700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdbe80a7740 0x7fdbe808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:44:21.755 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:21.746+0000 7fdbed44a700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fdbe807e720 0x7fdbe807eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:44:21.756 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:21.746+0000 7fdbed44a700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbe80a0710 con 0x7fdbe809f2e0 2019-12-05T06:44:21.756 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:21.746+0000 7fdbed44a700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdbe80a0b00 con 0x7fdbe807e720 2019-12-05T06:44:21.756 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:21.746+0000 7fdbed44a700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbe808e5b0 con 0x7fdbe80a7740 2019-12-05T06:44:21.756 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:21.746+0000 7fdbe7fff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fdbe809f2e0 0x7fdbe809f710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:34714/0 (socket says 172.21.15.42:34714) 2019-12-05T06:44:21.756 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:21.746+0000 7fdbe7fff700 1 -- 172.21.15.42:0/3034749786 learned_addr learned my addr 172.21.15.42:0/3034749786 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:44:21.756 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:21.746+0000 7fdbe77fe700 1 --2- 172.21.15.42:0/3034749786 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdbe80a7740 0x7fdbe808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:21.756 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:21.746+0000 7fdbe6ffd700 1 --2- 172.21.15.42:0/3034749786 >> v2:172.21.15.40:3300/0 conn(0x7fdbe807e720 0x7fdbe807eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:21.757 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:21.750+0000 7fdbe67fc700 1 -- 172.21.15.42:0/3034749786 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 733468504 0 0) 0x7fdbe80a0710 con 0x7fdbe809f2e0 2019-12-05T06:44:21.757 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:21.750+0000 7fdbe67fc700 1 -- 172.21.15.42:0/3034749786 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdbd00030b0 con 0x7fdbe809f2e0 2019-12-05T06:44:21.757 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:21.750+0000 7fdbe77fe700 1 -- 172.21.15.42:0/3034749786 >> v2:172.21.15.40:3300/0 conn(0x7fdbe807e720 msgr2=0x7fdbe807eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:44:21.757 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:21.750+0000 7fdbe77fe700 1 --2- 172.21.15.42:0/3034749786 >> v2:172.21.15.40:3300/0 conn(0x7fdbe807e720 0x7fdbe807eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:44:21.757 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:21.750+0000 7fdbe77fe700 1 -- 172.21.15.42:0/3034749786 >> v1:172.21.15.42:6789/0 conn(0x7fdbe809f2e0 legacy=0x7fdbe809f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:21.757 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:21.750+0000 7fdbe77fe700 1 -- 172.21.15.42:0/3034749786 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbe807f0a0 con 0x7fdbe80a7740 2019-12-05T06:44:21.757 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:21.750+0000 7fdbe6ffd700 1 --2- 172.21.15.42:0/3034749786 >> v2:172.21.15.40:3300/0 conn(0x7fdbe807e720 0x7fdbe807eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:44:21.758 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:21.750+0000 7fdbe77fe700 1 --2- 172.21.15.42:0/3034749786 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdbe80a7740 0x7fdbe808df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fdbd8008790 tx=0x7fdbd800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:21.758 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:21.750+0000 7fdbe77fe700 1 --2- 172.21.15.42:0/3034749786 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdbe80a7740 0x7fdbe808df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fdbd8008790 tx=0x7fdbd800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:21.758 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:21.750+0000 7fdbe77fe700 1 --2- 172.21.15.42:0/3034749786 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdbe80a7740 0x7fdbe808df80 secure :-1 s=READY pgs=681 cs=0 l=1 rx=0x7fdbd8008790 tx=0x7fdbd800fc70).ready entity=mon.2 client_cookie=662405d9f569f008 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:44:21.758 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:21.750+0000 7fdbe77fe700 1 --2- 172.21.15.42:0/3034749786 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdbe80a7740 0x7fdbe808df80 secure :-1 s=THROTTLE_DONE pgs=681 cs=0 l=1 rx=0x7fdbd8008790 tx=0x7fdbd800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:21.758 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:21.750+0000 7fdbe67fc700 1 -- 172.21.15.42:0/3034749786 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdbd80112f0 con 0x7fdbe80a7740 2019-12-05T06:44:21.758 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:21.750+0000 7fdbe77fe700 1 --2- 172.21.15.42:0/3034749786 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdbe80a7740 0x7fdbe808df80 secure :-1 s=THROTTLE_DONE pgs=681 cs=0 l=1 rx=0x7fdbd8008790 tx=0x7fdbd800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:21.759 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:21.750+0000 7fdbe77fe700 1 --2- 172.21.15.42:0/3034749786 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdbe80a7740 0x7fdbe808df80 secure :-1 s=THROTTLE_DONE pgs=681 cs=0 l=1 rx=0x7fdbd8008790 tx=0x7fdbd800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:21.759 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:21.750+0000 7fdbe67fc700 1 -- 172.21.15.42:0/3034749786 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbd8008ad0 con 0x7fdbe80a7740 2019-12-05T06:44:21.760 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:21.750+0000 7fdbe67fc700 1 -- 172.21.15.42:0/3034749786 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdbd80118d0 con 0x7fdbe80a7740 2019-12-05T06:44:21.760 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:21.750+0000 7fdbed44a700 1 -- 172.21.15.42:0/3034749786 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdbe80a7740 msgr2=0x7fdbe808df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:21.760 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:21.750+0000 7fdbed44a700 1 --2- 172.21.15.42:0/3034749786 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdbe80a7740 0x7fdbe808df80 secure :-1 s=READY pgs=681 cs=0 l=1 rx=0x7fdbd8008790 tx=0x7fdbd800fc70).stop 2019-12-05T06:44:21.760 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:21.750+0000 7fdbed44a700 1 -- 172.21.15.42:0/3034749786 shutdown_connections 2019-12-05T06:44:21.760 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:21.750+0000 7fdbed44a700 1 --2- 172.21.15.42:0/3034749786 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdbe80a7740 0x7fdbe808df80 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:21.760 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:21.750+0000 7fdbed44a700 1 --2- 172.21.15.42:0/3034749786 >> v2:172.21.15.40:3300/0 conn(0x7fdbe807e720 0x7fdbe807eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:44:21.761 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:21.750+0000 7fdbed44a700 1 -- 172.21.15.42:0/3034749786 shutdown_connections 2019-12-05T06:44:21.761 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:21.750+0000 7fdbed44a700 1 -- 172.21.15.42:0/3034749786 wait complete. 2019-12-05T06:44:21.761 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:21.750+0000 7fdbed44a700 1 -- 172.21.15.42:0/3034749786 >> 172.21.15.42:0/3034749786 conn(0x7fdbe80aabe0 msgr2=0x7fdbe80a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:44:21.781 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:21.754+0000 7fdbed44a700 1 Processor -- start 2019-12-05T06:44:21.781 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:21.754+0000 7fdbed44a700 1 -- start start 2019-12-05T06:44:21.781 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:21.754+0000 7fdbed44a700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdbe8076b00 0x7fdbe8078f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:44:21.781 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:21.754+0000 7fdbed44a700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbe8117a60 con 0x7fdbe80a3a20 2019-12-05T06:44:21.782 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:21.754+0000 7fdbed44a700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbe8118530 con 0x7fdbe811a970 2019-12-05T06:44:21.782 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:21.754+0000 7fdbed44a700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbe811ada0 con 0x7fdbe8076b00 2019-12-05T06:44:21.782 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:21.754+0000 7fdbe6ffd700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fdbe811a970 0x7fdbe80752f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:34718/0 (socket says 172.21.15.42:34718) 2019-12-05T06:44:21.782 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:21.754+0000 7fdbe6ffd700 1 -- 172.21.15.42:0/2836025958 learned_addr learned my addr 172.21.15.42:0/2836025958 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:44:21.783 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:21.754+0000 7fdbe77fe700 1 --1- 172.21.15.42:0/2836025958 >> v1:172.21.15.40:6789/0 conn(0x7fdbe80a3a20 0x7fdbe8117d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:44156/0 (socket says 172.21.15.42:44156) 2019-12-05T06:44:21.783 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:21.754+0000 7fdbe7fff700 1 --2- 172.21.15.42:0/2836025958 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdbe8076b00 0x7fdbe8078f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:21.783 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:21.754+0000 7fdbcffff700 1 -- 172.21.15.42:0/2836025958 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 872235920 0 0) 0x7fdbe8118530 con 0x7fdbe811a970 2019-12-05T06:44:21.784 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:21.754+0000 7fdbcffff700 1 -- 172.21.15.42:0/2836025958 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdbc80030b0 con 0x7fdbe811a970 2019-12-05T06:44:21.784 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:21.754+0000 7fdbcffff700 1 -- 172.21.15.42:0/2836025958 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1233188181 0 0) 0x7fdbe8117a60 con 0x7fdbe80a3a20 2019-12-05T06:44:21.784 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:21.754+0000 7fdbcffff700 1 -- 172.21.15.42:0/2836025958 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdbe8118530 con 0x7fdbe80a3a20 2019-12-05T06:44:21.785 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:21.754+0000 7fdbe7fff700 1 -- 172.21.15.42:0/2836025958 >> v1:172.21.15.42:6789/0 conn(0x7fdbe811a970 legacy=0x7fdbe80752f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:21.785 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:21.754+0000 7fdbcffff700 1 -- 172.21.15.42:0/2836025958 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 3881670179 0 0) 0x7fdbc80030b0 con 0x7fdbe811a970 2019-12-05T06:44:21.785 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:21.754+0000 7fdbe7fff700 1 -- 172.21.15.42:0/2836025958 >> v1:172.21.15.40:6789/0 conn(0x7fdbe80a3a20 legacy=0x7fdbe8117d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:21.785 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:21.754+0000 7fdbe7fff700 1 -- 172.21.15.42:0/2836025958 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbe811a010 con 0x7fdbe8076b00 2019-12-05T06:44:21.786 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:21.754+0000 7fdbe7fff700 1 --2- 172.21.15.42:0/2836025958 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdbe8076b00 0x7fdbe8078f40 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fdbdc0022d0 tx=0x7fdbdc007720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:21.786 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:21.754+0000 7fdbe7fff700 1 --2- 172.21.15.42:0/2836025958 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdbe8076b00 0x7fdbe8078f40 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fdbdc0022d0 tx=0x7fdbdc007720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:21.786 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:21.754+0000 7fdbe7fff700 1 --2- 172.21.15.42:0/2836025958 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdbe8076b00 0x7fdbe8078f40 secure :-1 s=READY pgs=682 cs=0 l=1 rx=0x7fdbdc0022d0 tx=0x7fdbdc007720).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:44:21.786 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:21.754+0000 7fdbe7fff700 1 --2- 172.21.15.42:0/2836025958 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdbe8076b00 0x7fdbe8078f40 secure :-1 s=THROTTLE_DONE pgs=682 cs=0 l=1 rx=0x7fdbdc0022d0 tx=0x7fdbdc007720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:21.787 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:21.754+0000 7fdbcffff700 1 -- 172.21.15.42:0/2836025958 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdbdc002500 con 0x7fdbe8076b00 2019-12-05T06:44:21.787 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:21.754+0000 7fdbe7fff700 1 --2- 172.21.15.42:0/2836025958 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdbe8076b00 0x7fdbe8078f40 secure :-1 s=THROTTLE_DONE pgs=682 cs=0 l=1 rx=0x7fdbdc0022d0 tx=0x7fdbdc007720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:21.787 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:21.758+0000 7fdbcffff700 1 -- 172.21.15.42:0/2836025958 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbdc002310 con 0x7fdbe8076b00 2019-12-05T06:44:21.788 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:21.758+0000 7fdbe7fff700 1 --2- 172.21.15.42:0/2836025958 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdbe8076b00 0x7fdbe8078f40 secure :-1 s=THROTTLE_DONE pgs=682 cs=0 l=1 rx=0x7fdbdc0022d0 tx=0x7fdbdc007720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:21.788 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:21.758+0000 7fdbed44a700 1 -- 172.21.15.42:0/2836025958 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbe811b5e0 con 0x7fdbe8076b00 2019-12-05T06:44:21.789 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:21.758+0000 7fdbcffff700 1 -- 172.21.15.42:0/2836025958 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdbe00062b0 con 0x7fdbe8076b00 2019-12-05T06:44:21.789 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:21.758+0000 7fdbed44a700 1 -- 172.21.15.42:0/2836025958 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbe8079490 con 0x7fdbe8076b00 2019-12-05T06:44:21.789 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:21.758+0000 7fdbe7fff700 1 --2- 172.21.15.42:0/2836025958 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdbe8076b00 0x7fdbe8078f40 secure :-1 s=THROTTLE_DONE pgs=682 cs=0 l=1 rx=0x7fdbdc0022d0 tx=0x7fdbdc007720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:21.789 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:21.758+0000 7fdbcffff700 1 -- 172.21.15.42:0/2836025958 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fdbe0006540 con 0x7fdbe8076b00 2019-12-05T06:44:21.790 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:21.758+0000 7fdbe7fff700 1 --2- 172.21.15.42:0/2836025958 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdbe8076b00 0x7fdbe8078f40 secure :-1 s=THROTTLE_DONE pgs=682 cs=0 l=1 rx=0x7fdbdc0022d0 tx=0x7fdbdc007720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:21.790 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:21.758+0000 7fdbcffff700 1 --2- 172.21.15.42:0/2836025958 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fdbc803b3a0 0x7fdbc803d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:44:21.790 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:21.758+0000 7fdbcffff700 1 -- 172.21.15.42:0/2836025958 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 11879+0+0 (secure 0 0 0) 0x7fdbe004abe0 con 0x7fdbe8076b00 2019-12-05T06:44:21.790 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:21.758+0000 7fdbe77fe700 1 --2- 172.21.15.42:0/2836025958 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fdbc803b3a0 0x7fdbc803d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:21.791 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:21.758+0000 7fdbe77fe700 1 --2- 172.21.15.42:0/2836025958 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fdbc803b3a0 0x7fdbc803d8b0 crc :-1 s=READY pgs=1310 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:44:21.791 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:21.762+0000 7fdbed44a700 1 -- 172.21.15.42:0/2836025958 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbe805d110 con 0x7fdbe8076b00 2019-12-05T06:44:21.791 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:21.770+0000 7fdbe7fff700 1 --2- 172.21.15.42:0/2836025958 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdbe8076b00 0x7fdbe8078f40 secure :-1 s=THROTTLE_DONE pgs=682 cs=0 l=1 rx=0x7fdbdc0022d0 tx=0x7fdbdc007720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:21.791 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:21.770+0000 7fdbcffff700 1 -- 172.21.15.42:0/2836025958 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fdbe0031c40 con 0x7fdbe8076b00 2019-12-05T06:44:21.945 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:21.942+0000 7fdbed44a700 1 -- 172.21.15.42:0/2836025958 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdbe805b8c0 con 0x7fdbe8076b00 2019-12-05T06:44:21.946 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:21.942+0000 7fdbe7fff700 1 --2- 172.21.15.42:0/2836025958 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdbe8076b00 0x7fdbe8078f40 secure :-1 s=THROTTLE_DONE pgs=682 cs=0 l=1 rx=0x7fdbdc0022d0 tx=0x7fdbdc007720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:21.946 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:21.942+0000 7fdbcffff700 1 -- 172.21.15.42:0/2836025958 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 377 v377) v1 ==== 95+0+2108 (secure 0 0 0) 0x7fdbe0031ea0 con 0x7fdbe8076b00 2019-12-05T06:44:21.948 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 377 2019-12-05T06:44:21.948 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:44:21.948 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":377,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":12303,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/2489540211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2489540211},{"type":"v1","addr":"172.21.15.42:6835","nonce":2489540211}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":377}],"filesystems":[{"mdsmap":{"epoch":376,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:43:33.543372+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":117,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12496},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12496":{"gid":12496,"name":"b","rank":0,"incarnation":373,"state":"up:active","state_seq":11,"addr":"172.21.15.40:6833/3599619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3599619},{"type":"v1","addr":"172.21.15.40:6833","nonce":3599619}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:44:21.957 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:21.946+0000 7fdbed44a700 1 -- 172.21.15.42:0/2836025958 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fdbc803b3a0 msgr2=0x7fdbc803d8b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:21.957 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:21.946+0000 7fdbed44a700 1 --2- 172.21.15.42:0/2836025958 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fdbc803b3a0 0x7fdbc803d8b0 crc :-1 s=READY pgs=1310 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:21.958 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:21.946+0000 7fdbed44a700 1 -- 172.21.15.42:0/2836025958 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdbe8076b00 msgr2=0x7fdbe8078f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:21.958 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:21.946+0000 7fdbed44a700 1 --2- 172.21.15.42:0/2836025958 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdbe8076b00 0x7fdbe8078f40 secure :-1 s=READY pgs=682 cs=0 l=1 rx=0x7fdbdc0022d0 tx=0x7fdbdc007720).stop 2019-12-05T06:44:21.958 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:21.946+0000 7fdbed44a700 1 -- 172.21.15.42:0/2836025958 shutdown_connections 2019-12-05T06:44:21.958 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:21.946+0000 7fdbed44a700 1 --2- 172.21.15.42:0/2836025958 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fdbc803b3a0 0x7fdbc803d8b0 unknown :-1 s=CLOSED pgs=1310 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:21.958 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:21.946+0000 7fdbed44a700 1 --2- 172.21.15.42:0/2836025958 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdbe8076b00 0x7fdbe8078f40 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:21.959 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:21.946+0000 7fdbed44a700 1 -- 172.21.15.42:0/2836025958 shutdown_connections 2019-12-05T06:44:21.959 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:21.946+0000 7fdbed44a700 1 -- 172.21.15.42:0/2836025958 wait complete. 2019-12-05T06:44:21.959 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:21.946+0000 7fdbed44a700 1 -- 172.21.15.42:0/2836025958 >> 172.21.15.42:0/2836025958 conn(0x7fdbe80a82a0 msgr2=0x7fdbe80a9790 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:44:21.965 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:44:23.966 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:44:24.069 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:24.058+0000 7f941f872700 1 Processor -- start 2019-12-05T06:44:24.069 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:24.058+0000 7f941f872700 1 -- start start 2019-12-05T06:44:24.069 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:24.058+0000 7f941f872700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f94180a7740 0x7f941808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:44:24.069 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:24.058+0000 7f941f872700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f941809fae0 0x7f941807eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:44:24.069 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:24.058+0000 7f941f872700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94180a0710 con 0x7f941807e720 2019-12-05T06:44:24.071 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:24.058+0000 7f941f872700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94180a0b00 con 0x7f94180a7740 2019-12-05T06:44:24.071 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:24.058+0000 7f941f872700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f941808e5b0 con 0x7f941809fae0 2019-12-05T06:44:24.072 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:24.058+0000 7f941e610700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f941809fae0 0x7f941807eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:24.072 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:24.058+0000 7f941de0f700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f94180a7740 0x7f941808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:24.072 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:24.058+0000 7f941e610700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f941809fae0 0x7f941807eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:35158/0 (socket says 172.21.15.42:35158) 2019-12-05T06:44:24.072 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:24.058+0000 7f941e610700 1 -- 172.21.15.42:0/1249515352 learned_addr learned my addr 172.21.15.42:0/1249515352 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:44:24.072 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:24.062+0000 7f941e610700 1 -- 172.21.15.42:0/1249515352 >> v2:172.21.15.40:3300/0 conn(0x7f94180a7740 msgr2=0x7f941808df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:44:24.072 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:24.062+0000 7f941e610700 1 --2- 172.21.15.42:0/1249515352 >> v2:172.21.15.40:3300/0 conn(0x7f94180a7740 0x7f941808df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:44:24.072 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:24.062+0000 7f941e610700 1 -- 172.21.15.42:0/1249515352 >> v1:172.21.15.40:6789/0 conn(0x7f941807e720 legacy=0x7f941809f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:24.073 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:24.062+0000 7f941e610700 1 -- 172.21.15.42:0/1249515352 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f941807f0a0 con 0x7f941809fae0 2019-12-05T06:44:24.073 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:24.062+0000 7f941e610700 1 --2- 172.21.15.42:0/1249515352 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f941809fae0 0x7f941807eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f9410008b10 tx=0x7f94100100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:24.073 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:24.062+0000 7f941e610700 1 --2- 172.21.15.42:0/1249515352 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f941809fae0 0x7f941807eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f9410008b10 tx=0x7f94100100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:24.073 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:24.062+0000 7f941e610700 1 --2- 172.21.15.42:0/1249515352 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f941809fae0 0x7f941807eb50 secure :-1 s=READY pgs=683 cs=0 l=1 rx=0x7f9410008b10 tx=0x7f94100100b0).ready entity=mon.2 client_cookie=a5fed376e1e17b36 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:44:24.073 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:24.062+0000 7f941e610700 1 --2- 172.21.15.42:0/1249515352 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f941809fae0 0x7f941807eb50 secure :-1 s=THROTTLE_DONE pgs=683 cs=0 l=1 rx=0x7f9410008b10 tx=0x7f94100100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:24.073 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:24.062+0000 7f941e610700 1 -- 172.21.15.42:0/1249515352 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f941809fae0 msgr2=0x7f941807eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2019-12-05T06:44:24.074 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:24.062+0000 7f941e610700 1 -- 172.21.15.42:0/1249515352 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f941809fae0 msgr2=0x7f941807eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2019-12-05T06:44:24.074 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:24.062+0000 7f941e610700 1 --2- 172.21.15.42:0/1249515352 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f941809fae0 0x7f941807eb50 secure :-1 s=READY pgs=683 cs=0 l=1 rx=0x7f9410008b10 tx=0x7f94100100b0).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-12-05T06:44:24.074 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:24.062+0000 7f941ce0d700 1 -- 172.21.15.42:0/1249515352 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9410010da0 con 0x7f941809fae0 2019-12-05T06:44:24.074 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:24.062+0000 7f941e610700 1 --2- 172.21.15.42:0/1249515352 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f941809fae0 0x7f941807eb50 secure :-1 s=READY pgs=683 cs=0 l=1 rx=0x7f9410008b10 tx=0x7f94100100b0).stop 2019-12-05T06:44:24.074 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:24.062+0000 7f941ce0d700 1 -- 172.21.15.42:0/1249515352 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f941809fae0 msgr2=0x7f941807eb50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2019-12-05T06:44:24.074 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:24.062+0000 7f941ce0d700 1 --2- 172.21.15.42:0/1249515352 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f941809fae0 0x7f941807eb50 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:24.074 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:24.062+0000 7f941ce0d700 1 --2- 172.21.15.42:0/1249515352 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9404047000 0x7f94040494a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:44:24.075 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:24.062+0000 7f941ce0d700 1 --2- 172.21.15.42:0/1249515352 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9404049ab0 0x7f940404bf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:44:24.075 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:24.062+0000 7f941ce0d700 1 --2- 172.21.15.42:0/1249515352 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f940404c710 0x7f940404ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:44:24.075 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:24.062+0000 7f941ce0d700 1 -- 172.21.15.42:0/1249515352 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f940404f300 con 0x7f9404047000 2019-12-05T06:44:24.075 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:24.062+0000 7f941ce0d700 1 -- 172.21.15.42:0/1249515352 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f940404f4f0 con 0x7f9404049ab0 2019-12-05T06:44:24.075 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:24.062+0000 7f941ce0d700 1 -- 172.21.15.42:0/1249515352 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f940404f730 con 0x7f940404c710 2019-12-05T06:44:24.075 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:24.062+0000 7f941de0f700 1 --2- 172.21.15.42:0/1249515352 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9404047000 0x7f94040494a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:24.075 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:24.062+0000 7f941e610700 1 --2- 172.21.15.42:0/1249515352 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f940404c710 0x7f940404ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:24.075 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:24.062+0000 7f941d60e700 1 --2- 172.21.15.42:0/1249515352 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9404049ab0 0x7f940404bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:24.076 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:24.062+0000 7f941d60e700 1 -- 172.21.15.42:0/1249515352 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f940404c710 msgr2=0x7f940404ebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:44:24.076 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:24.062+0000 7f941d60e700 1 --2- 172.21.15.42:0/1249515352 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f940404c710 0x7f940404ebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:44:24.076 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:24.062+0000 7f941d60e700 1 -- 172.21.15.42:0/1249515352 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9404047000 msgr2=0x7f94040494a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:44:24.077 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:24.062+0000 7f941d60e700 1 --2- 172.21.15.42:0/1249515352 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9404047000 0x7f94040494a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:44:24.077 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:24.062+0000 7f941d60e700 1 -- 172.21.15.42:0/1249515352 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=2+}) v3 -- 0x7f940404fa30 con 0x7f9404049ab0 2019-12-05T06:44:24.077 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:24.062+0000 7f941d60e700 1 --2- 172.21.15.42:0/1249515352 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9404049ab0 0x7f940404bf90 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f9408001000 tx=0x7f940800fcd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:24.077 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:24.062+0000 7f941d60e700 1 --2- 172.21.15.42:0/1249515352 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9404049ab0 0x7f940404bf90 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f9408001000 tx=0x7f940800fcd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:24.077 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:24.062+0000 7f941d60e700 1 --2- 172.21.15.42:0/1249515352 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9404049ab0 0x7f940404bf90 secure :-1 s=READY pgs=3140 cs=0 l=1 rx=0x7f9408001000 tx=0x7f940800fcd0).ready entity=mon.1 client_cookie=be013ec88f7fd131 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:44:24.077 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:24.062+0000 7f941d60e700 1 --2- 172.21.15.42:0/1249515352 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9404049ab0 0x7f940404bf90 secure :-1 s=THROTTLE_DONE pgs=3140 cs=0 l=1 rx=0x7f9408001000 tx=0x7f940800fcd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:24.078 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:24.062+0000 7f941d60e700 1 --2- 172.21.15.42:0/1249515352 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9404049ab0 0x7f940404bf90 secure :-1 s=THROTTLE_DONE pgs=3140 cs=0 l=1 rx=0x7f9408001000 tx=0x7f940800fcd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:24.078 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:24.062+0000 7f941ce0d700 1 -- 172.21.15.42:0/1249515352 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9408003540 con 0x7f9404049ab0 2019-12-05T06:44:24.078 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:24.062+0000 7f941ce0d700 1 -- 172.21.15.42:0/1249515352 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9408003160 con 0x7f9404049ab0 2019-12-05T06:44:24.078 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:24.062+0000 7f941f872700 1 -- 172.21.15.42:0/1249515352 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9404049ab0 msgr2=0x7f940404bf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:24.079 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:24.062+0000 7f941f872700 1 --2- 172.21.15.42:0/1249515352 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9404049ab0 0x7f940404bf90 secure :-1 s=READY pgs=3140 cs=0 l=1 rx=0x7f9408001000 tx=0x7f940800fcd0).stop 2019-12-05T06:44:24.079 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:24.062+0000 7f941e610700 1 -- 172.21.15.42:0/1249515352 reap_dead start 2019-12-05T06:44:24.079 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:24.066+0000 7f941f872700 1 -- 172.21.15.42:0/1249515352 shutdown_connections 2019-12-05T06:44:24.079 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:24.066+0000 7f941f872700 1 -- 172.21.15.42:0/1249515352 shutdown_connections 2019-12-05T06:44:24.079 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:24.066+0000 7f941f872700 1 -- 172.21.15.42:0/1249515352 wait complete. 2019-12-05T06:44:24.079 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:24.066+0000 7f941f872700 1 -- 172.21.15.42:0/1249515352 >> 172.21.15.42:0/1249515352 conn(0x7f94180aabe0 msgr2=0x7f94180a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:44:24.079 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:24.066+0000 7f941f872700 1 Processor -- start 2019-12-05T06:44:24.080 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:24.066+0000 7f941f872700 1 -- start start 2019-12-05T06:44:24.083 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:24.070+0000 7f941f872700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f941807d360 0x7f941807a360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:44:24.083 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:24.070+0000 7f941f872700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94180a4840 con 0x7f9418119cc0 2019-12-05T06:44:24.083 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:24.070+0000 7f941f872700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94180a4bd0 con 0x7f94180a50f0 2019-12-05T06:44:24.083 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:24.070+0000 7f941f872700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94180a5670 con 0x7f941807d360 2019-12-05T06:44:24.083 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:24.070+0000 7f941e610700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f94180a50f0 0x7f941813b1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:34742/0 (socket says 172.21.15.42:34742) 2019-12-05T06:44:24.083 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:24.070+0000 7f941e610700 1 -- 172.21.15.42:0/565132039 learned_addr learned my addr 172.21.15.42:0/565132039 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:44:24.084 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:24.070+0000 7f941d60e700 1 --2- 172.21.15.42:0/565132039 >> v2:172.21.15.40:3300/0 conn(0x7f941807d360 0x7f941807a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:24.084 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:24.070+0000 7f940e7fc700 1 -- 172.21.15.42:0/565132039 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2537098775 0 0) 0x7f94180a4bd0 con 0x7f94180a50f0 2019-12-05T06:44:24.084 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:24.070+0000 7f940e7fc700 1 -- 172.21.15.42:0/565132039 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94000030b0 con 0x7f94180a50f0 2019-12-05T06:44:24.084 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:24.070+0000 7f941d60e700 1 -- 172.21.15.42:0/565132039 >> v1:172.21.15.42:6789/0 conn(0x7f94180a50f0 legacy=0x7f941813b1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:24.084 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:24.070+0000 7f941d60e700 1 -- 172.21.15.42:0/565132039 >> v1:172.21.15.40:6789/0 conn(0x7f9418119cc0 legacy=0x7f941811b420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:24.084 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:24.070+0000 7f941d60e700 1 -- 172.21.15.42:0/565132039 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f941807a8b0 con 0x7f941807d360 2019-12-05T06:44:24.084 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:24.070+0000 7f941d60e700 1 --2- 172.21.15.42:0/565132039 >> v2:172.21.15.40:3300/0 conn(0x7f941807d360 0x7f941807a360 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f9408001000 tx=0x7f9408011cc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:24.085 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:24.070+0000 7f941d60e700 1 --2- 172.21.15.42:0/565132039 >> v2:172.21.15.40:3300/0 conn(0x7f941807d360 0x7f941807a360 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f9408001000 tx=0x7f9408011cc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:24.085 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:24.070+0000 7f941d60e700 1 --2- 172.21.15.42:0/565132039 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f941807d360 0x7f941807a360 secure :-1 s=READY pgs=2368 cs=0 l=1 rx=0x7f9408001000 tx=0x7f9408011cc0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:44:24.085 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:24.070+0000 7f941d60e700 1 --2- 172.21.15.42:0/565132039 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f941807d360 0x7f941807a360 secure :-1 s=THROTTLE_DONE pgs=2368 cs=0 l=1 rx=0x7f9408001000 tx=0x7f9408011cc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:24.085 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:24.070+0000 7f940e7fc700 1 -- 172.21.15.42:0/565132039 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9408006950 con 0x7f941807d360 2019-12-05T06:44:24.085 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:24.070+0000 7f941d60e700 1 --2- 172.21.15.42:0/565132039 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f941807d360 0x7f941807a360 secure :-1 s=THROTTLE_DONE pgs=2368 cs=0 l=1 rx=0x7f9408001000 tx=0x7f9408011cc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:24.085 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:24.070+0000 7f941d60e700 1 --2- 172.21.15.42:0/565132039 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f941807d360 0x7f941807a360 secure :-1 s=THROTTLE_DONE pgs=2368 cs=0 l=1 rx=0x7f9408001000 tx=0x7f9408011cc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:24.085 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:24.070+0000 7f940e7fc700 1 -- 172.21.15.42:0/565132039 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9408006570 con 0x7f941807d360 2019-12-05T06:44:24.086 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:24.070+0000 7f941f872700 1 -- 172.21.15.42:0/565132039 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f941811bc20 con 0x7f941807d360 2019-12-05T06:44:24.086 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:24.070+0000 7f940e7fc700 1 -- 172.21.15.42:0/565132039 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9408007450 con 0x7f941807d360 2019-12-05T06:44:24.086 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:24.070+0000 7f941f872700 1 -- 172.21.15.42:0/565132039 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f941807ab80 con 0x7f941807d360 2019-12-05T06:44:24.086 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:24.070+0000 7f941d60e700 1 --2- 172.21.15.42:0/565132039 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f941807d360 0x7f941807a360 secure :-1 s=THROTTLE_DONE pgs=2368 cs=0 l=1 rx=0x7f9408001000 tx=0x7f9408011cc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:24.086 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:24.074+0000 7f940e7fc700 1 -- 172.21.15.42:0/565132039 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f9408007760 con 0x7f941807d360 2019-12-05T06:44:24.086 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:24.074+0000 7f941d60e700 1 --2- 172.21.15.42:0/565132039 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f941807d360 0x7f941807a360 secure :-1 s=THROTTLE_DONE pgs=2368 cs=0 l=1 rx=0x7f9408001000 tx=0x7f9408011cc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:24.086 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:24.074+0000 7f940e7fc700 1 --2- 172.21.15.42:0/565132039 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f940003b250 0x7f940003d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:44:24.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:24.074+0000 7f940e7fc700 1 -- 172.21.15.42:0/565132039 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 11879+0+0 (secure 0 0 0) 0x7f940804ba20 con 0x7f941807d360 2019-12-05T06:44:24.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:24.074+0000 7f941de0f700 1 --2- 172.21.15.42:0/565132039 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f940003b250 0x7f940003d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:24.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:24.074+0000 7f941f872700 1 -- 172.21.15.42:0/565132039 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f941805d110 con 0x7f941807d360 2019-12-05T06:44:24.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:24.074+0000 7f941de0f700 1 --2- 172.21.15.42:0/565132039 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f940003b250 0x7f940003d760 crc :-1 s=READY pgs=1311 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:44:24.088 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:24.082+0000 7f941d60e700 1 --2- 172.21.15.42:0/565132039 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f941807d360 0x7f941807a360 secure :-1 s=THROTTLE_DONE pgs=2368 cs=0 l=1 rx=0x7f9408001000 tx=0x7f9408011cc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:24.088 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:24.082+0000 7f940e7fc700 1 -- 172.21.15.42:0/565132039 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f941811bc20 con 0x7f941807d360 2019-12-05T06:44:24.257 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:24.250+0000 7f941f872700 1 -- 172.21.15.42:0/565132039 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f941805b8c0 con 0x7f941807d360 2019-12-05T06:44:24.257 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:24.254+0000 7f941d60e700 1 --2- 172.21.15.42:0/565132039 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f941807d360 0x7f941807a360 secure :-1 s=THROTTLE_DONE pgs=2368 cs=0 l=1 rx=0x7f9408001000 tx=0x7f9408011cc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:24.258 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:24.254+0000 7f940e7fc700 1 -- 172.21.15.42:0/565132039 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 377 v377) v1 ==== 95+0+2108 (secure 0 0 0) 0x7f9408032a60 con 0x7f941807d360 2019-12-05T06:44:24.259 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 377 2019-12-05T06:44:24.259 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:44:24.259 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":377,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":12303,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/2489540211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2489540211},{"type":"v1","addr":"172.21.15.42:6835","nonce":2489540211}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":377}],"filesystems":[{"mdsmap":{"epoch":376,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:43:33.543372+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":117,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12496},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12496":{"gid":12496,"name":"b","rank":0,"incarnation":373,"state":"up:active","state_seq":11,"addr":"172.21.15.40:6833/3599619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3599619},{"type":"v1","addr":"172.21.15.40:6833","nonce":3599619}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:44:24.261 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:24.258+0000 7f941f872700 1 -- 172.21.15.42:0/565132039 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f940003b250 msgr2=0x7f940003d760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:24.262 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:24.258+0000 7f941f872700 1 --2- 172.21.15.42:0/565132039 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f940003b250 0x7f940003d760 crc :-1 s=READY pgs=1311 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:24.262 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:24.258+0000 7f941f872700 1 -- 172.21.15.42:0/565132039 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f941807d360 msgr2=0x7f941807a360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:24.262 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:24.258+0000 7f941f872700 1 --2- 172.21.15.42:0/565132039 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f941807d360 0x7f941807a360 secure :-1 s=READY pgs=2368 cs=0 l=1 rx=0x7f9408001000 tx=0x7f9408011cc0).stop 2019-12-05T06:44:24.262 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:24.258+0000 7f941f872700 1 -- 172.21.15.42:0/565132039 shutdown_connections 2019-12-05T06:44:24.262 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:24.258+0000 7f941f872700 1 --2- 172.21.15.42:0/565132039 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f940003b250 0x7f940003d760 unknown :-1 s=CLOSED pgs=1311 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:24.262 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:24.258+0000 7f941f872700 1 --2- 172.21.15.42:0/565132039 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f941807d360 0x7f941807a360 unknown :-1 s=CLOSED pgs=2368 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:24.263 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:24.258+0000 7f941f872700 1 -- 172.21.15.42:0/565132039 shutdown_connections 2019-12-05T06:44:24.263 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:24.258+0000 7f941f872700 1 -- 172.21.15.42:0/565132039 wait complete. 2019-12-05T06:44:24.263 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:24.258+0000 7f941f872700 1 -- 172.21.15.42:0/565132039 >> 172.21.15.42:0/565132039 conn(0x7f941808f6b0 msgr2=0x7f9418059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:44:24.272 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:44:26.272 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:44:26.358 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.346+0000 7f394fe89700 1 Processor -- start 2019-12-05T06:44:26.363 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.346+0000 7f394fe89700 1 -- start start 2019-12-05T06:44:26.363 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.346+0000 7f394fe89700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f39480a7740 0x7f394808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:44:26.363 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.346+0000 7f394fe89700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39480a0710 con 0x7f394807e720 2019-12-05T06:44:26.364 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.346+0000 7f394fe89700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39480a9ba0 con 0x7f394809fae0 2019-12-05T06:44:26.364 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.346+0000 7f394fe89700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39480a0b00 con 0x7f39480a7740 2019-12-05T06:44:26.364 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.346+0000 7f394ec27700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f394809fae0 0x7f394814efe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:34768/0 (socket says 172.21.15.42:34768) 2019-12-05T06:44:26.364 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.346+0000 7f394e426700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f39480a7740 0x7f394808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:26.364 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.346+0000 7f394ec27700 1 -- 172.21.15.42:0/2132194092 learned_addr learned my addr 172.21.15.42:0/2132194092 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:44:26.364 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.346+0000 7f394e426700 1 -- 172.21.15.42:0/2132194092 >> v1:172.21.15.42:6789/0 conn(0x7f394809fae0 legacy=0x7f394814efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:26.365 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.346+0000 7f394e426700 1 -- 172.21.15.42:0/2132194092 >> v1:172.21.15.40:6789/0 conn(0x7f394807e720 legacy=0x7f394809f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:26.365 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.346+0000 7f394e426700 1 -- 172.21.15.42:0/2132194092 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f394807eb50 con 0x7f39480a7740 2019-12-05T06:44:26.365 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.346+0000 7f394e426700 1 --2- 172.21.15.42:0/2132194092 >> v2:172.21.15.42:3300/0 conn(0x7f39480a7740 0x7f394808df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f3944008790 tx=0x7f394400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:26.365 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.346+0000 7f394e426700 1 --2- 172.21.15.42:0/2132194092 >> v2:172.21.15.42:3300/0 conn(0x7f39480a7740 0x7f394808df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f3944008790 tx=0x7f394400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:26.365 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.346+0000 7f394e426700 1 --2- 172.21.15.42:0/2132194092 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f39480a7740 0x7f394808df80 secure :-1 s=READY pgs=3143 cs=0 l=1 rx=0x7f3944008790 tx=0x7f394400fbf0).ready entity=mon.1 client_cookie=79ffb9011f9b119 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:44:26.365 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.346+0000 7f394e426700 1 --2- 172.21.15.42:0/2132194092 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f39480a7740 0x7f394808df80 secure :-1 s=THROTTLE_DONE pgs=3143 cs=0 l=1 rx=0x7f3944008790 tx=0x7f394400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:26.365 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.346+0000 7f394d424700 1 -- 172.21.15.42:0/2132194092 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3944011270 con 0x7f39480a7740 2019-12-05T06:44:26.366 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.346+0000 7f394e426700 1 --2- 172.21.15.42:0/2132194092 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f39480a7740 0x7f394808df80 secure :-1 s=THROTTLE_DONE pgs=3143 cs=0 l=1 rx=0x7f3944008790 tx=0x7f394400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:26.366 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.346+0000 7f394e426700 1 --2- 172.21.15.42:0/2132194092 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f39480a7740 0x7f394808df80 secure :-1 s=THROTTLE_DONE pgs=3143 cs=0 l=1 rx=0x7f3944008790 tx=0x7f394400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:26.366 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.346+0000 7f394d424700 1 -- 172.21.15.42:0/2132194092 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3944008bc0 con 0x7f39480a7740 2019-12-05T06:44:26.366 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.346+0000 7f394d424700 1 -- 172.21.15.42:0/2132194092 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3944011850 con 0x7f39480a7740 2019-12-05T06:44:26.366 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.346+0000 7f394fe89700 1 -- 172.21.15.42:0/2132194092 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f39480a7740 msgr2=0x7f394808df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:26.366 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.346+0000 7f394fe89700 1 --2- 172.21.15.42:0/2132194092 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f39480a7740 0x7f394808df80 secure :-1 s=READY pgs=3143 cs=0 l=1 rx=0x7f3944008790 tx=0x7f394400fbf0).stop 2019-12-05T06:44:26.366 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.346+0000 7f394fe89700 1 -- 172.21.15.42:0/2132194092 shutdown_connections 2019-12-05T06:44:26.367 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.346+0000 7f394fe89700 1 --2- 172.21.15.42:0/2132194092 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f39480a7740 0x7f394808df80 unknown :-1 s=CLOSED pgs=3143 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:26.367 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.346+0000 7f394fe89700 1 -- 172.21.15.42:0/2132194092 shutdown_connections 2019-12-05T06:44:26.367 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.346+0000 7f394fe89700 1 -- 172.21.15.42:0/2132194092 wait complete. 2019-12-05T06:44:26.367 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.346+0000 7f394fe89700 1 -- 172.21.15.42:0/2132194092 >> 172.21.15.42:0/2132194092 conn(0x7f39480aabe0 msgr2=0x7f39480a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:44:26.367 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.350+0000 7f394fe89700 1 Processor -- start 2019-12-05T06:44:26.367 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.350+0000 7f394fe89700 1 -- start start 2019-12-05T06:44:26.367 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.350+0000 7f394fe89700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f3948144b50 0x7f39480a3a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:44:26.368 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.350+0000 7f394fe89700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f394807a0f0 0x7f394807a520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:44:26.368 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.350+0000 7f394fe89700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39480a3470 con 0x7f39480a5f00 2019-12-05T06:44:26.368 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.350+0000 7f394fe89700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39480a2720 con 0x7f3948144b50 2019-12-05T06:44:26.368 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.350+0000 7f394fe89700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39480a37b0 con 0x7f394807a0f0 2019-12-05T06:44:26.368 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.350+0000 7f394dc25700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f39480a5f00 0x7f39480a1e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:34772/0 (socket says 172.21.15.42:34772) 2019-12-05T06:44:26.368 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.350+0000 7f394dc25700 1 -- 172.21.15.42:0/345635282 learned_addr learned my addr 172.21.15.42:0/345635282 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:44:26.369 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.350+0000 7f394e426700 1 --2- 172.21.15.42:0/345635282 >> v2:172.21.15.40:3300/0 conn(0x7f3948144b50 0x7f39480a3a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:26.369 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.350+0000 7f394ec27700 1 --2- 172.21.15.42:0/345635282 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f394807a0f0 0x7f394807a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:26.369 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.350+0000 7f393effd700 1 -- 172.21.15.42:0/345635282 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 625355342 0 0) 0x7f39480a3470 con 0x7f39480a5f00 2019-12-05T06:44:26.369 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.350+0000 7f393effd700 1 -- 172.21.15.42:0/345635282 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39300030b0 con 0x7f39480a5f00 2019-12-05T06:44:26.369 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.350+0000 7f393effd700 1 -- 172.21.15.42:0/345635282 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 424318666 0 0) 0x7f39300030b0 con 0x7f39480a5f00 2019-12-05T06:44:26.369 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.350+0000 7f393effd700 1 -- 172.21.15.42:0/345635282 --> v1:172.21.15.42:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f39480a3470 con 0x7f39480a5f00 2019-12-05T06:44:26.370 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.350+0000 7f393effd700 1 -- 172.21.15.42:0/345635282 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7f3938001a60 con 0x7f39480a5f00 2019-12-05T06:44:26.370 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.350+0000 7f393effd700 1 -- 172.21.15.42:0/345635282 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f394807a0f0 msgr2=0x7f394807a520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:26.370 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.350+0000 7f393effd700 1 --2- 172.21.15.42:0/345635282 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f394807a0f0 0x7f394807a520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:26.370 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.350+0000 7f393effd700 1 -- 172.21.15.42:0/345635282 >> v2:172.21.15.40:3300/0 conn(0x7f3948144b50 msgr2=0x7f39480a3a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:26.370 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.350+0000 7f393effd700 1 --2- 172.21.15.42:0/345635282 >> v2:172.21.15.40:3300/0 conn(0x7f3948144b50 0x7f39480a3a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:26.370 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.350+0000 7f393effd700 1 -- 172.21.15.42:0/345635282 >> v1:172.21.15.42:6789/0 conn(0x7f39480a5f00 legacy=0x7f39480a1e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:26.370 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.350+0000 7f394dc25700 1 -- 172.21.15.42:0/345635282 >> v1:172.21.15.42:6789/0 conn(0x7f39480a5f00 legacy=0x7f39480a1e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2019-12-05T06:44:26.371 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.350+0000 7f394dc25700 1 -- 172.21.15.42:0/345635282 >> v1:172.21.15.42:6789/0 conn(0x7f39480a5f00 legacy=0x7f39480a1e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2019-12-05T06:44:26.371 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.350+0000 7f394dc25700 1 --1- 172.21.15.42:0/345635282 >> v1:172.21.15.42:6789/0 conn(0x7f39480a5f00 0x7f39480a1e50 :-1 s=OPENED pgs=3144 cs=1 l=1).handle_message read tag failed 2019-12-05T06:44:26.371 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.350+0000 7f394dc25700 1 --1- 172.21.15.42:0/345635282 >> v1:172.21.15.42:6789/0 conn(0x7f39480a5f00 0x7f39480a1e50 :-1 s=OPENED pgs=3144 cs=1 l=1).fault on lossy channel, failing 2019-12-05T06:44:26.371 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.350+0000 7f393effd700 1 --2- 172.21.15.42:0/345635282 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f39300038a0 0x7f3930003cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:44:26.371 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.350+0000 7f393effd700 1 --2- 172.21.15.42:0/345635282 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3930004420 0x7f3930004850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:44:26.371 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.350+0000 7f393effd700 1 --2- 172.21.15.42:0/345635282 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3930004da0 0x7f393000b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:44:26.371 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.350+0000 7f393effd700 1 -- 172.21.15.42:0/345635282 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39480a2720 con 0x7f39300038a0 2019-12-05T06:44:26.371 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.350+0000 7f393effd700 1 -- 172.21.15.42:0/345635282 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39480a37b0 con 0x7f3930004da0 2019-12-05T06:44:26.372 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.350+0000 7f393effd700 1 -- 172.21.15.42:0/345635282 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f393000bbc0 con 0x7f3930004420 2019-12-05T06:44:26.372 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.350+0000 7f394ec27700 1 --2- 172.21.15.42:0/345635282 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3930004da0 0x7f393000b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:26.372 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.350+0000 7f394e426700 1 --2- 172.21.15.42:0/345635282 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f39300038a0 0x7f3930003cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:26.372 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.350+0000 7f394dc25700 1 --2- 172.21.15.42:0/345635282 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3930004420 0x7f3930004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:26.372 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.350+0000 7f394ec27700 1 -- 172.21.15.42:0/345635282 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3930004420 msgr2=0x7f3930004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:26.372 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.350+0000 7f394ec27700 1 --2- 172.21.15.42:0/345635282 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3930004420 0x7f3930004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:26.373 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.350+0000 7f394ec27700 1 -- 172.21.15.42:0/345635282 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f39300038a0 msgr2=0x7f3930003cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:26.373 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.350+0000 7f394ec27700 1 --2- 172.21.15.42:0/345635282 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f39300038a0 0x7f3930003cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:26.373 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.350+0000 7f394ec27700 1 -- 172.21.15.42:0/345635282 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39480a6330 con 0x7f3930004da0 2019-12-05T06:44:26.373 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.350+0000 7f394e426700 1 --2- 172.21.15.42:0/345635282 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f39300038a0 0x7f3930003cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:44:26.373 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.350+0000 7f394dc25700 1 --2- 172.21.15.42:0/345635282 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3930004420 0x7f3930004850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).send_auth_request state changed! 2019-12-05T06:44:26.373 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.350+0000 7f394ec27700 1 --2- 172.21.15.42:0/345635282 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3930004da0 0x7f393000b670 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f3940004720 tx=0x7f3940010f90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:26.373 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.350+0000 7f394fe89700 1 -- 172.21.15.42:0/345635282 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39480a4250 con 0x7f3930004da0 2019-12-05T06:44:26.374 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.350+0000 7f394fe89700 1 -- 172.21.15.42:0/345635282 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f394807a310 con 0x7f3930004da0 2019-12-05T06:44:26.374 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.350+0000 7f394ec27700 1 --2- 172.21.15.42:0/345635282 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3930004da0 0x7f393000b670 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f3940004720 tx=0x7f3940010f90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:26.374 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.350+0000 7f394ec27700 1 --2- 172.21.15.42:0/345635282 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3930004da0 0x7f393000b670 secure :-1 s=READY pgs=3145 cs=0 l=1 rx=0x7f3940004720 tx=0x7f3940010f90).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:44:26.374 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.350+0000 7f394ec27700 1 --2- 172.21.15.42:0/345635282 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3930004da0 0x7f393000b670 secure :-1 s=THROTTLE_DONE pgs=3145 cs=0 l=1 rx=0x7f3940004720 tx=0x7f3940010f90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:26.374 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.350+0000 7f393effd700 1 -- 172.21.15.42:0/345635282 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f39400047b0 con 0x7f3930004da0 2019-12-05T06:44:26.374 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.350+0000 7f394ec27700 1 --2- 172.21.15.42:0/345635282 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3930004da0 0x7f393000b670 secure :-1 s=THROTTLE_DONE pgs=3145 cs=0 l=1 rx=0x7f3940004720 tx=0x7f3940010f90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:26.374 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.350+0000 7f394ec27700 1 --2- 172.21.15.42:0/345635282 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3930004da0 0x7f393000b670 secure :-1 s=THROTTLE_DONE pgs=3145 cs=0 l=1 rx=0x7f3940004720 tx=0x7f3940010f90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:26.375 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.350+0000 7f393effd700 1 -- 172.21.15.42:0/345635282 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f394000cb30 con 0x7f3930004da0 2019-12-05T06:44:26.375 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.350+0000 7f393effd700 1 -- 172.21.15.42:0/345635282 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3940004d90 con 0x7f3930004da0 2019-12-05T06:44:26.375 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.354+0000 7f394ec27700 1 --2- 172.21.15.42:0/345635282 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3930004da0 0x7f393000b670 secure :-1 s=THROTTLE_DONE pgs=3145 cs=0 l=1 rx=0x7f3940004720 tx=0x7f3940010f90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:26.375 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.354+0000 7f393effd700 1 -- 172.21.15.42:0/345635282 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f3940006180 con 0x7f3930004da0 2019-12-05T06:44:26.375 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.354+0000 7f394ec27700 1 --2- 172.21.15.42:0/345635282 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3930004da0 0x7f393000b670 secure :-1 s=THROTTLE_DONE pgs=3145 cs=0 l=1 rx=0x7f3940004720 tx=0x7f3940010f90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:26.375 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.354+0000 7f393effd700 1 --2- 172.21.15.42:0/345635282 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f39300439c0 0x7f3930045ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:44:26.375 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.354+0000 7f393effd700 1 -- 172.21.15.42:0/345635282 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 11965+0+0 (secure 0 0 0) 0x7f394004aeb0 con 0x7f3930004da0 2019-12-05T06:44:26.376 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.354+0000 7f394e426700 1 --2- 172.21.15.42:0/345635282 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f39300439c0 0x7f3930045ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:26.376 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.354+0000 7f394e426700 1 --2- 172.21.15.42:0/345635282 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f39300439c0 0x7f3930045ed0 crc :-1 s=READY pgs=1312 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:44:26.377 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.362+0000 7f394fe89700 1 -- 172.21.15.42:0/345635282 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3924003200 con 0x7f3930004da0 2019-12-05T06:44:26.377 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.366+0000 7f394ec27700 1 --2- 172.21.15.42:0/345635282 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3930004da0 0x7f393000b670 secure :-1 s=THROTTLE_DONE pgs=3145 cs=0 l=1 rx=0x7f3940004720 tx=0x7f3940010f90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:26.377 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.366+0000 7f393effd700 1 -- 172.21.15.42:0/345635282 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f3940031f30 con 0x7f3930004da0 2019-12-05T06:44:26.541 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.538+0000 7f394fe89700 1 -- 172.21.15.42:0/345635282 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3924003630 con 0x7f3930004da0 2019-12-05T06:44:26.541 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.538+0000 7f394ec27700 1 --2- 172.21.15.42:0/345635282 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3930004da0 0x7f393000b670 secure :-1 s=THROTTLE_DONE pgs=3145 cs=0 l=1 rx=0x7f3940004720 tx=0x7f3940010f90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:26.542 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.538+0000 7f393effd700 1 -- 172.21.15.42:0/345635282 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 378 v378) v1 ==== 95+0+1778 (secure 0 0 0) 0x7f3940032190 con 0x7f3930004da0 2019-12-05T06:44:26.542 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 378 2019-12-05T06:44:26.542 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:44:26.543 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":378,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":378,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:44:25.277566+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":118,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12303},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12303":{"gid":12303,"name":"a","rank":0,"incarnation":378,"state":"up:replay","state_seq":2,"addr":"172.21.15.42:6835/2489540211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2489540211},{"type":"v1","addr":"172.21.15.42:6835","nonce":2489540211}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:44:26.544 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.542+0000 7f394fe89700 1 -- 172.21.15.42:0/345635282 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f39300439c0 msgr2=0x7f3930045ed0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:26.544 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.542+0000 7f394fe89700 1 --2- 172.21.15.42:0/345635282 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f39300439c0 0x7f3930045ed0 crc :-1 s=READY pgs=1312 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:26.545 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.542+0000 7f394ec27700 1 -- 172.21.15.42:0/345635282 reap_dead start 2019-12-05T06:44:26.545 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.542+0000 7f394fe89700 1 -- 172.21.15.42:0/345635282 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3930004da0 msgr2=0x7f393000b670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:26.545 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.542+0000 7f394fe89700 1 --2- 172.21.15.42:0/345635282 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3930004da0 0x7f393000b670 secure :-1 s=READY pgs=3145 cs=0 l=1 rx=0x7f3940004720 tx=0x7f3940010f90).stop 2019-12-05T06:44:26.545 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.542+0000 7f394fe89700 1 -- 172.21.15.42:0/345635282 shutdown_connections 2019-12-05T06:44:26.545 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.542+0000 7f394fe89700 1 --2- 172.21.15.42:0/345635282 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3930004da0 0x7f393000b670 unknown :-1 s=CLOSED pgs=3145 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:26.545 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.542+0000 7f394fe89700 1 -- 172.21.15.42:0/345635282 shutdown_connections 2019-12-05T06:44:26.546 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.542+0000 7f394fe89700 1 -- 172.21.15.42:0/345635282 wait complete. 2019-12-05T06:44:26.546 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.542+0000 7f394fe89700 1 -- 172.21.15.42:0/345635282 >> 172.21.15.42:0/345635282 conn(0x7f394807ee20 msgr2=0x7f394808eb70 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:44:26.555 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b down, removed from mdsmap 2019-12-05T06:44:26.555 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2019-12-05T06:44:26.555 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:44:26.658 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.642+0000 7f296e81a700 1 Processor -- start 2019-12-05T06:44:26.659 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.642+0000 7f296e81a700 1 -- start start 2019-12-05T06:44:26.659 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.642+0000 7f296e81a700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f296808df70 0x7f296809f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:44:26.659 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.642+0000 7f296e81a700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f296809fc50 con 0x7f29680a7730 2019-12-05T06:44:26.659 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.642+0000 7f296e81a700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29680a0700 con 0x7f296809f820 2019-12-05T06:44:26.660 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.642+0000 7f296e81a700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29680a0af0 con 0x7f296808df70 2019-12-05T06:44:26.660 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.642+0000 7f296d5b8700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f296809f820 0x7f296814efd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:34788/0 (socket says 172.21.15.42:34788) 2019-12-05T06:44:26.660 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.642+0000 7f296d5b8700 1 -- 172.21.15.42:0/2061863257 learned_addr learned my addr 172.21.15.42:0/2061863257 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:44:26.660 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.646+0000 7f2967fff700 1 --2- 172.21.15.42:0/2061863257 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f296808df70 0x7f296809f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:26.661 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.646+0000 7f29677fe700 1 -- 172.21.15.42:0/2061863257 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1484658241 0 0) 0x7f29680a0700 con 0x7f296809f820 2019-12-05T06:44:26.661 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.646+0000 7f29677fe700 1 -- 172.21.15.42:0/2061863257 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29500030b0 con 0x7f296809f820 2019-12-05T06:44:26.661 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.646+0000 7f29677fe700 1 -- 172.21.15.42:0/2061863257 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 448475529 0 0) 0x7f29500030b0 con 0x7f296809f820 2019-12-05T06:44:26.661 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.646+0000 7f29677fe700 1 -- 172.21.15.42:0/2061863257 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f296808df70 msgr2=0x7f296809f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:44:26.661 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.646+0000 7f29677fe700 1 --2- 172.21.15.42:0/2061863257 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f296808df70 0x7f296809f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:44:26.662 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.646+0000 7f29677fe700 1 -- 172.21.15.42:0/2061863257 >> v1:172.21.15.40:6789/0 conn(0x7f29680a7730 legacy=0x7f296807e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:26.662 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.646+0000 7f29677fe700 1 -- 172.21.15.42:0/2061863257 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f296807ef10 con 0x7f296809f820 2019-12-05T06:44:26.662 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.646+0000 7f29677fe700 1 -- 172.21.15.42:0/2061863257 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7f2960001ad0 con 0x7f296809f820 2019-12-05T06:44:26.662 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.646+0000 7f29677fe700 1 -- 172.21.15.42:0/2061863257 >> v1:172.21.15.42:6789/0 conn(0x7f296809f820 legacy=0x7f296814efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:26.662 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.646+0000 7f29677fe700 1 --2- 172.21.15.42:0/2061863257 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2950003710 0x7f2950003b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:44:26.663 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.646+0000 7f29677fe700 1 --2- 172.21.15.42:0/2061863257 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2950004420 0x7f2950004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:44:26.663 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.646+0000 7f29677fe700 1 --2- 172.21.15.42:0/2061863257 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2950004da0 0x7f295000b670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:44:26.663 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.646+0000 7f29677fe700 1 -- 172.21.15.42:0/2061863257 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29680a0af0 con 0x7f2950004420 2019-12-05T06:44:26.663 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.646+0000 7f296cdb7700 1 --2- 172.21.15.42:0/2061863257 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2950003710 0x7f2950003b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:26.664 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.646+0000 7f29677fe700 1 -- 172.21.15.42:0/2061863257 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2950005200 con 0x7f2950003710 2019-12-05T06:44:26.664 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.646+0000 7f29677fe700 1 -- 172.21.15.42:0/2061863257 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f295004fc00 con 0x7f2950004da0 2019-12-05T06:44:26.664 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.646+0000 7f2967fff700 1 --2- 172.21.15.42:0/2061863257 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2950004420 0x7f2950004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:26.664 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.646+0000 7f296d5b8700 1 --2- 172.21.15.42:0/2061863257 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2950004da0 0x7f295000b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:26.665 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.646+0000 7f296cdb7700 1 -- 172.21.15.42:0/2061863257 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2950004da0 msgr2=0x7f295000b670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:44:26.665 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.646+0000 7f296cdb7700 1 --2- 172.21.15.42:0/2061863257 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2950004da0 0x7f295000b670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:44:26.665 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.646+0000 7f296cdb7700 1 -- 172.21.15.42:0/2061863257 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2950004420 msgr2=0x7f2950004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:44:26.665 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.646+0000 7f296cdb7700 1 --2- 172.21.15.42:0/2061863257 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2950004420 0x7f2950004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:44:26.665 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.646+0000 7f296cdb7700 1 -- 172.21.15.42:0/2061863257 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f295004ff00 con 0x7f2950003710 2019-12-05T06:44:26.666 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.646+0000 7f296cdb7700 1 --2- 172.21.15.42:0/2061863257 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2950003710 0x7f2950003b40 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f2958000fe0 tx=0x7f295800fd80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:26.666 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.646+0000 7f296cdb7700 1 --2- 172.21.15.42:0/2061863257 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2950003710 0x7f2950003b40 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f2958000fe0 tx=0x7f295800fd80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:26.666 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.646+0000 7f296cdb7700 1 --2- 172.21.15.42:0/2061863257 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2950003710 0x7f2950003b40 secure :-1 s=READY pgs=3147 cs=0 l=1 rx=0x7f2958000fe0 tx=0x7f295800fd80).ready entity=mon.1 client_cookie=41f45e3f9cdf071b server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:44:26.666 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.646+0000 7f296cdb7700 1 --2- 172.21.15.42:0/2061863257 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2950003710 0x7f2950003b40 secure :-1 s=THROTTLE_DONE pgs=3147 cs=0 l=1 rx=0x7f2958000fe0 tx=0x7f295800fd80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:26.667 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.646+0000 7f29677fe700 1 -- 172.21.15.42:0/2061863257 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2958003540 con 0x7f2950003710 2019-12-05T06:44:26.667 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.646+0000 7f296cdb7700 1 --2- 172.21.15.42:0/2061863257 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2950003710 0x7f2950003b40 secure :-1 s=THROTTLE_DONE pgs=3147 cs=0 l=1 rx=0x7f2958000fe0 tx=0x7f295800fd80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:26.667 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.646+0000 7f296cdb7700 1 --2- 172.21.15.42:0/2061863257 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2950003710 0x7f2950003b40 secure :-1 s=THROTTLE_DONE pgs=3147 cs=0 l=1 rx=0x7f2958000fe0 tx=0x7f295800fd80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:26.667 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.646+0000 7f29677fe700 1 -- 172.21.15.42:0/2061863257 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2958003160 con 0x7f2950003710 2019-12-05T06:44:26.668 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.646+0000 7f29677fe700 1 -- 172.21.15.42:0/2061863257 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2958003b20 con 0x7f2950003710 2019-12-05T06:44:26.668 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.646+0000 7f296e81a700 1 -- 172.21.15.42:0/2061863257 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2950003710 msgr2=0x7f2950003b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:26.668 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.646+0000 7f296e81a700 1 --2- 172.21.15.42:0/2061863257 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2950003710 0x7f2950003b40 secure :-1 s=READY pgs=3147 cs=0 l=1 rx=0x7f2958000fe0 tx=0x7f295800fd80).stop 2019-12-05T06:44:26.668 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.646+0000 7f296d5b8700 1 -- 172.21.15.42:0/2061863257 reap_dead start 2019-12-05T06:44:26.668 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.646+0000 7f296e81a700 1 -- 172.21.15.42:0/2061863257 shutdown_connections 2019-12-05T06:44:26.669 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.646+0000 7f296e81a700 1 -- 172.21.15.42:0/2061863257 shutdown_connections 2019-12-05T06:44:26.669 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.646+0000 7f296e81a700 1 -- 172.21.15.42:0/2061863257 wait complete. 2019-12-05T06:44:26.669 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.646+0000 7f296e81a700 1 -- 172.21.15.42:0/2061863257 >> 172.21.15.42:0/2061863257 conn(0x7f29680aabd0 msgr2=0x7f29680a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:44:26.669 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.650+0000 7f296e81a700 1 Processor -- start 2019-12-05T06:44:26.669 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.650+0000 7f296e81a700 1 -- start start 2019-12-05T06:44:26.670 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.650+0000 7f296e81a700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f29680a2830 0x7f296808eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:44:26.670 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.650+0000 7f296e81a700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f29680a2f10 0x7f29680a3340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:44:26.670 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.650+0000 7f296e81a700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2968065670 con 0x7f2968063180 2019-12-05T06:44:26.670 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.650+0000 7f296e81a700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2968065c10 con 0x7f29680a2830 2019-12-05T06:44:26.671 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.650+0000 7f296e81a700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f296807c260 con 0x7f29680a2f10 2019-12-05T06:44:26.671 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.650+0000 7f296d5b8700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f2968063180 0x7f29680635b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:34800/0 (socket says 172.21.15.42:34800) 2019-12-05T06:44:26.671 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.650+0000 7f296d5b8700 1 -- 172.21.15.42:0/304661815 learned_addr learned my addr 172.21.15.42:0/304661815 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:44:26.671 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.650+0000 7f296cdb7700 1 --2- 172.21.15.42:0/304661815 >> v2:172.21.15.40:3300/0 conn(0x7f29680a2830 0x7f296808eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:26.672 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.650+0000 7f2967fff700 1 --2- 172.21.15.42:0/304661815 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f29680a2f10 0x7f29680a3340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:26.672 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.650+0000 7f29657fa700 1 -- 172.21.15.42:0/304661815 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 631525425 0 0) 0x7f2968065670 con 0x7f2968063180 2019-12-05T06:44:26.672 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.650+0000 7f29657fa700 1 -- 172.21.15.42:0/304661815 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29480030b0 con 0x7f2968063180 2019-12-05T06:44:26.672 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.650+0000 7f2967fff700 1 -- 172.21.15.42:0/304661815 >> v2:172.21.15.40:3300/0 conn(0x7f29680a2830 msgr2=0x7f296808eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:26.672 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.650+0000 7f2967fff700 1 --2- 172.21.15.42:0/304661815 >> v2:172.21.15.40:3300/0 conn(0x7f29680a2830 0x7f296808eb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:26.673 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.650+0000 7f2967fff700 1 -- 172.21.15.42:0/304661815 >> v1:172.21.15.42:6789/0 conn(0x7f2968063180 legacy=0x7f29680635b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:26.673 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.650+0000 7f2967fff700 1 -- 172.21.15.42:0/304661815 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29680a4830 con 0x7f29680a2f10 2019-12-05T06:44:26.673 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.650+0000 7f2967fff700 1 --2- 172.21.15.42:0/304661815 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f29680a2f10 0x7f29680a3340 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f295c002f30 tx=0x7f295c010b80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:26.673 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.650+0000 7f2967fff700 1 --2- 172.21.15.42:0/304661815 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f29680a2f10 0x7f29680a3340 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f295c002f30 tx=0x7f295c010b80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:26.673 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.650+0000 7f2967fff700 1 --2- 172.21.15.42:0/304661815 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f29680a2f10 0x7f29680a3340 secure :-1 s=READY pgs=684 cs=0 l=1 rx=0x7f295c002f30 tx=0x7f295c010b80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:44:26.674 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.650+0000 7f2967fff700 1 --2- 172.21.15.42:0/304661815 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f29680a2f10 0x7f29680a3340 secure :-1 s=THROTTLE_DONE pgs=684 cs=0 l=1 rx=0x7f295c002f30 tx=0x7f295c010b80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:26.674 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.650+0000 7f29657fa700 1 -- 172.21.15.42:0/304661815 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f295c005930 con 0x7f29680a2f10 2019-12-05T06:44:26.674 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.650+0000 7f2967fff700 1 --2- 172.21.15.42:0/304661815 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f29680a2f10 0x7f29680a3340 secure :-1 s=THROTTLE_DONE pgs=684 cs=0 l=1 rx=0x7f295c002f30 tx=0x7f295c010b80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:26.674 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.650+0000 7f296e81a700 1 -- 172.21.15.42:0/304661815 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29680a2c60 con 0x7f29680a2f10 2019-12-05T06:44:26.675 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.650+0000 7f29657fa700 1 -- 172.21.15.42:0/304661815 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f295c007160 con 0x7f29680a2f10 2019-12-05T06:44:26.675 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.650+0000 7f2967fff700 1 --2- 172.21.15.42:0/304661815 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f29680a2f10 0x7f29680a3340 secure :-1 s=THROTTLE_DONE pgs=684 cs=0 l=1 rx=0x7f295c002f30 tx=0x7f295c010b80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:26.675 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.650+0000 7f29657fa700 1 -- 172.21.15.42:0/304661815 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f295c008260 con 0x7f29680a2f10 2019-12-05T06:44:26.675 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.650+0000 7f296e81a700 1 -- 172.21.15.42:0/304661815 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f296808f0b0 con 0x7f29680a2f10 2019-12-05T06:44:26.676 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.650+0000 7f2967fff700 1 --2- 172.21.15.42:0/304661815 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f29680a2f10 0x7f29680a3340 secure :-1 s=THROTTLE_DONE pgs=684 cs=0 l=1 rx=0x7f295c002f30 tx=0x7f295c010b80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:26.676 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.654+0000 7f29657fa700 1 -- 172.21.15.42:0/304661815 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f295c008500 con 0x7f29680a2f10 2019-12-05T06:44:26.676 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.654+0000 7f2967fff700 1 --2- 172.21.15.42:0/304661815 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f29680a2f10 0x7f29680a3340 secure :-1 s=THROTTLE_DONE pgs=684 cs=0 l=1 rx=0x7f295c002f30 tx=0x7f295c010b80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:26.676 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.654+0000 7f29657fa700 1 --2- 172.21.15.42:0/304661815 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f294803b260 0x7f294803d770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:44:26.677 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.654+0000 7f29657fa700 1 -- 172.21.15.42:0/304661815 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 11965+0+0 (secure 0 0 0) 0x7f295c04b7a0 con 0x7f29680a2f10 2019-12-05T06:44:26.677 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.654+0000 7f296cdb7700 1 --2- 172.21.15.42:0/304661815 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f294803b260 0x7f294803d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:26.677 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.654+0000 7f296cdb7700 1 --2- 172.21.15.42:0/304661815 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f294803b260 0x7f294803d770 crc :-1 s=READY pgs=1313 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:44:26.678 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.658+0000 7f296e81a700 1 -- 172.21.15.42:0/304661815 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f296805d100 con 0x7f29680a2f10 2019-12-05T06:44:26.679 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.666+0000 7f2967fff700 1 --2- 172.21.15.42:0/304661815 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f29680a2f10 0x7f29680a3340 secure :-1 s=THROTTLE_DONE pgs=684 cs=0 l=1 rx=0x7f295c002f30 tx=0x7f295c010b80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:26.679 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.666+0000 7f29657fa700 1 -- 172.21.15.42:0/304661815 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f295c032800 con 0x7f29680a2f10 2019-12-05T06:44:26.831 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.826+0000 7f296e81a700 1 -- 172.21.15.42:0/304661815 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f296805b8b0 con 0x7f29680a2f10 2019-12-05T06:44:26.832 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.826+0000 7f2967fff700 1 --2- 172.21.15.42:0/304661815 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f29680a2f10 0x7f29680a3340 secure :-1 s=THROTTLE_DONE pgs=684 cs=0 l=1 rx=0x7f295c002f30 tx=0x7f295c010b80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:26.832 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.826+0000 7f29657fa700 1 -- 172.21.15.42:0/304661815 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 378 v378) v1 ==== 95+0+1778 (secure 0 0 0) 0x7f295c032a60 con 0x7f29680a2f10 2019-12-05T06:44:26.832 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 378 2019-12-05T06:44:26.832 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:44:26.833 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":378,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":378,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:44:25.277566+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":118,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12303},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12303":{"gid":12303,"name":"a","rank":0,"incarnation":378,"state":"up:replay","state_seq":2,"addr":"172.21.15.42:6835/2489540211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2489540211},{"type":"v1","addr":"172.21.15.42:6835","nonce":2489540211}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:44:26.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.830+0000 7f296e81a700 1 -- 172.21.15.42:0/304661815 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f294803b260 msgr2=0x7f294803d770 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:26.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.830+0000 7f296e81a700 1 --2- 172.21.15.42:0/304661815 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f294803b260 0x7f294803d770 crc :-1 s=READY pgs=1313 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:26.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.830+0000 7f296e81a700 1 -- 172.21.15.42:0/304661815 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f29680a2f10 msgr2=0x7f29680a3340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:26.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.830+0000 7f296e81a700 1 --2- 172.21.15.42:0/304661815 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f29680a2f10 0x7f29680a3340 secure :-1 s=READY pgs=684 cs=0 l=1 rx=0x7f295c002f30 tx=0x7f295c010b80).stop 2019-12-05T06:44:26.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.830+0000 7f296e81a700 1 -- 172.21.15.42:0/304661815 shutdown_connections 2019-12-05T06:44:26.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.830+0000 7f296e81a700 1 --2- 172.21.15.42:0/304661815 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f294803b260 0x7f294803d770 unknown :-1 s=CLOSED pgs=1313 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:26.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.830+0000 7f296e81a700 1 --2- 172.21.15.42:0/304661815 >> v2:172.21.15.40:3300/0 conn(0x7f29680a2830 0x7f296808eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:26.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.830+0000 7f296e81a700 1 --2- 172.21.15.42:0/304661815 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f29680a2f10 0x7f29680a3340 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:26.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.830+0000 7f296e81a700 1 -- 172.21.15.42:0/304661815 shutdown_connections 2019-12-05T06:44:26.837 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.830+0000 7f296e81a700 1 -- 172.21.15.42:0/304661815 wait complete. 2019-12-05T06:44:26.837 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:26.830+0000 7f296e81a700 1 -- 172.21.15.42:0/304661815 >> 172.21.15.42:0/304661815 conn(0x7f29680a7730 msgr2=0x7f296807c760 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:44:26.845 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 378, "feature_flags": { "enable_multiple": false, "ever_enabled_multiple": false }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2019-12-05T05:03:35.604738+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 378, "ever_allowed_features": 2, "explicitly_allowed_features": 2, "failed": [], "flags": 18, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_12303": { "addr": "172.21.15.42:6835/2489540211", "addrs": { "addrvec": [ { "addr": "172.21.15.42:6834", "nonce": 2489540211, "type": "v2" }, { "addr": "172.21.15.42:6835", "nonce": 2489540211, "type": "v1" } ] }, "export_targets": [], "features": 4540138292836696063, "flags": 0, "gid": 12303, "incarnation": 378, "name": "a", "rank": 0, "state": "up:replay", "state_seq": 2 } }, "last_failure": 0, "last_failure_osd_epoch": 118, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 2, "min_compat_client": "0 (unknown)", "modified": "2019-12-05T06:44:25.277566+0000", "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 12303 } } } ], "standbys": [] } 2019-12-05T06:44:28.847 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:44:28.936 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:28.930+0000 7f7ff6e83700 1 Processor -- start 2019-12-05T06:44:28.937 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:28.930+0000 7f7ff6e83700 1 -- start start 2019-12-05T06:44:28.937 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:28.930+0000 7f7ff6e83700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7ff00a7730 0x7f7ff008df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:44:28.937 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:28.930+0000 7f7ff6e83700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7ff009fad0 0x7f7ff007eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:44:28.937 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:28.930+0000 7f7ff6e83700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ff00a0700 con 0x7f7ff007e710 2019-12-05T06:44:28.937 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:28.930+0000 7f7ff6e83700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ff00a0af0 con 0x7f7ff009fad0 2019-12-05T06:44:28.938 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:28.930+0000 7f7ff6e83700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ff008e5a0 con 0x7f7ff00a7730 2019-12-05T06:44:28.938 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:28.930+0000 7f7ff5420700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7ff00a7730 0x7f7ff008df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:28.938 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:28.930+0000 7f7ff5420700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7ff00a7730 0x7f7ff008df70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:35234/0 (socket says 172.21.15.42:35234) 2019-12-05T06:44:28.938 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:28.930+0000 7f7ff5c21700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7ff009fad0 0x7f7ff007eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:28.938 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:28.930+0000 7f7ff5420700 1 -- 172.21.15.42:0/2789913701 learned_addr learned my addr 172.21.15.42:0/2789913701 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:44:28.938 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:28.930+0000 7f7ff5420700 1 -- 172.21.15.42:0/2789913701 >> v2:172.21.15.40:3300/0 conn(0x7f7ff009fad0 msgr2=0x7f7ff007eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:44:28.939 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:28.930+0000 7f7ff5420700 1 --2- 172.21.15.42:0/2789913701 >> v2:172.21.15.40:3300/0 conn(0x7f7ff009fad0 0x7f7ff007eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:44:28.939 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:28.930+0000 7f7ff5420700 1 -- 172.21.15.42:0/2789913701 >> v1:172.21.15.40:6789/0 conn(0x7f7ff007e710 legacy=0x7f7ff009f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:28.939 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:28.930+0000 7f7ff5420700 1 -- 172.21.15.42:0/2789913701 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ff007f090 con 0x7f7ff00a7730 2019-12-05T06:44:28.939 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:28.930+0000 7f7ff5420700 1 --2- 172.21.15.42:0/2789913701 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7ff00a7730 0x7f7ff008df70 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f7fec008790 tx=0x7f7fec00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:28.940 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:28.930+0000 7f7ff5420700 1 --2- 172.21.15.42:0/2789913701 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7ff00a7730 0x7f7ff008df70 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f7fec008790 tx=0x7f7fec00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:28.940 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:28.930+0000 7f7ff5420700 1 --2- 172.21.15.42:0/2789913701 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7ff00a7730 0x7f7ff008df70 secure :-1 s=READY pgs=685 cs=0 l=1 rx=0x7f7fec008790 tx=0x7f7fec00fd30).ready entity=mon.2 client_cookie=17d6a375b9b0fc37 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:44:28.940 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:28.930+0000 7f7ff5420700 1 --2- 172.21.15.42:0/2789913701 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7ff00a7730 0x7f7ff008df70 secure :-1 s=THROTTLE_DONE pgs=685 cs=0 l=1 rx=0x7f7fec008790 tx=0x7f7fec00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:28.940 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:28.930+0000 7f7fe7fff700 1 -- 172.21.15.42:0/2789913701 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7fec008c90 con 0x7f7ff00a7730 2019-12-05T06:44:28.940 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:28.930+0000 7f7ff5420700 1 --2- 172.21.15.42:0/2789913701 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7ff00a7730 0x7f7ff008df70 secure :-1 s=THROTTLE_DONE pgs=685 cs=0 l=1 rx=0x7f7fec008790 tx=0x7f7fec00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:28.940 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:28.930+0000 7f7fe7fff700 1 -- 172.21.15.42:0/2789913701 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fec0088b0 con 0x7f7ff00a7730 2019-12-05T06:44:28.941 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:28.930+0000 7f7ff5420700 1 --2- 172.21.15.42:0/2789913701 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7ff00a7730 0x7f7ff008df70 secure :-1 s=THROTTLE_DONE pgs=685 cs=0 l=1 rx=0x7f7fec008790 tx=0x7f7fec00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:28.941 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:28.930+0000 7f7fe7fff700 1 -- 172.21.15.42:0/2789913701 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7fec011b10 con 0x7f7ff00a7730 2019-12-05T06:44:28.941 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:28.930+0000 7f7ff6e83700 1 -- 172.21.15.42:0/2789913701 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7ff00a7730 msgr2=0x7f7ff008df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:28.941 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:28.930+0000 7f7ff6e83700 1 --2- 172.21.15.42:0/2789913701 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7ff00a7730 0x7f7ff008df70 secure :-1 s=READY pgs=685 cs=0 l=1 rx=0x7f7fec008790 tx=0x7f7fec00fd30).stop 2019-12-05T06:44:28.941 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:28.930+0000 7f7ff6e83700 1 -- 172.21.15.42:0/2789913701 shutdown_connections 2019-12-05T06:44:28.941 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:28.930+0000 7f7ff6e83700 1 --2- 172.21.15.42:0/2789913701 >> v2:172.21.15.40:3300/0 conn(0x7f7ff009fad0 0x7f7ff007eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:44:28.942 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:28.930+0000 7f7ff6e83700 1 --2- 172.21.15.42:0/2789913701 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7ff00a7730 0x7f7ff008df70 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:28.942 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:28.930+0000 7f7ff6e83700 1 -- 172.21.15.42:0/2789913701 shutdown_connections 2019-12-05T06:44:28.942 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:28.930+0000 7f7ff6e83700 1 -- 172.21.15.42:0/2789913701 wait complete. 2019-12-05T06:44:28.942 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:28.930+0000 7f7ff6e83700 1 -- 172.21.15.42:0/2789913701 >> 172.21.15.42:0/2789913701 conn(0x7f7ff00aabd0 msgr2=0x7f7ff00a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:44:28.945 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:28.934+0000 7f7ff6e83700 1 Processor -- start 2019-12-05T06:44:28.946 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:28.934+0000 7f7ff6e83700 1 -- start start 2019-12-05T06:44:28.946 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:28.934+0000 7f7ff6e83700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7ff0119dd0 0x7f7ff00a4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:44:28.946 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:28.934+0000 7f7ff6e83700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7ff0131bc0 0x7f7ff0131ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:44:28.946 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:28.934+0000 7f7ff6e83700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ff007b740 con 0x7f7ff0118690 2019-12-05T06:44:28.947 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:28.934+0000 7f7ff6e83700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ff00a5850 con 0x7f7ff0131bc0 2019-12-05T06:44:28.947 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:28.934+0000 7f7ff6e83700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ff0131920 con 0x7f7ff0119dd0 2019-12-05T06:44:28.947 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:28.934+0000 7f7ff5420700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7ff0119dd0 0x7f7ff00a4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:28.948 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:28.934+0000 7f7ff5420700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7ff0119dd0 0x7f7ff00a4f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:35242/0 (socket says 172.21.15.42:35242) 2019-12-05T06:44:28.948 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:28.934+0000 7f7ff5420700 1 -- 172.21.15.42:0/777320546 learned_addr learned my addr 172.21.15.42:0/777320546 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:44:28.948 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:28.934+0000 7f7ff5c21700 1 --2- 172.21.15.42:0/777320546 >> v2:172.21.15.40:3300/0 conn(0x7f7ff0131bc0 0x7f7ff0131ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:28.948 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:28.934+0000 7f7ff5420700 1 -- 172.21.15.42:0/777320546 >> v2:172.21.15.40:3300/0 conn(0x7f7ff0131bc0 msgr2=0x7f7ff0131ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:28.949 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:28.934+0000 7f7ff5420700 1 --2- 172.21.15.42:0/777320546 >> v2:172.21.15.40:3300/0 conn(0x7f7ff0131bc0 0x7f7ff0131ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:28.949 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:28.934+0000 7f7ff5420700 1 -- 172.21.15.42:0/777320546 >> v1:172.21.15.40:6789/0 conn(0x7f7ff0118690 legacy=0x7f7ff007af40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:28.949 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:28.934+0000 7f7ff5420700 1 -- 172.21.15.42:0/777320546 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ff0132540 con 0x7f7ff0119dd0 2019-12-05T06:44:28.950 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:28.934+0000 7f7ff5420700 1 --2- 172.21.15.42:0/777320546 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7ff0119dd0 0x7f7ff00a4f30 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f7fec008790 tx=0x7f7fec008f40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:28.950 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:28.934+0000 7f7ff5420700 1 --2- 172.21.15.42:0/777320546 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7ff0119dd0 0x7f7ff00a4f30 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f7fec008790 tx=0x7f7fec008f40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:28.950 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:28.934+0000 7f7ff5420700 1 --2- 172.21.15.42:0/777320546 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7ff0119dd0 0x7f7ff00a4f30 secure :-1 s=READY pgs=686 cs=0 l=1 rx=0x7f7fec008790 tx=0x7f7fec008f40).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:44:28.950 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:28.934+0000 7f7ff5420700 1 --2- 172.21.15.42:0/777320546 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7ff0119dd0 0x7f7ff00a4f30 secure :-1 s=THROTTLE_DONE pgs=686 cs=0 l=1 rx=0x7f7fec008790 tx=0x7f7fec008f40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:28.951 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:28.934+0000 7f7fe5ffb700 1 -- 172.21.15.42:0/777320546 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7fec00f470 con 0x7f7ff0119dd0 2019-12-05T06:44:28.951 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:28.934+0000 7f7ff5420700 1 --2- 172.21.15.42:0/777320546 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7ff0119dd0 0x7f7ff00a4f30 secure :-1 s=THROTTLE_DONE pgs=686 cs=0 l=1 rx=0x7f7fec008790 tx=0x7f7fec008f40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:28.951 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:28.934+0000 7f7ff6e83700 1 -- 172.21.15.42:0/777320546 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ff00a5480 con 0x7f7ff0119dd0 2019-12-05T06:44:28.951 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:28.934+0000 7f7fe5ffb700 1 -- 172.21.15.42:0/777320546 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fec011730 con 0x7f7ff0119dd0 2019-12-05T06:44:28.952 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:28.934+0000 7f7ff5420700 1 --2- 172.21.15.42:0/777320546 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7ff0119dd0 0x7f7ff00a4f30 secure :-1 s=THROTTLE_DONE pgs=686 cs=0 l=1 rx=0x7f7fec008790 tx=0x7f7fec008f40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:28.952 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:28.934+0000 7f7fe5ffb700 1 -- 172.21.15.42:0/777320546 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7fec005730 con 0x7f7ff0119dd0 2019-12-05T06:44:28.952 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:28.938+0000 7f7ff6e83700 1 -- 172.21.15.42:0/777320546 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ff0138e10 con 0x7f7ff0119dd0 2019-12-05T06:44:28.952 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:28.938+0000 7f7ff5420700 1 --2- 172.21.15.42:0/777320546 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7ff0119dd0 0x7f7ff00a4f30 secure :-1 s=THROTTLE_DONE pgs=686 cs=0 l=1 rx=0x7f7fec008790 tx=0x7f7fec008f40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:28.953 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:28.938+0000 7f7fe5ffb700 1 -- 172.21.15.42:0/777320546 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f7fec0059d0 con 0x7f7ff0119dd0 2019-12-05T06:44:28.953 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:28.938+0000 7f7ff5420700 1 --2- 172.21.15.42:0/777320546 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7ff0119dd0 0x7f7ff00a4f30 secure :-1 s=THROTTLE_DONE pgs=686 cs=0 l=1 rx=0x7f7fec008790 tx=0x7f7fec008f40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:28.953 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:28.938+0000 7f7fe5ffb700 1 --2- 172.21.15.42:0/777320546 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f7fd003adb0 0x7f7fd003d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:44:28.954 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:28.938+0000 7f7fe5ffb700 1 -- 172.21.15.42:0/777320546 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 11965+0+0 (secure 0 0 0) 0x7f7fec04b7c0 con 0x7f7ff0119dd0 2019-12-05T06:44:28.954 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:28.938+0000 7f7ff4c1f700 1 --2- 172.21.15.42:0/777320546 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f7fd003adb0 0x7f7fd003d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:28.954 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:28.938+0000 7f7ff4c1f700 1 --2- 172.21.15.42:0/777320546 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f7fd003adb0 0x7f7fd003d2c0 crc :-1 s=READY pgs=1315 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:44:28.955 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:28.946+0000 7f7ff6e83700 1 -- 172.21.15.42:0/777320546 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ff005d100 con 0x7f7ff0119dd0 2019-12-05T06:44:28.958 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:28.954+0000 7f7ff5420700 1 --2- 172.21.15.42:0/777320546 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7ff0119dd0 0x7f7ff00a4f30 secure :-1 s=THROTTLE_DONE pgs=686 cs=0 l=1 rx=0x7f7fec008790 tx=0x7f7fec008f40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:28.958 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:28.954+0000 7f7fe5ffb700 1 -- 172.21.15.42:0/777320546 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f7fec033630 con 0x7f7ff0119dd0 2019-12-05T06:44:29.125 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:29.122+0000 7f7ff6e83700 1 -- 172.21.15.42:0/777320546 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7ff005b8b0 con 0x7f7ff0119dd0 2019-12-05T06:44:29.127 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:29.122+0000 7f7ff5420700 1 --2- 172.21.15.42:0/777320546 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7ff0119dd0 0x7f7ff00a4f30 secure :-1 s=THROTTLE_DONE pgs=686 cs=0 l=1 rx=0x7f7fec008790 tx=0x7f7fec008f40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:29.127 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:29.122+0000 7f7fe5ffb700 1 -- 172.21.15.42:0/777320546 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 378 v378) v1 ==== 95+0+1778 (secure 0 0 0) 0x7f7fec032d40 con 0x7f7ff0119dd0 2019-12-05T06:44:29.127 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 378 2019-12-05T06:44:29.128 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:44:29.128 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":378,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":378,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:44:25.277566+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":118,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12303},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12303":{"gid":12303,"name":"a","rank":0,"incarnation":378,"state":"up:replay","state_seq":2,"addr":"172.21.15.42:6835/2489540211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2489540211},{"type":"v1","addr":"172.21.15.42:6835","nonce":2489540211}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:44:29.130 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:29.126+0000 7f7ff6e83700 1 -- 172.21.15.42:0/777320546 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f7fd003adb0 msgr2=0x7f7fd003d2c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:29.130 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:29.126+0000 7f7ff6e83700 1 --2- 172.21.15.42:0/777320546 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f7fd003adb0 0x7f7fd003d2c0 crc :-1 s=READY pgs=1315 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:29.130 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:29.126+0000 7f7ff6e83700 1 -- 172.21.15.42:0/777320546 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7ff0119dd0 msgr2=0x7f7ff00a4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:29.131 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:29.126+0000 7f7ff6e83700 1 --2- 172.21.15.42:0/777320546 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7ff0119dd0 0x7f7ff00a4f30 secure :-1 s=READY pgs=686 cs=0 l=1 rx=0x7f7fec008790 tx=0x7f7fec008f40).stop 2019-12-05T06:44:29.131 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:29.126+0000 7f7ff6e83700 1 -- 172.21.15.42:0/777320546 shutdown_connections 2019-12-05T06:44:29.131 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:29.126+0000 7f7ff6e83700 1 --2- 172.21.15.42:0/777320546 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f7fd003adb0 0x7f7fd003d2c0 unknown :-1 s=CLOSED pgs=1315 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:29.131 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:29.126+0000 7f7ff6e83700 1 --2- 172.21.15.42:0/777320546 >> v2:172.21.15.40:3300/0 conn(0x7f7ff0131bc0 0x7f7ff0131ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:29.131 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:29.126+0000 7f7ff6e83700 1 --2- 172.21.15.42:0/777320546 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7ff0119dd0 0x7f7ff00a4f30 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:29.132 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:29.126+0000 7f7ff6e83700 1 -- 172.21.15.42:0/777320546 shutdown_connections 2019-12-05T06:44:29.132 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:29.126+0000 7f7ff6e83700 1 -- 172.21.15.42:0/777320546 wait complete. 2019-12-05T06:44:29.132 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:29.126+0000 7f7ff6e83700 1 -- 172.21.15.42:0/777320546 >> 172.21.15.42:0/777320546 conn(0x7f7ff00aabd0 msgr2=0x7f7ff00a9780 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:44:29.138 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2019-12-05T06:44:31.138 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:44:31.217 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:31.209+0000 7f42ac190700 1 Processor -- start 2019-12-05T06:44:31.217 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:31.209+0000 7f42ac190700 1 -- start start 2019-12-05T06:44:31.217 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:31.209+0000 7f42ac190700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f42a40a7740 0x7f42a408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:44:31.219 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:31.209+0000 7f42ac190700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f42a409fae0 0x7f42a407eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:44:31.219 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:31.209+0000 7f42ac190700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42a40a0710 con 0x7f42a407e720 2019-12-05T06:44:31.219 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:31.209+0000 7f42ac190700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42a40a0b00 con 0x7f42a40a7740 2019-12-05T06:44:31.220 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:31.209+0000 7f42ac190700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42a408e5b0 con 0x7f42a409fae0 2019-12-05T06:44:31.220 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:31.209+0000 7f42aaf2e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f42a409fae0 0x7f42a407eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:31.220 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:31.209+0000 7f42aaf2e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f42a409fae0 0x7f42a407eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:34788/0 (socket says 172.21.15.42:34788) 2019-12-05T06:44:31.220 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:31.209+0000 7f42aaf2e700 1 -- 172.21.15.42:0/3651238845 learned_addr learned my addr 172.21.15.42:0/3651238845 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:44:31.220 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:31.209+0000 7f42aa72d700 1 --2- 172.21.15.42:0/3651238845 >> v2:172.21.15.40:3300/0 conn(0x7f42a40a7740 0x7f42a408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:31.220 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:31.209+0000 7f42aaf2e700 1 -- 172.21.15.42:0/3651238845 >> v2:172.21.15.40:3300/0 conn(0x7f42a40a7740 msgr2=0x7f42a408df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:44:31.221 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:31.209+0000 7f42aaf2e700 1 --2- 172.21.15.42:0/3651238845 >> v2:172.21.15.40:3300/0 conn(0x7f42a40a7740 0x7f42a408df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:44:31.221 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:31.209+0000 7f42aaf2e700 1 -- 172.21.15.42:0/3651238845 >> v1:172.21.15.42:6789/0 conn(0x7f42a407e720 legacy=0x7f42a409f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:31.221 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:31.209+0000 7f42aaf2e700 1 -- 172.21.15.42:0/3651238845 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42a407f0a0 con 0x7f42a409fae0 2019-12-05T06:44:31.221 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:31.209+0000 7f42aa72d700 1 --2- 172.21.15.42:0/3651238845 >> v2:172.21.15.40:3300/0 conn(0x7f42a40a7740 0x7f42a408df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:44:31.221 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:31.209+0000 7f42aaf2e700 1 --2- 172.21.15.42:0/3651238845 >> v2:172.21.15.42:3300/0 conn(0x7f42a409fae0 0x7f42a407eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f429c0032b0 tx=0x7f429c0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:31.222 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:31.209+0000 7f42aaf2e700 1 --2- 172.21.15.42:0/3651238845 >> v2:172.21.15.42:3300/0 conn(0x7f42a409fae0 0x7f42a407eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f429c0032b0 tx=0x7f429c0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:31.222 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:31.209+0000 7f42aaf2e700 1 --2- 172.21.15.42:0/3651238845 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f42a409fae0 0x7f42a407eb50 secure :-1 s=READY pgs=3150 cs=0 l=1 rx=0x7f429c0032b0 tx=0x7f429c0100b0).ready entity=mon.1 client_cookie=ee86195911bfa587 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:44:31.222 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:31.209+0000 7f42aaf2e700 1 --2- 172.21.15.42:0/3651238845 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f42a409fae0 0x7f42a407eb50 secure :-1 s=THROTTLE_DONE pgs=3150 cs=0 l=1 rx=0x7f429c0032b0 tx=0x7f429c0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:31.222 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:31.209+0000 7f42a972b700 1 -- 172.21.15.42:0/3651238845 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f429c010d60 con 0x7f42a409fae0 2019-12-05T06:44:31.222 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:31.209+0000 7f42aaf2e700 1 --2- 172.21.15.42:0/3651238845 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f42a409fae0 0x7f42a407eb50 secure :-1 s=THROTTLE_DONE pgs=3150 cs=0 l=1 rx=0x7f429c0032b0 tx=0x7f429c0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:31.223 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:31.209+0000 7f42aaf2e700 1 --2- 172.21.15.42:0/3651238845 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f42a409fae0 0x7f42a407eb50 secure :-1 s=THROTTLE_DONE pgs=3150 cs=0 l=1 rx=0x7f429c0032b0 tx=0x7f429c0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:31.223 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:31.209+0000 7f42a972b700 1 -- 172.21.15.42:0/3651238845 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f429c010980 con 0x7f42a409fae0 2019-12-05T06:44:31.223 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:31.209+0000 7f42a972b700 1 -- 172.21.15.42:0/3651238845 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f429c0114a0 con 0x7f42a409fae0 2019-12-05T06:44:31.223 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:31.209+0000 7f42ac190700 1 -- 172.21.15.42:0/3651238845 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f42a409fae0 msgr2=0x7f42a407eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:31.223 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:31.209+0000 7f42ac190700 1 --2- 172.21.15.42:0/3651238845 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f42a409fae0 0x7f42a407eb50 secure :-1 s=READY pgs=3150 cs=0 l=1 rx=0x7f429c0032b0 tx=0x7f429c0100b0).stop 2019-12-05T06:44:31.223 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:31.213+0000 7f42ac190700 1 -- 172.21.15.42:0/3651238845 shutdown_connections 2019-12-05T06:44:31.224 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:31.213+0000 7f42ac190700 1 --2- 172.21.15.42:0/3651238845 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f42a409fae0 0x7f42a407eb50 unknown :-1 s=CLOSED pgs=3150 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:31.224 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:31.213+0000 7f42ac190700 1 --2- 172.21.15.42:0/3651238845 >> v2:172.21.15.40:3300/0 conn(0x7f42a40a7740 0x7f42a408df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:44:31.224 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:31.213+0000 7f42ac190700 1 -- 172.21.15.42:0/3651238845 shutdown_connections 2019-12-05T06:44:31.224 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:31.213+0000 7f42ac190700 1 -- 172.21.15.42:0/3651238845 wait complete. 2019-12-05T06:44:31.224 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:31.213+0000 7f42ac190700 1 -- 172.21.15.42:0/3651238845 >> 172.21.15.42:0/3651238845 conn(0x7f42a40aabe0 msgr2=0x7f42a40a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:44:31.229 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:31.213+0000 7f42ac190700 1 Processor -- start 2019-12-05T06:44:31.230 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:31.213+0000 7f42ac190700 1 -- start start 2019-12-05T06:44:31.230 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:31.213+0000 7f42ac190700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f42a408ee90 0x7f42a4063dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:44:31.230 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:31.213+0000 7f42ac190700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f42a4065030 0x7f42a412a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:44:31.230 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:31.213+0000 7f42ac190700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42a4065460 con 0x7f42a412af10 2019-12-05T06:44:31.230 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:31.213+0000 7f42ac190700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42a4065730 con 0x7f42a4065030 2019-12-05T06:44:31.230 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:31.213+0000 7f42ac190700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42a40659d0 con 0x7f42a408ee90 2019-12-05T06:44:31.230 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:31.213+0000 7f42aa72d700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f42a408ee90 0x7f42a4063dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:31.231 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:31.213+0000 7f42a9f2c700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f42a4065030 0x7f42a412a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:31.231 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:31.213+0000 7f42aaf2e700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f42a412af10 0x7f42a41132d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:44270/0 (socket says 172.21.15.42:44270) 2019-12-05T06:44:31.231 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:31.213+0000 7f42aaf2e700 1 -- 172.21.15.42:0/3479846969 learned_addr learned my addr 172.21.15.42:0/3479846969 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:44:31.231 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:31.217+0000 7f42aa72d700 1 -- 172.21.15.42:0/3479846969 >> v2:172.21.15.40:3300/0 conn(0x7f42a4065030 msgr2=0x7f42a412a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:31.231 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:31.217+0000 7f42aa72d700 1 --2- 172.21.15.42:0/3479846969 >> v2:172.21.15.40:3300/0 conn(0x7f42a4065030 0x7f42a412a9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:31.232 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:31.217+0000 7f42aa72d700 1 -- 172.21.15.42:0/3479846969 >> v1:172.21.15.40:6789/0 conn(0x7f42a412af10 legacy=0x7f42a41132d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:31.232 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:31.217+0000 7f42aa72d700 1 -- 172.21.15.42:0/3479846969 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42a4113ad0 con 0x7f42a408ee90 2019-12-05T06:44:31.232 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:31.217+0000 7f42aa72d700 1 --2- 172.21.15.42:0/3479846969 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f42a408ee90 0x7f42a4063dc0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f4294002f30 tx=0x7f42940108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:31.232 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:31.217+0000 7f42aa72d700 1 --2- 172.21.15.42:0/3479846969 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f42a408ee90 0x7f42a4063dc0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f4294002f30 tx=0x7f42940108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:31.232 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:31.217+0000 7f42aa72d700 1 --2- 172.21.15.42:0/3479846969 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f42a408ee90 0x7f42a4063dc0 secure :-1 s=READY pgs=687 cs=0 l=1 rx=0x7f4294002f30 tx=0x7f42940108f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:44:31.233 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:31.217+0000 7f42aa72d700 1 --2- 172.21.15.42:0/3479846969 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f42a408ee90 0x7f42a4063dc0 secure :-1 s=THROTTLE_DONE pgs=687 cs=0 l=1 rx=0x7f4294002f30 tx=0x7f42940108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:31.233 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:31.217+0000 7f42a2ffd700 1 -- 172.21.15.42:0/3479846969 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4294005930 con 0x7f42a408ee90 2019-12-05T06:44:31.233 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:31.217+0000 7f42aa72d700 1 --2- 172.21.15.42:0/3479846969 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f42a408ee90 0x7f42a4063dc0 secure :-1 s=THROTTLE_DONE pgs=687 cs=0 l=1 rx=0x7f4294002f30 tx=0x7f42940108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:31.233 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:31.217+0000 7f42aa72d700 1 --2- 172.21.15.42:0/3479846969 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f42a408ee90 0x7f42a4063dc0 secure :-1 s=THROTTLE_DONE pgs=687 cs=0 l=1 rx=0x7f4294002f30 tx=0x7f42940108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:31.233 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:31.217+0000 7f42a2ffd700 1 -- 172.21.15.42:0/3479846969 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4294007160 con 0x7f42a408ee90 2019-12-05T06:44:31.233 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:31.217+0000 7f42ac190700 1 -- 172.21.15.42:0/3479846969 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42a41316d0 con 0x7f42a408ee90 2019-12-05T06:44:31.233 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:31.217+0000 7f42a2ffd700 1 -- 172.21.15.42:0/3479846969 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f42940060c0 con 0x7f42a408ee90 2019-12-05T06:44:31.234 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:31.217+0000 7f42ac190700 1 -- 172.21.15.42:0/3479846969 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42a4131a60 con 0x7f42a408ee90 2019-12-05T06:44:31.234 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:31.217+0000 7f42aa72d700 1 --2- 172.21.15.42:0/3479846969 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f42a408ee90 0x7f42a4063dc0 secure :-1 s=THROTTLE_DONE pgs=687 cs=0 l=1 rx=0x7f4294002f30 tx=0x7f42940108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:31.234 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:31.217+0000 7f42a2ffd700 1 -- 172.21.15.42:0/3479846969 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f4294007b30 con 0x7f42a408ee90 2019-12-05T06:44:31.234 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:31.217+0000 7f42aa72d700 1 --2- 172.21.15.42:0/3479846969 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f42a408ee90 0x7f42a4063dc0 secure :-1 s=THROTTLE_DONE pgs=687 cs=0 l=1 rx=0x7f4294002f30 tx=0x7f42940108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:31.234 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:31.217+0000 7f42a2ffd700 1 --2- 172.21.15.42:0/3479846969 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f428403adb0 0x7f428403d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:44:31.234 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:31.217+0000 7f42a2ffd700 1 -- 172.21.15.42:0/3479846969 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 11965+0+0 (secure 0 0 0) 0x7f429404abb0 con 0x7f42a408ee90 2019-12-05T06:44:31.234 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:31.217+0000 7f42a9f2c700 1 --2- 172.21.15.42:0/3479846969 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f428403adb0 0x7f428403d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:31.235 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:31.217+0000 7f42a9f2c700 1 --2- 172.21.15.42:0/3479846969 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f428403adb0 0x7f428403d2c0 crc :-1 s=READY pgs=1316 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:44:31.235 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:31.225+0000 7f42ac190700 1 -- 172.21.15.42:0/3479846969 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42a405d110 con 0x7f42a408ee90 2019-12-05T06:44:31.237 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:31.233+0000 7f42aa72d700 1 --2- 172.21.15.42:0/3479846969 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f42a408ee90 0x7f42a4063dc0 secure :-1 s=THROTTLE_DONE pgs=687 cs=0 l=1 rx=0x7f4294002f30 tx=0x7f42940108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:31.237 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:31.233+0000 7f42a2ffd700 1 -- 172.21.15.42:0/3479846969 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f4294031c30 con 0x7f42a408ee90 2019-12-05T06:44:31.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:31.397+0000 7f42ac190700 1 -- 172.21.15.42:0/3479846969 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f42a405b8c0 con 0x7f42a408ee90 2019-12-05T06:44:31.404 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:31.401+0000 7f42aa72d700 1 --2- 172.21.15.42:0/3479846969 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f42a408ee90 0x7f42a4063dc0 secure :-1 s=THROTTLE_DONE pgs=687 cs=0 l=1 rx=0x7f4294002f30 tx=0x7f42940108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:31.404 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:31.401+0000 7f42a2ffd700 1 -- 172.21.15.42:0/3479846969 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 379 v379) v1 ==== 95+0+1782 (secure 0 0 0) 0x7f4294031e90 con 0x7f42a408ee90 2019-12-05T06:44:31.406 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 379 2019-12-05T06:44:31.406 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:44:31.407 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":379,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":379,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:44:31.303175+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":118,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12303},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12303":{"gid":12303,"name":"a","rank":0,"incarnation":378,"state":"up:reconnect","state_seq":14,"addr":"172.21.15.42:6835/2489540211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2489540211},{"type":"v1","addr":"172.21.15.42:6835","nonce":2489540211}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:44:31.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:31.405+0000 7f42ac190700 1 -- 172.21.15.42:0/3479846969 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f428403adb0 msgr2=0x7f428403d2c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:31.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:31.405+0000 7f42ac190700 1 --2- 172.21.15.42:0/3479846969 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f428403adb0 0x7f428403d2c0 crc :-1 s=READY pgs=1316 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:31.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:31.405+0000 7f42ac190700 1 -- 172.21.15.42:0/3479846969 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f42a408ee90 msgr2=0x7f42a4063dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:31.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:31.405+0000 7f42ac190700 1 --2- 172.21.15.42:0/3479846969 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f42a408ee90 0x7f42a4063dc0 secure :-1 s=READY pgs=687 cs=0 l=1 rx=0x7f4294002f30 tx=0x7f42940108f0).stop 2019-12-05T06:44:31.409 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:31.405+0000 7f42ac190700 1 -- 172.21.15.42:0/3479846969 shutdown_connections 2019-12-05T06:44:31.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:31.405+0000 7f42ac190700 1 --2- 172.21.15.42:0/3479846969 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f428403adb0 0x7f428403d2c0 unknown :-1 s=CLOSED pgs=1316 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:31.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:31.405+0000 7f42ac190700 1 --2- 172.21.15.42:0/3479846969 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f42a408ee90 0x7f42a4063dc0 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:31.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:31.405+0000 7f42ac190700 1 --2- 172.21.15.42:0/3479846969 >> v2:172.21.15.40:3300/0 conn(0x7f42a4065030 0x7f42a412a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:31.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:31.405+0000 7f42ac190700 1 -- 172.21.15.42:0/3479846969 shutdown_connections 2019-12-05T06:44:31.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:31.405+0000 7f42ac190700 1 -- 172.21.15.42:0/3479846969 wait complete. 2019-12-05T06:44:31.410 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:31.405+0000 7f42ac190700 1 -- 172.21.15.42:0/3479846969 >> 172.21.15.42:0/3479846969 conn(0x7f42a40a6ef0 msgr2=0x7f42a407c450 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:44:31.418 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2019-12-05T06:44:33.418 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:44:33.513 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:33.501+0000 7f42805f9700 1 Processor -- start 2019-12-05T06:44:33.513 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:33.505+0000 7f42805f9700 1 -- start start 2019-12-05T06:44:33.514 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:33.505+0000 7f42805f9700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f42780a7740 0x7f427808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:44:33.514 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:33.505+0000 7f42805f9700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f427807e720 0x7f427807eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:44:33.514 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:33.505+0000 7f42805f9700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42780a0710 con 0x7f427809f2e0 2019-12-05T06:44:33.514 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:33.505+0000 7f42805f9700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42780a0b00 con 0x7f42780a7740 2019-12-05T06:44:33.514 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:33.505+0000 7f42805f9700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f427808e5b0 con 0x7f427807e720 2019-12-05T06:44:33.514 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:33.505+0000 7f427e395700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f427807e720 0x7f427807eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:33.515 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:33.505+0000 7f427e395700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f427807e720 0x7f427807eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:34812/0 (socket says 172.21.15.42:34812) 2019-12-05T06:44:33.515 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:33.505+0000 7f427e395700 1 -- 172.21.15.42:0/3257871763 learned_addr learned my addr 172.21.15.42:0/3257871763 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:44:33.515 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:33.505+0000 7f427eb96700 1 --2- 172.21.15.42:0/3257871763 >> v2:172.21.15.40:3300/0 conn(0x7f42780a7740 0x7f427808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:33.515 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:33.505+0000 7f427e395700 1 -- 172.21.15.42:0/3257871763 >> v2:172.21.15.40:3300/0 conn(0x7f42780a7740 msgr2=0x7f427808df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:44:33.515 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:33.505+0000 7f427e395700 1 --2- 172.21.15.42:0/3257871763 >> v2:172.21.15.40:3300/0 conn(0x7f42780a7740 0x7f427808df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:44:33.515 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:33.505+0000 7f427e395700 1 -- 172.21.15.42:0/3257871763 >> v1:172.21.15.40:6789/0 conn(0x7f427809f2e0 legacy=0x7f427809f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:33.516 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:33.505+0000 7f427e395700 1 -- 172.21.15.42:0/3257871763 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f427807f0a0 con 0x7f427807e720 2019-12-05T06:44:33.516 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:33.505+0000 7f427e395700 1 --2- 172.21.15.42:0/3257871763 >> v2:172.21.15.42:3300/0 conn(0x7f427807e720 0x7f427807eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f42680032b0 tx=0x7f42680100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:33.516 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:33.505+0000 7f427e395700 1 --2- 172.21.15.42:0/3257871763 >> v2:172.21.15.42:3300/0 conn(0x7f427807e720 0x7f427807eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f42680032b0 tx=0x7f42680100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:33.516 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:33.505+0000 7f427e395700 1 --2- 172.21.15.42:0/3257871763 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f427807e720 0x7f427807eb50 secure :-1 s=READY pgs=3151 cs=0 l=1 rx=0x7f42680032b0 tx=0x7f42680100b0).ready entity=mon.1 client_cookie=96074e019b09b65e server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:44:33.516 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:33.505+0000 7f427e395700 1 --2- 172.21.15.42:0/3257871763 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f427807e720 0x7f427807eb50 secure :-1 s=THROTTLE_DONE pgs=3151 cs=0 l=1 rx=0x7f42680032b0 tx=0x7f42680100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:33.516 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:33.505+0000 7f427db94700 1 -- 172.21.15.42:0/3257871763 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4268011dd0 con 0x7f427807e720 2019-12-05T06:44:33.517 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:33.505+0000 7f427e395700 1 --2- 172.21.15.42:0/3257871763 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f427807e720 0x7f427807eb50 secure :-1 s=THROTTLE_DONE pgs=3151 cs=0 l=1 rx=0x7f42680032b0 tx=0x7f42680100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:33.517 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:33.505+0000 7f427db94700 1 -- 172.21.15.42:0/3257871763 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42680119f0 con 0x7f427807e720 2019-12-05T06:44:33.517 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:33.505+0000 7f427e395700 1 --2- 172.21.15.42:0/3257871763 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f427807e720 0x7f427807eb50 secure :-1 s=THROTTLE_DONE pgs=3151 cs=0 l=1 rx=0x7f42680032b0 tx=0x7f42680100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:33.517 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:33.505+0000 7f427db94700 1 -- 172.21.15.42:0/3257871763 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f426800a540 con 0x7f427807e720 2019-12-05T06:44:33.517 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:33.505+0000 7f42805f9700 1 -- 172.21.15.42:0/3257871763 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f427807e720 msgr2=0x7f427807eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:33.517 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:33.505+0000 7f42805f9700 1 --2- 172.21.15.42:0/3257871763 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f427807e720 0x7f427807eb50 secure :-1 s=READY pgs=3151 cs=0 l=1 rx=0x7f42680032b0 tx=0x7f42680100b0).stop 2019-12-05T06:44:33.518 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:33.509+0000 7f42805f9700 1 -- 172.21.15.42:0/3257871763 shutdown_connections 2019-12-05T06:44:33.518 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:33.509+0000 7f42805f9700 1 --2- 172.21.15.42:0/3257871763 >> v2:172.21.15.40:3300/0 conn(0x7f42780a7740 0x7f427808df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:44:33.518 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:33.509+0000 7f42805f9700 1 --2- 172.21.15.42:0/3257871763 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f427807e720 0x7f427807eb50 unknown :-1 s=CLOSED pgs=3151 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:33.518 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:33.509+0000 7f42805f9700 1 -- 172.21.15.42:0/3257871763 shutdown_connections 2019-12-05T06:44:33.518 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:33.509+0000 7f42805f9700 1 -- 172.21.15.42:0/3257871763 wait complete. 2019-12-05T06:44:33.518 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:33.509+0000 7f42805f9700 1 -- 172.21.15.42:0/3257871763 >> 172.21.15.42:0/3257871763 conn(0x7f42780aabe0 msgr2=0x7f42780a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:44:33.523 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:33.509+0000 7f42805f9700 1 Processor -- start 2019-12-05T06:44:33.524 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:33.509+0000 7f42805f9700 1 -- start start 2019-12-05T06:44:33.524 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:33.509+0000 7f42805f9700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f427811c1a0 0x7f427811b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:44:33.524 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:33.509+0000 7f42805f9700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f427807c550 0x7f427808f350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:44:33.525 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:33.509+0000 7f42805f9700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f427807d930 con 0x7f4278116360 2019-12-05T06:44:33.525 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:33.509+0000 7f42805f9700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4278116790 con 0x7f427807c550 2019-12-05T06:44:33.525 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:33.509+0000 7f42805f9700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f427811bc60 con 0x7f427811c1a0 2019-12-05T06:44:33.525 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:33.513+0000 7f427e395700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f4278116360 0x7f427808eb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:34858/0 (socket says 172.21.15.42:34858) 2019-12-05T06:44:33.525 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:33.513+0000 7f427e395700 1 -- 172.21.15.42:0/1804009513 learned_addr learned my addr 172.21.15.42:0/1804009513 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:44:33.526 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:33.513+0000 7f427eb96700 1 --2- 172.21.15.42:0/1804009513 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f427811c1a0 0x7f427811b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:33.526 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:33.513+0000 7f427f397700 1 --2- 172.21.15.42:0/1804009513 >> v2:172.21.15.40:3300/0 conn(0x7f427807c550 0x7f427808f350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:33.526 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:33.513+0000 7f426f7fe700 1 -- 172.21.15.42:0/1804009513 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 356197577 0 0) 0x7f427807d930 con 0x7f4278116360 2019-12-05T06:44:33.526 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:33.513+0000 7f426f7fe700 1 -- 172.21.15.42:0/1804009513 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42580030b0 con 0x7f4278116360 2019-12-05T06:44:33.526 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:33.513+0000 7f426f7fe700 1 -- 172.21.15.42:0/1804009513 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 3997351338 0 0) 0x7f42580030b0 con 0x7f4278116360 2019-12-05T06:44:33.526 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:33.513+0000 7f426f7fe700 1 -- 172.21.15.42:0/1804009513 --> v1:172.21.15.42:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f427807d930 con 0x7f4278116360 2019-12-05T06:44:33.527 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:33.513+0000 7f426f7fe700 1 -- 172.21.15.42:0/1804009513 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7f42680087f0 con 0x7f4278116360 2019-12-05T06:44:33.527 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:33.513+0000 7f426f7fe700 1 -- 172.21.15.42:0/1804009513 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f427811c1a0 msgr2=0x7f427811b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:33.527 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:33.513+0000 7f426f7fe700 1 --2- 172.21.15.42:0/1804009513 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f427811c1a0 0x7f427811b580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:33.527 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:33.513+0000 7f426f7fe700 1 -- 172.21.15.42:0/1804009513 >> v2:172.21.15.40:3300/0 conn(0x7f427807c550 msgr2=0x7f427808f350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:33.527 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:33.513+0000 7f426f7fe700 1 --2- 172.21.15.42:0/1804009513 >> v2:172.21.15.40:3300/0 conn(0x7f427807c550 0x7f427808f350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:33.527 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:33.513+0000 7f426f7fe700 1 -- 172.21.15.42:0/1804009513 >> v1:172.21.15.42:6789/0 conn(0x7f4278116360 legacy=0x7f427808eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:33.527 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:33.513+0000 7f426f7fe700 1 --2- 172.21.15.42:0/1804009513 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f42580038a0 0x7f4258003d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:44:33.528 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:33.513+0000 7f426f7fe700 1 --2- 172.21.15.42:0/1804009513 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4258004420 0x7f4258004850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:44:33.528 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:33.513+0000 7f426f7fe700 1 --2- 172.21.15.42:0/1804009513 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4258004e40 0x7f425800b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:44:33.528 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:33.513+0000 7f426f7fe700 1 -- 172.21.15.42:0/1804009513 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4278116790 con 0x7f42580038a0 2019-12-05T06:44:33.528 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:33.513+0000 7f426f7fe700 1 -- 172.21.15.42:0/1804009513 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f427811bc60 con 0x7f4258004420 2019-12-05T06:44:33.528 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:33.513+0000 7f426f7fe700 1 -- 172.21.15.42:0/1804009513 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f425800bcd0 con 0x7f4258004e40 2019-12-05T06:44:33.528 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:33.513+0000 7f427f397700 1 --2- 172.21.15.42:0/1804009513 >> v2:172.21.15.40:3300/0 conn(0x7f427807c550 0x7f427808f350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:44:33.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:33.513+0000 7f427eb96700 1 --2- 172.21.15.42:0/1804009513 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f427811c1a0 0x7f427811b580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:44:33.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:33.513+0000 7f427e395700 1 --2- 172.21.15.42:0/1804009513 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4258004420 0x7f4258004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:33.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:33.513+0000 7f427f397700 1 --2- 172.21.15.42:0/1804009513 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4258004e40 0x7f425800b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:33.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:33.513+0000 7f427eb96700 1 --2- 172.21.15.42:0/1804009513 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f42580038a0 0x7f4258003d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:33.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:33.513+0000 7f427e395700 1 -- 172.21.15.42:0/1804009513 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4258004e40 msgr2=0x7f425800b670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:33.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:33.513+0000 7f427e395700 1 --2- 172.21.15.42:0/1804009513 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4258004e40 0x7f425800b670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:33.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:33.513+0000 7f427e395700 1 -- 172.21.15.42:0/1804009513 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f42580038a0 msgr2=0x7f4258003d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:33.530 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:33.513+0000 7f427e395700 1 --2- 172.21.15.42:0/1804009513 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f42580038a0 0x7f4258003d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:33.530 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:33.513+0000 7f427e395700 1 -- 172.21.15.42:0/1804009513 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f427808f8a0 con 0x7f4258004420 2019-12-05T06:44:33.530 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:33.513+0000 7f42805f9700 1 -- 172.21.15.42:0/1804009513 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f427811a5f0 con 0x7f4258004420 2019-12-05T06:44:33.530 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:33.513+0000 7f427e395700 1 --2- 172.21.15.42:0/1804009513 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4258004420 0x7f4258004850 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f4268003270 tx=0x7f426800f5e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:33.530 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:33.513+0000 7f42805f9700 1 -- 172.21.15.42:0/1804009513 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f427807d5b0 con 0x7f4258004420 2019-12-05T06:44:33.530 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:33.513+0000 7f427e395700 1 --2- 172.21.15.42:0/1804009513 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4258004420 0x7f4258004850 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f4268003270 tx=0x7f426800f5e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:33.531 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:33.513+0000 7f427e395700 1 --2- 172.21.15.42:0/1804009513 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4258004420 0x7f4258004850 secure :-1 s=READY pgs=3153 cs=0 l=1 rx=0x7f4268003270 tx=0x7f426800f5e0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:44:33.531 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:33.513+0000 7f427e395700 1 --2- 172.21.15.42:0/1804009513 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4258004420 0x7f4258004850 secure :-1 s=THROTTLE_DONE pgs=3153 cs=0 l=1 rx=0x7f4268003270 tx=0x7f426800f5e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:33.531 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:33.513+0000 7f426f7fe700 1 -- 172.21.15.42:0/1804009513 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4268005ab0 con 0x7f4258004420 2019-12-05T06:44:33.531 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:33.513+0000 7f427e395700 1 --2- 172.21.15.42:0/1804009513 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4258004420 0x7f4258004850 secure :-1 s=THROTTLE_DONE pgs=3153 cs=0 l=1 rx=0x7f4268003270 tx=0x7f426800f5e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:33.531 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:33.513+0000 7f426f7fe700 1 -- 172.21.15.42:0/1804009513 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f426800fa30 con 0x7f4258004420 2019-12-05T06:44:33.531 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:33.513+0000 7f427e395700 1 --2- 172.21.15.42:0/1804009513 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4258004420 0x7f4258004850 secure :-1 s=THROTTLE_DONE pgs=3153 cs=0 l=1 rx=0x7f4268003270 tx=0x7f426800f5e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:33.532 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:33.513+0000 7f426f7fe700 1 -- 172.21.15.42:0/1804009513 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4268007340 con 0x7f4258004420 2019-12-05T06:44:33.532 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:33.513+0000 7f427e395700 1 --2- 172.21.15.42:0/1804009513 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4258004420 0x7f4258004850 secure :-1 s=THROTTLE_DONE pgs=3153 cs=0 l=1 rx=0x7f4268003270 tx=0x7f426800f5e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:33.532 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:33.517+0000 7f426f7fe700 1 -- 172.21.15.42:0/1804009513 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f4268007540 con 0x7f4258004420 2019-12-05T06:44:33.532 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:33.517+0000 7f427e395700 1 --2- 172.21.15.42:0/1804009513 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4258004420 0x7f4258004850 secure :-1 s=THROTTLE_DONE pgs=3153 cs=0 l=1 rx=0x7f4268003270 tx=0x7f426800f5e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:33.532 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:33.517+0000 7f426f7fe700 1 --2- 172.21.15.42:0/1804009513 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4258043ba0 0x7f42580460b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:44:33.533 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:33.517+0000 7f426f7fe700 1 -- 172.21.15.42:0/1804009513 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 11965+0+0 (secure 0 0 0) 0x7f42680906c0 con 0x7f4258004420 2019-12-05T06:44:33.533 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:33.517+0000 7f427eb96700 1 --2- 172.21.15.42:0/1804009513 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4258043ba0 0x7f42580460b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:33.533 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:33.517+0000 7f427eb96700 1 --2- 172.21.15.42:0/1804009513 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4258043ba0 0x7f42580460b0 crc :-1 s=READY pgs=1317 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:44:33.534 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:33.521+0000 7f42805f9700 1 -- 172.21.15.42:0/1804009513 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4254003200 con 0x7f4258004420 2019-12-05T06:44:33.534 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:33.525+0000 7f427e395700 1 --2- 172.21.15.42:0/1804009513 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4258004420 0x7f4258004850 secure :-1 s=THROTTLE_DONE pgs=3153 cs=0 l=1 rx=0x7f4268003270 tx=0x7f426800f5e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:33.534 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:33.525+0000 7f426f7fe700 1 -- 172.21.15.42:0/1804009513 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f4268077740 con 0x7f4258004420 2019-12-05T06:44:33.695 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:33.689+0000 7f42805f9700 1 -- 172.21.15.42:0/1804009513 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4254003630 con 0x7f4258004420 2019-12-05T06:44:33.696 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:33.693+0000 7f427e395700 1 --2- 172.21.15.42:0/1804009513 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4258004420 0x7f4258004850 secure :-1 s=THROTTLE_DONE pgs=3153 cs=0 l=1 rx=0x7f4268003270 tx=0x7f426800f5e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:33.696 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:33.693+0000 7f426f7fe700 1 -- 172.21.15.42:0/1804009513 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 381 v381) v1 ==== 95+0+1785 (secure 0 0 0) 0x7f42680779a0 con 0x7f4258004420 2019-12-05T06:44:33.698 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:44:33.698 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":381,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":381,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:44:33.315374+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":118,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12303},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12303":{"gid":12303,"name":"a","rank":0,"incarnation":378,"state":"up:clientreplay","state_seq":16,"addr":"172.21.15.42:6835/2489540211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2489540211},{"type":"v1","addr":"172.21.15.42:6835","nonce":2489540211}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:44:33.698 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 381 2019-12-05T06:44:33.700 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:33.697+0000 7f42805f9700 1 -- 172.21.15.42:0/1804009513 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4258043ba0 msgr2=0x7f42580460b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:33.701 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:33.697+0000 7f42805f9700 1 --2- 172.21.15.42:0/1804009513 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4258043ba0 0x7f42580460b0 crc :-1 s=READY pgs=1317 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:33.701 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:33.697+0000 7f42805f9700 1 -- 172.21.15.42:0/1804009513 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4258004420 msgr2=0x7f4258004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:33.701 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:33.697+0000 7f42805f9700 1 --2- 172.21.15.42:0/1804009513 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4258004420 0x7f4258004850 secure :-1 s=READY pgs=3153 cs=0 l=1 rx=0x7f4268003270 tx=0x7f426800f5e0).stop 2019-12-05T06:44:33.701 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:33.697+0000 7f427f397700 1 -- 172.21.15.42:0/1804009513 reap_dead start 2019-12-05T06:44:33.701 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:33.697+0000 7f427f397700 1 -- 172.21.15.42:0/1804009513 reap_dead start 2019-12-05T06:44:33.701 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:33.697+0000 7f42805f9700 1 -- 172.21.15.42:0/1804009513 shutdown_connections 2019-12-05T06:44:33.702 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:33.697+0000 7f42805f9700 1 -- 172.21.15.42:0/1804009513 shutdown_connections 2019-12-05T06:44:33.702 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:33.697+0000 7f42805f9700 1 -- 172.21.15.42:0/1804009513 wait complete. 2019-12-05T06:44:33.702 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:33.697+0000 7f42805f9700 1 -- 172.21.15.42:0/1804009513 >> 172.21.15.42:0/1804009513 conn(0x7f42780a7f90 msgr2=0x7f4278059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:44:33.714 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2019-12-05T06:44:35.720 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:44:35.826 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:35.813+0000 7fae93d51700 1 Processor -- start 2019-12-05T06:44:35.827 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:35.813+0000 7fae93d51700 1 -- start start 2019-12-05T06:44:35.827 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:35.813+0000 7fae93d51700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fae8c09fad0 0x7fae8c14ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:44:35.827 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:35.813+0000 7fae93d51700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae8c0a0700 con 0x7fae8c0a7730 2019-12-05T06:44:35.827 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:35.813+0000 7fae93d51700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae8c07ef10 con 0x7fae8c08df70 2019-12-05T06:44:35.828 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:35.813+0000 7fae93d51700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae8c0a0af0 con 0x7fae8c09fad0 2019-12-05T06:44:35.828 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:35.813+0000 7fae92aef700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fae8c09fad0 0x7fae8c14ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:35.828 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:35.813+0000 7fae92aef700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fae8c09fad0 0x7fae8c14ffe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:34846/0 (socket says 172.21.15.42:34846) 2019-12-05T06:44:35.828 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:35.813+0000 7fae91aed700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fae8c08df70 0x7fae8c09f2d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:34882/0 (socket says 172.21.15.42:34882) 2019-12-05T06:44:35.828 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:35.813+0000 7fae92aef700 1 -- 172.21.15.42:0/3448890583 learned_addr learned my addr 172.21.15.42:0/3448890583 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:44:35.829 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:35.813+0000 7fae92aef700 1 -- 172.21.15.42:0/3448890583 >> v1:172.21.15.42:6789/0 conn(0x7fae8c08df70 legacy=0x7fae8c09f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:35.829 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:35.813+0000 7fae92aef700 1 -- 172.21.15.42:0/3448890583 >> v1:172.21.15.40:6789/0 conn(0x7fae8c0a7730 legacy=0x7fae8c07e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:35.829 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:35.813+0000 7fae92aef700 1 -- 172.21.15.42:0/3448890583 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae8c0a9b90 con 0x7fae8c09fad0 2019-12-05T06:44:35.829 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:35.813+0000 7fae92aef700 1 --2- 172.21.15.42:0/3448890583 >> v2:172.21.15.42:3300/0 conn(0x7fae8c09fad0 0x7fae8c14ffe0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fae84008b10 tx=0x7fae84010030).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:35.829 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:35.813+0000 7fae92aef700 1 --2- 172.21.15.42:0/3448890583 >> v2:172.21.15.42:3300/0 conn(0x7fae8c09fad0 0x7fae8c14ffe0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fae84008b10 tx=0x7fae84010030).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:35.829 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:35.813+0000 7fae92aef700 1 --2- 172.21.15.42:0/3448890583 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fae8c09fad0 0x7fae8c14ffe0 secure :-1 s=READY pgs=3155 cs=0 l=1 rx=0x7fae84008b10 tx=0x7fae84010030).ready entity=mon.1 client_cookie=31c6bfbd1791d9fd server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:44:35.830 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:35.817+0000 7fae92aef700 1 --2- 172.21.15.42:0/3448890583 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fae8c09fad0 0x7fae8c14ffe0 secure :-1 s=THROTTLE_DONE pgs=3155 cs=0 l=1 rx=0x7fae84008b10 tx=0x7fae84010030).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:35.830 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:35.817+0000 7fae92aef700 1 --2- 172.21.15.42:0/3448890583 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fae8c09fad0 0x7fae8c14ffe0 secure :-1 s=THROTTLE_DONE pgs=3155 cs=0 l=1 rx=0x7fae84008b10 tx=0x7fae84010030).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:35.830 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:35.817+0000 7fae912ec700 1 -- 172.21.15.42:0/3448890583 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fae840110c0 con 0x7fae8c09fad0 2019-12-05T06:44:35.830 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:35.817+0000 7fae92aef700 1 --2- 172.21.15.42:0/3448890583 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fae8c09fad0 0x7fae8c14ffe0 secure :-1 s=THROTTLE_DONE pgs=3155 cs=0 l=1 rx=0x7fae84008b10 tx=0x7fae84010030).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:35.830 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:35.817+0000 7fae912ec700 1 -- 172.21.15.42:0/3448890583 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae84010a30 con 0x7fae8c09fad0 2019-12-05T06:44:35.830 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:35.817+0000 7fae912ec700 1 -- 172.21.15.42:0/3448890583 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fae840114b0 con 0x7fae8c09fad0 2019-12-05T06:44:35.831 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:35.817+0000 7fae93d51700 1 -- 172.21.15.42:0/3448890583 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fae8c09fad0 msgr2=0x7fae8c14ffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:35.831 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:35.817+0000 7fae93d51700 1 --2- 172.21.15.42:0/3448890583 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fae8c09fad0 0x7fae8c14ffe0 secure :-1 s=READY pgs=3155 cs=0 l=1 rx=0x7fae84008b10 tx=0x7fae84010030).stop 2019-12-05T06:44:35.831 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:35.817+0000 7fae93d51700 1 -- 172.21.15.42:0/3448890583 shutdown_connections 2019-12-05T06:44:35.831 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:35.817+0000 7fae93d51700 1 --2- 172.21.15.42:0/3448890583 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fae8c09fad0 0x7fae8c14ffe0 unknown :-1 s=CLOSED pgs=3155 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:35.831 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:35.817+0000 7fae93d51700 1 -- 172.21.15.42:0/3448890583 shutdown_connections 2019-12-05T06:44:35.831 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:35.817+0000 7fae93d51700 1 -- 172.21.15.42:0/3448890583 wait complete. 2019-12-05T06:44:35.831 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:35.817+0000 7fae93d51700 1 -- 172.21.15.42:0/3448890583 >> 172.21.15.42:0/3448890583 conn(0x7fae8c0aabd0 msgr2=0x7fae8c0a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:44:35.832 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:35.817+0000 7fae93d51700 1 Processor -- start 2019-12-05T06:44:35.832 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:35.817+0000 7fae93d51700 1 -- start start 2019-12-05T06:44:35.832 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:35.817+0000 7fae93d51700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fae8c115b10 0x7fae8c0a6c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:44:35.832 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:35.817+0000 7fae93d51700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fae8c064ad0 0x7fae8c0a5360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:44:35.832 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:35.817+0000 7fae93d51700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae8c0a6020 con 0x7fae8c0a58b0 2019-12-05T06:44:35.832 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:35.817+0000 7fae93d51700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae8c064f00 con 0x7fae8c115b10 2019-12-05T06:44:35.832 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:35.817+0000 7fae93d51700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae8c064590 con 0x7fae8c064ad0 2019-12-05T06:44:35.833 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:35.817+0000 7fae91aed700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fae8c064ad0 0x7fae8c0a5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:35.833 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:35.817+0000 7fae922ee700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fae8c115b10 0x7fae8c0a6c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:35.833 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:35.817+0000 7fae91aed700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fae8c064ad0 0x7fae8c0a5360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:35316/0 (socket says 172.21.15.42:35316) 2019-12-05T06:44:35.833 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:35.817+0000 7fae91aed700 1 -- 172.21.15.42:0/4115255175 learned_addr learned my addr 172.21.15.42:0/4115255175 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:44:35.833 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:35.817+0000 7fae91aed700 1 -- 172.21.15.42:0/4115255175 >> v2:172.21.15.40:3300/0 conn(0x7fae8c115b10 msgr2=0x7fae8c0a6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:35.833 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:35.817+0000 7fae91aed700 1 --2- 172.21.15.42:0/4115255175 >> v2:172.21.15.40:3300/0 conn(0x7fae8c115b10 0x7fae8c0a6c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:35.834 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:35.817+0000 7fae91aed700 1 -- 172.21.15.42:0/4115255175 >> v1:172.21.15.40:6789/0 conn(0x7fae8c0a58b0 legacy=0x7fae8c07b820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:35.834 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:35.817+0000 7fae91aed700 1 -- 172.21.15.42:0/4115255175 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae8c07c020 con 0x7fae8c064ad0 2019-12-05T06:44:35.834 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:35.817+0000 7fae91aed700 1 --2- 172.21.15.42:0/4115255175 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fae8c064ad0 0x7fae8c0a5360 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fae7c000b80 tx=0x7fae7c00c390).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:35.834 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:35.817+0000 7fae91aed700 1 --2- 172.21.15.42:0/4115255175 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fae8c064ad0 0x7fae8c0a5360 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fae7c000b80 tx=0x7fae7c00c390).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:35.834 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:35.821+0000 7fae91aed700 1 --2- 172.21.15.42:0/4115255175 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fae8c064ad0 0x7fae8c0a5360 secure :-1 s=READY pgs=688 cs=0 l=1 rx=0x7fae7c000b80 tx=0x7fae7c00c390).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:44:35.834 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:35.821+0000 7fae91aed700 1 --2- 172.21.15.42:0/4115255175 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fae8c064ad0 0x7fae8c0a5360 secure :-1 s=THROTTLE_DONE pgs=688 cs=0 l=1 rx=0x7fae7c000b80 tx=0x7fae7c00c390).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:35.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:35.821+0000 7fae82ffd700 1 -- 172.21.15.42:0/4115255175 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fae7c011620 con 0x7fae8c064ad0 2019-12-05T06:44:35.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:35.821+0000 7fae91aed700 1 --2- 172.21.15.42:0/4115255175 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fae8c064ad0 0x7fae8c0a5360 secure :-1 s=THROTTLE_DONE pgs=688 cs=0 l=1 rx=0x7fae7c000b80 tx=0x7fae7c00c390).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:35.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:35.821+0000 7fae91aed700 1 --2- 172.21.15.42:0/4115255175 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fae8c064ad0 0x7fae8c0a5360 secure :-1 s=THROTTLE_DONE pgs=688 cs=0 l=1 rx=0x7fae7c000b80 tx=0x7fae7c00c390).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:35.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:35.821+0000 7fae82ffd700 1 -- 172.21.15.42:0/4115255175 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae7c011240 con 0x7fae8c064ad0 2019-12-05T06:44:35.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:35.821+0000 7fae82ffd700 1 -- 172.21.15.42:0/4115255175 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fae7c011c00 con 0x7fae8c064ad0 2019-12-05T06:44:35.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:35.821+0000 7fae93d51700 1 -- 172.21.15.42:0/4115255175 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae8c064100 con 0x7fae8c064ad0 2019-12-05T06:44:35.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:35.821+0000 7fae93d51700 1 -- 172.21.15.42:0/4115255175 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae8c12f5f0 con 0x7fae8c064ad0 2019-12-05T06:44:35.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:35.821+0000 7fae91aed700 1 --2- 172.21.15.42:0/4115255175 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fae8c064ad0 0x7fae8c0a5360 secure :-1 s=THROTTLE_DONE pgs=688 cs=0 l=1 rx=0x7fae7c000b80 tx=0x7fae7c00c390).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:35.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:35.821+0000 7fae82ffd700 1 -- 172.21.15.42:0/4115255175 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fae7c006460 con 0x7fae8c064ad0 2019-12-05T06:44:35.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:35.821+0000 7fae91aed700 1 --2- 172.21.15.42:0/4115255175 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fae8c064ad0 0x7fae8c0a5360 secure :-1 s=THROTTLE_DONE pgs=688 cs=0 l=1 rx=0x7fae7c000b80 tx=0x7fae7c00c390).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:35.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:35.821+0000 7fae82ffd700 1 --2- 172.21.15.42:0/4115255175 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fae7403adf0 0x7fae7403d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:44:35.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:35.821+0000 7fae82ffd700 1 -- 172.21.15.42:0/4115255175 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 11965+0+0 (secure 0 0 0) 0x7fae7c04ade0 con 0x7fae8c064ad0 2019-12-05T06:44:35.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:35.821+0000 7fae922ee700 1 --2- 172.21.15.42:0/4115255175 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fae7403adf0 0x7fae7403d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:35.837 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:35.821+0000 7fae93d51700 1 -- 172.21.15.42:0/4115255175 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae8c05d100 con 0x7fae8c064ad0 2019-12-05T06:44:35.837 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:35.821+0000 7fae922ee700 1 --2- 172.21.15.42:0/4115255175 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fae7403adf0 0x7fae7403d300 crc :-1 s=READY pgs=1318 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:44:35.843 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:35.825+0000 7fae91aed700 1 --2- 172.21.15.42:0/4115255175 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fae8c064ad0 0x7fae8c0a5360 secure :-1 s=THROTTLE_DONE pgs=688 cs=0 l=1 rx=0x7fae7c000b80 tx=0x7fae7c00c390).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:35.843 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:35.825+0000 7fae82ffd700 1 -- 172.21.15.42:0/4115255175 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fae7c031e60 con 0x7fae8c064ad0 2019-12-05T06:44:35.987 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:35.981+0000 7fae93d51700 1 -- 172.21.15.42:0/4115255175 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fae8c05b8b0 con 0x7fae8c064ad0 2019-12-05T06:44:35.988 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:35.985+0000 7fae91aed700 1 --2- 172.21.15.42:0/4115255175 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fae8c064ad0 0x7fae8c0a5360 secure :-1 s=THROTTLE_DONE pgs=688 cs=0 l=1 rx=0x7fae7c000b80 tx=0x7fae7c00c390).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:35.988 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:35.985+0000 7fae82ffd700 1 -- 172.21.15.42:0/4115255175 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 382 v382) v1 ==== 95+0+1779 (secure 0 0 0) 0x7fae7c0320c0 con 0x7fae8c064ad0 2019-12-05T06:44:35.990 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 382 2019-12-05T06:44:35.990 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:44:35.991 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":382,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":382,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:44:34.328051+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":118,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12303},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12303":{"gid":12303,"name":"a","rank":0,"incarnation":378,"state":"up:active","state_seq":17,"addr":"172.21.15.42:6835/2489540211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2489540211},{"type":"v1","addr":"172.21.15.42:6835","nonce":2489540211}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:44:35.993 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:35.989+0000 7fae93d51700 1 -- 172.21.15.42:0/4115255175 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fae7403adf0 msgr2=0x7fae7403d300 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:35.993 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:35.989+0000 7fae93d51700 1 --2- 172.21.15.42:0/4115255175 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fae7403adf0 0x7fae7403d300 crc :-1 s=READY pgs=1318 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:35.993 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:35.989+0000 7fae93d51700 1 -- 172.21.15.42:0/4115255175 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fae8c064ad0 msgr2=0x7fae8c0a5360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:35.993 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:35.989+0000 7fae93d51700 1 --2- 172.21.15.42:0/4115255175 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fae8c064ad0 0x7fae8c0a5360 secure :-1 s=READY pgs=688 cs=0 l=1 rx=0x7fae7c000b80 tx=0x7fae7c00c390).stop 2019-12-05T06:44:35.994 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:35.989+0000 7fae93d51700 1 -- 172.21.15.42:0/4115255175 shutdown_connections 2019-12-05T06:44:35.994 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:35.989+0000 7fae93d51700 1 --2- 172.21.15.42:0/4115255175 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fae7403adf0 0x7fae7403d300 unknown :-1 s=CLOSED pgs=1318 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:35.994 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:35.989+0000 7fae93d51700 1 --2- 172.21.15.42:0/4115255175 >> v2:172.21.15.40:3300/0 conn(0x7fae8c115b10 0x7fae8c0a6c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:35.994 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:35.989+0000 7fae93d51700 1 --2- 172.21.15.42:0/4115255175 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fae8c064ad0 0x7fae8c0a5360 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:35.994 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:35.989+0000 7fae93d51700 1 -- 172.21.15.42:0/4115255175 shutdown_connections 2019-12-05T06:44:35.995 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:35.989+0000 7fae93d51700 1 -- 172.21.15.42:0/4115255175 wait complete. 2019-12-05T06:44:35.995 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:35.989+0000 7fae93d51700 1 -- 172.21.15.42:0/4115255175 >> 172.21.15.42:0/4115255175 conn(0x7fae8c07e900 msgr2=0x7fae8c07dbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:44:36.003 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a has gained rank=0, replacing gid=12496 2019-12-05T06:44:36.003 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 2 secs before reviving mds.b 2019-12-05T06:44:38.004 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.b 2019-12-05T06:44:38.004 INFO:tasks.ceph.mds.b:Restarting daemon 2019-12-05T06:44:38.004 INFO:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2019-12-05T06:44:38.008 INFO:tasks.ceph.mds.b:Started 2019-12-05T06:44:38.008 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:44:38.053 INFO:tasks.ceph.mds.b.smithi040.stdout:starting mds.b at 2019-12-05T06:44:38.091 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:38.085+0000 7fd82fe9b700 1 Processor -- start 2019-12-05T06:44:38.092 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:38.085+0000 7fd82fe9b700 1 -- start start 2019-12-05T06:44:38.092 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:38.085+0000 7fd82fe9b700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd82808df80 0x7fd82809f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:44:38.092 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:38.085+0000 7fd82fe9b700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd82809fc60 con 0x7fd8280a7740 2019-12-05T06:44:38.092 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:38.085+0000 7fd82fe9b700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8280a0710 con 0x7fd82809f830 2019-12-05T06:44:38.092 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:38.085+0000 7fd82fe9b700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8280a0b00 con 0x7fd82808df80 2019-12-05T06:44:38.092 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:38.085+0000 7fd82ec39700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fd82809f830 0x7fd82814efe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:34918/0 (socket says 172.21.15.42:34918) 2019-12-05T06:44:38.093 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:38.085+0000 7fd82ec39700 1 -- 172.21.15.42:0/2660094639 learned_addr learned my addr 172.21.15.42:0/2660094639 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:44:38.093 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:38.085+0000 7fd82dc37700 1 --2- 172.21.15.42:0/2660094639 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd82808df80 0x7fd82809f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:38.093 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:38.085+0000 7fd82d436700 1 -- 172.21.15.42:0/2660094639 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3271944461 0 0) 0x7fd8280a0710 con 0x7fd82809f830 2019-12-05T06:44:38.093 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:38.085+0000 7fd82d436700 1 -- 172.21.15.42:0/2660094639 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8140030b0 con 0x7fd82809f830 2019-12-05T06:44:38.093 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:38.085+0000 7fd82d436700 1 -- 172.21.15.42:0/2660094639 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2192075302 0 0) 0x7fd82809fc60 con 0x7fd8280a7740 2019-12-05T06:44:38.093 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:38.085+0000 7fd82d436700 1 -- 172.21.15.42:0/2660094639 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8280a0710 con 0x7fd8280a7740 2019-12-05T06:44:38.094 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:38.085+0000 7fd82dc37700 1 -- 172.21.15.42:0/2660094639 >> v1:172.21.15.42:6789/0 conn(0x7fd82809f830 legacy=0x7fd82814efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:38.094 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:38.085+0000 7fd82dc37700 1 -- 172.21.15.42:0/2660094639 >> v1:172.21.15.40:6789/0 conn(0x7fd8280a7740 legacy=0x7fd82807e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:38.094 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:38.085+0000 7fd82dc37700 1 -- 172.21.15.42:0/2660094639 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd82807ef20 con 0x7fd82808df80 2019-12-05T06:44:38.094 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:38.085+0000 7fd82dc37700 1 --2- 172.21.15.42:0/2660094639 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd82808df80 0x7fd82809f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fd818002f30 tx=0x7fd81800fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:38.094 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:38.085+0000 7fd82dc37700 1 --2- 172.21.15.42:0/2660094639 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd82808df80 0x7fd82809f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fd818002f30 tx=0x7fd81800fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:38.094 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:38.085+0000 7fd82dc37700 1 --2- 172.21.15.42:0/2660094639 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd82808df80 0x7fd82809f2e0 secure :-1 s=READY pgs=691 cs=0 l=1 rx=0x7fd818002f30 tx=0x7fd81800fbf0).ready entity=mon.2 client_cookie=de641883017d9379 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:44:38.094 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:38.085+0000 7fd82dc37700 1 --2- 172.21.15.42:0/2660094639 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd82808df80 0x7fd82809f2e0 secure :-1 s=THROTTLE_DONE pgs=691 cs=0 l=1 rx=0x7fd818002f30 tx=0x7fd81800fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:38.095 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:38.085+0000 7fd82d436700 1 -- 172.21.15.42:0/2660094639 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd8180112b0 con 0x7fd82808df80 2019-12-05T06:44:38.095 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:38.085+0000 7fd82dc37700 1 --2- 172.21.15.42:0/2660094639 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd82808df80 0x7fd82809f2e0 secure :-1 s=THROTTLE_DONE pgs=691 cs=0 l=1 rx=0x7fd818002f30 tx=0x7fd81800fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:38.095 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:38.085+0000 7fd82dc37700 1 --2- 172.21.15.42:0/2660094639 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd82808df80 0x7fd82809f2e0 secure :-1 s=THROTTLE_DONE pgs=691 cs=0 l=1 rx=0x7fd818002f30 tx=0x7fd81800fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:38.095 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:38.085+0000 7fd82d436700 1 -- 172.21.15.42:0/2660094639 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd818008bc0 con 0x7fd82808df80 2019-12-05T06:44:38.095 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:38.085+0000 7fd82d436700 1 -- 172.21.15.42:0/2660094639 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd818011890 con 0x7fd82808df80 2019-12-05T06:44:38.095 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:38.085+0000 7fd82fe9b700 1 -- 172.21.15.42:0/2660094639 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd82808df80 msgr2=0x7fd82809f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:38.095 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:38.085+0000 7fd82fe9b700 1 --2- 172.21.15.42:0/2660094639 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd82808df80 0x7fd82809f2e0 secure :-1 s=READY pgs=691 cs=0 l=1 rx=0x7fd818002f30 tx=0x7fd81800fbf0).stop 2019-12-05T06:44:38.096 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:38.085+0000 7fd82fe9b700 1 -- 172.21.15.42:0/2660094639 shutdown_connections 2019-12-05T06:44:38.096 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:38.085+0000 7fd82fe9b700 1 --2- 172.21.15.42:0/2660094639 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd82808df80 0x7fd82809f2e0 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:38.096 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:38.085+0000 7fd82fe9b700 1 -- 172.21.15.42:0/2660094639 shutdown_connections 2019-12-05T06:44:38.096 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:38.085+0000 7fd82fe9b700 1 -- 172.21.15.42:0/2660094639 wait complete. 2019-12-05T06:44:38.096 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:38.085+0000 7fd82fe9b700 1 -- 172.21.15.42:0/2660094639 >> 172.21.15.42:0/2660094639 conn(0x7fd8280aabe0 msgr2=0x7fd8280a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:44:38.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:38.089+0000 7fd82fe9b700 1 Processor -- start 2019-12-05T06:44:38.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:38.089+0000 7fd82fe9b700 1 -- start start 2019-12-05T06:44:38.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:38.089+0000 7fd82fe9b700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd8280a3a60 0x7fd8281462c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:44:38.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:38.089+0000 7fd82fe9b700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8281277b0 con 0x7fd828144200 2019-12-05T06:44:38.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:38.089+0000 7fd82fe9b700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd828144e30 con 0x7fd828146810 2019-12-05T06:44:38.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:38.089+0000 7fd82fe9b700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd828126a70 con 0x7fd8280a3a60 2019-12-05T06:44:38.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:38.089+0000 7fd82dc37700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fd828146810 0x7fd828126fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:34922/0 (socket says 172.21.15.42:34922) 2019-12-05T06:44:38.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:38.089+0000 7fd82dc37700 1 -- 172.21.15.42:0/3610328990 learned_addr learned my addr 172.21.15.42:0/3610328990 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:44:38.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:38.089+0000 7fd82e438700 1 --2- 172.21.15.42:0/3610328990 >> v2:172.21.15.40:3300/0 conn(0x7fd8280a3a60 0x7fd8281462c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:38.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:38.089+0000 7fd81effd700 1 -- 172.21.15.42:0/3610328990 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3239056929 0 0) 0x7fd828144e30 con 0x7fd828146810 2019-12-05T06:44:38.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:38.089+0000 7fd81effd700 1 -- 172.21.15.42:0/3610328990 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8100030b0 con 0x7fd828146810 2019-12-05T06:44:38.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:38.089+0000 7fd81effd700 1 -- 172.21.15.42:0/3610328990 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 278820957 0 0) 0x7fd8281277b0 con 0x7fd828144200 2019-12-05T06:44:38.103 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:38.089+0000 7fd81effd700 1 -- 172.21.15.42:0/3610328990 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd828144e30 con 0x7fd828144200 2019-12-05T06:44:38.103 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:38.089+0000 7fd81effd700 1 -- 172.21.15.42:0/3610328990 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 2161958144 0 0) 0x7fd8100030b0 con 0x7fd828146810 2019-12-05T06:44:38.103 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:38.089+0000 7fd81effd700 1 -- 172.21.15.42:0/3610328990 --> v1:172.21.15.42:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd8281277b0 con 0x7fd828146810 2019-12-05T06:44:38.103 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:38.089+0000 7fd81effd700 1 -- 172.21.15.42:0/3610328990 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7fd8180112b0 con 0x7fd828146810 2019-12-05T06:44:38.103 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:38.089+0000 7fd82e438700 1 -- 172.21.15.42:0/3610328990 >> v1:172.21.15.42:6789/0 conn(0x7fd828146810 legacy=0x7fd828126fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:38.103 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:38.089+0000 7fd82e438700 1 -- 172.21.15.42:0/3610328990 >> v1:172.21.15.40:6789/0 conn(0x7fd828144200 legacy=0x7fd828144630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:38.103 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:38.089+0000 7fd82e438700 1 -- 172.21.15.42:0/3610328990 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd828145060 con 0x7fd8280a3a60 2019-12-05T06:44:38.104 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:38.089+0000 7fd82e438700 1 --2- 172.21.15.42:0/3610328990 >> v2:172.21.15.40:3300/0 conn(0x7fd8280a3a60 0x7fd8281462c0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fd824000e90 tx=0x7fd8180082d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:38.104 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:38.089+0000 7fd82e438700 1 --2- 172.21.15.42:0/3610328990 >> v2:172.21.15.40:3300/0 conn(0x7fd8280a3a60 0x7fd8281462c0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fd824000e90 tx=0x7fd8180082d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:38.104 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:38.089+0000 7fd82e438700 1 --2- 172.21.15.42:0/3610328990 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd8280a3a60 0x7fd8281462c0 secure :-1 s=READY pgs=2384 cs=0 l=1 rx=0x7fd824000e90 tx=0x7fd8180082d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:44:38.104 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:38.089+0000 7fd82e438700 1 --2- 172.21.15.42:0/3610328990 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd8280a3a60 0x7fd8281462c0 secure :-1 s=THROTTLE_DONE pgs=2384 cs=0 l=1 rx=0x7fd824000e90 tx=0x7fd8180082d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:38.104 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:38.089+0000 7fd81effd700 1 -- 172.21.15.42:0/3610328990 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd82400c960 con 0x7fd8280a3a60 2019-12-05T06:44:38.104 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:38.089+0000 7fd82fe9b700 1 -- 172.21.15.42:0/3610328990 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8281452d0 con 0x7fd8280a3a60 2019-12-05T06:44:38.105 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:38.089+0000 7fd82e438700 1 --2- 172.21.15.42:0/3610328990 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd8280a3a60 0x7fd8281462c0 secure :-1 s=THROTTLE_DONE pgs=2384 cs=0 l=1 rx=0x7fd824000e90 tx=0x7fd8180082d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:38.105 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:38.089+0000 7fd82e438700 1 --2- 172.21.15.42:0/3610328990 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd8280a3a60 0x7fd8281462c0 secure :-1 s=THROTTLE_DONE pgs=2384 cs=0 l=1 rx=0x7fd824000e90 tx=0x7fd8180082d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:38.105 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:38.089+0000 7fd81effd700 1 -- 172.21.15.42:0/3610328990 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd82400c580 con 0x7fd8280a3a60 2019-12-05T06:44:38.105 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:38.089+0000 7fd82fe9b700 1 -- 172.21.15.42:0/3610328990 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd82812edc0 con 0x7fd8280a3a60 2019-12-05T06:44:38.105 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:38.089+0000 7fd81effd700 1 -- 172.21.15.42:0/3610328990 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd82400d490 con 0x7fd8280a3a60 2019-12-05T06:44:38.105 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:38.089+0000 7fd82e438700 1 --2- 172.21.15.42:0/3610328990 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd8280a3a60 0x7fd8281462c0 secure :-1 s=THROTTLE_DONE pgs=2384 cs=0 l=1 rx=0x7fd824000e90 tx=0x7fd8180082d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:38.105 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:38.089+0000 7fd81effd700 1 -- 172.21.15.42:0/3610328990 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fd82400d7b0 con 0x7fd8280a3a60 2019-12-05T06:44:38.106 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:38.089+0000 7fd82e438700 1 --2- 172.21.15.42:0/3610328990 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd8280a3a60 0x7fd8281462c0 secure :-1 s=THROTTLE_DONE pgs=2384 cs=0 l=1 rx=0x7fd824000e90 tx=0x7fd8180082d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:38.106 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:38.089+0000 7fd81effd700 1 --2- 172.21.15.42:0/3610328990 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd81003b870 0x7fd81003dd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:44:38.106 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:38.089+0000 7fd81effd700 1 -- 172.21.15.42:0/3610328990 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 11965+0+0 (secure 0 0 0) 0x7fd82404af10 con 0x7fd8280a3a60 2019-12-05T06:44:38.106 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:38.089+0000 7fd82dc37700 1 --2- 172.21.15.42:0/3610328990 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd81003b870 0x7fd81003dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:38.106 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:38.089+0000 7fd82fe9b700 1 -- 172.21.15.42:0/3610328990 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd82805d110 con 0x7fd8280a3a60 2019-12-05T06:44:38.106 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:38.093+0000 7fd82dc37700 1 --2- 172.21.15.42:0/3610328990 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd81003b870 0x7fd81003dd80 crc :-1 s=READY pgs=1319 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:44:38.106 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:38.097+0000 7fd82e438700 1 --2- 172.21.15.42:0/3610328990 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd8280a3a60 0x7fd8281462c0 secure :-1 s=THROTTLE_DONE pgs=2384 cs=0 l=1 rx=0x7fd824000e90 tx=0x7fd8180082d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:38.107 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:38.097+0000 7fd81effd700 1 -- 172.21.15.42:0/3610328990 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fd824031f70 con 0x7fd8280a3a60 2019-12-05T06:44:38.267 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:38.261+0000 7fd82fe9b700 1 -- 172.21.15.42:0/3610328990 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd82805b8c0 con 0x7fd8280a3a60 2019-12-05T06:44:38.268 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:38.265+0000 7fd82e438700 1 --2- 172.21.15.42:0/3610328990 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd8280a3a60 0x7fd8281462c0 secure :-1 s=THROTTLE_DONE pgs=2384 cs=0 l=1 rx=0x7fd824000e90 tx=0x7fd8180082d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:38.269 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:38.265+0000 7fd81effd700 1 -- 172.21.15.42:0/3610328990 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 382 v382) v1 ==== 95+0+1779 (secure 0 0 0) 0x7fd8240321d0 con 0x7fd8280a3a60 2019-12-05T06:44:38.271 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 382 2019-12-05T06:44:38.271 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:44:38.271 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":382,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":382,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:44:34.328051+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":118,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12303},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12303":{"gid":12303,"name":"a","rank":0,"incarnation":378,"state":"up:active","state_seq":17,"addr":"172.21.15.42:6835/2489540211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2489540211},{"type":"v1","addr":"172.21.15.42:6835","nonce":2489540211}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:44:38.273 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:38.269+0000 7fd82fe9b700 1 -- 172.21.15.42:0/3610328990 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd81003b870 msgr2=0x7fd81003dd80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:38.274 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:38.269+0000 7fd82fe9b700 1 --2- 172.21.15.42:0/3610328990 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd81003b870 0x7fd81003dd80 crc :-1 s=READY pgs=1319 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:38.274 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:38.269+0000 7fd82fe9b700 1 -- 172.21.15.42:0/3610328990 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd8280a3a60 msgr2=0x7fd8281462c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:38.274 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:38.269+0000 7fd82fe9b700 1 --2- 172.21.15.42:0/3610328990 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd8280a3a60 0x7fd8281462c0 secure :-1 s=READY pgs=2384 cs=0 l=1 rx=0x7fd824000e90 tx=0x7fd8180082d0).stop 2019-12-05T06:44:38.274 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:38.269+0000 7fd82fe9b700 1 -- 172.21.15.42:0/3610328990 shutdown_connections 2019-12-05T06:44:38.274 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:38.269+0000 7fd82fe9b700 1 --2- 172.21.15.42:0/3610328990 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd81003b870 0x7fd81003dd80 unknown :-1 s=CLOSED pgs=1319 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:38.275 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:38.269+0000 7fd82fe9b700 1 --2- 172.21.15.42:0/3610328990 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd8280a3a60 0x7fd8281462c0 unknown :-1 s=CLOSED pgs=2384 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:38.275 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:38.269+0000 7fd82fe9b700 1 -- 172.21.15.42:0/3610328990 shutdown_connections 2019-12-05T06:44:38.275 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:38.269+0000 7fd82fe9b700 1 -- 172.21.15.42:0/3610328990 wait complete. 2019-12-05T06:44:38.275 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:38.269+0000 7fd82fe9b700 1 -- 172.21.15.42:0/3610328990 >> 172.21.15.42:0/3610328990 conn(0x7fd8280a7f90 msgr2=0x7fd828059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:44:38.284 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2019-12-05T06:44:40.286 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:44:40.393 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:40.369+0000 7f8a077d1700 1 Processor -- start 2019-12-05T06:44:40.394 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:40.369+0000 7f8a077d1700 1 -- start start 2019-12-05T06:44:40.394 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:40.369+0000 7f8a077d1700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8a000a7740 0x7f8a0008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:44:40.394 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:40.369+0000 7f8a077d1700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f8a0007e720 0x7f8a0007eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:44:40.394 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:40.369+0000 7f8a077d1700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a000a0710 con 0x7f8a0009f2e0 2019-12-05T06:44:40.394 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:40.369+0000 7f8a077d1700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a000a0b00 con 0x7f8a000a7740 2019-12-05T06:44:40.394 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:40.369+0000 7f8a077d1700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a0008e5b0 con 0x7f8a0007e720 2019-12-05T06:44:40.395 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:40.369+0000 7f8a0556d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f8a0007e720 0x7f8a0007eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:40.395 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:40.369+0000 7f8a0556d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f8a0007e720 0x7f8a0007eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:34908/0 (socket says 172.21.15.42:34908) 2019-12-05T06:44:40.395 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:40.369+0000 7f8a0556d700 1 -- 172.21.15.42:0/4026887048 learned_addr learned my addr 172.21.15.42:0/4026887048 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:44:40.395 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:40.369+0000 7f8a05d6e700 1 --2- 172.21.15.42:0/4026887048 >> v2:172.21.15.40:3300/0 conn(0x7f8a000a7740 0x7f8a0008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:40.395 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:40.369+0000 7f8a0556d700 1 -- 172.21.15.42:0/4026887048 >> v2:172.21.15.40:3300/0 conn(0x7f8a000a7740 msgr2=0x7f8a0008df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:44:40.396 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:40.369+0000 7f8a0556d700 1 --2- 172.21.15.42:0/4026887048 >> v2:172.21.15.40:3300/0 conn(0x7f8a000a7740 0x7f8a0008df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:44:40.398 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:40.369+0000 7f8a0556d700 1 -- 172.21.15.42:0/4026887048 >> v1:172.21.15.40:6789/0 conn(0x7f8a0009f2e0 legacy=0x7f8a0009f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:40.398 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:40.369+0000 7f8a0556d700 1 -- 172.21.15.42:0/4026887048 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a0007f0a0 con 0x7f8a0007e720 2019-12-05T06:44:40.398 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:40.369+0000 7f8a0556d700 1 --2- 172.21.15.42:0/4026887048 >> v2:172.21.15.42:3300/0 conn(0x7f8a0007e720 0x7f8a0007eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f89f0002f30 tx=0x7f89f000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:40.398 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:40.369+0000 7f8a0556d700 1 --2- 172.21.15.42:0/4026887048 >> v2:172.21.15.42:3300/0 conn(0x7f8a0007e720 0x7f8a0007eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f89f0002f30 tx=0x7f89f000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:40.399 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:40.369+0000 7f8a0556d700 1 --2- 172.21.15.42:0/4026887048 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8a0007e720 0x7f8a0007eb50 secure :-1 s=READY pgs=3160 cs=0 l=1 rx=0x7f89f0002f30 tx=0x7f89f000fd30).ready entity=mon.1 client_cookie=dd1639624b2f8 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:44:40.399 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:40.369+0000 7f8a0556d700 1 --2- 172.21.15.42:0/4026887048 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8a0007e720 0x7f8a0007eb50 secure :-1 s=THROTTLE_DONE pgs=3160 cs=0 l=1 rx=0x7f89f0002f30 tx=0x7f89f000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:40.399 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:40.369+0000 7f8a0556d700 1 --2- 172.21.15.42:0/4026887048 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8a0007e720 0x7f8a0007eb50 secure :-1 s=THROTTLE_DONE pgs=3160 cs=0 l=1 rx=0x7f89f0002f30 tx=0x7f89f000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:40.400 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:40.369+0000 7f8a04d6c700 1 -- 172.21.15.42:0/4026887048 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f89f0011370 con 0x7f8a0007e720 2019-12-05T06:44:40.400 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:40.369+0000 7f8a0556d700 1 --2- 172.21.15.42:0/4026887048 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8a0007e720 0x7f8a0007eb50 secure :-1 s=THROTTLE_DONE pgs=3160 cs=0 l=1 rx=0x7f89f0002f30 tx=0x7f89f000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:40.400 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:40.369+0000 7f8a04d6c700 1 -- 172.21.15.42:0/4026887048 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89f0008bb0 con 0x7f8a0007e720 2019-12-05T06:44:40.400 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:40.369+0000 7f8a04d6c700 1 -- 172.21.15.42:0/4026887048 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f89f0011950 con 0x7f8a0007e720 2019-12-05T06:44:40.400 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:40.369+0000 7f8a077d1700 1 -- 172.21.15.42:0/4026887048 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8a0007e720 msgr2=0x7f8a0007eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:40.400 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:40.369+0000 7f8a077d1700 1 --2- 172.21.15.42:0/4026887048 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8a0007e720 0x7f8a0007eb50 secure :-1 s=READY pgs=3160 cs=0 l=1 rx=0x7f89f0002f30 tx=0x7f89f000fd30).stop 2019-12-05T06:44:40.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:40.369+0000 7f8a077d1700 1 -- 172.21.15.42:0/4026887048 shutdown_connections 2019-12-05T06:44:40.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:40.369+0000 7f8a077d1700 1 --2- 172.21.15.42:0/4026887048 >> v2:172.21.15.40:3300/0 conn(0x7f8a000a7740 0x7f8a0008df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:44:40.417 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:40.369+0000 7f8a077d1700 1 --2- 172.21.15.42:0/4026887048 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8a0007e720 0x7f8a0007eb50 unknown :-1 s=CLOSED pgs=3160 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:40.418 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:40.369+0000 7f8a077d1700 1 -- 172.21.15.42:0/4026887048 shutdown_connections 2019-12-05T06:44:40.418 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:40.369+0000 7f8a077d1700 1 -- 172.21.15.42:0/4026887048 wait complete. 2019-12-05T06:44:40.418 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:40.369+0000 7f8a077d1700 1 -- 172.21.15.42:0/4026887048 >> 172.21.15.42:0/4026887048 conn(0x7f8a000aabe0 msgr2=0x7f8a000a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:44:40.419 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:40.373+0000 7f8a077d1700 1 Processor -- start 2019-12-05T06:44:40.419 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:40.373+0000 7f8a077d1700 1 -- start start 2019-12-05T06:44:40.419 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:40.373+0000 7f8a077d1700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8a000a5ff0 0x7f8a000a6420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:44:40.420 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:40.373+0000 7f8a077d1700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8a00132980 0x7f8a00132db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:44:40.420 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:40.373+0000 7f8a077d1700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a001161c0 con 0x7f8a00133300 2019-12-05T06:44:40.420 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:40.373+0000 7f8a077d1700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a000a5810 con 0x7f8a000a5ff0 2019-12-05T06:44:40.421 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:40.373+0000 7f8a077d1700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a000a5ab0 con 0x7f8a00132980 2019-12-05T06:44:40.421 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:40.373+0000 7f8a05d6e700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8a000a5ff0 0x7f8a000a6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:40.421 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:40.373+0000 7f8a0556d700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8a00132980 0x7f8a00132db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:40.422 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:40.373+0000 7f8a05d6e700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8a000a5ff0 0x7f8a000a6420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.42:50960/0 (socket says 172.21.15.42:50960) 2019-12-05T06:44:40.422 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:40.373+0000 7f8a05d6e700 1 -- 172.21.15.42:0/2319776846 learned_addr learned my addr 172.21.15.42:0/2319776846 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:44:40.422 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:40.373+0000 7f8a05d6e700 1 -- 172.21.15.42:0/2319776846 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8a00132980 msgr2=0x7f8a00132db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:40.423 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:40.373+0000 7f89f67fc700 1 -- 172.21.15.42:0/2319776846 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2168637403 0 0) 0x7f8a001161c0 con 0x7f8a00133300 2019-12-05T06:44:40.423 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:40.373+0000 7f8a05d6e700 1 --2- 172.21.15.42:0/2319776846 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8a00132980 0x7f8a00132db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:40.423 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:40.373+0000 7f8a05d6e700 1 -- 172.21.15.42:0/2319776846 >> v1:172.21.15.40:6789/0 conn(0x7f8a00133300 legacy=0x7f8a00128b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:40.424 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:40.373+0000 7f8a05d6e700 1 -- 172.21.15.42:0/2319776846 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a00129390 con 0x7f8a000a5ff0 2019-12-05T06:44:40.424 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:40.373+0000 7f8a05d6e700 1 --2- 172.21.15.42:0/2319776846 >> v2:172.21.15.40:3300/0 conn(0x7f8a000a5ff0 0x7f8a000a6420 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f8a000a59d0 tx=0x7f89fc010b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:40.424 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:40.373+0000 7f8a05d6e700 1 --2- 172.21.15.42:0/2319776846 >> v2:172.21.15.40:3300/0 conn(0x7f8a000a5ff0 0x7f8a000a6420 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f8a000a59d0 tx=0x7f89fc010b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:40.425 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:40.373+0000 7f8a05d6e700 1 --2- 172.21.15.42:0/2319776846 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8a000a5ff0 0x7f8a000a6420 secure :-1 s=READY pgs=2387 cs=0 l=1 rx=0x7f8a000a59d0 tx=0x7f89fc010b00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:44:40.427 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:40.373+0000 7f8a05d6e700 1 --2- 172.21.15.42:0/2319776846 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8a000a5ff0 0x7f8a000a6420 secure :-1 s=THROTTLE_DONE pgs=2387 cs=0 l=1 rx=0x7f8a000a59d0 tx=0x7f89fc010b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:40.429 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:40.373+0000 7f89f67fc700 1 -- 172.21.15.42:0/2319776846 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f89fc012570 con 0x7f8a000a5ff0 2019-12-05T06:44:40.429 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:40.373+0000 7f8a05d6e700 1 --2- 172.21.15.42:0/2319776846 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8a000a5ff0 0x7f8a000a6420 secure :-1 s=THROTTLE_DONE pgs=2387 cs=0 l=1 rx=0x7f8a000a59d0 tx=0x7f89fc010b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:40.429 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:40.373+0000 7f89f67fc700 1 -- 172.21.15.42:0/2319776846 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89fc007150 con 0x7f8a000a5ff0 2019-12-05T06:44:40.429 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:40.373+0000 7f8a05d6e700 1 --2- 172.21.15.42:0/2319776846 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8a000a5ff0 0x7f8a000a6420 secure :-1 s=THROTTLE_DONE pgs=2387 cs=0 l=1 rx=0x7f8a000a59d0 tx=0x7f89fc010b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:40.430 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:40.373+0000 7f8a077d1700 1 -- 172.21.15.42:0/2319776846 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a001295a0 con 0x7f8a000a5ff0 2019-12-05T06:44:40.430 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:40.373+0000 7f89f67fc700 1 -- 172.21.15.42:0/2319776846 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f89fc012b50 con 0x7f8a000a5ff0 2019-12-05T06:44:40.431 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:40.373+0000 7f8a077d1700 1 -- 172.21.15.42:0/2319776846 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a000752e0 con 0x7f8a000a5ff0 2019-12-05T06:44:40.431 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:40.377+0000 7f8a05d6e700 1 --2- 172.21.15.42:0/2319776846 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8a000a5ff0 0x7f8a000a6420 secure :-1 s=THROTTLE_DONE pgs=2387 cs=0 l=1 rx=0x7f8a000a59d0 tx=0x7f89fc010b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:40.431 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:40.377+0000 7f89f67fc700 1 -- 172.21.15.42:0/2319776846 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f89fc005540 con 0x7f8a000a5ff0 2019-12-05T06:44:40.431 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:40.377+0000 7f8a05d6e700 1 --2- 172.21.15.42:0/2319776846 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8a000a5ff0 0x7f8a000a6420 secure :-1 s=THROTTLE_DONE pgs=2387 cs=0 l=1 rx=0x7f8a000a59d0 tx=0x7f89fc010b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:40.432 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:40.377+0000 7f89f67fc700 1 --2- 172.21.15.42:0/2319776846 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f89e803ac30 0x7f89e803d140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:44:40.432 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:40.377+0000 7f89f67fc700 1 -- 172.21.15.42:0/2319776846 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 11965+0+0 (secure 0 0 0) 0x7f89fc04b0b0 con 0x7f8a000a5ff0 2019-12-05T06:44:40.432 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:40.377+0000 7f8a0556d700 1 --2- 172.21.15.42:0/2319776846 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f89e803ac30 0x7f89e803d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:40.433 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:40.377+0000 7f8a0556d700 1 --2- 172.21.15.42:0/2319776846 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f89e803ac30 0x7f89e803d140 crc :-1 s=READY pgs=1321 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:44:40.433 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:40.385+0000 7f8a077d1700 1 -- 172.21.15.42:0/2319776846 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a000a24b0 con 0x7f8a000a5ff0 2019-12-05T06:44:40.434 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:40.393+0000 7f8a05d6e700 1 --2- 172.21.15.42:0/2319776846 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8a000a5ff0 0x7f8a000a6420 secure :-1 s=THROTTLE_DONE pgs=2387 cs=0 l=1 rx=0x7f8a000a59d0 tx=0x7f89fc010b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:40.434 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:40.393+0000 7f89f67fc700 1 -- 172.21.15.42:0/2319776846 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f89fc0320b0 con 0x7f8a000a5ff0 2019-12-05T06:44:40.611 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:40.605+0000 7f8a077d1700 1 -- 172.21.15.42:0/2319776846 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8a0005d110 con 0x7f8a000a5ff0 2019-12-05T06:44:40.613 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:40.605+0000 7f8a05d6e700 1 --2- 172.21.15.42:0/2319776846 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8a000a5ff0 0x7f8a000a6420 secure :-1 s=THROTTLE_DONE pgs=2387 cs=0 l=1 rx=0x7f8a000a59d0 tx=0x7f89fc010b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:40.613 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:40.605+0000 7f89f67fc700 1 -- 172.21.15.42:0/2319776846 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 383 v383) v1 ==== 95+0+2117 (secure 0 0 0) 0x7f89fc032310 con 0x7f8a000a5ff0 2019-12-05T06:44:40.614 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 383 2019-12-05T06:44:40.615 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:44:40.615 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":383,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":12429,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/4178556019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":4178556019},{"type":"v1","addr":"172.21.15.40:6833","nonce":4178556019}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":383}],"filesystems":[{"mdsmap":{"epoch":382,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:44:34.328051+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":118,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12303},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12303":{"gid":12303,"name":"a","rank":0,"incarnation":378,"state":"up:active","state_seq":17,"addr":"172.21.15.42:6835/2489540211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2489540211},{"type":"v1","addr":"172.21.15.42:6835","nonce":2489540211}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:44:40.616 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:40.609+0000 7f8a077d1700 1 -- 172.21.15.42:0/2319776846 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f89e803ac30 msgr2=0x7f89e803d140 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:40.617 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:40.609+0000 7f8a077d1700 1 --2- 172.21.15.42:0/2319776846 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f89e803ac30 0x7f89e803d140 crc :-1 s=READY pgs=1321 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:40.617 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:40.609+0000 7f8a077d1700 1 -- 172.21.15.42:0/2319776846 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8a000a5ff0 msgr2=0x7f8a000a6420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:40.617 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:40.609+0000 7f8a077d1700 1 --2- 172.21.15.42:0/2319776846 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8a000a5ff0 0x7f8a000a6420 secure :-1 s=READY pgs=2387 cs=0 l=1 rx=0x7f8a000a59d0 tx=0x7f89fc010b00).stop 2019-12-05T06:44:40.617 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:40.609+0000 7f8a077d1700 1 -- 172.21.15.42:0/2319776846 shutdown_connections 2019-12-05T06:44:40.618 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:40.609+0000 7f8a077d1700 1 --2- 172.21.15.42:0/2319776846 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f89e803ac30 0x7f89e803d140 unknown :-1 s=CLOSED pgs=1321 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:40.619 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:40.609+0000 7f8a077d1700 1 --2- 172.21.15.42:0/2319776846 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8a000a5ff0 0x7f8a000a6420 unknown :-1 s=CLOSED pgs=2387 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:40.619 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:40.609+0000 7f8a077d1700 1 --2- 172.21.15.42:0/2319776846 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8a00132980 0x7f8a00132db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:40.619 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:40.609+0000 7f8a077d1700 1 -- 172.21.15.42:0/2319776846 shutdown_connections 2019-12-05T06:44:40.619 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:40.609+0000 7f8a077d1700 1 -- 172.21.15.42:0/2319776846 wait complete. 2019-12-05T06:44:40.620 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:44:40.609+0000 7f8a077d1700 1 -- 172.21.15.42:0/2319776846 >> 172.21.15.42:0/2319776846 conn(0x7f8a0007dee0 msgr2=0x7f8a0009f620 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:44:40.624 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b reported in up:standby state 2019-12-05T06:44:40.624 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 58 secs before thrashing 2019-12-05T06:44:46.370 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:44:46.374 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:45:03.936 INFO:tasks.workunit.client.0.smithi042.stderr:Checking out files: 0% (25/9895) Checking out files: 0% (44/9895) Checking out files: 1% (99/9895) Checking out files: 1% (141/9895) Checking out files: 2% (198/9895) Checking out files: 2% (208/9895) Checking out files: 2% (284/9895) Checking out files: 3% (297/9895) Checking out files: 3% (326/9895) Checking out files: 3% (357/9895) Checking out files: 4% (396/9895) Checking out files: 4% (406/9895) Checking out files: 5% (495/9895) Checking out files: 5% (497/9895) Checking out files: 5% (566/9895) Checking out files: 6% (594/9895) Checking out files: 6% (596/9895) Checking out files: 6% (604/9895) Checking out files: 6% (686/9895) Checking out files: 7% (693/9895) Checking out files: 7% (783/9895) Checking out files: 8% (792/9895) Checking out files: 8% (845/9895) Checking out files: 8% (890/9895) Checking out files: 9% (891/9895) Checking out files: 9% (896/9895) Checking out files: 9% (958/9895) Checking out files: 10% (990/9895) Checking out files: 10% (1003/9895) Checking out files: 10% (1040/9895) Checking out files: 11% (1089/9895) Checking out files: 11% (1143/9895) Checking out files: 11% (1158/9895) Checking out files: 12% (1188/9895) Checking out files: 12% (1194/9895) Checking out files: 12% (1247/9895) Checking out files: 13% (1287/9895) Checking out files: 13% (1328/9895) Checking out files: 13% (1352/9895) Checking out files: 13% (1375/9895) Checking out files: 14% (1386/9895) Checking out files: 14% (1430/9895) Checking out files: 14% (1451/9895) Checking out files: 14% (1460/9895) Checking out files: 15% (1485/9895) Checking out files: 15% (1528/9895) Checking out files: 16% (1584/9895) Checking out files: 16% (1590/9895) Checking out files: 16% (1627/9895) Checking out files: 16% (1671/9895) Checking out files: 17% (1683/9895) Checking out files: 17% (1734/9895) Checking out files: 18% (1782/9895) Checking out files: 18% (1788/9895) Checking out files: 18% (1828/9895) Checking out files: 18% (1844/9895) Checking out files: 18% (1851/9895) Checking out files: 19% (1881/9895) Checking out files: 19% (1927/9895) Checking out files: 19% (1949/9895) Checking out files: 20% (1979/9895) Checking out files: 20% (1991/9895) Checking out files: 20% (2045/9895) Checking out files: 20% (2047/9895) Checking out files: 21% (2078/9895) Checking out files: 21% (2133/9895) Checking out files: 22% (2177/9895) Checking out files: 22% (2225/9895) Checking out files: 22% (2248/9895) Checking out files: 22% (2267/9895) Checking out files: 23% (2276/9895) Checking out files: 23% (2346/9895) Checking out files: 23% (2367/9895) Checking out files: 24% (2375/9895) Checking out files: 24% (2402/9895) Checking out files: 24% (2451/9895) Checking out files: 25% (2474/9895) Checking out files: 25% (2511/9895) Checking out files: 25% (2548/9895) Checking out files: 26% (2573/9895) Checking out files: 26% (2608/9895) Checking out files: 26% (2630/9895) Checking out files: 27% (2672/9895) Checking out files: 27% (2683/9895) Checking out files: 27% (2729/9895) Checking out files: 27% (2736/9895) Checking out files: 28% (2771/9895) Checking out files: 28% (2805/9895) Checking out files: 28% (2822/9895) Checking out files: 29% (2870/9895) Checking out files: 29% (2879/9895) Checking out files: 29% (2929/9895) Checking out files: 29% (2965/9895) Checking out files: 30% (2969/9895) Checking out files: 30% (3007/9895) Checking out files: 30% (3049/9895) Checking out files: 31% (3068/9895) Checking out files: 31% (3076/9895) Checking out files: 32% (3167/9895) Checking out files: 32% (3176/9895) Checking out files: 32% (3238/9895) Checking out files: 33% (3266/9895) Checking out files: 33% (3308/9895) Checking out files: 33% (3346/9895) Checking out files: 34% (3365/9895) Checking out files: 34% (3408/9895) Checking out files: 35% (3464/9895) Checking out files: 35% (3502/9895) Checking out files: 35% (3540/9895) Checking out files: 36% (3563/9895) Checking out files: 36% (3577/9895) Checking out files: 36% (3622/9895) Checking out files: 37% (3662/9895) Checking out files: 37% (3757/9895) Checking out files: 38% (3761/9895) Checking out files: 38% (3765/9895) Checking out files: 39% (3860/9895) Checking out files: 39% (3863/9895) Checking out files: 39% (3909/9895) Checking out files: 39% (3954/9895) Checking out files: 40% (3958/9895) Checking out files: 40% (3990/9895) Checking out files: 40% (4011/9895) Checking out files: 41% (4057/9895) Checking out files: 41% (4067/9895) Checking out files: 41% (4086/9895) Checking out files: 41% (4094/9895) Checking out files: 41% (4120/9895) Checking out files: 41% (4126/9895) Checking out files: 42% (4156/9895) Checking out files: 42% (4219/9895) Checking out files: 43% (4255/9895) Checking out files: 44% (4354/9895) Checking out files: 44% (4398/9895) Checking out files: 45% (4453/9895) Checking out files: 45% (4525/9895) Checking out files: 46% (4552/9895) Checking out files: 46% (4584/9895) Checking out files: 46% (4631/9895) Checking out files: 46% (4632/9895) Checking out files: 47% (4651/9895) Checking out files: 47% (4747/9895) Checking out files: 48% (4750/9895) Checking out files: 48% (4794/9895) Checking out files: 49% (4849/9895) Checking out files: 49% (4868/9895) Checking out files: 49% (4910/9895) Checking out files: 50% (4948/9895) Checking out files: 50% (4998/9895) Checking out files: 50% (5016/9895) Checking out files: 50% (5044/9895) Checking out files: 51% (5047/9895) Checking out files: 51% (5080/9895) Checking out files: 51% (5113/9895) Checking out files: 51% (5143/9895) Checking out files: 52% (5146/9895) Checking out files: 52% (5167/9895) Checking out files: 52% (5192/9895) Checking out files: 53% (5245/9895) Checking out files: 54% (5344/9895) Checking out files: 54% (5382/9895) Checking out files: 55% (5443/9895) Checking out files: 55% (5480/9895) Checking out files: 55% (5494/9895) Checking out files: 56% (5542/9895) Checking out files: 56% (5553/9895) Checking out files: 56% (5606/9895) Checking out files: 56% (5622/9895) Checking out files: 57% (5641/9895) Checking out files: 58% (5740/9895) Checking out files: 58% (5788/9895) Checking out files: 58% (5821/9895) Checking out files: 59% (5839/9895) Checking out files: 59% (5868/9895) Checking out files: 59% (5929/9895) Checking out files: 60% (5937/9895) Checking out files: 60% (5944/9895) Checking out files: 60% (5955/9895) Checking out files: 60% (5975/9895) Checking out files: 61% (6036/9895) Checking out files: 61% (6055/9895) Checking out files: 61% (6068/9895) Checking out files: 61% (6083/9895) Checking out files: 61% (6112/9895) Checking out files: 62% (6135/9895) Checking out files: 62% (6140/9895) Checking out files: 62% (6178/9895) Checking out files: 62% (6199/9895) Checking out files: 63% (6234/9895) Checking out files: 63% (6271/9895) Checking out files: 64% (6333/9895) Checking out files: 64% (6338/9895) Checking out files: 64% (6366/9895) Checking out files: 64% (6386/9895) Checking out files: 64% (6406/9895) Checking out files: 65% (6432/9895) Checking out files: 65% (6438/9895) Checking out files: 66% (6531/9895) Checking out files: 66% (6532/9895) Checking out files: 66% (6583/9895) Checking out files: 67% (6630/9895) Checking out files: 67% (6666/9895) Checking out files: 67% (6675/9895) Checking out files: 68% (6729/9895) Checking out files: 68% (6750/9895) Checking out files: 68% (6804/9895) Checking out files: 69% (6828/9895) Checking out files: 69% (6878/9895) Checking out files: 69% (6917/9895) Checking out files: 70% (6927/9895) Checking out files: 70% (6995/9895) Checking out files: 70% (7020/9895) Checking out files: 71% (7026/9895) Checking out files: 71% (7074/9895) Checking out files: 72% (7125/9895) Checking out files: 72% (7130/9895) Checking out files: 72% (7194/9895) Checking out files: 73% (7224/9895) Checking out files: 73% (7257/9895) Checking out files: 73% (7318/9895) Checking out files: 74% (7323/9895) Checking out files: 75% (7422/9895) Checking out files: 75% (7423/9895) Checking out files: 76% (7521/9895) Checking out files: 76% (7530/9895) Checking out files: 76% (7565/9895) Checking out files: 76% (7615/9895) Checking out files: 77% (7620/9895) Checking out files: 78% (7719/9895) Checking out files: 78% (7720/9895) Checking out files: 78% (7754/9895) Checking out files: 78% (7805/9895) Checking out files: 79% (7818/9895) Checking out files: 79% (7836/9895) Checking out files: 79% (7895/9895) Checking out files: 80% (7916/9895) Checking out files: 80% (7917/9895) Checking out files: 80% (7957/9895) Checking out files: 80% (7994/9895) Checking out files: 81% (8015/9895) Checking out files: 81% (8059/9895) Checking out files: 82% (8114/9895) Checking out files: 82% (8148/9895) Checking out files: 82% (8181/9895) Checking out files: 83% (8213/9895) Checking out files: 83% (8261/9895) Checking out files: 84% (8312/9895) Checking out files: 84% (8330/9895) Checking out files: 84% (8334/9895) Checking out files: 85% (8411/9895) Checking out files: 85% (8479/9895) Checking out files: 86% (8510/9895) Checking out files: 86% (8557/9895) Checking out files: 86% (8598/9895) Checking out files: 87% (8609/9895) Checking out files: 87% (8668/9895) Checking out files: 88% (8708/9895) Checking out files: 88% (8723/9895) Checking out files: 88% (8777/9895) Checking out files: 89% (8807/9895) Checking out files: 89% (8851/9895) Checking out files: 90% (8906/9895) Checking out files: 90% (8909/9895) Checking out files: 91% (9005/9895) Checking out files: 91% (9067/9895) Checking out files: 92% (9104/9895) Checking out files: 93% (9203/9895) Checking out files: 93% (9231/9895) Checking out files: 93% (9298/9895) Checking out files: 94% (9302/9895) Checking out files: 94% (9366/9895) Checking out files: 95% (9401/9895) Checking out files: 95% (9455/9895) Checking out files: 95% (9458/9895) Checking out files: 96% (9500/9895) Checking out files: 97% (9599/9895) Checking out files: 97% (9601/9895) Checking out files: 97% (9688/9895) Checking out files: 98% (9698/9895) Checking out files: 98% (9766/9895) Checking out files: 98% (9779/9895) Checking out files: 98% (9788/9895) Checking out files: 99% (9797/9895) Checking out files: 99% (9823/9895) Checking out files: 99% (9857/9895) Checking out files: 100% (9895/9895) Checking out files: 100% (9895/9895), done. 2019-12-05T06:45:05.147 INFO:tasks.workunit.client.0.smithi042.stdout:HEAD is now at 204c175eea msg: blobhash on entity_name_t causes strange badness... should look into that, but avoid for now 2019-12-05T06:45:05.157 INFO:tasks.workunit.client.0.smithi042.stderr:+ mkdir .snap/v0.1 2019-12-05T06:45:05.229 INFO:tasks.workunit.client.0.smithi042.stderr:+ ver=v0.2 2019-12-05T06:45:05.229 INFO:tasks.workunit.client.0.smithi042.stderr:+ echo v0.2 2019-12-05T06:45:05.229 INFO:tasks.workunit.client.0.smithi042.stderr:+ git reset --hard v0.2 2019-12-05T06:45:05.229 INFO:tasks.workunit.client.0.smithi042.stdout:v0.2 2019-12-05T06:45:09.635 INFO:tasks.workunit.client.0.smithi042.stderr:Checking out files: 62% (138/222) Checking out files: 63% (140/222) Checking out files: 64% (143/222) Checking out files: 65% (145/222) Checking out files: 66% (147/222) Checking out files: 67% (149/222) Checking out files: 68% (151/222) Checking out files: 69% (154/222) Checking out files: 70% (156/222) Checking out files: 71% (158/222) Checking out files: 72% (160/222) Checking out files: 73% (163/222) Checking out files: 74% (165/222) Checking out files: 75% (167/222) Checking out files: 76% (169/222) Checking out files: 77% (171/222) Checking out files: 78% (174/222) Checking out files: 79% (176/222) Checking out files: 80% (178/222) Checking out files: 81% (180/222) Checking out files: 82% (183/222) Checking out files: 83% (185/222) Checking out files: 84% (187/222) Checking out files: 85% (189/222) Checking out files: 86% (191/222) Checking out files: 87% (194/222) Checking out files: 88% (196/222) Checking out files: 89% (198/222) Checking out files: 90% (200/222) Checking out files: 91% (203/222) Checking out files: 92% (205/222) Checking out files: 93% (207/222) Checking out files: 94% (209/222) Checking out files: 95% (211/222) Checking out files: 96% (214/222) Checking out files: 97% (216/222) Checking out files: 98% (218/222) Checking out files: 99% (220/222) Checking out files: 100% (222/222) Checking out files: 100% (222/222), done. 2019-12-05T06:45:09.896 INFO:tasks.workunit.client.0.smithi042.stdout:HEAD is now at 2b5f76f48e readme 2019-12-05T06:45:09.901 INFO:tasks.workunit.client.0.smithi042.stderr:+ mkdir .snap/v0.2 2019-12-05T06:45:10.577 INFO:tasks.workunit.client.0.smithi042.stderr:+ ver=v0.3 2019-12-05T06:45:10.577 INFO:tasks.workunit.client.0.smithi042.stderr:+ echo v0.3 2019-12-05T06:45:10.577 INFO:tasks.workunit.client.0.smithi042.stderr:+ git reset --hard v0.3 2019-12-05T06:45:10.578 INFO:tasks.workunit.client.0.smithi042.stdout:v0.3 2019-12-05T06:45:15.706 INFO:tasks.workunit.client.0.smithi042.stderr:Checking out files: 5% (17/295) Checking out files: 6% (18/295) Checking out files: 7% (21/295) Checking out files: 8% (24/295) Checking out files: 9% (27/295) Checking out files: 10% (30/295) Checking out files: 11% (33/295) Checking out files: 12% (36/295) Checking out files: 13% (39/295) Checking out files: 14% (42/295) Checking out files: 15% (45/295) Checking out files: 16% (48/295) Checking out files: 17% (51/295) Checking out files: 18% (54/295) Checking out files: 19% (57/295) Checking out files: 20% (59/295) Checking out files: 21% (62/295) Checking out files: 22% (65/295) Checking out files: 23% (68/295) Checking out files: 24% (71/295) Checking out files: 25% (74/295) Checking out files: 26% (77/295) Checking out files: 27% (80/295) Checking out files: 28% (83/295) Checking out files: 29% (86/295) Checking out files: 30% (89/295) Checking out files: 31% (92/295) Checking out files: 32% (95/295) Checking out files: 33% (98/295) Checking out files: 34% (101/295) Checking out files: 35% (104/295) Checking out files: 36% (107/295) Checking out files: 37% (110/295) Checking out files: 38% (113/295) Checking out files: 39% (116/295) Checking out files: 40% (118/295) Checking out files: 41% (121/295) Checking out files: 42% (124/295) Checking out files: 43% (127/295) Checking out files: 44% (130/295) Checking out files: 45% (133/295) Checking out files: 46% (136/295) Checking out files: 47% (139/295) Checking out files: 48% (142/295) Checking out files: 49% (145/295) Checking out files: 50% (148/295) Checking out files: 51% (151/295) Checking out files: 52% (154/295) Checking out files: 53% (157/295) Checking out files: 54% (160/295) Checking out files: 55% (163/295) Checking out files: 56% (166/295) Checking out files: 57% (169/295) Checking out files: 58% (172/295) Checking out files: 59% (175/295) Checking out files: 60% (177/295) Checking out files: 61% (180/295) Checking out files: 62% (183/295) Checking out files: 63% (186/295) Checking out files: 64% (189/295) Checking out files: 65% (192/295) Checking out files: 66% (195/295) Checking out files: 67% (198/295) Checking out files: 68% (201/295) Checking out files: 69% (204/295) Checking out files: 70% (207/295) Checking out files: 71% (210/295) Checking out files: 72% (213/295) Checking out files: 73% (216/295) Checking out files: 74% (219/295) Checking out files: 75% (222/295) Checking out files: 76% (225/295) Checking out files: 77% (228/295) Checking out files: 78% (231/295) Checking out files: 79% (234/295) Checking out files: 80% (236/295) Checking out files: 81% (239/295) Checking out files: 82% (242/295) Checking out files: 83% (245/295) Checking out files: 84% (248/295) Checking out files: 85% (251/295) Checking out files: 86% (254/295) Checking out files: 87% (257/295) Checking out files: 88% (260/295) Checking out files: 89% (263/295) Checking out files: 90% (266/295) Checking out files: 91% (269/295) Checking out files: 92% (272/295) Checking out files: 93% (275/295) Checking out files: 94% (278/295) Checking out files: 95% (281/295) Checking out files: 96% (284/295) Checking out files: 96% (286/295) Checking out files: 97% (287/295) Checking out files: 98% (290/295) Checking out files: 99% (293/295) Checking out files: 100% (295/295) Checking out files: 100% (295/295), done. 2019-12-05T06:45:16.180 INFO:tasks.workunit.client.0.smithi042.stdout:HEAD is now at 2e986fe9dd include ceph.spec in configure.ac 2019-12-05T06:45:16.180 INFO:tasks.workunit.client.0.smithi042.stderr:+ mkdir .snap/v0.3 2019-12-05T06:45:16.235 INFO:tasks.workunit.client.0.smithi042.stderr:+ ver=v0.4 2019-12-05T06:45:16.236 INFO:tasks.workunit.client.0.smithi042.stderr:+ echo v0.4 2019-12-05T06:45:16.236 INFO:tasks.workunit.client.0.smithi042.stderr:+ git reset --hard v0.4 2019-12-05T06:45:16.236 INFO:tasks.workunit.client.0.smithi042.stdout:v0.4 2019-12-05T06:45:16.440 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:45:16.445 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:45:20.066 INFO:tasks.workunit.client.0.smithi042.stderr:Checking out files: 27% (69/247) Checking out files: 28% (70/247) Checking out files: 29% (72/247) Checking out files: 30% (75/247) Checking out files: 31% (77/247) Checking out files: 32% (80/247) Checking out files: 33% (82/247) Checking out files: 34% (84/247) Checking out files: 35% (87/247) Checking out files: 36% (89/247) Checking out files: 37% (92/247) Checking out files: 38% (94/247) Checking out files: 39% (97/247) Checking out files: 40% (99/247) Checking out files: 41% (102/247) Checking out files: 42% (104/247) Checking out files: 43% (107/247) Checking out files: 44% (109/247) Checking out files: 45% (112/247) Checking out files: 46% (114/247) Checking out files: 47% (117/247) Checking out files: 48% (119/247) Checking out files: 49% (122/247) Checking out files: 50% (124/247) Checking out files: 51% (126/247) Checking out files: 52% (129/247) Checking out files: 53% (131/247) Checking out files: 54% (134/247) Checking out files: 55% (136/247) Checking out files: 56% (139/247) Checking out files: 57% (141/247) Checking out files: 58% (144/247) Checking out files: 59% (146/247) Checking out files: 60% (149/247) Checking out files: 61% (151/247) Checking out files: 62% (154/247) Checking out files: 63% (156/247) Checking out files: 64% (159/247) Checking out files: 65% (161/247) Checking out files: 66% (164/247) Checking out files: 67% (166/247) Checking out files: 68% (168/247) Checking out files: 69% (171/247) Checking out files: 70% (173/247) Checking out files: 71% (176/247) Checking out files: 72% (178/247) Checking out files: 73% (181/247) Checking out files: 74% (183/247) Checking out files: 75% (186/247) Checking out files: 76% (188/247) Checking out files: 77% (191/247) Checking out files: 78% (193/247) Checking out files: 79% (196/247) Checking out files: 80% (198/247) Checking out files: 81% (201/247) Checking out files: 82% (203/247) Checking out files: 83% (206/247) Checking out files: 84% (208/247) Checking out files: 85% (210/247) Checking out files: 86% (213/247) Checking out files: 87% (215/247) Checking out files: 88% (218/247) Checking out files: 89% (220/247) Checking out files: 90% (223/247) Checking out files: 91% (225/247) Checking out files: 92% (228/247) Checking out files: 93% (230/247) Checking out files: 94% (233/247) Checking out files: 95% (235/247) Checking out files: 96% (238/247) Checking out files: 97% (240/247) Checking out files: 98% (243/247) Checking out files: 99% (245/247) Checking out files: 100% (247/247) Checking out files: 100% (247/247), done. 2019-12-05T06:45:20.447 INFO:tasks.workunit.client.0.smithi042.stdout:HEAD is now at 13b1bf7c49 kclient: initializing kaddr (merge fix) 2019-12-05T06:45:20.448 INFO:tasks.workunit.client.0.smithi042.stderr:+ mkdir .snap/v0.4 2019-12-05T06:45:20.552 INFO:tasks.workunit.client.0.smithi042.stderr:+ ver=v0.5 2019-12-05T06:45:20.552 INFO:tasks.workunit.client.0.smithi042.stderr:+ echo v0.5 2019-12-05T06:45:20.552 INFO:tasks.workunit.client.0.smithi042.stderr:+ git reset --hard v0.5 2019-12-05T06:45:20.552 INFO:tasks.workunit.client.0.smithi042.stdout:v0.5 2019-12-05T06:45:26.751 INFO:tasks.workunit.client.0.smithi042.stderr:Checking out files: 7% (19/245) Checking out files: 8% (20/245) Checking out files: 9% (23/245) Checking out files: 10% (25/245) Checking out files: 11% (27/245) Checking out files: 12% (30/245) Checking out files: 13% (32/245) Checking out files: 14% (35/245) Checking out files: 15% (37/245) Checking out files: 16% (40/245) Checking out files: 17% (42/245) Checking out files: 18% (45/245) Checking out files: 19% (47/245) Checking out files: 20% (49/245) Checking out files: 21% (52/245) Checking out files: 22% (54/245) Checking out files: 23% (57/245) Checking out files: 24% (59/245) Checking out files: 25% (62/245) Checking out files: 26% (64/245) Checking out files: 27% (67/245) Checking out files: 27% (68/245) Checking out files: 28% (69/245) Checking out files: 29% (72/245) Checking out files: 30% (74/245) Checking out files: 31% (76/245) Checking out files: 32% (79/245) Checking out files: 33% (81/245) Checking out files: 34% (84/245) Checking out files: 35% (86/245) Checking out files: 36% (89/245) Checking out files: 37% (91/245) Checking out files: 38% (94/245) Checking out files: 39% (96/245) Checking out files: 40% (98/245) Checking out files: 41% (101/245) Checking out files: 42% (103/245) Checking out files: 43% (106/245) Checking out files: 44% (108/245) Checking out files: 45% (111/245) Checking out files: 46% (113/245) Checking out files: 47% (116/245) Checking out files: 48% (118/245) Checking out files: 49% (121/245) Checking out files: 50% (123/245) Checking out files: 51% (125/245) Checking out files: 52% (128/245) Checking out files: 53% (130/245) Checking out files: 54% (133/245) Checking out files: 55% (135/245) Checking out files: 56% (138/245) Checking out files: 57% (140/245) Checking out files: 58% (143/245) Checking out files: 59% (145/245) Checking out files: 60% (147/245) Checking out files: 61% (150/245) Checking out files: 62% (152/245) Checking out files: 63% (155/245) Checking out files: 64% (157/245) Checking out files: 65% (160/245) Checking out files: 66% (162/245) Checking out files: 67% (165/245) Checking out files: 68% (167/245) Checking out files: 69% (170/245) Checking out files: 70% (172/245) Checking out files: 71% (174/245) Checking out files: 72% (177/245) Checking out files: 73% (179/245) Checking out files: 74% (182/245) Checking out files: 75% (184/245) Checking out files: 76% (187/245) Checking out files: 77% (189/245) Checking out files: 78% (192/245) Checking out files: 79% (194/245) Checking out files: 80% (196/245) Checking out files: 81% (199/245) Checking out files: 82% (201/245) Checking out files: 83% (204/245) Checking out files: 84% (206/245) Checking out files: 85% (209/245) Checking out files: 86% (211/245) Checking out files: 87% (214/245) Checking out files: 88% (216/245) Checking out files: 89% (219/245) Checking out files: 90% (221/245) Checking out files: 90% (222/245) Checking out files: 91% (223/245) Checking out files: 92% (226/245) Checking out files: 93% (228/245) Checking out files: 94% (231/245) Checking out files: 95% (233/245) Checking out files: 96% (236/245) Checking out files: 97% (238/245) Checking out files: 98% (241/245) Checking out files: 99% (243/245) Checking out files: 100% (245/245) Checking out files: 100% (245/245), done. 2019-12-05T06:45:26.858 INFO:tasks.workunit.client.0.smithi042.stdout:HEAD is now at 1dd420957f journal: detect size of raw block devices properly 2019-12-05T06:45:26.862 INFO:tasks.workunit.client.0.smithi042.stderr:+ mkdir .snap/v0.5 2019-12-05T06:45:27.810 INFO:tasks.workunit.client.0.smithi042.stderr:+ ver=v0.6 2019-12-05T06:45:27.810 INFO:tasks.workunit.client.0.smithi042.stderr:+ echo v0.6 2019-12-05T06:45:27.811 INFO:tasks.workunit.client.0.smithi042.stderr:+ git reset --hard v0.6 2019-12-05T06:45:27.811 INFO:tasks.workunit.client.0.smithi042.stdout:v0.6 2019-12-05T06:45:31.171 INFO:tasks.workunit.client.0.smithi042.stderr:Checking out files: 55% (125/227) Checking out files: 56% (128/227) Checking out files: 57% (130/227) Checking out files: 58% (132/227) Checking out files: 59% (134/227) Checking out files: 60% (137/227) Checking out files: 61% (139/227) Checking out files: 62% (141/227) Checking out files: 63% (144/227) Checking out files: 64% (146/227) Checking out files: 65% (148/227) Checking out files: 66% (150/227) Checking out files: 67% (153/227) Checking out files: 68% (155/227) Checking out files: 69% (157/227) Checking out files: 70% (159/227) Checking out files: 71% (162/227) Checking out files: 72% (164/227) Checking out files: 72% (165/227) Checking out files: 73% (166/227) Checking out files: 74% (168/227) Checking out files: 75% (171/227) Checking out files: 76% (173/227) Checking out files: 77% (175/227) Checking out files: 78% (178/227) Checking out files: 79% (180/227) Checking out files: 80% (182/227) Checking out files: 81% (184/227) Checking out files: 82% (187/227) Checking out files: 83% (189/227) Checking out files: 84% (191/227) Checking out files: 85% (193/227) Checking out files: 86% (196/227) Checking out files: 87% (198/227) Checking out files: 88% (200/227) Checking out files: 89% (203/227) Checking out files: 90% (205/227) Checking out files: 91% (207/227) Checking out files: 92% (209/227) Checking out files: 93% (212/227) Checking out files: 94% (214/227) Checking out files: 95% (216/227) Checking out files: 96% (218/227) Checking out files: 97% (221/227) Checking out files: 98% (223/227) Checking out files: 99% (225/227) Checking out files: 100% (227/227) Checking out files: 100% (227/227), done. 2019-12-05T06:45:31.279 INFO:tasks.workunit.client.0.smithi042.stdout:HEAD is now at a4c752df71 v0.6 2019-12-05T06:45:31.283 INFO:tasks.workunit.client.0.smithi042.stderr:+ mkdir .snap/v0.6 2019-12-05T06:45:31.479 INFO:tasks.workunit.client.0.smithi042.stderr:+ ver=v0.7 2019-12-05T06:45:31.480 INFO:tasks.workunit.client.0.smithi042.stderr:+ echo v0.7 2019-12-05T06:45:31.480 INFO:tasks.workunit.client.0.smithi042.stderr:+ git reset --hard v0.7 2019-12-05T06:45:31.480 INFO:tasks.workunit.client.0.smithi042.stdout:v0.7 2019-12-05T06:45:33.955 INFO:tasks.workunit.client.0.smithi042.stderr:Checking out files: 68% (138/201) Checking out files: 69% (139/201) Checking out files: 70% (141/201) Checking out files: 71% (143/201) Checking out files: 72% (145/201) Checking out files: 73% (147/201) Checking out files: 74% (149/201) Checking out files: 75% (151/201) Checking out files: 76% (153/201) Checking out files: 77% (155/201) Checking out files: 78% (157/201) Checking out files: 79% (159/201) Checking out files: 80% (161/201) Checking out files: 81% (163/201) Checking out files: 82% (165/201) Checking out files: 83% (167/201) Checking out files: 84% (169/201) Checking out files: 85% (171/201) Checking out files: 86% (173/201) Checking out files: 87% (175/201) Checking out files: 88% (177/201) Checking out files: 89% (179/201) Checking out files: 90% (181/201) Checking out files: 91% (183/201) Checking out files: 92% (185/201) Checking out files: 93% (187/201) Checking out files: 94% (189/201) Checking out files: 95% (191/201) Checking out files: 96% (193/201) Checking out files: 97% (195/201) Checking out files: 98% (197/201) Checking out files: 99% (199/201) Checking out files: 100% (201/201) Checking out files: 100% (201/201), done. 2019-12-05T06:45:34.045 INFO:tasks.workunit.client.0.smithi042.stdout:HEAD is now at c46e1b320c mkcephfs: btrfsctl -a between mkfs and mount 2019-12-05T06:45:34.051 INFO:tasks.workunit.client.0.smithi042.stderr:+ mkdir .snap/v0.7 2019-12-05T06:45:34.215 INFO:tasks.workunit.client.0.smithi042.stderr:+ ver=v0.8 2019-12-05T06:45:34.215 INFO:tasks.workunit.client.0.smithi042.stderr:+ echo v0.8 2019-12-05T06:45:34.215 INFO:tasks.workunit.client.0.smithi042.stderr:+ git reset --hard v0.8 2019-12-05T06:45:34.216 INFO:tasks.workunit.client.0.smithi042.stdout:v0.8 2019-12-05T06:45:37.994 INFO:tasks.workunit.client.0.smithi042.stderr:Checking out files: 38% (87/225) Checking out files: 39% (88/225) Checking out files: 40% (90/225) Checking out files: 41% (93/225) Checking out files: 42% (95/225) Checking out files: 43% (97/225) Checking out files: 44% (99/225) Checking out files: 45% (102/225) Checking out files: 46% (104/225) Checking out files: 47% (106/225) Checking out files: 48% (108/225) Checking out files: 49% (111/225) Checking out files: 50% (113/225) Checking out files: 51% (115/225) Checking out files: 52% (117/225) Checking out files: 53% (120/225) Checking out files: 54% (122/225) Checking out files: 55% (124/225) Checking out files: 56% (126/225) Checking out files: 57% (129/225) Checking out files: 58% (131/225) Checking out files: 59% (133/225) Checking out files: 60% (135/225) Checking out files: 61% (138/225) Checking out files: 62% (140/225) Checking out files: 63% (142/225) Checking out files: 64% (144/225) Checking out files: 65% (147/225) Checking out files: 66% (149/225) Checking out files: 67% (151/225) Checking out files: 68% (153/225) Checking out files: 69% (156/225) Checking out files: 70% (158/225) Checking out files: 71% (160/225) Checking out files: 72% (162/225) Checking out files: 73% (165/225) Checking out files: 74% (167/225) Checking out files: 75% (169/225) Checking out files: 76% (171/225) Checking out files: 77% (174/225) Checking out files: 78% (176/225) Checking out files: 79% (178/225) Checking out files: 80% (180/225) Checking out files: 81% (183/225) Checking out files: 82% (185/225) Checking out files: 83% (187/225) Checking out files: 84% (189/225) Checking out files: 85% (192/225) Checking out files: 86% (194/225) Checking out files: 87% (196/225) Checking out files: 88% (198/225) Checking out files: 89% (201/225) Checking out files: 90% (203/225) Checking out files: 91% (205/225) Checking out files: 92% (207/225) Checking out files: 93% (210/225) Checking out files: 94% (212/225) Checking out files: 95% (214/225) Checking out files: 96% (216/225) Checking out files: 97% (219/225) Checking out files: 98% (221/225) Checking out files: 99% (223/225) Checking out files: 100% (225/225) Checking out files: 100% (225/225), done. 2019-12-05T06:45:38.630 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:45:38.993 INFO:tasks.workunit.client.0.smithi042.stdout:HEAD is now at 284161c00a kclient: kill dput/d_drop debug cruft 2019-12-05T06:45:38.998 INFO:tasks.workunit.client.0.smithi042.stderr:+ mkdir .snap/v0.8 2019-12-05T06:45:39.423 INFO:tasks.workunit.client.0.smithi042.stderr:+ ver=v0.9 2019-12-05T06:45:39.423 INFO:tasks.workunit.client.0.smithi042.stderr:+ echo v0.9 2019-12-05T06:45:39.423 INFO:tasks.workunit.client.0.smithi042.stderr:+ git reset --hard v0.9 2019-12-05T06:45:39.423 INFO:tasks.workunit.client.0.smithi042.stdout:v0.9 2019-12-05T06:45:39.903 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:39.893+0000 7f0388747700 1 Processor -- start 2019-12-05T06:45:39.903 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:39.893+0000 7f0388747700 1 -- start start 2019-12-05T06:45:39.904 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:39.893+0000 7f0388747700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f03800a7740 0x7f038008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:45:39.904 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:39.893+0000 7f0388747700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f038009fae0 0x7f038007eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:45:39.904 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:39.893+0000 7f0388747700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03800a0710 con 0x7f038007e720 2019-12-05T06:45:39.904 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:39.893+0000 7f0388747700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03800a0b00 con 0x7f038009fae0 2019-12-05T06:45:39.904 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:39.893+0000 7f0388747700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f038008e5b0 con 0x7f03800a7740 2019-12-05T06:45:39.905 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:39.893+0000 7f0386ce4700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f03800a7740 0x7f038008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:45:39.905 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:39.893+0000 7f0386ce4700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f03800a7740 0x7f038008df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:36116/0 (socket says 172.21.15.42:36116) 2019-12-05T06:45:39.905 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:39.893+0000 7f0386ce4700 1 -- 172.21.15.42:0/3656010475 learned_addr learned my addr 172.21.15.42:0/3656010475 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:45:39.905 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:39.893+0000 7f03874e5700 1 --2- 172.21.15.42:0/3656010475 >> v2:172.21.15.40:3300/0 conn(0x7f038009fae0 0x7f038007eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:45:39.905 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:39.893+0000 7f0386ce4700 1 -- 172.21.15.42:0/3656010475 >> v2:172.21.15.40:3300/0 conn(0x7f038009fae0 msgr2=0x7f038007eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:45:39.905 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:39.893+0000 7f0386ce4700 1 --2- 172.21.15.42:0/3656010475 >> v2:172.21.15.40:3300/0 conn(0x7f038009fae0 0x7f038007eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:45:39.905 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:39.893+0000 7f0386ce4700 1 -- 172.21.15.42:0/3656010475 >> v1:172.21.15.40:6789/0 conn(0x7f038007e720 legacy=0x7f038009f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:45:39.906 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:39.893+0000 7f0386ce4700 1 -- 172.21.15.42:0/3656010475 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f038007f0a0 con 0x7f03800a7740 2019-12-05T06:45:39.906 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:39.893+0000 7f03874e5700 1 --2- 172.21.15.42:0/3656010475 >> v2:172.21.15.40:3300/0 conn(0x7f038009fae0 0x7f038007eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:45:39.906 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:39.893+0000 7f0386ce4700 1 --2- 172.21.15.42:0/3656010475 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f03800a7740 0x7f038008df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f037c008790 tx=0x7f037c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:39.906 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:39.897+0000 7f0386ce4700 1 --2- 172.21.15.42:0/3656010475 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f03800a7740 0x7f038008df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f037c008790 tx=0x7f037c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:39.906 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:39.897+0000 7f0386ce4700 1 --2- 172.21.15.42:0/3656010475 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f03800a7740 0x7f038008df80 secure :-1 s=READY pgs=698 cs=0 l=1 rx=0x7f037c008790 tx=0x7f037c00fd30).ready entity=mon.2 client_cookie=ec755273e31ef637 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:45:39.906 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:39.897+0000 7f0386ce4700 1 --2- 172.21.15.42:0/3656010475 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f03800a7740 0x7f038008df80 secure :-1 s=THROTTLE_DONE pgs=698 cs=0 l=1 rx=0x7f037c008790 tx=0x7f037c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:39.906 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:39.897+0000 7f0385ce2700 1 -- 172.21.15.42:0/3656010475 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f037c008c90 con 0x7f03800a7740 2019-12-05T06:45:39.907 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:39.897+0000 7f0386ce4700 1 --2- 172.21.15.42:0/3656010475 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f03800a7740 0x7f038008df80 secure :-1 s=THROTTLE_DONE pgs=698 cs=0 l=1 rx=0x7f037c008790 tx=0x7f037c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:39.907 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:39.897+0000 7f0386ce4700 1 --2- 172.21.15.42:0/3656010475 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f03800a7740 0x7f038008df80 secure :-1 s=THROTTLE_DONE pgs=698 cs=0 l=1 rx=0x7f037c008790 tx=0x7f037c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:39.907 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:39.897+0000 7f0385ce2700 1 -- 172.21.15.42:0/3656010475 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f037c0088b0 con 0x7f03800a7740 2019-12-05T06:45:39.907 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:39.897+0000 7f0385ce2700 1 -- 172.21.15.42:0/3656010475 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f037c011b10 con 0x7f03800a7740 2019-12-05T06:45:39.907 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:39.897+0000 7f0388747700 1 -- 172.21.15.42:0/3656010475 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f03800a7740 msgr2=0x7f038008df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:45:39.907 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:39.897+0000 7f0388747700 1 --2- 172.21.15.42:0/3656010475 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f03800a7740 0x7f038008df80 secure :-1 s=READY pgs=698 cs=0 l=1 rx=0x7f037c008790 tx=0x7f037c00fd30).stop 2019-12-05T06:45:39.907 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:39.897+0000 7f0388747700 1 -- 172.21.15.42:0/3656010475 shutdown_connections 2019-12-05T06:45:39.908 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:39.897+0000 7f0388747700 1 --2- 172.21.15.42:0/3656010475 >> v2:172.21.15.40:3300/0 conn(0x7f038009fae0 0x7f038007eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:45:39.908 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:39.897+0000 7f0388747700 1 --2- 172.21.15.42:0/3656010475 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f03800a7740 0x7f038008df80 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:45:39.908 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:39.897+0000 7f0388747700 1 -- 172.21.15.42:0/3656010475 shutdown_connections 2019-12-05T06:45:39.908 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:39.897+0000 7f0388747700 1 -- 172.21.15.42:0/3656010475 wait complete. 2019-12-05T06:45:39.908 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:39.897+0000 7f0388747700 1 -- 172.21.15.42:0/3656010475 >> 172.21.15.42:0/3656010475 conn(0x7f03800aabe0 msgr2=0x7f03800a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:45:39.912 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:39.897+0000 7f0388747700 1 Processor -- start 2019-12-05T06:45:39.912 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:39.897+0000 7f0388747700 1 -- start start 2019-12-05T06:45:39.912 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:39.901+0000 7f0388747700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f0380124ec0 0x7f0380124500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:45:39.912 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:39.901+0000 7f0388747700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0380123ab0 0x7f038013bc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:45:39.912 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:39.901+0000 7f0388747700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f038007c910 con 0x7f03801209e0 2019-12-05T06:45:39.913 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:39.901+0000 7f0388747700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03801203b0 con 0x7f0380124ec0 2019-12-05T06:45:39.913 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:39.901+0000 7f0388747700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0380121260 con 0x7f0380123ab0 2019-12-05T06:45:39.913 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:39.901+0000 7f03874e5700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0380123ab0 0x7f038013bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:45:39.913 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:39.901+0000 7f0386ce4700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f0380124ec0 0x7f0380124500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:45:39.913 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:39.901+0000 7f0386ce4700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f0380124ec0 0x7f0380124500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.42:51704/0 (socket says 172.21.15.42:51704) 2019-12-05T06:45:39.913 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:39.901+0000 7f0386ce4700 1 -- 172.21.15.42:0/3385962463 learned_addr learned my addr 172.21.15.42:0/3385962463 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:45:39.913 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:39.901+0000 7f03864e3700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f03801209e0 0x7f03801232b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:45136/0 (socket says 172.21.15.42:45136) 2019-12-05T06:45:39.914 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:39.901+0000 7f03874e5700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0380123ab0 0x7f038013bc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:36126/0 (socket says 172.21.15.42:36126) 2019-12-05T06:45:39.914 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:39.901+0000 7f0386ce4700 1 -- 172.21.15.42:0/3385962463 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0380123ab0 msgr2=0x7f038013bc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:45:39.914 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:39.901+0000 7f0386ce4700 1 --2- 172.21.15.42:0/3385962463 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0380123ab0 0x7f038013bc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:45:39.914 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:39.901+0000 7f0386ce4700 1 -- 172.21.15.42:0/3385962463 >> v1:172.21.15.40:6789/0 conn(0x7f03801209e0 legacy=0x7f03801232b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:45:39.914 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:39.901+0000 7f03777fe700 1 -- 172.21.15.42:0/3385962463 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 119235649 0 0) 0x7f038007c910 con 0x7f03801209e0 2019-12-05T06:45:39.914 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:39.901+0000 7f0386ce4700 1 -- 172.21.15.42:0/3385962463 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0380124a50 con 0x7f0380124ec0 2019-12-05T06:45:39.914 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:39.901+0000 7f0386ce4700 1 --2- 172.21.15.42:0/3385962463 >> v2:172.21.15.40:3300/0 conn(0x7f0380124ec0 0x7f0380124500 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f037c008790 tx=0x7f037c008ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:39.915 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:39.901+0000 7f0386ce4700 1 --2- 172.21.15.42:0/3385962463 >> v2:172.21.15.40:3300/0 conn(0x7f0380124ec0 0x7f0380124500 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f037c008790 tx=0x7f037c008ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:39.915 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:39.901+0000 7f0386ce4700 1 --2- 172.21.15.42:0/3385962463 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0380124ec0 0x7f0380124500 secure :-1 s=READY pgs=2393 cs=0 l=1 rx=0x7f037c008790 tx=0x7f037c008ae0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:45:39.915 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:39.901+0000 7f0386ce4700 1 --2- 172.21.15.42:0/3385962463 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0380124ec0 0x7f0380124500 secure :-1 s=THROTTLE_DONE pgs=2393 cs=0 l=1 rx=0x7f037c008790 tx=0x7f037c008ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:39.915 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:39.901+0000 7f03777fe700 1 -- 172.21.15.42:0/3385962463 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f037c00f470 con 0x7f0380124ec0 2019-12-05T06:45:39.915 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:39.901+0000 7f0386ce4700 1 --2- 172.21.15.42:0/3385962463 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0380124ec0 0x7f0380124500 secure :-1 s=THROTTLE_DONE pgs=2393 cs=0 l=1 rx=0x7f037c008790 tx=0x7f037c008ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:39.915 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:39.901+0000 7f0386ce4700 1 --2- 172.21.15.42:0/3385962463 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0380124ec0 0x7f0380124500 secure :-1 s=THROTTLE_DONE pgs=2393 cs=0 l=1 rx=0x7f037c008790 tx=0x7f037c008ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:39.915 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:39.901+0000 7f03777fe700 1 -- 172.21.15.42:0/3385962463 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f037c011730 con 0x7f0380124ec0 2019-12-05T06:45:39.916 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:39.901+0000 7f03777fe700 1 -- 172.21.15.42:0/3385962463 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f037c005730 con 0x7f0380124ec0 2019-12-05T06:45:39.916 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:39.901+0000 7f0388747700 1 -- 172.21.15.42:0/3385962463 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0380120e10 con 0x7f0380124ec0 2019-12-05T06:45:39.916 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:39.901+0000 7f0388747700 1 -- 172.21.15.42:0/3385962463 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f038013c170 con 0x7f0380124ec0 2019-12-05T06:45:39.916 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:39.901+0000 7f0386ce4700 1 --2- 172.21.15.42:0/3385962463 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0380124ec0 0x7f0380124500 secure :-1 s=THROTTLE_DONE pgs=2393 cs=0 l=1 rx=0x7f037c008790 tx=0x7f037c008ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:39.916 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:39.905+0000 7f03777fe700 1 -- 172.21.15.42:0/3385962463 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f037c005980 con 0x7f0380124ec0 2019-12-05T06:45:39.916 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:39.905+0000 7f0386ce4700 1 --2- 172.21.15.42:0/3385962463 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0380124ec0 0x7f0380124500 secure :-1 s=THROTTLE_DONE pgs=2393 cs=0 l=1 rx=0x7f037c008790 tx=0x7f037c008ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:39.916 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:39.905+0000 7f03777fe700 1 --2- 172.21.15.42:0/3385962463 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f036803ac30 0x7f036803d140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:45:39.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:39.905+0000 7f03777fe700 1 -- 172.21.15.42:0/3385962463 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 11965+0+0 (secure 0 0 0) 0x7f037c04b530 con 0x7f0380124ec0 2019-12-05T06:45:39.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:39.905+0000 7f03864e3700 1 --2- 172.21.15.42:0/3385962463 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f036803ac30 0x7f036803d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:45:39.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:39.905+0000 7f03864e3700 1 --2- 172.21.15.42:0/3385962463 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f036803ac30 0x7f036803d140 crc :-1 s=READY pgs=1324 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:45:39.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:39.909+0000 7f0388747700 1 -- 172.21.15.42:0/3385962463 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f038005d110 con 0x7f0380124ec0 2019-12-05T06:45:39.922 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:39.917+0000 7f0386ce4700 1 --2- 172.21.15.42:0/3385962463 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0380124ec0 0x7f0380124500 secure :-1 s=THROTTLE_DONE pgs=2393 cs=0 l=1 rx=0x7f037c008790 tx=0x7f037c008ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:39.922 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:39.917+0000 7f03777fe700 1 -- 172.21.15.42:0/3385962463 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f037c0327f0 con 0x7f0380124ec0 2019-12-05T06:45:40.109 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:40.105+0000 7f0388747700 1 -- 172.21.15.42:0/3385962463 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f038005b8c0 con 0x7f0380124ec0 2019-12-05T06:45:40.110 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:40.105+0000 7f0386ce4700 1 --2- 172.21.15.42:0/3385962463 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0380124ec0 0x7f0380124500 secure :-1 s=THROTTLE_DONE pgs=2393 cs=0 l=1 rx=0x7f037c008790 tx=0x7f037c008ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:40.110 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:40.105+0000 7f03777fe700 1 -- 172.21.15.42:0/3385962463 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 383 v383) v1 ==== 95+0+2117 (secure 0 0 0) 0x7f037c007f10 con 0x7f0380124ec0 2019-12-05T06:45:40.110 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 383 2019-12-05T06:45:40.110 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:45:40.110 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":383,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":12429,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/4178556019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":4178556019},{"type":"v1","addr":"172.21.15.40:6833","nonce":4178556019}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":383}],"filesystems":[{"mdsmap":{"epoch":382,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:44:34.328051+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":118,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12303},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12303":{"gid":12303,"name":"a","rank":0,"incarnation":378,"state":"up:active","state_seq":17,"addr":"172.21.15.42:6835/2489540211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2489540211},{"type":"v1","addr":"172.21.15.42:6835","nonce":2489540211}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:45:40.114 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:40.109+0000 7f0388747700 1 -- 172.21.15.42:0/3385962463 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f036803ac30 msgr2=0x7f036803d140 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:45:40.114 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:40.109+0000 7f0388747700 1 --2- 172.21.15.42:0/3385962463 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f036803ac30 0x7f036803d140 crc :-1 s=READY pgs=1324 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:45:40.114 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:40.109+0000 7f0388747700 1 -- 172.21.15.42:0/3385962463 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0380124ec0 msgr2=0x7f0380124500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:45:40.114 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:40.109+0000 7f0388747700 1 --2- 172.21.15.42:0/3385962463 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0380124ec0 0x7f0380124500 secure :-1 s=READY pgs=2393 cs=0 l=1 rx=0x7f037c008790 tx=0x7f037c008ae0).stop 2019-12-05T06:45:40.114 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:40.109+0000 7f0388747700 1 -- 172.21.15.42:0/3385962463 shutdown_connections 2019-12-05T06:45:40.114 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:40.109+0000 7f0388747700 1 --2- 172.21.15.42:0/3385962463 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f036803ac30 0x7f036803d140 unknown :-1 s=CLOSED pgs=1324 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:45:40.115 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:40.109+0000 7f0388747700 1 --2- 172.21.15.42:0/3385962463 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0380123ab0 0x7f038013bc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:45:40.115 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:40.109+0000 7f0388747700 1 --2- 172.21.15.42:0/3385962463 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0380124ec0 0x7f0380124500 unknown :-1 s=CLOSED pgs=2393 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:45:40.115 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:40.109+0000 7f0388747700 1 -- 172.21.15.42:0/3385962463 shutdown_connections 2019-12-05T06:45:40.115 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:40.109+0000 7f0388747700 1 -- 172.21.15.42:0/3385962463 wait complete. 2019-12-05T06:45:40.116 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:40.109+0000 7f0388747700 1 -- 172.21.15.42:0/3385962463 >> 172.21.15.42:0/3385962463 conn(0x7f03800a8410 msgr2=0x7f038009f580 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:45:40.127 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.a (rank=0) 2019-12-05T06:45:40.127 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2019-12-05T06:45:40.127 INFO:teuthology.orchestra.run:waiting for 300 2019-12-05T06:45:40.138 INFO:tasks.ceph.mds.a:Stopped 2019-12-05T06:45:40.139 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:45:40.317 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:40.297+0000 7f829ae0f700 1 Processor -- start 2019-12-05T06:45:40.318 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:40.297+0000 7f829ae0f700 1 -- start start 2019-12-05T06:45:40.318 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:40.297+0000 7f829ae0f700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f82940a7740 0x7f829408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:45:40.318 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:40.297+0000 7f829ae0f700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f829407e720 0x7f829407eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:45:40.318 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:40.297+0000 7f829ae0f700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82940a0710 con 0x7f829409f2e0 2019-12-05T06:45:40.318 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:40.297+0000 7f829ae0f700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82940a0b00 con 0x7f82940a7740 2019-12-05T06:45:40.318 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:40.297+0000 7f829ae0f700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f829408e5b0 con 0x7f829407e720 2019-12-05T06:45:40.319 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:40.297+0000 7f8299bad700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f829409f2e0 0x7f829409f710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:35708/0 (socket says 172.21.15.42:35708) 2019-12-05T06:45:40.319 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:40.297+0000 7f82993ac700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f82940a7740 0x7f829408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:45:40.319 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:40.297+0000 7f8299bad700 1 -- 172.21.15.42:0/3812933112 learned_addr learned my addr 172.21.15.42:0/3812933112 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:45:40.319 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:40.297+0000 7f8298bab700 1 --2- 172.21.15.42:0/3812933112 >> v2:172.21.15.42:3300/0 conn(0x7f829407e720 0x7f829407eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:45:40.319 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:40.301+0000 7f8298bab700 1 -- 172.21.15.42:0/3812933112 >> v2:172.21.15.40:3300/0 conn(0x7f82940a7740 msgr2=0x7f829408df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:45:40.319 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:40.301+0000 7f8298bab700 1 --2- 172.21.15.42:0/3812933112 >> v2:172.21.15.40:3300/0 conn(0x7f82940a7740 0x7f829408df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:45:40.320 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:40.301+0000 7f8298bab700 1 -- 172.21.15.42:0/3812933112 >> v1:172.21.15.42:6789/0 conn(0x7f829409f2e0 legacy=0x7f829409f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:45:40.320 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:40.301+0000 7f8298bab700 1 -- 172.21.15.42:0/3812933112 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f829407f0a0 con 0x7f829407e720 2019-12-05T06:45:40.320 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:40.301+0000 7f82993ac700 1 --2- 172.21.15.42:0/3812933112 >> v2:172.21.15.40:3300/0 conn(0x7f82940a7740 0x7f829408df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:45:40.320 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:40.301+0000 7f8298bab700 1 --2- 172.21.15.42:0/3812933112 >> v2:172.21.15.42:3300/0 conn(0x7f829407e720 0x7f829407eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f8290002f30 tx=0x7f829000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:40.320 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:40.301+0000 7f8298bab700 1 --2- 172.21.15.42:0/3812933112 >> v2:172.21.15.42:3300/0 conn(0x7f829407e720 0x7f829407eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f8290002f30 tx=0x7f829000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:40.320 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:40.301+0000 7f8298bab700 1 --2- 172.21.15.42:0/3812933112 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f829407e720 0x7f829407eb50 secure :-1 s=READY pgs=3164 cs=0 l=1 rx=0x7f8290002f30 tx=0x7f829000fc70).ready entity=mon.1 client_cookie=852b02000076e824 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:45:40.320 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:40.301+0000 7f8298bab700 1 --2- 172.21.15.42:0/3812933112 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f829407e720 0x7f829407eb50 secure :-1 s=THROTTLE_DONE pgs=3164 cs=0 l=1 rx=0x7f8290002f30 tx=0x7f829000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:40.321 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:40.301+0000 7f8298bab700 1 --2- 172.21.15.42:0/3812933112 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f829407e720 0x7f829407eb50 secure :-1 s=THROTTLE_DONE pgs=3164 cs=0 l=1 rx=0x7f8290002f30 tx=0x7f829000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:40.321 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:40.301+0000 7f828bfff700 1 -- 172.21.15.42:0/3812933112 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f82900112b0 con 0x7f829407e720 2019-12-05T06:45:40.321 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:40.301+0000 7f8298bab700 1 --2- 172.21.15.42:0/3812933112 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f829407e720 0x7f829407eb50 secure :-1 s=THROTTLE_DONE pgs=3164 cs=0 l=1 rx=0x7f8290002f30 tx=0x7f829000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:40.321 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:40.301+0000 7f828bfff700 1 -- 172.21.15.42:0/3812933112 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8290008ad0 con 0x7f829407e720 2019-12-05T06:45:40.321 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:40.301+0000 7f828bfff700 1 -- 172.21.15.42:0/3812933112 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8290011890 con 0x7f829407e720 2019-12-05T06:45:40.321 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:40.301+0000 7f829ae0f700 1 -- 172.21.15.42:0/3812933112 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f829407e720 msgr2=0x7f829407eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:45:40.321 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:40.301+0000 7f829ae0f700 1 --2- 172.21.15.42:0/3812933112 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f829407e720 0x7f829407eb50 secure :-1 s=READY pgs=3164 cs=0 l=1 rx=0x7f8290002f30 tx=0x7f829000fc70).stop 2019-12-05T06:45:40.322 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:40.301+0000 7f829ae0f700 1 -- 172.21.15.42:0/3812933112 shutdown_connections 2019-12-05T06:45:40.322 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:40.301+0000 7f829ae0f700 1 --2- 172.21.15.42:0/3812933112 >> v2:172.21.15.40:3300/0 conn(0x7f82940a7740 0x7f829408df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:45:40.322 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:40.301+0000 7f829ae0f700 1 --2- 172.21.15.42:0/3812933112 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f829407e720 0x7f829407eb50 unknown :-1 s=CLOSED pgs=3164 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:45:40.322 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:40.301+0000 7f829ae0f700 1 -- 172.21.15.42:0/3812933112 shutdown_connections 2019-12-05T06:45:40.322 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:40.301+0000 7f829ae0f700 1 -- 172.21.15.42:0/3812933112 wait complete. 2019-12-05T06:45:40.322 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:40.301+0000 7f829ae0f700 1 -- 172.21.15.42:0/3812933112 >> 172.21.15.42:0/3812933112 conn(0x7f82940aabe0 msgr2=0x7f82940a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:45:40.322 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:40.301+0000 7f829ae0f700 1 Processor -- start 2019-12-05T06:45:40.323 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:40.301+0000 7f829ae0f700 1 -- start start 2019-12-05T06:45:40.323 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:40.305+0000 7f829ae0f700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f82941289f0 0x7f8294124a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:45:40.323 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:40.305+0000 7f829ae0f700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8294128e20 con 0x7f82941469e0 2019-12-05T06:45:40.323 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:40.305+0000 7f829ae0f700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8294146e10 con 0x7f82940a3a60 2019-12-05T06:45:40.323 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:40.305+0000 7f829ae0f700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8294146130 con 0x7f82941289f0 2019-12-05T06:45:40.323 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:40.305+0000 7f82993ac700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f82940a3a60 0x7f8294125320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:35710/0 (socket says 172.21.15.42:35710) 2019-12-05T06:45:40.323 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:40.305+0000 7f82993ac700 1 -- 172.21.15.42:0/4280685286 learned_addr learned my addr 172.21.15.42:0/4280685286 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:45:40.324 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:40.305+0000 7f8298bab700 1 --2- 172.21.15.42:0/4280685286 >> v2:172.21.15.40:3300/0 conn(0x7f82941289f0 0x7f8294124a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:45:40.324 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:40.305+0000 7f8289ffb700 1 -- 172.21.15.42:0/4280685286 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1399222819 0 0) 0x7f8294146e10 con 0x7f82940a3a60 2019-12-05T06:45:40.324 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:40.305+0000 7f8289ffb700 1 -- 172.21.15.42:0/4280685286 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82740030b0 con 0x7f82940a3a60 2019-12-05T06:45:40.324 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:40.305+0000 7f8289ffb700 1 -- 172.21.15.42:0/4280685286 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 825665156 0 0) 0x7f82740030b0 con 0x7f82940a3a60 2019-12-05T06:45:40.324 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:40.305+0000 7f8289ffb700 1 -- 172.21.15.42:0/4280685286 --> v1:172.21.15.42:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8294146e10 con 0x7f82940a3a60 2019-12-05T06:45:40.324 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:40.305+0000 7f8289ffb700 1 -- 172.21.15.42:0/4280685286 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7f828c002840 con 0x7f82940a3a60 2019-12-05T06:45:40.325 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:40.305+0000 7f8298bab700 1 -- 172.21.15.42:0/4280685286 >> v1:172.21.15.42:6789/0 conn(0x7f82940a3a60 legacy=0x7f8294125320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:45:40.325 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:40.305+0000 7f8298bab700 1 -- 172.21.15.42:0/4280685286 >> v1:172.21.15.40:6789/0 conn(0x7f82941469e0 legacy=0x7f829412f430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:45:40.325 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:40.305+0000 7f8298bab700 1 -- 172.21.15.42:0/4280685286 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8294147040 con 0x7f82941289f0 2019-12-05T06:45:40.325 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:40.305+0000 7f8298bab700 1 --2- 172.21.15.42:0/4280685286 >> v2:172.21.15.40:3300/0 conn(0x7f82941289f0 0x7f8294124a20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f8284001000 tx=0x7f8284001040).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:40.325 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:40.305+0000 7f8298bab700 1 --2- 172.21.15.42:0/4280685286 >> v2:172.21.15.40:3300/0 conn(0x7f82941289f0 0x7f8294124a20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f8284001000 tx=0x7f8284001040).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:40.325 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:40.305+0000 7f8298bab700 1 --2- 172.21.15.42:0/4280685286 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f82941289f0 0x7f8294124a20 secure :-1 s=READY pgs=2395 cs=0 l=1 rx=0x7f8284001000 tx=0x7f8284001040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:45:40.325 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:40.305+0000 7f8298bab700 1 --2- 172.21.15.42:0/4280685286 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f82941289f0 0x7f8294124a20 secure :-1 s=THROTTLE_DONE pgs=2395 cs=0 l=1 rx=0x7f8284001000 tx=0x7f8284001040).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:40.326 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:40.305+0000 7f8289ffb700 1 -- 172.21.15.42:0/4280685286 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8290007340 con 0x7f82941289f0 2019-12-05T06:45:40.326 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:40.305+0000 7f829ae0f700 1 -- 172.21.15.42:0/4280685286 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8294130c40 con 0x7f82941289f0 2019-12-05T06:45:40.326 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:40.305+0000 7f8298bab700 1 --2- 172.21.15.42:0/4280685286 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f82941289f0 0x7f8294124a20 secure :-1 s=THROTTLE_DONE pgs=2395 cs=0 l=1 rx=0x7f8284001000 tx=0x7f8284001040).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:40.326 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:40.305+0000 7f8289ffb700 1 -- 172.21.15.42:0/4280685286 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82900114b0 con 0x7f82941289f0 2019-12-05T06:45:40.326 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:40.305+0000 7f8298bab700 1 --2- 172.21.15.42:0/4280685286 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f82941289f0 0x7f8294124a20 secure :-1 s=THROTTLE_DONE pgs=2395 cs=0 l=1 rx=0x7f8284001000 tx=0x7f8284001040).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:40.326 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:40.305+0000 7f829ae0f700 1 -- 172.21.15.42:0/4280685286 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8294131110 con 0x7f82941289f0 2019-12-05T06:45:40.326 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:40.305+0000 7f8289ffb700 1 -- 172.21.15.42:0/4280685286 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8290008ad0 con 0x7f82941289f0 2019-12-05T06:45:40.327 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:40.305+0000 7f8298bab700 1 --2- 172.21.15.42:0/4280685286 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f82941289f0 0x7f8294124a20 secure :-1 s=THROTTLE_DONE pgs=2395 cs=0 l=1 rx=0x7f8284001000 tx=0x7f8284001040).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:40.327 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:40.305+0000 7f8289ffb700 1 -- 172.21.15.42:0/4280685286 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f8290005540 con 0x7f82941289f0 2019-12-05T06:45:40.327 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:40.305+0000 7f8298bab700 1 --2- 172.21.15.42:0/4280685286 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f82941289f0 0x7f8294124a20 secure :-1 s=THROTTLE_DONE pgs=2395 cs=0 l=1 rx=0x7f8284001000 tx=0x7f8284001040).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:40.327 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:40.309+0000 7f8289ffb700 1 --2- 172.21.15.42:0/4280685286 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f827403b620 0x7f827403db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:45:40.327 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:40.309+0000 7f8289ffb700 1 -- 172.21.15.42:0/4280685286 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 11965+0+0 (secure 0 0 0) 0x7f829004b030 con 0x7f82941289f0 2019-12-05T06:45:40.327 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:40.309+0000 7f82993ac700 1 --2- 172.21.15.42:0/4280685286 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f827403b620 0x7f827403db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:45:40.327 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:40.309+0000 7f82993ac700 1 --2- 172.21.15.42:0/4280685286 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f827403b620 0x7f827403db30 crc :-1 s=READY pgs=1325 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:45:40.328 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:40.313+0000 7f829ae0f700 1 -- 172.21.15.42:0/4280685286 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f829405d110 con 0x7f82941289f0 2019-12-05T06:45:40.328 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:40.317+0000 7f8298bab700 1 --2- 172.21.15.42:0/4280685286 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f82941289f0 0x7f8294124a20 secure :-1 s=THROTTLE_DONE pgs=2395 cs=0 l=1 rx=0x7f8284001000 tx=0x7f8284001040).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:40.329 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:40.317+0000 7f8289ffb700 1 -- 172.21.15.42:0/4280685286 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f8290007f50 con 0x7f82941289f0 2019-12-05T06:45:40.521 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:40.513+0000 7f829ae0f700 1 -- 172.21.15.42:0/4280685286 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f829405b8c0 con 0x7f82941289f0 2019-12-05T06:45:40.521 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:40.517+0000 7f8298bab700 1 --2- 172.21.15.42:0/4280685286 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f82941289f0 0x7f8294124a20 secure :-1 s=THROTTLE_DONE pgs=2395 cs=0 l=1 rx=0x7f8284001000 tx=0x7f8284001040).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:40.521 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:40.517+0000 7f8289ffb700 1 -- 172.21.15.42:0/4280685286 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 383 v383) v1 ==== 95+0+2117 (secure 0 0 0) 0x7f8290012c00 con 0x7f82941289f0 2019-12-05T06:45:40.521 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 383 2019-12-05T06:45:40.522 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:45:40.522 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":383,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":12429,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/4178556019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":4178556019},{"type":"v1","addr":"172.21.15.40:6833","nonce":4178556019}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":383}],"filesystems":[{"mdsmap":{"epoch":382,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:44:34.328051+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":118,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12303},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12303":{"gid":12303,"name":"a","rank":0,"incarnation":378,"state":"up:active","state_seq":17,"addr":"172.21.15.42:6835/2489540211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2489540211},{"type":"v1","addr":"172.21.15.42:6835","nonce":2489540211}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:45:40.524 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:40.517+0000 7f829ae0f700 1 -- 172.21.15.42:0/4280685286 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f827403b620 msgr2=0x7f827403db30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:45:40.524 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:40.517+0000 7f829ae0f700 1 --2- 172.21.15.42:0/4280685286 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f827403b620 0x7f827403db30 crc :-1 s=READY pgs=1325 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:45:40.525 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:40.517+0000 7f829ae0f700 1 -- 172.21.15.42:0/4280685286 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f82941289f0 msgr2=0x7f8294124a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:45:40.525 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:40.517+0000 7f829ae0f700 1 --2- 172.21.15.42:0/4280685286 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f82941289f0 0x7f8294124a20 secure :-1 s=READY pgs=2395 cs=0 l=1 rx=0x7f8284001000 tx=0x7f8284001040).stop 2019-12-05T06:45:40.525 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:40.517+0000 7f829ae0f700 1 -- 172.21.15.42:0/4280685286 shutdown_connections 2019-12-05T06:45:40.525 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:40.517+0000 7f829ae0f700 1 --2- 172.21.15.42:0/4280685286 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f827403b620 0x7f827403db30 unknown :-1 s=CLOSED pgs=1325 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:45:40.525 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:40.517+0000 7f829ae0f700 1 --2- 172.21.15.42:0/4280685286 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f82941289f0 0x7f8294124a20 unknown :-1 s=CLOSED pgs=2395 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:45:40.525 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:40.517+0000 7f829ae0f700 1 -- 172.21.15.42:0/4280685286 shutdown_connections 2019-12-05T06:45:40.526 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:40.517+0000 7f829ae0f700 1 -- 172.21.15.42:0/4280685286 wait complete. 2019-12-05T06:45:40.526 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:40.517+0000 7f829ae0f700 1 -- 172.21.15.42:0/4280685286 >> 172.21.15.42:0/4280685286 conn(0x7f82940a9770 msgr2=0x7f8294059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:45:40.534 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:45:42.535 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:45:42.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:42.629+0000 7fb9e1fed700 1 Processor -- start 2019-12-05T06:45:42.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:42.629+0000 7fb9e1fed700 1 -- start start 2019-12-05T06:45:42.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:42.633+0000 7fb9e1fed700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb9dc0a7730 0x7fb9dc08df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:45:42.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:42.633+0000 7fb9e1fed700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb9dc07e710 0x7fb9dc07eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:45:42.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:42.633+0000 7fb9e1fed700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9dc0a0700 con 0x7fb9dc09f2d0 2019-12-05T06:45:42.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:42.633+0000 7fb9e1fed700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9dc0a0af0 con 0x7fb9dc0a7730 2019-12-05T06:45:42.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:42.633+0000 7fb9e1fed700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9dc08e5a0 con 0x7fb9dc07e710 2019-12-05T06:45:42.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:42.633+0000 7fb9db7fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb9dc07e710 0x7fb9dc07eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:45:42.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:42.633+0000 7fb9e0d8b700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fb9dc09f2d0 0x7fb9dc09f700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:35726/0 (socket says 172.21.15.42:35726) 2019-12-05T06:45:42.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:42.633+0000 7fb9e0d8b700 1 -- 172.21.15.42:0/2588418061 learned_addr learned my addr 172.21.15.42:0/2588418061 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:45:42.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:42.633+0000 7fb9dbfff700 1 --2- 172.21.15.42:0/2588418061 >> v2:172.21.15.40:3300/0 conn(0x7fb9dc0a7730 0x7fb9dc08df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:45:42.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:42.633+0000 7fb9db7fe700 1 -- 172.21.15.42:0/2588418061 >> v2:172.21.15.40:3300/0 conn(0x7fb9dc0a7730 msgr2=0x7fb9dc08df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:45:42.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:42.633+0000 7fb9daffd700 1 -- 172.21.15.42:0/2588418061 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3610014726 0 0) 0x7fb9dc0a0700 con 0x7fb9dc09f2d0 2019-12-05T06:45:42.650 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:42.633+0000 7fb9db7fe700 1 --2- 172.21.15.42:0/2588418061 >> v2:172.21.15.40:3300/0 conn(0x7fb9dc0a7730 0x7fb9dc08df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:45:42.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:42.633+0000 7fb9db7fe700 1 -- 172.21.15.42:0/2588418061 >> v1:172.21.15.42:6789/0 conn(0x7fb9dc09f2d0 legacy=0x7fb9dc09f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:45:42.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:42.633+0000 7fb9db7fe700 1 -- 172.21.15.42:0/2588418061 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9dc07f090 con 0x7fb9dc07e710 2019-12-05T06:45:42.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:42.633+0000 7fb9dbfff700 1 --2- 172.21.15.42:0/2588418061 >> v2:172.21.15.40:3300/0 conn(0x7fb9dc0a7730 0x7fb9dc08df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:45:42.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:42.633+0000 7fb9db7fe700 1 --2- 172.21.15.42:0/2588418061 >> v2:172.21.15.42:3300/0 conn(0x7fb9dc07e710 0x7fb9dc07eb40 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fb9c4002f30 tx=0x7fb9c400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:42.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:42.633+0000 7fb9db7fe700 1 --2- 172.21.15.42:0/2588418061 >> v2:172.21.15.42:3300/0 conn(0x7fb9dc07e710 0x7fb9dc07eb40 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fb9c4002f30 tx=0x7fb9c400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:42.655 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:42.633+0000 7fb9db7fe700 1 --2- 172.21.15.42:0/2588418061 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb9dc07e710 0x7fb9dc07eb40 secure :-1 s=READY pgs=3167 cs=0 l=1 rx=0x7fb9c4002f30 tx=0x7fb9c400fc70).ready entity=mon.1 client_cookie=d65fe1e70853ee28 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:45:42.656 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:42.633+0000 7fb9db7fe700 1 --2- 172.21.15.42:0/2588418061 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb9dc07e710 0x7fb9dc07eb40 secure :-1 s=THROTTLE_DONE pgs=3167 cs=0 l=1 rx=0x7fb9c4002f30 tx=0x7fb9c400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:42.656 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:42.633+0000 7fb9daffd700 1 -- 172.21.15.42:0/2588418061 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb9c40112b0 con 0x7fb9dc07e710 2019-12-05T06:45:42.656 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:42.633+0000 7fb9db7fe700 1 --2- 172.21.15.42:0/2588418061 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb9dc07e710 0x7fb9dc07eb40 secure :-1 s=THROTTLE_DONE pgs=3167 cs=0 l=1 rx=0x7fb9c4002f30 tx=0x7fb9c400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:42.656 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:42.633+0000 7fb9daffd700 1 -- 172.21.15.42:0/2588418061 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9c4008ad0 con 0x7fb9dc07e710 2019-12-05T06:45:42.658 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:42.633+0000 7fb9db7fe700 1 --2- 172.21.15.42:0/2588418061 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb9dc07e710 0x7fb9dc07eb40 secure :-1 s=THROTTLE_DONE pgs=3167 cs=0 l=1 rx=0x7fb9c4002f30 tx=0x7fb9c400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:42.658 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:42.633+0000 7fb9daffd700 1 -- 172.21.15.42:0/2588418061 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb9c4011890 con 0x7fb9dc07e710 2019-12-05T06:45:42.658 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:42.633+0000 7fb9e1fed700 1 -- 172.21.15.42:0/2588418061 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb9dc07e710 msgr2=0x7fb9dc07eb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:45:42.658 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:42.633+0000 7fb9e1fed700 1 --2- 172.21.15.42:0/2588418061 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb9dc07e710 0x7fb9dc07eb40 secure :-1 s=READY pgs=3167 cs=0 l=1 rx=0x7fb9c4002f30 tx=0x7fb9c400fc70).stop 2019-12-05T06:45:42.659 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:42.633+0000 7fb9e1fed700 1 -- 172.21.15.42:0/2588418061 shutdown_connections 2019-12-05T06:45:42.659 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:42.633+0000 7fb9e1fed700 1 --2- 172.21.15.42:0/2588418061 >> v2:172.21.15.40:3300/0 conn(0x7fb9dc0a7730 0x7fb9dc08df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:45:42.659 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:42.633+0000 7fb9e1fed700 1 --2- 172.21.15.42:0/2588418061 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb9dc07e710 0x7fb9dc07eb40 unknown :-1 s=CLOSED pgs=3167 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:45:42.659 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:42.633+0000 7fb9e1fed700 1 -- 172.21.15.42:0/2588418061 shutdown_connections 2019-12-05T06:45:42.660 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:42.633+0000 7fb9e1fed700 1 -- 172.21.15.42:0/2588418061 wait complete. 2019-12-05T06:45:42.660 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:42.633+0000 7fb9e1fed700 1 -- 172.21.15.42:0/2588418061 >> 172.21.15.42:0/2588418061 conn(0x7fb9dc0aabd0 msgr2=0x7fb9dc0a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:45:42.661 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:42.637+0000 7fb9e1fed700 1 Processor -- start 2019-12-05T06:45:42.661 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:42.637+0000 7fb9e1fed700 1 -- start start 2019-12-05T06:45:42.662 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:42.637+0000 7fb9e1fed700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb9dc133570 0x7fb9dc119a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:45:42.662 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:42.637+0000 7fb9e1fed700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9dc11c200 con 0x7fb9dc11b5d0 2019-12-05T06:45:42.663 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:42.637+0000 7fb9e1fed700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9dc1165e0 con 0x7fb9dc1161b0 2019-12-05T06:45:42.663 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:42.637+0000 7fb9e1fed700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9dc119f70 con 0x7fb9dc133570 2019-12-05T06:45:42.663 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:42.637+0000 7fb9db7fe700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fb9dc1161b0 0x7fb9dc118650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:35732/0 (socket says 172.21.15.42:35732) 2019-12-05T06:45:42.663 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:42.637+0000 7fb9dbfff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb9dc133570 0x7fb9dc119a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:45:42.664 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:42.637+0000 7fb9db7fe700 1 -- 172.21.15.42:0/2961108640 learned_addr learned my addr 172.21.15.42:0/2961108640 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:45:42.664 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:42.641+0000 7fb9d8ff9700 1 -- 172.21.15.42:0/2961108640 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2807297091 0 0) 0x7fb9dc1165e0 con 0x7fb9dc1161b0 2019-12-05T06:45:42.666 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:42.641+0000 7fb9d8ff9700 1 -- 172.21.15.42:0/2961108640 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9c00030b0 con 0x7fb9dc1161b0 2019-12-05T06:45:42.666 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:42.641+0000 7fb9d8ff9700 1 -- 172.21.15.42:0/2961108640 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3540828376 0 0) 0x7fb9dc11c200 con 0x7fb9dc11b5d0 2019-12-05T06:45:42.667 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:42.641+0000 7fb9d8ff9700 1 -- 172.21.15.42:0/2961108640 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9dc1165e0 con 0x7fb9dc11b5d0 2019-12-05T06:45:42.667 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:42.641+0000 7fb9dbfff700 1 -- 172.21.15.42:0/2961108640 >> v1:172.21.15.42:6789/0 conn(0x7fb9dc1161b0 legacy=0x7fb9dc118650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:45:42.667 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:42.641+0000 7fb9dbfff700 1 -- 172.21.15.42:0/2961108640 >> v1:172.21.15.40:6789/0 conn(0x7fb9dc11b5d0 legacy=0x7fb9dc11ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:45:42.667 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:42.641+0000 7fb9dbfff700 1 -- 172.21.15.42:0/2961108640 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9dc134420 con 0x7fb9dc133570 2019-12-05T06:45:42.668 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:42.641+0000 7fb9dbfff700 1 --2- 172.21.15.42:0/2961108640 >> v2:172.21.15.42:3300/0 conn(0x7fb9dc133570 0x7fb9dc119a20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fb9cc0032b0 tx=0x7fb9cc010c70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:42.668 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:42.641+0000 7fb9dbfff700 1 --2- 172.21.15.42:0/2961108640 >> v2:172.21.15.42:3300/0 conn(0x7fb9dc133570 0x7fb9dc119a20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fb9cc0032b0 tx=0x7fb9cc010c70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:42.668 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:42.641+0000 7fb9dbfff700 1 --2- 172.21.15.42:0/2961108640 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb9dc133570 0x7fb9dc119a20 secure :-1 s=READY pgs=3169 cs=0 l=1 rx=0x7fb9cc0032b0 tx=0x7fb9cc010c70).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:45:42.668 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:42.641+0000 7fb9dbfff700 1 --2- 172.21.15.42:0/2961108640 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb9dc133570 0x7fb9dc119a20 secure :-1 s=THROTTLE_DONE pgs=3169 cs=0 l=1 rx=0x7fb9cc0032b0 tx=0x7fb9cc010c70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:42.669 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:42.641+0000 7fb9d8ff9700 1 -- 172.21.15.42:0/2961108640 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb9cc005ca0 con 0x7fb9dc133570 2019-12-05T06:45:42.669 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:42.641+0000 7fb9e1fed700 1 -- 172.21.15.42:0/2961108640 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9dc1173d0 con 0x7fb9dc133570 2019-12-05T06:45:42.669 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:42.641+0000 7fb9dbfff700 1 --2- 172.21.15.42:0/2961108640 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb9dc133570 0x7fb9dc119a20 secure :-1 s=THROTTLE_DONE pgs=3169 cs=0 l=1 rx=0x7fb9cc0032b0 tx=0x7fb9cc010c70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:42.669 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:42.641+0000 7fb9dbfff700 1 --2- 172.21.15.42:0/2961108640 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb9dc133570 0x7fb9dc119a20 secure :-1 s=THROTTLE_DONE pgs=3169 cs=0 l=1 rx=0x7fb9cc0032b0 tx=0x7fb9cc010c70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:42.670 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:42.641+0000 7fb9e1fed700 1 -- 172.21.15.42:0/2961108640 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9dc078300 con 0x7fb9dc133570 2019-12-05T06:45:42.670 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:42.641+0000 7fb9d8ff9700 1 -- 172.21.15.42:0/2961108640 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9cc0058c0 con 0x7fb9dc133570 2019-12-05T06:45:42.670 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:42.641+0000 7fb9d8ff9700 1 -- 172.21.15.42:0/2961108640 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb9cc007530 con 0x7fb9dc133570 2019-12-05T06:45:42.672 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:42.641+0000 7fb9dbfff700 1 --2- 172.21.15.42:0/2961108640 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb9dc133570 0x7fb9dc119a20 secure :-1 s=THROTTLE_DONE pgs=3169 cs=0 l=1 rx=0x7fb9cc0032b0 tx=0x7fb9cc010c70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:42.672 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:42.641+0000 7fb9d8ff9700 1 -- 172.21.15.42:0/2961108640 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fb9cc0127a0 con 0x7fb9dc133570 2019-12-05T06:45:42.672 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:42.641+0000 7fb9dbfff700 1 --2- 172.21.15.42:0/2961108640 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb9dc133570 0x7fb9dc119a20 secure :-1 s=THROTTLE_DONE pgs=3169 cs=0 l=1 rx=0x7fb9cc0032b0 tx=0x7fb9cc010c70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:42.672 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:42.641+0000 7fb9d8ff9700 1 --2- 172.21.15.42:0/2961108640 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb9c003b4a0 0x7fb9c003d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:45:42.673 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:42.641+0000 7fb9d8ff9700 1 -- 172.21.15.42:0/2961108640 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 11965+0+0 (secure 0 0 0) 0x7fb9cc04b560 con 0x7fb9dc133570 2019-12-05T06:45:42.673 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:42.641+0000 7fb9db7fe700 1 --2- 172.21.15.42:0/2961108640 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb9c003b4a0 0x7fb9c003d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:45:42.674 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:42.641+0000 7fb9db7fe700 1 --2- 172.21.15.42:0/2961108640 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb9c003b4a0 0x7fb9c003d9b0 crc :-1 s=READY pgs=1326 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:45:42.675 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:42.645+0000 7fb9e1fed700 1 -- 172.21.15.42:0/2961108640 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9b4003200 con 0x7fb9dc133570 2019-12-05T06:45:42.675 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:42.653+0000 7fb9dbfff700 1 --2- 172.21.15.42:0/2961108640 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb9dc133570 0x7fb9dc119a20 secure :-1 s=THROTTLE_DONE pgs=3169 cs=0 l=1 rx=0x7fb9cc0032b0 tx=0x7fb9cc010c70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:42.675 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:42.653+0000 7fb9d8ff9700 1 -- 172.21.15.42:0/2961108640 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fb9cc032580 con 0x7fb9dc133570 2019-12-05T06:45:42.832 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:42.825+0000 7fb9e1fed700 1 -- 172.21.15.42:0/2961108640 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb9b4003630 con 0x7fb9dc133570 2019-12-05T06:45:42.840 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:42.825+0000 7fb9dbfff700 1 --2- 172.21.15.42:0/2961108640 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb9dc133570 0x7fb9dc119a20 secure :-1 s=THROTTLE_DONE pgs=3169 cs=0 l=1 rx=0x7fb9cc0032b0 tx=0x7fb9cc010c70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:42.840 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:42.825+0000 7fb9d8ff9700 1 -- 172.21.15.42:0/2961108640 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 383 v383) v1 ==== 95+0+2117 (secure 0 0 0) 0x7fb9cc0327e0 con 0x7fb9dc133570 2019-12-05T06:45:42.840 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 383 2019-12-05T06:45:42.840 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:45:42.841 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":383,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":12429,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/4178556019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":4178556019},{"type":"v1","addr":"172.21.15.40:6833","nonce":4178556019}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":383}],"filesystems":[{"mdsmap":{"epoch":382,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:44:34.328051+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":118,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12303},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12303":{"gid":12303,"name":"a","rank":0,"incarnation":378,"state":"up:active","state_seq":17,"addr":"172.21.15.42:6835/2489540211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2489540211},{"type":"v1","addr":"172.21.15.42:6835","nonce":2489540211}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:45:42.842 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:42.829+0000 7fb9e1fed700 1 -- 172.21.15.42:0/2961108640 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb9c003b4a0 msgr2=0x7fb9c003d9b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:45:42.842 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:42.829+0000 7fb9e1fed700 1 --2- 172.21.15.42:0/2961108640 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb9c003b4a0 0x7fb9c003d9b0 crc :-1 s=READY pgs=1326 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:45:42.842 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:42.829+0000 7fb9e1fed700 1 -- 172.21.15.42:0/2961108640 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb9dc133570 msgr2=0x7fb9dc119a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:45:42.851 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:42.829+0000 7fb9e1fed700 1 --2- 172.21.15.42:0/2961108640 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb9dc133570 0x7fb9dc119a20 secure :-1 s=READY pgs=3169 cs=0 l=1 rx=0x7fb9cc0032b0 tx=0x7fb9cc010c70).stop 2019-12-05T06:45:42.851 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:42.829+0000 7fb9e1fed700 1 -- 172.21.15.42:0/2961108640 shutdown_connections 2019-12-05T06:45:42.851 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:42.829+0000 7fb9e1fed700 1 --2- 172.21.15.42:0/2961108640 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb9c003b4a0 0x7fb9c003d9b0 unknown :-1 s=CLOSED pgs=1326 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:45:42.851 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:42.829+0000 7fb9e1fed700 1 --2- 172.21.15.42:0/2961108640 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb9dc133570 0x7fb9dc119a20 unknown :-1 s=CLOSED pgs=3169 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:45:42.851 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:42.829+0000 7fb9e1fed700 1 -- 172.21.15.42:0/2961108640 shutdown_connections 2019-12-05T06:45:42.852 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:42.829+0000 7fb9e1fed700 1 -- 172.21.15.42:0/2961108640 wait complete. 2019-12-05T06:45:42.852 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:42.829+0000 7fb9e1fed700 1 -- 172.21.15.42:0/2961108640 >> 172.21.15.42:0/2961108640 conn(0x7fb9dc0a8290 msgr2=0x7fb9dc0a9780 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:45:42.853 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:45:44.861 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:45:44.969 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:44.957+0000 7f1a2f7ea700 1 Processor -- start 2019-12-05T06:45:44.969 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:44.957+0000 7f1a2f7ea700 1 -- start start 2019-12-05T06:45:44.970 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:44.961+0000 7f1a2f7ea700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1a2808df70 0x7f1a2809f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:45:44.970 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:44.961+0000 7f1a2f7ea700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a2809fc50 con 0x7f1a280a7730 2019-12-05T06:45:44.970 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:44.961+0000 7f1a2f7ea700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a280a0700 con 0x7f1a2809f820 2019-12-05T06:45:44.970 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:44.961+0000 7f1a2f7ea700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a280a0af0 con 0x7f1a2808df70 2019-12-05T06:45:44.970 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:44.961+0000 7f1a2e588700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f1a2809f820 0x7f1a2814efd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:35744/0 (socket says 172.21.15.42:35744) 2019-12-05T06:45:44.970 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:44.961+0000 7f1a2e588700 1 -- 172.21.15.42:0/178299039 learned_addr learned my addr 172.21.15.42:0/178299039 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:45:44.971 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:44.961+0000 7f1a2d586700 1 --2- 172.21.15.42:0/178299039 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1a2808df70 0x7f1a2809f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:45:44.971 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:44.961+0000 7f1a2cd85700 1 -- 172.21.15.42:0/178299039 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2108819850 0 0) 0x7f1a280a0700 con 0x7f1a2809f820 2019-12-05T06:45:44.971 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:44.961+0000 7f1a2cd85700 1 -- 172.21.15.42:0/178299039 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a140030b0 con 0x7f1a2809f820 2019-12-05T06:45:44.971 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:44.961+0000 7f1a2cd85700 1 -- 172.21.15.42:0/178299039 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 593959443 0 0) 0x7f1a2809fc50 con 0x7f1a280a7730 2019-12-05T06:45:44.971 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:44.961+0000 7f1a2cd85700 1 -- 172.21.15.42:0/178299039 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a280a0700 con 0x7f1a280a7730 2019-12-05T06:45:44.971 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:44.961+0000 7f1a2d586700 1 -- 172.21.15.42:0/178299039 >> v1:172.21.15.42:6789/0 conn(0x7f1a2809f820 legacy=0x7f1a2814efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:45:44.972 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:44.961+0000 7f1a2d586700 1 -- 172.21.15.42:0/178299039 >> v1:172.21.15.40:6789/0 conn(0x7f1a280a7730 legacy=0x7f1a2807e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:45:44.972 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:44.961+0000 7f1a2d586700 1 -- 172.21.15.42:0/178299039 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a2807ef10 con 0x7f1a2808df70 2019-12-05T06:45:44.972 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:44.961+0000 7f1a2d586700 1 --2- 172.21.15.42:0/178299039 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1a2808df70 0x7f1a2809f2d0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f1a1c0032b0 tx=0x7f1a1c00ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:44.972 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:44.961+0000 7f1a2d586700 1 --2- 172.21.15.42:0/178299039 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1a2808df70 0x7f1a2809f2d0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f1a1c0032b0 tx=0x7f1a1c00ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:44.972 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:44.961+0000 7f1a2d586700 1 --2- 172.21.15.42:0/178299039 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1a2808df70 0x7f1a2809f2d0 secure :-1 s=READY pgs=699 cs=0 l=1 rx=0x7f1a1c0032b0 tx=0x7f1a1c00ff70).ready entity=mon.2 client_cookie=3c738dfa1b0efb38 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:45:44.973 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:44.961+0000 7f1a2d586700 1 --2- 172.21.15.42:0/178299039 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1a2808df70 0x7f1a2809f2d0 secure :-1 s=THROTTLE_DONE pgs=699 cs=0 l=1 rx=0x7f1a1c0032b0 tx=0x7f1a1c00ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:44.973 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:44.961+0000 7f1a2cd85700 1 -- 172.21.15.42:0/178299039 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1a1c011a10 con 0x7f1a2808df70 2019-12-05T06:45:44.973 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:44.961+0000 7f1a2d586700 1 --2- 172.21.15.42:0/178299039 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1a2808df70 0x7f1a2809f2d0 secure :-1 s=THROTTLE_DONE pgs=699 cs=0 l=1 rx=0x7f1a1c0032b0 tx=0x7f1a1c00ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:44.973 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:44.961+0000 7f1a2d586700 1 --2- 172.21.15.42:0/178299039 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1a2808df70 0x7f1a2809f2d0 secure :-1 s=THROTTLE_DONE pgs=699 cs=0 l=1 rx=0x7f1a1c0032b0 tx=0x7f1a1c00ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:44.973 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:44.961+0000 7f1a2cd85700 1 -- 172.21.15.42:0/178299039 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a1c011630 con 0x7f1a2808df70 2019-12-05T06:45:44.973 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:44.961+0000 7f1a2cd85700 1 -- 172.21.15.42:0/178299039 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1a1c00a160 con 0x7f1a2808df70 2019-12-05T06:45:44.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:44.961+0000 7f1a2f7ea700 1 -- 172.21.15.42:0/178299039 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1a2808df70 msgr2=0x7f1a2809f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:45:44.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:44.961+0000 7f1a2f7ea700 1 --2- 172.21.15.42:0/178299039 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1a2808df70 0x7f1a2809f2d0 secure :-1 s=READY pgs=699 cs=0 l=1 rx=0x7f1a1c0032b0 tx=0x7f1a1c00ff70).stop 2019-12-05T06:45:44.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:44.965+0000 7f1a2f7ea700 1 -- 172.21.15.42:0/178299039 shutdown_connections 2019-12-05T06:45:44.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:44.965+0000 7f1a2f7ea700 1 --2- 172.21.15.42:0/178299039 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1a2808df70 0x7f1a2809f2d0 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:45:44.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:44.965+0000 7f1a2f7ea700 1 -- 172.21.15.42:0/178299039 shutdown_connections 2019-12-05T06:45:44.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:44.965+0000 7f1a2f7ea700 1 -- 172.21.15.42:0/178299039 wait complete. 2019-12-05T06:45:44.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:44.965+0000 7f1a2f7ea700 1 -- 172.21.15.42:0/178299039 >> 172.21.15.42:0/178299039 conn(0x7f1a280aabd0 msgr2=0x7f1a280a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:45:44.980 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:44.965+0000 7f1a2f7ea700 1 Processor -- start 2019-12-05T06:45:44.980 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:44.965+0000 7f1a2f7ea700 1 -- start start 2019-12-05T06:45:44.980 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:44.965+0000 7f1a2f7ea700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f1a281447a0 0x7f1a280797f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:45:44.981 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:44.965+0000 7f1a2f7ea700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a2814f8c0 con 0x7f1a280a3180 2019-12-05T06:45:44.981 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:44.965+0000 7f1a2f7ea700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a2812ec20 con 0x7f1a281221f0 2019-12-05T06:45:44.981 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:44.965+0000 7f1a2f7ea700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a28144cb0 con 0x7f1a281447a0 2019-12-05T06:45:44.981 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:44.965+0000 7f1a2e588700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f1a281221f0 0x7f1a28122620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:35746/0 (socket says 172.21.15.42:35746) 2019-12-05T06:45:44.981 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:44.965+0000 7f1a2e588700 1 -- 172.21.15.42:0/3267230035 learned_addr learned my addr 172.21.15.42:0/3267230035 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:45:44.981 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:44.965+0000 7f1a2dd87700 1 --2- 172.21.15.42:0/3267230035 >> v2:172.21.15.40:3300/0 conn(0x7f1a281447a0 0x7f1a280797f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:45:44.982 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:44.969+0000 7f1a1a7fc700 1 -- 172.21.15.42:0/3267230035 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4281920389 0 0) 0x7f1a2812ec20 con 0x7f1a281221f0 2019-12-05T06:45:44.982 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:44.969+0000 7f1a1a7fc700 1 -- 172.21.15.42:0/3267230035 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a100030b0 con 0x7f1a281221f0 2019-12-05T06:45:44.982 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:44.969+0000 7f1a2dd87700 1 -- 172.21.15.42:0/3267230035 >> v1:172.21.15.42:6789/0 conn(0x7f1a281221f0 legacy=0x7f1a28122620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:45:44.982 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:44.969+0000 7f1a2dd87700 1 -- 172.21.15.42:0/3267230035 >> v1:172.21.15.40:6789/0 conn(0x7f1a280a3180 legacy=0x7f1a2814f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:45:44.982 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:44.969+0000 7f1a2dd87700 1 -- 172.21.15.42:0/3267230035 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a2812fe60 con 0x7f1a281447a0 2019-12-05T06:45:44.982 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:44.969+0000 7f1a2dd87700 1 --2- 172.21.15.42:0/3267230035 >> v2:172.21.15.40:3300/0 conn(0x7f1a281447a0 0x7f1a280797f0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f1a24000e90 tx=0x7f1a2400c2f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:44.983 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:44.969+0000 7f1a2dd87700 1 --2- 172.21.15.42:0/3267230035 >> v2:172.21.15.40:3300/0 conn(0x7f1a281447a0 0x7f1a280797f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f1a24000e90 tx=0x7f1a2400c2f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:44.983 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:44.969+0000 7f1a2dd87700 1 --2- 172.21.15.42:0/3267230035 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1a281447a0 0x7f1a280797f0 secure :-1 s=READY pgs=2399 cs=0 l=1 rx=0x7f1a24000e90 tx=0x7f1a2400c2f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:45:44.983 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:44.969+0000 7f1a2dd87700 1 --2- 172.21.15.42:0/3267230035 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1a281447a0 0x7f1a280797f0 secure :-1 s=THROTTLE_DONE pgs=2399 cs=0 l=1 rx=0x7f1a24000e90 tx=0x7f1a2400c2f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:44.983 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:44.969+0000 7f1a1a7fc700 1 -- 172.21.15.42:0/3267230035 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1a2400caa0 con 0x7f1a281447a0 2019-12-05T06:45:44.983 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:44.969+0000 7f1a2f7ea700 1 -- 172.21.15.42:0/3267230035 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a2812dc10 con 0x7f1a281447a0 2019-12-05T06:45:44.983 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:44.969+0000 7f1a2dd87700 1 --2- 172.21.15.42:0/3267230035 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1a281447a0 0x7f1a280797f0 secure :-1 s=THROTTLE_DONE pgs=2399 cs=0 l=1 rx=0x7f1a24000e90 tx=0x7f1a2400c2f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:44.984 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:44.969+0000 7f1a1a7fc700 1 -- 172.21.15.42:0/3267230035 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a2400c6c0 con 0x7f1a281447a0 2019-12-05T06:45:44.984 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:44.969+0000 7f1a2dd87700 1 --2- 172.21.15.42:0/3267230035 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1a281447a0 0x7f1a280797f0 secure :-1 s=THROTTLE_DONE pgs=2399 cs=0 l=1 rx=0x7f1a24000e90 tx=0x7f1a2400c2f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:44.984 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:44.969+0000 7f1a1a7fc700 1 -- 172.21.15.42:0/3267230035 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1a2400d6a0 con 0x7f1a281447a0 2019-12-05T06:45:44.984 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:44.969+0000 7f1a2f7ea700 1 -- 172.21.15.42:0/3267230035 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a2812e1c0 con 0x7f1a281447a0 2019-12-05T06:45:44.984 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:44.969+0000 7f1a2dd87700 1 --2- 172.21.15.42:0/3267230035 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1a281447a0 0x7f1a280797f0 secure :-1 s=THROTTLE_DONE pgs=2399 cs=0 l=1 rx=0x7f1a24000e90 tx=0x7f1a2400c2f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:44.984 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:44.969+0000 7f1a1a7fc700 1 -- 172.21.15.42:0/3267230035 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f1a2400d9c0 con 0x7f1a281447a0 2019-12-05T06:45:44.985 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:44.969+0000 7f1a2dd87700 1 --2- 172.21.15.42:0/3267230035 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1a281447a0 0x7f1a280797f0 secure :-1 s=THROTTLE_DONE pgs=2399 cs=0 l=1 rx=0x7f1a24000e90 tx=0x7f1a2400c2f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:44.985 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:44.969+0000 7f1a1a7fc700 1 --2- 172.21.15.42:0/3267230035 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f1a1003b210 0x7f1a1003d720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:45:44.985 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:44.969+0000 7f1a1a7fc700 1 -- 172.21.15.42:0/3267230035 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 11965+0+0 (secure 0 0 0) 0x7f1a2404d450 con 0x7f1a281447a0 2019-12-05T06:45:44.985 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:44.973+0000 7f1a2d586700 1 --2- 172.21.15.42:0/3267230035 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f1a1003b210 0x7f1a1003d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:45:44.985 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:44.973+0000 7f1a2d586700 1 --2- 172.21.15.42:0/3267230035 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f1a1003b210 0x7f1a1003d720 crc :-1 s=READY pgs=1327 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:45:44.985 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:44.977+0000 7f1a2f7ea700 1 -- 172.21.15.42:0/3267230035 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a28079d40 con 0x7f1a281447a0 2019-12-05T06:45:44.990 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:44.985+0000 7f1a2dd87700 1 --2- 172.21.15.42:0/3267230035 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1a281447a0 0x7f1a280797f0 secure :-1 s=THROTTLE_DONE pgs=2399 cs=0 l=1 rx=0x7f1a24000e90 tx=0x7f1a2400c2f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:44.990 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:44.985+0000 7f1a1a7fc700 1 -- 172.21.15.42:0/3267230035 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f1a240344b0 con 0x7f1a281447a0 2019-12-05T06:45:45.159 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:45.153+0000 7f1a2f7ea700 1 -- 172.21.15.42:0/3267230035 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1a2805b8b0 con 0x7f1a281447a0 2019-12-05T06:45:45.160 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:45.157+0000 7f1a2dd87700 1 --2- 172.21.15.42:0/3267230035 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1a281447a0 0x7f1a280797f0 secure :-1 s=THROTTLE_DONE pgs=2399 cs=0 l=1 rx=0x7f1a24000e90 tx=0x7f1a2400c2f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:45.160 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:45.157+0000 7f1a1a7fc700 1 -- 172.21.15.42:0/3267230035 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 383 v383) v1 ==== 95+0+2117 (secure 0 0 0) 0x7f1a24034710 con 0x7f1a281447a0 2019-12-05T06:45:45.162 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 383 2019-12-05T06:45:45.163 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:45:45.163 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":383,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":12429,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/4178556019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":4178556019},{"type":"v1","addr":"172.21.15.40:6833","nonce":4178556019}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":383}],"filesystems":[{"mdsmap":{"epoch":382,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:44:34.328051+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":118,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12303},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12303":{"gid":12303,"name":"a","rank":0,"incarnation":378,"state":"up:active","state_seq":17,"addr":"172.21.15.42:6835/2489540211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2489540211},{"type":"v1","addr":"172.21.15.42:6835","nonce":2489540211}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:45:45.167 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:45.161+0000 7f1a2f7ea700 1 -- 172.21.15.42:0/3267230035 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f1a1003b210 msgr2=0x7f1a1003d720 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:45:45.167 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:45.161+0000 7f1a2f7ea700 1 --2- 172.21.15.42:0/3267230035 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f1a1003b210 0x7f1a1003d720 crc :-1 s=READY pgs=1327 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:45:45.168 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:45.161+0000 7f1a2f7ea700 1 -- 172.21.15.42:0/3267230035 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1a281447a0 msgr2=0x7f1a280797f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:45:45.168 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:45.161+0000 7f1a2f7ea700 1 --2- 172.21.15.42:0/3267230035 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1a281447a0 0x7f1a280797f0 secure :-1 s=READY pgs=2399 cs=0 l=1 rx=0x7f1a24000e90 tx=0x7f1a2400c2f0).stop 2019-12-05T06:45:45.168 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:45.161+0000 7f1a2f7ea700 1 -- 172.21.15.42:0/3267230035 shutdown_connections 2019-12-05T06:45:45.168 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:45.161+0000 7f1a2f7ea700 1 --2- 172.21.15.42:0/3267230035 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f1a1003b210 0x7f1a1003d720 unknown :-1 s=CLOSED pgs=1327 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:45:45.169 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:45.161+0000 7f1a2f7ea700 1 --2- 172.21.15.42:0/3267230035 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1a281447a0 0x7f1a280797f0 unknown :-1 s=CLOSED pgs=2399 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:45:45.169 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:45.161+0000 7f1a2f7ea700 1 -- 172.21.15.42:0/3267230035 shutdown_connections 2019-12-05T06:45:45.169 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:45.161+0000 7f1a2f7ea700 1 -- 172.21.15.42:0/3267230035 wait complete. 2019-12-05T06:45:45.169 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:45.161+0000 7f1a2f7ea700 1 -- 172.21.15.42:0/3267230035 >> 172.21.15.42:0/3267230035 conn(0x7f1a280a7f80 msgr2=0x7f1a280a6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:45:45.176 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:45:47.177 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:45:47.487 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:47.269+0000 7f030709d700 1 Processor -- start 2019-12-05T06:45:47.487 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:47.269+0000 7f030709d700 1 -- start start 2019-12-05T06:45:47.487 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:47.273+0000 7f030709d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f03000a7740 0x7f030008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:45:47.488 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:47.273+0000 7f030709d700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03000a0710 con 0x7f030007e720 2019-12-05T06:45:47.488 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:47.273+0000 7f030709d700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03000a9ba0 con 0x7f030009fae0 2019-12-05T06:45:47.488 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:47.273+0000 7f030709d700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03000a0b00 con 0x7f03000a7740 2019-12-05T06:45:47.488 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:47.273+0000 7f030563a700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f03000a7740 0x7f030008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:45:47.488 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:47.273+0000 7f0305e3b700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f030009fae0 0x7f030014efe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:35762/0 (socket says 172.21.15.42:35762) 2019-12-05T06:45:47.488 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:47.273+0000 7f0305e3b700 1 -- 172.21.15.42:0/3408793575 learned_addr learned my addr 172.21.15.42:0/3408793575 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:45:47.489 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:47.273+0000 7f030563a700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f03000a7740 0x7f030008df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:35720/0 (socket says 172.21.15.42:35720) 2019-12-05T06:45:47.489 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:47.273+0000 7f030563a700 1 -- 172.21.15.42:0/3408793575 >> v1:172.21.15.42:6789/0 conn(0x7f030009fae0 legacy=0x7f030014efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:45:47.489 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:47.273+0000 7f030563a700 1 -- 172.21.15.42:0/3408793575 >> v1:172.21.15.40:6789/0 conn(0x7f030007e720 legacy=0x7f030009f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:45:47.489 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:47.273+0000 7f030563a700 1 -- 172.21.15.42:0/3408793575 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f030007eb50 con 0x7f03000a7740 2019-12-05T06:45:47.489 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:47.273+0000 7f030563a700 1 --2- 172.21.15.42:0/3408793575 >> v2:172.21.15.42:3300/0 conn(0x7f03000a7740 0x7f030008df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f02fc008790 tx=0x7f02fc00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:47.489 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:47.273+0000 7f030563a700 1 -- 172.21.15.42:0/3408793575 >> v2:172.21.15.42:3300/0 conn(0x7f03000a7740 msgr2=0x7f030008df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2019-12-05T06:45:47.489 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:47.273+0000 7f030563a700 1 -- 172.21.15.42:0/3408793575 >> v2:172.21.15.42:3300/0 conn(0x7f03000a7740 msgr2=0x7f030008df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2019-12-05T06:45:47.490 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:47.273+0000 7f030563a700 1 --2- 172.21.15.42:0/3408793575 >> v2:172.21.15.42:3300/0 conn(0x7f03000a7740 0x7f030008df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f02fc008790 tx=0x7f02fc00fbf0).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-12-05T06:45:47.490 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:47.273+0000 7f030563a700 1 --2- 172.21.15.42:0/3408793575 >> v2:172.21.15.42:3300/0 conn(0x7f03000a7740 0x7f030008df80 secure :-1 s=START_CONNECT pgs=0 cs=0 l=0 rx=0x7f02fc008790 tx=0x7f02fc00fbf0)._fault waiting 0.200000 2019-12-05T06:45:47.490 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:47.477+0000 7f030563a700 1 --2- 172.21.15.42:0/3408793575 >> v2:172.21.15.42:3300/0 conn(0x7f03000a7740 0x7f030008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:45:47.490 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:47.477+0000 7f030563a700 0 --2- 172.21.15.42:0/3408793575 >> v2:172.21.15.42:3300/0 conn(0x7f03000a7740 0x7f030008df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2019-12-05T06:45:47.490 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:47.477+0000 7f030563a700 1 --2- 172.21.15.42:0/3408793575 >> v2:172.21.15.42:3300/0 conn(0x7f03000a7740 0x7f030008df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:45:47.490 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:47.477+0000 7f02f7fff700 1 -- 172.21.15.42:0/3408793575 >> v2:172.21.15.42:3300/0 conn(0x7f03000a7740 msgr2=0x7f030008df80 unknown :-1 s=STATE_CLOSED l=0).mark_down 2019-12-05T06:45:47.490 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:47.477+0000 7f02f7fff700 1 --2- 172.21.15.42:0/3408793575 >> v2:172.21.15.42:3300/0 conn(0x7f03000a7740 0x7f030008df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:45:47.490 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:47.477+0000 7f02f7fff700 1 --2- 172.21.15.42:0/3408793575 >> v2:172.21.15.40:3300/0 conn(0x7f02ec003090 0x7f02ec005560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:45:47.491 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:47.477+0000 7f02f7fff700 1 --2- 172.21.15.42:0/3408793575 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f02ec005c80 0x7f02ec008190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:45:47.491 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:47.477+0000 7f02f7fff700 1 -- 172.21.15.42:0/3408793575 --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02ec008880 con 0x7f02ec003090 2019-12-05T06:45:47.491 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:47.477+0000 7f02f7fff700 1 -- 172.21.15.42:0/3408793575 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02ec008a70 con 0x7f02ec005c80 2019-12-05T06:45:47.491 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:47.477+0000 7f030563a700 1 --2- 172.21.15.42:0/3408793575 >> v2:172.21.15.40:3300/0 conn(0x7f02ec003090 0x7f02ec005560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:45:47.491 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:47.477+0000 7f0304e39700 1 --2- 172.21.15.42:0/3408793575 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f02ec005c80 0x7f02ec008190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:45:47.491 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:47.477+0000 7f030563a700 1 -- 172.21.15.42:0/3408793575 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f02ec005c80 msgr2=0x7f02ec008190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:45:47.491 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:47.477+0000 7f030563a700 1 --2- 172.21.15.42:0/3408793575 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f02ec005c80 0x7f02ec008190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:45:47.492 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:47.477+0000 7f030563a700 1 -- 172.21.15.42:0/3408793575 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02ec04cd80 con 0x7f02ec003090 2019-12-05T06:45:47.492 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:47.477+0000 7f030563a700 1 --2- 172.21.15.42:0/3408793575 >> v2:172.21.15.40:3300/0 conn(0x7f02ec003090 0x7f02ec005560 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f02fc002f30 tx=0x7f02fc008f10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:47.492 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:47.477+0000 7f030563a700 1 --2- 172.21.15.42:0/3408793575 >> v2:172.21.15.40:3300/0 conn(0x7f02ec003090 0x7f02ec005560 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f02fc002f30 tx=0x7f02fc008f10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:47.492 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:47.477+0000 7f030563a700 1 --2- 172.21.15.42:0/3408793575 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f02ec003090 0x7f02ec005560 secure :-1 s=READY pgs=2401 cs=0 l=1 rx=0x7f02fc002f30 tx=0x7f02fc008f10).ready entity=mon.0 client_cookie=e4129324eeadf221 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:45:47.509 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:47.477+0000 7f030563a700 1 --2- 172.21.15.42:0/3408793575 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f02ec003090 0x7f02ec005560 secure :-1 s=THROTTLE_DONE pgs=2401 cs=0 l=1 rx=0x7f02fc002f30 tx=0x7f02fc008f10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:47.509 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:47.477+0000 7f030563a700 1 --2- 172.21.15.42:0/3408793575 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f02ec003090 0x7f02ec005560 secure :-1 s=THROTTLE_DONE pgs=2401 cs=0 l=1 rx=0x7f02fc002f30 tx=0x7f02fc008f10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:47.509 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:47.477+0000 7f030563a700 1 --2- 172.21.15.42:0/3408793575 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f02ec003090 0x7f02ec005560 secure :-1 s=THROTTLE_DONE pgs=2401 cs=0 l=1 rx=0x7f02fc002f30 tx=0x7f02fc008f10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:47.509 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:47.477+0000 7f02f7fff700 1 -- 172.21.15.42:0/3408793575 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f02fc0133f0 con 0x7f02ec003090 2019-12-05T06:45:47.509 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:47.477+0000 7f02f7fff700 1 -- 172.21.15.42:0/3408793575 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02fc00f480 con 0x7f02ec003090 2019-12-05T06:45:47.510 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:47.477+0000 7f02f7fff700 1 -- 172.21.15.42:0/3408793575 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f02fc0139d0 con 0x7f02ec003090 2019-12-05T06:45:47.510 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:47.477+0000 7f030709d700 1 -- 172.21.15.42:0/3408793575 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f02ec003090 msgr2=0x7f02ec005560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:45:47.510 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:47.477+0000 7f030709d700 1 --2- 172.21.15.42:0/3408793575 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f02ec003090 0x7f02ec005560 secure :-1 s=READY pgs=2401 cs=0 l=1 rx=0x7f02fc002f30 tx=0x7f02fc008f10).stop 2019-12-05T06:45:47.510 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:47.477+0000 7f0305e3b700 1 -- 172.21.15.42:0/3408793575 reap_dead start 2019-12-05T06:45:47.510 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:47.481+0000 7f030709d700 1 -- 172.21.15.42:0/3408793575 shutdown_connections 2019-12-05T06:45:47.510 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:47.481+0000 7f030709d700 1 --2- 172.21.15.42:0/3408793575 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f02ec003090 0x7f02ec005560 unknown :-1 s=CLOSED pgs=2401 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:45:47.510 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:47.481+0000 7f030709d700 1 -- 172.21.15.42:0/3408793575 shutdown_connections 2019-12-05T06:45:47.511 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:47.481+0000 7f030709d700 1 -- 172.21.15.42:0/3408793575 wait complete. 2019-12-05T06:45:47.511 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:47.481+0000 7f030709d700 1 -- 172.21.15.42:0/3408793575 >> 172.21.15.42:0/3408793575 conn(0x7f03000aabe0 msgr2=0x7f03000a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:45:47.516 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:47.481+0000 7f030709d700 1 Processor -- start 2019-12-05T06:45:47.516 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:47.481+0000 7f030709d700 1 -- start start 2019-12-05T06:45:47.516 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:47.485+0000 7f030709d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0300116690 0x7f03000762f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:45:47.517 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:47.485+0000 7f030709d700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f030011a820 con 0x7f030011cd50 2019-12-05T06:45:47.517 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:47.485+0000 7f030709d700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f030011c2b0 con 0x7f0300142940 2019-12-05T06:45:47.517 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:47.485+0000 7f030709d700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0300142d70 con 0x7f0300116690 2019-12-05T06:45:47.517 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:47.485+0000 7f030563a700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f0300142940 0x7f03000a4b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:35770/0 (socket says 172.21.15.42:35770) 2019-12-05T06:45:47.517 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:47.485+0000 7f0305e3b700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0300116690 0x7f03000762f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:45:47.517 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:47.485+0000 7f030563a700 1 -- 172.21.15.42:0/2721759862 learned_addr learned my addr 172.21.15.42:0/2721759862 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:45:47.518 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:47.485+0000 7f02f5ffb700 1 -- 172.21.15.42:0/2721759862 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1050873224 0 0) 0x7f030011c2b0 con 0x7f0300142940 2019-12-05T06:45:47.518 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:47.485+0000 7f02f5ffb700 1 -- 172.21.15.42:0/2721759862 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02e00030b0 con 0x7f0300142940 2019-12-05T06:45:47.518 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:47.485+0000 7f02f5ffb700 1 -- 172.21.15.42:0/2721759862 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 101495347 0 0) 0x7f030011a820 con 0x7f030011cd50 2019-12-05T06:45:47.518 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:47.485+0000 7f02f5ffb700 1 -- 172.21.15.42:0/2721759862 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f030011c2b0 con 0x7f030011cd50 2019-12-05T06:45:47.518 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:47.485+0000 7f0305e3b700 1 -- 172.21.15.42:0/2721759862 >> v1:172.21.15.42:6789/0 conn(0x7f0300142940 legacy=0x7f03000a4b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:45:47.518 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:47.485+0000 7f0305e3b700 1 -- 172.21.15.42:0/2721759862 >> v1:172.21.15.40:6789/0 conn(0x7f030011cd50 legacy=0x7f03000a5320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:45:47.518 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:47.485+0000 7f0305e3b700 1 -- 172.21.15.42:0/2721759862 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03000a4840 con 0x7f0300116690 2019-12-05T06:45:47.518 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:47.485+0000 7f0305e3b700 1 --2- 172.21.15.42:0/2721759862 >> v2:172.21.15.42:3300/0 conn(0x7f0300116690 0x7f03000762f0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f030011d530 tx=0x7f02f80106d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:47.519 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:47.485+0000 7f0305e3b700 1 --2- 172.21.15.42:0/2721759862 >> v2:172.21.15.42:3300/0 conn(0x7f0300116690 0x7f03000762f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f030011d530 tx=0x7f02f80106d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:47.519 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:47.485+0000 7f0305e3b700 1 --2- 172.21.15.42:0/2721759862 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0300116690 0x7f03000762f0 secure :-1 s=READY pgs=3175 cs=0 l=1 rx=0x7f030011d530 tx=0x7f02f80106d0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:45:47.519 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:47.485+0000 7f0305e3b700 1 --2- 172.21.15.42:0/2721759862 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0300116690 0x7f03000762f0 secure :-1 s=THROTTLE_DONE pgs=3175 cs=0 l=1 rx=0x7f030011d530 tx=0x7f02f80106d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:47.519 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:47.485+0000 7f02f5ffb700 1 -- 172.21.15.42:0/2721759862 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f02f80060c0 con 0x7f0300116690 2019-12-05T06:45:47.519 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:47.485+0000 7f0305e3b700 1 --2- 172.21.15.42:0/2721759862 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0300116690 0x7f03000762f0 secure :-1 s=THROTTLE_DONE pgs=3175 cs=0 l=1 rx=0x7f030011d530 tx=0x7f02f80106d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:47.519 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:47.485+0000 7f02f5ffb700 1 -- 172.21.15.42:0/2721759862 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02f8005b50 con 0x7f0300116690 2019-12-05T06:45:47.519 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:47.485+0000 7f0305e3b700 1 --2- 172.21.15.42:0/2721759862 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0300116690 0x7f03000762f0 secure :-1 s=THROTTLE_DONE pgs=3175 cs=0 l=1 rx=0x7f030011d530 tx=0x7f02f80106d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:47.520 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:47.485+0000 7f030709d700 1 -- 172.21.15.42:0/2721759862 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f030011b690 con 0x7f0300116690 2019-12-05T06:45:47.520 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:47.485+0000 7f02f5ffb700 1 -- 172.21.15.42:0/2721759862 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f02f80066a0 con 0x7f0300116690 2019-12-05T06:45:47.520 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:47.485+0000 7f030709d700 1 -- 172.21.15.42:0/2721759862 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0300116ac0 con 0x7f0300116690 2019-12-05T06:45:47.520 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:47.489+0000 7f0305e3b700 1 --2- 172.21.15.42:0/2721759862 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0300116690 0x7f03000762f0 secure :-1 s=THROTTLE_DONE pgs=3175 cs=0 l=1 rx=0x7f030011d530 tx=0x7f02f80106d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:47.520 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:47.489+0000 7f02f5ffb700 1 -- 172.21.15.42:0/2721759862 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f02f80068f0 con 0x7f0300116690 2019-12-05T06:45:47.520 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:47.489+0000 7f0305e3b700 1 --2- 172.21.15.42:0/2721759862 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0300116690 0x7f03000762f0 secure :-1 s=THROTTLE_DONE pgs=3175 cs=0 l=1 rx=0x7f030011d530 tx=0x7f02f80106d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:47.520 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:47.489+0000 7f02f5ffb700 1 --2- 172.21.15.42:0/2721759862 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f02e003b460 0x7f02e003d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:45:47.521 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:47.489+0000 7f02f5ffb700 1 -- 172.21.15.42:0/2721759862 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 11965+0+0 (secure 0 0 0) 0x7f02f804ae50 con 0x7f0300116690 2019-12-05T06:45:47.521 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:47.489+0000 7f030563a700 1 --2- 172.21.15.42:0/2721759862 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f02e003b460 0x7f02e003d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:45:47.521 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:47.489+0000 7f030563a700 1 --2- 172.21.15.42:0/2721759862 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f02e003b460 0x7f02e003d970 crc :-1 s=READY pgs=1328 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:45:47.541 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:47.537+0000 7f030709d700 1 -- 172.21.15.42:0/2721759862 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0300142730 con 0x7f0300116690 2019-12-05T06:45:47.548 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:47.541+0000 7f0305e3b700 1 --2- 172.21.15.42:0/2721759862 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0300116690 0x7f03000762f0 secure :-1 s=THROTTLE_DONE pgs=3175 cs=0 l=1 rx=0x7f030011d530 tx=0x7f02f80106d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:47.548 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:47.545+0000 7f02f5ffb700 1 -- 172.21.15.42:0/2721759862 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f02f8031ed0 con 0x7f0300116690 2019-12-05T06:45:47.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:47.717+0000 7f030709d700 1 -- 172.21.15.42:0/2721759862 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f030005d110 con 0x7f0300116690 2019-12-05T06:45:47.724 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:47.717+0000 7f0305e3b700 1 --2- 172.21.15.42:0/2721759862 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0300116690 0x7f03000762f0 secure :-1 s=THROTTLE_DONE pgs=3175 cs=0 l=1 rx=0x7f030011d530 tx=0x7f02f80106d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:47.724 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:47.717+0000 7f02f5ffb700 1 -- 172.21.15.42:0/2721759862 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 383 v383) v1 ==== 95+0+2117 (secure 0 0 0) 0x7f02f8032130 con 0x7f0300116690 2019-12-05T06:45:47.724 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 383 2019-12-05T06:45:47.725 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:45:47.725 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":383,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":12429,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/4178556019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":4178556019},{"type":"v1","addr":"172.21.15.40:6833","nonce":4178556019}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":383}],"filesystems":[{"mdsmap":{"epoch":382,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:44:34.328051+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":118,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12303},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12303":{"gid":12303,"name":"a","rank":0,"incarnation":378,"state":"up:active","state_seq":17,"addr":"172.21.15.42:6835/2489540211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2489540211},{"type":"v1","addr":"172.21.15.42:6835","nonce":2489540211}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:45:47.727 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:47.721+0000 7f030709d700 1 -- 172.21.15.42:0/2721759862 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f02e003b460 msgr2=0x7f02e003d970 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:45:47.727 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:47.721+0000 7f030709d700 1 --2- 172.21.15.42:0/2721759862 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f02e003b460 0x7f02e003d970 crc :-1 s=READY pgs=1328 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:45:47.727 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:47.721+0000 7f030709d700 1 -- 172.21.15.42:0/2721759862 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0300116690 msgr2=0x7f03000762f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:45:47.727 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:47.721+0000 7f030709d700 1 --2- 172.21.15.42:0/2721759862 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0300116690 0x7f03000762f0 secure :-1 s=READY pgs=3175 cs=0 l=1 rx=0x7f030011d530 tx=0x7f02f80106d0).stop 2019-12-05T06:45:47.728 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:47.721+0000 7f030709d700 1 -- 172.21.15.42:0/2721759862 shutdown_connections 2019-12-05T06:45:47.728 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:47.721+0000 7f030709d700 1 --2- 172.21.15.42:0/2721759862 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f02e003b460 0x7f02e003d970 unknown :-1 s=CLOSED pgs=1328 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:45:47.728 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:47.721+0000 7f030709d700 1 --2- 172.21.15.42:0/2721759862 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0300116690 0x7f03000762f0 unknown :-1 s=CLOSED pgs=3175 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:45:47.729 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:47.721+0000 7f030709d700 1 -- 172.21.15.42:0/2721759862 shutdown_connections 2019-12-05T06:45:47.729 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:47.721+0000 7f030709d700 1 -- 172.21.15.42:0/2721759862 wait complete. 2019-12-05T06:45:47.729 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:47.721+0000 7f030709d700 1 -- 172.21.15.42:0/2721759862 >> 172.21.15.42:0/2721759862 conn(0x7f030008e4d0 msgr2=0x7f03000a6b60 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:45:47.749 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:45:48.390 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:45:48.394 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:45:49.750 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:45:49.854 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:49.845+0000 7f017dbf4700 1 Processor -- start 2019-12-05T06:45:49.854 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:49.845+0000 7f017dbf4700 1 -- start start 2019-12-05T06:45:49.854 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:49.845+0000 7f017dbf4700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f01780a7740 0x7f017808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:45:49.854 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:49.845+0000 7f017dbf4700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f017809fae0 0x7f017807eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:45:49.855 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:49.845+0000 7f017dbf4700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01780a0710 con 0x7f017807e720 2019-12-05T06:45:49.855 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:49.845+0000 7f017dbf4700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01780a0b00 con 0x7f017809fae0 2019-12-05T06:45:49.855 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:49.845+0000 7f017dbf4700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f017808e5b0 con 0x7f01780a7740 2019-12-05T06:45:49.855 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:49.845+0000 7f0177fff700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f01780a7740 0x7f017808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:45:49.855 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:49.845+0000 7f017c992700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f017809fae0 0x7f017807eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:45:49.855 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:49.845+0000 7f0177fff700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f01780a7740 0x7f017808df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:36208/0 (socket says 172.21.15.42:36208) 2019-12-05T06:45:49.856 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:49.845+0000 7f0177fff700 1 -- 172.21.15.42:0/591102000 learned_addr learned my addr 172.21.15.42:0/591102000 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:45:49.856 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:49.845+0000 7f0177fff700 1 -- 172.21.15.42:0/591102000 >> v2:172.21.15.40:3300/0 conn(0x7f017809fae0 msgr2=0x7f017807eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:45:49.856 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:49.845+0000 7f0177fff700 1 --2- 172.21.15.42:0/591102000 >> v2:172.21.15.40:3300/0 conn(0x7f017809fae0 0x7f017807eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:45:49.856 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:49.845+0000 7f0177fff700 1 -- 172.21.15.42:0/591102000 >> v1:172.21.15.40:6789/0 conn(0x7f017807e720 legacy=0x7f017809f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:45:49.859 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:49.845+0000 7f0177fff700 1 -- 172.21.15.42:0/591102000 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f017807f0a0 con 0x7f01780a7740 2019-12-05T06:45:49.862 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:49.845+0000 7f0177fff700 1 --2- 172.21.15.42:0/591102000 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f01780a7740 0x7f017808df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f0168008790 tx=0x7f016800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:49.862 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:49.845+0000 7f0177fff700 1 --2- 172.21.15.42:0/591102000 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f01780a7740 0x7f017808df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f0168008790 tx=0x7f016800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:49.862 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:49.845+0000 7f0177fff700 1 --2- 172.21.15.42:0/591102000 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f01780a7740 0x7f017808df80 secure :-1 s=READY pgs=700 cs=0 l=1 rx=0x7f0168008790 tx=0x7f016800fd30).ready entity=mon.2 client_cookie=ec376fee2650e93b server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:45:49.862 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:49.845+0000 7f0177fff700 1 --2- 172.21.15.42:0/591102000 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f01780a7740 0x7f017808df80 secure :-1 s=THROTTLE_DONE pgs=700 cs=0 l=1 rx=0x7f0168008790 tx=0x7f016800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:49.862 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:49.845+0000 7f0176ffd700 1 -- 172.21.15.42:0/591102000 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0168008c90 con 0x7f01780a7740 2019-12-05T06:45:49.862 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:49.845+0000 7f0177fff700 1 --2- 172.21.15.42:0/591102000 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f01780a7740 0x7f017808df80 secure :-1 s=THROTTLE_DONE pgs=700 cs=0 l=1 rx=0x7f0168008790 tx=0x7f016800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:49.864 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:49.845+0000 7f0176ffd700 1 -- 172.21.15.42:0/591102000 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01680088b0 con 0x7f01780a7740 2019-12-05T06:45:49.864 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:49.845+0000 7f0177fff700 1 --2- 172.21.15.42:0/591102000 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f01780a7740 0x7f017808df80 secure :-1 s=THROTTLE_DONE pgs=700 cs=0 l=1 rx=0x7f0168008790 tx=0x7f016800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:49.864 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:49.845+0000 7f0176ffd700 1 -- 172.21.15.42:0/591102000 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0168011b10 con 0x7f01780a7740 2019-12-05T06:45:49.864 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:49.845+0000 7f017dbf4700 1 -- 172.21.15.42:0/591102000 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f01780a7740 msgr2=0x7f017808df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:45:49.864 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:49.845+0000 7f017dbf4700 1 --2- 172.21.15.42:0/591102000 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f01780a7740 0x7f017808df80 secure :-1 s=READY pgs=700 cs=0 l=1 rx=0x7f0168008790 tx=0x7f016800fd30).stop 2019-12-05T06:45:49.865 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:49.849+0000 7f017dbf4700 1 -- 172.21.15.42:0/591102000 shutdown_connections 2019-12-05T06:45:49.866 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:49.849+0000 7f017dbf4700 1 --2- 172.21.15.42:0/591102000 >> v2:172.21.15.40:3300/0 conn(0x7f017809fae0 0x7f017807eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:45:49.866 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:49.849+0000 7f017dbf4700 1 --2- 172.21.15.42:0/591102000 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f01780a7740 0x7f017808df80 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:45:49.866 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:49.849+0000 7f017dbf4700 1 -- 172.21.15.42:0/591102000 shutdown_connections 2019-12-05T06:45:49.866 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:49.849+0000 7f017dbf4700 1 -- 172.21.15.42:0/591102000 wait complete. 2019-12-05T06:45:49.866 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:49.849+0000 7f017dbf4700 1 -- 172.21.15.42:0/591102000 >> 172.21.15.42:0/591102000 conn(0x7f01780aabe0 msgr2=0x7f01780a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:45:49.879 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:49.849+0000 7f017dbf4700 1 Processor -- start 2019-12-05T06:45:49.879 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:49.853+0000 7f017dbf4700 1 -- start start 2019-12-05T06:45:49.880 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:49.853+0000 7f017dbf4700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f01781187a0 0x7f017807bb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:45:49.880 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:49.853+0000 7f017dbf4700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f017807afe0 0x7f017811af50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:45:49.881 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:49.853+0000 7f017dbf4700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f017807c0d0 con 0x7f017807b410 2019-12-05T06:45:49.882 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:49.853+0000 7f017dbf4700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f017807d590 con 0x7f017807afe0 2019-12-05T06:45:49.884 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:49.853+0000 7f017dbf4700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f017807d830 con 0x7f01781187a0 2019-12-05T06:45:49.885 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:49.853+0000 7f0177fff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f01781187a0 0x7f017807bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:45:49.885 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:49.853+0000 7f0177fff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f01781187a0 0x7f017807bb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:35746/0 (socket says 172.21.15.42:35746) 2019-12-05T06:45:49.885 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:49.853+0000 7f0177fff700 1 -- 172.21.15.42:0/3541024212 learned_addr learned my addr 172.21.15.42:0/3541024212 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:45:49.885 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:49.853+0000 7f01777fe700 1 --2- 172.21.15.42:0/3541024212 >> v2:172.21.15.40:3300/0 conn(0x7f017807afe0 0x7f017811af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:45:49.886 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:49.853+0000 7f0177fff700 1 -- 172.21.15.42:0/3541024212 >> v2:172.21.15.40:3300/0 conn(0x7f017807afe0 msgr2=0x7f017811af50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:45:49.886 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:49.853+0000 7f0177fff700 1 --2- 172.21.15.42:0/3541024212 >> v2:172.21.15.40:3300/0 conn(0x7f017807afe0 0x7f017811af50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:45:49.886 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:49.853+0000 7f0177fff700 1 -- 172.21.15.42:0/3541024212 >> v1:172.21.15.40:6789/0 conn(0x7f017807b410 legacy=0x7f017811b4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:45:49.886 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:49.853+0000 7f0177fff700 1 -- 172.21.15.42:0/3541024212 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f017807d0f0 con 0x7f01781187a0 2019-12-05T06:45:49.886 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:49.853+0000 7f0177fff700 1 --2- 172.21.15.42:0/3541024212 >> v2:172.21.15.42:3300/0 conn(0x7f01781187a0 0x7f017807bb80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f0168008790 tx=0x7f0168008f40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:49.886 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:49.853+0000 7f0177fff700 1 --2- 172.21.15.42:0/3541024212 >> v2:172.21.15.42:3300/0 conn(0x7f01781187a0 0x7f017807bb80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f0168008790 tx=0x7f0168008f40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:49.887 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:49.853+0000 7f0177fff700 1 --2- 172.21.15.42:0/3541024212 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f01781187a0 0x7f017807bb80 secure :-1 s=READY pgs=3176 cs=0 l=1 rx=0x7f0168008790 tx=0x7f0168008f40).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:45:49.887 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:49.853+0000 7f0177fff700 1 --2- 172.21.15.42:0/3541024212 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f01781187a0 0x7f017807bb80 secure :-1 s=THROTTLE_DONE pgs=3176 cs=0 l=1 rx=0x7f0168008790 tx=0x7f0168008f40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:49.887 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:49.853+0000 7f0177fff700 1 --2- 172.21.15.42:0/3541024212 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f01781187a0 0x7f017807bb80 secure :-1 s=THROTTLE_DONE pgs=3176 cs=0 l=1 rx=0x7f0168008790 tx=0x7f0168008f40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:49.887 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:49.853+0000 7f0174ff9700 1 -- 172.21.15.42:0/3541024212 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f016800f470 con 0x7f01781187a0 2019-12-05T06:45:49.887 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:49.853+0000 7f0177fff700 1 --2- 172.21.15.42:0/3541024212 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f01781187a0 0x7f017807bb80 secure :-1 s=THROTTLE_DONE pgs=3176 cs=0 l=1 rx=0x7f0168008790 tx=0x7f0168008f40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:49.887 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:49.853+0000 7f0174ff9700 1 -- 172.21.15.42:0/3541024212 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0168011730 con 0x7f01781187a0 2019-12-05T06:45:49.887 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:49.853+0000 7f017dbf4700 1 -- 172.21.15.42:0/3541024212 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f017808eb70 con 0x7f01781187a0 2019-12-05T06:45:49.888 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:49.853+0000 7f0174ff9700 1 -- 172.21.15.42:0/3541024212 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f016805a810 con 0x7f01781187a0 2019-12-05T06:45:49.888 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:49.853+0000 7f017dbf4700 1 -- 172.21.15.42:0/3541024212 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f017808f120 con 0x7f01781187a0 2019-12-05T06:45:49.888 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:49.857+0000 7f0177fff700 1 --2- 172.21.15.42:0/3541024212 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f01781187a0 0x7f017807bb80 secure :-1 s=THROTTLE_DONE pgs=3176 cs=0 l=1 rx=0x7f0168008790 tx=0x7f0168008f40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:49.888 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:49.857+0000 7f0174ff9700 1 -- 172.21.15.42:0/3541024212 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f0168007180 con 0x7f01781187a0 2019-12-05T06:45:49.888 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:49.857+0000 7f0177fff700 1 --2- 172.21.15.42:0/3541024212 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f01781187a0 0x7f017807bb80 secure :-1 s=THROTTLE_DONE pgs=3176 cs=0 l=1 rx=0x7f0168008790 tx=0x7f0168008f40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:49.888 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:49.857+0000 7f0174ff9700 1 --2- 172.21.15.42:0/3541024212 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f015003ae10 0x7f015003d320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:45:49.888 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:49.857+0000 7f0174ff9700 1 -- 172.21.15.42:0/3541024212 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 11965+0+0 (secure 0 0 0) 0x7f01680915a0 con 0x7f01781187a0 2019-12-05T06:45:49.889 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:49.857+0000 7f01777fe700 1 --2- 172.21.15.42:0/3541024212 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f015003ae10 0x7f015003d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:45:49.889 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:49.857+0000 7f01777fe700 1 --2- 172.21.15.42:0/3541024212 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f015003ae10 0x7f015003d320 crc :-1 s=READY pgs=1329 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:45:49.889 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:49.861+0000 7f017dbf4700 1 -- 172.21.15.42:0/3541024212 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f017805d110 con 0x7f01781187a0 2019-12-05T06:45:49.889 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:49.869+0000 7f0177fff700 1 --2- 172.21.15.42:0/3541024212 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f01781187a0 0x7f017807bb80 secure :-1 s=THROTTLE_DONE pgs=3176 cs=0 l=1 rx=0x7f0168008790 tx=0x7f0168008f40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:49.889 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:49.869+0000 7f0174ff9700 1 -- 172.21.15.42:0/3541024212 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f016805ac20 con 0x7f01781187a0 2019-12-05T06:45:50.043 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:50.037+0000 7f017dbf4700 1 -- 172.21.15.42:0/3541024212 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f017805b8c0 con 0x7f01781187a0 2019-12-05T06:45:50.043 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:50.037+0000 7f0177fff700 1 --2- 172.21.15.42:0/3541024212 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f01781187a0 0x7f017807bb80 secure :-1 s=THROTTLE_DONE pgs=3176 cs=0 l=1 rx=0x7f0168008790 tx=0x7f0168008f40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:50.044 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:50.037+0000 7f0174ff9700 1 -- 172.21.15.42:0/3541024212 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 383 v383) v1 ==== 95+0+2117 (secure 0 0 0) 0x7f0168078880 con 0x7f01781187a0 2019-12-05T06:45:50.044 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 383 2019-12-05T06:45:50.044 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:45:50.045 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":383,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":12429,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/4178556019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":4178556019},{"type":"v1","addr":"172.21.15.40:6833","nonce":4178556019}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":383}],"filesystems":[{"mdsmap":{"epoch":382,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:44:34.328051+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":118,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12303},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12303":{"gid":12303,"name":"a","rank":0,"incarnation":378,"state":"up:active","state_seq":17,"addr":"172.21.15.42:6835/2489540211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2489540211},{"type":"v1","addr":"172.21.15.42:6835","nonce":2489540211}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:45:50.047 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:50.041+0000 7f017dbf4700 1 -- 172.21.15.42:0/3541024212 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f015003ae10 msgr2=0x7f015003d320 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:45:50.047 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:50.041+0000 7f017dbf4700 1 --2- 172.21.15.42:0/3541024212 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f015003ae10 0x7f015003d320 crc :-1 s=READY pgs=1329 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:45:50.047 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:50.041+0000 7f017dbf4700 1 -- 172.21.15.42:0/3541024212 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f01781187a0 msgr2=0x7f017807bb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:45:50.047 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:50.041+0000 7f017dbf4700 1 --2- 172.21.15.42:0/3541024212 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f01781187a0 0x7f017807bb80 secure :-1 s=READY pgs=3176 cs=0 l=1 rx=0x7f0168008790 tx=0x7f0168008f40).stop 2019-12-05T06:45:50.047 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:50.041+0000 7f017dbf4700 1 -- 172.21.15.42:0/3541024212 shutdown_connections 2019-12-05T06:45:50.047 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:50.041+0000 7f017dbf4700 1 --2- 172.21.15.42:0/3541024212 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f015003ae10 0x7f015003d320 unknown :-1 s=CLOSED pgs=1329 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:45:50.048 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:50.041+0000 7f017dbf4700 1 --2- 172.21.15.42:0/3541024212 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f01781187a0 0x7f017807bb80 unknown :-1 s=CLOSED pgs=3176 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:45:50.048 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:50.041+0000 7f017dbf4700 1 --2- 172.21.15.42:0/3541024212 >> v2:172.21.15.40:3300/0 conn(0x7f017807afe0 0x7f017811af50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:45:50.048 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:50.041+0000 7f017dbf4700 1 -- 172.21.15.42:0/3541024212 shutdown_connections 2019-12-05T06:45:50.048 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:50.041+0000 7f017dbf4700 1 -- 172.21.15.42:0/3541024212 wait complete. 2019-12-05T06:45:50.048 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:50.041+0000 7f017dbf4700 1 -- 172.21.15.42:0/3541024212 >> 172.21.15.42:0/3541024212 conn(0x7f01780aabe0 msgr2=0x7f01780a9790 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:45:50.056 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:45:52.058 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:45:52.166 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:52.157+0000 7f4863575700 1 Processor -- start 2019-12-05T06:45:52.166 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:52.157+0000 7f4863575700 1 -- start start 2019-12-05T06:45:52.168 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:52.157+0000 7f4863575700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f485c0a7740 0x7f485c08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:45:52.169 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:52.157+0000 7f4863575700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f485c07e720 0x7f485c07eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:45:52.169 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:52.157+0000 7f4863575700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f485c09f2e0 0x7f485c09f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:45:52.169 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:52.157+0000 7f4863575700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f485c0a0b00 con 0x7f485c07e720 2019-12-05T06:45:52.169 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:52.157+0000 7f4863575700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f485c08e5b0 con 0x7f485c0a7740 2019-12-05T06:45:52.169 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:52.157+0000 7f4863575700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f485c08e850 con 0x7f485c09f2e0 2019-12-05T06:45:52.170 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:52.157+0000 7f4861b12700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f485c0a7740 0x7f485c08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:45:52.170 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:52.157+0000 7f4861b12700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f485c0a7740 0x7f485c08df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:35756/0 (socket says 172.21.15.42:35756) 2019-12-05T06:45:52.170 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:52.157+0000 7f4861b12700 1 -- 172.21.15.42:0/2708287656 learned_addr learned my addr 172.21.15.42:0/2708287656 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:45:52.170 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:52.157+0000 7f4862313700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f485c09f2e0 0x7f485c09f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:45:52.170 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:52.157+0000 7f4861311700 1 --2- 172.21.15.42:0/2708287656 >> v2:172.21.15.40:3300/0 conn(0x7f485c07e720 0x7f485c07eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:45:52.170 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:52.157+0000 7f4861b12700 1 -- 172.21.15.42:0/2708287656 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f485c09f2e0 msgr2=0x7f485c09f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:45:52.170 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:52.157+0000 7f4861b12700 1 --2- 172.21.15.42:0/2708287656 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f485c09f2e0 0x7f485c09f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:45:52.171 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:52.157+0000 7f4861b12700 1 -- 172.21.15.42:0/2708287656 >> v2:172.21.15.40:3300/0 conn(0x7f485c07e720 msgr2=0x7f485c07eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:45:52.171 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:52.157+0000 7f4861b12700 1 --2- 172.21.15.42:0/2708287656 >> v2:172.21.15.40:3300/0 conn(0x7f485c07e720 0x7f485c07eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:45:52.171 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:52.157+0000 7f4861b12700 1 -- 172.21.15.42:0/2708287656 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f485c07f0a0 con 0x7f485c0a7740 2019-12-05T06:45:52.171 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:52.157+0000 7f4861b12700 1 --2- 172.21.15.42:0/2708287656 >> v2:172.21.15.42:3300/0 conn(0x7f485c0a7740 0x7f485c08df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f485c0a0f50 tx=0x7f485800fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:52.171 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:52.157+0000 7f4861b12700 1 --2- 172.21.15.42:0/2708287656 >> v2:172.21.15.42:3300/0 conn(0x7f485c0a7740 0x7f485c08df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f485c0a0f50 tx=0x7f485800fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:52.171 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:52.157+0000 7f4861b12700 1 --2- 172.21.15.42:0/2708287656 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f485c0a7740 0x7f485c08df80 secure :-1 s=READY pgs=3177 cs=0 l=1 rx=0x7f485c0a0f50 tx=0x7f485800fdb0).ready entity=mon.1 client_cookie=2f03f6769e449547 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:45:52.171 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:52.157+0000 7f4861b12700 1 --2- 172.21.15.42:0/2708287656 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f485c0a7740 0x7f485c08df80 secure :-1 s=THROTTLE_DONE pgs=3177 cs=0 l=1 rx=0x7f485c0a0f50 tx=0x7f485800fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:52.172 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:52.157+0000 7f4861b12700 1 --2- 172.21.15.42:0/2708287656 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f485c0a7740 0x7f485c08df80 secure :-1 s=THROTTLE_DONE pgs=3177 cs=0 l=1 rx=0x7f485c0a0f50 tx=0x7f485800fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:52.172 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:52.157+0000 7f4860b10700 1 -- 172.21.15.42:0/2708287656 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4858008c90 con 0x7f485c0a7740 2019-12-05T06:45:52.172 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:52.157+0000 7f4861b12700 1 --2- 172.21.15.42:0/2708287656 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f485c0a7740 0x7f485c08df80 secure :-1 s=THROTTLE_DONE pgs=3177 cs=0 l=1 rx=0x7f485c0a0f50 tx=0x7f485800fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:52.172 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:52.157+0000 7f4860b10700 1 -- 172.21.15.42:0/2708287656 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48580088b0 con 0x7f485c0a7740 2019-12-05T06:45:52.172 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:52.157+0000 7f4860b10700 1 -- 172.21.15.42:0/2708287656 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4858011b50 con 0x7f485c0a7740 2019-12-05T06:45:52.172 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:52.157+0000 7f4863575700 1 -- 172.21.15.42:0/2708287656 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f485c0a7740 msgr2=0x7f485c08df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:45:52.172 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:52.157+0000 7f4863575700 1 --2- 172.21.15.42:0/2708287656 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f485c0a7740 0x7f485c08df80 secure :-1 s=READY pgs=3177 cs=0 l=1 rx=0x7f485c0a0f50 tx=0x7f485800fdb0).stop 2019-12-05T06:45:52.173 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:52.157+0000 7f4863575700 1 -- 172.21.15.42:0/2708287656 shutdown_connections 2019-12-05T06:45:52.173 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:52.157+0000 7f4863575700 1 --2- 172.21.15.42:0/2708287656 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f485c09f2e0 0x7f485c09f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:45:52.173 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:52.157+0000 7f4863575700 1 --2- 172.21.15.42:0/2708287656 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f485c0a7740 0x7f485c08df80 unknown :-1 s=CLOSED pgs=3177 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:45:52.173 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:52.157+0000 7f4863575700 1 --2- 172.21.15.42:0/2708287656 >> v2:172.21.15.40:3300/0 conn(0x7f485c07e720 0x7f485c07eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:45:52.173 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:52.157+0000 7f4863575700 1 -- 172.21.15.42:0/2708287656 shutdown_connections 2019-12-05T06:45:52.173 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:52.157+0000 7f4863575700 1 -- 172.21.15.42:0/2708287656 wait complete. 2019-12-05T06:45:52.173 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:52.157+0000 7f4863575700 1 -- 172.21.15.42:0/2708287656 >> 172.21.15.42:0/2708287656 conn(0x7f485c0aabe0 msgr2=0x7f485c0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:45:52.174 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:52.161+0000 7f4863575700 1 Processor -- start 2019-12-05T06:45:52.174 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:52.161+0000 7f4863575700 1 -- start start 2019-12-05T06:45:52.174 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:52.161+0000 7f4863575700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f485c0a5cb0 0x7f485c11a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:45:52.174 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:52.161+0000 7f4863575700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f485c1222b0 con 0x7f485c120de0 2019-12-05T06:45:52.174 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:52.161+0000 7f4863575700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f485c073af0 con 0x7f485c11aa20 2019-12-05T06:45:52.174 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:52.161+0000 7f4863575700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f485c11b2a0 con 0x7f485c0a5cb0 2019-12-05T06:45:52.174 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:52.161+0000 7f4861b12700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f485c0a5cb0 0x7f485c11a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:45:52.175 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:52.161+0000 7f4861311700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f485c11aa20 0x7f485c1205e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:35800/0 (socket says 172.21.15.42:35800) 2019-12-05T06:45:52.175 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:52.161+0000 7f4861b12700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f485c0a5cb0 0x7f485c11a4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:35764/0 (socket says 172.21.15.42:35764) 2019-12-05T06:45:52.175 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:52.161+0000 7f4861311700 1 -- 172.21.15.42:0/1378459772 learned_addr learned my addr 172.21.15.42:0/1378459772 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:45:52.178 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:52.161+0000 7f4861b12700 1 -- 172.21.15.42:0/1378459772 >> v1:172.21.15.42:6789/0 conn(0x7f485c11aa20 legacy=0x7f485c1205e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:45:52.178 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:52.161+0000 7f4861b12700 1 -- 172.21.15.42:0/1378459772 >> v1:172.21.15.40:6789/0 conn(0x7f485c120de0 legacy=0x7f485c0732f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:45:52.178 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:52.161+0000 7f48527fc700 1 -- 172.21.15.42:0/1378459772 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2544023708 0 0) 0x7f485c073af0 con 0x7f485c11aa20 2019-12-05T06:45:52.178 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:52.161+0000 7f4861b12700 1 -- 172.21.15.42:0/1378459772 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f485c073d20 con 0x7f485c0a5cb0 2019-12-05T06:45:52.179 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:52.161+0000 7f4861b12700 1 --2- 172.21.15.42:0/1378459772 >> v2:172.21.15.42:3300/0 conn(0x7f485c0a5cb0 0x7f485c11a4d0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f485c120260 tx=0x7f48580088b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:52.179 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:52.161+0000 7f4861b12700 1 --2- 172.21.15.42:0/1378459772 >> v2:172.21.15.42:3300/0 conn(0x7f485c0a5cb0 0x7f485c11a4d0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f485c120260 tx=0x7f48580088b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:52.179 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:52.161+0000 7f4861b12700 1 --2- 172.21.15.42:0/1378459772 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f485c0a5cb0 0x7f485c11a4d0 secure :-1 s=READY pgs=3179 cs=0 l=1 rx=0x7f485c120260 tx=0x7f48580088b0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:45:52.179 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:52.161+0000 7f4861b12700 1 --2- 172.21.15.42:0/1378459772 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f485c0a5cb0 0x7f485c11a4d0 secure :-1 s=THROTTLE_DONE pgs=3179 cs=0 l=1 rx=0x7f485c120260 tx=0x7f48580088b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:52.180 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:52.161+0000 7f48527fc700 1 -- 172.21.15.42:0/1378459772 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f485800f280 con 0x7f485c0a5cb0 2019-12-05T06:45:52.180 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:52.161+0000 7f4863575700 1 -- 172.21.15.42:0/1378459772 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f485c073f90 con 0x7f485c0a5cb0 2019-12-05T06:45:52.180 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:52.161+0000 7f4861b12700 1 --2- 172.21.15.42:0/1378459772 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f485c0a5cb0 0x7f485c11a4d0 secure :-1 s=THROTTLE_DONE pgs=3179 cs=0 l=1 rx=0x7f485c120260 tx=0x7f48580088b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:52.180 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:52.161+0000 7f48527fc700 1 -- 172.21.15.42:0/1378459772 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4858011770 con 0x7f485c0a5cb0 2019-12-05T06:45:52.180 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:52.161+0000 7f4863575700 1 -- 172.21.15.42:0/1378459772 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f485c07b610 con 0x7f485c0a5cb0 2019-12-05T06:45:52.180 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:52.161+0000 7f4861b12700 1 --2- 172.21.15.42:0/1378459772 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f485c0a5cb0 0x7f485c11a4d0 secure :-1 s=THROTTLE_DONE pgs=3179 cs=0 l=1 rx=0x7f485c120260 tx=0x7f48580088b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:52.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:52.161+0000 7f48527fc700 1 -- 172.21.15.42:0/1378459772 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4858005540 con 0x7f485c0a5cb0 2019-12-05T06:45:52.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:52.165+0000 7f4861b12700 1 --2- 172.21.15.42:0/1378459772 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f485c0a5cb0 0x7f485c11a4d0 secure :-1 s=THROTTLE_DONE pgs=3179 cs=0 l=1 rx=0x7f485c120260 tx=0x7f48580088b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:52.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:52.165+0000 7f48527fc700 1 -- 172.21.15.42:0/1378459772 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f48580057e0 con 0x7f485c0a5cb0 2019-12-05T06:45:52.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:52.165+0000 7f4861b12700 1 --2- 172.21.15.42:0/1378459772 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f485c0a5cb0 0x7f485c11a4d0 secure :-1 s=THROTTLE_DONE pgs=3179 cs=0 l=1 rx=0x7f485c120260 tx=0x7f48580088b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:52.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:52.165+0000 7f48527fc700 1 --2- 172.21.15.42:0/1378459772 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f484403ac40 0x7f484403d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:45:52.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:52.165+0000 7f48527fc700 1 -- 172.21.15.42:0/1378459772 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 11965+0+0 (secure 0 0 0) 0x7f485808f2e0 con 0x7f485c0a5cb0 2019-12-05T06:45:52.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:52.165+0000 7f4861311700 1 --2- 172.21.15.42:0/1378459772 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f484403ac40 0x7f484403d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:45:52.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:52.165+0000 7f4861311700 1 --2- 172.21.15.42:0/1378459772 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f484403ac40 0x7f484403d150 crc :-1 s=READY pgs=1330 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:45:52.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:52.169+0000 7f4863575700 1 -- 172.21.15.42:0/1378459772 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f485c05d110 con 0x7f485c0a5cb0 2019-12-05T06:45:52.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:52.177+0000 7f4861b12700 1 --2- 172.21.15.42:0/1378459772 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f485c0a5cb0 0x7f485c11a4d0 secure :-1 s=THROTTLE_DONE pgs=3179 cs=0 l=1 rx=0x7f485c120260 tx=0x7f48580088b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:52.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:52.177+0000 7f48527fc700 1 -- 172.21.15.42:0/1378459772 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f4858076340 con 0x7f485c0a5cb0 2019-12-05T06:45:52.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:52.345+0000 7f4863575700 1 -- 172.21.15.42:0/1378459772 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f485c05b8c0 con 0x7f485c0a5cb0 2019-12-05T06:45:52.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:52.345+0000 7f4861b12700 1 --2- 172.21.15.42:0/1378459772 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f485c0a5cb0 0x7f485c11a4d0 secure :-1 s=THROTTLE_DONE pgs=3179 cs=0 l=1 rx=0x7f485c120260 tx=0x7f48580088b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:52.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:52.345+0000 7f48527fc700 1 -- 172.21.15.42:0/1378459772 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 383 v383) v1 ==== 95+0+2117 (secure 0 0 0) 0x7f48580765a0 con 0x7f485c0a5cb0 2019-12-05T06:45:52.353 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 383 2019-12-05T06:45:52.353 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:45:52.353 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":383,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":12429,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/4178556019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":4178556019},{"type":"v1","addr":"172.21.15.40:6833","nonce":4178556019}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":383}],"filesystems":[{"mdsmap":{"epoch":382,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:44:34.328051+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":118,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12303},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12303":{"gid":12303,"name":"a","rank":0,"incarnation":378,"state":"up:active","state_seq":17,"addr":"172.21.15.42:6835/2489540211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2489540211},{"type":"v1","addr":"172.21.15.42:6835","nonce":2489540211}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:45:52.374 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:52.353+0000 7f4863575700 1 -- 172.21.15.42:0/1378459772 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f484403ac40 msgr2=0x7f484403d150 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:45:52.374 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:52.353+0000 7f4863575700 1 --2- 172.21.15.42:0/1378459772 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f484403ac40 0x7f484403d150 crc :-1 s=READY pgs=1330 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:45:52.374 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:52.353+0000 7f4863575700 1 -- 172.21.15.42:0/1378459772 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f485c0a5cb0 msgr2=0x7f485c11a4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:45:52.375 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:52.353+0000 7f4863575700 1 --2- 172.21.15.42:0/1378459772 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f485c0a5cb0 0x7f485c11a4d0 secure :-1 s=READY pgs=3179 cs=0 l=1 rx=0x7f485c120260 tx=0x7f48580088b0).stop 2019-12-05T06:45:52.375 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:52.353+0000 7f4863575700 1 -- 172.21.15.42:0/1378459772 shutdown_connections 2019-12-05T06:45:52.375 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:52.353+0000 7f4863575700 1 --2- 172.21.15.42:0/1378459772 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f484403ac40 0x7f484403d150 unknown :-1 s=CLOSED pgs=1330 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:45:52.375 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:52.353+0000 7f4863575700 1 --2- 172.21.15.42:0/1378459772 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f485c0a5cb0 0x7f485c11a4d0 unknown :-1 s=CLOSED pgs=3179 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:45:52.375 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:52.353+0000 7f4863575700 1 -- 172.21.15.42:0/1378459772 shutdown_connections 2019-12-05T06:45:52.375 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:52.353+0000 7f4863575700 1 -- 172.21.15.42:0/1378459772 wait complete. 2019-12-05T06:45:52.376 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:52.353+0000 7f4863575700 1 -- 172.21.15.42:0/1378459772 >> 172.21.15.42:0/1378459772 conn(0x7f485c0a82a0 msgr2=0x7f485c0a9770 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:45:52.376 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:45:54.377 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:45:54.483 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:54.473+0000 7f59d46d1700 1 Processor -- start 2019-12-05T06:45:54.483 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:54.473+0000 7f59d46d1700 1 -- start start 2019-12-05T06:45:54.483 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:54.473+0000 7f59d46d1700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f59cc0a7740 0x7f59cc08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:45:54.484 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:54.473+0000 7f59d46d1700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f59cc09fae0 0x7f59cc07eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:45:54.484 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:54.473+0000 7f59d46d1700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59cc0a0710 con 0x7f59cc07e720 2019-12-05T06:45:54.484 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:54.473+0000 7f59d46d1700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59cc0a0b00 con 0x7f59cc09fae0 2019-12-05T06:45:54.484 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:54.473+0000 7f59d46d1700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59cc08e5b0 con 0x7f59cc0a7740 2019-12-05T06:45:54.485 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:54.473+0000 7f59d2c6e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f59cc0a7740 0x7f59cc08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:45:54.485 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:54.473+0000 7f59d2c6e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f59cc0a7740 0x7f59cc08df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:35776/0 (socket says 172.21.15.42:35776) 2019-12-05T06:45:54.485 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:54.473+0000 7f59d2c6e700 1 -- 172.21.15.42:0/145360944 learned_addr learned my addr 172.21.15.42:0/145360944 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:45:54.485 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:54.473+0000 7f59d346f700 1 --2- 172.21.15.42:0/145360944 >> v2:172.21.15.40:3300/0 conn(0x7f59cc09fae0 0x7f59cc07eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:45:54.486 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:54.473+0000 7f59d2c6e700 1 -- 172.21.15.42:0/145360944 >> v2:172.21.15.40:3300/0 conn(0x7f59cc09fae0 msgr2=0x7f59cc07eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:45:54.486 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:54.473+0000 7f59d2c6e700 1 --2- 172.21.15.42:0/145360944 >> v2:172.21.15.40:3300/0 conn(0x7f59cc09fae0 0x7f59cc07eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:45:54.486 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:54.473+0000 7f59d2c6e700 1 -- 172.21.15.42:0/145360944 >> v1:172.21.15.40:6789/0 conn(0x7f59cc07e720 legacy=0x7f59cc09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:45:54.486 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:54.473+0000 7f59d2c6e700 1 -- 172.21.15.42:0/145360944 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59cc07f0a0 con 0x7f59cc0a7740 2019-12-05T06:45:54.487 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:54.473+0000 7f59d2c6e700 1 --2- 172.21.15.42:0/145360944 >> v2:172.21.15.42:3300/0 conn(0x7f59cc0a7740 0x7f59cc08df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f59c8002f30 tx=0x7f59c800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:54.487 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:54.473+0000 7f59d2c6e700 1 --2- 172.21.15.42:0/145360944 >> v2:172.21.15.42:3300/0 conn(0x7f59cc0a7740 0x7f59cc08df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f59c8002f30 tx=0x7f59c800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:54.487 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:54.473+0000 7f59d2c6e700 1 --2- 172.21.15.42:0/145360944 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f59cc0a7740 0x7f59cc08df80 secure :-1 s=READY pgs=3180 cs=0 l=1 rx=0x7f59c8002f30 tx=0x7f59c800fd30).ready entity=mon.1 client_cookie=f6206c27bcbb8a2a server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:45:54.488 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:54.473+0000 7f59d2c6e700 1 --2- 172.21.15.42:0/145360944 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f59cc0a7740 0x7f59cc08df80 secure :-1 s=THROTTLE_DONE pgs=3180 cs=0 l=1 rx=0x7f59c8002f30 tx=0x7f59c800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:54.488 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:54.473+0000 7f59d1c6c700 1 -- 172.21.15.42:0/145360944 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f59c8008c90 con 0x7f59cc0a7740 2019-12-05T06:45:54.488 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:54.473+0000 7f59d2c6e700 1 --2- 172.21.15.42:0/145360944 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f59cc0a7740 0x7f59cc08df80 secure :-1 s=THROTTLE_DONE pgs=3180 cs=0 l=1 rx=0x7f59c8002f30 tx=0x7f59c800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:54.488 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:54.473+0000 7f59d2c6e700 1 --2- 172.21.15.42:0/145360944 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f59cc0a7740 0x7f59cc08df80 secure :-1 s=THROTTLE_DONE pgs=3180 cs=0 l=1 rx=0x7f59c8002f30 tx=0x7f59c800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:54.489 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:54.473+0000 7f59d1c6c700 1 -- 172.21.15.42:0/145360944 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59c80088b0 con 0x7f59cc0a7740 2019-12-05T06:45:54.489 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:54.473+0000 7f59d1c6c700 1 -- 172.21.15.42:0/145360944 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f59c8011ad0 con 0x7f59cc0a7740 2019-12-05T06:45:54.490 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:54.473+0000 7f59d46d1700 1 -- 172.21.15.42:0/145360944 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f59cc0a7740 msgr2=0x7f59cc08df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:45:54.490 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:54.477+0000 7f59d46d1700 1 --2- 172.21.15.42:0/145360944 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f59cc0a7740 0x7f59cc08df80 secure :-1 s=READY pgs=3180 cs=0 l=1 rx=0x7f59c8002f30 tx=0x7f59c800fd30).stop 2019-12-05T06:45:54.500 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:54.477+0000 7f59d46d1700 1 -- 172.21.15.42:0/145360944 shutdown_connections 2019-12-05T06:45:54.500 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:54.477+0000 7f59d46d1700 1 --2- 172.21.15.42:0/145360944 >> v2:172.21.15.40:3300/0 conn(0x7f59cc09fae0 0x7f59cc07eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:45:54.501 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:54.477+0000 7f59d46d1700 1 --2- 172.21.15.42:0/145360944 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f59cc0a7740 0x7f59cc08df80 unknown :-1 s=CLOSED pgs=3180 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:45:54.501 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:54.477+0000 7f59d46d1700 1 -- 172.21.15.42:0/145360944 shutdown_connections 2019-12-05T06:45:54.501 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:54.477+0000 7f59d46d1700 1 -- 172.21.15.42:0/145360944 wait complete. 2019-12-05T06:45:54.501 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:54.477+0000 7f59d46d1700 1 -- 172.21.15.42:0/145360944 >> 172.21.15.42:0/145360944 conn(0x7f59cc0aabe0 msgr2=0x7f59cc0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:45:54.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:54.481+0000 7f59d46d1700 1 Processor -- start 2019-12-05T06:45:54.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:54.481+0000 7f59d46d1700 1 -- start start 2019-12-05T06:45:54.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:54.481+0000 7f59d46d1700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f59cc07baa0 0x7f59cc07c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:45:54.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:54.481+0000 7f59d46d1700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59cc07c630 con 0x7f59cc126680 2019-12-05T06:45:54.539 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:54.481+0000 7f59d46d1700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59cc1272b0 con 0x7f59cc116550 2019-12-05T06:45:54.539 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:54.481+0000 7f59d46d1700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59cc07bed0 con 0x7f59cc07baa0 2019-12-05T06:45:54.539 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:54.481+0000 7f59d2c6e700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f59cc116550 0x7f59cc11b8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:35820/0 (socket says 172.21.15.42:35820) 2019-12-05T06:45:54.539 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:54.481+0000 7f59d2c6e700 1 -- 172.21.15.42:0/2941281567 learned_addr learned my addr 172.21.15.42:0/2941281567 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:45:54.539 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:54.481+0000 7f59d246d700 1 --2- 172.21.15.42:0/2941281567 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f59cc07baa0 0x7f59cc07c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:45:54.539 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:54.481+0000 7f59c37fe700 1 -- 172.21.15.42:0/2941281567 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2869107645 0 0) 0x7f59cc1272b0 con 0x7f59cc116550 2019-12-05T06:45:54.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:54.481+0000 7f59c37fe700 1 -- 172.21.15.42:0/2941281567 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59b40030b0 con 0x7f59cc116550 2019-12-05T06:45:54.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:54.481+0000 7f59c37fe700 1 -- 172.21.15.42:0/2941281567 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1469004869 0 0) 0x7f59cc07c630 con 0x7f59cc126680 2019-12-05T06:45:54.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:54.481+0000 7f59d246d700 1 -- 172.21.15.42:0/2941281567 >> v1:172.21.15.42:6789/0 conn(0x7f59cc116550 legacy=0x7f59cc11b8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:45:54.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:54.481+0000 7f59d246d700 1 -- 172.21.15.42:0/2941281567 >> v1:172.21.15.40:6789/0 conn(0x7f59cc126680 legacy=0x7f59cc126ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:45:54.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:54.481+0000 7f59d246d700 1 -- 172.21.15.42:0/2941281567 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59cc11acb0 con 0x7f59cc07baa0 2019-12-05T06:45:54.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:54.481+0000 7f59d246d700 1 --2- 172.21.15.42:0/2941281567 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f59cc07baa0 0x7f59cc07c0e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f59bc000e90 tx=0x7f59bc00c2f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:54.541 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:54.481+0000 7f59d246d700 1 --2- 172.21.15.42:0/2941281567 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f59cc07baa0 0x7f59cc07c0e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f59bc000e90 tx=0x7f59bc00c2f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:54.541 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:54.481+0000 7f59d246d700 1 --2- 172.21.15.42:0/2941281567 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f59cc07baa0 0x7f59cc07c0e0 secure :-1 s=READY pgs=701 cs=0 l=1 rx=0x7f59bc000e90 tx=0x7f59bc00c2f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:45:54.541 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:54.481+0000 7f59d246d700 1 --2- 172.21.15.42:0/2941281567 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f59cc07baa0 0x7f59cc07c0e0 secure :-1 s=THROTTLE_DONE pgs=701 cs=0 l=1 rx=0x7f59bc000e90 tx=0x7f59bc00c2f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:54.541 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:54.481+0000 7f59c37fe700 1 -- 172.21.15.42:0/2941281567 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f59bc00cb90 con 0x7f59cc07baa0 2019-12-05T06:45:54.541 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:54.481+0000 7f59d246d700 1 --2- 172.21.15.42:0/2941281567 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f59cc07baa0 0x7f59cc07c0e0 secure :-1 s=THROTTLE_DONE pgs=701 cs=0 l=1 rx=0x7f59bc000e90 tx=0x7f59bc00c2f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:54.541 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:54.481+0000 7f59d246d700 1 --2- 172.21.15.42:0/2941281567 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f59cc07baa0 0x7f59cc07c0e0 secure :-1 s=THROTTLE_DONE pgs=701 cs=0 l=1 rx=0x7f59bc000e90 tx=0x7f59bc00c2f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:54.541 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:54.481+0000 7f59c37fe700 1 -- 172.21.15.42:0/2941281567 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59bc00c7b0 con 0x7f59cc07baa0 2019-12-05T06:45:54.542 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:54.481+0000 7f59d46d1700 1 -- 172.21.15.42:0/2941281567 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59cc12e900 con 0x7f59cc07baa0 2019-12-05T06:45:54.542 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:54.481+0000 7f59c37fe700 1 -- 172.21.15.42:0/2941281567 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f59bc011070 con 0x7f59cc07baa0 2019-12-05T06:45:54.542 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:54.481+0000 7f59d46d1700 1 -- 172.21.15.42:0/2941281567 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59cc12ee30 con 0x7f59cc07baa0 2019-12-05T06:45:54.543 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:54.485+0000 7f59d246d700 1 --2- 172.21.15.42:0/2941281567 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f59cc07baa0 0x7f59cc07c0e0 secure :-1 s=THROTTLE_DONE pgs=701 cs=0 l=1 rx=0x7f59bc000e90 tx=0x7f59bc00c2f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:54.543 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:54.485+0000 7f59c37fe700 1 -- 172.21.15.42:0/2941281567 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f59bc011340 con 0x7f59cc07baa0 2019-12-05T06:45:54.545 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:54.485+0000 7f59d246d700 1 --2- 172.21.15.42:0/2941281567 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f59cc07baa0 0x7f59cc07c0e0 secure :-1 s=THROTTLE_DONE pgs=701 cs=0 l=1 rx=0x7f59bc000e90 tx=0x7f59bc00c2f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:54.545 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:54.485+0000 7f59c37fe700 1 --2- 172.21.15.42:0/2941281567 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f59b403b1c0 0x7f59b403d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:45:54.545 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:54.485+0000 7f59c37fe700 1 -- 172.21.15.42:0/2941281567 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 11965+0+0 (secure 0 0 0) 0x7f59bc04b110 con 0x7f59cc07baa0 2019-12-05T06:45:54.545 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:54.485+0000 7f59d2c6e700 1 --2- 172.21.15.42:0/2941281567 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f59b403b1c0 0x7f59b403d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:45:54.546 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:54.485+0000 7f59d2c6e700 1 --2- 172.21.15.42:0/2941281567 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f59b403b1c0 0x7f59b403d6d0 crc :-1 s=READY pgs=1331 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:45:54.546 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:54.489+0000 7f59d46d1700 1 -- 172.21.15.42:0/2941281567 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59cc05d110 con 0x7f59cc07baa0 2019-12-05T06:45:54.546 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:54.497+0000 7f59d246d700 1 --2- 172.21.15.42:0/2941281567 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f59cc07baa0 0x7f59cc07c0e0 secure :-1 s=THROTTLE_DONE pgs=701 cs=0 l=1 rx=0x7f59bc000e90 tx=0x7f59bc00c2f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:54.546 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:54.497+0000 7f59c37fe700 1 -- 172.21.15.42:0/2941281567 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f59bc032190 con 0x7f59cc07baa0 2019-12-05T06:45:54.676 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:54.669+0000 7f59d46d1700 1 -- 172.21.15.42:0/2941281567 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f59cc05b8c0 con 0x7f59cc07baa0 2019-12-05T06:45:54.676 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:54.669+0000 7f59d246d700 1 --2- 172.21.15.42:0/2941281567 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f59cc07baa0 0x7f59cc07c0e0 secure :-1 s=THROTTLE_DONE pgs=701 cs=0 l=1 rx=0x7f59bc000e90 tx=0x7f59bc00c2f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:54.677 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:54.669+0000 7f59c37fe700 1 -- 172.21.15.42:0/2941281567 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 383 v383) v1 ==== 95+0+2117 (secure 0 0 0) 0x7f59bc0323f0 con 0x7f59cc07baa0 2019-12-05T06:45:54.678 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 383 2019-12-05T06:45:54.678 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:45:54.678 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":383,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":12429,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/4178556019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":4178556019},{"type":"v1","addr":"172.21.15.40:6833","nonce":4178556019}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":383}],"filesystems":[{"mdsmap":{"epoch":382,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:44:34.328051+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":118,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12303},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12303":{"gid":12303,"name":"a","rank":0,"incarnation":378,"state":"up:active","state_seq":17,"addr":"172.21.15.42:6835/2489540211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":2489540211},{"type":"v1","addr":"172.21.15.42:6835","nonce":2489540211}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:45:54.681 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:54.673+0000 7f59d46d1700 1 -- 172.21.15.42:0/2941281567 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f59b403b1c0 msgr2=0x7f59b403d6d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:45:54.681 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:54.673+0000 7f59d46d1700 1 --2- 172.21.15.42:0/2941281567 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f59b403b1c0 0x7f59b403d6d0 crc :-1 s=READY pgs=1331 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:45:54.682 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:54.673+0000 7f59d46d1700 1 -- 172.21.15.42:0/2941281567 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f59cc07baa0 msgr2=0x7f59cc07c0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:45:54.682 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:54.673+0000 7f59d46d1700 1 --2- 172.21.15.42:0/2941281567 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f59cc07baa0 0x7f59cc07c0e0 secure :-1 s=READY pgs=701 cs=0 l=1 rx=0x7f59bc000e90 tx=0x7f59bc00c2f0).stop 2019-12-05T06:45:54.682 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:54.673+0000 7f59d46d1700 1 -- 172.21.15.42:0/2941281567 shutdown_connections 2019-12-05T06:45:54.682 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:54.673+0000 7f59d46d1700 1 --2- 172.21.15.42:0/2941281567 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f59b403b1c0 0x7f59b403d6d0 unknown :-1 s=CLOSED pgs=1331 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:45:54.683 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:54.673+0000 7f59d46d1700 1 --2- 172.21.15.42:0/2941281567 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f59cc07baa0 0x7f59cc07c0e0 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:45:54.683 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:54.677+0000 7f59d46d1700 1 -- 172.21.15.42:0/2941281567 shutdown_connections 2019-12-05T06:45:54.683 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:54.677+0000 7f59d46d1700 1 -- 172.21.15.42:0/2941281567 wait complete. 2019-12-05T06:45:54.683 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:54.677+0000 7f59d46d1700 1 -- 172.21.15.42:0/2941281567 >> 172.21.15.42:0/2941281567 conn(0x7f59cc08fda0 msgr2=0x7f59cc09f620 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:45:54.691 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:45:56.692 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:45:56.805 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:56.793+0000 7f98b1c02700 1 Processor -- start 2019-12-05T06:45:56.806 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:56.793+0000 7f98b1c02700 1 -- start start 2019-12-05T06:45:56.806 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:56.793+0000 7f98b1c02700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f98ac0a7740 0x7f98ac08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:45:56.806 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:56.793+0000 7f98b1c02700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f98ac09fae0 0x7f98ac07eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:45:56.806 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:56.793+0000 7f98b1c02700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98ac0a0710 con 0x7f98ac07e720 2019-12-05T06:45:56.806 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:56.793+0000 7f98b1c02700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98ac0a0b00 con 0x7f98ac0a7740 2019-12-05T06:45:56.806 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:56.793+0000 7f98b1c02700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98ac08e5b0 con 0x7f98ac09fae0 2019-12-05T06:45:56.807 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:56.793+0000 7f98b09a0700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f98ac09fae0 0x7f98ac07eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:45:56.807 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:56.793+0000 7f98b09a0700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f98ac09fae0 0x7f98ac07eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:35790/0 (socket says 172.21.15.42:35790) 2019-12-05T06:45:56.807 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:56.793+0000 7f98b09a0700 1 -- 172.21.15.42:0/885667692 learned_addr learned my addr 172.21.15.42:0/885667692 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:45:56.807 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:56.793+0000 7f98abfff700 1 --2- 172.21.15.42:0/885667692 >> v2:172.21.15.40:3300/0 conn(0x7f98ac0a7740 0x7f98ac08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:45:56.807 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:56.793+0000 7f98b09a0700 1 -- 172.21.15.42:0/885667692 >> v2:172.21.15.40:3300/0 conn(0x7f98ac0a7740 msgr2=0x7f98ac08df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:45:56.807 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:56.793+0000 7f98b09a0700 1 --2- 172.21.15.42:0/885667692 >> v2:172.21.15.40:3300/0 conn(0x7f98ac0a7740 0x7f98ac08df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:45:56.807 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:56.793+0000 7f98b09a0700 1 -- 172.21.15.42:0/885667692 >> v1:172.21.15.40:6789/0 conn(0x7f98ac07e720 legacy=0x7f98ac09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:45:56.808 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:56.793+0000 7f98b09a0700 1 -- 172.21.15.42:0/885667692 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98ac07f0a0 con 0x7f98ac09fae0 2019-12-05T06:45:56.808 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:56.793+0000 7f98b09a0700 1 --2- 172.21.15.42:0/885667692 >> v2:172.21.15.42:3300/0 conn(0x7f98ac09fae0 0x7f98ac07eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f98a40032b0 tx=0x7f98a40100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:56.808 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:56.797+0000 7f98b09a0700 1 --2- 172.21.15.42:0/885667692 >> v2:172.21.15.42:3300/0 conn(0x7f98ac09fae0 0x7f98ac07eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f98a40032b0 tx=0x7f98a40100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:56.808 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:56.797+0000 7f98b09a0700 1 --2- 172.21.15.42:0/885667692 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f98ac09fae0 0x7f98ac07eb50 secure :-1 s=READY pgs=3182 cs=0 l=1 rx=0x7f98a40032b0 tx=0x7f98a40100b0).ready entity=mon.1 client_cookie=f1e9f35a252b304b server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:45:56.808 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:56.797+0000 7f98b09a0700 1 --2- 172.21.15.42:0/885667692 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f98ac09fae0 0x7f98ac07eb50 secure :-1 s=THROTTLE_DONE pgs=3182 cs=0 l=1 rx=0x7f98a40032b0 tx=0x7f98a40100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:56.808 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:56.797+0000 7f98ab7fe700 1 -- 172.21.15.42:0/885667692 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f98a4010d60 con 0x7f98ac09fae0 2019-12-05T06:45:56.808 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:56.797+0000 7f98b09a0700 1 --2- 172.21.15.42:0/885667692 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f98ac09fae0 0x7f98ac07eb50 secure :-1 s=THROTTLE_DONE pgs=3182 cs=0 l=1 rx=0x7f98a40032b0 tx=0x7f98a40100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:56.809 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:56.797+0000 7f98ab7fe700 1 -- 172.21.15.42:0/885667692 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98a4010980 con 0x7f98ac09fae0 2019-12-05T06:45:56.809 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:56.797+0000 7f98b09a0700 1 --2- 172.21.15.42:0/885667692 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f98ac09fae0 0x7f98ac07eb50 secure :-1 s=THROTTLE_DONE pgs=3182 cs=0 l=1 rx=0x7f98a40032b0 tx=0x7f98a40100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:56.809 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:56.797+0000 7f98b1c02700 1 -- 172.21.15.42:0/885667692 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f98ac09fae0 msgr2=0x7f98ac07eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:45:56.809 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:56.797+0000 7f98ab7fe700 1 -- 172.21.15.42:0/885667692 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f98a40114a0 con 0x7f98ac09fae0 2019-12-05T06:45:56.809 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:56.797+0000 7f98b1c02700 1 --2- 172.21.15.42:0/885667692 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f98ac09fae0 0x7f98ac07eb50 secure :-1 s=READY pgs=3182 cs=0 l=1 rx=0x7f98a40032b0 tx=0x7f98a40100b0).stop 2019-12-05T06:45:56.809 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:56.797+0000 7f98b1c02700 1 -- 172.21.15.42:0/885667692 shutdown_connections 2019-12-05T06:45:56.809 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:56.797+0000 7f98b1c02700 1 --2- 172.21.15.42:0/885667692 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f98ac09fae0 0x7f98ac07eb50 unknown :-1 s=CLOSED pgs=3182 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:45:56.810 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:56.797+0000 7f98b1c02700 1 --2- 172.21.15.42:0/885667692 >> v2:172.21.15.40:3300/0 conn(0x7f98ac0a7740 0x7f98ac08df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:45:56.810 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:56.797+0000 7f98b1c02700 1 -- 172.21.15.42:0/885667692 shutdown_connections 2019-12-05T06:45:56.810 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:56.797+0000 7f98b1c02700 1 -- 172.21.15.42:0/885667692 wait complete. 2019-12-05T06:45:56.810 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:56.797+0000 7f98b1c02700 1 -- 172.21.15.42:0/885667692 >> 172.21.15.42:0/885667692 conn(0x7f98ac0aabe0 msgr2=0x7f98ac0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:45:56.810 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:56.801+0000 7f98b1c02700 1 Processor -- start 2019-12-05T06:45:56.810 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:56.801+0000 7f98b1c02700 1 -- start start 2019-12-05T06:45:56.810 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:56.801+0000 7f98b1c02700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f98ac0a5e40 0x7f98ac117170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:45:56.812 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:56.801+0000 7f98b1c02700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f98ac122ff0 0x7f98ac123420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:45:56.812 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:56.801+0000 7f98b1c02700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98ac1176c0 con 0x7f98ac119a50 2019-12-05T06:45:56.813 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:56.801+0000 7f98b1c02700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98ac117a40 con 0x7f98ac122ff0 2019-12-05T06:45:56.813 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:56.801+0000 7f98b1c02700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98ac122d50 con 0x7f98ac0a5e40 2019-12-05T06:45:56.813 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:56.801+0000 7f98a37fe700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f98ac119a50 0x7f98ac07db50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:45274/0 (socket says 172.21.15.42:45274) 2019-12-05T06:45:56.813 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:56.801+0000 7f98a37fe700 1 -- 172.21.15.42:0/2085085415 learned_addr learned my addr 172.21.15.42:0/2085085415 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:45:56.813 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:56.801+0000 7f98b09a0700 1 --2- 172.21.15.42:0/2085085415 >> v2:172.21.15.40:3300/0 conn(0x7f98ac122ff0 0x7f98ac123420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:45:56.813 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:56.801+0000 7f98abfff700 1 --2- 172.21.15.42:0/2085085415 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f98ac0a5e40 0x7f98ac117170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:45:56.817 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:56.801+0000 7f98a97fa700 1 -- 172.21.15.42:0/2085085415 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3186971396 0 0) 0x7f98ac1176c0 con 0x7f98ac119a50 2019-12-05T06:45:56.817 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:56.801+0000 7f98a97fa700 1 -- 172.21.15.42:0/2085085415 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98900030b0 con 0x7f98ac119a50 2019-12-05T06:45:56.817 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:56.801+0000 7f98a97fa700 1 -- 172.21.15.42:0/2085085415 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 2730692240 0 0) 0x7f98900030b0 con 0x7f98ac119a50 2019-12-05T06:45:56.817 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:56.801+0000 7f98b09a0700 1 -- 172.21.15.42:0/2085085415 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f98ac0a5e40 msgr2=0x7f98ac117170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:45:56.817 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:56.801+0000 7f98b09a0700 1 --2- 172.21.15.42:0/2085085415 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f98ac0a5e40 0x7f98ac117170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:45:56.818 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:56.801+0000 7f98b09a0700 1 -- 172.21.15.42:0/2085085415 >> v1:172.21.15.40:6789/0 conn(0x7f98ac119a50 legacy=0x7f98ac07db50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:45:56.818 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:56.801+0000 7f98b09a0700 1 -- 172.21.15.42:0/2085085415 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98ac1305e0 con 0x7f98ac122ff0 2019-12-05T06:45:56.818 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:56.801+0000 7f98abfff700 1 --2- 172.21.15.42:0/2085085415 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f98ac0a5e40 0x7f98ac117170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:45:56.818 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:56.801+0000 7f98b09a0700 1 --2- 172.21.15.42:0/2085085415 >> v2:172.21.15.40:3300/0 conn(0x7f98ac122ff0 0x7f98ac123420 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f9894001ca0 tx=0x7f9894001ce0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:56.818 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:56.801+0000 7f98b09a0700 1 --2- 172.21.15.42:0/2085085415 >> v2:172.21.15.40:3300/0 conn(0x7f98ac122ff0 0x7f98ac123420 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f9894001ca0 tx=0x7f9894001ce0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:56.818 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:56.801+0000 7f98b09a0700 1 --2- 172.21.15.42:0/2085085415 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f98ac122ff0 0x7f98ac123420 secure :-1 s=READY pgs=2410 cs=0 l=1 rx=0x7f9894001ca0 tx=0x7f9894001ce0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:45:56.819 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:56.801+0000 7f98b09a0700 1 --2- 172.21.15.42:0/2085085415 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f98ac122ff0 0x7f98ac123420 secure :-1 s=THROTTLE_DONE pgs=2410 cs=0 l=1 rx=0x7f9894001ca0 tx=0x7f9894001ce0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:56.819 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:56.801+0000 7f98a97fa700 1 -- 172.21.15.42:0/2085085415 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9894001a60 con 0x7f98ac122ff0 2019-12-05T06:45:56.819 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:56.801+0000 7f98b09a0700 1 --2- 172.21.15.42:0/2085085415 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f98ac122ff0 0x7f98ac123420 secure :-1 s=THROTTLE_DONE pgs=2410 cs=0 l=1 rx=0x7f9894001ca0 tx=0x7f9894001ce0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:56.819 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:56.801+0000 7f98b09a0700 1 --2- 172.21.15.42:0/2085085415 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f98ac122ff0 0x7f98ac123420 secure :-1 s=THROTTLE_DONE pgs=2410 cs=0 l=1 rx=0x7f9894001ca0 tx=0x7f9894001ce0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:56.819 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:56.801+0000 7f98a97fa700 1 -- 172.21.15.42:0/2085085415 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9894001870 con 0x7f98ac122ff0 2019-12-05T06:45:56.819 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:56.801+0000 7f98b1c02700 1 -- 172.21.15.42:0/2085085415 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98ac137a30 con 0x7f98ac122ff0 2019-12-05T06:45:56.819 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:56.801+0000 7f98a97fa700 1 -- 172.21.15.42:0/2085085415 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f98a4011db0 con 0x7f98ac122ff0 2019-12-05T06:45:56.820 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:56.801+0000 7f98b1c02700 1 -- 172.21.15.42:0/2085085415 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98ac137fe0 con 0x7f98ac122ff0 2019-12-05T06:45:56.820 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:56.805+0000 7f98b09a0700 1 --2- 172.21.15.42:0/2085085415 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f98ac122ff0 0x7f98ac123420 secure :-1 s=THROTTLE_DONE pgs=2410 cs=0 l=1 rx=0x7f9894001ca0 tx=0x7f9894001ce0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:56.820 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:56.805+0000 7f98a97fa700 1 -- 172.21.15.42:0/2085085415 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f98a40058c0 con 0x7f98ac122ff0 2019-12-05T06:45:56.820 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:56.805+0000 7f98b09a0700 1 --2- 172.21.15.42:0/2085085415 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f98ac122ff0 0x7f98ac123420 secure :-1 s=THROTTLE_DONE pgs=2410 cs=0 l=1 rx=0x7f9894001ca0 tx=0x7f9894001ce0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:56.820 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:56.805+0000 7f98a97fa700 1 --2- 172.21.15.42:0/2085085415 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f989003b210 0x7f989003d720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:45:56.820 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:56.805+0000 7f98a97fa700 1 -- 172.21.15.42:0/2085085415 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 12051+0+0 (secure 0 0 0) 0x7f98a404af30 con 0x7f98ac122ff0 2019-12-05T06:45:56.820 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:56.805+0000 7f98abfff700 1 --2- 172.21.15.42:0/2085085415 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f989003b210 0x7f989003d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:45:56.821 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:56.805+0000 7f98abfff700 1 --2- 172.21.15.42:0/2085085415 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f989003b210 0x7f989003d720 crc :-1 s=READY pgs=1332 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:45:56.821 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:56.809+0000 7f98b1c02700 1 -- 172.21.15.42:0/2085085415 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98ac05d110 con 0x7f98ac122ff0 2019-12-05T06:45:56.821 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:56.817+0000 7f98b09a0700 1 --2- 172.21.15.42:0/2085085415 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f98ac122ff0 0x7f98ac123420 secure :-1 s=THROTTLE_DONE pgs=2410 cs=0 l=1 rx=0x7f9894001ca0 tx=0x7f9894001ce0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:56.821 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:56.817+0000 7f98a97fa700 1 -- 172.21.15.42:0/2085085415 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f98a4031fb0 con 0x7f98ac122ff0 2019-12-05T06:45:56.987 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:56.981+0000 7f98b1c02700 1 -- 172.21.15.42:0/2085085415 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f98ac05b8c0 con 0x7f98ac122ff0 2019-12-05T06:45:56.988 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:56.981+0000 7f98b09a0700 1 --2- 172.21.15.42:0/2085085415 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f98ac122ff0 0x7f98ac123420 secure :-1 s=THROTTLE_DONE pgs=2410 cs=0 l=1 rx=0x7f9894001ca0 tx=0x7f9894001ce0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:56.988 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:56.981+0000 7f98a97fa700 1 -- 172.21.15.42:0/2085085415 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 384 v384) v1 ==== 95+0+1778 (secure 0 0 0) 0x7f98a4032210 con 0x7f98ac122ff0 2019-12-05T06:45:56.990 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:45:56.990 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":384,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":384,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:45:55.302477+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":119,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12429},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12429":{"gid":12429,"name":"b","rank":0,"incarnation":384,"state":"up:replay","state_seq":1,"addr":"172.21.15.40:6833/4178556019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":4178556019},{"type":"v1","addr":"172.21.15.40:6833","nonce":4178556019}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:45:56.990 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 384 2019-12-05T06:45:56.992 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:56.989+0000 7f98b1c02700 1 -- 172.21.15.42:0/2085085415 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f989003b210 msgr2=0x7f989003d720 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:45:56.993 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:56.989+0000 7f98b1c02700 1 --2- 172.21.15.42:0/2085085415 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f989003b210 0x7f989003d720 crc :-1 s=READY pgs=1332 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:45:56.993 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:56.989+0000 7f98b1c02700 1 -- 172.21.15.42:0/2085085415 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f98ac122ff0 msgr2=0x7f98ac123420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:45:56.993 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:56.989+0000 7f98b1c02700 1 --2- 172.21.15.42:0/2085085415 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f98ac122ff0 0x7f98ac123420 secure :-1 s=READY pgs=2410 cs=0 l=1 rx=0x7f9894001ca0 tx=0x7f9894001ce0).stop 2019-12-05T06:45:56.993 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:56.989+0000 7f98b1c02700 1 -- 172.21.15.42:0/2085085415 shutdown_connections 2019-12-05T06:45:56.994 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:56.989+0000 7f98b1c02700 1 --2- 172.21.15.42:0/2085085415 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f989003b210 0x7f989003d720 unknown :-1 s=CLOSED pgs=1332 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:45:56.994 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:56.989+0000 7f98b1c02700 1 --2- 172.21.15.42:0/2085085415 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f98ac122ff0 0x7f98ac123420 unknown :-1 s=CLOSED pgs=2410 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:45:56.994 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:56.989+0000 7f98b1c02700 1 --2- 172.21.15.42:0/2085085415 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f98ac0a5e40 0x7f98ac117170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:45:56.994 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:56.989+0000 7f98b1c02700 1 -- 172.21.15.42:0/2085085415 shutdown_connections 2019-12-05T06:45:56.994 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:56.989+0000 7f98b1c02700 1 -- 172.21.15.42:0/2085085415 wait complete. 2019-12-05T06:45:56.994 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:56.989+0000 7f98b1c02700 1 -- 172.21.15.42:0/2085085415 >> 172.21.15.42:0/2085085415 conn(0x7f98ac0aabe0 msgr2=0x7f98ac059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:45:57.008 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a down, removed from mdsmap 2019-12-05T06:45:57.009 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2019-12-05T06:45:57.009 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:45:57.129 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:57.117+0000 7fad6351e700 1 Processor -- start 2019-12-05T06:45:57.129 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:57.117+0000 7fad6351e700 1 -- start start 2019-12-05T06:45:57.129 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:57.121+0000 7fad6351e700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fad5c0a7740 0x7fad5c08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:45:57.130 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:57.121+0000 7fad6351e700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fad5c09fae0 0x7fad5c07eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:45:57.130 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:57.121+0000 7fad6351e700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad5c0a0710 con 0x7fad5c07e720 2019-12-05T06:45:57.130 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:57.121+0000 7fad6351e700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad5c0a0b00 con 0x7fad5c0a7740 2019-12-05T06:45:57.131 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:57.121+0000 7fad6351e700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad5c08e5b0 con 0x7fad5c09fae0 2019-12-05T06:45:57.131 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:57.121+0000 7fad61abb700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fad5c0a7740 0x7fad5c08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:45:57.131 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:57.121+0000 7fad612ba700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fad5c07e720 0x7fad5c09f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:35844/0 (socket says 172.21.15.42:35844) 2019-12-05T06:45:57.132 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:57.121+0000 7fad612ba700 1 -- 172.21.15.42:0/506590220 learned_addr learned my addr 172.21.15.42:0/506590220 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:45:57.132 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:57.121+0000 7fad622bc700 1 --2- 172.21.15.42:0/506590220 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fad5c09fae0 0x7fad5c07eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:45:57.132 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:57.121+0000 7fad60ab9700 1 -- 172.21.15.42:0/506590220 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1719217278 0 0) 0x7fad5c0a0710 con 0x7fad5c07e720 2019-12-05T06:45:57.133 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:57.121+0000 7fad61abb700 1 -- 172.21.15.42:0/506590220 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fad5c09fae0 msgr2=0x7fad5c07eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:45:57.133 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:57.121+0000 7fad61abb700 1 --2- 172.21.15.42:0/506590220 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fad5c09fae0 0x7fad5c07eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:45:57.133 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:57.121+0000 7fad61abb700 1 -- 172.21.15.42:0/506590220 >> v1:172.21.15.42:6789/0 conn(0x7fad5c07e720 legacy=0x7fad5c09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:45:57.133 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:57.121+0000 7fad61abb700 1 -- 172.21.15.42:0/506590220 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad5c07f0a0 con 0x7fad5c0a7740 2019-12-05T06:45:57.134 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:57.121+0000 7fad622bc700 1 --2- 172.21.15.42:0/506590220 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fad5c09fae0 0x7fad5c07eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:45:57.134 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:57.121+0000 7fad61abb700 1 --2- 172.21.15.42:0/506590220 >> v2:172.21.15.40:3300/0 conn(0x7fad5c0a7740 0x7fad5c08df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fad58008790 tx=0x7fad5800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:57.134 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:57.121+0000 7fad61abb700 1 --2- 172.21.15.42:0/506590220 >> v2:172.21.15.40:3300/0 conn(0x7fad5c0a7740 0x7fad5c08df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fad58008790 tx=0x7fad5800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:57.135 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:57.121+0000 7fad61abb700 1 --2- 172.21.15.42:0/506590220 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fad5c0a7740 0x7fad5c08df80 secure :-1 s=READY pgs=2411 cs=0 l=1 rx=0x7fad58008790 tx=0x7fad5800fc70).ready entity=mon.0 client_cookie=18df6baf84b04a65 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:45:57.135 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:57.121+0000 7fad61abb700 1 --2- 172.21.15.42:0/506590220 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fad5c0a7740 0x7fad5c08df80 secure :-1 s=THROTTLE_DONE pgs=2411 cs=0 l=1 rx=0x7fad58008790 tx=0x7fad5800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:57.135 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:57.121+0000 7fad60ab9700 1 -- 172.21.15.42:0/506590220 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fad58008bb0 con 0x7fad5c0a7740 2019-12-05T06:45:57.135 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:57.121+0000 7fad61abb700 1 --2- 172.21.15.42:0/506590220 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fad5c0a7740 0x7fad5c08df80 secure :-1 s=THROTTLE_DONE pgs=2411 cs=0 l=1 rx=0x7fad58008790 tx=0x7fad5800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:57.136 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:57.121+0000 7fad61abb700 1 --2- 172.21.15.42:0/506590220 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fad5c0a7740 0x7fad5c08df80 secure :-1 s=THROTTLE_DONE pgs=2411 cs=0 l=1 rx=0x7fad58008790 tx=0x7fad5800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:57.136 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:57.121+0000 7fad60ab9700 1 -- 172.21.15.42:0/506590220 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad580087d0 con 0x7fad5c0a7740 2019-12-05T06:45:57.136 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:57.121+0000 7fad60ab9700 1 -- 172.21.15.42:0/506590220 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fad58011860 con 0x7fad5c0a7740 2019-12-05T06:45:57.137 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:57.121+0000 7fad6351e700 1 -- 172.21.15.42:0/506590220 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fad5c0a7740 msgr2=0x7fad5c08df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:45:57.137 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:57.121+0000 7fad6351e700 1 --2- 172.21.15.42:0/506590220 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fad5c0a7740 0x7fad5c08df80 secure :-1 s=READY pgs=2411 cs=0 l=1 rx=0x7fad58008790 tx=0x7fad5800fc70).stop 2019-12-05T06:45:57.138 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:57.121+0000 7fad6351e700 1 -- 172.21.15.42:0/506590220 shutdown_connections 2019-12-05T06:45:57.138 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:57.121+0000 7fad6351e700 1 --2- 172.21.15.42:0/506590220 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fad5c09fae0 0x7fad5c07eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:45:57.138 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:57.121+0000 7fad6351e700 1 --2- 172.21.15.42:0/506590220 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fad5c0a7740 0x7fad5c08df80 unknown :-1 s=CLOSED pgs=2411 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:45:57.139 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:57.121+0000 7fad6351e700 1 -- 172.21.15.42:0/506590220 shutdown_connections 2019-12-05T06:45:57.139 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:57.121+0000 7fad6351e700 1 -- 172.21.15.42:0/506590220 wait complete. 2019-12-05T06:45:57.139 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:57.121+0000 7fad6351e700 1 -- 172.21.15.42:0/506590220 >> 172.21.15.42:0/506590220 conn(0x7fad5c0aabe0 msgr2=0x7fad5c0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:45:57.145 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:57.125+0000 7fad6351e700 1 Processor -- start 2019-12-05T06:45:57.145 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:57.125+0000 7fad6351e700 1 -- start start 2019-12-05T06:45:57.146 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:57.129+0000 7fad6351e700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fad5c1429f0 0x7fad5c11fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:45:57.146 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:57.129+0000 7fad6351e700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad5c1200e0 con 0x7fad5c144f10 2019-12-05T06:45:57.146 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:57.129+0000 7fad6351e700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad5c145340 con 0x7fad5c144ae0 2019-12-05T06:45:57.146 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:57.129+0000 7fad6351e700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad5c1314b0 con 0x7fad5c1429f0 2019-12-05T06:45:57.146 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:57.129+0000 7fad612ba700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fad5c144ae0 0x7fad5c11b020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:35848/0 (socket says 172.21.15.42:35848) 2019-12-05T06:45:57.148 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:57.129+0000 7fad612ba700 1 -- 172.21.15.42:0/294499046 learned_addr learned my addr 172.21.15.42:0/294499046 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:45:57.148 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:57.129+0000 7fad61abb700 1 --2- 172.21.15.42:0/294499046 >> v2:172.21.15.40:3300/0 conn(0x7fad5c1429f0 0x7fad5c11fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:45:57.148 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:57.129+0000 7fad527fc700 1 -- 172.21.15.42:0/294499046 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 595530855 0 0) 0x7fad5c145340 con 0x7fad5c144ae0 2019-12-05T06:45:57.148 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:57.129+0000 7fad527fc700 1 -- 172.21.15.42:0/294499046 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad440030b0 con 0x7fad5c144ae0 2019-12-05T06:45:57.148 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:57.129+0000 7fad527fc700 1 -- 172.21.15.42:0/294499046 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2095207833 0 0) 0x7fad5c1200e0 con 0x7fad5c144f10 2019-12-05T06:45:57.148 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:57.129+0000 7fad527fc700 1 -- 172.21.15.42:0/294499046 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad5c145340 con 0x7fad5c144f10 2019-12-05T06:45:57.149 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:57.129+0000 7fad527fc700 1 -- 172.21.15.42:0/294499046 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 1582760471 0 0) 0x7fad440030b0 con 0x7fad5c144ae0 2019-12-05T06:45:57.149 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:57.129+0000 7fad61abb700 1 -- 172.21.15.42:0/294499046 >> v1:172.21.15.42:6789/0 conn(0x7fad5c144ae0 legacy=0x7fad5c11b020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:45:57.149 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:57.129+0000 7fad61abb700 1 -- 172.21.15.42:0/294499046 >> v1:172.21.15.40:6789/0 conn(0x7fad5c144f10 legacy=0x7fad5c0752e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:45:57.149 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:57.129+0000 7fad61abb700 1 -- 172.21.15.42:0/294499046 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad5c11a520 con 0x7fad5c1429f0 2019-12-05T06:45:57.149 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:57.129+0000 7fad61abb700 1 --2- 172.21.15.42:0/294499046 >> v2:172.21.15.40:3300/0 conn(0x7fad5c1429f0 0x7fad5c11fb90 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fad4c001bc0 tx=0x7fad4c0016d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:57.149 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:57.129+0000 7fad61abb700 1 --2- 172.21.15.42:0/294499046 >> v2:172.21.15.40:3300/0 conn(0x7fad5c1429f0 0x7fad5c11fb90 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fad4c001bc0 tx=0x7fad4c0016d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:57.149 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:57.129+0000 7fad61abb700 1 --2- 172.21.15.42:0/294499046 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fad5c1429f0 0x7fad5c11fb90 secure :-1 s=READY pgs=2413 cs=0 l=1 rx=0x7fad4c001bc0 tx=0x7fad4c0016d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:45:57.150 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:57.129+0000 7fad61abb700 1 --2- 172.21.15.42:0/294499046 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fad5c1429f0 0x7fad5c11fb90 secure :-1 s=THROTTLE_DONE pgs=2413 cs=0 l=1 rx=0x7fad4c001bc0 tx=0x7fad4c0016d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:57.150 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:57.129+0000 7fad527fc700 1 -- 172.21.15.42:0/294499046 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fad4c001980 con 0x7fad5c1429f0 2019-12-05T06:45:57.150 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:57.129+0000 7fad61abb700 1 --2- 172.21.15.42:0/294499046 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fad5c1429f0 0x7fad5c11fb90 secure :-1 s=THROTTLE_DONE pgs=2413 cs=0 l=1 rx=0x7fad4c001bc0 tx=0x7fad4c0016d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:57.150 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:57.129+0000 7fad527fc700 1 -- 172.21.15.42:0/294499046 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad4c001790 con 0x7fad5c1429f0 2019-12-05T06:45:57.150 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:57.129+0000 7fad61abb700 1 --2- 172.21.15.42:0/294499046 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fad5c1429f0 0x7fad5c11fb90 secure :-1 s=THROTTLE_DONE pgs=2413 cs=0 l=1 rx=0x7fad4c001bc0 tx=0x7fad4c0016d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:57.150 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:57.129+0000 7fad527fc700 1 -- 172.21.15.42:0/294499046 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fad5800f660 con 0x7fad5c1429f0 2019-12-05T06:45:57.151 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:57.129+0000 7fad6351e700 1 -- 172.21.15.42:0/294499046 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad5c11b820 con 0x7fad5c1429f0 2019-12-05T06:45:57.151 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:57.129+0000 7fad6351e700 1 -- 172.21.15.42:0/294499046 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad5c121800 con 0x7fad5c1429f0 2019-12-05T06:45:57.151 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:57.129+0000 7fad61abb700 1 --2- 172.21.15.42:0/294499046 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fad5c1429f0 0x7fad5c11fb90 secure :-1 s=THROTTLE_DONE pgs=2413 cs=0 l=1 rx=0x7fad4c001bc0 tx=0x7fad4c0016d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:57.151 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:57.129+0000 7fad527fc700 1 -- 172.21.15.42:0/294499046 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fad58005540 con 0x7fad5c1429f0 2019-12-05T06:45:57.151 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:57.129+0000 7fad61abb700 1 --2- 172.21.15.42:0/294499046 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fad5c1429f0 0x7fad5c11fb90 secure :-1 s=THROTTLE_DONE pgs=2413 cs=0 l=1 rx=0x7fad4c001bc0 tx=0x7fad4c0016d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:57.151 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:57.133+0000 7fad527fc700 1 --2- 172.21.15.42:0/294499046 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fad4403b460 0x7fad4403d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:45:57.152 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:57.133+0000 7fad527fc700 1 -- 172.21.15.42:0/294499046 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 12051+0+0 (secure 0 0 0) 0x7fad5804b160 con 0x7fad5c1429f0 2019-12-05T06:45:57.152 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:57.133+0000 7fad612ba700 1 --2- 172.21.15.42:0/294499046 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fad4403b460 0x7fad4403d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:45:57.152 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:57.133+0000 7fad612ba700 1 --2- 172.21.15.42:0/294499046 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fad4403b460 0x7fad4403d970 crc :-1 s=READY pgs=1333 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:45:57.152 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:57.133+0000 7fad6351e700 1 -- 172.21.15.42:0/294499046 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad5c05d110 con 0x7fad5c1429f0 2019-12-05T06:45:57.153 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:57.141+0000 7fad61abb700 1 --2- 172.21.15.42:0/294499046 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fad5c1429f0 0x7fad5c11fb90 secure :-1 s=THROTTLE_DONE pgs=2413 cs=0 l=1 rx=0x7fad4c001bc0 tx=0x7fad4c0016d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:57.153 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:57.141+0000 7fad527fc700 1 -- 172.21.15.42:0/294499046 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fad580321c0 con 0x7fad5c1429f0 2019-12-05T06:45:57.310 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:57.305+0000 7fad6351e700 1 -- 172.21.15.42:0/294499046 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fad5c0a8cb0 con 0x7fad5c1429f0 2019-12-05T06:45:57.311 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:57.305+0000 7fad61abb700 1 --2- 172.21.15.42:0/294499046 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fad5c1429f0 0x7fad5c11fb90 secure :-1 s=THROTTLE_DONE pgs=2413 cs=0 l=1 rx=0x7fad4c001bc0 tx=0x7fad4c0016d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:57.311 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:57.305+0000 7fad527fc700 1 -- 172.21.15.42:0/294499046 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 384 v384) v1 ==== 95+0+1778 (secure 0 0 0) 0x7fad58032420 con 0x7fad5c1429f0 2019-12-05T06:45:57.311 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 384 2019-12-05T06:45:57.311 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:45:57.311 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":384,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":384,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:45:55.302477+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":119,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12429},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12429":{"gid":12429,"name":"b","rank":0,"incarnation":384,"state":"up:replay","state_seq":1,"addr":"172.21.15.40:6833/4178556019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":4178556019},{"type":"v1","addr":"172.21.15.40:6833","nonce":4178556019}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:45:57.315 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:57.309+0000 7fad6351e700 1 -- 172.21.15.42:0/294499046 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fad4403b460 msgr2=0x7fad4403d970 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:45:57.315 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:57.309+0000 7fad6351e700 1 --2- 172.21.15.42:0/294499046 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fad4403b460 0x7fad4403d970 crc :-1 s=READY pgs=1333 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:45:57.315 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:57.309+0000 7fad6351e700 1 -- 172.21.15.42:0/294499046 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fad5c1429f0 msgr2=0x7fad5c11fb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:45:57.315 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:57.309+0000 7fad6351e700 1 --2- 172.21.15.42:0/294499046 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fad5c1429f0 0x7fad5c11fb90 secure :-1 s=READY pgs=2413 cs=0 l=1 rx=0x7fad4c001bc0 tx=0x7fad4c0016d0).stop 2019-12-05T06:45:57.315 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:57.309+0000 7fad6351e700 1 -- 172.21.15.42:0/294499046 shutdown_connections 2019-12-05T06:45:57.316 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:57.309+0000 7fad6351e700 1 --2- 172.21.15.42:0/294499046 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fad4403b460 0x7fad4403d970 unknown :-1 s=CLOSED pgs=1333 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:45:57.316 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:57.309+0000 7fad6351e700 1 --2- 172.21.15.42:0/294499046 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fad5c1429f0 0x7fad5c11fb90 unknown :-1 s=CLOSED pgs=2413 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:45:57.316 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:57.309+0000 7fad6351e700 1 -- 172.21.15.42:0/294499046 shutdown_connections 2019-12-05T06:45:57.316 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:57.309+0000 7fad6351e700 1 -- 172.21.15.42:0/294499046 wait complete. 2019-12-05T06:45:57.317 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:57.309+0000 7fad6351e700 1 -- 172.21.15.42:0/294499046 >> 172.21.15.42:0/294499046 conn(0x7fad5c0a9b50 msgr2=0x7fad5c11c560 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:45:57.331 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 384, "feature_flags": { "enable_multiple": false, "ever_enabled_multiple": false }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2019-12-05T05:03:35.604738+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 384, "ever_allowed_features": 2, "explicitly_allowed_features": 2, "failed": [], "flags": 18, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_12429": { "addr": "172.21.15.40:6833/4178556019", "addrs": { "addrvec": [ { "addr": "172.21.15.40:6832", "nonce": 4178556019, "type": "v2" }, { "addr": "172.21.15.40:6833", "nonce": 4178556019, "type": "v1" } ] }, "export_targets": [], "features": 4540138292836696063, "flags": 0, "gid": 12429, "incarnation": 384, "name": "b", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 119, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 2, "min_compat_client": "0 (unknown)", "modified": "2019-12-05T06:45:55.302477+0000", "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 12429 } } } ], "standbys": [] } 2019-12-05T06:45:59.332 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:45:59.441 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:59.429+0000 7f1968cdd700 1 Processor -- start 2019-12-05T06:45:59.441 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:59.429+0000 7f1968cdd700 1 -- start start 2019-12-05T06:45:59.442 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:59.429+0000 7f1968cdd700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f19640a7740 0x7f196408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:45:59.442 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:59.429+0000 7f1968cdd700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f196407e720 0x7f196407eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:45:59.442 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:59.429+0000 7f1968cdd700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f196409f2e0 0x7f196409f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:45:59.442 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:59.429+0000 7f1968cdd700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19640a0b00 con 0x7f196407e720 2019-12-05T06:45:59.442 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:59.429+0000 7f1968cdd700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f196408e5b0 con 0x7f19640a7740 2019-12-05T06:45:59.442 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:59.429+0000 7f1968cdd700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f196408e850 con 0x7f196409f2e0 2019-12-05T06:45:59.443 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:59.429+0000 7f1962d9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f19640a7740 0x7f196408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:45:59.443 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:59.429+0000 7f196259c700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f196407e720 0x7f196407eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:45:59.443 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:59.429+0000 7f1962d9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f19640a7740 0x7f196408df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:35818/0 (socket says 172.21.15.42:35818) 2019-12-05T06:45:59.443 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:59.429+0000 7f1962d9d700 1 -- 172.21.15.42:0/4075952697 learned_addr learned my addr 172.21.15.42:0/4075952697 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:45:59.443 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:59.429+0000 7f196359e700 1 --2- 172.21.15.42:0/4075952697 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f196409f2e0 0x7f196409f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:45:59.443 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:59.429+0000 7f196259c700 1 -- 172.21.15.42:0/4075952697 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f196409f2e0 msgr2=0x7f196409f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:45:59.444 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:59.429+0000 7f196259c700 1 --2- 172.21.15.42:0/4075952697 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f196409f2e0 0x7f196409f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:45:59.444 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:59.429+0000 7f196259c700 1 -- 172.21.15.42:0/4075952697 >> v2:172.21.15.42:3300/0 conn(0x7f19640a7740 msgr2=0x7f196408df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:45:59.444 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:59.429+0000 7f196259c700 1 --2- 172.21.15.42:0/4075952697 >> v2:172.21.15.42:3300/0 conn(0x7f19640a7740 0x7f196408df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:45:59.444 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:59.429+0000 7f196259c700 1 -- 172.21.15.42:0/4075952697 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f196407f0a0 con 0x7f196407e720 2019-12-05T06:45:59.444 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:59.429+0000 7f1962d9d700 1 --2- 172.21.15.42:0/4075952697 >> v2:172.21.15.42:3300/0 conn(0x7f19640a7740 0x7f196408df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:45:59.444 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:59.429+0000 7f196259c700 1 --2- 172.21.15.42:0/4075952697 >> v2:172.21.15.40:3300/0 conn(0x7f196407e720 0x7f196407eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f19640a0f50 tx=0x7f194c010070).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:59.444 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:59.429+0000 7f196259c700 1 --2- 172.21.15.42:0/4075952697 >> v2:172.21.15.40:3300/0 conn(0x7f196407e720 0x7f196407eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f19640a0f50 tx=0x7f194c010070).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:59.445 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:59.429+0000 7f196259c700 1 --2- 172.21.15.42:0/4075952697 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f196407e720 0x7f196407eb50 secure :-1 s=READY pgs=2414 cs=0 l=1 rx=0x7f19640a0f50 tx=0x7f194c010070).ready entity=mon.0 client_cookie=962eac904bce38dd server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:45:59.445 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:59.429+0000 7f196259c700 1 --2- 172.21.15.42:0/4075952697 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f196407e720 0x7f196407eb50 secure :-1 s=THROTTLE_DONE pgs=2414 cs=0 l=1 rx=0x7f19640a0f50 tx=0x7f194c010070).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:59.445 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:59.429+0000 7f196259c700 1 --2- 172.21.15.42:0/4075952697 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f196407e720 0x7f196407eb50 secure :-1 s=THROTTLE_DONE pgs=2414 cs=0 l=1 rx=0x7f19640a0f50 tx=0x7f194c010070).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:59.445 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:59.429+0000 7f196259c700 1 --2- 172.21.15.42:0/4075952697 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f196407e720 0x7f196407eb50 secure :-1 s=THROTTLE_DONE pgs=2414 cs=0 l=1 rx=0x7f19640a0f50 tx=0x7f194c010070).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:59.445 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:59.429+0000 7f1961d9b700 1 -- 172.21.15.42:0/4075952697 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f194c010940 con 0x7f196407e720 2019-12-05T06:45:59.445 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:59.429+0000 7f1961d9b700 1 -- 172.21.15.42:0/4075952697 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f194c008b50 con 0x7f196407e720 2019-12-05T06:45:59.446 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:59.429+0000 7f1961d9b700 1 -- 172.21.15.42:0/4075952697 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f194c0110c0 con 0x7f196407e720 2019-12-05T06:45:59.446 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:59.429+0000 7f1968cdd700 1 -- 172.21.15.42:0/4075952697 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f196407e720 msgr2=0x7f196407eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:45:59.446 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:59.429+0000 7f1968cdd700 1 --2- 172.21.15.42:0/4075952697 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f196407e720 0x7f196407eb50 secure :-1 s=READY pgs=2414 cs=0 l=1 rx=0x7f19640a0f50 tx=0x7f194c010070).stop 2019-12-05T06:45:59.446 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:59.433+0000 7f1968cdd700 1 -- 172.21.15.42:0/4075952697 shutdown_connections 2019-12-05T06:45:59.446 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:59.433+0000 7f1968cdd700 1 --2- 172.21.15.42:0/4075952697 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f196409f2e0 0x7f196409f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:45:59.446 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:59.433+0000 7f1968cdd700 1 --2- 172.21.15.42:0/4075952697 >> v2:172.21.15.42:3300/0 conn(0x7f19640a7740 0x7f196408df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:45:59.447 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:59.433+0000 7f1968cdd700 1 --2- 172.21.15.42:0/4075952697 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f196407e720 0x7f196407eb50 unknown :-1 s=CLOSED pgs=2414 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:45:59.447 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:59.433+0000 7f1968cdd700 1 -- 172.21.15.42:0/4075952697 shutdown_connections 2019-12-05T06:45:59.447 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:59.433+0000 7f1968cdd700 1 -- 172.21.15.42:0/4075952697 wait complete. 2019-12-05T06:45:59.447 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:59.433+0000 7f1968cdd700 1 -- 172.21.15.42:0/4075952697 >> 172.21.15.42:0/4075952697 conn(0x7f19640aabe0 msgr2=0x7f19640a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:45:59.447 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:59.433+0000 7f1968cdd700 1 Processor -- start 2019-12-05T06:45:59.447 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:59.433+0000 7f1968cdd700 1 -- start start 2019-12-05T06:45:59.447 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:59.433+0000 7f1968cdd700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f19641176b0 0x7f196411be50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:45:59.447 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:59.433+0000 7f1968cdd700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f196411a3c0 0x7f1964125960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:45:59.448 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:59.433+0000 7f1968cdd700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f196411a7f0 con 0x7f1964123d40 2019-12-05T06:45:59.448 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:59.433+0000 7f1968cdd700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1964116ed0 con 0x7f19641176b0 2019-12-05T06:45:59.448 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:59.433+0000 7f1968cdd700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1964117170 con 0x7f196411a3c0 2019-12-05T06:45:59.448 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:59.433+0000 7f1962d9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f19641176b0 0x7f196411be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:45:59.448 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:59.433+0000 7f196259c700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f196411a3c0 0x7f1964125960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:45:59.448 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:59.433+0000 7f196359e700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f1964123d40 0x7f1964118f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:35864/0 (socket says 172.21.15.42:35864) 2019-12-05T06:45:59.449 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:59.433+0000 7f196359e700 1 -- 172.21.15.42:0/3294926711 learned_addr learned my addr 172.21.15.42:0/3294926711 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:45:59.449 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:59.437+0000 7f196259c700 1 -- 172.21.15.42:0/3294926711 >> v2:172.21.15.42:3300/0 conn(0x7f19641176b0 msgr2=0x7f196411be50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:45:59.449 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:59.437+0000 7f196259c700 1 --2- 172.21.15.42:0/3294926711 >> v2:172.21.15.42:3300/0 conn(0x7f19641176b0 0x7f196411be50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:45:59.449 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:59.437+0000 7f196259c700 1 -- 172.21.15.42:0/3294926711 >> v1:172.21.15.42:6789/0 conn(0x7f1964123d40 legacy=0x7f1964118f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:45:59.449 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:59.437+0000 7f196259c700 1 -- 172.21.15.42:0/3294926711 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1964125eb0 con 0x7f196411a3c0 2019-12-05T06:45:59.449 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:59.437+0000 7f196259c700 1 --2- 172.21.15.42:0/3294926711 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f196411a3c0 0x7f1964125960 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f19641186a0 tx=0x7f194c010a40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:59.449 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:59.437+0000 7f196259c700 1 --2- 172.21.15.42:0/3294926711 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f196411a3c0 0x7f1964125960 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f19641186a0 tx=0x7f194c010a40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:59.450 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:59.437+0000 7f196259c700 1 --2- 172.21.15.42:0/3294926711 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f196411a3c0 0x7f1964125960 secure :-1 s=READY pgs=702 cs=0 l=1 rx=0x7f19641186a0 tx=0x7f194c010a40).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:45:59.450 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:59.437+0000 7f196259c700 1 --2- 172.21.15.42:0/3294926711 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f196411a3c0 0x7f1964125960 secure :-1 s=THROTTLE_DONE pgs=702 cs=0 l=1 rx=0x7f19641186a0 tx=0x7f194c010a40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:59.450 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:59.437+0000 7f195b7fe700 1 -- 172.21.15.42:0/3294926711 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f194c005ab0 con 0x7f196411a3c0 2019-12-05T06:45:59.450 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:59.437+0000 7f196259c700 1 --2- 172.21.15.42:0/3294926711 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f196411a3c0 0x7f1964125960 secure :-1 s=THROTTLE_DONE pgs=702 cs=0 l=1 rx=0x7f19641186a0 tx=0x7f194c010a40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:59.450 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:59.437+0000 7f196259c700 1 --2- 172.21.15.42:0/3294926711 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f196411a3c0 0x7f1964125960 secure :-1 s=THROTTLE_DONE pgs=702 cs=0 l=1 rx=0x7f19641186a0 tx=0x7f194c010a40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:59.450 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:59.437+0000 7f195b7fe700 1 -- 172.21.15.42:0/3294926711 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f194c010b40 con 0x7f196411a3c0 2019-12-05T06:45:59.450 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:59.437+0000 7f195b7fe700 1 -- 172.21.15.42:0/3294926711 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f194c00f680 con 0x7f196411a3c0 2019-12-05T06:45:59.451 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:59.437+0000 7f1968cdd700 1 -- 172.21.15.42:0/3294926711 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f196407c4d0 con 0x7f196411a3c0 2019-12-05T06:45:59.451 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:59.437+0000 7f1968cdd700 1 -- 172.21.15.42:0/3294926711 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1964124170 con 0x7f196411a3c0 2019-12-05T06:45:59.452 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:59.437+0000 7f196259c700 1 --2- 172.21.15.42:0/3294926711 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f196411a3c0 0x7f1964125960 secure :-1 s=THROTTLE_DONE pgs=702 cs=0 l=1 rx=0x7f19641186a0 tx=0x7f194c010a40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:59.452 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:59.437+0000 7f195b7fe700 1 -- 172.21.15.42:0/3294926711 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f194c0113d0 con 0x7f196411a3c0 2019-12-05T06:45:59.452 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:59.437+0000 7f196259c700 1 --2- 172.21.15.42:0/3294926711 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f196411a3c0 0x7f1964125960 secure :-1 s=THROTTLE_DONE pgs=702 cs=0 l=1 rx=0x7f19641186a0 tx=0x7f194c010a40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:59.453 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:59.437+0000 7f195b7fe700 1 --2- 172.21.15.42:0/3294926711 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f194803adf0 0x7f194803d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:45:59.453 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:59.437+0000 7f195b7fe700 1 -- 172.21.15.42:0/3294926711 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 12051+0+0 (secure 0 0 0) 0x7f194c08f000 con 0x7f196411a3c0 2019-12-05T06:45:59.453 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:59.437+0000 7f1962d9d700 1 --2- 172.21.15.42:0/3294926711 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f194803adf0 0x7f194803d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:45:59.453 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:59.441+0000 7f1962d9d700 1 --2- 172.21.15.42:0/3294926711 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f194803adf0 0x7f194803d300 crc :-1 s=READY pgs=1334 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:45:59.454 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:59.445+0000 7f1968cdd700 1 -- 172.21.15.42:0/3294926711 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f196405d110 con 0x7f196411a3c0 2019-12-05T06:45:59.459 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:59.453+0000 7f196259c700 1 --2- 172.21.15.42:0/3294926711 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f196411a3c0 0x7f1964125960 secure :-1 s=THROTTLE_DONE pgs=702 cs=0 l=1 rx=0x7f19641186a0 tx=0x7f194c010a40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:59.459 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:59.453+0000 7f195b7fe700 1 -- 172.21.15.42:0/3294926711 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f194c00f920 con 0x7f196411a3c0 2019-12-05T06:45:59.633 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:59.629+0000 7f1968cdd700 1 -- 172.21.15.42:0/3294926711 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f196405b8c0 con 0x7f196411a3c0 2019-12-05T06:45:59.633 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:59.629+0000 7f196259c700 1 --2- 172.21.15.42:0/3294926711 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f196411a3c0 0x7f1964125960 secure :-1 s=THROTTLE_DONE pgs=702 cs=0 l=1 rx=0x7f19641186a0 tx=0x7f194c010a40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:59.634 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:59.629+0000 7f195b7fe700 1 -- 172.21.15.42:0/3294926711 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 384 v384) v1 ==== 95+0+1778 (secure 0 0 0) 0x7f194c011cd0 con 0x7f196411a3c0 2019-12-05T06:45:59.635 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:45:59.636 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":384,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":384,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:45:55.302477+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":119,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12429},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12429":{"gid":12429,"name":"b","rank":0,"incarnation":384,"state":"up:replay","state_seq":1,"addr":"172.21.15.40:6833/4178556019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":4178556019},{"type":"v1","addr":"172.21.15.40:6833","nonce":4178556019}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:45:59.636 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 384 2019-12-05T06:45:59.639 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:59.633+0000 7f1968cdd700 1 -- 172.21.15.42:0/3294926711 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f194803adf0 msgr2=0x7f194803d300 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:45:59.639 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:59.633+0000 7f1968cdd700 1 --2- 172.21.15.42:0/3294926711 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f194803adf0 0x7f194803d300 crc :-1 s=READY pgs=1334 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:45:59.639 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:59.633+0000 7f1968cdd700 1 -- 172.21.15.42:0/3294926711 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f196411a3c0 msgr2=0x7f1964125960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:45:59.639 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:59.633+0000 7f1968cdd700 1 --2- 172.21.15.42:0/3294926711 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f196411a3c0 0x7f1964125960 secure :-1 s=READY pgs=702 cs=0 l=1 rx=0x7f19641186a0 tx=0x7f194c010a40).stop 2019-12-05T06:45:59.640 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:59.633+0000 7f1968cdd700 1 -- 172.21.15.42:0/3294926711 shutdown_connections 2019-12-05T06:45:59.640 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:59.633+0000 7f1968cdd700 1 --2- 172.21.15.42:0/3294926711 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f194803adf0 0x7f194803d300 unknown :-1 s=CLOSED pgs=1334 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:45:59.640 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:59.633+0000 7f1968cdd700 1 --2- 172.21.15.42:0/3294926711 >> v2:172.21.15.42:3300/0 conn(0x7f19641176b0 0x7f196411be50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:45:59.640 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:59.633+0000 7f1968cdd700 1 --2- 172.21.15.42:0/3294926711 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f196411a3c0 0x7f1964125960 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:45:59.641 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:59.633+0000 7f1968cdd700 1 -- 172.21.15.42:0/3294926711 shutdown_connections 2019-12-05T06:45:59.641 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:59.633+0000 7f1968cdd700 1 -- 172.21.15.42:0/3294926711 wait complete. 2019-12-05T06:45:59.641 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:45:59.633+0000 7f1968cdd700 1 -- 172.21.15.42:0/3294926711 >> 172.21.15.42:0/3294926711 conn(0x7f196407e5d0 msgr2=0x7f196411cbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:45:59.649 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2019-12-05T06:46:01.650 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:46:01.760 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:01.749+0000 7f5141722700 1 Processor -- start 2019-12-05T06:46:01.760 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:01.749+0000 7f5141722700 1 -- start start 2019-12-05T06:46:01.760 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:01.749+0000 7f5141722700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f513c0a7740 0x7f513c08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:46:01.760 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:01.749+0000 7f5141722700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f513c09fae0 0x7f513c07eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:46:01.760 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:01.749+0000 7f5141722700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f513c0a0710 con 0x7f513c07e720 2019-12-05T06:46:01.761 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:01.749+0000 7f5141722700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f513c0a0b00 con 0x7f513c09fae0 2019-12-05T06:46:01.761 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:01.749+0000 7f5141722700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f513c08e5b0 con 0x7f513c0a7740 2019-12-05T06:46:01.761 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:01.749+0000 7f513b7fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f513c0a7740 0x7f513c08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:46:01.761 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:01.749+0000 7f513b7fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f513c0a7740 0x7f513c08df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:35840/0 (socket says 172.21.15.42:35840) 2019-12-05T06:46:01.761 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:01.749+0000 7f513b7fe700 1 -- 172.21.15.42:0/308417308 learned_addr learned my addr 172.21.15.42:0/308417308 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:46:01.761 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:01.749+0000 7f513bfff700 1 --2- 172.21.15.42:0/308417308 >> v2:172.21.15.40:3300/0 conn(0x7f513c09fae0 0x7f513c07eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:46:01.762 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:01.753+0000 7f513bfff700 1 -- 172.21.15.42:0/308417308 >> v2:172.21.15.42:3300/0 conn(0x7f513c0a7740 msgr2=0x7f513c08df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:46:01.762 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:01.753+0000 7f513bfff700 1 --2- 172.21.15.42:0/308417308 >> v2:172.21.15.42:3300/0 conn(0x7f513c0a7740 0x7f513c08df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:46:01.762 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:01.753+0000 7f513bfff700 1 -- 172.21.15.42:0/308417308 >> v1:172.21.15.42:6789/0 conn(0x7f513c07e720 legacy=0x7f513c09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:46:01.762 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:01.753+0000 7f513bfff700 1 -- 172.21.15.42:0/308417308 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f513c07f0a0 con 0x7f513c09fae0 2019-12-05T06:46:01.762 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:01.753+0000 7f513b7fe700 1 --2- 172.21.15.42:0/308417308 >> v2:172.21.15.42:3300/0 conn(0x7f513c0a7740 0x7f513c08df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:46:01.762 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:01.753+0000 7f513bfff700 1 --2- 172.21.15.42:0/308417308 >> v2:172.21.15.40:3300/0 conn(0x7f513c09fae0 0x7f513c07eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f5134002f30 tx=0x7f513400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:46:01.763 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:01.753+0000 7f513bfff700 1 --2- 172.21.15.42:0/308417308 >> v2:172.21.15.40:3300/0 conn(0x7f513c09fae0 0x7f513c07eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f5134002f30 tx=0x7f513400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:46:01.763 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:01.753+0000 7f513bfff700 1 --2- 172.21.15.42:0/308417308 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f513c09fae0 0x7f513c07eb50 secure :-1 s=READY pgs=2417 cs=0 l=1 rx=0x7f5134002f30 tx=0x7f513400fc70).ready entity=mon.0 client_cookie=b96420b553b12052 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:46:01.763 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:01.753+0000 7f513bfff700 1 --2- 172.21.15.42:0/308417308 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f513c09fae0 0x7f513c07eb50 secure :-1 s=THROTTLE_DONE pgs=2417 cs=0 l=1 rx=0x7f5134002f30 tx=0x7f513400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:46:01.763 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:01.753+0000 7f513bfff700 1 --2- 172.21.15.42:0/308417308 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f513c09fae0 0x7f513c07eb50 secure :-1 s=THROTTLE_DONE pgs=2417 cs=0 l=1 rx=0x7f5134002f30 tx=0x7f513400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:46:01.763 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:01.753+0000 7f513a7fc700 1 -- 172.21.15.42:0/308417308 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5134008bb0 con 0x7f513c09fae0 2019-12-05T06:46:01.763 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:01.753+0000 7f513bfff700 1 --2- 172.21.15.42:0/308417308 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f513c09fae0 0x7f513c07eb50 secure :-1 s=THROTTLE_DONE pgs=2417 cs=0 l=1 rx=0x7f5134002f30 tx=0x7f513400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:46:01.764 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:01.753+0000 7f513a7fc700 1 -- 172.21.15.42:0/308417308 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51340087d0 con 0x7f513c09fae0 2019-12-05T06:46:01.764 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:01.753+0000 7f513a7fc700 1 -- 172.21.15.42:0/308417308 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5134011820 con 0x7f513c09fae0 2019-12-05T06:46:01.764 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:01.753+0000 7f5141722700 1 -- 172.21.15.42:0/308417308 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f513c09fae0 msgr2=0x7f513c07eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:46:01.764 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:01.753+0000 7f5141722700 1 --2- 172.21.15.42:0/308417308 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f513c09fae0 0x7f513c07eb50 secure :-1 s=READY pgs=2417 cs=0 l=1 rx=0x7f5134002f30 tx=0x7f513400fc70).stop 2019-12-05T06:46:01.764 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:01.753+0000 7f5141722700 1 -- 172.21.15.42:0/308417308 shutdown_connections 2019-12-05T06:46:01.764 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:01.753+0000 7f5141722700 1 --2- 172.21.15.42:0/308417308 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f513c09fae0 0x7f513c07eb50 unknown :-1 s=CLOSED pgs=2417 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:46:01.765 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:01.753+0000 7f5141722700 1 --2- 172.21.15.42:0/308417308 >> v2:172.21.15.42:3300/0 conn(0x7f513c0a7740 0x7f513c08df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:46:01.765 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:01.753+0000 7f5141722700 1 -- 172.21.15.42:0/308417308 shutdown_connections 2019-12-05T06:46:01.765 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:01.753+0000 7f5141722700 1 -- 172.21.15.42:0/308417308 wait complete. 2019-12-05T06:46:01.765 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:01.753+0000 7f5141722700 1 -- 172.21.15.42:0/308417308 >> 172.21.15.42:0/308417308 conn(0x7f513c0aabe0 msgr2=0x7f513c0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:46:01.769 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:01.757+0000 7f5141722700 1 Processor -- start 2019-12-05T06:46:01.770 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:01.757+0000 7f5141722700 1 -- start start 2019-12-05T06:46:01.770 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:01.757+0000 7f5141722700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f513c07c810 0x7f513c126960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:46:01.770 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:01.757+0000 7f5141722700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f513c08ec20 con 0x7f513c126eb0 2019-12-05T06:46:01.770 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:01.757+0000 7f5141722700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f513c125340 con 0x7f513c116500 2019-12-05T06:46:01.770 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:01.757+0000 7f5141722700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f513c11ad90 con 0x7f513c07c810 2019-12-05T06:46:01.771 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:01.757+0000 7f513affd700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f513c07c810 0x7f513c126960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:46:01.771 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:01.757+0000 7f513b7fe700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f513c116500 0x7f513c07c010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:35880/0 (socket says 172.21.15.42:35880) 2019-12-05T06:46:01.771 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:01.757+0000 7f513b7fe700 1 -- 172.21.15.42:0/3872751160 learned_addr learned my addr 172.21.15.42:0/3872751160 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:46:01.771 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:01.757+0000 7f513b7fe700 1 -- 172.21.15.42:0/3872751160 >> v1:172.21.15.42:6789/0 conn(0x7f513c116500 legacy=0x7f513c07c010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2019-12-05T06:46:01.771 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:01.757+0000 7f513b7fe700 1 -- 172.21.15.42:0/3872751160 >> v1:172.21.15.42:6789/0 conn(0x7f513c116500 legacy=0x7f513c07c010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2019-12-05T06:46:01.771 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:01.757+0000 7f513b7fe700 1 --1- 172.21.15.42:0/3872751160 >> v1:172.21.15.42:6789/0 conn(0x7f513c116500 0x7f513c07c010 :-1 s=OPENED pgs=3187 cs=1 l=1).handle_message read tag failed 2019-12-05T06:46:01.772 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:01.757+0000 7f513b7fe700 1 --1- 172.21.15.42:0/3872751160 >> v1:172.21.15.42:6789/0 conn(0x7f513c116500 0x7f513c07c010 :-1 s=OPENED pgs=3187 cs=1 l=1).fault on lossy channel, failing 2019-12-05T06:46:01.772 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:01.757+0000 7f5123fff700 1 -- 172.21.15.42:0/3872751160 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2865445853 0 0) 0x7f513c08ec20 con 0x7f513c126eb0 2019-12-05T06:46:01.772 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:01.757+0000 7f5123fff700 1 -- 172.21.15.42:0/3872751160 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f511c0030b0 con 0x7f513c126eb0 2019-12-05T06:46:01.772 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:01.757+0000 7f513affd700 1 -- 172.21.15.42:0/3872751160 >> v1:172.21.15.42:6789/0 conn(0x7f513c116500 legacy=0x7f513c07c010 unknown :-1 s=STATE_CLOSED l=1).mark_down 2019-12-05T06:46:01.773 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:01.757+0000 7f513affd700 1 -- 172.21.15.42:0/3872751160 >> v1:172.21.15.40:6789/0 conn(0x7f513c126eb0 legacy=0x7f513c12dd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:46:01.773 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:01.757+0000 7f513affd700 1 -- 172.21.15.42:0/3872751160 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f513c08f7d0 con 0x7f513c07c810 2019-12-05T06:46:01.773 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:01.757+0000 7f513affd700 1 --2- 172.21.15.42:0/3872751160 >> v2:172.21.15.42:3300/0 conn(0x7f513c07c810 0x7f513c126960 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f5130000e90 tx=0x7f513000c2f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:46:01.773 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:01.757+0000 7f513affd700 1 --2- 172.21.15.42:0/3872751160 >> v2:172.21.15.42:3300/0 conn(0x7f513c07c810 0x7f513c126960 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f5130000e90 tx=0x7f513000c2f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:46:01.773 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:01.757+0000 7f513affd700 1 --2- 172.21.15.42:0/3872751160 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f513c07c810 0x7f513c126960 secure :-1 s=READY pgs=3188 cs=0 l=1 rx=0x7f5130000e90 tx=0x7f513000c2f0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:46:01.773 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:01.757+0000 7f513affd700 1 --2- 172.21.15.42:0/3872751160 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f513c07c810 0x7f513c126960 secure :-1 s=THROTTLE_DONE pgs=3188 cs=0 l=1 rx=0x7f5130000e90 tx=0x7f513000c2f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:46:01.774 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:01.757+0000 7f5123fff700 1 -- 172.21.15.42:0/3872751160 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f513000cb90 con 0x7f513c07c810 2019-12-05T06:46:01.774 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:01.757+0000 7f513affd700 1 --2- 172.21.15.42:0/3872751160 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f513c07c810 0x7f513c126960 secure :-1 s=THROTTLE_DONE pgs=3188 cs=0 l=1 rx=0x7f5130000e90 tx=0x7f513000c2f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:46:01.774 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:01.757+0000 7f513affd700 1 --2- 172.21.15.42:0/3872751160 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f513c07c810 0x7f513c126960 secure :-1 s=THROTTLE_DONE pgs=3188 cs=0 l=1 rx=0x7f5130000e90 tx=0x7f513000c2f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:46:01.775 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:01.757+0000 7f5123fff700 1 -- 172.21.15.42:0/3872751160 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f513000c7b0 con 0x7f513c07c810 2019-12-05T06:46:01.775 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:01.757+0000 7f5141722700 1 -- 172.21.15.42:0/3872751160 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f513c11a8f0 con 0x7f513c07c810 2019-12-05T06:46:01.775 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:01.757+0000 7f5123fff700 1 -- 172.21.15.42:0/3872751160 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5130011030 con 0x7f513c07c810 2019-12-05T06:46:01.776 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:01.757+0000 7f5141722700 1 -- 172.21.15.42:0/3872751160 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f513c1272e0 con 0x7f513c07c810 2019-12-05T06:46:01.776 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:01.761+0000 7f513affd700 1 --2- 172.21.15.42:0/3872751160 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f513c07c810 0x7f513c126960 secure :-1 s=THROTTLE_DONE pgs=3188 cs=0 l=1 rx=0x7f5130000e90 tx=0x7f513000c2f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:46:01.776 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:01.761+0000 7f5123fff700 1 -- 172.21.15.42:0/3872751160 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f5130011340 con 0x7f513c07c810 2019-12-05T06:46:01.777 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:01.761+0000 7f513affd700 1 --2- 172.21.15.42:0/3872751160 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f513c07c810 0x7f513c126960 secure :-1 s=THROTTLE_DONE pgs=3188 cs=0 l=1 rx=0x7f5130000e90 tx=0x7f513000c2f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:46:01.777 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:01.761+0000 7f5123fff700 1 --2- 172.21.15.42:0/3872751160 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f511c03b200 0x7f511c03d710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:46:01.777 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:01.761+0000 7f5123fff700 1 -- 172.21.15.42:0/3872751160 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 12051+0+0 (secure 0 0 0) 0x7f513004b160 con 0x7f513c07c810 2019-12-05T06:46:01.777 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:01.761+0000 7f513b7fe700 1 --2- 172.21.15.42:0/3872751160 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f511c03b200 0x7f511c03d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:46:01.778 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:01.761+0000 7f513b7fe700 1 --2- 172.21.15.42:0/3872751160 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f511c03b200 0x7f511c03d710 crc :-1 s=READY pgs=1336 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:46:01.778 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:01.765+0000 7f5141722700 1 -- 172.21.15.42:0/3872751160 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f513c05d110 con 0x7f513c07c810 2019-12-05T06:46:01.796 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:01.773+0000 7f513affd700 1 --2- 172.21.15.42:0/3872751160 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f513c07c810 0x7f513c126960 secure :-1 s=THROTTLE_DONE pgs=3188 cs=0 l=1 rx=0x7f5130000e90 tx=0x7f513000c2f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:46:01.797 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:01.773+0000 7f5123fff700 1 -- 172.21.15.42:0/3872751160 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f51300321e0 con 0x7f513c07c810 2019-12-05T06:46:01.947 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:01.941+0000 7f5141722700 1 -- 172.21.15.42:0/3872751160 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f513c05b8c0 con 0x7f513c07c810 2019-12-05T06:46:01.948 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:01.945+0000 7f513affd700 1 --2- 172.21.15.42:0/3872751160 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f513c07c810 0x7f513c126960 secure :-1 s=THROTTLE_DONE pgs=3188 cs=0 l=1 rx=0x7f5130000e90 tx=0x7f513000c2f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:46:01.948 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:01.945+0000 7f5123fff700 1 -- 172.21.15.42:0/3872751160 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 384 v384) v1 ==== 95+0+1778 (secure 0 0 0) 0x7f5130032440 con 0x7f513c07c810 2019-12-05T06:46:01.949 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 384 2019-12-05T06:46:01.949 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:46:01.949 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":384,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":384,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:45:55.302477+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":119,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12429},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12429":{"gid":12429,"name":"b","rank":0,"incarnation":384,"state":"up:replay","state_seq":1,"addr":"172.21.15.40:6833/4178556019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":4178556019},{"type":"v1","addr":"172.21.15.40:6833","nonce":4178556019}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:46:01.952 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:01.945+0000 7f5141722700 1 -- 172.21.15.42:0/3872751160 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f511c03b200 msgr2=0x7f511c03d710 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:46:01.953 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:01.945+0000 7f5141722700 1 --2- 172.21.15.42:0/3872751160 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f511c03b200 0x7f511c03d710 crc :-1 s=READY pgs=1336 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:46:01.953 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:01.945+0000 7f5141722700 1 -- 172.21.15.42:0/3872751160 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f513c07c810 msgr2=0x7f513c126960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:46:01.954 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:01.945+0000 7f5141722700 1 --2- 172.21.15.42:0/3872751160 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f513c07c810 0x7f513c126960 secure :-1 s=READY pgs=3188 cs=0 l=1 rx=0x7f5130000e90 tx=0x7f513000c2f0).stop 2019-12-05T06:46:01.954 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:01.945+0000 7f5141722700 1 -- 172.21.15.42:0/3872751160 shutdown_connections 2019-12-05T06:46:01.954 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:01.945+0000 7f5141722700 1 --2- 172.21.15.42:0/3872751160 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f511c03b200 0x7f511c03d710 unknown :-1 s=CLOSED pgs=1336 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:46:01.954 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:01.945+0000 7f5141722700 1 --2- 172.21.15.42:0/3872751160 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f513c07c810 0x7f513c126960 unknown :-1 s=CLOSED pgs=3188 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:46:01.954 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:01.945+0000 7f5141722700 1 -- 172.21.15.42:0/3872751160 shutdown_connections 2019-12-05T06:46:01.954 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:01.945+0000 7f5141722700 1 -- 172.21.15.42:0/3872751160 wait complete. 2019-12-05T06:46:01.954 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:01.945+0000 7f5141722700 1 -- 172.21.15.42:0/3872751160 >> 172.21.15.42:0/3872751160 conn(0x7f513c0a82a0 msgr2=0x7f513c059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:46:01.961 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2019-12-05T06:46:03.962 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:46:04.068 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:04.057+0000 7fbe99a84700 1 Processor -- start 2019-12-05T06:46:04.068 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:04.057+0000 7fbe99a84700 1 -- start start 2019-12-05T06:46:04.068 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:04.057+0000 7fbe99a84700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fbe940a7740 0x7fbe9408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:46:04.069 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:04.057+0000 7fbe99a84700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbe9407e720 0x7fbe9407eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:46:04.069 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:04.057+0000 7fbe99a84700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe940a0710 con 0x7fbe9409f2e0 2019-12-05T06:46:04.069 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:04.057+0000 7fbe99a84700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe940a0b00 con 0x7fbe940a7740 2019-12-05T06:46:04.069 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:04.057+0000 7fbe99a84700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe9408e5b0 con 0x7fbe9407e720 2019-12-05T06:46:04.070 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:04.057+0000 7fbe8b7fe700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbe9407e720 0x7fbe9407eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:46:04.070 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:04.057+0000 7fbe93fff700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fbe940a7740 0x7fbe9408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:46:04.070 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:04.057+0000 7fbe8b7fe700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbe9407e720 0x7fbe9407eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:36328/0 (socket says 172.21.15.42:36328) 2019-12-05T06:46:04.071 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:04.057+0000 7fbe8b7fe700 1 -- 172.21.15.42:0/1357556520 learned_addr learned my addr 172.21.15.42:0/1357556520 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:46:04.071 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:04.057+0000 7fbe8b7fe700 1 -- 172.21.15.42:0/1357556520 >> v2:172.21.15.40:3300/0 conn(0x7fbe940a7740 msgr2=0x7fbe9408df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:46:04.072 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:04.057+0000 7fbe8b7fe700 1 --2- 172.21.15.42:0/1357556520 >> v2:172.21.15.40:3300/0 conn(0x7fbe940a7740 0x7fbe9408df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:46:04.072 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:04.057+0000 7fbe8b7fe700 1 -- 172.21.15.42:0/1357556520 >> v1:172.21.15.40:6789/0 conn(0x7fbe9409f2e0 legacy=0x7fbe9409f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:46:04.072 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:04.057+0000 7fbe8b7fe700 1 -- 172.21.15.42:0/1357556520 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe9407f0a0 con 0x7fbe9407e720 2019-12-05T06:46:04.072 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:04.057+0000 7fbe93fff700 1 --2- 172.21.15.42:0/1357556520 >> v2:172.21.15.40:3300/0 conn(0x7fbe940a7740 0x7fbe9408df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:46:04.073 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:04.061+0000 7fbe8b7fe700 1 --2- 172.21.15.42:0/1357556520 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbe9407e720 0x7fbe9407eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fbe7c008790 tx=0x7fbe7c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:46:04.073 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:04.061+0000 7fbe8b7fe700 1 --2- 172.21.15.42:0/1357556520 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbe9407e720 0x7fbe9407eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fbe7c008790 tx=0x7fbe7c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:46:04.073 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:04.061+0000 7fbe8b7fe700 1 --2- 172.21.15.42:0/1357556520 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbe9407e720 0x7fbe9407eb50 secure :-1 s=READY pgs=703 cs=0 l=1 rx=0x7fbe7c008790 tx=0x7fbe7c00fd30).ready entity=mon.2 client_cookie=f7a4d5cf6f12cca3 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:46:04.073 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:04.061+0000 7fbe8b7fe700 1 --2- 172.21.15.42:0/1357556520 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbe9407e720 0x7fbe9407eb50 secure :-1 s=THROTTLE_DONE pgs=703 cs=0 l=1 rx=0x7fbe7c008790 tx=0x7fbe7c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:46:04.074 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:04.061+0000 7fbe937fe700 1 -- 172.21.15.42:0/1357556520 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbe7c0113b0 con 0x7fbe9407e720 2019-12-05T06:46:04.074 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:04.061+0000 7fbe8b7fe700 1 --2- 172.21.15.42:0/1357556520 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbe9407e720 0x7fbe9407eb50 secure :-1 s=THROTTLE_DONE pgs=703 cs=0 l=1 rx=0x7fbe7c008790 tx=0x7fbe7c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:46:04.074 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:04.061+0000 7fbe937fe700 1 -- 172.21.15.42:0/1357556520 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe7c008bb0 con 0x7fbe9407e720 2019-12-05T06:46:04.074 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:04.061+0000 7fbe8b7fe700 1 --2- 172.21.15.42:0/1357556520 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbe9407e720 0x7fbe9407eb50 secure :-1 s=THROTTLE_DONE pgs=703 cs=0 l=1 rx=0x7fbe7c008790 tx=0x7fbe7c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:46:04.074 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:04.061+0000 7fbe937fe700 1 -- 172.21.15.42:0/1357556520 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbe7c011990 con 0x7fbe9407e720 2019-12-05T06:46:04.075 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:04.061+0000 7fbe99a84700 1 -- 172.21.15.42:0/1357556520 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbe9407e720 msgr2=0x7fbe9407eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:46:04.075 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:04.061+0000 7fbe99a84700 1 --2- 172.21.15.42:0/1357556520 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbe9407e720 0x7fbe9407eb50 secure :-1 s=READY pgs=703 cs=0 l=1 rx=0x7fbe7c008790 tx=0x7fbe7c00fd30).stop 2019-12-05T06:46:04.075 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:04.061+0000 7fbe99a84700 1 -- 172.21.15.42:0/1357556520 shutdown_connections 2019-12-05T06:46:04.075 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:04.061+0000 7fbe99a84700 1 --2- 172.21.15.42:0/1357556520 >> v2:172.21.15.40:3300/0 conn(0x7fbe940a7740 0x7fbe9408df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:46:04.075 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:04.061+0000 7fbe99a84700 1 --2- 172.21.15.42:0/1357556520 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbe9407e720 0x7fbe9407eb50 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:46:04.075 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:04.061+0000 7fbe99a84700 1 -- 172.21.15.42:0/1357556520 shutdown_connections 2019-12-05T06:46:04.076 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:04.061+0000 7fbe99a84700 1 -- 172.21.15.42:0/1357556520 wait complete. 2019-12-05T06:46:04.076 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:04.061+0000 7fbe99a84700 1 -- 172.21.15.42:0/1357556520 >> 172.21.15.42:0/1357556520 conn(0x7fbe940aabe0 msgr2=0x7fbe940a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:46:04.081 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:04.065+0000 7fbe99a84700 1 Processor -- start 2019-12-05T06:46:04.081 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:04.065+0000 7fbe99a84700 1 -- start start 2019-12-05T06:46:04.081 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:04.065+0000 7fbe99a84700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbe9411bf30 0x7fbe9407ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:46:04.082 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:04.065+0000 7fbe99a84700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fbe940a6860 0x7fbe9407af50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:46:04.082 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:04.065+0000 7fbe99a84700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fbe940a56a0 0x7fbe9407b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:46:04.082 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:04.065+0000 7fbe99a84700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe940a1890 con 0x7fbe940a56a0 2019-12-05T06:46:04.082 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:04.065+0000 7fbe99a84700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe940a5400 con 0x7fbe940a6860 2019-12-05T06:46:04.082 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:04.065+0000 7fbe99a84700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe940a1510 con 0x7fbe9411bf30 2019-12-05T06:46:04.082 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:04.065+0000 7fbe8b7fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fbe940a6860 0x7fbe9407af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:46:04.083 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:04.065+0000 7fbe93fff700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbe9411bf30 0x7fbe9407ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:46:04.083 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:04.065+0000 7fbe98822700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fbe940a56a0 0x7fbe9407b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:46:04.083 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:04.065+0000 7fbe8b7fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fbe940a6860 0x7fbe9407af50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:35870/0 (socket says 172.21.15.42:35870) 2019-12-05T06:46:04.083 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:04.065+0000 7fbe93fff700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbe9411bf30 0x7fbe9407ced0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:36334/0 (socket says 172.21.15.42:36334) 2019-12-05T06:46:04.083 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:04.065+0000 7fbe8b7fe700 1 -- 172.21.15.42:0/1915813478 learned_addr learned my addr 172.21.15.42:0/1915813478 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:46:04.083 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:04.065+0000 7fbe8b7fe700 1 -- 172.21.15.42:0/1915813478 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbe9411bf30 msgr2=0x7fbe9407ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:46:04.083 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:04.065+0000 7fbe8b7fe700 1 --2- 172.21.15.42:0/1915813478 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbe9411bf30 0x7fbe9407ced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:46:04.084 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:04.065+0000 7fbe8b7fe700 1 -- 172.21.15.42:0/1915813478 >> v2:172.21.15.40:3300/0 conn(0x7fbe940a56a0 msgr2=0x7fbe9407b4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:46:04.084 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:04.065+0000 7fbe8b7fe700 1 --2- 172.21.15.42:0/1915813478 >> v2:172.21.15.40:3300/0 conn(0x7fbe940a56a0 0x7fbe9407b4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:46:04.084 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:04.065+0000 7fbe8b7fe700 1 -- 172.21.15.42:0/1915813478 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe9407d420 con 0x7fbe940a6860 2019-12-05T06:46:04.084 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:04.065+0000 7fbe98822700 1 --2- 172.21.15.42:0/1915813478 >> v2:172.21.15.40:3300/0 conn(0x7fbe940a56a0 0x7fbe9407b4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:46:04.084 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:04.069+0000 7fbe8b7fe700 1 --2- 172.21.15.42:0/1915813478 >> v2:172.21.15.42:3300/0 conn(0x7fbe940a6860 0x7fbe9407af50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fbe940a55c0 tx=0x7fbe7c008bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:46:04.085 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:04.069+0000 7fbe8b7fe700 1 --2- 172.21.15.42:0/1915813478 >> v2:172.21.15.42:3300/0 conn(0x7fbe940a6860 0x7fbe9407af50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fbe940a55c0 tx=0x7fbe7c008bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:46:04.085 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:04.069+0000 7fbe8b7fe700 1 --2- 172.21.15.42:0/1915813478 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbe940a6860 0x7fbe9407af50 secure :-1 s=READY pgs=3189 cs=0 l=1 rx=0x7fbe940a55c0 tx=0x7fbe7c008bb0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:46:04.085 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:04.069+0000 7fbe8b7fe700 1 --2- 172.21.15.42:0/1915813478 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbe940a6860 0x7fbe9407af50 secure :-1 s=THROTTLE_DONE pgs=3189 cs=0 l=1 rx=0x7fbe940a55c0 tx=0x7fbe7c008bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:46:04.085 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:04.069+0000 7fbe917fa700 1 -- 172.21.15.42:0/1915813478 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbe7c007340 con 0x7fbe940a6860 2019-12-05T06:46:04.085 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:04.069+0000 7fbe8b7fe700 1 --2- 172.21.15.42:0/1915813478 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbe940a6860 0x7fbe9407af50 secure :-1 s=THROTTLE_DONE pgs=3189 cs=0 l=1 rx=0x7fbe940a55c0 tx=0x7fbe7c008bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:46:04.085 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:04.069+0000 7fbe8b7fe700 1 --2- 172.21.15.42:0/1915813478 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbe940a6860 0x7fbe9407af50 secure :-1 s=THROTTLE_DONE pgs=3189 cs=0 l=1 rx=0x7fbe940a55c0 tx=0x7fbe7c008bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:46:04.085 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:04.069+0000 7fbe917fa700 1 -- 172.21.15.42:0/1915813478 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe7c0115b0 con 0x7fbe940a6860 2019-12-05T06:46:04.086 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:04.069+0000 7fbe99a84700 1 -- 172.21.15.42:0/1915813478 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe940a4de0 con 0x7fbe940a6860 2019-12-05T06:46:04.086 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:04.069+0000 7fbe917fa700 1 -- 172.21.15.42:0/1915813478 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbe7c005540 con 0x7fbe940a6860 2019-12-05T06:46:04.086 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:04.069+0000 7fbe99a84700 1 -- 172.21.15.42:0/1915813478 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe9407bcd0 con 0x7fbe940a6860 2019-12-05T06:46:04.086 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:04.069+0000 7fbe8b7fe700 1 --2- 172.21.15.42:0/1915813478 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbe940a6860 0x7fbe9407af50 secure :-1 s=THROTTLE_DONE pgs=3189 cs=0 l=1 rx=0x7fbe940a55c0 tx=0x7fbe7c008bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:46:04.086 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:04.069+0000 7fbe917fa700 1 -- 172.21.15.42:0/1915813478 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fbe7c005810 con 0x7fbe940a6860 2019-12-05T06:46:04.086 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:04.069+0000 7fbe8b7fe700 1 --2- 172.21.15.42:0/1915813478 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbe940a6860 0x7fbe9407af50 secure :-1 s=THROTTLE_DONE pgs=3189 cs=0 l=1 rx=0x7fbe940a55c0 tx=0x7fbe7c008bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:46:04.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:04.069+0000 7fbe917fa700 1 --2- 172.21.15.42:0/1915813478 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbe7803ae90 0x7fbe7803d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:46:04.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:04.069+0000 7fbe917fa700 1 -- 172.21.15.42:0/1915813478 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 12051+0+0 (secure 0 0 0) 0x7fbe7c04b1c0 con 0x7fbe940a6860 2019-12-05T06:46:04.088 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:04.069+0000 7fbe93fff700 1 --2- 172.21.15.42:0/1915813478 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbe7803ae90 0x7fbe7803d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:46:04.088 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:04.069+0000 7fbe93fff700 1 --2- 172.21.15.42:0/1915813478 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbe7803ae90 0x7fbe7803d3a0 crc :-1 s=READY pgs=1338 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:46:04.088 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:04.073+0000 7fbe99a84700 1 -- 172.21.15.42:0/1915813478 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe9405d110 con 0x7fbe940a6860 2019-12-05T06:46:04.088 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:04.077+0000 7fbe8b7fe700 1 --2- 172.21.15.42:0/1915813478 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbe940a6860 0x7fbe9407af50 secure :-1 s=THROTTLE_DONE pgs=3189 cs=0 l=1 rx=0x7fbe940a55c0 tx=0x7fbe7c008bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:46:04.088 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:04.077+0000 7fbe917fa700 1 -- 172.21.15.42:0/1915813478 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fbe7c032240 con 0x7fbe940a6860 2019-12-05T06:46:04.246 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:04.241+0000 7fbe99a84700 1 -- 172.21.15.42:0/1915813478 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbe9405b8c0 con 0x7fbe940a6860 2019-12-05T06:46:04.248 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:04.241+0000 7fbe8b7fe700 1 --2- 172.21.15.42:0/1915813478 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbe940a6860 0x7fbe9407af50 secure :-1 s=THROTTLE_DONE pgs=3189 cs=0 l=1 rx=0x7fbe940a55c0 tx=0x7fbe7c008bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:46:04.248 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:04.241+0000 7fbe917fa700 1 -- 172.21.15.42:0/1915813478 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 386 v386) v1 ==== 95+0+1779 (secure 0 0 0) 0x7fbe7c007f50 con 0x7fbe940a6860 2019-12-05T06:46:04.249 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 386 2019-12-05T06:46:04.249 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:46:04.249 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":386,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":386,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:46:03.581047+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":119,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12429},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12429":{"gid":12429,"name":"b","rank":0,"incarnation":384,"state":"up:rejoin","state_seq":24,"addr":"172.21.15.40:6833/4178556019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":4178556019},{"type":"v1","addr":"172.21.15.40:6833","nonce":4178556019}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:46:04.251 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:04.245+0000 7fbe99a84700 1 -- 172.21.15.42:0/1915813478 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbe7803ae90 msgr2=0x7fbe7803d3a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:46:04.252 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:04.245+0000 7fbe99a84700 1 --2- 172.21.15.42:0/1915813478 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbe7803ae90 0x7fbe7803d3a0 crc :-1 s=READY pgs=1338 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:46:04.252 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:04.245+0000 7fbe99a84700 1 -- 172.21.15.42:0/1915813478 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbe940a6860 msgr2=0x7fbe9407af50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:46:04.253 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:04.245+0000 7fbe99a84700 1 --2- 172.21.15.42:0/1915813478 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbe940a6860 0x7fbe9407af50 secure :-1 s=READY pgs=3189 cs=0 l=1 rx=0x7fbe940a55c0 tx=0x7fbe7c008bb0).stop 2019-12-05T06:46:04.253 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:04.245+0000 7fbe99a84700 1 -- 172.21.15.42:0/1915813478 shutdown_connections 2019-12-05T06:46:04.253 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:04.245+0000 7fbe99a84700 1 --2- 172.21.15.42:0/1915813478 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbe7803ae90 0x7fbe7803d3a0 unknown :-1 s=CLOSED pgs=1338 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:46:04.254 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:04.245+0000 7fbe99a84700 1 --2- 172.21.15.42:0/1915813478 >> v2:172.21.15.40:3300/0 conn(0x7fbe940a56a0 0x7fbe9407b4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:46:04.254 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:04.245+0000 7fbe99a84700 1 --2- 172.21.15.42:0/1915813478 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbe9411bf30 0x7fbe9407ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:46:04.254 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:04.245+0000 7fbe99a84700 1 --2- 172.21.15.42:0/1915813478 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbe940a6860 0x7fbe9407af50 unknown :-1 s=CLOSED pgs=3189 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:46:04.254 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:04.245+0000 7fbe99a84700 1 -- 172.21.15.42:0/1915813478 shutdown_connections 2019-12-05T06:46:04.255 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:04.245+0000 7fbe99a84700 1 -- 172.21.15.42:0/1915813478 wait complete. 2019-12-05T06:46:04.255 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:04.245+0000 7fbe99a84700 1 -- 172.21.15.42:0/1915813478 >> 172.21.15.42:0/1915813478 conn(0x7fbe940a6d70 msgr2=0x7fbe94059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:46:04.260 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2019-12-05T06:46:06.261 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:46:06.347 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:06.337+0000 7f5c5cf7a700 1 Processor -- start 2019-12-05T06:46:06.347 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:06.337+0000 7f5c5cf7a700 1 -- start start 2019-12-05T06:46:06.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:06.337+0000 7f5c5cf7a700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5c580a7740 0x7f5c5808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:46:06.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:06.337+0000 7f5c5cf7a700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5c5809fae0 0x7f5c5807eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:46:06.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:06.337+0000 7f5c5cf7a700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c580a0710 con 0x7f5c5807e720 2019-12-05T06:46:06.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:06.337+0000 7f5c5cf7a700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c580a0b00 con 0x7f5c580a7740 2019-12-05T06:46:06.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:06.337+0000 7f5c5cf7a700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c5808e5b0 con 0x7f5c5809fae0 2019-12-05T06:46:06.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:06.341+0000 7f5c4ffff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f5c5807e720 0x7f5c5809f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:35946/0 (socket says 172.21.15.42:35946) 2019-12-05T06:46:06.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:06.341+0000 7f5c4ffff700 1 -- 172.21.15.42:0/3187282424 learned_addr learned my addr 172.21.15.42:0/3187282424 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:46:06.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:06.341+0000 7f5c5759e700 1 --2- 172.21.15.42:0/3187282424 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5c5809fae0 0x7f5c5807eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:46:06.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:06.341+0000 7f5c56d9d700 1 --2- 172.21.15.42:0/3187282424 >> v2:172.21.15.40:3300/0 conn(0x7f5c580a7740 0x7f5c5808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:46:06.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:06.341+0000 7f5c5659c700 1 -- 172.21.15.42:0/3187282424 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4114181397 0 0) 0x7f5c580a0710 con 0x7f5c5807e720 2019-12-05T06:46:06.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:06.341+0000 7f5c5659c700 1 -- 172.21.15.42:0/3187282424 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c440030b0 con 0x7f5c5807e720 2019-12-05T06:46:06.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:06.341+0000 7f5c5759e700 1 -- 172.21.15.42:0/3187282424 >> v2:172.21.15.40:3300/0 conn(0x7f5c580a7740 msgr2=0x7f5c5808df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:46:06.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:06.341+0000 7f5c5759e700 1 --2- 172.21.15.42:0/3187282424 >> v2:172.21.15.40:3300/0 conn(0x7f5c580a7740 0x7f5c5808df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:46:06.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:06.341+0000 7f5c5759e700 1 -- 172.21.15.42:0/3187282424 >> v1:172.21.15.42:6789/0 conn(0x7f5c5807e720 legacy=0x7f5c5809f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:46:06.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:06.341+0000 7f5c5759e700 1 -- 172.21.15.42:0/3187282424 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c5807f0a0 con 0x7f5c5809fae0 2019-12-05T06:46:06.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:06.341+0000 7f5c56d9d700 1 --2- 172.21.15.42:0/3187282424 >> v2:172.21.15.40:3300/0 conn(0x7f5c580a7740 0x7f5c5808df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:46:06.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:06.341+0000 7f5c5759e700 1 --2- 172.21.15.42:0/3187282424 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5c5809fae0 0x7f5c5807eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f5c50008b10 tx=0x7f5c5000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:46:06.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:06.341+0000 7f5c5759e700 1 --2- 172.21.15.42:0/3187282424 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5c5809fae0 0x7f5c5807eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f5c50008b10 tx=0x7f5c5000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:46:06.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:06.341+0000 7f5c5759e700 1 --2- 172.21.15.42:0/3187282424 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5c5809fae0 0x7f5c5807eb50 secure :-1 s=READY pgs=707 cs=0 l=1 rx=0x7f5c50008b10 tx=0x7f5c5000fff0).ready entity=mon.2 client_cookie=e5f42ceb6b953c96 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:46:06.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:06.341+0000 7f5c5759e700 1 --2- 172.21.15.42:0/3187282424 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5c5809fae0 0x7f5c5807eb50 secure :-1 s=THROTTLE_DONE pgs=707 cs=0 l=1 rx=0x7f5c50008b10 tx=0x7f5c5000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:46:06.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:06.341+0000 7f5c5659c700 1 -- 172.21.15.42:0/3187282424 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5c50010900 con 0x7f5c5809fae0 2019-12-05T06:46:06.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:06.341+0000 7f5c5759e700 1 --2- 172.21.15.42:0/3187282424 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5c5809fae0 0x7f5c5807eb50 secure :-1 s=THROTTLE_DONE pgs=707 cs=0 l=1 rx=0x7f5c50008b10 tx=0x7f5c5000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:46:06.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:06.341+0000 7f5c5659c700 1 -- 172.21.15.42:0/3187282424 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c50008b50 con 0x7f5c5809fae0 2019-12-05T06:46:06.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:06.341+0000 7f5c5759e700 1 --2- 172.21.15.42:0/3187282424 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5c5809fae0 0x7f5c5807eb50 secure :-1 s=THROTTLE_DONE pgs=707 cs=0 l=1 rx=0x7f5c50008b10 tx=0x7f5c5000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:46:06.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:06.341+0000 7f5c5659c700 1 -- 172.21.15.42:0/3187282424 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5c500110c0 con 0x7f5c5809fae0 2019-12-05T06:46:06.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:06.341+0000 7f5c5cf7a700 1 -- 172.21.15.42:0/3187282424 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5c5809fae0 msgr2=0x7f5c5807eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:46:06.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:06.341+0000 7f5c5cf7a700 1 --2- 172.21.15.42:0/3187282424 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5c5809fae0 0x7f5c5807eb50 secure :-1 s=READY pgs=707 cs=0 l=1 rx=0x7f5c50008b10 tx=0x7f5c5000fff0).stop 2019-12-05T06:46:06.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:06.341+0000 7f5c5cf7a700 1 -- 172.21.15.42:0/3187282424 shutdown_connections 2019-12-05T06:46:06.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:06.341+0000 7f5c5cf7a700 1 --2- 172.21.15.42:0/3187282424 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5c5809fae0 0x7f5c5807eb50 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:46:06.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:06.341+0000 7f5c5cf7a700 1 --2- 172.21.15.42:0/3187282424 >> v2:172.21.15.40:3300/0 conn(0x7f5c580a7740 0x7f5c5808df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:46:06.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:06.341+0000 7f5c5cf7a700 1 -- 172.21.15.42:0/3187282424 shutdown_connections 2019-12-05T06:46:06.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:06.341+0000 7f5c5cf7a700 1 -- 172.21.15.42:0/3187282424 wait complete. 2019-12-05T06:46:06.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:06.341+0000 7f5c5cf7a700 1 -- 172.21.15.42:0/3187282424 >> 172.21.15.42:0/3187282424 conn(0x7f5c580aabe0 msgr2=0x7f5c580a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:46:06.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:06.341+0000 7f5c5cf7a700 1 Processor -- start 2019-12-05T06:46:06.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:06.341+0000 7f5c5cf7a700 1 -- start start 2019-12-05T06:46:06.363 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:06.341+0000 7f5c5cf7a700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5c5807d5a0 0x7f5c581181f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:46:06.364 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:06.341+0000 7f5c5cf7a700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5c5807b9d0 0x7f5c58130cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:46:06.364 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:06.341+0000 7f5c5cf7a700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c5807be00 con 0x7f5c5811ae30 2019-12-05T06:46:06.364 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:06.341+0000 7f5c5cf7a700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c5807d9d0 con 0x7f5c5807d5a0 2019-12-05T06:46:06.364 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:06.341+0000 7f5c5cf7a700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c5807c0e0 con 0x7f5c5807b9d0 2019-12-05T06:46:06.364 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:06.341+0000 7f5c56d9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5c5807d5a0 0x7f5c581181f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:46:06.364 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:06.341+0000 7f5c56d9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5c5807d5a0 0x7f5c581181f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:35914/0 (socket says 172.21.15.42:35914) 2019-12-05T06:46:06.365 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:06.341+0000 7f5c56d9d700 1 -- 172.21.15.42:0/3390891023 learned_addr learned my addr 172.21.15.42:0/3390891023 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:46:06.365 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:06.341+0000 7f5c4ffff700 1 --2- 172.21.15.42:0/3390891023 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5c5807b9d0 0x7f5c58130cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:46:06.365 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:06.345+0000 7f5c56d9d700 1 -- 172.21.15.42:0/3390891023 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5c5807b9d0 msgr2=0x7f5c58130cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:46:06.365 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:06.345+0000 7f5c56d9d700 1 --2- 172.21.15.42:0/3390891023 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5c5807b9d0 0x7f5c58130cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:46:06.365 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:06.345+0000 7f5c56d9d700 1 -- 172.21.15.42:0/3390891023 >> v1:172.21.15.40:6789/0 conn(0x7f5c5811ae30 legacy=0x7f5c5811b260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:46:06.365 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:06.345+0000 7f5c56d9d700 1 -- 172.21.15.42:0/3390891023 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c58131240 con 0x7f5c5807d5a0 2019-12-05T06:46:06.365 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:06.345+0000 7f5c56d9d700 1 --2- 172.21.15.42:0/3390891023 >> v2:172.21.15.42:3300/0 conn(0x7f5c5807d5a0 0x7f5c581181f0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f5c5807c2a0 tx=0x7f5c48012f40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:46:06.366 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:06.345+0000 7f5c56d9d700 1 --2- 172.21.15.42:0/3390891023 >> v2:172.21.15.42:3300/0 conn(0x7f5c5807d5a0 0x7f5c581181f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f5c5807c2a0 tx=0x7f5c48012f40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:46:06.366 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:06.345+0000 7f5c56d9d700 1 --2- 172.21.15.42:0/3390891023 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5c5807d5a0 0x7f5c581181f0 secure :-1 s=READY pgs=3191 cs=0 l=1 rx=0x7f5c5807c2a0 tx=0x7f5c48012f40).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:46:06.366 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:06.345+0000 7f5c56d9d700 1 --2- 172.21.15.42:0/3390891023 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5c5807d5a0 0x7f5c581181f0 secure :-1 s=THROTTLE_DONE pgs=3191 cs=0 l=1 rx=0x7f5c5807c2a0 tx=0x7f5c48012f40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:46:06.366 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:06.345+0000 7f5c4f7fe700 1 -- 172.21.15.42:0/3390891023 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5c48014880 con 0x7f5c5807d5a0 2019-12-05T06:46:06.366 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:06.345+0000 7f5c56d9d700 1 --2- 172.21.15.42:0/3390891023 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5c5807d5a0 0x7f5c581181f0 secure :-1 s=THROTTLE_DONE pgs=3191 cs=0 l=1 rx=0x7f5c5807c2a0 tx=0x7f5c48012f40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:46:06.366 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:06.345+0000 7f5c56d9d700 1 --2- 172.21.15.42:0/3390891023 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5c5807d5a0 0x7f5c581181f0 secure :-1 s=THROTTLE_DONE pgs=3191 cs=0 l=1 rx=0x7f5c5807c2a0 tx=0x7f5c48012f40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:46:06.366 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:06.345+0000 7f5c4f7fe700 1 -- 172.21.15.42:0/3390891023 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c48007160 con 0x7f5c5807d5a0 2019-12-05T06:46:06.367 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:06.345+0000 7f5c4f7fe700 1 -- 172.21.15.42:0/3390891023 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5c48005540 con 0x7f5c5807d5a0 2019-12-05T06:46:06.367 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:06.345+0000 7f5c5cf7a700 1 -- 172.21.15.42:0/3390891023 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c5808eb70 con 0x7f5c5807d5a0 2019-12-05T06:46:06.367 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:06.345+0000 7f5c5cf7a700 1 -- 172.21.15.42:0/3390891023 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c5808f0c0 con 0x7f5c5807d5a0 2019-12-05T06:46:06.367 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:06.345+0000 7f5c56d9d700 1 --2- 172.21.15.42:0/3390891023 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5c5807d5a0 0x7f5c581181f0 secure :-1 s=THROTTLE_DONE pgs=3191 cs=0 l=1 rx=0x7f5c5807c2a0 tx=0x7f5c48012f40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:46:06.367 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:06.345+0000 7f5c4f7fe700 1 -- 172.21.15.42:0/3390891023 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f5c48010180 con 0x7f5c5807d5a0 2019-12-05T06:46:06.367 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:06.345+0000 7f5c56d9d700 1 --2- 172.21.15.42:0/3390891023 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5c5807d5a0 0x7f5c581181f0 secure :-1 s=THROTTLE_DONE pgs=3191 cs=0 l=1 rx=0x7f5c5807c2a0 tx=0x7f5c48012f40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:46:06.367 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:06.345+0000 7f5c4f7fe700 1 --2- 172.21.15.42:0/3390891023 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5c3c03adc0 0x7f5c3c03d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:46:06.368 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:06.345+0000 7f5c4f7fe700 1 -- 172.21.15.42:0/3390891023 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 12051+0+0 (secure 0 0 0) 0x7f5c4804d060 con 0x7f5c5807d5a0 2019-12-05T06:46:06.368 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:06.345+0000 7f5c4ffff700 1 --2- 172.21.15.42:0/3390891023 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5c3c03adc0 0x7f5c3c03d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:46:06.368 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:06.345+0000 7f5c4ffff700 1 --2- 172.21.15.42:0/3390891023 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5c3c03adc0 0x7f5c3c03d2d0 crc :-1 s=READY pgs=1339 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:46:06.368 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:06.345+0000 7f5c5cf7a700 1 -- 172.21.15.42:0/3390891023 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c5805d110 con 0x7f5c5807d5a0 2019-12-05T06:46:06.368 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:06.345+0000 7f5c56d9d700 1 -- 172.21.15.42:0/3390891023 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5c5807d5a0 msgr2=0x7f5c581181f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2019-12-05T06:46:06.368 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:06.345+0000 7f5c56d9d700 1 -- 172.21.15.42:0/3390891023 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5c5807d5a0 msgr2=0x7f5c581181f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2019-12-05T06:46:06.368 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:06.345+0000 7f5c56d9d700 1 --2- 172.21.15.42:0/3390891023 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5c5807d5a0 0x7f5c581181f0 secure :-1 s=READY pgs=3191 cs=0 l=1 rx=0x7f5c5807c2a0 tx=0x7f5c48012f40).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-12-05T06:46:06.369 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:06.345+0000 7f5c56d9d700 1 --2- 172.21.15.42:0/3390891023 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5c5807d5a0 0x7f5c581181f0 secure :-1 s=READY pgs=3191 cs=0 l=1 rx=0x7f5c5807c2a0 tx=0x7f5c48012f40).stop 2019-12-05T06:46:06.369 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:06.345+0000 7f5c4f7fe700 1 -- 172.21.15.42:0/3390891023 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5c5807d5a0 msgr2=0x7f5c581181f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2019-12-05T06:46:06.369 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:06.345+0000 7f5c4f7fe700 1 --2- 172.21.15.42:0/3390891023 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5c5807d5a0 0x7f5c581181f0 unknown :-1 s=CLOSED pgs=3191 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:46:06.369 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:06.345+0000 7f5c4f7fe700 1 --2- 172.21.15.42:0/3390891023 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5c3c044b00 0x7f5c3c046f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:46:06.369 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:06.345+0000 7f5c4f7fe700 1 --2- 172.21.15.42:0/3390891023 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5c3c0474e0 0x7f5c3c0499a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:46:06.369 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:06.345+0000 7f5c4f7fe700 1 --2- 172.21.15.42:0/3390891023 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5c3c04a120 0x7f5c3c04c630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:46:06.369 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:06.345+0000 7f5c4f7fe700 1 -- 172.21.15.42:0/3390891023 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c3c04cdc0 con 0x7f5c3c0474e0 2019-12-05T06:46:06.370 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:06.345+0000 7f5c4f7fe700 1 -- 172.21.15.42:0/3390891023 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c3c04cfb0 con 0x7f5c3c04a120 2019-12-05T06:46:06.370 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:06.345+0000 7f5c4f7fe700 1 -- 172.21.15.42:0/3390891023 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c3c04d1f0 con 0x7f5c3c044b00 2019-12-05T06:46:06.370 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:06.345+0000 7f5c56d9d700 1 --2- 172.21.15.42:0/3390891023 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5c3c04a120 0x7f5c3c04c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:46:06.370 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:06.345+0000 7f5c56d9d700 1 --2- 172.21.15.42:0/3390891023 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5c3c044b00 0x7f5c3c046f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:46:06.370 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:06.345+0000 7f5c5759e700 1 --2- 172.21.15.42:0/3390891023 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5c3c0474e0 0x7f5c3c0499a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:46:06.370 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:06.345+0000 7f5c56d9d700 1 -- 172.21.15.42:0/3390891023 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5c3c044b00 msgr2=0x7f5c3c046f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:46:06.370 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:06.345+0000 7f5c56d9d700 1 --2- 172.21.15.42:0/3390891023 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5c3c044b00 0x7f5c3c046f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:46:06.370 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:06.345+0000 7f5c56d9d700 1 -- 172.21.15.42:0/3390891023 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5c3c0474e0 msgr2=0x7f5c3c0499a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:46:06.371 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:06.345+0000 7f5c56d9d700 1 --2- 172.21.15.42:0/3390891023 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5c3c0474e0 0x7f5c3c0499a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:46:06.371 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:06.345+0000 7f5c56d9d700 1 -- 172.21.15.42:0/3390891023 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f5c3c04d430 con 0x7f5c3c04a120 2019-12-05T06:46:06.371 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:06.345+0000 7f5c56d9d700 1 -- 172.21.15.42:0/3390891023 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c480312e0 con 0x7f5c3c04a120 2019-12-05T06:46:06.371 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:06.345+0000 7f5c56d9d700 1 --2- 172.21.15.42:0/3390891023 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5c3c04a120 0x7f5c3c04c630 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f5c48007540 tx=0x7f5c480314e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:46:06.371 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:06.345+0000 7f5c56d9d700 1 --2- 172.21.15.42:0/3390891023 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5c3c04a120 0x7f5c3c04c630 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f5c48007540 tx=0x7f5c480314e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:46:06.371 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:06.345+0000 7f5c56d9d700 1 --2- 172.21.15.42:0/3390891023 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5c3c04a120 0x7f5c3c04c630 secure :-1 s=READY pgs=3192 cs=0 l=1 rx=0x7f5c48007540 tx=0x7f5c480314e0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:46:06.371 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:06.349+0000 7f5c56d9d700 1 --2- 172.21.15.42:0/3390891023 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5c3c04a120 0x7f5c3c04c630 secure :-1 s=THROTTLE_DONE pgs=3192 cs=0 l=1 rx=0x7f5c48007540 tx=0x7f5c480314e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:46:06.372 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:06.349+0000 7f5c56d9d700 1 --2- 172.21.15.42:0/3390891023 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5c3c04a120 0x7f5c3c04c630 secure :-1 s=THROTTLE_DONE pgs=3192 cs=0 l=1 rx=0x7f5c48007540 tx=0x7f5c480314e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:46:06.372 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:06.349+0000 7f5c4f7fe700 1 -- 172.21.15.42:0/3390891023 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5c480312e0 con 0x7f5c3c04a120 2019-12-05T06:46:06.372 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:06.349+0000 7f5c4f7fe700 1 -- 172.21.15.42:0/3390891023 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c48014ac0 con 0x7f5c3c04a120 2019-12-05T06:46:06.372 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:06.349+0000 7f5c56d9d700 1 --2- 172.21.15.42:0/3390891023 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5c3c04a120 0x7f5c3c04c630 secure :-1 s=THROTTLE_DONE pgs=3192 cs=0 l=1 rx=0x7f5c48007540 tx=0x7f5c480314e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:46:06.372 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:06.349+0000 7f5c4f7fe700 1 -- 172.21.15.42:0/3390891023 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f5c48005850 con 0x7f5c3c04a120 2019-12-05T06:46:06.372 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:06.353+0000 7f5c56d9d700 1 --2- 172.21.15.42:0/3390891023 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5c3c04a120 0x7f5c3c04c630 secure :-1 s=THROTTLE_DONE pgs=3192 cs=0 l=1 rx=0x7f5c48007540 tx=0x7f5c480314e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:46:06.372 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:06.353+0000 7f5c4f7fe700 1 -- 172.21.15.42:0/3390891023 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f5c48031620 con 0x7f5c3c04a120 2019-12-05T06:46:06.525 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:06.521+0000 7f5c5cf7a700 1 -- 172.21.15.42:0/3390891023 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5c5805b8c0 con 0x7f5c3c04a120 2019-12-05T06:46:06.526 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:06.521+0000 7f5c56d9d700 1 --2- 172.21.15.42:0/3390891023 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5c3c04a120 0x7f5c3c04c630 secure :-1 s=THROTTLE_DONE pgs=3192 cs=0 l=1 rx=0x7f5c48007540 tx=0x7f5c480314e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:46:06.526 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:06.521+0000 7f5c4f7fe700 1 -- 172.21.15.42:0/3390891023 <== mon.1 v2:172.21.15.42:3300/0 5 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 388 v388) v1 ==== 95+0+1779 (secure 0 0 0) 0x7f5c48005b60 con 0x7f5c3c04a120 2019-12-05T06:46:06.526 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 388 2019-12-05T06:46:06.526 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:46:06.526 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":388,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":388,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:46:05.602252+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":119,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12429},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12429":{"gid":12429,"name":"b","rank":0,"incarnation":384,"state":"up:active","state_seq":26,"addr":"172.21.15.40:6833/4178556019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":4178556019},{"type":"v1","addr":"172.21.15.40:6833","nonce":4178556019}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:46:06.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:06.521+0000 7f5c5cf7a700 1 -- 172.21.15.42:0/3390891023 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5c3c03adc0 msgr2=0x7f5c3c03d2d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:46:06.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:06.521+0000 7f5c5cf7a700 1 --2- 172.21.15.42:0/3390891023 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5c3c03adc0 0x7f5c3c03d2d0 crc :-1 s=READY pgs=1339 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:46:06.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:06.525+0000 7f5c5cf7a700 1 -- 172.21.15.42:0/3390891023 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5c3c04a120 msgr2=0x7f5c3c04c630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:46:06.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:06.525+0000 7f5c5cf7a700 1 --2- 172.21.15.42:0/3390891023 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5c3c04a120 0x7f5c3c04c630 secure :-1 s=READY pgs=3192 cs=0 l=1 rx=0x7f5c48007540 tx=0x7f5c480314e0).stop 2019-12-05T06:46:06.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:06.525+0000 7f5c5759e700 1 -- 172.21.15.42:0/3390891023 reap_dead start 2019-12-05T06:46:06.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:06.525+0000 7f5c5759e700 1 -- 172.21.15.42:0/3390891023 reap_dead start 2019-12-05T06:46:06.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:06.525+0000 7f5c5cf7a700 1 -- 172.21.15.42:0/3390891023 shutdown_connections 2019-12-05T06:46:06.530 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:06.525+0000 7f5c5cf7a700 1 --2- 172.21.15.42:0/3390891023 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5c5807d5a0 0x7f5c581181f0 unknown :-1 s=CLOSED pgs=3191 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:46:06.530 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:06.525+0000 7f5c5cf7a700 1 -- 172.21.15.42:0/3390891023 shutdown_connections 2019-12-05T06:46:06.530 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:06.525+0000 7f5c5cf7a700 1 -- 172.21.15.42:0/3390891023 wait complete. 2019-12-05T06:46:06.530 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:06.525+0000 7f5c5cf7a700 1 -- 172.21.15.42:0/3390891023 >> 172.21.15.42:0/3390891023 conn(0x7f5c5807ec60 msgr2=0x7f5c58059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:46:06.538 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b has gained rank=0, replacing gid=12303 2019-12-05T06:46:06.538 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 2 secs before reviving mds.a 2019-12-05T06:46:08.540 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.a 2019-12-05T06:46:08.540 INFO:tasks.ceph.mds.a:Restarting daemon 2019-12-05T06:46:08.541 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2019-12-05T06:46:08.557 INFO:tasks.ceph.mds.a:Started 2019-12-05T06:46:08.557 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:46:08.613 INFO:tasks.ceph.mds.a.smithi042.stdout:starting mds.a at 2019-12-05T06:46:08.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:08.641+0000 7f35f02d7700 1 Processor -- start 2019-12-05T06:46:08.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:08.641+0000 7f35f02d7700 1 -- start start 2019-12-05T06:46:08.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:08.641+0000 7f35f02d7700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f35e80a7740 0x7f35e808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:46:08.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:08.641+0000 7f35f02d7700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f35e809fae0 0x7f35e807eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:46:08.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:08.641+0000 7f35f02d7700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35e80a0710 con 0x7f35e807e720 2019-12-05T06:46:08.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:08.641+0000 7f35f02d7700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35e80a0b00 con 0x7f35e809fae0 2019-12-05T06:46:08.650 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:08.641+0000 7f35f02d7700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35e808e5b0 con 0x7f35e80a7740 2019-12-05T06:46:08.650 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:08.641+0000 7f35ef075700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f35e809fae0 0x7f35e807eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:46:08.650 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:08.641+0000 7f35ee874700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f35e80a7740 0x7f35e808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:46:08.650 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:08.641+0000 7f35ef075700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f35e809fae0 0x7f35e807eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.42:52010/0 (socket says 172.21.15.42:52010) 2019-12-05T06:46:08.650 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:08.641+0000 7f35ef075700 1 -- 172.21.15.42:0/2390288223 learned_addr learned my addr 172.21.15.42:0/2390288223 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:46:08.650 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:08.641+0000 7f35ee874700 1 -- 172.21.15.42:0/2390288223 >> v2:172.21.15.40:3300/0 conn(0x7f35e809fae0 msgr2=0x7f35e807eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:46:08.650 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:08.641+0000 7f35ee874700 1 --2- 172.21.15.42:0/2390288223 >> v2:172.21.15.40:3300/0 conn(0x7f35e809fae0 0x7f35e807eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:46:08.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:08.641+0000 7f35ee874700 1 -- 172.21.15.42:0/2390288223 >> v1:172.21.15.42:6789/0 conn(0x7f35e807e720 legacy=0x7f35e809f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:46:08.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:08.641+0000 7f35ee874700 1 -- 172.21.15.42:0/2390288223 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35e807f0a0 con 0x7f35e80a7740 2019-12-05T06:46:08.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:08.641+0000 7f35ee874700 1 --2- 172.21.15.42:0/2390288223 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f35e80a7740 0x7f35e808df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f35e0008b10 tx=0x7f35e000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:46:08.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:08.641+0000 7f35ee874700 1 --2- 172.21.15.42:0/2390288223 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f35e80a7740 0x7f35e808df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f35e0008b10 tx=0x7f35e000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:46:08.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:08.641+0000 7f35ee874700 1 --2- 172.21.15.42:0/2390288223 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f35e80a7740 0x7f35e808df80 secure :-1 s=READY pgs=708 cs=0 l=1 rx=0x7f35e0008b10 tx=0x7f35e000fff0).ready entity=mon.2 client_cookie=48991618d2903da5 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:46:08.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:08.641+0000 7f35ee874700 1 --2- 172.21.15.42:0/2390288223 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f35e80a7740 0x7f35e808df80 secure :-1 s=THROTTLE_DONE pgs=708 cs=0 l=1 rx=0x7f35e0008b10 tx=0x7f35e000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:46:08.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:08.641+0000 7f35ee874700 1 --2- 172.21.15.42:0/2390288223 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f35e80a7740 0x7f35e808df80 secure :-1 s=THROTTLE_DONE pgs=708 cs=0 l=1 rx=0x7f35e0008b10 tx=0x7f35e000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:46:08.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:08.641+0000 7f35ed872700 1 -- 172.21.15.42:0/2390288223 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f35e00109f0 con 0x7f35e80a7740 2019-12-05T06:46:08.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:08.641+0000 7f35ee874700 1 --2- 172.21.15.42:0/2390288223 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f35e80a7740 0x7f35e808df80 secure :-1 s=THROTTLE_DONE pgs=708 cs=0 l=1 rx=0x7f35e0008b10 tx=0x7f35e000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:46:08.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:08.641+0000 7f35ed872700 1 -- 172.21.15.42:0/2390288223 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35e0008b50 con 0x7f35e80a7740 2019-12-05T06:46:08.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:08.641+0000 7f35ed872700 1 -- 172.21.15.42:0/2390288223 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f35e00110c0 con 0x7f35e80a7740 2019-12-05T06:46:08.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:08.641+0000 7f35f02d7700 1 -- 172.21.15.42:0/2390288223 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f35e80a7740 msgr2=0x7f35e808df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:46:08.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:08.641+0000 7f35f02d7700 1 --2- 172.21.15.42:0/2390288223 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f35e80a7740 0x7f35e808df80 secure :-1 s=READY pgs=708 cs=0 l=1 rx=0x7f35e0008b10 tx=0x7f35e000fff0).stop 2019-12-05T06:46:08.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:08.641+0000 7f35f02d7700 1 -- 172.21.15.42:0/2390288223 shutdown_connections 2019-12-05T06:46:08.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:08.641+0000 7f35f02d7700 1 --2- 172.21.15.42:0/2390288223 >> v2:172.21.15.40:3300/0 conn(0x7f35e809fae0 0x7f35e807eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:46:08.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:08.641+0000 7f35f02d7700 1 --2- 172.21.15.42:0/2390288223 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f35e80a7740 0x7f35e808df80 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:46:08.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:08.641+0000 7f35f02d7700 1 -- 172.21.15.42:0/2390288223 shutdown_connections 2019-12-05T06:46:08.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:08.641+0000 7f35f02d7700 1 -- 172.21.15.42:0/2390288223 wait complete. 2019-12-05T06:46:08.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:08.641+0000 7f35f02d7700 1 -- 172.21.15.42:0/2390288223 >> 172.21.15.42:0/2390288223 conn(0x7f35e80aabe0 msgr2=0x7f35e80a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:46:08.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:08.645+0000 7f35f02d7700 1 Processor -- start 2019-12-05T06:46:08.654 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:08.645+0000 7f35f02d7700 1 -- start start 2019-12-05T06:46:08.657 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:08.645+0000 7f35f02d7700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f35e811a930 0x7f35e81191b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:46:08.657 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:08.645+0000 7f35f02d7700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f35e8118c20 0x7f35e8119700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:46:08.657 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:08.645+0000 7f35f02d7700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35e811a090 con 0x7f35e808eb70 2019-12-05T06:46:08.657 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:08.645+0000 7f35f02d7700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35e811bce0 con 0x7f35e8118c20 2019-12-05T06:46:08.657 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:08.645+0000 7f35f02d7700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35e811bf80 con 0x7f35e811a930 2019-12-05T06:46:08.657 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:08.645+0000 7f35ee073700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f35e8118c20 0x7f35e8119700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:46:08.658 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:08.645+0000 7f35ee073700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f35e8118c20 0x7f35e8119700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.42:52018/0 (socket says 172.21.15.42:52018) 2019-12-05T06:46:08.658 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:08.645+0000 7f35ef075700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f35e808eb70 0x7f35e808efa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:45450/0 (socket says 172.21.15.42:45450) 2019-12-05T06:46:08.658 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:08.645+0000 7f35ee073700 1 -- 172.21.15.42:0/2792660594 learned_addr learned my addr 172.21.15.42:0/2792660594 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:46:08.658 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:08.645+0000 7f35ee874700 1 --2- 172.21.15.42:0/2792660594 >> v2:172.21.15.42:3300/0 conn(0x7f35e811a930 0x7f35e81191b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:46:08.658 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:08.645+0000 7f35ee073700 1 -- 172.21.15.42:0/2792660594 >> v2:172.21.15.42:3300/0 conn(0x7f35e811a930 msgr2=0x7f35e81191b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:46:08.658 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:08.645+0000 7f35ee073700 1 --2- 172.21.15.42:0/2792660594 >> v2:172.21.15.42:3300/0 conn(0x7f35e811a930 0x7f35e81191b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:46:08.658 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:08.645+0000 7f35ee073700 1 -- 172.21.15.42:0/2792660594 >> v1:172.21.15.40:6789/0 conn(0x7f35e808eb70 legacy=0x7f35e808efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:46:08.658 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:08.645+0000 7f35ee073700 1 -- 172.21.15.42:0/2792660594 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35e808f7a0 con 0x7f35e8118c20 2019-12-05T06:46:08.659 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:08.645+0000 7f35df7fe700 1 -- 172.21.15.42:0/2792660594 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 481505844 0 0) 0x7f35e811a090 con 0x7f35e808eb70 2019-12-05T06:46:08.659 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:08.645+0000 7f35ee073700 1 --2- 172.21.15.42:0/2792660594 >> v2:172.21.15.40:3300/0 conn(0x7f35e8118c20 0x7f35e8119700 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f35e8120c10 tx=0x7f35e40109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:46:08.659 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:08.645+0000 7f35ee073700 1 --2- 172.21.15.42:0/2792660594 >> v2:172.21.15.40:3300/0 conn(0x7f35e8118c20 0x7f35e8119700 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f35e8120c10 tx=0x7f35e40109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:46:08.659 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:08.645+0000 7f35ee073700 1 --2- 172.21.15.42:0/2792660594 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f35e8118c20 0x7f35e8119700 secure :-1 s=READY pgs=2430 cs=0 l=1 rx=0x7f35e8120c10 tx=0x7f35e40109a0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:46:08.659 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:08.645+0000 7f35ee073700 1 --2- 172.21.15.42:0/2792660594 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f35e8118c20 0x7f35e8119700 secure :-1 s=THROTTLE_DONE pgs=2430 cs=0 l=1 rx=0x7f35e8120c10 tx=0x7f35e40109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:46:08.659 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:08.645+0000 7f35df7fe700 1 -- 172.21.15.42:0/2792660594 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f35e400caa0 con 0x7f35e8118c20 2019-12-05T06:46:08.659 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:08.645+0000 7f35ee073700 1 --2- 172.21.15.42:0/2792660594 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f35e8118c20 0x7f35e8119700 secure :-1 s=THROTTLE_DONE pgs=2430 cs=0 l=1 rx=0x7f35e8120c10 tx=0x7f35e40109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:46:08.660 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:08.645+0000 7f35ee073700 1 --2- 172.21.15.42:0/2792660594 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f35e8118c20 0x7f35e8119700 secure :-1 s=THROTTLE_DONE pgs=2430 cs=0 l=1 rx=0x7f35e8120c10 tx=0x7f35e40109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:46:08.660 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:08.645+0000 7f35df7fe700 1 -- 172.21.15.42:0/2792660594 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35e400c6c0 con 0x7f35e8118c20 2019-12-05T06:46:08.660 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:08.645+0000 7f35df7fe700 1 -- 172.21.15.42:0/2792660594 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f35e4011160 con 0x7f35e8118c20 2019-12-05T06:46:08.660 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:08.645+0000 7f35f02d7700 1 -- 172.21.15.42:0/2792660594 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35e81166d0 con 0x7f35e8118c20 2019-12-05T06:46:08.660 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:08.645+0000 7f35f02d7700 1 -- 172.21.15.42:0/2792660594 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35e8074630 con 0x7f35e8118c20 2019-12-05T06:46:08.660 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:08.645+0000 7f35ee073700 1 --2- 172.21.15.42:0/2792660594 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f35e8118c20 0x7f35e8119700 secure :-1 s=THROTTLE_DONE pgs=2430 cs=0 l=1 rx=0x7f35e8120c10 tx=0x7f35e40109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:46:08.660 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:08.649+0000 7f35df7fe700 1 -- 172.21.15.42:0/2792660594 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f35e4011440 con 0x7f35e8118c20 2019-12-05T06:46:08.661 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:08.649+0000 7f35ee073700 1 --2- 172.21.15.42:0/2792660594 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f35e8118c20 0x7f35e8119700 secure :-1 s=THROTTLE_DONE pgs=2430 cs=0 l=1 rx=0x7f35e8120c10 tx=0x7f35e40109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:46:08.661 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:08.649+0000 7f35df7fe700 1 --2- 172.21.15.42:0/2792660594 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f35d003ac50 0x7f35d003d160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:46:08.661 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:08.649+0000 7f35df7fe700 1 -- 172.21.15.42:0/2792660594 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 12051+0+0 (secure 0 0 0) 0x7f35e404b280 con 0x7f35e8118c20 2019-12-05T06:46:08.661 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:08.649+0000 7f35ee874700 1 --2- 172.21.15.42:0/2792660594 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f35d003ac50 0x7f35d003d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:46:08.662 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:08.649+0000 7f35ee874700 1 --2- 172.21.15.42:0/2792660594 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f35d003ac50 0x7f35d003d160 crc :-1 s=READY pgs=1340 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:46:08.662 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:08.657+0000 7f35f02d7700 1 -- 172.21.15.42:0/2792660594 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35e805d110 con 0x7f35e8118c20 2019-12-05T06:46:08.668 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:08.661+0000 7f35ee073700 1 --2- 172.21.15.42:0/2792660594 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f35e8118c20 0x7f35e8119700 secure :-1 s=THROTTLE_DONE pgs=2430 cs=0 l=1 rx=0x7f35e8120c10 tx=0x7f35e40109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:46:08.668 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:08.661+0000 7f35df7fe700 1 -- 172.21.15.42:0/2792660594 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f35e40322e0 con 0x7f35e8118c20 2019-12-05T06:46:08.833 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:08.825+0000 7f35f02d7700 1 -- 172.21.15.42:0/2792660594 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f35e809fd00 con 0x7f35e8118c20 2019-12-05T06:46:08.834 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:08.829+0000 7f35ee073700 1 --2- 172.21.15.42:0/2792660594 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f35e8118c20 0x7f35e8119700 secure :-1 s=THROTTLE_DONE pgs=2430 cs=0 l=1 rx=0x7f35e8120c10 tx=0x7f35e40109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:46:08.834 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:08.829+0000 7f35df7fe700 1 -- 172.21.15.42:0/2792660594 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 388 v388) v1 ==== 95+0+1779 (secure 0 0 0) 0x7f35e4032540 con 0x7f35e8118c20 2019-12-05T06:46:08.835 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 388 2019-12-05T06:46:08.835 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:46:08.835 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":388,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":388,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:46:05.602252+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":119,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12429},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12429":{"gid":12429,"name":"b","rank":0,"incarnation":384,"state":"up:active","state_seq":26,"addr":"172.21.15.40:6833/4178556019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":4178556019},{"type":"v1","addr":"172.21.15.40:6833","nonce":4178556019}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:46:08.839 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:08.833+0000 7f35f02d7700 1 -- 172.21.15.42:0/2792660594 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f35d003ac50 msgr2=0x7f35d003d160 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:46:08.839 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:08.833+0000 7f35f02d7700 1 --2- 172.21.15.42:0/2792660594 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f35d003ac50 0x7f35d003d160 crc :-1 s=READY pgs=1340 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:46:08.839 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:08.833+0000 7f35f02d7700 1 -- 172.21.15.42:0/2792660594 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f35e8118c20 msgr2=0x7f35e8119700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:46:08.839 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:08.833+0000 7f35f02d7700 1 --2- 172.21.15.42:0/2792660594 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f35e8118c20 0x7f35e8119700 secure :-1 s=READY pgs=2430 cs=0 l=1 rx=0x7f35e8120c10 tx=0x7f35e40109a0).stop 2019-12-05T06:46:08.839 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:08.833+0000 7f35f02d7700 1 -- 172.21.15.42:0/2792660594 shutdown_connections 2019-12-05T06:46:08.839 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:08.833+0000 7f35f02d7700 1 --2- 172.21.15.42:0/2792660594 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f35d003ac50 0x7f35d003d160 unknown :-1 s=CLOSED pgs=1340 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:46:08.840 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:08.833+0000 7f35f02d7700 1 --2- 172.21.15.42:0/2792660594 >> v2:172.21.15.42:3300/0 conn(0x7f35e811a930 0x7f35e81191b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:46:08.840 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:08.833+0000 7f35f02d7700 1 --2- 172.21.15.42:0/2792660594 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f35e8118c20 0x7f35e8119700 unknown :-1 s=CLOSED pgs=2430 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:46:08.840 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:08.833+0000 7f35f02d7700 1 -- 172.21.15.42:0/2792660594 shutdown_connections 2019-12-05T06:46:08.840 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:08.833+0000 7f35f02d7700 1 -- 172.21.15.42:0/2792660594 wait complete. 2019-12-05T06:46:08.840 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:08.833+0000 7f35f02d7700 1 -- 172.21.15.42:0/2792660594 >> 172.21.15.42:0/2792660594 conn(0x7f35e811d380 msgr2=0x7f35e8059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:46:08.848 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2019-12-05T06:46:10.547 INFO:tasks.workunit.client.0.smithi042.stderr:Checking out files: 1% (3/193) Checking out files: 2% (4/193) Checking out files: 3% (6/193) Checking out files: 4% (8/193) Checking out files: 5% (10/193) Checking out files: 6% (12/193) Checking out files: 7% (14/193) Checking out files: 8% (16/193) Checking out files: 9% (18/193) Checking out files: 10% (20/193) Checking out files: 11% (22/193) Checking out files: 12% (24/193) Checking out files: 13% (26/193) Checking out files: 14% (28/193) Checking out files: 15% (29/193) Checking out files: 16% (31/193) Checking out files: 17% (33/193) Checking out files: 18% (35/193) Checking out files: 19% (37/193) Checking out files: 20% (39/193) Checking out files: 21% (41/193) Checking out files: 22% (43/193) Checking out files: 23% (45/193) Checking out files: 24% (47/193) Checking out files: 25% (49/193) Checking out files: 26% (51/193) Checking out files: 27% (53/193) Checking out files: 28% (55/193) Checking out files: 29% (56/193) Checking out files: 30% (58/193) Checking out files: 31% (60/193) Checking out files: 32% (62/193) Checking out files: 33% (64/193) Checking out files: 34% (66/193) Checking out files: 35% (68/193) Checking out files: 36% (70/193) Checking out files: 37% (72/193) Checking out files: 38% (74/193) Checking out files: 39% (76/193) Checking out files: 40% (78/193) Checking out files: 41% (80/193) Checking out files: 42% (82/193) Checking out files: 43% (83/193) Checking out files: 44% (85/193) Checking out files: 45% (87/193) Checking out files: 46% (89/193) Checking out files: 47% (91/193) Checking out files: 48% (93/193) Checking out files: 49% (95/193) Checking out files: 50% (97/193) Checking out files: 51% (99/193) Checking out files: 52% (101/193) Checking out files: 53% (103/193) Checking out files: 54% (105/193) Checking out files: 55% (107/193) Checking out files: 56% (109/193) Checking out files: 57% (111/193) Checking out files: 58% (112/193) Checking out files: 59% (114/193) Checking out files: 60% (116/193) Checking out files: 61% (118/193) Checking out files: 62% (120/193) Checking out files: 63% (122/193) Checking out files: 64% (124/193) Checking out files: 65% (126/193) Checking out files: 66% (128/193) Checking out files: 67% (130/193) Checking out files: 68% (132/193) Checking out files: 69% (134/193) Checking out files: 70% (136/193) Checking out files: 71% (138/193) Checking out files: 72% (139/193) Checking out files: 73% (141/193) Checking out files: 74% (143/193) Checking out files: 75% (145/193) Checking out files: 76% (147/193) Checking out files: 77% (149/193) Checking out files: 78% (151/193) Checking out files: 79% (153/193) Checking out files: 80% (155/193) Checking out files: 81% (157/193) Checking out files: 82% (159/193) Checking out files: 83% (161/193) Checking out files: 84% (163/193) Checking out files: 85% (165/193) Checking out files: 86% (166/193) Checking out files: 87% (168/193) Checking out files: 88% (170/193) Checking out files: 88% (171/193) Checking out files: 89% (172/193) Checking out files: 90% (174/193) Checking out files: 91% (176/193) Checking out files: 92% (178/193) Checking out files: 93% (180/193) Checking out files: 94% (182/193) Checking out files: 95% (184/193) Checking out files: 96% (186/193) Checking out files: 97% (188/193) Checking out files: 98% (190/193) Checking out files: 99% (192/193) Checking out files: 100% (193/193) Checking out files: 100% (193/193), done. 2019-12-05T06:46:10.741 INFO:tasks.workunit.client.0.smithi042.stdout:HEAD is now at 4148185392 kclient: strip out kernel version compatibility cruft 2019-12-05T06:46:10.747 INFO:tasks.workunit.client.0.smithi042.stderr:+ mkdir .snap/v0.9 2019-12-05T06:46:10.772 INFO:tasks.workunit.client.0.smithi042.stderr:+ ver=v0.10 2019-12-05T06:46:10.773 INFO:tasks.workunit.client.0.smithi042.stderr:+ echo v0.10 2019-12-05T06:46:10.773 INFO:tasks.workunit.client.0.smithi042.stderr:+ git reset --hard v0.10 2019-12-05T06:46:10.773 INFO:tasks.workunit.client.0.smithi042.stdout:v0.10 2019-12-05T06:46:10.849 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:46:10.957 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:10.945+0000 7fdb0f144700 1 Processor -- start 2019-12-05T06:46:10.957 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:10.945+0000 7fdb0f144700 1 -- start start 2019-12-05T06:46:10.957 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:10.949+0000 7fdb0f144700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fdb0808df80 0x7fdb0809f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:46:10.957 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:10.949+0000 7fdb0f144700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fdb0809f830 0x7fdb0814efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:46:10.957 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:10.949+0000 7fdb0f144700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb0809fc60 con 0x7fdb080a7740 2019-12-05T06:46:10.958 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:10.949+0000 7fdb0f144700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb080a0b00 con 0x7fdb0808df80 2019-12-05T06:46:10.958 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:10.949+0000 7fdb0f144700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb0808e5b0 con 0x7fdb0809f830 2019-12-05T06:46:10.958 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:10.949+0000 7fdb0dee2700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fdb0809f830 0x7fdb0814efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:46:10.958 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:10.949+0000 7fdb0dee2700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fdb0809f830 0x7fdb0814efe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:36016/0 (socket says 172.21.15.42:36016) 2019-12-05T06:46:10.958 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:10.949+0000 7fdb0dee2700 1 -- 172.21.15.42:0/397083604 learned_addr learned my addr 172.21.15.42:0/397083604 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:46:10.958 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:10.949+0000 7fdb0cee0700 1 --2- 172.21.15.42:0/397083604 >> v2:172.21.15.40:3300/0 conn(0x7fdb0808df80 0x7fdb0809f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:46:10.958 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:10.949+0000 7fdb0dee2700 1 -- 172.21.15.42:0/397083604 >> v2:172.21.15.40:3300/0 conn(0x7fdb0808df80 msgr2=0x7fdb0809f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:46:10.959 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:10.949+0000 7fdb0dee2700 1 --2- 172.21.15.42:0/397083604 >> v2:172.21.15.40:3300/0 conn(0x7fdb0808df80 0x7fdb0809f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:46:10.959 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:10.949+0000 7fdb0dee2700 1 -- 172.21.15.42:0/397083604 >> v1:172.21.15.40:6789/0 conn(0x7fdb080a7740 legacy=0x7fdb0807e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:46:10.959 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:10.949+0000 7fdb0dee2700 1 -- 172.21.15.42:0/397083604 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb080a0710 con 0x7fdb0809f830 2019-12-05T06:46:10.960 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:10.949+0000 7fdb0dee2700 1 --2- 172.21.15.42:0/397083604 >> v2:172.21.15.42:3300/0 conn(0x7fdb0809f830 0x7fdb0814efe0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fdb00002f30 tx=0x7fdb0000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:46:10.966 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:10.949+0000 7fdb0dee2700 1 --2- 172.21.15.42:0/397083604 >> v2:172.21.15.42:3300/0 conn(0x7fdb0809f830 0x7fdb0814efe0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fdb00002f30 tx=0x7fdb0000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:46:10.973 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:10.949+0000 7fdb0dee2700 1 --2- 172.21.15.42:0/397083604 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdb0809f830 0x7fdb0814efe0 secure :-1 s=READY pgs=3200 cs=0 l=1 rx=0x7fdb00002f30 tx=0x7fdb0000fd30).ready entity=mon.1 client_cookie=f59eee5bd3edc8ad server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:46:10.973 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:10.949+0000 7fdb0dee2700 1 --2- 172.21.15.42:0/397083604 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdb0809f830 0x7fdb0814efe0 secure :-1 s=THROTTLE_DONE pgs=3200 cs=0 l=1 rx=0x7fdb00002f30 tx=0x7fdb0000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:46:10.973 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:10.949+0000 7fdb0dee2700 1 --2- 172.21.15.42:0/397083604 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdb0809f830 0x7fdb0814efe0 secure :-1 s=THROTTLE_DONE pgs=3200 cs=0 l=1 rx=0x7fdb00002f30 tx=0x7fdb0000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:46:10.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:10.949+0000 7fdafffff700 1 -- 172.21.15.42:0/397083604 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdb00008c90 con 0x7fdb0809f830 2019-12-05T06:46:10.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:10.949+0000 7fdb0dee2700 1 --2- 172.21.15.42:0/397083604 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdb0809f830 0x7fdb0814efe0 secure :-1 s=THROTTLE_DONE pgs=3200 cs=0 l=1 rx=0x7fdb00002f30 tx=0x7fdb0000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:46:10.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:10.949+0000 7fdafffff700 1 -- 172.21.15.42:0/397083604 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb000088b0 con 0x7fdb0809f830 2019-12-05T06:46:10.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:10.949+0000 7fdafffff700 1 -- 172.21.15.42:0/397083604 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdb00011ad0 con 0x7fdb0809f830 2019-12-05T06:46:10.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:10.949+0000 7fdb0f144700 1 -- 172.21.15.42:0/397083604 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdb0809f830 msgr2=0x7fdb0814efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:46:10.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:10.949+0000 7fdb0f144700 1 --2- 172.21.15.42:0/397083604 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdb0809f830 0x7fdb0814efe0 secure :-1 s=READY pgs=3200 cs=0 l=1 rx=0x7fdb00002f30 tx=0x7fdb0000fd30).stop 2019-12-05T06:46:10.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:10.949+0000 7fdb0f144700 1 -- 172.21.15.42:0/397083604 shutdown_connections 2019-12-05T06:46:10.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:10.949+0000 7fdb0f144700 1 --2- 172.21.15.42:0/397083604 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdb0809f830 0x7fdb0814efe0 unknown :-1 s=CLOSED pgs=3200 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:46:10.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:10.949+0000 7fdb0f144700 1 --2- 172.21.15.42:0/397083604 >> v2:172.21.15.40:3300/0 conn(0x7fdb0808df80 0x7fdb0809f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:46:10.978 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:10.949+0000 7fdb0f144700 1 -- 172.21.15.42:0/397083604 shutdown_connections 2019-12-05T06:46:10.979 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:10.949+0000 7fdb0f144700 1 -- 172.21.15.42:0/397083604 wait complete. 2019-12-05T06:46:10.979 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:10.949+0000 7fdb0f144700 1 -- 172.21.15.42:0/397083604 >> 172.21.15.42:0/397083604 conn(0x7fdb080aabe0 msgr2=0x7fdb080a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:46:10.991 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:10.953+0000 7fdb0f144700 1 Processor -- start 2019-12-05T06:46:10.991 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:10.953+0000 7fdb0f144700 1 -- start start 2019-12-05T06:46:10.992 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:10.953+0000 7fdb0f144700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdb0808eeb0 0x7fdb08063cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:46:10.992 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:10.953+0000 7fdb0f144700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fdb080658f0 0x7fdb0807c020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:46:10.992 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:10.953+0000 7fdb0f144700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fdb0807c570 0x7fdb081265b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:46:10.995 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:10.953+0000 7fdb0f144700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb08064630 con 0x7fdb080658f0 2019-12-05T06:46:10.996 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:10.953+0000 7fdb0f144700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb08065110 con 0x7fdb0807c570 2019-12-05T06:46:10.996 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:10.953+0000 7fdb0f144700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb080653b0 con 0x7fdb0808eeb0 2019-12-05T06:46:10.996 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:10.953+0000 7fdb0dee2700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fdb0807c570 0x7fdb081265b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:46:10.996 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:10.953+0000 7fdb0dee2700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fdb0807c570 0x7fdb081265b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:36022/0 (socket says 172.21.15.42:36022) 2019-12-05T06:46:10.996 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:10.953+0000 7fdb0dee2700 1 -- 172.21.15.42:0/4209734689 learned_addr learned my addr 172.21.15.42:0/4209734689 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:46:10.996 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:10.953+0000 7fdb0cee0700 1 --2- 172.21.15.42:0/4209734689 >> v2:172.21.15.40:3300/0 conn(0x7fdb080658f0 0x7fdb0807c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:46:10.997 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:10.953+0000 7fdb0d6e1700 1 --2- 172.21.15.42:0/4209734689 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdb0808eeb0 0x7fdb08063cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:46:10.997 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:10.953+0000 7fdb0dee2700 1 -- 172.21.15.42:0/4209734689 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdb0808eeb0 msgr2=0x7fdb08063cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:46:10.997 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:10.953+0000 7fdb0dee2700 1 --2- 172.21.15.42:0/4209734689 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdb0808eeb0 0x7fdb08063cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:46:10.997 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:10.953+0000 7fdb0dee2700 1 -- 172.21.15.42:0/4209734689 >> v2:172.21.15.40:3300/0 conn(0x7fdb080658f0 msgr2=0x7fdb0807c020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:46:10.997 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:10.953+0000 7fdb0dee2700 1 --2- 172.21.15.42:0/4209734689 >> v2:172.21.15.40:3300/0 conn(0x7fdb080658f0 0x7fdb0807c020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:46:10.997 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:10.953+0000 7fdb0dee2700 1 -- 172.21.15.42:0/4209734689 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb08065d20 con 0x7fdb0807c570 2019-12-05T06:46:10.997 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:10.953+0000 7fdb0d6e1700 1 --2- 172.21.15.42:0/4209734689 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdb0808eeb0 0x7fdb08063cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:46:10.998 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:10.953+0000 7fdb0dee2700 1 --2- 172.21.15.42:0/4209734689 >> v2:172.21.15.42:3300/0 conn(0x7fdb0807c570 0x7fdb081265b0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fdb080652d0 tx=0x7fdb00008be0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:46:10.998 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:10.957+0000 7fdb0dee2700 1 --2- 172.21.15.42:0/4209734689 >> v2:172.21.15.42:3300/0 conn(0x7fdb0807c570 0x7fdb081265b0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fdb080652d0 tx=0x7fdb00008be0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:46:10.998 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:10.957+0000 7fdb0dee2700 1 --2- 172.21.15.42:0/4209734689 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdb0807c570 0x7fdb081265b0 secure :-1 s=READY pgs=3201 cs=0 l=1 rx=0x7fdb080652d0 tx=0x7fdb00008be0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:46:10.998 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:10.957+0000 7fdb0dee2700 1 --2- 172.21.15.42:0/4209734689 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdb0807c570 0x7fdb081265b0 secure :-1 s=THROTTLE_DONE pgs=3201 cs=0 l=1 rx=0x7fdb080652d0 tx=0x7fdb00008be0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:46:10.998 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:10.957+0000 7fdafdffb700 1 -- 172.21.15.42:0/4209734689 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdb0000f680 con 0x7fdb0807c570 2019-12-05T06:46:10.998 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:10.957+0000 7fdb0dee2700 1 --2- 172.21.15.42:0/4209734689 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdb0807c570 0x7fdb081265b0 secure :-1 s=THROTTLE_DONE pgs=3201 cs=0 l=1 rx=0x7fdb080652d0 tx=0x7fdb00008be0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:46:10.998 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:10.957+0000 7fdb0dee2700 1 --2- 172.21.15.42:0/4209734689 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdb0807c570 0x7fdb081265b0 secure :-1 s=THROTTLE_DONE pgs=3201 cs=0 l=1 rx=0x7fdb080652d0 tx=0x7fdb00008be0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:46:10.999 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:10.957+0000 7fdafdffb700 1 -- 172.21.15.42:0/4209734689 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb000116f0 con 0x7fdb0807c570 2019-12-05T06:46:10.999 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:10.957+0000 7fdb0f144700 1 -- 172.21.15.42:0/4209734689 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb08063490 con 0x7fdb0807c570 2019-12-05T06:46:10.999 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:10.957+0000 7fdafdffb700 1 -- 172.21.15.42:0/4209734689 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdb00005920 con 0x7fdb0807c570 2019-12-05T06:46:10.999 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:10.957+0000 7fdb0f144700 1 -- 172.21.15.42:0/4209734689 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb08126b00 con 0x7fdb0807c570 2019-12-05T06:46:10.999 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:10.957+0000 7fdb0dee2700 1 --2- 172.21.15.42:0/4209734689 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdb0807c570 0x7fdb081265b0 secure :-1 s=THROTTLE_DONE pgs=3201 cs=0 l=1 rx=0x7fdb080652d0 tx=0x7fdb00008be0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:46:10.999 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:10.957+0000 7fdafdffb700 1 -- 172.21.15.42:0/4209734689 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fdb00005c30 con 0x7fdb0807c570 2019-12-05T06:46:10.999 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:10.957+0000 7fdb0dee2700 1 --2- 172.21.15.42:0/4209734689 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdb0807c570 0x7fdb081265b0 secure :-1 s=THROTTLE_DONE pgs=3201 cs=0 l=1 rx=0x7fdb080652d0 tx=0x7fdb00008be0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:46:11.000 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:10.957+0000 7fdafdffb700 1 --2- 172.21.15.42:0/4209734689 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fdae803ada0 0x7fdae803d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:46:11.000 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:10.957+0000 7fdafdffb700 1 -- 172.21.15.42:0/4209734689 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 12051+0+0 (secure 0 0 0) 0x7fdb0004b9e0 con 0x7fdb0807c570 2019-12-05T06:46:11.000 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:10.957+0000 7fdb0d6e1700 1 --2- 172.21.15.42:0/4209734689 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fdae803ada0 0x7fdae803d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:46:11.000 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:10.957+0000 7fdb0d6e1700 1 --2- 172.21.15.42:0/4209734689 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fdae803ada0 0x7fdae803d2b0 crc :-1 s=READY pgs=1341 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:46:11.000 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:10.961+0000 7fdb0f144700 1 -- 172.21.15.42:0/4209734689 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb0805d110 con 0x7fdb0807c570 2019-12-05T06:46:11.000 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:10.973+0000 7fdb0dee2700 1 --2- 172.21.15.42:0/4209734689 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdb0807c570 0x7fdb081265b0 secure :-1 s=THROTTLE_DONE pgs=3201 cs=0 l=1 rx=0x7fdb080652d0 tx=0x7fdb00008be0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:46:11.000 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:10.973+0000 7fdafdffb700 1 -- 172.21.15.42:0/4209734689 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fdb00007660 con 0x7fdb0807c570 2019-12-05T06:46:11.143 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:11.137+0000 7fdb0f144700 1 -- 172.21.15.42:0/4209734689 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdb0805b8c0 con 0x7fdb0807c570 2019-12-05T06:46:11.144 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:11.141+0000 7fdb0dee2700 1 --2- 172.21.15.42:0/4209734689 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdb0807c570 0x7fdb081265b0 secure :-1 s=THROTTLE_DONE pgs=3201 cs=0 l=1 rx=0x7fdb080652d0 tx=0x7fdb00008be0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:46:11.144 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:11.141+0000 7fdafdffb700 1 -- 172.21.15.42:0/4209734689 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 388 v388) v1 ==== 95+0+1779 (secure 0 0 0) 0x7fdb000306a0 con 0x7fdb0807c570 2019-12-05T06:46:11.145 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 388 2019-12-05T06:46:11.145 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:46:11.145 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":388,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":388,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:46:05.602252+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":119,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12429},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12429":{"gid":12429,"name":"b","rank":0,"incarnation":384,"state":"up:active","state_seq":26,"addr":"172.21.15.40:6833/4178556019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":4178556019},{"type":"v1","addr":"172.21.15.40:6833","nonce":4178556019}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:46:11.148 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:11.141+0000 7fdb0f144700 1 -- 172.21.15.42:0/4209734689 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fdae803ada0 msgr2=0x7fdae803d2b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:46:11.148 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:11.141+0000 7fdb0f144700 1 --2- 172.21.15.42:0/4209734689 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fdae803ada0 0x7fdae803d2b0 crc :-1 s=READY pgs=1341 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:46:11.148 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:11.141+0000 7fdb0f144700 1 -- 172.21.15.42:0/4209734689 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdb0807c570 msgr2=0x7fdb081265b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:46:11.149 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:11.141+0000 7fdb0f144700 1 --2- 172.21.15.42:0/4209734689 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdb0807c570 0x7fdb081265b0 secure :-1 s=READY pgs=3201 cs=0 l=1 rx=0x7fdb080652d0 tx=0x7fdb00008be0).stop 2019-12-05T06:46:11.149 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:11.141+0000 7fdb0f144700 1 -- 172.21.15.42:0/4209734689 shutdown_connections 2019-12-05T06:46:11.149 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:11.141+0000 7fdb0f144700 1 --2- 172.21.15.42:0/4209734689 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fdae803ada0 0x7fdae803d2b0 unknown :-1 s=CLOSED pgs=1341 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:46:11.149 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:11.141+0000 7fdb0f144700 1 --2- 172.21.15.42:0/4209734689 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdb0807c570 0x7fdb081265b0 unknown :-1 s=CLOSED pgs=3201 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:46:11.149 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:11.141+0000 7fdb0f144700 1 --2- 172.21.15.42:0/4209734689 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdb0808eeb0 0x7fdb08063cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:46:11.150 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:11.141+0000 7fdb0f144700 1 --2- 172.21.15.42:0/4209734689 >> v2:172.21.15.40:3300/0 conn(0x7fdb080658f0 0x7fdb0807c020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:46:11.150 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:11.141+0000 7fdb0f144700 1 -- 172.21.15.42:0/4209734689 shutdown_connections 2019-12-05T06:46:11.150 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:11.141+0000 7fdb0f144700 1 -- 172.21.15.42:0/4209734689 wait complete. 2019-12-05T06:46:11.151 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:11.141+0000 7fdb0f144700 1 -- 172.21.15.42:0/4209734689 >> 172.21.15.42:0/4209734689 conn(0x7fdb080a7f90 msgr2=0x7fdb08113430 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:46:11.156 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2019-12-05T06:46:13.157 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:46:13.260 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:13.249+0000 7fc2c45b9700 1 Processor -- start 2019-12-05T06:46:13.260 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:13.249+0000 7fc2c45b9700 1 -- start start 2019-12-05T06:46:13.260 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:13.249+0000 7fc2c45b9700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc2bc08df80 0x7fc2bc09f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:46:13.260 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:13.249+0000 7fc2c45b9700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2bc09fc60 con 0x7fc2bc09f830 2019-12-05T06:46:13.261 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:13.249+0000 7fc2c45b9700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2bc0a0710 con 0x7fc2bc0a7740 2019-12-05T06:46:13.261 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:13.249+0000 7fc2c45b9700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2bc0a0b00 con 0x7fc2bc08df80 2019-12-05T06:46:13.261 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:13.249+0000 7fc2c2b56700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fc2bc0a7740 0x7fc2bc07e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:36076/0 (socket says 172.21.15.42:36076) 2019-12-05T06:46:13.261 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:13.249+0000 7fc2c2b56700 1 -- 172.21.15.42:0/3865991475 learned_addr learned my addr 172.21.15.42:0/3865991475 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:46:13.261 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:13.253+0000 7fc2c2355700 1 --2- 172.21.15.42:0/3865991475 >> v2:172.21.15.40:3300/0 conn(0x7fc2bc08df80 0x7fc2bc09f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:46:13.261 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:13.253+0000 7fc2c1b54700 1 -- 172.21.15.42:0/3865991475 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2927686408 0 0) 0x7fc2bc0a0710 con 0x7fc2bc0a7740 2019-12-05T06:46:13.261 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:13.253+0000 7fc2c1b54700 1 -- 172.21.15.42:0/3865991475 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2a80030b0 con 0x7fc2bc0a7740 2019-12-05T06:46:13.262 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:13.253+0000 7fc2c1b54700 1 -- 172.21.15.42:0/3865991475 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 852914877 0 0) 0x7fc2bc09fc60 con 0x7fc2bc09f830 2019-12-05T06:46:13.262 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:13.253+0000 7fc2c1b54700 1 -- 172.21.15.42:0/3865991475 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2bc0a0710 con 0x7fc2bc09f830 2019-12-05T06:46:13.262 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:13.253+0000 7fc2c2355700 1 -- 172.21.15.42:0/3865991475 >> v1:172.21.15.42:6789/0 conn(0x7fc2bc0a7740 legacy=0x7fc2bc07e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:46:13.262 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:13.253+0000 7fc2c2355700 1 -- 172.21.15.42:0/3865991475 >> v1:172.21.15.40:6789/0 conn(0x7fc2bc09f830 legacy=0x7fc2bc14efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:46:13.262 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:13.253+0000 7fc2c2355700 1 -- 172.21.15.42:0/3865991475 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2bc07ef20 con 0x7fc2bc08df80 2019-12-05T06:46:13.262 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:13.253+0000 7fc2c2355700 1 --2- 172.21.15.42:0/3865991475 >> v2:172.21.15.40:3300/0 conn(0x7fc2bc08df80 0x7fc2bc09f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fc2ac008790 tx=0x7fc2ac00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:46:13.262 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:13.253+0000 7fc2c2355700 1 --2- 172.21.15.42:0/3865991475 >> v2:172.21.15.40:3300/0 conn(0x7fc2bc08df80 0x7fc2bc09f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fc2ac008790 tx=0x7fc2ac00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:46:13.263 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:13.253+0000 7fc2c2355700 1 --2- 172.21.15.42:0/3865991475 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc2bc08df80 0x7fc2bc09f2e0 secure :-1 s=READY pgs=2433 cs=0 l=1 rx=0x7fc2ac008790 tx=0x7fc2ac00fbf0).ready entity=mon.0 client_cookie=fc4ac75eb1a71363 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:46:13.263 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:13.253+0000 7fc2c2355700 1 --2- 172.21.15.42:0/3865991475 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc2bc08df80 0x7fc2bc09f2e0 secure :-1 s=THROTTLE_DONE pgs=2433 cs=0 l=1 rx=0x7fc2ac008790 tx=0x7fc2ac00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:46:13.263 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:13.253+0000 7fc2c2355700 1 --2- 172.21.15.42:0/3865991475 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc2bc08df80 0x7fc2bc09f2e0 secure :-1 s=THROTTLE_DONE pgs=2433 cs=0 l=1 rx=0x7fc2ac008790 tx=0x7fc2ac00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:46:13.263 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:13.253+0000 7fc2c1b54700 1 -- 172.21.15.42:0/3865991475 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc2ac011270 con 0x7fc2bc08df80 2019-12-05T06:46:13.263 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:13.253+0000 7fc2c2355700 1 --2- 172.21.15.42:0/3865991475 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc2bc08df80 0x7fc2bc09f2e0 secure :-1 s=THROTTLE_DONE pgs=2433 cs=0 l=1 rx=0x7fc2ac008790 tx=0x7fc2ac00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:46:13.263 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:13.253+0000 7fc2c1b54700 1 -- 172.21.15.42:0/3865991475 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2ac008bc0 con 0x7fc2bc08df80 2019-12-05T06:46:13.263 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:13.253+0000 7fc2c1b54700 1 -- 172.21.15.42:0/3865991475 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc2ac011850 con 0x7fc2bc08df80 2019-12-05T06:46:13.264 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:13.253+0000 7fc2c45b9700 1 -- 172.21.15.42:0/3865991475 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc2bc08df80 msgr2=0x7fc2bc09f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:46:13.264 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:13.253+0000 7fc2c45b9700 1 --2- 172.21.15.42:0/3865991475 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc2bc08df80 0x7fc2bc09f2e0 secure :-1 s=READY pgs=2433 cs=0 l=1 rx=0x7fc2ac008790 tx=0x7fc2ac00fbf0).stop 2019-12-05T06:46:13.264 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:13.253+0000 7fc2c45b9700 1 -- 172.21.15.42:0/3865991475 shutdown_connections 2019-12-05T06:46:13.264 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:13.253+0000 7fc2c45b9700 1 --2- 172.21.15.42:0/3865991475 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc2bc08df80 0x7fc2bc09f2e0 unknown :-1 s=CLOSED pgs=2433 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:46:13.264 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:13.253+0000 7fc2c45b9700 1 -- 172.21.15.42:0/3865991475 shutdown_connections 2019-12-05T06:46:13.264 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:13.253+0000 7fc2c45b9700 1 -- 172.21.15.42:0/3865991475 wait complete. 2019-12-05T06:46:13.265 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:13.253+0000 7fc2c45b9700 1 -- 172.21.15.42:0/3865991475 >> 172.21.15.42:0/3865991475 conn(0x7fc2bc0aabe0 msgr2=0x7fc2bc0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:46:13.269 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:13.257+0000 7fc2c45b9700 1 Processor -- start 2019-12-05T06:46:13.269 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:13.257+0000 7fc2c45b9700 1 -- start start 2019-12-05T06:46:13.269 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:13.257+0000 7fc2c45b9700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc2bc144170 0x7fc2bc12d950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:46:13.270 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:13.257+0000 7fc2c45b9700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2bc1445a0 con 0x7fc2bc0793b0 2019-12-05T06:46:13.270 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:13.257+0000 7fc2c45b9700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2bc0797e0 con 0x7fc2bc143d40 2019-12-05T06:46:13.270 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:13.257+0000 7fc2c45b9700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2bc122160 con 0x7fc2bc144170 2019-12-05T06:46:13.270 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:13.257+0000 7fc2c2355700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fc2bc143d40 0x7fc2bc144ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:36082/0 (socket says 172.21.15.42:36082) 2019-12-05T06:46:13.271 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:13.257+0000 7fc2c2355700 1 -- 172.21.15.42:0/1726170506 learned_addr learned my addr 172.21.15.42:0/1726170506 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:46:13.271 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:13.257+0000 7fc2c3357700 1 --2- 172.21.15.42:0/1726170506 >> v2:172.21.15.40:3300/0 conn(0x7fc2bc144170 0x7fc2bc12d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:46:13.271 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:13.257+0000 7fc2b37fe700 1 -- 172.21.15.42:0/1726170506 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1875972573 0 0) 0x7fc2bc0797e0 con 0x7fc2bc143d40 2019-12-05T06:46:13.271 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:13.257+0000 7fc2b37fe700 1 -- 172.21.15.42:0/1726170506 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2a40030b0 con 0x7fc2bc143d40 2019-12-05T06:46:13.271 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:13.257+0000 7fc2b37fe700 1 -- 172.21.15.42:0/1726170506 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 1519268163 0 0) 0x7fc2a40030b0 con 0x7fc2bc143d40 2019-12-05T06:46:13.272 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:13.257+0000 7fc2b37fe700 1 -- 172.21.15.42:0/1726170506 --> v1:172.21.15.42:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc2bc0797e0 con 0x7fc2bc143d40 2019-12-05T06:46:13.272 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:13.257+0000 7fc2b37fe700 1 -- 172.21.15.42:0/1726170506 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7fc2ac011270 con 0x7fc2bc143d40 2019-12-05T06:46:13.273 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:13.257+0000 7fc2b37fe700 1 -- 172.21.15.42:0/1726170506 >> v2:172.21.15.40:3300/0 conn(0x7fc2bc144170 msgr2=0x7fc2bc12d950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:46:13.273 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:13.257+0000 7fc2b37fe700 1 --2- 172.21.15.42:0/1726170506 >> v2:172.21.15.40:3300/0 conn(0x7fc2bc144170 0x7fc2bc12d950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:46:13.273 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:13.257+0000 7fc2b37fe700 1 -- 172.21.15.42:0/1726170506 >> v1:172.21.15.42:6789/0 conn(0x7fc2bc143d40 legacy=0x7fc2bc144ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:46:13.273 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:13.257+0000 7fc2b37fe700 1 -- 172.21.15.42:0/1726170506 >> v1:172.21.15.40:6789/0 conn(0x7fc2bc0793b0 legacy=0x7fc2bc1226a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:46:13.274 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:13.257+0000 7fc2b37fe700 1 --2- 172.21.15.42:0/1726170506 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc2a40038a0 0x7fc2a4003cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:46:13.274 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:13.257+0000 7fc2b37fe700 1 --2- 172.21.15.42:0/1726170506 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc2a4004420 0x7fc2a4004850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:46:13.274 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:13.257+0000 7fc2b37fe700 1 --2- 172.21.15.42:0/1726170506 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc2a4004e40 0x7fc2a400b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:46:13.274 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:13.257+0000 7fc2b37fe700 1 -- 172.21.15.42:0/1726170506 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2bc122160 con 0x7fc2a40038a0 2019-12-05T06:46:13.274 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:13.257+0000 7fc2b37fe700 1 -- 172.21.15.42:0/1726170506 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2a400bc20 con 0x7fc2a4004420 2019-12-05T06:46:13.275 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:13.257+0000 7fc2b37fe700 1 -- 172.21.15.42:0/1726170506 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2a400be60 con 0x7fc2a4004e40 2019-12-05T06:46:13.275 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:13.257+0000 7fc2c3357700 1 --2- 172.21.15.42:0/1726170506 >> v2:172.21.15.40:3300/0 conn(0x7fc2bc144170 0x7fc2bc12d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:46:13.275 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:13.257+0000 7fc2c2355700 1 --2- 172.21.15.42:0/1726170506 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc2a4004420 0x7fc2a4004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:46:13.275 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:13.257+0000 7fc2c2b56700 1 --2- 172.21.15.42:0/1726170506 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc2a40038a0 0x7fc2a4003cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:46:13.276 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:13.261+0000 7fc2c3357700 1 --2- 172.21.15.42:0/1726170506 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc2a4004e40 0x7fc2a400b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:46:13.276 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:13.261+0000 7fc2c2355700 1 -- 172.21.15.42:0/1726170506 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc2a4004e40 msgr2=0x7fc2a400b670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:46:13.276 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:13.261+0000 7fc2c2355700 1 --2- 172.21.15.42:0/1726170506 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc2a4004e40 0x7fc2a400b670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:46:13.276 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:13.261+0000 7fc2c2355700 1 -- 172.21.15.42:0/1726170506 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc2a40038a0 msgr2=0x7fc2a4003cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:46:13.277 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:13.261+0000 7fc2c2355700 1 --2- 172.21.15.42:0/1726170506 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc2a40038a0 0x7fc2a4003cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:46:13.277 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:13.261+0000 7fc2c2355700 1 -- 172.21.15.42:0/1726170506 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2bc143560 con 0x7fc2a4004420 2019-12-05T06:46:13.277 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:13.261+0000 7fc2c3357700 1 -- 172.21.15.42:0/1726170506 reap_dead start 2019-12-05T06:46:13.277 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:13.261+0000 7fc2c2355700 1 --2- 172.21.15.42:0/1726170506 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc2a4004420 0x7fc2a4004850 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fc2ac007e50 tx=0x7fc2ac0119f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:46:13.278 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:13.261+0000 7fc2c45b9700 1 -- 172.21.15.42:0/1726170506 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2bc143d40 con 0x7fc2a4004420 2019-12-05T06:46:13.278 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:13.261+0000 7fc2c45b9700 1 -- 172.21.15.42:0/1726170506 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2bc0793b0 con 0x7fc2a4004420 2019-12-05T06:46:13.278 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:13.261+0000 7fc2c2355700 1 --2- 172.21.15.42:0/1726170506 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc2a4004420 0x7fc2a4004850 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fc2ac007e50 tx=0x7fc2ac0119f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:46:13.278 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:13.261+0000 7fc2c2355700 1 --2- 172.21.15.42:0/1726170506 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc2a4004420 0x7fc2a4004850 secure :-1 s=READY pgs=3204 cs=0 l=1 rx=0x7fc2ac007e50 tx=0x7fc2ac0119f0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:46:13.279 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:13.261+0000 7fc2c2355700 1 --2- 172.21.15.42:0/1726170506 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc2a4004420 0x7fc2a4004850 secure :-1 s=THROTTLE_DONE pgs=3204 cs=0 l=1 rx=0x7fc2ac007e50 tx=0x7fc2ac0119f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:46:13.279 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:13.261+0000 7fc2b37fe700 1 -- 172.21.15.42:0/1726170506 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc2ac007340 con 0x7fc2a4004420 2019-12-05T06:46:13.279 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:13.261+0000 7fc2c2355700 1 --2- 172.21.15.42:0/1726170506 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc2a4004420 0x7fc2a4004850 secure :-1 s=THROTTLE_DONE pgs=3204 cs=0 l=1 rx=0x7fc2ac007e50 tx=0x7fc2ac0119f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:46:13.280 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:13.261+0000 7fc2b37fe700 1 -- 172.21.15.42:0/1726170506 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2ac00f590 con 0x7fc2a4004420 2019-12-05T06:46:13.280 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:13.261+0000 7fc2c2355700 1 --2- 172.21.15.42:0/1726170506 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc2a4004420 0x7fc2a4004850 secure :-1 s=THROTTLE_DONE pgs=3204 cs=0 l=1 rx=0x7fc2ac007e50 tx=0x7fc2ac0119f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:46:13.280 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:13.261+0000 7fc2b37fe700 1 -- 172.21.15.42:0/1726170506 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc2ac005920 con 0x7fc2a4004420 2019-12-05T06:46:13.280 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:13.261+0000 7fc2c2355700 1 --2- 172.21.15.42:0/1726170506 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc2a4004420 0x7fc2a4004850 secure :-1 s=THROTTLE_DONE pgs=3204 cs=0 l=1 rx=0x7fc2ac007e50 tx=0x7fc2ac0119f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:46:13.281 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:13.261+0000 7fc2b37fe700 1 -- 172.21.15.42:0/1726170506 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fc2ac005b20 con 0x7fc2a4004420 2019-12-05T06:46:13.281 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:13.261+0000 7fc2c2355700 1 --2- 172.21.15.42:0/1726170506 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc2a4004420 0x7fc2a4004850 secure :-1 s=THROTTLE_DONE pgs=3204 cs=0 l=1 rx=0x7fc2ac007e50 tx=0x7fc2ac0119f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:46:13.281 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:13.261+0000 7fc2b37fe700 1 --2- 172.21.15.42:0/1726170506 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc2a403e900 0x7fc2a4040e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:46:13.282 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:13.261+0000 7fc2b37fe700 1 -- 172.21.15.42:0/1726170506 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 12051+0+0 (secure 0 0 0) 0x7fc2ac04bba0 con 0x7fc2a4004420 2019-12-05T06:46:13.282 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:13.261+0000 7fc2c2b56700 1 --2- 172.21.15.42:0/1726170506 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc2a403e900 0x7fc2a4040e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:46:13.282 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:13.261+0000 7fc2c2b56700 1 --2- 172.21.15.42:0/1726170506 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc2a403e900 0x7fc2a4040e10 crc :-1 s=READY pgs=1342 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:46:13.283 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:13.265+0000 7fc2c45b9700 1 -- 172.21.15.42:0/1726170506 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc298003200 con 0x7fc2a4004420 2019-12-05T06:46:13.284 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:13.269+0000 7fc2c2355700 1 --2- 172.21.15.42:0/1726170506 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc2a4004420 0x7fc2a4004850 secure :-1 s=THROTTLE_DONE pgs=3204 cs=0 l=1 rx=0x7fc2ac007e50 tx=0x7fc2ac0119f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:46:13.284 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:13.269+0000 7fc2b37fe700 1 -- 172.21.15.42:0/1726170506 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fc2ac032c00 con 0x7fc2a4004420 2019-12-05T06:46:13.438 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:13.433+0000 7fc2c45b9700 1 -- 172.21.15.42:0/1726170506 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc298003630 con 0x7fc2a4004420 2019-12-05T06:46:13.439 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:13.433+0000 7fc2c2355700 1 --2- 172.21.15.42:0/1726170506 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc2a4004420 0x7fc2a4004850 secure :-1 s=THROTTLE_DONE pgs=3204 cs=0 l=1 rx=0x7fc2ac007e50 tx=0x7fc2ac0119f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:46:13.440 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:13.433+0000 7fc2b37fe700 1 -- 172.21.15.42:0/1726170506 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 389 v389) v1 ==== 95+0+2117 (secure 0 0 0) 0x7fc2ac033720 con 0x7fc2a4004420 2019-12-05T06:46:13.440 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 389 2019-12-05T06:46:13.440 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:46:13.440 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":389,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":12847,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/1449317789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1449317789},{"type":"v1","addr":"172.21.15.42:6835","nonce":1449317789}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":389}],"filesystems":[{"mdsmap":{"epoch":388,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:46:05.602252+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":119,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12429},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12429":{"gid":12429,"name":"b","rank":0,"incarnation":384,"state":"up:active","state_seq":26,"addr":"172.21.15.40:6833/4178556019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":4178556019},{"type":"v1","addr":"172.21.15.40:6833","nonce":4178556019}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:46:13.442 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:13.437+0000 7fc2c45b9700 1 -- 172.21.15.42:0/1726170506 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc2a403e900 msgr2=0x7fc2a4040e10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:46:13.442 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:13.437+0000 7fc2c45b9700 1 --2- 172.21.15.42:0/1726170506 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc2a403e900 0x7fc2a4040e10 crc :-1 s=READY pgs=1342 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:46:13.442 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:13.437+0000 7fc2c45b9700 1 -- 172.21.15.42:0/1726170506 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc2a4004420 msgr2=0x7fc2a4004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:46:13.442 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:13.437+0000 7fc2c45b9700 1 --2- 172.21.15.42:0/1726170506 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc2a4004420 0x7fc2a4004850 secure :-1 s=READY pgs=3204 cs=0 l=1 rx=0x7fc2ac007e50 tx=0x7fc2ac0119f0).stop 2019-12-05T06:46:13.442 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:13.437+0000 7fc2c45b9700 1 -- 172.21.15.42:0/1726170506 shutdown_connections 2019-12-05T06:46:13.443 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:13.437+0000 7fc2c45b9700 1 --2- 172.21.15.42:0/1726170506 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc2a403e900 0x7fc2a4040e10 unknown :-1 s=CLOSED pgs=1342 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:46:13.443 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:13.437+0000 7fc2c45b9700 1 --2- 172.21.15.42:0/1726170506 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc2a4004420 0x7fc2a4004850 unknown :-1 s=CLOSED pgs=3204 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:46:13.443 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:13.437+0000 7fc2c45b9700 1 -- 172.21.15.42:0/1726170506 shutdown_connections 2019-12-05T06:46:13.443 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:13.437+0000 7fc2c45b9700 1 -- 172.21.15.42:0/1726170506 wait complete. 2019-12-05T06:46:13.443 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:46:13.437+0000 7fc2c45b9700 1 -- 172.21.15.42:0/1726170506 >> 172.21.15.42:0/1726170506 conn(0x7fc2bc0a6e40 msgr2=0x7fc2bc07e570 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:46:13.452 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a reported in up:standby state 2019-12-05T06:46:13.452 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 114 secs before thrashing 2019-12-05T06:46:18.455 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:46:18.459 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:46:19.694 INFO:tasks.workunit.client.0.smithi042.stderr:Checking out files: 59% (98/164) Checking out files: 60% (99/164) Checking out files: 61% (101/164) Checking out files: 62% (102/164) Checking out files: 63% (104/164) Checking out files: 64% (105/164) Checking out files: 65% (107/164) Checking out files: 66% (109/164) Checking out files: 67% (110/164) Checking out files: 68% (112/164) Checking out files: 69% (114/164) Checking out files: 70% (115/164) Checking out files: 71% (117/164) Checking out files: 72% (119/164) Checking out files: 73% (120/164) Checking out files: 74% (122/164) Checking out files: 75% (123/164) Checking out files: 76% (125/164) Checking out files: 77% (127/164) Checking out files: 78% (128/164) Checking out files: 79% (130/164) Checking out files: 79% (131/164) Checking out files: 80% (132/164) Checking out files: 81% (133/164) Checking out files: 82% (135/164) Checking out files: 83% (137/164) Checking out files: 84% (138/164) Checking out files: 85% (140/164) Checking out files: 86% (142/164) Checking out files: 87% (143/164) Checking out files: 87% (144/164) Checking out files: 88% (145/164) Checking out files: 89% (146/164) Checking out files: 90% (148/164) Checking out files: 91% (150/164) Checking out files: 92% (151/164) Checking out files: 93% (153/164) Checking out files: 94% (155/164) Checking out files: 95% (156/164) Checking out files: 96% (158/164) Checking out files: 97% (160/164) Checking out files: 98% (161/164) Checking out files: 99% (163/164) Checking out files: 100% (164/164) Checking out files: 100% (164/164), done. 2019-12-05T06:46:20.203 INFO:tasks.workunit.client.0.smithi042.stdout:HEAD is now at 007313fe84 v0.10 2019-12-05T06:46:20.208 INFO:tasks.workunit.client.0.smithi042.stderr:+ mkdir .snap/v0.10 2019-12-05T06:46:20.410 INFO:tasks.workunit.client.0.smithi042.stderr:+ ver=v0.11 2019-12-05T06:46:20.411 INFO:tasks.workunit.client.0.smithi042.stderr:+ echo v0.11 2019-12-05T06:46:20.411 INFO:tasks.workunit.client.0.smithi042.stderr:+ git reset --hard v0.11 2019-12-05T06:46:20.411 INFO:tasks.workunit.client.0.smithi042.stdout:v0.11 2019-12-05T06:46:23.019 INFO:tasks.workunit.client.0.smithi042.stderr:Checking out files: 71% (49/69) Checking out files: 72% (50/69) Checking out files: 73% (51/69) Checking out files: 75% (52/69) Checking out files: 76% (53/69) Checking out files: 78% (54/69) Checking out files: 79% (55/69) Checking out files: 81% (56/69) Checking out files: 82% (57/69) Checking out files: 84% (58/69) Checking out files: 85% (59/69) Checking out files: 86% (60/69) Checking out files: 88% (61/69) Checking out files: 89% (62/69) Checking out files: 91% (63/69) Checking out files: 92% (64/69) Checking out files: 94% (65/69) Checking out files: 95% (66/69) Checking out files: 97% (67/69) Checking out files: 98% (68/69) Checking out files: 100% (69/69) Checking out files: 100% (69/69), done. 2019-12-05T06:46:24.225 INFO:tasks.workunit.client.0.smithi042.stdout:HEAD is now at 9fa6f5e7d0 v0.11 2019-12-05T06:46:24.229 INFO:tasks.workunit.client.0.smithi042.stderr:+ mkdir .snap/v0.11 2019-12-05T06:46:24.294 INFO:tasks.workunit.client.0.smithi042.stdout:v0.12 2019-12-05T06:46:24.294 INFO:tasks.workunit.client.0.smithi042.stderr:+ ver=v0.12 2019-12-05T06:46:24.294 INFO:tasks.workunit.client.0.smithi042.stderr:+ echo v0.12 2019-12-05T06:46:24.295 INFO:tasks.workunit.client.0.smithi042.stderr:+ git reset --hard v0.12 2019-12-05T06:46:26.396 INFO:tasks.workunit.client.0.smithi042.stdout:HEAD is now at f524150822 v0.12 2019-12-05T06:46:26.397 INFO:tasks.workunit.client.0.smithi042.stderr:+ mkdir .snap/v0.12 2019-12-05T06:46:26.565 INFO:tasks.workunit.client.0.smithi042.stderr:+ ver=v0.13 2019-12-05T06:46:26.565 INFO:tasks.workunit.client.0.smithi042.stderr:+ echo v0.13 2019-12-05T06:46:26.565 INFO:tasks.workunit.client.0.smithi042.stderr:+ git reset --hard v0.13 2019-12-05T06:46:26.565 INFO:tasks.workunit.client.0.smithi042.stdout:v0.13 2019-12-05T06:46:31.526 INFO:tasks.workunit.client.0.smithi042.stderr:Checking out files: 12% (17/133) Checking out files: 13% (18/133) Checking out files: 14% (19/133) Checking out files: 15% (20/133) Checking out files: 16% (22/133) Checking out files: 17% (23/133) Checking out files: 18% (24/133) Checking out files: 19% (26/133) Checking out files: 20% (27/133) Checking out files: 21% (28/133) Checking out files: 22% (30/133) Checking out files: 23% (31/133) Checking out files: 24% (32/133) Checking out files: 25% (34/133) Checking out files: 26% (35/133) Checking out files: 27% (36/133) Checking out files: 28% (38/133) Checking out files: 29% (39/133) Checking out files: 30% (40/133) Checking out files: 31% (42/133) Checking out files: 32% (43/133) Checking out files: 33% (44/133) Checking out files: 34% (46/133) Checking out files: 35% (47/133) Checking out files: 36% (48/133) Checking out files: 37% (50/133) Checking out files: 38% (51/133) Checking out files: 39% (52/133) Checking out files: 40% (54/133) Checking out files: 41% (55/133) Checking out files: 42% (56/133) Checking out files: 43% (58/133) Checking out files: 44% (59/133) Checking out files: 45% (60/133) Checking out files: 46% (62/133) Checking out files: 47% (63/133) Checking out files: 48% (64/133) Checking out files: 49% (66/133) Checking out files: 50% (67/133) Checking out files: 51% (68/133) Checking out files: 52% (70/133) Checking out files: 53% (71/133) Checking out files: 54% (72/133) Checking out files: 55% (74/133) Checking out files: 56% (75/133) Checking out files: 57% (76/133) Checking out files: 58% (78/133) Checking out files: 59% (79/133) Checking out files: 60% (80/133) Checking out files: 61% (82/133) Checking out files: 62% (83/133) Checking out files: 63% (84/133) Checking out files: 64% (86/133) Checking out files: 65% (87/133) Checking out files: 66% (88/133) Checking out files: 67% (90/133) Checking out files: 68% (91/133) Checking out files: 69% (92/133) Checking out files: 70% (94/133) Checking out files: 71% (95/133) Checking out files: 72% (96/133) Checking out files: 73% (98/133) Checking out files: 74% (99/133) Checking out files: 75% (100/133) Checking out files: 76% (102/133) Checking out files: 77% (103/133) Checking out files: 78% (104/133) Checking out files: 79% (106/133) Checking out files: 80% (107/133) Checking out files: 81% (108/133) Checking out files: 82% (110/133) Checking out files: 83% (111/133) Checking out files: 84% (112/133) Checking out files: 85% (114/133) Checking out files: 86% (115/133) Checking out files: 87% (116/133) Checking out files: 88% (118/133) Checking out files: 89% (119/133) Checking out files: 90% (120/133) Checking out files: 91% (122/133) Checking out files: 92% (123/133) Checking out files: 93% (124/133) Checking out files: 94% (126/133) Checking out files: 95% (127/133) Checking out files: 96% (128/133) Checking out files: 97% (130/133) Checking out files: 98% (131/133) Checking out files: 99% (132/133) Checking out files: 100% (133/133) Checking out files: 100% (133/133), done. 2019-12-05T06:46:31.646 INFO:tasks.workunit.client.0.smithi042.stdout:HEAD is now at 2b32297267 v0.13 2019-12-05T06:46:31.651 INFO:tasks.workunit.client.0.smithi042.stderr:+ mkdir .snap/v0.13 2019-12-05T06:46:31.877 INFO:tasks.workunit.client.0.smithi042.stderr:+ ver=v0.14 2019-12-05T06:46:31.878 INFO:tasks.workunit.client.0.smithi042.stderr:+ echo v0.14 2019-12-05T06:46:31.878 INFO:tasks.workunit.client.0.smithi042.stderr:+ git reset --hard v0.14 2019-12-05T06:46:31.878 INFO:tasks.workunit.client.0.smithi042.stdout:v0.14 2019-12-05T06:46:35.177 INFO:tasks.workunit.client.0.smithi042.stderr:Checking out files: 52% (79/151) Checking out files: 53% (81/151) Checking out files: 54% (82/151) Checking out files: 55% (84/151) Checking out files: 56% (85/151) Checking out files: 57% (87/151) Checking out files: 58% (88/151) Checking out files: 59% (90/151) Checking out files: 60% (91/151) Checking out files: 61% (93/151) Checking out files: 62% (94/151) Checking out files: 63% (96/151) Checking out files: 64% (97/151) Checking out files: 65% (99/151) Checking out files: 66% (100/151) Checking out files: 67% (102/151) Checking out files: 68% (103/151) Checking out files: 69% (105/151) Checking out files: 70% (106/151) Checking out files: 71% (108/151) Checking out files: 72% (109/151) Checking out files: 73% (111/151) Checking out files: 74% (112/151) Checking out files: 75% (114/151) Checking out files: 76% (115/151) Checking out files: 77% (117/151) Checking out files: 78% (118/151) Checking out files: 79% (120/151) Checking out files: 80% (121/151) Checking out files: 81% (123/151) Checking out files: 82% (124/151) Checking out files: 83% (126/151) Checking out files: 84% (127/151) Checking out files: 85% (129/151) Checking out files: 86% (130/151) Checking out files: 87% (132/151) Checking out files: 88% (133/151) Checking out files: 89% (135/151) Checking out files: 90% (136/151) Checking out files: 91% (138/151) Checking out files: 92% (139/151) Checking out files: 93% (141/151) Checking out files: 94% (142/151) Checking out files: 95% (144/151) Checking out files: 96% (145/151) Checking out files: 97% (147/151) Checking out files: 98% (148/151) Checking out files: 99% (150/151) Checking out files: 100% (151/151) Checking out files: 100% (151/151), done. 2019-12-05T06:46:35.330 INFO:tasks.workunit.client.0.smithi042.stdout:HEAD is now at 1ec1e8d6fb v0.14 2019-12-05T06:46:35.335 INFO:tasks.workunit.client.0.smithi042.stderr:+ mkdir .snap/v0.14 2019-12-05T06:46:35.559 INFO:tasks.workunit.client.0.smithi042.stderr:+ ver=v0.15 2019-12-05T06:46:35.559 INFO:tasks.workunit.client.0.smithi042.stderr:+ echo v0.15 2019-12-05T06:46:35.559 INFO:tasks.workunit.client.0.smithi042.stderr:+ git reset --hard v0.15 2019-12-05T06:46:35.559 INFO:tasks.workunit.client.0.smithi042.stdout:v0.15 2019-12-05T06:46:39.004 INFO:tasks.workunit.client.0.smithi042.stderr:Checking out files: 90% (101/112) Checking out files: 91% (102/112) Checking out files: 91% (103/112) Checking out files: 92% (104/112) Checking out files: 93% (105/112) Checking out files: 94% (106/112) Checking out files: 95% (107/112) Checking out files: 96% (108/112) Checking out files: 97% (109/112) Checking out files: 98% (110/112) Checking out files: 99% (111/112) Checking out files: 100% (112/112) Checking out files: 100% (112/112), done. 2019-12-05T06:46:39.421 INFO:tasks.workunit.client.0.smithi042.stdout:HEAD is now at ab3c421eb9 v0.15 2019-12-05T06:46:39.424 INFO:tasks.workunit.client.0.smithi042.stderr:+ mkdir .snap/v0.15 2019-12-05T06:46:40.420 INFO:tasks.workunit.client.0.smithi042.stderr:+ ver=v0.16 2019-12-05T06:46:40.420 INFO:tasks.workunit.client.0.smithi042.stderr:+ echo v0.16 2019-12-05T06:46:40.420 INFO:tasks.workunit.client.0.smithi042.stderr:+ git reset --hard v0.16 2019-12-05T06:46:40.421 INFO:tasks.workunit.client.0.smithi042.stdout:v0.16 2019-12-05T06:46:43.482 INFO:tasks.workunit.client.0.smithi042.stderr:Checking out files: 66% (71/107) Checking out files: 67% (72/107) Checking out files: 68% (73/107) Checking out files: 69% (74/107) Checking out files: 70% (75/107) Checking out files: 71% (76/107) Checking out files: 72% (78/107) Checking out files: 73% (79/107) Checking out files: 74% (80/107) Checking out files: 75% (81/107) Checking out files: 76% (82/107) Checking out files: 77% (83/107) Checking out files: 78% (84/107) Checking out files: 79% (85/107) Checking out files: 80% (86/107) Checking out files: 81% (87/107) Checking out files: 82% (88/107) Checking out files: 83% (89/107) Checking out files: 84% (90/107) Checking out files: 85% (91/107) Checking out files: 86% (93/107) Checking out files: 87% (94/107) Checking out files: 88% (95/107) Checking out files: 89% (96/107) Checking out files: 90% (97/107) Checking out files: 91% (98/107) Checking out files: 92% (99/107) Checking out files: 93% (100/107) Checking out files: 94% (101/107) Checking out files: 95% (102/107) Checking out files: 96% (103/107) Checking out files: 97% (104/107) Checking out files: 98% (105/107) Checking out files: 99% (106/107) Checking out files: 100% (107/107) Checking out files: 100% (107/107), done. 2019-12-05T06:46:44.162 INFO:tasks.workunit.client.0.smithi042.stdout:HEAD is now at 38dbaa5175 v0.16 2019-12-05T06:46:44.167 INFO:tasks.workunit.client.0.smithi042.stderr:+ mkdir .snap/v0.16 2019-12-05T06:46:44.270 INFO:tasks.workunit.client.0.smithi042.stdout:v0.17 2019-12-05T06:46:44.271 INFO:tasks.workunit.client.0.smithi042.stderr:+ ver=v0.17 2019-12-05T06:46:44.271 INFO:tasks.workunit.client.0.smithi042.stderr:+ echo v0.17 2019-12-05T06:46:44.271 INFO:tasks.workunit.client.0.smithi042.stderr:+ git reset --hard v0.17 2019-12-05T06:46:47.549 INFO:tasks.workunit.client.0.smithi042.stderr:Checking out files: 55% (120/217) Checking out files: 56% (122/217) Checking out files: 57% (124/217) Checking out files: 58% (126/217) Checking out files: 59% (129/217) Checking out files: 60% (131/217) Checking out files: 61% (133/217) Checking out files: 62% (135/217) Checking out files: 63% (137/217) Checking out files: 64% (139/217) Checking out files: 65% (142/217) Checking out files: 66% (144/217) Checking out files: 67% (146/217) Checking out files: 68% (148/217) Checking out files: 69% (150/217) Checking out files: 70% (152/217) Checking out files: 71% (155/217) Checking out files: 72% (157/217) Checking out files: 73% (159/217) Checking out files: 74% (161/217) Checking out files: 75% (163/217) Checking out files: 76% (165/217) Checking out files: 77% (168/217) Checking out files: 78% (170/217) Checking out files: 79% (172/217) Checking out files: 80% (174/217) Checking out files: 81% (176/217) Checking out files: 82% (178/217) Checking out files: 83% (181/217) Checking out files: 84% (183/217) Checking out files: 85% (185/217) Checking out files: 86% (187/217) Checking out files: 87% (189/217) Checking out files: 88% (191/217) Checking out files: 89% (194/217) Checking out files: 90% (196/217) Checking out files: 91% (198/217) Checking out files: 92% (200/217) Checking out files: 93% (202/217) Checking out files: 94% (204/217) Checking out files: 95% (207/217) Checking out files: 96% (209/217) Checking out files: 97% (211/217) Checking out files: 98% (213/217) Checking out files: 99% (215/217) Checking out files: 100% (217/217) Checking out files: 100% (217/217), done. 2019-12-05T06:46:47.683 INFO:tasks.workunit.client.0.smithi042.stdout:HEAD is now at dac3a8c35f v0.17 2019-12-05T06:46:47.687 INFO:tasks.workunit.client.0.smithi042.stderr:+ mkdir .snap/v0.17 2019-12-05T06:46:48.036 INFO:tasks.workunit.client.0.smithi042.stderr:+ ver=v0.18 2019-12-05T06:46:48.037 INFO:tasks.workunit.client.0.smithi042.stderr:+ echo v0.18 2019-12-05T06:46:48.037 INFO:tasks.workunit.client.0.smithi042.stderr:+ git reset --hard v0.18 2019-12-05T06:46:48.037 INFO:tasks.workunit.client.0.smithi042.stdout:v0.18 2019-12-05T06:46:48.554 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:46:48.558 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:47:05.073 INFO:tasks.workunit.client.0.smithi042.stderr:Checking out files: 8% (68/779) Checking out files: 9% (71/779) Checking out files: 10% (78/779) Checking out files: 11% (86/779) Checking out files: 12% (94/779) Checking out files: 13% (102/779) Checking out files: 14% (110/779) Checking out files: 15% (117/779) Checking out files: 15% (121/779) Checking out files: 16% (125/779) Checking out files: 17% (133/779) Checking out files: 18% (141/779) Checking out files: 19% (149/779) Checking out files: 20% (156/779) Checking out files: 20% (160/779) Checking out files: 21% (164/779) Checking out files: 22% (172/779) Checking out files: 23% (180/779) Checking out files: 24% (187/779) Checking out files: 25% (195/779) Checking out files: 26% (203/779) Checking out files: 27% (211/779) Checking out files: 28% (219/779) Checking out files: 28% (222/779) Checking out files: 29% (226/779) Checking out files: 30% (234/779) Checking out files: 31% (242/779) Checking out files: 32% (250/779) Checking out files: 33% (258/779) Checking out files: 34% (265/779) Checking out files: 35% (273/779) Checking out files: 36% (281/779) Checking out files: 36% (284/779) Checking out files: 37% (289/779) Checking out files: 38% (297/779) Checking out files: 39% (304/779) Checking out files: 40% (312/779) Checking out files: 41% (320/779) Checking out files: 42% (328/779) Checking out files: 43% (335/779) Checking out files: 44% (343/779) Checking out files: 45% (351/779) Checking out files: 46% (359/779) Checking out files: 47% (367/779) Checking out files: 47% (369/779) Checking out files: 48% (374/779) Checking out files: 49% (382/779) Checking out files: 50% (390/779) Checking out files: 50% (391/779) Checking out files: 51% (398/779) Checking out files: 52% (406/779) Checking out files: 53% (413/779) Checking out files: 54% (421/779) Checking out files: 55% (429/779) Checking out files: 55% (434/779) Checking out files: 55% (436/779) Checking out files: 56% (437/779) Checking out files: 56% (443/779) Checking out files: 57% (445/779) Checking out files: 58% (452/779) Checking out files: 59% (460/779) Checking out files: 60% (468/779) Checking out files: 61% (476/779) Checking out files: 62% (483/779) Checking out files: 63% (491/779) Checking out files: 64% (499/779) Checking out files: 65% (507/779) Checking out files: 66% (515/779) Checking out files: 67% (522/779) Checking out files: 68% (530/779) Checking out files: 69% (538/779) Checking out files: 70% (546/779) Checking out files: 71% (554/779) Checking out files: 72% (561/779) Checking out files: 73% (569/779) Checking out files: 73% (575/779) Checking out files: 74% (577/779) Checking out files: 75% (585/779) Checking out files: 76% (593/779) Checking out files: 77% (600/779) Checking out files: 78% (608/779) Checking out files: 79% (616/779) Checking out files: 80% (624/779) Checking out files: 81% (631/779) Checking out files: 81% (638/779) Checking out files: 82% (639/779) Checking out files: 83% (647/779) Checking out files: 84% (655/779) Checking out files: 85% (663/779) Checking out files: 86% (670/779) Checking out files: 86% (674/779) Checking out files: 87% (678/779) Checking out files: 88% (686/779) Checking out files: 89% (694/779) Checking out files: 90% (702/779) Checking out files: 91% (709/779) Checking out files: 92% (717/779) Checking out files: 93% (725/779) Checking out files: 94% (733/779) Checking out files: 95% (741/779) Checking out files: 96% (748/779) Checking out files: 97% (756/779) Checking out files: 98% (764/779) Checking out files: 99% (772/779) Checking out files: 100% (779/779) Checking out files: 100% (779/779), done. 2019-12-05T06:47:06.306 INFO:tasks.workunit.client.0.smithi042.stdout:HEAD is now at 2f10137986 v0.18 2019-12-05T06:47:06.311 INFO:tasks.workunit.client.0.smithi042.stderr:+ mkdir .snap/v0.18 2019-12-05T06:47:06.367 INFO:tasks.workunit.client.0.smithi042.stdout:v0.19 2019-12-05T06:47:06.368 INFO:tasks.workunit.client.0.smithi042.stderr:+ ver=v0.19 2019-12-05T06:47:06.368 INFO:tasks.workunit.client.0.smithi042.stderr:+ echo v0.19 2019-12-05T06:47:06.368 INFO:tasks.workunit.client.0.smithi042.stderr:+ git reset --hard v0.19 2019-12-05T06:47:12.134 INFO:tasks.workunit.client.0.smithi042.stderr:Checking out files: 33% (73/217) Checking out files: 34% (74/217) Checking out files: 35% (76/217) Checking out files: 36% (79/217) Checking out files: 37% (81/217) Checking out files: 38% (83/217) Checking out files: 39% (85/217) Checking out files: 40% (87/217) Checking out files: 41% (89/217) Checking out files: 42% (92/217) Checking out files: 43% (94/217) Checking out files: 44% (96/217) Checking out files: 45% (98/217) Checking out files: 46% (100/217) Checking out files: 47% (102/217) Checking out files: 48% (105/217) Checking out files: 49% (107/217) Checking out files: 50% (109/217) Checking out files: 51% (111/217) Checking out files: 52% (113/217) Checking out files: 53% (116/217) Checking out files: 54% (118/217) Checking out files: 55% (120/217) Checking out files: 56% (122/217) Checking out files: 57% (124/217) Checking out files: 58% (126/217) Checking out files: 59% (129/217) Checking out files: 60% (131/217) Checking out files: 61% (133/217) Checking out files: 62% (135/217) Checking out files: 63% (137/217) Checking out files: 64% (139/217) Checking out files: 65% (142/217) Checking out files: 66% (144/217) Checking out files: 67% (146/217) Checking out files: 68% (148/217) Checking out files: 69% (150/217) Checking out files: 70% (152/217) Checking out files: 71% (155/217) Checking out files: 72% (157/217) Checking out files: 73% (159/217) Checking out files: 74% (161/217) Checking out files: 75% (163/217) Checking out files: 76% (165/217) Checking out files: 77% (168/217) Checking out files: 78% (170/217) Checking out files: 79% (172/217) Checking out files: 80% (174/217) Checking out files: 81% (176/217) Checking out files: 82% (178/217) Checking out files: 83% (181/217) Checking out files: 83% (182/217) Checking out files: 84% (183/217) Checking out files: 85% (185/217) Checking out files: 86% (187/217) Checking out files: 87% (189/217) Checking out files: 88% (191/217) Checking out files: 89% (194/217) Checking out files: 90% (196/217) Checking out files: 91% (198/217) Checking out files: 92% (200/217) Checking out files: 92% (201/217) Checking out files: 93% (202/217) Checking out files: 94% (204/217) Checking out files: 95% (207/217) Checking out files: 96% (209/217) Checking out files: 97% (211/217) Checking out files: 98% (213/217) Checking out files: 99% (215/217) Checking out files: 100% (217/217) Checking out files: 100% (217/217), done. 2019-12-05T06:47:12.289 INFO:tasks.workunit.client.0.smithi042.stdout:HEAD is now at 98f5be53f8 v0.19 2019-12-05T06:47:12.293 INFO:tasks.workunit.client.0.smithi042.stderr:+ mkdir .snap/v0.19 2019-12-05T06:47:12.396 INFO:tasks.workunit.client.0.smithi042.stdout:v0.20 2019-12-05T06:47:12.396 INFO:tasks.workunit.client.0.smithi042.stderr:+ ver=v0.20 2019-12-05T06:47:12.396 INFO:tasks.workunit.client.0.smithi042.stderr:+ echo v0.20 2019-12-05T06:47:12.397 INFO:tasks.workunit.client.0.smithi042.stderr:+ git reset --hard v0.20 2019-12-05T06:47:16.945 INFO:tasks.workunit.client.0.smithi042.stderr:Checking out files: 18% (45/245) Checking out files: 19% (47/245) Checking out files: 20% (49/245) Checking out files: 21% (52/245) Checking out files: 22% (54/245) Checking out files: 23% (57/245) Checking out files: 24% (59/245) Checking out files: 25% (62/245) Checking out files: 26% (64/245) Checking out files: 27% (67/245) Checking out files: 28% (69/245) Checking out files: 29% (72/245) Checking out files: 30% (74/245) Checking out files: 31% (76/245) Checking out files: 32% (79/245) Checking out files: 33% (81/245) Checking out files: 34% (84/245) Checking out files: 35% (86/245) Checking out files: 36% (89/245) Checking out files: 37% (91/245) Checking out files: 38% (94/245) Checking out files: 39% (96/245) Checking out files: 40% (98/245) Checking out files: 41% (101/245) Checking out files: 42% (103/245) Checking out files: 43% (106/245) Checking out files: 44% (108/245) Checking out files: 45% (111/245) Checking out files: 46% (113/245) Checking out files: 47% (116/245) Checking out files: 48% (118/245) Checking out files: 49% (121/245) Checking out files: 50% (123/245) Checking out files: 51% (125/245) Checking out files: 52% (128/245) Checking out files: 53% (130/245) Checking out files: 54% (133/245) Checking out files: 54% (134/245) Checking out files: 55% (135/245) Checking out files: 56% (138/245) Checking out files: 57% (140/245) Checking out files: 58% (143/245) Checking out files: 59% (145/245) Checking out files: 60% (147/245) Checking out files: 61% (150/245) Checking out files: 62% (152/245) Checking out files: 63% (155/245) Checking out files: 64% (157/245) Checking out files: 65% (160/245) Checking out files: 66% (162/245) Checking out files: 67% (165/245) Checking out files: 68% (167/245) Checking out files: 69% (170/245) Checking out files: 70% (172/245) Checking out files: 71% (174/245) Checking out files: 72% (177/245) Checking out files: 73% (179/245) Checking out files: 74% (182/245) Checking out files: 75% (184/245) Checking out files: 76% (187/245) Checking out files: 77% (189/245) Checking out files: 78% (192/245) Checking out files: 79% (194/245) Checking out files: 80% (196/245) Checking out files: 81% (199/245) Checking out files: 82% (201/245) Checking out files: 83% (204/245) Checking out files: 84% (206/245) Checking out files: 85% (209/245) Checking out files: 86% (211/245) Checking out files: 87% (214/245) Checking out files: 88% (216/245) Checking out files: 89% (219/245) Checking out files: 90% (221/245) Checking out files: 91% (223/245) Checking out files: 92% (226/245) Checking out files: 93% (228/245) Checking out files: 94% (231/245) Checking out files: 95% (233/245) Checking out files: 96% (236/245) Checking out files: 97% (238/245) Checking out files: 98% (241/245) Checking out files: 99% (243/245) Checking out files: 100% (245/245) Checking out files: 100% (245/245), done. 2019-12-05T06:47:17.035 INFO:tasks.workunit.client.0.smithi042.stdout:HEAD is now at 266d2076b6 debian: fix up debian scripts 2019-12-05T06:47:17.040 INFO:tasks.workunit.client.0.smithi042.stderr:+ mkdir .snap/v0.20 2019-12-05T06:47:17.621 INFO:tasks.workunit.client.0.smithi042.stdout:v0.21 2019-12-05T06:47:17.622 INFO:tasks.workunit.client.0.smithi042.stderr:+ ver=v0.21 2019-12-05T06:47:17.622 INFO:tasks.workunit.client.0.smithi042.stderr:+ echo v0.21 2019-12-05T06:47:17.622 INFO:tasks.workunit.client.0.smithi042.stderr:+ git reset --hard v0.21 2019-12-05T06:47:20.270 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:47:20.305 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:47:27.359 INFO:tasks.workunit.client.0.smithi042.stderr:Checking out files: 8% (46/537) Checking out files: 9% (49/537) Checking out files: 10% (54/537) Checking out files: 11% (60/537) Checking out files: 12% (65/537) Checking out files: 13% (70/537) Checking out files: 14% (76/537) Checking out files: 15% (81/537) Checking out files: 16% (86/537) Checking out files: 16% (87/537) Checking out files: 17% (92/537) Checking out files: 18% (97/537) Checking out files: 19% (103/537) Checking out files: 20% (108/537) Checking out files: 21% (113/537) Checking out files: 22% (119/537) Checking out files: 23% (124/537) Checking out files: 24% (129/537) Checking out files: 25% (135/537) Checking out files: 26% (140/537) Checking out files: 27% (145/537) Checking out files: 28% (151/537) Checking out files: 29% (156/537) Checking out files: 30% (162/537) Checking out files: 31% (167/537) Checking out files: 32% (172/537) Checking out files: 33% (178/537) Checking out files: 33% (181/537) Checking out files: 34% (183/537) Checking out files: 35% (188/537) Checking out files: 36% (194/537) Checking out files: 37% (199/537) Checking out files: 38% (205/537) Checking out files: 39% (210/537) Checking out files: 40% (215/537) Checking out files: 41% (221/537) Checking out files: 42% (226/537) Checking out files: 43% (231/537) Checking out files: 44% (237/537) Checking out files: 45% (242/537) Checking out files: 46% (248/537) Checking out files: 47% (253/537) Checking out files: 48% (258/537) Checking out files: 49% (264/537) Checking out files: 50% (269/537) Checking out files: 51% (274/537) Checking out files: 52% (280/537) Checking out files: 53% (285/537) Checking out files: 54% (290/537) Checking out files: 55% (296/537) Checking out files: 56% (301/537) Checking out files: 57% (307/537) Checking out files: 58% (312/537) Checking out files: 59% (317/537) Checking out files: 60% (323/537) Checking out files: 61% (328/537) Checking out files: 62% (333/537) Checking out files: 63% (339/537) Checking out files: 64% (344/537) Checking out files: 65% (350/537) Checking out files: 66% (355/537) Checking out files: 67% (360/537) Checking out files: 68% (366/537) Checking out files: 69% (371/537) Checking out files: 70% (376/537) Checking out files: 71% (382/537) Checking out files: 71% (386/537) Checking out files: 72% (387/537) Checking out files: 73% (393/537) Checking out files: 74% (398/537) Checking out files: 75% (403/537) Checking out files: 76% (409/537) Checking out files: 77% (414/537) Checking out files: 78% (419/537) Checking out files: 79% (425/537) Checking out files: 80% (430/537) Checking out files: 81% (435/537) Checking out files: 82% (441/537) Checking out files: 83% (446/537) Checking out files: 84% (452/537) Checking out files: 85% (457/537) Checking out files: 86% (462/537) Checking out files: 87% (468/537) Checking out files: 88% (473/537) Checking out files: 89% (478/537) Checking out files: 89% (483/537) Checking out files: 90% (484/537) Checking out files: 91% (489/537) Checking out files: 91% (491/537) Checking out files: 92% (495/537) Checking out files: 93% (500/537) Checking out files: 94% (505/537) Checking out files: 95% (511/537) Checking out files: 96% (516/537) Checking out files: 97% (521/537) Checking out files: 98% (527/537) Checking out files: 99% (532/537) Checking out files: 100% (537/537) Checking out files: 100% (537/537), done. 2019-12-05T06:47:28.116 INFO:tasks.workunit.client.0.smithi042.stdout:HEAD is now at 090436f56a v0.21 2019-12-05T06:47:28.121 INFO:tasks.workunit.client.0.smithi042.stderr:+ mkdir .snap/v0.21 2019-12-05T06:47:28.844 INFO:tasks.workunit.client.0.smithi042.stderr:+ ver=v0.1 2019-12-05T06:47:28.844 INFO:tasks.workunit.client.0.smithi042.stderr:+ echo checking v0.1 2019-12-05T06:47:28.844 INFO:tasks.workunit.client.0.smithi042.stderr:+ cd .snap/v0.1 2019-12-05T06:47:28.845 INFO:tasks.workunit.client.0.smithi042.stdout:checking v0.1 2019-12-05T06:47:29.053 INFO:tasks.workunit.client.0.smithi042.stderr:+ git diff --exit-code 2019-12-05T06:47:39.251 INFO:tasks.workunit.client.0.smithi042.stderr:+ cd ../.. 2019-12-05T06:47:39.252 INFO:tasks.workunit.client.0.smithi042.stderr:+ ver=v0.2 2019-12-05T06:47:39.252 INFO:tasks.workunit.client.0.smithi042.stderr:+ echo checking v0.2 2019-12-05T06:47:39.253 INFO:tasks.workunit.client.0.smithi042.stderr:+ cd .snap/v0.2 2019-12-05T06:47:39.253 INFO:tasks.workunit.client.0.smithi042.stdout:checking v0.2 2019-12-05T06:47:39.262 INFO:tasks.workunit.client.0.smithi042.stderr:+ git diff --exit-code 2019-12-05T06:47:46.931 INFO:tasks.workunit.client.0.smithi042.stderr:+ cd ../.. 2019-12-05T06:47:46.932 INFO:tasks.workunit.client.0.smithi042.stderr:+ ver=v0.3 2019-12-05T06:47:46.932 INFO:tasks.workunit.client.0.smithi042.stderr:+ echo checking v0.3 2019-12-05T06:47:46.932 INFO:tasks.workunit.client.0.smithi042.stderr:+ cd .snap/v0.3 2019-12-05T06:47:46.932 INFO:tasks.workunit.client.0.smithi042.stdout:checking v0.3 2019-12-05T06:47:46.937 INFO:tasks.workunit.client.0.smithi042.stderr:+ git diff --exit-code 2019-12-05T06:47:50.404 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:47:50.409 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:47:56.329 INFO:tasks.workunit.client.0.smithi042.stderr:+ cd ../.. 2019-12-05T06:47:56.330 INFO:tasks.workunit.client.0.smithi042.stderr:+ ver=v0.4 2019-12-05T06:47:56.330 INFO:tasks.workunit.client.0.smithi042.stderr:+ echo checking v0.4 2019-12-05T06:47:56.330 INFO:tasks.workunit.client.0.smithi042.stderr:+ cd .snap/v0.4 2019-12-05T06:47:56.331 INFO:tasks.workunit.client.0.smithi042.stdout:checking v0.4 2019-12-05T06:47:56.334 INFO:tasks.workunit.client.0.smithi042.stderr:+ git diff --exit-code 2019-12-05T06:48:02.781 INFO:tasks.workunit.client.0.smithi042.stderr:+ cd ../.. 2019-12-05T06:48:02.781 INFO:tasks.workunit.client.0.smithi042.stderr:+ ver=v0.5 2019-12-05T06:48:02.781 INFO:tasks.workunit.client.0.smithi042.stderr:+ echo checking v0.5 2019-12-05T06:48:02.781 INFO:tasks.workunit.client.0.smithi042.stderr:+ cd .snap/v0.5 2019-12-05T06:48:02.782 INFO:tasks.workunit.client.0.smithi042.stdout:checking v0.5 2019-12-05T06:48:02.783 INFO:tasks.workunit.client.0.smithi042.stderr:+ git diff --exit-code 2019-12-05T06:48:07.453 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:48:07.618 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.608+0000 7f4851cd0700 1 Processor -- start 2019-12-05T06:48:07.619 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.608+0000 7f4851cd0700 1 -- start start 2019-12-05T06:48:07.619 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.608+0000 7f4851cd0700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f484c08df80 0x7f484c09f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:48:07.619 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.608+0000 7f4851cd0700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f484c09f830 0x7f484c14efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:48:07.619 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.608+0000 7f4851cd0700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f484c09fc60 con 0x7f484c0a7740 2019-12-05T06:48:07.620 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.608+0000 7f4851cd0700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f484c0a0b00 con 0x7f484c08df80 2019-12-05T06:48:07.620 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.608+0000 7f4851cd0700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f484c08e5b0 con 0x7f484c09f830 2019-12-05T06:48:07.620 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.608+0000 7f4850a6e700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f484c09f830 0x7f484c14efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:07.620 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.608+0000 7f484b7fe700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f484c08df80 0x7f484c09f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:07.621 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.608+0000 7f4850a6e700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f484c09f830 0x7f484c14efe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:37530/0 (socket says 172.21.15.42:37530) 2019-12-05T06:48:07.621 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.608+0000 7f4850a6e700 1 -- 172.21.15.42:0/3174834199 learned_addr learned my addr 172.21.15.42:0/3174834199 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:48:07.621 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.608+0000 7f484bfff700 1 --1- 172.21.15.42:0/3174834199 >> v1:172.21.15.40:6789/0 conn(0x7f484c0a7740 0x7f484c07e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:46540/0 (socket says 172.21.15.42:46540) 2019-12-05T06:48:07.621 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.608+0000 7f4850a6e700 1 -- 172.21.15.42:0/3174834199 >> v2:172.21.15.40:3300/0 conn(0x7f484c08df80 msgr2=0x7f484c09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:48:07.622 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.608+0000 7f4850a6e700 1 --2- 172.21.15.42:0/3174834199 >> v2:172.21.15.40:3300/0 conn(0x7f484c08df80 0x7f484c09f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:48:07.622 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.608+0000 7f4850a6e700 1 -- 172.21.15.42:0/3174834199 >> v1:172.21.15.40:6789/0 conn(0x7f484c0a7740 legacy=0x7f484c07e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:07.622 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.608+0000 7f4850a6e700 1 -- 172.21.15.42:0/3174834199 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f484c0a0710 con 0x7f484c09f830 2019-12-05T06:48:07.622 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.608+0000 7f4850a6e700 1 --2- 172.21.15.42:0/3174834199 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f484c09f830 0x7f484c14efe0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f4844008b10 tx=0x7f48440100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:07.623 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.608+0000 7f4850a6e700 1 --2- 172.21.15.42:0/3174834199 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f484c09f830 0x7f484c14efe0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f4844008b10 tx=0x7f48440100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:07.623 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.608+0000 7f4850a6e700 1 --2- 172.21.15.42:0/3174834199 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f484c09f830 0x7f484c14efe0 secure :-1 s=READY pgs=712 cs=0 l=1 rx=0x7f4844008b10 tx=0x7f48440100b0).ready entity=mon.2 client_cookie=3ebe581f8a1e9ce9 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:48:07.623 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.608+0000 7f4850a6e700 1 --2- 172.21.15.42:0/3174834199 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f484c09f830 0x7f484c14efe0 secure :-1 s=THROTTLE_DONE pgs=712 cs=0 l=1 rx=0x7f4844008b10 tx=0x7f48440100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:07.623 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.608+0000 7f484affd700 1 -- 172.21.15.42:0/3174834199 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4844010da0 con 0x7f484c09f830 2019-12-05T06:48:07.624 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.608+0000 7f4850a6e700 1 --2- 172.21.15.42:0/3174834199 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f484c09f830 0x7f484c14efe0 secure :-1 s=THROTTLE_DONE pgs=712 cs=0 l=1 rx=0x7f4844008b10 tx=0x7f48440100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:07.624 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.608+0000 7f4850a6e700 1 --2- 172.21.15.42:0/3174834199 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f484c09f830 0x7f484c14efe0 secure :-1 s=THROTTLE_DONE pgs=712 cs=0 l=1 rx=0x7f4844008b10 tx=0x7f48440100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:07.624 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.608+0000 7f484affd700 1 -- 172.21.15.42:0/3174834199 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48440109c0 con 0x7f484c09f830 2019-12-05T06:48:07.624 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.608+0000 7f484affd700 1 -- 172.21.15.42:0/3174834199 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f48440114a0 con 0x7f484c09f830 2019-12-05T06:48:07.625 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.608+0000 7f4851cd0700 1 -- 172.21.15.42:0/3174834199 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f484c09f830 msgr2=0x7f484c14efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:07.625 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.608+0000 7f4851cd0700 1 --2- 172.21.15.42:0/3174834199 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f484c09f830 0x7f484c14efe0 secure :-1 s=READY pgs=712 cs=0 l=1 rx=0x7f4844008b10 tx=0x7f48440100b0).stop 2019-12-05T06:48:07.625 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.612+0000 7f4851cd0700 1 -- 172.21.15.42:0/3174834199 shutdown_connections 2019-12-05T06:48:07.626 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.612+0000 7f4851cd0700 1 --2- 172.21.15.42:0/3174834199 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f484c09f830 0x7f484c14efe0 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:07.626 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.612+0000 7f4851cd0700 1 --2- 172.21.15.42:0/3174834199 >> v2:172.21.15.40:3300/0 conn(0x7f484c08df80 0x7f484c09f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:48:07.626 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.612+0000 7f4851cd0700 1 -- 172.21.15.42:0/3174834199 shutdown_connections 2019-12-05T06:48:07.626 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.612+0000 7f4851cd0700 1 -- 172.21.15.42:0/3174834199 wait complete. 2019-12-05T06:48:07.627 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.612+0000 7f4851cd0700 1 -- 172.21.15.42:0/3174834199 >> 172.21.15.42:0/3174834199 conn(0x7f484c0aabe0 msgr2=0x7f484c0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:48:07.634 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.612+0000 7f4851cd0700 1 Processor -- start 2019-12-05T06:48:07.635 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.612+0000 7f4851cd0700 1 -- start start 2019-12-05T06:48:07.635 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.616+0000 7f4851cd0700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f484c0a17a0 0x7f484c0a3580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:48:07.635 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.616+0000 7f4851cd0700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f484c0a3ad0 con 0x7f484c0a2100 2019-12-05T06:48:07.635 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.616+0000 7f4851cd0700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f484c0a3d00 con 0x7f484c119c00 2019-12-05T06:48:07.636 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.616+0000 7f4851cd0700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f484c0a1bd0 con 0x7f484c0a17a0 2019-12-05T06:48:07.636 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.616+0000 7f484bfff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f484c119c00 0x7f484c0a5f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:37106/0 (socket says 172.21.15.42:37106) 2019-12-05T06:48:07.636 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.616+0000 7f484bfff700 1 -- 172.21.15.42:0/1110518352 learned_addr learned my addr 172.21.15.42:0/1110518352 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:48:07.636 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.616+0000 7f4850a6e700 1 --2- 172.21.15.42:0/1110518352 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f484c0a17a0 0x7f484c0a3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:07.637 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.616+0000 7f4848ff9700 1 -- 172.21.15.42:0/1110518352 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 653343097 0 0) 0x7f484c0a3d00 con 0x7f484c119c00 2019-12-05T06:48:07.637 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.616+0000 7f4848ff9700 1 -- 172.21.15.42:0/1110518352 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f482c0030b0 con 0x7f484c119c00 2019-12-05T06:48:07.637 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.616+0000 7f4848ff9700 1 -- 172.21.15.42:0/1110518352 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3877695704 0 0) 0x7f484c0a3ad0 con 0x7f484c0a2100 2019-12-05T06:48:07.637 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.616+0000 7f4850a6e700 1 -- 172.21.15.42:0/1110518352 >> v1:172.21.15.42:6789/0 conn(0x7f484c119c00 legacy=0x7f484c0a5f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:07.638 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.616+0000 7f4850a6e700 1 -- 172.21.15.42:0/1110518352 >> v1:172.21.15.40:6789/0 conn(0x7f484c0a2100 legacy=0x7f484c14e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:07.638 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.616+0000 7f4850a6e700 1 -- 172.21.15.42:0/1110518352 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f484c0a3f30 con 0x7f484c0a17a0 2019-12-05T06:48:07.638 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.616+0000 7f4850a6e700 1 --2- 172.21.15.42:0/1110518352 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f484c0a17a0 0x7f484c0a3580 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f4844008b10 tx=0x7f4844010c50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:07.638 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.616+0000 7f4850a6e700 1 --2- 172.21.15.42:0/1110518352 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f484c0a17a0 0x7f484c0a3580 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f4844008b10 tx=0x7f4844010c50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:07.638 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.616+0000 7f4850a6e700 1 --2- 172.21.15.42:0/1110518352 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f484c0a17a0 0x7f484c0a3580 secure :-1 s=READY pgs=713 cs=0 l=1 rx=0x7f4844008b10 tx=0x7f4844010c50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:48:07.639 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.616+0000 7f4850a6e700 1 --2- 172.21.15.42:0/1110518352 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f484c0a17a0 0x7f484c0a3580 secure :-1 s=THROTTLE_DONE pgs=713 cs=0 l=1 rx=0x7f4844008b10 tx=0x7f4844010c50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:07.640 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.616+0000 7f4848ff9700 1 -- 172.21.15.42:0/1110518352 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4844011bb0 con 0x7f484c0a17a0 2019-12-05T06:48:07.640 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.616+0000 7f4850a6e700 1 --2- 172.21.15.42:0/1110518352 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f484c0a17a0 0x7f484c0a3580 secure :-1 s=THROTTLE_DONE pgs=713 cs=0 l=1 rx=0x7f4844008b10 tx=0x7f4844010c50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:07.640 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.616+0000 7f4848ff9700 1 -- 172.21.15.42:0/1110518352 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48440110c0 con 0x7f484c0a17a0 2019-12-05T06:48:07.640 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.616+0000 7f4850a6e700 1 --2- 172.21.15.42:0/1110518352 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f484c0a17a0 0x7f484c0a3580 secure :-1 s=THROTTLE_DONE pgs=713 cs=0 l=1 rx=0x7f4844008b10 tx=0x7f4844010c50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:07.641 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.616+0000 7f4851cd0700 1 -- 172.21.15.42:0/1110518352 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f484c12ed90 con 0x7f484c0a17a0 2019-12-05T06:48:07.641 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.616+0000 7f4848ff9700 1 -- 172.21.15.42:0/1110518352 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4844005ab0 con 0x7f484c0a17a0 2019-12-05T06:48:07.641 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.616+0000 7f4851cd0700 1 -- 172.21.15.42:0/1110518352 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f484c12f2e0 con 0x7f484c0a17a0 2019-12-05T06:48:07.641 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.620+0000 7f4850a6e700 1 --2- 172.21.15.42:0/1110518352 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f484c0a17a0 0x7f484c0a3580 secure :-1 s=THROTTLE_DONE pgs=713 cs=0 l=1 rx=0x7f4844008b10 tx=0x7f4844010c50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:07.641 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.620+0000 7f4848ff9700 1 -- 172.21.15.42:0/1110518352 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f484400f280 con 0x7f484c0a17a0 2019-12-05T06:48:07.642 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.620+0000 7f4850a6e700 1 --2- 172.21.15.42:0/1110518352 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f484c0a17a0 0x7f484c0a3580 secure :-1 s=THROTTLE_DONE pgs=713 cs=0 l=1 rx=0x7f4844008b10 tx=0x7f4844010c50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:07.642 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.620+0000 7f4848ff9700 1 --2- 172.21.15.42:0/1110518352 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f482c03b1e0 0x7f482c03d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:48:07.642 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.620+0000 7f4848ff9700 1 -- 172.21.15.42:0/1110518352 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 12051+0+0 (secure 0 0 0) 0x7f484404b350 con 0x7f484c0a17a0 2019-12-05T06:48:07.642 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.620+0000 7f484bfff700 1 --2- 172.21.15.42:0/1110518352 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f482c03b1e0 0x7f482c03d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:07.643 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.620+0000 7f484bfff700 1 --2- 172.21.15.42:0/1110518352 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f482c03b1e0 0x7f482c03d6f0 crc :-1 s=READY pgs=1345 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:48:07.643 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.624+0000 7f4851cd0700 1 -- 172.21.15.42:0/1110518352 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f484c05d110 con 0x7f484c0a17a0 2019-12-05T06:48:07.644 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.632+0000 7f4850a6e700 1 --2- 172.21.15.42:0/1110518352 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f484c0a17a0 0x7f484c0a3580 secure :-1 s=THROTTLE_DONE pgs=713 cs=0 l=1 rx=0x7f4844008b10 tx=0x7f4844010c50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:07.645 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.632+0000 7f4848ff9700 1 -- 172.21.15.42:0/1110518352 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f4844005d50 con 0x7f484c0a17a0 2019-12-05T06:48:07.808 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.800+0000 7f4851cd0700 1 -- 172.21.15.42:0/1110518352 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f484c05b8c0 con 0x7f484c0a17a0 2019-12-05T06:48:07.809 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.804+0000 7f4850a6e700 1 --2- 172.21.15.42:0/1110518352 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f484c0a17a0 0x7f484c0a3580 secure :-1 s=THROTTLE_DONE pgs=713 cs=0 l=1 rx=0x7f4844008b10 tx=0x7f4844010c50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:07.809 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.804+0000 7f4848ff9700 1 -- 172.21.15.42:0/1110518352 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 389 v389) v1 ==== 95+0+2117 (secure 0 0 0) 0x7f48440331c0 con 0x7f484c0a17a0 2019-12-05T06:48:07.811 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 389 2019-12-05T06:48:07.811 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:48:07.812 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":389,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":12847,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/1449317789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1449317789},{"type":"v1","addr":"172.21.15.42:6835","nonce":1449317789}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":389}],"filesystems":[{"mdsmap":{"epoch":388,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:46:05.602252+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":119,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12429},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12429":{"gid":12429,"name":"b","rank":0,"incarnation":384,"state":"up:active","state_seq":26,"addr":"172.21.15.40:6833/4178556019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":4178556019},{"type":"v1","addr":"172.21.15.40:6833","nonce":4178556019}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:48:07.814 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.808+0000 7f4851cd0700 1 -- 172.21.15.42:0/1110518352 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f482c03b1e0 msgr2=0x7f482c03d6f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:07.814 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.808+0000 7f4851cd0700 1 --2- 172.21.15.42:0/1110518352 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f482c03b1e0 0x7f482c03d6f0 crc :-1 s=READY pgs=1345 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:07.814 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.808+0000 7f4851cd0700 1 -- 172.21.15.42:0/1110518352 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f484c0a17a0 msgr2=0x7f484c0a3580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:07.814 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.808+0000 7f4851cd0700 1 --2- 172.21.15.42:0/1110518352 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f484c0a17a0 0x7f484c0a3580 secure :-1 s=READY pgs=713 cs=0 l=1 rx=0x7f4844008b10 tx=0x7f4844010c50).stop 2019-12-05T06:48:07.814 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.808+0000 7f4851cd0700 1 -- 172.21.15.42:0/1110518352 shutdown_connections 2019-12-05T06:48:07.815 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.808+0000 7f4851cd0700 1 --2- 172.21.15.42:0/1110518352 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f482c03b1e0 0x7f482c03d6f0 unknown :-1 s=CLOSED pgs=1345 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:07.815 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.808+0000 7f4851cd0700 1 --2- 172.21.15.42:0/1110518352 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f484c0a17a0 0x7f484c0a3580 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:07.815 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.808+0000 7f4851cd0700 1 -- 172.21.15.42:0/1110518352 shutdown_connections 2019-12-05T06:48:07.815 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.808+0000 7f4851cd0700 1 -- 172.21.15.42:0/1110518352 wait complete. 2019-12-05T06:48:07.815 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.808+0000 7f4851cd0700 1 -- 172.21.15.42:0/1110518352 >> 172.21.15.42:0/1110518352 conn(0x7f484c0a9040 msgr2=0x7f484c07ce10 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:48:07.824 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.b (rank=0) 2019-12-05T06:48:07.824 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2019-12-05T06:48:07.824 INFO:teuthology.orchestra.run:waiting for 300 2019-12-05T06:48:07.841 INFO:tasks.ceph.mds.b:Stopped 2019-12-05T06:48:07.841 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:48:07.946 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.932+0000 7fd8813da700 1 Processor -- start 2019-12-05T06:48:07.947 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.932+0000 7fd8813da700 1 -- start start 2019-12-05T06:48:07.947 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.932+0000 7fd8813da700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd87c0a7740 0x7fd87c08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:48:07.947 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.932+0000 7fd8813da700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd87c07e720 0x7fd87c07eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:48:07.948 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.932+0000 7fd8813da700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd87c0a0710 con 0x7fd87c09f2e0 2019-12-05T06:48:07.948 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.932+0000 7fd8813da700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd87c0a0b00 con 0x7fd87c07e720 2019-12-05T06:48:07.948 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.932+0000 7fd8813da700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd87c08e5b0 con 0x7fd87c0a7740 2019-12-05T06:48:07.948 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.932+0000 7fd87affd700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd87c07e720 0x7fd87c07eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:07.949 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.932+0000 7fd87affd700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd87c07e720 0x7fd87c07eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:37078/0 (socket says 172.21.15.42:37078) 2019-12-05T06:48:07.949 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.932+0000 7fd87affd700 1 -- 172.21.15.42:0/1847431026 learned_addr learned my addr 172.21.15.42:0/1847431026 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:48:07.950 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.932+0000 7fd87b7fe700 1 --2- 172.21.15.42:0/1847431026 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd87c0a7740 0x7fd87c08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:07.950 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.932+0000 7fd87affd700 1 -- 172.21.15.42:0/1847431026 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd87c0a7740 msgr2=0x7fd87c08df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:48:07.950 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.932+0000 7fd87affd700 1 --2- 172.21.15.42:0/1847431026 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd87c0a7740 0x7fd87c08df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:48:07.950 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.932+0000 7fd87affd700 1 -- 172.21.15.42:0/1847431026 >> v1:172.21.15.40:6789/0 conn(0x7fd87c09f2e0 legacy=0x7fd87c09f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:07.951 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.932+0000 7fd87affd700 1 -- 172.21.15.42:0/1847431026 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd87c07f0a0 con 0x7fd87c07e720 2019-12-05T06:48:07.951 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.932+0000 7fd87affd700 1 --2- 172.21.15.42:0/1847431026 >> v2:172.21.15.42:3300/0 conn(0x7fd87c07e720 0x7fd87c07eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fd86c008790 tx=0x7fd86c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:07.951 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.932+0000 7fd87affd700 1 --2- 172.21.15.42:0/1847431026 >> v2:172.21.15.42:3300/0 conn(0x7fd87c07e720 0x7fd87c07eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fd86c008790 tx=0x7fd86c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:07.951 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.932+0000 7fd87affd700 1 --2- 172.21.15.42:0/1847431026 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd87c07e720 0x7fd87c07eb50 secure :-1 s=READY pgs=3208 cs=0 l=1 rx=0x7fd86c008790 tx=0x7fd86c00fd30).ready entity=mon.1 client_cookie=928b1c0c0b92a154 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:48:07.952 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.932+0000 7fd87affd700 1 --2- 172.21.15.42:0/1847431026 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd87c07e720 0x7fd87c07eb50 secure :-1 s=THROTTLE_DONE pgs=3208 cs=0 l=1 rx=0x7fd86c008790 tx=0x7fd86c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:07.953 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.932+0000 7fd87a7fc700 1 -- 172.21.15.42:0/1847431026 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd86c0113b0 con 0x7fd87c07e720 2019-12-05T06:48:07.954 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.932+0000 7fd87affd700 1 --2- 172.21.15.42:0/1847431026 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd87c07e720 0x7fd87c07eb50 secure :-1 s=THROTTLE_DONE pgs=3208 cs=0 l=1 rx=0x7fd86c008790 tx=0x7fd86c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:07.954 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.932+0000 7fd87a7fc700 1 -- 172.21.15.42:0/1847431026 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd86c008bb0 con 0x7fd87c07e720 2019-12-05T06:48:07.954 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.932+0000 7fd87affd700 1 --2- 172.21.15.42:0/1847431026 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd87c07e720 0x7fd87c07eb50 secure :-1 s=THROTTLE_DONE pgs=3208 cs=0 l=1 rx=0x7fd86c008790 tx=0x7fd86c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:07.954 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.932+0000 7fd87a7fc700 1 -- 172.21.15.42:0/1847431026 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd86c011990 con 0x7fd87c07e720 2019-12-05T06:48:07.955 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.932+0000 7fd8813da700 1 -- 172.21.15.42:0/1847431026 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd87c07e720 msgr2=0x7fd87c07eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:07.955 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.932+0000 7fd8813da700 1 --2- 172.21.15.42:0/1847431026 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd87c07e720 0x7fd87c07eb50 secure :-1 s=READY pgs=3208 cs=0 l=1 rx=0x7fd86c008790 tx=0x7fd86c00fd30).stop 2019-12-05T06:48:07.955 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.936+0000 7fd8813da700 1 -- 172.21.15.42:0/1847431026 shutdown_connections 2019-12-05T06:48:07.955 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.936+0000 7fd8813da700 1 --2- 172.21.15.42:0/1847431026 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd87c0a7740 0x7fd87c08df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:48:07.956 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.936+0000 7fd8813da700 1 --2- 172.21.15.42:0/1847431026 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd87c07e720 0x7fd87c07eb50 unknown :-1 s=CLOSED pgs=3208 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:07.956 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.936+0000 7fd8813da700 1 -- 172.21.15.42:0/1847431026 shutdown_connections 2019-12-05T06:48:07.956 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.936+0000 7fd8813da700 1 -- 172.21.15.42:0/1847431026 wait complete. 2019-12-05T06:48:07.956 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.936+0000 7fd8813da700 1 -- 172.21.15.42:0/1847431026 >> 172.21.15.42:0/1847431026 conn(0x7fd87c0aabe0 msgr2=0x7fd87c0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:48:07.957 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.936+0000 7fd8813da700 1 Processor -- start 2019-12-05T06:48:07.957 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.936+0000 7fd8813da700 1 -- start start 2019-12-05T06:48:07.957 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.936+0000 7fd8813da700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd87c08f080 0x7fd87c142de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:48:07.957 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.936+0000 7fd8813da700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd87c129350 0x7fd87c129780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:48:07.959 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.936+0000 7fd8813da700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd87c129cd0 con 0x7fd87c1229d0 2019-12-05T06:48:07.959 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.936+0000 7fd8813da700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd87c064b10 con 0x7fd87c129350 2019-12-05T06:48:07.959 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.936+0000 7fd8813da700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd87c064db0 con 0x7fd87c08f080 2019-12-05T06:48:07.959 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.936+0000 7fd87affd700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd87c129350 0x7fd87c129780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:07.960 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.936+0000 7fd87affd700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd87c129350 0x7fd87c129780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:37088/0 (socket says 172.21.15.42:37088) 2019-12-05T06:48:07.960 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.936+0000 7fd87affd700 1 -- 172.21.15.42:0/1837825530 learned_addr learned my addr 172.21.15.42:0/1837825530 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:48:07.960 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.936+0000 7fd87b7fe700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd87c08f080 0x7fd87c142de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:07.961 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.936+0000 7fd87affd700 1 -- 172.21.15.42:0/1837825530 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd87c08f080 msgr2=0x7fd87c142de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:07.961 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.936+0000 7fd87affd700 1 --2- 172.21.15.42:0/1837825530 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd87c08f080 0x7fd87c142de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:07.961 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.936+0000 7fd87affd700 1 -- 172.21.15.42:0/1837825530 >> v1:172.21.15.40:6789/0 conn(0x7fd87c1229d0 legacy=0x7fd87c122e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:07.961 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.936+0000 7fd873fff700 1 -- 172.21.15.42:0/1837825530 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 469108244 0 0) 0x7fd87c129cd0 con 0x7fd87c1229d0 2019-12-05T06:48:07.962 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.936+0000 7fd87affd700 1 -- 172.21.15.42:0/1837825530 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd87c0646e0 con 0x7fd87c129350 2019-12-05T06:48:07.962 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.936+0000 7fd87affd700 1 --2- 172.21.15.42:0/1837825530 >> v2:172.21.15.42:3300/0 conn(0x7fd87c129350 0x7fd87c129780 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fd87c064cd0 tx=0x7fd86c011f80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:07.962 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.936+0000 7fd87affd700 1 --2- 172.21.15.42:0/1837825530 >> v2:172.21.15.42:3300/0 conn(0x7fd87c129350 0x7fd87c129780 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fd87c064cd0 tx=0x7fd86c011f80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:07.963 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.936+0000 7fd87affd700 1 --2- 172.21.15.42:0/1837825530 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd87c129350 0x7fd87c129780 secure :-1 s=READY pgs=3209 cs=0 l=1 rx=0x7fd87c064cd0 tx=0x7fd86c011f80).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:48:07.963 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.936+0000 7fd87affd700 1 --2- 172.21.15.42:0/1837825530 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd87c129350 0x7fd87c129780 secure :-1 s=THROTTLE_DONE pgs=3209 cs=0 l=1 rx=0x7fd87c064cd0 tx=0x7fd86c011f80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:07.963 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.936+0000 7fd873fff700 1 -- 172.21.15.42:0/1837825530 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd86c007340 con 0x7fd87c129350 2019-12-05T06:48:07.964 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.936+0000 7fd87affd700 1 --2- 172.21.15.42:0/1837825530 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd87c129350 0x7fd87c129780 secure :-1 s=THROTTLE_DONE pgs=3209 cs=0 l=1 rx=0x7fd87c064cd0 tx=0x7fd86c011f80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:07.964 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.936+0000 7fd873fff700 1 -- 172.21.15.42:0/1837825530 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd86c0115b0 con 0x7fd87c129350 2019-12-05T06:48:07.964 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.936+0000 7fd8813da700 1 -- 172.21.15.42:0/1837825530 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd87c123600 con 0x7fd87c129350 2019-12-05T06:48:07.964 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.936+0000 7fd87affd700 1 --2- 172.21.15.42:0/1837825530 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd87c129350 0x7fd87c129780 secure :-1 s=THROTTLE_DONE pgs=3209 cs=0 l=1 rx=0x7fd87c064cd0 tx=0x7fd86c011f80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:07.965 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.940+0000 7fd873fff700 1 -- 172.21.15.42:0/1837825530 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd86c005540 con 0x7fd87c129350 2019-12-05T06:48:07.965 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.940+0000 7fd8813da700 1 -- 172.21.15.42:0/1837825530 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd87c130b20 con 0x7fd87c129350 2019-12-05T06:48:07.965 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.940+0000 7fd87affd700 1 --2- 172.21.15.42:0/1837825530 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd87c129350 0x7fd87c129780 secure :-1 s=THROTTLE_DONE pgs=3209 cs=0 l=1 rx=0x7fd87c064cd0 tx=0x7fd86c011f80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:07.966 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.940+0000 7fd873fff700 1 -- 172.21.15.42:0/1837825530 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fd86c0057e0 con 0x7fd87c129350 2019-12-05T06:48:07.966 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.940+0000 7fd87affd700 1 --2- 172.21.15.42:0/1837825530 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd87c129350 0x7fd87c129780 secure :-1 s=THROTTLE_DONE pgs=3209 cs=0 l=1 rx=0x7fd87c064cd0 tx=0x7fd86c011f80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:07.966 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.940+0000 7fd873fff700 1 --2- 172.21.15.42:0/1837825530 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd86003ac90 0x7fd86003d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:48:07.966 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.940+0000 7fd873fff700 1 -- 172.21.15.42:0/1837825530 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 12051+0+0 (secure 0 0 0) 0x7fd86c04b550 con 0x7fd87c129350 2019-12-05T06:48:07.967 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.940+0000 7fd87b7fe700 1 --2- 172.21.15.42:0/1837825530 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd86003ac90 0x7fd86003d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:07.967 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.940+0000 7fd87b7fe700 1 --2- 172.21.15.42:0/1837825530 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd86003ac90 0x7fd86003d1a0 crc :-1 s=READY pgs=1346 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:48:07.968 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.948+0000 7fd8813da700 1 -- 172.21.15.42:0/1837825530 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd87c05d110 con 0x7fd87c129350 2019-12-05T06:48:07.969 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.956+0000 7fd87affd700 1 --2- 172.21.15.42:0/1837825530 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd87c129350 0x7fd87c129780 secure :-1 s=THROTTLE_DONE pgs=3209 cs=0 l=1 rx=0x7fd87c064cd0 tx=0x7fd86c011f80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:07.969 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:07.956+0000 7fd873fff700 1 -- 172.21.15.42:0/1837825530 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fd86c0325b0 con 0x7fd87c129350 2019-12-05T06:48:08.132 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:08.128+0000 7fd8813da700 1 -- 172.21.15.42:0/1837825530 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd87c05b8c0 con 0x7fd87c129350 2019-12-05T06:48:08.139 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:08.128+0000 7fd87affd700 1 --2- 172.21.15.42:0/1837825530 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd87c129350 0x7fd87c129780 secure :-1 s=THROTTLE_DONE pgs=3209 cs=0 l=1 rx=0x7fd87c064cd0 tx=0x7fd86c011f80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:08.139 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:08.128+0000 7fd873fff700 1 -- 172.21.15.42:0/1837825530 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 389 v389) v1 ==== 95+0+2117 (secure 0 0 0) 0x7fd86c032810 con 0x7fd87c129350 2019-12-05T06:48:08.140 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 389 2019-12-05T06:48:08.140 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:08.128+0000 7fd8813da700 1 -- 172.21.15.42:0/1837825530 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd86003ac90 msgr2=0x7fd86003d1a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:08.140 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:08.128+0000 7fd8813da700 1 --2- 172.21.15.42:0/1837825530 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd86003ac90 0x7fd86003d1a0 crc :-1 s=READY pgs=1346 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:08.140 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:08.128+0000 7fd8813da700 1 -- 172.21.15.42:0/1837825530 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd87c129350 msgr2=0x7fd87c129780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:08.141 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:08.128+0000 7fd8813da700 1 --2- 172.21.15.42:0/1837825530 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd87c129350 0x7fd87c129780 secure :-1 s=READY pgs=3209 cs=0 l=1 rx=0x7fd87c064cd0 tx=0x7fd86c011f80).stop 2019-12-05T06:48:08.141 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:08.128+0000 7fd8813da700 1 -- 172.21.15.42:0/1837825530 shutdown_connections 2019-12-05T06:48:08.141 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:08.128+0000 7fd8813da700 1 --2- 172.21.15.42:0/1837825530 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd86003ac90 0x7fd86003d1a0 unknown :-1 s=CLOSED pgs=1346 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:08.141 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:08.128+0000 7fd8813da700 1 --2- 172.21.15.42:0/1837825530 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd87c08f080 0x7fd87c142de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:08.142 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:08.128+0000 7fd8813da700 1 --2- 172.21.15.42:0/1837825530 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd87c129350 0x7fd87c129780 unknown :-1 s=CLOSED pgs=3209 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:08.142 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:08.128+0000 7fd8813da700 1 -- 172.21.15.42:0/1837825530 shutdown_connections 2019-12-05T06:48:08.142 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:08.128+0000 7fd8813da700 1 -- 172.21.15.42:0/1837825530 wait complete. 2019-12-05T06:48:08.142 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:08.128+0000 7fd8813da700 1 -- 172.21.15.42:0/1837825530 >> 172.21.15.42:0/1837825530 conn(0x7fd87c0a7740 msgr2=0x7fd87c141360 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:48:08.143 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:48:08.143 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":389,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":12847,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/1449317789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1449317789},{"type":"v1","addr":"172.21.15.42:6835","nonce":1449317789}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":389}],"filesystems":[{"mdsmap":{"epoch":388,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:46:05.602252+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":119,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12429},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12429":{"gid":12429,"name":"b","rank":0,"incarnation":384,"state":"up:active","state_seq":26,"addr":"172.21.15.40:6833/4178556019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":4178556019},{"type":"v1","addr":"172.21.15.40:6833","nonce":4178556019}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:48:08.147 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:48:10.149 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:48:10.260 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:10.248+0000 7f5845b92700 1 Processor -- start 2019-12-05T06:48:10.260 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:10.248+0000 7f5845b92700 1 -- start start 2019-12-05T06:48:10.260 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:10.252+0000 7f5845b92700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f584008df80 0x7f584009f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:48:10.261 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:10.252+0000 7f5845b92700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f584009fc60 con 0x7f58400a7740 2019-12-05T06:48:10.261 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:10.252+0000 7f5845b92700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58400a0710 con 0x7f584009f830 2019-12-05T06:48:10.261 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:10.252+0000 7f5845b92700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58400a0b00 con 0x7f584008df80 2019-12-05T06:48:10.261 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:10.252+0000 7f583f7fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f584008df80 0x7f584009f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:10.261 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:10.252+0000 7f583f7fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f584008df80 0x7f584009f2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:37098/0 (socket says 172.21.15.42:37098) 2019-12-05T06:48:10.261 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:10.252+0000 7f583f7fe700 1 -- 172.21.15.42:0/2591153380 learned_addr learned my addr 172.21.15.42:0/2591153380 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:48:10.262 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:10.252+0000 7f5844930700 1 --1- 172.21.15.42:0/2591153380 >> v1:172.21.15.42:6789/0 conn(0x7f584009f830 0x7f584014efe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:37140/0 (socket says 172.21.15.42:37140) 2019-12-05T06:48:10.262 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:10.252+0000 7f583f7fe700 1 -- 172.21.15.42:0/2591153380 >> v1:172.21.15.42:6789/0 conn(0x7f584009f830 legacy=0x7f584014efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:10.262 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:10.252+0000 7f583f7fe700 1 -- 172.21.15.42:0/2591153380 >> v1:172.21.15.40:6789/0 conn(0x7f58400a7740 legacy=0x7f584007e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:48:10.262 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:10.252+0000 7f583f7fe700 1 -- 172.21.15.42:0/2591153380 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f584007ef20 con 0x7f584008df80 2019-12-05T06:48:10.262 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:10.252+0000 7f583f7fe700 1 --2- 172.21.15.42:0/2591153380 >> v2:172.21.15.42:3300/0 conn(0x7f584008df80 0x7f584009f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f5828008790 tx=0x7f582800fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:10.262 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:10.252+0000 7f583f7fe700 1 --2- 172.21.15.42:0/2591153380 >> v2:172.21.15.42:3300/0 conn(0x7f584008df80 0x7f584009f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f5828008790 tx=0x7f582800fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:10.262 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:10.252+0000 7f583f7fe700 1 --2- 172.21.15.42:0/2591153380 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f584008df80 0x7f584009f2e0 secure :-1 s=READY pgs=3211 cs=0 l=1 rx=0x7f5828008790 tx=0x7f582800fcb0).ready entity=mon.1 client_cookie=5c182fabff6430c8 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:48:10.263 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:10.252+0000 7f583f7fe700 1 --2- 172.21.15.42:0/2591153380 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f584008df80 0x7f584009f2e0 secure :-1 s=THROTTLE_DONE pgs=3211 cs=0 l=1 rx=0x7f5828008790 tx=0x7f582800fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:10.263 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:10.252+0000 7f583effd700 1 -- 172.21.15.42:0/2591153380 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5828011520 con 0x7f584008df80 2019-12-05T06:48:10.263 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:10.252+0000 7f583f7fe700 1 --2- 172.21.15.42:0/2591153380 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f584008df80 0x7f584009f2e0 secure :-1 s=THROTTLE_DONE pgs=3211 cs=0 l=1 rx=0x7f5828008790 tx=0x7f582800fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:10.263 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:10.252+0000 7f583effd700 1 -- 172.21.15.42:0/2591153380 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5828008ca0 con 0x7f584008df80 2019-12-05T06:48:10.263 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:10.252+0000 7f583f7fe700 1 --2- 172.21.15.42:0/2591153380 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f584008df80 0x7f584009f2e0 secure :-1 s=THROTTLE_DONE pgs=3211 cs=0 l=1 rx=0x7f5828008790 tx=0x7f582800fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:10.263 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:10.252+0000 7f583effd700 1 -- 172.21.15.42:0/2591153380 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5828011b00 con 0x7f584008df80 2019-12-05T06:48:10.264 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:10.252+0000 7f5845b92700 1 -- 172.21.15.42:0/2591153380 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f584008df80 msgr2=0x7f584009f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:10.264 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:10.252+0000 7f5845b92700 1 --2- 172.21.15.42:0/2591153380 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f584008df80 0x7f584009f2e0 secure :-1 s=READY pgs=3211 cs=0 l=1 rx=0x7f5828008790 tx=0x7f582800fcb0).stop 2019-12-05T06:48:10.264 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:10.252+0000 7f5845b92700 1 -- 172.21.15.42:0/2591153380 shutdown_connections 2019-12-05T06:48:10.264 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:10.252+0000 7f5845b92700 1 --2- 172.21.15.42:0/2591153380 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f584008df80 0x7f584009f2e0 unknown :-1 s=CLOSED pgs=3211 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:10.264 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:10.252+0000 7f5845b92700 1 -- 172.21.15.42:0/2591153380 shutdown_connections 2019-12-05T06:48:10.264 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:10.252+0000 7f5845b92700 1 -- 172.21.15.42:0/2591153380 wait complete. 2019-12-05T06:48:10.265 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:10.252+0000 7f5845b92700 1 -- 172.21.15.42:0/2591153380 >> 172.21.15.42:0/2591153380 conn(0x7f58400aabe0 msgr2=0x7f58400a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:48:10.269 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:10.256+0000 7f5845b92700 1 Processor -- start 2019-12-05T06:48:10.270 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:10.256+0000 7f5845b92700 1 -- start start 2019-12-05T06:48:10.270 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:10.256+0000 7f5845b92700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f58401462d0 0x7f5840064600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:48:10.270 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:10.256+0000 7f5845b92700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f584011f980 con 0x7f58400a3960 2019-12-05T06:48:10.270 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:10.256+0000 7f5845b92700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5840131e90 con 0x7f5840063680 2019-12-05T06:48:10.270 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:10.256+0000 7f5845b92700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5840064d30 con 0x7f58401462d0 2019-12-05T06:48:10.270 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:10.256+0000 7f5844930700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f5840063680 0x7f5840063ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:37146/0 (socket says 172.21.15.42:37146) 2019-12-05T06:48:10.271 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:10.256+0000 7f5844930700 1 -- 172.21.15.42:0/515376371 learned_addr learned my addr 172.21.15.42:0/515376371 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:48:10.271 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:10.256+0000 7f583f7fe700 1 --2- 172.21.15.42:0/515376371 >> v2:172.21.15.40:3300/0 conn(0x7f58401462d0 0x7f5840064600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:10.271 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:10.260+0000 7f583cff9700 1 -- 172.21.15.42:0/515376371 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3253864943 0 0) 0x7f5840131e90 con 0x7f5840063680 2019-12-05T06:48:10.271 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:10.260+0000 7f583cff9700 1 -- 172.21.15.42:0/515376371 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58240030b0 con 0x7f5840063680 2019-12-05T06:48:10.271 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:10.260+0000 7f583f7fe700 1 -- 172.21.15.42:0/515376371 >> v1:172.21.15.42:6789/0 conn(0x7f5840063680 legacy=0x7f5840063ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:10.271 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:10.260+0000 7f583cff9700 1 -- 172.21.15.42:0/515376371 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 1990171500 0 0) 0x7f58240030b0 con 0x7f5840063680 2019-12-05T06:48:10.272 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:10.260+0000 7f583f7fe700 1 -- 172.21.15.42:0/515376371 >> v1:172.21.15.40:6789/0 conn(0x7f58400a3960 legacy=0x7f584011f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:10.272 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:10.260+0000 7f583f7fe700 1 -- 172.21.15.42:0/515376371 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58401330d0 con 0x7f58401462d0 2019-12-05T06:48:10.272 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:10.260+0000 7f583f7fe700 1 --2- 172.21.15.42:0/515376371 >> v2:172.21.15.40:3300/0 conn(0x7f58401462d0 0x7f5840064600 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f5838001e40 tx=0x7f5838001950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:10.272 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:10.260+0000 7f583f7fe700 1 --2- 172.21.15.42:0/515376371 >> v2:172.21.15.40:3300/0 conn(0x7f58401462d0 0x7f5840064600 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f5838001e40 tx=0x7f5838001950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:10.272 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:10.260+0000 7f583f7fe700 1 --2- 172.21.15.42:0/515376371 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f58401462d0 0x7f5840064600 secure :-1 s=READY pgs=2442 cs=0 l=1 rx=0x7f5838001e40 tx=0x7f5838001950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:48:10.273 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:10.260+0000 7f583f7fe700 1 --2- 172.21.15.42:0/515376371 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f58401462d0 0x7f5840064600 secure :-1 s=THROTTLE_DONE pgs=2442 cs=0 l=1 rx=0x7f5838001e40 tx=0x7f5838001950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:10.273 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:10.260+0000 7f583cff9700 1 -- 172.21.15.42:0/515376371 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5838001c00 con 0x7f58401462d0 2019-12-05T06:48:10.273 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:10.260+0000 7f583f7fe700 1 --2- 172.21.15.42:0/515376371 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f58401462d0 0x7f5840064600 secure :-1 s=THROTTLE_DONE pgs=2442 cs=0 l=1 rx=0x7f5838001e40 tx=0x7f5838001950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:10.273 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:10.260+0000 7f583cff9700 1 -- 172.21.15.42:0/515376371 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5838001a10 con 0x7f58401462d0 2019-12-05T06:48:10.273 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:10.260+0000 7f583f7fe700 1 --2- 172.21.15.42:0/515376371 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f58401462d0 0x7f5840064600 secure :-1 s=THROTTLE_DONE pgs=2442 cs=0 l=1 rx=0x7f5838001e40 tx=0x7f5838001950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:10.273 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:10.260+0000 7f5845b92700 1 -- 172.21.15.42:0/515376371 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5840130e80 con 0x7f58401462d0 2019-12-05T06:48:10.274 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:10.260+0000 7f583cff9700 1 -- 172.21.15.42:0/515376371 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5828007340 con 0x7f58401462d0 2019-12-05T06:48:10.274 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:10.260+0000 7f5845b92700 1 -- 172.21.15.42:0/515376371 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58401313d0 con 0x7f58401462d0 2019-12-05T06:48:10.274 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:10.260+0000 7f583f7fe700 1 --2- 172.21.15.42:0/515376371 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f58401462d0 0x7f5840064600 secure :-1 s=THROTTLE_DONE pgs=2442 cs=0 l=1 rx=0x7f5838001e40 tx=0x7f5838001950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:10.274 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:10.260+0000 7f583cff9700 1 -- 172.21.15.42:0/515376371 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f5828005540 con 0x7f58401462d0 2019-12-05T06:48:10.274 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:10.260+0000 7f583f7fe700 1 --2- 172.21.15.42:0/515376371 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f58401462d0 0x7f5840064600 secure :-1 s=THROTTLE_DONE pgs=2442 cs=0 l=1 rx=0x7f5838001e40 tx=0x7f5838001950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:10.274 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:10.264+0000 7f583cff9700 1 --2- 172.21.15.42:0/515376371 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f582403b210 0x7f582403d720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:48:10.275 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:10.264+0000 7f583cff9700 1 -- 172.21.15.42:0/515376371 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 12051+0+0 (secure 0 0 0) 0x7f582804b160 con 0x7f58401462d0 2019-12-05T06:48:10.275 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:10.264+0000 7f583ffff700 1 --2- 172.21.15.42:0/515376371 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f582403b210 0x7f582403d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:10.275 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:10.264+0000 7f583ffff700 1 --2- 172.21.15.42:0/515376371 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f582403b210 0x7f582403d720 crc :-1 s=READY pgs=1347 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:48:10.276 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:10.264+0000 7f5845b92700 1 -- 172.21.15.42:0/515376371 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f584005d110 con 0x7f58401462d0 2019-12-05T06:48:10.282 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:10.276+0000 7f583f7fe700 1 --2- 172.21.15.42:0/515376371 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f58401462d0 0x7f5840064600 secure :-1 s=THROTTLE_DONE pgs=2442 cs=0 l=1 rx=0x7f5838001e40 tx=0x7f5838001950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:10.282 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:10.276+0000 7f583cff9700 1 -- 172.21.15.42:0/515376371 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f58280321c0 con 0x7f58401462d0 2019-12-05T06:48:10.452 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:10.448+0000 7f5845b92700 1 -- 172.21.15.42:0/515376371 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5840131c90 con 0x7f58401462d0 2019-12-05T06:48:10.453 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:10.448+0000 7f583f7fe700 1 --2- 172.21.15.42:0/515376371 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f58401462d0 0x7f5840064600 secure :-1 s=THROTTLE_DONE pgs=2442 cs=0 l=1 rx=0x7f5838001e40 tx=0x7f5838001950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:10.453 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:10.448+0000 7f583cff9700 1 -- 172.21.15.42:0/515376371 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 389 v389) v1 ==== 95+0+2117 (secure 0 0 0) 0x7f5828032420 con 0x7f58401462d0 2019-12-05T06:48:10.455 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 389 2019-12-05T06:48:10.455 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:48:10.456 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":389,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":12847,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/1449317789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1449317789},{"type":"v1","addr":"172.21.15.42:6835","nonce":1449317789}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":389}],"filesystems":[{"mdsmap":{"epoch":388,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:46:05.602252+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":119,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12429},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12429":{"gid":12429,"name":"b","rank":0,"incarnation":384,"state":"up:active","state_seq":26,"addr":"172.21.15.40:6833/4178556019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":4178556019},{"type":"v1","addr":"172.21.15.40:6833","nonce":4178556019}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:48:10.458 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:10.452+0000 7f5845b92700 1 -- 172.21.15.42:0/515376371 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f582403b210 msgr2=0x7f582403d720 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:10.458 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:10.452+0000 7f5845b92700 1 --2- 172.21.15.42:0/515376371 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f582403b210 0x7f582403d720 crc :-1 s=READY pgs=1347 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:10.458 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:10.452+0000 7f5845b92700 1 -- 172.21.15.42:0/515376371 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f58401462d0 msgr2=0x7f5840064600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:10.459 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:10.452+0000 7f5845b92700 1 --2- 172.21.15.42:0/515376371 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f58401462d0 0x7f5840064600 secure :-1 s=READY pgs=2442 cs=0 l=1 rx=0x7f5838001e40 tx=0x7f5838001950).stop 2019-12-05T06:48:10.463 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:10.452+0000 7f5845b92700 1 -- 172.21.15.42:0/515376371 shutdown_connections 2019-12-05T06:48:10.463 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:10.452+0000 7f5845b92700 1 --2- 172.21.15.42:0/515376371 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f582403b210 0x7f582403d720 unknown :-1 s=CLOSED pgs=1347 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:10.463 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:10.452+0000 7f5845b92700 1 --2- 172.21.15.42:0/515376371 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f58401462d0 0x7f5840064600 unknown :-1 s=CLOSED pgs=2442 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:10.463 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:10.452+0000 7f5845b92700 1 -- 172.21.15.42:0/515376371 shutdown_connections 2019-12-05T06:48:10.463 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:10.452+0000 7f5845b92700 1 -- 172.21.15.42:0/515376371 wait complete. 2019-12-05T06:48:10.464 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:10.452+0000 7f5845b92700 1 -- 172.21.15.42:0/515376371 >> 172.21.15.42:0/515376371 conn(0x7f584005a190 msgr2=0x7f58400a9770 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:48:10.471 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:48:12.472 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:48:12.579 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:12.564+0000 7fd9a568e700 1 Processor -- start 2019-12-05T06:48:12.579 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:12.568+0000 7fd9a568e700 1 -- start start 2019-12-05T06:48:12.579 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:12.568+0000 7fd9a568e700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd9a00a7740 0x7fd9a008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:48:12.580 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:12.568+0000 7fd9a568e700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd9a009fae0 0x7fd9a007eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:48:12.580 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:12.568+0000 7fd9a568e700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9a00a0710 con 0x7fd9a007e720 2019-12-05T06:48:12.580 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:12.568+0000 7fd9a568e700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9a00a0b00 con 0x7fd9a009fae0 2019-12-05T06:48:12.580 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:12.568+0000 7fd9a568e700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9a008e5b0 con 0x7fd9a00a7740 2019-12-05T06:48:12.580 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:12.568+0000 7fd99effd700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fd9a007e720 0x7fd9a009f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:37158/0 (socket says 172.21.15.42:37158) 2019-12-05T06:48:12.580 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:12.568+0000 7fd99effd700 1 -- 172.21.15.42:0/1227332116 learned_addr learned my addr 172.21.15.42:0/1227332116 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:48:12.581 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:12.568+0000 7fd99f7fe700 1 --2- 172.21.15.42:0/1227332116 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd9a00a7740 0x7fd9a008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:12.581 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:12.568+0000 7fd99ffff700 1 --2- 172.21.15.42:0/1227332116 >> v2:172.21.15.40:3300/0 conn(0x7fd9a009fae0 0x7fd9a007eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:12.581 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:12.568+0000 7fd99e7fc700 1 -- 172.21.15.42:0/1227332116 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1123730538 0 0) 0x7fd9a00a0710 con 0x7fd9a007e720 2019-12-05T06:48:12.581 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:12.568+0000 7fd99e7fc700 1 -- 172.21.15.42:0/1227332116 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9880030b0 con 0x7fd9a007e720 2019-12-05T06:48:12.581 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:12.568+0000 7fd99f7fe700 1 -- 172.21.15.42:0/1227332116 >> v2:172.21.15.40:3300/0 conn(0x7fd9a009fae0 msgr2=0x7fd9a007eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:48:12.581 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:12.568+0000 7fd99f7fe700 1 --2- 172.21.15.42:0/1227332116 >> v2:172.21.15.40:3300/0 conn(0x7fd9a009fae0 0x7fd9a007eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:48:12.582 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:12.568+0000 7fd99f7fe700 1 -- 172.21.15.42:0/1227332116 >> v1:172.21.15.42:6789/0 conn(0x7fd9a007e720 legacy=0x7fd9a009f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:12.582 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:12.568+0000 7fd99f7fe700 1 -- 172.21.15.42:0/1227332116 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9a007f0a0 con 0x7fd9a00a7740 2019-12-05T06:48:12.582 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:12.568+0000 7fd99f7fe700 1 --2- 172.21.15.42:0/1227332116 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd9a00a7740 0x7fd9a008df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fd998008b10 tx=0x7fd99800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:12.583 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:12.568+0000 7fd99f7fe700 1 --2- 172.21.15.42:0/1227332116 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd9a00a7740 0x7fd9a008df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fd998008b10 tx=0x7fd99800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:12.583 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:12.568+0000 7fd99f7fe700 1 --2- 172.21.15.42:0/1227332116 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd9a00a7740 0x7fd9a008df80 secure :-1 s=READY pgs=714 cs=0 l=1 rx=0x7fd998008b10 tx=0x7fd99800fff0).ready entity=mon.2 client_cookie=bae3b3ba8acd304d server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:48:12.583 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:12.568+0000 7fd99f7fe700 1 --2- 172.21.15.42:0/1227332116 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd9a00a7740 0x7fd9a008df80 secure :-1 s=THROTTLE_DONE pgs=714 cs=0 l=1 rx=0x7fd998008b10 tx=0x7fd99800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:12.583 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:12.568+0000 7fd99e7fc700 1 -- 172.21.15.42:0/1227332116 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd998010900 con 0x7fd9a00a7740 2019-12-05T06:48:12.584 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:12.568+0000 7fd99f7fe700 1 --2- 172.21.15.42:0/1227332116 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd9a00a7740 0x7fd9a008df80 secure :-1 s=THROTTLE_DONE pgs=714 cs=0 l=1 rx=0x7fd998008b10 tx=0x7fd99800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:12.584 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:12.568+0000 7fd99f7fe700 1 --2- 172.21.15.42:0/1227332116 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd9a00a7740 0x7fd9a008df80 secure :-1 s=THROTTLE_DONE pgs=714 cs=0 l=1 rx=0x7fd998008b10 tx=0x7fd99800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:12.585 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:12.568+0000 7fd99e7fc700 1 -- 172.21.15.42:0/1227332116 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd998008b50 con 0x7fd9a00a7740 2019-12-05T06:48:12.585 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:12.568+0000 7fd99e7fc700 1 -- 172.21.15.42:0/1227332116 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd9980110c0 con 0x7fd9a00a7740 2019-12-05T06:48:12.585 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:12.568+0000 7fd9a568e700 1 -- 172.21.15.42:0/1227332116 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd9a00a7740 msgr2=0x7fd9a008df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:12.585 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:12.568+0000 7fd9a568e700 1 --2- 172.21.15.42:0/1227332116 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd9a00a7740 0x7fd9a008df80 secure :-1 s=READY pgs=714 cs=0 l=1 rx=0x7fd998008b10 tx=0x7fd99800fff0).stop 2019-12-05T06:48:12.585 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:12.572+0000 7fd9a568e700 1 -- 172.21.15.42:0/1227332116 shutdown_connections 2019-12-05T06:48:12.585 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:12.572+0000 7fd9a568e700 1 --2- 172.21.15.42:0/1227332116 >> v2:172.21.15.40:3300/0 conn(0x7fd9a009fae0 0x7fd9a007eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:48:12.586 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:12.572+0000 7fd9a568e700 1 --2- 172.21.15.42:0/1227332116 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd9a00a7740 0x7fd9a008df80 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:12.586 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:12.572+0000 7fd9a568e700 1 -- 172.21.15.42:0/1227332116 shutdown_connections 2019-12-05T06:48:12.586 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:12.572+0000 7fd9a568e700 1 -- 172.21.15.42:0/1227332116 wait complete. 2019-12-05T06:48:12.586 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:12.572+0000 7fd9a568e700 1 -- 172.21.15.42:0/1227332116 >> 172.21.15.42:0/1227332116 conn(0x7fd9a00aabe0 msgr2=0x7fd9a00a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:48:12.592 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:12.572+0000 7fd9a568e700 1 Processor -- start 2019-12-05T06:48:12.592 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:12.572+0000 7fd9a568e700 1 -- start start 2019-12-05T06:48:12.592 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:12.576+0000 7fd9a568e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd9a0121fe0 0x7fd9a008f500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:48:12.592 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:12.576+0000 7fd9a568e700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd9a011b520 0x7fd9a011b950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:48:12.593 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:12.576+0000 7fd9a568e700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9a008fa50 con 0x7fd9a0141a40 2019-12-05T06:48:12.593 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:12.576+0000 7fd9a568e700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9a00a38d0 con 0x7fd9a011b520 2019-12-05T06:48:12.593 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:12.576+0000 7fd9a568e700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9a00a3b70 con 0x7fd9a0121fe0 2019-12-05T06:48:12.593 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:12.576+0000 7fd99f7fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd9a0121fe0 0x7fd9a008f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:12.593 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:12.576+0000 7fd99f7fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd9a0121fe0 0x7fd9a008f500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:37124/0 (socket says 172.21.15.42:37124) 2019-12-05T06:48:12.594 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:12.576+0000 7fd99f7fe700 1 -- 172.21.15.42:0/1357099263 learned_addr learned my addr 172.21.15.42:0/1357099263 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:48:12.594 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:12.576+0000 7fd99ffff700 1 --2- 172.21.15.42:0/1357099263 >> v2:172.21.15.40:3300/0 conn(0x7fd9a011b520 0x7fd9a011b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:12.594 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:12.576+0000 7fd99ffff700 1 -- 172.21.15.42:0/1357099263 >> v2:172.21.15.42:3300/0 conn(0x7fd9a0121fe0 msgr2=0x7fd9a008f500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:12.594 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:12.576+0000 7fd99ffff700 1 --2- 172.21.15.42:0/1357099263 >> v2:172.21.15.42:3300/0 conn(0x7fd9a0121fe0 0x7fd9a008f500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:12.595 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:12.576+0000 7fd99ffff700 1 -- 172.21.15.42:0/1357099263 >> v1:172.21.15.42:6789/0 conn(0x7fd9a0141a40 legacy=0x7fd9a01233b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:12.595 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:12.576+0000 7fd99ffff700 1 -- 172.21.15.42:0/1357099263 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9a0121130 con 0x7fd9a011b520 2019-12-05T06:48:12.595 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:12.576+0000 7fd99f7fe700 1 --2- 172.21.15.42:0/1357099263 >> v2:172.21.15.42:3300/0 conn(0x7fd9a0121fe0 0x7fd9a008f500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:48:12.595 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:12.576+0000 7fd99ffff700 1 --2- 172.21.15.42:0/1357099263 >> v2:172.21.15.40:3300/0 conn(0x7fd9a011b520 0x7fd9a011b950 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fd9a01232d0 tx=0x7fd990010980).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:12.596 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:12.576+0000 7fd99ffff700 1 --2- 172.21.15.42:0/1357099263 >> v2:172.21.15.40:3300/0 conn(0x7fd9a011b520 0x7fd9a011b950 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fd9a01232d0 tx=0x7fd990010980).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:12.596 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:12.576+0000 7fd99ffff700 1 --2- 172.21.15.42:0/1357099263 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd9a011b520 0x7fd9a011b950 secure :-1 s=READY pgs=2443 cs=0 l=1 rx=0x7fd9a01232d0 tx=0x7fd990010980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:48:12.596 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:12.576+0000 7fd99ffff700 1 --2- 172.21.15.42:0/1357099263 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd9a011b520 0x7fd9a011b950 secure :-1 s=THROTTLE_DONE pgs=2443 cs=0 l=1 rx=0x7fd9a01232d0 tx=0x7fd990010980).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:12.596 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:12.576+0000 7fd99ffff700 1 --2- 172.21.15.42:0/1357099263 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd9a011b520 0x7fd9a011b950 secure :-1 s=THROTTLE_DONE pgs=2443 cs=0 l=1 rx=0x7fd9a01232d0 tx=0x7fd990010980).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:12.596 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:12.576+0000 7fd987fff700 1 -- 172.21.15.42:0/1357099263 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd990002690 con 0x7fd9a011b520 2019-12-05T06:48:12.597 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:12.576+0000 7fd99ffff700 1 --2- 172.21.15.42:0/1357099263 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd9a011b520 0x7fd9a011b950 secure :-1 s=THROTTLE_DONE pgs=2443 cs=0 l=1 rx=0x7fd9a01232d0 tx=0x7fd990010980).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:12.597 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:12.576+0000 7fd987fff700 1 -- 172.21.15.42:0/1357099263 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9900022b0 con 0x7fd9a011b520 2019-12-05T06:48:12.597 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:12.576+0000 7fd987fff700 1 -- 172.21.15.42:0/1357099263 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd990002c70 con 0x7fd9a011b520 2019-12-05T06:48:12.597 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:12.576+0000 7fd9a568e700 1 -- 172.21.15.42:0/1357099263 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9a0131180 con 0x7fd9a011b520 2019-12-05T06:48:12.597 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:12.576+0000 7fd9a568e700 1 -- 172.21.15.42:0/1357099263 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9a0141e70 con 0x7fd9a011b520 2019-12-05T06:48:12.598 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:12.580+0000 7fd99ffff700 1 --2- 172.21.15.42:0/1357099263 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd9a011b520 0x7fd9a011b950 secure :-1 s=THROTTLE_DONE pgs=2443 cs=0 l=1 rx=0x7fd9a01232d0 tx=0x7fd990010980).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:12.598 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:12.580+0000 7fd987fff700 1 -- 172.21.15.42:0/1357099263 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fd990007170 con 0x7fd9a011b520 2019-12-05T06:48:12.598 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:12.580+0000 7fd99ffff700 1 --2- 172.21.15.42:0/1357099263 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd9a011b520 0x7fd9a011b950 secure :-1 s=THROTTLE_DONE pgs=2443 cs=0 l=1 rx=0x7fd9a01232d0 tx=0x7fd990010980).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:12.598 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:12.580+0000 7fd987fff700 1 --2- 172.21.15.42:0/1357099263 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd98003ae40 0x7fd98003d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:48:12.598 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:12.580+0000 7fd987fff700 1 -- 172.21.15.42:0/1357099263 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 12051+0+0 (secure 0 0 0) 0x7fd99004ab50 con 0x7fd9a011b520 2019-12-05T06:48:12.598 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:12.580+0000 7fd99f7fe700 1 --2- 172.21.15.42:0/1357099263 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd98003ae40 0x7fd98003d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:12.599 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:12.580+0000 7fd99f7fe700 1 --2- 172.21.15.42:0/1357099263 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd98003ae40 0x7fd98003d350 crc :-1 s=READY pgs=1348 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:48:12.599 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:12.580+0000 7fd9a568e700 1 -- 172.21.15.42:0/1357099263 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9a005d110 con 0x7fd9a011b520 2019-12-05T06:48:12.599 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:12.592+0000 7fd99ffff700 1 --2- 172.21.15.42:0/1357099263 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd9a011b520 0x7fd9a011b950 secure :-1 s=THROTTLE_DONE pgs=2443 cs=0 l=1 rx=0x7fd9a01232d0 tx=0x7fd990010980).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:12.600 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:12.592+0000 7fd987fff700 1 -- 172.21.15.42:0/1357099263 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fd990031bd0 con 0x7fd9a011b520 2019-12-05T06:48:12.767 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:12.760+0000 7fd9a568e700 1 -- 172.21.15.42:0/1357099263 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd9a005b8c0 con 0x7fd9a011b520 2019-12-05T06:48:12.768 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:12.760+0000 7fd99ffff700 1 --2- 172.21.15.42:0/1357099263 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd9a011b520 0x7fd9a011b950 secure :-1 s=THROTTLE_DONE pgs=2443 cs=0 l=1 rx=0x7fd9a01232d0 tx=0x7fd990010980).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:12.768 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:12.764+0000 7fd987fff700 1 -- 172.21.15.42:0/1357099263 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 389 v389) v1 ==== 95+0+2117 (secure 0 0 0) 0x7fd990031e30 con 0x7fd9a011b520 2019-12-05T06:48:12.770 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 389 2019-12-05T06:48:12.770 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:48:12.771 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":389,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":12847,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/1449317789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1449317789},{"type":"v1","addr":"172.21.15.42:6835","nonce":1449317789}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":389}],"filesystems":[{"mdsmap":{"epoch":388,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:46:05.602252+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":119,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12429},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12429":{"gid":12429,"name":"b","rank":0,"incarnation":384,"state":"up:active","state_seq":26,"addr":"172.21.15.40:6833/4178556019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":4178556019},{"type":"v1","addr":"172.21.15.40:6833","nonce":4178556019}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:48:12.780 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:12.768+0000 7fd9a568e700 1 -- 172.21.15.42:0/1357099263 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd98003ae40 msgr2=0x7fd98003d350 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:12.780 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:12.768+0000 7fd9a568e700 1 --2- 172.21.15.42:0/1357099263 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd98003ae40 0x7fd98003d350 crc :-1 s=READY pgs=1348 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:12.780 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:12.768+0000 7fd9a568e700 1 -- 172.21.15.42:0/1357099263 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd9a011b520 msgr2=0x7fd9a011b950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:12.780 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:12.768+0000 7fd9a568e700 1 --2- 172.21.15.42:0/1357099263 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd9a011b520 0x7fd9a011b950 secure :-1 s=READY pgs=2443 cs=0 l=1 rx=0x7fd9a01232d0 tx=0x7fd990010980).stop 2019-12-05T06:48:12.781 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:12.768+0000 7fd9a568e700 1 -- 172.21.15.42:0/1357099263 shutdown_connections 2019-12-05T06:48:12.781 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:12.768+0000 7fd9a568e700 1 --2- 172.21.15.42:0/1357099263 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd98003ae40 0x7fd98003d350 unknown :-1 s=CLOSED pgs=1348 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:12.781 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:12.768+0000 7fd9a568e700 1 --2- 172.21.15.42:0/1357099263 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd9a011b520 0x7fd9a011b950 unknown :-1 s=CLOSED pgs=2443 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:12.781 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:12.768+0000 7fd9a568e700 1 --2- 172.21.15.42:0/1357099263 >> v2:172.21.15.42:3300/0 conn(0x7fd9a0121fe0 0x7fd9a008f500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:12.781 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:12.768+0000 7fd9a568e700 1 -- 172.21.15.42:0/1357099263 shutdown_connections 2019-12-05T06:48:12.782 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:12.768+0000 7fd9a568e700 1 -- 172.21.15.42:0/1357099263 wait complete. 2019-12-05T06:48:12.782 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:12.768+0000 7fd9a568e700 1 -- 172.21.15.42:0/1357099263 >> 172.21.15.42:0/1357099263 conn(0x7fd9a00a7740 msgr2=0x7fd9a00a9790 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:48:12.784 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:48:14.785 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:48:14.891 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:14.876+0000 7ff000f57700 1 Processor -- start 2019-12-05T06:48:14.891 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:14.876+0000 7ff000f57700 1 -- start start 2019-12-05T06:48:14.891 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:14.880+0000 7ff000f57700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7feffc0a7740 0x7feffc08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:48:14.891 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:14.880+0000 7ff000f57700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7feffc09fae0 0x7feffc07eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:48:14.892 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:14.880+0000 7ff000f57700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feffc0a0710 con 0x7feffc07e720 2019-12-05T06:48:14.892 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:14.880+0000 7ff000f57700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feffc0a0b00 con 0x7feffc09fae0 2019-12-05T06:48:14.892 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:14.880+0000 7ff000f57700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feffc08e5b0 con 0x7feffc0a7740 2019-12-05T06:48:14.892 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:14.880+0000 7feffa59c700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7feffc07e720 0x7feffc09f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:37178/0 (socket says 172.21.15.42:37178) 2019-12-05T06:48:14.892 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:14.880+0000 7feffa59c700 1 -- 172.21.15.42:0/2725319327 learned_addr learned my addr 172.21.15.42:0/2725319327 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:48:14.892 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:14.880+0000 7feffb59e700 1 --2- 172.21.15.42:0/2725319327 >> v2:172.21.15.42:3300/0 conn(0x7feffc09fae0 0x7feffc07eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:14.892 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:14.880+0000 7feffad9d700 1 --2- 172.21.15.42:0/2725319327 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7feffc0a7740 0x7feffc08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:14.893 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:14.880+0000 7feff9d9b700 1 -- 172.21.15.42:0/2725319327 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2437987733 0 0) 0x7feffc0a0710 con 0x7feffc07e720 2019-12-05T06:48:14.893 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:14.880+0000 7feff9d9b700 1 -- 172.21.15.42:0/2725319327 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fefe40030b0 con 0x7feffc07e720 2019-12-05T06:48:14.893 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:14.880+0000 7feffb59e700 1 -- 172.21.15.42:0/2725319327 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7feffc0a7740 msgr2=0x7feffc08df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:48:14.893 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:14.880+0000 7feffb59e700 1 --2- 172.21.15.42:0/2725319327 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7feffc0a7740 0x7feffc08df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:48:14.893 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:14.880+0000 7feffb59e700 1 -- 172.21.15.42:0/2725319327 >> v1:172.21.15.42:6789/0 conn(0x7feffc07e720 legacy=0x7feffc09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:14.893 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:14.880+0000 7feffb59e700 1 -- 172.21.15.42:0/2725319327 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feffc07f0a0 con 0x7feffc09fae0 2019-12-05T06:48:14.893 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:14.880+0000 7feffad9d700 1 --2- 172.21.15.42:0/2725319327 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7feffc0a7740 0x7feffc08df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:48:14.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:14.880+0000 7feffb59e700 1 --2- 172.21.15.42:0/2725319327 >> v2:172.21.15.42:3300/0 conn(0x7feffc09fae0 0x7feffc07eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7feff4008790 tx=0x7feff400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:14.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:14.880+0000 7feffb59e700 1 --2- 172.21.15.42:0/2725319327 >> v2:172.21.15.42:3300/0 conn(0x7feffc09fae0 0x7feffc07eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7feff4008790 tx=0x7feff400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:14.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:14.880+0000 7feffb59e700 1 --2- 172.21.15.42:0/2725319327 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feffc09fae0 0x7feffc07eb50 secure :-1 s=READY pgs=3216 cs=0 l=1 rx=0x7feff4008790 tx=0x7feff400fc70).ready entity=mon.1 client_cookie=7e1eefdac9b02feb server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:48:14.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:14.880+0000 7feffb59e700 1 --2- 172.21.15.42:0/2725319327 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feffc09fae0 0x7feffc07eb50 secure :-1 s=THROTTLE_DONE pgs=3216 cs=0 l=1 rx=0x7feff4008790 tx=0x7feff400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:14.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:14.880+0000 7feff9d9b700 1 -- 172.21.15.42:0/2725319327 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7feff4008bb0 con 0x7feffc09fae0 2019-12-05T06:48:14.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:14.880+0000 7feffb59e700 1 --2- 172.21.15.42:0/2725319327 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feffc09fae0 0x7feffc07eb50 secure :-1 s=THROTTLE_DONE pgs=3216 cs=0 l=1 rx=0x7feff4008790 tx=0x7feff400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:14.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:14.880+0000 7feffb59e700 1 --2- 172.21.15.42:0/2725319327 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feffc09fae0 0x7feffc07eb50 secure :-1 s=THROTTLE_DONE pgs=3216 cs=0 l=1 rx=0x7feff4008790 tx=0x7feff400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:14.895 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:14.880+0000 7feff9d9b700 1 -- 172.21.15.42:0/2725319327 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feff40087d0 con 0x7feffc09fae0 2019-12-05T06:48:14.895 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:14.880+0000 7feff9d9b700 1 -- 172.21.15.42:0/2725319327 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7feff4011860 con 0x7feffc09fae0 2019-12-05T06:48:14.895 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:14.880+0000 7ff000f57700 1 -- 172.21.15.42:0/2725319327 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feffc09fae0 msgr2=0x7feffc07eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:14.895 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:14.884+0000 7ff000f57700 1 --2- 172.21.15.42:0/2725319327 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feffc09fae0 0x7feffc07eb50 secure :-1 s=READY pgs=3216 cs=0 l=1 rx=0x7feff4008790 tx=0x7feff400fc70).stop 2019-12-05T06:48:14.895 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:14.884+0000 7ff000f57700 1 -- 172.21.15.42:0/2725319327 shutdown_connections 2019-12-05T06:48:14.895 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:14.884+0000 7ff000f57700 1 --2- 172.21.15.42:0/2725319327 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feffc09fae0 0x7feffc07eb50 unknown :-1 s=CLOSED pgs=3216 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:14.895 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:14.884+0000 7ff000f57700 1 --2- 172.21.15.42:0/2725319327 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7feffc0a7740 0x7feffc08df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:48:14.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:14.884+0000 7ff000f57700 1 -- 172.21.15.42:0/2725319327 shutdown_connections 2019-12-05T06:48:14.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:14.884+0000 7ff000f57700 1 -- 172.21.15.42:0/2725319327 wait complete. 2019-12-05T06:48:14.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:14.884+0000 7ff000f57700 1 -- 172.21.15.42:0/2725319327 >> 172.21.15.42:0/2725319327 conn(0x7feffc0aabe0 msgr2=0x7feffc0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:48:14.899 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:14.888+0000 7ff000f57700 1 Processor -- start 2019-12-05T06:48:14.899 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:14.888+0000 7ff000f57700 1 -- start start 2019-12-05T06:48:14.899 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:14.888+0000 7ff000f57700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7feffc117710 0x7feffc147fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:48:14.900 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:14.888+0000 7ff000f57700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7feffc07d7c0 0x7feffc07dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:48:14.900 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:14.888+0000 7ff000f57700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feffc07e140 con 0x7feffc11e0a0 2019-12-05T06:48:14.900 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:14.888+0000 7ff000f57700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feffc148930 con 0x7feffc117710 2019-12-05T06:48:14.900 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:14.888+0000 7ff000f57700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feffc11a440 con 0x7feffc07d7c0 2019-12-05T06:48:14.900 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:14.888+0000 7feffad9d700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7feffc11e0a0 0x7feffc11a980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:37182/0 (socket says 172.21.15.42:37182) 2019-12-05T06:48:14.900 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:14.888+0000 7feffad9d700 1 -- 172.21.15.42:0/1301660487 learned_addr learned my addr 172.21.15.42:0/1301660487 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:48:14.900 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:14.888+0000 7feffb59e700 1 --2- 172.21.15.42:0/1301660487 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7feffc07d7c0 0x7feffc07dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:14.901 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:14.888+0000 7feffa59c700 1 --2- 172.21.15.42:0/1301660487 >> v2:172.21.15.40:3300/0 conn(0x7feffc117710 0x7feffc147fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:14.901 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:14.888+0000 7fefeb7fe700 1 -- 172.21.15.42:0/1301660487 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2870783866 0 0) 0x7feffc07e140 con 0x7feffc11e0a0 2019-12-05T06:48:14.901 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:14.888+0000 7fefeb7fe700 1 -- 172.21.15.42:0/1301660487 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fefe00030b0 con 0x7feffc11e0a0 2019-12-05T06:48:14.901 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:14.888+0000 7fefeb7fe700 1 -- 172.21.15.42:0/1301660487 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 3501875716 0 0) 0x7fefe00030b0 con 0x7feffc11e0a0 2019-12-05T06:48:14.901 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:14.888+0000 7feffb59e700 1 -- 172.21.15.42:0/1301660487 >> v2:172.21.15.40:3300/0 conn(0x7feffc117710 msgr2=0x7feffc147fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:14.901 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:14.888+0000 7feffb59e700 1 --2- 172.21.15.42:0/1301660487 >> v2:172.21.15.40:3300/0 conn(0x7feffc117710 0x7feffc147fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:14.901 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:14.888+0000 7feffb59e700 1 -- 172.21.15.42:0/1301660487 >> v1:172.21.15.42:6789/0 conn(0x7feffc11e0a0 legacy=0x7feffc11a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:14.902 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:14.888+0000 7feffb59e700 1 -- 172.21.15.42:0/1301660487 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feffc11c1e0 con 0x7feffc07d7c0 2019-12-05T06:48:14.902 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:14.888+0000 7feffb59e700 1 --2- 172.21.15.42:0/1301660487 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7feffc07d7c0 0x7feffc07dbf0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fefec002a40 tx=0x7fefec007dd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:14.902 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:14.888+0000 7feffb59e700 1 --2- 172.21.15.42:0/1301660487 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7feffc07d7c0 0x7feffc07dbf0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fefec002a40 tx=0x7fefec007dd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:14.902 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:14.888+0000 7feffb59e700 1 --2- 172.21.15.42:0/1301660487 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7feffc07d7c0 0x7feffc07dbf0 secure :-1 s=READY pgs=715 cs=0 l=1 rx=0x7fefec002a40 tx=0x7fefec007dd0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:48:14.903 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:14.888+0000 7feffb59e700 1 --2- 172.21.15.42:0/1301660487 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7feffc07d7c0 0x7feffc07dbf0 secure :-1 s=THROTTLE_DONE pgs=715 cs=0 l=1 rx=0x7fefec002a40 tx=0x7fefec007dd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:14.903 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:14.892+0000 7fefeb7fe700 1 -- 172.21.15.42:0/1301660487 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fefec002840 con 0x7feffc07d7c0 2019-12-05T06:48:14.903 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:14.892+0000 7feffb59e700 1 --2- 172.21.15.42:0/1301660487 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7feffc07d7c0 0x7feffc07dbf0 secure :-1 s=THROTTLE_DONE pgs=715 cs=0 l=1 rx=0x7fefec002a40 tx=0x7fefec007dd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:14.903 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:14.892+0000 7fefeb7fe700 1 -- 172.21.15.42:0/1301660487 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefec002650 con 0x7feffc07d7c0 2019-12-05T06:48:14.903 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:14.892+0000 7feffb59e700 1 --2- 172.21.15.42:0/1301660487 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7feffc07d7c0 0x7feffc07dbf0 secure :-1 s=THROTTLE_DONE pgs=715 cs=0 l=1 rx=0x7fefec002a40 tx=0x7fefec007dd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:14.903 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:14.892+0000 7fefeb7fe700 1 -- 172.21.15.42:0/1301660487 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7feff4008980 con 0x7feffc07d7c0 2019-12-05T06:48:14.903 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:14.892+0000 7ff000f57700 1 -- 172.21.15.42:0/1301660487 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feffc148500 con 0x7feffc07d7c0 2019-12-05T06:48:14.904 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:14.892+0000 7ff000f57700 1 -- 172.21.15.42:0/1301660487 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feffc134760 con 0x7feffc07d7c0 2019-12-05T06:48:14.904 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:14.892+0000 7feffb59e700 1 --2- 172.21.15.42:0/1301660487 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7feffc07d7c0 0x7feffc07dbf0 secure :-1 s=THROTTLE_DONE pgs=715 cs=0 l=1 rx=0x7fefec002a40 tx=0x7fefec007dd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:14.904 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:14.892+0000 7fefeb7fe700 1 -- 172.21.15.42:0/1301660487 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7feff4005540 con 0x7feffc07d7c0 2019-12-05T06:48:14.904 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:14.892+0000 7feffb59e700 1 --2- 172.21.15.42:0/1301660487 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7feffc07d7c0 0x7feffc07dbf0 secure :-1 s=THROTTLE_DONE pgs=715 cs=0 l=1 rx=0x7fefec002a40 tx=0x7fefec007dd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:14.904 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:14.892+0000 7fefeb7fe700 1 --2- 172.21.15.42:0/1301660487 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fefe003b1b0 0x7fefe003d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:48:14.904 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:14.892+0000 7fefeb7fe700 1 -- 172.21.15.42:0/1301660487 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 12051+0+0 (secure 0 0 0) 0x7feff404ace0 con 0x7feffc07d7c0 2019-12-05T06:48:14.904 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:14.892+0000 7feffad9d700 1 --2- 172.21.15.42:0/1301660487 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fefe003b1b0 0x7fefe003d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:14.905 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:14.892+0000 7feffad9d700 1 --2- 172.21.15.42:0/1301660487 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fefe003b1b0 0x7fefe003d6c0 crc :-1 s=READY pgs=1349 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:48:14.905 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:14.892+0000 7ff000f57700 1 -- 172.21.15.42:0/1301660487 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feffc05d110 con 0x7feffc07d7c0 2019-12-05T06:48:14.908 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:14.904+0000 7feffb59e700 1 --2- 172.21.15.42:0/1301660487 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7feffc07d7c0 0x7feffc07dbf0 secure :-1 s=THROTTLE_DONE pgs=715 cs=0 l=1 rx=0x7fefec002a40 tx=0x7fefec007dd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:14.908 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:14.904+0000 7fefeb7fe700 1 -- 172.21.15.42:0/1301660487 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7feff40076c0 con 0x7feffc07d7c0 2019-12-05T06:48:15.078 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:15.068+0000 7ff000f57700 1 -- 172.21.15.42:0/1301660487 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7feffc05b8c0 con 0x7feffc07d7c0 2019-12-05T06:48:15.078 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:15.068+0000 7feffb59e700 1 --2- 172.21.15.42:0/1301660487 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7feffc07d7c0 0x7feffc07dbf0 secure :-1 s=THROTTLE_DONE pgs=715 cs=0 l=1 rx=0x7fefec002a40 tx=0x7fefec007dd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:15.079 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:15.068+0000 7fefeb7fe700 1 -- 172.21.15.42:0/1301660487 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 389 v389) v1 ==== 95+0+2117 (secure 0 0 0) 0x7feff4007f10 con 0x7feffc07d7c0 2019-12-05T06:48:15.080 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 389 2019-12-05T06:48:15.080 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:48:15.081 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":389,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":12847,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/1449317789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1449317789},{"type":"v1","addr":"172.21.15.42:6835","nonce":1449317789}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":389}],"filesystems":[{"mdsmap":{"epoch":388,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:46:05.602252+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":119,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12429},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12429":{"gid":12429,"name":"b","rank":0,"incarnation":384,"state":"up:active","state_seq":26,"addr":"172.21.15.40:6833/4178556019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":4178556019},{"type":"v1","addr":"172.21.15.40:6833","nonce":4178556019}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:48:15.083 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:15.072+0000 7ff000f57700 1 -- 172.21.15.42:0/1301660487 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fefe003b1b0 msgr2=0x7fefe003d6c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:15.083 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:15.072+0000 7ff000f57700 1 --2- 172.21.15.42:0/1301660487 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fefe003b1b0 0x7fefe003d6c0 crc :-1 s=READY pgs=1349 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:15.083 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:15.072+0000 7ff000f57700 1 -- 172.21.15.42:0/1301660487 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7feffc07d7c0 msgr2=0x7feffc07dbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:15.084 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:15.072+0000 7ff000f57700 1 --2- 172.21.15.42:0/1301660487 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7feffc07d7c0 0x7feffc07dbf0 secure :-1 s=READY pgs=715 cs=0 l=1 rx=0x7fefec002a40 tx=0x7fefec007dd0).stop 2019-12-05T06:48:15.084 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:15.076+0000 7ff000f57700 1 -- 172.21.15.42:0/1301660487 shutdown_connections 2019-12-05T06:48:15.084 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:15.076+0000 7ff000f57700 1 --2- 172.21.15.42:0/1301660487 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fefe003b1b0 0x7fefe003d6c0 unknown :-1 s=CLOSED pgs=1349 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:15.084 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:15.076+0000 7ff000f57700 1 --2- 172.21.15.42:0/1301660487 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7feffc07d7c0 0x7feffc07dbf0 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:15.084 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:15.076+0000 7ff000f57700 1 --2- 172.21.15.42:0/1301660487 >> v2:172.21.15.40:3300/0 conn(0x7feffc117710 0x7feffc147fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:15.085 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:15.076+0000 7ff000f57700 1 -- 172.21.15.42:0/1301660487 shutdown_connections 2019-12-05T06:48:15.085 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:15.076+0000 7ff000f57700 1 -- 172.21.15.42:0/1301660487 wait complete. 2019-12-05T06:48:15.085 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:15.076+0000 7ff000f57700 1 -- 172.21.15.42:0/1301660487 >> 172.21.15.42:0/1301660487 conn(0x7feffc0a7f90 msgr2=0x7feffc0a9790 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:48:15.090 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:48:17.096 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:48:17.206 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:17.192+0000 7fee97a53700 1 Processor -- start 2019-12-05T06:48:17.206 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:17.192+0000 7fee97a53700 1 -- start start 2019-12-05T06:48:17.206 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:17.196+0000 7fee97a53700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fee900a7740 0x7fee9008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:48:17.206 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:17.196+0000 7fee97a53700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fee9007e720 0x7fee9007eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:48:17.206 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:17.196+0000 7fee97a53700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fee9009f2e0 0x7fee9009f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:48:17.207 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:17.196+0000 7fee97a53700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee900a0b00 con 0x7fee9009f2e0 2019-12-05T06:48:17.207 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:17.196+0000 7fee97a53700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee9008e5b0 con 0x7fee900a7740 2019-12-05T06:48:17.207 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:17.196+0000 7fee97a53700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee9008e850 con 0x7fee9007e720 2019-12-05T06:48:17.207 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:17.196+0000 7fee95ff0700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fee900a7740 0x7fee9008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:17.207 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:17.196+0000 7fee95ff0700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fee900a7740 0x7fee9008df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:37152/0 (socket says 172.21.15.42:37152) 2019-12-05T06:48:17.207 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:17.196+0000 7fee95ff0700 1 -- 172.21.15.42:0/2076638344 learned_addr learned my addr 172.21.15.42:0/2076638344 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:48:17.207 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:17.196+0000 7fee957ef700 1 --2- 172.21.15.42:0/2076638344 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fee9007e720 0x7fee9007eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:17.208 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:17.196+0000 7fee967f1700 1 --2- 172.21.15.42:0/2076638344 >> v2:172.21.15.40:3300/0 conn(0x7fee9009f2e0 0x7fee9009f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:17.208 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:17.196+0000 7fee95ff0700 1 -- 172.21.15.42:0/2076638344 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fee9007e720 msgr2=0x7fee9007eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:48:17.208 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:17.196+0000 7fee95ff0700 1 --2- 172.21.15.42:0/2076638344 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fee9007e720 0x7fee9007eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:48:17.208 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:17.196+0000 7fee95ff0700 1 -- 172.21.15.42:0/2076638344 >> v2:172.21.15.40:3300/0 conn(0x7fee9009f2e0 msgr2=0x7fee9009f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:48:17.208 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:17.196+0000 7fee95ff0700 1 --2- 172.21.15.42:0/2076638344 >> v2:172.21.15.40:3300/0 conn(0x7fee9009f2e0 0x7fee9009f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:48:17.208 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:17.196+0000 7fee95ff0700 1 -- 172.21.15.42:0/2076638344 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee9007f0a0 con 0x7fee900a7740 2019-12-05T06:48:17.208 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:17.196+0000 7fee95ff0700 1 --2- 172.21.15.42:0/2076638344 >> v2:172.21.15.42:3300/0 conn(0x7fee900a7740 0x7fee9008df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fee9008e770 tx=0x7fee8c010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:17.209 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:17.196+0000 7fee95ff0700 1 --2- 172.21.15.42:0/2076638344 >> v2:172.21.15.42:3300/0 conn(0x7fee900a7740 0x7fee9008df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fee9008e770 tx=0x7fee8c010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:17.209 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:17.196+0000 7fee95ff0700 1 --2- 172.21.15.42:0/2076638344 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fee900a7740 0x7fee9008df80 secure :-1 s=READY pgs=3218 cs=0 l=1 rx=0x7fee9008e770 tx=0x7fee8c010130).ready entity=mon.1 client_cookie=1928ffdbc0c7e50e server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:48:17.209 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:17.196+0000 7fee95ff0700 1 --2- 172.21.15.42:0/2076638344 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fee900a7740 0x7fee9008df80 secure :-1 s=THROTTLE_DONE pgs=3218 cs=0 l=1 rx=0x7fee9008e770 tx=0x7fee8c010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:17.209 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:17.196+0000 7fee94fee700 1 -- 172.21.15.42:0/2076638344 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fee8c010de0 con 0x7fee900a7740 2019-12-05T06:48:17.209 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:17.196+0000 7fee95ff0700 1 --2- 172.21.15.42:0/2076638344 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fee900a7740 0x7fee9008df80 secure :-1 s=THROTTLE_DONE pgs=3218 cs=0 l=1 rx=0x7fee9008e770 tx=0x7fee8c010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:17.210 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:17.196+0000 7fee95ff0700 1 --2- 172.21.15.42:0/2076638344 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fee900a7740 0x7fee9008df80 secure :-1 s=THROTTLE_DONE pgs=3218 cs=0 l=1 rx=0x7fee9008e770 tx=0x7fee8c010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:17.210 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:17.196+0000 7fee94fee700 1 -- 172.21.15.42:0/2076638344 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee8c010a00 con 0x7fee900a7740 2019-12-05T06:48:17.210 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:17.196+0000 7fee94fee700 1 -- 172.21.15.42:0/2076638344 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fee8c0114a0 con 0x7fee900a7740 2019-12-05T06:48:17.210 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:17.196+0000 7fee97a53700 1 -- 172.21.15.42:0/2076638344 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fee900a7740 msgr2=0x7fee9008df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:17.210 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:17.196+0000 7fee97a53700 1 --2- 172.21.15.42:0/2076638344 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fee900a7740 0x7fee9008df80 secure :-1 s=READY pgs=3218 cs=0 l=1 rx=0x7fee9008e770 tx=0x7fee8c010130).stop 2019-12-05T06:48:17.210 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:17.196+0000 7fee97a53700 1 -- 172.21.15.42:0/2076638344 shutdown_connections 2019-12-05T06:48:17.211 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:17.196+0000 7fee97a53700 1 --2- 172.21.15.42:0/2076638344 >> v2:172.21.15.40:3300/0 conn(0x7fee9009f2e0 0x7fee9009f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:48:17.211 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:17.196+0000 7fee97a53700 1 --2- 172.21.15.42:0/2076638344 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fee900a7740 0x7fee9008df80 unknown :-1 s=CLOSED pgs=3218 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:17.211 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:17.196+0000 7fee97a53700 1 --2- 172.21.15.42:0/2076638344 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fee9007e720 0x7fee9007eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:48:17.211 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:17.196+0000 7fee97a53700 1 -- 172.21.15.42:0/2076638344 shutdown_connections 2019-12-05T06:48:17.212 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:17.196+0000 7fee97a53700 1 -- 172.21.15.42:0/2076638344 wait complete. 2019-12-05T06:48:17.212 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:17.196+0000 7fee97a53700 1 -- 172.21.15.42:0/2076638344 >> 172.21.15.42:0/2076638344 conn(0x7fee900aabe0 msgr2=0x7fee900a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:48:17.212 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:17.200+0000 7fee97a53700 1 Processor -- start 2019-12-05T06:48:17.212 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:17.200+0000 7fee97a53700 1 -- start start 2019-12-05T06:48:17.216 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:17.200+0000 7fee97a53700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fee9008eeb0 0x7fee90146cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:48:17.216 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:17.200+0000 7fee97a53700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fee90130850 0x7fee90130c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:48:17.216 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:17.200+0000 7fee97a53700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee901311d0 con 0x7fee90145a70 2019-12-05T06:48:17.216 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:17.200+0000 7fee97a53700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee901474c0 con 0x7fee9008eeb0 2019-12-05T06:48:17.217 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:17.200+0000 7fee97a53700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee901305b0 con 0x7fee90130850 2019-12-05T06:48:17.217 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:17.200+0000 7fee957ef700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fee9008eeb0 0x7fee90146cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:17.217 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:17.200+0000 7fee957ef700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fee9008eeb0 0x7fee90146cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:37160/0 (socket says 172.21.15.42:37160) 2019-12-05T06:48:17.217 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:17.200+0000 7fee95ff0700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fee90145a70 0x7fee9007b850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:46636/0 (socket says 172.21.15.42:46636) 2019-12-05T06:48:17.217 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:17.200+0000 7fee957ef700 1 -- 172.21.15.42:0/1051127967 learned_addr learned my addr 172.21.15.42:0/1051127967 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:48:17.217 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:17.200+0000 7fee967f1700 1 --2- 172.21.15.42:0/1051127967 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fee90130850 0x7fee90130c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:17.217 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:17.200+0000 7fee867fc700 1 -- 172.21.15.42:0/1051127967 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4263205990 0 0) 0x7fee901311d0 con 0x7fee90145a70 2019-12-05T06:48:17.218 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:17.200+0000 7fee867fc700 1 -- 172.21.15.42:0/1051127967 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee780030b0 con 0x7fee90145a70 2019-12-05T06:48:17.218 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:17.204+0000 7fee957ef700 1 -- 172.21.15.42:0/1051127967 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fee90130850 msgr2=0x7fee90130c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:17.218 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:17.204+0000 7fee957ef700 1 --2- 172.21.15.42:0/1051127967 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fee90130850 0x7fee90130c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:17.218 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:17.204+0000 7fee957ef700 1 -- 172.21.15.42:0/1051127967 >> v1:172.21.15.40:6789/0 conn(0x7fee90145a70 legacy=0x7fee9007b850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:17.218 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:17.204+0000 7fee957ef700 1 -- 172.21.15.42:0/1051127967 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee9007c050 con 0x7fee9008eeb0 2019-12-05T06:48:17.219 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:17.204+0000 7fee957ef700 1 --2- 172.21.15.42:0/1051127967 >> v2:172.21.15.42:3300/0 conn(0x7fee9008eeb0 0x7fee90146cd0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fee90147680 tx=0x7fee80010bf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:17.219 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:17.204+0000 7fee957ef700 1 --2- 172.21.15.42:0/1051127967 >> v2:172.21.15.42:3300/0 conn(0x7fee9008eeb0 0x7fee90146cd0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fee90147680 tx=0x7fee80010bf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:17.219 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:17.204+0000 7fee957ef700 1 --2- 172.21.15.42:0/1051127967 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fee9008eeb0 0x7fee90146cd0 secure :-1 s=READY pgs=3219 cs=0 l=1 rx=0x7fee90147680 tx=0x7fee80010bf0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:48:17.219 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:17.204+0000 7fee957ef700 1 --2- 172.21.15.42:0/1051127967 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fee9008eeb0 0x7fee90146cd0 secure :-1 s=THROTTLE_DONE pgs=3219 cs=0 l=1 rx=0x7fee90147680 tx=0x7fee80010bf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:17.219 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:17.204+0000 7fee867fc700 1 -- 172.21.15.42:0/1051127967 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fee80005840 con 0x7fee9008eeb0 2019-12-05T06:48:17.219 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:17.204+0000 7fee957ef700 1 --2- 172.21.15.42:0/1051127967 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fee9008eeb0 0x7fee90146cd0 secure :-1 s=THROTTLE_DONE pgs=3219 cs=0 l=1 rx=0x7fee90147680 tx=0x7fee80010bf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:17.219 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:17.204+0000 7fee957ef700 1 --2- 172.21.15.42:0/1051127967 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fee9008eeb0 0x7fee90146cd0 secure :-1 s=THROTTLE_DONE pgs=3219 cs=0 l=1 rx=0x7fee90147680 tx=0x7fee80010bf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:17.220 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:17.204+0000 7fee867fc700 1 -- 172.21.15.42:0/1051127967 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee80007160 con 0x7fee9008eeb0 2019-12-05T06:48:17.220 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:17.204+0000 7fee97a53700 1 -- 172.21.15.42:0/1051127967 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee90137880 con 0x7fee9008eeb0 2019-12-05T06:48:17.220 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:17.204+0000 7fee867fc700 1 -- 172.21.15.42:0/1051127967 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fee80012230 con 0x7fee9008eeb0 2019-12-05T06:48:17.220 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:17.204+0000 7fee97a53700 1 -- 172.21.15.42:0/1051127967 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee90137e00 con 0x7fee9008eeb0 2019-12-05T06:48:17.220 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:17.204+0000 7fee957ef700 1 --2- 172.21.15.42:0/1051127967 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fee9008eeb0 0x7fee90146cd0 secure :-1 s=THROTTLE_DONE pgs=3219 cs=0 l=1 rx=0x7fee90147680 tx=0x7fee80010bf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:17.220 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:17.204+0000 7fee867fc700 1 -- 172.21.15.42:0/1051127967 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fee80012550 con 0x7fee9008eeb0 2019-12-05T06:48:17.220 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:17.204+0000 7fee957ef700 1 --2- 172.21.15.42:0/1051127967 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fee9008eeb0 0x7fee90146cd0 secure :-1 s=THROTTLE_DONE pgs=3219 cs=0 l=1 rx=0x7fee90147680 tx=0x7fee80010bf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:17.221 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:17.204+0000 7fee867fc700 1 --2- 172.21.15.42:0/1051127967 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fee7803b2b0 0x7fee7803d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:48:17.221 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:17.204+0000 7fee867fc700 1 -- 172.21.15.42:0/1051127967 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 12051+0+0 (secure 0 0 0) 0x7fee8004b1c0 con 0x7fee9008eeb0 2019-12-05T06:48:17.221 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:17.204+0000 7fee95ff0700 1 --2- 172.21.15.42:0/1051127967 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fee7803b2b0 0x7fee7803d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:17.221 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:17.208+0000 7fee95ff0700 1 --2- 172.21.15.42:0/1051127967 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fee7803b2b0 0x7fee7803d7c0 crc :-1 s=READY pgs=1350 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:48:17.221 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:17.208+0000 7fee97a53700 1 -- 172.21.15.42:0/1051127967 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee901476c0 con 0x7fee9008eeb0 2019-12-05T06:48:17.222 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:17.216+0000 7fee957ef700 1 --2- 172.21.15.42:0/1051127967 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fee9008eeb0 0x7fee90146cd0 secure :-1 s=THROTTLE_DONE pgs=3219 cs=0 l=1 rx=0x7fee90147680 tx=0x7fee80010bf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:17.222 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:17.216+0000 7fee867fc700 1 -- 172.21.15.42:0/1051127967 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fee800321c0 con 0x7fee9008eeb0 2019-12-05T06:48:17.394 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:17.388+0000 7fee97a53700 1 -- 172.21.15.42:0/1051127967 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fee9005d110 con 0x7fee9008eeb0 2019-12-05T06:48:17.394 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:17.388+0000 7fee957ef700 1 --2- 172.21.15.42:0/1051127967 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fee9008eeb0 0x7fee90146cd0 secure :-1 s=THROTTLE_DONE pgs=3219 cs=0 l=1 rx=0x7fee90147680 tx=0x7fee80010bf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:17.394 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:17.388+0000 7fee867fc700 1 -- 172.21.15.42:0/1051127967 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 389 v389) v1 ==== 95+0+2117 (secure 0 0 0) 0x7fee80032420 con 0x7fee9008eeb0 2019-12-05T06:48:17.394 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 389 2019-12-05T06:48:17.394 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:48:17.395 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":389,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":12847,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/1449317789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1449317789},{"type":"v1","addr":"172.21.15.42:6835","nonce":1449317789}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":389}],"filesystems":[{"mdsmap":{"epoch":388,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:46:05.602252+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":119,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12429},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12429":{"gid":12429,"name":"b","rank":0,"incarnation":384,"state":"up:active","state_seq":26,"addr":"172.21.15.40:6833/4178556019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":4178556019},{"type":"v1","addr":"172.21.15.40:6833","nonce":4178556019}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:48:17.398 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:17.388+0000 7fee97a53700 1 -- 172.21.15.42:0/1051127967 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fee7803b2b0 msgr2=0x7fee7803d7c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:17.398 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:17.388+0000 7fee97a53700 1 --2- 172.21.15.42:0/1051127967 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fee7803b2b0 0x7fee7803d7c0 crc :-1 s=READY pgs=1350 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:17.398 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:17.388+0000 7fee97a53700 1 -- 172.21.15.42:0/1051127967 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fee9008eeb0 msgr2=0x7fee90146cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:17.398 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:17.388+0000 7fee97a53700 1 --2- 172.21.15.42:0/1051127967 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fee9008eeb0 0x7fee90146cd0 secure :-1 s=READY pgs=3219 cs=0 l=1 rx=0x7fee90147680 tx=0x7fee80010bf0).stop 2019-12-05T06:48:17.398 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:17.388+0000 7fee97a53700 1 -- 172.21.15.42:0/1051127967 shutdown_connections 2019-12-05T06:48:17.398 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:17.388+0000 7fee97a53700 1 --2- 172.21.15.42:0/1051127967 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fee7803b2b0 0x7fee7803d7c0 unknown :-1 s=CLOSED pgs=1350 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:17.398 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:17.388+0000 7fee97a53700 1 --2- 172.21.15.42:0/1051127967 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fee90130850 0x7fee90130c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:17.399 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:17.388+0000 7fee97a53700 1 --2- 172.21.15.42:0/1051127967 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fee9008eeb0 0x7fee90146cd0 unknown :-1 s=CLOSED pgs=3219 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:17.399 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:17.392+0000 7fee97a53700 1 -- 172.21.15.42:0/1051127967 shutdown_connections 2019-12-05T06:48:17.399 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:17.392+0000 7fee97a53700 1 -- 172.21.15.42:0/1051127967 wait complete. 2019-12-05T06:48:17.399 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:17.392+0000 7fee97a53700 1 -- 172.21.15.42:0/1051127967 >> 172.21.15.42:0/1051127967 conn(0x7fee9007df60 msgr2=0x7fee9009f350 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:48:17.406 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:48:19.407 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:48:19.516 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:19.504+0000 7f75820f8700 1 Processor -- start 2019-12-05T06:48:19.516 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:19.504+0000 7f75820f8700 1 -- start start 2019-12-05T06:48:19.517 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:19.504+0000 7f75820f8700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f757c08df80 0x7f757c09f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:48:19.517 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:19.504+0000 7f75820f8700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f757c09f830 0x7f757c14efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:48:19.517 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:19.504+0000 7f75820f8700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f757c09fc60 con 0x7f757c0a7740 2019-12-05T06:48:19.517 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:19.504+0000 7f75820f8700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f757c0a0b00 con 0x7f757c08df80 2019-12-05T06:48:19.517 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:19.504+0000 7f75820f8700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f757c08e5b0 con 0x7f757c09f830 2019-12-05T06:48:19.518 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:19.504+0000 7f7580e96700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f757c09f830 0x7f757c14efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:19.518 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:19.504+0000 7f757b7fe700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f757c08df80 0x7f757c09f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:19.518 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:19.504+0000 7f757b7fe700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f757c08df80 0x7f757c09f2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.42:53216/0 (socket says 172.21.15.42:53216) 2019-12-05T06:48:19.518 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:19.504+0000 7f7580e96700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f757c09f830 0x7f757c14efe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:37636/0 (socket says 172.21.15.42:37636) 2019-12-05T06:48:19.518 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:19.504+0000 7f757b7fe700 1 -- 172.21.15.42:0/2711590773 learned_addr learned my addr 172.21.15.42:0/2711590773 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:48:19.519 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:19.504+0000 7f757bfff700 1 --1- 172.21.15.42:0/2711590773 >> v1:172.21.15.40:6789/0 conn(0x7f757c0a7740 0x7f757c07e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:46644/0 (socket says 172.21.15.42:46644) 2019-12-05T06:48:19.519 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:19.508+0000 7f7580e96700 1 -- 172.21.15.42:0/2711590773 >> v2:172.21.15.40:3300/0 conn(0x7f757c08df80 msgr2=0x7f757c09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:48:19.519 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:19.508+0000 7f7580e96700 1 --2- 172.21.15.42:0/2711590773 >> v2:172.21.15.40:3300/0 conn(0x7f757c08df80 0x7f757c09f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:48:19.519 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:19.508+0000 7f7580e96700 1 -- 172.21.15.42:0/2711590773 >> v1:172.21.15.40:6789/0 conn(0x7f757c0a7740 legacy=0x7f757c07e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:19.519 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:19.508+0000 7f7580e96700 1 -- 172.21.15.42:0/2711590773 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f757c0a0710 con 0x7f757c09f830 2019-12-05T06:48:19.519 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:19.508+0000 7f757b7fe700 1 --2- 172.21.15.42:0/2711590773 >> v2:172.21.15.40:3300/0 conn(0x7f757c08df80 0x7f757c09f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:48:19.520 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:19.508+0000 7f7580e96700 1 --2- 172.21.15.42:0/2711590773 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f757c09f830 0x7f757c14efe0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f7574008790 tx=0x7f757400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:19.520 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:19.508+0000 7f7580e96700 1 --2- 172.21.15.42:0/2711590773 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f757c09f830 0x7f757c14efe0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f7574008790 tx=0x7f757400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:19.520 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:19.508+0000 7f7580e96700 1 --2- 172.21.15.42:0/2711590773 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f757c09f830 0x7f757c14efe0 secure :-1 s=READY pgs=716 cs=0 l=1 rx=0x7f7574008790 tx=0x7f757400fc70).ready entity=mon.2 client_cookie=da4839060e11fe20 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:48:19.520 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:19.508+0000 7f7580e96700 1 --2- 172.21.15.42:0/2711590773 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f757c09f830 0x7f757c14efe0 secure :-1 s=THROTTLE_DONE pgs=716 cs=0 l=1 rx=0x7f7574008790 tx=0x7f757400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:19.520 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:19.508+0000 7f7580e96700 1 --2- 172.21.15.42:0/2711590773 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f757c09f830 0x7f757c14efe0 secure :-1 s=THROTTLE_DONE pgs=716 cs=0 l=1 rx=0x7f7574008790 tx=0x7f757400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:19.521 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:19.508+0000 7f757affd700 1 -- 172.21.15.42:0/2711590773 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7574008bb0 con 0x7f757c09f830 2019-12-05T06:48:19.521 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:19.508+0000 7f7580e96700 1 --2- 172.21.15.42:0/2711590773 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f757c09f830 0x7f757c14efe0 secure :-1 s=THROTTLE_DONE pgs=716 cs=0 l=1 rx=0x7f7574008790 tx=0x7f757400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:19.521 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:19.508+0000 7f757affd700 1 -- 172.21.15.42:0/2711590773 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75740087d0 con 0x7f757c09f830 2019-12-05T06:48:19.521 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:19.508+0000 7f757affd700 1 -- 172.21.15.42:0/2711590773 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7574011860 con 0x7f757c09f830 2019-12-05T06:48:19.521 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:19.508+0000 7f75820f8700 1 -- 172.21.15.42:0/2711590773 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f757c09f830 msgr2=0x7f757c14efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:19.522 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:19.508+0000 7f75820f8700 1 --2- 172.21.15.42:0/2711590773 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f757c09f830 0x7f757c14efe0 secure :-1 s=READY pgs=716 cs=0 l=1 rx=0x7f7574008790 tx=0x7f757400fc70).stop 2019-12-05T06:48:19.522 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:19.508+0000 7f75820f8700 1 -- 172.21.15.42:0/2711590773 shutdown_connections 2019-12-05T06:48:19.522 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:19.508+0000 7f75820f8700 1 --2- 172.21.15.42:0/2711590773 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f757c09f830 0x7f757c14efe0 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:19.522 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:19.508+0000 7f75820f8700 1 --2- 172.21.15.42:0/2711590773 >> v2:172.21.15.40:3300/0 conn(0x7f757c08df80 0x7f757c09f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:48:19.522 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:19.508+0000 7f75820f8700 1 -- 172.21.15.42:0/2711590773 shutdown_connections 2019-12-05T06:48:19.523 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:19.508+0000 7f75820f8700 1 -- 172.21.15.42:0/2711590773 wait complete. 2019-12-05T06:48:19.523 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:19.508+0000 7f75820f8700 1 -- 172.21.15.42:0/2711590773 >> 172.21.15.42:0/2711590773 conn(0x7f757c0aabe0 msgr2=0x7f757c0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:48:19.526 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:19.512+0000 7f75820f8700 1 Processor -- start 2019-12-05T06:48:19.527 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:19.512+0000 7f75820f8700 1 -- start start 2019-12-05T06:48:19.527 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:19.512+0000 7f75820f8700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f757c11c030 0x7f757c11b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:48:19.527 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:19.512+0000 7f75820f8700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f757c11a730 0x7f757c12d770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:48:19.527 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:19.512+0000 7f75820f8700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f757c07d730 con 0x7f757c07cb70 2019-12-05T06:48:19.527 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:19.512+0000 7f75820f8700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f757c12cf90 con 0x7f757c11a730 2019-12-05T06:48:19.528 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:19.512+0000 7f75820f8700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f757c12d230 con 0x7f757c11c030 2019-12-05T06:48:19.528 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:19.512+0000 7f757bfff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f757c11c030 0x7f757c11b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:19.528 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:19.512+0000 7f7580e96700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f757c07cb70 0x7f757c1304b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:37212/0 (socket says 172.21.15.42:37212) 2019-12-05T06:48:19.528 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:19.512+0000 7f7580e96700 1 -- 172.21.15.42:0/3064305066 learned_addr learned my addr 172.21.15.42:0/3064305066 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:48:19.528 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:19.512+0000 7f757b7fe700 1 --2- 172.21.15.42:0/3064305066 >> v2:172.21.15.40:3300/0 conn(0x7f757c11a730 0x7f757c12d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:19.528 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:19.512+0000 7f757bfff700 1 -- 172.21.15.42:0/3064305066 >> v2:172.21.15.40:3300/0 conn(0x7f757c11a730 msgr2=0x7f757c12d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:19.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:19.512+0000 7f757bfff700 1 --2- 172.21.15.42:0/3064305066 >> v2:172.21.15.40:3300/0 conn(0x7f757c11a730 0x7f757c12d770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:19.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:19.512+0000 7f7578ff9700 1 -- 172.21.15.42:0/3064305066 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3969914290 0 0) 0x7f757c07d730 con 0x7f757c07cb70 2019-12-05T06:48:19.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:19.512+0000 7f757bfff700 1 -- 172.21.15.42:0/3064305066 >> v1:172.21.15.42:6789/0 conn(0x7f757c07cb70 legacy=0x7f757c1304b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:19.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:19.512+0000 7f757bfff700 1 -- 172.21.15.42:0/3064305066 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f757c11b6c0 con 0x7f757c11c030 2019-12-05T06:48:19.530 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:19.512+0000 7f757b7fe700 1 --2- 172.21.15.42:0/3064305066 >> v2:172.21.15.40:3300/0 conn(0x7f757c11a730 0x7f757c12d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:48:19.530 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:19.512+0000 7f757bfff700 1 --2- 172.21.15.42:0/3064305066 >> v2:172.21.15.42:3300/0 conn(0x7f757c11c030 0x7f757c11b170 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f756c001210 tx=0x7f756c012ed0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:19.530 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:19.512+0000 7f757bfff700 1 --2- 172.21.15.42:0/3064305066 >> v2:172.21.15.42:3300/0 conn(0x7f757c11c030 0x7f757c11b170 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f756c001210 tx=0x7f756c012ed0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:19.530 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:19.512+0000 7f757bfff700 1 --2- 172.21.15.42:0/3064305066 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f757c11c030 0x7f757c11b170 secure :-1 s=READY pgs=3221 cs=0 l=1 rx=0x7f756c001210 tx=0x7f756c012ed0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:48:19.531 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:19.516+0000 7f757bfff700 1 --2- 172.21.15.42:0/3064305066 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f757c11c030 0x7f757c11b170 secure :-1 s=THROTTLE_DONE pgs=3221 cs=0 l=1 rx=0x7f756c001210 tx=0x7f756c012ed0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:19.531 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:19.516+0000 7f7578ff9700 1 -- 172.21.15.42:0/3064305066 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f756c014ca0 con 0x7f757c11c030 2019-12-05T06:48:19.531 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:19.516+0000 7f757bfff700 1 --2- 172.21.15.42:0/3064305066 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f757c11c030 0x7f757c11b170 secure :-1 s=THROTTLE_DONE pgs=3221 cs=0 l=1 rx=0x7f756c001210 tx=0x7f756c012ed0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:19.531 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:19.516+0000 7f757bfff700 1 --2- 172.21.15.42:0/3064305066 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f757c11c030 0x7f757c11b170 secure :-1 s=THROTTLE_DONE pgs=3221 cs=0 l=1 rx=0x7f756c001210 tx=0x7f756c012ed0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:19.532 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:19.516+0000 7f7578ff9700 1 -- 172.21.15.42:0/3064305066 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f756c0148c0 con 0x7f757c11c030 2019-12-05T06:48:19.532 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:19.516+0000 7f7578ff9700 1 -- 172.21.15.42:0/3064305066 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f756c00cb70 con 0x7f757c11c030 2019-12-05T06:48:19.532 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:19.516+0000 7f75820f8700 1 -- 172.21.15.42:0/3064305066 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f757c12dcc0 con 0x7f757c11c030 2019-12-05T06:48:19.533 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:19.516+0000 7f75820f8700 1 -- 172.21.15.42:0/3064305066 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f757c07cfa0 con 0x7f757c11c030 2019-12-05T06:48:19.533 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:19.516+0000 7f757bfff700 1 --2- 172.21.15.42:0/3064305066 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f757c11c030 0x7f757c11b170 secure :-1 s=THROTTLE_DONE pgs=3221 cs=0 l=1 rx=0x7f756c001210 tx=0x7f756c012ed0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:19.533 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:19.516+0000 7f7578ff9700 1 -- 172.21.15.42:0/3064305066 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f756c0046f0 con 0x7f757c11c030 2019-12-05T06:48:19.533 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:19.516+0000 7f757bfff700 1 --2- 172.21.15.42:0/3064305066 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f757c11c030 0x7f757c11b170 secure :-1 s=THROTTLE_DONE pgs=3221 cs=0 l=1 rx=0x7f756c001210 tx=0x7f756c012ed0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:19.534 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:19.516+0000 7f7578ff9700 1 --2- 172.21.15.42:0/3064305066 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f756003ad10 0x7f756003d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:48:19.534 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:19.516+0000 7f7578ff9700 1 -- 172.21.15.42:0/3064305066 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 12051+0+0 (secure 0 0 0) 0x7f756c04d0b0 con 0x7f757c11c030 2019-12-05T06:48:19.534 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:19.516+0000 7f757b7fe700 1 --2- 172.21.15.42:0/3064305066 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f756003ad10 0x7f756003d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:19.534 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:19.516+0000 7f757b7fe700 1 --2- 172.21.15.42:0/3064305066 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f756003ad10 0x7f756003d220 crc :-1 s=READY pgs=1351 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:48:19.534 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:19.520+0000 7f75820f8700 1 -- 172.21.15.42:0/3064305066 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f757c05d110 con 0x7f757c11c030 2019-12-05T06:48:19.535 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:19.528+0000 7f757bfff700 1 --2- 172.21.15.42:0/3064305066 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f757c11c030 0x7f757c11b170 secure :-1 s=THROTTLE_DONE pgs=3221 cs=0 l=1 rx=0x7f756c001210 tx=0x7f756c012ed0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:19.535 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:19.528+0000 7f7578ff9700 1 -- 172.21.15.42:0/3064305066 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f756c034130 con 0x7f757c11c030 2019-12-05T06:48:19.709 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:19.704+0000 7f75820f8700 1 -- 172.21.15.42:0/3064305066 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f757c05b8c0 con 0x7f757c11c030 2019-12-05T06:48:19.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:19.704+0000 7f757bfff700 1 --2- 172.21.15.42:0/3064305066 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f757c11c030 0x7f757c11b170 secure :-1 s=THROTTLE_DONE pgs=3221 cs=0 l=1 rx=0x7f756c001210 tx=0x7f756c012ed0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:19.711 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:19.704+0000 7f7578ff9700 1 -- 172.21.15.42:0/3064305066 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 389 v389) v1 ==== 95+0+2117 (secure 0 0 0) 0x7f756c034390 con 0x7f757c11c030 2019-12-05T06:48:19.711 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 389 2019-12-05T06:48:19.711 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:48:19.711 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":389,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":12847,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/1449317789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1449317789},{"type":"v1","addr":"172.21.15.42:6835","nonce":1449317789}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":389}],"filesystems":[{"mdsmap":{"epoch":388,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:46:05.602252+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":119,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12429},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12429":{"gid":12429,"name":"b","rank":0,"incarnation":384,"state":"up:active","state_seq":26,"addr":"172.21.15.40:6833/4178556019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":4178556019},{"type":"v1","addr":"172.21.15.40:6833","nonce":4178556019}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:48:19.713 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:19.708+0000 7f75820f8700 1 -- 172.21.15.42:0/3064305066 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f756003ad10 msgr2=0x7f756003d220 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:19.713 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:19.708+0000 7f75820f8700 1 --2- 172.21.15.42:0/3064305066 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f756003ad10 0x7f756003d220 crc :-1 s=READY pgs=1351 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:19.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:19.708+0000 7f75820f8700 1 -- 172.21.15.42:0/3064305066 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f757c11c030 msgr2=0x7f757c11b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:19.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:19.708+0000 7f75820f8700 1 --2- 172.21.15.42:0/3064305066 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f757c11c030 0x7f757c11b170 secure :-1 s=READY pgs=3221 cs=0 l=1 rx=0x7f756c001210 tx=0x7f756c012ed0).stop 2019-12-05T06:48:19.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:19.708+0000 7f75820f8700 1 -- 172.21.15.42:0/3064305066 shutdown_connections 2019-12-05T06:48:19.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:19.708+0000 7f75820f8700 1 --2- 172.21.15.42:0/3064305066 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f756003ad10 0x7f756003d220 unknown :-1 s=CLOSED pgs=1351 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:19.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:19.708+0000 7f75820f8700 1 --2- 172.21.15.42:0/3064305066 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f757c11c030 0x7f757c11b170 unknown :-1 s=CLOSED pgs=3221 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:19.715 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:19.708+0000 7f75820f8700 1 --2- 172.21.15.42:0/3064305066 >> v2:172.21.15.40:3300/0 conn(0x7f757c11a730 0x7f757c12d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:19.715 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:19.708+0000 7f75820f8700 1 -- 172.21.15.42:0/3064305066 shutdown_connections 2019-12-05T06:48:19.715 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:19.708+0000 7f75820f8700 1 -- 172.21.15.42:0/3064305066 wait complete. 2019-12-05T06:48:19.715 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:19.708+0000 7f75820f8700 1 -- 172.21.15.42:0/3064305066 >> 172.21.15.42:0/3064305066 conn(0x7f757c08fde0 msgr2=0x7f757c0a9870 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:48:19.727 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:48:20.474 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:48:20.478 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:48:21.728 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:48:21.817 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:21.808+0000 7fb3f2dfd700 1 Processor -- start 2019-12-05T06:48:21.817 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:21.808+0000 7fb3f2dfd700 1 -- start start 2019-12-05T06:48:21.817 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:21.808+0000 7fb3f2dfd700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb3ec0a7740 0x7fb3ec08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:48:21.817 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:21.808+0000 7fb3f2dfd700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb3ec07e720 0x7fb3ec07eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:48:21.817 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:21.808+0000 7fb3f2dfd700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb3ec09f2e0 0x7fb3ec09f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:48:21.818 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:21.808+0000 7fb3f2dfd700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3ec0a0b00 con 0x7fb3ec07e720 2019-12-05T06:48:21.818 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:21.808+0000 7fb3f2dfd700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3ec08e5b0 con 0x7fb3ec09f2e0 2019-12-05T06:48:21.818 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:21.808+0000 7fb3f2dfd700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3ec08e850 con 0x7fb3ec0a7740 2019-12-05T06:48:21.818 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:21.808+0000 7fb3f1b9b700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb3ec09f2e0 0x7fb3ec09f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:21.818 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:21.808+0000 7fb3f1b9b700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb3ec09f2e0 0x7fb3ec09f710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:37204/0 (socket says 172.21.15.42:37204) 2019-12-05T06:48:21.818 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:21.808+0000 7fb3f1b9b700 1 -- 172.21.15.42:0/1163784402 learned_addr learned my addr 172.21.15.42:0/1163784402 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:48:21.818 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:21.808+0000 7fb3f0b99700 1 --2- 172.21.15.42:0/1163784402 >> v2:172.21.15.40:3300/0 conn(0x7fb3ec07e720 0x7fb3ec07eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:21.819 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:21.808+0000 7fb3f139a700 1 --2- 172.21.15.42:0/1163784402 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb3ec0a7740 0x7fb3ec08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:21.819 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:21.808+0000 7fb3f1b9b700 1 -- 172.21.15.42:0/1163784402 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb3ec0a7740 msgr2=0x7fb3ec08df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:48:21.819 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:21.808+0000 7fb3f1b9b700 1 --2- 172.21.15.42:0/1163784402 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb3ec0a7740 0x7fb3ec08df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:48:21.819 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:21.808+0000 7fb3f1b9b700 1 -- 172.21.15.42:0/1163784402 >> v2:172.21.15.40:3300/0 conn(0x7fb3ec07e720 msgr2=0x7fb3ec07eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:48:21.819 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:21.808+0000 7fb3f1b9b700 1 --2- 172.21.15.42:0/1163784402 >> v2:172.21.15.40:3300/0 conn(0x7fb3ec07e720 0x7fb3ec07eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:48:21.819 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:21.808+0000 7fb3f1b9b700 1 -- 172.21.15.42:0/1163784402 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3ec07f0a0 con 0x7fb3ec09f2e0 2019-12-05T06:48:21.819 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:21.808+0000 7fb3f1b9b700 1 --2- 172.21.15.42:0/1163784402 >> v2:172.21.15.42:3300/0 conn(0x7fb3ec09f2e0 0x7fb3ec09f710 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fb3ec08e770 tx=0x7fb3dc00fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:21.820 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:21.808+0000 7fb3f1b9b700 1 --2- 172.21.15.42:0/1163784402 >> v2:172.21.15.42:3300/0 conn(0x7fb3ec09f2e0 0x7fb3ec09f710 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fb3ec08e770 tx=0x7fb3dc00fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:21.820 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:21.808+0000 7fb3f1b9b700 1 --2- 172.21.15.42:0/1163784402 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb3ec09f2e0 0x7fb3ec09f710 secure :-1 s=READY pgs=3225 cs=0 l=1 rx=0x7fb3ec08e770 tx=0x7fb3dc00fdb0).ready entity=mon.1 client_cookie=616ce3c9e20ba0ec server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:48:21.820 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:21.808+0000 7fb3f1b9b700 1 --2- 172.21.15.42:0/1163784402 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb3ec09f2e0 0x7fb3ec09f710 secure :-1 s=THROTTLE_DONE pgs=3225 cs=0 l=1 rx=0x7fb3ec08e770 tx=0x7fb3dc00fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:21.820 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:21.808+0000 7fb3ebfff700 1 -- 172.21.15.42:0/1163784402 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb3dc008c90 con 0x7fb3ec09f2e0 2019-12-05T06:48:21.824 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:21.808+0000 7fb3f1b9b700 1 --2- 172.21.15.42:0/1163784402 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb3ec09f2e0 0x7fb3ec09f710 secure :-1 s=THROTTLE_DONE pgs=3225 cs=0 l=1 rx=0x7fb3ec08e770 tx=0x7fb3dc00fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:21.825 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:21.808+0000 7fb3f1b9b700 1 --2- 172.21.15.42:0/1163784402 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb3ec09f2e0 0x7fb3ec09f710 secure :-1 s=THROTTLE_DONE pgs=3225 cs=0 l=1 rx=0x7fb3ec08e770 tx=0x7fb3dc00fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:21.825 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:21.808+0000 7fb3ebfff700 1 -- 172.21.15.42:0/1163784402 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3dc0088b0 con 0x7fb3ec09f2e0 2019-12-05T06:48:21.825 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:21.808+0000 7fb3ebfff700 1 -- 172.21.15.42:0/1163784402 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb3dc011b50 con 0x7fb3ec09f2e0 2019-12-05T06:48:21.825 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:21.808+0000 7fb3f2dfd700 1 -- 172.21.15.42:0/1163784402 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb3ec09f2e0 msgr2=0x7fb3ec09f710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:21.825 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:21.808+0000 7fb3f2dfd700 1 --2- 172.21.15.42:0/1163784402 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb3ec09f2e0 0x7fb3ec09f710 secure :-1 s=READY pgs=3225 cs=0 l=1 rx=0x7fb3ec08e770 tx=0x7fb3dc00fdb0).stop 2019-12-05T06:48:21.825 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:21.808+0000 7fb3f2dfd700 1 -- 172.21.15.42:0/1163784402 shutdown_connections 2019-12-05T06:48:21.825 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:21.808+0000 7fb3f2dfd700 1 --2- 172.21.15.42:0/1163784402 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb3ec09f2e0 0x7fb3ec09f710 unknown :-1 s=CLOSED pgs=3225 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:21.826 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:21.808+0000 7fb3f2dfd700 1 --2- 172.21.15.42:0/1163784402 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb3ec0a7740 0x7fb3ec08df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:48:21.826 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:21.808+0000 7fb3f2dfd700 1 --2- 172.21.15.42:0/1163784402 >> v2:172.21.15.40:3300/0 conn(0x7fb3ec07e720 0x7fb3ec07eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:48:21.826 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:21.808+0000 7fb3f2dfd700 1 -- 172.21.15.42:0/1163784402 shutdown_connections 2019-12-05T06:48:21.826 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:21.808+0000 7fb3f2dfd700 1 -- 172.21.15.42:0/1163784402 wait complete. 2019-12-05T06:48:21.826 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:21.808+0000 7fb3f2dfd700 1 -- 172.21.15.42:0/1163784402 >> 172.21.15.42:0/1163784402 conn(0x7fb3ec0aabe0 msgr2=0x7fb3ec0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:48:21.826 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:21.812+0000 7fb3f2dfd700 1 Processor -- start 2019-12-05T06:48:21.826 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:21.812+0000 7fb3f2dfd700 1 -- start start 2019-12-05T06:48:21.829 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:21.812+0000 7fb3f2dfd700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb3ec0752f0 0x7fb3ec077730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:48:21.829 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:21.812+0000 7fb3f2dfd700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3ec11be20 con 0x7fb3ec1161c0 2019-12-05T06:48:21.831 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:21.812+0000 7fb3f2dfd700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3ec1333d0 con 0x7fb3ec1217d0 2019-12-05T06:48:21.831 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:21.812+0000 7fb3f2dfd700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3ec11b0f0 con 0x7fb3ec0752f0 2019-12-05T06:48:21.831 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:21.812+0000 7fb3f139a700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fb3ec1217d0 0x7fb3ec11dcd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:37244/0 (socket says 172.21.15.42:37244) 2019-12-05T06:48:21.831 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:21.812+0000 7fb3f139a700 1 -- 172.21.15.42:0/3184097314 learned_addr learned my addr 172.21.15.42:0/3184097314 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:48:21.832 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:21.812+0000 7fb3f1b9b700 1 --2- 172.21.15.42:0/3184097314 >> v2:172.21.15.42:3300/0 conn(0x7fb3ec0752f0 0x7fb3ec077730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:21.832 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:21.812+0000 7fb3f1b9b700 1 -- 172.21.15.42:0/3184097314 >> v1:172.21.15.42:6789/0 conn(0x7fb3ec1217d0 legacy=0x7fb3ec11dcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:21.832 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:21.812+0000 7fb3f1b9b700 1 -- 172.21.15.42:0/3184097314 >> v1:172.21.15.40:6789/0 conn(0x7fb3ec1161c0 legacy=0x7fb3ec11b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:21.832 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:21.812+0000 7fb3f1b9b700 1 -- 172.21.15.42:0/3184097314 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3ec1165f0 con 0x7fb3ec0752f0 2019-12-05T06:48:21.833 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:21.812+0000 7fb3f1b9b700 1 --2- 172.21.15.42:0/3184097314 >> v2:172.21.15.42:3300/0 conn(0x7fb3ec0752f0 0x7fb3ec077730 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fb3ec08e770 tx=0x7fb3dc0086c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:21.833 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:21.812+0000 7fb3f1b9b700 1 --2- 172.21.15.42:0/3184097314 >> v2:172.21.15.42:3300/0 conn(0x7fb3ec0752f0 0x7fb3ec077730 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fb3ec08e770 tx=0x7fb3dc0086c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:21.833 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:21.812+0000 7fb3f1b9b700 1 --2- 172.21.15.42:0/3184097314 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb3ec0752f0 0x7fb3ec077730 secure :-1 s=READY pgs=3227 cs=0 l=1 rx=0x7fb3ec08e770 tx=0x7fb3dc0086c0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:48:21.834 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:21.812+0000 7fb3f1b9b700 1 --2- 172.21.15.42:0/3184097314 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb3ec0752f0 0x7fb3ec077730 secure :-1 s=THROTTLE_DONE pgs=3227 cs=0 l=1 rx=0x7fb3ec08e770 tx=0x7fb3dc0086c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:21.834 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:21.812+0000 7fb3f1b9b700 1 --2- 172.21.15.42:0/3184097314 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb3ec0752f0 0x7fb3ec077730 secure :-1 s=THROTTLE_DONE pgs=3227 cs=0 l=1 rx=0x7fb3ec08e770 tx=0x7fb3dc0086c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:21.834 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:21.812+0000 7fb3e9ffb700 1 -- 172.21.15.42:0/3184097314 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb3dc00f470 con 0x7fb3ec0752f0 2019-12-05T06:48:21.834 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:21.812+0000 7fb3f1b9b700 1 --2- 172.21.15.42:0/3184097314 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb3ec0752f0 0x7fb3ec077730 secure :-1 s=THROTTLE_DONE pgs=3227 cs=0 l=1 rx=0x7fb3ec08e770 tx=0x7fb3dc0086c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:21.834 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:21.812+0000 7fb3e9ffb700 1 -- 172.21.15.42:0/3184097314 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3dc011770 con 0x7fb3ec0752f0 2019-12-05T06:48:21.834 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:21.812+0000 7fb3f2dfd700 1 -- 172.21.15.42:0/3184097314 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3ec077c80 con 0x7fb3ec0752f0 2019-12-05T06:48:21.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:21.812+0000 7fb3e9ffb700 1 -- 172.21.15.42:0/3184097314 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb3dc005730 con 0x7fb3ec0752f0 2019-12-05T06:48:21.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:21.812+0000 7fb3f2dfd700 1 -- 172.21.15.42:0/3184097314 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3ec0781d0 con 0x7fb3ec0752f0 2019-12-05T06:48:21.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:21.812+0000 7fb3f1b9b700 1 --2- 172.21.15.42:0/3184097314 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb3ec0752f0 0x7fb3ec077730 secure :-1 s=THROTTLE_DONE pgs=3227 cs=0 l=1 rx=0x7fb3ec08e770 tx=0x7fb3dc0086c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:21.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:21.816+0000 7fb3e9ffb700 1 -- 172.21.15.42:0/3184097314 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fb3dc005a40 con 0x7fb3ec0752f0 2019-12-05T06:48:21.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:21.816+0000 7fb3f1b9b700 1 --2- 172.21.15.42:0/3184097314 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb3ec0752f0 0x7fb3ec077730 secure :-1 s=THROTTLE_DONE pgs=3227 cs=0 l=1 rx=0x7fb3ec08e770 tx=0x7fb3dc0086c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:21.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:21.816+0000 7fb3e9ffb700 1 --2- 172.21.15.42:0/3184097314 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb3cc03adf0 0x7fb3cc03d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:48:21.835 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:21.816+0000 7fb3e9ffb700 1 -- 172.21.15.42:0/3184097314 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 12137+0+0 (secure 0 0 0) 0x7fb3dc08f5b0 con 0x7fb3ec0752f0 2019-12-05T06:48:21.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:21.816+0000 7fb3f139a700 1 --2- 172.21.15.42:0/3184097314 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb3cc03adf0 0x7fb3cc03d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:21.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:21.816+0000 7fb3f2dfd700 1 -- 172.21.15.42:0/3184097314 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3ec05d110 con 0x7fb3ec0752f0 2019-12-05T06:48:21.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:21.816+0000 7fb3f139a700 1 --2- 172.21.15.42:0/3184097314 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb3cc03adf0 0x7fb3cc03d300 crc :-1 s=READY pgs=1352 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:48:21.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:21.820+0000 7fb3f1b9b700 1 --2- 172.21.15.42:0/3184097314 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb3ec0752f0 0x7fb3ec077730 secure :-1 s=THROTTLE_DONE pgs=3227 cs=0 l=1 rx=0x7fb3ec08e770 tx=0x7fb3dc0086c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:21.836 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:21.820+0000 7fb3e9ffb700 1 -- 172.21.15.42:0/3184097314 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fb3dc076890 con 0x7fb3ec0752f0 2019-12-05T06:48:21.984 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:21.976+0000 7fb3f2dfd700 1 -- 172.21.15.42:0/3184097314 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb3ec05b8c0 con 0x7fb3ec0752f0 2019-12-05T06:48:21.985 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:21.980+0000 7fb3f1b9b700 1 --2- 172.21.15.42:0/3184097314 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb3ec0752f0 0x7fb3ec077730 secure :-1 s=THROTTLE_DONE pgs=3227 cs=0 l=1 rx=0x7fb3ec08e770 tx=0x7fb3dc0086c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:21.985 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:21.980+0000 7fb3e9ffb700 1 -- 172.21.15.42:0/3184097314 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 390 v390) v1 ==== 95+0+1778 (secure 0 0 0) 0x7fb3dc005d50 con 0x7fb3ec0752f0 2019-12-05T06:48:21.985 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 390 2019-12-05T06:48:21.985 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:48:21.986 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":390,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":390,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:48:20.367393+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":120,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12847},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12847":{"gid":12847,"name":"a","rank":0,"incarnation":390,"state":"up:replay","state_seq":2,"addr":"172.21.15.42:6835/1449317789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1449317789},{"type":"v1","addr":"172.21.15.42:6835","nonce":1449317789}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:48:21.987 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:21.980+0000 7fb3f2dfd700 1 -- 172.21.15.42:0/3184097314 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb3cc03adf0 msgr2=0x7fb3cc03d300 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:21.988 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:21.980+0000 7fb3f2dfd700 1 --2- 172.21.15.42:0/3184097314 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb3cc03adf0 0x7fb3cc03d300 crc :-1 s=READY pgs=1352 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:21.988 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:21.980+0000 7fb3f2dfd700 1 -- 172.21.15.42:0/3184097314 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb3ec0752f0 msgr2=0x7fb3ec077730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:21.988 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:21.980+0000 7fb3f2dfd700 1 --2- 172.21.15.42:0/3184097314 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb3ec0752f0 0x7fb3ec077730 secure :-1 s=READY pgs=3227 cs=0 l=1 rx=0x7fb3ec08e770 tx=0x7fb3dc0086c0).stop 2019-12-05T06:48:21.988 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:21.980+0000 7fb3f2dfd700 1 -- 172.21.15.42:0/3184097314 shutdown_connections 2019-12-05T06:48:21.988 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:21.980+0000 7fb3f2dfd700 1 --2- 172.21.15.42:0/3184097314 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb3cc03adf0 0x7fb3cc03d300 unknown :-1 s=CLOSED pgs=1352 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:21.989 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:21.980+0000 7fb3f2dfd700 1 --2- 172.21.15.42:0/3184097314 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb3ec0752f0 0x7fb3ec077730 unknown :-1 s=CLOSED pgs=3227 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:21.989 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:21.980+0000 7fb3f2dfd700 1 -- 172.21.15.42:0/3184097314 shutdown_connections 2019-12-05T06:48:21.989 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:21.980+0000 7fb3f2dfd700 1 -- 172.21.15.42:0/3184097314 wait complete. 2019-12-05T06:48:21.989 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:21.980+0000 7fb3f2dfd700 1 -- 172.21.15.42:0/3184097314 >> 172.21.15.42:0/3184097314 conn(0x7fb3ec07e4f0 msgr2=0x7fb3ec11c610 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:48:21.997 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b down, removed from mdsmap 2019-12-05T06:48:21.998 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2019-12-05T06:48:21.998 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:48:22.100 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:22.088+0000 7f603a416700 1 Processor -- start 2019-12-05T06:48:22.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:22.088+0000 7f603a416700 1 -- start start 2019-12-05T06:48:22.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:22.092+0000 7f603a416700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f603408df80 0x7f603409f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:48:22.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:22.092+0000 7f603a416700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f603409f830 0x7f603414efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:48:22.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:22.092+0000 7f603a416700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f603409fc60 con 0x7f60340a7740 2019-12-05T06:48:22.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:22.092+0000 7f603a416700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60340a0b00 con 0x7f603408df80 2019-12-05T06:48:22.101 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:22.092+0000 7f603a416700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f603408e5b0 con 0x7f603409f830 2019-12-05T06:48:22.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:22.092+0000 7f60391b4700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f603409f830 0x7f603414efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:22.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:22.092+0000 7f6033fff700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f603408df80 0x7f603409f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:22.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:22.092+0000 7f60391b4700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f603409f830 0x7f603414efe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:37684/0 (socket says 172.21.15.42:37684) 2019-12-05T06:48:22.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:22.092+0000 7f60391b4700 1 -- 172.21.15.42:0/363696630 learned_addr learned my addr 172.21.15.42:0/363696630 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:48:22.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:22.092+0000 7f60391b4700 1 -- 172.21.15.42:0/363696630 >> v2:172.21.15.40:3300/0 conn(0x7f603408df80 msgr2=0x7f603409f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:48:22.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:22.092+0000 7f60391b4700 1 --2- 172.21.15.42:0/363696630 >> v2:172.21.15.40:3300/0 conn(0x7f603408df80 0x7f603409f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:48:22.103 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:22.092+0000 7f60391b4700 1 -- 172.21.15.42:0/363696630 >> v1:172.21.15.40:6789/0 conn(0x7f60340a7740 legacy=0x7f603407e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:22.103 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:22.092+0000 7f60391b4700 1 -- 172.21.15.42:0/363696630 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60340a0710 con 0x7f603409f830 2019-12-05T06:48:22.103 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:22.092+0000 7f60391b4700 1 --2- 172.21.15.42:0/363696630 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f603409f830 0x7f603414efe0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f602c008b10 tx=0x7f602c0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:22.103 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:22.092+0000 7f60391b4700 1 --2- 172.21.15.42:0/363696630 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f603409f830 0x7f603414efe0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f602c008b10 tx=0x7f602c0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:22.103 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:22.092+0000 7f60391b4700 1 --2- 172.21.15.42:0/363696630 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f603409f830 0x7f603414efe0 secure :-1 s=READY pgs=717 cs=0 l=1 rx=0x7f602c008b10 tx=0x7f602c0100b0).ready entity=mon.2 client_cookie=35a1621f15dcf445 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:48:22.103 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:22.092+0000 7f60391b4700 1 --2- 172.21.15.42:0/363696630 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f603409f830 0x7f603414efe0 secure :-1 s=THROTTLE_DONE pgs=717 cs=0 l=1 rx=0x7f602c008b10 tx=0x7f602c0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:22.103 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:22.092+0000 7f60337fe700 1 -- 172.21.15.42:0/363696630 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f602c010da0 con 0x7f603409f830 2019-12-05T06:48:22.104 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:22.092+0000 7f60391b4700 1 --2- 172.21.15.42:0/363696630 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f603409f830 0x7f603414efe0 secure :-1 s=THROTTLE_DONE pgs=717 cs=0 l=1 rx=0x7f602c008b10 tx=0x7f602c0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:22.104 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:22.092+0000 7f60337fe700 1 -- 172.21.15.42:0/363696630 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f602c0109c0 con 0x7f603409f830 2019-12-05T06:48:22.104 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:22.092+0000 7f60391b4700 1 --2- 172.21.15.42:0/363696630 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f603409f830 0x7f603414efe0 secure :-1 s=THROTTLE_DONE pgs=717 cs=0 l=1 rx=0x7f602c008b10 tx=0x7f602c0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:22.104 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:22.092+0000 7f603a416700 1 -- 172.21.15.42:0/363696630 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f603409f830 msgr2=0x7f603414efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:22.105 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:22.092+0000 7f60337fe700 1 -- 172.21.15.42:0/363696630 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f602c0114a0 con 0x7f603409f830 2019-12-05T06:48:22.105 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:22.092+0000 7f603a416700 1 --2- 172.21.15.42:0/363696630 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f603409f830 0x7f603414efe0 secure :-1 s=READY pgs=717 cs=0 l=1 rx=0x7f602c008b10 tx=0x7f602c0100b0).stop 2019-12-05T06:48:22.106 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:22.092+0000 7f603a416700 1 -- 172.21.15.42:0/363696630 shutdown_connections 2019-12-05T06:48:22.106 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:22.092+0000 7f603a416700 1 --2- 172.21.15.42:0/363696630 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f603409f830 0x7f603414efe0 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:22.106 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:22.092+0000 7f603a416700 1 --2- 172.21.15.42:0/363696630 >> v2:172.21.15.40:3300/0 conn(0x7f603408df80 0x7f603409f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:48:22.107 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:22.092+0000 7f603a416700 1 -- 172.21.15.42:0/363696630 shutdown_connections 2019-12-05T06:48:22.107 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:22.092+0000 7f603a416700 1 -- 172.21.15.42:0/363696630 wait complete. 2019-12-05T06:48:22.107 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:22.092+0000 7f603a416700 1 -- 172.21.15.42:0/363696630 >> 172.21.15.42:0/363696630 conn(0x7f60340aabe0 msgr2=0x7f60340a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:48:22.114 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:22.096+0000 7f603a416700 1 Processor -- start 2019-12-05T06:48:22.114 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:22.096+0000 7f603a416700 1 -- start start 2019-12-05T06:48:22.114 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:22.100+0000 7f603a416700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f60341185c0 0x7f603411be70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:48:22.115 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:22.100+0000 7f603a416700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f603407c9f0 0x7f6034137bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:48:22.115 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:22.100+0000 7f603a416700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60340a6cd0 con 0x7f603407c5c0 2019-12-05T06:48:22.116 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:22.100+0000 7f603a416700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60340a4ea0 con 0x7f60341185c0 2019-12-05T06:48:22.116 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:22.100+0000 7f603a416700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60340a5140 con 0x7f603407c9f0 2019-12-05T06:48:22.116 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:22.100+0000 7f60391b4700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f603407c9f0 0x7f6034137bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:22.116 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:22.100+0000 7f6033fff700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f603407c5c0 0x7f603407aee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:46700/0 (socket says 172.21.15.42:46700) 2019-12-05T06:48:22.117 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:22.100+0000 7f60389b3700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f60341185c0 0x7f603411be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:22.117 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:22.100+0000 7f60391b4700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f603407c9f0 0x7f6034137bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:37690/0 (socket says 172.21.15.42:37690) 2019-12-05T06:48:22.117 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:22.100+0000 7f6033fff700 1 -- 172.21.15.42:0/1104603695 learned_addr learned my addr 172.21.15.42:0/1104603695 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:48:22.117 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:22.100+0000 7f60317fa700 1 -- 172.21.15.42:0/1104603695 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3660527208 0 0) 0x7f60340a6cd0 con 0x7f603407c5c0 2019-12-05T06:48:22.118 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:22.100+0000 7f60317fa700 1 -- 172.21.15.42:0/1104603695 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60140030b0 con 0x7f603407c5c0 2019-12-05T06:48:22.118 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:22.100+0000 7f60389b3700 1 -- 172.21.15.42:0/1104603695 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f603407c9f0 msgr2=0x7f6034137bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:22.118 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:22.100+0000 7f60389b3700 1 --2- 172.21.15.42:0/1104603695 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f603407c9f0 0x7f6034137bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:22.118 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:22.100+0000 7f60389b3700 1 -- 172.21.15.42:0/1104603695 >> v1:172.21.15.40:6789/0 conn(0x7f603407c5c0 legacy=0x7f603407aee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:22.119 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:22.100+0000 7f60389b3700 1 -- 172.21.15.42:0/1104603695 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f603407b6e0 con 0x7f60341185c0 2019-12-05T06:48:22.119 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:22.100+0000 7f60391b4700 1 --2- 172.21.15.42:0/1104603695 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f603407c9f0 0x7f6034137bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:48:22.119 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:22.100+0000 7f60389b3700 1 --2- 172.21.15.42:0/1104603695 >> v2:172.21.15.40:3300/0 conn(0x7f60341185c0 0x7f603411be70 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f60340675c0 tx=0x7f60280109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:22.119 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:22.100+0000 7f60389b3700 1 --2- 172.21.15.42:0/1104603695 >> v2:172.21.15.40:3300/0 conn(0x7f60341185c0 0x7f603411be70 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f60340675c0 tx=0x7f60280109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:22.120 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:22.100+0000 7f60389b3700 1 --2- 172.21.15.42:0/1104603695 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f60341185c0 0x7f603411be70 secure :-1 s=READY pgs=2451 cs=0 l=1 rx=0x7f60340675c0 tx=0x7f60280109a0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:48:22.120 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:22.100+0000 7f60389b3700 1 --2- 172.21.15.42:0/1104603695 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f60341185c0 0x7f603411be70 secure :-1 s=THROTTLE_DONE pgs=2451 cs=0 l=1 rx=0x7f60340675c0 tx=0x7f60280109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:22.120 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:22.100+0000 7f60317fa700 1 -- 172.21.15.42:0/1104603695 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f602800caa0 con 0x7f60341185c0 2019-12-05T06:48:22.120 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:22.100+0000 7f603a416700 1 -- 172.21.15.42:0/1104603695 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f603411c3c0 con 0x7f60341185c0 2019-12-05T06:48:22.121 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:22.100+0000 7f60389b3700 1 --2- 172.21.15.42:0/1104603695 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f60341185c0 0x7f603411be70 secure :-1 s=THROTTLE_DONE pgs=2451 cs=0 l=1 rx=0x7f60340675c0 tx=0x7f60280109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:22.121 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:22.100+0000 7f603a416700 1 -- 172.21.15.42:0/1104603695 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60340a5680 con 0x7f60341185c0 2019-12-05T06:48:22.121 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:22.100+0000 7f60317fa700 1 -- 172.21.15.42:0/1104603695 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f602800c6c0 con 0x7f60341185c0 2019-12-05T06:48:22.122 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:22.100+0000 7f60389b3700 1 --2- 172.21.15.42:0/1104603695 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f60341185c0 0x7f603411be70 secure :-1 s=THROTTLE_DONE pgs=2451 cs=0 l=1 rx=0x7f60340675c0 tx=0x7f60280109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:22.122 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:22.100+0000 7f60317fa700 1 -- 172.21.15.42:0/1104603695 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6028011120 con 0x7f60341185c0 2019-12-05T06:48:22.122 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:22.104+0000 7f60389b3700 1 --2- 172.21.15.42:0/1104603695 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f60341185c0 0x7f603411be70 secure :-1 s=THROTTLE_DONE pgs=2451 cs=0 l=1 rx=0x7f60340675c0 tx=0x7f60280109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:22.122 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:22.104+0000 7f60317fa700 1 -- 172.21.15.42:0/1104603695 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f6028011440 con 0x7f60341185c0 2019-12-05T06:48:22.123 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:22.104+0000 7f60389b3700 1 --2- 172.21.15.42:0/1104603695 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f60341185c0 0x7f603411be70 secure :-1 s=THROTTLE_DONE pgs=2451 cs=0 l=1 rx=0x7f60340675c0 tx=0x7f60280109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:22.123 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:22.104+0000 7f60317fa700 1 --2- 172.21.15.42:0/1104603695 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f601403b200 0x7f601403d710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:48:22.123 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:22.104+0000 7f60317fa700 1 -- 172.21.15.42:0/1104603695 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 12137+0+0 (secure 0 0 0) 0x7f602804b2c0 con 0x7f60341185c0 2019-12-05T06:48:22.124 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:22.104+0000 7f6033fff700 1 --2- 172.21.15.42:0/1104603695 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f601403b200 0x7f601403d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:22.124 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:22.104+0000 7f6033fff700 1 --2- 172.21.15.42:0/1104603695 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f601403b200 0x7f601403d710 crc :-1 s=READY pgs=1353 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:48:22.124 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:22.104+0000 7f603a416700 1 -- 172.21.15.42:0/1104603695 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f603405d110 con 0x7f60341185c0 2019-12-05T06:48:22.125 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:22.112+0000 7f60389b3700 1 --2- 172.21.15.42:0/1104603695 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f60341185c0 0x7f603411be70 secure :-1 s=THROTTLE_DONE pgs=2451 cs=0 l=1 rx=0x7f60340675c0 tx=0x7f60280109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:22.126 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:22.116+0000 7f60317fa700 1 -- 172.21.15.42:0/1104603695 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f603407b6e0 con 0x7f60341185c0 2019-12-05T06:48:22.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:22.284+0000 7f603a416700 1 -- 172.21.15.42:0/1104603695 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f603405b8c0 con 0x7f60341185c0 2019-12-05T06:48:22.291 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:22.284+0000 7f60389b3700 1 --2- 172.21.15.42:0/1104603695 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f60341185c0 0x7f603411be70 secure :-1 s=THROTTLE_DONE pgs=2451 cs=0 l=1 rx=0x7f60340675c0 tx=0x7f60280109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:22.291 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:22.284+0000 7f60317fa700 1 -- 172.21.15.42:0/1104603695 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 390 v390) v1 ==== 95+0+1778 (secure 0 0 0) 0x7f6028032240 con 0x7f60341185c0 2019-12-05T06:48:22.293 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 390 2019-12-05T06:48:22.293 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:48:22.294 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":390,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":390,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:48:20.367393+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":120,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12847},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12847":{"gid":12847,"name":"a","rank":0,"incarnation":390,"state":"up:replay","state_seq":2,"addr":"172.21.15.42:6835/1449317789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1449317789},{"type":"v1","addr":"172.21.15.42:6835","nonce":1449317789}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:48:22.298 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:22.292+0000 7f603a416700 1 -- 172.21.15.42:0/1104603695 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f601403b200 msgr2=0x7f601403d710 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:22.298 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:22.292+0000 7f603a416700 1 --2- 172.21.15.42:0/1104603695 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f601403b200 0x7f601403d710 crc :-1 s=READY pgs=1353 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:22.298 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:22.292+0000 7f603a416700 1 -- 172.21.15.42:0/1104603695 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f60341185c0 msgr2=0x7f603411be70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:22.298 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:22.292+0000 7f603a416700 1 --2- 172.21.15.42:0/1104603695 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f60341185c0 0x7f603411be70 secure :-1 s=READY pgs=2451 cs=0 l=1 rx=0x7f60340675c0 tx=0x7f60280109a0).stop 2019-12-05T06:48:22.298 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:22.292+0000 7f603a416700 1 -- 172.21.15.42:0/1104603695 shutdown_connections 2019-12-05T06:48:22.299 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:22.292+0000 7f603a416700 1 --2- 172.21.15.42:0/1104603695 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f601403b200 0x7f601403d710 unknown :-1 s=CLOSED pgs=1353 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:22.299 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:22.292+0000 7f603a416700 1 --2- 172.21.15.42:0/1104603695 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f603407c9f0 0x7f6034137bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:22.299 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:22.292+0000 7f603a416700 1 --2- 172.21.15.42:0/1104603695 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f60341185c0 0x7f603411be70 unknown :-1 s=CLOSED pgs=2451 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:22.300 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:22.292+0000 7f603a416700 1 -- 172.21.15.42:0/1104603695 shutdown_connections 2019-12-05T06:48:22.300 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:22.292+0000 7f603a416700 1 -- 172.21.15.42:0/1104603695 wait complete. 2019-12-05T06:48:22.300 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:22.292+0000 7f603a416700 1 -- 172.21.15.42:0/1104603695 >> 172.21.15.42:0/1104603695 conn(0x7f603411d3f0 msgr2=0x7f6034059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:48:22.309 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 390, "feature_flags": { "enable_multiple": false, "ever_enabled_multiple": false }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2019-12-05T05:03:35.604738+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 390, "ever_allowed_features": 2, "explicitly_allowed_features": 2, "failed": [], "flags": 18, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_12847": { "addr": "172.21.15.42:6835/1449317789", "addrs": { "addrvec": [ { "addr": "172.21.15.42:6834", "nonce": 1449317789, "type": "v2" }, { "addr": "172.21.15.42:6835", "nonce": 1449317789, "type": "v1" } ] }, "export_targets": [], "features": 4540138292836696063, "flags": 0, "gid": 12847, "incarnation": 390, "name": "a", "rank": 0, "state": "up:replay", "state_seq": 2 } }, "last_failure": 0, "last_failure_osd_epoch": 120, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 2, "min_compat_client": "0 (unknown)", "modified": "2019-12-05T06:48:20.367393+0000", "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 12847 } } } ], "standbys": [] } 2019-12-05T06:48:24.310 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:48:24.433 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:24.424+0000 7fdb5cb12700 1 Processor -- start 2019-12-05T06:48:24.436 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:24.424+0000 7fdb5cb12700 1 -- start start 2019-12-05T06:48:24.436 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:24.424+0000 7fdb5cb12700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fdb580a7740 0x7fdb5808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:48:24.436 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:24.424+0000 7fdb5cb12700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fdb5807e720 0x7fdb5807eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:48:24.436 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:24.424+0000 7fdb5cb12700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb580a0710 con 0x7fdb5809f2e0 2019-12-05T06:48:24.437 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:24.424+0000 7fdb5cb12700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb580a0b00 con 0x7fdb5807e720 2019-12-05T06:48:24.437 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:24.424+0000 7fdb5cb12700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb5808e5b0 con 0x7fdb580a7740 2019-12-05T06:48:24.437 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:24.424+0000 7fdb56d9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fdb580a7740 0x7fdb5808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:24.437 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:24.424+0000 7fdb56d9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fdb580a7740 0x7fdb5808df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:37228/0 (socket says 172.21.15.42:37228) 2019-12-05T06:48:24.438 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:24.424+0000 7fdb56d9d700 1 -- 172.21.15.42:0/1852191951 learned_addr learned my addr 172.21.15.42:0/1852191951 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:48:24.438 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:24.424+0000 7fdb5659c700 1 --2- 172.21.15.42:0/1852191951 >> v2:172.21.15.40:3300/0 conn(0x7fdb5807e720 0x7fdb5807eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:24.438 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:24.424+0000 7fdb56d9d700 1 -- 172.21.15.42:0/1852191951 >> v2:172.21.15.40:3300/0 conn(0x7fdb5807e720 msgr2=0x7fdb5807eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:48:24.438 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:24.424+0000 7fdb56d9d700 1 --2- 172.21.15.42:0/1852191951 >> v2:172.21.15.40:3300/0 conn(0x7fdb5807e720 0x7fdb5807eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:48:24.439 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:24.424+0000 7fdb56d9d700 1 -- 172.21.15.42:0/1852191951 >> v1:172.21.15.42:6789/0 conn(0x7fdb5809f2e0 legacy=0x7fdb5809f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:24.439 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:24.424+0000 7fdb56d9d700 1 -- 172.21.15.42:0/1852191951 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb5807f0a0 con 0x7fdb580a7740 2019-12-05T06:48:24.440 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:24.424+0000 7fdb56d9d700 1 --2- 172.21.15.42:0/1852191951 >> v2:172.21.15.42:3300/0 conn(0x7fdb580a7740 0x7fdb5808df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fdb500032b0 tx=0x7fdb500100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:24.440 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:24.424+0000 7fdb56d9d700 1 --2- 172.21.15.42:0/1852191951 >> v2:172.21.15.42:3300/0 conn(0x7fdb580a7740 0x7fdb5808df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fdb500032b0 tx=0x7fdb500100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:24.440 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:24.424+0000 7fdb56d9d700 1 --2- 172.21.15.42:0/1852191951 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdb580a7740 0x7fdb5808df80 secure :-1 s=READY pgs=3229 cs=0 l=1 rx=0x7fdb500032b0 tx=0x7fdb500100b0).ready entity=mon.1 client_cookie=a4e90906eafda5b9 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:48:24.440 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:24.424+0000 7fdb56d9d700 1 --2- 172.21.15.42:0/1852191951 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdb580a7740 0x7fdb5808df80 secure :-1 s=THROTTLE_DONE pgs=3229 cs=0 l=1 rx=0x7fdb500032b0 tx=0x7fdb500100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:24.441 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:24.424+0000 7fdb55d9b700 1 -- 172.21.15.42:0/1852191951 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdb50011dd0 con 0x7fdb580a7740 2019-12-05T06:48:24.441 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:24.424+0000 7fdb56d9d700 1 --2- 172.21.15.42:0/1852191951 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdb580a7740 0x7fdb5808df80 secure :-1 s=THROTTLE_DONE pgs=3229 cs=0 l=1 rx=0x7fdb500032b0 tx=0x7fdb500100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:24.441 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:24.424+0000 7fdb56d9d700 1 --2- 172.21.15.42:0/1852191951 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdb580a7740 0x7fdb5808df80 secure :-1 s=THROTTLE_DONE pgs=3229 cs=0 l=1 rx=0x7fdb500032b0 tx=0x7fdb500100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:24.441 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:24.424+0000 7fdb55d9b700 1 -- 172.21.15.42:0/1852191951 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb500119f0 con 0x7fdb580a7740 2019-12-05T06:48:24.442 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:24.424+0000 7fdb55d9b700 1 -- 172.21.15.42:0/1852191951 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdb5000a540 con 0x7fdb580a7740 2019-12-05T06:48:24.442 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:24.424+0000 7fdb5cb12700 1 -- 172.21.15.42:0/1852191951 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdb580a7740 msgr2=0x7fdb5808df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:24.442 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:24.424+0000 7fdb5cb12700 1 --2- 172.21.15.42:0/1852191951 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdb580a7740 0x7fdb5808df80 secure :-1 s=READY pgs=3229 cs=0 l=1 rx=0x7fdb500032b0 tx=0x7fdb500100b0).stop 2019-12-05T06:48:24.442 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:24.424+0000 7fdb5cb12700 1 -- 172.21.15.42:0/1852191951 shutdown_connections 2019-12-05T06:48:24.443 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:24.424+0000 7fdb5cb12700 1 --2- 172.21.15.42:0/1852191951 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdb580a7740 0x7fdb5808df80 unknown :-1 s=CLOSED pgs=3229 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:24.443 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:24.424+0000 7fdb5cb12700 1 --2- 172.21.15.42:0/1852191951 >> v2:172.21.15.40:3300/0 conn(0x7fdb5807e720 0x7fdb5807eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:48:24.443 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:24.424+0000 7fdb5cb12700 1 -- 172.21.15.42:0/1852191951 shutdown_connections 2019-12-05T06:48:24.443 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:24.424+0000 7fdb5cb12700 1 -- 172.21.15.42:0/1852191951 wait complete. 2019-12-05T06:48:24.444 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:24.424+0000 7fdb5cb12700 1 -- 172.21.15.42:0/1852191951 >> 172.21.15.42:0/1852191951 conn(0x7fdb580aabe0 msgr2=0x7fdb580a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:48:24.444 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:24.428+0000 7fdb5cb12700 1 Processor -- start 2019-12-05T06:48:24.444 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:24.428+0000 7fdb5cb12700 1 -- start start 2019-12-05T06:48:24.451 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:24.428+0000 7fdb5cb12700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fdb580a6950 0x7fdb58063270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:48:24.452 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:24.428+0000 7fdb5cb12700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fdb580a2ea0 0x7fdb580647a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:48:24.452 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:24.428+0000 7fdb5cb12700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb58065b70 con 0x7fdb58065740 2019-12-05T06:48:24.453 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:24.428+0000 7fdb5cb12700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb58064f20 con 0x7fdb580a2ea0 2019-12-05T06:48:24.453 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:24.428+0000 7fdb5cb12700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb580654a0 con 0x7fdb580a6950 2019-12-05T06:48:24.453 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:24.428+0000 7fdb56d9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fdb580a6950 0x7fdb58063270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:24.453 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:24.428+0000 7fdb5659c700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fdb580a2ea0 0x7fdb580647a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:24.454 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:24.428+0000 7fdb56d9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fdb580a6950 0x7fdb58063270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:37234/0 (socket says 172.21.15.42:37234) 2019-12-05T06:48:24.454 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:24.428+0000 7fdb56d9d700 1 -- 172.21.15.42:0/840077473 learned_addr learned my addr 172.21.15.42:0/840077473 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:48:24.454 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:24.428+0000 7fdb5759e700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fdb58065740 0x7fdb5808eb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:37276/0 (socket says 172.21.15.42:37276) 2019-12-05T06:48:24.454 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:24.428+0000 7fdb56d9d700 1 -- 172.21.15.42:0/840077473 >> v2:172.21.15.40:3300/0 conn(0x7fdb580a2ea0 msgr2=0x7fdb580647a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:24.455 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:24.428+0000 7fdb56d9d700 1 --2- 172.21.15.42:0/840077473 >> v2:172.21.15.40:3300/0 conn(0x7fdb580a2ea0 0x7fdb580647a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:24.455 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:24.428+0000 7fdb56d9d700 1 -- 172.21.15.42:0/840077473 >> v1:172.21.15.42:6789/0 conn(0x7fdb58065740 legacy=0x7fdb5808eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:24.456 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:24.428+0000 7fdb56d9d700 1 -- 172.21.15.42:0/840077473 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb58065da0 con 0x7fdb580a6950 2019-12-05T06:48:24.456 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:24.428+0000 7fdb4f7fe700 1 -- 172.21.15.42:0/840077473 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3312381677 0 0) 0x7fdb58065b70 con 0x7fdb58065740 2019-12-05T06:48:24.456 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:24.428+0000 7fdb56d9d700 1 --2- 172.21.15.42:0/840077473 >> v2:172.21.15.42:3300/0 conn(0x7fdb580a6950 0x7fdb58063270 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fdb500032b0 tx=0x7fdb50011f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:24.456 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:24.428+0000 7fdb56d9d700 1 --2- 172.21.15.42:0/840077473 >> v2:172.21.15.42:3300/0 conn(0x7fdb580a6950 0x7fdb58063270 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fdb500032b0 tx=0x7fdb50011f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:24.457 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:24.428+0000 7fdb56d9d700 1 --2- 172.21.15.42:0/840077473 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdb580a6950 0x7fdb58063270 secure :-1 s=READY pgs=3231 cs=0 l=1 rx=0x7fdb500032b0 tx=0x7fdb50011f70).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:48:24.457 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:24.428+0000 7fdb56d9d700 1 --2- 172.21.15.42:0/840077473 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdb580a6950 0x7fdb58063270 secure :-1 s=THROTTLE_DONE pgs=3231 cs=0 l=1 rx=0x7fdb500032b0 tx=0x7fdb50011f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:24.457 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:24.428+0000 7fdb4f7fe700 1 -- 172.21.15.42:0/840077473 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdb5000add0 con 0x7fdb580a6950 2019-12-05T06:48:24.457 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:24.428+0000 7fdb5cb12700 1 -- 172.21.15.42:0/840077473 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb5808f350 con 0x7fdb580a6950 2019-12-05T06:48:24.458 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:24.428+0000 7fdb5cb12700 1 -- 172.21.15.42:0/840077473 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb5808f8d0 con 0x7fdb580a6950 2019-12-05T06:48:24.458 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:24.428+0000 7fdb56d9d700 1 --2- 172.21.15.42:0/840077473 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdb580a6950 0x7fdb58063270 secure :-1 s=THROTTLE_DONE pgs=3231 cs=0 l=1 rx=0x7fdb500032b0 tx=0x7fdb50011f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:24.458 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:24.428+0000 7fdb56d9d700 1 --2- 172.21.15.42:0/840077473 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdb580a6950 0x7fdb58063270 secure :-1 s=THROTTLE_DONE pgs=3231 cs=0 l=1 rx=0x7fdb500032b0 tx=0x7fdb50011f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:24.458 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:24.428+0000 7fdb4f7fe700 1 -- 172.21.15.42:0/840077473 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb5000a160 con 0x7fdb580a6950 2019-12-05T06:48:24.458 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:24.428+0000 7fdb4f7fe700 1 -- 172.21.15.42:0/840077473 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdb50005ca0 con 0x7fdb580a6950 2019-12-05T06:48:24.458 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:24.428+0000 7fdb56d9d700 1 --2- 172.21.15.42:0/840077473 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdb580a6950 0x7fdb58063270 secure :-1 s=THROTTLE_DONE pgs=3231 cs=0 l=1 rx=0x7fdb500032b0 tx=0x7fdb50011f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:24.459 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:24.432+0000 7fdb4f7fe700 1 -- 172.21.15.42:0/840077473 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fdb50007150 con 0x7fdb580a6950 2019-12-05T06:48:24.459 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:24.432+0000 7fdb56d9d700 1 --2- 172.21.15.42:0/840077473 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdb580a6950 0x7fdb58063270 secure :-1 s=THROTTLE_DONE pgs=3231 cs=0 l=1 rx=0x7fdb500032b0 tx=0x7fdb50011f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:24.459 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:24.432+0000 7fdb4f7fe700 1 --2- 172.21.15.42:0/840077473 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fdb3c03ac40 0x7fdb3c03d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:48:24.459 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:24.432+0000 7fdb4f7fe700 1 -- 172.21.15.42:0/840077473 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 12137+0+0 (secure 0 0 0) 0x7fdb5004bd50 con 0x7fdb580a6950 2019-12-05T06:48:24.459 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:24.432+0000 7fdb5659c700 1 --2- 172.21.15.42:0/840077473 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fdb3c03ac40 0x7fdb3c03d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:24.460 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:24.432+0000 7fdb5659c700 1 --2- 172.21.15.42:0/840077473 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fdb3c03ac40 0x7fdb3c03d150 crc :-1 s=READY pgs=1354 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:48:24.461 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:24.432+0000 7fdb5cb12700 1 -- 172.21.15.42:0/840077473 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb30003200 con 0x7fdb580a6950 2019-12-05T06:48:24.461 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:24.436+0000 7fdb56d9d700 1 --2- 172.21.15.42:0/840077473 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdb580a6950 0x7fdb58063270 secure :-1 s=THROTTLE_DONE pgs=3231 cs=0 l=1 rx=0x7fdb500032b0 tx=0x7fdb50011f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:24.462 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:24.436+0000 7fdb4f7fe700 1 -- 172.21.15.42:0/840077473 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fdb50032dd0 con 0x7fdb580a6950 2019-12-05T06:48:24.614 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:24.608+0000 7fdb5cb12700 1 -- 172.21.15.42:0/840077473 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdb30003630 con 0x7fdb580a6950 2019-12-05T06:48:24.615 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:24.608+0000 7fdb56d9d700 1 --2- 172.21.15.42:0/840077473 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdb580a6950 0x7fdb58063270 secure :-1 s=THROTTLE_DONE pgs=3231 cs=0 l=1 rx=0x7fdb500032b0 tx=0x7fdb50011f70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:24.615 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:24.608+0000 7fdb4f7fe700 1 -- 172.21.15.42:0/840077473 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 390 v390) v1 ==== 95+0+1778 (secure 0 0 0) 0x7fdb50033030 con 0x7fdb580a6950 2019-12-05T06:48:24.615 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 390 2019-12-05T06:48:24.615 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:48:24.616 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":390,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":390,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:48:20.367393+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":120,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12847},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12847":{"gid":12847,"name":"a","rank":0,"incarnation":390,"state":"up:replay","state_seq":2,"addr":"172.21.15.42:6835/1449317789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1449317789},{"type":"v1","addr":"172.21.15.42:6835","nonce":1449317789}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:48:24.617 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:24.612+0000 7fdb5cb12700 1 -- 172.21.15.42:0/840077473 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fdb3c03ac40 msgr2=0x7fdb3c03d150 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:24.618 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:24.612+0000 7fdb5cb12700 1 --2- 172.21.15.42:0/840077473 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fdb3c03ac40 0x7fdb3c03d150 crc :-1 s=READY pgs=1354 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:24.618 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:24.612+0000 7fdb5cb12700 1 -- 172.21.15.42:0/840077473 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdb580a6950 msgr2=0x7fdb58063270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:24.618 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:24.612+0000 7fdb5cb12700 1 --2- 172.21.15.42:0/840077473 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdb580a6950 0x7fdb58063270 secure :-1 s=READY pgs=3231 cs=0 l=1 rx=0x7fdb500032b0 tx=0x7fdb50011f70).stop 2019-12-05T06:48:24.618 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:24.612+0000 7fdb5cb12700 1 -- 172.21.15.42:0/840077473 shutdown_connections 2019-12-05T06:48:24.618 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:24.612+0000 7fdb5cb12700 1 --2- 172.21.15.42:0/840077473 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fdb3c03ac40 0x7fdb3c03d150 unknown :-1 s=CLOSED pgs=1354 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:24.619 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:24.612+0000 7fdb5cb12700 1 --2- 172.21.15.42:0/840077473 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdb580a6950 0x7fdb58063270 unknown :-1 s=CLOSED pgs=3231 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:24.619 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:24.612+0000 7fdb5cb12700 1 --2- 172.21.15.42:0/840077473 >> v2:172.21.15.40:3300/0 conn(0x7fdb580a2ea0 0x7fdb580647a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:24.619 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:24.612+0000 7fdb5cb12700 1 -- 172.21.15.42:0/840077473 shutdown_connections 2019-12-05T06:48:24.619 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:24.612+0000 7fdb5cb12700 1 -- 172.21.15.42:0/840077473 wait complete. 2019-12-05T06:48:24.619 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:24.612+0000 7fdb5cb12700 1 -- 172.21.15.42:0/840077473 >> 172.21.15.42:0/840077473 conn(0x7fdb580a4a50 msgr2=0x7fdb58059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:48:24.626 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2019-12-05T06:48:26.627 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:48:26.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:26.960+0000 7f534e02b700 1 Processor -- start 2019-12-05T06:48:26.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:26.960+0000 7f534e02b700 1 -- start start 2019-12-05T06:48:26.976 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:26.960+0000 7f534e02b700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f534808df80 0x7f534809f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:48:26.976 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:26.960+0000 7f534e02b700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f534809f830 0x7f534814efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:48:26.976 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:26.960+0000 7f534e02b700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f534809fc60 con 0x7f53480a7740 2019-12-05T06:48:26.976 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:26.960+0000 7f534e02b700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53480a0b00 con 0x7f534808df80 2019-12-05T06:48:26.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:26.960+0000 7f534e02b700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f534808e5b0 con 0x7f534809f830 2019-12-05T06:48:26.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:26.960+0000 7f5347fff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f53480a7740 0x7f534807e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:37286/0 (socket says 172.21.15.42:37286) 2019-12-05T06:48:26.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:26.960+0000 7f5347fff700 1 -- 172.21.15.42:0/1772518860 learned_addr learned my addr 172.21.15.42:0/1772518860 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:48:26.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:26.960+0000 7f534cdc9700 1 --2- 172.21.15.42:0/1772518860 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f534809f830 0x7f534814efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:26.978 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:26.960+0000 7f533ffff700 1 --2- 172.21.15.42:0/1772518860 >> v2:172.21.15.40:3300/0 conn(0x7f534808df80 0x7f534809f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:26.979 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:26.960+0000 7f53477fe700 1 -- 172.21.15.42:0/1772518860 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1228663279 0 0) 0x7f534809fc60 con 0x7f53480a7740 2019-12-05T06:48:26.979 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:26.960+0000 7f53477fe700 1 -- 172.21.15.42:0/1772518860 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53340030b0 con 0x7f53480a7740 2019-12-05T06:48:26.979 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:26.960+0000 7f53477fe700 1 -- 172.21.15.42:0/1772518860 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 402330208 0 0) 0x7f53340030b0 con 0x7f53480a7740 2019-12-05T06:48:26.979 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:26.960+0000 7f53477fe700 1 -- 172.21.15.42:0/1772518860 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f534809f830 msgr2=0x7f534814efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:48:26.980 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:26.960+0000 7f53477fe700 1 --2- 172.21.15.42:0/1772518860 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f534809f830 0x7f534814efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:48:26.980 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:26.960+0000 7f53477fe700 1 -- 172.21.15.42:0/1772518860 >> v2:172.21.15.40:3300/0 conn(0x7f534808df80 msgr2=0x7f534809f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:48:26.980 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:26.960+0000 7f53477fe700 1 --2- 172.21.15.42:0/1772518860 >> v2:172.21.15.40:3300/0 conn(0x7f534808df80 0x7f534809f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:48:26.980 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:26.960+0000 7f53477fe700 1 -- 172.21.15.42:0/1772518860 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53480a0710 con 0x7f53480a7740 2019-12-05T06:48:26.981 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:26.960+0000 7f53477fe700 1 -- 172.21.15.42:0/1772518860 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7f5338001750 con 0x7f53480a7740 2019-12-05T06:48:26.981 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:26.960+0000 7f53477fe700 1 -- 172.21.15.42:0/1772518860 >> v1:172.21.15.42:6789/0 conn(0x7f53480a7740 legacy=0x7f534807e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:26.981 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:26.960+0000 7f53477fe700 1 --2- 172.21.15.42:0/1772518860 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f53340036e0 0x7f5334003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:48:26.981 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:26.960+0000 7f53477fe700 1 --2- 172.21.15.42:0/1772518860 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5334004420 0x7f5334004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:48:26.982 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:26.960+0000 7f53477fe700 1 --2- 172.21.15.42:0/1772518860 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5334004da0 0x7f533404f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:48:26.982 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:26.960+0000 7f53477fe700 1 -- 172.21.15.42:0/1772518860 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53480a0b00 con 0x7f5334004420 2019-12-05T06:48:26.982 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:26.960+0000 7f53477fe700 1 -- 172.21.15.42:0/1772518860 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f534808e5b0 con 0x7f53340036e0 2019-12-05T06:48:26.982 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:26.960+0000 7f53477fe700 1 -- 172.21.15.42:0/1772518860 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5334005250 con 0x7f5334004da0 2019-12-05T06:48:26.983 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:26.960+0000 7f5347fff700 1 --2- 172.21.15.42:0/1772518860 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f53340036e0 0x7f5334003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:26.983 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:26.964+0000 7f534cdc9700 1 --2- 172.21.15.42:0/1772518860 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5334004da0 0x7f533404f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:26.983 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:26.964+0000 7f533ffff700 1 --2- 172.21.15.42:0/1772518860 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5334004420 0x7f5334004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:26.983 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:26.964+0000 7f5347fff700 1 -- 172.21.15.42:0/1772518860 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5334004da0 msgr2=0x7f533404f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:48:26.984 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:26.964+0000 7f5347fff700 1 --2- 172.21.15.42:0/1772518860 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5334004da0 0x7f533404f680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:48:26.984 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:26.964+0000 7f5347fff700 1 -- 172.21.15.42:0/1772518860 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5334004420 msgr2=0x7f5334004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:48:26.984 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:26.964+0000 7f5347fff700 1 --2- 172.21.15.42:0/1772518860 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5334004420 0x7f5334004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:48:26.984 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:26.964+0000 7f5347fff700 1 -- 172.21.15.42:0/1772518860 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f533404fd10 con 0x7f53340036e0 2019-12-05T06:48:26.984 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:26.964+0000 7f534cdc9700 1 --2- 172.21.15.42:0/1772518860 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5334004da0 0x7f533404f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:48:26.985 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:26.964+0000 7f533ffff700 1 --2- 172.21.15.42:0/1772518860 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5334004420 0x7f5334004850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:48:26.985 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:26.964+0000 7f5347fff700 1 --2- 172.21.15.42:0/1772518860 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f53340036e0 0x7f5334003b50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f5338001a50 tx=0x7f533800c1c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:26.985 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:26.964+0000 7f5347fff700 1 --2- 172.21.15.42:0/1772518860 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f53340036e0 0x7f5334003b50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f5338001a50 tx=0x7f533800c1c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:26.985 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:26.964+0000 7f5347fff700 1 --2- 172.21.15.42:0/1772518860 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f53340036e0 0x7f5334003b50 secure :-1 s=READY pgs=3233 cs=0 l=1 rx=0x7f5338001a50 tx=0x7f533800c1c0).ready entity=mon.1 client_cookie=c29b23d4a4317108 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:48:26.986 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:26.964+0000 7f5347fff700 1 --2- 172.21.15.42:0/1772518860 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f53340036e0 0x7f5334003b50 secure :-1 s=THROTTLE_DONE pgs=3233 cs=0 l=1 rx=0x7f5338001a50 tx=0x7f533800c1c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:26.986 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:26.964+0000 7f53477fe700 1 -- 172.21.15.42:0/1772518860 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f533800b260 con 0x7f53340036e0 2019-12-05T06:48:26.986 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:26.964+0000 7f5347fff700 1 --2- 172.21.15.42:0/1772518860 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f53340036e0 0x7f5334003b50 secure :-1 s=THROTTLE_DONE pgs=3233 cs=0 l=1 rx=0x7f5338001a50 tx=0x7f533800c1c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:26.986 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:26.964+0000 7f5347fff700 1 --2- 172.21.15.42:0/1772518860 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f53340036e0 0x7f5334003b50 secure :-1 s=THROTTLE_DONE pgs=3233 cs=0 l=1 rx=0x7f5338001a50 tx=0x7f533800c1c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:26.987 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:26.964+0000 7f53477fe700 1 -- 172.21.15.42:0/1772518860 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f533800edb0 con 0x7f53340036e0 2019-12-05T06:48:26.987 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:26.964+0000 7f53477fe700 1 -- 172.21.15.42:0/1772518860 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5338010580 con 0x7f53340036e0 2019-12-05T06:48:26.987 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:26.964+0000 7f534e02b700 1 -- 172.21.15.42:0/1772518860 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f53340036e0 msgr2=0x7f5334003b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:26.987 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:26.964+0000 7f534e02b700 1 --2- 172.21.15.42:0/1772518860 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f53340036e0 0x7f5334003b50 secure :-1 s=READY pgs=3233 cs=0 l=1 rx=0x7f5338001a50 tx=0x7f533800c1c0).stop 2019-12-05T06:48:26.988 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:26.964+0000 7f534cdc9700 1 -- 172.21.15.42:0/1772518860 reap_dead start 2019-12-05T06:48:26.988 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:26.964+0000 7f534e02b700 1 -- 172.21.15.42:0/1772518860 shutdown_connections 2019-12-05T06:48:27.000 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:26.964+0000 7f534e02b700 1 -- 172.21.15.42:0/1772518860 shutdown_connections 2019-12-05T06:48:27.000 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:26.964+0000 7f534e02b700 1 -- 172.21.15.42:0/1772518860 wait complete. 2019-12-05T06:48:27.000 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:26.964+0000 7f534e02b700 1 -- 172.21.15.42:0/1772518860 >> 172.21.15.42:0/1772518860 conn(0x7f53480aabe0 msgr2=0x7f53480a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:48:27.001 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:26.964+0000 7f534e02b700 1 Processor -- start 2019-12-05T06:48:27.001 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:26.964+0000 7f534e02b700 1 -- start start 2019-12-05T06:48:27.001 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:26.964+0000 7f534e02b700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f534811bc40 0x7f53480a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:48:27.001 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:26.964+0000 7f534e02b700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53480a5400 con 0x7f534807a640 2019-12-05T06:48:27.002 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:26.964+0000 7f534e02b700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53480a4840 con 0x7f5348133c80 2019-12-05T06:48:27.002 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:26.964+0000 7f534e02b700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53480a5780 con 0x7f534811bc40 2019-12-05T06:48:27.002 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:26.964+0000 7f5347fff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f534811bc40 0x7f53480a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:27.002 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:26.964+0000 7f533ffff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f5348133c80 0x7f53481340b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:37300/0 (socket says 172.21.15.42:37300) 2019-12-05T06:48:27.003 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:26.964+0000 7f533ffff700 1 -- 172.21.15.42:0/3661171494 learned_addr learned my addr 172.21.15.42:0/3661171494 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:48:27.003 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:26.968+0000 7f5347fff700 1 -- 172.21.15.42:0/3661171494 >> v1:172.21.15.42:6789/0 conn(0x7f5348133c80 legacy=0x7f53481340b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:27.003 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:26.968+0000 7f5347fff700 1 -- 172.21.15.42:0/3661171494 >> v1:172.21.15.40:6789/0 conn(0x7f534807a640 legacy=0x7f534807aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:27.003 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:26.968+0000 7f53457fa700 1 -- 172.21.15.42:0/3661171494 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3815155098 0 0) 0x7f53480a4840 con 0x7f5348133c80 2019-12-05T06:48:27.003 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:26.968+0000 7f5347fff700 1 -- 172.21.15.42:0/3661171494 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f534807b270 con 0x7f534811bc40 2019-12-05T06:48:27.004 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:26.968+0000 7f5347fff700 1 --2- 172.21.15.42:0/3661171494 >> v2:172.21.15.42:3300/0 conn(0x7f534811bc40 0x7f53480a4eb0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f5338001a50 tx=0x7f53380016b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:27.004 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:26.968+0000 7f5347fff700 1 --2- 172.21.15.42:0/3661171494 >> v2:172.21.15.42:3300/0 conn(0x7f534811bc40 0x7f53480a4eb0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f5338001a50 tx=0x7f53380016b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:27.004 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:26.968+0000 7f5347fff700 1 --2- 172.21.15.42:0/3661171494 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f534811bc40 0x7f53480a4eb0 secure :-1 s=READY pgs=3235 cs=0 l=1 rx=0x7f5338001a50 tx=0x7f53380016b0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:48:27.004 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:26.968+0000 7f5347fff700 1 --2- 172.21.15.42:0/3661171494 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f534811bc40 0x7f53480a4eb0 secure :-1 s=THROTTLE_DONE pgs=3235 cs=0 l=1 rx=0x7f5338001a50 tx=0x7f53380016b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:27.005 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:26.968+0000 7f53457fa700 1 -- 172.21.15.42:0/3661171494 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5338010790 con 0x7f534811bc40 2019-12-05T06:48:27.005 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:26.968+0000 7f5347fff700 1 --2- 172.21.15.42:0/3661171494 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f534811bc40 0x7f53480a4eb0 secure :-1 s=THROTTLE_DONE pgs=3235 cs=0 l=1 rx=0x7f5338001a50 tx=0x7f53380016b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:27.005 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:26.968+0000 7f5347fff700 1 --2- 172.21.15.42:0/3661171494 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f534811bc40 0x7f53480a4eb0 secure :-1 s=THROTTLE_DONE pgs=3235 cs=0 l=1 rx=0x7f5338001a50 tx=0x7f53380016b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:27.005 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:26.968+0000 7f53457fa700 1 -- 172.21.15.42:0/3661171494 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53380101a0 con 0x7f534811bc40 2019-12-05T06:48:27.006 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:26.968+0000 7f53457fa700 1 -- 172.21.15.42:0/3661171494 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f533800a4a0 con 0x7f534811bc40 2019-12-05T06:48:27.006 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:26.968+0000 7f534e02b700 1 -- 172.21.15.42:0/3661171494 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f534813bef0 con 0x7f534811bc40 2019-12-05T06:48:27.006 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:26.968+0000 7f534e02b700 1 -- 172.21.15.42:0/3661171494 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f534813c360 con 0x7f534811bc40 2019-12-05T06:48:27.006 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:26.968+0000 7f5347fff700 1 --2- 172.21.15.42:0/3661171494 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f534811bc40 0x7f53480a4eb0 secure :-1 s=THROTTLE_DONE pgs=3235 cs=0 l=1 rx=0x7f5338001a50 tx=0x7f53380016b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:27.007 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:26.968+0000 7f53457fa700 1 -- 172.21.15.42:0/3661171494 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f533800a770 con 0x7f534811bc40 2019-12-05T06:48:27.007 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:26.968+0000 7f5347fff700 1 --2- 172.21.15.42:0/3661171494 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f534811bc40 0x7f53480a4eb0 secure :-1 s=THROTTLE_DONE pgs=3235 cs=0 l=1 rx=0x7f5338001a50 tx=0x7f53380016b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:27.007 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:26.968+0000 7f53457fa700 1 --2- 172.21.15.42:0/3661171494 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f532c03abd0 0x7f532c03d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:48:27.007 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:26.968+0000 7f53457fa700 1 -- 172.21.15.42:0/3661171494 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 12137+0+0 (secure 0 0 0) 0x7f533804b260 con 0x7f534811bc40 2019-12-05T06:48:27.008 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:26.968+0000 7f533ffff700 1 --2- 172.21.15.42:0/3661171494 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f532c03abd0 0x7f532c03d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:27.008 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:26.968+0000 7f533ffff700 1 --2- 172.21.15.42:0/3661171494 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f532c03abd0 0x7f532c03d0e0 crc :-1 s=READY pgs=1355 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:48:27.009 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:26.976+0000 7f534e02b700 1 -- 172.21.15.42:0/3661171494 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f534805d110 con 0x7f534811bc40 2019-12-05T06:48:27.010 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:26.980+0000 7f5347fff700 1 --2- 172.21.15.42:0/3661171494 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f534811bc40 0x7f53480a4eb0 secure :-1 s=THROTTLE_DONE pgs=3235 cs=0 l=1 rx=0x7f5338001a50 tx=0x7f53380016b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:27.010 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:26.980+0000 7f53457fa700 1 -- 172.21.15.42:0/3661171494 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f53380322e0 con 0x7f534811bc40 2019-12-05T06:48:27.152 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:27.140+0000 7f534e02b700 1 -- 172.21.15.42:0/3661171494 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f534805b8c0 con 0x7f534811bc40 2019-12-05T06:48:27.152 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:27.140+0000 7f5347fff700 1 --2- 172.21.15.42:0/3661171494 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f534811bc40 0x7f53480a4eb0 secure :-1 s=THROTTLE_DONE pgs=3235 cs=0 l=1 rx=0x7f5338001a50 tx=0x7f53380016b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:27.153 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:27.140+0000 7f53457fa700 1 -- 172.21.15.42:0/3661171494 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 390 v390) v1 ==== 95+0+1778 (secure 0 0 0) 0x7f5338032540 con 0x7f534811bc40 2019-12-05T06:48:27.153 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 390 2019-12-05T06:48:27.153 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:27.144+0000 7f534e02b700 1 -- 172.21.15.42:0/3661171494 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f532c03abd0 msgr2=0x7f532c03d0e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:27.153 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:27.144+0000 7f534e02b700 1 --2- 172.21.15.42:0/3661171494 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f532c03abd0 0x7f532c03d0e0 crc :-1 s=READY pgs=1355 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:27.154 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:27.144+0000 7f534e02b700 1 -- 172.21.15.42:0/3661171494 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f534811bc40 msgr2=0x7f53480a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:27.154 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:27.144+0000 7f534e02b700 1 --2- 172.21.15.42:0/3661171494 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f534811bc40 0x7f53480a4eb0 secure :-1 s=READY pgs=3235 cs=0 l=1 rx=0x7f5338001a50 tx=0x7f53380016b0).stop 2019-12-05T06:48:27.154 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:27.144+0000 7f534e02b700 1 -- 172.21.15.42:0/3661171494 shutdown_connections 2019-12-05T06:48:27.154 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:27.144+0000 7f534e02b700 1 --2- 172.21.15.42:0/3661171494 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f532c03abd0 0x7f532c03d0e0 unknown :-1 s=CLOSED pgs=1355 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:27.175 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:27.144+0000 7f534e02b700 1 --2- 172.21.15.42:0/3661171494 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f534811bc40 0x7f53480a4eb0 unknown :-1 s=CLOSED pgs=3235 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:27.175 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:27.144+0000 7f534e02b700 1 -- 172.21.15.42:0/3661171494 shutdown_connections 2019-12-05T06:48:27.176 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:27.144+0000 7f534e02b700 1 -- 172.21.15.42:0/3661171494 wait complete. 2019-12-05T06:48:27.176 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:27.144+0000 7f534e02b700 1 -- 172.21.15.42:0/3661171494 >> 172.21.15.42:0/3661171494 conn(0x7f53480a9640 msgr2=0x7f534807e830 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:48:27.177 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:48:27.177 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":390,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":390,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:48:20.367393+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":120,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12847},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12847":{"gid":12847,"name":"a","rank":0,"incarnation":390,"state":"up:replay","state_seq":2,"addr":"172.21.15.42:6835/1449317789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1449317789},{"type":"v1","addr":"172.21.15.42:6835","nonce":1449317789}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:48:27.178 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2019-12-05T06:48:29.179 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:48:29.296 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:29.280+0000 7fc77e08d700 1 Processor -- start 2019-12-05T06:48:29.296 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:29.280+0000 7fc77e08d700 1 -- start start 2019-12-05T06:48:29.296 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:29.280+0000 7fc77e08d700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc7780a7740 0x7fc77808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:48:29.297 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:29.280+0000 7fc77e08d700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc77807e720 0x7fc77807eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:48:29.297 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:29.280+0000 7fc77e08d700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7780a0710 con 0x7fc77809f2e0 2019-12-05T06:48:29.297 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:29.280+0000 7fc77e08d700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7780a0b00 con 0x7fc77807e720 2019-12-05T06:48:29.297 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:29.280+0000 7fc77e08d700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc77808e5b0 con 0x7fc7780a7740 2019-12-05T06:48:29.297 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:29.280+0000 7fc77ce2b700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fc77809f2e0 0x7fc77809f710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:37310/0 (socket says 172.21.15.42:37310) 2019-12-05T06:48:29.297 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:29.280+0000 7fc77ce2b700 1 -- 172.21.15.42:0/3966138905 learned_addr learned my addr 172.21.15.42:0/3966138905 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:48:29.297 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:29.280+0000 7fc7777fe700 1 --2- 172.21.15.42:0/3966138905 >> v2:172.21.15.40:3300/0 conn(0x7fc77807e720 0x7fc77807eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:29.301 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:29.280+0000 7fc777fff700 1 --2- 172.21.15.42:0/3966138905 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc7780a7740 0x7fc77808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:29.301 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:29.280+0000 7fc776ffd700 1 -- 172.21.15.42:0/3966138905 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1732055553 0 0) 0x7fc7780a0710 con 0x7fc77809f2e0 2019-12-05T06:48:29.301 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:29.280+0000 7fc776ffd700 1 -- 172.21.15.42:0/3966138905 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7580030b0 con 0x7fc77809f2e0 2019-12-05T06:48:29.301 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:29.280+0000 7fc777fff700 1 -- 172.21.15.42:0/3966138905 >> v2:172.21.15.40:3300/0 conn(0x7fc77807e720 msgr2=0x7fc77807eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:48:29.301 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:29.280+0000 7fc777fff700 1 --2- 172.21.15.42:0/3966138905 >> v2:172.21.15.40:3300/0 conn(0x7fc77807e720 0x7fc77807eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:48:29.301 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:29.280+0000 7fc777fff700 1 -- 172.21.15.42:0/3966138905 >> v1:172.21.15.42:6789/0 conn(0x7fc77809f2e0 legacy=0x7fc77809f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:29.302 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:29.280+0000 7fc777fff700 1 -- 172.21.15.42:0/3966138905 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc77807f0a0 con 0x7fc7780a7740 2019-12-05T06:48:29.302 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:29.280+0000 7fc7777fe700 1 --2- 172.21.15.42:0/3966138905 >> v2:172.21.15.40:3300/0 conn(0x7fc77807e720 0x7fc77807eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:48:29.302 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:29.280+0000 7fc777fff700 1 --2- 172.21.15.42:0/3966138905 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc7780a7740 0x7fc77808df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fc760008b10 tx=0x7fc76000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:29.302 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:29.280+0000 7fc777fff700 1 --2- 172.21.15.42:0/3966138905 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc7780a7740 0x7fc77808df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fc760008b10 tx=0x7fc76000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:29.302 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:29.280+0000 7fc777fff700 1 --2- 172.21.15.42:0/3966138905 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc7780a7740 0x7fc77808df80 secure :-1 s=READY pgs=719 cs=0 l=1 rx=0x7fc760008b10 tx=0x7fc76000fff0).ready entity=mon.2 client_cookie=c8cd77902f969e16 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:48:29.302 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:29.280+0000 7fc777fff700 1 --2- 172.21.15.42:0/3966138905 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc7780a7740 0x7fc77808df80 secure :-1 s=THROTTLE_DONE pgs=719 cs=0 l=1 rx=0x7fc760008b10 tx=0x7fc76000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:29.302 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:29.280+0000 7fc776ffd700 1 -- 172.21.15.42:0/3966138905 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc760011a50 con 0x7fc7780a7740 2019-12-05T06:48:29.303 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:29.280+0000 7fc777fff700 1 --2- 172.21.15.42:0/3966138905 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc7780a7740 0x7fc77808df80 secure :-1 s=THROTTLE_DONE pgs=719 cs=0 l=1 rx=0x7fc760008b10 tx=0x7fc76000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:29.303 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:29.280+0000 7fc777fff700 1 --2- 172.21.15.42:0/3966138905 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc7780a7740 0x7fc77808df80 secure :-1 s=THROTTLE_DONE pgs=719 cs=0 l=1 rx=0x7fc760008b10 tx=0x7fc76000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:29.303 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:29.280+0000 7fc776ffd700 1 -- 172.21.15.42:0/3966138905 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc760011670 con 0x7fc7780a7740 2019-12-05T06:48:29.303 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:29.280+0000 7fc776ffd700 1 -- 172.21.15.42:0/3966138905 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc76000a350 con 0x7fc7780a7740 2019-12-05T06:48:29.303 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:29.280+0000 7fc77e08d700 1 -- 172.21.15.42:0/3966138905 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc7780a7740 msgr2=0x7fc77808df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:29.303 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:29.280+0000 7fc77e08d700 1 --2- 172.21.15.42:0/3966138905 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc7780a7740 0x7fc77808df80 secure :-1 s=READY pgs=719 cs=0 l=1 rx=0x7fc760008b10 tx=0x7fc76000fff0).stop 2019-12-05T06:48:29.303 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:29.284+0000 7fc77e08d700 1 -- 172.21.15.42:0/3966138905 shutdown_connections 2019-12-05T06:48:29.304 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:29.284+0000 7fc77e08d700 1 --2- 172.21.15.42:0/3966138905 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc7780a7740 0x7fc77808df80 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:29.304 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:29.284+0000 7fc77e08d700 1 --2- 172.21.15.42:0/3966138905 >> v2:172.21.15.40:3300/0 conn(0x7fc77807e720 0x7fc77807eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:48:29.304 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:29.284+0000 7fc77e08d700 1 -- 172.21.15.42:0/3966138905 shutdown_connections 2019-12-05T06:48:29.304 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:29.284+0000 7fc77e08d700 1 -- 172.21.15.42:0/3966138905 wait complete. 2019-12-05T06:48:29.304 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:29.284+0000 7fc77e08d700 1 -- 172.21.15.42:0/3966138905 >> 172.21.15.42:0/3966138905 conn(0x7fc7780aabe0 msgr2=0x7fc7780a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:48:29.304 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:29.284+0000 7fc77e08d700 1 Processor -- start 2019-12-05T06:48:29.305 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:29.284+0000 7fc77e08d700 1 -- start start 2019-12-05T06:48:29.305 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:29.288+0000 7fc77e08d700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc7781253c0 0x7fc7781269c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:48:29.305 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:29.288+0000 7fc77e08d700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc77807d9e0 0x7fc77807de10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:48:29.306 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:29.288+0000 7fc77e08d700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7781242f0 con 0x7fc778137e60 2019-12-05T06:48:29.306 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:29.288+0000 7fc77e08d700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc778124670 con 0x7fc7781253c0 2019-12-05T06:48:29.306 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:29.288+0000 7fc77e08d700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc778137bc0 con 0x7fc77807d9e0 2019-12-05T06:48:29.306 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:29.288+0000 7fc7777fe700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc77807d9e0 0x7fc77807de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:29.306 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:29.288+0000 7fc777fff700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc7781253c0 0x7fc7781269c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:29.306 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:29.288+0000 7fc77ce2b700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fc778137e60 0x7fc778138290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:46756/0 (socket says 172.21.15.42:46756) 2019-12-05T06:48:29.306 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:29.288+0000 7fc77ce2b700 1 -- 172.21.15.42:0/1822572490 learned_addr learned my addr 172.21.15.42:0/1822572490 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:48:29.307 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:29.288+0000 7fc774ff9700 1 -- 172.21.15.42:0/1822572490 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 684430838 0 0) 0x7fc7781242f0 con 0x7fc778137e60 2019-12-05T06:48:29.307 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:29.288+0000 7fc774ff9700 1 -- 172.21.15.42:0/1822572490 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7500030b0 con 0x7fc778137e60 2019-12-05T06:48:29.307 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:29.288+0000 7fc7777fe700 1 -- 172.21.15.42:0/1822572490 >> v2:172.21.15.40:3300/0 conn(0x7fc7781253c0 msgr2=0x7fc7781269c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:29.307 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:29.288+0000 7fc7777fe700 1 --2- 172.21.15.42:0/1822572490 >> v2:172.21.15.40:3300/0 conn(0x7fc7781253c0 0x7fc7781269c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:29.307 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:29.288+0000 7fc7777fe700 1 -- 172.21.15.42:0/1822572490 >> v1:172.21.15.40:6789/0 conn(0x7fc778137e60 legacy=0x7fc778138290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:29.307 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:29.288+0000 7fc7777fe700 1 -- 172.21.15.42:0/1822572490 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc778126f10 con 0x7fc77807d9e0 2019-12-05T06:48:29.308 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:29.288+0000 7fc777fff700 1 --2- 172.21.15.42:0/1822572490 >> v2:172.21.15.40:3300/0 conn(0x7fc7781253c0 0x7fc7781269c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:48:29.308 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:29.288+0000 7fc7777fe700 1 --2- 172.21.15.42:0/1822572490 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc77807d9e0 0x7fc77807de10 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fc770002f30 tx=0x7fc770010ca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:29.308 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:29.288+0000 7fc7777fe700 1 --2- 172.21.15.42:0/1822572490 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc77807d9e0 0x7fc77807de10 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fc770002f30 tx=0x7fc770010ca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:29.308 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:29.288+0000 7fc7777fe700 1 --2- 172.21.15.42:0/1822572490 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc77807d9e0 0x7fc77807de10 secure :-1 s=READY pgs=720 cs=0 l=1 rx=0x7fc770002f30 tx=0x7fc770010ca0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:48:29.309 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:29.288+0000 7fc7777fe700 1 --2- 172.21.15.42:0/1822572490 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc77807d9e0 0x7fc77807de10 secure :-1 s=THROTTLE_DONE pgs=720 cs=0 l=1 rx=0x7fc770002f30 tx=0x7fc770010ca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:29.309 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:29.288+0000 7fc774ff9700 1 -- 172.21.15.42:0/1822572490 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc770005840 con 0x7fc77807d9e0 2019-12-05T06:48:29.309 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:29.288+0000 7fc7777fe700 1 --2- 172.21.15.42:0/1822572490 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc77807d9e0 0x7fc77807de10 secure :-1 s=THROTTLE_DONE pgs=720 cs=0 l=1 rx=0x7fc770002f30 tx=0x7fc770010ca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:29.309 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:29.288+0000 7fc7777fe700 1 --2- 172.21.15.42:0/1822572490 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc77807d9e0 0x7fc77807de10 secure :-1 s=THROTTLE_DONE pgs=720 cs=0 l=1 rx=0x7fc770002f30 tx=0x7fc770010ca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:29.309 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:29.288+0000 7fc774ff9700 1 -- 172.21.15.42:0/1822572490 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc770007160 con 0x7fc77807d9e0 2019-12-05T06:48:29.309 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:29.288+0000 7fc77e08d700 1 -- 172.21.15.42:0/1822572490 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc77813ee90 con 0x7fc77807d9e0 2019-12-05T06:48:29.309 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:29.288+0000 7fc774ff9700 1 -- 172.21.15.42:0/1822572490 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc770012360 con 0x7fc77807d9e0 2019-12-05T06:48:29.310 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:29.288+0000 7fc77e08d700 1 -- 172.21.15.42:0/1822572490 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc77813f440 con 0x7fc77807d9e0 2019-12-05T06:48:29.311 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:29.292+0000 7fc7777fe700 1 --2- 172.21.15.42:0/1822572490 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc77807d9e0 0x7fc77807de10 secure :-1 s=THROTTLE_DONE pgs=720 cs=0 l=1 rx=0x7fc770002f30 tx=0x7fc770010ca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:29.311 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:29.292+0000 7fc774ff9700 1 -- 172.21.15.42:0/1822572490 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fc770012670 con 0x7fc77807d9e0 2019-12-05T06:48:29.311 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:29.292+0000 7fc7777fe700 1 --2- 172.21.15.42:0/1822572490 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc77807d9e0 0x7fc77807de10 secure :-1 s=THROTTLE_DONE pgs=720 cs=0 l=1 rx=0x7fc770002f30 tx=0x7fc770010ca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:29.311 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:29.292+0000 7fc774ff9700 1 --2- 172.21.15.42:0/1822572490 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc75003b260 0x7fc75003d770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:48:29.311 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:29.292+0000 7fc774ff9700 1 -- 172.21.15.42:0/1822572490 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 12137+0+0 (secure 0 0 0) 0x7fc77004b7a0 con 0x7fc77807d9e0 2019-12-05T06:48:29.312 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:29.292+0000 7fc777fff700 1 --2- 172.21.15.42:0/1822572490 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc75003b260 0x7fc75003d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:29.312 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:29.292+0000 7fc777fff700 1 --2- 172.21.15.42:0/1822572490 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc75003b260 0x7fc75003d770 crc :-1 s=READY pgs=1356 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:48:29.312 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:29.296+0000 7fc77e08d700 1 -- 172.21.15.42:0/1822572490 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc77805d110 con 0x7fc77807d9e0 2019-12-05T06:48:29.312 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:29.304+0000 7fc7777fe700 1 --2- 172.21.15.42:0/1822572490 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc77807d9e0 0x7fc77807de10 secure :-1 s=THROTTLE_DONE pgs=720 cs=0 l=1 rx=0x7fc770002f30 tx=0x7fc770010ca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:29.312 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:29.304+0000 7fc774ff9700 1 -- 172.21.15.42:0/1822572490 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fc770032820 con 0x7fc77807d9e0 2019-12-05T06:48:29.487 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:29.480+0000 7fc77e08d700 1 -- 172.21.15.42:0/1822572490 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc77805b8c0 con 0x7fc77807d9e0 2019-12-05T06:48:29.488 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:29.480+0000 7fc7777fe700 1 --2- 172.21.15.42:0/1822572490 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc77807d9e0 0x7fc77807de10 secure :-1 s=THROTTLE_DONE pgs=720 cs=0 l=1 rx=0x7fc770002f30 tx=0x7fc770010ca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:29.488 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:29.480+0000 7fc774ff9700 1 -- 172.21.15.42:0/1822572490 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 391 v391) v1 ==== 95+0+1782 (secure 0 0 0) 0x7fc770032a80 con 0x7fc77807d9e0 2019-12-05T06:48:29.490 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:48:29.490 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":391,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":391,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:48:29.380198+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":120,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12847},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12847":{"gid":12847,"name":"a","rank":0,"incarnation":390,"state":"up:reconnect","state_seq":36,"addr":"172.21.15.42:6835/1449317789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1449317789},{"type":"v1","addr":"172.21.15.42:6835","nonce":1449317789}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:48:29.491 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 391 2019-12-05T06:48:29.493 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:29.488+0000 7fc77e08d700 1 -- 172.21.15.42:0/1822572490 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc75003b260 msgr2=0x7fc75003d770 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:29.493 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:29.488+0000 7fc77e08d700 1 --2- 172.21.15.42:0/1822572490 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc75003b260 0x7fc75003d770 crc :-1 s=READY pgs=1356 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:29.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:29.488+0000 7fc77e08d700 1 -- 172.21.15.42:0/1822572490 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc77807d9e0 msgr2=0x7fc77807de10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:29.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:29.488+0000 7fc77e08d700 1 --2- 172.21.15.42:0/1822572490 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc77807d9e0 0x7fc77807de10 secure :-1 s=READY pgs=720 cs=0 l=1 rx=0x7fc770002f30 tx=0x7fc770010ca0).stop 2019-12-05T06:48:29.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:29.488+0000 7fc77e08d700 1 -- 172.21.15.42:0/1822572490 shutdown_connections 2019-12-05T06:48:29.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:29.488+0000 7fc77e08d700 1 --2- 172.21.15.42:0/1822572490 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc75003b260 0x7fc75003d770 unknown :-1 s=CLOSED pgs=1356 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:29.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:29.488+0000 7fc77e08d700 1 --2- 172.21.15.42:0/1822572490 >> v2:172.21.15.40:3300/0 conn(0x7fc7781253c0 0x7fc7781269c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:29.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:29.488+0000 7fc77e08d700 1 --2- 172.21.15.42:0/1822572490 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc77807d9e0 0x7fc77807de10 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:29.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:29.488+0000 7fc77e08d700 1 -- 172.21.15.42:0/1822572490 shutdown_connections 2019-12-05T06:48:29.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:29.488+0000 7fc77e08d700 1 -- 172.21.15.42:0/1822572490 wait complete. 2019-12-05T06:48:29.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:29.488+0000 7fc77e08d700 1 -- 172.21.15.42:0/1822572490 >> 172.21.15.42:0/1822572490 conn(0x7fc7780a9790 msgr2=0x7fc77809f380 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:48:29.502 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2019-12-05T06:48:31.504 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:48:31.615 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:31.600+0000 7fcb162f1700 1 Processor -- start 2019-12-05T06:48:31.615 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:31.600+0000 7fcb162f1700 1 -- start start 2019-12-05T06:48:31.615 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:31.604+0000 7fcb162f1700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fcb1008df80 0x7fcb1009f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:48:31.615 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:31.604+0000 7fcb162f1700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcb1009f830 0x7fcb1014efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:48:31.615 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:31.604+0000 7fcb162f1700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb1009fc60 con 0x7fcb100a7740 2019-12-05T06:48:31.616 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:31.604+0000 7fcb162f1700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb100a0b00 con 0x7fcb1008df80 2019-12-05T06:48:31.616 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:31.604+0000 7fcb162f1700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb1008e5b0 con 0x7fcb1009f830 2019-12-05T06:48:31.616 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:31.604+0000 7fcb0ffff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fcb1008df80 0x7fcb1009f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:31.616 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:31.604+0000 7fcb1488e700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fcb100a7740 0x7fcb1007e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:37324/0 (socket says 172.21.15.42:37324) 2019-12-05T06:48:31.616 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:31.604+0000 7fcb1488e700 1 -- 172.21.15.42:0/2014556032 learned_addr learned my addr 172.21.15.42:0/2014556032 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:48:31.616 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:31.604+0000 7fcb1508f700 1 --2- 172.21.15.42:0/2014556032 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcb1009f830 0x7fcb1014efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:31.616 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:31.604+0000 7fcb0ffff700 1 -- 172.21.15.42:0/2014556032 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcb1009f830 msgr2=0x7fcb1014efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:48:31.617 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:31.604+0000 7fcb0ffff700 1 --2- 172.21.15.42:0/2014556032 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcb1009f830 0x7fcb1014efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:48:31.617 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:31.604+0000 7fcb0ffff700 1 -- 172.21.15.42:0/2014556032 >> v1:172.21.15.42:6789/0 conn(0x7fcb100a7740 legacy=0x7fcb1007e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:31.617 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:31.604+0000 7fcb0f7fe700 1 -- 172.21.15.42:0/2014556032 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2445603367 0 0) 0x7fcb1009fc60 con 0x7fcb100a7740 2019-12-05T06:48:31.617 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:31.604+0000 7fcb0ffff700 1 -- 172.21.15.42:0/2014556032 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb100a0710 con 0x7fcb1008df80 2019-12-05T06:48:31.617 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:31.604+0000 7fcb0ffff700 1 --2- 172.21.15.42:0/2014556032 >> v2:172.21.15.42:3300/0 conn(0x7fcb1008df80 0x7fcb1009f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fcaf8008790 tx=0x7fcaf800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:31.617 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:31.604+0000 7fcb0ffff700 1 --2- 172.21.15.42:0/2014556032 >> v2:172.21.15.42:3300/0 conn(0x7fcb1008df80 0x7fcb1009f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fcaf8008790 tx=0x7fcaf800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:31.618 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:31.604+0000 7fcb0ffff700 1 --2- 172.21.15.42:0/2014556032 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcb1008df80 0x7fcb1009f2e0 secure :-1 s=READY pgs=3238 cs=0 l=1 rx=0x7fcaf8008790 tx=0x7fcaf800fc70).ready entity=mon.1 client_cookie=56751216e61df61 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:48:31.618 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:31.604+0000 7fcb0ffff700 1 --2- 172.21.15.42:0/2014556032 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcb1008df80 0x7fcb1009f2e0 secure :-1 s=THROTTLE_DONE pgs=3238 cs=0 l=1 rx=0x7fcaf8008790 tx=0x7fcaf800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:31.618 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:31.604+0000 7fcb0ffff700 1 --2- 172.21.15.42:0/2014556032 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcb1008df80 0x7fcb1009f2e0 secure :-1 s=THROTTLE_DONE pgs=3238 cs=0 l=1 rx=0x7fcaf8008790 tx=0x7fcaf800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:31.618 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:31.604+0000 7fcb0f7fe700 1 -- 172.21.15.42:0/2014556032 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcaf8008bb0 con 0x7fcb1008df80 2019-12-05T06:48:31.618 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:31.604+0000 7fcb0ffff700 1 --2- 172.21.15.42:0/2014556032 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcb1008df80 0x7fcb1009f2e0 secure :-1 s=THROTTLE_DONE pgs=3238 cs=0 l=1 rx=0x7fcaf8008790 tx=0x7fcaf800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:31.618 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:31.608+0000 7fcb0f7fe700 1 -- 172.21.15.42:0/2014556032 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcaf80087d0 con 0x7fcb1008df80 2019-12-05T06:48:31.618 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:31.608+0000 7fcb0f7fe700 1 -- 172.21.15.42:0/2014556032 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcaf8011860 con 0x7fcb1008df80 2019-12-05T06:48:31.619 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:31.608+0000 7fcb162f1700 1 -- 172.21.15.42:0/2014556032 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcb1008df80 msgr2=0x7fcb1009f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:31.619 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:31.608+0000 7fcb162f1700 1 --2- 172.21.15.42:0/2014556032 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcb1008df80 0x7fcb1009f2e0 secure :-1 s=READY pgs=3238 cs=0 l=1 rx=0x7fcaf8008790 tx=0x7fcaf800fc70).stop 2019-12-05T06:48:31.619 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:31.608+0000 7fcb162f1700 1 -- 172.21.15.42:0/2014556032 shutdown_connections 2019-12-05T06:48:31.619 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:31.608+0000 7fcb162f1700 1 --2- 172.21.15.42:0/2014556032 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcb1009f830 0x7fcb1014efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:48:31.619 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:31.608+0000 7fcb162f1700 1 --2- 172.21.15.42:0/2014556032 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcb1008df80 0x7fcb1009f2e0 unknown :-1 s=CLOSED pgs=3238 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:31.619 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:31.608+0000 7fcb162f1700 1 -- 172.21.15.42:0/2014556032 shutdown_connections 2019-12-05T06:48:31.620 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:31.608+0000 7fcb162f1700 1 -- 172.21.15.42:0/2014556032 wait complete. 2019-12-05T06:48:31.620 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:31.608+0000 7fcb162f1700 1 -- 172.21.15.42:0/2014556032 >> 172.21.15.42:0/2014556032 conn(0x7fcb100aabe0 msgr2=0x7fcb100a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:48:31.623 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:31.612+0000 7fcb162f1700 1 Processor -- start 2019-12-05T06:48:31.623 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:31.612+0000 7fcb162f1700 1 -- start start 2019-12-05T06:48:31.624 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:31.612+0000 7fcb162f1700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fcb10116500 0x7fcb1007ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:48:31.624 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:31.612+0000 7fcb162f1700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fcb1007cd20 0x7fcb100a65d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:48:31.624 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:31.612+0000 7fcb162f1700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcb10133610 0x7fcb10133a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:48:31.624 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:31.612+0000 7fcb162f1700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb1007c1c0 con 0x7fcb1007cd20 2019-12-05T06:48:31.624 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:31.612+0000 7fcb162f1700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb100a7110 con 0x7fcb10116500 2019-12-05T06:48:31.624 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:31.612+0000 7fcb162f1700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb1007c460 con 0x7fcb10133610 2019-12-05T06:48:31.625 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:31.612+0000 7fcb1488e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fcb10116500 0x7fcb1007ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:31.625 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:31.612+0000 7fcb1508f700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcb10133610 0x7fcb10133a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:31.625 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:31.612+0000 7fcb0ffff700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fcb1007cd20 0x7fcb100a65d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:31.625 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:31.612+0000 7fcb1488e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fcb10116500 0x7fcb1007ba20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:37292/0 (socket says 172.21.15.42:37292) 2019-12-05T06:48:31.625 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:31.612+0000 7fcb1508f700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcb10133610 0x7fcb10133a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:37762/0 (socket says 172.21.15.42:37762) 2019-12-05T06:48:31.625 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:31.612+0000 7fcb1508f700 1 -- 172.21.15.42:0/2374420566 learned_addr learned my addr 172.21.15.42:0/2374420566 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:48:31.625 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:31.612+0000 7fcb1488e700 1 -- 172.21.15.42:0/2374420566 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcb10133610 msgr2=0x7fcb10133a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:31.626 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:31.612+0000 7fcb1488e700 1 --2- 172.21.15.42:0/2374420566 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcb10133610 0x7fcb10133a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:31.626 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:31.612+0000 7fcb1488e700 1 -- 172.21.15.42:0/2374420566 >> v2:172.21.15.40:3300/0 conn(0x7fcb1007cd20 msgr2=0x7fcb100a65d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:31.626 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:31.612+0000 7fcb1488e700 1 --2- 172.21.15.42:0/2374420566 >> v2:172.21.15.40:3300/0 conn(0x7fcb1007cd20 0x7fcb100a65d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:31.626 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:31.612+0000 7fcb1488e700 1 -- 172.21.15.42:0/2374420566 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb1007d150 con 0x7fcb10116500 2019-12-05T06:48:31.626 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:31.612+0000 7fcb0ffff700 1 --2- 172.21.15.42:0/2374420566 >> v2:172.21.15.40:3300/0 conn(0x7fcb1007cd20 0x7fcb100a65d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:48:31.626 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:31.612+0000 7fcb1488e700 1 --2- 172.21.15.42:0/2374420566 >> v2:172.21.15.42:3300/0 conn(0x7fcb10116500 0x7fcb1007ba20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fcb08004720 tx=0x7fcb08010ca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:31.626 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:31.612+0000 7fcb1488e700 1 --2- 172.21.15.42:0/2374420566 >> v2:172.21.15.42:3300/0 conn(0x7fcb10116500 0x7fcb1007ba20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fcb08004720 tx=0x7fcb08010ca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:31.627 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:31.612+0000 7fcb1488e700 1 --2- 172.21.15.42:0/2374420566 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcb10116500 0x7fcb1007ba20 secure :-1 s=READY pgs=3239 cs=0 l=1 rx=0x7fcb08004720 tx=0x7fcb08010ca0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:48:31.627 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:31.612+0000 7fcb1488e700 1 --2- 172.21.15.42:0/2374420566 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcb10116500 0x7fcb1007ba20 secure :-1 s=THROTTLE_DONE pgs=3239 cs=0 l=1 rx=0x7fcb08004720 tx=0x7fcb08010ca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:31.627 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:31.612+0000 7fcb1488e700 1 --2- 172.21.15.42:0/2374420566 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcb10116500 0x7fcb1007ba20 secure :-1 s=THROTTLE_DONE pgs=3239 cs=0 l=1 rx=0x7fcb08004720 tx=0x7fcb08010ca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:31.627 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:31.612+0000 7fcb0d7fa700 1 -- 172.21.15.42:0/2374420566 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcb08004b90 con 0x7fcb10116500 2019-12-05T06:48:31.627 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:31.612+0000 7fcb1488e700 1 --2- 172.21.15.42:0/2374420566 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcb10116500 0x7fcb1007ba20 secure :-1 s=THROTTLE_DONE pgs=3239 cs=0 l=1 rx=0x7fcb08004720 tx=0x7fcb08010ca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:31.627 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:31.612+0000 7fcb0d7fa700 1 -- 172.21.15.42:0/2374420566 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb080047b0 con 0x7fcb10116500 2019-12-05T06:48:31.627 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:31.612+0000 7fcb162f1700 1 -- 172.21.15.42:0/2374420566 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb100a6b20 con 0x7fcb10116500 2019-12-05T06:48:31.628 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:31.612+0000 7fcb0d7fa700 1 -- 172.21.15.42:0/2374420566 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcb08012850 con 0x7fcb10116500 2019-12-05T06:48:31.628 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:31.612+0000 7fcb162f1700 1 -- 172.21.15.42:0/2374420566 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb1008eb70 con 0x7fcb10116500 2019-12-05T06:48:31.628 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:31.616+0000 7fcb1488e700 1 --2- 172.21.15.42:0/2374420566 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcb10116500 0x7fcb1007ba20 secure :-1 s=THROTTLE_DONE pgs=3239 cs=0 l=1 rx=0x7fcb08004720 tx=0x7fcb08010ca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:31.628 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:31.616+0000 7fcb0d7fa700 1 -- 172.21.15.42:0/2374420566 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fcb08012b60 con 0x7fcb10116500 2019-12-05T06:48:31.628 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:31.616+0000 7fcb1488e700 1 --2- 172.21.15.42:0/2374420566 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcb10116500 0x7fcb1007ba20 secure :-1 s=THROTTLE_DONE pgs=3239 cs=0 l=1 rx=0x7fcb08004720 tx=0x7fcb08010ca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:31.628 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:31.616+0000 7fcb0d7fa700 1 --2- 172.21.15.42:0/2374420566 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fcaf403aec0 0x7fcaf403d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:48:31.628 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:31.616+0000 7fcb0d7fa700 1 -- 172.21.15.42:0/2374420566 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 12137+0+0 (secure 0 0 0) 0x7fcb0804b4e0 con 0x7fcb10116500 2019-12-05T06:48:31.629 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:31.616+0000 7fcb0ffff700 1 --2- 172.21.15.42:0/2374420566 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fcaf403aec0 0x7fcaf403d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:31.629 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:31.616+0000 7fcb0ffff700 1 --2- 172.21.15.42:0/2374420566 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fcaf403aec0 0x7fcaf403d3d0 crc :-1 s=READY pgs=1357 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:48:31.629 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:31.616+0000 7fcb162f1700 1 -- 172.21.15.42:0/2374420566 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb1005d110 con 0x7fcb10116500 2019-12-05T06:48:31.629 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:31.624+0000 7fcb1488e700 1 --2- 172.21.15.42:0/2374420566 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcb10116500 0x7fcb1007ba20 secure :-1 s=THROTTLE_DONE pgs=3239 cs=0 l=1 rx=0x7fcb08004720 tx=0x7fcb08010ca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:31.630 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:31.624+0000 7fcb0d7fa700 1 -- 172.21.15.42:0/2374420566 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fcb08032560 con 0x7fcb10116500 2019-12-05T06:48:31.806 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:31.800+0000 7fcb162f1700 1 -- 172.21.15.42:0/2374420566 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcb1005b8c0 con 0x7fcb10116500 2019-12-05T06:48:31.807 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:31.800+0000 7fcb1488e700 1 --2- 172.21.15.42:0/2374420566 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcb10116500 0x7fcb1007ba20 secure :-1 s=THROTTLE_DONE pgs=3239 cs=0 l=1 rx=0x7fcb08004720 tx=0x7fcb08010ca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:31.807 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:31.800+0000 7fcb0d7fa700 1 -- 172.21.15.42:0/2374420566 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 393 v393) v1 ==== 95+0+1779 (secure 0 0 0) 0x7fcb080327c0 con 0x7fcb10116500 2019-12-05T06:48:31.808 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 393 2019-12-05T06:48:31.808 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:48:31.808 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":393,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":393,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:48:31.390793+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":120,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12847},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12847":{"gid":12847,"name":"a","rank":0,"incarnation":390,"state":"up:active","state_seq":38,"addr":"172.21.15.42:6835/1449317789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1449317789},{"type":"v1","addr":"172.21.15.42:6835","nonce":1449317789}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:48:31.810 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:31.804+0000 7fcb162f1700 1 -- 172.21.15.42:0/2374420566 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fcaf403aec0 msgr2=0x7fcaf403d3d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:31.810 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:31.804+0000 7fcb162f1700 1 --2- 172.21.15.42:0/2374420566 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fcaf403aec0 0x7fcaf403d3d0 crc :-1 s=READY pgs=1357 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:31.810 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:31.804+0000 7fcb162f1700 1 -- 172.21.15.42:0/2374420566 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcb10116500 msgr2=0x7fcb1007ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:31.810 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:31.804+0000 7fcb162f1700 1 --2- 172.21.15.42:0/2374420566 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcb10116500 0x7fcb1007ba20 secure :-1 s=READY pgs=3239 cs=0 l=1 rx=0x7fcb08004720 tx=0x7fcb08010ca0).stop 2019-12-05T06:48:31.811 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:31.804+0000 7fcb162f1700 1 -- 172.21.15.42:0/2374420566 shutdown_connections 2019-12-05T06:48:31.811 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:31.804+0000 7fcb162f1700 1 --2- 172.21.15.42:0/2374420566 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fcaf403aec0 0x7fcaf403d3d0 unknown :-1 s=CLOSED pgs=1357 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:31.811 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:31.804+0000 7fcb162f1700 1 --2- 172.21.15.42:0/2374420566 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcb10133610 0x7fcb10133a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:31.811 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:31.804+0000 7fcb162f1700 1 --2- 172.21.15.42:0/2374420566 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcb10116500 0x7fcb1007ba20 unknown :-1 s=CLOSED pgs=3239 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:31.811 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:31.804+0000 7fcb162f1700 1 --2- 172.21.15.42:0/2374420566 >> v2:172.21.15.40:3300/0 conn(0x7fcb1007cd20 0x7fcb100a65d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:31.811 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:31.804+0000 7fcb162f1700 1 -- 172.21.15.42:0/2374420566 shutdown_connections 2019-12-05T06:48:31.811 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:31.804+0000 7fcb162f1700 1 -- 172.21.15.42:0/2374420566 wait complete. 2019-12-05T06:48:31.812 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:31.804+0000 7fcb162f1700 1 -- 172.21.15.42:0/2374420566 >> 172.21.15.42:0/2374420566 conn(0x7fcb1008fde0 msgr2=0x7fcb10059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:48:31.820 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a has gained rank=0, replacing gid=12429 2019-12-05T06:48:31.820 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 3 secs before reviving mds.b 2019-12-05T06:48:32.656 INFO:tasks.workunit.client.0.smithi042.stderr:+ cd ../.. 2019-12-05T06:48:32.656 INFO:tasks.workunit.client.0.smithi042.stderr:+ ver=v0.6 2019-12-05T06:48:32.657 INFO:tasks.workunit.client.0.smithi042.stderr:+ echo checking v0.6 2019-12-05T06:48:32.657 INFO:tasks.workunit.client.0.smithi042.stderr:+ cd .snap/v0.6 2019-12-05T06:48:32.657 INFO:tasks.workunit.client.0.smithi042.stdout:checking v0.6 2019-12-05T06:48:32.660 INFO:tasks.workunit.client.0.smithi042.stderr:+ git diff --exit-code 2019-12-05T06:48:34.821 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.b 2019-12-05T06:48:34.821 INFO:tasks.ceph.mds.b:Restarting daemon 2019-12-05T06:48:34.821 INFO:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2019-12-05T06:48:34.834 INFO:tasks.ceph.mds.b:Started 2019-12-05T06:48:34.834 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:48:34.896 INFO:tasks.ceph.mds.b.smithi040.stdout:starting mds.b at 2019-12-05T06:48:34.951 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:34.939+0000 7f3de6e5c700 1 Processor -- start 2019-12-05T06:48:34.951 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:34.939+0000 7f3de6e5c700 1 -- start start 2019-12-05T06:48:34.951 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:34.939+0000 7f3de6e5c700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f3de00a7740 0x7f3de008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:48:34.951 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:34.939+0000 7f3de6e5c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3de007e720 0x7f3de007eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:48:34.952 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:34.939+0000 7f3de6e5c700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3de00a0710 con 0x7f3de009f2e0 2019-12-05T06:48:34.952 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:34.939+0000 7f3de6e5c700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3de00a0b00 con 0x7f3de00a7740 2019-12-05T06:48:34.952 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:34.939+0000 7f3de6e5c700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3de008e5b0 con 0x7f3de007e720 2019-12-05T06:48:34.952 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:34.939+0000 7f3de4bf8700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3de007e720 0x7f3de007eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:34.952 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:34.939+0000 7f3de5bfa700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f3de009f2e0 0x7f3de009f710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:37372/0 (socket says 172.21.15.42:37372) 2019-12-05T06:48:34.952 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:34.939+0000 7f3de5bfa700 1 -- 172.21.15.42:0/3292358079 learned_addr learned my addr 172.21.15.42:0/3292358079 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:48:34.953 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:34.943+0000 7f3de53f9700 1 --2- 172.21.15.42:0/3292358079 >> v2:172.21.15.40:3300/0 conn(0x7f3de00a7740 0x7f3de008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:34.953 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:34.943+0000 7f3dd7fff700 1 -- 172.21.15.42:0/3292358079 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3176874423 0 0) 0x7f3de00a0710 con 0x7f3de009f2e0 2019-12-05T06:48:34.953 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:34.943+0000 7f3dd7fff700 1 -- 172.21.15.42:0/3292358079 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3dcc0030b0 con 0x7f3de009f2e0 2019-12-05T06:48:34.953 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:34.943+0000 7f3de4bf8700 1 -- 172.21.15.42:0/3292358079 >> v2:172.21.15.40:3300/0 conn(0x7f3de00a7740 msgr2=0x7f3de008df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:48:34.953 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:34.943+0000 7f3de4bf8700 1 --2- 172.21.15.42:0/3292358079 >> v2:172.21.15.40:3300/0 conn(0x7f3de00a7740 0x7f3de008df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:48:34.953 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:34.943+0000 7f3de4bf8700 1 -- 172.21.15.42:0/3292358079 >> v1:172.21.15.42:6789/0 conn(0x7f3de009f2e0 legacy=0x7f3de009f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:34.953 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:34.943+0000 7f3de4bf8700 1 -- 172.21.15.42:0/3292358079 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3de007f0a0 con 0x7f3de007e720 2019-12-05T06:48:34.954 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:34.943+0000 7f3de4bf8700 1 --2- 172.21.15.42:0/3292358079 >> v2:172.21.15.42:3300/0 conn(0x7f3de007e720 0x7f3de007eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f3dd0002f30 tx=0x7f3dd000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:34.954 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:34.943+0000 7f3de53f9700 1 --2- 172.21.15.42:0/3292358079 >> v2:172.21.15.40:3300/0 conn(0x7f3de00a7740 0x7f3de008df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:48:34.954 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:34.943+0000 7f3de4bf8700 1 --2- 172.21.15.42:0/3292358079 >> v2:172.21.15.42:3300/0 conn(0x7f3de007e720 0x7f3de007eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f3dd0002f30 tx=0x7f3dd000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:34.954 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:34.943+0000 7f3de4bf8700 1 --2- 172.21.15.42:0/3292358079 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3de007e720 0x7f3de007eb50 secure :-1 s=READY pgs=3242 cs=0 l=1 rx=0x7f3dd0002f30 tx=0x7f3dd000fc70).ready entity=mon.1 client_cookie=bd94fa498926c566 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:48:34.954 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:34.943+0000 7f3de4bf8700 1 --2- 172.21.15.42:0/3292358079 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3de007e720 0x7f3de007eb50 secure :-1 s=THROTTLE_DONE pgs=3242 cs=0 l=1 rx=0x7f3dd0002f30 tx=0x7f3dd000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:34.955 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:34.943+0000 7f3dd7fff700 1 -- 172.21.15.42:0/3292358079 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3dd00112b0 con 0x7f3de007e720 2019-12-05T06:48:34.955 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:34.943+0000 7f3de4bf8700 1 --2- 172.21.15.42:0/3292358079 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3de007e720 0x7f3de007eb50 secure :-1 s=THROTTLE_DONE pgs=3242 cs=0 l=1 rx=0x7f3dd0002f30 tx=0x7f3dd000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:34.955 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:34.943+0000 7f3dd7fff700 1 -- 172.21.15.42:0/3292358079 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3dd0008bc0 con 0x7f3de007e720 2019-12-05T06:48:34.955 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:34.943+0000 7f3de4bf8700 1 --2- 172.21.15.42:0/3292358079 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3de007e720 0x7f3de007eb50 secure :-1 s=THROTTLE_DONE pgs=3242 cs=0 l=1 rx=0x7f3dd0002f30 tx=0x7f3dd000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:34.955 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:34.943+0000 7f3de6e5c700 1 -- 172.21.15.42:0/3292358079 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3de007e720 msgr2=0x7f3de007eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:34.963 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:34.943+0000 7f3dd7fff700 1 -- 172.21.15.42:0/3292358079 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3dd0011890 con 0x7f3de007e720 2019-12-05T06:48:34.963 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:34.943+0000 7f3de6e5c700 1 --2- 172.21.15.42:0/3292358079 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3de007e720 0x7f3de007eb50 secure :-1 s=READY pgs=3242 cs=0 l=1 rx=0x7f3dd0002f30 tx=0x7f3dd000fc70).stop 2019-12-05T06:48:34.963 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:34.943+0000 7f3de6e5c700 1 -- 172.21.15.42:0/3292358079 shutdown_connections 2019-12-05T06:48:34.963 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:34.943+0000 7f3de6e5c700 1 --2- 172.21.15.42:0/3292358079 >> v2:172.21.15.40:3300/0 conn(0x7f3de00a7740 0x7f3de008df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:48:34.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:34.943+0000 7f3de6e5c700 1 --2- 172.21.15.42:0/3292358079 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3de007e720 0x7f3de007eb50 unknown :-1 s=CLOSED pgs=3242 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:34.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:34.943+0000 7f3de6e5c700 1 -- 172.21.15.42:0/3292358079 shutdown_connections 2019-12-05T06:48:34.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:34.943+0000 7f3de6e5c700 1 -- 172.21.15.42:0/3292358079 wait complete. 2019-12-05T06:48:34.978 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:34.943+0000 7f3de6e5c700 1 -- 172.21.15.42:0/3292358079 >> 172.21.15.42:0/3292358079 conn(0x7f3de00aabe0 msgr2=0x7f3de00a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:48:34.981 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:34.947+0000 7f3de6e5c700 1 Processor -- start 2019-12-05T06:48:34.981 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:34.947+0000 7f3de6e5c700 1 -- start start 2019-12-05T06:48:34.982 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:34.947+0000 7f3de6e5c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3de0133ec0 0x7f3de00a6d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:48:34.982 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:34.947+0000 7f3de6e5c700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3de01161c0 0x7f3de013d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:48:34.982 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:34.947+0000 7f3de6e5c700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3de007b4c0 con 0x7f3de00a6070 2019-12-05T06:48:34.982 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:34.947+0000 7f3de6e5c700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3de00a6740 con 0x7f3de0133ec0 2019-12-05T06:48:34.982 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:34.947+0000 7f3de6e5c700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3de00a69e0 con 0x7f3de01161c0 2019-12-05T06:48:34.982 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:34.947+0000 7f3de4bf8700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3de0133ec0 0x7f3de00a6d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:34.983 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:34.947+0000 7f3de53f9700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f3de00a6070 0x7f3de01336c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:37374/0 (socket says 172.21.15.42:37374) 2019-12-05T06:48:34.983 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:34.947+0000 7f3de53f9700 1 -- 172.21.15.42:0/1664253464 learned_addr learned my addr 172.21.15.42:0/1664253464 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:48:34.992 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:34.947+0000 7f3de5bfa700 1 --2- 172.21.15.42:0/1664253464 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3de01161c0 0x7f3de013d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:34.993 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:34.947+0000 7f3dd5ffb700 1 -- 172.21.15.42:0/1664253464 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1569420648 0 0) 0x7f3de007b4c0 con 0x7f3de00a6070 2019-12-05T06:48:34.993 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:34.947+0000 7f3dd5ffb700 1 -- 172.21.15.42:0/1664253464 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3dc00030b0 con 0x7f3de00a6070 2019-12-05T06:48:34.993 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:34.947+0000 7f3de4bf8700 1 -- 172.21.15.42:0/1664253464 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3de01161c0 msgr2=0x7f3de013d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:34.993 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:34.947+0000 7f3de4bf8700 1 --2- 172.21.15.42:0/1664253464 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3de01161c0 0x7f3de013d9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:34.993 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:34.947+0000 7f3de4bf8700 1 -- 172.21.15.42:0/1664253464 >> v1:172.21.15.42:6789/0 conn(0x7f3de00a6070 legacy=0x7f3de01336c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:34.993 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:34.947+0000 7f3de4bf8700 1 -- 172.21.15.42:0/1664253464 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3de01165f0 con 0x7f3de0133ec0 2019-12-05T06:48:34.993 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:34.947+0000 7f3de4bf8700 1 --2- 172.21.15.42:0/1664253464 >> v2:172.21.15.42:3300/0 conn(0x7f3de0133ec0 0x7f3de00a6d90 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f3de00a6900 tx=0x7f3dd0008bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:34.994 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:34.947+0000 7f3de4bf8700 1 --2- 172.21.15.42:0/1664253464 >> v2:172.21.15.42:3300/0 conn(0x7f3de0133ec0 0x7f3de00a6d90 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f3de00a6900 tx=0x7f3dd0008bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:34.994 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:34.947+0000 7f3de4bf8700 1 --2- 172.21.15.42:0/1664253464 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3de0133ec0 0x7f3de00a6d90 secure :-1 s=READY pgs=3244 cs=0 l=1 rx=0x7f3de00a6900 tx=0x7f3dd0008bc0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:48:34.994 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:34.947+0000 7f3de4bf8700 1 --2- 172.21.15.42:0/1664253464 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3de0133ec0 0x7f3de00a6d90 secure :-1 s=THROTTLE_DONE pgs=3244 cs=0 l=1 rx=0x7f3de00a6900 tx=0x7f3dd0008bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:34.994 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:34.947+0000 7f3de4bf8700 1 --2- 172.21.15.42:0/1664253464 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3de0133ec0 0x7f3de00a6d90 secure :-1 s=THROTTLE_DONE pgs=3244 cs=0 l=1 rx=0x7f3de00a6900 tx=0x7f3dd0008bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:34.994 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:34.947+0000 7f3de4bf8700 1 --2- 172.21.15.42:0/1664253464 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3de0133ec0 0x7f3de00a6d90 secure :-1 s=THROTTLE_DONE pgs=3244 cs=0 l=1 rx=0x7f3de00a6900 tx=0x7f3dd0008bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:34.994 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:34.947+0000 7f3dd5ffb700 1 -- 172.21.15.42:0/1664253464 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3dd0007340 con 0x7f3de0133ec0 2019-12-05T06:48:34.995 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:34.947+0000 7f3dd5ffb700 1 -- 172.21.15.42:0/1664253464 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3dd00114b0 con 0x7f3de0133ec0 2019-12-05T06:48:34.995 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:34.947+0000 7f3dd5ffb700 1 -- 172.21.15.42:0/1664253464 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3dd000f360 con 0x7f3de0133ec0 2019-12-05T06:48:34.995 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:34.947+0000 7f3de6e5c700 1 -- 172.21.15.42:0/1664253464 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3de013df10 con 0x7f3de0133ec0 2019-12-05T06:48:34.995 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:34.947+0000 7f3de6e5c700 1 -- 172.21.15.42:0/1664253464 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3de013e4c0 con 0x7f3de0133ec0 2019-12-05T06:48:34.995 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:34.947+0000 7f3de4bf8700 1 --2- 172.21.15.42:0/1664253464 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3de0133ec0 0x7f3de00a6d90 secure :-1 s=THROTTLE_DONE pgs=3244 cs=0 l=1 rx=0x7f3de00a6900 tx=0x7f3dd0008bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:34.995 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:34.951+0000 7f3dd5ffb700 1 -- 172.21.15.42:0/1664253464 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f3dd0005540 con 0x7f3de0133ec0 2019-12-05T06:48:34.995 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:34.951+0000 7f3de4bf8700 1 --2- 172.21.15.42:0/1664253464 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3de0133ec0 0x7f3de00a6d90 secure :-1 s=THROTTLE_DONE pgs=3244 cs=0 l=1 rx=0x7f3de00a6900 tx=0x7f3dd0008bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:34.996 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:34.951+0000 7f3dd5ffb700 1 --2- 172.21.15.42:0/1664253464 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3dc003b2b0 0x7f3dc003d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:48:34.996 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:34.951+0000 7f3dd5ffb700 1 -- 172.21.15.42:0/1664253464 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 12137+0+0 (secure 0 0 0) 0x7f3dd004b340 con 0x7f3de0133ec0 2019-12-05T06:48:34.996 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:34.951+0000 7f3de53f9700 1 --2- 172.21.15.42:0/1664253464 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3dc003b2b0 0x7f3dc003d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:34.996 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:34.951+0000 7f3de53f9700 1 --2- 172.21.15.42:0/1664253464 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3dc003b2b0 0x7f3dc003d7c0 crc :-1 s=READY pgs=1358 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:48:34.996 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:34.955+0000 7f3de6e5c700 1 -- 172.21.15.42:0/1664253464 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3de005d110 con 0x7f3de0133ec0 2019-12-05T06:48:34.996 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:34.959+0000 7f3de4bf8700 1 --2- 172.21.15.42:0/1664253464 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3de0133ec0 0x7f3de00a6d90 secure :-1 s=THROTTLE_DONE pgs=3244 cs=0 l=1 rx=0x7f3de00a6900 tx=0x7f3dd0008bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:34.996 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:34.959+0000 7f3dd5ffb700 1 -- 172.21.15.42:0/1664253464 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f3dd00323c0 con 0x7f3de0133ec0 2019-12-05T06:48:35.193 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:35.179+0000 7f3de6e5c700 1 -- 172.21.15.42:0/1664253464 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3de005b8c0 con 0x7f3de0133ec0 2019-12-05T06:48:35.194 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:35.183+0000 7f3de4bf8700 1 --2- 172.21.15.42:0/1664253464 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3de0133ec0 0x7f3de00a6d90 secure :-1 s=THROTTLE_DONE pgs=3244 cs=0 l=1 rx=0x7f3de00a6900 tx=0x7f3dd0008bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:35.194 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:35.183+0000 7f3dd5ffb700 1 -- 172.21.15.42:0/1664253464 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 393 v393) v1 ==== 95+0+1779 (secure 0 0 0) 0x7f3dd0032620 con 0x7f3de0133ec0 2019-12-05T06:48:35.194 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 393 2019-12-05T06:48:35.194 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:35.183+0000 7f3de6e5c700 1 -- 172.21.15.42:0/1664253464 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3dc003b2b0 msgr2=0x7f3dc003d7c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:35.194 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:35.183+0000 7f3de6e5c700 1 --2- 172.21.15.42:0/1664253464 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3dc003b2b0 0x7f3dc003d7c0 crc :-1 s=READY pgs=1358 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:35.194 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:35.183+0000 7f3de6e5c700 1 -- 172.21.15.42:0/1664253464 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3de0133ec0 msgr2=0x7f3de00a6d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:35.195 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:35.183+0000 7f3de6e5c700 1 --2- 172.21.15.42:0/1664253464 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3de0133ec0 0x7f3de00a6d90 secure :-1 s=READY pgs=3244 cs=0 l=1 rx=0x7f3de00a6900 tx=0x7f3dd0008bc0).stop 2019-12-05T06:48:35.195 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:35.183+0000 7f3de6e5c700 1 -- 172.21.15.42:0/1664253464 shutdown_connections 2019-12-05T06:48:35.195 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:35.183+0000 7f3de6e5c700 1 --2- 172.21.15.42:0/1664253464 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3dc003b2b0 0x7f3dc003d7c0 unknown :-1 s=CLOSED pgs=1358 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:35.195 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:35.183+0000 7f3de6e5c700 1 --2- 172.21.15.42:0/1664253464 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3de01161c0 0x7f3de013d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:35.195 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:35.183+0000 7f3de6e5c700 1 --2- 172.21.15.42:0/1664253464 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3de0133ec0 0x7f3de00a6d90 unknown :-1 s=CLOSED pgs=3244 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:35.195 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:35.183+0000 7f3de6e5c700 1 -- 172.21.15.42:0/1664253464 shutdown_connections 2019-12-05T06:48:35.195 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:35.183+0000 7f3de6e5c700 1 -- 172.21.15.42:0/1664253464 wait complete. 2019-12-05T06:48:35.196 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:35.183+0000 7f3de6e5c700 1 -- 172.21.15.42:0/1664253464 >> 172.21.15.42:0/1664253464 conn(0x7f3de007e6b0 msgr2=0x7f3de0059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:48:35.196 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:48:35.196 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":393,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":393,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:48:31.390793+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":120,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12847},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12847":{"gid":12847,"name":"a","rank":0,"incarnation":390,"state":"up:active","state_seq":38,"addr":"172.21.15.42:6835/1449317789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1449317789},{"type":"v1","addr":"172.21.15.42:6835","nonce":1449317789}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:48:35.203 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2019-12-05T06:48:37.204 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:48:37.312 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:37.295+0000 7fb222404700 1 Processor -- start 2019-12-05T06:48:37.312 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:37.295+0000 7fb222404700 1 -- start start 2019-12-05T06:48:37.312 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:37.295+0000 7fb222404700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb21c08df80 0x7fb21c09f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:48:37.313 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:37.295+0000 7fb222404700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb21c09f830 0x7fb21c14efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:48:37.313 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:37.295+0000 7fb222404700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb21c09fc60 con 0x7fb21c0a7740 2019-12-05T06:48:37.313 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:37.295+0000 7fb222404700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb21c0a0b00 con 0x7fb21c09f830 2019-12-05T06:48:37.313 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:37.295+0000 7fb222404700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb21c08e5b0 con 0x7fb21c08df80 2019-12-05T06:48:37.314 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:37.295+0000 7fb2211a2700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb21c09f830 0x7fb21c14efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:37.314 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:37.295+0000 7fb2211a2700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb21c09f830 0x7fb21c14efe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:37366/0 (socket says 172.21.15.42:37366) 2019-12-05T06:48:37.314 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:37.295+0000 7fb2211a2700 1 -- 172.21.15.42:0/3505982875 learned_addr learned my addr 172.21.15.42:0/3505982875 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:48:37.314 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:37.295+0000 7fb21bfff700 1 --2- 172.21.15.42:0/3505982875 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb21c08df80 0x7fb21c09f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:37.314 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:37.295+0000 7fb2211a2700 1 -- 172.21.15.42:0/3505982875 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb21c08df80 msgr2=0x7fb21c09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:48:37.314 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:37.295+0000 7fb2211a2700 1 --2- 172.21.15.42:0/3505982875 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb21c08df80 0x7fb21c09f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:48:37.315 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:37.295+0000 7fb2211a2700 1 -- 172.21.15.42:0/3505982875 >> v1:172.21.15.40:6789/0 conn(0x7fb21c0a7740 legacy=0x7fb21c07e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:37.315 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:37.295+0000 7fb2211a2700 1 -- 172.21.15.42:0/3505982875 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb21c0a0710 con 0x7fb21c09f830 2019-12-05T06:48:37.315 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:37.295+0000 7fb2211a2700 1 --2- 172.21.15.42:0/3505982875 >> v2:172.21.15.42:3300/0 conn(0x7fb21c09f830 0x7fb21c14efe0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fb214008b10 tx=0x7fb2140100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:37.315 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:37.295+0000 7fb2211a2700 1 --2- 172.21.15.42:0/3505982875 >> v2:172.21.15.42:3300/0 conn(0x7fb21c09f830 0x7fb21c14efe0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fb214008b10 tx=0x7fb2140100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:37.315 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:37.295+0000 7fb2211a2700 1 --2- 172.21.15.42:0/3505982875 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb21c09f830 0x7fb21c14efe0 secure :-1 s=READY pgs=3245 cs=0 l=1 rx=0x7fb214008b10 tx=0x7fb2140100b0).ready entity=mon.1 client_cookie=df7a759f439a1454 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:48:37.316 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:37.295+0000 7fb2211a2700 1 --2- 172.21.15.42:0/3505982875 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb21c09f830 0x7fb21c14efe0 secure :-1 s=THROTTLE_DONE pgs=3245 cs=0 l=1 rx=0x7fb214008b10 tx=0x7fb2140100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:37.316 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:37.299+0000 7fb21b7fe700 1 -- 172.21.15.42:0/3505982875 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb214010da0 con 0x7fb21c09f830 2019-12-05T06:48:37.316 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:37.299+0000 7fb2211a2700 1 --2- 172.21.15.42:0/3505982875 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb21c09f830 0x7fb21c14efe0 secure :-1 s=THROTTLE_DONE pgs=3245 cs=0 l=1 rx=0x7fb214008b10 tx=0x7fb2140100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:37.316 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:37.299+0000 7fb21b7fe700 1 -- 172.21.15.42:0/3505982875 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2140109c0 con 0x7fb21c09f830 2019-12-05T06:48:37.316 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:37.299+0000 7fb2211a2700 1 --2- 172.21.15.42:0/3505982875 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb21c09f830 0x7fb21c14efe0 secure :-1 s=THROTTLE_DONE pgs=3245 cs=0 l=1 rx=0x7fb214008b10 tx=0x7fb2140100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:37.316 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:37.299+0000 7fb222404700 1 -- 172.21.15.42:0/3505982875 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb21c09f830 msgr2=0x7fb21c14efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:37.317 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:37.299+0000 7fb21b7fe700 1 -- 172.21.15.42:0/3505982875 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb2140114a0 con 0x7fb21c09f830 2019-12-05T06:48:37.317 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:37.299+0000 7fb222404700 1 --2- 172.21.15.42:0/3505982875 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb21c09f830 0x7fb21c14efe0 secure :-1 s=READY pgs=3245 cs=0 l=1 rx=0x7fb214008b10 tx=0x7fb2140100b0).stop 2019-12-05T06:48:37.317 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:37.299+0000 7fb222404700 1 -- 172.21.15.42:0/3505982875 shutdown_connections 2019-12-05T06:48:37.317 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:37.299+0000 7fb222404700 1 --2- 172.21.15.42:0/3505982875 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb21c09f830 0x7fb21c14efe0 unknown :-1 s=CLOSED pgs=3245 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:37.318 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:37.299+0000 7fb222404700 1 --2- 172.21.15.42:0/3505982875 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb21c08df80 0x7fb21c09f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:48:37.318 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:37.299+0000 7fb222404700 1 -- 172.21.15.42:0/3505982875 shutdown_connections 2019-12-05T06:48:37.318 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:37.299+0000 7fb222404700 1 -- 172.21.15.42:0/3505982875 wait complete. 2019-12-05T06:48:37.318 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:37.299+0000 7fb222404700 1 -- 172.21.15.42:0/3505982875 >> 172.21.15.42:0/3505982875 conn(0x7fb21c0aabe0 msgr2=0x7fb21c0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:48:37.319 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:37.299+0000 7fb222404700 1 Processor -- start 2019-12-05T06:48:37.319 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:37.299+0000 7fb222404700 1 -- start start 2019-12-05T06:48:37.319 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:37.299+0000 7fb222404700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb21c064650 0x7fb21c0a1a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:48:37.319 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:37.299+0000 7fb222404700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb21c1202c0 0x7fb21c1206f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:48:37.320 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:37.299+0000 7fb222404700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb21c120c40 con 0x7fb21c145860 2019-12-05T06:48:37.320 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:37.299+0000 7fb222404700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb21c0a1510 con 0x7fb21c1202c0 2019-12-05T06:48:37.320 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:37.299+0000 7fb222404700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb21c145e50 con 0x7fb21c064650 2019-12-05T06:48:37.320 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:37.299+0000 7fb2211a2700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb21c1202c0 0x7fb21c1206f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:37.321 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:37.299+0000 7fb2211a2700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb21c1202c0 0x7fb21c1206f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:37372/0 (socket says 172.21.15.42:37372) 2019-12-05T06:48:37.321 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:37.299+0000 7fb2211a2700 1 -- 172.21.15.42:0/3929468441 learned_addr learned my addr 172.21.15.42:0/3929468441 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:48:37.322 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:37.299+0000 7fb2209a1700 1 --2- 172.21.15.42:0/3929468441 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb21c064650 0x7fb21c0a1a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:37.322 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:37.303+0000 7fb2211a2700 1 -- 172.21.15.42:0/3929468441 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb21c064650 msgr2=0x7fb21c0a1a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:37.322 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:37.303+0000 7fb2211a2700 1 --2- 172.21.15.42:0/3929468441 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb21c064650 0x7fb21c0a1a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:37.323 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:37.303+0000 7fb2211a2700 1 -- 172.21.15.42:0/3929468441 >> v1:172.21.15.40:6789/0 conn(0x7fb21c145860 legacy=0x7fb21c0a2280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:37.323 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:37.303+0000 7fb2197fa700 1 -- 172.21.15.42:0/3929468441 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4067873211 0 0) 0x7fb21c120c40 con 0x7fb21c145860 2019-12-05T06:48:37.323 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:37.303+0000 7fb2211a2700 1 -- 172.21.15.42:0/3929468441 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb21c0a7120 con 0x7fb21c1202c0 2019-12-05T06:48:37.324 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:37.303+0000 7fb2211a2700 1 --2- 172.21.15.42:0/3929468441 >> v2:172.21.15.42:3300/0 conn(0x7fb21c1202c0 0x7fb21c1206f0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fb214008b10 tx=0x7fb214010cd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:37.324 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:37.303+0000 7fb2211a2700 1 --2- 172.21.15.42:0/3929468441 >> v2:172.21.15.42:3300/0 conn(0x7fb21c1202c0 0x7fb21c1206f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fb214008b10 tx=0x7fb214010cd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:37.324 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:37.303+0000 7fb2211a2700 1 --2- 172.21.15.42:0/3929468441 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb21c1202c0 0x7fb21c1206f0 secure :-1 s=READY pgs=3246 cs=0 l=1 rx=0x7fb214008b10 tx=0x7fb214010cd0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:48:37.325 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:37.303+0000 7fb2211a2700 1 --2- 172.21.15.42:0/3929468441 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb21c1202c0 0x7fb21c1206f0 secure :-1 s=THROTTLE_DONE pgs=3246 cs=0 l=1 rx=0x7fb214008b10 tx=0x7fb214010cd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:37.325 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:37.303+0000 7fb2197fa700 1 -- 172.21.15.42:0/3929468441 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb214011b80 con 0x7fb21c1202c0 2019-12-05T06:48:37.325 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:37.303+0000 7fb2211a2700 1 --2- 172.21.15.42:0/3929468441 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb21c1202c0 0x7fb21c1206f0 secure :-1 s=THROTTLE_DONE pgs=3246 cs=0 l=1 rx=0x7fb214008b10 tx=0x7fb214010cd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:37.325 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:37.303+0000 7fb2211a2700 1 --2- 172.21.15.42:0/3929468441 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb21c1202c0 0x7fb21c1206f0 secure :-1 s=THROTTLE_DONE pgs=3246 cs=0 l=1 rx=0x7fb214008b10 tx=0x7fb214010cd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:37.325 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:37.303+0000 7fb2197fa700 1 -- 172.21.15.42:0/3929468441 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2140110c0 con 0x7fb21c1202c0 2019-12-05T06:48:37.325 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:37.303+0000 7fb222404700 1 -- 172.21.15.42:0/3929468441 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb21c0a2a80 con 0x7fb21c1202c0 2019-12-05T06:48:37.325 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:37.303+0000 7fb2197fa700 1 -- 172.21.15.42:0/3929468441 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb214005ab0 con 0x7fb21c1202c0 2019-12-05T06:48:37.326 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:37.303+0000 7fb222404700 1 -- 172.21.15.42:0/3929468441 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb21c137a00 con 0x7fb21c1202c0 2019-12-05T06:48:37.326 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:37.303+0000 7fb2211a2700 1 --2- 172.21.15.42:0/3929468441 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb21c1202c0 0x7fb21c1206f0 secure :-1 s=THROTTLE_DONE pgs=3246 cs=0 l=1 rx=0x7fb214008b10 tx=0x7fb214010cd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:37.326 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:37.303+0000 7fb2197fa700 1 -- 172.21.15.42:0/3929468441 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fb21400f280 con 0x7fb21c1202c0 2019-12-05T06:48:37.326 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:37.303+0000 7fb2211a2700 1 --2- 172.21.15.42:0/3929468441 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb21c1202c0 0x7fb21c1206f0 secure :-1 s=THROTTLE_DONE pgs=3246 cs=0 l=1 rx=0x7fb214008b10 tx=0x7fb214010cd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:37.326 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:37.303+0000 7fb2197fa700 1 --2- 172.21.15.42:0/3929468441 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb1f403ac90 0x7fb1f403d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:48:37.326 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:37.303+0000 7fb2197fa700 1 -- 172.21.15.42:0/3929468441 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 12137+0+0 (secure 0 0 0) 0x7fb21404b420 con 0x7fb21c1202c0 2019-12-05T06:48:37.327 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:37.303+0000 7fb2209a1700 1 --2- 172.21.15.42:0/3929468441 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb1f403ac90 0x7fb1f403d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:37.327 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:37.303+0000 7fb2209a1700 1 --2- 172.21.15.42:0/3929468441 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb1f403ac90 0x7fb1f403d1a0 crc :-1 s=READY pgs=1359 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:48:37.327 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:37.311+0000 7fb222404700 1 -- 172.21.15.42:0/3929468441 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb21c05d110 con 0x7fb21c1202c0 2019-12-05T06:48:37.328 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:37.319+0000 7fb2211a2700 1 --2- 172.21.15.42:0/3929468441 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb21c1202c0 0x7fb21c1206f0 secure :-1 s=THROTTLE_DONE pgs=3246 cs=0 l=1 rx=0x7fb214008b10 tx=0x7fb214010cd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:37.328 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:37.319+0000 7fb2197fa700 1 -- 172.21.15.42:0/3929468441 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fb214005dc0 con 0x7fb21c1202c0 2019-12-05T06:48:37.490 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:37.483+0000 7fb222404700 1 -- 172.21.15.42:0/3929468441 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb21c05b8c0 con 0x7fb21c1202c0 2019-12-05T06:48:37.490 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:37.483+0000 7fb2211a2700 1 --2- 172.21.15.42:0/3929468441 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb21c1202c0 0x7fb21c1206f0 secure :-1 s=THROTTLE_DONE pgs=3246 cs=0 l=1 rx=0x7fb214008b10 tx=0x7fb214010cd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:37.490 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:37.483+0000 7fb2197fa700 1 -- 172.21.15.42:0/3929468441 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 393 v393) v1 ==== 95+0+1779 (secure 0 0 0) 0x7fb2140332b0 con 0x7fb21c1202c0 2019-12-05T06:48:37.496 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:48:37.496 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":393,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":393,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:48:31.390793+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":120,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12847},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12847":{"gid":12847,"name":"a","rank":0,"incarnation":390,"state":"up:active","state_seq":38,"addr":"172.21.15.42:6835/1449317789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1449317789},{"type":"v1","addr":"172.21.15.42:6835","nonce":1449317789}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:48:37.496 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 393 2019-12-05T06:48:37.496 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:37.487+0000 7fb222404700 1 -- 172.21.15.42:0/3929468441 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb1f403ac90 msgr2=0x7fb1f403d1a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:37.497 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:37.487+0000 7fb222404700 1 --2- 172.21.15.42:0/3929468441 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb1f403ac90 0x7fb1f403d1a0 crc :-1 s=READY pgs=1359 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:37.497 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:37.487+0000 7fb222404700 1 -- 172.21.15.42:0/3929468441 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb21c1202c0 msgr2=0x7fb21c1206f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:37.497 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:37.487+0000 7fb222404700 1 --2- 172.21.15.42:0/3929468441 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb21c1202c0 0x7fb21c1206f0 secure :-1 s=READY pgs=3246 cs=0 l=1 rx=0x7fb214008b10 tx=0x7fb214010cd0).stop 2019-12-05T06:48:37.497 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:37.487+0000 7fb222404700 1 -- 172.21.15.42:0/3929468441 shutdown_connections 2019-12-05T06:48:37.497 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:37.487+0000 7fb222404700 1 --2- 172.21.15.42:0/3929468441 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb1f403ac90 0x7fb1f403d1a0 unknown :-1 s=CLOSED pgs=1359 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:37.497 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:37.487+0000 7fb222404700 1 --2- 172.21.15.42:0/3929468441 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb21c1202c0 0x7fb21c1206f0 unknown :-1 s=CLOSED pgs=3246 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:37.498 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:37.487+0000 7fb222404700 1 --2- 172.21.15.42:0/3929468441 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb21c064650 0x7fb21c0a1a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:37.498 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:37.487+0000 7fb222404700 1 -- 172.21.15.42:0/3929468441 shutdown_connections 2019-12-05T06:48:37.498 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:37.487+0000 7fb222404700 1 -- 172.21.15.42:0/3929468441 wait complete. 2019-12-05T06:48:37.498 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:37.487+0000 7fb222404700 1 -- 172.21.15.42:0/3929468441 >> 172.21.15.42:0/3929468441 conn(0x7fb21c08eb70 msgr2=0x7fb21c0a97c0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:48:37.505 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2019-12-05T06:48:37.869 INFO:tasks.workunit.client.0.smithi042.stderr:+ cd ../.. 2019-12-05T06:48:37.870 INFO:tasks.workunit.client.0.smithi042.stderr:+ ver=v0.7 2019-12-05T06:48:37.870 INFO:tasks.workunit.client.0.smithi042.stderr:+ echo checking v0.7 2019-12-05T06:48:37.870 INFO:tasks.workunit.client.0.smithi042.stderr:+ cd .snap/v0.7 2019-12-05T06:48:37.871 INFO:tasks.workunit.client.0.smithi042.stdout:checking v0.7 2019-12-05T06:48:37.873 INFO:tasks.workunit.client.0.smithi042.stderr:+ git diff --exit-code 2019-12-05T06:48:39.506 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:48:39.614 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:39.603+0000 7f3c7a2c1700 1 Processor -- start 2019-12-05T06:48:39.614 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:39.603+0000 7f3c7a2c1700 1 -- start start 2019-12-05T06:48:39.614 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:39.603+0000 7f3c7a2c1700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3c7409fae0 0x7f3c7414fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:48:39.615 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:39.603+0000 7f3c7a2c1700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c740a0710 con 0x7f3c740a7740 2019-12-05T06:48:39.615 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:39.603+0000 7f3c7a2c1700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c7407ef20 con 0x7f3c7408df80 2019-12-05T06:48:39.615 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:39.603+0000 7f3c7a2c1700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c740a0b00 con 0x7f3c7409fae0 2019-12-05T06:48:39.615 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:39.603+0000 7f3c6bfff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f3c7408df80 0x7f3c7409f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:37434/0 (socket says 172.21.15.42:37434) 2019-12-05T06:48:39.615 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:39.603+0000 7f3c6bfff700 1 -- 172.21.15.42:0/574519696 learned_addr learned my addr 172.21.15.42:0/574519696 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:48:39.616 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:39.603+0000 7f3c7905f700 1 --2- 172.21.15.42:0/574519696 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3c7409fae0 0x7f3c7414fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:39.616 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:39.603+0000 7f3c6b7fe700 1 -- 172.21.15.42:0/574519696 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4273681168 0 0) 0x7f3c7407ef20 con 0x7f3c7408df80 2019-12-05T06:48:39.616 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:39.603+0000 7f3c6b7fe700 1 -- 172.21.15.42:0/574519696 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c5c0030b0 con 0x7f3c7408df80 2019-12-05T06:48:39.616 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:39.603+0000 7f3c6b7fe700 1 -- 172.21.15.42:0/574519696 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1382050017 0 0) 0x7f3c740a0710 con 0x7f3c740a7740 2019-12-05T06:48:39.616 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:39.603+0000 7f3c7905f700 1 -- 172.21.15.42:0/574519696 >> v1:172.21.15.42:6789/0 conn(0x7f3c7408df80 legacy=0x7f3c7409f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:39.617 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:39.603+0000 7f3c7905f700 1 -- 172.21.15.42:0/574519696 >> v1:172.21.15.40:6789/0 conn(0x7f3c740a7740 legacy=0x7f3c7407e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:39.617 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:39.603+0000 7f3c7905f700 1 -- 172.21.15.42:0/574519696 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c740a9ba0 con 0x7f3c7409fae0 2019-12-05T06:48:39.617 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:39.603+0000 7f3c7905f700 1 --2- 172.21.15.42:0/574519696 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3c7409fae0 0x7f3c7414fff0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f3c6c0032b0 tx=0x7f3c6c00ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:39.617 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:39.607+0000 7f3c7905f700 1 --2- 172.21.15.42:0/574519696 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3c7409fae0 0x7f3c7414fff0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f3c6c0032b0 tx=0x7f3c6c00ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:39.617 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:39.607+0000 7f3c7905f700 1 --2- 172.21.15.42:0/574519696 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3c7409fae0 0x7f3c7414fff0 secure :-1 s=READY pgs=721 cs=0 l=1 rx=0x7f3c6c0032b0 tx=0x7f3c6c00ff70).ready entity=mon.2 client_cookie=5a2dd2ee8430678a server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:48:39.617 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:39.607+0000 7f3c7905f700 1 --2- 172.21.15.42:0/574519696 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3c7409fae0 0x7f3c7414fff0 secure :-1 s=THROTTLE_DONE pgs=721 cs=0 l=1 rx=0x7f3c6c0032b0 tx=0x7f3c6c00ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:39.617 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:39.607+0000 7f3c7905f700 1 --2- 172.21.15.42:0/574519696 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3c7409fae0 0x7f3c7414fff0 secure :-1 s=THROTTLE_DONE pgs=721 cs=0 l=1 rx=0x7f3c6c0032b0 tx=0x7f3c6c00ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:39.618 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:39.607+0000 7f3c6b7fe700 1 -- 172.21.15.42:0/574519696 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3c6c0109b0 con 0x7f3c7409fae0 2019-12-05T06:48:39.618 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:39.607+0000 7f3c6b7fe700 1 -- 172.21.15.42:0/574519696 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c6c008b50 con 0x7f3c7409fae0 2019-12-05T06:48:39.618 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:39.607+0000 7f3c7a2c1700 1 -- 172.21.15.42:0/574519696 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3c7409fae0 msgr2=0x7f3c7414fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:39.618 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:39.607+0000 7f3c7905f700 1 --2- 172.21.15.42:0/574519696 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3c7409fae0 0x7f3c7414fff0 secure :-1 s=THROTTLE_DONE pgs=721 cs=0 l=1 rx=0x7f3c6c0032b0 tx=0x7f3c6c00ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:39.618 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:39.607+0000 7f3c6b7fe700 1 -- 172.21.15.42:0/574519696 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3c6c0110c0 con 0x7f3c7409fae0 2019-12-05T06:48:39.619 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:39.607+0000 7f3c7a2c1700 1 --2- 172.21.15.42:0/574519696 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3c7409fae0 0x7f3c7414fff0 secure :-1 s=READY pgs=721 cs=0 l=1 rx=0x7f3c6c0032b0 tx=0x7f3c6c00ff70).stop 2019-12-05T06:48:39.619 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:39.607+0000 7f3c7a2c1700 1 -- 172.21.15.42:0/574519696 shutdown_connections 2019-12-05T06:48:39.619 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:39.607+0000 7f3c7a2c1700 1 --2- 172.21.15.42:0/574519696 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3c7409fae0 0x7f3c7414fff0 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:39.619 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:39.607+0000 7f3c7a2c1700 1 -- 172.21.15.42:0/574519696 shutdown_connections 2019-12-05T06:48:39.619 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:39.607+0000 7f3c7a2c1700 1 -- 172.21.15.42:0/574519696 wait complete. 2019-12-05T06:48:39.620 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:39.607+0000 7f3c7a2c1700 1 -- 172.21.15.42:0/574519696 >> 172.21.15.42:0/574519696 conn(0x7f3c740aabe0 msgr2=0x7f3c740a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:48:39.626 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:39.607+0000 7f3c7a2c1700 1 Processor -- start 2019-12-05T06:48:39.626 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:39.607+0000 7f3c7a2c1700 1 -- start start 2019-12-05T06:48:39.626 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:39.611+0000 7f3c7a2c1700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f3c74147020 0x7f3c7412d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:48:39.626 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:39.611+0000 7f3c7a2c1700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3c74147d70 0x7f3c741481a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:48:39.626 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:39.611+0000 7f3c7a2c1700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c741486f0 con 0x7f3c74121150 2019-12-05T06:48:39.627 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:39.611+0000 7f3c7a2c1700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c7412d620 con 0x7f3c74147020 2019-12-05T06:48:39.627 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:39.611+0000 7f3c7a2c1700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c74146840 con 0x7f3c74147d70 2019-12-05T06:48:39.627 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:39.611+0000 7f3c7905f700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3c74147d70 0x7f3c741481a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:39.627 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:39.611+0000 7f3c7905f700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3c74147d70 0x7f3c741481a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:37406/0 (socket says 172.21.15.42:37406) 2019-12-05T06:48:39.627 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:39.611+0000 7f3c7905f700 1 -- 172.21.15.42:0/392091216 learned_addr learned my addr 172.21.15.42:0/392091216 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:48:39.627 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:39.611+0000 7f3c6bfff700 1 --2- 172.21.15.42:0/392091216 >> v2:172.21.15.40:3300/0 conn(0x7f3c74147020 0x7f3c7412d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:39.628 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:39.611+0000 7f3c7905f700 1 -- 172.21.15.42:0/392091216 >> v2:172.21.15.40:3300/0 conn(0x7f3c74147020 msgr2=0x7f3c7412d0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:39.628 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:39.611+0000 7f3c7905f700 1 --2- 172.21.15.42:0/392091216 >> v2:172.21.15.40:3300/0 conn(0x7f3c74147020 0x7f3c7412d0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:39.628 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:39.611+0000 7f3c7905f700 1 -- 172.21.15.42:0/392091216 >> v1:172.21.15.40:6789/0 conn(0x7f3c74121150 legacy=0x7f3c7407d920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:39.628 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:39.611+0000 7f3c7905f700 1 -- 172.21.15.42:0/392091216 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c74147450 con 0x7f3c74147d70 2019-12-05T06:48:39.628 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:39.611+0000 7f3c7905f700 1 --2- 172.21.15.42:0/392091216 >> v2:172.21.15.42:3300/0 conn(0x7f3c74147d70 0x7f3c741481a0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f3c6c0032b0 tx=0x7f3c6c0087f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:39.628 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:39.611+0000 7f3c7905f700 1 --2- 172.21.15.42:0/392091216 >> v2:172.21.15.42:3300/0 conn(0x7f3c74147d70 0x7f3c741481a0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f3c6c0032b0 tx=0x7f3c6c0087f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:39.629 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:39.611+0000 7f3c7905f700 1 --2- 172.21.15.42:0/392091216 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3c74147d70 0x7f3c741481a0 secure :-1 s=READY pgs=3248 cs=0 l=1 rx=0x7f3c6c0032b0 tx=0x7f3c6c0087f0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:48:39.629 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:39.611+0000 7f3c7905f700 1 --2- 172.21.15.42:0/392091216 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3c74147d70 0x7f3c741481a0 secure :-1 s=THROTTLE_DONE pgs=3248 cs=0 l=1 rx=0x7f3c6c0032b0 tx=0x7f3c6c0087f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:39.629 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:39.611+0000 7f3c697fa700 1 -- 172.21.15.42:0/392091216 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3c6c005ab0 con 0x7f3c74147d70 2019-12-05T06:48:39.629 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:39.611+0000 7f3c7a2c1700 1 -- 172.21.15.42:0/392091216 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c7407e120 con 0x7f3c74147d70 2019-12-05T06:48:39.629 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:39.611+0000 7f3c7a2c1700 1 -- 172.21.15.42:0/392091216 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c74134520 con 0x7f3c74147d70 2019-12-05T06:48:39.629 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:39.611+0000 7f3c7905f700 1 --2- 172.21.15.42:0/392091216 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3c74147d70 0x7f3c741481a0 secure :-1 s=THROTTLE_DONE pgs=3248 cs=0 l=1 rx=0x7f3c6c0032b0 tx=0x7f3c6c0087f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:39.630 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:39.611+0000 7f3c7905f700 1 -- 172.21.15.42:0/392091216 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3c74147d70 msgr2=0x7f3c741481a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2019-12-05T06:48:39.630 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:39.611+0000 7f3c7905f700 1 -- 172.21.15.42:0/392091216 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3c74147d70 msgr2=0x7f3c741481a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2019-12-05T06:48:39.630 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:39.611+0000 7f3c7905f700 1 --2- 172.21.15.42:0/392091216 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3c74147d70 0x7f3c741481a0 secure :-1 s=READY pgs=3248 cs=0 l=1 rx=0x7f3c6c0032b0 tx=0x7f3c6c0087f0).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-12-05T06:48:39.630 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:39.611+0000 7f3c7905f700 1 --2- 172.21.15.42:0/392091216 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3c74147d70 0x7f3c741481a0 secure :-1 s=READY pgs=3248 cs=0 l=1 rx=0x7f3c6c0032b0 tx=0x7f3c6c0087f0).stop 2019-12-05T06:48:39.630 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:39.611+0000 7f3c697fa700 1 -- 172.21.15.42:0/392091216 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3c6c010bb0 con 0x7f3c74147d70 2019-12-05T06:48:39.631 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:39.611+0000 7f3c697fa700 1 -- 172.21.15.42:0/392091216 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3c74147d70 msgr2=0x7f3c741481a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2019-12-05T06:48:39.631 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:39.611+0000 7f3c697fa700 1 --2- 172.21.15.42:0/392091216 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3c74147d70 0x7f3c741481a0 unknown :-1 s=CLOSED pgs=3248 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:39.631 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:39.611+0000 7f3c697fa700 1 --2- 172.21.15.42:0/392091216 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3c54003080 0x7f3c54005520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:48:39.631 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:39.611+0000 7f3c697fa700 1 --2- 172.21.15.42:0/392091216 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3c54005c00 0x7f3c540080e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:48:39.631 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:39.611+0000 7f3c697fa700 1 --2- 172.21.15.42:0/392091216 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3c540088c0 0x7f3c5400add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:48:39.632 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:39.611+0000 7f3c697fa700 1 -- 172.21.15.42:0/392091216 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c5400b560 con 0x7f3c540088c0 2019-12-05T06:48:39.632 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:39.611+0000 7f3c697fa700 1 -- 172.21.15.42:0/392091216 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c5400b750 con 0x7f3c54003080 2019-12-05T06:48:39.632 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:39.611+0000 7f3c697fa700 1 -- 172.21.15.42:0/392091216 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c5400b990 con 0x7f3c54005c00 2019-12-05T06:48:39.632 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:39.611+0000 7f3c7885e700 1 --2- 172.21.15.42:0/392091216 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3c54003080 0x7f3c54005520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:39.632 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:39.611+0000 7f3c7905f700 1 --2- 172.21.15.42:0/392091216 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3c540088c0 0x7f3c5400add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:39.633 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:39.611+0000 7f3c6bfff700 1 --2- 172.21.15.42:0/392091216 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3c54005c00 0x7f3c540080e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:39.633 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:39.611+0000 7f3c7885e700 1 -- 172.21.15.42:0/392091216 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3c54005c00 msgr2=0x7f3c540080e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:39.633 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:39.611+0000 7f3c7885e700 1 --2- 172.21.15.42:0/392091216 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3c54005c00 0x7f3c540080e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:39.633 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:39.611+0000 7f3c7885e700 1 -- 172.21.15.42:0/392091216 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3c540088c0 msgr2=0x7f3c5400add0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:39.633 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:39.611+0000 7f3c7885e700 1 --2- 172.21.15.42:0/392091216 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3c540088c0 0x7f3c5400add0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:39.633 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:39.611+0000 7f3c7905f700 1 -- 172.21.15.42:0/392091216 reap_dead start 2019-12-05T06:48:39.634 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:39.611+0000 7f3c7885e700 1 -- 172.21.15.42:0/392091216 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+,osdmap=0}) v3 -- 0x7f3c5400bcf0 con 0x7f3c54003080 2019-12-05T06:48:39.634 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:39.611+0000 7f3c7885e700 1 --2- 172.21.15.42:0/392091216 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3c54003080 0x7f3c54005520 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f3c70002e90 tx=0x7f3c70010a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:39.634 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:39.611+0000 7f3c7885e700 1 --2- 172.21.15.42:0/392091216 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3c54003080 0x7f3c54005520 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f3c70002e90 tx=0x7f3c70010a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:39.634 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:39.611+0000 7f3c7885e700 1 --2- 172.21.15.42:0/392091216 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3c54003080 0x7f3c54005520 secure :-1 s=READY pgs=3249 cs=0 l=1 rx=0x7f3c70002e90 tx=0x7f3c70010a20).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:48:39.634 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:39.611+0000 7f3c7885e700 1 --2- 172.21.15.42:0/392091216 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3c54003080 0x7f3c54005520 secure :-1 s=THROTTLE_DONE pgs=3249 cs=0 l=1 rx=0x7f3c70002e90 tx=0x7f3c70010a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:39.635 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:39.611+0000 7f3c7885e700 1 --2- 172.21.15.42:0/392091216 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3c54003080 0x7f3c54005520 secure :-1 s=THROTTLE_DONE pgs=3249 cs=0 l=1 rx=0x7f3c70002e90 tx=0x7f3c70010a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:39.635 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:39.611+0000 7f3c697fa700 1 -- 172.21.15.42:0/392091216 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3c70012800 con 0x7f3c54003080 2019-12-05T06:48:39.635 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:39.611+0000 7f3c697fa700 1 -- 172.21.15.42:0/392091216 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c70012420 con 0x7f3c54003080 2019-12-05T06:48:39.635 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:39.611+0000 7f3c7885e700 1 --2- 172.21.15.42:0/392091216 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3c54003080 0x7f3c54005520 secure :-1 s=THROTTLE_DONE pgs=3249 cs=0 l=1 rx=0x7f3c70002e90 tx=0x7f3c70010a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:39.635 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:39.611+0000 7f3c697fa700 1 -- 172.21.15.42:0/392091216 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f3c70012a40 con 0x7f3c54003080 2019-12-05T06:48:39.635 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:39.611+0000 7f3c7885e700 1 --2- 172.21.15.42:0/392091216 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3c54003080 0x7f3c54005520 secure :-1 s=THROTTLE_DONE pgs=3249 cs=0 l=1 rx=0x7f3c70002e90 tx=0x7f3c70010a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:39.636 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:39.615+0000 7f3c697fa700 1 --2- 172.21.15.42:0/392091216 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3c5403e890 0x7f3c54040da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:48:39.636 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:39.615+0000 7f3c697fa700 1 -- 172.21.15.42:0/392091216 <== mon.1 v2:172.21.15.42:3300/0 4 ==== osd_map(120..120 src has 1..120) v4 ==== 12137+0+0 (secure 0 0 0) 0x7f3c7004a620 con 0x7f3c54003080 2019-12-05T06:48:39.636 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:39.615+0000 7f3c6bfff700 1 --2- 172.21.15.42:0/392091216 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3c5403e890 0x7f3c54040da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:39.636 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:39.615+0000 7f3c6bfff700 1 --2- 172.21.15.42:0/392091216 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3c5403e890 0x7f3c54040da0 crc :-1 s=READY pgs=1361 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:48:39.636 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:39.619+0000 7f3c7a2c1700 1 -- 172.21.15.42:0/392091216 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c4c003200 con 0x7f3c54003080 2019-12-05T06:48:39.637 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:39.623+0000 7f3c7885e700 1 --2- 172.21.15.42:0/392091216 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3c54003080 0x7f3c54005520 secure :-1 s=THROTTLE_DONE pgs=3249 cs=0 l=1 rx=0x7f3c70002e90 tx=0x7f3c70010a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:39.637 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:39.623+0000 7f3c697fa700 1 -- 172.21.15.42:0/392091216 <== mon.1 v2:172.21.15.42:3300/0 5 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f3c700316a0 con 0x7f3c54003080 2019-12-05T06:48:39.838 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:39.831+0000 7f3c7a2c1700 1 -- 172.21.15.42:0/392091216 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3c4c003630 con 0x7f3c54003080 2019-12-05T06:48:39.839 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:39.831+0000 7f3c7885e700 1 --2- 172.21.15.42:0/392091216 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3c54003080 0x7f3c54005520 secure :-1 s=THROTTLE_DONE pgs=3249 cs=0 l=1 rx=0x7f3c70002e90 tx=0x7f3c70010a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:39.839 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:39.831+0000 7f3c697fa700 1 -- 172.21.15.42:0/392091216 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 394 v394) v1 ==== 95+0+2114 (secure 0 0 0) 0x7f3c700101c0 con 0x7f3c54003080 2019-12-05T06:48:39.839 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 394 2019-12-05T06:48:39.839 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:48:39.839 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":394,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":12687,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/815316041","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":815316041},{"type":"v1","addr":"172.21.15.40:6833","nonce":815316041}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":394}],"filesystems":[{"mdsmap":{"epoch":393,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:48:31.390793+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":120,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12847},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12847":{"gid":12847,"name":"a","rank":0,"incarnation":390,"state":"up:active","state_seq":38,"addr":"172.21.15.42:6835/1449317789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1449317789},{"type":"v1","addr":"172.21.15.42:6835","nonce":1449317789}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:48:39.841 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:39.835+0000 7f3c7a2c1700 1 -- 172.21.15.42:0/392091216 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3c5403e890 msgr2=0x7f3c54040da0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:39.841 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:39.835+0000 7f3c7a2c1700 1 --2- 172.21.15.42:0/392091216 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3c5403e890 0x7f3c54040da0 crc :-1 s=READY pgs=1361 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:39.842 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:39.835+0000 7f3c7a2c1700 1 -- 172.21.15.42:0/392091216 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3c54003080 msgr2=0x7f3c54005520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:39.842 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:39.835+0000 7f3c7a2c1700 1 --2- 172.21.15.42:0/392091216 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3c54003080 0x7f3c54005520 secure :-1 s=READY pgs=3249 cs=0 l=1 rx=0x7f3c70002e90 tx=0x7f3c70010a20).stop 2019-12-05T06:48:39.842 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:39.835+0000 7f3c7a2c1700 1 -- 172.21.15.42:0/392091216 shutdown_connections 2019-12-05T06:48:39.842 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:39.835+0000 7f3c7a2c1700 1 --2- 172.21.15.42:0/392091216 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3c5403e890 0x7f3c54040da0 unknown :-1 s=CLOSED pgs=1361 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:39.843 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:39.835+0000 7f3c7a2c1700 1 --2- 172.21.15.42:0/392091216 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3c54003080 0x7f3c54005520 unknown :-1 s=CLOSED pgs=3249 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:39.843 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:39.835+0000 7f3c7a2c1700 1 --2- 172.21.15.42:0/392091216 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3c74147d70 0x7f3c741481a0 unknown :-1 s=CLOSED pgs=3248 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:39.843 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:39.835+0000 7f3c7a2c1700 1 -- 172.21.15.42:0/392091216 shutdown_connections 2019-12-05T06:48:39.843 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:39.835+0000 7f3c7a2c1700 1 -- 172.21.15.42:0/392091216 wait complete. 2019-12-05T06:48:39.843 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:39.835+0000 7f3c7a2c1700 1 -- 172.21.15.42:0/392091216 >> 172.21.15.42:0/392091216 conn(0x7f3c740a8f60 msgr2=0x7f3c7407e720 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:48:39.853 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b reported in up:standby state 2019-12-05T06:48:39.853 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 3 secs before thrashing 2019-12-05T06:48:42.855 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:48:42.962 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:42.951+0000 7f7b364b3700 1 Processor -- start 2019-12-05T06:48:42.963 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:42.951+0000 7f7b364b3700 1 -- start start 2019-12-05T06:48:42.963 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:42.951+0000 7f7b364b3700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f7b3009fad0 0x7f7b3014ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:48:42.963 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:42.951+0000 7f7b364b3700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b300a0700 con 0x7f7b3008df70 2019-12-05T06:48:42.963 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:42.951+0000 7f7b364b3700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b3007ef10 con 0x7f7b300a7730 2019-12-05T06:48:42.963 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:42.951+0000 7f7b364b3700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b300a0af0 con 0x7f7b3009fad0 2019-12-05T06:48:42.963 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:42.951+0000 7f7b34a50700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f7b300a7730 0x7f7b3007e710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:37476/0 (socket says 172.21.15.42:37476) 2019-12-05T06:48:42.964 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:42.951+0000 7f7b34a50700 1 -- 172.21.15.42:0/3958970960 learned_addr learned my addr 172.21.15.42:0/3958970960 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:48:42.964 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:42.951+0000 7f7b35251700 1 --2- 172.21.15.42:0/3958970960 >> v2:172.21.15.42:3300/0 conn(0x7f7b3009fad0 0x7f7b3014ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:42.964 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:42.951+0000 7f7b35251700 1 -- 172.21.15.42:0/3958970960 >> v1:172.21.15.42:6789/0 conn(0x7f7b300a7730 legacy=0x7f7b3007e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:42.964 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:42.951+0000 7f7b277fe700 1 -- 172.21.15.42:0/3958970960 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3016585201 0 0) 0x7f7b300a0700 con 0x7f7b3008df70 2019-12-05T06:48:42.964 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:42.951+0000 7f7b35251700 1 -- 172.21.15.42:0/3958970960 >> v1:172.21.15.40:6789/0 conn(0x7f7b3008df70 legacy=0x7f7b3009f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:42.964 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:42.951+0000 7f7b35251700 1 -- 172.21.15.42:0/3958970960 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b300a9b90 con 0x7f7b3009fad0 2019-12-05T06:48:42.965 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:42.951+0000 7f7b35251700 1 --2- 172.21.15.42:0/3958970960 >> v2:172.21.15.42:3300/0 conn(0x7f7b3009fad0 0x7f7b3014ffe0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f7b28008b10 tx=0x7f7b2c001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:42.965 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:42.955+0000 7f7b35251700 1 --2- 172.21.15.42:0/3958970960 >> v2:172.21.15.42:3300/0 conn(0x7f7b3009fad0 0x7f7b3014ffe0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f7b28008b10 tx=0x7f7b2c001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:42.965 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:42.955+0000 7f7b35251700 1 --2- 172.21.15.42:0/3958970960 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7b3009fad0 0x7f7b3014ffe0 secure :-1 s=READY pgs=3251 cs=0 l=1 rx=0x7f7b28008b10 tx=0x7f7b2c001150).ready entity=mon.1 client_cookie=8b0e9ac82b6f6ffe server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:48:42.965 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:42.955+0000 7f7b35251700 1 --2- 172.21.15.42:0/3958970960 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7b3009fad0 0x7f7b3014ffe0 secure :-1 s=THROTTLE_DONE pgs=3251 cs=0 l=1 rx=0x7f7b28008b10 tx=0x7f7b2c001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:42.965 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:42.955+0000 7f7b277fe700 1 -- 172.21.15.42:0/3958970960 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7b28010870 con 0x7f7b3009fad0 2019-12-05T06:48:42.965 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:42.955+0000 7f7b35251700 1 --2- 172.21.15.42:0/3958970960 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7b3009fad0 0x7f7b3014ffe0 secure :-1 s=THROTTLE_DONE pgs=3251 cs=0 l=1 rx=0x7f7b28008b10 tx=0x7f7b2c001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:42.966 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:42.955+0000 7f7b277fe700 1 -- 172.21.15.42:0/3958970960 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b28008b50 con 0x7f7b3009fad0 2019-12-05T06:48:42.966 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:42.955+0000 7f7b35251700 1 --2- 172.21.15.42:0/3958970960 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7b3009fad0 0x7f7b3014ffe0 secure :-1 s=THROTTLE_DONE pgs=3251 cs=0 l=1 rx=0x7f7b28008b10 tx=0x7f7b2c001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:42.966 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:42.955+0000 7f7b364b3700 1 -- 172.21.15.42:0/3958970960 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7b3009fad0 msgr2=0x7f7b3014ffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:42.966 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:42.955+0000 7f7b277fe700 1 -- 172.21.15.42:0/3958970960 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7b280110c0 con 0x7f7b3009fad0 2019-12-05T06:48:42.967 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:42.955+0000 7f7b364b3700 1 --2- 172.21.15.42:0/3958970960 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7b3009fad0 0x7f7b3014ffe0 secure :-1 s=READY pgs=3251 cs=0 l=1 rx=0x7f7b28008b10 tx=0x7f7b2c001150).stop 2019-12-05T06:48:42.967 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:42.955+0000 7f7b364b3700 1 -- 172.21.15.42:0/3958970960 shutdown_connections 2019-12-05T06:48:42.967 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:42.955+0000 7f7b364b3700 1 --2- 172.21.15.42:0/3958970960 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7b3009fad0 0x7f7b3014ffe0 unknown :-1 s=CLOSED pgs=3251 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:42.967 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:42.955+0000 7f7b364b3700 1 -- 172.21.15.42:0/3958970960 shutdown_connections 2019-12-05T06:48:42.967 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:42.955+0000 7f7b364b3700 1 -- 172.21.15.42:0/3958970960 wait complete. 2019-12-05T06:48:42.967 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:42.955+0000 7f7b364b3700 1 -- 172.21.15.42:0/3958970960 >> 172.21.15.42:0/3958970960 conn(0x7f7b300aabd0 msgr2=0x7f7b300a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:48:42.972 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:42.955+0000 7f7b364b3700 1 Processor -- start 2019-12-05T06:48:42.972 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:42.955+0000 7f7b364b3700 1 -- start start 2019-12-05T06:48:42.972 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:42.959+0000 7f7b364b3700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7b30124870 0x7f7b30118430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:48:42.972 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:42.959+0000 7f7b364b3700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7b3011b950 0x7f7b300702b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:48:42.972 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:42.959+0000 7f7b364b3700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b30118980 con 0x7f7b3011b520 2019-12-05T06:48:42.973 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:42.959+0000 7f7b364b3700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b30117ef0 con 0x7f7b30124870 2019-12-05T06:48:42.973 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:42.959+0000 7f7b364b3700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b30118190 con 0x7f7b3011b950 2019-12-05T06:48:42.973 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:42.959+0000 7f7b27fff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f7b3011b520 0x7f7b3011ab00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:37484/0 (socket says 172.21.15.42:37484) 2019-12-05T06:48:42.973 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:42.959+0000 7f7b27fff700 1 -- 172.21.15.42:0/1623286720 learned_addr learned my addr 172.21.15.42:0/1623286720 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:48:42.973 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:42.959+0000 7f7b34a50700 1 --2- 172.21.15.42:0/1623286720 >> v2:172.21.15.40:3300/0 conn(0x7f7b30124870 0x7f7b30118430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:42.973 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:42.959+0000 7f7b35251700 1 --2- 172.21.15.42:0/1623286720 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7b3011b950 0x7f7b300702b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:42.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:42.959+0000 7f7b257fa700 1 -- 172.21.15.42:0/1623286720 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1287513753 0 0) 0x7f7b30118980 con 0x7f7b3011b520 2019-12-05T06:48:42.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:42.959+0000 7f7b257fa700 1 -- 172.21.15.42:0/1623286720 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b100030b0 con 0x7f7b3011b520 2019-12-05T06:48:42.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:42.959+0000 7f7b257fa700 1 -- 172.21.15.42:0/1623286720 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 1541798337 0 0) 0x7f7b100030b0 con 0x7f7b3011b520 2019-12-05T06:48:42.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:42.959+0000 7f7b34a50700 1 -- 172.21.15.42:0/1623286720 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7b3011b950 msgr2=0x7f7b300702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:42.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:42.959+0000 7f7b34a50700 1 --2- 172.21.15.42:0/1623286720 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7b3011b950 0x7f7b300702b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:42.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:42.959+0000 7f7b34a50700 1 -- 172.21.15.42:0/1623286720 >> v1:172.21.15.42:6789/0 conn(0x7f7b3011b520 legacy=0x7f7b3011ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:42.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:42.959+0000 7f7b34a50700 1 -- 172.21.15.42:0/1623286720 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b30116b30 con 0x7f7b30124870 2019-12-05T06:48:42.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:42.959+0000 7f7b35251700 1 --2- 172.21.15.42:0/1623286720 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7b3011b950 0x7f7b300702b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:48:42.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:42.959+0000 7f7b34a50700 1 --2- 172.21.15.42:0/1623286720 >> v2:172.21.15.40:3300/0 conn(0x7f7b30124870 0x7f7b30118430 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f7b20001c60 tx=0x7f7b20001ca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:42.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:42.959+0000 7f7b34a50700 1 --2- 172.21.15.42:0/1623286720 >> v2:172.21.15.40:3300/0 conn(0x7f7b30124870 0x7f7b30118430 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f7b20001c60 tx=0x7f7b20001ca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:42.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:42.959+0000 7f7b34a50700 1 --2- 172.21.15.42:0/1623286720 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7b30124870 0x7f7b30118430 secure :-1 s=READY pgs=2466 cs=0 l=1 rx=0x7f7b20001c60 tx=0x7f7b20001ca0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:48:42.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:42.959+0000 7f7b34a50700 1 --2- 172.21.15.42:0/1623286720 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7b30124870 0x7f7b30118430 secure :-1 s=THROTTLE_DONE pgs=2466 cs=0 l=1 rx=0x7f7b20001c60 tx=0x7f7b20001ca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:42.976 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:42.959+0000 7f7b257fa700 1 -- 172.21.15.42:0/1623286720 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7b20001a20 con 0x7f7b30124870 2019-12-05T06:48:42.976 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:42.959+0000 7f7b34a50700 1 --2- 172.21.15.42:0/1623286720 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7b30124870 0x7f7b30118430 secure :-1 s=THROTTLE_DONE pgs=2466 cs=0 l=1 rx=0x7f7b20001c60 tx=0x7f7b20001ca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:42.976 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:42.959+0000 7f7b34a50700 1 --2- 172.21.15.42:0/1623286720 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7b30124870 0x7f7b30118430 secure :-1 s=THROTTLE_DONE pgs=2466 cs=0 l=1 rx=0x7f7b20001c60 tx=0x7f7b20001ca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:42.976 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:42.959+0000 7f7b257fa700 1 -- 172.21.15.42:0/1623286720 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b20001830 con 0x7f7b30124870 2019-12-05T06:48:42.976 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:42.959+0000 7f7b257fa700 1 -- 172.21.15.42:0/1623286720 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7b2c0117e0 con 0x7f7b30124870 2019-12-05T06:48:42.976 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:42.959+0000 7f7b364b3700 1 -- 172.21.15.42:0/1623286720 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b30070800 con 0x7f7b30124870 2019-12-05T06:48:42.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:42.959+0000 7f7b364b3700 1 -- 172.21.15.42:0/1623286720 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b30070d80 con 0x7f7b30124870 2019-12-05T06:48:42.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:42.963+0000 7f7b34a50700 1 --2- 172.21.15.42:0/1623286720 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7b30124870 0x7f7b30118430 secure :-1 s=THROTTLE_DONE pgs=2466 cs=0 l=1 rx=0x7f7b20001c60 tx=0x7f7b20001ca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:42.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:42.963+0000 7f7b257fa700 1 -- 172.21.15.42:0/1623286720 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f7b2c011ab0 con 0x7f7b30124870 2019-12-05T06:48:42.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:42.963+0000 7f7b34a50700 1 --2- 172.21.15.42:0/1623286720 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7b30124870 0x7f7b30118430 secure :-1 s=THROTTLE_DONE pgs=2466 cs=0 l=1 rx=0x7f7b20001c60 tx=0x7f7b20001ca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:42.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:42.963+0000 7f7b257fa700 1 --2- 172.21.15.42:0/1623286720 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f7b1003b280 0x7f7b1003d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:48:42.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:42.963+0000 7f7b257fa700 1 -- 172.21.15.42:0/1623286720 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 12137+0+0 (secure 0 0 0) 0x7f7b2c04a7b0 con 0x7f7b30124870 2019-12-05T06:48:42.978 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:42.963+0000 7f7b27fff700 1 --2- 172.21.15.42:0/1623286720 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f7b1003b280 0x7f7b1003d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:42.978 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:42.963+0000 7f7b27fff700 1 --2- 172.21.15.42:0/1623286720 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f7b1003b280 0x7f7b1003d790 crc :-1 s=READY pgs=1362 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:48:42.978 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:42.967+0000 7f7b364b3700 1 -- 172.21.15.42:0/1623286720 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b3005d100 con 0x7f7b30124870 2019-12-05T06:48:42.991 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:42.975+0000 7f7b34a50700 1 --2- 172.21.15.42:0/1623286720 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7b30124870 0x7f7b30118430 secure :-1 s=THROTTLE_DONE pgs=2466 cs=0 l=1 rx=0x7f7b20001c60 tx=0x7f7b20001ca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:42.992 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:42.975+0000 7f7b257fa700 1 -- 172.21.15.42:0/1623286720 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f7b2c0317d0 con 0x7f7b30124870 2019-12-05T06:48:43.154 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:43.147+0000 7f7b364b3700 1 -- 172.21.15.42:0/1623286720 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7b3005b8b0 con 0x7f7b30124870 2019-12-05T06:48:43.155 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:43.147+0000 7f7b34a50700 1 --2- 172.21.15.42:0/1623286720 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7b30124870 0x7f7b30118430 secure :-1 s=THROTTLE_DONE pgs=2466 cs=0 l=1 rx=0x7f7b20001c60 tx=0x7f7b20001ca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:43.155 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:43.147+0000 7f7b257fa700 1 -- 172.21.15.42:0/1623286720 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 394 v394) v1 ==== 95+0+2114 (secure 0 0 0) 0x7f7b2c031a30 con 0x7f7b30124870 2019-12-05T06:48:43.157 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:48:43.157 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":394,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":12687,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/815316041","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":815316041},{"type":"v1","addr":"172.21.15.40:6833","nonce":815316041}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":394}],"filesystems":[{"mdsmap":{"epoch":393,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:48:31.390793+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":120,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12847},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12847":{"gid":12847,"name":"a","rank":0,"incarnation":390,"state":"up:active","state_seq":38,"addr":"172.21.15.42:6835/1449317789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1449317789},{"type":"v1","addr":"172.21.15.42:6835","nonce":1449317789}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:48:43.157 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 394 2019-12-05T06:48:43.160 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:43.151+0000 7f7b364b3700 1 -- 172.21.15.42:0/1623286720 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f7b1003b280 msgr2=0x7f7b1003d790 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:43.160 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:43.151+0000 7f7b364b3700 1 --2- 172.21.15.42:0/1623286720 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f7b1003b280 0x7f7b1003d790 crc :-1 s=READY pgs=1362 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:43.160 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:43.151+0000 7f7b364b3700 1 -- 172.21.15.42:0/1623286720 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7b30124870 msgr2=0x7f7b30118430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:43.160 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:43.151+0000 7f7b364b3700 1 --2- 172.21.15.42:0/1623286720 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7b30124870 0x7f7b30118430 secure :-1 s=READY pgs=2466 cs=0 l=1 rx=0x7f7b20001c60 tx=0x7f7b20001ca0).stop 2019-12-05T06:48:43.161 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:43.151+0000 7f7b364b3700 1 -- 172.21.15.42:0/1623286720 shutdown_connections 2019-12-05T06:48:43.161 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:43.151+0000 7f7b364b3700 1 --2- 172.21.15.42:0/1623286720 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f7b1003b280 0x7f7b1003d790 unknown :-1 s=CLOSED pgs=1362 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:43.161 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:43.151+0000 7f7b364b3700 1 --2- 172.21.15.42:0/1623286720 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7b3011b950 0x7f7b300702b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:43.161 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:43.151+0000 7f7b364b3700 1 --2- 172.21.15.42:0/1623286720 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7b30124870 0x7f7b30118430 unknown :-1 s=CLOSED pgs=2466 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:43.162 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:43.151+0000 7f7b364b3700 1 -- 172.21.15.42:0/1623286720 shutdown_connections 2019-12-05T06:48:43.162 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:43.151+0000 7f7b364b3700 1 -- 172.21.15.42:0/1623286720 wait complete. 2019-12-05T06:48:43.162 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:43.151+0000 7f7b364b3700 1 -- 172.21.15.42:0/1623286720 >> 172.21.15.42:0/1623286720 conn(0x7f7b300a6ba0 msgr2=0x7f7b3007e4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:48:43.170 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.a (rank=0) 2019-12-05T06:48:43.171 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2019-12-05T06:48:43.171 INFO:teuthology.orchestra.run:waiting for 300 2019-12-05T06:48:43.186 INFO:tasks.ceph.mds.a:Stopped 2019-12-05T06:48:43.186 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:48:43.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:43.339+0000 7f48ccafc700 1 Processor -- start 2019-12-05T06:48:43.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:43.339+0000 7f48ccafc700 1 -- start start 2019-12-05T06:48:43.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:43.343+0000 7f48ccafc700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f48c80a7740 0x7f48c808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:48:43.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:43.343+0000 7f48ccafc700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f48c809fae0 0x7f48c807eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:48:43.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:43.343+0000 7f48ccafc700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48c80a0710 con 0x7f48c807e720 2019-12-05T06:48:43.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:43.343+0000 7f48ccafc700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48c80a0b00 con 0x7f48c809fae0 2019-12-05T06:48:43.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:43.343+0000 7f48ccafc700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48c808e5b0 con 0x7f48c80a7740 2019-12-05T06:48:43.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:43.343+0000 7f48c659c700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f48c807e720 0x7f48c809f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:37494/0 (socket says 172.21.15.42:37494) 2019-12-05T06:48:43.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:43.343+0000 7f48c659c700 1 -- 172.21.15.42:0/3057323374 learned_addr learned my addr 172.21.15.42:0/3057323374 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:48:43.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:43.343+0000 7f48c759e700 1 --2- 172.21.15.42:0/3057323374 >> v2:172.21.15.42:3300/0 conn(0x7f48c809fae0 0x7f48c807eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:43.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:43.343+0000 7f48c6d9d700 1 --2- 172.21.15.42:0/3057323374 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f48c80a7740 0x7f48c808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:43.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:43.343+0000 7f48c759e700 1 -- 172.21.15.42:0/3057323374 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f48c80a7740 msgr2=0x7f48c808df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:48:43.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:43.343+0000 7f48c759e700 1 --2- 172.21.15.42:0/3057323374 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f48c80a7740 0x7f48c808df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:48:43.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:43.343+0000 7f48c759e700 1 -- 172.21.15.42:0/3057323374 >> v1:172.21.15.42:6789/0 conn(0x7f48c807e720 legacy=0x7f48c809f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:43.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:43.343+0000 7f48c5d9b700 1 -- 172.21.15.42:0/3057323374 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2948308795 0 0) 0x7f48c80a0710 con 0x7f48c807e720 2019-12-05T06:48:43.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:43.343+0000 7f48c759e700 1 -- 172.21.15.42:0/3057323374 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48c807f0a0 con 0x7f48c809fae0 2019-12-05T06:48:43.358 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:43.343+0000 7f48c6d9d700 1 --2- 172.21.15.42:0/3057323374 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f48c80a7740 0x7f48c808df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:48:43.358 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:43.343+0000 7f48c759e700 1 --2- 172.21.15.42:0/3057323374 >> v2:172.21.15.42:3300/0 conn(0x7f48c809fae0 0x7f48c807eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f48c0008790 tx=0x7f48c000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:43.358 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:43.343+0000 7f48c759e700 1 --2- 172.21.15.42:0/3057323374 >> v2:172.21.15.42:3300/0 conn(0x7f48c809fae0 0x7f48c807eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f48c0008790 tx=0x7f48c000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:43.358 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:43.343+0000 7f48c759e700 1 --2- 172.21.15.42:0/3057323374 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f48c809fae0 0x7f48c807eb50 secure :-1 s=READY pgs=3254 cs=0 l=1 rx=0x7f48c0008790 tx=0x7f48c000fc70).ready entity=mon.1 client_cookie=f61f24479494e1c3 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:48:43.358 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:43.343+0000 7f48c759e700 1 --2- 172.21.15.42:0/3057323374 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f48c809fae0 0x7f48c807eb50 secure :-1 s=THROTTLE_DONE pgs=3254 cs=0 l=1 rx=0x7f48c0008790 tx=0x7f48c000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:43.358 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:43.343+0000 7f48c5d9b700 1 -- 172.21.15.42:0/3057323374 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f48c0008bb0 con 0x7f48c809fae0 2019-12-05T06:48:43.359 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:43.343+0000 7f48c759e700 1 --2- 172.21.15.42:0/3057323374 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f48c809fae0 0x7f48c807eb50 secure :-1 s=THROTTLE_DONE pgs=3254 cs=0 l=1 rx=0x7f48c0008790 tx=0x7f48c000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:43.359 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:43.343+0000 7f48c759e700 1 --2- 172.21.15.42:0/3057323374 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f48c809fae0 0x7f48c807eb50 secure :-1 s=THROTTLE_DONE pgs=3254 cs=0 l=1 rx=0x7f48c0008790 tx=0x7f48c000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:43.359 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:43.343+0000 7f48c5d9b700 1 -- 172.21.15.42:0/3057323374 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48c00087d0 con 0x7f48c809fae0 2019-12-05T06:48:43.359 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:43.343+0000 7f48c5d9b700 1 -- 172.21.15.42:0/3057323374 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f48c0011860 con 0x7f48c809fae0 2019-12-05T06:48:43.359 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:43.343+0000 7f48ccafc700 1 -- 172.21.15.42:0/3057323374 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f48c809fae0 msgr2=0x7f48c807eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:43.359 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:43.343+0000 7f48ccafc700 1 --2- 172.21.15.42:0/3057323374 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f48c809fae0 0x7f48c807eb50 secure :-1 s=READY pgs=3254 cs=0 l=1 rx=0x7f48c0008790 tx=0x7f48c000fc70).stop 2019-12-05T06:48:43.360 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:43.347+0000 7f48ccafc700 1 -- 172.21.15.42:0/3057323374 shutdown_connections 2019-12-05T06:48:43.360 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:43.347+0000 7f48ccafc700 1 --2- 172.21.15.42:0/3057323374 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f48c809fae0 0x7f48c807eb50 unknown :-1 s=CLOSED pgs=3254 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:43.360 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:43.347+0000 7f48ccafc700 1 --2- 172.21.15.42:0/3057323374 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f48c80a7740 0x7f48c808df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:48:43.360 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:43.347+0000 7f48ccafc700 1 -- 172.21.15.42:0/3057323374 shutdown_connections 2019-12-05T06:48:43.360 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:43.347+0000 7f48ccafc700 1 -- 172.21.15.42:0/3057323374 wait complete. 2019-12-05T06:48:43.360 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:43.347+0000 7f48ccafc700 1 -- 172.21.15.42:0/3057323374 >> 172.21.15.42:0/3057323374 conn(0x7f48c80aabe0 msgr2=0x7f48c80a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:48:43.365 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:43.347+0000 7f48ccafc700 1 Processor -- start 2019-12-05T06:48:43.365 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:43.347+0000 7f48ccafc700 1 -- start start 2019-12-05T06:48:43.365 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:43.351+0000 7f48ccafc700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f48c811b050 0x7f48c808f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:48:43.365 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:43.351+0000 7f48ccafc700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f48c8144a80 0x7f48c8144eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:48:43.366 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:43.351+0000 7f48ccafc700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48c8145400 con 0x7f48c8116500 2019-12-05T06:48:43.366 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:43.351+0000 7f48ccafc700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48c811b680 con 0x7f48c811b050 2019-12-05T06:48:43.366 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:43.351+0000 7f48ccafc700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48c811b920 con 0x7f48c8144a80 2019-12-05T06:48:43.366 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:43.351+0000 7f48c659c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f48c811b050 0x7f48c808f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:43.366 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:43.351+0000 7f48c659c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f48c811b050 0x7f48c808f5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:37464/0 (socket says 172.21.15.42:37464) 2019-12-05T06:48:43.367 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:43.351+0000 7f48c659c700 1 -- 172.21.15.42:0/1667389664 learned_addr learned my addr 172.21.15.42:0/1667389664 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:48:43.367 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:43.351+0000 7f48c759e700 1 -- 172.21.15.42:0/1667389664 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f48c8144a80 msgr2=0x7f48c8144eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2019-12-05T06:48:43.367 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:43.351+0000 7f48c759e700 1 -- 172.21.15.42:0/1667389664 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f48c8144a80 msgr2=0x7f48c8144eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2019-12-05T06:48:43.367 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:43.351+0000 7f48c759e700 1 --2- 172.21.15.42:0/1667389664 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f48c8144a80 0x7f48c8144eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner read peer banner failed r=-1 ((1) Operation not permitted) 2019-12-05T06:48:43.368 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:43.351+0000 7f48c759e700 1 --2- 172.21.15.42:0/1667389664 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f48c8144a80 0x7f48c8144eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rx=0 tx=0)._fault waiting 0.200000 2019-12-05T06:48:43.368 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:43.351+0000 7f48c659c700 1 -- 172.21.15.42:0/1667389664 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f48c8144a80 msgr2=0x7f48c8144eb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2019-12-05T06:48:43.368 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:43.351+0000 7f48c659c700 1 --2- 172.21.15.42:0/1667389664 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f48c8144a80 0x7f48c8144eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:43.368 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:43.351+0000 7f48c659c700 1 -- 172.21.15.42:0/1667389664 >> v1:172.21.15.40:6789/0 conn(0x7f48c8116500 legacy=0x7f48c8142c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:43.368 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:43.351+0000 7f48c659c700 1 -- 172.21.15.42:0/1667389664 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48c8143480 con 0x7f48c811b050 2019-12-05T06:48:43.368 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:43.351+0000 7f48c659c700 1 --2- 172.21.15.42:0/1667389664 >> v2:172.21.15.42:3300/0 conn(0x7f48c811b050 0x7f48c808f5a0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f48c811b840 tx=0x7f48bc010cf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:43.368 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:43.351+0000 7f48c659c700 1 --2- 172.21.15.42:0/1667389664 >> v2:172.21.15.42:3300/0 conn(0x7f48c811b050 0x7f48c808f5a0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f48c811b840 tx=0x7f48bc010cf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:43.369 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:43.351+0000 7f48c659c700 1 --2- 172.21.15.42:0/1667389664 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f48c811b050 0x7f48c808f5a0 secure :-1 s=READY pgs=3255 cs=0 l=1 rx=0x7f48c811b840 tx=0x7f48bc010cf0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:48:43.369 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:43.351+0000 7f48c659c700 1 --2- 172.21.15.42:0/1667389664 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f48c811b050 0x7f48c808f5a0 secure :-1 s=THROTTLE_DONE pgs=3255 cs=0 l=1 rx=0x7f48c811b840 tx=0x7f48bc010cf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:43.369 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:43.351+0000 7f48b77fe700 1 -- 172.21.15.42:0/1667389664 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f48bc0119e0 con 0x7f48c811b050 2019-12-05T06:48:43.369 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:43.351+0000 7f48c659c700 1 --2- 172.21.15.42:0/1667389664 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f48c811b050 0x7f48c808f5a0 secure :-1 s=THROTTLE_DONE pgs=3255 cs=0 l=1 rx=0x7f48c811b840 tx=0x7f48bc010cf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:43.369 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:43.351+0000 7f48c659c700 1 --2- 172.21.15.42:0/1667389664 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f48c811b050 0x7f48c808f5a0 secure :-1 s=THROTTLE_DONE pgs=3255 cs=0 l=1 rx=0x7f48c811b840 tx=0x7f48bc010cf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:43.369 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:43.351+0000 7f48b77fe700 1 -- 172.21.15.42:0/1667389664 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48bc011600 con 0x7f48c811b050 2019-12-05T06:48:43.370 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:43.351+0000 7f48ccafc700 1 -- 172.21.15.42:0/1667389664 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48c8130380 con 0x7f48c811b050 2019-12-05T06:48:43.370 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:43.351+0000 7f48b77fe700 1 -- 172.21.15.42:0/1667389664 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f48bc00d0c0 con 0x7f48c811b050 2019-12-05T06:48:43.370 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:43.351+0000 7f48ccafc700 1 -- 172.21.15.42:0/1667389664 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48c81379f0 con 0x7f48c811b050 2019-12-05T06:48:43.370 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:43.351+0000 7f48c659c700 1 --2- 172.21.15.42:0/1667389664 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f48c811b050 0x7f48c808f5a0 secure :-1 s=THROTTLE_DONE pgs=3255 cs=0 l=1 rx=0x7f48c811b840 tx=0x7f48bc010cf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:43.370 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:43.351+0000 7f48b77fe700 1 -- 172.21.15.42:0/1667389664 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f48bc00d3a0 con 0x7f48c811b050 2019-12-05T06:48:43.370 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:43.351+0000 7f48c659c700 1 --2- 172.21.15.42:0/1667389664 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f48c811b050 0x7f48c808f5a0 secure :-1 s=THROTTLE_DONE pgs=3255 cs=0 l=1 rx=0x7f48c811b840 tx=0x7f48bc010cf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:43.370 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:43.351+0000 7f48b77fe700 1 --2- 172.21.15.42:0/1667389664 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f48ac03adc0 0x7f48ac03d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:48:43.371 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:43.351+0000 7f48b77fe700 1 -- 172.21.15.42:0/1667389664 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 12137+0+0 (secure 0 0 0) 0x7f48bc04b440 con 0x7f48c811b050 2019-12-05T06:48:43.371 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:43.351+0000 7f48c6d9d700 1 --2- 172.21.15.42:0/1667389664 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f48ac03adc0 0x7f48ac03d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:43.371 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:43.351+0000 7f48c6d9d700 1 --2- 172.21.15.42:0/1667389664 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f48ac03adc0 0x7f48ac03d2d0 crc :-1 s=READY pgs=1363 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:48:43.371 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:43.355+0000 7f48ccafc700 1 -- 172.21.15.42:0/1667389664 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48c811b480 con 0x7f48c811b050 2019-12-05T06:48:43.372 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:43.363+0000 7f48c659c700 1 --2- 172.21.15.42:0/1667389664 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f48c811b050 0x7f48c808f5a0 secure :-1 s=THROTTLE_DONE pgs=3255 cs=0 l=1 rx=0x7f48c811b840 tx=0x7f48bc010cf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:43.372 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:43.363+0000 7f48b77fe700 1 -- 172.21.15.42:0/1667389664 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f48bc032440 con 0x7f48c811b050 2019-12-05T06:48:43.531 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:43.523+0000 7f48ccafc700 1 -- 172.21.15.42:0/1667389664 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f48c805d110 con 0x7f48c811b050 2019-12-05T06:48:43.532 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:43.527+0000 7f48c659c700 1 --2- 172.21.15.42:0/1667389664 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f48c811b050 0x7f48c808f5a0 secure :-1 s=THROTTLE_DONE pgs=3255 cs=0 l=1 rx=0x7f48c811b840 tx=0x7f48bc010cf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:43.533 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:43.527+0000 7f48b77fe700 1 -- 172.21.15.42:0/1667389664 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 394 v394) v1 ==== 95+0+2114 (secure 0 0 0) 0x7f48bc0326a0 con 0x7f48c811b050 2019-12-05T06:48:43.533 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 394 2019-12-05T06:48:43.533 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:48:43.533 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":394,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":12687,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/815316041","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":815316041},{"type":"v1","addr":"172.21.15.40:6833","nonce":815316041}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":394}],"filesystems":[{"mdsmap":{"epoch":393,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:48:31.390793+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":120,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12847},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12847":{"gid":12847,"name":"a","rank":0,"incarnation":390,"state":"up:active","state_seq":38,"addr":"172.21.15.42:6835/1449317789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1449317789},{"type":"v1","addr":"172.21.15.42:6835","nonce":1449317789}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:48:43.535 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:43.527+0000 7f48ccafc700 1 -- 172.21.15.42:0/1667389664 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f48ac03adc0 msgr2=0x7f48ac03d2d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:43.536 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:43.527+0000 7f48ccafc700 1 --2- 172.21.15.42:0/1667389664 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f48ac03adc0 0x7f48ac03d2d0 crc :-1 s=READY pgs=1363 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:43.536 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:43.527+0000 7f48ccafc700 1 -- 172.21.15.42:0/1667389664 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f48c811b050 msgr2=0x7f48c808f5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:43.536 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:43.527+0000 7f48ccafc700 1 --2- 172.21.15.42:0/1667389664 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f48c811b050 0x7f48c808f5a0 secure :-1 s=READY pgs=3255 cs=0 l=1 rx=0x7f48c811b840 tx=0x7f48bc010cf0).stop 2019-12-05T06:48:43.536 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:43.527+0000 7f48ccafc700 1 -- 172.21.15.42:0/1667389664 shutdown_connections 2019-12-05T06:48:43.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:43.527+0000 7f48ccafc700 1 --2- 172.21.15.42:0/1667389664 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f48ac03adc0 0x7f48ac03d2d0 unknown :-1 s=CLOSED pgs=1363 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:43.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:43.527+0000 7f48ccafc700 1 --2- 172.21.15.42:0/1667389664 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f48c8144a80 0x7f48c8144eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:43.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:43.527+0000 7f48ccafc700 1 --2- 172.21.15.42:0/1667389664 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f48c811b050 0x7f48c808f5a0 unknown :-1 s=CLOSED pgs=3255 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:43.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:43.527+0000 7f48ccafc700 1 -- 172.21.15.42:0/1667389664 shutdown_connections 2019-12-05T06:48:43.537 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:43.527+0000 7f48ccafc700 1 -- 172.21.15.42:0/1667389664 wait complete. 2019-12-05T06:48:43.538 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:43.527+0000 7f48ccafc700 1 -- 172.21.15.42:0/1667389664 >> 172.21.15.42:0/1667389664 conn(0x7f48c807e4f0 msgr2=0x7f48c811c610 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:48:43.545 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:48:45.548 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:48:45.670 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:45.643+0000 7fa7a9cef700 1 Processor -- start 2019-12-05T06:48:45.671 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:45.643+0000 7fa7a9cef700 1 -- start start 2019-12-05T06:48:45.671 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:45.647+0000 7fa7a9cef700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa7a40a7740 0x7fa7a408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:48:45.671 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:45.647+0000 7fa7a9cef700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa7a407e720 0x7fa7a407eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:48:45.671 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:45.647+0000 7fa7a9cef700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fa7a409f2e0 0x7fa7a409f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:48:45.672 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:45.647+0000 7fa7a9cef700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7a40a0b00 con 0x7fa7a407e720 2019-12-05T06:48:45.672 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:45.647+0000 7fa7a9cef700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7a408e5b0 con 0x7fa7a409f2e0 2019-12-05T06:48:45.672 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:45.647+0000 7fa7a9cef700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7a408e850 con 0x7fa7a40a7740 2019-12-05T06:48:45.674 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:45.647+0000 7fa7a3fff700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa7a40a7740 0x7fa7a408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:45.674 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:45.647+0000 7fa7a3fff700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa7a40a7740 0x7fa7a408df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:37942/0 (socket says 172.21.15.42:37942) 2019-12-05T06:48:45.675 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:45.647+0000 7fa7a3fff700 1 -- 172.21.15.42:0/894885384 learned_addr learned my addr 172.21.15.42:0/894885384 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:48:45.675 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:45.647+0000 7fa7a8a8d700 1 --2- 172.21.15.42:0/894885384 >> v2:172.21.15.42:3300/0 conn(0x7fa7a409f2e0 0x7fa7a409f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:45.675 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:45.647+0000 7fa7a37fe700 1 --2- 172.21.15.42:0/894885384 >> v2:172.21.15.40:3300/0 conn(0x7fa7a407e720 0x7fa7a407eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:45.675 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:45.647+0000 7fa7a3fff700 1 -- 172.21.15.42:0/894885384 >> v2:172.21.15.42:3300/0 conn(0x7fa7a409f2e0 msgr2=0x7fa7a409f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:48:45.675 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:45.647+0000 7fa7a3fff700 1 --2- 172.21.15.42:0/894885384 >> v2:172.21.15.42:3300/0 conn(0x7fa7a409f2e0 0x7fa7a409f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:48:45.676 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:45.647+0000 7fa7a3fff700 1 -- 172.21.15.42:0/894885384 >> v2:172.21.15.40:3300/0 conn(0x7fa7a407e720 msgr2=0x7fa7a407eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:48:45.676 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:45.647+0000 7fa7a3fff700 1 --2- 172.21.15.42:0/894885384 >> v2:172.21.15.40:3300/0 conn(0x7fa7a407e720 0x7fa7a407eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:48:45.676 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:45.647+0000 7fa7a3fff700 1 -- 172.21.15.42:0/894885384 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7a407f0a0 con 0x7fa7a40a7740 2019-12-05T06:48:45.676 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:45.647+0000 7fa7a3fff700 1 --2- 172.21.15.42:0/894885384 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa7a40a7740 0x7fa7a408df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fa7940032b0 tx=0x7fa794010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:45.676 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:45.647+0000 7fa7a3fff700 1 --2- 172.21.15.42:0/894885384 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa7a40a7740 0x7fa7a408df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fa7940032b0 tx=0x7fa794010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:45.677 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:45.647+0000 7fa7a3fff700 1 --2- 172.21.15.42:0/894885384 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa7a40a7740 0x7fa7a408df80 secure :-1 s=READY pgs=722 cs=0 l=1 rx=0x7fa7940032b0 tx=0x7fa794010130).ready entity=mon.2 client_cookie=e28a49f63fd5fd9e server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:48:45.677 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:45.647+0000 7fa7a3fff700 1 --2- 172.21.15.42:0/894885384 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa7a40a7740 0x7fa7a408df80 secure :-1 s=THROTTLE_DONE pgs=722 cs=0 l=1 rx=0x7fa7940032b0 tx=0x7fa794010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:45.677 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:45.647+0000 7fa7a2ffd700 1 -- 172.21.15.42:0/894885384 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa794010de0 con 0x7fa7a40a7740 2019-12-05T06:48:45.677 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:45.647+0000 7fa7a3fff700 1 --2- 172.21.15.42:0/894885384 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa7a40a7740 0x7fa7a408df80 secure :-1 s=THROTTLE_DONE pgs=722 cs=0 l=1 rx=0x7fa7940032b0 tx=0x7fa794010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:45.678 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:45.647+0000 7fa7a3fff700 1 --2- 172.21.15.42:0/894885384 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa7a40a7740 0x7fa7a408df80 secure :-1 s=THROTTLE_DONE pgs=722 cs=0 l=1 rx=0x7fa7940032b0 tx=0x7fa794010130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:45.678 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:45.647+0000 7fa7a2ffd700 1 -- 172.21.15.42:0/894885384 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa794010a00 con 0x7fa7a40a7740 2019-12-05T06:48:45.678 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:45.647+0000 7fa7a2ffd700 1 -- 172.21.15.42:0/894885384 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa7940114a0 con 0x7fa7a40a7740 2019-12-05T06:48:45.678 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:45.647+0000 7fa7a9cef700 1 -- 172.21.15.42:0/894885384 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa7a40a7740 msgr2=0x7fa7a408df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:45.678 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:45.647+0000 7fa7a9cef700 1 --2- 172.21.15.42:0/894885384 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa7a40a7740 0x7fa7a408df80 secure :-1 s=READY pgs=722 cs=0 l=1 rx=0x7fa7940032b0 tx=0x7fa794010130).stop 2019-12-05T06:48:45.679 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:45.647+0000 7fa7a9cef700 1 -- 172.21.15.42:0/894885384 shutdown_connections 2019-12-05T06:48:45.679 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:45.647+0000 7fa7a9cef700 1 --2- 172.21.15.42:0/894885384 >> v2:172.21.15.42:3300/0 conn(0x7fa7a409f2e0 0x7fa7a409f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:48:45.679 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:45.647+0000 7fa7a9cef700 1 --2- 172.21.15.42:0/894885384 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa7a40a7740 0x7fa7a408df80 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:45.679 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:45.647+0000 7fa7a9cef700 1 --2- 172.21.15.42:0/894885384 >> v2:172.21.15.40:3300/0 conn(0x7fa7a407e720 0x7fa7a407eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:48:45.679 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:45.647+0000 7fa7a9cef700 1 -- 172.21.15.42:0/894885384 shutdown_connections 2019-12-05T06:48:45.680 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:45.647+0000 7fa7a9cef700 1 -- 172.21.15.42:0/894885384 wait complete. 2019-12-05T06:48:45.680 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:45.647+0000 7fa7a9cef700 1 -- 172.21.15.42:0/894885384 >> 172.21.15.42:0/894885384 conn(0x7fa7a40aabe0 msgr2=0x7fa7a40a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:48:45.680 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:45.651+0000 7fa7a9cef700 1 Processor -- start 2019-12-05T06:48:45.680 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:45.651+0000 7fa7a9cef700 1 -- start start 2019-12-05T06:48:45.681 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:45.651+0000 7fa7a9cef700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa7a4124970 0x7fa7a411f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:48:45.681 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:45.651+0000 7fa7a9cef700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa7a4120130 0x7fa7a411f630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:48:45.681 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:45.651+0000 7fa7a9cef700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7a411fde0 con 0x7fa7a411a060 2019-12-05T06:48:45.681 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:45.651+0000 7fa7a9cef700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7a411bbf0 con 0x7fa7a4124970 2019-12-05T06:48:45.681 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:45.651+0000 7fa7a9cef700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7a411be90 con 0x7fa7a4120130 2019-12-05T06:48:45.682 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:45.655+0000 7fa7a37fe700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa7a4124970 0x7fa7a411f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:45.682 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:45.655+0000 7fa7a37fe700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa7a4124970 0x7fa7a411f0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.42:53532/0 (socket says 172.21.15.42:53532) 2019-12-05T06:48:45.682 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:45.655+0000 7fa7a37fe700 1 -- 172.21.15.42:0/2903706732 learned_addr learned my addr 172.21.15.42:0/2903706732 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:48:45.682 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:45.655+0000 7fa7a8a8d700 1 --2- 172.21.15.42:0/2903706732 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa7a4120130 0x7fa7a411f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:45.683 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:45.655+0000 7fa7a37fe700 1 -- 172.21.15.42:0/2903706732 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa7a4120130 msgr2=0x7fa7a411f630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:45.683 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:45.655+0000 7fa7a37fe700 1 --2- 172.21.15.42:0/2903706732 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa7a4120130 0x7fa7a411f630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:45.683 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:45.655+0000 7fa7a37fe700 1 -- 172.21.15.42:0/2903706732 >> v1:172.21.15.40:6789/0 conn(0x7fa7a411a060 legacy=0x7fa7a407db50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:45.683 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:45.655+0000 7fa7a0ff9700 1 -- 172.21.15.42:0/2903706732 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 116476444 0 0) 0x7fa7a411fde0 con 0x7fa7a411a060 2019-12-05T06:48:45.683 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:45.655+0000 7fa7a37fe700 1 -- 172.21.15.42:0/2903706732 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7a4138500 con 0x7fa7a4124970 2019-12-05T06:48:45.684 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:45.655+0000 7fa7a37fe700 1 --2- 172.21.15.42:0/2903706732 >> v2:172.21.15.40:3300/0 conn(0x7fa7a4124970 0x7fa7a411f0e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fa7a411bdb0 tx=0x7fa798010bf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:45.684 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:45.655+0000 7fa7a37fe700 1 --2- 172.21.15.42:0/2903706732 >> v2:172.21.15.40:3300/0 conn(0x7fa7a4124970 0x7fa7a411f0e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fa7a411bdb0 tx=0x7fa798010bf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:45.684 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:45.655+0000 7fa7a37fe700 1 --2- 172.21.15.42:0/2903706732 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7a4124970 0x7fa7a411f0e0 secure :-1 s=READY pgs=2469 cs=0 l=1 rx=0x7fa7a411bdb0 tx=0x7fa798010bf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:48:45.684 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:45.655+0000 7fa7a37fe700 1 --2- 172.21.15.42:0/2903706732 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7a4124970 0x7fa7a411f0e0 secure :-1 s=THROTTLE_DONE pgs=2469 cs=0 l=1 rx=0x7fa7a411bdb0 tx=0x7fa798010bf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:45.685 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:45.655+0000 7fa7a0ff9700 1 -- 172.21.15.42:0/2903706732 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa798005840 con 0x7fa7a4124970 2019-12-05T06:48:45.685 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:45.655+0000 7fa7a37fe700 1 --2- 172.21.15.42:0/2903706732 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7a4124970 0x7fa7a411f0e0 secure :-1 s=THROTTLE_DONE pgs=2469 cs=0 l=1 rx=0x7fa7a411bdb0 tx=0x7fa798010bf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:45.685 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:45.655+0000 7fa7a0ff9700 1 -- 172.21.15.42:0/2903706732 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa798007160 con 0x7fa7a4124970 2019-12-05T06:48:45.685 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:45.655+0000 7fa7a37fe700 1 --2- 172.21.15.42:0/2903706732 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7a4124970 0x7fa7a411f0e0 secure :-1 s=THROTTLE_DONE pgs=2469 cs=0 l=1 rx=0x7fa7a411bdb0 tx=0x7fa798010bf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:45.686 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:45.655+0000 7fa7a0ff9700 1 -- 172.21.15.42:0/2903706732 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa798012230 con 0x7fa7a4124970 2019-12-05T06:48:45.686 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:45.655+0000 7fa7a9cef700 1 -- 172.21.15.42:0/2903706732 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7a41374f0 con 0x7fa7a4124970 2019-12-05T06:48:45.686 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:45.655+0000 7fa7a9cef700 1 -- 172.21.15.42:0/2903706732 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7a4137aa0 con 0x7fa7a4124970 2019-12-05T06:48:45.686 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:45.655+0000 7fa7a37fe700 1 --2- 172.21.15.42:0/2903706732 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7a4124970 0x7fa7a411f0e0 secure :-1 s=THROTTLE_DONE pgs=2469 cs=0 l=1 rx=0x7fa7a411bdb0 tx=0x7fa798010bf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:45.686 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:45.655+0000 7fa7a0ff9700 1 -- 172.21.15.42:0/2903706732 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fa798012550 con 0x7fa7a4124970 2019-12-05T06:48:45.687 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:45.655+0000 7fa7a37fe700 1 --2- 172.21.15.42:0/2903706732 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7a4124970 0x7fa7a411f0e0 secure :-1 s=THROTTLE_DONE pgs=2469 cs=0 l=1 rx=0x7fa7a411bdb0 tx=0x7fa798010bf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:45.687 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:45.655+0000 7fa7a0ff9700 1 --2- 172.21.15.42:0/2903706732 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa78403acd0 0x7fa78403d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:48:45.687 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:45.655+0000 7fa7a0ff9700 1 -- 172.21.15.42:0/2903706732 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 12137+0+0 (secure 0 0 0) 0x7fa79804b1f0 con 0x7fa7a4124970 2019-12-05T06:48:45.687 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:45.659+0000 7fa7a3fff700 1 --2- 172.21.15.42:0/2903706732 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa78403acd0 0x7fa78403d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:45.688 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:45.659+0000 7fa7a3fff700 1 --2- 172.21.15.42:0/2903706732 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa78403acd0 0x7fa78403d1e0 crc :-1 s=READY pgs=1364 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:48:45.688 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:45.659+0000 7fa7a9cef700 1 -- 172.21.15.42:0/2903706732 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7a405d110 con 0x7fa7a4124970 2019-12-05T06:48:45.689 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:45.667+0000 7fa7a37fe700 1 --2- 172.21.15.42:0/2903706732 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7a4124970 0x7fa7a411f0e0 secure :-1 s=THROTTLE_DONE pgs=2469 cs=0 l=1 rx=0x7fa7a411bdb0 tx=0x7fa798010bf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:45.689 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:45.667+0000 7fa7a0ff9700 1 -- 172.21.15.42:0/2903706732 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fa7980321f0 con 0x7fa7a4124970 2019-12-05T06:48:45.842 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:45.835+0000 7fa7a9cef700 1 -- 172.21.15.42:0/2903706732 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa7a405b8c0 con 0x7fa7a4124970 2019-12-05T06:48:45.843 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:45.835+0000 7fa7a37fe700 1 --2- 172.21.15.42:0/2903706732 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7a4124970 0x7fa7a411f0e0 secure :-1 s=THROTTLE_DONE pgs=2469 cs=0 l=1 rx=0x7fa7a411bdb0 tx=0x7fa798010bf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:45.843 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:45.835+0000 7fa7a0ff9700 1 -- 172.21.15.42:0/2903706732 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 394 v394) v1 ==== 95+0+2114 (secure 0 0 0) 0x7fa798032450 con 0x7fa7a4124970 2019-12-05T06:48:45.845 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:48:45.845 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":394,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":12687,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/815316041","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":815316041},{"type":"v1","addr":"172.21.15.40:6833","nonce":815316041}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":394}],"filesystems":[{"mdsmap":{"epoch":393,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:48:31.390793+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":120,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12847},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12847":{"gid":12847,"name":"a","rank":0,"incarnation":390,"state":"up:active","state_seq":38,"addr":"172.21.15.42:6835/1449317789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1449317789},{"type":"v1","addr":"172.21.15.42:6835","nonce":1449317789}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:48:45.846 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 394 2019-12-05T06:48:45.848 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:45.839+0000 7fa7a9cef700 1 -- 172.21.15.42:0/2903706732 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa78403acd0 msgr2=0x7fa78403d1e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:45.848 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:45.839+0000 7fa7a9cef700 1 --2- 172.21.15.42:0/2903706732 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa78403acd0 0x7fa78403d1e0 crc :-1 s=READY pgs=1364 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:45.849 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:45.839+0000 7fa7a9cef700 1 -- 172.21.15.42:0/2903706732 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7a4124970 msgr2=0x7fa7a411f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:45.849 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:45.839+0000 7fa7a9cef700 1 --2- 172.21.15.42:0/2903706732 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7a4124970 0x7fa7a411f0e0 secure :-1 s=READY pgs=2469 cs=0 l=1 rx=0x7fa7a411bdb0 tx=0x7fa798010bf0).stop 2019-12-05T06:48:45.849 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:45.839+0000 7fa7a9cef700 1 -- 172.21.15.42:0/2903706732 shutdown_connections 2019-12-05T06:48:45.849 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:45.839+0000 7fa7a9cef700 1 --2- 172.21.15.42:0/2903706732 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fa78403acd0 0x7fa78403d1e0 unknown :-1 s=CLOSED pgs=1364 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:45.849 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:45.839+0000 7fa7a9cef700 1 --2- 172.21.15.42:0/2903706732 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa7a4120130 0x7fa7a411f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:45.849 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:45.839+0000 7fa7a9cef700 1 --2- 172.21.15.42:0/2903706732 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7a4124970 0x7fa7a411f0e0 unknown :-1 s=CLOSED pgs=2469 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:45.850 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:45.839+0000 7fa7a9cef700 1 -- 172.21.15.42:0/2903706732 shutdown_connections 2019-12-05T06:48:45.850 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:45.843+0000 7fa7a9cef700 1 -- 172.21.15.42:0/2903706732 wait complete. 2019-12-05T06:48:45.850 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:45.843+0000 7fa7a9cef700 1 -- 172.21.15.42:0/2903706732 >> 172.21.15.42:0/2903706732 conn(0x7fa7a40aabe0 msgr2=0x7fa7a40a9790 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:48:45.858 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:48:47.859 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:48:47.964 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:47.951+0000 7f14a730c700 1 Processor -- start 2019-12-05T06:48:47.965 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:47.951+0000 7f14a730c700 1 -- start start 2019-12-05T06:48:47.965 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:47.955+0000 7f14a730c700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f14a00a7740 0x7f14a008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:48:47.965 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:47.955+0000 7f14a730c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f14a007e720 0x7f14a007eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:48:47.965 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:47.955+0000 7f14a730c700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f14a009f2e0 0x7f14a009f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:48:47.965 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:47.955+0000 7f14a730c700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14a00a0b00 con 0x7f14a009f2e0 2019-12-05T06:48:47.965 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:47.955+0000 7f14a730c700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14a008e5b0 con 0x7f14a007e720 2019-12-05T06:48:47.966 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:47.955+0000 7f14a730c700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14a008e850 con 0x7f14a00a7740 2019-12-05T06:48:47.966 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:47.955+0000 7f14a50a8700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f14a007e720 0x7f14a007eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:47.966 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:47.955+0000 7f14a58a9700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f14a00a7740 0x7f14a008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:47.966 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:47.955+0000 7f14a50a8700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f14a007e720 0x7f14a007eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:37502/0 (socket says 172.21.15.42:37502) 2019-12-05T06:48:47.966 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:47.955+0000 7f14a50a8700 1 -- 172.21.15.42:0/624892291 learned_addr learned my addr 172.21.15.42:0/624892291 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:48:47.966 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:47.955+0000 7f14a60aa700 1 --2- 172.21.15.42:0/624892291 >> v2:172.21.15.40:3300/0 conn(0x7f14a009f2e0 0x7f14a009f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:47.966 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:47.955+0000 7f14a50a8700 1 -- 172.21.15.42:0/624892291 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f14a00a7740 msgr2=0x7f14a008df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:48:47.967 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:47.955+0000 7f14a50a8700 1 --2- 172.21.15.42:0/624892291 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f14a00a7740 0x7f14a008df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:48:47.967 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:47.955+0000 7f14a50a8700 1 -- 172.21.15.42:0/624892291 >> v2:172.21.15.40:3300/0 conn(0x7f14a009f2e0 msgr2=0x7f14a009f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:48:47.967 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:47.955+0000 7f14a50a8700 1 --2- 172.21.15.42:0/624892291 >> v2:172.21.15.40:3300/0 conn(0x7f14a009f2e0 0x7f14a009f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:48:47.967 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:47.955+0000 7f14a50a8700 1 -- 172.21.15.42:0/624892291 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14a007f0a0 con 0x7f14a007e720 2019-12-05T06:48:47.967 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:47.955+0000 7f14a50a8700 1 --2- 172.21.15.42:0/624892291 >> v2:172.21.15.42:3300/0 conn(0x7f14a007e720 0x7f14a007eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f14a008e770 tx=0x7f149c00fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:47.967 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:47.955+0000 7f14a50a8700 1 --2- 172.21.15.42:0/624892291 >> v2:172.21.15.42:3300/0 conn(0x7f14a007e720 0x7f14a007eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f14a008e770 tx=0x7f149c00fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:47.967 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:47.955+0000 7f14a50a8700 1 --2- 172.21.15.42:0/624892291 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f14a007e720 0x7f14a007eb50 secure :-1 s=READY pgs=3256 cs=0 l=1 rx=0x7f14a008e770 tx=0x7f149c00fdb0).ready entity=mon.1 client_cookie=123951bf1966b28e server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:48:47.968 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:47.955+0000 7f14a50a8700 1 --2- 172.21.15.42:0/624892291 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f14a007e720 0x7f14a007eb50 secure :-1 s=THROTTLE_DONE pgs=3256 cs=0 l=1 rx=0x7f14a008e770 tx=0x7f149c00fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:47.968 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:47.955+0000 7f14a48a7700 1 -- 172.21.15.42:0/624892291 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f149c008c90 con 0x7f14a007e720 2019-12-05T06:48:47.968 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:47.955+0000 7f14a50a8700 1 --2- 172.21.15.42:0/624892291 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f14a007e720 0x7f14a007eb50 secure :-1 s=THROTTLE_DONE pgs=3256 cs=0 l=1 rx=0x7f14a008e770 tx=0x7f149c00fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:47.968 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:47.955+0000 7f14a50a8700 1 --2- 172.21.15.42:0/624892291 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f14a007e720 0x7f14a007eb50 secure :-1 s=THROTTLE_DONE pgs=3256 cs=0 l=1 rx=0x7f14a008e770 tx=0x7f149c00fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:47.968 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:47.955+0000 7f14a48a7700 1 -- 172.21.15.42:0/624892291 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f149c0088b0 con 0x7f14a007e720 2019-12-05T06:48:47.968 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:47.955+0000 7f14a48a7700 1 -- 172.21.15.42:0/624892291 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f149c011b50 con 0x7f14a007e720 2019-12-05T06:48:47.969 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:47.955+0000 7f14a730c700 1 -- 172.21.15.42:0/624892291 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f14a007e720 msgr2=0x7f14a007eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:47.969 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:47.955+0000 7f14a730c700 1 --2- 172.21.15.42:0/624892291 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f14a007e720 0x7f14a007eb50 secure :-1 s=READY pgs=3256 cs=0 l=1 rx=0x7f14a008e770 tx=0x7f149c00fdb0).stop 2019-12-05T06:48:47.969 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:47.955+0000 7f14a730c700 1 -- 172.21.15.42:0/624892291 shutdown_connections 2019-12-05T06:48:47.969 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:47.955+0000 7f14a730c700 1 --2- 172.21.15.42:0/624892291 >> v2:172.21.15.40:3300/0 conn(0x7f14a009f2e0 0x7f14a009f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:48:47.969 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:47.955+0000 7f14a730c700 1 --2- 172.21.15.42:0/624892291 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f14a00a7740 0x7f14a008df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:48:47.969 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:47.955+0000 7f14a730c700 1 --2- 172.21.15.42:0/624892291 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f14a007e720 0x7f14a007eb50 unknown :-1 s=CLOSED pgs=3256 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:47.969 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:47.955+0000 7f14a730c700 1 -- 172.21.15.42:0/624892291 shutdown_connections 2019-12-05T06:48:47.970 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:47.955+0000 7f14a730c700 1 -- 172.21.15.42:0/624892291 wait complete. 2019-12-05T06:48:47.970 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:47.955+0000 7f14a730c700 1 -- 172.21.15.42:0/624892291 >> 172.21.15.42:0/624892291 conn(0x7f14a00aabe0 msgr2=0x7f14a00a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:48:47.973 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:47.959+0000 7f14a730c700 1 Processor -- start 2019-12-05T06:48:47.973 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:47.959+0000 7f14a730c700 1 -- start start 2019-12-05T06:48:47.973 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:47.963+0000 7f14a730c700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f14a011b610 0x7f14a01186e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:48:47.973 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:47.963+0000 7f14a730c700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f14a0135560 0x7f14a01379a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:48:47.973 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:47.963+0000 7f14a730c700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14a0145400 con 0x7f14a01241e0 2019-12-05T06:48:47.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:47.963+0000 7f14a730c700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14a0118d10 con 0x7f14a0135560 2019-12-05T06:48:47.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:47.963+0000 7f14a730c700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14a01433e0 con 0x7f14a011b610 2019-12-05T06:48:47.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:47.963+0000 7f14a50a8700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f14a01241e0 0x7f14a0142be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:37546/0 (socket says 172.21.15.42:37546) 2019-12-05T06:48:47.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:47.963+0000 7f14a50a8700 1 -- 172.21.15.42:0/118923946 learned_addr learned my addr 172.21.15.42:0/118923946 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:48:47.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:47.963+0000 7f14a60aa700 1 --2- 172.21.15.42:0/118923946 >> v2:172.21.15.40:3300/0 conn(0x7f14a0135560 0x7f14a01379a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:47.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:47.963+0000 7f14a58a9700 1 --2- 172.21.15.42:0/118923946 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f14a011b610 0x7f14a01186e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:47.974 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:47.963+0000 7f14927fc700 1 -- 172.21.15.42:0/118923946 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3351275019 0 0) 0x7f14a0145400 con 0x7f14a01241e0 2019-12-05T06:48:47.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:47.963+0000 7f14927fc700 1 -- 172.21.15.42:0/118923946 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14880030b0 con 0x7f14a01241e0 2019-12-05T06:48:47.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:47.963+0000 7f14a58a9700 1 -- 172.21.15.42:0/118923946 >> v2:172.21.15.40:3300/0 conn(0x7f14a0135560 msgr2=0x7f14a01379a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:47.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:47.963+0000 7f14a58a9700 1 --2- 172.21.15.42:0/118923946 >> v2:172.21.15.40:3300/0 conn(0x7f14a0135560 0x7f14a01379a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:47.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:47.963+0000 7f14a58a9700 1 -- 172.21.15.42:0/118923946 >> v1:172.21.15.42:6789/0 conn(0x7f14a01241e0 legacy=0x7f14a0142be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:47.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:47.963+0000 7f14a58a9700 1 -- 172.21.15.42:0/118923946 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14a007dfe0 con 0x7f14a011b610 2019-12-05T06:48:47.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:47.963+0000 7f14a60aa700 1 --2- 172.21.15.42:0/118923946 >> v2:172.21.15.40:3300/0 conn(0x7f14a0135560 0x7f14a01379a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:48:47.975 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:47.963+0000 7f14a58a9700 1 --2- 172.21.15.42:0/118923946 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f14a011b610 0x7f14a01186e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f1494002f30 tx=0x7f1494012bf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:47.976 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:47.963+0000 7f14a58a9700 1 --2- 172.21.15.42:0/118923946 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f14a011b610 0x7f14a01186e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f1494002f30 tx=0x7f1494012bf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:47.976 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:47.963+0000 7f14a58a9700 1 --2- 172.21.15.42:0/118923946 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f14a011b610 0x7f14a01186e0 secure :-1 s=READY pgs=723 cs=0 l=1 rx=0x7f1494002f30 tx=0x7f1494012bf0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:48:47.976 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:47.963+0000 7f14a58a9700 1 --2- 172.21.15.42:0/118923946 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f14a011b610 0x7f14a01186e0 secure :-1 s=THROTTLE_DONE pgs=723 cs=0 l=1 rx=0x7f1494002f30 tx=0x7f1494012bf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:47.976 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:47.963+0000 7f14927fc700 1 -- 172.21.15.42:0/118923946 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1494014ba0 con 0x7f14a011b610 2019-12-05T06:48:47.976 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:47.963+0000 7f14a58a9700 1 --2- 172.21.15.42:0/118923946 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f14a011b610 0x7f14a01186e0 secure :-1 s=THROTTLE_DONE pgs=723 cs=0 l=1 rx=0x7f1494002f30 tx=0x7f1494012bf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:47.976 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:47.963+0000 7f14a58a9700 1 --2- 172.21.15.42:0/118923946 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f14a011b610 0x7f14a01186e0 secure :-1 s=THROTTLE_DONE pgs=723 cs=0 l=1 rx=0x7f1494002f30 tx=0x7f1494012bf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:47.976 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:47.963+0000 7f14927fc700 1 -- 172.21.15.42:0/118923946 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14940147c0 con 0x7f14a011b610 2019-12-05T06:48:47.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:47.963+0000 7f14a730c700 1 -- 172.21.15.42:0/118923946 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14a0124610 con 0x7f14a011b610 2019-12-05T06:48:47.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:47.963+0000 7f14927fc700 1 -- 172.21.15.42:0/118923946 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1494005730 con 0x7f14a011b610 2019-12-05T06:48:47.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:47.963+0000 7f14a730c700 1 -- 172.21.15.42:0/118923946 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14a008eb70 con 0x7f14a011b610 2019-12-05T06:48:47.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:47.963+0000 7f14a58a9700 1 --2- 172.21.15.42:0/118923946 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f14a011b610 0x7f14a01186e0 secure :-1 s=THROTTLE_DONE pgs=723 cs=0 l=1 rx=0x7f1494002f30 tx=0x7f1494012bf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:47.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:47.967+0000 7f14927fc700 1 -- 172.21.15.42:0/118923946 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f1494005a50 con 0x7f14a011b610 2019-12-05T06:48:47.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:47.967+0000 7f14a58a9700 1 --2- 172.21.15.42:0/118923946 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f14a011b610 0x7f14a01186e0 secure :-1 s=THROTTLE_DONE pgs=723 cs=0 l=1 rx=0x7f1494002f30 tx=0x7f1494012bf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:47.977 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:47.967+0000 7f14927fc700 1 --2- 172.21.15.42:0/118923946 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f148803b210 0x7f148803d720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:48:47.978 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:47.967+0000 7f14927fc700 1 -- 172.21.15.42:0/118923946 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 12137+0+0 (secure 0 0 0) 0x7f149404d2d0 con 0x7f14a011b610 2019-12-05T06:48:47.978 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:47.967+0000 7f14a50a8700 1 --2- 172.21.15.42:0/118923946 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f148803b210 0x7f148803d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:47.978 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:47.967+0000 7f14a50a8700 1 --2- 172.21.15.42:0/118923946 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f148803b210 0x7f148803d720 crc :-1 s=READY pgs=1365 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:48:47.978 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:47.967+0000 7f14a730c700 1 -- 172.21.15.42:0/118923946 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14a005d110 con 0x7f14a011b610 2019-12-05T06:48:47.990 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:47.975+0000 7f14a58a9700 1 --2- 172.21.15.42:0/118923946 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f14a011b610 0x7f14a01186e0 secure :-1 s=THROTTLE_DONE pgs=723 cs=0 l=1 rx=0x7f1494002f30 tx=0x7f1494012bf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:47.990 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:47.975+0000 7f14927fc700 1 -- 172.21.15.42:0/118923946 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f14940342d0 con 0x7f14a011b610 2019-12-05T06:48:48.154 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:48.147+0000 7f14a730c700 1 -- 172.21.15.42:0/118923946 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f14a005b8c0 con 0x7f14a011b610 2019-12-05T06:48:48.155 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:48.147+0000 7f14a58a9700 1 --2- 172.21.15.42:0/118923946 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f14a011b610 0x7f14a01186e0 secure :-1 s=THROTTLE_DONE pgs=723 cs=0 l=1 rx=0x7f1494002f30 tx=0x7f1494012bf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:48.155 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:48.147+0000 7f14927fc700 1 -- 172.21.15.42:0/118923946 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 394 v394) v1 ==== 95+0+2114 (secure 0 0 0) 0x7f1494005d60 con 0x7f14a011b610 2019-12-05T06:48:48.157 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 394 2019-12-05T06:48:48.157 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:48:48.157 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":394,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":12687,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/815316041","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":815316041},{"type":"v1","addr":"172.21.15.40:6833","nonce":815316041}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":394}],"filesystems":[{"mdsmap":{"epoch":393,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:48:31.390793+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":120,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12847},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12847":{"gid":12847,"name":"a","rank":0,"incarnation":390,"state":"up:active","state_seq":38,"addr":"172.21.15.42:6835/1449317789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1449317789},{"type":"v1","addr":"172.21.15.42:6835","nonce":1449317789}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:48:48.160 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:48.151+0000 7f14a730c700 1 -- 172.21.15.42:0/118923946 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f148803b210 msgr2=0x7f148803d720 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:48.160 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:48.151+0000 7f14a730c700 1 --2- 172.21.15.42:0/118923946 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f148803b210 0x7f148803d720 crc :-1 s=READY pgs=1365 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:48.160 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:48.151+0000 7f14a730c700 1 -- 172.21.15.42:0/118923946 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f14a011b610 msgr2=0x7f14a01186e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:48.160 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:48.151+0000 7f14a730c700 1 --2- 172.21.15.42:0/118923946 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f14a011b610 0x7f14a01186e0 secure :-1 s=READY pgs=723 cs=0 l=1 rx=0x7f1494002f30 tx=0x7f1494012bf0).stop 2019-12-05T06:48:48.160 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:48.151+0000 7f14a730c700 1 -- 172.21.15.42:0/118923946 shutdown_connections 2019-12-05T06:48:48.160 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:48.151+0000 7f14a730c700 1 --2- 172.21.15.42:0/118923946 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f148803b210 0x7f148803d720 unknown :-1 s=CLOSED pgs=1365 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:48.161 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:48.151+0000 7f14a730c700 1 --2- 172.21.15.42:0/118923946 >> v2:172.21.15.40:3300/0 conn(0x7f14a0135560 0x7f14a01379a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:48.161 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:48.151+0000 7f14a730c700 1 --2- 172.21.15.42:0/118923946 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f14a011b610 0x7f14a01186e0 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:48.161 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:48.151+0000 7f14a730c700 1 -- 172.21.15.42:0/118923946 shutdown_connections 2019-12-05T06:48:48.161 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:48.151+0000 7f14a730c700 1 -- 172.21.15.42:0/118923946 wait complete. 2019-12-05T06:48:48.161 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:48.151+0000 7f14a730c700 1 -- 172.21.15.42:0/118923946 >> 172.21.15.42:0/118923946 conn(0x7f14a007e5d0 msgr2=0x7f14a011c520 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:48:48.175 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:48:50.176 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:48:50.285 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:50.271+0000 7f6a4eb47700 1 Processor -- start 2019-12-05T06:48:50.285 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:50.271+0000 7f6a4eb47700 1 -- start start 2019-12-05T06:48:50.285 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:50.271+0000 7f6a4eb47700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f6a480a7740 0x7f6a4808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:48:50.286 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:50.271+0000 7f6a4eb47700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a480a0710 con 0x7f6a4809fae0 2019-12-05T06:48:50.286 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:50.271+0000 7f6a4eb47700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a480a9ba0 con 0x7f6a4807e720 2019-12-05T06:48:50.286 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:50.271+0000 7f6a4eb47700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a480a0b00 con 0x7f6a480a7740 2019-12-05T06:48:50.286 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:50.271+0000 7f6a4c8e3700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f6a4807e720 0x7f6a4809f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:37554/0 (socket says 172.21.15.42:37554) 2019-12-05T06:48:50.286 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:50.271+0000 7f6a4c8e3700 1 -- 172.21.15.42:0/2009986738 learned_addr learned my addr 172.21.15.42:0/2009986738 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:48:50.286 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:50.271+0000 7f6a4d0e4700 1 --2- 172.21.15.42:0/2009986738 >> v2:172.21.15.40:3300/0 conn(0x7f6a480a7740 0x7f6a4808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:50.286 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:50.271+0000 7f6a3ffff700 1 -- 172.21.15.42:0/2009986738 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3116743568 0 0) 0x7f6a480a9ba0 con 0x7f6a4807e720 2019-12-05T06:48:50.287 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:50.271+0000 7f6a3ffff700 1 -- 172.21.15.42:0/2009986738 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a340030b0 con 0x7f6a4807e720 2019-12-05T06:48:50.287 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:50.275+0000 7f6a3ffff700 1 -- 172.21.15.42:0/2009986738 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2896772073 0 0) 0x7f6a480a0710 con 0x7f6a4809fae0 2019-12-05T06:48:50.287 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:50.275+0000 7f6a3ffff700 1 -- 172.21.15.42:0/2009986738 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a480a9ba0 con 0x7f6a4809fae0 2019-12-05T06:48:50.287 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:50.275+0000 7f6a4d0e4700 1 -- 172.21.15.42:0/2009986738 >> v1:172.21.15.42:6789/0 conn(0x7f6a4807e720 legacy=0x7f6a4809f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:50.287 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:50.275+0000 7f6a4d0e4700 1 -- 172.21.15.42:0/2009986738 >> v1:172.21.15.40:6789/0 conn(0x7f6a4809fae0 legacy=0x7f6a4814efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:50.287 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:50.275+0000 7f6a4d0e4700 1 -- 172.21.15.42:0/2009986738 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a4807eb50 con 0x7f6a480a7740 2019-12-05T06:48:50.288 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:50.275+0000 7f6a4d0e4700 1 --2- 172.21.15.42:0/2009986738 >> v2:172.21.15.40:3300/0 conn(0x7f6a480a7740 0x7f6a4808df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f6a44008790 tx=0x7f6a4400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:50.288 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:50.275+0000 7f6a4d0e4700 1 --2- 172.21.15.42:0/2009986738 >> v2:172.21.15.40:3300/0 conn(0x7f6a480a7740 0x7f6a4808df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f6a44008790 tx=0x7f6a4400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:50.288 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:50.275+0000 7f6a4d0e4700 1 --2- 172.21.15.42:0/2009986738 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6a480a7740 0x7f6a4808df80 secure :-1 s=READY pgs=2471 cs=0 l=1 rx=0x7f6a44008790 tx=0x7f6a4400fbf0).ready entity=mon.0 client_cookie=f7c16a65975e821c server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:48:50.288 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:50.275+0000 7f6a4d0e4700 1 --2- 172.21.15.42:0/2009986738 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6a480a7740 0x7f6a4808df80 secure :-1 s=THROTTLE_DONE pgs=2471 cs=0 l=1 rx=0x7f6a44008790 tx=0x7f6a4400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:50.288 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:50.275+0000 7f6a3ffff700 1 -- 172.21.15.42:0/2009986738 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6a44011270 con 0x7f6a480a7740 2019-12-05T06:48:50.288 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:50.275+0000 7f6a4d0e4700 1 --2- 172.21.15.42:0/2009986738 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6a480a7740 0x7f6a4808df80 secure :-1 s=THROTTLE_DONE pgs=2471 cs=0 l=1 rx=0x7f6a44008790 tx=0x7f6a4400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:50.288 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:50.275+0000 7f6a4d0e4700 1 --2- 172.21.15.42:0/2009986738 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6a480a7740 0x7f6a4808df80 secure :-1 s=THROTTLE_DONE pgs=2471 cs=0 l=1 rx=0x7f6a44008790 tx=0x7f6a4400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:50.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:50.275+0000 7f6a3ffff700 1 -- 172.21.15.42:0/2009986738 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a44008bc0 con 0x7f6a480a7740 2019-12-05T06:48:50.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:50.275+0000 7f6a3ffff700 1 -- 172.21.15.42:0/2009986738 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6a44011850 con 0x7f6a480a7740 2019-12-05T06:48:50.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:50.275+0000 7f6a4eb47700 1 -- 172.21.15.42:0/2009986738 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6a480a7740 msgr2=0x7f6a4808df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:50.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:50.275+0000 7f6a4eb47700 1 --2- 172.21.15.42:0/2009986738 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6a480a7740 0x7f6a4808df80 secure :-1 s=READY pgs=2471 cs=0 l=1 rx=0x7f6a44008790 tx=0x7f6a4400fbf0).stop 2019-12-05T06:48:50.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:50.275+0000 7f6a4eb47700 1 -- 172.21.15.42:0/2009986738 shutdown_connections 2019-12-05T06:48:50.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:50.275+0000 7f6a4eb47700 1 --2- 172.21.15.42:0/2009986738 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6a480a7740 0x7f6a4808df80 unknown :-1 s=CLOSED pgs=2471 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:50.289 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:50.275+0000 7f6a4eb47700 1 -- 172.21.15.42:0/2009986738 shutdown_connections 2019-12-05T06:48:50.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:50.275+0000 7f6a4eb47700 1 -- 172.21.15.42:0/2009986738 wait complete. 2019-12-05T06:48:50.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:50.275+0000 7f6a4eb47700 1 -- 172.21.15.42:0/2009986738 >> 172.21.15.42:0/2009986738 conn(0x7f6a480aabe0 msgr2=0x7f6a480a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:48:50.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:50.279+0000 7f6a4eb47700 1 Processor -- start 2019-12-05T06:48:50.290 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:50.279+0000 7f6a4eb47700 1 -- start start 2019-12-05T06:48:50.318 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:50.279+0000 7f6a4eb47700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6a481434a0 0x7f6a480a1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:48:50.318 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:50.279+0000 7f6a4eb47700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f6a48122140 0x7f6a48143da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:48:50.318 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:50.279+0000 7f6a4eb47700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a48122570 con 0x7f6a48142cc0 2019-12-05T06:48:50.322 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:50.279+0000 7f6a4eb47700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a48143b90 con 0x7f6a48122140 2019-12-05T06:48:50.322 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:50.279+0000 7f6a4eb47700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a481438d0 con 0x7f6a481434a0 2019-12-05T06:48:50.323 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:50.279+0000 7f6a4d8e5700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f6a48122140 0x7f6a48143da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:50.323 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:50.279+0000 7f6a4d0e4700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f6a48142cc0 0x7f6a4814f900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:47000/0 (socket says 172.21.15.42:47000) 2019-12-05T06:48:50.323 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:50.279+0000 7f6a4d0e4700 1 -- 172.21.15.42:0/2616022963 learned_addr learned my addr 172.21.15.42:0/2616022963 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:48:50.323 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:50.279+0000 7f6a4c8e3700 1 --2- 172.21.15.42:0/2616022963 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6a481434a0 0x7f6a480a1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:50.323 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:50.279+0000 7f6a4c8e3700 1 -- 172.21.15.42:0/2616022963 >> v2:172.21.15.40:3300/0 conn(0x7f6a48122140 msgr2=0x7f6a48143da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:50.323 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:50.279+0000 7f6a4c8e3700 1 --2- 172.21.15.42:0/2616022963 >> v2:172.21.15.40:3300/0 conn(0x7f6a48122140 0x7f6a48143da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:50.323 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:50.279+0000 7f6a4c8e3700 1 -- 172.21.15.42:0/2616022963 >> v1:172.21.15.40:6789/0 conn(0x7f6a48142cc0 legacy=0x7f6a4814f900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:50.324 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:50.279+0000 7f6a3dffb700 1 -- 172.21.15.42:0/2616022963 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2588735225 0 0) 0x7f6a48122570 con 0x7f6a48142cc0 2019-12-05T06:48:50.324 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:50.279+0000 7f6a4c8e3700 1 -- 172.21.15.42:0/2616022963 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a481442f0 con 0x7f6a481434a0 2019-12-05T06:48:50.324 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:50.279+0000 7f6a4d8e5700 1 --2- 172.21.15.42:0/2616022963 >> v2:172.21.15.40:3300/0 conn(0x7f6a48122140 0x7f6a48143da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:48:50.324 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:50.279+0000 7f6a4c8e3700 1 --2- 172.21.15.42:0/2616022963 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6a481434a0 0x7f6a480a1cc0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f6a380043a0 tx=0x7f6a380108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:50.324 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:50.279+0000 7f6a4c8e3700 1 --2- 172.21.15.42:0/2616022963 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6a481434a0 0x7f6a480a1cc0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f6a380043a0 tx=0x7f6a380108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:50.324 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:50.279+0000 7f6a4c8e3700 1 --2- 172.21.15.42:0/2616022963 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6a481434a0 0x7f6a480a1cc0 secure :-1 s=READY pgs=724 cs=0 l=1 rx=0x7f6a380043a0 tx=0x7f6a380108f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:48:50.324 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:50.283+0000 7f6a4c8e3700 1 --2- 172.21.15.42:0/2616022963 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6a481434a0 0x7f6a480a1cc0 secure :-1 s=THROTTLE_DONE pgs=724 cs=0 l=1 rx=0x7f6a380043a0 tx=0x7f6a380108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:50.325 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:50.283+0000 7f6a3dffb700 1 -- 172.21.15.42:0/2616022963 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6a38005b40 con 0x7f6a481434a0 2019-12-05T06:48:50.325 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:50.283+0000 7f6a4c8e3700 1 --2- 172.21.15.42:0/2616022963 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6a481434a0 0x7f6a480a1cc0 secure :-1 s=THROTTLE_DONE pgs=724 cs=0 l=1 rx=0x7f6a380043a0 tx=0x7f6a380108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:50.325 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:50.283+0000 7f6a4eb47700 1 -- 172.21.15.42:0/2616022963 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a481445c0 con 0x7f6a481434a0 2019-12-05T06:48:50.325 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:50.283+0000 7f6a3dffb700 1 -- 172.21.15.42:0/2616022963 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a38005760 con 0x7f6a481434a0 2019-12-05T06:48:50.325 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:50.283+0000 7f6a4c8e3700 1 --2- 172.21.15.42:0/2616022963 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6a481434a0 0x7f6a480a1cc0 secure :-1 s=THROTTLE_DONE pgs=724 cs=0 l=1 rx=0x7f6a380043a0 tx=0x7f6a380108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:50.325 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:50.283+0000 7f6a3dffb700 1 -- 172.21.15.42:0/2616022963 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6a380062b0 con 0x7f6a481434a0 2019-12-05T06:48:50.325 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:50.283+0000 7f6a4eb47700 1 -- 172.21.15.42:0/2616022963 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a4808eb70 con 0x7f6a481434a0 2019-12-05T06:48:50.326 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:50.283+0000 7f6a4c8e3700 1 --2- 172.21.15.42:0/2616022963 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6a481434a0 0x7f6a480a1cc0 secure :-1 s=THROTTLE_DONE pgs=724 cs=0 l=1 rx=0x7f6a380043a0 tx=0x7f6a380108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:50.326 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:50.283+0000 7f6a3dffb700 1 -- 172.21.15.42:0/2616022963 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f6a38006590 con 0x7f6a481434a0 2019-12-05T06:48:50.326 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:50.283+0000 7f6a4c8e3700 1 --2- 172.21.15.42:0/2616022963 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6a481434a0 0x7f6a480a1cc0 secure :-1 s=THROTTLE_DONE pgs=724 cs=0 l=1 rx=0x7f6a380043a0 tx=0x7f6a380108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:50.326 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:50.283+0000 7f6a3dffb700 1 --2- 172.21.15.42:0/2616022963 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6a2803ad10 0x7f6a2803d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:48:50.326 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:50.283+0000 7f6a3dffb700 1 -- 172.21.15.42:0/2616022963 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 12137+0+0 (secure 0 0 0) 0x7f6a3804afc0 con 0x7f6a481434a0 2019-12-05T06:48:50.326 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:50.283+0000 7f6a4d0e4700 1 --2- 172.21.15.42:0/2616022963 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6a2803ad10 0x7f6a2803d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:50.326 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:50.283+0000 7f6a4d0e4700 1 --2- 172.21.15.42:0/2616022963 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6a2803ad10 0x7f6a2803d220 crc :-1 s=READY pgs=1366 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:48:50.327 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:50.287+0000 7f6a4eb47700 1 -- 172.21.15.42:0/2616022963 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a4805d110 con 0x7f6a481434a0 2019-12-05T06:48:50.327 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:50.295+0000 7f6a4c8e3700 1 --2- 172.21.15.42:0/2616022963 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6a481434a0 0x7f6a480a1cc0 secure :-1 s=THROTTLE_DONE pgs=724 cs=0 l=1 rx=0x7f6a380043a0 tx=0x7f6a380108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:50.327 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:50.295+0000 7f6a3dffb700 1 -- 172.21.15.42:0/2616022963 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f6a38031f80 con 0x7f6a481434a0 2019-12-05T06:48:50.471 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:50.463+0000 7f6a4eb47700 1 -- 172.21.15.42:0/2616022963 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6a4805b8c0 con 0x7f6a481434a0 2019-12-05T06:48:50.473 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:50.467+0000 7f6a4c8e3700 1 --2- 172.21.15.42:0/2616022963 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6a481434a0 0x7f6a480a1cc0 secure :-1 s=THROTTLE_DONE pgs=724 cs=0 l=1 rx=0x7f6a380043a0 tx=0x7f6a380108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:50.473 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:50.467+0000 7f6a3dffb700 1 -- 172.21.15.42:0/2616022963 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 394 v394) v1 ==== 95+0+2114 (secure 0 0 0) 0x7f6a380321e0 con 0x7f6a481434a0 2019-12-05T06:48:50.475 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 394 2019-12-05T06:48:50.475 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:48:50.475 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":394,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":12687,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/815316041","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":815316041},{"type":"v1","addr":"172.21.15.40:6833","nonce":815316041}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":394}],"filesystems":[{"mdsmap":{"epoch":393,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:48:31.390793+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":120,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12847},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12847":{"gid":12847,"name":"a","rank":0,"incarnation":390,"state":"up:active","state_seq":38,"addr":"172.21.15.42:6835/1449317789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1449317789},{"type":"v1","addr":"172.21.15.42:6835","nonce":1449317789}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:48:50.477 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:50.471+0000 7f6a4eb47700 1 -- 172.21.15.42:0/2616022963 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6a2803ad10 msgr2=0x7f6a2803d220 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:50.478 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:50.471+0000 7f6a4eb47700 1 --2- 172.21.15.42:0/2616022963 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6a2803ad10 0x7f6a2803d220 crc :-1 s=READY pgs=1366 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:50.478 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:50.471+0000 7f6a4eb47700 1 -- 172.21.15.42:0/2616022963 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6a481434a0 msgr2=0x7f6a480a1cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:50.478 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:50.471+0000 7f6a4eb47700 1 --2- 172.21.15.42:0/2616022963 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6a481434a0 0x7f6a480a1cc0 secure :-1 s=READY pgs=724 cs=0 l=1 rx=0x7f6a380043a0 tx=0x7f6a380108f0).stop 2019-12-05T06:48:50.478 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:50.471+0000 7f6a4eb47700 1 -- 172.21.15.42:0/2616022963 shutdown_connections 2019-12-05T06:48:50.479 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:50.471+0000 7f6a4eb47700 1 --2- 172.21.15.42:0/2616022963 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6a2803ad10 0x7f6a2803d220 unknown :-1 s=CLOSED pgs=1366 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:50.479 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:50.471+0000 7f6a4eb47700 1 --2- 172.21.15.42:0/2616022963 >> v2:172.21.15.40:3300/0 conn(0x7f6a48122140 0x7f6a48143da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:50.479 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:50.471+0000 7f6a4eb47700 1 --2- 172.21.15.42:0/2616022963 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6a481434a0 0x7f6a480a1cc0 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:50.479 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:50.471+0000 7f6a4eb47700 1 -- 172.21.15.42:0/2616022963 shutdown_connections 2019-12-05T06:48:50.479 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:50.471+0000 7f6a4eb47700 1 -- 172.21.15.42:0/2616022963 wait complete. 2019-12-05T06:48:50.479 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:50.471+0000 7f6a4eb47700 1 -- 172.21.15.42:0/2616022963 >> 172.21.15.42:0/2616022963 conn(0x7f6a480aabe0 msgr2=0x7f6a4809f5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:48:50.496 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:48:50.548 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:48:50.552 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:48:52.497 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:48:52.611 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:52.599+0000 7fd254d4a700 1 Processor -- start 2019-12-05T06:48:52.612 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:52.599+0000 7fd254d4a700 1 -- start start 2019-12-05T06:48:52.612 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:52.599+0000 7fd254d4a700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd25009fae0 0x7fd25014fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:48:52.612 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:52.599+0000 7fd254d4a700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2500a0710 con 0x7fd25008df80 2019-12-05T06:48:52.612 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:52.599+0000 7fd254d4a700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd25007ef20 con 0x7fd2500a7740 2019-12-05T06:48:52.612 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:52.599+0000 7fd254d4a700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2500a0b00 con 0x7fd25009fae0 2019-12-05T06:48:52.612 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:52.599+0000 7fd24ed9d700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fd2500a7740 0x7fd25007e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:37574/0 (socket says 172.21.15.42:37574) 2019-12-05T06:48:52.613 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:52.599+0000 7fd24ed9d700 1 -- 172.21.15.42:0/3725982741 learned_addr learned my addr 172.21.15.42:0/3725982741 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:48:52.613 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:52.599+0000 7fd24f59e700 1 --2- 172.21.15.42:0/3725982741 >> v2:172.21.15.40:3300/0 conn(0x7fd25009fae0 0x7fd25014fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:52.613 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:52.599+0000 7fd24dd9b700 1 -- 172.21.15.42:0/3725982741 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 428741961 0 0) 0x7fd25007ef20 con 0x7fd2500a7740 2019-12-05T06:48:52.613 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:52.599+0000 7fd24dd9b700 1 -- 172.21.15.42:0/3725982741 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd23c0030b0 con 0x7fd2500a7740 2019-12-05T06:48:52.613 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:52.599+0000 7fd24dd9b700 1 -- 172.21.15.42:0/3725982741 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1057487013 0 0) 0x7fd2500a0710 con 0x7fd25008df80 2019-12-05T06:48:52.613 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:52.599+0000 7fd24f59e700 1 -- 172.21.15.42:0/3725982741 >> v1:172.21.15.42:6789/0 conn(0x7fd2500a7740 legacy=0x7fd25007e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:52.613 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:52.603+0000 7fd24f59e700 1 -- 172.21.15.42:0/3725982741 >> v1:172.21.15.40:6789/0 conn(0x7fd25008df80 legacy=0x7fd25009f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:52.614 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:52.603+0000 7fd24f59e700 1 -- 172.21.15.42:0/3725982741 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2500a9ba0 con 0x7fd25009fae0 2019-12-05T06:48:52.614 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:52.603+0000 7fd24f59e700 1 --2- 172.21.15.42:0/3725982741 >> v2:172.21.15.40:3300/0 conn(0x7fd25009fae0 0x7fd25014fff0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fd240001c90 tx=0x7fd238001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:52.614 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:52.603+0000 7fd24f59e700 1 --2- 172.21.15.42:0/3725982741 >> v2:172.21.15.40:3300/0 conn(0x7fd25009fae0 0x7fd25014fff0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fd240001c90 tx=0x7fd238001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:52.614 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:52.603+0000 7fd24f59e700 1 --2- 172.21.15.42:0/3725982741 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd25009fae0 0x7fd25014fff0 secure :-1 s=READY pgs=2474 cs=0 l=1 rx=0x7fd240001c90 tx=0x7fd238001150).ready entity=mon.0 client_cookie=289e3f65382faf28 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:48:52.614 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:52.603+0000 7fd24f59e700 1 --2- 172.21.15.42:0/3725982741 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd25009fae0 0x7fd25014fff0 secure :-1 s=THROTTLE_DONE pgs=2474 cs=0 l=1 rx=0x7fd240001c90 tx=0x7fd238001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:52.614 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:52.603+0000 7fd24dd9b700 1 -- 172.21.15.42:0/3725982741 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd240001a50 con 0x7fd25009fae0 2019-12-05T06:48:52.614 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:52.603+0000 7fd24f59e700 1 --2- 172.21.15.42:0/3725982741 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd25009fae0 0x7fd25014fff0 secure :-1 s=THROTTLE_DONE pgs=2474 cs=0 l=1 rx=0x7fd240001c90 tx=0x7fd238001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:52.615 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:52.603+0000 7fd24f59e700 1 --2- 172.21.15.42:0/3725982741 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd25009fae0 0x7fd25014fff0 secure :-1 s=THROTTLE_DONE pgs=2474 cs=0 l=1 rx=0x7fd240001c90 tx=0x7fd238001150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:52.620 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:52.603+0000 7fd24dd9b700 1 -- 172.21.15.42:0/3725982741 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd240001860 con 0x7fd25009fae0 2019-12-05T06:48:52.621 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:52.603+0000 7fd24dd9b700 1 -- 172.21.15.42:0/3725982741 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd248008b70 con 0x7fd25009fae0 2019-12-05T06:48:52.623 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:52.603+0000 7fd254d4a700 1 -- 172.21.15.42:0/3725982741 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd25009fae0 msgr2=0x7fd25014fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:52.623 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:52.603+0000 7fd254d4a700 1 --2- 172.21.15.42:0/3725982741 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd25009fae0 0x7fd25014fff0 secure :-1 s=READY pgs=2474 cs=0 l=1 rx=0x7fd240001c90 tx=0x7fd238001150).stop 2019-12-05T06:48:52.623 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:52.603+0000 7fd254d4a700 1 -- 172.21.15.42:0/3725982741 shutdown_connections 2019-12-05T06:48:52.623 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:52.603+0000 7fd254d4a700 1 --2- 172.21.15.42:0/3725982741 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd25009fae0 0x7fd25014fff0 unknown :-1 s=CLOSED pgs=2474 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:52.623 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:52.603+0000 7fd254d4a700 1 -- 172.21.15.42:0/3725982741 shutdown_connections 2019-12-05T06:48:52.624 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:52.603+0000 7fd254d4a700 1 -- 172.21.15.42:0/3725982741 wait complete. 2019-12-05T06:48:52.624 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:52.603+0000 7fd254d4a700 1 -- 172.21.15.42:0/3725982741 >> 172.21.15.42:0/3725982741 conn(0x7fd2500aabe0 msgr2=0x7fd2500a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:48:52.627 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:52.607+0000 7fd254d4a700 1 Processor -- start 2019-12-05T06:48:52.627 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:52.607+0000 7fd254d4a700 1 -- start start 2019-12-05T06:48:52.627 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:52.607+0000 7fd254d4a700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd2501333c0 0x7fd25011bed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:48:52.627 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:52.607+0000 7fd254d4a700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd2501161c0 0x7fd25011aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:48:52.628 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:52.607+0000 7fd254d4a700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd25011b020 0x7fd25008eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:48:52.628 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:52.607+0000 7fd254d4a700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2501337f0 con 0x7fd25011b020 2019-12-05T06:48:52.628 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:52.607+0000 7fd254d4a700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd25011c420 con 0x7fd2501161c0 2019-12-05T06:48:52.637 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:52.607+0000 7fd254d4a700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd25011b990 con 0x7fd2501333c0 2019-12-05T06:48:52.638 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:52.607+0000 7fd24e59c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd2501161c0 0x7fd25011aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:52.638 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:52.607+0000 7fd24e59c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd2501161c0 0x7fd25011aad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:37542/0 (socket says 172.21.15.42:37542) 2019-12-05T06:48:52.638 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:52.607+0000 7fd24e59c700 1 -- 172.21.15.42:0/3042453656 learned_addr learned my addr 172.21.15.42:0/3042453656 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:48:52.638 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:52.607+0000 7fd24ed9d700 1 --2- 172.21.15.42:0/3042453656 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd2501333c0 0x7fd25011bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:52.638 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:52.607+0000 7fd24f59e700 1 --2- 172.21.15.42:0/3042453656 >> v2:172.21.15.40:3300/0 conn(0x7fd25011b020 0x7fd25008eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:52.638 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:52.607+0000 7fd24e59c700 1 -- 172.21.15.42:0/3042453656 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd2501333c0 msgr2=0x7fd25011bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:52.639 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:52.607+0000 7fd24e59c700 1 --2- 172.21.15.42:0/3042453656 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd2501333c0 0x7fd25011bed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:52.639 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:52.607+0000 7fd24e59c700 1 -- 172.21.15.42:0/3042453656 >> v2:172.21.15.40:3300/0 conn(0x7fd25011b020 msgr2=0x7fd25008eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:52.639 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:52.607+0000 7fd24e59c700 1 --2- 172.21.15.42:0/3042453656 >> v2:172.21.15.40:3300/0 conn(0x7fd25011b020 0x7fd25008eb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:52.639 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:52.607+0000 7fd24e59c700 1 -- 172.21.15.42:0/3042453656 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd250117500 con 0x7fd2501161c0 2019-12-05T06:48:52.639 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:52.607+0000 7fd24e59c700 1 --2- 172.21.15.42:0/3042453656 >> v2:172.21.15.42:3300/0 conn(0x7fd2501161c0 0x7fd25011aad0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fd238000b80 tx=0x7fd238010a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:52.639 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:52.607+0000 7fd24e59c700 1 --2- 172.21.15.42:0/3042453656 >> v2:172.21.15.42:3300/0 conn(0x7fd2501161c0 0x7fd25011aad0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fd238000b80 tx=0x7fd238010a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:52.640 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:52.607+0000 7fd24e59c700 1 --2- 172.21.15.42:0/3042453656 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd2501161c0 0x7fd25011aad0 secure :-1 s=READY pgs=3260 cs=0 l=1 rx=0x7fd238000b80 tx=0x7fd238010a20).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:48:52.640 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:52.607+0000 7fd24e59c700 1 --2- 172.21.15.42:0/3042453656 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd2501161c0 0x7fd25011aad0 secure :-1 s=THROTTLE_DONE pgs=3260 cs=0 l=1 rx=0x7fd238000b80 tx=0x7fd238010a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:52.640 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:52.607+0000 7fd24e59c700 1 --2- 172.21.15.42:0/3042453656 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd2501161c0 0x7fd25011aad0 secure :-1 s=THROTTLE_DONE pgs=3260 cs=0 l=1 rx=0x7fd238000b80 tx=0x7fd238010a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:52.640 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:52.607+0000 7fd24e59c700 1 --2- 172.21.15.42:0/3042453656 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd2501161c0 0x7fd25011aad0 secure :-1 s=THROTTLE_DONE pgs=3260 cs=0 l=1 rx=0x7fd238000b80 tx=0x7fd238010a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:52.640 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:52.607+0000 7fd2477fe700 1 -- 172.21.15.42:0/3042453656 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd23800caa0 con 0x7fd2501161c0 2019-12-05T06:48:52.641 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:52.607+0000 7fd2477fe700 1 -- 172.21.15.42:0/3042453656 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd23800c6c0 con 0x7fd2501161c0 2019-12-05T06:48:52.641 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:52.607+0000 7fd254d4a700 1 -- 172.21.15.42:0/3042453656 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd25011b680 con 0x7fd2501161c0 2019-12-05T06:48:52.641 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:52.607+0000 7fd2477fe700 1 -- 172.21.15.42:0/3042453656 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd238012290 con 0x7fd2501161c0 2019-12-05T06:48:52.641 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:52.607+0000 7fd254d4a700 1 -- 172.21.15.42:0/3042453656 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2501165f0 con 0x7fd2501161c0 2019-12-05T06:48:52.641 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:52.611+0000 7fd24e59c700 1 --2- 172.21.15.42:0/3042453656 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd2501161c0 0x7fd25011aad0 secure :-1 s=THROTTLE_DONE pgs=3260 cs=0 l=1 rx=0x7fd238000b80 tx=0x7fd238010a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:52.641 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:52.611+0000 7fd2477fe700 1 -- 172.21.15.42:0/3042453656 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fd2380125a0 con 0x7fd2501161c0 2019-12-05T06:48:52.642 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:52.611+0000 7fd24e59c700 1 --2- 172.21.15.42:0/3042453656 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd2501161c0 0x7fd25011aad0 secure :-1 s=THROTTLE_DONE pgs=3260 cs=0 l=1 rx=0x7fd238000b80 tx=0x7fd238010a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:52.642 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:52.611+0000 7fd2477fe700 1 --2- 172.21.15.42:0/3042453656 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd23403aec0 0x7fd23403d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:48:52.642 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:52.611+0000 7fd2477fe700 1 -- 172.21.15.42:0/3042453656 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 12137+0+0 (secure 0 0 0) 0x7fd23804b3e0 con 0x7fd2501161c0 2019-12-05T06:48:52.642 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:52.611+0000 7fd24ed9d700 1 --2- 172.21.15.42:0/3042453656 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd23403aec0 0x7fd23403d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:52.642 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:52.611+0000 7fd24ed9d700 1 --2- 172.21.15.42:0/3042453656 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd23403aec0 0x7fd23403d3d0 crc :-1 s=READY pgs=1367 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:48:52.642 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:52.615+0000 7fd254d4a700 1 -- 172.21.15.42:0/3042453656 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd25005d110 con 0x7fd2501161c0 2019-12-05T06:48:52.643 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:52.623+0000 7fd24e59c700 1 --2- 172.21.15.42:0/3042453656 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd2501161c0 0x7fd25011aad0 secure :-1 s=THROTTLE_DONE pgs=3260 cs=0 l=1 rx=0x7fd238000b80 tx=0x7fd238010a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:52.643 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:52.623+0000 7fd2477fe700 1 -- 172.21.15.42:0/3042453656 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fd250117500 con 0x7fd2501161c0 2019-12-05T06:48:52.793 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:52.787+0000 7fd254d4a700 1 -- 172.21.15.42:0/3042453656 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd25005b8c0 con 0x7fd2501161c0 2019-12-05T06:48:52.795 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:52.787+0000 7fd24e59c700 1 --2- 172.21.15.42:0/3042453656 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd2501161c0 0x7fd25011aad0 secure :-1 s=THROTTLE_DONE pgs=3260 cs=0 l=1 rx=0x7fd238000b80 tx=0x7fd238010a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:52.795 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:52.787+0000 7fd2477fe700 1 -- 172.21.15.42:0/3042453656 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 394 v394) v1 ==== 95+0+2114 (secure 0 0 0) 0x7fd238032420 con 0x7fd2501161c0 2019-12-05T06:48:52.795 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 394 2019-12-05T06:48:52.795 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:48:52.797 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":394,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":12687,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/815316041","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":815316041},{"type":"v1","addr":"172.21.15.40:6833","nonce":815316041}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":394}],"filesystems":[{"mdsmap":{"epoch":393,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:48:31.390793+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":120,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12847},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12847":{"gid":12847,"name":"a","rank":0,"incarnation":390,"state":"up:active","state_seq":38,"addr":"172.21.15.42:6835/1449317789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1449317789},{"type":"v1","addr":"172.21.15.42:6835","nonce":1449317789}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:48:52.798 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:52.791+0000 7fd254d4a700 1 -- 172.21.15.42:0/3042453656 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd23403aec0 msgr2=0x7fd23403d3d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:52.799 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:52.791+0000 7fd254d4a700 1 --2- 172.21.15.42:0/3042453656 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd23403aec0 0x7fd23403d3d0 crc :-1 s=READY pgs=1367 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:52.799 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:52.791+0000 7fd254d4a700 1 -- 172.21.15.42:0/3042453656 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd2501161c0 msgr2=0x7fd25011aad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:52.799 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:52.791+0000 7fd254d4a700 1 --2- 172.21.15.42:0/3042453656 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd2501161c0 0x7fd25011aad0 secure :-1 s=READY pgs=3260 cs=0 l=1 rx=0x7fd238000b80 tx=0x7fd238010a20).stop 2019-12-05T06:48:52.799 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:52.791+0000 7fd254d4a700 1 -- 172.21.15.42:0/3042453656 shutdown_connections 2019-12-05T06:48:52.799 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:52.791+0000 7fd254d4a700 1 --2- 172.21.15.42:0/3042453656 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd23403aec0 0x7fd23403d3d0 unknown :-1 s=CLOSED pgs=1367 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:52.799 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:52.791+0000 7fd254d4a700 1 --2- 172.21.15.42:0/3042453656 >> v2:172.21.15.40:3300/0 conn(0x7fd25011b020 0x7fd25008eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:52.800 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:52.791+0000 7fd254d4a700 1 --2- 172.21.15.42:0/3042453656 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd2501333c0 0x7fd25011bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:52.800 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:52.791+0000 7fd254d4a700 1 --2- 172.21.15.42:0/3042453656 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd2501161c0 0x7fd25011aad0 unknown :-1 s=CLOSED pgs=3260 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:52.800 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:52.791+0000 7fd254d4a700 1 -- 172.21.15.42:0/3042453656 shutdown_connections 2019-12-05T06:48:52.800 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:52.791+0000 7fd254d4a700 1 -- 172.21.15.42:0/3042453656 wait complete. 2019-12-05T06:48:52.800 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:52.791+0000 7fd254d4a700 1 -- 172.21.15.42:0/3042453656 >> 172.21.15.42:0/3042453656 conn(0x7fd2500a82a0 msgr2=0x7fd25007e650 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:48:52.808 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:48:54.812 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:48:54.916 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:54.903+0000 7f5f3db58700 1 Processor -- start 2019-12-05T06:48:54.916 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:54.903+0000 7f5f3db58700 1 -- start start 2019-12-05T06:48:54.916 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:54.903+0000 7f5f3db58700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5f3808df80 0x7f5f3809f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:48:54.916 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:54.903+0000 7f5f3db58700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5f3809f830 0x7f5f3814efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:48:54.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:54.903+0000 7f5f3db58700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f3809fc60 con 0x7f5f380a7740 2019-12-05T06:48:54.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:54.903+0000 7f5f3db58700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f380a0b00 con 0x7f5f3809f830 2019-12-05T06:48:54.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:54.903+0000 7f5f3db58700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f3808e5b0 con 0x7f5f3808df80 2019-12-05T06:48:54.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:54.907+0000 7f5f3c8f6700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5f3809f830 0x7f5f3814efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:54.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:54.907+0000 7f5f3c8f6700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5f3809f830 0x7f5f3814efe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:37556/0 (socket says 172.21.15.42:37556) 2019-12-05T06:48:54.917 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:54.907+0000 7f5f3c8f6700 1 -- 172.21.15.42:0/3257972657 learned_addr learned my addr 172.21.15.42:0/3257972657 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:48:54.918 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:54.907+0000 7f5f377fe700 1 --2- 172.21.15.42:0/3257972657 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5f3808df80 0x7f5f3809f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:54.918 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:54.907+0000 7f5f3c8f6700 1 -- 172.21.15.42:0/3257972657 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5f3808df80 msgr2=0x7f5f3809f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:48:54.918 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:54.907+0000 7f5f3c8f6700 1 --2- 172.21.15.42:0/3257972657 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5f3808df80 0x7f5f3809f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:48:54.918 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:54.907+0000 7f5f3c8f6700 1 -- 172.21.15.42:0/3257972657 >> v1:172.21.15.40:6789/0 conn(0x7f5f380a7740 legacy=0x7f5f3807e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:54.918 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:54.907+0000 7f5f3c8f6700 1 -- 172.21.15.42:0/3257972657 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f380a0710 con 0x7f5f3809f830 2019-12-05T06:48:54.918 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:54.907+0000 7f5f3c8f6700 1 --2- 172.21.15.42:0/3257972657 >> v2:172.21.15.42:3300/0 conn(0x7f5f3809f830 0x7f5f3814efe0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f5f30008790 tx=0x7f5f3000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:54.919 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:54.907+0000 7f5f3c8f6700 1 --2- 172.21.15.42:0/3257972657 >> v2:172.21.15.42:3300/0 conn(0x7f5f3809f830 0x7f5f3814efe0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f5f30008790 tx=0x7f5f3000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:54.919 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:54.907+0000 7f5f3c8f6700 1 --2- 172.21.15.42:0/3257972657 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5f3809f830 0x7f5f3814efe0 secure :-1 s=READY pgs=3261 cs=0 l=1 rx=0x7f5f30008790 tx=0x7f5f3000fd30).ready entity=mon.1 client_cookie=22a2ba0e483f98df server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:48:54.919 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:54.907+0000 7f5f3c8f6700 1 --2- 172.21.15.42:0/3257972657 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5f3809f830 0x7f5f3814efe0 secure :-1 s=THROTTLE_DONE pgs=3261 cs=0 l=1 rx=0x7f5f30008790 tx=0x7f5f3000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:54.919 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:54.907+0000 7f5f3c8f6700 1 --2- 172.21.15.42:0/3257972657 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5f3809f830 0x7f5f3814efe0 secure :-1 s=THROTTLE_DONE pgs=3261 cs=0 l=1 rx=0x7f5f30008790 tx=0x7f5f3000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:54.920 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:54.907+0000 7f5f36ffd700 1 -- 172.21.15.42:0/3257972657 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5f30008c90 con 0x7f5f3809f830 2019-12-05T06:48:54.920 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:54.907+0000 7f5f3c8f6700 1 --2- 172.21.15.42:0/3257972657 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5f3809f830 0x7f5f3814efe0 secure :-1 s=THROTTLE_DONE pgs=3261 cs=0 l=1 rx=0x7f5f30008790 tx=0x7f5f3000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:54.920 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:54.907+0000 7f5f36ffd700 1 -- 172.21.15.42:0/3257972657 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f300088b0 con 0x7f5f3809f830 2019-12-05T06:48:54.920 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:54.907+0000 7f5f36ffd700 1 -- 172.21.15.42:0/3257972657 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5f30011b10 con 0x7f5f3809f830 2019-12-05T06:48:54.920 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:54.907+0000 7f5f3db58700 1 -- 172.21.15.42:0/3257972657 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5f3809f830 msgr2=0x7f5f3814efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:54.921 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:54.907+0000 7f5f3db58700 1 --2- 172.21.15.42:0/3257972657 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5f3809f830 0x7f5f3814efe0 secure :-1 s=READY pgs=3261 cs=0 l=1 rx=0x7f5f30008790 tx=0x7f5f3000fd30).stop 2019-12-05T06:48:54.921 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:54.907+0000 7f5f3db58700 1 -- 172.21.15.42:0/3257972657 shutdown_connections 2019-12-05T06:48:54.921 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:54.907+0000 7f5f3db58700 1 --2- 172.21.15.42:0/3257972657 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5f3809f830 0x7f5f3814efe0 unknown :-1 s=CLOSED pgs=3261 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:54.921 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:54.907+0000 7f5f3db58700 1 --2- 172.21.15.42:0/3257972657 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5f3808df80 0x7f5f3809f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:48:54.921 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:54.907+0000 7f5f3db58700 1 -- 172.21.15.42:0/3257972657 shutdown_connections 2019-12-05T06:48:54.921 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:54.907+0000 7f5f3db58700 1 -- 172.21.15.42:0/3257972657 wait complete. 2019-12-05T06:48:54.922 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:54.907+0000 7f5f3db58700 1 -- 172.21.15.42:0/3257972657 >> 172.21.15.42:0/3257972657 conn(0x7f5f380aabe0 msgr2=0x7f5f380a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:48:54.925 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:54.911+0000 7f5f3db58700 1 Processor -- start 2019-12-05T06:48:54.925 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:54.911+0000 7f5f3db58700 1 -- start start 2019-12-05T06:48:54.926 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:54.911+0000 7f5f3db58700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5f381194d0 0x7f5f3811ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:48:54.926 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:54.911+0000 7f5f3db58700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5f3811b520 0x7f5f3811b950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:48:54.926 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:54.911+0000 7f5f3db58700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f381170e0 con 0x7f5f38142830 2019-12-05T06:48:54.926 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:54.911+0000 7f5f3db58700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f38143050 con 0x7f5f381194d0 2019-12-05T06:48:54.926 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:54.911+0000 7f5f3db58700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f381432f0 con 0x7f5f3811b520 2019-12-05T06:48:54.926 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:54.911+0000 7f5f3c8f6700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5f3811b520 0x7f5f3811b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:54.927 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:54.911+0000 7f5f3c8f6700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5f3811b520 0x7f5f3811b950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:37562/0 (socket says 172.21.15.42:37562) 2019-12-05T06:48:54.927 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:54.911+0000 7f5f3c8f6700 1 -- 172.21.15.42:0/3524166505 learned_addr learned my addr 172.21.15.42:0/3524166505 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:48:54.927 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:54.911+0000 7f5f37fff700 1 --2- 172.21.15.42:0/3524166505 >> v2:172.21.15.40:3300/0 conn(0x7f5f381194d0 0x7f5f3811ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:54.927 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:54.911+0000 7f5f3c8f6700 1 -- 172.21.15.42:0/3524166505 >> v2:172.21.15.40:3300/0 conn(0x7f5f381194d0 msgr2=0x7f5f3811ac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:54.928 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:54.911+0000 7f5f3c8f6700 1 --2- 172.21.15.42:0/3524166505 >> v2:172.21.15.40:3300/0 conn(0x7f5f381194d0 0x7f5f3811ac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:54.928 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:54.911+0000 7f5f3c8f6700 1 -- 172.21.15.42:0/3524166505 >> v1:172.21.15.40:6789/0 conn(0x7f5f38142830 legacy=0x7f5f3807d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:54.929 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:54.911+0000 7f5f3c8f6700 1 -- 172.21.15.42:0/3524166505 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f3811b260 con 0x7f5f3811b520 2019-12-05T06:48:54.929 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:54.911+0000 7f5f37fff700 1 --2- 172.21.15.42:0/3524166505 >> v2:172.21.15.40:3300/0 conn(0x7f5f381194d0 0x7f5f3811ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:48:54.929 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:54.911+0000 7f5f3c8f6700 1 --2- 172.21.15.42:0/3524166505 >> v2:172.21.15.42:3300/0 conn(0x7f5f3811b520 0x7f5f3811b950 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f5f38069ad0 tx=0x7f5f30008b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:54.929 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:54.911+0000 7f5f3c8f6700 1 --2- 172.21.15.42:0/3524166505 >> v2:172.21.15.42:3300/0 conn(0x7f5f3811b520 0x7f5f3811b950 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f5f38069ad0 tx=0x7f5f30008b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:54.929 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:54.911+0000 7f5f3c8f6700 1 --2- 172.21.15.42:0/3524166505 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5f3811b520 0x7f5f3811b950 secure :-1 s=READY pgs=3262 cs=0 l=1 rx=0x7f5f38069ad0 tx=0x7f5f30008b60).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:48:54.930 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:54.911+0000 7f5f3c8f6700 1 --2- 172.21.15.42:0/3524166505 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5f3811b520 0x7f5f3811b950 secure :-1 s=THROTTLE_DONE pgs=3262 cs=0 l=1 rx=0x7f5f38069ad0 tx=0x7f5f30008b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:54.930 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:54.911+0000 7f5f3c8f6700 1 --2- 172.21.15.42:0/3524166505 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5f3811b520 0x7f5f3811b950 secure :-1 s=THROTTLE_DONE pgs=3262 cs=0 l=1 rx=0x7f5f38069ad0 tx=0x7f5f30008b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:54.930 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:54.911+0000 7f5f34ff9700 1 -- 172.21.15.42:0/3524166505 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5f3000f470 con 0x7f5f3811b520 2019-12-05T06:48:54.930 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:54.911+0000 7f5f3c8f6700 1 --2- 172.21.15.42:0/3524166505 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5f3811b520 0x7f5f3811b950 secure :-1 s=THROTTLE_DONE pgs=3262 cs=0 l=1 rx=0x7f5f38069ad0 tx=0x7f5f30008b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:54.930 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:54.911+0000 7f5f34ff9700 1 -- 172.21.15.42:0/3524166505 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f30011730 con 0x7f5f3811b520 2019-12-05T06:48:54.931 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:54.911+0000 7f5f34ff9700 1 -- 172.21.15.42:0/3524166505 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5f30005730 con 0x7f5f3811b520 2019-12-05T06:48:54.931 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:54.911+0000 7f5f3db58700 1 -- 172.21.15.42:0/3524166505 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f3807e1f0 con 0x7f5f3811b520 2019-12-05T06:48:54.931 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:54.911+0000 7f5f3db58700 1 -- 172.21.15.42:0/3524166505 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f38142ce0 con 0x7f5f3811b520 2019-12-05T06:48:54.931 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:54.915+0000 7f5f3c8f6700 1 --2- 172.21.15.42:0/3524166505 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5f3811b520 0x7f5f3811b950 secure :-1 s=THROTTLE_DONE pgs=3262 cs=0 l=1 rx=0x7f5f38069ad0 tx=0x7f5f30008b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:54.931 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:54.915+0000 7f5f34ff9700 1 -- 172.21.15.42:0/3524166505 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f5f30005a40 con 0x7f5f3811b520 2019-12-05T06:48:54.931 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:54.915+0000 7f5f3c8f6700 1 --2- 172.21.15.42:0/3524166505 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5f3811b520 0x7f5f3811b950 secure :-1 s=THROTTLE_DONE pgs=3262 cs=0 l=1 rx=0x7f5f38069ad0 tx=0x7f5f30008b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:54.931 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:54.915+0000 7f5f34ff9700 1 --2- 172.21.15.42:0/3524166505 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5f1c03ae20 0x7f5f1c03d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:48:54.932 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:54.915+0000 7f5f34ff9700 1 -- 172.21.15.42:0/3524166505 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 12137+0+0 (secure 0 0 0) 0x7f5f3004b640 con 0x7f5f3811b520 2019-12-05T06:48:54.932 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:54.915+0000 7f5f37fff700 1 --2- 172.21.15.42:0/3524166505 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5f1c03ae20 0x7f5f1c03d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:54.932 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:54.915+0000 7f5f37fff700 1 --2- 172.21.15.42:0/3524166505 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5f1c03ae20 0x7f5f1c03d330 crc :-1 s=READY pgs=1368 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:48:54.932 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:54.919+0000 7f5f3db58700 1 -- 172.21.15.42:0/3524166505 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f3805d110 con 0x7f5f3811b520 2019-12-05T06:48:54.933 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:54.927+0000 7f5f3c8f6700 1 --2- 172.21.15.42:0/3524166505 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5f3811b520 0x7f5f3811b950 secure :-1 s=THROTTLE_DONE pgs=3262 cs=0 l=1 rx=0x7f5f38069ad0 tx=0x7f5f30008b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:54.933 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:54.927+0000 7f5f34ff9700 1 -- 172.21.15.42:0/3524166505 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f5f3811b260 con 0x7f5f3811b520 2019-12-05T06:48:55.099 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:55.091+0000 7f5f3db58700 1 -- 172.21.15.42:0/3524166505 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5f3805b8c0 con 0x7f5f3811b520 2019-12-05T06:48:55.109 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:55.091+0000 7f5f3c8f6700 1 --2- 172.21.15.42:0/3524166505 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5f3811b520 0x7f5f3811b950 secure :-1 s=THROTTLE_DONE pgs=3262 cs=0 l=1 rx=0x7f5f38069ad0 tx=0x7f5f30008b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:55.109 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:55.091+0000 7f5f34ff9700 1 -- 172.21.15.42:0/3524166505 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 394 v394) v1 ==== 95+0+2114 (secure 0 0 0) 0x7f5f30007660 con 0x7f5f3811b520 2019-12-05T06:48:55.109 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 394 2019-12-05T06:48:55.109 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:55.099+0000 7f5f3db58700 1 -- 172.21.15.42:0/3524166505 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5f1c03ae20 msgr2=0x7f5f1c03d330 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:55.110 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:55.099+0000 7f5f3db58700 1 --2- 172.21.15.42:0/3524166505 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5f1c03ae20 0x7f5f1c03d330 crc :-1 s=READY pgs=1368 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:55.110 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:55.099+0000 7f5f3db58700 1 -- 172.21.15.42:0/3524166505 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5f3811b520 msgr2=0x7f5f3811b950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:55.110 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:55.099+0000 7f5f3db58700 1 --2- 172.21.15.42:0/3524166505 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5f3811b520 0x7f5f3811b950 secure :-1 s=READY pgs=3262 cs=0 l=1 rx=0x7f5f38069ad0 tx=0x7f5f30008b60).stop 2019-12-05T06:48:55.110 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:55.099+0000 7f5f3db58700 1 -- 172.21.15.42:0/3524166505 shutdown_connections 2019-12-05T06:48:55.110 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:55.099+0000 7f5f3db58700 1 --2- 172.21.15.42:0/3524166505 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5f1c03ae20 0x7f5f1c03d330 unknown :-1 s=CLOSED pgs=1368 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:55.110 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:55.099+0000 7f5f3db58700 1 --2- 172.21.15.42:0/3524166505 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5f3811b520 0x7f5f3811b950 unknown :-1 s=CLOSED pgs=3262 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:55.110 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:55.099+0000 7f5f3db58700 1 --2- 172.21.15.42:0/3524166505 >> v2:172.21.15.40:3300/0 conn(0x7f5f381194d0 0x7f5f3811ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:55.111 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:55.099+0000 7f5f3db58700 1 -- 172.21.15.42:0/3524166505 shutdown_connections 2019-12-05T06:48:55.111 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:55.099+0000 7f5f3db58700 1 -- 172.21.15.42:0/3524166505 wait complete. 2019-12-05T06:48:55.111 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:55.099+0000 7f5f3db58700 1 -- 172.21.15.42:0/3524166505 >> 172.21.15.42:0/3524166505 conn(0x7f5f380a7f90 msgr2=0x7f5f38059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:48:55.111 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:48:55.111 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":394,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":12687,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/815316041","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":815316041},{"type":"v1","addr":"172.21.15.40:6833","nonce":815316041}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":394}],"filesystems":[{"mdsmap":{"epoch":393,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:48:31.390793+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":120,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12847},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12847":{"gid":12847,"name":"a","rank":0,"incarnation":390,"state":"up:active","state_seq":38,"addr":"172.21.15.42:6835/1449317789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1449317789},{"type":"v1","addr":"172.21.15.42:6835","nonce":1449317789}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:48:55.114 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:48:57.116 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:48:57.227 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:57.215+0000 7fd6076ca700 1 Processor -- start 2019-12-05T06:48:57.228 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:57.215+0000 7fd6076ca700 1 -- start start 2019-12-05T06:48:57.228 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:57.215+0000 7fd6076ca700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd60008df80 0x7fd60009f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:48:57.228 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:57.215+0000 7fd6076ca700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd60009f830 0x7fd60014efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:48:57.228 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:57.215+0000 7fd6076ca700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd60009fc60 con 0x7fd6000a7740 2019-12-05T06:48:57.228 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:57.215+0000 7fd6076ca700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6000a0b00 con 0x7fd60009f830 2019-12-05T06:48:57.228 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:57.215+0000 7fd6076ca700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd60008e5b0 con 0x7fd60008df80 2019-12-05T06:48:57.229 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:57.215+0000 7fd605c67700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fd6000a7740 0x7fd60007e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:37610/0 (socket says 172.21.15.42:37610) 2019-12-05T06:48:57.229 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:57.215+0000 7fd605c67700 1 -- 172.21.15.42:0/3328128635 learned_addr learned my addr 172.21.15.42:0/3328128635 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:48:57.229 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:57.215+0000 7fd605466700 1 --2- 172.21.15.42:0/3328128635 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd60008df80 0x7fd60009f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:57.229 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:57.215+0000 7fd606468700 1 --2- 172.21.15.42:0/3328128635 >> v2:172.21.15.40:3300/0 conn(0x7fd60009f830 0x7fd60014efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:57.229 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:57.219+0000 7fd604c65700 1 -- 172.21.15.42:0/3328128635 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 554860696 0 0) 0x7fd60009fc60 con 0x7fd6000a7740 2019-12-05T06:48:57.229 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:57.219+0000 7fd604c65700 1 -- 172.21.15.42:0/3328128635 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5ec0030b0 con 0x7fd6000a7740 2019-12-05T06:48:57.230 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:57.219+0000 7fd606468700 1 -- 172.21.15.42:0/3328128635 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd60008df80 msgr2=0x7fd60009f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:48:57.230 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:57.219+0000 7fd606468700 1 --2- 172.21.15.42:0/3328128635 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd60008df80 0x7fd60009f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:48:57.230 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:57.219+0000 7fd606468700 1 -- 172.21.15.42:0/3328128635 >> v1:172.21.15.42:6789/0 conn(0x7fd6000a7740 legacy=0x7fd60007e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:57.230 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:57.219+0000 7fd606468700 1 -- 172.21.15.42:0/3328128635 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6000a0710 con 0x7fd60009f830 2019-12-05T06:48:57.230 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:57.219+0000 7fd605466700 1 --2- 172.21.15.42:0/3328128635 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd60008df80 0x7fd60009f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:48:57.230 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:57.219+0000 7fd606468700 1 --2- 172.21.15.42:0/3328128635 >> v2:172.21.15.40:3300/0 conn(0x7fd60009f830 0x7fd60014efe0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fd5f8008b10 tx=0x7fd5f800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:57.230 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:57.219+0000 7fd606468700 1 --2- 172.21.15.42:0/3328128635 >> v2:172.21.15.40:3300/0 conn(0x7fd60009f830 0x7fd60014efe0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fd5f8008b10 tx=0x7fd5f800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:57.231 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:57.219+0000 7fd606468700 1 --2- 172.21.15.42:0/3328128635 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd60009f830 0x7fd60014efe0 secure :-1 s=READY pgs=2477 cs=0 l=1 rx=0x7fd5f8008b10 tx=0x7fd5f800fff0).ready entity=mon.0 client_cookie=8383e37201856766 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:48:57.231 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:57.219+0000 7fd606468700 1 --2- 172.21.15.42:0/3328128635 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd60009f830 0x7fd60014efe0 secure :-1 s=THROTTLE_DONE pgs=2477 cs=0 l=1 rx=0x7fd5f8008b10 tx=0x7fd5f800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:57.231 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:57.219+0000 7fd604c65700 1 -- 172.21.15.42:0/3328128635 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd5f8010900 con 0x7fd60009f830 2019-12-05T06:48:57.232 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:57.219+0000 7fd606468700 1 --2- 172.21.15.42:0/3328128635 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd60009f830 0x7fd60014efe0 secure :-1 s=THROTTLE_DONE pgs=2477 cs=0 l=1 rx=0x7fd5f8008b10 tx=0x7fd5f800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:57.232 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:57.219+0000 7fd606468700 1 --2- 172.21.15.42:0/3328128635 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd60009f830 0x7fd60014efe0 secure :-1 s=THROTTLE_DONE pgs=2477 cs=0 l=1 rx=0x7fd5f8008b10 tx=0x7fd5f800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:57.232 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:57.219+0000 7fd604c65700 1 -- 172.21.15.42:0/3328128635 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5f8008b50 con 0x7fd60009f830 2019-12-05T06:48:57.232 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:57.219+0000 7fd604c65700 1 -- 172.21.15.42:0/3328128635 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd5f80110c0 con 0x7fd60009f830 2019-12-05T06:48:57.233 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:57.219+0000 7fd6076ca700 1 -- 172.21.15.42:0/3328128635 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd60009f830 msgr2=0x7fd60014efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:57.233 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:57.219+0000 7fd6076ca700 1 --2- 172.21.15.42:0/3328128635 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd60009f830 0x7fd60014efe0 secure :-1 s=READY pgs=2477 cs=0 l=1 rx=0x7fd5f8008b10 tx=0x7fd5f800fff0).stop 2019-12-05T06:48:57.233 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:57.219+0000 7fd6076ca700 1 -- 172.21.15.42:0/3328128635 shutdown_connections 2019-12-05T06:48:57.233 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:57.219+0000 7fd6076ca700 1 --2- 172.21.15.42:0/3328128635 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd60009f830 0x7fd60014efe0 unknown :-1 s=CLOSED pgs=2477 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:57.233 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:57.219+0000 7fd6076ca700 1 --2- 172.21.15.42:0/3328128635 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd60008df80 0x7fd60009f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:48:57.233 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:57.219+0000 7fd6076ca700 1 -- 172.21.15.42:0/3328128635 shutdown_connections 2019-12-05T06:48:57.233 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:57.219+0000 7fd6076ca700 1 -- 172.21.15.42:0/3328128635 wait complete. 2019-12-05T06:48:57.234 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:57.219+0000 7fd6076ca700 1 -- 172.21.15.42:0/3328128635 >> 172.21.15.42:0/3328128635 conn(0x7fd6000aabe0 msgr2=0x7fd6000a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:48:57.237 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:57.223+0000 7fd6076ca700 1 Processor -- start 2019-12-05T06:48:57.238 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:57.223+0000 7fd6076ca700 1 -- start start 2019-12-05T06:48:57.238 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:57.223+0000 7fd6076ca700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd600122110 0x7fd600122540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:48:57.238 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:57.223+0000 7fd6076ca700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd600138890 0x7fd60013acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:48:57.238 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:57.223+0000 7fd6076ca700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd60007dfe0 con 0x7fd60011b4f0 2019-12-05T06:48:57.238 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:57.223+0000 7fd6076ca700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6001239c0 con 0x7fd600138890 2019-12-05T06:48:57.239 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:57.223+0000 7fd6076ca700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd600123c60 con 0x7fd600122110 2019-12-05T06:48:57.239 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:57.223+0000 7fd606468700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd600138890 0x7fd60013acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:57.239 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:57.223+0000 7fd605466700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd600122110 0x7fd600122540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:57.239 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:57.223+0000 7fd606468700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd600138890 0x7fd60013acd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.42:53630/0 (socket says 172.21.15.42:53630) 2019-12-05T06:48:57.239 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:57.223+0000 7fd606468700 1 -- 172.21.15.42:0/2146706085 learned_addr learned my addr 172.21.15.42:0/2146706085 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:48:57.245 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:57.223+0000 7fd606468700 1 -- 172.21.15.42:0/2146706085 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd600122110 msgr2=0x7fd600122540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:57.245 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:57.223+0000 7fd5f67fc700 1 -- 172.21.15.42:0/2146706085 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 693992328 0 0) 0x7fd60007dfe0 con 0x7fd60011b4f0 2019-12-05T06:48:57.245 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:57.223+0000 7fd606468700 1 --2- 172.21.15.42:0/2146706085 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd600122110 0x7fd600122540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:57.245 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:57.223+0000 7fd606468700 1 -- 172.21.15.42:0/2146706085 >> v1:172.21.15.40:6789/0 conn(0x7fd60011b4f0 legacy=0x7fd6001191b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:57.246 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:57.223+0000 7fd606468700 1 -- 172.21.15.42:0/2146706085 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd60011b920 con 0x7fd600138890 2019-12-05T06:48:57.246 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:57.223+0000 7fd605466700 1 --2- 172.21.15.42:0/2146706085 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd600122110 0x7fd600122540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:48:57.246 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:57.223+0000 7fd606468700 1 --2- 172.21.15.42:0/2146706085 >> v2:172.21.15.40:3300/0 conn(0x7fd600138890 0x7fd60013acd0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fd600123b80 tx=0x7fd5f8008870).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:57.246 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:57.223+0000 7fd606468700 1 --2- 172.21.15.42:0/2146706085 >> v2:172.21.15.40:3300/0 conn(0x7fd600138890 0x7fd60013acd0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fd600123b80 tx=0x7fd5f8008870).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:57.246 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:57.223+0000 7fd606468700 1 --2- 172.21.15.42:0/2146706085 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd600138890 0x7fd60013acd0 secure :-1 s=READY pgs=2479 cs=0 l=1 rx=0x7fd600123b80 tx=0x7fd5f8008870).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:48:57.247 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:57.223+0000 7fd606468700 1 --2- 172.21.15.42:0/2146706085 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd600138890 0x7fd60013acd0 secure :-1 s=THROTTLE_DONE pgs=2479 cs=0 l=1 rx=0x7fd600123b80 tx=0x7fd5f8008870).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:57.247 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:57.223+0000 7fd5f67fc700 1 -- 172.21.15.42:0/2146706085 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd5f8005ab0 con 0x7fd600138890 2019-12-05T06:48:57.247 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:57.223+0000 7fd606468700 1 --2- 172.21.15.42:0/2146706085 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd600138890 0x7fd60013acd0 secure :-1 s=THROTTLE_DONE pgs=2479 cs=0 l=1 rx=0x7fd600123b80 tx=0x7fd5f8008870).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:57.247 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:57.223+0000 7fd606468700 1 --2- 172.21.15.42:0/2146706085 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd600138890 0x7fd60013acd0 secure :-1 s=THROTTLE_DONE pgs=2479 cs=0 l=1 rx=0x7fd600123b80 tx=0x7fd5f8008870).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:57.247 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:57.223+0000 7fd5f67fc700 1 -- 172.21.15.42:0/2146706085 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5f8010b00 con 0x7fd600138890 2019-12-05T06:48:57.247 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:57.223+0000 7fd6076ca700 1 -- 172.21.15.42:0/2146706085 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd600123f00 con 0x7fd600138890 2019-12-05T06:48:57.248 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:57.223+0000 7fd5f67fc700 1 -- 172.21.15.42:0/2146706085 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd5f800f470 con 0x7fd600138890 2019-12-05T06:48:57.248 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:57.223+0000 7fd6076ca700 1 -- 172.21.15.42:0/2146706085 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd60011ac60 con 0x7fd600138890 2019-12-05T06:48:57.248 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:57.227+0000 7fd606468700 1 --2- 172.21.15.42:0/2146706085 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd600138890 0x7fd60013acd0 secure :-1 s=THROTTLE_DONE pgs=2479 cs=0 l=1 rx=0x7fd600123b80 tx=0x7fd5f8008870).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:57.248 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:57.227+0000 7fd5f67fc700 1 -- 172.21.15.42:0/2146706085 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fd5f800f670 con 0x7fd600138890 2019-12-05T06:48:57.248 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:57.227+0000 7fd606468700 1 --2- 172.21.15.42:0/2146706085 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd600138890 0x7fd60013acd0 secure :-1 s=THROTTLE_DONE pgs=2479 cs=0 l=1 rx=0x7fd600123b80 tx=0x7fd5f8008870).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:57.248 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:57.227+0000 7fd5f67fc700 1 --2- 172.21.15.42:0/2146706085 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd5e803ac60 0x7fd5e803d170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:48:57.249 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:57.227+0000 7fd5f67fc700 1 -- 172.21.15.42:0/2146706085 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 12137+0+0 (secure 0 0 0) 0x7fd5f804ac50 con 0x7fd600138890 2019-12-05T06:48:57.249 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:57.227+0000 7fd605c67700 1 --2- 172.21.15.42:0/2146706085 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd5e803ac60 0x7fd5e803d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:57.249 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:57.227+0000 7fd605c67700 1 --2- 172.21.15.42:0/2146706085 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd5e803ac60 0x7fd5e803d170 crc :-1 s=READY pgs=1369 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:48:57.249 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:57.231+0000 7fd6076ca700 1 -- 172.21.15.42:0/2146706085 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6001237b0 con 0x7fd600138890 2019-12-05T06:48:57.250 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:57.239+0000 7fd606468700 1 --2- 172.21.15.42:0/2146706085 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd600138890 0x7fd60013acd0 secure :-1 s=THROTTLE_DONE pgs=2479 cs=0 l=1 rx=0x7fd600123b80 tx=0x7fd5f8008870).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:57.250 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:57.239+0000 7fd5f67fc700 1 -- 172.21.15.42:0/2146706085 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fd5f8031cb0 con 0x7fd600138890 2019-12-05T06:48:57.418 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:57.411+0000 7fd6076ca700 1 -- 172.21.15.42:0/2146706085 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd60005d110 con 0x7fd600138890 2019-12-05T06:48:57.419 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:57.411+0000 7fd606468700 1 --2- 172.21.15.42:0/2146706085 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd600138890 0x7fd60013acd0 secure :-1 s=THROTTLE_DONE pgs=2479 cs=0 l=1 rx=0x7fd600123b80 tx=0x7fd5f8008870).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:57.420 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:57.411+0000 7fd5f67fc700 1 -- 172.21.15.42:0/2146706085 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 394 v394) v1 ==== 95+0+2114 (secure 0 0 0) 0x7fd5f800f980 con 0x7fd600138890 2019-12-05T06:48:57.422 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 394 2019-12-05T06:48:57.422 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:48:57.422 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":394,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":12687,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/815316041","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":815316041},{"type":"v1","addr":"172.21.15.40:6833","nonce":815316041}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":394}],"filesystems":[{"mdsmap":{"epoch":393,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:48:31.390793+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":120,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12847},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12847":{"gid":12847,"name":"a","rank":0,"incarnation":390,"state":"up:active","state_seq":38,"addr":"172.21.15.42:6835/1449317789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1449317789},{"type":"v1","addr":"172.21.15.42:6835","nonce":1449317789}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:48:57.425 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:57.415+0000 7fd6076ca700 1 -- 172.21.15.42:0/2146706085 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd5e803ac60 msgr2=0x7fd5e803d170 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:57.426 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:57.415+0000 7fd6076ca700 1 --2- 172.21.15.42:0/2146706085 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd5e803ac60 0x7fd5e803d170 crc :-1 s=READY pgs=1369 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:57.426 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:57.415+0000 7fd6076ca700 1 -- 172.21.15.42:0/2146706085 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd600138890 msgr2=0x7fd60013acd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:57.426 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:57.415+0000 7fd6076ca700 1 --2- 172.21.15.42:0/2146706085 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd600138890 0x7fd60013acd0 secure :-1 s=READY pgs=2479 cs=0 l=1 rx=0x7fd600123b80 tx=0x7fd5f8008870).stop 2019-12-05T06:48:57.426 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:57.419+0000 7fd6076ca700 1 -- 172.21.15.42:0/2146706085 shutdown_connections 2019-12-05T06:48:57.427 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:57.419+0000 7fd6076ca700 1 --2- 172.21.15.42:0/2146706085 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd5e803ac60 0x7fd5e803d170 unknown :-1 s=CLOSED pgs=1369 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:57.428 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:57.419+0000 7fd6076ca700 1 --2- 172.21.15.42:0/2146706085 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd600138890 0x7fd60013acd0 unknown :-1 s=CLOSED pgs=2479 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:57.428 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:57.419+0000 7fd6076ca700 1 --2- 172.21.15.42:0/2146706085 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd600122110 0x7fd600122540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:57.428 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:57.419+0000 7fd6076ca700 1 -- 172.21.15.42:0/2146706085 shutdown_connections 2019-12-05T06:48:57.429 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:57.419+0000 7fd6076ca700 1 -- 172.21.15.42:0/2146706085 wait complete. 2019-12-05T06:48:57.429 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:57.419+0000 7fd6076ca700 1 -- 172.21.15.42:0/2146706085 >> 172.21.15.42:0/2146706085 conn(0x7fd6000a9bc0 msgr2=0x7fd600059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:48:57.437 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:48:59.437 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:48:59.543 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:59.531+0000 7f4832455700 1 Processor -- start 2019-12-05T06:48:59.544 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:59.531+0000 7f4832455700 1 -- start start 2019-12-05T06:48:59.544 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:59.531+0000 7f4832455700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f482c0a7740 0x7f482c08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:48:59.544 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:59.531+0000 7f4832455700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f482c07e720 0x7f482c07eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:48:59.545 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:59.531+0000 7f4832455700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f482c0a0710 con 0x7f482c09f2e0 2019-12-05T06:48:59.545 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:59.531+0000 7f4832455700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f482c0a0b00 con 0x7f482c07e720 2019-12-05T06:48:59.545 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:59.531+0000 7f4832455700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f482c08e5b0 con 0x7f482c0a7740 2019-12-05T06:48:59.545 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:59.531+0000 7f482bfff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f482c07e720 0x7f482c07eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:59.546 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:59.531+0000 7f48311f3700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f482c09f2e0 0x7f482c09f710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:37626/0 (socket says 172.21.15.42:37626) 2019-12-05T06:48:59.546 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:59.531+0000 7f48311f3700 1 -- 172.21.15.42:0/1793591571 learned_addr learned my addr 172.21.15.42:0/1793591571 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:48:59.546 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:59.531+0000 7f48309f2700 1 --2- 172.21.15.42:0/1793591571 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f482c0a7740 0x7f482c08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:59.547 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:59.535+0000 7f482bfff700 1 -- 172.21.15.42:0/1793591571 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f482c0a7740 msgr2=0x7f482c08df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:48:59.547 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:59.535+0000 7f482bfff700 1 --2- 172.21.15.42:0/1793591571 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f482c0a7740 0x7f482c08df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:48:59.547 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:59.535+0000 7f482bfff700 1 -- 172.21.15.42:0/1793591571 >> v1:172.21.15.42:6789/0 conn(0x7f482c09f2e0 legacy=0x7f482c09f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:59.547 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:59.535+0000 7f482b7fe700 1 -- 172.21.15.42:0/1793591571 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2430681426 0 0) 0x7f482c0a0710 con 0x7f482c09f2e0 2019-12-05T06:48:59.548 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:59.535+0000 7f482bfff700 1 -- 172.21.15.42:0/1793591571 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f482c07f0a0 con 0x7f482c07e720 2019-12-05T06:48:59.548 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:59.535+0000 7f482bfff700 1 --2- 172.21.15.42:0/1793591571 >> v2:172.21.15.42:3300/0 conn(0x7f482c07e720 0x7f482c07eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f4820008790 tx=0x7f482000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:59.548 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:59.535+0000 7f482bfff700 1 --2- 172.21.15.42:0/1793591571 >> v2:172.21.15.42:3300/0 conn(0x7f482c07e720 0x7f482c07eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f4820008790 tx=0x7f482000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:59.549 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:59.535+0000 7f482bfff700 1 --2- 172.21.15.42:0/1793591571 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f482c07e720 0x7f482c07eb50 secure :-1 s=READY pgs=3265 cs=0 l=1 rx=0x7f4820008790 tx=0x7f482000fc70).ready entity=mon.1 client_cookie=b144ba7ae94d9b94 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:48:59.549 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:59.535+0000 7f482bfff700 1 --2- 172.21.15.42:0/1793591571 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f482c07e720 0x7f482c07eb50 secure :-1 s=THROTTLE_DONE pgs=3265 cs=0 l=1 rx=0x7f4820008790 tx=0x7f482000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:59.549 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:59.535+0000 7f482bfff700 1 --2- 172.21.15.42:0/1793591571 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f482c07e720 0x7f482c07eb50 secure :-1 s=THROTTLE_DONE pgs=3265 cs=0 l=1 rx=0x7f4820008790 tx=0x7f482000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:59.549 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:59.535+0000 7f482b7fe700 1 -- 172.21.15.42:0/1793591571 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f48200112f0 con 0x7f482c07e720 2019-12-05T06:48:59.550 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:59.535+0000 7f482bfff700 1 --2- 172.21.15.42:0/1793591571 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f482c07e720 0x7f482c07eb50 secure :-1 s=THROTTLE_DONE pgs=3265 cs=0 l=1 rx=0x7f4820008790 tx=0x7f482000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:59.550 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:59.535+0000 7f482b7fe700 1 -- 172.21.15.42:0/1793591571 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4820008ad0 con 0x7f482c07e720 2019-12-05T06:48:59.551 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:59.535+0000 7f482b7fe700 1 -- 172.21.15.42:0/1793591571 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f48200118d0 con 0x7f482c07e720 2019-12-05T06:48:59.551 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:59.535+0000 7f4832455700 1 -- 172.21.15.42:0/1793591571 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f482c07e720 msgr2=0x7f482c07eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:59.551 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:59.535+0000 7f4832455700 1 --2- 172.21.15.42:0/1793591571 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f482c07e720 0x7f482c07eb50 secure :-1 s=READY pgs=3265 cs=0 l=1 rx=0x7f4820008790 tx=0x7f482000fc70).stop 2019-12-05T06:48:59.552 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:59.535+0000 7f4832455700 1 -- 172.21.15.42:0/1793591571 shutdown_connections 2019-12-05T06:48:59.552 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:59.535+0000 7f4832455700 1 --2- 172.21.15.42:0/1793591571 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f482c0a7740 0x7f482c08df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:48:59.552 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:59.535+0000 7f4832455700 1 --2- 172.21.15.42:0/1793591571 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f482c07e720 0x7f482c07eb50 unknown :-1 s=CLOSED pgs=3265 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:59.552 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:59.535+0000 7f4832455700 1 -- 172.21.15.42:0/1793591571 shutdown_connections 2019-12-05T06:48:59.553 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:59.535+0000 7f4832455700 1 -- 172.21.15.42:0/1793591571 wait complete. 2019-12-05T06:48:59.553 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:59.535+0000 7f4832455700 1 -- 172.21.15.42:0/1793591571 >> 172.21.15.42:0/1793591571 conn(0x7f482c0aabe0 msgr2=0x7f482c0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:48:59.560 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:59.539+0000 7f4832455700 1 Processor -- start 2019-12-05T06:48:59.560 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:59.539+0000 7f4832455700 1 -- start start 2019-12-05T06:48:59.560 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:59.539+0000 7f4832455700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f482c117a40 0x7f482c08f260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:48:59.561 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:59.539+0000 7f4832455700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f482c122620 0x7f482c122a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:48:59.561 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:59.539+0000 7f4832455700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f482c08f7b0 con 0x7f482c120620 2019-12-05T06:48:59.561 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:59.539+0000 7f4832455700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f482c118270 con 0x7f482c122620 2019-12-05T06:48:59.561 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:59.539+0000 7f4832455700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f482c122380 con 0x7f482c117a40 2019-12-05T06:48:59.561 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:59.539+0000 7f482bfff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f482c122620 0x7f482c122a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:59.562 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:59.539+0000 7f482bfff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f482c122620 0x7f482c122a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:37596/0 (socket says 172.21.15.42:37596) 2019-12-05T06:48:59.562 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:59.539+0000 7f482bfff700 1 -- 172.21.15.42:0/769054443 learned_addr learned my addr 172.21.15.42:0/769054443 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:48:59.562 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:59.539+0000 7f48309f2700 1 --2- 172.21.15.42:0/769054443 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f482c117a40 0x7f482c08f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:59.562 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:59.539+0000 7f482bfff700 1 -- 172.21.15.42:0/769054443 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f482c117a40 msgr2=0x7f482c08f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:59.563 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:59.539+0000 7f482bfff700 1 --2- 172.21.15.42:0/769054443 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f482c117a40 0x7f482c08f260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:59.563 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:59.539+0000 7f482bfff700 1 -- 172.21.15.42:0/769054443 >> v1:172.21.15.40:6789/0 conn(0x7f482c120620 legacy=0x7f482c120a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:59.563 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:59.539+0000 7f48297fa700 1 -- 172.21.15.42:0/769054443 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3742453933 0 0) 0x7f482c08f7b0 con 0x7f482c120620 2019-12-05T06:48:59.563 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:59.539+0000 7f482bfff700 1 -- 172.21.15.42:0/769054443 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f482c07e160 con 0x7f482c122620 2019-12-05T06:48:59.564 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:59.543+0000 7f482bfff700 1 --2- 172.21.15.42:0/769054443 >> v2:172.21.15.42:3300/0 conn(0x7f482c122620 0x7f482c122a50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f482c118430 tx=0x7f4820011a70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:59.564 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:59.543+0000 7f482bfff700 1 --2- 172.21.15.42:0/769054443 >> v2:172.21.15.42:3300/0 conn(0x7f482c122620 0x7f482c122a50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f482c118430 tx=0x7f4820011a70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:59.564 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:59.543+0000 7f482bfff700 1 --2- 172.21.15.42:0/769054443 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f482c122620 0x7f482c122a50 secure :-1 s=READY pgs=3266 cs=0 l=1 rx=0x7f482c118430 tx=0x7f4820011a70).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:48:59.565 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:59.543+0000 7f482bfff700 1 --2- 172.21.15.42:0/769054443 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f482c122620 0x7f482c122a50 secure :-1 s=THROTTLE_DONE pgs=3266 cs=0 l=1 rx=0x7f482c118430 tx=0x7f4820011a70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:59.565 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:59.543+0000 7f48297fa700 1 -- 172.21.15.42:0/769054443 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4820007340 con 0x7f482c122620 2019-12-05T06:48:59.565 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:59.543+0000 7f482bfff700 1 --2- 172.21.15.42:0/769054443 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f482c122620 0x7f482c122a50 secure :-1 s=THROTTLE_DONE pgs=3266 cs=0 l=1 rx=0x7f482c118430 tx=0x7f4820011a70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:59.565 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:59.543+0000 7f482bfff700 1 --2- 172.21.15.42:0/769054443 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f482c122620 0x7f482c122a50 secure :-1 s=THROTTLE_DONE pgs=3266 cs=0 l=1 rx=0x7f482c118430 tx=0x7f4820011a70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:59.566 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:59.543+0000 7f48297fa700 1 -- 172.21.15.42:0/769054443 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48200114f0 con 0x7f482c122620 2019-12-05T06:48:59.566 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:59.543+0000 7f4832455700 1 -- 172.21.15.42:0/769054443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f482c117e70 con 0x7f482c122620 2019-12-05T06:48:59.566 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:59.543+0000 7f48297fa700 1 -- 172.21.15.42:0/769054443 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4820005540 con 0x7f482c122620 2019-12-05T06:48:59.567 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:59.543+0000 7f4832455700 1 -- 172.21.15.42:0/769054443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f482c123870 con 0x7f482c122620 2019-12-05T06:48:59.567 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:59.543+0000 7f482bfff700 1 --2- 172.21.15.42:0/769054443 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f482c122620 0x7f482c122a50 secure :-1 s=THROTTLE_DONE pgs=3266 cs=0 l=1 rx=0x7f482c118430 tx=0x7f4820011a70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:59.567 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:59.543+0000 7f48297fa700 1 -- 172.21.15.42:0/769054443 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f4820005850 con 0x7f482c122620 2019-12-05T06:48:59.567 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:59.543+0000 7f482bfff700 1 --2- 172.21.15.42:0/769054443 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f482c122620 0x7f482c122a50 secure :-1 s=THROTTLE_DONE pgs=3266 cs=0 l=1 rx=0x7f482c118430 tx=0x7f4820011a70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:59.567 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:59.543+0000 7f48297fa700 1 --2- 172.21.15.42:0/769054443 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f480c03ad10 0x7f480c03d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:48:59.568 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:59.543+0000 7f48297fa700 1 -- 172.21.15.42:0/769054443 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 12137+0+0 (secure 0 0 0) 0x7f482004b990 con 0x7f482c122620 2019-12-05T06:48:59.568 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:59.543+0000 7f48309f2700 1 --2- 172.21.15.42:0/769054443 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f480c03ad10 0x7f480c03d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:59.568 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:59.543+0000 7f48309f2700 1 --2- 172.21.15.42:0/769054443 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f480c03ad10 0x7f480c03d220 crc :-1 s=READY pgs=1370 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:48:59.569 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:59.547+0000 7f4832455700 1 -- 172.21.15.42:0/769054443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f482c05d110 con 0x7f482c122620 2019-12-05T06:48:59.570 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:59.555+0000 7f482bfff700 1 --2- 172.21.15.42:0/769054443 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f482c122620 0x7f482c122a50 secure :-1 s=THROTTLE_DONE pgs=3266 cs=0 l=1 rx=0x7f482c118430 tx=0x7f4820011a70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:59.570 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:59.555+0000 7f48297fa700 1 -- 172.21.15.42:0/769054443 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f4820032c70 con 0x7f482c122620 2019-12-05T06:48:59.732 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:59.723+0000 7f4832455700 1 -- 172.21.15.42:0/769054443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f482c05b8c0 con 0x7f482c122620 2019-12-05T06:48:59.733 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:59.727+0000 7f482bfff700 1 --2- 172.21.15.42:0/769054443 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f482c122620 0x7f482c122a50 secure :-1 s=THROTTLE_DONE pgs=3266 cs=0 l=1 rx=0x7f482c118430 tx=0x7f4820011a70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:59.733 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:59.727+0000 7f48297fa700 1 -- 172.21.15.42:0/769054443 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 394 v394) v1 ==== 95+0+2114 (secure 0 0 0) 0x7f4820012c40 con 0x7f482c122620 2019-12-05T06:48:59.734 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 394 2019-12-05T06:48:59.734 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:48:59.734 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":394,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":12687,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/815316041","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":815316041},{"type":"v1","addr":"172.21.15.40:6833","nonce":815316041}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":394}],"filesystems":[{"mdsmap":{"epoch":393,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:48:31.390793+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":120,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12847},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12847":{"gid":12847,"name":"a","rank":0,"incarnation":390,"state":"up:active","state_seq":38,"addr":"172.21.15.42:6835/1449317789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1449317789},{"type":"v1","addr":"172.21.15.42:6835","nonce":1449317789}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:48:59.735 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:59.727+0000 7f4832455700 1 -- 172.21.15.42:0/769054443 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f480c03ad10 msgr2=0x7f480c03d220 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:59.736 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:59.727+0000 7f4832455700 1 --2- 172.21.15.42:0/769054443 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f480c03ad10 0x7f480c03d220 crc :-1 s=READY pgs=1370 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:59.736 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:59.727+0000 7f4832455700 1 -- 172.21.15.42:0/769054443 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f482c122620 msgr2=0x7f482c122a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:59.736 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:59.727+0000 7f4832455700 1 --2- 172.21.15.42:0/769054443 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f482c122620 0x7f482c122a50 secure :-1 s=READY pgs=3266 cs=0 l=1 rx=0x7f482c118430 tx=0x7f4820011a70).stop 2019-12-05T06:48:59.736 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:59.727+0000 7f4832455700 1 -- 172.21.15.42:0/769054443 shutdown_connections 2019-12-05T06:48:59.736 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:59.727+0000 7f4832455700 1 --2- 172.21.15.42:0/769054443 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f480c03ad10 0x7f480c03d220 unknown :-1 s=CLOSED pgs=1370 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:59.736 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:59.727+0000 7f4832455700 1 --2- 172.21.15.42:0/769054443 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f482c117a40 0x7f482c08f260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:59.737 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:59.727+0000 7f4832455700 1 --2- 172.21.15.42:0/769054443 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f482c122620 0x7f482c122a50 unknown :-1 s=CLOSED pgs=3266 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:59.737 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:59.727+0000 7f4832455700 1 -- 172.21.15.42:0/769054443 shutdown_connections 2019-12-05T06:48:59.737 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:59.727+0000 7f4832455700 1 -- 172.21.15.42:0/769054443 wait complete. 2019-12-05T06:48:59.737 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:48:59.727+0000 7f4832455700 1 -- 172.21.15.42:0/769054443 >> 172.21.15.42:0/769054443 conn(0x7f482c0a8410 msgr2=0x7f482c059ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:48:59.746 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:49:01.746 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:49:01.848 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:01.835+0000 7f7aaf333700 1 Processor -- start 2019-12-05T06:49:01.848 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:01.835+0000 7f7aaf333700 1 -- start start 2019-12-05T06:49:01.849 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:01.839+0000 7f7aaf333700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7aa808df70 0x7f7aa809f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:49:01.849 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:01.839+0000 7f7aaf333700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7aa809f820 0x7f7aa814efd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:49:01.849 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:01.839+0000 7f7aaf333700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7aa809fc50 con 0x7f7aa80a7730 2019-12-05T06:49:01.849 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:01.839+0000 7f7aaf333700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7aa80a0af0 con 0x7f7aa808df70 2019-12-05T06:49:01.849 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:01.839+0000 7f7aaf333700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7aa808e5a0 con 0x7f7aa809f820 2019-12-05T06:49:01.849 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:01.839+0000 7f7aad8d0700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f7aa80a7730 0x7f7aa807e710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:37638/0 (socket says 172.21.15.42:37638) 2019-12-05T06:49:01.850 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:01.839+0000 7f7aad8d0700 1 -- 172.21.15.42:0/1590190721 learned_addr learned my addr 172.21.15.42:0/1590190721 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:49:01.850 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:01.839+0000 7f7aad0cf700 1 --2- 172.21.15.42:0/1590190721 >> v2:172.21.15.40:3300/0 conn(0x7f7aa808df70 0x7f7aa809f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:49:01.850 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:01.839+0000 7f7aae0d1700 1 --2- 172.21.15.42:0/1590190721 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7aa809f820 0x7f7aa814efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:49:01.850 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:01.839+0000 7f7aac8ce700 1 -- 172.21.15.42:0/1590190721 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 56761719 0 0) 0x7f7aa809fc50 con 0x7f7aa80a7730 2019-12-05T06:49:01.850 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:01.839+0000 7f7aac8ce700 1 -- 172.21.15.42:0/1590190721 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a940030b0 con 0x7f7aa80a7730 2019-12-05T06:49:01.850 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:01.839+0000 7f7aad0cf700 1 -- 172.21.15.42:0/1590190721 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7aa809f820 msgr2=0x7f7aa814efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:49:01.851 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:01.839+0000 7f7aad0cf700 1 --2- 172.21.15.42:0/1590190721 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7aa809f820 0x7f7aa814efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:49:01.851 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:01.839+0000 7f7aad0cf700 1 -- 172.21.15.42:0/1590190721 >> v1:172.21.15.42:6789/0 conn(0x7f7aa80a7730 legacy=0x7f7aa807e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:49:01.851 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:01.839+0000 7f7aad0cf700 1 -- 172.21.15.42:0/1590190721 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7aa80a0700 con 0x7f7aa808df70 2019-12-05T06:49:01.851 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:01.839+0000 7f7aad0cf700 1 --2- 172.21.15.42:0/1590190721 >> v2:172.21.15.40:3300/0 conn(0x7f7aa808df70 0x7f7aa809f2d0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f7a9c008790 tx=0x7f7a9c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:01.851 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:01.839+0000 7f7aad0cf700 1 --2- 172.21.15.42:0/1590190721 >> v2:172.21.15.40:3300/0 conn(0x7f7aa808df70 0x7f7aa809f2d0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f7a9c008790 tx=0x7f7a9c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:01.852 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:01.839+0000 7f7aad0cf700 1 --2- 172.21.15.42:0/1590190721 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7aa808df70 0x7f7aa809f2d0 secure :-1 s=READY pgs=2481 cs=0 l=1 rx=0x7f7a9c008790 tx=0x7f7a9c00fc70).ready entity=mon.0 client_cookie=7634058f49e3cb79 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:49:01.852 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:01.839+0000 7f7aad0cf700 1 --2- 172.21.15.42:0/1590190721 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7aa808df70 0x7f7aa809f2d0 secure :-1 s=THROTTLE_DONE pgs=2481 cs=0 l=1 rx=0x7f7a9c008790 tx=0x7f7a9c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:01.852 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:01.839+0000 7f7aac8ce700 1 -- 172.21.15.42:0/1590190721 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7a9c008bb0 con 0x7f7aa808df70 2019-12-05T06:49:01.852 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:01.839+0000 7f7aad0cf700 1 --2- 172.21.15.42:0/1590190721 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7aa808df70 0x7f7aa809f2d0 secure :-1 s=THROTTLE_DONE pgs=2481 cs=0 l=1 rx=0x7f7a9c008790 tx=0x7f7a9c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:01.853 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:01.839+0000 7f7aad0cf700 1 --2- 172.21.15.42:0/1590190721 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7aa808df70 0x7f7aa809f2d0 secure :-1 s=THROTTLE_DONE pgs=2481 cs=0 l=1 rx=0x7f7a9c008790 tx=0x7f7a9c00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:01.853 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:01.839+0000 7f7aac8ce700 1 -- 172.21.15.42:0/1590190721 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a9c0087d0 con 0x7f7aa808df70 2019-12-05T06:49:01.853 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:01.839+0000 7f7aac8ce700 1 -- 172.21.15.42:0/1590190721 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7a9c011860 con 0x7f7aa808df70 2019-12-05T06:49:01.853 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:01.839+0000 7f7aaf333700 1 -- 172.21.15.42:0/1590190721 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7aa808df70 msgr2=0x7f7aa809f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:49:01.853 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:01.839+0000 7f7aaf333700 1 --2- 172.21.15.42:0/1590190721 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7aa808df70 0x7f7aa809f2d0 secure :-1 s=READY pgs=2481 cs=0 l=1 rx=0x7f7a9c008790 tx=0x7f7a9c00fc70).stop 2019-12-05T06:49:01.853 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:01.839+0000 7f7aaf333700 1 -- 172.21.15.42:0/1590190721 shutdown_connections 2019-12-05T06:49:01.854 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:01.839+0000 7f7aaf333700 1 --2- 172.21.15.42:0/1590190721 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7aa809f820 0x7f7aa814efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:49:01.854 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:01.839+0000 7f7aaf333700 1 --2- 172.21.15.42:0/1590190721 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7aa808df70 0x7f7aa809f2d0 unknown :-1 s=CLOSED pgs=2481 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:49:01.854 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:01.839+0000 7f7aaf333700 1 -- 172.21.15.42:0/1590190721 shutdown_connections 2019-12-05T06:49:01.855 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:01.839+0000 7f7aaf333700 1 -- 172.21.15.42:0/1590190721 wait complete. 2019-12-05T06:49:01.855 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:01.839+0000 7f7aaf333700 1 -- 172.21.15.42:0/1590190721 >> 172.21.15.42:0/1590190721 conn(0x7f7aa80aabd0 msgr2=0x7f7aa80a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:49:01.859 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:01.843+0000 7f7aaf333700 1 Processor -- start 2019-12-05T06:49:01.859 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:01.843+0000 7f7aaf333700 1 -- start start 2019-12-05T06:49:01.859 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:01.847+0000 7f7aaf333700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7aa8130f20 0x7f7aa8138890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:49:01.859 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:01.847+0000 7f7aaf333700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7aa80a53e0 con 0x7f7aa81183c0 2019-12-05T06:49:01.859 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:01.847+0000 7f7aaf333700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7aa812ce70 con 0x7f7aa8130af0 2019-12-05T06:49:01.860 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:01.847+0000 7f7aaf333700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7aa811c4d0 con 0x7f7aa8130f20 2019-12-05T06:49:01.860 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:01.847+0000 7f7aad0cf700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f7aa8130af0 0x7f7aa80a4be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:37646/0 (socket says 172.21.15.42:37646) 2019-12-05T06:49:01.860 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:01.847+0000 7f7aad0cf700 1 -- 172.21.15.42:0/3873029570 learned_addr learned my addr 172.21.15.42:0/3873029570 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:49:01.860 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:01.847+0000 7f7aae0d1700 1 --2- 172.21.15.42:0/3873029570 >> v2:172.21.15.40:3300/0 conn(0x7f7aa8130f20 0x7f7aa8138890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:49:01.860 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:01.847+0000 7f7a9a7fc700 1 -- 172.21.15.42:0/3873029570 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2099401137 0 0) 0x7f7aa80a53e0 con 0x7f7aa81183c0 2019-12-05T06:49:01.860 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:01.847+0000 7f7a9a7fc700 1 -- 172.21.15.42:0/3873029570 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a900030b0 con 0x7f7aa81183c0 2019-12-05T06:49:01.860 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:01.847+0000 7f7a9a7fc700 1 -- 172.21.15.42:0/3873029570 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 249004767 0 0) 0x7f7aa812ce70 con 0x7f7aa8130af0 2019-12-05T06:49:01.861 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:01.847+0000 7f7a9a7fc700 1 -- 172.21.15.42:0/3873029570 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7aa80a53e0 con 0x7f7aa8130af0 2019-12-05T06:49:01.861 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:01.847+0000 7f7aae0d1700 1 -- 172.21.15.42:0/3873029570 >> v1:172.21.15.42:6789/0 conn(0x7f7aa8130af0 legacy=0x7f7aa80a4be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:49:01.861 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:01.847+0000 7f7a9a7fc700 1 -- 172.21.15.42:0/3873029570 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 2266916196 0 0) 0x7f7a900030b0 con 0x7f7aa81183c0 2019-12-05T06:49:01.861 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:01.847+0000 7f7aae0d1700 1 -- 172.21.15.42:0/3873029570 >> v1:172.21.15.40:6789/0 conn(0x7f7aa81183c0 legacy=0x7f7aa812d720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:49:01.861 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:01.847+0000 7f7aae0d1700 1 -- 172.21.15.42:0/3873029570 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7aa813ae00 con 0x7f7aa8130f20 2019-12-05T06:49:01.862 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:01.847+0000 7f7aae0d1700 1 --2- 172.21.15.42:0/3873029570 >> v2:172.21.15.40:3300/0 conn(0x7f7aa8130f20 0x7f7aa8138890 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f7aa40016a0 tx=0x7f7aa4001ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:01.862 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:01.847+0000 7f7aae0d1700 1 --2- 172.21.15.42:0/3873029570 >> v2:172.21.15.40:3300/0 conn(0x7f7aa8130f20 0x7f7aa8138890 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f7aa40016a0 tx=0x7f7aa4001ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:01.862 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:01.847+0000 7f7aae0d1700 1 --2- 172.21.15.42:0/3873029570 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7aa8130f20 0x7f7aa8138890 secure :-1 s=READY pgs=2483 cs=0 l=1 rx=0x7f7aa40016a0 tx=0x7f7aa4001ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:49:01.862 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:01.847+0000 7f7aae0d1700 1 --2- 172.21.15.42:0/3873029570 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7aa8130f20 0x7f7aa8138890 secure :-1 s=THROTTLE_DONE pgs=2483 cs=0 l=1 rx=0x7f7aa40016a0 tx=0x7f7aa4001ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:01.862 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:01.847+0000 7f7aae0d1700 1 --2- 172.21.15.42:0/3873029570 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7aa8130f20 0x7f7aa8138890 secure :-1 s=THROTTLE_DONE pgs=2483 cs=0 l=1 rx=0x7f7aa40016a0 tx=0x7f7aa4001ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:01.863 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:01.847+0000 7f7a9a7fc700 1 -- 172.21.15.42:0/3873029570 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7aa40018d0 con 0x7f7aa8130f20 2019-12-05T06:49:01.863 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:01.847+0000 7f7aae0d1700 1 --2- 172.21.15.42:0/3873029570 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7aa8130f20 0x7f7aa8138890 secure :-1 s=THROTTLE_DONE pgs=2483 cs=0 l=1 rx=0x7f7aa40016a0 tx=0x7f7aa4001ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:01.863 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:01.847+0000 7f7a9a7fc700 1 -- 172.21.15.42:0/3873029570 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7aa40016e0 con 0x7f7aa8130f20 2019-12-05T06:49:01.863 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:01.847+0000 7f7aaf333700 1 -- 172.21.15.42:0/3873029570 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7aa8138de0 con 0x7f7aa8130f20 2019-12-05T06:49:01.863 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:01.847+0000 7f7a9a7fc700 1 -- 172.21.15.42:0/3873029570 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7aa0012680 con 0x7f7aa8130f20 2019-12-05T06:49:01.863 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:01.847+0000 7f7aaf333700 1 -- 172.21.15.42:0/3873029570 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7aa8139330 con 0x7f7aa8130f20 2019-12-05T06:49:01.863 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:01.847+0000 7f7aae0d1700 1 --2- 172.21.15.42:0/3873029570 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7aa8130f20 0x7f7aa8138890 secure :-1 s=THROTTLE_DONE pgs=2483 cs=0 l=1 rx=0x7f7aa40016a0 tx=0x7f7aa4001ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:01.864 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:01.847+0000 7f7a9a7fc700 1 -- 172.21.15.42:0/3873029570 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f7aa0012990 con 0x7f7aa8130f20 2019-12-05T06:49:01.864 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:01.847+0000 7f7aae0d1700 1 --2- 172.21.15.42:0/3873029570 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7aa8130f20 0x7f7aa8138890 secure :-1 s=THROTTLE_DONE pgs=2483 cs=0 l=1 rx=0x7f7aa40016a0 tx=0x7f7aa4001ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:01.864 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:01.851+0000 7f7a9a7fc700 1 --2- 172.21.15.42:0/3873029570 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f7a9003b470 0x7f7a9003d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:49:01.864 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:01.851+0000 7f7a9a7fc700 1 -- 172.21.15.42:0/3873029570 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 12223+0+0 (secure 0 0 0) 0x7f7aa004aed0 con 0x7f7aa8130f20 2019-12-05T06:49:01.864 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:01.851+0000 7f7aad8d0700 1 --2- 172.21.15.42:0/3873029570 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f7a9003b470 0x7f7a9003d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:49:01.864 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:01.851+0000 7f7aad8d0700 1 --2- 172.21.15.42:0/3873029570 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f7a9003b470 0x7f7a9003d980 crc :-1 s=READY pgs=1371 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:49:01.865 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:01.851+0000 7f7aaf333700 1 -- 172.21.15.42:0/3873029570 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7aa805d100 con 0x7f7aa8130f20 2019-12-05T06:49:01.865 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:01.859+0000 7f7aae0d1700 1 --2- 172.21.15.42:0/3873029570 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7aa8130f20 0x7f7aa8138890 secure :-1 s=THROTTLE_DONE pgs=2483 cs=0 l=1 rx=0x7f7aa40016a0 tx=0x7f7aa4001ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:01.865 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:01.859+0000 7f7a9a7fc700 1 -- 172.21.15.42:0/3873029570 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f7aa0031f50 con 0x7f7aa8130f20 2019-12-05T06:49:02.029 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:02.023+0000 7f7aaf333700 1 -- 172.21.15.42:0/3873029570 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7aa805b8b0 con 0x7f7aa8130f20 2019-12-05T06:49:02.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:02.023+0000 7f7aae0d1700 1 --2- 172.21.15.42:0/3873029570 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7aa8130f20 0x7f7aa8138890 secure :-1 s=THROTTLE_DONE pgs=2483 cs=0 l=1 rx=0x7f7aa40016a0 tx=0x7f7aa4001ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:02.032 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:02.023+0000 7f7a9a7fc700 1 -- 172.21.15.42:0/3873029570 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 395 v395) v1 ==== 95+0+1775 (secure 0 0 0) 0x7f7aa00321b0 con 0x7f7aa8130f20 2019-12-05T06:49:02.032 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 395 2019-12-05T06:49:02.032 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:49:02.032 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":395,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":395,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:49:00.381005+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":121,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12687},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12687":{"gid":12687,"name":"b","rank":0,"incarnation":395,"state":"up:replay","state_seq":2,"addr":"172.21.15.40:6833/815316041","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":815316041},{"type":"v1","addr":"172.21.15.40:6833","nonce":815316041}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:49:02.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:02.027+0000 7f7aaf333700 1 -- 172.21.15.42:0/3873029570 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f7a9003b470 msgr2=0x7f7a9003d980 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:49:02.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:02.027+0000 7f7aaf333700 1 --2- 172.21.15.42:0/3873029570 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f7a9003b470 0x7f7a9003d980 crc :-1 s=READY pgs=1371 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:49:02.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:02.027+0000 7f7aaf333700 1 -- 172.21.15.42:0/3873029570 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7aa8130f20 msgr2=0x7f7aa8138890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:49:02.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:02.027+0000 7f7aaf333700 1 --2- 172.21.15.42:0/3873029570 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7aa8130f20 0x7f7aa8138890 secure :-1 s=READY pgs=2483 cs=0 l=1 rx=0x7f7aa40016a0 tx=0x7f7aa4001ad0).stop 2019-12-05T06:49:02.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:02.027+0000 7f7aaf333700 1 -- 172.21.15.42:0/3873029570 shutdown_connections 2019-12-05T06:49:02.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:02.027+0000 7f7aaf333700 1 --2- 172.21.15.42:0/3873029570 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f7a9003b470 0x7f7a9003d980 unknown :-1 s=CLOSED pgs=1371 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:49:02.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:02.027+0000 7f7aaf333700 1 --2- 172.21.15.42:0/3873029570 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7aa8130f20 0x7f7aa8138890 unknown :-1 s=CLOSED pgs=2483 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:49:02.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:02.027+0000 7f7aaf333700 1 -- 172.21.15.42:0/3873029570 shutdown_connections 2019-12-05T06:49:02.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:02.027+0000 7f7aaf333700 1 -- 172.21.15.42:0/3873029570 wait complete. 2019-12-05T06:49:02.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:02.027+0000 7f7aaf333700 1 -- 172.21.15.42:0/3873029570 >> 172.21.15.42:0/3873029570 conn(0x7f7aa80a9760 msgr2=0x7f7aa807e860 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:49:02.043 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a down, removed from mdsmap 2019-12-05T06:49:02.043 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2019-12-05T06:49:02.043 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:49:02.138 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:02.127+0000 7f1e3317b700 1 Processor -- start 2019-12-05T06:49:02.138 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:02.127+0000 7f1e3317b700 1 -- start start 2019-12-05T06:49:02.138 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:02.127+0000 7f1e3317b700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f1e2c0a7740 0x7f1e2c08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:49:02.138 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:02.127+0000 7f1e3317b700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f1e2c07e720 0x7f1e2c07eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:49:02.139 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:02.127+0000 7f1e3317b700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e2c0a0710 con 0x7f1e2c09f2e0 2019-12-05T06:49:02.139 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:02.127+0000 7f1e3317b700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e2c0a0b00 con 0x7f1e2c07e720 2019-12-05T06:49:02.139 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:02.127+0000 7f1e3317b700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e2c08e5b0 con 0x7f1e2c0a7740 2019-12-05T06:49:02.139 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:02.127+0000 7f1e31718700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f1e2c0a7740 0x7f1e2c08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:49:02.139 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:02.127+0000 7f1e30f17700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f1e2c07e720 0x7f1e2c07eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:49:02.139 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:02.127+0000 7f1e30f17700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f1e2c07e720 0x7f1e2c07eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.42:53664/0 (socket says 172.21.15.42:53664) 2019-12-05T06:49:02.139 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:02.127+0000 7f1e31f19700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f1e2c09f2e0 0x7f1e2c09f710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:47096/0 (socket says 172.21.15.42:47096) 2019-12-05T06:49:02.140 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:02.127+0000 7f1e30f17700 1 -- 172.21.15.42:0/1240408392 learned_addr learned my addr 172.21.15.42:0/1240408392 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:49:02.140 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:02.127+0000 7f1e31718700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f1e2c0a7740 0x7f1e2c08df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:37614/0 (socket says 172.21.15.42:37614) 2019-12-05T06:49:02.140 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:02.127+0000 7f1e30f17700 1 -- 172.21.15.42:0/1240408392 >> v2:172.21.15.42:3300/0 conn(0x7f1e2c0a7740 msgr2=0x7f1e2c08df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:49:02.140 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:02.127+0000 7f1e30f17700 1 --2- 172.21.15.42:0/1240408392 >> v2:172.21.15.42:3300/0 conn(0x7f1e2c0a7740 0x7f1e2c08df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:49:02.140 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:02.127+0000 7f1e30f17700 1 -- 172.21.15.42:0/1240408392 >> v1:172.21.15.40:6789/0 conn(0x7f1e2c09f2e0 legacy=0x7f1e2c09f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:49:02.140 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:02.127+0000 7f1e30f17700 1 -- 172.21.15.42:0/1240408392 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e2c07f0a0 con 0x7f1e2c07e720 2019-12-05T06:49:02.140 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:02.127+0000 7f1e30f17700 1 --2- 172.21.15.42:0/1240408392 >> v2:172.21.15.40:3300/0 conn(0x7f1e2c07e720 0x7f1e2c07eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f1e1c002f30 tx=0x7f1e1c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:02.141 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:02.127+0000 7f1e30f17700 1 --2- 172.21.15.42:0/1240408392 >> v2:172.21.15.40:3300/0 conn(0x7f1e2c07e720 0x7f1e2c07eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f1e1c002f30 tx=0x7f1e1c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:02.141 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:02.127+0000 7f1e30f17700 1 --2- 172.21.15.42:0/1240408392 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e2c07e720 0x7f1e2c07eb50 secure :-1 s=READY pgs=2485 cs=0 l=1 rx=0x7f1e1c002f30 tx=0x7f1e1c00fd30).ready entity=mon.0 client_cookie=8047af4d436f6262 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:49:02.141 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:02.127+0000 7f1e30f17700 1 --2- 172.21.15.42:0/1240408392 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e2c07e720 0x7f1e2c07eb50 secure :-1 s=THROTTLE_DONE pgs=2485 cs=0 l=1 rx=0x7f1e1c002f30 tx=0x7f1e1c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:02.141 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:02.127+0000 7f1e2bfff700 1 -- 172.21.15.42:0/1240408392 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1e1c011370 con 0x7f1e2c07e720 2019-12-05T06:49:02.141 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:02.127+0000 7f1e30f17700 1 --2- 172.21.15.42:0/1240408392 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e2c07e720 0x7f1e2c07eb50 secure :-1 s=THROTTLE_DONE pgs=2485 cs=0 l=1 rx=0x7f1e1c002f30 tx=0x7f1e1c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:02.141 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:02.127+0000 7f1e2bfff700 1 -- 172.21.15.42:0/1240408392 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e1c008bb0 con 0x7f1e2c07e720 2019-12-05T06:49:02.141 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:02.127+0000 7f1e30f17700 1 --2- 172.21.15.42:0/1240408392 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e2c07e720 0x7f1e2c07eb50 secure :-1 s=THROTTLE_DONE pgs=2485 cs=0 l=1 rx=0x7f1e1c002f30 tx=0x7f1e1c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:02.142 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:02.127+0000 7f1e3317b700 1 -- 172.21.15.42:0/1240408392 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e2c07e720 msgr2=0x7f1e2c07eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:49:02.142 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:02.127+0000 7f1e2bfff700 1 -- 172.21.15.42:0/1240408392 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1e1c011950 con 0x7f1e2c07e720 2019-12-05T06:49:02.142 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:02.127+0000 7f1e3317b700 1 --2- 172.21.15.42:0/1240408392 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e2c07e720 0x7f1e2c07eb50 secure :-1 s=READY pgs=2485 cs=0 l=1 rx=0x7f1e1c002f30 tx=0x7f1e1c00fd30).stop 2019-12-05T06:49:02.142 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:02.131+0000 7f1e3317b700 1 -- 172.21.15.42:0/1240408392 shutdown_connections 2019-12-05T06:49:02.142 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:02.131+0000 7f1e3317b700 1 --2- 172.21.15.42:0/1240408392 >> v2:172.21.15.42:3300/0 conn(0x7f1e2c0a7740 0x7f1e2c08df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:49:02.142 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:02.131+0000 7f1e3317b700 1 --2- 172.21.15.42:0/1240408392 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e2c07e720 0x7f1e2c07eb50 unknown :-1 s=CLOSED pgs=2485 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:49:02.142 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:02.131+0000 7f1e3317b700 1 -- 172.21.15.42:0/1240408392 shutdown_connections 2019-12-05T06:49:02.143 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:02.131+0000 7f1e3317b700 1 -- 172.21.15.42:0/1240408392 wait complete. 2019-12-05T06:49:02.143 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:02.131+0000 7f1e3317b700 1 -- 172.21.15.42:0/1240408392 >> 172.21.15.42:0/1240408392 conn(0x7f1e2c0aabe0 msgr2=0x7f1e2c0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:49:02.149 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:02.131+0000 7f1e3317b700 1 Processor -- start 2019-12-05T06:49:02.150 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:02.131+0000 7f1e3317b700 1 -- start start 2019-12-05T06:49:02.150 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:02.131+0000 7f1e3317b700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f1e2c0752f0 0x7f1e2c077730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:49:02.150 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:02.131+0000 7f1e3317b700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e2c1185d0 con 0x7f1e2c133650 2019-12-05T06:49:02.150 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:02.131+0000 7f1e3317b700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e2c133a80 con 0x7f1e2c116500 2019-12-05T06:49:02.150 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:02.131+0000 7f1e3317b700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e2c118910 con 0x7f1e2c0752f0 2019-12-05T06:49:02.150 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:02.131+0000 7f1e31718700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f1e2c116500 0x7f1e2c119810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:37658/0 (socket says 172.21.15.42:37658) 2019-12-05T06:49:02.150 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:02.131+0000 7f1e31718700 1 -- 172.21.15.42:0/385199372 learned_addr learned my addr 172.21.15.42:0/385199372 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:49:02.151 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:02.135+0000 7f1e31f19700 1 --2- 172.21.15.42:0/385199372 >> v2:172.21.15.42:3300/0 conn(0x7f1e2c0752f0 0x7f1e2c077730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:49:02.151 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:02.135+0000 7f1e29ffb700 1 -- 172.21.15.42:0/385199372 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1698821671 0 0) 0x7f1e2c1185d0 con 0x7f1e2c133650 2019-12-05T06:49:02.151 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:02.135+0000 7f1e29ffb700 1 -- 172.21.15.42:0/385199372 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e0c0030b0 con 0x7f1e2c133650 2019-12-05T06:49:02.151 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:02.135+0000 7f1e29ffb700 1 -- 172.21.15.42:0/385199372 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3353076989 0 0) 0x7f1e2c133a80 con 0x7f1e2c116500 2019-12-05T06:49:02.151 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:02.135+0000 7f1e29ffb700 1 -- 172.21.15.42:0/385199372 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e2c1185d0 con 0x7f1e2c116500 2019-12-05T06:49:02.151 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:02.135+0000 7f1e31f19700 1 -- 172.21.15.42:0/385199372 >> v1:172.21.15.42:6789/0 conn(0x7f1e2c116500 legacy=0x7f1e2c119810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:49:02.151 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:02.135+0000 7f1e31f19700 1 -- 172.21.15.42:0/385199372 >> v1:172.21.15.40:6789/0 conn(0x7f1e2c133650 legacy=0x7f1e2c118bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:49:02.152 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:02.135+0000 7f1e31f19700 1 -- 172.21.15.42:0/385199372 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e2c11c300 con 0x7f1e2c0752f0 2019-12-05T06:49:02.152 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:02.135+0000 7f1e31f19700 1 --2- 172.21.15.42:0/385199372 >> v2:172.21.15.42:3300/0 conn(0x7f1e2c0752f0 0x7f1e2c077730 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f1e24001210 tx=0x7f1e24010ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:02.152 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:02.135+0000 7f1e31f19700 1 --2- 172.21.15.42:0/385199372 >> v2:172.21.15.42:3300/0 conn(0x7f1e2c0752f0 0x7f1e2c077730 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f1e24001210 tx=0x7f1e24010ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:02.152 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:02.135+0000 7f1e31f19700 1 --2- 172.21.15.42:0/385199372 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1e2c0752f0 0x7f1e2c077730 secure :-1 s=READY pgs=3270 cs=0 l=1 rx=0x7f1e24001210 tx=0x7f1e24010ad0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:49:02.152 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:02.135+0000 7f1e31f19700 1 --2- 172.21.15.42:0/385199372 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1e2c0752f0 0x7f1e2c077730 secure :-1 s=THROTTLE_DONE pgs=3270 cs=0 l=1 rx=0x7f1e24001210 tx=0x7f1e24010ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:02.152 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:02.135+0000 7f1e31f19700 1 --2- 172.21.15.42:0/385199372 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1e2c0752f0 0x7f1e2c077730 secure :-1 s=THROTTLE_DONE pgs=3270 cs=0 l=1 rx=0x7f1e24001210 tx=0x7f1e24010ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:02.153 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:02.135+0000 7f1e31f19700 1 --2- 172.21.15.42:0/385199372 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1e2c0752f0 0x7f1e2c077730 secure :-1 s=THROTTLE_DONE pgs=3270 cs=0 l=1 rx=0x7f1e24001210 tx=0x7f1e24010ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:02.153 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:02.135+0000 7f1e29ffb700 1 -- 172.21.15.42:0/385199372 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1e2400d0c0 con 0x7f1e2c0752f0 2019-12-05T06:49:02.153 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:02.135+0000 7f1e29ffb700 1 -- 172.21.15.42:0/385199372 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e240115b0 con 0x7f1e2c0752f0 2019-12-05T06:49:02.153 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:02.135+0000 7f1e3317b700 1 -- 172.21.15.42:0/385199372 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e2c077c80 con 0x7f1e2c0752f0 2019-12-05T06:49:02.153 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:02.135+0000 7f1e29ffb700 1 -- 172.21.15.42:0/385199372 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1e2400d6a0 con 0x7f1e2c0752f0 2019-12-05T06:49:02.153 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:02.135+0000 7f1e3317b700 1 -- 172.21.15.42:0/385199372 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e2c0781b0 con 0x7f1e2c0752f0 2019-12-05T06:49:02.153 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:02.135+0000 7f1e31f19700 1 -- 172.21.15.42:0/385199372 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1e2c0752f0 msgr2=0x7f1e2c077730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2019-12-05T06:49:02.154 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:02.135+0000 7f1e31f19700 1 -- 172.21.15.42:0/385199372 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1e2c0752f0 msgr2=0x7f1e2c077730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2019-12-05T06:49:02.154 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:02.135+0000 7f1e31f19700 1 --2- 172.21.15.42:0/385199372 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1e2c0752f0 0x7f1e2c077730 secure :-1 s=READY pgs=3270 cs=0 l=1 rx=0x7f1e24001210 tx=0x7f1e24010ad0).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-12-05T06:49:02.154 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:02.135+0000 7f1e31f19700 1 --2- 172.21.15.42:0/385199372 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1e2c0752f0 0x7f1e2c077730 secure :-1 s=READY pgs=3270 cs=0 l=1 rx=0x7f1e24001210 tx=0x7f1e24010ad0).stop 2019-12-05T06:49:02.154 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:02.135+0000 7f1e29ffb700 1 -- 172.21.15.42:0/385199372 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1e2c0752f0 msgr2=0x7f1e2c077730 unknown :-1 s=STATE_CLOSED l=1).mark_down 2019-12-05T06:49:02.154 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:02.135+0000 7f1e29ffb700 1 --2- 172.21.15.42:0/385199372 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1e2c0752f0 0x7f1e2c077730 unknown :-1 s=CLOSED pgs=3270 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:49:02.154 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:02.135+0000 7f1e29ffb700 1 --2- 172.21.15.42:0/385199372 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e0c004650 0x7f1e0c004a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:49:02.154 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:02.135+0000 7f1e29ffb700 1 --2- 172.21.15.42:0/385199372 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1e0c004fd0 0x7f1e0c003440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:49:02.155 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:02.135+0000 7f1e29ffb700 1 --2- 172.21.15.42:0/385199372 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1e0c003aa0 0x7f1e0c00b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:49:02.155 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:02.135+0000 7f1e29ffb700 1 -- 172.21.15.42:0/385199372 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e0c0041e0 con 0x7f1e0c004650 2019-12-05T06:49:02.155 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:02.135+0000 7f1e29ffb700 1 -- 172.21.15.42:0/385199372 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e0c00bbc0 con 0x7f1e0c004fd0 2019-12-05T06:49:02.155 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:02.135+0000 7f1e29ffb700 1 -- 172.21.15.42:0/385199372 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e0c00be00 con 0x7f1e0c003aa0 2019-12-05T06:49:02.155 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:02.135+0000 7f1e30f17700 1 --2- 172.21.15.42:0/385199372 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1e0c004fd0 0x7f1e0c003440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:49:02.155 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:02.135+0000 7f1e31718700 1 --2- 172.21.15.42:0/385199372 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e0c004650 0x7f1e0c004a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:49:02.155 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:02.135+0000 7f1e31f19700 1 --2- 172.21.15.42:0/385199372 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1e0c003aa0 0x7f1e0c00b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:49:02.156 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:02.135+0000 7f1e30f17700 1 -- 172.21.15.42:0/385199372 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1e0c003aa0 msgr2=0x7f1e0c00b670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:49:02.157 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:02.135+0000 7f1e30f17700 1 --2- 172.21.15.42:0/385199372 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1e0c003aa0 0x7f1e0c00b670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:49:02.157 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:02.135+0000 7f1e30f17700 1 -- 172.21.15.42:0/385199372 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e0c004650 msgr2=0x7f1e0c004a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:49:02.157 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:02.135+0000 7f1e30f17700 1 --2- 172.21.15.42:0/385199372 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e0c004650 0x7f1e0c004a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:49:02.157 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:02.135+0000 7f1e31f19700 1 -- 172.21.15.42:0/385199372 reap_dead start 2019-12-05T06:49:02.158 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:02.135+0000 7f1e30f17700 1 -- 172.21.15.42:0/385199372 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+,osdmap=0}) v3 -- 0x7f1e0c00c1c0 con 0x7f1e0c004fd0 2019-12-05T06:49:02.158 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:02.135+0000 7f1e30f17700 1 --2- 172.21.15.42:0/385199372 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1e0c004fd0 0x7f1e0c003440 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f1e1c000b80 tx=0x7f1e1c011f60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:02.158 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:02.135+0000 7f1e30f17700 1 --2- 172.21.15.42:0/385199372 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1e0c004fd0 0x7f1e0c003440 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f1e1c000b80 tx=0x7f1e1c011f60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:02.158 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:02.135+0000 7f1e30f17700 1 --2- 172.21.15.42:0/385199372 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1e0c004fd0 0x7f1e0c003440 secure :-1 s=READY pgs=3271 cs=0 l=1 rx=0x7f1e1c000b80 tx=0x7f1e1c011f60).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:49:02.158 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:02.135+0000 7f1e30f17700 1 --2- 172.21.15.42:0/385199372 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1e0c004fd0 0x7f1e0c003440 secure :-1 s=THROTTLE_DONE pgs=3271 cs=0 l=1 rx=0x7f1e1c000b80 tx=0x7f1e1c011f60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:02.159 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:02.135+0000 7f1e29ffb700 1 -- 172.21.15.42:0/385199372 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1e1c00f550 con 0x7f1e0c004fd0 2019-12-05T06:49:02.159 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:02.135+0000 7f1e30f17700 1 --2- 172.21.15.42:0/385199372 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1e0c004fd0 0x7f1e0c003440 secure :-1 s=THROTTLE_DONE pgs=3271 cs=0 l=1 rx=0x7f1e1c000b80 tx=0x7f1e1c011f60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:02.159 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:02.135+0000 7f1e29ffb700 1 -- 172.21.15.42:0/385199372 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e1c011570 con 0x7f1e0c004fd0 2019-12-05T06:49:02.159 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:02.135+0000 7f1e30f17700 1 --2- 172.21.15.42:0/385199372 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1e0c004fd0 0x7f1e0c003440 secure :-1 s=THROTTLE_DONE pgs=3271 cs=0 l=1 rx=0x7f1e1c000b80 tx=0x7f1e1c011f60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:02.159 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:02.139+0000 7f1e29ffb700 1 -- 172.21.15.42:0/385199372 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f1e1c056410 con 0x7f1e0c004fd0 2019-12-05T06:49:02.159 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:02.139+0000 7f1e30f17700 1 --2- 172.21.15.42:0/385199372 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1e0c004fd0 0x7f1e0c003440 secure :-1 s=THROTTLE_DONE pgs=3271 cs=0 l=1 rx=0x7f1e1c000b80 tx=0x7f1e1c011f60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:02.160 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:02.139+0000 7f1e29ffb700 1 --2- 172.21.15.42:0/385199372 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f1e0c03ed50 0x7f1e0c041260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:49:02.160 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:02.139+0000 7f1e29ffb700 1 -- 172.21.15.42:0/385199372 <== mon.1 v2:172.21.15.42:3300/0 4 ==== osd_map(121..121 src has 1..121) v4 ==== 12223+0+0 (secure 0 0 0) 0x7f1e1c08ecf0 con 0x7f1e0c004fd0 2019-12-05T06:49:02.160 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:02.139+0000 7f1e31718700 1 --2- 172.21.15.42:0/385199372 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f1e0c03ed50 0x7f1e0c041260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:49:02.160 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:02.139+0000 7f1e31718700 1 --2- 172.21.15.42:0/385199372 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f1e0c03ed50 0x7f1e0c041260 crc :-1 s=READY pgs=1372 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:49:02.160 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:02.143+0000 7f1e3317b700 1 -- 172.21.15.42:0/385199372 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e2c05d110 con 0x7f1e0c004fd0 2019-12-05T06:49:02.161 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:02.151+0000 7f1e30f17700 1 --2- 172.21.15.42:0/385199372 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1e0c004fd0 0x7f1e0c003440 secure :-1 s=THROTTLE_DONE pgs=3271 cs=0 l=1 rx=0x7f1e1c000b80 tx=0x7f1e1c011f60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:02.161 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:02.151+0000 7f1e29ffb700 1 -- 172.21.15.42:0/385199372 <== mon.1 v2:172.21.15.42:3300/0 5 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f1e1c075fd0 con 0x7f1e0c004fd0 2019-12-05T06:49:02.328 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:02.323+0000 7f1e3317b700 1 -- 172.21.15.42:0/385199372 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1e2c05b8c0 con 0x7f1e0c004fd0 2019-12-05T06:49:02.329 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:02.323+0000 7f1e30f17700 1 --2- 172.21.15.42:0/385199372 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1e0c004fd0 0x7f1e0c003440 secure :-1 s=THROTTLE_DONE pgs=3271 cs=0 l=1 rx=0x7f1e1c000b80 tx=0x7f1e1c011f60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:02.329 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:02.323+0000 7f1e29ffb700 1 -- 172.21.15.42:0/385199372 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 395 v395) v1 ==== 95+0+1775 (secure 0 0 0) 0x7f1e1c056d60 con 0x7f1e0c004fd0 2019-12-05T06:49:02.329 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 395 2019-12-05T06:49:02.330 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:49:02.330 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":395,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":395,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:49:00.381005+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":121,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12687},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12687":{"gid":12687,"name":"b","rank":0,"incarnation":395,"state":"up:replay","state_seq":2,"addr":"172.21.15.40:6833/815316041","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":815316041},{"type":"v1","addr":"172.21.15.40:6833","nonce":815316041}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:49:02.334 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:02.323+0000 7f1e3317b700 1 -- 172.21.15.42:0/385199372 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f1e0c03ed50 msgr2=0x7f1e0c041260 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:49:02.335 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:02.323+0000 7f1e3317b700 1 --2- 172.21.15.42:0/385199372 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f1e0c03ed50 0x7f1e0c041260 crc :-1 s=READY pgs=1372 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:49:02.335 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:02.323+0000 7f1e3317b700 1 -- 172.21.15.42:0/385199372 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1e0c004fd0 msgr2=0x7f1e0c003440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:49:02.335 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:02.323+0000 7f1e3317b700 1 --2- 172.21.15.42:0/385199372 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1e0c004fd0 0x7f1e0c003440 secure :-1 s=READY pgs=3271 cs=0 l=1 rx=0x7f1e1c000b80 tx=0x7f1e1c011f60).stop 2019-12-05T06:49:02.335 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:02.323+0000 7f1e3317b700 1 -- 172.21.15.42:0/385199372 shutdown_connections 2019-12-05T06:49:02.335 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:02.323+0000 7f1e3317b700 1 --2- 172.21.15.42:0/385199372 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f1e0c03ed50 0x7f1e0c041260 unknown :-1 s=CLOSED pgs=1372 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:49:02.335 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:02.323+0000 7f1e3317b700 1 --2- 172.21.15.42:0/385199372 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1e0c004fd0 0x7f1e0c003440 unknown :-1 s=CLOSED pgs=3271 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:49:02.335 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:02.323+0000 7f1e3317b700 1 --2- 172.21.15.42:0/385199372 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1e2c0752f0 0x7f1e2c077730 unknown :-1 s=CLOSED pgs=3270 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:49:02.336 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:02.323+0000 7f1e3317b700 1 -- 172.21.15.42:0/385199372 shutdown_connections 2019-12-05T06:49:02.336 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:02.323+0000 7f1e3317b700 1 -- 172.21.15.42:0/385199372 wait complete. 2019-12-05T06:49:02.336 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:02.323+0000 7f1e3317b700 1 -- 172.21.15.42:0/385199372 >> 172.21.15.42:0/385199372 conn(0x7f1e2c09ffb0 msgr2=0x7f1e2c0a6bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:49:02.350 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 395, "feature_flags": { "enable_multiple": false, "ever_enabled_multiple": false }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2019-12-05T05:03:35.604738+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 395, "ever_allowed_features": 2, "explicitly_allowed_features": 2, "failed": [], "flags": 18, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_12687": { "addr": "172.21.15.40:6833/815316041", "addrs": { "addrvec": [ { "addr": "172.21.15.40:6832", "nonce": 815316041, "type": "v2" }, { "addr": "172.21.15.40:6833", "nonce": 815316041, "type": "v1" } ] }, "export_targets": [], "features": 4540138292836696063, "flags": 0, "gid": 12687, "incarnation": 395, "name": "b", "rank": 0, "state": "up:replay", "state_seq": 2 } }, "last_failure": 0, "last_failure_osd_epoch": 121, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 2, "min_compat_client": "0 (unknown)", "modified": "2019-12-05T06:49:00.381005+0000", "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 12687 } } } ], "standbys": [] } 2019-12-05T06:49:04.350 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:49:04.459 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:04.447+0000 7f74ca62b700 1 Processor -- start 2019-12-05T06:49:04.460 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:04.447+0000 7f74ca62b700 1 -- start start 2019-12-05T06:49:04.460 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:04.447+0000 7f74ca62b700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f74c40a7740 0x7f74c408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:49:04.460 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:04.447+0000 7f74ca62b700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f74c407e720 0x7f74c407eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:49:04.460 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:04.447+0000 7f74ca62b700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74c40a0710 con 0x7f74c409f2e0 2019-12-05T06:49:04.460 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:04.447+0000 7f74ca62b700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74c40a0b00 con 0x7f74c40a7740 2019-12-05T06:49:04.460 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:04.447+0000 7f74ca62b700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74c408e5b0 con 0x7f74c407e720 2019-12-05T06:49:04.460 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:04.447+0000 7f74c8bc8700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f74c40a7740 0x7f74c408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:49:04.461 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:04.447+0000 7f74c8bc8700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f74c40a7740 0x7f74c408df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:37640/0 (socket says 172.21.15.42:37640) 2019-12-05T06:49:04.461 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:04.447+0000 7f74c8bc8700 1 -- 172.21.15.42:0/1839296432 learned_addr learned my addr 172.21.15.42:0/1839296432 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:49:04.461 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:04.447+0000 7f74c93c9700 1 --1- 172.21.15.42:0/1839296432 >> v1:172.21.15.40:6789/0 conn(0x7f74c409f2e0 0x7f74c409f710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:47122/0 (socket says 172.21.15.42:47122) 2019-12-05T06:49:04.461 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:04.447+0000 7f74c3fff700 1 --2- 172.21.15.42:0/1839296432 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f74c407e720 0x7f74c407eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:49:04.461 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:04.451+0000 7f74c8bc8700 1 -- 172.21.15.42:0/1839296432 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f74c407e720 msgr2=0x7f74c407eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:49:04.461 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:04.451+0000 7f74c8bc8700 1 --2- 172.21.15.42:0/1839296432 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f74c407e720 0x7f74c407eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:49:04.461 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:04.451+0000 7f74c37fe700 1 -- 172.21.15.42:0/1839296432 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 788302550 0 0) 0x7f74c40a0710 con 0x7f74c409f2e0 2019-12-05T06:49:04.463 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:04.451+0000 7f74c8bc8700 1 -- 172.21.15.42:0/1839296432 >> v1:172.21.15.40:6789/0 conn(0x7f74c409f2e0 legacy=0x7f74c409f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:49:04.463 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:04.451+0000 7f74c8bc8700 1 -- 172.21.15.42:0/1839296432 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74c407f0a0 con 0x7f74c40a7740 2019-12-05T06:49:04.463 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:04.451+0000 7f74c8bc8700 1 --2- 172.21.15.42:0/1839296432 >> v2:172.21.15.42:3300/0 conn(0x7f74c40a7740 0x7f74c408df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f74b4008790 tx=0x7f74b400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:04.463 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:04.451+0000 7f74c8bc8700 1 --2- 172.21.15.42:0/1839296432 >> v2:172.21.15.42:3300/0 conn(0x7f74c40a7740 0x7f74c408df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f74b4008790 tx=0x7f74b400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:04.464 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:04.451+0000 7f74c8bc8700 1 --2- 172.21.15.42:0/1839296432 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f74c40a7740 0x7f74c408df80 secure :-1 s=READY pgs=3272 cs=0 l=1 rx=0x7f74b4008790 tx=0x7f74b400fd30).ready entity=mon.1 client_cookie=64fdb5e3856bae19 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:49:04.464 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:04.451+0000 7f74c8bc8700 1 --2- 172.21.15.42:0/1839296432 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f74c40a7740 0x7f74c408df80 secure :-1 s=THROTTLE_DONE pgs=3272 cs=0 l=1 rx=0x7f74b4008790 tx=0x7f74b400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:04.464 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:04.451+0000 7f74c37fe700 1 -- 172.21.15.42:0/1839296432 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f74b40113b0 con 0x7f74c40a7740 2019-12-05T06:49:04.464 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:04.451+0000 7f74c8bc8700 1 --2- 172.21.15.42:0/1839296432 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f74c40a7740 0x7f74c408df80 secure :-1 s=THROTTLE_DONE pgs=3272 cs=0 l=1 rx=0x7f74b4008790 tx=0x7f74b400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:04.464 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:04.451+0000 7f74c8bc8700 1 --2- 172.21.15.42:0/1839296432 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f74c40a7740 0x7f74c408df80 secure :-1 s=THROTTLE_DONE pgs=3272 cs=0 l=1 rx=0x7f74b4008790 tx=0x7f74b400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:04.464 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:04.451+0000 7f74c37fe700 1 -- 172.21.15.42:0/1839296432 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74b4008bb0 con 0x7f74c40a7740 2019-12-05T06:49:04.465 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:04.451+0000 7f74c37fe700 1 -- 172.21.15.42:0/1839296432 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f74b4011990 con 0x7f74c40a7740 2019-12-05T06:49:04.465 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:04.451+0000 7f74ca62b700 1 -- 172.21.15.42:0/1839296432 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f74c40a7740 msgr2=0x7f74c408df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:49:04.465 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:04.451+0000 7f74ca62b700 1 --2- 172.21.15.42:0/1839296432 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f74c40a7740 0x7f74c408df80 secure :-1 s=READY pgs=3272 cs=0 l=1 rx=0x7f74b4008790 tx=0x7f74b400fd30).stop 2019-12-05T06:49:04.465 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:04.451+0000 7f74ca62b700 1 -- 172.21.15.42:0/1839296432 shutdown_connections 2019-12-05T06:49:04.465 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:04.451+0000 7f74ca62b700 1 --2- 172.21.15.42:0/1839296432 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f74c40a7740 0x7f74c408df80 unknown :-1 s=CLOSED pgs=3272 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:49:04.465 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:04.451+0000 7f74ca62b700 1 --2- 172.21.15.42:0/1839296432 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f74c407e720 0x7f74c407eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:49:04.465 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:04.451+0000 7f74ca62b700 1 -- 172.21.15.42:0/1839296432 shutdown_connections 2019-12-05T06:49:04.466 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:04.451+0000 7f74ca62b700 1 -- 172.21.15.42:0/1839296432 wait complete. 2019-12-05T06:49:04.466 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:04.451+0000 7f74ca62b700 1 -- 172.21.15.42:0/1839296432 >> 172.21.15.42:0/1839296432 conn(0x7f74c40aabe0 msgr2=0x7f74c40a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:49:04.469 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:04.451+0000 7f74ca62b700 1 Processor -- start 2019-12-05T06:49:04.469 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:04.451+0000 7f74ca62b700 1 -- start start 2019-12-05T06:49:04.469 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:04.455+0000 7f74ca62b700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f74c4059f50 0x7f74c4117800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:49:04.470 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:04.455+0000 7f74ca62b700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f74c4143720 0x7f74c4116830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:49:04.470 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:04.455+0000 7f74ca62b700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74c41327c0 con 0x7f74c4116d80 2019-12-05T06:49:04.470 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:04.455+0000 7f74ca62b700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74c4143f00 con 0x7f74c4059f50 2019-12-05T06:49:04.470 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:04.455+0000 7f74ca62b700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74c41441a0 con 0x7f74c4143720 2019-12-05T06:49:04.470 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:04.455+0000 7f74c93c9700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f74c4116d80 0x7f74c40a59c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:37686/0 (socket says 172.21.15.42:37686) 2019-12-05T06:49:04.470 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:04.455+0000 7f74c93c9700 1 -- 172.21.15.42:0/559200564 learned_addr learned my addr 172.21.15.42:0/559200564 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:49:04.470 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:04.455+0000 7f74c8bc8700 1 --2- 172.21.15.42:0/559200564 >> v2:172.21.15.42:3300/0 conn(0x7f74c4059f50 0x7f74c4117800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:49:04.471 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:04.455+0000 7f74c3fff700 1 --2- 172.21.15.42:0/559200564 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f74c4143720 0x7f74c4116830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:49:04.471 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:04.455+0000 7f74c8bc8700 1 -- 172.21.15.42:0/559200564 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f74c4143720 msgr2=0x7f74c4116830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:49:04.471 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:04.455+0000 7f74c8bc8700 1 --2- 172.21.15.42:0/559200564 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f74c4143720 0x7f74c4116830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:49:04.471 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:04.455+0000 7f74c8bc8700 1 -- 172.21.15.42:0/559200564 >> v1:172.21.15.42:6789/0 conn(0x7f74c4116d80 legacy=0x7f74c40a59c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:49:04.471 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:04.455+0000 7f74c8bc8700 1 -- 172.21.15.42:0/559200564 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74c4143b50 con 0x7f74c4059f50 2019-12-05T06:49:04.471 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:04.455+0000 7f74c17fa700 1 -- 172.21.15.42:0/559200564 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 965088297 0 0) 0x7f74c41327c0 con 0x7f74c4116d80 2019-12-05T06:49:04.472 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:04.455+0000 7f74c3fff700 1 --2- 172.21.15.42:0/559200564 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f74c4143720 0x7f74c4116830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:49:04.472 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:04.455+0000 7f74c8bc8700 1 --2- 172.21.15.42:0/559200564 >> v2:172.21.15.42:3300/0 conn(0x7f74c4059f50 0x7f74c4117800 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f74c41440c0 tx=0x7f74b4011f20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:04.472 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:04.455+0000 7f74c8bc8700 1 --2- 172.21.15.42:0/559200564 >> v2:172.21.15.42:3300/0 conn(0x7f74c4059f50 0x7f74c4117800 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f74c41440c0 tx=0x7f74b4011f20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:04.472 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:04.455+0000 7f74c8bc8700 1 --2- 172.21.15.42:0/559200564 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f74c4059f50 0x7f74c4117800 secure :-1 s=READY pgs=3274 cs=0 l=1 rx=0x7f74c41440c0 tx=0x7f74b4011f20).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:49:04.472 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:04.455+0000 7f74c8bc8700 1 --2- 172.21.15.42:0/559200564 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f74c4059f50 0x7f74c4117800 secure :-1 s=THROTTLE_DONE pgs=3274 cs=0 l=1 rx=0x7f74c41440c0 tx=0x7f74b4011f20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:04.472 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:04.455+0000 7f74c17fa700 1 -- 172.21.15.42:0/559200564 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f74b4007340 con 0x7f74c4059f50 2019-12-05T06:49:04.472 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:04.455+0000 7f74c8bc8700 1 --2- 172.21.15.42:0/559200564 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f74c4059f50 0x7f74c4117800 secure :-1 s=THROTTLE_DONE pgs=3274 cs=0 l=1 rx=0x7f74c41440c0 tx=0x7f74b4011f20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:04.473 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:04.455+0000 7f74c8bc8700 1 --2- 172.21.15.42:0/559200564 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f74c4059f50 0x7f74c4117800 secure :-1 s=THROTTLE_DONE pgs=3274 cs=0 l=1 rx=0x7f74c41440c0 tx=0x7f74b4011f20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:04.473 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:04.455+0000 7f74c17fa700 1 -- 172.21.15.42:0/559200564 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74b40115b0 con 0x7f74c4059f50 2019-12-05T06:49:04.473 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:04.455+0000 7f74ca62b700 1 -- 172.21.15.42:0/559200564 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74c40a61c0 con 0x7f74c4059f50 2019-12-05T06:49:04.473 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:04.455+0000 7f74c17fa700 1 -- 172.21.15.42:0/559200564 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f74b4005540 con 0x7f74c4059f50 2019-12-05T06:49:04.473 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:04.455+0000 7f74ca62b700 1 -- 172.21.15.42:0/559200564 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74c40a66f0 con 0x7f74c4059f50 2019-12-05T06:49:04.473 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:04.455+0000 7f74c8bc8700 1 --2- 172.21.15.42:0/559200564 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f74c4059f50 0x7f74c4117800 secure :-1 s=THROTTLE_DONE pgs=3274 cs=0 l=1 rx=0x7f74c41440c0 tx=0x7f74b4011f20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:04.473 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:04.459+0000 7f74c17fa700 1 -- 172.21.15.42:0/559200564 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f74b4005820 con 0x7f74c4059f50 2019-12-05T06:49:04.474 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:04.459+0000 7f74c8bc8700 1 --2- 172.21.15.42:0/559200564 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f74c4059f50 0x7f74c4117800 secure :-1 s=THROTTLE_DONE pgs=3274 cs=0 l=1 rx=0x7f74c41440c0 tx=0x7f74b4011f20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:04.474 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:04.459+0000 7f74c17fa700 1 --2- 172.21.15.42:0/559200564 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f74a403ac30 0x7f74a403d140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:49:04.474 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:04.459+0000 7f74c17fa700 1 -- 172.21.15.42:0/559200564 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 12223+0+0 (secure 0 0 0) 0x7f74b408f380 con 0x7f74c4059f50 2019-12-05T06:49:04.474 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:04.459+0000 7f74c3fff700 1 --2- 172.21.15.42:0/559200564 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f74a403ac30 0x7f74a403d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:49:04.474 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:04.459+0000 7f74c3fff700 1 --2- 172.21.15.42:0/559200564 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f74a403ac30 0x7f74a403d140 crc :-1 s=READY pgs=1373 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:49:04.478 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:04.463+0000 7f74ca62b700 1 -- 172.21.15.42:0/559200564 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74c405d110 con 0x7f74c4059f50 2019-12-05T06:49:04.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:04.471+0000 7f74c8bc8700 1 --2- 172.21.15.42:0/559200564 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f74c4059f50 0x7f74c4117800 secure :-1 s=THROTTLE_DONE pgs=3274 cs=0 l=1 rx=0x7f74c41440c0 tx=0x7f74b4011f20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:04.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:04.471+0000 7f74c17fa700 1 -- 172.21.15.42:0/559200564 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f74b40763e0 con 0x7f74c4059f50 2019-12-05T06:49:04.643 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:04.635+0000 7f74ca62b700 1 -- 172.21.15.42:0/559200564 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f74c405b8c0 con 0x7f74c4059f50 2019-12-05T06:49:04.644 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:04.635+0000 7f74c8bc8700 1 --2- 172.21.15.42:0/559200564 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f74c4059f50 0x7f74c4117800 secure :-1 s=THROTTLE_DONE pgs=3274 cs=0 l=1 rx=0x7f74c41440c0 tx=0x7f74b4011f20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:04.644 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:04.635+0000 7f74c17fa700 1 -- 172.21.15.42:0/559200564 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 395 v395) v1 ==== 95+0+1775 (secure 0 0 0) 0x7f74b4076640 con 0x7f74c4059f50 2019-12-05T06:49:04.644 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 395 2019-12-05T06:49:04.644 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:49:04.644 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":395,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":395,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:49:00.381005+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":121,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12687},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12687":{"gid":12687,"name":"b","rank":0,"incarnation":395,"state":"up:replay","state_seq":2,"addr":"172.21.15.40:6833/815316041","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":815316041},{"type":"v1","addr":"172.21.15.40:6833","nonce":815316041}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:49:04.647 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:04.639+0000 7f74ca62b700 1 -- 172.21.15.42:0/559200564 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f74a403ac30 msgr2=0x7f74a403d140 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:49:04.647 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:04.639+0000 7f74ca62b700 1 --2- 172.21.15.42:0/559200564 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f74a403ac30 0x7f74a403d140 crc :-1 s=READY pgs=1373 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:49:04.647 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:04.639+0000 7f74ca62b700 1 -- 172.21.15.42:0/559200564 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f74c4059f50 msgr2=0x7f74c4117800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:49:04.647 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:04.639+0000 7f74ca62b700 1 --2- 172.21.15.42:0/559200564 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f74c4059f50 0x7f74c4117800 secure :-1 s=READY pgs=3274 cs=0 l=1 rx=0x7f74c41440c0 tx=0x7f74b4011f20).stop 2019-12-05T06:49:04.647 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:04.639+0000 7f74ca62b700 1 -- 172.21.15.42:0/559200564 shutdown_connections 2019-12-05T06:49:04.647 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:04.639+0000 7f74ca62b700 1 --2- 172.21.15.42:0/559200564 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f74a403ac30 0x7f74a403d140 unknown :-1 s=CLOSED pgs=1373 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:49:04.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:04.639+0000 7f74ca62b700 1 --2- 172.21.15.42:0/559200564 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f74c4059f50 0x7f74c4117800 unknown :-1 s=CLOSED pgs=3274 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:49:04.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:04.639+0000 7f74ca62b700 1 --2- 172.21.15.42:0/559200564 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f74c4143720 0x7f74c4116830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:49:04.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:04.639+0000 7f74ca62b700 1 -- 172.21.15.42:0/559200564 shutdown_connections 2019-12-05T06:49:04.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:04.639+0000 7f74ca62b700 1 -- 172.21.15.42:0/559200564 wait complete. 2019-12-05T06:49:04.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:04.639+0000 7f74ca62b700 1 -- 172.21.15.42:0/559200564 >> 172.21.15.42:0/559200564 conn(0x7f74c407de90 msgr2=0x7f74c411d550 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:49:04.661 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2019-12-05T06:49:06.662 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:49:06.766 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:06.751+0000 7fbe14b45700 1 Processor -- start 2019-12-05T06:49:06.766 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:06.751+0000 7fbe14b45700 1 -- start start 2019-12-05T06:49:06.766 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:06.755+0000 7fbe14b45700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbe1008df80 0x7fbe1009f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:49:06.766 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:06.755+0000 7fbe14b45700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe1009fc60 con 0x7fbe100a7740 2019-12-05T06:49:06.766 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:06.755+0000 7fbe14b45700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe100a0710 con 0x7fbe1009f830 2019-12-05T06:49:06.767 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:06.755+0000 7fbe14b45700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe100a0b00 con 0x7fbe1008df80 2019-12-05T06:49:06.767 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:06.755+0000 7fbe0ed9d700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fbe100a7740 0x7fbe1007e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:47132/0 (socket says 172.21.15.42:47132) 2019-12-05T06:49:06.767 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:06.755+0000 7fbe0ed9d700 1 -- 172.21.15.42:0/3727624749 learned_addr learned my addr 172.21.15.42:0/3727624749 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:49:06.767 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:06.755+0000 7fbe0e59c700 1 --2- 172.21.15.42:0/3727624749 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbe1008df80 0x7fbe1009f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:49:06.767 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:06.755+0000 7fbe0dd9b700 1 -- 172.21.15.42:0/3727624749 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3085320039 0 0) 0x7fbe1009fc60 con 0x7fbe100a7740 2019-12-05T06:49:06.768 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:06.755+0000 7fbe0dd9b700 1 -- 172.21.15.42:0/3727624749 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbdf80030b0 con 0x7fbe100a7740 2019-12-05T06:49:06.768 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:06.755+0000 7fbe0dd9b700 1 -- 172.21.15.42:0/3727624749 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 340652862 0 0) 0x7fbe100a0710 con 0x7fbe1009f830 2019-12-05T06:49:06.768 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:06.755+0000 7fbe0dd9b700 1 -- 172.21.15.42:0/3727624749 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe1009fc60 con 0x7fbe1009f830 2019-12-05T06:49:06.768 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:06.755+0000 7fbe0f59e700 1 -- 172.21.15.42:0/3727624749 >> v1:172.21.15.42:6789/0 conn(0x7fbe1009f830 legacy=0x7fbe1014efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2019-12-05T06:49:06.768 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:06.755+0000 7fbe0f59e700 1 -- 172.21.15.42:0/3727624749 >> v1:172.21.15.42:6789/0 conn(0x7fbe1009f830 legacy=0x7fbe1014efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2019-12-05T06:49:06.768 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:06.755+0000 7fbe0f59e700 1 --1- 172.21.15.42:0/3727624749 >> v1:172.21.15.42:6789/0 conn(0x7fbe1009f830 0x7fbe1014efe0 :-1 s=OPENED pgs=3275 cs=1 l=1).handle_message read tag failed 2019-12-05T06:49:06.769 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:06.755+0000 7fbe0f59e700 1 --1- 172.21.15.42:0/3727624749 >> v1:172.21.15.42:6789/0 conn(0x7fbe1009f830 0x7fbe1014efe0 :-1 s=OPENED pgs=3275 cs=1 l=1).fault on lossy channel, failing 2019-12-05T06:49:06.769 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:06.755+0000 7fbe0e59c700 1 -- 172.21.15.42:0/3727624749 >> v1:172.21.15.42:6789/0 conn(0x7fbe1009f830 legacy=0x7fbe1014efe0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2019-12-05T06:49:06.769 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:06.755+0000 7fbe0e59c700 1 -- 172.21.15.42:0/3727624749 >> v1:172.21.15.40:6789/0 conn(0x7fbe100a7740 legacy=0x7fbe1007e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:49:06.769 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:06.755+0000 7fbe0e59c700 1 -- 172.21.15.42:0/3727624749 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe1007ef20 con 0x7fbe1008df80 2019-12-05T06:49:06.769 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:06.755+0000 7fbe0e59c700 1 --2- 172.21.15.42:0/3727624749 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbe1008df80 0x7fbe1009f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fbe04008790 tx=0x7fbe0400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:06.769 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:06.755+0000 7fbe0e59c700 1 --2- 172.21.15.42:0/3727624749 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbe1008df80 0x7fbe1009f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fbe04008790 tx=0x7fbe0400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:06.770 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:06.755+0000 7fbe0e59c700 1 --2- 172.21.15.42:0/3727624749 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbe1008df80 0x7fbe1009f2e0 secure :-1 s=READY pgs=725 cs=0 l=1 rx=0x7fbe04008790 tx=0x7fbe0400fbf0).ready entity=mon.2 client_cookie=523708beed639f24 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:49:06.770 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:06.755+0000 7fbe0e59c700 1 --2- 172.21.15.42:0/3727624749 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbe1008df80 0x7fbe1009f2e0 secure :-1 s=THROTTLE_DONE pgs=725 cs=0 l=1 rx=0x7fbe04008790 tx=0x7fbe0400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:06.770 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:06.755+0000 7fbe0e59c700 1 --2- 172.21.15.42:0/3727624749 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbe1008df80 0x7fbe1009f2e0 secure :-1 s=THROTTLE_DONE pgs=725 cs=0 l=1 rx=0x7fbe04008790 tx=0x7fbe0400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:06.770 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:06.755+0000 7fbe0dd9b700 1 -- 172.21.15.42:0/3727624749 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbe04008bb0 con 0x7fbe1008df80 2019-12-05T06:49:06.770 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:06.755+0000 7fbe0e59c700 1 --2- 172.21.15.42:0/3727624749 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbe1008df80 0x7fbe1009f2e0 secure :-1 s=THROTTLE_DONE pgs=725 cs=0 l=1 rx=0x7fbe04008790 tx=0x7fbe0400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:06.770 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:06.755+0000 7fbe0dd9b700 1 -- 172.21.15.42:0/3727624749 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe040087d0 con 0x7fbe1008df80 2019-12-05T06:49:06.770 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:06.755+0000 7fbe0dd9b700 1 -- 172.21.15.42:0/3727624749 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbe040118d0 con 0x7fbe1008df80 2019-12-05T06:49:06.771 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:06.755+0000 7fbe14b45700 1 -- 172.21.15.42:0/3727624749 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbe1008df80 msgr2=0x7fbe1009f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:49:06.771 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:06.755+0000 7fbe14b45700 1 --2- 172.21.15.42:0/3727624749 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbe1008df80 0x7fbe1009f2e0 secure :-1 s=READY pgs=725 cs=0 l=1 rx=0x7fbe04008790 tx=0x7fbe0400fbf0).stop 2019-12-05T06:49:06.771 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:06.759+0000 7fbe14b45700 1 -- 172.21.15.42:0/3727624749 shutdown_connections 2019-12-05T06:49:06.771 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:06.759+0000 7fbe14b45700 1 --2- 172.21.15.42:0/3727624749 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbe1008df80 0x7fbe1009f2e0 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:49:06.771 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:06.759+0000 7fbe14b45700 1 -- 172.21.15.42:0/3727624749 shutdown_connections 2019-12-05T06:49:06.771 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:06.759+0000 7fbe14b45700 1 -- 172.21.15.42:0/3727624749 wait complete. 2019-12-05T06:49:06.772 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:06.759+0000 7fbe14b45700 1 -- 172.21.15.42:0/3727624749 >> 172.21.15.42:0/3727624749 conn(0x7fbe100aabe0 msgr2=0x7fbe100a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:49:06.776 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:06.759+0000 7fbe14b45700 1 Processor -- start 2019-12-05T06:49:06.776 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:06.759+0000 7fbe14b45700 1 -- start start 2019-12-05T06:49:06.776 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:06.763+0000 7fbe14b45700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fbe1011ae60 0x7fbe1012cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:49:06.776 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:06.763+0000 7fbe14b45700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe1011b290 con 0x7fbe100a57a0 2019-12-05T06:49:06.777 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:06.763+0000 7fbe14b45700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe100a6f70 con 0x7fbe1011dfb0 2019-12-05T06:49:06.777 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:06.763+0000 7fbe14b45700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe100a5260 con 0x7fbe1011ae60 2019-12-05T06:49:06.777 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:06.763+0000 7fbe0f59e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fbe1011ae60 0x7fbe1012cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:49:06.777 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:06.763+0000 7fbe0f59e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fbe1011ae60 0x7fbe1012cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:37664/0 (socket says 172.21.15.42:37664) 2019-12-05T06:49:06.777 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:06.763+0000 7fbe0f59e700 1 -- 172.21.15.42:0/956665927 learned_addr learned my addr 172.21.15.42:0/956665927 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:49:06.778 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:06.763+0000 7fbe0f59e700 1 -- 172.21.15.42:0/956665927 >> v1:172.21.15.42:6789/0 conn(0x7fbe1011dfb0 legacy=0x7fbe10129540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:49:06.778 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:06.763+0000 7fbe0f59e700 1 -- 172.21.15.42:0/956665927 >> v1:172.21.15.40:6789/0 conn(0x7fbe100a57a0 legacy=0x7fbe1011b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:49:06.778 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:06.763+0000 7fbe0f59e700 1 -- 172.21.15.42:0/956665927 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe1011d6a0 con 0x7fbe1011ae60 2019-12-05T06:49:06.778 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:06.763+0000 7fbdff7fe700 1 -- 172.21.15.42:0/956665927 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 915676265 0 0) 0x7fbe100a6f70 con 0x7fbe1011dfb0 2019-12-05T06:49:06.778 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:06.763+0000 7fbe0f59e700 1 --2- 172.21.15.42:0/956665927 >> v2:172.21.15.42:3300/0 conn(0x7fbe1011ae60 0x7fbe1012cd60 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fbe1011a770 tx=0x7fbe08010700).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:06.779 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:06.763+0000 7fbe0f59e700 1 --2- 172.21.15.42:0/956665927 >> v2:172.21.15.42:3300/0 conn(0x7fbe1011ae60 0x7fbe1012cd60 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fbe1011a770 tx=0x7fbe08010700).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:06.779 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:06.763+0000 7fbe0f59e700 1 --2- 172.21.15.42:0/956665927 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbe1011ae60 0x7fbe1012cd60 secure :-1 s=READY pgs=3277 cs=0 l=1 rx=0x7fbe1011a770 tx=0x7fbe08010700).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:49:06.779 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:06.763+0000 7fbe0f59e700 1 --2- 172.21.15.42:0/956665927 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbe1011ae60 0x7fbe1012cd60 secure :-1 s=THROTTLE_DONE pgs=3277 cs=0 l=1 rx=0x7fbe1011a770 tx=0x7fbe08010700).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:06.779 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:06.763+0000 7fbdff7fe700 1 -- 172.21.15.42:0/956665927 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbe08005b20 con 0x7fbe1011ae60 2019-12-05T06:49:06.779 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:06.763+0000 7fbe0f59e700 1 --2- 172.21.15.42:0/956665927 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbe1011ae60 0x7fbe1012cd60 secure :-1 s=THROTTLE_DONE pgs=3277 cs=0 l=1 rx=0x7fbe1011a770 tx=0x7fbe08010700).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:06.779 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:06.763+0000 7fbe0f59e700 1 --2- 172.21.15.42:0/956665927 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbe1011ae60 0x7fbe1012cd60 secure :-1 s=THROTTLE_DONE pgs=3277 cs=0 l=1 rx=0x7fbe1011a770 tx=0x7fbe08010700).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:06.780 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:06.763+0000 7fbdff7fe700 1 -- 172.21.15.42:0/956665927 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe08005740 con 0x7fbe1011ae60 2019-12-05T06:49:06.780 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:06.763+0000 7fbe14b45700 1 -- 172.21.15.42:0/956665927 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe1012d2b0 con 0x7fbe1011ae60 2019-12-05T06:49:06.780 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:06.763+0000 7fbdff7fe700 1 -- 172.21.15.42:0/956665927 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbe08006630 con 0x7fbe1011ae60 2019-12-05T06:49:06.780 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:06.763+0000 7fbe14b45700 1 -- 172.21.15.42:0/956665927 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe1012d830 con 0x7fbe1011ae60 2019-12-05T06:49:06.780 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:06.763+0000 7fbe0f59e700 1 --2- 172.21.15.42:0/956665927 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbe1011ae60 0x7fbe1012cd60 secure :-1 s=THROTTLE_DONE pgs=3277 cs=0 l=1 rx=0x7fbe1011a770 tx=0x7fbe08010700).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:06.780 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:06.763+0000 7fbdff7fe700 1 -- 172.21.15.42:0/956665927 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fbe08006940 con 0x7fbe1011ae60 2019-12-05T06:49:06.781 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:06.763+0000 7fbe0f59e700 1 --2- 172.21.15.42:0/956665927 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbe1011ae60 0x7fbe1012cd60 secure :-1 s=THROTTLE_DONE pgs=3277 cs=0 l=1 rx=0x7fbe1011a770 tx=0x7fbe08010700).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:06.781 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:06.763+0000 7fbdff7fe700 1 --2- 172.21.15.42:0/956665927 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbdf403abc0 0x7fbdf403d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:49:06.781 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:06.767+0000 7fbdff7fe700 1 -- 172.21.15.42:0/956665927 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 12223+0+0 (secure 0 0 0) 0x7fbe0804b230 con 0x7fbe1011ae60 2019-12-05T06:49:06.781 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:06.767+0000 7fbe0ed9d700 1 --2- 172.21.15.42:0/956665927 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbdf403abc0 0x7fbdf403d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:49:06.782 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:06.767+0000 7fbe0ed9d700 1 --2- 172.21.15.42:0/956665927 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbdf403abc0 0x7fbdf403d0d0 crc :-1 s=READY pgs=1374 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:49:06.782 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:06.767+0000 7fbe14b45700 1 -- 172.21.15.42:0/956665927 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe1005d110 con 0x7fbe1011ae60 2019-12-05T06:49:06.783 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:06.775+0000 7fbe0f59e700 1 --2- 172.21.15.42:0/956665927 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbe1011ae60 0x7fbe1012cd60 secure :-1 s=THROTTLE_DONE pgs=3277 cs=0 l=1 rx=0x7fbe1011a770 tx=0x7fbe08010700).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:06.783 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:06.775+0000 7fbdff7fe700 1 -- 172.21.15.42:0/956665927 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fbe080322b0 con 0x7fbe1011ae60 2019-12-05T06:49:06.942 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:06.935+0000 7fbe14b45700 1 -- 172.21.15.42:0/956665927 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbe1005b8c0 con 0x7fbe1011ae60 2019-12-05T06:49:06.943 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:06.935+0000 7fbe0f59e700 1 --2- 172.21.15.42:0/956665927 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbe1011ae60 0x7fbe1012cd60 secure :-1 s=THROTTLE_DONE pgs=3277 cs=0 l=1 rx=0x7fbe1011a770 tx=0x7fbe08010700).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:06.943 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:06.935+0000 7fbdff7fe700 1 -- 172.21.15.42:0/956665927 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 395 v395) v1 ==== 95+0+1775 (secure 0 0 0) 0x7fbe08032510 con 0x7fbe1011ae60 2019-12-05T06:49:06.943 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 395 2019-12-05T06:49:06.943 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:49:06.944 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":395,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":395,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:49:00.381005+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":121,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12687},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12687":{"gid":12687,"name":"b","rank":0,"incarnation":395,"state":"up:replay","state_seq":2,"addr":"172.21.15.40:6833/815316041","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":815316041},{"type":"v1","addr":"172.21.15.40:6833","nonce":815316041}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:49:06.946 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:06.939+0000 7fbe14b45700 1 -- 172.21.15.42:0/956665927 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbdf403abc0 msgr2=0x7fbdf403d0d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:49:06.946 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:06.939+0000 7fbe14b45700 1 --2- 172.21.15.42:0/956665927 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbdf403abc0 0x7fbdf403d0d0 crc :-1 s=READY pgs=1374 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:49:06.947 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:06.939+0000 7fbe14b45700 1 -- 172.21.15.42:0/956665927 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbe1011ae60 msgr2=0x7fbe1012cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:49:06.947 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:06.939+0000 7fbe14b45700 1 --2- 172.21.15.42:0/956665927 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbe1011ae60 0x7fbe1012cd60 secure :-1 s=READY pgs=3277 cs=0 l=1 rx=0x7fbe1011a770 tx=0x7fbe08010700).stop 2019-12-05T06:49:06.947 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:06.939+0000 7fbe14b45700 1 -- 172.21.15.42:0/956665927 shutdown_connections 2019-12-05T06:49:06.947 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:06.939+0000 7fbe14b45700 1 --2- 172.21.15.42:0/956665927 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbdf403abc0 0x7fbdf403d0d0 unknown :-1 s=CLOSED pgs=1374 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:49:06.947 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:06.939+0000 7fbe14b45700 1 --2- 172.21.15.42:0/956665927 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbe1011ae60 0x7fbe1012cd60 unknown :-1 s=CLOSED pgs=3277 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:49:06.948 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:06.939+0000 7fbe14b45700 1 -- 172.21.15.42:0/956665927 shutdown_connections 2019-12-05T06:49:06.948 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:06.939+0000 7fbe14b45700 1 -- 172.21.15.42:0/956665927 wait complete. 2019-12-05T06:49:06.948 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:06.939+0000 7fbe14b45700 1 -- 172.21.15.42:0/956665927 >> 172.21.15.42:0/956665927 conn(0x7fbe100a7f90 msgr2=0x7fbe1007e5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:49:06.956 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2019-12-05T06:49:08.957 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:49:09.064 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:09.051+0000 7f574ef06700 1 Processor -- start 2019-12-05T06:49:09.064 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:09.051+0000 7f574ef06700 1 -- start start 2019-12-05T06:49:09.065 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:09.051+0000 7f574ef06700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f57480a7740 0x7f574808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:49:09.065 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:09.051+0000 7f574ef06700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f574809fae0 0x7f574807eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:49:09.065 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:09.051+0000 7f574ef06700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57480a0710 con 0x7f574807e720 2019-12-05T06:49:09.065 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:09.051+0000 7f574ef06700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57480a0b00 con 0x7f574809fae0 2019-12-05T06:49:09.065 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:09.051+0000 7f574ef06700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f574808e5b0 con 0x7f57480a7740 2019-12-05T06:49:09.065 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:09.055+0000 7f574d4a3700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f57480a7740 0x7f574808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:49:09.065 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:09.055+0000 7f574d4a3700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f57480a7740 0x7f574808df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:37674/0 (socket says 172.21.15.42:37674) 2019-12-05T06:49:09.067 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:09.055+0000 7f574d4a3700 1 -- 172.21.15.42:0/164087222 learned_addr learned my addr 172.21.15.42:0/164087222 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:49:09.067 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:09.055+0000 7f574dca4700 1 --2- 172.21.15.42:0/164087222 >> v2:172.21.15.40:3300/0 conn(0x7f574809fae0 0x7f574807eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:49:09.068 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:09.055+0000 7f574d4a3700 1 -- 172.21.15.42:0/164087222 >> v2:172.21.15.40:3300/0 conn(0x7f574809fae0 msgr2=0x7f574807eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:49:09.068 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:09.055+0000 7f574d4a3700 1 --2- 172.21.15.42:0/164087222 >> v2:172.21.15.40:3300/0 conn(0x7f574809fae0 0x7f574807eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:49:09.068 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:09.055+0000 7f574d4a3700 1 -- 172.21.15.42:0/164087222 >> v1:172.21.15.40:6789/0 conn(0x7f574807e720 legacy=0x7f574809f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:49:09.068 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:09.055+0000 7f574d4a3700 1 -- 172.21.15.42:0/164087222 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f574807f0a0 con 0x7f57480a7740 2019-12-05T06:49:09.068 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:09.055+0000 7f574d4a3700 1 --2- 172.21.15.42:0/164087222 >> v2:172.21.15.42:3300/0 conn(0x7f57480a7740 0x7f574808df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f5744002f30 tx=0x7f574400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:09.068 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:09.055+0000 7f574d4a3700 1 --2- 172.21.15.42:0/164087222 >> v2:172.21.15.42:3300/0 conn(0x7f57480a7740 0x7f574808df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f5744002f30 tx=0x7f574400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:09.069 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:09.055+0000 7f574d4a3700 1 --2- 172.21.15.42:0/164087222 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f57480a7740 0x7f574808df80 secure :-1 s=READY pgs=3278 cs=0 l=1 rx=0x7f5744002f30 tx=0x7f574400fd30).ready entity=mon.1 client_cookie=428142a4de78bbdd server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:49:09.069 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:09.055+0000 7f574d4a3700 1 --2- 172.21.15.42:0/164087222 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f57480a7740 0x7f574808df80 secure :-1 s=THROTTLE_DONE pgs=3278 cs=0 l=1 rx=0x7f5744002f30 tx=0x7f574400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:09.069 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:09.055+0000 7f574d4a3700 1 --2- 172.21.15.42:0/164087222 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f57480a7740 0x7f574808df80 secure :-1 s=THROTTLE_DONE pgs=3278 cs=0 l=1 rx=0x7f5744002f30 tx=0x7f574400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:09.069 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:09.055+0000 7f574d4a3700 1 --2- 172.21.15.42:0/164087222 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f57480a7740 0x7f574808df80 secure :-1 s=THROTTLE_DONE pgs=3278 cs=0 l=1 rx=0x7f5744002f30 tx=0x7f574400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:09.069 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:09.055+0000 7f573ffff700 1 -- 172.21.15.42:0/164087222 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5744008c90 con 0x7f57480a7740 2019-12-05T06:49:09.069 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:09.055+0000 7f573ffff700 1 -- 172.21.15.42:0/164087222 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57440088b0 con 0x7f57480a7740 2019-12-05T06:49:09.069 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:09.055+0000 7f573ffff700 1 -- 172.21.15.42:0/164087222 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5744011ad0 con 0x7f57480a7740 2019-12-05T06:49:09.086 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:09.055+0000 7f574ef06700 1 -- 172.21.15.42:0/164087222 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f57480a7740 msgr2=0x7f574808df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:49:09.086 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:09.055+0000 7f574ef06700 1 --2- 172.21.15.42:0/164087222 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f57480a7740 0x7f574808df80 secure :-1 s=READY pgs=3278 cs=0 l=1 rx=0x7f5744002f30 tx=0x7f574400fd30).stop 2019-12-05T06:49:09.086 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:09.055+0000 7f574ef06700 1 -- 172.21.15.42:0/164087222 shutdown_connections 2019-12-05T06:49:09.086 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:09.055+0000 7f574ef06700 1 --2- 172.21.15.42:0/164087222 >> v2:172.21.15.40:3300/0 conn(0x7f574809fae0 0x7f574807eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:49:09.086 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:09.055+0000 7f574ef06700 1 --2- 172.21.15.42:0/164087222 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f57480a7740 0x7f574808df80 unknown :-1 s=CLOSED pgs=3278 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:49:09.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:09.055+0000 7f574ef06700 1 -- 172.21.15.42:0/164087222 shutdown_connections 2019-12-05T06:49:09.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:09.055+0000 7f574ef06700 1 -- 172.21.15.42:0/164087222 wait complete. 2019-12-05T06:49:09.087 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:09.055+0000 7f574ef06700 1 -- 172.21.15.42:0/164087222 >> 172.21.15.42:0/164087222 conn(0x7f57480aabe0 msgr2=0x7f57480a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:49:09.092 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:09.059+0000 7f574ef06700 1 Processor -- start 2019-12-05T06:49:09.092 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:09.059+0000 7f574ef06700 1 -- start start 2019-12-05T06:49:09.092 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:09.059+0000 7f574ef06700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5748063ac0 0x7f5748126c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:49:09.092 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:09.059+0000 7f574ef06700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5748063ef0 con 0x7f5748127170 2019-12-05T06:49:09.093 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:09.059+0000 7f574ef06700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5748065b80 con 0x7f574808ee90 2019-12-05T06:49:09.093 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:09.059+0000 7f574ef06700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f574811ba30 con 0x7f5748063ac0 2019-12-05T06:49:09.093 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:09.059+0000 7f574d4a3700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f574808ee90 0x7f574811b230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:37714/0 (socket says 172.21.15.42:37714) 2019-12-05T06:49:09.093 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:09.059+0000 7f574d4a3700 1 -- 172.21.15.42:0/1484651500 learned_addr learned my addr 172.21.15.42:0/1484651500 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:49:09.093 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:09.063+0000 7f574cca2700 1 --2- 172.21.15.42:0/1484651500 >> v2:172.21.15.42:3300/0 conn(0x7f5748063ac0 0x7f5748126c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:49:09.093 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:09.063+0000 7f573dffb700 1 -- 172.21.15.42:0/1484651500 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1941177764 0 0) 0x7f5748063ef0 con 0x7f5748127170 2019-12-05T06:49:09.093 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:09.063+0000 7f574cca2700 1 -- 172.21.15.42:0/1484651500 >> v1:172.21.15.42:6789/0 conn(0x7f574808ee90 legacy=0x7f574811b230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:49:09.094 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:09.063+0000 7f574cca2700 1 -- 172.21.15.42:0/1484651500 >> v1:172.21.15.40:6789/0 conn(0x7f5748127170 legacy=0x7f5748131630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:49:09.094 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:09.063+0000 7f574cca2700 1 -- 172.21.15.42:0/1484651500 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57481275a0 con 0x7f5748063ac0 2019-12-05T06:49:09.094 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:09.063+0000 7f574cca2700 1 --2- 172.21.15.42:0/1484651500 >> v2:172.21.15.42:3300/0 conn(0x7f5748063ac0 0x7f5748126c20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f5738001430 tx=0x7f574400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:09.094 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:09.063+0000 7f574cca2700 1 --2- 172.21.15.42:0/1484651500 >> v2:172.21.15.42:3300/0 conn(0x7f5748063ac0 0x7f5748126c20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f5738001430 tx=0x7f574400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:09.094 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:09.063+0000 7f574cca2700 1 --2- 172.21.15.42:0/1484651500 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5748063ac0 0x7f5748126c20 secure :-1 s=READY pgs=3280 cs=0 l=1 rx=0x7f5738001430 tx=0x7f574400fd30).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:49:09.094 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:09.063+0000 7f574cca2700 1 --2- 172.21.15.42:0/1484651500 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5748063ac0 0x7f5748126c20 secure :-1 s=THROTTLE_DONE pgs=3280 cs=0 l=1 rx=0x7f5738001430 tx=0x7f574400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:09.095 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:09.063+0000 7f573dffb700 1 -- 172.21.15.42:0/1484651500 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f573800c9a0 con 0x7f5748063ac0 2019-12-05T06:49:09.095 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:09.063+0000 7f574cca2700 1 --2- 172.21.15.42:0/1484651500 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5748063ac0 0x7f5748126c20 secure :-1 s=THROTTLE_DONE pgs=3280 cs=0 l=1 rx=0x7f5738001430 tx=0x7f574400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:09.095 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:09.063+0000 7f574cca2700 1 --2- 172.21.15.42:0/1484651500 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5748063ac0 0x7f5748126c20 secure :-1 s=THROTTLE_DONE pgs=3280 cs=0 l=1 rx=0x7f5738001430 tx=0x7f574400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:09.095 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:09.063+0000 7f573dffb700 1 -- 172.21.15.42:0/1484651500 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5738011e10 con 0x7f5748063ac0 2019-12-05T06:49:09.095 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:09.063+0000 7f574ef06700 1 -- 172.21.15.42:0/1484651500 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5748064da0 con 0x7f5748063ac0 2019-12-05T06:49:09.105 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:09.063+0000 7f573dffb700 1 -- 172.21.15.42:0/1484651500 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5738010e00 con 0x7f5748063ac0 2019-12-05T06:49:09.105 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:09.063+0000 7f574ef06700 1 -- 172.21.15.42:0/1484651500 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5748132e40 con 0x7f5748063ac0 2019-12-05T06:49:09.105 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:09.063+0000 7f574cca2700 1 --2- 172.21.15.42:0/1484651500 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5748063ac0 0x7f5748126c20 secure :-1 s=THROTTLE_DONE pgs=3280 cs=0 l=1 rx=0x7f5738001430 tx=0x7f574400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:09.105 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:09.063+0000 7f573dffb700 1 -- 172.21.15.42:0/1484651500 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f5738007430 con 0x7f5748063ac0 2019-12-05T06:49:09.106 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:09.063+0000 7f574cca2700 1 --2- 172.21.15.42:0/1484651500 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5748063ac0 0x7f5748126c20 secure :-1 s=THROTTLE_DONE pgs=3280 cs=0 l=1 rx=0x7f5738001430 tx=0x7f574400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:09.106 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:09.063+0000 7f573dffb700 1 --2- 172.21.15.42:0/1484651500 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f572803acc0 0x7f572803d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:49:09.106 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:09.063+0000 7f573dffb700 1 -- 172.21.15.42:0/1484651500 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 12223+0+0 (secure 0 0 0) 0x7f573804afe0 con 0x7f5748063ac0 2019-12-05T06:49:09.106 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:09.063+0000 7f574d4a3700 1 --2- 172.21.15.42:0/1484651500 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f572803acc0 0x7f572803d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:49:09.106 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:09.067+0000 7f574d4a3700 1 --2- 172.21.15.42:0/1484651500 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f572803acc0 0x7f572803d1d0 crc :-1 s=READY pgs=1375 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:49:09.107 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:09.067+0000 7f574ef06700 1 -- 172.21.15.42:0/1484651500 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f574805d110 con 0x7f5748063ac0 2019-12-05T06:49:09.107 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:09.075+0000 7f574cca2700 1 --2- 172.21.15.42:0/1484651500 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5748063ac0 0x7f5748126c20 secure :-1 s=THROTTLE_DONE pgs=3280 cs=0 l=1 rx=0x7f5738001430 tx=0x7f574400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:09.107 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:09.075+0000 7f573dffb700 1 -- 172.21.15.42:0/1484651500 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f573802f0c0 con 0x7f5748063ac0 2019-12-05T06:49:09.242 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:09.235+0000 7f574ef06700 1 -- 172.21.15.42:0/1484651500 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f574805b8c0 con 0x7f5748063ac0 2019-12-05T06:49:09.243 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:09.235+0000 7f574cca2700 1 --2- 172.21.15.42:0/1484651500 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5748063ac0 0x7f5748126c20 secure :-1 s=THROTTLE_DONE pgs=3280 cs=0 l=1 rx=0x7f5738001430 tx=0x7f574400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:09.243 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:09.235+0000 7f573dffb700 1 -- 172.21.15.42:0/1484651500 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 396 v396) v1 ==== 95+0+1779 (secure 0 0 0) 0x7f5738032000 con 0x7f5748063ac0 2019-12-05T06:49:09.245 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:49:09.245 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":396,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":396,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:49:08.390227+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":121,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12687},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12687":{"gid":12687,"name":"b","rank":0,"incarnation":395,"state":"up:reconnect","state_seq":10,"addr":"172.21.15.40:6833/815316041","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":815316041},{"type":"v1","addr":"172.21.15.40:6833","nonce":815316041}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:49:09.246 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 396 2019-12-05T06:49:09.249 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:09.239+0000 7f574ef06700 1 -- 172.21.15.42:0/1484651500 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f572803acc0 msgr2=0x7f572803d1d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:49:09.250 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:09.239+0000 7f574ef06700 1 --2- 172.21.15.42:0/1484651500 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f572803acc0 0x7f572803d1d0 crc :-1 s=READY pgs=1375 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:49:09.250 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:09.239+0000 7f574ef06700 1 -- 172.21.15.42:0/1484651500 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5748063ac0 msgr2=0x7f5748126c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:49:09.250 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:09.239+0000 7f574ef06700 1 --2- 172.21.15.42:0/1484651500 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5748063ac0 0x7f5748126c20 secure :-1 s=READY pgs=3280 cs=0 l=1 rx=0x7f5738001430 tx=0x7f574400fd30).stop 2019-12-05T06:49:09.250 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:09.239+0000 7f574ef06700 1 -- 172.21.15.42:0/1484651500 shutdown_connections 2019-12-05T06:49:09.250 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:09.239+0000 7f574ef06700 1 --2- 172.21.15.42:0/1484651500 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f572803acc0 0x7f572803d1d0 unknown :-1 s=CLOSED pgs=1375 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:49:09.251 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:09.239+0000 7f574ef06700 1 --2- 172.21.15.42:0/1484651500 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5748063ac0 0x7f5748126c20 unknown :-1 s=CLOSED pgs=3280 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:49:09.251 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:09.239+0000 7f574ef06700 1 -- 172.21.15.42:0/1484651500 shutdown_connections 2019-12-05T06:49:09.251 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:09.243+0000 7f574ef06700 1 -- 172.21.15.42:0/1484651500 wait complete. 2019-12-05T06:49:09.251 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:09.243+0000 7f574ef06700 1 -- 172.21.15.42:0/1484651500 >> 172.21.15.42:0/1484651500 conn(0x7f57480a89d0 msgr2=0x7f574809f620 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:49:09.261 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2019-12-05T06:49:10.560 INFO:tasks.workunit.client.0.smithi042.stderr:+ cd ../.. 2019-12-05T06:49:10.561 INFO:tasks.workunit.client.0.smithi042.stderr:+ ver=v0.8 2019-12-05T06:49:10.561 INFO:tasks.workunit.client.0.smithi042.stderr:+ echo checking v0.8 2019-12-05T06:49:10.561 INFO:tasks.workunit.client.0.smithi042.stderr:+ cd .snap/v0.8 2019-12-05T06:49:10.562 INFO:tasks.workunit.client.0.smithi042.stdout:checking v0.8 2019-12-05T06:49:10.565 INFO:tasks.workunit.client.0.smithi042.stderr:+ git diff --exit-code 2019-12-05T06:49:11.262 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:49:11.364 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:11.351+0000 7f90a39ee700 1 Processor -- start 2019-12-05T06:49:11.365 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:11.355+0000 7f90a39ee700 1 -- start start 2019-12-05T06:49:11.365 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:11.355+0000 7f90a39ee700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f909c08df70 0x7f909c09f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:49:11.365 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:11.355+0000 7f90a39ee700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f909c09fc50 con 0x7f909c09f820 2019-12-05T06:49:11.365 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:11.355+0000 7f90a39ee700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f909c0a0700 con 0x7f909c0a7730 2019-12-05T06:49:11.365 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:11.355+0000 7f90a39ee700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f909c0a0af0 con 0x7f909c08df70 2019-12-05T06:49:11.365 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:11.355+0000 7f90a1f8b700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f909c0a7730 0x7f909c07e710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:37740/0 (socket says 172.21.15.42:37740) 2019-12-05T06:49:11.366 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:11.355+0000 7f90a1f8b700 1 -- 172.21.15.42:0/199720900 learned_addr learned my addr 172.21.15.42:0/199720900 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:49:11.366 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:11.355+0000 7f90a178a700 1 --2- 172.21.15.42:0/199720900 >> v2:172.21.15.42:3300/0 conn(0x7f909c08df70 0x7f909c09f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:49:11.366 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:11.355+0000 7f90a178a700 1 -- 172.21.15.42:0/199720900 >> v1:172.21.15.42:6789/0 conn(0x7f909c0a7730 legacy=0x7f909c07e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:49:11.366 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:11.355+0000 7f90a178a700 1 -- 172.21.15.42:0/199720900 >> v1:172.21.15.40:6789/0 conn(0x7f909c09f820 legacy=0x7f909c14efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:49:11.366 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:11.355+0000 7f90a178a700 1 -- 172.21.15.42:0/199720900 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f909c07ef10 con 0x7f909c08df70 2019-12-05T06:49:11.366 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:11.355+0000 7f90a178a700 1 --2- 172.21.15.42:0/199720900 >> v2:172.21.15.42:3300/0 conn(0x7f909c08df70 0x7f909c09f2d0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f908c008790 tx=0x7f908c00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:11.367 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:11.355+0000 7f90a178a700 1 --2- 172.21.15.42:0/199720900 >> v2:172.21.15.42:3300/0 conn(0x7f909c08df70 0x7f909c09f2d0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f908c008790 tx=0x7f908c00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:11.367 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:11.355+0000 7f90a178a700 1 --2- 172.21.15.42:0/199720900 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f909c08df70 0x7f909c09f2d0 secure :-1 s=READY pgs=3282 cs=0 l=1 rx=0x7f908c008790 tx=0x7f908c00fbf0).ready entity=mon.1 client_cookie=26d2c95615f5dffb server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:49:11.367 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:11.355+0000 7f90a178a700 1 --2- 172.21.15.42:0/199720900 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f909c08df70 0x7f909c09f2d0 secure :-1 s=THROTTLE_DONE pgs=3282 cs=0 l=1 rx=0x7f908c008790 tx=0x7f908c00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:11.367 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:11.355+0000 7f90a0f89700 1 -- 172.21.15.42:0/199720900 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f908c011270 con 0x7f909c08df70 2019-12-05T06:49:11.367 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:11.355+0000 7f90a178a700 1 --2- 172.21.15.42:0/199720900 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f909c08df70 0x7f909c09f2d0 secure :-1 s=THROTTLE_DONE pgs=3282 cs=0 l=1 rx=0x7f908c008790 tx=0x7f908c00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:11.367 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:11.355+0000 7f90a178a700 1 --2- 172.21.15.42:0/199720900 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f909c08df70 0x7f909c09f2d0 secure :-1 s=THROTTLE_DONE pgs=3282 cs=0 l=1 rx=0x7f908c008790 tx=0x7f908c00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:11.368 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:11.355+0000 7f90a0f89700 1 -- 172.21.15.42:0/199720900 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f908c008bc0 con 0x7f909c08df70 2019-12-05T06:49:11.368 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:11.355+0000 7f90a0f89700 1 -- 172.21.15.42:0/199720900 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f908c011850 con 0x7f909c08df70 2019-12-05T06:49:11.368 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:11.355+0000 7f90a39ee700 1 -- 172.21.15.42:0/199720900 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f909c08df70 msgr2=0x7f909c09f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:49:11.368 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:11.355+0000 7f90a39ee700 1 --2- 172.21.15.42:0/199720900 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f909c08df70 0x7f909c09f2d0 secure :-1 s=READY pgs=3282 cs=0 l=1 rx=0x7f908c008790 tx=0x7f908c00fbf0).stop 2019-12-05T06:49:11.368 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:11.355+0000 7f90a39ee700 1 -- 172.21.15.42:0/199720900 shutdown_connections 2019-12-05T06:49:11.368 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:11.355+0000 7f90a39ee700 1 --2- 172.21.15.42:0/199720900 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f909c08df70 0x7f909c09f2d0 unknown :-1 s=CLOSED pgs=3282 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:49:11.368 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:11.355+0000 7f90a39ee700 1 -- 172.21.15.42:0/199720900 shutdown_connections 2019-12-05T06:49:11.369 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:11.355+0000 7f90a39ee700 1 -- 172.21.15.42:0/199720900 wait complete. 2019-12-05T06:49:11.369 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:11.355+0000 7f90a39ee700 1 -- 172.21.15.42:0/199720900 >> 172.21.15.42:0/199720900 conn(0x7f909c0aabd0 msgr2=0x7f909c0a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:49:11.374 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:11.359+0000 7f90a39ee700 1 Processor -- start 2019-12-05T06:49:11.374 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:11.359+0000 7f90a39ee700 1 -- start start 2019-12-05T06:49:11.375 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:11.359+0000 7f90a39ee700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f909c126e40 0x7f909c114160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:49:11.375 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:11.359+0000 7f90a39ee700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f909c1146b0 0x7f909c1236e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:49:11.375 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:11.359+0000 7f90a39ee700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f909c144400 con 0x7f909c123c30 2019-12-05T06:49:11.375 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:11.359+0000 7f90a39ee700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f909c127b80 con 0x7f909c126e40 2019-12-05T06:49:11.375 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:11.359+0000 7f90a39ee700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f909c127270 con 0x7f909c1146b0 2019-12-05T06:49:11.375 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:11.359+0000 7f90a278c700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f909c123c30 0x7f909c143c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:37750/0 (socket says 172.21.15.42:37750) 2019-12-05T06:49:11.376 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:11.359+0000 7f90a278c700 1 -- 172.21.15.42:0/473515902 learned_addr learned my addr 172.21.15.42:0/473515902 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:49:11.376 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:11.359+0000 7f90a1f8b700 1 --2- 172.21.15.42:0/473515902 >> v2:172.21.15.40:3300/0 conn(0x7f909c126e40 0x7f909c114160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:49:11.376 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:11.359+0000 7f90a178a700 1 --2- 172.21.15.42:0/473515902 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f909c1146b0 0x7f909c1236e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:49:11.376 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:11.359+0000 7f90927fc700 1 -- 172.21.15.42:0/473515902 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3984476541 0 0) 0x7f909c144400 con 0x7f909c123c30 2019-12-05T06:49:11.376 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:11.359+0000 7f90927fc700 1 -- 172.21.15.42:0/473515902 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90840030b0 con 0x7f909c123c30 2019-12-05T06:49:11.377 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:11.359+0000 7f90927fc700 1 -- 172.21.15.42:0/473515902 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 1538729941 0 0) 0x7f90840030b0 con 0x7f909c123c30 2019-12-05T06:49:11.377 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:11.359+0000 7f90927fc700 1 -- 172.21.15.42:0/473515902 --> v1:172.21.15.42:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f909c144400 con 0x7f909c123c30 2019-12-05T06:49:11.377 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:11.359+0000 7f90927fc700 1 -- 172.21.15.42:0/473515902 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7f9094001ca0 con 0x7f909c123c30 2019-12-05T06:49:11.377 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:11.359+0000 7f90927fc700 1 -- 172.21.15.42:0/473515902 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f909c1146b0 msgr2=0x7f909c1236e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:49:11.377 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:11.359+0000 7f90927fc700 1 --2- 172.21.15.42:0/473515902 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f909c1146b0 0x7f909c1236e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:49:11.378 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:11.359+0000 7f90927fc700 1 -- 172.21.15.42:0/473515902 >> v2:172.21.15.40:3300/0 conn(0x7f909c126e40 msgr2=0x7f909c114160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:49:11.378 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:11.359+0000 7f90927fc700 1 --2- 172.21.15.42:0/473515902 >> v2:172.21.15.40:3300/0 conn(0x7f909c126e40 0x7f909c114160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:49:11.378 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:11.359+0000 7f90927fc700 1 -- 172.21.15.42:0/473515902 >> v1:172.21.15.42:6789/0 conn(0x7f909c123c30 legacy=0x7f909c143c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:49:11.378 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:11.359+0000 7f90927fc700 1 --2- 172.21.15.42:0/473515902 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f90840038a0 0x7f9084003cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:49:11.378 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:11.359+0000 7f90927fc700 1 --2- 172.21.15.42:0/473515902 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9084004420 0x7f9084004850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:49:11.378 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:11.359+0000 7f90927fc700 1 --2- 172.21.15.42:0/473515902 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9084004e90 0x7f908400b5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:49:11.379 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:11.359+0000 7f90927fc700 1 -- 172.21.15.42:0/473515902 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f909c127b80 con 0x7f90840038a0 2019-12-05T06:49:11.379 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:11.359+0000 7f90927fc700 1 -- 172.21.15.42:0/473515902 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f909c127270 con 0x7f9084004420 2019-12-05T06:49:11.379 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:11.359+0000 7f90927fc700 1 -- 172.21.15.42:0/473515902 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f908400bbe0 con 0x7f9084004e90 2019-12-05T06:49:11.379 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:11.363+0000 7f90a178a700 1 --2- 172.21.15.42:0/473515902 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9084004420 0x7f9084004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:49:11.379 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:11.363+0000 7f90a1f8b700 1 --2- 172.21.15.42:0/473515902 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f90840038a0 0x7f9084003cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:49:11.379 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:11.363+0000 7f90a278c700 1 --2- 172.21.15.42:0/473515902 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9084004e90 0x7f908400b5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:49:11.379 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:11.363+0000 7f90a178a700 1 -- 172.21.15.42:0/473515902 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9084004e90 msgr2=0x7f908400b5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:49:11.380 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:11.363+0000 7f90a178a700 1 --2- 172.21.15.42:0/473515902 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9084004e90 0x7f908400b5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:49:11.380 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:11.363+0000 7f90a178a700 1 -- 172.21.15.42:0/473515902 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f90840038a0 msgr2=0x7f9084003cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:49:11.380 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:11.363+0000 7f90a178a700 1 --2- 172.21.15.42:0/473515902 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f90840038a0 0x7f9084003cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:49:11.380 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:11.363+0000 7f90a178a700 1 -- 172.21.15.42:0/473515902 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f909c144630 con 0x7f9084004420 2019-12-05T06:49:11.380 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:11.363+0000 7f90a278c700 1 --2- 172.21.15.42:0/473515902 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9084004e90 0x7f908400b5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).send_auth_request state changed! 2019-12-05T06:49:11.380 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:11.363+0000 7f90a39ee700 1 -- 172.21.15.42:0/473515902 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f909c1146b0 con 0x7f9084004420 2019-12-05T06:49:11.380 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:11.363+0000 7f90a178a700 1 --2- 172.21.15.42:0/473515902 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9084004420 0x7f9084004850 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f908c000b20 tx=0x7f908c011e50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:11.381 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:11.363+0000 7f90a39ee700 1 -- 172.21.15.42:0/473515902 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f909c114920 con 0x7f9084004420 2019-12-05T06:49:11.381 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:11.363+0000 7f90a178a700 1 --2- 172.21.15.42:0/473515902 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9084004420 0x7f9084004850 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f908c000b20 tx=0x7f908c011e50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:11.381 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:11.363+0000 7f90a178a700 1 --2- 172.21.15.42:0/473515902 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9084004420 0x7f9084004850 secure :-1 s=READY pgs=3284 cs=0 l=1 rx=0x7f908c000b20 tx=0x7f908c011e50).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:49:11.381 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:11.363+0000 7f90a178a700 1 --2- 172.21.15.42:0/473515902 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9084004420 0x7f9084004850 secure :-1 s=THROTTLE_DONE pgs=3284 cs=0 l=1 rx=0x7f908c000b20 tx=0x7f908c011e50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:11.381 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:11.363+0000 7f90a178a700 1 --2- 172.21.15.42:0/473515902 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9084004420 0x7f9084004850 secure :-1 s=THROTTLE_DONE pgs=3284 cs=0 l=1 rx=0x7f908c000b20 tx=0x7f908c011e50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:11.381 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:11.363+0000 7f90a178a700 1 --2- 172.21.15.42:0/473515902 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9084004420 0x7f9084004850 secure :-1 s=THROTTLE_DONE pgs=3284 cs=0 l=1 rx=0x7f908c000b20 tx=0x7f908c011e50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:11.382 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:11.363+0000 7f90927fc700 1 -- 172.21.15.42:0/473515902 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f908c0070b0 con 0x7f9084004420 2019-12-05T06:49:11.382 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:11.363+0000 7f90927fc700 1 -- 172.21.15.42:0/473515902 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f908c011470 con 0x7f9084004420 2019-12-05T06:49:11.382 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:11.363+0000 7f90927fc700 1 -- 172.21.15.42:0/473515902 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f908c0135c0 con 0x7f9084004420 2019-12-05T06:49:11.382 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:11.363+0000 7f90a178a700 1 --2- 172.21.15.42:0/473515902 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9084004420 0x7f9084004850 secure :-1 s=THROTTLE_DONE pgs=3284 cs=0 l=1 rx=0x7f908c000b20 tx=0x7f908c011e50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:11.382 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:11.363+0000 7f90927fc700 1 -- 172.21.15.42:0/473515902 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f908c0137c0 con 0x7f9084004420 2019-12-05T06:49:11.382 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:11.363+0000 7f90a178a700 1 --2- 172.21.15.42:0/473515902 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9084004420 0x7f9084004850 secure :-1 s=THROTTLE_DONE pgs=3284 cs=0 l=1 rx=0x7f908c000b20 tx=0x7f908c011e50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:11.382 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:11.363+0000 7f90927fc700 1 --2- 172.21.15.42:0/473515902 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9084043b20 0x7f9084046030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:49:11.383 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:11.363+0000 7f90927fc700 1 -- 172.21.15.42:0/473515902 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 12223+0+0 (secure 0 0 0) 0x7f908c04bd70 con 0x7f9084004420 2019-12-05T06:49:11.383 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:11.363+0000 7f90a1f8b700 1 --2- 172.21.15.42:0/473515902 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9084043b20 0x7f9084046030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:49:11.383 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:11.363+0000 7f90a1f8b700 1 --2- 172.21.15.42:0/473515902 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9084043b20 0x7f9084046030 crc :-1 s=READY pgs=1376 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:49:11.384 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:11.367+0000 7f90a39ee700 1 -- 172.21.15.42:0/473515902 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9078003200 con 0x7f9084004420 2019-12-05T06:49:11.384 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:11.375+0000 7f90a178a700 1 --2- 172.21.15.42:0/473515902 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9084004420 0x7f9084004850 secure :-1 s=THROTTLE_DONE pgs=3284 cs=0 l=1 rx=0x7f908c000b20 tx=0x7f908c011e50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:11.384 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:11.375+0000 7f90927fc700 1 -- 172.21.15.42:0/473515902 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f908c032e10 con 0x7f9084004420 2019-12-05T06:49:11.545 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:11.539+0000 7f90a39ee700 1 -- 172.21.15.42:0/473515902 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9078003630 con 0x7f9084004420 2019-12-05T06:49:11.546 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:11.539+0000 7f90a178a700 1 --2- 172.21.15.42:0/473515902 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9084004420 0x7f9084004850 secure :-1 s=THROTTLE_DONE pgs=3284 cs=0 l=1 rx=0x7f908c000b20 tx=0x7f908c011e50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:11.546 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:11.539+0000 7f90927fc700 1 -- 172.21.15.42:0/473515902 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 398 v398) v1 ==== 95+0+1776 (secure 0 0 0) 0x7f908c033070 con 0x7f9084004420 2019-12-05T06:49:11.546 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 398 2019-12-05T06:49:11.546 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:49:11.547 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":398,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":398,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:49:10.408613+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":121,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12687},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12687":{"gid":12687,"name":"b","rank":0,"incarnation":395,"state":"up:active","state_seq":12,"addr":"172.21.15.40:6833/815316041","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":815316041},{"type":"v1","addr":"172.21.15.40:6833","nonce":815316041}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:49:11.548 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:11.543+0000 7f90a39ee700 1 -- 172.21.15.42:0/473515902 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9084043b20 msgr2=0x7f9084046030 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:49:11.548 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:11.543+0000 7f90a39ee700 1 --2- 172.21.15.42:0/473515902 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f9084043b20 0x7f9084046030 crc :-1 s=READY pgs=1376 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:49:11.549 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:11.543+0000 7f90a39ee700 1 -- 172.21.15.42:0/473515902 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9084004420 msgr2=0x7f9084004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:49:11.549 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:11.543+0000 7f90a39ee700 1 --2- 172.21.15.42:0/473515902 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9084004420 0x7f9084004850 secure :-1 s=READY pgs=3284 cs=0 l=1 rx=0x7f908c000b20 tx=0x7f908c011e50).stop 2019-12-05T06:49:11.549 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:11.543+0000 7f90a278c700 1 -- 172.21.15.42:0/473515902 reap_dead start 2019-12-05T06:49:11.549 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:11.543+0000 7f90a39ee700 1 -- 172.21.15.42:0/473515902 shutdown_connections 2019-12-05T06:49:11.550 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:11.543+0000 7f90a39ee700 1 -- 172.21.15.42:0/473515902 shutdown_connections 2019-12-05T06:49:11.550 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:11.543+0000 7f90a39ee700 1 -- 172.21.15.42:0/473515902 wait complete. 2019-12-05T06:49:11.550 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:11.543+0000 7f90a39ee700 1 -- 172.21.15.42:0/473515902 >> 172.21.15.42:0/473515902 conn(0x7f909c0a8290 msgr2=0x7f909c116350 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:49:11.559 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b has gained rank=0, replacing gid=12847 2019-12-05T06:49:11.559 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 4 secs before reviving mds.a 2019-12-05T06:49:14.256 INFO:tasks.workunit.client.0.smithi042.stderr:+ cd ../.. 2019-12-05T06:49:14.256 INFO:tasks.workunit.client.0.smithi042.stdout:checking v0.9 2019-12-05T06:49:14.257 INFO:tasks.workunit.client.0.smithi042.stderr:+ ver=v0.9 2019-12-05T06:49:14.257 INFO:tasks.workunit.client.0.smithi042.stderr:+ echo checking v0.9 2019-12-05T06:49:14.257 INFO:tasks.workunit.client.0.smithi042.stderr:+ cd .snap/v0.9 2019-12-05T06:49:14.261 INFO:tasks.workunit.client.0.smithi042.stderr:+ git diff --exit-code 2019-12-05T06:49:15.560 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.a 2019-12-05T06:49:15.561 INFO:tasks.ceph.mds.a:Restarting daemon 2019-12-05T06:49:15.561 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2019-12-05T06:49:15.565 INFO:tasks.ceph.mds.a:Started 2019-12-05T06:49:15.566 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:49:15.644 INFO:tasks.ceph.mds.a.smithi042.stdout:starting mds.a at 2019-12-05T06:49:15.663 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:15.651+0000 7f7a2788b700 1 Processor -- start 2019-12-05T06:49:15.663 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:15.651+0000 7f7a2788b700 1 -- start start 2019-12-05T06:49:15.663 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:15.651+0000 7f7a2788b700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f7a200a7730 0x7f7a2008df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:49:15.663 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:15.651+0000 7f7a2788b700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7a2007e710 0x7f7a2007eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:49:15.664 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:15.651+0000 7f7a2788b700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a200a0700 con 0x7f7a2009f2d0 2019-12-05T06:49:15.664 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:15.651+0000 7f7a2788b700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a200a0af0 con 0x7f7a2007e710 2019-12-05T06:49:15.664 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:15.651+0000 7f7a2788b700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a2008e5a0 con 0x7f7a200a7730 2019-12-05T06:49:15.664 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:15.651+0000 7f7a25e28700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f7a200a7730 0x7f7a2008df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:49:15.664 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:15.651+0000 7f7a25e28700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f7a200a7730 0x7f7a2008df70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:37774/0 (socket says 172.21.15.42:37774) 2019-12-05T06:49:15.665 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:15.651+0000 7f7a25e28700 1 -- 172.21.15.42:0/3517232963 learned_addr learned my addr 172.21.15.42:0/3517232963 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:49:15.665 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:15.651+0000 7f7a25627700 1 --2- 172.21.15.42:0/3517232963 >> v2:172.21.15.40:3300/0 conn(0x7f7a2007e710 0x7f7a2007eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:49:15.665 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:15.651+0000 7f7a25627700 1 -- 172.21.15.42:0/3517232963 >> v2:172.21.15.40:3300/0 conn(0x7f7a2007e710 msgr2=0x7f7a2007eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2019-12-05T06:49:15.665 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:15.651+0000 7f7a25627700 1 -- 172.21.15.42:0/3517232963 >> v2:172.21.15.40:3300/0 conn(0x7f7a2007e710 msgr2=0x7f7a2007eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2019-12-05T06:49:15.665 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:15.651+0000 7f7a25627700 1 --2- 172.21.15.42:0/3517232963 >> v2:172.21.15.40:3300/0 conn(0x7f7a2007e710 0x7f7a2007eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-12-05T06:49:15.666 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:15.651+0000 7f7a25627700 1 --2- 172.21.15.42:0/3517232963 >> v2:172.21.15.40:3300/0 conn(0x7f7a2007e710 0x7f7a2007eb40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rx=0 tx=0)._fault waiting 0.200000 2019-12-05T06:49:15.666 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:15.651+0000 7f7a25e28700 1 -- 172.21.15.42:0/3517232963 >> v2:172.21.15.40:3300/0 conn(0x7f7a2007e710 msgr2=0x7f7a2007eb40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2019-12-05T06:49:15.666 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:15.651+0000 7f7a25e28700 1 --2- 172.21.15.42:0/3517232963 >> v2:172.21.15.40:3300/0 conn(0x7f7a2007e710 0x7f7a2007eb40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:49:15.666 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:15.651+0000 7f7a25e28700 1 -- 172.21.15.42:0/3517232963 >> v1:172.21.15.42:6789/0 conn(0x7f7a2009f2d0 legacy=0x7f7a2009f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:49:15.666 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:15.651+0000 7f7a25e28700 1 -- 172.21.15.42:0/3517232963 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a2007f090 con 0x7f7a200a7730 2019-12-05T06:49:15.666 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:15.655+0000 7f7a25e28700 1 --2- 172.21.15.42:0/3517232963 >> v2:172.21.15.42:3300/0 conn(0x7f7a200a7730 0x7f7a2008df70 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f7a1c002f30 tx=0x7f7a1c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:15.666 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:15.655+0000 7f7a25e28700 1 --2- 172.21.15.42:0/3517232963 >> v2:172.21.15.42:3300/0 conn(0x7f7a200a7730 0x7f7a2008df70 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f7a1c002f30 tx=0x7f7a1c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:15.667 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:15.655+0000 7f7a25e28700 1 --2- 172.21.15.42:0/3517232963 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7a200a7730 0x7f7a2008df70 secure :-1 s=READY pgs=3288 cs=0 l=1 rx=0x7f7a1c002f30 tx=0x7f7a1c00fd30).ready entity=mon.1 client_cookie=d07da802914f3e92 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:49:15.667 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:15.655+0000 7f7a25e28700 1 --2- 172.21.15.42:0/3517232963 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7a200a7730 0x7f7a2008df70 secure :-1 s=THROTTLE_DONE pgs=3288 cs=0 l=1 rx=0x7f7a1c002f30 tx=0x7f7a1c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:15.667 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:15.655+0000 7f7a25e28700 1 --2- 172.21.15.42:0/3517232963 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7a200a7730 0x7f7a2008df70 secure :-1 s=THROTTLE_DONE pgs=3288 cs=0 l=1 rx=0x7f7a1c002f30 tx=0x7f7a1c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:15.667 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:15.655+0000 7f7a24e26700 1 -- 172.21.15.42:0/3517232963 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7a1c011370 con 0x7f7a200a7730 2019-12-05T06:49:15.667 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:15.655+0000 7f7a25e28700 1 --2- 172.21.15.42:0/3517232963 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7a200a7730 0x7f7a2008df70 secure :-1 s=THROTTLE_DONE pgs=3288 cs=0 l=1 rx=0x7f7a1c002f30 tx=0x7f7a1c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:15.667 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:15.655+0000 7f7a24e26700 1 -- 172.21.15.42:0/3517232963 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a1c008bb0 con 0x7f7a200a7730 2019-12-05T06:49:15.668 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:15.655+0000 7f7a24e26700 1 -- 172.21.15.42:0/3517232963 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7a1c011950 con 0x7f7a200a7730 2019-12-05T06:49:15.668 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:15.655+0000 7f7a2788b700 1 -- 172.21.15.42:0/3517232963 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7a200a7730 msgr2=0x7f7a2008df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:49:15.668 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:15.655+0000 7f7a2788b700 1 --2- 172.21.15.42:0/3517232963 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7a200a7730 0x7f7a2008df70 secure :-1 s=READY pgs=3288 cs=0 l=1 rx=0x7f7a1c002f30 tx=0x7f7a1c00fd30).stop 2019-12-05T06:49:15.668 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:15.655+0000 7f7a2788b700 1 -- 172.21.15.42:0/3517232963 shutdown_connections 2019-12-05T06:49:15.669 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:15.655+0000 7f7a2788b700 1 --2- 172.21.15.42:0/3517232963 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7a200a7730 0x7f7a2008df70 unknown :-1 s=CLOSED pgs=3288 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:49:15.669 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:15.655+0000 7f7a2788b700 1 --2- 172.21.15.42:0/3517232963 >> v2:172.21.15.40:3300/0 conn(0x7f7a2007e710 0x7f7a2007eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:49:15.669 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:15.655+0000 7f7a2788b700 1 -- 172.21.15.42:0/3517232963 shutdown_connections 2019-12-05T06:49:15.669 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:15.655+0000 7f7a2788b700 1 -- 172.21.15.42:0/3517232963 wait complete. 2019-12-05T06:49:15.669 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:15.655+0000 7f7a2788b700 1 -- 172.21.15.42:0/3517232963 >> 172.21.15.42:0/3517232963 conn(0x7f7a200aabd0 msgr2=0x7f7a200a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:49:15.672 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:15.655+0000 7f7a2788b700 1 Processor -- start 2019-12-05T06:49:15.673 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:15.655+0000 7f7a2788b700 1 -- start start 2019-12-05T06:49:15.673 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:15.659+0000 7f7a2788b700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7a20117d70 0x7f7a200a6770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:49:15.673 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:15.659+0000 7f7a2788b700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7a2012e740 0x7f7a2008f200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:49:15.673 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:15.659+0000 7f7a2788b700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a2012df30 con 0x7f7a2008f750 2019-12-05T06:49:15.673 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:15.659+0000 7f7a2788b700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a2008ecc0 con 0x7f7a20117d70 2019-12-05T06:49:15.673 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:15.659+0000 7f7a2788b700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a2008ef60 con 0x7f7a2012e740 2019-12-05T06:49:15.674 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:15.659+0000 7f7a25627700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7a2012e740 0x7f7a2008f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:49:15.674 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:15.659+0000 7f7a25e28700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7a20117d70 0x7f7a200a6770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:49:15.674 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:15.659+0000 7f7a25627700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7a2012e740 0x7f7a2008f200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:38248/0 (socket says 172.21.15.42:38248) 2019-12-05T06:49:15.674 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:15.659+0000 7f7a25627700 1 -- 172.21.15.42:0/4098529564 learned_addr learned my addr 172.21.15.42:0/4098529564 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:49:15.674 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:15.659+0000 7f7a25627700 1 -- 172.21.15.42:0/4098529564 >> v2:172.21.15.40:3300/0 conn(0x7f7a20117d70 msgr2=0x7f7a200a6770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:49:15.675 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:15.659+0000 7f7a25627700 1 --2- 172.21.15.42:0/4098529564 >> v2:172.21.15.40:3300/0 conn(0x7f7a20117d70 0x7f7a200a6770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:49:15.675 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:15.659+0000 7f7a25627700 1 -- 172.21.15.42:0/4098529564 >> v1:172.21.15.40:6789/0 conn(0x7f7a2008f750 legacy=0x7f7a201314f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:49:15.675 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:15.659+0000 7f7a25627700 1 -- 172.21.15.42:0/4098529564 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a200a6e10 con 0x7f7a2012e740 2019-12-05T06:49:15.676 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:15.659+0000 7f7a25627700 1 --2- 172.21.15.42:0/4098529564 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7a2012e740 0x7f7a2008f200 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f7a10002f30 tx=0x7f7a10010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:15.676 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:15.659+0000 7f7a25627700 1 --2- 172.21.15.42:0/4098529564 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7a2012e740 0x7f7a2008f200 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f7a10002f30 tx=0x7f7a10010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:15.676 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:15.659+0000 7f7a25627700 1 --2- 172.21.15.42:0/4098529564 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7a2012e740 0x7f7a2008f200 secure :-1 s=READY pgs=727 cs=0 l=1 rx=0x7f7a10002f30 tx=0x7f7a10010b40).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:49:15.676 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:15.659+0000 7f7a25627700 1 --2- 172.21.15.42:0/4098529564 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7a2012e740 0x7f7a2008f200 secure :-1 s=THROTTLE_DONE pgs=727 cs=0 l=1 rx=0x7f7a10002f30 tx=0x7f7a10010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:15.677 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:15.659+0000 7f7a167fc700 1 -- 172.21.15.42:0/4098529564 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7a10005930 con 0x7f7a2012e740 2019-12-05T06:49:15.677 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:15.659+0000 7f7a25627700 1 --2- 172.21.15.42:0/4098529564 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7a2012e740 0x7f7a2008f200 secure :-1 s=THROTTLE_DONE pgs=727 cs=0 l=1 rx=0x7f7a10002f30 tx=0x7f7a10010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:15.677 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:15.659+0000 7f7a25627700 1 --2- 172.21.15.42:0/4098529564 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7a2012e740 0x7f7a2008f200 secure :-1 s=THROTTLE_DONE pgs=727 cs=0 l=1 rx=0x7f7a10002f30 tx=0x7f7a10010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:15.677 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:15.659+0000 7f7a167fc700 1 -- 172.21.15.42:0/4098529564 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a10007160 con 0x7f7a2012e740 2019-12-05T06:49:15.677 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:15.659+0000 7f7a2788b700 1 -- 172.21.15.42:0/4098529564 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a200a5de0 con 0x7f7a2012e740 2019-12-05T06:49:15.678 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:15.659+0000 7f7a167fc700 1 -- 172.21.15.42:0/4098529564 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7a10012200 con 0x7f7a2012e740 2019-12-05T06:49:15.678 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:15.659+0000 7f7a2788b700 1 -- 172.21.15.42:0/4098529564 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a2012ed10 con 0x7f7a2012e740 2019-12-05T06:49:15.678 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:15.659+0000 7f7a25627700 1 --2- 172.21.15.42:0/4098529564 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7a2012e740 0x7f7a2008f200 secure :-1 s=THROTTLE_DONE pgs=727 cs=0 l=1 rx=0x7f7a10002f30 tx=0x7f7a10010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:15.678 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:15.663+0000 7f7a167fc700 1 -- 172.21.15.42:0/4098529564 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f7a10012510 con 0x7f7a2012e740 2019-12-05T06:49:15.679 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:15.663+0000 7f7a25627700 1 --2- 172.21.15.42:0/4098529564 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7a2012e740 0x7f7a2008f200 secure :-1 s=THROTTLE_DONE pgs=727 cs=0 l=1 rx=0x7f7a10002f30 tx=0x7f7a10010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:15.679 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:15.663+0000 7f7a167fc700 1 --2- 172.21.15.42:0/4098529564 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f7a0803ae70 0x7f7a0803d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:49:15.679 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:15.663+0000 7f7a167fc700 1 -- 172.21.15.42:0/4098529564 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 12223+0+0 (secure 0 0 0) 0x7f7a1004b360 con 0x7f7a2012e740 2019-12-05T06:49:15.679 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:15.663+0000 7f7a25e28700 1 --2- 172.21.15.42:0/4098529564 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f7a0803ae70 0x7f7a0803d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:49:15.679 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:15.663+0000 7f7a25e28700 1 --2- 172.21.15.42:0/4098529564 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f7a0803ae70 0x7f7a0803d380 crc :-1 s=READY pgs=1377 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:49:15.680 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:15.667+0000 7f7a2788b700 1 -- 172.21.15.42:0/4098529564 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a2005d100 con 0x7f7a2012e740 2019-12-05T06:49:15.682 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:15.675+0000 7f7a25627700 1 --2- 172.21.15.42:0/4098529564 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7a2012e740 0x7f7a2008f200 secure :-1 s=THROTTLE_DONE pgs=727 cs=0 l=1 rx=0x7f7a10002f30 tx=0x7f7a10010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:15.682 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:15.675+0000 7f7a167fc700 1 -- 172.21.15.42:0/4098529564 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f7a200a6e10 con 0x7f7a2012e740 2019-12-05T06:49:15.857 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:15.851+0000 7f7a2788b700 1 -- 172.21.15.42:0/4098529564 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7a2005b8b0 con 0x7f7a2012e740 2019-12-05T06:49:15.858 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:15.851+0000 7f7a25627700 1 --2- 172.21.15.42:0/4098529564 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7a2012e740 0x7f7a2008f200 secure :-1 s=THROTTLE_DONE pgs=727 cs=0 l=1 rx=0x7f7a10002f30 tx=0x7f7a10010b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:15.858 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:15.851+0000 7f7a167fc700 1 -- 172.21.15.42:0/4098529564 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 398 v398) v1 ==== 95+0+1776 (secure 0 0 0) 0x7f7a10032640 con 0x7f7a2012e740 2019-12-05T06:49:15.861 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:49:15.861 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":398,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":398,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:49:10.408613+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":121,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12687},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12687":{"gid":12687,"name":"b","rank":0,"incarnation":395,"state":"up:active","state_seq":12,"addr":"172.21.15.40:6833/815316041","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":815316041},{"type":"v1","addr":"172.21.15.40:6833","nonce":815316041}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:49:15.861 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 398 2019-12-05T06:49:15.863 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:15.855+0000 7f7a2788b700 1 -- 172.21.15.42:0/4098529564 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f7a0803ae70 msgr2=0x7f7a0803d380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:49:15.863 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:15.855+0000 7f7a2788b700 1 --2- 172.21.15.42:0/4098529564 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f7a0803ae70 0x7f7a0803d380 crc :-1 s=READY pgs=1377 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:49:15.863 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:15.855+0000 7f7a2788b700 1 -- 172.21.15.42:0/4098529564 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7a2012e740 msgr2=0x7f7a2008f200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:49:15.863 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:15.855+0000 7f7a2788b700 1 --2- 172.21.15.42:0/4098529564 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7a2012e740 0x7f7a2008f200 secure :-1 s=READY pgs=727 cs=0 l=1 rx=0x7f7a10002f30 tx=0x7f7a10010b40).stop 2019-12-05T06:49:15.863 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:15.855+0000 7f7a2788b700 1 -- 172.21.15.42:0/4098529564 shutdown_connections 2019-12-05T06:49:15.863 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:15.855+0000 7f7a2788b700 1 --2- 172.21.15.42:0/4098529564 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f7a0803ae70 0x7f7a0803d380 unknown :-1 s=CLOSED pgs=1377 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:49:15.864 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:15.855+0000 7f7a2788b700 1 --2- 172.21.15.42:0/4098529564 >> v2:172.21.15.40:3300/0 conn(0x7f7a20117d70 0x7f7a200a6770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:49:15.864 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:15.855+0000 7f7a2788b700 1 --2- 172.21.15.42:0/4098529564 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7a2012e740 0x7f7a2008f200 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:49:15.864 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:15.855+0000 7f7a2788b700 1 -- 172.21.15.42:0/4098529564 shutdown_connections 2019-12-05T06:49:15.864 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:15.855+0000 7f7a2788b700 1 -- 172.21.15.42:0/4098529564 wait complete. 2019-12-05T06:49:15.864 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:15.855+0000 7f7a2788b700 1 -- 172.21.15.42:0/4098529564 >> 172.21.15.42:0/4098529564 conn(0x7f7a200a9760 msgr2=0x7f7a2009f570 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:49:15.873 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2019-12-05T06:49:17.874 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:49:17.992 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:17.979+0000 7f8b44642700 1 Processor -- start 2019-12-05T06:49:17.992 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:17.979+0000 7f8b44642700 1 -- start start 2019-12-05T06:49:17.992 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:17.979+0000 7f8b44642700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8b3c09fae0 0x7f8b3c14fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:49:17.992 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:17.979+0000 7f8b44642700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b3c0a0710 con 0x7f8b3c08df80 2019-12-05T06:49:17.992 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:17.979+0000 7f8b44642700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b3c07ef20 con 0x7f8b3c0a7740 2019-12-05T06:49:17.992 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:17.979+0000 7f8b44642700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b3c0a0b00 con 0x7f8b3c09fae0 2019-12-05T06:49:17.993 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:17.979+0000 7f8b42bdf700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f8b3c0a7740 0x7f8b3c07e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:37836/0 (socket says 172.21.15.42:37836) 2019-12-05T06:49:17.993 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:17.979+0000 7f8b42bdf700 1 -- 172.21.15.42:0/2886982453 learned_addr learned my addr 172.21.15.42:0/2886982453 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:49:17.993 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:17.979+0000 7f8b433e0700 1 --2- 172.21.15.42:0/2886982453 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8b3c09fae0 0x7f8b3c14fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:49:17.993 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:17.979+0000 7f8b41bdd700 1 -- 172.21.15.42:0/2886982453 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1295477883 0 0) 0x7f8b3c07ef20 con 0x7f8b3c0a7740 2019-12-05T06:49:17.993 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:17.979+0000 7f8b41bdd700 1 -- 172.21.15.42:0/2886982453 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b280030b0 con 0x7f8b3c0a7740 2019-12-05T06:49:17.993 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:17.979+0000 7f8b41bdd700 1 -- 172.21.15.42:0/2886982453 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1629853556 0 0) 0x7f8b3c0a0710 con 0x7f8b3c08df80 2019-12-05T06:49:17.993 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:17.979+0000 7f8b41bdd700 1 -- 172.21.15.42:0/2886982453 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b3c07ef20 con 0x7f8b3c08df80 2019-12-05T06:49:17.994 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:17.983+0000 7f8b433e0700 1 -- 172.21.15.42:0/2886982453 >> v1:172.21.15.42:6789/0 conn(0x7f8b3c0a7740 legacy=0x7f8b3c07e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:49:17.994 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:17.983+0000 7f8b433e0700 1 -- 172.21.15.42:0/2886982453 >> v1:172.21.15.40:6789/0 conn(0x7f8b3c08df80 legacy=0x7f8b3c09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:49:17.994 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:17.983+0000 7f8b433e0700 1 -- 172.21.15.42:0/2886982453 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b3c0a9ba0 con 0x7f8b3c09fae0 2019-12-05T06:49:17.994 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:17.983+0000 7f8b433e0700 1 --2- 172.21.15.42:0/2886982453 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8b3c09fae0 0x7f8b3c14fff0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f8b340032b0 tx=0x7f8b3400ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:17.994 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:17.983+0000 7f8b433e0700 1 --2- 172.21.15.42:0/2886982453 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8b3c09fae0 0x7f8b3c14fff0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f8b340032b0 tx=0x7f8b3400ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:17.994 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:17.983+0000 7f8b433e0700 1 --2- 172.21.15.42:0/2886982453 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8b3c09fae0 0x7f8b3c14fff0 secure :-1 s=READY pgs=728 cs=0 l=1 rx=0x7f8b340032b0 tx=0x7f8b3400ff70).ready entity=mon.2 client_cookie=c31170ec4fd6c0cc server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:49:17.994 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:17.983+0000 7f8b433e0700 1 --2- 172.21.15.42:0/2886982453 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8b3c09fae0 0x7f8b3c14fff0 secure :-1 s=THROTTLE_DONE pgs=728 cs=0 l=1 rx=0x7f8b340032b0 tx=0x7f8b3400ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:17.995 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:17.983+0000 7f8b41bdd700 1 -- 172.21.15.42:0/2886982453 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8b340109b0 con 0x7f8b3c09fae0 2019-12-05T06:49:17.995 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:17.983+0000 7f8b433e0700 1 --2- 172.21.15.42:0/2886982453 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8b3c09fae0 0x7f8b3c14fff0 secure :-1 s=THROTTLE_DONE pgs=728 cs=0 l=1 rx=0x7f8b340032b0 tx=0x7f8b3400ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:17.995 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:17.983+0000 7f8b41bdd700 1 -- 172.21.15.42:0/2886982453 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b34008b50 con 0x7f8b3c09fae0 2019-12-05T06:49:17.995 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:17.983+0000 7f8b433e0700 1 --2- 172.21.15.42:0/2886982453 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8b3c09fae0 0x7f8b3c14fff0 secure :-1 s=THROTTLE_DONE pgs=728 cs=0 l=1 rx=0x7f8b340032b0 tx=0x7f8b3400ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:17.995 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:17.983+0000 7f8b41bdd700 1 -- 172.21.15.42:0/2886982453 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8b340110c0 con 0x7f8b3c09fae0 2019-12-05T06:49:17.995 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:17.983+0000 7f8b44642700 1 -- 172.21.15.42:0/2886982453 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8b3c09fae0 msgr2=0x7f8b3c14fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:49:17.995 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:17.983+0000 7f8b44642700 1 --2- 172.21.15.42:0/2886982453 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8b3c09fae0 0x7f8b3c14fff0 secure :-1 s=READY pgs=728 cs=0 l=1 rx=0x7f8b340032b0 tx=0x7f8b3400ff70).stop 2019-12-05T06:49:17.996 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:17.983+0000 7f8b44642700 1 -- 172.21.15.42:0/2886982453 shutdown_connections 2019-12-05T06:49:17.996 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:17.983+0000 7f8b44642700 1 --2- 172.21.15.42:0/2886982453 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8b3c09fae0 0x7f8b3c14fff0 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:49:17.996 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:17.983+0000 7f8b44642700 1 -- 172.21.15.42:0/2886982453 shutdown_connections 2019-12-05T06:49:17.996 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:17.983+0000 7f8b44642700 1 -- 172.21.15.42:0/2886982453 wait complete. 2019-12-05T06:49:17.996 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:17.983+0000 7f8b44642700 1 -- 172.21.15.42:0/2886982453 >> 172.21.15.42:0/2886982453 conn(0x7f8b3c0aabe0 msgr2=0x7f8b3c0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:49:17.996 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:17.983+0000 7f8b44642700 1 Processor -- start 2019-12-05T06:49:17.996 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:17.983+0000 7f8b44642700 1 -- start start 2019-12-05T06:49:18.000 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:17.987+0000 7f8b44642700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8b3c065930 0x7f8b3c128670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:49:18.000 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:17.987+0000 7f8b44642700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b3c127b80 con 0x7f8b3c1245c0 2019-12-05T06:49:18.000 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:17.987+0000 7f8b44642700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b3c1249f0 con 0x7f8b3c07bab0 2019-12-05T06:49:18.001 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:17.987+0000 7f8b44642700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b3c127e90 con 0x7f8b3c065930 2019-12-05T06:49:18.001 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:17.987+0000 7f8b42bdf700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8b3c065930 0x7f8b3c128670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:49:18.001 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:17.987+0000 7f8b433e0700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f8b3c07bab0 0x7f8b3c12efd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:37844/0 (socket says 172.21.15.42:37844) 2019-12-05T06:49:18.001 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:17.987+0000 7f8b433e0700 1 -- 172.21.15.42:0/127263589 learned_addr learned my addr 172.21.15.42:0/127263589 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:49:18.001 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:17.987+0000 7f8b337fe700 1 -- 172.21.15.42:0/127263589 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1408698543 0 0) 0x7f8b3c1249f0 con 0x7f8b3c07bab0 2019-12-05T06:49:18.001 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:17.987+0000 7f8b337fe700 1 -- 172.21.15.42:0/127263589 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b240030b0 con 0x7f8b3c07bab0 2019-12-05T06:49:18.002 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:17.987+0000 7f8b42bdf700 1 -- 172.21.15.42:0/127263589 >> v1:172.21.15.42:6789/0 conn(0x7f8b3c07bab0 legacy=0x7f8b3c12efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:49:18.002 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:17.987+0000 7f8b42bdf700 1 -- 172.21.15.42:0/127263589 >> v1:172.21.15.40:6789/0 conn(0x7f8b3c1245c0 legacy=0x7f8b3c0a5b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:49:18.002 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:17.987+0000 7f8b337fe700 1 -- 172.21.15.42:0/127263589 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1968360885 0 0) 0x7f8b3c127b80 con 0x7f8b3c1245c0 2019-12-05T06:49:18.002 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:17.987+0000 7f8b42bdf700 1 -- 172.21.15.42:0/127263589 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b3c07bee0 con 0x7f8b3c065930 2019-12-05T06:49:18.002 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:17.987+0000 7f8b42bdf700 1 --2- 172.21.15.42:0/127263589 >> v2:172.21.15.40:3300/0 conn(0x7f8b3c065930 0x7f8b3c128670 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f8b38000b80 tx=0x7f8b38010810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:18.002 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:17.987+0000 7f8b42bdf700 1 --2- 172.21.15.42:0/127263589 >> v2:172.21.15.40:3300/0 conn(0x7f8b3c065930 0x7f8b3c128670 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f8b38000b80 tx=0x7f8b38010810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:18.003 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:17.987+0000 7f8b42bdf700 1 --2- 172.21.15.42:0/127263589 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8b3c065930 0x7f8b3c128670 secure :-1 s=READY pgs=2497 cs=0 l=1 rx=0x7f8b38000b80 tx=0x7f8b38010810).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:49:18.003 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:17.987+0000 7f8b42bdf700 1 --2- 172.21.15.42:0/127263589 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8b3c065930 0x7f8b3c128670 secure :-1 s=THROTTLE_DONE pgs=2497 cs=0 l=1 rx=0x7f8b38000b80 tx=0x7f8b38010810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:18.003 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:17.987+0000 7f8b337fe700 1 -- 172.21.15.42:0/127263589 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8b38005c30 con 0x7f8b3c065930 2019-12-05T06:49:18.003 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:17.987+0000 7f8b42bdf700 1 --2- 172.21.15.42:0/127263589 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8b3c065930 0x7f8b3c128670 secure :-1 s=THROTTLE_DONE pgs=2497 cs=0 l=1 rx=0x7f8b38000b80 tx=0x7f8b38010810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:18.003 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:17.987+0000 7f8b337fe700 1 -- 172.21.15.42:0/127263589 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b38005850 con 0x7f8b3c065930 2019-12-05T06:49:18.003 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:17.987+0000 7f8b42bdf700 1 --2- 172.21.15.42:0/127263589 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8b3c065930 0x7f8b3c128670 secure :-1 s=THROTTLE_DONE pgs=2497 cs=0 l=1 rx=0x7f8b38000b80 tx=0x7f8b38010810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:18.003 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:17.987+0000 7f8b337fe700 1 -- 172.21.15.42:0/127263589 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8b380064a0 con 0x7f8b3c065930 2019-12-05T06:49:18.004 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:17.987+0000 7f8b44642700 1 -- 172.21.15.42:0/127263589 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b3c1307e0 con 0x7f8b3c065930 2019-12-05T06:49:18.004 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:17.987+0000 7f8b44642700 1 -- 172.21.15.42:0/127263589 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b3c130d10 con 0x7f8b3c065930 2019-12-05T06:49:18.004 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:17.987+0000 7f8b42bdf700 1 --2- 172.21.15.42:0/127263589 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8b3c065930 0x7f8b3c128670 secure :-1 s=THROTTLE_DONE pgs=2497 cs=0 l=1 rx=0x7f8b38000b80 tx=0x7f8b38010810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:18.004 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:17.991+0000 7f8b337fe700 1 -- 172.21.15.42:0/127263589 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f8b38006780 con 0x7f8b3c065930 2019-12-05T06:49:18.004 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:17.991+0000 7f8b42bdf700 1 --2- 172.21.15.42:0/127263589 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8b3c065930 0x7f8b3c128670 secure :-1 s=THROTTLE_DONE pgs=2497 cs=0 l=1 rx=0x7f8b38000b80 tx=0x7f8b38010810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:18.005 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:17.991+0000 7f8b337fe700 1 --2- 172.21.15.42:0/127263589 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8b2403b1e0 0x7f8b2403d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:49:18.005 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:17.991+0000 7f8b337fe700 1 -- 172.21.15.42:0/127263589 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 12223+0+0 (secure 0 0 0) 0x7f8b3804afb0 con 0x7f8b3c065930 2019-12-05T06:49:18.005 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:17.991+0000 7f8b423de700 1 --2- 172.21.15.42:0/127263589 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8b2403b1e0 0x7f8b2403d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:49:18.005 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:17.991+0000 7f8b423de700 1 --2- 172.21.15.42:0/127263589 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8b2403b1e0 0x7f8b2403d6f0 crc :-1 s=READY pgs=1378 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:49:18.006 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:17.995+0000 7f8b44642700 1 -- 172.21.15.42:0/127263589 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b3c05d110 con 0x7f8b3c065930 2019-12-05T06:49:18.009 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:18.003+0000 7f8b42bdf700 1 --2- 172.21.15.42:0/127263589 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8b3c065930 0x7f8b3c128670 secure :-1 s=THROTTLE_DONE pgs=2497 cs=0 l=1 rx=0x7f8b38000b80 tx=0x7f8b38010810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:18.009 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:18.003+0000 7f8b337fe700 1 -- 172.21.15.42:0/127263589 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f8b38032010 con 0x7f8b3c065930 2019-12-05T06:49:18.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:18.175+0000 7f8b44642700 1 -- 172.21.15.42:0/127263589 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8b3c05b8c0 con 0x7f8b3c065930 2019-12-05T06:49:18.184 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:18.175+0000 7f8b42bdf700 1 --2- 172.21.15.42:0/127263589 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8b3c065930 0x7f8b3c128670 secure :-1 s=THROTTLE_DONE pgs=2497 cs=0 l=1 rx=0x7f8b38000b80 tx=0x7f8b38010810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:18.185 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:18.179+0000 7f8b337fe700 1 -- 172.21.15.42:0/127263589 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 398 v398) v1 ==== 95+0+1776 (secure 0 0 0) 0x7f8b38032270 con 0x7f8b3c065930 2019-12-05T06:49:18.185 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 398 2019-12-05T06:49:18.185 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:49:18.185 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":398,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":398,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:49:10.408613+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":121,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12687},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12687":{"gid":12687,"name":"b","rank":0,"incarnation":395,"state":"up:active","state_seq":12,"addr":"172.21.15.40:6833/815316041","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":815316041},{"type":"v1","addr":"172.21.15.40:6833","nonce":815316041}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:49:18.187 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:18.179+0000 7f8b44642700 1 -- 172.21.15.42:0/127263589 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8b2403b1e0 msgr2=0x7f8b2403d6f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:49:18.187 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:18.179+0000 7f8b44642700 1 --2- 172.21.15.42:0/127263589 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8b2403b1e0 0x7f8b2403d6f0 crc :-1 s=READY pgs=1378 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:49:18.187 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:18.179+0000 7f8b44642700 1 -- 172.21.15.42:0/127263589 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8b3c065930 msgr2=0x7f8b3c128670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:49:18.188 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:18.179+0000 7f8b44642700 1 --2- 172.21.15.42:0/127263589 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8b3c065930 0x7f8b3c128670 secure :-1 s=READY pgs=2497 cs=0 l=1 rx=0x7f8b38000b80 tx=0x7f8b38010810).stop 2019-12-05T06:49:18.188 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:18.179+0000 7f8b44642700 1 -- 172.21.15.42:0/127263589 shutdown_connections 2019-12-05T06:49:18.188 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:18.179+0000 7f8b44642700 1 --2- 172.21.15.42:0/127263589 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8b2403b1e0 0x7f8b2403d6f0 unknown :-1 s=CLOSED pgs=1378 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:49:18.188 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:18.179+0000 7f8b44642700 1 --2- 172.21.15.42:0/127263589 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8b3c065930 0x7f8b3c128670 unknown :-1 s=CLOSED pgs=2497 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:49:18.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:18.179+0000 7f8b44642700 1 -- 172.21.15.42:0/127263589 shutdown_connections 2019-12-05T06:49:18.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:18.179+0000 7f8b44642700 1 -- 172.21.15.42:0/127263589 wait complete. 2019-12-05T06:49:18.189 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:18.179+0000 7f8b44642700 1 -- 172.21.15.42:0/127263589 >> 172.21.15.42:0/127263589 conn(0x7f8b3c08fde0 msgr2=0x7f8b3c0a7f90 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:49:18.198 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2019-12-05T06:49:20.199 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:49:20.305 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:20.291+0000 7f960d3ee700 1 Processor -- start 2019-12-05T06:49:20.305 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:20.291+0000 7f960d3ee700 1 -- start start 2019-12-05T06:49:20.305 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:20.295+0000 7f960d3ee700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f960808df80 0x7f960809f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:49:20.306 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:20.295+0000 7f960d3ee700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f960809f830 0x7f960814efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:49:20.306 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:20.295+0000 7f960d3ee700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f960809fc60 con 0x7f96080a7740 2019-12-05T06:49:20.306 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:20.295+0000 7f960d3ee700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96080a0b00 con 0x7f960809f830 2019-12-05T06:49:20.306 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:20.295+0000 7f960d3ee700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f960808e5b0 con 0x7f960808df80 2019-12-05T06:49:20.306 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:20.295+0000 7f96077fe700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f96080a7740 0x7f960807e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:37864/0 (socket says 172.21.15.42:37864) 2019-12-05T06:49:20.306 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:20.295+0000 7f96077fe700 1 -- 172.21.15.42:0/2004328849 learned_addr learned my addr 172.21.15.42:0/2004328849 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:49:20.306 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:20.295+0000 7f95feffd700 1 --2- 172.21.15.42:0/2004328849 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f960808df80 0x7f960809f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:49:20.307 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:20.295+0000 7f9607fff700 1 --2- 172.21.15.42:0/2004328849 >> v2:172.21.15.40:3300/0 conn(0x7f960809f830 0x7f960814efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:49:20.307 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:20.295+0000 7f9606ffd700 1 -- 172.21.15.42:0/2004328849 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2270637957 0 0) 0x7f960809fc60 con 0x7f96080a7740 2019-12-05T06:49:20.307 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:20.295+0000 7f9606ffd700 1 -- 172.21.15.42:0/2004328849 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95f40030b0 con 0x7f96080a7740 2019-12-05T06:49:20.307 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:20.295+0000 7f9607fff700 1 -- 172.21.15.42:0/2004328849 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f960808df80 msgr2=0x7f960809f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:49:20.307 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:20.295+0000 7f9607fff700 1 --2- 172.21.15.42:0/2004328849 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f960808df80 0x7f960809f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:49:20.307 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:20.295+0000 7f9607fff700 1 -- 172.21.15.42:0/2004328849 >> v1:172.21.15.42:6789/0 conn(0x7f96080a7740 legacy=0x7f960807e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:49:20.307 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:20.295+0000 7f9607fff700 1 -- 172.21.15.42:0/2004328849 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96080a0710 con 0x7f960809f830 2019-12-05T06:49:20.308 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:20.295+0000 7f9607fff700 1 --2- 172.21.15.42:0/2004328849 >> v2:172.21.15.40:3300/0 conn(0x7f960809f830 0x7f960814efe0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f9600008b10 tx=0x7f960000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:20.308 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:20.295+0000 7f9607fff700 1 --2- 172.21.15.42:0/2004328849 >> v2:172.21.15.40:3300/0 conn(0x7f960809f830 0x7f960814efe0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f9600008b10 tx=0x7f960000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:20.308 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:20.295+0000 7f9607fff700 1 --2- 172.21.15.42:0/2004328849 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f960809f830 0x7f960814efe0 secure :-1 s=READY pgs=2498 cs=0 l=1 rx=0x7f9600008b10 tx=0x7f960000fff0).ready entity=mon.0 client_cookie=853c6966ed6880fc server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:49:20.308 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:20.295+0000 7f9607fff700 1 --2- 172.21.15.42:0/2004328849 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f960809f830 0x7f960814efe0 secure :-1 s=THROTTLE_DONE pgs=2498 cs=0 l=1 rx=0x7f9600008b10 tx=0x7f960000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:20.308 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:20.295+0000 7f9606ffd700 1 -- 172.21.15.42:0/2004328849 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f96000109f0 con 0x7f960809f830 2019-12-05T06:49:20.308 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:20.295+0000 7f9607fff700 1 --2- 172.21.15.42:0/2004328849 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f960809f830 0x7f960814efe0 secure :-1 s=THROTTLE_DONE pgs=2498 cs=0 l=1 rx=0x7f9600008b10 tx=0x7f960000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:20.309 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:20.295+0000 7f9606ffd700 1 -- 172.21.15.42:0/2004328849 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9600008b50 con 0x7f960809f830 2019-12-05T06:49:20.309 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:20.295+0000 7f9607fff700 1 --2- 172.21.15.42:0/2004328849 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f960809f830 0x7f960814efe0 secure :-1 s=THROTTLE_DONE pgs=2498 cs=0 l=1 rx=0x7f9600008b10 tx=0x7f960000fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:20.309 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:20.295+0000 7f9606ffd700 1 -- 172.21.15.42:0/2004328849 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f96000110c0 con 0x7f960809f830 2019-12-05T06:49:20.309 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:20.295+0000 7f960d3ee700 1 -- 172.21.15.42:0/2004328849 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f960809f830 msgr2=0x7f960814efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:49:20.309 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:20.295+0000 7f960d3ee700 1 --2- 172.21.15.42:0/2004328849 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f960809f830 0x7f960814efe0 secure :-1 s=READY pgs=2498 cs=0 l=1 rx=0x7f9600008b10 tx=0x7f960000fff0).stop 2019-12-05T06:49:20.309 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:20.299+0000 7f960d3ee700 1 -- 172.21.15.42:0/2004328849 shutdown_connections 2019-12-05T06:49:20.309 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:20.299+0000 7f960d3ee700 1 --2- 172.21.15.42:0/2004328849 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f960809f830 0x7f960814efe0 unknown :-1 s=CLOSED pgs=2498 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:49:20.310 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:20.299+0000 7f960d3ee700 1 --2- 172.21.15.42:0/2004328849 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f960808df80 0x7f960809f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:49:20.310 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:20.299+0000 7f960d3ee700 1 -- 172.21.15.42:0/2004328849 shutdown_connections 2019-12-05T06:49:20.310 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:20.299+0000 7f960d3ee700 1 -- 172.21.15.42:0/2004328849 wait complete. 2019-12-05T06:49:20.310 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:20.299+0000 7f960d3ee700 1 -- 172.21.15.42:0/2004328849 >> 172.21.15.42:0/2004328849 conn(0x7f96080aabe0 msgr2=0x7f96080a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:49:20.313 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:20.299+0000 7f960d3ee700 1 Processor -- start 2019-12-05T06:49:20.313 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:20.299+0000 7f960d3ee700 1 -- start start 2019-12-05T06:49:20.313 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:20.303+0000 7f960d3ee700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f960811b910 0x7f960813c120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:49:20.314 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:20.303+0000 7f960d3ee700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f960807e090 con 0x7f960811b4e0 2019-12-05T06:49:20.314 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:20.303+0000 7f960d3ee700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96081236a0 con 0x7f9608123170 2019-12-05T06:49:20.314 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:20.303+0000 7f960d3ee700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9608122c30 con 0x7f960811b910 2019-12-05T06:49:20.314 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:20.303+0000 7f96077fe700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f9608123170 0x7f960812ced0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:37870/0 (socket says 172.21.15.42:37870) 2019-12-05T06:49:20.314 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:20.303+0000 7f9607fff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f960811b910 0x7f960813c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:49:20.314 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:20.303+0000 7f96077fe700 1 -- 172.21.15.42:0/332796815 learned_addr learned my addr 172.21.15.42:0/332796815 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:49:20.314 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:20.303+0000 7f9607fff700 1 -- 172.21.15.42:0/332796815 >> v1:172.21.15.42:6789/0 conn(0x7f9608123170 legacy=0x7f960812ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:49:20.315 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:20.303+0000 7f9607fff700 1 -- 172.21.15.42:0/332796815 >> v1:172.21.15.40:6789/0 conn(0x7f960811b4e0 legacy=0x7f9608138900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:49:20.315 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:20.303+0000 7f9604ff9700 1 -- 172.21.15.42:0/332796815 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 871855382 0 0) 0x7f96081236a0 con 0x7f9608123170 2019-12-05T06:49:20.315 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:20.303+0000 7f9607fff700 1 -- 172.21.15.42:0/332796815 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f960811a100 con 0x7f960811b910 2019-12-05T06:49:20.315 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:20.303+0000 7f9607fff700 1 --2- 172.21.15.42:0/332796815 >> v2:172.21.15.42:3300/0 conn(0x7f960811b910 0x7f960813c120 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f95f0003070 tx=0x7f95f0007720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:20.315 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:20.303+0000 7f9607fff700 1 --2- 172.21.15.42:0/332796815 >> v2:172.21.15.42:3300/0 conn(0x7f960811b910 0x7f960813c120 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f95f0003070 tx=0x7f95f0007720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:20.315 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:20.303+0000 7f9607fff700 1 --2- 172.21.15.42:0/332796815 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f960811b910 0x7f960813c120 secure :-1 s=READY pgs=3293 cs=0 l=1 rx=0x7f95f0003070 tx=0x7f95f0007720).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:49:20.315 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:20.303+0000 7f9607fff700 1 --2- 172.21.15.42:0/332796815 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f960811b910 0x7f960813c120 secure :-1 s=THROTTLE_DONE pgs=3293 cs=0 l=1 rx=0x7f95f0003070 tx=0x7f95f0007720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:20.316 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:20.303+0000 7f9604ff9700 1 -- 172.21.15.42:0/332796815 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f96000058c0 con 0x7f960811b910 2019-12-05T06:49:20.316 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:20.303+0000 7f9607fff700 1 --2- 172.21.15.42:0/332796815 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f960811b910 0x7f960813c120 secure :-1 s=THROTTLE_DONE pgs=3293 cs=0 l=1 rx=0x7f95f0003070 tx=0x7f95f0007720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:20.316 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:20.303+0000 7f9607fff700 1 --2- 172.21.15.42:0/332796815 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f960811b910 0x7f960813c120 secure :-1 s=THROTTLE_DONE pgs=3293 cs=0 l=1 rx=0x7f95f0003070 tx=0x7f95f0007720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:20.316 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:20.303+0000 7f9604ff9700 1 -- 172.21.15.42:0/332796815 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9600010bf0 con 0x7f960811b910 2019-12-05T06:49:20.316 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:20.303+0000 7f960d3ee700 1 -- 172.21.15.42:0/332796815 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f960813c670 con 0x7f960811b910 2019-12-05T06:49:20.317 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:20.303+0000 7f9604ff9700 1 -- 172.21.15.42:0/332796815 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f960000f280 con 0x7f960811b910 2019-12-05T06:49:20.317 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:20.303+0000 7f960d3ee700 1 -- 172.21.15.42:0/332796815 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f960813ca20 con 0x7f960811b910 2019-12-05T06:49:20.317 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:20.303+0000 7f9607fff700 1 --2- 172.21.15.42:0/332796815 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f960811b910 0x7f960813c120 secure :-1 s=THROTTLE_DONE pgs=3293 cs=0 l=1 rx=0x7f95f0003070 tx=0x7f95f0007720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:20.317 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:20.307+0000 7f9604ff9700 1 -- 172.21.15.42:0/332796815 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f960000f520 con 0x7f960811b910 2019-12-05T06:49:20.318 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:20.307+0000 7f9607fff700 1 --2- 172.21.15.42:0/332796815 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f960811b910 0x7f960813c120 secure :-1 s=THROTTLE_DONE pgs=3293 cs=0 l=1 rx=0x7f95f0003070 tx=0x7f95f0007720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:20.318 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:20.307+0000 7f9604ff9700 1 --2- 172.21.15.42:0/332796815 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f95ec03ac20 0x7f95ec03d130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:49:20.318 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:20.307+0000 7f9604ff9700 1 -- 172.21.15.42:0/332796815 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 12223+0+0 (secure 0 0 0) 0x7f960004ad00 con 0x7f960811b910 2019-12-05T06:49:20.318 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:20.307+0000 7f96077fe700 1 --2- 172.21.15.42:0/332796815 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f95ec03ac20 0x7f95ec03d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:49:20.319 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:20.307+0000 7f96077fe700 1 --2- 172.21.15.42:0/332796815 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f95ec03ac20 0x7f95ec03d130 crc :-1 s=READY pgs=1379 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:49:20.319 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:20.307+0000 7f960d3ee700 1 -- 172.21.15.42:0/332796815 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f960805d110 con 0x7f960811b910 2019-12-05T06:49:20.322 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:20.315+0000 7f9607fff700 1 --2- 172.21.15.42:0/332796815 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f960811b910 0x7f960813c120 secure :-1 s=THROTTLE_DONE pgs=3293 cs=0 l=1 rx=0x7f95f0003070 tx=0x7f95f0007720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:20.322 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:20.315+0000 7f9604ff9700 1 -- 172.21.15.42:0/332796815 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f9600031d60 con 0x7f960811b910 2019-12-05T06:49:20.493 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:20.487+0000 7f960d3ee700 1 -- 172.21.15.42:0/332796815 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f960805b8c0 con 0x7f960811b910 2019-12-05T06:49:20.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:20.487+0000 7f9607fff700 1 --2- 172.21.15.42:0/332796815 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f960811b910 0x7f960813c120 secure :-1 s=THROTTLE_DONE pgs=3293 cs=0 l=1 rx=0x7f95f0003070 tx=0x7f95f0007720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:20.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:20.487+0000 7f9604ff9700 1 -- 172.21.15.42:0/332796815 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 398 v398) v1 ==== 95+0+1776 (secure 0 0 0) 0x7f9600031fc0 con 0x7f960811b910 2019-12-05T06:49:20.495 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 398 2019-12-05T06:49:20.495 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:49:20.495 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":398,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":398,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:49:10.408613+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":121,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12687},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12687":{"gid":12687,"name":"b","rank":0,"incarnation":395,"state":"up:active","state_seq":12,"addr":"172.21.15.40:6833/815316041","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":815316041},{"type":"v1","addr":"172.21.15.40:6833","nonce":815316041}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:49:20.499 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:20.491+0000 7f960d3ee700 1 -- 172.21.15.42:0/332796815 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f95ec03ac20 msgr2=0x7f95ec03d130 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:49:20.499 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:20.491+0000 7f960d3ee700 1 --2- 172.21.15.42:0/332796815 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f95ec03ac20 0x7f95ec03d130 crc :-1 s=READY pgs=1379 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:49:20.499 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:20.491+0000 7f960d3ee700 1 -- 172.21.15.42:0/332796815 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f960811b910 msgr2=0x7f960813c120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:49:20.499 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:20.491+0000 7f960d3ee700 1 --2- 172.21.15.42:0/332796815 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f960811b910 0x7f960813c120 secure :-1 s=READY pgs=3293 cs=0 l=1 rx=0x7f95f0003070 tx=0x7f95f0007720).stop 2019-12-05T06:49:20.499 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:20.491+0000 7f960d3ee700 1 -- 172.21.15.42:0/332796815 shutdown_connections 2019-12-05T06:49:20.500 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:20.491+0000 7f960d3ee700 1 --2- 172.21.15.42:0/332796815 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f95ec03ac20 0x7f95ec03d130 unknown :-1 s=CLOSED pgs=1379 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:49:20.500 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:20.491+0000 7f960d3ee700 1 --2- 172.21.15.42:0/332796815 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f960811b910 0x7f960813c120 unknown :-1 s=CLOSED pgs=3293 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:49:20.501 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:20.491+0000 7f960d3ee700 1 -- 172.21.15.42:0/332796815 shutdown_connections 2019-12-05T06:49:20.501 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:20.491+0000 7f960d3ee700 1 -- 172.21.15.42:0/332796815 wait complete. 2019-12-05T06:49:20.502 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:20.491+0000 7f960d3ee700 1 -- 172.21.15.42:0/332796815 >> 172.21.15.42:0/332796815 conn(0x7f96080a96b0 msgr2=0x7f96080a6a40 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:49:20.508 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2019-12-05T06:49:20.615 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:49:20.621 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:49:22.512 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:49:22.637 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:22.611+0000 7fd6138d6700 1 Processor -- start 2019-12-05T06:49:22.638 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:22.611+0000 7fd6138d6700 1 -- start start 2019-12-05T06:49:22.638 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:22.615+0000 7fd6138d6700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd60c0a7740 0x7fd60c08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:49:22.638 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:22.615+0000 7fd6138d6700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd60c07e720 0x7fd60c07eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:49:22.638 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:22.615+0000 7fd6138d6700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd60c09f2e0 0x7fd60c09f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:49:22.638 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:22.615+0000 7fd6138d6700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd60c0a0b00 con 0x7fd60c07e720 2019-12-05T06:49:22.639 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:22.615+0000 7fd6138d6700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd60c08e5b0 con 0x7fd60c0a7740 2019-12-05T06:49:22.639 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:22.615+0000 7fd6138d6700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd60c08e850 con 0x7fd60c09f2e0 2019-12-05T06:49:22.639 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:22.615+0000 7fd611e73700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd60c0a7740 0x7fd60c08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:49:22.639 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:22.615+0000 7fd611e73700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd60c0a7740 0x7fd60c08df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:37844/0 (socket says 172.21.15.42:37844) 2019-12-05T06:49:22.640 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:22.615+0000 7fd611e73700 1 -- 172.21.15.42:0/1851073866 learned_addr learned my addr 172.21.15.42:0/1851073866 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:49:22.640 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:22.615+0000 7fd611672700 1 --2- 172.21.15.42:0/1851073866 >> v2:172.21.15.40:3300/0 conn(0x7fd60c07e720 0x7fd60c07eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:49:22.640 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:22.615+0000 7fd612674700 1 --2- 172.21.15.42:0/1851073866 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd60c09f2e0 0x7fd60c09f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:49:22.640 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:22.615+0000 7fd611e73700 1 -- 172.21.15.42:0/1851073866 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd60c09f2e0 msgr2=0x7fd60c09f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:49:22.640 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:22.615+0000 7fd611e73700 1 --2- 172.21.15.42:0/1851073866 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd60c09f2e0 0x7fd60c09f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:49:22.641 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:22.615+0000 7fd611e73700 1 -- 172.21.15.42:0/1851073866 >> v2:172.21.15.40:3300/0 conn(0x7fd60c07e720 msgr2=0x7fd60c07eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:49:22.641 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:22.615+0000 7fd611e73700 1 --2- 172.21.15.42:0/1851073866 >> v2:172.21.15.40:3300/0 conn(0x7fd60c07e720 0x7fd60c07eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:49:22.641 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:22.615+0000 7fd611e73700 1 -- 172.21.15.42:0/1851073866 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd60c07f0a0 con 0x7fd60c0a7740 2019-12-05T06:49:22.641 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:22.615+0000 7fd611e73700 1 --2- 172.21.15.42:0/1851073866 >> v2:172.21.15.42:3300/0 conn(0x7fd60c0a7740 0x7fd60c08df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fd60c0a0f50 tx=0x7fd60800fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:22.642 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:22.615+0000 7fd611e73700 1 --2- 172.21.15.42:0/1851073866 >> v2:172.21.15.42:3300/0 conn(0x7fd60c0a7740 0x7fd60c08df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fd60c0a0f50 tx=0x7fd60800fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:22.644 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:22.615+0000 7fd611e73700 1 --2- 172.21.15.42:0/1851073866 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd60c0a7740 0x7fd60c08df80 secure :-1 s=READY pgs=3294 cs=0 l=1 rx=0x7fd60c0a0f50 tx=0x7fd60800fdb0).ready entity=mon.1 client_cookie=c419548ca599afc0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:49:22.644 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:22.615+0000 7fd611e73700 1 --2- 172.21.15.42:0/1851073866 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd60c0a7740 0x7fd60c08df80 secure :-1 s=THROTTLE_DONE pgs=3294 cs=0 l=1 rx=0x7fd60c0a0f50 tx=0x7fd60800fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:22.644 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:22.615+0000 7fd610e71700 1 -- 172.21.15.42:0/1851073866 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd608008c90 con 0x7fd60c0a7740 2019-12-05T06:49:22.645 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:22.615+0000 7fd611e73700 1 --2- 172.21.15.42:0/1851073866 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd60c0a7740 0x7fd60c08df80 secure :-1 s=THROTTLE_DONE pgs=3294 cs=0 l=1 rx=0x7fd60c0a0f50 tx=0x7fd60800fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:22.645 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:22.615+0000 7fd611e73700 1 --2- 172.21.15.42:0/1851073866 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd60c0a7740 0x7fd60c08df80 secure :-1 s=THROTTLE_DONE pgs=3294 cs=0 l=1 rx=0x7fd60c0a0f50 tx=0x7fd60800fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:22.645 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:22.615+0000 7fd610e71700 1 -- 172.21.15.42:0/1851073866 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6080088b0 con 0x7fd60c0a7740 2019-12-05T06:49:22.645 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:22.615+0000 7fd610e71700 1 -- 172.21.15.42:0/1851073866 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd608011b50 con 0x7fd60c0a7740 2019-12-05T06:49:22.646 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:22.615+0000 7fd6138d6700 1 -- 172.21.15.42:0/1851073866 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd60c0a7740 msgr2=0x7fd60c08df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:49:22.646 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:22.615+0000 7fd6138d6700 1 --2- 172.21.15.42:0/1851073866 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd60c0a7740 0x7fd60c08df80 secure :-1 s=READY pgs=3294 cs=0 l=1 rx=0x7fd60c0a0f50 tx=0x7fd60800fdb0).stop 2019-12-05T06:49:22.646 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:22.615+0000 7fd6138d6700 1 -- 172.21.15.42:0/1851073866 shutdown_connections 2019-12-05T06:49:22.646 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:22.615+0000 7fd6138d6700 1 --2- 172.21.15.42:0/1851073866 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd60c09f2e0 0x7fd60c09f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:49:22.646 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:22.615+0000 7fd6138d6700 1 --2- 172.21.15.42:0/1851073866 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd60c0a7740 0x7fd60c08df80 unknown :-1 s=CLOSED pgs=3294 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:49:22.646 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:22.615+0000 7fd6138d6700 1 --2- 172.21.15.42:0/1851073866 >> v2:172.21.15.40:3300/0 conn(0x7fd60c07e720 0x7fd60c07eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:49:22.647 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:22.615+0000 7fd6138d6700 1 -- 172.21.15.42:0/1851073866 shutdown_connections 2019-12-05T06:49:22.647 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:22.615+0000 7fd6138d6700 1 -- 172.21.15.42:0/1851073866 wait complete. 2019-12-05T06:49:22.647 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:22.615+0000 7fd6138d6700 1 -- 172.21.15.42:0/1851073866 >> 172.21.15.42:0/1851073866 conn(0x7fd60c0aabe0 msgr2=0x7fd60c0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:49:22.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:22.619+0000 7fd6138d6700 1 Processor -- start 2019-12-05T06:49:22.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:22.619+0000 7fd6138d6700 1 -- start start 2019-12-05T06:49:22.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:22.619+0000 7fd6138d6700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd60c120fc0 0x7fd60c11b590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:49:22.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:22.619+0000 7fd6138d6700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd60c1308c0 0x7fd60c130cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:49:22.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:22.619+0000 7fd6138d6700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd60c11d750 0x7fd60c11db80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:49:22.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:22.619+0000 7fd6138d6700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd60c131240 con 0x7fd60c1308c0 2019-12-05T06:49:22.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:22.619+0000 7fd6138d6700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd60c12cdc0 con 0x7fd60c11d750 2019-12-05T06:49:22.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:22.619+0000 7fd6138d6700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd60c11bea0 con 0x7fd60c120fc0 2019-12-05T06:49:22.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:22.619+0000 7fd612674700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd60c11d750 0x7fd60c11db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:49:22.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:22.619+0000 7fd612674700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd60c11d750 0x7fd60c11db80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:37852/0 (socket says 172.21.15.42:37852) 2019-12-05T06:49:22.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:22.619+0000 7fd612674700 1 -- 172.21.15.42:0/2893221513 learned_addr learned my addr 172.21.15.42:0/2893221513 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:49:22.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:22.619+0000 7fd611e73700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd60c120fc0 0x7fd60c11b590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:49:22.650 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:22.619+0000 7fd611672700 1 --2- 172.21.15.42:0/2893221513 >> v2:172.21.15.40:3300/0 conn(0x7fd60c1308c0 0x7fd60c130cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:49:22.650 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:22.619+0000 7fd612674700 1 -- 172.21.15.42:0/2893221513 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd60c120fc0 msgr2=0x7fd60c11b590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:49:22.650 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:22.623+0000 7fd612674700 1 --2- 172.21.15.42:0/2893221513 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd60c120fc0 0x7fd60c11b590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:49:22.650 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:22.623+0000 7fd612674700 1 -- 172.21.15.42:0/2893221513 >> v2:172.21.15.40:3300/0 conn(0x7fd60c1308c0 msgr2=0x7fd60c130cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:49:22.650 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:22.623+0000 7fd612674700 1 --2- 172.21.15.42:0/2893221513 >> v2:172.21.15.40:3300/0 conn(0x7fd60c1308c0 0x7fd60c130cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:49:22.650 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:22.623+0000 7fd612674700 1 -- 172.21.15.42:0/2893221513 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd60c11bae0 con 0x7fd60c11d750 2019-12-05T06:49:22.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:22.623+0000 7fd611e73700 1 --2- 172.21.15.42:0/2893221513 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd60c120fc0 0x7fd60c11b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:49:22.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:22.623+0000 7fd611672700 1 --2- 172.21.15.42:0/2893221513 >> v2:172.21.15.40:3300/0 conn(0x7fd60c1308c0 0x7fd60c130cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:49:22.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:22.623+0000 7fd612674700 1 --2- 172.21.15.42:0/2893221513 >> v2:172.21.15.42:3300/0 conn(0x7fd60c11d750 0x7fd60c11db80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fd60c11c060 tx=0x7fd604010d80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:22.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:22.623+0000 7fd612674700 1 --2- 172.21.15.42:0/2893221513 >> v2:172.21.15.42:3300/0 conn(0x7fd60c11d750 0x7fd60c11db80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fd60c11c060 tx=0x7fd604010d80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:22.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:22.623+0000 7fd612674700 1 --2- 172.21.15.42:0/2893221513 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd60c11d750 0x7fd60c11db80 secure :-1 s=READY pgs=3295 cs=0 l=1 rx=0x7fd60c11c060 tx=0x7fd604010d80).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:49:22.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:22.623+0000 7fd612674700 1 --2- 172.21.15.42:0/2893221513 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd60c11d750 0x7fd60c11db80 secure :-1 s=THROTTLE_DONE pgs=3295 cs=0 l=1 rx=0x7fd60c11c060 tx=0x7fd604010d80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:22.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:22.623+0000 7fd612674700 1 --2- 172.21.15.42:0/2893221513 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd60c11d750 0x7fd60c11db80 secure :-1 s=THROTTLE_DONE pgs=3295 cs=0 l=1 rx=0x7fd60c11c060 tx=0x7fd604010d80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:22.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:22.623+0000 7fd6027fc700 1 -- 172.21.15.42:0/2893221513 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd604012a70 con 0x7fd60c11d750 2019-12-05T06:49:22.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:22.623+0000 7fd612674700 1 -- 172.21.15.42:0/2893221513 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd60c11d750 msgr2=0x7fd60c11db80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2019-12-05T06:49:22.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:22.623+0000 7fd612674700 1 -- 172.21.15.42:0/2893221513 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd60c11d750 msgr2=0x7fd60c11db80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2019-12-05T06:49:22.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:22.623+0000 7fd612674700 1 --2- 172.21.15.42:0/2893221513 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd60c11d750 0x7fd60c11db80 secure :-1 s=READY pgs=3295 cs=0 l=1 rx=0x7fd60c11c060 tx=0x7fd604010d80).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-12-05T06:49:22.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:22.623+0000 7fd612674700 1 --2- 172.21.15.42:0/2893221513 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd60c11d750 0x7fd60c11db80 secure :-1 s=READY pgs=3295 cs=0 l=1 rx=0x7fd60c11c060 tx=0x7fd604010d80).stop 2019-12-05T06:49:22.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:22.623+0000 7fd6027fc700 1 -- 172.21.15.42:0/2893221513 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd60c11d750 msgr2=0x7fd60c11db80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2019-12-05T06:49:22.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:22.623+0000 7fd6027fc700 1 --2- 172.21.15.42:0/2893221513 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd60c11d750 0x7fd60c11db80 unknown :-1 s=CLOSED pgs=3295 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:49:22.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:22.623+0000 7fd6027fc700 1 --2- 172.21.15.42:0/2893221513 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd5f4003040 0x7fd5f40054e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:49:22.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:22.623+0000 7fd6027fc700 1 --2- 172.21.15.42:0/2893221513 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd5f4005c10 0x7fd5f40080f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:49:22.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:22.623+0000 7fd6027fc700 1 --2- 172.21.15.42:0/2893221513 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd5f4008870 0x7fd5f400acf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:49:22.654 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:22.623+0000 7fd6027fc700 1 -- 172.21.15.42:0/2893221513 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5f400b2f0 con 0x7fd5f4005c10 2019-12-05T06:49:22.654 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:22.623+0000 7fd6027fc700 1 -- 172.21.15.42:0/2893221513 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5f400b4e0 con 0x7fd5f4003040 2019-12-05T06:49:22.654 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:22.623+0000 7fd6027fc700 1 -- 172.21.15.42:0/2893221513 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5f400b720 con 0x7fd5f4008870 2019-12-05T06:49:22.654 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:22.623+0000 7fd611e73700 1 --2- 172.21.15.42:0/2893221513 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd5f4003040 0x7fd5f40054e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:49:22.654 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:22.623+0000 7fd611672700 1 --2- 172.21.15.42:0/2893221513 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd5f4005c10 0x7fd5f40080f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:49:22.654 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:22.623+0000 7fd612674700 1 --2- 172.21.15.42:0/2893221513 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd5f4008870 0x7fd5f400acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:49:22.655 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:22.623+0000 7fd611e73700 1 -- 172.21.15.42:0/2893221513 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd5f4008870 msgr2=0x7fd5f400acf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:49:22.655 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:22.623+0000 7fd611e73700 1 --2- 172.21.15.42:0/2893221513 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd5f4008870 0x7fd5f400acf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:49:22.655 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:22.623+0000 7fd611e73700 1 -- 172.21.15.42:0/2893221513 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd5f4005c10 msgr2=0x7fd5f40080f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:49:22.655 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:22.623+0000 7fd611e73700 1 --2- 172.21.15.42:0/2893221513 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd5f4005c10 0x7fd5f40080f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:49:22.655 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:22.623+0000 7fd611e73700 1 -- 172.21.15.42:0/2893221513 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=2+}) v3 -- 0x7fd5f400ba20 con 0x7fd5f4003040 2019-12-05T06:49:22.655 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:22.623+0000 7fd612674700 1 --2- 172.21.15.42:0/2893221513 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd5f4008870 0x7fd5f400acf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:49:22.656 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:22.623+0000 7fd6138d6700 1 -- 172.21.15.42:0/2893221513 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd60c123ea0 con 0x7fd5f4003040 2019-12-05T06:49:22.656 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:22.623+0000 7fd6138d6700 1 -- 172.21.15.42:0/2893221513 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd60c120fc0 con 0x7fd5f4003040 2019-12-05T06:49:22.656 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:22.623+0000 7fd611e73700 1 --2- 172.21.15.42:0/2893221513 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd5f4003040 0x7fd5f40054e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fd60c0a0f50 tx=0x7fd608008be0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:22.656 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:22.623+0000 7fd611e73700 1 --2- 172.21.15.42:0/2893221513 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd5f4003040 0x7fd5f40054e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fd60c0a0f50 tx=0x7fd608008be0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:22.656 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:22.623+0000 7fd611e73700 1 --2- 172.21.15.42:0/2893221513 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd5f4003040 0x7fd5f40054e0 secure :-1 s=READY pgs=3296 cs=0 l=1 rx=0x7fd60c0a0f50 tx=0x7fd608008be0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:49:22.656 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:22.623+0000 7fd611e73700 1 --2- 172.21.15.42:0/2893221513 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd5f4003040 0x7fd5f40054e0 secure :-1 s=THROTTLE_DONE pgs=3296 cs=0 l=1 rx=0x7fd60c0a0f50 tx=0x7fd608008be0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:22.657 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:22.623+0000 7fd6027fc700 1 -- 172.21.15.42:0/2893221513 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd60800f490 con 0x7fd5f4003040 2019-12-05T06:49:22.657 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:22.623+0000 7fd611e73700 1 --2- 172.21.15.42:0/2893221513 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd5f4003040 0x7fd5f40054e0 secure :-1 s=THROTTLE_DONE pgs=3296 cs=0 l=1 rx=0x7fd60c0a0f50 tx=0x7fd608008be0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:22.657 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:22.623+0000 7fd6027fc700 1 -- 172.21.15.42:0/2893221513 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd608011770 con 0x7fd5f4003040 2019-12-05T06:49:22.657 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:22.623+0000 7fd611e73700 1 --2- 172.21.15.42:0/2893221513 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd5f4003040 0x7fd5f40054e0 secure :-1 s=THROTTLE_DONE pgs=3296 cs=0 l=1 rx=0x7fd60c0a0f50 tx=0x7fd608008be0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:22.657 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:22.627+0000 7fd6027fc700 1 -- 172.21.15.42:0/2893221513 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fd608005540 con 0x7fd5f4003040 2019-12-05T06:49:22.658 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:22.627+0000 7fd611e73700 1 --2- 172.21.15.42:0/2893221513 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd5f4003040 0x7fd5f40054e0 secure :-1 s=THROTTLE_DONE pgs=3296 cs=0 l=1 rx=0x7fd60c0a0f50 tx=0x7fd608008be0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:22.665 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:22.627+0000 7fd6027fc700 1 --2- 172.21.15.42:0/2893221513 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd5f4043a50 0x7fd5f4045f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:49:22.665 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:22.627+0000 7fd6027fc700 1 -- 172.21.15.42:0/2893221513 <== mon.1 v2:172.21.15.42:3300/0 4 ==== osd_map(121..121 src has 1..121) v4 ==== 12223+0+0 (secure 0 0 0) 0x7fd60808ee10 con 0x7fd5f4003040 2019-12-05T06:49:22.666 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:22.627+0000 7fd611672700 1 --2- 172.21.15.42:0/2893221513 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd5f4043a50 0x7fd5f4045f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:49:22.666 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:22.627+0000 7fd611672700 1 --2- 172.21.15.42:0/2893221513 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd5f4043a50 0x7fd5f4045f60 crc :-1 s=READY pgs=1382 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:49:22.666 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:22.627+0000 7fd6138d6700 1 -- 172.21.15.42:0/2893221513 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5e8003200 con 0x7fd5f4003040 2019-12-05T06:49:22.666 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:22.635+0000 7fd611e73700 1 --2- 172.21.15.42:0/2893221513 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd5f4003040 0x7fd5f40054e0 secure :-1 s=THROTTLE_DONE pgs=3296 cs=0 l=1 rx=0x7fd60c0a0f50 tx=0x7fd608008be0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:22.667 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:22.635+0000 7fd6027fc700 1 -- 172.21.15.42:0/2893221513 <== mon.1 v2:172.21.15.42:3300/0 5 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fd6080760b0 con 0x7fd5f4003040 2019-12-05T06:49:22.701 INFO:tasks.workunit.client.0.smithi042.stderr:+ cd ../.. 2019-12-05T06:49:22.704 INFO:tasks.workunit.client.0.smithi042.stderr:+ ver=v0.10 2019-12-05T06:49:22.704 INFO:tasks.workunit.client.0.smithi042.stderr:+ echo checking v0.10 2019-12-05T06:49:22.704 INFO:tasks.workunit.client.0.smithi042.stderr:+ cd .snap/v0.10 2019-12-05T06:49:22.704 INFO:tasks.workunit.client.0.smithi042.stdout:checking v0.10 2019-12-05T06:49:22.706 INFO:tasks.workunit.client.0.smithi042.stderr:+ git diff --exit-code 2019-12-05T06:49:22.824 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:22.815+0000 7fd6138d6700 1 -- 172.21.15.42:0/2893221513 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd5e8003630 con 0x7fd5f4003040 2019-12-05T06:49:22.825 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:22.819+0000 7fd611e73700 1 --2- 172.21.15.42:0/2893221513 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd5f4003040 0x7fd5f40054e0 secure :-1 s=THROTTLE_DONE pgs=3296 cs=0 l=1 rx=0x7fd60c0a0f50 tx=0x7fd608008be0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:49:22.825 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:22.819+0000 7fd6027fc700 1 -- 172.21.15.42:0/2893221513 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 399 v399) v1 ==== 95+0+2113 (secure 0 0 0) 0x7fd60800f690 con 0x7fd5f4003040 2019-12-05T06:49:22.825 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 399 2019-12-05T06:49:22.825 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:49:22.826 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":399,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":8537,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/1881703925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1881703925},{"type":"v1","addr":"172.21.15.42:6835","nonce":1881703925}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":399}],"filesystems":[{"mdsmap":{"epoch":398,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:49:10.408613+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":121,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12687},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12687":{"gid":12687,"name":"b","rank":0,"incarnation":395,"state":"up:active","state_seq":12,"addr":"172.21.15.40:6833/815316041","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":815316041},{"type":"v1","addr":"172.21.15.40:6833","nonce":815316041}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:49:22.828 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:22.819+0000 7fd6138d6700 1 -- 172.21.15.42:0/2893221513 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd5f4043a50 msgr2=0x7fd5f4045f60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:49:22.828 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:22.819+0000 7fd6138d6700 1 --2- 172.21.15.42:0/2893221513 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd5f4043a50 0x7fd5f4045f60 crc :-1 s=READY pgs=1382 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:49:22.828 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:22.819+0000 7fd6138d6700 1 -- 172.21.15.42:0/2893221513 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd5f4003040 msgr2=0x7fd5f40054e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:49:22.828 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:22.819+0000 7fd612674700 1 -- 172.21.15.42:0/2893221513 reap_dead start 2019-12-05T06:49:22.828 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:22.819+0000 7fd6138d6700 1 --2- 172.21.15.42:0/2893221513 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd5f4003040 0x7fd5f40054e0 secure :-1 s=READY pgs=3296 cs=0 l=1 rx=0x7fd60c0a0f50 tx=0x7fd608008be0).stop 2019-12-05T06:49:22.829 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:22.819+0000 7fd6138d6700 1 -- 172.21.15.42:0/2893221513 shutdown_connections 2019-12-05T06:49:22.829 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:22.819+0000 7fd6138d6700 1 --2- 172.21.15.42:0/2893221513 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd5f4003040 0x7fd5f40054e0 unknown :-1 s=CLOSED pgs=3296 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:49:22.829 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:22.819+0000 7fd6138d6700 1 --2- 172.21.15.42:0/2893221513 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd60c11d750 0x7fd60c11db80 unknown :-1 s=CLOSED pgs=3295 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:49:22.829 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:22.819+0000 7fd6138d6700 1 -- 172.21.15.42:0/2893221513 shutdown_connections 2019-12-05T06:49:22.829 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:22.819+0000 7fd6138d6700 1 -- 172.21.15.42:0/2893221513 wait complete. 2019-12-05T06:49:22.829 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:49:22.819+0000 7fd6138d6700 1 -- 172.21.15.42:0/2893221513 >> 172.21.15.42:0/2893221513 conn(0x7fd60c0aabe0 msgr2=0x7fd60c0a9770 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:49:22.837 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a reported in up:standby state 2019-12-05T06:49:22.838 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 38 secs before thrashing 2019-12-05T06:49:29.268 INFO:tasks.workunit.client.0.smithi042.stderr:+ cd ../.. 2019-12-05T06:49:29.269 INFO:tasks.workunit.client.0.smithi042.stderr:+ ver=v0.11 2019-12-05T06:49:29.269 INFO:tasks.workunit.client.0.smithi042.stderr:+ echo checking v0.11 2019-12-05T06:49:29.270 INFO:tasks.workunit.client.0.smithi042.stderr:+ cd .snap/v0.11 2019-12-05T06:49:29.270 INFO:tasks.workunit.client.0.smithi042.stdout:checking v0.11 2019-12-05T06:49:29.273 INFO:tasks.workunit.client.0.smithi042.stderr:+ git diff --exit-code 2019-12-05T06:49:34.449 INFO:tasks.workunit.client.0.smithi042.stderr:+ cd ../.. 2019-12-05T06:49:34.450 INFO:tasks.workunit.client.0.smithi042.stderr:+ ver=v0.12 2019-12-05T06:49:34.450 INFO:tasks.workunit.client.0.smithi042.stderr:+ echo checking v0.12 2019-12-05T06:49:34.450 INFO:tasks.workunit.client.0.smithi042.stderr:+ cd .snap/v0.12 2019-12-05T06:49:34.450 INFO:tasks.workunit.client.0.smithi042.stdout:checking v0.12 2019-12-05T06:49:34.453 INFO:tasks.workunit.client.0.smithi042.stderr:+ git diff --exit-code 2019-12-05T06:49:42.034 INFO:tasks.workunit.client.0.smithi042.stderr:+ cd ../.. 2019-12-05T06:49:42.036 INFO:tasks.workunit.client.0.smithi042.stderr:+ ver=v0.13 2019-12-05T06:49:42.036 INFO:tasks.workunit.client.0.smithi042.stderr:+ echo checking v0.13 2019-12-05T06:49:42.036 INFO:tasks.workunit.client.0.smithi042.stderr:+ cd .snap/v0.13 2019-12-05T06:49:42.036 INFO:tasks.workunit.client.0.smithi042.stdout:checking v0.13 2019-12-05T06:49:42.041 INFO:tasks.workunit.client.0.smithi042.stderr:+ git diff --exit-code 2019-12-05T06:49:48.756 INFO:tasks.workunit.client.0.smithi042.stderr:+ cd ../.. 2019-12-05T06:49:48.758 INFO:tasks.workunit.client.0.smithi042.stderr:+ ver=v0.14 2019-12-05T06:49:48.758 INFO:tasks.workunit.client.0.smithi042.stderr:+ echo checking v0.14 2019-12-05T06:49:48.758 INFO:tasks.workunit.client.0.smithi042.stderr:+ cd .snap/v0.14 2019-12-05T06:49:48.768 INFO:tasks.workunit.client.0.smithi042.stderr:+ git diff --exit-code 2019-12-05T06:49:48.773 INFO:tasks.workunit.client.0.smithi042.stdout:checking v0.14 2019-12-05T06:49:50.688 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:49:50.695 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:49:53.441 INFO:tasks.workunit.client.0.smithi042.stderr:+ cd ../.. 2019-12-05T06:49:53.442 INFO:tasks.workunit.client.0.smithi042.stderr:+ ver=v0.15 2019-12-05T06:49:53.442 INFO:tasks.workunit.client.0.smithi042.stderr:+ echo checking v0.15 2019-12-05T06:49:53.442 INFO:tasks.workunit.client.0.smithi042.stderr:+ cd .snap/v0.15 2019-12-05T06:49:53.442 INFO:tasks.workunit.client.0.smithi042.stdout:checking v0.15 2019-12-05T06:49:53.446 INFO:tasks.workunit.client.0.smithi042.stderr:+ git diff --exit-code 2019-12-05T06:49:58.753 INFO:tasks.workunit.client.0.smithi042.stderr:+ cd ../.. 2019-12-05T06:49:58.754 INFO:tasks.workunit.client.0.smithi042.stderr:+ ver=v0.16 2019-12-05T06:49:58.754 INFO:tasks.workunit.client.0.smithi042.stderr:+ echo checking v0.16 2019-12-05T06:49:58.754 INFO:tasks.workunit.client.0.smithi042.stderr:+ cd .snap/v0.16 2019-12-05T06:49:58.755 INFO:tasks.workunit.client.0.smithi042.stdout:checking v0.16 2019-12-05T06:49:58.758 INFO:tasks.workunit.client.0.smithi042.stderr:+ git diff --exit-code 2019-12-05T06:50:00.839 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:50:01.029 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.003+0000 7f94efca9700 1 Processor -- start 2019-12-05T06:50:01.029 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.003+0000 7f94efca9700 1 -- start start 2019-12-05T06:50:01.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.003+0000 7f94efca9700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f94e808df80 0x7f94e809f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:50:01.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.003+0000 7f94efca9700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f94e809f830 0x7f94e814efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:50:01.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.003+0000 7f94efca9700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94e809fc60 con 0x7f94e80a7740 2019-12-05T06:50:01.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.003+0000 7f94efca9700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94e80a0b00 con 0x7f94e808df80 2019-12-05T06:50:01.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.003+0000 7f94efca9700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94e808e5b0 con 0x7f94e809f830 2019-12-05T06:50:01.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.003+0000 7f94eda45700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f94e808df80 0x7f94e809f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:01.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.003+0000 7f94eda45700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f94e808df80 0x7f94e809f2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:38198/0 (socket says 172.21.15.42:38198) 2019-12-05T06:50:01.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.003+0000 7f94eda45700 1 -- 172.21.15.42:0/1293080389 learned_addr learned my addr 172.21.15.42:0/1293080389 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:50:01.033 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.003+0000 7f94eea47700 1 --2- 172.21.15.42:0/1293080389 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f94e809f830 0x7f94e814efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:01.033 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.003+0000 7f94eda45700 1 -- 172.21.15.42:0/1293080389 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f94e809f830 msgr2=0x7f94e814efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:50:01.033 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.003+0000 7f94eda45700 1 --2- 172.21.15.42:0/1293080389 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f94e809f830 0x7f94e814efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:50:01.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.003+0000 7f94eda45700 1 -- 172.21.15.42:0/1293080389 >> v1:172.21.15.40:6789/0 conn(0x7f94e80a7740 legacy=0x7f94e807e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:01.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.003+0000 7f94eda45700 1 -- 172.21.15.42:0/1293080389 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94e80a0710 con 0x7f94e808df80 2019-12-05T06:50:01.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.003+0000 7f94eda45700 1 --2- 172.21.15.42:0/1293080389 >> v2:172.21.15.42:3300/0 conn(0x7f94e808df80 0x7f94e809f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f94d8008790 tx=0x7f94d800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:01.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.003+0000 7f94eda45700 1 --2- 172.21.15.42:0/1293080389 >> v2:172.21.15.42:3300/0 conn(0x7f94e808df80 0x7f94e809f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f94d8008790 tx=0x7f94d800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:01.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.003+0000 7f94eda45700 1 --2- 172.21.15.42:0/1293080389 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f94e808df80 0x7f94e809f2e0 secure :-1 s=READY pgs=3298 cs=0 l=1 rx=0x7f94d8008790 tx=0x7f94d800fd30).ready entity=mon.1 client_cookie=b0afb41ffbf933ad server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:50:01.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.003+0000 7f94eda45700 1 --2- 172.21.15.42:0/1293080389 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f94e808df80 0x7f94e809f2e0 secure :-1 s=THROTTLE_DONE pgs=3298 cs=0 l=1 rx=0x7f94d8008790 tx=0x7f94d800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:01.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.003+0000 7f94ed244700 1 -- 172.21.15.42:0/1293080389 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f94d8008c90 con 0x7f94e808df80 2019-12-05T06:50:01.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.003+0000 7f94eda45700 1 --2- 172.21.15.42:0/1293080389 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f94e808df80 0x7f94e809f2e0 secure :-1 s=THROTTLE_DONE pgs=3298 cs=0 l=1 rx=0x7f94d8008790 tx=0x7f94d800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:01.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.003+0000 7f94eda45700 1 --2- 172.21.15.42:0/1293080389 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f94e808df80 0x7f94e809f2e0 secure :-1 s=THROTTLE_DONE pgs=3298 cs=0 l=1 rx=0x7f94d8008790 tx=0x7f94d800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:01.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.003+0000 7f94ed244700 1 -- 172.21.15.42:0/1293080389 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94d80088b0 con 0x7f94e808df80 2019-12-05T06:50:01.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.003+0000 7f94ed244700 1 -- 172.21.15.42:0/1293080389 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f94d8011b10 con 0x7f94e808df80 2019-12-05T06:50:01.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.003+0000 7f94efca9700 1 -- 172.21.15.42:0/1293080389 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f94e808df80 msgr2=0x7f94e809f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:01.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.003+0000 7f94efca9700 1 --2- 172.21.15.42:0/1293080389 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f94e808df80 0x7f94e809f2e0 secure :-1 s=READY pgs=3298 cs=0 l=1 rx=0x7f94d8008790 tx=0x7f94d800fd30).stop 2019-12-05T06:50:01.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.003+0000 7f94efca9700 1 -- 172.21.15.42:0/1293080389 shutdown_connections 2019-12-05T06:50:01.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.003+0000 7f94efca9700 1 --2- 172.21.15.42:0/1293080389 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f94e809f830 0x7f94e814efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:50:01.039 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.003+0000 7f94efca9700 1 --2- 172.21.15.42:0/1293080389 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f94e808df80 0x7f94e809f2e0 unknown :-1 s=CLOSED pgs=3298 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:01.039 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.003+0000 7f94efca9700 1 -- 172.21.15.42:0/1293080389 shutdown_connections 2019-12-05T06:50:01.039 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.003+0000 7f94efca9700 1 -- 172.21.15.42:0/1293080389 wait complete. 2019-12-05T06:50:01.040 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.003+0000 7f94efca9700 1 -- 172.21.15.42:0/1293080389 >> 172.21.15.42:0/1293080389 conn(0x7f94e80aabe0 msgr2=0x7f94e80a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:50:01.040 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.007+0000 7f94efca9700 1 Processor -- start 2019-12-05T06:50:01.040 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.007+0000 7f94efca9700 1 -- start start 2019-12-05T06:50:01.041 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.007+0000 7f94efca9700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f94e807ad90 0x7f94e812fba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:50:01.041 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.007+0000 7f94efca9700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f94e81300f0 0x7f94e8136d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:50:01.041 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.007+0000 7f94efca9700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94e811b890 con 0x7f94e81179d0 2019-12-05T06:50:01.041 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.007+0000 7f94efca9700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94e81161c0 con 0x7f94e81300f0 2019-12-05T06:50:01.042 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.007+0000 7f94efca9700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94e8116420 con 0x7f94e807ad90 2019-12-05T06:50:01.042 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.007+0000 7f94ee246700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f94e81179d0 0x7f94e8078770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:38240/0 (socket says 172.21.15.42:38240) 2019-12-05T06:50:01.042 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.007+0000 7f94ee246700 1 -- 172.21.15.42:0/4015821205 learned_addr learned my addr 172.21.15.42:0/4015821205 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:50:01.042 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.007+0000 7f94eda45700 1 --2- 172.21.15.42:0/4015821205 >> v2:172.21.15.42:3300/0 conn(0x7f94e807ad90 0x7f94e812fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:01.043 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.007+0000 7f94eea47700 1 --2- 172.21.15.42:0/4015821205 >> v2:172.21.15.40:3300/0 conn(0x7f94e81300f0 0x7f94e8136d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:01.043 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.007+0000 7f94deffd700 1 -- 172.21.15.42:0/4015821205 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 39800555 0 0) 0x7f94e811b890 con 0x7f94e81179d0 2019-12-05T06:50:01.043 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.007+0000 7f94deffd700 1 -- 172.21.15.42:0/4015821205 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94d00030b0 con 0x7f94e81179d0 2019-12-05T06:50:01.043 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.007+0000 7f94eda45700 1 -- 172.21.15.42:0/4015821205 >> v2:172.21.15.40:3300/0 conn(0x7f94e81300f0 msgr2=0x7f94e8136d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:01.044 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.007+0000 7f94eda45700 1 --2- 172.21.15.42:0/4015821205 >> v2:172.21.15.40:3300/0 conn(0x7f94e81300f0 0x7f94e8136d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:01.044 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.007+0000 7f94eda45700 1 -- 172.21.15.42:0/4015821205 >> v1:172.21.15.42:6789/0 conn(0x7f94e81179d0 legacy=0x7f94e8078770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:01.044 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.007+0000 7f94eda45700 1 -- 172.21.15.42:0/4015821205 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94e807c4a0 con 0x7f94e807ad90 2019-12-05T06:50:01.045 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.007+0000 7f94eda45700 1 --2- 172.21.15.42:0/4015821205 >> v2:172.21.15.42:3300/0 conn(0x7f94e807ad90 0x7f94e812fba0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f94e81171b0 tx=0x7f94d8008b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:01.045 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.007+0000 7f94eda45700 1 --2- 172.21.15.42:0/4015821205 >> v2:172.21.15.42:3300/0 conn(0x7f94e807ad90 0x7f94e812fba0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f94e81171b0 tx=0x7f94d8008b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:01.046 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.007+0000 7f94eda45700 1 --2- 172.21.15.42:0/4015821205 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f94e807ad90 0x7f94e812fba0 secure :-1 s=READY pgs=3300 cs=0 l=1 rx=0x7f94e81171b0 tx=0x7f94d8008b60).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:50:01.046 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.007+0000 7f94eda45700 1 --2- 172.21.15.42:0/4015821205 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f94e807ad90 0x7f94e812fba0 secure :-1 s=THROTTLE_DONE pgs=3300 cs=0 l=1 rx=0x7f94e81171b0 tx=0x7f94d8008b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:01.046 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.007+0000 7f94deffd700 1 -- 172.21.15.42:0/4015821205 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f94d800f470 con 0x7f94e807ad90 2019-12-05T06:50:01.047 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.007+0000 7f94eda45700 1 --2- 172.21.15.42:0/4015821205 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f94e807ad90 0x7f94e812fba0 secure :-1 s=THROTTLE_DONE pgs=3300 cs=0 l=1 rx=0x7f94e81171b0 tx=0x7f94d8008b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:01.047 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.007+0000 7f94efca9700 1 -- 172.21.15.42:0/4015821205 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94e8078f70 con 0x7f94e807ad90 2019-12-05T06:50:01.047 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.007+0000 7f94deffd700 1 -- 172.21.15.42:0/4015821205 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94d8011730 con 0x7f94e807ad90 2019-12-05T06:50:01.047 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.007+0000 7f94eda45700 1 --2- 172.21.15.42:0/4015821205 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f94e807ad90 0x7f94e812fba0 secure :-1 s=THROTTLE_DONE pgs=3300 cs=0 l=1 rx=0x7f94e81171b0 tx=0x7f94d8008b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:01.047 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.007+0000 7f94efca9700 1 -- 172.21.15.42:0/4015821205 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94e8130520 con 0x7f94e807ad90 2019-12-05T06:50:01.048 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.007+0000 7f94deffd700 1 -- 172.21.15.42:0/4015821205 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f94d8005730 con 0x7f94e807ad90 2019-12-05T06:50:01.048 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.011+0000 7f94eda45700 1 --2- 172.21.15.42:0/4015821205 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f94e807ad90 0x7f94e812fba0 secure :-1 s=THROTTLE_DONE pgs=3300 cs=0 l=1 rx=0x7f94e81171b0 tx=0x7f94d8008b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:01.049 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.011+0000 7f94deffd700 1 -- 172.21.15.42:0/4015821205 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f94d80059d0 con 0x7f94e807ad90 2019-12-05T06:50:01.049 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.011+0000 7f94eda45700 1 --2- 172.21.15.42:0/4015821205 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f94e807ad90 0x7f94e812fba0 secure :-1 s=THROTTLE_DONE pgs=3300 cs=0 l=1 rx=0x7f94e81171b0 tx=0x7f94d8008b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:01.050 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.011+0000 7f94deffd700 1 --2- 172.21.15.42:0/4015821205 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f94d003b270 0x7f94d003d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:50:01.050 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.011+0000 7f94deffd700 1 -- 172.21.15.42:0/4015821205 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 12223+0+0 (secure 0 0 0) 0x7f94d804b780 con 0x7f94e807ad90 2019-12-05T06:50:01.050 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.011+0000 7f94ee246700 1 --2- 172.21.15.42:0/4015821205 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f94d003b270 0x7f94d003d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:01.051 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.011+0000 7f94ee246700 1 --2- 172.21.15.42:0/4015821205 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f94d003b270 0x7f94d003d780 crc :-1 s=READY pgs=1384 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:50:01.051 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.019+0000 7f94efca9700 1 -- 172.21.15.42:0/4015821205 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94e805d110 con 0x7f94e807ad90 2019-12-05T06:50:01.052 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.023+0000 7f94eda45700 1 --2- 172.21.15.42:0/4015821205 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f94e807ad90 0x7f94e812fba0 secure :-1 s=THROTTLE_DONE pgs=3300 cs=0 l=1 rx=0x7f94e81171b0 tx=0x7f94d8008b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:01.052 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.023+0000 7f94deffd700 1 -- 172.21.15.42:0/4015821205 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f94d8008de0 con 0x7f94e807ad90 2019-12-05T06:50:01.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.183+0000 7f94efca9700 1 -- 172.21.15.42:0/4015821205 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f94e8142080 con 0x7f94e807ad90 2019-12-05T06:50:01.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.183+0000 7f94eda45700 1 --2- 172.21.15.42:0/4015821205 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f94e807ad90 0x7f94e812fba0 secure :-1 s=THROTTLE_DONE pgs=3300 cs=0 l=1 rx=0x7f94e81171b0 tx=0x7f94d8008b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:01.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.183+0000 7f94deffd700 1 -- 172.21.15.42:0/4015821205 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 399 v399) v1 ==== 95+0+2113 (secure 0 0 0) 0x7f94d8013d80 con 0x7f94e807ad90 2019-12-05T06:50:01.191 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 399 2019-12-05T06:50:01.191 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:50:01.191 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":399,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":8537,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/1881703925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1881703925},{"type":"v1","addr":"172.21.15.42:6835","nonce":1881703925}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":399}],"filesystems":[{"mdsmap":{"epoch":398,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:49:10.408613+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":121,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12687},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12687":{"gid":12687,"name":"b","rank":0,"incarnation":395,"state":"up:active","state_seq":12,"addr":"172.21.15.40:6833/815316041","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":815316041},{"type":"v1","addr":"172.21.15.40:6833","nonce":815316041}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:50:01.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.187+0000 7f94efca9700 1 -- 172.21.15.42:0/4015821205 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f94d003b270 msgr2=0x7f94d003d780 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:01.193 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.187+0000 7f94efca9700 1 --2- 172.21.15.42:0/4015821205 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f94d003b270 0x7f94d003d780 crc :-1 s=READY pgs=1384 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:01.193 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.187+0000 7f94efca9700 1 -- 172.21.15.42:0/4015821205 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f94e807ad90 msgr2=0x7f94e812fba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:01.193 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.187+0000 7f94efca9700 1 --2- 172.21.15.42:0/4015821205 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f94e807ad90 0x7f94e812fba0 secure :-1 s=READY pgs=3300 cs=0 l=1 rx=0x7f94e81171b0 tx=0x7f94d8008b60).stop 2019-12-05T06:50:01.194 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.187+0000 7f94efca9700 1 -- 172.21.15.42:0/4015821205 shutdown_connections 2019-12-05T06:50:01.195 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.187+0000 7f94efca9700 1 --2- 172.21.15.42:0/4015821205 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f94d003b270 0x7f94d003d780 unknown :-1 s=CLOSED pgs=1384 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:01.195 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.187+0000 7f94efca9700 1 --2- 172.21.15.42:0/4015821205 >> v2:172.21.15.40:3300/0 conn(0x7f94e81300f0 0x7f94e8136d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:01.195 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.187+0000 7f94efca9700 1 --2- 172.21.15.42:0/4015821205 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f94e807ad90 0x7f94e812fba0 unknown :-1 s=CLOSED pgs=3300 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:01.195 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.187+0000 7f94efca9700 1 -- 172.21.15.42:0/4015821205 shutdown_connections 2019-12-05T06:50:01.196 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.187+0000 7f94efca9700 1 -- 172.21.15.42:0/4015821205 wait complete. 2019-12-05T06:50:01.196 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.187+0000 7f94efca9700 1 -- 172.21.15.42:0/4015821205 >> 172.21.15.42:0/4015821205 conn(0x7f94e807dff0 msgr2=0x7f94e807cb40 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:50:01.203 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.b (rank=0) 2019-12-05T06:50:01.203 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2019-12-05T06:50:01.204 INFO:teuthology.orchestra.run:waiting for 300 2019-12-05T06:50:01.217 INFO:tasks.ceph.mds.b:Stopped 2019-12-05T06:50:01.217 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:50:01.320 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.307+0000 7f75d3cf1700 1 Processor -- start 2019-12-05T06:50:01.320 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.307+0000 7f75d3cf1700 1 -- start start 2019-12-05T06:50:01.320 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.307+0000 7f75d3cf1700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f75cc0a7740 0x7f75cc08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:50:01.320 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.307+0000 7f75d3cf1700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75cc0a0710 con 0x7f75cc07e720 2019-12-05T06:50:01.320 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.307+0000 7f75d3cf1700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75cc0a9ba0 con 0x7f75cc09fae0 2019-12-05T06:50:01.321 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.307+0000 7f75d3cf1700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75cc0a0b00 con 0x7f75cc0a7740 2019-12-05T06:50:01.321 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.307+0000 7f75d2a8f700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f75cc09fae0 0x7f75cc14efe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:38254/0 (socket says 172.21.15.42:38254) 2019-12-05T06:50:01.321 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.307+0000 7f75d2a8f700 1 -- 172.21.15.42:0/2200923493 learned_addr learned my addr 172.21.15.42:0/2200923493 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:50:01.321 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.307+0000 7f75d228e700 1 --2- 172.21.15.42:0/2200923493 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f75cc0a7740 0x7f75cc08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:01.321 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.307+0000 7f75d128c700 1 -- 172.21.15.42:0/2200923493 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2626303946 0 0) 0x7f75cc0a9ba0 con 0x7f75cc09fae0 2019-12-05T06:50:01.321 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.307+0000 7f75d128c700 1 -- 172.21.15.42:0/2200923493 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75b80030b0 con 0x7f75cc09fae0 2019-12-05T06:50:01.322 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.307+0000 7f75d128c700 1 -- 172.21.15.42:0/2200923493 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3740711236 0 0) 0x7f75cc0a0710 con 0x7f75cc07e720 2019-12-05T06:50:01.322 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.307+0000 7f75d128c700 1 -- 172.21.15.42:0/2200923493 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75cc0a9ba0 con 0x7f75cc07e720 2019-12-05T06:50:01.322 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.307+0000 7f75d228e700 1 -- 172.21.15.42:0/2200923493 >> v1:172.21.15.42:6789/0 conn(0x7f75cc09fae0 legacy=0x7f75cc14efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:01.322 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.307+0000 7f75d228e700 1 -- 172.21.15.42:0/2200923493 >> v1:172.21.15.40:6789/0 conn(0x7f75cc07e720 legacy=0x7f75cc09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:01.322 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.311+0000 7f75d228e700 1 -- 172.21.15.42:0/2200923493 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75cc07eb50 con 0x7f75cc0a7740 2019-12-05T06:50:01.322 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.311+0000 7f75d228e700 1 --2- 172.21.15.42:0/2200923493 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f75cc0a7740 0x7f75cc08df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f75c8002f30 tx=0x7f75c800fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:01.323 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.311+0000 7f75d228e700 1 --2- 172.21.15.42:0/2200923493 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f75cc0a7740 0x7f75cc08df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f75c8002f30 tx=0x7f75c800fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:01.323 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.311+0000 7f75d228e700 1 --2- 172.21.15.42:0/2200923493 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f75cc0a7740 0x7f75cc08df80 secure :-1 s=READY pgs=729 cs=0 l=1 rx=0x7f75c8002f30 tx=0x7f75c800fbf0).ready entity=mon.2 client_cookie=4c4b34bf09bc5111 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:50:01.323 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.311+0000 7f75d228e700 1 --2- 172.21.15.42:0/2200923493 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f75cc0a7740 0x7f75cc08df80 secure :-1 s=THROTTLE_DONE pgs=729 cs=0 l=1 rx=0x7f75c8002f30 tx=0x7f75c800fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:01.323 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.311+0000 7f75d128c700 1 -- 172.21.15.42:0/2200923493 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f75c80112b0 con 0x7f75cc0a7740 2019-12-05T06:50:01.323 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.311+0000 7f75d228e700 1 --2- 172.21.15.42:0/2200923493 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f75cc0a7740 0x7f75cc08df80 secure :-1 s=THROTTLE_DONE pgs=729 cs=0 l=1 rx=0x7f75c8002f30 tx=0x7f75c800fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:01.323 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.311+0000 7f75d228e700 1 --2- 172.21.15.42:0/2200923493 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f75cc0a7740 0x7f75cc08df80 secure :-1 s=THROTTLE_DONE pgs=729 cs=0 l=1 rx=0x7f75c8002f30 tx=0x7f75c800fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:01.323 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.311+0000 7f75d128c700 1 -- 172.21.15.42:0/2200923493 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75c8008bc0 con 0x7f75cc0a7740 2019-12-05T06:50:01.324 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.311+0000 7f75d128c700 1 -- 172.21.15.42:0/2200923493 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f75c8011890 con 0x7f75cc0a7740 2019-12-05T06:50:01.324 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.311+0000 7f75d3cf1700 1 -- 172.21.15.42:0/2200923493 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f75cc0a7740 msgr2=0x7f75cc08df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:01.324 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.311+0000 7f75d3cf1700 1 --2- 172.21.15.42:0/2200923493 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f75cc0a7740 0x7f75cc08df80 secure :-1 s=READY pgs=729 cs=0 l=1 rx=0x7f75c8002f30 tx=0x7f75c800fbf0).stop 2019-12-05T06:50:01.324 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.311+0000 7f75d3cf1700 1 -- 172.21.15.42:0/2200923493 shutdown_connections 2019-12-05T06:50:01.324 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.311+0000 7f75d3cf1700 1 --2- 172.21.15.42:0/2200923493 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f75cc0a7740 0x7f75cc08df80 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:01.324 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.311+0000 7f75d3cf1700 1 -- 172.21.15.42:0/2200923493 shutdown_connections 2019-12-05T06:50:01.324 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.311+0000 7f75d3cf1700 1 -- 172.21.15.42:0/2200923493 wait complete. 2019-12-05T06:50:01.325 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.311+0000 7f75d3cf1700 1 -- 172.21.15.42:0/2200923493 >> 172.21.15.42:0/2200923493 conn(0x7f75cc0aabe0 msgr2=0x7f75cc0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:50:01.328 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.315+0000 7f75d3cf1700 1 Processor -- start 2019-12-05T06:50:01.328 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.315+0000 7f75d3cf1700 1 -- start start 2019-12-05T06:50:01.328 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.315+0000 7f75d3cf1700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f75cc126100 0x7f75cc0a3e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:50:01.328 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.315+0000 7f75d3cf1700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f75cc125680 0x7f75cc142530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:50:01.328 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.315+0000 7f75d3cf1700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75cc145360 con 0x7f75cc144f30 2019-12-05T06:50:01.328 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.315+0000 7f75d3cf1700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75cc126530 con 0x7f75cc125680 2019-12-05T06:50:01.329 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.315+0000 7f75d3cf1700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75cc142a80 con 0x7f75cc126100 2019-12-05T06:50:01.329 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.315+0000 7f75d228e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f75cc126100 0x7f75cc0a3e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:01.329 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.315+0000 7f75d228e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f75cc126100 0x7f75cc0a3e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:38218/0 (socket says 172.21.15.42:38218) 2019-12-05T06:50:01.329 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.315+0000 7f75d228e700 1 -- 172.21.15.42:0/3266861713 learned_addr learned my addr 172.21.15.42:0/3266861713 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:50:01.329 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.315+0000 7f75d2a8f700 1 --2- 172.21.15.42:0/3266861713 >> v2:172.21.15.40:3300/0 conn(0x7f75cc125680 0x7f75cc142530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:01.329 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.315+0000 7f75d228e700 1 -- 172.21.15.42:0/3266861713 >> v2:172.21.15.40:3300/0 conn(0x7f75cc125680 msgr2=0x7f75cc142530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:01.330 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.315+0000 7f75d228e700 1 --2- 172.21.15.42:0/3266861713 >> v2:172.21.15.40:3300/0 conn(0x7f75cc125680 0x7f75cc142530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:01.330 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.315+0000 7f75d228e700 1 -- 172.21.15.42:0/3266861713 >> v1:172.21.15.40:6789/0 conn(0x7f75cc144f30 legacy=0x7f75cc122900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:01.330 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.315+0000 7f75d228e700 1 -- 172.21.15.42:0/3266861713 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75cc125ab0 con 0x7f75cc126100 2019-12-05T06:50:01.330 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.315+0000 7f75d228e700 1 --2- 172.21.15.42:0/3266861713 >> v2:172.21.15.42:3300/0 conn(0x7f75cc126100 0x7f75cc0a3e70 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f75cc0a3680 tx=0x7f75c8007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:01.330 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.319+0000 7f75d228e700 1 --2- 172.21.15.42:0/3266861713 >> v2:172.21.15.42:3300/0 conn(0x7f75cc126100 0x7f75cc0a3e70 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f75cc0a3680 tx=0x7f75c8007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:01.330 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.319+0000 7f75d228e700 1 --2- 172.21.15.42:0/3266861713 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f75cc126100 0x7f75cc0a3e70 secure :-1 s=READY pgs=3302 cs=0 l=1 rx=0x7f75cc0a3680 tx=0x7f75c8007710).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:50:01.330 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.319+0000 7f75d228e700 1 --2- 172.21.15.42:0/3266861713 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f75cc126100 0x7f75cc0a3e70 secure :-1 s=THROTTLE_DONE pgs=3302 cs=0 l=1 rx=0x7f75cc0a3680 tx=0x7f75c8007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:01.331 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.319+0000 7f75d228e700 1 --2- 172.21.15.42:0/3266861713 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f75cc126100 0x7f75cc0a3e70 secure :-1 s=THROTTLE_DONE pgs=3302 cs=0 l=1 rx=0x7f75cc0a3680 tx=0x7f75c8007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:01.331 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.319+0000 7f75c2ffd700 1 -- 172.21.15.42:0/3266861713 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f75c8007340 con 0x7f75cc126100 2019-12-05T06:50:01.331 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.319+0000 7f75d228e700 1 --2- 172.21.15.42:0/3266861713 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f75cc126100 0x7f75cc0a3e70 secure :-1 s=THROTTLE_DONE pgs=3302 cs=0 l=1 rx=0x7f75cc0a3680 tx=0x7f75c8007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:01.331 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.319+0000 7f75c2ffd700 1 -- 172.21.15.42:0/3266861713 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75c80114b0 con 0x7f75cc126100 2019-12-05T06:50:01.331 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.319+0000 7f75c2ffd700 1 -- 172.21.15.42:0/3266861713 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f75c800f360 con 0x7f75cc126100 2019-12-05T06:50:01.331 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.319+0000 7f75d3cf1700 1 -- 172.21.15.42:0/3266861713 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75cc12cad0 con 0x7f75cc126100 2019-12-05T06:50:01.331 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.319+0000 7f75d3cf1700 1 -- 172.21.15.42:0/3266861713 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75cc12cfa0 con 0x7f75cc126100 2019-12-05T06:50:01.332 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.319+0000 7f75d228e700 1 --2- 172.21.15.42:0/3266861713 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f75cc126100 0x7f75cc0a3e70 secure :-1 s=THROTTLE_DONE pgs=3302 cs=0 l=1 rx=0x7f75cc0a3680 tx=0x7f75c8007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:01.332 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.319+0000 7f75c2ffd700 1 -- 172.21.15.42:0/3266861713 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f75c8005540 con 0x7f75cc126100 2019-12-05T06:50:01.332 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.319+0000 7f75d228e700 1 --2- 172.21.15.42:0/3266861713 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f75cc126100 0x7f75cc0a3e70 secure :-1 s=THROTTLE_DONE pgs=3302 cs=0 l=1 rx=0x7f75cc0a3680 tx=0x7f75c8007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:01.332 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.319+0000 7f75c2ffd700 1 --2- 172.21.15.42:0/3266861713 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f75b403ae70 0x7f75b403d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:50:01.332 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.319+0000 7f75c2ffd700 1 -- 172.21.15.42:0/3266861713 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 12223+0+0 (secure 0 0 0) 0x7f75c804b6d0 con 0x7f75cc126100 2019-12-05T06:50:01.332 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.319+0000 7f75d1a8d700 1 --2- 172.21.15.42:0/3266861713 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f75b403ae70 0x7f75b403d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:01.332 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.319+0000 7f75d1a8d700 1 --2- 172.21.15.42:0/3266861713 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f75b403ae70 0x7f75b403d380 crc :-1 s=READY pgs=1385 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:50:01.333 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.323+0000 7f75d3cf1700 1 -- 172.21.15.42:0/3266861713 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75cc05d110 con 0x7f75cc126100 2019-12-05T06:50:01.335 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.327+0000 7f75d228e700 1 --2- 172.21.15.42:0/3266861713 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f75cc126100 0x7f75cc0a3e70 secure :-1 s=THROTTLE_DONE pgs=3302 cs=0 l=1 rx=0x7f75cc0a3680 tx=0x7f75c8007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:01.336 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.327+0000 7f75c2ffd700 1 -- 172.21.15.42:0/3266861713 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f75c8033560 con 0x7f75cc126100 2019-12-05T06:50:01.498 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.491+0000 7f75d3cf1700 1 -- 172.21.15.42:0/3266861713 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f75cc05b8c0 con 0x7f75cc126100 2019-12-05T06:50:01.499 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.491+0000 7f75d228e700 1 --2- 172.21.15.42:0/3266861713 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f75cc126100 0x7f75cc0a3e70 secure :-1 s=THROTTLE_DONE pgs=3302 cs=0 l=1 rx=0x7f75cc0a3680 tx=0x7f75c8007710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:01.499 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.491+0000 7f75c2ffd700 1 -- 172.21.15.42:0/3266861713 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 399 v399) v1 ==== 95+0+2113 (secure 0 0 0) 0x7f75c8012c00 con 0x7f75cc126100 2019-12-05T06:50:01.500 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 399 2019-12-05T06:50:01.500 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:50:01.500 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":399,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":8537,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/1881703925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1881703925},{"type":"v1","addr":"172.21.15.42:6835","nonce":1881703925}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":399}],"filesystems":[{"mdsmap":{"epoch":398,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:49:10.408613+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":121,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12687},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12687":{"gid":12687,"name":"b","rank":0,"incarnation":395,"state":"up:active","state_seq":12,"addr":"172.21.15.40:6833/815316041","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":815316041},{"type":"v1","addr":"172.21.15.40:6833","nonce":815316041}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:50:01.502 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.495+0000 7f75d3cf1700 1 -- 172.21.15.42:0/3266861713 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f75b403ae70 msgr2=0x7f75b403d380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:01.503 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.495+0000 7f75d3cf1700 1 --2- 172.21.15.42:0/3266861713 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f75b403ae70 0x7f75b403d380 crc :-1 s=READY pgs=1385 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:01.503 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.495+0000 7f75d3cf1700 1 -- 172.21.15.42:0/3266861713 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f75cc126100 msgr2=0x7f75cc0a3e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:01.503 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.495+0000 7f75d3cf1700 1 --2- 172.21.15.42:0/3266861713 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f75cc126100 0x7f75cc0a3e70 secure :-1 s=READY pgs=3302 cs=0 l=1 rx=0x7f75cc0a3680 tx=0x7f75c8007710).stop 2019-12-05T06:50:01.503 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.495+0000 7f75d3cf1700 1 -- 172.21.15.42:0/3266861713 shutdown_connections 2019-12-05T06:50:01.503 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.495+0000 7f75d3cf1700 1 --2- 172.21.15.42:0/3266861713 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f75b403ae70 0x7f75b403d380 unknown :-1 s=CLOSED pgs=1385 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:01.503 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.495+0000 7f75d3cf1700 1 --2- 172.21.15.42:0/3266861713 >> v2:172.21.15.40:3300/0 conn(0x7f75cc125680 0x7f75cc142530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:01.503 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.495+0000 7f75d3cf1700 1 --2- 172.21.15.42:0/3266861713 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f75cc126100 0x7f75cc0a3e70 unknown :-1 s=CLOSED pgs=3302 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:01.504 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.495+0000 7f75d3cf1700 1 -- 172.21.15.42:0/3266861713 shutdown_connections 2019-12-05T06:50:01.504 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.495+0000 7f75d3cf1700 1 -- 172.21.15.42:0/3266861713 wait complete. 2019-12-05T06:50:01.504 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:01.495+0000 7f75d3cf1700 1 -- 172.21.15.42:0/3266861713 >> 172.21.15.42:0/3266861713 conn(0x7f75cc0a82a0 msgr2=0x7f75cc09f580 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:50:01.512 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:50:03.513 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:50:03.620 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:03.607+0000 7f00e38fc700 1 Processor -- start 2019-12-05T06:50:03.621 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:03.607+0000 7f00e38fc700 1 -- start start 2019-12-05T06:50:03.621 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:03.607+0000 7f00e38fc700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f00dc08df80 0x7f00dc09f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:50:03.621 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:03.607+0000 7f00e38fc700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00dc09fc60 con 0x7f00dc0a7740 2019-12-05T06:50:03.621 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:03.607+0000 7f00e38fc700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00dc0a0710 con 0x7f00dc09f830 2019-12-05T06:50:03.621 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:03.607+0000 7f00e38fc700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00dc0a0b00 con 0x7f00dc08df80 2019-12-05T06:50:03.621 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:03.607+0000 7f00e1698700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f00dc08df80 0x7f00dc09f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:03.621 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:03.607+0000 7f00e1e99700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f00dc0a7740 0x7f00dc07e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:47710/0 (socket says 172.21.15.42:47710) 2019-12-05T06:50:03.622 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:03.607+0000 7f00e269a700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f00dc09f830 0x7f00dc14efe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:38274/0 (socket says 172.21.15.42:38274) 2019-12-05T06:50:03.622 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:03.607+0000 7f00e1e99700 1 -- 172.21.15.42:0/1623067376 learned_addr learned my addr 172.21.15.42:0/1623067376 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:50:03.622 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:03.607+0000 7f00e0e97700 1 -- 172.21.15.42:0/1623067376 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4250223709 0 0) 0x7f00dc0a0710 con 0x7f00dc09f830 2019-12-05T06:50:03.622 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:03.607+0000 7f00e0e97700 1 -- 172.21.15.42:0/1623067376 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00c80030b0 con 0x7f00dc09f830 2019-12-05T06:50:03.622 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:03.607+0000 7f00e1698700 1 -- 172.21.15.42:0/1623067376 >> v1:172.21.15.42:6789/0 conn(0x7f00dc09f830 legacy=0x7f00dc14efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:03.622 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:03.607+0000 7f00e0e97700 1 -- 172.21.15.42:0/1623067376 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1243153345 0 0) 0x7f00dc09fc60 con 0x7f00dc0a7740 2019-12-05T06:50:03.622 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:03.607+0000 7f00e1698700 1 -- 172.21.15.42:0/1623067376 >> v1:172.21.15.40:6789/0 conn(0x7f00dc0a7740 legacy=0x7f00dc07e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:03.623 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:03.607+0000 7f00e1698700 1 -- 172.21.15.42:0/1623067376 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00dc07ef20 con 0x7f00dc08df80 2019-12-05T06:50:03.623 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:03.607+0000 7f00e1698700 1 --2- 172.21.15.42:0/1623067376 >> v2:172.21.15.42:3300/0 conn(0x7f00dc08df80 0x7f00dc09f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f00cc008790 tx=0x7f00cc00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:03.623 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:03.611+0000 7f00e1698700 1 --2- 172.21.15.42:0/1623067376 >> v2:172.21.15.42:3300/0 conn(0x7f00dc08df80 0x7f00dc09f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f00cc008790 tx=0x7f00cc00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:03.623 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:03.611+0000 7f00e1698700 1 --2- 172.21.15.42:0/1623067376 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f00dc08df80 0x7f00dc09f2e0 secure :-1 s=READY pgs=3304 cs=0 l=1 rx=0x7f00cc008790 tx=0x7f00cc00fbf0).ready entity=mon.1 client_cookie=ba4601402ae32a68 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:50:03.623 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:03.611+0000 7f00e1698700 1 --2- 172.21.15.42:0/1623067376 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f00dc08df80 0x7f00dc09f2e0 secure :-1 s=THROTTLE_DONE pgs=3304 cs=0 l=1 rx=0x7f00cc008790 tx=0x7f00cc00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:03.623 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:03.611+0000 7f00e0e97700 1 -- 172.21.15.42:0/1623067376 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f00cc011270 con 0x7f00dc08df80 2019-12-05T06:50:03.623 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:03.611+0000 7f00e1698700 1 --2- 172.21.15.42:0/1623067376 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f00dc08df80 0x7f00dc09f2e0 secure :-1 s=THROTTLE_DONE pgs=3304 cs=0 l=1 rx=0x7f00cc008790 tx=0x7f00cc00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:03.624 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:03.611+0000 7f00e1698700 1 --2- 172.21.15.42:0/1623067376 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f00dc08df80 0x7f00dc09f2e0 secure :-1 s=THROTTLE_DONE pgs=3304 cs=0 l=1 rx=0x7f00cc008790 tx=0x7f00cc00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:03.624 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:03.611+0000 7f00e0e97700 1 -- 172.21.15.42:0/1623067376 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00cc008bc0 con 0x7f00dc08df80 2019-12-05T06:50:03.624 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:03.611+0000 7f00e0e97700 1 -- 172.21.15.42:0/1623067376 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f00cc011850 con 0x7f00dc08df80 2019-12-05T06:50:03.624 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:03.611+0000 7f00e38fc700 1 -- 172.21.15.42:0/1623067376 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f00dc08df80 msgr2=0x7f00dc09f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:03.625 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:03.611+0000 7f00e38fc700 1 --2- 172.21.15.42:0/1623067376 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f00dc08df80 0x7f00dc09f2e0 secure :-1 s=READY pgs=3304 cs=0 l=1 rx=0x7f00cc008790 tx=0x7f00cc00fbf0).stop 2019-12-05T06:50:03.625 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:03.611+0000 7f00e38fc700 1 -- 172.21.15.42:0/1623067376 shutdown_connections 2019-12-05T06:50:03.626 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:03.611+0000 7f00e38fc700 1 --2- 172.21.15.42:0/1623067376 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f00dc08df80 0x7f00dc09f2e0 unknown :-1 s=CLOSED pgs=3304 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:03.626 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:03.611+0000 7f00e38fc700 1 -- 172.21.15.42:0/1623067376 shutdown_connections 2019-12-05T06:50:03.627 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:03.611+0000 7f00e38fc700 1 -- 172.21.15.42:0/1623067376 wait complete. 2019-12-05T06:50:03.627 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:03.611+0000 7f00e38fc700 1 -- 172.21.15.42:0/1623067376 >> 172.21.15.42:0/1623067376 conn(0x7f00dc0aabe0 msgr2=0x7f00dc0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:50:03.627 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:03.615+0000 7f00e38fc700 1 Processor -- start 2019-12-05T06:50:03.627 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:03.615+0000 7f00e38fc700 1 -- start start 2019-12-05T06:50:03.631 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:03.615+0000 7f00e38fc700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f00dc144550 0x7f00dc0a5150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:50:03.631 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:03.615+0000 7f00e38fc700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f00dc143620 0x7f00dc07be90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:50:03.632 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:03.615+0000 7f00e38fc700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f00dc0a3a60 0x7f00dc07c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:50:03.632 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:03.615+0000 7f00e38fc700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00dc144010 con 0x7f00dc144550 2019-12-05T06:50:03.632 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:03.615+0000 7f00e38fc700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00dc1442b0 con 0x7f00dc143620 2019-12-05T06:50:03.632 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:03.615+0000 7f00e38fc700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00dc143c20 con 0x7f00dc0a3a60 2019-12-05T06:50:03.632 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:03.615+0000 7f00e1698700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f00dc143620 0x7f00dc07be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:03.632 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:03.615+0000 7f00e1698700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f00dc143620 0x7f00dc07be90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:38240/0 (socket says 172.21.15.42:38240) 2019-12-05T06:50:03.633 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:03.615+0000 7f00e1698700 1 -- 172.21.15.42:0/678535825 learned_addr learned my addr 172.21.15.42:0/678535825 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:50:03.633 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:03.615+0000 7f00e1e99700 1 --2- 172.21.15.42:0/678535825 >> v2:172.21.15.40:3300/0 conn(0x7f00dc144550 0x7f00dc0a5150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:03.633 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:03.615+0000 7f00e269a700 1 --2- 172.21.15.42:0/678535825 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f00dc0a3a60 0x7f00dc07c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:03.633 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:03.615+0000 7f00e1698700 1 -- 172.21.15.42:0/678535825 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f00dc0a3a60 msgr2=0x7f00dc07c3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:03.634 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:03.615+0000 7f00e1698700 1 --2- 172.21.15.42:0/678535825 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f00dc0a3a60 0x7f00dc07c3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:03.634 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:03.615+0000 7f00e1698700 1 -- 172.21.15.42:0/678535825 >> v2:172.21.15.40:3300/0 conn(0x7f00dc144550 msgr2=0x7f00dc0a5150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:03.634 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:03.615+0000 7f00e1698700 1 --2- 172.21.15.42:0/678535825 >> v2:172.21.15.40:3300/0 conn(0x7f00dc144550 0x7f00dc0a5150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:03.634 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:03.615+0000 7f00e1698700 1 -- 172.21.15.42:0/678535825 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00dc150230 con 0x7f00dc143620 2019-12-05T06:50:03.634 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:03.615+0000 7f00e1698700 1 --2- 172.21.15.42:0/678535825 >> v2:172.21.15.42:3300/0 conn(0x7f00dc143620 0x7f00dc07be90 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f00cc008790 tx=0x7f00cc008ef0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:03.634 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:03.615+0000 7f00e1698700 1 --2- 172.21.15.42:0/678535825 >> v2:172.21.15.42:3300/0 conn(0x7f00dc143620 0x7f00dc07be90 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f00cc008790 tx=0x7f00cc008ef0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:03.634 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:03.615+0000 7f00e1698700 1 --2- 172.21.15.42:0/678535825 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f00dc143620 0x7f00dc07be90 secure :-1 s=READY pgs=3305 cs=0 l=1 rx=0x7f00cc008790 tx=0x7f00cc008ef0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:50:03.635 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:03.615+0000 7f00e1698700 1 --2- 172.21.15.42:0/678535825 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f00dc143620 0x7f00dc07be90 secure :-1 s=THROTTLE_DONE pgs=3305 cs=0 l=1 rx=0x7f00cc008790 tx=0x7f00cc008ef0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:03.635 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:03.615+0000 7f00d27fc700 1 -- 172.21.15.42:0/678535825 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f00cc007340 con 0x7f00dc143620 2019-12-05T06:50:03.635 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:03.615+0000 7f00e1698700 1 --2- 172.21.15.42:0/678535825 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f00dc143620 0x7f00dc07be90 secure :-1 s=THROTTLE_DONE pgs=3305 cs=0 l=1 rx=0x7f00cc008790 tx=0x7f00cc008ef0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:03.635 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:03.615+0000 7f00e1698700 1 --2- 172.21.15.42:0/678535825 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f00dc143620 0x7f00dc07be90 secure :-1 s=THROTTLE_DONE pgs=3305 cs=0 l=1 rx=0x7f00cc008790 tx=0x7f00cc008ef0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:03.635 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:03.615+0000 7f00d27fc700 1 -- 172.21.15.42:0/678535825 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00cc011470 con 0x7f00dc143620 2019-12-05T06:50:03.635 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:03.615+0000 7f00d27fc700 1 -- 172.21.15.42:0/678535825 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f00cc005540 con 0x7f00dc143620 2019-12-05T06:50:03.635 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:03.615+0000 7f00e38fc700 1 -- 172.21.15.42:0/678535825 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00dc145280 con 0x7f00dc143620 2019-12-05T06:50:03.636 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:03.615+0000 7f00e38fc700 1 -- 172.21.15.42:0/678535825 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00dc0a58c0 con 0x7f00dc143620 2019-12-05T06:50:03.636 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:03.619+0000 7f00e1698700 1 --2- 172.21.15.42:0/678535825 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f00dc143620 0x7f00dc07be90 secure :-1 s=THROTTLE_DONE pgs=3305 cs=0 l=1 rx=0x7f00cc008790 tx=0x7f00cc008ef0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:03.636 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:03.619+0000 7f00d27fc700 1 -- 172.21.15.42:0/678535825 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f00cc005790 con 0x7f00dc143620 2019-12-05T06:50:03.637 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:03.619+0000 7f00e1698700 1 --2- 172.21.15.42:0/678535825 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f00dc143620 0x7f00dc07be90 secure :-1 s=THROTTLE_DONE pgs=3305 cs=0 l=1 rx=0x7f00cc008790 tx=0x7f00cc008ef0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:03.638 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:03.619+0000 7f00d27fc700 1 --2- 172.21.15.42:0/678535825 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f00c403ae70 0x7f00c403d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:50:03.638 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:03.619+0000 7f00d27fc700 1 -- 172.21.15.42:0/678535825 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 12223+0+0 (secure 0 0 0) 0x7f00cc04bc70 con 0x7f00dc143620 2019-12-05T06:50:03.638 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:03.619+0000 7f00e1e99700 1 --2- 172.21.15.42:0/678535825 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f00c403ae70 0x7f00c403d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:03.638 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:03.619+0000 7f00e1e99700 1 --2- 172.21.15.42:0/678535825 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f00c403ae70 0x7f00c403d380 crc :-1 s=READY pgs=1386 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:50:03.638 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:03.619+0000 7f00e38fc700 1 -- 172.21.15.42:0/678535825 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00dc143e10 con 0x7f00dc143620 2019-12-05T06:50:03.639 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:03.627+0000 7f00e1698700 1 --2- 172.21.15.42:0/678535825 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f00dc143620 0x7f00dc07be90 secure :-1 s=THROTTLE_DONE pgs=3305 cs=0 l=1 rx=0x7f00cc008790 tx=0x7f00cc008ef0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:03.639 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:03.627+0000 7f00d27fc700 1 -- 172.21.15.42:0/678535825 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f00cc032cd0 con 0x7f00dc143620 2019-12-05T06:50:03.810 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:03.803+0000 7f00e38fc700 1 -- 172.21.15.42:0/678535825 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f00dc05d110 con 0x7f00dc143620 2019-12-05T06:50:03.812 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:03.803+0000 7f00e1698700 1 --2- 172.21.15.42:0/678535825 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f00dc143620 0x7f00dc07be90 secure :-1 s=THROTTLE_DONE pgs=3305 cs=0 l=1 rx=0x7f00cc008790 tx=0x7f00cc008ef0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:03.812 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:03.803+0000 7f00d27fc700 1 -- 172.21.15.42:0/678535825 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 399 v399) v1 ==== 95+0+2113 (secure 0 0 0) 0x7f00cc032f30 con 0x7f00dc143620 2019-12-05T06:50:03.812 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 399 2019-12-05T06:50:03.812 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:50:03.813 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":399,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":8537,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/1881703925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1881703925},{"type":"v1","addr":"172.21.15.42:6835","nonce":1881703925}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":399}],"filesystems":[{"mdsmap":{"epoch":398,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:49:10.408613+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":121,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12687},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12687":{"gid":12687,"name":"b","rank":0,"incarnation":395,"state":"up:active","state_seq":12,"addr":"172.21.15.40:6833/815316041","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":815316041},{"type":"v1","addr":"172.21.15.40:6833","nonce":815316041}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:50:03.814 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:03.807+0000 7f00e38fc700 1 -- 172.21.15.42:0/678535825 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f00c403ae70 msgr2=0x7f00c403d380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:03.815 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:03.807+0000 7f00e38fc700 1 --2- 172.21.15.42:0/678535825 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f00c403ae70 0x7f00c403d380 crc :-1 s=READY pgs=1386 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:03.815 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:03.807+0000 7f00e38fc700 1 -- 172.21.15.42:0/678535825 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f00dc143620 msgr2=0x7f00dc07be90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:03.815 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:03.807+0000 7f00e38fc700 1 --2- 172.21.15.42:0/678535825 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f00dc143620 0x7f00dc07be90 secure :-1 s=READY pgs=3305 cs=0 l=1 rx=0x7f00cc008790 tx=0x7f00cc008ef0).stop 2019-12-05T06:50:03.816 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:03.807+0000 7f00e38fc700 1 -- 172.21.15.42:0/678535825 shutdown_connections 2019-12-05T06:50:03.816 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:03.807+0000 7f00e38fc700 1 --2- 172.21.15.42:0/678535825 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f00c403ae70 0x7f00c403d380 unknown :-1 s=CLOSED pgs=1386 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:03.816 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:03.807+0000 7f00e38fc700 1 --2- 172.21.15.42:0/678535825 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f00dc0a3a60 0x7f00dc07c3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:03.816 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:03.807+0000 7f00e38fc700 1 --2- 172.21.15.42:0/678535825 >> v2:172.21.15.40:3300/0 conn(0x7f00dc144550 0x7f00dc0a5150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:03.817 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:03.807+0000 7f00e38fc700 1 --2- 172.21.15.42:0/678535825 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f00dc143620 0x7f00dc07be90 unknown :-1 s=CLOSED pgs=3305 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:03.817 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:03.807+0000 7f00e38fc700 1 -- 172.21.15.42:0/678535825 shutdown_connections 2019-12-05T06:50:03.817 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:03.807+0000 7f00e38fc700 1 -- 172.21.15.42:0/678535825 wait complete. 2019-12-05T06:50:03.818 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:03.807+0000 7f00e38fc700 1 -- 172.21.15.42:0/678535825 >> 172.21.15.42:0/678535825 conn(0x7f00dc08fde0 msgr2=0x7f00dc0a9770 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:50:03.824 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:50:05.826 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:50:05.933 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:05.919+0000 7f4bbe12e700 1 Processor -- start 2019-12-05T06:50:05.933 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:05.919+0000 7f4bbe12e700 1 -- start start 2019-12-05T06:50:05.933 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:05.923+0000 7f4bbe12e700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4bb80a7740 0x7f4bb808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:50:05.934 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:05.923+0000 7f4bbe12e700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f4bb809fae0 0x7f4bb807eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:50:05.934 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:05.923+0000 7f4bbe12e700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bb80a0710 con 0x7f4bb807e720 2019-12-05T06:50:05.934 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:05.923+0000 7f4bbe12e700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4bb80a0b00 con 0x7f4bb809fae0 2019-12-05T06:50:05.934 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:05.923+0000 7f4bbe12e700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bb808e5b0 con 0x7f4bb80a7740 2019-12-05T06:50:05.934 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:05.923+0000 7f4bb7fff700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4bb80a7740 0x7f4bb808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:05.935 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:05.923+0000 7f4bb7fff700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4bb80a7740 0x7f4bb808df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:38714/0 (socket says 172.21.15.42:38714) 2019-12-05T06:50:05.935 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:05.923+0000 7f4bb7fff700 1 -- 172.21.15.42:0/477387120 learned_addr learned my addr 172.21.15.42:0/477387120 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:50:05.936 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:05.923+0000 7f4bbcecc700 1 --2- 172.21.15.42:0/477387120 >> v2:172.21.15.40:3300/0 conn(0x7f4bb809fae0 0x7f4bb807eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:05.936 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:05.923+0000 7f4bb7fff700 1 -- 172.21.15.42:0/477387120 >> v2:172.21.15.40:3300/0 conn(0x7f4bb809fae0 msgr2=0x7f4bb807eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:50:05.936 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:05.923+0000 7f4bb7fff700 1 --2- 172.21.15.42:0/477387120 >> v2:172.21.15.40:3300/0 conn(0x7f4bb809fae0 0x7f4bb807eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:50:05.936 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:05.923+0000 7f4bb6ffd700 1 -- 172.21.15.42:0/477387120 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3715109872 0 0) 0x7f4bb80a0710 con 0x7f4bb807e720 2019-12-05T06:50:05.936 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:05.923+0000 7f4bb7fff700 1 -- 172.21.15.42:0/477387120 >> v1:172.21.15.40:6789/0 conn(0x7f4bb807e720 legacy=0x7f4bb809f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:05.937 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:05.923+0000 7f4bb7fff700 1 -- 172.21.15.42:0/477387120 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bb807f0a0 con 0x7f4bb80a7740 2019-12-05T06:50:05.937 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:05.923+0000 7f4bb7fff700 1 --2- 172.21.15.42:0/477387120 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4bb80a7740 0x7f4bb808df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f4ba8008790 tx=0x7f4ba800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:05.937 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:05.923+0000 7f4bb7fff700 1 --2- 172.21.15.42:0/477387120 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4bb80a7740 0x7f4bb808df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f4ba8008790 tx=0x7f4ba800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:05.939 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:05.923+0000 7f4bb7fff700 1 --2- 172.21.15.42:0/477387120 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4bb80a7740 0x7f4bb808df80 secure :-1 s=READY pgs=730 cs=0 l=1 rx=0x7f4ba8008790 tx=0x7f4ba800fd30).ready entity=mon.2 client_cookie=db3907da8d848604 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:50:05.940 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:05.923+0000 7f4bb7fff700 1 --2- 172.21.15.42:0/477387120 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4bb80a7740 0x7f4bb808df80 secure :-1 s=THROTTLE_DONE pgs=730 cs=0 l=1 rx=0x7f4ba8008790 tx=0x7f4ba800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:05.940 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:05.923+0000 7f4bb7fff700 1 --2- 172.21.15.42:0/477387120 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4bb80a7740 0x7f4bb808df80 secure :-1 s=THROTTLE_DONE pgs=730 cs=0 l=1 rx=0x7f4ba8008790 tx=0x7f4ba800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:05.940 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:05.923+0000 7f4bb6ffd700 1 -- 172.21.15.42:0/477387120 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4ba8008c90 con 0x7f4bb80a7740 2019-12-05T06:50:05.940 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:05.923+0000 7f4bb7fff700 1 --2- 172.21.15.42:0/477387120 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4bb80a7740 0x7f4bb808df80 secure :-1 s=THROTTLE_DONE pgs=730 cs=0 l=1 rx=0x7f4ba8008790 tx=0x7f4ba800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:05.941 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:05.923+0000 7f4bb6ffd700 1 -- 172.21.15.42:0/477387120 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ba80088b0 con 0x7f4bb80a7740 2019-12-05T06:50:05.941 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:05.923+0000 7f4bb6ffd700 1 -- 172.21.15.42:0/477387120 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4ba8011b10 con 0x7f4bb80a7740 2019-12-05T06:50:05.941 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:05.923+0000 7f4bbe12e700 1 -- 172.21.15.42:0/477387120 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4bb80a7740 msgr2=0x7f4bb808df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:05.942 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:05.923+0000 7f4bbe12e700 1 --2- 172.21.15.42:0/477387120 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4bb80a7740 0x7f4bb808df80 secure :-1 s=READY pgs=730 cs=0 l=1 rx=0x7f4ba8008790 tx=0x7f4ba800fd30).stop 2019-12-05T06:50:05.942 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:05.927+0000 7f4bbe12e700 1 -- 172.21.15.42:0/477387120 shutdown_connections 2019-12-05T06:50:05.942 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:05.927+0000 7f4bbe12e700 1 --2- 172.21.15.42:0/477387120 >> v2:172.21.15.40:3300/0 conn(0x7f4bb809fae0 0x7f4bb807eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:50:05.942 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:05.927+0000 7f4bbe12e700 1 --2- 172.21.15.42:0/477387120 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4bb80a7740 0x7f4bb808df80 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:05.943 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:05.927+0000 7f4bbe12e700 1 -- 172.21.15.42:0/477387120 shutdown_connections 2019-12-05T06:50:05.943 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:05.927+0000 7f4bbe12e700 1 -- 172.21.15.42:0/477387120 wait complete. 2019-12-05T06:50:05.943 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:05.927+0000 7f4bbe12e700 1 -- 172.21.15.42:0/477387120 >> 172.21.15.42:0/477387120 conn(0x7f4bb80aabe0 msgr2=0x7f4bb80a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:50:05.949 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:05.927+0000 7f4bbe12e700 1 Processor -- start 2019-12-05T06:50:05.949 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:05.927+0000 7f4bbe12e700 1 -- start start 2019-12-05T06:50:05.949 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:05.931+0000 7f4bbe12e700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4bb81192c0 0x7f4bb81188f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:50:05.950 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:05.931+0000 7f4bbe12e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f4bb807de10 0x7f4bb8117de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:50:05.950 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:05.931+0000 7f4bbe12e700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bb811b4e0 con 0x7f4bb807d9e0 2019-12-05T06:50:05.950 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:05.931+0000 7f4bbe12e700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4bb811b710 con 0x7f4bb807de10 2019-12-05T06:50:05.950 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:05.931+0000 7f4bbe12e700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bb811b240 con 0x7f4bb81192c0 2019-12-05T06:50:05.950 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:05.931+0000 7f4bbcecc700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f4bb807de10 0x7f4bb8117de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:05.951 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:05.931+0000 7f4bb7fff700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4bb81192c0 0x7f4bb81188f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:05.951 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:05.931+0000 7f4bb77fe700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f4bb807d9e0 0x7f4bb81175e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:38294/0 (socket says 172.21.15.42:38294) 2019-12-05T06:50:05.951 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:05.931+0000 7f4bb77fe700 1 -- 172.21.15.42:0/4002380539 learned_addr learned my addr 172.21.15.42:0/4002380539 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:50:05.951 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:05.931+0000 7f4bb7fff700 1 -- 172.21.15.42:0/4002380539 >> v2:172.21.15.42:3300/0 conn(0x7f4bb807de10 msgr2=0x7f4bb8117de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:05.951 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:05.931+0000 7f4bb7fff700 1 --2- 172.21.15.42:0/4002380539 >> v2:172.21.15.42:3300/0 conn(0x7f4bb807de10 0x7f4bb8117de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:05.952 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:05.931+0000 7f4bb7fff700 1 -- 172.21.15.42:0/4002380539 >> v1:172.21.15.42:6789/0 conn(0x7f4bb807d9e0 legacy=0x7f4bb81175e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:05.952 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:05.931+0000 7f4bb7fff700 1 -- 172.21.15.42:0/4002380539 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bb811bec0 con 0x7f4bb81192c0 2019-12-05T06:50:05.952 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:05.931+0000 7f4bbcecc700 1 --2- 172.21.15.42:0/4002380539 >> v2:172.21.15.42:3300/0 conn(0x7f4bb807de10 0x7f4bb8117de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:50:05.952 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:05.931+0000 7f4bb7fff700 1 --2- 172.21.15.42:0/4002380539 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4bb81192c0 0x7f4bb81188f0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f4ba8008790 tx=0x7f4ba8008ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:05.952 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:05.931+0000 7f4bb7fff700 1 --2- 172.21.15.42:0/4002380539 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4bb81192c0 0x7f4bb81188f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f4ba8008790 tx=0x7f4ba8008ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:05.952 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:05.931+0000 7f4bb7fff700 1 --2- 172.21.15.42:0/4002380539 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4bb81192c0 0x7f4bb81188f0 secure :-1 s=READY pgs=731 cs=0 l=1 rx=0x7f4ba8008790 tx=0x7f4ba8008ae0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:50:05.952 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:05.931+0000 7f4bb7fff700 1 --2- 172.21.15.42:0/4002380539 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4bb81192c0 0x7f4bb81188f0 secure :-1 s=THROTTLE_DONE pgs=731 cs=0 l=1 rx=0x7f4ba8008790 tx=0x7f4ba8008ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:05.953 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:05.931+0000 7f4bb7fff700 1 --2- 172.21.15.42:0/4002380539 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4bb81192c0 0x7f4bb81188f0 secure :-1 s=THROTTLE_DONE pgs=731 cs=0 l=1 rx=0x7f4ba8008790 tx=0x7f4ba8008ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:05.953 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:05.931+0000 7f4bb4ff9700 1 -- 172.21.15.42:0/4002380539 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4ba800f470 con 0x7f4bb81192c0 2019-12-05T06:50:05.953 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:05.931+0000 7f4bb7fff700 1 --2- 172.21.15.42:0/4002380539 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4bb81192c0 0x7f4bb81188f0 secure :-1 s=THROTTLE_DONE pgs=731 cs=0 l=1 rx=0x7f4ba8008790 tx=0x7f4ba8008ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:05.953 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:05.931+0000 7f4bb4ff9700 1 -- 172.21.15.42:0/4002380539 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ba8011730 con 0x7f4bb81192c0 2019-12-05T06:50:05.953 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:05.931+0000 7f4bbe12e700 1 -- 172.21.15.42:0/4002380539 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bb81375a0 con 0x7f4bb81192c0 2019-12-05T06:50:05.953 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:05.931+0000 7f4bb4ff9700 1 -- 172.21.15.42:0/4002380539 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4ba8005730 con 0x7f4bb81192c0 2019-12-05T06:50:05.953 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:05.931+0000 7f4bbe12e700 1 -- 172.21.15.42:0/4002380539 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bb8137ad0 con 0x7f4bb81192c0 2019-12-05T06:50:05.954 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:05.931+0000 7f4bb7fff700 1 --2- 172.21.15.42:0/4002380539 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4bb81192c0 0x7f4bb81188f0 secure :-1 s=THROTTLE_DONE pgs=731 cs=0 l=1 rx=0x7f4ba8008790 tx=0x7f4ba8008ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:05.954 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:05.931+0000 7f4bb4ff9700 1 -- 172.21.15.42:0/4002380539 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f4ba8005930 con 0x7f4bb81192c0 2019-12-05T06:50:05.954 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:05.931+0000 7f4bb7fff700 1 --2- 172.21.15.42:0/4002380539 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4bb81192c0 0x7f4bb81188f0 secure :-1 s=THROTTLE_DONE pgs=731 cs=0 l=1 rx=0x7f4ba8008790 tx=0x7f4ba8008ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:05.954 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:05.935+0000 7f4bb4ff9700 1 --2- 172.21.15.42:0/4002380539 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4b9c03ae70 0x7f4b9c03d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:50:05.954 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:05.935+0000 7f4bb4ff9700 1 -- 172.21.15.42:0/4002380539 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 12223+0+0 (secure 0 0 0) 0x7f4ba804b730 con 0x7f4bb81192c0 2019-12-05T06:50:05.954 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:05.935+0000 7f4bb77fe700 1 --2- 172.21.15.42:0/4002380539 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4b9c03ae70 0x7f4b9c03d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:05.954 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:05.935+0000 7f4bb77fe700 1 --2- 172.21.15.42:0/4002380539 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4b9c03ae70 0x7f4b9c03d380 crc :-1 s=READY pgs=1387 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:50:05.955 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:05.935+0000 7f4bbe12e700 1 -- 172.21.15.42:0/4002380539 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bb805d110 con 0x7f4bb81192c0 2019-12-05T06:50:05.955 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:05.947+0000 7f4bb7fff700 1 --2- 172.21.15.42:0/4002380539 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4bb81192c0 0x7f4bb81188f0 secure :-1 s=THROTTLE_DONE pgs=731 cs=0 l=1 rx=0x7f4ba8008790 tx=0x7f4ba8008ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:05.955 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:05.947+0000 7f4bb4ff9700 1 -- 172.21.15.42:0/4002380539 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f4ba8030420 con 0x7f4bb81192c0 2019-12-05T06:50:06.122 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:06.115+0000 7f4bbe12e700 1 -- 172.21.15.42:0/4002380539 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4bb805b8c0 con 0x7f4bb81192c0 2019-12-05T06:50:06.123 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:06.115+0000 7f4bb7fff700 1 --2- 172.21.15.42:0/4002380539 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4bb81192c0 0x7f4bb81188f0 secure :-1 s=THROTTLE_DONE pgs=731 cs=0 l=1 rx=0x7f4ba8008790 tx=0x7f4ba8008ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:06.134 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:06.115+0000 7f4bb4ff9700 1 -- 172.21.15.42:0/4002380539 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 399 v399) v1 ==== 95+0+2113 (secure 0 0 0) 0x7f4ba8005c40 con 0x7f4bb81192c0 2019-12-05T06:50:06.136 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:50:06.136 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":399,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":8537,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/1881703925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1881703925},{"type":"v1","addr":"172.21.15.42:6835","nonce":1881703925}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":399}],"filesystems":[{"mdsmap":{"epoch":398,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:49:10.408613+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":121,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12687},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12687":{"gid":12687,"name":"b","rank":0,"incarnation":395,"state":"up:active","state_seq":12,"addr":"172.21.15.40:6833/815316041","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":815316041},{"type":"v1","addr":"172.21.15.40:6833","nonce":815316041}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:50:06.136 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 399 2019-12-05T06:50:06.136 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:06.119+0000 7f4bbe12e700 1 -- 172.21.15.42:0/4002380539 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4b9c03ae70 msgr2=0x7f4b9c03d380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:06.137 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:06.119+0000 7f4bbe12e700 1 --2- 172.21.15.42:0/4002380539 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4b9c03ae70 0x7f4b9c03d380 crc :-1 s=READY pgs=1387 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:06.137 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:06.119+0000 7f4bbe12e700 1 -- 172.21.15.42:0/4002380539 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4bb81192c0 msgr2=0x7f4bb81188f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:06.137 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:06.119+0000 7f4bbe12e700 1 --2- 172.21.15.42:0/4002380539 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4bb81192c0 0x7f4bb81188f0 secure :-1 s=READY pgs=731 cs=0 l=1 rx=0x7f4ba8008790 tx=0x7f4ba8008ae0).stop 2019-12-05T06:50:06.137 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:06.119+0000 7f4bbe12e700 1 -- 172.21.15.42:0/4002380539 shutdown_connections 2019-12-05T06:50:06.137 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:06.119+0000 7f4bbe12e700 1 --2- 172.21.15.42:0/4002380539 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f4b9c03ae70 0x7f4b9c03d380 unknown :-1 s=CLOSED pgs=1387 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:06.137 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:06.119+0000 7f4bbe12e700 1 --2- 172.21.15.42:0/4002380539 >> v2:172.21.15.42:3300/0 conn(0x7f4bb807de10 0x7f4bb8117de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:06.137 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:06.119+0000 7f4bbe12e700 1 --2- 172.21.15.42:0/4002380539 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4bb81192c0 0x7f4bb81188f0 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:06.138 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:06.123+0000 7f4bbe12e700 1 -- 172.21.15.42:0/4002380539 shutdown_connections 2019-12-05T06:50:06.138 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:06.123+0000 7f4bbe12e700 1 -- 172.21.15.42:0/4002380539 wait complete. 2019-12-05T06:50:06.138 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:06.123+0000 7f4bbe12e700 1 -- 172.21.15.42:0/4002380539 >> 172.21.15.42:0/4002380539 conn(0x7f4bb80a7fd0 msgr2=0x7f4bb811c520 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:50:06.139 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:50:08.144 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:50:08.265 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:08.251+0000 7f226ffad700 1 Processor -- start 2019-12-05T06:50:08.266 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:08.251+0000 7f226ffad700 1 -- start start 2019-12-05T06:50:08.266 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:08.255+0000 7f226ffad700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f226809fae0 0x7f226814fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:50:08.266 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:08.255+0000 7f226ffad700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22680a0710 con 0x7f22680a7740 2019-12-05T06:50:08.266 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:08.255+0000 7f226ffad700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f226807ef20 con 0x7f226808df80 2019-12-05T06:50:08.267 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:08.255+0000 7f226ffad700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22680a0b00 con 0x7f226809fae0 2019-12-05T06:50:08.267 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:08.255+0000 7f226dd49700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f226808df80 0x7f226809f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:38308/0 (socket says 172.21.15.42:38308) 2019-12-05T06:50:08.267 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:08.255+0000 7f226ed4b700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f226809fae0 0x7f226814fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:08.267 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:08.255+0000 7f226dd49700 1 -- 172.21.15.42:0/3305340214 learned_addr learned my addr 172.21.15.42:0/3305340214 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:50:08.268 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:08.255+0000 7f226d548700 1 -- 172.21.15.42:0/3305340214 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 539677443 0 0) 0x7f22680a0710 con 0x7f22680a7740 2019-12-05T06:50:08.268 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:08.255+0000 7f226d548700 1 -- 172.21.15.42:0/3305340214 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22540030b0 con 0x7f22680a7740 2019-12-05T06:50:08.268 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:08.255+0000 7f226ed4b700 1 -- 172.21.15.42:0/3305340214 >> v1:172.21.15.42:6789/0 conn(0x7f226808df80 legacy=0x7f226809f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:08.268 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:08.255+0000 7f226d548700 1 -- 172.21.15.42:0/3305340214 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 559639664 0 0) 0x7f226807ef20 con 0x7f226808df80 2019-12-05T06:50:08.269 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:08.255+0000 7f226ed4b700 1 -- 172.21.15.42:0/3305340214 >> v1:172.21.15.40:6789/0 conn(0x7f22680a7740 legacy=0x7f226807e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:08.269 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:08.255+0000 7f226ed4b700 1 -- 172.21.15.42:0/3305340214 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22680a9ba0 con 0x7f226809fae0 2019-12-05T06:50:08.269 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:08.255+0000 7f226ed4b700 1 --2- 172.21.15.42:0/3305340214 >> v2:172.21.15.42:3300/0 conn(0x7f226809fae0 0x7f226814fff0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f2260008b10 tx=0x7f226000ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:08.270 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:08.255+0000 7f226ed4b700 1 --2- 172.21.15.42:0/3305340214 >> v2:172.21.15.42:3300/0 conn(0x7f226809fae0 0x7f226814fff0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f2260008b10 tx=0x7f226000ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:08.270 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:08.255+0000 7f226ed4b700 1 --2- 172.21.15.42:0/3305340214 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f226809fae0 0x7f226814fff0 secure :-1 s=READY pgs=3308 cs=0 l=1 rx=0x7f2260008b10 tx=0x7f226000ff70).ready entity=mon.1 client_cookie=e5c7ee56bac4e656 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:50:08.270 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:08.255+0000 7f226ed4b700 1 --2- 172.21.15.42:0/3305340214 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f226809fae0 0x7f226814fff0 secure :-1 s=THROTTLE_DONE pgs=3308 cs=0 l=1 rx=0x7f2260008b10 tx=0x7f226000ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:08.270 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:08.255+0000 7f226ed4b700 1 --2- 172.21.15.42:0/3305340214 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f226809fae0 0x7f226814fff0 secure :-1 s=THROTTLE_DONE pgs=3308 cs=0 l=1 rx=0x7f2260008b10 tx=0x7f226000ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:08.271 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:08.255+0000 7f226d548700 1 -- 172.21.15.42:0/3305340214 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2260010970 con 0x7f226809fae0 2019-12-05T06:50:08.271 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:08.255+0000 7f226ed4b700 1 --2- 172.21.15.42:0/3305340214 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f226809fae0 0x7f226814fff0 secure :-1 s=THROTTLE_DONE pgs=3308 cs=0 l=1 rx=0x7f2260008b10 tx=0x7f226000ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:08.271 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:08.255+0000 7f226d548700 1 -- 172.21.15.42:0/3305340214 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2260008b50 con 0x7f226809fae0 2019-12-05T06:50:08.271 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:08.255+0000 7f226d548700 1 -- 172.21.15.42:0/3305340214 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f22600110c0 con 0x7f226809fae0 2019-12-05T06:50:08.271 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:08.255+0000 7f226ffad700 1 -- 172.21.15.42:0/3305340214 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f226809fae0 msgr2=0x7f226814fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:08.272 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:08.255+0000 7f226ffad700 1 --2- 172.21.15.42:0/3305340214 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f226809fae0 0x7f226814fff0 secure :-1 s=READY pgs=3308 cs=0 l=1 rx=0x7f2260008b10 tx=0x7f226000ff70).stop 2019-12-05T06:50:08.272 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:08.255+0000 7f226ffad700 1 -- 172.21.15.42:0/3305340214 shutdown_connections 2019-12-05T06:50:08.272 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:08.259+0000 7f226ffad700 1 --2- 172.21.15.42:0/3305340214 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f226809fae0 0x7f226814fff0 unknown :-1 s=CLOSED pgs=3308 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:08.272 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:08.259+0000 7f226ffad700 1 -- 172.21.15.42:0/3305340214 shutdown_connections 2019-12-05T06:50:08.273 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:08.259+0000 7f226ffad700 1 -- 172.21.15.42:0/3305340214 wait complete. 2019-12-05T06:50:08.273 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:08.259+0000 7f226ffad700 1 -- 172.21.15.42:0/3305340214 >> 172.21.15.42:0/3305340214 conn(0x7f22680aabe0 msgr2=0x7f22680a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:50:08.278 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:08.259+0000 7f226ffad700 1 Processor -- start 2019-12-05T06:50:08.278 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:08.259+0000 7f226ffad700 1 -- start start 2019-12-05T06:50:08.278 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:08.259+0000 7f226ffad700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2268116500 0x7f2268145020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:50:08.278 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:08.259+0000 7f226ffad700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2268146400 0x7f2268147bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:50:08.279 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:08.259+0000 7f226ffad700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22681485f0 con 0x7f2268148120 2019-12-05T06:50:08.279 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:08.259+0000 7f226ffad700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2268144ae0 con 0x7f2268146400 2019-12-05T06:50:08.279 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:08.259+0000 7f226ffad700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2268144d80 con 0x7f2268116500 2019-12-05T06:50:08.279 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:08.263+0000 7f226e54a700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2268116500 0x7f2268145020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:08.279 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:08.263+0000 7f226dd49700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2268146400 0x7f2268147bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:08.280 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:08.263+0000 7f226e54a700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2268116500 0x7f2268145020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:38274/0 (socket says 172.21.15.42:38274) 2019-12-05T06:50:08.280 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:08.263+0000 7f226e54a700 1 -- 172.21.15.42:0/2735861797 learned_addr learned my addr 172.21.15.42:0/2735861797 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:50:08.280 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:08.263+0000 7f226e54a700 1 -- 172.21.15.42:0/2735861797 >> v2:172.21.15.40:3300/0 conn(0x7f2268146400 msgr2=0x7f2268147bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:08.280 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:08.263+0000 7f226e54a700 1 --2- 172.21.15.42:0/2735861797 >> v2:172.21.15.40:3300/0 conn(0x7f2268146400 0x7f2268147bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:08.281 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:08.263+0000 7f226e54a700 1 -- 172.21.15.42:0/2735861797 >> v1:172.21.15.40:6789/0 conn(0x7f2268148120 legacy=0x7f226807b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:08.281 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:08.263+0000 7f226e54a700 1 -- 172.21.15.42:0/2735861797 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2268146120 con 0x7f2268116500 2019-12-05T06:50:08.281 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:08.263+0000 7f226e54a700 1 --2- 172.21.15.42:0/2735861797 >> v2:172.21.15.42:3300/0 conn(0x7f2268116500 0x7f2268145020 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f2264000e90 tx=0x7f22640109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:08.281 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:08.263+0000 7f226e54a700 1 --2- 172.21.15.42:0/2735861797 >> v2:172.21.15.42:3300/0 conn(0x7f2268116500 0x7f2268145020 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f2264000e90 tx=0x7f22640109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:08.282 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:08.263+0000 7f226e54a700 1 --2- 172.21.15.42:0/2735861797 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2268116500 0x7f2268145020 secure :-1 s=READY pgs=3309 cs=0 l=1 rx=0x7f2264000e90 tx=0x7f22640109a0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:50:08.282 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:08.263+0000 7f226e54a700 1 --2- 172.21.15.42:0/2735861797 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2268116500 0x7f2268145020 secure :-1 s=THROTTLE_DONE pgs=3309 cs=0 l=1 rx=0x7f2264000e90 tx=0x7f22640109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:08.282 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:08.263+0000 7f226e54a700 1 --2- 172.21.15.42:0/2735861797 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2268116500 0x7f2268145020 secure :-1 s=THROTTLE_DONE pgs=3309 cs=0 l=1 rx=0x7f2264000e90 tx=0x7f22640109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:08.282 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:08.263+0000 7f225affd700 1 -- 172.21.15.42:0/2735861797 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f226400caa0 con 0x7f2268116500 2019-12-05T06:50:08.282 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:08.263+0000 7f226e54a700 1 --2- 172.21.15.42:0/2735861797 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2268116500 0x7f2268145020 secure :-1 s=THROTTLE_DONE pgs=3309 cs=0 l=1 rx=0x7f2264000e90 tx=0x7f22640109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:08.283 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:08.263+0000 7f225affd700 1 -- 172.21.15.42:0/2735861797 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f226400c6c0 con 0x7f2268116500 2019-12-05T06:50:08.283 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:08.263+0000 7f226ffad700 1 -- 172.21.15.42:0/2735861797 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f226807c120 con 0x7f2268116500 2019-12-05T06:50:08.283 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:08.263+0000 7f225affd700 1 -- 172.21.15.42:0/2735861797 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f226400d6a0 con 0x7f2268116500 2019-12-05T06:50:08.283 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:08.263+0000 7f226ffad700 1 -- 172.21.15.42:0/2735861797 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f226812ef80 con 0x7f2268116500 2019-12-05T06:50:08.284 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:08.263+0000 7f226e54a700 1 --2- 172.21.15.42:0/2735861797 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2268116500 0x7f2268145020 secure :-1 s=THROTTLE_DONE pgs=3309 cs=0 l=1 rx=0x7f2264000e90 tx=0x7f22640109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:08.284 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:08.263+0000 7f225affd700 1 -- 172.21.15.42:0/2735861797 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f226400d970 con 0x7f2268116500 2019-12-05T06:50:08.284 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:08.263+0000 7f226e54a700 1 --2- 172.21.15.42:0/2735861797 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2268116500 0x7f2268145020 secure :-1 s=THROTTLE_DONE pgs=3309 cs=0 l=1 rx=0x7f2264000e90 tx=0x7f22640109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:08.284 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:08.263+0000 7f225affd700 1 --2- 172.21.15.42:0/2735861797 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f225003aec0 0x7f225003d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:50:08.285 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:08.263+0000 7f225affd700 1 -- 172.21.15.42:0/2735861797 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 12223+0+0 (secure 0 0 0) 0x7f226404b350 con 0x7f2268116500 2019-12-05T06:50:08.285 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:08.267+0000 7f226dd49700 1 --2- 172.21.15.42:0/2735861797 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f225003aec0 0x7f225003d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:08.285 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:08.267+0000 7f226dd49700 1 --2- 172.21.15.42:0/2735861797 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f225003aec0 0x7f225003d3d0 crc :-1 s=READY pgs=1388 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:50:08.285 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:08.267+0000 7f226ffad700 1 -- 172.21.15.42:0/2735861797 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f226805d110 con 0x7f2268116500 2019-12-05T06:50:08.286 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:08.275+0000 7f226e54a700 1 --2- 172.21.15.42:0/2735861797 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2268116500 0x7f2268145020 secure :-1 s=THROTTLE_DONE pgs=3309 cs=0 l=1 rx=0x7f2264000e90 tx=0x7f22640109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:08.286 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:08.275+0000 7f225affd700 1 -- 172.21.15.42:0/2735861797 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f22640323d0 con 0x7f2268116500 2019-12-05T06:50:08.453 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:08.447+0000 7f226ffad700 1 -- 172.21.15.42:0/2735861797 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f226805b8c0 con 0x7f2268116500 2019-12-05T06:50:08.454 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:08.447+0000 7f226e54a700 1 --2- 172.21.15.42:0/2735861797 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2268116500 0x7f2268145020 secure :-1 s=THROTTLE_DONE pgs=3309 cs=0 l=1 rx=0x7f2264000e90 tx=0x7f22640109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:08.454 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:08.447+0000 7f225affd700 1 -- 172.21.15.42:0/2735861797 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 399 v399) v1 ==== 95+0+2113 (secure 0 0 0) 0x7f2264032630 con 0x7f2268116500 2019-12-05T06:50:08.454 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 399 2019-12-05T06:50:08.455 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:50:08.455 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":399,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":8537,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/1881703925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1881703925},{"type":"v1","addr":"172.21.15.42:6835","nonce":1881703925}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":399}],"filesystems":[{"mdsmap":{"epoch":398,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:49:10.408613+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":121,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12687},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12687":{"gid":12687,"name":"b","rank":0,"incarnation":395,"state":"up:active","state_seq":12,"addr":"172.21.15.40:6833/815316041","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":815316041},{"type":"v1","addr":"172.21.15.40:6833","nonce":815316041}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:50:08.470 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:08.447+0000 7f226ffad700 1 -- 172.21.15.42:0/2735861797 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f225003aec0 msgr2=0x7f225003d3d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:08.470 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:08.447+0000 7f226ffad700 1 --2- 172.21.15.42:0/2735861797 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f225003aec0 0x7f225003d3d0 crc :-1 s=READY pgs=1388 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:08.471 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:08.447+0000 7f226ffad700 1 -- 172.21.15.42:0/2735861797 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2268116500 msgr2=0x7f2268145020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:08.471 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:08.447+0000 7f226ffad700 1 --2- 172.21.15.42:0/2735861797 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2268116500 0x7f2268145020 secure :-1 s=READY pgs=3309 cs=0 l=1 rx=0x7f2264000e90 tx=0x7f22640109a0).stop 2019-12-05T06:50:08.471 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:08.447+0000 7f226ffad700 1 -- 172.21.15.42:0/2735861797 shutdown_connections 2019-12-05T06:50:08.471 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:08.447+0000 7f226ffad700 1 --2- 172.21.15.42:0/2735861797 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f225003aec0 0x7f225003d3d0 unknown :-1 s=CLOSED pgs=1388 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:08.472 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:08.451+0000 7f226ffad700 1 --2- 172.21.15.42:0/2735861797 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2268116500 0x7f2268145020 unknown :-1 s=CLOSED pgs=3309 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:08.472 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:08.451+0000 7f226ffad700 1 --2- 172.21.15.42:0/2735861797 >> v2:172.21.15.40:3300/0 conn(0x7f2268146400 0x7f2268147bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:08.472 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:08.451+0000 7f226ffad700 1 -- 172.21.15.42:0/2735861797 shutdown_connections 2019-12-05T06:50:08.472 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:08.451+0000 7f226ffad700 1 -- 172.21.15.42:0/2735861797 wait complete. 2019-12-05T06:50:08.473 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:08.451+0000 7f226ffad700 1 -- 172.21.15.42:0/2735861797 >> 172.21.15.42:0/2735861797 conn(0x7f22680a7740 msgr2=0x7f22680a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:50:08.473 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:50:10.475 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:50:10.582 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:10.571+0000 7f8b4c61e700 1 Processor -- start 2019-12-05T06:50:10.582 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:10.571+0000 7f8b4c61e700 1 -- start start 2019-12-05T06:50:10.583 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:10.571+0000 7f8b4c61e700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f8b440a7740 0x7f8b4408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:50:10.583 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:10.571+0000 7f8b4c61e700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8b4409fae0 0x7f8b4407eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:50:10.583 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:10.571+0000 7f8b4c61e700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b440a0710 con 0x7f8b4407e720 2019-12-05T06:50:10.583 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:10.571+0000 7f8b4c61e700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b440a0b00 con 0x7f8b440a7740 2019-12-05T06:50:10.584 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:10.571+0000 7f8b4c61e700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b4408e5b0 con 0x7f8b4409fae0 2019-12-05T06:50:10.584 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:10.571+0000 7f8b4abbb700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f8b440a7740 0x7f8b4408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:10.585 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:10.571+0000 7f8b4a3ba700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f8b4407e720 0x7f8b4409f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:38326/0 (socket says 172.21.15.42:38326) 2019-12-05T06:50:10.585 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:10.571+0000 7f8b4a3ba700 1 -- 172.21.15.42:0/98447666 learned_addr learned my addr 172.21.15.42:0/98447666 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:50:10.585 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:10.571+0000 7f8b4b3bc700 1 --2- 172.21.15.42:0/98447666 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8b4409fae0 0x7f8b4407eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:10.585 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:10.571+0000 7f8b4abbb700 1 -- 172.21.15.42:0/98447666 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8b4409fae0 msgr2=0x7f8b4407eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:50:10.585 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:10.571+0000 7f8b4abbb700 1 --2- 172.21.15.42:0/98447666 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8b4409fae0 0x7f8b4407eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:50:10.586 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:10.571+0000 7f8b4abbb700 1 -- 172.21.15.42:0/98447666 >> v1:172.21.15.42:6789/0 conn(0x7f8b4407e720 legacy=0x7f8b4409f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:10.586 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:10.571+0000 7f8b49bb9700 1 -- 172.21.15.42:0/98447666 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1982993154 0 0) 0x7f8b440a0710 con 0x7f8b4407e720 2019-12-05T06:50:10.586 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:10.571+0000 7f8b4abbb700 1 -- 172.21.15.42:0/98447666 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b4407f0a0 con 0x7f8b440a7740 2019-12-05T06:50:10.586 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:10.571+0000 7f8b4abbb700 1 --2- 172.21.15.42:0/98447666 >> v2:172.21.15.42:3300/0 conn(0x7f8b440a7740 0x7f8b4408df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f8b38008790 tx=0x7f8b3800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:10.587 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:10.571+0000 7f8b4abbb700 1 --2- 172.21.15.42:0/98447666 >> v2:172.21.15.42:3300/0 conn(0x7f8b440a7740 0x7f8b4408df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f8b38008790 tx=0x7f8b3800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:10.587 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:10.571+0000 7f8b4abbb700 1 --2- 172.21.15.42:0/98447666 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8b440a7740 0x7f8b4408df80 secure :-1 s=READY pgs=3311 cs=0 l=1 rx=0x7f8b38008790 tx=0x7f8b3800fc70).ready entity=mon.1 client_cookie=4ca0232898493867 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:50:10.587 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:10.571+0000 7f8b4abbb700 1 --2- 172.21.15.42:0/98447666 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8b440a7740 0x7f8b4408df80 secure :-1 s=THROTTLE_DONE pgs=3311 cs=0 l=1 rx=0x7f8b38008790 tx=0x7f8b3800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:10.587 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:10.571+0000 7f8b49bb9700 1 -- 172.21.15.42:0/98447666 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8b38008bb0 con 0x7f8b440a7740 2019-12-05T06:50:10.588 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:10.571+0000 7f8b4abbb700 1 --2- 172.21.15.42:0/98447666 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8b440a7740 0x7f8b4408df80 secure :-1 s=THROTTLE_DONE pgs=3311 cs=0 l=1 rx=0x7f8b38008790 tx=0x7f8b3800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:10.588 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:10.571+0000 7f8b4abbb700 1 --2- 172.21.15.42:0/98447666 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8b440a7740 0x7f8b4408df80 secure :-1 s=THROTTLE_DONE pgs=3311 cs=0 l=1 rx=0x7f8b38008790 tx=0x7f8b3800fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:10.588 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:10.571+0000 7f8b49bb9700 1 -- 172.21.15.42:0/98447666 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b380087d0 con 0x7f8b440a7740 2019-12-05T06:50:10.588 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:10.571+0000 7f8b49bb9700 1 -- 172.21.15.42:0/98447666 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8b38011860 con 0x7f8b440a7740 2019-12-05T06:50:10.589 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:10.571+0000 7f8b4c61e700 1 -- 172.21.15.42:0/98447666 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8b440a7740 msgr2=0x7f8b4408df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:10.589 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:10.571+0000 7f8b4c61e700 1 --2- 172.21.15.42:0/98447666 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8b440a7740 0x7f8b4408df80 secure :-1 s=READY pgs=3311 cs=0 l=1 rx=0x7f8b38008790 tx=0x7f8b3800fc70).stop 2019-12-05T06:50:10.589 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:10.575+0000 7f8b4c61e700 1 -- 172.21.15.42:0/98447666 shutdown_connections 2019-12-05T06:50:10.589 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:10.575+0000 7f8b4c61e700 1 --2- 172.21.15.42:0/98447666 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8b4409fae0 0x7f8b4407eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:50:10.590 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:10.575+0000 7f8b4c61e700 1 --2- 172.21.15.42:0/98447666 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8b440a7740 0x7f8b4408df80 unknown :-1 s=CLOSED pgs=3311 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:10.590 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:10.575+0000 7f8b4c61e700 1 -- 172.21.15.42:0/98447666 shutdown_connections 2019-12-05T06:50:10.590 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:10.575+0000 7f8b4c61e700 1 -- 172.21.15.42:0/98447666 wait complete. 2019-12-05T06:50:10.590 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:10.575+0000 7f8b4c61e700 1 -- 172.21.15.42:0/98447666 >> 172.21.15.42:0/98447666 conn(0x7f8b440aabe0 msgr2=0x7f8b440a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:50:10.597 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:10.575+0000 7f8b4c61e700 1 Processor -- start 2019-12-05T06:50:10.598 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:10.575+0000 7f8b4c61e700 1 -- start start 2019-12-05T06:50:10.598 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:10.579+0000 7f8b4c61e700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8b44124590 0x7f8b441249c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:50:10.598 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:10.579+0000 7f8b4c61e700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8b440722c0 0x7f8b44074700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:50:10.598 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:10.579+0000 7f8b4c61e700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b44124f10 con 0x7f8b440a5f10 2019-12-05T06:50:10.599 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:10.579+0000 7f8b4c61e700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b44119860 con 0x7f8b44124590 2019-12-05T06:50:10.599 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:10.579+0000 7f8b4c61e700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b44119b00 con 0x7f8b440722c0 2019-12-05T06:50:10.599 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:10.579+0000 7f8b4b3bc700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8b440722c0 0x7f8b44074700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:10.599 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:10.579+0000 7f8b4a3ba700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8b44124590 0x7f8b441249c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:10.600 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:10.579+0000 7f8b4a3ba700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8b44124590 0x7f8b441249c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.42:54342/0 (socket says 172.21.15.42:54342) 2019-12-05T06:50:10.600 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:10.579+0000 7f8b4b3bc700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8b440722c0 0x7f8b44074700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:38762/0 (socket says 172.21.15.42:38762) 2019-12-05T06:50:10.600 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:10.579+0000 7f8b4abbb700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f8b440a5f10 0x7f8b44122680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:47770/0 (socket says 172.21.15.42:47770) 2019-12-05T06:50:10.600 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:10.579+0000 7f8b4a3ba700 1 -- 172.21.15.42:0/2261728037 learned_addr learned my addr 172.21.15.42:0/2261728037 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:50:10.601 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:10.579+0000 7f8b377fe700 1 -- 172.21.15.42:0/2261728037 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1938177318 0 0) 0x7f8b44124f10 con 0x7f8b440a5f10 2019-12-05T06:50:10.601 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:10.579+0000 7f8b377fe700 1 -- 172.21.15.42:0/2261728037 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b2c0030b0 con 0x7f8b440a5f10 2019-12-05T06:50:10.601 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:10.579+0000 7f8b4b3bc700 1 -- 172.21.15.42:0/2261728037 >> v2:172.21.15.40:3300/0 conn(0x7f8b44124590 msgr2=0x7f8b441249c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:10.601 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:10.579+0000 7f8b4b3bc700 1 --2- 172.21.15.42:0/2261728037 >> v2:172.21.15.40:3300/0 conn(0x7f8b44124590 0x7f8b441249c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:10.601 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:10.579+0000 7f8b4b3bc700 1 -- 172.21.15.42:0/2261728037 >> v1:172.21.15.40:6789/0 conn(0x7f8b440a5f10 legacy=0x7f8b44122680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:10.602 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:10.579+0000 7f8b4b3bc700 1 -- 172.21.15.42:0/2261728037 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b44122e80 con 0x7f8b440722c0 2019-12-05T06:50:10.602 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:10.579+0000 7f8b4a3ba700 1 --2- 172.21.15.42:0/2261728037 >> v2:172.21.15.40:3300/0 conn(0x7f8b44124590 0x7f8b441249c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:50:10.602 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:10.579+0000 7f8b4b3bc700 1 --2- 172.21.15.42:0/2261728037 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8b440722c0 0x7f8b44074700 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f8b3c002f30 tx=0x7f8b3c010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:10.602 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:10.579+0000 7f8b4b3bc700 1 --2- 172.21.15.42:0/2261728037 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8b440722c0 0x7f8b44074700 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f8b3c002f30 tx=0x7f8b3c010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:10.603 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:10.579+0000 7f8b4b3bc700 1 --2- 172.21.15.42:0/2261728037 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8b440722c0 0x7f8b44074700 secure :-1 s=READY pgs=732 cs=0 l=1 rx=0x7f8b3c002f30 tx=0x7f8b3c010ac0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:50:10.603 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:10.579+0000 7f8b4b3bc700 1 --2- 172.21.15.42:0/2261728037 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8b440722c0 0x7f8b44074700 secure :-1 s=THROTTLE_DONE pgs=732 cs=0 l=1 rx=0x7f8b3c002f30 tx=0x7f8b3c010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:10.603 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:10.579+0000 7f8b377fe700 1 -- 172.21.15.42:0/2261728037 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8b3c012570 con 0x7f8b440722c0 2019-12-05T06:50:10.603 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:10.579+0000 7f8b4b3bc700 1 --2- 172.21.15.42:0/2261728037 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8b440722c0 0x7f8b44074700 secure :-1 s=THROTTLE_DONE pgs=732 cs=0 l=1 rx=0x7f8b3c002f30 tx=0x7f8b3c010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:10.604 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:10.579+0000 7f8b4b3bc700 1 --2- 172.21.15.42:0/2261728037 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8b440722c0 0x7f8b44074700 secure :-1 s=THROTTLE_DONE pgs=732 cs=0 l=1 rx=0x7f8b3c002f30 tx=0x7f8b3c010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:10.604 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:10.579+0000 7f8b377fe700 1 -- 172.21.15.42:0/2261728037 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b3c007150 con 0x7f8b440722c0 2019-12-05T06:50:10.604 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:10.579+0000 7f8b4c61e700 1 -- 172.21.15.42:0/2261728037 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b44074c50 con 0x7f8b440722c0 2019-12-05T06:50:10.604 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:10.579+0000 7f8b377fe700 1 -- 172.21.15.42:0/2261728037 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8b3c012b50 con 0x7f8b440722c0 2019-12-05T06:50:10.604 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:10.579+0000 7f8b4c61e700 1 -- 172.21.15.42:0/2261728037 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b44075200 con 0x7f8b440722c0 2019-12-05T06:50:10.605 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:10.579+0000 7f8b4b3bc700 1 --2- 172.21.15.42:0/2261728037 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8b440722c0 0x7f8b44074700 secure :-1 s=THROTTLE_DONE pgs=732 cs=0 l=1 rx=0x7f8b3c002f30 tx=0x7f8b3c010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:10.605 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:10.583+0000 7f8b377fe700 1 -- 172.21.15.42:0/2261728037 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f8b3c005540 con 0x7f8b440722c0 2019-12-05T06:50:10.606 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:10.583+0000 7f8b4b3bc700 1 --2- 172.21.15.42:0/2261728037 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8b440722c0 0x7f8b44074700 secure :-1 s=THROTTLE_DONE pgs=732 cs=0 l=1 rx=0x7f8b3c002f30 tx=0x7f8b3c010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:10.606 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:10.583+0000 7f8b377fe700 1 --2- 172.21.15.42:0/2261728037 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8b2c03b260 0x7f8b2c03d770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:50:10.606 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:10.583+0000 7f8b377fe700 1 -- 172.21.15.42:0/2261728037 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 12223+0+0 (secure 0 0 0) 0x7f8b3c04af40 con 0x7f8b440722c0 2019-12-05T06:50:10.606 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:10.583+0000 7f8b4abbb700 1 --2- 172.21.15.42:0/2261728037 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8b2c03b260 0x7f8b2c03d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:10.607 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:10.583+0000 7f8b4abbb700 1 --2- 172.21.15.42:0/2261728037 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8b2c03b260 0x7f8b2c03d770 crc :-1 s=READY pgs=1389 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:50:10.607 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:10.587+0000 7f8b4c61e700 1 -- 172.21.15.42:0/2261728037 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b4405d110 con 0x7f8b440722c0 2019-12-05T06:50:10.608 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:10.599+0000 7f8b4b3bc700 1 --2- 172.21.15.42:0/2261728037 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8b440722c0 0x7f8b44074700 secure :-1 s=THROTTLE_DONE pgs=732 cs=0 l=1 rx=0x7f8b3c002f30 tx=0x7f8b3c010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:10.608 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:10.599+0000 7f8b377fe700 1 -- 172.21.15.42:0/2261728037 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f8b44122e80 con 0x7f8b440722c0 2019-12-05T06:50:10.776 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:10.767+0000 7f8b4c61e700 1 -- 172.21.15.42:0/2261728037 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8b4405b8c0 con 0x7f8b440722c0 2019-12-05T06:50:10.777 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:10.771+0000 7f8b4b3bc700 1 --2- 172.21.15.42:0/2261728037 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8b440722c0 0x7f8b44074700 secure :-1 s=THROTTLE_DONE pgs=732 cs=0 l=1 rx=0x7f8b3c002f30 tx=0x7f8b3c010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:10.777 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:10.771+0000 7f8b377fe700 1 -- 172.21.15.42:0/2261728037 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 399 v399) v1 ==== 95+0+2113 (secure 0 0 0) 0x7f8b3c031f40 con 0x7f8b440722c0 2019-12-05T06:50:10.779 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:50:10.779 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":399,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":8537,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/1881703925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1881703925},{"type":"v1","addr":"172.21.15.42:6835","nonce":1881703925}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":399}],"filesystems":[{"mdsmap":{"epoch":398,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:49:10.408613+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":121,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12687},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12687":{"gid":12687,"name":"b","rank":0,"incarnation":395,"state":"up:active","state_seq":12,"addr":"172.21.15.40:6833/815316041","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":815316041},{"type":"v1","addr":"172.21.15.40:6833","nonce":815316041}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:50:10.780 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 399 2019-12-05T06:50:10.782 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:10.775+0000 7f8b4c61e700 1 -- 172.21.15.42:0/2261728037 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8b2c03b260 msgr2=0x7f8b2c03d770 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:10.782 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:10.775+0000 7f8b4c61e700 1 --2- 172.21.15.42:0/2261728037 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8b2c03b260 0x7f8b2c03d770 crc :-1 s=READY pgs=1389 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:10.783 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:10.775+0000 7f8b4c61e700 1 -- 172.21.15.42:0/2261728037 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8b440722c0 msgr2=0x7f8b44074700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:10.783 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:10.775+0000 7f8b4c61e700 1 --2- 172.21.15.42:0/2261728037 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8b440722c0 0x7f8b44074700 secure :-1 s=READY pgs=732 cs=0 l=1 rx=0x7f8b3c002f30 tx=0x7f8b3c010ac0).stop 2019-12-05T06:50:10.783 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:10.775+0000 7f8b4c61e700 1 -- 172.21.15.42:0/2261728037 shutdown_connections 2019-12-05T06:50:10.784 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:10.775+0000 7f8b4c61e700 1 --2- 172.21.15.42:0/2261728037 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8b2c03b260 0x7f8b2c03d770 unknown :-1 s=CLOSED pgs=1389 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:10.784 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:10.775+0000 7f8b4c61e700 1 --2- 172.21.15.42:0/2261728037 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8b440722c0 0x7f8b44074700 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:10.784 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:10.775+0000 7f8b4c61e700 1 --2- 172.21.15.42:0/2261728037 >> v2:172.21.15.40:3300/0 conn(0x7f8b44124590 0x7f8b441249c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:10.784 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:10.775+0000 7f8b4c61e700 1 -- 172.21.15.42:0/2261728037 shutdown_connections 2019-12-05T06:50:10.785 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:10.775+0000 7f8b4c61e700 1 -- 172.21.15.42:0/2261728037 wait complete. 2019-12-05T06:50:10.785 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:10.775+0000 7f8b4c61e700 1 -- 172.21.15.42:0/2261728037 >> 172.21.15.42:0/2261728037 conn(0x7f8b440a7740 msgr2=0x7f8b440a3720 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:50:10.810 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:50:12.812 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:50:12.930 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:12.919+0000 7fc66bf5a700 1 Processor -- start 2019-12-05T06:50:12.931 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:12.919+0000 7fc66bf5a700 1 -- start start 2019-12-05T06:50:12.931 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:12.919+0000 7fc66bf5a700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc6640a7740 0x7fc66408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:50:12.931 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:12.919+0000 7fc66bf5a700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc66407e720 0x7fc66407eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:50:12.931 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:12.919+0000 7fc66bf5a700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6640a0710 con 0x7fc66409f2e0 2019-12-05T06:50:12.931 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:12.919+0000 7fc66bf5a700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6640a0b00 con 0x7fc66407e720 2019-12-05T06:50:12.931 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:12.919+0000 7fc66bf5a700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc66408e5b0 con 0x7fc6640a7740 2019-12-05T06:50:12.932 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:12.919+0000 7fc66acf8700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fc66409f2e0 0x7fc66409f710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:38350/0 (socket says 172.21.15.42:38350) 2019-12-05T06:50:12.932 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:12.919+0000 7fc66acf8700 1 -- 172.21.15.42:0/4216563085 learned_addr learned my addr 172.21.15.42:0/4216563085 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:50:12.932 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:12.919+0000 7fc669cf6700 1 --2- 172.21.15.42:0/4216563085 >> v2:172.21.15.40:3300/0 conn(0x7fc66407e720 0x7fc66407eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:12.932 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:12.919+0000 7fc66a4f7700 1 --2- 172.21.15.42:0/4216563085 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc6640a7740 0x7fc66408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:12.932 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:12.919+0000 7fc6694f5700 1 -- 172.21.15.42:0/4216563085 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2756972461 0 0) 0x7fc6640a0710 con 0x7fc66409f2e0 2019-12-05T06:50:12.932 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:12.919+0000 7fc6694f5700 1 -- 172.21.15.42:0/4216563085 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6500030b0 con 0x7fc66409f2e0 2019-12-05T06:50:12.933 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:12.919+0000 7fc669cf6700 1 -- 172.21.15.42:0/4216563085 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc6640a7740 msgr2=0x7fc66408df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:50:12.933 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:12.919+0000 7fc669cf6700 1 --2- 172.21.15.42:0/4216563085 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc6640a7740 0x7fc66408df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:50:12.933 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:12.919+0000 7fc669cf6700 1 -- 172.21.15.42:0/4216563085 >> v1:172.21.15.42:6789/0 conn(0x7fc66409f2e0 legacy=0x7fc66409f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:12.933 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:12.919+0000 7fc669cf6700 1 -- 172.21.15.42:0/4216563085 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc66407f0a0 con 0x7fc66407e720 2019-12-05T06:50:12.933 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:12.923+0000 7fc669cf6700 1 --2- 172.21.15.42:0/4216563085 >> v2:172.21.15.40:3300/0 conn(0x7fc66407e720 0x7fc66407eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fc654008790 tx=0x7fc65400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:12.933 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:12.923+0000 7fc669cf6700 1 --2- 172.21.15.42:0/4216563085 >> v2:172.21.15.40:3300/0 conn(0x7fc66407e720 0x7fc66407eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fc654008790 tx=0x7fc65400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:12.934 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:12.923+0000 7fc669cf6700 1 --2- 172.21.15.42:0/4216563085 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc66407e720 0x7fc66407eb50 secure :-1 s=READY pgs=2509 cs=0 l=1 rx=0x7fc654008790 tx=0x7fc65400fc70).ready entity=mon.0 client_cookie=c5efbcad64dc6c87 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:50:12.934 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:12.923+0000 7fc669cf6700 1 --2- 172.21.15.42:0/4216563085 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc66407e720 0x7fc66407eb50 secure :-1 s=THROTTLE_DONE pgs=2509 cs=0 l=1 rx=0x7fc654008790 tx=0x7fc65400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:12.934 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:12.923+0000 7fc6694f5700 1 -- 172.21.15.42:0/4216563085 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc6540112f0 con 0x7fc66407e720 2019-12-05T06:50:12.934 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:12.923+0000 7fc669cf6700 1 --2- 172.21.15.42:0/4216563085 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc66407e720 0x7fc66407eb50 secure :-1 s=THROTTLE_DONE pgs=2509 cs=0 l=1 rx=0x7fc654008790 tx=0x7fc65400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:12.934 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:12.923+0000 7fc669cf6700 1 --2- 172.21.15.42:0/4216563085 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc66407e720 0x7fc66407eb50 secure :-1 s=THROTTLE_DONE pgs=2509 cs=0 l=1 rx=0x7fc654008790 tx=0x7fc65400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:12.934 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:12.923+0000 7fc6694f5700 1 -- 172.21.15.42:0/4216563085 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc654008ad0 con 0x7fc66407e720 2019-12-05T06:50:12.935 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:12.923+0000 7fc6694f5700 1 -- 172.21.15.42:0/4216563085 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc6540118d0 con 0x7fc66407e720 2019-12-05T06:50:12.935 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:12.923+0000 7fc66bf5a700 1 -- 172.21.15.42:0/4216563085 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc66407e720 msgr2=0x7fc66407eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:12.935 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:12.923+0000 7fc66bf5a700 1 --2- 172.21.15.42:0/4216563085 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc66407e720 0x7fc66407eb50 secure :-1 s=READY pgs=2509 cs=0 l=1 rx=0x7fc654008790 tx=0x7fc65400fc70).stop 2019-12-05T06:50:12.935 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:12.923+0000 7fc66bf5a700 1 -- 172.21.15.42:0/4216563085 shutdown_connections 2019-12-05T06:50:12.935 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:12.923+0000 7fc66bf5a700 1 --2- 172.21.15.42:0/4216563085 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc6640a7740 0x7fc66408df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:50:12.935 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:12.923+0000 7fc66bf5a700 1 --2- 172.21.15.42:0/4216563085 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc66407e720 0x7fc66407eb50 unknown :-1 s=CLOSED pgs=2509 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:12.935 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:12.923+0000 7fc66bf5a700 1 -- 172.21.15.42:0/4216563085 shutdown_connections 2019-12-05T06:50:12.936 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:12.923+0000 7fc66bf5a700 1 -- 172.21.15.42:0/4216563085 wait complete. 2019-12-05T06:50:12.936 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:12.923+0000 7fc66bf5a700 1 -- 172.21.15.42:0/4216563085 >> 172.21.15.42:0/4216563085 conn(0x7fc6640aabe0 msgr2=0x7fc6640a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:50:12.939 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:12.923+0000 7fc66bf5a700 1 Processor -- start 2019-12-05T06:50:12.939 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:12.923+0000 7fc66bf5a700 1 -- start start 2019-12-05T06:50:12.940 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:12.927+0000 7fc66bf5a700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc6641161c0 0x7fc664133480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:50:12.940 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:12.927+0000 7fc66bf5a700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc66411a760 con 0x7fc66411a330 2019-12-05T06:50:12.940 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:12.927+0000 7fc66bf5a700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc664124e30 con 0x7fc6641339d0 2019-12-05T06:50:12.940 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:12.927+0000 7fc66bf5a700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc66411aa70 con 0x7fc6641161c0 2019-12-05T06:50:12.940 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:12.927+0000 7fc669cf6700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc6641161c0 0x7fc664133480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:12.941 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:12.927+0000 7fc669cf6700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc6641161c0 0x7fc664133480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:38316/0 (socket says 172.21.15.42:38316) 2019-12-05T06:50:12.941 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:12.927+0000 7fc669cf6700 1 -- 172.21.15.42:0/231651254 learned_addr learned my addr 172.21.15.42:0/231651254 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:50:12.941 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:12.927+0000 7fc65affd700 1 -- 172.21.15.42:0/231651254 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 52380977 0 0) 0x7fc66411a760 con 0x7fc66411a330 2019-12-05T06:50:12.941 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:12.927+0000 7fc65affd700 1 -- 172.21.15.42:0/231651254 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc64c0030b0 con 0x7fc66411a330 2019-12-05T06:50:12.941 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:12.927+0000 7fc669cf6700 1 -- 172.21.15.42:0/231651254 >> v1:172.21.15.42:6789/0 conn(0x7fc6641339d0 legacy=0x7fc664077300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:12.942 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:12.927+0000 7fc669cf6700 1 -- 172.21.15.42:0/231651254 >> v1:172.21.15.40:6789/0 conn(0x7fc66411a330 legacy=0x7fc6640a1ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:12.942 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:12.927+0000 7fc669cf6700 1 -- 172.21.15.42:0/231651254 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6640a1cb0 con 0x7fc6641161c0 2019-12-05T06:50:12.942 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:12.927+0000 7fc669cf6700 1 --2- 172.21.15.42:0/231651254 >> v2:172.21.15.42:3300/0 conn(0x7fc6641161c0 0x7fc664133480 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fc654008790 tx=0x7fc654011df0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:12.942 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:12.927+0000 7fc669cf6700 1 --2- 172.21.15.42:0/231651254 >> v2:172.21.15.42:3300/0 conn(0x7fc6641161c0 0x7fc664133480 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fc654008790 tx=0x7fc654011df0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:12.942 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:12.927+0000 7fc669cf6700 1 --2- 172.21.15.42:0/231651254 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc6641161c0 0x7fc664133480 secure :-1 s=READY pgs=3314 cs=0 l=1 rx=0x7fc654008790 tx=0x7fc654011df0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:50:12.942 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:12.927+0000 7fc669cf6700 1 --2- 172.21.15.42:0/231651254 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc6641161c0 0x7fc664133480 secure :-1 s=THROTTLE_DONE pgs=3314 cs=0 l=1 rx=0x7fc654008790 tx=0x7fc654011df0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:12.943 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:12.927+0000 7fc65affd700 1 -- 172.21.15.42:0/231651254 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc654007150 con 0x7fc6641161c0 2019-12-05T06:50:12.943 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:12.927+0000 7fc669cf6700 1 --2- 172.21.15.42:0/231651254 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc6641161c0 0x7fc664133480 secure :-1 s=THROTTLE_DONE pgs=3314 cs=0 l=1 rx=0x7fc654008790 tx=0x7fc654011df0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:12.943 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:12.927+0000 7fc65affd700 1 -- 172.21.15.42:0/231651254 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6540114f0 con 0x7fc6641161c0 2019-12-05T06:50:12.943 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:12.927+0000 7fc669cf6700 1 --2- 172.21.15.42:0/231651254 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc6641161c0 0x7fc664133480 secure :-1 s=THROTTLE_DONE pgs=3314 cs=0 l=1 rx=0x7fc654008790 tx=0x7fc654011df0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:12.943 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:12.927+0000 7fc66bf5a700 1 -- 172.21.15.42:0/231651254 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc66411c270 con 0x7fc6641161c0 2019-12-05T06:50:12.943 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:12.927+0000 7fc65affd700 1 -- 172.21.15.42:0/231651254 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc654008cc0 con 0x7fc6641161c0 2019-12-05T06:50:12.944 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:12.927+0000 7fc66bf5a700 1 -- 172.21.15.42:0/231651254 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc66411af70 con 0x7fc6641161c0 2019-12-05T06:50:12.944 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:12.927+0000 7fc669cf6700 1 --2- 172.21.15.42:0/231651254 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc6641161c0 0x7fc664133480 secure :-1 s=THROTTLE_DONE pgs=3314 cs=0 l=1 rx=0x7fc654008790 tx=0x7fc654011df0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:12.944 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:12.931+0000 7fc65affd700 1 -- 172.21.15.42:0/231651254 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fc654005540 con 0x7fc6641161c0 2019-12-05T06:50:12.944 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:12.931+0000 7fc669cf6700 1 --2- 172.21.15.42:0/231651254 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc6641161c0 0x7fc664133480 secure :-1 s=THROTTLE_DONE pgs=3314 cs=0 l=1 rx=0x7fc654008790 tx=0x7fc654011df0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:12.944 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:12.931+0000 7fc65affd700 1 --2- 172.21.15.42:0/231651254 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc64c03b210 0x7fc64c03d720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:50:12.945 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:12.931+0000 7fc65affd700 1 -- 172.21.15.42:0/231651254 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 12223+0+0 (secure 0 0 0) 0x7fc65404b890 con 0x7fc6641161c0 2019-12-05T06:50:12.945 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:12.931+0000 7fc66a4f7700 1 --2- 172.21.15.42:0/231651254 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc64c03b210 0x7fc64c03d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:12.945 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:12.931+0000 7fc66a4f7700 1 --2- 172.21.15.42:0/231651254 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc64c03b210 0x7fc64c03d720 crc :-1 s=READY pgs=1390 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:50:12.945 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:12.935+0000 7fc66bf5a700 1 -- 172.21.15.42:0/231651254 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc66405d110 con 0x7fc6641161c0 2019-12-05T06:50:12.948 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:12.943+0000 7fc669cf6700 1 --2- 172.21.15.42:0/231651254 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc6641161c0 0x7fc664133480 secure :-1 s=THROTTLE_DONE pgs=3314 cs=0 l=1 rx=0x7fc654008790 tx=0x7fc654011df0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:12.949 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:12.943+0000 7fc65affd700 1 -- 172.21.15.42:0/231651254 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fc6540328f0 con 0x7fc6641161c0 2019-12-05T06:50:13.118 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:13.111+0000 7fc66bf5a700 1 -- 172.21.15.42:0/231651254 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc66405b8c0 con 0x7fc6641161c0 2019-12-05T06:50:13.118 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:13.111+0000 7fc669cf6700 1 --2- 172.21.15.42:0/231651254 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc6641161c0 0x7fc664133480 secure :-1 s=THROTTLE_DONE pgs=3314 cs=0 l=1 rx=0x7fc654008790 tx=0x7fc654011df0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:13.119 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:13.111+0000 7fc65affd700 1 -- 172.21.15.42:0/231651254 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 399 v399) v1 ==== 95+0+2113 (secure 0 0 0) 0x7fc654032b50 con 0x7fc6641161c0 2019-12-05T06:50:13.119 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 399 2019-12-05T06:50:13.119 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:50:13.119 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":399,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":8537,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.42:6835/1881703925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1881703925},{"type":"v1","addr":"172.21.15.42:6835","nonce":1881703925}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":399}],"filesystems":[{"mdsmap":{"epoch":398,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:49:10.408613+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":121,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12687},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12687":{"gid":12687,"name":"b","rank":0,"incarnation":395,"state":"up:active","state_seq":12,"addr":"172.21.15.40:6833/815316041","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":815316041},{"type":"v1","addr":"172.21.15.40:6833","nonce":815316041}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:50:13.127 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:13.115+0000 7fc66bf5a700 1 -- 172.21.15.42:0/231651254 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc64c03b210 msgr2=0x7fc64c03d720 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:13.127 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:13.115+0000 7fc66bf5a700 1 --2- 172.21.15.42:0/231651254 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc64c03b210 0x7fc64c03d720 crc :-1 s=READY pgs=1390 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:13.127 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:13.115+0000 7fc66bf5a700 1 -- 172.21.15.42:0/231651254 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc6641161c0 msgr2=0x7fc664133480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:13.127 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:13.115+0000 7fc66bf5a700 1 --2- 172.21.15.42:0/231651254 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc6641161c0 0x7fc664133480 secure :-1 s=READY pgs=3314 cs=0 l=1 rx=0x7fc654008790 tx=0x7fc654011df0).stop 2019-12-05T06:50:13.127 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:13.115+0000 7fc66bf5a700 1 -- 172.21.15.42:0/231651254 shutdown_connections 2019-12-05T06:50:13.127 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:13.115+0000 7fc66bf5a700 1 --2- 172.21.15.42:0/231651254 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc64c03b210 0x7fc64c03d720 unknown :-1 s=CLOSED pgs=1390 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:13.128 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:13.115+0000 7fc66bf5a700 1 --2- 172.21.15.42:0/231651254 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc6641161c0 0x7fc664133480 unknown :-1 s=CLOSED pgs=3314 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:13.128 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:13.115+0000 7fc66bf5a700 1 -- 172.21.15.42:0/231651254 shutdown_connections 2019-12-05T06:50:13.128 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:13.115+0000 7fc66bf5a700 1 -- 172.21.15.42:0/231651254 wait complete. 2019-12-05T06:50:13.128 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:13.115+0000 7fc66bf5a700 1 -- 172.21.15.42:0/231651254 >> 172.21.15.42:0/231651254 conn(0x7fc66407e670 msgr2=0x7fc6640a5c90 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:50:13.137 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:50:15.138 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:50:15.245 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.231+0000 7feead6e6700 1 Processor -- start 2019-12-05T06:50:15.245 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.231+0000 7feead6e6700 1 -- start start 2019-12-05T06:50:15.246 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.235+0000 7feead6e6700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7feea80a7740 0x7feea808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:50:15.246 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.235+0000 7feead6e6700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7feea807e720 0x7feea807eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:50:15.246 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.235+0000 7feead6e6700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7feea809f2e0 0x7feea809f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:50:15.246 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.235+0000 7feead6e6700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feea80a0b00 con 0x7feea807e720 2019-12-05T06:50:15.246 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.235+0000 7feead6e6700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feea808e5b0 con 0x7feea809f2e0 2019-12-05T06:50:15.246 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.235+0000 7feead6e6700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feea808e850 con 0x7feea80a7740 2019-12-05T06:50:15.246 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.235+0000 7feea7fff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7feea809f2e0 0x7feea809f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:15.247 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.235+0000 7feea77fe700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7feea80a7740 0x7feea808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:15.247 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.235+0000 7feea7fff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7feea809f2e0 0x7feea809f710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:38328/0 (socket says 172.21.15.42:38328) 2019-12-05T06:50:15.247 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.235+0000 7feea6ffd700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7feea807e720 0x7feea807eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:15.247 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.235+0000 7feea7fff700 1 -- 172.21.15.42:0/692127037 learned_addr learned my addr 172.21.15.42:0/692127037 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:50:15.247 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.235+0000 7feea7fff700 1 -- 172.21.15.42:0/692127037 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7feea80a7740 msgr2=0x7feea808df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:50:15.247 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.235+0000 7feea7fff700 1 --2- 172.21.15.42:0/692127037 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7feea80a7740 0x7feea808df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:50:15.248 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.235+0000 7feea7fff700 1 -- 172.21.15.42:0/692127037 >> v2:172.21.15.40:3300/0 conn(0x7feea807e720 msgr2=0x7feea807eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:50:15.248 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.235+0000 7feea7fff700 1 --2- 172.21.15.42:0/692127037 >> v2:172.21.15.40:3300/0 conn(0x7feea807e720 0x7feea807eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:50:15.248 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.235+0000 7feea7fff700 1 -- 172.21.15.42:0/692127037 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feea807f0a0 con 0x7feea809f2e0 2019-12-05T06:50:15.248 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.235+0000 7feea77fe700 1 --2- 172.21.15.42:0/692127037 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7feea80a7740 0x7feea808df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:50:15.248 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.235+0000 7feea7fff700 1 --2- 172.21.15.42:0/692127037 >> v2:172.21.15.42:3300/0 conn(0x7feea809f2e0 0x7feea809f710 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7feea808e770 tx=0x7feea000fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:15.249 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.235+0000 7feea7fff700 1 --2- 172.21.15.42:0/692127037 >> v2:172.21.15.42:3300/0 conn(0x7feea809f2e0 0x7feea809f710 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7feea808e770 tx=0x7feea000fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:15.249 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.235+0000 7feea7fff700 1 --2- 172.21.15.42:0/692127037 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feea809f2e0 0x7feea809f710 secure :-1 s=READY pgs=3315 cs=0 l=1 rx=0x7feea808e770 tx=0x7feea000fdb0).ready entity=mon.1 client_cookie=e72a270d6431823 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:50:15.249 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.235+0000 7feea7fff700 1 --2- 172.21.15.42:0/692127037 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feea809f2e0 0x7feea809f710 secure :-1 s=THROTTLE_DONE pgs=3315 cs=0 l=1 rx=0x7feea808e770 tx=0x7feea000fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:15.249 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.235+0000 7feea67fc700 1 -- 172.21.15.42:0/692127037 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7feea0008c90 con 0x7feea809f2e0 2019-12-05T06:50:15.249 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.235+0000 7feea7fff700 1 --2- 172.21.15.42:0/692127037 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feea809f2e0 0x7feea809f710 secure :-1 s=THROTTLE_DONE pgs=3315 cs=0 l=1 rx=0x7feea808e770 tx=0x7feea000fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:15.249 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.235+0000 7feea7fff700 1 --2- 172.21.15.42:0/692127037 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feea809f2e0 0x7feea809f710 secure :-1 s=THROTTLE_DONE pgs=3315 cs=0 l=1 rx=0x7feea808e770 tx=0x7feea000fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:15.249 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.235+0000 7feea67fc700 1 -- 172.21.15.42:0/692127037 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feea00088b0 con 0x7feea809f2e0 2019-12-05T06:50:15.250 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.235+0000 7feea67fc700 1 -- 172.21.15.42:0/692127037 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7feea0011b50 con 0x7feea809f2e0 2019-12-05T06:50:15.250 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.235+0000 7feead6e6700 1 -- 172.21.15.42:0/692127037 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feea809f2e0 msgr2=0x7feea809f710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:15.250 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.235+0000 7feead6e6700 1 --2- 172.21.15.42:0/692127037 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feea809f2e0 0x7feea809f710 secure :-1 s=READY pgs=3315 cs=0 l=1 rx=0x7feea808e770 tx=0x7feea000fdb0).stop 2019-12-05T06:50:15.250 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.239+0000 7feead6e6700 1 -- 172.21.15.42:0/692127037 shutdown_connections 2019-12-05T06:50:15.250 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.239+0000 7feead6e6700 1 --2- 172.21.15.42:0/692127037 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feea809f2e0 0x7feea809f710 unknown :-1 s=CLOSED pgs=3315 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:15.250 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.239+0000 7feead6e6700 1 --2- 172.21.15.42:0/692127037 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7feea80a7740 0x7feea808df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:50:15.251 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.239+0000 7feead6e6700 1 --2- 172.21.15.42:0/692127037 >> v2:172.21.15.40:3300/0 conn(0x7feea807e720 0x7feea807eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:50:15.251 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.239+0000 7feead6e6700 1 -- 172.21.15.42:0/692127037 shutdown_connections 2019-12-05T06:50:15.251 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.239+0000 7feead6e6700 1 -- 172.21.15.42:0/692127037 wait complete. 2019-12-05T06:50:15.251 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.239+0000 7feead6e6700 1 -- 172.21.15.42:0/692127037 >> 172.21.15.42:0/692127037 conn(0x7feea80aabe0 msgr2=0x7feea80a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:50:15.255 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.239+0000 7feead6e6700 1 Processor -- start 2019-12-05T06:50:15.255 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.239+0000 7feead6e6700 1 -- start start 2019-12-05T06:50:15.255 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.243+0000 7feead6e6700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7feea807d970 0x7feea807dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:50:15.255 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.243+0000 7feead6e6700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7feea8122cd0 0x7feea8123100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:50:15.256 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.243+0000 7feead6e6700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feea811a8a0 con 0x7feea81245a0 2019-12-05T06:50:15.256 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.243+0000 7feead6e6700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feea811acc0 con 0x7feea807d970 2019-12-05T06:50:15.256 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.243+0000 7feead6e6700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feea811af60 con 0x7feea8122cd0 2019-12-05T06:50:15.256 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.243+0000 7feea6ffd700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7feea807d970 0x7feea807dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:15.256 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.243+0000 7feea6ffd700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7feea807d970 0x7feea807dda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:38332/0 (socket says 172.21.15.42:38332) 2019-12-05T06:50:15.256 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.243+0000 7feea6ffd700 1 -- 172.21.15.42:0/2129940404 learned_addr learned my addr 172.21.15.42:0/2129940404 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:50:15.257 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.243+0000 7feea7fff700 1 --2- 172.21.15.42:0/2129940404 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7feea8122cd0 0x7feea8123100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:15.257 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.243+0000 7feea6ffd700 1 -- 172.21.15.42:0/2129940404 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7feea8122cd0 msgr2=0x7feea8123100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:15.257 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.243+0000 7feea6ffd700 1 --2- 172.21.15.42:0/2129940404 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7feea8122cd0 0x7feea8123100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:15.257 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.243+0000 7feea6ffd700 1 -- 172.21.15.42:0/2129940404 >> v1:172.21.15.40:6789/0 conn(0x7feea81245a0 legacy=0x7feea8122140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:15.257 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.243+0000 7fee9ffff700 1 -- 172.21.15.42:0/2129940404 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2231597013 0 0) 0x7feea811a8a0 con 0x7feea81245a0 2019-12-05T06:50:15.257 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.243+0000 7feea6ffd700 1 -- 172.21.15.42:0/2129940404 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feea8123650 con 0x7feea807d970 2019-12-05T06:50:15.257 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.243+0000 7feea6ffd700 1 --2- 172.21.15.42:0/2129940404 >> v2:172.21.15.42:3300/0 conn(0x7feea807d970 0x7feea807dda0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7feea8123d20 tx=0x7fee90010bf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:15.258 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.243+0000 7feea6ffd700 1 --2- 172.21.15.42:0/2129940404 >> v2:172.21.15.42:3300/0 conn(0x7feea807d970 0x7feea807dda0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7feea8123d20 tx=0x7fee90010bf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:15.258 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.243+0000 7feea6ffd700 1 --2- 172.21.15.42:0/2129940404 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feea807d970 0x7feea807dda0 secure :-1 s=READY pgs=3316 cs=0 l=1 rx=0x7feea8123d20 tx=0x7fee90010bf0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:50:15.258 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.243+0000 7feea6ffd700 1 --2- 172.21.15.42:0/2129940404 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feea807d970 0x7feea807dda0 secure :-1 s=THROTTLE_DONE pgs=3316 cs=0 l=1 rx=0x7feea8123d20 tx=0x7fee90010bf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:15.258 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.243+0000 7feea6ffd700 1 --2- 172.21.15.42:0/2129940404 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feea807d970 0x7feea807dda0 secure :-1 s=THROTTLE_DONE pgs=3316 cs=0 l=1 rx=0x7feea8123d20 tx=0x7fee90010bf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:15.258 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.243+0000 7fee9ffff700 1 -- 172.21.15.42:0/2129940404 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fee90013060 con 0x7feea807d970 2019-12-05T06:50:15.258 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.243+0000 7feea6ffd700 1 --2- 172.21.15.42:0/2129940404 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feea807d970 0x7feea807dda0 secure :-1 s=THROTTLE_DONE pgs=3316 cs=0 l=1 rx=0x7feea8123d20 tx=0x7fee90010bf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:15.258 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.243+0000 7fee9ffff700 1 -- 172.21.15.42:0/2129940404 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee90012070 con 0x7feea807d970 2019-12-05T06:50:15.259 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.243+0000 7feead6e6700 1 -- 172.21.15.42:0/2129940404 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feea808eb70 con 0x7feea807d970 2019-12-05T06:50:15.259 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.243+0000 7fee9ffff700 1 -- 172.21.15.42:0/2129940404 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fee90005c30 con 0x7feea807d970 2019-12-05T06:50:15.259 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.243+0000 7feead6e6700 1 -- 172.21.15.42:0/2129940404 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feea808eed0 con 0x7feea807d970 2019-12-05T06:50:15.259 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.243+0000 7feea6ffd700 1 --2- 172.21.15.42:0/2129940404 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feea807d970 0x7feea807dda0 secure :-1 s=THROTTLE_DONE pgs=3316 cs=0 l=1 rx=0x7feea8123d20 tx=0x7fee90010bf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:15.259 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.243+0000 7fee9ffff700 1 -- 172.21.15.42:0/2129940404 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fee90007180 con 0x7feea807d970 2019-12-05T06:50:15.260 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.243+0000 7feea6ffd700 1 --2- 172.21.15.42:0/2129940404 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feea807d970 0x7feea807dda0 secure :-1 s=THROTTLE_DONE pgs=3316 cs=0 l=1 rx=0x7feea8123d20 tx=0x7fee90010bf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:15.260 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.243+0000 7fee9ffff700 1 --2- 172.21.15.42:0/2129940404 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fee8c03ac90 0x7fee8c03d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:50:15.260 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.247+0000 7fee9ffff700 1 -- 172.21.15.42:0/2129940404 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 12223+0+0 (secure 0 0 0) 0x7fee9004add0 con 0x7feea807d970 2019-12-05T06:50:15.260 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.247+0000 7feea77fe700 1 --2- 172.21.15.42:0/2129940404 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fee8c03ac90 0x7fee8c03d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:15.260 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.247+0000 7feea77fe700 1 --2- 172.21.15.42:0/2129940404 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fee8c03ac90 0x7fee8c03d1a0 crc :-1 s=READY pgs=1391 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:50:15.260 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.247+0000 7feead6e6700 1 -- 172.21.15.42:0/2129940404 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feea805d110 con 0x7feea807d970 2019-12-05T06:50:15.261 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.251+0000 7feea6ffd700 1 --2- 172.21.15.42:0/2129940404 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feea807d970 0x7feea807dda0 secure :-1 s=THROTTLE_DONE pgs=3316 cs=0 l=1 rx=0x7feea8123d20 tx=0x7fee90010bf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:15.261 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.251+0000 7fee9ffff700 1 -- 172.21.15.42:0/2129940404 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fee90031e50 con 0x7feea807d970 2019-12-05T06:50:15.426 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.419+0000 7feead6e6700 1 -- 172.21.15.42:0/2129940404 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7feea805b8c0 con 0x7feea807d970 2019-12-05T06:50:15.431 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.423+0000 7feea6ffd700 1 --2- 172.21.15.42:0/2129940404 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feea807d970 0x7feea807dda0 secure :-1 s=THROTTLE_DONE pgs=3316 cs=0 l=1 rx=0x7feea8123d20 tx=0x7fee90010bf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:15.431 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.423+0000 7fee9ffff700 1 -- 172.21.15.42:0/2129940404 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 400 v400) v1 ==== 95+0+1775 (secure 0 0 0) 0x7fee900320b0 con 0x7feea807d970 2019-12-05T06:50:15.433 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 400 2019-12-05T06:50:15.433 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:50:15.434 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":400,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":400,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:50:15.404443+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":122,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8537},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8537":{"gid":8537,"name":"a","rank":0,"incarnation":400,"state":"up:replay","state_seq":2,"addr":"172.21.15.42:6835/1881703925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1881703925},{"type":"v1","addr":"172.21.15.42:6835","nonce":1881703925}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:50:15.437 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.431+0000 7feead6e6700 1 -- 172.21.15.42:0/2129940404 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fee8c03ac90 msgr2=0x7fee8c03d1a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:15.437 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.431+0000 7feead6e6700 1 --2- 172.21.15.42:0/2129940404 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fee8c03ac90 0x7fee8c03d1a0 crc :-1 s=READY pgs=1391 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:15.437 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.431+0000 7feead6e6700 1 -- 172.21.15.42:0/2129940404 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feea807d970 msgr2=0x7feea807dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:15.437 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.431+0000 7feead6e6700 1 --2- 172.21.15.42:0/2129940404 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feea807d970 0x7feea807dda0 secure :-1 s=READY pgs=3316 cs=0 l=1 rx=0x7feea8123d20 tx=0x7fee90010bf0).stop 2019-12-05T06:50:15.437 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.431+0000 7feead6e6700 1 -- 172.21.15.42:0/2129940404 shutdown_connections 2019-12-05T06:50:15.438 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.431+0000 7feead6e6700 1 --2- 172.21.15.42:0/2129940404 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fee8c03ac90 0x7fee8c03d1a0 unknown :-1 s=CLOSED pgs=1391 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:15.438 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.431+0000 7feead6e6700 1 --2- 172.21.15.42:0/2129940404 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7feea8122cd0 0x7feea8123100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:15.438 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.431+0000 7feead6e6700 1 --2- 172.21.15.42:0/2129940404 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feea807d970 0x7feea807dda0 unknown :-1 s=CLOSED pgs=3316 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:15.438 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.431+0000 7feead6e6700 1 -- 172.21.15.42:0/2129940404 shutdown_connections 2019-12-05T06:50:15.438 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.431+0000 7feead6e6700 1 -- 172.21.15.42:0/2129940404 wait complete. 2019-12-05T06:50:15.438 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.431+0000 7feead6e6700 1 -- 172.21.15.42:0/2129940404 >> 172.21.15.42:0/2129940404 conn(0x7feea807e4f0 msgr2=0x7feea80a9790 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:50:15.449 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b down, removed from mdsmap 2019-12-05T06:50:15.449 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2019-12-05T06:50:15.450 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:50:15.562 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.551+0000 7f55d63c2700 1 Processor -- start 2019-12-05T06:50:15.562 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.551+0000 7f55d63c2700 1 -- start start 2019-12-05T06:50:15.562 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.551+0000 7f55d63c2700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f55d00a7740 0x7f55d008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:50:15.563 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.551+0000 7f55d63c2700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f55d007e720 0x7f55d007eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:50:15.563 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.551+0000 7f55d63c2700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f55d009f2e0 0x7f55d009f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:50:15.563 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.551+0000 7f55d63c2700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55d00a0b00 con 0x7f55d009f2e0 2019-12-05T06:50:15.563 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.551+0000 7f55d63c2700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55d008e5b0 con 0x7f55d00a7740 2019-12-05T06:50:15.563 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.551+0000 7f55d63c2700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55d008e850 con 0x7f55d007e720 2019-12-05T06:50:15.563 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.551+0000 7f55d495f700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f55d00a7740 0x7f55d008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:15.563 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.551+0000 7f55d495f700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f55d00a7740 0x7f55d008df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:38354/0 (socket says 172.21.15.42:38354) 2019-12-05T06:50:15.564 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.551+0000 7f55d495f700 1 -- 172.21.15.42:0/3415278063 learned_addr learned my addr 172.21.15.42:0/3415278063 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:50:15.564 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.551+0000 7f55d5160700 1 --2- 172.21.15.42:0/3415278063 >> v2:172.21.15.40:3300/0 conn(0x7f55d009f2e0 0x7f55d009f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:15.564 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.551+0000 7f55cffff700 1 --2- 172.21.15.42:0/3415278063 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f55d007e720 0x7f55d007eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:15.564 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.551+0000 7f55d495f700 1 -- 172.21.15.42:0/3415278063 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f55d007e720 msgr2=0x7f55d007eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:50:15.564 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.551+0000 7f55d495f700 1 --2- 172.21.15.42:0/3415278063 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f55d007e720 0x7f55d007eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:50:15.564 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.551+0000 7f55d495f700 1 -- 172.21.15.42:0/3415278063 >> v2:172.21.15.40:3300/0 conn(0x7f55d009f2e0 msgr2=0x7f55d009f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:50:15.564 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.551+0000 7f55d495f700 1 --2- 172.21.15.42:0/3415278063 >> v2:172.21.15.40:3300/0 conn(0x7f55d009f2e0 0x7f55d009f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:50:15.565 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.551+0000 7f55d495f700 1 -- 172.21.15.42:0/3415278063 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55d007f0a0 con 0x7f55d00a7740 2019-12-05T06:50:15.565 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.551+0000 7f55d495f700 1 --2- 172.21.15.42:0/3415278063 >> v2:172.21.15.42:3300/0 conn(0x7f55d00a7740 0x7f55d008df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f55d008e770 tx=0x7f55c400fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:15.565 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.551+0000 7f55d495f700 1 --2- 172.21.15.42:0/3415278063 >> v2:172.21.15.42:3300/0 conn(0x7f55d00a7740 0x7f55d008df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f55d008e770 tx=0x7f55c400fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:15.565 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.551+0000 7f55d495f700 1 --2- 172.21.15.42:0/3415278063 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f55d00a7740 0x7f55d008df80 secure :-1 s=READY pgs=3317 cs=0 l=1 rx=0x7f55d008e770 tx=0x7f55c400fdb0).ready entity=mon.1 client_cookie=92f926cdfae303d9 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:50:15.565 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.551+0000 7f55d495f700 1 --2- 172.21.15.42:0/3415278063 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f55d00a7740 0x7f55d008df80 secure :-1 s=THROTTLE_DONE pgs=3317 cs=0 l=1 rx=0x7f55d008e770 tx=0x7f55c400fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:15.565 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.551+0000 7f55d495f700 1 --2- 172.21.15.42:0/3415278063 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f55d00a7740 0x7f55d008df80 secure :-1 s=THROTTLE_DONE pgs=3317 cs=0 l=1 rx=0x7f55d008e770 tx=0x7f55c400fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:15.565 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.551+0000 7f55d495f700 1 --2- 172.21.15.42:0/3415278063 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f55d00a7740 0x7f55d008df80 secure :-1 s=THROTTLE_DONE pgs=3317 cs=0 l=1 rx=0x7f55d008e770 tx=0x7f55c400fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:15.566 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.551+0000 7f55cf7fe700 1 -- 172.21.15.42:0/3415278063 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f55c4008c90 con 0x7f55d00a7740 2019-12-05T06:50:15.566 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.555+0000 7f55cf7fe700 1 -- 172.21.15.42:0/3415278063 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55c40088b0 con 0x7f55d00a7740 2019-12-05T06:50:15.566 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.555+0000 7f55cf7fe700 1 -- 172.21.15.42:0/3415278063 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f55c4011b50 con 0x7f55d00a7740 2019-12-05T06:50:15.566 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.555+0000 7f55d63c2700 1 -- 172.21.15.42:0/3415278063 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f55d00a7740 msgr2=0x7f55d008df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:15.566 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.555+0000 7f55d63c2700 1 --2- 172.21.15.42:0/3415278063 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f55d00a7740 0x7f55d008df80 secure :-1 s=READY pgs=3317 cs=0 l=1 rx=0x7f55d008e770 tx=0x7f55c400fdb0).stop 2019-12-05T06:50:15.566 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.555+0000 7f55d63c2700 1 -- 172.21.15.42:0/3415278063 shutdown_connections 2019-12-05T06:50:15.566 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.555+0000 7f55d63c2700 1 --2- 172.21.15.42:0/3415278063 >> v2:172.21.15.40:3300/0 conn(0x7f55d009f2e0 0x7f55d009f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:50:15.566 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.555+0000 7f55d63c2700 1 --2- 172.21.15.42:0/3415278063 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f55d00a7740 0x7f55d008df80 unknown :-1 s=CLOSED pgs=3317 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:15.567 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.555+0000 7f55d63c2700 1 --2- 172.21.15.42:0/3415278063 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f55d007e720 0x7f55d007eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:50:15.567 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.555+0000 7f55d63c2700 1 -- 172.21.15.42:0/3415278063 shutdown_connections 2019-12-05T06:50:15.567 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.555+0000 7f55d63c2700 1 -- 172.21.15.42:0/3415278063 wait complete. 2019-12-05T06:50:15.567 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.555+0000 7f55d63c2700 1 -- 172.21.15.42:0/3415278063 >> 172.21.15.42:0/3415278063 conn(0x7f55d00aabe0 msgr2=0x7f55d00a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:50:15.571 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.555+0000 7f55d63c2700 1 Processor -- start 2019-12-05T06:50:15.571 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.555+0000 7f55d63c2700 1 -- start start 2019-12-05T06:50:15.571 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.559+0000 7f55d63c2700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f55d00a5bd0 0x7f55d0077b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:50:15.571 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.559+0000 7f55d63c2700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55d01223a0 con 0x7f55d0123670 2019-12-05T06:50:15.572 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.559+0000 7f55d63c2700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55d011b140 con 0x7f55d0124c60 2019-12-05T06:50:15.572 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.559+0000 7f55d63c2700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55d011ac00 con 0x7f55d00a5bd0 2019-12-05T06:50:15.572 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.559+0000 7f55d495f700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f55d0124c60 0x7f55d0122e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:38398/0 (socket says 172.21.15.42:38398) 2019-12-05T06:50:15.572 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.559+0000 7f55d495f700 1 -- 172.21.15.42:0/4057684826 learned_addr learned my addr 172.21.15.42:0/4057684826 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:50:15.572 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.559+0000 7f55d5160700 1 --2- 172.21.15.42:0/4057684826 >> v2:172.21.15.40:3300/0 conn(0x7f55d00a5bd0 0x7f55d0077b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:15.572 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.559+0000 7f55cd7fa700 1 -- 172.21.15.42:0/4057684826 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3050716562 0 0) 0x7f55d011b140 con 0x7f55d0124c60 2019-12-05T06:50:15.572 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.559+0000 7f55cd7fa700 1 -- 172.21.15.42:0/4057684826 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55b00030b0 con 0x7f55d0124c60 2019-12-05T06:50:15.573 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.559+0000 7f55cd7fa700 1 -- 172.21.15.42:0/4057684826 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3316483959 0 0) 0x7f55d01223a0 con 0x7f55d0123670 2019-12-05T06:50:15.573 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.559+0000 7f55cd7fa700 1 -- 172.21.15.42:0/4057684826 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55d011b140 con 0x7f55d0123670 2019-12-05T06:50:15.573 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.559+0000 7f55d5160700 1 -- 172.21.15.42:0/4057684826 >> v1:172.21.15.42:6789/0 conn(0x7f55d0124c60 legacy=0x7f55d0122e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:15.573 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.559+0000 7f55d5160700 1 -- 172.21.15.42:0/4057684826 >> v1:172.21.15.40:6789/0 conn(0x7f55d0123670 legacy=0x7f55d00742e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:15.573 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.559+0000 7f55d5160700 1 -- 172.21.15.42:0/4057684826 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55d011a650 con 0x7f55d00a5bd0 2019-12-05T06:50:15.573 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.559+0000 7f55d5160700 1 --2- 172.21.15.42:0/4057684826 >> v2:172.21.15.40:3300/0 conn(0x7f55d00a5bd0 0x7f55d0077b00 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f55c80032b0 tx=0x7f55c8010c80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:15.573 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.559+0000 7f55d5160700 1 --2- 172.21.15.42:0/4057684826 >> v2:172.21.15.40:3300/0 conn(0x7f55d00a5bd0 0x7f55d0077b00 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f55c80032b0 tx=0x7f55c8010c80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:15.574 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.559+0000 7f55d5160700 1 --2- 172.21.15.42:0/4057684826 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f55d00a5bd0 0x7f55d0077b00 secure :-1 s=READY pgs=2513 cs=0 l=1 rx=0x7f55c80032b0 tx=0x7f55c8010c80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:50:15.574 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.559+0000 7f55d5160700 1 --2- 172.21.15.42:0/4057684826 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f55d00a5bd0 0x7f55d0077b00 secure :-1 s=THROTTLE_DONE pgs=2513 cs=0 l=1 rx=0x7f55c80032b0 tx=0x7f55c8010c80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:15.574 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.559+0000 7f55d5160700 1 --2- 172.21.15.42:0/4057684826 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f55d00a5bd0 0x7f55d0077b00 secure :-1 s=THROTTLE_DONE pgs=2513 cs=0 l=1 rx=0x7f55c80032b0 tx=0x7f55c8010c80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:15.574 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.559+0000 7f55cd7fa700 1 -- 172.21.15.42:0/4057684826 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f55c8005d40 con 0x7f55d00a5bd0 2019-12-05T06:50:15.574 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.559+0000 7f55d5160700 1 --2- 172.21.15.42:0/4057684826 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f55d00a5bd0 0x7f55d0077b00 secure :-1 s=THROTTLE_DONE pgs=2513 cs=0 l=1 rx=0x7f55c80032b0 tx=0x7f55c8010c80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:15.575 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.559+0000 7f55cd7fa700 1 -- 172.21.15.42:0/4057684826 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55c8005960 con 0x7f55d00a5bd0 2019-12-05T06:50:15.575 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.559+0000 7f55d63c2700 1 -- 172.21.15.42:0/4057684826 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55d011a8c0 con 0x7f55d00a5bd0 2019-12-05T06:50:15.575 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.559+0000 7f55d63c2700 1 -- 172.21.15.42:0/4057684826 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55d0078050 con 0x7f55d00a5bd0 2019-12-05T06:50:15.575 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.559+0000 7f55cd7fa700 1 -- 172.21.15.42:0/4057684826 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f55c8012be0 con 0x7f55d00a5bd0 2019-12-05T06:50:15.575 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.559+0000 7f55d5160700 1 --2- 172.21.15.42:0/4057684826 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f55d00a5bd0 0x7f55d0077b00 secure :-1 s=THROTTLE_DONE pgs=2513 cs=0 l=1 rx=0x7f55c80032b0 tx=0x7f55c8010c80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:15.575 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.559+0000 7f55cd7fa700 1 -- 172.21.15.42:0/4057684826 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f55c8013060 con 0x7f55d00a5bd0 2019-12-05T06:50:15.576 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.563+0000 7f55cd7fa700 1 --2- 172.21.15.42:0/4057684826 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f55b003b420 0x7f55b003d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:50:15.576 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.563+0000 7f55d5160700 1 --2- 172.21.15.42:0/4057684826 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f55d00a5bd0 0x7f55d0077b00 secure :-1 s=THROTTLE_DONE pgs=2513 cs=0 l=1 rx=0x7f55c80032b0 tx=0x7f55c8010c80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:15.576 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.563+0000 7f55cd7fa700 1 -- 172.21.15.42:0/4057684826 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 12309+0+0 (secure 0 0 0) 0x7f55c8032850 con 0x7f55d00a5bd0 2019-12-05T06:50:15.576 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.563+0000 7f55d495f700 1 --2- 172.21.15.42:0/4057684826 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f55b003b420 0x7f55b003d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:15.576 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.563+0000 7f55d495f700 1 --2- 172.21.15.42:0/4057684826 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f55b003b420 0x7f55b003d930 crc :-1 s=READY pgs=1392 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:50:15.577 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.567+0000 7f55d63c2700 1 -- 172.21.15.42:0/4057684826 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55d005d110 con 0x7f55d00a5bd0 2019-12-05T06:50:15.581 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.575+0000 7f55d5160700 1 --2- 172.21.15.42:0/4057684826 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f55d00a5bd0 0x7f55d0077b00 secure :-1 s=THROTTLE_DONE pgs=2513 cs=0 l=1 rx=0x7f55c80032b0 tx=0x7f55c8010c80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:15.581 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.575+0000 7f55cd7fa700 1 -- 172.21.15.42:0/4057684826 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f55c8032ab0 con 0x7f55d00a5bd0 2019-12-05T06:50:15.755 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.747+0000 7f55d63c2700 1 -- 172.21.15.42:0/4057684826 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f55d005b8c0 con 0x7f55d00a5bd0 2019-12-05T06:50:15.758 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.751+0000 7f55d5160700 1 --2- 172.21.15.42:0/4057684826 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f55d00a5bd0 0x7f55d0077b00 secure :-1 s=THROTTLE_DONE pgs=2513 cs=0 l=1 rx=0x7f55c80032b0 tx=0x7f55c8010c80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:15.758 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.751+0000 7f55cd7fa700 1 -- 172.21.15.42:0/4057684826 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 400 v400) v1 ==== 95+0+1775 (secure 0 0 0) 0x7f55c8033660 con 0x7f55d00a5bd0 2019-12-05T06:50:15.758 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 400 2019-12-05T06:50:15.758 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:50:15.759 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":400,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":400,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:50:15.404443+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":122,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8537},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8537":{"gid":8537,"name":"a","rank":0,"incarnation":400,"state":"up:replay","state_seq":2,"addr":"172.21.15.42:6835/1881703925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1881703925},{"type":"v1","addr":"172.21.15.42:6835","nonce":1881703925}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:50:15.761 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.751+0000 7f55d63c2700 1 -- 172.21.15.42:0/4057684826 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f55b003b420 msgr2=0x7f55b003d930 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:15.761 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.751+0000 7f55d63c2700 1 --2- 172.21.15.42:0/4057684826 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f55b003b420 0x7f55b003d930 crc :-1 s=READY pgs=1392 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:15.761 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.751+0000 7f55d63c2700 1 -- 172.21.15.42:0/4057684826 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f55d00a5bd0 msgr2=0x7f55d0077b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:15.762 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.751+0000 7f55d63c2700 1 --2- 172.21.15.42:0/4057684826 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f55d00a5bd0 0x7f55d0077b00 secure :-1 s=READY pgs=2513 cs=0 l=1 rx=0x7f55c80032b0 tx=0x7f55c8010c80).stop 2019-12-05T06:50:15.762 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.751+0000 7f55d63c2700 1 -- 172.21.15.42:0/4057684826 shutdown_connections 2019-12-05T06:50:15.762 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.751+0000 7f55d63c2700 1 --2- 172.21.15.42:0/4057684826 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f55b003b420 0x7f55b003d930 unknown :-1 s=CLOSED pgs=1392 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:15.762 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.751+0000 7f55d63c2700 1 --2- 172.21.15.42:0/4057684826 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f55d00a5bd0 0x7f55d0077b00 unknown :-1 s=CLOSED pgs=2513 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:15.762 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.751+0000 7f55d63c2700 1 -- 172.21.15.42:0/4057684826 shutdown_connections 2019-12-05T06:50:15.762 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.751+0000 7f55d63c2700 1 -- 172.21.15.42:0/4057684826 wait complete. 2019-12-05T06:50:15.763 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:15.751+0000 7f55d63c2700 1 -- 172.21.15.42:0/4057684826 >> 172.21.15.42:0/4057684826 conn(0x7f55d00aabe0 msgr2=0x7f55d00a9640 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:50:15.769 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 400, "feature_flags": { "enable_multiple": false, "ever_enabled_multiple": false }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2019-12-05T05:03:35.604738+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 400, "ever_allowed_features": 2, "explicitly_allowed_features": 2, "failed": [], "flags": 18, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_8537": { "addr": "172.21.15.42:6835/1881703925", "addrs": { "addrvec": [ { "addr": "172.21.15.42:6834", "nonce": 1881703925, "type": "v2" }, { "addr": "172.21.15.42:6835", "nonce": 1881703925, "type": "v1" } ] }, "export_targets": [], "features": 4540138292836696063, "flags": 0, "gid": 8537, "incarnation": 400, "name": "a", "rank": 0, "state": "up:replay", "state_seq": 2 } }, "last_failure": 0, "last_failure_osd_epoch": 122, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 2, "min_compat_client": "0 (unknown)", "modified": "2019-12-05T06:50:15.404443+0000", "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 8537 } } } ], "standbys": [] } 2019-12-05T06:50:17.770 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:50:17.868 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:17.859+0000 7f78c95ae700 1 Processor -- start 2019-12-05T06:50:17.869 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:17.859+0000 7f78c95ae700 1 -- start start 2019-12-05T06:50:17.869 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:17.859+0000 7f78c95ae700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f78c408df80 0x7f78c409f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:50:17.869 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:17.859+0000 7f78c95ae700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f78c409f830 0x7f78c414efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:50:17.869 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:17.859+0000 7f78c95ae700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78c409fc60 con 0x7f78c40a7740 2019-12-05T06:50:17.870 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:17.859+0000 7f78c95ae700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78c40a0b00 con 0x7f78c409f830 2019-12-05T06:50:17.870 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:17.859+0000 7f78c95ae700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78c408e5b0 con 0x7f78c408df80 2019-12-05T06:50:17.870 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:17.859+0000 7f78c2ffd700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f78c408df80 0x7f78c409f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:17.870 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:17.859+0000 7f78c2ffd700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f78c408df80 0x7f78c409f2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:38370/0 (socket says 172.21.15.42:38370) 2019-12-05T06:50:17.871 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:17.859+0000 7f78c2ffd700 1 -- 172.21.15.42:0/2749126002 learned_addr learned my addr 172.21.15.42:0/2749126002 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:50:17.871 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:17.859+0000 7f78c3fff700 1 --2- 172.21.15.42:0/2749126002 >> v2:172.21.15.40:3300/0 conn(0x7f78c409f830 0x7f78c414efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:17.871 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:17.859+0000 7f78c2ffd700 1 -- 172.21.15.42:0/2749126002 >> v2:172.21.15.40:3300/0 conn(0x7f78c409f830 msgr2=0x7f78c414efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:50:17.871 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:17.859+0000 7f78c2ffd700 1 --2- 172.21.15.42:0/2749126002 >> v2:172.21.15.40:3300/0 conn(0x7f78c409f830 0x7f78c414efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:50:17.871 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:17.859+0000 7f78c2ffd700 1 -- 172.21.15.42:0/2749126002 >> v1:172.21.15.42:6789/0 conn(0x7f78c40a7740 legacy=0x7f78c407e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:17.871 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:17.859+0000 7f78c27fc700 1 -- 172.21.15.42:0/2749126002 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3442601515 0 0) 0x7f78c409fc60 con 0x7f78c40a7740 2019-12-05T06:50:17.872 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:17.859+0000 7f78c2ffd700 1 -- 172.21.15.42:0/2749126002 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78c40a0710 con 0x7f78c408df80 2019-12-05T06:50:17.872 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:17.859+0000 7f78c2ffd700 1 --2- 172.21.15.42:0/2749126002 >> v2:172.21.15.42:3300/0 conn(0x7f78c408df80 0x7f78c409f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f78ac002f30 tx=0x7f78ac00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:17.872 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:17.859+0000 7f78c2ffd700 1 --2- 172.21.15.42:0/2749126002 >> v2:172.21.15.42:3300/0 conn(0x7f78c408df80 0x7f78c409f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f78ac002f30 tx=0x7f78ac00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:17.872 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:17.859+0000 7f78c2ffd700 1 --2- 172.21.15.42:0/2749126002 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f78c408df80 0x7f78c409f2e0 secure :-1 s=READY pgs=3320 cs=0 l=1 rx=0x7f78ac002f30 tx=0x7f78ac00fd30).ready entity=mon.1 client_cookie=793b5cba6b9e0202 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:50:17.872 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:17.859+0000 7f78c2ffd700 1 --2- 172.21.15.42:0/2749126002 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f78c408df80 0x7f78c409f2e0 secure :-1 s=THROTTLE_DONE pgs=3320 cs=0 l=1 rx=0x7f78ac002f30 tx=0x7f78ac00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:17.873 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:17.859+0000 7f78c27fc700 1 -- 172.21.15.42:0/2749126002 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f78ac008c90 con 0x7f78c408df80 2019-12-05T06:50:17.873 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:17.859+0000 7f78c2ffd700 1 --2- 172.21.15.42:0/2749126002 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f78c408df80 0x7f78c409f2e0 secure :-1 s=THROTTLE_DONE pgs=3320 cs=0 l=1 rx=0x7f78ac002f30 tx=0x7f78ac00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:17.873 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:17.859+0000 7f78c2ffd700 1 --2- 172.21.15.42:0/2749126002 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f78c408df80 0x7f78c409f2e0 secure :-1 s=THROTTLE_DONE pgs=3320 cs=0 l=1 rx=0x7f78ac002f30 tx=0x7f78ac00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:17.873 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:17.859+0000 7f78c27fc700 1 -- 172.21.15.42:0/2749126002 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78ac0088b0 con 0x7f78c408df80 2019-12-05T06:50:17.873 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:17.859+0000 7f78c27fc700 1 -- 172.21.15.42:0/2749126002 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f78ac011ad0 con 0x7f78c408df80 2019-12-05T06:50:17.873 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:17.859+0000 7f78c95ae700 1 -- 172.21.15.42:0/2749126002 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f78c408df80 msgr2=0x7f78c409f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:17.874 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:17.859+0000 7f78c95ae700 1 --2- 172.21.15.42:0/2749126002 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f78c408df80 0x7f78c409f2e0 secure :-1 s=READY pgs=3320 cs=0 l=1 rx=0x7f78ac002f30 tx=0x7f78ac00fd30).stop 2019-12-05T06:50:17.874 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:17.859+0000 7f78c95ae700 1 -- 172.21.15.42:0/2749126002 shutdown_connections 2019-12-05T06:50:17.874 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:17.859+0000 7f78c95ae700 1 --2- 172.21.15.42:0/2749126002 >> v2:172.21.15.40:3300/0 conn(0x7f78c409f830 0x7f78c414efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:50:17.874 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:17.859+0000 7f78c95ae700 1 --2- 172.21.15.42:0/2749126002 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f78c408df80 0x7f78c409f2e0 unknown :-1 s=CLOSED pgs=3320 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:17.874 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:17.859+0000 7f78c95ae700 1 -- 172.21.15.42:0/2749126002 shutdown_connections 2019-12-05T06:50:17.874 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:17.859+0000 7f78c95ae700 1 -- 172.21.15.42:0/2749126002 wait complete. 2019-12-05T06:50:17.875 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:17.859+0000 7f78c95ae700 1 -- 172.21.15.42:0/2749126002 >> 172.21.15.42:0/2749126002 conn(0x7f78c40aabe0 msgr2=0x7f78c40a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:50:17.882 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:17.863+0000 7f78c95ae700 1 Processor -- start 2019-12-05T06:50:17.882 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:17.863+0000 7f78c95ae700 1 -- start start 2019-12-05T06:50:17.883 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:17.863+0000 7f78c95ae700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f78c40a3080 0x7f78c40a1b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:50:17.883 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:17.863+0000 7f78c95ae700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f78c40a2080 0x7f78c41379c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:50:17.883 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:17.863+0000 7f78c95ae700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78c40a18b0 con 0x7f78c4067320 2019-12-05T06:50:17.883 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:17.863+0000 7f78c95ae700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78c4142840 con 0x7f78c40a2080 2019-12-05T06:50:17.884 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:17.863+0000 7f78c95ae700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78c4142ae0 con 0x7f78c40a3080 2019-12-05T06:50:17.884 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:17.863+0000 7f78c37fe700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f78c4067320 0x7f78c407c890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:38412/0 (socket says 172.21.15.42:38412) 2019-12-05T06:50:17.884 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:17.863+0000 7f78c37fe700 1 -- 172.21.15.42:0/4262202958 learned_addr learned my addr 172.21.15.42:0/4262202958 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:50:17.884 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:17.863+0000 7f78c3fff700 1 --2- 172.21.15.42:0/4262202958 >> v2:172.21.15.40:3300/0 conn(0x7f78c40a2080 0x7f78c41379c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:17.884 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:17.863+0000 7f78c2ffd700 1 --2- 172.21.15.42:0/4262202958 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f78c40a3080 0x7f78c40a1b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:17.885 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:17.863+0000 7f78bbfff700 1 -- 172.21.15.42:0/4262202958 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1373155867 0 0) 0x7f78c40a18b0 con 0x7f78c4067320 2019-12-05T06:50:17.885 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:17.863+0000 7f78bbfff700 1 -- 172.21.15.42:0/4262202958 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78a80030b0 con 0x7f78c4067320 2019-12-05T06:50:17.885 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:17.863+0000 7f78bbfff700 1 -- 172.21.15.42:0/4262202958 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 4265574160 0 0) 0x7f78a80030b0 con 0x7f78c4067320 2019-12-05T06:50:17.885 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:17.863+0000 7f78bbfff700 1 -- 172.21.15.42:0/4262202958 --> v1:172.21.15.42:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f78c40a18b0 con 0x7f78c4067320 2019-12-05T06:50:17.885 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:17.863+0000 7f78bbfff700 1 -- 172.21.15.42:0/4262202958 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7f78b4001a20 con 0x7f78c4067320 2019-12-05T06:50:17.886 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:17.863+0000 7f78bbfff700 1 -- 172.21.15.42:0/4262202958 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f78c40a3080 msgr2=0x7f78c40a1b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:17.886 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:17.863+0000 7f78bbfff700 1 --2- 172.21.15.42:0/4262202958 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f78c40a3080 0x7f78c40a1b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:17.887 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:17.863+0000 7f78bbfff700 1 -- 172.21.15.42:0/4262202958 >> v2:172.21.15.40:3300/0 conn(0x7f78c40a2080 msgr2=0x7f78c41379c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:17.887 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:17.863+0000 7f78bbfff700 1 --2- 172.21.15.42:0/4262202958 >> v2:172.21.15.40:3300/0 conn(0x7f78c40a2080 0x7f78c41379c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:17.887 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:17.863+0000 7f78bbfff700 1 -- 172.21.15.42:0/4262202958 >> v1:172.21.15.42:6789/0 conn(0x7f78c4067320 legacy=0x7f78c407c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:17.887 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:17.863+0000 7f78bbfff700 1 --2- 172.21.15.42:0/4262202958 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f78a80038a0 0x7f78a8003cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:50:17.887 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:17.863+0000 7f78bbfff700 1 --2- 172.21.15.42:0/4262202958 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f78a8004420 0x7f78a8004850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:50:17.888 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:17.863+0000 7f78bbfff700 1 --2- 172.21.15.42:0/4262202958 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f78a8004da0 0x7f78a800b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:50:17.888 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:17.863+0000 7f78bbfff700 1 -- 172.21.15.42:0/4262202958 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78c4142840 con 0x7f78a8004420 2019-12-05T06:50:17.888 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:17.863+0000 7f78bbfff700 1 -- 172.21.15.42:0/4262202958 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78c4142ae0 con 0x7f78a8004da0 2019-12-05T06:50:17.888 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:17.863+0000 7f78bbfff700 1 -- 172.21.15.42:0/4262202958 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78a800ddd0 con 0x7f78a80038a0 2019-12-05T06:50:17.888 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:17.863+0000 7f78c3fff700 1 --2- 172.21.15.42:0/4262202958 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f78a8004da0 0x7f78a800b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:17.889 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:17.863+0000 7f78c2ffd700 1 --2- 172.21.15.42:0/4262202958 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f78a8004420 0x7f78a8004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:17.889 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:17.863+0000 7f78c3fff700 1 -- 172.21.15.42:0/4262202958 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f78a80038a0 msgr2=0x7f78a8003cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:17.889 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:17.863+0000 7f78c3fff700 1 --2- 172.21.15.42:0/4262202958 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f78a80038a0 0x7f78a8003cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:17.889 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:17.863+0000 7f78c3fff700 1 -- 172.21.15.42:0/4262202958 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f78a8004420 msgr2=0x7f78a8004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:17.889 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:17.863+0000 7f78c3fff700 1 --2- 172.21.15.42:0/4262202958 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f78a8004420 0x7f78a8004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:17.890 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:17.863+0000 7f78c3fff700 1 -- 172.21.15.42:0/4262202958 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78c4142530 con 0x7f78a8004da0 2019-12-05T06:50:17.890 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:17.863+0000 7f78c95ae700 1 -- 172.21.15.42:0/4262202958 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78c40a24b0 con 0x7f78a8004da0 2019-12-05T06:50:17.890 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:17.863+0000 7f78c3fff700 1 --2- 172.21.15.42:0/4262202958 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f78a8004da0 0x7f78a800b670 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f78bc0032b0 tx=0x7f78bc011190).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:17.891 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:17.863+0000 7f78c95ae700 1 -- 172.21.15.42:0/4262202958 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78c407d090 con 0x7f78a8004da0 2019-12-05T06:50:17.891 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:17.863+0000 7f78c3fff700 1 --2- 172.21.15.42:0/4262202958 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f78a8004da0 0x7f78a800b670 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f78bc0032b0 tx=0x7f78bc011190).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:17.891 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:17.863+0000 7f78c3fff700 1 --2- 172.21.15.42:0/4262202958 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f78a8004da0 0x7f78a800b670 secure :-1 s=READY pgs=3322 cs=0 l=1 rx=0x7f78bc0032b0 tx=0x7f78bc011190).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:50:17.891 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:17.867+0000 7f78c3fff700 1 --2- 172.21.15.42:0/4262202958 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f78a8004da0 0x7f78a800b670 secure :-1 s=THROTTLE_DONE pgs=3322 cs=0 l=1 rx=0x7f78bc0032b0 tx=0x7f78bc011190).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:17.892 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:17.867+0000 7f78c3fff700 1 --2- 172.21.15.42:0/4262202958 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f78a8004da0 0x7f78a800b670 secure :-1 s=THROTTLE_DONE pgs=3322 cs=0 l=1 rx=0x7f78bc0032b0 tx=0x7f78bc011190).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:17.892 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:17.867+0000 7f78bbfff700 1 -- 172.21.15.42:0/4262202958 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f78bc00a450 con 0x7f78a8004da0 2019-12-05T06:50:17.892 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:17.867+0000 7f78c3fff700 1 --2- 172.21.15.42:0/4262202958 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f78a8004da0 0x7f78a800b670 secure :-1 s=THROTTLE_DONE pgs=3322 cs=0 l=1 rx=0x7f78bc0032b0 tx=0x7f78bc011190).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:17.892 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:17.867+0000 7f78bbfff700 1 -- 172.21.15.42:0/4262202958 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78bc011ae0 con 0x7f78a8004da0 2019-12-05T06:50:17.893 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:17.867+0000 7f78bbfff700 1 -- 172.21.15.42:0/4262202958 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f78bc00aa30 con 0x7f78a8004da0 2019-12-05T06:50:17.893 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:17.867+0000 7f78c3fff700 1 --2- 172.21.15.42:0/4262202958 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f78a8004da0 0x7f78a800b670 secure :-1 s=THROTTLE_DONE pgs=3322 cs=0 l=1 rx=0x7f78bc0032b0 tx=0x7f78bc011190).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:17.893 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:17.867+0000 7f78bbfff700 1 -- 172.21.15.42:0/4262202958 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f78bc00ac30 con 0x7f78a8004da0 2019-12-05T06:50:17.893 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:17.867+0000 7f78c3fff700 1 --2- 172.21.15.42:0/4262202958 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f78a8004da0 0x7f78a800b670 secure :-1 s=THROTTLE_DONE pgs=3322 cs=0 l=1 rx=0x7f78bc0032b0 tx=0x7f78bc011190).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:17.893 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:17.867+0000 7f78bbfff700 1 --2- 172.21.15.42:0/4262202958 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f78a8043af0 0x7f78a8046000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:50:17.893 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:17.867+0000 7f78bbfff700 1 -- 172.21.15.42:0/4262202958 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 12309+0+0 (secure 0 0 0) 0x7f78bc04b600 con 0x7f78a8004da0 2019-12-05T06:50:17.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:17.867+0000 7f78c37fe700 1 --2- 172.21.15.42:0/4262202958 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f78a8043af0 0x7f78a8046000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:17.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:17.867+0000 7f78c37fe700 1 --2- 172.21.15.42:0/4262202958 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f78a8043af0 0x7f78a8046000 crc :-1 s=READY pgs=1393 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:50:17.894 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:17.875+0000 7f78c95ae700 1 -- 172.21.15.42:0/4262202958 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f789c003200 con 0x7f78a8004da0 2019-12-05T06:50:17.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:17.879+0000 7f78c3fff700 1 --2- 172.21.15.42:0/4262202958 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f78a8004da0 0x7f78a800b670 secure :-1 s=THROTTLE_DONE pgs=3322 cs=0 l=1 rx=0x7f78bc0032b0 tx=0x7f78bc011190).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:17.896 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:17.879+0000 7f78bbfff700 1 -- 172.21.15.42:0/4262202958 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f78bc0326a0 con 0x7f78a8004da0 2019-12-05T06:50:18.040 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:18.035+0000 7f78c95ae700 1 -- 172.21.15.42:0/4262202958 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f789c003630 con 0x7f78a8004da0 2019-12-05T06:50:18.041 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:18.035+0000 7f78c3fff700 1 --2- 172.21.15.42:0/4262202958 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f78a8004da0 0x7f78a800b670 secure :-1 s=THROTTLE_DONE pgs=3322 cs=0 l=1 rx=0x7f78bc0032b0 tx=0x7f78bc011190).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:18.041 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:18.035+0000 7f78bbfff700 1 -- 172.21.15.42:0/4262202958 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 400 v400) v1 ==== 95+0+1775 (secure 0 0 0) 0x7f78bc032900 con 0x7f78a8004da0 2019-12-05T06:50:18.042 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 400 2019-12-05T06:50:18.042 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:50:18.042 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":400,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":400,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:50:15.404443+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":122,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8537},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8537":{"gid":8537,"name":"a","rank":0,"incarnation":400,"state":"up:replay","state_seq":2,"addr":"172.21.15.42:6835/1881703925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1881703925},{"type":"v1","addr":"172.21.15.42:6835","nonce":1881703925}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:50:18.044 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:18.035+0000 7f78c95ae700 1 -- 172.21.15.42:0/4262202958 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f78a8043af0 msgr2=0x7f78a8046000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:18.044 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:18.035+0000 7f78c95ae700 1 --2- 172.21.15.42:0/4262202958 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f78a8043af0 0x7f78a8046000 crc :-1 s=READY pgs=1393 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:18.044 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:18.035+0000 7f78c95ae700 1 -- 172.21.15.42:0/4262202958 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f78a8004da0 msgr2=0x7f78a800b670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:18.044 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:18.035+0000 7f78c3fff700 1 -- 172.21.15.42:0/4262202958 reap_dead start 2019-12-05T06:50:18.044 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:18.035+0000 7f78c95ae700 1 --2- 172.21.15.42:0/4262202958 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f78a8004da0 0x7f78a800b670 secure :-1 s=READY pgs=3322 cs=0 l=1 rx=0x7f78bc0032b0 tx=0x7f78bc011190).stop 2019-12-05T06:50:18.044 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:18.035+0000 7f78c95ae700 1 -- 172.21.15.42:0/4262202958 shutdown_connections 2019-12-05T06:50:18.045 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:18.035+0000 7f78c95ae700 1 --2- 172.21.15.42:0/4262202958 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f78a8004da0 0x7f78a800b670 unknown :-1 s=CLOSED pgs=3322 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:18.045 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:18.035+0000 7f78c95ae700 1 -- 172.21.15.42:0/4262202958 shutdown_connections 2019-12-05T06:50:18.045 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:18.035+0000 7f78c95ae700 1 -- 172.21.15.42:0/4262202958 wait complete. 2019-12-05T06:50:18.045 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:18.035+0000 7f78c95ae700 1 -- 172.21.15.42:0/4262202958 >> 172.21.15.42:0/4262202958 conn(0x7f78c40a82a0 msgr2=0x7f78c411ca60 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:50:18.053 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2019-12-05T06:50:20.054 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:50:20.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:20.171+0000 7fb358ffb700 1 Processor -- start 2019-12-05T06:50:20.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:20.171+0000 7fb358ffb700 1 -- start start 2019-12-05T06:50:20.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:20.171+0000 7fb358ffb700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb35408df80 0x7fb35409f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:50:20.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:20.171+0000 7fb358ffb700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb35409f830 0x7fb35414efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:50:20.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:20.171+0000 7fb358ffb700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb35409fc60 con 0x7fb3540a7740 2019-12-05T06:50:20.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:20.171+0000 7fb358ffb700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3540a0b00 con 0x7fb35409f830 2019-12-05T06:50:20.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:20.171+0000 7fb358ffb700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb35408e5b0 con 0x7fb35408df80 2019-12-05T06:50:20.184 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:20.171+0000 7fb352d9d700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fb3540a7740 0x7fb35407e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:38428/0 (socket says 172.21.15.42:38428) 2019-12-05T06:50:20.184 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:20.171+0000 7fb352d9d700 1 -- 172.21.15.42:0/1853756490 learned_addr learned my addr 172.21.15.42:0/1853756490 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:50:20.184 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:20.171+0000 7fb35259c700 1 --2- 172.21.15.42:0/1853756490 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb35408df80 0x7fb35409f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:20.184 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:20.171+0000 7fb35359e700 1 --2- 172.21.15.42:0/1853756490 >> v2:172.21.15.40:3300/0 conn(0x7fb35409f830 0x7fb35414efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:20.184 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:20.171+0000 7fb351d9b700 1 -- 172.21.15.42:0/1853756490 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 619542199 0 0) 0x7fb35409fc60 con 0x7fb3540a7740 2019-12-05T06:50:20.184 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:20.171+0000 7fb351d9b700 1 -- 172.21.15.42:0/1853756490 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3340030b0 con 0x7fb3540a7740 2019-12-05T06:50:20.184 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:20.171+0000 7fb35259c700 1 -- 172.21.15.42:0/1853756490 >> v2:172.21.15.40:3300/0 conn(0x7fb35409f830 msgr2=0x7fb35414efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:50:20.185 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:20.171+0000 7fb35259c700 1 --2- 172.21.15.42:0/1853756490 >> v2:172.21.15.40:3300/0 conn(0x7fb35409f830 0x7fb35414efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:50:20.185 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:20.171+0000 7fb35259c700 1 -- 172.21.15.42:0/1853756490 >> v1:172.21.15.42:6789/0 conn(0x7fb3540a7740 legacy=0x7fb35407e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:20.185 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:20.171+0000 7fb351d9b700 1 -- 172.21.15.42:0/1853756490 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2963299343 0 0) 0x7fb3340030b0 con 0x7fb3540a7740 2019-12-05T06:50:20.185 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:20.171+0000 7fb35259c700 1 -- 172.21.15.42:0/1853756490 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3540a0710 con 0x7fb35408df80 2019-12-05T06:50:20.185 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:20.171+0000 7fb35359e700 1 --2- 172.21.15.42:0/1853756490 >> v2:172.21.15.40:3300/0 conn(0x7fb35409f830 0x7fb35414efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:50:20.185 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:20.171+0000 7fb35259c700 1 --2- 172.21.15.42:0/1853756490 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb35408df80 0x7fb35409f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fb34c008b10 tx=0x7fb33c0019d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:20.185 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:20.175+0000 7fb35259c700 1 --2- 172.21.15.42:0/1853756490 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb35408df80 0x7fb35409f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fb34c008b10 tx=0x7fb33c0019d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:20.186 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:20.175+0000 7fb35259c700 1 --2- 172.21.15.42:0/1853756490 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb35408df80 0x7fb35409f2e0 secure :-1 s=READY pgs=734 cs=0 l=1 rx=0x7fb34c008b10 tx=0x7fb33c0019d0).ready entity=mon.2 client_cookie=8cab23173c6f457b server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:50:20.186 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:20.175+0000 7fb35259c700 1 --2- 172.21.15.42:0/1853756490 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb35408df80 0x7fb35409f2e0 secure :-1 s=THROTTLE_DONE pgs=734 cs=0 l=1 rx=0x7fb34c008b10 tx=0x7fb33c0019d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:20.186 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:20.175+0000 7fb351d9b700 1 -- 172.21.15.42:0/1853756490 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb33c001750 con 0x7fb35408df80 2019-12-05T06:50:20.186 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:20.175+0000 7fb35259c700 1 --2- 172.21.15.42:0/1853756490 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb35408df80 0x7fb35409f2e0 secure :-1 s=THROTTLE_DONE pgs=734 cs=0 l=1 rx=0x7fb34c008b10 tx=0x7fb33c0019d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:20.186 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:20.175+0000 7fb351d9b700 1 -- 172.21.15.42:0/1853756490 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb33c001560 con 0x7fb35408df80 2019-12-05T06:50:20.186 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:20.175+0000 7fb35259c700 1 --2- 172.21.15.42:0/1853756490 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb35408df80 0x7fb35409f2e0 secure :-1 s=THROTTLE_DONE pgs=734 cs=0 l=1 rx=0x7fb34c008b10 tx=0x7fb33c0019d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:20.187 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:20.175+0000 7fb351d9b700 1 -- 172.21.15.42:0/1853756490 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb34c0109f0 con 0x7fb35408df80 2019-12-05T06:50:20.187 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:20.175+0000 7fb358ffb700 1 -- 172.21.15.42:0/1853756490 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb35408df80 msgr2=0x7fb35409f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:20.187 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:20.175+0000 7fb358ffb700 1 --2- 172.21.15.42:0/1853756490 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb35408df80 0x7fb35409f2e0 secure :-1 s=READY pgs=734 cs=0 l=1 rx=0x7fb34c008b10 tx=0x7fb33c0019d0).stop 2019-12-05T06:50:20.187 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:20.175+0000 7fb358ffb700 1 -- 172.21.15.42:0/1853756490 shutdown_connections 2019-12-05T06:50:20.187 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:20.175+0000 7fb358ffb700 1 --2- 172.21.15.42:0/1853756490 >> v2:172.21.15.40:3300/0 conn(0x7fb35409f830 0x7fb35414efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:50:20.188 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:20.175+0000 7fb358ffb700 1 --2- 172.21.15.42:0/1853756490 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb35408df80 0x7fb35409f2e0 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:20.188 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:20.175+0000 7fb358ffb700 1 -- 172.21.15.42:0/1853756490 shutdown_connections 2019-12-05T06:50:20.188 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:20.175+0000 7fb358ffb700 1 -- 172.21.15.42:0/1853756490 wait complete. 2019-12-05T06:50:20.188 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:20.175+0000 7fb358ffb700 1 -- 172.21.15.42:0/1853756490 >> 172.21.15.42:0/1853756490 conn(0x7fb3540aabe0 msgr2=0x7fb3540a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:50:20.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:20.179+0000 7fb358ffb700 1 Processor -- start 2019-12-05T06:50:20.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:20.179+0000 7fb358ffb700 1 -- start start 2019-12-05T06:50:20.191 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:20.179+0000 7fb358ffb700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb35411ba70 0x7fb35412d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:50:20.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:20.179+0000 7fb358ffb700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb354120470 0x7fb3540a51d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:50:20.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:20.179+0000 7fb358ffb700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3540a4b00 con 0x7fb35412dbf0 2019-12-05T06:50:20.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:20.179+0000 7fb358ffb700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb35411bf80 con 0x7fb354120470 2019-12-05T06:50:20.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:20.179+0000 7fb358ffb700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb35411c220 con 0x7fb35411ba70 2019-12-05T06:50:20.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:20.179+0000 7fb35259c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb354120470 0x7fb3540a51d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:20.192 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:20.179+0000 7fb35259c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb354120470 0x7fb3540a51d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:38398/0 (socket says 172.21.15.42:38398) 2019-12-05T06:50:20.193 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:20.179+0000 7fb35259c700 1 -- 172.21.15.42:0/829730234 learned_addr learned my addr 172.21.15.42:0/829730234 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:50:20.193 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:20.179+0000 7fb352d9d700 1 --2- 172.21.15.42:0/829730234 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb35411ba70 0x7fb35412d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:20.193 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:20.179+0000 7fb35259c700 1 -- 172.21.15.42:0/829730234 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb35411ba70 msgr2=0x7fb35412d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:20.193 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:20.179+0000 7fb35259c700 1 --2- 172.21.15.42:0/829730234 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb35411ba70 0x7fb35412d6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:20.193 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:20.179+0000 7fb35259c700 1 -- 172.21.15.42:0/829730234 >> v1:172.21.15.40:6789/0 conn(0x7fb35412dbf0 legacy=0x7fb35411a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:20.193 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:20.179+0000 7fb35259c700 1 -- 172.21.15.42:0/829730234 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb35412cfc0 con 0x7fb354120470 2019-12-05T06:50:20.193 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:20.179+0000 7fb35259c700 1 --2- 172.21.15.42:0/829730234 >> v2:172.21.15.42:3300/0 conn(0x7fb354120470 0x7fb3540a51d0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fb35411c140 tx=0x7fb34c010f40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:20.194 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:20.179+0000 7fb35259c700 1 --2- 172.21.15.42:0/829730234 >> v2:172.21.15.42:3300/0 conn(0x7fb354120470 0x7fb3540a51d0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fb35411c140 tx=0x7fb34c010f40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:20.194 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:20.179+0000 7fb35259c700 1 --2- 172.21.15.42:0/829730234 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb354120470 0x7fb3540a51d0 secure :-1 s=READY pgs=3324 cs=0 l=1 rx=0x7fb35411c140 tx=0x7fb34c010f40).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:50:20.194 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:20.179+0000 7fb35259c700 1 --2- 172.21.15.42:0/829730234 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb354120470 0x7fb3540a51d0 secure :-1 s=THROTTLE_DONE pgs=3324 cs=0 l=1 rx=0x7fb35411c140 tx=0x7fb34c010f40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:20.194 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:20.179+0000 7fb35259c700 1 --2- 172.21.15.42:0/829730234 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb354120470 0x7fb3540a51d0 secure :-1 s=THROTTLE_DONE pgs=3324 cs=0 l=1 rx=0x7fb35411c140 tx=0x7fb34c010f40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:20.194 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:20.179+0000 7fb34b7fe700 1 -- 172.21.15.42:0/829730234 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb34c00f600 con 0x7fb354120470 2019-12-05T06:50:20.194 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:20.179+0000 7fb35259c700 1 --2- 172.21.15.42:0/829730234 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb354120470 0x7fb3540a51d0 secure :-1 s=THROTTLE_DONE pgs=3324 cs=0 l=1 rx=0x7fb35411c140 tx=0x7fb34c010f40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:20.194 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:20.179+0000 7fb34b7fe700 1 -- 172.21.15.42:0/829730234 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb34c008d40 con 0x7fb354120470 2019-12-05T06:50:20.195 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:20.179+0000 7fb34b7fe700 1 -- 172.21.15.42:0/829730234 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb34c0058c0 con 0x7fb354120470 2019-12-05T06:50:20.195 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:20.179+0000 7fb358ffb700 1 -- 172.21.15.42:0/829730234 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb354117010 con 0x7fb354120470 2019-12-05T06:50:20.195 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:20.179+0000 7fb358ffb700 1 -- 172.21.15.42:0/829730234 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3540a5940 con 0x7fb354120470 2019-12-05T06:50:20.195 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:20.179+0000 7fb35259c700 1 --2- 172.21.15.42:0/829730234 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb354120470 0x7fb3540a51d0 secure :-1 s=THROTTLE_DONE pgs=3324 cs=0 l=1 rx=0x7fb35411c140 tx=0x7fb34c010f40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:20.195 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:20.183+0000 7fb34b7fe700 1 -- 172.21.15.42:0/829730234 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fb34c005b90 con 0x7fb354120470 2019-12-05T06:50:20.195 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:20.183+0000 7fb35259c700 1 --2- 172.21.15.42:0/829730234 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb354120470 0x7fb3540a51d0 secure :-1 s=THROTTLE_DONE pgs=3324 cs=0 l=1 rx=0x7fb35411c140 tx=0x7fb34c010f40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:20.196 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:20.183+0000 7fb34b7fe700 1 --2- 172.21.15.42:0/829730234 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb32c03ae40 0x7fb32c03d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:50:20.196 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:20.183+0000 7fb34b7fe700 1 -- 172.21.15.42:0/829730234 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 12309+0+0 (secure 0 0 0) 0x7fb34c04aba0 con 0x7fb354120470 2019-12-05T06:50:20.196 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:20.183+0000 7fb352d9d700 1 --2- 172.21.15.42:0/829730234 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb32c03ae40 0x7fb32c03d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:20.196 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:20.183+0000 7fb352d9d700 1 --2- 172.21.15.42:0/829730234 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb32c03ae40 0x7fb32c03d350 crc :-1 s=READY pgs=1394 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:50:20.196 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:20.187+0000 7fb358ffb700 1 -- 172.21.15.42:0/829730234 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb35405d110 con 0x7fb354120470 2019-12-05T06:50:20.200 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:20.195+0000 7fb35259c700 1 --2- 172.21.15.42:0/829730234 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb354120470 0x7fb3540a51d0 secure :-1 s=THROTTLE_DONE pgs=3324 cs=0 l=1 rx=0x7fb35411c140 tx=0x7fb34c010f40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:20.201 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:20.195+0000 7fb34b7fe700 1 -- 172.21.15.42:0/829730234 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fb34c031e80 con 0x7fb354120470 2019-12-05T06:50:20.360 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:20.351+0000 7fb358ffb700 1 -- 172.21.15.42:0/829730234 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb35405b8c0 con 0x7fb354120470 2019-12-05T06:50:20.360 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:20.351+0000 7fb35259c700 1 --2- 172.21.15.42:0/829730234 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb354120470 0x7fb3540a51d0 secure :-1 s=THROTTLE_DONE pgs=3324 cs=0 l=1 rx=0x7fb35411c140 tx=0x7fb34c010f40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:20.360 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:20.355+0000 7fb34b7fe700 1 -- 172.21.15.42:0/829730234 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 400 v400) v1 ==== 95+0+1775 (secure 0 0 0) 0x7fb34c02f840 con 0x7fb354120470 2019-12-05T06:50:20.361 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 400 2019-12-05T06:50:20.361 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:50:20.361 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":400,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":400,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:50:15.404443+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":122,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8537},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8537":{"gid":8537,"name":"a","rank":0,"incarnation":400,"state":"up:replay","state_seq":2,"addr":"172.21.15.42:6835/1881703925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1881703925},{"type":"v1","addr":"172.21.15.42:6835","nonce":1881703925}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:50:20.365 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:20.355+0000 7fb358ffb700 1 -- 172.21.15.42:0/829730234 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb32c03ae40 msgr2=0x7fb32c03d350 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:20.365 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:20.355+0000 7fb358ffb700 1 --2- 172.21.15.42:0/829730234 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb32c03ae40 0x7fb32c03d350 crc :-1 s=READY pgs=1394 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:20.366 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:20.355+0000 7fb358ffb700 1 -- 172.21.15.42:0/829730234 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb354120470 msgr2=0x7fb3540a51d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:20.366 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:20.355+0000 7fb358ffb700 1 --2- 172.21.15.42:0/829730234 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb354120470 0x7fb3540a51d0 secure :-1 s=READY pgs=3324 cs=0 l=1 rx=0x7fb35411c140 tx=0x7fb34c010f40).stop 2019-12-05T06:50:20.366 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:20.355+0000 7fb358ffb700 1 -- 172.21.15.42:0/829730234 shutdown_connections 2019-12-05T06:50:20.366 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:20.355+0000 7fb358ffb700 1 --2- 172.21.15.42:0/829730234 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fb32c03ae40 0x7fb32c03d350 unknown :-1 s=CLOSED pgs=1394 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:20.367 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:20.355+0000 7fb358ffb700 1 --2- 172.21.15.42:0/829730234 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb35411ba70 0x7fb35412d6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:20.367 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:20.355+0000 7fb358ffb700 1 --2- 172.21.15.42:0/829730234 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb354120470 0x7fb3540a51d0 unknown :-1 s=CLOSED pgs=3324 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:20.367 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:20.355+0000 7fb358ffb700 1 -- 172.21.15.42:0/829730234 shutdown_connections 2019-12-05T06:50:20.367 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:20.355+0000 7fb358ffb700 1 -- 172.21.15.42:0/829730234 wait complete. 2019-12-05T06:50:20.367 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:20.355+0000 7fb358ffb700 1 -- 172.21.15.42:0/829730234 >> 172.21.15.42:0/829730234 conn(0x7fb3540aabe0 msgr2=0x7fb354059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:50:20.372 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2019-12-05T06:50:20.778 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:50:20.782 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:50:22.373 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:50:22.646 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:22.635+0000 7f11bd82d700 1 Processor -- start 2019-12-05T06:50:22.647 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:22.635+0000 7f11bd82d700 1 -- start start 2019-12-05T06:50:22.647 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:22.635+0000 7f11bd82d700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f11b808df70 0x7f11b809f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:50:22.647 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:22.635+0000 7f11bd82d700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f11b809f820 0x7f11b814efd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:50:22.647 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:22.635+0000 7f11bd82d700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11b809fc50 con 0x7f11b80a7730 2019-12-05T06:50:22.647 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:22.635+0000 7f11bd82d700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11b80a0af0 con 0x7f11b808df70 2019-12-05T06:50:22.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:22.635+0000 7f11bd82d700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11b808e5a0 con 0x7f11b809f820 2019-12-05T06:50:22.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:22.635+0000 7f11b77fe700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f11b80a7730 0x7f11b807e710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:38446/0 (socket says 172.21.15.42:38446) 2019-12-05T06:50:22.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:22.635+0000 7f11b77fe700 1 -- 172.21.15.42:0/3464116103 learned_addr learned my addr 172.21.15.42:0/3464116103 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:50:22.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:22.635+0000 7f11b6ffd700 1 --2- 172.21.15.42:0/3464116103 >> v2:172.21.15.40:3300/0 conn(0x7f11b808df70 0x7f11b809f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:22.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:22.635+0000 7f11b7fff700 1 --2- 172.21.15.42:0/3464116103 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f11b809f820 0x7f11b814efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:22.648 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:22.635+0000 7f11b67fc700 1 -- 172.21.15.42:0/3464116103 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1139123416 0 0) 0x7f11b809fc50 con 0x7f11b80a7730 2019-12-05T06:50:22.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:22.635+0000 7f11b67fc700 1 -- 172.21.15.42:0/3464116103 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11a40030b0 con 0x7f11b80a7730 2019-12-05T06:50:22.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:22.635+0000 7f11b67fc700 1 -- 172.21.15.42:0/3464116103 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1359585982 0 0) 0x7f11a40030b0 con 0x7f11b80a7730 2019-12-05T06:50:22.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:22.635+0000 7f11b67fc700 1 -- 172.21.15.42:0/3464116103 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f11b809f820 msgr2=0x7f11b814efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:50:22.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:22.635+0000 7f11b67fc700 1 --2- 172.21.15.42:0/3464116103 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f11b809f820 0x7f11b814efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:50:22.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:22.635+0000 7f11b67fc700 1 -- 172.21.15.42:0/3464116103 >> v2:172.21.15.40:3300/0 conn(0x7f11b808df70 msgr2=0x7f11b809f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:50:22.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:22.635+0000 7f11b67fc700 1 --2- 172.21.15.42:0/3464116103 >> v2:172.21.15.40:3300/0 conn(0x7f11b808df70 0x7f11b809f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:50:22.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:22.635+0000 7f11b67fc700 1 -- 172.21.15.42:0/3464116103 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11b80a0700 con 0x7f11b80a7730 2019-12-05T06:50:22.650 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:22.635+0000 7f11b67fc700 1 -- 172.21.15.42:0/3464116103 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7f11a8001ad0 con 0x7f11b80a7730 2019-12-05T06:50:22.650 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:22.635+0000 7f11b67fc700 1 -- 172.21.15.42:0/3464116103 >> v1:172.21.15.42:6789/0 conn(0x7f11b80a7730 legacy=0x7f11b807e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:22.650 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:22.635+0000 7f11b67fc700 1 --2- 172.21.15.42:0/3464116103 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f11a40036e0 0x7f11a4003b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:50:22.650 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:22.635+0000 7f11b67fc700 1 --2- 172.21.15.42:0/3464116103 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f11a4004420 0x7f11a4004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:50:22.650 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:22.635+0000 7f11b67fc700 1 --2- 172.21.15.42:0/3464116103 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f11a4004da0 0x7f11a404f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:50:22.650 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:22.635+0000 7f11b67fc700 1 -- 172.21.15.42:0/3464116103 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11b80a0af0 con 0x7f11a4004420 2019-12-05T06:50:22.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:22.635+0000 7f11b67fc700 1 -- 172.21.15.42:0/3464116103 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11b808e5a0 con 0x7f11a4004da0 2019-12-05T06:50:22.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:22.635+0000 7f11b67fc700 1 -- 172.21.15.42:0/3464116103 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11a4005250 con 0x7f11a40036e0 2019-12-05T06:50:22.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:22.639+0000 7f11b7fff700 1 --2- 172.21.15.42:0/3464116103 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f11a4004da0 0x7f11a404f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:22.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:22.639+0000 7f11b6ffd700 1 --2- 172.21.15.42:0/3464116103 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f11a4004420 0x7f11a4004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:22.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:22.639+0000 7f11b7fff700 1 -- 172.21.15.42:0/3464116103 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f11a40036e0 msgr2=0x7f11a4003b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:50:22.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:22.639+0000 7f11b77fe700 1 --2- 172.21.15.42:0/3464116103 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f11a40036e0 0x7f11a4003b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:22.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:22.639+0000 7f11b7fff700 1 --2- 172.21.15.42:0/3464116103 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f11a40036e0 0x7f11a4003b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:50:22.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:22.639+0000 7f11b7fff700 1 -- 172.21.15.42:0/3464116103 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f11a4004420 msgr2=0x7f11a4004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:50:22.652 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:22.639+0000 7f11b7fff700 1 --2- 172.21.15.42:0/3464116103 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f11a4004420 0x7f11a4004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:50:22.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:22.639+0000 7f11b7fff700 1 -- 172.21.15.42:0/3464116103 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11a404fce0 con 0x7f11a4004da0 2019-12-05T06:50:22.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:22.639+0000 7f11b6ffd700 1 --2- 172.21.15.42:0/3464116103 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f11a4004420 0x7f11a4004850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:50:22.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:22.639+0000 7f11b7fff700 1 --2- 172.21.15.42:0/3464116103 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f11a4004da0 0x7f11a404f680 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f11a4004390 tx=0x7f11b000ffb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:22.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:22.639+0000 7f11b7fff700 1 --2- 172.21.15.42:0/3464116103 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f11a4004da0 0x7f11a404f680 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f11a4004390 tx=0x7f11b000ffb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:22.653 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:22.639+0000 7f11b7fff700 1 --2- 172.21.15.42:0/3464116103 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f11a4004da0 0x7f11a404f680 secure :-1 s=READY pgs=3326 cs=0 l=1 rx=0x7f11a4004390 tx=0x7f11b000ffb0).ready entity=mon.1 client_cookie=5ac5e0377ef09a40 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:50:22.654 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:22.639+0000 7f11b7fff700 1 --2- 172.21.15.42:0/3464116103 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f11a4004da0 0x7f11a404f680 secure :-1 s=THROTTLE_DONE pgs=3326 cs=0 l=1 rx=0x7f11a4004390 tx=0x7f11b000ffb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:22.654 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:22.639+0000 7f11b67fc700 1 -- 172.21.15.42:0/3464116103 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f11b0010ca0 con 0x7f11a4004da0 2019-12-05T06:50:22.654 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:22.639+0000 7f11b7fff700 1 --2- 172.21.15.42:0/3464116103 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f11a4004da0 0x7f11a404f680 secure :-1 s=THROTTLE_DONE pgs=3326 cs=0 l=1 rx=0x7f11a4004390 tx=0x7f11b000ffb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:22.654 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:22.639+0000 7f11b7fff700 1 --2- 172.21.15.42:0/3464116103 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f11a4004da0 0x7f11a404f680 secure :-1 s=THROTTLE_DONE pgs=3326 cs=0 l=1 rx=0x7f11a4004390 tx=0x7f11b000ffb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:22.654 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:22.639+0000 7f11b67fc700 1 -- 172.21.15.42:0/3464116103 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11b00108c0 con 0x7f11a4004da0 2019-12-05T06:50:22.654 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:22.639+0000 7f11b67fc700 1 -- 172.21.15.42:0/3464116103 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f11b0008b90 con 0x7f11a4004da0 2019-12-05T06:50:22.655 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:22.639+0000 7f11bd82d700 1 -- 172.21.15.42:0/3464116103 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f11a4004da0 msgr2=0x7f11a404f680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:22.655 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:22.639+0000 7f11bd82d700 1 --2- 172.21.15.42:0/3464116103 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f11a4004da0 0x7f11a404f680 secure :-1 s=READY pgs=3326 cs=0 l=1 rx=0x7f11a4004390 tx=0x7f11b000ffb0).stop 2019-12-05T06:50:22.655 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:22.639+0000 7f11b7fff700 1 -- 172.21.15.42:0/3464116103 reap_dead start 2019-12-05T06:50:22.655 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:22.639+0000 7f11bd82d700 1 -- 172.21.15.42:0/3464116103 shutdown_connections 2019-12-05T06:50:22.655 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:22.639+0000 7f11bd82d700 1 -- 172.21.15.42:0/3464116103 shutdown_connections 2019-12-05T06:50:22.655 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:22.639+0000 7f11bd82d700 1 -- 172.21.15.42:0/3464116103 wait complete. 2019-12-05T06:50:22.656 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:22.639+0000 7f11bd82d700 1 -- 172.21.15.42:0/3464116103 >> 172.21.15.42:0/3464116103 conn(0x7f11b80aabd0 msgr2=0x7f11b80a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:50:22.659 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:22.639+0000 7f11bd82d700 1 Processor -- start 2019-12-05T06:50:22.660 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:22.639+0000 7f11bd82d700 1 -- start start 2019-12-05T06:50:22.660 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:22.643+0000 7f11bd82d700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f11b80a4b70 0x7f11b80a6330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:50:22.661 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:22.643+0000 7f11bd82d700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f11b80a6880 0x7f11b807ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:50:22.661 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:22.643+0000 7f11bd82d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f11b807d3e0 0x7f11b8116e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:50:22.661 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:22.643+0000 7f11bd82d700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11b80a6cb0 con 0x7f11b80a4b70 2019-12-05T06:50:22.661 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:22.643+0000 7f11bd82d700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11b81166b0 con 0x7f11b807d3e0 2019-12-05T06:50:22.661 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:22.643+0000 7f11bd82d700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11b8116950 con 0x7f11b80a6880 2019-12-05T06:50:22.662 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:22.643+0000 7f11b7fff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f11b807d3e0 0x7f11b8116e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:22.662 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:22.643+0000 7f11b77fe700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f11b80a4b70 0x7f11b80a6330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:22.663 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:22.643+0000 7f11b7fff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f11b807d3e0 0x7f11b8116e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:38424/0 (socket says 172.21.15.42:38424) 2019-12-05T06:50:22.663 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:22.643+0000 7f11b6ffd700 1 --2- 172.21.15.42:0/179261922 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f11b80a6880 0x7f11b807ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:22.663 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:22.643+0000 7f11b7fff700 1 -- 172.21.15.42:0/179261922 learned_addr learned my addr 172.21.15.42:0/179261922 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:50:22.663 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:22.643+0000 7f11b7fff700 1 -- 172.21.15.42:0/179261922 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f11b80a6880 msgr2=0x7f11b807ce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:22.663 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:22.643+0000 7f11b7fff700 1 --2- 172.21.15.42:0/179261922 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f11b80a6880 0x7f11b807ce90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:22.663 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:22.643+0000 7f11b7fff700 1 -- 172.21.15.42:0/179261922 >> v2:172.21.15.40:3300/0 conn(0x7f11b80a4b70 msgr2=0x7f11b80a6330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:22.664 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:22.643+0000 7f11b7fff700 1 --2- 172.21.15.42:0/179261922 >> v2:172.21.15.40:3300/0 conn(0x7f11b80a4b70 0x7f11b80a6330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:22.664 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:22.643+0000 7f11b7fff700 1 -- 172.21.15.42:0/179261922 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11b807d810 con 0x7f11b807d3e0 2019-12-05T06:50:22.664 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:22.643+0000 7f11b7fff700 1 --2- 172.21.15.42:0/179261922 >> v2:172.21.15.42:3300/0 conn(0x7f11b807d3e0 0x7f11b8116e90 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f11b80a0d50 tx=0x7f11b0007a40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:22.664 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:22.643+0000 7f11b7fff700 1 --2- 172.21.15.42:0/179261922 >> v2:172.21.15.42:3300/0 conn(0x7f11b807d3e0 0x7f11b8116e90 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f11b80a0d50 tx=0x7f11b0007a40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:22.664 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:22.643+0000 7f11b7fff700 1 --2- 172.21.15.42:0/179261922 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f11b807d3e0 0x7f11b8116e90 secure :-1 s=READY pgs=3327 cs=0 l=1 rx=0x7f11b80a0d50 tx=0x7f11b0007a40).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:50:22.665 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:22.643+0000 7f11b7fff700 1 --2- 172.21.15.42:0/179261922 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f11b807d3e0 0x7f11b8116e90 secure :-1 s=THROTTLE_DONE pgs=3327 cs=0 l=1 rx=0x7f11b80a0d50 tx=0x7f11b0007a40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:22.665 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:22.643+0000 7f11b7fff700 1 --2- 172.21.15.42:0/179261922 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f11b807d3e0 0x7f11b8116e90 secure :-1 s=THROTTLE_DONE pgs=3327 cs=0 l=1 rx=0x7f11b80a0d50 tx=0x7f11b0007a40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:22.665 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:22.643+0000 7f11affff700 1 -- 172.21.15.42:0/179261922 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f11b0010d40 con 0x7f11b807d3e0 2019-12-05T06:50:22.665 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:22.643+0000 7f11b7fff700 1 --2- 172.21.15.42:0/179261922 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f11b807d3e0 0x7f11b8116e90 secure :-1 s=THROTTLE_DONE pgs=3327 cs=0 l=1 rx=0x7f11b80a0d50 tx=0x7f11b0007a40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:22.666 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:22.643+0000 7f11affff700 1 -- 172.21.15.42:0/179261922 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11b00087b0 con 0x7f11b807d3e0 2019-12-05T06:50:22.666 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:22.643+0000 7f11affff700 1 -- 172.21.15.42:0/179261922 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f11b000f5d0 con 0x7f11b807d3e0 2019-12-05T06:50:22.666 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:22.643+0000 7f11bd82d700 1 -- 172.21.15.42:0/179261922 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11b81173e0 con 0x7f11b807d3e0 2019-12-05T06:50:22.666 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:22.643+0000 7f11bd82d700 1 -- 172.21.15.42:0/179261922 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11b8117650 con 0x7f11b807d3e0 2019-12-05T06:50:22.666 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:22.643+0000 7f11b7fff700 1 --2- 172.21.15.42:0/179261922 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f11b807d3e0 0x7f11b8116e90 secure :-1 s=THROTTLE_DONE pgs=3327 cs=0 l=1 rx=0x7f11b80a0d50 tx=0x7f11b0007a40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:22.666 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:22.643+0000 7f11affff700 1 -- 172.21.15.42:0/179261922 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f11b0005540 con 0x7f11b807d3e0 2019-12-05T06:50:22.667 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:22.643+0000 7f11b7fff700 1 --2- 172.21.15.42:0/179261922 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f11b807d3e0 0x7f11b8116e90 secure :-1 s=THROTTLE_DONE pgs=3327 cs=0 l=1 rx=0x7f11b80a0d50 tx=0x7f11b0007a40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:22.667 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:22.643+0000 7f11affff700 1 --2- 172.21.15.42:0/179261922 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f119c03ad70 0x7f119c03d280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:50:22.667 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:22.643+0000 7f11affff700 1 -- 172.21.15.42:0/179261922 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 12309+0+0 (secure 0 0 0) 0x7f11b004ac40 con 0x7f11b807d3e0 2019-12-05T06:50:22.667 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:22.643+0000 7f11b77fe700 1 --2- 172.21.15.42:0/179261922 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f119c03ad70 0x7f119c03d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:22.667 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:22.643+0000 7f11b77fe700 1 --2- 172.21.15.42:0/179261922 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f119c03ad70 0x7f119c03d280 crc :-1 s=READY pgs=1395 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:50:22.667 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:22.651+0000 7f11bd82d700 1 -- 172.21.15.42:0/179261922 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11b805d100 con 0x7f11b807d3e0 2019-12-05T06:50:22.668 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:22.655+0000 7f11b7fff700 1 --2- 172.21.15.42:0/179261922 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f11b807d3e0 0x7f11b8116e90 secure :-1 s=THROTTLE_DONE pgs=3327 cs=0 l=1 rx=0x7f11b80a0d50 tx=0x7f11b0007a40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:22.668 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:22.655+0000 7f11affff700 1 -- 172.21.15.42:0/179261922 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f11b0031f20 con 0x7f11b807d3e0 2019-12-05T06:50:22.820 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:22.815+0000 7f11bd82d700 1 -- 172.21.15.42:0/179261922 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f11b80a6f60 con 0x7f11b807d3e0 2019-12-05T06:50:22.821 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:22.815+0000 7f11b7fff700 1 --2- 172.21.15.42:0/179261922 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f11b807d3e0 0x7f11b8116e90 secure :-1 s=THROTTLE_DONE pgs=3327 cs=0 l=1 rx=0x7f11b80a0d50 tx=0x7f11b0007a40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:22.821 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:22.815+0000 7f11affff700 1 -- 172.21.15.42:0/179261922 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 400 v400) v1 ==== 95+0+1775 (secure 0 0 0) 0x7f11b0032ad0 con 0x7f11b807d3e0 2019-12-05T06:50:22.821 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 400 2019-12-05T06:50:22.822 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:50:22.822 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":400,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":400,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:50:15.404443+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":122,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8537},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8537":{"gid":8537,"name":"a","rank":0,"incarnation":400,"state":"up:replay","state_seq":2,"addr":"172.21.15.42:6835/1881703925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1881703925},{"type":"v1","addr":"172.21.15.42:6835","nonce":1881703925}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:50:22.824 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:22.815+0000 7f11bd82d700 1 -- 172.21.15.42:0/179261922 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f119c03ad70 msgr2=0x7f119c03d280 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:22.824 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:22.815+0000 7f11bd82d700 1 --2- 172.21.15.42:0/179261922 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f119c03ad70 0x7f119c03d280 crc :-1 s=READY pgs=1395 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:22.824 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:22.815+0000 7f11bd82d700 1 -- 172.21.15.42:0/179261922 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f11b807d3e0 msgr2=0x7f11b8116e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:22.824 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:22.815+0000 7f11bd82d700 1 --2- 172.21.15.42:0/179261922 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f11b807d3e0 0x7f11b8116e90 secure :-1 s=READY pgs=3327 cs=0 l=1 rx=0x7f11b80a0d50 tx=0x7f11b0007a40).stop 2019-12-05T06:50:22.825 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:22.815+0000 7f11bd82d700 1 -- 172.21.15.42:0/179261922 shutdown_connections 2019-12-05T06:50:22.825 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:22.815+0000 7f11bd82d700 1 --2- 172.21.15.42:0/179261922 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f119c03ad70 0x7f119c03d280 unknown :-1 s=CLOSED pgs=1395 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:22.825 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:22.815+0000 7f11bd82d700 1 --2- 172.21.15.42:0/179261922 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f11b807d3e0 0x7f11b8116e90 unknown :-1 s=CLOSED pgs=3327 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:22.825 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:22.815+0000 7f11bd82d700 1 --2- 172.21.15.42:0/179261922 >> v2:172.21.15.40:3300/0 conn(0x7f11b80a4b70 0x7f11b80a6330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:22.826 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:22.815+0000 7f11bd82d700 1 --2- 172.21.15.42:0/179261922 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f11b80a6880 0x7f11b807ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:22.826 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:22.815+0000 7f11bd82d700 1 -- 172.21.15.42:0/179261922 shutdown_connections 2019-12-05T06:50:22.826 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:22.815+0000 7f11bd82d700 1 -- 172.21.15.42:0/179261922 wait complete. 2019-12-05T06:50:22.826 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:22.815+0000 7f11bd82d700 1 -- 172.21.15.42:0/179261922 >> 172.21.15.42:0/179261922 conn(0x7f11b80a8290 msgr2=0x7f11b8118650 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:50:22.833 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2019-12-05T06:50:24.835 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:50:24.941 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:24.923+0000 7fac616ea700 1 Processor -- start 2019-12-05T06:50:24.941 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:24.923+0000 7fac616ea700 1 -- start start 2019-12-05T06:50:24.941 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:24.927+0000 7fac616ea700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fac5c08df80 0x7fac5c09f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:50:24.941 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:24.927+0000 7fac616ea700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fac5c09f830 0x7fac5c14efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:50:24.941 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:24.927+0000 7fac616ea700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac5c09fc60 con 0x7fac5c0a7740 2019-12-05T06:50:24.942 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:24.927+0000 7fac616ea700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac5c0a0b00 con 0x7fac5c09f830 2019-12-05T06:50:24.942 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:24.927+0000 7fac616ea700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac5c08e5b0 con 0x7fac5c08df80 2019-12-05T06:50:24.942 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:24.927+0000 7fac5bfff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fac5c09f830 0x7fac5c14efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:24.942 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:24.927+0000 7fac5b7fe700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fac5c0a7740 0x7fac5c07e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:38468/0 (socket says 172.21.15.42:38468) 2019-12-05T06:50:24.942 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:24.927+0000 7fac5b7fe700 1 -- 172.21.15.42:0/4191083096 learned_addr learned my addr 172.21.15.42:0/4191083096 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:50:24.942 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:24.927+0000 7fac52ffd700 1 --2- 172.21.15.42:0/4191083096 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fac5c08df80 0x7fac5c09f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:24.942 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:24.927+0000 7fac5affd700 1 -- 172.21.15.42:0/4191083096 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 982001737 0 0) 0x7fac5c09fc60 con 0x7fac5c0a7740 2019-12-05T06:50:24.943 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:24.927+0000 7fac5affd700 1 -- 172.21.15.42:0/4191083096 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac480030b0 con 0x7fac5c0a7740 2019-12-05T06:50:24.943 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:24.927+0000 7fac5bfff700 1 -- 172.21.15.42:0/4191083096 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fac5c08df80 msgr2=0x7fac5c09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:50:24.943 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:24.927+0000 7fac5bfff700 1 --2- 172.21.15.42:0/4191083096 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fac5c08df80 0x7fac5c09f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:50:24.943 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:24.927+0000 7fac5bfff700 1 -- 172.21.15.42:0/4191083096 >> v1:172.21.15.42:6789/0 conn(0x7fac5c0a7740 legacy=0x7fac5c07e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:24.943 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:24.927+0000 7fac5bfff700 1 -- 172.21.15.42:0/4191083096 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac5c0a0710 con 0x7fac5c09f830 2019-12-05T06:50:24.943 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:24.927+0000 7fac5bfff700 1 --2- 172.21.15.42:0/4191083096 >> v2:172.21.15.42:3300/0 conn(0x7fac5c09f830 0x7fac5c14efe0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fac54008b10 tx=0x7fac5400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:24.943 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:24.927+0000 7fac5bfff700 1 --2- 172.21.15.42:0/4191083096 >> v2:172.21.15.42:3300/0 conn(0x7fac5c09f830 0x7fac5c14efe0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fac54008b10 tx=0x7fac5400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:24.944 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:24.927+0000 7fac5bfff700 1 --2- 172.21.15.42:0/4191083096 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fac5c09f830 0x7fac5c14efe0 secure :-1 s=READY pgs=3329 cs=0 l=1 rx=0x7fac54008b10 tx=0x7fac5400fff0).ready entity=mon.1 client_cookie=f5e22fae51cbe24b server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:50:24.944 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:24.927+0000 7fac5bfff700 1 --2- 172.21.15.42:0/4191083096 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fac5c09f830 0x7fac5c14efe0 secure :-1 s=THROTTLE_DONE pgs=3329 cs=0 l=1 rx=0x7fac54008b10 tx=0x7fac5400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:24.944 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:24.927+0000 7fac5bfff700 1 --2- 172.21.15.42:0/4191083096 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fac5c09f830 0x7fac5c14efe0 secure :-1 s=THROTTLE_DONE pgs=3329 cs=0 l=1 rx=0x7fac54008b10 tx=0x7fac5400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:24.944 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:24.927+0000 7fac5affd700 1 -- 172.21.15.42:0/4191083096 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fac54010900 con 0x7fac5c09f830 2019-12-05T06:50:24.944 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:24.927+0000 7fac5bfff700 1 --2- 172.21.15.42:0/4191083096 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fac5c09f830 0x7fac5c14efe0 secure :-1 s=THROTTLE_DONE pgs=3329 cs=0 l=1 rx=0x7fac54008b10 tx=0x7fac5400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:24.944 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:24.927+0000 7fac5affd700 1 -- 172.21.15.42:0/4191083096 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac54008b50 con 0x7fac5c09f830 2019-12-05T06:50:24.944 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:24.927+0000 7fac5affd700 1 -- 172.21.15.42:0/4191083096 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fac540110c0 con 0x7fac5c09f830 2019-12-05T06:50:24.945 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:24.927+0000 7fac616ea700 1 -- 172.21.15.42:0/4191083096 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fac5c09f830 msgr2=0x7fac5c14efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:24.945 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:24.927+0000 7fac616ea700 1 --2- 172.21.15.42:0/4191083096 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fac5c09f830 0x7fac5c14efe0 secure :-1 s=READY pgs=3329 cs=0 l=1 rx=0x7fac54008b10 tx=0x7fac5400fff0).stop 2019-12-05T06:50:24.945 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:24.927+0000 7fac616ea700 1 -- 172.21.15.42:0/4191083096 shutdown_connections 2019-12-05T06:50:24.945 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:24.927+0000 7fac616ea700 1 --2- 172.21.15.42:0/4191083096 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fac5c09f830 0x7fac5c14efe0 unknown :-1 s=CLOSED pgs=3329 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:24.945 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:24.927+0000 7fac616ea700 1 --2- 172.21.15.42:0/4191083096 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fac5c08df80 0x7fac5c09f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:50:24.946 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:24.927+0000 7fac616ea700 1 -- 172.21.15.42:0/4191083096 shutdown_connections 2019-12-05T06:50:24.946 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:24.927+0000 7fac616ea700 1 -- 172.21.15.42:0/4191083096 wait complete. 2019-12-05T06:50:24.946 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:24.927+0000 7fac616ea700 1 -- 172.21.15.42:0/4191083096 >> 172.21.15.42:0/4191083096 conn(0x7fac5c0aabe0 msgr2=0x7fac5c0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:50:24.946 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:24.927+0000 7fac616ea700 1 Processor -- start 2019-12-05T06:50:24.946 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:24.931+0000 7fac616ea700 1 -- start start 2019-12-05T06:50:24.946 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:24.931+0000 7fac616ea700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fac5c1434a0 0x7fac5c1191f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:50:24.947 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:24.931+0000 7fac616ea700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fac5c1482e0 0x7fac5c07b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:50:24.947 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:24.931+0000 7fac616ea700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fac5c07bcf0 0x7fac5c129cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:50:24.947 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:24.931+0000 7fac616ea700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac5c1454c0 con 0x7fac5c1434a0 2019-12-05T06:50:24.947 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:24.931+0000 7fac616ea700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac5c1439b0 con 0x7fac5c07bcf0 2019-12-05T06:50:24.947 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:24.931+0000 7fac616ea700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac5c143c50 con 0x7fac5c1482e0 2019-12-05T06:50:24.948 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:24.931+0000 7fac5bfff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fac5c07bcf0 0x7fac5c129cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:24.948 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:24.931+0000 7fac5bfff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fac5c07bcf0 0x7fac5c129cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:38440/0 (socket says 172.21.15.42:38440) 2019-12-05T06:50:24.948 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:24.931+0000 7fac5bfff700 1 -- 172.21.15.42:0/1787611373 learned_addr learned my addr 172.21.15.42:0/1787611373 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:50:24.948 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:24.931+0000 7fac52ffd700 1 --2- 172.21.15.42:0/1787611373 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fac5c1482e0 0x7fac5c07b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:24.948 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:24.931+0000 7fac5b7fe700 1 --2- 172.21.15.42:0/1787611373 >> v2:172.21.15.40:3300/0 conn(0x7fac5c1434a0 0x7fac5c1191f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:24.949 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:24.931+0000 7fac5bfff700 1 -- 172.21.15.42:0/1787611373 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fac5c1482e0 msgr2=0x7fac5c07b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:24.949 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:24.931+0000 7fac5bfff700 1 --2- 172.21.15.42:0/1787611373 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fac5c1482e0 0x7fac5c07b7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:24.949 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:24.931+0000 7fac5bfff700 1 -- 172.21.15.42:0/1787611373 >> v2:172.21.15.40:3300/0 conn(0x7fac5c1434a0 msgr2=0x7fac5c1191f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:24.949 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:24.931+0000 7fac5bfff700 1 --2- 172.21.15.42:0/1787611373 >> v2:172.21.15.40:3300/0 conn(0x7fac5c1434a0 0x7fac5c1191f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:24.949 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:24.931+0000 7fac5bfff700 1 -- 172.21.15.42:0/1787611373 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac5c148710 con 0x7fac5c07bcf0 2019-12-05T06:50:24.949 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:24.931+0000 7fac5b7fe700 1 --2- 172.21.15.42:0/1787611373 >> v2:172.21.15.40:3300/0 conn(0x7fac5c1434a0 0x7fac5c1191f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:50:24.950 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:24.931+0000 7fac5bfff700 1 --2- 172.21.15.42:0/1787611373 >> v2:172.21.15.42:3300/0 conn(0x7fac5c07bcf0 0x7fac5c129cd0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fac5c143b70 tx=0x7fac54008870).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:24.950 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:24.931+0000 7fac5bfff700 1 --2- 172.21.15.42:0/1787611373 >> v2:172.21.15.42:3300/0 conn(0x7fac5c07bcf0 0x7fac5c129cd0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fac5c143b70 tx=0x7fac54008870).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:24.950 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:24.931+0000 7fac5bfff700 1 --2- 172.21.15.42:0/1787611373 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fac5c07bcf0 0x7fac5c129cd0 secure :-1 s=READY pgs=3330 cs=0 l=1 rx=0x7fac5c143b70 tx=0x7fac54008870).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:50:24.950 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:24.931+0000 7fac5bfff700 1 --2- 172.21.15.42:0/1787611373 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fac5c07bcf0 0x7fac5c129cd0 secure :-1 s=THROTTLE_DONE pgs=3330 cs=0 l=1 rx=0x7fac5c143b70 tx=0x7fac54008870).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:24.950 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:24.931+0000 7fac5bfff700 1 --2- 172.21.15.42:0/1787611373 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fac5c07bcf0 0x7fac5c129cd0 secure :-1 s=THROTTLE_DONE pgs=3330 cs=0 l=1 rx=0x7fac5c143b70 tx=0x7fac54008870).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:24.950 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:24.931+0000 7fac58ff9700 1 -- 172.21.15.42:0/1787611373 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fac54005ab0 con 0x7fac5c07bcf0 2019-12-05T06:50:24.950 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:24.931+0000 7fac5bfff700 1 --2- 172.21.15.42:0/1787611373 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fac5c07bcf0 0x7fac5c129cd0 secure :-1 s=THROTTLE_DONE pgs=3330 cs=0 l=1 rx=0x7fac5c143b70 tx=0x7fac54008870).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:24.950 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:24.931+0000 7fac58ff9700 1 -- 172.21.15.42:0/1787611373 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac54010b00 con 0x7fac5c07bcf0 2019-12-05T06:50:24.951 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:24.931+0000 7fac58ff9700 1 -- 172.21.15.42:0/1787611373 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fac5400f470 con 0x7fac5c07bcf0 2019-12-05T06:50:24.951 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:24.931+0000 7fac616ea700 1 -- 172.21.15.42:0/1787611373 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac5c07c120 con 0x7fac5c07bcf0 2019-12-05T06:50:24.951 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:24.931+0000 7fac616ea700 1 -- 172.21.15.42:0/1787611373 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac5c12a440 con 0x7fac5c07bcf0 2019-12-05T06:50:24.951 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:24.931+0000 7fac5bfff700 1 --2- 172.21.15.42:0/1787611373 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fac5c07bcf0 0x7fac5c129cd0 secure :-1 s=THROTTLE_DONE pgs=3330 cs=0 l=1 rx=0x7fac5c143b70 tx=0x7fac54008870).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:24.951 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:24.931+0000 7fac58ff9700 1 -- 172.21.15.42:0/1787611373 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fac5400f670 con 0x7fac5c07bcf0 2019-12-05T06:50:24.951 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:24.931+0000 7fac5bfff700 1 --2- 172.21.15.42:0/1787611373 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fac5c07bcf0 0x7fac5c129cd0 secure :-1 s=THROTTLE_DONE pgs=3330 cs=0 l=1 rx=0x7fac5c143b70 tx=0x7fac54008870).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:24.951 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:24.931+0000 7fac58ff9700 1 --2- 172.21.15.42:0/1787611373 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fac4003aec0 0x7fac4003d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:50:24.952 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:24.931+0000 7fac58ff9700 1 -- 172.21.15.42:0/1787611373 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 12309+0+0 (secure 0 0 0) 0x7fac5404af40 con 0x7fac5c07bcf0 2019-12-05T06:50:24.952 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:24.931+0000 7fac5b7fe700 1 --2- 172.21.15.42:0/1787611373 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fac4003aec0 0x7fac4003d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:24.952 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:24.931+0000 7fac616ea700 1 -- 172.21.15.42:0/1787611373 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac5c05d110 con 0x7fac5c07bcf0 2019-12-05T06:50:24.952 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:24.931+0000 7fac5b7fe700 1 --2- 172.21.15.42:0/1787611373 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fac4003aec0 0x7fac4003d3d0 crc :-1 s=READY pgs=1396 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:50:24.953 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:24.935+0000 7fac5bfff700 1 --2- 172.21.15.42:0/1787611373 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fac5c07bcf0 0x7fac5c129cd0 secure :-1 s=THROTTLE_DONE pgs=3330 cs=0 l=1 rx=0x7fac5c143b70 tx=0x7fac54008870).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:24.953 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:24.935+0000 7fac58ff9700 1 -- 172.21.15.42:0/1787611373 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fac5402f0c0 con 0x7fac5c07bcf0 2019-12-05T06:50:25.100 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:25.095+0000 7fac616ea700 1 -- 172.21.15.42:0/1787611373 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fac5c05b8c0 con 0x7fac5c07bcf0 2019-12-05T06:50:25.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:25.095+0000 7fac5bfff700 1 --2- 172.21.15.42:0/1787611373 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fac5c07bcf0 0x7fac5c129cd0 secure :-1 s=THROTTLE_DONE pgs=3330 cs=0 l=1 rx=0x7fac5c143b70 tx=0x7fac54008870).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:25.102 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:25.095+0000 7fac58ff9700 1 -- 172.21.15.42:0/1787611373 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 402 v402) v1 ==== 95+0+1776 (secure 0 0 0) 0x7fac54007b10 con 0x7fac5c07bcf0 2019-12-05T06:50:25.102 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 402 2019-12-05T06:50:25.102 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:50:25.103 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":402,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":402,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:50:24.882446+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":122,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8537},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8537":{"gid":8537,"name":"a","rank":0,"incarnation":400,"state":"up:rejoin","state_seq":19,"addr":"172.21.15.42:6835/1881703925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1881703925},{"type":"v1","addr":"172.21.15.42:6835","nonce":1881703925}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:50:25.105 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:25.095+0000 7fac616ea700 1 -- 172.21.15.42:0/1787611373 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fac4003aec0 msgr2=0x7fac4003d3d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:25.105 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:25.095+0000 7fac616ea700 1 --2- 172.21.15.42:0/1787611373 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fac4003aec0 0x7fac4003d3d0 crc :-1 s=READY pgs=1396 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:25.105 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:25.095+0000 7fac616ea700 1 -- 172.21.15.42:0/1787611373 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fac5c07bcf0 msgr2=0x7fac5c129cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:25.106 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:25.095+0000 7fac616ea700 1 --2- 172.21.15.42:0/1787611373 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fac5c07bcf0 0x7fac5c129cd0 secure :-1 s=READY pgs=3330 cs=0 l=1 rx=0x7fac5c143b70 tx=0x7fac54008870).stop 2019-12-05T06:50:25.106 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:25.095+0000 7fac616ea700 1 -- 172.21.15.42:0/1787611373 shutdown_connections 2019-12-05T06:50:25.106 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:25.095+0000 7fac616ea700 1 --2- 172.21.15.42:0/1787611373 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fac4003aec0 0x7fac4003d3d0 unknown :-1 s=CLOSED pgs=1396 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:25.106 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:25.095+0000 7fac616ea700 1 --2- 172.21.15.42:0/1787611373 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fac5c07bcf0 0x7fac5c129cd0 unknown :-1 s=CLOSED pgs=3330 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:25.107 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:25.095+0000 7fac616ea700 1 --2- 172.21.15.42:0/1787611373 >> v2:172.21.15.40:3300/0 conn(0x7fac5c1434a0 0x7fac5c1191f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:25.107 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:25.095+0000 7fac616ea700 1 --2- 172.21.15.42:0/1787611373 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fac5c1482e0 0x7fac5c07b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:25.107 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:25.095+0000 7fac616ea700 1 -- 172.21.15.42:0/1787611373 shutdown_connections 2019-12-05T06:50:25.107 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:25.095+0000 7fac616ea700 1 -- 172.21.15.42:0/1787611373 wait complete. 2019-12-05T06:50:25.108 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:25.095+0000 7fac616ea700 1 -- 172.21.15.42:0/1787611373 >> 172.21.15.42:0/1787611373 conn(0x7fac5c0aabe0 msgr2=0x7fac5c11c470 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:50:25.117 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2019-12-05T06:50:27.119 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:50:27.413 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:27.403+0000 7f1fb7b47700 1 Processor -- start 2019-12-05T06:50:27.413 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:27.403+0000 7f1fb7b47700 1 -- start start 2019-12-05T06:50:27.413 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:27.403+0000 7f1fb7b47700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f1fb00a7740 0x7f1fb008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:50:27.413 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:27.403+0000 7f1fb7b47700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f1fb007e720 0x7f1fb007eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:50:27.413 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:27.403+0000 7f1fb7b47700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fb00a0710 con 0x7f1fb009f2e0 2019-12-05T06:50:27.414 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:27.403+0000 7f1fb7b47700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1fb00a0b00 con 0x7f1fb007e720 2019-12-05T06:50:27.414 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:27.403+0000 7f1fb7b47700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1fb008e5b0 con 0x7f1fb00a7740 2019-12-05T06:50:27.414 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:27.403+0000 7f1fb60e4700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f1fb00a7740 0x7f1fb008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:27.414 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:27.403+0000 7f1fb68e5700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f1fb009f2e0 0x7f1fb009f710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:38524/0 (socket says 172.21.15.42:38524) 2019-12-05T06:50:27.414 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:27.403+0000 7f1fb68e5700 1 -- 172.21.15.42:0/2984955510 learned_addr learned my addr 172.21.15.42:0/2984955510 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:50:27.415 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:27.403+0000 7f1fb58e3700 1 --2- 172.21.15.42:0/2984955510 >> v2:172.21.15.40:3300/0 conn(0x7f1fb007e720 0x7f1fb007eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:27.415 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:27.403+0000 7f1fb60e4700 1 -- 172.21.15.42:0/2984955510 >> v2:172.21.15.40:3300/0 conn(0x7f1fb007e720 msgr2=0x7f1fb007eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:50:27.415 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:27.403+0000 7f1fb60e4700 1 --2- 172.21.15.42:0/2984955510 >> v2:172.21.15.40:3300/0 conn(0x7f1fb007e720 0x7f1fb007eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:50:27.415 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:27.403+0000 7f1fb60e4700 1 -- 172.21.15.42:0/2984955510 >> v1:172.21.15.42:6789/0 conn(0x7f1fb009f2e0 legacy=0x7f1fb009f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:27.415 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:27.403+0000 7f1fb60e4700 1 -- 172.21.15.42:0/2984955510 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fb007f0a0 con 0x7f1fb00a7740 2019-12-05T06:50:27.416 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:27.403+0000 7f1fb50e2700 1 -- 172.21.15.42:0/2984955510 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1174270805 0 0) 0x7f1fb00a0710 con 0x7f1fb009f2e0 2019-12-05T06:50:27.416 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:27.403+0000 7f1fb60e4700 1 --2- 172.21.15.42:0/2984955510 >> v2:172.21.15.42:3300/0 conn(0x7f1fb00a7740 0x7f1fb008df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f1fac002f30 tx=0x7f1fac00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:27.416 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:27.403+0000 7f1fb60e4700 1 --2- 172.21.15.42:0/2984955510 >> v2:172.21.15.42:3300/0 conn(0x7f1fb00a7740 0x7f1fb008df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f1fac002f30 tx=0x7f1fac00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:27.416 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:27.403+0000 7f1fb60e4700 1 --2- 172.21.15.42:0/2984955510 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1fb00a7740 0x7f1fb008df80 secure :-1 s=READY pgs=3333 cs=0 l=1 rx=0x7f1fac002f30 tx=0x7f1fac00fc70).ready entity=mon.1 client_cookie=200079237e20913d server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:50:27.416 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:27.403+0000 7f1fb60e4700 1 --2- 172.21.15.42:0/2984955510 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1fb00a7740 0x7f1fb008df80 secure :-1 s=THROTTLE_DONE pgs=3333 cs=0 l=1 rx=0x7f1fac002f30 tx=0x7f1fac00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:27.416 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:27.403+0000 7f1fb50e2700 1 -- 172.21.15.42:0/2984955510 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1fac0112b0 con 0x7f1fb00a7740 2019-12-05T06:50:27.417 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:27.403+0000 7f1fb60e4700 1 --2- 172.21.15.42:0/2984955510 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1fb00a7740 0x7f1fb008df80 secure :-1 s=THROTTLE_DONE pgs=3333 cs=0 l=1 rx=0x7f1fac002f30 tx=0x7f1fac00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:27.417 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:27.403+0000 7f1fb50e2700 1 -- 172.21.15.42:0/2984955510 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fac008ad0 con 0x7f1fb00a7740 2019-12-05T06:50:27.417 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:27.403+0000 7f1fb60e4700 1 --2- 172.21.15.42:0/2984955510 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1fb00a7740 0x7f1fb008df80 secure :-1 s=THROTTLE_DONE pgs=3333 cs=0 l=1 rx=0x7f1fac002f30 tx=0x7f1fac00fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:27.417 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:27.403+0000 7f1fb50e2700 1 -- 172.21.15.42:0/2984955510 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1fac011890 con 0x7f1fb00a7740 2019-12-05T06:50:27.417 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:27.403+0000 7f1fb7b47700 1 -- 172.21.15.42:0/2984955510 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1fb00a7740 msgr2=0x7f1fb008df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:27.417 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:27.403+0000 7f1fb7b47700 1 --2- 172.21.15.42:0/2984955510 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1fb00a7740 0x7f1fb008df80 secure :-1 s=READY pgs=3333 cs=0 l=1 rx=0x7f1fac002f30 tx=0x7f1fac00fc70).stop 2019-12-05T06:50:27.417 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:27.403+0000 7f1fb7b47700 1 -- 172.21.15.42:0/2984955510 shutdown_connections 2019-12-05T06:50:27.418 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:27.403+0000 7f1fb7b47700 1 --2- 172.21.15.42:0/2984955510 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1fb00a7740 0x7f1fb008df80 unknown :-1 s=CLOSED pgs=3333 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:27.418 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:27.403+0000 7f1fb7b47700 1 --2- 172.21.15.42:0/2984955510 >> v2:172.21.15.40:3300/0 conn(0x7f1fb007e720 0x7f1fb007eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:50:27.418 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:27.403+0000 7f1fb7b47700 1 -- 172.21.15.42:0/2984955510 shutdown_connections 2019-12-05T06:50:27.418 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:27.403+0000 7f1fb7b47700 1 -- 172.21.15.42:0/2984955510 wait complete. 2019-12-05T06:50:27.418 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:27.403+0000 7f1fb7b47700 1 -- 172.21.15.42:0/2984955510 >> 172.21.15.42:0/2984955510 conn(0x7f1fb00aabe0 msgr2=0x7f1fb00a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:50:27.419 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:27.407+0000 7f1fb7b47700 1 Processor -- start 2019-12-05T06:50:27.419 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:27.407+0000 7f1fb7b47700 1 -- start start 2019-12-05T06:50:27.422 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:27.407+0000 7f1fb7b47700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f1fb007cde0 0x7f1fb00a4b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:50:27.422 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:27.407+0000 7f1fb7b47700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fb0137750 con 0x7f1fb00a38b0 2019-12-05T06:50:27.422 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:27.407+0000 7f1fb7b47700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fb00a3ce0 con 0x7f1fb0136a30 2019-12-05T06:50:27.423 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:27.407+0000 7f1fb7b47700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1fb00a5c90 con 0x7f1fb007cde0 2019-12-05T06:50:27.423 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:27.407+0000 7f1fb60e4700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f1fb007cde0 0x7f1fb00a4b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:27.423 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:27.407+0000 7f1fb58e3700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f1fb0136a30 0x7f1fb00a5490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:38528/0 (socket says 172.21.15.42:38528) 2019-12-05T06:50:27.423 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:27.407+0000 7f1fb58e3700 1 -- 172.21.15.42:0/2920962342 learned_addr learned my addr 172.21.15.42:0/2920962342 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:50:27.423 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:27.407+0000 7f1fb60e4700 1 -- 172.21.15.42:0/2920962342 >> v1:172.21.15.42:6789/0 conn(0x7f1fb0136a30 legacy=0x7f1fb00a5490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:27.423 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:27.407+0000 7f1fb60e4700 1 -- 172.21.15.42:0/2920962342 >> v1:172.21.15.40:6789/0 conn(0x7f1fb00a38b0 legacy=0x7f1fb012aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:27.424 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:27.407+0000 7f1fb60e4700 1 -- 172.21.15.42:0/2920962342 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fb00a3f10 con 0x7f1fb007cde0 2019-12-05T06:50:27.424 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:27.407+0000 7f1fa6ffd700 1 -- 172.21.15.42:0/2920962342 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 953942989 0 0) 0x7f1fb00a3ce0 con 0x7f1fb0136a30 2019-12-05T06:50:27.424 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:27.407+0000 7f1fb60e4700 1 --2- 172.21.15.42:0/2920962342 >> v2:172.21.15.42:3300/0 conn(0x7f1fb007cde0 0x7f1fb00a4b30 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f1fb00a6890 tx=0x7f1fac011e20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:27.424 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:27.407+0000 7f1fb60e4700 1 --2- 172.21.15.42:0/2920962342 >> v2:172.21.15.42:3300/0 conn(0x7f1fb007cde0 0x7f1fb00a4b30 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f1fb00a6890 tx=0x7f1fac011e20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:27.424 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:27.407+0000 7f1fb60e4700 1 --2- 172.21.15.42:0/2920962342 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1fb007cde0 0x7f1fb00a4b30 secure :-1 s=READY pgs=3335 cs=0 l=1 rx=0x7f1fb00a6890 tx=0x7f1fac011e20).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:50:27.424 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:27.407+0000 7f1fb60e4700 1 --2- 172.21.15.42:0/2920962342 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1fb007cde0 0x7f1fb00a4b30 secure :-1 s=THROTTLE_DONE pgs=3335 cs=0 l=1 rx=0x7f1fb00a6890 tx=0x7f1fac011e20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:27.425 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:27.407+0000 7f1fa6ffd700 1 -- 172.21.15.42:0/2920962342 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1fac007150 con 0x7f1fb007cde0 2019-12-05T06:50:27.425 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:27.407+0000 7f1fb60e4700 1 --2- 172.21.15.42:0/2920962342 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1fb007cde0 0x7f1fb00a4b30 secure :-1 s=THROTTLE_DONE pgs=3335 cs=0 l=1 rx=0x7f1fb00a6890 tx=0x7f1fac011e20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:27.425 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:27.407+0000 7f1fb60e4700 1 --2- 172.21.15.42:0/2920962342 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1fb007cde0 0x7f1fb00a4b30 secure :-1 s=THROTTLE_DONE pgs=3335 cs=0 l=1 rx=0x7f1fb00a6890 tx=0x7f1fac011e20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:27.425 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:27.407+0000 7f1fa6ffd700 1 -- 172.21.15.42:0/2920962342 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fac0114b0 con 0x7f1fb007cde0 2019-12-05T06:50:27.425 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:27.407+0000 7f1fb7b47700 1 -- 172.21.15.42:0/2920962342 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fb012c210 con 0x7f1fb007cde0 2019-12-05T06:50:27.425 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:27.407+0000 7f1fa6ffd700 1 -- 172.21.15.42:0/2920962342 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1fac00f550 con 0x7f1fb007cde0 2019-12-05T06:50:27.426 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:27.407+0000 7f1fb7b47700 1 -- 172.21.15.42:0/2920962342 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fb012c760 con 0x7f1fb007cde0 2019-12-05T06:50:27.426 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:27.407+0000 7f1fb60e4700 1 --2- 172.21.15.42:0/2920962342 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1fb007cde0 0x7f1fb00a4b30 secure :-1 s=THROTTLE_DONE pgs=3335 cs=0 l=1 rx=0x7f1fb00a6890 tx=0x7f1fac011e20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:27.426 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:27.411+0000 7f1fa6ffd700 1 -- 172.21.15.42:0/2920962342 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f1fac005540 con 0x7f1fb007cde0 2019-12-05T06:50:27.426 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:27.411+0000 7f1fb60e4700 1 --2- 172.21.15.42:0/2920962342 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1fb007cde0 0x7f1fb00a4b30 secure :-1 s=THROTTLE_DONE pgs=3335 cs=0 l=1 rx=0x7f1fb00a6890 tx=0x7f1fac011e20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:27.426 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:27.411+0000 7f1fa6ffd700 1 --2- 172.21.15.42:0/2920962342 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f1f9803ac40 0x7f1f9803d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:50:27.426 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:27.411+0000 7f1fa6ffd700 1 -- 172.21.15.42:0/2920962342 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 12309+0+0 (secure 0 0 0) 0x7f1fac08f8c0 con 0x7f1fb007cde0 2019-12-05T06:50:27.427 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:27.411+0000 7f1fb58e3700 1 --2- 172.21.15.42:0/2920962342 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f1f9803ac40 0x7f1f9803d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:27.427 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:27.411+0000 7f1fb58e3700 1 --2- 172.21.15.42:0/2920962342 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f1f9803ac40 0x7f1f9803d150 crc :-1 s=READY pgs=1397 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:50:27.427 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:27.419+0000 7f1fb7b47700 1 -- 172.21.15.42:0/2920962342 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fb00a5080 con 0x7f1fb007cde0 2019-12-05T06:50:27.437 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:27.423+0000 7f1fb60e4700 1 --2- 172.21.15.42:0/2920962342 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1fb007cde0 0x7f1fb00a4b30 secure :-1 s=THROTTLE_DONE pgs=3335 cs=0 l=1 rx=0x7f1fb00a6890 tx=0x7f1fac011e20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:27.438 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:27.423+0000 7f1fa6ffd700 1 -- 172.21.15.42:0/2920962342 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f1fac0745f0 con 0x7f1fb007cde0 2019-12-05T06:50:27.605 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:27.599+0000 7f1fb7b47700 1 -- 172.21.15.42:0/2920962342 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1fb005d110 con 0x7f1fb007cde0 2019-12-05T06:50:27.606 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:27.599+0000 7f1fb60e4700 1 --2- 172.21.15.42:0/2920962342 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1fb007cde0 0x7f1fb00a4b30 secure :-1 s=THROTTLE_DONE pgs=3335 cs=0 l=1 rx=0x7f1fb00a6890 tx=0x7f1fac011e20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:27.606 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:27.599+0000 7f1fa6ffd700 1 -- 172.21.15.42:0/2920962342 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 403 v403) v1 ==== 95+0+1776 (secure 0 0 0) 0x7f1fac077750 con 0x7f1fb007cde0 2019-12-05T06:50:27.606 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 403 2019-12-05T06:50:27.607 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:50:27.607 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":403,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":403,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:50:25.893719+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":122,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8537},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8537":{"gid":8537,"name":"a","rank":0,"incarnation":400,"state":"up:active","state_seq":20,"addr":"172.21.15.42:6835/1881703925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1881703925},{"type":"v1","addr":"172.21.15.42:6835","nonce":1881703925}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:50:27.609 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:27.603+0000 7f1fb7b47700 1 -- 172.21.15.42:0/2920962342 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f1f9803ac40 msgr2=0x7f1f9803d150 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:27.609 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:27.603+0000 7f1fb7b47700 1 --2- 172.21.15.42:0/2920962342 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f1f9803ac40 0x7f1f9803d150 crc :-1 s=READY pgs=1397 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:27.609 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:27.603+0000 7f1fb7b47700 1 -- 172.21.15.42:0/2920962342 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1fb007cde0 msgr2=0x7f1fb00a4b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:27.610 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:27.603+0000 7f1fb7b47700 1 --2- 172.21.15.42:0/2920962342 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1fb007cde0 0x7f1fb00a4b30 secure :-1 s=READY pgs=3335 cs=0 l=1 rx=0x7f1fb00a6890 tx=0x7f1fac011e20).stop 2019-12-05T06:50:27.610 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:27.603+0000 7f1fb7b47700 1 -- 172.21.15.42:0/2920962342 shutdown_connections 2019-12-05T06:50:27.610 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:27.603+0000 7f1fb7b47700 1 --2- 172.21.15.42:0/2920962342 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f1f9803ac40 0x7f1f9803d150 unknown :-1 s=CLOSED pgs=1397 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:27.610 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:27.603+0000 7f1fb7b47700 1 --2- 172.21.15.42:0/2920962342 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1fb007cde0 0x7f1fb00a4b30 unknown :-1 s=CLOSED pgs=3335 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:27.610 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:27.603+0000 7f1fb7b47700 1 -- 172.21.15.42:0/2920962342 shutdown_connections 2019-12-05T06:50:27.611 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:27.603+0000 7f1fb7b47700 1 -- 172.21.15.42:0/2920962342 wait complete. 2019-12-05T06:50:27.611 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:27.603+0000 7f1fb7b47700 1 -- 172.21.15.42:0/2920962342 >> 172.21.15.42:0/2920962342 conn(0x7f1fb007dff0 msgr2=0x7f1fb009f510 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:50:27.619 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a has gained rank=0, replacing gid=12687 2019-12-05T06:50:27.619 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 2 secs before reviving mds.b 2019-12-05T06:50:28.708 INFO:tasks.workunit.client.0.smithi042.stderr:+ cd ../.. 2019-12-05T06:50:28.709 INFO:tasks.workunit.client.0.smithi042.stderr:+ ver=v0.17 2019-12-05T06:50:28.710 INFO:tasks.workunit.client.0.smithi042.stderr:+ echo checking v0.17 2019-12-05T06:50:28.710 INFO:tasks.workunit.client.0.smithi042.stderr:+ cd .snap/v0.17 2019-12-05T06:50:28.710 INFO:tasks.workunit.client.0.smithi042.stdout:checking v0.17 2019-12-05T06:50:28.713 INFO:tasks.workunit.client.0.smithi042.stderr:+ git diff --exit-code 2019-12-05T06:50:29.620 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.b 2019-12-05T06:50:29.620 INFO:tasks.ceph.mds.b:Restarting daemon 2019-12-05T06:50:29.620 INFO:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2019-12-05T06:50:29.626 INFO:tasks.ceph.mds.b:Started 2019-12-05T06:50:29.626 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:50:29.688 INFO:tasks.ceph.mds.b.smithi040.stdout:starting mds.b at 2019-12-05T06:50:29.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:29.707+0000 7f8c5eed7700 1 Processor -- start 2019-12-05T06:50:29.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:29.707+0000 7f8c5eed7700 1 -- start start 2019-12-05T06:50:29.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:29.711+0000 7f8c5eed7700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8c580a7740 0x7f8c5808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:50:29.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:29.711+0000 7f8c5eed7700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f8c5807e720 0x7f8c5807eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:50:29.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:29.711+0000 7f8c5eed7700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c580a0710 con 0x7f8c5809f2e0 2019-12-05T06:50:29.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:29.711+0000 7f8c5eed7700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c580a0b00 con 0x7f8c5807e720 2019-12-05T06:50:29.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:29.711+0000 7f8c5eed7700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c5808e5b0 con 0x7f8c580a7740 2019-12-05T06:50:29.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:29.711+0000 7f8c5cc73700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f8c5807e720 0x7f8c5807eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:29.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:29.711+0000 7f8c5cc73700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f8c5807e720 0x7f8c5807eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:38526/0 (socket says 172.21.15.42:38526) 2019-12-05T06:50:29.722 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:29.711+0000 7f8c5cc73700 1 -- 172.21.15.42:0/1022108861 learned_addr learned my addr 172.21.15.42:0/1022108861 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:50:29.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:29.711+0000 7f8c5d474700 1 --2- 172.21.15.42:0/1022108861 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8c580a7740 0x7f8c5808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:29.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:29.711+0000 7f8c5cc73700 1 -- 172.21.15.42:0/1022108861 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8c580a7740 msgr2=0x7f8c5808df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:50:29.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:29.711+0000 7f8c5cc73700 1 --2- 172.21.15.42:0/1022108861 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8c580a7740 0x7f8c5808df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:50:29.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:29.711+0000 7f8c5cc73700 1 -- 172.21.15.42:0/1022108861 >> v1:172.21.15.40:6789/0 conn(0x7f8c5809f2e0 legacy=0x7f8c5809f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:29.723 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:29.711+0000 7f8c5cc73700 1 -- 172.21.15.42:0/1022108861 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c5807f0a0 con 0x7f8c5807e720 2019-12-05T06:50:29.724 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:29.711+0000 7f8c5cc73700 1 --2- 172.21.15.42:0/1022108861 >> v2:172.21.15.42:3300/0 conn(0x7f8c5807e720 0x7f8c5807eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f8c54008790 tx=0x7f8c5400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:29.724 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:29.711+0000 7f8c5cc73700 1 --2- 172.21.15.42:0/1022108861 >> v2:172.21.15.42:3300/0 conn(0x7f8c5807e720 0x7f8c5807eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f8c54008790 tx=0x7f8c5400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:29.724 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:29.711+0000 7f8c5cc73700 1 --2- 172.21.15.42:0/1022108861 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8c5807e720 0x7f8c5807eb50 secure :-1 s=READY pgs=3337 cs=0 l=1 rx=0x7f8c54008790 tx=0x7f8c5400fd30).ready entity=mon.1 client_cookie=37ec241d80be45b0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:50:29.724 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:29.711+0000 7f8c5cc73700 1 --2- 172.21.15.42:0/1022108861 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8c5807e720 0x7f8c5807eb50 secure :-1 s=THROTTLE_DONE pgs=3337 cs=0 l=1 rx=0x7f8c54008790 tx=0x7f8c5400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:29.724 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:29.711+0000 7f8c4ffff700 1 -- 172.21.15.42:0/1022108861 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8c540113b0 con 0x7f8c5807e720 2019-12-05T06:50:29.725 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:29.711+0000 7f8c5cc73700 1 --2- 172.21.15.42:0/1022108861 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8c5807e720 0x7f8c5807eb50 secure :-1 s=THROTTLE_DONE pgs=3337 cs=0 l=1 rx=0x7f8c54008790 tx=0x7f8c5400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:29.725 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:29.711+0000 7f8c4ffff700 1 -- 172.21.15.42:0/1022108861 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c54008bb0 con 0x7f8c5807e720 2019-12-05T06:50:29.725 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:29.711+0000 7f8c5cc73700 1 --2- 172.21.15.42:0/1022108861 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8c5807e720 0x7f8c5807eb50 secure :-1 s=THROTTLE_DONE pgs=3337 cs=0 l=1 rx=0x7f8c54008790 tx=0x7f8c5400fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:29.725 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:29.711+0000 7f8c4ffff700 1 -- 172.21.15.42:0/1022108861 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8c54011990 con 0x7f8c5807e720 2019-12-05T06:50:29.725 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:29.711+0000 7f8c5eed7700 1 -- 172.21.15.42:0/1022108861 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8c5807e720 msgr2=0x7f8c5807eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:29.726 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:29.711+0000 7f8c5eed7700 1 --2- 172.21.15.42:0/1022108861 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8c5807e720 0x7f8c5807eb50 secure :-1 s=READY pgs=3337 cs=0 l=1 rx=0x7f8c54008790 tx=0x7f8c5400fd30).stop 2019-12-05T06:50:29.726 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:29.711+0000 7f8c5eed7700 1 -- 172.21.15.42:0/1022108861 shutdown_connections 2019-12-05T06:50:29.726 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:29.711+0000 7f8c5eed7700 1 --2- 172.21.15.42:0/1022108861 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8c580a7740 0x7f8c5808df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:50:29.726 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:29.711+0000 7f8c5eed7700 1 --2- 172.21.15.42:0/1022108861 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8c5807e720 0x7f8c5807eb50 unknown :-1 s=CLOSED pgs=3337 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:29.726 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:29.711+0000 7f8c5eed7700 1 -- 172.21.15.42:0/1022108861 shutdown_connections 2019-12-05T06:50:29.727 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:29.711+0000 7f8c5eed7700 1 -- 172.21.15.42:0/1022108861 wait complete. 2019-12-05T06:50:29.727 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:29.711+0000 7f8c5eed7700 1 -- 172.21.15.42:0/1022108861 >> 172.21.15.42:0/1022108861 conn(0x7f8c580aabe0 msgr2=0x7f8c580a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:50:29.732 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:29.715+0000 7f8c5eed7700 1 Processor -- start 2019-12-05T06:50:29.732 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:29.715+0000 7f8c5eed7700 1 -- start start 2019-12-05T06:50:29.732 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:29.715+0000 7f8c5eed7700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f8c5811b210 0x7f8c58112210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:50:29.733 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:29.715+0000 7f8c5eed7700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8c58113400 0x7f8c581275e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:50:29.733 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:29.715+0000 7f8c5eed7700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c580726b0 con 0x7f8c58127b30 2019-12-05T06:50:29.734 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:29.715+0000 7f8c5eed7700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c58118380 con 0x7f8c58113400 2019-12-05T06:50:29.734 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:29.715+0000 7f8c5eed7700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c58073500 con 0x7f8c5811b210 2019-12-05T06:50:29.734 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:29.715+0000 7f8c5d474700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f8c5811b210 0x7f8c58112210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:29.734 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:29.715+0000 7f8c5d474700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f8c5811b210 0x7f8c58112210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:38530/0 (socket says 172.21.15.42:38530) 2019-12-05T06:50:29.734 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:29.715+0000 7f8c5d474700 1 -- 172.21.15.42:0/3127330673 learned_addr learned my addr 172.21.15.42:0/3127330673 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:50:29.735 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:29.715+0000 7f8c5cc73700 1 --2- 172.21.15.42:0/3127330673 >> v2:172.21.15.40:3300/0 conn(0x7f8c58113400 0x7f8c581275e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:29.736 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:29.715+0000 7f8c5d474700 1 -- 172.21.15.42:0/3127330673 >> v2:172.21.15.40:3300/0 conn(0x7f8c58113400 msgr2=0x7f8c581275e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:29.736 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:29.715+0000 7f8c5d474700 1 --2- 172.21.15.42:0/3127330673 >> v2:172.21.15.40:3300/0 conn(0x7f8c58113400 0x7f8c581275e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:29.736 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:29.715+0000 7f8c5d474700 1 -- 172.21.15.42:0/3127330673 >> v1:172.21.15.40:6789/0 conn(0x7f8c58127b30 legacy=0x7f8c5808eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:29.736 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:29.715+0000 7f8c5d474700 1 -- 172.21.15.42:0/3127330673 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c58073a40 con 0x7f8c5811b210 2019-12-05T06:50:29.736 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:29.715+0000 7f8c5d474700 1 --2- 172.21.15.42:0/3127330673 >> v2:172.21.15.42:3300/0 conn(0x7f8c5811b210 0x7f8c58112210 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f8c48003070 tx=0x7f8c48010ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:29.736 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:29.715+0000 7f8c5d474700 1 --2- 172.21.15.42:0/3127330673 >> v2:172.21.15.42:3300/0 conn(0x7f8c5811b210 0x7f8c58112210 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f8c48003070 tx=0x7f8c48010ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:29.737 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:29.715+0000 7f8c5d474700 1 --2- 172.21.15.42:0/3127330673 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8c5811b210 0x7f8c58112210 secure :-1 s=READY pgs=3338 cs=0 l=1 rx=0x7f8c48003070 tx=0x7f8c48010ae0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:50:29.737 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:29.715+0000 7f8c5d474700 1 --2- 172.21.15.42:0/3127330673 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8c5811b210 0x7f8c58112210 secure :-1 s=THROTTLE_DONE pgs=3338 cs=0 l=1 rx=0x7f8c48003070 tx=0x7f8c48010ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:29.737 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:29.715+0000 7f8c5d474700 1 --2- 172.21.15.42:0/3127330673 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8c5811b210 0x7f8c58112210 secure :-1 s=THROTTLE_DONE pgs=3338 cs=0 l=1 rx=0x7f8c48003070 tx=0x7f8c48010ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:29.737 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:29.715+0000 7f8c5d474700 1 --2- 172.21.15.42:0/3127330673 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8c5811b210 0x7f8c58112210 secure :-1 s=THROTTLE_DONE pgs=3338 cs=0 l=1 rx=0x7f8c48003070 tx=0x7f8c48010ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:29.738 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:29.715+0000 7f8c4dffb700 1 -- 172.21.15.42:0/3127330673 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8c48005920 con 0x7f8c5811b210 2019-12-05T06:50:29.738 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:29.715+0000 7f8c4dffb700 1 -- 172.21.15.42:0/3127330673 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c48005540 con 0x7f8c5811b210 2019-12-05T06:50:29.738 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:29.715+0000 7f8c5eed7700 1 -- 172.21.15.42:0/3127330673 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c58117f50 con 0x7f8c5811b210 2019-12-05T06:50:29.738 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:29.715+0000 7f8c4dffb700 1 -- 172.21.15.42:0/3127330673 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8c4800e460 con 0x7f8c5811b210 2019-12-05T06:50:29.738 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:29.715+0000 7f8c5eed7700 1 -- 172.21.15.42:0/3127330673 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c58113830 con 0x7f8c5811b210 2019-12-05T06:50:29.738 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:29.715+0000 7f8c5d474700 1 --2- 172.21.15.42:0/3127330673 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8c5811b210 0x7f8c58112210 secure :-1 s=THROTTLE_DONE pgs=3338 cs=0 l=1 rx=0x7f8c48003070 tx=0x7f8c48010ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:29.739 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:29.715+0000 7f8c4dffb700 1 -- 172.21.15.42:0/3127330673 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f8c48007180 con 0x7f8c5811b210 2019-12-05T06:50:29.739 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:29.715+0000 7f8c5d474700 1 --2- 172.21.15.42:0/3127330673 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8c5811b210 0x7f8c58112210 secure :-1 s=THROTTLE_DONE pgs=3338 cs=0 l=1 rx=0x7f8c48003070 tx=0x7f8c48010ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:29.739 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:29.719+0000 7f8c4dffb700 1 --2- 172.21.15.42:0/3127330673 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8c3803ae00 0x7f8c3803d310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:50:29.739 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:29.719+0000 7f8c4dffb700 1 -- 172.21.15.42:0/3127330673 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 12309+0+0 (secure 0 0 0) 0x7f8c4804ae20 con 0x7f8c5811b210 2019-12-05T06:50:29.740 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:29.719+0000 7f8c5cc73700 1 --2- 172.21.15.42:0/3127330673 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8c3803ae00 0x7f8c3803d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:29.740 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:29.719+0000 7f8c5cc73700 1 --2- 172.21.15.42:0/3127330673 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8c3803ae00 0x7f8c3803d310 crc :-1 s=READY pgs=1398 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:50:29.740 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:29.723+0000 7f8c5eed7700 1 -- 172.21.15.42:0/3127330673 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c5805d110 con 0x7f8c5811b210 2019-12-05T06:50:29.741 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:29.727+0000 7f8c5d474700 1 --2- 172.21.15.42:0/3127330673 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8c5811b210 0x7f8c58112210 secure :-1 s=THROTTLE_DONE pgs=3338 cs=0 l=1 rx=0x7f8c48003070 tx=0x7f8c48010ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:29.741 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:29.731+0000 7f8c4dffb700 1 -- 172.21.15.42:0/3127330673 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f8c48031ea0 con 0x7f8c5811b210 2019-12-05T06:50:29.955 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:29.947+0000 7f8c5eed7700 1 -- 172.21.15.42:0/3127330673 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8c5805b8c0 con 0x7f8c5811b210 2019-12-05T06:50:29.955 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:29.947+0000 7f8c5d474700 1 --2- 172.21.15.42:0/3127330673 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8c5811b210 0x7f8c58112210 secure :-1 s=THROTTLE_DONE pgs=3338 cs=0 l=1 rx=0x7f8c48003070 tx=0x7f8c48010ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:29.955 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:29.947+0000 7f8c4dffb700 1 -- 172.21.15.42:0/3127330673 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 403 v403) v1 ==== 95+0+1776 (secure 0 0 0) 0x7f8c48032100 con 0x7f8c5811b210 2019-12-05T06:50:29.955 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 403 2019-12-05T06:50:29.956 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:50:29.956 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":403,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":403,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:50:25.893719+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":122,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8537},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8537":{"gid":8537,"name":"a","rank":0,"incarnation":400,"state":"up:active","state_seq":20,"addr":"172.21.15.42:6835/1881703925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1881703925},{"type":"v1","addr":"172.21.15.42:6835","nonce":1881703925}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:50:29.958 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:29.951+0000 7f8c5eed7700 1 -- 172.21.15.42:0/3127330673 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8c3803ae00 msgr2=0x7f8c3803d310 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:29.958 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:29.951+0000 7f8c5eed7700 1 --2- 172.21.15.42:0/3127330673 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8c3803ae00 0x7f8c3803d310 crc :-1 s=READY pgs=1398 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:29.958 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:29.951+0000 7f8c5eed7700 1 -- 172.21.15.42:0/3127330673 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8c5811b210 msgr2=0x7f8c58112210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:29.958 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:29.951+0000 7f8c5eed7700 1 --2- 172.21.15.42:0/3127330673 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8c5811b210 0x7f8c58112210 secure :-1 s=READY pgs=3338 cs=0 l=1 rx=0x7f8c48003070 tx=0x7f8c48010ae0).stop 2019-12-05T06:50:29.959 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:29.951+0000 7f8c5eed7700 1 -- 172.21.15.42:0/3127330673 shutdown_connections 2019-12-05T06:50:29.959 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:29.951+0000 7f8c5eed7700 1 --2- 172.21.15.42:0/3127330673 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8c3803ae00 0x7f8c3803d310 unknown :-1 s=CLOSED pgs=1398 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:29.959 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:29.951+0000 7f8c5eed7700 1 --2- 172.21.15.42:0/3127330673 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8c5811b210 0x7f8c58112210 unknown :-1 s=CLOSED pgs=3338 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:29.959 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:29.951+0000 7f8c5eed7700 1 --2- 172.21.15.42:0/3127330673 >> v2:172.21.15.40:3300/0 conn(0x7f8c58113400 0x7f8c581275e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:29.959 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:29.951+0000 7f8c5eed7700 1 -- 172.21.15.42:0/3127330673 shutdown_connections 2019-12-05T06:50:29.959 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:29.951+0000 7f8c5eed7700 1 -- 172.21.15.42:0/3127330673 wait complete. 2019-12-05T06:50:29.959 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:29.951+0000 7f8c5eed7700 1 -- 172.21.15.42:0/3127330673 >> 172.21.15.42:0/3127330673 conn(0x7f8c580a8230 msgr2=0x7f8c580a9870 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:50:29.969 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2019-12-05T06:50:31.970 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:50:32.173 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:32.159+0000 7f69955ad700 1 Processor -- start 2019-12-05T06:50:32.173 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:32.159+0000 7f69955ad700 1 -- start start 2019-12-05T06:50:32.173 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:32.159+0000 7f69955ad700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f69900a7740 0x7f699008df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:50:32.173 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:32.159+0000 7f69955ad700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f699007e720 0x7f699007eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:50:32.173 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:32.159+0000 7f69955ad700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69900a0710 con 0x7f699009f2e0 2019-12-05T06:50:32.173 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:32.159+0000 7f69955ad700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69900a0b00 con 0x7f69900a7740 2019-12-05T06:50:32.174 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:32.159+0000 7f69955ad700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f699008e5b0 con 0x7f699007e720 2019-12-05T06:50:32.174 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:32.159+0000 7f698ffff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f699009f2e0 0x7f699009f710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:38610/0 (socket says 172.21.15.42:38610) 2019-12-05T06:50:32.174 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:32.159+0000 7f698ffff700 1 -- 172.21.15.42:0/2846701752 learned_addr learned my addr 172.21.15.42:0/2846701752 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:50:32.174 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:32.159+0000 7f698f7fe700 1 --2- 172.21.15.42:0/2846701752 >> v2:172.21.15.40:3300/0 conn(0x7f69900a7740 0x7f699008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:32.174 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:32.159+0000 7f698effd700 1 --2- 172.21.15.42:0/2846701752 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f699007e720 0x7f699007eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:32.174 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:32.159+0000 7f698e7fc700 1 -- 172.21.15.42:0/2846701752 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 633770436 0 0) 0x7f69900a0710 con 0x7f699009f2e0 2019-12-05T06:50:32.174 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:32.159+0000 7f698e7fc700 1 -- 172.21.15.42:0/2846701752 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69780030b0 con 0x7f699009f2e0 2019-12-05T06:50:32.175 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:32.159+0000 7f698effd700 1 -- 172.21.15.42:0/2846701752 >> v2:172.21.15.40:3300/0 conn(0x7f69900a7740 msgr2=0x7f699008df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:50:32.175 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:32.159+0000 7f698effd700 1 --2- 172.21.15.42:0/2846701752 >> v2:172.21.15.40:3300/0 conn(0x7f69900a7740 0x7f699008df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:50:32.175 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:32.159+0000 7f698effd700 1 -- 172.21.15.42:0/2846701752 >> v1:172.21.15.42:6789/0 conn(0x7f699009f2e0 legacy=0x7f699009f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:32.175 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:32.159+0000 7f698effd700 1 -- 172.21.15.42:0/2846701752 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f699007f0a0 con 0x7f699007e720 2019-12-05T06:50:32.175 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:32.159+0000 7f698effd700 1 --2- 172.21.15.42:0/2846701752 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f699007e720 0x7f699007eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f6980008790 tx=0x7f698000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:32.175 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:32.159+0000 7f698effd700 1 --2- 172.21.15.42:0/2846701752 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f699007e720 0x7f699007eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f6980008790 tx=0x7f698000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:32.175 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:32.159+0000 7f698effd700 1 --2- 172.21.15.42:0/2846701752 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f699007e720 0x7f699007eb50 secure :-1 s=READY pgs=735 cs=0 l=1 rx=0x7f6980008790 tx=0x7f698000fc70).ready entity=mon.2 client_cookie=cb98689957982a33 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:50:32.176 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:32.163+0000 7f698effd700 1 --2- 172.21.15.42:0/2846701752 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f699007e720 0x7f699007eb50 secure :-1 s=THROTTLE_DONE pgs=735 cs=0 l=1 rx=0x7f6980008790 tx=0x7f698000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:32.176 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:32.163+0000 7f698e7fc700 1 -- 172.21.15.42:0/2846701752 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f69800112f0 con 0x7f699007e720 2019-12-05T06:50:32.176 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:32.163+0000 7f698effd700 1 --2- 172.21.15.42:0/2846701752 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f699007e720 0x7f699007eb50 secure :-1 s=THROTTLE_DONE pgs=735 cs=0 l=1 rx=0x7f6980008790 tx=0x7f698000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:32.176 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:32.163+0000 7f698e7fc700 1 -- 172.21.15.42:0/2846701752 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6980008ad0 con 0x7f699007e720 2019-12-05T06:50:32.176 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:32.163+0000 7f698effd700 1 --2- 172.21.15.42:0/2846701752 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f699007e720 0x7f699007eb50 secure :-1 s=THROTTLE_DONE pgs=735 cs=0 l=1 rx=0x7f6980008790 tx=0x7f698000fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:32.176 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:32.163+0000 7f698e7fc700 1 -- 172.21.15.42:0/2846701752 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f69800118d0 con 0x7f699007e720 2019-12-05T06:50:32.176 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:32.163+0000 7f69955ad700 1 -- 172.21.15.42:0/2846701752 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f699007e720 msgr2=0x7f699007eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:32.177 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:32.163+0000 7f69955ad700 1 --2- 172.21.15.42:0/2846701752 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f699007e720 0x7f699007eb50 secure :-1 s=READY pgs=735 cs=0 l=1 rx=0x7f6980008790 tx=0x7f698000fc70).stop 2019-12-05T06:50:32.177 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:32.163+0000 7f69955ad700 1 -- 172.21.15.42:0/2846701752 shutdown_connections 2019-12-05T06:50:32.177 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:32.163+0000 7f69955ad700 1 --2- 172.21.15.42:0/2846701752 >> v2:172.21.15.40:3300/0 conn(0x7f69900a7740 0x7f699008df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:50:32.177 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:32.163+0000 7f69955ad700 1 --2- 172.21.15.42:0/2846701752 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f699007e720 0x7f699007eb50 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:32.177 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:32.163+0000 7f69955ad700 1 -- 172.21.15.42:0/2846701752 shutdown_connections 2019-12-05T06:50:32.177 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:32.163+0000 7f69955ad700 1 -- 172.21.15.42:0/2846701752 wait complete. 2019-12-05T06:50:32.177 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:32.163+0000 7f69955ad700 1 -- 172.21.15.42:0/2846701752 >> 172.21.15.42:0/2846701752 conn(0x7f69900aabe0 msgr2=0x7f69900a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:50:32.178 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:32.163+0000 7f69955ad700 1 Processor -- start 2019-12-05T06:50:32.178 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:32.163+0000 7f69955ad700 1 -- start start 2019-12-05T06:50:32.178 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:32.167+0000 7f69955ad700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f699012a100 0x7f699012c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:50:32.178 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:32.167+0000 7f69955ad700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6990134740 con 0x7f6990134310 2019-12-05T06:50:32.178 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:32.167+0000 7f69955ad700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6990143b60 con 0x7f69900a6de0 2019-12-05T06:50:32.178 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:32.167+0000 7f69955ad700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6990134a90 con 0x7f699012a100 2019-12-05T06:50:32.178 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:32.167+0000 7f698f7fe700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f69900a6de0 0x7f6990144b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:38612/0 (socket says 172.21.15.42:38612) 2019-12-05T06:50:32.178 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:32.167+0000 7f698f7fe700 1 -- 172.21.15.42:0/2531981454 learned_addr learned my addr 172.21.15.42:0/2531981454 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:50:32.179 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:32.167+0000 7f698ffff700 1 --2- 172.21.15.42:0/2531981454 >> v2:172.21.15.40:3300/0 conn(0x7f699012a100 0x7f699012c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:32.179 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:32.167+0000 7f6977fff700 1 -- 172.21.15.42:0/2531981454 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3020680996 0 0) 0x7f6990143b60 con 0x7f69900a6de0 2019-12-05T06:50:32.179 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:32.167+0000 7f6977fff700 1 -- 172.21.15.42:0/2531981454 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69700030b0 con 0x7f69900a6de0 2019-12-05T06:50:32.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:32.167+0000 7f698ffff700 1 -- 172.21.15.42:0/2531981454 >> v1:172.21.15.42:6789/0 conn(0x7f69900a6de0 legacy=0x7f6990144b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:32.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:32.167+0000 7f6977fff700 1 -- 172.21.15.42:0/2531981454 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2235595956 0 0) 0x7f6990134740 con 0x7f6990134310 2019-12-05T06:50:32.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:32.167+0000 7f698ffff700 1 -- 172.21.15.42:0/2531981454 >> v1:172.21.15.40:6789/0 conn(0x7f6990134310 legacy=0x7f6990143360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:32.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:32.167+0000 7f698ffff700 1 -- 172.21.15.42:0/2531981454 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6990145860 con 0x7f699012a100 2019-12-05T06:50:32.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:32.167+0000 7f698ffff700 1 --2- 172.21.15.42:0/2531981454 >> v2:172.21.15.40:3300/0 conn(0x7f699012a100 0x7f699012c540 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f6988004720 tx=0x7f6988010b10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:32.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:32.167+0000 7f698ffff700 1 --2- 172.21.15.42:0/2531981454 >> v2:172.21.15.40:3300/0 conn(0x7f699012a100 0x7f699012c540 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f6988004720 tx=0x7f6988010b10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:32.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:32.167+0000 7f698ffff700 1 --2- 172.21.15.42:0/2531981454 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f699012a100 0x7f699012c540 secure :-1 s=READY pgs=2524 cs=0 l=1 rx=0x7f6988004720 tx=0x7f6988010b10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:50:32.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:32.167+0000 7f698ffff700 1 --2- 172.21.15.42:0/2531981454 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f699012a100 0x7f699012c540 secure :-1 s=THROTTLE_DONE pgs=2524 cs=0 l=1 rx=0x7f6988004720 tx=0x7f6988010b10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:32.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:32.167+0000 7f6977fff700 1 -- 172.21.15.42:0/2531981454 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6988013060 con 0x7f699012a100 2019-12-05T06:50:32.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:32.167+0000 7f698ffff700 1 --2- 172.21.15.42:0/2531981454 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f699012a100 0x7f699012c540 secure :-1 s=THROTTLE_DONE pgs=2524 cs=0 l=1 rx=0x7f6988004720 tx=0x7f6988010b10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:32.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:32.167+0000 7f698ffff700 1 --2- 172.21.15.42:0/2531981454 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f699012a100 0x7f699012c540 secure :-1 s=THROTTLE_DONE pgs=2524 cs=0 l=1 rx=0x7f6988004720 tx=0x7f6988010b10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:32.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:32.167+0000 7f6977fff700 1 -- 172.21.15.42:0/2531981454 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6988005bd0 con 0x7f699012a100 2019-12-05T06:50:32.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:32.167+0000 7f69955ad700 1 -- 172.21.15.42:0/2531981454 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f699012ca90 con 0x7f699012a100 2019-12-05T06:50:32.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:32.167+0000 7f6977fff700 1 -- 172.21.15.42:0/2531981454 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f69880125b0 con 0x7f699012a100 2019-12-05T06:50:32.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:32.167+0000 7f69955ad700 1 -- 172.21.15.42:0/2531981454 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f699012cf60 con 0x7f699012a100 2019-12-05T06:50:32.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:32.167+0000 7f698ffff700 1 --2- 172.21.15.42:0/2531981454 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f699012a100 0x7f699012c540 secure :-1 s=THROTTLE_DONE pgs=2524 cs=0 l=1 rx=0x7f6988004720 tx=0x7f6988010b10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:32.184 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:32.171+0000 7f6977fff700 1 -- 172.21.15.42:0/2531981454 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f69880127b0 con 0x7f699012a100 2019-12-05T06:50:32.184 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:32.171+0000 7f698ffff700 1 --2- 172.21.15.42:0/2531981454 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f699012a100 0x7f699012c540 secure :-1 s=THROTTLE_DONE pgs=2524 cs=0 l=1 rx=0x7f6988004720 tx=0x7f6988010b10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:32.184 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:32.171+0000 7f6977fff700 1 --2- 172.21.15.42:0/2531981454 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f697003b1c0 0x7f697003d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:50:32.184 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:32.171+0000 7f6977fff700 1 -- 172.21.15.42:0/2531981454 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 12309+0+0 (secure 0 0 0) 0x7f698804b0f0 con 0x7f699012a100 2019-12-05T06:50:32.184 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:32.171+0000 7f698f7fe700 1 --2- 172.21.15.42:0/2531981454 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f697003b1c0 0x7f697003d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:32.184 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:32.171+0000 7f698f7fe700 1 --2- 172.21.15.42:0/2531981454 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f697003b1c0 0x7f697003d6d0 crc :-1 s=READY pgs=1399 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:50:32.184 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:32.175+0000 7f69955ad700 1 -- 172.21.15.42:0/2531981454 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f699005d110 con 0x7f699012a100 2019-12-05T06:50:32.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:32.183+0000 7f698ffff700 1 --2- 172.21.15.42:0/2531981454 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f699012a100 0x7f699012c540 secure :-1 s=THROTTLE_DONE pgs=2524 cs=0 l=1 rx=0x7f6988004720 tx=0x7f6988010b10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:32.190 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:32.183+0000 7f6977fff700 1 -- 172.21.15.42:0/2531981454 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f6988032170 con 0x7f699012a100 2019-12-05T06:50:32.360 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:32.355+0000 7f69955ad700 1 -- 172.21.15.42:0/2531981454 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f699005b8c0 con 0x7f699012a100 2019-12-05T06:50:32.361 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:32.355+0000 7f698ffff700 1 --2- 172.21.15.42:0/2531981454 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f699012a100 0x7f699012c540 secure :-1 s=THROTTLE_DONE pgs=2524 cs=0 l=1 rx=0x7f6988004720 tx=0x7f6988010b10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:32.361 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:32.355+0000 7f6977fff700 1 -- 172.21.15.42:0/2531981454 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 403 v403) v1 ==== 95+0+1776 (secure 0 0 0) 0x7f69880323d0 con 0x7f699012a100 2019-12-05T06:50:32.363 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:50:32.363 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":403,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":403,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:50:25.893719+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":122,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8537},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8537":{"gid":8537,"name":"a","rank":0,"incarnation":400,"state":"up:active","state_seq":20,"addr":"172.21.15.42:6835/1881703925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1881703925},{"type":"v1","addr":"172.21.15.42:6835","nonce":1881703925}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:50:32.364 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 403 2019-12-05T06:50:32.378 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:32.359+0000 7f69955ad700 1 -- 172.21.15.42:0/2531981454 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f697003b1c0 msgr2=0x7f697003d6d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:32.378 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:32.359+0000 7f69955ad700 1 --2- 172.21.15.42:0/2531981454 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f697003b1c0 0x7f697003d6d0 crc :-1 s=READY pgs=1399 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:32.378 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:32.359+0000 7f69955ad700 1 -- 172.21.15.42:0/2531981454 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f699012a100 msgr2=0x7f699012c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:32.378 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:32.359+0000 7f69955ad700 1 --2- 172.21.15.42:0/2531981454 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f699012a100 0x7f699012c540 secure :-1 s=READY pgs=2524 cs=0 l=1 rx=0x7f6988004720 tx=0x7f6988010b10).stop 2019-12-05T06:50:32.378 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:32.359+0000 7f69955ad700 1 -- 172.21.15.42:0/2531981454 shutdown_connections 2019-12-05T06:50:32.378 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:32.359+0000 7f69955ad700 1 --2- 172.21.15.42:0/2531981454 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f697003b1c0 0x7f697003d6d0 unknown :-1 s=CLOSED pgs=1399 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:32.379 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:32.359+0000 7f69955ad700 1 --2- 172.21.15.42:0/2531981454 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f699012a100 0x7f699012c540 unknown :-1 s=CLOSED pgs=2524 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:32.379 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:32.359+0000 7f69955ad700 1 -- 172.21.15.42:0/2531981454 shutdown_connections 2019-12-05T06:50:32.379 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:32.359+0000 7f69955ad700 1 -- 172.21.15.42:0/2531981454 wait complete. 2019-12-05T06:50:32.379 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:32.359+0000 7f69955ad700 1 -- 172.21.15.42:0/2531981454 >> 172.21.15.42:0/2531981454 conn(0x7f699011b3d0 msgr2=0x7f699007e4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:50:32.379 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2019-12-05T06:50:34.382 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:50:34.489 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:34.479+0000 7f8ee459a700 1 Processor -- start 2019-12-05T06:50:34.490 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:34.479+0000 7f8ee459a700 1 -- start start 2019-12-05T06:50:34.490 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:34.479+0000 7f8ee459a700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8edc0a7740 0x7f8edc08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:50:34.490 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:34.479+0000 7f8ee459a700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f8edc09fae0 0x7f8edc07eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:50:34.490 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:34.479+0000 7f8ee459a700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8edc0a0710 con 0x7f8edc07e720 2019-12-05T06:50:34.490 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:34.479+0000 7f8ee459a700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8edc0a0b00 con 0x7f8edc0a7740 2019-12-05T06:50:34.491 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:34.479+0000 7f8ee459a700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8edc08e5b0 con 0x7f8edc09fae0 2019-12-05T06:50:34.491 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:34.479+0000 7f8ee3338700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f8edc09fae0 0x7f8edc07eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:34.491 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:34.479+0000 7f8ee3338700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f8edc09fae0 0x7f8edc07eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:38586/0 (socket says 172.21.15.42:38586) 2019-12-05T06:50:34.491 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:34.479+0000 7f8ee3338700 1 -- 172.21.15.42:0/632505525 learned_addr learned my addr 172.21.15.42:0/632505525 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:50:34.491 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:34.479+0000 7f8ee2b37700 1 --2- 172.21.15.42:0/632505525 >> v2:172.21.15.40:3300/0 conn(0x7f8edc0a7740 0x7f8edc08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:34.491 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:34.479+0000 7f8ee3338700 1 -- 172.21.15.42:0/632505525 >> v2:172.21.15.40:3300/0 conn(0x7f8edc0a7740 msgr2=0x7f8edc08df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:50:34.491 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:34.479+0000 7f8ee3338700 1 --2- 172.21.15.42:0/632505525 >> v2:172.21.15.40:3300/0 conn(0x7f8edc0a7740 0x7f8edc08df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:50:34.491 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:34.479+0000 7f8ee3338700 1 -- 172.21.15.42:0/632505525 >> v1:172.21.15.42:6789/0 conn(0x7f8edc07e720 legacy=0x7f8edc09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:34.492 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:34.479+0000 7f8ee3338700 1 -- 172.21.15.42:0/632505525 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8edc07f0a0 con 0x7f8edc09fae0 2019-12-05T06:50:34.492 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:34.479+0000 7f8ee1b35700 1 -- 172.21.15.42:0/632505525 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 966692990 0 0) 0x7f8edc0a0710 con 0x7f8edc07e720 2019-12-05T06:50:34.492 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:34.479+0000 7f8ee3338700 1 --2- 172.21.15.42:0/632505525 >> v2:172.21.15.42:3300/0 conn(0x7f8edc09fae0 0x7f8edc07eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f8ed40032b0 tx=0x7f8ed40100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:34.492 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:34.479+0000 7f8ee3338700 1 --2- 172.21.15.42:0/632505525 >> v2:172.21.15.42:3300/0 conn(0x7f8edc09fae0 0x7f8edc07eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f8ed40032b0 tx=0x7f8ed40100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:34.493 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:34.479+0000 7f8ee3338700 1 --2- 172.21.15.42:0/632505525 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8edc09fae0 0x7f8edc07eb50 secure :-1 s=READY pgs=3342 cs=0 l=1 rx=0x7f8ed40032b0 tx=0x7f8ed40100b0).ready entity=mon.1 client_cookie=6f9b73c4cf2f00ac server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:50:34.493 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:34.479+0000 7f8ee3338700 1 --2- 172.21.15.42:0/632505525 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8edc09fae0 0x7f8edc07eb50 secure :-1 s=THROTTLE_DONE pgs=3342 cs=0 l=1 rx=0x7f8ed40032b0 tx=0x7f8ed40100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:34.493 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:34.479+0000 7f8ee1b35700 1 -- 172.21.15.42:0/632505525 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8ed4010d60 con 0x7f8edc09fae0 2019-12-05T06:50:34.493 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:34.479+0000 7f8ee3338700 1 --2- 172.21.15.42:0/632505525 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8edc09fae0 0x7f8edc07eb50 secure :-1 s=THROTTLE_DONE pgs=3342 cs=0 l=1 rx=0x7f8ed40032b0 tx=0x7f8ed40100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:34.493 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:34.479+0000 7f8ee1b35700 1 -- 172.21.15.42:0/632505525 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ed4010980 con 0x7f8edc09fae0 2019-12-05T06:50:34.493 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:34.479+0000 7f8ee3338700 1 --2- 172.21.15.42:0/632505525 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8edc09fae0 0x7f8edc07eb50 secure :-1 s=THROTTLE_DONE pgs=3342 cs=0 l=1 rx=0x7f8ed40032b0 tx=0x7f8ed40100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:34.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:34.479+0000 7f8ee1b35700 1 -- 172.21.15.42:0/632505525 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8ed40114a0 con 0x7f8edc09fae0 2019-12-05T06:50:34.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:34.479+0000 7f8ee459a700 1 -- 172.21.15.42:0/632505525 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8edc09fae0 msgr2=0x7f8edc07eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:34.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:34.479+0000 7f8ee459a700 1 --2- 172.21.15.42:0/632505525 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8edc09fae0 0x7f8edc07eb50 secure :-1 s=READY pgs=3342 cs=0 l=1 rx=0x7f8ed40032b0 tx=0x7f8ed40100b0).stop 2019-12-05T06:50:34.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:34.483+0000 7f8ee459a700 1 -- 172.21.15.42:0/632505525 shutdown_connections 2019-12-05T06:50:34.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:34.483+0000 7f8ee459a700 1 --2- 172.21.15.42:0/632505525 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8edc09fae0 0x7f8edc07eb50 unknown :-1 s=CLOSED pgs=3342 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:34.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:34.483+0000 7f8ee459a700 1 --2- 172.21.15.42:0/632505525 >> v2:172.21.15.40:3300/0 conn(0x7f8edc0a7740 0x7f8edc08df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:50:34.494 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:34.483+0000 7f8ee459a700 1 -- 172.21.15.42:0/632505525 shutdown_connections 2019-12-05T06:50:34.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:34.483+0000 7f8ee459a700 1 -- 172.21.15.42:0/632505525 wait complete. 2019-12-05T06:50:34.495 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:34.483+0000 7f8ee459a700 1 -- 172.21.15.42:0/632505525 >> 172.21.15.42:0/632505525 conn(0x7f8edc0aabe0 msgr2=0x7f8edc0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:50:34.500 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:34.483+0000 7f8ee459a700 1 Processor -- start 2019-12-05T06:50:34.500 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:34.483+0000 7f8ee459a700 1 -- start start 2019-12-05T06:50:34.501 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:34.483+0000 7f8ee459a700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8edc123760 0x7f8edc11a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:50:34.501 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:34.483+0000 7f8ee459a700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8edc07c9f0 con 0x7f8edc08f350 2019-12-05T06:50:34.501 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:34.483+0000 7f8ee459a700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8edc121630 con 0x7f8edc11abf0 2019-12-05T06:50:34.501 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:34.483+0000 7f8ee459a700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8edc11bf80 con 0x7f8edc123760 2019-12-05T06:50:34.501 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:34.487+0000 7f8ee2336700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f8edc11abf0 0x7f8edc08eb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:38628/0 (socket says 172.21.15.42:38628) 2019-12-05T06:50:34.501 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:34.487+0000 7f8ee2336700 1 -- 172.21.15.42:0/921891407 learned_addr learned my addr 172.21.15.42:0/921891407 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:50:34.501 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:34.487+0000 7f8ee2b37700 1 --2- 172.21.15.42:0/921891407 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8edc123760 0x7f8edc11a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:34.502 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:34.487+0000 7f8ed37fe700 1 -- 172.21.15.42:0/921891407 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3336475178 0 0) 0x7f8edc121630 con 0x7f8edc11abf0 2019-12-05T06:50:34.502 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:34.487+0000 7f8ed37fe700 1 -- 172.21.15.42:0/921891407 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ec40030b0 con 0x7f8edc11abf0 2019-12-05T06:50:34.504 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:34.487+0000 7f8ed37fe700 1 -- 172.21.15.42:0/921891407 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3699147674 0 0) 0x7f8edc07c9f0 con 0x7f8edc08f350 2019-12-05T06:50:34.504 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:34.487+0000 7f8ed37fe700 1 -- 172.21.15.42:0/921891407 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8edc121630 con 0x7f8edc08f350 2019-12-05T06:50:34.504 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:34.487+0000 7f8ed37fe700 1 -- 172.21.15.42:0/921891407 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 2965169642 0 0) 0x7f8ec40030b0 con 0x7f8edc11abf0 2019-12-05T06:50:34.505 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:34.487+0000 7f8ed37fe700 1 -- 172.21.15.42:0/921891407 --> v1:172.21.15.42:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8edc07c9f0 con 0x7f8edc11abf0 2019-12-05T06:50:34.505 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:34.487+0000 7f8ed37fe700 1 -- 172.21.15.42:0/921891407 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7f8ecc001a20 con 0x7f8edc11abf0 2019-12-05T06:50:34.505 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:34.487+0000 7f8ed37fe700 1 -- 172.21.15.42:0/921891407 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8edc123760 msgr2=0x7f8edc11a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:34.505 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:34.487+0000 7f8ed37fe700 1 --2- 172.21.15.42:0/921891407 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8edc123760 0x7f8edc11a6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:34.505 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:34.487+0000 7f8ed37fe700 1 -- 172.21.15.42:0/921891407 >> v1:172.21.15.42:6789/0 conn(0x7f8edc11abf0 legacy=0x7f8edc08eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:34.505 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:34.487+0000 7f8ed37fe700 1 -- 172.21.15.42:0/921891407 >> v1:172.21.15.40:6789/0 conn(0x7f8edc08f350 legacy=0x7f8edc13f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:34.505 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:34.487+0000 7f8ed37fe700 1 --2- 172.21.15.42:0/921891407 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8ec4003ab0 0x7f8ec4004420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:50:34.506 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:34.487+0000 7f8ed37fe700 1 --2- 172.21.15.42:0/921891407 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8ec4003f90 0x7f8ec40049f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:50:34.506 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:34.487+0000 7f8ed37fe700 1 --2- 172.21.15.42:0/921891407 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8ec4009660 0x7f8ec400baa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:50:34.506 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:34.487+0000 7f8ed37fe700 1 -- 172.21.15.42:0/921891407 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8edc11bf80 con 0x7f8ec4003f90 2019-12-05T06:50:34.506 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:34.487+0000 7f8ed37fe700 1 -- 172.21.15.42:0/921891407 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ec400bff0 con 0x7f8ec4009660 2019-12-05T06:50:34.507 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:34.487+0000 7f8ed37fe700 1 -- 172.21.15.42:0/921891407 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ec400c1b0 con 0x7f8ec4003ab0 2019-12-05T06:50:34.507 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:34.487+0000 7f8ee3338700 1 --2- 172.21.15.42:0/921891407 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8ec4009660 0x7f8ec400baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:34.509 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:34.487+0000 7f8ee2b37700 1 --2- 172.21.15.42:0/921891407 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8ec4003ab0 0x7f8ec4004420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:34.509 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:34.487+0000 7f8ee2336700 1 --2- 172.21.15.42:0/921891407 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8ec4003f90 0x7f8ec40049f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:34.509 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:34.487+0000 7f8ee3338700 1 -- 172.21.15.42:0/921891407 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8ec4003ab0 msgr2=0x7f8ec4004420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:34.510 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:34.487+0000 7f8ee3338700 1 --2- 172.21.15.42:0/921891407 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8ec4003ab0 0x7f8ec4004420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:34.510 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:34.487+0000 7f8ee3338700 1 -- 172.21.15.42:0/921891407 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8ec4003f90 msgr2=0x7f8ec40049f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:34.510 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:34.487+0000 7f8ee3338700 1 --2- 172.21.15.42:0/921891407 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8ec4003f90 0x7f8ec40049f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:34.513 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:34.487+0000 7f8ee3338700 1 -- 172.21.15.42:0/921891407 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8edc07ae90 con 0x7f8ec4009660 2019-12-05T06:50:34.514 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:34.487+0000 7f8ee459a700 1 -- 172.21.15.42:0/921891407 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8edc07d890 con 0x7f8ec4009660 2019-12-05T06:50:34.514 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:34.487+0000 7f8ee3338700 1 --2- 172.21.15.42:0/921891407 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8ec4009660 0x7f8ec400baa0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f8ed40104c0 tx=0x7f8ed4011a70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:34.514 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:34.487+0000 7f8ee459a700 1 -- 172.21.15.42:0/921891407 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8edc08f780 con 0x7f8ec4009660 2019-12-05T06:50:34.514 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:34.487+0000 7f8ee3338700 1 --2- 172.21.15.42:0/921891407 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8ec4009660 0x7f8ec400baa0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f8ed40104c0 tx=0x7f8ed4011a70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:34.514 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:34.487+0000 7f8ee3338700 1 --2- 172.21.15.42:0/921891407 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8ec4009660 0x7f8ec400baa0 secure :-1 s=READY pgs=3344 cs=0 l=1 rx=0x7f8ed40104c0 tx=0x7f8ed4011a70).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:50:34.515 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:34.487+0000 7f8ee3338700 1 --2- 172.21.15.42:0/921891407 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8ec4009660 0x7f8ec400baa0 secure :-1 s=THROTTLE_DONE pgs=3344 cs=0 l=1 rx=0x7f8ed40104c0 tx=0x7f8ed4011a70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:34.515 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:34.487+0000 7f8ee3338700 1 --2- 172.21.15.42:0/921891407 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8ec4009660 0x7f8ec400baa0 secure :-1 s=THROTTLE_DONE pgs=3344 cs=0 l=1 rx=0x7f8ed40104c0 tx=0x7f8ed4011a70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:34.515 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:34.487+0000 7f8ed37fe700 1 -- 172.21.15.42:0/921891407 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8ed4005ab0 con 0x7f8ec4009660 2019-12-05T06:50:34.515 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:34.487+0000 7f8ee3338700 1 --2- 172.21.15.42:0/921891407 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8ec4009660 0x7f8ec400baa0 secure :-1 s=THROTTLE_DONE pgs=3344 cs=0 l=1 rx=0x7f8ed40104c0 tx=0x7f8ed4011a70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:34.515 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:34.487+0000 7f8ed37fe700 1 -- 172.21.15.42:0/921891407 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ed40110c0 con 0x7f8ec4009660 2019-12-05T06:50:34.515 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:34.487+0000 7f8ed37fe700 1 -- 172.21.15.42:0/921891407 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8ed400f470 con 0x7f8ec4009660 2019-12-05T06:50:34.516 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:34.487+0000 7f8ee3338700 1 --2- 172.21.15.42:0/921891407 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8ec4009660 0x7f8ec400baa0 secure :-1 s=THROTTLE_DONE pgs=3344 cs=0 l=1 rx=0x7f8ed40104c0 tx=0x7f8ed4011a70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:34.516 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:34.487+0000 7f8ee3338700 1 --2- 172.21.15.42:0/921891407 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8ec4009660 0x7f8ec400baa0 secure :-1 s=THROTTLE_DONE pgs=3344 cs=0 l=1 rx=0x7f8ed40104c0 tx=0x7f8ed4011a70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:34.517 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:34.491+0000 7f8ed37fe700 1 -- 172.21.15.42:0/921891407 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f8ed400f670 con 0x7f8ec4009660 2019-12-05T06:50:34.517 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:34.491+0000 7f8ed37fe700 1 --2- 172.21.15.42:0/921891407 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8ec4044050 0x7f8ec4046560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:50:34.517 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:34.491+0000 7f8ed37fe700 1 -- 172.21.15.42:0/921891407 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 12309+0+0 (secure 0 0 0) 0x7f8ed40150a0 con 0x7f8ec4009660 2019-12-05T06:50:34.517 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:34.491+0000 7f8ee2b37700 1 --2- 172.21.15.42:0/921891407 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8ec4044050 0x7f8ec4046560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:34.517 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:34.491+0000 7f8ee2b37700 1 --2- 172.21.15.42:0/921891407 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8ec4044050 0x7f8ec4046560 crc :-1 s=READY pgs=1400 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:50:34.517 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:34.491+0000 7f8ee459a700 1 -- 172.21.15.42:0/921891407 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8eb8003200 con 0x7f8ec4009660 2019-12-05T06:50:34.518 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:34.499+0000 7f8ee3338700 1 --2- 172.21.15.42:0/921891407 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8ec4009660 0x7f8ec400baa0 secure :-1 s=THROTTLE_DONE pgs=3344 cs=0 l=1 rx=0x7f8ed40104c0 tx=0x7f8ed4011a70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:34.518 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:34.499+0000 7f8ed37fe700 1 -- 172.21.15.42:0/921891407 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f8ed4032930 con 0x7f8ec4009660 2019-12-05T06:50:34.666 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:34.659+0000 7f8ee459a700 1 -- 172.21.15.42:0/921891407 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8eb8003630 con 0x7f8ec4009660 2019-12-05T06:50:34.667 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:34.659+0000 7f8ee3338700 1 --2- 172.21.15.42:0/921891407 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8ec4009660 0x7f8ec400baa0 secure :-1 s=THROTTLE_DONE pgs=3344 cs=0 l=1 rx=0x7f8ed40104c0 tx=0x7f8ed4011a70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:34.667 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:34.659+0000 7f8ed37fe700 1 -- 172.21.15.42:0/921891407 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 403 v403) v1 ==== 95+0+1776 (secure 0 0 0) 0x7f8ed40334e0 con 0x7f8ec4009660 2019-12-05T06:50:34.669 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 403 2019-12-05T06:50:34.670 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:50:34.670 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":403,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":403,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:50:25.893719+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":122,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8537},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8537":{"gid":8537,"name":"a","rank":0,"incarnation":400,"state":"up:active","state_seq":20,"addr":"172.21.15.42:6835/1881703925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1881703925},{"type":"v1","addr":"172.21.15.42:6835","nonce":1881703925}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:50:34.673 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:34.667+0000 7f8ee459a700 1 -- 172.21.15.42:0/921891407 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8ec4044050 msgr2=0x7f8ec4046560 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:34.674 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:34.667+0000 7f8ee459a700 1 --2- 172.21.15.42:0/921891407 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8ec4044050 0x7f8ec4046560 crc :-1 s=READY pgs=1400 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:34.674 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:34.667+0000 7f8ee459a700 1 -- 172.21.15.42:0/921891407 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8ec4009660 msgr2=0x7f8ec400baa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:34.674 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:34.667+0000 7f8ee459a700 1 --2- 172.21.15.42:0/921891407 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8ec4009660 0x7f8ec400baa0 secure :-1 s=READY pgs=3344 cs=0 l=1 rx=0x7f8ed40104c0 tx=0x7f8ed4011a70).stop 2019-12-05T06:50:34.674 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:34.667+0000 7f8ee3338700 1 -- 172.21.15.42:0/921891407 reap_dead start 2019-12-05T06:50:34.674 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:34.667+0000 7f8ee3338700 1 -- 172.21.15.42:0/921891407 reap_dead start 2019-12-05T06:50:34.675 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:34.667+0000 7f8ee459a700 1 -- 172.21.15.42:0/921891407 shutdown_connections 2019-12-05T06:50:34.675 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:34.667+0000 7f8ee459a700 1 -- 172.21.15.42:0/921891407 shutdown_connections 2019-12-05T06:50:34.675 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:34.667+0000 7f8ee459a700 1 -- 172.21.15.42:0/921891407 wait complete. 2019-12-05T06:50:34.675 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:34.667+0000 7f8ee459a700 1 -- 172.21.15.42:0/921891407 >> 172.21.15.42:0/921891407 conn(0x7f8edc0a8020 msgr2=0x7f8edc059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:50:34.687 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2019-12-05T06:50:35.133 INFO:tasks.workunit.client.0.smithi042.stderr:+ cd ../.. 2019-12-05T06:50:35.134 INFO:tasks.workunit.client.0.smithi042.stderr:+ ver=v0.18 2019-12-05T06:50:35.134 INFO:tasks.workunit.client.0.smithi042.stderr:+ echo checking v0.18 2019-12-05T06:50:35.134 INFO:tasks.workunit.client.0.smithi042.stderr:+ cd .snap/v0.18 2019-12-05T06:50:35.135 INFO:tasks.workunit.client.0.smithi042.stdout:checking v0.18 2019-12-05T06:50:35.138 INFO:tasks.workunit.client.0.smithi042.stderr:+ git diff --exit-code 2019-12-05T06:50:36.688 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:50:36.794 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:36.778+0000 7fbac972f700 1 Processor -- start 2019-12-05T06:50:36.794 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:36.778+0000 7fbac972f700 1 -- start start 2019-12-05T06:50:36.794 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:36.782+0000 7fbac972f700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fbac40a7740 0x7fbac408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:50:36.794 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:36.782+0000 7fbac972f700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbac409fae0 0x7fbac407eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:50:36.795 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:36.782+0000 7fbac972f700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbac40a0710 con 0x7fbac407e720 2019-12-05T06:50:36.795 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:36.782+0000 7fbac972f700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbac40a0b00 con 0x7fbac40a7740 2019-12-05T06:50:36.795 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:36.782+0000 7fbac972f700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbac408e5b0 con 0x7fbac409fae0 2019-12-05T06:50:36.795 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:36.782+0000 7fbac2ffd700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fbac407e720 0x7fbac409f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:38648/0 (socket says 172.21.15.42:38648) 2019-12-05T06:50:36.795 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:36.782+0000 7fbac2ffd700 1 -- 172.21.15.42:0/2269432221 learned_addr learned my addr 172.21.15.42:0/2269432221 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:50:36.795 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:36.782+0000 7fbac3fff700 1 --2- 172.21.15.42:0/2269432221 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbac409fae0 0x7fbac407eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:36.795 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:36.782+0000 7fbac37fe700 1 --2- 172.21.15.42:0/2269432221 >> v2:172.21.15.40:3300/0 conn(0x7fbac40a7740 0x7fbac408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:36.796 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:36.782+0000 7fbac27fc700 1 -- 172.21.15.42:0/2269432221 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 858986877 0 0) 0x7fbac40a0710 con 0x7fbac407e720 2019-12-05T06:50:36.796 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:36.782+0000 7fbac27fc700 1 -- 172.21.15.42:0/2269432221 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbab00030b0 con 0x7fbac407e720 2019-12-05T06:50:36.796 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:36.782+0000 7fbac27fc700 1 -- 172.21.15.42:0/2269432221 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 154540195 0 0) 0x7fbab00030b0 con 0x7fbac407e720 2019-12-05T06:50:36.796 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:36.782+0000 7fbac37fe700 1 -- 172.21.15.42:0/2269432221 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbac409fae0 msgr2=0x7fbac407eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:50:36.796 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:36.782+0000 7fbac37fe700 1 --2- 172.21.15.42:0/2269432221 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbac409fae0 0x7fbac407eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:50:36.796 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:36.782+0000 7fbac37fe700 1 -- 172.21.15.42:0/2269432221 >> v1:172.21.15.42:6789/0 conn(0x7fbac407e720 legacy=0x7fbac409f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:36.796 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:36.782+0000 7fbac37fe700 1 -- 172.21.15.42:0/2269432221 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbac407f0a0 con 0x7fbac40a7740 2019-12-05T06:50:36.797 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:36.782+0000 7fbac3fff700 1 --2- 172.21.15.42:0/2269432221 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbac409fae0 0x7fbac407eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:50:36.797 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:36.782+0000 7fbac37fe700 1 --2- 172.21.15.42:0/2269432221 >> v2:172.21.15.40:3300/0 conn(0x7fbac40a7740 0x7fbac408df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fbab4008b10 tx=0x7fbaac0019d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:36.797 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:36.782+0000 7fbac37fe700 1 --2- 172.21.15.42:0/2269432221 >> v2:172.21.15.40:3300/0 conn(0x7fbac40a7740 0x7fbac408df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fbab4008b10 tx=0x7fbaac0019d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:36.797 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:36.782+0000 7fbac37fe700 1 --2- 172.21.15.42:0/2269432221 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbac40a7740 0x7fbac408df80 secure :-1 s=READY pgs=2526 cs=0 l=1 rx=0x7fbab4008b10 tx=0x7fbaac0019d0).ready entity=mon.0 client_cookie=da4dc04e63e5329b server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:50:36.797 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:36.782+0000 7fbac37fe700 1 --2- 172.21.15.42:0/2269432221 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbac40a7740 0x7fbac408df80 secure :-1 s=THROTTLE_DONE pgs=2526 cs=0 l=1 rx=0x7fbab4008b10 tx=0x7fbaac0019d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:36.797 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:36.782+0000 7fbac27fc700 1 -- 172.21.15.42:0/2269432221 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbaac001750 con 0x7fbac40a7740 2019-12-05T06:50:36.797 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:36.782+0000 7fbac37fe700 1 --2- 172.21.15.42:0/2269432221 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbac40a7740 0x7fbac408df80 secure :-1 s=THROTTLE_DONE pgs=2526 cs=0 l=1 rx=0x7fbab4008b10 tx=0x7fbaac0019d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:36.798 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:36.782+0000 7fbac37fe700 1 --2- 172.21.15.42:0/2269432221 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbac40a7740 0x7fbac408df80 secure :-1 s=THROTTLE_DONE pgs=2526 cs=0 l=1 rx=0x7fbab4008b10 tx=0x7fbaac0019d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:36.798 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:36.782+0000 7fbac27fc700 1 -- 172.21.15.42:0/2269432221 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbaac001560 con 0x7fbac40a7740 2019-12-05T06:50:36.798 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:36.782+0000 7fbac27fc700 1 -- 172.21.15.42:0/2269432221 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbab40109f0 con 0x7fbac40a7740 2019-12-05T06:50:36.798 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:36.782+0000 7fbac972f700 1 -- 172.21.15.42:0/2269432221 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbac40a7740 msgr2=0x7fbac408df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:36.798 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:36.782+0000 7fbac972f700 1 --2- 172.21.15.42:0/2269432221 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbac40a7740 0x7fbac408df80 secure :-1 s=READY pgs=2526 cs=0 l=1 rx=0x7fbab4008b10 tx=0x7fbaac0019d0).stop 2019-12-05T06:50:36.798 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:36.782+0000 7fbac972f700 1 -- 172.21.15.42:0/2269432221 shutdown_connections 2019-12-05T06:50:36.798 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:36.782+0000 7fbac972f700 1 --2- 172.21.15.42:0/2269432221 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbac409fae0 0x7fbac407eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:50:36.799 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:36.782+0000 7fbac972f700 1 --2- 172.21.15.42:0/2269432221 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbac40a7740 0x7fbac408df80 unknown :-1 s=CLOSED pgs=2526 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:36.799 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:36.782+0000 7fbac972f700 1 -- 172.21.15.42:0/2269432221 shutdown_connections 2019-12-05T06:50:36.800 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:36.782+0000 7fbac972f700 1 -- 172.21.15.42:0/2269432221 wait complete. 2019-12-05T06:50:36.800 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:36.782+0000 7fbac972f700 1 -- 172.21.15.42:0/2269432221 >> 172.21.15.42:0/2269432221 conn(0x7fbac40aabe0 msgr2=0x7fbac40a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:50:36.801 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:36.786+0000 7fbac972f700 1 Processor -- start 2019-12-05T06:50:36.801 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:36.786+0000 7fbac972f700 1 -- start start 2019-12-05T06:50:36.805 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:36.786+0000 7fbac972f700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fbac411cb10 0x7fbac4136890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:50:36.805 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:36.786+0000 7fbac972f700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbac40a3f60 con 0x7fbac4136de0 2019-12-05T06:50:36.805 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:36.786+0000 7fbac972f700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbac412ed60 con 0x7fbac412e930 2019-12-05T06:50:36.805 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:36.786+0000 7fbac972f700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbac40a4a50 con 0x7fbac411cb10 2019-12-05T06:50:36.805 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:36.786+0000 7fbac37fe700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fbac412e930 0x7fbac412ae60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:38652/0 (socket says 172.21.15.42:38652) 2019-12-05T06:50:36.805 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:36.786+0000 7fbac37fe700 1 -- 172.21.15.42:0/508602112 learned_addr learned my addr 172.21.15.42:0/508602112 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:50:36.805 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:36.786+0000 7fbac2ffd700 1 --2- 172.21.15.42:0/508602112 >> v2:172.21.15.40:3300/0 conn(0x7fbac411cb10 0x7fbac4136890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:36.806 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:36.786+0000 7fbabbfff700 1 -- 172.21.15.42:0/508602112 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3162318885 0 0) 0x7fbac412ed60 con 0x7fbac412e930 2019-12-05T06:50:36.806 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:36.786+0000 7fbabbfff700 1 -- 172.21.15.42:0/508602112 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbaa80030b0 con 0x7fbac412e930 2019-12-05T06:50:36.806 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:36.790+0000 7fbac2ffd700 1 -- 172.21.15.42:0/508602112 >> v1:172.21.15.42:6789/0 conn(0x7fbac412e930 legacy=0x7fbac412ae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:36.806 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:36.790+0000 7fbabbfff700 1 -- 172.21.15.42:0/508602112 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 2143942027 0 0) 0x7fbaa80030b0 con 0x7fbac412e930 2019-12-05T06:50:36.806 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:36.790+0000 7fbac2ffd700 1 -- 172.21.15.42:0/508602112 >> v1:172.21.15.40:6789/0 conn(0x7fbac4136de0 legacy=0x7fbac4139220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:36.806 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:36.790+0000 7fbac2ffd700 1 -- 172.21.15.42:0/508602112 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbac40a4240 con 0x7fbac411cb10 2019-12-05T06:50:36.807 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:36.790+0000 7fbac2ffd700 1 --2- 172.21.15.42:0/508602112 >> v2:172.21.15.40:3300/0 conn(0x7fbac411cb10 0x7fbac4136890 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fbabc002ef0 tx=0x7fbabc007ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:36.807 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:36.790+0000 7fbac2ffd700 1 --2- 172.21.15.42:0/508602112 >> v2:172.21.15.40:3300/0 conn(0x7fbac411cb10 0x7fbac4136890 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fbabc002ef0 tx=0x7fbabc007ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:36.807 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:36.790+0000 7fbac2ffd700 1 --2- 172.21.15.42:0/508602112 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbac411cb10 0x7fbac4136890 secure :-1 s=READY pgs=2528 cs=0 l=1 rx=0x7fbabc002ef0 tx=0x7fbabc007ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:50:36.807 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:36.790+0000 7fbac2ffd700 1 --2- 172.21.15.42:0/508602112 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbac411cb10 0x7fbac4136890 secure :-1 s=THROTTLE_DONE pgs=2528 cs=0 l=1 rx=0x7fbabc002ef0 tx=0x7fbabc007ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:36.807 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:36.790+0000 7fbabbfff700 1 -- 172.21.15.42:0/508602112 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbab40087f0 con 0x7fbac411cb10 2019-12-05T06:50:36.807 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:36.790+0000 7fbac2ffd700 1 --2- 172.21.15.42:0/508602112 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbac411cb10 0x7fbac4136890 secure :-1 s=THROTTLE_DONE pgs=2528 cs=0 l=1 rx=0x7fbabc002ef0 tx=0x7fbabc007ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:36.807 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:36.790+0000 7fbac972f700 1 -- 172.21.15.42:0/508602112 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbac411b050 con 0x7fbac411cb10 2019-12-05T06:50:36.808 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:36.790+0000 7fbabbfff700 1 -- 172.21.15.42:0/508602112 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbab4008d40 con 0x7fbac411cb10 2019-12-05T06:50:36.808 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:36.790+0000 7fbac2ffd700 1 --2- 172.21.15.42:0/508602112 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbac411cb10 0x7fbac4136890 secure :-1 s=THROTTLE_DONE pgs=2528 cs=0 l=1 rx=0x7fbabc002ef0 tx=0x7fbabc007ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:36.808 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:36.790+0000 7fbabbfff700 1 -- 172.21.15.42:0/508602112 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbaac011780 con 0x7fbac411cb10 2019-12-05T06:50:36.808 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:36.790+0000 7fbac972f700 1 -- 172.21.15.42:0/508602112 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbac411cf40 con 0x7fbac411cb10 2019-12-05T06:50:36.808 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:36.790+0000 7fbac2ffd700 1 --2- 172.21.15.42:0/508602112 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbac411cb10 0x7fbac4136890 secure :-1 s=THROTTLE_DONE pgs=2528 cs=0 l=1 rx=0x7fbabc002ef0 tx=0x7fbabc007ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:36.808 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:36.790+0000 7fbabbfff700 1 -- 172.21.15.42:0/508602112 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fbaac011a60 con 0x7fbac411cb10 2019-12-05T06:50:36.808 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:36.790+0000 7fbac2ffd700 1 --2- 172.21.15.42:0/508602112 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbac411cb10 0x7fbac4136890 secure :-1 s=THROTTLE_DONE pgs=2528 cs=0 l=1 rx=0x7fbabc002ef0 tx=0x7fbabc007ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:36.809 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:36.790+0000 7fbabbfff700 1 --2- 172.21.15.42:0/508602112 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbaa803b160 0x7fbaa803d670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:50:36.809 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:36.790+0000 7fbabbfff700 1 -- 172.21.15.42:0/508602112 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 12309+0+0 (secure 0 0 0) 0x7fbaac04a830 con 0x7fbac411cb10 2019-12-05T06:50:36.809 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:36.790+0000 7fbac37fe700 1 --2- 172.21.15.42:0/508602112 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbaa803b160 0x7fbaa803d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:36.809 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:36.794+0000 7fbac37fe700 1 --2- 172.21.15.42:0/508602112 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbaa803b160 0x7fbaa803d670 crc :-1 s=READY pgs=1402 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:50:36.809 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:36.794+0000 7fbac972f700 1 -- 172.21.15.42:0/508602112 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbac405d110 con 0x7fbac411cb10 2019-12-05T06:50:36.810 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:36.802+0000 7fbac2ffd700 1 --2- 172.21.15.42:0/508602112 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbac411cb10 0x7fbac4136890 secure :-1 s=THROTTLE_DONE pgs=2528 cs=0 l=1 rx=0x7fbabc002ef0 tx=0x7fbabc007ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:36.810 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:36.802+0000 7fbabbfff700 1 -- 172.21.15.42:0/508602112 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fbaac031830 con 0x7fbac411cb10 2019-12-05T06:50:36.985 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:36.978+0000 7fbac972f700 1 -- 172.21.15.42:0/508602112 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbac405b8c0 con 0x7fbac411cb10 2019-12-05T06:50:36.986 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:36.978+0000 7fbac2ffd700 1 --2- 172.21.15.42:0/508602112 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbac411cb10 0x7fbac4136890 secure :-1 s=THROTTLE_DONE pgs=2528 cs=0 l=1 rx=0x7fbabc002ef0 tx=0x7fbabc007ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:36.986 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:36.978+0000 7fbabbfff700 1 -- 172.21.15.42:0/508602112 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 404 v404) v1 ==== 95+0+2111 (secure 0 0 0) 0x7fbaac031a90 con 0x7fbac411cb10 2019-12-05T06:50:36.988 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 404 2019-12-05T06:50:36.989 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:50:36.989 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":404,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":12948,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.40:6833/478622344","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":478622344},{"type":"v1","addr":"172.21.15.40:6833","nonce":478622344}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":404}],"filesystems":[{"mdsmap":{"epoch":403,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T05:03:35.604738+0000","modified":"2019-12-05T06:50:25.893719+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":122,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8537},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8537":{"gid":8537,"name":"a","rank":0,"incarnation":400,"state":"up:active","state_seq":20,"addr":"172.21.15.42:6835/1881703925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":1881703925},{"type":"v1","addr":"172.21.15.42:6835","nonce":1881703925}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:50:36.991 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:36.982+0000 7fbac972f700 1 -- 172.21.15.42:0/508602112 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbaa803b160 msgr2=0x7fbaa803d670 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:36.991 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:36.982+0000 7fbac972f700 1 --2- 172.21.15.42:0/508602112 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbaa803b160 0x7fbaa803d670 crc :-1 s=READY pgs=1402 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:36.991 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:36.982+0000 7fbac972f700 1 -- 172.21.15.42:0/508602112 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbac411cb10 msgr2=0x7fbac4136890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:36.992 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:36.982+0000 7fbac972f700 1 --2- 172.21.15.42:0/508602112 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbac411cb10 0x7fbac4136890 secure :-1 s=READY pgs=2528 cs=0 l=1 rx=0x7fbabc002ef0 tx=0x7fbabc007ad0).stop 2019-12-05T06:50:36.992 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:36.982+0000 7fbac972f700 1 -- 172.21.15.42:0/508602112 shutdown_connections 2019-12-05T06:50:36.992 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:36.982+0000 7fbac972f700 1 --2- 172.21.15.42:0/508602112 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fbaa803b160 0x7fbaa803d670 unknown :-1 s=CLOSED pgs=1402 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:36.992 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:36.982+0000 7fbac972f700 1 --2- 172.21.15.42:0/508602112 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbac411cb10 0x7fbac4136890 unknown :-1 s=CLOSED pgs=2528 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:36.992 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:36.982+0000 7fbac972f700 1 -- 172.21.15.42:0/508602112 shutdown_connections 2019-12-05T06:50:36.992 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:36.982+0000 7fbac972f700 1 -- 172.21.15.42:0/508602112 wait complete. 2019-12-05T06:50:36.993 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:50:36.982+0000 7fbac972f700 1 -- 172.21.15.42:0/508602112 >> 172.21.15.42:0/508602112 conn(0x7fbac407deb0 msgr2=0x7fbac407e4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:50:37.002 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b reported in up:standby state 2019-12-05T06:50:37.002 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 73 secs before thrashing 2019-12-05T06:50:43.986 INFO:tasks.workunit.client.0.smithi042.stderr:+ cd ../.. 2019-12-05T06:50:43.987 INFO:tasks.workunit.client.0.smithi042.stderr:+ ver=v0.19 2019-12-05T06:50:43.987 INFO:tasks.workunit.client.0.smithi042.stderr:+ echo checking v0.19 2019-12-05T06:50:43.988 INFO:tasks.workunit.client.0.smithi042.stderr:+ cd .snap/v0.19 2019-12-05T06:50:43.988 INFO:tasks.workunit.client.0.smithi042.stdout:checking v0.19 2019-12-05T06:50:43.991 INFO:tasks.workunit.client.0.smithi042.stderr:+ git diff --exit-code 2019-12-05T06:50:48.613 INFO:tasks.workunit.client.0.smithi042.stderr:+ cd ../.. 2019-12-05T06:50:48.614 INFO:tasks.workunit.client.0.smithi042.stderr:+ ver=v0.20 2019-12-05T06:50:48.615 INFO:tasks.workunit.client.0.smithi042.stderr:+ echo checking v0.20 2019-12-05T06:50:48.615 INFO:tasks.workunit.client.0.smithi042.stderr:+ cd .snap/v0.20 2019-12-05T06:50:48.615 INFO:tasks.workunit.client.0.smithi042.stdout:checking v0.20 2019-12-05T06:50:48.617 INFO:tasks.workunit.client.0.smithi042.stderr:+ git diff --exit-code 2019-12-05T06:50:50.836 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:50:50.840 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:50:58.869 INFO:tasks.workunit.client.0.smithi042.stderr:+ cd ../.. 2019-12-05T06:50:58.869 INFO:tasks.workunit.client.0.smithi042.stderr:+ ver=v0.21 2019-12-05T06:50:58.869 INFO:tasks.workunit.client.0.smithi042.stderr:+ echo checking v0.21 2019-12-05T06:50:58.869 INFO:tasks.workunit.client.0.smithi042.stderr:+ cd .snap/v0.21 2019-12-05T06:50:58.870 INFO:tasks.workunit.client.0.smithi042.stdout:checking v0.21 2019-12-05T06:50:58.872 INFO:tasks.workunit.client.0.smithi042.stderr:+ git diff --exit-code 2019-12-05T06:51:07.585 INFO:tasks.workunit.client.0.smithi042.stderr:+ cd ../.. 2019-12-05T06:51:07.586 INFO:tasks.workunit.client.0.smithi042.stderr:+ ver=v0.1 2019-12-05T06:51:07.586 INFO:tasks.workunit.client.0.smithi042.stderr:+ rmdir .snap/v0.1 2019-12-05T06:51:07.616 INFO:tasks.workunit.client.0.smithi042.stderr:+ ver=v0.2 2019-12-05T06:51:07.617 INFO:tasks.workunit.client.0.smithi042.stderr:+ rmdir .snap/v0.2 2019-12-05T06:51:07.662 INFO:tasks.workunit.client.0.smithi042.stderr:+ ver=v0.3 2019-12-05T06:51:07.663 INFO:tasks.workunit.client.0.smithi042.stderr:+ rmdir .snap/v0.3 2019-12-05T06:51:07.702 INFO:tasks.workunit.client.0.smithi042.stderr:+ ver=v0.4 2019-12-05T06:51:07.702 INFO:tasks.workunit.client.0.smithi042.stderr:+ rmdir .snap/v0.4 2019-12-05T06:51:07.745 INFO:tasks.workunit.client.0.smithi042.stderr:+ ver=v0.5 2019-12-05T06:51:07.746 INFO:tasks.workunit.client.0.smithi042.stderr:+ rmdir .snap/v0.5 2019-12-05T06:51:07.789 INFO:tasks.workunit.client.0.smithi042.stderr:+ ver=v0.6 2019-12-05T06:51:07.790 INFO:tasks.workunit.client.0.smithi042.stderr:+ rmdir .snap/v0.6 2019-12-05T06:51:07.822 INFO:tasks.workunit.client.0.smithi042.stderr:+ ver=v0.7 2019-12-05T06:51:07.822 INFO:tasks.workunit.client.0.smithi042.stderr:+ rmdir .snap/v0.7 2019-12-05T06:51:07.854 INFO:tasks.workunit.client.0.smithi042.stderr:+ ver=v0.8 2019-12-05T06:51:07.854 INFO:tasks.workunit.client.0.smithi042.stderr:+ rmdir .snap/v0.8 2019-12-05T06:51:07.886 INFO:tasks.workunit.client.0.smithi042.stderr:+ ver=v0.9 2019-12-05T06:51:07.886 INFO:tasks.workunit.client.0.smithi042.stderr:+ rmdir .snap/v0.9 2019-12-05T06:51:07.918 INFO:tasks.workunit.client.0.smithi042.stderr:+ ver=v0.10 2019-12-05T06:51:07.919 INFO:tasks.workunit.client.0.smithi042.stderr:+ rmdir .snap/v0.10 2019-12-05T06:51:07.950 INFO:tasks.workunit.client.0.smithi042.stderr:+ ver=v0.11 2019-12-05T06:51:07.950 INFO:tasks.workunit.client.0.smithi042.stderr:+ rmdir .snap/v0.11 2019-12-05T06:51:07.981 INFO:tasks.workunit.client.0.smithi042.stderr:+ ver=v0.12 2019-12-05T06:51:07.981 INFO:tasks.workunit.client.0.smithi042.stderr:+ rmdir .snap/v0.12 2019-12-05T06:51:08.013 INFO:tasks.workunit.client.0.smithi042.stderr:+ ver=v0.13 2019-12-05T06:51:08.014 INFO:tasks.workunit.client.0.smithi042.stderr:+ rmdir .snap/v0.13 2019-12-05T06:51:08.048 INFO:tasks.workunit.client.0.smithi042.stderr:+ ver=v0.14 2019-12-05T06:51:08.049 INFO:tasks.workunit.client.0.smithi042.stderr:+ rmdir .snap/v0.14 2019-12-05T06:51:08.100 INFO:tasks.workunit.client.0.smithi042.stderr:+ ver=v0.15 2019-12-05T06:51:08.100 INFO:tasks.workunit.client.0.smithi042.stderr:+ rmdir .snap/v0.15 2019-12-05T06:51:08.132 INFO:tasks.workunit.client.0.smithi042.stderr:+ ver=v0.16 2019-12-05T06:51:08.132 INFO:tasks.workunit.client.0.smithi042.stderr:+ rmdir .snap/v0.16 2019-12-05T06:51:08.178 INFO:tasks.workunit.client.0.smithi042.stderr:+ ver=v0.17 2019-12-05T06:51:08.178 INFO:tasks.workunit.client.0.smithi042.stderr:+ rmdir .snap/v0.17 2019-12-05T06:51:08.214 INFO:tasks.workunit.client.0.smithi042.stderr:+ ver=v0.18 2019-12-05T06:51:08.215 INFO:tasks.workunit.client.0.smithi042.stderr:+ rmdir .snap/v0.18 2019-12-05T06:51:08.258 INFO:tasks.workunit.client.0.smithi042.stderr:+ ver=v0.19 2019-12-05T06:51:08.259 INFO:tasks.workunit.client.0.smithi042.stderr:+ rmdir .snap/v0.19 2019-12-05T06:51:08.299 INFO:tasks.workunit.client.0.smithi042.stderr:+ ver=v0.20 2019-12-05T06:51:08.299 INFO:tasks.workunit.client.0.smithi042.stderr:+ rmdir .snap/v0.20 2019-12-05T06:51:09.189 INFO:tasks.workunit.client.0.smithi042.stderr:+ ver=v0.21 2019-12-05T06:51:09.189 INFO:tasks.workunit.client.0.smithi042.stderr:+ rmdir .snap/v0.21 2019-12-05T06:51:09.253 INFO:tasks.workunit.client.0.smithi042.stderr:+ echo OK 2019-12-05T06:51:09.253 INFO:tasks.workunit.client.0.smithi042.stdout:OK 2019-12-05T06:51:09.254 INFO:teuthology.orchestra.run:Running command with timeout 3600 2019-12-05T06:51:09.255 INFO:teuthology.orchestra.run.smithi042:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2019-12-05T06:51:11.436 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:11.429+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 122'5354 (114'2305,122'5354] local-lis/les=21/22 n=1163 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=122'5354 lcod 122'5353 mlcod 122'5353 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.437 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:11.429+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 122'5165 (113'2106,122'5165] local-lis/les=21/22 n=1104 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=122'5165 lcod 122'5164 mlcod 122'5164 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.439 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:11.433+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5356 (114'2305,123'5356] local-lis/les=21/22 n=1161 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=122'5354 crt=122'5354 lcod 122'5353 mlcod 122'5353 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.441 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:11.433+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5167 (113'2106,123'5167] local-lis/les=21/22 n=1102 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=122'5165 crt=122'5165 lcod 122'5164 mlcod 122'5164 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.446 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:11.438+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 122'5482 (115'2409,122'5482] local-lis/les=21/22 n=1178 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=122'5482 lcod 122'5481 mlcod 122'5481 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.446 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:11.438+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 122'5184 (113'2103,122'5184] local-lis/les=21/22 n=1035 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=122'5184 lcod 122'5183 mlcod 122'5183 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.447 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:11.442+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5484 (115'2409,123'5484] local-lis/les=21/22 n=1176 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=122'5482 crt=122'5482 lcod 122'5481 mlcod 122'5481 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.448 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:11.442+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5186 (113'2103,123'5186] local-lis/les=21/22 n=1033 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=122'5184 crt=122'5184 lcod 122'5183 mlcod 122'5183 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.449 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:11.446+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 122'5504 (115'2407,122'5504] local-lis/les=21/22 n=1185 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=122'5504 lcod 122'5503 mlcod 122'5503 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.449 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:11.446+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 122'5286 (114'2202,122'5286] local-lis/les=21/22 n=1179 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=122'5286 lcod 122'5285 mlcod 122'5285 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.452 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:11.446+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 122'5289 (114'2204,122'5289] local-lis/les=21/22 n=1100 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=122'5289 lcod 122'5288 mlcod 122'5288 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.453 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:11.446+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5506 (115'2407,123'5506] local-lis/les=21/22 n=1183 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=122'5504 crt=122'5504 lcod 122'5503 mlcod 122'5503 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.455 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:11.450+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5288 (114'2202,123'5288] local-lis/les=21/22 n=1177 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=122'5286 crt=122'5286 lcod 122'5285 mlcod 122'5285 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.456 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:11.445+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5358 (114'2305,123'5358] local-lis/les=21/22 n=1159 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=123'5358 lcod 123'5356 mlcod 123'5356 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.456 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:11.445+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5169 (113'2106,123'5169] local-lis/les=21/22 n=1100 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5169 lcod 123'5167 mlcod 123'5167 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.456 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:11.450+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5291 (114'2204,123'5291] local-lis/les=21/22 n=1098 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=122'5289 crt=122'5289 lcod 122'5288 mlcod 122'5288 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.457 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:11.450+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 122'5207 (114'2205,122'5207] local-lis/les=21/22 n=1188 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=122'5207 lcod 122'5206 mlcod 122'5206 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.457 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:11.449+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5171 (113'2106,123'5171] local-lis/les=21/22 n=1098 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5169 crt=123'5169 lcod 123'5167 mlcod 123'5167 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.461 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:11.453+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5360 (114'2305,123'5360] local-lis/les=21/22 n=1157 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5358 crt=123'5358 lcod 123'5356 mlcod 123'5356 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.465 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:11.458+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5209 (114'2205,123'5209] local-lis/les=21/22 n=1186 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=122'5207 crt=122'5207 lcod 122'5206 mlcod 122'5206 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.465 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:11.458+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5486 (115'2409,123'5486] local-lis/les=21/22 n=1174 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'5486 lcod 123'5484 mlcod 123'5484 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.466 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:11.458+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5188 (113'2103,123'5188] local-lis/les=21/22 n=1031 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5188 lcod 123'5186 mlcod 123'5186 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.466 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:11.462+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5488 (115'2409,123'5488] local-lis/les=21/22 n=1172 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'5486 crt=123'5486 lcod 123'5484 mlcod 123'5484 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.472 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:11.466+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5190 (113'2103,123'5190] local-lis/les=21/22 n=1029 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5188 crt=123'5188 lcod 123'5186 mlcod 123'5186 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.472 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:11.466+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5508 (115'2407,123'5508] local-lis/les=21/22 n=1181 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'5508 lcod 123'5506 mlcod 123'5506 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.481 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:11.474+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5294 (114'2204,123'5294] local-lis/les=21/22 n=1095 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5293 crt=123'5293 lcod 123'5291 mlcod 123'5291 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.481 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:11.474+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5296 (114'2204,123'5296] local-lis/les=21/22 n=1093 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5293 crt=123'5293 lcod 123'5291 mlcod 123'5291 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.482 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:11.470+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5510 (115'2508,123'5510] local-lis/les=21/22 n=1179 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'5508 crt=123'5508 lcod 123'5506 mlcod 123'5506 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.482 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:11.474+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5290 (114'2202,123'5290] local-lis/les=21/22 n=1175 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5290 lcod 123'5288 mlcod 123'5288 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.482 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:11.478+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5292 (114'2202,123'5292] local-lis/les=21/22 n=1173 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5290 crt=123'5290 lcod 123'5288 mlcod 123'5288 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.482 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:11.478+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5192 (113'2103,123'5192] local-lis/les=21/22 n=1027 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5192 lcod 123'5190 mlcod 123'5190 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.484 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:11.478+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5491 (115'2409,123'5491] local-lis/les=21/22 n=1169 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'5490 crt=123'5486 lcod 123'5488 mlcod 123'5488 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.485 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:11.482+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5194 (113'2103,123'5194] local-lis/les=21/22 n=1025 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5192 crt=123'5192 lcod 123'5190 mlcod 123'5190 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.485 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:11.482+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5211 (114'2205,123'5211] local-lis/les=21/22 n=1184 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5211 lcod 123'5209 mlcod 123'5209 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.488 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:11.481+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5362 (114'2305,123'5362] local-lis/les=21/22 n=1155 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=123'5362 lcod 123'5360 mlcod 123'5360 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.490 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:11.486+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5213 (114'2205,123'5213] local-lis/les=21/22 n=1182 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5211 crt=123'5211 lcod 123'5209 mlcod 123'5209 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.492 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:11.486+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5512 (115'2508,123'5512] local-lis/les=21/22 n=1177 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'5512 lcod 123'5510 mlcod 123'5510 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.492 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:11.490+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5514 (115'2508,123'5514] local-lis/les=21/22 n=1175 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'5512 crt=123'5512 lcod 123'5510 mlcod 123'5510 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.493 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:11.485+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5364 (114'2305,123'5364] local-lis/les=21/22 n=1153 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5362 crt=123'5362 lcod 123'5360 mlcod 123'5360 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.495 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:11.482+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5493 (115'2409,123'5493] local-lis/les=21/22 n=1167 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'5490 crt=123'5490 lcod 123'5488 mlcod 123'5488 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.496 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:11.494+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5294 (114'2202,123'5294] local-lis/les=21/22 n=1171 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5294 lcod 123'5292 mlcod 123'5292 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.497 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:11.490+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5196 (113'2103,123'5196] local-lis/les=21/22 n=1023 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5196 lcod 123'5194 mlcod 123'5194 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.501 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:11.494+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5198 (113'2103,123'5198] local-lis/les=21/22 n=1021 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5196 crt=123'5196 lcod 123'5194 mlcod 123'5194 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.501 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:11.494+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5296 (114'2202,123'5296] local-lis/les=21/22 n=1169 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5294 crt=123'5294 lcod 123'5292 mlcod 123'5292 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.504 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:11.498+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5298 (114'2204,123'5298] local-lis/les=21/22 n=1091 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5298 lcod 123'5296 mlcod 123'5296 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.516 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:11.505+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5367 (114'2305,123'5367] local-lis/les=21/22 n=1150 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=123'5367 lcod 123'5366 mlcod 123'5366 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.523 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:11.502+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5300 (114'2204,123'5300] local-lis/les=21/22 n=1089 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5298 crt=123'5298 lcod 123'5296 mlcod 123'5296 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.523 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:11.510+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5215 (114'2205,123'5215] local-lis/les=21/22 n=1180 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5215 lcod 123'5213 mlcod 123'5213 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.523 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:11.502+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5495 (115'2409,123'5495] local-lis/les=21/22 n=1165 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'5495 lcod 123'5493 mlcod 123'5493 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.523 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:11.506+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5497 (115'2409,123'5497] local-lis/les=21/22 n=1163 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'5495 crt=123'5495 lcod 123'5493 mlcod 123'5493 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.525 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:11.506+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5516 (115'2508,123'5516] local-lis/les=21/22 n=1173 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'5516 lcod 123'5514 mlcod 123'5514 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.525 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:11.506+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5518 (115'2508,123'5518] local-lis/les=21/22 n=1171 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'5516 crt=123'5516 lcod 123'5514 mlcod 123'5514 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.525 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:11.510+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5298 (114'2202,123'5298] local-lis/les=21/22 n=1167 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5298 lcod 123'5296 mlcod 123'5296 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.526 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:11.510+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5300 (114'2202,123'5300] local-lis/les=21/22 n=1165 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5298 crt=123'5298 lcod 123'5296 mlcod 123'5296 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.527 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:11.513+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5369 (114'2305,123'5369] local-lis/les=21/22 n=1148 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5367 crt=123'5367 lcod 123'5366 mlcod 123'5366 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.527 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:11.510+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5217 (114'2205,123'5217] local-lis/les=21/22 n=1178 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5215 crt=123'5215 lcod 123'5213 mlcod 123'5213 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.527 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:11.514+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5201 (113'2103,123'5201] local-lis/les=21/22 n=1018 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5201 lcod 123'5200 mlcod 123'5200 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.527 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:11.518+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5500 (115'2409,123'5500] local-lis/les=21/22 n=1160 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'5499 crt=123'5495 lcod 123'5497 mlcod 123'5497 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.528 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:11.522+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5203 (113'2103,123'5203] local-lis/les=21/22 n=1016 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5201 crt=123'5201 lcod 123'5200 mlcod 123'5200 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.528 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:11.522+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5502 (115'2409,123'5502] local-lis/les=21/22 n=1158 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'5499 crt=123'5499 lcod 123'5497 mlcod 123'5497 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.530 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:11.521+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5174 (113'2106,123'5174] local-lis/les=21/22 n=1095 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5173 crt=123'5171 lcod 123'5171 mlcod 123'5171 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.531 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:11.526+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5520 (115'2508,123'5520] local-lis/les=21/22 n=1169 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'5520 lcod 123'5518 mlcod 123'5518 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.532 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:11.522+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5302 (114'2204,123'5302] local-lis/les=21/22 n=1087 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5302 lcod 123'5300 mlcod 123'5300 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.533 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:11.530+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5302 (114'2202,123'5302] local-lis/les=21/22 n=1163 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5302 lcod 123'5300 mlcod 123'5300 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.534 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:11.525+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5176 (113'2106,123'5176] local-lis/les=21/22 n=1093 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5173 crt=123'5173 lcod 123'5171 mlcod 123'5171 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.535 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:11.530+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5522 (115'2508,123'5522] local-lis/les=21/22 n=1167 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'5520 crt=123'5520 lcod 123'5518 mlcod 123'5518 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.536 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:11.526+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5304 (114'2204,123'5304] local-lis/les=21/22 n=1085 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5302 crt=123'5302 lcod 123'5300 mlcod 123'5300 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.538 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:11.534+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5219 (114'2205,123'5219] local-lis/les=21/22 n=1176 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5219 lcod 123'5217 mlcod 123'5217 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.539 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:11.534+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5304 (115'2302,123'5304] local-lis/les=21/22 n=1161 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5302 crt=123'5302 lcod 123'5300 mlcod 123'5300 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.541 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:11.534+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5221 (114'2205,123'5221] local-lis/les=21/22 n=1174 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5219 crt=123'5219 lcod 123'5217 mlcod 123'5217 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.546 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:11.537+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5372 (114'2305,123'5372] local-lis/les=21/22 n=1145 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5371 crt=123'5371 lcod 123'5369 mlcod 123'5369 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.547 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:11.542+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5205 (114'2203,123'5205] local-lis/les=21/22 n=1014 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5205 lcod 123'5203 mlcod 123'5203 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.548 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:11.541+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5374 (114'2305,123'5374] local-lis/les=21/22 n=1143 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5371 crt=123'5371 lcod 123'5369 mlcod 123'5369 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.550 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:11.542+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5504 (115'2409,123'5504] local-lis/les=21/22 n=1156 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'5504 lcod 123'5502 mlcod 123'5502 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.551 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:11.546+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5524 (115'2508,123'5524] local-lis/les=21/22 n=1165 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'5524 lcod 123'5522 mlcod 123'5522 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.552 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:11.550+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5306 (114'2304,123'5306] local-lis/les=21/22 n=1083 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5306 lcod 123'5304 mlcod 123'5304 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.553 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:11.550+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5506 (115'2409,123'5506] local-lis/les=21/22 n=1154 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'5504 crt=123'5504 lcod 123'5502 mlcod 123'5502 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.553 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:11.550+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5207 (114'2203,123'5207] local-lis/les=21/22 n=1012 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5205 crt=123'5205 lcod 123'5203 mlcod 123'5203 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.556 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:11.550+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5308 (114'2304,123'5308] local-lis/les=21/22 n=1081 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5306 crt=123'5306 lcod 123'5304 mlcod 123'5304 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.556 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:11.550+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5526 (115'2508,123'5526] local-lis/les=21/22 n=1163 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'5524 crt=123'5524 lcod 123'5522 mlcod 123'5522 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.558 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:11.554+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5223 (114'2205,123'5223] local-lis/les=21/22 n=1172 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5223 lcod 123'5221 mlcod 123'5221 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.565 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:11.558+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5306 (115'2302,123'5306] local-lis/les=21/22 n=1159 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5306 lcod 123'5304 mlcod 123'5304 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.565 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:11.562+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5308 (115'2302,123'5308] local-lis/les=21/22 n=1157 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5306 crt=123'5306 lcod 123'5304 mlcod 123'5304 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.565 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:11.558+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5225 (114'2205,123'5225] local-lis/les=21/22 n=1170 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5223 crt=123'5223 lcod 123'5221 mlcod 123'5221 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.566 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:11.561+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5376 (114'2305,123'5376] local-lis/les=21/22 n=1141 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=123'5376 lcod 123'5374 mlcod 123'5374 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.566 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:11.562+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5528 (115'2508,123'5528] local-lis/les=21/22 n=1161 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'5528 lcod 123'5526 mlcod 123'5526 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.569 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:11.561+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5378 (114'2305,123'5378] local-lis/les=21/22 n=1139 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5376 crt=123'5376 lcod 123'5374 mlcod 123'5374 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.570 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:11.566+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5530 (115'2508,123'5530] local-lis/les=21/22 n=1159 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'5528 crt=123'5528 lcod 123'5526 mlcod 123'5526 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.570 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:11.566+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5310 (114'2304,123'5310] local-lis/les=21/22 n=1079 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5310 lcod 123'5308 mlcod 123'5308 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.571 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:11.566+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5509 (115'2409,123'5509] local-lis/les=21/22 n=1151 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'5508 crt=123'5504 lcod 123'5506 mlcod 123'5506 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.571 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:11.566+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5209 (114'2203,123'5209] local-lis/les=21/22 n=1010 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5209 lcod 123'5207 mlcod 123'5207 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.575 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:11.570+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5511 (115'2409,123'5511] local-lis/les=21/22 n=1149 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'5508 crt=123'5508 lcod 123'5506 mlcod 123'5506 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.576 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:11.570+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5312 (114'2304,123'5312] local-lis/les=21/22 n=1077 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5310 crt=123'5310 lcod 123'5308 mlcod 123'5308 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.577 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:11.570+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5228 (114'2205,123'5228] local-lis/les=21/22 n=1167 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5227 crt=123'5223 lcod 123'5225 mlcod 123'5225 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.581 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:11.578+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5230 (114'2205,123'5230] local-lis/les=21/22 n=1165 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5227 crt=123'5227 lcod 123'5225 mlcod 123'5225 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.582 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:11.574+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5211 (114'2203,123'5211] local-lis/les=21/22 n=1008 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5209 crt=123'5209 lcod 123'5207 mlcod 123'5207 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.585 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:11.574+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5310 (115'2302,123'5310] local-lis/les=21/22 n=1155 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5310 lcod 123'5308 mlcod 123'5308 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.586 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:11.582+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5312 (115'2302,123'5312] local-lis/les=21/22 n=1153 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5310 crt=123'5310 lcod 123'5308 mlcod 123'5308 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.587 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:11.581+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5380 (114'2305,123'5380] local-lis/les=21/22 n=1137 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=123'5380 lcod 123'5378 mlcod 123'5378 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.590 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:11.582+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5532 (115'2508,123'5532] local-lis/les=21/22 n=1157 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'5532 lcod 123'5530 mlcod 123'5530 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.590 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:11.586+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5534 (115'2508,123'5534] local-lis/les=21/22 n=1155 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'5532 crt=123'5532 lcod 123'5530 mlcod 123'5530 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.591 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:11.581+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5178 (113'2106,123'5178] local-lis/les=21/22 n=1091 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5178 lcod 123'5176 mlcod 123'5176 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.591 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:11.586+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5314 (114'2304,123'5314] local-lis/les=21/22 n=1075 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5314 lcod 123'5312 mlcod 123'5312 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.594 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:11.590+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5316 (114'2304,123'5316] local-lis/les=21/22 n=1073 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5314 crt=123'5314 lcod 123'5312 mlcod 123'5312 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.595 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:11.590+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5213 (114'2203,123'5213] local-lis/les=21/22 n=1006 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5213 lcod 123'5211 mlcod 123'5211 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.596 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:11.590+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5215 (114'2203,123'5215] local-lis/les=21/22 n=1004 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5213 crt=123'5213 lcod 123'5211 mlcod 123'5211 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.596 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:11.589+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5382 (114'2305,123'5382] local-lis/les=21/22 n=1135 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5380 crt=123'5380 lcod 123'5378 mlcod 123'5378 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.597 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:11.589+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5180 (113'2106,123'5180] local-lis/les=21/22 n=1089 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5178 crt=123'5178 lcod 123'5176 mlcod 123'5176 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.600 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:11.598+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5232 (114'2205,123'5232] local-lis/les=21/22 n=1163 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5232 lcod 123'5230 mlcod 123'5230 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.602 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:11.598+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5234 (114'2205,123'5234] local-lis/les=21/22 n=1161 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5232 crt=123'5232 lcod 123'5230 mlcod 123'5230 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.607 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:11.602+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5217 (114'2203,123'5217] local-lis/les=21/22 n=1002 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5217 lcod 123'5215 mlcod 123'5215 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.608 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:11.601+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5384 (114'2305,123'5384] local-lis/les=21/22 n=1133 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=123'5384 lcod 123'5382 mlcod 123'5382 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.609 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:11.602+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5314 (115'2302,123'5314] local-lis/les=21/22 n=1151 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5314 lcod 123'5312 mlcod 123'5312 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.609 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:11.606+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5536 (115'2508,123'5536] local-lis/les=21/22 n=1153 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'5536 lcod 123'5534 mlcod 123'5534 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.610 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:11.606+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5316 (115'2302,123'5316] local-lis/les=21/22 n=1149 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5314 crt=123'5314 lcod 123'5312 mlcod 123'5312 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.612 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:11.606+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5219 (114'2203,123'5219] local-lis/les=21/22 n=1000 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5217 crt=123'5217 lcod 123'5215 mlcod 123'5215 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.612 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:11.610+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5538 (115'2508,123'5538] local-lis/les=21/22 n=1151 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'5536 crt=123'5536 lcod 123'5534 mlcod 123'5534 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.614 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:11.605+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5386 (114'2305,123'5386] local-lis/les=21/22 n=1131 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5384 crt=123'5384 lcod 123'5382 mlcod 123'5382 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.617 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:11.609+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5183 (113'2106,123'5183] local-lis/les=21/22 n=1086 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5182 crt=123'5178 lcod 123'5180 mlcod 123'5180 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.621 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:11.613+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5185 (113'2106,123'5185] local-lis/les=21/22 n=1084 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5182 crt=123'5182 lcod 123'5180 mlcod 123'5180 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.622 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:11.618+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5318 (115'2302,123'5318] local-lis/les=21/22 n=1147 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5318 lcod 123'5316 mlcod 123'5316 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.628 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:11.614+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5236 (114'2205,123'5236] local-lis/les=21/22 n=1159 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5236 lcod 123'5234 mlcod 123'5234 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.629 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:11.618+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5238 (114'2205,123'5238] local-lis/les=21/22 n=1157 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5236 crt=123'5236 lcod 123'5234 mlcod 123'5234 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.631 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:11.622+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5318 (114'2304,123'5318] local-lis/les=21/22 n=1071 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5318 lcod 123'5316 mlcod 123'5316 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.631 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:11.626+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5320 (114'2304,123'5320] local-lis/les=21/22 n=1069 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5318 crt=123'5318 lcod 123'5316 mlcod 123'5316 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.631 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:11.618+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5221 (114'2203,123'5221] local-lis/les=21/22 n=998 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5221 lcod 123'5219 mlcod 123'5219 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.631 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:11.622+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5320 (115'2302,123'5320] local-lis/les=21/22 n=1145 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5318 crt=123'5318 lcod 123'5316 mlcod 123'5316 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.632 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:11.622+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5223 (114'2203,123'5223] local-lis/les=21/22 n=996 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5221 crt=123'5221 lcod 123'5219 mlcod 123'5219 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.632 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:11.626+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5540 (115'2508,123'5540] local-lis/les=21/22 n=1149 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'5540 lcod 123'5538 mlcod 123'5538 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.634 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:11.630+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5542 (115'2508,123'5542] local-lis/les=21/22 n=1147 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'5540 crt=123'5540 lcod 123'5538 mlcod 123'5538 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.635 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:11.625+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5388 (114'2305,123'5388] local-lis/les=21/22 n=1129 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=123'5388 lcod 123'5386 mlcod 123'5386 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.636 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:11.629+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5390 (114'2305,123'5390] local-lis/les=21/22 n=1127 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5388 crt=123'5388 lcod 123'5386 mlcod 123'5386 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.639 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:11.633+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5188 (113'2106,123'5188] local-lis/les=21/22 n=1081 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5187 crt=123'5183 lcod 123'5185 mlcod 123'5185 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.640 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:11.638+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5322 (115'2302,123'5322] local-lis/les=21/22 n=1143 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5322 lcod 123'5320 mlcod 123'5320 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.641 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:11.634+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5225 (114'2203,123'5225] local-lis/les=21/22 n=994 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5225 lcod 123'5223 mlcod 123'5223 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.642 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:11.634+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5240 (114'2205,123'5240] local-lis/les=21/22 n=1155 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5240 lcod 123'5238 mlcod 123'5238 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.642 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:11.634+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5242 (114'2205,123'5242] local-lis/les=21/22 n=1153 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5240 crt=123'5240 lcod 123'5238 mlcod 123'5238 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.643 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:11.638+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5324 (115'2302,123'5324] local-lis/les=21/22 n=1141 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5322 crt=123'5322 lcod 123'5320 mlcod 123'5320 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.645 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:11.637+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5190 (113'2106,123'5190] local-lis/les=21/22 n=1079 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5187 crt=123'5187 lcod 123'5185 mlcod 123'5185 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.649 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:11.642+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5227 (114'2203,123'5227] local-lis/les=21/22 n=992 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5225 crt=123'5225 lcod 123'5223 mlcod 123'5223 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.649 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:11.646+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5322 (114'2304,123'5322] local-lis/les=21/22 n=1067 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5322 lcod 123'5320 mlcod 123'5320 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.650 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:11.646+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5544 (115'2508,123'5544] local-lis/les=21/22 n=1145 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'5544 lcod 123'5542 mlcod 123'5542 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.652 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:11.646+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5324 (114'2304,123'5324] local-lis/les=21/22 n=1065 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5322 crt=123'5322 lcod 123'5320 mlcod 123'5320 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.655 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:11.649+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5392 (114'2305,123'5392] local-lis/les=21/22 n=1125 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=123'5392 lcod 123'5390 mlcod 123'5390 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.655 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:11.650+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5546 (115'2508,123'5546] local-lis/les=21/22 n=1143 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'5544 crt=123'5544 lcod 123'5542 mlcod 123'5542 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.658 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:11.654+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5327 (115'2302,123'5327] local-lis/les=21/22 n=1138 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5326 crt=123'5322 lcod 123'5324 mlcod 123'5324 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.658 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:11.654+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5244 (114'2205,123'5244] local-lis/les=21/22 n=1151 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5244 lcod 123'5242 mlcod 123'5242 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.660 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:11.658+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5246 (114'2205,123'5246] local-lis/les=21/22 n=1149 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5244 crt=123'5244 lcod 123'5242 mlcod 123'5242 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.661 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:11.658+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5329 (115'2302,123'5329] local-lis/les=21/22 n=1136 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5326 crt=123'5326 lcod 123'5324 mlcod 123'5324 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.664 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:11.657+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5394 (114'2305,123'5394] local-lis/les=21/22 n=1123 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5392 crt=123'5392 lcod 123'5390 mlcod 123'5390 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.664 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:11.658+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5548 (115'2508,123'5548] local-lis/les=21/22 n=1141 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'5548 lcod 123'5546 mlcod 123'5546 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.666 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:11.662+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5550 (115'2508,123'5550] local-lis/les=21/22 n=1139 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'5548 crt=123'5548 lcod 123'5546 mlcod 123'5546 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.669 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:11.662+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5229 (114'2203,123'5229] local-lis/les=21/22 n=990 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5229 lcod 123'5227 mlcod 123'5227 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.669 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:11.666+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5326 (114'2304,123'5326] local-lis/les=21/22 n=1063 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5326 lcod 123'5324 mlcod 123'5324 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.669 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:11.666+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5231 (114'2203,123'5231] local-lis/les=21/22 n=988 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5229 crt=123'5229 lcod 123'5227 mlcod 123'5227 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.672 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:11.670+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5328 (114'2304,123'5328] local-lis/les=21/22 n=1061 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5326 crt=123'5326 lcod 123'5324 mlcod 123'5324 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.675 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:11.669+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5396 (114'2305,123'5396] local-lis/les=21/22 n=1121 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=123'5396 lcod 123'5394 mlcod 123'5394 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.677 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:11.669+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5398 (114'2305,123'5398] local-lis/les=21/22 n=1119 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5396 crt=123'5396 lcod 123'5394 mlcod 123'5394 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.679 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:11.674+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5248 (114'2205,123'5248] local-lis/les=21/22 n=1147 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5248 lcod 123'5246 mlcod 123'5246 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.681 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:11.678+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5250 (114'2205,123'5250] local-lis/les=21/22 n=1145 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5248 crt=123'5248 lcod 123'5246 mlcod 123'5246 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.682 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:11.677+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5401 (114'2305,123'5401] local-lis/les=21/22 n=1116 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5400 crt=123'5396 lcod 123'5398 mlcod 123'5398 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.685 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:11.682+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5331 (115'2302,123'5331] local-lis/les=21/22 n=1134 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5331 lcod 123'5329 mlcod 123'5329 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.686 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:11.682+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5330 (114'2304,123'5330] local-lis/les=21/22 n=1059 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5330 lcod 123'5328 mlcod 123'5328 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.686 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:11.677+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5403 (114'2305,123'5403] local-lis/les=21/22 n=1114 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5400 crt=123'5400 lcod 123'5398 mlcod 123'5398 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.687 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:11.682+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5332 (114'2304,123'5332] local-lis/les=21/22 n=1057 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5330 crt=123'5330 lcod 123'5328 mlcod 123'5328 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.687 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:11.682+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5333 (115'2302,123'5333] local-lis/les=21/22 n=1132 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5331 crt=123'5331 lcod 123'5329 mlcod 123'5329 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.692 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:11.690+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5552 (115'2508,123'5552] local-lis/les=21/22 n=1137 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'5552 lcod 123'5550 mlcod 123'5550 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.700 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:11.690+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5233 (114'2203,123'5233] local-lis/les=21/22 n=986 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5233 lcod 123'5231 mlcod 123'5231 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.700 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:11.694+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5235 (114'2203,123'5235] local-lis/les=21/22 n=984 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5233 crt=123'5233 lcod 123'5231 mlcod 123'5231 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.701 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:11.694+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5554 (115'2508,123'5554] local-lis/les=21/22 n=1135 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'5552 crt=123'5552 lcod 123'5550 mlcod 123'5550 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.701 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:11.694+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5336 (115'2302,123'5336] local-lis/les=21/22 n=1129 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5335 crt=123'5331 lcod 123'5333 mlcod 123'5333 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.701 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:11.698+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5338 (115'2302,123'5338] local-lis/les=21/22 n=1127 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5335 crt=123'5335 lcod 123'5333 mlcod 123'5333 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.703 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:11.698+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5556 (115'2508,123'5556] local-lis/les=21/22 n=1133 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'5556 lcod 123'5554 mlcod 123'5554 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.706 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:11.702+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5558 (115'2508,123'5558] local-lis/les=21/22 n=1131 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'5556 crt=123'5556 lcod 123'5554 mlcod 123'5554 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.708 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:11.702+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5237 (114'2203,123'5237] local-lis/les=21/22 n=982 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5237 lcod 123'5235 mlcod 123'5235 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.708 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:11.706+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5340 (115'2302,123'5340] local-lis/les=21/22 n=1125 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5340 lcod 123'5338 mlcod 123'5338 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.709 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:11.706+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5239 (114'2203,123'5239] local-lis/les=21/22 n=980 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5237 crt=123'5237 lcod 123'5235 mlcod 123'5235 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.711 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:11.705+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5406 (115'2405,123'5406] local-lis/les=21/22 n=1111 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5405 crt=123'5403 lcod 123'5403 mlcod 123'5403 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.712 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:11.710+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5342 (115'2302,123'5342] local-lis/les=21/22 n=1123 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5340 crt=123'5340 lcod 123'5338 mlcod 123'5338 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.714 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:11.709+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5408 (115'2405,123'5408] local-lis/les=21/22 n=1109 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5405 crt=123'5405 lcod 123'5403 mlcod 123'5403 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.715 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:11.710+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5560 (115'2508,123'5560] local-lis/les=21/22 n=1129 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'5560 lcod 123'5558 mlcod 123'5558 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.720 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:11.714+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5562 (115'2508,123'5562] local-lis/les=21/22 n=1127 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'5560 crt=123'5560 lcod 123'5558 mlcod 123'5558 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.721 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:11.714+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5344 (115'2302,123'5344] local-lis/les=21/22 n=1121 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5344 lcod 123'5342 mlcod 123'5342 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.722 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:11.718+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5241 (114'2203,123'5241] local-lis/les=21/22 n=978 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5241 lcod 123'5239 mlcod 123'5239 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.723 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:11.718+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5346 (115'2302,123'5346] local-lis/les=21/22 n=1119 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5344 crt=123'5344 lcod 123'5342 mlcod 123'5342 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.725 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:11.722+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5243 (114'2203,123'5243] local-lis/les=21/22 n=976 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5241 crt=123'5241 lcod 123'5239 mlcod 123'5239 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.729 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:11.726+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5565 (115'2508,123'5565] local-lis/les=21/22 n=1124 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'5564 crt=123'5560 lcod 123'5562 mlcod 123'5562 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.730 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:11.726+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5245 (114'2203,123'5245] local-lis/les=21/22 n=974 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5245 lcod 123'5243 mlcod 123'5243 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.731 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:11.726+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5567 (115'2508,123'5567] local-lis/les=21/22 n=1122 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'5564 crt=123'5564 lcod 123'5562 mlcod 123'5562 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.733 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:11.730+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5348 (115'2302,123'5348] local-lis/les=21/22 n=1117 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5348 lcod 123'5346 mlcod 123'5346 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.736 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:11.734+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5247 (114'2203,123'5247] local-lis/les=21/22 n=972 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5245 crt=123'5245 lcod 123'5243 mlcod 123'5243 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.737 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:11.734+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5350 (115'2302,123'5350] local-lis/les=21/22 n=1115 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5348 crt=123'5348 lcod 123'5346 mlcod 123'5346 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.740 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:11.738+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5569 (115'2508,123'5569] local-lis/les=21/22 n=1120 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'5569 lcod 123'5567 mlcod 123'5567 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.745 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:11.738+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5249 (114'2203,123'5249] local-lis/les=21/22 n=970 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5249 lcod 123'5247 mlcod 123'5247 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.745 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:11.742+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5571 (115'2508,123'5571] local-lis/les=21/22 n=1118 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'5569 crt=123'5569 lcod 123'5567 mlcod 123'5567 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.746 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:11.742+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5251 (114'2203,123'5251] local-lis/les=21/22 n=968 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5249 crt=123'5249 lcod 123'5247 mlcod 123'5247 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.746 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:11.742+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5353 (115'2302,123'5353] local-lis/les=21/22 n=1112 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5352 crt=123'5348 lcod 123'5350 mlcod 123'5350 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.748 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:11.746+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5355 (115'2302,123'5355] local-lis/les=21/22 n=1110 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5352 crt=123'5352 lcod 123'5350 mlcod 123'5350 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.753 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:11.750+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5573 (115'2508,123'5573] local-lis/les=21/22 n=1116 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'5573 lcod 123'5571 mlcod 123'5571 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.756 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:11.750+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5253 (114'2203,123'5253] local-lis/les=21/22 n=966 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5253 lcod 123'5251 mlcod 123'5251 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.756 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:11.750+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5334 (114'2304,123'5334] local-lis/les=21/22 n=1055 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5334 lcod 123'5332 mlcod 123'5332 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.758 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:11.754+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5336 (114'2304,123'5336] local-lis/les=21/22 n=1053 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5334 crt=123'5334 lcod 123'5332 mlcod 123'5332 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.758 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:11.754+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5255 (114'2203,123'5255] local-lis/les=21/22 n=964 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5253 crt=123'5253 lcod 123'5251 mlcod 123'5251 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.759 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:11.754+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5252 (114'2205,123'5252] local-lis/les=21/22 n=1143 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5252 lcod 123'5250 mlcod 123'5250 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.760 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:11.754+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5575 (115'2508,123'5575] local-lis/les=21/22 n=1114 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'5573 crt=123'5573 lcod 123'5571 mlcod 123'5571 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.760 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:11.758+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5514 (115'2511,123'5514] local-lis/les=21/22 n=1146 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'5513 crt=123'5511 lcod 123'5511 mlcod 123'5511 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.761 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:11.758+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5357 (115'2302,123'5357] local-lis/les=21/22 n=1108 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5357 lcod 123'5355 mlcod 123'5355 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.762 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:11.758+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5254 (114'2205,123'5254] local-lis/les=21/22 n=1141 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5252 crt=123'5252 lcod 123'5250 mlcod 123'5250 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.762 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:11.758+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5516 (115'2511,123'5516] local-lis/les=21/22 n=1144 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'5513 crt=123'5513 lcod 123'5511 mlcod 123'5511 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.764 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:11.758+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5359 (115'2302,123'5359] local-lis/les=21/22 n=1106 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5357 crt=123'5357 lcod 123'5355 mlcod 123'5355 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.771 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:11.766+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5338 (114'2304,123'5338] local-lis/les=21/22 n=1051 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5338 lcod 123'5336 mlcod 123'5336 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.772 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:11.766+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5577 (115'2508,123'5577] local-lis/les=21/22 n=1112 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'5577 lcod 123'5575 mlcod 123'5575 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.799 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:11.789+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5192 (113'2106,123'5192] local-lis/les=21/22 n=1077 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5192 lcod 123'5190 mlcod 123'5190 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.800 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:11.770+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5340 (114'2304,123'5340] local-lis/les=21/22 n=1049 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5338 crt=123'5338 lcod 123'5336 mlcod 123'5336 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.800 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:11.774+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5256 (114'2205,123'5256] local-lis/les=21/22 n=1139 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5256 lcod 123'5254 mlcod 123'5254 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.800 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:11.774+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5258 (114'2205,123'5258] local-lis/les=21/22 n=1137 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5256 crt=123'5256 lcod 123'5254 mlcod 123'5254 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.800 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:11.778+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5342 (114'2304,123'5342] local-lis/les=21/22 n=1047 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5342 lcod 123'5340 mlcod 123'5340 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.800 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:11.782+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5344 (114'2304,123'5344] local-lis/les=21/22 n=1045 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5342 crt=123'5342 lcod 123'5340 mlcod 123'5340 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.800 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:11.786+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5260 (114'2205,123'5260] local-lis/les=21/22 n=1135 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5260 lcod 123'5258 mlcod 123'5258 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.800 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:11.790+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5262 (114'2205,123'5262] local-lis/les=21/22 n=1133 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5260 crt=123'5260 lcod 123'5258 mlcod 123'5258 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.801 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:11.790+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5346 (114'2304,123'5346] local-lis/les=21/22 n=1043 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5346 lcod 123'5344 mlcod 123'5344 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.801 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:11.794+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5348 (114'2304,123'5348] local-lis/les=21/22 n=1041 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5346 crt=123'5346 lcod 123'5344 mlcod 123'5344 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.801 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:11.770+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5257 (114'2203,123'5257] local-lis/les=21/22 n=962 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5257 lcod 123'5255 mlcod 123'5255 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.801 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:11.774+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5259 (114'2203,123'5259] local-lis/les=21/22 n=960 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5257 crt=123'5257 lcod 123'5255 mlcod 123'5255 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.801 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:11.786+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5261 (114'2203,123'5261] local-lis/les=21/22 n=958 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5261 lcod 123'5259 mlcod 123'5259 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.801 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:11.790+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5263 (114'2203,123'5263] local-lis/les=21/22 n=956 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5261 crt=123'5261 lcod 123'5259 mlcod 123'5259 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.802 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:11.774+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5579 (115'2508,123'5579] local-lis/les=21/22 n=1110 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'5577 crt=123'5577 lcod 123'5575 mlcod 123'5575 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.802 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:11.782+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5361 (115'2302,123'5361] local-lis/les=21/22 n=1104 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5361 lcod 123'5359 mlcod 123'5359 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.802 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:11.786+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5363 (115'2302,123'5363] local-lis/les=21/22 n=1102 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5361 crt=123'5361 lcod 123'5359 mlcod 123'5359 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.802 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:11.786+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5581 (115'2508,123'5581] local-lis/les=21/22 n=1108 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'5581 lcod 123'5579 mlcod 123'5579 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.804 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:11.789+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5194 (113'2106,123'5194] local-lis/les=21/22 n=1075 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5192 crt=123'5192 lcod 123'5190 mlcod 123'5190 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.804 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:11.794+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5583 (115'2508,123'5583] local-lis/les=21/22 n=1106 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'5581 crt=123'5581 lcod 123'5579 mlcod 123'5579 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.805 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:11.802+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5264 (114'2205,123'5264] local-lis/les=21/22 n=1131 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5264 lcod 123'5262 mlcod 123'5262 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.806 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:11.802+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5365 (115'2302,123'5365] local-lis/les=21/22 n=1100 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5365 lcod 123'5363 mlcod 123'5363 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.810 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:11.801+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5196 (113'2106,123'5196] local-lis/les=21/22 n=1073 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5196 lcod 123'5194 mlcod 123'5194 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.811 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:11.806+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5350 (114'2304,123'5350] local-lis/les=21/22 n=1039 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5350 lcod 123'5348 mlcod 123'5348 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.812 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:11.801+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5198 (113'2106,123'5198] local-lis/les=21/22 n=1071 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5196 crt=123'5196 lcod 123'5194 mlcod 123'5194 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.813 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:11.810+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5352 (114'2304,123'5352] local-lis/les=21/22 n=1037 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5350 crt=123'5350 lcod 123'5348 mlcod 123'5348 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.813 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:11.810+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5266 (114'2205,123'5266] local-lis/les=21/22 n=1129 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5264 crt=123'5264 lcod 123'5262 mlcod 123'5262 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.813 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:11.802+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5265 (114'2203,123'5265] local-lis/les=21/22 n=954 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5265 lcod 123'5263 mlcod 123'5263 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.813 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:11.810+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5585 (115'2508,123'5585] local-lis/les=21/22 n=1104 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'5585 lcod 123'5583 mlcod 123'5583 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.815 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:11.806+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5267 (114'2203,123'5267] local-lis/les=21/22 n=952 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5265 crt=123'5265 lcod 123'5263 mlcod 123'5263 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.815 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:11.810+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5367 (115'2302,123'5367] local-lis/les=21/22 n=1098 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5365 crt=123'5365 lcod 123'5363 mlcod 123'5363 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.818 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:11.814+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5587 (115'2508,123'5587] local-lis/les=21/22 n=1102 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'5585 crt=123'5585 lcod 123'5583 mlcod 123'5583 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.822 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:11.813+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5200 (113'2106,123'5200] local-lis/les=21/22 n=1069 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5200 lcod 123'5198 mlcod 123'5198 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.823 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:11.818+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5268 (114'2205,123'5268] local-lis/les=21/22 n=1127 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5268 lcod 123'5266 mlcod 123'5266 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.823 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:11.818+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5269 (114'2203,123'5269] local-lis/les=21/22 n=950 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5269 lcod 123'5267 mlcod 123'5267 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.824 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:11.817+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5202 (113'2106,123'5202] local-lis/les=21/22 n=1067 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5200 crt=123'5200 lcod 123'5198 mlcod 123'5198 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.826 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:11.822+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5271 (114'2203,123'5271] local-lis/les=21/22 n=948 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5269 crt=123'5269 lcod 123'5267 mlcod 123'5267 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.828 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:11.822+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5369 (115'2302,123'5369] local-lis/les=21/22 n=1096 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5369 lcod 123'5367 mlcod 123'5367 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.829 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:11.826+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5270 (114'2205,123'5270] local-lis/les=21/22 n=1125 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5268 crt=123'5268 lcod 123'5266 mlcod 123'5266 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.829 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:11.826+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5354 (114'2304,123'5354] local-lis/les=21/22 n=1035 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5354 lcod 123'5352 mlcod 123'5352 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.831 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:11.826+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5371 (115'2302,123'5371] local-lis/les=21/22 n=1094 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5369 crt=123'5369 lcod 123'5367 mlcod 123'5367 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.833 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:11.830+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5356 (114'2304,123'5356] local-lis/les=21/22 n=1033 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5354 crt=123'5354 lcod 123'5352 mlcod 123'5352 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.837 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:11.834+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5273 (114'2203,123'5273] local-lis/les=21/22 n=946 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5273 lcod 123'5271 mlcod 123'5271 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.837 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:11.830+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5589 (115'2508,123'5589] local-lis/les=21/22 n=1100 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'5589 lcod 123'5587 mlcod 123'5587 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.837 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:11.834+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5591 (115'2508,123'5591] local-lis/les=21/22 n=1098 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'5589 crt=123'5589 lcod 123'5587 mlcod 123'5587 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.839 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:11.834+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5275 (114'2203,123'5275] local-lis/les=21/22 n=944 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5273 crt=123'5273 lcod 123'5271 mlcod 123'5271 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.840 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:11.838+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5272 (114'2205,123'5272] local-lis/les=21/22 n=1123 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5272 lcod 123'5270 mlcod 123'5270 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.842 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:11.838+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5274 (114'2205,123'5274] local-lis/les=21/22 n=1121 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5272 crt=123'5272 lcod 123'5270 mlcod 123'5270 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.845 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:11.842+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5373 (115'2302,123'5373] local-lis/les=21/22 n=1092 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5373 lcod 123'5371 mlcod 123'5371 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.847 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:11.841+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5204 (113'2106,123'5204] local-lis/les=21/22 n=1065 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5204 lcod 123'5202 mlcod 123'5202 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.848 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:11.842+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5375 (115'2302,123'5375] local-lis/les=21/22 n=1090 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5373 crt=123'5373 lcod 123'5371 mlcod 123'5371 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.848 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:11.842+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5358 (114'2304,123'5358] local-lis/les=21/22 n=1031 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5358 lcod 123'5356 mlcod 123'5356 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.849 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:11.846+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5360 (114'2304,123'5360] local-lis/les=21/22 n=1029 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5358 crt=123'5358 lcod 123'5356 mlcod 123'5356 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.850 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:11.846+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5593 (115'2508,123'5593] local-lis/les=21/22 n=1096 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'5593 lcod 123'5591 mlcod 123'5591 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.851 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:11.846+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5595 (115'2508,123'5595] local-lis/les=21/22 n=1094 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'5593 crt=123'5593 lcod 123'5591 mlcod 123'5591 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.853 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:11.845+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5206 (113'2106,123'5206] local-lis/les=21/22 n=1063 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5204 crt=123'5204 lcod 123'5202 mlcod 123'5202 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.855 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:11.850+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5276 (114'2205,123'5276] local-lis/les=21/22 n=1119 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5276 lcod 123'5274 mlcod 123'5274 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.855 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:11.850+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5278 (114'2205,123'5278] local-lis/les=21/22 n=1117 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5276 crt=123'5276 lcod 123'5274 mlcod 123'5274 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.861 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:11.858+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5377 (115'2302,123'5377] local-lis/les=21/22 n=1088 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5377 lcod 123'5375 mlcod 123'5375 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.862 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:11.858+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5362 (114'2304,123'5362] local-lis/les=21/22 n=1027 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5362 lcod 123'5360 mlcod 123'5360 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.864 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:11.862+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5379 (115'2302,123'5379] local-lis/les=21/22 n=1086 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5377 crt=123'5377 lcod 123'5375 mlcod 123'5375 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.866 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:11.857+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5208 (114'2206,123'5208] local-lis/les=21/22 n=1061 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5208 lcod 123'5206 mlcod 123'5206 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.867 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:11.862+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5364 (114'2304,123'5364] local-lis/les=21/22 n=1025 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5362 crt=123'5362 lcod 123'5360 mlcod 123'5360 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.867 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:11.861+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5210 (114'2206,123'5210] local-lis/les=21/22 n=1059 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5208 crt=123'5208 lcod 123'5206 mlcod 123'5206 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.868 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:11.862+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5597 (115'2508,123'5597] local-lis/les=21/22 n=1092 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'5597 lcod 123'5595 mlcod 123'5595 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.868 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:11.862+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5280 (114'2205,123'5280] local-lis/les=21/22 n=1115 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5280 lcod 123'5278 mlcod 123'5278 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.870 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:11.866+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5599 (115'2508,123'5599] local-lis/les=21/22 n=1090 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'5597 crt=123'5597 lcod 123'5595 mlcod 123'5595 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.874 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:11.870+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5381 (115'2302,123'5381] local-lis/les=21/22 n=1084 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5381 lcod 123'5379 mlcod 123'5379 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.875 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:11.866+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5282 (114'2205,123'5282] local-lis/les=21/22 n=1113 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5280 crt=123'5280 lcod 123'5278 mlcod 123'5278 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.877 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:11.874+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5383 (115'2302,123'5383] local-lis/les=21/22 n=1082 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5381 crt=123'5381 lcod 123'5379 mlcod 123'5379 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.877 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:11.869+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5212 (114'2206,123'5212] local-lis/les=21/22 n=1057 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5212 lcod 123'5210 mlcod 123'5210 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.878 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:11.874+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5601 (115'2508,123'5601] local-lis/les=21/22 n=1088 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'5601 lcod 123'5599 mlcod 123'5599 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.879 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:11.874+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5366 (114'2304,123'5366] local-lis/les=21/22 n=1023 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5366 lcod 123'5364 mlcod 123'5364 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.880 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:11.873+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5214 (114'2206,123'5214] local-lis/les=21/22 n=1055 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5212 crt=123'5212 lcod 123'5210 mlcod 123'5210 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.882 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:11.878+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5368 (114'2304,123'5368] local-lis/les=21/22 n=1021 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5366 crt=123'5366 lcod 123'5364 mlcod 123'5364 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.882 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:11.878+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5603 (115'2508,123'5603] local-lis/les=21/22 n=1086 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'5601 crt=123'5601 lcod 123'5599 mlcod 123'5599 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.884 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:11.882+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5284 (114'2205,123'5284] local-lis/les=21/22 n=1111 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5284 lcod 123'5282 mlcod 123'5282 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.887 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:11.882+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5286 (114'2205,123'5286] local-lis/les=21/22 n=1109 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5284 crt=123'5284 lcod 123'5282 mlcod 123'5282 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.891 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:11.886+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5385 (115'2302,123'5385] local-lis/les=21/22 n=1080 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5385 lcod 123'5383 mlcod 123'5383 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.891 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:11.886+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5605 (115'2508,123'5605] local-lis/les=21/22 n=1084 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'5605 lcod 123'5603 mlcod 123'5603 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.892 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:11.886+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5370 (114'2304,123'5370] local-lis/les=21/22 n=1019 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5370 lcod 123'5368 mlcod 123'5368 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.893 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:11.885+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5216 (114'2206,123'5216] local-lis/les=21/22 n=1053 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5216 lcod 123'5214 mlcod 123'5214 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.894 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:11.890+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5372 (114'2304,123'5372] local-lis/les=21/22 n=1017 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5370 crt=123'5370 lcod 123'5368 mlcod 123'5368 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.895 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:11.890+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5387 (115'2302,123'5387] local-lis/les=21/22 n=1078 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5385 crt=123'5385 lcod 123'5383 mlcod 123'5383 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.895 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:11.890+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5607 (115'2508,123'5607] local-lis/les=21/22 n=1082 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'5605 crt=123'5605 lcod 123'5603 mlcod 123'5603 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.896 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:11.889+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5218 (114'2206,123'5218] local-lis/les=21/22 n=1051 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5216 crt=123'5216 lcod 123'5214 mlcod 123'5214 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.899 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:11.894+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5288 (114'2205,123'5288] local-lis/les=21/22 n=1107 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5288 lcod 123'5286 mlcod 123'5286 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.912 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:11.898+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5290 (114'2205,123'5290] local-lis/les=21/22 n=1105 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5288 crt=123'5288 lcod 123'5286 mlcod 123'5286 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.914 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:11.901+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5220 (114'2206,123'5220] local-lis/les=21/22 n=1049 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5220 lcod 123'5218 mlcod 123'5218 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.914 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:11.901+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5222 (114'2206,123'5222] local-lis/les=21/22 n=1047 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5220 crt=123'5220 lcod 123'5218 mlcod 123'5218 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.914 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:11.902+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5609 (115'2508,123'5609] local-lis/les=21/22 n=1080 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'5609 lcod 123'5607 mlcod 123'5607 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.915 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:11.906+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5389 (115'2302,123'5389] local-lis/les=21/22 n=1076 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5389 lcod 123'5387 mlcod 123'5387 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.915 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:11.906+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5611 (115'2609,123'5611] local-lis/les=21/22 n=1078 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'5609 crt=123'5609 lcod 123'5607 mlcod 123'5607 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.915 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:11.906+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5391 (115'2302,123'5391] local-lis/les=21/22 n=1074 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5389 crt=123'5389 lcod 123'5387 mlcod 123'5387 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.915 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:11.902+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5374 (114'2304,123'5374] local-lis/les=21/22 n=1015 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5374 lcod 123'5372 mlcod 123'5372 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.915 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:11.906+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5376 (114'2304,123'5376] local-lis/les=21/22 n=1013 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5374 crt=123'5374 lcod 123'5372 mlcod 123'5372 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.918 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:11.914+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5613 (115'2609,123'5613] local-lis/les=21/22 n=1076 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'5613 lcod 123'5611 mlcod 123'5611 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.919 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:11.914+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5292 (114'2205,123'5292] local-lis/les=21/22 n=1103 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5292 lcod 123'5290 mlcod 123'5290 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.920 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:11.918+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5393 (115'2302,123'5393] local-lis/les=21/22 n=1072 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5393 lcod 123'5391 mlcod 123'5391 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.920 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:11.918+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5615 (115'2609,123'5615] local-lis/les=21/22 n=1074 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'5613 crt=123'5613 lcod 123'5611 mlcod 123'5611 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.923 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:11.918+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5395 (115'2302,123'5395] local-lis/les=21/22 n=1070 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5393 crt=123'5393 lcod 123'5391 mlcod 123'5391 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.923 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:11.918+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5294 (114'2205,123'5294] local-lis/les=21/22 n=1101 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5292 crt=123'5292 lcod 123'5290 mlcod 123'5290 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.926 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:11.917+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5224 (114'2206,123'5224] local-lis/les=21/22 n=1045 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5224 lcod 123'5222 mlcod 123'5222 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.927 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:11.922+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5378 (114'2304,123'5378] local-lis/les=21/22 n=1011 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5378 lcod 123'5376 mlcod 123'5376 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.929 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:11.921+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5226 (114'2206,123'5226] local-lis/les=21/22 n=1043 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5224 crt=123'5224 lcod 123'5222 mlcod 123'5222 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.929 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:11.926+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5380 (114'2304,123'5380] local-lis/les=21/22 n=1009 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5378 crt=123'5378 lcod 123'5376 mlcod 123'5376 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.930 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:11.926+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5617 (115'2609,123'5617] local-lis/les=21/22 n=1072 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'5617 lcod 123'5615 mlcod 123'5615 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.933 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:11.930+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5619 (115'2609,123'5619] local-lis/les=21/22 n=1070 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'5617 crt=123'5617 lcod 123'5615 mlcod 123'5615 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.933 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:11.930+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5397 (115'2302,123'5397] local-lis/les=21/22 n=1068 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5397 lcod 123'5395 mlcod 123'5395 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.938 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:11.934+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5399 (115'2302,123'5399] local-lis/les=21/22 n=1066 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5397 crt=123'5397 lcod 123'5395 mlcod 123'5395 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.939 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:11.934+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5621 (115'2609,123'5621] local-lis/les=21/22 n=1068 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'5621 lcod 123'5619 mlcod 123'5619 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.941 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:11.930+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5296 (114'2205,123'5296] local-lis/les=21/22 n=1099 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5296 lcod 123'5294 mlcod 123'5294 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.941 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:11.934+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5298 (114'2205,123'5298] local-lis/les=21/22 n=1097 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5296 crt=123'5296 lcod 123'5294 mlcod 123'5294 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.942 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:11.938+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5623 (115'2609,123'5623] local-lis/les=21/22 n=1066 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'5621 crt=123'5621 lcod 123'5619 mlcod 123'5619 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.942 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:11.938+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5382 (114'2304,123'5382] local-lis/les=21/22 n=1007 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5382 lcod 123'5380 mlcod 123'5380 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.946 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:11.942+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5384 (114'2304,123'5384] local-lis/les=21/22 n=1005 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5382 crt=123'5382 lcod 123'5380 mlcod 123'5380 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.946 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:11.942+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5401 (115'2302,123'5401] local-lis/les=21/22 n=1064 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5401 lcod 123'5399 mlcod 123'5399 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.949 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:11.946+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5625 (115'2609,123'5625] local-lis/les=21/22 n=1064 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'5625 lcod 123'5623 mlcod 123'5623 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.949 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:11.946+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5403 (115'2302,123'5403] local-lis/les=21/22 n=1062 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5401 crt=123'5401 lcod 123'5399 mlcod 123'5399 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.950 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:11.946+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5627 (115'2609,123'5627] local-lis/les=21/22 n=1062 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'5625 crt=123'5625 lcod 123'5623 mlcod 123'5623 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.952 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:11.945+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5228 (114'2206,123'5228] local-lis/les=21/22 n=1041 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5228 lcod 123'5226 mlcod 123'5226 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.955 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:11.949+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5230 (114'2206,123'5230] local-lis/les=21/22 n=1039 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5228 crt=123'5228 lcod 123'5226 mlcod 123'5226 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.956 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:11.954+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5300 (114'2205,123'5300] local-lis/les=21/22 n=1095 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5300 lcod 123'5298 mlcod 123'5298 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.958 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:11.954+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5302 (114'2205,123'5302] local-lis/les=21/22 n=1093 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5300 crt=123'5300 lcod 123'5298 mlcod 123'5298 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.961 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:11.958+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5629 (115'2609,123'5629] local-lis/les=21/22 n=1060 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'5629 lcod 123'5627 mlcod 123'5627 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.963 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:11.958+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5405 (115'2403,123'5405] local-lis/les=21/22 n=1060 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5405 lcod 123'5403 mlcod 123'5403 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.966 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:11.962+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5407 (115'2403,123'5407] local-lis/les=21/22 n=1058 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5405 crt=123'5405 lcod 123'5403 mlcod 123'5403 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.967 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:11.962+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5631 (115'2609,123'5631] local-lis/les=21/22 n=1058 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'5629 crt=123'5629 lcod 123'5627 mlcod 123'5627 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.967 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:11.962+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5386 (114'2304,123'5386] local-lis/les=21/22 n=1003 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5386 lcod 123'5384 mlcod 123'5384 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.972 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:11.966+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5388 (114'2304,123'5388] local-lis/les=21/22 n=1001 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5386 crt=123'5386 lcod 123'5384 mlcod 123'5384 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.973 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:11.965+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5232 (114'2206,123'5232] local-lis/les=21/22 n=1037 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5232 lcod 123'5230 mlcod 123'5230 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.973 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:11.970+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5633 (115'2609,123'5633] local-lis/les=21/22 n=1056 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'5633 lcod 123'5631 mlcod 123'5631 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.976 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:11.969+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5234 (114'2206,123'5234] local-lis/les=21/22 n=1035 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5232 crt=123'5232 lcod 123'5230 mlcod 123'5230 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.977 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:11.970+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5409 (115'2403,123'5409] local-lis/les=21/22 n=1056 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5409 lcod 123'5407 mlcod 123'5407 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.978 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:11.974+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5635 (115'2609,123'5635] local-lis/les=21/22 n=1054 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'5633 crt=123'5633 lcod 123'5631 mlcod 123'5631 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.978 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:11.970+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5304 (114'2205,123'5304] local-lis/les=21/22 n=1091 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5304 lcod 123'5302 mlcod 123'5302 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.979 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:11.974+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5306 (114'2205,123'5306] local-lis/les=21/22 n=1089 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5304 crt=123'5304 lcod 123'5302 mlcod 123'5302 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.982 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:11.978+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5390 (114'2304,123'5390] local-lis/les=21/22 n=999 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5390 lcod 123'5388 mlcod 123'5388 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.983 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:11.978+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5411 (115'2403,123'5411] local-lis/les=21/22 n=1054 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5409 crt=123'5409 lcod 123'5407 mlcod 123'5407 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.984 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:11.982+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5637 (115'2609,123'5637] local-lis/les=21/22 n=1052 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'5637 lcod 123'5635 mlcod 123'5635 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.985 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:11.982+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5392 (114'2304,123'5392] local-lis/les=21/22 n=997 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5390 crt=123'5390 lcod 123'5388 mlcod 123'5388 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.986 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:11.982+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5639 (115'2609,123'5639] local-lis/les=21/22 n=1050 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'5637 crt=123'5637 lcod 123'5635 mlcod 123'5635 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.992 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:11.990+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5413 (115'2403,123'5413] local-lis/les=21/22 n=1052 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5413 lcod 123'5411 mlcod 123'5411 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.993 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:11.985+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5236 (114'2206,123'5236] local-lis/les=21/22 n=1033 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5236 lcod 123'5234 mlcod 123'5234 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.995 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:11.990+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5308 (115'2306,123'5308] local-lis/les=21/22 n=1087 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5308 lcod 123'5306 mlcod 123'5306 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.995 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:11.990+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5310 (115'2306,123'5310] local-lis/les=21/22 n=1085 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5308 crt=123'5308 lcod 123'5306 mlcod 123'5306 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.996 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:11.990+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5641 (115'2609,123'5641] local-lis/les=21/22 n=1048 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'5641 lcod 123'5639 mlcod 123'5639 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.996 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:11.990+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5415 (115'2403,123'5415] local-lis/les=21/22 n=1050 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5413 crt=123'5413 lcod 123'5411 mlcod 123'5411 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.997 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:11.989+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5238 (114'2206,123'5238] local-lis/les=21/22 n=1031 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5236 crt=123'5236 lcod 123'5234 mlcod 123'5234 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:11.998 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:11.994+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5394 (114'2304,123'5394] local-lis/les=21/22 n=995 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5394 lcod 123'5392 mlcod 123'5392 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.000 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:11.994+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5643 (115'2609,123'5643] local-lis/les=21/22 n=1046 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'5641 crt=123'5641 lcod 123'5639 mlcod 123'5639 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.003 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:11.998+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5396 (114'2304,123'5396] local-lis/les=21/22 n=993 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5394 crt=123'5394 lcod 123'5392 mlcod 123'5392 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.006 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:12.002+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5312 (115'2306,123'5312] local-lis/les=21/22 n=1083 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5312 lcod 123'5310 mlcod 123'5310 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.007 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:12.001+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5240 (114'2206,123'5240] local-lis/les=21/22 n=1029 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5240 lcod 123'5238 mlcod 123'5238 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.009 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.002+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5417 (115'2403,123'5417] local-lis/les=21/22 n=1048 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5417 lcod 123'5415 mlcod 123'5415 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.009 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:12.006+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5314 (115'2306,123'5314] local-lis/les=21/22 n=1081 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5312 crt=123'5312 lcod 123'5310 mlcod 123'5310 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.010 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:12.001+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5242 (114'2206,123'5242] local-lis/les=21/22 n=1027 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5240 crt=123'5240 lcod 123'5238 mlcod 123'5238 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.012 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.006+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5419 (115'2403,123'5419] local-lis/les=21/22 n=1046 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5417 crt=123'5417 lcod 123'5415 mlcod 123'5415 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.013 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.010+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5645 (115'2609,123'5645] local-lis/les=21/22 n=1044 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'5645 lcod 123'5643 mlcod 123'5643 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.015 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.010+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5647 (115'2609,123'5647] local-lis/les=21/22 n=1042 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'5645 crt=123'5645 lcod 123'5643 mlcod 123'5643 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.016 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:12.014+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5398 (114'2304,123'5398] local-lis/les=21/22 n=991 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5398 lcod 123'5396 mlcod 123'5396 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.019 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:12.014+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5400 (114'2304,123'5400] local-lis/les=21/22 n=989 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5398 crt=123'5398 lcod 123'5396 mlcod 123'5396 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.021 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:12.013+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5244 (114'2206,123'5244] local-lis/les=21/22 n=1025 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5244 lcod 123'5242 mlcod 123'5242 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.022 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.018+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5421 (115'2403,123'5421] local-lis/les=21/22 n=1044 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5421 lcod 123'5419 mlcod 123'5419 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.023 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:12.017+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5246 (114'2206,123'5246] local-lis/les=21/22 n=1023 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5244 crt=123'5244 lcod 123'5242 mlcod 123'5242 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.025 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.022+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5423 (115'2403,123'5423] local-lis/les=21/22 n=1042 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5421 crt=123'5421 lcod 123'5419 mlcod 123'5419 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.027 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.022+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5649 (115'2609,123'5649] local-lis/les=21/22 n=1040 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'5649 lcod 123'5647 mlcod 123'5647 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.029 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.026+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5651 (115'2609,123'5651] local-lis/les=21/22 n=1038 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'5649 crt=123'5649 lcod 123'5647 mlcod 123'5647 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.032 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.030+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5425 (115'2403,123'5425] local-lis/les=21/22 n=1040 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5425 lcod 123'5423 mlcod 123'5423 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.034 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.030+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5427 (115'2403,123'5427] local-lis/les=21/22 n=1038 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5425 crt=123'5425 lcod 123'5423 mlcod 123'5423 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.035 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.030+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5653 (115'2609,123'5653] local-lis/les=21/22 n=1036 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'5653 lcod 123'5651 mlcod 123'5651 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.037 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.034+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5655 (115'2609,123'5655] local-lis/les=21/22 n=1034 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'5653 crt=123'5653 lcod 123'5651 mlcod 123'5651 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.041 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.038+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5657 (115'2609,123'5657] local-lis/les=21/22 n=1032 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'5657 lcod 123'5655 mlcod 123'5655 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.042 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.038+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5429 (115'2403,123'5429] local-lis/les=21/22 n=1036 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5429 lcod 123'5427 mlcod 123'5427 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.043 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.038+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5659 (115'2609,123'5659] local-lis/les=21/22 n=1030 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'5657 crt=123'5657 lcod 123'5655 mlcod 123'5655 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.044 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.042+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5431 (115'2403,123'5431] local-lis/les=21/22 n=1034 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5429 crt=123'5429 lcod 123'5427 mlcod 123'5427 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.048 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.046+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5661 (115'2609,123'5661] local-lis/les=21/22 n=1028 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'5661 lcod 123'5659 mlcod 123'5659 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.050 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.046+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5663 (115'2609,123'5663] local-lis/les=21/22 n=1026 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'5661 crt=123'5661 lcod 123'5659 mlcod 123'5659 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.051 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.046+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5433 (115'2403,123'5433] local-lis/les=21/22 n=1032 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5433 lcod 123'5431 mlcod 123'5431 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.054 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.050+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5435 (115'2403,123'5435] local-lis/les=21/22 n=1030 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5433 crt=123'5433 lcod 123'5431 mlcod 123'5431 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.057 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.054+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5665 (115'2609,123'5665] local-lis/les=21/22 n=1024 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'5665 lcod 123'5663 mlcod 123'5663 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.059 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:12.053+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5248 (114'2206,123'5248] local-lis/les=21/22 n=1021 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5248 lcod 123'5246 mlcod 123'5246 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.060 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:12.054+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5402 (114'2304,123'5402] local-lis/les=21/22 n=987 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5402 lcod 123'5400 mlcod 123'5400 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.060 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:12.054+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5404 (114'2304,123'5404] local-lis/les=21/22 n=985 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5402 crt=123'5402 lcod 123'5400 mlcod 123'5400 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.061 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:12.053+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5250 (114'2206,123'5250] local-lis/les=21/22 n=1019 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5248 crt=123'5248 lcod 123'5246 mlcod 123'5246 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.062 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.058+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5667 (115'2609,123'5667] local-lis/les=21/22 n=1022 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'5665 crt=123'5665 lcod 123'5663 mlcod 123'5663 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.064 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.062+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5437 (115'2403,123'5437] local-lis/les=21/22 n=1028 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5437 lcod 123'5435 mlcod 123'5435 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.070 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:12.062+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5518 (115'2511,123'5518] local-lis/les=21/22 n=1142 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'5518 lcod 123'5516 mlcod 123'5516 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.071 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:12.066+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5520 (115'2511,123'5520] local-lis/les=21/22 n=1140 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'5518 crt=123'5518 lcod 123'5516 mlcod 123'5516 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.071 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.066+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5439 (115'2403,123'5439] local-lis/les=21/22 n=1026 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5437 crt=123'5437 lcod 123'5435 mlcod 123'5435 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.073 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.070+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5669 (115'2609,123'5669] local-lis/les=21/22 n=1020 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'5669 lcod 123'5667 mlcod 123'5667 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.075 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:12.070+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5277 (114'2203,123'5277] local-lis/les=21/22 n=942 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5277 lcod 123'5275 mlcod 123'5275 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.075 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:12.070+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5316 (115'2306,123'5316] local-lis/les=21/22 n=1079 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5316 lcod 123'5314 mlcod 123'5314 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.076 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:12.069+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5252 (114'2206,123'5252] local-lis/les=21/22 n=1017 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5252 lcod 123'5250 mlcod 123'5250 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.076 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:12.074+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5279 (114'2203,123'5279] local-lis/les=21/22 n=940 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5277 crt=123'5277 lcod 123'5275 mlcod 123'5275 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.079 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:12.073+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5254 (114'2206,123'5254] local-lis/les=21/22 n=1015 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5252 crt=123'5252 lcod 123'5250 mlcod 123'5250 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.080 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:12.078+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5318 (115'2306,123'5318] local-lis/les=21/22 n=1077 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5316 crt=123'5316 lcod 123'5314 mlcod 123'5314 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.082 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.078+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5441 (115'2403,123'5441] local-lis/les=21/22 n=1024 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5441 lcod 123'5439 mlcod 123'5439 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.083 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.078+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5671 (115'2609,123'5671] local-lis/les=21/22 n=1018 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'5669 crt=123'5669 lcod 123'5667 mlcod 123'5667 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.086 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.082+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5443 (115'2403,123'5443] local-lis/les=21/22 n=1022 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5441 crt=123'5441 lcod 123'5439 mlcod 123'5439 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.086 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:12.082+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5406 (115'2404,123'5406] local-lis/les=21/22 n=983 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5406 lcod 123'5404 mlcod 123'5404 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.087 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:12.082+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5522 (115'2511,123'5522] local-lis/les=21/22 n=1138 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'5522 lcod 123'5520 mlcod 123'5520 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.089 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:12.086+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5524 (115'2511,123'5524] local-lis/les=21/22 n=1136 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'5522 crt=123'5522 lcod 123'5520 mlcod 123'5520 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.091 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:12.086+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5408 (115'2404,123'5408] local-lis/les=21/22 n=981 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5406 crt=123'5406 lcod 123'5404 mlcod 123'5404 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.097 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.094+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5673 (115'2609,123'5673] local-lis/les=21/22 n=1016 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'5673 lcod 123'5671 mlcod 123'5671 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.097 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.094+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5445 (115'2403,123'5445] local-lis/les=21/22 n=1020 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5445 lcod 123'5443 mlcod 123'5443 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.098 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.094+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5675 (115'2609,123'5675] local-lis/les=21/22 n=1014 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'5673 crt=123'5673 lcod 123'5671 mlcod 123'5671 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.102 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.098+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5447 (115'2403,123'5447] local-lis/les=21/22 n=1018 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5445 crt=123'5445 lcod 123'5443 mlcod 123'5443 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.106 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.102+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5677 (115'2609,123'5677] local-lis/les=21/22 n=1012 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'5677 lcod 123'5675 mlcod 123'5675 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.108 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.106+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5449 (115'2403,123'5449] local-lis/les=21/22 n=1016 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5449 lcod 123'5447 mlcod 123'5447 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.109 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.106+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5679 (115'2609,123'5679] local-lis/les=21/22 n=1010 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'5677 crt=123'5677 lcod 123'5675 mlcod 123'5675 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.113 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.110+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5451 (115'2403,123'5451] local-lis/les=21/22 n=1014 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5449 crt=123'5449 lcod 123'5447 mlcod 123'5447 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.116 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.114+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5681 (115'2609,123'5681] local-lis/les=21/22 n=1008 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'5681 lcod 123'5679 mlcod 123'5679 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.119 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.114+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5683 (115'2609,123'5683] local-lis/les=21/22 n=1006 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'5681 crt=123'5681 lcod 123'5679 mlcod 123'5679 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.120 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.114+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5453 (115'2403,123'5453] local-lis/les=21/22 n=1012 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5453 lcod 123'5451 mlcod 123'5451 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.124 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.118+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5455 (115'2403,123'5455] local-lis/les=21/22 n=1010 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5453 crt=123'5453 lcod 123'5451 mlcod 123'5451 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.126 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.122+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5685 (115'2609,123'5685] local-lis/les=21/22 n=1004 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'5685 lcod 123'5683 mlcod 123'5683 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.128 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.126+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5687 (115'2609,123'5687] local-lis/les=21/22 n=1002 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'5685 crt=123'5685 lcod 123'5683 mlcod 123'5683 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.133 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.130+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5457 (115'2403,123'5457] local-lis/les=21/22 n=1008 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5457 lcod 123'5455 mlcod 123'5455 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.134 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.130+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5689 (115'2609,123'5689] local-lis/les=21/22 n=1000 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'5689 lcod 123'5687 mlcod 123'5687 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.135 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.130+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5459 (115'2403,123'5459] local-lis/les=21/22 n=1006 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5457 crt=123'5457 lcod 123'5455 mlcod 123'5455 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.137 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.134+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5691 (115'2609,123'5691] local-lis/les=21/22 n=998 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'5689 crt=123'5689 lcod 123'5687 mlcod 123'5687 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.142 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.138+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5693 (115'2609,123'5693] local-lis/les=21/22 n=996 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'5693 lcod 123'5691 mlcod 123'5691 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.142 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.138+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5461 (115'2403,123'5461] local-lis/les=21/22 n=1004 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5461 lcod 123'5459 mlcod 123'5459 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.146 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.142+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5463 (115'2403,123'5463] local-lis/les=21/22 n=1002 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5461 crt=123'5461 lcod 123'5459 mlcod 123'5459 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.146 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.142+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5695 (115'2609,123'5695] local-lis/les=21/22 n=994 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'5693 crt=123'5693 lcod 123'5691 mlcod 123'5691 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.150 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.146+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5465 (115'2403,123'5465] local-lis/les=21/22 n=1000 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5465 lcod 123'5463 mlcod 123'5463 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.153 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.150+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5697 (115'2609,123'5697] local-lis/les=21/22 n=992 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'5697 lcod 123'5695 mlcod 123'5695 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.153 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.150+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5467 (115'2403,123'5467] local-lis/les=21/22 n=998 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5465 crt=123'5465 lcod 123'5463 mlcod 123'5463 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.155 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.150+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5699 (115'2609,123'5699] local-lis/les=21/22 n=990 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'5697 crt=123'5697 lcod 123'5695 mlcod 123'5695 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.160 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.158+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5469 (115'2403,123'5469] local-lis/les=21/22 n=996 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5469 lcod 123'5467 mlcod 123'5467 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.163 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.158+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5701 (115'2609,123'5701] local-lis/les=21/22 n=988 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'5701 lcod 123'5699 mlcod 123'5699 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.164 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.158+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5471 (115'2403,123'5471] local-lis/les=21/22 n=994 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5469 crt=123'5469 lcod 123'5467 mlcod 123'5467 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.165 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.162+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5703 (115'2609,123'5703] local-lis/les=21/22 n=986 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'5701 crt=123'5701 lcod 123'5699 mlcod 123'5699 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.172 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.166+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5473 (115'2403,123'5473] local-lis/les=21/22 n=992 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5473 lcod 123'5471 mlcod 123'5471 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.173 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.166+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5705 (115'2609,123'5705] local-lis/les=21/22 n=984 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'5705 lcod 123'5703 mlcod 123'5703 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.173 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.170+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5475 (115'2403,123'5475] local-lis/les=21/22 n=990 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5473 crt=123'5473 lcod 123'5471 mlcod 123'5471 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.173 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.170+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5707 (115'2609,123'5707] local-lis/les=21/22 n=982 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'5705 crt=123'5705 lcod 123'5703 mlcod 123'5703 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.178 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.174+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5477 (115'2403,123'5477] local-lis/les=21/22 n=988 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5477 lcod 123'5475 mlcod 123'5475 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.180 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.178+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5709 (115'2609,123'5709] local-lis/les=21/22 n=980 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'5709 lcod 123'5707 mlcod 123'5707 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.180 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.178+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5479 (115'2403,123'5479] local-lis/les=21/22 n=986 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5477 crt=123'5477 lcod 123'5475 mlcod 123'5475 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.184 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.178+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5711 (115'2709,123'5711] local-lis/les=21/22 n=978 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'5709 crt=123'5709 lcod 123'5707 mlcod 123'5707 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.187 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.182+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5481 (115'2403,123'5481] local-lis/les=21/22 n=984 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5481 lcod 123'5479 mlcod 123'5479 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.189 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.186+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5483 (115'2403,123'5483] local-lis/les=21/22 n=982 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5481 crt=123'5481 lcod 123'5479 mlcod 123'5479 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.190 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.186+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5713 (115'2709,123'5713] local-lis/les=21/22 n=976 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'5713 lcod 123'5711 mlcod 123'5711 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.193 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.190+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5715 (115'2709,123'5715] local-lis/les=21/22 n=974 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'5713 crt=123'5713 lcod 123'5711 mlcod 123'5711 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.197 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.194+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5485 (115'2403,123'5485] local-lis/les=21/22 n=980 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5485 lcod 123'5483 mlcod 123'5483 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.198 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.194+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5717 (115'2709,123'5717] local-lis/les=21/22 n=972 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'5717 lcod 123'5715 mlcod 123'5715 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.199 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.194+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5487 (115'2403,123'5487] local-lis/les=21/22 n=978 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5485 crt=123'5485 lcod 123'5483 mlcod 123'5483 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.201 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.198+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5719 (115'2709,123'5719] local-lis/les=21/22 n=970 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'5717 crt=123'5717 lcod 123'5715 mlcod 123'5715 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.204 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:12.197+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5256 (114'2206,123'5256] local-lis/les=21/22 n=1013 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5256 lcod 123'5254 mlcod 123'5254 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.205 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:12.202+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5321 (115'2306,123'5321] local-lis/les=21/22 n=1074 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5320 crt=123'5318 lcod 123'5318 mlcod 123'5318 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.206 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:12.201+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5258 (114'2206,123'5258] local-lis/les=21/22 n=1011 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5256 crt=123'5256 lcod 123'5254 mlcod 123'5254 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.207 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:12.202+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5323 (115'2306,123'5323] local-lis/les=21/22 n=1072 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5320 crt=123'5320 lcod 123'5318 mlcod 123'5318 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.211 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:12.206+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5410 (115'2404,123'5410] local-lis/les=21/22 n=979 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5410 lcod 123'5408 mlcod 123'5408 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.212 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.206+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5721 (115'2709,123'5721] local-lis/les=21/22 n=968 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'5721 lcod 123'5719 mlcod 123'5719 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.212 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.206+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5489 (115'2403,123'5489] local-lis/les=21/22 n=976 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5489 lcod 123'5487 mlcod 123'5487 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.213 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:12.210+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5412 (115'2404,123'5412] local-lis/les=21/22 n=977 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5410 crt=123'5410 lcod 123'5408 mlcod 123'5408 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.215 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.210+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5723 (115'2709,123'5723] local-lis/les=21/22 n=966 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'5721 crt=123'5721 lcod 123'5719 mlcod 123'5719 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.215 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.210+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5491 (115'2403,123'5491] local-lis/les=21/22 n=974 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5489 crt=123'5489 lcod 123'5487 mlcod 123'5487 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.218 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:12.214+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5281 (114'2203,123'5281] local-lis/les=21/22 n=938 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5281 lcod 123'5279 mlcod 123'5279 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.227 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.218+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5493 (115'2403,123'5493] local-lis/les=21/22 n=972 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5493 lcod 123'5491 mlcod 123'5491 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.227 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.222+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5495 (115'2403,123'5495] local-lis/les=21/22 n=970 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5493 crt=123'5493 lcod 123'5491 mlcod 123'5491 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.228 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.222+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5725 (115'2709,123'5725] local-lis/les=21/22 n=964 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'5725 lcod 123'5723 mlcod 123'5723 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.229 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:12.218+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5283 (114'2203,123'5283] local-lis/les=21/22 n=936 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5281 crt=123'5281 lcod 123'5279 mlcod 123'5279 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.229 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:12.218+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5526 (115'2511,123'5526] local-lis/les=21/22 n=1134 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'5526 lcod 123'5524 mlcod 123'5524 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.229 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:12.222+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5528 (115'2511,123'5528] local-lis/les=21/22 n=1132 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'5526 crt=123'5526 lcod 123'5524 mlcod 123'5524 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.233 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.226+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5727 (115'2709,123'5727] local-lis/les=21/22 n=962 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'5725 crt=123'5725 lcod 123'5723 mlcod 123'5723 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.234 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.230+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5497 (115'2403,123'5497] local-lis/les=21/22 n=968 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5497 lcod 123'5495 mlcod 123'5495 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.236 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.234+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5499 (115'2403,123'5499] local-lis/les=21/22 n=966 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5497 crt=123'5497 lcod 123'5495 mlcod 123'5495 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.239 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.234+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5729 (115'2709,123'5729] local-lis/les=21/22 n=960 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'5729 lcod 123'5727 mlcod 123'5727 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.241 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.238+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5731 (115'2709,123'5731] local-lis/les=21/22 n=958 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'5729 crt=123'5729 lcod 123'5727 mlcod 123'5727 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.242 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.238+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5501 (115'2403,123'5501] local-lis/les=21/22 n=964 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5501 lcod 123'5499 mlcod 123'5499 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.244 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.242+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5503 (115'2403,123'5503] local-lis/les=21/22 n=962 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5501 crt=123'5501 lcod 123'5499 mlcod 123'5499 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.247 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.242+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5733 (115'2709,123'5733] local-lis/les=21/22 n=956 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'5733 lcod 123'5731 mlcod 123'5731 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.250 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.246+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5735 (115'2709,123'5735] local-lis/les=21/22 n=954 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'5733 crt=123'5733 lcod 123'5731 mlcod 123'5731 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.252 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.250+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5505 (115'2503,123'5505] local-lis/les=21/22 n=960 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5505 lcod 123'5503 mlcod 123'5503 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.256 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.250+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5507 (115'2503,123'5507] local-lis/les=21/22 n=958 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5505 crt=123'5505 lcod 123'5503 mlcod 123'5503 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.256 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.254+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5737 (115'2709,123'5737] local-lis/les=21/22 n=952 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'5737 lcod 123'5735 mlcod 123'5735 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.259 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.254+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5739 (115'2709,123'5739] local-lis/les=21/22 n=950 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'5737 crt=123'5737 lcod 123'5735 mlcod 123'5735 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.260 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.258+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5509 (115'2503,123'5509] local-lis/les=21/22 n=956 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5509 lcod 123'5507 mlcod 123'5507 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.264 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.262+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5741 (115'2709,123'5741] local-lis/les=21/22 n=948 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'5741 lcod 123'5739 mlcod 123'5739 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.266 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.262+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5743 (115'2709,123'5743] local-lis/les=21/22 n=946 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'5741 crt=123'5741 lcod 123'5739 mlcod 123'5739 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.267 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.262+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5511 (115'2503,123'5511] local-lis/les=21/22 n=954 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5511 lcod 123'5509 mlcod 123'5509 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:12.269 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.266+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5513 (115'2503,123'5513] local-lis/les=21/22 n=952 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5511 crt=123'5511 lcod 123'5509 mlcod 123'5509 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:12.273 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.270+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5745 (115'2709,123'5745] local-lis/les=21/22 n=944 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'5745 lcod 123'5743 mlcod 123'5743 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.274 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.270+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5515 (115'2503,123'5515] local-lis/les=21/22 n=950 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5515 lcod 123'5513 mlcod 123'5513 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:12.275 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.270+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5747 (115'2709,123'5747] local-lis/les=21/22 n=942 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'5745 crt=123'5745 lcod 123'5743 mlcod 123'5743 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.276 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.274+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5517 (115'2503,123'5517] local-lis/les=21/22 n=948 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5515 crt=123'5515 lcod 123'5513 mlcod 123'5513 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:12.283 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.278+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5519 (115'2503,123'5519] local-lis/les=21/22 n=946 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5519 lcod 123'5517 mlcod 123'5517 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:12.283 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.278+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5749 (115'2709,123'5749] local-lis/les=21/22 n=940 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'5749 lcod 123'5747 mlcod 123'5747 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:12.284 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:12.277+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5410 (115'2405,123'5410] local-lis/les=21/22 n=1107 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=123'5410 lcod 123'5408 mlcod 123'5408 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.285 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.282+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5521 (115'2503,123'5521] local-lis/les=21/22 n=944 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5519 crt=123'5519 lcod 123'5517 mlcod 123'5517 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:12.285 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.282+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5751 (115'2709,123'5751] local-lis/les=21/22 n=938 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'5749 crt=123'5749 lcod 123'5747 mlcod 123'5747 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:12.287 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:12.281+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5412 (115'2405,123'5412] local-lis/les=21/22 n=1105 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5410 crt=123'5410 lcod 123'5408 mlcod 123'5408 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.291 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.286+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5523 (115'2503,123'5523] local-lis/les=21/22 n=942 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5523 lcod 123'5521 mlcod 123'5521 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:12.292 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.286+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5525 (115'2503,123'5525] local-lis/les=21/22 n=940 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5523 crt=123'5523 lcod 123'5521 mlcod 123'5521 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:12.294 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.290+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5753 (115'2709,123'5753] local-lis/les=21/22 n=936 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'5753 lcod 123'5751 mlcod 123'5751 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:12.296 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.294+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5755 (115'2709,123'5755] local-lis/les=21/22 n=934 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'5753 crt=123'5753 lcod 123'5751 mlcod 123'5751 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:12.297 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.294+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5527 (115'2503,123'5527] local-lis/les=21/22 n=938 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5527 lcod 123'5525 mlcod 123'5525 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:12.299 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.294+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5529 (115'2503,123'5529] local-lis/les=21/22 n=936 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5527 crt=123'5527 lcod 123'5525 mlcod 123'5525 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:12.301 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.298+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5757 (115'2709,123'5757] local-lis/les=21/22 n=932 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'5757 lcod 123'5755 mlcod 123'5755 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:12.304 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.302+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5759 (115'2709,123'5759] local-lis/les=21/22 n=930 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'5757 crt=123'5757 lcod 123'5755 mlcod 123'5755 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:12.305 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.302+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5531 (115'2503,123'5531] local-lis/les=21/22 n=934 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5531 lcod 123'5529 mlcod 123'5529 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:12.308 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.302+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5533 (115'2503,123'5533] local-lis/les=21/22 n=932 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5531 crt=123'5531 lcod 123'5529 mlcod 123'5529 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:12.308 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.306+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5761 (115'2709,123'5761] local-lis/les=21/22 n=928 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'5761 lcod 123'5759 mlcod 123'5759 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:12.310 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.306+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5763 (115'2709,123'5763] local-lis/les=21/22 n=926 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'5761 crt=123'5761 lcod 123'5759 mlcod 123'5759 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:12.313 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.310+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5765 (115'2709,123'5765] local-lis/les=21/22 n=924 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'5765 lcod 123'5763 mlcod 123'5763 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:12.314 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.310+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5767 (115'2709,123'5767] local-lis/les=21/22 n=922 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'5765 crt=123'5765 lcod 123'5763 mlcod 123'5763 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:12.317 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.314+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5769 (115'2709,123'5769] local-lis/les=21/22 n=920 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'5769 lcod 123'5767 mlcod 123'5767 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:12.319 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.314+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5771 (115'2709,123'5771] local-lis/les=21/22 n=918 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'5769 crt=123'5769 lcod 123'5767 mlcod 123'5767 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:12.341 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:12.338+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5530 (115'2511,123'5530] local-lis/les=21/22 n=1130 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'5530 lcod 123'5528 mlcod 123'5528 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.342 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:12.338+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5532 (115'2511,123'5532] local-lis/les=21/22 n=1128 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'5530 crt=123'5530 lcod 123'5528 mlcod 123'5528 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.350 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:12.345+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5414 (115'2405,123'5414] local-lis/les=21/22 n=1103 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=123'5414 lcod 123'5412 mlcod 123'5412 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.352 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:12.345+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5416 (115'2405,123'5416] local-lis/les=21/22 n=1101 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5414 crt=123'5414 lcod 123'5412 mlcod 123'5412 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.352 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:12.350+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5534 (115'2511,123'5534] local-lis/les=21/22 n=1126 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'5534 lcod 123'5532 mlcod 123'5532 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.355 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:12.350+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5536 (115'2511,123'5536] local-lis/les=21/22 n=1124 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'5534 crt=123'5534 lcod 123'5532 mlcod 123'5532 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.415 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:12.410+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5285 (114'2203,123'5285] local-lis/les=21/22 n=934 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5285 lcod 123'5283 mlcod 123'5283 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.419 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:12.414+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5287 (114'2203,123'5287] local-lis/les=21/22 n=932 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5285 crt=123'5285 lcod 123'5283 mlcod 123'5283 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.419 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.414+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5535 (115'2503,123'5535] local-lis/les=21/22 n=930 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5535 lcod 123'5533 mlcod 123'5533 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:12.420 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:12.418+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5538 (115'2511,123'5538] local-lis/les=21/22 n=1122 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'5538 lcod 123'5536 mlcod 123'5536 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.424 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.418+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5537 (115'2503,123'5537] local-lis/les=21/22 n=928 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5535 crt=123'5535 lcod 123'5533 mlcod 123'5533 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:12.424 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.422+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5773 (115'2709,123'5773] local-lis/les=21/22 n=916 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'5773 lcod 123'5771 mlcod 123'5771 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:12.425 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:12.418+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5540 (115'2511,123'5540] local-lis/les=21/22 n=1120 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'5538 crt=123'5538 lcod 123'5536 mlcod 123'5536 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.427 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.422+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5775 (115'2709,123'5775] local-lis/les=21/22 n=914 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'5773 crt=123'5773 lcod 123'5771 mlcod 123'5771 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:12.431 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:12.426+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5289 (114'2203,123'5289] local-lis/les=21/22 n=930 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5289 lcod 123'5287 mlcod 123'5287 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.433 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:12.430+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5291 (114'2203,123'5291] local-lis/les=21/22 n=928 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5289 crt=123'5289 lcod 123'5287 mlcod 123'5287 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.438 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.434+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5539 (115'2503,123'5539] local-lis/les=21/22 n=926 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5539 lcod 123'5537 mlcod 123'5537 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:12.439 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:12.434+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5542 (115'2511,123'5542] local-lis/les=21/22 n=1118 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'5542 lcod 123'5540 mlcod 123'5540 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.440 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.438+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5541 (115'2503,123'5541] local-lis/les=21/22 n=924 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5539 crt=123'5539 lcod 123'5537 mlcod 123'5537 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:12.442 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:12.438+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5544 (115'2511,123'5544] local-lis/les=21/22 n=1116 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'5542 crt=123'5542 lcod 123'5540 mlcod 123'5540 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.447 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:12.442+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5293 (114'2203,123'5293] local-lis/les=21/22 n=926 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5293 lcod 123'5291 mlcod 123'5291 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.448 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.442+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5777 (115'2709,123'5777] local-lis/les=21/22 n=912 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'5777 lcod 123'5775 mlcod 123'5775 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:12.449 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.442+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5779 (115'2709,123'5779] local-lis/les=21/22 n=910 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'5777 crt=123'5777 lcod 123'5775 mlcod 123'5775 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:12.450 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:12.446+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5295 (114'2203,123'5295] local-lis/les=21/22 n=924 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5293 crt=123'5293 lcod 123'5291 mlcod 123'5291 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.452 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.450+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5543 (115'2503,123'5543] local-lis/les=21/22 n=922 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5543 lcod 123'5541 mlcod 123'5541 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:12.457 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:12.450+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5546 (115'2511,123'5546] local-lis/les=21/22 n=1114 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'5546 lcod 123'5544 mlcod 123'5544 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.458 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.454+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5545 (115'2503,123'5545] local-lis/les=21/22 n=920 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5543 crt=123'5543 lcod 123'5541 mlcod 123'5541 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:12.458 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:12.449+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5261 (114'2206,123'5261] local-lis/les=21/22 n=1008 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5260 crt=123'5258 lcod 123'5258 mlcod 123'5258 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.459 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:12.453+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5263 (114'2206,123'5263] local-lis/les=21/22 n=1006 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5260 crt=123'5260 lcod 123'5258 mlcod 123'5258 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.460 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:12.454+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5548 (115'2511,123'5548] local-lis/les=21/22 n=1112 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'5546 crt=123'5546 lcod 123'5544 mlcod 123'5544 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.468 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.458+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5781 (115'2709,123'5781] local-lis/les=21/22 n=908 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'5781 lcod 123'5779 mlcod 123'5779 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:12.469 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.462+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5783 (115'2709,123'5783] local-lis/les=21/22 n=906 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'5781 crt=123'5781 lcod 123'5779 mlcod 123'5779 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:12.469 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:12.466+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5414 (115'2404,123'5414] local-lis/les=21/22 n=975 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5414 lcod 123'5412 mlcod 123'5412 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.469 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:12.462+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5297 (114'2203,123'5297] local-lis/les=21/22 n=922 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5297 lcod 123'5295 mlcod 123'5295 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.470 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.466+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5548 (115'2503,123'5548] local-lis/les=21/22 n=917 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5548 lcod 123'5547 mlcod 123'5547 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:12.471 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:12.466+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5299 (114'2203,123'5299] local-lis/les=21/22 n=920 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5297 crt=123'5297 lcod 123'5295 mlcod 123'5295 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.472 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:12.466+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5416 (115'2404,123'5416] local-lis/les=21/22 n=973 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5414 crt=123'5414 lcod 123'5412 mlcod 123'5412 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.476 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.470+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5550 (115'2503,123'5550] local-lis/les=21/22 n=915 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5548 crt=123'5548 lcod 123'5547 mlcod 123'5547 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:12.477 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:12.465+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5265 (114'2206,123'5265] local-lis/les=21/22 n=1004 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5265 lcod 123'5263 mlcod 123'5263 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.477 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:12.470+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5550 (115'2511,123'5550] local-lis/les=21/22 n=1110 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'5550 lcod 123'5548 mlcod 123'5548 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.478 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:12.469+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5267 (114'2206,123'5267] local-lis/les=21/22 n=1002 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5265 crt=123'5265 lcod 123'5263 mlcod 123'5263 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.479 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:12.474+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5552 (115'2511,123'5552] local-lis/les=21/22 n=1108 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'5550 crt=123'5550 lcod 123'5548 mlcod 123'5548 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.479 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:12.473+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5419 (115'2405,123'5419] local-lis/les=21/22 n=1098 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5418 crt=123'5416 lcod 123'5416 mlcod 123'5416 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.480 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.478+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5785 (115'2709,123'5785] local-lis/les=21/22 n=904 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'5785 lcod 123'5783 mlcod 123'5783 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:12.483 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:12.473+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5421 (115'2405,123'5421] local-lis/les=21/22 n=1096 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5418 crt=123'5418 lcod 123'5416 mlcod 123'5416 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.484 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:12.478+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5302 (114'2203,123'5302] local-lis/les=21/22 n=917 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5301 crt=123'5297 lcod 123'5299 mlcod 123'5299 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.484 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.478+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5787 (115'2709,123'5787] local-lis/les=21/22 n=902 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'5785 crt=123'5785 lcod 123'5783 mlcod 123'5783 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:12.486 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:12.482+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5304 (114'2203,123'5304] local-lis/les=21/22 n=915 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5301 crt=123'5301 lcod 123'5299 mlcod 123'5299 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.488 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:12.486+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5418 (115'2404,123'5418] local-lis/les=21/22 n=971 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5418 lcod 123'5416 mlcod 123'5416 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.489 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:12.481+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5269 (114'2206,123'5269] local-lis/les=21/22 n=1000 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5269 lcod 123'5267 mlcod 123'5267 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.492 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.486+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5552 (115'2503,123'5552] local-lis/les=21/22 n=913 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5552 lcod 123'5550 mlcod 123'5550 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:12.492 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:12.486+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5420 (115'2404,123'5420] local-lis/les=21/22 n=969 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5418 crt=123'5418 lcod 123'5416 mlcod 123'5416 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.494 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:12.485+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5271 (114'2206,123'5271] local-lis/les=21/22 n=998 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5269 crt=123'5269 lcod 123'5267 mlcod 123'5267 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.495 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.490+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5554 (115'2503,123'5554] local-lis/les=21/22 n=911 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5552 crt=123'5552 lcod 123'5550 mlcod 123'5550 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:12.497 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.494+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5789 (115'2709,123'5789] local-lis/les=21/22 n=900 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'5789 lcod 123'5787 mlcod 123'5787 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:12.498 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:12.489+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5424 (115'2405,123'5424] local-lis/les=21/22 n=1093 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5423 crt=123'5419 lcod 123'5421 mlcod 123'5421 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.500 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:12.498+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5555 (115'2511,123'5555] local-lis/les=21/22 n=1105 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'5554 crt=123'5550 lcod 123'5552 mlcod 123'5552 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.501 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:12.493+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5426 (115'2405,123'5426] local-lis/les=21/22 n=1091 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5423 crt=123'5423 lcod 123'5421 mlcod 123'5421 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.503 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:12.498+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5557 (115'2511,123'5557] local-lis/les=21/22 n=1103 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'5554 crt=123'5554 lcod 123'5552 mlcod 123'5552 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.505 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:12.502+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5422 (115'2404,123'5422] local-lis/les=21/22 n=967 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5422 lcod 123'5420 mlcod 123'5420 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.506 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.502+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5556 (115'2503,123'5556] local-lis/les=21/22 n=909 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5556 lcod 123'5554 mlcod 123'5554 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:12.507 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:12.502+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5424 (115'2404,123'5424] local-lis/les=21/22 n=965 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5422 crt=123'5422 lcod 123'5420 mlcod 123'5420 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.508 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:12.501+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5274 (114'2206,123'5274] local-lis/les=21/22 n=995 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5273 crt=123'5273 lcod 123'5271 mlcod 123'5271 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.511 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:12.505+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5276 (114'2206,123'5276] local-lis/les=21/22 n=993 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5273 crt=123'5273 lcod 123'5271 mlcod 123'5271 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.514 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.510+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5791 (115'2709,123'5791] local-lis/les=21/22 n=898 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'5791 lcod 123'5789 mlcod 123'5789 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:12.517 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.514+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5793 (115'2709,123'5793] local-lis/les=21/22 n=896 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'5791 crt=123'5791 lcod 123'5789 mlcod 123'5789 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:12.518 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:12.514+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5306 (114'2304,123'5306] local-lis/les=21/22 n=913 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5306 lcod 123'5304 mlcod 123'5304 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.520 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:12.518+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5308 (114'2304,123'5308] local-lis/les=21/22 n=911 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5306 crt=123'5306 lcod 123'5304 mlcod 123'5304 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.523 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:12.518+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5426 (115'2404,123'5426] local-lis/les=21/22 n=963 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5426 lcod 123'5424 mlcod 123'5424 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.524 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:12.517+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5278 (114'2206,123'5278] local-lis/les=21/22 n=991 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5278 lcod 123'5276 mlcod 123'5276 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.526 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:12.522+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5428 (115'2404,123'5428] local-lis/les=21/22 n=961 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5426 crt=123'5426 lcod 123'5424 mlcod 123'5424 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.526 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.522+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5795 (115'2709,123'5795] local-lis/les=21/22 n=894 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'5795 lcod 123'5793 mlcod 123'5793 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:12.527 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:12.522+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5561 (115'2511,123'5561] local-lis/les=21/22 n=1099 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'5559 crt=123'5555 lcod 123'5557 mlcod 123'5557 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.529 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:12.522+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5563 (115'2511,123'5563] local-lis/les=21/22 n=1097 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'5559 crt=123'5559 lcod 123'5557 mlcod 123'5557 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.530 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:12.517+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5280 (114'2206,123'5280] local-lis/les=21/22 n=989 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5278 crt=123'5278 lcod 123'5276 mlcod 123'5276 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.530 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.526+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5558 (115'2503,123'5558] local-lis/les=21/22 n=907 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5558 lcod 123'5556 mlcod 123'5556 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:12.531 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.526+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5797 (115'2709,123'5797] local-lis/les=21/22 n=892 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'5795 crt=123'5795 lcod 123'5793 mlcod 123'5793 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:12.531 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.526+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5560 (115'2503,123'5560] local-lis/les=21/22 n=905 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5558 crt=123'5558 lcod 123'5556 mlcod 123'5556 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:12.538 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:12.529+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5282 (114'2206,123'5282] local-lis/les=21/22 n=987 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5282 lcod 123'5280 mlcod 123'5280 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.538 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:12.534+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5311 (114'2304,123'5311] local-lis/les=21/22 n=908 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5310 crt=123'5306 lcod 123'5308 mlcod 123'5308 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.539 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:12.533+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5429 (115'2405,123'5429] local-lis/les=21/22 n=1088 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5428 crt=123'5426 lcod 123'5426 mlcod 123'5426 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.540 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:12.534+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5313 (114'2304,123'5313] local-lis/les=21/22 n=906 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5310 crt=123'5310 lcod 123'5308 mlcod 123'5308 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.540 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:12.533+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5284 (114'2206,123'5284] local-lis/les=21/22 n=985 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5282 crt=123'5282 lcod 123'5280 mlcod 123'5280 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.541 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:12.533+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5431 (115'2405,123'5431] local-lis/les=21/22 n=1086 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5428 crt=123'5428 lcod 123'5426 mlcod 123'5426 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.542 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:12.538+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5430 (115'2404,123'5430] local-lis/les=21/22 n=959 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5430 lcod 123'5428 mlcod 123'5428 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.544 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:12.542+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5432 (115'2404,123'5432] local-lis/les=21/22 n=957 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5430 crt=123'5430 lcod 123'5428 mlcod 123'5428 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.545 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:12.542+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5565 (115'2511,123'5565] local-lis/les=21/22 n=1095 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'5565 lcod 123'5563 mlcod 123'5563 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.547 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:12.542+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5567 (115'2511,123'5567] local-lis/les=21/22 n=1093 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'5565 crt=123'5565 lcod 123'5563 mlcod 123'5563 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.548 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:12.542+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5326 (115'2306,123'5326] local-lis/les=21/22 n=1069 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5325 crt=123'5323 lcod 123'5323 mlcod 123'5323 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.550 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:12.546+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5328 (115'2306,123'5328] local-lis/les=21/22 n=1067 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5325 crt=123'5325 lcod 123'5323 mlcod 123'5323 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.554 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:12.545+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5286 (114'2206,123'5286] local-lis/les=21/22 n=983 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5286 lcod 123'5284 mlcod 123'5284 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.556 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.550+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5799 (115'2709,123'5799] local-lis/les=21/22 n=890 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'5799 lcod 123'5797 mlcod 123'5797 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:12.557 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:12.549+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5288 (114'2206,123'5288] local-lis/les=21/22 n=981 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5286 crt=123'5286 lcod 123'5284 mlcod 123'5284 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.559 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.550+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5801 (115'2709,123'5801] local-lis/les=21/22 n=888 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'5799 crt=123'5799 lcod 123'5797 mlcod 123'5797 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:12.559 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.554+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5562 (115'2503,123'5562] local-lis/les=21/22 n=903 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5562 lcod 123'5560 mlcod 123'5560 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:12.560 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.558+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5564 (115'2503,123'5564] local-lis/les=21/22 n=901 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5562 crt=123'5562 lcod 123'5560 mlcod 123'5560 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:12.561 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:12.558+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5332 (115'2306,123'5332] local-lis/les=21/22 n=1063 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5330 crt=123'5328 lcod 123'5328 mlcod 123'5328 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.565 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:12.562+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5434 (115'2404,123'5434] local-lis/les=21/22 n=955 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5434 lcod 123'5432 mlcod 123'5432 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.565 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:12.562+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5334 (115'2306,123'5334] local-lis/les=21/22 n=1061 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5330 crt=123'5330 lcod 123'5328 mlcod 123'5328 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.567 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:12.562+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5436 (115'2404,123'5436] local-lis/les=21/22 n=953 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5434 crt=123'5434 lcod 123'5432 mlcod 123'5432 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.568 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:12.561+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5290 (114'2206,123'5290] local-lis/les=21/22 n=979 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5290 lcod 123'5288 mlcod 123'5288 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.570 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:12.565+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5292 (114'2206,123'5292] local-lis/les=21/22 n=977 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5290 crt=123'5290 lcod 123'5288 mlcod 123'5288 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.573 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:12.570+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5337 (115'2306,123'5337] local-lis/les=21/22 n=1058 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5336 crt=123'5332 lcod 123'5334 mlcod 123'5334 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.574 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.570+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5567 (115'2503,123'5567] local-lis/les=21/22 n=898 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5566 crt=123'5564 lcod 123'5564 mlcod 123'5564 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:12.578 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:12.570+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5438 (115'2404,123'5438] local-lis/les=21/22 n=951 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5438 lcod 123'5436 mlcod 123'5436 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.578 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:12.574+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5339 (115'2306,123'5339] local-lis/les=21/22 n=1056 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5336 crt=123'5336 lcod 123'5334 mlcod 123'5334 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.579 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.574+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5569 (115'2503,123'5569] local-lis/les=21/22 n=896 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5566 crt=123'5566 lcod 123'5564 mlcod 123'5564 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:12.580 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:12.573+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5294 (114'2206,123'5294] local-lis/les=21/22 n=975 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5294 lcod 123'5292 mlcod 123'5292 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.580 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:12.574+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5440 (115'2404,123'5440] local-lis/les=21/22 n=949 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5438 crt=123'5438 lcod 123'5436 mlcod 123'5436 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.582 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:12.573+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5296 (114'2206,123'5296] local-lis/les=21/22 n=973 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5294 crt=123'5294 lcod 123'5292 mlcod 123'5292 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.586 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:12.582+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5316 (114'2304,123'5316] local-lis/les=21/22 n=903 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5315 crt=123'5315 lcod 123'5313 mlcod 123'5313 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.588 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:12.586+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5342 (115'2306,123'5342] local-lis/les=21/22 n=1053 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5341 crt=123'5339 lcod 123'5339 mlcod 123'5339 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.588 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:12.586+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5318 (114'2304,123'5318] local-lis/les=21/22 n=901 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5315 crt=123'5315 lcod 123'5313 mlcod 123'5313 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.590 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:12.586+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5442 (115'2404,123'5442] local-lis/les=21/22 n=947 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5442 lcod 123'5440 mlcod 123'5440 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.592 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:12.590+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5569 (115'2511,123'5569] local-lis/les=21/22 n=1091 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'5569 lcod 123'5567 mlcod 123'5567 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.594 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:12.589+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5298 (114'2206,123'5298] local-lis/les=21/22 n=971 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5298 lcod 123'5296 mlcod 123'5296 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.595 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:12.590+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5571 (115'2511,123'5571] local-lis/les=21/22 n=1089 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'5569 crt=123'5569 lcod 123'5567 mlcod 123'5567 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.596 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:12.590+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5444 (115'2404,123'5444] local-lis/les=21/22 n=945 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5442 crt=123'5442 lcod 123'5440 mlcod 123'5440 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.598 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:12.590+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5344 (115'2306,123'5344] local-lis/les=21/22 n=1051 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5341 crt=123'5341 lcod 123'5339 mlcod 123'5339 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.599 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:12.594+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5320 (114'2304,123'5320] local-lis/les=21/22 n=899 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5320 lcod 123'5318 mlcod 123'5318 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.600 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:12.593+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5300 (114'2206,123'5300] local-lis/les=21/22 n=969 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5298 crt=123'5298 lcod 123'5296 mlcod 123'5296 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.601 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:12.598+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5322 (114'2304,123'5322] local-lis/les=21/22 n=897 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5320 crt=123'5320 lcod 123'5318 mlcod 123'5318 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.602 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:12.598+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5347 (115'2306,123'5347] local-lis/les=21/22 n=1048 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5346 crt=123'5342 lcod 123'5344 mlcod 123'5344 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.605 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:12.602+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5446 (115'2404,123'5446] local-lis/les=21/22 n=943 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5446 lcod 123'5444 mlcod 123'5444 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.606 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:12.602+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5349 (115'2306,123'5349] local-lis/les=21/22 n=1046 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5346 crt=123'5346 lcod 123'5344 mlcod 123'5344 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.607 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:12.602+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5573 (115'2511,123'5573] local-lis/les=21/22 n=1087 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'5573 lcod 123'5571 mlcod 123'5571 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.608 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:12.601+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5302 (114'2206,123'5302] local-lis/les=21/22 n=967 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5302 lcod 123'5300 mlcod 123'5300 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.608 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:12.606+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5575 (115'2511,123'5575] local-lis/les=21/22 n=1085 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'5573 crt=123'5573 lcod 123'5571 mlcod 123'5571 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.612 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:12.610+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5324 (114'2304,123'5324] local-lis/les=21/22 n=895 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5324 lcod 123'5322 mlcod 123'5322 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.613 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:12.606+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5448 (115'2404,123'5448] local-lis/les=21/22 n=941 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5446 crt=123'5446 lcod 123'5444 mlcod 123'5444 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.616 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:12.609+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5304 (114'2206,123'5304] local-lis/les=21/22 n=965 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5302 crt=123'5302 lcod 123'5300 mlcod 123'5300 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.617 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:12.610+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5326 (114'2304,123'5326] local-lis/les=21/22 n=893 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5324 crt=123'5324 lcod 123'5322 mlcod 123'5322 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.618 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:12.614+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5577 (115'2511,123'5577] local-lis/les=21/22 n=1083 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'5577 lcod 123'5575 mlcod 123'5575 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.619 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:12.614+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5450 (115'2404,123'5450] local-lis/les=21/22 n=939 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5450 lcod 123'5448 mlcod 123'5448 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.622 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:12.618+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5579 (115'2511,123'5579] local-lis/les=21/22 n=1081 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'5577 crt=123'5577 lcod 123'5575 mlcod 123'5575 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.622 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:12.618+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5452 (115'2404,123'5452] local-lis/les=21/22 n=937 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5450 crt=123'5450 lcod 123'5448 mlcod 123'5448 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.624 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:12.617+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5306 (114'2206,123'5306] local-lis/les=21/22 n=963 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5306 lcod 123'5304 mlcod 123'5304 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.628 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:12.626+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5581 (115'2511,123'5581] local-lis/les=21/22 n=1079 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'5581 lcod 123'5579 mlcod 123'5579 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.628 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:12.621+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5308 (115'2306,123'5308] local-lis/les=21/22 n=961 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5306 crt=123'5306 lcod 123'5304 mlcod 123'5304 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.630 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:12.626+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5583 (115'2511,123'5583] local-lis/les=21/22 n=1077 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'5581 crt=123'5581 lcod 123'5579 mlcod 123'5579 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.631 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:12.626+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5454 (115'2404,123'5454] local-lis/les=21/22 n=935 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5454 lcod 123'5452 mlcod 123'5452 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.632 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:12.630+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5456 (115'2404,123'5456] local-lis/les=21/22 n=933 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5454 crt=123'5454 lcod 123'5452 mlcod 123'5452 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.636 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:12.629+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5310 (115'2306,123'5310] local-lis/les=21/22 n=959 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5310 lcod 123'5308 mlcod 123'5308 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.637 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:12.634+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5585 (115'2511,123'5585] local-lis/les=21/22 n=1075 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'5585 lcod 123'5583 mlcod 123'5583 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.639 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:12.633+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5312 (115'2306,123'5312] local-lis/les=21/22 n=957 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5310 crt=123'5310 lcod 123'5308 mlcod 123'5308 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.639 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:12.634+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5458 (115'2404,123'5458] local-lis/les=21/22 n=931 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5458 lcod 123'5456 mlcod 123'5456 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.640 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:12.638+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5587 (115'2511,123'5587] local-lis/les=21/22 n=1073 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'5585 crt=123'5585 lcod 123'5583 mlcod 123'5583 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.642 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:12.638+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5460 (115'2404,123'5460] local-lis/les=21/22 n=929 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5458 crt=123'5458 lcod 123'5456 mlcod 123'5456 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.645 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:12.637+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5314 (115'2306,123'5314] local-lis/les=21/22 n=955 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5314 lcod 123'5312 mlcod 123'5312 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.648 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:12.642+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5462 (115'2404,123'5462] local-lis/les=21/22 n=927 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5462 lcod 123'5460 mlcod 123'5460 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.649 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:12.641+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5316 (115'2306,123'5316] local-lis/les=21/22 n=953 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5314 crt=123'5314 lcod 123'5312 mlcod 123'5312 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.649 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:12.646+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5464 (115'2404,123'5464] local-lis/les=21/22 n=925 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5462 crt=123'5462 lcod 123'5460 mlcod 123'5460 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.651 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:12.645+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5318 (115'2306,123'5318] local-lis/les=21/22 n=951 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5318 lcod 123'5316 mlcod 123'5316 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.654 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:12.645+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5320 (115'2306,123'5320] local-lis/les=21/22 n=949 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5318 crt=123'5318 lcod 123'5316 mlcod 123'5316 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.656 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:12.654+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5466 (115'2404,123'5466] local-lis/les=21/22 n=923 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5466 lcod 123'5464 mlcod 123'5464 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.658 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:12.654+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5468 (115'2404,123'5468] local-lis/les=21/22 n=921 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5466 crt=123'5466 lcod 123'5464 mlcod 123'5464 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.659 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:12.653+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5322 (115'2306,123'5322] local-lis/les=21/22 n=947 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5322 lcod 123'5320 mlcod 123'5320 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.663 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:12.653+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5324 (115'2306,123'5324] local-lis/les=21/22 n=945 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5322 crt=123'5322 lcod 123'5320 mlcod 123'5320 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.663 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:12.658+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5470 (115'2404,123'5470] local-lis/les=21/22 n=919 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5470 lcod 123'5468 mlcod 123'5468 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.664 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:12.662+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5472 (115'2404,123'5472] local-lis/les=21/22 n=917 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5470 crt=123'5470 lcod 123'5468 mlcod 123'5468 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.668 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:12.661+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5326 (115'2306,123'5326] local-lis/les=21/22 n=943 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5326 lcod 123'5324 mlcod 123'5324 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.670 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:12.661+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5328 (115'2306,123'5328] local-lis/les=21/22 n=941 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5326 crt=123'5326 lcod 123'5324 mlcod 123'5324 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.670 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:12.666+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5474 (115'2404,123'5474] local-lis/les=21/22 n=915 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5474 lcod 123'5472 mlcod 123'5472 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.672 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:12.670+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5476 (115'2404,123'5476] local-lis/les=21/22 n=913 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5474 crt=123'5474 lcod 123'5472 mlcod 123'5472 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.675 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:12.669+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5330 (115'2306,123'5330] local-lis/les=21/22 n=939 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5330 lcod 123'5328 mlcod 123'5328 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.677 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:12.669+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5332 (115'2306,123'5332] local-lis/les=21/22 n=937 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5330 crt=123'5330 lcod 123'5328 mlcod 123'5328 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.678 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:12.674+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5478 (115'2404,123'5478] local-lis/les=21/22 n=911 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5478 lcod 123'5476 mlcod 123'5476 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.679 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:12.674+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5480 (115'2404,123'5480] local-lis/les=21/22 n=909 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5478 crt=123'5478 lcod 123'5476 mlcod 123'5476 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.683 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:12.677+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5334 (115'2306,123'5334] local-lis/les=21/22 n=935 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5334 lcod 123'5332 mlcod 123'5332 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.686 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:12.677+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5336 (115'2306,123'5336] local-lis/les=21/22 n=933 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5334 crt=123'5334 lcod 123'5332 mlcod 123'5332 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.687 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:12.682+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5482 (115'2404,123'5482] local-lis/les=21/22 n=907 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5482 lcod 123'5480 mlcod 123'5480 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.688 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:12.686+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5484 (115'2404,123'5484] local-lis/les=21/22 n=905 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5482 crt=123'5482 lcod 123'5480 mlcod 123'5480 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.692 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:12.685+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5338 (115'2306,123'5338] local-lis/les=21/22 n=931 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5338 lcod 123'5336 mlcod 123'5336 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.697 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:12.690+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5486 (115'2404,123'5486] local-lis/les=21/22 n=903 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5486 lcod 123'5484 mlcod 123'5484 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.698 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:12.685+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5340 (115'2306,123'5340] local-lis/les=21/22 n=929 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5338 crt=123'5338 lcod 123'5336 mlcod 123'5336 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.701 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:12.694+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5488 (115'2404,123'5488] local-lis/les=21/22 n=901 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5486 crt=123'5486 lcod 123'5484 mlcod 123'5484 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.701 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:12.693+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5342 (115'2306,123'5342] local-lis/les=21/22 n=927 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5342 lcod 123'5340 mlcod 123'5340 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.702 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:12.697+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5344 (115'2306,123'5344] local-lis/les=21/22 n=925 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5342 crt=123'5342 lcod 123'5340 mlcod 123'5340 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.705 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:12.698+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5490 (115'2404,123'5490] local-lis/les=21/22 n=899 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5490 lcod 123'5488 mlcod 123'5488 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.705 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:12.702+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5492 (115'2404,123'5492] local-lis/les=21/22 n=897 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5490 crt=123'5490 lcod 123'5488 mlcod 123'5488 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.709 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:12.701+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5346 (115'2306,123'5346] local-lis/les=21/22 n=923 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5346 lcod 123'5344 mlcod 123'5344 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.711 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:12.705+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5348 (115'2306,123'5348] local-lis/les=21/22 n=921 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5346 crt=123'5346 lcod 123'5344 mlcod 123'5344 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.713 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:12.710+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5494 (115'2404,123'5494] local-lis/les=21/22 n=895 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5494 lcod 123'5492 mlcod 123'5492 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.714 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:12.710+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5496 (115'2404,123'5496] local-lis/les=21/22 n=893 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5494 crt=123'5494 lcod 123'5492 mlcod 123'5492 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.718 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:12.713+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5350 (115'2306,123'5350] local-lis/les=21/22 n=919 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5350 lcod 123'5348 mlcod 123'5348 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.720 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:12.713+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5352 (115'2306,123'5352] local-lis/les=21/22 n=917 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5350 crt=123'5350 lcod 123'5348 mlcod 123'5348 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.721 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:12.718+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5498 (115'2404,123'5498] local-lis/les=21/22 n=891 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5498 lcod 123'5496 mlcod 123'5496 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.724 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:12.718+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5500 (115'2404,123'5500] local-lis/les=21/22 n=889 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5498 crt=123'5498 lcod 123'5496 mlcod 123'5496 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.727 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:12.721+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5354 (115'2306,123'5354] local-lis/les=21/22 n=915 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5354 lcod 123'5352 mlcod 123'5352 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.729 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:12.721+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5356 (115'2306,123'5356] local-lis/les=21/22 n=913 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5354 crt=123'5354 lcod 123'5352 mlcod 123'5352 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.729 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:12.726+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5502 (115'2404,123'5502] local-lis/les=21/22 n=887 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5502 lcod 123'5500 mlcod 123'5500 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.735 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:12.730+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5504 (115'2404,123'5504] local-lis/les=21/22 n=885 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5504 lcod 123'5502 mlcod 123'5502 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:12.736 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:12.729+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5358 (115'2306,123'5358] local-lis/les=21/22 n=911 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5358 lcod 123'5356 mlcod 123'5356 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.737 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:12.734+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5506 (115'2504,123'5506] local-lis/les=21/22 n=883 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5504 crt=123'5504 lcod 123'5502 mlcod 123'5502 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:12.738 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:12.729+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5360 (115'2306,123'5360] local-lis/les=21/22 n=909 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5358 crt=123'5358 lcod 123'5356 mlcod 123'5356 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.772 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.770+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5803 (115'2709,123'5803] local-lis/les=21/22 n=886 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'5803 lcod 123'5801 mlcod 123'5801 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:12.774 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.770+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5571 (115'2503,123'5571] local-lis/les=21/22 n=894 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5571 lcod 123'5569 mlcod 123'5569 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:12.776 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.774+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5573 (115'2503,123'5573] local-lis/les=21/22 n=892 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5571 crt=123'5571 lcod 123'5569 mlcod 123'5569 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:12.778 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.774+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5805 (115'2709,123'5805] local-lis/les=21/22 n=884 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'5803 crt=123'5803 lcod 123'5801 mlcod 123'5801 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:12.782 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.778+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5575 (115'2503,123'5575] local-lis/les=21/22 n=890 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5575 lcod 123'5573 mlcod 123'5573 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:12.784 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.778+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5577 (115'2503,123'5577] local-lis/les=21/22 n=888 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5575 crt=123'5575 lcod 123'5573 mlcod 123'5573 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:12.784 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.782+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5807 (115'2709,123'5807] local-lis/les=21/22 n=882 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'5807 lcod 123'5805 mlcod 123'5805 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:12.787 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.782+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5809 (115'2709,123'5809] local-lis/les=21/22 n=880 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'5807 crt=123'5807 lcod 123'5805 mlcod 123'5805 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:12.788 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.786+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5579 (115'2503,123'5579] local-lis/les=21/22 n=886 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5579 lcod 123'5577 mlcod 123'5577 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:12.793 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.786+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5581 (115'2503,123'5581] local-lis/les=21/22 n=884 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5579 crt=123'5579 lcod 123'5577 mlcod 123'5577 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:12.795 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.790+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5811 (115'2809,123'5811] local-lis/les=21/22 n=878 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'5811 lcod 123'5809 mlcod 123'5809 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:12.797 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.794+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5583 (115'2503,123'5583] local-lis/les=21/22 n=882 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5583 lcod 123'5581 mlcod 123'5581 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:12.797 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.794+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5813 (115'2809,123'5813] local-lis/les=21/22 n=876 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'5811 crt=123'5811 lcod 123'5809 mlcod 123'5809 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:12.803 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.798+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5585 (115'2503,123'5585] local-lis/les=21/22 n=880 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5583 crt=123'5583 lcod 123'5581 mlcod 123'5581 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:12.804 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.798+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5815 (115'2809,123'5815] local-lis/les=21/22 n=874 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'5815 lcod 123'5813 mlcod 123'5813 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:12.805 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.802+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5817 (115'2809,123'5817] local-lis/les=21/22 n=872 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'5815 crt=123'5815 lcod 123'5813 mlcod 123'5813 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:12.806 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.802+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5587 (115'2503,123'5587] local-lis/les=21/22 n=878 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5587 lcod 123'5585 mlcod 123'5585 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:12.808 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.806+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5589 (115'2503,123'5589] local-lis/les=21/22 n=876 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5587 crt=123'5587 lcod 123'5585 mlcod 123'5585 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:12.811 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.806+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5819 (115'2809,123'5819] local-lis/les=21/22 n=870 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'5819 lcod 123'5817 mlcod 123'5817 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:12.815 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.810+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5821 (115'2809,123'5821] local-lis/les=21/22 n=868 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'5819 crt=123'5819 lcod 123'5817 mlcod 123'5817 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:12.815 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.810+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5591 (115'2503,123'5591] local-lis/les=21/22 n=874 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5591 lcod 123'5589 mlcod 123'5589 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:12.817 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.814+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5593 (115'2503,123'5593] local-lis/les=21/22 n=872 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5591 crt=123'5591 lcod 123'5589 mlcod 123'5589 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:12.818 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.814+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5823 (115'2809,123'5823] local-lis/les=21/22 n=866 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'5823 lcod 123'5821 mlcod 123'5821 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:12.820 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.818+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5825 (115'2809,123'5825] local-lis/les=21/22 n=864 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'5823 crt=123'5823 lcod 123'5821 mlcod 123'5821 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:12.822 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.818+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5595 (115'2503,123'5595] local-lis/les=21/22 n=870 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5595 lcod 123'5593 mlcod 123'5593 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:12.824 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.822+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5597 (115'2503,123'5597] local-lis/les=21/22 n=868 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5595 crt=123'5595 lcod 123'5593 mlcod 123'5593 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:12.825 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.822+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5827 (115'2809,123'5827] local-lis/les=21/22 n=862 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'5827 lcod 123'5825 mlcod 123'5825 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:12.828 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.822+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5829 (115'2809,123'5829] local-lis/les=21/22 n=860 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'5827 crt=123'5827 lcod 123'5825 mlcod 123'5825 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:12.833 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.826+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5599 (115'2503,123'5599] local-lis/les=21/22 n=866 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5599 lcod 123'5597 mlcod 123'5597 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:12.836 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.830+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5601 (115'2503,123'5601] local-lis/les=21/22 n=864 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5599 crt=123'5599 lcod 123'5597 mlcod 123'5597 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:12.836 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.830+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5831 (115'2809,123'5831] local-lis/les=21/22 n=858 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'5831 lcod 123'5829 mlcod 123'5829 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:12.836 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:12.830+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5508 (115'2504,123'5508] local-lis/les=21/22 n=881 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5508 lcod 123'5506 mlcod 123'5506 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:12.837 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:12.834+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5510 (115'2504,123'5510] local-lis/les=21/22 n=879 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5508 crt=123'5508 lcod 123'5506 mlcod 123'5506 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:12.837 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.834+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5833 (115'2809,123'5833] local-lis/les=21/22 n=856 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'5831 crt=123'5831 lcod 123'5829 mlcod 123'5829 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:12.842 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.838+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5603 (115'2503,123'5603] local-lis/les=21/22 n=862 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5603 lcod 123'5601 mlcod 123'5601 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:12.843 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:12.838+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5512 (115'2504,123'5512] local-lis/les=21/22 n=877 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5512 lcod 123'5510 mlcod 123'5510 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:12.845 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:12.842+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5514 (115'2504,123'5514] local-lis/les=21/22 n=875 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5512 crt=123'5512 lcod 123'5510 mlcod 123'5510 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:12.847 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.842+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5605 (115'2603,123'5605] local-lis/les=21/22 n=860 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5603 crt=123'5603 lcod 123'5601 mlcod 123'5601 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:12.848 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.842+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5835 (115'2809,123'5835] local-lis/les=21/22 n=854 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'5835 lcod 123'5833 mlcod 123'5833 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:12.850 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.846+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5837 (115'2809,123'5837] local-lis/les=21/22 n=852 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'5835 crt=123'5835 lcod 123'5833 mlcod 123'5833 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:12.853 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:12.850+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5516 (115'2504,123'5516] local-lis/les=21/22 n=873 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5516 lcod 123'5514 mlcod 123'5514 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:12.854 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:12.850+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5518 (115'2504,123'5518] local-lis/les=21/22 n=871 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5516 crt=123'5516 lcod 123'5514 mlcod 123'5514 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:12.857 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.850+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5607 (115'2603,123'5607] local-lis/les=21/22 n=858 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5607 lcod 123'5605 mlcod 123'5605 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:12.857 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.854+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5609 (115'2603,123'5609] local-lis/les=21/22 n=856 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5607 crt=123'5607 lcod 123'5605 mlcod 123'5605 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:12.859 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.854+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5839 (115'2809,123'5839] local-lis/les=21/22 n=850 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'5839 lcod 123'5837 mlcod 123'5837 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:12.860 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:12.858+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5520 (115'2504,123'5520] local-lis/les=21/22 n=869 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5520 lcod 123'5518 mlcod 123'5518 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:12.862 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.858+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5841 (115'2809,123'5841] local-lis/les=21/22 n=848 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'5839 crt=123'5839 lcod 123'5837 mlcod 123'5837 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:12.862 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:12.858+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5522 (115'2504,123'5522] local-lis/les=21/22 n=867 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5520 crt=123'5520 lcod 123'5518 mlcod 123'5518 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:12.865 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.862+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5611 (115'2603,123'5611] local-lis/les=21/22 n=854 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5611 lcod 123'5609 mlcod 123'5609 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:12.866 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:12.862+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5524 (115'2504,123'5524] local-lis/les=21/22 n=865 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5524 lcod 123'5522 mlcod 123'5522 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:12.868 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.866+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5613 (115'2603,123'5613] local-lis/les=21/22 n=852 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5611 crt=123'5611 lcod 123'5609 mlcod 123'5609 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:12.869 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.866+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5843 (115'2809,123'5843] local-lis/les=21/22 n=846 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'5843 lcod 123'5841 mlcod 123'5841 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:12.870 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:12.866+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5526 (115'2504,123'5526] local-lis/les=21/22 n=863 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5524 crt=123'5524 lcod 123'5522 mlcod 123'5522 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:12.872 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.870+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5845 (115'2809,123'5845] local-lis/les=21/22 n=844 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'5843 crt=123'5843 lcod 123'5841 mlcod 123'5841 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:12.874 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:12.870+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5528 (115'2504,123'5528] local-lis/les=21/22 n=861 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5528 lcod 123'5526 mlcod 123'5526 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:12.877 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:12.874+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5530 (115'2504,123'5530] local-lis/les=21/22 n=859 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5528 crt=123'5528 lcod 123'5526 mlcod 123'5526 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:12.878 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.874+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5615 (115'2603,123'5615] local-lis/les=21/22 n=850 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5615 lcod 123'5613 mlcod 123'5613 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:12.879 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.874+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5617 (115'2603,123'5617] local-lis/les=21/22 n=848 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5615 crt=123'5615 lcod 123'5613 mlcod 123'5613 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:12.882 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:12.878+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5532 (115'2504,123'5532] local-lis/les=21/22 n=857 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5532 lcod 123'5530 mlcod 123'5530 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:12.883 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.878+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5847 (115'2809,123'5847] local-lis/les=21/22 n=842 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'5847 lcod 123'5845 mlcod 123'5845 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:12.884 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:12.882+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5534 (115'2504,123'5534] local-lis/les=21/22 n=855 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5532 crt=123'5532 lcod 123'5530 mlcod 123'5530 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:12.885 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.882+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5849 (115'2809,123'5849] local-lis/les=21/22 n=840 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'5847 crt=123'5847 lcod 123'5845 mlcod 123'5845 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:12.888 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.886+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5619 (115'2603,123'5619] local-lis/les=21/22 n=846 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5619 lcod 123'5617 mlcod 123'5617 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:12.890 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:12.886+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5536 (115'2504,123'5536] local-lis/les=21/22 n=853 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5536 lcod 123'5534 mlcod 123'5534 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:12.891 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.886+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5621 (115'2603,123'5621] local-lis/les=21/22 n=844 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5619 crt=123'5619 lcod 123'5617 mlcod 123'5617 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:12.891 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:12.886+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5538 (115'2504,123'5538] local-lis/les=21/22 n=851 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5536 crt=123'5536 lcod 123'5534 mlcod 123'5534 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:12.899 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.890+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5851 (115'2809,123'5851] local-lis/les=21/22 n=838 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'5851 lcod 123'5849 mlcod 123'5849 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:12.899 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.894+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5853 (115'2809,123'5853] local-lis/les=21/22 n=836 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'5851 crt=123'5851 lcod 123'5849 mlcod 123'5849 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:12.900 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:12.894+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5540 (115'2504,123'5540] local-lis/les=21/22 n=849 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5540 lcod 123'5538 mlcod 123'5538 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:12.900 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:12.894+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5542 (115'2504,123'5542] local-lis/les=21/22 n=847 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5540 crt=123'5540 lcod 123'5538 mlcod 123'5538 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:12.900 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.898+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5623 (115'2603,123'5623] local-lis/les=21/22 n=842 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5623 lcod 123'5621 mlcod 123'5621 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:12.902 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.898+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5625 (115'2603,123'5625] local-lis/les=21/22 n=840 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5623 crt=123'5623 lcod 123'5621 mlcod 123'5621 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:12.908 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:12.902+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5544 (115'2504,123'5544] local-lis/les=21/22 n=845 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5544 lcod 123'5542 mlcod 123'5542 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:12.908 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:12.902+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5546 (115'2504,123'5546] local-lis/les=21/22 n=843 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5544 crt=123'5544 lcod 123'5542 mlcod 123'5542 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:12.909 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.902+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5855 (115'2809,123'5855] local-lis/les=21/22 n=834 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'5855 lcod 123'5853 mlcod 123'5853 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:12.910 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.906+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5857 (115'2809,123'5857] local-lis/les=21/22 n=832 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'5855 crt=123'5855 lcod 123'5853 mlcod 123'5853 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:12.911 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:12.906+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5548 (115'2504,123'5548] local-lis/les=21/22 n=841 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5548 lcod 123'5546 mlcod 123'5546 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:12.911 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:12.906+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5550 (115'2504,123'5550] local-lis/les=21/22 n=839 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5548 crt=123'5548 lcod 123'5546 mlcod 123'5546 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:12.913 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.910+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5627 (115'2603,123'5627] local-lis/les=21/22 n=838 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5627 lcod 123'5625 mlcod 123'5625 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:12.917 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.914+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5629 (115'2603,123'5629] local-lis/les=21/22 n=836 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5627 crt=123'5627 lcod 123'5625 mlcod 123'5625 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:12.920 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.914+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5859 (115'2809,123'5859] local-lis/les=21/22 n=830 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'5859 lcod 123'5857 mlcod 123'5857 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:12.921 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:12.914+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5552 (115'2504,123'5552] local-lis/les=21/22 n=837 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5552 lcod 123'5550 mlcod 123'5550 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:12.929 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.918+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5861 (115'2809,123'5861] local-lis/les=21/22 n=828 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'5859 crt=123'5859 lcod 123'5857 mlcod 123'5857 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:12.931 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:12.918+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5554 (115'2504,123'5554] local-lis/les=21/22 n=835 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5552 crt=123'5552 lcod 123'5550 mlcod 123'5550 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:12.931 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:12.922+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5556 (115'2504,123'5556] local-lis/les=21/22 n=833 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5556 lcod 123'5554 mlcod 123'5554 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:12.931 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:12.926+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5558 (115'2504,123'5558] local-lis/les=21/22 n=831 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5556 crt=123'5556 lcod 123'5554 mlcod 123'5554 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:12.931 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:12.914+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5328 (114'2304,123'5328] local-lis/les=21/22 n=891 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5328 lcod 123'5326 mlcod 123'5326 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.931 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:12.918+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5330 (114'2304,123'5330] local-lis/les=21/22 n=889 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5328 crt=123'5328 lcod 123'5326 mlcod 123'5326 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.932 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.922+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5631 (115'2603,123'5631] local-lis/les=21/22 n=834 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5631 lcod 123'5629 mlcod 123'5629 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:12.932 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:12.930+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5332 (114'2304,123'5332] local-lis/les=21/22 n=887 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5332 lcod 123'5330 mlcod 123'5330 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.933 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.930+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5863 (115'2809,123'5863] local-lis/les=21/22 n=826 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'5863 lcod 123'5861 mlcod 123'5861 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:12.934 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:12.930+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5334 (114'2304,123'5334] local-lis/les=21/22 n=885 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5332 crt=123'5332 lcod 123'5330 mlcod 123'5330 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.935 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.930+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5865 (115'2809,123'5865] local-lis/les=21/22 n=824 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'5863 crt=123'5863 lcod 123'5861 mlcod 123'5861 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:12.936 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:12.930+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5560 (115'2504,123'5560] local-lis/les=21/22 n=829 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5560 lcod 123'5558 mlcod 123'5558 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:12.938 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:12.934+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5562 (115'2504,123'5562] local-lis/les=21/22 n=827 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5560 crt=123'5560 lcod 123'5558 mlcod 123'5558 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:12.939 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.934+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5633 (115'2603,123'5633] local-lis/les=21/22 n=832 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5633 lcod 123'5631 mlcod 123'5631 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:12.942 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:12.938+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5336 (114'2304,123'5336] local-lis/les=21/22 n=883 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5336 lcod 123'5334 mlcod 123'5334 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.944 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.938+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5635 (115'2603,123'5635] local-lis/les=21/22 n=830 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5633 crt=123'5633 lcod 123'5631 mlcod 123'5631 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:12.944 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:12.942+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5338 (114'2304,123'5338] local-lis/les=21/22 n=881 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5336 crt=123'5336 lcod 123'5334 mlcod 123'5334 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.945 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:12.942+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5564 (115'2504,123'5564] local-lis/les=21/22 n=825 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5564 lcod 123'5562 mlcod 123'5562 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:12.947 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:12.942+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5566 (115'2504,123'5566] local-lis/les=21/22 n=823 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5564 crt=123'5564 lcod 123'5562 mlcod 123'5562 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:12.950 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:12.946+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5340 (114'2304,123'5340] local-lis/les=21/22 n=879 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5340 lcod 123'5338 mlcod 123'5338 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.952 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:12.950+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5342 (114'2304,123'5342] local-lis/les=21/22 n=877 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5340 crt=123'5340 lcod 123'5338 mlcod 123'5338 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:12.952 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.950+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5867 (115'2809,123'5867] local-lis/les=21/22 n=822 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'5867 lcod 123'5865 mlcod 123'5865 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:12.954 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.950+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5869 (115'2809,123'5869] local-lis/les=21/22 n=820 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'5867 crt=123'5867 lcod 123'5865 mlcod 123'5865 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:12.959 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:12.954+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5344 (114'2304,123'5344] local-lis/les=21/22 n=875 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5344 lcod 123'5342 mlcod 123'5342 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:13.038 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:12.981+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5433 (115'2405,123'5433] local-lis/les=21/22 n=1084 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=123'5433 lcod 123'5431 mlcod 123'5431 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:13.038 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:12.985+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5435 (115'2405,123'5435] local-lis/les=21/22 n=1082 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5433 crt=123'5433 lcod 123'5431 mlcod 123'5431 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:13.039 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:12.993+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5437 (115'2405,123'5437] local-lis/les=21/22 n=1080 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=123'5437 lcod 123'5435 mlcod 123'5435 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:13.039 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:12.993+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5439 (115'2405,123'5439] local-lis/les=21/22 n=1078 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5437 crt=123'5437 lcod 123'5435 mlcod 123'5435 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:13.039 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:13.001+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5441 (115'2405,123'5441] local-lis/les=21/22 n=1076 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=123'5441 lcod 123'5439 mlcod 123'5439 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:13.039 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:13.005+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5443 (115'2405,123'5443] local-lis/les=21/22 n=1074 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5441 crt=123'5441 lcod 123'5439 mlcod 123'5439 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:13.039 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:13.013+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5445 (115'2405,123'5445] local-lis/les=21/22 n=1072 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=123'5445 lcod 123'5443 mlcod 123'5443 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:13.039 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:13.013+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5447 (115'2405,123'5447] local-lis/les=21/22 n=1070 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5445 crt=123'5445 lcod 123'5443 mlcod 123'5443 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:13.040 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.954+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5637 (115'2603,123'5637] local-lis/les=21/22 n=828 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5637 lcod 123'5635 mlcod 123'5635 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:13.040 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.954+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5639 (115'2603,123'5639] local-lis/les=21/22 n=826 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5637 crt=123'5637 lcod 123'5635 mlcod 123'5635 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:13.040 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.962+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5871 (115'2809,123'5871] local-lis/les=21/22 n=818 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'5871 lcod 123'5869 mlcod 123'5869 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:13.040 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.962+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5873 (115'2809,123'5873] local-lis/les=21/22 n=816 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'5871 crt=123'5871 lcod 123'5869 mlcod 123'5869 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:13.041 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.966+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5641 (115'2603,123'5641] local-lis/les=21/22 n=824 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5641 lcod 123'5639 mlcod 123'5639 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:13.041 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.970+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5643 (115'2603,123'5643] local-lis/les=21/22 n=822 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5641 crt=123'5641 lcod 123'5639 mlcod 123'5639 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:13.041 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.974+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5875 (115'2809,123'5875] local-lis/les=21/22 n=814 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'5875 lcod 123'5873 mlcod 123'5873 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:13.041 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.978+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5645 (115'2603,123'5645] local-lis/les=21/22 n=820 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5645 lcod 123'5643 mlcod 123'5643 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:13.042 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.982+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5647 (115'2603,123'5647] local-lis/les=21/22 n=818 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5645 crt=123'5645 lcod 123'5643 mlcod 123'5643 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:13.042 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.986+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5877 (115'2809,123'5877] local-lis/les=21/22 n=812 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'5877 lcod 123'5875 mlcod 123'5875 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:13.042 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.990+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5879 (115'2809,123'5879] local-lis/les=21/22 n=810 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'5877 crt=123'5877 lcod 123'5875 mlcod 123'5875 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:13.042 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.990+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5649 (115'2603,123'5649] local-lis/les=21/22 n=816 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5649 lcod 123'5647 mlcod 123'5647 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:13.042 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.994+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5651 (115'2603,123'5651] local-lis/les=21/22 n=814 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5649 crt=123'5649 lcod 123'5647 mlcod 123'5647 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:13.043 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:12.998+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5881 (115'2809,123'5881] local-lis/les=21/22 n=808 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'5881 lcod 123'5879 mlcod 123'5879 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:13.043 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:13.002+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5883 (115'2809,123'5883] local-lis/les=21/22 n=806 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'5881 crt=123'5881 lcod 123'5879 mlcod 123'5879 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:13.043 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:13.006+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5653 (115'2603,123'5653] local-lis/les=21/22 n=812 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5653 lcod 123'5651 mlcod 123'5651 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:13.043 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:13.010+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5655 (115'2603,123'5655] local-lis/les=21/22 n=810 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5653 crt=123'5653 lcod 123'5651 mlcod 123'5651 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:13.043 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:13.014+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5885 (115'2809,123'5885] local-lis/les=21/22 n=804 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'5885 lcod 123'5883 mlcod 123'5883 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:13.044 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:13.014+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5887 (115'2809,123'5887] local-lis/les=21/22 n=802 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'5885 crt=123'5885 lcod 123'5883 mlcod 123'5883 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:13.044 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:12.958+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5346 (114'2304,123'5346] local-lis/les=21/22 n=873 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5344 crt=123'5344 lcod 123'5342 mlcod 123'5342 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:13.044 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:12.962+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5348 (114'2304,123'5348] local-lis/les=21/22 n=871 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5348 lcod 123'5346 mlcod 123'5346 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:13.044 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:12.970+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5350 (114'2304,123'5350] local-lis/les=21/22 n=869 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5348 crt=123'5348 lcod 123'5346 mlcod 123'5346 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:13.044 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:12.974+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5352 (114'2304,123'5352] local-lis/les=21/22 n=867 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5352 lcod 123'5350 mlcod 123'5350 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:13.045 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:12.978+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5354 (114'2304,123'5354] local-lis/les=21/22 n=865 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5352 crt=123'5352 lcod 123'5350 mlcod 123'5350 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:13.045 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:12.986+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5356 (114'2304,123'5356] local-lis/les=21/22 n=863 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5356 lcod 123'5354 mlcod 123'5354 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:13.045 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:12.986+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5358 (114'2304,123'5358] local-lis/les=21/22 n=861 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5356 crt=123'5356 lcod 123'5354 mlcod 123'5354 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:13.045 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:12.994+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5360 (114'2304,123'5360] local-lis/les=21/22 n=859 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5360 lcod 123'5358 mlcod 123'5358 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:13.045 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:13.002+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5362 (114'2304,123'5362] local-lis/les=21/22 n=857 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5362 lcod 123'5360 mlcod 123'5360 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:13.045 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:13.002+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5364 (114'2304,123'5364] local-lis/les=21/22 n=855 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5362 crt=123'5362 lcod 123'5360 mlcod 123'5360 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:13.045 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:13.010+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5366 (114'2304,123'5366] local-lis/les=21/22 n=853 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5366 lcod 123'5364 mlcod 123'5364 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:13.046 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:13.014+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5368 (114'2304,123'5368] local-lis/les=21/22 n=851 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5366 crt=123'5366 lcod 123'5364 mlcod 123'5364 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:13.046 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:13.022+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5370 (114'2304,123'5370] local-lis/les=21/22 n=849 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5370 lcod 123'5368 mlcod 123'5368 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:13.046 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:13.022+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5372 (114'2304,123'5372] local-lis/les=21/22 n=847 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5370 crt=123'5370 lcod 123'5368 mlcod 123'5368 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:13.046 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:13.022+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5351 (115'2306,123'5351] local-lis/les=21/22 n=1044 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5351 lcod 123'5349 mlcod 123'5349 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:13.046 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:13.026+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5568 (115'2504,123'5568] local-lis/les=21/22 n=821 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5568 lcod 123'5566 mlcod 123'5566 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:13.046 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:13.026+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5353 (115'2306,123'5353] local-lis/les=21/22 n=1042 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5351 crt=123'5351 lcod 123'5349 mlcod 123'5349 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:13.047 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:13.026+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5570 (115'2504,123'5570] local-lis/les=21/22 n=819 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5568 crt=123'5568 lcod 123'5566 mlcod 123'5566 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:13.048 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:13.030+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5374 (114'2304,123'5374] local-lis/les=21/22 n=845 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5374 lcod 123'5372 mlcod 123'5372 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:13.048 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:13.030+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5376 (114'2304,123'5376] local-lis/les=21/22 n=843 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5374 crt=123'5374 lcod 123'5372 mlcod 123'5372 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:13.048 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:13.042+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5378 (114'2304,123'5378] local-lis/les=21/22 n=841 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5378 lcod 123'5376 mlcod 123'5376 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:13.049 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:13.030+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5657 (115'2603,123'5657] local-lis/les=21/22 n=808 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5657 lcod 123'5655 mlcod 123'5655 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:13.049 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:13.030+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5659 (115'2603,123'5659] local-lis/les=21/22 n=806 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5657 crt=123'5657 lcod 123'5655 mlcod 123'5655 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:13.049 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:13.038+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5889 (115'2809,123'5889] local-lis/les=21/22 n=800 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'5889 lcod 123'5887 mlcod 123'5887 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:13.049 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:13.042+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5891 (115'2809,123'5891] local-lis/les=21/22 n=798 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'5889 crt=123'5889 lcod 123'5887 mlcod 123'5887 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:13.049 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:13.034+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5355 (115'2306,123'5355] local-lis/les=21/22 n=1040 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5355 lcod 123'5353 mlcod 123'5353 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:13.049 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:13.038+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5357 (115'2306,123'5357] local-lis/les=21/22 n=1038 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5355 crt=123'5355 lcod 123'5353 mlcod 123'5353 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:13.050 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:13.038+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5572 (115'2504,123'5572] local-lis/les=21/22 n=817 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5572 lcod 123'5570 mlcod 123'5570 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:13.050 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:13.042+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5574 (115'2504,123'5574] local-lis/les=21/22 n=815 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5572 crt=123'5572 lcod 123'5570 mlcod 123'5570 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:13.050 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:13.046+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5380 (114'2304,123'5380] local-lis/les=21/22 n=839 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5378 crt=123'5378 lcod 123'5376 mlcod 123'5376 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:13.051 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:13.046+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5661 (115'2603,123'5661] local-lis/les=21/22 n=804 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5661 lcod 123'5659 mlcod 123'5659 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:13.052 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:13.050+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5576 (115'2504,123'5576] local-lis/les=21/22 n=813 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5576 lcod 123'5574 mlcod 123'5574 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:13.052 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:13.050+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5359 (115'2306,123'5359] local-lis/les=21/22 n=1036 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5359 lcod 123'5357 mlcod 123'5357 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:13.054 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:13.050+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5578 (115'2504,123'5578] local-lis/les=21/22 n=811 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5576 crt=123'5576 lcod 123'5574 mlcod 123'5574 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:13.056 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:13.050+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5663 (115'2603,123'5663] local-lis/les=21/22 n=802 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5661 crt=123'5661 lcod 123'5659 mlcod 123'5659 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:13.057 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:13.054+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5893 (115'2809,123'5893] local-lis/les=21/22 n=796 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'5893 lcod 123'5891 mlcod 123'5891 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:13.059 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:13.054+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5361 (115'2306,123'5361] local-lis/les=21/22 n=1034 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5359 crt=123'5359 lcod 123'5357 mlcod 123'5357 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:13.060 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:13.058+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5895 (115'2809,123'5895] local-lis/les=21/22 n=794 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'5893 crt=123'5893 lcod 123'5891 mlcod 123'5891 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:13.063 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:13.058+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5665 (115'2603,123'5665] local-lis/les=21/22 n=800 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5665 lcod 123'5663 mlcod 123'5663 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:13.066 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:13.062+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5667 (115'2603,123'5667] local-lis/les=21/22 n=798 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5665 crt=123'5665 lcod 123'5663 mlcod 123'5663 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:13.066 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:13.062+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5580 (115'2504,123'5580] local-lis/les=21/22 n=809 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5580 lcod 123'5578 mlcod 123'5578 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:13.067 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:13.062+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5582 (115'2504,123'5582] local-lis/les=21/22 n=807 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5580 crt=123'5580 lcod 123'5578 mlcod 123'5578 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:13.069 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:13.062+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5382 (114'2304,123'5382] local-lis/les=21/22 n=837 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5382 lcod 123'5380 mlcod 123'5380 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:13.070 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:13.066+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5897 (115'2809,123'5897] local-lis/les=21/22 n=792 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'5897 lcod 123'5895 mlcod 123'5895 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:13.071 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:13.066+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5363 (115'2306,123'5363] local-lis/les=21/22 n=1032 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5363 lcod 123'5361 mlcod 123'5361 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:13.071 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:13.066+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5365 (115'2306,123'5365] local-lis/les=21/22 n=1030 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5363 crt=123'5363 lcod 123'5361 mlcod 123'5361 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:13.072 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:13.070+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5384 (114'2304,123'5384] local-lis/les=21/22 n=835 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5382 crt=123'5382 lcod 123'5380 mlcod 123'5380 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:13.072 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:13.070+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5899 (115'2809,123'5899] local-lis/les=21/22 n=790 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'5897 crt=123'5897 lcod 123'5895 mlcod 123'5895 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:13.078 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:13.074+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5669 (115'2603,123'5669] local-lis/les=21/22 n=796 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5669 lcod 123'5667 mlcod 123'5667 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:13.081 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:13.078+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5386 (114'2304,123'5386] local-lis/les=21/22 n=833 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5386 lcod 123'5384 mlcod 123'5384 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:13.081 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:13.078+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5671 (115'2603,123'5671] local-lis/les=21/22 n=794 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5669 crt=123'5669 lcod 123'5667 mlcod 123'5667 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:13.082 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:13.078+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5388 (114'2304,123'5388] local-lis/les=21/22 n=831 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5386 crt=123'5386 lcod 123'5384 mlcod 123'5384 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:13.085 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:13.078+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5584 (115'2504,123'5584] local-lis/les=21/22 n=805 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5584 lcod 123'5582 mlcod 123'5582 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:13.086 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:13.082+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5586 (115'2504,123'5586] local-lis/les=21/22 n=803 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5584 crt=123'5584 lcod 123'5582 mlcod 123'5582 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:13.086 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:13.082+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5367 (115'2306,123'5367] local-lis/les=21/22 n=1028 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5367 lcod 123'5365 mlcod 123'5365 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:13.090 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:13.086+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5369 (115'2306,123'5369] local-lis/les=21/22 n=1026 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5367 crt=123'5367 lcod 123'5365 mlcod 123'5365 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:13.090 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:13.086+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5901 (115'2809,123'5901] local-lis/les=21/22 n=788 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'5901 lcod 123'5899 mlcod 123'5899 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:13.093 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:13.086+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5390 (114'2304,123'5390] local-lis/les=21/22 n=829 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5390 lcod 123'5388 mlcod 123'5388 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:13.093 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:13.090+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5392 (114'2304,123'5392] local-lis/les=21/22 n=827 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5390 crt=123'5390 lcod 123'5388 mlcod 123'5388 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:13.094 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:13.090+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5588 (115'2504,123'5588] local-lis/les=21/22 n=801 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5588 lcod 123'5586 mlcod 123'5586 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:13.095 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:13.090+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5903 (115'2809,123'5903] local-lis/les=21/22 n=786 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'5901 crt=123'5901 lcod 123'5899 mlcod 123'5899 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:13.097 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:13.094+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5673 (115'2603,123'5673] local-lis/les=21/22 n=792 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5673 lcod 123'5671 mlcod 123'5671 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:13.100 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:13.094+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5590 (115'2504,123'5590] local-lis/les=21/22 n=799 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5588 crt=123'5588 lcod 123'5586 mlcod 123'5586 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:13.100 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:13.094+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5675 (115'2603,123'5675] local-lis/les=21/22 n=790 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5673 crt=123'5673 lcod 123'5671 mlcod 123'5671 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:13.100 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:13.098+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5371 (115'2306,123'5371] local-lis/les=21/22 n=1024 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5371 lcod 123'5369 mlcod 123'5369 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:13.104 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:13.098+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5905 (115'2809,123'5905] local-lis/les=21/22 n=784 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'5905 lcod 123'5903 mlcod 123'5903 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:13.106 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:13.102+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5907 (115'2809,123'5907] local-lis/les=21/22 n=782 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'5905 crt=123'5905 lcod 123'5903 mlcod 123'5903 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:13.108 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:13.102+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5394 (114'2304,123'5394] local-lis/les=21/22 n=825 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5394 lcod 123'5392 mlcod 123'5392 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:13.108 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:13.102+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5592 (115'2504,123'5592] local-lis/les=21/22 n=797 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5592 lcod 123'5590 mlcod 123'5590 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:13.109 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:13.106+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5594 (115'2504,123'5594] local-lis/les=21/22 n=795 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5592 crt=123'5592 lcod 123'5590 mlcod 123'5590 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:13.110 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:13.106+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5373 (115'2306,123'5373] local-lis/les=21/22 n=1022 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5371 crt=123'5371 lcod 123'5369 mlcod 123'5369 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:13.110 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:13.106+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5396 (114'2304,123'5396] local-lis/les=21/22 n=823 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5394 crt=123'5394 lcod 123'5392 mlcod 123'5392 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:13.113 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:13.110+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5677 (115'2603,123'5677] local-lis/les=21/22 n=788 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5677 lcod 123'5675 mlcod 123'5675 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:13.116 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:13.110+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5679 (115'2603,123'5679] local-lis/les=21/22 n=786 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5677 crt=123'5677 lcod 123'5675 mlcod 123'5675 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:13.117 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:13.114+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5398 (114'2304,123'5398] local-lis/les=21/22 n=821 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5398 lcod 123'5396 mlcod 123'5396 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:13.120 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:13.118+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5400 (114'2304,123'5400] local-lis/les=21/22 n=819 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5398 crt=123'5398 lcod 123'5396 mlcod 123'5396 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:13.121 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:13.118+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5909 (115'2809,123'5909] local-lis/les=21/22 n=780 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'5909 lcod 123'5907 mlcod 123'5907 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:13.122 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:13.114+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5375 (115'2306,123'5375] local-lis/les=21/22 n=1020 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5375 lcod 123'5373 mlcod 123'5373 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:13.123 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:13.114+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5377 (115'2306,123'5377] local-lis/les=21/22 n=1018 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5375 crt=123'5375 lcod 123'5373 mlcod 123'5373 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:13.131 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:13.122+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5911 (115'2909,123'5911] local-lis/les=21/22 n=778 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'5909 crt=123'5909 lcod 123'5907 mlcod 123'5907 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:13.132 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:13.126+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5681 (115'2603,123'5681] local-lis/les=21/22 n=784 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5681 lcod 123'5679 mlcod 123'5679 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:13.132 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:13.122+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5379 (115'2306,123'5379] local-lis/les=21/22 n=1016 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5379 lcod 123'5377 mlcod 123'5377 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:13.132 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:13.126+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5381 (115'2306,123'5381] local-lis/les=21/22 n=1014 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5379 crt=123'5379 lcod 123'5377 mlcod 123'5377 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:13.132 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:13.126+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5402 (114'2304,123'5402] local-lis/les=21/22 n=817 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5402 lcod 123'5400 mlcod 123'5400 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:13.134 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:13.130+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5683 (115'2603,123'5683] local-lis/les=21/22 n=782 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5681 crt=123'5681 lcod 123'5679 mlcod 123'5679 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:13.134 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:13.130+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5404 (114'2304,123'5404] local-lis/les=21/22 n=815 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5402 crt=123'5402 lcod 123'5400 mlcod 123'5400 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:13.139 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:13.134+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5383 (115'2306,123'5383] local-lis/les=21/22 n=1012 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5383 lcod 123'5381 mlcod 123'5381 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:13.140 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:13.134+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5913 (115'2909,123'5913] local-lis/les=21/22 n=776 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'5913 lcod 123'5911 mlcod 123'5911 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:13.142 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:13.138+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5385 (115'2306,123'5385] local-lis/les=21/22 n=1010 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5383 crt=123'5383 lcod 123'5381 mlcod 123'5381 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:13.143 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:13.138+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5915 (115'2909,123'5915] local-lis/les=21/22 n=774 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'5913 crt=123'5913 lcod 123'5911 mlcod 123'5911 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:13.149 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:13.146+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5406 (115'2404,123'5406] local-lis/les=21/22 n=813 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5406 lcod 123'5404 mlcod 123'5404 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:13.149 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:13.146+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5387 (115'2306,123'5387] local-lis/les=21/22 n=1008 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5387 lcod 123'5385 mlcod 123'5385 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:13.151 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:13.146+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5408 (115'2404,123'5408] local-lis/les=21/22 n=811 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5406 crt=123'5406 lcod 123'5404 mlcod 123'5404 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:13.152 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:13.150+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5389 (115'2306,123'5389] local-lis/les=21/22 n=1006 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5387 crt=123'5387 lcod 123'5385 mlcod 123'5385 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:13.155 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:13.150+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5685 (115'2603,123'5685] local-lis/les=21/22 n=780 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5685 lcod 123'5683 mlcod 123'5683 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:13.157 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:13.154+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5687 (115'2603,123'5687] local-lis/les=21/22 n=778 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5685 crt=123'5685 lcod 123'5683 mlcod 123'5683 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:13.159 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:13.154+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5391 (115'2306,123'5391] local-lis/les=21/22 n=1004 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5391 lcod 123'5389 mlcod 123'5389 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:13.162 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:13.158+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5393 (115'2306,123'5393] local-lis/les=21/22 n=1002 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5391 crt=123'5391 lcod 123'5389 mlcod 123'5389 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:13.164 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:13.162+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5917 (115'2909,123'5917] local-lis/les=21/22 n=772 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'5917 lcod 123'5915 mlcod 123'5915 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:13.165 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:13.162+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5410 (115'2404,123'5410] local-lis/les=21/22 n=809 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5410 lcod 123'5408 mlcod 123'5408 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:13.168 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:13.162+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5919 (115'2909,123'5919] local-lis/les=21/22 n=770 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'5917 crt=123'5917 lcod 123'5915 mlcod 123'5915 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:13.168 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:13.166+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5412 (115'2404,123'5412] local-lis/les=21/22 n=807 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5410 crt=123'5410 lcod 123'5408 mlcod 123'5408 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:13.173 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:13.170+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5689 (115'2603,123'5689] local-lis/les=21/22 n=776 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5689 lcod 123'5687 mlcod 123'5687 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:13.175 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:13.170+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5395 (115'2306,123'5395] local-lis/les=21/22 n=1000 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5395 lcod 123'5393 mlcod 123'5393 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:13.176 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:13.174+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5691 (115'2603,123'5691] local-lis/les=21/22 n=774 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5689 crt=123'5689 lcod 123'5687 mlcod 123'5687 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:13.196 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:13.174+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5414 (115'2404,123'5414] local-lis/les=21/22 n=805 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5414 lcod 123'5412 mlcod 123'5412 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:13.199 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:13.178+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5397 (115'2306,123'5397] local-lis/les=21/22 n=998 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5395 crt=123'5395 lcod 123'5393 mlcod 123'5393 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:13.199 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:13.186+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5399 (115'2306,123'5399] local-lis/les=21/22 n=996 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5399 lcod 123'5397 mlcod 123'5397 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:13.199 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:13.186+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5401 (115'2306,123'5401] local-lis/les=21/22 n=994 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5399 crt=123'5399 lcod 123'5397 mlcod 123'5397 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:13.200 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:13.182+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5921 (115'2909,123'5921] local-lis/les=21/22 n=768 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'5921 lcod 123'5919 mlcod 123'5919 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:13.200 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:13.182+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5923 (115'2909,123'5923] local-lis/les=21/22 n=766 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'5921 crt=123'5921 lcod 123'5919 mlcod 123'5919 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:13.200 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:13.190+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5693 (115'2603,123'5693] local-lis/les=21/22 n=772 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5693 lcod 123'5691 mlcod 123'5691 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:13.200 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:13.190+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5695 (115'2603,123'5695] local-lis/les=21/22 n=770 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5693 crt=123'5693 lcod 123'5691 mlcod 123'5691 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:13.202 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:13.194+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5416 (115'2404,123'5416] local-lis/les=21/22 n=803 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5416 lcod 123'5414 mlcod 123'5414 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:13.202 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:13.194+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5403 (115'2306,123'5403] local-lis/les=21/22 n=992 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5403 lcod 123'5401 mlcod 123'5401 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:13.202 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:13.198+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5405 (115'2306,123'5405] local-lis/les=21/22 n=990 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5403 crt=123'5403 lcod 123'5401 mlcod 123'5401 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:13.204 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:13.198+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5418 (115'2404,123'5418] local-lis/les=21/22 n=801 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5416 crt=123'5416 lcod 123'5414 mlcod 123'5414 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:13.207 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:13.202+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5407 (115'2306,123'5407] local-lis/les=21/22 n=988 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5407 lcod 123'5405 mlcod 123'5405 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:13.209 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:13.206+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5409 (115'2407,123'5409] local-lis/les=21/22 n=986 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5407 crt=123'5407 lcod 123'5405 mlcod 123'5405 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:13.216 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:13.210+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5411 (115'2407,123'5411] local-lis/les=21/22 n=984 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5411 lcod 123'5409 mlcod 123'5409 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:13.216 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:13.214+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5413 (115'2407,123'5413] local-lis/les=21/22 n=982 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5411 crt=123'5411 lcod 123'5409 mlcod 123'5409 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:13.222 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:13.218+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5415 (115'2407,123'5415] local-lis/les=21/22 n=980 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5415 lcod 123'5413 mlcod 123'5413 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:13.223 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:13.218+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5417 (115'2407,123'5417] local-lis/les=21/22 n=978 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5415 crt=123'5415 lcod 123'5413 mlcod 123'5413 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:13.226 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:13.222+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5419 (115'2407,123'5419] local-lis/les=21/22 n=976 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5419 lcod 123'5417 mlcod 123'5417 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:13.229 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:13.226+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5421 (115'2407,123'5421] local-lis/les=21/22 n=974 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5419 crt=123'5419 lcod 123'5417 mlcod 123'5417 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:13.232 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:13.230+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5423 (115'2407,123'5423] local-lis/les=21/22 n=972 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5423 lcod 123'5421 mlcod 123'5421 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:13.235 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:13.230+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5425 (115'2407,123'5425] local-lis/les=21/22 n=970 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5423 crt=123'5423 lcod 123'5421 mlcod 123'5421 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:13.239 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:13.234+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5427 (115'2407,123'5427] local-lis/les=21/22 n=968 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5427 lcod 123'5425 mlcod 123'5425 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:13.240 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:13.238+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5429 (115'2407,123'5429] local-lis/les=21/22 n=966 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5427 crt=123'5427 lcod 123'5425 mlcod 123'5425 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:13.245 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:13.242+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5431 (115'2407,123'5431] local-lis/les=21/22 n=964 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5431 lcod 123'5429 mlcod 123'5429 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:13.247 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:13.242+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5433 (115'2407,123'5433] local-lis/les=21/22 n=962 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5431 crt=123'5431 lcod 123'5429 mlcod 123'5429 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:13.250 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:13.246+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5435 (115'2407,123'5435] local-lis/les=21/22 n=960 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5435 lcod 123'5433 mlcod 123'5433 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:13.251 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:13.246+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5437 (115'2407,123'5437] local-lis/les=21/22 n=958 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5435 crt=123'5435 lcod 123'5433 mlcod 123'5433 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:13.284 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:13.278+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5925 (115'2909,123'5925] local-lis/les=21/22 n=764 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'5925 lcod 123'5923 mlcod 123'5923 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:13.286 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:13.282+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5927 (115'2909,123'5927] local-lis/les=21/22 n=762 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'5925 crt=123'5925 lcod 123'5923 mlcod 123'5923 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:13.301 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:13.290+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5697 (115'2603,123'5697] local-lis/les=21/22 n=768 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5697 lcod 123'5695 mlcod 123'5695 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:13.301 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:13.294+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5699 (115'2603,123'5699] local-lis/les=21/22 n=766 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5697 crt=123'5697 lcod 123'5695 mlcod 123'5695 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:13.301 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:13.298+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5929 (115'2909,123'5929] local-lis/les=21/22 n=760 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'5929 lcod 123'5927 mlcod 123'5927 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:13.303 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:13.298+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5931 (115'2909,123'5931] local-lis/les=21/22 n=758 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'5929 crt=123'5929 lcod 123'5927 mlcod 123'5927 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:13.377 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:13.374+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5596 (115'2504,123'5596] local-lis/les=21/22 n=793 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5596 lcod 123'5594 mlcod 123'5594 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:13.381 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:13.378+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5598 (115'2504,123'5598] local-lis/les=21/22 n=791 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5596 crt=123'5596 lcod 123'5594 mlcod 123'5594 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:13.437 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:13.434+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5589 (115'2511,123'5589] local-lis/les=21/22 n=1071 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'5589 lcod 123'5587 mlcod 123'5587 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:13.440 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:13.438+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5591 (115'2511,123'5591] local-lis/les=21/22 n=1069 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'5589 crt=123'5589 lcod 123'5587 mlcod 123'5587 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:13.459 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:13.445+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5450 (115'2405,123'5450] local-lis/les=21/22 n=1067 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5449 crt=123'5447 lcod 123'5447 mlcod 123'5447 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:13.459 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:13.449+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5452 (115'2405,123'5452] local-lis/les=21/22 n=1065 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5449 crt=123'5449 lcod 123'5447 mlcod 123'5447 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:13.462 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:13.458+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5420 (115'2404,123'5420] local-lis/les=21/22 n=799 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5420 lcod 123'5418 mlcod 123'5418 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:13.463 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:13.458+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5934 (115'2909,123'5934] local-lis/les=21/22 n=755 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'5933 crt=123'5929 lcod 123'5931 mlcod 123'5931 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:13.463 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:13.458+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5702 (115'2603,123'5702] local-lis/les=21/22 n=763 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5701 crt=123'5699 lcod 123'5699 mlcod 123'5699 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:13.465 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:13.462+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5422 (115'2404,123'5422] local-lis/les=21/22 n=797 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5420 crt=123'5420 lcod 123'5418 mlcod 123'5418 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:13.465 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:13.462+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5936 (115'2909,123'5936] local-lis/les=21/22 n=753 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'5933 crt=123'5933 lcod 123'5931 mlcod 123'5931 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:13.469 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:13.461+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5454 (115'2405,123'5454] local-lis/les=21/22 n=1063 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=123'5454 lcod 123'5452 mlcod 123'5452 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:13.471 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:13.465+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5456 (115'2405,123'5456] local-lis/les=21/22 n=1061 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5454 crt=123'5454 lcod 123'5452 mlcod 123'5452 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:13.472 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:13.470+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5704 (115'2603,123'5704] local-lis/les=21/22 n=761 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5701 crt=123'5701 lcod 123'5699 mlcod 123'5699 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:13.475 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:13.470+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5424 (115'2404,123'5424] local-lis/les=21/22 n=795 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5424 lcod 123'5422 mlcod 123'5422 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:13.480 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:13.474+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5426 (115'2404,123'5426] local-lis/les=21/22 n=793 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5424 crt=123'5424 lcod 123'5422 mlcod 123'5422 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:13.481 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:13.473+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5363 (115'2306,123'5363] local-lis/les=21/22 n=906 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5362 crt=123'5360 lcod 123'5360 mlcod 123'5360 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:13.482 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:13.478+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5938 (115'2909,123'5938] local-lis/les=21/22 n=751 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'5938 lcod 123'5936 mlcod 123'5936 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:13.483 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:13.477+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5365 (115'2306,123'5365] local-lis/les=21/22 n=904 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5362 crt=123'5362 lcod 123'5360 mlcod 123'5360 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:13.484 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:13.478+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5707 (115'2704,123'5707] local-lis/les=21/22 n=758 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5706 crt=123'5702 lcod 123'5704 mlcod 123'5704 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:13.487 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:13.482+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5940 (115'2909,123'5940] local-lis/les=21/22 n=749 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'5938 crt=123'5938 lcod 123'5936 mlcod 123'5936 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:13.487 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:13.482+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5709 (115'2704,123'5709] local-lis/les=21/22 n=756 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5706 crt=123'5706 lcod 123'5704 mlcod 123'5704 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:13.490 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:13.486+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5428 (115'2404,123'5428] local-lis/les=21/22 n=791 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5428 lcod 123'5426 mlcod 123'5426 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:13.491 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:13.486+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5601 (115'2504,123'5601] local-lis/les=21/22 n=788 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5600 crt=123'5598 lcod 123'5598 mlcod 123'5598 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:13.492 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:13.490+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5603 (115'2504,123'5603] local-lis/les=21/22 n=786 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5600 crt=123'5600 lcod 123'5598 mlcod 123'5598 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:13.492 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:13.490+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5430 (115'2404,123'5430] local-lis/les=21/22 n=789 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5428 crt=123'5428 lcod 123'5426 mlcod 123'5426 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:13.496 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:13.494+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5942 (115'2909,123'5942] local-lis/les=21/22 n=747 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'5942 lcod 123'5940 mlcod 123'5940 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:13.500 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:13.494+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5712 (115'2704,123'5712] local-lis/les=21/22 n=753 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5711 crt=123'5707 lcod 123'5709 mlcod 123'5709 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:13.501 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:13.498+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5944 (115'2909,123'5944] local-lis/les=21/22 n=745 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'5942 crt=123'5942 lcod 123'5940 mlcod 123'5940 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:13.501 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:13.494+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5440 (115'2407,123'5440] local-lis/les=21/22 n=955 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5439 crt=123'5437 lcod 123'5437 mlcod 123'5437 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:13.501 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:13.498+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5442 (115'2407,123'5442] local-lis/les=21/22 n=953 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5439 crt=123'5439 lcod 123'5437 mlcod 123'5437 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:13.503 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:13.498+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5432 (115'2404,123'5432] local-lis/les=21/22 n=787 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5432 lcod 123'5430 mlcod 123'5430 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:13.506 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:13.502+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5434 (115'2404,123'5434] local-lis/les=21/22 n=785 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5432 crt=123'5432 lcod 123'5430 mlcod 123'5430 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:13.507 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:13.502+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5445 (115'2407,123'5445] local-lis/les=21/22 n=950 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5444 crt=123'5440 lcod 123'5442 mlcod 123'5442 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:13.510 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:13.506+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5447 (115'2407,123'5447] local-lis/les=21/22 n=948 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5444 crt=123'5444 lcod 123'5442 mlcod 123'5442 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:13.511 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:13.506+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5437 (115'2404,123'5437] local-lis/les=21/22 n=782 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5436 crt=123'5432 lcod 123'5434 mlcod 123'5434 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:13.515 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:13.510+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5439 (115'2404,123'5439] local-lis/les=21/22 n=780 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5436 crt=123'5436 lcod 123'5434 mlcod 123'5434 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:13.515 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:13.509+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5368 (115'2306,123'5368] local-lis/les=21/22 n=901 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5367 crt=123'5365 lcod 123'5365 mlcod 123'5365 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:13.517 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:13.509+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5370 (115'2306,123'5370] local-lis/les=21/22 n=899 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5367 crt=123'5367 lcod 123'5365 mlcod 123'5365 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:13.521 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:13.513+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5373 (115'2306,123'5373] local-lis/les=21/22 n=896 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5372 crt=123'5367 lcod 123'5370 mlcod 123'5370 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:13.524 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:13.517+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5375 (115'2306,123'5375] local-lis/les=21/22 n=894 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5372 crt=123'5372 lcod 123'5370 mlcod 123'5370 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:13.526 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:13.522+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5441 (115'2404,123'5441] local-lis/les=21/22 n=778 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5441 lcod 123'5439 mlcod 123'5439 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:13.527 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:13.522+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5715 (115'2704,123'5715] local-lis/les=21/22 n=750 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5714 crt=123'5712 lcod 123'5712 mlcod 123'5712 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:13.529 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:13.526+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5443 (115'2404,123'5443] local-lis/les=21/22 n=776 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5441 crt=123'5441 lcod 123'5439 mlcod 123'5439 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:13.531 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:13.526+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5717 (115'2704,123'5717] local-lis/les=21/22 n=748 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5714 crt=123'5714 lcod 123'5712 mlcod 123'5712 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:13.532 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:13.525+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5459 (115'2405,123'5459] local-lis/les=21/22 n=1058 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5458 crt=123'5456 lcod 123'5456 mlcod 123'5456 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:13.533 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:13.530+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5946 (115'2909,123'5946] local-lis/les=21/22 n=743 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'5946 lcod 123'5944 mlcod 123'5944 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:13.536 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:13.534+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5948 (115'2909,123'5948] local-lis/les=21/22 n=741 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'5946 crt=123'5946 lcod 123'5944 mlcod 123'5944 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:13.536 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:13.529+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5461 (115'2405,123'5461] local-lis/les=21/22 n=1056 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5458 crt=123'5458 lcod 123'5456 mlcod 123'5456 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:13.539 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:13.534+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5606 (115'2605,123'5606] local-lis/les=21/22 n=783 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5605 crt=123'5603 lcod 123'5603 mlcod 123'5603 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:13.541 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:13.538+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5608 (115'2605,123'5608] local-lis/les=21/22 n=781 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5605 crt=123'5605 lcod 123'5603 mlcod 123'5603 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:13.543 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:13.537+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5378 (115'2306,123'5378] local-lis/les=21/22 n=891 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5377 crt=123'5375 lcod 123'5375 mlcod 123'5375 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:13.546 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:13.537+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5380 (115'2306,123'5380] local-lis/les=21/22 n=889 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5377 crt=123'5377 lcod 123'5375 mlcod 123'5375 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:13.548 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:13.546+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5951 (115'2909,123'5951] local-lis/les=21/22 n=738 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'5950 crt=123'5948 lcod 123'5948 mlcod 123'5948 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:13.550 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:13.546+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5953 (115'2909,123'5953] local-lis/les=21/22 n=736 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'5950 crt=123'5950 lcod 123'5948 mlcod 123'5948 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:13.554 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:13.550+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5719 (115'2704,123'5719] local-lis/les=21/22 n=746 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5719 lcod 123'5717 mlcod 123'5717 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:13.555 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:13.550+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5445 (115'2404,123'5445] local-lis/les=21/22 n=774 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5445 lcod 123'5443 mlcod 123'5443 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:13.556 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:13.550+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5956 (115'2909,123'5956] local-lis/les=21/22 n=733 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'5955 crt=123'5950 lcod 123'5953 mlcod 123'5953 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:13.557 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:13.554+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5447 (115'2404,123'5447] local-lis/les=21/22 n=772 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5445 crt=123'5445 lcod 123'5443 mlcod 123'5443 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:13.558 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:13.554+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5958 (115'2909,123'5958] local-lis/les=21/22 n=731 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'5955 crt=123'5955 lcod 123'5953 mlcod 123'5953 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:13.558 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:13.553+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5383 (115'2306,123'5383] local-lis/les=21/22 n=886 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5382 crt=123'5380 lcod 123'5380 mlcod 123'5380 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:13.563 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:13.557+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5385 (115'2306,123'5385] local-lis/les=21/22 n=884 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5382 crt=123'5382 lcod 123'5380 mlcod 123'5380 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:13.563 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:13.558+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5721 (115'2704,123'5721] local-lis/les=21/22 n=744 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5719 crt=123'5719 lcod 123'5717 mlcod 123'5717 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:13.566 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:13.562+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5611 (115'2605,123'5611] local-lis/les=21/22 n=778 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5610 crt=123'5608 lcod 123'5608 mlcod 123'5608 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:13.568 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:13.566+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5613 (115'2605,123'5613] local-lis/les=21/22 n=776 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5610 crt=123'5610 lcod 123'5608 mlcod 123'5608 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:13.569 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:13.566+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5450 (115'2404,123'5450] local-lis/les=21/22 n=769 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5449 crt=123'5447 lcod 123'5447 mlcod 123'5447 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:13.571 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:13.566+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5452 (115'2404,123'5452] local-lis/les=21/22 n=767 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5449 crt=123'5449 lcod 123'5447 mlcod 123'5447 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:13.574 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:13.565+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5388 (115'2306,123'5388] local-lis/les=21/22 n=881 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5387 crt=123'5385 lcod 123'5385 mlcod 123'5385 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:13.576 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:13.569+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5390 (115'2306,123'5390] local-lis/les=21/22 n=879 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5387 crt=123'5387 lcod 123'5385 mlcod 123'5385 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:13.579 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:13.574+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5960 (115'2909,123'5960] local-lis/les=21/22 n=729 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'5960 lcod 123'5958 mlcod 123'5958 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:13.579 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:13.574+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5724 (115'2704,123'5724] local-lis/les=21/22 n=741 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5723 crt=123'5721 lcod 123'5721 mlcod 123'5721 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:13.581 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:13.578+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5962 (115'2909,123'5962] local-lis/les=21/22 n=727 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'5960 crt=123'5960 lcod 123'5958 mlcod 123'5958 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:13.581 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:13.578+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5726 (115'2704,123'5726] local-lis/les=21/22 n=739 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5723 crt=123'5723 lcod 123'5721 mlcod 123'5721 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:13.587 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:13.582+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5454 (115'2404,123'5454] local-lis/les=21/22 n=765 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5454 lcod 123'5452 mlcod 123'5452 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:13.589 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:13.586+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5456 (115'2404,123'5456] local-lis/les=21/22 n=763 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5454 crt=123'5454 lcod 123'5452 mlcod 123'5452 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:13.613 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:13.605+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5392 (115'2306,123'5392] local-lis/les=21/22 n=877 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5392 lcod 123'5390 mlcod 123'5390 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:13.615 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:13.609+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5394 (115'2306,123'5394] local-lis/les=21/22 n=875 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5392 crt=123'5392 lcod 123'5390 mlcod 123'5390 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:13.621 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:13.613+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5396 (115'2306,123'5396] local-lis/les=21/22 n=873 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5396 lcod 123'5394 mlcod 123'5394 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:13.622 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:13.617+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5398 (115'2306,123'5398] local-lis/les=21/22 n=871 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5396 crt=123'5396 lcod 123'5394 mlcod 123'5394 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:13.628 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:13.621+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5400 (115'2306,123'5400] local-lis/les=21/22 n=869 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5400 lcod 123'5398 mlcod 123'5398 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:13.630 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:13.621+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5402 (115'2306,123'5402] local-lis/les=21/22 n=867 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5400 crt=123'5400 lcod 123'5398 mlcod 123'5398 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:13.631 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:13.626+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5593 (115'2511,123'5593] local-lis/les=21/22 n=1067 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'5593 lcod 123'5591 mlcod 123'5591 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:13.633 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:13.630+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5595 (115'2511,123'5595] local-lis/les=21/22 n=1065 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'5593 crt=123'5593 lcod 123'5591 mlcod 123'5591 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:13.636 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:13.629+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5404 (115'2306,123'5404] local-lis/les=21/22 n=865 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5404 lcod 123'5402 mlcod 123'5402 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:13.637 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:13.629+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5406 (115'2306,123'5406] local-lis/les=21/22 n=863 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5404 crt=123'5404 lcod 123'5402 mlcod 123'5402 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:13.638 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:13.634+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5597 (115'2511,123'5597] local-lis/les=21/22 n=1063 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'5597 lcod 123'5595 mlcod 123'5595 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:13.640 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:13.638+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5599 (115'2511,123'5599] local-lis/les=21/22 n=1061 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'5597 crt=123'5597 lcod 123'5595 mlcod 123'5595 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:13.645 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:13.637+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5408 (115'2406,123'5408] local-lis/les=21/22 n=861 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5408 lcod 123'5406 mlcod 123'5406 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:13.646 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:13.641+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5410 (115'2406,123'5410] local-lis/les=21/22 n=859 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5408 crt=123'5408 lcod 123'5406 mlcod 123'5406 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:13.648 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:13.646+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5601 (115'2511,123'5601] local-lis/les=21/22 n=1059 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'5601 lcod 123'5599 mlcod 123'5599 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:13.650 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:13.646+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5603 (115'2511,123'5603] local-lis/les=21/22 n=1057 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'5601 crt=123'5601 lcod 123'5599 mlcod 123'5599 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:13.655 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:13.650+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5605 (115'2511,123'5605] local-lis/les=21/22 n=1055 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'5605 lcod 123'5603 mlcod 123'5603 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:13.656 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:13.654+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5607 (115'2511,123'5607] local-lis/les=21/22 n=1053 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'5605 crt=123'5605 lcod 123'5603 mlcod 123'5603 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:13.661 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:13.658+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5609 (115'2511,123'5609] local-lis/les=21/22 n=1051 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'5609 lcod 123'5607 mlcod 123'5607 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:13.662 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:13.658+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5611 (115'2511,123'5611] local-lis/les=21/22 n=1049 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'5609 crt=123'5609 lcod 123'5607 mlcod 123'5607 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:13.708 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:13.702+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5964 (115'2909,123'5964] local-lis/les=21/22 n=725 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'5964 lcod 123'5962 mlcod 123'5962 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:13.709 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:13.706+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5966 (115'2909,123'5966] local-lis/les=21/22 n=723 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'5964 crt=123'5964 lcod 123'5962 mlcod 123'5962 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:13.749 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:13.746+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5968 (115'2909,123'5968] local-lis/les=21/22 n=721 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'5968 lcod 123'5966 mlcod 123'5966 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:13.752 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:13.750+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5970 (115'2909,123'5970] local-lis/les=21/22 n=719 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'5968 crt=123'5968 lcod 123'5966 mlcod 123'5966 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:13.821 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:13.813+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5412 (115'2406,123'5412] local-lis/les=21/22 n=857 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5412 lcod 123'5410 mlcod 123'5410 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:13.822 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:13.818+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5449 (115'2407,123'5449] local-lis/les=21/22 n=946 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5449 lcod 123'5447 mlcod 123'5447 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:13.823 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:13.817+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5414 (115'2406,123'5414] local-lis/les=21/22 n=855 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5412 crt=123'5412 lcod 123'5410 mlcod 123'5410 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:13.826 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:13.822+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5451 (115'2407,123'5451] local-lis/les=21/22 n=944 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5449 crt=123'5449 lcod 123'5447 mlcod 123'5447 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:13.832 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:13.830+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5453 (115'2407,123'5453] local-lis/les=21/22 n=942 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5453 lcod 123'5451 mlcod 123'5451 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:13.833 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:13.825+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5416 (115'2406,123'5416] local-lis/les=21/22 n=853 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5416 lcod 123'5414 mlcod 123'5414 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:13.834 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:13.825+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5418 (115'2406,123'5418] local-lis/les=21/22 n=851 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5416 crt=123'5416 lcod 123'5414 mlcod 123'5414 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:13.835 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:13.830+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5455 (115'2407,123'5455] local-lis/les=21/22 n=940 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5453 crt=123'5453 lcod 123'5451 mlcod 123'5451 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:13.839 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:13.834+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5457 (115'2407,123'5457] local-lis/les=21/22 n=938 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5457 lcod 123'5455 mlcod 123'5455 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:13.840 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:13.833+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5420 (115'2406,123'5420] local-lis/les=21/22 n=849 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5420 lcod 123'5418 mlcod 123'5418 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:13.841 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:13.838+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5459 (115'2407,123'5459] local-lis/les=21/22 n=936 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5457 crt=123'5457 lcod 123'5455 mlcod 123'5455 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:13.842 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:13.833+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5422 (115'2406,123'5422] local-lis/les=21/22 n=847 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5420 crt=123'5420 lcod 123'5418 mlcod 123'5418 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:13.848 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:13.841+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5424 (115'2406,123'5424] local-lis/les=21/22 n=845 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5424 lcod 123'5422 mlcod 123'5422 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:13.848 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:13.842+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5461 (115'2407,123'5461] local-lis/les=21/22 n=934 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5461 lcod 123'5459 mlcod 123'5459 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:13.849 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:13.841+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5426 (115'2406,123'5426] local-lis/les=21/22 n=843 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5424 crt=123'5424 lcod 123'5422 mlcod 123'5422 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:13.850 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:13.846+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5463 (115'2407,123'5463] local-lis/les=21/22 n=932 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5461 crt=123'5461 lcod 123'5459 mlcod 123'5459 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:13.855 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:13.850+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5465 (115'2407,123'5465] local-lis/les=21/22 n=930 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5465 lcod 123'5463 mlcod 123'5463 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:13.857 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:13.849+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5428 (115'2406,123'5428] local-lis/les=21/22 n=841 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5428 lcod 123'5426 mlcod 123'5426 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:13.857 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:13.854+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5467 (115'2407,123'5467] local-lis/les=21/22 n=928 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5465 crt=123'5465 lcod 123'5463 mlcod 123'5463 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:13.858 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:13.849+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5430 (115'2406,123'5430] local-lis/les=21/22 n=839 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5428 crt=123'5428 lcod 123'5426 mlcod 123'5426 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:13.863 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:13.857+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5432 (115'2406,123'5432] local-lis/les=21/22 n=837 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5432 lcod 123'5430 mlcod 123'5430 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:13.863 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:13.858+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5469 (115'2407,123'5469] local-lis/les=21/22 n=926 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5469 lcod 123'5467 mlcod 123'5467 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:13.865 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:13.862+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5471 (115'2407,123'5471] local-lis/les=21/22 n=924 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5469 crt=123'5469 lcod 123'5467 mlcod 123'5467 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:13.866 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:13.857+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5434 (115'2406,123'5434] local-lis/les=21/22 n=835 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5432 crt=123'5432 lcod 123'5430 mlcod 123'5430 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:13.874 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:13.861+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5436 (115'2406,123'5436] local-lis/les=21/22 n=833 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5436 lcod 123'5434 mlcod 123'5434 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:13.874 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:13.865+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5438 (115'2406,123'5438] local-lis/les=21/22 n=831 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5436 crt=123'5436 lcod 123'5434 mlcod 123'5434 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:13.874 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:13.866+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5473 (115'2407,123'5473] local-lis/les=21/22 n=922 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5473 lcod 123'5471 mlcod 123'5471 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:13.875 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:13.870+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5475 (115'2407,123'5475] local-lis/les=21/22 n=920 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5473 crt=123'5473 lcod 123'5471 mlcod 123'5471 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:13.878 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:13.869+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5440 (115'2406,123'5440] local-lis/les=21/22 n=829 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5440 lcod 123'5438 mlcod 123'5438 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:13.878 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:13.874+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5477 (115'2407,123'5477] local-lis/les=21/22 n=918 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5477 lcod 123'5475 mlcod 123'5475 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:13.879 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:13.873+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5442 (115'2406,123'5442] local-lis/les=21/22 n=827 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5440 crt=123'5440 lcod 123'5438 mlcod 123'5438 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:13.880 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:13.874+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5479 (115'2407,123'5479] local-lis/les=21/22 n=916 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5477 crt=123'5477 lcod 123'5475 mlcod 123'5475 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:13.891 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:13.877+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5444 (115'2406,123'5444] local-lis/les=21/22 n=825 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5444 lcod 123'5442 mlcod 123'5442 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:13.892 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:13.881+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5446 (115'2406,123'5446] local-lis/les=21/22 n=823 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5444 crt=123'5444 lcod 123'5442 mlcod 123'5442 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:13.892 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:13.882+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5481 (115'2407,123'5481] local-lis/les=21/22 n=914 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5481 lcod 123'5479 mlcod 123'5479 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:13.892 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:13.882+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5483 (115'2407,123'5483] local-lis/les=21/22 n=912 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5481 crt=123'5481 lcod 123'5479 mlcod 123'5479 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:13.892 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:13.886+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5485 (115'2407,123'5485] local-lis/les=21/22 n=910 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5485 lcod 123'5483 mlcod 123'5483 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:13.894 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:13.885+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5448 (115'2406,123'5448] local-lis/les=21/22 n=821 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5448 lcod 123'5446 mlcod 123'5446 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:13.895 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:13.889+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5450 (115'2406,123'5450] local-lis/les=21/22 n=819 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5448 crt=123'5448 lcod 123'5446 mlcod 123'5446 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:13.898 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:13.894+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5487 (115'2407,123'5487] local-lis/les=21/22 n=908 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5487 lcod 123'5485 mlcod 123'5485 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:13.900 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:13.898+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5489 (115'2407,123'5489] local-lis/les=21/22 n=906 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5487 crt=123'5487 lcod 123'5485 mlcod 123'5485 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:13.901 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:13.893+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5452 (115'2406,123'5452] local-lis/les=21/22 n=817 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5452 lcod 123'5450 mlcod 123'5450 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:13.902 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:13.897+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5454 (115'2406,123'5454] local-lis/les=21/22 n=815 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5452 crt=123'5452 lcod 123'5450 mlcod 123'5450 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:13.907 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:13.902+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5491 (115'2407,123'5491] local-lis/les=21/22 n=904 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5491 lcod 123'5489 mlcod 123'5489 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:13.908 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:13.901+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5456 (115'2406,123'5456] local-lis/les=21/22 n=813 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5456 lcod 123'5454 mlcod 123'5454 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:13.909 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:13.906+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5493 (115'2407,123'5493] local-lis/les=21/22 n=902 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5491 crt=123'5491 lcod 123'5489 mlcod 123'5489 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:13.909 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:13.901+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5458 (115'2406,123'5458] local-lis/les=21/22 n=811 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5456 crt=123'5456 lcod 123'5454 mlcod 123'5454 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:13.914 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:13.910+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5495 (115'2407,123'5495] local-lis/les=21/22 n=900 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5495 lcod 123'5493 mlcod 123'5493 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:13.915 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:13.909+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5460 (115'2406,123'5460] local-lis/les=21/22 n=809 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5460 lcod 123'5458 mlcod 123'5458 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:13.916 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:13.914+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5497 (115'2407,123'5497] local-lis/les=21/22 n=898 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5495 crt=123'5495 lcod 123'5493 mlcod 123'5493 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:13.917 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:13.909+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5462 (115'2406,123'5462] local-lis/les=21/22 n=807 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5460 crt=123'5460 lcod 123'5458 mlcod 123'5458 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:13.924 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:13.918+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5499 (115'2407,123'5499] local-lis/les=21/22 n=896 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5499 lcod 123'5497 mlcod 123'5497 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:13.924 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:13.922+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5501 (115'2407,123'5501] local-lis/les=21/22 n=894 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5499 crt=123'5499 lcod 123'5497 mlcod 123'5497 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:13.925 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:13.917+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5464 (115'2406,123'5464] local-lis/les=21/22 n=805 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5464 lcod 123'5462 mlcod 123'5462 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:13.925 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:13.917+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5466 (115'2406,123'5466] local-lis/les=21/22 n=803 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5464 crt=123'5464 lcod 123'5462 mlcod 123'5462 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:13.929 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:13.921+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5468 (115'2406,123'5468] local-lis/les=21/22 n=801 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5468 lcod 123'5466 mlcod 123'5466 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:13.930 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:13.926+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5503 (115'2407,123'5503] local-lis/les=21/22 n=892 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5503 lcod 123'5501 mlcod 123'5501 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:13.931 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:13.925+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5470 (115'2406,123'5470] local-lis/les=21/22 n=799 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5468 crt=123'5468 lcod 123'5466 mlcod 123'5466 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:13.931 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:13.926+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5505 (115'2407,123'5505] local-lis/les=21/22 n=890 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5503 crt=123'5503 lcod 123'5501 mlcod 123'5501 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:13.936 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:13.934+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5507 (115'2407,123'5507] local-lis/les=21/22 n=888 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5507 lcod 123'5505 mlcod 123'5505 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:13.936 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:13.929+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5472 (115'2406,123'5472] local-lis/les=21/22 n=797 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5472 lcod 123'5470 mlcod 123'5470 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:13.938 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:13.929+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5474 (115'2406,123'5474] local-lis/les=21/22 n=795 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5472 crt=123'5472 lcod 123'5470 mlcod 123'5470 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:13.939 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:13.934+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5509 (115'2507,123'5509] local-lis/les=21/22 n=886 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5507 crt=123'5507 lcod 123'5505 mlcod 123'5505 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:13.945 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:13.937+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5476 (115'2406,123'5476] local-lis/les=21/22 n=793 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5476 lcod 123'5474 mlcod 123'5474 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:13.946 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:13.942+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5511 (115'2507,123'5511] local-lis/les=21/22 n=884 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5511 lcod 123'5509 mlcod 123'5509 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:13.947 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:13.941+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5478 (115'2406,123'5478] local-lis/les=21/22 n=791 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5476 crt=123'5476 lcod 123'5474 mlcod 123'5474 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:13.948 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:13.946+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5513 (115'2507,123'5513] local-lis/les=21/22 n=882 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5511 crt=123'5511 lcod 123'5509 mlcod 123'5509 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:14.034 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:14.030+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5972 (115'2909,123'5972] local-lis/les=21/22 n=717 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'5972 lcod 123'5970 mlcod 123'5970 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:14.036 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:14.034+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5974 (115'2909,123'5974] local-lis/les=21/22 n=715 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'5972 crt=123'5972 lcod 123'5970 mlcod 123'5970 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:14.041 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:14.038+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5976 (115'2909,123'5976] local-lis/les=21/22 n=713 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'5976 lcod 123'5974 mlcod 123'5974 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:14.045 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:14.042+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5978 (115'2909,123'5978] local-lis/les=21/22 n=711 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'5976 crt=123'5976 lcod 123'5974 mlcod 123'5974 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:14.051 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:14.046+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5980 (115'2909,123'5980] local-lis/les=21/22 n=709 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'5980 lcod 123'5978 mlcod 123'5978 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:14.053 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:14.050+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5982 (115'2909,123'5982] local-lis/les=21/22 n=707 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'5980 crt=123'5980 lcod 123'5978 mlcod 123'5978 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:14.059 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:14.054+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5984 (115'2909,123'5984] local-lis/les=21/22 n=705 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'5984 lcod 123'5982 mlcod 123'5982 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:14.060 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:14.054+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5728 (115'2704,123'5728] local-lis/les=21/22 n=737 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5728 lcod 123'5726 mlcod 123'5726 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:14.061 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:14.058+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5986 (115'2909,123'5986] local-lis/les=21/22 n=703 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'5984 crt=123'5984 lcod 123'5982 mlcod 123'5982 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:14.061 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:14.058+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5730 (115'2704,123'5730] local-lis/les=21/22 n=735 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5728 crt=123'5728 lcod 123'5726 mlcod 123'5726 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:14.062 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:14.054+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5458 (115'2404,123'5458] local-lis/les=21/22 n=761 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5458 lcod 123'5456 mlcod 123'5456 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:14.063 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:14.058+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5460 (115'2404,123'5460] local-lis/les=21/22 n=759 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5458 crt=123'5458 lcod 123'5456 mlcod 123'5456 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:14.066 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:14.062+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5613 (115'2611,123'5613] local-lis/les=21/22 n=1047 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'5613 lcod 123'5611 mlcod 123'5611 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:14.070 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:14.066+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5615 (115'2611,123'5615] local-lis/les=21/22 n=1045 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'5613 crt=123'5613 lcod 123'5611 mlcod 123'5611 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:14.073 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:14.070+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5462 (115'2404,123'5462] local-lis/les=21/22 n=757 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5462 lcod 123'5460 mlcod 123'5460 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:14.073 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:14.066+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5988 (115'2909,123'5988] local-lis/les=21/22 n=701 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'5988 lcod 123'5986 mlcod 123'5986 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:14.074 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:14.070+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5617 (115'2611,123'5617] local-lis/les=21/22 n=1043 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'5617 lcod 123'5615 mlcod 123'5615 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:14.077 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:14.070+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5464 (115'2404,123'5464] local-lis/les=21/22 n=755 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5462 crt=123'5462 lcod 123'5460 mlcod 123'5460 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:14.078 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:14.070+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5990 (115'2909,123'5990] local-lis/les=21/22 n=699 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'5988 crt=123'5988 lcod 123'5986 mlcod 123'5986 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:14.078 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:14.074+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5732 (115'2704,123'5732] local-lis/les=21/22 n=733 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5732 lcod 123'5730 mlcod 123'5730 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:14.078 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:14.074+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5734 (115'2704,123'5734] local-lis/les=21/22 n=731 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5732 crt=123'5732 lcod 123'5730 mlcod 123'5730 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:14.082 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:14.078+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5992 (115'2909,123'5992] local-lis/les=21/22 n=697 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'5992 lcod 123'5990 mlcod 123'5990 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:14.082 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:14.078+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5619 (115'2611,123'5619] local-lis/les=21/22 n=1041 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'5617 crt=123'5617 lcod 123'5615 mlcod 123'5615 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:14.084 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:14.082+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5994 (115'2909,123'5994] local-lis/les=21/22 n=695 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'5992 crt=123'5992 lcod 123'5990 mlcod 123'5990 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:14.092 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:14.090+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5466 (115'2404,123'5466] local-lis/les=21/22 n=753 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5466 lcod 123'5464 mlcod 123'5464 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:14.094 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:14.085+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5463 (115'2405,123'5463] local-lis/les=21/22 n=1054 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=123'5463 lcod 123'5461 mlcod 123'5461 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:14.096 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:14.089+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5465 (115'2405,123'5465] local-lis/les=21/22 n=1052 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5463 crt=123'5463 lcod 123'5461 mlcod 123'5461 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:14.097 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:14.090+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5468 (115'2404,123'5468] local-lis/les=21/22 n=751 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5466 crt=123'5466 lcod 123'5464 mlcod 123'5464 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:14.097 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:14.086+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5996 (115'2909,123'5996] local-lis/les=21/22 n=693 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'5996 lcod 123'5994 mlcod 123'5994 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:14.097 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:14.090+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'5998 (115'2909,123'5998] local-lis/les=21/22 n=691 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'5996 crt=123'5996 lcod 123'5994 mlcod 123'5994 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:14.097 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:14.086+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5615 (115'2605,123'5615] local-lis/les=21/22 n=774 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5615 lcod 123'5613 mlcod 123'5613 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:14.097 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:14.090+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5617 (115'2605,123'5617] local-lis/les=21/22 n=772 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5615 crt=123'5615 lcod 123'5613 mlcod 123'5613 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:14.099 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:14.094+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5736 (115'2704,123'5736] local-lis/les=21/22 n=729 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5736 lcod 123'5734 mlcod 123'5734 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:14.099 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:14.089+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5480 (115'2406,123'5480] local-lis/les=21/22 n=789 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5480 lcod 123'5478 mlcod 123'5478 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:14.100 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:14.098+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5738 (115'2704,123'5738] local-lis/les=21/22 n=727 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5736 crt=123'5736 lcod 123'5734 mlcod 123'5734 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:14.100 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:14.093+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5482 (115'2406,123'5482] local-lis/les=21/22 n=787 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5480 crt=123'5480 lcod 123'5478 mlcod 123'5478 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:14.106 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:14.102+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6000 (115'2909,123'6000] local-lis/les=21/22 n=689 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'6000 lcod 123'5998 mlcod 123'5998 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:14.107 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:14.102+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6002 (115'2909,123'6002] local-lis/les=21/22 n=687 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6000 crt=123'6000 lcod 123'5998 mlcod 123'5998 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:14.110 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:14.106+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5621 (115'2611,123'5621] local-lis/les=21/22 n=1039 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'5621 lcod 123'5619 mlcod 123'5619 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:14.114 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:14.110+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5623 (115'2611,123'5623] local-lis/les=21/22 n=1037 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'5621 crt=123'5621 lcod 123'5619 mlcod 123'5619 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:14.115 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:14.109+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5467 (115'2405,123'5467] local-lis/les=21/22 n=1050 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=123'5467 lcod 123'5465 mlcod 123'5465 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:14.119 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:14.114+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5470 (115'2404,123'5470] local-lis/les=21/22 n=749 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5470 lcod 123'5468 mlcod 123'5468 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:14.120 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:14.113+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5469 (115'2405,123'5469] local-lis/les=21/22 n=1048 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5467 crt=123'5467 lcod 123'5465 mlcod 123'5465 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:14.121 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:14.114+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5472 (115'2404,123'5472] local-lis/les=21/22 n=747 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5470 crt=123'5470 lcod 123'5468 mlcod 123'5468 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:14.122 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:14.114+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6004 (115'2909,123'6004] local-lis/les=21/22 n=685 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'6004 lcod 123'6002 mlcod 123'6002 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:14.123 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:14.118+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5740 (115'2704,123'5740] local-lis/les=21/22 n=725 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5740 lcod 123'5738 mlcod 123'5738 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:14.123 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:14.118+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6006 (115'2909,123'6006] local-lis/les=21/22 n=683 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6004 crt=123'6004 lcod 123'6002 mlcod 123'6002 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:14.124 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:14.118+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5742 (115'2704,123'5742] local-lis/les=21/22 n=723 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5740 crt=123'5740 lcod 123'5738 mlcod 123'5738 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:14.128 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:14.122+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5625 (115'2611,123'5625] local-lis/les=21/22 n=1035 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'5625 lcod 123'5623 mlcod 123'5623 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:14.129 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:14.126+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5627 (115'2611,123'5627] local-lis/les=21/22 n=1033 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'5625 crt=123'5625 lcod 123'5623 mlcod 123'5623 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:14.132 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:14.125+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5471 (115'2405,123'5471] local-lis/les=21/22 n=1046 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=123'5471 lcod 123'5469 mlcod 123'5469 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:14.134 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:14.125+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5473 (115'2405,123'5473] local-lis/les=21/22 n=1044 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5471 crt=123'5471 lcod 123'5469 mlcod 123'5469 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:14.135 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:14.130+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5474 (115'2404,123'5474] local-lis/les=21/22 n=745 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5474 lcod 123'5472 mlcod 123'5472 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:14.139 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:14.134+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5629 (115'2611,123'5629] local-lis/les=21/22 n=1031 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'5629 lcod 123'5627 mlcod 123'5627 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:14.143 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:14.137+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5475 (115'2405,123'5475] local-lis/les=21/22 n=1042 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=123'5475 lcod 123'5473 mlcod 123'5473 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:14.146 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:14.138+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5631 (115'2611,123'5631] local-lis/les=21/22 n=1029 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'5629 crt=123'5629 lcod 123'5627 mlcod 123'5627 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:14.147 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:14.134+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5744 (115'2704,123'5744] local-lis/les=21/22 n=721 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5744 lcod 123'5742 mlcod 123'5742 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:14.147 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:14.138+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5746 (115'2704,123'5746] local-lis/les=21/22 n=719 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5744 crt=123'5744 lcod 123'5742 mlcod 123'5742 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:14.147 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:14.137+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5477 (115'2405,123'5477] local-lis/les=21/22 n=1040 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5475 crt=123'5475 lcod 123'5473 mlcod 123'5473 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:14.155 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:14.145+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5479 (115'2405,123'5479] local-lis/les=21/22 n=1038 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=123'5479 lcod 123'5477 mlcod 123'5477 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:14.156 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:14.150+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5748 (115'2704,123'5748] local-lis/les=21/22 n=717 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5748 lcod 123'5746 mlcod 123'5746 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:14.156 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:14.150+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5476 (115'2404,123'5476] local-lis/les=21/22 n=743 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5476 lcod 123'5474 mlcod 123'5474 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:14.157 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:14.149+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5481 (115'2405,123'5481] local-lis/les=21/22 n=1036 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5479 crt=123'5479 lcod 123'5477 mlcod 123'5477 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:14.157 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:14.150+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5633 (115'2611,123'5633] local-lis/les=21/22 n=1027 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'5633 lcod 123'5631 mlcod 123'5631 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:14.157 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:14.150+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5478 (115'2404,123'5478] local-lis/les=21/22 n=741 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5476 crt=123'5476 lcod 123'5474 mlcod 123'5474 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:14.158 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:14.154+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5750 (115'2704,123'5750] local-lis/les=21/22 n=715 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5748 crt=123'5748 lcod 123'5746 mlcod 123'5746 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:14.161 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:14.154+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5635 (115'2611,123'5635] local-lis/les=21/22 n=1025 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'5633 crt=123'5633 lcod 123'5631 mlcod 123'5631 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:14.168 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:14.162+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5752 (115'2704,123'5752] local-lis/les=21/22 n=713 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5752 lcod 123'5750 mlcod 123'5750 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:14.168 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:14.166+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5754 (115'2704,123'5754] local-lis/les=21/22 n=711 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5752 crt=123'5752 lcod 123'5750 mlcod 123'5750 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:14.169 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:14.166+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5480 (115'2404,123'5480] local-lis/les=21/22 n=739 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5480 lcod 123'5478 mlcod 123'5478 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:14.171 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:14.166+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5482 (115'2404,123'5482] local-lis/les=21/22 n=737 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5480 crt=123'5480 lcod 123'5478 mlcod 123'5478 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:14.174 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:14.170+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5637 (115'2611,123'5637] local-lis/les=21/22 n=1023 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'5637 lcod 123'5635 mlcod 123'5635 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:14.175 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:14.169+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5483 (115'2405,123'5483] local-lis/les=21/22 n=1034 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=123'5483 lcod 123'5481 mlcod 123'5481 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:14.176 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:14.174+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5639 (115'2611,123'5639] local-lis/les=21/22 n=1021 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'5637 crt=123'5637 lcod 123'5635 mlcod 123'5635 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:14.177 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:14.169+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5485 (115'2405,123'5485] local-lis/les=21/22 n=1032 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5483 crt=123'5483 lcod 123'5481 mlcod 123'5481 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:14.179 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:14.174+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5756 (115'2704,123'5756] local-lis/les=21/22 n=709 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5756 lcod 123'5754 mlcod 123'5754 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:14.179 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:14.174+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5758 (115'2704,123'5758] local-lis/les=21/22 n=707 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5756 crt=123'5756 lcod 123'5754 mlcod 123'5754 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:14.183 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:14.178+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5484 (115'2404,123'5484] local-lis/les=21/22 n=735 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5484 lcod 123'5482 mlcod 123'5482 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:14.187 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:14.182+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5486 (115'2404,123'5486] local-lis/les=21/22 n=733 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5484 crt=123'5484 lcod 123'5482 mlcod 123'5482 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:14.192 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:14.186+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5641 (115'2611,123'5641] local-lis/les=21/22 n=1019 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'5641 lcod 123'5639 mlcod 123'5639 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:14.194 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:14.190+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5643 (115'2611,123'5643] local-lis/les=21/22 n=1017 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'5641 crt=123'5641 lcod 123'5639 mlcod 123'5639 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:14.195 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:14.189+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5487 (115'2405,123'5487] local-lis/les=21/22 n=1030 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=123'5487 lcod 123'5485 mlcod 123'5485 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:14.197 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:14.189+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5489 (115'2405,123'5489] local-lis/les=21/22 n=1028 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5487 crt=123'5487 lcod 123'5485 mlcod 123'5485 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:14.199 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:14.194+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5488 (115'2404,123'5488] local-lis/les=21/22 n=731 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5488 lcod 123'5486 mlcod 123'5486 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:14.199 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:14.194+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5760 (115'2704,123'5760] local-lis/les=21/22 n=705 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5760 lcod 123'5758 mlcod 123'5758 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:14.201 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:14.198+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5762 (115'2704,123'5762] local-lis/les=21/22 n=703 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5760 crt=123'5760 lcod 123'5758 mlcod 123'5758 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:14.203 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:14.198+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5490 (115'2404,123'5490] local-lis/les=21/22 n=729 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5488 crt=123'5488 lcod 123'5486 mlcod 123'5486 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:14.206 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:14.201+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5491 (115'2405,123'5491] local-lis/les=21/22 n=1026 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=123'5491 lcod 123'5489 mlcod 123'5489 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:14.207 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:14.202+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5645 (115'2611,123'5645] local-lis/les=21/22 n=1015 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'5645 lcod 123'5643 mlcod 123'5643 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:14.209 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:14.201+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5493 (115'2405,123'5493] local-lis/les=21/22 n=1024 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5491 crt=123'5491 lcod 123'5489 mlcod 123'5489 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:14.216 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:14.206+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5647 (115'2611,123'5647] local-lis/les=21/22 n=1013 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'5645 crt=123'5645 lcod 123'5643 mlcod 123'5643 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:14.216 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:14.210+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5492 (115'2404,123'5492] local-lis/les=21/22 n=727 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5492 lcod 123'5490 mlcod 123'5490 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:14.217 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:14.210+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5494 (115'2404,123'5494] local-lis/les=21/22 n=725 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5492 crt=123'5492 lcod 123'5490 mlcod 123'5490 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:14.217 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:14.210+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5764 (115'2704,123'5764] local-lis/les=21/22 n=701 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5764 lcod 123'5762 mlcod 123'5762 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:14.218 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:14.210+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5766 (115'2704,123'5766] local-lis/les=21/22 n=699 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5764 crt=123'5764 lcod 123'5762 mlcod 123'5762 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:14.220 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:14.213+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5495 (115'2405,123'5495] local-lis/les=21/22 n=1022 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=123'5495 lcod 123'5493 mlcod 123'5493 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:14.220 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:14.218+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5649 (115'2611,123'5649] local-lis/les=21/22 n=1011 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'5649 lcod 123'5647 mlcod 123'5647 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:14.221 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:14.213+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5497 (115'2405,123'5497] local-lis/les=21/22 n=1020 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5495 crt=123'5495 lcod 123'5493 mlcod 123'5493 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:14.224 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:14.222+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5768 (115'2704,123'5768] local-lis/les=21/22 n=697 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5768 lcod 123'5766 mlcod 123'5766 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:14.226 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:14.218+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5651 (115'2611,123'5651] local-lis/les=21/22 n=1009 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'5649 crt=123'5649 lcod 123'5647 mlcod 123'5647 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:14.227 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:14.222+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5496 (115'2404,123'5496] local-lis/les=21/22 n=723 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5496 lcod 123'5494 mlcod 123'5494 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:14.228 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:14.222+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5498 (115'2404,123'5498] local-lis/les=21/22 n=721 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5496 crt=123'5496 lcod 123'5494 mlcod 123'5494 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:14.229 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:14.226+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5770 (115'2704,123'5770] local-lis/les=21/22 n=695 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5768 crt=123'5768 lcod 123'5766 mlcod 123'5766 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:14.234 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:14.229+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5499 (115'2405,123'5499] local-lis/les=21/22 n=1018 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=123'5499 lcod 123'5497 mlcod 123'5497 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:14.236 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:14.229+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5501 (115'2405,123'5501] local-lis/les=21/22 n=1016 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5499 crt=123'5499 lcod 123'5497 mlcod 123'5497 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:14.238 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:14.230+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5653 (115'2611,123'5653] local-lis/les=21/22 n=1007 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'5653 lcod 123'5651 mlcod 123'5651 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:14.240 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:14.234+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5772 (115'2704,123'5772] local-lis/les=21/22 n=693 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5772 lcod 123'5770 mlcod 123'5770 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:14.240 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:14.234+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5500 (115'2404,123'5500] local-lis/les=21/22 n=719 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5500 lcod 123'5498 mlcod 123'5498 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:14.240 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:14.234+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5655 (115'2611,123'5655] local-lis/les=21/22 n=1005 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'5653 crt=123'5653 lcod 123'5651 mlcod 123'5651 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:14.241 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:14.238+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5502 (115'2404,123'5502] local-lis/les=21/22 n=717 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5500 crt=123'5500 lcod 123'5498 mlcod 123'5498 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:14.243 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:14.238+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5774 (115'2704,123'5774] local-lis/les=21/22 n=691 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5772 crt=123'5772 lcod 123'5770 mlcod 123'5770 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:14.250 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:14.245+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5503 (115'2405,123'5503] local-lis/les=21/22 n=1014 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=123'5503 lcod 123'5501 mlcod 123'5501 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:14.253 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:14.245+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5505 (115'2405,123'5505] local-lis/les=21/22 n=1012 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5503 crt=123'5503 lcod 123'5501 mlcod 123'5501 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:14.254 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:14.246+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5776 (115'2704,123'5776] local-lis/les=21/22 n=689 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5776 lcod 123'5774 mlcod 123'5774 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:14.255 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:14.246+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5657 (115'2611,123'5657] local-lis/les=21/22 n=1003 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'5657 lcod 123'5655 mlcod 123'5655 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:14.256 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:14.250+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5504 (115'2404,123'5504] local-lis/les=21/22 n=715 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5504 lcod 123'5502 mlcod 123'5502 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:14.256 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:14.250+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5659 (115'2611,123'5659] local-lis/les=21/22 n=1001 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'5657 crt=123'5657 lcod 123'5655 mlcod 123'5655 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:14.257 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:14.254+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5778 (115'2704,123'5778] local-lis/les=21/22 n=687 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5776 crt=123'5776 lcod 123'5774 mlcod 123'5774 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:14.257 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:14.254+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5506 (115'2504,123'5506] local-lis/les=21/22 n=713 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5504 crt=123'5504 lcod 123'5502 mlcod 123'5502 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:14.264 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:14.262+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5661 (115'2611,123'5661] local-lis/les=21/22 n=999 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'5661 lcod 123'5659 mlcod 123'5659 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:14.266 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:14.261+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5507 (115'2505,123'5507] local-lis/les=21/22 n=1010 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=123'5507 lcod 123'5505 mlcod 123'5505 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:14.267 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:14.262+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5508 (115'2504,123'5508] local-lis/les=21/22 n=711 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5508 lcod 123'5506 mlcod 123'5506 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:14.270 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:14.261+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5509 (115'2505,123'5509] local-lis/les=21/22 n=1008 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5507 crt=123'5507 lcod 123'5505 mlcod 123'5505 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:14.270 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:14.266+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5510 (115'2504,123'5510] local-lis/les=21/22 n=709 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5508 crt=123'5508 lcod 123'5506 mlcod 123'5506 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:14.275 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:14.270+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5665 (115'2611,123'5665] local-lis/les=21/22 n=996 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'5665 lcod 123'5663 mlcod 123'5663 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:14.276 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:14.269+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5511 (115'2505,123'5511] local-lis/les=21/22 n=1006 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=123'5511 lcod 123'5509 mlcod 123'5509 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:14.277 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:14.274+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5667 (115'2611,123'5667] local-lis/les=21/22 n=994 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'5665 crt=123'5665 lcod 123'5663 mlcod 123'5663 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:14.278 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:14.273+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5513 (115'2505,123'5513] local-lis/les=21/22 n=1004 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5511 crt=123'5511 lcod 123'5509 mlcod 123'5509 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:14.371 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:14.365+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5515 (115'2505,123'5515] local-lis/les=21/22 n=1002 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=123'5515 lcod 123'5513 mlcod 123'5513 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:14.373 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:14.365+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5517 (115'2505,123'5517] local-lis/les=21/22 n=1000 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5515 crt=123'5515 lcod 123'5513 mlcod 123'5513 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:14.379 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:14.373+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5519 (115'2505,123'5519] local-lis/les=21/22 n=998 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=123'5519 lcod 123'5517 mlcod 123'5517 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:14.380 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:14.373+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5521 (115'2505,123'5521] local-lis/les=21/22 n=996 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5519 crt=123'5519 lcod 123'5517 mlcod 123'5517 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:14.386 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:14.381+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5523 (115'2505,123'5523] local-lis/les=21/22 n=994 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=123'5523 lcod 123'5521 mlcod 123'5521 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:14.388 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:14.381+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5525 (115'2505,123'5525] local-lis/les=21/22 n=992 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5523 crt=123'5523 lcod 123'5521 mlcod 123'5521 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:14.392 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:14.385+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5527 (115'2505,123'5527] local-lis/les=21/22 n=990 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=123'5527 lcod 123'5525 mlcod 123'5525 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:14.395 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:14.389+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5529 (115'2505,123'5529] local-lis/les=21/22 n=988 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5527 crt=123'5527 lcod 123'5525 mlcod 123'5525 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:14.400 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:14.393+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5531 (115'2505,123'5531] local-lis/les=21/22 n=986 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=123'5531 lcod 123'5529 mlcod 123'5529 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:14.402 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:14.397+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5533 (115'2505,123'5533] local-lis/les=21/22 n=984 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5531 crt=123'5531 lcod 123'5529 mlcod 123'5529 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:14.406 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:14.401+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5535 (115'2505,123'5535] local-lis/les=21/22 n=982 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=123'5535 lcod 123'5533 mlcod 123'5533 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:14.408 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:14.401+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5537 (115'2505,123'5537] local-lis/les=21/22 n=980 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5535 crt=123'5535 lcod 123'5533 mlcod 123'5533 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:14.413 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:14.405+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5539 (115'2505,123'5539] local-lis/les=21/22 n=978 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=123'5539 lcod 123'5537 mlcod 123'5537 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:14.414 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:14.409+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5541 (115'2505,123'5541] local-lis/les=21/22 n=976 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5539 crt=123'5539 lcod 123'5537 mlcod 123'5537 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:14.419 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:14.413+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5543 (115'2505,123'5543] local-lis/les=21/22 n=974 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=123'5543 lcod 123'5541 mlcod 123'5541 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:14.421 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:14.413+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5484 (115'2406,123'5484] local-lis/les=21/22 n=785 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5484 lcod 123'5482 mlcod 123'5482 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:14.423 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:14.417+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5486 (115'2406,123'5486] local-lis/les=21/22 n=783 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5484 crt=123'5484 lcod 123'5482 mlcod 123'5482 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:14.423 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:14.413+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5545 (115'2505,123'5545] local-lis/les=21/22 n=972 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5543 crt=123'5543 lcod 123'5541 mlcod 123'5541 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:14.428 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:14.421+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5488 (115'2406,123'5488] local-lis/les=21/22 n=781 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5488 lcod 123'5486 mlcod 123'5486 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:14.431 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:14.421+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5547 (115'2505,123'5547] local-lis/les=21/22 n=970 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=123'5547 lcod 123'5545 mlcod 123'5545 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:14.431 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:14.425+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5490 (115'2406,123'5490] local-lis/les=21/22 n=779 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5488 crt=123'5488 lcod 123'5486 mlcod 123'5486 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:14.435 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:14.429+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5551 (115'2505,123'5551] local-lis/les=21/22 n=967 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=123'5551 lcod 123'5549 mlcod 123'5549 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:14.438 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:14.429+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5492 (115'2406,123'5492] local-lis/les=21/22 n=777 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5492 lcod 123'5490 mlcod 123'5490 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:14.439 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:14.429+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5553 (115'2505,123'5553] local-lis/les=21/22 n=965 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5551 crt=123'5551 lcod 123'5549 mlcod 123'5549 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:14.439 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:14.433+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5494 (115'2406,123'5494] local-lis/les=21/22 n=775 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5492 crt=123'5492 lcod 123'5490 mlcod 123'5490 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:14.444 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:14.437+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5555 (115'2505,123'5555] local-lis/les=21/22 n=963 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=123'5555 lcod 123'5553 mlcod 123'5553 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:14.444 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:14.437+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5496 (115'2406,123'5496] local-lis/les=21/22 n=773 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5496 lcod 123'5494 mlcod 123'5494 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:14.447 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:14.441+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5557 (115'2505,123'5557] local-lis/les=21/22 n=961 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5555 crt=123'5555 lcod 123'5553 mlcod 123'5553 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:14.447 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:14.441+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5498 (115'2406,123'5498] local-lis/les=21/22 n=771 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5496 crt=123'5496 lcod 123'5494 mlcod 123'5494 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:14.451 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:14.445+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5559 (115'2505,123'5559] local-lis/les=21/22 n=959 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=123'5559 lcod 123'5557 mlcod 123'5557 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:14.453 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:14.445+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5500 (115'2406,123'5500] local-lis/les=21/22 n=769 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5500 lcod 123'5498 mlcod 123'5498 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:14.456 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:14.445+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5561 (115'2505,123'5561] local-lis/les=21/22 n=957 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5559 crt=123'5559 lcod 123'5557 mlcod 123'5557 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:14.457 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:14.454+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5513 (115'2504,123'5513] local-lis/les=21/22 n=706 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5512 crt=123'5510 lcod 123'5510 mlcod 123'5510 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:14.459 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:14.454+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5516 (115'2507,123'5516] local-lis/les=21/22 n=879 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5515 crt=123'5513 lcod 123'5513 mlcod 123'5513 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:14.460 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:14.449+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5502 (115'2406,123'5502] local-lis/les=21/22 n=767 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5500 crt=123'5500 lcod 123'5498 mlcod 123'5498 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:14.462 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:14.454+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5515 (115'2504,123'5515] local-lis/les=21/22 n=704 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5512 crt=123'5512 lcod 123'5510 mlcod 123'5510 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:14.462 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:14.458+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5518 (115'2507,123'5518] local-lis/les=21/22 n=877 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5515 crt=123'5515 lcod 123'5513 mlcod 123'5513 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:14.463 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:14.458+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5669 (115'2611,123'5669] local-lis/les=21/22 n=992 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'5669 lcod 123'5667 mlcod 123'5667 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:14.468 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:14.462+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5671 (115'2611,123'5671] local-lis/les=21/22 n=990 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'5669 crt=123'5669 lcod 123'5667 mlcod 123'5667 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:14.469 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:14.466+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5781 (115'2704,123'5781] local-lis/les=21/22 n=684 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5780 crt=123'5778 lcod 123'5778 mlcod 123'5778 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:14.473 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:14.465+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5504 (115'2406,123'5504] local-lis/les=21/22 n=765 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5504 lcod 123'5502 mlcod 123'5502 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:14.473 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:14.470+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5783 (115'2704,123'5783] local-lis/les=21/22 n=682 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5780 crt=123'5780 lcod 123'5778 mlcod 123'5778 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:14.474 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:14.470+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5520 (115'2507,123'5520] local-lis/les=21/22 n=875 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5520 lcod 123'5518 mlcod 123'5518 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:14.476 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:14.470+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5522 (115'2507,123'5522] local-lis/les=21/22 n=873 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5520 crt=123'5520 lcod 123'5518 mlcod 123'5518 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:14.476 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:14.474+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5518 (115'2504,123'5518] local-lis/les=21/22 n=701 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5517 crt=123'5513 lcod 123'5515 mlcod 123'5515 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:14.481 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:14.478+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5520 (115'2504,123'5520] local-lis/les=21/22 n=699 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5517 crt=123'5517 lcod 123'5515 mlcod 123'5515 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:14.482 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:14.473+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5563 (115'2505,123'5563] local-lis/les=21/22 n=955 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=123'5563 lcod 123'5561 mlcod 123'5561 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:14.482 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:14.473+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5506 (115'2406,123'5506] local-lis/les=21/22 n=763 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5506 lcod 123'5504 mlcod 123'5504 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:14.484 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:14.477+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5565 (115'2505,123'5565] local-lis/les=21/22 n=953 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5563 crt=123'5563 lcod 123'5561 mlcod 123'5561 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:14.485 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:14.477+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5508 (115'2506,123'5508] local-lis/les=21/22 n=761 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5506 crt=123'5506 lcod 123'5504 mlcod 123'5504 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:14.489 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:14.486+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5525 (115'2507,123'5525] local-lis/les=21/22 n=870 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5524 crt=123'5520 lcod 123'5522 mlcod 123'5522 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:14.492 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:14.490+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5527 (115'2507,123'5527] local-lis/les=21/22 n=868 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5524 crt=123'5524 lcod 123'5522 mlcod 123'5522 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:14.494 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:14.490+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5674 (115'2611,123'5674] local-lis/les=21/22 n=987 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'5673 crt=123'5673 lcod 123'5671 mlcod 123'5671 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:14.496 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:14.489+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5510 (115'2506,123'5510] local-lis/les=21/22 n=759 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5510 lcod 123'5508 mlcod 123'5508 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:14.497 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:14.489+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5567 (115'2505,123'5567] local-lis/les=21/22 n=951 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=123'5567 lcod 123'5565 mlcod 123'5565 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:14.497 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:14.494+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5676 (115'2611,123'5676] local-lis/les=21/22 n=985 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'5673 crt=123'5673 lcod 123'5671 mlcod 123'5671 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:14.500 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:14.493+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5512 (115'2506,123'5512] local-lis/les=21/22 n=757 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5510 crt=123'5510 lcod 123'5508 mlcod 123'5508 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:14.500 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:14.498+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5529 (115'2507,123'5529] local-lis/les=21/22 n=866 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5529 lcod 123'5527 mlcod 123'5527 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:14.500 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:14.493+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5569 (115'2505,123'5569] local-lis/les=21/22 n=949 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5567 crt=123'5567 lcod 123'5565 mlcod 123'5565 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:14.503 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:14.498+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5531 (115'2507,123'5531] local-lis/les=21/22 n=864 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5529 crt=123'5529 lcod 123'5527 mlcod 123'5527 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:14.509 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:14.506+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5533 (115'2507,123'5533] local-lis/les=21/22 n=862 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5533 lcod 123'5531 mlcod 123'5531 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:14.510 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:14.506+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5678 (115'2611,123'5678] local-lis/les=21/22 n=983 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'5678 lcod 123'5676 mlcod 123'5676 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:14.510 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:14.501+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5514 (115'2506,123'5514] local-lis/les=21/22 n=755 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5514 lcod 123'5512 mlcod 123'5512 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:14.511 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:14.505+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5571 (115'2505,123'5571] local-lis/les=21/22 n=947 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=123'5571 lcod 123'5569 mlcod 123'5569 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:14.512 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:14.506+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5535 (115'2507,123'5535] local-lis/les=21/22 n=860 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5533 crt=123'5533 lcod 123'5531 mlcod 123'5531 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:14.513 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:14.505+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5573 (115'2505,123'5573] local-lis/les=21/22 n=945 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5571 crt=123'5571 lcod 123'5569 mlcod 123'5569 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:14.515 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:14.505+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5516 (115'2506,123'5516] local-lis/les=21/22 n=753 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5514 crt=123'5514 lcod 123'5512 mlcod 123'5512 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:14.517 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:14.506+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5680 (115'2611,123'5680] local-lis/les=21/22 n=981 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'5678 crt=123'5678 lcod 123'5676 mlcod 123'5676 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:14.528 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:14.517+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5518 (115'2506,123'5518] local-lis/les=21/22 n=751 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5518 lcod 123'5516 mlcod 123'5516 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:14.528 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:14.517+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5520 (115'2506,123'5520] local-lis/les=21/22 n=749 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5518 crt=123'5518 lcod 123'5516 mlcod 123'5516 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:14.529 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:14.514+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5537 (115'2507,123'5537] local-lis/les=21/22 n=858 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5537 lcod 123'5535 mlcod 123'5535 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:14.529 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:14.518+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5539 (115'2507,123'5539] local-lis/les=21/22 n=856 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5537 crt=123'5537 lcod 123'5535 mlcod 123'5535 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:14.529 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:14.518+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5682 (115'2611,123'5682] local-lis/les=21/22 n=979 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'5682 lcod 123'5680 mlcod 123'5680 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:14.529 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:14.522+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5684 (115'2611,123'5684] local-lis/les=21/22 n=977 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'5682 crt=123'5682 lcod 123'5680 mlcod 123'5680 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:14.530 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:14.526+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5541 (115'2507,123'5541] local-lis/les=21/22 n=854 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5541 lcod 123'5539 mlcod 123'5539 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:14.533 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:14.525+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5522 (115'2506,123'5522] local-lis/les=21/22 n=747 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5522 lcod 123'5520 mlcod 123'5520 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:14.534 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:14.530+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5543 (115'2507,123'5543] local-lis/les=21/22 n=852 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5541 crt=123'5541 lcod 123'5539 mlcod 123'5539 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:14.535 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:14.529+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5524 (115'2506,123'5524] local-lis/les=21/22 n=745 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5522 crt=123'5522 lcod 123'5520 mlcod 123'5520 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:14.537 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:14.530+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5686 (115'2611,123'5686] local-lis/les=21/22 n=975 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'5686 lcod 123'5684 mlcod 123'5684 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:14.537 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:14.534+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5688 (115'2611,123'5688] local-lis/les=21/22 n=973 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'5686 crt=123'5686 lcod 123'5684 mlcod 123'5684 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:14.540 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:14.533+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5526 (115'2506,123'5526] local-lis/les=21/22 n=743 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5526 lcod 123'5524 mlcod 123'5524 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:14.543 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:14.537+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5528 (115'2506,123'5528] local-lis/les=21/22 n=741 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5526 crt=123'5526 lcod 123'5524 mlcod 123'5524 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:14.543 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:14.538+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5545 (115'2507,123'5545] local-lis/les=21/22 n=850 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5545 lcod 123'5543 mlcod 123'5543 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:14.546 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:14.542+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5547 (115'2507,123'5547] local-lis/les=21/22 n=848 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5545 crt=123'5545 lcod 123'5543 mlcod 123'5543 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:14.549 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:14.541+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5530 (115'2506,123'5530] local-lis/les=21/22 n=739 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5530 lcod 123'5528 mlcod 123'5528 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:14.551 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:14.545+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5532 (115'2506,123'5532] local-lis/les=21/22 n=737 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5530 crt=123'5530 lcod 123'5528 mlcod 123'5528 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:14.553 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:14.550+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5549 (115'2507,123'5549] local-lis/les=21/22 n=846 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5549 lcod 123'5547 mlcod 123'5547 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:14.554 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:14.550+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5785 (115'2704,123'5785] local-lis/les=21/22 n=680 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5785 lcod 123'5783 mlcod 123'5783 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:14.557 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:14.554+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5787 (115'2704,123'5787] local-lis/les=21/22 n=678 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5785 crt=123'5785 lcod 123'5783 mlcod 123'5783 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:14.557 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:14.554+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5522 (115'2504,123'5522] local-lis/les=21/22 n=697 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5522 lcod 123'5520 mlcod 123'5520 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:14.559 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:14.554+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5551 (115'2507,123'5551] local-lis/les=21/22 n=844 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5549 crt=123'5549 lcod 123'5547 mlcod 123'5547 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:14.561 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:14.558+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5524 (115'2504,123'5524] local-lis/les=21/22 n=695 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5522 crt=123'5522 lcod 123'5520 mlcod 123'5520 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:14.561 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:14.558+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5690 (115'2611,123'5690] local-lis/les=21/22 n=971 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'5690 lcod 123'5688 mlcod 123'5688 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:14.564 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:14.557+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5534 (115'2506,123'5534] local-lis/les=21/22 n=735 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5534 lcod 123'5532 mlcod 123'5532 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:14.564 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:14.557+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5536 (115'2506,123'5536] local-lis/les=21/22 n=733 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5534 crt=123'5534 lcod 123'5532 mlcod 123'5532 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:14.565 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:14.562+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5692 (115'2611,123'5692] local-lis/les=21/22 n=969 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'5690 crt=123'5690 lcod 123'5688 mlcod 123'5688 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:14.572 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:14.566+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5789 (115'2704,123'5789] local-lis/les=21/22 n=676 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5789 lcod 123'5787 mlcod 123'5787 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:14.572 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:14.566+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5694 (115'2611,123'5694] local-lis/les=21/22 n=967 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'5694 lcod 123'5692 mlcod 123'5692 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:14.572 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:14.566+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5553 (115'2507,123'5553] local-lis/les=21/22 n=842 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5553 lcod 123'5551 mlcod 123'5551 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:14.575 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:14.569+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5538 (115'2506,123'5538] local-lis/les=21/22 n=731 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5538 lcod 123'5536 mlcod 123'5536 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:14.576 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:14.570+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5696 (115'2611,123'5696] local-lis/les=21/22 n=965 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'5694 crt=123'5694 lcod 123'5692 mlcod 123'5692 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:14.577 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:14.570+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5526 (115'2504,123'5526] local-lis/les=21/22 n=693 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5526 lcod 123'5524 mlcod 123'5524 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:14.578 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:14.569+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5540 (115'2506,123'5540] local-lis/les=21/22 n=729 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5538 crt=123'5538 lcod 123'5536 mlcod 123'5536 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:14.579 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:14.574+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5555 (115'2507,123'5555] local-lis/les=21/22 n=840 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5553 crt=123'5553 lcod 123'5551 mlcod 123'5551 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:14.579 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:14.574+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5791 (115'2704,123'5791] local-lis/les=21/22 n=674 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5789 crt=123'5789 lcod 123'5787 mlcod 123'5787 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:14.582 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:14.578+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5528 (115'2504,123'5528] local-lis/les=21/22 n=691 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5526 crt=123'5526 lcod 123'5524 mlcod 123'5524 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:14.587 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:14.582+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5619 (115'2605,123'5619] local-lis/les=21/22 n=770 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5619 lcod 123'5617 mlcod 123'5617 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:14.588 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:14.586+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5698 (115'2611,123'5698] local-lis/les=21/22 n=963 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'5698 lcod 123'5696 mlcod 123'5696 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:14.589 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:14.586+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5557 (115'2507,123'5557] local-lis/les=21/22 n=838 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5557 lcod 123'5555 mlcod 123'5555 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:14.590 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:14.586+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5621 (115'2605,123'5621] local-lis/les=21/22 n=768 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5619 crt=123'5619 lcod 123'5617 mlcod 123'5617 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:14.591 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:14.586+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5700 (115'2611,123'5700] local-lis/les=21/22 n=961 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'5698 crt=123'5698 lcod 123'5696 mlcod 123'5696 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:14.597 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:14.585+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5542 (115'2506,123'5542] local-lis/les=21/22 n=727 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5542 lcod 123'5540 mlcod 123'5540 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:14.597 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:14.594+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5623 (115'2605,123'5623] local-lis/les=21/22 n=766 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5623 lcod 123'5621 mlcod 123'5621 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:14.598 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:14.589+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5544 (115'2506,123'5544] local-lis/les=21/22 n=725 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5542 crt=123'5542 lcod 123'5540 mlcod 123'5540 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:14.601 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:14.598+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5793 (115'2704,123'5793] local-lis/les=21/22 n=672 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5793 lcod 123'5791 mlcod 123'5791 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:14.603 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:14.598+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5625 (115'2605,123'5625] local-lis/les=21/22 n=764 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5623 crt=123'5623 lcod 123'5621 mlcod 123'5621 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:14.605 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:14.594+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5702 (115'2611,123'5702] local-lis/les=21/22 n=959 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'5702 lcod 123'5700 mlcod 123'5700 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:14.605 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:14.594+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5530 (115'2504,123'5530] local-lis/les=21/22 n=689 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5530 lcod 123'5528 mlcod 123'5528 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:14.606 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:14.598+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5704 (115'2611,123'5704] local-lis/les=21/22 n=957 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'5702 crt=123'5702 lcod 123'5700 mlcod 123'5700 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:14.606 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:14.602+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5559 (115'2507,123'5559] local-lis/les=21/22 n=836 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5557 crt=123'5557 lcod 123'5555 mlcod 123'5555 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:14.608 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:14.606+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5795 (115'2704,123'5795] local-lis/les=21/22 n=670 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5793 crt=123'5793 lcod 123'5791 mlcod 123'5791 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:14.608 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:14.601+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5546 (115'2506,123'5546] local-lis/les=21/22 n=723 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5546 lcod 123'5544 mlcod 123'5544 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:14.613 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:14.610+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5532 (115'2504,123'5532] local-lis/les=21/22 n=687 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5532 lcod 123'5530 mlcod 123'5530 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:14.616 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:14.610+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5534 (115'2504,123'5534] local-lis/les=21/22 n=685 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5532 crt=123'5532 lcod 123'5530 mlcod 123'5530 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:14.616 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:14.610+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5706 (115'2611,123'5706] local-lis/les=21/22 n=955 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'5706 lcod 123'5704 mlcod 123'5704 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:14.617 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:14.610+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5627 (115'2605,123'5627] local-lis/les=21/22 n=762 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5627 lcod 123'5625 mlcod 123'5625 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:14.618 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:14.614+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5629 (115'2605,123'5629] local-lis/les=21/22 n=760 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5627 crt=123'5627 lcod 123'5625 mlcod 123'5625 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:14.620 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:14.614+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5561 (115'2507,123'5561] local-lis/les=21/22 n=834 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5561 lcod 123'5559 mlcod 123'5559 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:14.621 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:14.618+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5797 (115'2704,123'5797] local-lis/les=21/22 n=668 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5797 lcod 123'5795 mlcod 123'5795 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:14.622 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:14.618+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5563 (115'2507,123'5563] local-lis/les=21/22 n=832 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5561 crt=123'5561 lcod 123'5559 mlcod 123'5559 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:14.623 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:14.617+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5548 (115'2506,123'5548] local-lis/les=21/22 n=721 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5548 lcod 123'5546 mlcod 123'5546 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:14.624 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:14.622+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5708 (115'2611,123'5708] local-lis/les=21/22 n=953 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'5706 crt=123'5706 lcod 123'5704 mlcod 123'5704 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:14.625 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:14.617+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5550 (115'2506,123'5550] local-lis/les=21/22 n=719 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5548 crt=123'5548 lcod 123'5546 mlcod 123'5546 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:14.627 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:14.622+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5799 (115'2704,123'5799] local-lis/les=21/22 n=666 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5797 crt=123'5797 lcod 123'5795 mlcod 123'5795 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:14.633 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:14.630+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5565 (115'2507,123'5565] local-lis/les=21/22 n=830 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5565 lcod 123'5563 mlcod 123'5563 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:14.635 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:14.630+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5536 (115'2504,123'5536] local-lis/les=21/22 n=683 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5536 lcod 123'5534 mlcod 123'5534 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:14.636 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:14.630+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5631 (115'2605,123'5631] local-lis/les=21/22 n=758 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5631 lcod 123'5629 mlcod 123'5629 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:14.636 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:14.630+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5567 (115'2507,123'5567] local-lis/les=21/22 n=828 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5565 crt=123'5565 lcod 123'5563 mlcod 123'5563 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:14.637 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:14.634+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5633 (115'2605,123'5633] local-lis/les=21/22 n=756 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5631 crt=123'5631 lcod 123'5629 mlcod 123'5629 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:14.643 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:14.634+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5538 (115'2504,123'5538] local-lis/les=21/22 n=681 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5536 crt=123'5536 lcod 123'5534 mlcod 123'5534 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:14.644 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:14.638+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5710 (115'2611,123'5710] local-lis/les=21/22 n=951 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'5710 lcod 123'5708 mlcod 123'5708 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:14.644 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:14.637+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5552 (115'2506,123'5552] local-lis/les=21/22 n=717 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5552 lcod 123'5550 mlcod 123'5550 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:14.645 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:14.638+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5569 (115'2507,123'5569] local-lis/les=21/22 n=826 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5569 lcod 123'5567 mlcod 123'5567 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:14.645 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:14.642+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5635 (115'2605,123'5635] local-lis/les=21/22 n=754 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5635 lcod 123'5633 mlcod 123'5633 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:14.646 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:14.642+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5571 (115'2507,123'5571] local-lis/les=21/22 n=824 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5569 crt=123'5569 lcod 123'5567 mlcod 123'5567 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:14.647 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:14.642+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5637 (115'2605,123'5637] local-lis/les=21/22 n=752 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5635 crt=123'5635 lcod 123'5633 mlcod 123'5633 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:14.647 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:14.642+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5712 (115'2611,123'5712] local-lis/les=21/22 n=949 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'5710 crt=123'5710 lcod 123'5708 mlcod 123'5708 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:14.650 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:14.641+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5554 (115'2506,123'5554] local-lis/les=21/22 n=715 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5552 crt=123'5552 lcod 123'5550 mlcod 123'5550 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:14.654 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:14.650+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5540 (115'2504,123'5540] local-lis/les=21/22 n=679 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5540 lcod 123'5538 mlcod 123'5538 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:14.655 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:14.646+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5801 (115'2704,123'5801] local-lis/les=21/22 n=664 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5801 lcod 123'5799 mlcod 123'5799 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:14.655 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:14.650+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5803 (115'2704,123'5803] local-lis/les=21/22 n=662 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5801 crt=123'5801 lcod 123'5799 mlcod 123'5799 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:14.657 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:14.654+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5542 (115'2504,123'5542] local-lis/les=21/22 n=677 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5540 crt=123'5540 lcod 123'5538 mlcod 123'5538 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:14.660 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:14.658+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5573 (115'2507,123'5573] local-lis/les=21/22 n=822 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5573 lcod 123'5571 mlcod 123'5571 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:14.662 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:14.653+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5556 (115'2506,123'5556] local-lis/les=21/22 n=713 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5556 lcod 123'5554 mlcod 123'5554 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:14.663 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:14.658+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5639 (115'2605,123'5639] local-lis/les=21/22 n=750 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5639 lcod 123'5637 mlcod 123'5637 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:14.664 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:14.657+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5558 (115'2506,123'5558] local-lis/les=21/22 n=711 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5556 crt=123'5556 lcod 123'5554 mlcod 123'5554 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:14.670 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:14.662+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5805 (115'2704,123'5805] local-lis/les=21/22 n=660 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5805 lcod 123'5803 mlcod 123'5803 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:14.671 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:14.666+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5544 (115'2504,123'5544] local-lis/les=21/22 n=675 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5544 lcod 123'5542 mlcod 123'5542 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:14.671 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:14.662+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5575 (115'2507,123'5575] local-lis/les=21/22 n=820 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5573 crt=123'5573 lcod 123'5571 mlcod 123'5571 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:14.672 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:14.662+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5641 (115'2605,123'5641] local-lis/les=21/22 n=748 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5639 crt=123'5639 lcod 123'5637 mlcod 123'5637 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:14.673 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:14.666+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5546 (115'2504,123'5546] local-lis/les=21/22 n=673 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5544 crt=123'5544 lcod 123'5542 mlcod 123'5542 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:14.676 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:14.670+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5807 (115'2805,123'5807] local-lis/les=21/22 n=658 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5805 crt=123'5805 lcod 123'5803 mlcod 123'5803 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:14.677 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:14.670+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5577 (115'2507,123'5577] local-lis/les=21/22 n=818 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5577 lcod 123'5575 mlcod 123'5575 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:14.678 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:14.674+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5579 (115'2507,123'5579] local-lis/les=21/22 n=816 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5577 crt=123'5577 lcod 123'5575 mlcod 123'5575 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:14.679 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:14.674+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5643 (115'2605,123'5643] local-lis/les=21/22 n=746 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5643 lcod 123'5641 mlcod 123'5641 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:14.679 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:14.673+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5560 (115'2506,123'5560] local-lis/les=21/22 n=709 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5560 lcod 123'5558 mlcod 123'5558 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:14.681 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:14.673+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5562 (115'2506,123'5562] local-lis/les=21/22 n=707 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5560 crt=123'5560 lcod 123'5558 mlcod 123'5558 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:14.682 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:14.678+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5645 (115'2605,123'5645] local-lis/les=21/22 n=744 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5643 crt=123'5643 lcod 123'5641 mlcod 123'5641 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:14.688 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:14.682+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5809 (115'2805,123'5809] local-lis/les=21/22 n=656 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5809 lcod 123'5807 mlcod 123'5807 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:14.688 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:14.686+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5581 (115'2507,123'5581] local-lis/les=21/22 n=814 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5581 lcod 123'5579 mlcod 123'5579 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:14.690 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:14.686+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5811 (115'2805,123'5811] local-lis/les=21/22 n=654 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5809 crt=123'5809 lcod 123'5807 mlcod 123'5807 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:14.695 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:14.690+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5647 (115'2605,123'5647] local-lis/les=21/22 n=742 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5647 lcod 123'5645 mlcod 123'5645 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:14.695 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:14.686+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5548 (115'2504,123'5548] local-lis/les=21/22 n=671 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5548 lcod 123'5546 mlcod 123'5546 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:14.696 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:14.690+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5550 (115'2504,123'5550] local-lis/les=21/22 n=669 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5548 crt=123'5548 lcod 123'5546 mlcod 123'5546 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:14.696 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:14.689+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5564 (115'2506,123'5564] local-lis/les=21/22 n=705 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5564 lcod 123'5562 mlcod 123'5562 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:14.697 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:14.690+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5649 (115'2605,123'5649] local-lis/les=21/22 n=740 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5647 crt=123'5647 lcod 123'5645 mlcod 123'5645 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:14.698 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:14.690+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5583 (115'2507,123'5583] local-lis/les=21/22 n=812 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5581 crt=123'5581 lcod 123'5579 mlcod 123'5579 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:14.701 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:14.698+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5813 (115'2805,123'5813] local-lis/les=21/22 n=652 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5813 lcod 123'5811 mlcod 123'5811 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:14.702 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:14.693+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5566 (115'2506,123'5566] local-lis/les=21/22 n=703 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5564 crt=123'5564 lcod 123'5562 mlcod 123'5562 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:14.703 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:14.698+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5651 (115'2605,123'5651] local-lis/les=21/22 n=738 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5651 lcod 123'5649 mlcod 123'5649 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:14.705 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:14.702+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5585 (115'2507,123'5585] local-lis/les=21/22 n=810 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5585 lcod 123'5583 mlcod 123'5583 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:14.705 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:14.702+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5552 (115'2504,123'5552] local-lis/les=21/22 n=667 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5552 lcod 123'5550 mlcod 123'5550 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:14.706 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:14.702+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5653 (115'2605,123'5653] local-lis/les=21/22 n=736 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5651 crt=123'5651 lcod 123'5649 mlcod 123'5649 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:14.707 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:14.702+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5587 (115'2507,123'5587] local-lis/les=21/22 n=808 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5585 crt=123'5585 lcod 123'5583 mlcod 123'5583 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:14.710 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:14.702+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5554 (115'2504,123'5554] local-lis/les=21/22 n=665 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5552 crt=123'5552 lcod 123'5550 mlcod 123'5550 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:14.712 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:14.705+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5568 (115'2506,123'5568] local-lis/les=21/22 n=701 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5568 lcod 123'5566 mlcod 123'5566 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:14.714 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:14.705+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5570 (115'2506,123'5570] local-lis/les=21/22 n=699 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5568 crt=123'5568 lcod 123'5566 mlcod 123'5566 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:14.715 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:14.710+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5589 (115'2507,123'5589] local-lis/les=21/22 n=806 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5589 lcod 123'5587 mlcod 123'5587 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:14.717 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:14.714+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5591 (115'2507,123'5591] local-lis/les=21/22 n=804 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5589 crt=123'5589 lcod 123'5587 mlcod 123'5587 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:14.719 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:14.713+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5572 (115'2506,123'5572] local-lis/les=21/22 n=697 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5572 lcod 123'5570 mlcod 123'5570 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:14.721 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:14.713+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5574 (115'2506,123'5574] local-lis/les=21/22 n=695 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5572 crt=123'5572 lcod 123'5570 mlcod 123'5570 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:14.722 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:14.718+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5593 (115'2507,123'5593] local-lis/les=21/22 n=802 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5593 lcod 123'5591 mlcod 123'5591 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:14.723 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:14.718+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5595 (115'2507,123'5595] local-lis/les=21/22 n=800 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5593 crt=123'5593 lcod 123'5591 mlcod 123'5591 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:14.728 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:14.721+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5576 (115'2506,123'5576] local-lis/les=21/22 n=693 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5576 lcod 123'5574 mlcod 123'5574 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:14.728 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:14.721+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5578 (115'2506,123'5578] local-lis/les=21/22 n=691 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5576 crt=123'5576 lcod 123'5574 mlcod 123'5574 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:14.729 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:14.722+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5597 (115'2507,123'5597] local-lis/les=21/22 n=798 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5597 lcod 123'5595 mlcod 123'5595 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:14.729 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:14.726+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5599 (115'2507,123'5599] local-lis/les=21/22 n=796 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5597 crt=123'5597 lcod 123'5595 mlcod 123'5595 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:14.734 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:14.729+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5580 (115'2506,123'5580] local-lis/les=21/22 n=689 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5580 lcod 123'5578 mlcod 123'5578 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:14.735 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:14.730+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5601 (115'2507,123'5601] local-lis/les=21/22 n=794 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5601 lcod 123'5599 mlcod 123'5599 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:14.736 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:14.729+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5582 (115'2506,123'5582] local-lis/les=21/22 n=687 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5580 crt=123'5580 lcod 123'5578 mlcod 123'5578 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:14.737 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:14.734+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5603 (115'2507,123'5603] local-lis/les=21/22 n=792 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5601 crt=123'5601 lcod 123'5599 mlcod 123'5599 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:14.743 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:14.738+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5605 (115'2507,123'5605] local-lis/les=21/22 n=790 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5605 lcod 123'5603 mlcod 123'5603 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:14.743 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:14.737+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5584 (115'2506,123'5584] local-lis/les=21/22 n=685 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5584 lcod 123'5582 mlcod 123'5582 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:14.744 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:14.742+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5607 (115'2507,123'5607] local-lis/les=21/22 n=788 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5605 crt=123'5605 lcod 123'5603 mlcod 123'5603 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:14.745 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:14.737+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5586 (115'2506,123'5586] local-lis/les=21/22 n=683 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5584 crt=123'5584 lcod 123'5582 mlcod 123'5582 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:14.753 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:14.750+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5609 (115'2607,123'5609] local-lis/les=21/22 n=786 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5609 lcod 123'5607 mlcod 123'5607 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:14.753 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:14.745+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5588 (115'2506,123'5588] local-lis/les=21/22 n=681 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5588 lcod 123'5586 mlcod 123'5586 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:14.754 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:14.750+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5611 (115'2607,123'5611] local-lis/les=21/22 n=784 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5609 crt=123'5609 lcod 123'5607 mlcod 123'5607 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:14.755 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:14.749+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5590 (115'2506,123'5590] local-lis/les=21/22 n=679 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5588 crt=123'5588 lcod 123'5586 mlcod 123'5586 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:14.760 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:14.753+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5592 (115'2506,123'5592] local-lis/les=21/22 n=677 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5592 lcod 123'5590 mlcod 123'5590 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:14.761 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:14.758+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5613 (115'2607,123'5613] local-lis/les=21/22 n=782 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5613 lcod 123'5611 mlcod 123'5611 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:14.762 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:14.758+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5615 (115'2607,123'5615] local-lis/les=21/22 n=780 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5613 crt=123'5613 lcod 123'5611 mlcod 123'5611 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:14.763 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:14.757+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5594 (115'2506,123'5594] local-lis/les=21/22 n=675 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5592 crt=123'5592 lcod 123'5590 mlcod 123'5590 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:14.768 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:14.761+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5596 (115'2506,123'5596] local-lis/les=21/22 n=673 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5596 lcod 123'5594 mlcod 123'5594 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:14.768 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:14.766+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5617 (115'2607,123'5617] local-lis/les=21/22 n=778 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5617 lcod 123'5615 mlcod 123'5615 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:14.769 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:14.761+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5598 (115'2506,123'5598] local-lis/les=21/22 n=671 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5596 crt=123'5596 lcod 123'5594 mlcod 123'5594 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:14.773 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:14.765+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5600 (115'2506,123'5600] local-lis/les=21/22 n=669 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5600 lcod 123'5598 mlcod 123'5598 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:14.774 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:14.770+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5619 (115'2607,123'5619] local-lis/les=21/22 n=776 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5619 lcod 123'5617 mlcod 123'5617 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:14.775 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:14.769+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5602 (115'2506,123'5602] local-lis/les=21/22 n=667 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5600 crt=123'5600 lcod 123'5598 mlcod 123'5598 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:14.777 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:14.774+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5621 (115'2607,123'5621] local-lis/les=21/22 n=774 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5619 crt=123'5619 lcod 123'5617 mlcod 123'5617 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:14.783 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:14.778+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5623 (115'2607,123'5623] local-lis/les=21/22 n=772 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5623 lcod 123'5621 mlcod 123'5621 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:14.783 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:14.777+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5604 (115'2506,123'5604] local-lis/les=21/22 n=665 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5604 lcod 123'5602 mlcod 123'5602 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:14.785 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:14.782+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5625 (115'2607,123'5625] local-lis/les=21/22 n=770 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5623 crt=123'5623 lcod 123'5621 mlcod 123'5621 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:14.785 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:14.777+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5606 (115'2506,123'5606] local-lis/les=21/22 n=663 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5604 crt=123'5604 lcod 123'5602 mlcod 123'5602 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:14.792 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:14.785+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5608 (115'2606,123'5608] local-lis/les=21/22 n=661 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5608 lcod 123'5606 mlcod 123'5606 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:14.792 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:14.790+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5627 (115'2607,123'5627] local-lis/les=21/22 n=768 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5627 lcod 123'5625 mlcod 123'5625 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:14.793 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:14.785+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5610 (115'2606,123'5610] local-lis/les=21/22 n=659 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5608 crt=123'5608 lcod 123'5606 mlcod 123'5606 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:14.793 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:14.790+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5629 (115'2607,123'5629] local-lis/les=21/22 n=766 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5627 crt=123'5627 lcod 123'5625 mlcod 123'5625 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:14.804 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:14.798+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5631 (115'2607,123'5631] local-lis/les=21/22 n=764 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5631 lcod 123'5629 mlcod 123'5629 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:14.804 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:14.797+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5612 (115'2606,123'5612] local-lis/les=21/22 n=657 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5612 lcod 123'5610 mlcod 123'5610 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:14.806 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:14.802+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5633 (115'2607,123'5633] local-lis/les=21/22 n=762 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5631 crt=123'5631 lcod 123'5629 mlcod 123'5629 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:14.811 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:14.805+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5614 (115'2606,123'5614] local-lis/les=21/22 n=655 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5614 lcod 123'5612 mlcod 123'5612 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:14.811 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:14.806+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5635 (115'2607,123'5635] local-lis/les=21/22 n=760 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5635 lcod 123'5633 mlcod 123'5633 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:14.812 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:14.810+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5637 (115'2607,123'5637] local-lis/les=21/22 n=758 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5635 crt=123'5635 lcod 123'5633 mlcod 123'5633 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:14.813 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:14.805+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5616 (115'2606,123'5616] local-lis/les=21/22 n=653 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5614 crt=123'5614 lcod 123'5612 mlcod 123'5612 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:14.815 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:14.810+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5556 (115'2504,123'5556] local-lis/les=21/22 n=663 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5556 lcod 123'5554 mlcod 123'5554 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:14.818 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:14.814+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5558 (115'2504,123'5558] local-lis/les=21/22 n=661 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5556 crt=123'5556 lcod 123'5554 mlcod 123'5554 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:14.819 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:14.814+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6008 (115'2909,123'6008] local-lis/les=21/22 n=681 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'6008 lcod 123'6006 mlcod 123'6006 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:14.824 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:14.818+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6010 (115'2909,123'6010] local-lis/les=21/22 n=679 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6008 crt=123'6008 lcod 123'6006 mlcod 123'6006 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:14.825 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:14.822+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5560 (115'2504,123'5560] local-lis/les=21/22 n=659 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5560 lcod 123'5558 mlcod 123'5558 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:14.826 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:14.817+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5618 (115'2606,123'5618] local-lis/les=21/22 n=651 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5618 lcod 123'5616 mlcod 123'5616 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:14.827 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:14.821+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5620 (115'2606,123'5620] local-lis/les=21/22 n=649 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5618 crt=123'5618 lcod 123'5616 mlcod 123'5616 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:14.831 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:14.822+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5562 (115'2504,123'5562] local-lis/les=21/22 n=657 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5560 crt=123'5560 lcod 123'5558 mlcod 123'5558 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:14.831 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:14.822+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5639 (115'2607,123'5639] local-lis/les=21/22 n=756 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5639 lcod 123'5637 mlcod 123'5637 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:14.831 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:14.826+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5641 (115'2607,123'5641] local-lis/les=21/22 n=754 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5639 crt=123'5639 lcod 123'5637 mlcod 123'5637 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:14.832 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:14.822+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5815 (115'2805,123'5815] local-lis/les=21/22 n=650 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5815 lcod 123'5813 mlcod 123'5813 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:14.832 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:14.826+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5817 (115'2805,123'5817] local-lis/les=21/22 n=648 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5815 crt=123'5815 lcod 123'5813 mlcod 123'5813 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:14.835 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:14.830+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5655 (115'2605,123'5655] local-lis/les=21/22 n=734 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5655 lcod 123'5653 mlcod 123'5653 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:14.837 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:14.834+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5564 (115'2504,123'5564] local-lis/les=21/22 n=655 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5564 lcod 123'5562 mlcod 123'5562 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:14.838 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:14.834+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5657 (115'2605,123'5657] local-lis/les=21/22 n=732 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5655 crt=123'5655 lcod 123'5653 mlcod 123'5653 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:14.838 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:14.834+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5566 (115'2504,123'5566] local-lis/les=21/22 n=653 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5564 crt=123'5564 lcod 123'5562 mlcod 123'5562 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:14.843 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:14.837+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5622 (115'2606,123'5622] local-lis/les=21/22 n=647 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5622 lcod 123'5620 mlcod 123'5620 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:14.844 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:14.837+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5624 (115'2606,123'5624] local-lis/les=21/22 n=645 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5622 crt=123'5622 lcod 123'5620 mlcod 123'5620 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:14.845 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:14.838+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6012 (116'3010,123'6012] local-lis/les=21/22 n=677 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'6012 lcod 123'6010 mlcod 123'6010 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:14.848 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:14.842+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5659 (115'2605,123'5659] local-lis/les=21/22 n=730 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5659 lcod 123'5657 mlcod 123'5657 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:14.849 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:14.842+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5661 (115'2605,123'5661] local-lis/les=21/22 n=728 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5659 crt=123'5659 lcod 123'5657 mlcod 123'5657 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:14.849 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:14.842+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5568 (115'2504,123'5568] local-lis/les=21/22 n=651 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5568 lcod 123'5566 mlcod 123'5566 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:14.849 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:14.842+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6014 (116'3010,123'6014] local-lis/les=21/22 n=675 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6012 crt=123'6012 lcod 123'6010 mlcod 123'6010 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:14.850 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:14.842+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5819 (115'2805,123'5819] local-lis/les=21/22 n=646 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5819 lcod 123'5817 mlcod 123'5817 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:14.851 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:14.842+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5821 (115'2805,123'5821] local-lis/les=21/22 n=644 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5819 crt=123'5819 lcod 123'5817 mlcod 123'5817 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:14.854 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:14.850+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5570 (115'2504,123'5570] local-lis/les=21/22 n=649 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5568 crt=123'5568 lcod 123'5566 mlcod 123'5566 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:14.856 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:14.854+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6016 (116'3010,123'6016] local-lis/les=21/22 n=673 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'6016 lcod 123'6014 mlcod 123'6014 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:14.858 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:14.853+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5626 (115'2606,123'5626] local-lis/les=21/22 n=643 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5626 lcod 123'5624 mlcod 123'5624 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:14.860 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:14.854+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5823 (115'2805,123'5823] local-lis/les=21/22 n=642 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5823 lcod 123'5821 mlcod 123'5821 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:14.861 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:14.858+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6018 (116'3010,123'6018] local-lis/les=21/22 n=671 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6016 crt=123'6016 lcod 123'6014 mlcod 123'6014 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:14.861 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:14.858+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5825 (115'2805,123'5825] local-lis/les=21/22 n=640 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5823 crt=123'5823 lcod 123'5821 mlcod 123'5821 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:14.862 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:14.854+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5663 (115'2605,123'5663] local-lis/les=21/22 n=726 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5663 lcod 123'5661 mlcod 123'5661 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:14.862 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:14.858+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5665 (115'2605,123'5665] local-lis/les=21/22 n=724 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5663 crt=123'5663 lcod 123'5661 mlcod 123'5661 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:14.864 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:14.857+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5628 (115'2606,123'5628] local-lis/les=21/22 n=641 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5626 crt=123'5626 lcod 123'5624 mlcod 123'5624 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:14.865 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:14.862+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5572 (115'2504,123'5572] local-lis/les=21/22 n=647 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5572 lcod 123'5570 mlcod 123'5570 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:14.868 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:14.866+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5574 (115'2504,123'5574] local-lis/les=21/22 n=645 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5572 crt=123'5572 lcod 123'5570 mlcod 123'5570 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:14.870 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:14.866+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6020 (116'3010,123'6020] local-lis/les=21/22 n=669 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'6020 lcod 123'6018 mlcod 123'6018 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:14.871 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:14.866+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6022 (116'3010,123'6022] local-lis/les=21/22 n=667 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6020 crt=123'6020 lcod 123'6018 mlcod 123'6018 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:14.873 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:14.870+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5576 (115'2504,123'5576] local-lis/les=21/22 n=643 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5576 lcod 123'5574 mlcod 123'5574 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:14.875 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:14.870+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5578 (115'2504,123'5578] local-lis/les=21/22 n=641 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5576 crt=123'5576 lcod 123'5574 mlcod 123'5574 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:14.877 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:14.874+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5827 (115'2805,123'5827] local-lis/les=21/22 n=638 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5827 lcod 123'5825 mlcod 123'5825 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:14.878 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:14.874+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5829 (115'2805,123'5829] local-lis/les=21/22 n=636 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5827 crt=123'5827 lcod 123'5825 mlcod 123'5825 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:14.880 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:14.878+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5580 (115'2504,123'5580] local-lis/les=21/22 n=639 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5580 lcod 123'5578 mlcod 123'5578 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:14.883 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:14.878+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5582 (115'2504,123'5582] local-lis/les=21/22 n=637 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5580 crt=123'5580 lcod 123'5578 mlcod 123'5578 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:14.883 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:14.878+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6024 (116'3010,123'6024] local-lis/les=21/22 n=665 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'6024 lcod 123'6022 mlcod 123'6022 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:14.886 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:14.882+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6026 (116'3010,123'6026] local-lis/les=21/22 n=663 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6024 crt=123'6024 lcod 123'6022 mlcod 123'6022 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:14.887 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:14.882+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5831 (115'2805,123'5831] local-lis/les=21/22 n=634 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5831 lcod 123'5829 mlcod 123'5829 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:14.889 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:14.886+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5833 (115'2805,123'5833] local-lis/les=21/22 n=632 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5831 crt=123'5831 lcod 123'5829 mlcod 123'5829 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:14.894 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:14.890+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6028 (116'3010,123'6028] local-lis/les=21/22 n=661 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'6028 lcod 123'6026 mlcod 123'6026 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:14.895 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:14.890+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6030 (116'3010,123'6030] local-lis/les=21/22 n=659 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6028 crt=123'6028 lcod 123'6026 mlcod 123'6026 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:14.898 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:14.894+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5835 (115'2805,123'5835] local-lis/les=21/22 n=630 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5835 lcod 123'5833 mlcod 123'5833 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:14.901 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:14.898+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5837 (115'2805,123'5837] local-lis/les=21/22 n=628 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5835 crt=123'5835 lcod 123'5833 mlcod 123'5833 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:14.902 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:14.898+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6032 (116'3010,123'6032] local-lis/les=21/22 n=657 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'6032 lcod 123'6030 mlcod 123'6030 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:14.904 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:14.902+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6034 (116'3010,123'6034] local-lis/les=21/22 n=655 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6032 crt=123'6032 lcod 123'6030 mlcod 123'6030 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:14.908 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:14.906+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5839 (115'2805,123'5839] local-lis/les=21/22 n=626 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5839 lcod 123'5837 mlcod 123'5837 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:14.909 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:14.906+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5841 (115'2805,123'5841] local-lis/les=21/22 n=624 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5839 crt=123'5839 lcod 123'5837 mlcod 123'5837 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:14.913 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:14.910+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6036 (116'3010,123'6036] local-lis/les=21/22 n=653 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'6036 lcod 123'6034 mlcod 123'6034 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:14.915 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:14.910+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6038 (116'3010,123'6038] local-lis/les=21/22 n=651 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6036 crt=123'6036 lcod 123'6034 mlcod 123'6034 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:14.918 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:14.914+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5843 (115'2805,123'5843] local-lis/les=21/22 n=622 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5843 lcod 123'5841 mlcod 123'5841 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:14.919 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:14.914+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5845 (115'2805,123'5845] local-lis/les=21/22 n=620 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5843 crt=123'5843 lcod 123'5841 mlcod 123'5841 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:14.923 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:14.918+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6040 (116'3010,123'6040] local-lis/les=21/22 n=649 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'6040 lcod 123'6038 mlcod 123'6038 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:14.925 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:14.922+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6042 (116'3010,123'6042] local-lis/les=21/22 n=647 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6040 crt=123'6040 lcod 123'6038 mlcod 123'6038 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:14.927 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:14.922+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5847 (115'2805,123'5847] local-lis/les=21/22 n=618 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5847 lcod 123'5845 mlcod 123'5845 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:14.930 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:14.926+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5849 (115'2805,123'5849] local-lis/les=21/22 n=616 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5847 crt=123'5847 lcod 123'5845 mlcod 123'5845 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:14.938 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:14.934+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6044 (116'3010,123'6044] local-lis/les=21/22 n=645 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'6044 lcod 123'6042 mlcod 123'6042 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:14.940 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:14.938+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5584 (115'2504,123'5584] local-lis/les=21/22 n=635 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5584 lcod 123'5582 mlcod 123'5582 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:14.941 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:14.938+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6046 (116'3010,123'6046] local-lis/les=21/22 n=643 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6044 crt=123'6044 lcod 123'6042 mlcod 123'6042 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:14.945 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:14.938+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5586 (115'2504,123'5586] local-lis/les=21/22 n=633 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5584 crt=123'5584 lcod 123'5582 mlcod 123'5582 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:14.945 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:14.942+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5851 (115'2805,123'5851] local-lis/les=21/22 n=614 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5851 lcod 123'5849 mlcod 123'5849 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:14.947 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:14.942+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5853 (115'2805,123'5853] local-lis/les=21/22 n=612 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5851 crt=123'5851 lcod 123'5849 mlcod 123'5849 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:14.948 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:14.946+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5588 (115'2504,123'5588] local-lis/les=21/22 n=631 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5588 lcod 123'5586 mlcod 123'5586 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:14.950 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:14.946+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5590 (115'2504,123'5590] local-lis/les=21/22 n=629 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5588 crt=123'5588 lcod 123'5586 mlcod 123'5586 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:14.950 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:14.946+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6048 (116'3010,123'6048] local-lis/les=21/22 n=641 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'6048 lcod 123'6046 mlcod 123'6046 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:14.957 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:14.950+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6050 (116'3010,123'6050] local-lis/les=21/22 n=639 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6048 crt=123'6048 lcod 123'6046 mlcod 123'6046 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:14.957 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:14.954+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5592 (115'2504,123'5592] local-lis/les=21/22 n=627 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5592 lcod 123'5590 mlcod 123'5590 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:14.958 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:14.954+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5855 (115'2805,123'5855] local-lis/les=21/22 n=610 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5855 lcod 123'5853 mlcod 123'5853 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:14.959 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:14.954+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5594 (115'2504,123'5594] local-lis/les=21/22 n=625 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5592 crt=123'5592 lcod 123'5590 mlcod 123'5590 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:14.960 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:14.958+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5857 (115'2805,123'5857] local-lis/les=21/22 n=608 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5855 crt=123'5855 lcod 123'5853 mlcod 123'5853 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:14.964 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:14.962+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5596 (115'2504,123'5596] local-lis/les=21/22 n=623 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5596 lcod 123'5594 mlcod 123'5594 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:14.965 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:14.962+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6052 (116'3010,123'6052] local-lis/les=21/22 n=637 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'6052 lcod 123'6050 mlcod 123'6050 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:14.966 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:14.962+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5598 (115'2504,123'5598] local-lis/les=21/22 n=621 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5596 crt=123'5596 lcod 123'5594 mlcod 123'5594 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:14.967 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:14.962+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6054 (116'3010,123'6054] local-lis/les=21/22 n=635 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6052 crt=123'6052 lcod 123'6050 mlcod 123'6050 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:14.970 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:14.966+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5859 (115'2805,123'5859] local-lis/les=21/22 n=606 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5859 lcod 123'5857 mlcod 123'5857 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:14.972 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:14.970+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5861 (115'2805,123'5861] local-lis/les=21/22 n=604 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5859 crt=123'5859 lcod 123'5857 mlcod 123'5857 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:14.974 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:14.970+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5600 (115'2504,123'5600] local-lis/les=21/22 n=619 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5600 lcod 123'5598 mlcod 123'5598 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:14.975 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:14.970+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6056 (116'3010,123'6056] local-lis/les=21/22 n=633 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'6056 lcod 123'6054 mlcod 123'6054 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:14.975 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:14.970+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5602 (115'2504,123'5602] local-lis/les=21/22 n=617 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5600 crt=123'5600 lcod 123'5598 mlcod 123'5598 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:14.981 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:14.974+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6058 (116'3010,123'6058] local-lis/les=21/22 n=631 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6056 crt=123'6056 lcod 123'6054 mlcod 123'6054 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:14.981 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:14.978+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5604 (115'2504,123'5604] local-lis/les=21/22 n=615 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5604 lcod 123'5602 mlcod 123'5602 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:14.984 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:14.982+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5606 (115'2604,123'5606] local-lis/les=21/22 n=613 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5604 crt=123'5604 lcod 123'5602 mlcod 123'5602 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:14.985 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:14.978+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5863 (115'2805,123'5863] local-lis/les=21/22 n=602 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5863 lcod 123'5861 mlcod 123'5861 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:14.987 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:14.982+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5865 (115'2805,123'5865] local-lis/les=21/22 n=600 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5863 crt=123'5863 lcod 123'5861 mlcod 123'5861 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:14.990 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:14.986+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6060 (116'3010,123'6060] local-lis/les=21/22 n=629 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'6060 lcod 123'6058 mlcod 123'6058 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:14.992 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:14.986+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5608 (115'2604,123'5608] local-lis/les=21/22 n=611 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5608 lcod 123'5606 mlcod 123'5606 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:14.993 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:14.990+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6062 (116'3010,123'6062] local-lis/les=21/22 n=627 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6060 crt=123'6060 lcod 123'6058 mlcod 123'6058 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:14.993 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:14.990+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5610 (115'2604,123'5610] local-lis/les=21/22 n=609 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5608 crt=123'5608 lcod 123'5606 mlcod 123'5606 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:14.998 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:14.994+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5612 (115'2604,123'5612] local-lis/les=21/22 n=607 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5612 lcod 123'5610 mlcod 123'5610 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:15.001 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:14.998+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5614 (115'2604,123'5614] local-lis/les=21/22 n=605 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5612 crt=123'5612 lcod 123'5610 mlcod 123'5610 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:15.003 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:14.994+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5867 (115'2805,123'5867] local-lis/les=21/22 n=598 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5867 lcod 123'5865 mlcod 123'5865 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:15.003 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:14.998+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5869 (115'2805,123'5869] local-lis/les=21/22 n=596 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5867 crt=123'5867 lcod 123'5865 mlcod 123'5865 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:15.004 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:14.998+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6064 (116'3010,123'6064] local-lis/les=21/22 n=625 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'6064 lcod 123'6062 mlcod 123'6062 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:15.006 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.002+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5616 (115'2604,123'5616] local-lis/les=21/22 n=603 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5616 lcod 123'5614 mlcod 123'5614 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:15.008 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.002+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6066 (116'3010,123'6066] local-lis/les=21/22 n=623 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6064 crt=123'6064 lcod 123'6062 mlcod 123'6062 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:15.009 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.006+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5618 (115'2604,123'5618] local-lis/les=21/22 n=601 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5616 crt=123'5616 lcod 123'5614 mlcod 123'5614 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:15.012 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.010+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5871 (115'2805,123'5871] local-lis/les=21/22 n=594 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5871 lcod 123'5869 mlcod 123'5869 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:15.015 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.010+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5620 (115'2604,123'5620] local-lis/les=21/22 n=599 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5620 lcod 123'5618 mlcod 123'5618 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:15.016 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.014+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5622 (115'2604,123'5622] local-lis/les=21/22 n=597 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5620 crt=123'5620 lcod 123'5618 mlcod 123'5618 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:15.018 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.014+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6068 (116'3010,123'6068] local-lis/les=21/22 n=621 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'6068 lcod 123'6066 mlcod 123'6066 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:15.020 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.018+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6070 (116'3010,123'6070] local-lis/les=21/22 n=619 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6068 crt=123'6068 lcod 123'6066 mlcod 123'6066 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:15.023 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.018+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5873 (115'2805,123'5873] local-lis/les=21/22 n=592 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5873 lcod 123'5871 mlcod 123'5871 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:15.025 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.022+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5875 (115'2805,123'5875] local-lis/les=21/22 n=590 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5873 crt=123'5873 lcod 123'5871 mlcod 123'5871 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:15.029 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.026+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6072 (116'3010,123'6072] local-lis/les=21/22 n=617 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'6072 lcod 123'6070 mlcod 123'6070 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:15.030 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.026+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6074 (116'3010,123'6074] local-lis/les=21/22 n=615 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6072 crt=123'6072 lcod 123'6070 mlcod 123'6070 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:15.035 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.030+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5624 (115'2604,123'5624] local-lis/les=21/22 n=595 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5624 lcod 123'5622 mlcod 123'5622 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:15.039 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.034+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5877 (115'2805,123'5877] local-lis/les=21/22 n=588 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5877 lcod 123'5875 mlcod 123'5875 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:15.039 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.034+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5626 (115'2604,123'5626] local-lis/les=21/22 n=593 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5624 crt=123'5624 lcod 123'5622 mlcod 123'5622 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:15.040 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.038+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5879 (115'2805,123'5879] local-lis/les=21/22 n=586 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5877 crt=123'5877 lcod 123'5875 mlcod 123'5875 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:15.044 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.042+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6076 (116'3010,123'6076] local-lis/les=21/22 n=613 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'6076 lcod 123'6074 mlcod 123'6074 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:15.045 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.042+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5628 (115'2604,123'5628] local-lis/les=21/22 n=591 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5628 lcod 123'5626 mlcod 123'5626 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:15.051 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.046+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5630 (115'2604,123'5630] local-lis/les=21/22 n=589 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5630 lcod 123'5628 mlcod 123'5628 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:15.052 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.050+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5632 (115'2604,123'5632] local-lis/les=21/22 n=587 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5630 crt=123'5630 lcod 123'5628 mlcod 123'5628 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:15.055 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.050+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5634 (115'2604,123'5634] local-lis/les=21/22 n=585 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5634 lcod 123'5632 mlcod 123'5632 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:15.057 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.054+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5636 (115'2604,123'5636] local-lis/les=21/22 n=583 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5634 crt=123'5634 lcod 123'5632 mlcod 123'5632 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:15.060 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.058+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5638 (115'2604,123'5638] local-lis/les=21/22 n=581 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5638 lcod 123'5636 mlcod 123'5636 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:15.061 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.058+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5640 (115'2604,123'5640] local-lis/les=21/22 n=579 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5638 crt=123'5638 lcod 123'5636 mlcod 123'5636 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:15.064 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.062+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5642 (115'2604,123'5642] local-lis/les=21/22 n=577 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5642 lcod 123'5640 mlcod 123'5640 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:15.066 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.062+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5644 (115'2604,123'5644] local-lis/les=21/22 n=575 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5642 crt=123'5642 lcod 123'5640 mlcod 123'5640 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:15.069 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.066+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5646 (115'2604,123'5646] local-lis/les=21/22 n=573 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5646 lcod 123'5644 mlcod 123'5644 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:15.071 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.066+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5648 (115'2604,123'5648] local-lis/les=21/22 n=571 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5646 crt=123'5646 lcod 123'5644 mlcod 123'5644 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:15.075 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.070+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5650 (115'2604,123'5650] local-lis/les=21/22 n=569 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5650 lcod 123'5648 mlcod 123'5648 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:15.076 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.074+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5652 (115'2604,123'5652] local-lis/les=21/22 n=567 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5650 crt=123'5650 lcod 123'5648 mlcod 123'5648 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:15.080 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.078+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5654 (115'2604,123'5654] local-lis/les=21/22 n=565 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5654 lcod 123'5652 mlcod 123'5652 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:15.081 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.078+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5656 (115'2604,123'5656] local-lis/les=21/22 n=563 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5654 crt=123'5654 lcod 123'5652 mlcod 123'5652 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:15.085 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.082+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5658 (115'2604,123'5658] local-lis/les=21/22 n=561 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5658 lcod 123'5656 mlcod 123'5656 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:15.088 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.082+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5660 (115'2604,123'5660] local-lis/les=21/22 n=559 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5658 crt=123'5658 lcod 123'5656 mlcod 123'5656 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:15.088 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:15.086+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5667 (115'2605,123'5667] local-lis/les=21/22 n=722 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5667 lcod 123'5665 mlcod 123'5665 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:15.090 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:15.081+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5630 (115'2606,123'5630] local-lis/les=21/22 n=639 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5630 lcod 123'5628 mlcod 123'5628 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:15.091 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:15.081+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5632 (115'2606,123'5632] local-lis/les=21/22 n=637 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5630 crt=123'5630 lcod 123'5628 mlcod 123'5628 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:15.094 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.090+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5662 (115'2604,123'5662] local-lis/les=21/22 n=557 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5662 lcod 123'5660 mlcod 123'5660 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:15.097 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:15.089+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5634 (115'2606,123'5634] local-lis/les=21/22 n=635 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5634 lcod 123'5632 mlcod 123'5632 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:15.097 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.094+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5664 (115'2604,123'5664] local-lis/les=21/22 n=555 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5662 crt=123'5662 lcod 123'5660 mlcod 123'5660 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:15.098 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:15.093+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5636 (115'2606,123'5636] local-lis/les=21/22 n=633 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5634 crt=123'5634 lcod 123'5632 mlcod 123'5632 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:15.102 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:15.094+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5669 (115'2605,123'5669] local-lis/les=21/22 n=720 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5669 lcod 123'5667 mlcod 123'5667 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:15.102 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:15.098+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5671 (115'2605,123'5671] local-lis/les=21/22 n=718 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5669 crt=123'5669 lcod 123'5667 mlcod 123'5667 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:15.106 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:15.097+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5638 (115'2606,123'5638] local-lis/les=21/22 n=631 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5638 lcod 123'5636 mlcod 123'5636 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:15.106 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:15.102+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5673 (115'2605,123'5673] local-lis/les=21/22 n=716 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5673 lcod 123'5671 mlcod 123'5671 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:15.108 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:15.106+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5675 (115'2605,123'5675] local-lis/les=21/22 n=714 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5673 crt=123'5673 lcod 123'5671 mlcod 123'5671 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:15.109 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:15.101+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5640 (115'2606,123'5640] local-lis/les=21/22 n=629 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5638 crt=123'5638 lcod 123'5636 mlcod 123'5636 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:15.113 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:15.105+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5642 (115'2606,123'5642] local-lis/les=21/22 n=627 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5642 lcod 123'5640 mlcod 123'5640 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:15.113 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:15.110+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5677 (115'2605,123'5677] local-lis/les=21/22 n=712 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5677 lcod 123'5675 mlcod 123'5675 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:15.114 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:15.110+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5679 (115'2605,123'5679] local-lis/les=21/22 n=710 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5677 crt=123'5677 lcod 123'5675 mlcod 123'5675 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:15.115 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:15.109+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5644 (115'2606,123'5644] local-lis/les=21/22 n=625 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5642 crt=123'5642 lcod 123'5640 mlcod 123'5640 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:15.120 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:15.114+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5681 (115'2605,123'5681] local-lis/les=21/22 n=708 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5681 lcod 123'5679 mlcod 123'5679 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:15.120 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:15.113+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5646 (115'2606,123'5646] local-lis/les=21/22 n=623 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5646 lcod 123'5644 mlcod 123'5644 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:15.121 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:15.118+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5683 (115'2605,123'5683] local-lis/les=21/22 n=706 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5681 crt=123'5681 lcod 123'5679 mlcod 123'5679 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:15.121 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:15.113+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5648 (115'2606,123'5648] local-lis/les=21/22 n=621 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5646 crt=123'5646 lcod 123'5644 mlcod 123'5644 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:15.126 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.122+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5881 (115'2805,123'5881] local-lis/les=21/22 n=584 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5881 lcod 123'5879 mlcod 123'5879 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:15.129 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.126+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5883 (115'2805,123'5883] local-lis/les=21/22 n=582 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5881 crt=123'5881 lcod 123'5879 mlcod 123'5879 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:15.130 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:15.121+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5650 (115'2606,123'5650] local-lis/les=21/22 n=619 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5650 lcod 123'5648 mlcod 123'5648 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:15.131 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:15.126+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5685 (115'2605,123'5685] local-lis/les=21/22 n=704 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5685 lcod 123'5683 mlcod 123'5683 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:15.131 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:15.126+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5687 (115'2605,123'5687] local-lis/les=21/22 n=702 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5685 crt=123'5685 lcod 123'5683 mlcod 123'5683 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:15.133 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.130+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6078 (116'3010,123'6078] local-lis/les=21/22 n=611 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'6078 lcod 123'6076 mlcod 123'6076 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:15.135 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:15.129+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5652 (115'2606,123'5652] local-lis/les=21/22 n=617 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5650 crt=123'5650 lcod 123'5648 mlcod 123'5648 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:15.141 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.134+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6080 (116'3010,123'6080] local-lis/les=21/22 n=609 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6078 crt=123'6078 lcod 123'6076 mlcod 123'6076 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:15.141 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.134+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5885 (115'2805,123'5885] local-lis/les=21/22 n=580 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5885 lcod 123'5883 mlcod 123'5883 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:15.142 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:15.138+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5689 (115'2605,123'5689] local-lis/les=21/22 n=700 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5689 lcod 123'5687 mlcod 123'5687 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:15.142 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:15.133+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5654 (115'2606,123'5654] local-lis/les=21/22 n=615 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5654 lcod 123'5652 mlcod 123'5652 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:15.142 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.138+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5887 (115'2805,123'5887] local-lis/les=21/22 n=578 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5885 crt=123'5885 lcod 123'5883 mlcod 123'5883 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:15.144 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:15.138+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5691 (115'2605,123'5691] local-lis/les=21/22 n=698 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5689 crt=123'5689 lcod 123'5687 mlcod 123'5687 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:15.145 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:15.137+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5656 (115'2606,123'5656] local-lis/les=21/22 n=613 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5654 crt=123'5654 lcod 123'5652 mlcod 123'5652 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:15.147 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.142+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6082 (116'3010,123'6082] local-lis/les=21/22 n=607 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'6082 lcod 123'6080 mlcod 123'6080 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:15.148 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:15.146+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5693 (115'2605,123'5693] local-lis/les=21/22 n=696 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5693 lcod 123'5691 mlcod 123'5691 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:15.149 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.146+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6084 (116'3010,123'6084] local-lis/les=21/22 n=605 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6082 crt=123'6082 lcod 123'6080 mlcod 123'6080 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:15.152 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:15.150+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5695 (115'2605,123'5695] local-lis/les=21/22 n=694 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5693 crt=123'5693 lcod 123'5691 mlcod 123'5691 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:15.153 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:15.145+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5658 (115'2606,123'5658] local-lis/les=21/22 n=611 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5658 lcod 123'5656 mlcod 123'5656 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:15.154 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.150+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5889 (115'2805,123'5889] local-lis/les=21/22 n=576 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5889 lcod 123'5887 mlcod 123'5887 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:15.155 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:15.149+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5660 (115'2606,123'5660] local-lis/les=21/22 n=609 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5658 crt=123'5658 lcod 123'5656 mlcod 123'5656 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:15.157 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.154+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6086 (116'3010,123'6086] local-lis/les=21/22 n=603 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'6086 lcod 123'6084 mlcod 123'6084 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:15.158 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.154+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5891 (115'2805,123'5891] local-lis/les=21/22 n=574 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5889 crt=123'5889 lcod 123'5887 mlcod 123'5887 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:15.159 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.154+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6088 (116'3010,123'6088] local-lis/les=21/22 n=601 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6086 crt=123'6086 lcod 123'6084 mlcod 123'6084 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:15.161 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:15.153+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5662 (115'2606,123'5662] local-lis/les=21/22 n=607 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5662 lcod 123'5660 mlcod 123'5660 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:15.163 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:15.157+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5664 (115'2606,123'5664] local-lis/les=21/22 n=605 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5662 crt=123'5662 lcod 123'5660 mlcod 123'5660 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:15.165 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.162+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5893 (115'2805,123'5893] local-lis/les=21/22 n=572 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5893 lcod 123'5891 mlcod 123'5891 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:15.169 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:15.161+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5666 (115'2606,123'5666] local-lis/les=21/22 n=603 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5666 lcod 123'5664 mlcod 123'5664 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:15.169 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.166+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5895 (115'2805,123'5895] local-lis/les=21/22 n=570 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5893 crt=123'5893 lcod 123'5891 mlcod 123'5891 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:15.169 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:15.161+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5668 (115'2606,123'5668] local-lis/les=21/22 n=601 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5666 crt=123'5666 lcod 123'5664 mlcod 123'5664 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:15.171 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.166+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6090 (116'3010,123'6090] local-lis/les=21/22 n=599 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'6090 lcod 123'6088 mlcod 123'6088 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:15.173 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.170+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6092 (116'3010,123'6092] local-lis/les=21/22 n=597 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6090 crt=123'6090 lcod 123'6088 mlcod 123'6088 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:15.175 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:15.169+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5670 (115'2606,123'5670] local-lis/les=21/22 n=599 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5670 lcod 123'5668 mlcod 123'5668 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:15.178 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:15.169+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5672 (115'2606,123'5672] local-lis/les=21/22 n=597 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5670 crt=123'5670 lcod 123'5668 mlcod 123'5668 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:15.178 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.174+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5897 (115'2805,123'5897] local-lis/les=21/22 n=568 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5897 lcod 123'5895 mlcod 123'5895 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:15.180 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.178+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5899 (115'2805,123'5899] local-lis/les=21/22 n=566 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5897 crt=123'5897 lcod 123'5895 mlcod 123'5895 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:15.182 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:15.173+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5674 (115'2606,123'5674] local-lis/les=21/22 n=595 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5674 lcod 123'5672 mlcod 123'5672 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:15.183 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:15.177+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5676 (115'2606,123'5676] local-lis/les=21/22 n=593 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5674 crt=123'5674 lcod 123'5672 mlcod 123'5672 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:15.193 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:15.181+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5678 (115'2606,123'5678] local-lis/les=21/22 n=591 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5678 lcod 123'5676 mlcod 123'5676 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:15.193 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:15.185+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5680 (115'2606,123'5680] local-lis/les=21/22 n=589 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5678 crt=123'5678 lcod 123'5676 mlcod 123'5676 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:15.193 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.182+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6094 (116'3010,123'6094] local-lis/les=21/22 n=595 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'6094 lcod 123'6092 mlcod 123'6092 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:15.194 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.186+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6096 (116'3010,123'6096] local-lis/les=21/22 n=593 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6094 crt=123'6094 lcod 123'6092 mlcod 123'6092 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:15.194 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.186+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5901 (115'2805,123'5901] local-lis/les=21/22 n=564 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5901 lcod 123'5899 mlcod 123'5899 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:15.194 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.190+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5903 (115'2805,123'5903] local-lis/les=21/22 n=562 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5901 crt=123'5901 lcod 123'5899 mlcod 123'5899 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:15.197 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.194+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6098 (116'3010,123'6098] local-lis/les=21/22 n=591 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'6098 lcod 123'6096 mlcod 123'6096 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:15.199 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:15.189+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5682 (115'2606,123'5682] local-lis/les=21/22 n=587 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5682 lcod 123'5680 mlcod 123'5680 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:15.199 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.194+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6100 (116'3010,123'6100] local-lis/les=21/22 n=589 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6098 crt=123'6098 lcod 123'6096 mlcod 123'6096 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:15.200 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:15.193+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5684 (115'2606,123'5684] local-lis/les=21/22 n=585 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5682 crt=123'5682 lcod 123'5680 mlcod 123'5680 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:15.202 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.198+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5905 (115'2805,123'5905] local-lis/les=21/22 n=560 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5905 lcod 123'5903 mlcod 123'5903 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:15.205 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.202+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5907 (116'2905,123'5907] local-lis/les=21/22 n=558 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5905 crt=123'5905 lcod 123'5903 mlcod 123'5903 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:15.209 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:15.201+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5686 (115'2606,123'5686] local-lis/les=21/22 n=583 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5686 lcod 123'5684 mlcod 123'5684 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:15.211 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:15.205+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5688 (115'2606,123'5688] local-lis/les=21/22 n=581 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5686 crt=123'5686 lcod 123'5684 mlcod 123'5684 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:15.215 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.206+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6102 (116'3010,123'6102] local-lis/les=21/22 n=587 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'6102 lcod 123'6100 mlcod 123'6100 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:15.216 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:15.209+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5690 (115'2606,123'5690] local-lis/les=21/22 n=579 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5690 lcod 123'5688 mlcod 123'5688 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:15.216 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.210+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6104 (116'3010,123'6104] local-lis/les=21/22 n=585 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6102 crt=123'6102 lcod 123'6100 mlcod 123'6100 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:15.217 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:15.209+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5692 (115'2606,123'5692] local-lis/les=21/22 n=577 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5690 crt=123'5690 lcod 123'5688 mlcod 123'5688 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:15.218 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.214+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5909 (116'2905,123'5909] local-lis/les=21/22 n=556 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5909 lcod 123'5907 mlcod 123'5907 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:15.221 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.218+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5911 (116'2905,123'5911] local-lis/les=21/22 n=554 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5909 crt=123'5909 lcod 123'5907 mlcod 123'5907 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:15.221 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.218+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6106 (116'3010,123'6106] local-lis/les=21/22 n=583 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'6106 lcod 123'6104 mlcod 123'6104 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:15.224 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:15.217+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5694 (115'2606,123'5694] local-lis/les=21/22 n=575 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5694 lcod 123'5692 mlcod 123'5692 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:15.224 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.222+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6108 (116'3010,123'6108] local-lis/les=21/22 n=581 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6106 crt=123'6106 lcod 123'6104 mlcod 123'6104 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:15.225 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:15.217+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5696 (115'2606,123'5696] local-lis/les=21/22 n=573 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5694 crt=123'5694 lcod 123'5692 mlcod 123'5692 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:15.229 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.226+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5913 (116'2905,123'5913] local-lis/les=21/22 n=552 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5913 lcod 123'5911 mlcod 123'5911 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:15.231 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.226+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6110 (116'3010,123'6110] local-lis/les=21/22 n=579 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'6110 lcod 123'6108 mlcod 123'6108 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:15.232 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.230+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5915 (116'2905,123'5915] local-lis/les=21/22 n=550 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5913 crt=123'5913 lcod 123'5911 mlcod 123'5911 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:15.233 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:15.225+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5698 (115'2606,123'5698] local-lis/les=21/22 n=571 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5698 lcod 123'5696 mlcod 123'5696 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:15.234 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.230+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6112 (116'3110,123'6112] local-lis/les=21/22 n=577 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6110 crt=123'6110 lcod 123'6108 mlcod 123'6108 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:15.235 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:15.229+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5700 (115'2606,123'5700] local-lis/les=21/22 n=569 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5698 crt=123'5698 lcod 123'5696 mlcod 123'5696 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:15.235 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.230+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5714 (115'2712,123'5714] local-lis/les=21/22 n=947 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'5714 lcod 123'5712 mlcod 123'5712 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:15.239 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.234+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5716 (115'2712,123'5716] local-lis/les=21/22 n=945 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'5714 crt=123'5714 lcod 123'5712 mlcod 123'5712 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:15.241 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.238+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5666 (115'2604,123'5666] local-lis/les=21/22 n=553 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5666 lcod 123'5664 mlcod 123'5664 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:15.242 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.238+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5668 (115'2604,123'5668] local-lis/les=21/22 n=551 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5666 crt=123'5666 lcod 123'5664 mlcod 123'5664 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:15.245 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.242+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5718 (115'2712,123'5718] local-lis/les=21/22 n=943 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'5718 lcod 123'5716 mlcod 123'5716 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:15.248 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.246+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5720 (115'2712,123'5720] local-lis/les=21/22 n=941 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'5718 crt=123'5718 lcod 123'5716 mlcod 123'5716 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:15.248 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:15.241+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5702 (115'2606,123'5702] local-lis/les=21/22 n=567 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5702 lcod 123'5700 mlcod 123'5700 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:15.254 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.246+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5670 (115'2604,123'5670] local-lis/les=21/22 n=549 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5670 lcod 123'5668 mlcod 123'5668 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:15.255 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.242+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5917 (116'2905,123'5917] local-lis/les=21/22 n=548 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5917 lcod 123'5915 mlcod 123'5915 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:15.255 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.246+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5919 (116'2905,123'5919] local-lis/les=21/22 n=546 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5917 crt=123'5917 lcod 123'5915 mlcod 123'5915 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:15.255 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.246+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6114 (116'3110,123'6114] local-lis/les=21/22 n=575 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'6114 lcod 123'6112 mlcod 123'6112 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:15.255 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:15.245+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5704 (115'2606,123'5704] local-lis/les=21/22 n=565 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5702 crt=123'5702 lcod 123'5700 mlcod 123'5700 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:15.256 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.250+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5672 (115'2604,123'5672] local-lis/les=21/22 n=547 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5670 crt=123'5670 lcod 123'5668 mlcod 123'5668 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:15.257 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.250+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6116 (116'3110,123'6116] local-lis/les=21/22 n=573 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6114 crt=123'6114 lcod 123'6112 mlcod 123'6112 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:15.261 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.258+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5722 (115'2712,123'5722] local-lis/les=21/22 n=939 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'5722 lcod 123'5720 mlcod 123'5720 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:15.262 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:15.253+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5706 (115'2606,123'5706] local-lis/les=21/22 n=563 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5706 lcod 123'5704 mlcod 123'5704 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:15.264 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.258+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5674 (115'2604,123'5674] local-lis/les=21/22 n=545 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5674 lcod 123'5672 mlcod 123'5672 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:15.264 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.258+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5921 (116'2905,123'5921] local-lis/les=21/22 n=544 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5921 lcod 123'5919 mlcod 123'5919 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:15.265 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.262+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5724 (115'2712,123'5724] local-lis/les=21/22 n=937 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'5722 crt=123'5722 lcod 123'5720 mlcod 123'5720 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:15.266 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:15.261+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5708 (115'2706,123'5708] local-lis/les=21/22 n=561 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5706 crt=123'5706 lcod 123'5704 mlcod 123'5704 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:15.270 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.262+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5676 (115'2604,123'5676] local-lis/les=21/22 n=543 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5674 crt=123'5674 lcod 123'5672 mlcod 123'5672 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:15.278 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.262+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6118 (116'3110,123'6118] local-lis/les=21/22 n=571 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'6118 lcod 123'6116 mlcod 123'6116 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:15.278 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.266+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5923 (116'2905,123'5923] local-lis/les=21/22 n=542 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5921 crt=123'5921 lcod 123'5919 mlcod 123'5919 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:15.278 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.266+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6120 (116'3110,123'6120] local-lis/les=21/22 n=569 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6118 crt=123'6118 lcod 123'6116 mlcod 123'6116 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:15.278 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.270+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5726 (115'2712,123'5726] local-lis/les=21/22 n=935 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'5726 lcod 123'5724 mlcod 123'5724 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:15.279 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.274+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5728 (115'2712,123'5728] local-lis/les=21/22 n=933 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'5726 crt=123'5726 lcod 123'5724 mlcod 123'5724 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:15.280 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.274+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5678 (115'2604,123'5678] local-lis/les=21/22 n=541 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5678 lcod 123'5676 mlcod 123'5676 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:15.280 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.278+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6122 (116'3110,123'6122] local-lis/les=21/22 n=567 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'6122 lcod 123'6120 mlcod 123'6120 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:15.281 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.278+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5680 (115'2604,123'5680] local-lis/les=21/22 n=539 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5678 crt=123'5678 lcod 123'5676 mlcod 123'5676 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:15.284 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.278+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5925 (116'2905,123'5925] local-lis/les=21/22 n=540 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5925 lcod 123'5923 mlcod 123'5923 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:15.285 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.278+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5927 (116'2905,123'5927] local-lis/les=21/22 n=538 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5925 crt=123'5925 lcod 123'5923 mlcod 123'5923 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:15.286 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.282+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5730 (115'2712,123'5730] local-lis/les=21/22 n=931 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'5730 lcod 123'5728 mlcod 123'5728 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:15.286 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.282+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6124 (116'3110,123'6124] local-lis/les=21/22 n=565 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6122 crt=123'6122 lcod 123'6120 mlcod 123'6120 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:15.287 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.282+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5732 (115'2712,123'5732] local-lis/les=21/22 n=929 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'5730 crt=123'5730 lcod 123'5728 mlcod 123'5728 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:15.292 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:15.281+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5575 (115'2505,123'5575] local-lis/les=21/22 n=943 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=123'5575 lcod 123'5573 mlcod 123'5573 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:15.292 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:15.285+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5577 (115'2505,123'5577] local-lis/les=21/22 n=941 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5575 crt=123'5575 lcod 123'5573 mlcod 123'5573 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:15.292 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.286+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5929 (116'2905,123'5929] local-lis/les=21/22 n=536 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5929 lcod 123'5927 mlcod 123'5927 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:15.293 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.290+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5682 (115'2604,123'5682] local-lis/les=21/22 n=537 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5682 lcod 123'5680 mlcod 123'5680 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:15.294 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.290+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5931 (116'2905,123'5931] local-lis/les=21/22 n=534 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5929 crt=123'5929 lcod 123'5927 mlcod 123'5927 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:15.295 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.290+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5684 (115'2604,123'5684] local-lis/les=21/22 n=535 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5682 crt=123'5682 lcod 123'5680 mlcod 123'5680 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:15.298 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.294+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6126 (116'3110,123'6126] local-lis/les=21/22 n=563 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'6126 lcod 123'6124 mlcod 123'6124 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:15.299 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.294+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5734 (115'2712,123'5734] local-lis/les=21/22 n=927 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'5734 lcod 123'5732 mlcod 123'5732 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:15.301 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.298+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6128 (116'3110,123'6128] local-lis/les=21/22 n=561 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6126 crt=123'6126 lcod 123'6124 mlcod 123'6124 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:15.302 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.298+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5736 (115'2712,123'5736] local-lis/les=21/22 n=925 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'5734 crt=123'5734 lcod 123'5732 mlcod 123'5732 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:15.304 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:15.297+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5579 (115'2505,123'5579] local-lis/les=21/22 n=939 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=123'5579 lcod 123'5577 mlcod 123'5577 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:15.305 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.302+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5686 (115'2604,123'5686] local-lis/les=21/22 n=533 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5686 lcod 123'5684 mlcod 123'5684 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:15.306 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:15.297+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5581 (115'2505,123'5581] local-lis/les=21/22 n=937 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5579 crt=123'5579 lcod 123'5577 mlcod 123'5577 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:15.308 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.306+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5933 (116'2905,123'5933] local-lis/les=21/22 n=532 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5933 lcod 123'5931 mlcod 123'5931 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:15.308 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.306+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5688 (115'2604,123'5688] local-lis/les=21/22 n=531 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5686 crt=123'5686 lcod 123'5684 mlcod 123'5684 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:15.313 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.306+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6130 (116'3110,123'6130] local-lis/les=21/22 n=559 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'6130 lcod 123'6128 mlcod 123'6128 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:15.313 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.310+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5935 (116'2905,123'5935] local-lis/les=21/22 n=530 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5933 crt=123'5933 lcod 123'5931 mlcod 123'5931 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:15.314 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.310+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5738 (115'2712,123'5738] local-lis/les=21/22 n=923 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'5738 lcod 123'5736 mlcod 123'5736 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:15.315 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.310+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6132 (116'3110,123'6132] local-lis/les=21/22 n=557 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6130 crt=123'6130 lcod 123'6128 mlcod 123'6128 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:15.318 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:15.309+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5583 (115'2505,123'5583] local-lis/les=21/22 n=935 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=123'5583 lcod 123'5581 mlcod 123'5581 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:15.320 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:15.313+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5585 (115'2505,123'5585] local-lis/les=21/22 n=933 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5583 crt=123'5583 lcod 123'5581 mlcod 123'5581 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:15.321 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.314+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5690 (115'2604,123'5690] local-lis/les=21/22 n=529 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5690 lcod 123'5688 mlcod 123'5688 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:15.321 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.318+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5692 (115'2604,123'5692] local-lis/les=21/22 n=527 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5690 crt=123'5690 lcod 123'5688 mlcod 123'5688 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:15.326 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.322+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6134 (116'3110,123'6134] local-lis/les=21/22 n=555 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'6134 lcod 123'6132 mlcod 123'6132 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:15.327 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.322+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5937 (116'2905,123'5937] local-lis/les=21/22 n=528 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5937 lcod 123'5935 mlcod 123'5935 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:15.327 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.322+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6136 (116'3110,123'6136] local-lis/les=21/22 n=553 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6134 crt=123'6134 lcod 123'6132 mlcod 123'6132 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:15.328 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.322+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5742 (115'2712,123'5742] local-lis/les=21/22 n=920 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'5742 lcod 123'5740 mlcod 123'5740 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:15.329 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.326+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5939 (116'2905,123'5939] local-lis/les=21/22 n=526 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5937 crt=123'5937 lcod 123'5935 mlcod 123'5935 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:15.332 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.330+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5744 (115'2712,123'5744] local-lis/les=21/22 n=918 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'5742 crt=123'5742 lcod 123'5740 mlcod 123'5740 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:15.333 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:15.325+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5587 (115'2505,123'5587] local-lis/les=21/22 n=931 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=123'5587 lcod 123'5585 mlcod 123'5585 active+clean+snaptrim trimq=[1fe~2a]] removing snap head 2019-12-05T06:51:15.343 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.330+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6138 (116'3110,123'6138] local-lis/les=21/22 n=551 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'6138 lcod 123'6136 mlcod 123'6136 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:15.343 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.334+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6140 (116'3110,123'6140] local-lis/les=21/22 n=549 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6138 crt=123'6138 lcod 123'6136 mlcod 123'6136 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:15.343 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.334+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5694 (115'2604,123'5694] local-lis/les=21/22 n=525 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5694 lcod 123'5692 mlcod 123'5692 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:15.343 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.334+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5696 (115'2604,123'5696] local-lis/les=21/22 n=523 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5694 crt=123'5694 lcod 123'5692 mlcod 123'5692 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:15.344 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.338+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5941 (116'2905,123'5941] local-lis/les=21/22 n=524 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5941 lcod 123'5939 mlcod 123'5939 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:15.344 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:15.337+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5589 (115'2505,123'5589] local-lis/les=21/22 n=929 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=123'5589 lcod 123'5587 mlcod 123'5587 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:15.347 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.342+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5746 (115'2712,123'5746] local-lis/les=21/22 n=916 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'5746 lcod 123'5744 mlcod 123'5744 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:15.347 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.342+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5943 (116'2905,123'5943] local-lis/les=21/22 n=522 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5941 crt=123'5941 lcod 123'5939 mlcod 123'5939 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:15.348 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:15.341+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5591 (115'2505,123'5591] local-lis/les=21/22 n=927 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5589 crt=123'5589 lcod 123'5587 mlcod 123'5587 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:15.349 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.346+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5748 (115'2712,123'5748] local-lis/les=21/22 n=914 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'5746 crt=123'5746 lcod 123'5744 mlcod 123'5744 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:15.357 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.350+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6142 (116'3110,123'6142] local-lis/les=21/22 n=547 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'6142 lcod 123'6140 mlcod 123'6140 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:15.358 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.350+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5945 (116'2905,123'5945] local-lis/les=21/22 n=520 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5945 lcod 123'5943 mlcod 123'5943 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:15.358 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.354+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6144 (116'3110,123'6144] local-lis/les=21/22 n=545 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6142 crt=123'6142 lcod 123'6140 mlcod 123'6140 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:15.358 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.354+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5947 (116'2905,123'5947] local-lis/les=21/22 n=518 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5945 crt=123'5945 lcod 123'5943 mlcod 123'5943 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:15.358 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:15.350+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5643 (115'2607,123'5643] local-lis/les=21/22 n=752 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5643 lcod 123'5641 mlcod 123'5641 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:15.359 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:15.354+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5645 (115'2607,123'5645] local-lis/les=21/22 n=750 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5643 crt=123'5643 lcod 123'5641 mlcod 123'5641 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:15.359 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.354+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5698 (115'2604,123'5698] local-lis/les=21/22 n=521 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5698 lcod 123'5696 mlcod 123'5696 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:15.362 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.358+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5700 (115'2604,123'5700] local-lis/les=21/22 n=519 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5698 crt=123'5698 lcod 123'5696 mlcod 123'5696 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:15.366 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.362+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5750 (115'2712,123'5750] local-lis/les=21/22 n=912 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'5750 lcod 123'5748 mlcod 123'5748 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:15.370 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.366+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6146 (116'3110,123'6146] local-lis/les=21/22 n=543 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'6146 lcod 123'6144 mlcod 123'6144 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:15.371 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:15.366+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5647 (115'2607,123'5647] local-lis/les=21/22 n=748 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5647 lcod 123'5645 mlcod 123'5645 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:15.373 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.362+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5752 (115'2712,123'5752] local-lis/les=21/22 n=910 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'5750 crt=123'5750 lcod 123'5748 mlcod 123'5748 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:15.374 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:15.370+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5649 (115'2607,123'5649] local-lis/les=21/22 n=746 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5647 crt=123'5647 lcod 123'5645 mlcod 123'5645 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:15.375 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.370+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5949 (116'2905,123'5949] local-lis/les=21/22 n=516 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5949 lcod 123'5947 mlcod 123'5947 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:15.376 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.370+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5702 (115'2604,123'5702] local-lis/les=21/22 n=517 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5702 lcod 123'5700 mlcod 123'5700 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:15.377 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.374+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6148 (116'3110,123'6148] local-lis/les=21/22 n=541 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6146 crt=123'6146 lcod 123'6144 mlcod 123'6144 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:15.379 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:15.374+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5697 (115'2605,123'5697] local-lis/les=21/22 n=692 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5697 lcod 123'5695 mlcod 123'5695 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:15.379 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.374+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5704 (115'2604,123'5704] local-lis/les=21/22 n=515 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5702 crt=123'5702 lcod 123'5700 mlcod 123'5700 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:15.379 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.374+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5951 (116'2905,123'5951] local-lis/les=21/22 n=514 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5949 crt=123'5949 lcod 123'5947 mlcod 123'5947 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:15.384 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:15.378+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5699 (115'2605,123'5699] local-lis/les=21/22 n=690 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5697 crt=123'5697 lcod 123'5695 mlcod 123'5695 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:15.387 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.382+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5754 (115'2712,123'5754] local-lis/les=21/22 n=908 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'5754 lcod 123'5752 mlcod 123'5752 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:15.390 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.386+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5756 (115'2712,123'5756] local-lis/les=21/22 n=906 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'5754 crt=123'5754 lcod 123'5752 mlcod 123'5752 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:15.392 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.386+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6150 (116'3110,123'6150] local-lis/les=21/22 n=539 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'6150 lcod 123'6148 mlcod 123'6148 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:15.392 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:15.390+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5701 (115'2605,123'5701] local-lis/les=21/22 n=688 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5701 lcod 123'5699 mlcod 123'5699 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:15.392 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:15.390+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5651 (115'2607,123'5651] local-lis/les=21/22 n=744 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5651 lcod 123'5649 mlcod 123'5649 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:15.393 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.390+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6152 (116'3110,123'6152] local-lis/les=21/22 n=537 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6150 crt=123'6150 lcod 123'6148 mlcod 123'6148 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:15.394 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.390+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5953 (116'2905,123'5953] local-lis/les=21/22 n=512 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5953 lcod 123'5951 mlcod 123'5951 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:15.395 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:15.390+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5653 (115'2607,123'5653] local-lis/les=21/22 n=742 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5651 crt=123'5651 lcod 123'5649 mlcod 123'5649 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:15.395 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:15.390+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5703 (115'2605,123'5703] local-lis/les=21/22 n=686 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5701 crt=123'5701 lcod 123'5699 mlcod 123'5699 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:15.396 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.394+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5955 (116'2905,123'5955] local-lis/les=21/22 n=510 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5953 crt=123'5953 lcod 123'5951 mlcod 123'5951 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:15.405 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:15.398+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5705 (115'2605,123'5705] local-lis/les=21/22 n=684 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5705 lcod 123'5703 mlcod 123'5703 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:15.405 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.402+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6154 (116'3110,123'6154] local-lis/les=21/22 n=535 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'6154 lcod 123'6152 mlcod 123'6152 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:15.407 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.394+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5706 (115'2704,123'5706] local-lis/les=21/22 n=513 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5706 lcod 123'5704 mlcod 123'5704 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:15.407 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:15.402+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5707 (115'2705,123'5707] local-lis/les=21/22 n=682 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5705 crt=123'5705 lcod 123'5703 mlcod 123'5703 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:15.407 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.398+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5708 (115'2704,123'5708] local-lis/les=21/22 n=511 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5706 crt=123'5706 lcod 123'5704 mlcod 123'5704 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:15.410 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.406+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6156 (116'3110,123'6156] local-lis/les=21/22 n=533 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6154 crt=123'6154 lcod 123'6152 mlcod 123'6152 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:15.411 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:15.406+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5655 (115'2607,123'5655] local-lis/les=21/22 n=740 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5655 lcod 123'5653 mlcod 123'5653 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:15.414 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:15.410+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5657 (115'2607,123'5657] local-lis/les=21/22 n=738 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5655 crt=123'5655 lcod 123'5653 mlcod 123'5653 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:15.414 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.410+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5758 (115'2712,123'5758] local-lis/les=21/22 n=904 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'5758 lcod 123'5756 mlcod 123'5756 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:15.419 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.414+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5760 (115'2712,123'5760] local-lis/les=21/22 n=902 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'5758 crt=123'5758 lcod 123'5756 mlcod 123'5756 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:15.420 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.414+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5710 (115'2704,123'5710] local-lis/les=21/22 n=509 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5710 lcod 123'5708 mlcod 123'5708 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:15.422 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:15.418+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5659 (115'2607,123'5659] local-lis/les=21/22 n=736 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5659 lcod 123'5657 mlcod 123'5657 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:15.426 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.414+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5957 (116'2905,123'5957] local-lis/les=21/22 n=508 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5957 lcod 123'5955 mlcod 123'5955 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:15.426 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.414+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5959 (116'2905,123'5959] local-lis/les=21/22 n=506 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5957 crt=123'5957 lcod 123'5955 mlcod 123'5955 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:15.427 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.422+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5712 (115'2704,123'5712] local-lis/les=21/22 n=507 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5710 crt=123'5710 lcod 123'5708 mlcod 123'5708 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:15.428 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:15.422+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5709 (115'2705,123'5709] local-lis/les=21/22 n=680 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5709 lcod 123'5707 mlcod 123'5707 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:15.428 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:15.422+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5661 (115'2607,123'5661] local-lis/les=21/22 n=734 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5659 crt=123'5659 lcod 123'5657 mlcod 123'5657 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:15.430 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:15.426+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5711 (115'2705,123'5711] local-lis/les=21/22 n=678 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5709 crt=123'5709 lcod 123'5707 mlcod 123'5707 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:15.432 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.430+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5762 (115'2712,123'5762] local-lis/les=21/22 n=900 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'5762 lcod 123'5760 mlcod 123'5760 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:15.434 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.430+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6158 (116'3110,123'6158] local-lis/les=21/22 n=531 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'6158 lcod 123'6156 mlcod 123'6156 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:15.437 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.430+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5764 (115'2712,123'5764] local-lis/les=21/22 n=898 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'5762 crt=123'5762 lcod 123'5760 mlcod 123'5760 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:15.438 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.434+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6160 (116'3110,123'6160] local-lis/les=21/22 n=529 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6158 crt=123'6158 lcod 123'6156 mlcod 123'6156 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:15.440 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:15.434+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5663 (115'2607,123'5663] local-lis/les=21/22 n=732 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5663 lcod 123'5661 mlcod 123'5661 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:15.441 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:15.438+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5713 (115'2705,123'5713] local-lis/les=21/22 n=676 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5713 lcod 123'5711 mlcod 123'5711 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:15.442 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:15.438+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5665 (115'2607,123'5665] local-lis/les=21/22 n=730 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5663 crt=123'5663 lcod 123'5661 mlcod 123'5661 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:15.445 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:15.442+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5715 (115'2705,123'5715] local-lis/les=21/22 n=674 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5713 crt=123'5713 lcod 123'5711 mlcod 123'5711 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:15.447 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.442+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5766 (115'2712,123'5766] local-lis/les=21/22 n=896 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'5766 lcod 123'5764 mlcod 123'5764 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:15.449 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.446+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5961 (116'2905,123'5961] local-lis/les=21/22 n=504 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5961 lcod 123'5959 mlcod 123'5959 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:15.454 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.446+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5714 (115'2704,123'5714] local-lis/les=21/22 n=505 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5714 lcod 123'5712 mlcod 123'5712 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:15.455 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.450+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5716 (115'2704,123'5716] local-lis/les=21/22 n=503 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5714 crt=123'5714 lcod 123'5712 mlcod 123'5712 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:15.455 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.450+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5768 (115'2712,123'5768] local-lis/les=21/22 n=894 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'5766 crt=123'5766 lcod 123'5764 mlcod 123'5764 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:15.458 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:15.450+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5717 (115'2705,123'5717] local-lis/les=21/22 n=672 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5717 lcod 123'5715 mlcod 123'5715 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:15.459 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.450+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5963 (116'2905,123'5963] local-lis/les=21/22 n=502 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5961 crt=123'5961 lcod 123'5959 mlcod 123'5959 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:15.459 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.450+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6162 (116'3110,123'6162] local-lis/les=21/22 n=527 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'6162 lcod 123'6160 mlcod 123'6160 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:15.459 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:15.450+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5719 (115'2705,123'5719] local-lis/les=21/22 n=670 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5717 crt=123'5717 lcod 123'5715 mlcod 123'5715 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:15.462 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.458+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6164 (116'3110,123'6164] local-lis/les=21/22 n=525 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6162 crt=123'6162 lcod 123'6160 mlcod 123'6160 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:15.462 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:15.458+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5668 (115'2607,123'5668] local-lis/les=21/22 n=727 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5667 crt=123'5667 lcod 123'5665 mlcod 123'5665 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:15.464 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.462+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5771 (115'2712,123'5771] local-lis/les=21/22 n=891 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'5770 crt=123'5766 lcod 123'5768 mlcod 123'5768 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:15.465 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:15.462+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5670 (115'2607,123'5670] local-lis/les=21/22 n=725 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5667 crt=123'5667 lcod 123'5665 mlcod 123'5665 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:15.467 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:15.462+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5721 (115'2705,123'5721] local-lis/les=21/22 n=668 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5721 lcod 123'5719 mlcod 123'5719 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:15.468 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.466+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5965 (116'2905,123'5965] local-lis/les=21/22 n=500 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5965 lcod 123'5963 mlcod 123'5963 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:15.470 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.466+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5718 (115'2704,123'5718] local-lis/les=21/22 n=501 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5718 lcod 123'5716 mlcod 123'5716 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:15.473 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.466+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5967 (116'2905,123'5967] local-lis/les=21/22 n=498 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5965 crt=123'5965 lcod 123'5963 mlcod 123'5963 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:15.475 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:15.470+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5723 (115'2705,123'5723] local-lis/les=21/22 n=666 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5721 crt=123'5721 lcod 123'5719 mlcod 123'5719 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:15.476 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.474+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6166 (116'3110,123'6166] local-lis/les=21/22 n=523 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'6166 lcod 123'6164 mlcod 123'6164 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:15.476 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.474+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5720 (115'2704,123'5720] local-lis/les=21/22 n=499 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5718 crt=123'5718 lcod 123'5716 mlcod 123'5716 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:15.478 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.474+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5773 (115'2712,123'5773] local-lis/les=21/22 n=889 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'5770 crt=123'5770 lcod 123'5768 mlcod 123'5768 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:15.479 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.474+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6168 (116'3110,123'6168] local-lis/les=21/22 n=521 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6166 crt=123'6166 lcod 123'6164 mlcod 123'6164 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:15.482 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:15.478+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5672 (115'2607,123'5672] local-lis/les=21/22 n=723 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5672 lcod 123'5670 mlcod 123'5670 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:15.483 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.478+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5969 (116'2905,123'5969] local-lis/les=21/22 n=496 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5969 lcod 123'5967 mlcod 123'5967 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:15.486 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.482+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5971 (116'2905,123'5971] local-lis/les=21/22 n=494 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5969 crt=123'5969 lcod 123'5967 mlcod 123'5967 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:15.487 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.482+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5723 (115'2704,123'5723] local-lis/les=21/22 n=496 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5722 crt=123'5718 lcod 123'5720 mlcod 123'5720 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:15.488 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.486+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6170 (116'3110,123'6170] local-lis/les=21/22 n=519 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'6170 lcod 123'6168 mlcod 123'6168 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:15.491 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.486+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6172 (116'3110,123'6172] local-lis/les=21/22 n=517 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6170 crt=123'6170 lcod 123'6168 mlcod 123'6168 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:15.492 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.490+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5725 (115'2704,123'5725] local-lis/les=21/22 n=494 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5722 crt=123'5722 lcod 123'5720 mlcod 123'5720 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:15.496 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:15.490+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5674 (115'2607,123'5674] local-lis/les=21/22 n=721 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5674 lcod 123'5672 mlcod 123'5672 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:15.496 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:15.489+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5594 (115'2505,123'5594] local-lis/les=21/22 n=924 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5593 crt=123'5591 lcod 123'5591 mlcod 123'5591 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:15.498 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.494+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5973 (116'2905,123'5973] local-lis/les=21/22 n=492 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5973 lcod 123'5971 mlcod 123'5971 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:15.501 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.494+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6174 (116'3110,123'6174] local-lis/les=21/22 n=515 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'6174 lcod 123'6172 mlcod 123'6172 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:15.501 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.498+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5975 (116'2905,123'5975] local-lis/les=21/22 n=490 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5973 crt=123'5973 lcod 123'5971 mlcod 123'5971 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:15.502 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:15.498+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5676 (115'2607,123'5676] local-lis/les=21/22 n=719 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5674 crt=123'5674 lcod 123'5672 mlcod 123'5672 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:15.503 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:15.493+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5596 (115'2505,123'5596] local-lis/les=21/22 n=922 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5593 crt=123'5593 lcod 123'5591 mlcod 123'5591 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:15.503 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.498+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6176 (116'3110,123'6176] local-lis/les=21/22 n=513 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6174 crt=123'6174 lcod 123'6172 mlcod 123'6172 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:15.512 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.510+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5775 (115'2712,123'5775] local-lis/les=21/22 n=887 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'5775 lcod 123'5773 mlcod 123'5773 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:15.515 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.510+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5727 (115'2704,123'5727] local-lis/les=21/22 n=492 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5727 lcod 123'5725 mlcod 123'5725 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:15.516 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:15.510+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5680 (115'2607,123'5680] local-lis/les=21/22 n=715 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5679 crt=123'5674 lcod 123'5678 mlcod 123'5678 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:15.516 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.514+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6178 (116'3110,123'6178] local-lis/les=21/22 n=511 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'6178 lcod 123'6176 mlcod 123'6176 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:15.518 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.514+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6180 (116'3110,123'6180] local-lis/les=21/22 n=509 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6178 crt=123'6178 lcod 123'6176 mlcod 123'6176 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:15.519 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:15.514+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5682 (115'2607,123'5682] local-lis/les=21/22 n=713 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5679 crt=123'5679 lcod 123'5678 mlcod 123'5678 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:15.521 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:15.513+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5711 (115'2706,123'5711] local-lis/les=21/22 n=558 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5710 crt=123'5708 lcod 123'5708 mlcod 123'5708 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:15.522 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.518+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5777 (115'2712,123'5777] local-lis/les=21/22 n=885 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'5775 crt=123'5775 lcod 123'5773 mlcod 123'5773 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:15.523 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.518+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5977 (116'2905,123'5977] local-lis/les=21/22 n=488 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5977 lcod 123'5975 mlcod 123'5975 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:15.525 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:15.517+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5713 (115'2706,123'5713] local-lis/les=21/22 n=556 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5710 crt=123'5710 lcod 123'5708 mlcod 123'5708 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:15.526 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.522+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5979 (116'2905,123'5979] local-lis/les=21/22 n=486 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5977 crt=123'5977 lcod 123'5975 mlcod 123'5975 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:15.526 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:15.521+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5599 (115'2505,123'5599] local-lis/les=21/22 n=919 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5598 crt=123'5596 lcod 123'5596 mlcod 123'5596 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:15.531 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.526+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5729 (115'2704,123'5729] local-lis/les=21/22 n=490 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5727 crt=123'5727 lcod 123'5725 mlcod 123'5725 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:15.531 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:15.521+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5601 (115'2505,123'5601] local-lis/les=21/22 n=917 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5598 crt=123'5598 lcod 123'5596 mlcod 123'5596 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:15.532 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:15.526+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5685 (115'2607,123'5685] local-lis/les=21/22 n=710 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5684 crt=123'5680 lcod 123'5682 mlcod 123'5682 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:15.536 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:15.530+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5687 (115'2607,123'5687] local-lis/les=21/22 n=708 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5684 crt=123'5684 lcod 123'5682 mlcod 123'5682 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:15.537 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.530+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6182 (116'3110,123'6182] local-lis/les=21/22 n=507 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'6182 lcod 123'6180 mlcod 123'6180 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:15.538 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.534+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6184 (116'3110,123'6184] local-lis/les=21/22 n=505 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6182 crt=123'6182 lcod 123'6180 mlcod 123'6180 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:15.538 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:15.533+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5716 (115'2706,123'5716] local-lis/les=21/22 n=553 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5715 crt=123'5713 lcod 123'5713 mlcod 123'5713 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:15.540 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:15.533+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5718 (115'2706,123'5718] local-lis/les=21/22 n=551 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5715 crt=123'5715 lcod 123'5713 mlcod 123'5713 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:15.544 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.542+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5731 (115'2704,123'5731] local-lis/les=21/22 n=488 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5731 lcod 123'5729 mlcod 123'5729 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:15.547 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:15.542+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5689 (115'2607,123'5689] local-lis/les=21/22 n=706 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5689 lcod 123'5687 mlcod 123'5687 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:15.548 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.542+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5982 (116'2905,123'5982] local-lis/les=21/22 n=483 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5981 crt=123'5979 lcod 123'5979 mlcod 123'5979 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:15.550 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.542+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5779 (115'2712,123'5779] local-lis/les=21/22 n=883 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'5779 lcod 123'5777 mlcod 123'5777 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:15.550 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.542+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5733 (115'2704,123'5733] local-lis/les=21/22 n=486 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5731 crt=123'5731 lcod 123'5729 mlcod 123'5729 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:15.551 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:15.546+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5691 (115'2607,123'5691] local-lis/les=21/22 n=704 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5689 crt=123'5689 lcod 123'5687 mlcod 123'5687 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:15.551 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.546+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5781 (115'2712,123'5781] local-lis/les=21/22 n=881 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'5779 crt=123'5779 lcod 123'5777 mlcod 123'5777 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:15.554 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.550+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5984 (116'2905,123'5984] local-lis/les=21/22 n=481 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5981 crt=123'5981 lcod 123'5979 mlcod 123'5979 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:15.560 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.558+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6186 (116'3110,123'6186] local-lis/les=21/22 n=503 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'6186 lcod 123'6184 mlcod 123'6184 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:15.565 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.562+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5783 (115'2712,123'5783] local-lis/les=21/22 n=879 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'5783 lcod 123'5781 mlcod 123'5781 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:15.566 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.562+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5986 (116'2905,123'5986] local-lis/les=21/22 n=479 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5986 lcod 123'5984 mlcod 123'5984 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:15.568 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.566+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5785 (115'2712,123'5785] local-lis/les=21/22 n=877 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'5783 crt=123'5783 lcod 123'5781 mlcod 123'5781 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:15.569 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.566+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5988 (116'2905,123'5988] local-lis/les=21/22 n=477 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5986 crt=123'5986 lcod 123'5984 mlcod 123'5984 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:15.572 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.566+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5737 (115'2704,123'5737] local-lis/les=21/22 n=482 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5737 lcod 123'5736 mlcod 123'5736 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:15.576 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:15.566+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5726 (115'2705,123'5726] local-lis/les=21/22 n=663 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5725 crt=123'5723 lcod 123'5723 mlcod 123'5723 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:15.576 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.574+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5739 (115'2704,123'5739] local-lis/les=21/22 n=480 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5737 crt=123'5737 lcod 123'5736 mlcod 123'5736 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:15.577 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:15.570+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5728 (115'2705,123'5728] local-lis/les=21/22 n=661 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5725 crt=123'5725 lcod 123'5723 mlcod 123'5723 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:15.577 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:15.570+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5693 (115'2607,123'5693] local-lis/les=21/22 n=702 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5693 lcod 123'5691 mlcod 123'5691 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:15.578 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:15.570+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5695 (115'2607,123'5695] local-lis/les=21/22 n=700 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5693 crt=123'5693 lcod 123'5691 mlcod 123'5691 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:15.581 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.578+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5990 (116'2905,123'5990] local-lis/les=21/22 n=475 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'5990 lcod 123'5988 mlcod 123'5988 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:15.585 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.582+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5788 (115'2712,123'5788] local-lis/les=21/22 n=874 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'5787 crt=123'5785 lcod 123'5785 mlcod 123'5785 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:15.586 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.582+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6189 (116'3110,123'6189] local-lis/les=21/22 n=500 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6188 crt=123'6188 lcod 123'6186 mlcod 123'6186 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:15.586 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.582+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5992 (116'2905,123'5992] local-lis/les=21/22 n=473 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5990 crt=123'5990 lcod 123'5988 mlcod 123'5988 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:15.588 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.586+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6191 (116'3110,123'6191] local-lis/les=21/22 n=498 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6188 crt=123'6188 lcod 123'6186 mlcod 123'6186 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:15.590 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.586+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5790 (115'2712,123'5790] local-lis/les=21/22 n=872 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'5787 crt=123'5787 lcod 123'5785 mlcod 123'5785 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:15.591 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.586+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5741 (115'2704,123'5741] local-lis/les=21/22 n=478 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5741 lcod 123'5739 mlcod 123'5739 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:15.595 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.590+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5743 (115'2704,123'5743] local-lis/les=21/22 n=476 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5741 crt=123'5741 lcod 123'5739 mlcod 123'5739 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:15.606 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:15.593+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5721 (115'2706,123'5721] local-lis/les=21/22 n=548 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5720 crt=123'5718 lcod 123'5718 mlcod 123'5718 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:15.606 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:15.597+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5723 (115'2706,123'5723] local-lis/les=21/22 n=546 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5720 crt=123'5720 lcod 123'5718 mlcod 123'5718 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:15.606 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.594+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6193 (116'3110,123'6193] local-lis/les=21/22 n=496 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'6193 lcod 123'6191 mlcod 123'6191 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:15.606 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.594+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5995 (116'2905,123'5995] local-lis/les=21/22 n=470 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5994 crt=123'5992 lcod 123'5992 mlcod 123'5992 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:15.607 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.594+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6195 (116'3110,123'6195] local-lis/les=21/22 n=494 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6193 crt=123'6193 lcod 123'6191 mlcod 123'6191 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:15.607 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.598+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'5997 (116'2905,123'5997] local-lis/les=21/22 n=468 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5994 crt=123'5994 lcod 123'5992 mlcod 123'5992 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:15.607 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:15.598+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5697 (115'2607,123'5697] local-lis/les=21/22 n=698 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5697 lcod 123'5695 mlcod 123'5695 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:15.607 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:15.598+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5699 (115'2607,123'5699] local-lis/les=21/22 n=696 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5697 crt=123'5697 lcod 123'5695 mlcod 123'5695 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:15.610 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.606+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5745 (115'2704,123'5745] local-lis/les=21/22 n=474 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5745 lcod 123'5743 mlcod 123'5743 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:15.615 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.606+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6197 (116'3110,123'6197] local-lis/les=21/22 n=492 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'6197 lcod 123'6195 mlcod 123'6195 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:15.615 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.610+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5747 (115'2704,123'5747] local-lis/les=21/22 n=472 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5745 crt=123'5745 lcod 123'5743 mlcod 123'5743 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:15.616 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.610+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6199 (116'3110,123'6199] local-lis/les=21/22 n=490 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6197 crt=123'6197 lcod 123'6195 mlcod 123'6195 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:15.618 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.614+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6000 (116'2905,123'6000] local-lis/les=21/22 n=465 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5999 crt=123'5994 lcod 123'5997 mlcod 123'5997 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:15.619 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.614+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5750 (115'2704,123'5750] local-lis/les=21/22 n=469 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5749 crt=123'5745 lcod 123'5747 mlcod 123'5747 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:15.622 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.618+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6002 (116'2905,123'6002] local-lis/les=21/22 n=463 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'5999 crt=123'5999 lcod 123'5997 mlcod 123'5997 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:15.622 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.618+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5752 (115'2704,123'5752] local-lis/les=21/22 n=467 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5749 crt=123'5749 lcod 123'5747 mlcod 123'5747 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:15.624 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.622+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6201 (116'3110,123'6201] local-lis/les=21/22 n=488 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'6201 lcod 123'6199 mlcod 123'6199 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:15.627 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.622+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6203 (116'3110,123'6203] local-lis/les=21/22 n=486 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6201 crt=123'6201 lcod 123'6199 mlcod 123'6199 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:15.630 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.626+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5755 (115'2704,123'5755] local-lis/les=21/22 n=464 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5754 crt=123'5750 lcod 123'5752 mlcod 123'5752 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:15.633 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.630+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5757 (115'2704,123'5757] local-lis/les=21/22 n=462 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5754 crt=123'5754 lcod 123'5752 mlcod 123'5752 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:15.635 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.630+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6205 (116'3110,123'6205] local-lis/les=21/22 n=484 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'6205 lcod 123'6203 mlcod 123'6203 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:15.636 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.634+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6005 (116'2905,123'6005] local-lis/les=21/22 n=460 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'6004 crt=123'6002 lcod 123'6002 mlcod 123'6002 active+clean+snaptrim trimq=[208~20] ps=[1fe~a]] removing snap head 2019-12-05T06:51:15.640 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.634+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6007 (116'2905,123'6007] local-lis/les=21/22 n=458 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'6004 crt=123'6004 lcod 123'6002 mlcod 123'6002 active+clean+snaptrim trimq=[208~20] ps=[1fe~a]] removing snap head 2019-12-05T06:51:15.641 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.638+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6207 (116'3110,123'6207] local-lis/les=21/22 n=482 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6205 crt=123'6205 lcod 123'6203 mlcod 123'6203 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:15.645 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.642+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5759 (115'2704,123'5759] local-lis/les=21/22 n=460 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5759 lcod 123'5757 mlcod 123'5757 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:15.653 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.650+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5761 (115'2704,123'5761] local-lis/les=21/22 n=458 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5759 crt=123'5759 lcod 123'5757 mlcod 123'5757 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:15.656 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.654+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6009 (117'3007,123'6009] local-lis/les=21/22 n=456 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'6009 lcod 123'6007 mlcod 123'6007 active+clean+snaptrim trimq=[208~20] ps=[1fe~a]] removing snap head 2019-12-05T06:51:15.661 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.658+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6011 (117'3007,123'6011] local-lis/les=21/22 n=454 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'6009 crt=123'6009 lcod 123'6007 mlcod 123'6007 active+clean+snaptrim trimq=[208~20] ps=[1fe~a]] removing snap head 2019-12-05T06:51:15.663 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.658+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5763 (115'2704,123'5763] local-lis/les=21/22 n=456 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5763 lcod 123'5761 mlcod 123'5761 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:15.664 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.662+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5765 (115'2704,123'5765] local-lis/les=21/22 n=454 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5763 crt=123'5763 lcod 123'5761 mlcod 123'5761 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:15.667 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:15.661+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5604 (115'2505,123'5604] local-lis/les=21/22 n=914 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5603 crt=123'5601 lcod 123'5601 mlcod 123'5601 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:15.669 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:15.661+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5606 (115'2505,123'5606] local-lis/les=21/22 n=912 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5603 crt=123'5603 lcod 123'5601 mlcod 123'5601 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:15.670 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:15.665+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5725 (115'2706,123'5725] local-lis/les=21/22 n=544 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5725 lcod 123'5723 mlcod 123'5723 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:15.674 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:15.665+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5727 (115'2706,123'5727] local-lis/les=21/22 n=542 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5725 crt=123'5725 lcod 123'5723 mlcod 123'5723 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:15.675 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.670+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6013 (117'3007,123'6013] local-lis/les=21/22 n=452 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'6013 lcod 123'6011 mlcod 123'6011 active+clean+snaptrim trimq=[208~20] ps=[1fe~a]] removing snap head 2019-12-05T06:51:15.678 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.674+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6211 (117'3210,123'6211] local-lis/les=21/22 n=478 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'6211 lcod 123'6210 mlcod 123'6210 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:15.679 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.674+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6213 (117'3210,123'6213] local-lis/les=21/22 n=476 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6211 crt=123'6211 lcod 123'6210 mlcod 123'6210 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:15.680 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.678+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6015 (117'3007,123'6015] local-lis/les=21/22 n=450 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'6013 crt=123'6013 lcod 123'6011 mlcod 123'6011 active+clean+snaptrim trimq=[208~20] ps=[1fe~a]] removing snap head 2019-12-05T06:51:15.681 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:15.673+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5609 (115'2606,123'5609] local-lis/les=21/22 n=909 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5608 crt=123'5606 lcod 123'5606 mlcod 123'5606 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:15.682 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.674+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5768 (115'2704,123'5768] local-lis/les=21/22 n=451 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5767 crt=123'5767 lcod 123'5765 mlcod 123'5765 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:15.682 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.678+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5770 (115'2704,123'5770] local-lis/les=21/22 n=449 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5767 crt=123'5767 lcod 123'5765 mlcod 123'5765 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:15.684 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:15.677+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5611 (115'2606,123'5611] local-lis/les=21/22 n=907 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5608 crt=123'5608 lcod 123'5606 mlcod 123'5606 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:15.690 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.686+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6017 (117'3007,123'6017] local-lis/les=21/22 n=448 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'6017 lcod 123'6015 mlcod 123'6015 active+clean+snaptrim trimq=[208~20] ps=[1fe~a]] removing snap head 2019-12-05T06:51:15.693 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.686+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5772 (115'2704,123'5772] local-lis/les=21/22 n=447 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5772 lcod 123'5770 mlcod 123'5770 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:15.694 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.690+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6019 (117'3007,123'6019] local-lis/les=21/22 n=446 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'6017 crt=123'6017 lcod 123'6015 mlcod 123'6015 active+clean+snaptrim trimq=[208~20] ps=[1fe~a]] removing snap head 2019-12-05T06:51:15.695 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.690+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5774 (115'2704,123'5774] local-lis/les=21/22 n=445 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5772 crt=123'5772 lcod 123'5770 mlcod 123'5770 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:15.697 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.694+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6216 (117'3210,123'6216] local-lis/les=21/22 n=473 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6215 crt=123'6215 lcod 123'6213 mlcod 123'6213 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:15.701 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.698+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6218 (117'3210,123'6218] local-lis/les=21/22 n=471 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6215 crt=123'6215 lcod 123'6213 mlcod 123'6213 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:15.702 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.698+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6021 (117'3007,123'6021] local-lis/les=21/22 n=444 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'6021 lcod 123'6019 mlcod 123'6019 active+clean+snaptrim trimq=[208~20] ps=[1fe~a]] removing snap head 2019-12-05T06:51:15.704 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.702+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6023 (117'3007,123'6023] local-lis/les=21/22 n=442 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'6021 crt=123'6021 lcod 123'6019 mlcod 123'6019 active+clean+snaptrim trimq=[208~20] ps=[1fe~a]] removing snap head 2019-12-05T06:51:15.709 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.706+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5777 (115'2704,123'5777] local-lis/les=21/22 n=442 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5776 crt=123'5774 lcod 123'5774 mlcod 123'5774 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:15.709 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.706+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6220 (117'3210,123'6220] local-lis/les=21/22 n=469 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'6220 lcod 123'6218 mlcod 123'6218 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:15.711 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.706+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6222 (117'3210,123'6222] local-lis/les=21/22 n=467 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6220 crt=123'6220 lcod 123'6218 mlcod 123'6218 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:15.711 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.706+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5779 (115'2704,123'5779] local-lis/les=21/22 n=440 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5776 crt=123'5776 lcod 123'5774 mlcod 123'5774 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:15.716 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.714+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6025 (117'3007,123'6025] local-lis/les=21/22 n=440 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'6025 lcod 123'6023 mlcod 123'6023 active+clean+snaptrim trimq=[208~20] ps=[1fe~a]] removing snap head 2019-12-05T06:51:15.718 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.714+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6027 (117'3007,123'6027] local-lis/les=21/22 n=438 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'6025 crt=123'6025 lcod 123'6023 mlcod 123'6023 active+clean+snaptrim trimq=[208~20] ps=[1fe~a]] removing snap head 2019-12-05T06:51:15.719 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:15.713+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5730 (115'2706,123'5730] local-lis/les=21/22 n=539 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5729 crt=123'5727 lcod 123'5727 mlcod 123'5727 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:15.720 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:15.713+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5732 (115'2706,123'5732] local-lis/les=21/22 n=537 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5729 crt=123'5729 lcod 123'5727 mlcod 123'5727 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:15.722 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.718+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6224 (117'3210,123'6224] local-lis/les=21/22 n=465 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'6224 lcod 123'6222 mlcod 123'6222 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:15.728 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.722+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6226 (117'3210,123'6226] local-lis/les=21/22 n=463 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6224 crt=123'6224 lcod 123'6222 mlcod 123'6222 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:15.728 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.722+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6029 (117'3007,123'6029] local-lis/les=21/22 n=436 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'6029 lcod 123'6027 mlcod 123'6027 active+clean+snaptrim trimq=[208~20] ps=[1fe~a]] removing snap head 2019-12-05T06:51:15.728 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:15.721+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5614 (115'2606,123'5614] local-lis/les=21/22 n=904 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5613 crt=123'5611 lcod 123'5611 mlcod 123'5611 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:15.729 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.726+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6031 (117'3007,123'6031] local-lis/les=21/22 n=434 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'6029 crt=123'6029 lcod 123'6027 mlcod 123'6027 active+clean+snaptrim trimq=[208~20] ps=[1fe~a]] removing snap head 2019-12-05T06:51:15.731 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:15.721+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5616 (115'2606,123'5616] local-lis/les=21/22 n=902 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5613 crt=123'5613 lcod 123'5611 mlcod 123'5611 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:15.736 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.730+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6228 (117'3210,123'6228] local-lis/les=21/22 n=461 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'6228 lcod 123'6226 mlcod 123'6226 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:15.737 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.734+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6033 (117'3007,123'6033] local-lis/les=21/22 n=432 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'6033 lcod 123'6031 mlcod 123'6031 active+clean+snaptrim trimq=[209~1f] ps=[1fe~b]] removing snap head 2019-12-05T06:51:15.738 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:15.729+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5735 (115'2706,123'5735] local-lis/les=21/22 n=534 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5734 crt=123'5732 lcod 123'5732 mlcod 123'5732 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:15.739 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:15.733+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5737 (115'2706,123'5737] local-lis/les=21/22 n=532 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5734 crt=123'5734 lcod 123'5732 mlcod 123'5732 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:15.742 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.738+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6035 (117'3007,123'6035] local-lis/les=21/22 n=430 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'6033 crt=123'6033 lcod 123'6031 mlcod 123'6031 active+clean+snaptrim trimq=[209~1f] ps=[1fe~b]] removing snap head 2019-12-05T06:51:15.746 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:15.742+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5701 (115'2607,123'5701] local-lis/les=21/22 n=694 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5701 lcod 123'5699 mlcod 123'5699 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:15.749 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:15.746+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5703 (115'2607,123'5703] local-lis/les=21/22 n=692 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5701 crt=123'5701 lcod 123'5699 mlcod 123'5699 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:15.750 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.742+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5794 (115'2712,123'5794] local-lis/les=21/22 n=868 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'5792 crt=123'5790 lcod 123'5790 mlcod 123'5790 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:15.751 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.746+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5796 (115'2712,123'5796] local-lis/les=21/22 n=866 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'5792 crt=123'5792 lcod 123'5790 mlcod 123'5790 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:15.751 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.742+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6230 (117'3210,123'6230] local-lis/les=21/22 n=459 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'6230 lcod 123'6228 mlcod 123'6228 active+clean+snaptrim trimq=[208~20] ps=[1fe~a]] removing snap head 2019-12-05T06:51:15.751 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.742+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6232 (117'3210,123'6232] local-lis/les=21/22 n=457 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6230 crt=123'6230 lcod 123'6228 mlcod 123'6228 active+clean+snaptrim trimq=[208~20] ps=[1fe~a]] removing snap head 2019-12-05T06:51:15.753 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.750+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6037 (117'3007,123'6037] local-lis/les=21/22 n=428 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'6037 lcod 123'6035 mlcod 123'6035 active+clean+snaptrim trimq=[209~1f] ps=[1fe~b]] removing snap head 2019-12-05T06:51:15.756 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.754+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6235 (117'3210,123'6235] local-lis/les=21/22 n=454 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6234 crt=123'6230 lcod 123'6232 mlcod 123'6232 active+clean+snaptrim trimq=[208~20] ps=[1fe~a]] removing snap head 2019-12-05T06:51:15.756 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.754+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6039 (117'3007,123'6039] local-lis/les=21/22 n=426 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'6037 crt=123'6037 lcod 123'6035 mlcod 123'6035 active+clean+snaptrim trimq=[209~1f] ps=[1fe~b]] removing snap head 2019-12-05T06:51:15.760 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.754+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6237 (117'3210,123'6237] local-lis/les=21/22 n=452 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6234 crt=123'6234 lcod 123'6232 mlcod 123'6232 active+clean+snaptrim trimq=[208~20] ps=[1fe~a]] removing snap head 2019-12-05T06:51:15.764 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.762+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6043 (117'3007,123'6043] local-lis/les=21/22 n=422 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'6042 crt=123'6037 lcod 123'6041 mlcod 123'6041 active+clean+snaptrim trimq=[209~1f] ps=[1fe~b]] removing snap head 2019-12-05T06:51:15.766 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.762+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6239 (117'3210,123'6239] local-lis/les=21/22 n=450 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'6239 lcod 123'6237 mlcod 123'6237 active+clean+snaptrim trimq=[208~20] ps=[1fe~a]] removing snap head 2019-12-05T06:51:15.766 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.762+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6045 (117'3007,123'6045] local-lis/les=21/22 n=420 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'6042 crt=123'6042 lcod 123'6041 mlcod 123'6041 active+clean+snaptrim trimq=[209~1f] ps=[1fe~b]] removing snap head 2019-12-05T06:51:15.768 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.766+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6241 (117'3210,123'6241] local-lis/les=21/22 n=448 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6239 crt=123'6239 lcod 123'6237 mlcod 123'6237 active+clean+snaptrim trimq=[208~20] ps=[1fe~a]] removing snap head 2019-12-05T06:51:15.772 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.770+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6048 (117'3007,123'6048] local-lis/les=21/22 n=417 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'6047 crt=123'6043 lcod 123'6045 mlcod 123'6045 active+clean+snaptrim trimq=[209~1f] ps=[1fe~b]] removing snap head 2019-12-05T06:51:15.773 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.770+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6050 (117'3007,123'6050] local-lis/les=21/22 n=415 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'6047 crt=123'6047 lcod 123'6045 mlcod 123'6045 active+clean+snaptrim trimq=[209~1f] ps=[1fe~b]] removing snap head 2019-12-05T06:51:15.780 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.774+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6243 (117'3210,123'6243] local-lis/les=21/22 n=446 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'6243 lcod 123'6241 mlcod 123'6241 active+clean+snaptrim trimq=[208~20] ps=[1fe~a]] removing snap head 2019-12-05T06:51:15.781 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.778+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6245 (117'3210,123'6245] local-lis/les=21/22 n=444 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6243 crt=123'6243 lcod 123'6241 mlcod 123'6241 active+clean+snaptrim trimq=[208~20] ps=[1fe~a]] removing snap head 2019-12-05T06:51:15.781 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.778+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6052 (117'3007,123'6052] local-lis/les=21/22 n=413 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'6052 lcod 123'6050 mlcod 123'6050 active+clean+snaptrim trimq=[209~1f] ps=[1fe~b]] removing snap head 2019-12-05T06:51:15.784 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.778+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6054 (117'3007,123'6054] local-lis/les=21/22 n=411 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'6052 crt=123'6052 lcod 123'6050 mlcod 123'6050 active+clean+snaptrim trimq=[209~1f] ps=[1fe~b]] removing snap head 2019-12-05T06:51:15.784 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:15.778+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5731 (115'2705,123'5731] local-lis/les=21/22 n=658 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5730 crt=123'5728 lcod 123'5728 mlcod 123'5728 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:15.784 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:15.778+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5733 (115'2705,123'5733] local-lis/les=21/22 n=656 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5730 crt=123'5730 lcod 123'5728 mlcod 123'5728 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:15.787 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.782+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6247 (117'3210,123'6247] local-lis/les=21/22 n=442 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'6247 lcod 123'6245 mlcod 123'6245 active+clean+snaptrim trimq=[208~20] ps=[1fe~a]] removing snap head 2019-12-05T06:51:15.789 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.786+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6249 (117'3210,123'6249] local-lis/les=21/22 n=440 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6247 crt=123'6247 lcod 123'6245 mlcod 123'6245 active+clean+snaptrim trimq=[208~20] ps=[1fe~a]] removing snap head 2019-12-05T06:51:15.790 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.786+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6056 (117'3007,123'6056] local-lis/les=21/22 n=409 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'6056 lcod 123'6054 mlcod 123'6054 active+clean+snaptrim trimq=[20a~1e] ps=[1fe~c]] removing snap head 2019-12-05T06:51:15.793 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.790+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6058 (117'3007,123'6058] local-lis/les=21/22 n=407 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'6056 crt=123'6056 lcod 123'6054 mlcod 123'6054 active+clean+snaptrim trimq=[20a~1e] ps=[1fe~c]] removing snap head 2019-12-05T06:51:15.797 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.794+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6252 (117'3210,123'6252] local-lis/les=21/22 n=437 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'6252 lcod 123'6251 mlcod 123'6251 active+clean+snaptrim trimq=[208~20] ps=[1fe~a]] removing snap head 2019-12-05T06:51:15.812 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:15.793+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5619 (115'2606,123'5619] local-lis/les=21/22 n=899 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5618 crt=123'5616 lcod 123'5616 mlcod 123'5616 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:15.812 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:15.793+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5621 (115'2606,123'5621] local-lis/les=21/22 n=897 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5618 crt=123'5618 lcod 123'5616 mlcod 123'5616 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:15.812 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.794+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6060 (117'3007,123'6060] local-lis/les=21/22 n=405 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'6060 lcod 123'6058 mlcod 123'6058 active+clean+snaptrim trimq=[20a~1e] ps=[1fe~c]] removing snap head 2019-12-05T06:51:15.812 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.794+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6254 (117'3210,123'6254] local-lis/les=21/22 n=435 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6252 crt=123'6252 lcod 123'6251 mlcod 123'6251 active+clean+snaptrim trimq=[208~20] ps=[1fe~a]] removing snap head 2019-12-05T06:51:15.813 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.798+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6062 (117'3007,123'6062] local-lis/les=21/22 n=403 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'6060 crt=123'6060 lcod 123'6058 mlcod 123'6058 active+clean+snaptrim trimq=[20a~1e] ps=[1fe~c]] removing snap head 2019-12-05T06:51:15.813 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.806+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6256 (117'3210,123'6256] local-lis/les=21/22 n=433 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'6256 lcod 123'6254 mlcod 123'6254 active+clean+snaptrim trimq=[208~20] ps=[1fe~a]] removing snap head 2019-12-05T06:51:15.813 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.806+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5800 (115'2712,123'5800] local-lis/les=21/22 n=862 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'5798 crt=123'5796 lcod 123'5796 mlcod 123'5796 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:15.813 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:15.806+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5705 (115'2607,123'5705] local-lis/les=21/22 n=690 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5705 lcod 123'5703 mlcod 123'5703 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:15.813 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:15.806+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5707 (115'2607,123'5707] local-lis/les=21/22 n=688 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5705 crt=123'5705 lcod 123'5703 mlcod 123'5703 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:15.814 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.806+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5802 (115'2712,123'5802] local-lis/les=21/22 n=860 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'5798 crt=123'5798 lcod 123'5796 mlcod 123'5796 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:15.837 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.810+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6258 (117'3210,123'6258] local-lis/les=21/22 n=431 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6256 crt=123'6256 lcod 123'6254 mlcod 123'6254 active+clean+snaptrim trimq=[208~20] ps=[1fe~a]] removing snap head 2019-12-05T06:51:15.838 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.810+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6064 (117'3007,123'6064] local-lis/les=21/22 n=401 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'6064 lcod 123'6062 mlcod 123'6062 active+clean+snaptrim trimq=[20a~1e] ps=[1fe~c]] removing snap head 2019-12-05T06:51:15.838 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.814+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6066 (117'3007,123'6066] local-lis/les=21/22 n=399 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'6064 crt=123'6064 lcod 123'6062 mlcod 123'6062 active+clean+snaptrim trimq=[20a~1e] ps=[1fe~c]] removing snap head 2019-12-05T06:51:15.838 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.818+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6260 (117'3210,123'6260] local-lis/les=21/22 n=429 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'6260 lcod 123'6258 mlcod 123'6258 active+clean+snaptrim trimq=[208~20] ps=[1fe~a]] removing snap head 2019-12-05T06:51:15.838 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.822+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6262 (117'3210,123'6262] local-lis/les=21/22 n=427 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6260 crt=123'6260 lcod 123'6258 mlcod 123'6258 active+clean+snaptrim trimq=[208~20] ps=[1fe~a]] removing snap head 2019-12-05T06:51:15.838 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:15.822+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5711 (115'2707,123'5711] local-lis/les=21/22 n=684 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5709 crt=123'5705 lcod 123'5707 mlcod 123'5707 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:15.838 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:15.822+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5713 (115'2707,123'5713] local-lis/les=21/22 n=682 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5709 crt=123'5709 lcod 123'5707 mlcod 123'5707 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:15.838 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:15.830+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5716 (115'2707,123'5716] local-lis/les=21/22 n=679 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5715 crt=123'5711 lcod 123'5713 mlcod 123'5713 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:15.839 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.830+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5805 (115'2712,123'5805] local-lis/les=21/22 n=857 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'5804 crt=123'5802 lcod 123'5802 mlcod 123'5802 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:15.845 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:15.834+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5718 (115'2707,123'5718] local-lis/les=21/22 n=677 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5715 crt=123'5715 lcod 123'5713 mlcod 123'5713 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:15.845 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.838+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6069 (117'3007,123'6069] local-lis/les=21/22 n=396 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'6068 crt=123'6066 lcod 123'6066 mlcod 123'6066 active+clean+snaptrim trimq=[20a~1e] ps=[1fe~c]] removing snap head 2019-12-05T06:51:15.845 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.838+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6264 (117'3210,123'6264] local-lis/les=21/22 n=425 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'6264 lcod 123'6262 mlcod 123'6262 active+clean+snaptrim trimq=[208~20] ps=[1fe~a]] removing snap head 2019-12-05T06:51:15.845 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.838+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6071 (117'3007,123'6071] local-lis/les=21/22 n=394 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'6068 crt=123'6068 lcod 123'6066 mlcod 123'6066 active+clean+snaptrim trimq=[20a~1e] ps=[1fe~c]] removing snap head 2019-12-05T06:51:15.847 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.842+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6266 (117'3210,123'6266] local-lis/les=21/22 n=423 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6264 crt=123'6264 lcod 123'6262 mlcod 123'6262 active+clean+snaptrim trimq=[208~20] ps=[1fe~a]] removing snap head 2019-12-05T06:51:15.850 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.846+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5783 (115'2704,123'5783] local-lis/les=21/22 n=436 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5781 crt=123'5779 lcod 123'5779 mlcod 123'5779 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:15.852 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.850+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5785 (115'2704,123'5785] local-lis/les=21/22 n=434 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5781 crt=123'5781 lcod 123'5779 mlcod 123'5779 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:15.853 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.850+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6073 (117'3007,123'6073] local-lis/les=21/22 n=392 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'6073 lcod 123'6071 mlcod 123'6071 active+clean+snaptrim trimq=[20a~1e] ps=[1fe~c]] removing snap head 2019-12-05T06:51:15.857 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.854+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6075 (117'3007,123'6075] local-lis/les=21/22 n=390 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'6073 crt=123'6073 lcod 123'6071 mlcod 123'6071 active+clean+snaptrim trimq=[20a~1e] ps=[1fe~c]] removing snap head 2019-12-05T06:51:15.858 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.854+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6268 (117'3210,123'6268] local-lis/les=21/22 n=421 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'6268 lcod 123'6266 mlcod 123'6266 active+clean+snaptrim trimq=[209~1f] ps=[1fe~b]] removing snap head 2019-12-05T06:51:15.860 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.854+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5810 (115'2712,123'5810] local-lis/les=21/22 n=853 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'5809 crt=123'5807 lcod 123'5807 mlcod 123'5807 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:15.861 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.858+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6270 (117'3210,123'6270] local-lis/les=21/22 n=419 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6268 crt=123'6268 lcod 123'6266 mlcod 123'6266 active+clean+snaptrim trimq=[209~1f] ps=[1fe~b]] removing snap head 2019-12-05T06:51:15.862 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.858+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5812 (115'2712,123'5812] local-lis/les=21/22 n=851 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'5809 crt=123'5809 lcod 123'5807 mlcod 123'5807 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:15.863 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.858+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6077 (117'3007,123'6077] local-lis/les=21/22 n=388 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'6077 lcod 123'6075 mlcod 123'6075 active+clean+snaptrim trimq=[20a~1e] ps=[1fe~c]] removing snap head 2019-12-05T06:51:15.864 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:15.862+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5721 (115'2707,123'5721] local-lis/les=21/22 n=674 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5720 crt=123'5716 lcod 123'5718 mlcod 123'5718 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:15.866 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:15.862+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5723 (115'2707,123'5723] local-lis/les=21/22 n=672 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5720 crt=123'5720 lcod 123'5718 mlcod 123'5718 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:15.872 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.866+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5787 (115'2704,123'5787] local-lis/les=21/22 n=432 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5787 lcod 123'5785 mlcod 123'5785 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:15.875 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.870+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5789 (115'2704,123'5789] local-lis/les=21/22 n=430 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5787 crt=123'5787 lcod 123'5785 mlcod 123'5785 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:15.876 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:15.869+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5624 (115'2606,123'5624] local-lis/les=21/22 n=894 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5623 crt=123'5621 lcod 123'5621 mlcod 123'5621 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:15.878 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:15.873+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5626 (115'2606,123'5626] local-lis/les=21/22 n=892 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5623 crt=123'5623 lcod 123'5621 mlcod 123'5621 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:15.878 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.870+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6272 (117'3210,123'6272] local-lis/les=21/22 n=417 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'6272 lcod 123'6270 mlcod 123'6270 active+clean+snaptrim trimq=[209~1f] ps=[1fe~b]] removing snap head 2019-12-05T06:51:15.879 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.870+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6274 (117'3210,123'6274] local-lis/les=21/22 n=415 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6272 crt=123'6272 lcod 123'6270 mlcod 123'6270 active+clean+snaptrim trimq=[209~1f] ps=[1fe~b]] removing snap head 2019-12-05T06:51:15.882 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.878+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5814 (115'2812,123'5814] local-lis/les=21/22 n=849 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'5814 lcod 123'5812 mlcod 123'5812 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:15.884 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.882+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6079 (117'3007,123'6079] local-lis/les=21/22 n=386 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'6079 lcod 123'6077 mlcod 123'6077 active+clean+snaptrim trimq=[20b~1d] ps=[1fe~d]] removing snap head 2019-12-05T06:51:15.884 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.882+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5816 (115'2812,123'5816] local-lis/les=21/22 n=847 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'5814 crt=123'5814 lcod 123'5812 mlcod 123'5812 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:15.885 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.882+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5792 (115'2704,123'5792] local-lis/les=21/22 n=427 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5791 crt=123'5787 lcod 123'5789 mlcod 123'5789 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:15.886 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:15.882+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5736 (115'2705,123'5736] local-lis/les=21/22 n=653 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5735 crt=123'5733 lcod 123'5733 mlcod 123'5733 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:15.889 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.886+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5794 (115'2704,123'5794] local-lis/les=21/22 n=425 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5791 crt=123'5791 lcod 123'5789 mlcod 123'5789 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:15.889 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.886+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6276 (117'3210,123'6276] local-lis/les=21/22 n=413 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'6276 lcod 123'6274 mlcod 123'6274 active+clean+snaptrim trimq=[209~1f] ps=[1fe~b]] removing snap head 2019-12-05T06:51:15.889 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.886+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6081 (117'3007,123'6081] local-lis/les=21/22 n=384 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'6079 crt=123'6079 lcod 123'6077 mlcod 123'6077 active+clean+snaptrim trimq=[20b~1d] ps=[1fe~d]] removing snap head 2019-12-05T06:51:15.890 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:15.886+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5738 (115'2705,123'5738] local-lis/les=21/22 n=651 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5735 crt=123'5735 lcod 123'5733 mlcod 123'5733 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:15.894 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.890+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6278 (117'3210,123'6278] local-lis/les=21/22 n=411 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6276 crt=123'6276 lcod 123'6274 mlcod 123'6274 active+clean+snaptrim trimq=[209~1f] ps=[1fe~b]] removing snap head 2019-12-05T06:51:15.898 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.894+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5818 (115'2812,123'5818] local-lis/les=21/22 n=845 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'5818 lcod 123'5816 mlcod 123'5816 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:15.899 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:15.894+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5741 (115'2705,123'5741] local-lis/les=21/22 n=648 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5740 crt=123'5736 lcod 123'5738 mlcod 123'5738 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:15.899 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.894+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6083 (117'3007,123'6083] local-lis/les=21/22 n=382 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'6083 lcod 123'6081 mlcod 123'6081 active+clean+snaptrim trimq=[20b~1d] ps=[1fe~d]] removing snap head 2019-12-05T06:51:15.900 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.898+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5796 (115'2704,123'5796] local-lis/les=21/22 n=423 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5796 lcod 123'5794 mlcod 123'5794 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:15.901 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:15.893+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5740 (115'2706,123'5740] local-lis/les=21/22 n=529 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5739 crt=123'5737 lcod 123'5737 mlcod 123'5737 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:15.903 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:15.898+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5743 (115'2705,123'5743] local-lis/les=21/22 n=646 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5740 crt=123'5740 lcod 123'5738 mlcod 123'5738 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:15.903 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.898+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5798 (115'2704,123'5798] local-lis/les=21/22 n=421 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5796 crt=123'5796 lcod 123'5794 mlcod 123'5794 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:15.904 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.902+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6280 (117'3210,123'6280] local-lis/les=21/22 n=409 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'6280 lcod 123'6278 mlcod 123'6278 active+clean+snaptrim trimq=[209~1f] ps=[1fe~b]] removing snap head 2019-12-05T06:51:15.905 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:15.893+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5742 (115'2706,123'5742] local-lis/les=21/22 n=527 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5739 crt=123'5739 lcod 123'5737 mlcod 123'5737 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:15.906 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.902+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5820 (115'2812,123'5820] local-lis/les=21/22 n=843 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'5818 crt=123'5818 lcod 123'5816 mlcod 123'5816 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:15.907 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.902+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6085 (117'3007,123'6085] local-lis/les=21/22 n=380 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'6083 crt=123'6083 lcod 123'6081 mlcod 123'6081 active+clean+snaptrim trimq=[20b~1d] ps=[1fe~d]] removing snap head 2019-12-05T06:51:15.907 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.902+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6282 (117'3210,123'6282] local-lis/les=21/22 n=407 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6280 crt=123'6280 lcod 123'6278 mlcod 123'6278 active+clean+snaptrim trimq=[209~1f] ps=[1fe~b]] removing snap head 2019-12-05T06:51:15.911 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:15.905+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5629 (115'2606,123'5629] local-lis/les=21/22 n=889 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5628 crt=123'5626 lcod 123'5626 mlcod 123'5626 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:15.913 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:15.905+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5631 (115'2606,123'5631] local-lis/les=21/22 n=887 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5628 crt=123'5628 lcod 123'5626 mlcod 123'5626 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:15.915 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.910+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6087 (117'3007,123'6087] local-lis/les=21/22 n=378 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'6087 lcod 123'6085 mlcod 123'6085 active+clean+snaptrim trimq=[20b~1d] ps=[1fe~d]] removing snap head 2019-12-05T06:51:15.917 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.914+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6089 (117'3007,123'6089] local-lis/les=21/22 n=376 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'6087 crt=123'6087 lcod 123'6085 mlcod 123'6085 active+clean+snaptrim trimq=[20b~1d] ps=[1fe~d]] removing snap head 2019-12-05T06:51:15.917 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.914+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5800 (115'2704,123'5800] local-lis/les=21/22 n=419 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5800 lcod 123'5798 mlcod 123'5798 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:15.921 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.918+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5802 (115'2704,123'5802] local-lis/les=21/22 n=417 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5800 crt=123'5800 lcod 123'5798 mlcod 123'5798 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:15.922 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.918+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5823 (115'2812,123'5823] local-lis/les=21/22 n=840 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'5823 lcod 123'5822 mlcod 123'5822 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:15.925 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.922+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6284 (117'3210,123'6284] local-lis/les=21/22 n=405 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'6284 lcod 123'6282 mlcod 123'6282 active+clean+snaptrim trimq=[209~1f] ps=[1fe~b]] removing snap head 2019-12-05T06:51:15.927 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.922+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5825 (115'2812,123'5825] local-lis/les=21/22 n=838 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'5823 crt=123'5823 lcod 123'5822 mlcod 123'5822 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:15.929 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.926+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6091 (117'3007,123'6091] local-lis/les=21/22 n=374 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'6091 lcod 123'6089 mlcod 123'6089 active+clean+snaptrim trimq=[20b~1d] ps=[1fe~d]] removing snap head 2019-12-05T06:51:15.930 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.926+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6286 (117'3210,123'6286] local-lis/les=21/22 n=403 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6284 crt=123'6284 lcod 123'6282 mlcod 123'6282 active+clean+snaptrim trimq=[209~1f] ps=[1fe~b]] removing snap head 2019-12-05T06:51:15.933 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.926+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5806 (115'2804,123'5806] local-lis/les=21/22 n=413 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5804 crt=123'5800 lcod 123'5802 mlcod 123'5802 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:15.933 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.930+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6093 (117'3007,123'6093] local-lis/les=21/22 n=372 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'6091 crt=123'6091 lcod 123'6089 mlcod 123'6089 active+clean+snaptrim trimq=[20b~1d] ps=[1fe~d]] removing snap head 2019-12-05T06:51:15.936 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.930+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5808 (115'2804,123'5808] local-lis/les=21/22 n=411 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5804 crt=123'5804 lcod 123'5802 mlcod 123'5802 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:15.938 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.934+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5827 (115'2812,123'5827] local-lis/les=21/22 n=836 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'5827 lcod 123'5825 mlcod 123'5825 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:15.938 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.934+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6288 (117'3210,123'6288] local-lis/les=21/22 n=401 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'6288 lcod 123'6286 mlcod 123'6286 active+clean+snaptrim trimq=[209~1f] ps=[1fe~b]] removing snap head 2019-12-05T06:51:15.941 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.938+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5829 (115'2812,123'5829] local-lis/les=21/22 n=834 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'5827 crt=123'5827 lcod 123'5825 mlcod 123'5825 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:15.941 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.938+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6290 (117'3210,123'6290] local-lis/les=21/22 n=399 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6288 crt=123'6288 lcod 123'6286 mlcod 123'6286 active+clean+snaptrim trimq=[209~1f] ps=[1fe~b]] removing snap head 2019-12-05T06:51:15.942 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.938+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6095 (117'3007,123'6095] local-lis/les=21/22 n=370 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'6095 lcod 123'6093 mlcod 123'6093 active+clean+snaptrim trimq=[20b~1d] ps=[1fe~d]] removing snap head 2019-12-05T06:51:15.947 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:15.941+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5745 (115'2706,123'5745] local-lis/les=21/22 n=524 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5744 crt=123'5742 lcod 123'5742 mlcod 123'5742 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:15.948 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.942+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6097 (117'3007,123'6097] local-lis/les=21/22 n=368 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'6095 crt=123'6095 lcod 123'6093 mlcod 123'6093 active+clean+snaptrim trimq=[20b~1d] ps=[1fe~d]] removing snap head 2019-12-05T06:51:15.948 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.946+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5810 (115'2804,123'5810] local-lis/les=21/22 n=409 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5810 lcod 123'5808 mlcod 123'5808 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:15.950 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:15.941+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5747 (115'2706,123'5747] local-lis/les=21/22 n=522 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5744 crt=123'5744 lcod 123'5742 mlcod 123'5742 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:15.954 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.950+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5831 (115'2812,123'5831] local-lis/les=21/22 n=832 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'5831 lcod 123'5829 mlcod 123'5829 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:15.954 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.950+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6292 (117'3210,123'6292] local-lis/les=21/22 n=397 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'6292 lcod 123'6290 mlcod 123'6290 active+clean+snaptrim trimq=[209~1f] ps=[1fe~b]] removing snap head 2019-12-05T06:51:15.956 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.954+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5812 (115'2804,123'5812] local-lis/les=21/22 n=407 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5810 crt=123'5810 lcod 123'5808 mlcod 123'5808 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:15.957 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.954+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6294 (117'3210,123'6294] local-lis/les=21/22 n=395 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6292 crt=123'6292 lcod 123'6290 mlcod 123'6290 active+clean+snaptrim trimq=[209~1f] ps=[1fe~b]] removing snap head 2019-12-05T06:51:15.957 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.954+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6100 (117'3007,123'6100] local-lis/les=21/22 n=365 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'6099 crt=123'6095 lcod 123'6097 mlcod 123'6097 active+clean+snaptrim trimq=[20b~1d] ps=[1fe~d]] removing snap head 2019-12-05T06:51:15.958 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:15.954+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5746 (115'2705,123'5746] local-lis/les=21/22 n=643 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5745 crt=123'5743 lcod 123'5743 mlcod 123'5743 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:15.961 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:15.958+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5748 (115'2705,123'5748] local-lis/les=21/22 n=641 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5745 crt=123'5745 lcod 123'5743 mlcod 123'5743 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:15.962 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.958+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6102 (117'3007,123'6102] local-lis/les=21/22 n=363 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'6099 crt=123'6099 lcod 123'6097 mlcod 123'6097 active+clean+snaptrim trimq=[20b~1d] ps=[1fe~d]] removing snap head 2019-12-05T06:51:15.969 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.958+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5833 (115'2812,123'5833] local-lis/les=21/22 n=830 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'5833 lcod 123'5831 mlcod 123'5831 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:15.970 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.962+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5814 (115'2804,123'5814] local-lis/les=21/22 n=405 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5814 lcod 123'5812 mlcod 123'5812 active+clean+snaptrim trimq=[208~20] ps=[1fe~a]] removing snap head 2019-12-05T06:51:15.970 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.966+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5835 (115'2812,123'5835] local-lis/les=21/22 n=828 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'5833 crt=123'5833 lcod 123'5831 mlcod 123'5831 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:15.971 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.966+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5816 (115'2804,123'5816] local-lis/les=21/22 n=403 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5814 crt=123'5814 lcod 123'5812 mlcod 123'5812 active+clean+snaptrim trimq=[208~20] ps=[1fe~a]] removing snap head 2019-12-05T06:51:15.973 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:15.970+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5751 (115'2705,123'5751] local-lis/les=21/22 n=638 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5750 crt=123'5746 lcod 123'5748 mlcod 123'5748 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:15.974 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.970+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6297 (117'3210,123'6297] local-lis/les=21/22 n=392 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6296 crt=123'6296 lcod 123'6294 mlcod 123'6294 active+clean+snaptrim trimq=[20a~1e] ps=[1fe~c]] removing snap head 2019-12-05T06:51:15.976 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.970+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6104 (117'3007,123'6104] local-lis/les=21/22 n=361 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'6104 lcod 123'6102 mlcod 123'6102 active+clean+snaptrim trimq=[20b~1d] ps=[1fe~d]] removing snap head 2019-12-05T06:51:15.977 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:15.970+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5753 (115'2705,123'5753] local-lis/les=21/22 n=636 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5750 crt=123'5750 lcod 123'5748 mlcod 123'5748 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:15.979 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.974+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5837 (115'2812,123'5837] local-lis/les=21/22 n=826 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'5837 lcod 123'5835 mlcod 123'5835 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:15.984 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.978+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5839 (115'2812,123'5839] local-lis/les=21/22 n=824 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'5837 crt=123'5837 lcod 123'5835 mlcod 123'5835 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:15.984 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.974+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6106 (117'3007,123'6106] local-lis/les=21/22 n=359 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'6104 crt=123'6104 lcod 123'6102 mlcod 123'6102 active+clean+snaptrim trimq=[20b~1d] ps=[1fe~d]] removing snap head 2019-12-05T06:51:15.984 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.974+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6299 (117'3210,123'6299] local-lis/les=21/22 n=390 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6296 crt=123'6296 lcod 123'6294 mlcod 123'6294 active+clean+snaptrim trimq=[20a~1e] ps=[1fe~c]] removing snap head 2019-12-05T06:51:15.985 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.982+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5818 (115'2804,123'5818] local-lis/les=21/22 n=401 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5818 lcod 123'5816 mlcod 123'5816 active+clean+snaptrim trimq=[208~20] ps=[1fe~a]] removing snap head 2019-12-05T06:51:15.986 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:15.982+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5756 (115'2705,123'5756] local-lis/les=21/22 n=633 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5755 crt=123'5751 lcod 123'5753 mlcod 123'5753 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:15.987 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.982+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5820 (115'2804,123'5820] local-lis/les=21/22 n=399 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5818 crt=123'5818 lcod 123'5816 mlcod 123'5816 active+clean+snaptrim trimq=[208~20] ps=[1fe~a]] removing snap head 2019-12-05T06:51:15.987 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:15.982+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5758 (115'2705,123'5758] local-lis/les=21/22 n=631 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5755 crt=123'5755 lcod 123'5753 mlcod 123'5753 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:15.990 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:15.986+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5726 (115'2707,123'5726] local-lis/les=21/22 n=669 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5725 crt=123'5723 lcod 123'5723 mlcod 123'5723 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:15.991 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:15.986+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5728 (115'2707,123'5728] local-lis/les=21/22 n=667 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5725 crt=123'5725 lcod 123'5723 mlcod 123'5723 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:15.994 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.986+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5841 (115'2812,123'5841] local-lis/les=21/22 n=822 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'5841 lcod 123'5839 mlcod 123'5839 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:15.994 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.990+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5843 (115'2812,123'5843] local-lis/les=21/22 n=820 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'5841 crt=123'5841 lcod 123'5839 mlcod 123'5839 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:15.996 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.986+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6108 (117'3007,123'6108] local-lis/les=21/22 n=357 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'6108 lcod 123'6106 mlcod 123'6106 active+clean+snaptrim trimq=[20b~1d] ps=[1fe~d]] removing snap head 2019-12-05T06:51:15.997 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.990+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6301 (117'3210,123'6301] local-lis/les=21/22 n=388 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'6301 lcod 123'6299 mlcod 123'6299 active+clean+snaptrim trimq=[20a~1e] ps=[1fe~c]] removing snap head 2019-12-05T06:51:15.998 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.994+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6110 (117'3108,123'6110] local-lis/les=21/22 n=355 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'6108 crt=123'6108 lcod 123'6106 mlcod 123'6106 active+clean+snaptrim trimq=[20b~1d] ps=[1fe~d]] removing snap head 2019-12-05T06:51:15.998 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:15.994+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6303 (117'3210,123'6303] local-lis/les=21/22 n=386 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6301 crt=123'6301 lcod 123'6299 mlcod 123'6299 active+clean+snaptrim trimq=[20a~1e] ps=[1fe~c]] removing snap head 2019-12-05T06:51:16.003 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:15.997+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5634 (115'2606,123'5634] local-lis/les=21/22 n=884 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5633 crt=123'5631 lcod 123'5631 mlcod 123'5631 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:16.003 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:15.998+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5822 (115'2804,123'5822] local-lis/les=21/22 n=397 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5822 lcod 123'5820 mlcod 123'5820 active+clean+snaptrim trimq=[208~20] ps=[1fe~a]] removing snap head 2019-12-05T06:51:16.007 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:16.002+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5845 (115'2812,123'5845] local-lis/les=21/22 n=818 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'5845 lcod 123'5843 mlcod 123'5843 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:16.007 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:16.002+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6305 (117'3210,123'6305] local-lis/les=21/22 n=384 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'6305 lcod 123'6303 mlcod 123'6303 active+clean+snaptrim trimq=[20a~1e] ps=[1fe~c]] removing snap head 2019-12-05T06:51:16.011 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:16.006+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5847 (115'2812,123'5847] local-lis/les=21/22 n=816 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'5845 crt=123'5845 lcod 123'5843 mlcod 123'5843 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:16.012 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:16.006+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6307 (117'3210,123'6307] local-lis/les=21/22 n=382 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6305 crt=123'6305 lcod 123'6303 mlcod 123'6303 active+clean+snaptrim trimq=[20a~1e] ps=[1fe~c]] removing snap head 2019-12-05T06:51:16.015 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:16.010+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5824 (115'2804,123'5824] local-lis/les=21/22 n=395 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5824 lcod 123'5822 mlcod 123'5822 active+clean+snaptrim trimq=[209~1f] ps=[1fe~b]] removing snap head 2019-12-05T06:51:16.016 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:16.010+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6112 (117'3108,123'6112] local-lis/les=21/22 n=353 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'6112 lcod 123'6110 mlcod 123'6110 active+clean+snaptrim trimq=[20b~1d] ps=[1fe~d]] removing snap head 2019-12-05T06:51:16.016 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:16.010+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5731 (115'2707,123'5731] local-lis/les=21/22 n=664 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5730 crt=123'5728 lcod 123'5728 mlcod 123'5728 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:16.017 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:16.014+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5826 (115'2804,123'5826] local-lis/les=21/22 n=393 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5824 crt=123'5824 lcod 123'5822 mlcod 123'5822 active+clean+snaptrim trimq=[209~1f] ps=[1fe~b]] removing snap head 2019-12-05T06:51:16.017 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:16.014+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6114 (117'3108,123'6114] local-lis/les=21/22 n=351 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'6112 crt=123'6112 lcod 123'6110 mlcod 123'6110 active+clean+snaptrim trimq=[20b~1d] ps=[1fe~d]] removing snap head 2019-12-05T06:51:16.020 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:16.014+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5733 (115'2707,123'5733] local-lis/les=21/22 n=662 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5730 crt=123'5730 lcod 123'5728 mlcod 123'5728 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:16.023 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:16.017+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5639 (115'2606,123'5639] local-lis/les=21/22 n=880 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5638 crt=123'5636 lcod 123'5636 mlcod 123'5636 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:16.024 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:16.022+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5849 (115'2812,123'5849] local-lis/les=21/22 n=814 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'5849 lcod 123'5847 mlcod 123'5847 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:16.025 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:16.017+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5641 (115'2606,123'5641] local-lis/les=21/22 n=878 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5638 crt=123'5638 lcod 123'5636 mlcod 123'5636 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:16.026 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:16.022+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5851 (115'2812,123'5851] local-lis/les=21/22 n=812 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'5849 crt=123'5849 lcod 123'5847 mlcod 123'5847 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:16.028 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:16.022+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6309 (117'3210,123'6309] local-lis/les=21/22 n=380 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'6309 lcod 123'6307 mlcod 123'6307 active+clean+snaptrim trimq=[20a~1e] ps=[1fe~c]] removing snap head 2019-12-05T06:51:16.028 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:16.022+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6311 (117'3210,123'6311] local-lis/les=21/22 n=378 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6309 crt=123'6309 lcod 123'6307 mlcod 123'6307 active+clean+snaptrim trimq=[20a~1e] ps=[1fe~c]] removing snap head 2019-12-05T06:51:16.035 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:16.030+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5761 (115'2705,123'5761] local-lis/les=21/22 n=628 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5760 crt=123'5758 lcod 123'5758 mlcod 123'5758 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:16.036 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:16.034+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5853 (115'2812,123'5853] local-lis/les=21/22 n=810 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'5853 lcod 123'5851 mlcod 123'5851 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:16.037 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:16.034+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5763 (115'2705,123'5763] local-lis/les=21/22 n=626 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5760 crt=123'5760 lcod 123'5758 mlcod 123'5758 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:16.038 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:16.034+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5855 (115'2812,123'5855] local-lis/les=21/22 n=808 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'5853 crt=123'5853 lcod 123'5851 mlcod 123'5851 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:16.042 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:16.038+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5857 (115'2812,123'5857] local-lis/les=21/22 n=806 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'5857 lcod 123'5855 mlcod 123'5855 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:16.043 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:16.038+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5736 (115'2707,123'5736] local-lis/les=21/22 n=659 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5735 crt=123'5733 lcod 123'5733 mlcod 123'5733 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:16.044 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:16.042+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5859 (115'2812,123'5859] local-lis/les=21/22 n=804 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'5857 crt=123'5857 lcod 123'5855 mlcod 123'5855 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:16.046 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:16.042+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5738 (115'2707,123'5738] local-lis/les=21/22 n=657 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5735 crt=123'5735 lcod 123'5733 mlcod 123'5733 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:16.049 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:16.046+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5861 (115'2812,123'5861] local-lis/les=21/22 n=802 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'5861 lcod 123'5859 mlcod 123'5859 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:16.051 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:16.046+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5863 (115'2812,123'5863] local-lis/les=21/22 n=800 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'5861 crt=123'5861 lcod 123'5859 mlcod 123'5859 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:16.052 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:16.050+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5829 (115'2804,123'5829] local-lis/les=21/22 n=390 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5828 crt=123'5826 lcod 123'5826 mlcod 123'5826 active+clean+snaptrim trimq=[209~1f] ps=[1fe~b]] removing snap head 2019-12-05T06:51:16.055 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:16.050+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6313 (117'3311,123'6313] local-lis/les=21/22 n=376 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'6313 lcod 123'6311 mlcod 123'6311 active+clean+snaptrim trimq=[20a~1e] ps=[1fe~c]] removing snap head 2019-12-05T06:51:16.055 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:16.050+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5831 (115'2804,123'5831] local-lis/les=21/22 n=388 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5828 crt=123'5828 lcod 123'5826 mlcod 123'5826 active+clean+snaptrim trimq=[209~1f] ps=[1fe~b]] removing snap head 2019-12-05T06:51:16.056 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:16.054+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6116 (117'3108,123'6116] local-lis/les=21/22 n=349 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'6116 lcod 123'6114 mlcod 123'6114 active+clean+snaptrim trimq=[20b~1d] ps=[1fe~d]] removing snap head 2019-12-05T06:51:16.057 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:16.054+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6315 (117'3311,123'6315] local-lis/les=21/22 n=374 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6313 crt=123'6313 lcod 123'6311 mlcod 123'6311 active+clean+snaptrim trimq=[20a~1e] ps=[1fe~c]] removing snap head 2019-12-05T06:51:16.060 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:16.053+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5643 (115'2606,123'5643] local-lis/les=21/22 n=876 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=123'5643 lcod 123'5641 mlcod 123'5641 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:16.061 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:16.058+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6118 (117'3108,123'6118] local-lis/les=21/22 n=347 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'6116 crt=123'6116 lcod 123'6114 mlcod 123'6114 active+clean+snaptrim trimq=[20b~1d] ps=[1fe~d]] removing snap head 2019-12-05T06:51:16.062 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:16.057+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5645 (115'2606,123'5645] local-lis/les=21/22 n=874 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5643 crt=123'5643 lcod 123'5641 mlcod 123'5641 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:16.063 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:16.058+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5865 (115'2812,123'5865] local-lis/les=21/22 n=798 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'5865 lcod 123'5863 mlcod 123'5863 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:16.064 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:16.062+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5867 (115'2812,123'5867] local-lis/les=21/22 n=796 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'5865 crt=123'5865 lcod 123'5863 mlcod 123'5863 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:16.067 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:16.062+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6317 (117'3311,123'6317] local-lis/les=21/22 n=372 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'6317 lcod 123'6315 mlcod 123'6315 active+clean+snaptrim trimq=[20a~1e] ps=[1fe~c]] removing snap head 2019-12-05T06:51:16.069 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:16.066+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6319 (117'3311,123'6319] local-lis/les=21/22 n=370 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6317 crt=123'6317 lcod 123'6315 mlcod 123'6315 active+clean+snaptrim trimq=[20a~1e] ps=[1fe~c]] removing snap head 2019-12-05T06:51:16.070 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:16.061+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5648 (115'2606,123'5648] local-lis/les=21/22 n=871 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5647 crt=123'5643 lcod 123'5645 mlcod 123'5645 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:16.072 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:16.065+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5650 (115'2606,123'5650] local-lis/les=21/22 n=869 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5647 crt=123'5647 lcod 123'5645 mlcod 123'5645 active+clean+snaptrim trimq=[1ff~29] ps=[1fe~1]] removing snap head 2019-12-05T06:51:16.072 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:16.070+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5869 (115'2812,123'5869] local-lis/les=21/22 n=794 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'5869 lcod 123'5867 mlcod 123'5867 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:16.074 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:16.070+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5871 (115'2812,123'5871] local-lis/les=21/22 n=792 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'5869 crt=123'5869 lcod 123'5867 mlcod 123'5867 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:16.081 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:16.073+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5653 (115'2606,123'5653] local-lis/les=21/22 n=866 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=123'5653 lcod 123'5652 mlcod 123'5652 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:16.082 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:16.078+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5873 (115'2812,123'5873] local-lis/les=21/22 n=790 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'5873 lcod 123'5871 mlcod 123'5871 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:16.083 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:16.077+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5655 (115'2606,123'5655] local-lis/les=21/22 n=864 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5653 crt=123'5653 lcod 123'5652 mlcod 123'5652 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:16.083 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:16.078+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5875 (115'2812,123'5875] local-lis/les=21/22 n=788 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'5873 crt=123'5873 lcod 123'5871 mlcod 123'5871 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:16.088 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:16.081+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5658 (115'2606,123'5658] local-lis/les=21/22 n=861 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5657 crt=123'5653 lcod 123'5655 mlcod 123'5655 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:16.089 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:16.086+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5877 (115'2812,123'5877] local-lis/les=21/22 n=786 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'5877 lcod 123'5875 mlcod 123'5875 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:16.091 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:16.086+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5879 (115'2812,123'5879] local-lis/les=21/22 n=784 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'5877 crt=123'5877 lcod 123'5875 mlcod 123'5875 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:16.092 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:16.085+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5660 (115'2606,123'5660] local-lis/les=21/22 n=859 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5657 crt=123'5657 lcod 123'5655 mlcod 123'5655 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:16.096 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:16.094+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5881 (115'2812,123'5881] local-lis/les=21/22 n=782 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'5881 lcod 123'5879 mlcod 123'5879 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:16.097 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:16.089+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5663 (115'2606,123'5663] local-lis/les=21/22 n=856 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5662 crt=123'5658 lcod 123'5660 mlcod 123'5660 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:16.098 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:16.094+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5883 (115'2812,123'5883] local-lis/les=21/22 n=780 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'5881 crt=123'5881 lcod 123'5879 mlcod 123'5879 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:16.100 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:16.093+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5665 (115'2606,123'5665] local-lis/les=21/22 n=854 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5662 crt=123'5662 lcod 123'5660 mlcod 123'5660 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:16.106 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:16.097+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5667 (115'2606,123'5667] local-lis/les=21/22 n=852 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=123'5667 lcod 123'5665 mlcod 123'5665 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:16.108 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:16.102+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5885 (115'2812,123'5885] local-lis/les=21/22 n=778 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'5885 lcod 123'5883 mlcod 123'5883 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:16.108 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:16.106+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6121 (117'3108,123'6121] local-lis/les=21/22 n=344 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'6120 crt=123'6118 lcod 123'6118 mlcod 123'6118 active+clean+snaptrim trimq=[20c~1c] ps=[1fe~e]] removing snap head 2019-12-05T06:51:16.109 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:16.101+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5669 (115'2606,123'5669] local-lis/les=21/22 n=850 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5667 crt=123'5667 lcod 123'5665 mlcod 123'5665 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:16.110 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:16.106+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6321 (117'3311,123'6321] local-lis/les=21/22 n=368 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'6321 lcod 123'6319 mlcod 123'6319 active+clean+snaptrim trimq=[20a~1e] ps=[1fe~c]] removing snap head 2019-12-05T06:51:16.111 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:16.106+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6123 (117'3108,123'6123] local-lis/les=21/22 n=342 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'6120 crt=123'6120 lcod 123'6118 mlcod 123'6118 active+clean+snaptrim trimq=[20c~1c] ps=[1fe~e]] removing snap head 2019-12-05T06:51:16.112 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:16.106+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6323 (117'3311,123'6323] local-lis/les=21/22 n=366 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6321 crt=123'6321 lcod 123'6319 mlcod 123'6319 active+clean+snaptrim trimq=[20a~1e] ps=[1fe~c]] removing snap head 2019-12-05T06:51:16.114 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:16.106+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5887 (115'2812,123'5887] local-lis/les=21/22 n=776 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'5885 crt=123'5885 lcod 123'5883 mlcod 123'5883 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:16.118 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:16.114+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5766 (115'2705,123'5766] local-lis/les=21/22 n=623 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5765 crt=123'5763 lcod 123'5763 mlcod 123'5763 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:16.118 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:16.113+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5671 (115'2606,123'5671] local-lis/les=21/22 n=848 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=123'5671 lcod 123'5669 mlcod 123'5669 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:16.119 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:16.114+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5889 (115'2812,123'5889] local-lis/les=21/22 n=774 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'5889 lcod 123'5887 mlcod 123'5887 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:16.120 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:16.118+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5768 (115'2705,123'5768] local-lis/les=21/22 n=621 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5765 crt=123'5765 lcod 123'5763 mlcod 123'5763 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:16.121 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:16.113+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5673 (115'2606,123'5673] local-lis/les=21/22 n=846 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5671 crt=123'5671 lcod 123'5669 mlcod 123'5669 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:16.127 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:16.121+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5750 (115'2706,123'5750] local-lis/les=21/22 n=519 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5749 crt=123'5747 lcod 123'5747 mlcod 123'5747 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:16.130 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:16.121+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5675 (115'2606,123'5675] local-lis/les=21/22 n=844 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=123'5675 lcod 123'5673 mlcod 123'5673 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:16.130 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:16.121+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5752 (115'2706,123'5752] local-lis/les=21/22 n=517 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5749 crt=123'5749 lcod 123'5747 mlcod 123'5747 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:16.132 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:16.125+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5677 (115'2606,123'5677] local-lis/les=21/22 n=842 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5675 crt=123'5675 lcod 123'5673 mlcod 123'5673 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:16.136 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:16.134+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5771 (115'2705,123'5771] local-lis/les=21/22 n=618 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5770 crt=123'5768 lcod 123'5768 mlcod 123'5768 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:16.137 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:16.134+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5773 (115'2705,123'5773] local-lis/les=21/22 n=616 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5770 crt=123'5770 lcod 123'5768 mlcod 123'5768 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:16.138 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:16.129+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5679 (115'2606,123'5679] local-lis/les=21/22 n=840 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=123'5679 lcod 123'5677 mlcod 123'5677 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:16.140 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:16.133+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5681 (115'2606,123'5681] local-lis/les=21/22 n=838 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5679 crt=123'5679 lcod 123'5677 mlcod 123'5677 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:16.143 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:16.137+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5755 (115'2706,123'5755] local-lis/les=21/22 n=514 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5754 crt=123'5752 lcod 123'5752 mlcod 123'5752 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:16.145 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:16.137+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5757 (115'2706,123'5757] local-lis/les=21/22 n=512 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5754 crt=123'5754 lcod 123'5752 mlcod 123'5752 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:16.146 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:16.141+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5683 (115'2606,123'5683] local-lis/les=21/22 n=836 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=123'5683 lcod 123'5681 mlcod 123'5681 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:16.149 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:16.141+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5685 (115'2606,123'5685] local-lis/les=21/22 n=834 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5683 crt=123'5683 lcod 123'5681 mlcod 123'5681 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:16.150 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:16.146+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6126 (117'3108,123'6126] local-lis/les=21/22 n=339 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'6125 crt=123'6123 lcod 123'6123 mlcod 123'6123 active+clean+snaptrim trimq=[20c~1c] ps=[1fe~e]] removing snap head 2019-12-05T06:51:16.153 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:16.150+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6128 (117'3108,123'6128] local-lis/les=21/22 n=337 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'6125 crt=123'6125 lcod 123'6123 mlcod 123'6123 active+clean+snaptrim trimq=[20c~1c] ps=[1fe~e]] removing snap head 2019-12-05T06:51:16.153 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:16.150+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6325 (117'3311,123'6325] local-lis/les=21/22 n=364 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'6325 lcod 123'6323 mlcod 123'6323 active+clean+snaptrim trimq=[20a~1e] ps=[1fe~c]] removing snap head 2019-12-05T06:51:16.155 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:16.150+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6327 (117'3311,123'6327] local-lis/les=21/22 n=362 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6325 crt=123'6325 lcod 123'6323 mlcod 123'6323 active+clean+snaptrim trimq=[20a~1e] ps=[1fe~c]] removing snap head 2019-12-05T06:51:16.157 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:16.149+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5687 (115'2606,123'5687] local-lis/les=21/22 n=832 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=123'5687 lcod 123'5685 mlcod 123'5685 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:16.158 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:16.153+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5689 (115'2606,123'5689] local-lis/les=21/22 n=830 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5687 crt=123'5687 lcod 123'5685 mlcod 123'5685 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:16.159 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:16.154+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5894 (115'2812,123'5894] local-lis/les=21/22 n=770 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'5893 crt=123'5891 lcod 123'5891 mlcod 123'5891 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:16.161 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:16.158+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5896 (115'2812,123'5896] local-lis/les=21/22 n=768 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'5893 crt=123'5893 lcod 123'5891 mlcod 123'5891 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:16.165 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:16.157+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5691 (115'2606,123'5691] local-lis/les=21/22 n=828 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=123'5691 lcod 123'5689 mlcod 123'5689 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:16.167 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:16.162+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5899 (115'2812,123'5899] local-lis/les=21/22 n=765 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'5898 crt=123'5894 lcod 123'5896 mlcod 123'5896 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:16.167 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:16.161+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5693 (115'2606,123'5693] local-lis/les=21/22 n=826 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5691 crt=123'5691 lcod 123'5689 mlcod 123'5689 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:16.168 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:16.166+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5901 (115'2812,123'5901] local-lis/les=21/22 n=763 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'5898 crt=123'5898 lcod 123'5896 mlcod 123'5896 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:16.173 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:16.165+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5695 (115'2606,123'5695] local-lis/les=21/22 n=824 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=123'5695 lcod 123'5693 mlcod 123'5693 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:16.175 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:16.170+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5904 (115'2812,123'5904] local-lis/les=21/22 n=760 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'5903 crt=123'5899 lcod 123'5901 mlcod 123'5901 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:16.176 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:16.169+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5697 (115'2606,123'5697] local-lis/les=21/22 n=822 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5695 crt=123'5695 lcod 123'5693 mlcod 123'5693 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:16.180 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:16.173+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5699 (115'2606,123'5699] local-lis/les=21/22 n=820 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=123'5699 lcod 123'5697 mlcod 123'5697 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:16.181 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:16.178+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5776 (115'2705,123'5776] local-lis/les=21/22 n=613 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5775 crt=123'5773 lcod 123'5773 mlcod 123'5773 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:16.182 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:16.178+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5778 (115'2705,123'5778] local-lis/les=21/22 n=611 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5775 crt=123'5775 lcod 123'5773 mlcod 123'5773 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:16.182 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:16.177+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5701 (115'2606,123'5701] local-lis/les=21/22 n=818 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5699 crt=123'5699 lcod 123'5697 mlcod 123'5697 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:16.188 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:16.186+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6130 (117'3108,123'6130] local-lis/les=21/22 n=335 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'6130 lcod 123'6128 mlcod 123'6128 active+clean+snaptrim trimq=[20c~1c] ps=[1fe~e]] removing snap head 2019-12-05T06:51:16.188 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:16.186+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5834 (115'2804,123'5834] local-lis/les=21/22 n=385 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5833 crt=123'5831 lcod 123'5831 mlcod 123'5831 active+clean+snaptrim trimq=[209~1f] ps=[1fe~b]] removing snap head 2019-12-05T06:51:16.190 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:16.186+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6132 (117'3108,123'6132] local-lis/les=21/22 n=333 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'6130 crt=123'6130 lcod 123'6128 mlcod 123'6128 active+clean+snaptrim trimq=[20c~1c] ps=[1fe~e]] removing snap head 2019-12-05T06:51:16.192 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:16.186+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5836 (115'2804,123'5836] local-lis/les=21/22 n=383 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5833 crt=123'5833 lcod 123'5831 mlcod 123'5831 active+clean+snaptrim trimq=[209~1f] ps=[1fe~b]] removing snap head 2019-12-05T06:51:16.192 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:16.190+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5741 (115'2707,123'5741] local-lis/les=21/22 n=654 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5740 crt=123'5738 lcod 123'5738 mlcod 123'5738 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:16.195 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:16.190+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6329 (117'3311,123'6329] local-lis/les=21/22 n=360 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'6329 lcod 123'6327 mlcod 123'6327 active+clean+snaptrim trimq=[20a~1e] ps=[1fe~c]] removing snap head 2019-12-05T06:51:16.196 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:16.194+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6331 (117'3311,123'6331] local-lis/les=21/22 n=358 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6329 crt=123'6329 lcod 123'6327 mlcod 123'6327 active+clean+snaptrim trimq=[20a~1e] ps=[1fe~c]] removing snap head 2019-12-05T06:51:16.202 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:16.198+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5907 (115'2812,123'5907] local-lis/les=21/22 n=757 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'5906 crt=123'5904 lcod 123'5904 mlcod 123'5904 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:16.202 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:16.198+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6334 (117'3311,123'6334] local-lis/les=21/22 n=355 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6333 crt=123'6329 lcod 123'6331 mlcod 123'6331 active+clean+snaptrim trimq=[20a~1e] ps=[1fe~c]] removing snap head 2019-12-05T06:51:16.205 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:16.202+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5909 (115'2812,123'5909] local-lis/les=21/22 n=755 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'5906 crt=123'5906 lcod 123'5904 mlcod 123'5904 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:16.207 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:16.202+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6336 (117'3311,123'6336] local-lis/les=21/22 n=353 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6333 crt=123'6333 lcod 123'6331 mlcod 123'6331 active+clean+snaptrim trimq=[20a~1e] ps=[1fe~c]] removing snap head 2019-12-05T06:51:16.212 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:16.210+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5744 (115'2707,123'5744] local-lis/les=21/22 n=651 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5743 crt=123'5741 lcod 123'5741 mlcod 123'5741 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:16.213 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:16.210+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6339 (117'3311,123'6339] local-lis/les=21/22 n=350 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6338 crt=123'6336 lcod 123'6336 mlcod 123'6336 active+clean+snaptrim trimq=[20a~1e] ps=[1fe~c]] removing snap head 2019-12-05T06:51:16.214 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:16.210+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5746 (115'2707,123'5746] local-lis/les=21/22 n=649 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5743 crt=123'5743 lcod 123'5741 mlcod 123'5741 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:16.219 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:16.213+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5704 (115'2606,123'5704] local-lis/les=21/22 n=815 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5703 crt=123'5701 lcod 123'5701 mlcod 123'5701 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:16.220 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:16.218+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5912 (115'2812,123'5912] local-lis/les=21/22 n=752 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'5911 crt=123'5909 lcod 123'5909 mlcod 123'5909 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:16.221 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:16.213+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5706 (115'2606,123'5706] local-lis/les=21/22 n=813 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5703 crt=123'5703 lcod 123'5701 mlcod 123'5701 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:16.222 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:16.218+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5914 (115'2812,123'5914] local-lis/les=21/22 n=750 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'5911 crt=123'5911 lcod 123'5909 mlcod 123'5909 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:16.231 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:16.226+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6135 (117'3108,123'6135] local-lis/les=21/22 n=330 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'6134 crt=123'6132 lcod 123'6132 mlcod 123'6132 active+clean+snaptrim trimq=[20c~1c] ps=[1fe~e]] removing snap head 2019-12-05T06:51:16.233 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:16.230+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6137 (117'3108,123'6137] local-lis/les=21/22 n=328 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'6134 crt=123'6134 lcod 123'6132 mlcod 123'6132 active+clean+snaptrim trimq=[20c~1c] ps=[1fe~e]] removing snap head 2019-12-05T06:51:16.235 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:16.226+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5781 (115'2705,123'5781] local-lis/les=21/22 n=608 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5780 crt=123'5778 lcod 123'5778 mlcod 123'5778 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:16.236 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:16.230+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5783 (115'2705,123'5783] local-lis/les=21/22 n=606 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5780 crt=123'5780 lcod 123'5778 mlcod 123'5778 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:16.240 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:16.234+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5785 (115'2705,123'5785] local-lis/les=21/22 n=604 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5785 lcod 123'5783 mlcod 123'5783 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:16.241 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:16.238+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5787 (115'2705,123'5787] local-lis/les=21/22 n=602 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5785 crt=123'5785 lcod 123'5783 mlcod 123'5783 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:16.242 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:16.238+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5917 (115'2914,123'5917] local-lis/les=21/22 n=747 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'5916 crt=123'5914 lcod 123'5914 mlcod 123'5914 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:16.242 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:16.238+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5839 (115'2804,123'5839] local-lis/les=21/22 n=380 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5838 crt=123'5836 lcod 123'5836 mlcod 123'5836 active+clean+snaptrim trimq=[209~1f] ps=[1fe~b]] removing snap head 2019-12-05T06:51:16.243 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:16.238+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5841 (115'2804,123'5841] local-lis/les=21/22 n=378 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5838 crt=123'5838 lcod 123'5836 mlcod 123'5836 active+clean+snaptrim trimq=[209~1f] ps=[1fe~b]] removing snap head 2019-12-05T06:51:16.245 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:16.242+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5919 (115'2914,123'5919] local-lis/les=21/22 n=745 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'5916 crt=123'5916 lcod 123'5914 mlcod 123'5914 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:16.249 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:16.241+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5759 (115'2706,123'5759] local-lis/les=21/22 n=510 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5759 lcod 123'5757 mlcod 123'5757 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:16.250 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:16.245+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5761 (115'2706,123'5761] local-lis/les=21/22 n=508 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5759 crt=123'5759 lcod 123'5757 mlcod 123'5757 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:16.253 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:16.250+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6140 (117'3108,123'6140] local-lis/les=21/22 n=325 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'6139 crt=123'6137 lcod 123'6137 mlcod 123'6137 active+clean+snaptrim trimq=[20c~1c] ps=[1fe~e]] removing snap head 2019-12-05T06:51:16.255 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:16.250+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6142 (117'3108,123'6142] local-lis/les=21/22 n=323 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'6139 crt=123'6139 lcod 123'6137 mlcod 123'6137 active+clean+snaptrim trimq=[20c~1c] ps=[1fe~e]] removing snap head 2019-12-05T06:51:16.256 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:16.254+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5790 (115'2705,123'5790] local-lis/les=21/22 n=599 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5790 lcod 123'5789 mlcod 123'5789 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:16.258 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:16.253+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5763 (115'2706,123'5763] local-lis/les=21/22 n=506 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5763 lcod 123'5761 mlcod 123'5761 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:16.259 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:16.254+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5792 (115'2705,123'5792] local-lis/les=21/22 n=597 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5790 crt=123'5790 lcod 123'5789 mlcod 123'5789 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:16.260 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:16.253+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5765 (115'2706,123'5765] local-lis/les=21/22 n=504 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5763 crt=123'5763 lcod 123'5761 mlcod 123'5761 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:16.267 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:16.261+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5767 (115'2706,123'5767] local-lis/les=21/22 n=502 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5767 lcod 123'5765 mlcod 123'5765 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:16.267 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:16.262+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5795 (115'2705,123'5795] local-lis/les=21/22 n=594 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5794 crt=123'5790 lcod 123'5792 mlcod 123'5792 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:16.268 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:16.266+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5797 (115'2705,123'5797] local-lis/les=21/22 n=592 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5794 crt=123'5794 lcod 123'5792 mlcod 123'5792 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:16.269 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:16.261+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5769 (115'2706,123'5769] local-lis/les=21/22 n=500 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5767 crt=123'5767 lcod 123'5765 mlcod 123'5765 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:16.273 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:16.270+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5843 (115'2804,123'5843] local-lis/les=21/22 n=376 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5843 lcod 123'5841 mlcod 123'5841 active+clean+snaptrim trimq=[209~1f] ps=[1fe~b]] removing snap head 2019-12-05T06:51:16.275 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:16.269+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5771 (115'2706,123'5771] local-lis/les=21/22 n=498 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5771 lcod 123'5769 mlcod 123'5769 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:16.276 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:16.274+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5845 (115'2804,123'5845] local-lis/les=21/22 n=374 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5843 crt=123'5843 lcod 123'5841 mlcod 123'5841 active+clean+snaptrim trimq=[209~1f] ps=[1fe~b]] removing snap head 2019-12-05T06:51:16.277 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:16.269+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5773 (115'2706,123'5773] local-lis/les=21/22 n=496 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5771 crt=123'5771 lcod 123'5769 mlcod 123'5769 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:16.281 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:16.273+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5775 (115'2706,123'5775] local-lis/les=21/22 n=494 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5775 lcod 123'5773 mlcod 123'5773 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:16.282 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:16.277+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5777 (115'2706,123'5777] local-lis/les=21/22 n=492 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5775 crt=123'5775 lcod 123'5773 mlcod 123'5773 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:16.286 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:16.281+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5779 (115'2706,123'5779] local-lis/les=21/22 n=490 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5779 lcod 123'5777 mlcod 123'5777 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:16.288 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:16.281+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5781 (115'2706,123'5781] local-lis/les=21/22 n=488 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5779 crt=123'5779 lcod 123'5777 mlcod 123'5777 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:16.291 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:16.285+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5783 (115'2706,123'5783] local-lis/les=21/22 n=486 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5783 lcod 123'5781 mlcod 123'5781 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:16.292 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:16.285+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5785 (115'2706,123'5785] local-lis/les=21/22 n=484 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5783 crt=123'5783 lcod 123'5781 mlcod 123'5781 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:16.296 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:16.289+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5787 (115'2706,123'5787] local-lis/les=21/22 n=482 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5787 lcod 123'5785 mlcod 123'5785 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:16.297 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:16.289+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5789 (115'2706,123'5789] local-lis/les=21/22 n=480 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5787 crt=123'5787 lcod 123'5785 mlcod 123'5785 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:16.302 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:16.293+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5791 (115'2706,123'5791] local-lis/les=21/22 n=478 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5791 lcod 123'5789 mlcod 123'5789 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:16.303 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:16.297+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5793 (115'2706,123'5793] local-lis/les=21/22 n=476 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5791 crt=123'5791 lcod 123'5789 mlcod 123'5789 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:16.307 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:16.301+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5795 (115'2706,123'5795] local-lis/les=21/22 n=474 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5795 lcod 123'5793 mlcod 123'5793 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:16.309 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:16.301+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5797 (115'2706,123'5797] local-lis/les=21/22 n=472 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5795 crt=123'5795 lcod 123'5793 mlcod 123'5793 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:16.312 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:16.305+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5799 (115'2706,123'5799] local-lis/les=21/22 n=470 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5799 lcod 123'5797 mlcod 123'5797 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:16.314 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:16.309+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5801 (115'2706,123'5801] local-lis/les=21/22 n=468 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5799 crt=123'5799 lcod 123'5797 mlcod 123'5797 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:16.318 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:16.309+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5803 (115'2706,123'5803] local-lis/les=21/22 n=466 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5803 lcod 123'5801 mlcod 123'5801 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:16.319 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:16.313+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5805 (115'2706,123'5805] local-lis/les=21/22 n=464 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5803 crt=123'5803 lcod 123'5801 mlcod 123'5801 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:16.323 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:16.317+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5807 (115'2706,123'5807] local-lis/les=21/22 n=462 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5807 lcod 123'5805 mlcod 123'5805 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:16.325 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:16.317+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5809 (115'2807,123'5809] local-lis/les=21/22 n=460 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5807 crt=123'5807 lcod 123'5805 mlcod 123'5805 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:16.330 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:16.325+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5811 (115'2807,123'5811] local-lis/les=21/22 n=458 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5811 lcod 123'5809 mlcod 123'5809 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:16.331 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:16.325+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5813 (115'2807,123'5813] local-lis/les=21/22 n=456 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5811 crt=123'5811 lcod 123'5809 mlcod 123'5809 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:16.463 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:16.458+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6144 (117'3108,123'6144] local-lis/les=21/22 n=321 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'6144 lcod 123'6142 mlcod 123'6142 active+clean+snaptrim trimq=[20d~1b] ps=[1fe~f]] removing snap head 2019-12-05T06:51:16.464 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:16.462+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6341 (117'3311,123'6341] local-lis/les=21/22 n=348 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'6341 lcod 123'6339 mlcod 123'6339 active+clean+snaptrim trimq=[20b~1d] ps=[1fe~d]] removing snap head 2019-12-05T06:51:16.468 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:16.462+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6146 (117'3108,123'6146] local-lis/les=21/22 n=319 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'6144 crt=123'6144 lcod 123'6142 mlcod 123'6142 active+clean+snaptrim trimq=[20d~1b] ps=[1fe~f]] removing snap head 2019-12-05T06:51:16.469 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:16.466+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6343 (117'3311,123'6343] local-lis/les=21/22 n=346 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6341 crt=123'6341 lcod 123'6339 mlcod 123'6339 active+clean+snaptrim trimq=[20b~1d] ps=[1fe~d]] removing snap head 2019-12-05T06:51:16.477 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:16.474+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6148 (117'3108,123'6148] local-lis/les=21/22 n=317 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'6148 lcod 123'6146 mlcod 123'6146 active+clean+snaptrim trimq=[20d~1b] ps=[1fe~f]] removing snap head 2019-12-05T06:51:16.479 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:16.474+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6150 (117'3108,123'6150] local-lis/les=21/22 n=315 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'6148 crt=123'6148 lcod 123'6146 mlcod 123'6146 active+clean+snaptrim trimq=[20d~1b] ps=[1fe~f]] removing snap head 2019-12-05T06:51:16.481 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:16.474+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5847 (115'2804,123'5847] local-lis/les=21/22 n=372 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5847 lcod 123'5845 mlcod 123'5845 active+clean+snaptrim trimq=[20a~1e] ps=[1fe~c]] removing snap head 2019-12-05T06:51:16.482 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:16.474+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5849 (115'2804,123'5849] local-lis/les=21/22 n=370 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5847 crt=123'5847 lcod 123'5845 mlcod 123'5845 active+clean+snaptrim trimq=[20a~1e] ps=[1fe~c]] removing snap head 2019-12-05T06:51:16.485 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:16.482+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6345 (117'3311,123'6345] local-lis/les=21/22 n=344 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'6345 lcod 123'6343 mlcod 123'6343 active+clean+snaptrim trimq=[20b~1d] ps=[1fe~d]] removing snap head 2019-12-05T06:51:16.487 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:16.482+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5851 (115'2804,123'5851] local-lis/les=21/22 n=368 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5851 lcod 123'5849 mlcod 123'5849 active+clean+snaptrim trimq=[20a~1e] ps=[1fe~c]] removing snap head 2019-12-05T06:51:16.497 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:16.486+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6347 (117'3311,123'6347] local-lis/les=21/22 n=342 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6345 crt=123'6345 lcod 123'6343 mlcod 123'6343 active+clean+snaptrim trimq=[20b~1d] ps=[1fe~d]] removing snap head 2019-12-05T06:51:16.497 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:16.490+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6152 (117'3108,123'6152] local-lis/les=21/22 n=313 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'6152 lcod 123'6150 mlcod 123'6150 active+clean+snaptrim trimq=[20d~1b] ps=[1fe~f]] removing snap head 2019-12-05T06:51:16.497 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:16.494+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6154 (117'3108,123'6154] local-lis/les=21/22 n=311 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'6152 crt=123'6152 lcod 123'6150 mlcod 123'6150 active+clean+snaptrim trimq=[20d~1b] ps=[1fe~f]] removing snap head 2019-12-05T06:51:16.497 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:16.486+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5853 (115'2804,123'5853] local-lis/les=21/22 n=366 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5851 crt=123'5851 lcod 123'5849 mlcod 123'5849 active+clean+snaptrim trimq=[20a~1e] ps=[1fe~c]] removing snap head 2019-12-05T06:51:16.501 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:16.498+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5855 (115'2804,123'5855] local-lis/les=21/22 n=364 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5855 lcod 123'5853 mlcod 123'5853 active+clean+snaptrim trimq=[20a~1e] ps=[1fe~c]] removing snap head 2019-12-05T06:51:16.504 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:16.498+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5857 (115'2804,123'5857] local-lis/les=21/22 n=362 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5855 crt=123'5855 lcod 123'5853 mlcod 123'5853 active+clean+snaptrim trimq=[20a~1e] ps=[1fe~c]] removing snap head 2019-12-05T06:51:16.507 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:16.502+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5800 (115'2705,123'5800] local-lis/les=21/22 n=589 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5799 crt=123'5797 lcod 123'5797 mlcod 123'5797 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:16.508 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:16.497+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5816 (115'2807,123'5816] local-lis/les=21/22 n=453 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5815 crt=123'5813 lcod 123'5813 mlcod 123'5813 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:16.509 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:16.501+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5818 (115'2807,123'5818] local-lis/les=21/22 n=451 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5815 crt=123'5815 lcod 123'5813 mlcod 123'5813 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:16.510 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:16.506+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5802 (115'2705,123'5802] local-lis/les=21/22 n=587 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5799 crt=123'5799 lcod 123'5797 mlcod 123'5797 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:16.514 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:16.509+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5820 (115'2807,123'5820] local-lis/les=21/22 n=449 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5820 lcod 123'5818 mlcod 123'5818 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:16.519 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:16.513+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5822 (115'2807,123'5822] local-lis/les=21/22 n=447 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5820 crt=123'5820 lcod 123'5818 mlcod 123'5818 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:16.520 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:16.510+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6350 (117'3311,123'6350] local-lis/les=21/22 n=339 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6349 crt=123'6347 lcod 123'6347 mlcod 123'6347 active+clean+snaptrim trimq=[20b~1d] ps=[1fe~d]] removing snap head 2019-12-05T06:51:16.520 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:16.514+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6352 (117'3311,123'6352] local-lis/les=21/22 n=337 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6349 crt=123'6349 lcod 123'6347 mlcod 123'6347 active+clean+snaptrim trimq=[20b~1d] ps=[1fe~d]] removing snap head 2019-12-05T06:51:16.523 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:16.518+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6156 (117'3108,123'6156] local-lis/les=21/22 n=309 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'6156 lcod 123'6154 mlcod 123'6154 active+clean+snaptrim trimq=[20d~1b] ps=[1fe~f]] removing snap head 2019-12-05T06:51:16.526 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:16.518+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5860 (115'2804,123'5860] local-lis/les=21/22 n=359 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5859 crt=123'5857 lcod 123'5857 mlcod 123'5857 active+clean+snaptrim trimq=[20a~1e] ps=[1fe~c]] removing snap head 2019-12-05T06:51:16.526 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:16.522+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5862 (115'2804,123'5862] local-lis/les=21/22 n=357 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5859 crt=123'5859 lcod 123'5857 mlcod 123'5857 active+clean+snaptrim trimq=[20a~1e] ps=[1fe~c]] removing snap head 2019-12-05T06:51:16.526 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:16.522+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6158 (117'3108,123'6158] local-lis/les=21/22 n=307 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'6156 crt=123'6156 lcod 123'6154 mlcod 123'6154 active+clean+snaptrim trimq=[20d~1b] ps=[1fe~f]] removing snap head 2019-12-05T06:51:16.534 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:16.530+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5864 (115'2804,123'5864] local-lis/les=21/22 n=355 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5864 lcod 123'5862 mlcod 123'5862 active+clean+snaptrim trimq=[20a~1e] ps=[1fe~c]] removing snap head 2019-12-05T06:51:16.535 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:16.530+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6162 (117'3108,123'6162] local-lis/les=21/22 n=303 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'6160 crt=123'6156 lcod 123'6158 mlcod 123'6158 active+clean+snaptrim trimq=[20d~1b] ps=[1fe~f]] removing snap head 2019-12-05T06:51:16.538 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:16.534+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5866 (115'2804,123'5866] local-lis/les=21/22 n=353 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5864 crt=123'5864 lcod 123'5862 mlcod 123'5862 active+clean+snaptrim trimq=[20a~1e] ps=[1fe~c]] removing snap head 2019-12-05T06:51:16.539 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:16.534+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6164 (117'3108,123'6164] local-lis/les=21/22 n=301 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'6160 crt=123'6160 lcod 123'6158 mlcod 123'6158 active+clean+snaptrim trimq=[20d~1b] ps=[1fe~f]] removing snap head 2019-12-05T06:51:16.544 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:16.542+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5922 (115'2914,123'5922] local-lis/les=21/22 n=742 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'5921 crt=123'5919 lcod 123'5919 mlcod 123'5919 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:16.546 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:16.542+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6167 (117'3108,123'6167] local-lis/les=21/22 n=298 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'6166 crt=123'6162 lcod 123'6164 mlcod 123'6164 active+clean+snaptrim trimq=[20d~1b] ps=[1fe~f]] removing snap head 2019-12-05T06:51:16.547 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:16.542+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5924 (115'2914,123'5924] local-lis/les=21/22 n=740 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'5921 crt=123'5921 lcod 123'5919 mlcod 123'5919 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:16.550 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:16.542+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5868 (115'2804,123'5868] local-lis/les=21/22 n=351 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5868 lcod 123'5866 mlcod 123'5866 active+clean+snaptrim trimq=[20a~1e] ps=[1fe~c]] removing snap head 2019-12-05T06:51:16.551 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:16.546+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5870 (115'2804,123'5870] local-lis/les=21/22 n=349 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5868 crt=123'5868 lcod 123'5866 mlcod 123'5866 active+clean+snaptrim trimq=[20a~1e] ps=[1fe~c]] removing snap head 2019-12-05T06:51:16.552 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:16.546+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6169 (117'3108,123'6169] local-lis/les=21/22 n=296 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'6166 crt=123'6166 lcod 123'6164 mlcod 123'6164 active+clean+snaptrim trimq=[20d~1b] ps=[1fe~f]] removing snap head 2019-12-05T06:51:16.574 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:16.553+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5709 (115'2706,123'5709] local-lis/les=21/22 n=810 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5708 crt=123'5706 lcod 123'5706 mlcod 123'5706 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:16.574 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:16.553+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5711 (115'2706,123'5711] local-lis/les=21/22 n=808 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5708 crt=123'5708 lcod 123'5706 mlcod 123'5706 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:16.576 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:16.561+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5714 (115'2706,123'5714] local-lis/les=21/22 n=805 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5713 crt=123'5709 lcod 123'5711 mlcod 123'5711 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:16.576 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:16.565+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5716 (115'2706,123'5716] local-lis/les=21/22 n=803 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5713 crt=123'5713 lcod 123'5711 mlcod 123'5711 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:16.576 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:16.554+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6172 (117'3108,123'6172] local-lis/les=21/22 n=293 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'6171 crt=123'6166 lcod 123'6169 mlcod 123'6169 active+clean+snaptrim trimq=[20d~1b] ps=[1fe~f]] removing snap head 2019-12-05T06:51:16.576 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:16.554+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6174 (117'3108,123'6174] local-lis/les=21/22 n=291 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'6171 crt=123'6171 lcod 123'6169 mlcod 123'6169 active+clean+snaptrim trimq=[20d~1b] ps=[1fe~f]] removing snap head 2019-12-05T06:51:16.577 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:16.562+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6177 (117'3108,123'6177] local-lis/les=21/22 n=288 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'6176 crt=123'6172 lcod 123'6174 mlcod 123'6174 active+clean+snaptrim trimq=[20d~1b] ps=[1fe~f]] removing snap head 2019-12-05T06:51:16.577 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:16.558+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5872 (115'2804,123'5872] local-lis/les=21/22 n=347 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5872 lcod 123'5870 mlcod 123'5870 active+clean+snaptrim trimq=[20a~1e] ps=[1fe~c]] removing snap head 2019-12-05T06:51:16.577 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:16.566+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5874 (115'2804,123'5874] local-lis/les=21/22 n=345 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5872 crt=123'5872 lcod 123'5870 mlcod 123'5870 active+clean+snaptrim trimq=[20a~1e] ps=[1fe~c]] removing snap head 2019-12-05T06:51:16.580 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:16.569+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5719 (115'2706,123'5719] local-lis/les=21/22 n=800 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5718 crt=123'5716 lcod 123'5716 mlcod 123'5716 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:16.580 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:16.573+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5721 (115'2706,123'5721] local-lis/les=21/22 n=798 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5718 crt=123'5718 lcod 123'5716 mlcod 123'5716 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:16.581 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:16.574+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5927 (115'2914,123'5927] local-lis/les=21/22 n=737 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'5926 crt=123'5924 lcod 123'5924 mlcod 123'5924 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:16.581 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:16.574+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5929 (115'2914,123'5929] local-lis/les=21/22 n=735 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'5926 crt=123'5926 lcod 123'5924 mlcod 123'5924 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:16.590 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:16.586+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5876 (115'2804,123'5876] local-lis/les=21/22 n=343 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5876 lcod 123'5874 mlcod 123'5874 active+clean+snaptrim trimq=[20b~1d] ps=[1fe~d]] removing snap head 2019-12-05T06:51:16.590 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:16.586+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6180 (117'3108,123'6180] local-lis/les=21/22 n=285 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'6179 crt=123'6177 lcod 123'6177 mlcod 123'6177 active+clean+snaptrim trimq=[20e~1a] ps=[1fe~10]] removing snap head 2019-12-05T06:51:16.590 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:16.581+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5723 (115'2706,123'5723] local-lis/les=21/22 n=796 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=123'5723 lcod 123'5721 mlcod 123'5721 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:16.592 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:16.577+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5825 (115'2807,123'5825] local-lis/les=21/22 n=444 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5824 crt=123'5822 lcod 123'5822 mlcod 123'5822 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:16.592 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:16.581+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5827 (115'2807,123'5827] local-lis/les=21/22 n=442 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5824 crt=123'5824 lcod 123'5822 mlcod 123'5822 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:16.593 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:16.590+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6182 (117'3108,123'6182] local-lis/les=21/22 n=283 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'6179 crt=123'6179 lcod 123'6177 mlcod 123'6177 active+clean+snaptrim trimq=[20e~1a] ps=[1fe~10]] removing snap head 2019-12-05T06:51:16.594 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:16.590+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5878 (115'2804,123'5878] local-lis/les=21/22 n=341 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5876 crt=123'5876 lcod 123'5874 mlcod 123'5874 active+clean+snaptrim trimq=[20b~1d] ps=[1fe~d]] removing snap head 2019-12-05T06:51:16.598 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:16.589+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5727 (115'2706,123'5727] local-lis/les=21/22 n=793 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=123'5727 lcod 123'5725 mlcod 123'5725 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:16.598 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:16.589+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5829 (115'2807,123'5829] local-lis/les=21/22 n=440 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5829 lcod 123'5827 mlcod 123'5827 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:16.600 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:16.590+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5749 (115'2707,123'5749] local-lis/les=21/22 n=646 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5748 crt=123'5746 lcod 123'5746 mlcod 123'5746 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:16.600 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:16.594+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5751 (115'2707,123'5751] local-lis/les=21/22 n=644 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5748 crt=123'5748 lcod 123'5746 mlcod 123'5746 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:16.602 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:16.593+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5729 (115'2706,123'5729] local-lis/les=21/22 n=791 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5727 crt=123'5727 lcod 123'5725 mlcod 123'5725 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:16.602 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:16.593+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5831 (115'2807,123'5831] local-lis/les=21/22 n=438 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5829 crt=123'5829 lcod 123'5827 mlcod 123'5827 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:16.607 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:16.601+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5732 (115'2706,123'5732] local-lis/les=21/22 n=788 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5731 crt=123'5727 lcod 123'5729 mlcod 123'5729 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:16.609 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:16.601+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5734 (115'2706,123'5734] local-lis/les=21/22 n=786 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5731 crt=123'5731 lcod 123'5729 mlcod 123'5729 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:16.615 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:16.609+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5833 (115'2807,123'5833] local-lis/les=21/22 n=436 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5833 lcod 123'5831 mlcod 123'5831 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:16.615 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:16.610+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5754 (115'2707,123'5754] local-lis/les=21/22 n=641 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5753 crt=123'5751 lcod 123'5751 mlcod 123'5751 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:16.617 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:16.609+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5736 (115'2706,123'5736] local-lis/les=21/22 n=784 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=123'5736 lcod 123'5734 mlcod 123'5734 active+clean+snaptrim trimq=[200~28] ps=[1fe~2]] removing snap head 2019-12-05T06:51:16.617 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:16.609+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5835 (115'2807,123'5835] local-lis/les=21/22 n=434 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5833 crt=123'5833 lcod 123'5831 mlcod 123'5831 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:16.619 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:16.614+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5756 (115'2707,123'5756] local-lis/les=21/22 n=639 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5753 crt=123'5753 lcod 123'5751 mlcod 123'5751 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:16.624 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:16.622+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6355 (117'3311,123'6355] local-lis/les=21/22 n=334 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6354 crt=123'6352 lcod 123'6352 mlcod 123'6352 active+clean+snaptrim trimq=[20b~1d] ps=[1fe~d]] removing snap head 2019-12-05T06:51:16.627 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:16.622+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6357 (117'3311,123'6357] local-lis/les=21/22 n=332 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6354 crt=123'6354 lcod 123'6352 mlcod 123'6352 active+clean+snaptrim trimq=[20b~1d] ps=[1fe~d]] removing snap head 2019-12-05T06:51:16.632 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:16.625+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5838 (115'2807,123'5838] local-lis/les=21/22 n=431 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5837 crt=123'5835 lcod 123'5835 mlcod 123'5835 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:16.634 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:16.625+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5840 (115'2807,123'5840] local-lis/les=21/22 n=429 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5837 crt=123'5837 lcod 123'5835 mlcod 123'5835 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:16.639 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:16.634+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6360 (117'3311,123'6360] local-lis/les=21/22 n=329 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6359 crt=123'6357 lcod 123'6357 mlcod 123'6357 active+clean+snaptrim trimq=[20b~1d] ps=[1fe~d]] removing snap head 2019-12-05T06:51:16.641 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:16.638+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6362 (117'3311,123'6362] local-lis/les=21/22 n=327 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6359 crt=123'6359 lcod 123'6357 mlcod 123'6357 active+clean+snaptrim trimq=[20b~1d] ps=[1fe~d]] removing snap head 2019-12-05T06:51:16.646 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:16.637+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5843 (115'2807,123'5843] local-lis/les=21/22 n=426 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5842 crt=123'5840 lcod 123'5840 mlcod 123'5840 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:16.647 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:16.641+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5845 (115'2807,123'5845] local-lis/les=21/22 n=424 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5842 crt=123'5842 lcod 123'5840 mlcod 123'5840 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:16.657 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:16.654+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5932 (115'2914,123'5932] local-lis/les=21/22 n=732 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'5932 lcod 123'5931 mlcod 123'5931 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:16.658 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:16.654+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5934 (115'2914,123'5934] local-lis/les=21/22 n=730 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'5932 crt=123'5932 lcod 123'5931 mlcod 123'5931 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:16.660 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:16.653+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5739 (115'2706,123'5739] local-lis/les=21/22 n=781 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5738 crt=123'5736 lcod 123'5736 mlcod 123'5736 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:16.662 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:16.657+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5741 (115'2706,123'5741] local-lis/les=21/22 n=779 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5738 crt=123'5738 lcod 123'5736 mlcod 123'5736 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:16.663 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:16.658+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5936 (115'2914,123'5936] local-lis/les=21/22 n=728 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'5936 lcod 123'5934 mlcod 123'5934 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:16.666 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:16.662+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5938 (115'2914,123'5938] local-lis/les=21/22 n=726 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'5936 crt=123'5936 lcod 123'5934 mlcod 123'5934 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:16.670 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:16.666+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5941 (115'2914,123'5941] local-lis/les=21/22 n=723 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'5940 crt=123'5936 lcod 123'5938 mlcod 123'5938 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:16.672 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:16.670+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5943 (115'2914,123'5943] local-lis/les=21/22 n=721 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'5940 crt=123'5940 lcod 123'5938 mlcod 123'5938 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:16.680 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:16.674+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5945 (115'2914,123'5945] local-lis/les=21/22 n=719 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'5945 lcod 123'5943 mlcod 123'5943 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:16.685 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:16.673+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5847 (115'2807,123'5847] local-lis/les=21/22 n=422 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5847 lcod 123'5845 mlcod 123'5845 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:16.687 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:16.674+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5947 (115'2914,123'5947] local-lis/les=21/22 n=717 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'5945 crt=123'5945 lcod 123'5943 mlcod 123'5943 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:16.687 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:16.674+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6365 (117'3311,123'6365] local-lis/les=21/22 n=324 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6364 crt=123'6362 lcod 123'6362 mlcod 123'6362 active+clean+snaptrim trimq=[20b~1d] ps=[1fe~d]] removing snap head 2019-12-05T06:51:16.688 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:16.678+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6367 (117'3311,123'6367] local-lis/les=21/22 n=322 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6364 crt=123'6364 lcod 123'6362 mlcod 123'6362 active+clean+snaptrim trimq=[20b~1d] ps=[1fe~d]] removing snap head 2019-12-05T06:51:16.689 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:16.674+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5804 (115'2705,123'5804] local-lis/les=21/22 n=585 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5804 lcod 123'5802 mlcod 123'5802 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:16.689 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:16.678+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5806 (115'2705,123'5806] local-lis/les=21/22 n=583 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5804 crt=123'5804 lcod 123'5802 mlcod 123'5802 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:16.689 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:16.686+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5949 (115'2914,123'5949] local-lis/les=21/22 n=715 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'5949 lcod 123'5947 mlcod 123'5947 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:16.690 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:16.686+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5951 (115'2914,123'5951] local-lis/les=21/22 n=713 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'5949 crt=123'5949 lcod 123'5947 mlcod 123'5947 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:16.693 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:16.685+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5744 (115'2706,123'5744] local-lis/les=21/22 n=776 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5743 crt=123'5741 lcod 123'5741 mlcod 123'5741 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:16.694 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:16.690+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5808 (115'2806,123'5808] local-lis/les=21/22 n=581 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5808 lcod 123'5806 mlcod 123'5806 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:16.700 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:16.689+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5746 (115'2706,123'5746] local-lis/les=21/22 n=774 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5743 crt=123'5743 lcod 123'5741 mlcod 123'5741 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:16.700 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:16.689+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5849 (115'2807,123'5849] local-lis/les=21/22 n=420 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5849 lcod 123'5847 mlcod 123'5847 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:16.701 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:16.690+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5810 (115'2806,123'5810] local-lis/les=21/22 n=579 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5808 crt=123'5808 lcod 123'5806 mlcod 123'5806 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:16.701 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:16.694+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5953 (115'2914,123'5953] local-lis/les=21/22 n=711 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'5953 lcod 123'5951 mlcod 123'5951 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:16.702 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:16.698+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5955 (115'2914,123'5955] local-lis/les=21/22 n=709 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'5953 crt=123'5953 lcod 123'5951 mlcod 123'5951 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:16.703 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:16.693+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5851 (115'2807,123'5851] local-lis/les=21/22 n=418 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5849 crt=123'5849 lcod 123'5847 mlcod 123'5847 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:16.709 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:16.702+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6370 (117'3311,123'6370] local-lis/les=21/22 n=319 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6369 crt=123'6367 lcod 123'6367 mlcod 123'6367 active+clean+snaptrim trimq=[20b~1d] ps=[1fe~d]] removing snap head 2019-12-05T06:51:16.709 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:16.702+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5812 (115'2806,123'5812] local-lis/les=21/22 n=577 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5812 lcod 123'5810 mlcod 123'5810 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:16.710 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:16.706+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6372 (117'3311,123'6372] local-lis/les=21/22 n=317 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6369 crt=123'6369 lcod 123'6367 mlcod 123'6367 active+clean+snaptrim trimq=[20b~1d] ps=[1fe~d]] removing snap head 2019-12-05T06:51:16.710 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:16.706+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5957 (115'2914,123'5957] local-lis/les=21/22 n=707 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'5957 lcod 123'5955 mlcod 123'5955 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:16.712 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:16.706+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5814 (115'2806,123'5814] local-lis/les=21/22 n=575 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5812 crt=123'5812 lcod 123'5810 mlcod 123'5810 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:16.713 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:16.710+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5959 (115'2914,123'5959] local-lis/les=21/22 n=705 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'5957 crt=123'5957 lcod 123'5955 mlcod 123'5955 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:16.716 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:16.709+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5854 (115'2807,123'5854] local-lis/les=21/22 n=415 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5853 crt=123'5851 lcod 123'5851 mlcod 123'5851 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:16.720 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:16.713+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5856 (115'2807,123'5856] local-lis/les=21/22 n=413 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5853 crt=123'5853 lcod 123'5851 mlcod 123'5851 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:16.720 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:16.714+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5961 (115'2914,123'5961] local-lis/les=21/22 n=703 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'5961 lcod 123'5959 mlcod 123'5959 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:16.721 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:16.718+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5816 (115'2806,123'5816] local-lis/les=21/22 n=573 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5816 lcod 123'5814 mlcod 123'5814 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:16.721 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:16.718+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5963 (115'2914,123'5963] local-lis/les=21/22 n=701 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'5961 crt=123'5961 lcod 123'5959 mlcod 123'5959 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:16.724 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:16.722+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5818 (115'2806,123'5818] local-lis/les=21/22 n=571 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5816 crt=123'5816 lcod 123'5814 mlcod 123'5814 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:16.726 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:16.722+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5965 (115'2914,123'5965] local-lis/les=21/22 n=699 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'5965 lcod 123'5963 mlcod 123'5963 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:16.728 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:16.726+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5967 (115'2914,123'5967] local-lis/les=21/22 n=697 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'5965 crt=123'5965 lcod 123'5963 mlcod 123'5963 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:16.732 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:16.730+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5969 (115'2914,123'5969] local-lis/les=21/22 n=695 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'5969 lcod 123'5967 mlcod 123'5967 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:16.734 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:16.730+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5971 (115'2914,123'5971] local-lis/les=21/22 n=693 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'5969 crt=123'5969 lcod 123'5967 mlcod 123'5967 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:16.740 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:16.738+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5975 (115'2914,123'5975] local-lis/les=21/22 n=690 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'5973 crt=123'5973 lcod 123'5971 mlcod 123'5971 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:16.744 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:16.742+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5977 (115'2914,123'5977] local-lis/les=21/22 n=688 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'5977 lcod 123'5975 mlcod 123'5975 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:16.746 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:16.742+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5979 (115'2914,123'5979] local-lis/les=21/22 n=686 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'5977 crt=123'5977 lcod 123'5975 mlcod 123'5975 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:16.755 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:16.746+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5981 (115'2914,123'5981] local-lis/les=21/22 n=684 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'5981 lcod 123'5979 mlcod 123'5979 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:16.756 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:16.754+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5983 (115'2914,123'5983] local-lis/les=21/22 n=682 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'5983 lcod 123'5981 mlcod 123'5981 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:16.758 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:16.754+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5985 (115'2914,123'5985] local-lis/les=21/22 n=680 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'5983 crt=123'5983 lcod 123'5981 mlcod 123'5981 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:16.763 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:16.758+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5987 (115'2914,123'5987] local-lis/les=21/22 n=678 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'5987 lcod 123'5985 mlcod 123'5985 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:16.764 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:16.762+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5989 (115'2914,123'5989] local-lis/les=21/22 n=676 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'5987 crt=123'5987 lcod 123'5985 mlcod 123'5985 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:16.766 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:16.762+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5758 (115'2707,123'5758] local-lis/les=21/22 n=637 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5758 lcod 123'5756 mlcod 123'5756 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:16.767 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:16.762+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5760 (115'2707,123'5760] local-lis/les=21/22 n=635 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5758 crt=123'5758 lcod 123'5756 mlcod 123'5756 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:16.769 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:16.766+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5820 (115'2806,123'5820] local-lis/les=21/22 n=569 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5820 lcod 123'5818 mlcod 123'5818 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:16.770 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:16.766+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5822 (115'2806,123'5822] local-lis/les=21/22 n=567 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5820 crt=123'5820 lcod 123'5818 mlcod 123'5818 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:16.774 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:16.770+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5762 (115'2707,123'5762] local-lis/les=21/22 n=633 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5762 lcod 123'5760 mlcod 123'5760 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:16.785 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:16.774+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5764 (115'2707,123'5764] local-lis/les=21/22 n=631 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5762 crt=123'5762 lcod 123'5760 mlcod 123'5760 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:16.785 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:16.774+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5824 (115'2806,123'5824] local-lis/les=21/22 n=565 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5824 lcod 123'5822 mlcod 123'5822 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:16.786 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:16.774+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5826 (115'2806,123'5826] local-lis/les=21/22 n=563 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5824 crt=123'5824 lcod 123'5822 mlcod 123'5822 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:16.786 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:16.778+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5766 (115'2707,123'5766] local-lis/les=21/22 n=629 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5766 lcod 123'5764 mlcod 123'5764 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:16.786 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:16.778+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5768 (115'2707,123'5768] local-lis/les=21/22 n=627 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5766 crt=123'5766 lcod 123'5764 mlcod 123'5764 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:16.805 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:16.802+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6184 (117'3108,123'6184] local-lis/les=21/22 n=281 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'6184 lcod 123'6182 mlcod 123'6182 active+clean+snaptrim trimq=[20e~1a] ps=[1fe~10]] removing snap head 2019-12-05T06:51:16.805 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:16.802+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5880 (115'2804,123'5880] local-lis/les=21/22 n=339 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5880 lcod 123'5878 mlcod 123'5878 active+clean+snaptrim trimq=[20b~1d] ps=[1fe~d]] removing snap head 2019-12-05T06:51:16.808 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:16.802+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6186 (117'3108,123'6186] local-lis/les=21/22 n=279 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'6184 crt=123'6184 lcod 123'6182 mlcod 123'6182 active+clean+snaptrim trimq=[20e~1a] ps=[1fe~10]] removing snap head 2019-12-05T06:51:16.812 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:16.802+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5882 (115'2804,123'5882] local-lis/les=21/22 n=337 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5880 crt=123'5880 lcod 123'5878 mlcod 123'5878 active+clean+snaptrim trimq=[20b~1d] ps=[1fe~d]] removing snap head 2019-12-05T06:51:16.812 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:16.806+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5991 (115'2914,123'5991] local-lis/les=21/22 n=674 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'5991 lcod 123'5989 mlcod 123'5989 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:16.813 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:16.806+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5993 (115'2914,123'5993] local-lis/les=21/22 n=672 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'5991 crt=123'5991 lcod 123'5989 mlcod 123'5989 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:16.816 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:16.814+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6188 (117'3108,123'6188] local-lis/les=21/22 n=277 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'6188 lcod 123'6186 mlcod 123'6186 active+clean+snaptrim trimq=[20e~1a] ps=[1fe~10]] removing snap head 2019-12-05T06:51:16.816 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:16.814+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5884 (115'2804,123'5884] local-lis/les=21/22 n=335 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5884 lcod 123'5882 mlcod 123'5882 active+clean+snaptrim trimq=[20b~1d] ps=[1fe~d]] removing snap head 2019-12-05T06:51:16.818 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:16.814+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6190 (117'3108,123'6190] local-lis/les=21/22 n=275 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'6188 crt=123'6188 lcod 123'6186 mlcod 123'6186 active+clean+snaptrim trimq=[20e~1a] ps=[1fe~10]] removing snap head 2019-12-05T06:51:16.818 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:16.814+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5886 (115'2804,123'5886] local-lis/les=21/22 n=333 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5884 crt=123'5884 lcod 123'5882 mlcod 123'5882 active+clean+snaptrim trimq=[20b~1d] ps=[1fe~d]] removing snap head 2019-12-05T06:51:16.822 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:16.818+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5995 (115'2914,123'5995] local-lis/les=21/22 n=670 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'5995 lcod 123'5993 mlcod 123'5993 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:16.825 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:16.822+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5997 (115'2914,123'5997] local-lis/les=21/22 n=668 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'5995 crt=123'5995 lcod 123'5993 mlcod 123'5993 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:16.828 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:16.826+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'5999 (115'2914,123'5999] local-lis/les=21/22 n=666 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'5999 lcod 123'5997 mlcod 123'5997 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:16.830 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:16.826+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6001 (115'2914,123'6001] local-lis/les=21/22 n=664 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'5999 crt=123'5999 lcod 123'5997 mlcod 123'5997 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:16.833 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:16.830+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6003 (115'2914,123'6003] local-lis/les=21/22 n=662 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'6003 lcod 123'6001 mlcod 123'6001 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:16.834 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:16.830+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6005 (115'2914,123'6005] local-lis/les=21/22 n=660 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'6003 crt=123'6003 lcod 123'6001 mlcod 123'6001 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:16.838 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:16.834+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6007 (115'2914,123'6007] local-lis/les=21/22 n=658 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'6007 lcod 123'6005 mlcod 123'6005 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:16.839 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:16.834+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6009 (115'2914,123'6009] local-lis/les=21/22 n=656 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'6007 crt=123'6007 lcod 123'6005 mlcod 123'6005 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:16.842 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:16.838+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6011 (115'2914,123'6011] local-lis/les=21/22 n=654 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'6011 lcod 123'6009 mlcod 123'6009 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:16.844 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:16.842+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6013 (115'2914,123'6013] local-lis/les=21/22 n=652 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'6011 crt=123'6011 lcod 123'6009 mlcod 123'6009 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:16.985 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:16.966+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6374 (117'3311,123'6374] local-lis/les=21/22 n=315 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'6374 lcod 123'6372 mlcod 123'6372 active+clean+snaptrim trimq=[20b~1d] ps=[1fe~d]] removing snap head 2019-12-05T06:51:16.985 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:16.970+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6376 (117'3311,123'6376] local-lis/les=21/22 n=313 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6374 crt=123'6374 lcod 123'6372 mlcod 123'6372 active+clean+snaptrim trimq=[20b~1d] ps=[1fe~d]] removing snap head 2019-12-05T06:51:16.985 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:16.970+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6192 (117'3108,123'6192] local-lis/les=21/22 n=273 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'6192 lcod 123'6190 mlcod 123'6190 active+clean+snaptrim trimq=[20e~1a] ps=[1fe~10]] removing snap head 2019-12-05T06:51:16.986 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:16.970+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5888 (115'2804,123'5888] local-lis/les=21/22 n=331 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5888 lcod 123'5886 mlcod 123'5886 active+clean+snaptrim trimq=[20b~1d] ps=[1fe~d]] removing snap head 2019-12-05T06:51:16.987 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:16.974+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5890 (115'2804,123'5890] local-lis/les=21/22 n=329 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5888 crt=123'5888 lcod 123'5886 mlcod 123'5886 active+clean+snaptrim trimq=[20b~1d] ps=[1fe~d]] removing snap head 2019-12-05T06:51:16.987 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:16.978+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6015 (115'2914,123'6015] local-lis/les=21/22 n=650 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'6015 lcod 123'6013 mlcod 123'6013 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:16.987 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:16.978+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6194 (117'3108,123'6194] local-lis/les=21/22 n=271 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'6192 crt=123'6192 lcod 123'6190 mlcod 123'6190 active+clean+snaptrim trimq=[20e~1a] ps=[1fe~10]] removing snap head 2019-12-05T06:51:16.987 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:16.982+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6378 (117'3311,123'6378] local-lis/les=21/22 n=311 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'6378 lcod 123'6376 mlcod 123'6376 active+clean+snaptrim trimq=[20b~1d] ps=[1fe~d]] removing snap head 2019-12-05T06:51:16.989 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:16.986+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6380 (117'3311,123'6380] local-lis/les=21/22 n=309 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6378 crt=123'6378 lcod 123'6376 mlcod 123'6376 active+clean+snaptrim trimq=[20b~1d] ps=[1fe~d]] removing snap head 2019-12-05T06:51:16.993 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:16.986+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6196 (117'3108,123'6196] local-lis/les=21/22 n=269 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'6196 lcod 123'6194 mlcod 123'6194 active+clean+snaptrim trimq=[20e~1a] ps=[1fe~10]] removing snap head 2019-12-05T06:51:16.993 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:16.986+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6017 (116'3015,123'6017] local-lis/les=21/22 n=648 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'6015 crt=123'6015 lcod 123'6013 mlcod 123'6013 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:16.995 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:16.990+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6198 (117'3108,123'6198] local-lis/les=21/22 n=267 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'6196 crt=123'6196 lcod 123'6194 mlcod 123'6194 active+clean+snaptrim trimq=[20e~1a] ps=[1fe~10]] removing snap head 2019-12-05T06:51:16.995 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:16.990+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5892 (115'2804,123'5892] local-lis/les=21/22 n=327 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5892 lcod 123'5890 mlcod 123'5890 active+clean+snaptrim trimq=[20b~1d] ps=[1fe~d]] removing snap head 2019-12-05T06:51:16.997 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:16.994+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5894 (115'2804,123'5894] local-lis/les=21/22 n=325 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5892 crt=123'5892 lcod 123'5890 mlcod 123'5890 active+clean+snaptrim trimq=[20b~1d] ps=[1fe~d]] removing snap head 2019-12-05T06:51:16.999 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:16.994+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6382 (117'3311,123'6382] local-lis/les=21/22 n=307 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'6382 lcod 123'6380 mlcod 123'6380 active+clean+snaptrim trimq=[20c~1c] ps=[1fe~e]] removing snap head 2019-12-05T06:51:17.002 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:16.998+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6384 (117'3311,123'6384] local-lis/les=21/22 n=305 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6382 crt=123'6382 lcod 123'6380 mlcod 123'6380 active+clean+snaptrim trimq=[20c~1c] ps=[1fe~e]] removing snap head 2019-12-05T06:51:17.002 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:16.998+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6019 (116'3015,123'6019] local-lis/les=21/22 n=646 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'6019 lcod 123'6017 mlcod 123'6017 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:17.007 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:17.002+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6200 (117'3108,123'6200] local-lis/les=21/22 n=265 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'6200 lcod 123'6198 mlcod 123'6198 active+clean+snaptrim trimq=[20e~1a] ps=[1fe~10]] removing snap head 2019-12-05T06:51:17.007 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.002+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6021 (116'3015,123'6021] local-lis/les=21/22 n=644 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'6019 crt=123'6019 lcod 123'6017 mlcod 123'6017 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:17.008 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.002+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5896 (115'2804,123'5896] local-lis/les=21/22 n=323 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5896 lcod 123'5894 mlcod 123'5894 active+clean+snaptrim trimq=[20b~1d] ps=[1fe~d]] removing snap head 2019-12-05T06:51:17.009 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:17.006+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6202 (117'3108,123'6202] local-lis/les=21/22 n=263 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'6200 crt=123'6200 lcod 123'6198 mlcod 123'6198 active+clean+snaptrim trimq=[20e~1a] ps=[1fe~10]] removing snap head 2019-12-05T06:51:17.010 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.006+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5898 (115'2804,123'5898] local-lis/les=21/22 n=321 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5896 crt=123'5896 lcod 123'5894 mlcod 123'5894 active+clean+snaptrim trimq=[20b~1d] ps=[1fe~d]] removing snap head 2019-12-05T06:51:17.013 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:17.010+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6386 (117'3311,123'6386] local-lis/les=21/22 n=303 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'6386 lcod 123'6384 mlcod 123'6384 active+clean+snaptrim trimq=[20c~1c] ps=[1fe~e]] removing snap head 2019-12-05T06:51:17.015 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:17.010+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6388 (117'3311,123'6388] local-lis/les=21/22 n=301 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6386 crt=123'6386 lcod 123'6384 mlcod 123'6384 active+clean+snaptrim trimq=[20c~1c] ps=[1fe~e]] removing snap head 2019-12-05T06:51:17.016 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.014+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6023 (116'3015,123'6023] local-lis/les=21/22 n=642 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'6023 lcod 123'6021 mlcod 123'6021 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:17.019 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.014+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5900 (115'2804,123'5900] local-lis/les=21/22 n=319 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5900 lcod 123'5898 mlcod 123'5898 active+clean+snaptrim trimq=[20b~1d] ps=[1fe~d]] removing snap head 2019-12-05T06:51:17.020 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:17.014+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6204 (117'3108,123'6204] local-lis/les=21/22 n=261 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'6204 lcod 123'6202 mlcod 123'6202 active+clean+snaptrim trimq=[20e~1a] ps=[1fe~10]] removing snap head 2019-12-05T06:51:17.021 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:17.018+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6206 (117'3108,123'6206] local-lis/les=21/22 n=259 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'6204 crt=123'6204 lcod 123'6202 mlcod 123'6202 active+clean+snaptrim trimq=[20e~1a] ps=[1fe~10]] removing snap head 2019-12-05T06:51:17.022 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.018+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5902 (115'2804,123'5902] local-lis/les=21/22 n=317 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5900 crt=123'5900 lcod 123'5898 mlcod 123'5898 active+clean+snaptrim trimq=[20b~1d] ps=[1fe~d]] removing snap head 2019-12-05T06:51:17.024 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.022+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6025 (116'3015,123'6025] local-lis/les=21/22 n=640 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'6023 crt=123'6023 lcod 123'6021 mlcod 123'6021 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:17.026 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:17.022+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6390 (117'3311,123'6390] local-lis/les=21/22 n=299 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'6390 lcod 123'6388 mlcod 123'6388 active+clean+snaptrim trimq=[20c~1c] ps=[1fe~e]] removing snap head 2019-12-05T06:51:17.028 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:17.026+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6392 (117'3311,123'6392] local-lis/les=21/22 n=297 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6390 crt=123'6390 lcod 123'6388 mlcod 123'6388 active+clean+snaptrim trimq=[20c~1c] ps=[1fe~e]] removing snap head 2019-12-05T06:51:17.032 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.026+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5904 (115'2804,123'5904] local-lis/les=21/22 n=315 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5904 lcod 123'5902 mlcod 123'5902 active+clean+snaptrim trimq=[20b~1d] ps=[1fe~d]] removing snap head 2019-12-05T06:51:17.032 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:17.026+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6208 (117'3108,123'6208] local-lis/les=21/22 n=257 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'6208 lcod 123'6206 mlcod 123'6206 active+clean+snaptrim trimq=[20e~1a] ps=[1fe~10]] removing snap head 2019-12-05T06:51:17.033 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.030+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6027 (116'3015,123'6027] local-lis/les=21/22 n=638 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'6027 lcod 123'6025 mlcod 123'6025 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:17.035 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.030+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6029 (116'3015,123'6029] local-lis/les=21/22 n=636 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'6027 crt=123'6027 lcod 123'6025 mlcod 123'6025 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:17.036 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:17.034+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6210 (117'3208,123'6210] local-lis/les=21/22 n=255 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'6208 crt=123'6208 lcod 123'6206 mlcod 123'6206 active+clean+snaptrim trimq=[20e~1a] ps=[1fe~10]] removing snap head 2019-12-05T06:51:17.039 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:17.034+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6394 (117'3311,123'6394] local-lis/les=21/22 n=295 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'6394 lcod 123'6392 mlcod 123'6392 active+clean+snaptrim trimq=[20c~1c] ps=[1fe~e]] removing snap head 2019-12-05T06:51:17.040 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.034+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5906 (115'2904,123'5906] local-lis/les=21/22 n=313 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5904 crt=123'5904 lcod 123'5902 mlcod 123'5902 active+clean+snaptrim trimq=[20b~1d] ps=[1fe~d]] removing snap head 2019-12-05T06:51:17.045 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:17.042+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6396 (117'3311,123'6396] local-lis/les=21/22 n=293 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6394 crt=123'6394 lcod 123'6392 mlcod 123'6392 active+clean+snaptrim trimq=[20c~1c] ps=[1fe~e]] removing snap head 2019-12-05T06:51:17.046 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.038+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6031 (116'3015,123'6031] local-lis/les=21/22 n=634 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'6031 lcod 123'6029 mlcod 123'6029 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:17.047 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.042+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6033 (116'3015,123'6033] local-lis/les=21/22 n=632 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'6031 crt=123'6031 lcod 123'6029 mlcod 123'6029 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:17.049 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:17.046+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6212 (117'3208,123'6212] local-lis/les=21/22 n=253 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'6212 lcod 123'6210 mlcod 123'6210 active+clean+snaptrim trimq=[20e~1a] ps=[1fe~10]] removing snap head 2019-12-05T06:51:17.051 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:17.046+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6214 (117'3208,123'6214] local-lis/les=21/22 n=251 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'6212 crt=123'6212 lcod 123'6210 mlcod 123'6210 active+clean+snaptrim trimq=[20e~1a] ps=[1fe~10]] removing snap head 2019-12-05T06:51:17.052 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.046+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5908 (115'2904,123'5908] local-lis/les=21/22 n=311 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5908 lcod 123'5906 mlcod 123'5906 active+clean+snaptrim trimq=[20b~1d] ps=[1fe~d]] removing snap head 2019-12-05T06:51:17.052 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:17.050+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6398 (117'3311,123'6398] local-lis/les=21/22 n=291 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'6398 lcod 123'6396 mlcod 123'6396 active+clean+snaptrim trimq=[20c~1c] ps=[1fe~e]] removing snap head 2019-12-05T06:51:17.056 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:17.054+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6400 (117'3311,123'6400] local-lis/les=21/22 n=289 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6398 crt=123'6398 lcod 123'6396 mlcod 123'6396 active+clean+snaptrim trimq=[20c~1c] ps=[1fe~e]] removing snap head 2019-12-05T06:51:17.057 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.054+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5910 (115'2904,123'5910] local-lis/les=21/22 n=309 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5908 crt=123'5908 lcod 123'5906 mlcod 123'5906 active+clean+snaptrim trimq=[20b~1d] ps=[1fe~d]] removing snap head 2019-12-05T06:51:17.060 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.058+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6037 (116'3015,123'6037] local-lis/les=21/22 n=629 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'6035 crt=123'6035 lcod 123'6033 mlcod 123'6033 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:17.061 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:17.058+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6216 (117'3208,123'6216] local-lis/les=21/22 n=249 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'6216 lcod 123'6214 mlcod 123'6214 active+clean+snaptrim trimq=[20e~1a] ps=[1fe~10]] removing snap head 2019-12-05T06:51:17.065 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.062+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5912 (115'2904,123'5912] local-lis/les=21/22 n=307 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5912 lcod 123'5910 mlcod 123'5910 active+clean+snaptrim trimq=[20b~1d] ps=[1fe~d]] removing snap head 2019-12-05T06:51:17.066 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:17.062+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6402 (117'3311,123'6402] local-lis/les=21/22 n=287 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'6402 lcod 123'6400 mlcod 123'6400 active+clean+snaptrim trimq=[20c~1c] ps=[1fe~e]] removing snap head 2019-12-05T06:51:17.067 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.062+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5914 (115'2904,123'5914] local-lis/les=21/22 n=305 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5912 crt=123'5912 lcod 123'5910 mlcod 123'5910 active+clean+snaptrim trimq=[20b~1d] ps=[1fe~d]] removing snap head 2019-12-05T06:51:17.070 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:17.066+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6404 (117'3311,123'6404] local-lis/les=21/22 n=285 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6402 crt=123'6402 lcod 123'6400 mlcod 123'6400 active+clean+snaptrim trimq=[20c~1c] ps=[1fe~e]] removing snap head 2019-12-05T06:51:17.074 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.066+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6039 (116'3015,123'6039] local-lis/les=21/22 n=627 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'6039 lcod 123'6037 mlcod 123'6037 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:17.075 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.070+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5916 (115'2904,123'5916] local-lis/les=21/22 n=303 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5916 lcod 123'5914 mlcod 123'5914 active+clean+snaptrim trimq=[20b~1d] ps=[1fe~d]] removing snap head 2019-12-05T06:51:17.077 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:17.074+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6218 (117'3208,123'6218] local-lis/les=21/22 n=247 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'6218 lcod 123'6216 mlcod 123'6216 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:17.079 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:17.074+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6220 (117'3208,123'6220] local-lis/les=21/22 n=245 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'6218 crt=123'6218 lcod 123'6216 mlcod 123'6216 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:17.080 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.078+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5918 (115'2904,123'5918] local-lis/les=21/22 n=301 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5918 lcod 123'5916 mlcod 123'5916 active+clean+snaptrim trimq=[20c~1c] ps=[1fe~e]] removing snap head 2019-12-05T06:51:17.080 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.078+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6041 (116'3015,123'6041] local-lis/les=21/22 n=625 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'6041 lcod 123'6039 mlcod 123'6039 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:17.083 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.078+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5920 (115'2904,123'5920] local-lis/les=21/22 n=299 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5918 crt=123'5918 lcod 123'5916 mlcod 123'5916 active+clean+snaptrim trimq=[20c~1c] ps=[1fe~e]] removing snap head 2019-12-05T06:51:17.087 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.082+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6043 (116'3015,123'6043] local-lis/les=21/22 n=623 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'6041 crt=123'6041 lcod 123'6039 mlcod 123'6039 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:17.087 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:17.082+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6406 (117'3311,123'6406] local-lis/les=21/22 n=283 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'6406 lcod 123'6404 mlcod 123'6404 active+clean+snaptrim trimq=[20c~1c] ps=[1fe~e]] removing snap head 2019-12-05T06:51:17.088 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:17.082+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6222 (117'3208,123'6222] local-lis/les=21/22 n=243 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'6222 lcod 123'6220 mlcod 123'6220 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:17.089 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:17.086+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6408 (117'3311,123'6408] local-lis/les=21/22 n=281 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6406 crt=123'6406 lcod 123'6404 mlcod 123'6404 active+clean+snaptrim trimq=[20c~1c] ps=[1fe~e]] removing snap head 2019-12-05T06:51:17.090 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.086+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5922 (115'2904,123'5922] local-lis/les=21/22 n=297 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5922 lcod 123'5920 mlcod 123'5920 active+clean+snaptrim trimq=[20c~1c] ps=[1fe~e]] removing snap head 2019-12-05T06:51:17.090 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.086+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6045 (116'3015,123'6045] local-lis/les=21/22 n=621 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'6045 lcod 123'6043 mlcod 123'6043 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:17.093 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.090+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6047 (116'3015,123'6047] local-lis/les=21/22 n=619 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'6045 crt=123'6045 lcod 123'6043 mlcod 123'6043 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:17.094 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:17.090+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6224 (117'3208,123'6224] local-lis/les=21/22 n=241 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'6222 crt=123'6222 lcod 123'6220 mlcod 123'6220 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:17.094 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.090+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5924 (115'2904,123'5924] local-lis/les=21/22 n=295 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5922 crt=123'5922 lcod 123'5920 mlcod 123'5920 active+clean+snaptrim trimq=[20c~1c] ps=[1fe~e]] removing snap head 2019-12-05T06:51:17.098 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.094+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6049 (116'3015,123'6049] local-lis/les=21/22 n=617 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'6049 lcod 123'6047 mlcod 123'6047 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:17.102 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:17.098+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6226 (117'3208,123'6226] local-lis/les=21/22 n=239 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'6226 lcod 123'6224 mlcod 123'6224 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:17.102 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:17.098+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6410 (117'3311,123'6410] local-lis/les=21/22 n=279 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'6410 lcod 123'6408 mlcod 123'6408 active+clean+snaptrim trimq=[20c~1c] ps=[1fe~e]] removing snap head 2019-12-05T06:51:17.103 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.098+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6051 (116'3015,123'6051] local-lis/les=21/22 n=615 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'6049 crt=123'6049 lcod 123'6047 mlcod 123'6047 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:17.106 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.102+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5770 (115'2707,123'5770] local-lis/les=21/22 n=625 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5770 lcod 123'5768 mlcod 123'5768 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:17.106 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:17.102+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6228 (117'3208,123'6228] local-lis/les=21/22 n=237 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'6226 crt=123'6226 lcod 123'6224 mlcod 123'6224 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:17.106 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:17.102+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6412 (117'3311,123'6412] local-lis/les=21/22 n=277 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6410 crt=123'6410 lcod 123'6408 mlcod 123'6408 active+clean+snaptrim trimq=[20c~1c] ps=[1fe~e]] removing snap head 2019-12-05T06:51:17.106 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.102+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5772 (115'2707,123'5772] local-lis/les=21/22 n=623 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5770 crt=123'5770 lcod 123'5768 mlcod 123'5768 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:17.111 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.106+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5926 (115'2904,123'5926] local-lis/les=21/22 n=293 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5926 lcod 123'5924 mlcod 123'5924 active+clean+snaptrim trimq=[20c~1c] ps=[1fe~e]] removing snap head 2019-12-05T06:51:17.113 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.110+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5928 (115'2904,123'5928] local-lis/les=21/22 n=291 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5926 crt=123'5926 lcod 123'5924 mlcod 123'5924 active+clean+snaptrim trimq=[20c~1c] ps=[1fe~e]] removing snap head 2019-12-05T06:51:17.115 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:17.110+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6230 (117'3208,123'6230] local-lis/les=21/22 n=235 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'6230 lcod 123'6228 mlcod 123'6228 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:17.116 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:17.110+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6232 (117'3208,123'6232] local-lis/les=21/22 n=233 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'6230 crt=123'6230 lcod 123'6228 mlcod 123'6228 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:17.117 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.110+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6053 (116'3015,123'6053] local-lis/les=21/22 n=613 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'6053 lcod 123'6051 mlcod 123'6051 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:17.119 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.114+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6055 (116'3015,123'6055] local-lis/les=21/22 n=611 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'6053 crt=123'6053 lcod 123'6051 mlcod 123'6051 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:17.120 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.118+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5930 (115'2904,123'5930] local-lis/les=21/22 n=289 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5930 lcod 123'5928 mlcod 123'5928 active+clean+snaptrim trimq=[20c~1c] ps=[1fe~e]] removing snap head 2019-12-05T06:51:17.122 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.118+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5932 (115'2904,123'5932] local-lis/les=21/22 n=287 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5930 crt=123'5930 lcod 123'5928 mlcod 123'5928 active+clean+snaptrim trimq=[20c~1c] ps=[1fe~e]] removing snap head 2019-12-05T06:51:17.125 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.122+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6057 (116'3015,123'6057] local-lis/les=21/22 n=609 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'6057 lcod 123'6055 mlcod 123'6055 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:17.126 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.122+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5934 (115'2904,123'5934] local-lis/les=21/22 n=285 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5934 lcod 123'5932 mlcod 123'5932 active+clean+snaptrim trimq=[20c~1c] ps=[1fe~e]] removing snap head 2019-12-05T06:51:17.128 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.122+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6059 (116'3015,123'6059] local-lis/les=21/22 n=607 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'6057 crt=123'6057 lcod 123'6055 mlcod 123'6055 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:17.128 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.126+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5936 (115'2904,123'5936] local-lis/les=21/22 n=283 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5934 crt=123'5934 lcod 123'5932 mlcod 123'5932 active+clean+snaptrim trimq=[20c~1c] ps=[1fe~e]] removing snap head 2019-12-05T06:51:17.132 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.130+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6061 (116'3015,123'6061] local-lis/les=21/22 n=605 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'6061 lcod 123'6059 mlcod 123'6059 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:17.133 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.130+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5938 (115'2904,123'5938] local-lis/les=21/22 n=281 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5938 lcod 123'5936 mlcod 123'5936 active+clean+snaptrim trimq=[20c~1c] ps=[1fe~e]] removing snap head 2019-12-05T06:51:17.134 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.130+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6063 (116'3015,123'6063] local-lis/les=21/22 n=603 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'6061 crt=123'6061 lcod 123'6059 mlcod 123'6059 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:17.135 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.130+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5940 (115'2904,123'5940] local-lis/les=21/22 n=279 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5938 crt=123'5938 lcod 123'5936 mlcod 123'5936 active+clean+snaptrim trimq=[20c~1c] ps=[1fe~e]] removing snap head 2019-12-05T06:51:17.138 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:17.129+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5858 (115'2807,123'5858] local-lis/les=21/22 n=411 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5858 lcod 123'5856 mlcod 123'5856 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:17.138 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:17.133+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5860 (115'2807,123'5860] local-lis/les=21/22 n=409 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5858 crt=123'5858 lcod 123'5856 mlcod 123'5856 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:17.141 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.138+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5942 (115'2904,123'5942] local-lis/les=21/22 n=277 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5942 lcod 123'5940 mlcod 123'5940 active+clean+snaptrim trimq=[20c~1c] ps=[1fe~e]] removing snap head 2019-12-05T06:51:17.143 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.138+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6065 (116'3015,123'6065] local-lis/les=21/22 n=601 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'6065 lcod 123'6063 mlcod 123'6063 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:17.143 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:17.137+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5862 (115'2807,123'5862] local-lis/les=21/22 n=407 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5862 lcod 123'5860 mlcod 123'5860 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:17.144 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.142+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5944 (115'2904,123'5944] local-lis/les=21/22 n=275 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5942 crt=123'5942 lcod 123'5940 mlcod 123'5940 active+clean+snaptrim trimq=[20c~1c] ps=[1fe~e]] removing snap head 2019-12-05T06:51:17.144 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.142+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6067 (116'3015,123'6067] local-lis/les=21/22 n=599 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'6065 crt=123'6065 lcod 123'6063 mlcod 123'6063 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:17.145 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:17.137+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5864 (115'2807,123'5864] local-lis/les=21/22 n=405 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5862 crt=123'5862 lcod 123'5860 mlcod 123'5860 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:17.150 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.146+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5946 (115'2904,123'5946] local-lis/les=21/22 n=273 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5946 lcod 123'5944 mlcod 123'5944 active+clean+snaptrim trimq=[20c~1c] ps=[1fe~e]] removing snap head 2019-12-05T06:51:17.152 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.150+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5948 (115'2904,123'5948] local-lis/les=21/22 n=271 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5946 crt=123'5946 lcod 123'5944 mlcod 123'5944 active+clean+snaptrim trimq=[20c~1c] ps=[1fe~e]] removing snap head 2019-12-05T06:51:17.152 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.150+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6069 (116'3015,123'6069] local-lis/les=21/22 n=597 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'6069 lcod 123'6067 mlcod 123'6067 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:17.154 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.150+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6071 (116'3015,123'6071] local-lis/les=21/22 n=595 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'6069 crt=123'6069 lcod 123'6067 mlcod 123'6067 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:17.159 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.154+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5950 (115'2904,123'5950] local-lis/les=21/22 n=269 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5950 lcod 123'5948 mlcod 123'5948 active+clean+snaptrim trimq=[20c~1c] ps=[1fe~e]] removing snap head 2019-12-05T06:51:17.159 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.154+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6073 (116'3015,123'6073] local-lis/les=21/22 n=593 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'6073 lcod 123'6071 mlcod 123'6071 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:17.160 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.158+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6075 (116'3015,123'6075] local-lis/les=21/22 n=591 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'6073 crt=123'6073 lcod 123'6071 mlcod 123'6071 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:17.164 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.158+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5952 (115'2904,123'5952] local-lis/les=21/22 n=267 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5952 lcod 123'5950 mlcod 123'5950 active+clean+snaptrim trimq=[20d~1b] ps=[1fe~f]] removing snap head 2019-12-05T06:51:17.165 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.162+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5954 (115'2904,123'5954] local-lis/les=21/22 n=265 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5952 crt=123'5952 lcod 123'5950 mlcod 123'5950 active+clean+snaptrim trimq=[20d~1b] ps=[1fe~f]] removing snap head 2019-12-05T06:51:17.168 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.166+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6077 (116'3015,123'6077] local-lis/les=21/22 n=589 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'6077 lcod 123'6075 mlcod 123'6075 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:17.173 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.170+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5956 (115'2904,123'5956] local-lis/les=21/22 n=263 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5956 lcod 123'5954 mlcod 123'5954 active+clean+snaptrim trimq=[20d~1b] ps=[1fe~f]] removing snap head 2019-12-05T06:51:17.175 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.170+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5958 (115'2904,123'5958] local-lis/les=21/22 n=261 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5956 crt=123'5956 lcod 123'5954 mlcod 123'5954 active+clean+snaptrim trimq=[20d~1b] ps=[1fe~f]] removing snap head 2019-12-05T06:51:17.177 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.174+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6081 (116'3015,123'6081] local-lis/les=21/22 n=586 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'6081 lcod 123'6079 mlcod 123'6079 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:17.179 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.174+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6083 (116'3015,123'6083] local-lis/les=21/22 n=584 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'6081 crt=123'6081 lcod 123'6079 mlcod 123'6079 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:17.181 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.178+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5960 (115'2904,123'5960] local-lis/les=21/22 n=259 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5960 lcod 123'5958 mlcod 123'5958 active+clean+snaptrim trimq=[20d~1b] ps=[1fe~f]] removing snap head 2019-12-05T06:51:17.183 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.178+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5962 (115'2904,123'5962] local-lis/les=21/22 n=257 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5960 crt=123'5960 lcod 123'5958 mlcod 123'5958 active+clean+snaptrim trimq=[20d~1b] ps=[1fe~f]] removing snap head 2019-12-05T06:51:17.187 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.182+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6085 (116'3015,123'6085] local-lis/les=21/22 n=582 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'6085 lcod 123'6083 mlcod 123'6083 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:17.188 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.186+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6087 (116'3015,123'6087] local-lis/les=21/22 n=580 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'6085 crt=123'6085 lcod 123'6083 mlcod 123'6083 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:17.191 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.186+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5964 (115'2904,123'5964] local-lis/les=21/22 n=255 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5964 lcod 123'5962 mlcod 123'5962 active+clean+snaptrim trimq=[20d~1b] ps=[1fe~f]] removing snap head 2019-12-05T06:51:17.196 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.190+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5966 (115'2904,123'5966] local-lis/les=21/22 n=253 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5964 crt=123'5964 lcod 123'5962 mlcod 123'5962 active+clean+snaptrim trimq=[20d~1b] ps=[1fe~f]] removing snap head 2019-12-05T06:51:17.196 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.190+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6089 (116'3015,123'6089] local-lis/les=21/22 n=578 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'6089 lcod 123'6087 mlcod 123'6087 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:17.199 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.194+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5968 (115'2904,123'5968] local-lis/les=21/22 n=251 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5968 lcod 123'5966 mlcod 123'5966 active+clean+snaptrim trimq=[20d~1b] ps=[1fe~f]] removing snap head 2019-12-05T06:51:17.201 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.198+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5970 (115'2904,123'5970] local-lis/les=21/22 n=249 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5968 crt=123'5968 lcod 123'5966 mlcod 123'5966 active+clean+snaptrim trimq=[20d~1b] ps=[1fe~f]] removing snap head 2019-12-05T06:51:17.204 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.202+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6091 (116'3015,123'6091] local-lis/les=21/22 n=576 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'6091 lcod 123'6089 mlcod 123'6089 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:17.205 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.202+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6093 (116'3015,123'6093] local-lis/les=21/22 n=574 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'6091 crt=123'6091 lcod 123'6089 mlcod 123'6089 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:17.209 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.206+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5972 (115'2904,123'5972] local-lis/les=21/22 n=247 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5972 lcod 123'5970 mlcod 123'5970 active+clean+snaptrim trimq=[20d~1b] ps=[1fe~f]] removing snap head 2019-12-05T06:51:17.212 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.210+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6095 (116'3015,123'6095] local-lis/les=21/22 n=572 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'6095 lcod 123'6093 mlcod 123'6093 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:17.214 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.210+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6097 (116'3015,123'6097] local-lis/les=21/22 n=570 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'6095 crt=123'6095 lcod 123'6093 mlcod 123'6093 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:17.218 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.214+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5974 (115'2904,123'5974] local-lis/les=21/22 n=245 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5974 lcod 123'5972 mlcod 123'5972 active+clean+snaptrim trimq=[20e~1a] ps=[1fe~10]] removing snap head 2019-12-05T06:51:17.218 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.214+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6099 (116'3015,123'6099] local-lis/les=21/22 n=568 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'6099 lcod 123'6097 mlcod 123'6097 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:17.219 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.214+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5976 (115'2904,123'5976] local-lis/les=21/22 n=243 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5974 crt=123'5974 lcod 123'5972 mlcod 123'5972 active+clean+snaptrim trimq=[20e~1a] ps=[1fe~10]] removing snap head 2019-12-05T06:51:17.222 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.218+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6101 (116'3015,123'6101] local-lis/les=21/22 n=566 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'6099 crt=123'6099 lcod 123'6097 mlcod 123'6097 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:17.225 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.222+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5978 (115'2904,123'5978] local-lis/les=21/22 n=241 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5978 lcod 123'5976 mlcod 123'5976 active+clean+snaptrim trimq=[20e~1a] ps=[1fe~10]] removing snap head 2019-12-05T06:51:17.225 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.222+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6103 (116'3015,123'6103] local-lis/les=21/22 n=564 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'6103 lcod 123'6101 mlcod 123'6101 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:17.227 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.222+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5980 (115'2904,123'5980] local-lis/les=21/22 n=239 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5978 crt=123'5978 lcod 123'5976 mlcod 123'5976 active+clean+snaptrim trimq=[20e~1a] ps=[1fe~10]] removing snap head 2019-12-05T06:51:17.228 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.222+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6105 (116'3015,123'6105] local-lis/les=21/22 n=562 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'6103 crt=123'6103 lcod 123'6101 mlcod 123'6101 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:17.231 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.226+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6107 (116'3015,123'6107] local-lis/les=21/22 n=560 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'6107 lcod 123'6105 mlcod 123'6105 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:17.232 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.230+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5982 (115'2904,123'5982] local-lis/les=21/22 n=237 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5982 lcod 123'5980 mlcod 123'5980 active+clean+snaptrim trimq=[20e~1a] ps=[1fe~10]] removing snap head 2019-12-05T06:51:17.233 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.230+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6109 (116'3015,123'6109] local-lis/les=21/22 n=558 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'6107 crt=123'6107 lcod 123'6105 mlcod 123'6105 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:17.235 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.230+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5984 (115'2904,123'5984] local-lis/les=21/22 n=235 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5982 crt=123'5982 lcod 123'5980 mlcod 123'5980 active+clean+snaptrim trimq=[20e~1a] ps=[1fe~10]] removing snap head 2019-12-05T06:51:17.238 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.234+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6111 (116'3015,123'6111] local-lis/les=21/22 n=556 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'6111 lcod 123'6109 mlcod 123'6109 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:17.242 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.234+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5986 (115'2904,123'5986] local-lis/les=21/22 n=233 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5986 lcod 123'5984 mlcod 123'5984 active+clean+snaptrim trimq=[20e~1a] ps=[1fe~10]] removing snap head 2019-12-05T06:51:17.242 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.234+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6113 (116'3015,123'6113] local-lis/les=21/22 n=554 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'6111 crt=123'6111 lcod 123'6109 mlcod 123'6109 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:17.242 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.238+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5988 (115'2904,123'5988] local-lis/les=21/22 n=231 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5986 crt=123'5986 lcod 123'5984 mlcod 123'5984 active+clean+snaptrim trimq=[20e~1a] ps=[1fe~10]] removing snap head 2019-12-05T06:51:17.245 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.242+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6115 (116'3015,123'6115] local-lis/les=21/22 n=552 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'6115 lcod 123'6113 mlcod 123'6113 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:17.247 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.242+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5990 (115'2904,123'5990] local-lis/les=21/22 n=229 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5990 lcod 123'5988 mlcod 123'5988 active+clean+snaptrim trimq=[20e~1a] ps=[1fe~10]] removing snap head 2019-12-05T06:51:17.248 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.246+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6117 (116'3115,123'6117] local-lis/les=21/22 n=550 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'6115 crt=123'6115 lcod 123'6113 mlcod 123'6113 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:17.249 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.246+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5992 (115'2904,123'5992] local-lis/les=21/22 n=227 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5990 crt=123'5990 lcod 123'5988 mlcod 123'5988 active+clean+snaptrim trimq=[20e~1a] ps=[1fe~10]] removing snap head 2019-12-05T06:51:17.268 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.250+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6119 (116'3115,123'6119] local-lis/les=21/22 n=548 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'6119 lcod 123'6117 mlcod 123'6117 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:17.268 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.250+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6121 (116'3115,123'6121] local-lis/les=21/22 n=546 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'6119 crt=123'6119 lcod 123'6117 mlcod 123'6117 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:17.268 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.254+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5994 (115'2904,123'5994] local-lis/les=21/22 n=225 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5994 lcod 123'5992 mlcod 123'5992 active+clean+snaptrim trimq=[20e~1a] ps=[1fe~10]] removing snap head 2019-12-05T06:51:17.268 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.258+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6123 (116'3115,123'6123] local-lis/les=21/22 n=544 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'6123 lcod 123'6121 mlcod 123'6121 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:17.268 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.258+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5996 (115'2904,123'5996] local-lis/les=21/22 n=223 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'5996 lcod 123'5994 mlcod 123'5994 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:17.268 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.262+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'5998 (115'2904,123'5998] local-lis/les=21/22 n=221 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'5996 crt=123'5996 lcod 123'5994 mlcod 123'5994 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:17.269 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.266+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6127 (116'3115,123'6127] local-lis/les=21/22 n=541 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'6127 lcod 123'6125 mlcod 123'6125 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:17.286 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.266+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'6000 (115'2904,123'6000] local-lis/les=21/22 n=219 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'6000 lcod 123'5998 mlcod 123'5998 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:17.286 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.266+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6129 (116'3115,123'6129] local-lis/les=21/22 n=539 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'6127 crt=123'6127 lcod 123'6125 mlcod 123'6125 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:17.286 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.270+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'6002 (115'2904,123'6002] local-lis/les=21/22 n=217 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'6000 crt=123'6000 lcod 123'5998 mlcod 123'5998 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:17.300 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.294+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5828 (115'2806,123'5828] local-lis/les=21/22 n=561 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5828 lcod 123'5826 mlcod 123'5826 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:17.301 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.294+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5830 (115'2806,123'5830] local-lis/les=21/22 n=559 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5828 crt=123'5828 lcod 123'5826 mlcod 123'5826 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:17.301 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:17.289+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5866 (115'2807,123'5866] local-lis/les=21/22 n=403 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5866 lcod 123'5864 mlcod 123'5864 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:17.301 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:17.293+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5868 (115'2807,123'5868] local-lis/les=21/22 n=401 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5866 crt=123'5866 lcod 123'5864 mlcod 123'5864 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:17.306 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.302+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5832 (115'2806,123'5832] local-lis/les=21/22 n=557 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5832 lcod 123'5830 mlcod 123'5830 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:17.306 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.302+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5774 (115'2707,123'5774] local-lis/les=21/22 n=621 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5774 lcod 123'5772 mlcod 123'5772 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:17.307 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:17.301+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5870 (115'2807,123'5870] local-lis/les=21/22 n=399 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5870 lcod 123'5868 mlcod 123'5868 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:17.309 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:17.301+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5872 (115'2807,123'5872] local-lis/les=21/22 n=397 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5870 crt=123'5870 lcod 123'5868 mlcod 123'5868 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:17.309 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.306+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5834 (115'2806,123'5834] local-lis/les=21/22 n=555 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5832 crt=123'5832 lcod 123'5830 mlcod 123'5830 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:17.309 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.306+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5776 (115'2707,123'5776] local-lis/les=21/22 n=619 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5774 crt=123'5774 lcod 123'5772 mlcod 123'5772 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:17.315 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.310+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5836 (115'2806,123'5836] local-lis/les=21/22 n=553 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5836 lcod 123'5834 mlcod 123'5834 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:17.316 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.314+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5778 (115'2707,123'5778] local-lis/les=21/22 n=617 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5778 lcod 123'5776 mlcod 123'5776 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:17.317 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:17.309+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5874 (115'2807,123'5874] local-lis/les=21/22 n=395 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5874 lcod 123'5872 mlcod 123'5872 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:17.317 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.314+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5838 (115'2806,123'5838] local-lis/les=21/22 n=551 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5836 crt=123'5836 lcod 123'5834 mlcod 123'5834 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:17.319 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.314+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5780 (115'2707,123'5780] local-lis/les=21/22 n=615 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5778 crt=123'5778 lcod 123'5776 mlcod 123'5776 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:17.320 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:17.313+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5876 (115'2807,123'5876] local-lis/les=21/22 n=393 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5874 crt=123'5874 lcod 123'5872 mlcod 123'5872 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:17.324 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.322+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5840 (115'2806,123'5840] local-lis/les=21/22 n=549 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5840 lcod 123'5838 mlcod 123'5838 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:17.325 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:17.317+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5878 (115'2807,123'5878] local-lis/les=21/22 n=391 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5878 lcod 123'5876 mlcod 123'5876 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:17.326 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.322+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5842 (115'2806,123'5842] local-lis/les=21/22 n=547 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5840 crt=123'5840 lcod 123'5838 mlcod 123'5838 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:17.326 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.322+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5782 (115'2707,123'5782] local-lis/les=21/22 n=613 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5782 lcod 123'5780 mlcod 123'5780 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:17.328 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.326+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5784 (115'2707,123'5784] local-lis/les=21/22 n=611 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5782 crt=123'5782 lcod 123'5780 mlcod 123'5780 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:17.333 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:17.325+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5880 (115'2807,123'5880] local-lis/les=21/22 n=389 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5880 lcod 123'5878 mlcod 123'5878 active+clean+snaptrim trimq=[208~20] ps=[1fe~a]] removing snap head 2019-12-05T06:51:17.335 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:17.329+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5882 (115'2807,123'5882] local-lis/les=21/22 n=387 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5880 crt=123'5880 lcod 123'5878 mlcod 123'5878 active+clean+snaptrim trimq=[208~20] ps=[1fe~a]] removing snap head 2019-12-05T06:51:17.336 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.330+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5844 (115'2806,123'5844] local-lis/les=21/22 n=545 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5844 lcod 123'5842 mlcod 123'5842 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:17.336 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.334+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5846 (115'2806,123'5846] local-lis/les=21/22 n=543 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5844 crt=123'5844 lcod 123'5842 mlcod 123'5842 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:17.337 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.334+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5786 (115'2707,123'5786] local-lis/les=21/22 n=609 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5786 lcod 123'5784 mlcod 123'5784 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:17.342 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.338+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5788 (115'2707,123'5788] local-lis/les=21/22 n=607 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5788 lcod 123'5786 mlcod 123'5786 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:17.343 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.338+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5848 (115'2806,123'5848] local-lis/les=21/22 n=541 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5848 lcod 123'5846 mlcod 123'5846 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:17.343 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.338+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5790 (115'2707,123'5790] local-lis/les=21/22 n=605 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5788 crt=123'5788 lcod 123'5786 mlcod 123'5786 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:17.345 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.342+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5850 (115'2806,123'5850] local-lis/les=21/22 n=539 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5848 crt=123'5848 lcod 123'5846 mlcod 123'5846 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:17.349 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.346+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5792 (115'2707,123'5792] local-lis/les=21/22 n=603 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5792 lcod 123'5790 mlcod 123'5790 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:17.349 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.346+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5852 (115'2806,123'5852] local-lis/les=21/22 n=537 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5852 lcod 123'5850 mlcod 123'5850 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:17.351 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.346+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5794 (115'2707,123'5794] local-lis/les=21/22 n=601 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5792 crt=123'5792 lcod 123'5790 mlcod 123'5790 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:17.351 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.346+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5854 (115'2806,123'5854] local-lis/les=21/22 n=535 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5852 crt=123'5852 lcod 123'5850 mlcod 123'5850 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:17.355 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.350+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5856 (115'2806,123'5856] local-lis/les=21/22 n=533 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5856 lcod 123'5854 mlcod 123'5854 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:17.356 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.354+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5796 (115'2707,123'5796] local-lis/les=21/22 n=599 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5796 lcod 123'5794 mlcod 123'5794 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:17.356 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.354+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5858 (115'2806,123'5858] local-lis/les=21/22 n=531 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5856 crt=123'5856 lcod 123'5854 mlcod 123'5854 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:17.358 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.354+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5798 (115'2707,123'5798] local-lis/les=21/22 n=597 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5796 crt=123'5796 lcod 123'5794 mlcod 123'5794 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:17.363 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.358+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5860 (115'2806,123'5860] local-lis/les=21/22 n=529 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5860 lcod 123'5858 mlcod 123'5858 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:17.363 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.358+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5800 (115'2707,123'5800] local-lis/les=21/22 n=595 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5800 lcod 123'5798 mlcod 123'5798 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:17.365 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.362+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5862 (115'2806,123'5862] local-lis/les=21/22 n=527 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5860 crt=123'5860 lcod 123'5858 mlcod 123'5858 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:17.365 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.362+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5802 (115'2707,123'5802] local-lis/les=21/22 n=593 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5800 crt=123'5800 lcod 123'5798 mlcod 123'5798 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:17.381 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.366+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5864 (115'2806,123'5864] local-lis/les=21/22 n=525 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5864 lcod 123'5862 mlcod 123'5862 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:17.387 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.366+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5804 (115'2707,123'5804] local-lis/les=21/22 n=591 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5804 lcod 123'5802 mlcod 123'5802 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:17.387 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.366+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5866 (115'2806,123'5866] local-lis/les=21/22 n=523 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5864 crt=123'5864 lcod 123'5862 mlcod 123'5862 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:17.387 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.366+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5806 (115'2707,123'5806] local-lis/les=21/22 n=589 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5804 crt=123'5804 lcod 123'5802 mlcod 123'5802 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:17.387 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.370+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5808 (115'2707,123'5808] local-lis/les=21/22 n=587 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5808 lcod 123'5806 mlcod 123'5806 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:17.388 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.374+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5868 (115'2806,123'5868] local-lis/les=21/22 n=521 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5868 lcod 123'5866 mlcod 123'5866 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:17.388 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.374+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5810 (116'2808,123'5810] local-lis/les=21/22 n=585 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5808 crt=123'5808 lcod 123'5806 mlcod 123'5806 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:17.388 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.374+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5870 (115'2806,123'5870] local-lis/les=21/22 n=519 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5868 crt=123'5868 lcod 123'5866 mlcod 123'5866 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:17.389 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.382+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5812 (116'2808,123'5812] local-lis/les=21/22 n=583 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5812 lcod 123'5810 mlcod 123'5810 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:17.389 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.382+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5872 (115'2806,123'5872] local-lis/les=21/22 n=517 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5872 lcod 123'5870 mlcod 123'5870 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:17.389 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.382+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5814 (116'2808,123'5814] local-lis/les=21/22 n=581 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5812 crt=123'5812 lcod 123'5810 mlcod 123'5810 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:17.389 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.382+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5874 (115'2806,123'5874] local-lis/les=21/22 n=515 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5872 crt=123'5872 lcod 123'5870 mlcod 123'5870 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:17.393 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.386+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5816 (116'2808,123'5816] local-lis/les=21/22 n=579 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5816 lcod 123'5814 mlcod 123'5814 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:17.393 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.390+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5818 (116'2808,123'5818] local-lis/les=21/22 n=577 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5816 crt=123'5816 lcod 123'5814 mlcod 123'5814 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:17.394 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.390+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5876 (115'2806,123'5876] local-lis/les=21/22 n=513 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5876 lcod 123'5874 mlcod 123'5874 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:17.395 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.390+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5878 (115'2806,123'5878] local-lis/les=21/22 n=511 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5876 crt=123'5876 lcod 123'5874 mlcod 123'5874 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:17.396 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:17.389+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5748 (115'2706,123'5748] local-lis/les=21/22 n=772 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=123'5748 lcod 123'5746 mlcod 123'5746 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:17.399 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.394+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'6004 (115'2904,123'6004] local-lis/les=21/22 n=215 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'6004 lcod 123'6002 mlcod 123'6002 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:17.403 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:17.393+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5750 (115'2706,123'5750] local-lis/les=21/22 n=770 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5748 crt=123'5748 lcod 123'5746 mlcod 123'5746 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:17.404 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.398+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5880 (115'2806,123'5880] local-lis/les=21/22 n=509 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5880 lcod 123'5878 mlcod 123'5878 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:17.404 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.402+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5820 (116'2808,123'5820] local-lis/les=21/22 n=575 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5820 lcod 123'5818 mlcod 123'5818 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:17.404 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.398+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'6006 (116'3004,123'6006] local-lis/les=21/22 n=213 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'6004 crt=123'6004 lcod 123'6002 mlcod 123'6002 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:17.406 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.402+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5822 (116'2808,123'5822] local-lis/les=21/22 n=573 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5820 crt=123'5820 lcod 123'5818 mlcod 123'5818 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:17.406 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.402+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5882 (115'2806,123'5882] local-lis/les=21/22 n=507 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5880 crt=123'5880 lcod 123'5878 mlcod 123'5878 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:17.412 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.410+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'6008 (116'3004,123'6008] local-lis/les=21/22 n=211 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'6008 lcod 123'6006 mlcod 123'6006 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:17.413 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:17.405+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5752 (115'2706,123'5752] local-lis/les=21/22 n=768 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=123'5752 lcod 123'5750 mlcod 123'5750 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:17.413 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:17.405+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5754 (115'2706,123'5754] local-lis/les=21/22 n=766 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5752 crt=123'5752 lcod 123'5750 mlcod 123'5750 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:17.414 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.410+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'6010 (116'3004,123'6010] local-lis/les=21/22 n=209 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'6008 crt=123'6008 lcod 123'6006 mlcod 123'6006 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:17.415 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.410+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5824 (116'2808,123'5824] local-lis/les=21/22 n=571 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5824 lcod 123'5822 mlcod 123'5822 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:17.415 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.410+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5884 (115'2806,123'5884] local-lis/les=21/22 n=505 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5884 lcod 123'5882 mlcod 123'5882 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:17.417 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.414+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5826 (116'2808,123'5826] local-lis/les=21/22 n=569 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5824 crt=123'5824 lcod 123'5822 mlcod 123'5822 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:17.417 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.414+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5886 (115'2806,123'5886] local-lis/les=21/22 n=503 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5884 crt=123'5884 lcod 123'5882 mlcod 123'5882 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:17.422 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.418+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'6012 (116'3004,123'6012] local-lis/les=21/22 n=207 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'6012 lcod 123'6010 mlcod 123'6010 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:17.424 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.422+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'6014 (116'3004,123'6014] local-lis/les=21/22 n=205 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'6012 crt=123'6012 lcod 123'6010 mlcod 123'6010 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:17.425 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:17.417+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5756 (115'2706,123'5756] local-lis/les=21/22 n=764 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=123'5756 lcod 123'5754 mlcod 123'5754 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:17.426 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.422+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5828 (116'2808,123'5828] local-lis/les=21/22 n=567 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5828 lcod 123'5826 mlcod 123'5826 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:17.428 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.422+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5830 (116'2808,123'5830] local-lis/les=21/22 n=565 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5828 crt=123'5828 lcod 123'5826 mlcod 123'5826 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:17.430 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.426+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5888 (115'2806,123'5888] local-lis/les=21/22 n=501 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5888 lcod 123'5886 mlcod 123'5886 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:17.431 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.426+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'6016 (116'3004,123'6016] local-lis/les=21/22 n=203 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'6016 lcod 123'6014 mlcod 123'6014 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:17.431 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:17.421+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5758 (115'2706,123'5758] local-lis/les=21/22 n=762 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5756 crt=123'5756 lcod 123'5754 mlcod 123'5754 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:17.432 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.430+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5890 (115'2806,123'5890] local-lis/les=21/22 n=499 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5888 crt=123'5888 lcod 123'5886 mlcod 123'5886 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:17.435 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.430+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'6018 (116'3004,123'6018] local-lis/les=21/22 n=201 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'6016 crt=123'6016 lcod 123'6014 mlcod 123'6014 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:17.435 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:17.425+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5884 (115'2807,123'5884] local-lis/les=21/22 n=385 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5884 lcod 123'5882 mlcod 123'5882 active+clean+snaptrim trimq=[208~20] ps=[1fe~a]] removing snap head 2019-12-05T06:51:17.436 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:17.429+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5886 (115'2807,123'5886] local-lis/les=21/22 n=383 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5884 crt=123'5884 lcod 123'5882 mlcod 123'5882 active+clean+snaptrim trimq=[208~20] ps=[1fe~a]] removing snap head 2019-12-05T06:51:17.437 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:17.429+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5760 (115'2706,123'5760] local-lis/les=21/22 n=760 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=123'5760 lcod 123'5758 mlcod 123'5758 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:17.440 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:17.433+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5762 (115'2706,123'5762] local-lis/les=21/22 n=758 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5760 crt=123'5760 lcod 123'5758 mlcod 123'5758 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:17.443 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.438+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5832 (116'2808,123'5832] local-lis/les=21/22 n=563 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5832 lcod 123'5830 mlcod 123'5830 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:17.444 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.438+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'6020 (116'3004,123'6020] local-lis/les=21/22 n=199 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'6020 lcod 123'6018 mlcod 123'6018 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:17.445 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.438+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'6022 (116'3004,123'6022] local-lis/les=21/22 n=197 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'6020 crt=123'6020 lcod 123'6018 mlcod 123'6018 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:17.445 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.442+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5834 (116'2808,123'5834] local-lis/les=21/22 n=561 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5832 crt=123'5832 lcod 123'5830 mlcod 123'5830 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:17.447 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:17.441+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5764 (115'2706,123'5764] local-lis/les=21/22 n=756 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=123'5764 lcod 123'5762 mlcod 123'5762 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:17.450 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.446+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5892 (115'2806,123'5892] local-lis/les=21/22 n=497 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5892 lcod 123'5890 mlcod 123'5890 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:17.452 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.446+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5894 (115'2806,123'5894] local-lis/les=21/22 n=495 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5892 crt=123'5892 lcod 123'5890 mlcod 123'5890 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:17.452 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:17.445+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5888 (115'2807,123'5888] local-lis/les=21/22 n=381 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5888 lcod 123'5886 mlcod 123'5886 active+clean+snaptrim trimq=[208~20] ps=[1fe~a]] removing snap head 2019-12-05T06:51:17.453 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:17.445+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5766 (115'2706,123'5766] local-lis/les=21/22 n=754 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5764 crt=123'5764 lcod 123'5762 mlcod 123'5762 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:17.454 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:17.445+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5890 (115'2807,123'5890] local-lis/les=21/22 n=379 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5888 crt=123'5888 lcod 123'5886 mlcod 123'5886 active+clean+snaptrim trimq=[208~20] ps=[1fe~a]] removing snap head 2019-12-05T06:51:17.454 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:17.450+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6414 (117'3412,123'6414] local-lis/les=21/22 n=275 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'6414 lcod 123'6412 mlcod 123'6412 active+clean+snaptrim trimq=[20c~1c] ps=[1fe~e]] removing snap head 2019-12-05T06:51:17.454 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.450+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'6024 (116'3004,123'6024] local-lis/les=21/22 n=195 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'6024 lcod 123'6022 mlcod 123'6022 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:17.455 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:17.450+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6416 (117'3412,123'6416] local-lis/les=21/22 n=273 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6414 crt=123'6414 lcod 123'6412 mlcod 123'6412 active+clean+snaptrim trimq=[20c~1c] ps=[1fe~e]] removing snap head 2019-12-05T06:51:17.456 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.454+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'6026 (116'3004,123'6026] local-lis/les=21/22 n=193 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'6024 crt=123'6024 lcod 123'6022 mlcod 123'6022 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:17.458 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.454+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6131 (116'3115,123'6131] local-lis/les=21/22 n=537 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'6131 lcod 123'6129 mlcod 123'6129 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:17.463 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.458+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6133 (116'3115,123'6133] local-lis/les=21/22 n=535 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'6131 crt=123'6131 lcod 123'6129 mlcod 123'6129 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:17.464 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:17.458+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6234 (117'3208,123'6234] local-lis/les=21/22 n=231 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'6234 lcod 123'6232 mlcod 123'6232 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:17.464 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:17.458+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6236 (117'3208,123'6236] local-lis/les=21/22 n=229 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'6234 crt=123'6234 lcod 123'6232 mlcod 123'6232 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:17.468 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.466+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5836 (116'2808,123'5836] local-lis/les=21/22 n=559 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5836 lcod 123'5834 mlcod 123'5834 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:17.470 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:17.461+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5768 (115'2706,123'5768] local-lis/les=21/22 n=752 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=123'5768 lcod 123'5766 mlcod 123'5766 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:17.471 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:17.465+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5892 (115'2807,123'5892] local-lis/les=21/22 n=377 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5892 lcod 123'5890 mlcod 123'5890 active+clean+snaptrim trimq=[208~20] ps=[1fe~a]] removing snap head 2019-12-05T06:51:17.471 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.466+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5838 (116'2808,123'5838] local-lis/les=21/22 n=557 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5836 crt=123'5836 lcod 123'5834 mlcod 123'5834 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:17.472 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:17.465+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5770 (115'2706,123'5770] local-lis/les=21/22 n=750 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5768 crt=123'5768 lcod 123'5766 mlcod 123'5766 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:17.475 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:17.465+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5894 (115'2807,123'5894] local-lis/les=21/22 n=375 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5892 crt=123'5892 lcod 123'5890 mlcod 123'5890 active+clean+snaptrim trimq=[208~20] ps=[1fe~a]] removing snap head 2019-12-05T06:51:17.478 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.474+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'6028 (116'3004,123'6028] local-lis/les=21/22 n=191 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'6028 lcod 123'6026 mlcod 123'6026 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:17.479 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.474+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5896 (115'2806,123'5896] local-lis/les=21/22 n=493 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5896 lcod 123'5894 mlcod 123'5894 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:17.480 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:17.478+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6418 (117'3412,123'6418] local-lis/les=21/22 n=271 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'6418 lcod 123'6416 mlcod 123'6416 active+clean+snaptrim trimq=[20d~1b] ps=[1fe~f]] removing snap head 2019-12-05T06:51:17.481 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.478+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5898 (115'2806,123'5898] local-lis/les=21/22 n=491 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5896 crt=123'5896 lcod 123'5894 mlcod 123'5894 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:17.482 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.478+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6135 (116'3115,123'6135] local-lis/les=21/22 n=533 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'6135 lcod 123'6133 mlcod 123'6133 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:17.483 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:17.478+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6420 (117'3412,123'6420] local-lis/les=21/22 n=269 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6418 crt=123'6418 lcod 123'6416 mlcod 123'6416 active+clean+snaptrim trimq=[20d~1b] ps=[1fe~f]] removing snap head 2019-12-05T06:51:17.483 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.478+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'6030 (116'3004,123'6030] local-lis/les=21/22 n=189 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'6028 crt=123'6028 lcod 123'6026 mlcod 123'6026 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:17.493 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:17.486+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6238 (117'3208,123'6238] local-lis/les=21/22 n=227 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'6238 lcod 123'6236 mlcod 123'6236 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:17.494 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:17.485+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5772 (115'2706,123'5772] local-lis/les=21/22 n=748 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=123'5772 lcod 123'5770 mlcod 123'5770 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:17.494 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:17.486+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6422 (117'3412,123'6422] local-lis/les=21/22 n=267 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'6422 lcod 123'6420 mlcod 123'6420 active+clean+snaptrim trimq=[20d~1b] ps=[1fe~f]] removing snap head 2019-12-05T06:51:17.494 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:17.490+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6240 (117'3208,123'6240] local-lis/les=21/22 n=225 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'6238 crt=123'6238 lcod 123'6236 mlcod 123'6236 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:17.496 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:17.489+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5774 (115'2706,123'5774] local-lis/les=21/22 n=746 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5772 crt=123'5772 lcod 123'5770 mlcod 123'5770 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:17.498 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.490+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6137 (116'3115,123'6137] local-lis/les=21/22 n=531 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'6137 lcod 123'6135 mlcod 123'6135 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:17.498 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.490+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'6032 (116'3004,123'6032] local-lis/les=21/22 n=187 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'6032 lcod 123'6030 mlcod 123'6030 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:17.499 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.494+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5840 (116'2808,123'5840] local-lis/les=21/22 n=555 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5840 lcod 123'5838 mlcod 123'5838 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:17.499 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.494+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6139 (116'3115,123'6139] local-lis/les=21/22 n=529 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'6137 crt=123'6137 lcod 123'6135 mlcod 123'6135 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:17.503 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:17.497+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5896 (115'2807,123'5896] local-lis/les=21/22 n=373 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5896 lcod 123'5894 mlcod 123'5894 active+clean+snaptrim trimq=[208~20] ps=[1fe~a]] removing snap head 2019-12-05T06:51:17.503 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.498+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'6034 (116'3004,123'6034] local-lis/les=21/22 n=185 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'6032 crt=123'6032 lcod 123'6030 mlcod 123'6030 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:17.508 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:17.502+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6424 (117'3412,123'6424] local-lis/les=21/22 n=265 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6422 crt=123'6422 lcod 123'6420 mlcod 123'6420 active+clean+snaptrim trimq=[20d~1b] ps=[1fe~f]] removing snap head 2019-12-05T06:51:17.509 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:17.502+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6243 (117'3208,123'6243] local-lis/les=21/22 n=222 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'6243 lcod 123'6242 mlcod 123'6242 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:17.510 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:17.506+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6245 (117'3208,123'6245] local-lis/les=21/22 n=220 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'6243 crt=123'6243 lcod 123'6242 mlcod 123'6242 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:17.512 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.502+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5900 (115'2806,123'5900] local-lis/les=21/22 n=489 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5900 lcod 123'5898 mlcod 123'5898 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:17.512 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.502+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5842 (116'2808,123'5842] local-lis/les=21/22 n=553 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5840 crt=123'5840 lcod 123'5838 mlcod 123'5838 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:17.513 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.510+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5902 (115'2806,123'5902] local-lis/les=21/22 n=487 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5900 crt=123'5900 lcod 123'5898 mlcod 123'5898 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:17.516 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.510+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6141 (116'3115,123'6141] local-lis/les=21/22 n=527 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'6141 lcod 123'6139 mlcod 123'6139 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:17.516 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:17.509+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5776 (115'2706,123'5776] local-lis/les=21/22 n=744 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=123'5776 lcod 123'5774 mlcod 123'5774 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:17.517 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.514+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'6037 (116'3004,123'6037] local-lis/les=21/22 n=182 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'6036 crt=123'6032 lcod 123'6034 mlcod 123'6034 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:17.518 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:17.509+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5778 (115'2706,123'5778] local-lis/les=21/22 n=742 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5776 crt=123'5776 lcod 123'5774 mlcod 123'5774 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:17.521 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.518+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5844 (116'2808,123'5844] local-lis/les=21/22 n=551 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5844 lcod 123'5842 mlcod 123'5842 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:17.522 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.518+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5846 (116'2808,123'5846] local-lis/les=21/22 n=549 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5844 crt=123'5844 lcod 123'5842 mlcod 123'5842 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:17.524 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:17.517+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5898 (115'2807,123'5898] local-lis/les=21/22 n=371 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5898 lcod 123'5896 mlcod 123'5896 active+clean+snaptrim trimq=[209~1f] ps=[1fe~b]] removing snap head 2019-12-05T06:51:17.526 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:17.522+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6426 (117'3412,123'6426] local-lis/les=21/22 n=263 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'6426 lcod 123'6424 mlcod 123'6424 active+clean+snaptrim trimq=[20d~1b] ps=[1fe~f]] removing snap head 2019-12-05T06:51:17.527 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.522+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6143 (116'3115,123'6143] local-lis/les=21/22 n=525 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'6141 crt=123'6141 lcod 123'6139 mlcod 123'6139 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:17.527 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.522+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'6039 (116'3004,123'6039] local-lis/les=21/22 n=180 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'6036 crt=123'6036 lcod 123'6034 mlcod 123'6034 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:17.529 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:17.526+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6247 (117'3208,123'6247] local-lis/les=21/22 n=218 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'6247 lcod 123'6245 mlcod 123'6245 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:17.529 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:17.517+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5900 (115'2807,123'5900] local-lis/les=21/22 n=369 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5898 crt=123'5898 lcod 123'5896 mlcod 123'5896 active+clean+snaptrim trimq=[209~1f] ps=[1fe~b]] removing snap head 2019-12-05T06:51:17.530 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:17.526+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6249 (117'3208,123'6249] local-lis/les=21/22 n=216 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'6247 crt=123'6247 lcod 123'6245 mlcod 123'6245 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:17.531 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.526+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5904 (115'2806,123'5904] local-lis/les=21/22 n=485 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5904 lcod 123'5902 mlcod 123'5902 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:17.536 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:17.530+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6428 (117'3412,123'6428] local-lis/les=21/22 n=261 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6426 crt=123'6426 lcod 123'6424 mlcod 123'6424 active+clean+snaptrim trimq=[20d~1b] ps=[1fe~f]] removing snap head 2019-12-05T06:51:17.536 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:17.529+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5902 (115'2807,123'5902] local-lis/les=21/22 n=367 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5902 lcod 123'5900 mlcod 123'5900 active+clean+snaptrim trimq=[209~1f] ps=[1fe~b]] removing snap head 2019-12-05T06:51:17.537 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:17.529+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5780 (115'2706,123'5780] local-lis/les=21/22 n=740 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=123'5780 lcod 123'5778 mlcod 123'5778 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:17.540 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:17.533+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5904 (115'2807,123'5904] local-lis/les=21/22 n=365 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5902 crt=123'5902 lcod 123'5900 mlcod 123'5900 active+clean+snaptrim trimq=[209~1f] ps=[1fe~b]] removing snap head 2019-12-05T06:51:17.540 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.530+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5906 (115'2806,123'5906] local-lis/les=21/22 n=483 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5904 crt=123'5904 lcod 123'5902 mlcod 123'5902 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:17.541 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:17.533+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5782 (115'2706,123'5782] local-lis/les=21/22 n=738 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5780 crt=123'5780 lcod 123'5778 mlcod 123'5778 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:17.544 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.538+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'6041 (116'3004,123'6041] local-lis/les=21/22 n=178 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'6041 lcod 123'6039 mlcod 123'6039 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:17.545 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:17.538+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6431 (117'3412,123'6431] local-lis/les=21/22 n=258 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6430 crt=123'6428 lcod 123'6428 mlcod 123'6428 active+clean+snaptrim trimq=[20e~1a] ps=[1fe~10]] removing snap head 2019-12-05T06:51:17.546 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:17.542+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6433 (117'3412,123'6433] local-lis/les=21/22 n=256 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6430 crt=123'6430 lcod 123'6428 mlcod 123'6428 active+clean+snaptrim trimq=[20e~1a] ps=[1fe~10]] removing snap head 2019-12-05T06:51:17.547 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:17.542+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6251 (117'3208,123'6251] local-lis/les=21/22 n=214 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'6251 lcod 123'6249 mlcod 123'6249 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:17.549 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:17.546+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6253 (117'3208,123'6253] local-lis/les=21/22 n=212 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'6251 crt=123'6251 lcod 123'6249 mlcod 123'6249 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:17.549 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.542+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5848 (116'2808,123'5848] local-lis/les=21/22 n=547 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5848 lcod 123'5846 mlcod 123'5846 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:17.550 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.546+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5850 (116'2808,123'5850] local-lis/les=21/22 n=545 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5848 crt=123'5848 lcod 123'5846 mlcod 123'5846 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:17.552 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.550+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'6043 (116'3004,123'6043] local-lis/les=21/22 n=176 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'6041 crt=123'6041 lcod 123'6039 mlcod 123'6039 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:17.554 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.550+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5908 (115'2906,123'5908] local-lis/les=21/22 n=481 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5908 lcod 123'5906 mlcod 123'5906 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:17.556 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.554+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5910 (115'2906,123'5910] local-lis/les=21/22 n=479 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5908 crt=123'5908 lcod 123'5906 mlcod 123'5906 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:17.557 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.554+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6145 (116'3115,123'6145] local-lis/les=21/22 n=523 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'6145 lcod 123'6143 mlcod 123'6143 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:17.558 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:17.554+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6435 (117'3412,123'6435] local-lis/les=21/22 n=254 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'6435 lcod 123'6433 mlcod 123'6433 active+clean+snaptrim trimq=[20e~1a] ps=[1fe~10]] removing snap head 2019-12-05T06:51:17.559 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:17.553+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5906 (115'2807,123'5906] local-lis/les=21/22 n=363 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5906 lcod 123'5904 mlcod 123'5904 active+clean+snaptrim trimq=[209~1f] ps=[1fe~b]] removing snap head 2019-12-05T06:51:17.560 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:17.553+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5784 (115'2706,123'5784] local-lis/les=21/22 n=736 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=123'5784 lcod 123'5782 mlcod 123'5782 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:17.562 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:17.553+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5786 (115'2706,123'5786] local-lis/les=21/22 n=734 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5784 crt=123'5784 lcod 123'5782 mlcod 123'5782 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:17.563 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:17.558+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6437 (117'3412,123'6437] local-lis/les=21/22 n=252 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6435 crt=123'6435 lcod 123'6433 mlcod 123'6433 active+clean+snaptrim trimq=[20e~1a] ps=[1fe~10]] removing snap head 2019-12-05T06:51:17.568 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.558+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6147 (116'3115,123'6147] local-lis/les=21/22 n=521 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'6145 crt=123'6145 lcod 123'6143 mlcod 123'6143 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:17.569 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:17.557+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5908 (115'2807,123'5908] local-lis/les=21/22 n=361 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5906 crt=123'5906 lcod 123'5904 mlcod 123'5904 active+clean+snaptrim trimq=[209~1f] ps=[1fe~b]] removing snap head 2019-12-05T06:51:17.569 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:17.566+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6255 (117'3208,123'6255] local-lis/les=21/22 n=210 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'6255 lcod 123'6253 mlcod 123'6253 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:17.570 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.562+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'6045 (116'3004,123'6045] local-lis/les=21/22 n=174 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'6045 lcod 123'6043 mlcod 123'6043 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:17.570 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.562+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'6047 (116'3004,123'6047] local-lis/les=21/22 n=172 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'6045 crt=123'6045 lcod 123'6043 mlcod 123'6043 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:17.571 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:17.566+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6257 (117'3208,123'6257] local-lis/les=21/22 n=208 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'6255 crt=123'6255 lcod 123'6253 mlcod 123'6253 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:17.571 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.566+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5852 (116'2808,123'5852] local-lis/les=21/22 n=543 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5852 lcod 123'5850 mlcod 123'5850 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:17.575 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.570+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5854 (116'2808,123'5854] local-lis/les=21/22 n=541 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5852 crt=123'5852 lcod 123'5850 mlcod 123'5850 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:17.578 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:17.574+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6439 (117'3412,123'6439] local-lis/les=21/22 n=250 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'6439 lcod 123'6437 mlcod 123'6437 active+clean+snaptrim trimq=[20e~1a] ps=[1fe~10]] removing snap head 2019-12-05T06:51:17.580 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.574+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5913 (115'2906,123'5913] local-lis/les=21/22 n=476 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5913 lcod 123'5912 mlcod 123'5912 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:17.582 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.578+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5915 (115'2906,123'5915] local-lis/les=21/22 n=474 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5913 crt=123'5913 lcod 123'5912 mlcod 123'5912 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:17.583 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:17.577+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5910 (116'2908,123'5910] local-lis/les=21/22 n=359 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5910 lcod 123'5908 mlcod 123'5908 active+clean+snaptrim trimq=[209~1f] ps=[1fe~b]] removing snap head 2019-12-05T06:51:17.583 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:17.577+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5788 (115'2706,123'5788] local-lis/les=21/22 n=732 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=123'5788 lcod 123'5786 mlcod 123'5786 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:17.583 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.578+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6149 (116'3115,123'6149] local-lis/les=21/22 n=519 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'6149 lcod 123'6147 mlcod 123'6147 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:17.584 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.578+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6151 (116'3115,123'6151] local-lis/les=21/22 n=517 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'6149 crt=123'6149 lcod 123'6147 mlcod 123'6147 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:17.585 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:17.577+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5790 (115'2706,123'5790] local-lis/les=21/22 n=730 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5788 crt=123'5788 lcod 123'5786 mlcod 123'5786 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:17.585 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:17.578+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6441 (117'3412,123'6441] local-lis/les=21/22 n=248 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6439 crt=123'6439 lcod 123'6437 mlcod 123'6437 active+clean+snaptrim trimq=[20e~1a] ps=[1fe~10]] removing snap head 2019-12-05T06:51:17.591 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.586+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'6049 (116'3004,123'6049] local-lis/les=21/22 n=170 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'6049 lcod 123'6047 mlcod 123'6047 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:17.591 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:17.577+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5912 (116'2908,123'5912] local-lis/les=21/22 n=357 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5910 crt=123'5910 lcod 123'5908 mlcod 123'5908 active+clean+snaptrim trimq=[209~1f] ps=[1fe~b]] removing snap head 2019-12-05T06:51:17.592 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:17.590+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6259 (117'3208,123'6259] local-lis/les=21/22 n=206 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'6259 lcod 123'6257 mlcod 123'6257 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:17.594 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.590+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5856 (116'2808,123'5856] local-lis/les=21/22 n=539 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5856 lcod 123'5854 mlcod 123'5854 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:17.596 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:17.590+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6261 (117'3208,123'6261] local-lis/les=21/22 n=204 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'6259 crt=123'6259 lcod 123'6257 mlcod 123'6257 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:17.598 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.590+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'6051 (116'3004,123'6051] local-lis/les=21/22 n=168 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'6049 crt=123'6049 lcod 123'6047 mlcod 123'6047 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:17.598 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.590+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5858 (116'2808,123'5858] local-lis/les=21/22 n=537 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5856 crt=123'5856 lcod 123'5854 mlcod 123'5854 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:17.599 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:17.594+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6444 (117'3412,123'6444] local-lis/les=21/22 n=245 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6443 crt=123'6439 lcod 123'6441 mlcod 123'6441 active+clean+snaptrim trimq=[20e~1a] ps=[1fe~10]] removing snap head 2019-12-05T06:51:17.603 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:17.597+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5915 (116'2908,123'5915] local-lis/les=21/22 n=354 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5915 lcod 123'5914 mlcod 123'5914 active+clean+snaptrim trimq=[20a~1e] ps=[1fe~c]] removing snap head 2019-12-05T06:51:17.604 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.598+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6153 (116'3115,123'6153] local-lis/les=21/22 n=515 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'6153 lcod 123'6151 mlcod 123'6151 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:17.605 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.602+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5917 (115'2906,123'5917] local-lis/les=21/22 n=472 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5917 lcod 123'5915 mlcod 123'5915 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:17.605 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:17.602+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6446 (117'3412,123'6446] local-lis/les=21/22 n=243 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6443 crt=123'6443 lcod 123'6441 mlcod 123'6441 active+clean+snaptrim trimq=[20e~1a] ps=[1fe~10]] removing snap head 2019-12-05T06:51:17.609 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:17.601+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5793 (115'2706,123'5793] local-lis/les=21/22 n=727 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5792 crt=123'5792 lcod 123'5790 mlcod 123'5790 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:17.609 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.606+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5919 (115'2906,123'5919] local-lis/les=21/22 n=470 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5917 crt=123'5917 lcod 123'5915 mlcod 123'5915 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:17.610 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.606+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6155 (116'3115,123'6155] local-lis/les=21/22 n=513 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'6153 crt=123'6153 lcod 123'6151 mlcod 123'6151 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:17.611 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.606+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'6053 (116'3004,123'6053] local-lis/les=21/22 n=166 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'6053 lcod 123'6051 mlcod 123'6051 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:17.612 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:17.605+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5917 (116'2908,123'5917] local-lis/les=21/22 n=352 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5915 crt=123'5915 lcod 123'5914 mlcod 123'5914 active+clean+snaptrim trimq=[20a~1e] ps=[1fe~c]] removing snap head 2019-12-05T06:51:17.614 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:17.609+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5795 (115'2706,123'5795] local-lis/les=21/22 n=725 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5792 crt=123'5792 lcod 123'5790 mlcod 123'5790 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:17.618 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:17.614+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6448 (117'3412,123'6448] local-lis/les=21/22 n=241 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'6448 lcod 123'6446 mlcod 123'6446 active+clean+snaptrim trimq=[20e~1a] ps=[1fe~10]] removing snap head 2019-12-05T06:51:17.618 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.614+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5860 (116'2808,123'5860] local-lis/les=21/22 n=535 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5860 lcod 123'5858 mlcod 123'5858 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:17.622 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:17.618+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6264 (117'3208,123'6264] local-lis/les=21/22 n=201 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'6263 crt=123'6261 lcod 123'6261 mlcod 123'6261 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:17.624 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:17.617+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5798 (115'2706,123'5798] local-lis/les=21/22 n=722 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5797 crt=123'5795 lcod 123'5795 mlcod 123'5795 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:17.624 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:17.618+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6450 (117'3412,123'6450] local-lis/les=21/22 n=239 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6448 crt=123'6448 lcod 123'6446 mlcod 123'6446 active+clean+snaptrim trimq=[20e~1a] ps=[1fe~10]] removing snap head 2019-12-05T06:51:17.624 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.618+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5921 (115'2906,123'5921] local-lis/les=21/22 n=468 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5921 lcod 123'5919 mlcod 123'5919 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:17.626 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:17.621+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5800 (115'2706,123'5800] local-lis/les=21/22 n=720 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5797 crt=123'5797 lcod 123'5795 mlcod 123'5795 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:17.627 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:17.622+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6266 (117'3208,123'6266] local-lis/les=21/22 n=199 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'6263 crt=123'6263 lcod 123'6261 mlcod 123'6261 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:17.628 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.618+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6157 (116'3115,123'6157] local-lis/les=21/22 n=511 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'6157 lcod 123'6155 mlcod 123'6155 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:17.629 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.618+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6159 (116'3115,123'6159] local-lis/les=21/22 n=509 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'6157 crt=123'6157 lcod 123'6155 mlcod 123'6155 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:17.631 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.622+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5923 (115'2906,123'5923] local-lis/les=21/22 n=466 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5921 crt=123'5921 lcod 123'5919 mlcod 123'5919 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:17.631 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.626+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5862 (116'2808,123'5862] local-lis/les=21/22 n=533 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5860 crt=123'5860 lcod 123'5858 mlcod 123'5858 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:17.638 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:17.629+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5802 (115'2706,123'5802] local-lis/les=21/22 n=718 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=123'5802 lcod 123'5800 mlcod 123'5800 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:17.638 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:17.634+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6452 (117'3412,123'6452] local-lis/les=21/22 n=237 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'6452 lcod 123'6450 mlcod 123'6450 active+clean+snaptrim trimq=[20e~1a] ps=[1fe~10]] removing snap head 2019-12-05T06:51:17.640 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.638+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5925 (115'2906,123'5925] local-lis/les=21/22 n=464 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5925 lcod 123'5923 mlcod 123'5923 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:17.640 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.634+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6163 (116'3115,123'6163] local-lis/les=21/22 n=505 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'6161 crt=123'6159 lcod 123'6159 mlcod 123'6159 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:17.641 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.634+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6165 (116'3115,123'6165] local-lis/les=21/22 n=503 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'6161 crt=123'6161 lcod 123'6159 mlcod 123'6159 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:17.642 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:17.638+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6268 (117'3208,123'6268] local-lis/les=21/22 n=197 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'6268 lcod 123'6266 mlcod 123'6266 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:17.648 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.642+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'6055 (116'3004,123'6055] local-lis/les=21/22 n=164 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'6055 lcod 123'6053 mlcod 123'6053 active+clean+snaptrim trimq=[210~18] ps=[1fe~12]] removing snap head 2019-12-05T06:51:17.648 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:17.642+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6270 (117'3208,123'6270] local-lis/les=21/22 n=195 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'6268 crt=123'6268 lcod 123'6266 mlcod 123'6266 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:17.649 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.646+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5927 (115'2906,123'5927] local-lis/les=21/22 n=462 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5925 crt=123'5925 lcod 123'5923 mlcod 123'5923 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:17.650 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.646+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5864 (116'2808,123'5864] local-lis/les=21/22 n=531 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5864 lcod 123'5862 mlcod 123'5862 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:17.651 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:17.646+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6454 (117'3412,123'6454] local-lis/les=21/22 n=235 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'6454 lcod 123'6452 mlcod 123'6452 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:17.651 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.646+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'6057 (116'3004,123'6057] local-lis/les=21/22 n=162 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'6055 crt=123'6055 lcod 123'6053 mlcod 123'6053 active+clean+snaptrim trimq=[210~18] ps=[1fe~12]] removing snap head 2019-12-05T06:51:17.656 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.650+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5866 (116'2808,123'5866] local-lis/les=21/22 n=529 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5864 crt=123'5864 lcod 123'5862 mlcod 123'5862 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:17.657 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.650+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5930 (115'2906,123'5930] local-lis/les=21/22 n=459 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5929 crt=123'5925 lcod 123'5927 mlcod 123'5927 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:17.657 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.654+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5932 (115'2906,123'5932] local-lis/les=21/22 n=457 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5929 crt=123'5929 lcod 123'5927 mlcod 123'5927 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:17.658 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:17.654+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6456 (117'3412,123'6456] local-lis/les=21/22 n=233 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6454 crt=123'6454 lcod 123'6452 mlcod 123'6452 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:17.659 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.654+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6167 (116'3115,123'6167] local-lis/les=21/22 n=501 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'6167 lcod 123'6165 mlcod 123'6165 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:17.665 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.662+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5868 (116'2808,123'5868] local-lis/les=21/22 n=527 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5868 lcod 123'5866 mlcod 123'5866 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:17.668 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.662+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5934 (115'2906,123'5934] local-lis/les=21/22 n=455 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5934 lcod 123'5932 mlcod 123'5932 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:17.668 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.662+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5870 (116'2808,123'5870] local-lis/les=21/22 n=525 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5868 crt=123'5868 lcod 123'5866 mlcod 123'5866 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:17.670 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:17.666+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6459 (117'3412,123'6459] local-lis/les=21/22 n=230 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6458 crt=123'6454 lcod 123'6456 mlcod 123'6456 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:17.671 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:17.657+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5806 (115'2706,123'5806] local-lis/les=21/22 n=715 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=123'5806 lcod 123'5804 mlcod 123'5804 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:17.674 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:17.666+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6273 (117'3208,123'6273] local-lis/les=21/22 n=192 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'6272 crt=123'6270 lcod 123'6270 mlcod 123'6270 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:17.675 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:17.669+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5808 (115'2806,123'5808] local-lis/les=21/22 n=713 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5806 crt=123'5806 lcod 123'5804 mlcod 123'5804 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:17.676 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:17.670+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6461 (117'3412,123'6461] local-lis/les=21/22 n=228 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6458 crt=123'6458 lcod 123'6456 mlcod 123'6456 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:17.676 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.670+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5936 (115'2906,123'5936] local-lis/les=21/22 n=453 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5934 crt=123'5934 lcod 123'5932 mlcod 123'5932 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:17.678 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.674+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'6059 (116'3004,123'6059] local-lis/les=21/22 n=160 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'6059 lcod 123'6057 mlcod 123'6057 active+clean+snaptrim trimq=[210~18] ps=[1fe~12]] removing snap head 2019-12-05T06:51:17.703 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:17.685+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5810 (115'2806,123'5810] local-lis/les=21/22 n=711 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=123'5810 lcod 123'5808 mlcod 123'5808 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:17.703 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.678+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'6061 (116'3004,123'6061] local-lis/les=21/22 n=158 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'6059 crt=123'6059 lcod 123'6057 mlcod 123'6057 active+clean+snaptrim trimq=[210~18] ps=[1fe~12]] removing snap head 2019-12-05T06:51:17.704 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.682+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6171 (116'3115,123'6171] local-lis/les=21/22 n=498 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'6171 lcod 123'6169 mlcod 123'6169 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:17.704 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.682+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6173 (116'3115,123'6173] local-lis/les=21/22 n=496 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'6171 crt=123'6171 lcod 123'6169 mlcod 123'6169 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:17.704 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.694+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6176 (116'3115,123'6176] local-lis/les=21/22 n=493 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'6175 crt=123'6171 lcod 123'6173 mlcod 123'6173 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:17.704 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:17.678+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6463 (117'3412,123'6463] local-lis/les=21/22 n=226 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'6463 lcod 123'6461 mlcod 123'6461 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:17.704 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:17.682+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6465 (117'3412,123'6465] local-lis/les=21/22 n=224 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6463 crt=123'6463 lcod 123'6461 mlcod 123'6461 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:17.705 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.678+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5872 (116'2808,123'5872] local-lis/les=21/22 n=523 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5872 lcod 123'5870 mlcod 123'5870 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:17.705 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.682+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5940 (115'2906,123'5940] local-lis/les=21/22 n=449 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5939 crt=123'5936 lcod 123'5938 mlcod 123'5938 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:17.705 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.686+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5874 (116'2808,123'5874] local-lis/les=21/22 n=521 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5872 crt=123'5872 lcod 123'5870 mlcod 123'5870 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:17.705 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.686+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5942 (115'2906,123'5942] local-lis/les=21/22 n=447 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5939 crt=123'5939 lcod 123'5938 mlcod 123'5938 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:17.705 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.694+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5876 (116'2808,123'5876] local-lis/les=21/22 n=519 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5876 lcod 123'5874 mlcod 123'5874 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:17.705 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.698+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5878 (116'2808,123'5878] local-lis/les=21/22 n=517 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5876 crt=123'5876 lcod 123'5874 mlcod 123'5874 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:17.706 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:17.685+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5812 (115'2806,123'5812] local-lis/les=21/22 n=709 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5810 crt=123'5810 lcod 123'5808 mlcod 123'5808 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:17.706 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:17.697+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5814 (115'2806,123'5814] local-lis/les=21/22 n=707 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=123'5814 lcod 123'5812 mlcod 123'5812 active+clean+snaptrim trimq=[201~27] ps=[1fe~3]] removing snap head 2019-12-05T06:51:17.707 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.698+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6178 (116'3115,123'6178] local-lis/les=21/22 n=491 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'6175 crt=123'6175 lcod 123'6173 mlcod 123'6173 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:17.707 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.698+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5944 (115'2906,123'5944] local-lis/les=21/22 n=445 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5944 lcod 123'5942 mlcod 123'5942 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:17.707 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.702+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5946 (115'2906,123'5946] local-lis/les=21/22 n=443 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5944 crt=123'5944 lcod 123'5942 mlcod 123'5942 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:17.707 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:17.698+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6276 (117'3208,123'6276] local-lis/les=21/22 n=189 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'6275 crt=123'6273 lcod 123'6273 mlcod 123'6273 active+clean+snaptrim trimq=[210~18] ps=[1fe~12]] removing snap head 2019-12-05T06:51:17.708 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:17.702+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6278 (117'3208,123'6278] local-lis/les=21/22 n=187 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'6275 crt=123'6275 lcod 123'6273 mlcod 123'6273 active+clean+snaptrim trimq=[210~18] ps=[1fe~12]] removing snap head 2019-12-05T06:51:17.713 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.710+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5880 (116'2808,123'5880] local-lis/les=21/22 n=515 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5880 lcod 123'5878 mlcod 123'5878 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:17.714 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:17.709+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5816 (115'2806,123'5816] local-lis/les=21/22 n=705 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=123'5816 lcod 123'5814 mlcod 123'5814 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:17.718 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.710+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5948 (115'2906,123'5948] local-lis/les=21/22 n=441 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5948 lcod 123'5946 mlcod 123'5946 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:17.718 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.714+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5882 (116'2808,123'5882] local-lis/les=21/22 n=513 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5880 crt=123'5880 lcod 123'5878 mlcod 123'5878 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:17.719 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.710+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6180 (116'3115,123'6180] local-lis/les=21/22 n=489 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'6180 lcod 123'6178 mlcod 123'6178 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:17.719 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:17.714+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6281 (117'3208,123'6281] local-lis/les=21/22 n=184 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'6280 crt=123'6275 lcod 123'6278 mlcod 123'6278 active+clean+snaptrim trimq=[210~18] ps=[1fe~12]] removing snap head 2019-12-05T06:51:17.719 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:17.709+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5818 (115'2806,123'5818] local-lis/les=21/22 n=703 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5816 crt=123'5816 lcod 123'5814 mlcod 123'5814 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:17.719 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:17.714+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6283 (117'3208,123'6283] local-lis/les=21/22 n=182 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'6280 crt=123'6280 lcod 123'6278 mlcod 123'6278 active+clean+snaptrim trimq=[210~18] ps=[1fe~12]] removing snap head 2019-12-05T06:51:17.720 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.718+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5950 (115'2906,123'5950] local-lis/les=21/22 n=439 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5948 crt=123'5948 lcod 123'5946 mlcod 123'5946 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:17.723 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.714+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6182 (116'3115,123'6182] local-lis/les=21/22 n=487 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'6180 crt=123'6180 lcod 123'6178 mlcod 123'6178 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:17.730 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.722+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'6063 (116'3004,123'6063] local-lis/les=21/22 n=156 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'6063 lcod 123'6061 mlcod 123'6061 active+clean+snaptrim trimq=[210~18] ps=[1fe~12]] removing snap head 2019-12-05T06:51:17.732 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.730+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'6065 (116'3004,123'6065] local-lis/les=21/22 n=154 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'6063 crt=123'6063 lcod 123'6061 mlcod 123'6061 active+clean+snaptrim trimq=[210~18] ps=[1fe~12]] removing snap head 2019-12-05T06:51:17.734 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:17.729+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5820 (115'2806,123'5820] local-lis/les=21/22 n=701 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=123'5820 lcod 123'5818 mlcod 123'5818 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:17.735 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.730+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6184 (116'3115,123'6184] local-lis/les=21/22 n=485 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'6184 lcod 123'6182 mlcod 123'6182 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:17.736 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:17.729+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5822 (115'2806,123'5822] local-lis/les=21/22 n=699 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5820 crt=123'5820 lcod 123'5818 mlcod 123'5818 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:17.740 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.738+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6186 (116'3115,123'6186] local-lis/les=21/22 n=483 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'6184 crt=123'6184 lcod 123'6182 mlcod 123'6182 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:17.743 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.738+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5886 (116'2808,123'5886] local-lis/les=21/22 n=509 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5885 crt=123'5885 lcod 123'5884 mlcod 123'5884 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:17.743 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.738+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5952 (115'2906,123'5952] local-lis/les=21/22 n=437 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5952 lcod 123'5950 mlcod 123'5950 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:17.744 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:17.742+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6467 (117'3412,123'6467] local-lis/les=21/22 n=222 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'6467 lcod 123'6465 mlcod 123'6465 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:17.746 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.742+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5954 (115'2906,123'5954] local-lis/les=21/22 n=435 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5952 crt=123'5952 lcod 123'5950 mlcod 123'5950 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:17.746 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.742+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5888 (116'2808,123'5888] local-lis/les=21/22 n=507 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5885 crt=123'5885 lcod 123'5884 mlcod 123'5884 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:17.746 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:17.742+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6469 (117'3412,123'6469] local-lis/les=21/22 n=220 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6467 crt=123'6467 lcod 123'6465 mlcod 123'6465 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:17.748 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.742+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'6067 (116'3004,123'6067] local-lis/les=21/22 n=152 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'6067 lcod 123'6065 mlcod 123'6065 active+clean+snaptrim trimq=[210~18] ps=[1fe~12]] removing snap head 2019-12-05T06:51:17.749 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.746+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'6069 (116'3004,123'6069] local-lis/les=21/22 n=150 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'6067 crt=123'6067 lcod 123'6065 mlcod 123'6065 active+clean+snaptrim trimq=[210~18] ps=[1fe~12]] removing snap head 2019-12-05T06:51:17.754 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:17.745+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5824 (115'2806,123'5824] local-lis/les=21/22 n=697 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=123'5824 lcod 123'5822 mlcod 123'5822 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:17.755 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.750+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6188 (116'3115,123'6188] local-lis/les=21/22 n=481 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'6188 lcod 123'6186 mlcod 123'6186 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:17.758 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.754+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5956 (115'2906,123'5956] local-lis/les=21/22 n=433 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5956 lcod 123'5954 mlcod 123'5954 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:17.758 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:17.749+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5826 (115'2806,123'5826] local-lis/les=21/22 n=695 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5824 crt=123'5824 lcod 123'5822 mlcod 123'5822 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:17.760 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:17.754+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6287 (117'3208,123'6287] local-lis/les=21/22 n=178 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'6286 crt=123'6286 lcod 123'6285 mlcod 123'6285 active+clean+snaptrim trimq=[210~18] ps=[1fe~12]] removing snap head 2019-12-05T06:51:17.763 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.758+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5958 (115'2906,123'5958] local-lis/les=21/22 n=431 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5956 crt=123'5956 lcod 123'5954 mlcod 123'5954 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:17.763 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.758+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5891 (116'2808,123'5891] local-lis/les=21/22 n=504 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5890 crt=123'5888 lcod 123'5888 mlcod 123'5888 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:17.764 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.758+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6190 (116'3115,123'6190] local-lis/les=21/22 n=479 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'6188 crt=123'6188 lcod 123'6186 mlcod 123'6186 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:17.765 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.762+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5893 (116'2808,123'5893] local-lis/les=21/22 n=502 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5890 crt=123'5890 lcod 123'5888 mlcod 123'5888 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:17.766 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:17.758+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6289 (117'3208,123'6289] local-lis/les=21/22 n=176 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'6286 crt=123'6286 lcod 123'6285 mlcod 123'6285 active+clean+snaptrim trimq=[210~18] ps=[1fe~12]] removing snap head 2019-12-05T06:51:17.769 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.766+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'6071 (116'3004,123'6071] local-lis/les=21/22 n=148 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'6071 lcod 123'6069 mlcod 123'6069 active+clean+snaptrim trimq=[210~18] ps=[1fe~12]] removing snap head 2019-12-05T06:51:17.770 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:17.765+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5828 (115'2806,123'5828] local-lis/les=21/22 n=693 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=123'5828 lcod 123'5826 mlcod 123'5826 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:17.772 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:17.766+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6471 (117'3412,123'6471] local-lis/les=21/22 n=218 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'6471 lcod 123'6469 mlcod 123'6469 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:17.773 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:17.770+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6473 (117'3412,123'6473] local-lis/les=21/22 n=216 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6471 crt=123'6471 lcod 123'6469 mlcod 123'6469 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:17.774 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:17.765+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5830 (115'2806,123'5830] local-lis/les=21/22 n=691 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5828 crt=123'5828 lcod 123'5826 mlcod 123'5826 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:17.775 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.770+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5960 (115'2906,123'5960] local-lis/les=21/22 n=429 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5960 lcod 123'5958 mlcod 123'5958 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:17.777 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.774+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5962 (115'2906,123'5962] local-lis/les=21/22 n=427 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5960 crt=123'5960 lcod 123'5958 mlcod 123'5958 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:17.778 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.770+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'6073 (116'3004,123'6073] local-lis/les=21/22 n=146 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'6071 crt=123'6071 lcod 123'6069 mlcod 123'6069 active+clean+snaptrim trimq=[210~18] ps=[1fe~12]] removing snap head 2019-12-05T06:51:17.778 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.774+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6192 (116'3115,123'6192] local-lis/les=21/22 n=477 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'6192 lcod 123'6190 mlcod 123'6190 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:17.779 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.774+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6194 (116'3115,123'6194] local-lis/les=21/22 n=475 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'6192 crt=123'6192 lcod 123'6190 mlcod 123'6190 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:17.784 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.782+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5895 (116'2808,123'5895] local-lis/les=21/22 n=500 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5895 lcod 123'5893 mlcod 123'5893 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:17.786 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.782+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5897 (116'2808,123'5897] local-lis/les=21/22 n=498 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5895 crt=123'5895 lcod 123'5893 mlcod 123'5893 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:17.786 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:17.777+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5832 (115'2806,123'5832] local-lis/les=21/22 n=689 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=123'5832 lcod 123'5830 mlcod 123'5830 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:17.787 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:17.781+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5834 (115'2806,123'5834] local-lis/les=21/22 n=687 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5832 crt=123'5832 lcod 123'5830 mlcod 123'5830 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:17.791 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.786+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6196 (116'3115,123'6196] local-lis/les=21/22 n=473 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'6196 lcod 123'6194 mlcod 123'6194 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:17.794 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.790+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6198 (116'3115,123'6198] local-lis/les=21/22 n=471 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'6196 crt=123'6196 lcod 123'6194 mlcod 123'6194 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:17.796 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:17.789+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5836 (115'2806,123'5836] local-lis/les=21/22 n=685 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=123'5836 lcod 123'5834 mlcod 123'5834 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:17.797 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.794+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5965 (115'2906,123'5965] local-lis/les=21/22 n=424 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5965 lcod 123'5964 mlcod 123'5964 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:17.797 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.794+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5900 (116'2808,123'5900] local-lis/les=21/22 n=495 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5899 crt=123'5895 lcod 123'5897 mlcod 123'5897 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:17.799 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.794+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5967 (115'2906,123'5967] local-lis/les=21/22 n=422 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5965 crt=123'5965 lcod 123'5964 mlcod 123'5964 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:17.800 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.794+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5902 (116'2808,123'5902] local-lis/les=21/22 n=493 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5899 crt=123'5899 lcod 123'5897 mlcod 123'5897 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:17.800 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:17.789+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5838 (115'2806,123'5838] local-lis/les=21/22 n=683 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5836 crt=123'5836 lcod 123'5834 mlcod 123'5834 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:17.807 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.802+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5904 (116'2808,123'5904] local-lis/les=21/22 n=491 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5904 lcod 123'5902 mlcod 123'5902 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:17.807 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:17.801+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5920 (116'2908,123'5920] local-lis/les=21/22 n=349 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5919 crt=123'5917 lcod 123'5917 mlcod 123'5917 active+clean+snaptrim trimq=[20a~1e] ps=[1fe~c]] removing snap head 2019-12-05T06:51:17.809 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.806+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5906 (116'2808,123'5906] local-lis/les=21/22 n=489 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5904 crt=123'5904 lcod 123'5902 mlcod 123'5902 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:17.811 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:17.801+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5922 (116'2908,123'5922] local-lis/les=21/22 n=347 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5919 crt=123'5919 lcod 123'5917 mlcod 123'5917 active+clean+snaptrim trimq=[20a~1e] ps=[1fe~c]] removing snap head 2019-12-05T06:51:17.812 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.806+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5969 (115'2906,123'5969] local-lis/les=21/22 n=420 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5969 lcod 123'5967 mlcod 123'5967 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:17.812 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.810+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5971 (115'2906,123'5971] local-lis/les=21/22 n=418 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5969 crt=123'5969 lcod 123'5967 mlcod 123'5967 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:17.817 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.814+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6201 (116'3115,123'6201] local-lis/les=21/22 n=468 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'6200 crt=123'6198 lcod 123'6198 mlcod 123'6198 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:17.818 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.814+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5908 (116'2808,123'5908] local-lis/les=21/22 n=487 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5908 lcod 123'5906 mlcod 123'5906 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:17.819 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:17.813+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5840 (115'2806,123'5840] local-lis/les=21/22 n=681 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=123'5840 lcod 123'5838 mlcod 123'5838 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:17.820 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.818+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6203 (116'3115,123'6203] local-lis/les=21/22 n=466 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'6200 crt=123'6200 lcod 123'6198 mlcod 123'6198 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:17.822 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:17.813+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5842 (115'2806,123'5842] local-lis/les=21/22 n=679 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5840 crt=123'5840 lcod 123'5838 mlcod 123'5838 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:17.829 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:17.821+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5845 (115'2806,123'5845] local-lis/les=21/22 n=676 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5844 crt=123'5840 lcod 123'5842 mlcod 123'5842 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:17.832 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:17.825+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5847 (115'2806,123'5847] local-lis/les=21/22 n=674 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5844 crt=123'5844 lcod 123'5842 mlcod 123'5842 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:17.833 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.830+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6205 (116'3115,123'6205] local-lis/les=21/22 n=464 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'6205 lcod 123'6203 mlcod 123'6203 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:17.835 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:17.830+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6207 (116'3115,123'6207] local-lis/les=21/22 n=462 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'6205 crt=123'6205 lcod 123'6203 mlcod 123'6203 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:17.836 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.834+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5973 (115'2906,123'5973] local-lis/les=21/22 n=416 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'5973 lcod 123'5971 mlcod 123'5971 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:17.838 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.834+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5975 (115'2906,123'5975] local-lis/les=21/22 n=414 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5973 crt=123'5973 lcod 123'5971 mlcod 123'5971 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:17.943 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.938+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5911 (116'2908,123'5911] local-lis/les=21/22 n=484 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5910 crt=123'5908 lcod 123'5908 mlcod 123'5908 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:17.945 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:17.942+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5913 (116'2908,123'5913] local-lis/les=21/22 n=482 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5910 crt=123'5910 lcod 123'5908 mlcod 123'5908 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:18.003 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:17.998+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6476 (117'3412,123'6476] local-lis/les=21/22 n=213 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6475 crt=123'6471 lcod 123'6473 mlcod 123'6473 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:18.007 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:18.002+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6478 (117'3412,123'6478] local-lis/les=21/22 n=211 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6475 crt=123'6475 lcod 123'6473 mlcod 123'6473 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:18.065 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:18.062+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6291 (117'3208,123'6291] local-lis/les=21/22 n=174 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'6291 lcod 123'6289 mlcod 123'6289 active+clean+snaptrim trimq=[210~18] ps=[1fe~12]] removing snap head 2019-12-05T06:51:18.065 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:18.062+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'6075 (116'3004,123'6075] local-lis/les=21/22 n=144 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'6075 lcod 123'6073 mlcod 123'6073 active+clean+snaptrim trimq=[210~18] ps=[1fe~12]] removing snap head 2019-12-05T06:51:18.067 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:18.062+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6293 (117'3208,123'6293] local-lis/les=21/22 n=172 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'6291 crt=123'6291 lcod 123'6289 mlcod 123'6289 active+clean+snaptrim trimq=[210~18] ps=[1fe~12]] removing snap head 2019-12-05T06:51:18.069 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:18.066+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'6077 (116'3004,123'6077] local-lis/les=21/22 n=142 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'6075 crt=123'6075 lcod 123'6073 mlcod 123'6073 active+clean+snaptrim trimq=[210~18] ps=[1fe~12]] removing snap head 2019-12-05T06:51:18.129 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:18.121+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5849 (115'2806,123'5849] local-lis/les=21/22 n=672 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=123'5849 lcod 123'5847 mlcod 123'5847 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:18.129 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:18.121+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5924 (116'2908,123'5924] local-lis/les=21/22 n=345 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5924 lcod 123'5922 mlcod 123'5922 active+clean+snaptrim trimq=[20a~1e] ps=[1fe~c]] removing snap head 2019-12-05T06:51:18.131 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:18.125+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5926 (116'2908,123'5926] local-lis/les=21/22 n=343 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5924 crt=123'5924 lcod 123'5922 mlcod 123'5922 active+clean+snaptrim trimq=[20a~1e] ps=[1fe~c]] removing snap head 2019-12-05T06:51:18.131 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:18.125+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5851 (115'2806,123'5851] local-lis/les=21/22 n=670 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5849 crt=123'5849 lcod 123'5847 mlcod 123'5847 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:18.136 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:18.134+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6209 (116'3115,123'6209] local-lis/les=21/22 n=460 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'6209 lcod 123'6207 mlcod 123'6207 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:18.137 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:18.134+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6480 (117'3412,123'6480] local-lis/les=21/22 n=209 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'6480 lcod 123'6478 mlcod 123'6478 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:18.140 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:18.134+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'6079 (116'3004,123'6079] local-lis/les=21/22 n=140 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'6079 lcod 123'6077 mlcod 123'6077 active+clean+snaptrim trimq=[210~18] ps=[1fe~12]] removing snap head 2019-12-05T06:51:18.140 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:18.134+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6482 (117'3412,123'6482] local-lis/les=21/22 n=207 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6480 crt=123'6480 lcod 123'6478 mlcod 123'6478 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:18.141 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:18.138+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'6081 (116'3004,123'6081] local-lis/les=21/22 n=138 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'6079 crt=123'6079 lcod 123'6077 mlcod 123'6077 active+clean+snaptrim trimq=[210~18] ps=[1fe~12]] removing snap head 2019-12-05T06:51:18.141 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:18.138+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5916 (116'2908,123'5916] local-lis/les=21/22 n=479 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5915 crt=123'5911 lcod 123'5913 mlcod 123'5913 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:18.143 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:18.138+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6295 (117'3208,123'6295] local-lis/les=21/22 n=170 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'6295 lcod 123'6293 mlcod 123'6293 active+clean+snaptrim trimq=[210~18] ps=[1fe~12]] removing snap head 2019-12-05T06:51:18.145 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:18.137+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5928 (116'2908,123'5928] local-lis/les=21/22 n=341 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5928 lcod 123'5926 mlcod 123'5926 active+clean+snaptrim trimq=[20a~1e] ps=[1fe~c]] removing snap head 2019-12-05T06:51:18.146 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:18.142+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5918 (116'2908,123'5918] local-lis/les=21/22 n=477 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5915 crt=123'5915 lcod 123'5913 mlcod 123'5913 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:18.146 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:18.137+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5930 (116'2908,123'5930] local-lis/les=21/22 n=339 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5928 crt=123'5928 lcod 123'5926 mlcod 123'5926 active+clean+snaptrim trimq=[20a~1e] ps=[1fe~c]] removing snap head 2019-12-05T06:51:18.147 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:18.141+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5854 (115'2806,123'5854] local-lis/les=21/22 n=667 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5853 crt=123'5849 lcod 123'5851 mlcod 123'5851 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:18.149 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:18.142+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6297 (117'3208,123'6297] local-lis/les=21/22 n=168 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'6295 crt=123'6295 lcod 123'6293 mlcod 123'6293 active+clean+snaptrim trimq=[210~18] ps=[1fe~12]] removing snap head 2019-12-05T06:51:18.158 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:18.150+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6213 (116'3115,123'6213] local-lis/les=21/22 n=457 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'6213 lcod 123'6211 mlcod 123'6211 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:18.159 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:18.149+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5932 (116'2908,123'5932] local-lis/les=21/22 n=337 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5932 lcod 123'5930 mlcod 123'5930 active+clean+snaptrim trimq=[20a~1e] ps=[1fe~c]] removing snap head 2019-12-05T06:51:18.160 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:18.154+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6215 (116'3115,123'6215] local-lis/les=21/22 n=455 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'6213 crt=123'6213 lcod 123'6211 mlcod 123'6211 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:18.160 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:18.154+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5978 (115'2906,123'5978] local-lis/les=21/22 n=411 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5977 crt=123'5975 lcod 123'5975 mlcod 123'5975 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:18.160 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:18.158+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'6084 (116'3004,123'6084] local-lis/les=21/22 n=135 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'6083 crt=123'6081 lcod 123'6081 mlcod 123'6081 active+clean+snaptrim trimq=[210~18] ps=[1fe~12]] removing snap head 2019-12-05T06:51:18.162 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:18.158+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5980 (115'2906,123'5980] local-lis/les=21/22 n=409 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5977 crt=123'5977 lcod 123'5975 mlcod 123'5975 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:18.165 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:18.158+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'6086 (116'3004,123'6086] local-lis/les=21/22 n=133 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'6083 crt=123'6083 lcod 123'6081 mlcod 123'6081 active+clean+snaptrim trimq=[210~18] ps=[1fe~12]] removing snap head 2019-12-05T06:51:18.167 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:18.161+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5858 (115'2806,123'5858] local-lis/les=21/22 n=664 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=123'5858 lcod 123'5856 mlcod 123'5856 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:18.168 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:18.161+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5934 (116'2908,123'5934] local-lis/les=21/22 n=335 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5932 crt=123'5932 lcod 123'5930 mlcod 123'5930 active+clean+snaptrim trimq=[20a~1e] ps=[1fe~c]] removing snap head 2019-12-05T06:51:18.172 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:18.165+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5860 (115'2806,123'5860] local-lis/les=21/22 n=662 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5858 crt=123'5858 lcod 123'5856 mlcod 123'5856 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:18.175 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:18.170+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5983 (115'2906,123'5983] local-lis/les=21/22 n=406 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5982 crt=123'5978 lcod 123'5980 mlcod 123'5980 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:18.177 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:18.174+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5985 (115'2906,123'5985] local-lis/les=21/22 n=404 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5982 crt=123'5982 lcod 123'5980 mlcod 123'5980 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:18.178 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:18.174+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5920 (116'2908,123'5920] local-lis/les=21/22 n=475 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5920 lcod 123'5918 mlcod 123'5918 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:18.179 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:18.173+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5936 (116'2908,123'5936] local-lis/les=21/22 n=333 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5936 lcod 123'5934 mlcod 123'5934 active+clean+snaptrim trimq=[20a~1e] ps=[1fe~c]] removing snap head 2019-12-05T06:51:18.180 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:18.178+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5922 (116'2908,123'5922] local-lis/les=21/22 n=473 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5920 crt=123'5920 lcod 123'5918 mlcod 123'5918 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:18.181 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:18.178+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6218 (117'3215,123'6218] local-lis/les=21/22 n=452 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'6217 crt=123'6215 lcod 123'6215 mlcod 123'6215 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:18.183 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:18.173+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5938 (116'2908,123'5938] local-lis/les=21/22 n=331 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5936 crt=123'5936 lcod 123'5934 mlcod 123'5934 active+clean+snaptrim trimq=[20a~1e] ps=[1fe~c]] removing snap head 2019-12-05T06:51:18.185 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:18.182+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6220 (117'3215,123'6220] local-lis/les=21/22 n=450 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'6217 crt=123'6217 lcod 123'6215 mlcod 123'6215 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:18.187 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:18.181+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5862 (115'2806,123'5862] local-lis/les=21/22 n=660 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=123'5862 lcod 123'5860 mlcod 123'5860 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:18.190 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:18.181+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5864 (115'2806,123'5864] local-lis/les=21/22 n=658 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5862 crt=123'5862 lcod 123'5860 mlcod 123'5860 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:18.194 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:18.185+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5940 (116'2908,123'5940] local-lis/les=21/22 n=329 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5940 lcod 123'5938 mlcod 123'5938 active+clean+snaptrim trimq=[20b~1d] ps=[1fe~d]] removing snap head 2019-12-05T06:51:18.198 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:18.194+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5925 (116'2908,123'5925] local-lis/les=21/22 n=470 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5924 crt=123'5920 lcod 123'5922 mlcod 123'5922 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:18.198 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:18.189+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5867 (115'2806,123'5867] local-lis/les=21/22 n=655 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5866 crt=123'5862 lcod 123'5864 mlcod 123'5864 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:18.198 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:18.189+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5942 (116'2908,123'5942] local-lis/les=21/22 n=327 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5940 crt=123'5940 lcod 123'5938 mlcod 123'5938 active+clean+snaptrim trimq=[20b~1d] ps=[1fe~d]] removing snap head 2019-12-05T06:51:18.199 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:18.194+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6222 (117'3215,123'6222] local-lis/les=21/22 n=448 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'6222 lcod 123'6220 mlcod 123'6220 active+clean+snaptrim trimq=[208~20] ps=[1fe~a]] removing snap head 2019-12-05T06:51:18.199 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:18.194+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5927 (116'2908,123'5927] local-lis/les=21/22 n=468 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5924 crt=123'5924 lcod 123'5922 mlcod 123'5922 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:18.202 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:18.193+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5869 (115'2806,123'5869] local-lis/les=21/22 n=653 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5866 crt=123'5866 lcod 123'5864 mlcod 123'5864 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:18.204 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:18.198+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6224 (117'3215,123'6224] local-lis/les=21/22 n=446 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'6222 crt=123'6222 lcod 123'6220 mlcod 123'6220 active+clean+snaptrim trimq=[208~20] ps=[1fe~a]] removing snap head 2019-12-05T06:51:18.207 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:18.202+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5988 (115'2906,123'5988] local-lis/les=21/22 n=401 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5987 crt=123'5985 lcod 123'5985 mlcod 123'5985 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:18.209 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:18.206+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5990 (115'2906,123'5990] local-lis/les=21/22 n=399 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5987 crt=123'5987 lcod 123'5985 mlcod 123'5985 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:18.210 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:18.206+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5929 (116'2908,123'5929] local-lis/les=21/22 n=466 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5929 lcod 123'5927 mlcod 123'5927 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:18.212 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:18.206+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6484 (117'3412,123'6484] local-lis/les=21/22 n=205 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'6484 lcod 123'6482 mlcod 123'6482 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:18.213 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:18.210+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5931 (116'2908,123'5931] local-lis/les=21/22 n=464 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5929 crt=123'5929 lcod 123'5927 mlcod 123'5927 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:18.214 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:18.205+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5944 (116'2908,123'5944] local-lis/les=21/22 n=325 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5944 lcod 123'5942 mlcod 123'5942 active+clean+snaptrim trimq=[20b~1d] ps=[1fe~d]] removing snap head 2019-12-05T06:51:18.214 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:18.210+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6226 (117'3215,123'6226] local-lis/les=21/22 n=444 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'6226 lcod 123'6224 mlcod 123'6224 active+clean+snaptrim trimq=[208~20] ps=[1fe~a]] removing snap head 2019-12-05T06:51:18.216 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:18.209+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5946 (116'2908,123'5946] local-lis/les=21/22 n=323 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5944 crt=123'5944 lcod 123'5942 mlcod 123'5942 active+clean+snaptrim trimq=[20b~1d] ps=[1fe~d]] removing snap head 2019-12-05T06:51:18.217 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:18.209+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5872 (115'2806,123'5872] local-lis/les=21/22 n=650 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5871 crt=123'5869 lcod 123'5869 mlcod 123'5869 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:18.218 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:18.214+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6228 (117'3215,123'6228] local-lis/les=21/22 n=442 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'6226 crt=123'6226 lcod 123'6224 mlcod 123'6224 active+clean+snaptrim trimq=[208~20] ps=[1fe~a]] removing snap head 2019-12-05T06:51:18.223 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:18.213+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5874 (115'2806,123'5874] local-lis/les=21/22 n=648 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5871 crt=123'5871 lcod 123'5869 mlcod 123'5869 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:18.224 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:18.210+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6486 (117'3412,123'6486] local-lis/les=21/22 n=203 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6484 crt=123'6484 lcod 123'6482 mlcod 123'6482 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:18.225 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:18.214+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6299 (117'3208,123'6299] local-lis/les=21/22 n=166 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'6299 lcod 123'6297 mlcod 123'6297 active+clean+snaptrim trimq=[210~18] ps=[1fe~12]] removing snap head 2019-12-05T06:51:18.225 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:18.222+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6301 (117'3208,123'6301] local-lis/les=21/22 n=164 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'6299 crt=123'6299 lcod 123'6297 mlcod 123'6297 active+clean+snaptrim trimq=[210~18] ps=[1fe~12]] removing snap head 2019-12-05T06:51:18.228 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:18.226+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'6088 (116'3004,123'6088] local-lis/les=21/22 n=131 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'6088 lcod 123'6086 mlcod 123'6086 active+clean+snaptrim trimq=[210~18] ps=[1fe~12]] removing snap head 2019-12-05T06:51:18.229 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:18.221+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5949 (116'2908,123'5949] local-lis/les=21/22 n=320 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5948 crt=123'5946 lcod 123'5946 mlcod 123'5946 active+clean+snaptrim trimq=[20b~1d] ps=[1fe~d]] removing snap head 2019-12-05T06:51:18.230 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:18.226+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'6090 (116'3004,123'6090] local-lis/les=21/22 n=129 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'6088 crt=123'6088 lcod 123'6086 mlcod 123'6086 active+clean+snaptrim trimq=[210~18] ps=[1fe~12]] removing snap head 2019-12-05T06:51:18.232 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:18.226+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6231 (117'3215,123'6231] local-lis/les=21/22 n=439 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'6230 crt=123'6226 lcod 123'6228 mlcod 123'6228 active+clean+snaptrim trimq=[208~20] ps=[1fe~a]] removing snap head 2019-12-05T06:51:18.232 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:18.226+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5933 (116'2908,123'5933] local-lis/les=21/22 n=462 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5933 lcod 123'5931 mlcod 123'5931 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:18.250 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:18.225+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5951 (116'2908,123'5951] local-lis/les=21/22 n=318 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5948 crt=123'5948 lcod 123'5946 mlcod 123'5946 active+clean+snaptrim trimq=[20b~1d] ps=[1fe~d]] removing snap head 2019-12-05T06:51:18.250 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:18.229+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5876 (115'2806,123'5876] local-lis/les=21/22 n=646 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=123'5876 lcod 123'5874 mlcod 123'5874 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:18.250 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:18.233+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5878 (115'2806,123'5878] local-lis/les=21/22 n=644 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5876 crt=123'5876 lcod 123'5874 mlcod 123'5874 active+clean+snaptrim trimq=[202~26] ps=[1fe~4]] removing snap head 2019-12-05T06:51:18.251 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:18.230+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5935 (116'2908,123'5935] local-lis/les=21/22 n=460 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5933 crt=123'5933 lcod 123'5931 mlcod 123'5931 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:18.251 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:18.242+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5993 (115'2906,123'5993] local-lis/les=21/22 n=396 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5992 crt=123'5990 lcod 123'5990 mlcod 123'5990 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:18.251 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:18.242+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5938 (116'2908,123'5938] local-lis/les=21/22 n=457 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5937 crt=123'5933 lcod 123'5935 mlcod 123'5935 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:18.251 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:18.246+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5940 (116'2908,123'5940] local-lis/les=21/22 n=455 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5937 crt=123'5937 lcod 123'5935 mlcod 123'5935 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:18.252 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:18.226+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6488 (117'3412,123'6488] local-lis/les=21/22 n=201 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'6488 lcod 123'6486 mlcod 123'6486 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:18.252 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:18.230+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6490 (117'3412,123'6490] local-lis/les=21/22 n=199 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6488 crt=123'6488 lcod 123'6486 mlcod 123'6486 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:18.252 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:18.238+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6303 (117'3208,123'6303] local-lis/les=21/22 n=162 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'6303 lcod 123'6301 mlcod 123'6301 active+clean+snaptrim trimq=[210~18] ps=[1fe~12]] removing snap head 2019-12-05T06:51:18.252 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:18.242+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6305 (117'3208,123'6305] local-lis/les=21/22 n=160 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'6303 crt=123'6303 lcod 123'6301 mlcod 123'6301 active+clean+snaptrim trimq=[210~18] ps=[1fe~12]] removing snap head 2019-12-05T06:51:18.252 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:18.230+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6233 (117'3215,123'6233] local-lis/les=21/22 n=437 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'6230 crt=123'6230 lcod 123'6228 mlcod 123'6228 active+clean+snaptrim trimq=[208~20] ps=[1fe~a]] removing snap head 2019-12-05T06:51:18.253 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:18.246+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'6092 (116'3004,123'6092] local-lis/les=21/22 n=127 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'6092 lcod 123'6090 mlcod 123'6090 active+clean+snaptrim trimq=[210~18] ps=[1fe~12]] removing snap head 2019-12-05T06:51:18.254 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:18.250+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6492 (117'3412,123'6492] local-lis/les=21/22 n=197 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'6492 lcod 123'6490 mlcod 123'6490 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:18.254 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:18.246+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'6094 (116'3004,123'6094] local-lis/les=21/22 n=125 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'6092 crt=123'6092 lcod 123'6090 mlcod 123'6090 active+clean+snaptrim trimq=[210~18] ps=[1fe~12]] removing snap head 2019-12-05T06:51:18.254 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:18.250+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5995 (115'2906,123'5995] local-lis/les=21/22 n=394 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5992 crt=123'5992 lcod 123'5990 mlcod 123'5990 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:18.255 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:18.245+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5953 (116'2908,123'5953] local-lis/les=21/22 n=316 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5953 lcod 123'5951 mlcod 123'5951 active+clean+snaptrim trimq=[20b~1d] ps=[1fe~d]] removing snap head 2019-12-05T06:51:18.255 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:18.250+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6494 (117'3412,123'6494] local-lis/les=21/22 n=195 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6492 crt=123'6492 lcod 123'6490 mlcod 123'6490 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:18.270 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:18.254+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6235 (117'3215,123'6235] local-lis/les=21/22 n=435 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'6235 lcod 123'6233 mlcod 123'6233 active+clean+snaptrim trimq=[208~20] ps=[1fe~a]] removing snap head 2019-12-05T06:51:18.270 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:18.258+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6237 (117'3215,123'6237] local-lis/les=21/22 n=433 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'6235 crt=123'6235 lcod 123'6233 mlcod 123'6233 active+clean+snaptrim trimq=[208~20] ps=[1fe~a]] removing snap head 2019-12-05T06:51:18.270 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:18.254+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6307 (117'3208,123'6307] local-lis/les=21/22 n=158 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'6307 lcod 123'6305 mlcod 123'6305 active+clean+snaptrim trimq=[210~18] ps=[1fe~12]] removing snap head 2019-12-05T06:51:18.271 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:18.262+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6309 (117'3208,123'6309] local-lis/les=21/22 n=156 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'6307 crt=123'6307 lcod 123'6305 mlcod 123'6305 active+clean+snaptrim trimq=[210~18] ps=[1fe~12]] removing snap head 2019-12-05T06:51:18.271 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:18.249+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5955 (116'2908,123'5955] local-lis/les=21/22 n=314 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5953 crt=123'5953 lcod 123'5951 mlcod 123'5951 active+clean+snaptrim trimq=[20b~1d] ps=[1fe~d]] removing snap head 2019-12-05T06:51:18.271 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:18.257+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5880 (115'2806,123'5880] local-lis/les=21/22 n=642 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=123'5880 lcod 123'5878 mlcod 123'5878 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:18.271 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:18.261+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5882 (115'2806,123'5882] local-lis/les=21/22 n=640 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5880 crt=123'5880 lcod 123'5878 mlcod 123'5878 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:18.273 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:18.262+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6497 (117'3412,123'6497] local-lis/les=21/22 n=192 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6496 crt=123'6494 lcod 123'6494 mlcod 123'6494 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:18.273 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:18.266+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5942 (116'2908,123'5942] local-lis/les=21/22 n=453 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5942 lcod 123'5940 mlcod 123'5940 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:18.273 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:18.270+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5944 (116'2908,123'5944] local-lis/les=21/22 n=451 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5942 crt=123'5942 lcod 123'5940 mlcod 123'5940 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:18.274 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:18.262+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'6096 (116'3004,123'6096] local-lis/les=21/22 n=123 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'6096 lcod 123'6094 mlcod 123'6094 active+clean+snaptrim trimq=[210~18] ps=[1fe~12]] removing snap head 2019-12-05T06:51:18.274 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:18.262+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'6098 (116'3004,123'6098] local-lis/les=21/22 n=121 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'6096 crt=123'6096 lcod 123'6094 mlcod 123'6094 active+clean+snaptrim trimq=[210~18] ps=[1fe~12]] removing snap head 2019-12-05T06:51:18.274 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:18.270+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6499 (117'3412,123'6499] local-lis/les=21/22 n=190 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6496 crt=123'6496 lcod 123'6494 mlcod 123'6494 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:18.278 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:18.274+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6311 (118'3309,123'6311] local-lis/les=21/22 n=154 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'6311 lcod 123'6309 mlcod 123'6309 active+clean+snaptrim trimq=[210~18] ps=[1fe~12]] removing snap head 2019-12-05T06:51:18.278 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:18.273+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5957 (116'2908,123'5957] local-lis/les=21/22 n=312 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5957 lcod 123'5955 mlcod 123'5955 active+clean+snaptrim trimq=[20b~1d] ps=[1fe~d]] removing snap head 2019-12-05T06:51:18.280 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:18.273+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5959 (116'2908,123'5959] local-lis/les=21/22 n=310 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5957 crt=123'5957 lcod 123'5955 mlcod 123'5955 active+clean+snaptrim trimq=[20b~1d] ps=[1fe~d]] removing snap head 2019-12-05T06:51:18.282 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:18.278+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6241 (117'3215,123'6241] local-lis/les=21/22 n=430 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'6239 crt=123'6239 lcod 123'6237 mlcod 123'6237 active+clean+snaptrim trimq=[208~20] ps=[1fe~a]] removing snap head 2019-12-05T06:51:18.286 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:18.278+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6313 (118'3309,123'6313] local-lis/les=21/22 n=152 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'6311 crt=123'6311 lcod 123'6309 mlcod 123'6309 active+clean+snaptrim trimq=[210~18] ps=[1fe~12]] removing snap head 2019-12-05T06:51:18.286 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:18.278+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5946 (116'2908,123'5946] local-lis/les=21/22 n=449 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5946 lcod 123'5944 mlcod 123'5944 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:18.286 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:18.282+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5948 (116'2908,123'5948] local-lis/les=21/22 n=447 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5946 crt=123'5946 lcod 123'5944 mlcod 123'5944 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:18.287 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:18.281+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5884 (115'2806,123'5884] local-lis/les=21/22 n=638 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=123'5884 lcod 123'5882 mlcod 123'5882 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:18.289 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:18.286+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'6101 (116'3004,123'6101] local-lis/les=21/22 n=118 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'6101 lcod 123'6100 mlcod 123'6100 active+clean+snaptrim trimq=[210~18] ps=[1fe~12]] removing snap head 2019-12-05T06:51:18.289 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:18.286+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6501 (117'3412,123'6501] local-lis/les=21/22 n=188 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'6501 lcod 123'6499 mlcod 123'6499 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:18.290 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:18.281+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5886 (115'2806,123'5886] local-lis/les=21/22 n=636 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5884 crt=123'5884 lcod 123'5882 mlcod 123'5882 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:18.293 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:18.290+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6503 (117'3412,123'6503] local-lis/les=21/22 n=186 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6501 crt=123'6501 lcod 123'6499 mlcod 123'6499 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:18.294 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:18.290+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'6103 (116'3004,123'6103] local-lis/les=21/22 n=116 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'6101 crt=123'6101 lcod 123'6100 mlcod 123'6100 active+clean+snaptrim trimq=[210~18] ps=[1fe~12]] removing snap head 2019-12-05T06:51:18.294 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:18.290+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6315 (118'3309,123'6315] local-lis/les=21/22 n=150 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'6315 lcod 123'6313 mlcod 123'6313 active+clean+snaptrim trimq=[210~18] ps=[1fe~12]] removing snap head 2019-12-05T06:51:18.298 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:18.293+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5962 (116'2908,123'5962] local-lis/les=21/22 n=307 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5961 crt=123'5957 lcod 123'5959 mlcod 123'5959 active+clean+snaptrim trimq=[20b~1d] ps=[1fe~d]] removing snap head 2019-12-05T06:51:18.299 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:18.294+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6317 (118'3309,123'6317] local-lis/les=21/22 n=148 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'6315 crt=123'6315 lcod 123'6313 mlcod 123'6313 active+clean+snaptrim trimq=[210~18] ps=[1fe~12]] removing snap head 2019-12-05T06:51:18.299 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:18.294+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6506 (117'3412,123'6506] local-lis/les=21/22 n=183 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6505 crt=123'6501 lcod 123'6503 mlcod 123'6503 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:18.301 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:18.293+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5964 (116'2908,123'5964] local-lis/les=21/22 n=305 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5961 crt=123'5961 lcod 123'5959 mlcod 123'5959 active+clean+snaptrim trimq=[20b~1d] ps=[1fe~d]] removing snap head 2019-12-05T06:51:18.301 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:18.298+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6508 (117'3412,123'6508] local-lis/les=21/22 n=181 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6505 crt=123'6505 lcod 123'6503 mlcod 123'6503 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:18.303 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:18.298+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5950 (116'2908,123'5950] local-lis/les=21/22 n=445 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5950 lcod 123'5948 mlcod 123'5948 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:18.305 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:18.302+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6319 (118'3309,123'6319] local-lis/les=21/22 n=146 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'6319 lcod 123'6317 mlcod 123'6317 active+clean+snaptrim trimq=[210~18] ps=[1fe~12]] removing snap head 2019-12-05T06:51:18.306 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:18.302+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5952 (116'2908,123'5952] local-lis/les=21/22 n=443 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5950 crt=123'5950 lcod 123'5948 mlcod 123'5948 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:18.310 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:18.302+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6321 (118'3309,123'6321] local-lis/les=21/22 n=144 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'6319 crt=123'6319 lcod 123'6317 mlcod 123'6317 active+clean+snaptrim trimq=[210~18] ps=[1fe~12]] removing snap head 2019-12-05T06:51:18.310 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:18.306+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6510 (117'3412,123'6510] local-lis/les=21/22 n=179 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'6510 lcod 123'6508 mlcod 123'6508 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:18.312 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:18.310+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'5999 (115'2906,123'5999] local-lis/les=21/22 n=390 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5997 crt=123'5995 lcod 123'5995 mlcod 123'5995 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:18.313 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:18.310+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6512 (117'3412,123'6512] local-lis/les=21/22 n=177 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6510 crt=123'6510 lcod 123'6508 mlcod 123'6508 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:18.316 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:18.314+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'6001 (115'2906,123'6001] local-lis/les=21/22 n=388 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'5997 crt=123'5997 lcod 123'5995 mlcod 123'5995 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:18.318 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:18.314+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6323 (118'3309,123'6323] local-lis/les=21/22 n=142 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'6323 lcod 123'6321 mlcod 123'6321 active+clean+snaptrim trimq=[210~18] ps=[1fe~12]] removing snap head 2019-12-05T06:51:18.321 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:18.318+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6325 (118'3309,123'6325] local-lis/les=21/22 n=140 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'6323 crt=123'6323 lcod 123'6321 mlcod 123'6321 active+clean+snaptrim trimq=[210~18] ps=[1fe~12]] removing snap head 2019-12-05T06:51:18.324 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:18.318+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5954 (116'2908,123'5954] local-lis/les=21/22 n=441 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5954 lcod 123'5952 mlcod 123'5952 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:18.325 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:18.317+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5966 (116'2908,123'5966] local-lis/les=21/22 n=303 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5966 lcod 123'5964 mlcod 123'5964 active+clean+snaptrim trimq=[20b~1d] ps=[1fe~d]] removing snap head 2019-12-05T06:51:18.326 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:18.322+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6515 (118'3512,123'6515] local-lis/les=21/22 n=174 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6514 crt=123'6512 lcod 123'6512 mlcod 123'6512 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:18.326 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:18.322+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5956 (116'2908,123'5956] local-lis/les=21/22 n=439 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5954 crt=123'5954 lcod 123'5952 mlcod 123'5952 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:18.328 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:18.321+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5968 (116'2908,123'5968] local-lis/les=21/22 n=301 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5966 crt=123'5966 lcod 123'5964 mlcod 123'5964 active+clean+snaptrim trimq=[20b~1d] ps=[1fe~d]] removing snap head 2019-12-05T06:51:18.330 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:18.326+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6517 (118'3512,123'6517] local-lis/les=21/22 n=172 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6514 crt=123'6514 lcod 123'6512 mlcod 123'6512 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:18.332 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:18.330+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6328 (118'3309,123'6328] local-lis/les=21/22 n=137 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'6328 lcod 123'6327 mlcod 123'6327 active+clean+snaptrim trimq=[210~18] ps=[1fe~12]] removing snap head 2019-12-05T06:51:18.335 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:18.330+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5958 (116'2908,123'5958] local-lis/les=21/22 n=437 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5958 lcod 123'5956 mlcod 123'5956 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:18.336 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:18.330+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6330 (118'3309,123'6330] local-lis/les=21/22 n=135 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'6328 crt=123'6328 lcod 123'6327 mlcod 123'6327 active+clean+snaptrim trimq=[210~18] ps=[1fe~12]] removing snap head 2019-12-05T06:51:18.338 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:18.333+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5888 (115'2806,123'5888] local-lis/les=21/22 n=634 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=123'5888 lcod 123'5886 mlcod 123'5886 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:18.341 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:18.333+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5890 (115'2806,123'5890] local-lis/les=21/22 n=632 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5888 crt=123'5888 lcod 123'5886 mlcod 123'5886 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:18.343 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:18.338+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5960 (116'2908,123'5960] local-lis/les=21/22 n=435 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5958 crt=123'5958 lcod 123'5956 mlcod 123'5956 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:18.343 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:18.334+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6520 (118'3512,123'6520] local-lis/les=21/22 n=169 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6519 crt=123'6514 lcod 123'6517 mlcod 123'6517 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:18.343 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:18.338+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6522 (118'3512,123'6522] local-lis/les=21/22 n=167 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6519 crt=123'6519 lcod 123'6517 mlcod 123'6517 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:18.344 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:18.334+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6244 (117'3215,123'6244] local-lis/les=21/22 n=427 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'6243 crt=123'6241 lcod 123'6241 mlcod 123'6241 active+clean+snaptrim trimq=[208~20] ps=[1fe~a]] removing snap head 2019-12-05T06:51:18.345 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:18.334+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'6105 (116'3004,123'6105] local-lis/les=21/22 n=114 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'6105 lcod 123'6103 mlcod 123'6103 active+clean+snaptrim trimq=[210~18] ps=[1fe~12]] removing snap head 2019-12-05T06:51:18.345 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:18.342+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'6107 (117'3105,123'6107] local-lis/les=21/22 n=112 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'6105 crt=123'6105 lcod 123'6103 mlcod 123'6103 active+clean+snaptrim trimq=[210~18] ps=[1fe~12]] removing snap head 2019-12-05T06:51:18.347 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:18.341+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5970 (116'2908,123'5970] local-lis/les=21/22 n=299 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5970 lcod 123'5968 mlcod 123'5968 active+clean+snaptrim trimq=[20c~1c] ps=[1fe~e]] removing snap head 2019-12-05T06:51:18.351 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:18.341+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5972 (116'2908,123'5972] local-lis/les=21/22 n=297 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5970 crt=123'5970 lcod 123'5968 mlcod 123'5968 active+clean+snaptrim trimq=[20c~1c] ps=[1fe~e]] removing snap head 2019-12-05T06:51:18.353 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:18.350+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6332 (118'3309,123'6332] local-lis/les=21/22 n=133 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'6332 lcod 123'6330 mlcod 123'6330 active+clean+snaptrim trimq=[210~18] ps=[1fe~12]] removing snap head 2019-12-05T06:51:18.354 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:18.350+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5962 (116'2908,123'5962] local-lis/les=21/22 n=433 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5962 lcod 123'5960 mlcod 123'5960 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:18.357 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:18.354+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5964 (116'2908,123'5964] local-lis/les=21/22 n=431 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5962 crt=123'5962 lcod 123'5960 mlcod 123'5960 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:18.357 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:18.350+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6334 (118'3309,123'6334] local-lis/les=21/22 n=131 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'6332 crt=123'6332 lcod 123'6330 mlcod 123'6330 active+clean+snaptrim trimq=[210~18] ps=[1fe~12]] removing snap head 2019-12-05T06:51:18.358 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:18.354+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6524 (118'3512,123'6524] local-lis/les=21/22 n=165 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'6524 lcod 123'6522 mlcod 123'6522 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:18.361 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:18.358+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6526 (118'3512,123'6526] local-lis/les=21/22 n=163 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6524 crt=123'6524 lcod 123'6522 mlcod 123'6522 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:18.362 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:18.354+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'6004 (115'2906,123'6004] local-lis/les=21/22 n=385 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'6003 crt=123'6001 lcod 123'6001 mlcod 123'6001 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:18.362 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:18.353+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5975 (116'2908,123'5975] local-lis/les=21/22 n=294 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5974 crt=123'5970 lcod 123'5972 mlcod 123'5972 active+clean+snaptrim trimq=[20c~1c] ps=[1fe~e]] removing snap head 2019-12-05T06:51:18.363 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:18.358+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'6006 (115'2906,123'6006] local-lis/les=21/22 n=383 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'6003 crt=123'6003 lcod 123'6001 mlcod 123'6001 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:18.368 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:18.361+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5892 (115'2806,123'5892] local-lis/les=21/22 n=630 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=123'5892 lcod 123'5890 mlcod 123'5890 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:18.368 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:18.366+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'6109 (117'3105,123'6109] local-lis/les=21/22 n=110 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'6109 lcod 123'6107 mlcod 123'6107 active+clean+snaptrim trimq=[210~18] ps=[1fe~12]] removing snap head 2019-12-05T06:51:18.371 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:18.366+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6529 (118'3512,123'6529] local-lis/les=21/22 n=160 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6528 crt=123'6526 lcod 123'6526 mlcod 123'6526 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:18.371 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:18.366+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5966 (116'2908,123'5966] local-lis/les=21/22 n=429 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5966 lcod 123'5964 mlcod 123'5964 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:18.372 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:18.365+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5894 (115'2806,123'5894] local-lis/les=21/22 n=628 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5892 crt=123'5892 lcod 123'5890 mlcod 123'5890 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:18.373 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:18.370+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6531 (118'3512,123'6531] local-lis/les=21/22 n=158 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6528 crt=123'6528 lcod 123'6526 mlcod 123'6526 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:18.374 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:18.370+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5968 (116'2908,123'5968] local-lis/les=21/22 n=427 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5966 crt=123'5966 lcod 123'5964 mlcod 123'5964 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:18.375 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:18.366+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6247 (117'3215,123'6247] local-lis/les=21/22 n=424 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'6246 crt=123'6244 lcod 123'6244 mlcod 123'6244 active+clean+snaptrim trimq=[209~1f] ps=[1fe~b]] removing snap head 2019-12-05T06:51:18.376 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:18.370+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6249 (117'3215,123'6249] local-lis/les=21/22 n=422 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'6246 crt=123'6246 lcod 123'6244 mlcod 123'6244 active+clean+snaptrim trimq=[209~1f] ps=[1fe~b]] removing snap head 2019-12-05T06:51:18.376 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:18.374+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6336 (118'3309,123'6336] local-lis/les=21/22 n=129 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'6336 lcod 123'6334 mlcod 123'6334 active+clean+snaptrim trimq=[211~17] ps=[1fe~13]] removing snap head 2019-12-05T06:51:18.377 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:18.374+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'6111 (117'3105,123'6111] local-lis/les=21/22 n=108 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'6109 crt=123'6109 lcod 123'6107 mlcod 123'6107 active+clean+snaptrim trimq=[210~18] ps=[1fe~12]] removing snap head 2019-12-05T06:51:18.380 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:18.374+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6338 (118'3309,123'6338] local-lis/les=21/22 n=127 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'6336 crt=123'6336 lcod 123'6334 mlcod 123'6334 active+clean+snaptrim trimq=[211~17] ps=[1fe~13]] removing snap head 2019-12-05T06:51:18.381 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:18.373+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5978 (116'2908,123'5978] local-lis/les=21/22 n=291 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5977 crt=123'5977 lcod 123'5975 mlcod 123'5975 active+clean+snaptrim trimq=[20d~1b] ps=[1fe~f]] removing snap head 2019-12-05T06:51:18.384 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:18.377+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5980 (116'2908,123'5980] local-lis/les=21/22 n=289 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5977 crt=123'5977 lcod 123'5975 mlcod 123'5975 active+clean+snaptrim trimq=[20d~1b] ps=[1fe~f]] removing snap head 2019-12-05T06:51:18.385 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:18.382+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6533 (118'3512,123'6533] local-lis/les=21/22 n=156 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'6533 lcod 123'6531 mlcod 123'6531 active+clean+snaptrim trimq=[210~18] ps=[1fe~12]] removing snap head 2019-12-05T06:51:18.387 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:18.382+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5970 (116'2908,123'5970] local-lis/les=21/22 n=425 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5970 lcod 123'5968 mlcod 123'5968 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:18.388 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:18.382+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6535 (118'3512,123'6535] local-lis/les=21/22 n=154 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6533 crt=123'6533 lcod 123'6531 mlcod 123'6531 active+clean+snaptrim trimq=[210~18] ps=[1fe~12]] removing snap head 2019-12-05T06:51:18.391 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:18.385+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5982 (116'2908,123'5982] local-lis/les=21/22 n=287 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5982 lcod 123'5980 mlcod 123'5980 active+clean+snaptrim trimq=[20d~1b] ps=[1fe~f]] removing snap head 2019-12-05T06:51:18.394 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:18.385+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5984 (116'2908,123'5984] local-lis/les=21/22 n=285 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5982 crt=123'5982 lcod 123'5980 mlcod 123'5980 active+clean+snaptrim trimq=[20d~1b] ps=[1fe~f]] removing snap head 2019-12-05T06:51:18.397 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:18.390+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6341 (118'3309,123'6341] local-lis/les=21/22 n=124 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'6340 crt=123'6340 lcod 123'6338 mlcod 123'6338 active+clean+snaptrim trimq=[211~17] ps=[1fe~13]] removing snap head 2019-12-05T06:51:18.399 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:18.390+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5972 (116'2908,123'5972] local-lis/les=21/22 n=423 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5972 lcod 123'5970 mlcod 123'5970 active+clean+snaptrim trimq=[208~20] ps=[1fe~a]] removing snap head 2019-12-05T06:51:18.399 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:18.394+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5974 (116'2908,123'5974] local-lis/les=21/22 n=421 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5972 crt=123'5972 lcod 123'5970 mlcod 123'5970 active+clean+snaptrim trimq=[208~20] ps=[1fe~a]] removing snap head 2019-12-05T06:51:18.400 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:18.394+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6343 (118'3309,123'6343] local-lis/les=21/22 n=122 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'6340 crt=123'6340 lcod 123'6338 mlcod 123'6338 active+clean+snaptrim trimq=[211~17] ps=[1fe~13]] removing snap head 2019-12-05T06:51:18.400 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:18.394+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6538 (118'3512,123'6538] local-lis/les=21/22 n=151 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'6538 lcod 123'6537 mlcod 123'6537 active+clean+snaptrim trimq=[210~18] ps=[1fe~12]] removing snap head 2019-12-05T06:51:18.402 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:18.398+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6540 (118'3512,123'6540] local-lis/les=21/22 n=149 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6538 crt=123'6538 lcod 123'6537 mlcod 123'6537 active+clean+snaptrim trimq=[210~18] ps=[1fe~12]] removing snap head 2019-12-05T06:51:18.404 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:18.397+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5986 (116'2908,123'5986] local-lis/les=21/22 n=283 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5986 lcod 123'5984 mlcod 123'5984 active+clean+snaptrim trimq=[20d~1b] ps=[1fe~f]] removing snap head 2019-12-05T06:51:18.407 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:18.402+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5976 (116'2908,123'5976] local-lis/les=21/22 n=419 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5976 lcod 123'5974 mlcod 123'5974 active+clean+snaptrim trimq=[208~20] ps=[1fe~a]] removing snap head 2019-12-05T06:51:18.407 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:18.397+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5988 (116'2908,123'5988] local-lis/les=21/22 n=281 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5986 crt=123'5986 lcod 123'5984 mlcod 123'5984 active+clean+snaptrim trimq=[20d~1b] ps=[1fe~f]] removing snap head 2019-12-05T06:51:18.409 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:18.406+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6346 (118'3309,123'6346] local-lis/les=21/22 n=119 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'6345 crt=123'6340 lcod 123'6343 mlcod 123'6343 active+clean+snaptrim trimq=[211~17] ps=[1fe~13]] removing snap head 2019-12-05T06:51:18.412 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:18.406+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5978 (116'2908,123'5978] local-lis/les=21/22 n=417 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5976 crt=123'5976 lcod 123'5974 mlcod 123'5974 active+clean+snaptrim trimq=[208~20] ps=[1fe~a]] removing snap head 2019-12-05T06:51:18.412 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:18.406+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6348 (118'3309,123'6348] local-lis/les=21/22 n=117 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'6345 crt=123'6345 lcod 123'6343 mlcod 123'6343 active+clean+snaptrim trimq=[211~17] ps=[1fe~13]] removing snap head 2019-12-05T06:51:18.418 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:18.405+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5990 (116'2908,123'5990] local-lis/les=21/22 n=279 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5990 lcod 123'5988 mlcod 123'5988 active+clean+snaptrim trimq=[20d~1b] ps=[1fe~f]] removing snap head 2019-12-05T06:51:18.419 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:18.409+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5992 (116'2908,123'5992] local-lis/les=21/22 n=277 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5990 crt=123'5990 lcod 123'5988 mlcod 123'5988 active+clean+snaptrim trimq=[20d~1b] ps=[1fe~f]] removing snap head 2019-12-05T06:51:18.420 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:18.410+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6542 (118'3512,123'6542] local-lis/les=21/22 n=147 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'6542 lcod 123'6540 mlcod 123'6540 active+clean+snaptrim trimq=[210~18] ps=[1fe~12]] removing snap head 2019-12-05T06:51:18.420 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:18.414+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6544 (118'3512,123'6544] local-lis/les=21/22 n=145 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6542 crt=123'6542 lcod 123'6540 mlcod 123'6540 active+clean+snaptrim trimq=[210~18] ps=[1fe~12]] removing snap head 2019-12-05T06:51:18.422 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:18.414+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5981 (116'2908,123'5981] local-lis/les=21/22 n=414 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5980 crt=123'5978 lcod 123'5978 mlcod 123'5978 active+clean+snaptrim trimq=[208~20] ps=[1fe~a]] removing snap head 2019-12-05T06:51:18.423 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:18.418+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5983 (116'2908,123'5983] local-lis/les=21/22 n=412 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5980 crt=123'5980 lcod 123'5978 mlcod 123'5978 active+clean+snaptrim trimq=[208~20] ps=[1fe~a]] removing snap head 2019-12-05T06:51:18.423 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:18.418+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6351 (118'3309,123'6351] local-lis/les=21/22 n=114 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'6350 crt=123'6348 lcod 123'6348 mlcod 123'6348 active+clean+snaptrim trimq=[211~17] ps=[1fe~13]] removing snap head 2019-12-05T06:51:18.425 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:18.422+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6353 (118'3309,123'6353] local-lis/les=21/22 n=112 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'6350 crt=123'6350 lcod 123'6348 mlcod 123'6348 active+clean+snaptrim trimq=[211~17] ps=[1fe~13]] removing snap head 2019-12-05T06:51:18.429 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:18.421+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5994 (116'2908,123'5994] local-lis/les=21/22 n=275 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5994 lcod 123'5992 mlcod 123'5992 active+clean+snaptrim trimq=[20d~1b] ps=[1fe~f]] removing snap head 2019-12-05T06:51:18.429 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:18.426+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5985 (116'2908,123'5985] local-lis/les=21/22 n=410 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5985 lcod 123'5983 mlcod 123'5983 active+clean+snaptrim trimq=[208~20] ps=[1fe~a]] removing snap head 2019-12-05T06:51:18.430 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:18.425+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5996 (116'2908,123'5996] local-lis/les=21/22 n=273 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5994 crt=123'5994 lcod 123'5992 mlcod 123'5992 active+clean+snaptrim trimq=[20d~1b] ps=[1fe~f]] removing snap head 2019-12-05T06:51:18.431 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:18.426+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5987 (116'2908,123'5987] local-lis/les=21/22 n=408 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5985 crt=123'5985 lcod 123'5983 mlcod 123'5983 active+clean+snaptrim trimq=[208~20] ps=[1fe~a]] removing snap head 2019-12-05T06:51:18.435 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:18.429+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'5998 (116'2908,123'5998] local-lis/les=21/22 n=271 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'5998 lcod 123'5996 mlcod 123'5996 active+clean+snaptrim trimq=[20d~1b] ps=[1fe~f]] removing snap head 2019-12-05T06:51:18.437 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:18.434+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5989 (116'2908,123'5989] local-lis/les=21/22 n=406 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5989 lcod 123'5987 mlcod 123'5987 active+clean+snaptrim trimq=[209~1f] ps=[1fe~b]] removing snap head 2019-12-05T06:51:18.437 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:18.429+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'6000 (116'2908,123'6000] local-lis/les=21/22 n=269 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'5998 crt=123'5998 lcod 123'5996 mlcod 123'5996 active+clean+snaptrim trimq=[20d~1b] ps=[1fe~f]] removing snap head 2019-12-05T06:51:18.438 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:18.434+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5991 (116'2908,123'5991] local-lis/les=21/22 n=404 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5989 crt=123'5989 lcod 123'5987 mlcod 123'5987 active+clean+snaptrim trimq=[209~1f] ps=[1fe~b]] removing snap head 2019-12-05T06:51:18.443 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:18.437+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'6002 (116'2908,123'6002] local-lis/les=21/22 n=267 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'6002 lcod 123'6000 mlcod 123'6000 active+clean+snaptrim trimq=[20d~1b] ps=[1fe~f]] removing snap head 2019-12-05T06:51:18.445 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:18.438+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5993 (116'2908,123'5993] local-lis/les=21/22 n=402 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5993 lcod 123'5991 mlcod 123'5991 active+clean+snaptrim trimq=[209~1f] ps=[1fe~b]] removing snap head 2019-12-05T06:51:18.446 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:18.442+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5995 (116'2908,123'5995] local-lis/les=21/22 n=400 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5993 crt=123'5993 lcod 123'5991 mlcod 123'5991 active+clean+snaptrim trimq=[209~1f] ps=[1fe~b]] removing snap head 2019-12-05T06:51:18.446 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:18.437+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'6004 (116'2908,123'6004] local-lis/les=21/22 n=265 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'6002 crt=123'6002 lcod 123'6000 mlcod 123'6000 active+clean+snaptrim trimq=[20d~1b] ps=[1fe~f]] removing snap head 2019-12-05T06:51:18.449 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:18.446+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5997 (116'2908,123'5997] local-lis/les=21/22 n=398 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'5997 lcod 123'5995 mlcod 123'5995 active+clean+snaptrim trimq=[209~1f] ps=[1fe~b]] removing snap head 2019-12-05T06:51:18.450 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:18.441+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'6006 (116'2908,123'6006] local-lis/les=21/22 n=263 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'6006 lcod 123'6004 mlcod 123'6004 active+clean+snaptrim trimq=[20e~1a] ps=[1fe~10]] removing snap head 2019-12-05T06:51:18.451 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:18.446+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'5999 (116'2908,123'5999] local-lis/les=21/22 n=396 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'5997 crt=123'5997 lcod 123'5995 mlcod 123'5995 active+clean+snaptrim trimq=[209~1f] ps=[1fe~b]] removing snap head 2019-12-05T06:51:18.452 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:18.445+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'6008 (116'2908,123'6008] local-lis/les=21/22 n=261 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'6006 crt=123'6006 lcod 123'6004 mlcod 123'6004 active+clean+snaptrim trimq=[20e~1a] ps=[1fe~10]] removing snap head 2019-12-05T06:51:18.458 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:18.454+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'6001 (116'2908,123'6001] local-lis/les=21/22 n=394 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'6001 lcod 123'5999 mlcod 123'5999 active+clean+snaptrim trimq=[209~1f] ps=[1fe~b]] removing snap head 2019-12-05T06:51:18.459 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:18.454+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'6003 (116'2908,123'6003] local-lis/les=21/22 n=392 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'6001 crt=123'6001 lcod 123'5999 mlcod 123'5999 active+clean+snaptrim trimq=[209~1f] ps=[1fe~b]] removing snap head 2019-12-05T06:51:18.466 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:18.457+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'6011 (117'3008,123'6011] local-lis/les=21/22 n=258 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'6010 crt=123'6008 lcod 123'6008 mlcod 123'6008 active+clean+snaptrim trimq=[20e~1a] ps=[1fe~10]] removing snap head 2019-12-05T06:51:18.466 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:18.462+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'6005 (116'2908,123'6005] local-lis/les=21/22 n=390 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'6005 lcod 123'6003 mlcod 123'6003 active+clean+snaptrim trimq=[209~1f] ps=[1fe~b]] removing snap head 2019-12-05T06:51:18.467 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:18.462+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'6009 (116'3006,123'6009] local-lis/les=21/22 n=380 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'6008 crt=123'6006 lcod 123'6006 mlcod 123'6006 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:18.467 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:18.462+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'6007 (116'2908,123'6007] local-lis/les=21/22 n=388 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'6005 crt=123'6005 lcod 123'6003 mlcod 123'6003 active+clean+snaptrim trimq=[209~1f] ps=[1fe~b]] removing snap head 2019-12-05T06:51:18.468 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:18.461+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'6013 (117'3008,123'6013] local-lis/les=21/22 n=256 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'6010 crt=123'6010 lcod 123'6008 mlcod 123'6008 active+clean+snaptrim trimq=[20e~1a] ps=[1fe~10]] removing snap head 2019-12-05T06:51:18.469 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:18.466+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'6011 (116'3006,123'6011] local-lis/les=21/22 n=378 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'6008 crt=123'6008 lcod 123'6006 mlcod 123'6006 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:18.473 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:18.470+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'6113 (117'3105,123'6113] local-lis/les=21/22 n=106 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'6113 lcod 123'6111 mlcod 123'6111 active+clean+snaptrim trimq=[210~18] ps=[1fe~12]] removing snap head 2019-12-05T06:51:18.477 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:18.465+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5897 (115'2806,123'5897] local-lis/les=21/22 n=625 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5896 crt=123'5894 lcod 123'5894 mlcod 123'5894 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:18.477 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:18.470+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'6009 (116'2908,123'6009] local-lis/les=21/22 n=386 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'6009 lcod 123'6007 mlcod 123'6007 active+clean+snaptrim trimq=[209~1f] ps=[1fe~b]] removing snap head 2019-12-05T06:51:18.477 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:18.474+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'6115 (117'3105,123'6115] local-lis/les=21/22 n=104 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'6113 crt=123'6113 lcod 123'6111 mlcod 123'6111 active+clean+snaptrim trimq=[210~18] ps=[1fe~12]] removing snap head 2019-12-05T06:51:18.478 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:18.469+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5899 (115'2806,123'5899] local-lis/les=21/22 n=623 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5896 crt=123'5896 lcod 123'5894 mlcod 123'5894 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:18.478 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:18.474+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'6014 (116'3006,123'6014] local-lis/les=21/22 n=375 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'6013 crt=123'6009 lcod 123'6011 mlcod 123'6011 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:18.486 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:18.482+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6252 (117'3215,123'6252] local-lis/les=21/22 n=419 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'6251 crt=123'6249 lcod 123'6249 mlcod 123'6249 active+clean+snaptrim trimq=[209~1f] ps=[1fe~b]] removing snap head 2019-12-05T06:51:18.487 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:18.481+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5901 (115'2806,123'5901] local-lis/les=21/22 n=621 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=123'5901 lcod 123'5899 mlcod 123'5899 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:18.487 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:18.482+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'6011 (117'3009,123'6011] local-lis/les=21/22 n=384 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'6011 lcod 123'6009 mlcod 123'6009 active+clean+snaptrim trimq=[20a~1e] ps=[1fe~c]] removing snap head 2019-12-05T06:51:18.492 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:18.486+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6254 (117'3215,123'6254] local-lis/les=21/22 n=417 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'6251 crt=123'6251 lcod 123'6249 mlcod 123'6249 active+clean+snaptrim trimq=[209~1f] ps=[1fe~b]] removing snap head 2019-12-05T06:51:18.493 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:18.486+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6547 (118'3512,123'6547] local-lis/les=21/22 n=142 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6546 crt=123'6544 lcod 123'6544 mlcod 123'6544 active+clean+snaptrim trimq=[210~18] ps=[1fe~12]] removing snap head 2019-12-05T06:51:18.493 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:18.486+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6549 (118'3512,123'6549] local-lis/les=21/22 n=140 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6546 crt=123'6546 lcod 123'6544 mlcod 123'6544 active+clean+snaptrim trimq=[210~18] ps=[1fe~12]] removing snap head 2019-12-05T06:51:18.493 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:18.486+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'6013 (117'3009,123'6013] local-lis/les=21/22 n=382 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'6011 crt=123'6011 lcod 123'6009 mlcod 123'6009 active+clean+snaptrim trimq=[20a~1e] ps=[1fe~c]] removing snap head 2019-12-05T06:51:18.494 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:18.481+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5903 (115'2806,123'5903] local-lis/les=21/22 n=619 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5901 crt=123'5901 lcod 123'5899 mlcod 123'5899 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:18.497 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:18.494+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6355 (118'3309,123'6355] local-lis/les=21/22 n=110 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'6355 lcod 123'6353 mlcod 123'6353 active+clean+snaptrim trimq=[211~17] ps=[1fe~13]] removing snap head 2019-12-05T06:51:18.498 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:18.489+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'6016 (117'3008,123'6016] local-lis/les=21/22 n=253 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'6015 crt=123'6013 lcod 123'6013 mlcod 123'6013 active+clean+snaptrim trimq=[20e~1a] ps=[1fe~10]] removing snap head 2019-12-05T06:51:18.499 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:18.494+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6357 (118'3309,123'6357] local-lis/les=21/22 n=108 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'6355 crt=123'6355 lcod 123'6353 mlcod 123'6353 active+clean+snaptrim trimq=[211~17] ps=[1fe~13]] removing snap head 2019-12-05T06:51:18.501 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:18.493+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'6018 (117'3008,123'6018] local-lis/les=21/22 n=251 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'6015 crt=123'6015 lcod 123'6013 mlcod 123'6013 active+clean+snaptrim trimq=[20e~1a] ps=[1fe~10]] removing snap head 2019-12-05T06:51:18.502 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:18.498+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'6015 (117'3009,123'6015] local-lis/les=21/22 n=380 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'6015 lcod 123'6013 mlcod 123'6013 active+clean+snaptrim trimq=[20a~1e] ps=[1fe~c]] removing snap head 2019-12-05T06:51:18.503 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:18.498+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'6017 (117'3009,123'6017] local-lis/les=21/22 n=378 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'6015 crt=123'6015 lcod 123'6013 mlcod 123'6013 active+clean+snaptrim trimq=[20a~1e] ps=[1fe~c]] removing snap head 2019-12-05T06:51:18.505 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:18.502+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6551 (118'3512,123'6551] local-lis/les=21/22 n=138 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'6551 lcod 123'6549 mlcod 123'6549 active+clean+snaptrim trimq=[210~18] ps=[1fe~12]] removing snap head 2019-12-05T06:51:18.509 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:18.506+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6553 (118'3512,123'6553] local-lis/les=21/22 n=136 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6551 crt=123'6551 lcod 123'6549 mlcod 123'6549 active+clean+snaptrim trimq=[210~18] ps=[1fe~12]] removing snap head 2019-12-05T06:51:18.513 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:18.510+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6360 (118'3309,123'6360] local-lis/les=21/22 n=105 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'6359 crt=123'6355 lcod 123'6357 mlcod 123'6357 active+clean+snaptrim trimq=[211~17] ps=[1fe~13]] removing snap head 2019-12-05T06:51:18.513 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:18.506+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'6017 (116'3006,123'6017] local-lis/les=21/22 n=372 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'6016 crt=123'6014 lcod 123'6014 mlcod 123'6014 active+clean+snaptrim trimq=[208~20] ps=[1fe~a]] removing snap head 2019-12-05T06:51:18.514 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:18.506+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'6019 (116'3006,123'6019] local-lis/les=21/22 n=370 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'6016 crt=123'6016 lcod 123'6014 mlcod 123'6014 active+clean+snaptrim trimq=[208~20] ps=[1fe~a]] removing snap head 2019-12-05T06:51:18.514 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:18.510+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'6019 (117'3009,123'6019] local-lis/les=21/22 n=376 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'6019 lcod 123'6017 mlcod 123'6017 active+clean+snaptrim trimq=[20a~1e] ps=[1fe~c]] removing snap head 2019-12-05T06:51:18.515 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:18.510+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'6117 (117'3105,123'6117] local-lis/les=21/22 n=102 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'6117 lcod 123'6115 mlcod 123'6115 active+clean+snaptrim trimq=[210~18] ps=[1fe~12]] removing snap head 2019-12-05T06:51:18.516 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:18.514+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'6021 (117'3009,123'6021] local-lis/les=21/22 n=374 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'6019 crt=123'6019 lcod 123'6017 mlcod 123'6017 active+clean+snaptrim trimq=[20a~1e] ps=[1fe~c]] removing snap head 2019-12-05T06:51:18.519 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:18.514+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6362 (118'3309,123'6362] local-lis/les=21/22 n=103 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'6359 crt=123'6359 lcod 123'6357 mlcod 123'6357 active+clean+snaptrim trimq=[211~17] ps=[1fe~13]] removing snap head 2019-12-05T06:51:18.522 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:18.509+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5905 (115'2806,123'5905] local-lis/les=21/22 n=617 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=123'5905 lcod 123'5903 mlcod 123'5903 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:18.522 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:18.513+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5907 (115'2806,123'5907] local-lis/les=21/22 n=615 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5905 crt=123'5905 lcod 123'5903 mlcod 123'5903 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:18.523 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:18.518+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'6023 (117'3009,123'6023] local-lis/les=21/22 n=372 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'6023 lcod 123'6021 mlcod 123'6021 active+clean+snaptrim trimq=[20a~1e] ps=[1fe~c]] removing snap head 2019-12-05T06:51:18.525 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:18.522+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'6025 (117'3009,123'6025] local-lis/les=21/22 n=370 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'6023 crt=123'6023 lcod 123'6021 mlcod 123'6021 active+clean+snaptrim trimq=[20a~1e] ps=[1fe~c]] removing snap head 2019-12-05T06:51:18.526 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:18.522+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'6119 (117'3105,123'6119] local-lis/les=21/22 n=100 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'6119 lcod 123'6117 mlcod 123'6117 active+clean+snaptrim trimq=[211~17] ps=[1fe~13]] removing snap head 2019-12-05T06:51:18.529 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:18.526+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'6121 (117'3105,123'6121] local-lis/les=21/22 n=98 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'6119 crt=123'6119 lcod 123'6117 mlcod 123'6117 active+clean+snaptrim trimq=[211~17] ps=[1fe~13]] removing snap head 2019-12-05T06:51:18.531 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:18.525+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5911 (115'2907,123'5911] local-lis/les=21/22 n=611 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5910 crt=123'5905 lcod 123'5909 mlcod 123'5909 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:18.534 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:18.530+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'6027 (117'3009,123'6027] local-lis/les=21/22 n=368 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'6027 lcod 123'6025 mlcod 123'6025 active+clean+snaptrim trimq=[20a~1e] ps=[1fe~c]] removing snap head 2019-12-05T06:51:18.535 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:18.529+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5913 (115'2907,123'5913] local-lis/les=21/22 n=609 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5910 crt=123'5910 lcod 123'5909 mlcod 123'5909 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:18.550 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:18.534+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'6029 (117'3009,123'6029] local-lis/les=21/22 n=366 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'6027 crt=123'6027 lcod 123'6025 mlcod 123'6025 active+clean+snaptrim trimq=[20a~1e] ps=[1fe~c]] removing snap head 2019-12-05T06:51:18.550 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:18.538+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'6031 (117'3009,123'6031] local-lis/les=21/22 n=364 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'6031 lcod 123'6029 mlcod 123'6029 active+clean+snaptrim trimq=[20a~1e] ps=[1fe~c]] removing snap head 2019-12-05T06:51:18.550 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:18.538+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'6033 (117'3009,123'6033] local-lis/les=21/22 n=362 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'6031 crt=123'6031 lcod 123'6029 mlcod 123'6029 active+clean+snaptrim trimq=[20a~1e] ps=[1fe~c]] removing snap head 2019-12-05T06:51:18.550 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:18.538+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'6125 (117'3105,123'6125] local-lis/les=21/22 n=94 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'6123 crt=123'6121 lcod 123'6121 mlcod 123'6121 active+clean+snaptrim trimq=[211~17] ps=[1fe~13]] removing snap head 2019-12-05T06:51:18.550 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:18.538+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'6127 (117'3105,123'6127] local-lis/les=21/22 n=92 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'6123 crt=123'6123 lcod 123'6121 mlcod 123'6121 active+clean+snaptrim trimq=[211~17] ps=[1fe~13]] removing snap head 2019-12-05T06:51:18.551 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:18.542+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6555 (118'3512,123'6555] local-lis/les=21/22 n=134 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'6555 lcod 123'6553 mlcod 123'6553 active+clean+snaptrim trimq=[210~18] ps=[1fe~12]] removing snap head 2019-12-05T06:51:18.551 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:18.542+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6557 (118'3512,123'6557] local-lis/les=21/22 n=132 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6555 crt=123'6555 lcod 123'6553 mlcod 123'6553 active+clean+snaptrim trimq=[210~18] ps=[1fe~12]] removing snap head 2019-12-05T06:51:18.552 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:18.550+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6364 (118'3309,123'6364] local-lis/les=21/22 n=101 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'6364 lcod 123'6362 mlcod 123'6362 active+clean+snaptrim trimq=[211~17] ps=[1fe~13]] removing snap head 2019-12-05T06:51:18.556 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:18.550+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'6130 (117'3105,123'6130] local-lis/les=21/22 n=89 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'6129 crt=123'6125 lcod 123'6127 mlcod 123'6127 active+clean+snaptrim trimq=[211~17] ps=[1fe~13]] removing snap head 2019-12-05T06:51:18.556 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:18.550+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'6036 (117'3009,123'6036] local-lis/les=21/22 n=359 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'6035 crt=123'6035 lcod 123'6033 mlcod 123'6033 active+clean+snaptrim trimq=[20a~1e] ps=[1fe~c]] removing snap head 2019-12-05T06:51:18.556 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:18.554+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'6038 (117'3009,123'6038] local-lis/les=21/22 n=357 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'6035 crt=123'6035 lcod 123'6033 mlcod 123'6033 active+clean+snaptrim trimq=[20a~1e] ps=[1fe~c]] removing snap head 2019-12-05T06:51:18.558 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:18.550+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6366 (118'3309,123'6366] local-lis/les=21/22 n=99 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'6364 crt=123'6364 lcod 123'6362 mlcod 123'6362 active+clean+snaptrim trimq=[211~17] ps=[1fe~13]] removing snap head 2019-12-05T06:51:18.558 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:18.554+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'6132 (117'3105,123'6132] local-lis/les=21/22 n=87 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'6129 crt=123'6129 lcod 123'6127 mlcod 123'6127 active+clean+snaptrim trimq=[211~17] ps=[1fe~13]] removing snap head 2019-12-05T06:51:18.559 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:18.554+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6559 (118'3512,123'6559] local-lis/les=21/22 n=130 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'6559 lcod 123'6557 mlcod 123'6557 active+clean+snaptrim trimq=[210~18] ps=[1fe~12]] removing snap head 2019-12-05T06:51:18.560 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:18.558+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6561 (118'3512,123'6561] local-lis/les=21/22 n=128 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6559 crt=123'6559 lcod 123'6557 mlcod 123'6557 active+clean+snaptrim trimq=[210~18] ps=[1fe~12]] removing snap head 2019-12-05T06:51:18.564 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:18.557+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5916 (115'2907,123'5916] local-lis/les=21/22 n=606 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5915 crt=123'5913 lcod 123'5913 mlcod 123'5913 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:18.566 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:18.562+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'6040 (117'3009,123'6040] local-lis/les=21/22 n=355 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'6040 lcod 123'6038 mlcod 123'6038 active+clean+snaptrim trimq=[20a~1e] ps=[1fe~c]] removing snap head 2019-12-05T06:51:18.567 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:18.562+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'6042 (117'3009,123'6042] local-lis/les=21/22 n=353 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'6040 crt=123'6040 lcod 123'6038 mlcod 123'6038 active+clean+snaptrim trimq=[20a~1e] ps=[1fe~c]] removing snap head 2019-12-05T06:51:18.569 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:18.561+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5918 (115'2907,123'5918] local-lis/les=21/22 n=604 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5915 crt=123'5915 lcod 123'5913 mlcod 123'5913 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:18.573 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:18.565+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5921 (115'2907,123'5921] local-lis/les=21/22 n=601 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5920 crt=123'5916 lcod 123'5918 mlcod 123'5918 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:18.574 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:18.570+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'6044 (117'3009,123'6044] local-lis/les=21/22 n=351 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'6044 lcod 123'6042 mlcod 123'6042 active+clean+snaptrim trimq=[20a~1e] ps=[1fe~c]] removing snap head 2019-12-05T06:51:18.575 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:18.569+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5923 (115'2907,123'5923] local-lis/les=21/22 n=599 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5920 crt=123'5920 lcod 123'5918 mlcod 123'5918 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:18.576 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:18.574+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'6046 (117'3009,123'6046] local-lis/les=21/22 n=349 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'6044 crt=123'6044 lcod 123'6042 mlcod 123'6042 active+clean+snaptrim trimq=[20a~1e] ps=[1fe~c]] removing snap head 2019-12-05T06:51:18.581 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:18.578+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6565 (118'3512,123'6565] local-lis/les=21/22 n=124 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6563 crt=123'6561 lcod 123'6561 mlcod 123'6561 active+clean+snaptrim trimq=[210~18] ps=[1fe~12]] removing snap head 2019-12-05T06:51:18.582 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:18.578+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'6048 (117'3009,123'6048] local-lis/les=21/22 n=347 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'6048 lcod 123'6046 mlcod 123'6046 active+clean+snaptrim trimq=[20a~1e] ps=[1fe~c]] removing snap head 2019-12-05T06:51:18.583 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:18.578+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6567 (118'3512,123'6567] local-lis/les=21/22 n=122 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6563 crt=123'6563 lcod 123'6561 mlcod 123'6561 active+clean+snaptrim trimq=[210~18] ps=[1fe~12]] removing snap head 2019-12-05T06:51:18.584 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:18.582+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'6050 (117'3009,123'6050] local-lis/les=21/22 n=345 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'6048 crt=123'6048 lcod 123'6046 mlcod 123'6046 active+clean+snaptrim trimq=[20a~1e] ps=[1fe~c]] removing snap head 2019-12-05T06:51:18.590 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:18.586+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6257 (117'3215,123'6257] local-lis/les=21/22 n=414 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'6256 crt=123'6254 lcod 123'6254 mlcod 123'6254 active+clean+snaptrim trimq=[209~1f] ps=[1fe~b]] removing snap head 2019-12-05T06:51:18.592 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:18.585+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5928 (115'2907,123'5928] local-lis/les=21/22 n=595 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5925 crt=123'5925 lcod 123'5923 mlcod 123'5923 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:18.599 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:18.594+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6570 (118'3512,123'6570] local-lis/les=21/22 n=119 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6569 crt=123'6565 lcod 123'6567 mlcod 123'6567 active+clean+snaptrim trimq=[210~18] ps=[1fe~12]] removing snap head 2019-12-05T06:51:18.600 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:18.598+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6369 (118'3309,123'6369] local-lis/les=21/22 n=96 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'6368 crt=123'6366 lcod 123'6366 mlcod 123'6366 active+clean+snaptrim trimq=[211~17] ps=[1fe~13]] removing snap head 2019-12-05T06:51:18.602 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:18.598+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6572 (118'3512,123'6572] local-lis/les=21/22 n=117 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6569 crt=123'6569 lcod 123'6567 mlcod 123'6567 active+clean+snaptrim trimq=[210~18] ps=[1fe~12]] removing snap head 2019-12-05T06:51:18.602 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:18.598+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'6134 (117'3105,123'6134] local-lis/les=21/22 n=85 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'6134 lcod 123'6132 mlcod 123'6132 active+clean+snaptrim trimq=[211~17] ps=[1fe~13]] removing snap head 2019-12-05T06:51:18.605 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:18.598+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6371 (118'3309,123'6371] local-lis/les=21/22 n=94 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'6368 crt=123'6368 lcod 123'6366 mlcod 123'6366 active+clean+snaptrim trimq=[211~17] ps=[1fe~13]] removing snap head 2019-12-05T06:51:18.605 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:18.602+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'6136 (117'3105,123'6136] local-lis/les=21/22 n=83 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'6134 crt=123'6134 lcod 123'6132 mlcod 123'6132 active+clean+snaptrim trimq=[211~17] ps=[1fe~13]] removing snap head 2019-12-05T06:51:18.610 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:18.606+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6574 (118'3512,123'6574] local-lis/les=21/22 n=115 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'6574 lcod 123'6572 mlcod 123'6572 active+clean+snaptrim trimq=[210~18] ps=[1fe~12]] removing snap head 2019-12-05T06:51:18.611 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:18.606+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6374 (118'3309,123'6374] local-lis/les=21/22 n=91 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'6373 crt=123'6369 lcod 123'6371 mlcod 123'6371 active+clean+snaptrim trimq=[211~17] ps=[1fe~13]] removing snap head 2019-12-05T06:51:18.613 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:18.610+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6576 (118'3512,123'6576] local-lis/les=21/22 n=113 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6574 crt=123'6574 lcod 123'6572 mlcod 123'6572 active+clean+snaptrim trimq=[210~18] ps=[1fe~12]] removing snap head 2019-12-05T06:51:18.614 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:18.610+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6376 (118'3309,123'6376] local-lis/les=21/22 n=89 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'6373 crt=123'6373 lcod 123'6371 mlcod 123'6371 active+clean+snaptrim trimq=[211~17] ps=[1fe~13]] removing snap head 2019-12-05T06:51:18.619 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:18.614+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6579 (118'3512,123'6579] local-lis/les=21/22 n=110 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6578 crt=123'6576 lcod 123'6576 mlcod 123'6576 active+clean+snaptrim trimq=[210~18] ps=[1fe~12]] removing snap head 2019-12-05T06:51:18.619 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:18.614+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'6139 (117'3105,123'6139] local-lis/les=21/22 n=80 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'6138 crt=123'6134 lcod 123'6136 mlcod 123'6136 active+clean+snaptrim trimq=[211~17] ps=[1fe~13]] removing snap head 2019-12-05T06:51:18.620 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:18.614+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'6141 (117'3105,123'6141] local-lis/les=21/22 n=78 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'6138 crt=123'6138 lcod 123'6136 mlcod 123'6136 active+clean+snaptrim trimq=[211~17] ps=[1fe~13]] removing snap head 2019-12-05T06:51:18.621 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:18.618+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6581 (118'3512,123'6581] local-lis/les=21/22 n=108 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6578 crt=123'6578 lcod 123'6576 mlcod 123'6576 active+clean+snaptrim trimq=[210~18] ps=[1fe~12]] removing snap head 2019-12-05T06:51:18.624 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:18.622+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6378 (118'3309,123'6378] local-lis/les=21/22 n=87 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'6378 lcod 123'6376 mlcod 123'6376 active+clean+snaptrim trimq=[211~17] ps=[1fe~13]] removing snap head 2019-12-05T06:51:18.626 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:18.622+0000 7fd862c4c700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6380 (118'3309,123'6380] local-lis/les=21/22 n=85 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'6378 crt=123'6378 lcod 123'6376 mlcod 123'6376 active+clean+snaptrim trimq=[211~17] ps=[1fe~13]] removing snap head 2019-12-05T06:51:18.632 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:18.630+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'6054 (117'3009,123'6054] local-lis/les=21/22 n=341 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'6053 crt=123'6052 lcod 123'6052 mlcod 123'6052 active+clean+snaptrim trimq=[20b~1d] ps=[1fe~d]] removing snap head 2019-12-05T06:51:18.636 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:18.630+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'6144 (117'3105,123'6144] local-lis/les=21/22 n=75 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'6143 crt=123'6141 lcod 123'6141 mlcod 123'6141 active+clean+snaptrim trimq=[211~17] ps=[1fe~13]] removing snap head 2019-12-05T06:51:18.637 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:18.630+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'6056 (117'3009,123'6056] local-lis/les=21/22 n=339 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'6053 crt=123'6053 lcod 123'6052 mlcod 123'6052 active+clean+snaptrim trimq=[20b~1d] ps=[1fe~d]] removing snap head 2019-12-05T06:51:18.637 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:18.630+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6583 (118'3512,123'6583] local-lis/les=21/22 n=106 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'6583 lcod 123'6581 mlcod 123'6581 active+clean+snaptrim trimq=[211~17] ps=[1fe~13]] removing snap head 2019-12-05T06:51:18.638 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:18.634+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'6146 (117'3105,123'6146] local-lis/les=21/22 n=73 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'6143 crt=123'6143 lcod 123'6141 mlcod 123'6141 active+clean+snaptrim trimq=[211~17] ps=[1fe~13]] removing snap head 2019-12-05T06:51:18.640 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:18.638+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6585 (118'3512,123'6585] local-lis/les=21/22 n=104 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6583 crt=123'6583 lcod 123'6581 mlcod 123'6581 active+clean+snaptrim trimq=[211~17] ps=[1fe~13]] removing snap head 2019-12-05T06:51:18.641 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:18.638+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'6059 (117'3009,123'6059] local-lis/les=21/22 n=336 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'6058 crt=123'6056 lcod 123'6056 mlcod 123'6056 active+clean+snaptrim trimq=[20b~1d] ps=[1fe~d]] removing snap head 2019-12-05T06:51:18.643 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:18.638+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6382 (118'3309,123'6382] local-lis/les=21/22 n=83 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'6382 lcod 123'6380 mlcod 123'6380 active+clean+snaptrim trimq=[211~17] ps=[1fe~13]] removing snap head 2019-12-05T06:51:18.644 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:18.638+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6384 (118'3309,123'6384] local-lis/les=21/22 n=81 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'6382 crt=123'6382 lcod 123'6380 mlcod 123'6380 active+clean+snaptrim trimq=[211~17] ps=[1fe~13]] removing snap head 2019-12-05T06:51:18.644 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:18.638+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'6061 (117'3009,123'6061] local-lis/les=21/22 n=334 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'6058 crt=123'6058 lcod 123'6056 mlcod 123'6056 active+clean+snaptrim trimq=[20b~1d] ps=[1fe~d]] removing snap head 2019-12-05T06:51:18.649 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:18.646+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'6022 (116'3006,123'6022] local-lis/les=21/22 n=367 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'6021 crt=123'6019 lcod 123'6019 mlcod 123'6019 active+clean+snaptrim trimq=[208~20] ps=[1fe~a]] removing snap head 2019-12-05T06:51:18.649 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:18.646+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6587 (118'3512,123'6587] local-lis/les=21/22 n=102 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'6587 lcod 123'6585 mlcod 123'6585 active+clean+snaptrim trimq=[211~17] ps=[1fe~13]] removing snap head 2019-12-05T06:51:18.651 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:18.646+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6589 (118'3512,123'6589] local-lis/les=21/22 n=100 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6587 crt=123'6587 lcod 123'6585 mlcod 123'6585 active+clean+snaptrim trimq=[211~17] ps=[1fe~13]] removing snap head 2019-12-05T06:51:18.652 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:18.650+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6262 (117'3215,123'6262] local-lis/les=21/22 n=410 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'6261 crt=123'6259 lcod 123'6259 mlcod 123'6259 active+clean+snaptrim trimq=[209~1f] ps=[1fe~b]] removing snap head 2019-12-05T06:51:18.655 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:18.650+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6264 (117'3215,123'6264] local-lis/les=21/22 n=408 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'6261 crt=123'6261 lcod 123'6259 mlcod 123'6259 active+clean+snaptrim trimq=[209~1f] ps=[1fe~b]] removing snap head 2019-12-05T06:51:18.657 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:18.654+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6386 (118'3309,123'6386] local-lis/les=21/22 n=79 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 crt=123'6386 lcod 123'6384 mlcod 123'6384 active+clean+snaptrim trimq=[211~17] ps=[1fe~13]] removing snap head 2019-12-05T06:51:18.662 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:18.654+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6388 (118'3309,123'6388] local-lis/les=21/22 n=77 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'6386 crt=123'6386 lcod 123'6384 mlcod 123'6384 active+clean+snaptrim trimq=[211~17] ps=[1fe~13]] removing snap head 2019-12-05T06:51:18.664 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:18.658+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6591 (118'3512,123'6591] local-lis/les=21/22 n=98 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'6591 lcod 123'6589 mlcod 123'6589 active+clean+snaptrim trimq=[211~17] ps=[1fe~13]] removing snap head 2019-12-05T06:51:18.664 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:18.658+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6593 (118'3512,123'6593] local-lis/les=21/22 n=96 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6591 crt=123'6591 lcod 123'6589 mlcod 123'6589 active+clean+snaptrim trimq=[211~17] ps=[1fe~13]] removing snap head 2019-12-05T06:51:18.664 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:18.658+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6267 (117'3215,123'6267] local-lis/les=21/22 n=405 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'6266 crt=123'6264 lcod 123'6264 mlcod 123'6264 active+clean+snaptrim trimq=[209~1f] ps=[1fe~b]] removing snap head 2019-12-05T06:51:18.667 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:18.662+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6391 (118'3309,123'6391] local-lis/les=21/22 n=74 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'6390 crt=123'6386 lcod 123'6388 mlcod 123'6388 active+clean+snaptrim trimq=[211~17] ps=[1fe~13]] removing snap head 2019-12-05T06:51:18.674 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:18.666+0000 7fd85ec44700 -1 osd.1 pg_epoch: 123 pg[3.4s0( v 123'6393 (118'3309,123'6393] local-lis/les=21/22 n=72 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [1,2,0,5]p1(0) r=0 lpr=21 luod=123'6390 crt=123'6390 lcod 123'6388 mlcod 123'6388 active+clean+snaptrim trimq=[211~17] ps=[1fe~13]] removing snap head 2019-12-05T06:51:18.675 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:18.665+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5931 (115'2907,123'5931] local-lis/les=21/22 n=592 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5930 crt=123'5928 lcod 123'5928 mlcod 123'5928 active+clean+snaptrim trimq=[203~25] ps=[1fe~5]] removing snap head 2019-12-05T06:51:18.675 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:18.670+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6269 (117'3215,123'6269] local-lis/les=21/22 n=403 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'6269 lcod 123'6267 mlcod 123'6267 active+clean+snaptrim trimq=[20a~1e] ps=[1fe~c]] removing snap head 2019-12-05T06:51:18.676 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:18.670+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6271 (117'3215,123'6271] local-lis/les=21/22 n=401 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'6269 crt=123'6269 lcod 123'6267 mlcod 123'6267 active+clean+snaptrim trimq=[20a~1e] ps=[1fe~c]] removing snap head 2019-12-05T06:51:18.681 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:18.670+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6595 (118'3512,123'6595] local-lis/les=21/22 n=94 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'6595 lcod 123'6593 mlcod 123'6593 active+clean+snaptrim trimq=[211~17] ps=[1fe~13]] removing snap head 2019-12-05T06:51:18.682 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:18.674+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6597 (118'3512,123'6597] local-lis/les=21/22 n=92 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6595 crt=123'6595 lcod 123'6593 mlcod 123'6593 active+clean+snaptrim trimq=[211~17] ps=[1fe~13]] removing snap head 2019-12-05T06:51:18.682 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:18.673+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5934 (115'2907,123'5934] local-lis/les=21/22 n=589 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5933 crt=123'5930 lcod 123'5931 mlcod 123'5931 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:18.685 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:18.678+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'6024 (116'3006,123'6024] local-lis/les=21/22 n=365 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'6024 lcod 123'6022 mlcod 123'6022 active+clean+snaptrim trimq=[209~1f] ps=[1fe~b]] removing snap head 2019-12-05T06:51:18.686 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:18.678+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'6064 (117'3009,123'6064] local-lis/les=21/22 n=331 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'6063 crt=123'6061 lcod 123'6061 mlcod 123'6061 active+clean+snaptrim trimq=[20b~1d] ps=[1fe~d]] removing snap head 2019-12-05T06:51:18.686 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:18.682+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'6026 (116'3006,123'6026] local-lis/les=21/22 n=363 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'6024 crt=123'6024 lcod 123'6022 mlcod 123'6022 active+clean+snaptrim trimq=[209~1f] ps=[1fe~b]] removing snap head 2019-12-05T06:51:18.686 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:18.682+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'6066 (117'3009,123'6066] local-lis/les=21/22 n=329 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'6063 crt=123'6063 lcod 123'6061 mlcod 123'6061 active+clean+snaptrim trimq=[20b~1d] ps=[1fe~d]] removing snap head 2019-12-05T06:51:18.687 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:18.681+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5936 (115'2907,123'5936] local-lis/les=21/22 n=587 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5933 crt=123'5933 lcod 123'5931 mlcod 123'5931 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:18.688 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:18.678+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6273 (117'3215,123'6273] local-lis/les=21/22 n=399 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'6273 lcod 123'6271 mlcod 123'6271 active+clean+snaptrim trimq=[20a~1e] ps=[1fe~c]] removing snap head 2019-12-05T06:51:18.690 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:18.686+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6275 (117'3215,123'6275] local-lis/les=21/22 n=397 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'6273 crt=123'6273 lcod 123'6271 mlcod 123'6271 active+clean+snaptrim trimq=[20a~1e] ps=[1fe~c]] removing snap head 2019-12-05T06:51:18.694 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:18.686+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6599 (118'3512,123'6599] local-lis/les=21/22 n=90 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'6599 lcod 123'6597 mlcod 123'6597 active+clean+snaptrim trimq=[211~17] ps=[1fe~13]] removing snap head 2019-12-05T06:51:18.694 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:18.690+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6601 (118'3512,123'6601] local-lis/les=21/22 n=88 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6599 crt=123'6599 lcod 123'6597 mlcod 123'6597 active+clean+snaptrim trimq=[211~17] ps=[1fe~13]] removing snap head 2019-12-05T06:51:18.697 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:18.694+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'6069 (117'3009,123'6069] local-lis/les=21/22 n=326 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'6068 crt=123'6064 lcod 123'6066 mlcod 123'6066 active+clean+snaptrim trimq=[20b~1d] ps=[1fe~d]] removing snap head 2019-12-05T06:51:18.697 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:18.694+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'6028 (116'3006,123'6028] local-lis/les=21/22 n=361 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'6028 lcod 123'6026 mlcod 123'6026 active+clean+snaptrim trimq=[209~1f] ps=[1fe~b]] removing snap head 2019-12-05T06:51:18.698 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:18.693+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5939 (115'2907,123'5939] local-lis/les=21/22 n=584 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5938 crt=123'5934 lcod 123'5936 mlcod 123'5936 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:18.699 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:18.694+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'6071 (117'3009,123'6071] local-lis/les=21/22 n=324 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'6068 crt=123'6068 lcod 123'6066 mlcod 123'6066 active+clean+snaptrim trimq=[20b~1d] ps=[1fe~d]] removing snap head 2019-12-05T06:51:18.699 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:18.694+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6277 (117'3215,123'6277] local-lis/les=21/22 n=395 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'6277 lcod 123'6275 mlcod 123'6275 active+clean+snaptrim trimq=[20a~1e] ps=[1fe~c]] removing snap head 2019-12-05T06:51:18.701 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:18.693+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5941 (115'2907,123'5941] local-lis/les=21/22 n=582 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5938 crt=123'5938 lcod 123'5936 mlcod 123'5936 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:18.702 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:18.698+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6279 (117'3215,123'6279] local-lis/les=21/22 n=393 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'6277 crt=123'6277 lcod 123'6275 mlcod 123'6275 active+clean+snaptrim trimq=[20a~1e] ps=[1fe~c]] removing snap head 2019-12-05T06:51:18.704 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:18.702+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'6030 (116'3006,123'6030] local-lis/les=21/22 n=359 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'6028 crt=123'6028 lcod 123'6026 mlcod 123'6026 active+clean+snaptrim trimq=[209~1f] ps=[1fe~b]] removing snap head 2019-12-05T06:51:18.710 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:18.706+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'6032 (116'3006,123'6032] local-lis/les=21/22 n=357 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'6032 lcod 123'6030 mlcod 123'6030 active+clean+snaptrim trimq=[209~1f] ps=[1fe~b]] removing snap head 2019-12-05T06:51:18.712 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:18.706+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6282 (117'3215,123'6282] local-lis/les=21/22 n=390 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'6281 crt=123'6277 lcod 123'6279 mlcod 123'6279 active+clean+snaptrim trimq=[20a~1e] ps=[1fe~c]] removing snap head 2019-12-05T06:51:18.713 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:18.710+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'6034 (116'3006,123'6034] local-lis/les=21/22 n=355 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'6032 crt=123'6032 lcod 123'6030 mlcod 123'6030 active+clean+snaptrim trimq=[209~1f] ps=[1fe~b]] removing snap head 2019-12-05T06:51:18.713 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:18.710+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6284 (117'3215,123'6284] local-lis/les=21/22 n=388 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'6281 crt=123'6281 lcod 123'6279 mlcod 123'6279 active+clean+snaptrim trimq=[20a~1e] ps=[1fe~c]] removing snap head 2019-12-05T06:51:18.715 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:18.709+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'6021 (117'3008,123'6021] local-lis/les=21/22 n=248 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'6020 crt=123'6018 lcod 123'6018 mlcod 123'6018 active+clean+snaptrim trimq=[20e~1a] ps=[1fe~10]] removing snap head 2019-12-05T06:51:18.717 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:18.709+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'6023 (117'3008,123'6023] local-lis/les=21/22 n=246 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'6020 crt=123'6020 lcod 123'6018 mlcod 123'6018 active+clean+snaptrim trimq=[20e~1a] ps=[1fe~10]] removing snap head 2019-12-05T06:51:18.720 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:18.718+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6286 (117'3215,123'6286] local-lis/les=21/22 n=386 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'6286 lcod 123'6284 mlcod 123'6284 active+clean+snaptrim trimq=[20a~1e] ps=[1fe~c]] removing snap head 2019-12-05T06:51:18.720 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:18.718+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'6036 (116'3006,123'6036] local-lis/les=21/22 n=353 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'6036 lcod 123'6034 mlcod 123'6034 active+clean+snaptrim trimq=[209~1f] ps=[1fe~b]] removing snap head 2019-12-05T06:51:18.724 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:18.717+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'6026 (117'3008,123'6026] local-lis/les=21/22 n=243 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'6025 crt=123'6020 lcod 123'6023 mlcod 123'6023 active+clean+snaptrim trimq=[20e~1a] ps=[1fe~10]] removing snap head 2019-12-05T06:51:18.726 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:18.722+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6288 (117'3215,123'6288] local-lis/les=21/22 n=384 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'6286 crt=123'6286 lcod 123'6284 mlcod 123'6284 active+clean+snaptrim trimq=[20a~1e] ps=[1fe~c]] removing snap head 2019-12-05T06:51:18.726 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:18.721+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'6028 (117'3008,123'6028] local-lis/les=21/22 n=241 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'6025 crt=123'6025 lcod 123'6023 mlcod 123'6023 active+clean+snaptrim trimq=[20e~1a] ps=[1fe~10]] removing snap head 2019-12-05T06:51:18.727 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:18.722+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'6038 (116'3006,123'6038] local-lis/les=21/22 n=351 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'6038 lcod 123'6036 mlcod 123'6036 active+clean+snaptrim trimq=[20a~1e] ps=[1fe~c]] removing snap head 2019-12-05T06:51:18.729 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:18.726+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'6040 (116'3006,123'6040] local-lis/les=21/22 n=349 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'6038 crt=123'6038 lcod 123'6036 mlcod 123'6036 active+clean+snaptrim trimq=[20a~1e] ps=[1fe~c]] removing snap head 2019-12-05T06:51:18.732 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:18.730+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6290 (117'3215,123'6290] local-lis/les=21/22 n=382 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'6290 lcod 123'6288 mlcod 123'6288 active+clean+snaptrim trimq=[20a~1e] ps=[1fe~c]] removing snap head 2019-12-05T06:51:18.734 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:18.730+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6292 (117'3215,123'6292] local-lis/les=21/22 n=380 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'6290 crt=123'6290 lcod 123'6288 mlcod 123'6288 active+clean+snaptrim trimq=[20a~1e] ps=[1fe~c]] removing snap head 2019-12-05T06:51:18.734 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:18.725+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'6031 (117'3008,123'6031] local-lis/les=21/22 n=238 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'6030 crt=123'6028 lcod 123'6028 mlcod 123'6028 active+clean+snaptrim trimq=[20e~1a] ps=[1fe~10]] removing snap head 2019-12-05T06:51:18.737 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:18.729+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'6033 (117'3008,123'6033] local-lis/les=21/22 n=236 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'6030 crt=123'6030 lcod 123'6028 mlcod 123'6028 active+clean+snaptrim trimq=[20e~1a] ps=[1fe~10]] removing snap head 2019-12-05T06:51:18.739 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:18.734+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'6042 (116'3006,123'6042] local-lis/les=21/22 n=347 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'6042 lcod 123'6040 mlcod 123'6040 active+clean+snaptrim trimq=[20a~1e] ps=[1fe~c]] removing snap head 2019-12-05T06:51:18.740 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:18.734+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'6044 (116'3006,123'6044] local-lis/les=21/22 n=345 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'6042 crt=123'6042 lcod 123'6040 mlcod 123'6040 active+clean+snaptrim trimq=[20a~1e] ps=[1fe~c]] removing snap head 2019-12-05T06:51:18.744 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:18.742+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6604 (118'3512,123'6604] local-lis/les=21/22 n=85 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6603 crt=123'6601 lcod 123'6601 mlcod 123'6601 active+clean+snaptrim trimq=[211~17] ps=[1fe~13]] removing snap head 2019-12-05T06:51:18.748 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:18.742+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6606 (118'3512,123'6606] local-lis/les=21/22 n=83 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6603 crt=123'6603 lcod 123'6601 mlcod 123'6601 active+clean+snaptrim trimq=[211~17] ps=[1fe~13]] removing snap head 2019-12-05T06:51:18.749 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:18.742+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6296 (117'3215,123'6296] local-lis/les=21/22 n=377 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'6296 lcod 123'6294 mlcod 123'6294 active+clean+snaptrim trimq=[20b~1d] ps=[1fe~d]] removing snap head 2019-12-05T06:51:18.750 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:18.746+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6298 (117'3215,123'6298] local-lis/les=21/22 n=375 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'6296 crt=123'6296 lcod 123'6294 mlcod 123'6294 active+clean+snaptrim trimq=[20b~1d] ps=[1fe~d]] removing snap head 2019-12-05T06:51:18.755 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:18.750+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6301 (117'3215,123'6301] local-lis/les=21/22 n=372 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'6300 crt=123'6298 lcod 123'6298 mlcod 123'6298 active+clean+snaptrim trimq=[20b~1d] ps=[1fe~d]] removing snap head 2019-12-05T06:51:18.755 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:18.750+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6303 (117'3215,123'6303] local-lis/les=21/22 n=370 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'6300 crt=123'6300 lcod 123'6298 mlcod 123'6298 active+clean+snaptrim trimq=[20b~1d] ps=[1fe~d]] removing snap head 2019-12-05T06:51:18.759 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:18.754+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6305 (117'3215,123'6305] local-lis/les=21/22 n=368 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'6305 lcod 123'6303 mlcod 123'6303 active+clean+snaptrim trimq=[20b~1d] ps=[1fe~d]] removing snap head 2019-12-05T06:51:18.760 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:18.758+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6307 (117'3215,123'6307] local-lis/les=21/22 n=366 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'6305 crt=123'6305 lcod 123'6303 mlcod 123'6303 active+clean+snaptrim trimq=[20b~1d] ps=[1fe~d]] removing snap head 2019-12-05T06:51:18.761 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:18.753+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5944 (115'2907,123'5944] local-lis/les=21/22 n=579 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5943 crt=123'5941 lcod 123'5941 mlcod 123'5941 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:18.764 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:18.757+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5946 (115'2907,123'5946] local-lis/les=21/22 n=577 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5943 crt=123'5943 lcod 123'5941 mlcod 123'5941 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:18.764 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:18.762+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6309 (117'3215,123'6309] local-lis/les=21/22 n=364 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'6309 lcod 123'6307 mlcod 123'6307 active+clean+snaptrim trimq=[20b~1d] ps=[1fe~d]] removing snap head 2019-12-05T06:51:18.765 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:18.762+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6311 (117'3215,123'6311] local-lis/les=21/22 n=362 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'6309 crt=123'6309 lcod 123'6307 mlcod 123'6307 active+clean+snaptrim trimq=[20b~1d] ps=[1fe~d]] removing snap head 2019-12-05T06:51:18.769 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:18.766+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6313 (117'3215,123'6313] local-lis/les=21/22 n=360 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'6313 lcod 123'6311 mlcod 123'6311 active+clean+snaptrim trimq=[20b~1d] ps=[1fe~d]] removing snap head 2019-12-05T06:51:18.770 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:18.765+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'6036 (117'3008,123'6036] local-lis/les=21/22 n=233 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'6035 crt=123'6033 lcod 123'6033 mlcod 123'6033 active+clean+snaptrim trimq=[20e~1a] ps=[1fe~10]] removing snap head 2019-12-05T06:51:18.771 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:18.766+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6315 (117'3215,123'6315] local-lis/les=21/22 n=358 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'6313 crt=123'6313 lcod 123'6311 mlcod 123'6311 active+clean+snaptrim trimq=[20b~1d] ps=[1fe~d]] removing snap head 2019-12-05T06:51:18.775 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:18.770+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'6046 (116'3006,123'6046] local-lis/les=21/22 n=343 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'6046 lcod 123'6044 mlcod 123'6044 active+clean+snaptrim trimq=[20a~1e] ps=[1fe~c]] removing snap head 2019-12-05T06:51:18.775 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:18.770+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'6048 (116'3006,123'6048] local-lis/les=21/22 n=341 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'6046 crt=123'6046 lcod 123'6044 mlcod 123'6044 active+clean+snaptrim trimq=[20a~1e] ps=[1fe~c]] removing snap head 2019-12-05T06:51:18.779 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:18.774+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6317 (117'3315,123'6317] local-lis/les=21/22 n=356 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'6317 lcod 123'6315 mlcod 123'6315 active+clean+snaptrim trimq=[20b~1d] ps=[1fe~d]] removing snap head 2019-12-05T06:51:18.784 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:18.778+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'6149 (117'3105,123'6149] local-lis/les=21/22 n=70 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'6148 crt=123'6146 lcod 123'6146 mlcod 123'6146 active+clean+snaptrim trimq=[211~17] ps=[1fe~13]] removing snap head 2019-12-05T06:51:18.785 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:18.778+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6319 (117'3315,123'6319] local-lis/les=21/22 n=354 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'6317 crt=123'6317 lcod 123'6315 mlcod 123'6315 active+clean+snaptrim trimq=[20b~1d] ps=[1fe~d]] removing snap head 2019-12-05T06:51:18.785 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:18.778+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'6151 (117'3105,123'6151] local-lis/les=21/22 n=68 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'6148 crt=123'6148 lcod 123'6146 mlcod 123'6146 active+clean+snaptrim trimq=[211~17] ps=[1fe~13]] removing snap head 2019-12-05T06:51:18.786 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:18.782+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6321 (117'3315,123'6321] local-lis/les=21/22 n=352 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'6321 lcod 123'6319 mlcod 123'6319 active+clean+snaptrim trimq=[20b~1d] ps=[1fe~d]] removing snap head 2019-12-05T06:51:18.788 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:18.786+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6323 (117'3315,123'6323] local-lis/les=21/22 n=350 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'6321 crt=123'6321 lcod 123'6319 mlcod 123'6319 active+clean+snaptrim trimq=[20b~1d] ps=[1fe~d]] removing snap head 2019-12-05T06:51:18.793 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:18.790+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6326 (117'3315,123'6326] local-lis/les=21/22 n=347 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'6326 lcod 123'6325 mlcod 123'6325 active+clean+snaptrim trimq=[20b~1d] ps=[1fe~d]] removing snap head 2019-12-05T06:51:18.795 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:18.790+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6328 (117'3315,123'6328] local-lis/les=21/22 n=345 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'6326 crt=123'6326 lcod 123'6325 mlcod 123'6325 active+clean+snaptrim trimq=[20b~1d] ps=[1fe~d]] removing snap head 2019-12-05T06:51:18.795 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:18.789+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5949 (115'2907,123'5949] local-lis/les=21/22 n=574 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5948 crt=123'5946 lcod 123'5946 mlcod 123'5946 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:18.797 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:18.789+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5951 (115'2907,123'5951] local-lis/les=21/22 n=572 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5948 crt=123'5948 lcod 123'5946 mlcod 123'5946 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:18.801 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:18.798+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6332 (117'3315,123'6332] local-lis/les=21/22 n=342 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'6330 crt=123'6330 lcod 123'6328 mlcod 123'6328 active+clean+snaptrim trimq=[20b~1d] ps=[1fe~d]] removing snap head 2019-12-05T06:51:18.805 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:18.802+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'6051 (116'3006,123'6051] local-lis/les=21/22 n=338 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'6050 crt=123'6048 lcod 123'6048 mlcod 123'6048 active+clean+snaptrim trimq=[20a~1e] ps=[1fe~c]] removing snap head 2019-12-05T06:51:18.807 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:18.802+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6334 (117'3315,123'6334] local-lis/les=21/22 n=340 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'6334 lcod 123'6332 mlcod 123'6332 active+clean+snaptrim trimq=[20b~1d] ps=[1fe~d]] removing snap head 2019-12-05T06:51:18.807 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:18.802+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'6053 (116'3006,123'6053] local-lis/les=21/22 n=336 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'6050 crt=123'6050 lcod 123'6048 mlcod 123'6048 active+clean+snaptrim trimq=[20a~1e] ps=[1fe~c]] removing snap head 2019-12-05T06:51:18.809 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:18.806+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6336 (117'3315,123'6336] local-lis/les=21/22 n=338 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'6334 crt=123'6334 lcod 123'6332 mlcod 123'6332 active+clean+snaptrim trimq=[20b~1d] ps=[1fe~d]] removing snap head 2019-12-05T06:51:18.811 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:18.806+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'6055 (116'3006,123'6055] local-lis/les=21/22 n=334 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'6055 lcod 123'6053 mlcod 123'6053 active+clean+snaptrim trimq=[20a~1e] ps=[1fe~c]] removing snap head 2019-12-05T06:51:18.812 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:18.810+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'6057 (116'3006,123'6057] local-lis/les=21/22 n=332 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'6055 crt=123'6055 lcod 123'6053 mlcod 123'6053 active+clean+snaptrim trimq=[20a~1e] ps=[1fe~c]] removing snap head 2019-12-05T06:51:18.816 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:18.814+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'6059 (116'3006,123'6059] local-lis/les=21/22 n=330 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'6059 lcod 123'6057 mlcod 123'6057 active+clean+snaptrim trimq=[20a~1e] ps=[1fe~c]] removing snap head 2019-12-05T06:51:18.818 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:18.814+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'6061 (116'3006,123'6061] local-lis/les=21/22 n=328 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'6059 crt=123'6059 lcod 123'6057 mlcod 123'6057 active+clean+snaptrim trimq=[20a~1e] ps=[1fe~c]] removing snap head 2019-12-05T06:51:18.822 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:18.818+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'6063 (116'3006,123'6063] local-lis/les=21/22 n=326 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'6063 lcod 123'6061 mlcod 123'6061 active+clean+snaptrim trimq=[20a~1e] ps=[1fe~c]] removing snap head 2019-12-05T06:51:18.824 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:18.822+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'6065 (116'3006,123'6065] local-lis/les=21/22 n=324 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'6063 crt=123'6063 lcod 123'6061 mlcod 123'6061 active+clean+snaptrim trimq=[20a~1e] ps=[1fe~c]] removing snap head 2019-12-05T06:51:18.830 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:18.826+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'6067 (116'3006,123'6067] local-lis/les=21/22 n=322 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'6067 lcod 123'6065 mlcod 123'6065 active+clean+snaptrim trimq=[20a~1e] ps=[1fe~c]] removing snap head 2019-12-05T06:51:18.835 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:18.830+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'6069 (116'3006,123'6069] local-lis/les=21/22 n=320 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'6069 lcod 123'6067 mlcod 123'6067 active+clean+snaptrim trimq=[20b~1d] ps=[1fe~d]] removing snap head 2019-12-05T06:51:18.836 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:18.834+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'6071 (116'3006,123'6071] local-lis/les=21/22 n=318 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'6069 crt=123'6069 lcod 123'6067 mlcod 123'6067 active+clean+snaptrim trimq=[20b~1d] ps=[1fe~d]] removing snap head 2019-12-05T06:51:18.841 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:18.838+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'6073 (116'3006,123'6073] local-lis/les=21/22 n=316 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'6073 lcod 123'6071 mlcod 123'6071 active+clean+snaptrim trimq=[20b~1d] ps=[1fe~d]] removing snap head 2019-12-05T06:51:18.842 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:18.838+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'6075 (116'3006,123'6075] local-lis/les=21/22 n=314 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'6073 crt=123'6073 lcod 123'6071 mlcod 123'6071 active+clean+snaptrim trimq=[20b~1d] ps=[1fe~d]] removing snap head 2019-12-05T06:51:18.847 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:18.842+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'6077 (116'3006,123'6077] local-lis/les=21/22 n=312 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'6077 lcod 123'6075 mlcod 123'6075 active+clean+snaptrim trimq=[20b~1d] ps=[1fe~d]] removing snap head 2019-12-05T06:51:18.848 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:18.846+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'6079 (116'3006,123'6079] local-lis/les=21/22 n=310 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'6077 crt=123'6077 lcod 123'6075 mlcod 123'6075 active+clean+snaptrim trimq=[20b~1d] ps=[1fe~d]] removing snap head 2019-12-05T06:51:18.853 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:18.850+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'6081 (116'3006,123'6081] local-lis/les=21/22 n=308 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'6081 lcod 123'6079 mlcod 123'6079 active+clean+snaptrim trimq=[20b~1d] ps=[1fe~d]] removing snap head 2019-12-05T06:51:18.854 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:18.850+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'6083 (116'3006,123'6083] local-lis/les=21/22 n=306 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'6081 crt=123'6081 lcod 123'6079 mlcod 123'6079 active+clean+snaptrim trimq=[20b~1d] ps=[1fe~d]] removing snap head 2019-12-05T06:51:18.858 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:18.854+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'6085 (116'3006,123'6085] local-lis/les=21/22 n=304 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'6085 lcod 123'6083 mlcod 123'6083 active+clean+snaptrim trimq=[20b~1d] ps=[1fe~d]] removing snap head 2019-12-05T06:51:18.860 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:18.854+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'6087 (116'3006,123'6087] local-lis/les=21/22 n=302 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'6085 crt=123'6085 lcod 123'6083 mlcod 123'6083 active+clean+snaptrim trimq=[20b~1d] ps=[1fe~d]] removing snap head 2019-12-05T06:51:18.864 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:18.858+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'6089 (116'3006,123'6089] local-lis/les=21/22 n=300 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'6089 lcod 123'6087 mlcod 123'6087 active+clean+snaptrim trimq=[20b~1d] ps=[1fe~d]] removing snap head 2019-12-05T06:51:18.867 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:18.862+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'6091 (116'3006,123'6091] local-lis/les=21/22 n=298 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'6091 lcod 123'6089 mlcod 123'6089 active+clean+snaptrim trimq=[20c~1c] ps=[1fe~e]] removing snap head 2019-12-05T06:51:18.868 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:18.866+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'6093 (116'3006,123'6093] local-lis/les=21/22 n=296 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'6091 crt=123'6091 lcod 123'6089 mlcod 123'6089 active+clean+snaptrim trimq=[20c~1c] ps=[1fe~e]] removing snap head 2019-12-05T06:51:18.872 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:18.870+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'6095 (116'3006,123'6095] local-lis/les=21/22 n=294 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'6095 lcod 123'6093 mlcod 123'6093 active+clean+snaptrim trimq=[20c~1c] ps=[1fe~e]] removing snap head 2019-12-05T06:51:18.874 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:18.870+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'6097 (116'3006,123'6097] local-lis/les=21/22 n=292 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'6095 crt=123'6095 lcod 123'6093 mlcod 123'6093 active+clean+snaptrim trimq=[20c~1c] ps=[1fe~e]] removing snap head 2019-12-05T06:51:19.191 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:19.186+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'6099 (116'3006,123'6099] local-lis/les=21/22 n=290 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'6099 lcod 123'6097 mlcod 123'6097 active+clean+snaptrim trimq=[20c~1c] ps=[1fe~e]] removing snap head 2019-12-05T06:51:19.193 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:19.185+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'6038 (117'3008,123'6038] local-lis/les=21/22 n=231 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'6038 lcod 123'6036 mlcod 123'6036 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:19.195 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:19.190+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'6101 (116'3006,123'6101] local-lis/les=21/22 n=288 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'6099 crt=123'6099 lcod 123'6097 mlcod 123'6097 active+clean+snaptrim trimq=[20c~1c] ps=[1fe~e]] removing snap head 2019-12-05T06:51:19.199 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:19.193+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'6040 (117'3008,123'6040] local-lis/les=21/22 n=229 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'6038 crt=123'6038 lcod 123'6036 mlcod 123'6036 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:19.205 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:19.202+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6608 (118'3512,123'6608] local-lis/les=21/22 n=81 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'6608 lcod 123'6606 mlcod 123'6606 active+clean+snaptrim trimq=[211~17] ps=[1fe~13]] removing snap head 2019-12-05T06:51:19.208 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:19.202+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6610 (118'3512,123'6610] local-lis/les=21/22 n=79 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6608 crt=123'6608 lcod 123'6606 mlcod 123'6606 active+clean+snaptrim trimq=[211~17] ps=[1fe~13]] removing snap head 2019-12-05T06:51:19.220 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:19.218+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'6103 (116'3006,123'6103] local-lis/les=21/22 n=286 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'6103 lcod 123'6101 mlcod 123'6101 active+clean+snaptrim trimq=[20c~1c] ps=[1fe~e]] removing snap head 2019-12-05T06:51:19.221 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:19.213+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'6042 (117'3008,123'6042] local-lis/les=21/22 n=227 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'6042 lcod 123'6040 mlcod 123'6040 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:19.223 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:19.218+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'6105 (116'3006,123'6105] local-lis/les=21/22 n=284 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'6103 crt=123'6103 lcod 123'6101 mlcod 123'6101 active+clean+snaptrim trimq=[20c~1c] ps=[1fe~e]] removing snap head 2019-12-05T06:51:19.223 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:19.217+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'6044 (117'3008,123'6044] local-lis/les=21/22 n=225 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'6042 crt=123'6042 lcod 123'6040 mlcod 123'6040 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:19.250 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:19.241+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'6046 (117'3008,123'6046] local-lis/les=21/22 n=223 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'6046 lcod 123'6044 mlcod 123'6044 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:19.251 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:19.246+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'6107 (116'3006,123'6107] local-lis/les=21/22 n=282 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'6107 lcod 123'6105 mlcod 123'6105 active+clean+snaptrim trimq=[20c~1c] ps=[1fe~e]] removing snap head 2019-12-05T06:51:19.253 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:19.245+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'6048 (117'3008,123'6048] local-lis/les=21/22 n=221 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'6046 crt=123'6046 lcod 123'6044 mlcod 123'6044 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:19.254 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:19.250+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'6153 (117'3105,123'6153] local-lis/les=21/22 n=66 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'6153 lcod 123'6151 mlcod 123'6151 active+clean+snaptrim trimq=[211~17] ps=[1fe~13]] removing snap head 2019-12-05T06:51:19.254 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:19.250+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'6109 (116'3107,123'6109] local-lis/les=21/22 n=280 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'6107 crt=123'6107 lcod 123'6105 mlcod 123'6105 active+clean+snaptrim trimq=[20c~1c] ps=[1fe~e]] removing snap head 2019-12-05T06:51:19.255 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:19.250+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'6155 (117'3105,123'6155] local-lis/les=21/22 n=64 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'6153 crt=123'6153 lcod 123'6151 mlcod 123'6151 active+clean+snaptrim trimq=[211~17] ps=[1fe~13]] removing snap head 2019-12-05T06:51:19.275 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:19.270+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6612 (118'3512,123'6612] local-lis/les=21/22 n=77 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'6612 lcod 123'6610 mlcod 123'6610 active+clean+snaptrim trimq=[211~17] ps=[1fe~13]] removing snap head 2019-12-05T06:51:19.278 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:19.274+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6614 (118'3612,123'6614] local-lis/les=21/22 n=75 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6612 crt=123'6612 lcod 123'6610 mlcod 123'6610 active+clean+snaptrim trimq=[211~17] ps=[1fe~13]] removing snap head 2019-12-05T06:51:19.284 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:19.282+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'6111 (116'3107,123'6111] local-lis/les=21/22 n=278 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'6111 lcod 123'6109 mlcod 123'6109 active+clean+snaptrim trimq=[20c~1c] ps=[1fe~e]] removing snap head 2019-12-05T06:51:19.285 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:19.277+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'6050 (117'3008,123'6050] local-lis/les=21/22 n=219 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'6050 lcod 123'6048 mlcod 123'6048 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:19.287 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:19.282+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'6113 (116'3107,123'6113] local-lis/les=21/22 n=276 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'6111 crt=123'6111 lcod 123'6109 mlcod 123'6109 active+clean+snaptrim trimq=[20c~1c] ps=[1fe~e]] removing snap head 2019-12-05T06:51:19.288 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:19.281+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'6052 (117'3008,123'6052] local-lis/les=21/22 n=217 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'6050 crt=123'6050 lcod 123'6048 mlcod 123'6048 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:19.290 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:19.286+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6616 (118'3612,123'6616] local-lis/les=21/22 n=73 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'6616 lcod 123'6614 mlcod 123'6614 active+clean+snaptrim trimq=[211~17] ps=[1fe~13]] removing snap head 2019-12-05T06:51:19.292 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:19.290+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6618 (118'3612,123'6618] local-lis/les=21/22 n=71 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6616 crt=123'6616 lcod 123'6614 mlcod 123'6614 active+clean+snaptrim trimq=[211~17] ps=[1fe~13]] removing snap head 2019-12-05T06:51:19.298 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:19.294+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6620 (118'3612,123'6620] local-lis/les=21/22 n=69 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'6620 lcod 123'6618 mlcod 123'6618 active+clean+snaptrim trimq=[211~17] ps=[1fe~13]] removing snap head 2019-12-05T06:51:19.300 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:19.298+0000 7fd864c50700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6622 (118'3612,123'6622] local-lis/les=21/22 n=67 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6620 crt=123'6620 lcod 123'6618 mlcod 123'6618 active+clean+snaptrim trimq=[211~17] ps=[1fe~13]] removing snap head 2019-12-05T06:51:19.305 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:19.302+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6624 (118'3612,123'6624] local-lis/les=21/22 n=65 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'6624 lcod 123'6622 mlcod 123'6622 active+clean+snaptrim trimq=[211~17] ps=[1fe~13]] removing snap head 2019-12-05T06:51:19.307 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:19.302+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6626 (118'3612,123'6626] local-lis/les=21/22 n=63 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6624 crt=123'6624 lcod 123'6622 mlcod 123'6622 active+clean+snaptrim trimq=[211~17] ps=[1fe~13]] removing snap head 2019-12-05T06:51:19.313 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:19.310+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6628 (118'3612,123'6628] local-lis/les=21/22 n=61 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'6628 lcod 123'6626 mlcod 123'6626 active+clean+snaptrim trimq=[211~17] ps=[1fe~13]] removing snap head 2019-12-05T06:51:19.315 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:19.310+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6630 (118'3612,123'6630] local-lis/les=21/22 n=59 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 luod=123'6628 crt=123'6628 lcod 123'6626 mlcod 123'6626 active+clean+snaptrim trimq=[211~17] ps=[1fe~13]] removing snap head 2019-12-05T06:51:19.321 INFO:tasks.ceph.osd.1.smithi042.stderr:2019-12-05T06:51:19.318+0000 7fd860c48700 -1 osd.1 pg_epoch: 123 pg[3.0s0( v 123'6632 (118'3612,123'6632] local-lis/les=21/22 n=57 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [1,2,6,5]p1(0) r=0 lpr=21 crt=123'6632 lcod 123'6630 mlcod 123'6630 active+clean+snaptrim trimq=[211~17] ps=[1fe~13]] removing snap head 2019-12-05T06:51:19.328 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:19.321+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'6054 (117'3008,123'6054] local-lis/les=21/22 n=215 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'6054 lcod 123'6052 mlcod 123'6052 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:19.331 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:19.325+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'6056 (117'3008,123'6056] local-lis/les=21/22 n=213 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'6054 crt=123'6054 lcod 123'6052 mlcod 123'6052 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:19.332 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:19.326+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'6115 (116'3107,123'6115] local-lis/les=21/22 n=274 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'6115 lcod 123'6113 mlcod 123'6113 active+clean+snaptrim trimq=[20c~1c] ps=[1fe~e]] removing snap head 2019-12-05T06:51:19.332 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:19.326+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'6117 (116'3107,123'6117] local-lis/les=21/22 n=272 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'6115 crt=123'6115 lcod 123'6113 mlcod 123'6113 active+clean+snaptrim trimq=[20c~1c] ps=[1fe~e]] removing snap head 2019-12-05T06:51:19.376 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:19.369+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5953 (115'2907,123'5953] local-lis/les=21/22 n=570 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=123'5953 lcod 123'5951 mlcod 123'5951 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:19.378 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:19.373+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5955 (115'2907,123'5955] local-lis/les=21/22 n=568 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5953 crt=123'5953 lcod 123'5951 mlcod 123'5951 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:19.387 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:19.377+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5957 (115'2907,123'5957] local-lis/les=21/22 n=566 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=123'5957 lcod 123'5955 mlcod 123'5955 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:19.388 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:19.377+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5959 (115'2907,123'5959] local-lis/les=21/22 n=564 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5957 crt=123'5957 lcod 123'5955 mlcod 123'5955 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:19.388 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:19.381+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5961 (115'2907,123'5961] local-lis/les=21/22 n=562 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=123'5961 lcod 123'5959 mlcod 123'5959 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:19.391 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:19.381+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5963 (115'2907,123'5963] local-lis/les=21/22 n=560 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5961 crt=123'5961 lcod 123'5959 mlcod 123'5959 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:19.396 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:19.390+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'6119 (116'3107,123'6119] local-lis/les=21/22 n=270 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'6119 lcod 123'6117 mlcod 123'6117 active+clean+snaptrim trimq=[20c~1c] ps=[1fe~e]] removing snap head 2019-12-05T06:51:19.396 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:19.394+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'6157 (117'3105,123'6157] local-lis/les=21/22 n=62 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'6157 lcod 123'6155 mlcod 123'6155 active+clean+snaptrim trimq=[211~17] ps=[1fe~13]] removing snap head 2019-12-05T06:51:19.396 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:19.394+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'6121 (116'3107,123'6121] local-lis/les=21/22 n=268 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'6119 crt=123'6119 lcod 123'6117 mlcod 123'6117 active+clean+snaptrim trimq=[20c~1c] ps=[1fe~e]] removing snap head 2019-12-05T06:51:19.397 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:19.385+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'6058 (117'3008,123'6058] local-lis/les=21/22 n=211 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'6058 lcod 123'6056 mlcod 123'6056 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:19.397 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:19.389+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'6060 (117'3008,123'6060] local-lis/les=21/22 n=209 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'6058 crt=123'6058 lcod 123'6056 mlcod 123'6056 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:19.398 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:19.389+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5965 (115'2907,123'5965] local-lis/les=21/22 n=558 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=123'5965 lcod 123'5963 mlcod 123'5963 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:19.399 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:19.394+0000 7fbf8749e700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'6159 (117'3105,123'6159] local-lis/les=21/22 n=60 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 luod=123'6157 crt=123'6157 lcod 123'6155 mlcod 123'6155 active+clean+snaptrim trimq=[211~17] ps=[1fe~13]] removing snap head 2019-12-05T06:51:19.401 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:19.393+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5967 (115'2907,123'5967] local-lis/les=21/22 n=556 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5965 crt=123'5965 lcod 123'5963 mlcod 123'5963 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:19.405 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:19.402+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'6123 (116'3107,123'6123] local-lis/les=21/22 n=266 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'6123 lcod 123'6121 mlcod 123'6121 active+clean+snaptrim trimq=[20c~1c] ps=[1fe~e]] removing snap head 2019-12-05T06:51:19.406 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:19.402+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'6125 (116'3107,123'6125] local-lis/les=21/22 n=264 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'6123 crt=123'6123 lcod 123'6121 mlcod 123'6121 active+clean+snaptrim trimq=[20c~1c] ps=[1fe~e]] removing snap head 2019-12-05T06:51:19.408 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:19.401+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5969 (115'2907,123'5969] local-lis/les=21/22 n=554 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=123'5969 lcod 123'5967 mlcod 123'5967 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:19.410 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:19.406+0000 7fbf83496700 -1 osd.0 pg_epoch: 123 pg[3.6s0( v 123'6161 (117'3105,123'6161] local-lis/les=21/22 n=58 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,1,4,7]p0(0) r=0 lpr=21 crt=123'6161 lcod 123'6159 mlcod 123'6159 active+clean+snaptrim trimq=[211~17] ps=[1fe~13]] removing snap head 2019-12-05T06:51:19.413 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:19.405+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5971 (115'2907,123'5971] local-lis/les=21/22 n=552 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5969 crt=123'5969 lcod 123'5967 mlcod 123'5967 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:19.415 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:19.410+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'6127 (116'3107,123'6127] local-lis/les=21/22 n=262 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'6127 lcod 123'6125 mlcod 123'6125 active+clean+snaptrim trimq=[20c~1c] ps=[1fe~e]] removing snap head 2019-12-05T06:51:19.420 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:19.413+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5973 (115'2907,123'5973] local-lis/les=21/22 n=550 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=123'5973 lcod 123'5971 mlcod 123'5971 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:19.427 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:19.418+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'6129 (116'3107,123'6129] local-lis/les=21/22 n=260 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'6129 lcod 123'6127 mlcod 123'6127 active+clean+snaptrim trimq=[20d~1b] ps=[1fe~f]] removing snap head 2019-12-05T06:51:19.427 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:19.422+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'6131 (116'3107,123'6131] local-lis/les=21/22 n=258 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'6129 crt=123'6129 lcod 123'6127 mlcod 123'6127 active+clean+snaptrim trimq=[20d~1b] ps=[1fe~f]] removing snap head 2019-12-05T06:51:19.431 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:19.426+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'6133 (116'3107,123'6133] local-lis/les=21/22 n=256 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'6133 lcod 123'6131 mlcod 123'6131 active+clean+snaptrim trimq=[20d~1b] ps=[1fe~f]] removing snap head 2019-12-05T06:51:19.432 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:19.430+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'6135 (116'3107,123'6135] local-lis/les=21/22 n=254 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'6133 crt=123'6133 lcod 123'6131 mlcod 123'6131 active+clean+snaptrim trimq=[20d~1b] ps=[1fe~f]] removing snap head 2019-12-05T06:51:19.436 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:19.429+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5977 (115'2907,123'5977] local-lis/les=21/22 n=547 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=123'5977 lcod 123'5975 mlcod 123'5975 active+clean+snaptrim trimq=[204~24] ps=[1fe~6]] removing snap head 2019-12-05T06:51:19.437 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:19.434+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'6137 (116'3107,123'6137] local-lis/les=21/22 n=252 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'6137 lcod 123'6135 mlcod 123'6135 active+clean+snaptrim trimq=[20d~1b] ps=[1fe~f]] removing snap head 2019-12-05T06:51:19.440 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:19.434+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'6139 (116'3107,123'6139] local-lis/les=21/22 n=250 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'6137 crt=123'6137 lcod 123'6135 mlcod 123'6135 active+clean+snaptrim trimq=[20d~1b] ps=[1fe~f]] removing snap head 2019-12-05T06:51:19.444 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:19.442+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'6141 (116'3107,123'6141] local-lis/les=21/22 n=248 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'6141 lcod 123'6139 mlcod 123'6139 active+clean+snaptrim trimq=[20e~1a] ps=[1fe~10]] removing snap head 2019-12-05T06:51:19.445 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:19.437+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5979 (115'2907,123'5979] local-lis/les=21/22 n=545 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=123'5979 lcod 123'5977 mlcod 123'5977 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:19.446 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:19.442+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'6143 (116'3107,123'6143] local-lis/les=21/22 n=246 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'6141 crt=123'6141 lcod 123'6139 mlcod 123'6139 active+clean+snaptrim trimq=[20e~1a] ps=[1fe~10]] removing snap head 2019-12-05T06:51:19.447 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:19.441+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5981 (115'2907,123'5981] local-lis/les=21/22 n=543 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5979 crt=123'5979 lcod 123'5977 mlcod 123'5977 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:19.447 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:19.441+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'6062 (117'3008,123'6062] local-lis/les=21/22 n=207 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'6062 lcod 123'6060 mlcod 123'6060 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:19.456 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:19.450+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'6145 (116'3107,123'6145] local-lis/les=21/22 n=244 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'6145 lcod 123'6143 mlcod 123'6143 active+clean+snaptrim trimq=[20e~1a] ps=[1fe~10]] removing snap head 2019-12-05T06:51:19.457 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:19.450+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'6147 (116'3107,123'6147] local-lis/les=21/22 n=242 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'6145 crt=123'6145 lcod 123'6143 mlcod 123'6143 active+clean+snaptrim trimq=[20e~1a] ps=[1fe~10]] removing snap head 2019-12-05T06:51:19.457 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:19.445+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'6064 (117'3008,123'6064] local-lis/les=21/22 n=205 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'6062 crt=123'6062 lcod 123'6060 mlcod 123'6060 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:19.457 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:19.449+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5983 (115'2907,123'5983] local-lis/les=21/22 n=541 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=123'5983 lcod 123'5981 mlcod 123'5981 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:19.458 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:19.449+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5985 (115'2907,123'5985] local-lis/les=21/22 n=539 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5983 crt=123'5983 lcod 123'5981 mlcod 123'5981 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:19.459 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:19.453+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'6066 (117'3008,123'6066] local-lis/les=21/22 n=203 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'6066 lcod 123'6064 mlcod 123'6064 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:19.462 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:19.453+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'6068 (117'3008,123'6068] local-lis/les=21/22 n=201 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'6066 crt=123'6066 lcod 123'6064 mlcod 123'6064 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:19.465 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:19.457+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5987 (115'2907,123'5987] local-lis/les=21/22 n=537 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=123'5987 lcod 123'5985 mlcod 123'5985 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:19.471 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:19.462+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'6149 (116'3107,123'6149] local-lis/les=21/22 n=240 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'6149 lcod 123'6147 mlcod 123'6147 active+clean+snaptrim trimq=[20e~1a] ps=[1fe~10]] removing snap head 2019-12-05T06:51:19.472 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:19.462+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'6151 (116'3107,123'6151] local-lis/les=21/22 n=238 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'6149 crt=123'6149 lcod 123'6147 mlcod 123'6147 active+clean+snaptrim trimq=[20e~1a] ps=[1fe~10]] removing snap head 2019-12-05T06:51:19.472 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:19.461+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'6070 (117'3008,123'6070] local-lis/les=21/22 n=199 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'6070 lcod 123'6068 mlcod 123'6068 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:19.473 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:19.465+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'6072 (117'3008,123'6072] local-lis/les=21/22 n=197 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'6070 crt=123'6070 lcod 123'6068 mlcod 123'6068 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:19.473 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:19.465+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5989 (115'2907,123'5989] local-lis/les=21/22 n=535 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5987 crt=123'5987 lcod 123'5985 mlcod 123'5985 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:19.475 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:19.466+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6339 (117'3315,123'6339] local-lis/les=21/22 n=335 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'6338 crt=123'6336 lcod 123'6336 mlcod 123'6336 active+clean+snaptrim trimq=[20b~1d] ps=[1fe~d]] removing snap head 2019-12-05T06:51:19.475 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:19.470+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6341 (117'3315,123'6341] local-lis/les=21/22 n=333 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'6338 crt=123'6338 lcod 123'6336 mlcod 123'6336 active+clean+snaptrim trimq=[20b~1d] ps=[1fe~d]] removing snap head 2019-12-05T06:51:19.478 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:19.474+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'6153 (116'3107,123'6153] local-lis/les=21/22 n=236 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'6153 lcod 123'6151 mlcod 123'6151 active+clean+snaptrim trimq=[20e~1a] ps=[1fe~10]] removing snap head 2019-12-05T06:51:19.483 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:19.477+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5993 (115'2907,123'5993] local-lis/les=21/22 n=531 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5991 crt=123'5989 lcod 123'5989 mlcod 123'5989 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:19.483 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:19.478+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'6155 (116'3107,123'6155] local-lis/les=21/22 n=234 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'6153 crt=123'6153 lcod 123'6151 mlcod 123'6151 active+clean+snaptrim trimq=[20e~1a] ps=[1fe~10]] removing snap head 2019-12-05T06:51:19.485 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:19.477+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'6074 (117'3008,123'6074] local-lis/les=21/22 n=195 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'6074 lcod 123'6072 mlcod 123'6072 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:19.486 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:19.481+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5995 (115'2907,123'5995] local-lis/les=21/22 n=529 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5991 crt=123'5991 lcod 123'5989 mlcod 123'5989 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:19.487 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:19.482+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'6157 (116'3107,123'6157] local-lis/les=21/22 n=232 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'6157 lcod 123'6155 mlcod 123'6155 active+clean+snaptrim trimq=[20e~1a] ps=[1fe~10]] removing snap head 2019-12-05T06:51:19.488 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:19.481+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'6076 (117'3008,123'6076] local-lis/les=21/22 n=193 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'6074 crt=123'6074 lcod 123'6072 mlcod 123'6072 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:19.490 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:19.486+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'6159 (116'3107,123'6159] local-lis/les=21/22 n=230 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'6157 crt=123'6157 lcod 123'6155 mlcod 123'6155 active+clean+snaptrim trimq=[20e~1a] ps=[1fe~10]] removing snap head 2019-12-05T06:51:19.499 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:19.485+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5997 (115'2907,123'5997] local-lis/les=21/22 n=527 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=123'5997 lcod 123'5995 mlcod 123'5995 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:19.500 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:19.489+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'5999 (115'2907,123'5999] local-lis/les=21/22 n=525 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'5997 crt=123'5997 lcod 123'5995 mlcod 123'5995 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:19.500 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:19.489+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'6078 (117'3008,123'6078] local-lis/les=21/22 n=191 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'6078 lcod 123'6076 mlcod 123'6076 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:19.501 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:19.498+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'6162 (116'3107,123'6162] local-lis/les=21/22 n=227 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'6162 lcod 123'6161 mlcod 123'6161 active+clean+snaptrim trimq=[20e~1a] ps=[1fe~10]] removing snap head 2019-12-05T06:51:19.501 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:19.493+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'6080 (117'3008,123'6080] local-lis/les=21/22 n=189 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'6078 crt=123'6078 lcod 123'6076 mlcod 123'6076 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:19.503 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:19.498+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'6164 (116'3107,123'6164] local-lis/les=21/22 n=225 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'6162 crt=123'6162 lcod 123'6161 mlcod 123'6161 active+clean+snaptrim trimq=[20e~1a] ps=[1fe~10]] removing snap head 2019-12-05T06:51:19.509 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:19.501+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'6001 (115'2907,123'6001] local-lis/les=21/22 n=523 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=123'6001 lcod 123'5999 mlcod 123'5999 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:19.509 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:19.497+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'6083 (117'3008,123'6083] local-lis/les=21/22 n=186 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'6082 crt=123'6080 lcod 123'6080 mlcod 123'6080 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:19.510 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:19.501+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'6085 (117'3008,123'6085] local-lis/les=21/22 n=184 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'6082 crt=123'6082 lcod 123'6080 mlcod 123'6080 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:19.512 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:19.506+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'6166 (116'3107,123'6166] local-lis/les=21/22 n=223 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'6166 lcod 123'6164 mlcod 123'6164 active+clean+snaptrim trimq=[20e~1a] ps=[1fe~10]] removing snap head 2019-12-05T06:51:19.515 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:19.505+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'6003 (115'2907,123'6003] local-lis/les=21/22 n=521 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'6001 crt=123'6001 lcod 123'5999 mlcod 123'5999 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:19.515 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:19.510+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'6168 (116'3107,123'6168] local-lis/les=21/22 n=221 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'6166 crt=123'6166 lcod 123'6164 mlcod 123'6164 active+clean+snaptrim trimq=[20e~1a] ps=[1fe~10]] removing snap head 2019-12-05T06:51:19.515 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:19.510+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'6074 (117'3009,123'6074] local-lis/les=21/22 n=321 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'6073 crt=123'6071 lcod 123'6071 mlcod 123'6071 active+clean+snaptrim trimq=[20b~1d] ps=[1fe~d]] removing snap head 2019-12-05T06:51:19.516 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:19.509+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'6087 (117'3008,123'6087] local-lis/les=21/22 n=182 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'6087 lcod 123'6085 mlcod 123'6085 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:19.517 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:19.514+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'6076 (117'3009,123'6076] local-lis/les=21/22 n=319 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'6073 crt=123'6073 lcod 123'6071 mlcod 123'6071 active+clean+snaptrim trimq=[20b~1d] ps=[1fe~d]] removing snap head 2019-12-05T06:51:19.523 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:19.513+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'6089 (117'3008,123'6089] local-lis/les=21/22 n=180 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'6087 crt=123'6087 lcod 123'6085 mlcod 123'6085 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:19.525 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:19.517+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'6005 (115'2907,123'6005] local-lis/les=21/22 n=519 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=123'6005 lcod 123'6003 mlcod 123'6003 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:19.529 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:19.522+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'6079 (117'3009,123'6079] local-lis/les=21/22 n=316 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'6078 crt=123'6074 lcod 123'6076 mlcod 123'6076 active+clean+snaptrim trimq=[20b~1d] ps=[1fe~d]] removing snap head 2019-12-05T06:51:19.529 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:19.522+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'6170 (116'3107,123'6170] local-lis/les=21/22 n=219 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'6170 lcod 123'6168 mlcod 123'6168 active+clean+snaptrim trimq=[20e~1a] ps=[1fe~10]] removing snap head 2019-12-05T06:51:19.529 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:19.522+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'6081 (117'3009,123'6081] local-lis/les=21/22 n=314 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'6078 crt=123'6078 lcod 123'6076 mlcod 123'6076 active+clean+snaptrim trimq=[20b~1d] ps=[1fe~d]] removing snap head 2019-12-05T06:51:19.529 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:19.526+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'6172 (116'3107,123'6172] local-lis/les=21/22 n=217 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'6170 crt=123'6170 lcod 123'6168 mlcod 123'6168 active+clean+snaptrim trimq=[20e~1a] ps=[1fe~10]] removing snap head 2019-12-05T06:51:19.530 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:19.517+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'6007 (115'2907,123'6007] local-lis/les=21/22 n=517 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'6005 crt=123'6005 lcod 123'6003 mlcod 123'6003 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:19.535 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:19.529+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'6010 (116'3007,123'6010] local-lis/les=21/22 n=514 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'6009 crt=123'6007 lcod 123'6007 mlcod 123'6007 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:19.537 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:19.534+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'6174 (116'3107,123'6174] local-lis/les=21/22 n=215 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'6174 lcod 123'6172 mlcod 123'6172 active+clean+snaptrim trimq=[20e~1a] ps=[1fe~10]] removing snap head 2019-12-05T06:51:19.538 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:19.529+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'6012 (116'3007,123'6012] local-lis/les=21/22 n=512 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'6009 crt=123'6009 lcod 123'6007 mlcod 123'6007 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:19.544 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:19.537+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'6014 (116'3007,123'6014] local-lis/les=21/22 n=510 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=123'6014 lcod 123'6012 mlcod 123'6012 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:19.546 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:19.542+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'6177 (116'3107,123'6177] local-lis/les=21/22 n=212 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'6177 lcod 123'6176 mlcod 123'6176 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:19.548 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:19.546+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'6179 (116'3107,123'6179] local-lis/les=21/22 n=210 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'6177 crt=123'6177 lcod 123'6176 mlcod 123'6176 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:19.548 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:19.541+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'6016 (116'3007,123'6016] local-lis/les=21/22 n=508 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'6014 crt=123'6014 lcod 123'6012 mlcod 123'6012 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:19.549 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:19.541+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'6092 (117'3008,123'6092] local-lis/les=21/22 n=177 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'6091 crt=123'6089 lcod 123'6089 mlcod 123'6089 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:19.553 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:19.550+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'6083 (117'3009,123'6083] local-lis/les=21/22 n=312 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'6083 lcod 123'6081 mlcod 123'6081 active+clean+snaptrim trimq=[20b~1d] ps=[1fe~d]] removing snap head 2019-12-05T06:51:19.560 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:19.550+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'6085 (117'3009,123'6085] local-lis/les=21/22 n=310 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'6083 crt=123'6083 lcod 123'6081 mlcod 123'6081 active+clean+snaptrim trimq=[20b~1d] ps=[1fe~d]] removing snap head 2019-12-05T06:51:19.560 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:19.554+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'6182 (116'3107,123'6182] local-lis/les=21/22 n=207 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'6181 crt=123'6179 lcod 123'6179 mlcod 123'6179 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:19.561 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:19.549+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'6018 (116'3007,123'6018] local-lis/les=21/22 n=506 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=123'6018 lcod 123'6016 mlcod 123'6016 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:19.561 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:19.553+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'6020 (116'3007,123'6020] local-lis/les=21/22 n=504 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'6018 crt=123'6018 lcod 123'6016 mlcod 123'6016 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:19.561 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:19.558+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'6184 (116'3107,123'6184] local-lis/les=21/22 n=205 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'6181 crt=123'6181 lcod 123'6179 mlcod 123'6179 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:19.564 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:19.562+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'6087 (117'3009,123'6087] local-lis/les=21/22 n=308 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'6087 lcod 123'6085 mlcod 123'6085 active+clean+snaptrim trimq=[20b~1d] ps=[1fe~d]] removing snap head 2019-12-05T06:51:19.566 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:19.562+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'6089 (117'3009,123'6089] local-lis/les=21/22 n=306 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'6087 crt=123'6087 lcod 123'6085 mlcod 123'6085 active+clean+snaptrim trimq=[20b~1d] ps=[1fe~d]] removing snap head 2019-12-05T06:51:19.566 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:19.557+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'6022 (116'3007,123'6022] local-lis/les=21/22 n=502 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=123'6022 lcod 123'6020 mlcod 123'6020 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:19.568 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:19.561+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'6024 (116'3007,123'6024] local-lis/les=21/22 n=500 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'6022 crt=123'6022 lcod 123'6020 mlcod 123'6020 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:19.572 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:19.566+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'6186 (116'3107,123'6186] local-lis/les=21/22 n=203 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'6186 lcod 123'6184 mlcod 123'6184 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:19.573 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:19.570+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'6188 (116'3107,123'6188] local-lis/les=21/22 n=201 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'6186 crt=123'6186 lcod 123'6184 mlcod 123'6184 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:19.574 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:19.570+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'6091 (117'3009,123'6091] local-lis/les=21/22 n=304 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'6091 lcod 123'6089 mlcod 123'6089 active+clean+snaptrim trimq=[20b~1d] ps=[1fe~d]] removing snap head 2019-12-05T06:51:19.576 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:19.570+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'6093 (117'3009,123'6093] local-lis/les=21/22 n=302 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'6091 crt=123'6091 lcod 123'6089 mlcod 123'6089 active+clean+snaptrim trimq=[20b~1d] ps=[1fe~d]] removing snap head 2019-12-05T06:51:19.576 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:19.569+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'6026 (116'3007,123'6026] local-lis/les=21/22 n=498 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=123'6026 lcod 123'6024 mlcod 123'6024 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:19.589 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:19.581+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'6030 (116'3007,123'6030] local-lis/les=21/22 n=495 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=123'6030 lcod 123'6028 mlcod 123'6028 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:19.590 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:19.578+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'6190 (116'3107,123'6190] local-lis/les=21/22 n=199 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'6190 lcod 123'6188 mlcod 123'6188 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:19.590 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:19.578+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'6192 (116'3107,123'6192] local-lis/les=21/22 n=197 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'6190 crt=123'6190 lcod 123'6188 mlcod 123'6188 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:19.590 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:19.582+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'6095 (117'3009,123'6095] local-lis/les=21/22 n=300 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'6095 lcod 123'6093 mlcod 123'6093 active+clean+snaptrim trimq=[20c~1c] ps=[1fe~e]] removing snap head 2019-12-05T06:51:19.590 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:19.581+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'6032 (116'3007,123'6032] local-lis/les=21/22 n=493 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'6030 crt=123'6030 lcod 123'6028 mlcod 123'6028 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:19.591 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:19.586+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'6097 (117'3009,123'6097] local-lis/les=21/22 n=298 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'6095 crt=123'6095 lcod 123'6093 mlcod 123'6093 active+clean+snaptrim trimq=[20c~1c] ps=[1fe~e]] removing snap head 2019-12-05T06:51:19.592 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:19.585+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'6094 (117'3008,123'6094] local-lis/les=21/22 n=175 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'6094 lcod 123'6092 mlcod 123'6092 active+clean+snaptrim trimq=[210~18] ps=[1fe~12]] removing snap head 2019-12-05T06:51:19.593 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:19.586+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6343 (117'3315,123'6343] local-lis/les=21/22 n=331 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'6343 lcod 123'6341 mlcod 123'6341 active+clean+snaptrim trimq=[20b~1d] ps=[1fe~d]] removing snap head 2019-12-05T06:51:19.597 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:19.590+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'6194 (116'3107,123'6194] local-lis/les=21/22 n=195 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'6194 lcod 123'6192 mlcod 123'6192 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:19.598 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:19.594+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'6196 (116'3107,123'6196] local-lis/les=21/22 n=193 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'6194 crt=123'6194 lcod 123'6192 mlcod 123'6192 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:19.599 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:19.589+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'6096 (117'3008,123'6096] local-lis/les=21/22 n=173 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'6094 crt=123'6094 lcod 123'6092 mlcod 123'6092 active+clean+snaptrim trimq=[210~18] ps=[1fe~12]] removing snap head 2019-12-05T06:51:19.599 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:19.594+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'6099 (117'3009,123'6099] local-lis/les=21/22 n=296 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'6099 lcod 123'6097 mlcod 123'6097 active+clean+snaptrim trimq=[20c~1c] ps=[1fe~e]] removing snap head 2019-12-05T06:51:19.600 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:19.593+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'6034 (116'3007,123'6034] local-lis/les=21/22 n=491 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=123'6034 lcod 123'6032 mlcod 123'6032 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:19.602 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:19.598+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'6101 (117'3009,123'6101] local-lis/les=21/22 n=294 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'6099 crt=123'6099 lcod 123'6097 mlcod 123'6097 active+clean+snaptrim trimq=[20c~1c] ps=[1fe~e]] removing snap head 2019-12-05T06:51:19.605 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:19.597+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'6036 (116'3007,123'6036] local-lis/les=21/22 n=489 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'6034 crt=123'6034 lcod 123'6032 mlcod 123'6032 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:19.607 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:19.602+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'6199 (116'3107,123'6199] local-lis/les=21/22 n=190 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'6198 crt=123'6194 lcod 123'6196 mlcod 123'6196 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:19.609 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:19.606+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'6201 (116'3107,123'6201] local-lis/les=21/22 n=188 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'6198 crt=123'6198 lcod 123'6196 mlcod 123'6196 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:19.612 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:19.610+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'6103 (117'3009,123'6103] local-lis/les=21/22 n=292 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'6103 lcod 123'6101 mlcod 123'6101 active+clean+snaptrim trimq=[20c~1c] ps=[1fe~e]] removing snap head 2019-12-05T06:51:19.613 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:19.605+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'6038 (116'3007,123'6038] local-lis/les=21/22 n=487 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=123'6038 lcod 123'6036 mlcod 123'6036 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:19.614 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:19.610+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'6105 (117'3009,123'6105] local-lis/les=21/22 n=290 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'6103 crt=123'6103 lcod 123'6101 mlcod 123'6101 active+clean+snaptrim trimq=[20c~1c] ps=[1fe~e]] removing snap head 2019-12-05T06:51:19.615 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:19.609+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'6040 (116'3007,123'6040] local-lis/les=21/22 n=485 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'6038 crt=123'6038 lcod 123'6036 mlcod 123'6036 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:19.617 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:19.614+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6346 (117'3315,123'6346] local-lis/les=21/22 n=328 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'6345 crt=123'6343 lcod 123'6343 mlcod 123'6343 active+clean+snaptrim trimq=[20c~1c] ps=[1fe~e]] removing snap head 2019-12-05T06:51:19.622 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:19.618+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6348 (117'3315,123'6348] local-lis/les=21/22 n=326 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'6345 crt=123'6345 lcod 123'6343 mlcod 123'6343 active+clean+snaptrim trimq=[20c~1c] ps=[1fe~e]] removing snap head 2019-12-05T06:51:19.623 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:19.618+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'6203 (116'3107,123'6203] local-lis/les=21/22 n=186 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'6203 lcod 123'6201 mlcod 123'6201 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:19.625 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:19.622+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'6205 (116'3107,123'6205] local-lis/les=21/22 n=184 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'6203 crt=123'6203 lcod 123'6201 mlcod 123'6201 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:19.626 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:19.617+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'6042 (116'3007,123'6042] local-lis/les=21/22 n=483 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=123'6042 lcod 123'6040 mlcod 123'6040 active+clean+snaptrim trimq=[205~23] ps=[1fe~7]] removing snap head 2019-12-05T06:51:19.627 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:19.622+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'6108 (117'3009,123'6108] local-lis/les=21/22 n=287 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'6107 crt=123'6105 lcod 123'6105 mlcod 123'6105 active+clean+snaptrim trimq=[20c~1c] ps=[1fe~e]] removing snap head 2019-12-05T06:51:19.632 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:19.626+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'6110 (117'3009,123'6110] local-lis/les=21/22 n=285 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'6107 crt=123'6107 lcod 123'6105 mlcod 123'6105 active+clean+snaptrim trimq=[20c~1c] ps=[1fe~e]] removing snap head 2019-12-05T06:51:19.632 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:19.625+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'6044 (116'3007,123'6044] local-lis/les=21/22 n=481 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=123'6044 lcod 123'6042 mlcod 123'6042 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:19.634 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:19.630+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'6207 (116'3107,123'6207] local-lis/les=21/22 n=182 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'6207 lcod 123'6205 mlcod 123'6205 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:19.635 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:19.629+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'6046 (116'3007,123'6046] local-lis/les=21/22 n=479 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'6044 crt=123'6044 lcod 123'6042 mlcod 123'6042 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:19.637 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:19.629+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'6099 (117'3008,123'6099] local-lis/les=21/22 n=170 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'6098 crt=123'6096 lcod 123'6096 mlcod 123'6096 active+clean+snaptrim trimq=[210~18] ps=[1fe~12]] removing snap head 2019-12-05T06:51:19.638 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:19.634+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'6209 (117'3207,123'6209] local-lis/les=21/22 n=180 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'6207 crt=123'6207 lcod 123'6205 mlcod 123'6205 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:19.642 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:19.633+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'6101 (117'3008,123'6101] local-lis/les=21/22 n=168 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'6098 crt=123'6098 lcod 123'6096 mlcod 123'6096 active+clean+snaptrim trimq=[210~18] ps=[1fe~12]] removing snap head 2019-12-05T06:51:19.642 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:19.637+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'6048 (116'3007,123'6048] local-lis/les=21/22 n=477 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=123'6048 lcod 123'6046 mlcod 123'6046 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:19.644 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:19.637+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'6050 (116'3007,123'6050] local-lis/les=21/22 n=475 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'6048 crt=123'6048 lcod 123'6046 mlcod 123'6046 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:19.652 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:19.646+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6350 (117'3315,123'6350] local-lis/les=21/22 n=324 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'6350 lcod 123'6348 mlcod 123'6348 active+clean+snaptrim trimq=[20c~1c] ps=[1fe~e]] removing snap head 2019-12-05T06:51:19.653 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:19.645+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'6052 (116'3007,123'6052] local-lis/les=21/22 n=473 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=123'6052 lcod 123'6050 mlcod 123'6050 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:19.663 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:19.649+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'6054 (116'3007,123'6054] local-lis/les=21/22 n=471 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'6052 crt=123'6052 lcod 123'6050 mlcod 123'6050 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:19.666 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:19.650+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6352 (117'3315,123'6352] local-lis/les=21/22 n=322 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'6350 crt=123'6350 lcod 123'6348 mlcod 123'6348 active+clean+snaptrim trimq=[20c~1c] ps=[1fe~e]] removing snap head 2019-12-05T06:51:19.667 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:19.657+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'6056 (116'3007,123'6056] local-lis/les=21/22 n=469 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=123'6056 lcod 123'6054 mlcod 123'6054 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:19.667 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:19.658+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6354 (117'3315,123'6354] local-lis/les=21/22 n=320 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'6354 lcod 123'6352 mlcod 123'6352 active+clean+snaptrim trimq=[20c~1c] ps=[1fe~e]] removing snap head 2019-12-05T06:51:19.668 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:19.662+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6356 (117'3315,123'6356] local-lis/les=21/22 n=318 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'6354 crt=123'6354 lcod 123'6352 mlcod 123'6352 active+clean+snaptrim trimq=[20c~1c] ps=[1fe~e]] removing snap head 2019-12-05T06:51:19.678 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:19.670+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6358 (117'3315,123'6358] local-lis/les=21/22 n=316 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'6358 lcod 123'6356 mlcod 123'6356 active+clean+snaptrim trimq=[20c~1c] ps=[1fe~e]] removing snap head 2019-12-05T06:51:19.678 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:19.674+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'6211 (117'3207,123'6211] local-lis/les=21/22 n=178 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'6211 lcod 123'6209 mlcod 123'6209 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:19.678 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:19.661+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'6058 (116'3007,123'6058] local-lis/les=21/22 n=467 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'6056 crt=123'6056 lcod 123'6054 mlcod 123'6054 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:19.687 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:19.674+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6360 (117'3315,123'6360] local-lis/les=21/22 n=314 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'6358 crt=123'6358 lcod 123'6356 mlcod 123'6356 active+clean+snaptrim trimq=[20c~1c] ps=[1fe~e]] removing snap head 2019-12-05T06:51:19.688 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:19.674+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'6113 (117'3110,123'6113] local-lis/les=21/22 n=282 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'6112 crt=123'6110 lcod 123'6110 mlcod 123'6110 active+clean+snaptrim trimq=[20c~1c] ps=[1fe~e]] removing snap head 2019-12-05T06:51:19.689 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:19.678+0000 7fc9c444d700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'6213 (117'3207,123'6213] local-lis/les=21/22 n=176 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'6211 crt=123'6211 lcod 123'6209 mlcod 123'6209 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:19.689 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:19.678+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'6115 (117'3110,123'6115] local-lis/les=21/22 n=280 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'6112 crt=123'6112 lcod 123'6110 mlcod 123'6110 active+clean+snaptrim trimq=[20c~1c] ps=[1fe~e]] removing snap head 2019-12-05T06:51:19.689 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:19.686+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'6216 (117'3207,123'6216] local-lis/les=21/22 n=173 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'6215 crt=123'6211 lcod 123'6213 mlcod 123'6213 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:19.689 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:19.682+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6364 (117'3315,123'6364] local-lis/les=21/22 n=311 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'6362 crt=123'6362 lcod 123'6360 mlcod 123'6360 active+clean+snaptrim trimq=[20c~1c] ps=[1fe~e]] removing snap head 2019-12-05T06:51:19.690 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:19.686+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'6218 (117'3207,123'6218] local-lis/les=21/22 n=171 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'6215 crt=123'6215 lcod 123'6213 mlcod 123'6213 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:19.691 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:19.686+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6366 (117'3315,123'6366] local-lis/les=21/22 n=309 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'6366 lcod 123'6364 mlcod 123'6364 active+clean+snaptrim trimq=[20c~1c] ps=[1fe~e]] removing snap head 2019-12-05T06:51:19.695 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:19.690+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6368 (117'3315,123'6368] local-lis/les=21/22 n=307 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'6366 crt=123'6366 lcod 123'6364 mlcod 123'6364 active+clean+snaptrim trimq=[20c~1c] ps=[1fe~e]] removing snap head 2019-12-05T06:51:19.697 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:19.689+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'6061 (116'3007,123'6061] local-lis/les=21/22 n=464 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'6060 crt=123'6058 lcod 123'6058 mlcod 123'6058 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:19.699 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:19.694+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6370 (117'3315,123'6370] local-lis/les=21/22 n=305 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'6370 lcod 123'6368 mlcod 123'6368 active+clean+snaptrim trimq=[20c~1c] ps=[1fe~e]] removing snap head 2019-12-05T06:51:19.699 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:19.693+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'6063 (116'3007,123'6063] local-lis/les=21/22 n=462 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'6060 crt=123'6060 lcod 123'6058 mlcod 123'6058 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:19.707 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:19.702+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6372 (117'3315,123'6372] local-lis/les=21/22 n=303 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'6372 lcod 123'6370 mlcod 123'6370 active+clean+snaptrim trimq=[20d~1b] ps=[1fe~f]] removing snap head 2019-12-05T06:51:19.710 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:19.706+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6374 (117'3315,123'6374] local-lis/les=21/22 n=301 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'6372 crt=123'6372 lcod 123'6370 mlcod 123'6370 active+clean+snaptrim trimq=[20d~1b] ps=[1fe~f]] removing snap head 2019-12-05T06:51:19.715 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:19.710+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6377 (117'3315,123'6377] local-lis/les=21/22 n=298 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'6376 crt=123'6372 lcod 123'6374 mlcod 123'6374 active+clean+snaptrim trimq=[20d~1b] ps=[1fe~f]] removing snap head 2019-12-05T06:51:19.716 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:19.714+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6379 (117'3315,123'6379] local-lis/les=21/22 n=296 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'6376 crt=123'6376 lcod 123'6374 mlcod 123'6374 active+clean+snaptrim trimq=[20d~1b] ps=[1fe~f]] removing snap head 2019-12-05T06:51:19.723 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:19.718+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6381 (117'3315,123'6381] local-lis/les=21/22 n=294 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'6381 lcod 123'6379 mlcod 123'6379 active+clean+snaptrim trimq=[20d~1b] ps=[1fe~f]] removing snap head 2019-12-05T06:51:19.725 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:19.722+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6383 (117'3315,123'6383] local-lis/les=21/22 n=292 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'6381 crt=123'6381 lcod 123'6379 mlcod 123'6379 active+clean+snaptrim trimq=[20d~1b] ps=[1fe~f]] removing snap head 2019-12-05T06:51:19.726 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:19.722+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'6118 (117'3110,123'6118] local-lis/les=21/22 n=277 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'6117 crt=123'6115 lcod 123'6115 mlcod 123'6115 active+clean+snaptrim trimq=[20c~1c] ps=[1fe~e]] removing snap head 2019-12-05T06:51:19.729 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:19.726+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'6120 (117'3110,123'6120] local-lis/les=21/22 n=275 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'6117 crt=123'6117 lcod 123'6115 mlcod 123'6115 active+clean+snaptrim trimq=[20c~1c] ps=[1fe~e]] removing snap head 2019-12-05T06:51:19.730 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:19.726+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6385 (117'3315,123'6385] local-lis/les=21/22 n=290 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'6385 lcod 123'6383 mlcod 123'6383 active+clean+snaptrim trimq=[20d~1b] ps=[1fe~f]] removing snap head 2019-12-05T06:51:19.732 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:19.730+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6387 (117'3315,123'6387] local-lis/les=21/22 n=288 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'6385 crt=123'6385 lcod 123'6383 mlcod 123'6383 active+clean+snaptrim trimq=[20d~1b] ps=[1fe~f]] removing snap head 2019-12-05T06:51:19.734 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:19.729+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'6066 (116'3007,123'6066] local-lis/les=21/22 n=459 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'6065 crt=123'6063 lcod 123'6063 mlcod 123'6063 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:19.737 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:19.729+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'6068 (116'3007,123'6068] local-lis/les=21/22 n=457 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'6065 crt=123'6065 lcod 123'6063 mlcod 123'6063 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:19.737 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:19.734+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6389 (117'3315,123'6389] local-lis/les=21/22 n=286 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'6389 lcod 123'6387 mlcod 123'6387 active+clean+snaptrim trimq=[20d~1b] ps=[1fe~f]] removing snap head 2019-12-05T06:51:19.739 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:19.734+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6391 (117'3315,123'6391] local-lis/les=21/22 n=284 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'6389 crt=123'6389 lcod 123'6387 mlcod 123'6387 active+clean+snaptrim trimq=[20d~1b] ps=[1fe~f]] removing snap head 2019-12-05T06:51:19.743 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:19.738+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'6221 (117'3207,123'6221] local-lis/les=21/22 n=168 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'6220 crt=123'6218 lcod 123'6218 mlcod 123'6218 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:19.745 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:19.742+0000 7fc9c0445700 -1 osd.3 pg_epoch: 123 pg[3.2s0( v 123'6223 (117'3207,123'6223] local-lis/les=21/22 n=166 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'6220 crt=123'6220 lcod 123'6218 mlcod 123'6218 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:19.746 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:19.742+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6393 (117'3315,123'6393] local-lis/les=21/22 n=282 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'6393 lcod 123'6391 mlcod 123'6391 active+clean+snaptrim trimq=[20d~1b] ps=[1fe~f]] removing snap head 2019-12-05T06:51:19.747 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:19.742+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6395 (117'3315,123'6395] local-lis/les=21/22 n=280 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'6393 crt=123'6393 lcod 123'6391 mlcod 123'6391 active+clean+snaptrim trimq=[20d~1b] ps=[1fe~f]] removing snap head 2019-12-05T06:51:19.754 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:19.750+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6397 (117'3315,123'6397] local-lis/les=21/22 n=278 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'6397 lcod 123'6395 mlcod 123'6395 active+clean+snaptrim trimq=[20d~1b] ps=[1fe~f]] removing snap head 2019-12-05T06:51:19.757 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:19.754+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6399 (117'3315,123'6399] local-lis/les=21/22 n=276 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'6397 crt=123'6397 lcod 123'6395 mlcod 123'6395 active+clean+snaptrim trimq=[20d~1b] ps=[1fe~f]] removing snap head 2019-12-05T06:51:19.762 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:19.753+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'6071 (116'3007,123'6071] local-lis/les=21/22 n=454 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'6070 crt=123'6068 lcod 123'6068 mlcod 123'6068 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:19.764 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:19.757+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'6073 (116'3007,123'6073] local-lis/les=21/22 n=452 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'6070 crt=123'6070 lcod 123'6068 mlcod 123'6068 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:19.765 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:19.762+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6401 (117'3315,123'6401] local-lis/les=21/22 n=274 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'6401 lcod 123'6399 mlcod 123'6399 active+clean+snaptrim trimq=[20d~1b] ps=[1fe~f]] removing snap head 2019-12-05T06:51:19.767 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:19.762+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6403 (117'3315,123'6403] local-lis/les=21/22 n=272 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'6401 crt=123'6401 lcod 123'6399 mlcod 123'6399 active+clean+snaptrim trimq=[20d~1b] ps=[1fe~f]] removing snap head 2019-12-05T06:51:19.772 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:19.766+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'6123 (117'3110,123'6123] local-lis/les=21/22 n=272 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'6122 crt=123'6120 lcod 123'6120 mlcod 123'6120 active+clean+snaptrim trimq=[20c~1c] ps=[1fe~e]] removing snap head 2019-12-05T06:51:19.772 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:19.770+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'6125 (117'3110,123'6125] local-lis/les=21/22 n=270 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'6122 crt=123'6122 lcod 123'6120 mlcod 123'6120 active+clean+snaptrim trimq=[20c~1c] ps=[1fe~e]] removing snap head 2019-12-05T06:51:19.773 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:19.770+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6405 (117'3315,123'6405] local-lis/les=21/22 n=270 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'6405 lcod 123'6403 mlcod 123'6403 active+clean+snaptrim trimq=[20d~1b] ps=[1fe~f]] removing snap head 2019-12-05T06:51:19.775 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:19.770+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6407 (117'3315,123'6407] local-lis/les=21/22 n=268 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'6405 crt=123'6405 lcod 123'6403 mlcod 123'6403 active+clean+snaptrim trimq=[20d~1b] ps=[1fe~f]] removing snap head 2019-12-05T06:51:19.783 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:19.777+0000 7f4272b1c700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'6076 (116'3007,123'6076] local-lis/les=21/22 n=449 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'6075 crt=123'6073 lcod 123'6073 mlcod 123'6073 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:19.784 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:19.782+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6411 (117'3315,123'6411] local-lis/les=21/22 n=265 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'6409 crt=123'6409 lcod 123'6407 mlcod 123'6407 active+clean+snaptrim trimq=[20d~1b] ps=[1fe~f]] removing snap head 2019-12-05T06:51:20.118 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:20.109+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'6081 (116'3007,123'6081] local-lis/les=21/22 n=445 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'6080 crt=123'6076 lcod 123'6078 mlcod 123'6078 active+clean+snaptrim trimq=[206~22] ps=[1fe~8]] removing snap head 2019-12-05T06:51:20.131 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:20.125+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'6084 (116'3007,123'6084] local-lis/les=21/22 n=442 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'6083 crt=123'6081 lcod 123'6081 mlcod 123'6081 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:20.133 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:20.125+0000 7f426eb14700 -1 osd.4 pg_epoch: 123 pg[3.3s0( v 123'6086 (116'3007,123'6086] local-lis/les=21/22 n=440 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'6083 crt=123'6083 lcod 123'6081 mlcod 123'6081 active+clean+snaptrim trimq=[207~21] ps=[1fe~9]] removing snap head 2019-12-05T06:51:20.240 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:20.234+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'6127 (117'3110,123'6127] local-lis/les=21/22 n=268 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'6127 lcod 123'6125 mlcod 123'6125 active+clean+snaptrim trimq=[20c~1c] ps=[1fe~e]] removing snap head 2019-12-05T06:51:20.243 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:20.238+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6413 (117'3315,123'6413] local-lis/les=21/22 n=263 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'6413 lcod 123'6411 mlcod 123'6411 active+clean+snaptrim trimq=[20d~1b] ps=[1fe~f]] removing snap head 2019-12-05T06:51:20.243 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:20.238+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'6129 (117'3110,123'6129] local-lis/les=21/22 n=266 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'6127 crt=123'6127 lcod 123'6125 mlcod 123'6125 active+clean+snaptrim trimq=[20c~1c] ps=[1fe~e]] removing snap head 2019-12-05T06:51:20.249 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:20.246+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6415 (117'3315,123'6415] local-lis/les=21/22 n=261 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'6415 lcod 123'6413 mlcod 123'6413 active+clean+snaptrim trimq=[20e~1a] ps=[1fe~10]] removing snap head 2019-12-05T06:51:20.252 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:20.250+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6417 (117'3415,123'6417] local-lis/les=21/22 n=259 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'6415 crt=123'6415 lcod 123'6413 mlcod 123'6413 active+clean+snaptrim trimq=[20e~1a] ps=[1fe~10]] removing snap head 2019-12-05T06:51:20.254 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:20.250+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'6131 (117'3110,123'6131] local-lis/les=21/22 n=264 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'6131 lcod 123'6129 mlcod 123'6129 active+clean+snaptrim trimq=[20c~1c] ps=[1fe~e]] removing snap head 2019-12-05T06:51:20.261 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:20.258+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6419 (117'3415,123'6419] local-lis/les=21/22 n=257 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'6419 lcod 123'6417 mlcod 123'6417 active+clean+snaptrim trimq=[20e~1a] ps=[1fe~10]] removing snap head 2019-12-05T06:51:20.262 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:20.258+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'6133 (117'3110,123'6133] local-lis/les=21/22 n=262 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'6133 lcod 123'6131 mlcod 123'6131 active+clean+snaptrim trimq=[20d~1b] ps=[1fe~f]] removing snap head 2019-12-05T06:51:20.263 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:20.258+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6421 (117'3415,123'6421] local-lis/les=21/22 n=255 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'6419 crt=123'6419 lcod 123'6417 mlcod 123'6417 active+clean+snaptrim trimq=[20e~1a] ps=[1fe~10]] removing snap head 2019-12-05T06:51:20.264 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:20.262+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'6135 (117'3110,123'6135] local-lis/les=21/22 n=260 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'6133 crt=123'6133 lcod 123'6131 mlcod 123'6131 active+clean+snaptrim trimq=[20d~1b] ps=[1fe~f]] removing snap head 2019-12-05T06:51:20.271 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:20.266+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'6137 (117'3110,123'6137] local-lis/les=21/22 n=258 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'6137 lcod 123'6135 mlcod 123'6135 active+clean+snaptrim trimq=[20d~1b] ps=[1fe~f]] removing snap head 2019-12-05T06:51:20.272 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:20.270+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6423 (117'3415,123'6423] local-lis/les=21/22 n=253 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'6423 lcod 123'6421 mlcod 123'6421 active+clean+snaptrim trimq=[20e~1a] ps=[1fe~10]] removing snap head 2019-12-05T06:51:20.273 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:20.270+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'6139 (117'3110,123'6139] local-lis/les=21/22 n=256 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'6137 crt=123'6137 lcod 123'6135 mlcod 123'6135 active+clean+snaptrim trimq=[20d~1b] ps=[1fe~f]] removing snap head 2019-12-05T06:51:20.277 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:20.270+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6425 (117'3415,123'6425] local-lis/les=21/22 n=251 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'6423 crt=123'6423 lcod 123'6421 mlcod 123'6421 active+clean+snaptrim trimq=[20e~1a] ps=[1fe~10]] removing snap head 2019-12-05T06:51:20.279 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:20.274+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'6141 (117'3110,123'6141] local-lis/les=21/22 n=254 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'6141 lcod 123'6139 mlcod 123'6139 active+clean+snaptrim trimq=[20d~1b] ps=[1fe~f]] removing snap head 2019-12-05T06:51:20.280 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:20.274+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6427 (117'3415,123'6427] local-lis/les=21/22 n=249 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'6427 lcod 123'6425 mlcod 123'6425 active+clean+snaptrim trimq=[20e~1a] ps=[1fe~10]] removing snap head 2019-12-05T06:51:20.282 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:20.278+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6429 (117'3415,123'6429] local-lis/les=21/22 n=247 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'6427 crt=123'6427 lcod 123'6425 mlcod 123'6425 active+clean+snaptrim trimq=[20e~1a] ps=[1fe~10]] removing snap head 2019-12-05T06:51:20.282 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:20.278+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'6143 (117'3110,123'6143] local-lis/les=21/22 n=252 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'6141 crt=123'6141 lcod 123'6139 mlcod 123'6139 active+clean+snaptrim trimq=[20d~1b] ps=[1fe~f]] removing snap head 2019-12-05T06:51:20.306 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:20.286+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'6145 (117'3110,123'6145] local-lis/les=21/22 n=250 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'6145 lcod 123'6143 mlcod 123'6143 active+clean+snaptrim trimq=[20d~1b] ps=[1fe~f]] removing snap head 2019-12-05T06:51:20.306 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:20.290+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'6147 (117'3110,123'6147] local-lis/les=21/22 n=248 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'6145 crt=123'6145 lcod 123'6143 mlcod 123'6143 active+clean+snaptrim trimq=[20d~1b] ps=[1fe~f]] removing snap head 2019-12-05T06:51:20.306 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:20.290+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6431 (117'3415,123'6431] local-lis/les=21/22 n=245 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'6431 lcod 123'6429 mlcod 123'6429 active+clean+snaptrim trimq=[20e~1a] ps=[1fe~10]] removing snap head 2019-12-05T06:51:20.307 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:20.290+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6433 (117'3415,123'6433] local-lis/les=21/22 n=243 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'6431 crt=123'6431 lcod 123'6429 mlcod 123'6429 active+clean+snaptrim trimq=[20e~1a] ps=[1fe~10]] removing snap head 2019-12-05T06:51:20.307 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:20.298+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6435 (117'3415,123'6435] local-lis/les=21/22 n=241 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'6435 lcod 123'6433 mlcod 123'6433 active+clean+snaptrim trimq=[20e~1a] ps=[1fe~10]] removing snap head 2019-12-05T06:51:20.308 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:20.298+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'6149 (117'3110,123'6149] local-lis/les=21/22 n=246 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'6149 lcod 123'6147 mlcod 123'6147 active+clean+snaptrim trimq=[20d~1b] ps=[1fe~f]] removing snap head 2019-12-05T06:51:20.308 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:20.298+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'6151 (117'3110,123'6151] local-lis/les=21/22 n=244 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'6149 crt=123'6149 lcod 123'6147 mlcod 123'6147 active+clean+snaptrim trimq=[20d~1b] ps=[1fe~f]] removing snap head 2019-12-05T06:51:20.320 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:20.306+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'6153 (117'3110,123'6153] local-lis/les=21/22 n=242 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'6153 lcod 123'6151 mlcod 123'6151 active+clean+snaptrim trimq=[20d~1b] ps=[1fe~f]] removing snap head 2019-12-05T06:51:20.320 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:20.310+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'6155 (117'3110,123'6155] local-lis/les=21/22 n=240 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'6153 crt=123'6153 lcod 123'6151 mlcod 123'6151 active+clean+snaptrim trimq=[20d~1b] ps=[1fe~f]] removing snap head 2019-12-05T06:51:20.320 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:20.306+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6439 (117'3415,123'6439] local-lis/les=21/22 n=238 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'6439 lcod 123'6437 mlcod 123'6437 active+clean+snaptrim trimq=[20e~1a] ps=[1fe~10]] removing snap head 2019-12-05T06:51:20.320 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:20.310+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6441 (117'3415,123'6441] local-lis/les=21/22 n=236 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'6439 crt=123'6439 lcod 123'6437 mlcod 123'6437 active+clean+snaptrim trimq=[20e~1a] ps=[1fe~10]] removing snap head 2019-12-05T06:51:20.321 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:20.318+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6443 (117'3415,123'6443] local-lis/les=21/22 n=234 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'6443 lcod 123'6441 mlcod 123'6441 active+clean+snaptrim trimq=[20e~1a] ps=[1fe~10]] removing snap head 2019-12-05T06:51:20.332 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:20.318+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'6157 (117'3110,123'6157] local-lis/les=21/22 n=238 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'6157 lcod 123'6155 mlcod 123'6155 active+clean+snaptrim trimq=[20d~1b] ps=[1fe~f]] removing snap head 2019-12-05T06:51:20.333 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:20.318+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'6159 (117'3110,123'6159] local-lis/les=21/22 n=236 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'6157 crt=123'6157 lcod 123'6155 mlcod 123'6155 active+clean+snaptrim trimq=[20d~1b] ps=[1fe~f]] removing snap head 2019-12-05T06:51:20.333 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:20.326+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'6161 (117'3110,123'6161] local-lis/les=21/22 n=234 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'6161 lcod 123'6159 mlcod 123'6159 active+clean+snaptrim trimq=[20d~1b] ps=[1fe~f]] removing snap head 2019-12-05T06:51:20.333 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:20.330+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'6163 (117'3110,123'6163] local-lis/les=21/22 n=232 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'6161 crt=123'6161 lcod 123'6159 mlcod 123'6159 active+clean+snaptrim trimq=[20d~1b] ps=[1fe~f]] removing snap head 2019-12-05T06:51:20.334 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:20.318+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6445 (117'3415,123'6445] local-lis/les=21/22 n=232 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'6443 crt=123'6443 lcod 123'6441 mlcod 123'6441 active+clean+snaptrim trimq=[20e~1a] ps=[1fe~10]] removing snap head 2019-12-05T06:51:20.334 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:20.326+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6447 (117'3415,123'6447] local-lis/les=21/22 n=230 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'6447 lcod 123'6445 mlcod 123'6445 active+clean+snaptrim trimq=[20e~1a] ps=[1fe~10]] removing snap head 2019-12-05T06:51:20.345 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:20.338+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'6165 (117'3110,123'6165] local-lis/les=21/22 n=230 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'6165 lcod 123'6163 mlcod 123'6163 active+clean+snaptrim trimq=[20d~1b] ps=[1fe~f]] removing snap head 2019-12-05T06:51:20.346 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:20.338+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'6167 (117'3110,123'6167] local-lis/les=21/22 n=228 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'6165 crt=123'6165 lcod 123'6163 mlcod 123'6163 active+clean+snaptrim trimq=[20d~1b] ps=[1fe~f]] removing snap head 2019-12-05T06:51:20.346 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:20.338+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6449 (117'3415,123'6449] local-lis/les=21/22 n=228 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'6449 lcod 123'6447 mlcod 123'6447 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:20.346 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:20.338+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6451 (117'3415,123'6451] local-lis/les=21/22 n=226 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'6449 crt=123'6449 lcod 123'6447 mlcod 123'6447 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:20.353 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:20.346+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6453 (117'3415,123'6453] local-lis/les=21/22 n=224 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'6453 lcod 123'6451 mlcod 123'6451 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:20.353 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:20.350+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6455 (117'3415,123'6455] local-lis/les=21/22 n=222 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'6453 crt=123'6453 lcod 123'6451 mlcod 123'6451 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:20.353 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:20.346+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'6169 (117'3110,123'6169] local-lis/les=21/22 n=226 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'6169 lcod 123'6167 mlcod 123'6167 active+clean+snaptrim trimq=[20d~1b] ps=[1fe~f]] removing snap head 2019-12-05T06:51:20.361 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:20.358+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'6171 (117'3110,123'6171] local-lis/les=21/22 n=224 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'6171 lcod 123'6169 mlcod 123'6169 active+clean+snaptrim trimq=[20e~1a] ps=[1fe~10]] removing snap head 2019-12-05T06:51:20.363 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:20.358+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6457 (117'3415,123'6457] local-lis/les=21/22 n=220 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'6457 lcod 123'6455 mlcod 123'6455 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:20.364 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:20.358+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'6173 (117'3110,123'6173] local-lis/les=21/22 n=222 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'6171 crt=123'6171 lcod 123'6169 mlcod 123'6169 active+clean+snaptrim trimq=[20e~1a] ps=[1fe~10]] removing snap head 2019-12-05T06:51:20.366 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:20.362+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6459 (117'3415,123'6459] local-lis/les=21/22 n=218 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'6457 crt=123'6457 lcod 123'6455 mlcod 123'6455 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:20.372 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:20.370+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'6175 (117'3110,123'6175] local-lis/les=21/22 n=220 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'6175 lcod 123'6173 mlcod 123'6173 active+clean+snaptrim trimq=[20e~1a] ps=[1fe~10]] removing snap head 2019-12-05T06:51:20.374 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:20.370+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 123 pg[3.7s0( v 123'6177 (117'3110,123'6177] local-lis/les=21/22 n=218 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'6175 crt=123'6175 lcod 123'6173 mlcod 123'6173 active+clean+snaptrim trimq=[20e~1a] ps=[1fe~10]] removing snap head 2019-12-05T06:51:20.375 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:20.370+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6461 (117'3415,123'6461] local-lis/les=21/22 n=216 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'6461 lcod 123'6459 mlcod 123'6459 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:20.377 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:20.374+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 123 pg[3.1s0( v 123'6463 (117'3415,123'6463] local-lis/les=21/22 n=214 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'6461 crt=123'6461 lcod 123'6459 mlcod 123'6459 active+clean+snaptrim trimq=[20f~19] ps=[1fe~11]] removing snap head 2019-12-05T06:51:20.402 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:20.393+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'6103 (117'3008,123'6103] local-lis/les=21/22 n=166 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'6103 lcod 123'6101 mlcod 123'6101 active+clean+snaptrim trimq=[210~18] ps=[1fe~12]] removing snap head 2019-12-05T06:51:20.404 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:20.397+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'6105 (117'3008,123'6105] local-lis/les=21/22 n=164 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'6103 crt=123'6103 lcod 123'6101 mlcod 123'6101 active+clean+snaptrim trimq=[210~18] ps=[1fe~12]] removing snap head 2019-12-05T06:51:20.409 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:20.401+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'6107 (117'3008,123'6107] local-lis/les=21/22 n=162 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'6107 lcod 123'6105 mlcod 123'6105 active+clean+snaptrim trimq=[210~18] ps=[1fe~12]] removing snap head 2019-12-05T06:51:20.411 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:20.405+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'6109 (117'3008,123'6109] local-lis/les=21/22 n=160 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'6107 crt=123'6107 lcod 123'6105 mlcod 123'6105 active+clean+snaptrim trimq=[210~18] ps=[1fe~12]] removing snap head 2019-12-05T06:51:20.418 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:20.409+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'6111 (117'3109,123'6111] local-lis/les=21/22 n=158 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'6111 lcod 123'6109 mlcod 123'6109 active+clean+snaptrim trimq=[210~18] ps=[1fe~12]] removing snap head 2019-12-05T06:51:20.419 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:20.413+0000 7fc712838700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'6113 (117'3109,123'6113] local-lis/les=21/22 n=156 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'6111 crt=123'6111 lcod 123'6109 mlcod 123'6109 active+clean+snaptrim trimq=[210~18] ps=[1fe~12]] removing snap head 2019-12-05T06:51:20.423 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:20.417+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'6115 (117'3109,123'6115] local-lis/les=21/22 n=154 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'6115 lcod 123'6113 mlcod 123'6113 active+clean+snaptrim trimq=[210~18] ps=[1fe~12]] removing snap head 2019-12-05T06:51:20.424 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:20.417+0000 7fc70e830700 -1 osd.5 pg_epoch: 123 pg[3.5s0( v 123'6117 (117'3109,123'6117] local-lis/les=21/22 n=152 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'6115 crt=123'6115 lcod 123'6113 mlcod 123'6113 active+clean+snaptrim trimq=[210~18] ps=[1fe~12]] removing snap head 2019-12-05T06:51:20.439 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:20.433+0000 7fc712838700 -1 osd.5 pg_epoch: 124 pg[3.5s0( v 123'6119 (117'3109,123'6119] local-lis/les=21/22 n=150 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=123'6119 lcod 123'6117 mlcod 123'6117 active+clean+snaptrim trimq=[210~18] ps=[204~c]] removing snap head 2019-12-05T06:51:20.440 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:20.433+0000 7fc712838700 -1 osd.5 pg_epoch: 124 pg[3.5s0( v 124'6121 (117'3109,124'6121] local-lis/les=21/22 n=148 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=123'6119 crt=123'6119 lcod 123'6117 mlcod 123'6117 active+clean+snaptrim trimq=[210~18] ps=[204~c]] removing snap head 2019-12-05T06:51:20.477 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:20.470+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 124 pg[3.7s0( v 123'6179 (117'3110,123'6179] local-lis/les=21/22 n=216 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=123'6179 lcod 123'6177 mlcod 123'6177 active+clean+snaptrim trimq=[20e~1a] ps=[204~a]] removing snap head 2019-12-05T06:51:20.478 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:20.470+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 124 pg[3.1s0( v 123'6465 (117'3415,123'6465] local-lis/les=21/22 n=212 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=123'6465 lcod 123'6463 mlcod 123'6463 active+clean+snaptrim trimq=[20f~19] ps=[204~b]] removing snap head 2019-12-05T06:51:20.478 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:20.474+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 124 pg[3.1s0( v 124'6467 (117'3415,124'6467] local-lis/les=21/22 n=210 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=123'6465 crt=123'6465 lcod 123'6463 mlcod 123'6463 active+clean+snaptrim trimq=[20f~19] ps=[204~b]] removing snap head 2019-12-05T06:51:20.484 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:20.474+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 124 pg[3.7s0( v 124'6181 (117'3110,124'6181] local-lis/les=21/22 n=214 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=123'6179 crt=123'6179 lcod 123'6177 mlcod 123'6177 active+clean+snaptrim trimq=[20e~1a] ps=[204~a]] removing snap head 2019-12-05T06:51:20.485 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:20.477+0000 7f4272b1c700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6089 (116'3007,124'6089] local-lis/les=21/22 n=437 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'6088 crt=123'6086 lcod 123'6086 mlcod 123'6086 active+clean+snaptrim trimq=[207~21] ps=[204~3]] removing snap head 2019-12-05T06:51:20.488 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:20.481+0000 7f4272b1c700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6091 (116'3007,124'6091] local-lis/les=21/22 n=435 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=123'6088 crt=123'6088 lcod 123'6086 mlcod 123'6086 active+clean+snaptrim trimq=[207~21] ps=[204~3]] removing snap head 2019-12-05T06:51:20.490 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:20.485+0000 7fc70e830700 -1 osd.5 pg_epoch: 124 pg[3.5s0( v 124'6123 (117'3109,124'6123] local-lis/les=21/22 n=146 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=124'6123 lcod 124'6121 mlcod 124'6121 active+clean+snaptrim trimq=[210~18] ps=[204~c]] removing snap head 2019-12-05T06:51:20.494 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:20.485+0000 7fc70e830700 -1 osd.5 pg_epoch: 124 pg[3.5s0( v 124'6125 (117'3109,124'6125] local-lis/les=21/22 n=144 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=124'6123 crt=124'6123 lcod 124'6121 mlcod 124'6121 active+clean+snaptrim trimq=[210~18] ps=[204~c]] removing snap head 2019-12-05T06:51:20.497 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:20.489+0000 7f426eb14700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6094 (116'3007,124'6094] local-lis/les=21/22 n=432 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=124'6093 crt=124'6091 lcod 124'6091 mlcod 124'6091 active+clean+snaptrim trimq=[207~21] ps=[204~3]] removing snap head 2019-12-05T06:51:20.499 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:20.493+0000 7fc70e830700 -1 osd.5 pg_epoch: 124 pg[3.5s0( v 124'6127 (117'3109,124'6127] local-lis/les=21/22 n=142 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=124'6127 lcod 124'6125 mlcod 124'6125 active+clean+snaptrim trimq=[210~18] ps=[204~c]] removing snap head 2019-12-05T06:51:20.499 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:20.493+0000 7f426eb14700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6096 (116'3007,124'6096] local-lis/les=21/22 n=430 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=124'6093 crt=124'6093 lcod 124'6091 mlcod 124'6091 active+clean+snaptrim trimq=[207~21] ps=[204~3]] removing snap head 2019-12-05T06:51:20.501 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:20.493+0000 7fc70e830700 -1 osd.5 pg_epoch: 124 pg[3.5s0( v 124'6129 (117'3109,124'6129] local-lis/les=21/22 n=140 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=124'6127 crt=124'6127 lcod 124'6125 mlcod 124'6125 active+clean+snaptrim trimq=[210~18] ps=[204~c]] removing snap head 2019-12-05T06:51:20.506 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:20.497+0000 7fc70e830700 -1 osd.5 pg_epoch: 124 pg[3.5s0( v 124'6131 (117'3109,124'6131] local-lis/les=21/22 n=138 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=124'6131 lcod 124'6129 mlcod 124'6129 active+clean+snaptrim trimq=[210~18] ps=[204~c]] removing snap head 2019-12-05T06:51:20.509 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:20.501+0000 7fc70e830700 -1 osd.5 pg_epoch: 124 pg[3.5s0( v 124'6133 (117'3109,124'6133] local-lis/les=21/22 n=136 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=124'6131 crt=124'6131 lcod 124'6129 mlcod 124'6129 active+clean+snaptrim trimq=[210~18] ps=[204~c]] removing snap head 2019-12-05T06:51:20.509 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:20.506+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 124 pg[3.7s0( v 124'6183 (117'3110,124'6183] local-lis/les=21/22 n=212 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=124'6183 lcod 124'6181 mlcod 124'6181 active+clean+snaptrim trimq=[20e~1a] ps=[204~a]] removing snap head 2019-12-05T06:51:20.510 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:20.506+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 124 pg[3.1s0( v 124'6470 (117'3415,124'6470] local-lis/les=21/22 n=207 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=124'6469 crt=123'6465 lcod 124'6467 mlcod 124'6467 active+clean+snaptrim trimq=[20f~19] ps=[204~b]] removing snap head 2019-12-05T06:51:20.510 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:20.506+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 124 pg[3.7s0( v 124'6185 (117'3110,124'6185] local-lis/les=21/22 n=210 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=124'6183 crt=124'6183 lcod 124'6181 mlcod 124'6181 active+clean+snaptrim trimq=[20e~1a] ps=[204~a]] removing snap head 2019-12-05T06:51:20.513 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:20.505+0000 7fc70e830700 -1 osd.5 pg_epoch: 124 pg[3.5s0( v 124'6135 (117'3109,124'6135] local-lis/les=21/22 n=134 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=124'6135 lcod 124'6133 mlcod 124'6133 active+clean+snaptrim trimq=[210~18] ps=[204~c]] removing snap head 2019-12-05T06:51:20.515 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:20.510+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 124 pg[3.1s0( v 124'6472 (117'3415,124'6472] local-lis/les=21/22 n=205 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=124'6469 crt=124'6469 lcod 124'6467 mlcod 124'6467 active+clean+snaptrim trimq=[20f~19] ps=[204~b]] removing snap head 2019-12-05T06:51:20.515 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:20.509+0000 7fc70e830700 -1 osd.5 pg_epoch: 124 pg[3.5s0( v 124'6137 (117'3109,124'6137] local-lis/les=21/22 n=132 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=124'6135 crt=124'6135 lcod 124'6133 mlcod 124'6133 active+clean+snaptrim trimq=[210~18] ps=[204~c]] removing snap head 2019-12-05T06:51:20.518 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:20.513+0000 7fc712838700 -1 osd.5 pg_epoch: 124 pg[3.5s0( v 124'6139 (117'3109,124'6139] local-lis/les=21/22 n=130 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=124'6139 lcod 124'6137 mlcod 124'6137 active+clean+snaptrim trimq=[210~18] ps=[204~c]] removing snap head 2019-12-05T06:51:20.520 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:20.513+0000 7fc712838700 -1 osd.5 pg_epoch: 124 pg[3.5s0( v 124'6141 (117'3109,124'6141] local-lis/les=21/22 n=128 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=124'6139 crt=124'6139 lcod 124'6137 mlcod 124'6137 active+clean+snaptrim trimq=[210~18] ps=[204~c]] removing snap head 2019-12-05T06:51:20.522 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:20.518+0000 7fc9c444d700 -1 osd.3 pg_epoch: 124 pg[3.2s0( v 123'6225 (117'3207,123'6225] local-lis/les=21/22 n=164 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=123'6225 lcod 123'6223 mlcod 123'6223 active+clean+snaptrim trimq=[20f~19] ps=[204~b]] removing snap head 2019-12-05T06:51:20.524 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:20.522+0000 7fc9c444d700 -1 osd.3 pg_epoch: 124 pg[3.2s0( v 124'6227 (117'3207,124'6227] local-lis/les=21/22 n=162 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=123'6225 crt=123'6225 lcod 123'6223 mlcod 123'6223 active+clean+snaptrim trimq=[20f~19] ps=[204~b]] removing snap head 2019-12-05T06:51:20.527 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:20.521+0000 7fc70e830700 -1 osd.5 pg_epoch: 124 pg[3.5s0( v 124'6143 (117'3109,124'6143] local-lis/les=21/22 n=126 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=124'6143 lcod 124'6141 mlcod 124'6141 active+clean+snaptrim trimq=[210~18] ps=[204~c]] removing snap head 2019-12-05T06:51:20.529 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:20.526+0000 7fc9c0445700 -1 osd.3 pg_epoch: 124 pg[3.2s0( v 124'6229 (117'3207,124'6229] local-lis/les=21/22 n=160 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=124'6229 lcod 124'6227 mlcod 124'6227 active+clean+snaptrim trimq=[20f~19] ps=[204~b]] removing snap head 2019-12-05T06:51:20.530 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:20.521+0000 7fc70e830700 -1 osd.5 pg_epoch: 124 pg[3.5s0( v 124'6145 (117'3109,124'6145] local-lis/les=21/22 n=124 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=124'6143 crt=124'6143 lcod 124'6141 mlcod 124'6141 active+clean+snaptrim trimq=[210~18] ps=[204~c]] removing snap head 2019-12-05T06:51:20.531 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:20.526+0000 7fc9c0445700 -1 osd.3 pg_epoch: 124 pg[3.2s0( v 124'6231 (117'3207,124'6231] local-lis/les=21/22 n=158 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=124'6229 crt=124'6229 lcod 124'6227 mlcod 124'6227 active+clean+snaptrim trimq=[20f~19] ps=[204~b]] removing snap head 2019-12-05T06:51:20.536 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:20.530+0000 7fc9c444d700 -1 osd.3 pg_epoch: 124 pg[3.2s0( v 124'6233 (117'3207,124'6233] local-lis/les=21/22 n=156 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=124'6233 lcod 124'6231 mlcod 124'6231 active+clean+snaptrim trimq=[20f~19] ps=[204~b]] removing snap head 2019-12-05T06:51:20.536 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:20.529+0000 7fc712838700 -1 osd.5 pg_epoch: 124 pg[3.5s0( v 124'6147 (117'3109,124'6147] local-lis/les=21/22 n=122 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=124'6147 lcod 124'6145 mlcod 124'6145 active+clean+snaptrim trimq=[210~18] ps=[204~c]] removing snap head 2019-12-05T06:51:20.537 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:20.534+0000 7fc9c444d700 -1 osd.3 pg_epoch: 124 pg[3.2s0( v 124'6235 (117'3207,124'6235] local-lis/les=21/22 n=154 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=124'6233 crt=124'6233 lcod 124'6231 mlcod 124'6231 active+clean+snaptrim trimq=[20f~19] ps=[204~b]] removing snap head 2019-12-05T06:51:20.537 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:20.529+0000 7fc712838700 -1 osd.5 pg_epoch: 124 pg[3.5s0( v 124'6149 (117'3109,124'6149] local-lis/les=21/22 n=120 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=124'6147 crt=124'6147 lcod 124'6145 mlcod 124'6145 active+clean+snaptrim trimq=[210~18] ps=[204~c]] removing snap head 2019-12-05T06:51:20.543 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:20.538+0000 7fc9c444d700 -1 osd.3 pg_epoch: 124 pg[3.2s0( v 124'6237 (117'3207,124'6237] local-lis/les=21/22 n=152 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=124'6237 lcod 124'6235 mlcod 124'6235 active+clean+snaptrim trimq=[20f~19] ps=[204~b]] removing snap head 2019-12-05T06:51:20.543 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:20.537+0000 7fc70e830700 -1 osd.5 pg_epoch: 124 pg[3.5s0( v 124'6151 (117'3109,124'6151] local-lis/les=21/22 n=118 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=124'6151 lcod 124'6149 mlcod 124'6149 active+clean+snaptrim trimq=[210~18] ps=[204~c]] removing snap head 2019-12-05T06:51:20.545 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:20.542+0000 7fc9c444d700 -1 osd.3 pg_epoch: 124 pg[3.2s0( v 124'6239 (117'3207,124'6239] local-lis/les=21/22 n=150 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=124'6237 crt=124'6237 lcod 124'6235 mlcod 124'6235 active+clean+snaptrim trimq=[20f~19] ps=[204~b]] removing snap head 2019-12-05T06:51:20.546 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:20.537+0000 7fc70e830700 -1 osd.5 pg_epoch: 124 pg[3.5s0( v 124'6153 (117'3109,124'6153] local-lis/les=21/22 n=116 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=124'6151 crt=124'6151 lcod 124'6149 mlcod 124'6149 active+clean+snaptrim trimq=[210~18] ps=[204~c]] removing snap head 2019-12-05T06:51:20.550 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:20.541+0000 7fc712838700 -1 osd.5 pg_epoch: 124 pg[3.5s0( v 124'6155 (117'3109,124'6155] local-lis/les=21/22 n=114 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=124'6155 lcod 124'6153 mlcod 124'6153 active+clean+snaptrim trimq=[210~18] ps=[204~c]] removing snap head 2019-12-05T06:51:20.551 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:20.545+0000 7fc712838700 -1 osd.5 pg_epoch: 124 pg[3.5s0( v 124'6157 (117'3109,124'6157] local-lis/les=21/22 n=112 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=124'6155 crt=124'6155 lcod 124'6153 mlcod 124'6153 active+clean+snaptrim trimq=[210~18] ps=[204~c]] removing snap head 2019-12-05T06:51:20.557 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:20.549+0000 7fc712838700 -1 osd.5 pg_epoch: 124 pg[3.5s0( v 124'6159 (117'3109,124'6159] local-lis/les=21/22 n=110 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=124'6159 lcod 124'6157 mlcod 124'6157 active+clean+snaptrim trimq=[210~18] ps=[204~c]] removing snap head 2019-12-05T06:51:20.558 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:20.553+0000 7fc712838700 -1 osd.5 pg_epoch: 124 pg[3.5s0( v 124'6161 (117'3109,124'6161] local-lis/les=21/22 n=108 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=124'6161 lcod 124'6159 mlcod 124'6159 active+clean+snaptrim trimq=[211~17] ps=[204~d]] removing snap head 2019-12-05T06:51:20.560 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:20.553+0000 7fc712838700 -1 osd.5 pg_epoch: 124 pg[3.5s0( v 124'6163 (117'3109,124'6163] local-lis/les=21/22 n=106 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=124'6161 crt=124'6161 lcod 124'6159 mlcod 124'6159 active+clean+snaptrim trimq=[211~17] ps=[204~d]] removing snap head 2019-12-05T06:51:20.563 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:20.557+0000 7fc70e830700 -1 osd.5 pg_epoch: 124 pg[3.5s0( v 124'6165 (117'3109,124'6165] local-lis/les=21/22 n=104 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=124'6165 lcod 124'6163 mlcod 124'6163 active+clean+snaptrim trimq=[211~17] ps=[204~d]] removing snap head 2019-12-05T06:51:20.564 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:20.557+0000 7fc70e830700 -1 osd.5 pg_epoch: 124 pg[3.5s0( v 124'6167 (117'3109,124'6167] local-lis/les=21/22 n=102 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=124'6165 crt=124'6165 lcod 124'6163 mlcod 124'6163 active+clean+snaptrim trimq=[211~17] ps=[204~d]] removing snap head 2019-12-05T06:51:20.567 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:20.561+0000 7fc712838700 -1 osd.5 pg_epoch: 124 pg[3.5s0( v 124'6169 (117'3109,124'6169] local-lis/les=21/22 n=100 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=124'6169 lcod 124'6167 mlcod 124'6167 active+clean+snaptrim trimq=[211~17] ps=[204~d]] removing snap head 2019-12-05T06:51:20.568 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:20.561+0000 7fc712838700 -1 osd.5 pg_epoch: 124 pg[3.5s0( v 124'6171 (117'3109,124'6171] local-lis/les=21/22 n=98 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=124'6169 crt=124'6169 lcod 124'6167 mlcod 124'6167 active+clean+snaptrim trimq=[211~17] ps=[204~d]] removing snap head 2019-12-05T06:51:20.576 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:20.569+0000 7f426eb14700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6098 (116'3007,124'6098] local-lis/les=21/22 n=428 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=124'6098 lcod 124'6096 mlcod 124'6096 active+clean+snaptrim trimq=[207~21] ps=[204~3]] removing snap head 2019-12-05T06:51:20.578 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:20.573+0000 7f426eb14700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6100 (116'3007,124'6100] local-lis/les=21/22 n=426 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=124'6098 crt=124'6098 lcod 124'6096 mlcod 124'6096 active+clean+snaptrim trimq=[207~21] ps=[204~3]] removing snap head 2019-12-05T06:51:20.593 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:20.577+0000 7f426eb14700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6102 (116'3007,124'6102] local-lis/les=21/22 n=424 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=124'6102 lcod 124'6100 mlcod 124'6100 active+clean+snaptrim trimq=[207~21] ps=[204~3]] removing snap head 2019-12-05T06:51:20.593 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:20.581+0000 7f426eb14700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6104 (116'3007,124'6104] local-lis/les=21/22 n=422 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=124'6102 crt=124'6102 lcod 124'6100 mlcod 124'6100 active+clean+snaptrim trimq=[207~21] ps=[204~3]] removing snap head 2019-12-05T06:51:20.594 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:20.589+0000 7f4272b1c700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6108 (117'3107,124'6108] local-lis/les=21/22 n=418 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=124'6106 crt=124'6106 lcod 124'6104 mlcod 124'6104 active+clean+snaptrim trimq=[207~21] ps=[204~3]] removing snap head 2019-12-05T06:51:20.596 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:20.589+0000 7f4272b1c700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6110 (117'3107,124'6110] local-lis/les=21/22 n=416 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=124'6106 crt=124'6106 lcod 124'6104 mlcod 124'6104 active+clean+snaptrim trimq=[207~21] ps=[204~3]] removing snap head 2019-12-05T06:51:20.603 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:20.597+0000 7f4272b1c700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6112 (117'3107,124'6112] local-lis/les=21/22 n=414 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=124'6112 lcod 124'6110 mlcod 124'6110 active+clean+snaptrim trimq=[207~21] ps=[204~3]] removing snap head 2019-12-05T06:51:20.604 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:20.598+0000 7fc9c444d700 -1 osd.3 pg_epoch: 124 pg[3.2s0( v 124'6242 (117'3207,124'6242] local-lis/les=21/22 n=147 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=124'6241 crt=124'6239 lcod 124'6239 mlcod 124'6239 active+clean+snaptrim trimq=[20f~19] ps=[204~b]] removing snap head 2019-12-05T06:51:20.605 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:20.602+0000 7fc9c444d700 -1 osd.3 pg_epoch: 124 pg[3.2s0( v 124'6244 (117'3207,124'6244] local-lis/les=21/22 n=145 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=124'6241 crt=124'6241 lcod 124'6239 mlcod 124'6239 active+clean+snaptrim trimq=[20f~19] ps=[204~b]] removing snap head 2019-12-05T06:51:20.606 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:20.597+0000 7f4272b1c700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6114 (117'3107,124'6114] local-lis/les=21/22 n=412 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=124'6112 crt=124'6112 lcod 124'6110 mlcod 124'6110 active+clean+snaptrim trimq=[207~21] ps=[204~3]] removing snap head 2019-12-05T06:51:20.614 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:20.605+0000 7fc712838700 -1 osd.5 pg_epoch: 124 pg[3.5s0( v 124'6174 (117'3109,124'6174] local-lis/les=21/22 n=95 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=124'6173 crt=124'6171 lcod 124'6171 mlcod 124'6171 active+clean+snaptrim trimq=[211~17] ps=[204~d]] removing snap head 2019-12-05T06:51:20.614 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:20.605+0000 7fc712838700 -1 osd.5 pg_epoch: 124 pg[3.5s0( v 124'6176 (117'3109,124'6176] local-lis/les=21/22 n=93 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=124'6173 crt=124'6173 lcod 124'6171 mlcod 124'6171 active+clean+snaptrim trimq=[211~17] ps=[204~d]] removing snap head 2019-12-05T06:51:20.619 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:20.613+0000 7f4272b1c700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6116 (117'3107,124'6116] local-lis/les=21/22 n=410 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=124'6116 lcod 124'6114 mlcod 124'6114 active+clean+snaptrim trimq=[207~21] ps=[204~3]] removing snap head 2019-12-05T06:51:20.630 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:20.621+0000 7f426eb14700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6121 (117'3107,124'6121] local-lis/les=21/22 n=406 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=124'6120 crt=124'6118 lcod 124'6118 mlcod 124'6118 active+clean+snaptrim trimq=[207~21] ps=[204~3]] removing snap head 2019-12-05T06:51:20.631 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:20.625+0000 7f426eb14700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6123 (117'3107,124'6123] local-lis/les=21/22 n=404 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=124'6120 crt=124'6120 lcod 124'6118 mlcod 124'6118 active+clean+snaptrim trimq=[207~21] ps=[204~3]] removing snap head 2019-12-05T06:51:20.652 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:20.645+0000 7fc712838700 -1 osd.5 pg_epoch: 124 pg[3.5s0( v 124'6178 (117'3109,124'6178] local-lis/les=21/22 n=91 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=124'6178 lcod 124'6176 mlcod 124'6176 active+clean+snaptrim trimq=[211~17] ps=[204~d]] removing snap head 2019-12-05T06:51:20.654 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:20.649+0000 7fc712838700 -1 osd.5 pg_epoch: 124 pg[3.5s0( v 124'6180 (117'3109,124'6180] local-lis/les=21/22 n=89 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=124'6178 crt=124'6178 lcod 124'6176 mlcod 124'6176 active+clean+snaptrim trimq=[211~17] ps=[204~d]] removing snap head 2019-12-05T06:51:20.659 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:20.654+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 124 pg[3.1s0( v 124'6474 (117'3415,124'6474] local-lis/les=21/22 n=203 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=124'6474 lcod 124'6472 mlcod 124'6472 active+clean+snaptrim trimq=[20f~19] ps=[204~b]] removing snap head 2019-12-05T06:51:20.659 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:20.654+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 124 pg[3.7s0( v 124'6188 (117'3110,124'6188] local-lis/les=21/22 n=207 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=124'6187 crt=124'6185 lcod 124'6185 mlcod 124'6185 active+clean+snaptrim trimq=[20e~1a] ps=[204~a]] removing snap head 2019-12-05T06:51:20.661 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:20.658+0000 7fc9c0445700 -1 osd.3 pg_epoch: 124 pg[3.2s0( v 124'6247 (117'3207,124'6247] local-lis/les=21/22 n=142 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=124'6246 crt=124'6244 lcod 124'6244 mlcod 124'6244 active+clean+snaptrim trimq=[20f~19] ps=[204~b]] removing snap head 2019-12-05T06:51:20.672 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:20.658+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 124 pg[3.1s0( v 124'6476 (117'3415,124'6476] local-lis/les=21/22 n=201 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=124'6474 crt=124'6474 lcod 124'6472 mlcod 124'6472 active+clean+snaptrim trimq=[20f~19] ps=[204~b]] removing snap head 2019-12-05T06:51:20.676 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:20.669+0000 7fc70e830700 -1 osd.5 pg_epoch: 124 pg[3.5s0( v 124'6182 (117'3109,124'6182] local-lis/les=21/22 n=87 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=124'6182 lcod 124'6180 mlcod 124'6180 active+clean+snaptrim trimq=[211~17] ps=[204~d]] removing snap head 2019-12-05T06:51:20.678 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:20.673+0000 7fc70e830700 -1 osd.5 pg_epoch: 124 pg[3.5s0( v 124'6184 (117'3109,124'6184] local-lis/les=21/22 n=85 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=124'6182 crt=124'6182 lcod 124'6180 mlcod 124'6180 active+clean+snaptrim trimq=[211~17] ps=[204~d]] removing snap head 2019-12-05T06:51:20.679 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:20.673+0000 7f4272b1c700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6126 (117'3107,124'6126] local-lis/les=21/22 n=401 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=124'6125 crt=124'6123 lcod 124'6123 mlcod 124'6123 active+clean+snaptrim trimq=[207~21] ps=[204~3]] removing snap head 2019-12-05T06:51:20.687 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:20.682+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 124 pg[3.7s0( v 124'6190 (117'3110,124'6190] local-lis/les=21/22 n=205 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=124'6190 lcod 124'6188 mlcod 124'6188 active+clean+snaptrim trimq=[20f~19] ps=[204~b]] removing snap head 2019-12-05T06:51:20.687 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:20.682+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 124 pg[3.1s0( v 124'6479 (117'3415,124'6479] local-lis/les=21/22 n=198 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=124'6478 crt=124'6474 lcod 124'6476 mlcod 124'6476 active+clean+snaptrim trimq=[20f~19] ps=[204~b]] removing snap head 2019-12-05T06:51:20.689 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:20.686+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 124 pg[3.7s0( v 124'6192 (117'3110,124'6192] local-lis/les=21/22 n=203 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=124'6190 crt=124'6190 lcod 124'6188 mlcod 124'6188 active+clean+snaptrim trimq=[20f~19] ps=[204~b]] removing snap head 2019-12-05T06:51:20.690 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:20.686+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 124 pg[3.1s0( v 124'6481 (117'3415,124'6481] local-lis/les=21/22 n=196 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=124'6478 crt=124'6478 lcod 124'6476 mlcod 124'6476 active+clean+snaptrim trimq=[20f~19] ps=[204~b]] removing snap head 2019-12-05T06:51:20.706 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:20.702+0000 7fc9c444d700 -1 osd.3 pg_epoch: 124 pg[3.2s0( v 124'6250 (117'3207,124'6250] local-lis/les=21/22 n=139 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=124'6249 crt=124'6247 lcod 124'6247 mlcod 124'6247 active+clean+snaptrim trimq=[210~18] ps=[204~c]] removing snap head 2019-12-05T06:51:20.709 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:20.706+0000 7fc9c444d700 -1 osd.3 pg_epoch: 124 pg[3.2s0( v 124'6252 (117'3207,124'6252] local-lis/les=21/22 n=137 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=124'6249 crt=124'6249 lcod 124'6247 mlcod 124'6247 active+clean+snaptrim trimq=[210~18] ps=[204~c]] removing snap head 2019-12-05T06:51:20.715 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:20.709+0000 7fc712838700 -1 osd.5 pg_epoch: 124 pg[3.5s0( v 124'6186 (117'3109,124'6186] local-lis/les=21/22 n=83 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=124'6186 lcod 124'6184 mlcod 124'6184 active+clean+snaptrim trimq=[211~17] ps=[204~d]] removing snap head 2019-12-05T06:51:20.717 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:20.709+0000 7fc712838700 -1 osd.5 pg_epoch: 124 pg[3.5s0( v 124'6188 (117'3109,124'6188] local-lis/les=21/22 n=81 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=124'6186 crt=124'6186 lcod 124'6184 mlcod 124'6184 active+clean+snaptrim trimq=[211~17] ps=[204~d]] removing snap head 2019-12-05T06:51:20.719 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:20.714+0000 7fc9c0445700 -1 osd.3 pg_epoch: 124 pg[3.2s0( v 124'6255 (117'3207,124'6255] local-lis/les=21/22 n=134 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=124'6254 crt=124'6252 lcod 124'6252 mlcod 124'6252 active+clean+snaptrim trimq=[210~18] ps=[204~c]] removing snap head 2019-12-05T06:51:20.721 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:20.718+0000 7fc9c0445700 -1 osd.3 pg_epoch: 124 pg[3.2s0( v 124'6257 (117'3207,124'6257] local-lis/les=21/22 n=132 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=124'6254 crt=124'6254 lcod 124'6252 mlcod 124'6252 active+clean+snaptrim trimq=[210~18] ps=[204~c]] removing snap head 2019-12-05T06:51:20.730 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:20.721+0000 7f4272b1c700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6129 (117'3107,124'6129] local-lis/les=21/22 n=398 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=124'6128 crt=124'6126 lcod 124'6126 mlcod 124'6126 active+clean+snaptrim trimq=[208~20] ps=[204~4]] removing snap head 2019-12-05T06:51:20.732 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:20.725+0000 7f4272b1c700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6131 (117'3107,124'6131] local-lis/les=21/22 n=396 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=124'6128 crt=124'6128 lcod 124'6126 mlcod 124'6126 active+clean+snaptrim trimq=[208~20] ps=[204~4]] removing snap head 2019-12-05T06:51:20.832 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:20.830+0000 7fc9c444d700 -1 osd.3 pg_epoch: 124 pg[3.2s0( v 124'6259 (117'3207,124'6259] local-lis/les=21/22 n=130 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=124'6259 lcod 124'6257 mlcod 124'6257 active+clean+snaptrim trimq=[210~18] ps=[204~c]] removing snap head 2019-12-05T06:51:20.834 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:20.830+0000 7fc9c444d700 -1 osd.3 pg_epoch: 124 pg[3.2s0( v 124'6261 (117'3207,124'6261] local-lis/les=21/22 n=128 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=124'6259 crt=124'6259 lcod 124'6257 mlcod 124'6257 active+clean+snaptrim trimq=[210~18] ps=[204~c]] removing snap head 2019-12-05T06:51:20.839 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:20.834+0000 7fc9c0445700 -1 osd.3 pg_epoch: 124 pg[3.2s0( v 124'6263 (117'3207,124'6263] local-lis/les=21/22 n=126 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=124'6263 lcod 124'6261 mlcod 124'6261 active+clean+snaptrim trimq=[210~18] ps=[204~c]] removing snap head 2019-12-05T06:51:20.841 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:20.838+0000 7fc9c0445700 -1 osd.3 pg_epoch: 124 pg[3.2s0( v 124'6265 (117'3207,124'6265] local-lis/les=21/22 n=124 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=124'6263 crt=124'6263 lcod 124'6261 mlcod 124'6261 active+clean+snaptrim trimq=[210~18] ps=[204~c]] removing snap head 2019-12-05T06:51:20.847 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:20.842+0000 7fc9c0445700 -1 osd.3 pg_epoch: 124 pg[3.2s0( v 124'6267 (117'3207,124'6267] local-lis/les=21/22 n=122 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=124'6267 lcod 124'6265 mlcod 124'6265 active+clean+snaptrim trimq=[210~18] ps=[204~c]] removing snap head 2019-12-05T06:51:20.851 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:20.846+0000 7fc9c0445700 -1 osd.3 pg_epoch: 124 pg[3.2s0( v 124'6269 (117'3207,124'6269] local-lis/les=21/22 n=120 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=124'6267 crt=124'6267 lcod 124'6265 mlcod 124'6265 active+clean+snaptrim trimq=[210~18] ps=[204~c]] removing snap head 2019-12-05T06:51:20.856 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:20.854+0000 7fc9c0445700 -1 osd.3 pg_epoch: 124 pg[3.2s0( v 124'6271 (117'3207,124'6271] local-lis/les=21/22 n=118 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=124'6271 lcod 124'6269 mlcod 124'6269 active+clean+snaptrim trimq=[210~18] ps=[204~c]] removing snap head 2019-12-05T06:51:20.860 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:20.858+0000 7fc9c0445700 -1 osd.3 pg_epoch: 124 pg[3.2s0( v 124'6273 (117'3207,124'6273] local-lis/les=21/22 n=116 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=124'6271 crt=124'6271 lcod 124'6269 mlcod 124'6269 active+clean+snaptrim trimq=[210~18] ps=[204~c]] removing snap head 2019-12-05T06:51:20.864 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:20.862+0000 7fc9c444d700 -1 osd.3 pg_epoch: 124 pg[3.2s0( v 124'6275 (117'3207,124'6275] local-lis/les=21/22 n=114 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=124'6275 lcod 124'6273 mlcod 124'6273 active+clean+snaptrim trimq=[210~18] ps=[204~c]] removing snap head 2019-12-05T06:51:20.866 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:20.862+0000 7fc9c444d700 -1 osd.3 pg_epoch: 124 pg[3.2s0( v 124'6277 (117'3207,124'6277] local-lis/les=21/22 n=112 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=124'6275 crt=124'6275 lcod 124'6273 mlcod 124'6273 active+clean+snaptrim trimq=[210~18] ps=[204~c]] removing snap head 2019-12-05T06:51:20.872 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:20.870+0000 7fc9c444d700 -1 osd.3 pg_epoch: 124 pg[3.2s0( v 124'6279 (117'3207,124'6279] local-lis/les=21/22 n=110 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=124'6279 lcod 124'6277 mlcod 124'6277 active+clean+snaptrim trimq=[211~17] ps=[204~d]] removing snap head 2019-12-05T06:51:20.874 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:20.870+0000 7fc9c444d700 -1 osd.3 pg_epoch: 124 pg[3.2s0( v 124'6281 (117'3207,124'6281] local-lis/les=21/22 n=108 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=124'6279 crt=124'6279 lcod 124'6277 mlcod 124'6277 active+clean+snaptrim trimq=[211~17] ps=[204~d]] removing snap head 2019-12-05T06:51:20.879 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:20.874+0000 7fc9c0445700 -1 osd.3 pg_epoch: 124 pg[3.2s0( v 124'6283 (117'3207,124'6283] local-lis/les=21/22 n=106 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=124'6283 lcod 124'6281 mlcod 124'6281 active+clean+snaptrim trimq=[211~17] ps=[204~d]] removing snap head 2019-12-05T06:51:20.881 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:20.878+0000 7fc9c0445700 -1 osd.3 pg_epoch: 124 pg[3.2s0( v 124'6285 (117'3207,124'6285] local-lis/les=21/22 n=104 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=124'6283 crt=124'6283 lcod 124'6281 mlcod 124'6281 active+clean+snaptrim trimq=[211~17] ps=[204~d]] removing snap head 2019-12-05T06:51:20.886 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:20.882+0000 7fc9c444d700 -1 osd.3 pg_epoch: 124 pg[3.2s0( v 124'6287 (117'3207,124'6287] local-lis/les=21/22 n=102 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=124'6287 lcod 124'6285 mlcod 124'6285 active+clean+snaptrim trimq=[211~17] ps=[204~d]] removing snap head 2019-12-05T06:51:20.887 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:20.882+0000 7fc9c444d700 -1 osd.3 pg_epoch: 124 pg[3.2s0( v 124'6289 (117'3207,124'6289] local-lis/les=21/22 n=100 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=124'6287 crt=124'6287 lcod 124'6285 mlcod 124'6285 active+clean+snaptrim trimq=[211~17] ps=[204~d]] removing snap head 2019-12-05T06:51:20.891 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:20.886+0000 7fc9c444d700 -1 osd.3 pg_epoch: 124 pg[3.2s0( v 124'6291 (117'3207,124'6291] local-lis/les=21/22 n=98 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=124'6291 lcod 124'6289 mlcod 124'6289 active+clean+snaptrim trimq=[211~17] ps=[204~d]] removing snap head 2019-12-05T06:51:20.893 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:20.890+0000 7fc9c444d700 -1 osd.3 pg_epoch: 124 pg[3.2s0( v 124'6293 (117'3207,124'6293] local-lis/les=21/22 n=96 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=124'6291 crt=124'6291 lcod 124'6289 mlcod 124'6289 active+clean+snaptrim trimq=[211~17] ps=[204~d]] removing snap head 2019-12-05T06:51:20.898 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:20.894+0000 7fc9c444d700 -1 osd.3 pg_epoch: 124 pg[3.2s0( v 124'6295 (117'3207,124'6295] local-lis/les=21/22 n=94 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=124'6295 lcod 124'6293 mlcod 124'6293 active+clean+snaptrim trimq=[211~17] ps=[204~d]] removing snap head 2019-12-05T06:51:20.900 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:20.898+0000 7fc9c444d700 -1 osd.3 pg_epoch: 124 pg[3.2s0( v 124'6297 (117'3207,124'6297] local-lis/les=21/22 n=92 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=124'6295 crt=124'6295 lcod 124'6293 mlcod 124'6293 active+clean+snaptrim trimq=[211~17] ps=[204~d]] removing snap head 2019-12-05T06:51:20.904 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:20.902+0000 7fc9c0445700 -1 osd.3 pg_epoch: 124 pg[3.2s0( v 124'6299 (117'3207,124'6299] local-lis/les=21/22 n=90 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=124'6299 lcod 124'6297 mlcod 124'6297 active+clean+snaptrim trimq=[211~17] ps=[204~d]] removing snap head 2019-12-05T06:51:20.906 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:20.902+0000 7fc9c0445700 -1 osd.3 pg_epoch: 124 pg[3.2s0( v 124'6301 (117'3207,124'6301] local-lis/les=21/22 n=88 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=124'6299 crt=124'6299 lcod 124'6297 mlcod 124'6297 active+clean+snaptrim trimq=[211~17] ps=[204~d]] removing snap head 2019-12-05T06:51:20.912 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:20.906+0000 7fc9c0445700 -1 osd.3 pg_epoch: 124 pg[3.2s0( v 124'6303 (117'3207,124'6303] local-lis/les=21/22 n=86 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=124'6303 lcod 124'6301 mlcod 124'6301 active+clean+snaptrim trimq=[211~17] ps=[204~d]] removing snap head 2019-12-05T06:51:20.913 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:20.910+0000 7fc9c0445700 -1 osd.3 pg_epoch: 124 pg[3.2s0( v 124'6305 (117'3207,124'6305] local-lis/les=21/22 n=84 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=124'6303 crt=124'6303 lcod 124'6301 mlcod 124'6301 active+clean+snaptrim trimq=[211~17] ps=[204~d]] removing snap head 2019-12-05T06:51:20.917 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:20.914+0000 7fc9c444d700 -1 osd.3 pg_epoch: 124 pg[3.2s0( v 124'6307 (117'3207,124'6307] local-lis/les=21/22 n=82 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=124'6307 lcod 124'6305 mlcod 124'6305 active+clean+snaptrim trimq=[211~17] ps=[204~d]] removing snap head 2019-12-05T06:51:20.919 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:20.914+0000 7fc9c444d700 -1 osd.3 pg_epoch: 124 pg[3.2s0( v 124'6309 (117'3307,124'6309] local-lis/les=21/22 n=80 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=124'6307 crt=124'6307 lcod 124'6305 mlcod 124'6305 active+clean+snaptrim trimq=[211~17] ps=[204~d]] removing snap head 2019-12-05T06:51:20.920 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:51:20.931 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:20.922+0000 7fc9c0445700 -1 osd.3 pg_epoch: 124 pg[3.2s0( v 124'6311 (117'3307,124'6311] local-lis/les=21/22 n=78 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=124'6311 lcod 124'6309 mlcod 124'6309 active+clean+snaptrim trimq=[211~17] ps=[204~d]] removing snap head 2019-12-05T06:51:20.931 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:20.922+0000 7fc9c0445700 -1 osd.3 pg_epoch: 124 pg[3.2s0( v 124'6313 (117'3307,124'6313] local-lis/les=21/22 n=76 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=124'6311 crt=124'6311 lcod 124'6309 mlcod 124'6309 active+clean+snaptrim trimq=[211~17] ps=[204~d]] removing snap head 2019-12-05T06:51:20.931 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:51:20.988 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:20.986+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 124 pg[3.7s0( v 124'6195 (117'3110,124'6195] local-lis/les=21/22 n=200 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=124'6194 crt=124'6192 lcod 124'6192 mlcod 124'6192 active+clean+snaptrim trimq=[20f~19] ps=[204~b]] removing snap head 2019-12-05T06:51:20.990 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:20.986+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 124 pg[3.7s0( v 124'6197 (117'3110,124'6197] local-lis/les=21/22 n=198 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=124'6194 crt=124'6194 lcod 124'6192 mlcod 124'6192 active+clean+snaptrim trimq=[20f~19] ps=[204~b]] removing snap head 2019-12-05T06:51:20.991 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:20.986+0000 7fc9c0445700 -1 osd.3 pg_epoch: 124 pg[3.2s0( v 124'6315 (117'3307,124'6315] local-lis/les=21/22 n=74 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=124'6315 lcod 124'6313 mlcod 124'6313 active+clean+snaptrim trimq=[211~17] ps=[204~d]] removing snap head 2019-12-05T06:51:20.993 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:20.986+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 124 pg[3.1s0( v 124'6483 (117'3415,124'6483] local-lis/les=21/22 n=194 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=124'6483 lcod 124'6481 mlcod 124'6481 active+clean+snaptrim trimq=[20f~19] ps=[204~b]] removing snap head 2019-12-05T06:51:20.993 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:20.990+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 124 pg[3.1s0( v 124'6485 (117'3415,124'6485] local-lis/les=21/22 n=192 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=124'6483 crt=124'6483 lcod 124'6481 mlcod 124'6481 active+clean+snaptrim trimq=[20f~19] ps=[204~b]] removing snap head 2019-12-05T06:51:20.994 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:20.990+0000 7fc9c0445700 -1 osd.3 pg_epoch: 124 pg[3.2s0( v 124'6317 (117'3307,124'6317] local-lis/les=21/22 n=72 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=124'6315 crt=124'6315 lcod 124'6313 mlcod 124'6313 active+clean+snaptrim trimq=[211~17] ps=[204~d]] removing snap head 2019-12-05T06:51:21.002 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:20.997+0000 7f4272b1c700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6134 (117'3107,124'6134] local-lis/les=21/22 n=393 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=124'6133 crt=124'6131 lcod 124'6131 mlcod 124'6131 active+clean+snaptrim trimq=[208~20] ps=[204~4]] removing snap head 2019-12-05T06:51:21.003 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:20.998+0000 7fc9c0445700 -1 osd.3 pg_epoch: 124 pg[3.2s0( v 124'6320 (117'3307,124'6320] local-lis/les=21/22 n=69 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=124'6319 crt=124'6315 lcod 124'6317 mlcod 124'6317 active+clean+snaptrim trimq=[211~17] ps=[204~d]] removing snap head 2019-12-05T06:51:21.006 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:21.002+0000 7fc9c0445700 -1 osd.3 pg_epoch: 124 pg[3.2s0( v 124'6322 (117'3307,124'6322] local-lis/les=21/22 n=67 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=124'6319 crt=124'6319 lcod 124'6317 mlcod 124'6317 active+clean+snaptrim trimq=[211~17] ps=[204~d]] removing snap head 2019-12-05T06:51:21.006 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:21.002+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 124 pg[3.1s0( v 124'6487 (117'3415,124'6487] local-lis/les=21/22 n=190 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=124'6487 lcod 124'6485 mlcod 124'6485 active+clean+snaptrim trimq=[20f~19] ps=[204~b]] removing snap head 2019-12-05T06:51:21.008 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:21.006+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 124 pg[3.1s0( v 124'6489 (117'3415,124'6489] local-lis/les=21/22 n=188 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=124'6487 crt=124'6487 lcod 124'6485 mlcod 124'6485 active+clean+snaptrim trimq=[20f~19] ps=[204~b]] removing snap head 2019-12-05T06:51:21.010 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:21.006+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 124 pg[3.7s0( v 124'6199 (117'3110,124'6199] local-lis/les=21/22 n=196 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=124'6199 lcod 124'6197 mlcod 124'6197 active+clean+snaptrim trimq=[20f~19] ps=[204~b]] removing snap head 2019-12-05T06:51:21.011 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:20.997+0000 7f4272b1c700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6136 (117'3107,124'6136] local-lis/les=21/22 n=391 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=124'6133 crt=124'6133 lcod 124'6131 mlcod 124'6131 active+clean+snaptrim trimq=[208~20] ps=[204~4]] removing snap head 2019-12-05T06:51:21.012 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:21.006+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 124 pg[3.7s0( v 124'6201 (117'3110,124'6201] local-lis/les=21/22 n=194 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=124'6199 crt=124'6199 lcod 124'6197 mlcod 124'6197 active+clean+snaptrim trimq=[20f~19] ps=[204~b]] removing snap head 2019-12-05T06:51:21.017 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:21.014+0000 7fc9c444d700 -1 osd.3 pg_epoch: 124 pg[3.2s0( v 124'6324 (117'3307,124'6324] local-lis/les=21/22 n=65 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=124'6324 lcod 124'6322 mlcod 124'6322 active+clean+snaptrim trimq=[211~17] ps=[204~d]] removing snap head 2019-12-05T06:51:21.019 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:21.014+0000 7fc9c444d700 -1 osd.3 pg_epoch: 124 pg[3.2s0( v 124'6326 (117'3307,124'6326] local-lis/les=21/22 n=63 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=124'6324 crt=124'6324 lcod 124'6322 mlcod 124'6322 active+clean+snaptrim trimq=[211~17] ps=[204~d]] removing snap head 2019-12-05T06:51:21.020 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:21.018+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 124 pg[3.7s0( v 124'6204 (117'3110,124'6204] local-lis/les=21/22 n=191 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=124'6203 crt=124'6199 lcod 124'6201 mlcod 124'6201 active+clean+snaptrim trimq=[20f~19] ps=[204~b]] removing snap head 2019-12-05T06:51:21.022 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:21.018+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 124 pg[3.7s0( v 124'6206 (117'3110,124'6206] local-lis/les=21/22 n=189 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=124'6203 crt=124'6203 lcod 124'6201 mlcod 124'6201 active+clean+snaptrim trimq=[20f~19] ps=[204~b]] removing snap head 2019-12-05T06:51:21.033 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:21.030+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 124 pg[3.7s0( v 124'6209 (117'3110,124'6209] local-lis/les=21/22 n=186 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=124'6208 crt=124'6206 lcod 124'6206 mlcod 124'6206 active+clean+snaptrim trimq=[20f~19] ps=[204~b]] removing snap head 2019-12-05T06:51:21.035 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:21.030+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 124 pg[3.7s0( v 124'6211 (117'3110,124'6211] local-lis/les=21/22 n=184 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=124'6208 crt=124'6208 lcod 124'6206 mlcod 124'6206 active+clean+snaptrim trimq=[20f~19] ps=[204~b]] removing snap head 2019-12-05T06:51:21.043 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:21.038+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 124 pg[3.1s0( v 124'6491 (117'3415,124'6491] local-lis/les=21/22 n=186 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=124'6491 lcod 124'6489 mlcod 124'6489 active+clean+snaptrim trimq=[20f~19] ps=[204~b]] removing snap head 2019-12-05T06:51:21.044 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:21.042+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 124 pg[3.7s0( v 124'6213 (118'3211,124'6213] local-lis/les=21/22 n=182 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=124'6213 lcod 124'6211 mlcod 124'6211 active+clean+snaptrim trimq=[20f~19] ps=[204~b]] removing snap head 2019-12-05T06:51:21.045 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:21.042+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 124 pg[3.1s0( v 124'6493 (117'3415,124'6493] local-lis/les=21/22 n=184 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=124'6491 crt=124'6491 lcod 124'6489 mlcod 124'6489 active+clean+snaptrim trimq=[20f~19] ps=[204~b]] removing snap head 2019-12-05T06:51:21.046 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:21.042+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 124 pg[3.7s0( v 124'6215 (118'3211,124'6215] local-lis/les=21/22 n=180 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=124'6213 crt=124'6213 lcod 124'6211 mlcod 124'6211 active+clean+snaptrim trimq=[20f~19] ps=[204~b]] removing snap head 2019-12-05T06:51:21.072 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:21.050+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 124 pg[3.1s0( v 124'6495 (117'3415,124'6495] local-lis/les=21/22 n=182 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=124'6495 lcod 124'6493 mlcod 124'6493 active+clean+snaptrim trimq=[20f~19] ps=[204~b]] removing snap head 2019-12-05T06:51:21.073 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:21.054+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 124 pg[3.1s0( v 124'6497 (117'3415,124'6497] local-lis/les=21/22 n=180 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=124'6495 crt=124'6495 lcod 124'6493 mlcod 124'6493 active+clean+snaptrim trimq=[20f~19] ps=[204~b]] removing snap head 2019-12-05T06:51:21.073 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:21.054+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 124 pg[3.7s0( v 124'6217 (118'3211,124'6217] local-lis/les=21/22 n=178 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=124'6217 lcod 124'6215 mlcod 124'6215 active+clean+snaptrim trimq=[20f~19] ps=[204~b]] removing snap head 2019-12-05T06:51:21.073 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:21.058+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 124 pg[3.7s0( v 124'6219 (118'3211,124'6219] local-lis/les=21/22 n=176 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=124'6217 crt=124'6217 lcod 124'6215 mlcod 124'6215 active+clean+snaptrim trimq=[20f~19] ps=[204~b]] removing snap head 2019-12-05T06:51:21.087 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:21.082+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 124 pg[3.7s0( v 124'6221 (118'3211,124'6221] local-lis/les=21/22 n=174 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=124'6221 lcod 124'6219 mlcod 124'6219 active+clean+snaptrim trimq=[20f~19] ps=[204~b]] removing snap head 2019-12-05T06:51:21.089 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:21.086+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 124 pg[3.7s0( v 124'6223 (118'3211,124'6223] local-lis/les=21/22 n=172 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=124'6221 crt=124'6221 lcod 124'6219 mlcod 124'6219 active+clean+snaptrim trimq=[20f~19] ps=[204~b]] removing snap head 2019-12-05T06:51:21.154 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:21.145+0000 7fc712838700 -1 osd.5 pg_epoch: 124 pg[3.5s0( v 124'6190 (117'3109,124'6190] local-lis/les=21/22 n=79 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=124'6190 lcod 124'6188 mlcod 124'6188 active+clean+snaptrim trimq=[211~17] ps=[204~d]] removing snap head 2019-12-05T06:51:21.157 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:21.149+0000 7fc712838700 -1 osd.5 pg_epoch: 124 pg[3.5s0( v 124'6192 (117'3109,124'6192] local-lis/les=21/22 n=77 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=124'6190 crt=124'6190 lcod 124'6188 mlcod 124'6188 active+clean+snaptrim trimq=[211~17] ps=[204~d]] removing snap head 2019-12-05T06:51:21.164 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:21.157+0000 7fc70e830700 -1 osd.5 pg_epoch: 124 pg[3.5s0( v 124'6194 (117'3109,124'6194] local-lis/les=21/22 n=75 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=124'6194 lcod 124'6192 mlcod 124'6192 active+clean+snaptrim trimq=[211~17] ps=[204~d]] removing snap head 2019-12-05T06:51:21.166 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:21.157+0000 7fc70e830700 -1 osd.5 pg_epoch: 124 pg[3.5s0( v 124'6196 (117'3109,124'6196] local-lis/les=21/22 n=73 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=124'6194 crt=124'6194 lcod 124'6192 mlcod 124'6192 active+clean+snaptrim trimq=[211~17] ps=[204~d]] removing snap head 2019-12-05T06:51:21.171 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:21.165+0000 7fc70e830700 -1 osd.5 pg_epoch: 124 pg[3.5s0( v 124'6198 (117'3109,124'6198] local-lis/les=21/22 n=71 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=124'6198 lcod 124'6196 mlcod 124'6196 active+clean+snaptrim trimq=[211~17] ps=[204~d]] removing snap head 2019-12-05T06:51:21.173 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:21.165+0000 7fc70e830700 -1 osd.5 pg_epoch: 124 pg[3.5s0( v 124'6200 (117'3109,124'6200] local-lis/les=21/22 n=69 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=124'6198 crt=124'6198 lcod 124'6196 mlcod 124'6196 active+clean+snaptrim trimq=[211~17] ps=[204~d]] removing snap head 2019-12-05T06:51:21.177 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:21.169+0000 7fc70e830700 -1 osd.5 pg_epoch: 124 pg[3.5s0( v 124'6202 (117'3109,124'6202] local-lis/les=21/22 n=67 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 crt=124'6202 lcod 124'6200 mlcod 124'6200 active+clean+snaptrim trimq=[211~17] ps=[204~d]] removing snap head 2019-12-05T06:51:21.179 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:21.173+0000 7fc70e830700 -1 osd.5 pg_epoch: 124 pg[3.5s0( v 124'6204 (117'3109,124'6204] local-lis/les=21/22 n=65 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=124'6202 crt=124'6202 lcod 124'6200 mlcod 124'6200 active+clean+snaptrim trimq=[211~17] ps=[204~d]] removing snap head 2019-12-05T06:51:21.222 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:21.218+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 124 pg[3.1s0( v 124'6500 (117'3415,124'6500] local-lis/les=21/22 n=177 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=124'6499 crt=124'6495 lcod 124'6497 mlcod 124'6497 active+clean+snaptrim trimq=[20f~19] ps=[204~b]] removing snap head 2019-12-05T06:51:21.225 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:21.222+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 124 pg[3.1s0( v 124'6502 (117'3415,124'6502] local-lis/les=21/22 n=175 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=124'6499 crt=124'6499 lcod 124'6497 mlcod 124'6497 active+clean+snaptrim trimq=[20f~19] ps=[204~b]] removing snap head 2019-12-05T06:51:21.235 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:21.229+0000 7f4272b1c700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6142 (117'3107,124'6142] local-lis/les=21/22 n=386 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=124'6141 crt=124'6139 lcod 124'6139 mlcod 124'6139 active+clean+snaptrim trimq=[209~1f] ps=[204~5]] removing snap head 2019-12-05T06:51:21.236 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:21.234+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 124 pg[3.1s0( v 124'6504 (117'3415,124'6504] local-lis/les=21/22 n=173 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=124'6504 lcod 124'6502 mlcod 124'6502 active+clean+snaptrim trimq=[20f~19] ps=[204~b]] removing snap head 2019-12-05T06:51:21.238 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:21.234+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 124 pg[3.1s0( v 124'6506 (117'3415,124'6506] local-lis/les=21/22 n=171 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=124'6504 crt=124'6504 lcod 124'6502 mlcod 124'6502 active+clean+snaptrim trimq=[20f~19] ps=[204~b]] removing snap head 2019-12-05T06:51:21.238 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:21.234+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 124 pg[3.7s0( v 124'6225 (118'3211,124'6225] local-lis/les=21/22 n=170 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=124'6225 lcod 124'6223 mlcod 124'6223 active+clean+snaptrim trimq=[20f~19] ps=[204~b]] removing snap head 2019-12-05T06:51:21.240 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:21.233+0000 7f4272b1c700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6144 (117'3107,124'6144] local-lis/les=21/22 n=384 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=124'6141 crt=124'6141 lcod 124'6139 mlcod 124'6139 active+clean+snaptrim trimq=[209~1f] ps=[204~5]] removing snap head 2019-12-05T06:51:21.241 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:21.238+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 124 pg[3.7s0( v 124'6227 (118'3211,124'6227] local-lis/les=21/22 n=168 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=124'6225 crt=124'6225 lcod 124'6223 mlcod 124'6223 active+clean+snaptrim trimq=[20f~19] ps=[204~b]] removing snap head 2019-12-05T06:51:21.249 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:21.241+0000 7f426eb14700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6147 (117'3107,124'6147] local-lis/les=21/22 n=381 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=124'6146 crt=124'6144 lcod 124'6144 mlcod 124'6144 active+clean+snaptrim trimq=[209~1f] ps=[204~5]] removing snap head 2019-12-05T06:51:21.250 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:21.246+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 124 pg[3.1s0( v 124'6510 (117'3415,124'6510] local-lis/les=21/22 n=168 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=124'6508 crt=124'6508 lcod 124'6506 mlcod 124'6506 active+clean+snaptrim trimq=[20f~19] ps=[204~b]] removing snap head 2019-12-05T06:51:21.252 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:21.245+0000 7f426eb14700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6149 (117'3107,124'6149] local-lis/les=21/22 n=379 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=124'6146 crt=124'6146 lcod 124'6144 mlcod 124'6144 active+clean+snaptrim trimq=[209~1f] ps=[204~5]] removing snap head 2019-12-05T06:51:21.254 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:21.250+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 124 pg[3.7s0( v 124'6229 (118'3211,124'6229] local-lis/les=21/22 n=166 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=124'6229 lcod 124'6227 mlcod 124'6227 active+clean+snaptrim trimq=[20f~19] ps=[204~b]] removing snap head 2019-12-05T06:51:21.256 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:21.254+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 124 pg[3.7s0( v 124'6231 (118'3211,124'6231] local-lis/les=21/22 n=164 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=124'6229 crt=124'6229 lcod 124'6227 mlcod 124'6227 active+clean+snaptrim trimq=[20f~19] ps=[204~b]] removing snap head 2019-12-05T06:51:21.257 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:21.254+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 124 pg[3.1s0( v 124'6512 (117'3415,124'6512] local-lis/les=21/22 n=166 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=124'6512 lcod 124'6510 mlcod 124'6510 active+clean+snaptrim trimq=[20f~19] ps=[204~b]] removing snap head 2019-12-05T06:51:21.260 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:21.254+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 124 pg[3.1s0( v 124'6514 (117'3415,124'6514] local-lis/les=21/22 n=164 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=124'6512 crt=124'6512 lcod 124'6510 mlcod 124'6510 active+clean+snaptrim trimq=[20f~19] ps=[204~b]] removing snap head 2019-12-05T06:51:21.262 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:21.253+0000 7fc70e830700 -1 osd.5 pg_epoch: 124 pg[3.5s0( v 124'6207 (117'3109,124'6207] local-lis/les=21/22 n=62 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=124'6206 crt=124'6204 lcod 124'6204 mlcod 124'6204 active+clean+snaptrim trimq=[211~17] ps=[204~d]] removing snap head 2019-12-05T06:51:21.265 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:21.257+0000 7fc70e830700 -1 osd.5 pg_epoch: 124 pg[3.5s0( v 124'6209 (117'3109,124'6209] local-lis/les=21/22 n=60 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=124'6206 crt=124'6206 lcod 124'6204 mlcod 124'6204 active+clean+snaptrim trimq=[211~17] ps=[204~d]] removing snap head 2019-12-05T06:51:21.265 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:21.262+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 124 pg[3.7s0( v 124'6233 (118'3211,124'6233] local-lis/les=21/22 n=162 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=124'6233 lcod 124'6231 mlcod 124'6231 active+clean+snaptrim trimq=[20f~19] ps=[204~b]] removing snap head 2019-12-05T06:51:21.267 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:21.262+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 124 pg[3.7s0( v 124'6235 (118'3211,124'6235] local-lis/les=21/22 n=160 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=124'6233 crt=124'6233 lcod 124'6231 mlcod 124'6231 active+clean+snaptrim trimq=[20f~19] ps=[204~b]] removing snap head 2019-12-05T06:51:21.272 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:21.265+0000 7f426eb14700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6152 (117'3107,124'6152] local-lis/les=21/22 n=376 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=124'6151 crt=124'6149 lcod 124'6149 mlcod 124'6149 active+clean+snaptrim trimq=[209~1f] ps=[204~5]] removing snap head 2019-12-05T06:51:21.273 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:21.270+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 124 pg[3.7s0( v 124'6237 (118'3211,124'6237] local-lis/les=21/22 n=158 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=124'6237 lcod 124'6235 mlcod 124'6235 active+clean+snaptrim trimq=[20f~19] ps=[204~b]] removing snap head 2019-12-05T06:51:21.274 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:21.265+0000 7f426eb14700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6154 (117'3107,124'6154] local-lis/les=21/22 n=374 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=124'6151 crt=124'6151 lcod 124'6149 mlcod 124'6149 active+clean+snaptrim trimq=[209~1f] ps=[204~5]] removing snap head 2019-12-05T06:51:21.279 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:21.274+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 124 pg[3.1s0( v 124'6516 (117'3415,124'6516] local-lis/les=21/22 n=162 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=124'6516 lcod 124'6514 mlcod 124'6514 active+clean+snaptrim trimq=[20f~19] ps=[204~b]] removing snap head 2019-12-05T06:51:21.281 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:21.278+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 124 pg[3.7s0( v 124'6240 (118'3211,124'6240] local-lis/les=21/22 n=155 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=124'6239 crt=124'6239 lcod 124'6237 mlcod 124'6237 active+clean+snaptrim trimq=[210~18] ps=[204~c]] removing snap head 2019-12-05T06:51:21.282 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:21.278+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 124 pg[3.1s0( v 124'6518 (118'3516,124'6518] local-lis/les=21/22 n=160 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=124'6516 crt=124'6516 lcod 124'6514 mlcod 124'6514 active+clean+snaptrim trimq=[20f~19] ps=[204~b]] removing snap head 2019-12-05T06:51:21.284 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:21.282+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 124 pg[3.7s0( v 124'6242 (118'3211,124'6242] local-lis/les=21/22 n=153 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=124'6239 crt=124'6239 lcod 124'6237 mlcod 124'6237 active+clean+snaptrim trimq=[210~18] ps=[204~c]] removing snap head 2019-12-05T06:51:21.289 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:21.286+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 124 pg[3.7s0( v 124'6245 (118'3211,124'6245] local-lis/les=21/22 n=150 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=124'6244 crt=124'6242 lcod 124'6242 mlcod 124'6242 active+clean+snaptrim trimq=[210~18] ps=[204~c]] removing snap head 2019-12-05T06:51:21.291 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:21.286+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 124 pg[3.7s0( v 124'6247 (118'3211,124'6247] local-lis/les=21/22 n=148 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=124'6244 crt=124'6244 lcod 124'6242 mlcod 124'6242 active+clean+snaptrim trimq=[210~18] ps=[204~c]] removing snap head 2019-12-05T06:51:21.293 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:21.285+0000 7f426eb14700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6156 (117'3107,124'6156] local-lis/les=21/22 n=372 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=124'6156 lcod 124'6154 mlcod 124'6154 active+clean+snaptrim trimq=[209~1f] ps=[204~5]] removing snap head 2019-12-05T06:51:21.296 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:21.294+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 124 pg[3.7s0( v 124'6249 (118'3211,124'6249] local-lis/les=21/22 n=146 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=124'6249 lcod 124'6247 mlcod 124'6247 active+clean+snaptrim trimq=[210~18] ps=[204~c]] removing snap head 2019-12-05T06:51:21.296 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:21.289+0000 7f426eb14700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6158 (117'3107,124'6158] local-lis/les=21/22 n=370 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=124'6156 crt=124'6156 lcod 124'6154 mlcod 124'6154 active+clean+snaptrim trimq=[209~1f] ps=[204~5]] removing snap head 2019-12-05T06:51:21.298 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:21.294+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 124 pg[3.7s0( v 124'6251 (118'3211,124'6251] local-lis/les=21/22 n=144 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=124'6249 crt=124'6249 lcod 124'6247 mlcod 124'6247 active+clean+snaptrim trimq=[210~18] ps=[204~c]] removing snap head 2019-12-05T06:51:21.304 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:21.298+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 124 pg[3.1s0( v 124'6521 (118'3516,124'6521] local-lis/les=21/22 n=157 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=124'6520 crt=124'6516 lcod 124'6518 mlcod 124'6518 active+clean+snaptrim trimq=[20f~19] ps=[204~b]] removing snap head 2019-12-05T06:51:21.304 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:21.302+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 124 pg[3.1s0( v 124'6523 (118'3516,124'6523] local-lis/les=21/22 n=155 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=124'6520 crt=124'6520 lcod 124'6518 mlcod 124'6518 active+clean+snaptrim trimq=[20f~19] ps=[204~b]] removing snap head 2019-12-05T06:51:21.305 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:21.302+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 124 pg[3.7s0( v 124'6253 (118'3211,124'6253] local-lis/les=21/22 n=142 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=124'6253 lcod 124'6251 mlcod 124'6251 active+clean+snaptrim trimq=[210~18] ps=[204~c]] removing snap head 2019-12-05T06:51:21.307 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:21.301+0000 7f426eb14700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6162 (117'3107,124'6162] local-lis/les=21/22 n=367 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=124'6160 crt=124'6160 lcod 124'6158 mlcod 124'6158 active+clean+snaptrim trimq=[209~1f] ps=[204~5]] removing snap head 2019-12-05T06:51:21.308 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:21.302+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 124 pg[3.7s0( v 124'6255 (118'3211,124'6255] local-lis/les=21/22 n=140 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=124'6253 crt=124'6253 lcod 124'6251 mlcod 124'6251 active+clean+snaptrim trimq=[210~18] ps=[204~c]] removing snap head 2019-12-05T06:51:21.311 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:21.306+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 124 pg[3.1s0( v 124'6525 (118'3516,124'6525] local-lis/les=21/22 n=153 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=124'6525 lcod 124'6523 mlcod 124'6523 active+clean+snaptrim trimq=[20f~19] ps=[204~b]] removing snap head 2019-12-05T06:51:21.315 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:21.310+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 124 pg[3.7s0( v 124'6257 (118'3211,124'6257] local-lis/les=21/22 n=138 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=124'6257 lcod 124'6255 mlcod 124'6255 active+clean+snaptrim trimq=[210~18] ps=[204~c]] removing snap head 2019-12-05T06:51:21.316 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:21.309+0000 7f4272b1c700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6164 (117'3107,124'6164] local-lis/les=21/22 n=365 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=124'6164 lcod 124'6162 mlcod 124'6162 active+clean+snaptrim trimq=[20a~1e] ps=[204~6]] removing snap head 2019-12-05T06:51:21.317 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:21.314+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 124 pg[3.7s0( v 124'6259 (118'3211,124'6259] local-lis/les=21/22 n=136 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=124'6257 crt=124'6257 lcod 124'6255 mlcod 124'6255 active+clean+snaptrim trimq=[210~18] ps=[204~c]] removing snap head 2019-12-05T06:51:21.318 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:21.313+0000 7f4272b1c700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6166 (117'3107,124'6166] local-lis/les=21/22 n=363 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=124'6164 crt=124'6164 lcod 124'6162 mlcod 124'6162 active+clean+snaptrim trimq=[20a~1e] ps=[204~6]] removing snap head 2019-12-05T06:51:21.319 INFO:tasks.ceph.osd.5.smithi040.stderr:2019-12-05T06:51:21.309+0000 7fc712838700 -1 osd.5 pg_epoch: 124 pg[3.5s0( v 124'6212 (117'3209,124'6212] local-lis/les=21/22 n=57 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [5,3,2,6]p5(0) r=0 lpr=21 luod=124'6211 crt=124'6209 lcod 124'6209 mlcod 124'6209 active+clean+snaptrim trimq=[211~17] ps=[204~d]] removing snap head 2019-12-05T06:51:21.320 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:21.314+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 124 pg[3.1s0( v 124'6527 (118'3516,124'6527] local-lis/les=21/22 n=151 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=124'6527 lcod 124'6525 mlcod 124'6525 active+clean+snaptrim trimq=[210~18] ps=[204~c]] removing snap head 2019-12-05T06:51:21.323 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:21.318+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 124 pg[3.1s0( v 124'6529 (118'3516,124'6529] local-lis/les=21/22 n=149 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=124'6527 crt=124'6527 lcod 124'6525 mlcod 124'6525 active+clean+snaptrim trimq=[210~18] ps=[204~c]] removing snap head 2019-12-05T06:51:21.324 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:21.322+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 124 pg[3.7s0( v 124'6262 (118'3211,124'6262] local-lis/les=21/22 n=133 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=124'6261 crt=124'6257 lcod 124'6259 mlcod 124'6259 active+clean+snaptrim trimq=[210~18] ps=[204~c]] removing snap head 2019-12-05T06:51:21.326 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:21.322+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 124 pg[3.7s0( v 124'6264 (118'3211,124'6264] local-lis/les=21/22 n=131 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=124'6261 crt=124'6261 lcod 124'6259 mlcod 124'6259 active+clean+snaptrim trimq=[210~18] ps=[204~c]] removing snap head 2019-12-05T06:51:21.329 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:21.326+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 124 pg[3.1s0( v 124'6531 (118'3516,124'6531] local-lis/les=21/22 n=147 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=124'6531 lcod 124'6529 mlcod 124'6529 active+clean+snaptrim trimq=[210~18] ps=[204~c]] removing snap head 2019-12-05T06:51:21.329 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:21.321+0000 7f426eb14700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6168 (117'3107,124'6168] local-lis/les=21/22 n=361 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=124'6168 lcod 124'6166 mlcod 124'6166 active+clean+snaptrim trimq=[20a~1e] ps=[204~6]] removing snap head 2019-12-05T06:51:21.330 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:21.321+0000 7f426eb14700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6170 (117'3107,124'6170] local-lis/les=21/22 n=359 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=124'6168 crt=124'6168 lcod 124'6166 mlcod 124'6166 active+clean+snaptrim trimq=[20a~1e] ps=[204~6]] removing snap head 2019-12-05T06:51:21.330 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:21.326+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 124 pg[3.1s0( v 124'6533 (118'3516,124'6533] local-lis/les=21/22 n=145 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=124'6531 crt=124'6531 lcod 124'6529 mlcod 124'6529 active+clean+snaptrim trimq=[210~18] ps=[204~c]] removing snap head 2019-12-05T06:51:21.334 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:21.330+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 124 pg[3.7s0( v 124'6266 (118'3211,124'6266] local-lis/les=21/22 n=129 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=124'6266 lcod 124'6264 mlcod 124'6264 active+clean+snaptrim trimq=[210~18] ps=[204~c]] removing snap head 2019-12-05T06:51:21.335 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:21.330+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 124 pg[3.7s0( v 124'6268 (118'3211,124'6268] local-lis/les=21/22 n=127 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=124'6266 crt=124'6266 lcod 124'6264 mlcod 124'6264 active+clean+snaptrim trimq=[210~18] ps=[204~c]] removing snap head 2019-12-05T06:51:21.339 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:21.334+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 124 pg[3.1s0( v 124'6536 (118'3516,124'6536] local-lis/les=21/22 n=142 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=124'6535 crt=124'6535 lcod 124'6533 mlcod 124'6533 active+clean+snaptrim trimq=[210~18] ps=[204~c]] removing snap head 2019-12-05T06:51:21.340 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:21.333+0000 7f4272b1c700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6172 (117'3107,124'6172] local-lis/les=21/22 n=357 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=124'6172 lcod 124'6170 mlcod 124'6170 active+clean+snaptrim trimq=[20a~1e] ps=[204~6]] removing snap head 2019-12-05T06:51:21.342 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:21.338+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 124 pg[3.1s0( v 124'6538 (118'3516,124'6538] local-lis/les=21/22 n=140 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=124'6535 crt=124'6535 lcod 124'6533 mlcod 124'6533 active+clean+snaptrim trimq=[210~18] ps=[204~c]] removing snap head 2019-12-05T06:51:21.343 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:21.337+0000 7f4272b1c700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6174 (117'3107,124'6174] local-lis/les=21/22 n=355 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=124'6172 crt=124'6172 lcod 124'6170 mlcod 124'6170 active+clean+snaptrim trimq=[20a~1e] ps=[204~6]] removing snap head 2019-12-05T06:51:21.345 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:21.342+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 124 pg[3.7s0( v 124'6270 (118'3211,124'6270] local-lis/les=21/22 n=125 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=124'6270 lcod 124'6268 mlcod 124'6268 active+clean+snaptrim trimq=[210~18] ps=[204~c]] removing snap head 2019-12-05T06:51:21.348 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:21.342+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 124 pg[3.7s0( v 124'6272 (118'3211,124'6272] local-lis/les=21/22 n=123 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=124'6270 crt=124'6270 lcod 124'6268 mlcod 124'6268 active+clean+snaptrim trimq=[210~18] ps=[204~c]] removing snap head 2019-12-05T06:51:21.349 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:21.341+0000 7f4272b1c700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6176 (117'3107,124'6176] local-lis/les=21/22 n=353 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=124'6176 lcod 124'6174 mlcod 124'6174 active+clean+snaptrim trimq=[20a~1e] ps=[204~6]] removing snap head 2019-12-05T06:51:21.350 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:21.346+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 124 pg[3.1s0( v 124'6540 (118'3516,124'6540] local-lis/les=21/22 n=138 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=124'6540 lcod 124'6538 mlcod 124'6538 active+clean+snaptrim trimq=[210~18] ps=[204~c]] removing snap head 2019-12-05T06:51:21.354 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:21.345+0000 7f4272b1c700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6178 (117'3107,124'6178] local-lis/les=21/22 n=351 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=124'6176 crt=124'6176 lcod 124'6174 mlcod 124'6174 active+clean+snaptrim trimq=[20a~1e] ps=[204~6]] removing snap head 2019-12-05T06:51:21.354 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:21.350+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 124 pg[3.1s0( v 124'6542 (118'3516,124'6542] local-lis/les=21/22 n=136 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=124'6540 crt=124'6540 lcod 124'6538 mlcod 124'6538 active+clean+snaptrim trimq=[210~18] ps=[204~c]] removing snap head 2019-12-05T06:51:21.355 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:21.350+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 124 pg[3.7s0( v 124'6275 (118'3211,124'6275] local-lis/les=21/22 n=120 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=124'6274 crt=124'6270 lcod 124'6272 mlcod 124'6272 active+clean+snaptrim trimq=[210~18] ps=[204~c]] removing snap head 2019-12-05T06:51:21.357 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:21.349+0000 7f4272b1c700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6180 (117'3107,124'6180] local-lis/les=21/22 n=349 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=124'6180 lcod 124'6178 mlcod 124'6178 active+clean+snaptrim trimq=[20a~1e] ps=[204~6]] removing snap head 2019-12-05T06:51:21.358 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:21.354+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 124 pg[3.7s0( v 124'6277 (118'3211,124'6277] local-lis/les=21/22 n=118 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=124'6274 crt=124'6274 lcod 124'6272 mlcod 124'6272 active+clean+snaptrim trimq=[210~18] ps=[204~c]] removing snap head 2019-12-05T06:51:21.359 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:21.353+0000 7f4272b1c700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6182 (117'3107,124'6182] local-lis/les=21/22 n=347 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=124'6180 crt=124'6180 lcod 124'6178 mlcod 124'6178 active+clean+snaptrim trimq=[20a~1e] ps=[204~6]] removing snap head 2019-12-05T06:51:21.364 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:21.362+0000 7fc9c444d700 -1 osd.3 pg_epoch: 124 pg[3.2s0( v 124'6330 (117'3307,124'6330] local-lis/les=21/22 n=59 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=124'6328 crt=124'6326 lcod 124'6326 mlcod 124'6326 active+clean+snaptrim trimq=[211~17] ps=[204~d]] removing snap head 2019-12-05T06:51:21.365 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:21.362+0000 7fc9c444d700 -1 osd.3 pg_epoch: 124 pg[3.2s0( v 124'6332 (117'3307,124'6332] local-lis/les=21/22 n=57 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=124'6328 crt=124'6328 lcod 124'6326 mlcod 124'6326 active+clean+snaptrim trimq=[211~17] ps=[204~d]] removing snap head 2019-12-05T06:51:21.369 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:21.366+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 124 pg[3.7s0( v 124'6279 (118'3211,124'6279] local-lis/les=21/22 n=116 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=124'6279 lcod 124'6277 mlcod 124'6277 active+clean+snaptrim trimq=[210~18] ps=[204~c]] removing snap head 2019-12-05T06:51:21.371 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:21.366+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 124 pg[3.7s0( v 124'6281 (118'3211,124'6281] local-lis/les=21/22 n=114 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=124'6279 crt=124'6279 lcod 124'6277 mlcod 124'6277 active+clean+snaptrim trimq=[210~18] ps=[204~c]] removing snap head 2019-12-05T06:51:21.376 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:21.374+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 124 pg[3.7s0( v 124'6284 (118'3211,124'6284] local-lis/les=21/22 n=111 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=124'6283 crt=124'6279 lcod 124'6281 mlcod 124'6281 active+clean+snaptrim trimq=[210~18] ps=[204~c]] removing snap head 2019-12-05T06:51:21.378 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:21.374+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 124 pg[3.7s0( v 124'6286 (118'3211,124'6286] local-lis/les=21/22 n=109 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=124'6283 crt=124'6283 lcod 124'6281 mlcod 124'6281 active+clean+snaptrim trimq=[210~18] ps=[204~c]] removing snap head 2019-12-05T06:51:21.384 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:21.382+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 124 pg[3.1s0( v 124'6545 (118'3516,124'6545] local-lis/les=21/22 n=133 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=124'6544 crt=124'6542 lcod 124'6542 mlcod 124'6542 active+clean+snaptrim trimq=[210~18] ps=[204~c]] removing snap head 2019-12-05T06:51:21.384 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:21.382+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 124 pg[3.7s0( v 124'6289 (118'3211,124'6289] local-lis/les=21/22 n=106 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=124'6288 crt=124'6286 lcod 124'6286 mlcod 124'6286 active+clean+snaptrim trimq=[210~18] ps=[204~c]] removing snap head 2019-12-05T06:51:21.387 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:21.382+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 124 pg[3.1s0( v 124'6547 (118'3516,124'6547] local-lis/les=21/22 n=131 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=124'6544 crt=124'6544 lcod 124'6542 mlcod 124'6542 active+clean+snaptrim trimq=[210~18] ps=[204~c]] removing snap head 2019-12-05T06:51:21.392 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:21.390+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 124 pg[3.7s0( v 124'6291 (118'3211,124'6291] local-lis/les=21/22 n=104 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=124'6291 lcod 124'6289 mlcod 124'6289 active+clean+snaptrim trimq=[211~17] ps=[204~d]] removing snap head 2019-12-05T06:51:21.395 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:21.390+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 124 pg[3.7s0( v 124'6293 (118'3211,124'6293] local-lis/les=21/22 n=102 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=124'6291 crt=124'6291 lcod 124'6289 mlcod 124'6289 active+clean+snaptrim trimq=[211~17] ps=[204~d]] removing snap head 2019-12-05T06:51:21.402 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:21.398+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 124 pg[3.7s0( v 124'6295 (118'3211,124'6295] local-lis/les=21/22 n=100 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=124'6295 lcod 124'6293 mlcod 124'6293 active+clean+snaptrim trimq=[211~17] ps=[204~d]] removing snap head 2019-12-05T06:51:21.404 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:21.402+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 124 pg[3.7s0( v 124'6297 (118'3211,124'6297] local-lis/les=21/22 n=98 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=124'6295 crt=124'6295 lcod 124'6293 mlcod 124'6293 active+clean+snaptrim trimq=[211~17] ps=[204~d]] removing snap head 2019-12-05T06:51:21.405 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:21.393+0000 7f4272b1c700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6185 (117'3107,124'6185] local-lis/les=21/22 n=344 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=124'6184 crt=124'6182 lcod 124'6182 mlcod 124'6182 active+clean+snaptrim trimq=[20a~1e] ps=[204~6]] removing snap head 2019-12-05T06:51:21.406 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:21.397+0000 7f4272b1c700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6187 (117'3107,124'6187] local-lis/les=21/22 n=342 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=124'6184 crt=124'6184 lcod 124'6182 mlcod 124'6182 active+clean+snaptrim trimq=[20a~1e] ps=[204~6]] removing snap head 2019-12-05T06:51:21.409 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:21.406+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 124 pg[3.7s0( v 124'6300 (118'3211,124'6300] local-lis/les=21/22 n=95 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=124'6299 crt=124'6295 lcod 124'6297 mlcod 124'6297 active+clean+snaptrim trimq=[211~17] ps=[204~d]] removing snap head 2019-12-05T06:51:21.412 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:21.406+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 124 pg[3.7s0( v 124'6302 (118'3211,124'6302] local-lis/les=21/22 n=93 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=124'6299 crt=124'6299 lcod 124'6297 mlcod 124'6297 active+clean+snaptrim trimq=[211~17] ps=[204~d]] removing snap head 2019-12-05T06:51:21.416 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:21.414+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 124 pg[3.7s0( v 124'6305 (118'3211,124'6305] local-lis/les=21/22 n=90 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=124'6304 crt=124'6300 lcod 124'6302 mlcod 124'6302 active+clean+snaptrim trimq=[211~17] ps=[204~d]] removing snap head 2019-12-05T06:51:21.417 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:21.414+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 124 pg[3.7s0( v 124'6307 (118'3211,124'6307] local-lis/les=21/22 n=88 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=124'6304 crt=124'6304 lcod 124'6302 mlcod 124'6302 active+clean+snaptrim trimq=[211~17] ps=[204~d]] removing snap head 2019-12-05T06:51:21.422 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:21.418+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 124 pg[3.7s0( v 124'6309 (118'3211,124'6309] local-lis/les=21/22 n=86 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=124'6309 lcod 124'6307 mlcod 124'6307 active+clean+snaptrim trimq=[211~17] ps=[204~d]] removing snap head 2019-12-05T06:51:21.423 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:21.418+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 124 pg[3.7s0( v 124'6311 (118'3211,124'6311] local-lis/les=21/22 n=84 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=124'6309 crt=124'6309 lcod 124'6307 mlcod 124'6307 active+clean+snaptrim trimq=[211~17] ps=[204~d]] removing snap head 2019-12-05T06:51:21.428 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:21.426+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 124 pg[3.1s0( v 124'6551 (118'3516,124'6551] local-lis/les=21/22 n=127 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=124'6549 crt=124'6547 lcod 124'6547 mlcod 124'6547 active+clean+snaptrim trimq=[210~18] ps=[204~c]] removing snap head 2019-12-05T06:51:21.430 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:21.426+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 124 pg[3.1s0( v 124'6553 (118'3516,124'6553] local-lis/les=21/22 n=125 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=124'6549 crt=124'6549 lcod 124'6547 mlcod 124'6547 active+clean+snaptrim trimq=[210~18] ps=[204~c]] removing snap head 2019-12-05T06:51:21.432 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:21.426+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 124 pg[3.7s0( v 124'6313 (118'3311,124'6313] local-lis/les=21/22 n=82 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=124'6313 lcod 124'6311 mlcod 124'6311 active+clean+snaptrim trimq=[211~17] ps=[204~d]] removing snap head 2019-12-05T06:51:21.434 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:21.430+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 124 pg[3.7s0( v 124'6315 (118'3311,124'6315] local-lis/les=21/22 n=80 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=124'6313 crt=124'6313 lcod 124'6311 mlcod 124'6311 active+clean+snaptrim trimq=[211~17] ps=[204~d]] removing snap head 2019-12-05T06:51:21.443 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:21.434+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 124 pg[3.1s0( v 124'6555 (118'3516,124'6555] local-lis/les=21/22 n=123 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=124'6555 lcod 124'6553 mlcod 124'6553 active+clean+snaptrim trimq=[210~18] ps=[204~c]] removing snap head 2019-12-05T06:51:21.444 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:21.438+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 124 pg[3.1s0( v 124'6557 (118'3516,124'6557] local-lis/les=21/22 n=121 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=124'6555 crt=124'6555 lcod 124'6553 mlcod 124'6553 active+clean+snaptrim trimq=[210~18] ps=[204~c]] removing snap head 2019-12-05T06:51:21.444 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:21.442+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 124 pg[3.7s0( v 124'6317 (118'3311,124'6317] local-lis/les=21/22 n=78 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=124'6317 lcod 124'6315 mlcod 124'6315 active+clean+snaptrim trimq=[211~17] ps=[204~d]] removing snap head 2019-12-05T06:51:21.448 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:21.446+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 124 pg[3.7s0( v 124'6319 (118'3311,124'6319] local-lis/les=21/22 n=76 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=124'6317 crt=124'6317 lcod 124'6315 mlcod 124'6315 active+clean+snaptrim trimq=[211~17] ps=[204~d]] removing snap head 2019-12-05T06:51:21.453 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:21.450+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 124 pg[3.1s0( v 124'6559 (118'3516,124'6559] local-lis/les=21/22 n=119 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=124'6559 lcod 124'6557 mlcod 124'6557 active+clean+snaptrim trimq=[210~18] ps=[204~c]] removing snap head 2019-12-05T06:51:21.456 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:21.450+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 124 pg[3.1s0( v 124'6561 (118'3516,124'6561] local-lis/les=21/22 n=117 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=124'6559 crt=124'6559 lcod 124'6557 mlcod 124'6557 active+clean+snaptrim trimq=[210~18] ps=[204~c]] removing snap head 2019-12-05T06:51:21.458 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:21.454+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 124 pg[3.7s0( v 124'6321 (118'3311,124'6321] local-lis/les=21/22 n=74 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=124'6321 lcod 124'6319 mlcod 124'6319 active+clean+snaptrim trimq=[211~17] ps=[204~d]] removing snap head 2019-12-05T06:51:21.461 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:21.458+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 124 pg[3.7s0( v 124'6323 (118'3311,124'6323] local-lis/les=21/22 n=72 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=124'6321 crt=124'6321 lcod 124'6319 mlcod 124'6319 active+clean+snaptrim trimq=[211~17] ps=[204~d]] removing snap head 2019-12-05T06:51:21.462 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:21.458+0000 7fc9c0445700 -1 osd.3 pg_epoch: 124 pg[3.2s0( v 124'6336 (117'3307,124'6336] local-lis/les=21/22 n=53 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=124'6334 crt=124'6332 lcod 124'6332 mlcod 124'6332 active+clean+snaptrim trimq=[211~17] ps=[204~d]] removing snap head 2019-12-05T06:51:21.464 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:21.462+0000 7fc9c0445700 -1 osd.3 pg_epoch: 124 pg[3.2s0( v 124'6338 (117'3307,124'6338] local-lis/les=21/22 n=51 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=124'6334 crt=124'6334 lcod 124'6332 mlcod 124'6332 active+clean+snaptrim trimq=[211~17] ps=[204~d]] removing snap head 2019-12-05T06:51:21.465 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:21.462+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 124 pg[3.1s0( v 124'6563 (118'3516,124'6563] local-lis/les=21/22 n=115 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=124'6563 lcod 124'6561 mlcod 124'6561 active+clean+snaptrim trimq=[210~18] ps=[204~c]] removing snap head 2019-12-05T06:51:21.467 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:21.462+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 124 pg[3.1s0( v 124'6565 (118'3516,124'6565] local-lis/les=21/22 n=113 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=124'6563 crt=124'6563 lcod 124'6561 mlcod 124'6561 active+clean+snaptrim trimq=[210~18] ps=[204~c]] removing snap head 2019-12-05T06:51:21.475 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:21.470+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 124 pg[3.7s0( v 124'6325 (118'3311,124'6325] local-lis/les=21/22 n=70 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=124'6325 lcod 124'6323 mlcod 124'6323 active+clean+snaptrim trimq=[211~17] ps=[204~d]] removing snap head 2019-12-05T06:51:21.475 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:21.470+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 124 pg[3.1s0( v 124'6567 (118'3516,124'6567] local-lis/les=21/22 n=111 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=124'6567 lcod 124'6565 mlcod 124'6565 active+clean+snaptrim trimq=[210~18] ps=[204~c]] removing snap head 2019-12-05T06:51:21.479 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:21.474+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 124 pg[3.7s0( v 124'6327 (118'3311,124'6327] local-lis/les=21/22 n=68 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=124'6325 crt=124'6325 lcod 124'6323 mlcod 124'6323 active+clean+snaptrim trimq=[211~17] ps=[204~d]] removing snap head 2019-12-05T06:51:21.488 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:21.482+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 124 pg[3.1s0( v 124'6571 (118'3516,124'6571] local-lis/les=21/22 n=108 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=124'6571 lcod 124'6569 mlcod 124'6569 active+clean+snaptrim trimq=[210~18] ps=[204~c]] removing snap head 2019-12-05T06:51:21.489 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:21.486+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 124 pg[3.1s0( v 124'6573 (118'3516,124'6573] local-lis/les=21/22 n=106 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=124'6571 crt=124'6571 lcod 124'6569 mlcod 124'6569 active+clean+snaptrim trimq=[210~18] ps=[204~c]] removing snap head 2019-12-05T06:51:21.491 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:21.486+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 124 pg[3.7s0( v 124'6330 (118'3311,124'6330] local-lis/les=21/22 n=65 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=124'6330 lcod 124'6329 mlcod 124'6329 active+clean+snaptrim trimq=[211~17] ps=[204~d]] removing snap head 2019-12-05T06:51:21.492 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:21.486+0000 7fc9c444d700 -1 osd.3 pg_epoch: 124 pg[3.2s0( v 124'6341 (117'3307,124'6341] local-lis/les=21/22 n=48 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=124'6340 crt=124'6338 lcod 124'6338 mlcod 124'6338 active+clean+snaptrim trimq=[211~17] ps=[204~d]] removing snap head 2019-12-05T06:51:21.494 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:21.490+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 124 pg[3.7s0( v 124'6332 (118'3311,124'6332] local-lis/les=21/22 n=63 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=124'6330 crt=124'6330 lcod 124'6329 mlcod 124'6329 active+clean+snaptrim trimq=[211~17] ps=[204~d]] removing snap head 2019-12-05T06:51:21.494 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:21.490+0000 7fc9c444d700 -1 osd.3 pg_epoch: 124 pg[3.2s0( v 124'6343 (117'3307,124'6343] local-lis/les=21/22 n=46 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 luod=124'6340 crt=124'6340 lcod 124'6338 mlcod 124'6338 active+clean+snaptrim trimq=[211~17] ps=[204~d]] removing snap head 2019-12-05T06:51:21.496 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:21.494+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 124 pg[3.1s0( v 124'6575 (118'3516,124'6575] local-lis/les=21/22 n=104 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=124'6575 lcod 124'6573 mlcod 124'6573 active+clean+snaptrim trimq=[210~18] ps=[204~c]] removing snap head 2019-12-05T06:51:21.498 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:21.494+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 124 pg[3.1s0( v 124'6577 (118'3516,124'6577] local-lis/les=21/22 n=102 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=124'6575 crt=124'6575 lcod 124'6573 mlcod 124'6573 active+clean+snaptrim trimq=[210~18] ps=[204~c]] removing snap head 2019-12-05T06:51:21.498 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:21.489+0000 7f426eb14700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6189 (117'3107,124'6189] local-lis/les=21/22 n=340 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=124'6189 lcod 124'6187 mlcod 124'6187 active+clean+snaptrim trimq=[20a~1e] ps=[204~6]] removing snap head 2019-12-05T06:51:21.506 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:21.502+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 124 pg[3.7s0( v 124'6334 (118'3311,124'6334] local-lis/les=21/22 n=61 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=124'6334 lcod 124'6332 mlcod 124'6332 active+clean+snaptrim trimq=[211~17] ps=[204~d]] removing snap head 2019-12-05T06:51:21.509 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:21.502+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 124 pg[3.1s0( v 124'6580 (118'3516,124'6580] local-lis/les=21/22 n=99 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=124'6580 lcod 124'6579 mlcod 124'6579 active+clean+snaptrim trimq=[210~18] ps=[204~c]] removing snap head 2019-12-05T06:51:21.510 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:21.502+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 124 pg[3.7s0( v 124'6336 (118'3311,124'6336] local-lis/les=21/22 n=59 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=124'6334 crt=124'6334 lcod 124'6332 mlcod 124'6332 active+clean+snaptrim trimq=[211~17] ps=[204~d]] removing snap head 2019-12-05T06:51:21.512 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:21.501+0000 7f4272b1c700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6193 (117'3107,124'6193] local-lis/les=21/22 n=337 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=124'6193 lcod 124'6191 mlcod 124'6191 active+clean+snaptrim trimq=[20a~1e] ps=[204~6]] removing snap head 2019-12-05T06:51:21.512 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:21.501+0000 7f4272b1c700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6195 (117'3107,124'6195] local-lis/les=21/22 n=335 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=124'6193 crt=124'6193 lcod 124'6191 mlcod 124'6191 active+clean+snaptrim trimq=[20a~1e] ps=[204~6]] removing snap head 2019-12-05T06:51:21.512 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:21.506+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 124 pg[3.1s0( v 124'6582 (118'3516,124'6582] local-lis/les=21/22 n=97 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=124'6580 crt=124'6580 lcod 124'6579 mlcod 124'6579 active+clean+snaptrim trimq=[210~18] ps=[204~c]] removing snap head 2019-12-05T06:51:21.515 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:21.510+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 124 pg[3.7s0( v 124'6339 (118'3311,124'6339] local-lis/les=21/22 n=56 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=124'6338 crt=124'6334 lcod 124'6336 mlcod 124'6336 active+clean+snaptrim trimq=[211~17] ps=[204~d]] removing snap head 2019-12-05T06:51:21.517 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:21.514+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 124 pg[3.1s0( v 124'6584 (118'3516,124'6584] local-lis/les=21/22 n=95 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=124'6584 lcod 124'6582 mlcod 124'6582 active+clean+snaptrim trimq=[211~17] ps=[204~d]] removing snap head 2019-12-05T06:51:21.518 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:21.509+0000 7f426eb14700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6197 (117'3107,124'6197] local-lis/les=21/22 n=333 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=124'6197 lcod 124'6195 mlcod 124'6195 active+clean+snaptrim trimq=[20b~1d] ps=[204~7]] removing snap head 2019-12-05T06:51:21.519 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:21.514+0000 7fc9c1c48700 -1 osd.3 pg_epoch: 124 pg[3.7s0( v 124'6341 (118'3311,124'6341] local-lis/les=21/22 n=54 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=124'6338 crt=124'6338 lcod 124'6336 mlcod 124'6336 active+clean+snaptrim trimq=[211~17] ps=[204~d]] removing snap head 2019-12-05T06:51:21.519 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:21.514+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 124 pg[3.1s0( v 124'6586 (118'3516,124'6586] local-lis/les=21/22 n=93 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=124'6584 crt=124'6584 lcod 124'6582 mlcod 124'6582 active+clean+snaptrim trimq=[211~17] ps=[204~d]] removing snap head 2019-12-05T06:51:21.522 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:21.513+0000 7f426eb14700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6199 (117'3107,124'6199] local-lis/les=21/22 n=331 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=124'6197 crt=124'6197 lcod 124'6195 mlcod 124'6195 active+clean+snaptrim trimq=[20b~1d] ps=[204~7]] removing snap head 2019-12-05T06:51:21.526 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:21.517+0000 7f4272b1c700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6201 (117'3107,124'6201] local-lis/les=21/22 n=329 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=124'6201 lcod 124'6199 mlcod 124'6199 active+clean+snaptrim trimq=[20b~1d] ps=[204~7]] removing snap head 2019-12-05T06:51:21.528 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:21.521+0000 7f4272b1c700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6203 (117'3107,124'6203] local-lis/les=21/22 n=327 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=124'6201 crt=124'6201 lcod 124'6199 mlcod 124'6199 active+clean+snaptrim trimq=[20b~1d] ps=[204~7]] removing snap head 2019-12-05T06:51:21.529 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:21.526+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 124 pg[3.7s0( v 124'6343 (118'3311,124'6343] local-lis/les=21/22 n=52 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=124'6343 lcod 124'6341 mlcod 124'6341 active+clean+snaptrim trimq=[211~17] ps=[204~d]] removing snap head 2019-12-05T06:51:21.531 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:21.526+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 124 pg[3.7s0( v 124'6345 (118'3311,124'6345] local-lis/les=21/22 n=50 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=124'6343 crt=124'6343 lcod 124'6341 mlcod 124'6341 active+clean+snaptrim trimq=[211~17] ps=[204~d]] removing snap head 2019-12-05T06:51:21.538 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:21.534+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 124 pg[3.7s0( v 124'6347 (118'3311,124'6347] local-lis/les=21/22 n=48 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=124'6347 lcod 124'6345 mlcod 124'6345 active+clean+snaptrim trimq=[211~17] ps=[204~d]] removing snap head 2019-12-05T06:51:21.539 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:21.534+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 124 pg[3.7s0( v 124'6349 (118'3311,124'6349] local-lis/les=21/22 n=46 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 luod=124'6347 crt=124'6347 lcod 124'6345 mlcod 124'6345 active+clean+snaptrim trimq=[211~17] ps=[204~d]] removing snap head 2019-12-05T06:51:21.547 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:21.542+0000 7fc9bdc40700 -1 osd.3 pg_epoch: 124 pg[3.7s0( v 124'6351 (118'3311,124'6351] local-lis/les=21/22 n=44 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [3,5,7,0]p3(0) r=0 lpr=21 crt=124'6351 lcod 124'6349 mlcod 124'6349 active+clean+snaptrim trimq=[211~17] ps=[204~d]] removing snap head 2019-12-05T06:51:21.859 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:21.849+0000 7f426eb14700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6205 (117'3107,124'6205] local-lis/les=21/22 n=325 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=124'6205 lcod 124'6203 mlcod 124'6203 active+clean+snaptrim trimq=[20b~1d] ps=[204~7]] removing snap head 2019-12-05T06:51:21.859 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:21.853+0000 7f426eb14700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6207 (117'3107,124'6207] local-lis/les=21/22 n=323 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=124'6205 crt=124'6205 lcod 124'6203 mlcod 124'6203 active+clean+snaptrim trimq=[20b~1d] ps=[204~7]] removing snap head 2019-12-05T06:51:21.867 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:21.862+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 124 pg[3.1s0( v 124'6589 (118'3516,124'6589] local-lis/les=21/22 n=90 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=124'6588 crt=124'6586 lcod 124'6586 mlcod 124'6586 active+clean+snaptrim trimq=[211~17] ps=[204~d]] removing snap head 2019-12-05T06:51:21.868 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:21.866+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 124 pg[3.1s0( v 124'6591 (118'3516,124'6591] local-lis/les=21/22 n=88 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=124'6588 crt=124'6588 lcod 124'6586 mlcod 124'6586 active+clean+snaptrim trimq=[211~17] ps=[204~d]] removing snap head 2019-12-05T06:51:21.870 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:21.861+0000 7f4272b1c700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6209 (117'3207,124'6209] local-lis/les=21/22 n=321 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=124'6209 lcod 124'6207 mlcod 124'6207 active+clean+snaptrim trimq=[20b~1d] ps=[204~7]] removing snap head 2019-12-05T06:51:21.874 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:21.865+0000 7f4272b1c700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6211 (117'3207,124'6211] local-lis/les=21/22 n=319 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=124'6209 crt=124'6209 lcod 124'6207 mlcod 124'6207 active+clean+snaptrim trimq=[20b~1d] ps=[204~7]] removing snap head 2019-12-05T06:51:22.072 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:22.070+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 124 pg[3.1s0( v 124'6593 (118'3516,124'6593] local-lis/les=21/22 n=86 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=124'6593 lcod 124'6591 mlcod 124'6591 active+clean+snaptrim trimq=[211~17] ps=[204~d]] removing snap head 2019-12-05T06:51:22.073 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:22.065+0000 7f4272b1c700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6213 (117'3207,124'6213] local-lis/les=21/22 n=317 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=124'6213 lcod 124'6211 mlcod 124'6211 active+clean+snaptrim trimq=[20b~1d] ps=[204~7]] removing snap head 2019-12-05T06:51:22.076 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:22.074+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 124 pg[3.1s0( v 124'6595 (118'3516,124'6595] local-lis/les=21/22 n=84 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=124'6593 crt=124'6593 lcod 124'6591 mlcod 124'6591 active+clean+snaptrim trimq=[211~17] ps=[204~d]] removing snap head 2019-12-05T06:51:22.078 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:22.069+0000 7f4272b1c700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6215 (117'3207,124'6215] local-lis/les=21/22 n=315 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=124'6213 crt=124'6213 lcod 124'6211 mlcod 124'6211 active+clean+snaptrim trimq=[20b~1d] ps=[204~7]] removing snap head 2019-12-05T06:51:22.084 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:22.078+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 124 pg[3.1s0( v 124'6597 (118'3516,124'6597] local-lis/les=21/22 n=82 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=124'6597 lcod 124'6595 mlcod 124'6595 active+clean+snaptrim trimq=[211~17] ps=[204~d]] removing snap head 2019-12-05T06:51:22.084 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:22.077+0000 7f4272b1c700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6217 (117'3207,124'6217] local-lis/les=21/22 n=313 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=124'6217 lcod 124'6215 mlcod 124'6215 active+clean+snaptrim trimq=[20b~1d] ps=[204~7]] removing snap head 2019-12-05T06:51:22.086 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:22.082+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 124 pg[3.1s0( v 124'6599 (118'3516,124'6599] local-lis/les=21/22 n=80 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=124'6597 crt=124'6597 lcod 124'6595 mlcod 124'6595 active+clean+snaptrim trimq=[211~17] ps=[204~d]] removing snap head 2019-12-05T06:51:22.088 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:22.081+0000 7f4272b1c700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6219 (117'3207,124'6219] local-lis/les=21/22 n=311 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=124'6217 crt=124'6217 lcod 124'6215 mlcod 124'6215 active+clean+snaptrim trimq=[20b~1d] ps=[204~7]] removing snap head 2019-12-05T06:51:22.094 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:22.089+0000 7f426eb14700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6221 (117'3207,124'6221] local-lis/les=21/22 n=309 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=124'6221 lcod 124'6219 mlcod 124'6219 active+clean+snaptrim trimq=[20b~1d] ps=[204~7]] removing snap head 2019-12-05T06:51:22.095 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:22.090+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 124 pg[3.1s0( v 124'6601 (118'3516,124'6601] local-lis/les=21/22 n=78 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=124'6601 lcod 124'6599 mlcod 124'6599 active+clean+snaptrim trimq=[211~17] ps=[204~d]] removing snap head 2019-12-05T06:51:22.097 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:22.089+0000 7f426eb14700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6223 (117'3207,124'6223] local-lis/les=21/22 n=307 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=124'6221 crt=124'6221 lcod 124'6219 mlcod 124'6219 active+clean+snaptrim trimq=[20b~1d] ps=[204~7]] removing snap head 2019-12-05T06:51:22.098 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:22.094+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 124 pg[3.1s0( v 124'6603 (118'3516,124'6603] local-lis/les=21/22 n=76 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=124'6601 crt=124'6601 lcod 124'6599 mlcod 124'6599 active+clean+snaptrim trimq=[211~17] ps=[204~d]] removing snap head 2019-12-05T06:51:22.106 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:22.102+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 124 pg[3.1s0( v 124'6605 (118'3516,124'6605] local-lis/les=21/22 n=74 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=124'6605 lcod 124'6603 mlcod 124'6603 active+clean+snaptrim trimq=[211~17] ps=[204~d]] removing snap head 2019-12-05T06:51:22.107 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:22.101+0000 7f4272b1c700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6225 (117'3207,124'6225] local-lis/les=21/22 n=305 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=124'6225 lcod 124'6223 mlcod 124'6223 active+clean+snaptrim trimq=[20b~1d] ps=[204~7]] removing snap head 2019-12-05T06:51:22.109 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:22.101+0000 7f4272b1c700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6227 (117'3207,124'6227] local-lis/les=21/22 n=303 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=124'6225 crt=124'6225 lcod 124'6223 mlcod 124'6223 active+clean+snaptrim trimq=[20b~1d] ps=[204~7]] removing snap head 2019-12-05T06:51:22.109 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:22.106+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 124 pg[3.1s0( v 124'6607 (118'3516,124'6607] local-lis/les=21/22 n=72 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=124'6605 crt=124'6605 lcod 124'6603 mlcod 124'6603 active+clean+snaptrim trimq=[211~17] ps=[204~d]] removing snap head 2019-12-05T06:51:22.116 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:22.109+0000 7f426eb14700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6229 (117'3207,124'6229] local-lis/les=21/22 n=301 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=124'6229 lcod 124'6227 mlcod 124'6227 active+clean+snaptrim trimq=[20b~1d] ps=[204~7]] removing snap head 2019-12-05T06:51:22.116 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:22.114+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 124 pg[3.1s0( v 124'6609 (118'3516,124'6609] local-lis/les=21/22 n=70 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=124'6609 lcod 124'6607 mlcod 124'6607 active+clean+snaptrim trimq=[211~17] ps=[204~d]] removing snap head 2019-12-05T06:51:22.119 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:22.109+0000 7f426eb14700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6231 (117'3207,124'6231] local-lis/les=21/22 n=299 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=124'6229 crt=124'6229 lcod 124'6227 mlcod 124'6227 active+clean+snaptrim trimq=[20b~1d] ps=[204~7]] removing snap head 2019-12-05T06:51:22.121 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:22.114+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 124 pg[3.1s0( v 124'6611 (118'3516,124'6611] local-lis/les=21/22 n=68 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=124'6609 crt=124'6609 lcod 124'6607 mlcod 124'6607 active+clean+snaptrim trimq=[211~17] ps=[204~d]] removing snap head 2019-12-05T06:51:22.126 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:22.121+0000 7f4272b1c700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6233 (117'3207,124'6233] local-lis/les=21/22 n=297 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=124'6233 lcod 124'6231 mlcod 124'6231 active+clean+snaptrim trimq=[20b~1d] ps=[204~7]] removing snap head 2019-12-05T06:51:22.127 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:22.122+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 124 pg[3.1s0( v 124'6613 (118'3516,124'6613] local-lis/les=21/22 n=66 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=124'6613 lcod 124'6611 mlcod 124'6611 active+clean+snaptrim trimq=[211~17] ps=[204~d]] removing snap head 2019-12-05T06:51:22.133 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:22.126+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 124 pg[3.1s0( v 124'6615 (118'3516,124'6615] local-lis/les=21/22 n=64 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=124'6613 crt=124'6613 lcod 124'6611 mlcod 124'6611 active+clean+snaptrim trimq=[211~17] ps=[204~d]] removing snap head 2019-12-05T06:51:22.136 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:22.129+0000 7f4272b1c700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6237 (117'3207,124'6237] local-lis/les=21/22 n=294 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=124'6237 lcod 124'6235 mlcod 124'6235 active+clean+snaptrim trimq=[20b~1d] ps=[204~7]] removing snap head 2019-12-05T06:51:22.137 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:22.134+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 124 pg[3.1s0( v 124'6617 (118'3516,124'6617] local-lis/les=21/22 n=62 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=124'6617 lcod 124'6615 mlcod 124'6615 active+clean+snaptrim trimq=[211~17] ps=[204~d]] removing snap head 2019-12-05T06:51:22.137 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:22.129+0000 7f4272b1c700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6239 (117'3207,124'6239] local-lis/les=21/22 n=292 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=124'6237 crt=124'6237 lcod 124'6235 mlcod 124'6235 active+clean+snaptrim trimq=[20b~1d] ps=[204~7]] removing snap head 2019-12-05T06:51:22.140 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:22.138+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 124 pg[3.1s0( v 124'6619 (118'3617,124'6619] local-lis/les=21/22 n=60 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=124'6617 crt=124'6617 lcod 124'6615 mlcod 124'6615 active+clean+snaptrim trimq=[211~17] ps=[204~d]] removing snap head 2019-12-05T06:51:22.149 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:22.141+0000 7f4272b1c700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6241 (117'3207,124'6241] local-lis/les=21/22 n=290 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=124'6241 lcod 124'6239 mlcod 124'6239 active+clean+snaptrim trimq=[20c~1c] ps=[204~8]] removing snap head 2019-12-05T06:51:22.150 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:22.146+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 124 pg[3.1s0( v 124'6621 (118'3617,124'6621] local-lis/les=21/22 n=58 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=124'6621 lcod 124'6619 mlcod 124'6619 active+clean+snaptrim trimq=[211~17] ps=[204~d]] removing snap head 2019-12-05T06:51:22.151 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:22.146+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 124 pg[3.1s0( v 124'6623 (118'3617,124'6623] local-lis/les=21/22 n=56 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=124'6621 crt=124'6621 lcod 124'6619 mlcod 124'6619 active+clean+snaptrim trimq=[211~17] ps=[204~d]] removing snap head 2019-12-05T06:51:22.151 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:22.145+0000 7f4272b1c700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6243 (117'3207,124'6243] local-lis/les=21/22 n=288 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=124'6241 crt=124'6241 lcod 124'6239 mlcod 124'6239 active+clean+snaptrim trimq=[20c~1c] ps=[204~8]] removing snap head 2019-12-05T06:51:22.162 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:22.158+0000 7fbf89ca3700 -1 osd.0 pg_epoch: 124 pg[3.1s0( v 124'6627 (118'3617,124'6627] local-lis/les=21/22 n=53 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=124'6625 crt=124'6625 lcod 124'6623 mlcod 124'6623 active+clean+snaptrim trimq=[211~17] ps=[204~d]] removing snap head 2019-12-05T06:51:22.417 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:22.414+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 124 pg[3.1s0( v 124'6629 (118'3617,124'6629] local-lis/les=21/22 n=51 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=124'6629 lcod 124'6627 mlcod 124'6627 active+clean+snaptrim trimq=[211~17] ps=[204~d]] removing snap head 2019-12-05T06:51:22.420 INFO:tasks.ceph.osd.0.smithi042.stderr:2019-12-05T06:51:22.418+0000 7fbf85c9b700 -1 osd.0 pg_epoch: 124 pg[3.1s0( v 124'6631 (118'3617,124'6631] local-lis/les=21/22 n=49 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 luod=124'6629 crt=124'6629 lcod 124'6627 mlcod 124'6627 active+clean+snaptrim trimq=[211~17] ps=[204~d]] removing snap head 2019-12-05T06:51:22.489 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:22.481+0000 7f4272b1c700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6246 (117'3207,124'6246] local-lis/les=21/22 n=285 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=124'6245 crt=124'6243 lcod 124'6243 mlcod 124'6243 active+clean+snaptrim trimq=[20c~1c] ps=[204~8]] removing snap head 2019-12-05T06:51:22.491 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:22.485+0000 7f4272b1c700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6248 (117'3207,124'6248] local-lis/les=21/22 n=283 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=124'6245 crt=124'6245 lcod 124'6243 mlcod 124'6243 active+clean+snaptrim trimq=[20c~1c] ps=[204~8]] removing snap head 2019-12-05T06:51:22.547 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:22.541+0000 7f426eb14700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6251 (117'3207,124'6251] local-lis/les=21/22 n=280 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=124'6250 crt=124'6248 lcod 124'6248 mlcod 124'6248 active+clean+snaptrim trimq=[20c~1c] ps=[204~8]] removing snap head 2019-12-05T06:51:22.550 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:22.545+0000 7f426eb14700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6253 (117'3207,124'6253] local-lis/les=21/22 n=278 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=124'6250 crt=124'6250 lcod 124'6248 mlcod 124'6248 active+clean+snaptrim trimq=[20c~1c] ps=[204~8]] removing snap head 2019-12-05T06:51:22.605 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:22.597+0000 7f426eb14700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6256 (117'3207,124'6256] local-lis/les=21/22 n=275 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=124'6255 crt=124'6253 lcod 124'6253 mlcod 124'6253 active+clean+snaptrim trimq=[20c~1c] ps=[204~8]] removing snap head 2019-12-05T06:51:22.606 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:22.601+0000 7f426eb14700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6258 (117'3207,124'6258] local-lis/les=21/22 n=273 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=124'6255 crt=124'6255 lcod 124'6253 mlcod 124'6253 active+clean+snaptrim trimq=[20c~1c] ps=[204~8]] removing snap head 2019-12-05T06:51:22.794 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:22.785+0000 7f4272b1c700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6261 (117'3207,124'6261] local-lis/les=21/22 n=270 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=124'6260 crt=124'6258 lcod 124'6258 mlcod 124'6258 active+clean+snaptrim trimq=[20c~1c] ps=[204~8]] removing snap head 2019-12-05T06:51:22.798 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:22.789+0000 7f4272b1c700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6263 (117'3207,124'6263] local-lis/les=21/22 n=268 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=124'6260 crt=124'6260 lcod 124'6258 mlcod 124'6258 active+clean+snaptrim trimq=[20c~1c] ps=[204~8]] removing snap head 2019-12-05T06:51:22.806 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:22.801+0000 7f426eb14700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6266 (117'3207,124'6266] local-lis/les=21/22 n=265 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=124'6265 crt=124'6263 lcod 124'6263 mlcod 124'6263 active+clean+snaptrim trimq=[20c~1c] ps=[204~8]] removing snap head 2019-12-05T06:51:22.837 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:22.829+0000 7f4272b1c700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6271 (117'3207,124'6271] local-lis/les=21/22 n=261 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=124'6270 crt=124'6268 lcod 124'6268 mlcod 124'6268 active+clean+snaptrim trimq=[20c~1c] ps=[204~8]] removing snap head 2019-12-05T06:51:22.841 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:22.833+0000 7f4272b1c700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6273 (117'3207,124'6273] local-lis/les=21/22 n=259 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=124'6270 crt=124'6270 lcod 124'6268 mlcod 124'6268 active+clean+snaptrim trimq=[20c~1c] ps=[204~8]] removing snap head 2019-12-05T06:51:22.855 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:22.845+0000 7f426eb14700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6276 (117'3207,124'6276] local-lis/les=21/22 n=256 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=124'6275 crt=124'6273 lcod 124'6273 mlcod 124'6273 active+clean+snaptrim trimq=[20c~1c] ps=[204~8]] removing snap head 2019-12-05T06:51:22.873 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:22.861+0000 7f426eb14700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6279 (117'3207,124'6279] local-lis/les=21/22 n=253 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=124'6278 crt=124'6276 lcod 124'6276 mlcod 124'6276 active+clean+snaptrim trimq=[20d~1b] ps=[204~9]] removing snap head 2019-12-05T06:51:22.874 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:22.861+0000 7f426eb14700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6281 (117'3207,124'6281] local-lis/les=21/22 n=251 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=124'6278 crt=124'6278 lcod 124'6276 mlcod 124'6276 active+clean+snaptrim trimq=[20d~1b] ps=[204~9]] removing snap head 2019-12-05T06:51:22.886 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:22.877+0000 7f4272b1c700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6284 (117'3207,124'6284] local-lis/les=21/22 n=248 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=124'6283 crt=124'6281 lcod 124'6281 mlcod 124'6281 active+clean+snaptrim trimq=[20d~1b] ps=[204~9]] removing snap head 2019-12-05T06:51:22.888 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:22.881+0000 7f4272b1c700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6286 (117'3207,124'6286] local-lis/les=21/22 n=246 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=124'6283 crt=124'6283 lcod 124'6281 mlcod 124'6281 active+clean+snaptrim trimq=[20d~1b] ps=[204~9]] removing snap head 2019-12-05T06:51:22.902 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:22.893+0000 7f4272b1c700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6289 (117'3207,124'6289] local-lis/les=21/22 n=243 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=124'6288 crt=124'6286 lcod 124'6286 mlcod 124'6286 active+clean+snaptrim trimq=[20d~1b] ps=[204~9]] removing snap head 2019-12-05T06:51:22.904 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:22.897+0000 7f4272b1c700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6291 (117'3207,124'6291] local-lis/les=21/22 n=241 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=124'6288 crt=124'6288 lcod 124'6286 mlcod 124'6286 active+clean+snaptrim trimq=[20d~1b] ps=[204~9]] removing snap head 2019-12-05T06:51:23.304 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:23.293+0000 7f4272b1c700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6293 (117'3207,124'6293] local-lis/les=21/22 n=239 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=124'6293 lcod 124'6291 mlcod 124'6291 active+clean+snaptrim trimq=[20d~1b] ps=[204~9]] removing snap head 2019-12-05T06:51:23.304 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:23.293+0000 7f4272b1c700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6295 (117'3207,124'6295] local-lis/les=21/22 n=237 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=124'6293 crt=124'6293 lcod 124'6291 mlcod 124'6291 active+clean+snaptrim trimq=[20d~1b] ps=[204~9]] removing snap head 2019-12-05T06:51:23.309 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:23.301+0000 7f426eb14700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6297 (117'3207,124'6297] local-lis/les=21/22 n=235 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=124'6297 lcod 124'6295 mlcod 124'6295 active+clean+snaptrim trimq=[20d~1b] ps=[204~9]] removing snap head 2019-12-05T06:51:23.310 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:23.301+0000 7f426eb14700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6299 (117'3207,124'6299] local-lis/les=21/22 n=233 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=124'6297 crt=124'6297 lcod 124'6295 mlcod 124'6295 active+clean+snaptrim trimq=[20d~1b] ps=[204~9]] removing snap head 2019-12-05T06:51:23.316 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:23.309+0000 7f426eb14700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6301 (117'3207,124'6301] local-lis/les=21/22 n=231 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=124'6301 lcod 124'6299 mlcod 124'6299 active+clean+snaptrim trimq=[20d~1b] ps=[204~9]] removing snap head 2019-12-05T06:51:23.324 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:23.317+0000 7f4272b1c700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6305 (117'3207,124'6305] local-lis/les=21/22 n=228 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=124'6305 lcod 124'6303 mlcod 124'6303 active+clean+snaptrim trimq=[20d~1b] ps=[204~9]] removing snap head 2019-12-05T06:51:23.326 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:23.321+0000 7f4272b1c700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6307 (117'3207,124'6307] local-lis/les=21/22 n=226 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=124'6305 crt=124'6305 lcod 124'6303 mlcod 124'6303 active+clean+snaptrim trimq=[20d~1b] ps=[204~9]] removing snap head 2019-12-05T06:51:23.334 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:23.325+0000 7f4272b1c700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6309 (117'3307,124'6309] local-lis/les=21/22 n=224 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=124'6309 lcod 124'6307 mlcod 124'6307 active+clean+snaptrim trimq=[20d~1b] ps=[204~9]] removing snap head 2019-12-05T06:51:23.340 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:23.333+0000 7f4272b1c700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6311 (117'3307,124'6311] local-lis/les=21/22 n=222 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=124'6311 lcod 124'6309 mlcod 124'6309 active+clean+snaptrim trimq=[20e~1a] ps=[204~a]] removing snap head 2019-12-05T06:51:23.341 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:23.333+0000 7f4272b1c700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6313 (117'3307,124'6313] local-lis/les=21/22 n=220 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=124'6311 crt=124'6311 lcod 124'6309 mlcod 124'6309 active+clean+snaptrim trimq=[20e~1a] ps=[204~a]] removing snap head 2019-12-05T06:51:23.347 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:23.341+0000 7f426eb14700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6315 (117'3307,124'6315] local-lis/les=21/22 n=218 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=124'6315 lcod 124'6313 mlcod 124'6313 active+clean+snaptrim trimq=[20e~1a] ps=[204~a]] removing snap head 2019-12-05T06:51:23.348 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:23.341+0000 7f426eb14700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6317 (117'3307,124'6317] local-lis/les=21/22 n=216 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=124'6315 crt=124'6315 lcod 124'6313 mlcod 124'6313 active+clean+snaptrim trimq=[20e~1a] ps=[204~a]] removing snap head 2019-12-05T06:51:23.353 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:23.345+0000 7f426eb14700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6319 (117'3307,124'6319] local-lis/les=21/22 n=214 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=124'6319 lcod 124'6317 mlcod 124'6317 active+clean+snaptrim trimq=[20e~1a] ps=[204~a]] removing snap head 2019-12-05T06:51:23.355 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:23.349+0000 7f426eb14700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6321 (117'3307,124'6321] local-lis/les=21/22 n=212 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=124'6319 crt=124'6319 lcod 124'6317 mlcod 124'6317 active+clean+snaptrim trimq=[20e~1a] ps=[204~a]] removing snap head 2019-12-05T06:51:23.363 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:23.353+0000 7f426eb14700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6323 (117'3307,124'6323] local-lis/les=21/22 n=210 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=124'6323 lcod 124'6321 mlcod 124'6321 active+clean+snaptrim trimq=[20e~1a] ps=[204~a]] removing snap head 2019-12-05T06:51:23.364 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:23.357+0000 7f426eb14700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6325 (117'3307,124'6325] local-lis/les=21/22 n=208 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=124'6323 crt=124'6323 lcod 124'6321 mlcod 124'6321 active+clean+snaptrim trimq=[20e~1a] ps=[204~a]] removing snap head 2019-12-05T06:51:23.367 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:23.361+0000 7f4272b1c700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6327 (117'3307,124'6327] local-lis/les=21/22 n=206 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=124'6327 lcod 124'6325 mlcod 124'6325 active+clean+snaptrim trimq=[20e~1a] ps=[204~a]] removing snap head 2019-12-05T06:51:23.369 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:23.361+0000 7f4272b1c700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6329 (117'3307,124'6329] local-lis/les=21/22 n=204 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=124'6327 crt=124'6327 lcod 124'6325 mlcod 124'6325 active+clean+snaptrim trimq=[20e~1a] ps=[204~a]] removing snap head 2019-12-05T06:51:23.375 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:23.369+0000 7f4272b1c700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6331 (117'3307,124'6331] local-lis/les=21/22 n=202 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=124'6331 lcod 124'6329 mlcod 124'6329 active+clean+snaptrim trimq=[20e~1a] ps=[204~a]] removing snap head 2019-12-05T06:51:23.378 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:23.369+0000 7f4272b1c700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6333 (117'3307,124'6333] local-lis/les=21/22 n=200 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=124'6331 crt=124'6331 lcod 124'6329 mlcod 124'6329 active+clean+snaptrim trimq=[20e~1a] ps=[204~a]] removing snap head 2019-12-05T06:51:23.394 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:23.389+0000 7f4272b1c700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6338 (117'3307,124'6338] local-lis/les=21/22 n=196 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=124'6338 lcod 124'6337 mlcod 124'6337 active+clean+snaptrim trimq=[20f~19] ps=[204~b]] removing snap head 2019-12-05T06:51:23.396 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:23.389+0000 7f4272b1c700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6340 (117'3307,124'6340] local-lis/les=21/22 n=194 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=124'6338 crt=124'6338 lcod 124'6337 mlcod 124'6337 active+clean+snaptrim trimq=[20f~19] ps=[204~b]] removing snap head 2019-12-05T06:51:23.402 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:23.393+0000 7f426eb14700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6342 (117'3307,124'6342] local-lis/les=21/22 n=192 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=124'6342 lcod 124'6340 mlcod 124'6340 active+clean+snaptrim trimq=[20f~19] ps=[204~b]] removing snap head 2019-12-05T06:51:23.404 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:23.397+0000 7f426eb14700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6344 (117'3307,124'6344] local-lis/les=21/22 n=190 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=124'6342 crt=124'6342 lcod 124'6340 mlcod 124'6340 active+clean+snaptrim trimq=[20f~19] ps=[204~b]] removing snap head 2019-12-05T06:51:23.408 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:23.401+0000 7f4272b1c700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6346 (117'3307,124'6346] local-lis/les=21/22 n=188 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=124'6346 lcod 124'6344 mlcod 124'6344 active+clean+snaptrim trimq=[20f~19] ps=[204~b]] removing snap head 2019-12-05T06:51:23.409 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:23.401+0000 7f4272b1c700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6348 (117'3307,124'6348] local-lis/les=21/22 n=186 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=124'6346 crt=124'6346 lcod 124'6344 mlcod 124'6344 active+clean+snaptrim trimq=[20f~19] ps=[204~b]] removing snap head 2019-12-05T06:51:23.415 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:23.409+0000 7f4272b1c700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6350 (117'3307,124'6350] local-lis/les=21/22 n=184 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=124'6350 lcod 124'6348 mlcod 124'6348 active+clean+snaptrim trimq=[20f~19] ps=[204~b]] removing snap head 2019-12-05T06:51:23.417 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:23.409+0000 7f4272b1c700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6352 (117'3307,124'6352] local-lis/les=21/22 n=182 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=124'6350 crt=124'6350 lcod 124'6348 mlcod 124'6348 active+clean+snaptrim trimq=[20f~19] ps=[204~b]] removing snap head 2019-12-05T06:51:23.423 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:23.417+0000 7f426eb14700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6354 (117'3307,124'6354] local-lis/les=21/22 n=180 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=124'6354 lcod 124'6352 mlcod 124'6352 active+clean+snaptrim trimq=[20f~19] ps=[204~b]] removing snap head 2019-12-05T06:51:23.426 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:23.417+0000 7f426eb14700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6356 (117'3307,124'6356] local-lis/les=21/22 n=178 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=124'6354 crt=124'6354 lcod 124'6352 mlcod 124'6352 active+clean+snaptrim trimq=[20f~19] ps=[204~b]] removing snap head 2019-12-05T06:51:23.431 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:23.425+0000 7f426eb14700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6358 (117'3307,124'6358] local-lis/les=21/22 n=176 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=124'6358 lcod 124'6356 mlcod 124'6356 active+clean+snaptrim trimq=[20f~19] ps=[204~b]] removing snap head 2019-12-05T06:51:23.433 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:23.425+0000 7f426eb14700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6360 (117'3307,124'6360] local-lis/les=21/22 n=174 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=124'6358 crt=124'6358 lcod 124'6356 mlcod 124'6356 active+clean+snaptrim trimq=[20f~19] ps=[204~b]] removing snap head 2019-12-05T06:51:23.467 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:23.437+0000 7f426eb14700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6362 (117'3307,124'6362] local-lis/les=21/22 n=172 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=124'6362 lcod 124'6360 mlcod 124'6360 active+clean+snaptrim trimq=[20f~19] ps=[204~b]] removing snap head 2019-12-05T06:51:23.467 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:23.441+0000 7f426eb14700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6364 (117'3307,124'6364] local-lis/les=21/22 n=170 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=124'6362 crt=124'6362 lcod 124'6360 mlcod 124'6360 active+clean+snaptrim trimq=[20f~19] ps=[204~b]] removing snap head 2019-12-05T06:51:23.536 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:23.529+0000 7f4272b1c700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6367 (117'3307,124'6367] local-lis/les=21/22 n=167 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=124'6366 crt=124'6364 lcod 124'6364 mlcod 124'6364 active+clean+snaptrim trimq=[20f~19] ps=[204~b]] removing snap head 2019-12-05T06:51:23.539 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:23.533+0000 7f4272b1c700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6369 (117'3307,124'6369] local-lis/les=21/22 n=165 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=124'6366 crt=124'6366 lcod 124'6364 mlcod 124'6364 active+clean+snaptrim trimq=[20f~19] ps=[204~b]] removing snap head 2019-12-05T06:51:23.544 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:23.537+0000 7f426eb14700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6372 (117'3307,124'6372] local-lis/les=21/22 n=162 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=124'6371 crt=124'6369 lcod 124'6369 mlcod 124'6369 active+clean+snaptrim trimq=[20f~19] ps=[204~b]] removing snap head 2019-12-05T06:51:23.560 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:23.553+0000 7f426eb14700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6376 (117'3307,124'6376] local-lis/les=21/22 n=159 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=124'6376 lcod 124'6374 mlcod 124'6374 active+clean+snaptrim trimq=[20f~19] ps=[204~b]] removing snap head 2019-12-05T06:51:23.561 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:23.553+0000 7f426eb14700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6378 (117'3307,124'6378] local-lis/les=21/22 n=157 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=124'6376 crt=124'6376 lcod 124'6374 mlcod 124'6374 active+clean+snaptrim trimq=[20f~19] ps=[204~b]] removing snap head 2019-12-05T06:51:23.780 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:23.773+0000 7f4272b1c700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6380 (117'3307,124'6380] local-lis/les=21/22 n=155 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=124'6380 lcod 124'6378 mlcod 124'6378 active+clean+snaptrim trimq=[20f~19] ps=[204~b]] removing snap head 2019-12-05T06:51:23.782 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:23.777+0000 7f4272b1c700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6382 (117'3307,124'6382] local-lis/les=21/22 n=153 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=124'6380 crt=124'6380 lcod 124'6378 mlcod 124'6378 active+clean+snaptrim trimq=[20f~19] ps=[204~b]] removing snap head 2019-12-05T06:51:23.857 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:23.849+0000 7f4272b1c700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6385 (117'3307,124'6385] local-lis/les=21/22 n=150 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=124'6384 crt=124'6382 lcod 124'6382 mlcod 124'6382 active+clean+snaptrim trimq=[20f~19] ps=[204~b]] removing snap head 2019-12-05T06:51:23.965 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:23.957+0000 7f4272b1c700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6388 (117'3307,124'6388] local-lis/les=21/22 n=147 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=124'6387 crt=124'6385 lcod 124'6385 mlcod 124'6385 active+clean+snaptrim trimq=[210~18] ps=[204~c]] removing snap head 2019-12-05T06:51:23.966 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:23.957+0000 7f4272b1c700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6390 (117'3307,124'6390] local-lis/les=21/22 n=145 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=124'6387 crt=124'6387 lcod 124'6385 mlcod 124'6385 active+clean+snaptrim trimq=[210~18] ps=[204~c]] removing snap head 2019-12-05T06:51:24.405 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:24.397+0000 7f4272b1c700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6392 (117'3307,124'6392] local-lis/les=21/22 n=143 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=124'6392 lcod 124'6390 mlcod 124'6390 active+clean+snaptrim trimq=[210~18] ps=[204~c]] removing snap head 2019-12-05T06:51:24.407 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:24.401+0000 7f4272b1c700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6394 (117'3307,124'6394] local-lis/les=21/22 n=141 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=124'6392 crt=124'6392 lcod 124'6390 mlcod 124'6390 active+clean+snaptrim trimq=[210~18] ps=[204~c]] removing snap head 2019-12-05T06:51:24.413 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:24.405+0000 7f426eb14700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6396 (117'3307,124'6396] local-lis/les=21/22 n=139 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=124'6396 lcod 124'6394 mlcod 124'6394 active+clean+snaptrim trimq=[210~18] ps=[204~c]] removing snap head 2019-12-05T06:51:24.415 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:24.409+0000 7f426eb14700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6398 (117'3307,124'6398] local-lis/les=21/22 n=137 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=124'6396 crt=124'6396 lcod 124'6394 mlcod 124'6394 active+clean+snaptrim trimq=[210~18] ps=[204~c]] removing snap head 2019-12-05T06:51:24.422 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:24.413+0000 7f4272b1c700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6400 (117'3307,124'6400] local-lis/les=21/22 n=135 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=124'6400 lcod 124'6398 mlcod 124'6398 active+clean+snaptrim trimq=[210~18] ps=[204~c]] removing snap head 2019-12-05T06:51:24.426 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:24.417+0000 7f4272b1c700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6402 (117'3307,124'6402] local-lis/les=21/22 n=133 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=124'6400 crt=124'6400 lcod 124'6398 mlcod 124'6398 active+clean+snaptrim trimq=[210~18] ps=[204~c]] removing snap head 2019-12-05T06:51:24.431 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:24.425+0000 7f4272b1c700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6404 (117'3307,124'6404] local-lis/les=21/22 n=131 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=124'6404 lcod 124'6402 mlcod 124'6402 active+clean+snaptrim trimq=[210~18] ps=[204~c]] removing snap head 2019-12-05T06:51:24.432 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:24.425+0000 7f4272b1c700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6406 (117'3307,124'6406] local-lis/les=21/22 n=129 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=124'6404 crt=124'6404 lcod 124'6402 mlcod 124'6402 active+clean+snaptrim trimq=[210~18] ps=[204~c]] removing snap head 2019-12-05T06:51:24.438 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:24.429+0000 7f4272b1c700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6408 (117'3307,124'6408] local-lis/les=21/22 n=127 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=124'6408 lcod 124'6406 mlcod 124'6406 active+clean+snaptrim trimq=[210~18] ps=[204~c]] removing snap head 2019-12-05T06:51:24.440 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:24.433+0000 7f4272b1c700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6410 (118'3408,124'6410] local-lis/les=21/22 n=125 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=124'6408 crt=124'6408 lcod 124'6406 mlcod 124'6406 active+clean+snaptrim trimq=[210~18] ps=[204~c]] removing snap head 2019-12-05T06:51:24.447 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:24.441+0000 7f426eb14700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6412 (118'3408,124'6412] local-lis/les=21/22 n=123 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=124'6412 lcod 124'6410 mlcod 124'6410 active+clean+snaptrim trimq=[210~18] ps=[204~c]] removing snap head 2019-12-05T06:51:24.451 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:24.441+0000 7f426eb14700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6414 (118'3408,124'6414] local-lis/les=21/22 n=121 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=124'6412 crt=124'6412 lcod 124'6410 mlcod 124'6410 active+clean+snaptrim trimq=[210~18] ps=[204~c]] removing snap head 2019-12-05T06:51:24.454 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:24.445+0000 7f426eb14700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6416 (118'3408,124'6416] local-lis/les=21/22 n=119 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=124'6416 lcod 124'6414 mlcod 124'6414 active+clean+snaptrim trimq=[210~18] ps=[204~c]] removing snap head 2019-12-05T06:51:24.455 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:24.449+0000 7f426eb14700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6418 (118'3408,124'6418] local-lis/les=21/22 n=117 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=124'6416 crt=124'6416 lcod 124'6414 mlcod 124'6414 active+clean+snaptrim trimq=[210~18] ps=[204~c]] removing snap head 2019-12-05T06:51:24.459 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:24.453+0000 7f4272b1c700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6420 (118'3408,124'6420] local-lis/les=21/22 n=115 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=124'6420 lcod 124'6418 mlcod 124'6418 active+clean+snaptrim trimq=[210~18] ps=[204~c]] removing snap head 2019-12-05T06:51:24.461 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:24.453+0000 7f4272b1c700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6422 (118'3408,124'6422] local-lis/les=21/22 n=113 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=124'6420 crt=124'6420 lcod 124'6418 mlcod 124'6418 active+clean+snaptrim trimq=[210~18] ps=[204~c]] removing snap head 2019-12-05T06:51:24.465 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:24.457+0000 7f426eb14700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6424 (118'3408,124'6424] local-lis/les=21/22 n=111 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=124'6424 lcod 124'6422 mlcod 124'6422 active+clean+snaptrim trimq=[210~18] ps=[204~c]] removing snap head 2019-12-05T06:51:24.467 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:24.461+0000 7f426eb14700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6426 (118'3408,124'6426] local-lis/les=21/22 n=109 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=124'6424 crt=124'6424 lcod 124'6422 mlcod 124'6422 active+clean+snaptrim trimq=[210~18] ps=[204~c]] removing snap head 2019-12-05T06:51:24.471 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:24.465+0000 7f4272b1c700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6428 (118'3408,124'6428] local-lis/les=21/22 n=107 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=124'6428 lcod 124'6426 mlcod 124'6426 active+clean+snaptrim trimq=[210~18] ps=[204~c]] removing snap head 2019-12-05T06:51:24.473 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:24.465+0000 7f4272b1c700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6430 (118'3408,124'6430] local-lis/les=21/22 n=105 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=124'6428 crt=124'6428 lcod 124'6426 mlcod 124'6426 active+clean+snaptrim trimq=[210~18] ps=[204~c]] removing snap head 2019-12-05T06:51:24.479 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:24.473+0000 7f4272b1c700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6432 (118'3408,124'6432] local-lis/les=21/22 n=103 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=124'6432 lcod 124'6430 mlcod 124'6430 active+clean+snaptrim trimq=[210~18] ps=[204~c]] removing snap head 2019-12-05T06:51:24.484 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:24.473+0000 7f4272b1c700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6434 (118'3408,124'6434] local-lis/les=21/22 n=101 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=124'6432 crt=124'6432 lcod 124'6430 mlcod 124'6430 active+clean+snaptrim trimq=[210~18] ps=[204~c]] removing snap head 2019-12-05T06:51:24.488 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:24.481+0000 7f426eb14700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6436 (118'3408,124'6436] local-lis/les=21/22 n=99 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=124'6436 lcod 124'6434 mlcod 124'6434 active+clean+snaptrim trimq=[210~18] ps=[204~c]] removing snap head 2019-12-05T06:51:24.489 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:24.481+0000 7f426eb14700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6438 (118'3408,124'6438] local-lis/les=21/22 n=97 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=124'6436 crt=124'6436 lcod 124'6434 mlcod 124'6434 active+clean+snaptrim trimq=[210~18] ps=[204~c]] removing snap head 2019-12-05T06:51:24.495 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:24.489+0000 7f426eb14700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6442 (118'3408,124'6442] local-lis/les=21/22 n=94 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=124'6440 crt=124'6440 lcod 124'6438 mlcod 124'6438 active+clean+snaptrim trimq=[210~18] ps=[204~c]] removing snap head 2019-12-05T06:51:24.503 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:24.497+0000 7f426eb14700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6444 (118'3408,124'6444] local-lis/les=21/22 n=92 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=124'6444 lcod 124'6442 mlcod 124'6442 active+clean+snaptrim trimq=[211~17] ps=[204~d]] removing snap head 2019-12-05T06:51:24.506 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:24.497+0000 7f426eb14700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6446 (118'3408,124'6446] local-lis/les=21/22 n=90 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=124'6444 crt=124'6444 lcod 124'6442 mlcod 124'6442 active+clean+snaptrim trimq=[211~17] ps=[204~d]] removing snap head 2019-12-05T06:51:24.511 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:24.505+0000 7f4272b1c700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6448 (118'3408,124'6448] local-lis/les=21/22 n=88 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=124'6448 lcod 124'6446 mlcod 124'6446 active+clean+snaptrim trimq=[211~17] ps=[204~d]] removing snap head 2019-12-05T06:51:24.514 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:24.505+0000 7f4272b1c700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6450 (118'3408,124'6450] local-lis/les=21/22 n=86 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=124'6448 crt=124'6448 lcod 124'6446 mlcod 124'6446 active+clean+snaptrim trimq=[211~17] ps=[204~d]] removing snap head 2019-12-05T06:51:24.522 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:24.509+0000 7f426eb14700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6452 (118'3408,124'6452] local-lis/les=21/22 n=84 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=124'6452 lcod 124'6450 mlcod 124'6450 active+clean+snaptrim trimq=[211~17] ps=[204~d]] removing snap head 2019-12-05T06:51:24.524 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:24.513+0000 7f426eb14700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6454 (118'3408,124'6454] local-lis/les=21/22 n=82 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=124'6452 crt=124'6452 lcod 124'6450 mlcod 124'6450 active+clean+snaptrim trimq=[211~17] ps=[204~d]] removing snap head 2019-12-05T06:51:24.524 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:24.517+0000 7f426eb14700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6456 (118'3408,124'6456] local-lis/les=21/22 n=80 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=124'6456 lcod 124'6454 mlcod 124'6454 active+clean+snaptrim trimq=[211~17] ps=[204~d]] removing snap head 2019-12-05T06:51:24.526 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:24.517+0000 7f426eb14700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6458 (118'3408,124'6458] local-lis/les=21/22 n=78 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=124'6456 crt=124'6456 lcod 124'6454 mlcod 124'6454 active+clean+snaptrim trimq=[211~17] ps=[204~d]] removing snap head 2019-12-05T06:51:24.531 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:24.525+0000 7f426eb14700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6460 (118'3408,124'6460] local-lis/les=21/22 n=76 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=124'6460 lcod 124'6458 mlcod 124'6458 active+clean+snaptrim trimq=[211~17] ps=[204~d]] removing snap head 2019-12-05T06:51:24.532 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:24.525+0000 7f426eb14700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6462 (118'3408,124'6462] local-lis/les=21/22 n=74 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=124'6460 crt=124'6460 lcod 124'6458 mlcod 124'6458 active+clean+snaptrim trimq=[211~17] ps=[204~d]] removing snap head 2019-12-05T06:51:24.538 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:24.529+0000 7f426eb14700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6464 (118'3408,124'6464] local-lis/les=21/22 n=72 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=124'6464 lcod 124'6462 mlcod 124'6462 active+clean+snaptrim trimq=[211~17] ps=[204~d]] removing snap head 2019-12-05T06:51:24.541 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:24.533+0000 7f426eb14700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6466 (118'3408,124'6466] local-lis/les=21/22 n=70 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=124'6464 crt=124'6464 lcod 124'6462 mlcod 124'6462 active+clean+snaptrim trimq=[211~17] ps=[204~d]] removing snap head 2019-12-05T06:51:24.544 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:24.537+0000 7f4272b1c700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6468 (118'3408,124'6468] local-lis/les=21/22 n=68 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=124'6468 lcod 124'6466 mlcod 124'6466 active+clean+snaptrim trimq=[211~17] ps=[204~d]] removing snap head 2019-12-05T06:51:24.545 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:24.537+0000 7f4272b1c700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6470 (118'3408,124'6470] local-lis/les=21/22 n=66 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=124'6468 crt=124'6468 lcod 124'6466 mlcod 124'6466 active+clean+snaptrim trimq=[211~17] ps=[204~d]] removing snap head 2019-12-05T06:51:24.550 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:24.545+0000 7f426eb14700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6472 (118'3408,124'6472] local-lis/les=21/22 n=64 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=124'6472 lcod 124'6470 mlcod 124'6470 active+clean+snaptrim trimq=[211~17] ps=[204~d]] removing snap head 2019-12-05T06:51:24.552 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:24.545+0000 7f426eb14700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6474 (118'3408,124'6474] local-lis/les=21/22 n=62 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=124'6472 crt=124'6472 lcod 124'6470 mlcod 124'6470 active+clean+snaptrim trimq=[211~17] ps=[204~d]] removing snap head 2019-12-05T06:51:24.557 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:24.549+0000 7f4272b1c700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6476 (118'3408,124'6476] local-lis/les=21/22 n=60 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=124'6476 lcod 124'6474 mlcod 124'6474 active+clean+snaptrim trimq=[211~17] ps=[204~d]] removing snap head 2019-12-05T06:51:24.558 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:24.553+0000 7f4272b1c700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6478 (118'3408,124'6478] local-lis/les=21/22 n=58 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=124'6476 crt=124'6476 lcod 124'6474 mlcod 124'6474 active+clean+snaptrim trimq=[211~17] ps=[204~d]] removing snap head 2019-12-05T06:51:24.563 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:24.557+0000 7f426eb14700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6480 (118'3408,124'6480] local-lis/les=21/22 n=56 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=124'6480 lcod 124'6478 mlcod 124'6478 active+clean+snaptrim trimq=[211~17] ps=[204~d]] removing snap head 2019-12-05T06:51:24.565 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:24.557+0000 7f426eb14700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6482 (118'3408,124'6482] local-lis/les=21/22 n=54 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=124'6480 crt=124'6480 lcod 124'6478 mlcod 124'6478 active+clean+snaptrim trimq=[211~17] ps=[204~d]] removing snap head 2019-12-05T06:51:24.571 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:24.565+0000 7f4272b1c700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6484 (118'3408,124'6484] local-lis/les=21/22 n=52 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=124'6484 lcod 124'6482 mlcod 124'6482 active+clean+snaptrim trimq=[211~17] ps=[204~d]] removing snap head 2019-12-05T06:51:24.573 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:24.565+0000 7f4272b1c700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6486 (118'3408,124'6486] local-lis/les=21/22 n=50 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=124'6484 crt=124'6484 lcod 124'6482 mlcod 124'6482 active+clean+snaptrim trimq=[211~17] ps=[204~d]] removing snap head 2019-12-05T06:51:24.580 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:24.573+0000 7f426eb14700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6488 (118'3408,124'6488] local-lis/les=21/22 n=48 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=124'6488 lcod 124'6486 mlcod 124'6486 active+clean+snaptrim trimq=[211~17] ps=[204~d]] removing snap head 2019-12-05T06:51:24.581 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:24.573+0000 7f426eb14700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6490 (118'3408,124'6490] local-lis/les=21/22 n=46 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=124'6488 crt=124'6488 lcod 124'6486 mlcod 124'6486 active+clean+snaptrim trimq=[211~17] ps=[204~d]] removing snap head 2019-12-05T06:51:24.589 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:24.581+0000 7f426eb14700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6494 (118'3408,124'6494] local-lis/les=21/22 n=43 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=124'6492 crt=124'6492 lcod 124'6490 mlcod 124'6490 active+clean+snaptrim trimq=[211~17] ps=[204~d]] removing snap head 2019-12-05T06:51:24.595 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:24.589+0000 7f4272b1c700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6496 (118'3408,124'6496] local-lis/les=21/22 n=41 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=124'6496 lcod 124'6494 mlcod 124'6494 active+clean+snaptrim trimq=[211~17] ps=[204~d]] removing snap head 2019-12-05T06:51:24.597 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:24.589+0000 7f4272b1c700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6498 (118'3408,124'6498] local-lis/les=21/22 n=39 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=124'6496 crt=124'6496 lcod 124'6494 mlcod 124'6494 active+clean+snaptrim trimq=[211~17] ps=[204~d]] removing snap head 2019-12-05T06:51:24.603 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:24.597+0000 7f426eb14700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6500 (118'3408,124'6500] local-lis/les=21/22 n=37 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=124'6500 lcod 124'6498 mlcod 124'6498 active+clean+snaptrim trimq=[211~17] ps=[204~d]] removing snap head 2019-12-05T06:51:24.604 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:24.597+0000 7f426eb14700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6502 (118'3408,124'6502] local-lis/les=21/22 n=35 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=124'6500 crt=124'6500 lcod 124'6498 mlcod 124'6498 active+clean+snaptrim trimq=[211~17] ps=[204~d]] removing snap head 2019-12-05T06:51:24.611 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:24.601+0000 7f426eb14700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6504 (118'3408,124'6504] local-lis/les=21/22 n=33 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 crt=124'6504 lcod 124'6502 mlcod 124'6502 active+clean+snaptrim trimq=[211~17] ps=[204~d]] removing snap head 2019-12-05T06:51:24.612 INFO:tasks.ceph.osd.4.smithi040.stderr:2019-12-05T06:51:24.605+0000 7f426eb14700 -1 osd.4 pg_epoch: 124 pg[3.3s0( v 124'6506 (118'3408,124'6506] local-lis/les=21/22 n=31 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [4,5,0,6]p4(0) r=0 lpr=21 luod=124'6504 crt=124'6504 lcod 124'6502 mlcod 124'6502 active+clean+snaptrim trimq=[211~17] ps=[204~d]] removing snap head 2019-12-05T06:51:25.757 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-hardlink.sh... 2019-12-05T06:51:25.758 INFO:teuthology.orchestra.run.smithi042:workunit test fs/snaps/snaptest-hardlink.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c5290805033483ba0cec165c06bde07dae5f19ec TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-hardlink.sh 2019-12-05T06:51:25.835 INFO:tasks.workunit.client.0.smithi042.stderr:+ set -e 2019-12-05T06:51:25.836 INFO:tasks.workunit.client.0.smithi042.stderr:+ ceph fs set cephfs allow_new_snaps true --yes-i-really-mean-it 2019-12-05T06:51:25.953 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:25.930+0000 7f146b329700 1 Processor -- start 2019-12-05T06:51:25.953 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:25.930+0000 7f146b329700 1 -- start start 2019-12-05T06:51:25.953 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:25.930+0000 7f146b329700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f14640a69f0 0x7f14640a2840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:51:25.953 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:25.930+0000 7f146b329700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f14640a8030 0x7f14640652f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:51:25.953 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:25.930+0000 7f146b329700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1464065840 con 0x7f1464144fe0 2019-12-05T06:51:25.953 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:25.930+0000 7f146b329700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1464065b20 con 0x7f14640a8030 2019-12-05T06:51:25.954 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:25.930+0000 7f146b329700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14640a25a0 con 0x7f14640a69f0 2019-12-05T06:51:25.954 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:25.930+0000 7f14698c6700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f14640a69f0 0x7f14640a2840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:51:25.954 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:25.930+0000 7f14698c6700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f14640a69f0 0x7f14640a2840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:39138/0 (socket says 172.21.15.42:39138) 2019-12-05T06:51:25.954 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:25.930+0000 7f14698c6700 1 -- 172.21.15.42:0/771461105 learned_addr learned my addr 172.21.15.42:0/771461105 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:51:25.954 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:25.930+0000 7f14690c5700 1 --2- 172.21.15.42:0/771461105 >> v2:172.21.15.40:3300/0 conn(0x7f14640a8030 0x7f14640652f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:51:25.954 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:25.930+0000 7f14698c6700 1 -- 172.21.15.42:0/771461105 >> v2:172.21.15.40:3300/0 conn(0x7f14640a8030 msgr2=0x7f14640652f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:51:25.955 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:25.930+0000 7f14698c6700 1 --2- 172.21.15.42:0/771461105 >> v2:172.21.15.40:3300/0 conn(0x7f14640a8030 0x7f14640652f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:51:25.955 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:25.930+0000 7f14698c6700 1 -- 172.21.15.42:0/771461105 >> v1:172.21.15.40:6789/0 conn(0x7f1464144fe0 legacy=0x7f146409c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:51:25.955 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:25.930+0000 7f14698c6700 1 -- 172.21.15.42:0/771461105 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14640a7b60 con 0x7f14640a69f0 2019-12-05T06:51:25.955 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:25.930+0000 7f14698c6700 1 --2- 172.21.15.42:0/771461105 >> v2:172.21.15.42:3300/0 conn(0x7f14640a69f0 0x7f14640a2840 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f1460008790 tx=0x7f146000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:25.955 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:25.930+0000 7f14698c6700 1 --2- 172.21.15.42:0/771461105 >> v2:172.21.15.42:3300/0 conn(0x7f14640a69f0 0x7f14640a2840 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f1460008790 tx=0x7f146000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:25.955 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:25.930+0000 7f14698c6700 1 --2- 172.21.15.42:0/771461105 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f14640a69f0 0x7f14640a2840 secure :-1 s=READY pgs=3351 cs=0 l=1 rx=0x7f1460008790 tx=0x7f146000fd30).ready entity=mon.1 client_cookie=da1093198414020a server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:51:25.960 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:25.930+0000 7f14698c6700 1 --2- 172.21.15.42:0/771461105 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f14640a69f0 0x7f14640a2840 secure :-1 s=THROTTLE_DONE pgs=3351 cs=0 l=1 rx=0x7f1460008790 tx=0x7f146000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:25.960 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:25.930+0000 7f14688c4700 1 -- 172.21.15.42:0/771461105 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1460011370 con 0x7f14640a69f0 2019-12-05T06:51:25.960 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:25.930+0000 7f14698c6700 1 --2- 172.21.15.42:0/771461105 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f14640a69f0 0x7f14640a2840 secure :-1 s=THROTTLE_DONE pgs=3351 cs=0 l=1 rx=0x7f1460008790 tx=0x7f146000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:25.960 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:25.930+0000 7f14698c6700 1 --2- 172.21.15.42:0/771461105 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f14640a69f0 0x7f14640a2840 secure :-1 s=THROTTLE_DONE pgs=3351 cs=0 l=1 rx=0x7f1460008790 tx=0x7f146000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:25.960 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:25.930+0000 7f14688c4700 1 -- 172.21.15.42:0/771461105 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1460008bb0 con 0x7f14640a69f0 2019-12-05T06:51:25.961 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:25.930+0000 7f14688c4700 1 -- 172.21.15.42:0/771461105 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1460011950 con 0x7f14640a69f0 2019-12-05T06:51:25.961 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:25.934+0000 7f146b329700 1 -- 172.21.15.42:0/771461105 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f14640a69f0 msgr2=0x7f14640a2840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:25.961 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:25.934+0000 7f146b329700 1 --2- 172.21.15.42:0/771461105 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f14640a69f0 0x7f14640a2840 secure :-1 s=READY pgs=3351 cs=0 l=1 rx=0x7f1460008790 tx=0x7f146000fd30).stop 2019-12-05T06:51:25.961 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:25.934+0000 7f146b329700 1 -- 172.21.15.42:0/771461105 shutdown_connections 2019-12-05T06:51:25.961 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:25.934+0000 7f146b329700 1 --2- 172.21.15.42:0/771461105 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f14640a69f0 0x7f14640a2840 unknown :-1 s=CLOSED pgs=3351 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:51:25.962 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:25.934+0000 7f146b329700 1 --2- 172.21.15.42:0/771461105 >> v2:172.21.15.40:3300/0 conn(0x7f14640a8030 0x7f14640652f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:51:25.962 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:25.934+0000 7f146b329700 1 -- 172.21.15.42:0/771461105 shutdown_connections 2019-12-05T06:51:25.962 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:25.934+0000 7f146b329700 1 -- 172.21.15.42:0/771461105 wait complete. 2019-12-05T06:51:25.962 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:25.934+0000 7f146b329700 1 -- 172.21.15.42:0/771461105 >> 172.21.15.42:0/771461105 conn(0x7f1464059570 msgr2=0x7f14640599a0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:51:25.962 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:25.934+0000 7f146b329700 1 Processor -- start 2019-12-05T06:51:25.962 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:25.934+0000 7f146b329700 1 -- start start 2019-12-05T06:51:25.962 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:25.934+0000 7f146b329700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f146409fe90 0x7f1464077f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:51:25.963 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:25.938+0000 7f146b329700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f146407ac90 con 0x7f14640784e0 2019-12-05T06:51:25.963 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:25.938+0000 7f146b329700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14640a0e10 con 0x7f1464079420 2019-12-05T06:51:25.963 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:25.938+0000 7f146b329700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1464078910 con 0x7f146409fe90 2019-12-05T06:51:25.963 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:25.938+0000 7f14690c5700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f146409fe90 0x7f1464077f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:51:25.963 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:25.938+0000 7f14690c5700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f146409fe90 0x7f1464077f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:39144/0 (socket says 172.21.15.42:39144) 2019-12-05T06:51:25.963 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:25.938+0000 7f14690c5700 1 -- 172.21.15.42:0/2632786719 learned_addr learned my addr 172.21.15.42:0/2632786719 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:51:25.963 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:25.938+0000 7f14690c5700 1 -- 172.21.15.42:0/2632786719 >> v1:172.21.15.42:6789/0 conn(0x7f1464079420 legacy=0x7f146407a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:25.964 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:25.938+0000 7f14690c5700 1 -- 172.21.15.42:0/2632786719 >> v1:172.21.15.40:6789/0 conn(0x7f14640784e0 legacy=0x7f146407b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:25.965 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:25.938+0000 7f14567fc700 1 -- 172.21.15.42:0/2632786719 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3953235386 0 0) 0x7f14640a0e10 con 0x7f1464079420 2019-12-05T06:51:25.965 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:25.938+0000 7f14690c5700 1 -- 172.21.15.42:0/2632786719 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f146407ba50 con 0x7f146409fe90 2019-12-05T06:51:25.965 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:25.938+0000 7f14690c5700 1 --2- 172.21.15.42:0/2632786719 >> v2:172.21.15.42:3300/0 conn(0x7f146409fe90 0x7f1464077f90 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f14640a6a20 tx=0x7f14580109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:25.965 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:25.938+0000 7f14690c5700 1 --2- 172.21.15.42:0/2632786719 >> v2:172.21.15.42:3300/0 conn(0x7f146409fe90 0x7f1464077f90 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f14640a6a20 tx=0x7f14580109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:25.965 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:25.938+0000 7f14690c5700 1 --2- 172.21.15.42:0/2632786719 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f146409fe90 0x7f1464077f90 secure :-1 s=READY pgs=3353 cs=0 l=1 rx=0x7f14640a6a20 tx=0x7f14580109a0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:51:25.969 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:25.938+0000 7f14690c5700 1 --2- 172.21.15.42:0/2632786719 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f146409fe90 0x7f1464077f90 secure :-1 s=THROTTLE_DONE pgs=3353 cs=0 l=1 rx=0x7f14640a6a20 tx=0x7f14580109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:25.969 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:25.938+0000 7f14567fc700 1 -- 172.21.15.42:0/2632786719 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1458005930 con 0x7f146409fe90 2019-12-05T06:51:25.969 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:25.938+0000 7f14690c5700 1 --2- 172.21.15.42:0/2632786719 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f146409fe90 0x7f1464077f90 secure :-1 s=THROTTLE_DONE pgs=3353 cs=0 l=1 rx=0x7f14640a6a20 tx=0x7f14580109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:25.969 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:25.938+0000 7f14690c5700 1 --2- 172.21.15.42:0/2632786719 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f146409fe90 0x7f1464077f90 secure :-1 s=THROTTLE_DONE pgs=3353 cs=0 l=1 rx=0x7f14640a6a20 tx=0x7f14580109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:25.970 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:25.938+0000 7f14567fc700 1 -- 172.21.15.42:0/2632786719 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1458007160 con 0x7f146409fe90 2019-12-05T06:51:25.970 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:25.938+0000 7f146b329700 1 -- 172.21.15.42:0/2632786719 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f146407bd20 con 0x7f146409fe90 2019-12-05T06:51:25.970 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:25.938+0000 7f14567fc700 1 -- 172.21.15.42:0/2632786719 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f14580060c0 con 0x7f146409fe90 2019-12-05T06:51:25.970 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:25.938+0000 7f146b329700 1 -- 172.21.15.42:0/2632786719 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1464132070 con 0x7f146409fe90 2019-12-05T06:51:25.970 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:25.938+0000 7f14690c5700 1 --2- 172.21.15.42:0/2632786719 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f146409fe90 0x7f1464077f90 secure :-1 s=THROTTLE_DONE pgs=3353 cs=0 l=1 rx=0x7f14640a6a20 tx=0x7f14580109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:25.970 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:25.938+0000 7f14567fc700 1 -- 172.21.15.42:0/2632786719 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f1458007ae0 con 0x7f146409fe90 2019-12-05T06:51:25.970 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:25.938+0000 7f14690c5700 1 --2- 172.21.15.42:0/2632786719 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f146409fe90 0x7f1464077f90 secure :-1 s=THROTTLE_DONE pgs=3353 cs=0 l=1 rx=0x7f14640a6a20 tx=0x7f14580109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:25.971 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:25.938+0000 7f14567fc700 1 --2- 172.21.15.42:0/2632786719 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f144c03ac10 0x7f144c03d120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:51:25.971 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:25.938+0000 7f14567fc700 1 -- 172.21.15.42:0/2632786719 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 12365+0+0 (secure 0 0 0) 0x7f145804aaf0 con 0x7f146409fe90 2019-12-05T06:51:25.971 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:25.938+0000 7f14698c6700 1 --2- 172.21.15.42:0/2632786719 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f144c03ac10 0x7f144c03d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:51:25.972 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:25.942+0000 7f14698c6700 1 --2- 172.21.15.42:0/2632786719 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f144c03ac10 0x7f144c03d120 crc :-1 s=READY pgs=1404 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:51:25.972 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:25.946+0000 7f146b329700 1 -- 172.21.15.42:0/2632786719 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f146405d2c0 con 0x7f146409fe90 2019-12-05T06:51:25.973 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:25.954+0000 7f14690c5700 1 --2- 172.21.15.42:0/2632786719 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f146409fe90 0x7f1464077f90 secure :-1 s=THROTTLE_DONE pgs=3353 cs=0 l=1 rx=0x7f14640a6a20 tx=0x7f14580109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:25.973 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:25.954+0000 7f14567fc700 1 -- 172.21.15.42:0/2632786719 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f1458031dd0 con 0x7f146409fe90 2019-12-05T06:51:26.132 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:26.126+0000 7f146b329700 1 -- 172.21.15.42:0/2632786719 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"var": "allow_new_snaps", "prefix": "fs set", "fs_name": "cephfs", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f146414fff0 con 0x7f146409fe90 2019-12-05T06:51:26.447 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:26.442+0000 7f14690c5700 1 --2- 172.21.15.42:0/2632786719 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f146409fe90 0x7f1464077f90 secure :-1 s=THROTTLE_DONE pgs=3353 cs=0 l=1 rx=0x7f14640a6a20 tx=0x7f14580109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:26.447 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:26.442+0000 7f14567fc700 1 -- 172.21.15.42:0/2632786719 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"var": "allow_new_snaps", "prefix": "fs set", "fs_name": "cephfs", "val": "true", "yes_i_really_mean_it": true}]=0 enabled new snapshots v405) v1 ==== 167+0+0 (secure 0 0 0) 0x7f1458032980 con 0x7f146409fe90 2019-12-05T06:51:26.611 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:26.606+0000 7f146b329700 1 -- 172.21.15.42:0/2632786719 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"var": "allow_new_snaps", "prefix": "fs set", "fs_name": "cephfs", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f146405ba70 con 0x7f146409fe90 2019-12-05T06:51:27.454 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:27.450+0000 7f14690c5700 1 --2- 172.21.15.42:0/2632786719 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f146409fe90 0x7f1464077f90 secure :-1 s=THROTTLE_DONE pgs=3353 cs=0 l=1 rx=0x7f14640a6a20 tx=0x7f14580109a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:27.454 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:27.450+0000 7f14567fc700 1 -- 172.21.15.42:0/2632786719 <== mon.1 v2:172.21.15.42:3300/0 8 ==== mon_command_ack([{"var": "allow_new_snaps", "prefix": "fs set", "fs_name": "cephfs", "val": "true", "yes_i_really_mean_it": true}]=0 enabled new snapshots v406) v1 ==== 167+0+0 (secure 0 0 0) 0x7f1458033460 con 0x7f146409fe90 2019-12-05T06:51:27.476 INFO:tasks.workunit.client.0.smithi042.stderr:enabled new snapshots 2019-12-05T06:51:27.480 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:27.474+0000 7f146b329700 1 -- 172.21.15.42:0/2632786719 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f144c03ac10 msgr2=0x7f144c03d120 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:27.481 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:27.474+0000 7f146b329700 1 --2- 172.21.15.42:0/2632786719 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f144c03ac10 0x7f144c03d120 crc :-1 s=READY pgs=1404 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:51:27.481 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:27.474+0000 7f146b329700 1 -- 172.21.15.42:0/2632786719 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f146409fe90 msgr2=0x7f1464077f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:27.481 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:27.474+0000 7f146b329700 1 --2- 172.21.15.42:0/2632786719 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f146409fe90 0x7f1464077f90 secure :-1 s=READY pgs=3353 cs=0 l=1 rx=0x7f14640a6a20 tx=0x7f14580109a0).stop 2019-12-05T06:51:27.481 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:27.478+0000 7f146b329700 1 -- 172.21.15.42:0/2632786719 shutdown_connections 2019-12-05T06:51:27.481 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:27.478+0000 7f146b329700 1 --2- 172.21.15.42:0/2632786719 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f144c03ac10 0x7f144c03d120 unknown :-1 s=CLOSED pgs=1404 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:51:27.481 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:27.478+0000 7f146b329700 1 --2- 172.21.15.42:0/2632786719 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f146409fe90 0x7f1464077f90 unknown :-1 s=CLOSED pgs=3353 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:51:27.482 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:27.478+0000 7f146b329700 1 -- 172.21.15.42:0/2632786719 shutdown_connections 2019-12-05T06:51:27.482 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:27.478+0000 7f146b329700 1 -- 172.21.15.42:0/2632786719 wait complete. 2019-12-05T06:51:27.482 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:27.478+0000 7f146b329700 1 -- 172.21.15.42:0/2632786719 >> 172.21.15.42:0/2632786719 conn(0x7f14640a2020 msgr2=0x7f14641193c0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:51:27.493 INFO:tasks.workunit.client.0.smithi042.stderr:+ mkdir 1 2 2019-12-05T06:51:27.503 INFO:tasks.workunit.client.0.smithi042.stderr:+ echo asdf 2019-12-05T06:51:27.505 INFO:tasks.workunit.client.0.smithi042.stderr:+ echo asdf 2019-12-05T06:51:27.505 INFO:tasks.workunit.client.0.smithi042.stderr:+ ln 1/file1 2/file1 2019-12-05T06:51:27.514 INFO:tasks.workunit.client.0.smithi042.stderr:+ ln 1/file2 2/file2 2019-12-05T06:51:27.521 INFO:tasks.workunit.client.0.smithi042.stderr:+ mkdir 2/.snap/s1 2019-12-05T06:51:27.560 INFO:tasks.workunit.client.0.smithi042.stderr:+ echo qwer 2019-12-05T06:51:27.563 INFO:tasks.workunit.client.0.smithi042.stderr:+ grep asdf 2/.snap/s1/file1 2019-12-05T06:51:27.588 INFO:tasks.workunit.client.0.smithi042.stdout:asdf 2019-12-05T06:51:27.588 INFO:tasks.workunit.client.0.smithi042.stderr:+ rm -f 1/file2 2019-12-05T06:51:27.591 INFO:tasks.workunit.client.0.smithi042.stderr:+ grep asdf 2/.snap/s1/file2 2019-12-05T06:51:27.607 INFO:tasks.workunit.client.0.smithi042.stdout:asdf 2019-12-05T06:51:27.607 INFO:tasks.workunit.client.0.smithi042.stderr:+ rm -f 2/file2 2019-12-05T06:51:27.620 INFO:tasks.workunit.client.0.smithi042.stderr:+ grep asdf 2/.snap/s1/file2 2019-12-05T06:51:27.630 INFO:tasks.workunit.client.0.smithi042.stdout:asdf 2019-12-05T06:51:27.631 INFO:tasks.workunit.client.0.smithi042.stderr:+ rmdir 2/.snap/s1 2019-12-05T06:51:27.639 INFO:tasks.workunit.client.0.smithi042.stderr:+ rm -rf 1 2 2019-12-05T06:51:27.670 INFO:tasks.workunit.client.0.smithi042.stdout:OK 2019-12-05T06:51:27.672 INFO:tasks.workunit.client.0.smithi042.stderr:+ echo OK 2019-12-05T06:51:27.672 INFO:teuthology.orchestra.run:Running command with timeout 3600 2019-12-05T06:51:27.672 INFO:teuthology.orchestra.run.smithi042:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2019-12-05T06:51:27.774 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-intodir.sh... 2019-12-05T06:51:27.775 INFO:teuthology.orchestra.run.smithi042:workunit test fs/snaps/snaptest-intodir.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c5290805033483ba0cec165c06bde07dae5f19ec TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-intodir.sh 2019-12-05T06:51:27.865 INFO:tasks.workunit.client.0.smithi042.stderr:+ ceph fs set cephfs allow_new_snaps true --yes-i-really-mean-it 2019-12-05T06:51:27.990 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:27.966+0000 7fcae74cc700 1 Processor -- start 2019-12-05T06:51:27.990 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:27.966+0000 7fcae74cc700 1 -- start start 2019-12-05T06:51:27.990 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:27.970+0000 7fcae74cc700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fcae00a69e0 0x7fcae00a2830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:51:27.990 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:27.970+0000 7fcae74cc700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fcae00652e0 0x7fcae014f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:51:27.990 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:27.970+0000 7fcae74cc700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcae00a7b50 con 0x7fcae00a8020 2019-12-05T06:51:27.991 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:27.970+0000 7fcae74cc700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcae0065b10 con 0x7fcae00652e0 2019-12-05T06:51:27.991 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:27.970+0000 7fcae74cc700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcae00a2590 con 0x7fcae00a69e0 2019-12-05T06:51:27.991 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:27.970+0000 7fcae5a69700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fcae00a69e0 0x7fcae00a2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:51:27.991 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:27.970+0000 7fcae5a69700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fcae00a69e0 0x7fcae00a2830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:39168/0 (socket says 172.21.15.42:39168) 2019-12-05T06:51:27.991 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:27.970+0000 7fcae5a69700 1 -- 172.21.15.42:0/3936230285 learned_addr learned my addr 172.21.15.42:0/3936230285 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:51:27.992 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:27.970+0000 7fcae626a700 1 --2- 172.21.15.42:0/3936230285 >> v2:172.21.15.40:3300/0 conn(0x7fcae00652e0 0x7fcae014f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:51:27.992 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:27.970+0000 7fcae5a69700 1 -- 172.21.15.42:0/3936230285 >> v2:172.21.15.40:3300/0 conn(0x7fcae00652e0 msgr2=0x7fcae014f1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:51:27.994 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:27.970+0000 7fcae5a69700 1 --2- 172.21.15.42:0/3936230285 >> v2:172.21.15.40:3300/0 conn(0x7fcae00652e0 0x7fcae014f1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:51:27.994 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:27.970+0000 7fcae5a69700 1 -- 172.21.15.42:0/3936230285 >> v1:172.21.15.40:6789/0 conn(0x7fcae00a8020 legacy=0x7fcae0144fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:51:27.994 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:27.970+0000 7fcae5a69700 1 -- 172.21.15.42:0/3936230285 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcae00a8450 con 0x7fcae00a69e0 2019-12-05T06:51:27.994 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:27.970+0000 7fcae626a700 1 --2- 172.21.15.42:0/3936230285 >> v2:172.21.15.40:3300/0 conn(0x7fcae00652e0 0x7fcae014f1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:51:27.994 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:27.970+0000 7fcae5a69700 1 --2- 172.21.15.42:0/3936230285 >> v2:172.21.15.42:3300/0 conn(0x7fcae00a69e0 0x7fcae00a2830 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fcae00a2010 tx=0x7fcadc00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:27.995 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:27.970+0000 7fcae5a69700 1 --2- 172.21.15.42:0/3936230285 >> v2:172.21.15.42:3300/0 conn(0x7fcae00a69e0 0x7fcae00a2830 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fcae00a2010 tx=0x7fcadc00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:27.995 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:27.970+0000 7fcae5a69700 1 --2- 172.21.15.42:0/3936230285 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcae00a69e0 0x7fcae00a2830 secure :-1 s=READY pgs=3354 cs=0 l=1 rx=0x7fcae00a2010 tx=0x7fcadc00fd30).ready entity=mon.1 client_cookie=9f246c6a24044512 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:51:27.995 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:27.970+0000 7fcae5a69700 1 --2- 172.21.15.42:0/3936230285 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcae00a69e0 0x7fcae00a2830 secure :-1 s=THROTTLE_DONE pgs=3354 cs=0 l=1 rx=0x7fcae00a2010 tx=0x7fcadc00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:27.995 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:27.970+0000 7fcae5a69700 1 --2- 172.21.15.42:0/3936230285 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcae00a69e0 0x7fcae00a2830 secure :-1 s=THROTTLE_DONE pgs=3354 cs=0 l=1 rx=0x7fcae00a2010 tx=0x7fcadc00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:27.995 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:27.970+0000 7fcae4a67700 1 -- 172.21.15.42:0/3936230285 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcadc011330 con 0x7fcae00a69e0 2019-12-05T06:51:27.995 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:27.970+0000 7fcae5a69700 1 --2- 172.21.15.42:0/3936230285 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcae00a69e0 0x7fcae00a2830 secure :-1 s=THROTTLE_DONE pgs=3354 cs=0 l=1 rx=0x7fcae00a2010 tx=0x7fcadc00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:27.996 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:27.970+0000 7fcae4a67700 1 -- 172.21.15.42:0/3936230285 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcadc008bb0 con 0x7fcae00a69e0 2019-12-05T06:51:27.996 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:27.970+0000 7fcae4a67700 1 -- 172.21.15.42:0/3936230285 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcadc011910 con 0x7fcae00a69e0 2019-12-05T06:51:27.996 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:27.970+0000 7fcae74cc700 1 -- 172.21.15.42:0/3936230285 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcae00a69e0 msgr2=0x7fcae00a2830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:27.996 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:27.970+0000 7fcae74cc700 1 --2- 172.21.15.42:0/3936230285 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcae00a69e0 0x7fcae00a2830 secure :-1 s=READY pgs=3354 cs=0 l=1 rx=0x7fcae00a2010 tx=0x7fcadc00fd30).stop 2019-12-05T06:51:27.996 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:27.970+0000 7fcae74cc700 1 -- 172.21.15.42:0/3936230285 shutdown_connections 2019-12-05T06:51:27.996 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:27.970+0000 7fcae74cc700 1 --2- 172.21.15.42:0/3936230285 >> v2:172.21.15.40:3300/0 conn(0x7fcae00652e0 0x7fcae014f1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:51:27.997 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:27.970+0000 7fcae74cc700 1 --2- 172.21.15.42:0/3936230285 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcae00a69e0 0x7fcae00a2830 unknown :-1 s=CLOSED pgs=3354 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:51:27.997 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:27.970+0000 7fcae74cc700 1 -- 172.21.15.42:0/3936230285 shutdown_connections 2019-12-05T06:51:27.997 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:27.970+0000 7fcae74cc700 1 -- 172.21.15.42:0/3936230285 wait complete. 2019-12-05T06:51:27.997 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:27.970+0000 7fcae74cc700 1 -- 172.21.15.42:0/3936230285 >> 172.21.15.42:0/3936230285 conn(0x7fcae0059560 msgr2=0x7fcae0059990 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:51:27.998 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:27.974+0000 7fcae74cc700 1 Processor -- start 2019-12-05T06:51:27.998 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:27.974+0000 7fcae74cc700 1 -- start start 2019-12-05T06:51:27.998 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:27.974+0000 7fcae74cc700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fcae005eed0 0x7fcae0132690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:51:27.998 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:27.974+0000 7fcae74cc700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcae011ecf0 con 0x7fcae00aa410 2019-12-05T06:51:27.998 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:27.974+0000 7fcae74cc700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcae005f300 con 0x7fcae012a230 2019-12-05T06:51:27.998 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:27.974+0000 7fcae74cc700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcae005f680 con 0x7fcae005eed0 2019-12-05T06:51:27.999 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:27.974+0000 7fcae5268700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fcae012a230 0x7fcae014ee60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:39212/0 (socket says 172.21.15.42:39212) 2019-12-05T06:51:27.999 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:27.974+0000 7fcae5268700 1 -- 172.21.15.42:0/3669729972 learned_addr learned my addr 172.21.15.42:0/3669729972 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:51:27.999 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:27.974+0000 7fcae626a700 1 --2- 172.21.15.42:0/3669729972 >> v2:172.21.15.40:3300/0 conn(0x7fcae005eed0 0x7fcae0132690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:51:27.999 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:27.974+0000 7fcad27fc700 1 -- 172.21.15.42:0/3669729972 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1972703396 0 0) 0x7fcae005f300 con 0x7fcae012a230 2019-12-05T06:51:27.999 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:27.974+0000 7fcad27fc700 1 -- 172.21.15.42:0/3669729972 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcac80030b0 con 0x7fcae012a230 2019-12-05T06:51:27.999 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:27.974+0000 7fcae626a700 1 -- 172.21.15.42:0/3669729972 >> v1:172.21.15.42:6789/0 conn(0x7fcae012a230 legacy=0x7fcae014ee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:28.000 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:27.974+0000 7fcad27fc700 1 -- 172.21.15.42:0/3669729972 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2160425602 0 0) 0x7fcae011ecf0 con 0x7fcae00aa410 2019-12-05T06:51:28.000 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:27.974+0000 7fcae626a700 1 -- 172.21.15.42:0/3669729972 >> v1:172.21.15.40:6789/0 conn(0x7fcae00aa410 legacy=0x7fcae0061d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:28.000 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:27.974+0000 7fcae626a700 1 -- 172.21.15.42:0/3669729972 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcae005f940 con 0x7fcae005eed0 2019-12-05T06:51:28.001 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:27.974+0000 7fcae626a700 1 --2- 172.21.15.42:0/3669729972 >> v2:172.21.15.40:3300/0 conn(0x7fcae005eed0 0x7fcae0132690 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fcad80032b0 tx=0x7fcad8010bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:28.001 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:27.978+0000 7fcae626a700 1 --2- 172.21.15.42:0/3669729972 >> v2:172.21.15.40:3300/0 conn(0x7fcae005eed0 0x7fcae0132690 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fcad80032b0 tx=0x7fcad8010bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:28.001 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:27.978+0000 7fcae626a700 1 --2- 172.21.15.42:0/3669729972 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcae005eed0 0x7fcae0132690 secure :-1 s=READY pgs=2537 cs=0 l=1 rx=0x7fcad80032b0 tx=0x7fcad8010bb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:51:28.001 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:27.978+0000 7fcae626a700 1 --2- 172.21.15.42:0/3669729972 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcae005eed0 0x7fcae0132690 secure :-1 s=THROTTLE_DONE pgs=2537 cs=0 l=1 rx=0x7fcad80032b0 tx=0x7fcad8010bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:28.002 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:27.978+0000 7fcad27fc700 1 -- 172.21.15.42:0/3669729972 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcad8005ca0 con 0x7fcae005eed0 2019-12-05T06:51:28.002 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:27.978+0000 7fcae626a700 1 --2- 172.21.15.42:0/3669729972 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcae005eed0 0x7fcae0132690 secure :-1 s=THROTTLE_DONE pgs=2537 cs=0 l=1 rx=0x7fcad80032b0 tx=0x7fcad8010bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:28.002 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:27.978+0000 7fcae626a700 1 --2- 172.21.15.42:0/3669729972 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcae005eed0 0x7fcae0132690 secure :-1 s=THROTTLE_DONE pgs=2537 cs=0 l=1 rx=0x7fcad80032b0 tx=0x7fcad8010bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:28.003 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:27.978+0000 7fcad27fc700 1 -- 172.21.15.42:0/3669729972 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcad80058c0 con 0x7fcae005eed0 2019-12-05T06:51:28.003 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:27.978+0000 7fcad27fc700 1 -- 172.21.15.42:0/3669729972 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcad8007530 con 0x7fcae005eed0 2019-12-05T06:51:28.003 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:27.978+0000 7fcae74cc700 1 -- 172.21.15.42:0/3669729972 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcae0132be0 con 0x7fcae005eed0 2019-12-05T06:51:28.004 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:27.978+0000 7fcae74cc700 1 -- 172.21.15.42:0/3669729972 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcae0132f10 con 0x7fcae005eed0 2019-12-05T06:51:28.004 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:27.978+0000 7fcae626a700 1 -- 172.21.15.42:0/3669729972 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcae005eed0 msgr2=0x7fcae0132690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2019-12-05T06:51:28.004 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:27.978+0000 7fcae626a700 1 -- 172.21.15.42:0/3669729972 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcae005eed0 msgr2=0x7fcae0132690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2019-12-05T06:51:28.004 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:27.978+0000 7fcae626a700 1 --2- 172.21.15.42:0/3669729972 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcae005eed0 0x7fcae0132690 secure :-1 s=READY pgs=2537 cs=0 l=1 rx=0x7fcad80032b0 tx=0x7fcad8010bb0).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-12-05T06:51:28.005 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:27.978+0000 7fcae626a700 1 --2- 172.21.15.42:0/3669729972 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcae005eed0 0x7fcae0132690 secure :-1 s=READY pgs=2537 cs=0 l=1 rx=0x7fcad80032b0 tx=0x7fcad8010bb0).stop 2019-12-05T06:51:28.005 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:27.978+0000 7fcad27fc700 1 -- 172.21.15.42:0/3669729972 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcae005eed0 msgr2=0x7fcae0132690 unknown :-1 s=STATE_CLOSED l=1).mark_down 2019-12-05T06:51:28.005 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:27.978+0000 7fcad27fc700 1 --2- 172.21.15.42:0/3669729972 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcae005eed0 0x7fcae0132690 unknown :-1 s=CLOSED pgs=2537 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:51:28.005 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:27.978+0000 7fcad27fc700 1 --2- 172.21.15.42:0/3669729972 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcac80033c0 0x7fcac80037f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:51:28.005 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:27.978+0000 7fcad27fc700 1 --2- 172.21.15.42:0/3669729972 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcac8003d40 0x7fcac8004420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:51:28.006 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:27.978+0000 7fcad27fc700 1 --2- 172.21.15.42:0/3669729972 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcac8004970 0x7fcac8004da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:51:28.006 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:27.978+0000 7fcad27fc700 1 -- 172.21.15.42:0/3669729972 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcac800b700 con 0x7fcac8004970 2019-12-05T06:51:28.006 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:27.978+0000 7fcad27fc700 1 -- 172.21.15.42:0/3669729972 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcac800b8f0 con 0x7fcac80033c0 2019-12-05T06:51:28.006 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:27.978+0000 7fcad27fc700 1 -- 172.21.15.42:0/3669729972 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcac800bb30 con 0x7fcac8003d40 2019-12-05T06:51:28.007 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:27.978+0000 7fcae5a69700 1 --2- 172.21.15.42:0/3669729972 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcac80033c0 0x7fcac80037f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:51:28.007 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:27.978+0000 7fcae626a700 1 --2- 172.21.15.42:0/3669729972 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcac8004970 0x7fcac8004da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:51:28.007 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:27.978+0000 7fcae5268700 1 --2- 172.21.15.42:0/3669729972 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcac8003d40 0x7fcac8004420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:51:28.007 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:27.978+0000 7fcae5a69700 1 -- 172.21.15.42:0/3669729972 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcac8003d40 msgr2=0x7fcac8004420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:28.008 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:27.978+0000 7fcae5a69700 1 --2- 172.21.15.42:0/3669729972 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcac8003d40 0x7fcac8004420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:51:28.008 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:27.978+0000 7fcae5a69700 1 -- 172.21.15.42:0/3669729972 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcac8004970 msgr2=0x7fcac8004da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:28.008 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:27.978+0000 7fcae5a69700 1 --2- 172.21.15.42:0/3669729972 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcac8004970 0x7fcac8004da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:51:28.008 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:27.978+0000 7fcae626a700 1 -- 172.21.15.42:0/3669729972 reap_dead start 2019-12-05T06:51:28.008 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:27.978+0000 7fcae5a69700 1 -- 172.21.15.42:0/3669729972 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+,osdmap=0}) v3 -- 0x7fcac800bef0 con 0x7fcac80033c0 2019-12-05T06:51:28.008 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:27.978+0000 7fcae5a69700 1 --2- 172.21.15.42:0/3669729972 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcac80033c0 0x7fcac80037f0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fcadc008790 tx=0x7fcadc0079f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:28.009 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:27.978+0000 7fcae5a69700 1 --2- 172.21.15.42:0/3669729972 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcac80033c0 0x7fcac80037f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fcadc008790 tx=0x7fcadc0079f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:28.009 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:27.978+0000 7fcae5a69700 1 --2- 172.21.15.42:0/3669729972 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcac80033c0 0x7fcac80037f0 secure :-1 s=READY pgs=3356 cs=0 l=1 rx=0x7fcadc008790 tx=0x7fcadc0079f0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:51:28.009 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:27.978+0000 7fcae5a69700 1 --2- 172.21.15.42:0/3669729972 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcac80033c0 0x7fcac80037f0 secure :-1 s=THROTTLE_DONE pgs=3356 cs=0 l=1 rx=0x7fcadc008790 tx=0x7fcadc0079f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:28.009 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:27.978+0000 7fcad27fc700 1 -- 172.21.15.42:0/3669729972 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcadc007170 con 0x7fcac80033c0 2019-12-05T06:51:28.009 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:27.978+0000 7fcae5a69700 1 --2- 172.21.15.42:0/3669729972 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcac80033c0 0x7fcac80037f0 secure :-1 s=THROTTLE_DONE pgs=3356 cs=0 l=1 rx=0x7fcadc008790 tx=0x7fcadc0079f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:28.010 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:27.978+0000 7fcad27fc700 1 -- 172.21.15.42:0/3669729972 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcadc011530 con 0x7fcac80033c0 2019-12-05T06:51:28.010 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:27.978+0000 7fcae5a69700 1 --2- 172.21.15.42:0/3669729972 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcac80033c0 0x7fcac80037f0 secure :-1 s=THROTTLE_DONE pgs=3356 cs=0 l=1 rx=0x7fcadc008790 tx=0x7fcadc0079f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:28.010 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:27.982+0000 7fcad27fc700 1 -- 172.21.15.42:0/3669729972 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fcadc005540 con 0x7fcac80033c0 2019-12-05T06:51:28.010 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:27.982+0000 7fcae5a69700 1 --2- 172.21.15.42:0/3669729972 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcac80033c0 0x7fcac80037f0 secure :-1 s=THROTTLE_DONE pgs=3356 cs=0 l=1 rx=0x7fcadc008790 tx=0x7fcadc0079f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:28.010 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:27.982+0000 7fcad27fc700 1 --2- 172.21.15.42:0/3669729972 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fcac803ea10 0x7fcac8040f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:51:28.011 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:27.982+0000 7fcad27fc700 1 -- 172.21.15.42:0/3669729972 <== mon.1 v2:172.21.15.42:3300/0 4 ==== osd_map(124..124 src has 1..124) v4 ==== 12365+0+0 (secure 0 0 0) 0x7fcadc04abc0 con 0x7fcac80033c0 2019-12-05T06:51:28.011 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:27.982+0000 7fcae5268700 1 --2- 172.21.15.42:0/3669729972 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fcac803ea10 0x7fcac8040f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:51:28.011 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:27.982+0000 7fcae5268700 1 --2- 172.21.15.42:0/3669729972 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fcac803ea10 0x7fcac8040f20 crc :-1 s=READY pgs=1405 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:51:28.011 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:27.986+0000 7fcae74cc700 1 -- 172.21.15.42:0/3669729972 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcae005d2b0 con 0x7fcac80033c0 2019-12-05T06:51:28.012 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:27.990+0000 7fcae5a69700 1 --2- 172.21.15.42:0/3669729972 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcac80033c0 0x7fcac80037f0 secure :-1 s=THROTTLE_DONE pgs=3356 cs=0 l=1 rx=0x7fcadc008790 tx=0x7fcadc0079f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:28.012 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:27.990+0000 7fcad27fc700 1 -- 172.21.15.42:0/3669729972 <== mon.1 v2:172.21.15.42:3300/0 5 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fcadc007370 con 0x7fcac80033c0 2019-12-05T06:51:28.159 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:28.154+0000 7fcae74cc700 1 -- 172.21.15.42:0/3669729972 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"var": "allow_new_snaps", "prefix": "fs set", "fs_name": "cephfs", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fcae005ba60 con 0x7fcac80033c0 2019-12-05T06:51:28.463 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:28.458+0000 7fcae5a69700 1 --2- 172.21.15.42:0/3669729972 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcac80033c0 0x7fcac80037f0 secure :-1 s=THROTTLE_DONE pgs=3356 cs=0 l=1 rx=0x7fcadc008790 tx=0x7fcadc0079f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:28.463 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:28.458+0000 7fcad27fc700 1 -- 172.21.15.42:0/3669729972 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"var": "allow_new_snaps", "prefix": "fs set", "fs_name": "cephfs", "val": "true", "yes_i_really_mean_it": true}]=0 enabled new snapshots v407) v1 ==== 167+0+0 (secure 0 0 0) 0x7fcadc02f880 con 0x7fcac80033c0 2019-12-05T06:51:28.591 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:28.586+0000 7fcae74cc700 1 -- 172.21.15.42:0/3669729972 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"var": "allow_new_snaps", "prefix": "fs set", "fs_name": "cephfs", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fcae00aa410 con 0x7fcac80033c0 2019-12-05T06:51:29.466 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:29.462+0000 7fcae5a69700 1 --2- 172.21.15.42:0/3669729972 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcac80033c0 0x7fcac80037f0 secure :-1 s=THROTTLE_DONE pgs=3356 cs=0 l=1 rx=0x7fcadc008790 tx=0x7fcadc0079f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:29.466 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:29.462+0000 7fcad27fc700 1 -- 172.21.15.42:0/3669729972 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"var": "allow_new_snaps", "prefix": "fs set", "fs_name": "cephfs", "val": "true", "yes_i_really_mean_it": true}]=0 enabled new snapshots v408) v1 ==== 167+0+0 (secure 0 0 0) 0x7fcadc032a50 con 0x7fcac80033c0 2019-12-05T06:51:29.507 INFO:tasks.workunit.client.0.smithi042.stderr:enabled new snapshots 2019-12-05T06:51:29.512 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:29.506+0000 7fcae74cc700 1 -- 172.21.15.42:0/3669729972 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fcac803ea10 msgr2=0x7fcac8040f20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:29.512 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:29.506+0000 7fcae74cc700 1 --2- 172.21.15.42:0/3669729972 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fcac803ea10 0x7fcac8040f20 crc :-1 s=READY pgs=1405 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:51:29.512 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:29.506+0000 7fcae74cc700 1 -- 172.21.15.42:0/3669729972 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcac80033c0 msgr2=0x7fcac80037f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:29.512 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:29.506+0000 7fcae74cc700 1 --2- 172.21.15.42:0/3669729972 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcac80033c0 0x7fcac80037f0 secure :-1 s=READY pgs=3356 cs=0 l=1 rx=0x7fcadc008790 tx=0x7fcadc0079f0).stop 2019-12-05T06:51:29.512 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:29.506+0000 7fcae74cc700 1 -- 172.21.15.42:0/3669729972 shutdown_connections 2019-12-05T06:51:29.513 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:29.506+0000 7fcae74cc700 1 --2- 172.21.15.42:0/3669729972 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fcac803ea10 0x7fcac8040f20 unknown :-1 s=CLOSED pgs=1405 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:51:29.513 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:29.506+0000 7fcae74cc700 1 --2- 172.21.15.42:0/3669729972 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcac80033c0 0x7fcac80037f0 unknown :-1 s=CLOSED pgs=3356 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:51:29.513 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:29.506+0000 7fcae74cc700 1 --2- 172.21.15.42:0/3669729972 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcae005eed0 0x7fcae0132690 unknown :-1 s=CLOSED pgs=2537 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:51:29.513 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:29.506+0000 7fcae74cc700 1 -- 172.21.15.42:0/3669729972 shutdown_connections 2019-12-05T06:51:29.513 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:29.506+0000 7fcae74cc700 1 -- 172.21.15.42:0/3669729972 wait complete. 2019-12-05T06:51:29.513 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:29.506+0000 7fcae74cc700 1 -- 172.21.15.42:0/3669729972 >> 172.21.15.42:0/3669729972 conn(0x7fcae0145590 msgr2=0x7fcae011abc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:51:29.523 INFO:tasks.workunit.client.0.smithi042.stderr:+ mkdir foo 2019-12-05T06:51:29.530 INFO:tasks.workunit.client.0.smithi042.stderr:+ mkdir foo/.snap/one 2019-12-05T06:51:29.553 INFO:tasks.workunit.client.0.smithi042.stderr:+ touch bar 2019-12-05T06:51:29.557 INFO:tasks.workunit.client.0.smithi042.stderr:+ mv bar foo 2019-12-05T06:51:29.583 INFO:tasks.workunit.client.0.smithi042.stderr:+ sync 2019-12-05T06:51:32.151 INFO:tasks.ceph.osd.3.smithi042.stderr:2019-12-05T06:51:32.146+0000 7fc9c0445700 -1 osd.3 pg_epoch: 126 pg[3.2s0( v 124'6394 (117'3307,124'6394] local-lis/les=21/22 n=2 ec=21/21 lis/c=21/21 les/c/f=22/23/0 sis=21) [3,5,6,4]p3(0) r=0 lpr=21 crt=124'6394 lcod 124'6393 mlcod 124'6393 active+clean+snaptrim trimq=[228~2] ps=[210~18]] removing snap head 2019-12-05T06:51:33.403 INFO:tasks.workunit.client.0.smithi042.stderr:+ mkdir baz 2019-12-05T06:51:33.409 INFO:tasks.workunit.client.0.smithi042.stderr:+ mkdir baz/.snap/two 2019-12-05T06:51:33.436 INFO:tasks.workunit.client.0.smithi042.stderr:+ mv baz foo 2019-12-05T06:51:33.485 INFO:tasks.workunit.client.0.smithi042.stderr:+ sync 2019-12-05T06:51:33.508 INFO:tasks.workunit.client.0.smithi042.stderr:+ rmdir foo/baz/.snap/two 2019-12-05T06:51:33.530 INFO:tasks.workunit.client.0.smithi042.stderr:+ rmdir foo/.snap/one 2019-12-05T06:51:33.542 INFO:tasks.workunit.client.0.smithi042.stderr:+ rm -r foo 2019-12-05T06:51:33.565 INFO:tasks.workunit.client.0.smithi042.stderr:+ echo OK 2019-12-05T06:51:33.565 INFO:tasks.workunit.client.0.smithi042.stdout:OK 2019-12-05T06:51:33.566 INFO:teuthology.orchestra.run:Running command with timeout 3600 2019-12-05T06:51:33.566 INFO:teuthology.orchestra.run.smithi042:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2019-12-05T06:51:33.666 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-multiple-capsnaps.sh... 2019-12-05T06:51:33.666 INFO:teuthology.orchestra.run.smithi042:workunit test fs/snaps/snaptest-multiple-capsnaps.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c5290805033483ba0cec165c06bde07dae5f19ec TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-multiple-capsnaps.sh 2019-12-05T06:51:33.754 INFO:tasks.workunit.client.0.smithi042.stderr:+ set -e 2019-12-05T06:51:33.754 INFO:tasks.workunit.client.0.smithi042.stderr:+ ceph fs set cephfs allow_new_snaps true --yes-i-really-mean-it 2019-12-05T06:51:33.870 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:33.858+0000 7f2b51241700 1 Processor -- start 2019-12-05T06:51:33.870 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:33.858+0000 7f2b51241700 1 -- start start 2019-12-05T06:51:33.870 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:33.858+0000 7f2b51241700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2b4c0a69f0 0x7f2b4c0a2840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:51:33.871 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:33.858+0000 7f2b51241700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2b4c0a8030 0x7f2b4c0652f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:51:33.871 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:33.858+0000 7f2b51241700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b4c065840 con 0x7f2b4c144fe0 2019-12-05T06:51:33.871 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:33.858+0000 7f2b51241700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b4c065b20 con 0x7f2b4c0a69f0 2019-12-05T06:51:33.871 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:33.858+0000 7f2b51241700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b4c0a25a0 con 0x7f2b4c0a8030 2019-12-05T06:51:33.872 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:33.862+0000 7f2b42d9d700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2b4c0a8030 0x7f2b4c0652f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:51:33.872 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:33.862+0000 7f2b4b59e700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2b4c0a69f0 0x7f2b4c0a2840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:51:33.872 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:33.862+0000 7f2b50a40700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f2b4c144fe0 0x7f2b4c09c8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:48668/0 (socket says 172.21.15.42:48668) 2019-12-05T06:51:33.872 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:33.862+0000 7f2b50a40700 1 -- 172.21.15.42:0/291038421 learned_addr learned my addr 172.21.15.42:0/291038421 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:51:33.872 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:33.862+0000 7f2b42d9d700 1 -- 172.21.15.42:0/291038421 >> v2:172.21.15.40:3300/0 conn(0x7f2b4c0a69f0 msgr2=0x7f2b4c0a2840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:51:33.873 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:33.862+0000 7f2b42d9d700 1 --2- 172.21.15.42:0/291038421 >> v2:172.21.15.40:3300/0 conn(0x7f2b4c0a69f0 0x7f2b4c0a2840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:51:33.873 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:33.862+0000 7f2b42d9d700 1 -- 172.21.15.42:0/291038421 >> v1:172.21.15.40:6789/0 conn(0x7f2b4c144fe0 legacy=0x7f2b4c09c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:33.881 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:33.862+0000 7f2b42d9d700 1 -- 172.21.15.42:0/291038421 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b4c0a7b60 con 0x7f2b4c0a8030 2019-12-05T06:51:33.881 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:33.862+0000 7f2b4b59e700 1 --2- 172.21.15.42:0/291038421 >> v2:172.21.15.40:3300/0 conn(0x7f2b4c0a69f0 0x7f2b4c0a2840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:51:33.881 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:33.862+0000 7f2b42d9d700 1 --2- 172.21.15.42:0/291038421 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2b4c0a8030 0x7f2b4c0652f0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f2b34008790 tx=0x7f2b3400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:33.882 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:33.862+0000 7f2b42d9d700 1 --2- 172.21.15.42:0/291038421 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2b4c0a8030 0x7f2b4c0652f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f2b34008790 tx=0x7f2b3400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:33.882 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:33.862+0000 7f2b42d9d700 1 --2- 172.21.15.42:0/291038421 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2b4c0a8030 0x7f2b4c0652f0 secure :-1 s=READY pgs=739 cs=0 l=1 rx=0x7f2b34008790 tx=0x7f2b3400fc70).ready entity=mon.2 client_cookie=7abd28cd4690ba0c server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:51:33.882 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:33.862+0000 7f2b42d9d700 1 --2- 172.21.15.42:0/291038421 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2b4c0a8030 0x7f2b4c0652f0 secure :-1 s=THROTTLE_DONE pgs=739 cs=0 l=1 rx=0x7f2b34008790 tx=0x7f2b3400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:33.882 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:33.862+0000 7f2b4ad9d700 1 -- 172.21.15.42:0/291038421 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2b340112f0 con 0x7f2b4c0a8030 2019-12-05T06:51:33.882 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:33.862+0000 7f2b42d9d700 1 --2- 172.21.15.42:0/291038421 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2b4c0a8030 0x7f2b4c0652f0 secure :-1 s=THROTTLE_DONE pgs=739 cs=0 l=1 rx=0x7f2b34008790 tx=0x7f2b3400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:33.883 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:33.862+0000 7f2b4ad9d700 1 -- 172.21.15.42:0/291038421 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b34008ad0 con 0x7f2b4c0a8030 2019-12-05T06:51:33.883 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:33.862+0000 7f2b42d9d700 1 --2- 172.21.15.42:0/291038421 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2b4c0a8030 0x7f2b4c0652f0 secure :-1 s=THROTTLE_DONE pgs=739 cs=0 l=1 rx=0x7f2b34008790 tx=0x7f2b3400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:33.888 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:33.862+0000 7f2b4ad9d700 1 -- 172.21.15.42:0/291038421 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2b340118d0 con 0x7f2b4c0a8030 2019-12-05T06:51:33.888 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:33.862+0000 7f2b51241700 1 -- 172.21.15.42:0/291038421 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2b4c0a8030 msgr2=0x7f2b4c0652f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:33.888 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:33.862+0000 7f2b51241700 1 --2- 172.21.15.42:0/291038421 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2b4c0a8030 0x7f2b4c0652f0 secure :-1 s=READY pgs=739 cs=0 l=1 rx=0x7f2b34008790 tx=0x7f2b3400fc70).stop 2019-12-05T06:51:33.888 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:33.862+0000 7f2b51241700 1 -- 172.21.15.42:0/291038421 shutdown_connections 2019-12-05T06:51:33.888 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:33.862+0000 7f2b51241700 1 --2- 172.21.15.42:0/291038421 >> v2:172.21.15.40:3300/0 conn(0x7f2b4c0a69f0 0x7f2b4c0a2840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:51:33.888 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:33.862+0000 7f2b51241700 1 --2- 172.21.15.42:0/291038421 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2b4c0a8030 0x7f2b4c0652f0 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:51:33.889 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:33.862+0000 7f2b51241700 1 -- 172.21.15.42:0/291038421 shutdown_connections 2019-12-05T06:51:33.889 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:33.862+0000 7f2b51241700 1 -- 172.21.15.42:0/291038421 wait complete. 2019-12-05T06:51:33.889 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:33.862+0000 7f2b51241700 1 -- 172.21.15.42:0/291038421 >> 172.21.15.42:0/291038421 conn(0x7f2b4c059570 msgr2=0x7f2b4c0599a0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:51:33.889 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:33.866+0000 7f2b51241700 1 Processor -- start 2019-12-05T06:51:33.889 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:33.866+0000 7f2b51241700 1 -- start start 2019-12-05T06:51:33.894 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:33.866+0000 7f2b51241700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2b4c120b20 0x7f2b4c0a4670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:51:33.894 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:33.866+0000 7f2b51241700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2b4c11fd80 0x7f2b4c1201b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:51:33.894 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:33.866+0000 7f2b51241700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b4c1360c0 con 0x7f2b4c0a98f0 2019-12-05T06:51:33.895 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:33.866+0000 7f2b51241700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b4c063400 con 0x7f2b4c11fd80 2019-12-05T06:51:33.895 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:33.866+0000 7f2b51241700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b4c0636a0 con 0x7f2b4c120b20 2019-12-05T06:51:33.895 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:33.870+0000 7f2b4b59e700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2b4c120b20 0x7f2b4c0a4670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:51:33.895 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:33.870+0000 7f2b50a40700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2b4c11fd80 0x7f2b4c1201b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:51:33.895 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:33.870+0000 7f2b4b59e700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2b4c120b20 0x7f2b4c0a4670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:39658/0 (socket says 172.21.15.42:39658) 2019-12-05T06:51:33.896 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:33.870+0000 7f2b4b59e700 1 -- 172.21.15.42:0/2298646806 learned_addr learned my addr 172.21.15.42:0/2298646806 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:51:33.896 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:33.870+0000 7f2b4b59e700 1 -- 172.21.15.42:0/2298646806 >> v2:172.21.15.40:3300/0 conn(0x7f2b4c11fd80 msgr2=0x7f2b4c1201b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:33.896 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:33.870+0000 7f2b4b59e700 1 --2- 172.21.15.42:0/2298646806 >> v2:172.21.15.40:3300/0 conn(0x7f2b4c11fd80 0x7f2b4c1201b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:51:33.896 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:33.870+0000 7f2b4b59e700 1 -- 172.21.15.42:0/2298646806 >> v1:172.21.15.40:6789/0 conn(0x7f2b4c0a98f0 legacy=0x7f2b4c1368f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:33.896 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:33.870+0000 7f2b4b59e700 1 -- 172.21.15.42:0/2298646806 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b4c13e520 con 0x7f2b4c120b20 2019-12-05T06:51:33.896 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:33.870+0000 7f2b4b59e700 1 --2- 172.21.15.42:0/2298646806 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2b4c120b20 0x7f2b4c0a4670 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f2b3c002ef0 tx=0x7f2b3c010db0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:33.897 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:33.870+0000 7f2b4b59e700 1 --2- 172.21.15.42:0/2298646806 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2b4c120b20 0x7f2b4c0a4670 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f2b3c002ef0 tx=0x7f2b3c010db0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:33.897 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:33.870+0000 7f2b4b59e700 1 --2- 172.21.15.42:0/2298646806 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2b4c120b20 0x7f2b4c0a4670 secure :-1 s=READY pgs=740 cs=0 l=1 rx=0x7f2b3c002ef0 tx=0x7f2b3c010db0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:51:33.897 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:33.870+0000 7f2b4b59e700 1 --2- 172.21.15.42:0/2298646806 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2b4c120b20 0x7f2b4c0a4670 secure :-1 s=THROTTLE_DONE pgs=740 cs=0 l=1 rx=0x7f2b3c002ef0 tx=0x7f2b3c010db0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:33.897 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:33.870+0000 7f2b4b59e700 1 --2- 172.21.15.42:0/2298646806 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2b4c120b20 0x7f2b4c0a4670 secure :-1 s=THROTTLE_DONE pgs=740 cs=0 l=1 rx=0x7f2b3c002ef0 tx=0x7f2b3c010db0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:33.897 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:33.870+0000 7f2b48d99700 1 -- 172.21.15.42:0/2298646806 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2b3c005890 con 0x7f2b4c120b20 2019-12-05T06:51:33.897 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:33.870+0000 7f2b4b59e700 1 --2- 172.21.15.42:0/2298646806 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2b4c120b20 0x7f2b4c0a4670 secure :-1 s=THROTTLE_DONE pgs=740 cs=0 l=1 rx=0x7f2b3c002ef0 tx=0x7f2b3c010db0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:33.898 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:33.870+0000 7f2b48d99700 1 -- 172.21.15.42:0/2298646806 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b3c007160 con 0x7f2b4c120b20 2019-12-05T06:51:33.898 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:33.870+0000 7f2b51241700 1 -- 172.21.15.42:0/2298646806 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b4c13d510 con 0x7f2b4c120b20 2019-12-05T06:51:33.898 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:33.870+0000 7f2b48d99700 1 -- 172.21.15.42:0/2298646806 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2b3c012430 con 0x7f2b4c120b20 2019-12-05T06:51:33.898 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:33.870+0000 7f2b51241700 1 -- 172.21.15.42:0/2298646806 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b4c13da40 con 0x7f2b4c120b20 2019-12-05T06:51:33.898 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:33.870+0000 7f2b4b59e700 1 --2- 172.21.15.42:0/2298646806 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2b4c120b20 0x7f2b4c0a4670 secure :-1 s=THROTTLE_DONE pgs=740 cs=0 l=1 rx=0x7f2b3c002ef0 tx=0x7f2b3c010db0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:33.898 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:33.874+0000 7f2b48d99700 1 -- 172.21.15.42:0/2298646806 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f2b3c010180 con 0x7f2b4c120b20 2019-12-05T06:51:33.899 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:33.874+0000 7f2b4b59e700 1 --2- 172.21.15.42:0/2298646806 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2b4c120b20 0x7f2b4c0a4670 secure :-1 s=THROTTLE_DONE pgs=740 cs=0 l=1 rx=0x7f2b3c002ef0 tx=0x7f2b3c010db0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:33.899 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:33.874+0000 7f2b48d99700 1 --2- 172.21.15.42:0/2298646806 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2b3003ae10 0x7f2b3003d320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:51:33.899 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:33.874+0000 7f2b48d99700 1 -- 172.21.15.42:0/2298646806 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 12365+0+0 (secure 0 0 0) 0x7f2b3c04b0a0 con 0x7f2b4c120b20 2019-12-05T06:51:33.899 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:33.874+0000 7f2b42d9d700 1 --2- 172.21.15.42:0/2298646806 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2b3003ae10 0x7f2b3003d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:51:33.899 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:33.874+0000 7f2b42d9d700 1 --2- 172.21.15.42:0/2298646806 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2b3003ae10 0x7f2b3003d320 crc :-1 s=READY pgs=1406 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:51:33.899 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:33.874+0000 7f2b51241700 1 -- 172.21.15.42:0/2298646806 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b4c05d2c0 con 0x7f2b4c120b20 2019-12-05T06:51:33.900 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:33.882+0000 7f2b4b59e700 1 --2- 172.21.15.42:0/2298646806 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2b4c120b20 0x7f2b4c0a4670 secure :-1 s=THROTTLE_DONE pgs=740 cs=0 l=1 rx=0x7f2b3c002ef0 tx=0x7f2b3c010db0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:33.900 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:33.882+0000 7f2b48d99700 1 -- 172.21.15.42:0/2298646806 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f2b3c032120 con 0x7f2b4c120b20 2019-12-05T06:51:34.063 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:34.058+0000 7f2b51241700 1 -- 172.21.15.42:0/2298646806 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"var": "allow_new_snaps", "prefix": "fs set", "fs_name": "cephfs", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f2b4c05ba70 con 0x7f2b4c120b20 2019-12-05T06:51:34.545 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:34.538+0000 7f2b4b59e700 1 --2- 172.21.15.42:0/2298646806 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2b4c120b20 0x7f2b4c0a4670 secure :-1 s=THROTTLE_DONE pgs=740 cs=0 l=1 rx=0x7f2b3c002ef0 tx=0x7f2b3c010db0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:34.545 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:34.538+0000 7f2b48d99700 1 -- 172.21.15.42:0/2298646806 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"var": "allow_new_snaps", "prefix": "fs set", "fs_name": "cephfs", "val": "true", "yes_i_really_mean_it": true}]=0 enabled new snapshots v409) v1 ==== 167+0+0 (secure 0 0 0) 0x7f2b3c032380 con 0x7f2b4c120b20 2019-12-05T06:51:34.698 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:34.694+0000 7f2b51241700 1 -- 172.21.15.42:0/2298646806 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"var": "allow_new_snaps", "prefix": "fs set", "fs_name": "cephfs", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f2b4c13e110 con 0x7f2b4c120b20 2019-12-05T06:51:35.544 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:35.542+0000 7f2b4b59e700 1 --2- 172.21.15.42:0/2298646806 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2b4c120b20 0x7f2b4c0a4670 secure :-1 s=THROTTLE_DONE pgs=740 cs=0 l=1 rx=0x7f2b3c002ef0 tx=0x7f2b3c010db0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:35.545 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:35.542+0000 7f2b48d99700 1 -- 172.21.15.42:0/2298646806 <== mon.2 v2:172.21.15.40:3301/0 8 ==== mon_command_ack([{"var": "allow_new_snaps", "prefix": "fs set", "fs_name": "cephfs", "val": "true", "yes_i_really_mean_it": true}]=0 enabled new snapshots v410) v1 ==== 167+0+0 (secure 0 0 0) 0x7f2b3c032f30 con 0x7f2b4c120b20 2019-12-05T06:51:35.564 INFO:tasks.workunit.client.0.smithi042.stderr:enabled new snapshots 2019-12-05T06:51:35.568 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:35.566+0000 7f2b51241700 1 -- 172.21.15.42:0/2298646806 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2b3003ae10 msgr2=0x7f2b3003d320 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:35.569 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:35.566+0000 7f2b51241700 1 --2- 172.21.15.42:0/2298646806 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2b3003ae10 0x7f2b3003d320 crc :-1 s=READY pgs=1406 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:51:35.569 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:35.566+0000 7f2b51241700 1 -- 172.21.15.42:0/2298646806 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2b4c120b20 msgr2=0x7f2b4c0a4670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:35.569 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:35.566+0000 7f2b51241700 1 --2- 172.21.15.42:0/2298646806 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2b4c120b20 0x7f2b4c0a4670 secure :-1 s=READY pgs=740 cs=0 l=1 rx=0x7f2b3c002ef0 tx=0x7f2b3c010db0).stop 2019-12-05T06:51:35.569 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:35.566+0000 7f2b51241700 1 -- 172.21.15.42:0/2298646806 shutdown_connections 2019-12-05T06:51:35.569 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:35.566+0000 7f2b51241700 1 --2- 172.21.15.42:0/2298646806 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2b3003ae10 0x7f2b3003d320 unknown :-1 s=CLOSED pgs=1406 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:51:35.569 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:35.566+0000 7f2b51241700 1 --2- 172.21.15.42:0/2298646806 >> v2:172.21.15.40:3300/0 conn(0x7f2b4c11fd80 0x7f2b4c1201b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:51:35.570 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:35.566+0000 7f2b51241700 1 --2- 172.21.15.42:0/2298646806 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2b4c120b20 0x7f2b4c0a4670 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:51:35.570 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:35.566+0000 7f2b51241700 1 -- 172.21.15.42:0/2298646806 shutdown_connections 2019-12-05T06:51:35.570 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:35.566+0000 7f2b51241700 1 -- 172.21.15.42:0/2298646806 wait complete. 2019-12-05T06:51:35.570 INFO:tasks.workunit.client.0.smithi042.stderr:2019-12-05T06:51:35.566+0000 7f2b51241700 1 -- 172.21.15.42:0/2298646806 >> 172.21.15.42:0/2298646806 conn(0x7f2b4c0688a0 msgr2=0x7f2b4c144db0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:51:35.585 INFO:tasks.workunit.client.0.smithi042.stderr:+ echo asdf 2019-12-05T06:51:35.585 INFO:tasks.workunit.client.0.smithi042.stderr:+ mkdir .snap/1 2019-12-05T06:51:35.605 INFO:tasks.workunit.client.0.smithi042.stderr:+ chmod 777 a 2019-12-05T06:51:35.619 INFO:tasks.workunit.client.0.smithi042.stderr:+ mkdir .snap/2 2019-12-05T06:51:35.638 INFO:tasks.workunit.client.0.smithi042.stderr:+ echo qwer 2019-12-05T06:51:35.644 INFO:tasks.workunit.client.0.smithi042.stderr:+ mkdir .snap/3 2019-12-05T06:51:35.662 INFO:tasks.workunit.client.0.smithi042.stderr:+ chmod 666 a 2019-12-05T06:51:35.663 INFO:tasks.workunit.client.0.smithi042.stderr:+ mkdir .snap/4 2019-12-05T06:51:35.678 INFO:tasks.workunit.client.0.smithi042.stderr:+ echo zxcv 2019-12-05T06:51:35.687 INFO:tasks.workunit.client.0.smithi042.stderr:+ mkdir .snap/5 2019-12-05T06:51:36.101 INFO:tasks.workunit.client.0.smithi042.stderr:+ ls -al .snap/1/a .snap/2/a .snap/3/a .snap/4/a .snap/5/a 2019-12-05T06:51:36.145 INFO:tasks.workunit.client.0.smithi042.stdout:-rw-rw-r-- 1 ubuntu ubuntu 5 Dec 5 06:51 .snap/1/a 2019-12-05T06:51:36.145 INFO:tasks.workunit.client.0.smithi042.stdout:-rwxrwxrwx 1 ubuntu ubuntu 5 Dec 5 06:51 .snap/2/a 2019-12-05T06:51:36.145 INFO:tasks.workunit.client.0.smithi042.stdout:-rw-rw-rw- 1 ubuntu ubuntu 5 Dec 5 06:51 .snap/3/a 2019-12-05T06:51:36.146 INFO:tasks.workunit.client.0.smithi042.stdout:-rw-rw-rw- 1 ubuntu ubuntu 5 Dec 5 06:51 .snap/4/a 2019-12-05T06:51:36.146 INFO:tasks.workunit.client.0.smithi042.stdout:-rw-rw-rw- 1 ubuntu ubuntu 5 Dec 5 06:51 .snap/5/a 2019-12-05T06:51:36.146 INFO:tasks.workunit.client.0.smithi042.stderr:+ grep asdf .snap/1/a 2019-12-05T06:51:36.152 INFO:tasks.workunit.client.0.smithi042.stdout:asdf 2019-12-05T06:51:36.153 INFO:tasks.workunit.client.0.smithi042.stderr:+ stat .snap/1/a 2019-12-05T06:51:36.153 INFO:tasks.workunit.client.0.smithi042.stderr:+ grep Size: 5 2019-12-05T06:51:36.180 INFO:tasks.workunit.client.0.smithi042.stdout: Size: 5 Blocks: 1 IO Block: 4194304 regular file 2019-12-05T06:51:36.181 INFO:tasks.workunit.client.0.smithi042.stderr:+ grep asdf .snap/2/a 2019-12-05T06:51:36.185 INFO:tasks.workunit.client.0.smithi042.stdout:asdf 2019-12-05T06:51:36.186 INFO:tasks.workunit.client.0.smithi042.stderr:+ stat .snap/2/a 2019-12-05T06:51:36.186 INFO:tasks.workunit.client.0.smithi042.stderr:+ grep Size: 5 2019-12-05T06:51:36.190 INFO:tasks.workunit.client.0.smithi042.stdout: Size: 5 Blocks: 1 IO Block: 4194304 regular file 2019-12-05T06:51:36.191 INFO:tasks.workunit.client.0.smithi042.stderr:+ stat .snap/2/a 2019-12-05T06:51:36.191 INFO:tasks.workunit.client.0.smithi042.stderr:+ grep -- -rwxrwxrwx 2019-12-05T06:51:36.195 INFO:tasks.workunit.client.0.smithi042.stdout:Access: (0777/-rwxrwxrwx) Uid: ( 1000/ ubuntu) Gid: ( 1216/ ubuntu) 2019-12-05T06:51:36.195 INFO:tasks.workunit.client.0.smithi042.stderr:+ grep qwer .snap/3/a 2019-12-05T06:51:36.202 INFO:tasks.workunit.client.0.smithi042.stdout:qwer 2019-12-05T06:51:36.203 INFO:tasks.workunit.client.0.smithi042.stderr:+ stat .snap/3/a 2019-12-05T06:51:36.203 INFO:tasks.workunit.client.0.smithi042.stderr:+ grep Size: 5 2019-12-05T06:51:36.206 INFO:tasks.workunit.client.0.smithi042.stdout: Size: 5 Blocks: 1 IO Block: 4194304 regular file 2019-12-05T06:51:36.207 INFO:tasks.workunit.client.0.smithi042.stderr:+ + stat .snap/3/a 2019-12-05T06:51:36.207 INFO:tasks.workunit.client.0.smithi042.stderr:grep -- -rwxrwxrwx 2019-12-05T06:51:36.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2019-12-05T06:51:36.212 INFO:tasks.workunit:Stopping ['fs/snaps'] on client.0... 2019-12-05T06:51:36.212 INFO:teuthology.orchestra.run.smithi042:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2019-12-05T06:51:36.425 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 86, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 65, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-pdonnell-testing-20191204.231200/qa/tasks/workunit.py", line 136, in task cleanup=cleanup) File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-pdonnell-testing-20191204.231200/qa/tasks/workunit.py", line 286, in _spawn_on_all_clients timeout=timeout) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/parallel.py", line 87, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/parallel.py", line 101, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/parallel.py", line 37, in resurrect_traceback reraise(*exc_info) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/parallel.py", line 24, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-pdonnell-testing-20191204.231200/qa/tasks/workunit.py", line 409, in _run_tests label="workunit test {workunit}".format(workunit=workunit) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/remote.py", line 198, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 433, in run r.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 158, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 180, in _raise_for_status node=self.hostname, label=self.label CommandFailedError: Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithi042 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c5290805033483ba0cec165c06bde07dae5f19ec TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-multiple-capsnaps.sh' 2019-12-05T06:51:36.426 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse 2019-12-05T06:51:36.436 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients... 2019-12-05T06:51:36.437 INFO:teuthology.orchestra.run:Running command with timeout 900 2019-12-05T06:51:36.437 INFO:teuthology.orchestra.run.smithi042:> stat --file-system '--printf=%T 2019-12-05T06:51:36.437 INFO:teuthology.orchestra.run.smithi042:> ' -- /home/ubuntu/cephtest/mnt.0 2019-12-05T06:51:36.449 INFO:teuthology.orchestra.run.smithi042.stdout:fuseblk 2019-12-05T06:51:36.449 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2019-12-05T06:51:36.449 INFO:teuthology.orchestra.run:Running command with timeout 900 2019-12-05T06:51:36.449 INFO:teuthology.orchestra.run.smithi042:> stat --file-system '--printf=%T 2019-12-05T06:51:36.449 INFO:teuthology.orchestra.run.smithi042:> ' -- /home/ubuntu/cephtest/mnt.0 2019-12-05T06:51:36.524 INFO:teuthology.orchestra.run.smithi042.stdout:fuseblk 2019-12-05T06:51:36.525 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2019-12-05T06:51:36.525 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi042.front.sepia.ceph.com... 2019-12-05T06:51:36.525 INFO:teuthology.orchestra.run:Running command with timeout 1800 2019-12-05T06:51:36.525 INFO:teuthology.orchestra.run.smithi042:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2019-12-05T06:51:36.639 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi042.stderr:ceph-fuse[14738]: fuse finished with error 0 and tester_r 0 2019-12-05T06:51:36.659 INFO:teuthology.orchestra.run:Running command with timeout 900 2019-12-05T06:51:36.659 INFO:teuthology.orchestra.run.smithi042:> stat --file-system '--printf=%T 2019-12-05T06:51:36.660 INFO:teuthology.orchestra.run.smithi042:> ' -- /home/ubuntu/cephtest/mnt.0 2019-12-05T06:51:36.668 INFO:teuthology.orchestra.run.smithi042.stdout:ext2/ext3 2019-12-05T06:51:36.668 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2019-12-05T06:51:36.668 INFO:teuthology.orchestra.run:waiting for 900 2019-12-05T06:51:36.743 INFO:teuthology.orchestra.run:Running command with timeout 300 2019-12-05T06:51:36.743 INFO:teuthology.orchestra.run.smithi042:> rmdir -- /home/ubuntu/cephtest/mnt.0 2019-12-05T06:51:36.755 DEBUG:teuthology.run_tasks:Unwinding manager mds_thrash 2019-12-05T06:51:36.818 INFO:tasks.mds_thrash:joining mds_thrasher 2019-12-05T06:51:36.819 INFO:tasks.mds_thrash.fs.[cephfs]:stat['deactivate'] = 0 2019-12-05T06:51:36.819 INFO:tasks.mds_thrash.fs.[cephfs]:stat['kill'] = 73 2019-12-05T06:51:36.819 INFO:tasks.mds_thrash.fs.[cephfs]:stat['max_mds'] = 0 2019-12-05T06:51:36.820 INFO:tasks.mds_thrash:done joining 2019-12-05T06:51:36.820 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2019-12-05T06:51:38.270 INFO:tasks.ceph.ceph_manager.ceph:Canceling any pending splits or merges... 2019-12-05T06:51:38.270 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2019-12-05T06:51:38.379 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.370+0000 7f3bc95bd700 1 Processor -- start 2019-12-05T06:51:38.379 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.370+0000 7f3bc95bd700 1 -- start start 2019-12-05T06:51:38.379 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.374+0000 7f3bc95bd700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f3bc40a7740 0x7f3bc408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:51:38.380 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.374+0000 7f3bc95bd700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3bc409fae0 0x7f3bc407eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:51:38.380 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.374+0000 7f3bc95bd700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bc40a0710 con 0x7f3bc407e720 2019-12-05T06:51:38.380 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.374+0000 7f3bc95bd700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3bc40a0b00 con 0x7f3bc40a7740 2019-12-05T06:51:38.380 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.374+0000 7f3bc95bd700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3bc408e5b0 con 0x7f3bc409fae0 2019-12-05T06:51:38.380 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.374+0000 7f3bc3fff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3bc409fae0 0x7f3bc407eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:51:38.380 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.374+0000 7f3bc3fff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3bc409fae0 0x7f3bc407eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:39210/0 (socket says 172.21.15.42:39210) 2019-12-05T06:51:38.380 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.374+0000 7f3bc3fff700 1 -- 172.21.15.42:0/2740039201 learned_addr learned my addr 172.21.15.42:0/2740039201 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:51:38.381 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.374+0000 7f3bc37fe700 1 --2- 172.21.15.42:0/2740039201 >> v2:172.21.15.40:3300/0 conn(0x7f3bc40a7740 0x7f3bc408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:51:38.381 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.374+0000 7f3bc3fff700 1 -- 172.21.15.42:0/2740039201 >> v2:172.21.15.40:3300/0 conn(0x7f3bc40a7740 msgr2=0x7f3bc408df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:51:38.381 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.374+0000 7f3bc3fff700 1 --2- 172.21.15.42:0/2740039201 >> v2:172.21.15.40:3300/0 conn(0x7f3bc40a7740 0x7f3bc408df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:51:38.381 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.374+0000 7f3bc3fff700 1 -- 172.21.15.42:0/2740039201 >> v1:172.21.15.40:6789/0 conn(0x7f3bc407e720 legacy=0x7f3bc409f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:51:38.381 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.374+0000 7f3bc3fff700 1 -- 172.21.15.42:0/2740039201 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bc407f0a0 con 0x7f3bc409fae0 2019-12-05T06:51:38.381 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.374+0000 7f3bc3fff700 1 --2- 172.21.15.42:0/2740039201 >> v2:172.21.15.42:3300/0 conn(0x7f3bc409fae0 0x7f3bc407eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f3bbc0032b0 tx=0x7f3bbc0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:38.381 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.374+0000 7f3bc3fff700 1 --2- 172.21.15.42:0/2740039201 >> v2:172.21.15.42:3300/0 conn(0x7f3bc409fae0 0x7f3bc407eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f3bbc0032b0 tx=0x7f3bbc0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:38.382 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.374+0000 7f3bc3fff700 1 --2- 172.21.15.42:0/2740039201 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3bc409fae0 0x7f3bc407eb50 secure :-1 s=READY pgs=3358 cs=0 l=1 rx=0x7f3bbc0032b0 tx=0x7f3bbc0100b0).ready entity=mon.1 client_cookie=a32cf837ae06db1c server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:51:38.382 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.374+0000 7f3bc3fff700 1 --2- 172.21.15.42:0/2740039201 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3bc409fae0 0x7f3bc407eb50 secure :-1 s=THROTTLE_DONE pgs=3358 cs=0 l=1 rx=0x7f3bbc0032b0 tx=0x7f3bbc0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:38.382 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.374+0000 7f3bc3fff700 1 --2- 172.21.15.42:0/2740039201 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3bc409fae0 0x7f3bc407eb50 secure :-1 s=THROTTLE_DONE pgs=3358 cs=0 l=1 rx=0x7f3bbc0032b0 tx=0x7f3bbc0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:38.382 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.374+0000 7f3bc27fc700 1 -- 172.21.15.42:0/2740039201 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3bbc010d60 con 0x7f3bc409fae0 2019-12-05T06:51:38.382 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.374+0000 7f3bc3fff700 1 --2- 172.21.15.42:0/2740039201 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3bc409fae0 0x7f3bc407eb50 secure :-1 s=THROTTLE_DONE pgs=3358 cs=0 l=1 rx=0x7f3bbc0032b0 tx=0x7f3bbc0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:38.382 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.374+0000 7f3bc27fc700 1 -- 172.21.15.42:0/2740039201 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bbc010980 con 0x7f3bc409fae0 2019-12-05T06:51:38.383 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.374+0000 7f3bc27fc700 1 -- 172.21.15.42:0/2740039201 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3bbc0114a0 con 0x7f3bc409fae0 2019-12-05T06:51:38.383 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.374+0000 7f3bc95bd700 1 -- 172.21.15.42:0/2740039201 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3bc409fae0 msgr2=0x7f3bc407eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:38.383 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.374+0000 7f3bc95bd700 1 --2- 172.21.15.42:0/2740039201 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3bc409fae0 0x7f3bc407eb50 secure :-1 s=READY pgs=3358 cs=0 l=1 rx=0x7f3bbc0032b0 tx=0x7f3bbc0100b0).stop 2019-12-05T06:51:38.383 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.374+0000 7f3bc95bd700 1 -- 172.21.15.42:0/2740039201 shutdown_connections 2019-12-05T06:51:38.383 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.374+0000 7f3bc95bd700 1 --2- 172.21.15.42:0/2740039201 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3bc409fae0 0x7f3bc407eb50 unknown :-1 s=CLOSED pgs=3358 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:51:38.383 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.374+0000 7f3bc95bd700 1 --2- 172.21.15.42:0/2740039201 >> v2:172.21.15.40:3300/0 conn(0x7f3bc40a7740 0x7f3bc408df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:51:38.383 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.374+0000 7f3bc95bd700 1 -- 172.21.15.42:0/2740039201 shutdown_connections 2019-12-05T06:51:38.384 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.374+0000 7f3bc95bd700 1 -- 172.21.15.42:0/2740039201 wait complete. 2019-12-05T06:51:38.384 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.374+0000 7f3bc95bd700 1 -- 172.21.15.42:0/2740039201 >> 172.21.15.42:0/2740039201 conn(0x7f3bc40aabe0 msgr2=0x7f3bc40a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:51:38.387 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.378+0000 7f3bc95bd700 1 Processor -- start 2019-12-05T06:51:38.387 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.378+0000 7f3bc95bd700 1 -- start start 2019-12-05T06:51:38.387 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.378+0000 7f3bc95bd700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3bc4118080 0x7f3bc41156d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:51:38.387 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.378+0000 7f3bc95bd700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bc4113b50 con 0x7f3bc41271f0 2019-12-05T06:51:38.387 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.378+0000 7f3bc95bd700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bc4077380 con 0x7f3bc4115c20 2019-12-05T06:51:38.388 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.378+0000 7f3bc95bd700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bc4126f50 con 0x7f3bc4118080 2019-12-05T06:51:38.388 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.378+0000 7f3bc2ffd700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f3bc4115c20 0x7f3bc4076b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:39254/0 (socket says 172.21.15.42:39254) 2019-12-05T06:51:38.388 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.378+0000 7f3bc2ffd700 1 -- 172.21.15.42:0/2215796070 learned_addr learned my addr 172.21.15.42:0/2215796070 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:51:38.388 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.378+0000 7f3bc37fe700 1 --2- 172.21.15.42:0/2215796070 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3bc4118080 0x7f3bc41156d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:51:38.389 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.378+0000 7f3babfff700 1 -- 172.21.15.42:0/2215796070 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1531953128 0 0) 0x7f3bc4077380 con 0x7f3bc4115c20 2019-12-05T06:51:38.389 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.378+0000 7f3babfff700 1 -- 172.21.15.42:0/2215796070 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ba40030b0 con 0x7f3bc4115c20 2019-12-05T06:51:38.389 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.378+0000 7f3babfff700 1 -- 172.21.15.42:0/2215796070 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2655778940 0 0) 0x7f3bc4113b50 con 0x7f3bc41271f0 2019-12-05T06:51:38.389 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.378+0000 7f3babfff700 1 -- 172.21.15.42:0/2215796070 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3bc4077380 con 0x7f3bc41271f0 2019-12-05T06:51:38.389 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.382+0000 7f3bc37fe700 1 -- 172.21.15.42:0/2215796070 >> v1:172.21.15.42:6789/0 conn(0x7f3bc4115c20 legacy=0x7f3bc4076b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:38.389 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.382+0000 7f3bc37fe700 1 -- 172.21.15.42:0/2215796070 >> v1:172.21.15.40:6789/0 conn(0x7f3bc41271f0 legacy=0x7f3bc4127620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:38.390 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.382+0000 7f3bc37fe700 1 -- 172.21.15.42:0/2215796070 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bc4131250 con 0x7f3bc4118080 2019-12-05T06:51:38.390 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.382+0000 7f3bc37fe700 1 --2- 172.21.15.42:0/2215796070 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3bc4118080 0x7f3bc41156d0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f3bb4002f30 tx=0x7f3bb40108b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:38.390 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.382+0000 7f3bc37fe700 1 --2- 172.21.15.42:0/2215796070 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3bc4118080 0x7f3bc41156d0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f3bb4002f30 tx=0x7f3bb40108b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:38.390 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.382+0000 7f3bc37fe700 1 --2- 172.21.15.42:0/2215796070 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3bc4118080 0x7f3bc41156d0 secure :-1 s=READY pgs=741 cs=0 l=1 rx=0x7f3bb4002f30 tx=0x7f3bb40108b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:51:38.390 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.382+0000 7f3bc37fe700 1 --2- 172.21.15.42:0/2215796070 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3bc4118080 0x7f3bc41156d0 secure :-1 s=THROTTLE_DONE pgs=741 cs=0 l=1 rx=0x7f3bb4002f30 tx=0x7f3bb40108b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:38.390 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.382+0000 7f3babfff700 1 -- 172.21.15.42:0/2215796070 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3bb4005930 con 0x7f3bc4118080 2019-12-05T06:51:38.390 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.382+0000 7f3bc37fe700 1 --2- 172.21.15.42:0/2215796070 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3bc4118080 0x7f3bc41156d0 secure :-1 s=THROTTLE_DONE pgs=741 cs=0 l=1 rx=0x7f3bb4002f30 tx=0x7f3bb40108b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:38.391 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.382+0000 7f3bc37fe700 1 --2- 172.21.15.42:0/2215796070 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3bc4118080 0x7f3bc41156d0 secure :-1 s=THROTTLE_DONE pgs=741 cs=0 l=1 rx=0x7f3bb4002f30 tx=0x7f3bb40108b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:38.391 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.382+0000 7f3babfff700 1 -- 172.21.15.42:0/2215796070 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bb4007160 con 0x7f3bc4118080 2019-12-05T06:51:38.391 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.382+0000 7f3babfff700 1 -- 172.21.15.42:0/2215796070 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3bb40060c0 con 0x7f3bc4118080 2019-12-05T06:51:38.391 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.382+0000 7f3bc95bd700 1 -- 172.21.15.42:0/2215796070 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bc412f230 con 0x7f3bc4118080 2019-12-05T06:51:38.391 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.382+0000 7f3bc95bd700 1 -- 172.21.15.42:0/2215796070 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bc412f760 con 0x7f3bc4118080 2019-12-05T06:51:38.391 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.382+0000 7f3bc37fe700 1 --2- 172.21.15.42:0/2215796070 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3bc4118080 0x7f3bc41156d0 secure :-1 s=THROTTLE_DONE pgs=741 cs=0 l=1 rx=0x7f3bb4002f30 tx=0x7f3bb40108b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:38.392 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.382+0000 7f3babfff700 1 -- 172.21.15.42:0/2215796070 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f3bb4007940 con 0x7f3bc4118080 2019-12-05T06:51:38.393 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.382+0000 7f3bc37fe700 1 --2- 172.21.15.42:0/2215796070 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3bc4118080 0x7f3bc41156d0 secure :-1 s=THROTTLE_DONE pgs=741 cs=0 l=1 rx=0x7f3bb4002f30 tx=0x7f3bb40108b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:38.393 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.386+0000 7f3babfff700 1 --2- 172.21.15.42:0/2215796070 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3ba403b4c0 0x7f3ba403d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:51:38.393 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.386+0000 7f3babfff700 1 -- 172.21.15.42:0/2215796070 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 12365+0+0 (secure 0 0 0) 0x7f3bb404aac0 con 0x7f3bc4118080 2019-12-05T06:51:38.393 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.386+0000 7f3bc2ffd700 1 --2- 172.21.15.42:0/2215796070 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3ba403b4c0 0x7f3ba403d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:51:38.393 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.386+0000 7f3bc2ffd700 1 --2- 172.21.15.42:0/2215796070 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3ba403b4c0 0x7f3ba403d9d0 crc :-1 s=READY pgs=1407 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:51:38.393 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.386+0000 7f3bc95bd700 1 -- 172.21.15.42:0/2215796070 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bc405d110 con 0x7f3bc4118080 2019-12-05T06:51:38.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.398+0000 7f3bc37fe700 1 --2- 172.21.15.42:0/2215796070 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3bc4118080 0x7f3bc41156d0 secure :-1 s=THROTTLE_DONE pgs=741 cs=0 l=1 rx=0x7f3bb4002f30 tx=0x7f3bb40108b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:38.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.398+0000 7f3babfff700 1 -- 172.21.15.42:0/2215796070 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f3bb4031b40 con 0x7f3bc4118080 2019-12-05T06:51:38.574 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.570+0000 7f3bc95bd700 1 -- 172.21.15.42:0/2215796070 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3bc405b8c0 con 0x7f3bc4118080 2019-12-05T06:51:38.577 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.574+0000 7f3bc37fe700 1 --2- 172.21.15.42:0/2215796070 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3bc4118080 0x7f3bc41156d0 secure :-1 s=THROTTLE_DONE pgs=741 cs=0 l=1 rx=0x7f3bb4002f30 tx=0x7f3bb40108b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:38.578 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.574+0000 7f3babfff700 1 -- 172.21.15.42:0/2215796070 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v127) v1 ==== 74+0+24452 (secure 0 0 0) 0x7f3bb4031da0 con 0x7f3bc4118080 2019-12-05T06:51:38.578 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:51:38.578 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":127,"fsid":"5d836cfc-a83b-4433-9b3e-b9b2f31c9050","created":"2019-12-05T05:03:13.215602+0000","modified":"2019-12-05T06:51:36.545494+0000","last_up_change":"2019-12-05T05:03:22.664783+0000","last_in_change":"2019-12-05T05:03:22.664783+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2019-12-05T05:03:27.394605+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2019-12-05T05:03:30.419845+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2019-12-05T05:03:33.249075+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"127","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":557,"snap_epoch":127,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e3fc49bb-dcbc-43ff-ac7c-9eb0ea0bd257","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6810","nonce":12118},{"type":"v1","addr":"172.21.15.42:6811","nonce":12118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6812","nonce":12118},{"type":"v1","addr":"172.21.15.42:6813","nonce":12118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6816","nonce":12118},{"type":"v1","addr":"172.21.15.42:6817","nonce":12118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6814","nonce":12118},{"type":"v1","addr":"172.21.15.42:6815","nonce":12118}]},"public_addr":"172.21.15.42:6811/12118","cluster_addr":"172.21.15.42:6813/12118","heartbeat_back_addr":"172.21.15.42:6817/12118","heartbeat_front_addr":"172.21.15.42:6815/12118","state":["exists","up"]},{"osd":1,"uuid":"902543fb-f5a2-4321-9c9c-26f9b9ca8629","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6826","nonce":12119},{"type":"v1","addr":"172.21.15.42:6827","nonce":12119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6828","nonce":12119},{"type":"v1","addr":"172.21.15.42:6829","nonce":12119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6832","nonce":12119},{"type":"v1","addr":"172.21.15.42:6833","nonce":12119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6830","nonce":12119},{"type":"v1","addr":"172.21.15.42:6831","nonce":12119}]},"public_addr":"172.21.15.42:6827/12119","cluster_addr":"172.21.15.42:6829/12119","heartbeat_back_addr":"172.21.15.42:6833/12119","heartbeat_front_addr":"172.21.15.42:6831/12119","state":["exists","up"]},{"osd":2,"uuid":"28739ff7-bed7-4b75-8ea5-b57d088f1250","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6802","nonce":12117},{"type":"v1","addr":"172.21.15.42:6803","nonce":12117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6804","nonce":12117},{"type":"v1","addr":"172.21.15.42:6805","nonce":12117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6808","nonce":12117},{"type":"v1","addr":"172.21.15.42:6809","nonce":12117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6806","nonce":12117},{"type":"v1","addr":"172.21.15.42:6807","nonce":12117}]},"public_addr":"172.21.15.42:6803/12117","cluster_addr":"172.21.15.42:6805/12117","heartbeat_back_addr":"172.21.15.42:6809/12117","heartbeat_front_addr":"172.21.15.42:6807/12117","state":["exists","up"]},{"osd":3,"uuid":"1960f66b-2d40-4bb9-a289-ce8cd0c1c142","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6818","nonce":12116},{"type":"v1","addr":"172.21.15.42:6819","nonce":12116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6820","nonce":12116},{"type":"v1","addr":"172.21.15.42:6821","nonce":12116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6824","nonce":12116},{"type":"v1","addr":"172.21.15.42:6825","nonce":12116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6822","nonce":12116},{"type":"v1","addr":"172.21.15.42:6823","nonce":12116}]},"public_addr":"172.21.15.42:6819/12116","cluster_addr":"172.21.15.42:6821/12116","heartbeat_back_addr":"172.21.15.42:6825/12116","heartbeat_front_addr":"172.21.15.42:6823/12116","state":["exists","up"]},{"osd":4,"uuid":"45bbf1bd-7c8a-4a0d-b76b-afa6148a855c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":12488},{"type":"v1","addr":"172.21.15.40:6809","nonce":12488}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":12488},{"type":"v1","addr":"172.21.15.40:6812","nonce":12488}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":12488},{"type":"v1","addr":"172.21.15.40:6825","nonce":12488}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":12488},{"type":"v1","addr":"172.21.15.40:6819","nonce":12488}]},"public_addr":"172.21.15.40:6809/12488","cluster_addr":"172.21.15.40:6812/12488","heartbeat_back_addr":"172.21.15.40:6825/12488","heartbeat_front_addr":"172.21.15.40:6819/12488","state":["exists","up"]},{"osd":5,"uuid":"8a5511cd-925f-4869-a49e-924383e4ce53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":12489},{"type":"v1","addr":"172.21.15.40:6801","nonce":12489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":12489},{"type":"v1","addr":"172.21.15.40:6803","nonce":12489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":12489},{"type":"v1","addr":"172.21.15.40:6807","nonce":12489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":12489},{"type":"v1","addr":"172.21.15.40:6805","nonce":12489}]},"public_addr":"172.21.15.40:6801/12489","cluster_addr":"172.21.15.40:6803/12489","heartbeat_back_addr":"172.21.15.40:6807/12489","heartbeat_front_addr":"172.21.15.40:6805/12489","state":["exists","up"]},{"osd":6,"uuid":"d692c179-379e-446c-8484-a219a9241e04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6811","nonce":12491},{"type":"v1","addr":"172.21.15.40:6814","nonce":12491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6817","nonce":12491},{"type":"v1","addr":"172.21.15.40:6818","nonce":12491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":12491},{"type":"v1","addr":"172.21.15.40:6828","nonce":12491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":12491},{"type":"v1","addr":"172.21.15.40:6824","nonce":12491}]},"public_addr":"172.21.15.40:6814/12491","cluster_addr":"172.21.15.40:6818/12491","heartbeat_back_addr":"172.21.15.40:6828/12491","heartbeat_front_addr":"172.21.15.40:6824/12491","state":["exists","up"]},{"osd":7,"uuid":"79c3c243-190e-4e35-b769-319a2077e7cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6813","nonce":12490},{"type":"v1","addr":"172.21.15.40:6816","nonce":12490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":12490},{"type":"v1","addr":"172.21.15.40:6823","nonce":12490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":12490},{"type":"v1","addr":"172.21.15.40:6831","nonce":12490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6827","nonce":12490},{"type":"v1","addr":"172.21.15.40:6829","nonce":12490}]},"public_addr":"172.21.15.40:6816/12490","cluster_addr":"172.21.15.40:6823/12490","heartbeat_back_addr":"172.21.15.40:6831/12490","heartbeat_front_addr":"172.21.15.40:6829/12490","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T05:03:21.349150+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T05:03:21.644281+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T05:03:21.394040+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T05:03:21.587096+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T05:03:21.656497+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T05:03:21.510547+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T05:03:21.659358+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T05:03:21.628063+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{"172.21.15.40:6833/815316041":"2019-12-06T06:50:15.403188+0000","172.21.15.40:6833/4178556019":"2019-12-06T06:48:20.366134+0000","172.21.15.40:6832/4178556019":"2019-12-06T06:48:20.366134+0000","172.21.15.42:6835/1449317789":"2019-12-06T06:49:00.379730+0000","172.21.15.42:6835/2489540211":"2019-12-06T06:45:55.301261+0000","172.21.15.40:6833/3599619":"2019-12-06T06:44:25.276320+0000","172.21.15.40:6832/3599619":"2019-12-06T06:44:25.276320+0000","172.21.15.42:6835/1922398668":"2019-12-06T06:43:25.247354+0000","172.21.15.42:6834/1922398668":"2019-12-06T06:43:25.247354+0000","172.21.15.40:6833/474728597":"2019-12-06T06:42:45.234540+0000","172.21.15.40:6832/474728597":"2019-12-06T06:42:45.234540+0000","172.21.15.42:6835/406829134":"2019-12-06T06:40:40.197779+0000","172.21.15.42:6834/406829134":"2019-12-06T06:40:40.197779+0000","172.21.15.40:6832/1777669201":"2019-12-06T06:40:05.186898+0000","172.21.15.42:6834/958796360":"2019-12-06T06:39:30.176520+0000","172.21.15.42:6835/958796360":"2019-12-06T06:39:30.176520+0000","172.21.15.40:6832/4253974783":"2019-12-06T06:37:55.150270+0000","172.21.15.40:6832/2055706790":"2019-12-06T06:33:25.051960+0000","172.21.15.42:6834/1985322937":"2019-12-06T06:31:14.985625+0000","172.21.15.40:6832/3155851600":"2019-12-06T06:30:14.967933+0000","172.21.15.42:6835/3427104185":"2019-12-06T06:28:24.936048+0000","172.21.15.42:6834/3427104185":"2019-12-06T06:28:24.936048+0000","172.21.15.42:6835/3964426552":"2019-12-06T06:24:34.851396+0000","172.21.15.40:6832/2324657302":"2019-12-06T06:26:39.889893+0000","172.21.15.42:6834/3964426552":"2019-12-06T06:24:34.851396+0000","172.21.15.42:6834/2532474042":"2019-12-06T06:35:35.090613+0000","172.21.15.42:6834/2638662429":"2019-12-06T06:21:34.795475+0000","172.21.15.42:6834/1449317789":"2019-12-06T06:49:00.379730+0000","172.21.15.40:6832/1473620421":"2019-12-06T06:19:49.763362+0000","172.21.15.42:6834/3228549071":"2019-12-06T06:19:09.750176+0000","172.21.15.42:6835/4032647860":"2019-12-06T06:14:34.627961+0000","172.21.15.42:6835/2532474042":"2019-12-06T06:35:35.090613+0000","172.21.15.42:6834/4032647860":"2019-12-06T06:14:34.627961+0000","172.21.15.40:6833/3785523496":"2019-12-06T06:13:14.603886+0000","172.21.15.40:6832/3785523496":"2019-12-06T06:13:14.603886+0000","172.21.15.42:6835/1601334502":"2019-12-06T05:34:33.762032+0000","172.21.15.42:6835/2456922143":"2019-12-06T05:50:49.115731+0000","172.21.15.42:6834/1601334502":"2019-12-06T05:34:33.762032+0000","172.21.15.42:6835/3228549071":"2019-12-06T06:19:09.750176+0000","172.21.15.40:6832/3543033403":"2019-12-06T05:32:48.730013+0000","172.21.15.40:6832/815316041":"2019-12-06T06:50:15.403188+0000","172.21.15.42:6835/2636699468":"2019-12-06T06:10:09.528777+0000","172.21.15.42:6835/2638662429":"2019-12-06T06:21:34.795475+0000","172.21.15.40:6833/3168718372":"2019-12-06T05:30:48.693050+0000","172.21.15.40:6832/3789081777":"2019-12-06T06:05:29.441993+0000","172.21.15.40:6833/3061061478":"2019-12-06T05:23:03.537466+0000","172.21.15.40:6833/2667648531":"2019-12-06T06:16:59.698225+0000","172.21.15.40:6833/3789081777":"2019-12-06T06:05:29.441993+0000","172.21.15.42:6835/1233824070":"2019-12-06T05:42:43.928051+0000","172.21.15.40:6833/1149478696":"2019-12-06T05:28:13.631131+0000","172.21.15.40:6832/1149478696":"2019-12-06T05:28:13.631131+0000","172.21.15.40:6832/90248709":"2019-12-06T05:09:23.257810+0000","172.21.15.40:6832/2146240383":"2019-12-06T05:49:19.088366+0000","172.21.15.42:6835/613535662":"2019-12-06T06:04:04.415621+0000","172.21.15.42:6835/1455108684":"2019-12-06T05:27:08.611758+0000","172.21.15.40:6832/3789641053":"2019-12-06T05:37:08.826019+0000","172.21.15.42:6834/1455108684":"2019-12-06T05:27:08.611758+0000","172.21.15.42:6834/3264829400":"2019-12-06T05:17:13.419048+0000","172.21.15.40:6833/3208416000":"2019-12-06T05:25:33.582715+0000","172.21.15.40:6832/526579407":"2019-12-06T06:23:44.835535+0000","172.21.15.40:6832/2667648531":"2019-12-06T06:16:59.698225+0000","172.21.15.42:6835/2544447110":"2019-12-06T05:29:18.666496+0000","172.21.15.40:6832/3208416000":"2019-12-06T05:25:33.582715+0000","172.21.15.42:6835/1214659805":"2019-12-06T05:23:53.552042+0000","172.21.15.40:6833/3789641053":"2019-12-06T05:37:08.826019+0000","172.21.15.42:6835/1414371821":"2019-12-06T05:20:43.495003+0000","172.21.15.42:6834/3757738536":"2019-12-06T05:47:14.050547+0000","172.21.15.40:6833/211584380":"2019-12-06T05:11:18.295434+0000","172.21.15.40:6832/2698421979":"2019-12-06T05:18:48.447318+0000","172.21.15.40:6833/90248709":"2019-12-06T05:09:23.257810+0000","172.21.15.42:6835/1368848584":"2019-12-06T05:59:44.294812+0000","172.21.15.40:6832/3296198322":"2019-12-06T05:04:43.166959+0000","172.21.15.40:6833/4253974783":"2019-12-06T06:37:55.150270+0000","172.21.15.42:6834/5991897":"2019-12-06T05:52:59.155806+0000","172.21.15.42:6835/3028851337":"2019-12-06T05:31:48.711252+0000","172.21.15.40:6833/814397242":"2019-12-06T05:13:08.346622+0000","172.21.15.40:6832/814397242":"2019-12-06T05:13:08.346622+0000","172.21.15.42:6835/2019200199":"2019-12-06T05:07:08.208550+0000","172.21.15.42:6835/211479572":"2019-12-06T05:14:13.365106+0000","172.21.15.40:6833/526579407":"2019-12-06T06:23:44.835535+0000","172.21.15.42:6834/1214659805":"2019-12-06T05:23:53.552042+0000","172.21.15.42:6834/2019200199":"2019-12-06T05:07:08.208550+0000","172.21.15.42:6835/448869597":"2019-12-06T06:00:59.318439+0000","172.21.15.42:6835/3757738536":"2019-12-06T05:47:14.050547+0000","172.21.15.40:6833/1473620421":"2019-12-06T06:19:49.763362+0000","172.21.15.42:6834/3176146895":"2019-12-06T05:12:43.339069+0000","172.21.15.40:6832/123054133":"2019-12-06T05:40:58.896726+0000","172.21.15.42:6835/3176146895":"2019-12-06T05:12:43.339069+0000","172.21.15.40:6833/3150831963":"2019-12-06T05:15:08.381589+0000","172.21.15.42:6834/2253267554":"2019-12-06T06:06:59.469462+0000","172.21.15.40:6832/3150831963":"2019-12-06T05:15:08.381589+0000","172.21.15.40:6833/2324657302":"2019-12-06T06:26:39.889893+0000","172.21.15.42:6834/2636699468":"2019-12-06T06:10:09.528777+0000","172.21.15.40:6832/3061061478":"2019-12-06T05:23:03.537466+0000","172.21.15.40:6833/2698421979":"2019-12-06T05:18:48.447318+0000","172.21.15.42:6835/3650564018":"2019-12-06T05:10:03.269449+0000","172.21.15.42:6834/1414371821":"2019-12-06T05:20:43.495003+0000","172.21.15.40:6833/1777669201":"2019-12-06T06:40:05.186898+0000","172.21.15.40:6833/3296198322":"2019-12-06T05:04:43.166959+0000","172.21.15.42:6835/3264829400":"2019-12-06T05:17:13.419048+0000","172.21.15.40:6833/3543033403":"2019-12-06T05:32:48.730013+0000","172.21.15.42:6834/3838877789":"2019-12-06T05:39:08.862108+0000","172.21.15.42:6834/2456922143":"2019-12-06T05:50:49.115731+0000","172.21.15.42:6834/448869597":"2019-12-06T06:00:59.318439+0000","172.21.15.42:6834/3923117701":"2019-12-06T06:12:09.582993+0000","172.21.15.42:6834/2489540211":"2019-12-06T06:45:55.301261+0000","172.21.15.40:6833/3155851600":"2019-12-06T06:30:14.967933+0000","172.21.15.42:6834/3650564018":"2019-12-06T05:10:03.269449+0000","172.21.15.42:6834/2544447110":"2019-12-06T05:29:18.666496+0000","172.21.15.40:6833/123054133":"2019-12-06T05:40:58.896726+0000","172.21.15.42:6834/1233824070":"2019-12-06T05:42:43.928051+0000","172.21.15.40:6832/211584380":"2019-12-06T05:11:18.295434+0000","172.21.15.42:6834/211479572":"2019-12-06T05:14:13.365106+0000","172.21.15.40:6832/1029013337":"2019-12-06T05:44:03.952842+0000","172.21.15.40:6833/2055706790":"2019-12-06T06:33:25.051960+0000","172.21.15.42:6834/613535662":"2019-12-06T06:04:04.415621+0000","172.21.15.40:6833/2146240383":"2019-12-06T05:49:19.088366+0000","172.21.15.40:6832/2780168808":"2019-12-06T06:01:39.357366+0000","172.21.15.40:6833/1029013337":"2019-12-06T05:44:03.952842+0000","172.21.15.42:6834/3028851337":"2019-12-06T05:31:48.711252+0000","172.21.15.42:6835/3923117701":"2019-12-06T06:12:09.582993+0000","172.21.15.42:6834/358497296":"2019-12-06T05:44:43.965329+0000","172.21.15.42:6835/358497296":"2019-12-06T05:44:43.965329+0000","172.21.15.40:6832/1127133319":"2019-12-06T05:45:39.022579+0000","172.21.15.40:6833/1127133319":"2019-12-06T05:45:39.022579+0000","172.21.15.42:6835/5991897":"2019-12-06T05:52:59.155806+0000","172.21.15.40:6832/2081111170":"2019-12-06T05:51:19.125597+0000","172.21.15.40:6833/2081111170":"2019-12-06T05:51:19.125597+0000","172.21.15.40:6832/1523046776":"2019-12-06T05:54:59.208157+0000","172.21.15.40:6833/1523046776":"2019-12-06T05:54:59.208157+0000","172.21.15.42:6835/3838877789":"2019-12-06T05:39:08.862108+0000","172.21.15.42:6834/2511123581":"2019-12-06T05:55:39.220889+0000","172.21.15.40:6832/3168718372":"2019-12-06T05:30:48.693050+0000","172.21.15.42:6835/2511123581":"2019-12-06T05:55:39.220889+0000","172.21.15.40:6833/3549993215":"2019-12-06T05:57:24.253912+0000","172.21.15.42:6834/1368848584":"2019-12-06T05:59:44.294812+0000","172.21.15.42:6835/1985322937":"2019-12-06T06:31:14.985625+0000","172.21.15.40:6832/1082666540":"2019-12-06T06:00:14.304550+0000","172.21.15.40:6833/1082666540":"2019-12-06T06:00:14.304550+0000","172.21.15.40:6833/2780168808":"2019-12-06T06:01:39.357366+0000","172.21.15.40:6832/2968227232":"2019-12-06T06:08:09.492099+0000","172.21.15.42:6835/2253267554":"2019-12-06T06:06:59.469462+0000","172.21.15.40:6833/2968227232":"2019-12-06T06:08:09.492099+0000","172.21.15.40:6832/3355679996":"2019-12-06T06:11:29.570475+0000","172.21.15.40:6832/3549993215":"2019-12-06T05:57:24.253912+0000","172.21.15.40:6833/3355679996":"2019-12-06T06:11:29.570475+0000"},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":528,"length":30}]}],"new_removed_snaps":[{"pool":3,"snaps":[{"begin":554,"length":4}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2019-12-05T06:51:38.580 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.574+0000 7f3bc95bd700 1 -- 172.21.15.42:0/2215796070 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3ba403b4c0 msgr2=0x7f3ba403d9d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:38.580 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.574+0000 7f3bc95bd700 1 --2- 172.21.15.42:0/2215796070 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3ba403b4c0 0x7f3ba403d9d0 crc :-1 s=READY pgs=1407 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:51:38.580 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.574+0000 7f3bc95bd700 1 -- 172.21.15.42:0/2215796070 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3bc4118080 msgr2=0x7f3bc41156d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:38.580 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.574+0000 7f3bc95bd700 1 --2- 172.21.15.42:0/2215796070 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3bc4118080 0x7f3bc41156d0 secure :-1 s=READY pgs=741 cs=0 l=1 rx=0x7f3bb4002f30 tx=0x7f3bb40108b0).stop 2019-12-05T06:51:38.580 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.574+0000 7f3bc95bd700 1 -- 172.21.15.42:0/2215796070 shutdown_connections 2019-12-05T06:51:38.580 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.574+0000 7f3bc95bd700 1 --2- 172.21.15.42:0/2215796070 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f3ba403b4c0 0x7f3ba403d9d0 unknown :-1 s=CLOSED pgs=1407 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:51:38.581 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.574+0000 7f3bc95bd700 1 --2- 172.21.15.42:0/2215796070 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3bc4118080 0x7f3bc41156d0 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:51:38.581 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.574+0000 7f3bc95bd700 1 -- 172.21.15.42:0/2215796070 shutdown_connections 2019-12-05T06:51:38.581 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.574+0000 7f3bc95bd700 1 -- 172.21.15.42:0/2215796070 wait complete. 2019-12-05T06:51:38.581 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.574+0000 7f3bc95bd700 1 -- 172.21.15.42:0/2215796070 >> 172.21.15.42:0/2215796070 conn(0x7f3bc4066fe0 msgr2=0x7f3bc40a9790 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:51:38.600 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2019-12-05T06:51:38.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.698+0000 7fd17e3d6700 1 Processor -- start 2019-12-05T06:51:38.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.698+0000 7fd17e3d6700 1 -- start start 2019-12-05T06:51:38.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.698+0000 7fd17e3d6700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd1780a7730 0x7fd17808df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:51:38.711 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.698+0000 7fd17e3d6700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1780a0700 con 0x7fd17807e710 2019-12-05T06:51:38.711 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.698+0000 7fd17e3d6700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1780a9b90 con 0x7fd17809fad0 2019-12-05T06:51:38.711 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.698+0000 7fd17e3d6700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1780a0af0 con 0x7fd1780a7730 2019-12-05T06:51:38.712 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.698+0000 7fd17c973700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd1780a7730 0x7fd17808df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:51:38.712 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.698+0000 7fd17c973700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd1780a7730 0x7fd17808df70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:39224/0 (socket says 172.21.15.42:39224) 2019-12-05T06:51:38.712 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.698+0000 7fd17c973700 1 -- 172.21.15.42:0/2835439923 learned_addr learned my addr 172.21.15.42:0/2835439923 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:51:38.713 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.698+0000 7fd17d174700 1 --1- 172.21.15.42:0/2835439923 >> v1:172.21.15.42:6789/0 conn(0x7fd17809fad0 0x7fd17814efd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:39264/0 (socket says 172.21.15.42:39264) 2019-12-05T06:51:38.713 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.702+0000 7fd17c973700 1 -- 172.21.15.42:0/2835439923 >> v1:172.21.15.42:6789/0 conn(0x7fd17809fad0 legacy=0x7fd17814efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:38.713 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.702+0000 7fd17c973700 1 -- 172.21.15.42:0/2835439923 >> v1:172.21.15.40:6789/0 conn(0x7fd17807e710 legacy=0x7fd17809f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:38.713 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.702+0000 7fd1777fe700 1 -- 172.21.15.42:0/2835439923 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3892063447 0 0) 0x7fd1780a0700 con 0x7fd17807e710 2019-12-05T06:51:38.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.702+0000 7fd17c973700 1 -- 172.21.15.42:0/2835439923 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd17807eb40 con 0x7fd1780a7730 2019-12-05T06:51:38.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.702+0000 7fd17c973700 1 --2- 172.21.15.42:0/2835439923 >> v2:172.21.15.42:3300/0 conn(0x7fd1780a7730 0x7fd17808df70 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fd160008790 tx=0x7fd16000fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:38.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.702+0000 7fd17c973700 1 --2- 172.21.15.42:0/2835439923 >> v2:172.21.15.42:3300/0 conn(0x7fd1780a7730 0x7fd17808df70 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fd160008790 tx=0x7fd16000fcb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:38.714 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.702+0000 7fd17c973700 1 --2- 172.21.15.42:0/2835439923 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd1780a7730 0x7fd17808df70 secure :-1 s=READY pgs=3361 cs=0 l=1 rx=0x7fd160008790 tx=0x7fd16000fcb0).ready entity=mon.1 client_cookie=f895ae3a73701d88 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:51:38.715 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.702+0000 7fd17c973700 1 -- 172.21.15.42:0/2835439923 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd1780a7730 msgr2=0x7fd17808df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2019-12-05T06:51:38.715 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.702+0000 7fd17c973700 1 -- 172.21.15.42:0/2835439923 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd1780a7730 msgr2=0x7fd17808df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2019-12-05T06:51:38.715 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.702+0000 7fd17c973700 1 --2- 172.21.15.42:0/2835439923 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd1780a7730 0x7fd17808df70 secure :-1 s=READY pgs=3361 cs=0 l=1 rx=0x7fd160008790 tx=0x7fd16000fcb0).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-12-05T06:51:38.715 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.702+0000 7fd17c973700 1 --2- 172.21.15.42:0/2835439923 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd1780a7730 0x7fd17808df70 secure :-1 s=READY pgs=3361 cs=0 l=1 rx=0x7fd160008790 tx=0x7fd16000fcb0).stop 2019-12-05T06:51:38.716 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.702+0000 7fd1777fe700 1 -- 172.21.15.42:0/2835439923 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd1780a7730 msgr2=0x7fd17808df70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2019-12-05T06:51:38.716 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.702+0000 7fd1777fe700 1 --2- 172.21.15.42:0/2835439923 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd1780a7730 0x7fd17808df70 unknown :-1 s=CLOSED pgs=3361 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:51:38.716 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.702+0000 7fd1777fe700 1 --2- 172.21.15.42:0/2835439923 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd164003050 0x7fd1640054f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:51:38.716 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.702+0000 7fd1777fe700 1 --2- 172.21.15.42:0/2835439923 >> v2:172.21.15.40:3300/0 conn(0x7fd164005b80 0x7fd164008090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:51:38.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.702+0000 7fd1777fe700 1 -- 172.21.15.42:0/2835439923 --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1640087b0 con 0x7fd164005b80 2019-12-05T06:51:38.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.702+0000 7fd1777fe700 1 -- 172.21.15.42:0/2835439923 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1640089a0 con 0x7fd164003050 2019-12-05T06:51:38.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.702+0000 7fd17c973700 1 --2- 172.21.15.42:0/2835439923 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd164003050 0x7fd1640054f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:51:38.717 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.702+0000 7fd177fff700 1 --2- 172.21.15.42:0/2835439923 >> v2:172.21.15.40:3300/0 conn(0x7fd164005b80 0x7fd164008090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:51:38.718 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.702+0000 7fd17c973700 1 -- 172.21.15.42:0/2835439923 >> v2:172.21.15.40:3300/0 conn(0x7fd164005b80 msgr2=0x7fd164008090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:51:38.718 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.702+0000 7fd17c973700 1 --2- 172.21.15.42:0/2835439923 >> v2:172.21.15.40:3300/0 conn(0x7fd164005b80 0x7fd164008090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:51:38.718 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.702+0000 7fd17c973700 1 -- 172.21.15.42:0/2835439923 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd16404cc60 con 0x7fd164003050 2019-12-05T06:51:38.718 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.702+0000 7fd17c973700 1 --2- 172.21.15.42:0/2835439923 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd164003050 0x7fd1640054f0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fd160008790 tx=0x7fd16000f540).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:38.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.702+0000 7fd17c973700 1 --2- 172.21.15.42:0/2835439923 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd164003050 0x7fd1640054f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fd160008790 tx=0x7fd16000f540).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:38.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.702+0000 7fd17c973700 1 --2- 172.21.15.42:0/2835439923 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd164003050 0x7fd1640054f0 secure :-1 s=READY pgs=742 cs=0 l=1 rx=0x7fd160008790 tx=0x7fd16000f540).ready entity=mon.2 client_cookie=a518195d8df4e410 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:51:38.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.702+0000 7fd17c973700 1 --2- 172.21.15.42:0/2835439923 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd164003050 0x7fd1640054f0 secure :-1 s=THROTTLE_DONE pgs=742 cs=0 l=1 rx=0x7fd160008790 tx=0x7fd16000f540).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:38.719 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.702+0000 7fd1777fe700 1 -- 172.21.15.42:0/2835439923 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd160013650 con 0x7fd164003050 2019-12-05T06:51:38.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.702+0000 7fd17c973700 1 --2- 172.21.15.42:0/2835439923 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd164003050 0x7fd1640054f0 secure :-1 s=THROTTLE_DONE pgs=742 cs=0 l=1 rx=0x7fd160008790 tx=0x7fd16000f540).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:38.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.702+0000 7fd1777fe700 1 -- 172.21.15.42:0/2835439923 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1600110c0 con 0x7fd164003050 2019-12-05T06:51:38.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.702+0000 7fd17c973700 1 --2- 172.21.15.42:0/2835439923 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd164003050 0x7fd1640054f0 secure :-1 s=THROTTLE_DONE pgs=742 cs=0 l=1 rx=0x7fd160008790 tx=0x7fd16000f540).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:38.720 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.702+0000 7fd1777fe700 1 -- 172.21.15.42:0/2835439923 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd160013c30 con 0x7fd164003050 2019-12-05T06:51:38.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.702+0000 7fd17e3d6700 1 -- 172.21.15.42:0/2835439923 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd164003050 msgr2=0x7fd1640054f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:38.721 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.702+0000 7fd17e3d6700 1 --2- 172.21.15.42:0/2835439923 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd164003050 0x7fd1640054f0 secure :-1 s=READY pgs=742 cs=0 l=1 rx=0x7fd160008790 tx=0x7fd16000f540).stop 2019-12-05T06:51:38.724 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.702+0000 7fd17d174700 1 -- 172.21.15.42:0/2835439923 reap_dead start 2019-12-05T06:51:38.725 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.702+0000 7fd17e3d6700 1 -- 172.21.15.42:0/2835439923 shutdown_connections 2019-12-05T06:51:38.725 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.706+0000 7fd17e3d6700 1 --2- 172.21.15.42:0/2835439923 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd1780a7730 0x7fd17808df70 unknown :-1 s=CLOSED pgs=3361 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:51:38.725 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.706+0000 7fd17e3d6700 1 -- 172.21.15.42:0/2835439923 shutdown_connections 2019-12-05T06:51:38.725 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.706+0000 7fd17e3d6700 1 -- 172.21.15.42:0/2835439923 wait complete. 2019-12-05T06:51:38.726 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.706+0000 7fd17e3d6700 1 -- 172.21.15.42:0/2835439923 >> 172.21.15.42:0/2835439923 conn(0x7fd1780aabd0 msgr2=0x7fd1780a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:51:38.729 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.706+0000 7fd17e3d6700 1 Processor -- start 2019-12-05T06:51:38.729 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.706+0000 7fd17e3d6700 1 -- start start 2019-12-05T06:51:38.729 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.710+0000 7fd17e3d6700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd178117f40 0x7fd178118ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:51:38.730 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.710+0000 7fd17e3d6700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1781193f0 con 0x7fd17811f420 2019-12-05T06:51:38.730 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.710+0000 7fd17e3d6700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd178120050 con 0x7fd1781165f0 2019-12-05T06:51:38.730 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.710+0000 7fd17e3d6700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd17811e820 con 0x7fd178117f40 2019-12-05T06:51:38.730 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.710+0000 7fd17c973700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fd1781165f0 0x7fd1781186a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:39270/0 (socket says 172.21.15.42:39270) 2019-12-05T06:51:38.730 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.710+0000 7fd17c973700 1 -- 172.21.15.42:0/2730346498 learned_addr learned my addr 172.21.15.42:0/2730346498 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:51:38.730 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.710+0000 7fd177fff700 1 --2- 172.21.15.42:0/2730346498 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd178117f40 0x7fd178118ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:51:38.730 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.710+0000 7fd1757fa700 1 -- 172.21.15.42:0/2730346498 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2878942039 0 0) 0x7fd178120050 con 0x7fd1781165f0 2019-12-05T06:51:38.731 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.710+0000 7fd1757fa700 1 -- 172.21.15.42:0/2730346498 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd15c0030b0 con 0x7fd1781165f0 2019-12-05T06:51:38.731 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.710+0000 7fd1757fa700 1 -- 172.21.15.42:0/2730346498 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2420963937 0 0) 0x7fd1781193f0 con 0x7fd17811f420 2019-12-05T06:51:38.731 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.710+0000 7fd1757fa700 1 -- 172.21.15.42:0/2730346498 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd178120050 con 0x7fd17811f420 2019-12-05T06:51:38.731 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.710+0000 7fd177fff700 1 -- 172.21.15.42:0/2730346498 >> v1:172.21.15.42:6789/0 conn(0x7fd1781165f0 legacy=0x7fd1781186a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:38.731 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.710+0000 7fd177fff700 1 -- 172.21.15.42:0/2730346498 >> v1:172.21.15.40:6789/0 conn(0x7fd17811f420 legacy=0x7fd17811f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:38.731 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.710+0000 7fd177fff700 1 -- 172.21.15.42:0/2730346498 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd178120280 con 0x7fd178117f40 2019-12-05T06:51:38.731 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.710+0000 7fd177fff700 1 --2- 172.21.15.42:0/2730346498 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd178117f40 0x7fd178118ea0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fd168001380 tx=0x7fd168010f30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:38.732 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.710+0000 7fd177fff700 1 --2- 172.21.15.42:0/2730346498 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd178117f40 0x7fd178118ea0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fd168001380 tx=0x7fd168010f30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:38.732 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.710+0000 7fd177fff700 1 --2- 172.21.15.42:0/2730346498 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd178117f40 0x7fd178118ea0 secure :-1 s=READY pgs=743 cs=0 l=1 rx=0x7fd168001380 tx=0x7fd168010f30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:51:38.732 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.710+0000 7fd177fff700 1 --2- 172.21.15.42:0/2730346498 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd178117f40 0x7fd178118ea0 secure :-1 s=THROTTLE_DONE pgs=743 cs=0 l=1 rx=0x7fd168001380 tx=0x7fd168010f30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:38.732 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.710+0000 7fd1757fa700 1 -- 172.21.15.42:0/2730346498 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd168012dc0 con 0x7fd178117f40 2019-12-05T06:51:38.732 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.710+0000 7fd177fff700 1 --2- 172.21.15.42:0/2730346498 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd178117f40 0x7fd178118ea0 secure :-1 s=THROTTLE_DONE pgs=743 cs=0 l=1 rx=0x7fd168001380 tx=0x7fd168010f30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:38.732 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.710+0000 7fd177fff700 1 --2- 172.21.15.42:0/2730346498 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd178117f40 0x7fd178118ea0 secure :-1 s=THROTTLE_DONE pgs=743 cs=0 l=1 rx=0x7fd168001380 tx=0x7fd168010f30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:38.733 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.710+0000 7fd1757fa700 1 -- 172.21.15.42:0/2730346498 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1680129e0 con 0x7fd178117f40 2019-12-05T06:51:38.733 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.710+0000 7fd17e3d6700 1 -- 172.21.15.42:0/2730346498 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd17808eb60 con 0x7fd178117f40 2019-12-05T06:51:38.733 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.710+0000 7fd1757fa700 1 -- 172.21.15.42:0/2730346498 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd1680034c0 con 0x7fd178117f40 2019-12-05T06:51:38.733 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.710+0000 7fd17e3d6700 1 -- 172.21.15.42:0/2730346498 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd17808f060 con 0x7fd178117f40 2019-12-05T06:51:38.733 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.714+0000 7fd177fff700 1 --2- 172.21.15.42:0/2730346498 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd178117f40 0x7fd178118ea0 secure :-1 s=THROTTLE_DONE pgs=743 cs=0 l=1 rx=0x7fd168001380 tx=0x7fd168010f30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:38.733 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.714+0000 7fd1757fa700 1 -- 172.21.15.42:0/2730346498 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fd1680037a0 con 0x7fd178117f40 2019-12-05T06:51:38.733 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.714+0000 7fd177fff700 1 --2- 172.21.15.42:0/2730346498 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd178117f40 0x7fd178118ea0 secure :-1 s=THROTTLE_DONE pgs=743 cs=0 l=1 rx=0x7fd168001380 tx=0x7fd168010f30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:38.734 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.714+0000 7fd1757fa700 1 --2- 172.21.15.42:0/2730346498 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd15c03c550 0x7fd15c03ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:51:38.734 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.714+0000 7fd1757fa700 1 -- 172.21.15.42:0/2730346498 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 12365+0+0 (secure 0 0 0) 0x7fd16804ae90 con 0x7fd178117f40 2019-12-05T06:51:38.734 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.714+0000 7fd17c973700 1 --2- 172.21.15.42:0/2730346498 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd15c03c550 0x7fd15c03ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:51:38.734 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.714+0000 7fd17c973700 1 --2- 172.21.15.42:0/2730346498 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd15c03c550 0x7fd15c03ea60 crc :-1 s=READY pgs=1408 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:51:38.734 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.714+0000 7fd17e3d6700 1 -- 172.21.15.42:0/2730346498 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd17805d100 con 0x7fd178117f40 2019-12-05T06:51:38.734 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.722+0000 7fd177fff700 1 --2- 172.21.15.42:0/2730346498 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd178117f40 0x7fd178118ea0 secure :-1 s=THROTTLE_DONE pgs=743 cs=0 l=1 rx=0x7fd168001380 tx=0x7fd168010f30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:38.735 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.722+0000 7fd1757fa700 1 -- 172.21.15.42:0/2730346498 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fd168032170 con 0x7fd178117f40 2019-12-05T06:51:38.902 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.898+0000 7fd17e3d6700 1 -- 172.21.15.42:0/2730346498 --> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd17805b8b0 con 0x7fd15c03c550 2019-12-05T06:51:38.910 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.906+0000 7fd1757fa700 1 -- 172.21.15.42:0/2730346498 <== mgr.4108 v2:172.21.15.42:6800/11930 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+92129 (crc 0 0 0) 0x7fd17805b8b0 con 0x7fd15c03c550 2019-12-05T06:51:38.927 INFO:teuthology.orchestra.run.smithi042.stderr:dumped all 2019-12-05T06:51:38.927 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.918+0000 7fd17e3d6700 1 -- 172.21.15.42:0/2730346498 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd15c03c550 msgr2=0x7fd15c03ea60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:38.927 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.918+0000 7fd17e3d6700 1 --2- 172.21.15.42:0/2730346498 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd15c03c550 0x7fd15c03ea60 crc :-1 s=READY pgs=1408 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:51:38.927 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.918+0000 7fd17e3d6700 1 -- 172.21.15.42:0/2730346498 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd178117f40 msgr2=0x7fd178118ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:38.928 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.918+0000 7fd17e3d6700 1 --2- 172.21.15.42:0/2730346498 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd178117f40 0x7fd178118ea0 secure :-1 s=READY pgs=743 cs=0 l=1 rx=0x7fd168001380 tx=0x7fd168010f30).stop 2019-12-05T06:51:38.928 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.918+0000 7fd17e3d6700 1 -- 172.21.15.42:0/2730346498 shutdown_connections 2019-12-05T06:51:38.928 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.918+0000 7fd17e3d6700 1 --2- 172.21.15.42:0/2730346498 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fd15c03c550 0x7fd15c03ea60 unknown :-1 s=CLOSED pgs=1408 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:51:38.928 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.918+0000 7fd17e3d6700 1 --2- 172.21.15.42:0/2730346498 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd178117f40 0x7fd178118ea0 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:51:38.929 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.918+0000 7fd17e3d6700 1 -- 172.21.15.42:0/2730346498 shutdown_connections 2019-12-05T06:51:38.929 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.918+0000 7fd17e3d6700 1 -- 172.21.15.42:0/2730346498 wait complete. 2019-12-05T06:51:38.929 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:38.918+0000 7fd17e3d6700 1 -- 172.21.15.42:0/2730346498 >> 172.21.15.42:0/2730346498 conn(0x7fd17805a180 msgr2=0x7fd17808f3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:51:38.929 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:51:38.951 INFO:teuthology.orchestra.run.smithi042.stdout:{"pg_ready":true,"pg_map":{"version":3374,"stamp":"2019-12-05T06:51:37.614799+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":412667787,"num_objects":150,"num_object_clones":0,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":150,"num_whiteouts":0,"num_read":9489,"num_read_kb":12806378,"num_write":49932,"num_write_kb":1020810,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39667,"ondisk_log_size":39667,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":9208240,"kb_used_data":819568,"kb_used_omap":0,"kb_used_meta":8388608,"kb_avail":745766480,"statfs":{"total":773094113280,"available":763664875520,"internally_reserved":8589934592,"allocated":839237632,"data_stored":833414953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8589934592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":369308,"num_objects":-2,"num_object_clones":-1,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":-1,"num_read":1,"num_read_kb":1,"num_write":24,"num_write_kb":376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.163494"},"pg_stats":[{"pgid":"3.7","version":"124'6395","reported_seq":"8167","reported_epoch":"126","state":"active+clean","last_fresh":"2019-12-05T06:51:32.150271+0000","last_change":"2019-12-05T06:51:32.151497+0000","last_active":"2019-12-05T06:51:32.150271+0000","last_peered":"2019-12-05T06:51:32.150271+0000","last_clean":"2019-12-05T06:51:32.150271+0000","last_became_active":"2019-12-05T05:03:35.332717+0000","last_became_peered":"2019-12-05T05:03:35.332717+0000","last_unstale":"2019-12-05T06:51:32.150271+0000","last_undegraded":"2019-12-05T06:51:32.150271+0000","last_fullsized":"2019-12-05T06:51:32.150271+0000","mapping_epoch":21,"log_start":"118'3311","ondisk_log_start":"118'3311","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:34.235090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:34.235090+0000","last_clean_scrub_stamp":"2019-12-05T05:03:34.235090+0000","log_size":3084,"ondisk_log_size":3084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":245,"num_read_kb":1968,"num_write":4375,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7,0],"acting":[3,5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"210","length":"1a"}]},{"pgid":"1.5","version":"0'0","reported_seq":"118","reported_epoch":"125","state":"active+clean","last_fresh":"2019-12-05T06:51:30.436732+0000","last_change":"2019-12-05T05:03:29.184885+0000","last_active":"2019-12-05T06:51:30.436732+0000","last_peered":"2019-12-05T06:51:30.436732+0000","last_clean":"2019-12-05T06:51:30.436732+0000","last_became_active":"2019-12-05T05:03:29.184431+0000","last_became_peered":"2019-12-05T05:03:29.184431+0000","last_unstale":"2019-12-05T06:51:30.436732+0000","last_undegraded":"2019-12-05T06:51:30.436732+0000","last_fullsized":"2019-12-05T06:51:30.436732+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:28.161818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:28.161818+0000","last_clean_scrub_stamp":"2019-12-05T05:03:28.161818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"125'2438","reported_seq":"3004","reported_epoch":"125","state":"active+clean","last_fresh":"2019-12-05T06:51:31.171975+0000","last_change":"2019-12-05T05:03:32.225175+0000","last_active":"2019-12-05T06:51:31.171975+0000","last_peered":"2019-12-05T06:51:31.171975+0000","last_clean":"2019-12-05T06:51:31.171975+0000","last_became_active":"2019-12-05T05:03:32.224827+0000","last_became_peered":"2019-12-05T05:03:32.224827+0000","last_unstale":"2019-12-05T06:51:31.171975+0000","last_undegraded":"2019-12-05T06:51:31.171975+0000","last_fullsized":"2019-12-05T06:51:31.171975+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:31.200243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:31.200243+0000","last_clean_scrub_stamp":"2019-12-05T05:03:31.200243+0000","log_size":2438,"ondisk_log_size":2438,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":67108864,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":679,"num_read_kb":1201860,"num_write":2584,"num_write_kb":72174,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"124'6219","reported_seq":"8082","reported_epoch":"126","state":"active+clean","last_fresh":"2019-12-05T06:51:32.415412+0000","last_change":"2019-12-05T06:51:32.428623+0000","last_active":"2019-12-05T06:51:32.415412+0000","last_peered":"2019-12-05T06:51:32.415412+0000","last_clean":"2019-12-05T06:51:32.415412+0000","last_became_active":"2019-12-05T05:03:35.332648+0000","last_became_peered":"2019-12-05T05:03:35.332648+0000","last_unstale":"2019-12-05T06:51:32.415412+0000","last_undegraded":"2019-12-05T06:51:32.415412+0000","last_fullsized":"2019-12-05T06:51:32.415412+0000","mapping_epoch":21,"log_start":"117'3205","ondisk_log_start":"117'3205","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:34.235090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:34.235090+0000","last_clean_scrub_stamp":"2019-12-05T05:03:34.235090+0000","log_size":3014,"ondisk_log_size":3014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":233,"num_read_kb":6040,"num_write":4390,"num_write_kb":51866,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,7],"acting":[0,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"210","length":"1a"}]},{"pgid":"1.4","version":"0'0","reported_seq":"118","reported_epoch":"125","state":"active+clean","last_fresh":"2019-12-05T06:51:30.436614+0000","last_change":"2019-12-05T05:03:29.182769+0000","last_active":"2019-12-05T06:51:30.436614+0000","last_peered":"2019-12-05T06:51:30.436614+0000","last_clean":"2019-12-05T06:51:30.436614+0000","last_became_active":"2019-12-05T05:03:29.182409+0000","last_became_peered":"2019-12-05T05:03:29.182409+0000","last_unstale":"2019-12-05T06:51:30.436614+0000","last_undegraded":"2019-12-05T06:51:30.436614+0000","last_fullsized":"2019-12-05T06:51:30.436614+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:28.161818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:28.161818+0000","last_clean_scrub_stamp":"2019-12-05T05:03:28.161818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"125'1441","reported_seq":"2093","reported_epoch":"125","state":"active+clean","last_fresh":"2019-12-05T06:51:30.464688+0000","last_change":"2019-12-05T05:03:32.228176+0000","last_active":"2019-12-05T06:51:30.464688+0000","last_peered":"2019-12-05T06:51:30.464688+0000","last_clean":"2019-12-05T06:51:30.464688+0000","last_became_active":"2019-12-05T05:03:32.227690+0000","last_became_peered":"2019-12-05T05:03:32.227690+0000","last_unstale":"2019-12-05T06:51:30.464688+0000","last_undegraded":"2019-12-05T06:51:30.464688+0000","last_fullsized":"2019-12-05T06:51:30.464688+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:31.200243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:31.200243+0000","last_clean_scrub_stamp":"2019-12-05T05:03:31.200243+0000","log_size":1441,"ondisk_log_size":1441,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":25166390,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":832,"num_read_kb":914148,"num_write":1232,"num_write_kb":29663,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"124'6680","reported_seq":"8434","reported_epoch":"126","state":"active+clean","last_fresh":"2019-12-05T06:51:32.413658+0000","last_change":"2019-12-05T06:51:32.428649+0000","last_active":"2019-12-05T06:51:32.413658+0000","last_peered":"2019-12-05T06:51:32.413658+0000","last_clean":"2019-12-05T06:51:32.413658+0000","last_became_active":"2019-12-05T05:03:35.266235+0000","last_became_peered":"2019-12-05T05:03:35.266235+0000","last_unstale":"2019-12-05T06:51:32.413658+0000","last_undegraded":"2019-12-05T06:51:32.413658+0000","last_fullsized":"2019-12-05T06:51:32.413658+0000","mapping_epoch":21,"log_start":"118'3617","ondisk_log_start":"118'3617","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:34.235090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:34.235090+0000","last_clean_scrub_stamp":"2019-12-05T05:03:34.235090+0000","log_size":3063,"ondisk_log_size":3063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":237,"num_read_kb":18614,"num_write":4619,"num_write_kb":74596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3,5],"acting":[0,4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"210","length":"1a"}]},{"pgid":"1.3","version":"0'0","reported_seq":"118","reported_epoch":"125","state":"active+clean","last_fresh":"2019-12-05T06:51:30.437111+0000","last_change":"2019-12-05T05:03:29.181115+0000","last_active":"2019-12-05T06:51:30.437111+0000","last_peered":"2019-12-05T06:51:30.437111+0000","last_clean":"2019-12-05T06:51:30.437111+0000","last_became_active":"2019-12-05T05:03:29.180832+0000","last_became_peered":"2019-12-05T05:03:29.180832+0000","last_unstale":"2019-12-05T06:51:30.437111+0000","last_undegraded":"2019-12-05T06:51:30.437111+0000","last_fullsized":"2019-12-05T06:51:30.437111+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:28.161818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:28.161818+0000","last_clean_scrub_stamp":"2019-12-05T05:03:28.161818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"124'2400","reported_seq":"3412","reported_epoch":"125","state":"active+clean","last_fresh":"2019-12-05T06:51:30.465449+0000","last_change":"2019-12-05T05:03:32.227259+0000","last_active":"2019-12-05T06:51:30.465449+0000","last_peered":"2019-12-05T06:51:30.465449+0000","last_clean":"2019-12-05T06:51:30.465449+0000","last_became_active":"2019-12-05T05:03:32.226895+0000","last_became_peered":"2019-12-05T05:03:32.226895+0000","last_unstale":"2019-12-05T06:51:30.465449+0000","last_undegraded":"2019-12-05T06:51:30.465449+0000","last_fullsized":"2019-12-05T06:51:30.465449+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:31.200243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:31.200243+0000","last_clean_scrub_stamp":"2019-12-05T05:03:31.200243+0000","log_size":2400,"ondisk_log_size":2400,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":50331648,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":1503,"num_read_kb":2307020,"num_write":2198,"num_write_kb":62851,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":11,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"124'6690","reported_seq":"8454","reported_epoch":"126","state":"active+clean","last_fresh":"2019-12-05T06:51:32.412203+0000","last_change":"2019-12-05T06:51:32.422135+0000","last_active":"2019-12-05T06:51:32.412203+0000","last_peered":"2019-12-05T06:51:32.412203+0000","last_clean":"2019-12-05T06:51:32.412203+0000","last_became_active":"2019-12-05T05:03:35.627352+0000","last_became_peered":"2019-12-05T05:03:35.627352+0000","last_unstale":"2019-12-05T06:51:32.412203+0000","last_undegraded":"2019-12-05T06:51:32.412203+0000","last_fullsized":"2019-12-05T06:51:32.412203+0000","mapping_epoch":21,"log_start":"118'3612","ondisk_log_start":"118'3612","created":21,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:34.235090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:34.235090+0000","last_clean_scrub_stamp":"2019-12-05T05:03:34.235090+0000","log_size":3078,"ondisk_log_size":3078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":249,"num_read_kb":14990,"num_write":4595,"num_write_kb":60350,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6,5],"acting":[1,2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"210","length":"1a"}]},{"pgid":"1.2","version":"0'0","reported_seq":"118","reported_epoch":"125","state":"active+clean","last_fresh":"2019-12-05T06:51:30.465437+0000","last_change":"2019-12-05T05:03:29.408129+0000","last_active":"2019-12-05T06:51:30.465437+0000","last_peered":"2019-12-05T06:51:30.465437+0000","last_clean":"2019-12-05T06:51:30.465437+0000","last_became_active":"2019-12-05T05:03:29.407108+0000","last_became_peered":"2019-12-05T05:03:29.407108+0000","last_unstale":"2019-12-05T06:51:30.465437+0000","last_undegraded":"2019-12-05T06:51:30.465437+0000","last_fullsized":"2019-12-05T06:51:30.465437+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:28.161818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:28.161818+0000","last_clean_scrub_stamp":"2019-12-05T05:03:28.161818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"125'1572","reported_seq":"2602","reported_epoch":"125","state":"active+clean","last_fresh":"2019-12-05T06:51:30.457097+0000","last_change":"2019-12-05T05:03:32.636592+0000","last_active":"2019-12-05T06:51:30.457097+0000","last_peered":"2019-12-05T06:51:30.457097+0000","last_clean":"2019-12-05T06:51:30.457097+0000","last_became_active":"2019-12-05T05:03:32.635937+0000","last_became_peered":"2019-12-05T05:03:32.635937+0000","last_unstale":"2019-12-05T06:51:30.457097+0000","last_undegraded":"2019-12-05T06:51:30.457097+0000","last_fullsized":"2019-12-05T06:51:30.457097+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:31.200243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:31.200243+0000","last_clean_scrub_stamp":"2019-12-05T05:03:31.200243+0000","log_size":1572,"ondisk_log_size":1572,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":35827610,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":918,"num_read_kb":599869,"num_write":1405,"num_write_kb":35964,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"126'6396","reported_seq":"8232","reported_epoch":"126","state":"active+clean","last_fresh":"2019-12-05T06:51:32.176031+0000","last_change":"2019-12-05T06:51:32.185952+0000","last_active":"2019-12-05T06:51:32.176031+0000","last_peered":"2019-12-05T06:51:32.176031+0000","last_clean":"2019-12-05T06:51:32.176031+0000","last_became_active":"2019-12-05T05:03:36.314006+0000","last_became_peered":"2019-12-05T05:03:36.314006+0000","last_unstale":"2019-12-05T06:51:32.176031+0000","last_undegraded":"2019-12-05T06:51:32.176031+0000","last_fullsized":"2019-12-05T06:51:32.176031+0000","mapping_epoch":21,"log_start":"117'3307","ondisk_log_start":"117'3307","created":21,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:34.235090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:34.235090+0000","last_clean_scrub_stamp":"2019-12-05T05:03:34.235090+0000","log_size":3089,"ondisk_log_size":3089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":253,"num_read_kb":6039,"num_write":4487,"num_write_kb":82964,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6,4],"acting":[3,5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"210","length":"1a"}]},{"pgid":"1.0","version":"0'0","reported_seq":"118","reported_epoch":"125","state":"active+clean","last_fresh":"2019-12-05T06:51:30.465574+0000","last_change":"2019-12-05T05:03:29.407882+0000","last_active":"2019-12-05T06:51:30.465574+0000","last_peered":"2019-12-05T06:51:30.465574+0000","last_clean":"2019-12-05T06:51:30.465574+0000","last_became_active":"2019-12-05T05:03:29.406770+0000","last_became_peered":"2019-12-05T05:03:29.406770+0000","last_unstale":"2019-12-05T06:51:30.465574+0000","last_undegraded":"2019-12-05T06:51:30.465574+0000","last_fullsized":"2019-12-05T06:51:30.465574+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:28.161818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:28.161818+0000","last_clean_scrub_stamp":"2019-12-05T05:03:28.161818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"126'1767","reported_seq":"2585","reported_epoch":"126","state":"active+clean","last_fresh":"2019-12-05T06:51:33.558614+0000","last_change":"2019-12-05T05:03:32.227654+0000","last_active":"2019-12-05T06:51:33.558614+0000","last_peered":"2019-12-05T06:51:33.558614+0000","last_clean":"2019-12-05T06:51:33.558614+0000","last_became_active":"2019-12-05T05:03:32.227052+0000","last_became_peered":"2019-12-05T05:03:32.227052+0000","last_unstale":"2019-12-05T06:51:33.558614+0000","last_undegraded":"2019-12-05T06:51:33.558614+0000","last_fullsized":"2019-12-05T06:51:33.558614+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:31.200243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:31.200243+0000","last_clean_scrub_stamp":"2019-12-05T05:03:31.200243+0000","log_size":1767,"ondisk_log_size":1767,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":45487964,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":922,"num_read_kb":1904128,"num_write":1570,"num_write_kb":61737,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"124'6539","reported_seq":"8338","reported_epoch":"126","state":"active+clean","last_fresh":"2019-12-05T06:51:32.168256+0000","last_change":"2019-12-05T06:51:32.179014+0000","last_active":"2019-12-05T06:51:32.168256+0000","last_peered":"2019-12-05T06:51:32.168256+0000","last_clean":"2019-12-05T06:51:32.168256+0000","last_became_active":"2019-12-05T05:03:35.732405+0000","last_became_peered":"2019-12-05T05:03:35.732405+0000","last_unstale":"2019-12-05T06:51:32.168256+0000","last_undegraded":"2019-12-05T06:51:32.168256+0000","last_fullsized":"2019-12-05T06:51:32.168256+0000","mapping_epoch":21,"log_start":"118'3508","ondisk_log_start":"118'3508","created":21,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:34.235090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:34.235090+0000","last_clean_scrub_stamp":"2019-12-05T05:03:34.235090+0000","log_size":3031,"ondisk_log_size":3031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":262,"num_read_kb":13412,"num_write":4550,"num_write_kb":85060,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0,6],"acting":[4,5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"210","length":"1a"}]},{"pgid":"1.1","version":"0'0","reported_seq":"118","reported_epoch":"125","state":"active+clean","last_fresh":"2019-12-05T06:51:30.650248+0000","last_change":"2019-12-05T05:03:29.407539+0000","last_active":"2019-12-05T06:51:30.650248+0000","last_peered":"2019-12-05T06:51:30.650248+0000","last_clean":"2019-12-05T06:51:30.650248+0000","last_became_active":"2019-12-05T05:03:29.406903+0000","last_became_peered":"2019-12-05T05:03:29.406903+0000","last_unstale":"2019-12-05T06:51:30.650248+0000","last_undegraded":"2019-12-05T06:51:30.650248+0000","last_fullsized":"2019-12-05T06:51:30.650248+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:28.161818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:28.161818+0000","last_clean_scrub_stamp":"2019-12-05T05:03:28.161818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"124'1332","reported_seq":"2007","reported_epoch":"125","state":"active+clean","last_fresh":"2019-12-05T06:51:30.650629+0000","last_change":"2019-12-05T05:03:32.227838+0000","last_active":"2019-12-05T06:51:30.650629+0000","last_peered":"2019-12-05T06:51:30.650629+0000","last_clean":"2019-12-05T06:51:30.650629+0000","last_became_active":"2019-12-05T05:03:32.227050+0000","last_became_peered":"2019-12-05T05:03:32.227050+0000","last_unstale":"2019-12-05T06:51:30.650629+0000","last_undegraded":"2019-12-05T06:51:30.650629+0000","last_fullsized":"2019-12-05T06:51:30.650629+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:31.200243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:31.200243+0000","last_clean_scrub_stamp":"2019-12-05T05:03:31.200243+0000","log_size":1332,"ondisk_log_size":1332,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":67108864,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":576,"num_read_kb":2216360,"num_write":1199,"num_write_kb":74709,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"124'6466","reported_seq":"8232","reported_epoch":"126","state":"active+clean","last_fresh":"2019-12-05T06:51:32.412130+0000","last_change":"2019-12-05T06:51:32.426291+0000","last_active":"2019-12-05T06:51:32.412130+0000","last_peered":"2019-12-05T06:51:32.412130+0000","last_clean":"2019-12-05T06:51:32.412130+0000","last_became_active":"2019-12-05T05:03:35.266868+0000","last_became_peered":"2019-12-05T05:03:35.266868+0000","last_unstale":"2019-12-05T06:51:32.412130+0000","last_undegraded":"2019-12-05T06:51:32.412130+0000","last_fullsized":"2019-12-05T06:51:32.412130+0000","mapping_epoch":21,"log_start":"118'3409","ondisk_log_start":"118'3409","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:34.235090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:34.235090+0000","last_clean_scrub_stamp":"2019-12-05T05:03:34.235090+0000","log_size":3057,"ondisk_log_size":3057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":226,"num_read_kb":10307,"num_write":4503,"num_write_kb":69746,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0,5],"acting":[1,2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"210","length":"1a"}]},{"pgid":"1.6","version":"0'0","reported_seq":"117","reported_epoch":"125","state":"active+clean","last_fresh":"2019-12-05T06:51:30.460954+0000","last_change":"2019-12-05T05:03:29.407541+0000","last_active":"2019-12-05T06:51:30.460954+0000","last_peered":"2019-12-05T06:51:30.460954+0000","last_clean":"2019-12-05T06:51:30.460954+0000","last_became_active":"2019-12-05T05:03:29.406857+0000","last_became_peered":"2019-12-05T05:03:29.406857+0000","last_unstale":"2019-12-05T06:51:30.460954+0000","last_undegraded":"2019-12-05T06:51:30.460954+0000","last_fullsized":"2019-12-05T06:51:30.460954+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:28.161818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:28.161818+0000","last_clean_scrub_stamp":"2019-12-05T05:03:28.161818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"125'2881","reported_seq":"3847","reported_epoch":"125","state":"active+clean","last_fresh":"2019-12-05T06:51:30.470276+0000","last_change":"2019-12-05T05:03:32.480464+0000","last_active":"2019-12-05T06:51:30.470276+0000","last_peered":"2019-12-05T06:51:30.470276+0000","last_clean":"2019-12-05T06:51:30.470276+0000","last_became_active":"2019-12-05T05:03:32.479565+0000","last_became_peered":"2019-12-05T05:03:32.479565+0000","last_unstale":"2019-12-05T06:51:30.470276+0000","last_undegraded":"2019-12-05T06:51:30.470276+0000","last_fullsized":"2019-12-05T06:51:30.470276+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:31.200243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:31.200243+0000","last_clean_scrub_stamp":"2019-12-05T05:03:31.200243+0000","log_size":2881,"ondisk_log_size":2881,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":83887077,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":1148,"num_read_kb":2218837,"num_write":2637,"num_write_kb":91696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.5","version":"124'6269","reported_seq":"8128","reported_epoch":"126","state":"active+clean","last_fresh":"2019-12-05T06:51:31.452283+0000","last_change":"2019-12-05T06:51:31.453472+0000","last_active":"2019-12-05T06:51:31.452283+0000","last_peered":"2019-12-05T06:51:31.452283+0000","last_clean":"2019-12-05T06:51:31.452283+0000","last_became_active":"2019-12-05T05:03:35.629300+0000","last_became_peered":"2019-12-05T05:03:35.629300+0000","last_unstale":"2019-12-05T06:51:31.452283+0000","last_undegraded":"2019-12-05T06:51:31.452283+0000","last_fullsized":"2019-12-05T06:51:31.452283+0000","mapping_epoch":21,"log_start":"117'3209","ondisk_log_start":"117'3209","created":21,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:34.235090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:34.235090+0000","last_clean_scrub_stamp":"2019-12-05T05:03:34.235090+0000","log_size":3060,"ondisk_log_size":3060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":275,"num_read_kb":10198,"num_write":4396,"num_write_kb":76174,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2,6],"acting":[5,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"210","length":"1a"}]},{"pgid":"1.7","version":"0'0","reported_seq":"117","reported_epoch":"125","state":"active+clean","last_fresh":"2019-12-05T06:51:30.461106+0000","last_change":"2019-12-05T05:03:29.185245+0000","last_active":"2019-12-05T06:51:30.461106+0000","last_peered":"2019-12-05T06:51:30.461106+0000","last_clean":"2019-12-05T06:51:30.461106+0000","last_became_active":"2019-12-05T05:03:29.184716+0000","last_became_peered":"2019-12-05T05:03:29.184716+0000","last_unstale":"2019-12-05T06:51:30.461106+0000","last_undegraded":"2019-12-05T06:51:30.461106+0000","last_fullsized":"2019-12-05T06:51:30.461106+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:28.161818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:28.161818+0000","last_clean_scrub_stamp":"2019-12-05T05:03:28.161818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"124'1360","reported_seq":"2114","reported_epoch":"125","state":"active+clean","last_fresh":"2019-12-05T06:51:30.438072+0000","last_change":"2019-12-05T05:03:32.476846+0000","last_active":"2019-12-05T06:51:30.438072+0000","last_peered":"2019-12-05T06:51:30.438072+0000","last_clean":"2019-12-05T06:51:30.438072+0000","last_became_active":"2019-12-05T05:03:32.476140+0000","last_became_peered":"2019-12-05T05:03:32.476140+0000","last_unstale":"2019-12-05T06:51:30.438072+0000","last_undegraded":"2019-12-05T06:51:30.438072+0000","last_fullsized":"2019-12-05T06:51:30.438072+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:31.200243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:31.200243+0000","last_clean_scrub_stamp":"2019-12-05T05:03:31.200243+0000","log_size":1360,"ondisk_log_size":1360,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":37749370,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":931,"num_read_kb":1362588,"num_write":1192,"num_write_kb":41852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1980,"num_read_kb":81568,"num_write":35915,"num_write_kb":550164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24476,"ondisk_log_size":24476,"up":32,"acting":32,"num_store_stats":8},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":412667787,"num_objects":150,"num_object_clones":0,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":150,"num_whiteouts":0,"num_read":7509,"num_read_kb":12724810,"num_write":14017,"num_write_kb":470646,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":825393152,"data_stored":825335574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":15191,"ondisk_log_size":15191,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576159,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1205944,"kb_used_data":157360,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93165896,"statfs":{"total":96636764160,"available":95401877504,"internally_reserved":1073741824,"allocated":161136640,"data_stored":160401379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Dec 5 06:51:04 2019","interfaces":[{"interface":"back","average":{"1min":0.872,"5min":10.013,"15min":7.459},"min":{"1min":0.425,"5min":0.287,"15min":0.226},"max":{"1min":1.461,"5min":501.337,"15min":825.441},"last":1.064},{"interface":"front","average":{"1min":0.870,"5min":2.647,"15min":4.935},"min":{"1min":0.354,"5min":0.321,"15min":0.318},"max":{"1min":1.431,"5min":150.493,"15min":817.919},"last":0.942}]},{"osd":1,"last update":"Thu Dec 5 06:51:04 2019","interfaces":[{"interface":"back","average":{"1min":0.909,"5min":8.193,"15min":7.424},"min":{"1min":0.512,"5min":0.300,"15min":0.241},"max":{"1min":1.455,"5min":501.235,"15min":735.242},"last":1.172},{"interface":"front","average":{"1min":1.021,"5min":0.925,"15min":5.173},"min":{"1min":0.546,"5min":0.387,"15min":0.322},"max":{"1min":1.495,"5min":2.264,"15min":501.253},"last":1.030}]},{"osd":2,"last update":"Thu Dec 5 06:51:10 2019","interfaces":[{"interface":"back","average":{"1min":0.962,"5min":0.933,"15min":6.753},"min":{"1min":0.651,"5min":0.468,"15min":0.292},"max":{"1min":1.366,"5min":2.133,"15min":982.285},"last":1.221},{"interface":"front","average":{"1min":1.027,"5min":5.943,"15min":10.961},"min":{"1min":0.703,"5min":0.425,"15min":0.272},"max":{"1min":1.527,"5min":419.518,"15min":842.181},"last":1.306}]},{"osd":3,"last update":"Thu Dec 5 06:51:04 2019","interfaces":[{"interface":"back","average":{"1min":1.095,"5min":1.005,"15min":1.545},"min":{"1min":0.733,"5min":0.341,"15min":0.309},"max":{"1min":1.351,"5min":2.072,"15min":157.003},"last":0.588},{"interface":"front","average":{"1min":1.228,"5min":1.101,"15min":1.825},"min":{"1min":0.795,"5min":0.458,"15min":0.384},"max":{"1min":1.399,"5min":2.103,"15min":240.924},"last":0.831}]},{"osd":4,"last update":"Thu Dec 5 06:51:04 2019","interfaces":[{"interface":"back","average":{"1min":0.859,"5min":27.486,"15min":13.538},"min":{"1min":0.542,"5min":0.312,"15min":0.312},"max":{"1min":1.314,"5min":994.097,"15min":994.097},"last":691.616},{"interface":"front","average":{"1min":1.063,"5min":1.049,"15min":4.392},"min":{"1min":0.673,"5min":0.358,"15min":0.358},"max":{"1min":1.401,"5min":2.367,"15min":586.973},"last":0.899}]},{"osd":5,"last update":"Thu Dec 5 06:51:04 2019","interfaces":[{"interface":"back","average":{"1min":1.179,"5min":1.090,"15min":1.100},"min":{"1min":0.796,"5min":0.423,"15min":0.419},"max":{"1min":1.636,"5min":2.455,"15min":2.455},"last":0.781},{"interface":"front","average":{"1min":1.243,"5min":1.174,"15min":5.572},"min":{"1min":0.751,"5min":0.522,"15min":0.489},"max":{"1min":1.547,"5min":2.425,"15min":791.026},"last":1.114}]},{"osd":6,"last update":"Thu Dec 5 06:51:04 2019","interfaces":[{"interface":"back","average":{"1min":1.159,"5min":1.083,"15min":3.761},"min":{"1min":0.773,"5min":0.458,"15min":0.332},"max":{"1min":1.589,"5min":2.317,"15min":878.805},"last":1.356},{"interface":"front","average":{"1min":1.305,"5min":5.742,"15min":8.220},"min":{"1min":0.822,"5min":0.549,"15min":0.519},"max":{"1min":1.695,"5min":450.021,"15min":840.923},"last":1.270}]}]},{"osd":6,"up_from":13,"seq":55834576155,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1140400,"kb_used_data":91816,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93231440,"statfs":{"total":96636764160,"available":95468994560,"internally_reserved":1073741824,"allocated":94019584,"data_stored":93291518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Dec 5 06:51:03 2019","interfaces":[{"interface":"back","average":{"1min":24.785,"5min":5.553,"15min":9.441},"min":{"1min":0.571,"5min":0.254,"15min":0.254},"max":{"1min":501.477,"5min":501.477,"15min":986.070},"last":1.225},{"interface":"front","average":{"1min":56.903,"5min":18.312,"15min":12.733},"min":{"1min":0.762,"5min":0.335,"15min":0.298},"max":{"1min":639.525,"5min":639.525,"15min":872.211},"last":1.292}]},{"osd":1,"last update":"Thu Dec 5 06:51:00 2019","interfaces":[{"interface":"back","average":{"1min":0.952,"5min":9.793,"15min":3.799},"min":{"1min":0.690,"5min":0.357,"15min":0.277},"max":{"1min":1.364,"5min":758.181,"15min":758.181},"last":1.176},{"interface":"front","average":{"1min":41.988,"5min":9.041,"15min":7.092},"min":{"1min":0.678,"5min":0.356,"15min":0.322},"max":{"1min":501.424,"5min":501.424,"15min":501.424},"last":0.916}]},{"osd":2,"last update":"Thu Dec 5 06:51:00 2019","interfaces":[{"interface":"back","average":{"1min":0.934,"5min":0.914,"15min":8.858},"min":{"1min":0.623,"5min":0.392,"15min":0.354},"max":{"1min":1.352,"5min":1.352,"15min":775.503},"last":0.884},{"interface":"front","average":{"1min":1.082,"5min":1.012,"15min":1.002},"min":{"1min":0.531,"5min":0.433,"15min":0.325},"max":{"1min":1.530,"5min":1.530,"15min":1.784},"last":1.133}]},{"osd":3,"last update":"Thu Dec 5 06:51:00 2019","interfaces":[{"interface":"back","average":{"1min":0.982,"5min":11.337,"15min":7.358},"min":{"1min":0.481,"5min":0.418,"15min":0.408},"max":{"1min":1.426,"5min":777.425,"15min":777.425},"last":1.008},{"interface":"front","average":{"1min":1.127,"5min":1.124,"15min":1.282},"min":{"1min":0.738,"5min":0.551,"15min":0.501},"max":{"1min":1.547,"5min":1.568,"15min":65.098},"last":1.098}]},{"osd":4,"last update":"Thu Dec 5 06:51:00 2019","interfaces":[{"interface":"back","average":{"1min":1.115,"5min":6.122,"15min":10.663},"min":{"1min":0.743,"5min":0.426,"15min":0.420},"max":{"1min":1.731,"5min":380.358,"15min":929.519},"last":0.821},{"interface":"front","average":{"1min":41.074,"5min":8.911,"15min":5.622},"min":{"1min":0.576,"5min":0.374,"15min":0.346},"max":{"1min":843.286,"5min":843.286,"15min":843.286},"last":0.749}]},{"osd":5,"last update":"Thu Dec 5 06:51:00 2019","interfaces":[{"interface":"back","average":{"1min":1.119,"5min":1.059,"15min":4.110},"min":{"1min":0.473,"5min":0.360,"15min":0.360},"max":{"1min":1.536,"5min":1.536,"15min":879.085},"last":0.643},{"interface":"front","average":{"1min":1.248,"5min":9.004,"15min":3.765},"min":{"1min":0.807,"5min":0.518,"15min":0.484},"max":{"1min":1.530,"5min":666.419,"15min":666.419},"last":1.051}]},{"osd":7,"last update":"Thu Dec 5 06:51:00 2019","interfaces":[{"interface":"back","average":{"1min":6.301,"5min":3.582,"15min":4.398},"min":{"1min":0.998,"5min":0.509,"15min":0.408},"max":{"1min":111.548,"5min":127.801,"15min":629.008},"last":1.258},{"interface":"front","average":{"1min":1.368,"5min":1.222,"15min":5.762},"min":{"1min":1.032,"5min":0.363,"15min":0.363},"max":{"1min":1.754,"5min":1.754,"15min":633.862},"last":1.329}]}]},{"osd":1,"up_from":13,"seq":55834576153,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1267376,"kb_used_data":218792,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93104464,"statfs":{"total":96636764160,"available":95338971136,"internally_reserved":1073741824,"allocated":224043008,"data_stored":223315010,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Dec 5 06:51:02 2019","interfaces":[{"interface":"back","average":{"1min":0.727,"5min":0.680,"15min":2.496},"min":{"1min":0.442,"5min":0.212,"15min":0.197},"max":{"1min":1.373,"5min":1.373,"15min":381.600},"last":0.541},{"interface":"front","average":{"1min":0.782,"5min":3.100,"15min":6.834},"min":{"1min":0.450,"5min":0.310,"15min":0.226},"max":{"1min":1.417,"5min":266.748,"15min":705.387},"last":0.717}]},{"osd":2,"last update":"Thu Dec 5 06:51:02 2019","interfaces":[{"interface":"back","average":{"1min":0.869,"5min":0.731,"15min":4.849},"min":{"1min":0.354,"5min":0.251,"15min":0.251},"max":{"1min":1.342,"5min":1.833,"15min":806.146},"last":1.082},{"interface":"front","average":{"1min":0.697,"5min":7.330,"15min":8.813},"min":{"1min":0.285,"5min":0.245,"15min":0.245},"max":{"1min":1.041,"5min":660.160,"15min":769.148},"last":1.189}]},{"osd":3,"last update":"Thu Dec 5 06:51:02 2019","interfaces":[{"interface":"back","average":{"1min":0.928,"5min":9.283,"15min":3.696},"min":{"1min":0.474,"5min":0.338,"15min":0.332},"max":{"1min":1.468,"5min":884.578,"15min":884.578},"last":0.500},{"interface":"front","average":{"1min":1.021,"5min":0.934,"15min":7.565},"min":{"1min":0.486,"5min":0.312,"15min":0.312},"max":{"1min":1.512,"5min":1.949,"15min":854.098},"last":0.625}]},{"osd":4,"last update":"Thu Dec 5 06:51:14 2019","interfaces":[{"interface":"back","average":{"1min":23.321,"5min":5.825,"15min":11.220},"min":{"1min":0.366,"5min":0.366,"15min":0.366},"max":{"1min":584.815,"5min":584.815,"15min":584.815},"last":0.990},{"interface":"front","average":{"1min":0.955,"5min":7.867,"15min":9.542},"min":{"1min":0.275,"5min":0.275,"15min":0.275},"max":{"1min":1.295,"5min":686.649,"15min":778.089},"last":0.906}]},{"osd":5,"last update":"Thu Dec 5 06:51:02 2019","interfaces":[{"interface":"back","average":{"1min":1.077,"5min":1.001,"15min":4.159},"min":{"1min":0.633,"5min":0.482,"15min":0.403},"max":{"1min":1.362,"5min":1.769,"15min":839.045},"last":0.841},{"interface":"front","average":{"1min":1.102,"5min":11.793,"15min":6.461},"min":{"1min":0.569,"5min":0.375,"15min":0.375},"max":{"1min":1.610,"5min":598.889,"15min":598.889},"last":1.242}]},{"osd":6,"last update":"Thu Dec 5 06:51:02 2019","interfaces":[{"interface":"back","average":{"1min":33.387,"5min":7.490,"15min":3.777},"min":{"1min":0.818,"5min":0.365,"15min":0.360},"max":{"1min":807.184,"5min":807.184,"15min":807.184},"last":1.296},{"interface":"front","average":{"1min":1.134,"5min":1.112,"15min":14.638},"min":{"1min":0.684,"5min":0.410,"15min":0.410},"max":{"1min":1.532,"5min":1.623,"15min":848.805},"last":1.151}]},{"osd":7,"last update":"Thu Dec 5 06:51:02 2019","interfaces":[{"interface":"back","average":{"1min":1.268,"5min":3.514,"15min":4.600},"min":{"1min":0.737,"5min":0.409,"15min":0.409},"max":{"1min":1.783,"5min":274.305,"15min":680.492},"last":1.053},{"interface":"front","average":{"1min":31.226,"5min":20.849,"15min":9.395},"min":{"1min":0.611,"5min":0.523,"15min":0.518},"max":{"1min":501.595,"5min":792.863,"15min":792.863},"last":1.358}]}]},{"osd":0,"up_from":13,"seq":55834576155,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1169080,"kb_used_data":120496,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93202760,"statfs":{"total":96636764160,"available":95439626240,"internally_reserved":1073741824,"allocated":123387904,"data_stored":122652711,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Dec 5 06:50:46 2019","interfaces":[{"interface":"back","average":{"1min":0.686,"5min":7.582,"15min":4.606},"min":{"1min":0.441,"5min":0.274,"15min":0.228},"max":{"1min":1.025,"5min":587.293,"15min":587.293},"last":0.869},{"interface":"front","average":{"1min":48.594,"5min":16.982,"15min":12.839},"min":{"1min":0.474,"5min":0.261,"15min":0.261},"max":{"1min":861.119,"5min":861.119,"15min":966.609},"last":0.821}]},{"osd":2,"last update":"Thu Dec 5 06:50:46 2019","interfaces":[{"interface":"back","average":{"1min":0.718,"5min":16.878,"15min":9.147},"min":{"1min":0.409,"5min":0.335,"15min":0.246},"max":{"1min":0.945,"5min":933.957,"15min":933.957},"last":0.614},{"interface":"front","average":{"1min":0.800,"5min":11.038,"15min":5.446},"min":{"1min":0.500,"5min":0.331,"15min":0.280},"max":{"1min":1.107,"5min":868.889,"15min":868.889},"last":0.896}]},{"osd":3,"last update":"Thu Dec 5 06:50:46 2019","interfaces":[{"interface":"back","average":{"1min":0.726,"5min":0.755,"15min":4.589},"min":{"1min":0.503,"5min":0.310,"15min":0.255},"max":{"1min":0.951,"5min":1.541,"15min":994.147},"last":0.718},{"interface":"front","average":{"1min":39.198,"5min":8.544,"15min":8.714},"min":{"1min":0.368,"5min":0.303,"15min":0.303},"max":{"1min":730.339,"5min":730.339,"15min":854.568},"last":0.783}]},{"osd":4,"last update":"Thu Dec 5 06:50:46 2019","interfaces":[{"interface":"back","average":{"1min":51.043,"5min":11.006,"15min":9.007},"min":{"1min":0.555,"5min":0.471,"15min":0.340},"max":{"1min":492.774,"5min":492.774,"15min":983.581},"last":1.065},{"interface":"front","average":{"1min":48.797,"5min":19.380,"15min":8.384},"min":{"1min":0.709,"5min":0.403,"15min":0.400},"max":{"1min":909.368,"5min":909.368,"15min":909.368},"last":0.948}]},{"osd":5,"last update":"Thu Dec 5 06:51:30 2019","interfaces":[{"interface":"back","average":{"1min":1.019,"5min":1.049,"15min":6.033},"min":{"1min":0.365,"5min":0.365,"15min":0.365},"max":{"1min":1.529,"5min":1.533,"15min":475.167},"last":1.255},{"interface":"front","average":{"1min":0.990,"5min":27.033,"15min":13.790},"min":{"1min":0.410,"5min":0.410,"15min":0.410},"max":{"1min":1.573,"5min":827.052,"15min":827.052},"last":1.117}]},{"osd":6,"last update":"Thu Dec 5 06:50:46 2019","interfaces":[{"interface":"back","average":{"1min":1.103,"5min":1.092,"15min":4.976},"min":{"1min":0.689,"5min":0.539,"15min":0.322},"max":{"1min":1.517,"5min":1.549,"15min":893.025},"last":237.675},{"interface":"front","average":{"1min":1.203,"5min":1.176,"15min":1.135},"min":{"1min":0.788,"5min":0.509,"15min":0.430},"max":{"1min":1.677,"5min":1.677,"15min":1.812},"last":1.196}]},{"osd":7,"last update":"Thu Dec 5 06:51:11 2019","interfaces":[{"interface":"back","average":{"1min":1.039,"5min":9.348,"15min":5.897},"min":{"1min":0.628,"5min":0.450,"15min":0.450},"max":{"1min":1.632,"5min":743.880,"15min":743.880},"last":0.934},{"interface":"front","average":{"1min":1.187,"5min":1.205,"15min":7.028},"min":{"1min":0.455,"5min":0.455,"15min":0.455},"max":{"1min":1.742,"5min":1.742,"15min":920.552},"last":1.352}]}]},{"osd":2,"up_from":13,"seq":55834576151,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1129648,"kb_used_data":81064,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93242192,"statfs":{"total":96636764160,"available":95480004608,"internally_reserved":1073741824,"allocated":83009536,"data_stored":82281105,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Dec 5 06:50:45 2019","interfaces":[{"interface":"back","average":{"1min":0.651,"5min":18.625,"15min":6.641},"min":{"1min":0.466,"5min":0.238,"15min":0.235},"max":{"1min":0.924,"5min":917.307,"15min":917.307},"last":0.728},{"interface":"front","average":{"1min":0.742,"5min":6.302,"15min":2.553},"min":{"1min":0.555,"5min":0.235,"15min":0.213},"max":{"1min":0.947,"5min":423.686,"15min":423.686},"last":0.951}]},{"osd":1,"last update":"Thu Dec 5 06:50:42 2019","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.742,"15min":3.716},"min":{"1min":0.438,"5min":0.331,"15min":0.284},"max":{"1min":1.103,"5min":1.316,"15min":501.424},"last":0.796},{"interface":"front","average":{"1min":51.618,"5min":11.016,"15min":9.907},"min":{"1min":0.416,"5min":0.385,"15min":0.234},"max":{"1min":814.005,"5min":814.005,"15min":814.005},"last":1.005}]},{"osd":3,"last update":"Thu Dec 5 06:50:50 2019","interfaces":[{"interface":"back","average":{"1min":0.784,"5min":5.153,"15min":2.200},"min":{"1min":0.456,"5min":0.288,"15min":0.262},"max":{"1min":1.029,"5min":474.268,"15min":474.268},"last":0.884},{"interface":"front","average":{"1min":0.812,"5min":0.840,"15min":1.716},"min":{"1min":0.528,"5min":0.334,"15min":0.303},"max":{"1min":1.126,"5min":1.291,"15min":245.318},"last":0.675}]},{"osd":4,"last update":"Thu Dec 5 06:50:42 2019","interfaces":[{"interface":"back","average":{"1min":1.045,"5min":8.455,"15min":3.953},"min":{"1min":0.615,"5min":0.332,"15min":0.323},"max":{"1min":1.361,"5min":632.509,"15min":632.509},"last":1.204},{"interface":"front","average":{"1min":1.099,"5min":1.111,"15min":4.268},"min":{"1min":0.757,"5min":0.610,"15min":0.365},"max":{"1min":1.333,"5min":2.178,"15min":991.418},"last":1.384}]},{"osd":5,"last update":"Thu Dec 5 06:50:42 2019","interfaces":[{"interface":"back","average":{"1min":1.015,"5min":9.320,"15min":5.649},"min":{"1min":0.649,"5min":0.316,"15min":0.316},"max":{"1min":1.501,"5min":965.993,"15min":965.993},"last":1.137},{"interface":"front","average":{"1min":0.961,"5min":11.954,"15min":6.908},"min":{"1min":0.743,"5min":0.381,"15min":0.381},"max":{"1min":1.201,"5min":700.890,"15min":760.855},"last":1.259}]},{"osd":6,"last update":"Thu Dec 5 06:50:45 2019","interfaces":[{"interface":"back","average":{"1min":1.180,"5min":1.142,"15min":6.080},"min":{"1min":0.791,"5min":0.415,"15min":0.344},"max":{"1min":1.416,"5min":2.126,"15min":652.287},"last":1.498},{"interface":"front","average":{"1min":63.325,"5min":15.337,"15min":10.066},"min":{"1min":0.983,"5min":0.444,"15min":0.420},"max":{"1min":934.081,"5min":934.081,"15min":934.081},"last":1.448}]},{"osd":7,"last update":"Thu Dec 5 06:50:42 2019","interfaces":[{"interface":"back","average":{"1min":1.197,"5min":5.507,"15min":2.611},"min":{"1min":0.807,"5min":0.440,"15min":0.440},"max":{"1min":1.380,"5min":419.900,"15min":419.900},"last":1.618},{"interface":"front","average":{"1min":1.132,"5min":1.104,"15min":1.084},"min":{"1min":0.896,"5min":0.408,"15min":0.408},"max":{"1min":1.456,"5min":1.620,"15min":1.794},"last":1.574}]}]},{"osd":3,"up_from":13,"seq":55834576154,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1085268,"kb_used_data":36684,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93286572,"statfs":{"total":96636764160,"available":95525449728,"internally_reserved":1073741824,"allocated":37564416,"data_stored":36843874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Dec 5 06:50:34 2019","interfaces":[{"interface":"back","average":{"1min":0.648,"5min":18.297,"15min":13.063},"min":{"1min":0.378,"5min":0.197,"15min":0.196},"max":{"1min":1.040,"5min":802.154,"15min":802.154},"last":0.788},{"interface":"front","average":{"1min":0.799,"5min":7.919,"15min":9.090},"min":{"1min":0.464,"5min":0.242,"15min":0.242},"max":{"1min":1.111,"5min":828.266,"15min":991.916},"last":0.882}]},{"osd":1,"last update":"Thu Dec 5 06:50:44 2019","interfaces":[{"interface":"back","average":{"1min":2.653,"5min":8.851,"15min":17.518},"min":{"1min":0.420,"5min":0.264,"15min":0.211},"max":{"1min":44.449,"5min":699.405,"15min":957.811},"last":0.830},{"interface":"front","average":{"1min":0.775,"5min":0.837,"15min":13.986},"min":{"1min":0.480,"5min":0.311,"15min":0.282},"max":{"1min":0.999,"5min":1.396,"15min":730.364},"last":1.116}]},{"osd":2,"last update":"Thu Dec 5 06:50:59 2019","interfaces":[{"interface":"back","average":{"1min":0.830,"5min":0.878,"15min":4.117},"min":{"1min":0.524,"5min":0.266,"15min":0.266},"max":{"1min":1.512,"5min":1.512,"15min":444.359},"last":0.655},{"interface":"front","average":{"1min":0.932,"5min":0.867,"15min":9.156},"min":{"1min":0.431,"5min":0.340,"15min":0.297},"max":{"1min":1.586,"5min":1.586,"15min":814.674},"last":1.080}]},{"osd":4,"last update":"Thu Dec 5 06:50:34 2019","interfaces":[{"interface":"back","average":{"1min":0.928,"5min":3.523,"15min":1.927},"min":{"1min":0.653,"5min":0.402,"15min":0.325},"max":{"1min":1.194,"5min":216.348,"15min":216.348},"last":0.983},{"interface":"front","average":{"1min":1.170,"5min":0.985,"15min":1.594},"min":{"1min":0.658,"5min":0.335,"15min":0.335},"max":{"1min":1.669,"5min":1.669,"15min":192.285},"last":1.212}]},{"osd":5,"last update":"Thu Dec 5 06:50:34 2019","interfaces":[{"interface":"back","average":{"1min":1.065,"5min":8.695,"15min":4.569},"min":{"1min":0.675,"5min":0.372,"15min":0.328},"max":{"1min":1.595,"5min":880.020,"15min":880.020},"last":1.307},{"interface":"front","average":{"1min":1.107,"5min":1.085,"15min":3.870},"min":{"1min":0.679,"5min":0.458,"15min":0.364},"max":{"1min":1.408,"5min":1.554,"15min":632.864},"last":1.034}]},{"osd":6,"last update":"Thu Dec 5 06:50:34 2019","interfaces":[{"interface":"back","average":{"1min":1.259,"5min":1.070,"15min":2.907},"min":{"1min":0.845,"5min":0.423,"15min":0.413},"max":{"1min":1.750,"5min":1.750,"15min":494.397},"last":1.254},{"interface":"front","average":{"1min":1.322,"5min":8.120,"15min":3.854},"min":{"1min":0.897,"5min":0.475,"15min":0.465},"max":{"1min":1.789,"5min":558.711,"15min":558.711},"last":1.181}]},{"osd":7,"last update":"Thu Dec 5 06:50:50 2019","interfaces":[{"interface":"back","average":{"1min":1.067,"5min":2.388,"15min":10.379},"min":{"1min":0.657,"5min":0.523,"15min":0.365},"max":{"1min":1.465,"5min":97.130,"15min":877.034},"last":1.383},{"interface":"front","average":{"1min":1.134,"5min":1.213,"15min":1.166},"min":{"1min":0.695,"5min":0.499,"15min":0.435},"max":{"1min":1.510,"5min":1.784,"15min":1.979},"last":1.444}]}]},{"osd":4,"up_from":13,"seq":55834576152,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1050280,"kb_used_data":1696,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93321560,"statfs":{"total":96636764160,"available":95561277440,"internally_reserved":1073741824,"allocated":1736704,"data_stored":1016264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Dec 5 06:51:05 2019","interfaces":[{"interface":"back","average":{"1min":0.881,"5min":7.022,"15min":5.481},"min":{"1min":0.542,"5min":0.283,"15min":0.264},"max":{"1min":1.366,"5min":439.064,"15min":439.064},"last":1.111},{"interface":"front","average":{"1min":31.716,"5min":12.524,"15min":5.380},"min":{"1min":0.662,"5min":0.225,"15min":0.225},"max":{"1min":524.030,"5min":524.030,"15min":524.030},"last":0.873}]},{"osd":1,"last update":"Thu Dec 5 06:51:05 2019","interfaces":[{"interface":"back","average":{"1min":0.962,"5min":1.204,"15min":4.257},"min":{"1min":0.551,"5min":0.303,"15min":0.303},"max":{"1min":1.309,"5min":31.305,"15min":501.402},"last":0.664},{"interface":"front","average":{"1min":0.982,"5min":0.929,"15min":4.757},"min":{"1min":0.648,"5min":0.304,"15min":0.304},"max":{"1min":1.576,"5min":1.576,"15min":501.754},"last":1.075}]},{"osd":2,"last update":"Thu Dec 5 06:51:14 2019","interfaces":[{"interface":"back","average":{"1min":43.387,"5min":31.525,"15min":12.886},"min":{"1min":0.397,"5min":0.359,"15min":0.268},"max":{"1min":850.354,"5min":897.315,"15min":897.315},"last":0.792},{"interface":"front","average":{"1min":1.010,"5min":0.982,"15min":3.965},"min":{"1min":0.287,"5min":0.287,"15min":0.287},"max":{"1min":1.397,"5min":1.397,"15min":632.767},"last":0.751}]},{"osd":3,"last update":"Thu Dec 5 06:51:05 2019","interfaces":[{"interface":"back","average":{"1min":0.863,"5min":6.934,"15min":3.446},"min":{"1min":0.694,"5min":0.340,"15min":0.340},"max":{"1min":1.045,"5min":501.658,"15min":501.658},"last":0.768},{"interface":"front","average":{"1min":9.928,"5min":2.858,"15min":2.715},"min":{"1min":0.934,"5min":0.432,"15min":0.348},"max":{"1min":149.969,"5min":149.969,"15min":308.851},"last":0.902}]},{"osd":5,"last update":"Thu Dec 5 06:51:05 2019","interfaces":[{"interface":"back","average":{"1min":1.098,"5min":0.906,"15min":3.992},"min":{"1min":0.704,"5min":0.325,"15min":0.325},"max":{"1min":1.682,"5min":1.682,"15min":840.089},"last":0.814},{"interface":"front","average":{"1min":1.224,"5min":16.912,"15min":9.261},"min":{"1min":0.918,"5min":0.385,"15min":0.369},"max":{"1min":1.729,"5min":660.266,"15min":748.692},"last":0.948}]},{"osd":6,"last update":"Thu Dec 5 06:51:05 2019","interfaces":[{"interface":"back","average":{"1min":0.900,"5min":18.011,"15min":10.852},"min":{"1min":0.583,"5min":0.354,"15min":0.314},"max":{"1min":1.139,"5min":817.106,"15min":861.297},"last":0.691},{"interface":"front","average":{"1min":1.258,"5min":1.114,"15min":3.380},"min":{"1min":0.708,"5min":0.390,"15min":0.374},"max":{"1min":1.808,"5min":3.181,"15min":690.644},"last":1.008}]},{"osd":7,"last update":"Thu Dec 5 06:51:21 2019","interfaces":[{"interface":"back","average":{"1min":1.120,"5min":1.135,"15min":6.627},"min":{"1min":0.481,"5min":0.425,"15min":0.351},"max":{"1min":1.534,"5min":2.624,"15min":507.089},"last":0.984},{"interface":"front","average":{"1min":1.194,"5min":1.191,"15min":3.242},"min":{"1min":0.520,"5min":0.464,"15min":0.464},"max":{"1min":1.635,"5min":1.670,"15min":452.400},"last":1.040}]}]},{"osd":5,"up_from":13,"seq":55834576157,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1160244,"kb_used_data":111660,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93211596,"statfs":{"total":96636764160,"available":95448674304,"internally_reserved":1073741824,"allocated":114339840,"data_stored":113613092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Dec 5 06:50:48 2019","interfaces":[{"interface":"back","average":{"1min":18.174,"5min":4.336,"15min":1.990},"min":{"1min":0.333,"5min":0.333,"15min":0.235},"max":{"1min":331.975,"5min":331.975,"15min":331.975},"last":1.194},{"interface":"front","average":{"1min":0.786,"5min":0.855,"15min":3.602},"min":{"1min":0.471,"5min":0.351,"15min":0.278},"max":{"1min":1.137,"5min":2.878,"15min":676.393},"last":0.698}]},{"osd":1,"last update":"Thu Dec 5 06:50:58 2019","interfaces":[{"interface":"back","average":{"1min":0.854,"5min":0.773,"15min":2.216},"min":{"1min":0.544,"5min":0.308,"15min":0.225},"max":{"1min":1.145,"5min":1.274,"15min":251.684},"last":0.845},{"interface":"front","average":{"1min":38.990,"5min":8.478,"15min":5.469},"min":{"1min":0.721,"5min":0.388,"15min":0.245},"max":{"1min":722.849,"5min":722.849,"15min":722.849},"last":0.969}]},{"osd":2,"last update":"Thu Dec 5 06:50:45 2019","interfaces":[{"interface":"back","average":{"1min":0.850,"5min":17.271,"15min":8.269},"min":{"1min":0.612,"5min":0.457,"15min":0.281},"max":{"1min":1.198,"5min":934.880,"15min":934.880},"last":0.923},{"interface":"front","average":{"1min":0.978,"5min":9.447,"15min":10.949},"min":{"1min":0.696,"5min":0.488,"15min":0.409},"max":{"1min":1.315,"5min":676.797,"15min":794.102},"last":1.082}]},{"osd":3,"last update":"Thu Dec 5 06:51:24 2019","interfaces":[{"interface":"back","average":{"1min":1.037,"5min":1.093,"15min":3.676},"min":{"1min":0.468,"5min":0.468,"15min":0.340},"max":{"1min":1.318,"5min":2.942,"15min":683.842},"last":1.296},{"interface":"front","average":{"1min":42.174,"5min":19.109,"15min":8.696},"min":{"1min":0.500,"5min":0.500,"15min":0.387},"max":{"1min":740.171,"5min":740.171,"15min":740.171},"last":1.344}]},{"osd":4,"last update":"Thu Dec 5 06:50:48 2019","interfaces":[{"interface":"back","average":{"1min":1.096,"5min":1.087,"15min":7.581},"min":{"1min":0.730,"5min":0.441,"15min":0.327},"max":{"1min":1.417,"5min":2.914,"15min":633.428},"last":1.036},{"interface":"front","average":{"1min":1.072,"5min":0.956,"15min":5.140},"min":{"1min":0.593,"5min":0.330,"15min":0.309},"max":{"1min":1.344,"5min":1.409,"15min":792.823},"last":1.132}]},{"osd":6,"last update":"Thu Dec 5 06:50:48 2019","interfaces":[{"interface":"back","average":{"1min":1.157,"5min":1.008,"15min":6.845},"min":{"1min":0.902,"5min":0.370,"15min":0.318},"max":{"1min":1.470,"5min":1.637,"15min":891.609},"last":1.244},{"interface":"front","average":{"1min":34.677,"5min":19.775,"15min":10.319},"min":{"1min":1.115,"5min":0.409,"15min":0.409},"max":{"1min":635.191,"5min":650.798,"15min":916.939},"last":1.406}]},{"osd":7,"last update":"Thu Dec 5 06:50:48 2019","interfaces":[{"interface":"back","average":{"1min":1.314,"5min":8.254,"15min":6.479},"min":{"1min":1.049,"5min":0.586,"15min":0.401},"max":{"1min":1.511,"5min":735.632,"15min":735.632},"last":1.468},{"interface":"front","average":{"1min":1.396,"5min":8.665,"15min":6.319},"min":{"1min":1.028,"5min":0.677,"15min":0.380},"max":{"1min":1.646,"5min":405.552,"15min":405.552},"last":1.503}]}]}]}} 2019-12-05T06:51:38.959 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2019-12-05T06:51:39.055 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:39.046+0000 7f895e7b3700 1 Processor -- start 2019-12-05T06:51:39.056 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:39.046+0000 7f895e7b3700 1 -- start start 2019-12-05T06:51:39.056 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:39.046+0000 7f895e7b3700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f89580a7740 0x7f895808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:51:39.056 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:39.046+0000 7f895e7b3700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f895809fae0 0x7f895807eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:51:39.056 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:39.046+0000 7f895e7b3700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89580a0710 con 0x7f895807e720 2019-12-05T06:51:39.057 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:39.046+0000 7f895e7b3700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89580a0b00 con 0x7f895809fae0 2019-12-05T06:51:39.057 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:39.046+0000 7f895e7b3700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f895808e5b0 con 0x7f89580a7740 2019-12-05T06:51:39.057 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:39.046+0000 7f895cd50700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f89580a7740 0x7f895808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:51:39.057 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:39.046+0000 7f895d551700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f895809fae0 0x7f895807eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:51:39.058 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:39.046+0000 7f895cd50700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f89580a7740 0x7f895808df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:39706/0 (socket says 172.21.15.42:39706) 2019-12-05T06:51:39.058 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:39.046+0000 7f895cd50700 1 -- 172.21.15.42:0/3578838887 learned_addr learned my addr 172.21.15.42:0/3578838887 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:51:39.058 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:39.050+0000 7f895cd50700 1 -- 172.21.15.42:0/3578838887 >> v2:172.21.15.40:3300/0 conn(0x7f895809fae0 msgr2=0x7f895807eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:51:39.059 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:39.050+0000 7f895cd50700 1 --2- 172.21.15.42:0/3578838887 >> v2:172.21.15.40:3300/0 conn(0x7f895809fae0 0x7f895807eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:51:39.059 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:39.050+0000 7f895cd50700 1 -- 172.21.15.42:0/3578838887 >> v1:172.21.15.40:6789/0 conn(0x7f895807e720 legacy=0x7f895809f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:39.059 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:39.050+0000 7f895cd50700 1 -- 172.21.15.42:0/3578838887 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f895807f0a0 con 0x7f89580a7740 2019-12-05T06:51:39.059 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:39.050+0000 7f895cd50700 1 --2- 172.21.15.42:0/3578838887 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f89580a7740 0x7f895808df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f8940008790 tx=0x7f894000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:39.060 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:39.050+0000 7f895cd50700 1 --2- 172.21.15.42:0/3578838887 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f89580a7740 0x7f895808df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f8940008790 tx=0x7f894000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:39.060 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:39.050+0000 7f895cd50700 1 --2- 172.21.15.42:0/3578838887 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f89580a7740 0x7f895808df80 secure :-1 s=READY pgs=744 cs=0 l=1 rx=0x7f8940008790 tx=0x7f894000fd30).ready entity=mon.2 client_cookie=2c8ee4a0004ca741 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:51:39.060 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:39.050+0000 7f895cd50700 1 --2- 172.21.15.42:0/3578838887 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f89580a7740 0x7f895808df80 secure :-1 s=THROTTLE_DONE pgs=744 cs=0 l=1 rx=0x7f8940008790 tx=0x7f894000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:39.060 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:39.050+0000 7f89577fe700 1 -- 172.21.15.42:0/3578838887 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8940008c90 con 0x7f89580a7740 2019-12-05T06:51:39.060 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:39.050+0000 7f895cd50700 1 --2- 172.21.15.42:0/3578838887 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f89580a7740 0x7f895808df80 secure :-1 s=THROTTLE_DONE pgs=744 cs=0 l=1 rx=0x7f8940008790 tx=0x7f894000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:39.061 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:39.050+0000 7f89577fe700 1 -- 172.21.15.42:0/3578838887 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89400088b0 con 0x7f89580a7740 2019-12-05T06:51:39.061 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:39.050+0000 7f895cd50700 1 --2- 172.21.15.42:0/3578838887 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f89580a7740 0x7f895808df80 secure :-1 s=THROTTLE_DONE pgs=744 cs=0 l=1 rx=0x7f8940008790 tx=0x7f894000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:39.061 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:39.050+0000 7f89577fe700 1 -- 172.21.15.42:0/3578838887 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8940011b10 con 0x7f89580a7740 2019-12-05T06:51:39.061 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:39.050+0000 7f895e7b3700 1 -- 172.21.15.42:0/3578838887 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f89580a7740 msgr2=0x7f895808df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:39.062 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:39.050+0000 7f895e7b3700 1 --2- 172.21.15.42:0/3578838887 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f89580a7740 0x7f895808df80 secure :-1 s=READY pgs=744 cs=0 l=1 rx=0x7f8940008790 tx=0x7f894000fd30).stop 2019-12-05T06:51:39.062 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:39.050+0000 7f895e7b3700 1 -- 172.21.15.42:0/3578838887 shutdown_connections 2019-12-05T06:51:39.062 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:39.050+0000 7f895e7b3700 1 --2- 172.21.15.42:0/3578838887 >> v2:172.21.15.40:3300/0 conn(0x7f895809fae0 0x7f895807eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:51:39.062 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:39.050+0000 7f895e7b3700 1 --2- 172.21.15.42:0/3578838887 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f89580a7740 0x7f895808df80 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:51:39.063 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:39.050+0000 7f895e7b3700 1 -- 172.21.15.42:0/3578838887 shutdown_connections 2019-12-05T06:51:39.063 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:39.050+0000 7f895e7b3700 1 -- 172.21.15.42:0/3578838887 wait complete. 2019-12-05T06:51:39.063 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:39.050+0000 7f895e7b3700 1 -- 172.21.15.42:0/3578838887 >> 172.21.15.42:0/3578838887 conn(0x7f89580aabe0 msgr2=0x7f89580a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:51:39.068 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:39.054+0000 7f895e7b3700 1 Processor -- start 2019-12-05T06:51:39.068 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:39.054+0000 7f895e7b3700 1 -- start start 2019-12-05T06:51:39.069 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:39.054+0000 7f895e7b3700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f89581187a0 0x7f895807bb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:51:39.069 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:39.054+0000 7f895e7b3700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f895807c0d0 con 0x7f895807afe0 2019-12-05T06:51:39.070 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:39.054+0000 7f895e7b3700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f895807d0f0 con 0x7f895807b410 2019-12-05T06:51:39.070 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:39.054+0000 7f895e7b3700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f895807d320 con 0x7f89581187a0 2019-12-05T06:51:39.071 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:39.054+0000 7f895cd50700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f89581187a0 0x7f895807bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:51:39.071 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:39.054+0000 7f895d551700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f895807b410 0x7f8958137890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:39288/0 (socket says 172.21.15.42:39288) 2019-12-05T06:51:39.072 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:39.054+0000 7f895d551700 1 -- 172.21.15.42:0/13984114 learned_addr learned my addr 172.21.15.42:0/13984114 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:51:39.072 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:39.054+0000 7f89557fa700 1 -- 172.21.15.42:0/13984114 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3583310732 0 0) 0x7f89580a0710 con 0x7f895807afe0 2019-12-05T06:51:39.072 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:39.054+0000 7f89557fa700 1 -- 172.21.15.42:0/13984114 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f893c0030b0 con 0x7f895807afe0 2019-12-05T06:51:39.072 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:39.054+0000 7f895cd50700 1 -- 172.21.15.42:0/13984114 >> v1:172.21.15.42:6789/0 conn(0x7f895807b410 legacy=0x7f8958137890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:39.073 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:39.054+0000 7f89557fa700 1 -- 172.21.15.42:0/13984114 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3397777712 0 0) 0x7f895807d0f0 con 0x7f895807b410 2019-12-05T06:51:39.073 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:39.058+0000 7f895cd50700 1 -- 172.21.15.42:0/13984114 >> v1:172.21.15.40:6789/0 conn(0x7f895807afe0 legacy=0x7f895811af50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:39.073 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:39.058+0000 7f895cd50700 1 -- 172.21.15.42:0/13984114 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f895811b750 con 0x7f89581187a0 2019-12-05T06:51:39.073 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:39.058+0000 7f895cd50700 1 --2- 172.21.15.42:0/13984114 >> v2:172.21.15.42:3300/0 conn(0x7f89581187a0 0x7f895807bb80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f8940008790 tx=0x7f8940008f40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:39.074 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:39.058+0000 7f895cd50700 1 --2- 172.21.15.42:0/13984114 >> v2:172.21.15.42:3300/0 conn(0x7f89581187a0 0x7f895807bb80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f8940008790 tx=0x7f8940008f40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:39.074 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:39.058+0000 7f895cd50700 1 --2- 172.21.15.42:0/13984114 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f89581187a0 0x7f895807bb80 secure :-1 s=READY pgs=3364 cs=0 l=1 rx=0x7f8940008790 tx=0x7f8940008f40).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:51:39.074 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:39.058+0000 7f895cd50700 1 --2- 172.21.15.42:0/13984114 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f89581187a0 0x7f895807bb80 secure :-1 s=THROTTLE_DONE pgs=3364 cs=0 l=1 rx=0x7f8940008790 tx=0x7f8940008f40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:39.074 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:39.058+0000 7f89557fa700 1 -- 172.21.15.42:0/13984114 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f894000f470 con 0x7f89581187a0 2019-12-05T06:51:39.074 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:39.058+0000 7f895cd50700 1 --2- 172.21.15.42:0/13984114 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f89581187a0 0x7f895807bb80 secure :-1 s=THROTTLE_DONE pgs=3364 cs=0 l=1 rx=0x7f8940008790 tx=0x7f8940008f40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:39.075 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:39.058+0000 7f895cd50700 1 --2- 172.21.15.42:0/13984114 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f89581187a0 0x7f895807bb80 secure :-1 s=THROTTLE_DONE pgs=3364 cs=0 l=1 rx=0x7f8940008790 tx=0x7f8940008f40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:39.075 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:39.058+0000 7f89557fa700 1 -- 172.21.15.42:0/13984114 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8940011730 con 0x7f89581187a0 2019-12-05T06:51:39.075 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:39.058+0000 7f895e7b3700 1 -- 172.21.15.42:0/13984114 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f895811b9c0 con 0x7f89581187a0 2019-12-05T06:51:39.075 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:39.058+0000 7f89557fa700 1 -- 172.21.15.42:0/13984114 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8940016800 con 0x7f89581187a0 2019-12-05T06:51:39.075 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:39.058+0000 7f895e7b3700 1 -- 172.21.15.42:0/13984114 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f895808eb70 con 0x7f89581187a0 2019-12-05T06:51:39.076 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:39.058+0000 7f895cd50700 1 --2- 172.21.15.42:0/13984114 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f89581187a0 0x7f895807bb80 secure :-1 s=THROTTLE_DONE pgs=3364 cs=0 l=1 rx=0x7f8940008790 tx=0x7f8940008f40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:39.076 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:39.058+0000 7f89557fa700 1 -- 172.21.15.42:0/13984114 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f8940016a50 con 0x7f89581187a0 2019-12-05T06:51:39.076 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:39.058+0000 7f895cd50700 1 --2- 172.21.15.42:0/13984114 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f89581187a0 0x7f895807bb80 secure :-1 s=THROTTLE_DONE pgs=3364 cs=0 l=1 rx=0x7f8940008790 tx=0x7f8940008f40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:39.076 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:39.058+0000 7f89557fa700 1 --2- 172.21.15.42:0/13984114 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f893c03b200 0x7f893c03d710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:51:39.076 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:39.058+0000 7f89557fa700 1 -- 172.21.15.42:0/13984114 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 12365+0+0 (secure 0 0 0) 0x7f894004d860 con 0x7f89581187a0 2019-12-05T06:51:39.076 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:39.058+0000 7f8957fff700 1 --2- 172.21.15.42:0/13984114 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f893c03b200 0x7f893c03d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:51:39.076 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:39.058+0000 7f8957fff700 1 --2- 172.21.15.42:0/13984114 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f893c03b200 0x7f893c03d710 crc :-1 s=READY pgs=1409 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:51:39.077 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:39.062+0000 7f895e7b3700 1 -- 172.21.15.42:0/13984114 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f895805d110 con 0x7f89581187a0 2019-12-05T06:51:39.077 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:39.070+0000 7f895cd50700 1 --2- 172.21.15.42:0/13984114 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f89581187a0 0x7f895807bb80 secure :-1 s=THROTTLE_DONE pgs=3364 cs=0 l=1 rx=0x7f8940008790 tx=0x7f8940008f40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:39.077 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:39.070+0000 7f89557fa700 1 -- 172.21.15.42:0/13984114 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f89400348e0 con 0x7f89581187a0 2019-12-05T06:51:39.241 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:39.238+0000 7f895e7b3700 1 -- 172.21.15.42:0/13984114 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f895805b8c0 con 0x7f89581187a0 2019-12-05T06:51:39.243 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:39.238+0000 7f895cd50700 1 --2- 172.21.15.42:0/13984114 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f89581187a0 0x7f895807bb80 secure :-1 s=THROTTLE_DONE pgs=3364 cs=0 l=1 rx=0x7f8940008790 tx=0x7f8940008f40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:39.244 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:39.238+0000 7f89557fa700 1 -- 172.21.15.42:0/13984114 <== mon.1 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v127) v1 ==== 74+0+24452 (secure 0 0 0) 0x7f8940016d60 con 0x7f89581187a0 2019-12-05T06:51:39.245 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:51:39.245 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":127,"fsid":"5d836cfc-a83b-4433-9b3e-b9b2f31c9050","created":"2019-12-05T05:03:13.215602+0000","modified":"2019-12-05T06:51:36.545494+0000","last_up_change":"2019-12-05T05:03:22.664783+0000","last_in_change":"2019-12-05T05:03:22.664783+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2019-12-05T05:03:27.394605+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2019-12-05T05:03:30.419845+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2019-12-05T05:03:33.249075+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"127","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":557,"snap_epoch":127,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e3fc49bb-dcbc-43ff-ac7c-9eb0ea0bd257","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6810","nonce":12118},{"type":"v1","addr":"172.21.15.42:6811","nonce":12118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6812","nonce":12118},{"type":"v1","addr":"172.21.15.42:6813","nonce":12118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6816","nonce":12118},{"type":"v1","addr":"172.21.15.42:6817","nonce":12118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6814","nonce":12118},{"type":"v1","addr":"172.21.15.42:6815","nonce":12118}]},"public_addr":"172.21.15.42:6811/12118","cluster_addr":"172.21.15.42:6813/12118","heartbeat_back_addr":"172.21.15.42:6817/12118","heartbeat_front_addr":"172.21.15.42:6815/12118","state":["exists","up"]},{"osd":1,"uuid":"902543fb-f5a2-4321-9c9c-26f9b9ca8629","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6826","nonce":12119},{"type":"v1","addr":"172.21.15.42:6827","nonce":12119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6828","nonce":12119},{"type":"v1","addr":"172.21.15.42:6829","nonce":12119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6832","nonce":12119},{"type":"v1","addr":"172.21.15.42:6833","nonce":12119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6830","nonce":12119},{"type":"v1","addr":"172.21.15.42:6831","nonce":12119}]},"public_addr":"172.21.15.42:6827/12119","cluster_addr":"172.21.15.42:6829/12119","heartbeat_back_addr":"172.21.15.42:6833/12119","heartbeat_front_addr":"172.21.15.42:6831/12119","state":["exists","up"]},{"osd":2,"uuid":"28739ff7-bed7-4b75-8ea5-b57d088f1250","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6802","nonce":12117},{"type":"v1","addr":"172.21.15.42:6803","nonce":12117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6804","nonce":12117},{"type":"v1","addr":"172.21.15.42:6805","nonce":12117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6808","nonce":12117},{"type":"v1","addr":"172.21.15.42:6809","nonce":12117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6806","nonce":12117},{"type":"v1","addr":"172.21.15.42:6807","nonce":12117}]},"public_addr":"172.21.15.42:6803/12117","cluster_addr":"172.21.15.42:6805/12117","heartbeat_back_addr":"172.21.15.42:6809/12117","heartbeat_front_addr":"172.21.15.42:6807/12117","state":["exists","up"]},{"osd":3,"uuid":"1960f66b-2d40-4bb9-a289-ce8cd0c1c142","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6818","nonce":12116},{"type":"v1","addr":"172.21.15.42:6819","nonce":12116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6820","nonce":12116},{"type":"v1","addr":"172.21.15.42:6821","nonce":12116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6824","nonce":12116},{"type":"v1","addr":"172.21.15.42:6825","nonce":12116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6822","nonce":12116},{"type":"v1","addr":"172.21.15.42:6823","nonce":12116}]},"public_addr":"172.21.15.42:6819/12116","cluster_addr":"172.21.15.42:6821/12116","heartbeat_back_addr":"172.21.15.42:6825/12116","heartbeat_front_addr":"172.21.15.42:6823/12116","state":["exists","up"]},{"osd":4,"uuid":"45bbf1bd-7c8a-4a0d-b76b-afa6148a855c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":12488},{"type":"v1","addr":"172.21.15.40:6809","nonce":12488}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":12488},{"type":"v1","addr":"172.21.15.40:6812","nonce":12488}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":12488},{"type":"v1","addr":"172.21.15.40:6825","nonce":12488}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":12488},{"type":"v1","addr":"172.21.15.40:6819","nonce":12488}]},"public_addr":"172.21.15.40:6809/12488","cluster_addr":"172.21.15.40:6812/12488","heartbeat_back_addr":"172.21.15.40:6825/12488","heartbeat_front_addr":"172.21.15.40:6819/12488","state":["exists","up"]},{"osd":5,"uuid":"8a5511cd-925f-4869-a49e-924383e4ce53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":12489},{"type":"v1","addr":"172.21.15.40:6801","nonce":12489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":12489},{"type":"v1","addr":"172.21.15.40:6803","nonce":12489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":12489},{"type":"v1","addr":"172.21.15.40:6807","nonce":12489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":12489},{"type":"v1","addr":"172.21.15.40:6805","nonce":12489}]},"public_addr":"172.21.15.40:6801/12489","cluster_addr":"172.21.15.40:6803/12489","heartbeat_back_addr":"172.21.15.40:6807/12489","heartbeat_front_addr":"172.21.15.40:6805/12489","state":["exists","up"]},{"osd":6,"uuid":"d692c179-379e-446c-8484-a219a9241e04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6811","nonce":12491},{"type":"v1","addr":"172.21.15.40:6814","nonce":12491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6817","nonce":12491},{"type":"v1","addr":"172.21.15.40:6818","nonce":12491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":12491},{"type":"v1","addr":"172.21.15.40:6828","nonce":12491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":12491},{"type":"v1","addr":"172.21.15.40:6824","nonce":12491}]},"public_addr":"172.21.15.40:6814/12491","cluster_addr":"172.21.15.40:6818/12491","heartbeat_back_addr":"172.21.15.40:6828/12491","heartbeat_front_addr":"172.21.15.40:6824/12491","state":["exists","up"]},{"osd":7,"uuid":"79c3c243-190e-4e35-b769-319a2077e7cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6813","nonce":12490},{"type":"v1","addr":"172.21.15.40:6816","nonce":12490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":12490},{"type":"v1","addr":"172.21.15.40:6823","nonce":12490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":12490},{"type":"v1","addr":"172.21.15.40:6831","nonce":12490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6827","nonce":12490},{"type":"v1","addr":"172.21.15.40:6829","nonce":12490}]},"public_addr":"172.21.15.40:6816/12490","cluster_addr":"172.21.15.40:6823/12490","heartbeat_back_addr":"172.21.15.40:6831/12490","heartbeat_front_addr":"172.21.15.40:6829/12490","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T05:03:21.349150+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T05:03:21.644281+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T05:03:21.394040+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T05:03:21.587096+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T05:03:21.656497+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T05:03:21.510547+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T05:03:21.659358+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T05:03:21.628063+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{"172.21.15.40:6833/815316041":"2019-12-06T06:50:15.403188+0000","172.21.15.40:6833/4178556019":"2019-12-06T06:48:20.366134+0000","172.21.15.40:6832/4178556019":"2019-12-06T06:48:20.366134+0000","172.21.15.42:6835/1449317789":"2019-12-06T06:49:00.379730+0000","172.21.15.42:6835/2489540211":"2019-12-06T06:45:55.301261+0000","172.21.15.40:6833/3599619":"2019-12-06T06:44:25.276320+0000","172.21.15.40:6832/3599619":"2019-12-06T06:44:25.276320+0000","172.21.15.42:6835/1922398668":"2019-12-06T06:43:25.247354+0000","172.21.15.42:6834/1922398668":"2019-12-06T06:43:25.247354+0000","172.21.15.40:6833/474728597":"2019-12-06T06:42:45.234540+0000","172.21.15.40:6832/474728597":"2019-12-06T06:42:45.234540+0000","172.21.15.42:6835/406829134":"2019-12-06T06:40:40.197779+0000","172.21.15.42:6834/406829134":"2019-12-06T06:40:40.197779+0000","172.21.15.40:6832/1777669201":"2019-12-06T06:40:05.186898+0000","172.21.15.42:6834/958796360":"2019-12-06T06:39:30.176520+0000","172.21.15.42:6835/958796360":"2019-12-06T06:39:30.176520+0000","172.21.15.40:6832/4253974783":"2019-12-06T06:37:55.150270+0000","172.21.15.40:6832/2055706790":"2019-12-06T06:33:25.051960+0000","172.21.15.42:6834/1985322937":"2019-12-06T06:31:14.985625+0000","172.21.15.40:6832/3155851600":"2019-12-06T06:30:14.967933+0000","172.21.15.42:6835/3427104185":"2019-12-06T06:28:24.936048+0000","172.21.15.42:6834/3427104185":"2019-12-06T06:28:24.936048+0000","172.21.15.42:6835/3964426552":"2019-12-06T06:24:34.851396+0000","172.21.15.40:6832/2324657302":"2019-12-06T06:26:39.889893+0000","172.21.15.42:6834/3964426552":"2019-12-06T06:24:34.851396+0000","172.21.15.42:6834/2532474042":"2019-12-06T06:35:35.090613+0000","172.21.15.42:6834/2638662429":"2019-12-06T06:21:34.795475+0000","172.21.15.42:6834/1449317789":"2019-12-06T06:49:00.379730+0000","172.21.15.40:6832/1473620421":"2019-12-06T06:19:49.763362+0000","172.21.15.42:6834/3228549071":"2019-12-06T06:19:09.750176+0000","172.21.15.42:6835/4032647860":"2019-12-06T06:14:34.627961+0000","172.21.15.42:6835/2532474042":"2019-12-06T06:35:35.090613+0000","172.21.15.42:6834/4032647860":"2019-12-06T06:14:34.627961+0000","172.21.15.40:6833/3785523496":"2019-12-06T06:13:14.603886+0000","172.21.15.40:6832/3785523496":"2019-12-06T06:13:14.603886+0000","172.21.15.42:6835/1601334502":"2019-12-06T05:34:33.762032+0000","172.21.15.42:6835/2456922143":"2019-12-06T05:50:49.115731+0000","172.21.15.42:6834/1601334502":"2019-12-06T05:34:33.762032+0000","172.21.15.42:6835/3228549071":"2019-12-06T06:19:09.750176+0000","172.21.15.40:6832/3543033403":"2019-12-06T05:32:48.730013+0000","172.21.15.40:6832/815316041":"2019-12-06T06:50:15.403188+0000","172.21.15.42:6835/2636699468":"2019-12-06T06:10:09.528777+0000","172.21.15.42:6835/2638662429":"2019-12-06T06:21:34.795475+0000","172.21.15.40:6833/3168718372":"2019-12-06T05:30:48.693050+0000","172.21.15.40:6832/3789081777":"2019-12-06T06:05:29.441993+0000","172.21.15.40:6833/3061061478":"2019-12-06T05:23:03.537466+0000","172.21.15.40:6833/2667648531":"2019-12-06T06:16:59.698225+0000","172.21.15.40:6833/3789081777":"2019-12-06T06:05:29.441993+0000","172.21.15.42:6835/1233824070":"2019-12-06T05:42:43.928051+0000","172.21.15.40:6833/1149478696":"2019-12-06T05:28:13.631131+0000","172.21.15.40:6832/1149478696":"2019-12-06T05:28:13.631131+0000","172.21.15.40:6832/90248709":"2019-12-06T05:09:23.257810+0000","172.21.15.40:6832/2146240383":"2019-12-06T05:49:19.088366+0000","172.21.15.42:6835/613535662":"2019-12-06T06:04:04.415621+0000","172.21.15.42:6835/1455108684":"2019-12-06T05:27:08.611758+0000","172.21.15.40:6832/3789641053":"2019-12-06T05:37:08.826019+0000","172.21.15.42:6834/1455108684":"2019-12-06T05:27:08.611758+0000","172.21.15.42:6834/3264829400":"2019-12-06T05:17:13.419048+0000","172.21.15.40:6833/3208416000":"2019-12-06T05:25:33.582715+0000","172.21.15.40:6832/526579407":"2019-12-06T06:23:44.835535+0000","172.21.15.40:6832/2667648531":"2019-12-06T06:16:59.698225+0000","172.21.15.42:6835/2544447110":"2019-12-06T05:29:18.666496+0000","172.21.15.40:6832/3208416000":"2019-12-06T05:25:33.582715+0000","172.21.15.42:6835/1214659805":"2019-12-06T05:23:53.552042+0000","172.21.15.40:6833/3789641053":"2019-12-06T05:37:08.826019+0000","172.21.15.42:6835/1414371821":"2019-12-06T05:20:43.495003+0000","172.21.15.42:6834/3757738536":"2019-12-06T05:47:14.050547+0000","172.21.15.40:6833/211584380":"2019-12-06T05:11:18.295434+0000","172.21.15.40:6832/2698421979":"2019-12-06T05:18:48.447318+0000","172.21.15.40:6833/90248709":"2019-12-06T05:09:23.257810+0000","172.21.15.42:6835/1368848584":"2019-12-06T05:59:44.294812+0000","172.21.15.40:6832/3296198322":"2019-12-06T05:04:43.166959+0000","172.21.15.40:6833/4253974783":"2019-12-06T06:37:55.150270+0000","172.21.15.42:6834/5991897":"2019-12-06T05:52:59.155806+0000","172.21.15.42:6835/3028851337":"2019-12-06T05:31:48.711252+0000","172.21.15.40:6833/814397242":"2019-12-06T05:13:08.346622+0000","172.21.15.40:6832/814397242":"2019-12-06T05:13:08.346622+0000","172.21.15.42:6835/2019200199":"2019-12-06T05:07:08.208550+0000","172.21.15.42:6835/211479572":"2019-12-06T05:14:13.365106+0000","172.21.15.40:6833/526579407":"2019-12-06T06:23:44.835535+0000","172.21.15.42:6834/1214659805":"2019-12-06T05:23:53.552042+0000","172.21.15.42:6834/2019200199":"2019-12-06T05:07:08.208550+0000","172.21.15.42:6835/448869597":"2019-12-06T06:00:59.318439+0000","172.21.15.42:6835/3757738536":"2019-12-06T05:47:14.050547+0000","172.21.15.40:6833/1473620421":"2019-12-06T06:19:49.763362+0000","172.21.15.42:6834/3176146895":"2019-12-06T05:12:43.339069+0000","172.21.15.40:6832/123054133":"2019-12-06T05:40:58.896726+0000","172.21.15.42:6835/3176146895":"2019-12-06T05:12:43.339069+0000","172.21.15.40:6833/3150831963":"2019-12-06T05:15:08.381589+0000","172.21.15.42:6834/2253267554":"2019-12-06T06:06:59.469462+0000","172.21.15.40:6832/3150831963":"2019-12-06T05:15:08.381589+0000","172.21.15.40:6833/2324657302":"2019-12-06T06:26:39.889893+0000","172.21.15.42:6834/2636699468":"2019-12-06T06:10:09.528777+0000","172.21.15.40:6832/3061061478":"2019-12-06T05:23:03.537466+0000","172.21.15.40:6833/2698421979":"2019-12-06T05:18:48.447318+0000","172.21.15.42:6835/3650564018":"2019-12-06T05:10:03.269449+0000","172.21.15.42:6834/1414371821":"2019-12-06T05:20:43.495003+0000","172.21.15.40:6833/1777669201":"2019-12-06T06:40:05.186898+0000","172.21.15.40:6833/3296198322":"2019-12-06T05:04:43.166959+0000","172.21.15.42:6835/3264829400":"2019-12-06T05:17:13.419048+0000","172.21.15.40:6833/3543033403":"2019-12-06T05:32:48.730013+0000","172.21.15.42:6834/3838877789":"2019-12-06T05:39:08.862108+0000","172.21.15.42:6834/2456922143":"2019-12-06T05:50:49.115731+0000","172.21.15.42:6834/448869597":"2019-12-06T06:00:59.318439+0000","172.21.15.42:6834/3923117701":"2019-12-06T06:12:09.582993+0000","172.21.15.42:6834/2489540211":"2019-12-06T06:45:55.301261+0000","172.21.15.40:6833/3155851600":"2019-12-06T06:30:14.967933+0000","172.21.15.42:6834/3650564018":"2019-12-06T05:10:03.269449+0000","172.21.15.42:6834/2544447110":"2019-12-06T05:29:18.666496+0000","172.21.15.40:6833/123054133":"2019-12-06T05:40:58.896726+0000","172.21.15.42:6834/1233824070":"2019-12-06T05:42:43.928051+0000","172.21.15.40:6832/211584380":"2019-12-06T05:11:18.295434+0000","172.21.15.42:6834/211479572":"2019-12-06T05:14:13.365106+0000","172.21.15.40:6832/1029013337":"2019-12-06T05:44:03.952842+0000","172.21.15.40:6833/2055706790":"2019-12-06T06:33:25.051960+0000","172.21.15.42:6834/613535662":"2019-12-06T06:04:04.415621+0000","172.21.15.40:6833/2146240383":"2019-12-06T05:49:19.088366+0000","172.21.15.40:6832/2780168808":"2019-12-06T06:01:39.357366+0000","172.21.15.40:6833/1029013337":"2019-12-06T05:44:03.952842+0000","172.21.15.42:6834/3028851337":"2019-12-06T05:31:48.711252+0000","172.21.15.42:6835/3923117701":"2019-12-06T06:12:09.582993+0000","172.21.15.42:6834/358497296":"2019-12-06T05:44:43.965329+0000","172.21.15.42:6835/358497296":"2019-12-06T05:44:43.965329+0000","172.21.15.40:6832/1127133319":"2019-12-06T05:45:39.022579+0000","172.21.15.40:6833/1127133319":"2019-12-06T05:45:39.022579+0000","172.21.15.42:6835/5991897":"2019-12-06T05:52:59.155806+0000","172.21.15.40:6832/2081111170":"2019-12-06T05:51:19.125597+0000","172.21.15.40:6833/2081111170":"2019-12-06T05:51:19.125597+0000","172.21.15.40:6832/1523046776":"2019-12-06T05:54:59.208157+0000","172.21.15.40:6833/1523046776":"2019-12-06T05:54:59.208157+0000","172.21.15.42:6835/3838877789":"2019-12-06T05:39:08.862108+0000","172.21.15.42:6834/2511123581":"2019-12-06T05:55:39.220889+0000","172.21.15.40:6832/3168718372":"2019-12-06T05:30:48.693050+0000","172.21.15.42:6835/2511123581":"2019-12-06T05:55:39.220889+0000","172.21.15.40:6833/3549993215":"2019-12-06T05:57:24.253912+0000","172.21.15.42:6834/1368848584":"2019-12-06T05:59:44.294812+0000","172.21.15.42:6835/1985322937":"2019-12-06T06:31:14.985625+0000","172.21.15.40:6832/1082666540":"2019-12-06T06:00:14.304550+0000","172.21.15.40:6833/1082666540":"2019-12-06T06:00:14.304550+0000","172.21.15.40:6833/2780168808":"2019-12-06T06:01:39.357366+0000","172.21.15.40:6832/2968227232":"2019-12-06T06:08:09.492099+0000","172.21.15.42:6835/2253267554":"2019-12-06T06:06:59.469462+0000","172.21.15.40:6833/2968227232":"2019-12-06T06:08:09.492099+0000","172.21.15.40:6832/3355679996":"2019-12-06T06:11:29.570475+0000","172.21.15.40:6832/3549993215":"2019-12-06T05:57:24.253912+0000","172.21.15.40:6833/3355679996":"2019-12-06T06:11:29.570475+0000"},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":528,"length":30}]}],"new_removed_snaps":[{"pool":3,"snaps":[{"begin":554,"length":4}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2019-12-05T06:51:39.247 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:39.242+0000 7f895e7b3700 1 -- 172.21.15.42:0/13984114 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f893c03b200 msgr2=0x7f893c03d710 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:39.247 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:39.242+0000 7f895e7b3700 1 --2- 172.21.15.42:0/13984114 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f893c03b200 0x7f893c03d710 crc :-1 s=READY pgs=1409 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:51:39.247 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:39.242+0000 7f895e7b3700 1 -- 172.21.15.42:0/13984114 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f89581187a0 msgr2=0x7f895807bb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:39.247 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:39.242+0000 7f895e7b3700 1 --2- 172.21.15.42:0/13984114 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f89581187a0 0x7f895807bb80 secure :-1 s=READY pgs=3364 cs=0 l=1 rx=0x7f8940008790 tx=0x7f8940008f40).stop 2019-12-05T06:51:39.247 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:39.242+0000 7f895e7b3700 1 -- 172.21.15.42:0/13984114 shutdown_connections 2019-12-05T06:51:39.247 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:39.242+0000 7f895e7b3700 1 --2- 172.21.15.42:0/13984114 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f893c03b200 0x7f893c03d710 unknown :-1 s=CLOSED pgs=1409 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:51:39.248 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:39.242+0000 7f895e7b3700 1 --2- 172.21.15.42:0/13984114 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f89581187a0 0x7f895807bb80 unknown :-1 s=CLOSED pgs=3364 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:51:39.248 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:39.242+0000 7f895e7b3700 1 -- 172.21.15.42:0/13984114 shutdown_connections 2019-12-05T06:51:39.248 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:39.242+0000 7f895e7b3700 1 -- 172.21.15.42:0/13984114 wait complete. 2019-12-05T06:51:39.248 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:39.242+0000 7f895e7b3700 1 -- 172.21.15.42:0/13984114 >> 172.21.15.42:0/13984114 conn(0x7f89580aabe0 msgr2=0x7f89580a9790 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:51:40.259 INFO:tasks.ceph:Scrubbing osd.0 2019-12-05T06:51:40.259 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2019-12-05T06:51:40.370 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.358+0000 7f6d718ab700 1 Processor -- start 2019-12-05T06:51:40.370 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.358+0000 7f6d718ab700 1 -- start start 2019-12-05T06:51:40.370 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.362+0000 7f6d718ab700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6d6c09f550 0x7f6d6c144630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:51:40.370 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.362+0000 7f6d718ab700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f6d6c0a9a00 0x7f6d6c07e880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:51:40.370 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.362+0000 7f6d718ab700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d6c0a1000 con 0x7f6d6c07edd0 2019-12-05T06:51:40.371 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.362+0000 7f6d718ab700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d6c0a8f80 con 0x7f6d6c0a9a00 2019-12-05T06:51:40.371 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.362+0000 7f6d718ab700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d6c0a9220 con 0x7f6d6c09f550 2019-12-05T06:51:40.371 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.362+0000 7f6d6b7fe700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6d6c09f550 0x7f6d6c144630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:51:40.371 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.362+0000 7f6d6affd700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f6d6c0a9a00 0x7f6d6c07e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:51:40.371 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.362+0000 7f6d6b7fe700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6d6c09f550 0x7f6d6c144630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:39720/0 (socket says 172.21.15.42:39720) 2019-12-05T06:51:40.372 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.362+0000 7f6d6b7fe700 1 -- 172.21.15.42:0/1238894737 learned_addr learned my addr 172.21.15.42:0/1238894737 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:51:40.372 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.362+0000 7f6d6b7fe700 1 -- 172.21.15.42:0/1238894737 >> v2:172.21.15.40:3300/0 conn(0x7f6d6c0a9a00 msgr2=0x7f6d6c07e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:51:40.372 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.362+0000 7f6d6b7fe700 1 --2- 172.21.15.42:0/1238894737 >> v2:172.21.15.40:3300/0 conn(0x7f6d6c0a9a00 0x7f6d6c07e880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:51:40.372 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.362+0000 7f6d6b7fe700 1 -- 172.21.15.42:0/1238894737 >> v1:172.21.15.40:6789/0 conn(0x7f6d6c07edd0 legacy=0x7f6d6c147b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:40.372 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.362+0000 7f6d6b7fe700 1 -- 172.21.15.42:0/1238894737 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d6c144b80 con 0x7f6d6c09f550 2019-12-05T06:51:40.372 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.362+0000 7f6d6b7fe700 1 --2- 172.21.15.42:0/1238894737 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6d6c09f550 0x7f6d6c144630 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f6d5c008790 tx=0x7f6d5c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:40.373 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.362+0000 7f6d6b7fe700 1 --2- 172.21.15.42:0/1238894737 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6d6c09f550 0x7f6d6c144630 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f6d5c008790 tx=0x7f6d5c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:40.373 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.362+0000 7f6d6b7fe700 1 --2- 172.21.15.42:0/1238894737 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6d6c09f550 0x7f6d6c144630 secure :-1 s=READY pgs=745 cs=0 l=1 rx=0x7f6d5c008790 tx=0x7f6d5c00fd30).ready entity=mon.2 client_cookie=fba28b21ea327214 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:51:40.373 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.362+0000 7f6d6b7fe700 1 --2- 172.21.15.42:0/1238894737 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6d6c09f550 0x7f6d6c144630 secure :-1 s=THROTTLE_DONE pgs=745 cs=0 l=1 rx=0x7f6d5c008790 tx=0x7f6d5c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:40.373 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.362+0000 7f6d6a7fc700 1 -- 172.21.15.42:0/1238894737 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6d5c0113b0 con 0x7f6d6c09f550 2019-12-05T06:51:40.373 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.362+0000 7f6d6b7fe700 1 --2- 172.21.15.42:0/1238894737 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6d6c09f550 0x7f6d6c144630 secure :-1 s=THROTTLE_DONE pgs=745 cs=0 l=1 rx=0x7f6d5c008790 tx=0x7f6d5c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:40.373 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.362+0000 7f6d6a7fc700 1 -- 172.21.15.42:0/1238894737 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d5c008bb0 con 0x7f6d6c09f550 2019-12-05T06:51:40.374 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.362+0000 7f6d6b7fe700 1 --2- 172.21.15.42:0/1238894737 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6d6c09f550 0x7f6d6c144630 secure :-1 s=THROTTLE_DONE pgs=745 cs=0 l=1 rx=0x7f6d5c008790 tx=0x7f6d5c00fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:40.374 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.362+0000 7f6d6a7fc700 1 -- 172.21.15.42:0/1238894737 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6d5c011990 con 0x7f6d6c09f550 2019-12-05T06:51:40.374 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.362+0000 7f6d718ab700 1 -- 172.21.15.42:0/1238894737 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6d6c09f550 msgr2=0x7f6d6c144630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:40.374 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.362+0000 7f6d718ab700 1 --2- 172.21.15.42:0/1238894737 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6d6c09f550 0x7f6d6c144630 secure :-1 s=READY pgs=745 cs=0 l=1 rx=0x7f6d5c008790 tx=0x7f6d5c00fd30).stop 2019-12-05T06:51:40.374 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.366+0000 7f6d718ab700 1 -- 172.21.15.42:0/1238894737 shutdown_connections 2019-12-05T06:51:40.375 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.366+0000 7f6d718ab700 1 --2- 172.21.15.42:0/1238894737 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6d6c09f550 0x7f6d6c144630 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:51:40.375 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.366+0000 7f6d718ab700 1 --2- 172.21.15.42:0/1238894737 >> v2:172.21.15.40:3300/0 conn(0x7f6d6c0a9a00 0x7f6d6c07e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:51:40.375 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.366+0000 7f6d718ab700 1 -- 172.21.15.42:0/1238894737 shutdown_connections 2019-12-05T06:51:40.375 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.366+0000 7f6d718ab700 1 -- 172.21.15.42:0/1238894737 wait complete. 2019-12-05T06:51:40.375 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.366+0000 7f6d718ab700 1 -- 172.21.15.42:0/1238894737 >> 172.21.15.42:0/1238894737 conn(0x7f6d6c0a9ef0 msgr2=0x7f6d6c0aa320 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:51:40.380 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.366+0000 7f6d718ab700 1 Processor -- start 2019-12-05T06:51:40.380 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.366+0000 7f6d718ab700 1 -- start start 2019-12-05T06:51:40.381 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.370+0000 7f6d718ab700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6d6c0a3480 0x7f6d6c0a1230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:51:40.381 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.370+0000 7f6d718ab700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f6d6c064840 0x7f6d6c0a1d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:51:40.381 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.370+0000 7f6d718ab700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d6c0a22c0 con 0x7f6d6c0a6cf0 2019-12-05T06:51:40.382 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.370+0000 7f6d718ab700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d6c0a1830 con 0x7f6d6c064840 2019-12-05T06:51:40.382 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.370+0000 7f6d718ab700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d6c0a1ad0 con 0x7f6d6c0a3480 2019-12-05T06:51:40.382 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.370+0000 7f6d6b7fe700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6d6c0a3480 0x7f6d6c0a1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:51:40.383 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.370+0000 7f6d6affd700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f6d6c0a6cf0 0x7f6d6c143510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:39300/0 (socket says 172.21.15.42:39300) 2019-12-05T06:51:40.383 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.370+0000 7f6d6affd700 1 -- 172.21.15.42:0/2747410099 learned_addr learned my addr 172.21.15.42:0/2747410099 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:51:40.383 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.370+0000 7f6d6bfff700 1 --2- 172.21.15.42:0/2747410099 >> v2:172.21.15.40:3300/0 conn(0x7f6d6c064840 0x7f6d6c0a1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:51:40.383 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.370+0000 7f6d53fff700 1 -- 172.21.15.42:0/2747410099 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1790659861 0 0) 0x7f6d6c0a22c0 con 0x7f6d6c0a6cf0 2019-12-05T06:51:40.383 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.370+0000 7f6d53fff700 1 -- 172.21.15.42:0/2747410099 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d4c0030b0 con 0x7f6d6c0a6cf0 2019-12-05T06:51:40.383 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.370+0000 7f6d6b7fe700 1 -- 172.21.15.42:0/2747410099 >> v2:172.21.15.40:3300/0 conn(0x7f6d6c064840 msgr2=0x7f6d6c0a1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:40.384 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.370+0000 7f6d6b7fe700 1 --2- 172.21.15.42:0/2747410099 >> v2:172.21.15.40:3300/0 conn(0x7f6d6c064840 0x7f6d6c0a1d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:51:40.386 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.370+0000 7f6d6b7fe700 1 -- 172.21.15.42:0/2747410099 >> v1:172.21.15.42:6789/0 conn(0x7f6d6c0a6cf0 legacy=0x7f6d6c143510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:40.386 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.370+0000 7f6d6b7fe700 1 -- 172.21.15.42:0/2747410099 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d6c0a4050 con 0x7f6d6c0a3480 2019-12-05T06:51:40.386 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.370+0000 7f6d6b7fe700 1 --2- 172.21.15.42:0/2747410099 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6d6c0a3480 0x7f6d6c0a1230 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f6d5c008790 tx=0x7f6d5c008bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:40.386 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.370+0000 7f6d6b7fe700 1 --2- 172.21.15.42:0/2747410099 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6d6c0a3480 0x7f6d6c0a1230 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f6d5c008790 tx=0x7f6d5c008bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:40.386 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.370+0000 7f6d6b7fe700 1 --2- 172.21.15.42:0/2747410099 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6d6c0a3480 0x7f6d6c0a1230 secure :-1 s=READY pgs=746 cs=0 l=1 rx=0x7f6d5c008790 tx=0x7f6d5c008bb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:51:40.387 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.370+0000 7f6d6b7fe700 1 --2- 172.21.15.42:0/2747410099 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6d6c0a3480 0x7f6d6c0a1230 secure :-1 s=THROTTLE_DONE pgs=746 cs=0 l=1 rx=0x7f6d5c008790 tx=0x7f6d5c008bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:40.389 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.370+0000 7f6d53fff700 1 -- 172.21.15.42:0/2747410099 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6d5c007340 con 0x7f6d6c0a3480 2019-12-05T06:51:40.389 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.370+0000 7f6d6b7fe700 1 --2- 172.21.15.42:0/2747410099 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6d6c0a3480 0x7f6d6c0a1230 secure :-1 s=THROTTLE_DONE pgs=746 cs=0 l=1 rx=0x7f6d5c008790 tx=0x7f6d5c008bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:40.389 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.370+0000 7f6d6b7fe700 1 --2- 172.21.15.42:0/2747410099 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6d6c0a3480 0x7f6d6c0a1230 secure :-1 s=THROTTLE_DONE pgs=746 cs=0 l=1 rx=0x7f6d5c008790 tx=0x7f6d5c008bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:40.389 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.370+0000 7f6d53fff700 1 -- 172.21.15.42:0/2747410099 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d5c0115b0 con 0x7f6d6c0a3480 2019-12-05T06:51:40.389 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.370+0000 7f6d718ab700 1 -- 172.21.15.42:0/2747410099 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d6c064c70 con 0x7f6d6c0a3480 2019-12-05T06:51:40.389 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.370+0000 7f6d53fff700 1 -- 172.21.15.42:0/2747410099 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6d5c005540 con 0x7f6d6c0a3480 2019-12-05T06:51:40.389 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.370+0000 7f6d718ab700 1 -- 172.21.15.42:0/2747410099 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d6c143f30 con 0x7f6d6c0a3480 2019-12-05T06:51:40.390 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.374+0000 7f6d6b7fe700 1 --2- 172.21.15.42:0/2747410099 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6d6c0a3480 0x7f6d6c0a1230 secure :-1 s=THROTTLE_DONE pgs=746 cs=0 l=1 rx=0x7f6d5c008790 tx=0x7f6d5c008bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:40.390 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.374+0000 7f6d53fff700 1 -- 172.21.15.42:0/2747410099 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f6d5c005850 con 0x7f6d6c0a3480 2019-12-05T06:51:40.390 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.374+0000 7f6d6b7fe700 1 --2- 172.21.15.42:0/2747410099 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6d6c0a3480 0x7f6d6c0a1230 secure :-1 s=THROTTLE_DONE pgs=746 cs=0 l=1 rx=0x7f6d5c008790 tx=0x7f6d5c008bb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:40.390 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.374+0000 7f6d53fff700 1 --2- 172.21.15.42:0/2747410099 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6d4c03b2a0 0x7f6d4c03d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:51:40.390 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.374+0000 7f6d53fff700 1 -- 172.21.15.42:0/2747410099 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 12365+0+0 (secure 0 0 0) 0x7f6d5c04b3a0 con 0x7f6d6c0a3480 2019-12-05T06:51:40.390 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.374+0000 7f6d6affd700 1 --2- 172.21.15.42:0/2747410099 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6d4c03b2a0 0x7f6d4c03d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:51:40.392 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.374+0000 7f6d6affd700 1 --2- 172.21.15.42:0/2747410099 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6d4c03b2a0 0x7f6d4c03d7b0 crc :-1 s=READY pgs=1410 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:51:40.392 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.374+0000 7f6d718ab700 1 --2- 172.21.15.42:0/2747410099 >> [v2:172.21.15.42:6810/12118,v1:172.21.15.42:6811/12118] conn(0x7f6d6c054260 0x7f6d6c054690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:51:40.393 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.374+0000 7f6d718ab700 1 -- 172.21.15.42:0/2747410099 --> [v2:172.21.15.42:6810/12118,v1:172.21.15.42:6811/12118] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6d6c0a93e0 con 0x7f6d6c054260 2019-12-05T06:51:40.393 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.374+0000 7f6d6bfff700 1 --2- 172.21.15.42:0/2747410099 >> [v2:172.21.15.42:6810/12118,v1:172.21.15.42:6811/12118] conn(0x7f6d6c054260 0x7f6d6c054690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:51:40.396 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.374+0000 7f6d6bfff700 1 -- 172.21.15.42:0/2747410099 >> [v2:172.21.15.42:6810/12118,v1:172.21.15.42:6811/12118] conn(0x7f6d6c054260 msgr2=0x7f6d6c054690 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2019-12-05T06:51:40.396 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.374+0000 7f6d6bfff700 1 --2- 172.21.15.42:0/2747410099 >> [v2:172.21.15.42:6810/12118,v1:172.21.15.42:6811/12118] conn(0x7f6d6c054260 0x7f6d6c054690 crc :-1 s=READY pgs=105 cs=0 l=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:51:40.397 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.378+0000 7f6d53fff700 1 -- 172.21.15.42:0/2747410099 <== osd.0 v2:172.21.15.42:6810/12118 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23014 (crc 0 0 0) 0x7f6d6c0a93e0 con 0x7f6d6c054260 2019-12-05T06:51:40.415 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.410+0000 7f6d718ab700 1 -- 172.21.15.42:0/2747410099 --> [v2:172.21.15.42:6810/12118,v1:172.21.15.42:6811/12118] -- command(tid 2: {"var": "osd_debug_deep_scrub_sleep", "prefix": "config set", "val": ["0"]}) v1 -- 0x7f6d6c05b8c0 con 0x7f6d6c054260 2019-12-05T06:51:40.415 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.410+0000 7f6d53fff700 1 -- 172.21.15.42:0/2747410099 <== osd.0 v2:172.21.15.42:6810/12118 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f6d6c05b8c0 con 0x7f6d6c054260 2019-12-05T06:51:40.416 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2019-12-05T06:51:40.416 INFO:teuthology.orchestra.run.smithi042.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2019-12-05T06:51:40.416 INFO:teuthology.orchestra.run.smithi042.stdout:} 2019-12-05T06:51:40.418 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.414+0000 7f6d718ab700 1 -- 172.21.15.42:0/2747410099 >> [v2:172.21.15.42:6810/12118,v1:172.21.15.42:6811/12118] conn(0x7f6d6c054260 msgr2=0x7f6d6c054690 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:40.418 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.414+0000 7f6d718ab700 1 --2- 172.21.15.42:0/2747410099 >> [v2:172.21.15.42:6810/12118,v1:172.21.15.42:6811/12118] conn(0x7f6d6c054260 0x7f6d6c054690 crc :-1 s=READY pgs=105 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:51:40.418 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.414+0000 7f6d718ab700 1 -- 172.21.15.42:0/2747410099 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6d4c03b2a0 msgr2=0x7f6d4c03d7b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:40.418 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.414+0000 7f6d718ab700 1 --2- 172.21.15.42:0/2747410099 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6d4c03b2a0 0x7f6d4c03d7b0 crc :-1 s=READY pgs=1410 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:51:40.419 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.414+0000 7f6d718ab700 1 -- 172.21.15.42:0/2747410099 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6d6c0a3480 msgr2=0x7f6d6c0a1230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:40.419 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.414+0000 7f6d718ab700 1 --2- 172.21.15.42:0/2747410099 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6d6c0a3480 0x7f6d6c0a1230 secure :-1 s=READY pgs=746 cs=0 l=1 rx=0x7f6d5c008790 tx=0x7f6d5c008bb0).stop 2019-12-05T06:51:40.419 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.414+0000 7f6d6bfff700 1 -- 172.21.15.42:0/2747410099 reap_dead start 2019-12-05T06:51:40.419 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.414+0000 7f6d718ab700 1 -- 172.21.15.42:0/2747410099 shutdown_connections 2019-12-05T06:51:40.419 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.414+0000 7f6d718ab700 1 -- 172.21.15.42:0/2747410099 shutdown_connections 2019-12-05T06:51:40.419 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.414+0000 7f6d718ab700 1 -- 172.21.15.42:0/2747410099 wait complete. 2019-12-05T06:51:40.420 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.414+0000 7f6d718ab700 1 -- 172.21.15.42:0/2747410099 >> 172.21.15.42:0/2747410099 conn(0x7f6d6c07dff0 msgr2=0x7f6d6c07c590 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:51:40.428 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2019-12-05T06:51:40.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.518+0000 7f20101b7700 1 Processor -- start 2019-12-05T06:51:40.529 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.518+0000 7f20101b7700 1 -- start start 2019-12-05T06:51:40.530 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.518+0000 7f20101b7700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f20080a7740 0x7f200808df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:51:40.530 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.518+0000 7f20101b7700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20080a0710 con 0x7f200807e720 2019-12-05T06:51:40.530 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.518+0000 7f20101b7700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20080a9ba0 con 0x7f200809fae0 2019-12-05T06:51:40.530 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.518+0000 7f20101b7700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20080a0b00 con 0x7f20080a7740 2019-12-05T06:51:40.530 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.518+0000 7f200e754700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f20080a7740 0x7f200808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:51:40.531 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.522+0000 7f200ef55700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f200809fae0 0x7f200814efe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:39312/0 (socket says 172.21.15.42:39312) 2019-12-05T06:51:40.531 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.522+0000 7f200ef55700 1 -- 172.21.15.42:0/3270101004 learned_addr learned my addr 172.21.15.42:0/3270101004 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:51:40.531 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.522+0000 7f200d752700 1 -- 172.21.15.42:0/3270101004 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4022951849 0 0) 0x7f20080a0710 con 0x7f200807e720 2019-12-05T06:51:40.531 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.522+0000 7f200e754700 1 -- 172.21.15.42:0/3270101004 >> v1:172.21.15.42:6789/0 conn(0x7f200809fae0 legacy=0x7f200814efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:40.531 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.522+0000 7f200e754700 1 -- 172.21.15.42:0/3270101004 >> v1:172.21.15.40:6789/0 conn(0x7f200807e720 legacy=0x7f200809f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:40.531 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.522+0000 7f200e754700 1 -- 172.21.15.42:0/3270101004 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f200807eb50 con 0x7f20080a7740 2019-12-05T06:51:40.532 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.522+0000 7f200e754700 1 --2- 172.21.15.42:0/3270101004 >> v2:172.21.15.42:3300/0 conn(0x7f20080a7740 0x7f200808df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f2004008790 tx=0x7f200400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:40.532 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.522+0000 7f200e754700 1 --2- 172.21.15.42:0/3270101004 >> v2:172.21.15.42:3300/0 conn(0x7f20080a7740 0x7f200808df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f2004008790 tx=0x7f200400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:40.532 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.522+0000 7f200e754700 1 --2- 172.21.15.42:0/3270101004 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f20080a7740 0x7f200808df80 secure :-1 s=READY pgs=3367 cs=0 l=1 rx=0x7f2004008790 tx=0x7f200400fbf0).ready entity=mon.1 client_cookie=8add5a526936ad55 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:51:40.532 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.522+0000 7f200e754700 1 --2- 172.21.15.42:0/3270101004 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f20080a7740 0x7f200808df80 secure :-1 s=THROTTLE_DONE pgs=3367 cs=0 l=1 rx=0x7f2004008790 tx=0x7f200400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:40.532 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.522+0000 7f200d752700 1 -- 172.21.15.42:0/3270101004 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2004011270 con 0x7f20080a7740 2019-12-05T06:51:40.533 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.522+0000 7f200e754700 1 --2- 172.21.15.42:0/3270101004 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f20080a7740 0x7f200808df80 secure :-1 s=THROTTLE_DONE pgs=3367 cs=0 l=1 rx=0x7f2004008790 tx=0x7f200400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:40.533 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.522+0000 7f200e754700 1 --2- 172.21.15.42:0/3270101004 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f20080a7740 0x7f200808df80 secure :-1 s=THROTTLE_DONE pgs=3367 cs=0 l=1 rx=0x7f2004008790 tx=0x7f200400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:40.533 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.522+0000 7f200d752700 1 -- 172.21.15.42:0/3270101004 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2004008bc0 con 0x7f20080a7740 2019-12-05T06:51:40.533 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.522+0000 7f200d752700 1 -- 172.21.15.42:0/3270101004 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2004011850 con 0x7f20080a7740 2019-12-05T06:51:40.533 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.522+0000 7f20101b7700 1 -- 172.21.15.42:0/3270101004 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f20080a7740 msgr2=0x7f200808df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:40.533 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.522+0000 7f20101b7700 1 --2- 172.21.15.42:0/3270101004 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f20080a7740 0x7f200808df80 secure :-1 s=READY pgs=3367 cs=0 l=1 rx=0x7f2004008790 tx=0x7f200400fbf0).stop 2019-12-05T06:51:40.534 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.522+0000 7f20101b7700 1 -- 172.21.15.42:0/3270101004 shutdown_connections 2019-12-05T06:51:40.534 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.522+0000 7f20101b7700 1 --2- 172.21.15.42:0/3270101004 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f20080a7740 0x7f200808df80 unknown :-1 s=CLOSED pgs=3367 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:51:40.534 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.522+0000 7f20101b7700 1 -- 172.21.15.42:0/3270101004 shutdown_connections 2019-12-05T06:51:40.534 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.522+0000 7f20101b7700 1 -- 172.21.15.42:0/3270101004 wait complete. 2019-12-05T06:51:40.534 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.522+0000 7f20101b7700 1 -- 172.21.15.42:0/3270101004 >> 172.21.15.42:0/3270101004 conn(0x7f20080aabe0 msgr2=0x7f20080a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:51:40.535 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.526+0000 7f20101b7700 1 Processor -- start 2019-12-05T06:51:40.535 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.526+0000 7f20101b7700 1 -- start start 2019-12-05T06:51:40.539 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.526+0000 7f20101b7700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f200811b8c0 0x7f2008116500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:51:40.539 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.526+0000 7f20101b7700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f20081217d0 0x7f2008134030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:51:40.539 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.526+0000 7f20101b7700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f200811a700 0x7f200811ab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:51:40.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.526+0000 7f20101b7700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f200811be00 con 0x7f20081217d0 2019-12-05T06:51:40.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.526+0000 7f20101b7700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f200811c0a0 con 0x7f200811b8c0 2019-12-05T06:51:40.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.526+0000 7f20101b7700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f200811c340 con 0x7f200811a700 2019-12-05T06:51:40.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.526+0000 7f200e754700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f200811b8c0 0x7f2008116500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:51:40.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.526+0000 7f200e754700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f200811b8c0 0x7f2008116500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:39276/0 (socket says 172.21.15.42:39276) 2019-12-05T06:51:40.540 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.526+0000 7f200e754700 1 -- 172.21.15.42:0/914963536 learned_addr learned my addr 172.21.15.42:0/914963536 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:51:40.541 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.526+0000 7f200ef55700 1 --2- 172.21.15.42:0/914963536 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f200811a700 0x7f200811ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:51:40.541 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.526+0000 7f200df53700 1 --2- 172.21.15.42:0/914963536 >> v2:172.21.15.40:3300/0 conn(0x7f20081217d0 0x7f2008134030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:51:40.541 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.526+0000 7f200e754700 1 -- 172.21.15.42:0/914963536 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f200811a700 msgr2=0x7f200811ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:40.542 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.526+0000 7f200e754700 1 --2- 172.21.15.42:0/914963536 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f200811a700 0x7f200811ab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:51:40.542 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.526+0000 7f200e754700 1 -- 172.21.15.42:0/914963536 >> v2:172.21.15.40:3300/0 conn(0x7f20081217d0 msgr2=0x7f2008134030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:40.542 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.526+0000 7f200e754700 1 --2- 172.21.15.42:0/914963536 >> v2:172.21.15.40:3300/0 conn(0x7f20081217d0 0x7f2008134030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:51:40.543 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.526+0000 7f200e754700 1 -- 172.21.15.42:0/914963536 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2008134580 con 0x7f200811b8c0 2019-12-05T06:51:40.543 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.526+0000 7f200e754700 1 --2- 172.21.15.42:0/914963536 >> v2:172.21.15.42:3300/0 conn(0x7f200811b8c0 0x7f2008116500 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f200811bfc0 tx=0x7f2004011b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:40.543 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.526+0000 7f200e754700 1 --2- 172.21.15.42:0/914963536 >> v2:172.21.15.42:3300/0 conn(0x7f200811b8c0 0x7f2008116500 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f200811bfc0 tx=0x7f2004011b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:40.544 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.526+0000 7f200e754700 1 --2- 172.21.15.42:0/914963536 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f200811b8c0 0x7f2008116500 secure :-1 s=READY pgs=3368 cs=0 l=1 rx=0x7f200811bfc0 tx=0x7f2004011b30).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:51:40.544 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.526+0000 7f200e754700 1 --2- 172.21.15.42:0/914963536 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f200811b8c0 0x7f2008116500 secure :-1 s=THROTTLE_DONE pgs=3368 cs=0 l=1 rx=0x7f200811bfc0 tx=0x7f2004011b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:40.544 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.526+0000 7f200e754700 1 --2- 172.21.15.42:0/914963536 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f200811b8c0 0x7f2008116500 secure :-1 s=THROTTLE_DONE pgs=3368 cs=0 l=1 rx=0x7f200811bfc0 tx=0x7f2004011b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:40.544 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.526+0000 7f1ffeffd700 1 -- 172.21.15.42:0/914963536 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f200400f550 con 0x7f200811b8c0 2019-12-05T06:51:40.544 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.526+0000 7f200e754700 1 --2- 172.21.15.42:0/914963536 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f200811b8c0 0x7f2008116500 secure :-1 s=THROTTLE_DONE pgs=3368 cs=0 l=1 rx=0x7f200811bfc0 tx=0x7f2004011b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:40.545 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.526+0000 7f1ffeffd700 1 -- 172.21.15.42:0/914963536 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2004011470 con 0x7f200811b8c0 2019-12-05T06:51:40.545 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.526+0000 7f1ffeffd700 1 -- 172.21.15.42:0/914963536 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2004005540 con 0x7f200811b8c0 2019-12-05T06:51:40.545 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.526+0000 7f20101b7700 1 -- 172.21.15.42:0/914963536 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f200811b080 con 0x7f200811b8c0 2019-12-05T06:51:40.545 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.526+0000 7f20101b7700 1 -- 172.21.15.42:0/914963536 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f200808eb70 con 0x7f200811b8c0 2019-12-05T06:51:40.545 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.530+0000 7f200e754700 1 --2- 172.21.15.42:0/914963536 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f200811b8c0 0x7f2008116500 secure :-1 s=THROTTLE_DONE pgs=3368 cs=0 l=1 rx=0x7f200811bfc0 tx=0x7f2004011b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:40.545 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.530+0000 7f1ffeffd700 1 -- 172.21.15.42:0/914963536 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f2004005850 con 0x7f200811b8c0 2019-12-05T06:51:40.546 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.530+0000 7f200e754700 1 --2- 172.21.15.42:0/914963536 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f200811b8c0 0x7f2008116500 secure :-1 s=THROTTLE_DONE pgs=3368 cs=0 l=1 rx=0x7f200811bfc0 tx=0x7f2004011b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:40.546 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.530+0000 7f1ffeffd700 1 --2- 172.21.15.42:0/914963536 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f1ff003aec0 0x7f1ff003d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:51:40.546 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.530+0000 7f1ffeffd700 1 -- 172.21.15.42:0/914963536 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 12365+0+0 (secure 0 0 0) 0x7f200404bc00 con 0x7f200811b8c0 2019-12-05T06:51:40.546 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.530+0000 7f200df53700 1 --2- 172.21.15.42:0/914963536 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f1ff003aec0 0x7f1ff003d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:51:40.547 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.530+0000 7f200df53700 1 --2- 172.21.15.42:0/914963536 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f1ff003aec0 0x7f1ff003d3d0 crc :-1 s=READY pgs=1411 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:51:40.547 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.534+0000 7f20101b7700 1 -- 172.21.15.42:0/914963536 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f200805d110 con 0x7f200811b8c0 2019-12-05T06:51:40.547 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.538+0000 7f200e754700 1 --2- 172.21.15.42:0/914963536 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f200811b8c0 0x7f2008116500 secure :-1 s=THROTTLE_DONE pgs=3368 cs=0 l=1 rx=0x7f200811bfc0 tx=0x7f2004011b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:40.548 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.538+0000 7f1ffeffd700 1 -- 172.21.15.42:0/914963536 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f20040308e0 con 0x7f200811b8c0 2019-12-05T06:51:40.702 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.698+0000 7f20101b7700 1 -- 172.21.15.42:0/914963536 --> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) v1 -- 0x7f200805b8c0 con 0x7f1ff003aec0 2019-12-05T06:51:40.702 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.698+0000 7f1ffeffd700 1 -- 172.21.15.42:0/914963536 <== mgr.4108 v2:172.21.15.42:6800/11930 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) v1 ==== 41+0+0 (crc 0 0 0) 0x7f200805b8c0 con 0x7f1ff003aec0 2019-12-05T06:51:40.703 INFO:teuthology.orchestra.run.smithi042.stderr:instructed osd(s) 0 to deep-scrub 2019-12-05T06:51:40.705 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.702+0000 7f20101b7700 1 -- 172.21.15.42:0/914963536 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f1ff003aec0 msgr2=0x7f1ff003d3d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:40.705 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.702+0000 7f20101b7700 1 --2- 172.21.15.42:0/914963536 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f1ff003aec0 0x7f1ff003d3d0 crc :-1 s=READY pgs=1411 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:51:40.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.702+0000 7f20101b7700 1 -- 172.21.15.42:0/914963536 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f200811b8c0 msgr2=0x7f2008116500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:40.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.702+0000 7f20101b7700 1 --2- 172.21.15.42:0/914963536 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f200811b8c0 0x7f2008116500 secure :-1 s=READY pgs=3368 cs=0 l=1 rx=0x7f200811bfc0 tx=0x7f2004011b30).stop 2019-12-05T06:51:40.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.702+0000 7f20101b7700 1 -- 172.21.15.42:0/914963536 shutdown_connections 2019-12-05T06:51:40.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.702+0000 7f20101b7700 1 --2- 172.21.15.42:0/914963536 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f1ff003aec0 0x7f1ff003d3d0 unknown :-1 s=CLOSED pgs=1411 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:51:40.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.702+0000 7f20101b7700 1 --2- 172.21.15.42:0/914963536 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f200811a700 0x7f200811ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:51:40.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.702+0000 7f20101b7700 1 --2- 172.21.15.42:0/914963536 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f200811b8c0 0x7f2008116500 unknown :-1 s=CLOSED pgs=3368 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:51:40.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.702+0000 7f20101b7700 1 --2- 172.21.15.42:0/914963536 >> v2:172.21.15.40:3300/0 conn(0x7f20081217d0 0x7f2008134030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:51:40.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.702+0000 7f20101b7700 1 -- 172.21.15.42:0/914963536 shutdown_connections 2019-12-05T06:51:40.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.702+0000 7f20101b7700 1 -- 172.21.15.42:0/914963536 wait complete. 2019-12-05T06:51:40.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.702+0000 7f20101b7700 1 -- 172.21.15.42:0/914963536 >> 172.21.15.42:0/914963536 conn(0x7f200811d2d0 msgr2=0x7f2008059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:51:40.719 INFO:tasks.ceph:Scrubbing osd.1 2019-12-05T06:51:40.719 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2019-12-05T06:51:40.826 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.818+0000 7f56c72cb700 1 Processor -- start 2019-12-05T06:51:40.827 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.818+0000 7f56c72cb700 1 -- start start 2019-12-05T06:51:40.827 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.818+0000 7f56c72cb700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f56c009f550 0x7f56c0144630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:51:40.827 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.818+0000 7f56c72cb700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f56c014cf50 0x7f56c014f390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:51:40.828 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.818+0000 7f56c72cb700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56c007f080 con 0x7f56c00a9a00 2019-12-05T06:51:40.828 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.818+0000 7f56c72cb700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56c00a8f80 con 0x7f56c009f550 2019-12-05T06:51:40.828 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.818+0000 7f56c72cb700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56c00a9220 con 0x7f56c014cf50 2019-12-05T06:51:40.828 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.818+0000 7f56c5868700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f56c009f550 0x7f56c0144630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:51:40.828 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.818+0000 7f56c5868700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f56c009f550 0x7f56c0144630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:39286/0 (socket says 172.21.15.42:39286) 2019-12-05T06:51:40.829 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.818+0000 7f56c5868700 1 -- 172.21.15.42:0/1527885322 learned_addr learned my addr 172.21.15.42:0/1527885322 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:51:40.829 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.818+0000 7f56c6069700 1 --2- 172.21.15.42:0/1527885322 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f56c014cf50 0x7f56c014f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:51:40.829 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.818+0000 7f56c5868700 1 -- 172.21.15.42:0/1527885322 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f56c014cf50 msgr2=0x7f56c014f390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:51:40.829 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.818+0000 7f56c5868700 1 --2- 172.21.15.42:0/1527885322 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f56c014cf50 0x7f56c014f390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:51:40.830 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.818+0000 7f56c5868700 1 -- 172.21.15.42:0/1527885322 >> v1:172.21.15.40:6789/0 conn(0x7f56c00a9a00 legacy=0x7f56c007e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:40.830 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.818+0000 7f56c5868700 1 -- 172.21.15.42:0/1527885322 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56c00a1000 con 0x7f56c009f550 2019-12-05T06:51:40.830 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.818+0000 7f56c5868700 1 --2- 172.21.15.42:0/1527885322 >> v2:172.21.15.42:3300/0 conn(0x7f56c009f550 0x7f56c0144630 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f56c01421e0 tx=0x7f56bc0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:40.830 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.818+0000 7f56c5868700 1 --2- 172.21.15.42:0/1527885322 >> v2:172.21.15.42:3300/0 conn(0x7f56c009f550 0x7f56c0144630 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f56c01421e0 tx=0x7f56bc0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:40.830 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.818+0000 7f56c5868700 1 --2- 172.21.15.42:0/1527885322 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f56c009f550 0x7f56c0144630 secure :-1 s=READY pgs=3369 cs=0 l=1 rx=0x7f56c01421e0 tx=0x7f56bc0100b0).ready entity=mon.1 client_cookie=5ccb3604cf63923a server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:51:40.830 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.818+0000 7f56c5868700 1 --2- 172.21.15.42:0/1527885322 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f56c009f550 0x7f56c0144630 secure :-1 s=THROTTLE_DONE pgs=3369 cs=0 l=1 rx=0x7f56c01421e0 tx=0x7f56bc0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:40.831 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.822+0000 7f56c5868700 1 --2- 172.21.15.42:0/1527885322 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f56c009f550 0x7f56c0144630 secure :-1 s=THROTTLE_DONE pgs=3369 cs=0 l=1 rx=0x7f56c01421e0 tx=0x7f56bc0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:40.831 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.822+0000 7f56c4866700 1 -- 172.21.15.42:0/1527885322 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f56bc011dd0 con 0x7f56c009f550 2019-12-05T06:51:40.831 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.822+0000 7f56c5868700 1 --2- 172.21.15.42:0/1527885322 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f56c009f550 0x7f56c0144630 secure :-1 s=THROTTLE_DONE pgs=3369 cs=0 l=1 rx=0x7f56c01421e0 tx=0x7f56bc0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:40.831 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.822+0000 7f56c4866700 1 -- 172.21.15.42:0/1527885322 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56bc0119f0 con 0x7f56c009f550 2019-12-05T06:51:40.831 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.822+0000 7f56c4866700 1 -- 172.21.15.42:0/1527885322 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f56bc00a540 con 0x7f56c009f550 2019-12-05T06:51:40.832 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.822+0000 7f56c72cb700 1 -- 172.21.15.42:0/1527885322 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f56c009f550 msgr2=0x7f56c0144630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:40.832 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.822+0000 7f56c72cb700 1 --2- 172.21.15.42:0/1527885322 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f56c009f550 0x7f56c0144630 secure :-1 s=READY pgs=3369 cs=0 l=1 rx=0x7f56c01421e0 tx=0x7f56bc0100b0).stop 2019-12-05T06:51:40.832 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.822+0000 7f56c72cb700 1 -- 172.21.15.42:0/1527885322 shutdown_connections 2019-12-05T06:51:40.832 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.822+0000 7f56c72cb700 1 --2- 172.21.15.42:0/1527885322 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f56c014cf50 0x7f56c014f390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:51:40.832 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.822+0000 7f56c72cb700 1 --2- 172.21.15.42:0/1527885322 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f56c009f550 0x7f56c0144630 unknown :-1 s=CLOSED pgs=3369 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:51:40.833 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.822+0000 7f56c72cb700 1 -- 172.21.15.42:0/1527885322 shutdown_connections 2019-12-05T06:51:40.833 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.822+0000 7f56c72cb700 1 -- 172.21.15.42:0/1527885322 wait complete. 2019-12-05T06:51:40.833 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.822+0000 7f56c72cb700 1 -- 172.21.15.42:0/1527885322 >> 172.21.15.42:0/1527885322 conn(0x7f56c00a9ef0 msgr2=0x7f56c00aa320 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:51:40.833 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.822+0000 7f56c72cb700 1 Processor -- start 2019-12-05T06:51:40.833 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.822+0000 7f56c72cb700 1 -- start start 2019-12-05T06:51:40.837 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.822+0000 7f56c72cb700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f56c0118860 0x7f56c0119170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:51:40.837 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.822+0000 7f56c72cb700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f56c0144240 0x7f56c007dad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:51:40.838 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.822+0000 7f56c72cb700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f56c007e020 0x7f56c00a1b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:51:40.838 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.822+0000 7f56c72cb700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56c00a1730 con 0x7f56c007e020 2019-12-05T06:51:40.838 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.822+0000 7f56c72cb700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56c0143d00 con 0x7f56c0144240 2019-12-05T06:51:40.838 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.822+0000 7f56c72cb700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56c0143fa0 con 0x7f56c0118860 2019-12-05T06:51:40.838 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.822+0000 7f56c5067700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f56c0144240 0x7f56c007dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:51:40.838 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.826+0000 7f56c5868700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f56c0118860 0x7f56c0119170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:51:40.839 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.826+0000 7f56c6069700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f56c007e020 0x7f56c00a1b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:51:40.839 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.826+0000 7f56c5868700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f56c0118860 0x7f56c0119170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:39756/0 (socket says 172.21.15.42:39756) 2019-12-05T06:51:40.839 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.826+0000 7f56c5868700 1 -- 172.21.15.42:0/3640443372 learned_addr learned my addr 172.21.15.42:0/3640443372 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:51:40.839 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.826+0000 7f56c5067700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f56c0144240 0x7f56c007dad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:39294/0 (socket says 172.21.15.42:39294) 2019-12-05T06:51:40.839 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.826+0000 7f56c5868700 1 -- 172.21.15.42:0/3640443372 >> v2:172.21.15.42:3300/0 conn(0x7f56c0144240 msgr2=0x7f56c007dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:40.839 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.826+0000 7f56c5868700 1 --2- 172.21.15.42:0/3640443372 >> v2:172.21.15.42:3300/0 conn(0x7f56c0144240 0x7f56c007dad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:51:40.840 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.826+0000 7f56c5868700 1 -- 172.21.15.42:0/3640443372 >> v2:172.21.15.40:3300/0 conn(0x7f56c007e020 msgr2=0x7f56c00a1b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:40.840 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.826+0000 7f56c5868700 1 --2- 172.21.15.42:0/3640443372 >> v2:172.21.15.40:3300/0 conn(0x7f56c007e020 0x7f56c00a1b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:51:40.840 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.826+0000 7f56c5868700 1 -- 172.21.15.42:0/3640443372 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56c01196c0 con 0x7f56c0118860 2019-12-05T06:51:40.840 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.826+0000 7f56c6069700 1 --2- 172.21.15.42:0/3640443372 >> v2:172.21.15.40:3300/0 conn(0x7f56c007e020 0x7f56c00a1b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:51:40.841 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.826+0000 7f56c5868700 1 --2- 172.21.15.42:0/3640443372 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f56c0118860 0x7f56c0119170 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f56c01421e0 tx=0x7f56bc00a8d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:40.841 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.826+0000 7f56c5868700 1 --2- 172.21.15.42:0/3640443372 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f56c0118860 0x7f56c0119170 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f56c01421e0 tx=0x7f56bc00a8d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:40.841 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.826+0000 7f56c5868700 1 --2- 172.21.15.42:0/3640443372 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f56c0118860 0x7f56c0119170 secure :-1 s=READY pgs=747 cs=0 l=1 rx=0x7f56c01421e0 tx=0x7f56bc00a8d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:51:40.841 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.826+0000 7f56c5868700 1 --2- 172.21.15.42:0/3640443372 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f56c0118860 0x7f56c0119170 secure :-1 s=THROTTLE_DONE pgs=747 cs=0 l=1 rx=0x7f56c01421e0 tx=0x7f56bc00a8d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:40.841 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.826+0000 7f56c5868700 1 --2- 172.21.15.42:0/3640443372 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f56c0118860 0x7f56c0119170 secure :-1 s=THROTTLE_DONE pgs=747 cs=0 l=1 rx=0x7f56c01421e0 tx=0x7f56bc00a8d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:40.842 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.826+0000 7f56b67fc700 1 -- 172.21.15.42:0/3640443372 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f56bc0058c0 con 0x7f56c0118860 2019-12-05T06:51:40.842 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.826+0000 7f56c5868700 1 --2- 172.21.15.42:0/3640443372 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f56c0118860 0x7f56c0119170 secure :-1 s=THROTTLE_DONE pgs=747 cs=0 l=1 rx=0x7f56c01421e0 tx=0x7f56bc00a8d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:40.842 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.826+0000 7f56b67fc700 1 -- 172.21.15.42:0/3640443372 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56bc00a160 con 0x7f56c0118860 2019-12-05T06:51:40.842 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.826+0000 7f56c72cb700 1 -- 172.21.15.42:0/3640443372 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56c00a2070 con 0x7f56c0118860 2019-12-05T06:51:40.842 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.826+0000 7f56c72cb700 1 -- 172.21.15.42:0/3640443372 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56c01277b0 con 0x7f56c0118860 2019-12-05T06:51:40.842 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.826+0000 7f56b67fc700 1 -- 172.21.15.42:0/3640443372 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f56bc007150 con 0x7f56c0118860 2019-12-05T06:51:40.843 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.826+0000 7f56c5868700 1 --2- 172.21.15.42:0/3640443372 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f56c0118860 0x7f56c0119170 secure :-1 s=THROTTLE_DONE pgs=747 cs=0 l=1 rx=0x7f56c01421e0 tx=0x7f56bc00a8d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:40.843 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.826+0000 7f56b67fc700 1 -- 172.21.15.42:0/3640443372 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f56bc007460 con 0x7f56c0118860 2019-12-05T06:51:40.843 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.826+0000 7f56c5868700 1 --2- 172.21.15.42:0/3640443372 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f56c0118860 0x7f56c0119170 secure :-1 s=THROTTLE_DONE pgs=747 cs=0 l=1 rx=0x7f56c01421e0 tx=0x7f56bc00a8d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:40.843 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.830+0000 7f56b67fc700 1 --2- 172.21.15.42:0/3640443372 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f56a803ae60 0x7f56a803d370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:51:40.843 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.830+0000 7f56b67fc700 1 -- 172.21.15.42:0/3640443372 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 12365+0+0 (secure 0 0 0) 0x7f56bc04be90 con 0x7f56c0118860 2019-12-05T06:51:40.844 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.830+0000 7f56c5067700 1 --2- 172.21.15.42:0/3640443372 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f56a803ae60 0x7f56a803d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:51:40.844 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.830+0000 7f56c5067700 1 --2- 172.21.15.42:0/3640443372 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f56a803ae60 0x7f56a803d370 crc :-1 s=READY pgs=1412 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:51:40.844 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.834+0000 7f56c72cb700 1 --2- 172.21.15.42:0/3640443372 >> [v2:172.21.15.42:6826/12119,v1:172.21.15.42:6827/12119] conn(0x7f56c005a3f0 0x7f56c0127e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:51:40.844 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.834+0000 7f56c72cb700 1 -- 172.21.15.42:0/3640443372 --> [v2:172.21.15.42:6826/12119,v1:172.21.15.42:6827/12119] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f56c005d110 con 0x7f56c005a3f0 2019-12-05T06:51:40.845 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.834+0000 7f56c6069700 1 --2- 172.21.15.42:0/3640443372 >> [v2:172.21.15.42:6826/12119,v1:172.21.15.42:6827/12119] conn(0x7f56c005a3f0 0x7f56c0127e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:51:40.846 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.834+0000 7f56c6069700 1 -- 172.21.15.42:0/3640443372 >> [v2:172.21.15.42:6826/12119,v1:172.21.15.42:6827/12119] conn(0x7f56c005a3f0 msgr2=0x7f56c0127e80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2019-12-05T06:51:40.846 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.834+0000 7f56c6069700 1 --2- 172.21.15.42:0/3640443372 >> [v2:172.21.15.42:6826/12119,v1:172.21.15.42:6827/12119] conn(0x7f56c005a3f0 0x7f56c0127e80 crc :-1 s=READY pgs=208 cs=0 l=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:51:40.846 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.834+0000 7f56b67fc700 1 -- 172.21.15.42:0/3640443372 <== osd.1 v2:172.21.15.42:6826/12119 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23014 (crc 0 0 0) 0x7f56c005d110 con 0x7f56c005a3f0 2019-12-05T06:51:40.872 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.866+0000 7f56c72cb700 1 -- 172.21.15.42:0/3640443372 --> [v2:172.21.15.42:6826/12119,v1:172.21.15.42:6827/12119] -- command(tid 2: {"var": "osd_debug_deep_scrub_sleep", "prefix": "config set", "val": ["0"]}) v1 -- 0x7f56c005b8c0 con 0x7f56c005a3f0 2019-12-05T06:51:40.872 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.870+0000 7f56b67fc700 1 -- 172.21.15.42:0/3640443372 <== osd.1 v2:172.21.15.42:6826/12119 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f56c005b8c0 con 0x7f56c005a3f0 2019-12-05T06:51:40.873 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2019-12-05T06:51:40.873 INFO:teuthology.orchestra.run.smithi042.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2019-12-05T06:51:40.873 INFO:teuthology.orchestra.run.smithi042.stdout:} 2019-12-05T06:51:40.874 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.870+0000 7f56c72cb700 1 -- 172.21.15.42:0/3640443372 >> [v2:172.21.15.42:6826/12119,v1:172.21.15.42:6827/12119] conn(0x7f56c005a3f0 msgr2=0x7f56c0127e80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:40.874 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.870+0000 7f56c72cb700 1 --2- 172.21.15.42:0/3640443372 >> [v2:172.21.15.42:6826/12119,v1:172.21.15.42:6827/12119] conn(0x7f56c005a3f0 0x7f56c0127e80 crc :-1 s=READY pgs=208 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:51:40.875 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.870+0000 7f56c72cb700 1 -- 172.21.15.42:0/3640443372 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f56a803ae60 msgr2=0x7f56a803d370 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:40.875 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.870+0000 7f56c72cb700 1 --2- 172.21.15.42:0/3640443372 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f56a803ae60 0x7f56a803d370 crc :-1 s=READY pgs=1412 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:51:40.875 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.870+0000 7f56c72cb700 1 -- 172.21.15.42:0/3640443372 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f56c0118860 msgr2=0x7f56c0119170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:40.875 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.870+0000 7f56c72cb700 1 --2- 172.21.15.42:0/3640443372 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f56c0118860 0x7f56c0119170 secure :-1 s=READY pgs=747 cs=0 l=1 rx=0x7f56c01421e0 tx=0x7f56bc00a8d0).stop 2019-12-05T06:51:40.875 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.870+0000 7f56c6069700 1 -- 172.21.15.42:0/3640443372 reap_dead start 2019-12-05T06:51:40.875 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.870+0000 7f56c72cb700 1 -- 172.21.15.42:0/3640443372 shutdown_connections 2019-12-05T06:51:40.876 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.870+0000 7f56c72cb700 1 -- 172.21.15.42:0/3640443372 shutdown_connections 2019-12-05T06:51:40.876 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.870+0000 7f56c72cb700 1 -- 172.21.15.42:0/3640443372 wait complete. 2019-12-05T06:51:40.876 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.870+0000 7f56c72cb700 1 -- 172.21.15.42:0/3640443372 >> 172.21.15.42:0/3640443372 conn(0x7f56c00a7420 msgr2=0x7f56c007e760 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:51:40.884 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2019-12-05T06:51:40.991 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.982+0000 7f8de474f700 1 Processor -- start 2019-12-05T06:51:40.991 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.982+0000 7f8de474f700 1 -- start start 2019-12-05T06:51:40.991 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.982+0000 7f8de474f700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8ddc0a7740 0x7f8ddc08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:51:40.991 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.982+0000 7f8de474f700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8ddc09fae0 0x7f8ddc07eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:51:40.992 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.982+0000 7f8de474f700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ddc0a0710 con 0x7f8ddc07e720 2019-12-05T06:51:40.992 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.982+0000 7f8de474f700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ddc0a0b00 con 0x7f8ddc09fae0 2019-12-05T06:51:40.992 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.982+0000 7f8de474f700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ddc08e5b0 con 0x7f8ddc0a7740 2019-12-05T06:51:40.992 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.982+0000 7f8de2cec700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8ddc0a7740 0x7f8ddc08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:51:40.992 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.982+0000 7f8de34ed700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8ddc09fae0 0x7f8ddc07eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:51:40.993 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.982+0000 7f8de2cec700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8ddc0a7740 0x7f8ddc08df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:39766/0 (socket says 172.21.15.42:39766) 2019-12-05T06:51:40.993 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.982+0000 7f8de2cec700 1 -- 172.21.15.42:0/3736614759 learned_addr learned my addr 172.21.15.42:0/3736614759 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:51:40.993 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.982+0000 7f8de2cec700 1 -- 172.21.15.42:0/3736614759 >> v2:172.21.15.40:3300/0 conn(0x7f8ddc09fae0 msgr2=0x7f8ddc07eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:51:40.993 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.982+0000 7f8de2cec700 1 --2- 172.21.15.42:0/3736614759 >> v2:172.21.15.40:3300/0 conn(0x7f8ddc09fae0 0x7f8ddc07eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:51:40.994 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.982+0000 7f8de2cec700 1 -- 172.21.15.42:0/3736614759 >> v1:172.21.15.40:6789/0 conn(0x7f8ddc07e720 legacy=0x7f8ddc09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:40.994 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.982+0000 7f8de2cec700 1 -- 172.21.15.42:0/3736614759 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ddc07f0a0 con 0x7f8ddc0a7740 2019-12-05T06:51:40.994 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.982+0000 7f8de2cec700 1 --2- 172.21.15.42:0/3736614759 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8ddc0a7740 0x7f8ddc08df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f8dd8008790 tx=0x7f8dd800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:40.994 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.982+0000 7f8de2cec700 1 --2- 172.21.15.42:0/3736614759 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8ddc0a7740 0x7f8ddc08df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f8dd8008790 tx=0x7f8dd800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:40.994 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.982+0000 7f8de2cec700 1 --2- 172.21.15.42:0/3736614759 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8ddc0a7740 0x7f8ddc08df80 secure :-1 s=READY pgs=748 cs=0 l=1 rx=0x7f8dd8008790 tx=0x7f8dd800fd30).ready entity=mon.2 client_cookie=1ed2d74ccd1f0487 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:51:40.995 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.982+0000 7f8de2cec700 1 --2- 172.21.15.42:0/3736614759 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8ddc0a7740 0x7f8ddc08df80 secure :-1 s=THROTTLE_DONE pgs=748 cs=0 l=1 rx=0x7f8dd8008790 tx=0x7f8dd800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:40.995 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.982+0000 7f8de2cec700 1 --2- 172.21.15.42:0/3736614759 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8ddc0a7740 0x7f8ddc08df80 secure :-1 s=THROTTLE_DONE pgs=748 cs=0 l=1 rx=0x7f8dd8008790 tx=0x7f8dd800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:40.995 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.982+0000 7f8de1cea700 1 -- 172.21.15.42:0/3736614759 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8dd8008c90 con 0x7f8ddc0a7740 2019-12-05T06:51:40.996 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.982+0000 7f8de2cec700 1 --2- 172.21.15.42:0/3736614759 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8ddc0a7740 0x7f8ddc08df80 secure :-1 s=THROTTLE_DONE pgs=748 cs=0 l=1 rx=0x7f8dd8008790 tx=0x7f8dd800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:40.996 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.982+0000 7f8de1cea700 1 -- 172.21.15.42:0/3736614759 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8dd80088b0 con 0x7f8ddc0a7740 2019-12-05T06:51:40.996 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.982+0000 7f8de1cea700 1 -- 172.21.15.42:0/3736614759 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8dd8011b10 con 0x7f8ddc0a7740 2019-12-05T06:51:40.996 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.982+0000 7f8de474f700 1 -- 172.21.15.42:0/3736614759 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8ddc0a7740 msgr2=0x7f8ddc08df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:41.000 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.982+0000 7f8de474f700 1 --2- 172.21.15.42:0/3736614759 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8ddc0a7740 0x7f8ddc08df80 secure :-1 s=READY pgs=748 cs=0 l=1 rx=0x7f8dd8008790 tx=0x7f8dd800fd30).stop 2019-12-05T06:51:41.001 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.986+0000 7f8de474f700 1 -- 172.21.15.42:0/3736614759 shutdown_connections 2019-12-05T06:51:41.001 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.986+0000 7f8de474f700 1 --2- 172.21.15.42:0/3736614759 >> v2:172.21.15.40:3300/0 conn(0x7f8ddc09fae0 0x7f8ddc07eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:51:41.001 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.986+0000 7f8de474f700 1 --2- 172.21.15.42:0/3736614759 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8ddc0a7740 0x7f8ddc08df80 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:51:41.001 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.986+0000 7f8de474f700 1 -- 172.21.15.42:0/3736614759 shutdown_connections 2019-12-05T06:51:41.001 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.986+0000 7f8de474f700 1 -- 172.21.15.42:0/3736614759 wait complete. 2019-12-05T06:51:41.001 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.986+0000 7f8de474f700 1 -- 172.21.15.42:0/3736614759 >> 172.21.15.42:0/3736614759 conn(0x7f8ddc0aabe0 msgr2=0x7f8ddc0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:51:41.006 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.986+0000 7f8de474f700 1 Processor -- start 2019-12-05T06:51:41.006 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.990+0000 7f8de474f700 1 -- start start 2019-12-05T06:51:41.006 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.990+0000 7f8de474f700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8ddc07bb80 0x7f8ddc11af50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:51:41.006 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.990+0000 7f8de474f700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ddc07bfb0 con 0x7f8ddc1187a0 2019-12-05T06:51:41.006 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.990+0000 7f8de474f700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ddc11b8d0 con 0x7f8ddc11b4a0 2019-12-05T06:51:41.007 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.990+0000 7f8de474f700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ddc07d590 con 0x7f8ddc07bb80 2019-12-05T06:51:41.007 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.990+0000 7f8de24eb700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8ddc07bb80 0x7f8ddc11af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:51:41.007 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.990+0000 7f8de2cec700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f8ddc1187a0 0x7f8ddc07afe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:48784/0 (socket says 172.21.15.42:48784) 2019-12-05T06:51:41.007 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.990+0000 7f8de2cec700 1 -- 172.21.15.42:0/3968697657 learned_addr learned my addr 172.21.15.42:0/3968697657 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:51:41.007 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.990+0000 7f8dd37fe700 1 -- 172.21.15.42:0/3968697657 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2114040687 0 0) 0x7f8ddc11b8d0 con 0x7f8ddc11b4a0 2019-12-05T06:51:41.007 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.990+0000 7f8dd37fe700 1 -- 172.21.15.42:0/3968697657 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8dc40030b0 con 0x7f8ddc11b4a0 2019-12-05T06:51:41.008 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.990+0000 7f8dd37fe700 1 -- 172.21.15.42:0/3968697657 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3153221219 0 0) 0x7f8ddc07bfb0 con 0x7f8ddc1187a0 2019-12-05T06:51:41.008 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.990+0000 7f8de24eb700 1 -- 172.21.15.42:0/3968697657 >> v1:172.21.15.42:6789/0 conn(0x7f8ddc11b4a0 legacy=0x7f8ddc137890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:41.008 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.990+0000 7f8de24eb700 1 -- 172.21.15.42:0/3968697657 >> v1:172.21.15.40:6789/0 conn(0x7f8ddc1187a0 legacy=0x7f8ddc07afe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:41.008 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.990+0000 7f8de24eb700 1 -- 172.21.15.42:0/3968697657 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ddc07d0f0 con 0x7f8ddc07bb80 2019-12-05T06:51:41.008 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.990+0000 7f8de24eb700 1 --2- 172.21.15.42:0/3968697657 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8ddc07bb80 0x7f8ddc11af50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f8ddc11ad20 tx=0x7f8dcc0108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:41.009 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.990+0000 7f8de24eb700 1 --2- 172.21.15.42:0/3968697657 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8ddc07bb80 0x7f8ddc11af50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f8ddc11ad20 tx=0x7f8dcc0108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:41.009 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.990+0000 7f8de24eb700 1 --2- 172.21.15.42:0/3968697657 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8ddc07bb80 0x7f8ddc11af50 secure :-1 s=READY pgs=749 cs=0 l=1 rx=0x7f8ddc11ad20 tx=0x7f8dcc0108f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:51:41.009 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.990+0000 7f8de24eb700 1 --2- 172.21.15.42:0/3968697657 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8ddc07bb80 0x7f8ddc11af50 secure :-1 s=THROTTLE_DONE pgs=749 cs=0 l=1 rx=0x7f8ddc11ad20 tx=0x7f8dcc0108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:41.009 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.990+0000 7f8dd37fe700 1 -- 172.21.15.42:0/3968697657 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8dcc00cb60 con 0x7f8ddc07bb80 2019-12-05T06:51:41.009 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.990+0000 7f8de24eb700 1 --2- 172.21.15.42:0/3968697657 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8ddc07bb80 0x7f8ddc11af50 secure :-1 s=THROTTLE_DONE pgs=749 cs=0 l=1 rx=0x7f8ddc11ad20 tx=0x7f8dcc0108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:41.009 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.990+0000 7f8de24eb700 1 --2- 172.21.15.42:0/3968697657 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8ddc07bb80 0x7f8ddc11af50 secure :-1 s=THROTTLE_DONE pgs=749 cs=0 l=1 rx=0x7f8ddc11ad20 tx=0x7f8dcc0108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:41.010 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.990+0000 7f8dd37fe700 1 -- 172.21.15.42:0/3968697657 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8dcc00c780 con 0x7f8ddc07bb80 2019-12-05T06:51:41.010 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.990+0000 7f8de474f700 1 -- 172.21.15.42:0/3968697657 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ddc1390a0 con 0x7f8ddc07bb80 2019-12-05T06:51:41.010 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.994+0000 7f8dd37fe700 1 -- 172.21.15.42:0/3968697657 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8dcc0110b0 con 0x7f8ddc07bb80 2019-12-05T06:51:41.010 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.994+0000 7f8de474f700 1 -- 172.21.15.42:0/3968697657 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ddc139650 con 0x7f8ddc07bb80 2019-12-05T06:51:41.010 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.994+0000 7f8de24eb700 1 --2- 172.21.15.42:0/3968697657 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8ddc07bb80 0x7f8ddc11af50 secure :-1 s=THROTTLE_DONE pgs=749 cs=0 l=1 rx=0x7f8ddc11ad20 tx=0x7f8dcc0108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:41.011 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.994+0000 7f8dd37fe700 1 -- 172.21.15.42:0/3968697657 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f8dcc0113c0 con 0x7f8ddc07bb80 2019-12-05T06:51:41.011 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.994+0000 7f8de24eb700 1 --2- 172.21.15.42:0/3968697657 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8ddc07bb80 0x7f8ddc11af50 secure :-1 s=THROTTLE_DONE pgs=749 cs=0 l=1 rx=0x7f8ddc11ad20 tx=0x7f8dcc0108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:41.011 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.994+0000 7f8dd37fe700 1 --2- 172.21.15.42:0/3968697657 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8dc403b1b0 0x7f8dc403d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:51:41.011 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.994+0000 7f8dd37fe700 1 -- 172.21.15.42:0/3968697657 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 12365+0+0 (secure 0 0 0) 0x7f8dcc04b070 con 0x7f8ddc07bb80 2019-12-05T06:51:41.011 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.994+0000 7f8de2cec700 1 --2- 172.21.15.42:0/3968697657 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8dc403b1b0 0x7f8dc403d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:51:41.012 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.994+0000 7f8de2cec700 1 --2- 172.21.15.42:0/3968697657 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8dc403b1b0 0x7f8dc403d6c0 crc :-1 s=READY pgs=1413 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:51:41.012 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:40.998+0000 7f8de474f700 1 -- 172.21.15.42:0/3968697657 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ddc05d110 con 0x7f8ddc07bb80 2019-12-05T06:51:41.013 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.006+0000 7f8de24eb700 1 --2- 172.21.15.42:0/3968697657 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8ddc07bb80 0x7f8ddc11af50 secure :-1 s=THROTTLE_DONE pgs=749 cs=0 l=1 rx=0x7f8ddc11ad20 tx=0x7f8dcc0108f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:41.013 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.006+0000 7f8dd37fe700 1 -- 172.21.15.42:0/3968697657 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f8dcc0320f0 con 0x7f8ddc07bb80 2019-12-05T06:51:41.180 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.174+0000 7f8de474f700 1 -- 172.21.15.42:0/3968697657 --> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7f8ddc05b8c0 con 0x7f8dc403b1b0 2019-12-05T06:51:41.180 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.174+0000 7f8dd37fe700 1 -- 172.21.15.42:0/3968697657 <== mgr.4108 v2:172.21.15.42:6800/11930 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) v1 ==== 41+0+0 (crc 0 0 0) 0x7f8ddc05b8c0 con 0x7f8dc403b1b0 2019-12-05T06:51:41.180 INFO:teuthology.orchestra.run.smithi042.stderr:instructed osd(s) 1 to deep-scrub 2019-12-05T06:51:41.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.178+0000 7f8de474f700 1 -- 172.21.15.42:0/3968697657 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8dc403b1b0 msgr2=0x7f8dc403d6c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:41.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.178+0000 7f8de474f700 1 --2- 172.21.15.42:0/3968697657 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8dc403b1b0 0x7f8dc403d6c0 crc :-1 s=READY pgs=1413 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:51:41.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.178+0000 7f8de474f700 1 -- 172.21.15.42:0/3968697657 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8ddc07bb80 msgr2=0x7f8ddc11af50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:41.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.178+0000 7f8de474f700 1 --2- 172.21.15.42:0/3968697657 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8ddc07bb80 0x7f8ddc11af50 secure :-1 s=READY pgs=749 cs=0 l=1 rx=0x7f8ddc11ad20 tx=0x7f8dcc0108f0).stop 2019-12-05T06:51:41.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.178+0000 7f8de474f700 1 -- 172.21.15.42:0/3968697657 shutdown_connections 2019-12-05T06:51:41.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.178+0000 7f8de474f700 1 --2- 172.21.15.42:0/3968697657 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f8dc403b1b0 0x7f8dc403d6c0 unknown :-1 s=CLOSED pgs=1413 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:51:41.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.178+0000 7f8de474f700 1 --2- 172.21.15.42:0/3968697657 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8ddc07bb80 0x7f8ddc11af50 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:51:41.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.178+0000 7f8de474f700 1 -- 172.21.15.42:0/3968697657 shutdown_connections 2019-12-05T06:51:41.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.178+0000 7f8de474f700 1 -- 172.21.15.42:0/3968697657 wait complete. 2019-12-05T06:51:41.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.178+0000 7f8de474f700 1 -- 172.21.15.42:0/3968697657 >> 172.21.15.42:0/3968697657 conn(0x7f8ddc0aabe0 msgr2=0x7f8ddc0a9790 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:51:41.192 INFO:tasks.ceph:Scrubbing osd.2 2019-12-05T06:51:41.192 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2019-12-05T06:51:41.303 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.290+0000 7fcf8aafd700 1 Processor -- start 2019-12-05T06:51:41.304 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.290+0000 7fcf8aafd700 1 -- start start 2019-12-05T06:51:41.304 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.290+0000 7fcf8aafd700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fcf8409f550 0x7fcf84144630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:51:41.304 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.290+0000 7fcf8aafd700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fcf840a9a00 0x7fcf8407e880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:51:41.304 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.290+0000 7fcf8aafd700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcf8407edd0 0x7fcf84147b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:51:41.305 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.290+0000 7fcf8aafd700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf840a8f80 con 0x7fcf8409f550 2019-12-05T06:51:41.305 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.290+0000 7fcf8aafd700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf840a9220 con 0x7fcf840a9a00 2019-12-05T06:51:41.305 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.290+0000 7fcf8aafd700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf840a94c0 con 0x7fcf8407edd0 2019-12-05T06:51:41.305 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.290+0000 7fcf88899700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fcf840a9a00 0x7fcf8407e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:51:41.306 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.290+0000 7fcf88899700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fcf840a9a00 0x7fcf8407e880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:39314/0 (socket says 172.21.15.42:39314) 2019-12-05T06:51:41.306 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.290+0000 7fcf88899700 1 -- 172.21.15.42:0/3370654369 learned_addr learned my addr 172.21.15.42:0/3370654369 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:51:41.307 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.290+0000 7fcf8989b700 1 --2- 172.21.15.42:0/3370654369 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcf8407edd0 0x7fcf84147b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:51:41.307 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.290+0000 7fcf8909a700 1 --2- 172.21.15.42:0/3370654369 >> v2:172.21.15.40:3300/0 conn(0x7fcf8409f550 0x7fcf84144630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:51:41.307 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.290+0000 7fcf88899700 1 -- 172.21.15.42:0/3370654369 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcf8407edd0 msgr2=0x7fcf84147b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:51:41.308 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.290+0000 7fcf88899700 1 --2- 172.21.15.42:0/3370654369 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcf8407edd0 0x7fcf84147b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:51:41.308 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.290+0000 7fcf88899700 1 -- 172.21.15.42:0/3370654369 >> v2:172.21.15.40:3300/0 conn(0x7fcf8409f550 msgr2=0x7fcf84144630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:51:41.308 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.290+0000 7fcf88899700 1 --2- 172.21.15.42:0/3370654369 >> v2:172.21.15.40:3300/0 conn(0x7fcf8409f550 0x7fcf84144630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:51:41.308 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.290+0000 7fcf88899700 1 -- 172.21.15.42:0/3370654369 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf840a1000 con 0x7fcf840a9a00 2019-12-05T06:51:41.308 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.290+0000 7fcf88899700 1 --2- 172.21.15.42:0/3370654369 >> v2:172.21.15.42:3300/0 conn(0x7fcf840a9a00 0x7fcf8407e880 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fcf840a9140 tx=0x7fcf7400fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:41.310 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.290+0000 7fcf88899700 1 --2- 172.21.15.42:0/3370654369 >> v2:172.21.15.42:3300/0 conn(0x7fcf840a9a00 0x7fcf8407e880 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fcf840a9140 tx=0x7fcf7400fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:41.310 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.290+0000 7fcf88899700 1 --2- 172.21.15.42:0/3370654369 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcf840a9a00 0x7fcf8407e880 secure :-1 s=READY pgs=3371 cs=0 l=1 rx=0x7fcf840a9140 tx=0x7fcf7400fdb0).ready entity=mon.1 client_cookie=f05e07b899256d86 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:51:41.310 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.290+0000 7fcf88899700 1 --2- 172.21.15.42:0/3370654369 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcf840a9a00 0x7fcf8407e880 secure :-1 s=THROTTLE_DONE pgs=3371 cs=0 l=1 rx=0x7fcf840a9140 tx=0x7fcf7400fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:41.310 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.290+0000 7fcf88899700 1 --2- 172.21.15.42:0/3370654369 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcf840a9a00 0x7fcf8407e880 secure :-1 s=THROTTLE_DONE pgs=3371 cs=0 l=1 rx=0x7fcf840a9140 tx=0x7fcf7400fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:41.310 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.290+0000 7fcf7bfff700 1 -- 172.21.15.42:0/3370654369 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcf740113f0 con 0x7fcf840a9a00 2019-12-05T06:51:41.310 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.290+0000 7fcf88899700 1 --2- 172.21.15.42:0/3370654369 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcf840a9a00 0x7fcf8407e880 secure :-1 s=THROTTLE_DONE pgs=3371 cs=0 l=1 rx=0x7fcf840a9140 tx=0x7fcf7400fdb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:41.311 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.290+0000 7fcf7bfff700 1 -- 172.21.15.42:0/3370654369 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf74008bb0 con 0x7fcf840a9a00 2019-12-05T06:51:41.311 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.290+0000 7fcf7bfff700 1 -- 172.21.15.42:0/3370654369 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcf740119d0 con 0x7fcf840a9a00 2019-12-05T06:51:41.311 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.290+0000 7fcf8aafd700 1 -- 172.21.15.42:0/3370654369 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcf840a9a00 msgr2=0x7fcf8407e880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:41.311 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.290+0000 7fcf8aafd700 1 --2- 172.21.15.42:0/3370654369 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcf840a9a00 0x7fcf8407e880 secure :-1 s=READY pgs=3371 cs=0 l=1 rx=0x7fcf840a9140 tx=0x7fcf7400fdb0).stop 2019-12-05T06:51:41.311 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.294+0000 7fcf8aafd700 1 -- 172.21.15.42:0/3370654369 shutdown_connections 2019-12-05T06:51:41.311 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.294+0000 7fcf8aafd700 1 --2- 172.21.15.42:0/3370654369 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcf8407edd0 0x7fcf84147b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:51:41.312 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.294+0000 7fcf8aafd700 1 --2- 172.21.15.42:0/3370654369 >> v2:172.21.15.40:3300/0 conn(0x7fcf8409f550 0x7fcf84144630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:51:41.312 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.294+0000 7fcf8aafd700 1 --2- 172.21.15.42:0/3370654369 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcf840a9a00 0x7fcf8407e880 unknown :-1 s=CLOSED pgs=3371 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:51:41.312 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.294+0000 7fcf8aafd700 1 -- 172.21.15.42:0/3370654369 shutdown_connections 2019-12-05T06:51:41.312 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.294+0000 7fcf8aafd700 1 -- 172.21.15.42:0/3370654369 wait complete. 2019-12-05T06:51:41.312 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.294+0000 7fcf8aafd700 1 -- 172.21.15.42:0/3370654369 >> 172.21.15.42:0/3370654369 conn(0x7fcf840a9ef0 msgr2=0x7fcf840aa320 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:51:41.312 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.294+0000 7fcf8aafd700 1 Processor -- start 2019-12-05T06:51:41.312 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.294+0000 7fcf8aafd700 1 -- start start 2019-12-05T06:51:41.313 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.294+0000 7fcf8aafd700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fcf840a1af0 0x7fcf840a3260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:51:41.313 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.294+0000 7fcf8aafd700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fcf84147440 0x7fcf84147870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:51:41.313 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.294+0000 7fcf8aafd700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf8412be60 con 0x7fcf8412ba30 2019-12-05T06:51:41.313 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.294+0000 7fcf8aafd700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf840a1f20 con 0x7fcf84147440 2019-12-05T06:51:41.313 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.294+0000 7fcf8aafd700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf8412b250 con 0x7fcf840a1af0 2019-12-05T06:51:41.313 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.294+0000 7fcf8909a700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fcf840a1af0 0x7fcf840a3260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:51:41.314 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.294+0000 7fcf88899700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fcf8412ba30 0x7fcf84148360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:39360/0 (socket says 172.21.15.42:39360) 2019-12-05T06:51:41.314 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.294+0000 7fcf88899700 1 -- 172.21.15.42:0/4214886532 learned_addr learned my addr 172.21.15.42:0/4214886532 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:51:41.314 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.294+0000 7fcf8989b700 1 --2- 172.21.15.42:0/4214886532 >> v2:172.21.15.40:3300/0 conn(0x7fcf84147440 0x7fcf84147870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:51:41.314 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.294+0000 7fcf8909a700 1 -- 172.21.15.42:0/4214886532 >> v2:172.21.15.40:3300/0 conn(0x7fcf84147440 msgr2=0x7fcf84147870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:41.314 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.294+0000 7fcf8909a700 1 --2- 172.21.15.42:0/4214886532 >> v2:172.21.15.40:3300/0 conn(0x7fcf84147440 0x7fcf84147870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:51:41.314 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.294+0000 7fcf79ffb700 1 -- 172.21.15.42:0/4214886532 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2551384368 0 0) 0x7fcf8412be60 con 0x7fcf8412ba30 2019-12-05T06:51:41.315 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.294+0000 7fcf8909a700 1 -- 172.21.15.42:0/4214886532 >> v1:172.21.15.42:6789/0 conn(0x7fcf8412ba30 legacy=0x7fcf84148360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:41.320 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.294+0000 7fcf8909a700 1 -- 172.21.15.42:0/4214886532 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf84136a80 con 0x7fcf840a1af0 2019-12-05T06:51:41.320 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.294+0000 7fcf8909a700 1 --2- 172.21.15.42:0/4214886532 >> v2:172.21.15.42:3300/0 conn(0x7fcf840a1af0 0x7fcf840a3260 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fcf80002f30 tx=0x7fcf80010a50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:41.321 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.298+0000 7fcf8909a700 1 --2- 172.21.15.42:0/4214886532 >> v2:172.21.15.42:3300/0 conn(0x7fcf840a1af0 0x7fcf840a3260 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fcf80002f30 tx=0x7fcf80010a50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:41.321 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.298+0000 7fcf8909a700 1 --2- 172.21.15.42:0/4214886532 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcf840a1af0 0x7fcf840a3260 secure :-1 s=READY pgs=3373 cs=0 l=1 rx=0x7fcf80002f30 tx=0x7fcf80010a50).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:51:41.321 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.298+0000 7fcf8909a700 1 --2- 172.21.15.42:0/4214886532 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcf840a1af0 0x7fcf840a3260 secure :-1 s=THROTTLE_DONE pgs=3373 cs=0 l=1 rx=0x7fcf80002f30 tx=0x7fcf80010a50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:41.321 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.298+0000 7fcf8909a700 1 --2- 172.21.15.42:0/4214886532 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcf840a1af0 0x7fcf840a3260 secure :-1 s=THROTTLE_DONE pgs=3373 cs=0 l=1 rx=0x7fcf80002f30 tx=0x7fcf80010a50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:41.321 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.298+0000 7fcf79ffb700 1 -- 172.21.15.42:0/4214886532 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcf80005920 con 0x7fcf840a1af0 2019-12-05T06:51:41.322 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.298+0000 7fcf8909a700 1 --2- 172.21.15.42:0/4214886532 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcf840a1af0 0x7fcf840a3260 secure :-1 s=THROTTLE_DONE pgs=3373 cs=0 l=1 rx=0x7fcf80002f30 tx=0x7fcf80010a50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:41.322 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.298+0000 7fcf79ffb700 1 -- 172.21.15.42:0/4214886532 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf80005540 con 0x7fcf840a1af0 2019-12-05T06:51:41.322 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.298+0000 7fcf79ffb700 1 -- 172.21.15.42:0/4214886532 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcf80012500 con 0x7fcf840a1af0 2019-12-05T06:51:41.322 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.298+0000 7fcf8aafd700 1 -- 172.21.15.42:0/4214886532 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf84135a70 con 0x7fcf840a1af0 2019-12-05T06:51:41.322 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.298+0000 7fcf8aafd700 1 -- 172.21.15.42:0/4214886532 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf84135fc0 con 0x7fcf840a1af0 2019-12-05T06:51:41.322 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.298+0000 7fcf8909a700 1 --2- 172.21.15.42:0/4214886532 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcf840a1af0 0x7fcf840a3260 secure :-1 s=THROTTLE_DONE pgs=3373 cs=0 l=1 rx=0x7fcf80002f30 tx=0x7fcf80010a50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:41.322 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.298+0000 7fcf79ffb700 1 -- 172.21.15.42:0/4214886532 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fcf80007180 con 0x7fcf840a1af0 2019-12-05T06:51:41.323 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.298+0000 7fcf8909a700 1 --2- 172.21.15.42:0/4214886532 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcf840a1af0 0x7fcf840a3260 secure :-1 s=THROTTLE_DONE pgs=3373 cs=0 l=1 rx=0x7fcf80002f30 tx=0x7fcf80010a50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:41.323 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.298+0000 7fcf79ffb700 1 --2- 172.21.15.42:0/4214886532 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fcf6403ac80 0x7fcf6403d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:51:41.323 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.298+0000 7fcf79ffb700 1 -- 172.21.15.42:0/4214886532 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 12365+0+0 (secure 0 0 0) 0x7fcf8004ae50 con 0x7fcf840a1af0 2019-12-05T06:51:41.323 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.298+0000 7fcf88899700 1 --2- 172.21.15.42:0/4214886532 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fcf6403ac80 0x7fcf6403d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:51:41.323 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.298+0000 7fcf88899700 1 --2- 172.21.15.42:0/4214886532 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fcf6403ac80 0x7fcf6403d190 crc :-1 s=READY pgs=1414 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:51:41.324 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.306+0000 7fcf8aafd700 1 --2- 172.21.15.42:0/4214886532 >> [v2:172.21.15.42:6802/12117,v1:172.21.15.42:6803/12117] conn(0x7fcf841222b0 0x7fcf84054260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:51:41.324 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.306+0000 7fcf8aafd700 1 -- 172.21.15.42:0/4214886532 --> [v2:172.21.15.42:6802/12117,v1:172.21.15.42:6803/12117] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcf8412b060 con 0x7fcf841222b0 2019-12-05T06:51:41.325 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.306+0000 7fcf8989b700 1 --2- 172.21.15.42:0/4214886532 >> [v2:172.21.15.42:6802/12117,v1:172.21.15.42:6803/12117] conn(0x7fcf841222b0 0x7fcf84054260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:51:41.325 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.306+0000 7fcf8989b700 1 -- 172.21.15.42:0/4214886532 >> [v2:172.21.15.42:6802/12117,v1:172.21.15.42:6803/12117] conn(0x7fcf841222b0 msgr2=0x7fcf84054260 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2019-12-05T06:51:41.325 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.306+0000 7fcf8989b700 1 --2- 172.21.15.42:0/4214886532 >> [v2:172.21.15.42:6802/12117,v1:172.21.15.42:6803/12117] conn(0x7fcf841222b0 0x7fcf84054260 crc :-1 s=READY pgs=120 cs=0 l=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:51:41.325 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.306+0000 7fcf79ffb700 1 -- 172.21.15.42:0/4214886532 <== osd.2 v2:172.21.15.42:6802/12117 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23014 (crc 0 0 0) 0x7fcf7c007260 con 0x7fcf841222b0 2019-12-05T06:51:41.345 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.342+0000 7fcf8aafd700 1 -- 172.21.15.42:0/4214886532 --> [v2:172.21.15.42:6802/12117,v1:172.21.15.42:6803/12117] -- command(tid 2: {"var": "osd_debug_deep_scrub_sleep", "prefix": "config set", "val": ["0"]}) v1 -- 0x7fcf840a37b0 con 0x7fcf841222b0 2019-12-05T06:51:41.346 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.342+0000 7fcf79ffb700 1 -- 172.21.15.42:0/4214886532 <== osd.2 v2:172.21.15.42:6802/12117 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7fcf840a37b0 con 0x7fcf841222b0 2019-12-05T06:51:41.346 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2019-12-05T06:51:41.346 INFO:teuthology.orchestra.run.smithi042.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2019-12-05T06:51:41.346 INFO:teuthology.orchestra.run.smithi042.stdout:} 2019-12-05T06:51:41.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.342+0000 7fcf8aafd700 1 -- 172.21.15.42:0/4214886532 >> [v2:172.21.15.42:6802/12117,v1:172.21.15.42:6803/12117] conn(0x7fcf841222b0 msgr2=0x7fcf84054260 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:41.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.342+0000 7fcf8aafd700 1 --2- 172.21.15.42:0/4214886532 >> [v2:172.21.15.42:6802/12117,v1:172.21.15.42:6803/12117] conn(0x7fcf841222b0 0x7fcf84054260 crc :-1 s=READY pgs=120 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:51:41.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.342+0000 7fcf8aafd700 1 -- 172.21.15.42:0/4214886532 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fcf6403ac80 msgr2=0x7fcf6403d190 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:41.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.342+0000 7fcf8aafd700 1 --2- 172.21.15.42:0/4214886532 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fcf6403ac80 0x7fcf6403d190 crc :-1 s=READY pgs=1414 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:51:41.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.342+0000 7fcf8aafd700 1 -- 172.21.15.42:0/4214886532 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcf840a1af0 msgr2=0x7fcf840a3260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:41.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.342+0000 7fcf8aafd700 1 --2- 172.21.15.42:0/4214886532 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcf840a1af0 0x7fcf840a3260 secure :-1 s=READY pgs=3373 cs=0 l=1 rx=0x7fcf80002f30 tx=0x7fcf80010a50).stop 2019-12-05T06:51:41.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.342+0000 7fcf8989b700 1 -- 172.21.15.42:0/4214886532 reap_dead start 2019-12-05T06:51:41.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.342+0000 7fcf8aafd700 1 -- 172.21.15.42:0/4214886532 shutdown_connections 2019-12-05T06:51:41.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.342+0000 7fcf8aafd700 1 --2- 172.21.15.42:0/4214886532 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcf840a1af0 0x7fcf840a3260 unknown :-1 s=CLOSED pgs=3373 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:51:41.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.342+0000 7fcf8aafd700 1 -- 172.21.15.42:0/4214886532 shutdown_connections 2019-12-05T06:51:41.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.342+0000 7fcf8aafd700 1 -- 172.21.15.42:0/4214886532 wait complete. 2019-12-05T06:51:41.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.342+0000 7fcf8aafd700 1 -- 172.21.15.42:0/4214886532 >> 172.21.15.42:0/4214886532 conn(0x7fcf840a9960 msgr2=0x7fcf840a6480 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:51:41.356 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2019-12-05T06:51:41.461 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.450+0000 7f2819716700 1 Processor -- start 2019-12-05T06:51:41.461 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.450+0000 7f2819716700 1 -- start start 2019-12-05T06:51:41.461 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.454+0000 7f2819716700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f28140a7730 0x7f281408df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:51:41.462 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.454+0000 7f2819716700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f281407e710 0x7f281407eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:51:41.462 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.454+0000 7f2819716700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f281409f2d0 0x7f281409f700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:51:41.462 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.454+0000 7f2819716700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28140a0af0 con 0x7f281409f2d0 2019-12-05T06:51:41.462 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.454+0000 7f2819716700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f281408e5a0 con 0x7f281407e710 2019-12-05T06:51:41.462 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.454+0000 7f2819716700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f281408e840 con 0x7f28140a7730 2019-12-05T06:51:41.462 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.454+0000 7f2812ffd700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f281407e710 0x7f281407eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:51:41.463 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.454+0000 7f2812ffd700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f281407e710 0x7f281407eb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:39332/0 (socket says 172.21.15.42:39332) 2019-12-05T06:51:41.463 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.454+0000 7f2812ffd700 1 -- 172.21.15.42:0/729588135 learned_addr learned my addr 172.21.15.42:0/729588135 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:51:41.463 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.454+0000 7f28137fe700 1 --2- 172.21.15.42:0/729588135 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f28140a7730 0x7f281408df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:51:41.463 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.454+0000 7f2813fff700 1 --2- 172.21.15.42:0/729588135 >> v2:172.21.15.40:3300/0 conn(0x7f281409f2d0 0x7f281409f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:51:41.463 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.454+0000 7f28137fe700 1 -- 172.21.15.42:0/729588135 >> v2:172.21.15.42:3300/0 conn(0x7f281407e710 msgr2=0x7f281407eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:51:41.464 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.454+0000 7f28137fe700 1 --2- 172.21.15.42:0/729588135 >> v2:172.21.15.42:3300/0 conn(0x7f281407e710 0x7f281407eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:51:41.464 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.454+0000 7f28137fe700 1 -- 172.21.15.42:0/729588135 >> v2:172.21.15.40:3300/0 conn(0x7f281409f2d0 msgr2=0x7f281409f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:51:41.464 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.454+0000 7f28137fe700 1 --2- 172.21.15.42:0/729588135 >> v2:172.21.15.40:3300/0 conn(0x7f281409f2d0 0x7f281409f700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:51:41.464 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.454+0000 7f28137fe700 1 -- 172.21.15.42:0/729588135 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f281407f090 con 0x7f28140a7730 2019-12-05T06:51:41.464 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.454+0000 7f2812ffd700 1 --2- 172.21.15.42:0/729588135 >> v2:172.21.15.42:3300/0 conn(0x7f281407e710 0x7f281407eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:51:41.464 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.454+0000 7f28137fe700 1 --2- 172.21.15.42:0/729588135 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f28140a7730 0x7f281408df70 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f280c0032b0 tx=0x7f280c010070).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:41.464 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.454+0000 7f28137fe700 1 --2- 172.21.15.42:0/729588135 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f28140a7730 0x7f281408df70 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f280c0032b0 tx=0x7f280c010070).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:41.465 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.454+0000 7f28137fe700 1 --2- 172.21.15.42:0/729588135 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f28140a7730 0x7f281408df70 secure :-1 s=READY pgs=750 cs=0 l=1 rx=0x7f280c0032b0 tx=0x7f280c010070).ready entity=mon.2 client_cookie=2f2ca9cffdcf39d0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:51:41.465 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.454+0000 7f28137fe700 1 --2- 172.21.15.42:0/729588135 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f28140a7730 0x7f281408df70 secure :-1 s=THROTTLE_DONE pgs=750 cs=0 l=1 rx=0x7f280c0032b0 tx=0x7f280c010070).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:41.465 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.454+0000 7f28137fe700 1 --2- 172.21.15.42:0/729588135 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f28140a7730 0x7f281408df70 secure :-1 s=THROTTLE_DONE pgs=750 cs=0 l=1 rx=0x7f280c0032b0 tx=0x7f280c010070).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:41.465 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.454+0000 7f28137fe700 1 --2- 172.21.15.42:0/729588135 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f28140a7730 0x7f281408df70 secure :-1 s=THROTTLE_DONE pgs=750 cs=0 l=1 rx=0x7f280c0032b0 tx=0x7f280c010070).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:41.465 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.454+0000 7f28127fc700 1 -- 172.21.15.42:0/729588135 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f280c010940 con 0x7f28140a7730 2019-12-05T06:51:41.465 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.454+0000 7f28127fc700 1 -- 172.21.15.42:0/729588135 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f280c008b50 con 0x7f28140a7730 2019-12-05T06:51:41.465 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.454+0000 7f28127fc700 1 -- 172.21.15.42:0/729588135 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f280c0110c0 con 0x7f28140a7730 2019-12-05T06:51:41.466 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.454+0000 7f2819716700 1 -- 172.21.15.42:0/729588135 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f28140a7730 msgr2=0x7f281408df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:41.466 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.454+0000 7f2819716700 1 --2- 172.21.15.42:0/729588135 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f28140a7730 0x7f281408df70 secure :-1 s=READY pgs=750 cs=0 l=1 rx=0x7f280c0032b0 tx=0x7f280c010070).stop 2019-12-05T06:51:41.466 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.454+0000 7f2819716700 1 -- 172.21.15.42:0/729588135 shutdown_connections 2019-12-05T06:51:41.466 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.454+0000 7f2819716700 1 --2- 172.21.15.42:0/729588135 >> v2:172.21.15.40:3300/0 conn(0x7f281409f2d0 0x7f281409f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:51:41.466 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.454+0000 7f2819716700 1 --2- 172.21.15.42:0/729588135 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f28140a7730 0x7f281408df70 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:51:41.466 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.454+0000 7f2819716700 1 --2- 172.21.15.42:0/729588135 >> v2:172.21.15.42:3300/0 conn(0x7f281407e710 0x7f281407eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:51:41.466 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.454+0000 7f2819716700 1 -- 172.21.15.42:0/729588135 shutdown_connections 2019-12-05T06:51:41.467 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.454+0000 7f2819716700 1 -- 172.21.15.42:0/729588135 wait complete. 2019-12-05T06:51:41.467 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.454+0000 7f2819716700 1 -- 172.21.15.42:0/729588135 >> 172.21.15.42:0/729588135 conn(0x7f28140aabd0 msgr2=0x7f28140a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:51:41.476 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.458+0000 7f2819716700 1 Processor -- start 2019-12-05T06:51:41.476 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.458+0000 7f2819716700 1 -- start start 2019-12-05T06:51:41.476 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.458+0000 7f2819716700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f28141191e0 0x7f2814124670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:51:41.477 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.458+0000 7f2819716700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2814120c50 0x7f2814131760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:51:41.477 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.458+0000 7f2819716700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28141320e0 con 0x7f2814131cb0 2019-12-05T06:51:41.477 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.458+0000 7f2819716700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28141211d0 con 0x7f2814120c50 2019-12-05T06:51:41.477 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.458+0000 7f2819716700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2814121470 con 0x7f28141191e0 2019-12-05T06:51:41.477 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.458+0000 7f28137fe700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f28141191e0 0x7f2814124670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:51:41.477 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.458+0000 7f28137fe700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f28141191e0 0x7f2814124670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:39802/0 (socket says 172.21.15.42:39802) 2019-12-05T06:51:41.477 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.458+0000 7f28137fe700 1 -- 172.21.15.42:0/1528210487 learned_addr learned my addr 172.21.15.42:0/1528210487 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:51:41.478 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.458+0000 7f2812ffd700 1 --2- 172.21.15.42:0/1528210487 >> v2:172.21.15.40:3300/0 conn(0x7f2814120c50 0x7f2814131760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:51:41.478 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.462+0000 7f28137fe700 1 -- 172.21.15.42:0/1528210487 >> v2:172.21.15.40:3300/0 conn(0x7f2814120c50 msgr2=0x7f2814131760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:41.478 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.462+0000 7f28137fe700 1 --2- 172.21.15.42:0/1528210487 >> v2:172.21.15.40:3300/0 conn(0x7f2814120c50 0x7f2814131760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:51:41.478 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.462+0000 7f28137fe700 1 -- 172.21.15.42:0/1528210487 >> v1:172.21.15.40:6789/0 conn(0x7f2814131cb0 legacy=0x7f281408eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:41.478 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.462+0000 7f28137fe700 1 -- 172.21.15.42:0/1528210487 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f281408f360 con 0x7f28141191e0 2019-12-05T06:51:41.478 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.462+0000 7f28137fe700 1 --2- 172.21.15.42:0/1528210487 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f28141191e0 0x7f2814124670 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f280c0032b0 tx=0x7f280c007960).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:41.478 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.462+0000 7f28137fe700 1 --2- 172.21.15.42:0/1528210487 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f28141191e0 0x7f2814124670 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f280c0032b0 tx=0x7f280c007960).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:41.479 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.462+0000 7f28137fe700 1 --2- 172.21.15.42:0/1528210487 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f28141191e0 0x7f2814124670 secure :-1 s=READY pgs=751 cs=0 l=1 rx=0x7f280c0032b0 tx=0x7f280c007960).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:51:41.479 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.462+0000 7f28137fe700 1 --2- 172.21.15.42:0/1528210487 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f28141191e0 0x7f2814124670 secure :-1 s=THROTTLE_DONE pgs=751 cs=0 l=1 rx=0x7f280c0032b0 tx=0x7f280c007960).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:41.479 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.462+0000 7f280bfff700 1 -- 172.21.15.42:0/1528210487 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f280c005cc0 con 0x7f28141191e0 2019-12-05T06:51:41.479 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.462+0000 7f28137fe700 1 --2- 172.21.15.42:0/1528210487 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f28141191e0 0x7f2814124670 secure :-1 s=THROTTLE_DONE pgs=751 cs=0 l=1 rx=0x7f280c0032b0 tx=0x7f280c007960).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:41.479 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.462+0000 7f280bfff700 1 -- 172.21.15.42:0/1528210487 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f280c010b40 con 0x7f28141191e0 2019-12-05T06:51:41.479 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.462+0000 7f28137fe700 1 --2- 172.21.15.42:0/1528210487 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f28141191e0 0x7f2814124670 secure :-1 s=THROTTLE_DONE pgs=751 cs=0 l=1 rx=0x7f280c0032b0 tx=0x7f280c007960).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:41.479 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.462+0000 7f2819716700 1 -- 172.21.15.42:0/1528210487 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f281408f5d0 con 0x7f28141191e0 2019-12-05T06:51:41.479 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.462+0000 7f280bfff700 1 -- 172.21.15.42:0/1528210487 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f280c007540 con 0x7f28141191e0 2019-12-05T06:51:41.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.462+0000 7f2819716700 1 -- 172.21.15.42:0/1528210487 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28140752d0 con 0x7f28141191e0 2019-12-05T06:51:41.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.462+0000 7f28137fe700 1 --2- 172.21.15.42:0/1528210487 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f28141191e0 0x7f2814124670 secure :-1 s=THROTTLE_DONE pgs=751 cs=0 l=1 rx=0x7f280c0032b0 tx=0x7f280c007960).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:41.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.462+0000 7f280bfff700 1 -- 172.21.15.42:0/1528210487 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f280c00f220 con 0x7f28141191e0 2019-12-05T06:51:41.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.462+0000 7f28137fe700 1 --2- 172.21.15.42:0/1528210487 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f28141191e0 0x7f2814124670 secure :-1 s=THROTTLE_DONE pgs=751 cs=0 l=1 rx=0x7f280c0032b0 tx=0x7f280c007960).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:41.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.462+0000 7f280bfff700 1 --2- 172.21.15.42:0/1528210487 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f27f803adc0 0x7f27f803d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:51:41.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.462+0000 7f280bfff700 1 -- 172.21.15.42:0/1528210487 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 12365+0+0 (secure 0 0 0) 0x7f280c08edb0 con 0x7f28141191e0 2019-12-05T06:51:41.480 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.462+0000 7f2812ffd700 1 --2- 172.21.15.42:0/1528210487 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f27f803adc0 0x7f27f803d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:51:41.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.462+0000 7f2812ffd700 1 --2- 172.21.15.42:0/1528210487 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f27f803adc0 0x7f27f803d2d0 crc :-1 s=READY pgs=1415 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:51:41.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.466+0000 7f2819716700 1 -- 172.21.15.42:0/1528210487 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f281405d100 con 0x7f28141191e0 2019-12-05T06:51:41.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.474+0000 7f28137fe700 1 --2- 172.21.15.42:0/1528210487 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f28141191e0 0x7f2814124670 secure :-1 s=THROTTLE_DONE pgs=751 cs=0 l=1 rx=0x7f280c0032b0 tx=0x7f280c007960).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:41.481 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.474+0000 7f280bfff700 1 -- 172.21.15.42:0/1528210487 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f280c0082d0 con 0x7f28141191e0 2019-12-05T06:51:41.646 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.642+0000 7f2819716700 1 -- 172.21.15.42:0/1528210487 --> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) v1 -- 0x7f281405b8b0 con 0x7f27f803adc0 2019-12-05T06:51:41.646 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.642+0000 7f280bfff700 1 -- 172.21.15.42:0/1528210487 <== mgr.4108 v2:172.21.15.42:6800/11930 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) v1 ==== 41+0+0 (crc 0 0 0) 0x7f281405b8b0 con 0x7f27f803adc0 2019-12-05T06:51:41.646 INFO:teuthology.orchestra.run.smithi042.stderr:instructed osd(s) 2 to deep-scrub 2019-12-05T06:51:41.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.646+0000 7f2819716700 1 -- 172.21.15.42:0/1528210487 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f27f803adc0 msgr2=0x7f27f803d2d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:41.649 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.646+0000 7f2819716700 1 --2- 172.21.15.42:0/1528210487 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f27f803adc0 0x7f27f803d2d0 crc :-1 s=READY pgs=1415 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:51:41.650 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.646+0000 7f2819716700 1 -- 172.21.15.42:0/1528210487 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f28141191e0 msgr2=0x7f2814124670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:41.650 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.646+0000 7f2819716700 1 --2- 172.21.15.42:0/1528210487 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f28141191e0 0x7f2814124670 secure :-1 s=READY pgs=751 cs=0 l=1 rx=0x7f280c0032b0 tx=0x7f280c007960).stop 2019-12-05T06:51:41.650 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.646+0000 7f2819716700 1 -- 172.21.15.42:0/1528210487 shutdown_connections 2019-12-05T06:51:41.650 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.646+0000 7f2819716700 1 --2- 172.21.15.42:0/1528210487 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f27f803adc0 0x7f27f803d2d0 unknown :-1 s=CLOSED pgs=1415 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:51:41.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.646+0000 7f2819716700 1 --2- 172.21.15.42:0/1528210487 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f28141191e0 0x7f2814124670 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:51:41.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.646+0000 7f2819716700 1 --2- 172.21.15.42:0/1528210487 >> v2:172.21.15.40:3300/0 conn(0x7f2814120c50 0x7f2814131760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:51:41.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.646+0000 7f2819716700 1 -- 172.21.15.42:0/1528210487 shutdown_connections 2019-12-05T06:51:41.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.646+0000 7f2819716700 1 -- 172.21.15.42:0/1528210487 wait complete. 2019-12-05T06:51:41.651 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.646+0000 7f2819716700 1 -- 172.21.15.42:0/1528210487 >> 172.21.15.42:0/1528210487 conn(0x7f28140a8290 msgr2=0x7f28140a9780 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:51:41.660 INFO:tasks.ceph:Scrubbing osd.3 2019-12-05T06:51:41.660 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2019-12-05T06:51:41.774 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.762+0000 7fc900b1a700 1 Processor -- start 2019-12-05T06:51:41.775 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.762+0000 7fc900b1a700 1 -- start start 2019-12-05T06:51:41.775 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.762+0000 7fc900b1a700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc8fc09f550 0x7fc8fc144630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:51:41.775 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.762+0000 7fc900b1a700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc8fc0a9a00 0x7fc8fc07e880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:51:41.775 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.762+0000 7fc900b1a700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8fc0a1000 con 0x7fc8fc07edd0 2019-12-05T06:51:41.775 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.762+0000 7fc900b1a700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8fc0a8f80 con 0x7fc8fc09f550 2019-12-05T06:51:41.775 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.762+0000 7fc900b1a700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8fc0a9220 con 0x7fc8fc0a9a00 2019-12-05T06:51:41.776 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.762+0000 7fc8fad9d700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc8fc09f550 0x7fc8fc144630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:51:41.776 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.762+0000 7fc8fa59c700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc8fc0a9a00 0x7fc8fc07e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:51:41.776 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.762+0000 7fc8fad9d700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc8fc09f550 0x7fc8fc144630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.42:55392/0 (socket says 172.21.15.42:55392) 2019-12-05T06:51:41.776 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.762+0000 7fc8fad9d700 1 -- 172.21.15.42:0/97121153 learned_addr learned my addr 172.21.15.42:0/97121153 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:51:41.776 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.762+0000 7fc8fa59c700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc8fc0a9a00 0x7fc8fc07e880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:39812/0 (socket says 172.21.15.42:39812) 2019-12-05T06:51:41.776 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.762+0000 7fc8fad9d700 1 -- 172.21.15.42:0/97121153 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc8fc0a9a00 msgr2=0x7fc8fc07e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:51:41.776 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.762+0000 7fc8fad9d700 1 --2- 172.21.15.42:0/97121153 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc8fc0a9a00 0x7fc8fc07e880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:51:41.777 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.762+0000 7fc8fad9d700 1 -- 172.21.15.42:0/97121153 >> v1:172.21.15.40:6789/0 conn(0x7fc8fc07edd0 legacy=0x7fc8fc147b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:41.777 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.762+0000 7fc8fad9d700 1 -- 172.21.15.42:0/97121153 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8fc144b80 con 0x7fc8fc09f550 2019-12-05T06:51:41.777 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.762+0000 7fc8fa59c700 1 --2- 172.21.15.42:0/97121153 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc8fc0a9a00 0x7fc8fc07e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:51:41.777 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.762+0000 7fc8fad9d700 1 --2- 172.21.15.42:0/97121153 >> v2:172.21.15.40:3300/0 conn(0x7fc8fc09f550 0x7fc8fc144630 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fc8fc0a9140 tx=0x7fc8f40100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:41.777 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.762+0000 7fc8fad9d700 1 --2- 172.21.15.42:0/97121153 >> v2:172.21.15.40:3300/0 conn(0x7fc8fc09f550 0x7fc8fc144630 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fc8fc0a9140 tx=0x7fc8f40100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:41.777 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.762+0000 7fc8fad9d700 1 --2- 172.21.15.42:0/97121153 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc8fc09f550 0x7fc8fc144630 secure :-1 s=READY pgs=2558 cs=0 l=1 rx=0x7fc8fc0a9140 tx=0x7fc8f40100b0).ready entity=mon.0 client_cookie=b5d4b8a9e9ff26d8 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:51:41.777 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.762+0000 7fc8fad9d700 1 --2- 172.21.15.42:0/97121153 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc8fc09f550 0x7fc8fc144630 secure :-1 s=THROTTLE_DONE pgs=2558 cs=0 l=1 rx=0x7fc8fc0a9140 tx=0x7fc8f40100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:41.778 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.762+0000 7fc8f9d9b700 1 -- 172.21.15.42:0/97121153 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc8f400a160 con 0x7fc8fc09f550 2019-12-05T06:51:41.778 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.762+0000 7fc8fad9d700 1 --2- 172.21.15.42:0/97121153 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc8fc09f550 0x7fc8fc144630 secure :-1 s=THROTTLE_DONE pgs=2558 cs=0 l=1 rx=0x7fc8fc0a9140 tx=0x7fc8f40100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:41.778 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.762+0000 7fc8f9d9b700 1 -- 172.21.15.42:0/97121153 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8f4011a30 con 0x7fc8fc09f550 2019-12-05T06:51:41.778 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.762+0000 7fc8fad9d700 1 --2- 172.21.15.42:0/97121153 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc8fc09f550 0x7fc8fc144630 secure :-1 s=THROTTLE_DONE pgs=2558 cs=0 l=1 rx=0x7fc8fc0a9140 tx=0x7fc8f40100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:41.778 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.762+0000 7fc8f9d9b700 1 -- 172.21.15.42:0/97121153 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc8f400a550 con 0x7fc8fc09f550 2019-12-05T06:51:41.778 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.762+0000 7fc900b1a700 1 -- 172.21.15.42:0/97121153 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc8fc09f550 msgr2=0x7fc8fc144630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:41.778 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.762+0000 7fc900b1a700 1 --2- 172.21.15.42:0/97121153 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc8fc09f550 0x7fc8fc144630 secure :-1 s=READY pgs=2558 cs=0 l=1 rx=0x7fc8fc0a9140 tx=0x7fc8f40100b0).stop 2019-12-05T06:51:41.779 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.766+0000 7fc900b1a700 1 -- 172.21.15.42:0/97121153 shutdown_connections 2019-12-05T06:51:41.779 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.766+0000 7fc900b1a700 1 --2- 172.21.15.42:0/97121153 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc8fc09f550 0x7fc8fc144630 unknown :-1 s=CLOSED pgs=2558 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:51:41.779 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.766+0000 7fc900b1a700 1 --2- 172.21.15.42:0/97121153 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc8fc0a9a00 0x7fc8fc07e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:51:41.779 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.766+0000 7fc900b1a700 1 -- 172.21.15.42:0/97121153 shutdown_connections 2019-12-05T06:51:41.779 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.766+0000 7fc900b1a700 1 -- 172.21.15.42:0/97121153 wait complete. 2019-12-05T06:51:41.779 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.766+0000 7fc900b1a700 1 -- 172.21.15.42:0/97121153 >> 172.21.15.42:0/97121153 conn(0x7fc8fc0a9ef0 msgr2=0x7fc8fc0aa320 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:51:41.779 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.766+0000 7fc900b1a700 1 Processor -- start 2019-12-05T06:51:41.780 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.766+0000 7fc900b1a700 1 -- start start 2019-12-05T06:51:41.780 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.766+0000 7fc900b1a700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc8fc0a6380 0x7fc8fc08dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:51:41.780 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.766+0000 7fc900b1a700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc8fc08ed60 0x7fc8fc078210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:51:41.780 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.766+0000 7fc900b1a700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8fc08f190 con 0x7fc8fc078760 2019-12-05T06:51:41.780 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.766+0000 7fc900b1a700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8fc08f500 con 0x7fc8fc08ed60 2019-12-05T06:51:41.780 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.766+0000 7fc900b1a700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8fc08f7a0 con 0x7fc8fc0a6380 2019-12-05T06:51:41.780 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.766+0000 7fc8fad9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc8fc0a6380 0x7fc8fc08dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:51:41.781 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.770+0000 7fc8fad9d700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc8fc0a6380 0x7fc8fc08dd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:39350/0 (socket says 172.21.15.42:39350) 2019-12-05T06:51:41.781 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.770+0000 7fc8fad9d700 1 -- 172.21.15.42:0/1129670049 learned_addr learned my addr 172.21.15.42:0/1129670049 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:51:41.781 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.770+0000 7fc8fa59c700 1 --2- 172.21.15.42:0/1129670049 >> v2:172.21.15.40:3300/0 conn(0x7fc8fc08ed60 0x7fc8fc078210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:51:41.781 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.770+0000 7fc8fad9d700 1 -- 172.21.15.42:0/1129670049 >> v2:172.21.15.40:3300/0 conn(0x7fc8fc08ed60 msgr2=0x7fc8fc078210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:41.784 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.770+0000 7fc8fad9d700 1 --2- 172.21.15.42:0/1129670049 >> v2:172.21.15.40:3300/0 conn(0x7fc8fc08ed60 0x7fc8fc078210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:51:41.784 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.770+0000 7fc8fad9d700 1 -- 172.21.15.42:0/1129670049 >> v1:172.21.15.42:6789/0 conn(0x7fc8fc078760 legacy=0x7fc8fc1318b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:41.784 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.770+0000 7fc8fad9d700 1 -- 172.21.15.42:0/1129670049 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8fc1320b0 con 0x7fc8fc0a6380 2019-12-05T06:51:41.784 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.770+0000 7fc8fa59c700 1 --2- 172.21.15.42:0/1129670049 >> v2:172.21.15.40:3300/0 conn(0x7fc8fc08ed60 0x7fc8fc078210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:51:41.785 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.770+0000 7fc8f37fe700 1 -- 172.21.15.42:0/1129670049 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 700818954 0 0) 0x7fc8fc08f190 con 0x7fc8fc078760 2019-12-05T06:51:41.785 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.770+0000 7fc8fad9d700 1 --2- 172.21.15.42:0/1129670049 >> v2:172.21.15.42:3300/0 conn(0x7fc8fc0a6380 0x7fc8fc08dd20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fc8fc0a9140 tx=0x7fc8f4011ce0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:41.785 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.770+0000 7fc8fad9d700 1 --2- 172.21.15.42:0/1129670049 >> v2:172.21.15.42:3300/0 conn(0x7fc8fc0a6380 0x7fc8fc08dd20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fc8fc0a9140 tx=0x7fc8f4011ce0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:41.785 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.770+0000 7fc8fad9d700 1 --2- 172.21.15.42:0/1129670049 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc8fc0a6380 0x7fc8fc08dd20 secure :-1 s=READY pgs=3375 cs=0 l=1 rx=0x7fc8fc0a9140 tx=0x7fc8f4011ce0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:51:41.785 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.770+0000 7fc8fad9d700 1 --2- 172.21.15.42:0/1129670049 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc8fc0a6380 0x7fc8fc08dd20 secure :-1 s=THROTTLE_DONE pgs=3375 cs=0 l=1 rx=0x7fc8fc0a9140 tx=0x7fc8f4011ce0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:41.785 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.770+0000 7fc8f37fe700 1 -- 172.21.15.42:0/1129670049 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc8f400acf0 con 0x7fc8fc0a6380 2019-12-05T06:51:41.785 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.770+0000 7fc8fad9d700 1 --2- 172.21.15.42:0/1129670049 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc8fc0a6380 0x7fc8fc08dd20 secure :-1 s=THROTTLE_DONE pgs=3375 cs=0 l=1 rx=0x7fc8fc0a9140 tx=0x7fc8f4011ce0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:41.786 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.770+0000 7fc8f37fe700 1 -- 172.21.15.42:0/1129670049 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8f4011e10 con 0x7fc8fc0a6380 2019-12-05T06:51:41.786 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.770+0000 7fc8fad9d700 1 --2- 172.21.15.42:0/1129670049 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc8fc0a6380 0x7fc8fc08dd20 secure :-1 s=THROTTLE_DONE pgs=3375 cs=0 l=1 rx=0x7fc8fc0a9140 tx=0x7fc8f4011ce0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:41.786 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.770+0000 7fc900b1a700 1 -- 172.21.15.42:0/1129670049 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8fc132380 con 0x7fc8fc0a6380 2019-12-05T06:51:41.786 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.770+0000 7fc8f37fe700 1 -- 172.21.15.42:0/1129670049 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc8f4005ca0 con 0x7fc8fc0a6380 2019-12-05T06:51:41.786 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.770+0000 7fc900b1a700 1 -- 172.21.15.42:0/1129670049 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8fc138da0 con 0x7fc8fc0a6380 2019-12-05T06:51:41.786 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.770+0000 7fc8fad9d700 1 --2- 172.21.15.42:0/1129670049 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc8fc0a6380 0x7fc8fc08dd20 secure :-1 s=THROTTLE_DONE pgs=3375 cs=0 l=1 rx=0x7fc8fc0a9140 tx=0x7fc8f4011ce0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:41.788 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.770+0000 7fc8f37fe700 1 -- 172.21.15.42:0/1129670049 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fc8f4007150 con 0x7fc8fc0a6380 2019-12-05T06:51:41.788 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.770+0000 7fc8fad9d700 1 --2- 172.21.15.42:0/1129670049 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc8fc0a6380 0x7fc8fc08dd20 secure :-1 s=THROTTLE_DONE pgs=3375 cs=0 l=1 rx=0x7fc8fc0a9140 tx=0x7fc8f4011ce0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:41.789 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.770+0000 7fc8f37fe700 1 --2- 172.21.15.42:0/1129670049 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc8e003ac10 0x7fc8e003d120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:51:41.789 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.770+0000 7fc8f37fe700 1 -- 172.21.15.42:0/1129670049 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 12365+0+0 (secure 0 0 0) 0x7fc8f404ba80 con 0x7fc8fc0a6380 2019-12-05T06:51:41.789 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.770+0000 7fc8fa59c700 1 --2- 172.21.15.42:0/1129670049 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc8e003ac10 0x7fc8e003d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:51:41.789 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.770+0000 7fc8fa59c700 1 --2- 172.21.15.42:0/1129670049 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc8e003ac10 0x7fc8e003d120 crc :-1 s=READY pgs=1416 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:51:41.789 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.778+0000 7fc900b1a700 1 --2- 172.21.15.42:0/1129670049 >> [v2:172.21.15.42:6818/12116,v1:172.21.15.42:6819/12116] conn(0x7fc8fc054260 0x7fc8fc054690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:51:41.790 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.778+0000 7fc900b1a700 1 -- 172.21.15.42:0/1129670049 --> [v2:172.21.15.42:6818/12116,v1:172.21.15.42:6819/12116] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc8fc05d110 con 0x7fc8fc054260 2019-12-05T06:51:41.790 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.778+0000 7fc8fb59e700 1 --2- 172.21.15.42:0/1129670049 >> [v2:172.21.15.42:6818/12116,v1:172.21.15.42:6819/12116] conn(0x7fc8fc054260 0x7fc8fc054690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:51:41.790 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.778+0000 7fc8fb59e700 1 -- 172.21.15.42:0/1129670049 >> [v2:172.21.15.42:6818/12116,v1:172.21.15.42:6819/12116] conn(0x7fc8fc054260 msgr2=0x7fc8fc054690 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2019-12-05T06:51:41.790 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.778+0000 7fc8fb59e700 1 --2- 172.21.15.42:0/1129670049 >> [v2:172.21.15.42:6818/12116,v1:172.21.15.42:6819/12116] conn(0x7fc8fc054260 0x7fc8fc054690 crc :-1 s=READY pgs=99 cs=0 l=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:51:41.790 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.778+0000 7fc8f37fe700 1 -- 172.21.15.42:0/1129670049 <== osd.3 v2:172.21.15.42:6818/12116 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23014 (crc 0 0 0) 0x7fc8fc05d110 con 0x7fc8fc054260 2019-12-05T06:51:41.815 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.810+0000 7fc900b1a700 1 -- 172.21.15.42:0/1129670049 --> [v2:172.21.15.42:6818/12116,v1:172.21.15.42:6819/12116] -- command(tid 2: {"var": "osd_debug_deep_scrub_sleep", "prefix": "config set", "val": ["0"]}) v1 -- 0x7fc8fc08f9a0 con 0x7fc8fc054260 2019-12-05T06:51:41.815 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.810+0000 7fc8f37fe700 1 -- 172.21.15.42:0/1129670049 <== osd.3 v2:172.21.15.42:6818/12116 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7fc8fc08f9a0 con 0x7fc8fc054260 2019-12-05T06:51:41.816 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2019-12-05T06:51:41.816 INFO:teuthology.orchestra.run.smithi042.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2019-12-05T06:51:41.816 INFO:teuthology.orchestra.run.smithi042.stdout:} 2019-12-05T06:51:41.817 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.810+0000 7fc900b1a700 1 -- 172.21.15.42:0/1129670049 >> [v2:172.21.15.42:6818/12116,v1:172.21.15.42:6819/12116] conn(0x7fc8fc054260 msgr2=0x7fc8fc054690 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:41.817 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.810+0000 7fc900b1a700 1 --2- 172.21.15.42:0/1129670049 >> [v2:172.21.15.42:6818/12116,v1:172.21.15.42:6819/12116] conn(0x7fc8fc054260 0x7fc8fc054690 crc :-1 s=READY pgs=99 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:51:41.817 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.810+0000 7fc900b1a700 1 -- 172.21.15.42:0/1129670049 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc8e003ac10 msgr2=0x7fc8e003d120 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:41.817 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.810+0000 7fc900b1a700 1 --2- 172.21.15.42:0/1129670049 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7fc8e003ac10 0x7fc8e003d120 crc :-1 s=READY pgs=1416 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:51:41.817 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.810+0000 7fc900b1a700 1 -- 172.21.15.42:0/1129670049 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc8fc0a6380 msgr2=0x7fc8fc08dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:41.817 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.810+0000 7fc900b1a700 1 --2- 172.21.15.42:0/1129670049 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc8fc0a6380 0x7fc8fc08dd20 secure :-1 s=READY pgs=3375 cs=0 l=1 rx=0x7fc8fc0a9140 tx=0x7fc8f4011ce0).stop 2019-12-05T06:51:41.818 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.814+0000 7fc8fb59e700 1 -- 172.21.15.42:0/1129670049 reap_dead start 2019-12-05T06:51:41.818 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.814+0000 7fc900b1a700 1 -- 172.21.15.42:0/1129670049 shutdown_connections 2019-12-05T06:51:41.818 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.814+0000 7fc900b1a700 1 --2- 172.21.15.42:0/1129670049 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc8fc0a6380 0x7fc8fc08dd20 unknown :-1 s=CLOSED pgs=3375 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:51:41.818 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.814+0000 7fc900b1a700 1 -- 172.21.15.42:0/1129670049 shutdown_connections 2019-12-05T06:51:41.818 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.814+0000 7fc900b1a700 1 -- 172.21.15.42:0/1129670049 wait complete. 2019-12-05T06:51:41.818 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.814+0000 7fc900b1a700 1 -- 172.21.15.42:0/1129670049 >> 172.21.15.42:0/1129670049 conn(0x7fc8fc0a8b30 msgr2=0x7fc8fc0a9c10 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:51:41.826 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2019-12-05T06:51:41.927 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.918+0000 7f2b081f8700 1 Processor -- start 2019-12-05T06:51:41.927 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.918+0000 7f2b081f8700 1 -- start start 2019-12-05T06:51:41.927 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.918+0000 7f2b081f8700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2b0009fae0 0x7f2b0014fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:51:41.928 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.918+0000 7f2b081f8700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b000a0710 con 0x7f2b000a7740 2019-12-05T06:51:41.928 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.918+0000 7f2b081f8700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b0007ef20 con 0x7f2b0008df80 2019-12-05T06:51:41.928 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.918+0000 7f2b081f8700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b000a0b00 con 0x7f2b0009fae0 2019-12-05T06:51:41.928 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.918+0000 7f2b05f94700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f2b0008df80 0x7f2b0009f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:39398/0 (socket says 172.21.15.42:39398) 2019-12-05T06:51:41.928 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.918+0000 7f2b05f94700 1 -- 172.21.15.42:0/2030899551 learned_addr learned my addr 172.21.15.42:0/2030899551 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:51:41.928 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.918+0000 7f2b06f96700 1 --2- 172.21.15.42:0/2030899551 >> v2:172.21.15.40:3300/0 conn(0x7f2b0009fae0 0x7f2b0014fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:51:41.928 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.918+0000 7f2b05793700 1 -- 172.21.15.42:0/2030899551 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 367448312 0 0) 0x7f2b0007ef20 con 0x7f2b0008df80 2019-12-05T06:51:41.929 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.918+0000 7f2b05793700 1 -- 172.21.15.42:0/2030899551 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2aec0030b0 con 0x7f2b0008df80 2019-12-05T06:51:41.929 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.918+0000 7f2b05793700 1 -- 172.21.15.42:0/2030899551 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 540199096 0 0) 0x7f2b000a0710 con 0x7f2b000a7740 2019-12-05T06:51:41.929 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.918+0000 7f2b05793700 1 -- 172.21.15.42:0/2030899551 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b0007ef20 con 0x7f2b000a7740 2019-12-05T06:51:41.929 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.918+0000 7f2b06f96700 1 -- 172.21.15.42:0/2030899551 >> v1:172.21.15.42:6789/0 conn(0x7f2b0008df80 legacy=0x7f2b0009f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:41.929 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.918+0000 7f2b06f96700 1 -- 172.21.15.42:0/2030899551 >> v1:172.21.15.40:6789/0 conn(0x7f2b000a7740 legacy=0x7f2b0007e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:41.929 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.918+0000 7f2b06f96700 1 -- 172.21.15.42:0/2030899551 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b000a9ba0 con 0x7f2b0009fae0 2019-12-05T06:51:41.929 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.918+0000 7f2b06f96700 1 --2- 172.21.15.42:0/2030899551 >> v2:172.21.15.40:3300/0 conn(0x7f2b0009fae0 0x7f2b0014fff0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f2af8008b10 tx=0x7f2af800ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:41.930 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.922+0000 7f2b06f96700 1 --2- 172.21.15.42:0/2030899551 >> v2:172.21.15.40:3300/0 conn(0x7f2b0009fae0 0x7f2b0014fff0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f2af8008b10 tx=0x7f2af800ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:41.930 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.922+0000 7f2b06f96700 1 --2- 172.21.15.42:0/2030899551 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2b0009fae0 0x7f2b0014fff0 secure :-1 s=READY pgs=2560 cs=0 l=1 rx=0x7f2af8008b10 tx=0x7f2af800ff70).ready entity=mon.0 client_cookie=4dc4a3179a4d81a9 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:51:41.930 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.922+0000 7f2b06f96700 1 --2- 172.21.15.42:0/2030899551 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2b0009fae0 0x7f2b0014fff0 secure :-1 s=THROTTLE_DONE pgs=2560 cs=0 l=1 rx=0x7f2af8008b10 tx=0x7f2af800ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:41.930 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.922+0000 7f2b05793700 1 -- 172.21.15.42:0/2030899551 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2af8010970 con 0x7f2b0009fae0 2019-12-05T06:51:41.930 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.922+0000 7f2b06f96700 1 --2- 172.21.15.42:0/2030899551 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2b0009fae0 0x7f2b0014fff0 secure :-1 s=THROTTLE_DONE pgs=2560 cs=0 l=1 rx=0x7f2af8008b10 tx=0x7f2af800ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:41.930 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.922+0000 7f2b06f96700 1 --2- 172.21.15.42:0/2030899551 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2b0009fae0 0x7f2b0014fff0 secure :-1 s=THROTTLE_DONE pgs=2560 cs=0 l=1 rx=0x7f2af8008b10 tx=0x7f2af800ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:41.931 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.922+0000 7f2b05793700 1 -- 172.21.15.42:0/2030899551 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2af8008b50 con 0x7f2b0009fae0 2019-12-05T06:51:41.931 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.922+0000 7f2b05793700 1 -- 172.21.15.42:0/2030899551 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2af80110c0 con 0x7f2b0009fae0 2019-12-05T06:51:41.931 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.922+0000 7f2b081f8700 1 -- 172.21.15.42:0/2030899551 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2b0009fae0 msgr2=0x7f2b0014fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:41.931 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.922+0000 7f2b081f8700 1 --2- 172.21.15.42:0/2030899551 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2b0009fae0 0x7f2b0014fff0 secure :-1 s=READY pgs=2560 cs=0 l=1 rx=0x7f2af8008b10 tx=0x7f2af800ff70).stop 2019-12-05T06:51:41.931 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.922+0000 7f2b081f8700 1 -- 172.21.15.42:0/2030899551 shutdown_connections 2019-12-05T06:51:41.931 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.922+0000 7f2b081f8700 1 --2- 172.21.15.42:0/2030899551 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2b0009fae0 0x7f2b0014fff0 unknown :-1 s=CLOSED pgs=2560 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:51:41.931 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.922+0000 7f2b081f8700 1 -- 172.21.15.42:0/2030899551 shutdown_connections 2019-12-05T06:51:41.932 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.922+0000 7f2b081f8700 1 -- 172.21.15.42:0/2030899551 wait complete. 2019-12-05T06:51:41.932 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.922+0000 7f2b081f8700 1 -- 172.21.15.42:0/2030899551 >> 172.21.15.42:0/2030899551 conn(0x7f2b000aabe0 msgr2=0x7f2b000a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:51:41.935 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.926+0000 7f2b081f8700 1 Processor -- start 2019-12-05T06:51:41.936 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.926+0000 7f2b081f8700 1 -- start start 2019-12-05T06:51:41.936 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.926+0000 7f2b081f8700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2b000a5b10 0x7f2b00142be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:51:41.936 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.926+0000 7f2b081f8700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2b00144e40 0x7f2b0011b180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:51:41.936 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.926+0000 7f2b081f8700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2b00145270 0x7f2b0011b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:51:41.936 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.926+0000 7f2b081f8700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b00143410 con 0x7f2b00144e40 2019-12-05T06:51:41.936 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.926+0000 7f2b081f8700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b0011aee0 con 0x7f2b000a5b10 2019-12-05T06:51:41.937 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.926+0000 7f2b081f8700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b00143130 con 0x7f2b00145270 2019-12-05T06:51:41.937 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.926+0000 7f2b06795700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2b000a5b10 0x7f2b00142be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:51:41.937 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.926+0000 7f2b06795700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2b000a5b10 0x7f2b00142be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:39366/0 (socket says 172.21.15.42:39366) 2019-12-05T06:51:41.937 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.926+0000 7f2b06795700 1 -- 172.21.15.42:0/1394969216 learned_addr learned my addr 172.21.15.42:0/1394969216 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:51:41.937 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.926+0000 7f2b06f96700 1 --2- 172.21.15.42:0/1394969216 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2b00145270 0x7f2b0011b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:51:41.937 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.926+0000 7f2b05f94700 1 --2- 172.21.15.42:0/1394969216 >> v2:172.21.15.40:3300/0 conn(0x7f2b00144e40 0x7f2b0011b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:51:41.937 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.926+0000 7f2b06f96700 1 -- 172.21.15.42:0/1394969216 >> v2:172.21.15.42:3300/0 conn(0x7f2b000a5b10 msgr2=0x7f2b00142be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:41.938 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.926+0000 7f2b06f96700 1 --2- 172.21.15.42:0/1394969216 >> v2:172.21.15.42:3300/0 conn(0x7f2b000a5b10 0x7f2b00142be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:51:41.938 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.926+0000 7f2b06f96700 1 -- 172.21.15.42:0/1394969216 >> v2:172.21.15.40:3300/0 conn(0x7f2b00144e40 msgr2=0x7f2b0011b180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:41.938 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.926+0000 7f2b06f96700 1 --2- 172.21.15.42:0/1394969216 >> v2:172.21.15.40:3300/0 conn(0x7f2b00144e40 0x7f2b0011b180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:51:41.938 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.926+0000 7f2b06f96700 1 -- 172.21.15.42:0/1394969216 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b00117100 con 0x7f2b00145270 2019-12-05T06:51:41.938 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.926+0000 7f2b06f96700 1 --2- 172.21.15.42:0/1394969216 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2b00145270 0x7f2b0011b6d0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f2b00142900 tx=0x7f2af8008870).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:41.938 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.926+0000 7f2b06f96700 1 --2- 172.21.15.42:0/1394969216 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2b00145270 0x7f2b0011b6d0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f2b00142900 tx=0x7f2af8008870).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:41.938 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.926+0000 7f2b06f96700 1 --2- 172.21.15.42:0/1394969216 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2b00145270 0x7f2b0011b6d0 secure :-1 s=READY pgs=752 cs=0 l=1 rx=0x7f2b00142900 tx=0x7f2af8008870).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:51:41.938 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.926+0000 7f2b06f96700 1 --2- 172.21.15.42:0/1394969216 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2b00145270 0x7f2b0011b6d0 secure :-1 s=THROTTLE_DONE pgs=752 cs=0 l=1 rx=0x7f2b00142900 tx=0x7f2af8008870).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:41.939 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.926+0000 7f2b06f96700 1 --2- 172.21.15.42:0/1394969216 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2b00145270 0x7f2b0011b6d0 secure :-1 s=THROTTLE_DONE pgs=752 cs=0 l=1 rx=0x7f2b00142900 tx=0x7f2af8008870).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:41.939 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.926+0000 7f2af6ffd700 1 -- 172.21.15.42:0/1394969216 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2af800f620 con 0x7f2b00145270 2019-12-05T06:51:41.939 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.926+0000 7f2b06f96700 1 --2- 172.21.15.42:0/1394969216 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2b00145270 0x7f2b0011b6d0 secure :-1 s=THROTTLE_DONE pgs=752 cs=0 l=1 rx=0x7f2b00142900 tx=0x7f2af8008870).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:41.939 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.926+0000 7f2af6ffd700 1 -- 172.21.15.42:0/1394969216 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2af8010b70 con 0x7f2b00145270 2019-12-05T06:51:41.939 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.926+0000 7f2af6ffd700 1 -- 172.21.15.42:0/1394969216 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2af8011580 con 0x7f2b00145270 2019-12-05T06:51:41.939 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.926+0000 7f2b081f8700 1 -- 172.21.15.42:0/1394969216 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b0011a6a0 con 0x7f2b00145270 2019-12-05T06:51:41.940 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.930+0000 7f2b081f8700 1 -- 172.21.15.42:0/1394969216 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b00130550 con 0x7f2b00145270 2019-12-05T06:51:41.940 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.930+0000 7f2b06f96700 1 --2- 172.21.15.42:0/1394969216 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2b00145270 0x7f2b0011b6d0 secure :-1 s=THROTTLE_DONE pgs=752 cs=0 l=1 rx=0x7f2b00142900 tx=0x7f2af8008870).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:41.940 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.930+0000 7f2af6ffd700 1 -- 172.21.15.42:0/1394969216 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f2af8011780 con 0x7f2b00145270 2019-12-05T06:51:41.940 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.930+0000 7f2b06f96700 1 --2- 172.21.15.42:0/1394969216 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2b00145270 0x7f2b0011b6d0 secure :-1 s=THROTTLE_DONE pgs=752 cs=0 l=1 rx=0x7f2b00142900 tx=0x7f2af8008870).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:41.940 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.930+0000 7f2af6ffd700 1 --2- 172.21.15.42:0/1394969216 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2ae803ae20 0x7f2ae803d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:51:41.940 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.930+0000 7f2af6ffd700 1 -- 172.21.15.42:0/1394969216 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 12365+0+0 (secure 0 0 0) 0x7f2af804d100 con 0x7f2b00145270 2019-12-05T06:51:41.940 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.930+0000 7f2b06795700 1 --2- 172.21.15.42:0/1394969216 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2ae803ae20 0x7f2ae803d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:51:41.941 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.930+0000 7f2b06795700 1 --2- 172.21.15.42:0/1394969216 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2ae803ae20 0x7f2ae803d330 crc :-1 s=READY pgs=1417 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:51:41.941 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.930+0000 7f2b081f8700 1 -- 172.21.15.42:0/1394969216 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b0005d110 con 0x7f2b00145270 2019-12-05T06:51:41.945 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.942+0000 7f2b06f96700 1 --2- 172.21.15.42:0/1394969216 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2b00145270 0x7f2b0011b6d0 secure :-1 s=THROTTLE_DONE pgs=752 cs=0 l=1 rx=0x7f2b00142900 tx=0x7f2af8008870).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:41.945 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:41.942+0000 7f2af6ffd700 1 -- 172.21.15.42:0/1394969216 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f2af8034180 con 0x7f2b00145270 2019-12-05T06:51:42.110 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.106+0000 7f2b081f8700 1 -- 172.21.15.42:0/1394969216 --> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) v1 -- 0x7f2b0005b8c0 con 0x7f2ae803ae20 2019-12-05T06:51:42.111 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.106+0000 7f2af6ffd700 1 -- 172.21.15.42:0/1394969216 <== mgr.4108 v2:172.21.15.42:6800/11930 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) v1 ==== 41+0+0 (crc 0 0 0) 0x7f2b0005b8c0 con 0x7f2ae803ae20 2019-12-05T06:51:42.111 INFO:teuthology.orchestra.run.smithi042.stderr:instructed osd(s) 3 to deep-scrub 2019-12-05T06:51:42.114 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.110+0000 7f2b081f8700 1 -- 172.21.15.42:0/1394969216 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2ae803ae20 msgr2=0x7f2ae803d330 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:42.114 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.110+0000 7f2b081f8700 1 --2- 172.21.15.42:0/1394969216 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2ae803ae20 0x7f2ae803d330 crc :-1 s=READY pgs=1417 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:51:42.114 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.110+0000 7f2b081f8700 1 -- 172.21.15.42:0/1394969216 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2b00145270 msgr2=0x7f2b0011b6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:42.114 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.110+0000 7f2b081f8700 1 --2- 172.21.15.42:0/1394969216 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2b00145270 0x7f2b0011b6d0 secure :-1 s=READY pgs=752 cs=0 l=1 rx=0x7f2b00142900 tx=0x7f2af8008870).stop 2019-12-05T06:51:42.115 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.110+0000 7f2b081f8700 1 -- 172.21.15.42:0/1394969216 shutdown_connections 2019-12-05T06:51:42.115 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.110+0000 7f2b081f8700 1 --2- 172.21.15.42:0/1394969216 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f2ae803ae20 0x7f2ae803d330 unknown :-1 s=CLOSED pgs=1417 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:51:42.115 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.110+0000 7f2b081f8700 1 --2- 172.21.15.42:0/1394969216 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2b00145270 0x7f2b0011b6d0 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:51:42.115 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.110+0000 7f2b081f8700 1 --2- 172.21.15.42:0/1394969216 >> v2:172.21.15.42:3300/0 conn(0x7f2b000a5b10 0x7f2b00142be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:51:42.115 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.110+0000 7f2b081f8700 1 --2- 172.21.15.42:0/1394969216 >> v2:172.21.15.40:3300/0 conn(0x7f2b00144e40 0x7f2b0011b180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:51:42.115 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.110+0000 7f2b081f8700 1 -- 172.21.15.42:0/1394969216 shutdown_connections 2019-12-05T06:51:42.115 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.110+0000 7f2b081f8700 1 -- 172.21.15.42:0/1394969216 wait complete. 2019-12-05T06:51:42.116 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.110+0000 7f2b081f8700 1 -- 172.21.15.42:0/1394969216 >> 172.21.15.42:0/1394969216 conn(0x7f2b0008fde0 msgr2=0x7f2b00059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:51:42.124 INFO:tasks.ceph:Scrubbing osd.4 2019-12-05T06:51:42.124 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2019-12-05T06:51:42.229 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.218+0000 7f81fd45a700 1 Processor -- start 2019-12-05T06:51:42.229 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.218+0000 7f81fd45a700 1 -- start start 2019-12-05T06:51:42.229 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.222+0000 7f81fd45a700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f81f809f550 0x7f81f8144630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:51:42.229 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.222+0000 7f81fd45a700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f81f814cf50 0x7f81f814f390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:51:42.229 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.222+0000 7f81fd45a700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81f807f080 con 0x7f81f80a9a00 2019-12-05T06:51:42.230 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.222+0000 7f81fd45a700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81f80a8f80 con 0x7f81f814cf50 2019-12-05T06:51:42.230 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.222+0000 7f81fd45a700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81f80a9220 con 0x7f81f809f550 2019-12-05T06:51:42.230 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.222+0000 7f81f77fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f81f809f550 0x7f81f8144630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:51:42.230 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.222+0000 7f81f77fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f81f809f550 0x7f81f8144630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:39374/0 (socket says 172.21.15.42:39374) 2019-12-05T06:51:42.230 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.222+0000 7f81f77fe700 1 -- 172.21.15.42:0/2516280836 learned_addr learned my addr 172.21.15.42:0/2516280836 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:51:42.230 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.222+0000 7f81f7fff700 1 --2- 172.21.15.42:0/2516280836 >> v2:172.21.15.40:3300/0 conn(0x7f81f814cf50 0x7f81f814f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:51:42.230 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.222+0000 7f81f77fe700 1 -- 172.21.15.42:0/2516280836 >> v2:172.21.15.40:3300/0 conn(0x7f81f814cf50 msgr2=0x7f81f814f390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:51:42.231 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.222+0000 7f81f77fe700 1 --2- 172.21.15.42:0/2516280836 >> v2:172.21.15.40:3300/0 conn(0x7f81f814cf50 0x7f81f814f390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:51:42.231 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.222+0000 7f81f77fe700 1 -- 172.21.15.42:0/2516280836 >> v1:172.21.15.42:6789/0 conn(0x7f81f80a9a00 legacy=0x7f81f807e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:42.231 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.222+0000 7f81f77fe700 1 -- 172.21.15.42:0/2516280836 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81f80a1000 con 0x7f81f809f550 2019-12-05T06:51:42.231 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.222+0000 7f81f77fe700 1 --2- 172.21.15.42:0/2516280836 >> v2:172.21.15.42:3300/0 conn(0x7f81f809f550 0x7f81f8144630 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f81e8002f30 tx=0x7f81e800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:42.231 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.222+0000 7f81f77fe700 1 --2- 172.21.15.42:0/2516280836 >> v2:172.21.15.42:3300/0 conn(0x7f81f809f550 0x7f81f8144630 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f81e8002f30 tx=0x7f81e800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:42.231 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.222+0000 7f81f77fe700 1 --2- 172.21.15.42:0/2516280836 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f81f809f550 0x7f81f8144630 secure :-1 s=READY pgs=3378 cs=0 l=1 rx=0x7f81e8002f30 tx=0x7f81e800fd30).ready entity=mon.1 client_cookie=e6320d75a75ec2b9 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:51:42.231 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.222+0000 7f81f77fe700 1 --2- 172.21.15.42:0/2516280836 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f81f809f550 0x7f81f8144630 secure :-1 s=THROTTLE_DONE pgs=3378 cs=0 l=1 rx=0x7f81e8002f30 tx=0x7f81e800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:42.232 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.222+0000 7f81f77fe700 1 --2- 172.21.15.42:0/2516280836 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f81f809f550 0x7f81f8144630 secure :-1 s=THROTTLE_DONE pgs=3378 cs=0 l=1 rx=0x7f81e8002f30 tx=0x7f81e800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:42.232 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.222+0000 7f81f67fc700 1 -- 172.21.15.42:0/2516280836 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f81e8008c90 con 0x7f81f809f550 2019-12-05T06:51:42.232 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.222+0000 7f81f77fe700 1 --2- 172.21.15.42:0/2516280836 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f81f809f550 0x7f81f8144630 secure :-1 s=THROTTLE_DONE pgs=3378 cs=0 l=1 rx=0x7f81e8002f30 tx=0x7f81e800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:42.232 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.222+0000 7f81f67fc700 1 -- 172.21.15.42:0/2516280836 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81e80088b0 con 0x7f81f809f550 2019-12-05T06:51:42.232 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.222+0000 7f81f67fc700 1 -- 172.21.15.42:0/2516280836 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f81e8011ad0 con 0x7f81f809f550 2019-12-05T06:51:42.232 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.222+0000 7f81fd45a700 1 -- 172.21.15.42:0/2516280836 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f81f809f550 msgr2=0x7f81f8144630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:42.233 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.222+0000 7f81fd45a700 1 --2- 172.21.15.42:0/2516280836 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f81f809f550 0x7f81f8144630 secure :-1 s=READY pgs=3378 cs=0 l=1 rx=0x7f81e8002f30 tx=0x7f81e800fd30).stop 2019-12-05T06:51:42.233 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.222+0000 7f81fd45a700 1 -- 172.21.15.42:0/2516280836 shutdown_connections 2019-12-05T06:51:42.233 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.222+0000 7f81fd45a700 1 --2- 172.21.15.42:0/2516280836 >> v2:172.21.15.40:3300/0 conn(0x7f81f814cf50 0x7f81f814f390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:51:42.233 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.222+0000 7f81fd45a700 1 --2- 172.21.15.42:0/2516280836 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f81f809f550 0x7f81f8144630 unknown :-1 s=CLOSED pgs=3378 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:51:42.233 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.222+0000 7f81fd45a700 1 -- 172.21.15.42:0/2516280836 shutdown_connections 2019-12-05T06:51:42.233 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.222+0000 7f81fd45a700 1 -- 172.21.15.42:0/2516280836 wait complete. 2019-12-05T06:51:42.233 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.222+0000 7f81fd45a700 1 -- 172.21.15.42:0/2516280836 >> 172.21.15.42:0/2516280836 conn(0x7f81f80a9ef0 msgr2=0x7f81f80aa320 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:51:42.234 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.226+0000 7f81fd45a700 1 Processor -- start 2019-12-05T06:51:42.234 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.226+0000 7f81fd45a700 1 -- start start 2019-12-05T06:51:42.237 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.226+0000 7f81fd45a700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f81f80a54b0 0x7f81f80a5a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:51:42.237 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.226+0000 7f81fd45a700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f81f8125720 0x7f81f8131390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:51:42.237 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.226+0000 7f81fd45a700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81f8125b50 con 0x7f81f81318e0 2019-12-05T06:51:42.237 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.226+0000 7f81fd45a700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81f80a61b0 con 0x7f81f8125720 2019-12-05T06:51:42.237 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.226+0000 7f81fd45a700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81f80a6450 con 0x7f81f80a54b0 2019-12-05T06:51:42.237 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.226+0000 7f81f6ffd700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f81f8125720 0x7f81f8131390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:51:42.237 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.226+0000 7f81f77fe700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f81f80a54b0 0x7f81f80a5a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:51:42.238 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.226+0000 7f81f6ffd700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f81f8125720 0x7f81f8131390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.42:55430/0 (socket says 172.21.15.42:55430) 2019-12-05T06:51:42.238 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.226+0000 7f81f6ffd700 1 -- 172.21.15.42:0/3751740147 learned_addr learned my addr 172.21.15.42:0/3751740147 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:51:42.238 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.226+0000 7f81f77fe700 1 -- 172.21.15.42:0/3751740147 >> v2:172.21.15.40:3300/0 conn(0x7f81f8125720 msgr2=0x7f81f8131390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:42.238 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.226+0000 7f81f77fe700 1 --2- 172.21.15.42:0/3751740147 >> v2:172.21.15.40:3300/0 conn(0x7f81f8125720 0x7f81f8131390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:51:42.238 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.226+0000 7f81f77fe700 1 -- 172.21.15.42:0/3751740147 >> v1:172.21.15.40:6789/0 conn(0x7f81f81318e0 legacy=0x7f81f8059dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:42.238 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.226+0000 7f81effff700 1 -- 172.21.15.42:0/3751740147 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 294435813 0 0) 0x7f81f8125b50 con 0x7f81f81318e0 2019-12-05T06:51:42.239 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.226+0000 7f81f77fe700 1 -- 172.21.15.42:0/3751740147 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81f8131d10 con 0x7f81f80a54b0 2019-12-05T06:51:42.239 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.226+0000 7f81f6ffd700 1 --2- 172.21.15.42:0/3751740147 >> v2:172.21.15.40:3300/0 conn(0x7f81f8125720 0x7f81f8131390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:51:42.239 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.226+0000 7f81f77fe700 1 --2- 172.21.15.42:0/3751740147 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f81f80a54b0 0x7f81f80a5a10 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f81e8002f30 tx=0x7f81e8008b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:42.239 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.226+0000 7f81f77fe700 1 --2- 172.21.15.42:0/3751740147 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f81f80a54b0 0x7f81f80a5a10 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f81e8002f30 tx=0x7f81e8008b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:42.239 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.226+0000 7f81f77fe700 1 --2- 172.21.15.42:0/3751740147 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f81f80a54b0 0x7f81f80a5a10 secure :-1 s=READY pgs=753 cs=0 l=1 rx=0x7f81e8002f30 tx=0x7f81e8008b40).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:51:42.239 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.226+0000 7f81f77fe700 1 --2- 172.21.15.42:0/3751740147 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f81f80a54b0 0x7f81f80a5a10 secure :-1 s=THROTTLE_DONE pgs=753 cs=0 l=1 rx=0x7f81e8002f30 tx=0x7f81e8008b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:42.239 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.226+0000 7f81effff700 1 -- 172.21.15.42:0/3751740147 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f81e800f490 con 0x7f81f80a54b0 2019-12-05T06:51:42.240 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.226+0000 7f81f77fe700 1 --2- 172.21.15.42:0/3751740147 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f81f80a54b0 0x7f81f80a5a10 secure :-1 s=THROTTLE_DONE pgs=753 cs=0 l=1 rx=0x7f81e8002f30 tx=0x7f81e8008b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:42.240 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.230+0000 7f81fd45a700 1 -- 172.21.15.42:0/3751740147 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81f805a5c0 con 0x7f81f80a54b0 2019-12-05T06:51:42.240 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.230+0000 7f81effff700 1 -- 172.21.15.42:0/3751740147 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81e80116f0 con 0x7f81f80a54b0 2019-12-05T06:51:42.240 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.230+0000 7f81f77fe700 1 --2- 172.21.15.42:0/3751740147 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f81f80a54b0 0x7f81f80a5a10 secure :-1 s=THROTTLE_DONE pgs=753 cs=0 l=1 rx=0x7f81e8002f30 tx=0x7f81e8008b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:42.240 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.230+0000 7f81fd45a700 1 -- 172.21.15.42:0/3751740147 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81f805ab40 con 0x7f81f80a54b0 2019-12-05T06:51:42.240 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.230+0000 7f81effff700 1 -- 172.21.15.42:0/3751740147 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f81e8005730 con 0x7f81f80a54b0 2019-12-05T06:51:42.241 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.230+0000 7f81f77fe700 1 --2- 172.21.15.42:0/3751740147 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f81f80a54b0 0x7f81f80a5a10 secure :-1 s=THROTTLE_DONE pgs=753 cs=0 l=1 rx=0x7f81e8002f30 tx=0x7f81e8008b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:42.241 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.230+0000 7f81effff700 1 -- 172.21.15.42:0/3751740147 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f81e80059d0 con 0x7f81f80a54b0 2019-12-05T06:51:42.241 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.230+0000 7f81f77fe700 1 --2- 172.21.15.42:0/3751740147 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f81f80a54b0 0x7f81f80a5a10 secure :-1 s=THROTTLE_DONE pgs=753 cs=0 l=1 rx=0x7f81e8002f30 tx=0x7f81e8008b40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:42.241 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.230+0000 7f81effff700 1 --2- 172.21.15.42:0/3751740147 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f81dc03ac40 0x7f81dc03d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:51:42.241 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.230+0000 7f81effff700 1 -- 172.21.15.42:0/3751740147 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 12365+0+0 (secure 0 0 0) 0x7f81e804bb00 con 0x7f81f80a54b0 2019-12-05T06:51:42.241 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.230+0000 7f81f6ffd700 1 --2- 172.21.15.42:0/3751740147 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f81dc03ac40 0x7f81dc03d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:51:42.241 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.230+0000 7f81f6ffd700 1 --2- 172.21.15.42:0/3751740147 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f81dc03ac40 0x7f81dc03d150 crc :-1 s=READY pgs=1418 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:51:42.242 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.234+0000 7f81fd45a700 1 --2- 172.21.15.42:0/3751740147 >> [v2:172.21.15.40:6808/12488,v1:172.21.15.40:6809/12488] conn(0x7f81f8054260 0x7f81f8054690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:51:42.243 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.234+0000 7f81fd45a700 1 -- 172.21.15.42:0/3751740147 --> [v2:172.21.15.40:6808/12488,v1:172.21.15.40:6809/12488] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f81f805d110 con 0x7f81f8054260 2019-12-05T06:51:42.243 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.238+0000 7f81f7fff700 1 --2- 172.21.15.42:0/3751740147 >> [v2:172.21.15.40:6808/12488,v1:172.21.15.40:6809/12488] conn(0x7f81f8054260 0x7f81f8054690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:51:42.243 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.238+0000 7f81f7fff700 1 -- 172.21.15.42:0/3751740147 >> [v2:172.21.15.40:6808/12488,v1:172.21.15.40:6809/12488] conn(0x7f81f8054260 msgr2=0x7f81f8054690 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2019-12-05T06:51:42.243 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.238+0000 7f81f7fff700 1 --2- 172.21.15.42:0/3751740147 >> [v2:172.21.15.40:6808/12488,v1:172.21.15.40:6809/12488] conn(0x7f81f8054260 0x7f81f8054690 crc :-1 s=READY pgs=81 cs=0 l=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:51:42.243 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.238+0000 7f81effff700 1 -- 172.21.15.42:0/3751740147 <== osd.4 v2:172.21.15.40:6808/12488 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23014 (crc 0 0 0) 0x7f81f805d110 con 0x7f81f8054260 2019-12-05T06:51:42.275 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.270+0000 7f81fd45a700 1 -- 172.21.15.42:0/3751740147 --> [v2:172.21.15.40:6808/12488,v1:172.21.15.40:6809/12488] -- command(tid 2: {"var": "osd_debug_deep_scrub_sleep", "prefix": "config set", "val": ["0"]}) v1 -- 0x7f81f805b8c0 con 0x7f81f8054260 2019-12-05T06:51:42.276 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.274+0000 7f81effff700 1 -- 172.21.15.42:0/3751740147 <== osd.4 v2:172.21.15.40:6808/12488 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f81f805b8c0 con 0x7f81f8054260 2019-12-05T06:51:42.277 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2019-12-05T06:51:42.277 INFO:teuthology.orchestra.run.smithi042.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2019-12-05T06:51:42.277 INFO:teuthology.orchestra.run.smithi042.stdout:} 2019-12-05T06:51:42.278 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.274+0000 7f81fd45a700 1 -- 172.21.15.42:0/3751740147 >> [v2:172.21.15.40:6808/12488,v1:172.21.15.40:6809/12488] conn(0x7f81f8054260 msgr2=0x7f81f8054690 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:42.278 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.274+0000 7f81fd45a700 1 --2- 172.21.15.42:0/3751740147 >> [v2:172.21.15.40:6808/12488,v1:172.21.15.40:6809/12488] conn(0x7f81f8054260 0x7f81f8054690 crc :-1 s=READY pgs=81 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:51:42.278 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.274+0000 7f81fd45a700 1 -- 172.21.15.42:0/3751740147 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f81dc03ac40 msgr2=0x7f81dc03d150 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:42.279 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.274+0000 7f81fd45a700 1 --2- 172.21.15.42:0/3751740147 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f81dc03ac40 0x7f81dc03d150 crc :-1 s=READY pgs=1418 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:51:42.279 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.274+0000 7f81fd45a700 1 -- 172.21.15.42:0/3751740147 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f81f80a54b0 msgr2=0x7f81f80a5a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:42.279 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.274+0000 7f81fd45a700 1 --2- 172.21.15.42:0/3751740147 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f81f80a54b0 0x7f81f80a5a10 secure :-1 s=READY pgs=753 cs=0 l=1 rx=0x7f81e8002f30 tx=0x7f81e8008b40).stop 2019-12-05T06:51:42.279 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.274+0000 7f81f7fff700 1 -- 172.21.15.42:0/3751740147 reap_dead start 2019-12-05T06:51:42.279 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.274+0000 7f81fd45a700 1 -- 172.21.15.42:0/3751740147 shutdown_connections 2019-12-05T06:51:42.279 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.274+0000 7f81fd45a700 1 -- 172.21.15.42:0/3751740147 shutdown_connections 2019-12-05T06:51:42.279 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.274+0000 7f81fd45a700 1 -- 172.21.15.42:0/3751740147 wait complete. 2019-12-05T06:51:42.280 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.274+0000 7f81fd45a700 1 -- 172.21.15.42:0/3751740147 >> 172.21.15.42:0/3751740147 conn(0x7f81f807ea80 msgr2=0x7f81f80a9960 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:51:42.288 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2019-12-05T06:51:42.394 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.386+0000 7f361a125700 1 Processor -- start 2019-12-05T06:51:42.395 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.386+0000 7f361a125700 1 -- start start 2019-12-05T06:51:42.395 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.386+0000 7f361a125700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f36140a7740 0x7f361408df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:51:42.395 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.386+0000 7f361a125700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f361409fae0 0x7f361407eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:51:42.395 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.386+0000 7f361a125700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36140a0710 con 0x7f361407e720 2019-12-05T06:51:42.395 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.386+0000 7f361a125700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36140a0b00 con 0x7f361409fae0 2019-12-05T06:51:42.395 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.386+0000 7f361a125700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f361408e5b0 con 0x7f36140a7740 2019-12-05T06:51:42.396 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.386+0000 7f3613fff700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f36140a7740 0x7f361408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:51:42.396 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.386+0000 7f3613fff700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f36140a7740 0x7f361408df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:39856/0 (socket says 172.21.15.42:39856) 2019-12-05T06:51:42.396 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.386+0000 7f3613fff700 1 -- 172.21.15.42:0/846254443 learned_addr learned my addr 172.21.15.42:0/846254443 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:51:42.396 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.386+0000 7f3618ec3700 1 --2- 172.21.15.42:0/846254443 >> v2:172.21.15.40:3300/0 conn(0x7f361409fae0 0x7f361407eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:51:42.396 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.386+0000 7f3613fff700 1 -- 172.21.15.42:0/846254443 >> v2:172.21.15.40:3300/0 conn(0x7f361409fae0 msgr2=0x7f361407eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:51:42.396 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.386+0000 7f3613fff700 1 --2- 172.21.15.42:0/846254443 >> v2:172.21.15.40:3300/0 conn(0x7f361409fae0 0x7f361407eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:51:42.396 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.386+0000 7f3613fff700 1 -- 172.21.15.42:0/846254443 >> v1:172.21.15.40:6789/0 conn(0x7f361407e720 legacy=0x7f361409f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:42.397 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.386+0000 7f3613fff700 1 -- 172.21.15.42:0/846254443 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f361407f0a0 con 0x7f36140a7740 2019-12-05T06:51:42.397 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.386+0000 7f3613fff700 1 --2- 172.21.15.42:0/846254443 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f36140a7740 0x7f361408df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f360c008b10 tx=0x7f360c0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:42.397 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.386+0000 7f3613fff700 1 --2- 172.21.15.42:0/846254443 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f36140a7740 0x7f361408df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f360c008b10 tx=0x7f360c0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:42.397 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.386+0000 7f3613fff700 1 --2- 172.21.15.42:0/846254443 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f36140a7740 0x7f361408df80 secure :-1 s=READY pgs=754 cs=0 l=1 rx=0x7f360c008b10 tx=0x7f360c0100b0).ready entity=mon.2 client_cookie=a4c0701418fd57f7 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:51:42.397 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.386+0000 7f3613fff700 1 --2- 172.21.15.42:0/846254443 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f36140a7740 0x7f361408df80 secure :-1 s=THROTTLE_DONE pgs=754 cs=0 l=1 rx=0x7f360c008b10 tx=0x7f360c0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:42.397 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.386+0000 7f3612ffd700 1 -- 172.21.15.42:0/846254443 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f360c010da0 con 0x7f36140a7740 2019-12-05T06:51:42.397 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.386+0000 7f3613fff700 1 --2- 172.21.15.42:0/846254443 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f36140a7740 0x7f361408df80 secure :-1 s=THROTTLE_DONE pgs=754 cs=0 l=1 rx=0x7f360c008b10 tx=0x7f360c0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:42.398 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.386+0000 7f3613fff700 1 --2- 172.21.15.42:0/846254443 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f36140a7740 0x7f361408df80 secure :-1 s=THROTTLE_DONE pgs=754 cs=0 l=1 rx=0x7f360c008b10 tx=0x7f360c0100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:42.398 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.386+0000 7f3612ffd700 1 -- 172.21.15.42:0/846254443 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f360c0109c0 con 0x7f36140a7740 2019-12-05T06:51:42.398 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.386+0000 7f3612ffd700 1 -- 172.21.15.42:0/846254443 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f360c0114a0 con 0x7f36140a7740 2019-12-05T06:51:42.398 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.386+0000 7f361a125700 1 -- 172.21.15.42:0/846254443 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f36140a7740 msgr2=0x7f361408df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:42.398 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.386+0000 7f361a125700 1 --2- 172.21.15.42:0/846254443 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f36140a7740 0x7f361408df80 secure :-1 s=READY pgs=754 cs=0 l=1 rx=0x7f360c008b10 tx=0x7f360c0100b0).stop 2019-12-05T06:51:42.398 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.390+0000 7f361a125700 1 -- 172.21.15.42:0/846254443 shutdown_connections 2019-12-05T06:51:42.398 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.390+0000 7f361a125700 1 --2- 172.21.15.42:0/846254443 >> v2:172.21.15.40:3300/0 conn(0x7f361409fae0 0x7f361407eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:51:42.399 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.390+0000 7f361a125700 1 --2- 172.21.15.42:0/846254443 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f36140a7740 0x7f361408df80 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:51:42.399 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.390+0000 7f361a125700 1 -- 172.21.15.42:0/846254443 shutdown_connections 2019-12-05T06:51:42.399 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.390+0000 7f361a125700 1 -- 172.21.15.42:0/846254443 wait complete. 2019-12-05T06:51:42.399 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.390+0000 7f361a125700 1 -- 172.21.15.42:0/846254443 >> 172.21.15.42:0/846254443 conn(0x7f36140aabe0 msgr2=0x7f36140a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:51:42.399 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.390+0000 7f361a125700 1 Processor -- start 2019-12-05T06:51:42.399 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.390+0000 7f361a125700 1 -- start start 2019-12-05T06:51:42.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.390+0000 7f361a125700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f3614117210 0x7f3614117640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:51:42.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.390+0000 7f361a125700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f361411a700 con 0x7f361412f0f0 2019-12-05T06:51:42.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.390+0000 7f361a125700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3614118cb0 con 0x7f361411ab10 2019-12-05T06:51:42.404 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.390+0000 7f361a125700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f361411af40 con 0x7f3614117210 2019-12-05T06:51:42.404 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.390+0000 7f3613fff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f361411ab10 0x7f3614119d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:39434/0 (socket says 172.21.15.42:39434) 2019-12-05T06:51:42.404 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.390+0000 7f3613fff700 1 -- 172.21.15.42:0/614486786 learned_addr learned my addr 172.21.15.42:0/614486786 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:51:42.404 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.390+0000 7f36137fe700 1 --2- 172.21.15.42:0/614486786 >> v2:172.21.15.40:3300/0 conn(0x7f3614117210 0x7f3614117640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:51:42.404 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.394+0000 7f3610ff9700 1 -- 172.21.15.42:0/614486786 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3245213540 0 0) 0x7f3614118cb0 con 0x7f361411ab10 2019-12-05T06:51:42.404 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.394+0000 7f3610ff9700 1 -- 172.21.15.42:0/614486786 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35f80030b0 con 0x7f361411ab10 2019-12-05T06:51:42.404 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.394+0000 7f36137fe700 1 -- 172.21.15.42:0/614486786 >> v1:172.21.15.42:6789/0 conn(0x7f361411ab10 legacy=0x7f3614119d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:42.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.394+0000 7f3610ff9700 1 -- 172.21.15.42:0/614486786 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 924455890 0 0) 0x7f361411a700 con 0x7f361412f0f0 2019-12-05T06:51:42.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.394+0000 7f36137fe700 1 -- 172.21.15.42:0/614486786 >> v1:172.21.15.40:6789/0 conn(0x7f361412f0f0 legacy=0x7f361412f520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:42.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.394+0000 7f36137fe700 1 -- 172.21.15.42:0/614486786 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f361412fd20 con 0x7f3614117210 2019-12-05T06:51:42.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.394+0000 7f36137fe700 1 --2- 172.21.15.42:0/614486786 >> v2:172.21.15.40:3300/0 conn(0x7f3614117210 0x7f3614117640 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f35fc001430 tx=0x7f35fc00c320).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:42.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.394+0000 7f36137fe700 1 --2- 172.21.15.42:0/614486786 >> v2:172.21.15.40:3300/0 conn(0x7f3614117210 0x7f3614117640 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f35fc001430 tx=0x7f35fc00c320).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:42.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.394+0000 7f36137fe700 1 --2- 172.21.15.42:0/614486786 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3614117210 0x7f3614117640 secure :-1 s=READY pgs=2564 cs=0 l=1 rx=0x7f35fc001430 tx=0x7f35fc00c320).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:51:42.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.394+0000 7f36137fe700 1 --2- 172.21.15.42:0/614486786 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3614117210 0x7f3614117640 secure :-1 s=THROTTLE_DONE pgs=2564 cs=0 l=1 rx=0x7f35fc001430 tx=0x7f35fc00c320).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:42.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.394+0000 7f36137fe700 1 --2- 172.21.15.42:0/614486786 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3614117210 0x7f3614117640 secure :-1 s=THROTTLE_DONE pgs=2564 cs=0 l=1 rx=0x7f35fc001430 tx=0x7f35fc00c320).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:42.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.394+0000 7f3610ff9700 1 -- 172.21.15.42:0/614486786 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f35fc00cb70 con 0x7f3614117210 2019-12-05T06:51:42.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.394+0000 7f36137fe700 1 --2- 172.21.15.42:0/614486786 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3614117210 0x7f3614117640 secure :-1 s=THROTTLE_DONE pgs=2564 cs=0 l=1 rx=0x7f35fc001430 tx=0x7f35fc00c320).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:42.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.394+0000 7f3610ff9700 1 -- 172.21.15.42:0/614486786 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35fc00c790 con 0x7f3614117210 2019-12-05T06:51:42.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.394+0000 7f3610ff9700 1 -- 172.21.15.42:0/614486786 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f35fc011070 con 0x7f3614117210 2019-12-05T06:51:42.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.394+0000 7f361a125700 1 -- 172.21.15.42:0/614486786 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f361412fff0 con 0x7f3614117210 2019-12-05T06:51:42.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.394+0000 7f361a125700 1 -- 172.21.15.42:0/614486786 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3614078310 con 0x7f3614117210 2019-12-05T06:51:42.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.394+0000 7f36137fe700 1 --2- 172.21.15.42:0/614486786 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3614117210 0x7f3614117640 secure :-1 s=THROTTLE_DONE pgs=2564 cs=0 l=1 rx=0x7f35fc001430 tx=0x7f35fc00c320).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:42.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.394+0000 7f3610ff9700 1 -- 172.21.15.42:0/614486786 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f35fc011340 con 0x7f3614117210 2019-12-05T06:51:42.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.394+0000 7f36137fe700 1 --2- 172.21.15.42:0/614486786 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3614117210 0x7f3614117640 secure :-1 s=THROTTLE_DONE pgs=2564 cs=0 l=1 rx=0x7f35fc001430 tx=0x7f35fc00c320).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:42.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.394+0000 7f3610ff9700 1 --2- 172.21.15.42:0/614486786 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f35f803b1c0 0x7f35f803d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:51:42.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.394+0000 7f3610ff9700 1 -- 172.21.15.42:0/614486786 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 12365+0+0 (secure 0 0 0) 0x7f35fc04b1e0 con 0x7f3614117210 2019-12-05T06:51:42.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.398+0000 7f3613fff700 1 --2- 172.21.15.42:0/614486786 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f35f803b1c0 0x7f35f803d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:51:42.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.398+0000 7f3613fff700 1 --2- 172.21.15.42:0/614486786 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f35f803b1c0 0x7f35f803d6d0 crc :-1 s=READY pgs=1419 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:51:42.408 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.402+0000 7f361a125700 1 -- 172.21.15.42:0/614486786 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f361405d110 con 0x7f3614117210 2019-12-05T06:51:42.414 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.410+0000 7f36137fe700 1 --2- 172.21.15.42:0/614486786 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3614117210 0x7f3614117640 secure :-1 s=THROTTLE_DONE pgs=2564 cs=0 l=1 rx=0x7f35fc001430 tx=0x7f35fc00c320).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:42.415 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.410+0000 7f3610ff9700 1 -- 172.21.15.42:0/614486786 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f35fc032260 con 0x7f3614117210 2019-12-05T06:51:42.582 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.578+0000 7f361a125700 1 -- 172.21.15.42:0/614486786 --> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7f361405b8c0 con 0x7f35f803b1c0 2019-12-05T06:51:42.582 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.578+0000 7f3610ff9700 1 -- 172.21.15.42:0/614486786 <== mgr.4108 v2:172.21.15.42:6800/11930 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) v1 ==== 41+0+0 (crc 0 0 0) 0x7f361405b8c0 con 0x7f35f803b1c0 2019-12-05T06:51:42.583 INFO:teuthology.orchestra.run.smithi042.stderr:instructed osd(s) 4 to deep-scrub 2019-12-05T06:51:42.585 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.582+0000 7f361a125700 1 -- 172.21.15.42:0/614486786 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f35f803b1c0 msgr2=0x7f35f803d6d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:42.586 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.582+0000 7f361a125700 1 --2- 172.21.15.42:0/614486786 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f35f803b1c0 0x7f35f803d6d0 crc :-1 s=READY pgs=1419 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:51:42.586 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.582+0000 7f361a125700 1 -- 172.21.15.42:0/614486786 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3614117210 msgr2=0x7f3614117640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:42.586 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.582+0000 7f361a125700 1 --2- 172.21.15.42:0/614486786 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3614117210 0x7f3614117640 secure :-1 s=READY pgs=2564 cs=0 l=1 rx=0x7f35fc001430 tx=0x7f35fc00c320).stop 2019-12-05T06:51:42.586 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.582+0000 7f361a125700 1 -- 172.21.15.42:0/614486786 shutdown_connections 2019-12-05T06:51:42.586 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.582+0000 7f361a125700 1 --2- 172.21.15.42:0/614486786 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f35f803b1c0 0x7f35f803d6d0 unknown :-1 s=CLOSED pgs=1419 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:51:42.586 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.582+0000 7f361a125700 1 --2- 172.21.15.42:0/614486786 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3614117210 0x7f3614117640 unknown :-1 s=CLOSED pgs=2564 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:51:42.587 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.582+0000 7f361a125700 1 -- 172.21.15.42:0/614486786 shutdown_connections 2019-12-05T06:51:42.587 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.582+0000 7f361a125700 1 -- 172.21.15.42:0/614486786 wait complete. 2019-12-05T06:51:42.587 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.582+0000 7f361a125700 1 -- 172.21.15.42:0/614486786 >> 172.21.15.42:0/614486786 conn(0x7f361408fde0 msgr2=0x7f3614141470 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:51:42.595 INFO:tasks.ceph:Scrubbing osd.5 2019-12-05T06:51:42.596 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2019-12-05T06:51:42.697 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.690+0000 7f70b1a48700 1 Processor -- start 2019-12-05T06:51:42.697 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.690+0000 7f70b1a48700 1 -- start start 2019-12-05T06:51:42.697 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.690+0000 7f70b1a48700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f70ac14df60 0x7f70ac1503a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:51:42.698 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.690+0000 7f70b1a48700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70ac07f080 con 0x7f70ac09f550 2019-12-05T06:51:42.698 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.690+0000 7f70b1a48700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70ac0a1000 con 0x7f70ac0a9a00 2019-12-05T06:51:42.698 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.690+0000 7f70b1a48700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70ac0a8f80 con 0x7f70ac14df60 2019-12-05T06:51:42.698 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.690+0000 7f70aaffd700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f70ac0a9a00 0x7f70ac07e880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:39442/0 (socket says 172.21.15.42:39442) 2019-12-05T06:51:42.698 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.690+0000 7f70aaffd700 1 -- 172.21.15.42:0/2558115370 learned_addr learned my addr 172.21.15.42:0/2558115370 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:51:42.699 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.690+0000 7f70abfff700 1 --2- 172.21.15.42:0/2558115370 >> v2:172.21.15.42:3300/0 conn(0x7f70ac14df60 0x7f70ac1503a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:51:42.699 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.690+0000 7f70aa7fc700 1 -- 172.21.15.42:0/2558115370 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1770266847 0 0) 0x7f70ac0a1000 con 0x7f70ac0a9a00 2019-12-05T06:51:42.699 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.690+0000 7f70aa7fc700 1 -- 172.21.15.42:0/2558115370 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f708c0030b0 con 0x7f70ac0a9a00 2019-12-05T06:51:42.699 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.690+0000 7f70abfff700 1 -- 172.21.15.42:0/2558115370 >> v1:172.21.15.42:6789/0 conn(0x7f70ac0a9a00 legacy=0x7f70ac07e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:42.700 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.690+0000 7f70abfff700 1 -- 172.21.15.42:0/2558115370 >> v1:172.21.15.40:6789/0 conn(0x7f70ac09f550 legacy=0x7f70ac144630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:42.700 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.690+0000 7f70abfff700 1 -- 172.21.15.42:0/2558115370 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70ac09ee30 con 0x7f70ac14df60 2019-12-05T06:51:42.700 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.690+0000 7f70abfff700 1 --2- 172.21.15.42:0/2558115370 >> v2:172.21.15.42:3300/0 conn(0x7f70ac14df60 0x7f70ac1503a0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f70a4002f30 tx=0x7f70a400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:42.700 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.690+0000 7f70abfff700 1 --2- 172.21.15.42:0/2558115370 >> v2:172.21.15.42:3300/0 conn(0x7f70ac14df60 0x7f70ac1503a0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f70a4002f30 tx=0x7f70a400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:42.700 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.690+0000 7f70abfff700 1 --2- 172.21.15.42:0/2558115370 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f70ac14df60 0x7f70ac1503a0 secure :-1 s=READY pgs=3381 cs=0 l=1 rx=0x7f70a4002f30 tx=0x7f70a400fbf0).ready entity=mon.1 client_cookie=1487704ebd9f3a8c server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:51:42.700 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.690+0000 7f70abfff700 1 --2- 172.21.15.42:0/2558115370 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f70ac14df60 0x7f70ac1503a0 secure :-1 s=THROTTLE_DONE pgs=3381 cs=0 l=1 rx=0x7f70a4002f30 tx=0x7f70a400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:42.701 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.690+0000 7f70aa7fc700 1 -- 172.21.15.42:0/2558115370 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f70a4012060 con 0x7f70ac14df60 2019-12-05T06:51:42.701 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.690+0000 7f70abfff700 1 --2- 172.21.15.42:0/2558115370 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f70ac14df60 0x7f70ac1503a0 secure :-1 s=THROTTLE_DONE pgs=3381 cs=0 l=1 rx=0x7f70a4002f30 tx=0x7f70a400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:42.701 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.690+0000 7f70abfff700 1 --2- 172.21.15.42:0/2558115370 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f70ac14df60 0x7f70ac1503a0 secure :-1 s=THROTTLE_DONE pgs=3381 cs=0 l=1 rx=0x7f70a4002f30 tx=0x7f70a400fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:42.701 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.690+0000 7f70aa7fc700 1 -- 172.21.15.42:0/2558115370 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70a40110c0 con 0x7f70ac14df60 2019-12-05T06:51:42.701 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.690+0000 7f70aa7fc700 1 -- 172.21.15.42:0/2558115370 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f70a4008da0 con 0x7f70ac14df60 2019-12-05T06:51:42.701 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.690+0000 7f70b1a48700 1 -- 172.21.15.42:0/2558115370 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f70ac14df60 msgr2=0x7f70ac1503a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:42.701 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.690+0000 7f70b1a48700 1 --2- 172.21.15.42:0/2558115370 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f70ac14df60 0x7f70ac1503a0 secure :-1 s=READY pgs=3381 cs=0 l=1 rx=0x7f70a4002f30 tx=0x7f70a400fbf0).stop 2019-12-05T06:51:42.702 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.694+0000 7f70b1a48700 1 -- 172.21.15.42:0/2558115370 shutdown_connections 2019-12-05T06:51:42.702 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.694+0000 7f70b1a48700 1 --2- 172.21.15.42:0/2558115370 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f70ac14df60 0x7f70ac1503a0 unknown :-1 s=CLOSED pgs=3381 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:51:42.702 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.694+0000 7f70b1a48700 1 -- 172.21.15.42:0/2558115370 shutdown_connections 2019-12-05T06:51:42.702 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.694+0000 7f70b1a48700 1 -- 172.21.15.42:0/2558115370 wait complete. 2019-12-05T06:51:42.702 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.694+0000 7f70b1a48700 1 -- 172.21.15.42:0/2558115370 >> 172.21.15.42:0/2558115370 conn(0x7f70ac0a9ef0 msgr2=0x7f70ac0aa320 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:51:42.705 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.694+0000 7f70b1a48700 1 Processor -- start 2019-12-05T06:51:42.705 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.694+0000 7f70b1a48700 1 -- start start 2019-12-05T06:51:42.705 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.694+0000 7f70b1a48700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f70ac143850 0x7f70ac111fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:51:42.705 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.694+0000 7f70b1a48700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f70ac07d390 0x7f70ac07d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:51:42.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.694+0000 7f70b1a48700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70ac112530 con 0x7f70ac08e680 2019-12-05T06:51:42.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.694+0000 7f70b1a48700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70ac065360 con 0x7f70ac07d390 2019-12-05T06:51:42.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.694+0000 7f70b1a48700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70ac065600 con 0x7f70ac143850 2019-12-05T06:51:42.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.694+0000 7f70abfff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f70ac07d390 0x7f70ac07d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:51:42.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.694+0000 7f70abfff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f70ac07d390 0x7f70ac07d7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:39414/0 (socket says 172.21.15.42:39414) 2019-12-05T06:51:42.706 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.694+0000 7f70abfff700 1 -- 172.21.15.42:0/3924064021 learned_addr learned my addr 172.21.15.42:0/3924064021 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:51:42.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.694+0000 7f70ab7fe700 1 --2- 172.21.15.42:0/3924064021 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f70ac143850 0x7f70ac111fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:51:42.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.694+0000 7f70abfff700 1 -- 172.21.15.42:0/3924064021 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f70ac143850 msgr2=0x7f70ac111fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:42.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.694+0000 7f70abfff700 1 --2- 172.21.15.42:0/3924064021 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f70ac143850 0x7f70ac111fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:51:42.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.694+0000 7f70abfff700 1 -- 172.21.15.42:0/3924064021 >> v1:172.21.15.40:6789/0 conn(0x7f70ac08e680 legacy=0x7f70ac08eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:42.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.694+0000 7f70abfff700 1 -- 172.21.15.42:0/3924064021 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70ac064f00 con 0x7f70ac07d390 2019-12-05T06:51:42.707 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.694+0000 7f70abfff700 1 --2- 172.21.15.42:0/3924064021 >> v2:172.21.15.42:3300/0 conn(0x7f70ac07d390 0x7f70ac07d7c0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f70ac065520 tx=0x7f70a40086a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:42.708 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.694+0000 7f70abfff700 1 --2- 172.21.15.42:0/3924064021 >> v2:172.21.15.42:3300/0 conn(0x7f70ac07d390 0x7f70ac07d7c0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f70ac065520 tx=0x7f70a40086a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:42.708 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.694+0000 7f70abfff700 1 --2- 172.21.15.42:0/3924064021 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f70ac07d390 0x7f70ac07d7c0 secure :-1 s=READY pgs=3382 cs=0 l=1 rx=0x7f70ac065520 tx=0x7f70a40086a0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:51:42.708 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.698+0000 7f70abfff700 1 --2- 172.21.15.42:0/3924064021 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f70ac07d390 0x7f70ac07d7c0 secure :-1 s=THROTTLE_DONE pgs=3382 cs=0 l=1 rx=0x7f70ac065520 tx=0x7f70a40086a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:42.708 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.698+0000 7f70a3fff700 1 -- 172.21.15.42:0/3924064021 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f70a4007350 con 0x7f70ac07d390 2019-12-05T06:51:42.708 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.698+0000 7f70abfff700 1 --2- 172.21.15.42:0/3924064021 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f70ac07d390 0x7f70ac07d7c0 secure :-1 s=THROTTLE_DONE pgs=3382 cs=0 l=1 rx=0x7f70ac065520 tx=0x7f70a40086a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:42.708 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.698+0000 7f70abfff700 1 --2- 172.21.15.42:0/3924064021 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f70ac07d390 0x7f70ac07d7c0 secure :-1 s=THROTTLE_DONE pgs=3382 cs=0 l=1 rx=0x7f70ac065520 tx=0x7f70a40086a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:42.709 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.698+0000 7f70a3fff700 1 -- 172.21.15.42:0/3924064021 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70a40089c0 con 0x7f70ac07d390 2019-12-05T06:51:42.709 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.698+0000 7f70b1a48700 1 -- 172.21.15.42:0/3924064021 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70ac07dd10 con 0x7f70ac07d390 2019-12-05T06:51:42.709 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.698+0000 7f70a3fff700 1 -- 172.21.15.42:0/3924064021 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f70a4011a80 con 0x7f70ac07d390 2019-12-05T06:51:42.709 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.698+0000 7f70b1a48700 1 -- 172.21.15.42:0/3924064021 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70ac12fde0 con 0x7f70ac07d390 2019-12-05T06:51:42.709 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.698+0000 7f70abfff700 1 --2- 172.21.15.42:0/3924064021 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f70ac07d390 0x7f70ac07d7c0 secure :-1 s=THROTTLE_DONE pgs=3382 cs=0 l=1 rx=0x7f70ac065520 tx=0x7f70a40086a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:42.709 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.698+0000 7f70a3fff700 1 -- 172.21.15.42:0/3924064021 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f70a4005540 con 0x7f70ac07d390 2019-12-05T06:51:42.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.698+0000 7f70abfff700 1 --2- 172.21.15.42:0/3924064021 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f70ac07d390 0x7f70ac07d7c0 secure :-1 s=THROTTLE_DONE pgs=3382 cs=0 l=1 rx=0x7f70ac065520 tx=0x7f70a40086a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:42.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.698+0000 7f70a3fff700 1 --2- 172.21.15.42:0/3924064021 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f708403acc0 0x7f708403d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:51:42.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.698+0000 7f70a3fff700 1 -- 172.21.15.42:0/3924064021 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 12365+0+0 (secure 0 0 0) 0x7f70a404ac40 con 0x7f70ac07d390 2019-12-05T06:51:42.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.698+0000 7f70ab7fe700 1 --2- 172.21.15.42:0/3924064021 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f708403acc0 0x7f708403d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:51:42.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.698+0000 7f70ab7fe700 1 --2- 172.21.15.42:0/3924064021 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f708403acc0 0x7f708403d1d0 crc :-1 s=READY pgs=1420 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:51:42.711 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.706+0000 7f70b1a48700 1 --2- 172.21.15.42:0/3924064021 >> [v2:172.21.15.40:6800/12489,v1:172.21.15.40:6801/12489] conn(0x7f70ac054260 0x7f70ac054690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:51:42.711 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.706+0000 7f70b1a48700 1 -- 172.21.15.42:0/3924064021 --> [v2:172.21.15.40:6800/12489,v1:172.21.15.40:6801/12489] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f70ac05d110 con 0x7f70ac054260 2019-12-05T06:51:42.711 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.706+0000 7f70aaffd700 1 --2- 172.21.15.42:0/3924064021 >> [v2:172.21.15.40:6800/12489,v1:172.21.15.40:6801/12489] conn(0x7f70ac054260 0x7f70ac054690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:51:42.711 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.706+0000 7f70aaffd700 1 -- 172.21.15.42:0/3924064021 >> [v2:172.21.15.40:6800/12489,v1:172.21.15.40:6801/12489] conn(0x7f70ac054260 msgr2=0x7f70ac054690 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2019-12-05T06:51:42.712 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.706+0000 7f70aaffd700 1 --2- 172.21.15.42:0/3924064021 >> [v2:172.21.15.40:6800/12489,v1:172.21.15.40:6801/12489] conn(0x7f70ac054260 0x7f70ac054690 crc :-1 s=READY pgs=208 cs=0 l=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:51:42.712 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.706+0000 7f70a3fff700 1 -- 172.21.15.42:0/3924064021 <== osd.5 v2:172.21.15.40:6800/12489 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23014 (crc 0 0 0) 0x7f70ac05d110 con 0x7f70ac054260 2019-12-05T06:51:42.743 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.738+0000 7f70b1a48700 1 -- 172.21.15.42:0/3924064021 --> [v2:172.21.15.40:6800/12489,v1:172.21.15.40:6801/12489] -- command(tid 2: {"var": "osd_debug_deep_scrub_sleep", "prefix": "config set", "val": ["0"]}) v1 -- 0x7f70ac05b8c0 con 0x7f70ac054260 2019-12-05T06:51:42.743 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.738+0000 7f70a3fff700 1 -- 172.21.15.42:0/3924064021 <== osd.5 v2:172.21.15.40:6800/12489 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f70ac05b8c0 con 0x7f70ac054260 2019-12-05T06:51:42.744 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2019-12-05T06:51:42.744 INFO:teuthology.orchestra.run.smithi042.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2019-12-05T06:51:42.744 INFO:teuthology.orchestra.run.smithi042.stdout:} 2019-12-05T06:51:42.745 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.742+0000 7f70b1a48700 1 -- 172.21.15.42:0/3924064021 >> [v2:172.21.15.40:6800/12489,v1:172.21.15.40:6801/12489] conn(0x7f70ac054260 msgr2=0x7f70ac054690 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:42.745 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.742+0000 7f70b1a48700 1 --2- 172.21.15.42:0/3924064021 >> [v2:172.21.15.40:6800/12489,v1:172.21.15.40:6801/12489] conn(0x7f70ac054260 0x7f70ac054690 crc :-1 s=READY pgs=208 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:51:42.746 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.742+0000 7f70b1a48700 1 -- 172.21.15.42:0/3924064021 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f708403acc0 msgr2=0x7f708403d1d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:42.746 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.742+0000 7f70b1a48700 1 --2- 172.21.15.42:0/3924064021 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f708403acc0 0x7f708403d1d0 crc :-1 s=READY pgs=1420 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:51:42.746 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.742+0000 7f70b1a48700 1 -- 172.21.15.42:0/3924064021 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f70ac07d390 msgr2=0x7f70ac07d7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:42.746 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.742+0000 7f70b1a48700 1 --2- 172.21.15.42:0/3924064021 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f70ac07d390 0x7f70ac07d7c0 secure :-1 s=READY pgs=3382 cs=0 l=1 rx=0x7f70ac065520 tx=0x7f70a40086a0).stop 2019-12-05T06:51:42.746 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.742+0000 7f70abfff700 1 -- 172.21.15.42:0/3924064021 reap_dead start 2019-12-05T06:51:42.746 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.742+0000 7f70b1a48700 1 -- 172.21.15.42:0/3924064021 shutdown_connections 2019-12-05T06:51:42.746 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.742+0000 7f70b1a48700 1 --2- 172.21.15.42:0/3924064021 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f70ac07d390 0x7f70ac07d7c0 unknown :-1 s=CLOSED pgs=3382 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:51:42.747 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.742+0000 7f70b1a48700 1 -- 172.21.15.42:0/3924064021 shutdown_connections 2019-12-05T06:51:42.747 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.742+0000 7f70b1a48700 1 -- 172.21.15.42:0/3924064021 wait complete. 2019-12-05T06:51:42.747 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.742+0000 7f70b1a48700 1 -- 172.21.15.42:0/3924064021 >> 172.21.15.42:0/3924064021 conn(0x7f70ac09f550 msgr2=0x7f70ac0a4ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:51:42.755 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2019-12-05T06:51:42.858 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.846+0000 7f7f93a0a700 1 Processor -- start 2019-12-05T06:51:42.858 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.846+0000 7f7f93a0a700 1 -- start start 2019-12-05T06:51:42.858 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.846+0000 7f7f93a0a700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7f8c0a7740 0x7f7f8c08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:51:42.858 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.846+0000 7f7f93a0a700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f7f8c09fae0 0x7f7f8c07eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:51:42.858 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.846+0000 7f7f93a0a700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f8c0a0710 con 0x7f7f8c07e720 2019-12-05T06:51:42.858 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.846+0000 7f7f93a0a700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f8c0a0b00 con 0x7f7f8c0a7740 2019-12-05T06:51:42.859 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.846+0000 7f7f93a0a700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f8c08e5b0 con 0x7f7f8c09fae0 2019-12-05T06:51:42.859 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.846+0000 7f7f917a6700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f7f8c07e720 0x7f7f8c09f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:39460/0 (socket says 172.21.15.42:39460) 2019-12-05T06:51:42.859 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.846+0000 7f7f917a6700 1 -- 172.21.15.42:0/1278204640 learned_addr learned my addr 172.21.15.42:0/1278204640 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:51:42.859 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.846+0000 7f7f927a8700 1 --2- 172.21.15.42:0/1278204640 >> v2:172.21.15.42:3300/0 conn(0x7f7f8c09fae0 0x7f7f8c07eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:51:42.859 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.846+0000 7f7f91fa7700 1 --2- 172.21.15.42:0/1278204640 >> v2:172.21.15.40:3300/0 conn(0x7f7f8c0a7740 0x7f7f8c08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:51:42.859 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.846+0000 7f7f90fa5700 1 -- 172.21.15.42:0/1278204640 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1308671532 0 0) 0x7f7f8c0a0710 con 0x7f7f8c07e720 2019-12-05T06:51:42.859 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.846+0000 7f7f90fa5700 1 -- 172.21.15.42:0/1278204640 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f740030b0 con 0x7f7f8c07e720 2019-12-05T06:51:42.860 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.846+0000 7f7f927a8700 1 -- 172.21.15.42:0/1278204640 >> v2:172.21.15.40:3300/0 conn(0x7f7f8c0a7740 msgr2=0x7f7f8c08df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:51:42.860 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.846+0000 7f7f927a8700 1 --2- 172.21.15.42:0/1278204640 >> v2:172.21.15.40:3300/0 conn(0x7f7f8c0a7740 0x7f7f8c08df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:51:42.860 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.846+0000 7f7f927a8700 1 -- 172.21.15.42:0/1278204640 >> v1:172.21.15.42:6789/0 conn(0x7f7f8c07e720 legacy=0x7f7f8c09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:42.860 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.846+0000 7f7f90fa5700 1 -- 172.21.15.42:0/1278204640 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2766515594 0 0) 0x7f7f740030b0 con 0x7f7f8c07e720 2019-12-05T06:51:42.861 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.846+0000 7f7f927a8700 1 -- 172.21.15.42:0/1278204640 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f8c07f0a0 con 0x7f7f8c09fae0 2019-12-05T06:51:42.861 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.846+0000 7f7f927a8700 1 --2- 172.21.15.42:0/1278204640 >> v2:172.21.15.42:3300/0 conn(0x7f7f8c09fae0 0x7f7f8c07eb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f7f80001990 tx=0x7f7f800019d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:42.862 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.846+0000 7f7f927a8700 1 --2- 172.21.15.42:0/1278204640 >> v2:172.21.15.42:3300/0 conn(0x7f7f8c09fae0 0x7f7f8c07eb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f7f80001990 tx=0x7f7f800019d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:42.862 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.846+0000 7f7f927a8700 1 --2- 172.21.15.42:0/1278204640 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7f8c09fae0 0x7f7f8c07eb50 secure :-1 s=READY pgs=3384 cs=0 l=1 rx=0x7f7f80001990 tx=0x7f7f800019d0).ready entity=mon.1 client_cookie=e8e22f4a8f988c0d server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:51:42.862 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.846+0000 7f7f927a8700 1 --2- 172.21.15.42:0/1278204640 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7f8c09fae0 0x7f7f8c07eb50 secure :-1 s=THROTTLE_DONE pgs=3384 cs=0 l=1 rx=0x7f7f80001990 tx=0x7f7f800019d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:42.862 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.846+0000 7f7f90fa5700 1 -- 172.21.15.42:0/1278204640 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7f80001750 con 0x7f7f8c09fae0 2019-12-05T06:51:42.862 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.846+0000 7f7f927a8700 1 --2- 172.21.15.42:0/1278204640 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7f8c09fae0 0x7f7f8c07eb50 secure :-1 s=THROTTLE_DONE pgs=3384 cs=0 l=1 rx=0x7f7f80001990 tx=0x7f7f800019d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:42.862 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.846+0000 7f7f90fa5700 1 -- 172.21.15.42:0/1278204640 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f80001560 con 0x7f7f8c09fae0 2019-12-05T06:51:42.862 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.846+0000 7f7f927a8700 1 --2- 172.21.15.42:0/1278204640 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7f8c09fae0 0x7f7f8c07eb50 secure :-1 s=THROTTLE_DONE pgs=3384 cs=0 l=1 rx=0x7f7f80001990 tx=0x7f7f800019d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:42.863 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.846+0000 7f7f93a0a700 1 -- 172.21.15.42:0/1278204640 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7f8c09fae0 msgr2=0x7f7f8c07eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:42.863 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.846+0000 7f7f90fa5700 1 -- 172.21.15.42:0/1278204640 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7f7c011480 con 0x7f7f8c09fae0 2019-12-05T06:51:42.863 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.846+0000 7f7f93a0a700 1 --2- 172.21.15.42:0/1278204640 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7f8c09fae0 0x7f7f8c07eb50 secure :-1 s=READY pgs=3384 cs=0 l=1 rx=0x7f7f80001990 tx=0x7f7f800019d0).stop 2019-12-05T06:51:42.863 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.850+0000 7f7f93a0a700 1 -- 172.21.15.42:0/1278204640 shutdown_connections 2019-12-05T06:51:42.863 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.850+0000 7f7f93a0a700 1 --2- 172.21.15.42:0/1278204640 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7f8c09fae0 0x7f7f8c07eb50 unknown :-1 s=CLOSED pgs=3384 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:51:42.863 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.850+0000 7f7f93a0a700 1 --2- 172.21.15.42:0/1278204640 >> v2:172.21.15.40:3300/0 conn(0x7f7f8c0a7740 0x7f7f8c08df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:51:42.863 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.850+0000 7f7f93a0a700 1 -- 172.21.15.42:0/1278204640 shutdown_connections 2019-12-05T06:51:42.864 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.850+0000 7f7f93a0a700 1 -- 172.21.15.42:0/1278204640 wait complete. 2019-12-05T06:51:42.864 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.850+0000 7f7f93a0a700 1 -- 172.21.15.42:0/1278204640 >> 172.21.15.42:0/1278204640 conn(0x7f7f8c0aabe0 msgr2=0x7f7f8c0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:51:42.864 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.850+0000 7f7f93a0a700 1 Processor -- start 2019-12-05T06:51:42.864 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.850+0000 7f7f93a0a700 1 -- start start 2019-12-05T06:51:42.864 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.850+0000 7f7f93a0a700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7f8c11c000 0x7f7f8c0752e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:51:42.864 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.850+0000 7f7f93a0a700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f8c1234c0 con 0x7f7f8c124c90 2019-12-05T06:51:42.865 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.850+0000 7f7f93a0a700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f8c076840 con 0x7f7f8c125440 2019-12-05T06:51:42.865 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.850+0000 7f7f93a0a700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f8c1236f0 con 0x7f7f8c11c000 2019-12-05T06:51:42.865 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.850+0000 7f7f917a6700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f7f8c125440 0x7f7f8c11b800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:39466/0 (socket says 172.21.15.42:39466) 2019-12-05T06:51:42.865 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.850+0000 7f7f917a6700 1 -- 172.21.15.42:0/2705640302 learned_addr learned my addr 172.21.15.42:0/2705640302 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:51:42.865 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.850+0000 7f7f927a8700 1 --2- 172.21.15.42:0/2705640302 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7f8c11c000 0x7f7f8c0752e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:51:42.865 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.850+0000 7f7f8a7fc700 1 -- 172.21.15.42:0/2705640302 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3532828426 0 0) 0x7f7f8c076840 con 0x7f7f8c125440 2019-12-05T06:51:42.865 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.850+0000 7f7f8a7fc700 1 -- 172.21.15.42:0/2705640302 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f6c0030b0 con 0x7f7f8c125440 2019-12-05T06:51:42.866 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.850+0000 7f7f8a7fc700 1 -- 172.21.15.42:0/2705640302 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4278128815 0 0) 0x7f7f8c1234c0 con 0x7f7f8c124c90 2019-12-05T06:51:42.866 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.850+0000 7f7f8a7fc700 1 -- 172.21.15.42:0/2705640302 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f8c076840 con 0x7f7f8c124c90 2019-12-05T06:51:42.866 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.850+0000 7f7f927a8700 1 -- 172.21.15.42:0/2705640302 >> v1:172.21.15.42:6789/0 conn(0x7f7f8c125440 legacy=0x7f7f8c11b800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:42.866 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.850+0000 7f7f927a8700 1 -- 172.21.15.42:0/2705640302 >> v1:172.21.15.40:6789/0 conn(0x7f7f8c124c90 legacy=0x7f7f8c141930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:42.867 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.854+0000 7f7f927a8700 1 -- 172.21.15.42:0/2705640302 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f8c077a80 con 0x7f7f8c11c000 2019-12-05T06:51:42.867 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.854+0000 7f7f927a8700 1 --2- 172.21.15.42:0/2705640302 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7f8c11c000 0x7f7f8c0752e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f7f80001990 tx=0x7f7f7c008a70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:42.867 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.854+0000 7f7f927a8700 1 --2- 172.21.15.42:0/2705640302 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7f8c11c000 0x7f7f8c0752e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f7f80001990 tx=0x7f7f7c008a70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:42.868 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.854+0000 7f7f927a8700 1 --2- 172.21.15.42:0/2705640302 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7f8c11c000 0x7f7f8c0752e0 secure :-1 s=READY pgs=755 cs=0 l=1 rx=0x7f7f80001990 tx=0x7f7f7c008a70).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:51:42.868 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.854+0000 7f7f927a8700 1 --2- 172.21.15.42:0/2705640302 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7f8c11c000 0x7f7f8c0752e0 secure :-1 s=THROTTLE_DONE pgs=755 cs=0 l=1 rx=0x7f7f80001990 tx=0x7f7f7c008a70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:42.868 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.854+0000 7f7f8a7fc700 1 -- 172.21.15.42:0/2705640302 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7f7c007350 con 0x7f7f8c11c000 2019-12-05T06:51:42.868 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.854+0000 7f7f927a8700 1 --2- 172.21.15.42:0/2705640302 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7f8c11c000 0x7f7f8c0752e0 secure :-1 s=THROTTLE_DONE pgs=755 cs=0 l=1 rx=0x7f7f80001990 tx=0x7f7f7c008a70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:42.868 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.854+0000 7f7f8a7fc700 1 -- 172.21.15.42:0/2705640302 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f7c008b60 con 0x7f7f8c11c000 2019-12-05T06:51:42.868 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.854+0000 7f7f927a8700 1 --2- 172.21.15.42:0/2705640302 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7f8c11c000 0x7f7f8c0752e0 secure :-1 s=THROTTLE_DONE pgs=755 cs=0 l=1 rx=0x7f7f80001990 tx=0x7f7f7c008a70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:42.868 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.854+0000 7f7f93a0a700 1 -- 172.21.15.42:0/2705640302 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f8c075830 con 0x7f7f8c11c000 2019-12-05T06:51:42.869 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.854+0000 7f7f8a7fc700 1 -- 172.21.15.42:0/2705640302 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7f7c011b00 con 0x7f7f8c11c000 2019-12-05T06:51:42.869 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.854+0000 7f7f93a0a700 1 -- 172.21.15.42:0/2705640302 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f8c075d30 con 0x7f7f8c11c000 2019-12-05T06:51:42.869 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.854+0000 7f7f927a8700 1 --2- 172.21.15.42:0/2705640302 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7f8c11c000 0x7f7f8c0752e0 secure :-1 s=THROTTLE_DONE pgs=755 cs=0 l=1 rx=0x7f7f80001990 tx=0x7f7f7c008a70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:42.870 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.854+0000 7f7f8a7fc700 1 -- 172.21.15.42:0/2705640302 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f7f7c005540 con 0x7f7f8c11c000 2019-12-05T06:51:42.870 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.854+0000 7f7f927a8700 1 --2- 172.21.15.42:0/2705640302 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7f8c11c000 0x7f7f8c0752e0 secure :-1 s=THROTTLE_DONE pgs=755 cs=0 l=1 rx=0x7f7f80001990 tx=0x7f7f7c008a70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:42.871 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.854+0000 7f7f8a7fc700 1 --2- 172.21.15.42:0/2705640302 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f7f6c03b450 0x7f7f6c03d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:51:42.871 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.854+0000 7f7f8a7fc700 1 -- 172.21.15.42:0/2705640302 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 12365+0+0 (secure 0 0 0) 0x7f7f7c04b200 con 0x7f7f8c11c000 2019-12-05T06:51:42.871 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.854+0000 7f7f91fa7700 1 --2- 172.21.15.42:0/2705640302 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f7f6c03b450 0x7f7f6c03d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:51:42.871 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.858+0000 7f7f91fa7700 1 --2- 172.21.15.42:0/2705640302 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f7f6c03b450 0x7f7f6c03d960 crc :-1 s=READY pgs=1421 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:51:42.871 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.862+0000 7f7f93a0a700 1 -- 172.21.15.42:0/2705640302 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f8c124920 con 0x7f7f8c11c000 2019-12-05T06:51:42.874 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.870+0000 7f7f927a8700 1 --2- 172.21.15.42:0/2705640302 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7f8c11c000 0x7f7f8c0752e0 secure :-1 s=THROTTLE_DONE pgs=755 cs=0 l=1 rx=0x7f7f80001990 tx=0x7f7f7c008a70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:42.874 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:42.870+0000 7f7f8a7fc700 1 -- 172.21.15.42:0/2705640302 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f7f7c032240 con 0x7f7f8c11c000 2019-12-05T06:51:43.043 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.038+0000 7f7f93a0a700 1 -- 172.21.15.42:0/2705640302 --> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) v1 -- 0x7f7f8c05d110 con 0x7f7f6c03b450 2019-12-05T06:51:43.044 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.038+0000 7f7f8a7fc700 1 -- 172.21.15.42:0/2705640302 <== mgr.4108 v2:172.21.15.42:6800/11930 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) v1 ==== 41+0+0 (crc 0 0 0) 0x7f7f8c05d110 con 0x7f7f6c03b450 2019-12-05T06:51:43.044 INFO:teuthology.orchestra.run.smithi042.stderr:instructed osd(s) 5 to deep-scrub 2019-12-05T06:51:43.047 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.042+0000 7f7f93a0a700 1 -- 172.21.15.42:0/2705640302 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f7f6c03b450 msgr2=0x7f7f6c03d960 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:43.047 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.042+0000 7f7f93a0a700 1 --2- 172.21.15.42:0/2705640302 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f7f6c03b450 0x7f7f6c03d960 crc :-1 s=READY pgs=1421 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:51:43.047 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.042+0000 7f7f93a0a700 1 -- 172.21.15.42:0/2705640302 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7f8c11c000 msgr2=0x7f7f8c0752e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:43.047 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.042+0000 7f7f93a0a700 1 --2- 172.21.15.42:0/2705640302 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7f8c11c000 0x7f7f8c0752e0 secure :-1 s=READY pgs=755 cs=0 l=1 rx=0x7f7f80001990 tx=0x7f7f7c008a70).stop 2019-12-05T06:51:43.047 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.042+0000 7f7f93a0a700 1 -- 172.21.15.42:0/2705640302 shutdown_connections 2019-12-05T06:51:43.047 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.042+0000 7f7f93a0a700 1 --2- 172.21.15.42:0/2705640302 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f7f6c03b450 0x7f7f6c03d960 unknown :-1 s=CLOSED pgs=1421 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:51:43.047 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.042+0000 7f7f93a0a700 1 --2- 172.21.15.42:0/2705640302 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7f8c11c000 0x7f7f8c0752e0 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:51:43.048 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.042+0000 7f7f93a0a700 1 -- 172.21.15.42:0/2705640302 shutdown_connections 2019-12-05T06:51:43.048 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.042+0000 7f7f93a0a700 1 -- 172.21.15.42:0/2705640302 wait complete. 2019-12-05T06:51:43.048 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.042+0000 7f7f93a0a700 1 -- 172.21.15.42:0/2705640302 >> 172.21.15.42:0/2705640302 conn(0x7f7f8c0a7f90 msgr2=0x7f7f8c0a3720 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:51:43.062 INFO:tasks.ceph:Scrubbing osd.6 2019-12-05T06:51:43.062 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2019-12-05T06:51:43.172 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.162+0000 7f7071e2b700 1 Processor -- start 2019-12-05T06:51:43.173 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.162+0000 7f7071e2b700 1 -- start start 2019-12-05T06:51:43.173 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.166+0000 7f7071e2b700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f706c09f550 0x7f706c144630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:51:43.173 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.166+0000 7f7071e2b700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f706c0a9a00 0x7f706c07e880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:51:43.173 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.166+0000 7f7071e2b700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f706c0a1000 con 0x7f706c07edd0 2019-12-05T06:51:43.173 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.166+0000 7f7071e2b700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f706c0a8f80 con 0x7f706c09f550 2019-12-05T06:51:43.173 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.166+0000 7f7071e2b700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f706c0a9220 con 0x7f706c0a9a00 2019-12-05T06:51:43.174 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.166+0000 7f706b7fe700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f706c0a9a00 0x7f706c07e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:51:43.174 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.166+0000 7f706bfff700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f706c09f550 0x7f706c144630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:51:43.174 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.166+0000 7f706b7fe700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f706c0a9a00 0x7f706c07e880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.42:39902/0 (socket says 172.21.15.42:39902) 2019-12-05T06:51:43.174 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.166+0000 7f706b7fe700 1 -- 172.21.15.42:0/322241147 learned_addr learned my addr 172.21.15.42:0/322241147 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:51:43.174 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.166+0000 7f706b7fe700 1 -- 172.21.15.42:0/322241147 >> v2:172.21.15.40:3300/0 conn(0x7f706c09f550 msgr2=0x7f706c144630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:51:43.174 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.166+0000 7f706b7fe700 1 --2- 172.21.15.42:0/322241147 >> v2:172.21.15.40:3300/0 conn(0x7f706c09f550 0x7f706c144630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:51:43.174 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.166+0000 7f706b7fe700 1 -- 172.21.15.42:0/322241147 >> v1:172.21.15.40:6789/0 conn(0x7f706c07edd0 legacy=0x7f706c147b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:43.175 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.166+0000 7f706b7fe700 1 -- 172.21.15.42:0/322241147 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f706c144b80 con 0x7f706c0a9a00 2019-12-05T06:51:43.175 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.166+0000 7f706b7fe700 1 --2- 172.21.15.42:0/322241147 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f706c0a9a00 0x7f706c07e880 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f7060008790 tx=0x7f706000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:43.175 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.166+0000 7f706b7fe700 1 --2- 172.21.15.42:0/322241147 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f706c0a9a00 0x7f706c07e880 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f7060008790 tx=0x7f706000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:43.175 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.166+0000 7f706b7fe700 1 --2- 172.21.15.42:0/322241147 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f706c0a9a00 0x7f706c07e880 secure :-1 s=READY pgs=756 cs=0 l=1 rx=0x7f7060008790 tx=0x7f706000fd30).ready entity=mon.2 client_cookie=65506e78f1c98a25 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:51:43.175 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.166+0000 7f706b7fe700 1 --2- 172.21.15.42:0/322241147 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f706c0a9a00 0x7f706c07e880 secure :-1 s=THROTTLE_DONE pgs=756 cs=0 l=1 rx=0x7f7060008790 tx=0x7f706000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:43.175 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.166+0000 7f706b7fe700 1 --2- 172.21.15.42:0/322241147 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f706c0a9a00 0x7f706c07e880 secure :-1 s=THROTTLE_DONE pgs=756 cs=0 l=1 rx=0x7f7060008790 tx=0x7f706000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:43.175 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.166+0000 7f706affd700 1 -- 172.21.15.42:0/322241147 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f70600113b0 con 0x7f706c0a9a00 2019-12-05T06:51:43.176 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.166+0000 7f706b7fe700 1 --2- 172.21.15.42:0/322241147 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f706c0a9a00 0x7f706c07e880 secure :-1 s=THROTTLE_DONE pgs=756 cs=0 l=1 rx=0x7f7060008790 tx=0x7f706000fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:43.176 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.166+0000 7f706affd700 1 -- 172.21.15.42:0/322241147 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7060008bb0 con 0x7f706c0a9a00 2019-12-05T06:51:43.176 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.166+0000 7f706affd700 1 -- 172.21.15.42:0/322241147 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7060011990 con 0x7f706c0a9a00 2019-12-05T06:51:43.176 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.166+0000 7f7071e2b700 1 -- 172.21.15.42:0/322241147 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f706c0a9a00 msgr2=0x7f706c07e880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:43.177 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.166+0000 7f7071e2b700 1 --2- 172.21.15.42:0/322241147 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f706c0a9a00 0x7f706c07e880 secure :-1 s=READY pgs=756 cs=0 l=1 rx=0x7f7060008790 tx=0x7f706000fd30).stop 2019-12-05T06:51:43.177 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.166+0000 7f7071e2b700 1 -- 172.21.15.42:0/322241147 shutdown_connections 2019-12-05T06:51:43.177 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.166+0000 7f7071e2b700 1 --2- 172.21.15.42:0/322241147 >> v2:172.21.15.40:3300/0 conn(0x7f706c09f550 0x7f706c144630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:51:43.177 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.166+0000 7f7071e2b700 1 --2- 172.21.15.42:0/322241147 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f706c0a9a00 0x7f706c07e880 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:51:43.177 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.166+0000 7f7071e2b700 1 -- 172.21.15.42:0/322241147 shutdown_connections 2019-12-05T06:51:43.177 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.166+0000 7f7071e2b700 1 -- 172.21.15.42:0/322241147 wait complete. 2019-12-05T06:51:43.177 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.166+0000 7f7071e2b700 1 -- 172.21.15.42:0/322241147 >> 172.21.15.42:0/322241147 conn(0x7f706c0a9ef0 msgr2=0x7f706c0aa320 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:51:43.180 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.170+0000 7f7071e2b700 1 Processor -- start 2019-12-05T06:51:43.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.170+0000 7f7071e2b700 1 -- start start 2019-12-05T06:51:43.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.170+0000 7f7071e2b700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f706c125dc0 0x7f706c133ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:51:43.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.170+0000 7f7071e2b700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f706c06eaa0 con 0x7f706c124ec0 2019-12-05T06:51:43.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.170+0000 7f7071e2b700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f706c06ecd0 con 0x7f706c11c2a0 2019-12-05T06:51:43.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.170+0000 7f7071e2b700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f706c11b550 con 0x7f706c125dc0 2019-12-05T06:51:43.181 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.170+0000 7f706b7fe700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f706c11c2a0 0x7f706c134000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:39482/0 (socket says 172.21.15.42:39482) 2019-12-05T06:51:43.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.170+0000 7f706b7fe700 1 -- 172.21.15.42:0/2415827962 learned_addr learned my addr 172.21.15.42:0/2415827962 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:51:43.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.170+0000 7f706bfff700 1 --2- 172.21.15.42:0/2415827962 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f706c125dc0 0x7f706c133ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:51:43.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.170+0000 7f7068ff9700 1 -- 172.21.15.42:0/2415827962 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1919240911 0 0) 0x7f706c06ecd0 con 0x7f706c11c2a0 2019-12-05T06:51:43.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.170+0000 7f7068ff9700 1 -- 172.21.15.42:0/2415827962 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f704c0030b0 con 0x7f706c11c2a0 2019-12-05T06:51:43.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.170+0000 7f7068ff9700 1 -- 172.21.15.42:0/2415827962 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 364552402 0 0) 0x7f706c06eaa0 con 0x7f706c124ec0 2019-12-05T06:51:43.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.170+0000 7f7068ff9700 1 -- 172.21.15.42:0/2415827962 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f706c06ecd0 con 0x7f706c124ec0 2019-12-05T06:51:43.182 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.170+0000 7f706bfff700 1 -- 172.21.15.42:0/2415827962 >> v1:172.21.15.42:6789/0 conn(0x7f706c11c2a0 legacy=0x7f706c134000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:43.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.170+0000 7f706bfff700 1 -- 172.21.15.42:0/2415827962 >> v1:172.21.15.40:6789/0 conn(0x7f706c124ec0 legacy=0x7f706c06e2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:43.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.170+0000 7f706bfff700 1 -- 172.21.15.42:0/2415827962 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f706c06ef00 con 0x7f706c125dc0 2019-12-05T06:51:43.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.170+0000 7f706bfff700 1 --2- 172.21.15.42:0/2415827962 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f706c125dc0 0x7f706c133ab0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f705c002f30 tx=0x7f705c010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:43.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.174+0000 7f706bfff700 1 --2- 172.21.15.42:0/2415827962 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f706c125dc0 0x7f706c133ab0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f705c002f30 tx=0x7f705c010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:43.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.174+0000 7f706bfff700 1 --2- 172.21.15.42:0/2415827962 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f706c125dc0 0x7f706c133ab0 secure :-1 s=READY pgs=757 cs=0 l=1 rx=0x7f705c002f30 tx=0x7f705c010aa0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:51:43.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.174+0000 7f706bfff700 1 --2- 172.21.15.42:0/2415827962 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f706c125dc0 0x7f706c133ab0 secure :-1 s=THROTTLE_DONE pgs=757 cs=0 l=1 rx=0x7f705c002f30 tx=0x7f705c010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:43.183 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.174+0000 7f7068ff9700 1 -- 172.21.15.42:0/2415827962 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f705c012550 con 0x7f706c125dc0 2019-12-05T06:51:43.184 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.174+0000 7f706bfff700 1 --2- 172.21.15.42:0/2415827962 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f706c125dc0 0x7f706c133ab0 secure :-1 s=THROTTLE_DONE pgs=757 cs=0 l=1 rx=0x7f705c002f30 tx=0x7f705c010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:43.184 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.174+0000 7f7068ff9700 1 -- 172.21.15.42:0/2415827962 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f705c007150 con 0x7f706c125dc0 2019-12-05T06:51:43.184 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.174+0000 7f706bfff700 1 --2- 172.21.15.42:0/2415827962 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f706c125dc0 0x7f706c133ab0 secure :-1 s=THROTTLE_DONE pgs=757 cs=0 l=1 rx=0x7f705c002f30 tx=0x7f705c010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:43.184 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.174+0000 7f7068ff9700 1 -- 172.21.15.42:0/2415827962 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f705c012b30 con 0x7f706c125dc0 2019-12-05T06:51:43.184 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.174+0000 7f7071e2b700 1 -- 172.21.15.42:0/2415827962 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f706c076520 con 0x7f706c125dc0 2019-12-05T06:51:43.184 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.174+0000 7f7071e2b700 1 -- 172.21.15.42:0/2415827962 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f706c076ad0 con 0x7f706c125dc0 2019-12-05T06:51:43.184 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.174+0000 7f706bfff700 1 --2- 172.21.15.42:0/2415827962 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f706c125dc0 0x7f706c133ab0 secure :-1 s=THROTTLE_DONE pgs=757 cs=0 l=1 rx=0x7f705c002f30 tx=0x7f705c010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:43.185 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.174+0000 7f7068ff9700 1 -- 172.21.15.42:0/2415827962 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f705c005540 con 0x7f706c125dc0 2019-12-05T06:51:43.185 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.174+0000 7f706bfff700 1 --2- 172.21.15.42:0/2415827962 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f706c125dc0 0x7f706c133ab0 secure :-1 s=THROTTLE_DONE pgs=757 cs=0 l=1 rx=0x7f705c002f30 tx=0x7f705c010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:43.185 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.174+0000 7f7068ff9700 1 --2- 172.21.15.42:0/2415827962 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f704c03b420 0x7f704c03d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:51:43.185 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.174+0000 7f7068ff9700 1 -- 172.21.15.42:0/2415827962 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 12365+0+0 (secure 0 0 0) 0x7f705c04ae70 con 0x7f706c125dc0 2019-12-05T06:51:43.185 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.174+0000 7f706b7fe700 1 --2- 172.21.15.42:0/2415827962 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f704c03b420 0x7f704c03d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:51:43.186 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.174+0000 7f706b7fe700 1 --2- 172.21.15.42:0/2415827962 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f704c03b420 0x7f704c03d930 crc :-1 s=READY pgs=1422 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:51:43.186 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.178+0000 7f7071e2b700 1 --2- 172.21.15.42:0/2415827962 >> [v2:172.21.15.40:6811/12491,v1:172.21.15.40:6814/12491] conn(0x7f706c11d400 0x7f706c054260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:51:43.187 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.178+0000 7f7071e2b700 1 -- 172.21.15.42:0/2415827962 --> [v2:172.21.15.40:6811/12491,v1:172.21.15.40:6814/12491] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f706c05d110 con 0x7f706c11d400 2019-12-05T06:51:43.187 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.178+0000 7f7070bc9700 1 --2- 172.21.15.42:0/2415827962 >> [v2:172.21.15.40:6811/12491,v1:172.21.15.40:6814/12491] conn(0x7f706c11d400 0x7f706c054260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:51:43.187 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.178+0000 7f7070bc9700 1 -- 172.21.15.42:0/2415827962 >> [v2:172.21.15.40:6811/12491,v1:172.21.15.40:6814/12491] conn(0x7f706c11d400 msgr2=0x7f706c054260 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2019-12-05T06:51:43.187 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.178+0000 7f7070bc9700 1 -- 172.21.15.42:0/2415827962 >> [v2:172.21.15.40:6811/12491,v1:172.21.15.40:6814/12491] conn(0x7f706c11d400 msgr2=0x7f706c054260 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2019-12-05T06:51:43.187 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.178+0000 7f7070bc9700 1 --2- 172.21.15.42:0/2415827962 >> [v2:172.21.15.40:6811/12491,v1:172.21.15.40:6814/12491] conn(0x7f706c11d400 0x7f706c054260 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-12-05T06:51:43.187 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.178+0000 7f7070bc9700 1 --2- 172.21.15.42:0/2415827962 >> [v2:172.21.15.40:6811/12491,v1:172.21.15.40:6814/12491] conn(0x7f706c11d400 0x7f706c054260 crc :-1 s=START_CONNECT pgs=0 cs=0 l=1 rx=0 tx=0)._fault waiting 0.200000 2019-12-05T06:51:43.384 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.382+0000 7f7070bc9700 1 --2- 172.21.15.42:0/2415827962 >> [v2:172.21.15.40:6811/12491,v1:172.21.15.40:6814/12491] conn(0x7f706c11d400 0x7f706c054260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:51:43.385 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.382+0000 7f7070bc9700 1 -- 172.21.15.42:0/2415827962 >> [v2:172.21.15.40:6811/12491,v1:172.21.15.40:6814/12491] conn(0x7f706c11d400 msgr2=0x7f706c054260 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2019-12-05T06:51:43.385 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.382+0000 7f7070bc9700 1 --2- 172.21.15.42:0/2415827962 >> [v2:172.21.15.40:6811/12491,v1:172.21.15.40:6814/12491] conn(0x7f706c11d400 0x7f706c054260 crc :-1 s=READY pgs=126 cs=0 l=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:51:43.387 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.382+0000 7f7068ff9700 1 -- 172.21.15.42:0/2415827962 <== osd.6 v2:172.21.15.40:6811/12491 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23014 (crc 0 0 0) 0x7f706c05d110 con 0x7f706c11d400 2019-12-05T06:51:43.429 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.426+0000 7f7071e2b700 1 -- 172.21.15.42:0/2415827962 --> [v2:172.21.15.40:6811/12491,v1:172.21.15.40:6814/12491] -- command(tid 2: {"var": "osd_debug_deep_scrub_sleep", "prefix": "config set", "val": ["0"]}) v1 -- 0x7f706c05b8c0 con 0x7f706c11d400 2019-12-05T06:51:43.430 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.426+0000 7f7068ff9700 1 -- 172.21.15.42:0/2415827962 <== osd.6 v2:172.21.15.40:6811/12491 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f706c05b8c0 con 0x7f706c11d400 2019-12-05T06:51:43.430 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2019-12-05T06:51:43.430 INFO:teuthology.orchestra.run.smithi042.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2019-12-05T06:51:43.430 INFO:teuthology.orchestra.run.smithi042.stdout:} 2019-12-05T06:51:43.432 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.426+0000 7f7071e2b700 1 -- 172.21.15.42:0/2415827962 >> [v2:172.21.15.40:6811/12491,v1:172.21.15.40:6814/12491] conn(0x7f706c11d400 msgr2=0x7f706c054260 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:43.432 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.426+0000 7f7071e2b700 1 --2- 172.21.15.42:0/2415827962 >> [v2:172.21.15.40:6811/12491,v1:172.21.15.40:6814/12491] conn(0x7f706c11d400 0x7f706c054260 crc :-1 s=READY pgs=126 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:51:43.432 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.426+0000 7f7071e2b700 1 -- 172.21.15.42:0/2415827962 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f704c03b420 msgr2=0x7f704c03d930 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:43.432 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.426+0000 7f7071e2b700 1 --2- 172.21.15.42:0/2415827962 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f704c03b420 0x7f704c03d930 crc :-1 s=READY pgs=1422 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:51:43.432 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.426+0000 7f7071e2b700 1 -- 172.21.15.42:0/2415827962 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f706c125dc0 msgr2=0x7f706c133ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:43.432 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.426+0000 7f7071e2b700 1 --2- 172.21.15.42:0/2415827962 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f706c125dc0 0x7f706c133ab0 secure :-1 s=READY pgs=757 cs=0 l=1 rx=0x7f705c002f30 tx=0x7f705c010aa0).stop 2019-12-05T06:51:43.433 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.426+0000 7f7070bc9700 1 -- 172.21.15.42:0/2415827962 reap_dead start 2019-12-05T06:51:43.433 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.426+0000 7f7071e2b700 1 -- 172.21.15.42:0/2415827962 shutdown_connections 2019-12-05T06:51:43.433 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.426+0000 7f7071e2b700 1 -- 172.21.15.42:0/2415827962 shutdown_connections 2019-12-05T06:51:43.433 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.426+0000 7f7071e2b700 1 -- 172.21.15.42:0/2415827962 wait complete. 2019-12-05T06:51:43.433 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.426+0000 7f7071e2b700 1 -- 172.21.15.42:0/2415827962 >> 172.21.15.42:0/2415827962 conn(0x7f706c0a71a0 msgr2=0x7f706c0a9960 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:51:43.442 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2019-12-05T06:51:43.549 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.542+0000 7f1ab92ba700 1 Processor -- start 2019-12-05T06:51:43.550 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.542+0000 7f1ab92ba700 1 -- start start 2019-12-05T06:51:43.550 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.542+0000 7f1ab92ba700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1ab408df80 0x7f1ab409f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:51:43.550 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.542+0000 7f1ab92ba700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f1ab409f830 0x7f1ab414efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:51:43.550 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.542+0000 7f1ab92ba700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ab409fc60 con 0x7f1ab40a7740 2019-12-05T06:51:43.551 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.542+0000 7f1ab92ba700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ab40a0b00 con 0x7f1ab409f830 2019-12-05T06:51:43.551 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.542+0000 7f1ab92ba700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ab408e5b0 con 0x7f1ab408df80 2019-12-05T06:51:43.551 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.542+0000 7f1ab37fe700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f1ab40a7740 0x7f1ab407e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:39490/0 (socket says 172.21.15.42:39490) 2019-12-05T06:51:43.551 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.542+0000 7f1ab37fe700 1 -- 172.21.15.42:0/1207762715 learned_addr learned my addr 172.21.15.42:0/1207762715 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:51:43.551 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.542+0000 7f1ab3fff700 1 --2- 172.21.15.42:0/1207762715 >> v2:172.21.15.40:3300/0 conn(0x7f1ab409f830 0x7f1ab414efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:51:43.551 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.542+0000 7f1ab37fe700 1 -- 172.21.15.42:0/1207762715 >> v1:172.21.15.42:6789/0 conn(0x7f1ab40a7740 legacy=0x7f1ab407e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2019-12-05T06:51:43.551 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.542+0000 7f1ab2ffd700 1 --2- 172.21.15.42:0/1207762715 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1ab408df80 0x7f1ab409f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:51:43.552 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.542+0000 7f1ab37fe700 1 -- 172.21.15.42:0/1207762715 >> v1:172.21.15.42:6789/0 conn(0x7f1ab40a7740 legacy=0x7f1ab407e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2019-12-05T06:51:43.552 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.542+0000 7f1ab37fe700 1 --1- 172.21.15.42:0/1207762715 >> v1:172.21.15.42:6789/0 conn(0x7f1ab40a7740 0x7f1ab407e720 :-1 s=OPENED pgs=3387 cs=1 l=1).handle_message read tag failed 2019-12-05T06:51:43.552 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.542+0000 7f1ab37fe700 1 --1- 172.21.15.42:0/1207762715 >> v1:172.21.15.42:6789/0 conn(0x7f1ab40a7740 0x7f1ab407e720 :-1 s=OPENED pgs=3387 cs=1 l=1).fault on lossy channel, failing 2019-12-05T06:51:43.552 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.542+0000 7f1ab3fff700 1 -- 172.21.15.42:0/1207762715 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1ab408df80 msgr2=0x7f1ab409f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:51:43.552 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.542+0000 7f1ab3fff700 1 --2- 172.21.15.42:0/1207762715 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1ab408df80 0x7f1ab409f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:51:43.552 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.542+0000 7f1ab3fff700 1 -- 172.21.15.42:0/1207762715 >> v1:172.21.15.42:6789/0 conn(0x7f1ab40a7740 legacy=0x7f1ab407e720 unknown :-1 s=STATE_CLOSED l=1).mark_down 2019-12-05T06:51:43.553 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.542+0000 7f1ab3fff700 1 -- 172.21.15.42:0/1207762715 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ab40a0710 con 0x7f1ab409f830 2019-12-05T06:51:43.553 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.542+0000 7f1ab2ffd700 1 --2- 172.21.15.42:0/1207762715 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1ab408df80 0x7f1ab409f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:51:43.553 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.542+0000 7f1ab3fff700 1 --2- 172.21.15.42:0/1207762715 >> v2:172.21.15.40:3300/0 conn(0x7f1ab409f830 0x7f1ab414efe0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f1aa40032b0 tx=0x7f1aa400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:43.553 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.542+0000 7f1ab3fff700 1 --2- 172.21.15.42:0/1207762715 >> v2:172.21.15.40:3300/0 conn(0x7f1ab409f830 0x7f1ab414efe0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f1aa40032b0 tx=0x7f1aa400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:43.553 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.542+0000 7f1ab3fff700 1 --2- 172.21.15.42:0/1207762715 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ab409f830 0x7f1ab414efe0 secure :-1 s=READY pgs=2570 cs=0 l=1 rx=0x7f1aa40032b0 tx=0x7f1aa400fff0).ready entity=mon.0 client_cookie=76c711863384a676 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:51:43.553 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.542+0000 7f1ab3fff700 1 --2- 172.21.15.42:0/1207762715 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ab409f830 0x7f1ab414efe0 secure :-1 s=THROTTLE_DONE pgs=2570 cs=0 l=1 rx=0x7f1aa40032b0 tx=0x7f1aa400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:43.553 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.542+0000 7f1ab27fc700 1 -- 172.21.15.42:0/1207762715 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1aa40108c0 con 0x7f1ab409f830 2019-12-05T06:51:43.554 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.542+0000 7f1ab3fff700 1 --2- 172.21.15.42:0/1207762715 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ab409f830 0x7f1ab414efe0 secure :-1 s=THROTTLE_DONE pgs=2570 cs=0 l=1 rx=0x7f1aa40032b0 tx=0x7f1aa400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:43.554 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.542+0000 7f1ab27fc700 1 -- 172.21.15.42:0/1207762715 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1aa4008b50 con 0x7f1ab409f830 2019-12-05T06:51:43.554 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.542+0000 7f1ab3fff700 1 --2- 172.21.15.42:0/1207762715 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ab409f830 0x7f1ab414efe0 secure :-1 s=THROTTLE_DONE pgs=2570 cs=0 l=1 rx=0x7f1aa40032b0 tx=0x7f1aa400fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:43.554 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.542+0000 7f1ab92ba700 1 -- 172.21.15.42:0/1207762715 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ab409f830 msgr2=0x7f1ab414efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:43.554 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.542+0000 7f1ab27fc700 1 -- 172.21.15.42:0/1207762715 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1aa40110c0 con 0x7f1ab409f830 2019-12-05T06:51:43.554 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.542+0000 7f1ab92ba700 1 --2- 172.21.15.42:0/1207762715 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ab409f830 0x7f1ab414efe0 secure :-1 s=READY pgs=2570 cs=0 l=1 rx=0x7f1aa40032b0 tx=0x7f1aa400fff0).stop 2019-12-05T06:51:43.555 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.546+0000 7f1ab92ba700 1 -- 172.21.15.42:0/1207762715 shutdown_connections 2019-12-05T06:51:43.555 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.546+0000 7f1ab92ba700 1 --2- 172.21.15.42:0/1207762715 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ab409f830 0x7f1ab414efe0 unknown :-1 s=CLOSED pgs=2570 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:51:43.555 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.546+0000 7f1ab92ba700 1 --2- 172.21.15.42:0/1207762715 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1ab408df80 0x7f1ab409f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:51:43.555 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.546+0000 7f1ab92ba700 1 -- 172.21.15.42:0/1207762715 shutdown_connections 2019-12-05T06:51:43.555 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.546+0000 7f1ab92ba700 1 -- 172.21.15.42:0/1207762715 wait complete. 2019-12-05T06:51:43.555 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.546+0000 7f1ab92ba700 1 -- 172.21.15.42:0/1207762715 >> 172.21.15.42:0/1207762715 conn(0x7f1ab40aabe0 msgr2=0x7f1ab40a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:51:43.559 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.546+0000 7f1ab92ba700 1 Processor -- start 2019-12-05T06:51:43.559 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.546+0000 7f1ab92ba700 1 -- start start 2019-12-05T06:51:43.559 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.546+0000 7f1ab92ba700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f1ab4133b00 0x7f1ab4133f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:51:43.559 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.546+0000 7f1ab92ba700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f1ab40742e0 0x7f1ab4076720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:51:43.559 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.546+0000 7f1ab92ba700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ab411a110 con 0x7f1ab411d0c0 2019-12-05T06:51:43.560 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.546+0000 7f1ab92ba700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ab4116600 con 0x7f1ab40742e0 2019-12-05T06:51:43.560 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.546+0000 7f1ab92ba700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ab411d4f0 con 0x7f1ab4133b00 2019-12-05T06:51:43.560 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.546+0000 7f1ab2ffd700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f1ab4133b00 0x7f1ab4133f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:51:43.560 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.546+0000 7f1ab2ffd700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f1ab4133b00 0x7f1ab4133f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:39460/0 (socket says 172.21.15.42:39460) 2019-12-05T06:51:43.560 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.546+0000 7f1ab2ffd700 1 -- 172.21.15.42:0/3440859776 learned_addr learned my addr 172.21.15.42:0/3440859776 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:51:43.560 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.546+0000 7f1ab37fe700 1 --1- 172.21.15.42:0/3440859776 >> v1:172.21.15.42:6789/0 conn(0x7f1ab411d0c0 0x7f1ab4121dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:39496/0 (socket says 172.21.15.42:39496) 2019-12-05T06:51:43.561 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.546+0000 7f1ab3fff700 1 --2- 172.21.15.42:0/3440859776 >> v2:172.21.15.40:3300/0 conn(0x7f1ab40742e0 0x7f1ab4076720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:51:43.561 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.550+0000 7f1ab2ffd700 1 -- 172.21.15.42:0/3440859776 >> v2:172.21.15.40:3300/0 conn(0x7f1ab40742e0 msgr2=0x7f1ab4076720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:43.561 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.550+0000 7f1ab2ffd700 1 --2- 172.21.15.42:0/3440859776 >> v2:172.21.15.40:3300/0 conn(0x7f1ab40742e0 0x7f1ab4076720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:51:43.561 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.550+0000 7f1ab2ffd700 1 -- 172.21.15.42:0/3440859776 >> v1:172.21.15.42:6789/0 conn(0x7f1ab411d0c0 legacy=0x7f1ab4121dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:43.561 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.550+0000 7f1ab2ffd700 1 -- 172.21.15.42:0/3440859776 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ab4134480 con 0x7f1ab4133b00 2019-12-05T06:51:43.561 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.550+0000 7f1ab2ffd700 1 --2- 172.21.15.42:0/3440859776 >> v2:172.21.15.42:3300/0 conn(0x7f1ab4133b00 0x7f1ab4133f30 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f1a9c002ef0 tx=0x7f1a9c010db0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:43.561 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.550+0000 7f1ab2ffd700 1 --2- 172.21.15.42:0/3440859776 >> v2:172.21.15.42:3300/0 conn(0x7f1ab4133b00 0x7f1ab4133f30 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f1a9c002ef0 tx=0x7f1a9c010db0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:43.562 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.550+0000 7f1ab2ffd700 1 --2- 172.21.15.42:0/3440859776 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1ab4133b00 0x7f1ab4133f30 secure :-1 s=READY pgs=3389 cs=0 l=1 rx=0x7f1a9c002ef0 tx=0x7f1a9c010db0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:51:43.562 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.550+0000 7f1ab2ffd700 1 --2- 172.21.15.42:0/3440859776 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1ab4133b00 0x7f1ab4133f30 secure :-1 s=THROTTLE_DONE pgs=3389 cs=0 l=1 rx=0x7f1a9c002ef0 tx=0x7f1a9c010db0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:43.562 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.550+0000 7f1ab2ffd700 1 --2- 172.21.15.42:0/3440859776 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1ab4133b00 0x7f1ab4133f30 secure :-1 s=THROTTLE_DONE pgs=3389 cs=0 l=1 rx=0x7f1a9c002ef0 tx=0x7f1a9c010db0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:43.562 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.550+0000 7f1ab2ffd700 1 --2- 172.21.15.42:0/3440859776 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1ab4133b00 0x7f1ab4133f30 secure :-1 s=THROTTLE_DONE pgs=3389 cs=0 l=1 rx=0x7f1a9c002ef0 tx=0x7f1a9c010db0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:43.562 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.550+0000 7f1aabfff700 1 -- 172.21.15.42:0/3440859776 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1a9c005ad0 con 0x7f1ab4133b00 2019-12-05T06:51:43.562 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.550+0000 7f1aabfff700 1 -- 172.21.15.42:0/3440859776 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a9c007200 con 0x7f1ab4133b00 2019-12-05T06:51:43.562 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.550+0000 7f1aabfff700 1 -- 172.21.15.42:0/3440859776 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1a9c012620 con 0x7f1ab4133b00 2019-12-05T06:51:43.563 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.550+0000 7f1ab92ba700 1 -- 172.21.15.42:0/3440859776 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ab4076c70 con 0x7f1ab4133b00 2019-12-05T06:51:43.563 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.550+0000 7f1ab92ba700 1 -- 172.21.15.42:0/3440859776 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ab4077220 con 0x7f1ab4133b00 2019-12-05T06:51:43.563 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.550+0000 7f1ab2ffd700 1 --2- 172.21.15.42:0/3440859776 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1ab4133b00 0x7f1ab4133f30 secure :-1 s=THROTTLE_DONE pgs=3389 cs=0 l=1 rx=0x7f1a9c002ef0 tx=0x7f1a9c010db0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:43.563 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.550+0000 7f1aabfff700 1 -- 172.21.15.42:0/3440859776 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f1a9c010180 con 0x7f1ab4133b00 2019-12-05T06:51:43.563 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.550+0000 7f1ab2ffd700 1 --2- 172.21.15.42:0/3440859776 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1ab4133b00 0x7f1ab4133f30 secure :-1 s=THROTTLE_DONE pgs=3389 cs=0 l=1 rx=0x7f1a9c002ef0 tx=0x7f1a9c010db0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:43.563 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.550+0000 7f1aabfff700 1 --2- 172.21.15.42:0/3440859776 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f1a9803ad50 0x7f1a9803d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:51:43.564 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.550+0000 7f1aabfff700 1 -- 172.21.15.42:0/3440859776 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 12365+0+0 (secure 0 0 0) 0x7f1a9c04b1d0 con 0x7f1ab4133b00 2019-12-05T06:51:43.564 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.550+0000 7f1ab37fe700 1 --2- 172.21.15.42:0/3440859776 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f1a9803ad50 0x7f1a9803d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:51:43.564 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.550+0000 7f1ab37fe700 1 --2- 172.21.15.42:0/3440859776 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f1a9803ad50 0x7f1a9803d260 crc :-1 s=READY pgs=1423 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:51:43.564 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.558+0000 7f1ab92ba700 1 -- 172.21.15.42:0/3440859776 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ab405d110 con 0x7f1ab4133b00 2019-12-05T06:51:43.567 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.562+0000 7f1ab2ffd700 1 --2- 172.21.15.42:0/3440859776 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1ab4133b00 0x7f1ab4133f30 secure :-1 s=THROTTLE_DONE pgs=3389 cs=0 l=1 rx=0x7f1a9c002ef0 tx=0x7f1a9c010db0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:43.567 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.562+0000 7f1aabfff700 1 -- 172.21.15.42:0/3440859776 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f1a9c0324b0 con 0x7f1ab4133b00 2019-12-05T06:51:43.733 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.730+0000 7f1ab92ba700 1 -- 172.21.15.42:0/3440859776 --> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) v1 -- 0x7f1ab405b8c0 con 0x7f1a9803ad50 2019-12-05T06:51:43.734 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.730+0000 7f1aabfff700 1 -- 172.21.15.42:0/3440859776 <== mgr.4108 v2:172.21.15.42:6800/11930 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) v1 ==== 41+0+0 (crc 0 0 0) 0x7f1ab405b8c0 con 0x7f1a9803ad50 2019-12-05T06:51:43.734 INFO:teuthology.orchestra.run.smithi042.stderr:instructed osd(s) 6 to deep-scrub 2019-12-05T06:51:43.736 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.734+0000 7f1ab92ba700 1 -- 172.21.15.42:0/3440859776 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f1a9803ad50 msgr2=0x7f1a9803d260 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:43.737 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.734+0000 7f1ab92ba700 1 --2- 172.21.15.42:0/3440859776 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f1a9803ad50 0x7f1a9803d260 crc :-1 s=READY pgs=1423 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:51:43.737 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.734+0000 7f1ab92ba700 1 -- 172.21.15.42:0/3440859776 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1ab4133b00 msgr2=0x7f1ab4133f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:43.737 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.734+0000 7f1ab92ba700 1 --2- 172.21.15.42:0/3440859776 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1ab4133b00 0x7f1ab4133f30 secure :-1 s=READY pgs=3389 cs=0 l=1 rx=0x7f1a9c002ef0 tx=0x7f1a9c010db0).stop 2019-12-05T06:51:43.737 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.734+0000 7f1ab92ba700 1 -- 172.21.15.42:0/3440859776 shutdown_connections 2019-12-05T06:51:43.737 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.734+0000 7f1ab92ba700 1 --2- 172.21.15.42:0/3440859776 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f1a9803ad50 0x7f1a9803d260 unknown :-1 s=CLOSED pgs=1423 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:51:43.737 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.734+0000 7f1ab92ba700 1 --2- 172.21.15.42:0/3440859776 >> v2:172.21.15.40:3300/0 conn(0x7f1ab40742e0 0x7f1ab4076720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:51:43.737 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.734+0000 7f1ab92ba700 1 --2- 172.21.15.42:0/3440859776 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1ab4133b00 0x7f1ab4133f30 unknown :-1 s=CLOSED pgs=3389 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:51:43.738 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.734+0000 7f1ab92ba700 1 -- 172.21.15.42:0/3440859776 shutdown_connections 2019-12-05T06:51:43.738 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.734+0000 7f1ab92ba700 1 -- 172.21.15.42:0/3440859776 wait complete. 2019-12-05T06:51:43.738 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.734+0000 7f1ab92ba700 1 -- 172.21.15.42:0/3440859776 >> 172.21.15.42:0/3440859776 conn(0x7f1ab40a7f90 msgr2=0x7f1ab40a3720 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:51:43.747 INFO:tasks.ceph:Scrubbing osd.7 2019-12-05T06:51:43.747 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2019-12-05T06:51:43.870 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.862+0000 7f5a7ede4700 1 Processor -- start 2019-12-05T06:51:43.870 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.862+0000 7f5a7ede4700 1 -- start start 2019-12-05T06:51:43.870 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.862+0000 7f5a7ede4700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5a7809f550 0x7f5a78144630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:51:43.871 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.862+0000 7f5a7ede4700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5a7814cf50 0x7f5a7814f390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:51:43.871 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.862+0000 7f5a7ede4700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a7807f080 con 0x7f5a780a9a00 2019-12-05T06:51:43.871 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.862+0000 7f5a7ede4700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a780a8f80 con 0x7f5a7809f550 2019-12-05T06:51:43.871 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.862+0000 7f5a7ede4700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a780a9220 con 0x7f5a7814cf50 2019-12-05T06:51:43.871 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.862+0000 7f5a7db82700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5a7814cf50 0x7f5a7814f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:51:43.871 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.862+0000 7f5a7db82700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5a7814cf50 0x7f5a7814f390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:39470/0 (socket says 172.21.15.42:39470) 2019-12-05T06:51:43.872 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.862+0000 7f5a7db82700 1 -- 172.21.15.42:0/2467262682 learned_addr learned my addr 172.21.15.42:0/2467262682 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:51:43.872 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.862+0000 7f5a7d381700 1 --2- 172.21.15.42:0/2467262682 >> v2:172.21.15.40:3300/0 conn(0x7f5a7809f550 0x7f5a78144630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:51:43.872 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.862+0000 7f5a7db82700 1 -- 172.21.15.42:0/2467262682 >> v2:172.21.15.40:3300/0 conn(0x7f5a7809f550 msgr2=0x7f5a78144630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:51:43.872 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.862+0000 7f5a7db82700 1 --2- 172.21.15.42:0/2467262682 >> v2:172.21.15.40:3300/0 conn(0x7f5a7809f550 0x7f5a78144630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:51:43.872 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.862+0000 7f5a7db82700 1 -- 172.21.15.42:0/2467262682 >> v1:172.21.15.40:6789/0 conn(0x7f5a780a9a00 legacy=0x7f5a7807e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:43.872 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.862+0000 7f5a7db82700 1 -- 172.21.15.42:0/2467262682 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a780a1000 con 0x7f5a7814cf50 2019-12-05T06:51:43.872 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.862+0000 7f5a7db82700 1 --2- 172.21.15.42:0/2467262682 >> v2:172.21.15.42:3300/0 conn(0x7f5a7814cf50 0x7f5a7814f390 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f5a700032b0 tx=0x7f5a700100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:43.873 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.862+0000 7f5a7db82700 1 --2- 172.21.15.42:0/2467262682 >> v2:172.21.15.42:3300/0 conn(0x7f5a7814cf50 0x7f5a7814f390 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f5a700032b0 tx=0x7f5a700100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:43.873 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.862+0000 7f5a7db82700 1 --2- 172.21.15.42:0/2467262682 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5a7814cf50 0x7f5a7814f390 secure :-1 s=READY pgs=3390 cs=0 l=1 rx=0x7f5a700032b0 tx=0x7f5a700100b0).ready entity=mon.1 client_cookie=60bdf8f52d51430 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:51:43.873 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.862+0000 7f5a7db82700 1 --2- 172.21.15.42:0/2467262682 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5a7814cf50 0x7f5a7814f390 secure :-1 s=THROTTLE_DONE pgs=3390 cs=0 l=1 rx=0x7f5a700032b0 tx=0x7f5a700100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:43.873 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.862+0000 7f5a7db82700 1 --2- 172.21.15.42:0/2467262682 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5a7814cf50 0x7f5a7814f390 secure :-1 s=THROTTLE_DONE pgs=3390 cs=0 l=1 rx=0x7f5a700032b0 tx=0x7f5a700100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:43.873 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.862+0000 7f5a6ffff700 1 -- 172.21.15.42:0/2467262682 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5a70010d60 con 0x7f5a7814cf50 2019-12-05T06:51:43.873 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.862+0000 7f5a7db82700 1 --2- 172.21.15.42:0/2467262682 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5a7814cf50 0x7f5a7814f390 secure :-1 s=THROTTLE_DONE pgs=3390 cs=0 l=1 rx=0x7f5a700032b0 tx=0x7f5a700100b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:43.873 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.862+0000 7f5a6ffff700 1 -- 172.21.15.42:0/2467262682 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a70010980 con 0x7f5a7814cf50 2019-12-05T06:51:43.874 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.862+0000 7f5a6ffff700 1 -- 172.21.15.42:0/2467262682 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5a700114a0 con 0x7f5a7814cf50 2019-12-05T06:51:43.874 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.866+0000 7f5a7ede4700 1 -- 172.21.15.42:0/2467262682 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5a7814cf50 msgr2=0x7f5a7814f390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:43.874 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.866+0000 7f5a7ede4700 1 --2- 172.21.15.42:0/2467262682 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5a7814cf50 0x7f5a7814f390 secure :-1 s=READY pgs=3390 cs=0 l=1 rx=0x7f5a700032b0 tx=0x7f5a700100b0).stop 2019-12-05T06:51:43.874 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.866+0000 7f5a7ede4700 1 -- 172.21.15.42:0/2467262682 shutdown_connections 2019-12-05T06:51:43.874 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.866+0000 7f5a7ede4700 1 --2- 172.21.15.42:0/2467262682 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5a7814cf50 0x7f5a7814f390 unknown :-1 s=CLOSED pgs=3390 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:51:43.874 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.866+0000 7f5a7ede4700 1 --2- 172.21.15.42:0/2467262682 >> v2:172.21.15.40:3300/0 conn(0x7f5a7809f550 0x7f5a78144630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:51:43.874 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.866+0000 7f5a7ede4700 1 -- 172.21.15.42:0/2467262682 shutdown_connections 2019-12-05T06:51:43.874 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.866+0000 7f5a7ede4700 1 -- 172.21.15.42:0/2467262682 wait complete. 2019-12-05T06:51:43.875 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.866+0000 7f5a7ede4700 1 -- 172.21.15.42:0/2467262682 >> 172.21.15.42:0/2467262682 conn(0x7f5a780a9ef0 msgr2=0x7f5a780aa320 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:51:43.878 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.866+0000 7f5a7ede4700 1 Processor -- start 2019-12-05T06:51:43.878 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.870+0000 7f5a7ede4700 1 -- start start 2019-12-05T06:51:43.878 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.870+0000 7f5a7ede4700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5a78064f90 0x7f5a78063cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:51:43.878 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.870+0000 7f5a7ede4700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a78064ae0 con 0x7f5a78143850 2019-12-05T06:51:43.879 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.870+0000 7f5a7ede4700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a780653c0 con 0x7f5a781165a0 2019-12-05T06:51:43.879 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.870+0000 7f5a7ede4700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a781169d0 con 0x7f5a78064f90 2019-12-05T06:51:43.879 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.870+0000 7f5a7cb80700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f5a781165a0 0x7f5a7812eac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:39510/0 (socket says 172.21.15.42:39510) 2019-12-05T06:51:43.879 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.870+0000 7f5a7cb80700 1 -- 172.21.15.42:0/2563917882 learned_addr learned my addr 172.21.15.42:0/2563917882 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:51:43.879 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.870+0000 7f5a7db82700 1 --2- 172.21.15.42:0/2563917882 >> v2:172.21.15.40:3300/0 conn(0x7f5a78064f90 0x7f5a78063cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:51:43.880 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.870+0000 7f5a6dffb700 1 -- 172.21.15.42:0/2563917882 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2685380888 0 0) 0x7f5a780653c0 con 0x7f5a781165a0 2019-12-05T06:51:43.880 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.870+0000 7f5a6dffb700 1 -- 172.21.15.42:0/2563917882 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a580030b0 con 0x7f5a781165a0 2019-12-05T06:51:43.880 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.870+0000 7f5a7db82700 1 -- 172.21.15.42:0/2563917882 >> v1:172.21.15.42:6789/0 conn(0x7f5a781165a0 legacy=0x7f5a7812eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:43.880 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.870+0000 7f5a7db82700 1 -- 172.21.15.42:0/2563917882 >> v1:172.21.15.40:6789/0 conn(0x7f5a78143850 legacy=0x7f5a7807d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:43.880 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.870+0000 7f5a6dffb700 1 -- 172.21.15.42:0/2563917882 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3900471379 0 0) 0x7f5a78064ae0 con 0x7f5a78143850 2019-12-05T06:51:43.880 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.870+0000 7f5a7db82700 1 -- 172.21.15.42:0/2563917882 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a78064240 con 0x7f5a78064f90 2019-12-05T06:51:43.881 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.870+0000 7f5a7db82700 1 --2- 172.21.15.42:0/2563917882 >> v2:172.21.15.40:3300/0 conn(0x7f5a78064f90 0x7f5a78063cf0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f5a700032b0 tx=0x7f5a70010c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:43.881 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.870+0000 7f5a7db82700 1 --2- 172.21.15.42:0/2563917882 >> v2:172.21.15.40:3300/0 conn(0x7f5a78064f90 0x7f5a78063cf0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f5a700032b0 tx=0x7f5a70010c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:43.881 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.870+0000 7f5a7db82700 1 --2- 172.21.15.42:0/2563917882 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5a78064f90 0x7f5a78063cf0 secure :-1 s=READY pgs=2573 cs=0 l=1 rx=0x7f5a700032b0 tx=0x7f5a70010c30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:51:43.881 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.870+0000 7f5a7db82700 1 --2- 172.21.15.42:0/2563917882 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5a78064f90 0x7f5a78063cf0 secure :-1 s=THROTTLE_DONE pgs=2573 cs=0 l=1 rx=0x7f5a700032b0 tx=0x7f5a70010c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:43.881 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.870+0000 7f5a6dffb700 1 -- 172.21.15.42:0/2563917882 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5a70011bb0 con 0x7f5a78064f90 2019-12-05T06:51:43.882 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.870+0000 7f5a7db82700 1 --2- 172.21.15.42:0/2563917882 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5a78064f90 0x7f5a78063cf0 secure :-1 s=THROTTLE_DONE pgs=2573 cs=0 l=1 rx=0x7f5a700032b0 tx=0x7f5a70010c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:43.882 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.870+0000 7f5a7ede4700 1 -- 172.21.15.42:0/2563917882 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a7808fdd0 con 0x7f5a78064f90 2019-12-05T06:51:43.882 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.870+0000 7f5a6dffb700 1 -- 172.21.15.42:0/2563917882 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a700110c0 con 0x7f5a78064f90 2019-12-05T06:51:43.882 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.870+0000 7f5a7db82700 1 --2- 172.21.15.42:0/2563917882 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5a78064f90 0x7f5a78063cf0 secure :-1 s=THROTTLE_DONE pgs=2573 cs=0 l=1 rx=0x7f5a700032b0 tx=0x7f5a70010c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:43.882 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.870+0000 7f5a6dffb700 1 -- 172.21.15.42:0/2563917882 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5a70005ab0 con 0x7f5a78064f90 2019-12-05T06:51:43.882 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.870+0000 7f5a7ede4700 1 -- 172.21.15.42:0/2563917882 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a7807e1f0 con 0x7f5a78064f90 2019-12-05T06:51:43.883 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.874+0000 7f5a7db82700 1 --2- 172.21.15.42:0/2563917882 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5a78064f90 0x7f5a78063cf0 secure :-1 s=THROTTLE_DONE pgs=2573 cs=0 l=1 rx=0x7f5a700032b0 tx=0x7f5a70010c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:43.883 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.874+0000 7f5a6dffb700 1 -- 172.21.15.42:0/2563917882 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f5a7000f280 con 0x7f5a78064f90 2019-12-05T06:51:43.883 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.874+0000 7f5a7db82700 1 --2- 172.21.15.42:0/2563917882 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5a78064f90 0x7f5a78063cf0 secure :-1 s=THROTTLE_DONE pgs=2573 cs=0 l=1 rx=0x7f5a700032b0 tx=0x7f5a70010c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:43.883 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.874+0000 7f5a6dffb700 1 --2- 172.21.15.42:0/2563917882 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5a5803b1c0 0x7f5a5803d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:51:43.883 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.874+0000 7f5a6dffb700 1 -- 172.21.15.42:0/2563917882 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 12365+0+0 (secure 0 0 0) 0x7f5a7004b3b0 con 0x7f5a78064f90 2019-12-05T06:51:43.883 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.874+0000 7f5a7d381700 1 --2- 172.21.15.42:0/2563917882 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5a5803b1c0 0x7f5a5803d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:51:43.884 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.874+0000 7f5a7d381700 1 -- 172.21.15.42:0/2563917882 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5a5803b1c0 msgr2=0x7f5a5803d6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2019-12-05T06:51:43.885 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.874+0000 7f5a7d381700 1 -- 172.21.15.42:0/2563917882 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5a5803b1c0 msgr2=0x7f5a5803d6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2019-12-05T06:51:43.885 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.874+0000 7f5a7d381700 1 --2- 172.21.15.42:0/2563917882 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5a5803b1c0 0x7f5a5803d6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-12-05T06:51:43.885 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.874+0000 7f5a7d381700 1 --2- 172.21.15.42:0/2563917882 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5a5803b1c0 0x7f5a5803d6d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rx=0 tx=0)._fault waiting 0.200000 2019-12-05T06:51:43.885 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.874+0000 7f5a7ede4700 1 --2- 172.21.15.42:0/2563917882 >> [v2:172.21.15.40:6813/12490,v1:172.21.15.40:6816/12490] conn(0x7f5a78054260 0x7f5a78054690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:51:43.885 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.878+0000 7f5a7ede4700 1 -- 172.21.15.42:0/2563917882 --> [v2:172.21.15.40:6813/12490,v1:172.21.15.40:6816/12490] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5a7805b8c0 con 0x7f5a78054260 2019-12-05T06:51:43.885 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.878+0000 7f5a7cb80700 1 --2- 172.21.15.42:0/2563917882 >> [v2:172.21.15.40:6813/12490,v1:172.21.15.40:6816/12490] conn(0x7f5a78054260 0x7f5a78054690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:51:43.885 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.878+0000 7f5a7cb80700 1 -- 172.21.15.42:0/2563917882 >> [v2:172.21.15.40:6813/12490,v1:172.21.15.40:6816/12490] conn(0x7f5a78054260 msgr2=0x7f5a78054690 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2019-12-05T06:51:43.886 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.878+0000 7f5a7cb80700 1 --2- 172.21.15.42:0/2563917882 >> [v2:172.21.15.40:6813/12490,v1:172.21.15.40:6816/12490] conn(0x7f5a78054260 0x7f5a78054690 crc :-1 s=READY pgs=178 cs=0 l=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:51:43.886 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.878+0000 7f5a6dffb700 1 -- 172.21.15.42:0/2563917882 <== osd.7 v2:172.21.15.40:6813/12490 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23014 (crc 0 0 0) 0x7f5a7805b8c0 con 0x7f5a78054260 2019-12-05T06:51:43.916 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.914+0000 7f5a7ede4700 1 -- 172.21.15.42:0/2563917882 --> [v2:172.21.15.40:6813/12490,v1:172.21.15.40:6816/12490] -- command(tid 2: {"var": "osd_debug_deep_scrub_sleep", "prefix": "config set", "val": ["0"]}) v1 -- 0x7f5a78054be0 con 0x7f5a78054260 2019-12-05T06:51:43.919 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.914+0000 7f5a6dffb700 1 -- 172.21.15.42:0/2563917882 <== osd.7 v2:172.21.15.40:6813/12490 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f5a78054be0 con 0x7f5a78054260 2019-12-05T06:51:43.919 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.914+0000 7f5a7ede4700 1 -- 172.21.15.42:0/2563917882 >> [v2:172.21.15.40:6813/12490,v1:172.21.15.40:6816/12490] conn(0x7f5a78054260 msgr2=0x7f5a78054690 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:43.919 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.914+0000 7f5a7ede4700 1 --2- 172.21.15.42:0/2563917882 >> [v2:172.21.15.40:6813/12490,v1:172.21.15.40:6816/12490] conn(0x7f5a78054260 0x7f5a78054690 crc :-1 s=READY pgs=178 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:51:43.919 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.914+0000 7f5a7ede4700 1 -- 172.21.15.42:0/2563917882 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5a5803b1c0 msgr2=0x7f5a5803d6d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2019-12-05T06:51:43.919 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.914+0000 7f5a7ede4700 1 --2- 172.21.15.42:0/2563917882 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f5a5803b1c0 0x7f5a5803d6d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:51:43.919 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.914+0000 7f5a7ede4700 1 -- 172.21.15.42:0/2563917882 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5a78064f90 msgr2=0x7f5a78063cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:43.920 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.914+0000 7f5a7ede4700 1 --2- 172.21.15.42:0/2563917882 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5a78064f90 0x7f5a78063cf0 secure :-1 s=READY pgs=2573 cs=0 l=1 rx=0x7f5a700032b0 tx=0x7f5a70010c30).stop 2019-12-05T06:51:43.920 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.914+0000 7f5a7db82700 1 -- 172.21.15.42:0/2563917882 reap_dead start 2019-12-05T06:51:43.920 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.914+0000 7f5a7ede4700 1 -- 172.21.15.42:0/2563917882 shutdown_connections 2019-12-05T06:51:43.920 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.914+0000 7f5a7ede4700 1 --2- 172.21.15.42:0/2563917882 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5a78064f90 0x7f5a78063cf0 unknown :-1 s=CLOSED pgs=2573 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:51:43.920 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.914+0000 7f5a7ede4700 1 -- 172.21.15.42:0/2563917882 shutdown_connections 2019-12-05T06:51:43.920 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.914+0000 7f5a7ede4700 1 -- 172.21.15.42:0/2563917882 wait complete. 2019-12-05T06:51:43.921 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:43.914+0000 7f5a7ede4700 1 -- 172.21.15.42:0/2563917882 >> 172.21.15.42:0/2563917882 conn(0x7f5a780a8b30 msgr2=0x7f5a78059700 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:51:43.921 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2019-12-05T06:51:43.921 INFO:teuthology.orchestra.run.smithi042.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2019-12-05T06:51:43.921 INFO:teuthology.orchestra.run.smithi042.stdout:} 2019-12-05T06:51:43.929 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2019-12-05T06:51:44.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.026+0000 7f7c23a72700 1 Processor -- start 2019-12-05T06:51:44.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.026+0000 7f7c23a72700 1 -- start start 2019-12-05T06:51:44.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.026+0000 7f7c23a72700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f7c1c0a7740 0x7f7c1c08df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:51:44.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.026+0000 7f7c23a72700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7c1c07e720 0x7f7c1c07eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:51:44.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.026+0000 7f7c23a72700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c1c0a0710 con 0x7f7c1c09f2e0 2019-12-05T06:51:44.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.026+0000 7f7c23a72700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c1c0a0b00 con 0x7f7c1c07e720 2019-12-05T06:51:44.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.026+0000 7f7c23a72700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c1c08e5b0 con 0x7f7c1c0a7740 2019-12-05T06:51:44.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.026+0000 7f7c2200f700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f7c1c0a7740 0x7f7c1c08df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:51:44.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.026+0000 7f7c22810700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f7c1c09f2e0 0x7f7c1c09f710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:39524/0 (socket says 172.21.15.42:39524) 2019-12-05T06:51:44.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.026+0000 7f7c22810700 1 -- 172.21.15.42:0/2657731853 learned_addr learned my addr 172.21.15.42:0/2657731853 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:51:44.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.026+0000 7f7c2180e700 1 --2- 172.21.15.42:0/2657731853 >> v2:172.21.15.40:3300/0 conn(0x7f7c1c07e720 0x7f7c1c07eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:51:44.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.026+0000 7f7c2200f700 1 -- 172.21.15.42:0/2657731853 >> v2:172.21.15.40:3300/0 conn(0x7f7c1c07e720 msgr2=0x7f7c1c07eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:51:44.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.026+0000 7f7c2200f700 1 --2- 172.21.15.42:0/2657731853 >> v2:172.21.15.40:3300/0 conn(0x7f7c1c07e720 0x7f7c1c07eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:51:44.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.026+0000 7f7c2200f700 1 -- 172.21.15.42:0/2657731853 >> v1:172.21.15.42:6789/0 conn(0x7f7c1c09f2e0 legacy=0x7f7c1c09f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:44.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.026+0000 7f7c2200f700 1 -- 172.21.15.42:0/2657731853 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c1c07f0a0 con 0x7f7c1c0a7740 2019-12-05T06:51:44.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.026+0000 7f7c2100d700 1 -- 172.21.15.42:0/2657731853 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 99958896 0 0) 0x7f7c1c0a0710 con 0x7f7c1c09f2e0 2019-12-05T06:51:44.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.026+0000 7f7c2180e700 1 --2- 172.21.15.42:0/2657731853 >> v2:172.21.15.40:3300/0 conn(0x7f7c1c07e720 0x7f7c1c07eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:51:44.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.026+0000 7f7c2200f700 1 --2- 172.21.15.42:0/2657731853 >> v2:172.21.15.42:3300/0 conn(0x7f7c1c0a7740 0x7f7c1c08df80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f7c14002f30 tx=0x7f7c1400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:44.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.026+0000 7f7c2200f700 1 --2- 172.21.15.42:0/2657731853 >> v2:172.21.15.42:3300/0 conn(0x7f7c1c0a7740 0x7f7c1c08df80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f7c14002f30 tx=0x7f7c1400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:44.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.026+0000 7f7c2200f700 1 --2- 172.21.15.42:0/2657731853 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7c1c0a7740 0x7f7c1c08df80 secure :-1 s=READY pgs=3393 cs=0 l=1 rx=0x7f7c14002f30 tx=0x7f7c1400fc70).ready entity=mon.1 client_cookie=9e05d9a2ac1284c server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:51:44.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.026+0000 7f7c2200f700 1 --2- 172.21.15.42:0/2657731853 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7c1c0a7740 0x7f7c1c08df80 secure :-1 s=THROTTLE_DONE pgs=3393 cs=0 l=1 rx=0x7f7c14002f30 tx=0x7f7c1400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:44.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.026+0000 7f7c2100d700 1 -- 172.21.15.42:0/2657731853 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7c140112b0 con 0x7f7c1c0a7740 2019-12-05T06:51:44.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.026+0000 7f7c2200f700 1 --2- 172.21.15.42:0/2657731853 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7c1c0a7740 0x7f7c1c08df80 secure :-1 s=THROTTLE_DONE pgs=3393 cs=0 l=1 rx=0x7f7c14002f30 tx=0x7f7c1400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:44.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.026+0000 7f7c2200f700 1 --2- 172.21.15.42:0/2657731853 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7c1c0a7740 0x7f7c1c08df80 secure :-1 s=THROTTLE_DONE pgs=3393 cs=0 l=1 rx=0x7f7c14002f30 tx=0x7f7c1400fc70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:44.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.026+0000 7f7c2100d700 1 -- 172.21.15.42:0/2657731853 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c14008ad0 con 0x7f7c1c0a7740 2019-12-05T06:51:44.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.026+0000 7f7c2100d700 1 -- 172.21.15.42:0/2657731853 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7c14011890 con 0x7f7c1c0a7740 2019-12-05T06:51:44.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.026+0000 7f7c23a72700 1 -- 172.21.15.42:0/2657731853 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7c1c0a7740 msgr2=0x7f7c1c08df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:44.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.026+0000 7f7c23a72700 1 --2- 172.21.15.42:0/2657731853 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7c1c0a7740 0x7f7c1c08df80 secure :-1 s=READY pgs=3393 cs=0 l=1 rx=0x7f7c14002f30 tx=0x7f7c1400fc70).stop 2019-12-05T06:51:44.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.030+0000 7f7c23a72700 1 -- 172.21.15.42:0/2657731853 shutdown_connections 2019-12-05T06:51:44.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.030+0000 7f7c23a72700 1 --2- 172.21.15.42:0/2657731853 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7c1c0a7740 0x7f7c1c08df80 unknown :-1 s=CLOSED pgs=3393 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:51:44.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.030+0000 7f7c23a72700 1 --2- 172.21.15.42:0/2657731853 >> v2:172.21.15.40:3300/0 conn(0x7f7c1c07e720 0x7f7c1c07eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:51:44.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.030+0000 7f7c23a72700 1 -- 172.21.15.42:0/2657731853 shutdown_connections 2019-12-05T06:51:44.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.030+0000 7f7c23a72700 1 -- 172.21.15.42:0/2657731853 wait complete. 2019-12-05T06:51:44.039 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.030+0000 7f7c23a72700 1 -- 172.21.15.42:0/2657731853 >> 172.21.15.42:0/2657731853 conn(0x7f7c1c0aabe0 msgr2=0x7f7c1c0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:51:44.042 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.030+0000 7f7c23a72700 1 Processor -- start 2019-12-05T06:51:44.042 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.030+0000 7f7c23a72700 1 -- start start 2019-12-05T06:51:44.042 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.030+0000 7f7c23a72700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7c1c118f60 0x7f7c1c119ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:51:44.042 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.030+0000 7f7c23a72700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c1c125830 con 0x7f7c1c118230 2019-12-05T06:51:44.042 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.030+0000 7f7c23a72700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c1c11a0f0 con 0x7f7c1c142870 2019-12-05T06:51:44.043 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.030+0000 7f7c23a72700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c1c118780 con 0x7f7c1c118f60 2019-12-05T06:51:44.043 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.030+0000 7f7c2180e700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f7c1c142870 0x7f7c1c12b1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:39530/0 (socket says 172.21.15.42:39530) 2019-12-05T06:51:44.043 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.030+0000 7f7c2180e700 1 -- 172.21.15.42:0/1680210844 learned_addr learned my addr 172.21.15.42:0/1680210844 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:51:44.043 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.030+0000 7f7c2200f700 1 --2- 172.21.15.42:0/1680210844 >> v2:172.21.15.40:3300/0 conn(0x7f7c1c118f60 0x7f7c1c119ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:51:44.043 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.030+0000 7f7c1affd700 1 -- 172.21.15.42:0/1680210844 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2124477789 0 0) 0x7f7c1c11a0f0 con 0x7f7c1c142870 2019-12-05T06:51:44.043 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.034+0000 7f7c1affd700 1 -- 172.21.15.42:0/1680210844 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7bfc0030b0 con 0x7f7c1c142870 2019-12-05T06:51:44.044 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.034+0000 7f7c1affd700 1 -- 172.21.15.42:0/1680210844 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1181112823 0 0) 0x7f7c1c125830 con 0x7f7c1c118230 2019-12-05T06:51:44.044 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.034+0000 7f7c1affd700 1 -- 172.21.15.42:0/1680210844 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c1c11a0f0 con 0x7f7c1c118230 2019-12-05T06:51:44.044 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.034+0000 7f7c2200f700 1 -- 172.21.15.42:0/1680210844 >> v1:172.21.15.42:6789/0 conn(0x7f7c1c142870 legacy=0x7f7c1c12b1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:44.044 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.034+0000 7f7c2200f700 1 -- 172.21.15.42:0/1680210844 >> v1:172.21.15.40:6789/0 conn(0x7f7c1c118230 legacy=0x7f7c1c125030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:44.044 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.034+0000 7f7c2200f700 1 -- 172.21.15.42:0/1680210844 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c1c138c20 con 0x7f7c1c118f60 2019-12-05T06:51:44.044 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.034+0000 7f7c2200f700 1 --2- 172.21.15.42:0/1680210844 >> v2:172.21.15.40:3300/0 conn(0x7f7c1c118f60 0x7f7c1c119ba0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f7c14002f30 tx=0x7f7c14011950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:44.044 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.034+0000 7f7c2200f700 1 --2- 172.21.15.42:0/1680210844 >> v2:172.21.15.40:3300/0 conn(0x7f7c1c118f60 0x7f7c1c119ba0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f7c14002f30 tx=0x7f7c14011950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:44.045 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.034+0000 7f7c2200f700 1 --2- 172.21.15.42:0/1680210844 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7c1c118f60 0x7f7c1c119ba0 secure :-1 s=READY pgs=2575 cs=0 l=1 rx=0x7f7c14002f30 tx=0x7f7c14011950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:51:44.045 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.034+0000 7f7c2200f700 1 --2- 172.21.15.42:0/1680210844 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7c1c118f60 0x7f7c1c119ba0 secure :-1 s=THROTTLE_DONE pgs=2575 cs=0 l=1 rx=0x7f7c14002f30 tx=0x7f7c14011950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:44.045 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.034+0000 7f7c1affd700 1 -- 172.21.15.42:0/1680210844 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7c14007150 con 0x7f7c1c118f60 2019-12-05T06:51:44.045 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.034+0000 7f7c2200f700 1 --2- 172.21.15.42:0/1680210844 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7c1c118f60 0x7f7c1c119ba0 secure :-1 s=THROTTLE_DONE pgs=2575 cs=0 l=1 rx=0x7f7c14002f30 tx=0x7f7c14011950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:44.045 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.034+0000 7f7c2200f700 1 --2- 172.21.15.42:0/1680210844 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7c1c118f60 0x7f7c1c119ba0 secure :-1 s=THROTTLE_DONE pgs=2575 cs=0 l=1 rx=0x7f7c14002f30 tx=0x7f7c14011950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:44.045 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.034+0000 7f7c1affd700 1 -- 172.21.15.42:0/1680210844 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c140114b0 con 0x7f7c1c118f60 2019-12-05T06:51:44.045 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.034+0000 7f7c23a72700 1 -- 172.21.15.42:0/1680210844 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c1c136c00 con 0x7f7c1c118f60 2019-12-05T06:51:44.046 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.034+0000 7f7c1affd700 1 -- 172.21.15.42:0/1680210844 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7c1400f550 con 0x7f7c1c118f60 2019-12-05T06:51:44.046 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.034+0000 7f7c23a72700 1 -- 172.21.15.42:0/1680210844 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c1c137130 con 0x7f7c1c118f60 2019-12-05T06:51:44.046 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.034+0000 7f7c2200f700 1 --2- 172.21.15.42:0/1680210844 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7c1c118f60 0x7f7c1c119ba0 secure :-1 s=THROTTLE_DONE pgs=2575 cs=0 l=1 rx=0x7f7c14002f30 tx=0x7f7c14011950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:44.046 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.038+0000 7f7c1affd700 1 -- 172.21.15.42:0/1680210844 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f7c14005540 con 0x7f7c1c118f60 2019-12-05T06:51:44.047 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.038+0000 7f7c2200f700 1 --2- 172.21.15.42:0/1680210844 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7c1c118f60 0x7f7c1c119ba0 secure :-1 s=THROTTLE_DONE pgs=2575 cs=0 l=1 rx=0x7f7c14002f30 tx=0x7f7c14011950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:44.047 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.038+0000 7f7c1affd700 1 --2- 172.21.15.42:0/1680210844 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f7bfc03b450 0x7f7bfc03d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:51:44.047 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.038+0000 7f7c1affd700 1 -- 172.21.15.42:0/1680210844 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 12365+0+0 (secure 0 0 0) 0x7f7c1404b820 con 0x7f7c1c118f60 2019-12-05T06:51:44.047 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.038+0000 7f7c2180e700 1 --2- 172.21.15.42:0/1680210844 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f7bfc03b450 0x7f7bfc03d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:51:44.047 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.038+0000 7f7c2180e700 1 --2- 172.21.15.42:0/1680210844 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f7bfc03b450 0x7f7bfc03d960 crc :-1 s=READY pgs=1424 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:51:44.048 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.042+0000 7f7c23a72700 1 -- 172.21.15.42:0/1680210844 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c1c05d110 con 0x7f7c1c118f60 2019-12-05T06:51:44.054 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.050+0000 7f7c2200f700 1 --2- 172.21.15.42:0/1680210844 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7c1c118f60 0x7f7c1c119ba0 secure :-1 s=THROTTLE_DONE pgs=2575 cs=0 l=1 rx=0x7f7c14002f30 tx=0x7f7c14011950).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:44.054 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.050+0000 7f7c1affd700 1 -- 172.21.15.42:0/1680210844 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f7c14032b00 con 0x7f7c1c118f60 2019-12-05T06:51:44.225 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.222+0000 7f7c23a72700 1 -- 172.21.15.42:0/1680210844 --> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) v1 -- 0x7f7c1c05b8c0 con 0x7f7bfc03b450 2019-12-05T06:51:44.225 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.222+0000 7f7c1affd700 1 -- 172.21.15.42:0/1680210844 <== mgr.4108 v2:172.21.15.42:6800/11930 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) v1 ==== 41+0+0 (crc 0 0 0) 0x7f7c1c05b8c0 con 0x7f7bfc03b450 2019-12-05T06:51:44.225 INFO:teuthology.orchestra.run.smithi042.stderr:instructed osd(s) 7 to deep-scrub 2019-12-05T06:51:44.228 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.226+0000 7f7c23a72700 1 -- 172.21.15.42:0/1680210844 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f7bfc03b450 msgr2=0x7f7bfc03d960 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:44.229 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.226+0000 7f7c23a72700 1 --2- 172.21.15.42:0/1680210844 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f7bfc03b450 0x7f7bfc03d960 crc :-1 s=READY pgs=1424 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:51:44.229 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.226+0000 7f7c23a72700 1 -- 172.21.15.42:0/1680210844 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7c1c118f60 msgr2=0x7f7c1c119ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:44.229 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.226+0000 7f7c23a72700 1 --2- 172.21.15.42:0/1680210844 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7c1c118f60 0x7f7c1c119ba0 secure :-1 s=READY pgs=2575 cs=0 l=1 rx=0x7f7c14002f30 tx=0x7f7c14011950).stop 2019-12-05T06:51:44.229 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.226+0000 7f7c23a72700 1 -- 172.21.15.42:0/1680210844 shutdown_connections 2019-12-05T06:51:44.229 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.226+0000 7f7c23a72700 1 --2- 172.21.15.42:0/1680210844 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f7bfc03b450 0x7f7bfc03d960 unknown :-1 s=CLOSED pgs=1424 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:51:44.229 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.226+0000 7f7c23a72700 1 --2- 172.21.15.42:0/1680210844 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7c1c118f60 0x7f7c1c119ba0 unknown :-1 s=CLOSED pgs=2575 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:51:44.229 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.226+0000 7f7c23a72700 1 -- 172.21.15.42:0/1680210844 shutdown_connections 2019-12-05T06:51:44.230 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.226+0000 7f7c23a72700 1 -- 172.21.15.42:0/1680210844 wait complete. 2019-12-05T06:51:44.230 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.226+0000 7f7c23a72700 1 -- 172.21.15.42:0/1680210844 >> 172.21.15.42:0/1680210844 conn(0x7f7c1c07de90 msgr2=0x7f7c1c09f5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:51:44.239 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2019-12-05T06:51:44.337 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.326+0000 7f6ba3482700 1 Processor -- start 2019-12-05T06:51:44.337 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.326+0000 7f6ba3482700 1 -- start start 2019-12-05T06:51:44.337 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.330+0000 7f6ba3482700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6b9c09fae0 0x7f6b9c14fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:51:44.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.330+0000 7f6ba3482700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b9c0a0710 con 0x7f6b9c0a7740 2019-12-05T06:51:44.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.330+0000 7f6ba3482700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b9c07ef20 con 0x7f6b9c08df80 2019-12-05T06:51:44.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.330+0000 7f6ba3482700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b9c0a0b00 con 0x7f6b9c09fae0 2019-12-05T06:51:44.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.330+0000 7f6ba2220700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6b9c09fae0 0x7f6b9c14fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:51:44.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.330+0000 7f6ba121e700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f6b9c08df80 0x7f6b9c09f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:39534/0 (socket says 172.21.15.42:39534) 2019-12-05T06:51:44.338 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.330+0000 7f6ba121e700 1 -- 172.21.15.42:0/1963759814 learned_addr learned my addr 172.21.15.42:0/1963759814 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:51:44.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.330+0000 7f6ba2220700 1 -- 172.21.15.42:0/1963759814 >> v1:172.21.15.42:6789/0 conn(0x7f6b9c08df80 legacy=0x7f6b9c09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:44.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.330+0000 7f6ba2220700 1 -- 172.21.15.42:0/1963759814 >> v1:172.21.15.40:6789/0 conn(0x7f6b9c0a7740 legacy=0x7f6b9c07e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:44.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.330+0000 7f6ba0a1d700 1 -- 172.21.15.42:0/1963759814 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3120926416 0 0) 0x7f6b9c07ef20 con 0x7f6b9c08df80 2019-12-05T06:51:44.339 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.330+0000 7f6ba2220700 1 -- 172.21.15.42:0/1963759814 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b9c0a9ba0 con 0x7f6b9c09fae0 2019-12-05T06:51:44.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.330+0000 7f6ba2220700 1 --2- 172.21.15.42:0/1963759814 >> v2:172.21.15.42:3300/0 conn(0x7f6b9c09fae0 0x7f6b9c14fff0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f6b94008b10 tx=0x7f6b9400ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:44.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.330+0000 7f6ba2220700 1 --2- 172.21.15.42:0/1963759814 >> v2:172.21.15.42:3300/0 conn(0x7f6b9c09fae0 0x7f6b9c14fff0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f6b94008b10 tx=0x7f6b9400ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:44.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.330+0000 7f6ba2220700 1 --2- 172.21.15.42:0/1963759814 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6b9c09fae0 0x7f6b9c14fff0 secure :-1 s=READY pgs=3396 cs=0 l=1 rx=0x7f6b94008b10 tx=0x7f6b9400ff70).ready entity=mon.1 client_cookie=16b3c5920e112b24 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:51:44.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.330+0000 7f6ba2220700 1 --2- 172.21.15.42:0/1963759814 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6b9c09fae0 0x7f6b9c14fff0 secure :-1 s=THROTTLE_DONE pgs=3396 cs=0 l=1 rx=0x7f6b94008b10 tx=0x7f6b9400ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:44.340 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.330+0000 7f6ba0a1d700 1 -- 172.21.15.42:0/1963759814 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6b94010970 con 0x7f6b9c09fae0 2019-12-05T06:51:44.341 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.330+0000 7f6ba2220700 1 --2- 172.21.15.42:0/1963759814 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6b9c09fae0 0x7f6b9c14fff0 secure :-1 s=THROTTLE_DONE pgs=3396 cs=0 l=1 rx=0x7f6b94008b10 tx=0x7f6b9400ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:44.341 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.330+0000 7f6ba2220700 1 --2- 172.21.15.42:0/1963759814 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6b9c09fae0 0x7f6b9c14fff0 secure :-1 s=THROTTLE_DONE pgs=3396 cs=0 l=1 rx=0x7f6b94008b10 tx=0x7f6b9400ff70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:44.341 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.330+0000 7f6ba0a1d700 1 -- 172.21.15.42:0/1963759814 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b94008b50 con 0x7f6b9c09fae0 2019-12-05T06:51:44.341 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.330+0000 7f6ba0a1d700 1 -- 172.21.15.42:0/1963759814 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6b940110c0 con 0x7f6b9c09fae0 2019-12-05T06:51:44.341 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.330+0000 7f6ba3482700 1 -- 172.21.15.42:0/1963759814 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6b9c09fae0 msgr2=0x7f6b9c14fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:44.341 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.330+0000 7f6ba3482700 1 --2- 172.21.15.42:0/1963759814 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6b9c09fae0 0x7f6b9c14fff0 secure :-1 s=READY pgs=3396 cs=0 l=1 rx=0x7f6b94008b10 tx=0x7f6b9400ff70).stop 2019-12-05T06:51:44.342 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.330+0000 7f6ba3482700 1 -- 172.21.15.42:0/1963759814 shutdown_connections 2019-12-05T06:51:44.342 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.330+0000 7f6ba3482700 1 --2- 172.21.15.42:0/1963759814 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6b9c09fae0 0x7f6b9c14fff0 unknown :-1 s=CLOSED pgs=3396 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:51:44.342 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.330+0000 7f6ba3482700 1 -- 172.21.15.42:0/1963759814 shutdown_connections 2019-12-05T06:51:44.342 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.330+0000 7f6ba3482700 1 -- 172.21.15.42:0/1963759814 wait complete. 2019-12-05T06:51:44.343 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.330+0000 7f6ba3482700 1 -- 172.21.15.42:0/1963759814 >> 172.21.15.42:0/1963759814 conn(0x7f6b9c0aabe0 msgr2=0x7f6b9c0a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:51:44.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.334+0000 7f6ba3482700 1 Processor -- start 2019-12-05T06:51:44.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.334+0000 7f6ba3482700 1 -- start start 2019-12-05T06:51:44.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.334+0000 7f6ba3482700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f6b9c11bbd0 0x7f6b9c11c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:51:44.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.334+0000 7f6ba3482700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6b9c0a4930 0x7f6b9c11a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:51:44.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.334+0000 7f6ba3482700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b9c0742d0 con 0x7f6b9c0a4d60 2019-12-05T06:51:44.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.334+0000 7f6ba3482700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b9c11c6d0 con 0x7f6b9c11bbd0 2019-12-05T06:51:44.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.334+0000 7f6ba3482700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b9c11c970 con 0x7f6b9c0a4930 2019-12-05T06:51:44.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.334+0000 7f6ba2220700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f6b9c0a4d60 0x7f6b9c11adb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:39544/0 (socket says 172.21.15.42:39544) 2019-12-05T06:51:44.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.334+0000 7f6ba2220700 1 -- 172.21.15.42:0/3038851073 learned_addr learned my addr 172.21.15.42:0/3038851073 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:51:44.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.334+0000 7f6ba121e700 1 --2- 172.21.15.42:0/3038851073 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6b9c0a4930 0x7f6b9c11a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:51:44.352 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.334+0000 7f6ba1a1f700 1 --2- 172.21.15.42:0/3038851073 >> v2:172.21.15.40:3300/0 conn(0x7f6b9c11bbd0 0x7f6b9c11c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:51:44.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.334+0000 7f6b927fc700 1 -- 172.21.15.42:0/3038851073 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2563221145 0 0) 0x7f6b9c0742d0 con 0x7f6b9c0a4d60 2019-12-05T06:51:44.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.334+0000 7f6b927fc700 1 -- 172.21.15.42:0/3038851073 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b840030b0 con 0x7f6b9c0a4d60 2019-12-05T06:51:44.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.334+0000 7f6b927fc700 1 -- 172.21.15.42:0/3038851073 <== mon.1 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 2369412457 0 0) 0x7f6b840030b0 con 0x7f6b9c0a4d60 2019-12-05T06:51:44.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.334+0000 7f6b927fc700 1 -- 172.21.15.42:0/3038851073 --> v1:172.21.15.42:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6b9c0742d0 con 0x7f6b9c0a4d60 2019-12-05T06:51:44.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.334+0000 7f6b927fc700 1 -- 172.21.15.42:0/3038851073 <== mon.1 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1876945237 0 0) 0x7f6b94010970 con 0x7f6b9c0a4d60 2019-12-05T06:51:44.353 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.334+0000 7f6b927fc700 1 -- 172.21.15.42:0/3038851073 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6b9c0a4930 msgr2=0x7f6b9c11a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:44.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.334+0000 7f6b927fc700 1 --2- 172.21.15.42:0/3038851073 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6b9c0a4930 0x7f6b9c11a860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:51:44.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.334+0000 7f6b927fc700 1 -- 172.21.15.42:0/3038851073 >> v2:172.21.15.40:3300/0 conn(0x7f6b9c11bbd0 msgr2=0x7f6b9c11c0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:44.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.334+0000 7f6b927fc700 1 --2- 172.21.15.42:0/3038851073 >> v2:172.21.15.40:3300/0 conn(0x7f6b9c11bbd0 0x7f6b9c11c0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:51:44.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.334+0000 7f6b927fc700 1 -- 172.21.15.42:0/3038851073 >> v1:172.21.15.42:6789/0 conn(0x7f6b9c0a4d60 legacy=0x7f6b9c11adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:44.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.334+0000 7f6b927fc700 1 --2- 172.21.15.42:0/3038851073 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6b840038a0 0x7f6b84003d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:51:44.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.334+0000 7f6b927fc700 1 --2- 172.21.15.42:0/3038851073 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6b84004420 0x7f6b84004850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:51:44.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.334+0000 7f6b927fc700 1 --2- 172.21.15.42:0/3038851073 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6b84004da0 0x7f6b8400b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:51:44.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.334+0000 7f6b927fc700 1 -- 172.21.15.42:0/3038851073 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b9c11c6d0 con 0x7f6b840038a0 2019-12-05T06:51:44.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.334+0000 7f6b927fc700 1 -- 172.21.15.42:0/3038851073 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b9c11c970 con 0x7f6b84004da0 2019-12-05T06:51:44.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.334+0000 7f6b927fc700 1 -- 172.21.15.42:0/3038851073 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b8400bbc0 con 0x7f6b84004420 2019-12-05T06:51:44.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.334+0000 7f6ba2220700 1 --2- 172.21.15.42:0/3038851073 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6b84004da0 0x7f6b8400b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:51:44.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.338+0000 7f6ba1a1f700 1 --2- 172.21.15.42:0/3038851073 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6b840038a0 0x7f6b84003d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:51:44.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.338+0000 7f6ba121e700 1 --2- 172.21.15.42:0/3038851073 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6b84004420 0x7f6b84004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:51:44.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.338+0000 7f6ba2220700 1 -- 172.21.15.42:0/3038851073 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6b84004420 msgr2=0x7f6b84004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:44.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.338+0000 7f6ba2220700 1 --2- 172.21.15.42:0/3038851073 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6b84004420 0x7f6b84004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:51:44.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.338+0000 7f6ba2220700 1 -- 172.21.15.42:0/3038851073 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6b840038a0 msgr2=0x7f6b84003d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:44.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.338+0000 7f6ba2220700 1 --2- 172.21.15.42:0/3038851073 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6b840038a0 0x7f6b84003d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:51:44.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.338+0000 7f6ba2220700 1 -- 172.21.15.42:0/3038851073 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b9c13f150 con 0x7f6b84004da0 2019-12-05T06:51:44.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.338+0000 7f6ba3482700 1 -- 172.21.15.42:0/3038851073 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b9c0a4930 con 0x7f6b84004da0 2019-12-05T06:51:44.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.338+0000 7f6ba2220700 1 --2- 172.21.15.42:0/3038851073 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6b84004da0 0x7f6b8400b670 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f6b940103c0 tx=0x7f6b9400f9b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:44.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.338+0000 7f6ba2220700 1 --2- 172.21.15.42:0/3038851073 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6b84004da0 0x7f6b8400b670 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f6b940103c0 tx=0x7f6b9400f9b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:44.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.338+0000 7f6ba3482700 1 -- 172.21.15.42:0/3038851073 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b9c11a860 con 0x7f6b84004da0 2019-12-05T06:51:44.358 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.338+0000 7f6ba2220700 1 --2- 172.21.15.42:0/3038851073 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6b84004da0 0x7f6b8400b670 secure :-1 s=READY pgs=3398 cs=0 l=1 rx=0x7f6b940103c0 tx=0x7f6b9400f9b0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:51:44.358 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.338+0000 7f6ba2220700 1 --2- 172.21.15.42:0/3038851073 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6b84004da0 0x7f6b8400b670 secure :-1 s=THROTTLE_DONE pgs=3398 cs=0 l=1 rx=0x7f6b940103c0 tx=0x7f6b9400f9b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:44.358 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.338+0000 7f6b927fc700 1 -- 172.21.15.42:0/3038851073 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6b94007a40 con 0x7f6b84004da0 2019-12-05T06:51:44.358 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.338+0000 7f6ba2220700 1 --2- 172.21.15.42:0/3038851073 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6b84004da0 0x7f6b8400b670 secure :-1 s=THROTTLE_DONE pgs=3398 cs=0 l=1 rx=0x7f6b940103c0 tx=0x7f6b9400f9b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:44.358 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.338+0000 7f6b927fc700 1 -- 172.21.15.42:0/3038851073 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b94005ad0 con 0x7f6b84004da0 2019-12-05T06:51:44.358 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.338+0000 7f6ba2220700 1 --2- 172.21.15.42:0/3038851073 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6b84004da0 0x7f6b8400b670 secure :-1 s=THROTTLE_DONE pgs=3398 cs=0 l=1 rx=0x7f6b940103c0 tx=0x7f6b9400f9b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:44.359 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.338+0000 7f6b927fc700 1 -- 172.21.15.42:0/3038851073 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6b94007540 con 0x7f6b84004da0 2019-12-05T06:51:44.359 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.338+0000 7f6ba2220700 1 --2- 172.21.15.42:0/3038851073 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6b84004da0 0x7f6b8400b670 secure :-1 s=THROTTLE_DONE pgs=3398 cs=0 l=1 rx=0x7f6b940103c0 tx=0x7f6b9400f9b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:44.359 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.338+0000 7f6b927fc700 1 -- 172.21.15.42:0/3038851073 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f6b940113d0 con 0x7f6b84004da0 2019-12-05T06:51:44.359 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.338+0000 7f6ba2220700 1 --2- 172.21.15.42:0/3038851073 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6b84004da0 0x7f6b8400b670 secure :-1 s=THROTTLE_DONE pgs=3398 cs=0 l=1 rx=0x7f6b940103c0 tx=0x7f6b9400f9b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:44.359 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.338+0000 7f6b927fc700 1 --2- 172.21.15.42:0/3038851073 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6b84043a80 0x7f6b84045f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:51:44.359 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.338+0000 7f6b927fc700 1 -- 172.21.15.42:0/3038851073 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 12365+0+0 (secure 0 0 0) 0x7f6b9404bbe0 con 0x7f6b84004da0 2019-12-05T06:51:44.359 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.338+0000 7f6ba1a1f700 1 --2- 172.21.15.42:0/3038851073 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6b84043a80 0x7f6b84045f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:51:44.360 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.338+0000 7f6ba1a1f700 1 --2- 172.21.15.42:0/3038851073 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6b84043a80 0x7f6b84045f90 crc :-1 s=READY pgs=1425 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:51:44.360 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.346+0000 7f6ba3482700 1 -- 172.21.15.42:0/3038851073 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b78003200 con 0x7f6b84004da0 2019-12-05T06:51:44.361 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.354+0000 7f6ba2220700 1 --2- 172.21.15.42:0/3038851073 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6b84004da0 0x7f6b8400b670 secure :-1 s=THROTTLE_DONE pgs=3398 cs=0 l=1 rx=0x7f6b940103c0 tx=0x7f6b9400f9b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:44.361 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.354+0000 7f6b927fc700 1 -- 172.21.15.42:0/3038851073 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f6b94007740 con 0x7f6b84004da0 2019-12-05T06:51:44.533 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.530+0000 7f6ba3482700 1 -- 172.21.15.42:0/3038851073 --> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f6b78002010 con 0x7f6b84043a80 2019-12-05T06:51:44.535 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.530+0000 7f6b927fc700 1 -- 172.21.15.42:0/3038851073 <== mgr.4108 v2:172.21.15.42:6800/11930 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+92174 (crc 0 0 0) 0x7f6b78002010 con 0x7f6b84043a80 2019-12-05T06:51:44.543 INFO:teuthology.orchestra.run.smithi042.stderr:dumped all 2019-12-05T06:51:44.544 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.538+0000 7f6ba3482700 1 -- 172.21.15.42:0/3038851073 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6b84043a80 msgr2=0x7f6b84045f90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:44.544 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.538+0000 7f6ba3482700 1 --2- 172.21.15.42:0/3038851073 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6b84043a80 0x7f6b84045f90 crc :-1 s=READY pgs=1425 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:51:44.544 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.538+0000 7f6ba3482700 1 -- 172.21.15.42:0/3038851073 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6b84004da0 msgr2=0x7f6b8400b670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:44.546 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.538+0000 7f6ba3482700 1 --2- 172.21.15.42:0/3038851073 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6b84004da0 0x7f6b8400b670 secure :-1 s=READY pgs=3398 cs=0 l=1 rx=0x7f6b940103c0 tx=0x7f6b9400f9b0).stop 2019-12-05T06:51:44.546 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.538+0000 7f6ba2220700 1 -- 172.21.15.42:0/3038851073 reap_dead start 2019-12-05T06:51:44.547 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.538+0000 7f6ba3482700 1 -- 172.21.15.42:0/3038851073 shutdown_connections 2019-12-05T06:51:44.547 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.538+0000 7f6ba3482700 1 --2- 172.21.15.42:0/3038851073 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6b84004da0 0x7f6b8400b670 unknown :-1 s=CLOSED pgs=3398 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:51:44.547 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.538+0000 7f6ba3482700 1 -- 172.21.15.42:0/3038851073 shutdown_connections 2019-12-05T06:51:44.548 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.538+0000 7f6ba3482700 1 -- 172.21.15.42:0/3038851073 wait complete. 2019-12-05T06:51:44.548 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:51:44.538+0000 7f6ba3482700 1 -- 172.21.15.42:0/3038851073 >> 172.21.15.42:0/3038851073 conn(0x7f6b9c0aabe0 msgr2=0x7f6b9c1418a0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:51:44.549 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:51:44.557 INFO:teuthology.orchestra.run.smithi042.stdout:{"pg_ready":true,"pg_map":{"version":3378,"stamp":"2019-12-05T06:51:43.616796+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":413387478,"num_objects":154,"num_object_clones":2,"num_object_copies":314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":154,"num_whiteouts":0,"num_read":9494,"num_read_kb":12806383,"num_write":49969,"num_write_kb":1021536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39707,"ondisk_log_size":39707,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":9209912,"kb_used_data":821240,"kb_used_omap":0,"kb_used_meta":8388608,"kb_avail":745764808,"statfs":{"total":773094113280,"available":763663163392,"internally_reserved":8589934592,"allocated":840949760,"data_stored":835057830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8589934592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":719691,"num_objects":4,"num_object_clones":2,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":37,"num_write_kb":726,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002538"},"pg_stats":[{"pgid":"3.7","version":"126'6396","reported_seq":"8185","reported_epoch":"127","state":"active+clean","last_fresh":"2019-12-05T06:51:36.572352+0000","last_change":"2019-12-05T06:51:36.575408+0000","last_active":"2019-12-05T06:51:36.572352+0000","last_peered":"2019-12-05T06:51:36.572352+0000","last_clean":"2019-12-05T06:51:36.572352+0000","last_became_active":"2019-12-05T05:03:35.332717+0000","last_became_peered":"2019-12-05T05:03:35.332717+0000","last_unstale":"2019-12-05T06:51:36.572352+0000","last_undegraded":"2019-12-05T06:51:36.572352+0000","last_fullsized":"2019-12-05T06:51:36.572352+0000","mapping_epoch":21,"log_start":"118'3311","ondisk_log_start":"118'3311","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:34.235090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:34.235090+0000","last_clean_scrub_stamp":"2019-12-05T05:03:34.235090+0000","log_size":3085,"ondisk_log_size":3085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":245,"num_read_kb":1968,"num_write":4375,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7,0],"acting":[3,5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"210","length":"1e"}]},{"pgid":"1.5","version":"0'0","reported_seq":"119","reported_epoch":"126","state":"active+clean","last_fresh":"2019-12-05T06:51:32.154298+0000","last_change":"2019-12-05T05:03:29.184885+0000","last_active":"2019-12-05T06:51:32.154298+0000","last_peered":"2019-12-05T06:51:32.154298+0000","last_clean":"2019-12-05T06:51:32.154298+0000","last_became_active":"2019-12-05T05:03:29.184431+0000","last_became_peered":"2019-12-05T05:03:29.184431+0000","last_unstale":"2019-12-05T06:51:32.154298+0000","last_undegraded":"2019-12-05T06:51:32.154298+0000","last_fullsized":"2019-12-05T06:51:32.154298+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:28.161818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:28.161818+0000","last_clean_scrub_stamp":"2019-12-05T05:03:28.161818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"126'2439","reported_seq":"3006","reported_epoch":"126","state":"active+clean","last_fresh":"2019-12-05T06:51:35.648728+0000","last_change":"2019-12-05T05:03:32.225175+0000","last_active":"2019-12-05T06:51:35.648728+0000","last_peered":"2019-12-05T06:51:35.648728+0000","last_clean":"2019-12-05T06:51:35.648728+0000","last_became_active":"2019-12-05T05:03:32.224827+0000","last_became_peered":"2019-12-05T05:03:32.224827+0000","last_unstale":"2019-12-05T06:51:35.648728+0000","last_undegraded":"2019-12-05T06:51:35.648728+0000","last_fullsized":"2019-12-05T06:51:35.648728+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:31.200243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:31.200243+0000","last_clean_scrub_stamp":"2019-12-05T05:03:31.200243+0000","log_size":2439,"ondisk_log_size":2439,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":67108864,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":679,"num_read_kb":1201860,"num_write":2587,"num_write_kb":72175,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"124'6219","reported_seq":"8099","reported_epoch":"127","state":"active+clean","last_fresh":"2019-12-05T06:51:36.996048+0000","last_change":"2019-12-05T06:51:36.996591+0000","last_active":"2019-12-05T06:51:36.996048+0000","last_peered":"2019-12-05T06:51:36.996048+0000","last_clean":"2019-12-05T06:51:36.996048+0000","last_became_active":"2019-12-05T05:03:35.332648+0000","last_became_peered":"2019-12-05T05:03:35.332648+0000","last_unstale":"2019-12-05T06:51:36.996048+0000","last_undegraded":"2019-12-05T06:51:36.996048+0000","last_fullsized":"2019-12-05T06:51:36.996048+0000","mapping_epoch":21,"log_start":"117'3205","ondisk_log_start":"117'3205","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:34.235090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:34.235090+0000","last_clean_scrub_stamp":"2019-12-05T05:03:34.235090+0000","log_size":3014,"ondisk_log_size":3014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":233,"num_read_kb":6040,"num_write":4390,"num_write_kb":51866,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,7],"acting":[0,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"210","length":"1e"}]},{"pgid":"1.4","version":"0'0","reported_seq":"119","reported_epoch":"126","state":"active+clean","last_fresh":"2019-12-05T06:51:32.149298+0000","last_change":"2019-12-05T05:03:29.182769+0000","last_active":"2019-12-05T06:51:32.149298+0000","last_peered":"2019-12-05T06:51:32.149298+0000","last_clean":"2019-12-05T06:51:32.149298+0000","last_became_active":"2019-12-05T05:03:29.182409+0000","last_became_peered":"2019-12-05T05:03:29.182409+0000","last_unstale":"2019-12-05T06:51:32.149298+0000","last_undegraded":"2019-12-05T06:51:32.149298+0000","last_fullsized":"2019-12-05T06:51:32.149298+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:28.161818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:28.161818+0000","last_clean_scrub_stamp":"2019-12-05T05:03:28.161818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"125'1441","reported_seq":"2094","reported_epoch":"126","state":"active+clean","last_fresh":"2019-12-05T06:51:31.448966+0000","last_change":"2019-12-05T05:03:32.228176+0000","last_active":"2019-12-05T06:51:31.448966+0000","last_peered":"2019-12-05T06:51:31.448966+0000","last_clean":"2019-12-05T06:51:31.448966+0000","last_became_active":"2019-12-05T05:03:32.227690+0000","last_became_peered":"2019-12-05T05:03:32.227690+0000","last_unstale":"2019-12-05T06:51:31.448966+0000","last_undegraded":"2019-12-05T06:51:31.448966+0000","last_fullsized":"2019-12-05T06:51:31.448966+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:31.200243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:31.200243+0000","last_clean_scrub_stamp":"2019-12-05T05:03:31.200243+0000","log_size":1441,"ondisk_log_size":1441,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":25166390,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":832,"num_read_kb":914148,"num_write":1232,"num_write_kb":29663,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"126'6687","reported_seq":"8459","reported_epoch":"127","state":"active+clean","last_fresh":"2019-12-05T06:51:36.995882+0000","last_change":"2019-12-05T06:51:36.996364+0000","last_active":"2019-12-05T06:51:36.995882+0000","last_peered":"2019-12-05T06:51:36.995882+0000","last_clean":"2019-12-05T06:51:36.995882+0000","last_became_active":"2019-12-05T05:03:35.266235+0000","last_became_peered":"2019-12-05T05:03:35.266235+0000","last_unstale":"2019-12-05T06:51:36.995882+0000","last_undegraded":"2019-12-05T06:51:36.995882+0000","last_fullsized":"2019-12-05T06:51:36.995882+0000","mapping_epoch":21,"log_start":"118'3617","ondisk_log_start":"118'3617","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:34.235090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:34.235090+0000","last_clean_scrub_stamp":"2019-12-05T05:03:34.235090+0000","log_size":3070,"ondisk_log_size":3070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":15,"num_objects":3,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":240,"num_read_kb":18617,"num_write":4624,"num_write_kb":74599,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3,5],"acting":[0,4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"210","length":"1e"}]},{"pgid":"1.3","version":"0'0","reported_seq":"119","reported_epoch":"126","state":"active+clean","last_fresh":"2019-12-05T06:51:32.396898+0000","last_change":"2019-12-05T05:03:29.181115+0000","last_active":"2019-12-05T06:51:32.396898+0000","last_peered":"2019-12-05T06:51:32.396898+0000","last_clean":"2019-12-05T06:51:32.396898+0000","last_became_active":"2019-12-05T05:03:29.180832+0000","last_became_peered":"2019-12-05T05:03:29.180832+0000","last_unstale":"2019-12-05T06:51:32.396898+0000","last_undegraded":"2019-12-05T06:51:32.396898+0000","last_fullsized":"2019-12-05T06:51:32.396898+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:28.161818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:28.161818+0000","last_clean_scrub_stamp":"2019-12-05T05:03:28.161818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"127'2405","reported_seq":"3418","reported_epoch":"127","state":"active+clean","last_fresh":"2019-12-05T06:51:36.660638+0000","last_change":"2019-12-05T05:03:32.227259+0000","last_active":"2019-12-05T06:51:36.660638+0000","last_peered":"2019-12-05T06:51:36.660638+0000","last_clean":"2019-12-05T06:51:36.660638+0000","last_became_active":"2019-12-05T05:03:32.226895+0000","last_became_peered":"2019-12-05T05:03:32.226895+0000","last_unstale":"2019-12-05T06:51:36.660638+0000","last_undegraded":"2019-12-05T06:51:36.660638+0000","last_fullsized":"2019-12-05T06:51:36.660638+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:31.200243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:31.200243+0000","last_clean_scrub_stamp":"2019-12-05T05:03:31.200243+0000","log_size":2405,"ondisk_log_size":2405,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":50401708,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":1503,"num_read_kb":2307020,"num_write":2202,"num_write_kb":62922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":11,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"124'6690","reported_seq":"8471","reported_epoch":"127","state":"active+clean","last_fresh":"2019-12-05T06:51:36.681787+0000","last_change":"2019-12-05T06:51:36.689587+0000","last_active":"2019-12-05T06:51:36.681787+0000","last_peered":"2019-12-05T06:51:36.681787+0000","last_clean":"2019-12-05T06:51:36.681787+0000","last_became_active":"2019-12-05T05:03:35.627352+0000","last_became_peered":"2019-12-05T05:03:35.627352+0000","last_unstale":"2019-12-05T06:51:36.681787+0000","last_undegraded":"2019-12-05T06:51:36.681787+0000","last_fullsized":"2019-12-05T06:51:36.681787+0000","mapping_epoch":21,"log_start":"118'3612","ondisk_log_start":"118'3612","created":21,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:34.235090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:34.235090+0000","last_clean_scrub_stamp":"2019-12-05T05:03:34.235090+0000","log_size":3078,"ondisk_log_size":3078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":249,"num_read_kb":14990,"num_write":4595,"num_write_kb":60350,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6,5],"acting":[1,2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"210","length":"1e"}]},{"pgid":"1.2","version":"0'0","reported_seq":"119","reported_epoch":"126","state":"active+clean","last_fresh":"2019-12-05T06:51:31.448645+0000","last_change":"2019-12-05T05:03:29.408129+0000","last_active":"2019-12-05T06:51:31.448645+0000","last_peered":"2019-12-05T06:51:31.448645+0000","last_clean":"2019-12-05T06:51:31.448645+0000","last_became_active":"2019-12-05T05:03:29.407108+0000","last_became_peered":"2019-12-05T05:03:29.407108+0000","last_unstale":"2019-12-05T06:51:31.448645+0000","last_undegraded":"2019-12-05T06:51:31.448645+0000","last_fullsized":"2019-12-05T06:51:31.448645+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:28.161818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:28.161818+0000","last_clean_scrub_stamp":"2019-12-05T05:03:28.161818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"126'1574","reported_seq":"2607","reported_epoch":"126","state":"active+clean","last_fresh":"2019-12-05T06:51:36.450976+0000","last_change":"2019-12-05T05:03:32.636592+0000","last_active":"2019-12-05T06:51:36.450976+0000","last_peered":"2019-12-05T06:51:36.450976+0000","last_clean":"2019-12-05T06:51:36.450976+0000","last_became_active":"2019-12-05T05:03:32.635937+0000","last_became_peered":"2019-12-05T05:03:32.635937+0000","last_unstale":"2019-12-05T06:51:36.450976+0000","last_undegraded":"2019-12-05T06:51:36.450976+0000","last_fullsized":"2019-12-05T06:51:36.450976+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:31.200243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:31.200243+0000","last_clean_scrub_stamp":"2019-12-05T05:03:31.200243+0000","log_size":1574,"ondisk_log_size":1574,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":35827812,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":920,"num_read_kb":599871,"num_write":1407,"num_write_kb":35966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"126'6396","reported_seq":"8249","reported_epoch":"127","state":"active+clean","last_fresh":"2019-12-05T06:51:36.576145+0000","last_change":"2019-12-05T06:51:36.579050+0000","last_active":"2019-12-05T06:51:36.576145+0000","last_peered":"2019-12-05T06:51:36.576145+0000","last_clean":"2019-12-05T06:51:36.576145+0000","last_became_active":"2019-12-05T05:03:36.314006+0000","last_became_peered":"2019-12-05T05:03:36.314006+0000","last_unstale":"2019-12-05T06:51:36.576145+0000","last_undegraded":"2019-12-05T06:51:36.576145+0000","last_fullsized":"2019-12-05T06:51:36.576145+0000","mapping_epoch":21,"log_start":"117'3307","ondisk_log_start":"117'3307","created":21,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:34.235090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:34.235090+0000","last_clean_scrub_stamp":"2019-12-05T05:03:34.235090+0000","log_size":3089,"ondisk_log_size":3089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":253,"num_read_kb":6039,"num_write":4487,"num_write_kb":82964,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6,4],"acting":[3,5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"210","length":"1e"}]},{"pgid":"1.0","version":"0'0","reported_seq":"119","reported_epoch":"126","state":"active+clean","last_fresh":"2019-12-05T06:51:31.448567+0000","last_change":"2019-12-05T05:03:29.407882+0000","last_active":"2019-12-05T06:51:31.448567+0000","last_peered":"2019-12-05T06:51:31.448567+0000","last_clean":"2019-12-05T06:51:31.448567+0000","last_became_active":"2019-12-05T05:03:29.406770+0000","last_became_peered":"2019-12-05T05:03:29.406770+0000","last_unstale":"2019-12-05T06:51:31.448567+0000","last_undegraded":"2019-12-05T06:51:31.448567+0000","last_fullsized":"2019-12-05T06:51:31.448567+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:28.161818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:28.161818+0000","last_clean_scrub_stamp":"2019-12-05T05:03:28.161818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"126'1789","reported_seq":"2608","reported_epoch":"126","state":"active+clean","last_fresh":"2019-12-05T06:51:35.696904+0000","last_change":"2019-12-05T05:03:32.227654+0000","last_active":"2019-12-05T06:51:35.696904+0000","last_peered":"2019-12-05T06:51:35.696904+0000","last_clean":"2019-12-05T06:51:35.696904+0000","last_became_active":"2019-12-05T05:03:32.227052+0000","last_became_peered":"2019-12-05T05:03:32.227052+0000","last_unstale":"2019-12-05T06:51:35.696904+0000","last_undegraded":"2019-12-05T06:51:35.696904+0000","last_fullsized":"2019-12-05T06:51:35.696904+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:31.200243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:31.200243+0000","last_clean_scrub_stamp":"2019-12-05T05:03:31.200243+0000","log_size":1789,"ondisk_log_size":1789,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46137378,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":922,"num_read_kb":1904128,"num_write":1592,"num_write_kb":62385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"124'6539","reported_seq":"8355","reported_epoch":"127","state":"active+clean","last_fresh":"2019-12-05T06:51:36.875661+0000","last_change":"2019-12-05T06:51:36.877074+0000","last_active":"2019-12-05T06:51:36.875661+0000","last_peered":"2019-12-05T06:51:36.875661+0000","last_clean":"2019-12-05T06:51:36.875661+0000","last_became_active":"2019-12-05T05:03:35.732405+0000","last_became_peered":"2019-12-05T05:03:35.732405+0000","last_unstale":"2019-12-05T06:51:36.875661+0000","last_undegraded":"2019-12-05T06:51:36.875661+0000","last_fullsized":"2019-12-05T06:51:36.875661+0000","mapping_epoch":21,"log_start":"118'3508","ondisk_log_start":"118'3508","created":21,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:34.235090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:34.235090+0000","last_clean_scrub_stamp":"2019-12-05T05:03:34.235090+0000","log_size":3031,"ondisk_log_size":3031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":262,"num_read_kb":13412,"num_write":4550,"num_write_kb":85060,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0,6],"acting":[4,5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"210","length":"1e"}]},{"pgid":"1.1","version":"0'0","reported_seq":"119","reported_epoch":"126","state":"active+clean","last_fresh":"2019-12-05T06:51:31.448785+0000","last_change":"2019-12-05T05:03:29.407539+0000","last_active":"2019-12-05T06:51:31.448785+0000","last_peered":"2019-12-05T06:51:31.448785+0000","last_clean":"2019-12-05T06:51:31.448785+0000","last_became_active":"2019-12-05T05:03:29.406903+0000","last_became_peered":"2019-12-05T05:03:29.406903+0000","last_unstale":"2019-12-05T06:51:31.448785+0000","last_undegraded":"2019-12-05T06:51:31.448785+0000","last_fullsized":"2019-12-05T06:51:31.448785+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:28.161818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:28.161818+0000","last_clean_scrub_stamp":"2019-12-05T05:03:28.161818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"126'1333","reported_seq":"2009","reported_epoch":"126","state":"active+clean","last_fresh":"2019-12-05T06:51:35.698073+0000","last_change":"2019-12-05T05:03:32.227838+0000","last_active":"2019-12-05T06:51:35.698073+0000","last_peered":"2019-12-05T06:51:35.698073+0000","last_clean":"2019-12-05T06:51:35.698073+0000","last_became_active":"2019-12-05T05:03:32.227050+0000","last_became_peered":"2019-12-05T05:03:32.227050+0000","last_unstale":"2019-12-05T06:51:35.698073+0000","last_undegraded":"2019-12-05T06:51:35.698073+0000","last_fullsized":"2019-12-05T06:51:35.698073+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:31.200243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:31.200243+0000","last_clean_scrub_stamp":"2019-12-05T05:03:31.200243+0000","log_size":1333,"ondisk_log_size":1333,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":67108864,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":576,"num_read_kb":2216360,"num_write":1199,"num_write_kb":74709,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"124'6466","reported_seq":"8249","reported_epoch":"127","state":"active+clean","last_fresh":"2019-12-05T06:51:36.673591+0000","last_change":"2019-12-05T06:51:36.683171+0000","last_active":"2019-12-05T06:51:36.673591+0000","last_peered":"2019-12-05T06:51:36.673591+0000","last_clean":"2019-12-05T06:51:36.673591+0000","last_became_active":"2019-12-05T05:03:35.266868+0000","last_became_peered":"2019-12-05T05:03:35.266868+0000","last_unstale":"2019-12-05T06:51:36.673591+0000","last_undegraded":"2019-12-05T06:51:36.673591+0000","last_fullsized":"2019-12-05T06:51:36.673591+0000","mapping_epoch":21,"log_start":"118'3409","ondisk_log_start":"118'3409","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:34.235090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:34.235090+0000","last_clean_scrub_stamp":"2019-12-05T05:03:34.235090+0000","log_size":3057,"ondisk_log_size":3057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":226,"num_read_kb":10307,"num_write":4503,"num_write_kb":69746,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0,5],"acting":[1,2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"210","length":"1e"}]},{"pgid":"1.6","version":"0'0","reported_seq":"118","reported_epoch":"126","state":"active+clean","last_fresh":"2019-12-05T06:51:31.449349+0000","last_change":"2019-12-05T05:03:29.407541+0000","last_active":"2019-12-05T06:51:31.449349+0000","last_peered":"2019-12-05T06:51:31.449349+0000","last_clean":"2019-12-05T06:51:31.449349+0000","last_became_active":"2019-12-05T05:03:29.406857+0000","last_became_peered":"2019-12-05T05:03:29.406857+0000","last_unstale":"2019-12-05T06:51:31.449349+0000","last_undegraded":"2019-12-05T06:51:31.449349+0000","last_fullsized":"2019-12-05T06:51:31.449349+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:28.161818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:28.161818+0000","last_clean_scrub_stamp":"2019-12-05T05:03:28.161818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"126'2882","reported_seq":"3849","reported_epoch":"126","state":"active+clean","last_fresh":"2019-12-05T06:51:36.461394+0000","last_change":"2019-12-05T05:03:32.480464+0000","last_active":"2019-12-05T06:51:36.461394+0000","last_peered":"2019-12-05T06:51:36.461394+0000","last_clean":"2019-12-05T06:51:36.461394+0000","last_became_active":"2019-12-05T05:03:32.479565+0000","last_became_peered":"2019-12-05T05:03:32.479565+0000","last_unstale":"2019-12-05T06:51:36.461394+0000","last_undegraded":"2019-12-05T06:51:36.461394+0000","last_fullsized":"2019-12-05T06:51:36.461394+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:31.200243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:31.200243+0000","last_clean_scrub_stamp":"2019-12-05T05:03:31.200243+0000","log_size":2882,"ondisk_log_size":2882,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":83887077,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":1148,"num_read_kb":2218837,"num_write":2638,"num_write_kb":91697,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.5","version":"124'6269","reported_seq":"8145","reported_epoch":"127","state":"active+clean","last_fresh":"2019-12-05T06:51:36.915937+0000","last_change":"2019-12-05T06:51:36.920210+0000","last_active":"2019-12-05T06:51:36.915937+0000","last_peered":"2019-12-05T06:51:36.915937+0000","last_clean":"2019-12-05T06:51:36.915937+0000","last_became_active":"2019-12-05T05:03:35.629300+0000","last_became_peered":"2019-12-05T05:03:35.629300+0000","last_unstale":"2019-12-05T06:51:36.915937+0000","last_undegraded":"2019-12-05T06:51:36.915937+0000","last_fullsized":"2019-12-05T06:51:36.915937+0000","mapping_epoch":21,"log_start":"117'3209","ondisk_log_start":"117'3209","created":21,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:34.235090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:34.235090+0000","last_clean_scrub_stamp":"2019-12-05T05:03:34.235090+0000","log_size":3060,"ondisk_log_size":3060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":275,"num_read_kb":10198,"num_write":4396,"num_write_kb":76174,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2,6],"acting":[5,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"210","length":"1e"}]},{"pgid":"1.7","version":"0'0","reported_seq":"118","reported_epoch":"126","state":"active+clean","last_fresh":"2019-12-05T06:51:31.449101+0000","last_change":"2019-12-05T05:03:29.185245+0000","last_active":"2019-12-05T06:51:31.449101+0000","last_peered":"2019-12-05T06:51:31.449101+0000","last_clean":"2019-12-05T06:51:31.449101+0000","last_became_active":"2019-12-05T05:03:29.184716+0000","last_became_peered":"2019-12-05T05:03:29.184716+0000","last_unstale":"2019-12-05T06:51:31.449101+0000","last_undegraded":"2019-12-05T06:51:31.449101+0000","last_fullsized":"2019-12-05T06:51:31.449101+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:28.161818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:28.161818+0000","last_clean_scrub_stamp":"2019-12-05T05:03:28.161818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"124'1360","reported_seq":"2115","reported_epoch":"126","state":"active+clean","last_fresh":"2019-12-05T06:51:32.396232+0000","last_change":"2019-12-05T05:03:32.476846+0000","last_active":"2019-12-05T06:51:32.396232+0000","last_peered":"2019-12-05T06:51:32.396232+0000","last_clean":"2019-12-05T06:51:32.396232+0000","last_became_active":"2019-12-05T05:03:32.476140+0000","last_became_peered":"2019-12-05T05:03:32.476140+0000","last_unstale":"2019-12-05T06:51:32.396232+0000","last_undegraded":"2019-12-05T06:51:32.396232+0000","last_fullsized":"2019-12-05T06:51:32.396232+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:31.200243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:31.200243+0000","last_clean_scrub_stamp":"2019-12-05T05:03:31.200243+0000","log_size":1360,"ondisk_log_size":1360,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":37749370,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":931,"num_read_kb":1362588,"num_write":1192,"num_write_kb":41852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":8,"stat_sum":{"num_bytes":15,"num_objects":3,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1983,"num_read_kb":81571,"num_write":35920,"num_write_kb":550167,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":49152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24484,"ondisk_log_size":24484,"up":32,"acting":32,"num_store_stats":8},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":413387463,"num_objects":151,"num_object_clones":0,"num_object_copies":302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":151,"num_whiteouts":0,"num_read":7511,"num_read_kb":12724812,"num_write":14049,"num_write_kb":471369,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":826843136,"data_stored":826774926,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":15223,"ondisk_log_size":15223,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576160,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1206036,"kb_used_data":157452,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93165804,"statfs":{"total":96636764160,"available":95401783296,"internally_reserved":1073741824,"allocated":161230848,"data_stored":160484394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Dec 5 06:51:04 2019","interfaces":[{"interface":"back","average":{"1min":0.872,"5min":10.013,"15min":7.459},"min":{"1min":0.425,"5min":0.287,"15min":0.226},"max":{"1min":1.461,"5min":501.337,"15min":825.441},"last":0.617},{"interface":"front","average":{"1min":0.870,"5min":2.647,"15min":4.935},"min":{"1min":0.354,"5min":0.321,"15min":0.318},"max":{"1min":1.431,"5min":150.493,"15min":817.919},"last":0.838}]},{"osd":1,"last update":"Thu Dec 5 06:51:04 2019","interfaces":[{"interface":"back","average":{"1min":0.909,"5min":8.193,"15min":7.424},"min":{"1min":0.512,"5min":0.300,"15min":0.241},"max":{"1min":1.455,"5min":501.235,"15min":735.242},"last":1.030},{"interface":"front","average":{"1min":1.021,"5min":0.925,"15min":5.173},"min":{"1min":0.546,"5min":0.387,"15min":0.322},"max":{"1min":1.495,"5min":2.264,"15min":501.253},"last":0.726}]},{"osd":2,"last update":"Thu Dec 5 06:51:10 2019","interfaces":[{"interface":"back","average":{"1min":0.962,"5min":0.933,"15min":6.753},"min":{"1min":0.651,"5min":0.468,"15min":0.292},"max":{"1min":1.366,"5min":2.133,"15min":982.285},"last":1.126},{"interface":"front","average":{"1min":1.027,"5min":5.943,"15min":10.961},"min":{"1min":0.703,"5min":0.425,"15min":0.272},"max":{"1min":1.527,"5min":419.518,"15min":842.181},"last":1.168}]},{"osd":3,"last update":"Thu Dec 5 06:51:04 2019","interfaces":[{"interface":"back","average":{"1min":1.095,"5min":1.005,"15min":1.545},"min":{"1min":0.733,"5min":0.341,"15min":0.309},"max":{"1min":1.351,"5min":2.072,"15min":157.003},"last":1.242},{"interface":"front","average":{"1min":1.228,"5min":1.101,"15min":1.825},"min":{"1min":0.795,"5min":0.458,"15min":0.384},"max":{"1min":1.399,"5min":2.103,"15min":240.924},"last":1.301}]},{"osd":4,"last update":"Thu Dec 5 06:51:04 2019","interfaces":[{"interface":"back","average":{"1min":0.859,"5min":27.486,"15min":13.538},"min":{"1min":0.542,"5min":0.312,"15min":0.312},"max":{"1min":1.314,"5min":994.097,"15min":994.097},"last":1.337},{"interface":"front","average":{"1min":1.063,"5min":1.049,"15min":4.392},"min":{"1min":0.673,"5min":0.358,"15min":0.358},"max":{"1min":1.401,"5min":2.367,"15min":586.973},"last":0.918}]},{"osd":5,"last update":"Thu Dec 5 06:51:04 2019","interfaces":[{"interface":"back","average":{"1min":1.179,"5min":1.090,"15min":1.100},"min":{"1min":0.796,"5min":0.423,"15min":0.419},"max":{"1min":1.636,"5min":2.455,"15min":2.455},"last":1.201},{"interface":"front","average":{"1min":1.243,"5min":1.174,"15min":5.572},"min":{"1min":0.751,"5min":0.522,"15min":0.489},"max":{"1min":1.547,"5min":2.425,"15min":791.026},"last":1.388}]},{"osd":6,"last update":"Thu Dec 5 06:51:04 2019","interfaces":[{"interface":"back","average":{"1min":1.159,"5min":1.083,"15min":3.761},"min":{"1min":0.773,"5min":0.458,"15min":0.332},"max":{"1min":1.589,"5min":2.317,"15min":878.805},"last":1.272},{"interface":"front","average":{"1min":1.305,"5min":5.742,"15min":8.220},"min":{"1min":0.822,"5min":0.549,"15min":0.519},"max":{"1min":1.695,"5min":450.021,"15min":840.923},"last":1.428}]}]},{"osd":6,"up_from":13,"seq":55834576156,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1140420,"kb_used_data":91836,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93231420,"statfs":{"total":96636764160,"available":95468974080,"internally_reserved":1073741824,"allocated":94040064,"data_stored":93304473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Dec 5 06:51:03 2019","interfaces":[{"interface":"back","average":{"1min":24.785,"5min":5.553,"15min":9.441},"min":{"1min":0.571,"5min":0.254,"15min":0.254},"max":{"1min":501.477,"5min":501.477,"15min":986.070},"last":1.225},{"interface":"front","average":{"1min":56.903,"5min":18.312,"15min":12.733},"min":{"1min":0.762,"5min":0.335,"15min":0.298},"max":{"1min":639.525,"5min":639.525,"15min":872.211},"last":1.292}]},{"osd":1,"last update":"Thu Dec 5 06:51:00 2019","interfaces":[{"interface":"back","average":{"1min":0.952,"5min":9.793,"15min":3.799},"min":{"1min":0.690,"5min":0.357,"15min":0.277},"max":{"1min":1.364,"5min":758.181,"15min":758.181},"last":1.176},{"interface":"front","average":{"1min":41.988,"5min":9.041,"15min":7.092},"min":{"1min":0.678,"5min":0.356,"15min":0.322},"max":{"1min":501.424,"5min":501.424,"15min":501.424},"last":0.916}]},{"osd":2,"last update":"Thu Dec 5 06:51:00 2019","interfaces":[{"interface":"back","average":{"1min":0.934,"5min":0.914,"15min":8.858},"min":{"1min":0.623,"5min":0.392,"15min":0.354},"max":{"1min":1.352,"5min":1.352,"15min":775.503},"last":0.884},{"interface":"front","average":{"1min":1.082,"5min":1.012,"15min":1.002},"min":{"1min":0.531,"5min":0.433,"15min":0.325},"max":{"1min":1.530,"5min":1.530,"15min":1.784},"last":1.133}]},{"osd":3,"last update":"Thu Dec 5 06:51:00 2019","interfaces":[{"interface":"back","average":{"1min":0.982,"5min":11.337,"15min":7.358},"min":{"1min":0.481,"5min":0.418,"15min":0.408},"max":{"1min":1.426,"5min":777.425,"15min":777.425},"last":1.008},{"interface":"front","average":{"1min":1.127,"5min":1.124,"15min":1.282},"min":{"1min":0.738,"5min":0.551,"15min":0.501},"max":{"1min":1.547,"5min":1.568,"15min":65.098},"last":1.098}]},{"osd":4,"last update":"Thu Dec 5 06:51:00 2019","interfaces":[{"interface":"back","average":{"1min":1.115,"5min":6.122,"15min":10.663},"min":{"1min":0.743,"5min":0.426,"15min":0.420},"max":{"1min":1.731,"5min":380.358,"15min":929.519},"last":0.821},{"interface":"front","average":{"1min":41.074,"5min":8.911,"15min":5.622},"min":{"1min":0.576,"5min":0.374,"15min":0.346},"max":{"1min":843.286,"5min":843.286,"15min":843.286},"last":0.749}]},{"osd":5,"last update":"Thu Dec 5 06:51:00 2019","interfaces":[{"interface":"back","average":{"1min":1.119,"5min":1.059,"15min":4.110},"min":{"1min":0.473,"5min":0.360,"15min":0.360},"max":{"1min":1.536,"5min":1.536,"15min":879.085},"last":0.643},{"interface":"front","average":{"1min":1.248,"5min":9.004,"15min":3.765},"min":{"1min":0.807,"5min":0.518,"15min":0.484},"max":{"1min":1.530,"5min":666.419,"15min":666.419},"last":1.051}]},{"osd":7,"last update":"Thu Dec 5 06:51:00 2019","interfaces":[{"interface":"back","average":{"1min":6.301,"5min":3.582,"15min":4.398},"min":{"1min":0.998,"5min":0.509,"15min":0.408},"max":{"1min":111.548,"5min":127.801,"15min":629.008},"last":1.258},{"interface":"front","average":{"1min":1.368,"5min":1.222,"15min":5.762},"min":{"1min":1.032,"5min":0.363,"15min":0.363},"max":{"1min":1.754,"5min":1.754,"15min":633.862},"last":1.329}]}]},{"osd":1,"up_from":13,"seq":55834576154,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1267468,"kb_used_data":218884,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93104372,"statfs":{"total":96636764160,"available":95338876928,"internally_reserved":1073741824,"allocated":224137216,"data_stored":223398025,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Dec 5 06:51:02 2019","interfaces":[{"interface":"back","average":{"1min":0.727,"5min":0.680,"15min":2.496},"min":{"1min":0.442,"5min":0.212,"15min":0.197},"max":{"1min":1.373,"5min":1.373,"15min":381.600},"last":1.052},{"interface":"front","average":{"1min":0.782,"5min":3.100,"15min":6.834},"min":{"1min":0.450,"5min":0.310,"15min":0.226},"max":{"1min":1.417,"5min":266.748,"15min":705.387},"last":1.026}]},{"osd":2,"last update":"Thu Dec 5 06:51:02 2019","interfaces":[{"interface":"back","average":{"1min":0.869,"5min":0.731,"15min":4.849},"min":{"1min":0.354,"5min":0.251,"15min":0.251},"max":{"1min":1.342,"5min":1.833,"15min":806.146},"last":0.721},{"interface":"front","average":{"1min":0.697,"5min":7.330,"15min":8.813},"min":{"1min":0.285,"5min":0.245,"15min":0.245},"max":{"1min":1.041,"5min":660.160,"15min":769.148},"last":0.868}]},{"osd":3,"last update":"Thu Dec 5 06:51:02 2019","interfaces":[{"interface":"back","average":{"1min":0.928,"5min":9.283,"15min":3.696},"min":{"1min":0.474,"5min":0.338,"15min":0.332},"max":{"1min":1.468,"5min":884.578,"15min":884.578},"last":0.609},{"interface":"front","average":{"1min":1.021,"5min":0.934,"15min":7.565},"min":{"1min":0.486,"5min":0.312,"15min":0.312},"max":{"1min":1.512,"5min":1.949,"15min":854.098},"last":0.789}]},{"osd":4,"last update":"Thu Dec 5 06:51:14 2019","interfaces":[{"interface":"back","average":{"1min":23.321,"5min":5.825,"15min":11.220},"min":{"1min":0.366,"5min":0.366,"15min":0.366},"max":{"1min":584.815,"5min":584.815,"15min":584.815},"last":1.186},{"interface":"front","average":{"1min":0.955,"5min":7.867,"15min":9.542},"min":{"1min":0.275,"5min":0.275,"15min":0.275},"max":{"1min":1.295,"5min":686.649,"15min":778.089},"last":1.218}]},{"osd":5,"last update":"Thu Dec 5 06:51:02 2019","interfaces":[{"interface":"back","average":{"1min":1.077,"5min":1.001,"15min":4.159},"min":{"1min":0.633,"5min":0.482,"15min":0.403},"max":{"1min":1.362,"5min":1.769,"15min":839.045},"last":1.245},{"interface":"front","average":{"1min":1.102,"5min":11.793,"15min":6.461},"min":{"1min":0.569,"5min":0.375,"15min":0.375},"max":{"1min":1.610,"5min":598.889,"15min":598.889},"last":0.967}]},{"osd":6,"last update":"Thu Dec 5 06:51:02 2019","interfaces":[{"interface":"back","average":{"1min":33.387,"5min":7.490,"15min":3.777},"min":{"1min":0.818,"5min":0.365,"15min":0.360},"max":{"1min":807.184,"5min":807.184,"15min":807.184},"last":1.299},{"interface":"front","average":{"1min":1.134,"5min":1.112,"15min":14.638},"min":{"1min":0.684,"5min":0.410,"15min":0.410},"max":{"1min":1.532,"5min":1.623,"15min":848.805},"last":1.118}]},{"osd":7,"last update":"Thu Dec 5 06:51:02 2019","interfaces":[{"interface":"back","average":{"1min":1.268,"5min":3.514,"15min":4.600},"min":{"1min":0.737,"5min":0.409,"15min":0.409},"max":{"1min":1.783,"5min":274.305,"15min":680.492},"last":1.089},{"interface":"front","average":{"1min":31.226,"5min":20.849,"15min":9.395},"min":{"1min":0.611,"5min":0.523,"15min":0.518},"max":{"1min":501.595,"5min":792.863,"15min":792.863},"last":1.153}]}]},{"osd":0,"up_from":13,"seq":55834576156,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1169112,"kb_used_data":120528,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93202728,"statfs":{"total":96636764160,"available":95439593472,"internally_reserved":1073741824,"allocated":123420672,"data_stored":122677954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Dec 5 06:50:46 2019","interfaces":[{"interface":"back","average":{"1min":0.686,"5min":7.582,"15min":4.606},"min":{"1min":0.441,"5min":0.274,"15min":0.228},"max":{"1min":1.025,"5min":587.293,"15min":587.293},"last":0.648},{"interface":"front","average":{"1min":48.594,"5min":16.982,"15min":12.839},"min":{"1min":0.474,"5min":0.261,"15min":0.261},"max":{"1min":861.119,"5min":861.119,"15min":966.609},"last":0.692}]},{"osd":2,"last update":"Thu Dec 5 06:50:46 2019","interfaces":[{"interface":"back","average":{"1min":0.718,"5min":16.878,"15min":9.147},"min":{"1min":0.409,"5min":0.335,"15min":0.246},"max":{"1min":0.945,"5min":933.957,"15min":933.957},"last":1.153},{"interface":"front","average":{"1min":0.800,"5min":11.038,"15min":5.446},"min":{"1min":0.500,"5min":0.331,"15min":0.280},"max":{"1min":1.107,"5min":868.889,"15min":868.889},"last":1.071}]},{"osd":3,"last update":"Thu Dec 5 06:50:46 2019","interfaces":[{"interface":"back","average":{"1min":0.726,"5min":0.755,"15min":4.589},"min":{"1min":0.503,"5min":0.310,"15min":0.255},"max":{"1min":0.951,"5min":1.541,"15min":994.147},"last":0.551},{"interface":"front","average":{"1min":39.198,"5min":8.544,"15min":8.714},"min":{"1min":0.368,"5min":0.303,"15min":0.303},"max":{"1min":730.339,"5min":730.339,"15min":854.568},"last":0.513}]},{"osd":4,"last update":"Thu Dec 5 06:50:46 2019","interfaces":[{"interface":"back","average":{"1min":51.043,"5min":11.006,"15min":9.007},"min":{"1min":0.555,"5min":0.471,"15min":0.340},"max":{"1min":492.774,"5min":492.774,"15min":983.581},"last":1.239},{"interface":"front","average":{"1min":48.797,"5min":19.380,"15min":8.384},"min":{"1min":0.709,"5min":0.403,"15min":0.400},"max":{"1min":909.368,"5min":909.368,"15min":909.368},"last":0.843}]},{"osd":5,"last update":"Thu Dec 5 06:51:30 2019","interfaces":[{"interface":"back","average":{"1min":1.019,"5min":1.049,"15min":6.033},"min":{"1min":0.365,"5min":0.365,"15min":0.365},"max":{"1min":1.529,"5min":1.533,"15min":475.167},"last":0.937},{"interface":"front","average":{"1min":0.990,"5min":27.033,"15min":13.790},"min":{"1min":0.410,"5min":0.410,"15min":0.410},"max":{"1min":1.573,"5min":827.052,"15min":827.052},"last":1.118}]},{"osd":6,"last update":"Thu Dec 5 06:50:46 2019","interfaces":[{"interface":"back","average":{"1min":1.103,"5min":1.092,"15min":4.976},"min":{"1min":0.689,"5min":0.539,"15min":0.322},"max":{"1min":1.517,"5min":1.549,"15min":893.025},"last":1.399},{"interface":"front","average":{"1min":1.203,"5min":1.176,"15min":1.135},"min":{"1min":0.788,"5min":0.509,"15min":0.430},"max":{"1min":1.677,"5min":1.677,"15min":1.812},"last":1.325}]},{"osd":7,"last update":"Thu Dec 5 06:51:11 2019","interfaces":[{"interface":"back","average":{"1min":1.039,"5min":9.348,"15min":5.897},"min":{"1min":0.628,"5min":0.450,"15min":0.450},"max":{"1min":1.632,"5min":743.880,"15min":743.880},"last":0.900},{"interface":"front","average":{"1min":1.187,"5min":1.205,"15min":7.028},"min":{"1min":0.455,"5min":0.455,"15min":0.455},"max":{"1min":1.742,"5min":1.742,"15min":920.552},"last":1.025}]}]},{"osd":2,"up_from":13,"seq":55834576152,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1130352,"kb_used_data":81768,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93241488,"statfs":{"total":96636764160,"available":95479283712,"internally_reserved":1073741824,"allocated":83730432,"data_stored":82994409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Dec 5 06:50:45 2019","interfaces":[{"interface":"back","average":{"1min":0.651,"5min":18.625,"15min":6.641},"min":{"1min":0.466,"5min":0.238,"15min":0.235},"max":{"1min":0.924,"5min":917.307,"15min":917.307},"last":0.924},{"interface":"front","average":{"1min":0.742,"5min":6.302,"15min":2.553},"min":{"1min":0.555,"5min":0.235,"15min":0.213},"max":{"1min":0.947,"5min":423.686,"15min":423.686},"last":0.614}]},{"osd":1,"last update":"Thu Dec 5 06:50:42 2019","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.742,"15min":3.716},"min":{"1min":0.438,"5min":0.331,"15min":0.284},"max":{"1min":1.103,"5min":1.316,"15min":501.424},"last":0.550},{"interface":"front","average":{"1min":51.618,"5min":11.016,"15min":9.907},"min":{"1min":0.416,"5min":0.385,"15min":0.234},"max":{"1min":814.005,"5min":814.005,"15min":814.005},"last":0.882}]},{"osd":3,"last update":"Thu Dec 5 06:50:50 2019","interfaces":[{"interface":"back","average":{"1min":0.784,"5min":5.153,"15min":2.200},"min":{"1min":0.456,"5min":0.288,"15min":0.262},"max":{"1min":1.029,"5min":474.268,"15min":474.268},"last":0.723},{"interface":"front","average":{"1min":0.812,"5min":0.840,"15min":1.716},"min":{"1min":0.528,"5min":0.334,"15min":0.303},"max":{"1min":1.126,"5min":1.291,"15min":245.318},"last":0.825}]},{"osd":4,"last update":"Thu Dec 5 06:50:42 2019","interfaces":[{"interface":"back","average":{"1min":1.045,"5min":8.455,"15min":3.953},"min":{"1min":0.615,"5min":0.332,"15min":0.323},"max":{"1min":1.361,"5min":632.509,"15min":632.509},"last":1.187},{"interface":"front","average":{"1min":1.099,"5min":1.111,"15min":4.268},"min":{"1min":0.757,"5min":0.610,"15min":0.365},"max":{"1min":1.333,"5min":2.178,"15min":991.418},"last":1.303}]},{"osd":5,"last update":"Thu Dec 5 06:50:42 2019","interfaces":[{"interface":"back","average":{"1min":1.015,"5min":9.320,"15min":5.649},"min":{"1min":0.649,"5min":0.316,"15min":0.316},"max":{"1min":1.501,"5min":965.993,"15min":965.993},"last":1.451},{"interface":"front","average":{"1min":0.961,"5min":11.954,"15min":6.908},"min":{"1min":0.743,"5min":0.381,"15min":0.381},"max":{"1min":1.201,"5min":700.890,"15min":760.855},"last":1.548}]},{"osd":6,"last update":"Thu Dec 5 06:50:45 2019","interfaces":[{"interface":"back","average":{"1min":1.180,"5min":1.142,"15min":6.080},"min":{"1min":0.791,"5min":0.415,"15min":0.344},"max":{"1min":1.416,"5min":2.126,"15min":652.287},"last":1.499},{"interface":"front","average":{"1min":63.325,"5min":15.337,"15min":10.066},"min":{"1min":0.983,"5min":0.444,"15min":0.420},"max":{"1min":934.081,"5min":934.081,"15min":934.081},"last":1.102}]},{"osd":7,"last update":"Thu Dec 5 06:50:42 2019","interfaces":[{"interface":"back","average":{"1min":1.197,"5min":5.507,"15min":2.611},"min":{"1min":0.807,"5min":0.440,"15min":0.440},"max":{"1min":1.380,"5min":419.900,"15min":419.900},"last":1.402},{"interface":"front","average":{"1min":1.132,"5min":1.104,"15min":1.084},"min":{"1min":0.896,"5min":0.408,"15min":0.408},"max":{"1min":1.456,"5min":1.620,"15min":1.794},"last":1.256}]}]},{"osd":3,"up_from":13,"seq":55834576155,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1085304,"kb_used_data":36720,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93286536,"statfs":{"total":96636764160,"available":95525412864,"internally_reserved":1073741824,"allocated":37601280,"data_stored":36869319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Dec 5 06:51:36 2019","interfaces":[{"interface":"back","average":{"1min":0.662,"5min":18.296,"15min":13.058},"min":{"1min":0.296,"5min":0.197,"15min":0.196},"max":{"1min":1.334,"5min":802.154,"15min":802.154},"last":0.813},{"interface":"front","average":{"1min":0.750,"5min":7.929,"15min":9.078},"min":{"1min":0.319,"5min":0.259,"15min":0.242},"max":{"1min":1.454,"5min":828.266,"15min":991.916},"last":0.920}]},{"osd":1,"last update":"Thu Dec 5 06:50:44 2019","interfaces":[{"interface":"back","average":{"1min":2.653,"5min":8.851,"15min":17.518},"min":{"1min":0.420,"5min":0.264,"15min":0.211},"max":{"1min":44.449,"5min":699.405,"15min":957.811},"last":0.644},{"interface":"front","average":{"1min":0.775,"5min":0.837,"15min":13.986},"min":{"1min":0.480,"5min":0.311,"15min":0.282},"max":{"1min":0.999,"5min":1.396,"15min":730.364},"last":0.858}]},{"osd":2,"last update":"Thu Dec 5 06:50:59 2019","interfaces":[{"interface":"back","average":{"1min":0.830,"5min":0.878,"15min":4.117},"min":{"1min":0.524,"5min":0.266,"15min":0.266},"max":{"1min":1.512,"5min":1.512,"15min":444.359},"last":0.576},{"interface":"front","average":{"1min":0.932,"5min":0.867,"15min":9.156},"min":{"1min":0.431,"5min":0.340,"15min":0.297},"max":{"1min":1.586,"5min":1.586,"15min":814.674},"last":0.742}]},{"osd":4,"last update":"Thu Dec 5 06:51:36 2019","interfaces":[{"interface":"back","average":{"1min":1.004,"5min":0.989,"15min":1.926},"min":{"1min":0.618,"5min":0.402,"15min":0.325},"max":{"1min":2.287,"5min":2.287,"15min":216.348},"last":1.213},{"interface":"front","average":{"1min":1.072,"5min":1.012,"15min":1.582},"min":{"1min":0.659,"5min":0.335,"15min":0.335},"max":{"1min":1.392,"5min":1.669,"15min":192.285},"last":1.155}]},{"osd":5,"last update":"Thu Dec 5 06:51:36 2019","interfaces":[{"interface":"back","average":{"1min":1.133,"5min":8.721,"15min":4.576},"min":{"1min":0.690,"5min":0.372,"15min":0.328},"max":{"1min":2.345,"5min":880.020,"15min":880.020},"last":0.960},{"interface":"front","average":{"1min":0.984,"5min":1.075,"15min":1.045},"min":{"1min":0.414,"5min":0.414,"15min":0.364},"max":{"1min":1.327,"5min":1.532,"15min":1.936},"last":0.701}]},{"osd":6,"last update":"Thu Dec 5 06:51:35 2019","interfaces":[{"interface":"back","average":{"1min":51.161,"5min":11.110,"15min":6.234},"min":{"1min":0.557,"5min":0.430,"15min":0.413},"max":{"1min":1001.571,"5min":1001.571,"15min":1001.571},"last":1.035},{"interface":"front","average":{"1min":1.016,"5min":8.110,"15min":3.832},"min":{"1min":0.265,"5min":0.265,"15min":0.265},"max":{"1min":2.065,"5min":558.711,"15min":558.711},"last":0.787}]},{"osd":7,"last update":"Thu Dec 5 06:50:50 2019","interfaces":[{"interface":"back","average":{"1min":1.067,"5min":2.388,"15min":10.379},"min":{"1min":0.657,"5min":0.523,"15min":0.365},"max":{"1min":1.465,"5min":97.130,"15min":877.034},"last":1.094},{"interface":"front","average":{"1min":1.134,"5min":1.213,"15min":1.166},"min":{"1min":0.695,"5min":0.499,"15min":0.435},"max":{"1min":1.510,"5min":1.784,"15min":1.979},"last":1.329}]}]},{"osd":4,"up_from":13,"seq":55834576153,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1050312,"kb_used_data":1728,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93321528,"statfs":{"total":96636764160,"available":95561244672,"internally_reserved":1073741824,"allocated":1769472,"data_stored":1041507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Dec 5 06:51:05 2019","interfaces":[{"interface":"back","average":{"1min":0.881,"5min":7.022,"15min":5.481},"min":{"1min":0.542,"5min":0.283,"15min":0.264},"max":{"1min":1.366,"5min":439.064,"15min":439.064},"last":1.251},{"interface":"front","average":{"1min":31.716,"5min":12.524,"15min":5.380},"min":{"1min":0.662,"5min":0.225,"15min":0.225},"max":{"1min":524.030,"5min":524.030,"15min":524.030},"last":0.680}]},{"osd":1,"last update":"Thu Dec 5 06:51:05 2019","interfaces":[{"interface":"back","average":{"1min":0.962,"5min":1.204,"15min":4.257},"min":{"1min":0.551,"5min":0.303,"15min":0.303},"max":{"1min":1.309,"5min":31.305,"15min":501.402},"last":0.996},{"interface":"front","average":{"1min":0.982,"5min":0.929,"15min":4.757},"min":{"1min":0.648,"5min":0.304,"15min":0.304},"max":{"1min":1.576,"5min":1.576,"15min":501.754},"last":1.165}]},{"osd":2,"last update":"Thu Dec 5 06:51:14 2019","interfaces":[{"interface":"back","average":{"1min":43.387,"5min":31.525,"15min":12.886},"min":{"1min":0.397,"5min":0.359,"15min":0.268},"max":{"1min":850.354,"5min":897.315,"15min":897.315},"last":0.619},{"interface":"front","average":{"1min":1.010,"5min":0.982,"15min":3.965},"min":{"1min":0.287,"5min":0.287,"15min":0.287},"max":{"1min":1.397,"5min":1.397,"15min":632.767},"last":0.765}]},{"osd":3,"last update":"Thu Dec 5 06:51:05 2019","interfaces":[{"interface":"back","average":{"1min":0.863,"5min":6.934,"15min":3.446},"min":{"1min":0.694,"5min":0.340,"15min":0.340},"max":{"1min":1.045,"5min":501.658,"15min":501.658},"last":0.922},{"interface":"front","average":{"1min":9.928,"5min":2.858,"15min":2.715},"min":{"1min":0.934,"5min":0.432,"15min":0.348},"max":{"1min":149.969,"5min":149.969,"15min":308.851},"last":1.093}]},{"osd":5,"last update":"Thu Dec 5 06:51:05 2019","interfaces":[{"interface":"back","average":{"1min":1.098,"5min":0.906,"15min":3.992},"min":{"1min":0.704,"5min":0.325,"15min":0.325},"max":{"1min":1.682,"5min":1.682,"15min":840.089},"last":0.882},{"interface":"front","average":{"1min":1.224,"5min":16.912,"15min":9.261},"min":{"1min":0.918,"5min":0.385,"15min":0.369},"max":{"1min":1.729,"5min":660.266,"15min":748.692},"last":1.212}]},{"osd":6,"last update":"Thu Dec 5 06:51:05 2019","interfaces":[{"interface":"back","average":{"1min":0.900,"5min":18.011,"15min":10.852},"min":{"1min":0.583,"5min":0.354,"15min":0.314},"max":{"1min":1.139,"5min":817.106,"15min":861.297},"last":0.839},{"interface":"front","average":{"1min":1.258,"5min":1.114,"15min":3.380},"min":{"1min":0.708,"5min":0.390,"15min":0.374},"max":{"1min":1.808,"5min":3.181,"15min":690.644},"last":1.050}]},{"osd":7,"last update":"Thu Dec 5 06:51:21 2019","interfaces":[{"interface":"back","average":{"1min":1.120,"5min":1.135,"15min":6.627},"min":{"1min":0.481,"5min":0.425,"15min":0.351},"max":{"1min":1.534,"5min":2.624,"15min":507.089},"last":1.289},{"interface":"front","average":{"1min":1.194,"5min":1.191,"15min":3.242},"min":{"1min":0.520,"5min":0.464,"15min":0.464},"max":{"1min":1.635,"5min":1.670,"15min":452.400},"last":1.372}]}]},{"osd":5,"up_from":13,"seq":55834576158,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1160908,"kb_used_data":112324,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93210932,"statfs":{"total":96636764160,"available":95447994368,"internally_reserved":1073741824,"allocated":115019776,"data_stored":114287749,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Dec 5 06:50:48 2019","interfaces":[{"interface":"back","average":{"1min":18.174,"5min":4.336,"15min":1.990},"min":{"1min":0.333,"5min":0.333,"15min":0.235},"max":{"1min":331.975,"5min":331.975,"15min":331.975},"last":1.170},{"interface":"front","average":{"1min":0.786,"5min":0.855,"15min":3.602},"min":{"1min":0.471,"5min":0.351,"15min":0.278},"max":{"1min":1.137,"5min":2.878,"15min":676.393},"last":0.640}]},{"osd":1,"last update":"Thu Dec 5 06:50:58 2019","interfaces":[{"interface":"back","average":{"1min":0.854,"5min":0.773,"15min":2.216},"min":{"1min":0.544,"5min":0.308,"15min":0.225},"max":{"1min":1.145,"5min":1.274,"15min":251.684},"last":1.044},{"interface":"front","average":{"1min":38.990,"5min":8.478,"15min":5.469},"min":{"1min":0.721,"5min":0.388,"15min":0.245},"max":{"1min":722.849,"5min":722.849,"15min":722.849},"last":1.213}]},{"osd":2,"last update":"Thu Dec 5 06:50:45 2019","interfaces":[{"interface":"back","average":{"1min":0.850,"5min":17.271,"15min":8.269},"min":{"1min":0.612,"5min":0.457,"15min":0.281},"max":{"1min":1.198,"5min":934.880,"15min":934.880},"last":0.936},{"interface":"front","average":{"1min":0.978,"5min":9.447,"15min":10.949},"min":{"1min":0.696,"5min":0.488,"15min":0.409},"max":{"1min":1.315,"5min":676.797,"15min":794.102},"last":1.085}]},{"osd":3,"last update":"Thu Dec 5 06:51:24 2019","interfaces":[{"interface":"back","average":{"1min":1.037,"5min":1.093,"15min":3.676},"min":{"1min":0.468,"5min":0.468,"15min":0.340},"max":{"1min":1.318,"5min":2.942,"15min":683.842},"last":0.990},{"interface":"front","average":{"1min":42.174,"5min":19.109,"15min":8.696},"min":{"1min":0.500,"5min":0.500,"15min":0.387},"max":{"1min":740.171,"5min":740.171,"15min":740.171},"last":1.257}]},{"osd":4,"last update":"Thu Dec 5 06:50:48 2019","interfaces":[{"interface":"back","average":{"1min":1.096,"5min":1.087,"15min":7.581},"min":{"1min":0.730,"5min":0.441,"15min":0.327},"max":{"1min":1.417,"5min":2.914,"15min":633.428},"last":1.302},{"interface":"front","average":{"1min":1.072,"5min":0.956,"15min":5.140},"min":{"1min":0.593,"5min":0.330,"15min":0.309},"max":{"1min":1.344,"5min":1.409,"15min":792.823},"last":0.821}]},{"osd":6,"last update":"Thu Dec 5 06:50:48 2019","interfaces":[{"interface":"back","average":{"1min":1.157,"5min":1.008,"15min":6.845},"min":{"1min":0.902,"5min":0.370,"15min":0.318},"max":{"1min":1.470,"5min":1.637,"15min":891.609},"last":1.337},{"interface":"front","average":{"1min":34.677,"5min":19.775,"15min":10.319},"min":{"1min":1.115,"5min":0.409,"15min":0.409},"max":{"1min":635.191,"5min":650.798,"15min":916.939},"last":1.382}]},{"osd":7,"last update":"Thu Dec 5 06:50:48 2019","interfaces":[{"interface":"back","average":{"1min":1.314,"5min":8.254,"15min":6.479},"min":{"1min":1.049,"5min":0.586,"15min":0.401},"max":{"1min":1.511,"5min":735.632,"15min":735.632},"last":1.416},{"interface":"front","average":{"1min":1.396,"5min":8.665,"15min":6.319},"min":{"1min":1.028,"5min":0.677,"15min":0.380},"max":{"1min":1.646,"5min":405.552,"15min":405.552},"last":1.455}]}]}]}} 2019-12-05T06:51:44.568 INFO:tasks.ceph:pgid 3.7 last_scrub_stamp 2019-12-05T05:03:34.235090+0000 time.struct_time(tm_year=2019, tm_mon=12, tm_mday=5, tm_hour=5, tm_min=3, tm_sec=34, tm_wday=3, tm_yday=339, tm_isdst=-1) <= time.struct_time(tm_year=2019, tm_mon=12, tm_mday=5, tm_hour=6, tm_min=51, tm_sec=39, tm_wday=3, tm_yday=339, tm_isdst=0) 2019-12-05T06:51:44.568 INFO:tasks.ceph:pgid 1.5 last_scrub_stamp 2019-12-05T05:03:28.161818+0000 time.struct_time(tm_year=2019, tm_mon=12, tm_mday=5, tm_hour=5, tm_min=3, tm_sec=28, tm_wday=3, tm_yday=339, tm_isdst=-1) <= time.struct_time(tm_year=2019, tm_mon=12, tm_mday=5, tm_hour=6, tm_min=51, tm_sec=39, tm_wday=3, tm_yday=339, tm_isdst=0) 2019-12-05T06:51:44.568 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2019-12-05T05:03:31.200243+0000 time.struct_time(tm_year=2019, tm_mon=12, tm_mday=5, tm_hour=5, tm_min=3, tm_sec=31, tm_wday=3, tm_yday=339, tm_isdst=-1) <= time.struct_time(tm_year=2019, tm_mon=12, tm_mday=5, tm_hour=6, tm_min=51, tm_sec=39, tm_wday=3, tm_yday=339, tm_isdst=0) 2019-12-05T06:51:44.569 INFO:tasks.ceph:pgid 3.6 last_scrub_stamp 2019-12-05T05:03:34.235090+0000 time.struct_time(tm_year=2019, tm_mon=12, tm_mday=5, tm_hour=5, tm_min=3, tm_sec=34, tm_wday=3, tm_yday=339, tm_isdst=-1) <= time.struct_time(tm_year=2019, tm_mon=12, tm_mday=5, tm_hour=6, tm_min=51, tm_sec=39, tm_wday=3, tm_yday=339, tm_isdst=0) 2019-12-05T06:51:44.569 INFO:tasks.ceph:pgid 1.4 last_scrub_stamp 2019-12-05T05:03:28.161818+0000 time.struct_time(tm_year=2019, tm_mon=12, tm_mday=5, tm_hour=5, tm_min=3, tm_sec=28, tm_wday=3, tm_yday=339, tm_isdst=-1) <= time.struct_time(tm_year=2019, tm_mon=12, tm_mday=5, tm_hour=6, tm_min=51, tm_sec=39, tm_wday=3, tm_yday=339, tm_isdst=0) 2019-12-05T06:51:44.569 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2019-12-05T05:03:31.200243+0000 time.struct_time(tm_year=2019, tm_mon=12, tm_mday=5, tm_hour=5, tm_min=3, tm_sec=31, tm_wday=3, tm_yday=339, tm_isdst=-1) <= time.struct_time(tm_year=2019, tm_mon=12, tm_mday=5, tm_hour=6, tm_min=51, tm_sec=39, tm_wday=3, tm_yday=339, tm_isdst=0) 2019-12-05T06:51:44.570 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2019-12-05T05:03:34.235090+0000 time.struct_time(tm_year=2019, tm_mon=12, tm_mday=5, tm_hour=5, tm_min=3, tm_sec=34, tm_wday=3, tm_yday=339, tm_isdst=-1) <= time.struct_time(tm_year=2019, tm_mon=12, tm_mday=5, tm_hour=6, tm_min=51, tm_sec=39, tm_wday=3, tm_yday=339, tm_isdst=0) 2019-12-05T06:51:44.570 INFO:tasks.ceph:pgid 1.3 last_scrub_stamp 2019-12-05T05:03:28.161818+0000 time.struct_time(tm_year=2019, tm_mon=12, tm_mday=5, tm_hour=5, tm_min=3, tm_sec=28, tm_wday=3, tm_yday=339, tm_isdst=-1) <= time.struct_time(tm_year=2019, tm_mon=12, tm_mday=5, tm_hour=6, tm_min=51, tm_sec=39, tm_wday=3, tm_yday=339, tm_isdst=0) 2019-12-05T06:51:44.570 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2019-12-05T05:03:31.200243+0000 time.struct_time(tm_year=2019, tm_mon=12, tm_mday=5, tm_hour=5, tm_min=3, tm_sec=31, tm_wday=3, tm_yday=339, tm_isdst=-1) <= time.struct_time(tm_year=2019, tm_mon=12, tm_mday=5, tm_hour=6, tm_min=51, tm_sec=39, tm_wday=3, tm_yday=339, tm_isdst=0) 2019-12-05T06:51:44.571 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2019-12-05T05:03:34.235090+0000 time.struct_time(tm_year=2019, tm_mon=12, tm_mday=5, tm_hour=5, tm_min=3, tm_sec=34, tm_wday=3, tm_yday=339, tm_isdst=-1) <= time.struct_time(tm_year=2019, tm_mon=12, tm_mday=5, tm_hour=6, tm_min=51, tm_sec=39, tm_wday=3, tm_yday=339, tm_isdst=0) 2019-12-05T06:51:44.571 INFO:tasks.ceph:pgid 1.2 last_scrub_stamp 2019-12-05T05:03:28.161818+0000 time.struct_time(tm_year=2019, tm_mon=12, tm_mday=5, tm_hour=5, tm_min=3, tm_sec=28, tm_wday=3, tm_yday=339, tm_isdst=-1) <= time.struct_time(tm_year=2019, tm_mon=12, tm_mday=5, tm_hour=6, tm_min=51, tm_sec=39, tm_wday=3, tm_yday=339, tm_isdst=0) 2019-12-05T06:51:44.571 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2019-12-05T05:03:31.200243+0000 time.struct_time(tm_year=2019, tm_mon=12, tm_mday=5, tm_hour=5, tm_min=3, tm_sec=31, tm_wday=3, tm_yday=339, tm_isdst=-1) <= time.struct_time(tm_year=2019, tm_mon=12, tm_mday=5, tm_hour=6, tm_min=51, tm_sec=39, tm_wday=3, tm_yday=339, tm_isdst=0) 2019-12-05T06:51:44.571 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2019-12-05T05:03:34.235090+0000 time.struct_time(tm_year=2019, tm_mon=12, tm_mday=5, tm_hour=5, tm_min=3, tm_sec=34, tm_wday=3, tm_yday=339, tm_isdst=-1) <= time.struct_time(tm_year=2019, tm_mon=12, tm_mday=5, tm_hour=6, tm_min=51, tm_sec=39, tm_wday=3, tm_yday=339, tm_isdst=0) 2019-12-05T06:51:44.572 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2019-12-05T05:03:28.161818+0000 time.struct_time(tm_year=2019, tm_mon=12, tm_mday=5, tm_hour=5, tm_min=3, tm_sec=28, tm_wday=3, tm_yday=339, tm_isdst=-1) <= time.struct_time(tm_year=2019, tm_mon=12, tm_mday=5, tm_hour=6, tm_min=51, tm_sec=39, tm_wday=3, tm_yday=339, tm_isdst=0) 2019-12-05T06:51:44.572 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2019-12-05T05:03:31.200243+0000 time.struct_time(tm_year=2019, tm_mon=12, tm_mday=5, tm_hour=5, tm_min=3, tm_sec=31, tm_wday=3, tm_yday=339, tm_isdst=-1) <= time.struct_time(tm_year=2019, tm_mon=12, tm_mday=5, tm_hour=6, tm_min=51, tm_sec=39, tm_wday=3, tm_yday=339, tm_isdst=0) 2019-12-05T06:51:44.572 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2019-12-05T05:03:34.235090+0000 time.struct_time(tm_year=2019, tm_mon=12, tm_mday=5, tm_hour=5, tm_min=3, tm_sec=34, tm_wday=3, tm_yday=339, tm_isdst=-1) <= time.struct_time(tm_year=2019, tm_mon=12, tm_mday=5, tm_hour=6, tm_min=51, tm_sec=39, tm_wday=3, tm_yday=339, tm_isdst=0) 2019-12-05T06:51:44.573 INFO:tasks.ceph:pgid 1.1 last_scrub_stamp 2019-12-05T05:03:28.161818+0000 time.struct_time(tm_year=2019, tm_mon=12, tm_mday=5, tm_hour=5, tm_min=3, tm_sec=28, tm_wday=3, tm_yday=339, tm_isdst=-1) <= time.struct_time(tm_year=2019, tm_mon=12, tm_mday=5, tm_hour=6, tm_min=51, tm_sec=39, tm_wday=3, tm_yday=339, tm_isdst=0) 2019-12-05T06:51:44.573 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2019-12-05T05:03:31.200243+0000 time.struct_time(tm_year=2019, tm_mon=12, tm_mday=5, tm_hour=5, tm_min=3, tm_sec=31, tm_wday=3, tm_yday=339, tm_isdst=-1) <= time.struct_time(tm_year=2019, tm_mon=12, tm_mday=5, tm_hour=6, tm_min=51, tm_sec=39, tm_wday=3, tm_yday=339, tm_isdst=0) 2019-12-05T06:51:44.573 INFO:tasks.ceph:pgid 3.4 last_scrub_stamp 2019-12-05T05:03:34.235090+0000 time.struct_time(tm_year=2019, tm_mon=12, tm_mday=5, tm_hour=5, tm_min=3, tm_sec=34, tm_wday=3, tm_yday=339, tm_isdst=-1) <= time.struct_time(tm_year=2019, tm_mon=12, tm_mday=5, tm_hour=6, tm_min=51, tm_sec=39, tm_wday=3, tm_yday=339, tm_isdst=0) 2019-12-05T06:51:44.573 INFO:tasks.ceph:pgid 1.6 last_scrub_stamp 2019-12-05T05:03:28.161818+0000 time.struct_time(tm_year=2019, tm_mon=12, tm_mday=5, tm_hour=5, tm_min=3, tm_sec=28, tm_wday=3, tm_yday=339, tm_isdst=-1) <= time.struct_time(tm_year=2019, tm_mon=12, tm_mday=5, tm_hour=6, tm_min=51, tm_sec=39, tm_wday=3, tm_yday=339, tm_isdst=0) 2019-12-05T06:51:44.574 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2019-12-05T05:03:31.200243+0000 time.struct_time(tm_year=2019, tm_mon=12, tm_mday=5, tm_hour=5, tm_min=3, tm_sec=31, tm_wday=3, tm_yday=339, tm_isdst=-1) <= time.struct_time(tm_year=2019, tm_mon=12, tm_mday=5, tm_hour=6, tm_min=51, tm_sec=39, tm_wday=3, tm_yday=339, tm_isdst=0) 2019-12-05T06:51:44.574 INFO:tasks.ceph:pgid 3.5 last_scrub_stamp 2019-12-05T05:03:34.235090+0000 time.struct_time(tm_year=2019, tm_mon=12, tm_mday=5, tm_hour=5, tm_min=3, tm_sec=34, tm_wday=3, tm_yday=339, tm_isdst=-1) <= time.struct_time(tm_year=2019, tm_mon=12, tm_mday=5, tm_hour=6, tm_min=51, tm_sec=39, tm_wday=3, tm_yday=339, tm_isdst=0) 2019-12-05T06:51:44.574 INFO:tasks.ceph:pgid 1.7 last_scrub_stamp 2019-12-05T05:03:28.161818+0000 time.struct_time(tm_year=2019, tm_mon=12, tm_mday=5, tm_hour=5, tm_min=3, tm_sec=28, tm_wday=3, tm_yday=339, tm_isdst=-1) <= time.struct_time(tm_year=2019, tm_mon=12, tm_mday=5, tm_hour=6, tm_min=51, tm_sec=39, tm_wday=3, tm_yday=339, tm_isdst=0) 2019-12-05T06:51:44.575 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2019-12-05T05:03:31.200243+0000 time.struct_time(tm_year=2019, tm_mon=12, tm_mday=5, tm_hour=5, tm_min=3, tm_sec=31, tm_wday=3, tm_yday=339, tm_isdst=-1) <= time.struct_time(tm_year=2019, tm_mon=12, tm_mday=5, tm_hour=6, tm_min=51, tm_sec=39, tm_wday=3, tm_yday=339, tm_isdst=0) 2019-12-05T06:51:44.575 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2019-12-05T06:51:51.034 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:51:51.037 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:52:04.583 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2019-12-05T06:52:04.691 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:04.682+0000 7f374b000700 1 Processor -- start 2019-12-05T06:52:04.691 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:04.682+0000 7f374b000700 1 -- start start 2019-12-05T06:52:04.692 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:04.686+0000 7f374b000700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f374408df70 0x7f374409f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:52:04.692 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:04.686+0000 7f374b000700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f374409f820 0x7f374414efd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:52:04.692 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:04.686+0000 7f374b000700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f374409fc50 con 0x7f37440a7730 2019-12-05T06:52:04.692 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:04.686+0000 7f374b000700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37440a0af0 con 0x7f374409f820 2019-12-05T06:52:04.692 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:04.686+0000 7f374b000700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f374408e5a0 con 0x7f374408df70 2019-12-05T06:52:04.692 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:04.686+0000 7f3748d9c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f374408df70 0x7f374409f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:52:04.692 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:04.686+0000 7f3748d9c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f374408df70 0x7f374409f2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:39526/0 (socket says 172.21.15.42:39526) 2019-12-05T06:52:04.693 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:04.686+0000 7f3748d9c700 1 -- 172.21.15.42:0/3826016496 learned_addr learned my addr 172.21.15.42:0/3826016496 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:52:04.693 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:04.686+0000 7f3749d9e700 1 --2- 172.21.15.42:0/3826016496 >> v2:172.21.15.40:3300/0 conn(0x7f374409f820 0x7f374414efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:52:04.693 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:04.686+0000 7f3748d9c700 1 -- 172.21.15.42:0/3826016496 >> v2:172.21.15.40:3300/0 conn(0x7f374409f820 msgr2=0x7f374414efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:52:04.693 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:04.686+0000 7f3748d9c700 1 --2- 172.21.15.42:0/3826016496 >> v2:172.21.15.40:3300/0 conn(0x7f374409f820 0x7f374414efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:52:04.693 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:04.686+0000 7f3748d9c700 1 -- 172.21.15.42:0/3826016496 >> v1:172.21.15.40:6789/0 conn(0x7f37440a7730 legacy=0x7f374407e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:52:04.693 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:04.686+0000 7f3748d9c700 1 -- 172.21.15.42:0/3826016496 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37440a0700 con 0x7f374408df70 2019-12-05T06:52:04.693 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:04.686+0000 7f3748d9c700 1 --2- 172.21.15.42:0/3826016496 >> v2:172.21.15.42:3300/0 conn(0x7f374408df70 0x7f374409f2d0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f3738002f30 tx=0x7f373800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:04.694 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:04.686+0000 7f3748d9c700 1 --2- 172.21.15.42:0/3826016496 >> v2:172.21.15.42:3300/0 conn(0x7f374408df70 0x7f374409f2d0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f3738002f30 tx=0x7f373800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:04.694 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:04.686+0000 7f3748d9c700 1 --2- 172.21.15.42:0/3826016496 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f374408df70 0x7f374409f2d0 secure :-1 s=READY pgs=3400 cs=0 l=1 rx=0x7f3738002f30 tx=0x7f373800fd30).ready entity=mon.1 client_cookie=42b8f04fec6f7084 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:52:04.694 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:04.686+0000 7f3748d9c700 1 --2- 172.21.15.42:0/3826016496 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f374408df70 0x7f374409f2d0 secure :-1 s=THROTTLE_DONE pgs=3400 cs=0 l=1 rx=0x7f3738002f30 tx=0x7f373800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:04.694 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:04.686+0000 7f3737fff700 1 -- 172.21.15.42:0/3826016496 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3738008c90 con 0x7f374408df70 2019-12-05T06:52:04.694 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:04.686+0000 7f3748d9c700 1 --2- 172.21.15.42:0/3826016496 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f374408df70 0x7f374409f2d0 secure :-1 s=THROTTLE_DONE pgs=3400 cs=0 l=1 rx=0x7f3738002f30 tx=0x7f373800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:04.694 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:04.686+0000 7f3748d9c700 1 --2- 172.21.15.42:0/3826016496 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f374408df70 0x7f374409f2d0 secure :-1 s=THROTTLE_DONE pgs=3400 cs=0 l=1 rx=0x7f3738002f30 tx=0x7f373800fd30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:04.694 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:04.686+0000 7f3737fff700 1 -- 172.21.15.42:0/3826016496 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37380088b0 con 0x7f374408df70 2019-12-05T06:52:04.695 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:04.686+0000 7f3737fff700 1 -- 172.21.15.42:0/3826016496 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3738011ad0 con 0x7f374408df70 2019-12-05T06:52:04.695 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:04.686+0000 7f374b000700 1 -- 172.21.15.42:0/3826016496 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f374408df70 msgr2=0x7f374409f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:52:04.695 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:04.686+0000 7f374b000700 1 --2- 172.21.15.42:0/3826016496 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f374408df70 0x7f374409f2d0 secure :-1 s=READY pgs=3400 cs=0 l=1 rx=0x7f3738002f30 tx=0x7f373800fd30).stop 2019-12-05T06:52:04.695 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:04.686+0000 7f374b000700 1 -- 172.21.15.42:0/3826016496 shutdown_connections 2019-12-05T06:52:04.695 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:04.686+0000 7f374b000700 1 --2- 172.21.15.42:0/3826016496 >> v2:172.21.15.40:3300/0 conn(0x7f374409f820 0x7f374414efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:52:04.695 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:04.686+0000 7f374b000700 1 --2- 172.21.15.42:0/3826016496 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f374408df70 0x7f374409f2d0 unknown :-1 s=CLOSED pgs=3400 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:52:04.695 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:04.686+0000 7f374b000700 1 -- 172.21.15.42:0/3826016496 shutdown_connections 2019-12-05T06:52:04.696 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:04.686+0000 7f374b000700 1 -- 172.21.15.42:0/3826016496 wait complete. 2019-12-05T06:52:04.696 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:04.686+0000 7f374b000700 1 -- 172.21.15.42:0/3826016496 >> 172.21.15.42:0/3826016496 conn(0x7f37440aabd0 msgr2=0x7f37440a6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:52:04.699 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:04.690+0000 7f374b000700 1 Processor -- start 2019-12-05T06:52:04.699 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:04.690+0000 7f374b000700 1 -- start start 2019-12-05T06:52:04.699 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:04.690+0000 7f374b000700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f374412d170 0x7f374412f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:52:04.699 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:04.690+0000 7f374b000700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f374407a770 con 0x7f37440a3a50 2019-12-05T06:52:04.699 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:04.690+0000 7f374b000700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3744130b10 con 0x7f374407a340 2019-12-05T06:52:04.700 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:04.690+0000 7f374b000700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3744145450 con 0x7f374412d170 2019-12-05T06:52:04.700 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:04.690+0000 7f3748d9c700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f374407a340 0x7f374407c1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:39568/0 (socket says 172.21.15.42:39568) 2019-12-05T06:52:04.700 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:04.690+0000 7f3748d9c700 1 -- 172.21.15.42:0/1968761663 learned_addr learned my addr 172.21.15.42:0/1968761663 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:52:04.700 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:04.690+0000 7f3749d9e700 1 --2- 172.21.15.42:0/1968761663 >> v2:172.21.15.40:3300/0 conn(0x7f374412d170 0x7f374412f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:52:04.701 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:04.690+0000 7f3735ffb700 1 -- 172.21.15.42:0/1968761663 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 491312963 0 0) 0x7f3744130b10 con 0x7f374407a340 2019-12-05T06:52:04.701 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:04.690+0000 7f3735ffb700 1 -- 172.21.15.42:0/1968761663 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37240030b0 con 0x7f374407a340 2019-12-05T06:52:04.701 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:04.694+0000 7f3749d9e700 1 -- 172.21.15.42:0/1968761663 >> v1:172.21.15.42:6789/0 conn(0x7f374407a340 legacy=0x7f374407c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:52:04.701 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:04.694+0000 7f3749d9e700 1 -- 172.21.15.42:0/1968761663 >> v1:172.21.15.40:6789/0 conn(0x7f37440a3a50 legacy=0x7f3744144bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:52:04.701 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:04.694+0000 7f3735ffb700 1 -- 172.21.15.42:0/1968761663 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3808520741 0 0) 0x7f374407a770 con 0x7f37440a3a50 2019-12-05T06:52:04.701 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:04.694+0000 7f3749d9e700 1 -- 172.21.15.42:0/1968761663 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3744131d50 con 0x7f374412d170 2019-12-05T06:52:04.702 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:04.694+0000 7f3749d9e700 1 --2- 172.21.15.42:0/1968761663 >> v2:172.21.15.40:3300/0 conn(0x7f374412d170 0x7f374412f5b0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f3738011d30 tx=0x7f373c010bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:04.702 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:04.694+0000 7f3749d9e700 1 --2- 172.21.15.42:0/1968761663 >> v2:172.21.15.40:3300/0 conn(0x7f374412d170 0x7f374412f5b0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f3738011d30 tx=0x7f373c010bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:04.702 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:04.694+0000 7f3749d9e700 1 --2- 172.21.15.42:0/1968761663 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f374412d170 0x7f374412f5b0 secure :-1 s=READY pgs=2583 cs=0 l=1 rx=0x7f3738011d30 tx=0x7f373c010bc0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:52:04.702 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:04.694+0000 7f3749d9e700 1 --2- 172.21.15.42:0/1968761663 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f374412d170 0x7f374412f5b0 secure :-1 s=THROTTLE_DONE pgs=2583 cs=0 l=1 rx=0x7f3738011d30 tx=0x7f373c010bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:04.702 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:04.694+0000 7f3749d9e700 1 --2- 172.21.15.42:0/1968761663 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f374412d170 0x7f374412f5b0 secure :-1 s=THROTTLE_DONE pgs=2583 cs=0 l=1 rx=0x7f3738011d30 tx=0x7f373c010bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:04.702 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:04.694+0000 7f3735ffb700 1 -- 172.21.15.42:0/1968761663 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3738011de0 con 0x7f374412d170 2019-12-05T06:52:04.702 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:04.694+0000 7f3749d9e700 1 --2- 172.21.15.42:0/1968761663 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f374412d170 0x7f374412f5b0 secure :-1 s=THROTTLE_DONE pgs=2583 cs=0 l=1 rx=0x7f3738011d30 tx=0x7f373c010bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:04.703 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:04.694+0000 7f3735ffb700 1 -- 172.21.15.42:0/1968761663 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37380116f0 con 0x7f374412d170 2019-12-05T06:52:04.703 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:04.694+0000 7f374b000700 1 -- 172.21.15.42:0/1968761663 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f374412fb00 con 0x7f374412d170 2019-12-05T06:52:04.703 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:04.694+0000 7f3735ffb700 1 -- 172.21.15.42:0/1968761663 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f373c007150 con 0x7f374412d170 2019-12-05T06:52:04.703 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:04.694+0000 7f374b000700 1 -- 172.21.15.42:0/1968761663 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f374412ffd0 con 0x7f374412d170 2019-12-05T06:52:04.703 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:04.694+0000 7f3749d9e700 1 --2- 172.21.15.42:0/1968761663 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f374412d170 0x7f374412f5b0 secure :-1 s=THROTTLE_DONE pgs=2583 cs=0 l=1 rx=0x7f3738011d30 tx=0x7f373c010bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:04.704 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:04.694+0000 7f3735ffb700 1 -- 172.21.15.42:0/1968761663 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f373c007420 con 0x7f374412d170 2019-12-05T06:52:04.704 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:04.694+0000 7f3749d9e700 1 --2- 172.21.15.42:0/1968761663 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f374412d170 0x7f374412f5b0 secure :-1 s=THROTTLE_DONE pgs=2583 cs=0 l=1 rx=0x7f3738011d30 tx=0x7f373c010bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:04.704 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:04.694+0000 7f3735ffb700 1 --2- 172.21.15.42:0/1968761663 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f372403b1e0 0x7f372403d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:52:04.705 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:04.698+0000 7f3735ffb700 1 -- 172.21.15.42:0/1968761663 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 12337+0+0 (secure 0 0 0) 0x7f373c04ac00 con 0x7f374412d170 2019-12-05T06:52:04.705 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:04.698+0000 7f374959d700 1 --2- 172.21.15.42:0/1968761663 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f372403b1e0 0x7f372403d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:52:04.705 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:04.698+0000 7f374959d700 1 --2- 172.21.15.42:0/1968761663 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f372403b1e0 0x7f372403d6f0 crc :-1 s=READY pgs=1426 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:52:04.705 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:04.698+0000 7f374b000700 1 -- 172.21.15.42:0/1968761663 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f374405d100 con 0x7f374412d170 2019-12-05T06:52:04.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:04.706+0000 7f3749d9e700 1 --2- 172.21.15.42:0/1968761663 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f374412d170 0x7f374412f5b0 secure :-1 s=THROTTLE_DONE pgs=2583 cs=0 l=1 rx=0x7f3738011d30 tx=0x7f373c010bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:04.710 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:04.706+0000 7f3735ffb700 1 -- 172.21.15.42:0/1968761663 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f373c031c80 con 0x7f374412d170 2019-12-05T06:52:04.882 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:04.878+0000 7f374b000700 1 -- 172.21.15.42:0/1968761663 --> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f374405b8b0 con 0x7f372403b1e0 2019-12-05T06:52:04.890 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:04.886+0000 7f3735ffb700 1 -- 172.21.15.42:0/1968761663 <== mgr.4108 v2:172.21.15.42:6800/11930 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+92070 (crc 0 0 0) 0x7f374405b8b0 con 0x7f372403b1e0 2019-12-05T06:52:04.899 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:52:04.899 INFO:teuthology.orchestra.run.smithi042.stderr:dumped all 2019-12-05T06:52:04.901 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:04.898+0000 7f374b000700 1 -- 172.21.15.42:0/1968761663 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f372403b1e0 msgr2=0x7f372403d6f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:52:04.901 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:04.898+0000 7f374b000700 1 --2- 172.21.15.42:0/1968761663 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f372403b1e0 0x7f372403d6f0 crc :-1 s=READY pgs=1426 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:52:04.901 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:04.898+0000 7f374b000700 1 -- 172.21.15.42:0/1968761663 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f374412d170 msgr2=0x7f374412f5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:52:04.901 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:04.898+0000 7f374b000700 1 --2- 172.21.15.42:0/1968761663 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f374412d170 0x7f374412f5b0 secure :-1 s=READY pgs=2583 cs=0 l=1 rx=0x7f3738011d30 tx=0x7f373c010bc0).stop 2019-12-05T06:52:04.901 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:04.898+0000 7f374b000700 1 -- 172.21.15.42:0/1968761663 shutdown_connections 2019-12-05T06:52:04.902 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:04.898+0000 7f374b000700 1 --2- 172.21.15.42:0/1968761663 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f372403b1e0 0x7f372403d6f0 unknown :-1 s=CLOSED pgs=1426 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:52:04.902 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:04.898+0000 7f374b000700 1 --2- 172.21.15.42:0/1968761663 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f374412d170 0x7f374412f5b0 unknown :-1 s=CLOSED pgs=2583 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:52:04.902 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:04.898+0000 7f374b000700 1 -- 172.21.15.42:0/1968761663 shutdown_connections 2019-12-05T06:52:04.902 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:04.898+0000 7f374b000700 1 -- 172.21.15.42:0/1968761663 wait complete. 2019-12-05T06:52:04.902 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:04.898+0000 7f374b000700 1 -- 172.21.15.42:0/1968761663 >> 172.21.15.42:0/1968761663 conn(0x7f37440a8110 msgr2=0x7f37440a6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:52:04.913 INFO:teuthology.orchestra.run.smithi042.stdout:{"pg_ready":true,"pg_map":{"version":3389,"stamp":"2019-12-05T06:52:03.623570+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":409363217,"num_objects":153,"num_object_clones":2,"num_object_copies":312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":153,"num_whiteouts":0,"num_read":9494,"num_read_kb":12806383,"num_write":49976,"num_write_kb":1021704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":192298,"num_omap_keys":296,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39821,"ondisk_log_size":39821,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":9202368,"kb_used_data":813696,"kb_used_omap":0,"kb_used_meta":8388608,"kb_avail":745772352,"statfs":{"total":773094113280,"available":763670888448,"internally_reserved":8589934592,"allocated":833224704,"data_stored":827210444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8589934592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":164798,"num_omap_keys":241,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004083"},"pg_stats":[{"pgid":"3.7","version":"126'6396","reported_seq":"8192","reported_epoch":"129","state":"active+clean","last_fresh":"2019-12-05T06:51:50.444739+0000","last_change":"2019-12-05T06:51:47.140998+0000","last_active":"2019-12-05T06:51:50.444739+0000","last_peered":"2019-12-05T06:51:50.444739+0000","last_clean":"2019-12-05T06:51:50.444739+0000","last_became_active":"2019-12-05T05:03:35.332717+0000","last_became_peered":"2019-12-05T05:03:35.332717+0000","last_unstale":"2019-12-05T06:51:50.444739+0000","last_undegraded":"2019-12-05T06:51:50.444739+0000","last_fullsized":"2019-12-05T06:51:50.444739+0000","mapping_epoch":21,"log_start":"118'3311","ondisk_log_start":"118'3311","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'6396","last_scrub_stamp":"2019-12-05T06:51:47.140916+0000","last_deep_scrub":"126'6396","last_deep_scrub_stamp":"2019-12-05T06:51:47.140916+0000","last_clean_scrub_stamp":"2019-12-05T06:51:47.140916+0000","log_size":3085,"ondisk_log_size":3085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":245,"num_read_kb":1968,"num_write":4375,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7,0],"acting":[3,5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":"124","reported_epoch":"128","state":"active+clean","last_fresh":"2019-12-05T06:51:44.867884+0000","last_change":"2019-12-05T06:51:44.867884+0000","last_active":"2019-12-05T06:51:44.867884+0000","last_peered":"2019-12-05T06:51:44.867884+0000","last_clean":"2019-12-05T06:51:44.867884+0000","last_became_active":"2019-12-05T05:03:29.184431+0000","last_became_peered":"2019-12-05T05:03:29.184431+0000","last_unstale":"2019-12-05T06:51:44.867884+0000","last_undegraded":"2019-12-05T06:51:44.867884+0000","last_fullsized":"2019-12-05T06:51:44.867884+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T06:51:44.867695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T06:51:44.867695+0000","last_clean_scrub_stamp":"2019-12-05T06:51:44.867695+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"128'2461","reported_seq":"3033","reported_epoch":"128","state":"active+clean","last_fresh":"2019-12-05T06:51:49.340434+0000","last_change":"2019-12-05T06:51:49.340434+0000","last_active":"2019-12-05T06:51:49.340434+0000","last_peered":"2019-12-05T06:51:49.340434+0000","last_clean":"2019-12-05T06:51:49.340434+0000","last_became_active":"2019-12-05T05:03:32.224827+0000","last_became_peered":"2019-12-05T05:03:32.224827+0000","last_unstale":"2019-12-05T06:51:49.340434+0000","last_undegraded":"2019-12-05T06:51:49.340434+0000","last_fullsized":"2019-12-05T06:51:49.340434+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"128'2461","last_scrub_stamp":"2019-12-05T06:51:49.340234+0000","last_deep_scrub":"128'2461","last_deep_scrub_stamp":"2019-12-05T06:51:49.340234+0000","last_clean_scrub_stamp":"2019-12-05T06:51:49.340234+0000","log_size":2461,"ondisk_log_size":2461,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":67108864,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":679,"num_read_kb":1201860,"num_write":2589,"num_write_kb":72175,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25462,"num_omap_keys":51,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"124'6219","reported_seq":"8103","reported_epoch":"129","state":"active+clean","last_fresh":"2019-12-05T06:51:50.506361+0000","last_change":"2019-12-05T06:51:36.996591+0000","last_active":"2019-12-05T06:51:50.506361+0000","last_peered":"2019-12-05T06:51:50.506361+0000","last_clean":"2019-12-05T06:51:50.506361+0000","last_became_active":"2019-12-05T05:03:35.332648+0000","last_became_peered":"2019-12-05T05:03:35.332648+0000","last_unstale":"2019-12-05T06:51:50.506361+0000","last_undegraded":"2019-12-05T06:51:50.506361+0000","last_fullsized":"2019-12-05T06:51:50.506361+0000","mapping_epoch":21,"log_start":"117'3205","ondisk_log_start":"117'3205","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:34.235090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:34.235090+0000","last_clean_scrub_stamp":"2019-12-05T05:03:34.235090+0000","log_size":3014,"ondisk_log_size":3014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":233,"num_read_kb":6040,"num_write":4390,"num_write_kb":51866,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,7],"acting":[0,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":"124","reported_epoch":"128","state":"active+clean","last_fresh":"2019-12-05T06:51:44.038732+0000","last_change":"2019-12-05T06:51:44.038732+0000","last_active":"2019-12-05T06:51:44.038732+0000","last_peered":"2019-12-05T06:51:44.038732+0000","last_clean":"2019-12-05T06:51:44.038732+0000","last_became_active":"2019-12-05T05:03:29.182409+0000","last_became_peered":"2019-12-05T05:03:29.182409+0000","last_unstale":"2019-12-05T06:51:44.038732+0000","last_undegraded":"2019-12-05T06:51:44.038732+0000","last_fullsized":"2019-12-05T06:51:44.038732+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T06:51:44.038545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T06:51:44.038545+0000","last_clean_scrub_stamp":"2019-12-05T06:51:44.038545+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"129'1454","reported_seq":"2112","reported_epoch":"129","state":"active+clean","last_fresh":"2019-12-05T06:51:55.902173+0000","last_change":"2019-12-05T06:51:55.902254+0000","last_active":"2019-12-05T06:51:55.902173+0000","last_peered":"2019-12-05T06:51:55.902173+0000","last_clean":"2019-12-05T06:51:55.902173+0000","last_became_active":"2019-12-05T05:03:32.227690+0000","last_became_peered":"2019-12-05T05:03:32.227690+0000","last_unstale":"2019-12-05T06:51:55.902173+0000","last_undegraded":"2019-12-05T06:51:55.902173+0000","last_fullsized":"2019-12-05T06:51:55.902173+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'1454","last_scrub_stamp":"2019-12-05T06:51:55.902169+0000","last_deep_scrub":"129'1454","last_deep_scrub_stamp":"2019-12-05T06:51:55.902169+0000","last_clean_scrub_stamp":"2019-12-05T06:51:55.902169+0000","log_size":1454,"ondisk_log_size":1454,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":25166390,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":832,"num_read_kb":914148,"num_write":1232,"num_write_kb":29663,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":65753,"num_omap_keys":49,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"126'6687","reported_seq":"8466","reported_epoch":"129","state":"active+clean","last_fresh":"2019-12-05T06:51:58.649365+0000","last_change":"2019-12-05T06:51:58.649654+0000","last_active":"2019-12-05T06:51:58.649365+0000","last_peered":"2019-12-05T06:51:58.649365+0000","last_clean":"2019-12-05T06:51:58.649365+0000","last_became_active":"2019-12-05T05:03:35.266235+0000","last_became_peered":"2019-12-05T05:03:35.266235+0000","last_unstale":"2019-12-05T06:51:58.649365+0000","last_undegraded":"2019-12-05T06:51:58.649365+0000","last_fullsized":"2019-12-05T06:51:58.649365+0000","mapping_epoch":21,"log_start":"118'3617","ondisk_log_start":"118'3617","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'6687","last_scrub_stamp":"2019-12-05T06:51:58.649342+0000","last_deep_scrub":"126'6687","last_deep_scrub_stamp":"2019-12-05T06:51:58.649342+0000","last_clean_scrub_stamp":"2019-12-05T06:51:58.649342+0000","log_size":3070,"ondisk_log_size":3070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":15,"num_objects":3,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":240,"num_read_kb":18617,"num_write":4624,"num_write_kb":74599,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3,5],"acting":[0,4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":"124","reported_epoch":"128","state":"active+clean","last_fresh":"2019-12-05T06:51:44.155842+0000","last_change":"2019-12-05T06:51:44.155842+0000","last_active":"2019-12-05T06:51:44.155842+0000","last_peered":"2019-12-05T06:51:44.155842+0000","last_clean":"2019-12-05T06:51:44.155842+0000","last_became_active":"2019-12-05T05:03:29.180832+0000","last_became_peered":"2019-12-05T05:03:29.180832+0000","last_unstale":"2019-12-05T06:51:44.155842+0000","last_undegraded":"2019-12-05T06:51:44.155842+0000","last_fullsized":"2019-12-05T06:51:44.155842+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T06:51:44.155709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T06:51:44.155709+0000","last_clean_scrub_stamp":"2019-12-05T06:51:44.155709+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"129'2432","reported_seq":"3450","reported_epoch":"129","state":"active+clean","last_fresh":"2019-12-05T06:51:51.936396+0000","last_change":"2019-12-05T06:51:51.936467+0000","last_active":"2019-12-05T06:51:51.936396+0000","last_peered":"2019-12-05T06:51:51.936396+0000","last_clean":"2019-12-05T06:51:51.936396+0000","last_became_active":"2019-12-05T05:03:32.226895+0000","last_became_peered":"2019-12-05T05:03:32.226895+0000","last_unstale":"2019-12-05T06:51:51.936396+0000","last_undegraded":"2019-12-05T06:51:51.936396+0000","last_fullsized":"2019-12-05T06:51:51.936396+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'2432","last_scrub_stamp":"2019-12-05T06:51:51.936389+0000","last_deep_scrub":"129'2432","last_deep_scrub_stamp":"2019-12-05T06:51:51.936389+0000","last_clean_scrub_stamp":"2019-12-05T06:51:51.936389+0000","log_size":2432,"ondisk_log_size":2432,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46377447,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":1503,"num_read_kb":2307020,"num_write":2206,"num_write_kb":63089,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":11,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":49583,"num_omap_keys":93,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"124'6690","reported_seq":"8478","reported_epoch":"129","state":"active+clean","last_fresh":"2019-12-05T06:51:51.439679+0000","last_change":"2019-12-05T06:51:51.439765+0000","last_active":"2019-12-05T06:51:51.439679+0000","last_peered":"2019-12-05T06:51:51.439679+0000","last_clean":"2019-12-05T06:51:51.439679+0000","last_became_active":"2019-12-05T05:03:35.627352+0000","last_became_peered":"2019-12-05T05:03:35.627352+0000","last_unstale":"2019-12-05T06:51:51.439679+0000","last_undegraded":"2019-12-05T06:51:51.439679+0000","last_fullsized":"2019-12-05T06:51:51.439679+0000","mapping_epoch":21,"log_start":"118'3612","ondisk_log_start":"118'3612","created":21,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'6690","last_scrub_stamp":"2019-12-05T06:51:51.439674+0000","last_deep_scrub":"124'6690","last_deep_scrub_stamp":"2019-12-05T06:51:51.439674+0000","last_clean_scrub_stamp":"2019-12-05T06:51:51.439674+0000","log_size":3078,"ondisk_log_size":3078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":249,"num_read_kb":14990,"num_write":4595,"num_write_kb":60350,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6,5],"acting":[1,2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":"124","reported_epoch":"128","state":"active+clean","last_fresh":"2019-12-05T06:51:47.723043+0000","last_change":"2019-12-05T06:51:47.723043+0000","last_active":"2019-12-05T06:51:47.723043+0000","last_peered":"2019-12-05T06:51:47.723043+0000","last_clean":"2019-12-05T06:51:47.723043+0000","last_became_active":"2019-12-05T05:03:29.407108+0000","last_became_peered":"2019-12-05T05:03:29.407108+0000","last_unstale":"2019-12-05T06:51:47.723043+0000","last_undegraded":"2019-12-05T06:51:47.723043+0000","last_fullsized":"2019-12-05T06:51:47.723043+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T06:51:47.722916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T06:51:47.722916+0000","last_clean_scrub_stamp":"2019-12-05T06:51:47.722916+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"128'1584","reported_seq":"2622","reported_epoch":"128","state":"active+clean","last_fresh":"2019-12-05T06:51:47.524078+0000","last_change":"2019-12-05T06:51:47.524078+0000","last_active":"2019-12-05T06:51:47.524078+0000","last_peered":"2019-12-05T06:51:47.524078+0000","last_clean":"2019-12-05T06:51:47.524078+0000","last_became_active":"2019-12-05T05:03:32.635937+0000","last_became_peered":"2019-12-05T05:03:32.635937+0000","last_unstale":"2019-12-05T06:51:47.524078+0000","last_undegraded":"2019-12-05T06:51:47.524078+0000","last_fullsized":"2019-12-05T06:51:47.524078+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"128'1584","last_scrub_stamp":"2019-12-05T06:51:47.523996+0000","last_deep_scrub":"128'1584","last_deep_scrub_stamp":"2019-12-05T06:51:47.523996+0000","last_clean_scrub_stamp":"2019-12-05T06:51:47.523996+0000","log_size":1584,"ondisk_log_size":1584,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":35827812,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":920,"num_read_kb":599871,"num_write":1407,"num_write_kb":35966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":12000,"num_omap_keys":24,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"126'6396","reported_seq":"8256","reported_epoch":"129","state":"active+clean","last_fresh":"2019-12-05T06:51:50.445104+0000","last_change":"2019-12-05T06:51:45.089230+0000","last_active":"2019-12-05T06:51:50.445104+0000","last_peered":"2019-12-05T06:51:50.445104+0000","last_clean":"2019-12-05T06:51:50.445104+0000","last_became_active":"2019-12-05T05:03:36.314006+0000","last_became_peered":"2019-12-05T05:03:36.314006+0000","last_unstale":"2019-12-05T06:51:50.445104+0000","last_undegraded":"2019-12-05T06:51:50.445104+0000","last_fullsized":"2019-12-05T06:51:50.445104+0000","mapping_epoch":21,"log_start":"117'3307","ondisk_log_start":"117'3307","created":21,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'6396","last_scrub_stamp":"2019-12-05T06:51:45.089137+0000","last_deep_scrub":"126'6396","last_deep_scrub_stamp":"2019-12-05T06:51:45.089137+0000","last_clean_scrub_stamp":"2019-12-05T06:51:45.089137+0000","log_size":3089,"ondisk_log_size":3089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":253,"num_read_kb":6039,"num_write":4487,"num_write_kb":82964,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6,4],"acting":[3,5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"124","reported_epoch":"128","state":"active+clean","last_fresh":"2019-12-05T06:51:44.789603+0000","last_change":"2019-12-05T06:51:44.789603+0000","last_active":"2019-12-05T06:51:44.789603+0000","last_peered":"2019-12-05T06:51:44.789603+0000","last_clean":"2019-12-05T06:51:44.789603+0000","last_became_active":"2019-12-05T05:03:29.406770+0000","last_became_peered":"2019-12-05T05:03:29.406770+0000","last_unstale":"2019-12-05T06:51:44.789603+0000","last_undegraded":"2019-12-05T06:51:44.789603+0000","last_fullsized":"2019-12-05T06:51:44.789603+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T06:51:44.789407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T06:51:44.789407+0000","last_clean_scrub_stamp":"2019-12-05T06:51:44.789407+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"126'1789","reported_seq":"2610","reported_epoch":"128","state":"active+clean","last_fresh":"2019-12-05T06:51:40.914161+0000","last_change":"2019-12-05T05:03:32.227654+0000","last_active":"2019-12-05T06:51:40.914161+0000","last_peered":"2019-12-05T06:51:40.914161+0000","last_clean":"2019-12-05T06:51:40.914161+0000","last_became_active":"2019-12-05T05:03:32.227052+0000","last_became_peered":"2019-12-05T05:03:32.227052+0000","last_unstale":"2019-12-05T06:51:40.914161+0000","last_undegraded":"2019-12-05T06:51:40.914161+0000","last_fullsized":"2019-12-05T06:51:40.914161+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:31.200243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:31.200243+0000","last_clean_scrub_stamp":"2019-12-05T05:03:31.200243+0000","log_size":1789,"ondisk_log_size":1789,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46137378,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":922,"num_read_kb":1904128,"num_write":1592,"num_write_kb":62385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"124'6539","reported_seq":"8362","reported_epoch":"129","state":"active+clean","last_fresh":"2019-12-05T06:51:50.713422+0000","last_change":"2019-12-05T06:51:46.818890+0000","last_active":"2019-12-05T06:51:50.713422+0000","last_peered":"2019-12-05T06:51:50.713422+0000","last_clean":"2019-12-05T06:51:50.713422+0000","last_became_active":"2019-12-05T05:03:35.732405+0000","last_became_peered":"2019-12-05T05:03:35.732405+0000","last_unstale":"2019-12-05T06:51:50.713422+0000","last_undegraded":"2019-12-05T06:51:50.713422+0000","last_fullsized":"2019-12-05T06:51:50.713422+0000","mapping_epoch":21,"log_start":"118'3508","ondisk_log_start":"118'3508","created":21,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'6539","last_scrub_stamp":"2019-12-05T06:51:46.818781+0000","last_deep_scrub":"124'6539","last_deep_scrub_stamp":"2019-12-05T06:51:46.818781+0000","last_clean_scrub_stamp":"2019-12-05T06:51:46.818781+0000","log_size":3031,"ondisk_log_size":3031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":262,"num_read_kb":13412,"num_write":4550,"num_write_kb":85060,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0,6],"acting":[4,5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":"124","reported_epoch":"128","state":"active+clean","last_fresh":"2019-12-05T06:51:45.824965+0000","last_change":"2019-12-05T06:51:45.824965+0000","last_active":"2019-12-05T06:51:45.824965+0000","last_peered":"2019-12-05T06:51:45.824965+0000","last_clean":"2019-12-05T06:51:45.824965+0000","last_became_active":"2019-12-05T05:03:29.406903+0000","last_became_peered":"2019-12-05T05:03:29.406903+0000","last_unstale":"2019-12-05T06:51:45.824965+0000","last_undegraded":"2019-12-05T06:51:45.824965+0000","last_fullsized":"2019-12-05T06:51:45.824965+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T06:51:45.824777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T06:51:45.824777+0000","last_clean_scrub_stamp":"2019-12-05T06:51:45.824777+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"126'1333","reported_seq":"2011","reported_epoch":"128","state":"active+clean","last_fresh":"2019-12-05T06:51:40.913644+0000","last_change":"2019-12-05T05:03:32.227838+0000","last_active":"2019-12-05T06:51:40.913644+0000","last_peered":"2019-12-05T06:51:40.913644+0000","last_clean":"2019-12-05T06:51:40.913644+0000","last_became_active":"2019-12-05T05:03:32.227050+0000","last_became_peered":"2019-12-05T05:03:32.227050+0000","last_unstale":"2019-12-05T06:51:40.913644+0000","last_undegraded":"2019-12-05T06:51:40.913644+0000","last_fullsized":"2019-12-05T06:51:40.913644+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:31.200243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:31.200243+0000","last_clean_scrub_stamp":"2019-12-05T05:03:31.200243+0000","log_size":1333,"ondisk_log_size":1333,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":67108864,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":576,"num_read_kb":2216360,"num_write":1199,"num_write_kb":74709,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"124'6466","reported_seq":"8256","reported_epoch":"129","state":"active+clean","last_fresh":"2019-12-05T06:51:53.151766+0000","last_change":"2019-12-05T06:51:53.151861+0000","last_active":"2019-12-05T06:51:53.151766+0000","last_peered":"2019-12-05T06:51:53.151766+0000","last_clean":"2019-12-05T06:51:53.151766+0000","last_became_active":"2019-12-05T05:03:35.266868+0000","last_became_peered":"2019-12-05T05:03:35.266868+0000","last_unstale":"2019-12-05T06:51:53.151766+0000","last_undegraded":"2019-12-05T06:51:53.151766+0000","last_fullsized":"2019-12-05T06:51:53.151766+0000","mapping_epoch":21,"log_start":"118'3409","ondisk_log_start":"118'3409","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'6466","last_scrub_stamp":"2019-12-05T06:51:53.151758+0000","last_deep_scrub":"124'6466","last_deep_scrub_stamp":"2019-12-05T06:51:53.151758+0000","last_clean_scrub_stamp":"2019-12-05T06:51:53.151758+0000","log_size":3057,"ondisk_log_size":3057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":226,"num_read_kb":10307,"num_write":4503,"num_write_kb":69746,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0,5],"acting":[1,2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":"123","reported_epoch":"128","state":"active+clean","last_fresh":"2019-12-05T06:51:49.713098+0000","last_change":"2019-12-05T06:51:49.713098+0000","last_active":"2019-12-05T06:51:49.713098+0000","last_peered":"2019-12-05T06:51:49.713098+0000","last_clean":"2019-12-05T06:51:49.713098+0000","last_became_active":"2019-12-05T05:03:29.406857+0000","last_became_peered":"2019-12-05T05:03:29.406857+0000","last_unstale":"2019-12-05T06:51:49.713098+0000","last_undegraded":"2019-12-05T06:51:49.713098+0000","last_fullsized":"2019-12-05T06:51:49.713098+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T06:51:49.712908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T06:51:49.712908+0000","last_clean_scrub_stamp":"2019-12-05T06:51:49.712908+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"129'2908","reported_seq":"3880","reported_epoch":"129","state":"active+clean","last_fresh":"2019-12-05T06:51:52.982864+0000","last_change":"2019-12-05T06:51:52.983093+0000","last_active":"2019-12-05T06:51:52.982864+0000","last_peered":"2019-12-05T06:51:52.982864+0000","last_clean":"2019-12-05T06:51:52.982864+0000","last_became_active":"2019-12-05T05:03:32.479565+0000","last_became_peered":"2019-12-05T05:03:32.479565+0000","last_unstale":"2019-12-05T06:51:52.982864+0000","last_undegraded":"2019-12-05T06:51:52.982864+0000","last_fullsized":"2019-12-05T06:51:52.982864+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'2908","last_scrub_stamp":"2019-12-05T06:51:52.982843+0000","last_deep_scrub":"129'2908","last_deep_scrub_stamp":"2019-12-05T06:51:52.982843+0000","last_clean_scrub_stamp":"2019-12-05T06:51:52.982843+0000","log_size":2908,"ondisk_log_size":2908,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":83887077,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":1148,"num_read_kb":2218837,"num_write":2638,"num_write_kb":91697,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":24000,"num_omap_keys":48,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.5","version":"124'6269","reported_seq":"8149","reported_epoch":"129","state":"active+clean","last_fresh":"2019-12-05T06:51:50.832868+0000","last_change":"2019-12-05T06:51:36.920210+0000","last_active":"2019-12-05T06:51:50.832868+0000","last_peered":"2019-12-05T06:51:50.832868+0000","last_clean":"2019-12-05T06:51:50.832868+0000","last_became_active":"2019-12-05T05:03:35.629300+0000","last_became_peered":"2019-12-05T05:03:35.629300+0000","last_unstale":"2019-12-05T06:51:50.832868+0000","last_undegraded":"2019-12-05T06:51:50.832868+0000","last_fullsized":"2019-12-05T06:51:50.832868+0000","mapping_epoch":21,"log_start":"117'3209","ondisk_log_start":"117'3209","created":21,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T05:03:34.235090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T05:03:34.235090+0000","last_clean_scrub_stamp":"2019-12-05T05:03:34.235090+0000","log_size":3060,"ondisk_log_size":3060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":275,"num_read_kb":10198,"num_write":4396,"num_write_kb":76174,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2,6],"acting":[5,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":"123","reported_epoch":"129","state":"active+clean","last_fresh":"2019-12-05T06:51:51.742615+0000","last_change":"2019-12-05T06:51:51.742806+0000","last_active":"2019-12-05T06:51:51.742615+0000","last_peered":"2019-12-05T06:51:51.742615+0000","last_clean":"2019-12-05T06:51:51.742615+0000","last_became_active":"2019-12-05T05:03:29.184716+0000","last_became_peered":"2019-12-05T05:03:29.184716+0000","last_unstale":"2019-12-05T06:51:51.742615+0000","last_undegraded":"2019-12-05T06:51:51.742615+0000","last_fullsized":"2019-12-05T06:51:51.742615+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T06:51:51.742606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T06:51:51.742606+0000","last_clean_scrub_stamp":"2019-12-05T06:51:51.742606+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"128'1376","reported_seq":"2136","reported_epoch":"128","state":"active+clean","last_fresh":"2019-12-05T06:51:48.182277+0000","last_change":"2019-12-05T06:51:48.182277+0000","last_active":"2019-12-05T06:51:48.182277+0000","last_peered":"2019-12-05T06:51:48.182277+0000","last_clean":"2019-12-05T06:51:48.182277+0000","last_became_active":"2019-12-05T05:03:32.476140+0000","last_became_peered":"2019-12-05T05:03:32.476140+0000","last_unstale":"2019-12-05T06:51:48.182277+0000","last_undegraded":"2019-12-05T06:51:48.182277+0000","last_fullsized":"2019-12-05T06:51:48.182277+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"128'1376","last_scrub_stamp":"2019-12-05T06:51:48.182181+0000","last_deep_scrub":"128'1376","last_deep_scrub_stamp":"2019-12-05T06:51:48.182181+0000","last_clean_scrub_stamp":"2019-12-05T06:51:48.182181+0000","log_size":1376,"ondisk_log_size":1376,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":37749370,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":931,"num_read_kb":1362588,"num_write":1193,"num_write_kb":41853,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15500,"num_omap_keys":31,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":8,"stat_sum":{"num_bytes":15,"num_objects":3,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1983,"num_read_kb":81571,"num_write":35920,"num_write_kb":550167,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":49152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24484,"ondisk_log_size":24484,"up":32,"acting":32,"num_store_stats":8},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":409363202,"num_objects":150,"num_object_clones":0,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":150,"num_whiteouts":0,"num_read":7511,"num_read_kb":12724812,"num_write":14056,"num_write_kb":471537,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":192298,"num_omap_keys":296,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":818790400,"data_stored":818726404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":15337,"ondisk_log_size":15337,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576164,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1202144,"kb_used_data":153560,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93169696,"statfs":{"total":96636764160,"available":95405768704,"internally_reserved":1073741824,"allocated":157245440,"data_stored":156485275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Dec 5 06:51:04 2019","interfaces":[{"interface":"back","average":{"1min":0.872,"5min":10.013,"15min":7.459},"min":{"1min":0.425,"5min":0.287,"15min":0.226},"max":{"1min":1.461,"5min":501.337,"15min":825.441},"last":0.745},{"interface":"front","average":{"1min":0.870,"5min":2.647,"15min":4.935},"min":{"1min":0.354,"5min":0.321,"15min":0.318},"max":{"1min":1.431,"5min":150.493,"15min":817.919},"last":1.152}]},{"osd":1,"last update":"Thu Dec 5 06:51:04 2019","interfaces":[{"interface":"back","average":{"1min":0.909,"5min":8.193,"15min":7.424},"min":{"1min":0.512,"5min":0.300,"15min":0.241},"max":{"1min":1.455,"5min":501.235,"15min":735.242},"last":0.981},{"interface":"front","average":{"1min":1.021,"5min":0.925,"15min":5.173},"min":{"1min":0.546,"5min":0.387,"15min":0.322},"max":{"1min":1.495,"5min":2.264,"15min":501.253},"last":0.894}]},{"osd":2,"last update":"Thu Dec 5 06:51:10 2019","interfaces":[{"interface":"back","average":{"1min":0.962,"5min":0.933,"15min":6.753},"min":{"1min":0.651,"5min":0.468,"15min":0.292},"max":{"1min":1.366,"5min":2.133,"15min":982.285},"last":1.040},{"interface":"front","average":{"1min":1.027,"5min":5.943,"15min":10.961},"min":{"1min":0.703,"5min":0.425,"15min":0.272},"max":{"1min":1.527,"5min":419.518,"15min":842.181},"last":0.850}]},{"osd":3,"last update":"Thu Dec 5 06:51:04 2019","interfaces":[{"interface":"back","average":{"1min":1.095,"5min":1.005,"15min":1.545},"min":{"1min":0.733,"5min":0.341,"15min":0.309},"max":{"1min":1.351,"5min":2.072,"15min":157.003},"last":1.299},{"interface":"front","average":{"1min":1.228,"5min":1.101,"15min":1.825},"min":{"1min":0.795,"5min":0.458,"15min":0.384},"max":{"1min":1.399,"5min":2.103,"15min":240.924},"last":1.340}]},{"osd":4,"last update":"Thu Dec 5 06:51:04 2019","interfaces":[{"interface":"back","average":{"1min":0.859,"5min":27.486,"15min":13.538},"min":{"1min":0.542,"5min":0.312,"15min":0.312},"max":{"1min":1.314,"5min":994.097,"15min":994.097},"last":0.922},{"interface":"front","average":{"1min":1.063,"5min":1.049,"15min":4.392},"min":{"1min":0.673,"5min":0.358,"15min":0.358},"max":{"1min":1.401,"5min":2.367,"15min":586.973},"last":1.248}]},{"osd":5,"last update":"Thu Dec 5 06:51:04 2019","interfaces":[{"interface":"back","average":{"1min":1.179,"5min":1.090,"15min":1.100},"min":{"1min":0.796,"5min":0.423,"15min":0.419},"max":{"1min":1.636,"5min":2.455,"15min":2.455},"last":1.106},{"interface":"front","average":{"1min":1.243,"5min":1.174,"15min":5.572},"min":{"1min":0.751,"5min":0.522,"15min":0.489},"max":{"1min":1.547,"5min":2.425,"15min":791.026},"last":1.374}]},{"osd":6,"last update":"Thu Dec 5 06:51:04 2019","interfaces":[{"interface":"back","average":{"1min":1.159,"5min":1.083,"15min":3.761},"min":{"1min":0.773,"5min":0.458,"15min":0.332},"max":{"1min":1.589,"5min":2.317,"15min":878.805},"last":1.207},{"interface":"front","average":{"1min":1.305,"5min":5.742,"15min":8.220},"min":{"1min":0.822,"5min":0.549,"15min":0.519},"max":{"1min":1.695,"5min":450.021,"15min":840.923},"last":1.422}]}]},{"osd":6,"up_from":13,"seq":55834576160,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1140460,"kb_used_data":91876,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93231380,"statfs":{"total":96636764160,"available":95468933120,"internally_reserved":1073741824,"allocated":94081024,"data_stored":93329615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Dec 5 06:51:03 2019","interfaces":[{"interface":"back","average":{"1min":24.785,"5min":5.553,"15min":9.441},"min":{"1min":0.571,"5min":0.254,"15min":0.254},"max":{"1min":501.477,"5min":501.477,"15min":986.070},"last":0.759},{"interface":"front","average":{"1min":56.903,"5min":18.312,"15min":12.733},"min":{"1min":0.762,"5min":0.335,"15min":0.298},"max":{"1min":639.525,"5min":639.525,"15min":872.211},"last":0.701}]},{"osd":1,"last update":"Thu Dec 5 06:51:00 2019","interfaces":[{"interface":"back","average":{"1min":0.952,"5min":9.793,"15min":3.799},"min":{"1min":0.690,"5min":0.357,"15min":0.277},"max":{"1min":1.364,"5min":758.181,"15min":758.181},"last":0.837},{"interface":"front","average":{"1min":41.988,"5min":9.041,"15min":7.092},"min":{"1min":0.678,"5min":0.356,"15min":0.322},"max":{"1min":501.424,"5min":501.424,"15min":501.424},"last":1.060}]},{"osd":2,"last update":"Thu Dec 5 06:51:00 2019","interfaces":[{"interface":"back","average":{"1min":0.934,"5min":0.914,"15min":8.858},"min":{"1min":0.623,"5min":0.392,"15min":0.354},"max":{"1min":1.352,"5min":1.352,"15min":775.503},"last":0.896},{"interface":"front","average":{"1min":1.082,"5min":1.012,"15min":1.002},"min":{"1min":0.531,"5min":0.433,"15min":0.325},"max":{"1min":1.530,"5min":1.530,"15min":1.784},"last":1.223}]},{"osd":3,"last update":"Thu Dec 5 06:51:00 2019","interfaces":[{"interface":"back","average":{"1min":0.982,"5min":11.337,"15min":7.358},"min":{"1min":0.481,"5min":0.418,"15min":0.408},"max":{"1min":1.426,"5min":777.425,"15min":777.425},"last":1.172},{"interface":"front","average":{"1min":1.127,"5min":1.124,"15min":1.282},"min":{"1min":0.738,"5min":0.551,"15min":0.501},"max":{"1min":1.547,"5min":1.568,"15min":65.098},"last":1.357}]},{"osd":4,"last update":"Thu Dec 5 06:51:00 2019","interfaces":[{"interface":"back","average":{"1min":1.115,"5min":6.122,"15min":10.663},"min":{"1min":0.743,"5min":0.426,"15min":0.420},"max":{"1min":1.731,"5min":380.358,"15min":929.519},"last":1.012},{"interface":"front","average":{"1min":41.074,"5min":8.911,"15min":5.622},"min":{"1min":0.576,"5min":0.374,"15min":0.346},"max":{"1min":843.286,"5min":843.286,"15min":843.286},"last":1.404}]},{"osd":5,"last update":"Thu Dec 5 06:51:00 2019","interfaces":[{"interface":"back","average":{"1min":1.119,"5min":1.059,"15min":4.110},"min":{"1min":0.473,"5min":0.360,"15min":0.360},"max":{"1min":1.536,"5min":1.536,"15min":879.085},"last":1.120},{"interface":"front","average":{"1min":1.248,"5min":9.004,"15min":3.765},"min":{"1min":0.807,"5min":0.518,"15min":0.484},"max":{"1min":1.530,"5min":666.419,"15min":666.419},"last":1.260}]},{"osd":7,"last update":"Thu Dec 5 06:51:00 2019","interfaces":[{"interface":"back","average":{"1min":6.301,"5min":3.582,"15min":4.398},"min":{"1min":0.998,"5min":0.509,"15min":0.408},"max":{"1min":111.548,"5min":127.801,"15min":629.008},"last":1.473},{"interface":"front","average":{"1min":1.368,"5min":1.222,"15min":5.762},"min":{"1min":1.032,"5min":0.363,"15min":0.363},"max":{"1min":1.754,"5min":1.754,"15min":633.862},"last":1.505}]}]},{"osd":1,"up_from":13,"seq":55834576158,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1263576,"kb_used_data":214992,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93108264,"statfs":{"total":96636764160,"available":95342862336,"internally_reserved":1073741824,"allocated":220151808,"data_stored":219398906,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Dec 5 06:51:02 2019","interfaces":[{"interface":"back","average":{"1min":0.727,"5min":0.680,"15min":2.496},"min":{"1min":0.442,"5min":0.212,"15min":0.197},"max":{"1min":1.373,"5min":1.373,"15min":381.600},"last":0.607},{"interface":"front","average":{"1min":0.782,"5min":3.100,"15min":6.834},"min":{"1min":0.450,"5min":0.310,"15min":0.226},"max":{"1min":1.417,"5min":266.748,"15min":705.387},"last":0.679}]},{"osd":2,"last update":"Thu Dec 5 06:51:02 2019","interfaces":[{"interface":"back","average":{"1min":0.869,"5min":0.731,"15min":4.849},"min":{"1min":0.354,"5min":0.251,"15min":0.251},"max":{"1min":1.342,"5min":1.833,"15min":806.146},"last":0.896},{"interface":"front","average":{"1min":0.697,"5min":7.330,"15min":8.813},"min":{"1min":0.285,"5min":0.245,"15min":0.245},"max":{"1min":1.041,"5min":660.160,"15min":769.148},"last":1.101}]},{"osd":3,"last update":"Thu Dec 5 06:51:02 2019","interfaces":[{"interface":"back","average":{"1min":0.928,"5min":9.283,"15min":3.696},"min":{"1min":0.474,"5min":0.338,"15min":0.332},"max":{"1min":1.468,"5min":884.578,"15min":884.578},"last":630.113},{"interface":"front","average":{"1min":1.021,"5min":0.934,"15min":7.565},"min":{"1min":0.486,"5min":0.312,"15min":0.312},"max":{"1min":1.512,"5min":1.949,"15min":854.098},"last":1.043}]},{"osd":4,"last update":"Thu Dec 5 06:51:14 2019","interfaces":[{"interface":"back","average":{"1min":23.321,"5min":5.825,"15min":11.220},"min":{"1min":0.366,"5min":0.366,"15min":0.366},"max":{"1min":584.815,"5min":584.815,"15min":584.815},"last":0.952},{"interface":"front","average":{"1min":0.955,"5min":7.867,"15min":9.542},"min":{"1min":0.275,"5min":0.275,"15min":0.275},"max":{"1min":1.295,"5min":686.649,"15min":778.089},"last":0.781}]},{"osd":5,"last update":"Thu Dec 5 06:51:02 2019","interfaces":[{"interface":"back","average":{"1min":1.077,"5min":1.001,"15min":4.159},"min":{"1min":0.633,"5min":0.482,"15min":0.403},"max":{"1min":1.362,"5min":1.769,"15min":839.045},"last":1.256},{"interface":"front","average":{"1min":1.102,"5min":11.793,"15min":6.461},"min":{"1min":0.569,"5min":0.375,"15min":0.375},"max":{"1min":1.610,"5min":598.889,"15min":598.889},"last":0.845}]},{"osd":6,"last update":"Thu Dec 5 06:51:02 2019","interfaces":[{"interface":"back","average":{"1min":33.387,"5min":7.490,"15min":3.777},"min":{"1min":0.818,"5min":0.365,"15min":0.360},"max":{"1min":807.184,"5min":807.184,"15min":807.184},"last":1.375},{"interface":"front","average":{"1min":1.134,"5min":1.112,"15min":14.638},"min":{"1min":0.684,"5min":0.410,"15min":0.410},"max":{"1min":1.532,"5min":1.623,"15min":848.805},"last":1.170}]},{"osd":7,"last update":"Thu Dec 5 06:51:02 2019","interfaces":[{"interface":"back","average":{"1min":1.268,"5min":3.514,"15min":4.600},"min":{"1min":0.737,"5min":0.409,"15min":0.409},"max":{"1min":1.783,"5min":274.305,"15min":680.492},"last":1.221},{"interface":"front","average":{"1min":31.226,"5min":20.849,"15min":9.395},"min":{"1min":0.611,"5min":0.523,"15min":0.518},"max":{"1min":501.595,"5min":792.863,"15min":792.863},"last":696.822}]}]},{"osd":0,"up_from":13,"seq":55834576160,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1169152,"kb_used_data":120568,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93202688,"statfs":{"total":96636764160,"available":95439552512,"internally_reserved":1073741824,"allocated":123461632,"data_stored":122703096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Dec 5 06:51:48 2019","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":7.575,"15min":4.600},"min":{"1min":0.209,"5min":0.209,"15min":0.209},"max":{"1min":0.869,"5min":587.293,"15min":587.293},"last":0.546},{"interface":"front","average":{"1min":0.770,"5min":10.355,"15min":12.828},"min":{"1min":0.310,"5min":0.261,"15min":0.261},"max":{"1min":1.190,"5min":861.119,"15min":966.609},"last":0.933}]},{"osd":2,"last update":"Thu Dec 5 06:51:48 2019","interfaces":[{"interface":"back","average":{"1min":0.700,"5min":16.877,"15min":9.139},"min":{"1min":0.225,"5min":0.225,"15min":0.225},"max":{"1min":1.166,"5min":933.957,"15min":933.957},"last":0.837},{"interface":"front","average":{"1min":0.759,"5min":11.025,"15min":4.963},"min":{"1min":0.272,"5min":0.272,"15min":0.272},"max":{"1min":1.254,"5min":868.889,"15min":868.889},"last":1.062}]},{"osd":3,"last update":"Thu Dec 5 06:51:48 2019","interfaces":[{"interface":"back","average":{"1min":0.701,"5min":0.735,"15min":0.691},"min":{"1min":0.267,"5min":0.267,"15min":0.255},"max":{"1min":1.151,"5min":1.472,"15min":2.652},"last":0.680},{"interface":"front","average":{"1min":28.670,"5min":14.126,"15min":10.568},"min":{"1min":0.337,"5min":0.322,"15min":0.303},"max":{"1min":447.144,"5min":730.339,"15min":854.568},"last":0.764}]},{"osd":4,"last update":"Thu Dec 5 06:51:48 2019","interfaces":[{"interface":"back","average":{"1min":1.020,"5min":11.009,"15min":9.005},"min":{"1min":0.283,"5min":0.283,"15min":0.283},"max":{"1min":1.671,"5min":492.774,"15min":983.581},"last":1.186},{"interface":"front","average":{"1min":0.924,"5min":15.606,"15min":8.376},"min":{"1min":0.300,"5min":0.300,"15min":0.300},"max":{"1min":1.436,"5min":909.368,"15min":909.368},"last":0.902}]},{"osd":5,"last update":"Thu Dec 5 06:51:30 2019","interfaces":[{"interface":"back","average":{"1min":1.019,"5min":1.049,"15min":6.033},"min":{"1min":0.365,"5min":0.365,"15min":0.365},"max":{"1min":1.529,"5min":1.533,"15min":475.167},"last":0.999},{"interface":"front","average":{"1min":0.990,"5min":27.033,"15min":13.790},"min":{"1min":0.410,"5min":0.410,"15min":0.410},"max":{"1min":1.573,"5min":827.052,"15min":827.052},"last":1.228}]},{"osd":6,"last update":"Thu Dec 5 06:51:49 2019","interfaces":[{"interface":"back","average":{"1min":68.182,"5min":14.510,"15min":9.445},"min":{"1min":0.414,"5min":0.414,"15min":0.322},"max":{"1min":837.447,"5min":837.447,"15min":893.025},"last":1.288},{"interface":"front","average":{"1min":55.264,"5min":12.003,"15min":4.735},"min":{"1min":0.392,"5min":0.392,"15min":0.392},"max":{"1min":867.178,"5min":867.178,"15min":867.178},"last":1.367}]},{"osd":7,"last update":"Thu Dec 5 06:51:11 2019","interfaces":[{"interface":"back","average":{"1min":1.039,"5min":9.348,"15min":5.897},"min":{"1min":0.628,"5min":0.450,"15min":0.450},"max":{"1min":1.632,"5min":743.880,"15min":743.880},"last":1.113},{"interface":"front","average":{"1min":1.187,"5min":1.205,"15min":7.028},"min":{"1min":0.455,"5min":0.455,"15min":0.455},"max":{"1min":1.742,"5min":1.742,"15min":920.552},"last":1.322}]}]},{"osd":2,"up_from":13,"seq":55834576156,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1130392,"kb_used_data":81808,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93241448,"statfs":{"total":96636764160,"available":95479242752,"internally_reserved":1073741824,"allocated":83771392,"data_stored":83019551,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Dec 5 06:51:49 2019","interfaces":[{"interface":"back","average":{"1min":0.684,"5min":18.639,"15min":6.640},"min":{"1min":0.215,"5min":0.215,"15min":0.215},"max":{"1min":1.145,"5min":917.307,"15min":917.307},"last":0.667},{"interface":"front","average":{"1min":0.679,"5min":6.311,"15min":2.547},"min":{"1min":0.198,"5min":0.198,"15min":0.198},"max":{"1min":1.138,"5min":423.686,"15min":423.686},"last":0.804}]},{"osd":1,"last update":"Thu Dec 5 06:51:43 2019","interfaces":[{"interface":"back","average":{"1min":0.682,"5min":0.726,"15min":3.704},"min":{"1min":0.272,"5min":0.272,"15min":0.272},"max":{"1min":1.125,"5min":1.316,"15min":501.424},"last":0.556},{"interface":"front","average":{"1min":0.787,"5min":11.001,"15min":9.897},"min":{"1min":0.335,"5min":0.335,"15min":0.234},"max":{"1min":1.250,"5min":814.005,"15min":814.005},"last":0.605}]},{"osd":3,"last update":"Thu Dec 5 06:51:55 2019","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":5.131,"15min":2.197},"min":{"1min":0.306,"5min":0.306,"15min":0.262},"max":{"1min":1.075,"5min":474.268,"15min":474.268},"last":0.756},{"interface":"front","average":{"1min":0.716,"5min":0.820,"15min":0.837},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":1.110,"5min":1.254,"15min":3.965},"last":0.721}]},{"osd":4,"last update":"Thu Dec 5 06:51:43 2019","interfaces":[{"interface":"back","average":{"1min":1.006,"5min":8.466,"15min":3.949},"min":{"1min":0.375,"5min":0.332,"15min":0.323},"max":{"1min":1.513,"5min":632.509,"15min":632.509},"last":0.880},{"interface":"front","average":{"1min":1.084,"5min":1.117,"15min":4.261},"min":{"1min":0.398,"5min":0.398,"15min":0.365},"max":{"1min":1.550,"5min":2.178,"15min":991.418},"last":0.919}]},{"osd":5,"last update":"Thu Dec 5 06:51:43 2019","interfaces":[{"interface":"back","average":{"1min":1.014,"5min":0.950,"15min":5.645},"min":{"1min":0.386,"5min":0.316,"15min":0.316},"max":{"1min":1.451,"5min":1.501,"15min":965.993},"last":1.210},{"interface":"front","average":{"1min":35.947,"5min":18.944,"15min":9.226},"min":{"1min":0.410,"5min":0.381,"15min":0.381},"max":{"1min":698.069,"5min":700.890,"15min":760.855},"last":1.241}]},{"osd":6,"last update":"Thu Dec 5 06:51:50 2019","interfaces":[{"interface":"back","average":{"1min":19.400,"5min":4.800,"15min":6.753},"min":{"1min":0.464,"5min":0.415,"15min":0.344},"max":{"1min":385.427,"5min":385.427,"15min":652.287},"last":1.148},{"interface":"front","average":{"1min":15.332,"5min":18.194,"15min":11.007},"min":{"1min":0.355,"5min":0.355,"15min":0.355},"max":{"1min":300.514,"5min":934.081,"15min":934.081},"last":1.033}]},{"osd":7,"last update":"Thu Dec 5 06:51:43 2019","interfaces":[{"interface":"back","average":{"1min":1.167,"5min":5.526,"15min":2.606},"min":{"1min":0.493,"5min":0.440,"15min":0.440},"max":{"1min":1.618,"5min":419.900,"15min":419.900},"last":1.075},{"interface":"front","average":{"1min":1.079,"5min":1.128,"15min":1.074},"min":{"1min":0.429,"5min":0.408,"15min":0.408},"max":{"1min":1.574,"5min":1.620,"15min":1.794},"last":1.104}]}]},{"osd":3,"up_from":13,"seq":55834576159,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1085344,"kb_used_data":36760,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93286496,"statfs":{"total":96636764160,"available":95525371904,"internally_reserved":1073741824,"allocated":37642240,"data_stored":36894461,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Dec 5 06:51:36 2019","interfaces":[{"interface":"back","average":{"1min":0.662,"5min":18.296,"15min":13.058},"min":{"1min":0.296,"5min":0.197,"15min":0.196},"max":{"1min":1.334,"5min":802.154,"15min":802.154},"last":0.844},{"interface":"front","average":{"1min":0.750,"5min":7.929,"15min":9.078},"min":{"1min":0.319,"5min":0.259,"15min":0.242},"max":{"1min":1.454,"5min":828.266,"15min":991.916},"last":0.931}]},{"osd":1,"last update":"Thu Dec 5 06:51:45 2019","interfaces":[{"interface":"back","average":{"1min":0.803,"5min":8.874,"15min":16.543},"min":{"1min":0.415,"5min":0.264,"15min":0.211},"max":{"1min":2.417,"5min":699.405,"15min":957.811},"last":0.741},{"interface":"front","average":{"1min":0.959,"5min":0.867,"15min":7.839},"min":{"1min":0.522,"5min":0.311,"15min":0.282},"max":{"1min":2.441,"5min":2.441,"15min":501.932},"last":0.879}]},{"osd":2,"last update":"Thu Dec 5 06:50:59 2019","interfaces":[{"interface":"back","average":{"1min":0.830,"5min":0.878,"15min":4.117},"min":{"1min":0.524,"5min":0.266,"15min":0.266},"max":{"1min":1.512,"5min":1.512,"15min":444.359},"last":0.654},{"interface":"front","average":{"1min":0.932,"5min":0.867,"15min":9.156},"min":{"1min":0.431,"5min":0.340,"15min":0.297},"max":{"1min":1.586,"5min":1.586,"15min":814.674},"last":1.251}]},{"osd":4,"last update":"Thu Dec 5 06:51:36 2019","interfaces":[{"interface":"back","average":{"1min":1.004,"5min":0.989,"15min":1.926},"min":{"1min":0.618,"5min":0.402,"15min":0.325},"max":{"1min":2.287,"5min":2.287,"15min":216.348},"last":1.206},{"interface":"front","average":{"1min":1.072,"5min":1.012,"15min":1.582},"min":{"1min":0.659,"5min":0.335,"15min":0.335},"max":{"1min":1.392,"5min":1.669,"15min":192.285},"last":1.169}]},{"osd":5,"last update":"Thu Dec 5 06:51:36 2019","interfaces":[{"interface":"back","average":{"1min":1.133,"5min":8.721,"15min":4.576},"min":{"1min":0.690,"5min":0.372,"15min":0.328},"max":{"1min":2.345,"5min":880.020,"15min":880.020},"last":1.124},{"interface":"front","average":{"1min":0.984,"5min":1.075,"15min":1.045},"min":{"1min":0.414,"5min":0.414,"15min":0.364},"max":{"1min":1.327,"5min":1.532,"15min":1.936},"last":0.803}]},{"osd":6,"last update":"Thu Dec 5 06:51:35 2019","interfaces":[{"interface":"back","average":{"1min":51.161,"5min":11.110,"15min":6.234},"min":{"1min":0.557,"5min":0.430,"15min":0.413},"max":{"1min":1001.571,"5min":1001.571,"15min":1001.571},"last":1.084},{"interface":"front","average":{"1min":1.016,"5min":8.110,"15min":3.832},"min":{"1min":0.265,"5min":0.265,"15min":0.265},"max":{"1min":2.065,"5min":558.711,"15min":558.711},"last":0.972}]},{"osd":7,"last update":"Thu Dec 5 06:51:50 2019","interfaces":[{"interface":"back","average":{"1min":1.246,"5min":2.412,"15min":10.388},"min":{"1min":0.841,"5min":0.523,"15min":0.365},"max":{"1min":2.765,"5min":97.130,"15min":877.034},"last":1.037},{"interface":"front","average":{"1min":1.304,"5min":1.231,"15min":1.161},"min":{"1min":0.869,"5min":0.499,"15min":0.435},"max":{"1min":2.717,"5min":2.717,"15min":2.717},"last":1.291}]}]},{"osd":4,"up_from":13,"seq":55834576157,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1050352,"kb_used_data":1768,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93321488,"statfs":{"total":96636764160,"available":95561203712,"internally_reserved":1073741824,"allocated":1810432,"data_stored":1066649,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Dec 5 06:51:05 2019","interfaces":[{"interface":"back","average":{"1min":0.881,"5min":7.022,"15min":5.481},"min":{"1min":0.542,"5min":0.283,"15min":0.264},"max":{"1min":1.366,"5min":439.064,"15min":439.064},"last":0.874},{"interface":"front","average":{"1min":31.716,"5min":12.524,"15min":5.380},"min":{"1min":0.662,"5min":0.225,"15min":0.225},"max":{"1min":524.030,"5min":524.030,"15min":524.030},"last":1.169}]},{"osd":1,"last update":"Thu Dec 5 06:51:05 2019","interfaces":[{"interface":"back","average":{"1min":0.962,"5min":1.204,"15min":4.257},"min":{"1min":0.551,"5min":0.303,"15min":0.303},"max":{"1min":1.309,"5min":31.305,"15min":501.402},"last":1.214},{"interface":"front","average":{"1min":0.982,"5min":0.929,"15min":4.757},"min":{"1min":0.648,"5min":0.304,"15min":0.304},"max":{"1min":1.576,"5min":1.576,"15min":501.754},"last":1.027}]},{"osd":2,"last update":"Thu Dec 5 06:51:14 2019","interfaces":[{"interface":"back","average":{"1min":43.387,"5min":31.525,"15min":12.886},"min":{"1min":0.397,"5min":0.359,"15min":0.268},"max":{"1min":850.354,"5min":897.315,"15min":897.315},"last":0.729},{"interface":"front","average":{"1min":1.010,"5min":0.982,"15min":3.965},"min":{"1min":0.287,"5min":0.287,"15min":0.287},"max":{"1min":1.397,"5min":1.397,"15min":632.767},"last":0.782}]},{"osd":3,"last update":"Thu Dec 5 06:51:05 2019","interfaces":[{"interface":"back","average":{"1min":0.863,"5min":6.934,"15min":3.446},"min":{"1min":0.694,"5min":0.340,"15min":0.340},"max":{"1min":1.045,"5min":501.658,"15min":501.658},"last":0.687},{"interface":"front","average":{"1min":9.928,"5min":2.858,"15min":2.715},"min":{"1min":0.934,"5min":0.432,"15min":0.348},"max":{"1min":149.969,"5min":149.969,"15min":308.851},"last":0.959}]},{"osd":5,"last update":"Thu Dec 5 06:51:05 2019","interfaces":[{"interface":"back","average":{"1min":1.098,"5min":0.906,"15min":3.992},"min":{"1min":0.704,"5min":0.325,"15min":0.325},"max":{"1min":1.682,"5min":1.682,"15min":840.089},"last":1.067},{"interface":"front","average":{"1min":1.224,"5min":16.912,"15min":9.261},"min":{"1min":0.918,"5min":0.385,"15min":0.369},"max":{"1min":1.729,"5min":660.266,"15min":748.692},"last":1.300}]},{"osd":6,"last update":"Thu Dec 5 06:51:05 2019","interfaces":[{"interface":"back","average":{"1min":0.900,"5min":18.011,"15min":10.852},"min":{"1min":0.583,"5min":0.354,"15min":0.314},"max":{"1min":1.139,"5min":817.106,"15min":861.297},"last":0.833},{"interface":"front","average":{"1min":1.258,"5min":1.114,"15min":3.380},"min":{"1min":0.708,"5min":0.390,"15min":0.374},"max":{"1min":1.808,"5min":3.181,"15min":690.644},"last":1.257}]},{"osd":7,"last update":"Thu Dec 5 06:51:21 2019","interfaces":[{"interface":"back","average":{"1min":1.120,"5min":1.135,"15min":6.627},"min":{"1min":0.481,"5min":0.425,"15min":0.351},"max":{"1min":1.534,"5min":2.624,"15min":507.089},"last":1.146},{"interface":"front","average":{"1min":1.194,"5min":1.191,"15min":3.242},"min":{"1min":0.520,"5min":0.464,"15min":0.464},"max":{"1min":1.635,"5min":1.670,"15min":452.400},"last":1.345}]}]},{"osd":5,"up_from":13,"seq":55834576162,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1160948,"kb_used_data":112364,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93210892,"statfs":{"total":96636764160,"available":95447953408,"internally_reserved":1073741824,"allocated":115060736,"data_stored":114312891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Dec 5 06:51:52 2019","interfaces":[{"interface":"back","average":{"1min":0.849,"5min":4.329,"15min":1.981},"min":{"1min":0.261,"5min":0.261,"15min":0.235},"max":{"1min":1.463,"5min":331.975,"15min":331.975},"last":0.595},{"interface":"front","average":{"1min":0.698,"5min":0.829,"15min":3.592},"min":{"1min":0.285,"5min":0.285,"15min":0.278},"max":{"1min":1.062,"5min":2.878,"15min":676.393},"last":0.748}]},{"osd":1,"last update":"Thu Dec 5 06:50:58 2019","interfaces":[{"interface":"back","average":{"1min":0.854,"5min":0.773,"15min":2.216},"min":{"1min":0.544,"5min":0.308,"15min":0.225},"max":{"1min":1.145,"5min":1.274,"15min":251.684},"last":1.003},{"interface":"front","average":{"1min":38.990,"5min":8.478,"15min":5.469},"min":{"1min":0.721,"5min":0.388,"15min":0.245},"max":{"1min":722.849,"5min":722.849,"15min":722.849},"last":1.049}]},{"osd":2,"last update":"Thu Dec 5 06:51:45 2019","interfaces":[{"interface":"back","average":{"1min":0.821,"5min":17.263,"15min":7.170},"min":{"1min":0.362,"5min":0.362,"15min":0.281},"max":{"1min":1.164,"5min":934.880,"15min":934.880},"last":0.913},{"interface":"front","average":{"1min":0.948,"5min":9.446,"15min":8.045},"min":{"1min":0.407,"5min":0.407,"15min":0.407},"max":{"1min":1.326,"5min":676.797,"15min":794.102},"last":1.226}]},{"osd":3,"last update":"Thu Dec 5 06:51:24 2019","interfaces":[{"interface":"back","average":{"1min":1.037,"5min":1.093,"15min":3.676},"min":{"1min":0.468,"5min":0.468,"15min":0.340},"max":{"1min":1.318,"5min":2.942,"15min":683.842},"last":1.376},{"interface":"front","average":{"1min":42.174,"5min":19.109,"15min":8.696},"min":{"1min":0.500,"5min":0.500,"15min":0.387},"max":{"1min":740.171,"5min":740.171,"15min":740.171},"last":1.307}]},{"osd":4,"last update":"Thu Dec 5 06:51:52 2019","interfaces":[{"interface":"back","average":{"1min":1.037,"5min":1.096,"15min":7.585},"min":{"1min":0.341,"5min":0.341,"15min":0.327},"max":{"1min":1.476,"5min":2.914,"15min":633.428},"last":1.103},{"interface":"front","average":{"1min":0.964,"5min":0.988,"15min":5.125},"min":{"1min":0.302,"5min":0.302,"15min":0.302},"max":{"1min":1.507,"5min":1.507,"15min":792.823},"last":0.870}]},{"osd":6,"last update":"Thu Dec 5 06:51:52 2019","interfaces":[{"interface":"back","average":{"1min":1.088,"5min":1.044,"15min":6.845},"min":{"1min":0.282,"5min":0.282,"15min":0.282},"max":{"1min":1.829,"5min":1.829,"15min":891.609},"last":0.950},{"interface":"front","average":{"1min":1.210,"5min":13.625,"15min":10.318},"min":{"1min":0.394,"5min":0.394,"15min":0.394},"max":{"1min":1.721,"5min":635.191,"15min":916.939},"last":1.177}]},{"osd":7,"last update":"Thu Dec 5 06:51:52 2019","interfaces":[{"interface":"back","average":{"1min":1.230,"5min":8.263,"15min":6.474},"min":{"1min":0.601,"5min":0.586,"15min":0.401},"max":{"1min":1.766,"5min":735.632,"15min":735.632},"last":1.448},{"interface":"front","average":{"1min":1.303,"5min":5.207,"15min":6.313},"min":{"1min":0.618,"5min":0.618,"15min":0.380},"max":{"1min":1.874,"5min":405.552,"15min":405.552},"last":1.510}]}]}]}} 2019-12-05T06:52:04.919 INFO:tasks.ceph:pgid 3.6 last_scrub_stamp 2019-12-05T05:03:34.235090+0000 time.struct_time(tm_year=2019, tm_mon=12, tm_mday=5, tm_hour=5, tm_min=3, tm_sec=34, tm_wday=3, tm_yday=339, tm_isdst=-1) <= time.struct_time(tm_year=2019, tm_mon=12, tm_mday=5, tm_hour=6, tm_min=51, tm_sec=39, tm_wday=3, tm_yday=339, tm_isdst=0) 2019-12-05T06:52:04.920 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2019-12-05T05:03:31.200243+0000 time.struct_time(tm_year=2019, tm_mon=12, tm_mday=5, tm_hour=5, tm_min=3, tm_sec=31, tm_wday=3, tm_yday=339, tm_isdst=-1) <= time.struct_time(tm_year=2019, tm_mon=12, tm_mday=5, tm_hour=6, tm_min=51, tm_sec=39, tm_wday=3, tm_yday=339, tm_isdst=0) 2019-12-05T06:52:04.920 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2019-12-05T05:03:31.200243+0000 time.struct_time(tm_year=2019, tm_mon=12, tm_mday=5, tm_hour=5, tm_min=3, tm_sec=31, tm_wday=3, tm_yday=339, tm_isdst=-1) <= time.struct_time(tm_year=2019, tm_mon=12, tm_mday=5, tm_hour=6, tm_min=51, tm_sec=39, tm_wday=3, tm_yday=339, tm_isdst=0) 2019-12-05T06:52:04.920 INFO:tasks.ceph:pgid 3.5 last_scrub_stamp 2019-12-05T05:03:34.235090+0000 time.struct_time(tm_year=2019, tm_mon=12, tm_mday=5, tm_hour=5, tm_min=3, tm_sec=34, tm_wday=3, tm_yday=339, tm_isdst=-1) <= time.struct_time(tm_year=2019, tm_mon=12, tm_mday=5, tm_hour=6, tm_min=51, tm_sec=39, tm_wday=3, tm_yday=339, tm_isdst=0) 2019-12-05T06:52:04.921 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2019-12-05T06:52:21.108 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:52:21.112 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:52:24.922 INFO:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2019-12-05T06:52:25.026 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.018+0000 7f6ba673c700 1 Processor -- start 2019-12-05T06:52:25.026 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.018+0000 7f6ba673c700 1 -- start start 2019-12-05T06:52:25.026 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.018+0000 7f6ba673c700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6ba008df80 0x7f6ba009f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:52:25.026 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.018+0000 7f6ba673c700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ba009fc60 con 0x7f6ba009f830 2019-12-05T06:52:25.027 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.018+0000 7f6ba673c700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ba00a0710 con 0x7f6ba00a7740 2019-12-05T06:52:25.027 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.018+0000 7f6ba673c700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ba00a0b00 con 0x7f6ba008df80 2019-12-05T06:52:25.027 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.018+0000 7f6ba4cd9700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f6ba00a7740 0x7f6ba007e720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:39586/0 (socket says 172.21.15.42:39586) 2019-12-05T06:52:25.027 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.018+0000 7f6ba4cd9700 1 -- 172.21.15.42:0/1609616935 learned_addr learned my addr 172.21.15.42:0/1609616935 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:52:25.027 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.018+0000 7f6b9ffff700 1 --2- 172.21.15.42:0/1609616935 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6ba008df80 0x7f6ba009f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:52:25.027 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.018+0000 7f6b9f7fe700 1 -- 172.21.15.42:0/1609616935 <== mon.1 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1896658996 0 0) 0x7f6ba00a0710 con 0x7f6ba00a7740 2019-12-05T06:52:25.028 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.018+0000 7f6b9f7fe700 1 -- 172.21.15.42:0/1609616935 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b900030b0 con 0x7f6ba00a7740 2019-12-05T06:52:25.028 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.018+0000 7f6b9f7fe700 1 -- 172.21.15.42:0/1609616935 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 320211790 0 0) 0x7f6ba009fc60 con 0x7f6ba009f830 2019-12-05T06:52:25.028 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.018+0000 7f6b9f7fe700 1 -- 172.21.15.42:0/1609616935 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ba00a0710 con 0x7f6ba009f830 2019-12-05T06:52:25.028 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.018+0000 7f6b9ffff700 1 -- 172.21.15.42:0/1609616935 >> v1:172.21.15.42:6789/0 conn(0x7f6ba00a7740 legacy=0x7f6ba007e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:52:25.028 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.018+0000 7f6b9ffff700 1 -- 172.21.15.42:0/1609616935 >> v1:172.21.15.40:6789/0 conn(0x7f6ba009f830 legacy=0x7f6ba014efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:52:25.028 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.018+0000 7f6b9ffff700 1 -- 172.21.15.42:0/1609616935 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ba007ef20 con 0x7f6ba008df80 2019-12-05T06:52:25.028 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.018+0000 7f6b9ffff700 1 --2- 172.21.15.42:0/1609616935 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6ba008df80 0x7f6ba009f2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f6b8c002f30 tx=0x7f6b8c00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:25.029 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.018+0000 7f6b9ffff700 1 --2- 172.21.15.42:0/1609616935 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6ba008df80 0x7f6ba009f2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f6b8c002f30 tx=0x7f6b8c00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:25.029 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.018+0000 7f6b9ffff700 1 --2- 172.21.15.42:0/1609616935 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6ba008df80 0x7f6ba009f2e0 secure :-1 s=READY pgs=758 cs=0 l=1 rx=0x7f6b8c002f30 tx=0x7f6b8c00fbf0).ready entity=mon.2 client_cookie=2cc1335bcd2327c server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:52:25.029 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.018+0000 7f6b9ffff700 1 --2- 172.21.15.42:0/1609616935 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6ba008df80 0x7f6ba009f2e0 secure :-1 s=THROTTLE_DONE pgs=758 cs=0 l=1 rx=0x7f6b8c002f30 tx=0x7f6b8c00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:25.029 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.022+0000 7f6b9f7fe700 1 -- 172.21.15.42:0/1609616935 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6b8c0112b0 con 0x7f6ba008df80 2019-12-05T06:52:25.029 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.022+0000 7f6b9ffff700 1 --2- 172.21.15.42:0/1609616935 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6ba008df80 0x7f6ba009f2e0 secure :-1 s=THROTTLE_DONE pgs=758 cs=0 l=1 rx=0x7f6b8c002f30 tx=0x7f6b8c00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:25.029 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.022+0000 7f6b9f7fe700 1 -- 172.21.15.42:0/1609616935 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b8c008bc0 con 0x7f6ba008df80 2019-12-05T06:52:25.029 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.022+0000 7f6b9ffff700 1 --2- 172.21.15.42:0/1609616935 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6ba008df80 0x7f6ba009f2e0 secure :-1 s=THROTTLE_DONE pgs=758 cs=0 l=1 rx=0x7f6b8c002f30 tx=0x7f6b8c00fbf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:25.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.022+0000 7f6b9f7fe700 1 -- 172.21.15.42:0/1609616935 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6b8c011890 con 0x7f6ba008df80 2019-12-05T06:52:25.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.022+0000 7f6ba673c700 1 -- 172.21.15.42:0/1609616935 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6ba008df80 msgr2=0x7f6ba009f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:52:25.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.022+0000 7f6ba673c700 1 --2- 172.21.15.42:0/1609616935 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6ba008df80 0x7f6ba009f2e0 secure :-1 s=READY pgs=758 cs=0 l=1 rx=0x7f6b8c002f30 tx=0x7f6b8c00fbf0).stop 2019-12-05T06:52:25.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.022+0000 7f6ba673c700 1 -- 172.21.15.42:0/1609616935 shutdown_connections 2019-12-05T06:52:25.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.022+0000 7f6ba673c700 1 --2- 172.21.15.42:0/1609616935 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6ba008df80 0x7f6ba009f2e0 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:52:25.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.022+0000 7f6ba673c700 1 -- 172.21.15.42:0/1609616935 shutdown_connections 2019-12-05T06:52:25.030 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.022+0000 7f6ba673c700 1 -- 172.21.15.42:0/1609616935 wait complete. 2019-12-05T06:52:25.031 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.022+0000 7f6ba673c700 1 -- 172.21.15.42:0/1609616935 >> 172.21.15.42:0/1609616935 conn(0x7f6ba00aabe0 msgr2=0x7f6ba00a6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:52:25.033 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.026+0000 7f6ba673c700 1 Processor -- start 2019-12-05T06:52:25.033 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.026+0000 7f6ba673c700 1 -- start start 2019-12-05T06:52:25.033 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.026+0000 7f6ba673c700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f6ba007b090 0x7f6ba00a30b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:52:25.033 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.026+0000 7f6ba673c700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6ba0143810 0x7f6ba0143c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:52:25.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.026+0000 7f6ba673c700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ba00a3600 con 0x7f6ba0122160 2019-12-05T06:52:25.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.026+0000 7f6ba673c700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ba01443c0 con 0x7f6ba007b090 2019-12-05T06:52:25.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.026+0000 7f6ba673c700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ba0144660 con 0x7f6ba0143810 2019-12-05T06:52:25.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.026+0000 7f6b9ffff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6ba0143810 0x7f6ba0143c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:52:25.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.026+0000 7f6b9ffff700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6ba0143810 0x7f6ba0143c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:39558/0 (socket says 172.21.15.42:39558) 2019-12-05T06:52:25.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.026+0000 7f6ba4cd9700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f6ba007b090 0x7f6ba00a30b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:52:25.034 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.026+0000 7f6b9ffff700 1 -- 172.21.15.42:0/2582955420 learned_addr learned my addr 172.21.15.42:0/2582955420 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:52:25.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.026+0000 7f6b9ffff700 1 -- 172.21.15.42:0/2582955420 >> v2:172.21.15.40:3300/0 conn(0x7f6ba007b090 msgr2=0x7f6ba00a30b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:52:25.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.026+0000 7f6b9ffff700 1 --2- 172.21.15.42:0/2582955420 >> v2:172.21.15.40:3300/0 conn(0x7f6ba007b090 0x7f6ba00a30b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:52:25.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.026+0000 7f6b9ffff700 1 -- 172.21.15.42:0/2582955420 >> v1:172.21.15.42:6789/0 conn(0x7f6ba0122160 legacy=0x7f6ba0122590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:52:25.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.026+0000 7f6b9ffff700 1 -- 172.21.15.42:0/2582955420 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ba0145280 con 0x7f6ba0143810 2019-12-05T06:52:25.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.026+0000 7f6b9ffff700 1 --2- 172.21.15.42:0/2582955420 >> v2:172.21.15.42:3300/0 conn(0x7f6ba0143810 0x7f6ba0143c40 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f6b8c002f30 tx=0x7f6b8c008bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:25.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.026+0000 7f6b9ffff700 1 --2- 172.21.15.42:0/2582955420 >> v2:172.21.15.42:3300/0 conn(0x7f6ba0143810 0x7f6ba0143c40 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f6b8c002f30 tx=0x7f6b8c008bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:25.035 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.026+0000 7f6b9ffff700 1 --2- 172.21.15.42:0/2582955420 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6ba0143810 0x7f6ba0143c40 secure :-1 s=READY pgs=3406 cs=0 l=1 rx=0x7f6b8c002f30 tx=0x7f6b8c008bc0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:52:25.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.026+0000 7f6b9ffff700 1 --2- 172.21.15.42:0/2582955420 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6ba0143810 0x7f6ba0143c40 secure :-1 s=THROTTLE_DONE pgs=3406 cs=0 l=1 rx=0x7f6b8c002f30 tx=0x7f6b8c008bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:25.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.026+0000 7f6b9d7fa700 1 -- 172.21.15.42:0/2582955420 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6b8c011d80 con 0x7f6ba0143810 2019-12-05T06:52:25.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.026+0000 7f6b9ffff700 1 --2- 172.21.15.42:0/2582955420 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6ba0143810 0x7f6ba0143c40 secure :-1 s=THROTTLE_DONE pgs=3406 cs=0 l=1 rx=0x7f6b8c002f30 tx=0x7f6b8c008bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:25.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.026+0000 7f6b9ffff700 1 --2- 172.21.15.42:0/2582955420 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6ba0143810 0x7f6ba0143c40 secure :-1 s=THROTTLE_DONE pgs=3406 cs=0 l=1 rx=0x7f6b8c002f30 tx=0x7f6b8c008bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:25.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.026+0000 7f6b9d7fa700 1 -- 172.21.15.42:0/2582955420 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b8c0114b0 con 0x7f6ba0143810 2019-12-05T06:52:25.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.026+0000 7f6ba673c700 1 -- 172.21.15.42:0/2582955420 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ba008eb70 con 0x7f6ba0143810 2019-12-05T06:52:25.036 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.026+0000 7f6b9d7fa700 1 -- 172.21.15.42:0/2582955420 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6b8c016570 con 0x7f6ba0143810 2019-12-05T06:52:25.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.026+0000 7f6ba673c700 1 -- 172.21.15.42:0/2582955420 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ba008f120 con 0x7f6ba0143810 2019-12-05T06:52:25.037 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.030+0000 7f6b9ffff700 1 --2- 172.21.15.42:0/2582955420 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6ba0143810 0x7f6ba0143c40 secure :-1 s=THROTTLE_DONE pgs=3406 cs=0 l=1 rx=0x7f6b8c002f30 tx=0x7f6b8c008bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:25.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.030+0000 7f6b9d7fa700 1 -- 172.21.15.42:0/2582955420 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f6b8c007170 con 0x7f6ba0143810 2019-12-05T06:52:25.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.030+0000 7f6b9ffff700 1 --2- 172.21.15.42:0/2582955420 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6ba0143810 0x7f6ba0143c40 secure :-1 s=THROTTLE_DONE pgs=3406 cs=0 l=1 rx=0x7f6b8c002f30 tx=0x7f6b8c008bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:25.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.030+0000 7f6b9d7fa700 1 --2- 172.21.15.42:0/2582955420 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6b8003acf0 0x7f6b8003d200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:52:25.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.030+0000 7f6b9d7fa700 1 -- 172.21.15.42:0/2582955420 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 12337+0+0 (secure 0 0 0) 0x7f6b8c04d620 con 0x7f6ba0143810 2019-12-05T06:52:25.038 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.030+0000 7f6ba4cd9700 1 --2- 172.21.15.42:0/2582955420 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6b8003acf0 0x7f6b8003d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:52:25.039 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.030+0000 7f6ba4cd9700 1 --2- 172.21.15.42:0/2582955420 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6b8003acf0 0x7f6b8003d200 crc :-1 s=READY pgs=1427 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:52:25.039 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.034+0000 7f6ba673c700 1 -- 172.21.15.42:0/2582955420 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ba005d110 con 0x7f6ba0143810 2019-12-05T06:52:25.045 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.042+0000 7f6b9ffff700 1 --2- 172.21.15.42:0/2582955420 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6ba0143810 0x7f6ba0143c40 secure :-1 s=THROTTLE_DONE pgs=3406 cs=0 l=1 rx=0x7f6b8c002f30 tx=0x7f6b8c008bc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:25.045 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.042+0000 7f6b9d7fa700 1 -- 172.21.15.42:0/2582955420 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f6b8c0346a0 con 0x7f6ba0143810 2019-12-05T06:52:25.217 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.214+0000 7f6ba673c700 1 -- 172.21.15.42:0/2582955420 --> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f6ba005b8c0 con 0x7f6b8003acf0 2019-12-05T06:52:25.220 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.218+0000 7f6b9d7fa700 1 -- 172.21.15.42:0/2582955420 <== mgr.4108 v2:172.21.15.42:6800/11930 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+92121 (crc 0 0 0) 0x7f6ba005b8c0 con 0x7f6b8003acf0 2019-12-05T06:52:25.224 INFO:teuthology.orchestra.run.smithi042.stderr:dumped all 2019-12-05T06:52:25.226 INFO:teuthology.orchestra.run.smithi042.stdout: 2019-12-05T06:52:25.228 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.222+0000 7f6ba673c700 1 -- 172.21.15.42:0/2582955420 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6b8003acf0 msgr2=0x7f6b8003d200 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:52:25.228 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.222+0000 7f6ba673c700 1 --2- 172.21.15.42:0/2582955420 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6b8003acf0 0x7f6b8003d200 crc :-1 s=READY pgs=1427 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:52:25.228 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.222+0000 7f6ba673c700 1 -- 172.21.15.42:0/2582955420 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6ba0143810 msgr2=0x7f6ba0143c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:52:25.228 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.222+0000 7f6ba673c700 1 --2- 172.21.15.42:0/2582955420 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6ba0143810 0x7f6ba0143c40 secure :-1 s=READY pgs=3406 cs=0 l=1 rx=0x7f6b8c002f30 tx=0x7f6b8c008bc0).stop 2019-12-05T06:52:25.228 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.222+0000 7f6ba673c700 1 -- 172.21.15.42:0/2582955420 shutdown_connections 2019-12-05T06:52:25.229 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.222+0000 7f6ba673c700 1 --2- 172.21.15.42:0/2582955420 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f6b8003acf0 0x7f6b8003d200 unknown :-1 s=CLOSED pgs=1427 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:52:25.229 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.222+0000 7f6ba673c700 1 --2- 172.21.15.42:0/2582955420 >> v2:172.21.15.40:3300/0 conn(0x7f6ba007b090 0x7f6ba00a30b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:52:25.229 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.222+0000 7f6ba673c700 1 --2- 172.21.15.42:0/2582955420 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6ba0143810 0x7f6ba0143c40 unknown :-1 s=CLOSED pgs=3406 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:52:25.229 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.222+0000 7f6ba673c700 1 -- 172.21.15.42:0/2582955420 shutdown_connections 2019-12-05T06:52:25.229 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.222+0000 7f6ba673c700 1 -- 172.21.15.42:0/2582955420 wait complete. 2019-12-05T06:52:25.229 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.226+0000 7f6ba673c700 1 -- 172.21.15.42:0/2582955420 >> 172.21.15.42:0/2582955420 conn(0x7f6ba00a96f0 msgr2=0x7f6ba007e570 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:52:25.240 INFO:teuthology.orchestra.run.smithi042.stdout:{"pg_ready":true,"pg_map":{"version":3399,"stamp":"2019-12-05T06:52:23.630008+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":409363217,"num_objects":153,"num_object_clones":2,"num_object_copies":312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":153,"num_whiteouts":0,"num_read":9494,"num_read_kb":12806383,"num_write":49976,"num_write_kb":1021704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":235566,"num_omap_keys":385,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39857,"ondisk_log_size":39857,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":9202368,"kb_used_data":813696,"kb_used_omap":0,"kb_used_meta":8388608,"kb_avail":745772352,"statfs":{"total":773094113280,"available":763670888448,"internally_reserved":8589934592,"allocated":833224704,"data_stored":827210444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8589934592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003749"},"pg_stats":[{"pgid":"3.7","version":"126'6396","reported_seq":"8192","reported_epoch":"129","state":"active+clean","last_fresh":"2019-12-05T06:51:50.444739+0000","last_change":"2019-12-05T06:51:47.140998+0000","last_active":"2019-12-05T06:51:50.444739+0000","last_peered":"2019-12-05T06:51:50.444739+0000","last_clean":"2019-12-05T06:51:50.444739+0000","last_became_active":"2019-12-05T05:03:35.332717+0000","last_became_peered":"2019-12-05T05:03:35.332717+0000","last_unstale":"2019-12-05T06:51:50.444739+0000","last_undegraded":"2019-12-05T06:51:50.444739+0000","last_fullsized":"2019-12-05T06:51:50.444739+0000","mapping_epoch":21,"log_start":"118'3311","ondisk_log_start":"118'3311","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'6396","last_scrub_stamp":"2019-12-05T06:51:47.140916+0000","last_deep_scrub":"126'6396","last_deep_scrub_stamp":"2019-12-05T06:51:47.140916+0000","last_clean_scrub_stamp":"2019-12-05T06:51:47.140916+0000","log_size":3085,"ondisk_log_size":3085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":245,"num_read_kb":1968,"num_write":4375,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7,0],"acting":[3,5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":"124","reported_epoch":"128","state":"active+clean","last_fresh":"2019-12-05T06:51:44.867884+0000","last_change":"2019-12-05T06:51:44.867884+0000","last_active":"2019-12-05T06:51:44.867884+0000","last_peered":"2019-12-05T06:51:44.867884+0000","last_clean":"2019-12-05T06:51:44.867884+0000","last_became_active":"2019-12-05T05:03:29.184431+0000","last_became_peered":"2019-12-05T05:03:29.184431+0000","last_unstale":"2019-12-05T06:51:44.867884+0000","last_undegraded":"2019-12-05T06:51:44.867884+0000","last_fullsized":"2019-12-05T06:51:44.867884+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T06:51:44.867695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T06:51:44.867695+0000","last_clean_scrub_stamp":"2019-12-05T06:51:44.867695+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"128'2461","reported_seq":"3033","reported_epoch":"128","state":"active+clean","last_fresh":"2019-12-05T06:51:49.340434+0000","last_change":"2019-12-05T06:51:49.340434+0000","last_active":"2019-12-05T06:51:49.340434+0000","last_peered":"2019-12-05T06:51:49.340434+0000","last_clean":"2019-12-05T06:51:49.340434+0000","last_became_active":"2019-12-05T05:03:32.224827+0000","last_became_peered":"2019-12-05T05:03:32.224827+0000","last_unstale":"2019-12-05T06:51:49.340434+0000","last_undegraded":"2019-12-05T06:51:49.340434+0000","last_fullsized":"2019-12-05T06:51:49.340434+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"128'2461","last_scrub_stamp":"2019-12-05T06:51:49.340234+0000","last_deep_scrub":"128'2461","last_deep_scrub_stamp":"2019-12-05T06:51:49.340234+0000","last_clean_scrub_stamp":"2019-12-05T06:51:49.340234+0000","log_size":2461,"ondisk_log_size":2461,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":67108864,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":679,"num_read_kb":1201860,"num_write":2589,"num_write_kb":72175,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25462,"num_omap_keys":51,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"124'6219","reported_seq":"8106","reported_epoch":"129","state":"active+clean","last_fresh":"2019-12-05T06:51:59.608061+0000","last_change":"2019-12-05T06:51:59.608206+0000","last_active":"2019-12-05T06:51:59.608061+0000","last_peered":"2019-12-05T06:51:59.608061+0000","last_clean":"2019-12-05T06:51:59.608061+0000","last_became_active":"2019-12-05T05:03:35.332648+0000","last_became_peered":"2019-12-05T05:03:35.332648+0000","last_unstale":"2019-12-05T06:51:59.608061+0000","last_undegraded":"2019-12-05T06:51:59.608061+0000","last_fullsized":"2019-12-05T06:51:59.608061+0000","mapping_epoch":21,"log_start":"117'3205","ondisk_log_start":"117'3205","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'6219","last_scrub_stamp":"2019-12-05T06:51:59.608055+0000","last_deep_scrub":"124'6219","last_deep_scrub_stamp":"2019-12-05T06:51:59.608055+0000","last_clean_scrub_stamp":"2019-12-05T06:51:59.608055+0000","log_size":3014,"ondisk_log_size":3014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":233,"num_read_kb":6040,"num_write":4390,"num_write_kb":51866,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,7],"acting":[0,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":"124","reported_epoch":"128","state":"active+clean","last_fresh":"2019-12-05T06:51:44.038732+0000","last_change":"2019-12-05T06:51:44.038732+0000","last_active":"2019-12-05T06:51:44.038732+0000","last_peered":"2019-12-05T06:51:44.038732+0000","last_clean":"2019-12-05T06:51:44.038732+0000","last_became_active":"2019-12-05T05:03:29.182409+0000","last_became_peered":"2019-12-05T05:03:29.182409+0000","last_unstale":"2019-12-05T06:51:44.038732+0000","last_undegraded":"2019-12-05T06:51:44.038732+0000","last_fullsized":"2019-12-05T06:51:44.038732+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T06:51:44.038545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T06:51:44.038545+0000","last_clean_scrub_stamp":"2019-12-05T06:51:44.038545+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"129'1454","reported_seq":"2112","reported_epoch":"129","state":"active+clean","last_fresh":"2019-12-05T06:51:55.902173+0000","last_change":"2019-12-05T06:51:55.902254+0000","last_active":"2019-12-05T06:51:55.902173+0000","last_peered":"2019-12-05T06:51:55.902173+0000","last_clean":"2019-12-05T06:51:55.902173+0000","last_became_active":"2019-12-05T05:03:32.227690+0000","last_became_peered":"2019-12-05T05:03:32.227690+0000","last_unstale":"2019-12-05T06:51:55.902173+0000","last_undegraded":"2019-12-05T06:51:55.902173+0000","last_fullsized":"2019-12-05T06:51:55.902173+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'1454","last_scrub_stamp":"2019-12-05T06:51:55.902169+0000","last_deep_scrub":"129'1454","last_deep_scrub_stamp":"2019-12-05T06:51:55.902169+0000","last_clean_scrub_stamp":"2019-12-05T06:51:55.902169+0000","log_size":1454,"ondisk_log_size":1454,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":25166390,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":832,"num_read_kb":914148,"num_write":1232,"num_write_kb":29663,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":65753,"num_omap_keys":49,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"126'6687","reported_seq":"8466","reported_epoch":"129","state":"active+clean","last_fresh":"2019-12-05T06:51:58.649365+0000","last_change":"2019-12-05T06:51:58.649654+0000","last_active":"2019-12-05T06:51:58.649365+0000","last_peered":"2019-12-05T06:51:58.649365+0000","last_clean":"2019-12-05T06:51:58.649365+0000","last_became_active":"2019-12-05T05:03:35.266235+0000","last_became_peered":"2019-12-05T05:03:35.266235+0000","last_unstale":"2019-12-05T06:51:58.649365+0000","last_undegraded":"2019-12-05T06:51:58.649365+0000","last_fullsized":"2019-12-05T06:51:58.649365+0000","mapping_epoch":21,"log_start":"118'3617","ondisk_log_start":"118'3617","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'6687","last_scrub_stamp":"2019-12-05T06:51:58.649342+0000","last_deep_scrub":"126'6687","last_deep_scrub_stamp":"2019-12-05T06:51:58.649342+0000","last_clean_scrub_stamp":"2019-12-05T06:51:58.649342+0000","log_size":3070,"ondisk_log_size":3070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":15,"num_objects":3,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":240,"num_read_kb":18617,"num_write":4624,"num_write_kb":74599,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3,5],"acting":[0,4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":"124","reported_epoch":"128","state":"active+clean","last_fresh":"2019-12-05T06:51:44.155842+0000","last_change":"2019-12-05T06:51:44.155842+0000","last_active":"2019-12-05T06:51:44.155842+0000","last_peered":"2019-12-05T06:51:44.155842+0000","last_clean":"2019-12-05T06:51:44.155842+0000","last_became_active":"2019-12-05T05:03:29.180832+0000","last_became_peered":"2019-12-05T05:03:29.180832+0000","last_unstale":"2019-12-05T06:51:44.155842+0000","last_undegraded":"2019-12-05T06:51:44.155842+0000","last_fullsized":"2019-12-05T06:51:44.155842+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T06:51:44.155709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T06:51:44.155709+0000","last_clean_scrub_stamp":"2019-12-05T06:51:44.155709+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"129'2432","reported_seq":"3450","reported_epoch":"129","state":"active+clean","last_fresh":"2019-12-05T06:51:51.936396+0000","last_change":"2019-12-05T06:51:51.936467+0000","last_active":"2019-12-05T06:51:51.936396+0000","last_peered":"2019-12-05T06:51:51.936396+0000","last_clean":"2019-12-05T06:51:51.936396+0000","last_became_active":"2019-12-05T05:03:32.226895+0000","last_became_peered":"2019-12-05T05:03:32.226895+0000","last_unstale":"2019-12-05T06:51:51.936396+0000","last_undegraded":"2019-12-05T06:51:51.936396+0000","last_fullsized":"2019-12-05T06:51:51.936396+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'2432","last_scrub_stamp":"2019-12-05T06:51:51.936389+0000","last_deep_scrub":"129'2432","last_deep_scrub_stamp":"2019-12-05T06:51:51.936389+0000","last_clean_scrub_stamp":"2019-12-05T06:51:51.936389+0000","log_size":2432,"ondisk_log_size":2432,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46377447,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":1503,"num_read_kb":2307020,"num_write":2206,"num_write_kb":63089,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":11,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":49583,"num_omap_keys":93,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"124'6690","reported_seq":"8478","reported_epoch":"129","state":"active+clean","last_fresh":"2019-12-05T06:51:51.439679+0000","last_change":"2019-12-05T06:51:51.439765+0000","last_active":"2019-12-05T06:51:51.439679+0000","last_peered":"2019-12-05T06:51:51.439679+0000","last_clean":"2019-12-05T06:51:51.439679+0000","last_became_active":"2019-12-05T05:03:35.627352+0000","last_became_peered":"2019-12-05T05:03:35.627352+0000","last_unstale":"2019-12-05T06:51:51.439679+0000","last_undegraded":"2019-12-05T06:51:51.439679+0000","last_fullsized":"2019-12-05T06:51:51.439679+0000","mapping_epoch":21,"log_start":"118'3612","ondisk_log_start":"118'3612","created":21,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'6690","last_scrub_stamp":"2019-12-05T06:51:51.439674+0000","last_deep_scrub":"124'6690","last_deep_scrub_stamp":"2019-12-05T06:51:51.439674+0000","last_clean_scrub_stamp":"2019-12-05T06:51:51.439674+0000","log_size":3078,"ondisk_log_size":3078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":249,"num_read_kb":14990,"num_write":4595,"num_write_kb":60350,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6,5],"acting":[1,2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":"124","reported_epoch":"128","state":"active+clean","last_fresh":"2019-12-05T06:51:47.723043+0000","last_change":"2019-12-05T06:51:47.723043+0000","last_active":"2019-12-05T06:51:47.723043+0000","last_peered":"2019-12-05T06:51:47.723043+0000","last_clean":"2019-12-05T06:51:47.723043+0000","last_became_active":"2019-12-05T05:03:29.407108+0000","last_became_peered":"2019-12-05T05:03:29.407108+0000","last_unstale":"2019-12-05T06:51:47.723043+0000","last_undegraded":"2019-12-05T06:51:47.723043+0000","last_fullsized":"2019-12-05T06:51:47.723043+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T06:51:47.722916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T06:51:47.722916+0000","last_clean_scrub_stamp":"2019-12-05T06:51:47.722916+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"128'1584","reported_seq":"2622","reported_epoch":"128","state":"active+clean","last_fresh":"2019-12-05T06:51:47.524078+0000","last_change":"2019-12-05T06:51:47.524078+0000","last_active":"2019-12-05T06:51:47.524078+0000","last_peered":"2019-12-05T06:51:47.524078+0000","last_clean":"2019-12-05T06:51:47.524078+0000","last_became_active":"2019-12-05T05:03:32.635937+0000","last_became_peered":"2019-12-05T05:03:32.635937+0000","last_unstale":"2019-12-05T06:51:47.524078+0000","last_undegraded":"2019-12-05T06:51:47.524078+0000","last_fullsized":"2019-12-05T06:51:47.524078+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"128'1584","last_scrub_stamp":"2019-12-05T06:51:47.523996+0000","last_deep_scrub":"128'1584","last_deep_scrub_stamp":"2019-12-05T06:51:47.523996+0000","last_clean_scrub_stamp":"2019-12-05T06:51:47.523996+0000","log_size":1584,"ondisk_log_size":1584,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":35827812,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":920,"num_read_kb":599871,"num_write":1407,"num_write_kb":35966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":12000,"num_omap_keys":24,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"126'6396","reported_seq":"8256","reported_epoch":"129","state":"active+clean","last_fresh":"2019-12-05T06:51:50.445104+0000","last_change":"2019-12-05T06:51:45.089230+0000","last_active":"2019-12-05T06:51:50.445104+0000","last_peered":"2019-12-05T06:51:50.445104+0000","last_clean":"2019-12-05T06:51:50.445104+0000","last_became_active":"2019-12-05T05:03:36.314006+0000","last_became_peered":"2019-12-05T05:03:36.314006+0000","last_unstale":"2019-12-05T06:51:50.445104+0000","last_undegraded":"2019-12-05T06:51:50.445104+0000","last_fullsized":"2019-12-05T06:51:50.445104+0000","mapping_epoch":21,"log_start":"117'3307","ondisk_log_start":"117'3307","created":21,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'6396","last_scrub_stamp":"2019-12-05T06:51:45.089137+0000","last_deep_scrub":"126'6396","last_deep_scrub_stamp":"2019-12-05T06:51:45.089137+0000","last_clean_scrub_stamp":"2019-12-05T06:51:45.089137+0000","log_size":3089,"ondisk_log_size":3089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":253,"num_read_kb":6039,"num_write":4487,"num_write_kb":82964,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6,4],"acting":[3,5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"124","reported_epoch":"128","state":"active+clean","last_fresh":"2019-12-05T06:51:44.789603+0000","last_change":"2019-12-05T06:51:44.789603+0000","last_active":"2019-12-05T06:51:44.789603+0000","last_peered":"2019-12-05T06:51:44.789603+0000","last_clean":"2019-12-05T06:51:44.789603+0000","last_became_active":"2019-12-05T05:03:29.406770+0000","last_became_peered":"2019-12-05T05:03:29.406770+0000","last_unstale":"2019-12-05T06:51:44.789603+0000","last_undegraded":"2019-12-05T06:51:44.789603+0000","last_fullsized":"2019-12-05T06:51:44.789603+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T06:51:44.789407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T06:51:44.789407+0000","last_clean_scrub_stamp":"2019-12-05T06:51:44.789407+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"129'1804","reported_seq":"2628","reported_epoch":"129","state":"active+clean","last_fresh":"2019-12-05T06:52:01.880449+0000","last_change":"2019-12-05T06:52:01.880518+0000","last_active":"2019-12-05T06:52:01.880449+0000","last_peered":"2019-12-05T06:52:01.880449+0000","last_clean":"2019-12-05T06:52:01.880449+0000","last_became_active":"2019-12-05T05:03:32.227052+0000","last_became_peered":"2019-12-05T05:03:32.227052+0000","last_unstale":"2019-12-05T06:52:01.880449+0000","last_undegraded":"2019-12-05T06:52:01.880449+0000","last_fullsized":"2019-12-05T06:52:01.880449+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'1804","last_scrub_stamp":"2019-12-05T06:52:01.880444+0000","last_deep_scrub":"129'1804","last_deep_scrub_stamp":"2019-12-05T06:52:01.880444+0000","last_clean_scrub_stamp":"2019-12-05T06:52:01.880444+0000","log_size":1804,"ondisk_log_size":1804,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46137378,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":922,"num_read_kb":1904128,"num_write":1592,"num_write_kb":62385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":17534,"num_omap_keys":36,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"124'6539","reported_seq":"8362","reported_epoch":"129","state":"active+clean","last_fresh":"2019-12-05T06:51:50.713422+0000","last_change":"2019-12-05T06:51:46.818890+0000","last_active":"2019-12-05T06:51:50.713422+0000","last_peered":"2019-12-05T06:51:50.713422+0000","last_clean":"2019-12-05T06:51:50.713422+0000","last_became_active":"2019-12-05T05:03:35.732405+0000","last_became_peered":"2019-12-05T05:03:35.732405+0000","last_unstale":"2019-12-05T06:51:50.713422+0000","last_undegraded":"2019-12-05T06:51:50.713422+0000","last_fullsized":"2019-12-05T06:51:50.713422+0000","mapping_epoch":21,"log_start":"118'3508","ondisk_log_start":"118'3508","created":21,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'6539","last_scrub_stamp":"2019-12-05T06:51:46.818781+0000","last_deep_scrub":"124'6539","last_deep_scrub_stamp":"2019-12-05T06:51:46.818781+0000","last_clean_scrub_stamp":"2019-12-05T06:51:46.818781+0000","log_size":3031,"ondisk_log_size":3031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":262,"num_read_kb":13412,"num_write":4550,"num_write_kb":85060,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0,6],"acting":[4,5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":"124","reported_epoch":"128","state":"active+clean","last_fresh":"2019-12-05T06:51:45.824965+0000","last_change":"2019-12-05T06:51:45.824965+0000","last_active":"2019-12-05T06:51:45.824965+0000","last_peered":"2019-12-05T06:51:45.824965+0000","last_clean":"2019-12-05T06:51:45.824965+0000","last_became_active":"2019-12-05T05:03:29.406903+0000","last_became_peered":"2019-12-05T05:03:29.406903+0000","last_unstale":"2019-12-05T06:51:45.824965+0000","last_undegraded":"2019-12-05T06:51:45.824965+0000","last_fullsized":"2019-12-05T06:51:45.824965+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T06:51:45.824777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T06:51:45.824777+0000","last_clean_scrub_stamp":"2019-12-05T06:51:45.824777+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"129'1354","reported_seq":"2035","reported_epoch":"129","state":"active+clean","last_fresh":"2019-12-05T06:51:59.866232+0000","last_change":"2019-12-05T06:51:59.866302+0000","last_active":"2019-12-05T06:51:59.866232+0000","last_peered":"2019-12-05T06:51:59.866232+0000","last_clean":"2019-12-05T06:51:59.866232+0000","last_became_active":"2019-12-05T05:03:32.227050+0000","last_became_peered":"2019-12-05T05:03:32.227050+0000","last_unstale":"2019-12-05T06:51:59.866232+0000","last_undegraded":"2019-12-05T06:51:59.866232+0000","last_fullsized":"2019-12-05T06:51:59.866232+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'1354","last_scrub_stamp":"2019-12-05T06:51:59.866227+0000","last_deep_scrub":"129'1354","last_deep_scrub_stamp":"2019-12-05T06:51:59.866227+0000","last_clean_scrub_stamp":"2019-12-05T06:51:59.866227+0000","log_size":1354,"ondisk_log_size":1354,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":67108864,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":576,"num_read_kb":2216360,"num_write":1199,"num_write_kb":74709,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25734,"num_omap_keys":53,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"124'6466","reported_seq":"8256","reported_epoch":"129","state":"active+clean","last_fresh":"2019-12-05T06:51:53.151766+0000","last_change":"2019-12-05T06:51:53.151861+0000","last_active":"2019-12-05T06:51:53.151766+0000","last_peered":"2019-12-05T06:51:53.151766+0000","last_clean":"2019-12-05T06:51:53.151766+0000","last_became_active":"2019-12-05T05:03:35.266868+0000","last_became_peered":"2019-12-05T05:03:35.266868+0000","last_unstale":"2019-12-05T06:51:53.151766+0000","last_undegraded":"2019-12-05T06:51:53.151766+0000","last_fullsized":"2019-12-05T06:51:53.151766+0000","mapping_epoch":21,"log_start":"118'3409","ondisk_log_start":"118'3409","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'6466","last_scrub_stamp":"2019-12-05T06:51:53.151758+0000","last_deep_scrub":"124'6466","last_deep_scrub_stamp":"2019-12-05T06:51:53.151758+0000","last_clean_scrub_stamp":"2019-12-05T06:51:53.151758+0000","log_size":3057,"ondisk_log_size":3057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":226,"num_read_kb":10307,"num_write":4503,"num_write_kb":69746,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0,5],"acting":[1,2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":"123","reported_epoch":"128","state":"active+clean","last_fresh":"2019-12-05T06:51:49.713098+0000","last_change":"2019-12-05T06:51:49.713098+0000","last_active":"2019-12-05T06:51:49.713098+0000","last_peered":"2019-12-05T06:51:49.713098+0000","last_clean":"2019-12-05T06:51:49.713098+0000","last_became_active":"2019-12-05T05:03:29.406857+0000","last_became_peered":"2019-12-05T05:03:29.406857+0000","last_unstale":"2019-12-05T06:51:49.713098+0000","last_undegraded":"2019-12-05T06:51:49.713098+0000","last_fullsized":"2019-12-05T06:51:49.713098+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T06:51:49.712908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T06:51:49.712908+0000","last_clean_scrub_stamp":"2019-12-05T06:51:49.712908+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"129'2908","reported_seq":"3880","reported_epoch":"129","state":"active+clean","last_fresh":"2019-12-05T06:51:52.982864+0000","last_change":"2019-12-05T06:51:52.983093+0000","last_active":"2019-12-05T06:51:52.982864+0000","last_peered":"2019-12-05T06:51:52.982864+0000","last_clean":"2019-12-05T06:51:52.982864+0000","last_became_active":"2019-12-05T05:03:32.479565+0000","last_became_peered":"2019-12-05T05:03:32.479565+0000","last_unstale":"2019-12-05T06:51:52.982864+0000","last_undegraded":"2019-12-05T06:51:52.982864+0000","last_fullsized":"2019-12-05T06:51:52.982864+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'2908","last_scrub_stamp":"2019-12-05T06:51:52.982843+0000","last_deep_scrub":"129'2908","last_deep_scrub_stamp":"2019-12-05T06:51:52.982843+0000","last_clean_scrub_stamp":"2019-12-05T06:51:52.982843+0000","log_size":2908,"ondisk_log_size":2908,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":83887077,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":1148,"num_read_kb":2218837,"num_write":2638,"num_write_kb":91697,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":24000,"num_omap_keys":48,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.5","version":"124'6269","reported_seq":"8152","reported_epoch":"129","state":"active+clean","last_fresh":"2019-12-05T06:52:03.752874+0000","last_change":"2019-12-05T06:52:03.753027+0000","last_active":"2019-12-05T06:52:03.752874+0000","last_peered":"2019-12-05T06:52:03.752874+0000","last_clean":"2019-12-05T06:52:03.752874+0000","last_became_active":"2019-12-05T05:03:35.629300+0000","last_became_peered":"2019-12-05T05:03:35.629300+0000","last_unstale":"2019-12-05T06:52:03.752874+0000","last_undegraded":"2019-12-05T06:52:03.752874+0000","last_fullsized":"2019-12-05T06:52:03.752874+0000","mapping_epoch":21,"log_start":"117'3209","ondisk_log_start":"117'3209","created":21,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'6269","last_scrub_stamp":"2019-12-05T06:52:03.752864+0000","last_deep_scrub":"124'6269","last_deep_scrub_stamp":"2019-12-05T06:52:03.752864+0000","last_clean_scrub_stamp":"2019-12-05T06:52:03.752864+0000","log_size":3060,"ondisk_log_size":3060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":275,"num_read_kb":10198,"num_write":4396,"num_write_kb":76174,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2,6],"acting":[5,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":"123","reported_epoch":"129","state":"active+clean","last_fresh":"2019-12-05T06:51:51.742615+0000","last_change":"2019-12-05T06:51:51.742806+0000","last_active":"2019-12-05T06:51:51.742615+0000","last_peered":"2019-12-05T06:51:51.742615+0000","last_clean":"2019-12-05T06:51:51.742615+0000","last_became_active":"2019-12-05T05:03:29.184716+0000","last_became_peered":"2019-12-05T05:03:29.184716+0000","last_unstale":"2019-12-05T06:51:51.742615+0000","last_undegraded":"2019-12-05T06:51:51.742615+0000","last_fullsized":"2019-12-05T06:51:51.742615+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T06:51:51.742606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T06:51:51.742606+0000","last_clean_scrub_stamp":"2019-12-05T06:51:51.742606+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"128'1376","reported_seq":"2136","reported_epoch":"128","state":"active+clean","last_fresh":"2019-12-05T06:51:48.182277+0000","last_change":"2019-12-05T06:51:48.182277+0000","last_active":"2019-12-05T06:51:48.182277+0000","last_peered":"2019-12-05T06:51:48.182277+0000","last_clean":"2019-12-05T06:51:48.182277+0000","last_became_active":"2019-12-05T05:03:32.476140+0000","last_became_peered":"2019-12-05T05:03:32.476140+0000","last_unstale":"2019-12-05T06:51:48.182277+0000","last_undegraded":"2019-12-05T06:51:48.182277+0000","last_fullsized":"2019-12-05T06:51:48.182277+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"128'1376","last_scrub_stamp":"2019-12-05T06:51:48.182181+0000","last_deep_scrub":"128'1376","last_deep_scrub_stamp":"2019-12-05T06:51:48.182181+0000","last_clean_scrub_stamp":"2019-12-05T06:51:48.182181+0000","log_size":1376,"ondisk_log_size":1376,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":37749370,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":931,"num_read_kb":1362588,"num_write":1193,"num_write_kb":41853,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15500,"num_omap_keys":31,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":8,"stat_sum":{"num_bytes":15,"num_objects":3,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1983,"num_read_kb":81571,"num_write":35920,"num_write_kb":550167,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":49152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24484,"ondisk_log_size":24484,"up":32,"acting":32,"num_store_stats":8},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":409363202,"num_objects":150,"num_object_clones":0,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":150,"num_whiteouts":0,"num_read":7511,"num_read_kb":12724812,"num_write":14056,"num_write_kb":471537,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":235566,"num_omap_keys":385,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":818790400,"data_stored":818726404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":15373,"ondisk_log_size":15373,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576168,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1202144,"kb_used_data":153560,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93169696,"statfs":{"total":96636764160,"available":95405768704,"internally_reserved":1073741824,"allocated":157245440,"data_stored":156485275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Dec 5 06:52:05 2019","interfaces":[{"interface":"back","average":{"1min":0.742,"5min":10.008,"15min":7.445},"min":{"1min":0.209,"5min":0.209,"15min":0.209},"max":{"1min":1.147,"5min":501.337,"15min":825.441},"last":0.850},{"interface":"front","average":{"1min":0.799,"5min":2.631,"15min":1.509},"min":{"1min":0.269,"5min":0.269,"15min":0.269},"max":{"1min":1.391,"5min":150.493,"15min":150.493},"last":1.006}]},{"osd":1,"last update":"Thu Dec 5 06:52:06 2019","interfaces":[{"interface":"back","average":{"1min":41.362,"5min":16.321,"15min":10.124},"min":{"1min":0.324,"5min":0.324,"15min":0.241},"max":{"1min":808.913,"5min":808.913,"15min":808.913},"last":1.152},{"interface":"front","average":{"1min":43.670,"5min":9.498,"15min":8.013},"min":{"1min":0.385,"5min":0.385,"15min":0.322},"max":{"1min":857.617,"5min":857.617,"15min":857.617},"last":0.683}]},{"osd":2,"last update":"Thu Dec 5 06:52:11 2019","interfaces":[{"interface":"back","average":{"1min":1.021,"5min":0.962,"15min":6.762},"min":{"1min":0.457,"5min":0.457,"15min":0.292},"max":{"1min":1.587,"5min":2.133,"15min":982.285},"last":1.094},{"interface":"front","average":{"1min":1.106,"5min":5.990,"15min":10.962},"min":{"1min":0.408,"5min":0.408,"15min":0.272},"max":{"1min":1.562,"5min":419.518,"15min":842.181},"last":0.962}]},{"osd":3,"last update":"Thu Dec 5 06:52:06 2019","interfaces":[{"interface":"back","average":{"1min":47.385,"5min":10.320,"15min":4.632},"min":{"1min":0.380,"5min":0.380,"15min":0.309},"max":{"1min":928.321,"5min":928.321,"15min":928.321},"last":1.296},{"interface":"front","average":{"1min":31.923,"5min":7.321,"15min":3.865},"min":{"1min":0.458,"5min":0.458,"15min":0.384},"max":{"1min":616.362,"5min":616.362,"15min":616.362},"last":1.397}]},{"osd":4,"last update":"Thu Dec 5 06:52:05 2019","interfaces":[{"interface":"back","average":{"1min":35.300,"5min":23.356,"15min":12.995},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":691.616,"5min":865.275,"15min":994.097},"last":0.810},{"interface":"front","average":{"1min":0.894,"5min":1.043,"15min":3.486},"min":{"1min":0.304,"5min":0.304,"15min":0.304},"max":{"1min":1.248,"5min":2.367,"15min":586.973},"last":0.918}]},{"osd":5,"last update":"Thu Dec 5 06:52:05 2019","interfaces":[{"interface":"back","average":{"1min":0.921,"5min":1.100,"15min":1.072},"min":{"1min":0.349,"5min":0.349,"15min":0.349},"max":{"1min":1.487,"5min":2.455,"15min":2.455},"last":1.069},{"interface":"front","average":{"1min":1.127,"5min":1.208,"15min":5.550},"min":{"1min":0.367,"5min":0.367,"15min":0.367},"max":{"1min":1.520,"5min":2.425,"15min":791.026},"last":1.196}]},{"osd":6,"last update":"Thu Dec 5 06:52:05 2019","interfaces":[{"interface":"back","average":{"1min":0.987,"5min":1.085,"15min":3.760},"min":{"1min":0.323,"5min":0.323,"15min":0.323},"max":{"1min":1.648,"5min":2.317,"15min":878.805},"last":1.233},{"interface":"front","average":{"1min":1.175,"5min":5.762,"15min":8.200},"min":{"1min":0.429,"5min":0.429,"15min":0.429},"max":{"1min":1.694,"5min":450.021,"15min":840.923},"last":1.343}]}]},{"osd":6,"up_from":13,"seq":55834576164,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1140460,"kb_used_data":91876,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93231380,"statfs":{"total":96636764160,"available":95468933120,"internally_reserved":1073741824,"allocated":94081024,"data_stored":93329615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Dec 5 06:52:05 2019","interfaces":[{"interface":"back","average":{"1min":0.939,"5min":5.590,"15min":9.439},"min":{"1min":0.230,"5min":0.230,"15min":0.230},"max":{"1min":1.787,"5min":501.477,"15min":986.070},"last":0.660},{"interface":"front","average":{"1min":0.926,"5min":13.784,"15min":12.720},"min":{"1min":0.492,"5min":0.335,"15min":0.298},"max":{"1min":1.827,"5min":639.525,"15min":872.211},"last":0.761}]},{"osd":1,"last update":"Thu Dec 5 06:52:02 2019","interfaces":[{"interface":"back","average":{"1min":0.805,"5min":9.775,"15min":3.788},"min":{"1min":0.378,"5min":0.357,"15min":0.277},"max":{"1min":1.212,"5min":758.181,"15min":758.181},"last":0.857},{"interface":"front","average":{"1min":0.726,"5min":9.019,"15min":6.958},"min":{"1min":0.391,"5min":0.356,"15min":0.322},"max":{"1min":1.060,"5min":501.424,"15min":501.424},"last":1.090}]},{"osd":2,"last update":"Thu Dec 5 06:52:02 2019","interfaces":[{"interface":"back","average":{"1min":0.951,"5min":0.930,"15min":8.853},"min":{"1min":0.273,"5min":0.273,"15min":0.273},"max":{"1min":1.766,"5min":1.766,"15min":775.503},"last":0.996},{"interface":"front","average":{"1min":0.985,"5min":1.004,"15min":0.998},"min":{"1min":0.414,"5min":0.414,"15min":0.325},"max":{"1min":1.804,"5min":1.804,"15min":1.804},"last":0.933}]},{"osd":3,"last update":"Thu Dec 5 06:52:02 2019","interfaces":[{"interface":"back","average":{"1min":0.881,"5min":11.316,"15min":7.356},"min":{"1min":0.339,"5min":0.339,"15min":0.339},"max":{"1min":1.234,"5min":777.425,"15min":777.425},"last":0.887},{"interface":"front","average":{"1min":1.035,"5min":1.106,"15min":1.280},"min":{"1min":0.414,"5min":0.414,"15min":0.414},"max":{"1min":1.550,"5min":1.568,"15min":65.098},"last":1.040}]},{"osd":4,"last update":"Thu Dec 5 06:52:02 2019","interfaces":[{"interface":"back","average":{"1min":0.848,"5min":6.082,"15min":9.543},"min":{"1min":0.367,"5min":0.367,"15min":0.367},"max":{"1min":1.209,"5min":380.358,"15min":929.519},"last":0.819},{"interface":"front","average":{"1min":0.800,"5min":8.893,"15min":5.613},"min":{"1min":0.323,"5min":0.323,"15min":0.323},"max":{"1min":1.404,"5min":843.286,"15min":843.286},"last":1.215}]},{"osd":5,"last update":"Thu Dec 5 06:52:02 2019","interfaces":[{"interface":"back","average":{"1min":1.038,"5min":1.065,"15min":4.099},"min":{"1min":0.364,"5min":0.360,"15min":0.360},"max":{"1min":2.011,"5min":2.011,"15min":879.085},"last":1.173},{"interface":"front","average":{"1min":1.147,"5min":1.184,"15min":3.755},"min":{"1min":0.471,"5min":0.471,"15min":0.471},"max":{"1min":1.743,"5min":1.743,"15min":666.419},"last":1.350}]},{"osd":7,"last update":"Thu Dec 5 06:52:02 2019","interfaces":[{"interface":"back","average":{"1min":1.084,"5min":3.607,"15min":4.394},"min":{"1min":0.429,"5min":0.429,"15min":0.408},"max":{"1min":1.488,"5min":127.801,"15min":629.008},"last":1.147},{"interface":"front","average":{"1min":1.161,"5min":1.230,"15min":3.276},"min":{"1min":0.438,"5min":0.363,"15min":0.363},"max":{"1min":1.671,"5min":1.754,"15min":600.711},"last":1.300}]}]},{"osd":1,"up_from":13,"seq":55834576162,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1263576,"kb_used_data":214992,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93108264,"statfs":{"total":96636764160,"available":95342862336,"internally_reserved":1073741824,"allocated":220151808,"data_stored":219398906,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Dec 5 06:52:06 2019","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.688,"15min":2.496},"min":{"1min":0.223,"5min":0.223,"15min":0.197},"max":{"1min":1.052,"5min":1.373,"15min":381.600},"last":0.518},{"interface":"front","average":{"1min":0.772,"5min":3.106,"15min":6.831},"min":{"1min":0.293,"5min":0.293,"15min":0.226},"max":{"1min":1.576,"5min":266.748,"15min":705.387},"last":0.895}]},{"osd":2,"last update":"Thu Dec 5 06:52:06 2019","interfaces":[{"interface":"back","average":{"1min":1.062,"5min":0.826,"15min":4.869},"min":{"1min":0.355,"5min":0.354,"15min":0.251},"max":{"1min":4.211,"5min":4.211,"15min":806.146},"last":0.651},{"interface":"front","average":{"1min":1.113,"5min":0.820,"15min":8.823},"min":{"1min":0.478,"5min":0.285,"15min":0.245},"max":{"1min":4.226,"5min":4.226,"15min":769.148},"last":0.684}]},{"osd":3,"last update":"Thu Dec 5 06:52:08 2019","interfaces":[{"interface":"back","average":{"1min":54.618,"5min":20.034,"15min":7.277},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":630.113,"5min":884.578,"15min":884.578},"last":0.592},{"interface":"front","average":{"1min":25.232,"5min":5.821,"15min":6.193},"min":{"1min":0.283,"5min":0.283,"15min":0.283},"max":{"1min":501.694,"5min":501.694,"15min":523.567},"last":0.751}]},{"osd":4,"last update":"Thu Dec 5 06:52:14 2019","interfaces":[{"interface":"back","average":{"1min":1.177,"5min":5.877,"15min":9.677},"min":{"1min":0.405,"5min":0.366,"15min":0.366},"max":{"1min":4.135,"5min":584.815,"15min":584.815},"last":0.795},{"interface":"front","average":{"1min":1.004,"5min":7.870,"15min":9.545},"min":{"1min":0.349,"5min":0.275,"15min":0.275},"max":{"1min":4.184,"5min":686.649,"15min":778.089},"last":1.129}]},{"osd":5,"last update":"Thu Dec 5 06:52:06 2019","interfaces":[{"interface":"back","average":{"1min":4.835,"5min":1.794,"15min":4.397},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":77.222,"5min":77.222,"15min":839.045},"last":0.951},{"interface":"front","average":{"1min":1.025,"5min":5.540,"15min":6.456},"min":{"1min":0.343,"5min":0.343,"15min":0.343},"max":{"1min":4.248,"5min":450.346,"15min":598.889},"last":0.825}]},{"osd":6,"last update":"Thu Dec 5 06:52:06 2019","interfaces":[{"interface":"back","average":{"1min":1.282,"5min":7.584,"15min":3.784},"min":{"1min":0.496,"5min":0.496,"15min":0.360},"max":{"1min":4.171,"5min":807.184,"15min":807.184},"last":1.012},{"interface":"front","average":{"1min":1.159,"5min":1.169,"15min":14.631},"min":{"1min":0.517,"5min":0.517,"15min":0.410},"max":{"1min":4.272,"5min":4.272,"15min":848.805},"last":0.990}]},{"osd":7,"last update":"Thu Dec 5 06:52:06 2019","interfaces":[{"interface":"back","average":{"1min":1.203,"5min":3.560,"15min":4.594},"min":{"1min":0.471,"5min":0.471,"15min":0.409},"max":{"1min":4.114,"5min":274.305,"15min":680.492},"last":1.041},{"interface":"front","average":{"1min":34.365,"5min":19.590,"15min":11.593},"min":{"1min":0.549,"5min":0.523,"15min":0.518},"max":{"1min":696.822,"5min":696.822,"15min":792.863},"last":1.077}]}]},{"osd":0,"up_from":13,"seq":55834576164,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1169152,"kb_used_data":120568,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93202688,"statfs":{"total":96636764160,"available":95439552512,"internally_reserved":1073741824,"allocated":123461632,"data_stored":122703096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Dec 5 06:51:48 2019","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":7.575,"15min":4.600},"min":{"1min":0.209,"5min":0.209,"15min":0.209},"max":{"1min":0.869,"5min":587.293,"15min":587.293},"last":0.787},{"interface":"front","average":{"1min":0.770,"5min":10.355,"15min":12.828},"min":{"1min":0.310,"5min":0.261,"15min":0.261},"max":{"1min":1.190,"5min":861.119,"15min":966.609},"last":0.632}]},{"osd":2,"last update":"Thu Dec 5 06:51:48 2019","interfaces":[{"interface":"back","average":{"1min":0.700,"5min":16.877,"15min":9.139},"min":{"1min":0.225,"5min":0.225,"15min":0.225},"max":{"1min":1.166,"5min":933.957,"15min":933.957},"last":0.841},{"interface":"front","average":{"1min":0.759,"5min":11.025,"15min":4.963},"min":{"1min":0.272,"5min":0.272,"15min":0.272},"max":{"1min":1.254,"5min":868.889,"15min":868.889},"last":1.107}]},{"osd":3,"last update":"Thu Dec 5 06:51:48 2019","interfaces":[{"interface":"back","average":{"1min":0.701,"5min":0.735,"15min":0.691},"min":{"1min":0.267,"5min":0.267,"15min":0.255},"max":{"1min":1.151,"5min":1.472,"15min":2.652},"last":0.682},{"interface":"front","average":{"1min":28.670,"5min":14.126,"15min":10.568},"min":{"1min":0.337,"5min":0.322,"15min":0.303},"max":{"1min":447.144,"5min":730.339,"15min":854.568},"last":1.022}]},{"osd":4,"last update":"Thu Dec 5 06:51:48 2019","interfaces":[{"interface":"back","average":{"1min":1.020,"5min":11.009,"15min":9.005},"min":{"1min":0.283,"5min":0.283,"15min":0.283},"max":{"1min":1.671,"5min":492.774,"15min":983.581},"last":1.264},{"interface":"front","average":{"1min":0.924,"5min":15.606,"15min":8.376},"min":{"1min":0.300,"5min":0.300,"15min":0.300},"max":{"1min":1.436,"5min":909.368,"15min":909.368},"last":0.911}]},{"osd":5,"last update":"Thu Dec 5 06:51:30 2019","interfaces":[{"interface":"back","average":{"1min":1.019,"5min":1.049,"15min":6.033},"min":{"1min":0.365,"5min":0.365,"15min":0.365},"max":{"1min":1.529,"5min":1.533,"15min":475.167},"last":0.971},{"interface":"front","average":{"1min":0.990,"5min":27.033,"15min":13.790},"min":{"1min":0.410,"5min":0.410,"15min":0.410},"max":{"1min":1.573,"5min":827.052,"15min":827.052},"last":0.944}]},{"osd":6,"last update":"Thu Dec 5 06:51:49 2019","interfaces":[{"interface":"back","average":{"1min":68.182,"5min":14.510,"15min":9.445},"min":{"1min":0.414,"5min":0.414,"15min":0.322},"max":{"1min":837.447,"5min":837.447,"15min":893.025},"last":1.323},{"interface":"front","average":{"1min":55.264,"5min":12.003,"15min":4.735},"min":{"1min":0.392,"5min":0.392,"15min":0.392},"max":{"1min":867.178,"5min":867.178,"15min":867.178},"last":1.362}]},{"osd":7,"last update":"Thu Dec 5 06:52:13 2019","interfaces":[{"interface":"back","average":{"1min":1.050,"5min":9.339,"15min":5.882},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":1.631,"5min":743.880,"15min":743.880},"last":1.217},{"interface":"front","average":{"1min":1.226,"5min":1.220,"15min":7.025},"min":{"1min":0.510,"5min":0.455,"15min":0.455},"max":{"1min":1.663,"5min":1.742,"15min":920.552},"last":1.171}]}]},{"osd":2,"up_from":13,"seq":55834576160,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1130392,"kb_used_data":81808,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93241448,"statfs":{"total":96636764160,"available":95479242752,"internally_reserved":1073741824,"allocated":83771392,"data_stored":83019551,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Dec 5 06:51:49 2019","interfaces":[{"interface":"back","average":{"1min":0.684,"5min":18.639,"15min":6.640},"min":{"1min":0.215,"5min":0.215,"15min":0.215},"max":{"1min":1.145,"5min":917.307,"15min":917.307},"last":0.554},{"interface":"front","average":{"1min":0.679,"5min":6.311,"15min":2.547},"min":{"1min":0.198,"5min":0.198,"15min":0.198},"max":{"1min":1.138,"5min":423.686,"15min":423.686},"last":1.015}]},{"osd":1,"last update":"Thu Dec 5 06:51:43 2019","interfaces":[{"interface":"back","average":{"1min":0.682,"5min":0.726,"15min":3.704},"min":{"1min":0.272,"5min":0.272,"15min":0.272},"max":{"1min":1.125,"5min":1.316,"15min":501.424},"last":0.665},{"interface":"front","average":{"1min":0.787,"5min":11.001,"15min":9.897},"min":{"1min":0.335,"5min":0.335,"15min":0.234},"max":{"1min":1.250,"5min":814.005,"15min":814.005},"last":0.911}]},{"osd":3,"last update":"Thu Dec 5 06:51:55 2019","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":5.131,"15min":2.197},"min":{"1min":0.306,"5min":0.306,"15min":0.262},"max":{"1min":1.075,"5min":474.268,"15min":474.268},"last":0.832},{"interface":"front","average":{"1min":0.716,"5min":0.820,"15min":0.837},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":1.110,"5min":1.254,"15min":3.965},"last":0.767}]},{"osd":4,"last update":"Thu Dec 5 06:51:43 2019","interfaces":[{"interface":"back","average":{"1min":1.006,"5min":8.466,"15min":3.949},"min":{"1min":0.375,"5min":0.332,"15min":0.323},"max":{"1min":1.513,"5min":632.509,"15min":632.509},"last":1.098},{"interface":"front","average":{"1min":1.084,"5min":1.117,"15min":4.261},"min":{"1min":0.398,"5min":0.398,"15min":0.365},"max":{"1min":1.550,"5min":2.178,"15min":991.418},"last":1.205}]},{"osd":5,"last update":"Thu Dec 5 06:51:43 2019","interfaces":[{"interface":"back","average":{"1min":1.014,"5min":0.950,"15min":5.645},"min":{"1min":0.386,"5min":0.316,"15min":0.316},"max":{"1min":1.451,"5min":1.501,"15min":965.993},"last":1.160},{"interface":"front","average":{"1min":35.947,"5min":18.944,"15min":9.226},"min":{"1min":0.410,"5min":0.381,"15min":0.381},"max":{"1min":698.069,"5min":700.890,"15min":760.855},"last":1.274}]},{"osd":6,"last update":"Thu Dec 5 06:51:50 2019","interfaces":[{"interface":"back","average":{"1min":19.400,"5min":4.800,"15min":6.753},"min":{"1min":0.464,"5min":0.415,"15min":0.344},"max":{"1min":385.427,"5min":385.427,"15min":652.287},"last":1.326},{"interface":"front","average":{"1min":15.332,"5min":18.194,"15min":11.007},"min":{"1min":0.355,"5min":0.355,"15min":0.355},"max":{"1min":300.514,"5min":934.081,"15min":934.081},"last":1.062}]},{"osd":7,"last update":"Thu Dec 5 06:51:43 2019","interfaces":[{"interface":"back","average":{"1min":1.167,"5min":5.526,"15min":2.606},"min":{"1min":0.493,"5min":0.440,"15min":0.440},"max":{"1min":1.618,"5min":419.900,"15min":419.900},"last":1.380},{"interface":"front","average":{"1min":1.079,"5min":1.128,"15min":1.074},"min":{"1min":0.429,"5min":0.408,"15min":0.408},"max":{"1min":1.574,"5min":1.620,"15min":1.794},"last":0.962}]}]},{"osd":3,"up_from":13,"seq":55834576163,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1085344,"kb_used_data":36760,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93286496,"statfs":{"total":96636764160,"available":95525371904,"internally_reserved":1073741824,"allocated":37642240,"data_stored":36894461,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Dec 5 06:51:36 2019","interfaces":[{"interface":"back","average":{"1min":0.662,"5min":18.296,"15min":13.058},"min":{"1min":0.296,"5min":0.197,"15min":0.196},"max":{"1min":1.334,"5min":802.154,"15min":802.154},"last":212.998},{"interface":"front","average":{"1min":0.750,"5min":7.929,"15min":9.078},"min":{"1min":0.319,"5min":0.259,"15min":0.242},"max":{"1min":1.454,"5min":828.266,"15min":991.916},"last":0.523}]},{"osd":1,"last update":"Thu Dec 5 06:51:45 2019","interfaces":[{"interface":"back","average":{"1min":0.803,"5min":8.874,"15min":16.543},"min":{"1min":0.415,"5min":0.264,"15min":0.211},"max":{"1min":2.417,"5min":699.405,"15min":957.811},"last":0.681},{"interface":"front","average":{"1min":0.959,"5min":0.867,"15min":7.839},"min":{"1min":0.522,"5min":0.311,"15min":0.282},"max":{"1min":2.441,"5min":2.441,"15min":501.932},"last":0.795}]},{"osd":2,"last update":"Thu Dec 5 06:52:03 2019","interfaces":[{"interface":"back","average":{"1min":0.829,"5min":0.865,"15min":4.116},"min":{"1min":0.460,"5min":0.443,"15min":0.266},"max":{"1min":1.286,"5min":1.512,"15min":444.359},"last":0.729},{"interface":"front","average":{"1min":18.619,"5min":4.432,"15min":6.711},"min":{"1min":0.487,"5min":0.340,"15min":0.297},"max":{"1min":387.829,"5min":387.829,"15min":628.956},"last":1.139}]},{"osd":4,"last update":"Thu Dec 5 06:51:36 2019","interfaces":[{"interface":"back","average":{"1min":1.004,"5min":0.989,"15min":1.926},"min":{"1min":0.618,"5min":0.402,"15min":0.325},"max":{"1min":2.287,"5min":2.287,"15min":216.348},"last":1.233},{"interface":"front","average":{"1min":1.072,"5min":1.012,"15min":1.582},"min":{"1min":0.659,"5min":0.335,"15min":0.335},"max":{"1min":1.392,"5min":1.669,"15min":192.285},"last":0.887}]},{"osd":5,"last update":"Thu Dec 5 06:51:36 2019","interfaces":[{"interface":"back","average":{"1min":1.133,"5min":8.721,"15min":4.576},"min":{"1min":0.690,"5min":0.372,"15min":0.328},"max":{"1min":2.345,"5min":880.020,"15min":880.020},"last":1.077},{"interface":"front","average":{"1min":0.984,"5min":1.075,"15min":1.045},"min":{"1min":0.414,"5min":0.414,"15min":0.364},"max":{"1min":1.327,"5min":1.532,"15min":1.936},"last":1.003}]},{"osd":6,"last update":"Thu Dec 5 06:51:35 2019","interfaces":[{"interface":"back","average":{"1min":51.161,"5min":11.110,"15min":6.234},"min":{"1min":0.557,"5min":0.430,"15min":0.413},"max":{"1min":1001.571,"5min":1001.571,"15min":1001.571},"last":0.919},{"interface":"front","average":{"1min":1.016,"5min":8.110,"15min":3.832},"min":{"1min":0.265,"5min":0.265,"15min":0.265},"max":{"1min":2.065,"5min":558.711,"15min":558.711},"last":1.185}]},{"osd":7,"last update":"Thu Dec 5 06:51:50 2019","interfaces":[{"interface":"back","average":{"1min":1.246,"5min":2.412,"15min":10.388},"min":{"1min":0.841,"5min":0.523,"15min":0.365},"max":{"1min":2.765,"5min":97.130,"15min":877.034},"last":1.315},{"interface":"front","average":{"1min":1.304,"5min":1.231,"15min":1.161},"min":{"1min":0.869,"5min":0.499,"15min":0.435},"max":{"1min":2.717,"5min":2.717,"15min":2.717},"last":1.383}]}]},{"osd":4,"up_from":13,"seq":55834576161,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1050352,"kb_used_data":1768,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93321488,"statfs":{"total":96636764160,"available":95561203712,"internally_reserved":1073741824,"allocated":1810432,"data_stored":1066649,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Dec 5 06:52:06 2019","interfaces":[{"interface":"back","average":{"1min":42.612,"5min":15.401,"15min":8.269},"min":{"1min":0.313,"5min":0.283,"15min":0.264},"max":{"1min":878.533,"5min":878.533,"15min":878.533},"last":1.194},{"interface":"front","average":{"1min":0.762,"5min":12.530,"15min":5.365},"min":{"1min":0.335,"5min":0.335,"15min":0.225},"max":{"1min":1.169,"5min":524.030,"15min":524.030},"last":0.953}]},{"osd":1,"last update":"Thu Dec 5 06:52:06 2019","interfaces":[{"interface":"back","average":{"1min":0.781,"5min":1.198,"15min":3.749},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":1.329,"5min":31.305,"15min":501.402},"last":0.851},{"interface":"front","average":{"1min":0.897,"5min":0.939,"15min":4.737},"min":{"1min":0.332,"5min":0.332,"15min":0.304},"max":{"1min":1.589,"5min":1.589,"15min":501.754},"last":1.019}]},{"osd":2,"last update":"Thu Dec 5 06:52:16 2019","interfaces":[{"interface":"back","average":{"1min":0.885,"5min":16.331,"15min":12.885},"min":{"1min":0.291,"5min":0.291,"15min":0.268},"max":{"1min":1.454,"5min":850.354,"15min":897.315},"last":0.770},{"interface":"front","average":{"1min":0.964,"5min":0.997,"15min":3.958},"min":{"1min":0.445,"5min":0.287,"15min":0.287},"max":{"1min":1.399,"5min":1.399,"15min":632.767},"last":0.900}]},{"osd":3,"last update":"Thu Dec 5 06:52:06 2019","interfaces":[{"interface":"back","average":{"1min":0.944,"5min":6.942,"15min":3.440},"min":{"1min":0.356,"5min":0.340,"15min":0.340},"max":{"1min":4.620,"5min":501.658,"15min":501.658},"last":1.136},{"interface":"front","average":{"1min":4.951,"5min":3.646,"15min":2.967},"min":{"1min":0.339,"5min":0.339,"15min":0.339},"max":{"1min":83.095,"5min":149.969,"15min":308.851},"last":1.102}]},{"osd":5,"last update":"Thu Dec 5 06:52:06 2019","interfaces":[{"interface":"back","average":{"1min":21.400,"5min":5.023,"15min":5.351},"min":{"1min":0.402,"5min":0.334,"15min":0.325},"max":{"1min":412.849,"5min":412.849,"15min":840.089},"last":1.304},{"interface":"front","average":{"1min":0.996,"5min":16.931,"15min":9.249},"min":{"1min":0.353,"5min":0.353,"15min":0.353},"max":{"1min":1.457,"5min":660.266,"15min":748.692},"last":1.469}]},{"osd":6,"last update":"Thu Dec 5 06:52:06 2019","interfaces":[{"interface":"back","average":{"1min":0.950,"5min":0.854,"15min":10.833},"min":{"1min":0.306,"5min":0.306,"15min":0.306},"max":{"1min":4.652,"5min":4.652,"15min":861.297},"last":1.266},{"interface":"front","average":{"1min":1.003,"5min":1.125,"15min":3.387},"min":{"1min":0.429,"5min":0.390,"15min":0.374},"max":{"1min":1.651,"5min":1.808,"15min":690.644},"last":1.773}]},{"osd":7,"last update":"Thu Dec 5 06:51:21 2019","interfaces":[{"interface":"back","average":{"1min":1.120,"5min":1.135,"15min":6.627},"min":{"1min":0.481,"5min":0.425,"15min":0.351},"max":{"1min":1.534,"5min":2.624,"15min":507.089},"last":1.568},{"interface":"front","average":{"1min":1.194,"5min":1.191,"15min":3.242},"min":{"1min":0.520,"5min":0.464,"15min":0.464},"max":{"1min":1.635,"5min":1.670,"15min":452.400},"last":1.634}]}]},{"osd":5,"up_from":13,"seq":55834576166,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1160948,"kb_used_data":112364,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93210892,"statfs":{"total":96636764160,"available":95447953408,"internally_reserved":1073741824,"allocated":115060736,"data_stored":114312891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Dec 5 06:51:52 2019","interfaces":[{"interface":"back","average":{"1min":0.849,"5min":4.329,"15min":1.981},"min":{"1min":0.261,"5min":0.261,"15min":0.235},"max":{"1min":1.463,"5min":331.975,"15min":331.975},"last":0.784},{"interface":"front","average":{"1min":0.698,"5min":0.829,"15min":3.592},"min":{"1min":0.285,"5min":0.285,"15min":0.278},"max":{"1min":1.062,"5min":2.878,"15min":676.393},"last":1.040}]},{"osd":1,"last update":"Thu Dec 5 06:52:01 2019","interfaces":[{"interface":"back","average":{"1min":0.913,"5min":0.813,"15min":2.211},"min":{"1min":0.343,"5min":0.308,"15min":0.225},"max":{"1min":1.321,"5min":1.321,"15min":251.684},"last":0.993},{"interface":"front","average":{"1min":1.022,"5min":8.528,"15min":5.471},"min":{"1min":0.359,"5min":0.359,"15min":0.245},"max":{"1min":1.423,"5min":722.849,"15min":722.849},"last":1.112}]},{"osd":2,"last update":"Thu Dec 5 06:51:45 2019","interfaces":[{"interface":"back","average":{"1min":0.821,"5min":17.263,"15min":7.170},"min":{"1min":0.362,"5min":0.362,"15min":0.281},"max":{"1min":1.164,"5min":934.880,"15min":934.880},"last":0.715},{"interface":"front","average":{"1min":0.948,"5min":9.446,"15min":8.045},"min":{"1min":0.407,"5min":0.407,"15min":0.407},"max":{"1min":1.326,"5min":676.797,"15min":794.102},"last":1.322}]},{"osd":3,"last update":"Thu Dec 5 06:51:24 2019","interfaces":[{"interface":"back","average":{"1min":1.037,"5min":1.093,"15min":3.676},"min":{"1min":0.468,"5min":0.468,"15min":0.340},"max":{"1min":1.318,"5min":2.942,"15min":683.842},"last":1.159},{"interface":"front","average":{"1min":42.174,"5min":19.109,"15min":8.696},"min":{"1min":0.500,"5min":0.500,"15min":0.387},"max":{"1min":740.171,"5min":740.171,"15min":740.171},"last":1.245}]},{"osd":4,"last update":"Thu Dec 5 06:51:52 2019","interfaces":[{"interface":"back","average":{"1min":1.037,"5min":1.096,"15min":7.585},"min":{"1min":0.341,"5min":0.341,"15min":0.327},"max":{"1min":1.476,"5min":2.914,"15min":633.428},"last":0.870},{"interface":"front","average":{"1min":0.964,"5min":0.988,"15min":5.125},"min":{"1min":0.302,"5min":0.302,"15min":0.302},"max":{"1min":1.507,"5min":1.507,"15min":792.823},"last":0.906}]},{"osd":6,"last update":"Thu Dec 5 06:51:52 2019","interfaces":[{"interface":"back","average":{"1min":1.088,"5min":1.044,"15min":6.845},"min":{"1min":0.282,"5min":0.282,"15min":0.282},"max":{"1min":1.829,"5min":1.829,"15min":891.609},"last":0.825},{"interface":"front","average":{"1min":1.210,"5min":13.625,"15min":10.318},"min":{"1min":0.394,"5min":0.394,"15min":0.394},"max":{"1min":1.721,"5min":635.191,"15min":916.939},"last":0.956}]},{"osd":7,"last update":"Thu Dec 5 06:51:52 2019","interfaces":[{"interface":"back","average":{"1min":1.230,"5min":8.263,"15min":6.474},"min":{"1min":0.601,"5min":0.586,"15min":0.401},"max":{"1min":1.766,"5min":735.632,"15min":735.632},"last":1.199},{"interface":"front","average":{"1min":1.303,"5min":5.207,"15min":6.313},"min":{"1min":0.618,"5min":0.618,"15min":0.380},"max":{"1min":1.874,"5min":405.552,"15min":405.552},"last":1.285}]}]}]}} 2019-12-05T06:52:25.244 INFO:teuthology.orchestra.run.smithi042:> sudo ceph --cluster ceph --mon-client-directed-command-retry 5 tell 'mon.*' injectargs -- --no-mon-health-to-clog 2019-12-05T06:52:25.346 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.334+0000 7f7206021700 1 Processor -- start 2019-12-05T06:52:25.346 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.334+0000 7f7206021700 1 -- start start 2019-12-05T06:52:25.346 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.338+0000 7f7206021700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f72000a7570 0x7f720009ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:52:25.347 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.338+0000 7f7206021700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f7200144c90 0x7f72001450c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:52:25.347 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.338+0000 7f7206021700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f720007f050 con 0x7f720009f370 2019-12-05T06:52:25.347 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.338+0000 7f7206021700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f720009f850 con 0x7f7200144c90 2019-12-05T06:52:25.347 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.338+0000 7f7206021700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7200144750 con 0x7f72000a7570 2019-12-05T06:52:25.347 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.338+0000 7f7204dbf700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f7200144c90 0x7f72001450c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:52:25.347 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.338+0000 7f71ff7fe700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f720009f370 0x7f720007e850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.42:49040/0 (socket says 172.21.15.42:49040) 2019-12-05T06:52:25.347 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.338+0000 7f71ff7fe700 1 -- 172.21.15.42:0/2262895797 learned_addr learned my addr 172.21.15.42:0/2262895797 (peer_addr_for_me v1:172.21.15.42:0/0) 2019-12-05T06:52:25.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.338+0000 7f7204dbf700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f7200144c90 0x7f72001450c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:39566/0 (socket says 172.21.15.42:39566) 2019-12-05T06:52:25.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.338+0000 7f71fffff700 1 --2- 172.21.15.42:0/2262895797 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f72000a7570 0x7f720009ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:52:25.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.338+0000 7f7204dbf700 1 -- 172.21.15.42:0/2262895797 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f72000a7570 msgr2=0x7f720009ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:52:25.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.338+0000 7f7204dbf700 1 --2- 172.21.15.42:0/2262895797 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f72000a7570 0x7f720009ee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:52:25.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.338+0000 7f7204dbf700 1 -- 172.21.15.42:0/2262895797 >> v1:172.21.15.40:6789/0 conn(0x7f720009f370 legacy=0x7f720007e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:52:25.348 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.338+0000 7f71feffd700 1 -- 172.21.15.42:0/2262895797 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3003813949 0 0) 0x7f720007f050 con 0x7f720009f370 2019-12-05T06:52:25.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.338+0000 7f7204dbf700 1 -- 172.21.15.42:0/2262895797 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72000a7df0 con 0x7f7200144c90 2019-12-05T06:52:25.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.338+0000 7f7204dbf700 1 --2- 172.21.15.42:0/2262895797 >> v2:172.21.15.42:3300/0 conn(0x7f7200144c90 0x7f72001450c0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f720005aba0 tx=0x7f71f800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:25.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.338+0000 7f7204dbf700 1 --2- 172.21.15.42:0/2262895797 >> v2:172.21.15.42:3300/0 conn(0x7f7200144c90 0x7f72001450c0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f720005aba0 tx=0x7f71f800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:25.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.338+0000 7f7204dbf700 1 --2- 172.21.15.42:0/2262895797 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7200144c90 0x7f72001450c0 secure :-1 s=READY pgs=3407 cs=0 l=1 rx=0x7f720005aba0 tx=0x7f71f800fff0).ready entity=mon.1 client_cookie=8ab5e3a2ac124e4c server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:52:25.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.338+0000 7f7204dbf700 1 --2- 172.21.15.42:0/2262895797 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7200144c90 0x7f72001450c0 secure :-1 s=THROTTLE_DONE pgs=3407 cs=0 l=1 rx=0x7f720005aba0 tx=0x7f71f800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:25.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.338+0000 7f71feffd700 1 -- 172.21.15.42:0/2262895797 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f71f8011a10 con 0x7f7200144c90 2019-12-05T06:52:25.349 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.338+0000 7f7204dbf700 1 --2- 172.21.15.42:0/2262895797 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7200144c90 0x7f72001450c0 secure :-1 s=THROTTLE_DONE pgs=3407 cs=0 l=1 rx=0x7f720005aba0 tx=0x7f71f800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:25.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.338+0000 7f7204dbf700 1 --2- 172.21.15.42:0/2262895797 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7200144c90 0x7f72001450c0 secure :-1 s=THROTTLE_DONE pgs=3407 cs=0 l=1 rx=0x7f720005aba0 tx=0x7f71f800fff0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:25.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.338+0000 7f71feffd700 1 -- 172.21.15.42:0/2262895797 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71f8011630 con 0x7f7200144c90 2019-12-05T06:52:25.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.338+0000 7f71feffd700 1 -- 172.21.15.42:0/2262895797 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f71f800a160 con 0x7f7200144c90 2019-12-05T06:52:25.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.338+0000 7f7206021700 1 -- 172.21.15.42:0/2262895797 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7200144c90 msgr2=0x7f72001450c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:52:25.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.338+0000 7f7206021700 1 --2- 172.21.15.42:0/2262895797 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7200144c90 0x7f72001450c0 secure :-1 s=READY pgs=3407 cs=0 l=1 rx=0x7f720005aba0 tx=0x7f71f800fff0).stop 2019-12-05T06:52:25.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.338+0000 7f7206021700 1 -- 172.21.15.42:0/2262895797 shutdown_connections 2019-12-05T06:52:25.350 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.338+0000 7f7206021700 1 --2- 172.21.15.42:0/2262895797 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7200144c90 0x7f72001450c0 unknown :-1 s=CLOSED pgs=3407 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:52:25.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.338+0000 7f7206021700 1 --2- 172.21.15.42:0/2262895797 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f72000a7570 0x7f720009ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:52:25.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.338+0000 7f7206021700 1 -- 172.21.15.42:0/2262895797 shutdown_connections 2019-12-05T06:52:25.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.338+0000 7f7206021700 1 -- 172.21.15.42:0/2262895797 wait complete. 2019-12-05T06:52:25.351 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.338+0000 7f7206021700 1 -- 172.21.15.42:0/2262895797 >> 172.21.15.42:0/2262895797 conn(0x7f72000aaa40 msgr2=0x7f72000a6a30 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:52:25.354 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.342+0000 7f7206021700 1 Processor -- start 2019-12-05T06:52:25.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.342+0000 7f7206021700 1 -- start start 2019-12-05T06:52:25.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.342+0000 7f7206021700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f720011a1a0 0x7f720011c570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:52:25.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.342+0000 7f7206021700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f720011ada0 0x7f720011b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:52:25.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.342+0000 7f7206021700 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72000a8560 con 0x7f72000a8cb0 2019-12-05T06:52:25.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.342+0000 7f7206021700 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f720011bee0 con 0x7f720011a1a0 2019-12-05T06:52:25.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.342+0000 7f7206021700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f720011cce0 con 0x7f720011ada0 2019-12-05T06:52:25.355 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.342+0000 7f71ff7fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f720011a1a0 0x7f720011c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:52:25.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.342+0000 7f71ff7fe700 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f720011a1a0 0x7f720011c570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:39570/0 (socket says 172.21.15.42:39570) 2019-12-05T06:52:25.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.342+0000 7f71fffff700 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f72000a8cb0 0x7f720007bba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:39606/0 (socket says 172.21.15.42:39606) 2019-12-05T06:52:25.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.342+0000 7f71ff7fe700 1 -- 172.21.15.42:0/3262361680 learned_addr learned my addr 172.21.15.42:0/3262361680 (peer_addr_for_me v2:172.21.15.42:0/0) 2019-12-05T06:52:25.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.342+0000 7f7204dbf700 1 --2- 172.21.15.42:0/3262361680 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f720011ada0 0x7f720011b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:52:25.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.342+0000 7f71ff7fe700 1 -- 172.21.15.42:0/3262361680 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f720011ada0 msgr2=0x7f720011b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:52:25.356 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.342+0000 7f71ff7fe700 1 --2- 172.21.15.42:0/3262361680 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f720011ada0 0x7f720011b1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:52:25.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.342+0000 7f71ff7fe700 1 -- 172.21.15.42:0/3262361680 >> v1:172.21.15.42:6789/0 conn(0x7f72000a8cb0 legacy=0x7f720007bba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:52:25.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.342+0000 7f71ff7fe700 1 -- 172.21.15.42:0/3262361680 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f720011b720 con 0x7f720011a1a0 2019-12-05T06:52:25.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.342+0000 7f71ff7fe700 1 --2- 172.21.15.42:0/3262361680 >> v2:172.21.15.42:3300/0 conn(0x7f720011a1a0 0x7f720011c570 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f71f4000b80 tx=0x7f71f4010940).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:25.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.346+0000 7f71ff7fe700 1 --2- 172.21.15.42:0/3262361680 >> v2:172.21.15.42:3300/0 conn(0x7f720011a1a0 0x7f720011c570 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f71f4000b80 tx=0x7f71f4010940).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:25.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.346+0000 7f71ff7fe700 1 --2- 172.21.15.42:0/3262361680 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f720011a1a0 0x7f720011c570 secure :-1 s=READY pgs=3409 cs=0 l=1 rx=0x7f71f4000b80 tx=0x7f71f4010940).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:52:25.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.346+0000 7f71ff7fe700 1 --2- 172.21.15.42:0/3262361680 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f720011a1a0 0x7f720011c570 secure :-1 s=THROTTLE_DONE pgs=3409 cs=0 l=1 rx=0x7f71f4000b80 tx=0x7f71f4010940).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:25.357 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.346+0000 7f71ff7fe700 1 --2- 172.21.15.42:0/3262361680 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f720011a1a0 0x7f720011c570 secure :-1 s=THROTTLE_DONE pgs=3409 cs=0 l=1 rx=0x7f71f4000b80 tx=0x7f71f4010940).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:25.358 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.346+0000 7f71fcff9700 1 -- 172.21.15.42:0/3262361680 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f71f4011670 con 0x7f720011a1a0 2019-12-05T06:52:25.358 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.346+0000 7f71ff7fe700 1 --2- 172.21.15.42:0/3262361680 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f720011a1a0 0x7f720011c570 secure :-1 s=THROTTLE_DONE pgs=3409 cs=0 l=1 rx=0x7f71f4000b80 tx=0x7f71f4010940).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:25.358 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.346+0000 7f71fcff9700 1 -- 172.21.15.42:0/3262361680 <== mon.1 v2:172.21.15.42:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71f4011290 con 0x7f720011a1a0 2019-12-05T06:52:25.358 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.346+0000 7f7206021700 1 -- 172.21.15.42:0/3262361680 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7200138bf0 con 0x7f720011a1a0 2019-12-05T06:52:25.358 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.346+0000 7f71fcff9700 1 -- 172.21.15.42:0/3262361680 <== mon.1 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f71f4011c50 con 0x7f720011a1a0 2019-12-05T06:52:25.358 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.346+0000 7f7206021700 1 -- 172.21.15.42:0/3262361680 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7200138f50 con 0x7f720011a1a0 2019-12-05T06:52:25.358 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.346+0000 7f71ff7fe700 1 --2- 172.21.15.42:0/3262361680 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f720011a1a0 0x7f720011c570 secure :-1 s=THROTTLE_DONE pgs=3409 cs=0 l=1 rx=0x7f71f4000b80 tx=0x7f71f4010940).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:25.359 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.346+0000 7f71fcff9700 1 -- 172.21.15.42:0/3262361680 <== mon.1 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f71f400b330 con 0x7f720011a1a0 2019-12-05T06:52:25.359 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.346+0000 7f71ff7fe700 1 --2- 172.21.15.42:0/3262361680 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f720011a1a0 0x7f720011c570 secure :-1 s=THROTTLE_DONE pgs=3409 cs=0 l=1 rx=0x7f71f4000b80 tx=0x7f71f4010940).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:25.359 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.346+0000 7f71fcff9700 1 --2- 172.21.15.42:0/3262361680 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f71e003ad80 0x7f71e003d290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:52:25.359 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.346+0000 7f71fcff9700 1 -- 172.21.15.42:0/3262361680 <== mon.1 v2:172.21.15.42:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 12337+0+0 (secure 0 0 0) 0x7f71f404b210 con 0x7f720011a1a0 2019-12-05T06:52:25.359 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.346+0000 7f71fffff700 1 --2- 172.21.15.42:0/3262361680 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f71e003ad80 0x7f71e003d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:52:25.359 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.346+0000 7f71fffff700 1 --2- 172.21.15.42:0/3262361680 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f71e003ad80 0x7f71e003d290 crc :-1 s=READY pgs=1428 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:52:25.359 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.350+0000 7f7206021700 1 -- 172.21.15.42:0/3262361680 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f720005d100 con 0x7f720011a1a0 2019-12-05T06:52:25.359 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.350+0000 7f71ff7fe700 1 --2- 172.21.15.42:0/3262361680 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f720011a1a0 0x7f720011c570 secure :-1 s=THROTTLE_DONE pgs=3409 cs=0 l=1 rx=0x7f71f4000b80 tx=0x7f71f4010940).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:25.360 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.350+0000 7f71fcff9700 1 -- 172.21.15.42:0/3262361680 <== mon.1 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+1051 (secure 0 0 0) 0x7f71f4032290 con 0x7f720011a1a0 2019-12-05T06:52:25.360 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.350+0000 7f7206021700 1 --2- 172.21.15.42:0/3262361680 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f720011f600 0x7f7200054250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:52:25.360 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.350+0000 7f7206021700 1 -- 172.21.15.42:0/3262361680 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f720011cac0 con 0x7f720011f600 2019-12-05T06:52:25.361 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.350+0000 7f7204dbf700 1 --2- 172.21.15.42:0/3262361680 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f720011f600 0x7f7200054250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:52:25.362 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.350+0000 7f7204dbf700 1 -- 172.21.15.42:0/3262361680 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- command(tid 2: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7200090090 con 0x7f720011f600 2019-12-05T06:52:25.362 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.350+0000 7f7204dbf700 1 --2- 172.21.15.42:0/3262361680 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f720011f600 0x7f7200054250 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f720005aba0 tx=0x7f71f80118c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:25.362 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.354+0000 7f7204dbf700 1 --2- 172.21.15.42:0/3262361680 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f720011f600 0x7f7200054250 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f720005aba0 tx=0x7f71f80118c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:25.362 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.354+0000 7f7204dbf700 1 --2- 172.21.15.42:0/3262361680 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f720011f600 0x7f7200054250 secure :-1 s=READY pgs=2586 cs=0 l=1 rx=0x7f720005aba0 tx=0x7f71f80118c0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:52:25.362 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.354+0000 7f7204dbf700 1 --2- 172.21.15.42:0/3262361680 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f720011f600 0x7f7200054250 secure :-1 s=THROTTLE_DONE pgs=2586 cs=0 l=1 rx=0x7f720005aba0 tx=0x7f71f80118c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:25.362 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.354+0000 7f71fcff9700 1 -- 172.21.15.42:0/3262361680 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f71f800f760 con 0x7f720011f600 2019-12-05T06:52:25.362 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.354+0000 7f7204dbf700 1 --2- 172.21.15.42:0/3262361680 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f720011f600 0x7f7200054250 secure :-1 s=THROTTLE_DONE pgs=2586 cs=0 l=1 rx=0x7f720005aba0 tx=0x7f71f80118c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:25.363 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.354+0000 7f71fcff9700 1 -- 172.21.15.42:0/3262361680 <== mon.0 v2:172.21.15.40:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+7841 (secure 0 0 0) 0x7f71f800f960 con 0x7f720011f600 2019-12-05T06:52:25.363 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.354+0000 7f71fcff9700 1 -- 172.21.15.42:0/3262361680 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f720011f600 msgr2=0x7f7200054250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:52:25.363 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.354+0000 7f71fcff9700 1 --2- 172.21.15.42:0/3262361680 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f720011f600 0x7f7200054250 secure :-1 s=READY pgs=2586 cs=0 l=1 rx=0x7f720005aba0 tx=0x7f71f80118c0).stop 2019-12-05T06:52:25.363 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.354+0000 7f71fcff9700 1 -- 172.21.15.42:0/3262361680 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f720011f600 msgr2=0x7f7200054250 secure :-1 s=STATE_CLOSED l=1).mark_down 2019-12-05T06:52:25.363 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.354+0000 7f71fcff9700 1 --2- 172.21.15.42:0/3262361680 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f720011f600 0x7f7200054250 secure :-1 s=CLOSED pgs=2586 cs=0 l=1 rx=0x7f720005aba0 tx=0x7f71f80118c0).stop 2019-12-05T06:52:25.372 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.366+0000 7f7206021700 1 --2- 172.21.15.42:0/3262361680 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f72000547a0 0x7f7200054bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:52:25.373 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.366+0000 7f7206021700 1 -- 172.21.15.42:0/3262361680 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f720005b8b0 con 0x7f72000547a0 2019-12-05T06:52:25.373 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.366+0000 7f7204dbf700 1 --2- 172.21.15.42:0/3262361680 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f72000547a0 0x7f7200054bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:52:25.373 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.370+0000 7f7204dbf700 1 -- 172.21.15.42:0/3262361680 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- command(tid 3: {"injected_args": ["--no-mon-health-to-clog"], "prefix": "injectargs"}) v1 -- 0x7f7200042680 con 0x7f72000547a0 2019-12-05T06:52:25.373 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.370+0000 7f7204dbf700 1 --2- 172.21.15.42:0/3262361680 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f72000547a0 0x7f7200054bd0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f720005aba0 tx=0x7f71f80076d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:25.374 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.370+0000 7f7204dbf700 1 --2- 172.21.15.42:0/3262361680 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f72000547a0 0x7f7200054bd0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f720005aba0 tx=0x7f71f80076d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:25.374 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.370+0000 7f7204dbf700 1 --2- 172.21.15.42:0/3262361680 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f72000547a0 0x7f7200054bd0 secure :-1 s=READY pgs=2587 cs=0 l=1 rx=0x7f720005aba0 tx=0x7f71f80076d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:52:25.375 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.370+0000 7f7204dbf700 1 --2- 172.21.15.42:0/3262361680 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f72000547a0 0x7f7200054bd0 secure :-1 s=THROTTLE_DONE pgs=2587 cs=0 l=1 rx=0x7f720005aba0 tx=0x7f71f80076d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:25.375 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.370+0000 7f71fcff9700 1 -- 172.21.15.42:0/3262361680 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f71f800a6c0 con 0x7f72000547a0 2019-12-05T06:52:25.376 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.374+0000 7f7204dbf700 1 --2- 172.21.15.42:0/3262361680 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f72000547a0 0x7f7200054bd0 secure :-1 s=THROTTLE_DONE pgs=2587 cs=0 l=1 rx=0x7f720005aba0 tx=0x7f71f80076d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:25.376 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.374+0000 7f71fcff9700 1 -- 172.21.15.42:0/3262361680 <== mon.0 v2:172.21.15.40:3300/0 2 ==== command_reply(tid 3: 0 mon_health_to_clog = 'false' ) v1 ==== 37+0+3 (secure 0 0 0) 0x7f71f800a8c0 con 0x7f72000547a0 2019-12-05T06:52:25.377 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.374+0000 7f71fcff9700 1 -- 172.21.15.42:0/3262361680 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f72000547a0 msgr2=0x7f7200054bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:52:25.377 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.374+0000 7f71fcff9700 1 --2- 172.21.15.42:0/3262361680 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f72000547a0 0x7f7200054bd0 secure :-1 s=READY pgs=2587 cs=0 l=1 rx=0x7f720005aba0 tx=0x7f71f80076d0).stop 2019-12-05T06:52:25.377 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.374+0000 7f71fcff9700 1 -- 172.21.15.42:0/3262361680 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f72000547a0 msgr2=0x7f7200054bd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2019-12-05T06:52:25.377 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.374+0000 7f71fcff9700 1 --2- 172.21.15.42:0/3262361680 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f72000547a0 0x7f7200054bd0 unknown :-1 s=CLOSED pgs=2587 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:52:25.379 INFO:teuthology.orchestra.run.smithi042.stderr:mon.b: mon_health_to_clog = 'false' 2019-12-05T06:52:25.379 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.374+0000 7f7206021700 1 --2- 172.21.15.42:0/3262361680 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7200141410 0x7f7200141840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:52:25.379 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.374+0000 7f7206021700 1 -- 172.21.15.42:0/3262361680 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f720005d3e0 con 0x7f7200141410 2019-12-05T06:52:25.379 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.374+0000 7f7204dbf700 1 --2- 172.21.15.42:0/3262361680 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7200141410 0x7f7200141840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:52:25.380 INFO:teuthology.orchestra.run.smithi042.stdout:mon.b: {} 2019-12-05T06:52:25.381 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.374+0000 7f7204dbf700 1 -- 172.21.15.42:0/3262361680 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 4: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7200055120 con 0x7f7200141410 2019-12-05T06:52:25.381 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.374+0000 7f7204dbf700 1 --2- 172.21.15.42:0/3262361680 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7200141410 0x7f7200141840 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f720005aba0 tx=0x7f71f8007750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:25.382 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.378+0000 7f7204dbf700 1 --2- 172.21.15.42:0/3262361680 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7200141410 0x7f7200141840 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f720005aba0 tx=0x7f71f8007750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:25.382 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.378+0000 7f7204dbf700 1 --2- 172.21.15.42:0/3262361680 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7200141410 0x7f7200141840 secure :-1 s=READY pgs=3410 cs=0 l=1 rx=0x7f720005aba0 tx=0x7f71f8007750).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:52:25.382 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.378+0000 7f7204dbf700 1 --2- 172.21.15.42:0/3262361680 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7200141410 0x7f7200141840 secure :-1 s=THROTTLE_DONE pgs=3410 cs=0 l=1 rx=0x7f720005aba0 tx=0x7f71f8007750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:25.382 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.378+0000 7f71fcff9700 1 -- 172.21.15.42:0/3262361680 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f71f800dbc0 con 0x7f7200141410 2019-12-05T06:52:25.382 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.378+0000 7f7204dbf700 1 --2- 172.21.15.42:0/3262361680 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7200141410 0x7f7200141840 secure :-1 s=THROTTLE_DONE pgs=3410 cs=0 l=1 rx=0x7f720005aba0 tx=0x7f71f8007750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:25.382 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.378+0000 7f71fcff9700 1 -- 172.21.15.42:0/3262361680 <== mon.1 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 4: 0 ) v1 ==== 8+0+7841 (secure 0 0 0) 0x7f71f800c330 con 0x7f7200141410 2019-12-05T06:52:25.382 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.378+0000 7f71fcff9700 1 -- 172.21.15.42:0/3262361680 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7200141410 msgr2=0x7f7200141840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:52:25.383 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.378+0000 7f71fcff9700 1 --2- 172.21.15.42:0/3262361680 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7200141410 0x7f7200141840 secure :-1 s=READY pgs=3410 cs=0 l=1 rx=0x7f720005aba0 tx=0x7f71f8007750).stop 2019-12-05T06:52:25.383 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.378+0000 7f71fcff9700 1 -- 172.21.15.42:0/3262361680 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7200141410 msgr2=0x7f7200141840 unknown :-1 s=STATE_CLOSED l=1).mark_down 2019-12-05T06:52:25.383 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.378+0000 7f7204dbf700 1 -- 172.21.15.42:0/3262361680 reap_dead start 2019-12-05T06:52:25.383 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.378+0000 7f71fcff9700 1 --2- 172.21.15.42:0/3262361680 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7200141410 0x7f7200141840 unknown :-1 s=CLOSED pgs=3410 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:52:25.393 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.390+0000 7f7206021700 1 --2- 172.21.15.42:0/3262361680 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f72000a8cb0 0x7f720007bba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:52:25.393 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.390+0000 7f7206021700 1 -- 172.21.15.42:0/3262361680 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72000ab120 con 0x7f72000a8cb0 2019-12-05T06:52:25.393 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.390+0000 7f7204dbf700 1 --2- 172.21.15.42:0/3262361680 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f72000a8cb0 0x7f720007bba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:52:25.394 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.390+0000 7f7204dbf700 1 -- 172.21.15.42:0/3262361680 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 5: {"injected_args": ["--no-mon-health-to-clog"], "prefix": "injectargs"}) v1 -- 0x7f720007c0f0 con 0x7f72000a8cb0 2019-12-05T06:52:25.395 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.390+0000 7f7204dbf700 1 --2- 172.21.15.42:0/3262361680 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f72000a8cb0 0x7f720007bba0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f72000a68f0 tx=0x7f71f8007150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:25.395 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.390+0000 7f7204dbf700 1 --2- 172.21.15.42:0/3262361680 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f72000a8cb0 0x7f720007bba0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f72000a68f0 tx=0x7f71f8007150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:25.395 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.390+0000 7f7204dbf700 1 --2- 172.21.15.42:0/3262361680 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f72000a8cb0 0x7f720007bba0 secure :-1 s=READY pgs=3411 cs=0 l=1 rx=0x7f72000a68f0 tx=0x7f71f8007150).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:52:25.395 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.390+0000 7f7204dbf700 1 --2- 172.21.15.42:0/3262361680 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f72000a8cb0 0x7f720007bba0 secure :-1 s=THROTTLE_DONE pgs=3411 cs=0 l=1 rx=0x7f72000a68f0 tx=0x7f71f8007150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:25.395 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.390+0000 7f71fcff9700 1 -- 172.21.15.42:0/3262361680 <== mon.1 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f71f800c910 con 0x7f72000a8cb0 2019-12-05T06:52:25.397 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.394+0000 7f7204dbf700 1 --2- 172.21.15.42:0/3262361680 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f72000a8cb0 0x7f720007bba0 secure :-1 s=THROTTLE_DONE pgs=3411 cs=0 l=1 rx=0x7f72000a68f0 tx=0x7f71f8007150).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:25.397 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.394+0000 7f71fcff9700 1 -- 172.21.15.42:0/3262361680 <== mon.1 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 5: 0 mon_health_to_clog = 'false' ) v1 ==== 37+0+3 (secure 0 0 0) 0x7f71f800cb10 con 0x7f72000a8cb0 2019-12-05T06:52:25.397 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.394+0000 7f71fcff9700 1 -- 172.21.15.42:0/3262361680 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f72000a8cb0 msgr2=0x7f720007bba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:52:25.397 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.394+0000 7f71fcff9700 1 --2- 172.21.15.42:0/3262361680 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f72000a8cb0 0x7f720007bba0 secure :-1 s=READY pgs=3411 cs=0 l=1 rx=0x7f72000a68f0 tx=0x7f71f8007150).stop 2019-12-05T06:52:25.398 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.394+0000 7f71fcff9700 1 -- 172.21.15.42:0/3262361680 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f72000a8cb0 msgr2=0x7f720007bba0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2019-12-05T06:52:25.398 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.394+0000 7f71fcff9700 1 --2- 172.21.15.42:0/3262361680 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f72000a8cb0 0x7f720007bba0 unknown :-1 s=CLOSED pgs=3411 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:52:25.401 INFO:teuthology.orchestra.run.smithi042.stdout:mon.a: {} 2019-12-05T06:52:25.401 INFO:teuthology.orchestra.run.smithi042.stderr:mon.a: mon_health_to_clog = 'false' 2019-12-05T06:52:25.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.398+0000 7f7206021700 1 --2- 172.21.15.42:0/3262361680 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7200141d90 0x7f720011ab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:52:25.401 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.398+0000 7f7206021700 1 -- 172.21.15.42:0/3262361680 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f720011b0a0 con 0x7f7200141d90 2019-12-05T06:52:25.402 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.398+0000 7f7204dbf700 1 --2- 172.21.15.42:0/3262361680 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7200141d90 0x7f720011ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:52:25.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.398+0000 7f7204dbf700 1 -- 172.21.15.42:0/3262361680 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- command(tid 6: {"prefix": "get_command_descriptions"}) v1 -- 0x7f720011b2e0 con 0x7f7200141d90 2019-12-05T06:52:25.403 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.398+0000 7f7204dbf700 1 --2- 172.21.15.42:0/3262361680 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7200141d90 0x7f720011ab50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f72000a68f0 tx=0x7f71f800d600).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:25.404 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.398+0000 7f7204dbf700 1 --2- 172.21.15.42:0/3262361680 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7200141d90 0x7f720011ab50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f72000a68f0 tx=0x7f71f800d600).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:25.404 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.398+0000 7f7204dbf700 1 --2- 172.21.15.42:0/3262361680 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7200141d90 0x7f720011ab50 secure :-1 s=READY pgs=759 cs=0 l=1 rx=0x7f72000a68f0 tx=0x7f71f800d600).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:52:25.404 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.398+0000 7f7204dbf700 1 --2- 172.21.15.42:0/3262361680 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7200141d90 0x7f720011ab50 secure :-1 s=THROTTLE_DONE pgs=759 cs=0 l=1 rx=0x7f72000a68f0 tx=0x7f71f800d600).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:25.405 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.398+0000 7f71fcff9700 1 -- 172.21.15.42:0/3262361680 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f71f8021440 con 0x7f7200141d90 2019-12-05T06:52:25.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.398+0000 7f7204dbf700 1 --2- 172.21.15.42:0/3262361680 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7200141d90 0x7f720011ab50 secure :-1 s=THROTTLE_DONE pgs=759 cs=0 l=1 rx=0x7f72000a68f0 tx=0x7f71f800d600).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:25.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.398+0000 7f71fcff9700 1 -- 172.21.15.42:0/3262361680 <== mon.2 v2:172.21.15.40:3301/0 2 ==== command_reply(tid 6: 0 ) v1 ==== 8+0+7841 (secure 0 0 0) 0x7f71f8021640 con 0x7f7200141d90 2019-12-05T06:52:25.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.398+0000 7f71fcff9700 1 -- 172.21.15.42:0/3262361680 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7200141d90 msgr2=0x7f720011ab50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:52:25.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.398+0000 7f71fcff9700 1 --2- 172.21.15.42:0/3262361680 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7200141d90 0x7f720011ab50 secure :-1 s=READY pgs=759 cs=0 l=1 rx=0x7f72000a68f0 tx=0x7f71f800d600).stop 2019-12-05T06:52:25.406 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.398+0000 7f71fcff9700 1 -- 172.21.15.42:0/3262361680 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7200141d90 msgr2=0x7f720011ab50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2019-12-05T06:52:25.407 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.398+0000 7f71fcff9700 1 --2- 172.21.15.42:0/3262361680 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7200141d90 0x7f720011ab50 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:52:25.416 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.410+0000 7f7206021700 1 --2- 172.21.15.42:0/3262361680 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f720008eca0 0x7f720008f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:52:25.416 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.410+0000 7f7206021700 1 -- 172.21.15.42:0/3262361680 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f720011b520 con 0x7f720008eca0 2019-12-05T06:52:25.416 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.414+0000 7f7204dbf700 1 --2- 172.21.15.42:0/3262361680 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f720008eca0 0x7f720008f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:52:25.417 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.414+0000 7f7204dbf700 1 -- 172.21.15.42:0/3262361680 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- command(tid 7: {"injected_args": ["--no-mon-health-to-clog"], "prefix": "injectargs"}) v1 -- 0x7f720008f620 con 0x7f720008eca0 2019-12-05T06:52:25.417 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.414+0000 7f7204dbf700 1 --2- 172.21.15.42:0/3262361680 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f720008eca0 0x7f720008f0d0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f71f8011630 tx=0x7f71f8005d40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:25.418 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.414+0000 7f7204dbf700 1 --2- 172.21.15.42:0/3262361680 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f720008eca0 0x7f720008f0d0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f71f8011630 tx=0x7f71f8005d40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:25.418 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.414+0000 7f7204dbf700 1 --2- 172.21.15.42:0/3262361680 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f720008eca0 0x7f720008f0d0 secure :-1 s=READY pgs=760 cs=0 l=1 rx=0x7f71f8011630 tx=0x7f71f8005d40).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:52:25.418 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.414+0000 7f7204dbf700 1 --2- 172.21.15.42:0/3262361680 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f720008eca0 0x7f720008f0d0 secure :-1 s=THROTTLE_DONE pgs=760 cs=0 l=1 rx=0x7f71f8011630 tx=0x7f71f8005d40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:25.418 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.414+0000 7f71fcff9700 1 -- 172.21.15.42:0/3262361680 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f71f8021870 con 0x7f720008eca0 2019-12-05T06:52:25.420 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.418+0000 7f7204dbf700 1 --2- 172.21.15.42:0/3262361680 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f720008eca0 0x7f720008f0d0 secure :-1 s=THROTTLE_DONE pgs=760 cs=0 l=1 rx=0x7f71f8011630 tx=0x7f71f8005d40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:25.420 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.418+0000 7f71fcff9700 1 -- 172.21.15.42:0/3262361680 <== mon.2 v2:172.21.15.40:3301/0 2 ==== command_reply(tid 7: 0 mon_health_to_clog = 'false' ) v1 ==== 37+0+3 (secure 0 0 0) 0x7f71f800d450 con 0x7f720008eca0 2019-12-05T06:52:25.421 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.418+0000 7f71fcff9700 1 -- 172.21.15.42:0/3262361680 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f720008eca0 msgr2=0x7f720008f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:52:25.421 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.418+0000 7f71fcff9700 1 --2- 172.21.15.42:0/3262361680 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f720008eca0 0x7f720008f0d0 secure :-1 s=READY pgs=760 cs=0 l=1 rx=0x7f71f8011630 tx=0x7f71f8005d40).stop 2019-12-05T06:52:25.421 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.418+0000 7f71fcff9700 1 -- 172.21.15.42:0/3262361680 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f720008eca0 msgr2=0x7f720008f0d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2019-12-05T06:52:25.421 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.418+0000 7f71fcff9700 1 --2- 172.21.15.42:0/3262361680 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f720008eca0 0x7f720008f0d0 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:52:25.423 INFO:teuthology.orchestra.run.smithi042.stdout:mon.c: {} 2019-12-05T06:52:25.423 INFO:teuthology.orchestra.run.smithi042.stderr:mon.c: mon_health_to_clog = 'false' 2019-12-05T06:52:25.424 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.418+0000 7f7206021700 1 -- 172.21.15.42:0/3262361680 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f71e003ad80 msgr2=0x7f71e003d290 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:52:25.425 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.418+0000 7f7206021700 1 --2- 172.21.15.42:0/3262361680 >> [v2:172.21.15.42:6800/11930,v1:172.21.15.42:6801/11930] conn(0x7f71e003ad80 0x7f71e003d290 crc :-1 s=READY pgs=1428 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:52:25.425 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.418+0000 7f7206021700 1 -- 172.21.15.42:0/3262361680 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f720011a1a0 msgr2=0x7f720011c570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:52:25.425 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.418+0000 7f7206021700 1 --2- 172.21.15.42:0/3262361680 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f720011a1a0 0x7f720011c570 secure :-1 s=READY pgs=3409 cs=0 l=1 rx=0x7f71f4000b80 tx=0x7f71f4010940).stop 2019-12-05T06:52:25.425 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.418+0000 7f7204dbf700 1 -- 172.21.15.42:0/3262361680 reap_dead start 2019-12-05T06:52:25.425 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.422+0000 7f7206021700 1 -- 172.21.15.42:0/3262361680 shutdown_connections 2019-12-05T06:52:25.425 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.422+0000 7f7206021700 1 --2- 172.21.15.42:0/3262361680 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f720011a1a0 0x7f720011c570 unknown :-1 s=CLOSED pgs=3409 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:52:25.426 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.422+0000 7f7206021700 1 -- 172.21.15.42:0/3262361680 shutdown_connections 2019-12-05T06:52:25.426 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.422+0000 7f7206021700 1 -- 172.21.15.42:0/3262361680 wait complete. 2019-12-05T06:52:25.426 INFO:teuthology.orchestra.run.smithi042.stderr:2019-12-05T06:52:25.422+0000 7f7206021700 1 -- 172.21.15.42:0/3262361680 >> 172.21.15.42:0/3262361680 conn(0x7f72000a00a0 msgr2=0x7f72000596f0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:52:25.439 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/contextutil.py", line 34, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-pdonnell-testing-20191204.231200/qa/tasks/ceph.py", line 1910, in task yield File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 86, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 65, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-pdonnell-testing-20191204.231200/qa/tasks/workunit.py", line 136, in task cleanup=cleanup) File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-pdonnell-testing-20191204.231200/qa/tasks/workunit.py", line 286, in _spawn_on_all_clients timeout=timeout) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/parallel.py", line 87, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/parallel.py", line 101, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/parallel.py", line 37, in resurrect_traceback reraise(*exc_info) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/parallel.py", line 24, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-pdonnell-testing-20191204.231200/qa/tasks/workunit.py", line 409, in _run_tests label="workunit test {workunit}".format(workunit=workunit) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/remote.py", line 198, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 433, in run r.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 158, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 180, in _raise_for_status node=self.hostname, label=self.label CommandFailedError: Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithi042 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c5290805033483ba0cec165c06bde07dae5f19ec TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-multiple-capsnaps.sh' 2019-12-05T06:52:25.442 INFO:teuthology.misc:Shutting down mds daemons... 2019-12-05T06:52:25.443 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2019-12-05T06:52:25.443 INFO:teuthology.orchestra.run:waiting for 300 2019-12-05T06:52:25.459 INFO:tasks.ceph.mds.a:Stopped 2019-12-05T06:52:25.459 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2019-12-05T06:52:25.459 INFO:teuthology.orchestra.run:waiting for 300 2019-12-05T06:52:25.471 INFO:tasks.ceph.mds.b:Stopped 2019-12-05T06:52:25.471 INFO:teuthology.misc:Shutting down osd daemons... 2019-12-05T06:52:25.471 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2019-12-05T06:52:25.471 INFO:teuthology.orchestra.run:waiting for 300 2019-12-05T06:52:28.240 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2019-12-05T06:52:31.573 INFO:tasks.ceph.osd.1:Stopped 2019-12-05T06:52:31.574 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2019-12-05T06:52:31.574 INFO:teuthology.orchestra.run:waiting for 300 2019-12-05T06:52:31.651 INFO:tasks.ceph.osd.0:Stopped 2019-12-05T06:52:31.652 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2019-12-05T06:52:31.652 INFO:teuthology.orchestra.run:waiting for 300 2019-12-05T06:52:31.732 INFO:tasks.ceph.osd.3:Stopped 2019-12-05T06:52:31.732 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2019-12-05T06:52:31.732 INFO:teuthology.orchestra.run:waiting for 300 2019-12-05T06:52:31.833 INFO:tasks.ceph.osd.2:Stopped 2019-12-05T06:52:31.833 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2019-12-05T06:52:31.833 INFO:teuthology.orchestra.run:waiting for 300 2019-12-05T06:52:31.920 INFO:tasks.ceph.osd.5:Stopped 2019-12-05T06:52:31.920 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2019-12-05T06:52:31.921 INFO:teuthology.orchestra.run:waiting for 300 2019-12-05T06:52:32.001 INFO:tasks.ceph.osd.4:Stopped 2019-12-05T06:52:32.002 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2019-12-05T06:52:32.002 INFO:teuthology.orchestra.run:waiting for 300 2019-12-05T06:52:32.080 INFO:tasks.ceph.osd.7:Stopped 2019-12-05T06:52:32.081 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2019-12-05T06:52:32.081 INFO:teuthology.orchestra.run:waiting for 300 2019-12-05T06:52:32.167 INFO:tasks.ceph.osd.6:Stopped 2019-12-05T06:52:32.168 INFO:teuthology.misc:Shutting down mgr daemons... 2019-12-05T06:52:32.168 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2019-12-05T06:52:32.168 INFO:teuthology.orchestra.run:waiting for 300 2019-12-05T06:52:32.189 INFO:tasks.ceph.mgr.y:Stopped 2019-12-05T06:52:32.190 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2019-12-05T06:52:32.190 INFO:teuthology.orchestra.run:waiting for 300 2019-12-05T06:52:32.215 INFO:tasks.ceph.mgr.x:Stopped 2019-12-05T06:52:32.215 INFO:teuthology.misc:Shutting down mon daemons... 2019-12-05T06:52:32.215 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2019-12-05T06:52:32.215 INFO:teuthology.orchestra.run:waiting for 300 2019-12-05T06:52:32.249 INFO:tasks.ceph.mon.a:Stopped 2019-12-05T06:52:32.250 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2019-12-05T06:52:32.250 INFO:teuthology.orchestra.run:waiting for 300 2019-12-05T06:52:32.282 INFO:tasks.ceph.mon.c:Stopped 2019-12-05T06:52:32.282 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2019-12-05T06:52:32.282 INFO:teuthology.orchestra.run:waiting for 300 2019-12-05T06:52:32.353 INFO:tasks.ceph.mon.b:Stopped 2019-12-05T06:52:32.354 INFO:tasks.ceph:Checking cluster log for badness... 2019-12-05T06:52:32.354 INFO:teuthology.orchestra.run.smithi042:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'not responding, replacing' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2019-12-05T06:52:32.433 INFO:teuthology.orchestra.run.smithi042.stdout:2019-12-05T06:41:51.221874+0000 mon.b (mon.0) 1510 : cluster [WRN] Health check failed: 1 clients failing to respond to capability release (MDS_CLIENT_LATE_RELEASE) 2019-12-05T06:52:32.434 WARNING:tasks.ceph:Found errors (ERR|WRN|SEC) in cluster log 2019-12-05T06:52:32.434 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi042.front.sepia.ceph.com 2019-12-05T06:52:32.434 INFO:teuthology.orchestra.run.smithi042:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2019-12-05T06:52:32.816 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi042.front.sepia.ceph.com 2019-12-05T06:52:32.816 INFO:teuthology.orchestra.run.smithi042:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2019-12-05T06:52:33.020 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi042.front.sepia.ceph.com 2019-12-05T06:52:33.020 INFO:teuthology.orchestra.run.smithi042:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2019-12-05T06:52:33.177 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi042.front.sepia.ceph.com 2019-12-05T06:52:33.177 INFO:teuthology.orchestra.run.smithi042:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2019-12-05T06:52:33.240 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 has been restored 2019-12-05T06:52:33.337 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi040.front.sepia.ceph.com 2019-12-05T06:52:33.338 INFO:teuthology.orchestra.run.smithi040:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2019-12-05T06:52:33.632 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi040.front.sepia.ceph.com 2019-12-05T06:52:33.632 INFO:teuthology.orchestra.run.smithi040:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2019-12-05T06:52:33.975 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi040.front.sepia.ceph.com 2019-12-05T06:52:33.975 INFO:teuthology.orchestra.run.smithi040:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2019-12-05T06:52:34.139 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi040.front.sepia.ceph.com 2019-12-05T06:52:34.140 INFO:teuthology.orchestra.run.smithi040:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2019-12-05T06:52:34.313 INFO:tasks.ceph:Archiving mon data... 2019-12-05T06:52:34.315 DEBUG:teuthology.misc:Transferring archived files from smithi042:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/pdonnell-2019-12-05_02:12:53-fs-wip-pdonnell-testing-20191204.231200-distro-basic-smithi/4567617/data/mon.a.tgz 2019-12-05T06:52:34.315 INFO:teuthology.orchestra.run.smithi042:> mktemp 2019-12-05T06:52:34.347 INFO:teuthology.orchestra.run.smithi042.stdout:/tmp/tmp.ZCOoCU5myO 2019-12-05T06:52:34.348 INFO:teuthology.orchestra.run.smithi042:> sudo tar cz -f /tmp/tmp.ZCOoCU5myO -C /var/lib/ceph/mon/ceph-a -- . 2019-12-05T06:52:34.654 INFO:teuthology.orchestra.run.smithi042:> sudo chmod 0666 /tmp/tmp.ZCOoCU5myO 2019-12-05T06:52:34.771 DEBUG:teuthology.orchestra.remote:smithi042:/tmp/tmp.ZCOoCU5myO is 764KB 2019-12-05T06:52:34.883 INFO:teuthology.orchestra.run.smithi042:> rm -fr /tmp/tmp.ZCOoCU5myO 2019-12-05T06:52:34.891 DEBUG:teuthology.misc:Transferring archived files from smithi040:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/pdonnell-2019-12-05_02:12:53-fs-wip-pdonnell-testing-20191204.231200-distro-basic-smithi/4567617/data/mon.b.tgz 2019-12-05T06:52:34.891 INFO:teuthology.orchestra.run.smithi040:> mktemp 2019-12-05T06:52:34.900 INFO:teuthology.orchestra.run.smithi040.stdout:/tmp/tmp.EHg13f9KFE 2019-12-05T06:52:34.900 INFO:teuthology.orchestra.run.smithi040:> sudo tar cz -f /tmp/tmp.EHg13f9KFE -C /var/lib/ceph/mon/ceph-b -- . 2019-12-05T06:52:35.192 INFO:teuthology.orchestra.run.smithi040:> sudo chmod 0666 /tmp/tmp.EHg13f9KFE 2019-12-05T06:52:35.280 DEBUG:teuthology.orchestra.remote:smithi040:/tmp/tmp.EHg13f9KFE is 754KB 2019-12-05T06:52:35.388 INFO:teuthology.orchestra.run.smithi040:> rm -fr /tmp/tmp.EHg13f9KFE 2019-12-05T06:52:35.396 DEBUG:teuthology.misc:Transferring archived files from smithi040:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/pdonnell-2019-12-05_02:12:53-fs-wip-pdonnell-testing-20191204.231200-distro-basic-smithi/4567617/data/mon.c.tgz 2019-12-05T06:52:35.396 INFO:teuthology.orchestra.run.smithi040:> mktemp 2019-12-05T06:52:35.474 INFO:teuthology.orchestra.run.smithi040.stdout:/tmp/tmp.1QY9d1DFCQ 2019-12-05T06:52:35.474 INFO:teuthology.orchestra.run.smithi040:> sudo tar cz -f /tmp/tmp.1QY9d1DFCQ -C /var/lib/ceph/mon/ceph-c -- . 2019-12-05T06:52:35.764 INFO:teuthology.orchestra.run.smithi040:> sudo chmod 0666 /tmp/tmp.1QY9d1DFCQ 2019-12-05T06:52:35.853 DEBUG:teuthology.orchestra.remote:smithi040:/tmp/tmp.1QY9d1DFCQ is 764KB 2019-12-05T06:52:35.929 INFO:teuthology.orchestra.run.smithi040:> rm -fr /tmp/tmp.1QY9d1DFCQ 2019-12-05T06:52:35.937 INFO:tasks.ceph:Cleaning ceph cluster... 2019-12-05T06:52:35.937 INFO:teuthology.orchestra.run.smithi040:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2019-12-05T06:52:36.004 INFO:teuthology.orchestra.run.smithi042:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2019-12-05T06:52:36.020 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2019-12-05T06:52:36.020 INFO:teuthology.orchestra.run.smithi042:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2019-12-05T06:52:36.091 INFO:teuthology.orchestra.run.smithi040:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2019-12-05T06:52:36.129 INFO:teuthology.orchestra.run.smithi040.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2019-12-05T06:52:36.131 INFO:teuthology.orchestra.run.smithi042.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2019-12-05T06:52:36.147 INFO:tasks.ceph:Archiving crash dumps... 2019-12-05T06:52:36.149 DEBUG:teuthology.misc:Transferring archived files from smithi042:/var/lib/ceph/crash to /home/teuthworker/archive/pdonnell-2019-12-05_02:12:53-fs-wip-pdonnell-testing-20191204.231200-distro-basic-smithi/4567617/remote/smithi042/crash 2019-12-05T06:52:36.151 INFO:teuthology.orchestra.run.smithi042:> sudo tar cz -f - -C /var/lib/ceph/crash -- . 2019-12-05T06:52:36.236 DEBUG:teuthology.misc:Transferring archived files from smithi040:/var/lib/ceph/crash to /home/teuthworker/archive/pdonnell-2019-12-05_02:12:53-fs-wip-pdonnell-testing-20191204.231200-distro-basic-smithi/4567617/remote/smithi040/crash 2019-12-05T06:52:36.237 INFO:teuthology.orchestra.run.smithi040:> sudo tar cz -f - -C /var/lib/ceph/crash -- . 2019-12-05T06:52:36.258 INFO:tasks.ceph:Shutting down logrotate 2019-12-05T06:52:36.259 INFO:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:52:36.327 INFO:teuthology.orchestra.run.smithi042:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:52:36.399 INFO:teuthology.orchestra.run.smithi040:> sudo rm /etc/logrotate.d/ceph-test.conf 2019-12-05T06:52:36.419 INFO:teuthology.orchestra.run.smithi042:> sudo rm /etc/logrotate.d/ceph-test.conf 2019-12-05T06:52:36.435 INFO:tasks.ceph:Compressing logs... 2019-12-05T06:52:36.436 INFO:teuthology.orchestra.run.smithi040:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2019-12-05T06:52:36.471 INFO:teuthology.orchestra.run.smithi042:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2019-12-05T06:57:20.226 INFO:tasks.ceph:Archiving logs... 2019-12-05T06:57:20.227 DEBUG:teuthology.misc:Transferring archived files from smithi042:/var/log/ceph to /home/teuthworker/archive/pdonnell-2019-12-05_02:12:53-fs-wip-pdonnell-testing-20191204.231200-distro-basic-smithi/4567617/remote/smithi042/log 2019-12-05T06:57:20.228 INFO:teuthology.orchestra.run.smithi042:> sudo tar cz -f - -C /var/log/ceph -- . 2019-12-05T06:58:15.379 DEBUG:teuthology.misc:Transferring archived files from smithi040:/var/log/ceph to /home/teuthworker/archive/pdonnell-2019-12-05_02:12:53-fs-wip-pdonnell-testing-20191204.231200-distro-basic-smithi/4567617/remote/smithi040/log 2019-12-05T06:58:15.381 INFO:teuthology.orchestra.run.smithi040:> sudo tar cz -f - -C /var/log/ceph -- . 2019-12-05T06:58:47.519 DEBUG:teuthology.run_tasks:Unwinding manager install 2019-12-05T06:58:47.527 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/contextutil.py", line 34, in nested yield vars File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/install/__init__.py", line 598, in task yield File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 86, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 65, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-pdonnell-testing-20191204.231200/qa/tasks/workunit.py", line 136, in task cleanup=cleanup) File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-pdonnell-testing-20191204.231200/qa/tasks/workunit.py", line 286, in _spawn_on_all_clients timeout=timeout) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/parallel.py", line 87, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/parallel.py", line 101, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/parallel.py", line 37, in resurrect_traceback reraise(*exc_info) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/parallel.py", line 24, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-pdonnell-testing-20191204.231200/qa/tasks/workunit.py", line 409, in _run_tests label="workunit test {workunit}".format(workunit=workunit) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/remote.py", line 198, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 433, in run r.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 158, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 180, in _raise_for_status node=self.hostname, label=self.label CommandFailedError: Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithi042 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c5290805033483ba0cec165c06bde07dae5f19ec TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-multiple-capsnaps.sh' 2019-12-05T06:58:47.535 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits... 2019-12-05T06:58:47.536 INFO:teuthology.orchestra.run.smithi040:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits 2019-12-05T06:58:47.570 INFO:teuthology.orchestra.run.smithi042:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits 2019-12-05T06:58:47.605 INFO:teuthology.orchestra.run.smithi042:> sudo lsb_release -is 2019-12-05T06:58:47.899 INFO:teuthology.orchestra.run.smithi042.stdout:Ubuntu 2019-12-05T06:58:47.906 DEBUG:teuthology.misc:System to be installed: Ubuntu 2019-12-05T06:58:47.906 INFO:teuthology.orchestra.run.smithi040:> sudo lsb_release -is 2019-12-05T06:58:47.906 INFO:teuthology.task.install.deb:Removing packages: ceph, ceph-daemon, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, radosgw, python-ceph, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, python3-cephfs, cephfs-shell on Debian system. 2019-12-05T06:58:47.907 INFO:teuthology.orchestra.run.smithi042:> for d in ceph ceph-daemon ceph-mds ceph-mgr ceph-common ceph-fuse ceph-test radosgw python-ceph libcephfs2 libcephfs-dev librados2 librbd1 rbd-fuse python3-cephfs cephfs-shell ; do sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" purge $d || true ; done 2019-12-05T06:58:47.958 INFO:teuthology.orchestra.run.smithi040.stdout:Ubuntu 2019-12-05T06:58:47.963 DEBUG:teuthology.misc:System to be installed: Ubuntu 2019-12-05T06:58:47.963 INFO:teuthology.task.install.deb:Removing packages: ceph, ceph-daemon, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, radosgw, python-ceph, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, python3-cephfs, cephfs-shell on Debian system. 2019-12-05T06:58:47.963 INFO:teuthology.orchestra.run.smithi040:> for d in ceph ceph-daemon ceph-mds ceph-mgr ceph-common ceph-fuse ceph-test radosgw python-ceph libcephfs2 libcephfs-dev librados2 librbd1 rbd-fuse python3-cephfs cephfs-shell ; do sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" purge $d || true ; done 2019-12-05T06:58:48.047 INFO:teuthology.orchestra.run.smithi040.stdout:Reading package lists... 2019-12-05T06:58:48.302 INFO:teuthology.orchestra.run.smithi040.stdout:Building dependency tree... 2019-12-05T06:58:48.302 INFO:teuthology.orchestra.run.smithi040.stdout:Reading state information... 2019-12-05T06:58:48.489 INFO:teuthology.orchestra.run.smithi040.stdout:The following packages were automatically installed and are no longer required: 2019-12-05T06:58:48.490 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mon ceph-osd 2019-12-05T06:58:48.490 INFO:teuthology.orchestra.run.smithi040.stdout:Use 'sudo apt autoremove' to remove them. 2019-12-05T06:58:48.511 INFO:teuthology.orchestra.run.smithi040.stdout:The following packages will be REMOVED: 2019-12-05T06:58:48.512 INFO:teuthology.orchestra.run.smithi040.stdout: ceph* 2019-12-05T06:58:48.553 INFO:teuthology.orchestra.run.smithi042.stdout:Reading package lists... 2019-12-05T06:58:48.803 INFO:teuthology.orchestra.run.smithi042.stdout:Building dependency tree... 2019-12-05T06:58:48.813 INFO:teuthology.orchestra.run.smithi042.stdout:Reading state information... 2019-12-05T06:58:48.868 INFO:teuthology.orchestra.run.smithi040.stdout:0 upgraded, 0 newly installed, 1 to remove and 51 not upgraded. 2019-12-05T06:58:48.868 INFO:teuthology.orchestra.run.smithi040.stdout:After this operation, 9,216 B disk space will be freed. 2019-12-05T06:58:48.974 INFO:teuthology.orchestra.run.smithi040.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 144013 files and directories currently installed.) 2019-12-05T06:58:48.978 INFO:teuthology.orchestra.run.smithi040.stdout:Removing ceph (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T06:58:48.983 INFO:teuthology.orchestra.run.smithi042.stdout:The following packages were automatically installed and are no longer required: 2019-12-05T06:58:48.985 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-mon ceph-osd 2019-12-05T06:58:48.985 INFO:teuthology.orchestra.run.smithi042.stdout:Use 'sudo apt autoremove' to remove them. 2019-12-05T06:58:49.196 INFO:teuthology.orchestra.run.smithi042.stdout:The following packages will be REMOVED: 2019-12-05T06:58:49.197 INFO:teuthology.orchestra.run.smithi042.stdout: ceph* 2019-12-05T06:58:49.752 INFO:teuthology.orchestra.run.smithi040.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2019-12-05T06:58:49.828 INFO:teuthology.orchestra.run.smithi040.stdout:Reading package lists... 2019-12-05T06:58:49.842 INFO:teuthology.orchestra.run.smithi042.stdout:0 upgraded, 0 newly installed, 1 to remove and 51 not upgraded. 2019-12-05T06:58:49.843 INFO:teuthology.orchestra.run.smithi042.stdout:After this operation, 9,216 B disk space will be freed. 2019-12-05T06:58:50.080 INFO:teuthology.orchestra.run.smithi040.stdout:Building dependency tree... 2019-12-05T06:58:50.081 INFO:teuthology.orchestra.run.smithi040.stdout:Reading state information... 2019-12-05T06:58:50.254 INFO:teuthology.orchestra.run.smithi040.stdout:The following packages were automatically installed and are no longer required: 2019-12-05T06:58:50.255 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mon ceph-osd 2019-12-05T06:58:50.255 INFO:teuthology.orchestra.run.smithi040.stdout:Use 'sudo apt autoremove' to remove them. 2019-12-05T06:58:50.286 INFO:teuthology.orchestra.run.smithi040.stdout:The following packages will be REMOVED: 2019-12-05T06:58:50.287 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-daemon* ceph-mgr-ssh* 2019-12-05T06:58:50.666 INFO:teuthology.orchestra.run.smithi040.stdout:0 upgraded, 0 newly installed, 2 to remove and 51 not upgraded. 2019-12-05T06:58:50.666 INFO:teuthology.orchestra.run.smithi040.stdout:After this operation, 169 kB disk space will be freed. 2019-12-05T06:58:50.771 INFO:teuthology.orchestra.run.smithi040.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 144012 files and directories currently installed.) 2019-12-05T06:58:50.775 INFO:teuthology.orchestra.run.smithi040.stdout:Removing ceph-mgr-ssh (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T06:58:50.991 INFO:teuthology.orchestra.run.smithi040.stdout:dpkg: warning: while removing ceph-mgr-ssh, directory '/usr/share/ceph/mgr/ssh' not empty so not removed 2019-12-05T06:58:51.149 INFO:teuthology.orchestra.run.smithi040.stdout:Removing ceph-daemon (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T06:58:51.207 INFO:teuthology.orchestra.run.smithi042.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 144013 files and directories currently installed.) 2019-12-05T06:58:51.211 INFO:teuthology.orchestra.run.smithi042.stdout:Removing ceph (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T06:58:51.322 INFO:teuthology.orchestra.run.smithi040.stdout:Looking for files to backup/remove ... 2019-12-05T06:58:51.326 INFO:teuthology.orchestra.run.smithi040.stdout:Removing files ... 2019-12-05T06:58:51.333 INFO:teuthology.orchestra.run.smithi040.stdout:Removing user `cephdaemon' ... 2019-12-05T06:58:51.333 INFO:teuthology.orchestra.run.smithi040.stdout:Warning: group `nogroup' has no more members. 2019-12-05T06:58:51.466 INFO:teuthology.orchestra.run.smithi040.stdout:Done. 2019-12-05T06:58:51.894 INFO:teuthology.orchestra.run.smithi040.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 143999 files and directories currently installed.) 2019-12-05T06:58:51.898 INFO:teuthology.orchestra.run.smithi040.stdout:Purging configuration files for ceph-daemon (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T06:58:52.374 INFO:teuthology.orchestra.run.smithi042.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2019-12-05T06:58:52.456 INFO:teuthology.orchestra.run.smithi042.stdout:Reading package lists... 2019-12-05T06:58:52.613 INFO:teuthology.orchestra.run.smithi040.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2019-12-05T06:58:52.698 INFO:teuthology.orchestra.run.smithi040.stdout:Reading package lists... 2019-12-05T06:58:52.712 INFO:teuthology.orchestra.run.smithi042.stdout:Building dependency tree... 2019-12-05T06:58:52.712 INFO:teuthology.orchestra.run.smithi042.stdout:Reading state information... 2019-12-05T06:58:52.885 INFO:teuthology.orchestra.run.smithi042.stdout:The following packages were automatically installed and are no longer required: 2019-12-05T06:58:52.886 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-mon ceph-osd 2019-12-05T06:58:52.886 INFO:teuthology.orchestra.run.smithi042.stdout:Use 'sudo apt autoremove' to remove them. 2019-12-05T06:58:52.916 INFO:teuthology.orchestra.run.smithi042.stdout:The following packages will be REMOVED: 2019-12-05T06:58:52.917 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-daemon* ceph-mgr-ssh* 2019-12-05T06:58:52.954 INFO:teuthology.orchestra.run.smithi040.stdout:Building dependency tree... 2019-12-05T06:58:52.955 INFO:teuthology.orchestra.run.smithi040.stdout:Reading state information... 2019-12-05T06:58:53.127 INFO:teuthology.orchestra.run.smithi040.stdout:The following packages were automatically installed and are no longer required: 2019-12-05T06:58:53.128 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mon ceph-osd 2019-12-05T06:58:53.128 INFO:teuthology.orchestra.run.smithi040.stdout:Use 'sudo apt autoremove' to remove them. 2019-12-05T06:58:53.149 INFO:teuthology.orchestra.run.smithi040.stdout:The following packages will be REMOVED: 2019-12-05T06:58:53.149 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mds* 2019-12-05T06:58:53.293 INFO:teuthology.orchestra.run.smithi042.stdout:0 upgraded, 0 newly installed, 2 to remove and 51 not upgraded. 2019-12-05T06:58:53.294 INFO:teuthology.orchestra.run.smithi042.stdout:After this operation, 169 kB disk space will be freed. 2019-12-05T06:58:53.409 INFO:teuthology.orchestra.run.smithi042.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 144012 files and directories currently installed.) 2019-12-05T06:58:53.413 INFO:teuthology.orchestra.run.smithi042.stdout:Removing ceph-mgr-ssh (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T06:58:53.507 INFO:teuthology.orchestra.run.smithi040.stdout:0 upgraded, 0 newly installed, 1 to remove and 51 not upgraded. 2019-12-05T06:58:53.508 INFO:teuthology.orchestra.run.smithi040.stdout:After this operation, 6,106 kB disk space will be freed. 2019-12-05T06:58:53.629 INFO:teuthology.orchestra.run.smithi042.stdout:dpkg: warning: while removing ceph-mgr-ssh, directory '/usr/share/ceph/mgr/ssh' not empty so not removed 2019-12-05T06:58:53.637 INFO:teuthology.orchestra.run.smithi040.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 143998 files and directories currently installed.) 2019-12-05T06:58:53.642 INFO:teuthology.orchestra.run.smithi040.stdout:Removing ceph-mds (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T06:58:53.779 INFO:teuthology.orchestra.run.smithi042.stdout:Removing ceph-daemon (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T06:58:54.020 INFO:teuthology.orchestra.run.smithi042.stdout:Looking for files to backup/remove ... 2019-12-05T06:58:54.028 INFO:teuthology.orchestra.run.smithi042.stdout:Removing files ... 2019-12-05T06:58:54.055 INFO:teuthology.orchestra.run.smithi042.stdout:Removing user `cephdaemon' ... 2019-12-05T06:58:54.056 INFO:teuthology.orchestra.run.smithi042.stdout:Warning: group `nogroup' has no more members. 2019-12-05T06:58:54.235 INFO:teuthology.orchestra.run.smithi042.stdout:Done. 2019-12-05T06:58:54.411 INFO:teuthology.orchestra.run.smithi040.stdout:Processing triggers for man-db (2.8.3-2ubuntu0.1) ... 2019-12-05T06:58:54.665 INFO:teuthology.orchestra.run.smithi042.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 143999 files and directories currently installed.) 2019-12-05T06:58:54.669 INFO:teuthology.orchestra.run.smithi042.stdout:Purging configuration files for ceph-daemon (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T06:58:55.150 INFO:teuthology.orchestra.run.smithi040.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 143992 files and directories currently installed.) 2019-12-05T06:58:55.154 INFO:teuthology.orchestra.run.smithi040.stdout:Purging configuration files for ceph-mds (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T06:58:55.370 INFO:teuthology.orchestra.run.smithi042.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2019-12-05T06:58:55.449 INFO:teuthology.orchestra.run.smithi042.stdout:Reading package lists... 2019-12-05T06:58:55.697 INFO:teuthology.orchestra.run.smithi042.stdout:Building dependency tree... 2019-12-05T06:58:55.698 INFO:teuthology.orchestra.run.smithi042.stdout:Reading state information... 2019-12-05T06:58:55.782 INFO:teuthology.orchestra.run.smithi040.stdout:dpkg: warning: while removing ceph-mds, directory '/var/lib/ceph/mds' not empty so not removed 2019-12-05T06:58:55.867 INFO:teuthology.orchestra.run.smithi042.stdout:The following packages were automatically installed and are no longer required: 2019-12-05T06:58:55.869 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-mon ceph-osd 2019-12-05T06:58:55.869 INFO:teuthology.orchestra.run.smithi042.stdout:Use 'sudo apt autoremove' to remove them. 2019-12-05T06:58:55.887 INFO:teuthology.orchestra.run.smithi042.stdout:The following packages will be REMOVED: 2019-12-05T06:58:55.888 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-mds* 2019-12-05T06:58:56.280 INFO:teuthology.orchestra.run.smithi042.stdout:0 upgraded, 0 newly installed, 1 to remove and 51 not upgraded. 2019-12-05T06:58:56.280 INFO:teuthology.orchestra.run.smithi042.stdout:After this operation, 6,106 kB disk space will be freed. 2019-12-05T06:58:56.299 INFO:teuthology.orchestra.run.smithi040.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2019-12-05T06:58:56.378 INFO:teuthology.orchestra.run.smithi040.stdout:Reading package lists... 2019-12-05T06:58:56.395 INFO:teuthology.orchestra.run.smithi042.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 143998 files and directories currently installed.) 2019-12-05T06:58:56.399 INFO:teuthology.orchestra.run.smithi042.stdout:Removing ceph-mds (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T06:58:56.636 INFO:teuthology.orchestra.run.smithi040.stdout:Building dependency tree... 2019-12-05T06:58:56.636 INFO:teuthology.orchestra.run.smithi040.stdout:Reading state information... 2019-12-05T06:58:56.817 INFO:teuthology.orchestra.run.smithi040.stdout:The following packages were automatically installed and are no longer required: 2019-12-05T06:58:56.819 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mon ceph-osd formencode-i18n libjs-sphinxdoc libjs-underscore 2019-12-05T06:58:56.819 INFO:teuthology.orchestra.run.smithi040.stdout: python-attr python-bcrypt python-bs4 python-certifi python-chardet 2019-12-05T06:58:56.819 INFO:teuthology.orchestra.run.smithi040.stdout: python-cherrypy3 python-dnspython python-fasteners python-formencode 2019-12-05T06:58:56.820 INFO:teuthology.orchestra.run.smithi040.stdout: python-funcsigs python-html5lib python-httplib2 python-joblib python-jwt 2019-12-05T06:58:56.820 INFO:teuthology.orchestra.run.smithi040.stdout: python-kubernetes python-logutils python-lxml python-mako python-markupsafe 2019-12-05T06:58:56.820 INFO:teuthology.orchestra.run.smithi040.stdout: python-monotonic python-oauth2client python-openid python-paste 2019-12-05T06:58:56.820 INFO:teuthology.orchestra.run.smithi040.stdout: python-pastedeploy python-pastedeploy-tpl python-pastescript python-pecan 2019-12-05T06:58:56.820 INFO:teuthology.orchestra.run.smithi040.stdout: python-pluggy python-py python-pyasn1 python-pyasn1-modules python-pyinotify 2019-12-05T06:58:56.820 INFO:teuthology.orchestra.run.smithi040.stdout: python-pytest python-repoze.lru python-requests python-routes python-rsa 2019-12-05T06:58:56.820 INFO:teuthology.orchestra.run.smithi040.stdout: python-scgi python-simplegeneric python-simplejson python-singledispatch 2019-12-05T06:58:56.821 INFO:teuthology.orchestra.run.smithi040.stdout: python-sklearn python-sklearn-lib python-sqlalchemy python-sqlalchemy-ext 2019-12-05T06:58:56.821 INFO:teuthology.orchestra.run.smithi040.stdout: python-tempita python-uritemplate python-urllib3 python-waitress 2019-12-05T06:58:56.821 INFO:teuthology.orchestra.run.smithi040.stdout: python-webencodings python-webob python-websocket python-webtest 2019-12-05T06:58:56.821 INFO:teuthology.orchestra.run.smithi040.stdout: python-werkzeug python-yaml 2019-12-05T06:58:56.821 INFO:teuthology.orchestra.run.smithi040.stdout:Use 'sudo apt autoremove' to remove them. 2019-12-05T06:58:56.900 INFO:teuthology.orchestra.run.smithi040.stdout:The following packages will be REMOVED: 2019-12-05T06:58:56.900 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr* ceph-mgr-dashboard* ceph-mgr-diskprediction-cloud* 2019-12-05T06:58:56.901 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-diskprediction-local* ceph-mgr-k8sevents* ceph-mgr-rook* 2019-12-05T06:58:57.266 INFO:teuthology.orchestra.run.smithi040.stdout:0 upgraded, 0 newly installed, 6 to remove and 51 not upgraded. 2019-12-05T06:58:57.266 INFO:teuthology.orchestra.run.smithi040.stdout:After this operation, 93.2 MB disk space will be freed. 2019-12-05T06:58:57.317 INFO:teuthology.orchestra.run.smithi042.stdout:Processing triggers for man-db (2.8.3-2ubuntu0.1) ... 2019-12-05T06:58:57.391 INFO:teuthology.orchestra.run.smithi040.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 143991 files and directories currently installed.) 2019-12-05T06:58:57.396 INFO:teuthology.orchestra.run.smithi040.stdout:Removing ceph-mgr-rook (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T06:58:57.622 INFO:teuthology.orchestra.run.smithi040.stdout:dpkg: warning: while removing ceph-mgr-rook, directory '/usr/share/ceph/mgr/rook' not empty so not removed 2019-12-05T06:58:57.926 INFO:teuthology.orchestra.run.smithi040.stdout:Removing ceph-mgr-k8sevents (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T06:58:58.140 INFO:teuthology.orchestra.run.smithi042.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 143992 files and directories currently installed.) 2019-12-05T06:58:58.144 INFO:teuthology.orchestra.run.smithi042.stdout:Purging configuration files for ceph-mds (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T06:58:58.302 INFO:teuthology.orchestra.run.smithi040.stdout:Removing ceph-mgr-dashboard (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T06:58:58.548 INFO:teuthology.orchestra.run.smithi040.stdout:dpkg: warning: while removing ceph-mgr-dashboard, directory '/usr/share/ceph/mgr/dashboard/services' not empty so not removed 2019-12-05T06:58:58.549 INFO:teuthology.orchestra.run.smithi040.stdout:dpkg: warning: while removing ceph-mgr-dashboard, directory '/usr/share/ceph/mgr/dashboard/plugins' not empty so not removed 2019-12-05T06:58:58.549 INFO:teuthology.orchestra.run.smithi040.stdout:dpkg: warning: while removing ceph-mgr-dashboard, directory '/usr/share/ceph/mgr/dashboard/controllers' not empty so not removed 2019-12-05T06:58:58.731 INFO:teuthology.orchestra.run.smithi040.stdout:Removing ceph-mgr-diskprediction-cloud (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T06:58:58.752 INFO:teuthology.orchestra.run.smithi042.stdout:dpkg: warning: while removing ceph-mds, directory '/var/lib/ceph/mds' not empty so not removed 2019-12-05T06:58:58.935 INFO:teuthology.orchestra.run.smithi040.stdout:dpkg: warning: while removing ceph-mgr-diskprediction-cloud, directory '/usr/share/ceph/mgr/diskprediction_cloud/common' not empty so not removed 2019-12-05T06:58:58.936 INFO:teuthology.orchestra.run.smithi040.stdout:dpkg: warning: while removing ceph-mgr-diskprediction-cloud, directory '/usr/share/ceph/mgr/diskprediction_cloud/agent/metrics' not empty so not removed 2019-12-05T06:58:59.086 INFO:teuthology.orchestra.run.smithi040.stdout:Removing ceph-mgr-diskprediction-local (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T06:58:59.331 INFO:teuthology.orchestra.run.smithi040.stdout:dpkg: warning: while removing ceph-mgr-diskprediction-local, directory '/usr/share/ceph/mgr/diskprediction_local' not empty so not removed 2019-12-05T06:58:59.338 INFO:teuthology.orchestra.run.smithi042.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2019-12-05T06:58:59.421 INFO:teuthology.orchestra.run.smithi042.stdout:Reading package lists... 2019-12-05T06:58:59.513 INFO:teuthology.orchestra.run.smithi040.stdout:Removing ceph-mgr (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T06:58:59.680 INFO:teuthology.orchestra.run.smithi042.stdout:Building dependency tree... 2019-12-05T06:58:59.681 INFO:teuthology.orchestra.run.smithi042.stdout:Reading state information... 2019-12-05T06:58:59.855 INFO:teuthology.orchestra.run.smithi042.stdout:The following packages were automatically installed and are no longer required: 2019-12-05T06:58:59.858 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-mon ceph-osd formencode-i18n libjs-sphinxdoc libjs-underscore 2019-12-05T06:58:59.858 INFO:teuthology.orchestra.run.smithi042.stdout: python-attr python-bcrypt python-bs4 python-certifi python-chardet 2019-12-05T06:58:59.858 INFO:teuthology.orchestra.run.smithi042.stdout: python-cherrypy3 python-dnspython python-fasteners python-formencode 2019-12-05T06:58:59.858 INFO:teuthology.orchestra.run.smithi042.stdout: python-funcsigs python-html5lib python-httplib2 python-joblib python-jwt 2019-12-05T06:58:59.859 INFO:teuthology.orchestra.run.smithi042.stdout: python-kubernetes python-logutils python-lxml python-mako python-markupsafe 2019-12-05T06:58:59.859 INFO:teuthology.orchestra.run.smithi042.stdout: python-monotonic python-oauth2client python-openid python-paste 2019-12-05T06:58:59.860 INFO:teuthology.orchestra.run.smithi042.stdout: python-pastedeploy python-pastedeploy-tpl python-pastescript python-pecan 2019-12-05T06:58:59.860 INFO:teuthology.orchestra.run.smithi042.stdout: python-pluggy python-py python-pyasn1 python-pyasn1-modules python-pyinotify 2019-12-05T06:58:59.860 INFO:teuthology.orchestra.run.smithi042.stdout: python-pytest python-repoze.lru python-requests python-routes python-rsa 2019-12-05T06:58:59.860 INFO:teuthology.orchestra.run.smithi042.stdout: python-scgi python-simplegeneric python-simplejson python-singledispatch 2019-12-05T06:58:59.860 INFO:teuthology.orchestra.run.smithi042.stdout: python-sklearn python-sklearn-lib python-sqlalchemy python-sqlalchemy-ext 2019-12-05T06:58:59.860 INFO:teuthology.orchestra.run.smithi042.stdout: python-tempita python-uritemplate python-urllib3 python-waitress 2019-12-05T06:58:59.861 INFO:teuthology.orchestra.run.smithi042.stdout: python-webencodings python-webob python-websocket python-webtest 2019-12-05T06:58:59.861 INFO:teuthology.orchestra.run.smithi042.stdout: python-werkzeug python-yaml 2019-12-05T06:58:59.861 INFO:teuthology.orchestra.run.smithi042.stdout:Use 'sudo apt autoremove' to remove them. 2019-12-05T06:58:59.937 INFO:teuthology.orchestra.run.smithi042.stdout:The following packages will be REMOVED: 2019-12-05T06:58:59.937 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-mgr* ceph-mgr-dashboard* ceph-mgr-diskprediction-cloud* 2019-12-05T06:58:59.941 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-mgr-diskprediction-local* ceph-mgr-k8sevents* ceph-mgr-rook* 2019-12-05T06:59:00.300 INFO:teuthology.orchestra.run.smithi042.stdout:0 upgraded, 0 newly installed, 6 to remove and 51 not upgraded. 2019-12-05T06:59:00.300 INFO:teuthology.orchestra.run.smithi042.stdout:After this operation, 93.2 MB disk space will be freed. 2019-12-05T06:59:00.410 INFO:teuthology.orchestra.run.smithi042.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 143991 files and directories currently installed.) 2019-12-05T06:59:00.414 INFO:teuthology.orchestra.run.smithi042.stdout:Removing ceph-mgr-rook (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T06:59:00.460 INFO:teuthology.orchestra.run.smithi040.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 142518 files and directories currently installed.) 2019-12-05T06:59:00.464 INFO:teuthology.orchestra.run.smithi040.stdout:Purging configuration files for ceph-mgr (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T06:59:00.657 INFO:teuthology.orchestra.run.smithi042.stdout:dpkg: warning: while removing ceph-mgr-rook, directory '/usr/share/ceph/mgr/rook' not empty so not removed 2019-12-05T06:59:00.842 INFO:teuthology.orchestra.run.smithi042.stdout:Removing ceph-mgr-k8sevents (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T06:59:01.084 INFO:teuthology.orchestra.run.smithi040.stdout:dpkg: warning: while removing ceph-mgr, directory '/var/lib/ceph/mgr' not empty so not removed 2019-12-05T06:59:01.084 INFO:teuthology.orchestra.run.smithi040.stdout:dpkg: warning: while removing ceph-mgr, directory '/usr/share/ceph/mgr/zabbix' not empty so not removed 2019-12-05T06:59:01.084 INFO:teuthology.orchestra.run.smithi040.stdout:dpkg: warning: while removing ceph-mgr, directory '/usr/share/ceph/mgr/volumes/fs' not empty so not removed 2019-12-05T06:59:01.084 INFO:teuthology.orchestra.run.smithi040.stdout:dpkg: warning: while removing ceph-mgr, directory '/usr/share/ceph/mgr/test_orchestrator' not empty so not removed 2019-12-05T06:59:01.084 INFO:teuthology.orchestra.run.smithi040.stdout:dpkg: warning: while removing ceph-mgr, directory '/usr/share/ceph/mgr/telemetry' not empty so not removed 2019-12-05T06:59:01.085 INFO:teuthology.orchestra.run.smithi040.stdout:dpkg: warning: while removing ceph-mgr, directory '/usr/share/ceph/mgr/telegraf' not empty so not removed 2019-12-05T06:59:01.085 INFO:teuthology.orchestra.run.smithi040.stdout:dpkg: warning: while removing ceph-mgr, directory '/usr/share/ceph/mgr/status' not empty so not removed 2019-12-05T06:59:01.085 INFO:teuthology.orchestra.run.smithi040.stdout:dpkg: warning: while removing ceph-mgr, directory '/usr/share/ceph/mgr/selftest' not empty so not removed 2019-12-05T06:59:01.085 INFO:teuthology.orchestra.run.smithi040.stdout:dpkg: warning: while removing ceph-mgr, directory '/usr/share/ceph/mgr/restful' not empty so not removed 2019-12-05T06:59:01.085 INFO:teuthology.orchestra.run.smithi040.stdout:dpkg: warning: while removing ceph-mgr, directory '/usr/share/ceph/mgr/rbd_support' not empty so not removed 2019-12-05T06:59:01.085 INFO:teuthology.orchestra.run.smithi040.stdout:dpkg: warning: while removing ceph-mgr, directory '/usr/share/ceph/mgr/prometheus' not empty so not removed 2019-12-05T06:59:01.085 INFO:teuthology.orchestra.run.smithi040.stdout:dpkg: warning: while removing ceph-mgr, directory '/usr/share/ceph/mgr/progress' not empty so not removed 2019-12-05T06:59:01.086 INFO:teuthology.orchestra.run.smithi040.stdout:dpkg: warning: while removing ceph-mgr, directory '/usr/share/ceph/mgr/pg_autoscaler' not empty so not removed 2019-12-05T06:59:01.086 INFO:teuthology.orchestra.run.smithi040.stdout:dpkg: warning: while removing ceph-mgr, directory '/usr/share/ceph/mgr/orchestrator_cli' not empty so not removed 2019-12-05T06:59:01.086 INFO:teuthology.orchestra.run.smithi040.stdout:dpkg: warning: while removing ceph-mgr, directory '/usr/share/ceph/mgr/localpool' not empty so not removed 2019-12-05T06:59:01.086 INFO:teuthology.orchestra.run.smithi040.stdout:dpkg: warning: while removing ceph-mgr, directory '/usr/share/ceph/mgr/iostat' not empty so not removed 2019-12-05T06:59:01.086 INFO:teuthology.orchestra.run.smithi040.stdout:dpkg: warning: while removing ceph-mgr, directory '/usr/share/ceph/mgr/insights' not empty so not removed 2019-12-05T06:59:01.086 INFO:teuthology.orchestra.run.smithi040.stdout:dpkg: warning: while removing ceph-mgr, directory '/usr/share/ceph/mgr/influx' not empty so not removed 2019-12-05T06:59:01.086 INFO:teuthology.orchestra.run.smithi040.stdout:dpkg: warning: while removing ceph-mgr, directory '/usr/share/ceph/mgr/devicehealth' not empty so not removed 2019-12-05T06:59:01.087 INFO:teuthology.orchestra.run.smithi040.stdout:dpkg: warning: while removing ceph-mgr, directory '/usr/share/ceph/mgr/deepsea' not empty so not removed 2019-12-05T06:59:01.087 INFO:teuthology.orchestra.run.smithi040.stdout:dpkg: warning: while removing ceph-mgr, directory '/usr/share/ceph/mgr/crash' not empty so not removed 2019-12-05T06:59:01.087 INFO:teuthology.orchestra.run.smithi040.stdout:dpkg: warning: while removing ceph-mgr, directory '/usr/share/ceph/mgr/balancer' not empty so not removed 2019-12-05T06:59:01.087 INFO:teuthology.orchestra.run.smithi040.stdout:dpkg: warning: while removing ceph-mgr, directory '/usr/share/ceph/mgr/ansible' not empty so not removed 2019-12-05T06:59:01.088 INFO:teuthology.orchestra.run.smithi040.stdout:dpkg: warning: while removing ceph-mgr, directory '/usr/share/ceph/mgr/alerts' not empty so not removed 2019-12-05T06:59:01.306 INFO:teuthology.orchestra.run.smithi042.stdout:Removing ceph-mgr-dashboard (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T06:59:01.561 INFO:teuthology.orchestra.run.smithi042.stdout:dpkg: warning: while removing ceph-mgr-dashboard, directory '/usr/share/ceph/mgr/dashboard/services' not empty so not removed 2019-12-05T06:59:01.561 INFO:teuthology.orchestra.run.smithi042.stdout:dpkg: warning: while removing ceph-mgr-dashboard, directory '/usr/share/ceph/mgr/dashboard/plugins' not empty so not removed 2019-12-05T06:59:01.561 INFO:teuthology.orchestra.run.smithi042.stdout:dpkg: warning: while removing ceph-mgr-dashboard, directory '/usr/share/ceph/mgr/dashboard/controllers' not empty so not removed 2019-12-05T06:59:01.602 INFO:teuthology.orchestra.run.smithi040.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2019-12-05T06:59:01.685 INFO:teuthology.orchestra.run.smithi040.stdout:Reading package lists... 2019-12-05T06:59:01.735 INFO:teuthology.orchestra.run.smithi042.stdout:Removing ceph-mgr-diskprediction-cloud (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T06:59:01.943 INFO:teuthology.orchestra.run.smithi040.stdout:Building dependency tree... 2019-12-05T06:59:01.944 INFO:teuthology.orchestra.run.smithi040.stdout:Reading state information... 2019-12-05T06:59:01.965 INFO:teuthology.orchestra.run.smithi042.stdout:dpkg: warning: while removing ceph-mgr-diskprediction-cloud, directory '/usr/share/ceph/mgr/diskprediction_cloud/common' not empty so not removed 2019-12-05T06:59:01.965 INFO:teuthology.orchestra.run.smithi042.stdout:dpkg: warning: while removing ceph-mgr-diskprediction-cloud, directory '/usr/share/ceph/mgr/diskprediction_cloud/agent/metrics' not empty so not removed 2019-12-05T06:59:02.113 INFO:teuthology.orchestra.run.smithi042.stdout:Removing ceph-mgr-diskprediction-local (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T06:59:02.130 INFO:teuthology.orchestra.run.smithi040.stdout:The following packages were automatically installed and are no longer required: 2019-12-05T06:59:02.131 INFO:teuthology.orchestra.run.smithi040.stdout: formencode-i18n jq libjq1 libjs-sphinxdoc libjs-underscore libleveldb1v5 2019-12-05T06:59:02.131 INFO:teuthology.orchestra.run.smithi040.stdout: liboath0 libonig4 libradosstriper1 libsnappy1v5 libxslt1.1 python-attr 2019-12-05T06:59:02.131 INFO:teuthology.orchestra.run.smithi040.stdout: python-bcrypt python-bs4 python-ceph-common python-certifi python-chardet 2019-12-05T06:59:02.131 INFO:teuthology.orchestra.run.smithi040.stdout: python-cherrypy3 python-dnspython python-fasteners python-formencode 2019-12-05T06:59:02.132 INFO:teuthology.orchestra.run.smithi040.stdout: python-funcsigs python-html5lib python-httplib2 python-joblib python-jwt 2019-12-05T06:59:02.132 INFO:teuthology.orchestra.run.smithi040.stdout: python-kubernetes python-logutils python-lxml python-mako python-markupsafe 2019-12-05T06:59:02.132 INFO:teuthology.orchestra.run.smithi040.stdout: python-monotonic python-oauth2client python-openid python-paste 2019-12-05T06:59:02.132 INFO:teuthology.orchestra.run.smithi040.stdout: python-pastedeploy python-pastedeploy-tpl python-pastescript python-pecan 2019-12-05T06:59:02.132 INFO:teuthology.orchestra.run.smithi040.stdout: python-pluggy python-prettytable python-py python-pyasn1 2019-12-05T06:59:02.132 INFO:teuthology.orchestra.run.smithi040.stdout: python-pyasn1-modules python-pyinotify python-pytest python-repoze.lru 2019-12-05T06:59:02.132 INFO:teuthology.orchestra.run.smithi040.stdout: python-requests python-routes python-rsa python-scgi python-simplegeneric 2019-12-05T06:59:02.133 INFO:teuthology.orchestra.run.smithi040.stdout: python-simplejson python-singledispatch python-sklearn python-sklearn-lib 2019-12-05T06:59:02.133 INFO:teuthology.orchestra.run.smithi040.stdout: python-sqlalchemy python-sqlalchemy-ext python-tempita python-uritemplate 2019-12-05T06:59:02.133 INFO:teuthology.orchestra.run.smithi040.stdout: python-urllib3 python-waitress python-webencodings python-webob 2019-12-05T06:59:02.133 INFO:teuthology.orchestra.run.smithi040.stdout: python-websocket python-webtest python-werkzeug python-yaml socat xmlstarlet 2019-12-05T06:59:02.133 INFO:teuthology.orchestra.run.smithi040.stdout:Use 'sudo apt autoremove' to remove them. 2019-12-05T06:59:02.212 INFO:teuthology.orchestra.run.smithi040.stdout:The following packages will be REMOVED: 2019-12-05T06:59:02.213 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-base* ceph-common* ceph-mon* ceph-osd* ceph-test* radosgw* 2019-12-05T06:59:02.342 INFO:teuthology.orchestra.run.smithi042.stdout:dpkg: warning: while removing ceph-mgr-diskprediction-local, directory '/usr/share/ceph/mgr/diskprediction_local' not empty so not removed 2019-12-05T06:59:02.533 INFO:teuthology.orchestra.run.smithi042.stdout:Removing ceph-mgr (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T06:59:02.567 INFO:teuthology.orchestra.run.smithi040.stdout:0 upgraded, 0 newly installed, 6 to remove and 51 not upgraded. 2019-12-05T06:59:02.567 INFO:teuthology.orchestra.run.smithi040.stdout:After this operation, 435 MB disk space will be freed. 2019-12-05T06:59:02.677 INFO:teuthology.orchestra.run.smithi040.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 142492 files and directories currently installed.) 2019-12-05T06:59:02.681 INFO:teuthology.orchestra.run.smithi040.stdout:Removing ceph-osd (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T06:59:03.496 INFO:teuthology.orchestra.run.smithi042.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 142518 files and directories currently installed.) 2019-12-05T06:59:03.500 INFO:teuthology.orchestra.run.smithi042.stdout:Purging configuration files for ceph-mgr (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T06:59:03.529 INFO:teuthology.orchestra.run.smithi040.stdout:Removing ceph-mon (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T06:59:04.103 INFO:teuthology.orchestra.run.smithi042.stdout:dpkg: warning: while removing ceph-mgr, directory '/var/lib/ceph/mgr' not empty so not removed 2019-12-05T06:59:04.103 INFO:teuthology.orchestra.run.smithi042.stdout:dpkg: warning: while removing ceph-mgr, directory '/usr/share/ceph/mgr/zabbix' not empty so not removed 2019-12-05T06:59:04.104 INFO:teuthology.orchestra.run.smithi042.stdout:dpkg: warning: while removing ceph-mgr, directory '/usr/share/ceph/mgr/volumes/fs' not empty so not removed 2019-12-05T06:59:04.104 INFO:teuthology.orchestra.run.smithi042.stdout:dpkg: warning: while removing ceph-mgr, directory '/usr/share/ceph/mgr/test_orchestrator' not empty so not removed 2019-12-05T06:59:04.104 INFO:teuthology.orchestra.run.smithi042.stdout:dpkg: warning: while removing ceph-mgr, directory '/usr/share/ceph/mgr/telemetry' not empty so not removed 2019-12-05T06:59:04.104 INFO:teuthology.orchestra.run.smithi042.stdout:dpkg: warning: while removing ceph-mgr, directory '/usr/share/ceph/mgr/telegraf' not empty so not removed 2019-12-05T06:59:04.104 INFO:teuthology.orchestra.run.smithi042.stdout:dpkg: warning: while removing ceph-mgr, directory '/usr/share/ceph/mgr/status' not empty so not removed 2019-12-05T06:59:04.104 INFO:teuthology.orchestra.run.smithi042.stdout:dpkg: warning: while removing ceph-mgr, directory '/usr/share/ceph/mgr/selftest' not empty so not removed 2019-12-05T06:59:04.105 INFO:teuthology.orchestra.run.smithi042.stdout:dpkg: warning: while removing ceph-mgr, directory '/usr/share/ceph/mgr/restful' not empty so not removed 2019-12-05T06:59:04.105 INFO:teuthology.orchestra.run.smithi042.stdout:dpkg: warning: while removing ceph-mgr, directory '/usr/share/ceph/mgr/rbd_support' not empty so not removed 2019-12-05T06:59:04.105 INFO:teuthology.orchestra.run.smithi042.stdout:dpkg: warning: while removing ceph-mgr, directory '/usr/share/ceph/mgr/prometheus' not empty so not removed 2019-12-05T06:59:04.105 INFO:teuthology.orchestra.run.smithi042.stdout:dpkg: warning: while removing ceph-mgr, directory '/usr/share/ceph/mgr/progress' not empty so not removed 2019-12-05T06:59:04.105 INFO:teuthology.orchestra.run.smithi042.stdout:dpkg: warning: while removing ceph-mgr, directory '/usr/share/ceph/mgr/pg_autoscaler' not empty so not removed 2019-12-05T06:59:04.106 INFO:teuthology.orchestra.run.smithi042.stdout:dpkg: warning: while removing ceph-mgr, directory '/usr/share/ceph/mgr/orchestrator_cli' not empty so not removed 2019-12-05T06:59:04.106 INFO:teuthology.orchestra.run.smithi042.stdout:dpkg: warning: while removing ceph-mgr, directory '/usr/share/ceph/mgr/localpool' not empty so not removed 2019-12-05T06:59:04.106 INFO:teuthology.orchestra.run.smithi042.stdout:dpkg: warning: while removing ceph-mgr, directory '/usr/share/ceph/mgr/iostat' not empty so not removed 2019-12-05T06:59:04.106 INFO:teuthology.orchestra.run.smithi042.stdout:dpkg: warning: while removing ceph-mgr, directory '/usr/share/ceph/mgr/insights' not empty so not removed 2019-12-05T06:59:04.106 INFO:teuthology.orchestra.run.smithi042.stdout:dpkg: warning: while removing ceph-mgr, directory '/usr/share/ceph/mgr/influx' not empty so not removed 2019-12-05T06:59:04.106 INFO:teuthology.orchestra.run.smithi042.stdout:dpkg: warning: while removing ceph-mgr, directory '/usr/share/ceph/mgr/devicehealth' not empty so not removed 2019-12-05T06:59:04.107 INFO:teuthology.orchestra.run.smithi042.stdout:dpkg: warning: while removing ceph-mgr, directory '/usr/share/ceph/mgr/deepsea' not empty so not removed 2019-12-05T06:59:04.107 INFO:teuthology.orchestra.run.smithi042.stdout:dpkg: warning: while removing ceph-mgr, directory '/usr/share/ceph/mgr/crash' not empty so not removed 2019-12-05T06:59:04.107 INFO:teuthology.orchestra.run.smithi042.stdout:dpkg: warning: while removing ceph-mgr, directory '/usr/share/ceph/mgr/balancer' not empty so not removed 2019-12-05T06:59:04.107 INFO:teuthology.orchestra.run.smithi042.stdout:dpkg: warning: while removing ceph-mgr, directory '/usr/share/ceph/mgr/ansible' not empty so not removed 2019-12-05T06:59:04.107 INFO:teuthology.orchestra.run.smithi042.stdout:dpkg: warning: while removing ceph-mgr, directory '/usr/share/ceph/mgr/alerts' not empty so not removed 2019-12-05T06:59:04.218 INFO:teuthology.orchestra.run.smithi040.stdout:Removing ceph-base (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T06:59:04.685 INFO:teuthology.orchestra.run.smithi042.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2019-12-05T06:59:04.765 INFO:teuthology.orchestra.run.smithi042.stdout:Reading package lists... 2019-12-05T06:59:04.920 INFO:teuthology.orchestra.run.smithi040.stdout:Removing radosgw (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T06:59:05.029 INFO:teuthology.orchestra.run.smithi042.stdout:Building dependency tree... 2019-12-05T06:59:05.029 INFO:teuthology.orchestra.run.smithi042.stdout:Reading state information... 2019-12-05T06:59:05.225 INFO:teuthology.orchestra.run.smithi042.stdout:The following packages were automatically installed and are no longer required: 2019-12-05T06:59:05.225 INFO:teuthology.orchestra.run.smithi042.stdout: formencode-i18n jq libjq1 libjs-sphinxdoc libjs-underscore libleveldb1v5 2019-12-05T06:59:05.226 INFO:teuthology.orchestra.run.smithi042.stdout: liboath0 libonig4 libradosstriper1 libsnappy1v5 libxslt1.1 python-attr 2019-12-05T06:59:05.226 INFO:teuthology.orchestra.run.smithi042.stdout: python-bcrypt python-bs4 python-ceph-common python-certifi python-chardet 2019-12-05T06:59:05.226 INFO:teuthology.orchestra.run.smithi042.stdout: python-cherrypy3 python-dnspython python-fasteners python-formencode 2019-12-05T06:59:05.226 INFO:teuthology.orchestra.run.smithi042.stdout: python-funcsigs python-html5lib python-httplib2 python-joblib python-jwt 2019-12-05T06:59:05.227 INFO:teuthology.orchestra.run.smithi042.stdout: python-kubernetes python-logutils python-lxml python-mako python-markupsafe 2019-12-05T06:59:05.227 INFO:teuthology.orchestra.run.smithi042.stdout: python-monotonic python-oauth2client python-openid python-paste 2019-12-05T06:59:05.227 INFO:teuthology.orchestra.run.smithi042.stdout: python-pastedeploy python-pastedeploy-tpl python-pastescript python-pecan 2019-12-05T06:59:05.227 INFO:teuthology.orchestra.run.smithi042.stdout: python-pluggy python-prettytable python-py python-pyasn1 2019-12-05T06:59:05.227 INFO:teuthology.orchestra.run.smithi042.stdout: python-pyasn1-modules python-pyinotify python-pytest python-repoze.lru 2019-12-05T06:59:05.227 INFO:teuthology.orchestra.run.smithi042.stdout: python-requests python-routes python-rsa python-scgi python-simplegeneric 2019-12-05T06:59:05.227 INFO:teuthology.orchestra.run.smithi042.stdout: python-simplejson python-singledispatch python-sklearn python-sklearn-lib 2019-12-05T06:59:05.228 INFO:teuthology.orchestra.run.smithi042.stdout: python-sqlalchemy python-sqlalchemy-ext python-tempita python-uritemplate 2019-12-05T06:59:05.228 INFO:teuthology.orchestra.run.smithi042.stdout: python-urllib3 python-waitress python-webencodings python-webob 2019-12-05T06:59:05.228 INFO:teuthology.orchestra.run.smithi042.stdout: python-websocket python-webtest python-werkzeug python-yaml socat xmlstarlet 2019-12-05T06:59:05.228 INFO:teuthology.orchestra.run.smithi042.stdout:Use 'sudo apt autoremove' to remove them. 2019-12-05T06:59:05.318 INFO:teuthology.orchestra.run.smithi042.stdout:The following packages will be REMOVED: 2019-12-05T06:59:05.319 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-base* ceph-common* ceph-mon* ceph-osd* ceph-test* radosgw* 2019-12-05T06:59:05.684 INFO:teuthology.orchestra.run.smithi042.stdout:0 upgraded, 0 newly installed, 6 to remove and 51 not upgraded. 2019-12-05T06:59:05.685 INFO:teuthology.orchestra.run.smithi042.stdout:After this operation, 435 MB disk space will be freed. 2019-12-05T06:59:05.756 INFO:teuthology.orchestra.run.smithi040.stdout:Removing ceph-test (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T06:59:05.789 INFO:teuthology.orchestra.run.smithi042.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 142492 files and directories currently installed.) 2019-12-05T06:59:05.794 INFO:teuthology.orchestra.run.smithi042.stdout:Removing ceph-osd (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T06:59:06.155 INFO:teuthology.orchestra.run.smithi040.stdout:Removing ceph-common (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T06:59:06.693 INFO:teuthology.orchestra.run.smithi042.stdout:Removing ceph-mon (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T06:59:06.808 INFO:teuthology.orchestra.run.smithi040.stdout:Processing triggers for man-db (2.8.3-2ubuntu0.1) ... 2019-12-05T06:59:07.312 INFO:teuthology.orchestra.run.smithi040.stdout:Processing triggers for libc-bin (2.27-3ubuntu1) ... 2019-12-05T06:59:07.401 INFO:teuthology.orchestra.run.smithi042.stdout:Removing ceph-base (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T06:59:07.667 INFO:teuthology.orchestra.run.smithi040.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 142107 files and directories currently installed.) 2019-12-05T06:59:07.671 INFO:teuthology.orchestra.run.smithi040.stdout:Purging configuration files for ceph-osd (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T06:59:08.090 INFO:teuthology.orchestra.run.smithi042.stdout:Removing radosgw (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T06:59:08.399 INFO:teuthology.orchestra.run.smithi040.stdout:dpkg: warning: while removing ceph-osd, directory '/var/lib/ceph/osd' not empty so not removed 2019-12-05T06:59:08.540 INFO:teuthology.orchestra.run.smithi040.stdout:Purging configuration files for ceph-mon (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T06:59:08.872 INFO:teuthology.orchestra.run.smithi042.stdout:Removing ceph-test (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T06:59:09.119 INFO:teuthology.orchestra.run.smithi040.stdout:dpkg: warning: while removing ceph-mon, directory '/var/lib/ceph/mon' not empty so not removed 2019-12-05T06:59:09.216 INFO:teuthology.orchestra.run.smithi042.stdout:Removing ceph-common (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T06:59:09.252 INFO:teuthology.orchestra.run.smithi040.stdout:Purging configuration files for ceph-base (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T06:59:09.886 INFO:teuthology.orchestra.run.smithi042.stdout:Processing triggers for man-db (2.8.3-2ubuntu0.1) ... 2019-12-05T06:59:10.098 INFO:teuthology.orchestra.run.smithi040.stdout:Purging configuration files for ceph-common (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T06:59:10.397 INFO:teuthology.orchestra.run.smithi042.stdout:Processing triggers for libc-bin (2.27-3ubuntu1) ... 2019-12-05T06:59:10.821 INFO:teuthology.orchestra.run.smithi042.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 142107 files and directories currently installed.) 2019-12-05T06:59:10.825 INFO:teuthology.orchestra.run.smithi042.stdout:Purging configuration files for ceph-osd (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T06:59:10.954 INFO:teuthology.orchestra.run.smithi040.stdout:dpkg: warning: while removing ceph-common, directory '/usr/share/ceph' not empty so not removed 2019-12-05T06:59:11.095 INFO:teuthology.orchestra.run.smithi040.stdout:Purging configuration files for radosgw (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T06:59:11.443 INFO:teuthology.orchestra.run.smithi042.stdout:dpkg: warning: while removing ceph-osd, directory '/var/lib/ceph/osd' not empty so not removed 2019-12-05T06:59:11.585 INFO:teuthology.orchestra.run.smithi042.stdout:Purging configuration files for ceph-mon (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T06:59:12.192 INFO:teuthology.orchestra.run.smithi040.stdout:Processing triggers for systemd (237-3ubuntu10.31) ... 2019-12-05T06:59:12.205 INFO:teuthology.orchestra.run.smithi042.stdout:dpkg: warning: while removing ceph-mon, directory '/var/lib/ceph/mon' not empty so not removed 2019-12-05T06:59:12.318 INFO:teuthology.orchestra.run.smithi042.stdout:Purging configuration files for ceph-base (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T06:59:12.411 INFO:teuthology.orchestra.run.smithi040.stdout:Processing triggers for ureadahead (0.100.0-21) ... 2019-12-05T06:59:12.795 INFO:teuthology.orchestra.run.smithi040.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2019-12-05T06:59:12.879 INFO:teuthology.orchestra.run.smithi040.stdout:Reading package lists... 2019-12-05T06:59:13.140 INFO:teuthology.orchestra.run.smithi042.stdout:Purging configuration files for ceph-common (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T06:59:13.142 INFO:teuthology.orchestra.run.smithi040.stdout:Building dependency tree... 2019-12-05T06:59:13.143 INFO:teuthology.orchestra.run.smithi040.stdout:Reading state information... 2019-12-05T06:59:13.334 INFO:teuthology.orchestra.run.smithi040.stdout:The following packages were automatically installed and are no longer required: 2019-12-05T06:59:13.335 INFO:teuthology.orchestra.run.smithi040.stdout: formencode-i18n jq libjq1 libjs-sphinxdoc libjs-underscore libleveldb1v5 2019-12-05T06:59:13.336 INFO:teuthology.orchestra.run.smithi040.stdout: liboath0 libonig4 libradosstriper1 libsnappy1v5 libxslt1.1 python-attr 2019-12-05T06:59:13.336 INFO:teuthology.orchestra.run.smithi040.stdout: python-bcrypt python-bs4 python-ceph-common python-certifi python-chardet 2019-12-05T06:59:13.336 INFO:teuthology.orchestra.run.smithi040.stdout: python-cherrypy3 python-dnspython python-fasteners python-formencode 2019-12-05T06:59:13.337 INFO:teuthology.orchestra.run.smithi040.stdout: python-funcsigs python-html5lib python-httplib2 python-joblib python-jwt 2019-12-05T06:59:13.337 INFO:teuthology.orchestra.run.smithi040.stdout: python-kubernetes python-logutils python-lxml python-mako python-markupsafe 2019-12-05T06:59:13.337 INFO:teuthology.orchestra.run.smithi040.stdout: python-monotonic python-oauth2client python-openid python-paste 2019-12-05T06:59:13.337 INFO:teuthology.orchestra.run.smithi040.stdout: python-pastedeploy python-pastedeploy-tpl python-pastescript python-pecan 2019-12-05T06:59:13.337 INFO:teuthology.orchestra.run.smithi040.stdout: python-pluggy python-prettytable python-py python-pyasn1 2019-12-05T06:59:13.337 INFO:teuthology.orchestra.run.smithi040.stdout: python-pyasn1-modules python-pyinotify python-pytest python-repoze.lru 2019-12-05T06:59:13.338 INFO:teuthology.orchestra.run.smithi040.stdout: python-requests python-routes python-rsa python-scgi python-simplegeneric 2019-12-05T06:59:13.338 INFO:teuthology.orchestra.run.smithi040.stdout: python-simplejson python-singledispatch python-sklearn python-sklearn-lib 2019-12-05T06:59:13.338 INFO:teuthology.orchestra.run.smithi040.stdout: python-sqlalchemy python-sqlalchemy-ext python-tempita python-uritemplate 2019-12-05T06:59:13.338 INFO:teuthology.orchestra.run.smithi040.stdout: python-urllib3 python-waitress python-webencodings python-webob 2019-12-05T06:59:13.338 INFO:teuthology.orchestra.run.smithi040.stdout: python-websocket python-webtest python-werkzeug python-yaml socat xmlstarlet 2019-12-05T06:59:13.338 INFO:teuthology.orchestra.run.smithi040.stdout:Use 'sudo apt autoremove' to remove them. 2019-12-05T06:59:13.356 INFO:teuthology.orchestra.run.smithi040.stdout:The following packages will be REMOVED: 2019-12-05T06:59:13.358 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-fuse* 2019-12-05T06:59:13.712 INFO:teuthology.orchestra.run.smithi040.stdout:0 upgraded, 0 newly installed, 1 to remove and 51 not upgraded. 2019-12-05T06:59:13.712 INFO:teuthology.orchestra.run.smithi040.stdout:After this operation, 1,591 kB disk space will be freed. 2019-12-05T06:59:13.822 INFO:teuthology.orchestra.run.smithi040.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 142090 files and directories currently installed.) 2019-12-05T06:59:13.826 INFO:teuthology.orchestra.run.smithi040.stdout:Removing ceph-fuse (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T06:59:14.065 INFO:teuthology.orchestra.run.smithi042.stdout:dpkg: warning: while removing ceph-common, directory '/usr/share/ceph' not empty so not removed 2019-12-05T06:59:14.189 INFO:teuthology.orchestra.run.smithi042.stdout:Purging configuration files for radosgw (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T06:59:14.442 INFO:teuthology.orchestra.run.smithi040.stdout:Processing triggers for man-db (2.8.3-2ubuntu0.1) ... 2019-12-05T06:59:15.210 INFO:teuthology.orchestra.run.smithi040.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 142083 files and directories currently installed.) 2019-12-05T06:59:15.217 INFO:teuthology.orchestra.run.smithi040.stdout:Purging configuration files for ceph-fuse (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T06:59:15.242 INFO:teuthology.orchestra.run.smithi042.stdout:Processing triggers for systemd (237-3ubuntu10.31) ... 2019-12-05T06:59:15.439 INFO:teuthology.orchestra.run.smithi042.stdout:Processing triggers for ureadahead (0.100.0-21) ... 2019-12-05T06:59:15.865 INFO:teuthology.orchestra.run.smithi042.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2019-12-05T06:59:15.945 INFO:teuthology.orchestra.run.smithi042.stdout:Reading package lists... 2019-12-05T06:59:16.204 INFO:teuthology.orchestra.run.smithi042.stdout:Building dependency tree... 2019-12-05T06:59:16.204 INFO:teuthology.orchestra.run.smithi042.stdout:Reading state information... 2019-12-05T06:59:16.310 INFO:teuthology.orchestra.run.smithi040.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2019-12-05T06:59:16.388 INFO:teuthology.orchestra.run.smithi042.stdout:The following packages were automatically installed and are no longer required: 2019-12-05T06:59:16.388 INFO:teuthology.orchestra.run.smithi042.stdout: formencode-i18n jq libjq1 libjs-sphinxdoc libjs-underscore libleveldb1v5 2019-12-05T06:59:16.389 INFO:teuthology.orchestra.run.smithi042.stdout: liboath0 libonig4 libradosstriper1 libsnappy1v5 libxslt1.1 python-attr 2019-12-05T06:59:16.389 INFO:teuthology.orchestra.run.smithi042.stdout: python-bcrypt python-bs4 python-ceph-common python-certifi python-chardet 2019-12-05T06:59:16.389 INFO:teuthology.orchestra.run.smithi042.stdout: python-cherrypy3 python-dnspython python-fasteners python-formencode 2019-12-05T06:59:16.389 INFO:teuthology.orchestra.run.smithi042.stdout: python-funcsigs python-html5lib python-httplib2 python-joblib python-jwt 2019-12-05T06:59:16.389 INFO:teuthology.orchestra.run.smithi042.stdout: python-kubernetes python-logutils python-lxml python-mako python-markupsafe 2019-12-05T06:59:16.390 INFO:teuthology.orchestra.run.smithi042.stdout: python-monotonic python-oauth2client python-openid python-paste 2019-12-05T06:59:16.390 INFO:teuthology.orchestra.run.smithi042.stdout: python-pastedeploy python-pastedeploy-tpl python-pastescript python-pecan 2019-12-05T06:59:16.390 INFO:teuthology.orchestra.run.smithi042.stdout: python-pluggy python-prettytable python-py python-pyasn1 2019-12-05T06:59:16.390 INFO:teuthology.orchestra.run.smithi042.stdout: python-pyasn1-modules python-pyinotify python-pytest python-repoze.lru 2019-12-05T06:59:16.390 INFO:teuthology.orchestra.run.smithi042.stdout: python-requests python-routes python-rsa python-scgi python-simplegeneric 2019-12-05T06:59:16.390 INFO:teuthology.orchestra.run.smithi042.stdout: python-simplejson python-singledispatch python-sklearn python-sklearn-lib 2019-12-05T06:59:16.390 INFO:teuthology.orchestra.run.smithi042.stdout: python-sqlalchemy python-sqlalchemy-ext python-tempita python-uritemplate 2019-12-05T06:59:16.391 INFO:teuthology.orchestra.run.smithi042.stdout: python-urllib3 python-waitress python-webencodings python-webob 2019-12-05T06:59:16.391 INFO:teuthology.orchestra.run.smithi042.stdout: python-websocket python-webtest python-werkzeug python-yaml socat xmlstarlet 2019-12-05T06:59:16.391 INFO:teuthology.orchestra.run.smithi042.stdout:Use 'sudo apt autoremove' to remove them. 2019-12-05T06:59:16.392 INFO:teuthology.orchestra.run.smithi040.stdout:Reading package lists... 2019-12-05T06:59:16.407 INFO:teuthology.orchestra.run.smithi042.stdout:The following packages will be REMOVED: 2019-12-05T06:59:16.409 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-fuse* 2019-12-05T06:59:16.645 INFO:teuthology.orchestra.run.smithi040.stdout:Building dependency tree... 2019-12-05T06:59:16.646 INFO:teuthology.orchestra.run.smithi040.stdout:Reading state information... 2019-12-05T06:59:16.793 INFO:teuthology.orchestra.run.smithi042.stdout:0 upgraded, 0 newly installed, 1 to remove and 51 not upgraded. 2019-12-05T06:59:16.794 INFO:teuthology.orchestra.run.smithi042.stdout:After this operation, 1,591 kB disk space will be freed. 2019-12-05T06:59:16.818 INFO:teuthology.orchestra.run.smithi040.stdout:Package 'ceph-test' is not installed, so not removed 2019-12-05T06:59:16.818 INFO:teuthology.orchestra.run.smithi040.stdout:The following packages were automatically installed and are no longer required: 2019-12-05T06:59:16.820 INFO:teuthology.orchestra.run.smithi040.stdout: formencode-i18n jq libjq1 libjs-sphinxdoc libjs-underscore libleveldb1v5 2019-12-05T06:59:16.820 INFO:teuthology.orchestra.run.smithi040.stdout: liboath0 libonig4 libradosstriper1 libsnappy1v5 libxslt1.1 python-attr 2019-12-05T06:59:16.820 INFO:teuthology.orchestra.run.smithi040.stdout: python-bcrypt python-bs4 python-ceph-common python-certifi python-chardet 2019-12-05T06:59:16.820 INFO:teuthology.orchestra.run.smithi040.stdout: python-cherrypy3 python-dnspython python-fasteners python-formencode 2019-12-05T06:59:16.821 INFO:teuthology.orchestra.run.smithi040.stdout: python-funcsigs python-html5lib python-httplib2 python-joblib python-jwt 2019-12-05T06:59:16.821 INFO:teuthology.orchestra.run.smithi040.stdout: python-kubernetes python-logutils python-lxml python-mako python-markupsafe 2019-12-05T06:59:16.821 INFO:teuthology.orchestra.run.smithi040.stdout: python-monotonic python-oauth2client python-openid python-paste 2019-12-05T06:59:16.821 INFO:teuthology.orchestra.run.smithi040.stdout: python-pastedeploy python-pastedeploy-tpl python-pastescript python-pecan 2019-12-05T06:59:16.821 INFO:teuthology.orchestra.run.smithi040.stdout: python-pluggy python-prettytable python-py python-pyasn1 2019-12-05T06:59:16.821 INFO:teuthology.orchestra.run.smithi040.stdout: python-pyasn1-modules python-pyinotify python-pytest python-repoze.lru 2019-12-05T06:59:16.821 INFO:teuthology.orchestra.run.smithi040.stdout: python-requests python-routes python-rsa python-scgi python-simplegeneric 2019-12-05T06:59:16.822 INFO:teuthology.orchestra.run.smithi040.stdout: python-simplejson python-singledispatch python-sklearn python-sklearn-lib 2019-12-05T06:59:16.822 INFO:teuthology.orchestra.run.smithi040.stdout: python-sqlalchemy python-sqlalchemy-ext python-tempita python-uritemplate 2019-12-05T06:59:16.822 INFO:teuthology.orchestra.run.smithi040.stdout: python-urllib3 python-waitress python-webencodings python-webob 2019-12-05T06:59:16.822 INFO:teuthology.orchestra.run.smithi040.stdout: python-websocket python-webtest python-werkzeug python-yaml socat xmlstarlet 2019-12-05T06:59:16.822 INFO:teuthology.orchestra.run.smithi040.stdout:Use 'sudo apt autoremove' to remove them. 2019-12-05T06:59:16.855 INFO:teuthology.orchestra.run.smithi040.stdout:0 upgraded, 0 newly installed, 0 to remove and 51 not upgraded. 2019-12-05T06:59:16.855 INFO:teuthology.orchestra.run.smithi040.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2019-12-05T06:59:16.912 INFO:teuthology.orchestra.run.smithi042.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 142090 files and directories currently installed.) 2019-12-05T06:59:16.916 INFO:teuthology.orchestra.run.smithi042.stdout:Removing ceph-fuse (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T06:59:16.933 INFO:teuthology.orchestra.run.smithi040.stdout:Reading package lists... 2019-12-05T06:59:17.162 INFO:teuthology.orchestra.run.smithi040.stdout:Building dependency tree... 2019-12-05T06:59:17.163 INFO:teuthology.orchestra.run.smithi040.stdout:Reading state information... 2019-12-05T06:59:17.332 INFO:teuthology.orchestra.run.smithi040.stdout:Package 'radosgw' is not installed, so not removed 2019-12-05T06:59:17.332 INFO:teuthology.orchestra.run.smithi040.stdout:The following packages were automatically installed and are no longer required: 2019-12-05T06:59:17.332 INFO:teuthology.orchestra.run.smithi040.stdout: formencode-i18n jq libjq1 libjs-sphinxdoc libjs-underscore libleveldb1v5 2019-12-05T06:59:17.334 INFO:teuthology.orchestra.run.smithi040.stdout: liboath0 libonig4 libradosstriper1 libsnappy1v5 libxslt1.1 python-attr 2019-12-05T06:59:17.334 INFO:teuthology.orchestra.run.smithi040.stdout: python-bcrypt python-bs4 python-ceph-common python-certifi python-chardet 2019-12-05T06:59:17.334 INFO:teuthology.orchestra.run.smithi040.stdout: python-cherrypy3 python-dnspython python-fasteners python-formencode 2019-12-05T06:59:17.334 INFO:teuthology.orchestra.run.smithi040.stdout: python-funcsigs python-html5lib python-httplib2 python-joblib python-jwt 2019-12-05T06:59:17.335 INFO:teuthology.orchestra.run.smithi040.stdout: python-kubernetes python-logutils python-lxml python-mako python-markupsafe 2019-12-05T06:59:17.335 INFO:teuthology.orchestra.run.smithi040.stdout: python-monotonic python-oauth2client python-openid python-paste 2019-12-05T06:59:17.335 INFO:teuthology.orchestra.run.smithi040.stdout: python-pastedeploy python-pastedeploy-tpl python-pastescript python-pecan 2019-12-05T06:59:17.335 INFO:teuthology.orchestra.run.smithi040.stdout: python-pluggy python-prettytable python-py python-pyasn1 2019-12-05T06:59:17.335 INFO:teuthology.orchestra.run.smithi040.stdout: python-pyasn1-modules python-pyinotify python-pytest python-repoze.lru 2019-12-05T06:59:17.335 INFO:teuthology.orchestra.run.smithi040.stdout: python-requests python-routes python-rsa python-scgi python-simplegeneric 2019-12-05T06:59:17.336 INFO:teuthology.orchestra.run.smithi040.stdout: python-simplejson python-singledispatch python-sklearn python-sklearn-lib 2019-12-05T06:59:17.336 INFO:teuthology.orchestra.run.smithi040.stdout: python-sqlalchemy python-sqlalchemy-ext python-tempita python-uritemplate 2019-12-05T06:59:17.336 INFO:teuthology.orchestra.run.smithi040.stdout: python-urllib3 python-waitress python-webencodings python-webob 2019-12-05T06:59:17.336 INFO:teuthology.orchestra.run.smithi040.stdout: python-websocket python-webtest python-werkzeug python-yaml socat xmlstarlet 2019-12-05T06:59:17.336 INFO:teuthology.orchestra.run.smithi040.stdout:Use 'sudo apt autoremove' to remove them. 2019-12-05T06:59:17.369 INFO:teuthology.orchestra.run.smithi040.stdout:0 upgraded, 0 newly installed, 0 to remove and 51 not upgraded. 2019-12-05T06:59:17.371 INFO:teuthology.orchestra.run.smithi040.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2019-12-05T06:59:17.452 INFO:teuthology.orchestra.run.smithi040.stdout:Reading package lists... 2019-12-05T06:59:17.585 INFO:teuthology.orchestra.run.smithi042.stdout:Processing triggers for man-db (2.8.3-2ubuntu0.1) ... 2019-12-05T06:59:17.705 INFO:teuthology.orchestra.run.smithi040.stdout:Building dependency tree... 2019-12-05T06:59:17.706 INFO:teuthology.orchestra.run.smithi040.stdout:Reading state information... 2019-12-05T06:59:17.875 INFO:teuthology.orchestra.run.smithi040.stdout:The following packages were automatically installed and are no longer required: 2019-12-05T06:59:17.876 INFO:teuthology.orchestra.run.smithi040.stdout: formencode-i18n jq libjq1 libjs-sphinxdoc libjs-underscore libleveldb1v5 2019-12-05T06:59:17.877 INFO:teuthology.orchestra.run.smithi040.stdout: liboath0 libonig4 librabbitmq4 libradosstriper1 librdkafka1 librgw2 2019-12-05T06:59:17.877 INFO:teuthology.orchestra.run.smithi040.stdout: libsnappy1v5 libxslt1.1 python-attr python-bcrypt python-bs4 2019-12-05T06:59:17.877 INFO:teuthology.orchestra.run.smithi040.stdout: python-ceph-argparse python-ceph-common python-cephfs python-certifi 2019-12-05T06:59:17.877 INFO:teuthology.orchestra.run.smithi040.stdout: python-chardet python-cherrypy3 python-dnspython python-fasteners 2019-12-05T06:59:17.877 INFO:teuthology.orchestra.run.smithi040.stdout: python-formencode python-funcsigs python-html5lib python-httplib2 2019-12-05T06:59:17.877 INFO:teuthology.orchestra.run.smithi040.stdout: python-joblib python-jwt python-kubernetes python-logutils python-lxml 2019-12-05T06:59:17.878 INFO:teuthology.orchestra.run.smithi040.stdout: python-mako python-markupsafe python-monotonic python-oauth2client 2019-12-05T06:59:17.878 INFO:teuthology.orchestra.run.smithi040.stdout: python-openid python-paste python-pastedeploy python-pastedeploy-tpl 2019-12-05T06:59:17.878 INFO:teuthology.orchestra.run.smithi040.stdout: python-pastescript python-pecan python-pluggy python-prettytable python-py 2019-12-05T06:59:17.878 INFO:teuthology.orchestra.run.smithi040.stdout: python-pyasn1 python-pyasn1-modules python-pyinotify python-pytest 2019-12-05T06:59:17.878 INFO:teuthology.orchestra.run.smithi040.stdout: python-rados python-rbd python-repoze.lru python-requests python-rgw 2019-12-05T06:59:17.878 INFO:teuthology.orchestra.run.smithi040.stdout: python-routes python-rsa python-scgi python-simplegeneric python-simplejson 2019-12-05T06:59:17.878 INFO:teuthology.orchestra.run.smithi040.stdout: python-singledispatch python-sklearn python-sklearn-lib python-sqlalchemy 2019-12-05T06:59:17.879 INFO:teuthology.orchestra.run.smithi040.stdout: python-sqlalchemy-ext python-tempita python-uritemplate python-urllib3 2019-12-05T06:59:17.879 INFO:teuthology.orchestra.run.smithi040.stdout: python-waitress python-webencodings python-webob python-websocket 2019-12-05T06:59:17.879 INFO:teuthology.orchestra.run.smithi040.stdout: python-webtest python-werkzeug python-yaml socat xmlstarlet 2019-12-05T06:59:17.879 INFO:teuthology.orchestra.run.smithi040.stdout:Use 'sudo apt autoremove' to remove them. 2019-12-05T06:59:17.896 INFO:teuthology.orchestra.run.smithi040.stdout:The following packages will be REMOVED: 2019-12-05T06:59:17.896 INFO:teuthology.orchestra.run.smithi040.stdout: python-ceph* 2019-12-05T06:59:18.260 INFO:teuthology.orchestra.run.smithi040.stdout:0 upgraded, 0 newly installed, 1 to remove and 51 not upgraded. 2019-12-05T06:59:18.261 INFO:teuthology.orchestra.run.smithi040.stdout:After this operation, 9,216 B disk space will be freed. 2019-12-05T06:59:18.344 INFO:teuthology.orchestra.run.smithi042.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 142083 files and directories currently installed.) 2019-12-05T06:59:18.348 INFO:teuthology.orchestra.run.smithi042.stdout:Purging configuration files for ceph-fuse (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T06:59:18.365 INFO:teuthology.orchestra.run.smithi040.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 142083 files and directories currently installed.) 2019-12-05T06:59:18.368 INFO:teuthology.orchestra.run.smithi040.stdout:Removing python-ceph (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T06:59:19.204 INFO:teuthology.orchestra.run.smithi040.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2019-12-05T06:59:19.282 INFO:teuthology.orchestra.run.smithi040.stdout:Reading package lists... 2019-12-05T06:59:19.538 INFO:teuthology.orchestra.run.smithi040.stdout:Building dependency tree... 2019-12-05T06:59:19.538 INFO:teuthology.orchestra.run.smithi040.stdout:Reading state information... 2019-12-05T06:59:19.712 INFO:teuthology.orchestra.run.smithi040.stdout:The following packages were automatically installed and are no longer required: 2019-12-05T06:59:19.713 INFO:teuthology.orchestra.run.smithi040.stdout: formencode-i18n jq libjq1 libjs-sphinxdoc libjs-underscore libleveldb1v5 2019-12-05T06:59:19.713 INFO:teuthology.orchestra.run.smithi040.stdout: liboath0 libonig4 librabbitmq4 libradosstriper1 librdkafka1 librgw2 2019-12-05T06:59:19.713 INFO:teuthology.orchestra.run.smithi040.stdout: libsnappy1v5 libxslt1.1 python-attr python-bcrypt python-bs4 2019-12-05T06:59:19.714 INFO:teuthology.orchestra.run.smithi040.stdout: python-ceph-argparse python-ceph-common python-certifi python-chardet 2019-12-05T06:59:19.714 INFO:teuthology.orchestra.run.smithi040.stdout: python-cherrypy3 python-dnspython python-fasteners python-formencode 2019-12-05T06:59:19.714 INFO:teuthology.orchestra.run.smithi040.stdout: python-funcsigs python-html5lib python-httplib2 python-joblib python-jwt 2019-12-05T06:59:19.714 INFO:teuthology.orchestra.run.smithi040.stdout: python-kubernetes python-logutils python-lxml python-mako python-markupsafe 2019-12-05T06:59:19.714 INFO:teuthology.orchestra.run.smithi040.stdout: python-monotonic python-oauth2client python-openid python-paste 2019-12-05T06:59:19.714 INFO:teuthology.orchestra.run.smithi040.stdout: python-pastedeploy python-pastedeploy-tpl python-pastescript python-pecan 2019-12-05T06:59:19.714 INFO:teuthology.orchestra.run.smithi040.stdout: python-pluggy python-prettytable python-py python-pyasn1 2019-12-05T06:59:19.715 INFO:teuthology.orchestra.run.smithi040.stdout: python-pyasn1-modules python-pyinotify python-pytest python-rados python-rbd 2019-12-05T06:59:19.715 INFO:teuthology.orchestra.run.smithi040.stdout: python-repoze.lru python-requests python-rgw python-routes python-rsa 2019-12-05T06:59:19.715 INFO:teuthology.orchestra.run.smithi040.stdout: python-scgi python-simplegeneric python-simplejson python-singledispatch 2019-12-05T06:59:19.715 INFO:teuthology.orchestra.run.smithi040.stdout: python-sklearn python-sklearn-lib python-sqlalchemy python-sqlalchemy-ext 2019-12-05T06:59:19.715 INFO:teuthology.orchestra.run.smithi040.stdout: python-tempita python-uritemplate python-urllib3 python-waitress 2019-12-05T06:59:19.715 INFO:teuthology.orchestra.run.smithi040.stdout: python-webencodings python-webob python-websocket python-webtest 2019-12-05T06:59:19.715 INFO:teuthology.orchestra.run.smithi040.stdout: python-werkzeug python-yaml python3-ceph-argparse python3-cmd2 2019-12-05T06:59:19.715 INFO:teuthology.orchestra.run.smithi040.stdout: python3-colorama python3-pyparsing python3-pyperclip python3-rados socat 2019-12-05T06:59:19.716 INFO:teuthology.orchestra.run.smithi040.stdout: xmlstarlet 2019-12-05T06:59:19.716 INFO:teuthology.orchestra.run.smithi040.stdout:Use 'sudo apt autoremove' to remove them. 2019-12-05T06:59:19.781 INFO:teuthology.orchestra.run.smithi040.stdout:The following packages will be REMOVED: 2019-12-05T06:59:19.781 INFO:teuthology.orchestra.run.smithi040.stdout: cephfs-shell* libcephfs-dev* libcephfs2* python-cephfs* python3-cephfs* 2019-12-05T06:59:19.948 INFO:teuthology.orchestra.run.smithi042.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2019-12-05T06:59:20.024 INFO:teuthology.orchestra.run.smithi042.stdout:Reading package lists... 2019-12-05T06:59:20.143 INFO:teuthology.orchestra.run.smithi040.stdout:0 upgraded, 0 newly installed, 5 to remove and 51 not upgraded. 2019-12-05T06:59:20.143 INFO:teuthology.orchestra.run.smithi040.stdout:After this operation, 2,243 kB disk space will be freed. 2019-12-05T06:59:20.257 INFO:teuthology.orchestra.run.smithi040.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 142081 files and directories currently installed.) 2019-12-05T06:59:20.260 INFO:teuthology.orchestra.run.smithi040.stdout:Removing cephfs-shell (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T06:59:20.279 INFO:teuthology.orchestra.run.smithi042.stdout:Building dependency tree... 2019-12-05T06:59:20.279 INFO:teuthology.orchestra.run.smithi042.stdout:Reading state information... 2019-12-05T06:59:20.455 INFO:teuthology.orchestra.run.smithi042.stdout:Package 'ceph-test' is not installed, so not removed 2019-12-05T06:59:20.455 INFO:teuthology.orchestra.run.smithi042.stdout:The following packages were automatically installed and are no longer required: 2019-12-05T06:59:20.456 INFO:teuthology.orchestra.run.smithi042.stdout: formencode-i18n jq libjq1 libjs-sphinxdoc libjs-underscore libleveldb1v5 2019-12-05T06:59:20.456 INFO:teuthology.orchestra.run.smithi042.stdout: liboath0 libonig4 libradosstriper1 libsnappy1v5 libxslt1.1 python-attr 2019-12-05T06:59:20.457 INFO:teuthology.orchestra.run.smithi042.stdout: python-bcrypt python-bs4 python-ceph-common python-certifi python-chardet 2019-12-05T06:59:20.457 INFO:teuthology.orchestra.run.smithi042.stdout: python-cherrypy3 python-dnspython python-fasteners python-formencode 2019-12-05T06:59:20.457 INFO:teuthology.orchestra.run.smithi042.stdout: python-funcsigs python-html5lib python-httplib2 python-joblib python-jwt 2019-12-05T06:59:20.457 INFO:teuthology.orchestra.run.smithi042.stdout: python-kubernetes python-logutils python-lxml python-mako python-markupsafe 2019-12-05T06:59:20.457 INFO:teuthology.orchestra.run.smithi042.stdout: python-monotonic python-oauth2client python-openid python-paste 2019-12-05T06:59:20.457 INFO:teuthology.orchestra.run.smithi042.stdout: python-pastedeploy python-pastedeploy-tpl python-pastescript python-pecan 2019-12-05T06:59:20.457 INFO:teuthology.orchestra.run.smithi042.stdout: python-pluggy python-prettytable python-py python-pyasn1 2019-12-05T06:59:20.458 INFO:teuthology.orchestra.run.smithi042.stdout: python-pyasn1-modules python-pyinotify python-pytest python-repoze.lru 2019-12-05T06:59:20.458 INFO:teuthology.orchestra.run.smithi042.stdout: python-requests python-routes python-rsa python-scgi python-simplegeneric 2019-12-05T06:59:20.458 INFO:teuthology.orchestra.run.smithi042.stdout: python-simplejson python-singledispatch python-sklearn python-sklearn-lib 2019-12-05T06:59:20.458 INFO:teuthology.orchestra.run.smithi042.stdout: python-sqlalchemy python-sqlalchemy-ext python-tempita python-uritemplate 2019-12-05T06:59:20.458 INFO:teuthology.orchestra.run.smithi042.stdout: python-urllib3 python-waitress python-webencodings python-webob 2019-12-05T06:59:20.458 INFO:teuthology.orchestra.run.smithi042.stdout: python-websocket python-webtest python-werkzeug python-yaml socat xmlstarlet 2019-12-05T06:59:20.459 INFO:teuthology.orchestra.run.smithi042.stdout:Use 'sudo apt autoremove' to remove them. 2019-12-05T06:59:20.493 INFO:teuthology.orchestra.run.smithi042.stdout:0 upgraded, 0 newly installed, 0 to remove and 51 not upgraded. 2019-12-05T06:59:20.493 INFO:teuthology.orchestra.run.smithi042.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2019-12-05T06:59:20.574 INFO:teuthology.orchestra.run.smithi042.stdout:Reading package lists... 2019-12-05T06:59:20.602 INFO:teuthology.orchestra.run.smithi040.stdout:Removing libcephfs-dev (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T06:59:20.806 INFO:teuthology.orchestra.run.smithi042.stdout:Building dependency tree... 2019-12-05T06:59:20.807 INFO:teuthology.orchestra.run.smithi042.stdout:Reading state information... 2019-12-05T06:59:20.979 INFO:teuthology.orchestra.run.smithi042.stdout:Package 'radosgw' is not installed, so not removed 2019-12-05T06:59:20.980 INFO:teuthology.orchestra.run.smithi042.stdout:The following packages were automatically installed and are no longer required: 2019-12-05T06:59:20.981 INFO:teuthology.orchestra.run.smithi042.stdout: formencode-i18n jq libjq1 libjs-sphinxdoc libjs-underscore libleveldb1v5 2019-12-05T06:59:20.981 INFO:teuthology.orchestra.run.smithi042.stdout: liboath0 libonig4 libradosstriper1 libsnappy1v5 libxslt1.1 python-attr 2019-12-05T06:59:20.981 INFO:teuthology.orchestra.run.smithi042.stdout: python-bcrypt python-bs4 python-ceph-common python-certifi python-chardet 2019-12-05T06:59:20.981 INFO:teuthology.orchestra.run.smithi042.stdout: python-cherrypy3 python-dnspython python-fasteners python-formencode 2019-12-05T06:59:20.981 INFO:teuthology.orchestra.run.smithi042.stdout: python-funcsigs python-html5lib python-httplib2 python-joblib python-jwt 2019-12-05T06:59:20.981 INFO:teuthology.orchestra.run.smithi042.stdout: python-kubernetes python-logutils python-lxml python-mako python-markupsafe 2019-12-05T06:59:20.982 INFO:teuthology.orchestra.run.smithi042.stdout: python-monotonic python-oauth2client python-openid python-paste 2019-12-05T06:59:20.982 INFO:teuthology.orchestra.run.smithi042.stdout: python-pastedeploy python-pastedeploy-tpl python-pastescript python-pecan 2019-12-05T06:59:20.982 INFO:teuthology.orchestra.run.smithi042.stdout: python-pluggy python-prettytable python-py python-pyasn1 2019-12-05T06:59:20.982 INFO:teuthology.orchestra.run.smithi042.stdout: python-pyasn1-modules python-pyinotify python-pytest python-repoze.lru 2019-12-05T06:59:20.982 INFO:teuthology.orchestra.run.smithi042.stdout: python-requests python-routes python-rsa python-scgi python-simplegeneric 2019-12-05T06:59:20.982 INFO:teuthology.orchestra.run.smithi042.stdout: python-simplejson python-singledispatch python-sklearn python-sklearn-lib 2019-12-05T06:59:20.982 INFO:teuthology.orchestra.run.smithi042.stdout: python-sqlalchemy python-sqlalchemy-ext python-tempita python-uritemplate 2019-12-05T06:59:20.982 INFO:teuthology.orchestra.run.smithi042.stdout: python-urllib3 python-waitress python-webencodings python-webob 2019-12-05T06:59:20.983 INFO:teuthology.orchestra.run.smithi042.stdout: python-websocket python-webtest python-werkzeug python-yaml socat xmlstarlet 2019-12-05T06:59:20.983 INFO:teuthology.orchestra.run.smithi042.stdout:Use 'sudo apt autoremove' to remove them. 2019-12-05T06:59:21.017 INFO:teuthology.orchestra.run.smithi042.stdout:0 upgraded, 0 newly installed, 0 to remove and 51 not upgraded. 2019-12-05T06:59:21.018 INFO:teuthology.orchestra.run.smithi042.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2019-12-05T06:59:21.098 INFO:teuthology.orchestra.run.smithi042.stdout:Reading package lists... 2019-12-05T06:59:21.330 INFO:teuthology.orchestra.run.smithi042.stdout:Building dependency tree... 2019-12-05T06:59:21.331 INFO:teuthology.orchestra.run.smithi042.stdout:Reading state information... 2019-12-05T06:59:21.414 INFO:teuthology.orchestra.run.smithi040.stdout:Removing python3-cephfs (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T06:59:21.505 INFO:teuthology.orchestra.run.smithi042.stdout:The following packages were automatically installed and are no longer required: 2019-12-05T06:59:21.505 INFO:teuthology.orchestra.run.smithi042.stdout: formencode-i18n jq libjq1 libjs-sphinxdoc libjs-underscore libleveldb1v5 2019-12-05T06:59:21.505 INFO:teuthology.orchestra.run.smithi042.stdout: liboath0 libonig4 librabbitmq4 libradosstriper1 librdkafka1 librgw2 2019-12-05T06:59:21.506 INFO:teuthology.orchestra.run.smithi042.stdout: libsnappy1v5 libxslt1.1 python-attr python-bcrypt python-bs4 2019-12-05T06:59:21.507 INFO:teuthology.orchestra.run.smithi042.stdout: python-ceph-argparse python-ceph-common python-cephfs python-certifi 2019-12-05T06:59:21.507 INFO:teuthology.orchestra.run.smithi042.stdout: python-chardet python-cherrypy3 python-dnspython python-fasteners 2019-12-05T06:59:21.507 INFO:teuthology.orchestra.run.smithi042.stdout: python-formencode python-funcsigs python-html5lib python-httplib2 2019-12-05T06:59:21.507 INFO:teuthology.orchestra.run.smithi042.stdout: python-joblib python-jwt python-kubernetes python-logutils python-lxml 2019-12-05T06:59:21.508 INFO:teuthology.orchestra.run.smithi042.stdout: python-mako python-markupsafe python-monotonic python-oauth2client 2019-12-05T06:59:21.508 INFO:teuthology.orchestra.run.smithi042.stdout: python-openid python-paste python-pastedeploy python-pastedeploy-tpl 2019-12-05T06:59:21.508 INFO:teuthology.orchestra.run.smithi042.stdout: python-pastescript python-pecan python-pluggy python-prettytable python-py 2019-12-05T06:59:21.509 INFO:teuthology.orchestra.run.smithi042.stdout: python-pyasn1 python-pyasn1-modules python-pyinotify python-pytest 2019-12-05T06:59:21.509 INFO:teuthology.orchestra.run.smithi042.stdout: python-rados python-rbd python-repoze.lru python-requests python-rgw 2019-12-05T06:59:21.509 INFO:teuthology.orchestra.run.smithi042.stdout: python-routes python-rsa python-scgi python-simplegeneric python-simplejson 2019-12-05T06:59:21.509 INFO:teuthology.orchestra.run.smithi042.stdout: python-singledispatch python-sklearn python-sklearn-lib python-sqlalchemy 2019-12-05T06:59:21.510 INFO:teuthology.orchestra.run.smithi042.stdout: python-sqlalchemy-ext python-tempita python-uritemplate python-urllib3 2019-12-05T06:59:21.510 INFO:teuthology.orchestra.run.smithi042.stdout: python-waitress python-webencodings python-webob python-websocket 2019-12-05T06:59:21.510 INFO:teuthology.orchestra.run.smithi042.stdout: python-webtest python-werkzeug python-yaml socat xmlstarlet 2019-12-05T06:59:21.511 INFO:teuthology.orchestra.run.smithi042.stdout:Use 'sudo apt autoremove' to remove them. 2019-12-05T06:59:21.525 INFO:teuthology.orchestra.run.smithi042.stdout:The following packages will be REMOVED: 2019-12-05T06:59:21.525 INFO:teuthology.orchestra.run.smithi042.stdout: python-ceph* 2019-12-05T06:59:21.885 INFO:teuthology.orchestra.run.smithi040.stdout:Removing python-cephfs (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T06:59:21.891 INFO:teuthology.orchestra.run.smithi042.stdout:0 upgraded, 0 newly installed, 1 to remove and 51 not upgraded. 2019-12-05T06:59:21.891 INFO:teuthology.orchestra.run.smithi042.stdout:After this operation, 9,216 B disk space will be freed. 2019-12-05T06:59:22.007 INFO:teuthology.orchestra.run.smithi042.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 142083 files and directories currently installed.) 2019-12-05T06:59:22.011 INFO:teuthology.orchestra.run.smithi042.stdout:Removing python-ceph (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T06:59:22.337 INFO:teuthology.orchestra.run.smithi040.stdout:Removing libcephfs2 (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T06:59:22.780 INFO:teuthology.orchestra.run.smithi042.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2019-12-05T06:59:22.858 INFO:teuthology.orchestra.run.smithi042.stdout:Reading package lists... 2019-12-05T06:59:22.926 INFO:teuthology.orchestra.run.smithi040.stdout:Processing triggers for libc-bin (2.27-3ubuntu1) ... 2019-12-05T06:59:23.112 INFO:teuthology.orchestra.run.smithi042.stdout:Building dependency tree... 2019-12-05T06:59:23.113 INFO:teuthology.orchestra.run.smithi042.stdout:Reading state information... 2019-12-05T06:59:23.286 INFO:teuthology.orchestra.run.smithi042.stdout:The following packages were automatically installed and are no longer required: 2019-12-05T06:59:23.288 INFO:teuthology.orchestra.run.smithi042.stdout: formencode-i18n jq libjq1 libjs-sphinxdoc libjs-underscore libleveldb1v5 2019-12-05T06:59:23.288 INFO:teuthology.orchestra.run.smithi042.stdout: liboath0 libonig4 librabbitmq4 libradosstriper1 librdkafka1 librgw2 2019-12-05T06:59:23.288 INFO:teuthology.orchestra.run.smithi042.stdout: libsnappy1v5 libxslt1.1 python-attr python-bcrypt python-bs4 2019-12-05T06:59:23.288 INFO:teuthology.orchestra.run.smithi042.stdout: python-ceph-argparse python-ceph-common python-certifi python-chardet 2019-12-05T06:59:23.288 INFO:teuthology.orchestra.run.smithi042.stdout: python-cherrypy3 python-dnspython python-fasteners python-formencode 2019-12-05T06:59:23.288 INFO:teuthology.orchestra.run.smithi042.stdout: python-funcsigs python-html5lib python-httplib2 python-joblib python-jwt 2019-12-05T06:59:23.289 INFO:teuthology.orchestra.run.smithi042.stdout: python-kubernetes python-logutils python-lxml python-mako python-markupsafe 2019-12-05T06:59:23.289 INFO:teuthology.orchestra.run.smithi042.stdout: python-monotonic python-oauth2client python-openid python-paste 2019-12-05T06:59:23.289 INFO:teuthology.orchestra.run.smithi042.stdout: python-pastedeploy python-pastedeploy-tpl python-pastescript python-pecan 2019-12-05T06:59:23.289 INFO:teuthology.orchestra.run.smithi042.stdout: python-pluggy python-prettytable python-py python-pyasn1 2019-12-05T06:59:23.289 INFO:teuthology.orchestra.run.smithi042.stdout: python-pyasn1-modules python-pyinotify python-pytest python-rados python-rbd 2019-12-05T06:59:23.289 INFO:teuthology.orchestra.run.smithi042.stdout: python-repoze.lru python-requests python-rgw python-routes python-rsa 2019-12-05T06:59:23.289 INFO:teuthology.orchestra.run.smithi042.stdout: python-scgi python-simplegeneric python-simplejson python-singledispatch 2019-12-05T06:59:23.290 INFO:teuthology.orchestra.run.smithi042.stdout: python-sklearn python-sklearn-lib python-sqlalchemy python-sqlalchemy-ext 2019-12-05T06:59:23.290 INFO:teuthology.orchestra.run.smithi042.stdout: python-tempita python-uritemplate python-urllib3 python-waitress 2019-12-05T06:59:23.290 INFO:teuthology.orchestra.run.smithi042.stdout: python-webencodings python-webob python-websocket python-webtest 2019-12-05T06:59:23.290 INFO:teuthology.orchestra.run.smithi042.stdout: python-werkzeug python-yaml python3-ceph-argparse python3-cmd2 2019-12-05T06:59:23.290 INFO:teuthology.orchestra.run.smithi042.stdout: python3-colorama python3-pyparsing python3-pyperclip python3-rados socat 2019-12-05T06:59:23.290 INFO:teuthology.orchestra.run.smithi042.stdout: xmlstarlet 2019-12-05T06:59:23.291 INFO:teuthology.orchestra.run.smithi042.stdout:Use 'sudo apt autoremove' to remove them. 2019-12-05T06:59:23.356 INFO:teuthology.orchestra.run.smithi042.stdout:The following packages will be REMOVED: 2019-12-05T06:59:23.357 INFO:teuthology.orchestra.run.smithi042.stdout: cephfs-shell* libcephfs-dev* libcephfs2* python-cephfs* python3-cephfs* 2019-12-05T06:59:23.430 INFO:teuthology.orchestra.run.smithi040.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2019-12-05T06:59:23.508 INFO:teuthology.orchestra.run.smithi040.stdout:Reading package lists... 2019-12-05T06:59:23.732 INFO:teuthology.orchestra.run.smithi042.stdout:0 upgraded, 0 newly installed, 5 to remove and 51 not upgraded. 2019-12-05T06:59:23.732 INFO:teuthology.orchestra.run.smithi042.stdout:After this operation, 2,243 kB disk space will be freed. 2019-12-05T06:59:23.768 INFO:teuthology.orchestra.run.smithi040.stdout:Building dependency tree... 2019-12-05T06:59:23.769 INFO:teuthology.orchestra.run.smithi040.stdout:Reading state information... 2019-12-05T06:59:23.845 INFO:teuthology.orchestra.run.smithi042.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 142081 files and directories currently installed.) 2019-12-05T06:59:23.849 INFO:teuthology.orchestra.run.smithi042.stdout:Removing cephfs-shell (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T06:59:23.972 INFO:teuthology.orchestra.run.smithi040.stdout:Package 'libcephfs-dev' is not installed, so not removed 2019-12-05T06:59:23.972 INFO:teuthology.orchestra.run.smithi040.stdout:The following packages were automatically installed and are no longer required: 2019-12-05T06:59:23.974 INFO:teuthology.orchestra.run.smithi040.stdout: formencode-i18n jq libjq1 libjs-sphinxdoc libjs-underscore libleveldb1v5 2019-12-05T06:59:23.974 INFO:teuthology.orchestra.run.smithi040.stdout: liboath0 libonig4 librabbitmq4 libradosstriper1 librdkafka1 librgw2 2019-12-05T06:59:23.974 INFO:teuthology.orchestra.run.smithi040.stdout: libsnappy1v5 libxslt1.1 python-attr python-bcrypt python-bs4 2019-12-05T06:59:23.974 INFO:teuthology.orchestra.run.smithi040.stdout: python-ceph-argparse python-ceph-common python-certifi python-chardet 2019-12-05T06:59:23.974 INFO:teuthology.orchestra.run.smithi040.stdout: python-cherrypy3 python-dnspython python-fasteners python-formencode 2019-12-05T06:59:23.975 INFO:teuthology.orchestra.run.smithi040.stdout: python-funcsigs python-html5lib python-httplib2 python-joblib python-jwt 2019-12-05T06:59:23.975 INFO:teuthology.orchestra.run.smithi040.stdout: python-kubernetes python-logutils python-lxml python-mako python-markupsafe 2019-12-05T06:59:23.975 INFO:teuthology.orchestra.run.smithi040.stdout: python-monotonic python-oauth2client python-openid python-paste 2019-12-05T06:59:23.975 INFO:teuthology.orchestra.run.smithi040.stdout: python-pastedeploy python-pastedeploy-tpl python-pastescript python-pecan 2019-12-05T06:59:23.975 INFO:teuthology.orchestra.run.smithi040.stdout: python-pluggy python-prettytable python-py python-pyasn1 2019-12-05T06:59:23.976 INFO:teuthology.orchestra.run.smithi040.stdout: python-pyasn1-modules python-pyinotify python-pytest python-rados python-rbd 2019-12-05T06:59:23.976 INFO:teuthology.orchestra.run.smithi040.stdout: python-repoze.lru python-requests python-rgw python-routes python-rsa 2019-12-05T06:59:23.976 INFO:teuthology.orchestra.run.smithi040.stdout: python-scgi python-simplegeneric python-simplejson python-singledispatch 2019-12-05T06:59:23.976 INFO:teuthology.orchestra.run.smithi040.stdout: python-sklearn python-sklearn-lib python-sqlalchemy python-sqlalchemy-ext 2019-12-05T06:59:23.976 INFO:teuthology.orchestra.run.smithi040.stdout: python-tempita python-uritemplate python-urllib3 python-waitress 2019-12-05T06:59:23.976 INFO:teuthology.orchestra.run.smithi040.stdout: python-webencodings python-webob python-websocket python-webtest 2019-12-05T06:59:23.976 INFO:teuthology.orchestra.run.smithi040.stdout: python-werkzeug python-yaml python3-ceph-argparse python3-cmd2 2019-12-05T06:59:23.977 INFO:teuthology.orchestra.run.smithi040.stdout: python3-colorama python3-pyparsing python3-pyperclip python3-rados socat 2019-12-05T06:59:23.977 INFO:teuthology.orchestra.run.smithi040.stdout: xmlstarlet 2019-12-05T06:59:23.977 INFO:teuthology.orchestra.run.smithi040.stdout:Use 'sudo apt autoremove' to remove them. 2019-12-05T06:59:23.993 INFO:teuthology.orchestra.run.smithi040.stdout:0 upgraded, 0 newly installed, 0 to remove and 51 not upgraded. 2019-12-05T06:59:23.993 INFO:teuthology.orchestra.run.smithi040.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2019-12-05T06:59:24.070 INFO:teuthology.orchestra.run.smithi040.stdout:Reading package lists... 2019-12-05T06:59:24.248 INFO:teuthology.orchestra.run.smithi042.stdout:Removing libcephfs-dev (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T06:59:24.306 INFO:teuthology.orchestra.run.smithi040.stdout:Building dependency tree... 2019-12-05T06:59:24.306 INFO:teuthology.orchestra.run.smithi040.stdout:Reading state information... 2019-12-05T06:59:24.490 INFO:teuthology.orchestra.run.smithi040.stdout:The following packages were automatically installed and are no longer required: 2019-12-05T06:59:24.490 INFO:teuthology.orchestra.run.smithi040.stdout: cpu-checker formencode-i18n gstreamer1.0-plugins-base 2019-12-05T06:59:24.490 INFO:teuthology.orchestra.run.smithi040.stdout: gstreamer1.0-plugins-good gstreamer1.0-x ipxe-qemu 2019-12-05T06:59:24.491 INFO:teuthology.orchestra.run.smithi040.stdout: ipxe-qemu-256k-compat-efi-roms jq libaa1 libavc1394-0 libbluetooth3 2019-12-05T06:59:24.491 INFO:teuthology.orchestra.run.smithi040.stdout: libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1 2019-12-05T06:59:24.491 INFO:teuthology.orchestra.run.smithi040.stdout: libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0 2019-12-05T06:59:24.491 INFO:teuthology.orchestra.run.smithi040.stdout: libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libjq1 2019-12-05T06:59:24.491 INFO:teuthology.orchestra.run.smithi040.stdout: libjs-sphinxdoc libjs-underscore libleveldb1v5 liblttng-ust0 libmp3lame0 2019-12-05T06:59:24.491 INFO:teuthology.orchestra.run.smithi040.stdout: libmpg123-0 liboath0 libonig4 libopus0 liborc-0.4-0 librabbitmq4 2019-12-05T06:59:24.492 INFO:teuthology.orchestra.run.smithi040.stdout: libraw1394-11 librdkafka1 libsamplerate0 libshout3 libsnappy1v5 libspeex1 2019-12-05T06:59:24.492 INFO:teuthology.orchestra.run.smithi040.stdout: libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0 libusbredirparser1 2019-12-05T06:59:24.492 INFO:teuthology.orchestra.run.smithi040.stdout: libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5 libwavpack1 libxen-4.9 2019-12-05T06:59:24.492 INFO:teuthology.orchestra.run.smithi040.stdout: libxenstore3.0 libxslt1.1 libyajl2 msr-tools python-attr python-bcrypt 2019-12-05T06:59:24.492 INFO:teuthology.orchestra.run.smithi040.stdout: python-bs4 python-ceph-argparse python-ceph-common python-certifi 2019-12-05T06:59:24.493 INFO:teuthology.orchestra.run.smithi040.stdout: python-chardet python-cherrypy3 python-dnspython python-fasteners 2019-12-05T06:59:24.493 INFO:teuthology.orchestra.run.smithi040.stdout: python-formencode python-funcsigs python-html5lib python-httplib2 2019-12-05T06:59:24.493 INFO:teuthology.orchestra.run.smithi040.stdout: python-joblib python-jwt python-kubernetes python-logutils python-lxml 2019-12-05T06:59:24.493 INFO:teuthology.orchestra.run.smithi040.stdout: python-mako python-markupsafe python-monotonic python-oauth2client 2019-12-05T06:59:24.493 INFO:teuthology.orchestra.run.smithi040.stdout: python-openid python-paste python-pastedeploy python-pastedeploy-tpl 2019-12-05T06:59:24.493 INFO:teuthology.orchestra.run.smithi040.stdout: python-pastescript python-pecan python-pluggy python-prettytable python-py 2019-12-05T06:59:24.493 INFO:teuthology.orchestra.run.smithi040.stdout: python-pyasn1 python-pyasn1-modules python-pyinotify python-pytest 2019-12-05T06:59:24.493 INFO:teuthology.orchestra.run.smithi040.stdout: python-repoze.lru python-requests python-routes python-rsa python-scgi 2019-12-05T06:59:24.494 INFO:teuthology.orchestra.run.smithi040.stdout: python-simplegeneric python-simplejson python-singledispatch python-sklearn 2019-12-05T06:59:24.494 INFO:teuthology.orchestra.run.smithi040.stdout: python-sklearn-lib python-sqlalchemy python-sqlalchemy-ext python-tempita 2019-12-05T06:59:24.494 INFO:teuthology.orchestra.run.smithi040.stdout: python-uritemplate python-urllib3 python-waitress python-webencodings 2019-12-05T06:59:24.494 INFO:teuthology.orchestra.run.smithi040.stdout: python-webob python-websocket python-webtest python-werkzeug python-yaml 2019-12-05T06:59:24.494 INFO:teuthology.orchestra.run.smithi040.stdout: python3-ceph-argparse python3-cmd2 python3-colorama python3-pyparsing 2019-12-05T06:59:24.494 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pyperclip seabios sharutils socat xmlstarlet 2019-12-05T06:59:24.494 INFO:teuthology.orchestra.run.smithi040.stdout:Use 'sudo apt autoremove' to remove them. 2019-12-05T06:59:24.618 INFO:teuthology.orchestra.run.smithi042.stdout:Removing python3-cephfs (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T06:59:24.664 INFO:teuthology.orchestra.run.smithi040.stdout:The following packages will be REMOVED: 2019-12-05T06:59:24.664 INFO:teuthology.orchestra.run.smithi040.stdout: librados2* libradosstriper1* librbd1* librgw2* python-rados* python-rbd* 2019-12-05T06:59:24.665 INFO:teuthology.orchestra.run.smithi040.stdout: python-rgw* python3-rados* qemu-block-extra* qemu-system-common* 2019-12-05T06:59:24.665 INFO:teuthology.orchestra.run.smithi040.stdout: qemu-system-x86* qemu-utils* rbd-fuse* 2019-12-05T06:59:25.030 INFO:teuthology.orchestra.run.smithi040.stdout:0 upgraded, 0 newly installed, 13 to remove and 51 not upgraded. 2019-12-05T06:59:25.031 INFO:teuthology.orchestra.run.smithi040.stdout:After this operation, 75.4 MB disk space will be freed. 2019-12-05T06:59:25.135 INFO:teuthology.orchestra.run.smithi040.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 142047 files and directories currently installed.) 2019-12-05T06:59:25.138 INFO:teuthology.orchestra.run.smithi040.stdout:Removing rbd-fuse (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T06:59:25.155 INFO:teuthology.orchestra.run.smithi042.stdout:Removing python-cephfs (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T06:59:25.615 INFO:teuthology.orchestra.run.smithi042.stdout:Removing libcephfs2 (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T06:59:25.700 INFO:teuthology.orchestra.run.smithi040.stdout:Removing python3-rados (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T06:59:26.082 INFO:teuthology.orchestra.run.smithi042.stdout:Processing triggers for libc-bin (2.27-3ubuntu1) ... 2019-12-05T06:59:26.095 INFO:teuthology.orchestra.run.smithi040.stdout:Removing libradosstriper1 (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T06:59:26.504 INFO:teuthology.orchestra.run.smithi040.stdout:Removing python-rbd (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T06:59:26.752 INFO:teuthology.orchestra.run.smithi042.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2019-12-05T06:59:26.829 INFO:teuthology.orchestra.run.smithi042.stdout:Reading package lists... 2019-12-05T06:59:26.899 INFO:teuthology.orchestra.run.smithi040.stdout:Removing qemu-utils (1:2.11+dfsg-1ubuntu7.21) ... 2019-12-05T06:59:27.087 INFO:teuthology.orchestra.run.smithi042.stdout:Building dependency tree... 2019-12-05T06:59:27.105 INFO:teuthology.orchestra.run.smithi042.stdout:Reading state information... 2019-12-05T06:59:27.267 INFO:teuthology.orchestra.run.smithi042.stdout:Package 'libcephfs-dev' is not installed, so not removed 2019-12-05T06:59:27.268 INFO:teuthology.orchestra.run.smithi042.stdout:The following packages were automatically installed and are no longer required: 2019-12-05T06:59:27.268 INFO:teuthology.orchestra.run.smithi042.stdout: formencode-i18n jq libjq1 libjs-sphinxdoc libjs-underscore libleveldb1v5 2019-12-05T06:59:27.268 INFO:teuthology.orchestra.run.smithi042.stdout: liboath0 libonig4 librabbitmq4 libradosstriper1 librdkafka1 librgw2 2019-12-05T06:59:27.269 INFO:teuthology.orchestra.run.smithi042.stdout: libsnappy1v5 libxslt1.1 python-attr python-bcrypt python-bs4 2019-12-05T06:59:27.270 INFO:teuthology.orchestra.run.smithi042.stdout: python-ceph-argparse python-ceph-common python-certifi python-chardet 2019-12-05T06:59:27.270 INFO:teuthology.orchestra.run.smithi042.stdout: python-cherrypy3 python-dnspython python-fasteners python-formencode 2019-12-05T06:59:27.270 INFO:teuthology.orchestra.run.smithi042.stdout: python-funcsigs python-html5lib python-httplib2 python-joblib python-jwt 2019-12-05T06:59:27.270 INFO:teuthology.orchestra.run.smithi042.stdout: python-kubernetes python-logutils python-lxml python-mako python-markupsafe 2019-12-05T06:59:27.271 INFO:teuthology.orchestra.run.smithi042.stdout: python-monotonic python-oauth2client python-openid python-paste 2019-12-05T06:59:27.271 INFO:teuthology.orchestra.run.smithi042.stdout: python-pastedeploy python-pastedeploy-tpl python-pastescript python-pecan 2019-12-05T06:59:27.271 INFO:teuthology.orchestra.run.smithi042.stdout: python-pluggy python-prettytable python-py python-pyasn1 2019-12-05T06:59:27.271 INFO:teuthology.orchestra.run.smithi042.stdout: python-pyasn1-modules python-pyinotify python-pytest python-rados python-rbd 2019-12-05T06:59:27.272 INFO:teuthology.orchestra.run.smithi042.stdout: python-repoze.lru python-requests python-rgw python-routes python-rsa 2019-12-05T06:59:27.272 INFO:teuthology.orchestra.run.smithi042.stdout: python-scgi python-simplegeneric python-simplejson python-singledispatch 2019-12-05T06:59:27.272 INFO:teuthology.orchestra.run.smithi042.stdout: python-sklearn python-sklearn-lib python-sqlalchemy python-sqlalchemy-ext 2019-12-05T06:59:27.272 INFO:teuthology.orchestra.run.smithi042.stdout: python-tempita python-uritemplate python-urllib3 python-waitress 2019-12-05T06:59:27.273 INFO:teuthology.orchestra.run.smithi042.stdout: python-webencodings python-webob python-websocket python-webtest 2019-12-05T06:59:27.273 INFO:teuthology.orchestra.run.smithi042.stdout: python-werkzeug python-yaml python3-ceph-argparse python3-cmd2 2019-12-05T06:59:27.273 INFO:teuthology.orchestra.run.smithi042.stdout: python3-colorama python3-pyparsing python3-pyperclip python3-rados socat 2019-12-05T06:59:27.273 INFO:teuthology.orchestra.run.smithi042.stdout: xmlstarlet 2019-12-05T06:59:27.274 INFO:teuthology.orchestra.run.smithi042.stdout:Use 'sudo apt autoremove' to remove them. 2019-12-05T06:59:27.301 INFO:teuthology.orchestra.run.smithi042.stdout:0 upgraded, 0 newly installed, 0 to remove and 51 not upgraded. 2019-12-05T06:59:27.301 INFO:teuthology.orchestra.run.smithi042.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2019-12-05T06:59:27.320 INFO:teuthology.orchestra.run.smithi040.stdout:Removing python-rgw (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T06:59:27.376 INFO:teuthology.orchestra.run.smithi042.stdout:Reading package lists... 2019-12-05T06:59:27.609 INFO:teuthology.orchestra.run.smithi042.stdout:Building dependency tree... 2019-12-05T06:59:27.610 INFO:teuthology.orchestra.run.smithi042.stdout:Reading state information... 2019-12-05T06:59:27.690 INFO:teuthology.orchestra.run.smithi040.stdout:Removing librgw2 (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T06:59:27.782 INFO:teuthology.orchestra.run.smithi042.stdout:The following packages were automatically installed and are no longer required: 2019-12-05T06:59:27.783 INFO:teuthology.orchestra.run.smithi042.stdout: cpu-checker formencode-i18n gstreamer1.0-plugins-base 2019-12-05T06:59:27.783 INFO:teuthology.orchestra.run.smithi042.stdout: gstreamer1.0-plugins-good gstreamer1.0-x ipxe-qemu 2019-12-05T06:59:27.783 INFO:teuthology.orchestra.run.smithi042.stdout: ipxe-qemu-256k-compat-efi-roms jq libaa1 libavc1394-0 libbluetooth3 2019-12-05T06:59:27.783 INFO:teuthology.orchestra.run.smithi042.stdout: libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1 2019-12-05T06:59:27.784 INFO:teuthology.orchestra.run.smithi042.stdout: libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0 2019-12-05T06:59:27.784 INFO:teuthology.orchestra.run.smithi042.stdout: libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libjq1 2019-12-05T06:59:27.784 INFO:teuthology.orchestra.run.smithi042.stdout: libjs-sphinxdoc libjs-underscore libleveldb1v5 liblttng-ust0 libmp3lame0 2019-12-05T06:59:27.785 INFO:teuthology.orchestra.run.smithi042.stdout: libmpg123-0 liboath0 libonig4 libopus0 liborc-0.4-0 librabbitmq4 2019-12-05T06:59:27.785 INFO:teuthology.orchestra.run.smithi042.stdout: libraw1394-11 librdkafka1 libsamplerate0 libshout3 libsnappy1v5 libspeex1 2019-12-05T06:59:27.786 INFO:teuthology.orchestra.run.smithi042.stdout: libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0 libusbredirparser1 2019-12-05T06:59:27.786 INFO:teuthology.orchestra.run.smithi042.stdout: libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5 libwavpack1 libxen-4.9 2019-12-05T06:59:27.786 INFO:teuthology.orchestra.run.smithi042.stdout: libxenstore3.0 libxslt1.1 libyajl2 msr-tools python-attr python-bcrypt 2019-12-05T06:59:27.787 INFO:teuthology.orchestra.run.smithi042.stdout: python-bs4 python-ceph-argparse python-ceph-common python-certifi 2019-12-05T06:59:27.787 INFO:teuthology.orchestra.run.smithi042.stdout: python-chardet python-cherrypy3 python-dnspython python-fasteners 2019-12-05T06:59:27.787 INFO:teuthology.orchestra.run.smithi042.stdout: python-formencode python-funcsigs python-html5lib python-httplib2 2019-12-05T06:59:27.787 INFO:teuthology.orchestra.run.smithi042.stdout: python-joblib python-jwt python-kubernetes python-logutils python-lxml 2019-12-05T06:59:27.787 INFO:teuthology.orchestra.run.smithi042.stdout: python-mako python-markupsafe python-monotonic python-oauth2client 2019-12-05T06:59:27.788 INFO:teuthology.orchestra.run.smithi042.stdout: python-openid python-paste python-pastedeploy python-pastedeploy-tpl 2019-12-05T06:59:27.788 INFO:teuthology.orchestra.run.smithi042.stdout: python-pastescript python-pecan python-pluggy python-prettytable python-py 2019-12-05T06:59:27.788 INFO:teuthology.orchestra.run.smithi042.stdout: python-pyasn1 python-pyasn1-modules python-pyinotify python-pytest 2019-12-05T06:59:27.788 INFO:teuthology.orchestra.run.smithi042.stdout: python-repoze.lru python-requests python-routes python-rsa python-scgi 2019-12-05T06:59:27.788 INFO:teuthology.orchestra.run.smithi042.stdout: python-simplegeneric python-simplejson python-singledispatch python-sklearn 2019-12-05T06:59:27.788 INFO:teuthology.orchestra.run.smithi042.stdout: python-sklearn-lib python-sqlalchemy python-sqlalchemy-ext python-tempita 2019-12-05T06:59:27.788 INFO:teuthology.orchestra.run.smithi042.stdout: python-uritemplate python-urllib3 python-waitress python-webencodings 2019-12-05T06:59:27.789 INFO:teuthology.orchestra.run.smithi042.stdout: python-webob python-websocket python-webtest python-werkzeug python-yaml 2019-12-05T06:59:27.789 INFO:teuthology.orchestra.run.smithi042.stdout: python3-ceph-argparse python3-cmd2 python3-colorama python3-pyparsing 2019-12-05T06:59:27.789 INFO:teuthology.orchestra.run.smithi042.stdout: python3-pyperclip seabios sharutils socat xmlstarlet 2019-12-05T06:59:27.789 INFO:teuthology.orchestra.run.smithi042.stdout:Use 'sudo apt autoremove' to remove them. 2019-12-05T06:59:27.955 INFO:teuthology.orchestra.run.smithi042.stdout:The following packages will be REMOVED: 2019-12-05T06:59:27.956 INFO:teuthology.orchestra.run.smithi042.stdout: librados2* libradosstriper1* librbd1* librgw2* python-rados* python-rbd* 2019-12-05T06:59:27.956 INFO:teuthology.orchestra.run.smithi042.stdout: python-rgw* python3-rados* qemu-block-extra* qemu-system-common* 2019-12-05T06:59:27.956 INFO:teuthology.orchestra.run.smithi042.stdout: qemu-system-x86* qemu-utils* rbd-fuse* 2019-12-05T06:59:28.052 INFO:teuthology.orchestra.run.smithi040.stdout:Removing python-rados (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T06:59:28.313 INFO:teuthology.orchestra.run.smithi042.stdout:0 upgraded, 0 newly installed, 13 to remove and 51 not upgraded. 2019-12-05T06:59:28.314 INFO:teuthology.orchestra.run.smithi042.stdout:After this operation, 75.4 MB disk space will be freed. 2019-12-05T06:59:28.414 INFO:teuthology.orchestra.run.smithi040.stdout:Removing qemu-system-x86 (1:2.11+dfsg-1ubuntu7.21) ... 2019-12-05T06:59:28.445 INFO:teuthology.orchestra.run.smithi042.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 142047 files and directories currently installed.) 2019-12-05T06:59:28.450 INFO:teuthology.orchestra.run.smithi042.stdout:Removing rbd-fuse (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T06:59:28.725 INFO:teuthology.orchestra.run.smithi040.stdout:Removing qemu-system-common (1:2.11+dfsg-1ubuntu7.21) ... 2019-12-05T06:59:28.879 INFO:teuthology.orchestra.run.smithi042.stdout:Removing python3-rados (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T06:59:29.266 INFO:teuthology.orchestra.run.smithi042.stdout:Removing libradosstriper1 (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T06:59:29.453 INFO:teuthology.orchestra.run.smithi040.stdout:Removing qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.21) ... 2019-12-05T06:59:29.728 INFO:teuthology.orchestra.run.smithi042.stdout:Removing python-rbd (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T06:59:29.822 INFO:teuthology.orchestra.run.smithi040.stdout:Removing librbd1 (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T06:59:30.082 INFO:teuthology.orchestra.run.smithi042.stdout:Removing qemu-utils (1:2.11+dfsg-1ubuntu7.21) ... 2019-12-05T06:59:30.205 INFO:teuthology.orchestra.run.smithi040.stdout:Removing librados2 (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T06:59:30.569 INFO:teuthology.orchestra.run.smithi042.stdout:Removing python-rgw (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T06:59:30.638 INFO:teuthology.orchestra.run.smithi040.stdout:Processing triggers for man-db (2.8.3-2ubuntu0.1) ... 2019-12-05T06:59:30.931 INFO:teuthology.orchestra.run.smithi042.stdout:Removing librgw2 (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T06:59:31.293 INFO:teuthology.orchestra.run.smithi042.stdout:Removing python-rados (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T06:59:31.513 INFO:teuthology.orchestra.run.smithi040.stdout:Processing triggers for libc-bin (2.27-3ubuntu1) ... 2019-12-05T06:59:31.724 INFO:teuthology.orchestra.run.smithi042.stdout:Removing qemu-system-x86 (1:2.11+dfsg-1ubuntu7.21) ... 2019-12-05T06:59:31.885 INFO:teuthology.orchestra.run.smithi040.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 141862 files and directories currently installed.) 2019-12-05T06:59:31.889 INFO:teuthology.orchestra.run.smithi040.stdout:Purging configuration files for qemu-system-x86 (1:2.11+dfsg-1ubuntu7.21) ... 2019-12-05T06:59:31.991 INFO:teuthology.orchestra.run.smithi042.stdout:Removing qemu-system-common (1:2.11+dfsg-1ubuntu7.21) ... 2019-12-05T06:59:32.336 INFO:teuthology.orchestra.run.smithi040.stdout:Purging configuration files for qemu-system-common (1:2.11+dfsg-1ubuntu7.21) ... 2019-12-05T06:59:32.726 INFO:teuthology.orchestra.run.smithi042.stdout:Removing qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.21) ... 2019-12-05T06:59:33.089 INFO:teuthology.orchestra.run.smithi042.stdout:Removing librbd1 (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T06:59:33.450 INFO:teuthology.orchestra.run.smithi042.stdout:Removing librados2 (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T06:59:33.721 INFO:teuthology.orchestra.run.smithi040.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2019-12-05T06:59:33.801 INFO:teuthology.orchestra.run.smithi040.stdout:Reading package lists... 2019-12-05T06:59:33.933 INFO:teuthology.orchestra.run.smithi042.stdout:Processing triggers for man-db (2.8.3-2ubuntu0.1) ... 2019-12-05T06:59:34.056 INFO:teuthology.orchestra.run.smithi040.stdout:Building dependency tree... 2019-12-05T06:59:34.057 INFO:teuthology.orchestra.run.smithi040.stdout:Reading state information... 2019-12-05T06:59:34.241 INFO:teuthology.orchestra.run.smithi040.stdout:Package 'librbd1' is not installed, so not removed 2019-12-05T06:59:34.241 INFO:teuthology.orchestra.run.smithi040.stdout:The following packages were automatically installed and are no longer required: 2019-12-05T06:59:34.241 INFO:teuthology.orchestra.run.smithi040.stdout: cpu-checker formencode-i18n gstreamer1.0-plugins-base 2019-12-05T06:59:34.241 INFO:teuthology.orchestra.run.smithi040.stdout: gstreamer1.0-plugins-good gstreamer1.0-x ipxe-qemu 2019-12-05T06:59:34.241 INFO:teuthology.orchestra.run.smithi040.stdout: ipxe-qemu-256k-compat-efi-roms jq libaa1 libavc1394-0 libbluetooth3 2019-12-05T06:59:34.242 INFO:teuthology.orchestra.run.smithi040.stdout: libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1 2019-12-05T06:59:34.242 INFO:teuthology.orchestra.run.smithi040.stdout: libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0 2019-12-05T06:59:34.242 INFO:teuthology.orchestra.run.smithi040.stdout: libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libjq1 2019-12-05T06:59:34.242 INFO:teuthology.orchestra.run.smithi040.stdout: libjs-sphinxdoc libjs-underscore libleveldb1v5 liblttng-ust0 libmp3lame0 2019-12-05T06:59:34.242 INFO:teuthology.orchestra.run.smithi040.stdout: libmpg123-0 liboath0 libonig4 libopus0 liborc-0.4-0 librabbitmq4 2019-12-05T06:59:34.243 INFO:teuthology.orchestra.run.smithi040.stdout: libraw1394-11 librdkafka1 libsamplerate0 libshout3 libsnappy1v5 libspeex1 2019-12-05T06:59:34.243 INFO:teuthology.orchestra.run.smithi040.stdout: libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0 libusbredirparser1 2019-12-05T06:59:34.243 INFO:teuthology.orchestra.run.smithi040.stdout: libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5 libwavpack1 libxen-4.9 2019-12-05T06:59:34.243 INFO:teuthology.orchestra.run.smithi040.stdout: libxenstore3.0 libxslt1.1 libyajl2 msr-tools python-attr python-bcrypt 2019-12-05T06:59:34.243 INFO:teuthology.orchestra.run.smithi040.stdout: python-bs4 python-ceph-argparse python-ceph-common python-certifi 2019-12-05T06:59:34.243 INFO:teuthology.orchestra.run.smithi040.stdout: python-chardet python-cherrypy3 python-dnspython python-fasteners 2019-12-05T06:59:34.243 INFO:teuthology.orchestra.run.smithi040.stdout: python-formencode python-funcsigs python-html5lib python-httplib2 2019-12-05T06:59:34.244 INFO:teuthology.orchestra.run.smithi040.stdout: python-joblib python-jwt python-kubernetes python-logutils python-lxml 2019-12-05T06:59:34.244 INFO:teuthology.orchestra.run.smithi040.stdout: python-mako python-markupsafe python-monotonic python-oauth2client 2019-12-05T06:59:34.244 INFO:teuthology.orchestra.run.smithi040.stdout: python-openid python-paste python-pastedeploy python-pastedeploy-tpl 2019-12-05T06:59:34.244 INFO:teuthology.orchestra.run.smithi040.stdout: python-pastescript python-pecan python-pluggy python-prettytable python-py 2019-12-05T06:59:34.244 INFO:teuthology.orchestra.run.smithi040.stdout: python-pyasn1 python-pyasn1-modules python-pyinotify python-pytest 2019-12-05T06:59:34.244 INFO:teuthology.orchestra.run.smithi040.stdout: python-repoze.lru python-requests python-routes python-rsa python-scgi 2019-12-05T06:59:34.244 INFO:teuthology.orchestra.run.smithi040.stdout: python-simplegeneric python-simplejson python-singledispatch python-sklearn 2019-12-05T06:59:34.245 INFO:teuthology.orchestra.run.smithi040.stdout: python-sklearn-lib python-sqlalchemy python-sqlalchemy-ext python-tempita 2019-12-05T06:59:34.245 INFO:teuthology.orchestra.run.smithi040.stdout: python-uritemplate python-urllib3 python-waitress python-webencodings 2019-12-05T06:59:34.245 INFO:teuthology.orchestra.run.smithi040.stdout: python-webob python-websocket python-webtest python-werkzeug python-yaml 2019-12-05T06:59:34.245 INFO:teuthology.orchestra.run.smithi040.stdout: python3-ceph-argparse python3-cmd2 python3-colorama python3-pyparsing 2019-12-05T06:59:34.245 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pyperclip seabios sharutils socat xmlstarlet 2019-12-05T06:59:34.245 INFO:teuthology.orchestra.run.smithi040.stdout:Use 'sudo apt autoremove' to remove them. 2019-12-05T06:59:34.278 INFO:teuthology.orchestra.run.smithi040.stdout:0 upgraded, 0 newly installed, 0 to remove and 51 not upgraded. 2019-12-05T06:59:34.278 INFO:teuthology.orchestra.run.smithi040.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2019-12-05T06:59:34.353 INFO:teuthology.orchestra.run.smithi040.stdout:Reading package lists... 2019-12-05T06:59:34.609 INFO:teuthology.orchestra.run.smithi040.stdout:Building dependency tree... 2019-12-05T06:59:34.610 INFO:teuthology.orchestra.run.smithi040.stdout:Reading state information... 2019-12-05T06:59:34.786 INFO:teuthology.orchestra.run.smithi040.stdout:Package 'rbd-fuse' is not installed, so not removed 2019-12-05T06:59:34.787 INFO:teuthology.orchestra.run.smithi040.stdout:The following packages were automatically installed and are no longer required: 2019-12-05T06:59:34.787 INFO:teuthology.orchestra.run.smithi040.stdout: cpu-checker formencode-i18n gstreamer1.0-plugins-base 2019-12-05T06:59:34.787 INFO:teuthology.orchestra.run.smithi040.stdout: gstreamer1.0-plugins-good gstreamer1.0-x ipxe-qemu 2019-12-05T06:59:34.787 INFO:teuthology.orchestra.run.smithi040.stdout: ipxe-qemu-256k-compat-efi-roms jq libaa1 libavc1394-0 libbluetooth3 2019-12-05T06:59:34.787 INFO:teuthology.orchestra.run.smithi040.stdout: libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1 2019-12-05T06:59:34.787 INFO:teuthology.orchestra.run.smithi040.stdout: libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0 2019-12-05T06:59:34.788 INFO:teuthology.orchestra.run.smithi040.stdout: libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libjq1 2019-12-05T06:59:34.788 INFO:teuthology.orchestra.run.smithi040.stdout: libjs-sphinxdoc libjs-underscore libleveldb1v5 liblttng-ust0 libmp3lame0 2019-12-05T06:59:34.788 INFO:teuthology.orchestra.run.smithi040.stdout: libmpg123-0 liboath0 libonig4 libopus0 liborc-0.4-0 librabbitmq4 2019-12-05T06:59:34.788 INFO:teuthology.orchestra.run.smithi040.stdout: libraw1394-11 librdkafka1 libsamplerate0 libshout3 libsnappy1v5 libspeex1 2019-12-05T06:59:34.790 INFO:teuthology.orchestra.run.smithi040.stdout: libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0 libusbredirparser1 2019-12-05T06:59:34.791 INFO:teuthology.orchestra.run.smithi040.stdout: libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5 libwavpack1 libxen-4.9 2019-12-05T06:59:34.791 INFO:teuthology.orchestra.run.smithi040.stdout: libxenstore3.0 libxslt1.1 libyajl2 msr-tools python-attr python-bcrypt 2019-12-05T06:59:34.791 INFO:teuthology.orchestra.run.smithi040.stdout: python-bs4 python-ceph-argparse python-ceph-common python-certifi 2019-12-05T06:59:34.791 INFO:teuthology.orchestra.run.smithi040.stdout: python-chardet python-cherrypy3 python-dnspython python-fasteners 2019-12-05T06:59:34.791 INFO:teuthology.orchestra.run.smithi040.stdout: python-formencode python-funcsigs python-html5lib python-httplib2 2019-12-05T06:59:34.791 INFO:teuthology.orchestra.run.smithi040.stdout: python-joblib python-jwt python-kubernetes python-logutils python-lxml 2019-12-05T06:59:34.792 INFO:teuthology.orchestra.run.smithi040.stdout: python-mako python-markupsafe python-monotonic python-oauth2client 2019-12-05T06:59:34.792 INFO:teuthology.orchestra.run.smithi040.stdout: python-openid python-paste python-pastedeploy python-pastedeploy-tpl 2019-12-05T06:59:34.792 INFO:teuthology.orchestra.run.smithi040.stdout: python-pastescript python-pecan python-pluggy python-prettytable python-py 2019-12-05T06:59:34.792 INFO:teuthology.orchestra.run.smithi040.stdout: python-pyasn1 python-pyasn1-modules python-pyinotify python-pytest 2019-12-05T06:59:34.792 INFO:teuthology.orchestra.run.smithi040.stdout: python-repoze.lru python-requests python-routes python-rsa python-scgi 2019-12-05T06:59:34.792 INFO:teuthology.orchestra.run.smithi040.stdout: python-simplegeneric python-simplejson python-singledispatch python-sklearn 2019-12-05T06:59:34.793 INFO:teuthology.orchestra.run.smithi040.stdout: python-sklearn-lib python-sqlalchemy python-sqlalchemy-ext python-tempita 2019-12-05T06:59:34.793 INFO:teuthology.orchestra.run.smithi040.stdout: python-uritemplate python-urllib3 python-waitress python-webencodings 2019-12-05T06:59:34.793 INFO:teuthology.orchestra.run.smithi040.stdout: python-webob python-websocket python-webtest python-werkzeug python-yaml 2019-12-05T06:59:34.793 INFO:teuthology.orchestra.run.smithi040.stdout: python3-ceph-argparse python3-cmd2 python3-colorama python3-pyparsing 2019-12-05T06:59:34.793 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pyperclip seabios sharutils socat xmlstarlet 2019-12-05T06:59:34.794 INFO:teuthology.orchestra.run.smithi040.stdout:Use 'sudo apt autoremove' to remove them. 2019-12-05T06:59:34.824 INFO:teuthology.orchestra.run.smithi040.stdout:0 upgraded, 0 newly installed, 0 to remove and 51 not upgraded. 2019-12-05T06:59:34.824 INFO:teuthology.orchestra.run.smithi040.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2019-12-05T06:59:34.899 INFO:teuthology.orchestra.run.smithi042.stdout:Processing triggers for libc-bin (2.27-3ubuntu1) ... 2019-12-05T06:59:34.903 INFO:teuthology.orchestra.run.smithi040.stdout:Reading package lists... 2019-12-05T06:59:35.136 INFO:teuthology.orchestra.run.smithi040.stdout:Building dependency tree... 2019-12-05T06:59:35.136 INFO:teuthology.orchestra.run.smithi040.stdout:Reading state information... 2019-12-05T06:59:35.247 INFO:teuthology.orchestra.run.smithi042.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 141862 files and directories currently installed.) 2019-12-05T06:59:35.251 INFO:teuthology.orchestra.run.smithi042.stdout:Purging configuration files for qemu-system-x86 (1:2.11+dfsg-1ubuntu7.21) ... 2019-12-05T06:59:35.319 INFO:teuthology.orchestra.run.smithi040.stdout:Package 'python3-cephfs' is not installed, so not removed 2019-12-05T06:59:35.319 INFO:teuthology.orchestra.run.smithi040.stdout:The following packages were automatically installed and are no longer required: 2019-12-05T06:59:35.319 INFO:teuthology.orchestra.run.smithi040.stdout: cpu-checker formencode-i18n gstreamer1.0-plugins-base 2019-12-05T06:59:35.319 INFO:teuthology.orchestra.run.smithi040.stdout: gstreamer1.0-plugins-good gstreamer1.0-x ipxe-qemu 2019-12-05T06:59:35.319 INFO:teuthology.orchestra.run.smithi040.stdout: ipxe-qemu-256k-compat-efi-roms jq libaa1 libavc1394-0 libbluetooth3 2019-12-05T06:59:35.319 INFO:teuthology.orchestra.run.smithi040.stdout: libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1 2019-12-05T06:59:35.320 INFO:teuthology.orchestra.run.smithi040.stdout: libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0 2019-12-05T06:59:35.320 INFO:teuthology.orchestra.run.smithi040.stdout: libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libjq1 2019-12-05T06:59:35.320 INFO:teuthology.orchestra.run.smithi040.stdout: libjs-sphinxdoc libjs-underscore libleveldb1v5 liblttng-ust0 libmp3lame0 2019-12-05T06:59:35.320 INFO:teuthology.orchestra.run.smithi040.stdout: libmpg123-0 liboath0 libonig4 libopus0 liborc-0.4-0 librabbitmq4 2019-12-05T06:59:35.320 INFO:teuthology.orchestra.run.smithi040.stdout: libraw1394-11 librdkafka1 libsamplerate0 libshout3 libsnappy1v5 libspeex1 2019-12-05T06:59:35.321 INFO:teuthology.orchestra.run.smithi040.stdout: libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0 libusbredirparser1 2019-12-05T06:59:35.321 INFO:teuthology.orchestra.run.smithi040.stdout: libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5 libwavpack1 libxen-4.9 2019-12-05T06:59:35.321 INFO:teuthology.orchestra.run.smithi040.stdout: libxenstore3.0 libxslt1.1 libyajl2 msr-tools python-attr python-bcrypt 2019-12-05T06:59:35.321 INFO:teuthology.orchestra.run.smithi040.stdout: python-bs4 python-ceph-argparse python-ceph-common python-certifi 2019-12-05T06:59:35.321 INFO:teuthology.orchestra.run.smithi040.stdout: python-chardet python-cherrypy3 python-dnspython python-fasteners 2019-12-05T06:59:35.321 INFO:teuthology.orchestra.run.smithi040.stdout: python-formencode python-funcsigs python-html5lib python-httplib2 2019-12-05T06:59:35.321 INFO:teuthology.orchestra.run.smithi040.stdout: python-joblib python-jwt python-kubernetes python-logutils python-lxml 2019-12-05T06:59:35.322 INFO:teuthology.orchestra.run.smithi040.stdout: python-mako python-markupsafe python-monotonic python-oauth2client 2019-12-05T06:59:35.322 INFO:teuthology.orchestra.run.smithi040.stdout: python-openid python-paste python-pastedeploy python-pastedeploy-tpl 2019-12-05T06:59:35.322 INFO:teuthology.orchestra.run.smithi040.stdout: python-pastescript python-pecan python-pluggy python-prettytable python-py 2019-12-05T06:59:35.322 INFO:teuthology.orchestra.run.smithi040.stdout: python-pyasn1 python-pyasn1-modules python-pyinotify python-pytest 2019-12-05T06:59:35.322 INFO:teuthology.orchestra.run.smithi040.stdout: python-repoze.lru python-requests python-routes python-rsa python-scgi 2019-12-05T06:59:35.322 INFO:teuthology.orchestra.run.smithi040.stdout: python-simplegeneric python-simplejson python-singledispatch python-sklearn 2019-12-05T06:59:35.322 INFO:teuthology.orchestra.run.smithi040.stdout: python-sklearn-lib python-sqlalchemy python-sqlalchemy-ext python-tempita 2019-12-05T06:59:35.323 INFO:teuthology.orchestra.run.smithi040.stdout: python-uritemplate python-urllib3 python-waitress python-webencodings 2019-12-05T06:59:35.323 INFO:teuthology.orchestra.run.smithi040.stdout: python-webob python-websocket python-webtest python-werkzeug python-yaml 2019-12-05T06:59:35.323 INFO:teuthology.orchestra.run.smithi040.stdout: python3-ceph-argparse python3-cmd2 python3-colorama python3-pyparsing 2019-12-05T06:59:35.323 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pyperclip seabios sharutils socat xmlstarlet 2019-12-05T06:59:35.323 INFO:teuthology.orchestra.run.smithi040.stdout:Use 'sudo apt autoremove' to remove them. 2019-12-05T06:59:35.356 INFO:teuthology.orchestra.run.smithi040.stdout:0 upgraded, 0 newly installed, 0 to remove and 51 not upgraded. 2019-12-05T06:59:35.356 INFO:teuthology.orchestra.run.smithi040.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2019-12-05T06:59:35.433 INFO:teuthology.orchestra.run.smithi040.stdout:Reading package lists... 2019-12-05T06:59:35.644 INFO:teuthology.orchestra.run.smithi042.stdout:Purging configuration files for qemu-system-common (1:2.11+dfsg-1ubuntu7.21) ... 2019-12-05T06:59:35.662 INFO:teuthology.orchestra.run.smithi040.stdout:Building dependency tree... 2019-12-05T06:59:35.663 INFO:teuthology.orchestra.run.smithi040.stdout:Reading state information... 2019-12-05T06:59:35.837 INFO:teuthology.orchestra.run.smithi040.stdout:Package 'cephfs-shell' is not installed, so not removed 2019-12-05T06:59:35.837 INFO:teuthology.orchestra.run.smithi040.stdout:The following packages were automatically installed and are no longer required: 2019-12-05T06:59:35.837 INFO:teuthology.orchestra.run.smithi040.stdout: cpu-checker formencode-i18n gstreamer1.0-plugins-base 2019-12-05T06:59:35.838 INFO:teuthology.orchestra.run.smithi040.stdout: gstreamer1.0-plugins-good gstreamer1.0-x ipxe-qemu 2019-12-05T06:59:35.838 INFO:teuthology.orchestra.run.smithi040.stdout: ipxe-qemu-256k-compat-efi-roms jq libaa1 libavc1394-0 libbluetooth3 2019-12-05T06:59:35.838 INFO:teuthology.orchestra.run.smithi040.stdout: libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1 2019-12-05T06:59:35.838 INFO:teuthology.orchestra.run.smithi040.stdout: libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0 2019-12-05T06:59:35.838 INFO:teuthology.orchestra.run.smithi040.stdout: libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libjq1 2019-12-05T06:59:35.838 INFO:teuthology.orchestra.run.smithi040.stdout: libjs-sphinxdoc libjs-underscore libleveldb1v5 liblttng-ust0 libmp3lame0 2019-12-05T06:59:35.839 INFO:teuthology.orchestra.run.smithi040.stdout: libmpg123-0 liboath0 libonig4 libopus0 liborc-0.4-0 librabbitmq4 2019-12-05T06:59:35.839 INFO:teuthology.orchestra.run.smithi040.stdout: libraw1394-11 librdkafka1 libsamplerate0 libshout3 libsnappy1v5 libspeex1 2019-12-05T06:59:35.839 INFO:teuthology.orchestra.run.smithi040.stdout: libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0 libusbredirparser1 2019-12-05T06:59:35.839 INFO:teuthology.orchestra.run.smithi040.stdout: libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5 libwavpack1 libxen-4.9 2019-12-05T06:59:35.839 INFO:teuthology.orchestra.run.smithi040.stdout: libxenstore3.0 libxslt1.1 libyajl2 msr-tools python-attr python-bcrypt 2019-12-05T06:59:35.840 INFO:teuthology.orchestra.run.smithi040.stdout: python-bs4 python-ceph-argparse python-ceph-common python-certifi 2019-12-05T06:59:35.840 INFO:teuthology.orchestra.run.smithi040.stdout: python-chardet python-cherrypy3 python-dnspython python-fasteners 2019-12-05T06:59:35.840 INFO:teuthology.orchestra.run.smithi040.stdout: python-formencode python-funcsigs python-html5lib python-httplib2 2019-12-05T06:59:35.840 INFO:teuthology.orchestra.run.smithi040.stdout: python-joblib python-jwt python-kubernetes python-logutils python-lxml 2019-12-05T06:59:35.840 INFO:teuthology.orchestra.run.smithi040.stdout: python-mako python-markupsafe python-monotonic python-oauth2client 2019-12-05T06:59:35.840 INFO:teuthology.orchestra.run.smithi040.stdout: python-openid python-paste python-pastedeploy python-pastedeploy-tpl 2019-12-05T06:59:35.840 INFO:teuthology.orchestra.run.smithi040.stdout: python-pastescript python-pecan python-pluggy python-prettytable python-py 2019-12-05T06:59:35.841 INFO:teuthology.orchestra.run.smithi040.stdout: python-pyasn1 python-pyasn1-modules python-pyinotify python-pytest 2019-12-05T06:59:35.841 INFO:teuthology.orchestra.run.smithi040.stdout: python-repoze.lru python-requests python-routes python-rsa python-scgi 2019-12-05T06:59:35.841 INFO:teuthology.orchestra.run.smithi040.stdout: python-simplegeneric python-simplejson python-singledispatch python-sklearn 2019-12-05T06:59:35.841 INFO:teuthology.orchestra.run.smithi040.stdout: python-sklearn-lib python-sqlalchemy python-sqlalchemy-ext python-tempita 2019-12-05T06:59:35.841 INFO:teuthology.orchestra.run.smithi040.stdout: python-uritemplate python-urllib3 python-waitress python-webencodings 2019-12-05T06:59:35.841 INFO:teuthology.orchestra.run.smithi040.stdout: python-webob python-websocket python-webtest python-werkzeug python-yaml 2019-12-05T06:59:35.842 INFO:teuthology.orchestra.run.smithi040.stdout: python3-ceph-argparse python3-cmd2 python3-colorama python3-pyparsing 2019-12-05T06:59:35.842 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pyperclip seabios sharutils socat xmlstarlet 2019-12-05T06:59:35.842 INFO:teuthology.orchestra.run.smithi040.stdout:Use 'sudo apt autoremove' to remove them. 2019-12-05T06:59:35.872 INFO:teuthology.orchestra.run.smithi040.stdout:0 upgraded, 0 newly installed, 0 to remove and 51 not upgraded. 2019-12-05T06:59:35.872 INFO:teuthology.orchestra.run.smithi040.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2019-12-05T06:59:35.874 INFO:teuthology.orchestra.run.smithi040:> dpkg -l | grep '^.\(U\|H\)R' | awk '{print $2}' | sudo xargs --no-run-if-empty dpkg -P --force-remove-reinstreq 2019-12-05T06:59:35.901 INFO:teuthology.orchestra.run.smithi040:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" autoremove 2019-12-05T06:59:36.055 INFO:teuthology.orchestra.run.smithi040.stdout:Reading package lists... 2019-12-05T06:59:36.282 INFO:teuthology.orchestra.run.smithi040.stdout:Building dependency tree... 2019-12-05T06:59:36.282 INFO:teuthology.orchestra.run.smithi040.stdout:Reading state information... 2019-12-05T06:59:36.471 INFO:teuthology.orchestra.run.smithi040.stdout:The following packages will be REMOVED: 2019-12-05T06:59:36.472 INFO:teuthology.orchestra.run.smithi040.stdout: cpu-checker formencode-i18n gstreamer1.0-plugins-base 2019-12-05T06:59:36.472 INFO:teuthology.orchestra.run.smithi040.stdout: gstreamer1.0-plugins-good gstreamer1.0-x ipxe-qemu 2019-12-05T06:59:36.472 INFO:teuthology.orchestra.run.smithi040.stdout: ipxe-qemu-256k-compat-efi-roms jq libaa1 libavc1394-0 libbluetooth3 2019-12-05T06:59:36.472 INFO:teuthology.orchestra.run.smithi040.stdout: libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1 2019-12-05T06:59:36.472 INFO:teuthology.orchestra.run.smithi040.stdout: libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0 2019-12-05T06:59:36.472 INFO:teuthology.orchestra.run.smithi040.stdout: libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libjq1 2019-12-05T06:59:36.473 INFO:teuthology.orchestra.run.smithi040.stdout: libjs-sphinxdoc libjs-underscore libleveldb1v5 liblttng-ust0 libmp3lame0 2019-12-05T06:59:36.473 INFO:teuthology.orchestra.run.smithi040.stdout: libmpg123-0 liboath0 libonig4 libopus0 liborc-0.4-0 librabbitmq4 2019-12-05T06:59:36.473 INFO:teuthology.orchestra.run.smithi040.stdout: libraw1394-11 librdkafka1 libsamplerate0 libshout3 libsnappy1v5 libspeex1 2019-12-05T06:59:36.473 INFO:teuthology.orchestra.run.smithi040.stdout: libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0 libusbredirparser1 2019-12-05T06:59:36.473 INFO:teuthology.orchestra.run.smithi040.stdout: libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5 libwavpack1 libxen-4.9 2019-12-05T06:59:36.473 INFO:teuthology.orchestra.run.smithi040.stdout: libxenstore3.0 libxslt1.1 libyajl2 msr-tools python-attr python-bcrypt 2019-12-05T06:59:36.473 INFO:teuthology.orchestra.run.smithi040.stdout: python-bs4 python-ceph-argparse python-ceph-common python-certifi 2019-12-05T06:59:36.474 INFO:teuthology.orchestra.run.smithi040.stdout: python-chardet python-cherrypy3 python-dnspython python-fasteners 2019-12-05T06:59:36.474 INFO:teuthology.orchestra.run.smithi040.stdout: python-formencode python-funcsigs python-html5lib python-httplib2 2019-12-05T06:59:36.474 INFO:teuthology.orchestra.run.smithi040.stdout: python-joblib python-jwt python-kubernetes python-logutils python-lxml 2019-12-05T06:59:36.474 INFO:teuthology.orchestra.run.smithi040.stdout: python-mako python-markupsafe python-monotonic python-oauth2client 2019-12-05T06:59:36.474 INFO:teuthology.orchestra.run.smithi040.stdout: python-openid python-paste python-pastedeploy python-pastedeploy-tpl 2019-12-05T06:59:36.474 INFO:teuthology.orchestra.run.smithi040.stdout: python-pastescript python-pecan python-pluggy python-prettytable python-py 2019-12-05T06:59:36.475 INFO:teuthology.orchestra.run.smithi040.stdout: python-pyasn1 python-pyasn1-modules python-pyinotify python-pytest 2019-12-05T06:59:36.475 INFO:teuthology.orchestra.run.smithi040.stdout: python-repoze.lru python-requests python-routes python-rsa python-scgi 2019-12-05T06:59:36.475 INFO:teuthology.orchestra.run.smithi040.stdout: python-simplegeneric python-simplejson python-singledispatch python-sklearn 2019-12-05T06:59:36.475 INFO:teuthology.orchestra.run.smithi040.stdout: python-sklearn-lib python-sqlalchemy python-sqlalchemy-ext python-tempita 2019-12-05T06:59:36.475 INFO:teuthology.orchestra.run.smithi040.stdout: python-uritemplate python-urllib3 python-waitress python-webencodings 2019-12-05T06:59:36.475 INFO:teuthology.orchestra.run.smithi040.stdout: python-webob python-websocket python-webtest python-werkzeug python-yaml 2019-12-05T06:59:36.475 INFO:teuthology.orchestra.run.smithi040.stdout: python3-ceph-argparse python3-cmd2 python3-colorama python3-pyparsing 2019-12-05T06:59:36.476 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pyperclip seabios sharutils socat xmlstarlet 2019-12-05T06:59:36.830 INFO:teuthology.orchestra.run.smithi040.stdout:0 upgraded, 0 newly installed, 122 to remove and 50 not upgraded. 2019-12-05T06:59:36.830 INFO:teuthology.orchestra.run.smithi040.stdout:After this operation, 83.5 MB disk space will be freed. 2019-12-05T06:59:36.942 INFO:teuthology.orchestra.run.smithi040.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 141858 files and directories currently installed.) 2019-12-05T06:59:36.946 INFO:teuthology.orchestra.run.smithi040.stdout:Removing cpu-checker (0.7-0ubuntu7) ... 2019-12-05T06:59:36.969 INFO:teuthology.orchestra.run.smithi042.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2019-12-05T06:59:37.041 INFO:teuthology.orchestra.run.smithi042.stdout:Reading package lists... 2019-12-05T06:59:37.299 INFO:teuthology.orchestra.run.smithi042.stdout:Building dependency tree... 2019-12-05T06:59:37.300 INFO:teuthology.orchestra.run.smithi042.stdout:Reading state information... 2019-12-05T06:59:37.335 INFO:teuthology.orchestra.run.smithi040.stdout:Removing python-pecan (1.2.1-2) ... 2019-12-05T06:59:37.481 INFO:teuthology.orchestra.run.smithi042.stdout:Package 'librbd1' is not installed, so not removed 2019-12-05T06:59:37.481 INFO:teuthology.orchestra.run.smithi042.stdout:The following packages were automatically installed and are no longer required: 2019-12-05T06:59:37.481 INFO:teuthology.orchestra.run.smithi042.stdout: cpu-checker formencode-i18n gstreamer1.0-plugins-base 2019-12-05T06:59:37.482 INFO:teuthology.orchestra.run.smithi042.stdout: gstreamer1.0-plugins-good gstreamer1.0-x ipxe-qemu 2019-12-05T06:59:37.482 INFO:teuthology.orchestra.run.smithi042.stdout: ipxe-qemu-256k-compat-efi-roms jq libaa1 libavc1394-0 libbluetooth3 2019-12-05T06:59:37.482 INFO:teuthology.orchestra.run.smithi042.stdout: libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1 2019-12-05T06:59:37.482 INFO:teuthology.orchestra.run.smithi042.stdout: libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0 2019-12-05T06:59:37.482 INFO:teuthology.orchestra.run.smithi042.stdout: libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libjq1 2019-12-05T06:59:37.482 INFO:teuthology.orchestra.run.smithi042.stdout: libjs-sphinxdoc libjs-underscore libleveldb1v5 liblttng-ust0 libmp3lame0 2019-12-05T06:59:37.482 INFO:teuthology.orchestra.run.smithi042.stdout: libmpg123-0 liboath0 libonig4 libopus0 liborc-0.4-0 librabbitmq4 2019-12-05T06:59:37.483 INFO:teuthology.orchestra.run.smithi042.stdout: libraw1394-11 librdkafka1 libsamplerate0 libshout3 libsnappy1v5 libspeex1 2019-12-05T06:59:37.483 INFO:teuthology.orchestra.run.smithi042.stdout: libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0 libusbredirparser1 2019-12-05T06:59:37.483 INFO:teuthology.orchestra.run.smithi042.stdout: libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5 libwavpack1 libxen-4.9 2019-12-05T06:59:37.483 INFO:teuthology.orchestra.run.smithi042.stdout: libxenstore3.0 libxslt1.1 libyajl2 msr-tools python-attr python-bcrypt 2019-12-05T06:59:37.483 INFO:teuthology.orchestra.run.smithi042.stdout: python-bs4 python-ceph-argparse python-ceph-common python-certifi 2019-12-05T06:59:37.483 INFO:teuthology.orchestra.run.smithi042.stdout: python-chardet python-cherrypy3 python-dnspython python-fasteners 2019-12-05T06:59:37.483 INFO:teuthology.orchestra.run.smithi042.stdout: python-formencode python-funcsigs python-html5lib python-httplib2 2019-12-05T06:59:37.484 INFO:teuthology.orchestra.run.smithi042.stdout: python-joblib python-jwt python-kubernetes python-logutils python-lxml 2019-12-05T06:59:37.484 INFO:teuthology.orchestra.run.smithi042.stdout: python-mako python-markupsafe python-monotonic python-oauth2client 2019-12-05T06:59:37.484 INFO:teuthology.orchestra.run.smithi042.stdout: python-openid python-paste python-pastedeploy python-pastedeploy-tpl 2019-12-05T06:59:37.484 INFO:teuthology.orchestra.run.smithi042.stdout: python-pastescript python-pecan python-pluggy python-prettytable python-py 2019-12-05T06:59:37.484 INFO:teuthology.orchestra.run.smithi042.stdout: python-pyasn1 python-pyasn1-modules python-pyinotify python-pytest 2019-12-05T06:59:37.484 INFO:teuthology.orchestra.run.smithi042.stdout: python-repoze.lru python-requests python-routes python-rsa python-scgi 2019-12-05T06:59:37.484 INFO:teuthology.orchestra.run.smithi042.stdout: python-simplegeneric python-simplejson python-singledispatch python-sklearn 2019-12-05T06:59:37.484 INFO:teuthology.orchestra.run.smithi042.stdout: python-sklearn-lib python-sqlalchemy python-sqlalchemy-ext python-tempita 2019-12-05T06:59:37.485 INFO:teuthology.orchestra.run.smithi042.stdout: python-uritemplate python-urllib3 python-waitress python-webencodings 2019-12-05T06:59:37.485 INFO:teuthology.orchestra.run.smithi042.stdout: python-webob python-websocket python-webtest python-werkzeug python-yaml 2019-12-05T06:59:37.485 INFO:teuthology.orchestra.run.smithi042.stdout: python3-ceph-argparse python3-cmd2 python3-colorama python3-pyparsing 2019-12-05T06:59:37.485 INFO:teuthology.orchestra.run.smithi042.stdout: python3-pyperclip seabios sharutils socat xmlstarlet 2019-12-05T06:59:37.485 INFO:teuthology.orchestra.run.smithi042.stdout:Use 'sudo apt autoremove' to remove them. 2019-12-05T06:59:37.517 INFO:teuthology.orchestra.run.smithi042.stdout:0 upgraded, 0 newly installed, 0 to remove and 51 not upgraded. 2019-12-05T06:59:37.517 INFO:teuthology.orchestra.run.smithi042.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2019-12-05T06:59:37.599 INFO:teuthology.orchestra.run.smithi042.stdout:Reading package lists... 2019-12-05T06:59:37.680 INFO:teuthology.orchestra.run.smithi040.stdout:Removing python-webtest (2.0.28-1ubuntu1) ... 2019-12-05T06:59:37.862 INFO:teuthology.orchestra.run.smithi042.stdout:Building dependency tree... 2019-12-05T06:59:37.869 INFO:teuthology.orchestra.run.smithi042.stdout:Reading state information... 2019-12-05T06:59:38.043 INFO:teuthology.orchestra.run.smithi042.stdout:Package 'rbd-fuse' is not installed, so not removed 2019-12-05T06:59:38.043 INFO:teuthology.orchestra.run.smithi042.stdout:The following packages were automatically installed and are no longer required: 2019-12-05T06:59:38.043 INFO:teuthology.orchestra.run.smithi042.stdout: cpu-checker formencode-i18n gstreamer1.0-plugins-base 2019-12-05T06:59:38.043 INFO:teuthology.orchestra.run.smithi042.stdout: gstreamer1.0-plugins-good gstreamer1.0-x ipxe-qemu 2019-12-05T06:59:38.043 INFO:teuthology.orchestra.run.smithi042.stdout: ipxe-qemu-256k-compat-efi-roms jq libaa1 libavc1394-0 libbluetooth3 2019-12-05T06:59:38.044 INFO:teuthology.orchestra.run.smithi042.stdout: libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1 2019-12-05T06:59:38.044 INFO:teuthology.orchestra.run.smithi042.stdout: libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0 2019-12-05T06:59:38.044 INFO:teuthology.orchestra.run.smithi042.stdout: libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libjq1 2019-12-05T06:59:38.044 INFO:teuthology.orchestra.run.smithi042.stdout: libjs-sphinxdoc libjs-underscore libleveldb1v5 liblttng-ust0 libmp3lame0 2019-12-05T06:59:38.044 INFO:teuthology.orchestra.run.smithi042.stdout: libmpg123-0 liboath0 libonig4 libopus0 liborc-0.4-0 librabbitmq4 2019-12-05T06:59:38.044 INFO:teuthology.orchestra.run.smithi042.stdout: libraw1394-11 librdkafka1 libsamplerate0 libshout3 libsnappy1v5 libspeex1 2019-12-05T06:59:38.044 INFO:teuthology.orchestra.run.smithi042.stdout: libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0 libusbredirparser1 2019-12-05T06:59:38.045 INFO:teuthology.orchestra.run.smithi042.stdout: libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5 libwavpack1 libxen-4.9 2019-12-05T06:59:38.045 INFO:teuthology.orchestra.run.smithi042.stdout: libxenstore3.0 libxslt1.1 libyajl2 msr-tools python-attr python-bcrypt 2019-12-05T06:59:38.045 INFO:teuthology.orchestra.run.smithi042.stdout: python-bs4 python-ceph-argparse python-ceph-common python-certifi 2019-12-05T06:59:38.045 INFO:teuthology.orchestra.run.smithi042.stdout: python-chardet python-cherrypy3 python-dnspython python-fasteners 2019-12-05T06:59:38.045 INFO:teuthology.orchestra.run.smithi042.stdout: python-formencode python-funcsigs python-html5lib python-httplib2 2019-12-05T06:59:38.045 INFO:teuthology.orchestra.run.smithi042.stdout: python-joblib python-jwt python-kubernetes python-logutils python-lxml 2019-12-05T06:59:38.045 INFO:teuthology.orchestra.run.smithi042.stdout: python-mako python-markupsafe python-monotonic python-oauth2client 2019-12-05T06:59:38.046 INFO:teuthology.orchestra.run.smithi042.stdout: python-openid python-paste python-pastedeploy python-pastedeploy-tpl 2019-12-05T06:59:38.046 INFO:teuthology.orchestra.run.smithi042.stdout: python-pastescript python-pecan python-pluggy python-prettytable python-py 2019-12-05T06:59:38.046 INFO:teuthology.orchestra.run.smithi042.stdout: python-pyasn1 python-pyasn1-modules python-pyinotify python-pytest 2019-12-05T06:59:38.046 INFO:teuthology.orchestra.run.smithi042.stdout: python-repoze.lru python-requests python-routes python-rsa python-scgi 2019-12-05T06:59:38.046 INFO:teuthology.orchestra.run.smithi042.stdout: python-simplegeneric python-simplejson python-singledispatch python-sklearn 2019-12-05T06:59:38.046 INFO:teuthology.orchestra.run.smithi042.stdout: python-sklearn-lib python-sqlalchemy python-sqlalchemy-ext python-tempita 2019-12-05T06:59:38.046 INFO:teuthology.orchestra.run.smithi042.stdout: python-uritemplate python-urllib3 python-waitress python-webencodings 2019-12-05T06:59:38.046 INFO:teuthology.orchestra.run.smithi042.stdout: python-webob python-websocket python-webtest python-werkzeug python-yaml 2019-12-05T06:59:38.047 INFO:teuthology.orchestra.run.smithi042.stdout: python3-ceph-argparse python3-cmd2 python3-colorama python3-pyparsing 2019-12-05T06:59:38.047 INFO:teuthology.orchestra.run.smithi042.stdout: python3-pyperclip seabios sharutils socat xmlstarlet 2019-12-05T06:59:38.047 INFO:teuthology.orchestra.run.smithi042.stdout:Use 'sudo apt autoremove' to remove them. 2019-12-05T06:59:38.071 INFO:teuthology.orchestra.run.smithi042.stdout:0 upgraded, 0 newly installed, 0 to remove and 51 not upgraded. 2019-12-05T06:59:38.072 INFO:teuthology.orchestra.run.smithi042.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2019-12-05T06:59:38.116 INFO:teuthology.orchestra.run.smithi040.stdout:Removing python-pastescript (2.0.2-2) ... 2019-12-05T06:59:38.149 INFO:teuthology.orchestra.run.smithi042.stdout:Reading package lists... 2019-12-05T06:59:38.402 INFO:teuthology.orchestra.run.smithi042.stdout:Building dependency tree... 2019-12-05T06:59:38.403 INFO:teuthology.orchestra.run.smithi042.stdout:Reading state information... 2019-12-05T06:59:38.584 INFO:teuthology.orchestra.run.smithi042.stdout:Package 'python3-cephfs' is not installed, so not removed 2019-12-05T06:59:38.584 INFO:teuthology.orchestra.run.smithi042.stdout:The following packages were automatically installed and are no longer required: 2019-12-05T06:59:38.584 INFO:teuthology.orchestra.run.smithi042.stdout: cpu-checker formencode-i18n gstreamer1.0-plugins-base 2019-12-05T06:59:38.584 INFO:teuthology.orchestra.run.smithi042.stdout: gstreamer1.0-plugins-good gstreamer1.0-x ipxe-qemu 2019-12-05T06:59:38.585 INFO:teuthology.orchestra.run.smithi042.stdout: ipxe-qemu-256k-compat-efi-roms jq libaa1 libavc1394-0 libbluetooth3 2019-12-05T06:59:38.585 INFO:teuthology.orchestra.run.smithi042.stdout: libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1 2019-12-05T06:59:38.585 INFO:teuthology.orchestra.run.smithi042.stdout: libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0 2019-12-05T06:59:38.585 INFO:teuthology.orchestra.run.smithi042.stdout: libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libjq1 2019-12-05T06:59:38.585 INFO:teuthology.orchestra.run.smithi042.stdout: libjs-sphinxdoc libjs-underscore libleveldb1v5 liblttng-ust0 libmp3lame0 2019-12-05T06:59:38.585 INFO:teuthology.orchestra.run.smithi042.stdout: libmpg123-0 liboath0 libonig4 libopus0 liborc-0.4-0 librabbitmq4 2019-12-05T06:59:38.585 INFO:teuthology.orchestra.run.smithi042.stdout: libraw1394-11 librdkafka1 libsamplerate0 libshout3 libsnappy1v5 libspeex1 2019-12-05T06:59:38.586 INFO:teuthology.orchestra.run.smithi042.stdout: libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0 libusbredirparser1 2019-12-05T06:59:38.586 INFO:teuthology.orchestra.run.smithi042.stdout: libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5 libwavpack1 libxen-4.9 2019-12-05T06:59:38.586 INFO:teuthology.orchestra.run.smithi042.stdout: libxenstore3.0 libxslt1.1 libyajl2 msr-tools python-attr python-bcrypt 2019-12-05T06:59:38.586 INFO:teuthology.orchestra.run.smithi042.stdout: python-bs4 python-ceph-argparse python-ceph-common python-certifi 2019-12-05T06:59:38.587 INFO:teuthology.orchestra.run.smithi042.stdout: python-chardet python-cherrypy3 python-dnspython python-fasteners 2019-12-05T06:59:38.587 INFO:teuthology.orchestra.run.smithi042.stdout: python-formencode python-funcsigs python-html5lib python-httplib2 2019-12-05T06:59:38.587 INFO:teuthology.orchestra.run.smithi042.stdout: python-joblib python-jwt python-kubernetes python-logutils python-lxml 2019-12-05T06:59:38.587 INFO:teuthology.orchestra.run.smithi042.stdout: python-mako python-markupsafe python-monotonic python-oauth2client 2019-12-05T06:59:38.587 INFO:teuthology.orchestra.run.smithi042.stdout: python-openid python-paste python-pastedeploy python-pastedeploy-tpl 2019-12-05T06:59:38.587 INFO:teuthology.orchestra.run.smithi042.stdout: python-pastescript python-pecan python-pluggy python-prettytable python-py 2019-12-05T06:59:38.587 INFO:teuthology.orchestra.run.smithi042.stdout: python-pyasn1 python-pyasn1-modules python-pyinotify python-pytest 2019-12-05T06:59:38.588 INFO:teuthology.orchestra.run.smithi042.stdout: python-repoze.lru python-requests python-routes python-rsa python-scgi 2019-12-05T06:59:38.588 INFO:teuthology.orchestra.run.smithi042.stdout: python-simplegeneric python-simplejson python-singledispatch python-sklearn 2019-12-05T06:59:38.588 INFO:teuthology.orchestra.run.smithi042.stdout: python-sklearn-lib python-sqlalchemy python-sqlalchemy-ext python-tempita 2019-12-05T06:59:38.588 INFO:teuthology.orchestra.run.smithi042.stdout: python-uritemplate python-urllib3 python-waitress python-webencodings 2019-12-05T06:59:38.588 INFO:teuthology.orchestra.run.smithi042.stdout: python-webob python-websocket python-webtest python-werkzeug python-yaml 2019-12-05T06:59:38.588 INFO:teuthology.orchestra.run.smithi042.stdout: python3-ceph-argparse python3-cmd2 python3-colorama python3-pyparsing 2019-12-05T06:59:38.589 INFO:teuthology.orchestra.run.smithi042.stdout: python3-pyperclip seabios sharutils socat xmlstarlet 2019-12-05T06:59:38.589 INFO:teuthology.orchestra.run.smithi042.stdout:Use 'sudo apt autoremove' to remove them. 2019-12-05T06:59:38.610 INFO:teuthology.orchestra.run.smithi042.stdout:0 upgraded, 0 newly installed, 0 to remove and 51 not upgraded. 2019-12-05T06:59:38.613 INFO:teuthology.orchestra.run.smithi042.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2019-12-05T06:59:38.688 INFO:teuthology.orchestra.run.smithi042.stdout:Reading package lists... 2019-12-05T06:59:38.882 INFO:teuthology.orchestra.run.smithi040.stdout:Removing gstreamer1.0-plugins-good:amd64 (1.14.5-0ubuntu1~18.04.1) ... 2019-12-05T06:59:38.916 INFO:teuthology.orchestra.run.smithi042.stdout:Building dependency tree... 2019-12-05T06:59:38.917 INFO:teuthology.orchestra.run.smithi042.stdout:Reading state information... 2019-12-05T06:59:39.090 INFO:teuthology.orchestra.run.smithi042.stdout:Package 'cephfs-shell' is not installed, so not removed 2019-12-05T06:59:39.090 INFO:teuthology.orchestra.run.smithi042.stdout:The following packages were automatically installed and are no longer required: 2019-12-05T06:59:39.090 INFO:teuthology.orchestra.run.smithi042.stdout: cpu-checker formencode-i18n gstreamer1.0-plugins-base 2019-12-05T06:59:39.090 INFO:teuthology.orchestra.run.smithi042.stdout: gstreamer1.0-plugins-good gstreamer1.0-x ipxe-qemu 2019-12-05T06:59:39.090 INFO:teuthology.orchestra.run.smithi042.stdout: ipxe-qemu-256k-compat-efi-roms jq libaa1 libavc1394-0 libbluetooth3 2019-12-05T06:59:39.090 INFO:teuthology.orchestra.run.smithi042.stdout: libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1 2019-12-05T06:59:39.091 INFO:teuthology.orchestra.run.smithi042.stdout: libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0 2019-12-05T06:59:39.091 INFO:teuthology.orchestra.run.smithi042.stdout: libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libjq1 2019-12-05T06:59:39.091 INFO:teuthology.orchestra.run.smithi042.stdout: libjs-sphinxdoc libjs-underscore libleveldb1v5 liblttng-ust0 libmp3lame0 2019-12-05T06:59:39.091 INFO:teuthology.orchestra.run.smithi042.stdout: libmpg123-0 liboath0 libonig4 libopus0 liborc-0.4-0 librabbitmq4 2019-12-05T06:59:39.091 INFO:teuthology.orchestra.run.smithi042.stdout: libraw1394-11 librdkafka1 libsamplerate0 libshout3 libsnappy1v5 libspeex1 2019-12-05T06:59:39.091 INFO:teuthology.orchestra.run.smithi042.stdout: libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0 libusbredirparser1 2019-12-05T06:59:39.092 INFO:teuthology.orchestra.run.smithi042.stdout: libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5 libwavpack1 libxen-4.9 2019-12-05T06:59:39.092 INFO:teuthology.orchestra.run.smithi042.stdout: libxenstore3.0 libxslt1.1 libyajl2 msr-tools python-attr python-bcrypt 2019-12-05T06:59:39.093 INFO:teuthology.orchestra.run.smithi042.stdout: python-bs4 python-ceph-argparse python-ceph-common python-certifi 2019-12-05T06:59:39.093 INFO:teuthology.orchestra.run.smithi042.stdout: python-chardet python-cherrypy3 python-dnspython python-fasteners 2019-12-05T06:59:39.093 INFO:teuthology.orchestra.run.smithi042.stdout: python-formencode python-funcsigs python-html5lib python-httplib2 2019-12-05T06:59:39.093 INFO:teuthology.orchestra.run.smithi042.stdout: python-joblib python-jwt python-kubernetes python-logutils python-lxml 2019-12-05T06:59:39.093 INFO:teuthology.orchestra.run.smithi042.stdout: python-mako python-markupsafe python-monotonic python-oauth2client 2019-12-05T06:59:39.093 INFO:teuthology.orchestra.run.smithi042.stdout: python-openid python-paste python-pastedeploy python-pastedeploy-tpl 2019-12-05T06:59:39.094 INFO:teuthology.orchestra.run.smithi042.stdout: python-pastescript python-pecan python-pluggy python-prettytable python-py 2019-12-05T06:59:39.094 INFO:teuthology.orchestra.run.smithi042.stdout: python-pyasn1 python-pyasn1-modules python-pyinotify python-pytest 2019-12-05T06:59:39.094 INFO:teuthology.orchestra.run.smithi042.stdout: python-repoze.lru python-requests python-routes python-rsa python-scgi 2019-12-05T06:59:39.094 INFO:teuthology.orchestra.run.smithi042.stdout: python-simplegeneric python-simplejson python-singledispatch python-sklearn 2019-12-05T06:59:39.094 INFO:teuthology.orchestra.run.smithi042.stdout: python-sklearn-lib python-sqlalchemy python-sqlalchemy-ext python-tempita 2019-12-05T06:59:39.094 INFO:teuthology.orchestra.run.smithi042.stdout: python-uritemplate python-urllib3 python-waitress python-webencodings 2019-12-05T06:59:39.094 INFO:teuthology.orchestra.run.smithi042.stdout: python-webob python-websocket python-webtest python-werkzeug python-yaml 2019-12-05T06:59:39.095 INFO:teuthology.orchestra.run.smithi042.stdout: python3-ceph-argparse python3-cmd2 python3-colorama python3-pyparsing 2019-12-05T06:59:39.095 INFO:teuthology.orchestra.run.smithi042.stdout: python3-pyperclip seabios sharutils socat xmlstarlet 2019-12-05T06:59:39.095 INFO:teuthology.orchestra.run.smithi042.stdout:Use 'sudo apt autoremove' to remove them. 2019-12-05T06:59:39.125 INFO:teuthology.orchestra.run.smithi042.stdout:0 upgraded, 0 newly installed, 0 to remove and 51 not upgraded. 2019-12-05T06:59:39.125 INFO:teuthology.orchestra.run.smithi042.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2019-12-05T06:59:39.127 INFO:teuthology.orchestra.run.smithi042:> dpkg -l | grep '^.\(U\|H\)R' | awk '{print $2}' | sudo xargs --no-run-if-empty dpkg -P --force-remove-reinstreq 2019-12-05T06:59:39.200 INFO:teuthology.orchestra.run.smithi042:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" autoremove 2019-12-05T06:59:39.228 INFO:teuthology.orchestra.run.smithi040.stdout:Removing gstreamer1.0-plugins-base:amd64 (1.14.5-0ubuntu1~18.04.1) ... 2019-12-05T06:59:39.286 INFO:teuthology.orchestra.run.smithi042.stdout:Reading package lists... 2019-12-05T06:59:39.542 INFO:teuthology.orchestra.run.smithi042.stdout:Building dependency tree... 2019-12-05T06:59:39.542 INFO:teuthology.orchestra.run.smithi042.stdout:Reading state information... 2019-12-05T06:59:39.624 INFO:teuthology.orchestra.run.smithi040.stdout:Removing gstreamer1.0-x:amd64 (1.14.5-0ubuntu1~18.04.1) ... 2019-12-05T06:59:39.739 INFO:teuthology.orchestra.run.smithi042.stdout:The following packages will be REMOVED: 2019-12-05T06:59:39.739 INFO:teuthology.orchestra.run.smithi042.stdout: cpu-checker formencode-i18n gstreamer1.0-plugins-base 2019-12-05T06:59:39.739 INFO:teuthology.orchestra.run.smithi042.stdout: gstreamer1.0-plugins-good gstreamer1.0-x ipxe-qemu 2019-12-05T06:59:39.739 INFO:teuthology.orchestra.run.smithi042.stdout: ipxe-qemu-256k-compat-efi-roms jq libaa1 libavc1394-0 libbluetooth3 2019-12-05T06:59:39.740 INFO:teuthology.orchestra.run.smithi042.stdout: libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1 2019-12-05T06:59:39.740 INFO:teuthology.orchestra.run.smithi042.stdout: libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0 2019-12-05T06:59:39.740 INFO:teuthology.orchestra.run.smithi042.stdout: libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libjq1 2019-12-05T06:59:39.740 INFO:teuthology.orchestra.run.smithi042.stdout: libjs-sphinxdoc libjs-underscore libleveldb1v5 liblttng-ust0 libmp3lame0 2019-12-05T06:59:39.740 INFO:teuthology.orchestra.run.smithi042.stdout: libmpg123-0 liboath0 libonig4 libopus0 liborc-0.4-0 librabbitmq4 2019-12-05T06:59:39.740 INFO:teuthology.orchestra.run.smithi042.stdout: libraw1394-11 librdkafka1 libsamplerate0 libshout3 libsnappy1v5 libspeex1 2019-12-05T06:59:39.740 INFO:teuthology.orchestra.run.smithi042.stdout: libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0 libusbredirparser1 2019-12-05T06:59:39.741 INFO:teuthology.orchestra.run.smithi042.stdout: libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5 libwavpack1 libxen-4.9 2019-12-05T06:59:39.741 INFO:teuthology.orchestra.run.smithi042.stdout: libxenstore3.0 libxslt1.1 libyajl2 msr-tools python-attr python-bcrypt 2019-12-05T06:59:39.741 INFO:teuthology.orchestra.run.smithi042.stdout: python-bs4 python-ceph-argparse python-ceph-common python-certifi 2019-12-05T06:59:39.741 INFO:teuthology.orchestra.run.smithi042.stdout: python-chardet python-cherrypy3 python-dnspython python-fasteners 2019-12-05T06:59:39.741 INFO:teuthology.orchestra.run.smithi042.stdout: python-formencode python-funcsigs python-html5lib python-httplib2 2019-12-05T06:59:39.741 INFO:teuthology.orchestra.run.smithi042.stdout: python-joblib python-jwt python-kubernetes python-logutils python-lxml 2019-12-05T06:59:39.741 INFO:teuthology.orchestra.run.smithi042.stdout: python-mako python-markupsafe python-monotonic python-oauth2client 2019-12-05T06:59:39.742 INFO:teuthology.orchestra.run.smithi042.stdout: python-openid python-paste python-pastedeploy python-pastedeploy-tpl 2019-12-05T06:59:39.742 INFO:teuthology.orchestra.run.smithi042.stdout: python-pastescript python-pecan python-pluggy python-prettytable python-py 2019-12-05T06:59:39.742 INFO:teuthology.orchestra.run.smithi042.stdout: python-pyasn1 python-pyasn1-modules python-pyinotify python-pytest 2019-12-05T06:59:39.742 INFO:teuthology.orchestra.run.smithi042.stdout: python-repoze.lru python-requests python-routes python-rsa python-scgi 2019-12-05T06:59:39.742 INFO:teuthology.orchestra.run.smithi042.stdout: python-simplegeneric python-simplejson python-singledispatch python-sklearn 2019-12-05T06:59:39.742 INFO:teuthology.orchestra.run.smithi042.stdout: python-sklearn-lib python-sqlalchemy python-sqlalchemy-ext python-tempita 2019-12-05T06:59:39.742 INFO:teuthology.orchestra.run.smithi042.stdout: python-uritemplate python-urllib3 python-waitress python-webencodings 2019-12-05T06:59:39.743 INFO:teuthology.orchestra.run.smithi042.stdout: python-webob python-websocket python-webtest python-werkzeug python-yaml 2019-12-05T06:59:39.743 INFO:teuthology.orchestra.run.smithi042.stdout: python3-ceph-argparse python3-cmd2 python3-colorama python3-pyparsing 2019-12-05T06:59:39.743 INFO:teuthology.orchestra.run.smithi042.stdout: python3-pyperclip seabios sharutils socat xmlstarlet 2019-12-05T06:59:39.987 INFO:teuthology.orchestra.run.smithi040.stdout:Removing ipxe-qemu (1.0.0+git-20180124.fbe8c52d-0ubuntu2.2) ... 2019-12-05T06:59:40.097 INFO:teuthology.orchestra.run.smithi042.stdout:0 upgraded, 0 newly installed, 122 to remove and 50 not upgraded. 2019-12-05T06:59:40.097 INFO:teuthology.orchestra.run.smithi042.stdout:After this operation, 83.5 MB disk space will be freed. 2019-12-05T06:59:40.227 INFO:teuthology.orchestra.run.smithi042.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 141858 files and directories currently installed.) 2019-12-05T06:59:40.231 INFO:teuthology.orchestra.run.smithi042.stdout:Removing cpu-checker (0.7-0ubuntu7) ... 2019-12-05T06:59:40.348 INFO:teuthology.orchestra.run.smithi040.stdout:Removing ipxe-qemu-256k-compat-efi-roms (1.0.0+git-20150424.a25a16d-0ubuntu2) ... 2019-12-05T06:59:40.639 INFO:teuthology.orchestra.run.smithi042.stdout:Removing python-pecan (1.2.1-2) ... 2019-12-05T06:59:40.734 INFO:teuthology.orchestra.run.smithi040.stdout:Removing jq (1.5+dfsg-2) ... 2019-12-05T06:59:41.040 INFO:teuthology.orchestra.run.smithi042.stdout:Removing python-webtest (2.0.28-1ubuntu1) ... 2019-12-05T06:59:41.079 INFO:teuthology.orchestra.run.smithi040.stdout:Removing libaa1:amd64 (1.4p5-44build2) ... 2019-12-05T06:59:41.513 INFO:teuthology.orchestra.run.smithi042.stdout:Removing python-pastescript (2.0.2-2) ... 2019-12-05T06:59:41.517 INFO:teuthology.orchestra.run.smithi040.stdout:Removing libavc1394-0:amd64 (0.5.4-4build1) ... 2019-12-05T06:59:41.865 INFO:teuthology.orchestra.run.smithi040.stdout:Removing libbluetooth3:amd64 (5.48-0ubuntu3.2) ... 2019-12-05T06:59:42.178 INFO:teuthology.orchestra.run.smithi042.stdout:Removing gstreamer1.0-plugins-good:amd64 (1.14.5-0ubuntu1~18.04.1) ... 2019-12-05T06:59:42.274 INFO:teuthology.orchestra.run.smithi040.stdout:Removing libbrlapi0.6:amd64 (5.5-4ubuntu2.0.1) ... 2019-12-05T06:59:42.627 INFO:teuthology.orchestra.run.smithi040.stdout:Removing libcacard0:amd64 (1:2.5.0-3) ... 2019-12-05T06:59:42.681 INFO:teuthology.orchestra.run.smithi042.stdout:Removing gstreamer1.0-plugins-base:amd64 (1.14.5-0ubuntu1~18.04.1) ... 2019-12-05T06:59:43.007 INFO:teuthology.orchestra.run.smithi040.stdout:Removing libcdparanoia0:amd64 (3.10.2+debian-13) ... 2019-12-05T06:59:43.022 INFO:teuthology.orchestra.run.smithi042.stdout:Removing gstreamer1.0-x:amd64 (1.14.5-0ubuntu1~18.04.1) ... 2019-12-05T06:59:43.393 INFO:teuthology.orchestra.run.smithi040.stdout:Removing libdv4:amd64 (1.0.0-11) ... 2019-12-05T06:59:43.405 INFO:teuthology.orchestra.run.smithi042.stdout:Removing ipxe-qemu (1.0.0+git-20180124.fbe8c52d-0ubuntu2.2) ... 2019-12-05T06:59:43.748 INFO:teuthology.orchestra.run.smithi040.stdout:Removing libfdt1:amd64 (1.4.5-3) ... 2019-12-05T06:59:43.791 INFO:teuthology.orchestra.run.smithi042.stdout:Removing ipxe-qemu-256k-compat-efi-roms (1.0.0+git-20150424.a25a16d-0ubuntu2) ... 2019-12-05T06:59:44.133 INFO:teuthology.orchestra.run.smithi040.stdout:Removing libspice-server1:amd64 (0.14.0-1ubuntu2.4) ... 2019-12-05T06:59:44.152 INFO:teuthology.orchestra.run.smithi042.stdout:Removing jq (1.5+dfsg-2) ... 2019-12-05T06:59:44.487 INFO:teuthology.orchestra.run.smithi040.stdout:Removing libgstreamer-plugins-good1.0-0:amd64 (1.14.5-0ubuntu1~18.04.1) ... 2019-12-05T06:59:44.556 INFO:teuthology.orchestra.run.smithi042.stdout:Removing libaa1:amd64 (1.4p5-44build2) ... 2019-12-05T06:59:44.865 INFO:teuthology.orchestra.run.smithi040.stdout:Removing libgstreamer-plugins-base1.0-0:amd64 (1.14.5-0ubuntu1~18.04.1) ... 2019-12-05T06:59:44.934 INFO:teuthology.orchestra.run.smithi042.stdout:Removing libavc1394-0:amd64 (0.5.4-4build1) ... 2019-12-05T06:59:45.227 INFO:teuthology.orchestra.run.smithi040.stdout:Removing libgstreamer1.0-0:amd64 (1.14.5-0ubuntu1~18.04.1) ... 2019-12-05T06:59:45.328 INFO:teuthology.orchestra.run.smithi042.stdout:Removing libbluetooth3:amd64 (5.48-0ubuntu3.2) ... 2019-12-05T06:59:45.590 INFO:teuthology.orchestra.run.smithi040.stdout:Removing libiec61883-0:amd64 (1.2.0-2) ... 2019-12-05T06:59:45.725 INFO:teuthology.orchestra.run.smithi042.stdout:Removing libbrlapi0.6:amd64 (5.5-4ubuntu2.0.1) ... 2019-12-05T06:59:45.968 INFO:teuthology.orchestra.run.smithi040.stdout:Removing libiscsi7:amd64 (1.17.0-1.1) ... 2019-12-05T06:59:46.111 INFO:teuthology.orchestra.run.smithi042.stdout:Removing libcacard0:amd64 (1:2.5.0-3) ... 2019-12-05T06:59:46.313 INFO:teuthology.orchestra.run.smithi040.stdout:Removing libjack-jackd2-0:amd64 (1.9.12~dfsg-2) ... 2019-12-05T06:59:46.482 INFO:teuthology.orchestra.run.smithi042.stdout:Removing libcdparanoia0:amd64 (3.10.2+debian-13) ... 2019-12-05T06:59:46.700 INFO:teuthology.orchestra.run.smithi040.stdout:Removing libjq1:amd64 (1.5+dfsg-2) ... 2019-12-05T06:59:46.853 INFO:teuthology.orchestra.run.smithi042.stdout:Removing libdv4:amd64 (1.0.0-11) ... 2019-12-05T06:59:47.046 INFO:teuthology.orchestra.run.smithi040.stdout:Removing python-routes (2.4.1-1) ... 2019-12-05T06:59:47.249 INFO:teuthology.orchestra.run.smithi042.stdout:Removing libfdt1:amd64 (1.4.5-3) ... 2019-12-05T06:59:47.497 INFO:teuthology.orchestra.run.smithi040.stdout:Removing libjs-sphinxdoc (1.6.7-1ubuntu1) ... 2019-12-05T06:59:47.612 INFO:teuthology.orchestra.run.smithi042.stdout:Removing libspice-server1:amd64 (0.14.0-1ubuntu2.4) ... 2019-12-05T06:59:47.887 INFO:teuthology.orchestra.run.smithi040.stdout:Removing libjs-underscore (1.8.3~dfsg-1) ... 2019-12-05T06:59:48.057 INFO:teuthology.orchestra.run.smithi042.stdout:Removing libgstreamer-plugins-good1.0-0:amd64 (1.14.5-0ubuntu1~18.04.1) ... 2019-12-05T06:59:48.249 INFO:teuthology.orchestra.run.smithi040.stdout:Removing libleveldb1v5:amd64 (1.20-2) ... 2019-12-05T06:59:48.412 INFO:teuthology.orchestra.run.smithi042.stdout:Removing libgstreamer-plugins-base1.0-0:amd64 (1.14.5-0ubuntu1~18.04.1) ... 2019-12-05T06:59:48.635 INFO:teuthology.orchestra.run.smithi040.stdout:Removing liblttng-ust0:amd64 (2.10.1-1) ... 2019-12-05T06:59:48.791 INFO:teuthology.orchestra.run.smithi042.stdout:Removing libgstreamer1.0-0:amd64 (1.14.5-0ubuntu1~18.04.1) ... 2019-12-05T06:59:48.981 INFO:teuthology.orchestra.run.smithi040.stdout:Removing libmp3lame0:amd64 (3.100-2) ... 2019-12-05T06:59:49.141 INFO:teuthology.orchestra.run.smithi042.stdout:Removing libiec61883-0:amd64 (1.2.0-2) ... 2019-12-05T06:59:49.368 INFO:teuthology.orchestra.run.smithi040.stdout:Removing libmpg123-0:amd64 (1.25.10-1) ... 2019-12-05T06:59:49.531 INFO:teuthology.orchestra.run.smithi042.stdout:Removing libiscsi7:amd64 (1.17.0-1.1) ... 2019-12-05T06:59:49.718 INFO:teuthology.orchestra.run.smithi040.stdout:Removing liboath0 (2.6.1-1) ... 2019-12-05T06:59:50.016 INFO:teuthology.orchestra.run.smithi040.stdout:Removing libonig4:amd64 (6.7.0-1) ... 2019-12-05T06:59:50.350 INFO:teuthology.orchestra.run.smithi042.stdout:Removing libjack-jackd2-0:amd64 (1.9.12~dfsg-2) ... 2019-12-05T06:59:50.363 INFO:teuthology.orchestra.run.smithi040.stdout:Removing libopus0:amd64 (1.1.2-1ubuntu1) ... 2019-12-05T06:59:50.729 INFO:teuthology.orchestra.run.smithi042.stdout:Removing libjq1:amd64 (1.5+dfsg-2) ... 2019-12-05T06:59:50.827 INFO:teuthology.orchestra.run.smithi040.stdout:Removing liborc-0.4-0:amd64 (1:0.4.28-1) ... 2019-12-05T06:59:51.127 INFO:teuthology.orchestra.run.smithi042.stdout:Removing python-routes (2.4.1-1) ... 2019-12-05T06:59:51.172 INFO:teuthology.orchestra.run.smithi040.stdout:Removing librabbitmq4:amd64 (0.8.0-1ubuntu0.18.04.1) ... 2019-12-05T06:59:51.541 INFO:teuthology.orchestra.run.smithi040.stdout:Removing libraw1394-11:amd64 (2.1.2-1) ... 2019-12-05T06:59:51.569 INFO:teuthology.orchestra.run.smithi042.stdout:Removing libjs-sphinxdoc (1.6.7-1ubuntu1) ... 2019-12-05T06:59:51.998 INFO:teuthology.orchestra.run.smithi042.stdout:Removing libjs-underscore (1.8.3~dfsg-1) ... 2019-12-05T06:59:52.243 INFO:teuthology.orchestra.run.smithi040.stdout:Removing librdkafka1:amd64 (0.11.3-1build1) ... 2019-12-05T06:59:52.370 INFO:teuthology.orchestra.run.smithi042.stdout:Removing libleveldb1v5:amd64 (1.20-2) ... 2019-12-05T06:59:52.599 INFO:teuthology.orchestra.run.smithi040.stdout:Removing libsamplerate0:amd64 (0.1.9-1) ... 2019-12-05T06:59:52.768 INFO:teuthology.orchestra.run.smithi042.stdout:Removing liblttng-ust0:amd64 (2.10.1-1) ... 2019-12-05T06:59:52.994 INFO:teuthology.orchestra.run.smithi040.stdout:Removing libshout3:amd64 (2.4.1-2build1) ... 2019-12-05T06:59:53.153 INFO:teuthology.orchestra.run.smithi042.stdout:Removing libmp3lame0:amd64 (3.100-2) ... 2019-12-05T06:59:53.331 INFO:teuthology.orchestra.run.smithi040.stdout:Removing libsnappy1v5:amd64 (1.1.7-1) ... 2019-12-05T06:59:53.583 INFO:teuthology.orchestra.run.smithi042.stdout:Removing libmpg123-0:amd64 (1.25.10-1) ... 2019-12-05T06:59:53.718 INFO:teuthology.orchestra.run.smithi040.stdout:Removing libspeex1:amd64 (1.2~rc1.2-1ubuntu2) ... 2019-12-05T06:59:53.979 INFO:teuthology.orchestra.run.smithi042.stdout:Removing liboath0 (2.6.1-1) ... 2019-12-05T06:59:54.071 INFO:teuthology.orchestra.run.smithi040.stdout:Removing libtag1v5:amd64 (1.11.1+dfsg.1-0.2build2) ... 2019-12-05T06:59:54.306 INFO:teuthology.orchestra.run.smithi042.stdout:Removing libonig4:amd64 (6.7.0-1) ... 2019-12-05T06:59:54.424 INFO:teuthology.orchestra.run.smithi040.stdout:Removing libtag1v5-vanilla:amd64 (1.11.1+dfsg.1-0.2build2) ... 2019-12-05T06:59:54.698 INFO:teuthology.orchestra.run.smithi042.stdout:Removing libopus0:amd64 (1.1.2-1ubuntu1) ... 2019-12-05T06:59:54.812 INFO:teuthology.orchestra.run.smithi040.stdout:Removing libtwolame0:amd64 (0.3.13-3) ... 2019-12-05T06:59:55.148 INFO:teuthology.orchestra.run.smithi040.stdout:Removing libusbredirparser1:amd64 (0.7.1-1) ... 2019-12-05T06:59:55.157 INFO:teuthology.orchestra.run.smithi042.stdout:Removing liborc-0.4-0:amd64 (1:0.4.28-1) ... 2019-12-05T06:59:55.523 INFO:teuthology.orchestra.run.smithi042.stdout:Removing librabbitmq4:amd64 (0.8.0-1ubuntu0.18.04.1) ... 2019-12-05T06:59:55.561 INFO:teuthology.orchestra.run.smithi040.stdout:Removing libv4l-0:amd64 (1.14.2-1) ... 2019-12-05T06:59:55.923 INFO:teuthology.orchestra.run.smithi040.stdout:Removing libv4lconvert0:amd64 (1.14.2-1) ... 2019-12-05T06:59:55.941 INFO:teuthology.orchestra.run.smithi042.stdout:Removing libraw1394-11:amd64 (2.1.2-1) ... 2019-12-05T06:59:56.283 INFO:teuthology.orchestra.run.smithi040.stdout:Removing libvisual-0.4-0:amd64 (0.4.0-11) ... 2019-12-05T06:59:56.298 INFO:teuthology.orchestra.run.smithi042.stdout:Removing librdkafka1:amd64 (0.11.3-1build1) ... 2019-12-05T06:59:56.679 INFO:teuthology.orchestra.run.smithi040.stdout:Removing libvpx5:amd64 (1.7.0-3) ... 2019-12-05T06:59:56.686 INFO:teuthology.orchestra.run.smithi042.stdout:Removing libsamplerate0:amd64 (0.1.9-1) ... 2019-12-05T06:59:57.039 INFO:teuthology.orchestra.run.smithi042.stdout:Removing libshout3:amd64 (2.4.1-2build1) ... 2019-12-05T06:59:57.183 INFO:teuthology.orchestra.run.smithi040.stdout:Removing libwavpack1:amd64 (5.1.0-2ubuntu1.4) ... 2019-12-05T06:59:57.409 INFO:teuthology.orchestra.run.smithi042.stdout:Removing libsnappy1v5:amd64 (1.1.7-1) ... 2019-12-05T06:59:57.578 INFO:teuthology.orchestra.run.smithi040.stdout:Removing libxen-4.9:amd64 (4.9.2-0ubuntu1) ... 2019-12-05T06:59:57.805 INFO:teuthology.orchestra.run.smithi042.stdout:Removing libspeex1:amd64 (1.2~rc1.2-1ubuntu2) ... 2019-12-05T06:59:57.931 INFO:teuthology.orchestra.run.smithi040.stdout:Removing libxenstore3.0:amd64 (4.9.2-0ubuntu1) ... 2019-12-05T06:59:58.174 INFO:teuthology.orchestra.run.smithi042.stdout:Removing libtag1v5:amd64 (1.11.1+dfsg.1-0.2build2) ... 2019-12-05T06:59:58.318 INFO:teuthology.orchestra.run.smithi040.stdout:Removing python-lxml:amd64 (4.2.1-1ubuntu0.1) ... 2019-12-05T06:59:58.555 INFO:teuthology.orchestra.run.smithi042.stdout:Removing libtag1v5-vanilla:amd64 (1.11.1+dfsg.1-0.2build2) ... 2019-12-05T06:59:58.878 INFO:teuthology.orchestra.run.smithi040.stdout:Removing xmlstarlet (1.6.1-2) ... 2019-12-05T06:59:58.931 INFO:teuthology.orchestra.run.smithi042.stdout:Removing libtwolame0:amd64 (0.3.13-3) ... 2019-12-05T06:59:59.259 INFO:teuthology.orchestra.run.smithi040.stdout:Removing libxslt1.1:amd64 (1.1.29-5ubuntu0.2) ... 2019-12-05T06:59:59.302 INFO:teuthology.orchestra.run.smithi042.stdout:Removing libusbredirparser1:amd64 (0.7.1-1) ... 2019-12-05T06:59:59.646 INFO:teuthology.orchestra.run.smithi040.stdout:Removing libyajl2:amd64 (2.1.0-2build1) ... 2019-12-05T06:59:59.664 INFO:teuthology.orchestra.run.smithi042.stdout:Removing libv4l-0:amd64 (1.14.2-1) ... 2019-12-05T06:59:59.995 INFO:teuthology.orchestra.run.smithi040.stdout:Removing msr-tools (1.3-2build1) ... 2019-12-05T07:00:00.067 INFO:teuthology.orchestra.run.smithi042.stdout:Removing libv4lconvert0:amd64 (1.14.2-1) ... 2019-12-05T07:00:00.379 INFO:teuthology.orchestra.run.smithi040.stdout:Removing python-pytest (3.3.2-2) ... 2019-12-05T07:00:00.471 INFO:teuthology.orchestra.run.smithi042.stdout:Removing libvisual-0.4-0:amd64 (0.4.0-11) ... 2019-12-05T07:00:00.800 INFO:teuthology.orchestra.run.smithi040.stdout:Removing python-attr (17.4.0-2) ... 2019-12-05T07:00:00.833 INFO:teuthology.orchestra.run.smithi042.stdout:Removing libvpx5:amd64 (1.7.0-3) ... 2019-12-05T07:00:01.234 INFO:teuthology.orchestra.run.smithi042.stdout:Removing libwavpack1:amd64 (5.1.0-2ubuntu1.4) ... 2019-12-05T07:00:01.252 INFO:teuthology.orchestra.run.smithi040.stdout:Removing python-bcrypt (3.1.4-2) ... 2019-12-05T07:00:01.642 INFO:teuthology.orchestra.run.smithi042.stdout:Removing libxen-4.9:amd64 (4.9.2-0ubuntu1) ... 2019-12-05T07:00:01.670 INFO:teuthology.orchestra.run.smithi040.stdout:Removing python-bs4 (4.6.0-1) ... 2019-12-05T07:00:02.063 INFO:teuthology.orchestra.run.smithi042.stdout:Removing libxenstore3.0:amd64 (4.9.2-0ubuntu1) ... 2019-12-05T07:00:02.128 INFO:teuthology.orchestra.run.smithi040.stdout:Removing python-ceph-argparse (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T07:00:02.425 INFO:teuthology.orchestra.run.smithi042.stdout:Removing python-lxml:amd64 (4.2.1-1ubuntu0.1) ... 2019-12-05T07:00:02.506 INFO:teuthology.orchestra.run.smithi040.stdout:Removing python-ceph-common (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T07:00:02.851 INFO:teuthology.orchestra.run.smithi040.stdout:Removing python-requests (2.18.4-2ubuntu0.1) ... 2019-12-05T07:00:02.902 INFO:teuthology.orchestra.run.smithi042.stdout:Removing xmlstarlet (1.6.1-2) ... 2019-12-05T07:00:03.275 INFO:teuthology.orchestra.run.smithi042.stdout:Removing libxslt1.1:amd64 (1.1.29-5ubuntu0.2) ... 2019-12-05T07:00:03.312 INFO:teuthology.orchestra.run.smithi040.stdout:Removing python-kubernetes (2.0.0-2ubuntu1) ... 2019-12-05T07:00:03.670 INFO:teuthology.orchestra.run.smithi042.stdout:Removing libyajl2:amd64 (2.1.0-2build1) ... 2019-12-05T07:00:03.793 INFO:teuthology.orchestra.run.smithi040.stdout:Removing python-certifi (2018.1.18-2) ... 2019-12-05T07:00:04.066 INFO:teuthology.orchestra.run.smithi042.stdout:Removing msr-tools (1.3-2build1) ... 2019-12-05T07:00:04.289 INFO:teuthology.orchestra.run.smithi040.stdout:Removing python-chardet (3.0.4-1) ... 2019-12-05T07:00:04.453 INFO:teuthology.orchestra.run.smithi042.stdout:Removing python-pytest (3.3.2-2) ... 2019-12-05T07:00:04.723 INFO:teuthology.orchestra.run.smithi040.stdout:Removing python-cherrypy3 (8.9.1-2) ... 2019-12-05T07:00:04.991 INFO:teuthology.orchestra.run.smithi042.stdout:Removing python-attr (17.4.0-2) ... 2019-12-05T07:00:05.286 INFO:teuthology.orchestra.run.smithi040.stdout:Removing python-oauth2client (4.1.2-3) ... 2019-12-05T07:00:05.469 INFO:teuthology.orchestra.run.smithi042.stdout:Removing python-bcrypt (3.1.4-2) ... 2019-12-05T07:00:05.816 INFO:teuthology.orchestra.run.smithi040.stdout:Removing python-fasteners (0.12.0-3) ... 2019-12-05T07:00:06.065 INFO:teuthology.orchestra.run.smithi042.stdout:Removing python-bs4 (4.6.0-1) ... 2019-12-05T07:00:06.287 INFO:teuthology.orchestra.run.smithi040.stdout:Removing python-funcsigs (1.0.2-4) ... 2019-12-05T07:00:06.534 INFO:teuthology.orchestra.run.smithi042.stdout:Removing python-ceph-argparse (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T07:00:06.699 INFO:teuthology.orchestra.run.smithi040.stdout:Removing python-html5lib (0.999999999-1) ... 2019-12-05T07:00:06.882 INFO:teuthology.orchestra.run.smithi042.stdout:Removing python-ceph-common (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T07:00:07.152 INFO:teuthology.orchestra.run.smithi040.stdout:Removing python-httplib2 (0.9.2+dfsg-1ubuntu0.1) ... 2019-12-05T07:00:07.225 INFO:teuthology.orchestra.run.smithi042.stdout:Removing python-requests (2.18.4-2ubuntu0.1) ... 2019-12-05T07:00:07.811 INFO:teuthology.orchestra.run.smithi042.stdout:Removing python-kubernetes (2.0.0-2ubuntu1) ... 2019-12-05T07:00:07.899 INFO:teuthology.orchestra.run.smithi040.stdout:Removing python-sklearn (0.19.1-3) ... 2019-12-05T07:00:08.363 INFO:teuthology.orchestra.run.smithi040.stdout:Removing python-joblib (0.11-1) ... 2019-12-05T07:00:08.461 INFO:teuthology.orchestra.run.smithi042.stdout:Removing python-certifi (2018.1.18-2) ... 2019-12-05T07:00:08.808 INFO:teuthology.orchestra.run.smithi040.stdout:Removing python-jwt (1.5.3+ds1-1) ... 2019-12-05T07:00:08.902 INFO:teuthology.orchestra.run.smithi042.stdout:Removing python-chardet (3.0.4-1) ... 2019-12-05T07:00:09.271 INFO:teuthology.orchestra.run.smithi040.stdout:Removing python-logutils (0.3.3-5) ... 2019-12-05T07:00:09.360 INFO:teuthology.orchestra.run.smithi042.stdout:Removing python-cherrypy3 (8.9.1-2) ... 2019-12-05T07:00:09.700 INFO:teuthology.orchestra.run.smithi040.stdout:Removing python-mako (1.0.7+ds1-1) ... 2019-12-05T07:00:09.914 INFO:teuthology.orchestra.run.smithi042.stdout:Removing python-oauth2client (4.1.2-3) ... 2019-12-05T07:00:10.165 INFO:teuthology.orchestra.run.smithi040.stdout:Removing python-markupsafe (1.0-1build1) ... 2019-12-05T07:00:10.581 INFO:teuthology.orchestra.run.smithi040.stdout:Removing python-monotonic (1.1-2) ... 2019-12-05T07:00:10.965 INFO:teuthology.orchestra.run.smithi042.stdout:Removing python-fasteners (0.12.0-3) ... 2019-12-05T07:00:11.036 INFO:teuthology.orchestra.run.smithi040.stdout:Removing python-openid (2.2.5-6) ... 2019-12-05T07:00:11.426 INFO:teuthology.orchestra.run.smithi042.stdout:Removing python-funcsigs (1.0.2-4) ... 2019-12-05T07:00:11.456 INFO:teuthology.orchestra.run.smithi040.stdout:Removing python-pastedeploy (1.5.2-4) ... 2019-12-05T07:00:11.913 INFO:teuthology.orchestra.run.smithi040.stdout:Removing python-pastedeploy-tpl (1.5.2-4) ... 2019-12-05T07:00:11.969 INFO:teuthology.orchestra.run.smithi042.stdout:Removing python-html5lib (0.999999999-1) ... 2019-12-05T07:00:12.274 INFO:teuthology.orchestra.run.smithi040.stdout:Removing python-pluggy (0.6.0-1) ... 2019-12-05T07:00:12.445 INFO:teuthology.orchestra.run.smithi042.stdout:Removing python-httplib2 (0.9.2+dfsg-1ubuntu0.1) ... 2019-12-05T07:00:12.716 INFO:teuthology.orchestra.run.smithi040.stdout:Removing python-prettytable (0.7.2-3) ... 2019-12-05T07:00:13.013 INFO:teuthology.orchestra.run.smithi042.stdout:Removing python-sklearn (0.19.1-3) ... 2019-12-05T07:00:13.172 INFO:teuthology.orchestra.run.smithi040.stdout:Removing python-py (1.5.2-1) ... 2019-12-05T07:00:13.488 INFO:teuthology.orchestra.run.smithi042.stdout:Removing python-joblib (0.11-1) ... 2019-12-05T07:00:13.608 INFO:teuthology.orchestra.run.smithi040.stdout:Removing python-rsa (3.4.2-1) ... 2019-12-05T07:00:13.984 INFO:teuthology.orchestra.run.smithi042.stdout:Removing python-jwt (1.5.3+ds1-1) ... 2019-12-05T07:00:14.031 INFO:teuthology.orchestra.run.smithi040.stdout:Removing python-pyasn1-modules (0.2.1-0.2) ... 2019-12-05T07:00:14.422 INFO:teuthology.orchestra.run.smithi042.stdout:Removing python-logutils (0.3.3-5) ... 2019-12-05T07:00:14.499 INFO:teuthology.orchestra.run.smithi040.stdout:Removing python-pyasn1 (0.4.2-3) ... 2019-12-05T07:00:14.928 INFO:teuthology.orchestra.run.smithi040.stdout:Removing python-pyinotify (0.9.6-1) ... 2019-12-05T07:00:14.987 INFO:teuthology.orchestra.run.smithi042.stdout:Removing python-mako (1.0.7+ds1-1) ... 2019-12-05T07:00:15.373 INFO:teuthology.orchestra.run.smithi040.stdout:Removing python-repoze.lru (0.7-1) ... 2019-12-05T07:00:15.466 INFO:teuthology.orchestra.run.smithi042.stdout:Removing python-markupsafe (1.0-1build1) ... 2019-12-05T07:00:15.886 INFO:teuthology.orchestra.run.smithi040.stdout:Removing python-scgi (1.13-1.1build1) ... 2019-12-05T07:00:15.912 INFO:teuthology.orchestra.run.smithi042.stdout:Removing python-monotonic (1.1-2) ... 2019-12-05T07:00:16.340 INFO:teuthology.orchestra.run.smithi040.stdout:Removing python-simplegeneric (0.8.1-1) ... 2019-12-05T07:00:16.404 INFO:teuthology.orchestra.run.smithi042.stdout:Removing python-openid (2.2.5-6) ... 2019-12-05T07:00:16.804 INFO:teuthology.orchestra.run.smithi040.stdout:Removing python-webob (1:1.7.3-2fakesync1) ... 2019-12-05T07:00:16.969 INFO:teuthology.orchestra.run.smithi042.stdout:Removing python-pastedeploy (1.5.2-4) ... 2019-12-05T07:00:17.259 INFO:teuthology.orchestra.run.smithi040.stdout:Removing python-simplejson (3.13.2-1) ... 2019-12-05T07:00:17.554 INFO:teuthology.orchestra.run.smithi042.stdout:Removing python-pastedeploy-tpl (1.5.2-4) ... 2019-12-05T07:00:17.704 INFO:teuthology.orchestra.run.smithi040.stdout:Removing python-singledispatch (3.4.0.3-2) ... 2019-12-05T07:00:17.918 INFO:teuthology.orchestra.run.smithi042.stdout:Removing python-pluggy (0.6.0-1) ... 2019-12-05T07:00:18.160 INFO:teuthology.orchestra.run.smithi040.stdout:Removing python-sklearn-lib (0.19.1-3) ... 2019-12-05T07:00:18.429 INFO:teuthology.orchestra.run.smithi042.stdout:Removing python-prettytable (0.7.2-3) ... 2019-12-05T07:00:18.523 INFO:teuthology.orchestra.run.smithi040.stdout:Removing python-sqlalchemy-ext (1.1.11+ds1-1ubuntu1) ... 2019-12-05T07:00:18.858 INFO:teuthology.orchestra.run.smithi042.stdout:Removing python-py (1.5.2-1) ... 2019-12-05T07:00:18.894 INFO:teuthology.orchestra.run.smithi040.stdout:Removing python-sqlalchemy (1.1.11+ds1-1ubuntu1) ... 2019-12-05T07:00:19.316 INFO:teuthology.orchestra.run.smithi042.stdout:Removing python-rsa (3.4.2-1) ... 2019-12-05T07:00:19.418 INFO:teuthology.orchestra.run.smithi040.stdout:Removing python-uritemplate (0.6-1ubuntu1) ... 2019-12-05T07:00:19.858 INFO:teuthology.orchestra.run.smithi042.stdout:Removing python-pyasn1-modules (0.2.1-0.2) ... 2019-12-05T07:00:19.874 INFO:teuthology.orchestra.run.smithi040.stdout:Removing python-urllib3 (1.22-1ubuntu0.18.04.1) ... 2019-12-05T07:00:20.330 INFO:teuthology.orchestra.run.smithi040.stdout:Removing python-waitress (1.0.1-1) ... 2019-12-05T07:00:20.355 INFO:teuthology.orchestra.run.smithi042.stdout:Removing python-pyasn1 (0.4.2-3) ... 2019-12-05T07:00:20.768 INFO:teuthology.orchestra.run.smithi040.stdout:Removing python-webencodings (0.5-2) ... 2019-12-05T07:00:21.093 INFO:teuthology.orchestra.run.smithi042.stdout:Removing python-pyinotify (0.9.6-1) ... 2019-12-05T07:00:21.220 INFO:teuthology.orchestra.run.smithi040.stdout:Removing python-websocket (0.44.0-0ubuntu2) ... 2019-12-05T07:00:21.553 INFO:teuthology.orchestra.run.smithi042.stdout:Removing python-repoze.lru (0.7-1) ... 2019-12-05T07:00:21.586 INFO:teuthology.orchestra.run.smithi040.stdout:Removing python-werkzeug (0.14.1+dfsg1-1) ... 2019-12-05T07:00:22.082 INFO:teuthology.orchestra.run.smithi042.stdout:Removing python-scgi (1.13-1.1build1) ... 2019-12-05T07:00:22.127 INFO:teuthology.orchestra.run.smithi040.stdout:Removing python-yaml (3.12-1build2) ... 2019-12-05T07:00:22.600 INFO:teuthology.orchestra.run.smithi040.stdout:Removing python3-ceph-argparse (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T07:00:22.655 INFO:teuthology.orchestra.run.smithi042.stdout:Removing python-simplegeneric (0.8.1-1) ... 2019-12-05T07:00:23.134 INFO:teuthology.orchestra.run.smithi042.stdout:Removing python-webob (1:1.7.3-2fakesync1) ... 2019-12-05T07:00:23.292 INFO:teuthology.orchestra.run.smithi040.stdout:Removing python3-cmd2 (0.7.9-0ubuntu1) ... 2019-12-05T07:00:23.603 INFO:teuthology.orchestra.run.smithi042.stdout:Removing python-simplejson (3.13.2-1) ... 2019-12-05T07:00:23.792 INFO:teuthology.orchestra.run.smithi040.stdout:Removing python3-colorama (0.3.7-1) ... 2019-12-05T07:00:24.198 INFO:teuthology.orchestra.run.smithi042.stdout:Removing python-singledispatch (3.4.0.3-2) ... 2019-12-05T07:00:24.273 INFO:teuthology.orchestra.run.smithi040.stdout:Removing python3-pyparsing (2.2.0+dfsg1-2) ... 2019-12-05T07:00:24.653 INFO:teuthology.orchestra.run.smithi042.stdout:Removing python-sklearn-lib (0.19.1-3) ... 2019-12-05T07:00:24.724 INFO:teuthology.orchestra.run.smithi040.stdout:Removing python3-pyperclip (1.6.0-1) ... 2019-12-05T07:00:25.041 INFO:teuthology.orchestra.run.smithi042.stdout:Removing python-sqlalchemy-ext (1.1.11+ds1-1ubuntu1) ... 2019-12-05T07:00:25.194 INFO:teuthology.orchestra.run.smithi040.stdout:Removing seabios (1.10.2-1ubuntu1) ... 2019-12-05T07:00:25.386 INFO:teuthology.orchestra.run.smithi042.stdout:Removing python-sqlalchemy (1.1.11+ds1-1ubuntu1) ... 2019-12-05T07:00:25.549 INFO:teuthology.orchestra.run.smithi040.stdout:Removing sharutils (1:4.15.2-3) ... 2019-12-05T07:00:25.914 INFO:teuthology.orchestra.run.smithi042.stdout:Removing python-uritemplate (0.6-1ubuntu1) ... 2019-12-05T07:00:25.934 INFO:teuthology.orchestra.run.smithi040.stdout:Removing socat (1.7.3.2-2ubuntu2) ... 2019-12-05T07:00:26.322 INFO:teuthology.orchestra.run.smithi040.stdout:Removing python-paste (2.0.3+dfsg-4ubuntu1) ... 2019-12-05T07:00:26.378 INFO:teuthology.orchestra.run.smithi042.stdout:Removing python-urllib3 (1.22-1ubuntu0.18.04.1) ... 2019-12-05T07:00:26.767 INFO:teuthology.orchestra.run.smithi040.stdout:Removing python-formencode (1.3.0-0ubuntu5) ... 2019-12-05T07:00:26.951 INFO:teuthology.orchestra.run.smithi042.stdout:Removing python-waitress (1.0.1-1) ... 2019-12-05T07:00:27.237 INFO:teuthology.orchestra.run.smithi040.stdout:Removing formencode-i18n (1.3.0-0ubuntu5) ... 2019-12-05T07:00:27.411 INFO:teuthology.orchestra.run.smithi042.stdout:Removing python-webencodings (0.5-2) ... 2019-12-05T07:00:27.584 INFO:teuthology.orchestra.run.smithi040.stdout:Removing python-dnspython (1.15.0-1) ... 2019-12-05T07:00:27.874 INFO:teuthology.orchestra.run.smithi042.stdout:Removing python-websocket (0.44.0-0ubuntu2) ... 2019-12-05T07:00:28.036 INFO:teuthology.orchestra.run.smithi040.stdout:Removing python-tempita (0.5.2-2) ... 2019-12-05T07:00:28.261 INFO:teuthology.orchestra.run.smithi042.stdout:Removing python-werkzeug (0.14.1+dfsg1-1) ... 2019-12-05T07:00:28.506 INFO:teuthology.orchestra.run.smithi040.stdout:Processing triggers for libc-bin (2.27-3ubuntu1) ... 2019-12-05T07:00:28.655 INFO:teuthology.orchestra.run.smithi040.stdout:Processing triggers for man-db (2.8.3-2ubuntu0.1) ... 2019-12-05T07:00:28.918 INFO:teuthology.orchestra.run.smithi042.stdout:Removing python-yaml (3.12-1build2) ... 2019-12-05T07:00:29.299 INFO:teuthology.orchestra.run.smithi040.stdout:Processing triggers for install-info (6.5.0.dfsg.1-2) ... 2019-12-05T07:00:29.345 INFO:teuthology.orchestra.run.smithi042.stdout:Removing python3-ceph-argparse (15.0.0-8014-gc529080-1bionic) ... 2019-12-05T07:00:29.822 INFO:teuthology.orchestra.run.smithi042.stdout:Removing python3-cmd2 (0.7.9-0ubuntu1) ... 2019-12-05T07:00:29.849 INFO:teuthology.orchestra.run.smithi040.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2019-12-05T07:00:29.852 DEBUG:teuthology.parallel:result is None 2019-12-05T07:00:30.312 INFO:teuthology.orchestra.run.smithi042.stdout:Removing python3-colorama (0.3.7-1) ... 2019-12-05T07:00:30.889 INFO:teuthology.orchestra.run.smithi042.stdout:Removing python3-pyparsing (2.2.0+dfsg1-2) ... 2019-12-05T07:00:31.346 INFO:teuthology.orchestra.run.smithi042.stdout:Removing python3-pyperclip (1.6.0-1) ... 2019-12-05T07:00:31.814 INFO:teuthology.orchestra.run.smithi042.stdout:Removing seabios (1.10.2-1ubuntu1) ... 2019-12-05T07:00:32.195 INFO:teuthology.orchestra.run.smithi042.stdout:Removing sharutils (1:4.15.2-3) ... 2019-12-05T07:00:32.601 INFO:teuthology.orchestra.run.smithi042.stdout:Removing socat (1.7.3.2-2ubuntu2) ... 2019-12-05T07:00:33.011 INFO:teuthology.orchestra.run.smithi042.stdout:Removing python-paste (2.0.3+dfsg-4ubuntu1) ... 2019-12-05T07:00:33.573 INFO:teuthology.orchestra.run.smithi042.stdout:Removing python-formencode (1.3.0-0ubuntu5) ... 2019-12-05T07:00:34.043 INFO:teuthology.orchestra.run.smithi042.stdout:Removing formencode-i18n (1.3.0-0ubuntu5) ... 2019-12-05T07:00:34.440 INFO:teuthology.orchestra.run.smithi042.stdout:Removing python-dnspython (1.15.0-1) ... 2019-12-05T07:00:34.919 INFO:teuthology.orchestra.run.smithi042.stdout:Removing python-tempita (0.5.2-2) ... 2019-12-05T07:00:35.507 INFO:teuthology.orchestra.run.smithi042.stdout:Processing triggers for libc-bin (2.27-3ubuntu1) ... 2019-12-05T07:00:35.642 INFO:teuthology.orchestra.run.smithi042.stdout:Processing triggers for man-db (2.8.3-2ubuntu0.1) ... 2019-12-05T07:00:36.352 INFO:teuthology.orchestra.run.smithi042.stdout:Processing triggers for install-info (6.5.0.dfsg.1-2) ... 2019-12-05T07:00:37.033 INFO:teuthology.orchestra.run.smithi042.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2019-12-05T07:00:37.035 DEBUG:teuthology.parallel:result is None 2019-12-05T07:00:37.036 INFO:teuthology.task.install:Removing ceph sources lists 2019-12-05T07:00:37.036 INFO:teuthology.orchestra.run.smithi042:> sudo lsb_release -is 2019-12-05T07:00:37.036 INFO:teuthology.orchestra.run.smithi040:> sudo lsb_release -is 2019-12-05T07:00:37.088 INFO:teuthology.orchestra.run.smithi042.stdout:Ubuntu 2019-12-05T07:00:37.090 INFO:teuthology.orchestra.run.smithi040.stdout:Ubuntu 2019-12-05T07:00:37.092 DEBUG:teuthology.misc:System to be installed: Ubuntu 2019-12-05T07:00:37.093 INFO:teuthology.orchestra.run.smithi042:> sudo rm -f /etc/apt/sources.list.d/ceph.list 2019-12-05T07:00:37.095 DEBUG:teuthology.misc:System to be installed: Ubuntu 2019-12-05T07:00:37.096 INFO:teuthology.orchestra.run.smithi040:> sudo rm -f /etc/apt/sources.list.d/ceph.list 2019-12-05T07:00:37.102 INFO:teuthology.orchestra.run.smithi042:> sudo apt-get update 2019-12-05T07:00:37.125 INFO:teuthology.orchestra.run.smithi040:> sudo apt-get update 2019-12-05T07:00:37.419 INFO:teuthology.orchestra.run.smithi040.stdout:Get:1 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB] 2019-12-05T07:00:37.431 INFO:teuthology.orchestra.run.smithi040.stdout:Hit:2 http://archive.ubuntu.com/ubuntu bionic InRelease 2019-12-05T07:00:37.442 INFO:teuthology.orchestra.run.smithi042.stdout:Hit:1 http://archive.ubuntu.com/ubuntu bionic InRelease 2019-12-05T07:00:37.442 INFO:teuthology.orchestra.run.smithi042.stdout:Get:2 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB] 2019-12-05T07:00:37.500 INFO:teuthology.orchestra.run.smithi040.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB] 2019-12-05T07:00:37.508 INFO:teuthology.orchestra.run.smithi042.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB] 2019-12-05T07:00:37.809 INFO:teuthology.orchestra.run.smithi040.stdout:Get:4 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB] 2019-12-05T07:00:37.814 INFO:teuthology.orchestra.run.smithi042.stdout:Get:4 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB] 2019-12-05T07:00:37.936 DEBUG:teuthology.exit:Got signal 15; running 2 handlers... 2019-12-05T07:00:37.937 DEBUG:teuthology.task.console_log:Killing console logger for smithi042 2019-12-05T07:00:37.937 DEBUG:teuthology.task.console_log:Killing console logger for smithi040 2019-12-05T07:00:37.938 DEBUG:teuthology.task.console_log:Killing console logger for smithi042 2019-12-05T07:00:37.938 DEBUG:teuthology.task.console_log:Killing console logger for smithi040 2019-12-05T07:00:37.939 DEBUG:teuthology.exit:Finished running handlers